11institutetext: University of Texas at Austin, Austin TX 78712, USA
11email<EMAIL_ADDRESS>

# Real-time Online Video Detection with Temporal Smoothing Transformers

Yue Zhao 11 0000-0003-2753-5921 Philipp Krähenbühl 11 0000-0002-9846-4369

###### Abstract

Streaming video recognition reasons about objects and their actions in every
frame of a video. A good streaming recognition model captures both long-term
dynamics and short-term changes of video. Unfortunately, in most existing
methods, the computational complexity grows linearly or quadratically with the
length of the considered dynamics. This issue is particularly pronounced in
transformer-based architectures. To address this issue, we reformulate the
cross-attention in a video transformer through the lens of kernel and apply
two kinds of temporal smoothing kernel: A box kernel or a Laplace kernel. The
resulting streaming attention reuses much of the computation from frame to
frame, and only requires a constant time update each frame. Based on this
idea, we build TeSTra, a Temporal Smoothing Transformer, that takes in
arbitrarily long inputs with constant caching and computing overhead.
Specifically, it runs $6\times$ faster than equivalent sliding-window based
transformers with 2,048 frames in a streaming setting. Furthermore, thanks to
the increased temporal span, TeSTra achieves state-of-the-art results on
THUMOS’14 and EPIC-Kitchen-100, two standard online action detection and
action anticipation datasets. A real-time version of TeSTra outperforms all
but one prior approaches on the THUMOS’14 dataset.

###### Keywords:

Online action detection, action anticipation, transformer, temporal smoothing
kernel

##  1 Introduction

The problem of online action detection [10] and anticipation [29] aims to
determine what action is happening or will happen shortly at each time step
without seeing the future. The challenge for online action detection is (1)
how to effectively retain both the long-term trends and short-term cues when
encoding the history and (2) how to efficiently compute at each time step in
the streaming setting when the history gets longer. Recurrent models such as
LSTM [22] and GRU [7] excel at updating the output recurrently but do not
benefit from increasing sequence length due to the training difficulty [50].
Attention-based models [44], like Long Short-Term Transformer (LSTR) [54], are
capable of handling sequences up to 8 minutes long with impressive prediction
results. However, in the streaming setting, the attention computation of the
long-term memory has to be recomputed for each streaming window considered.
Therefore, the computational cost per frame is proportional to the sequence
length.

Figure 1: A comparison of traditional attention computation (left) in
streaming videos and our streaming attention (right). Unlike traditional
approaches, our approach has a constant runtime per frame. Exponential
smoothing attention has a constant memory footprint as well

In this paper, we propose an effective and efficient approach, Temporal
Smoothing Transformers (TeSTra), to encode sufficiently long history with
constant inference cost at each time step. TeSTra relies on an efficient
attention that reuses much of the attention computation between consecutive
frames. We reformulate attention through a kernel perspective [38, 43] and
explore two temporal kernels: a Box kernel and a Laplace kernel. Both kernels
lead to an efficient streaming attention computation. A box kernel results in
a First In First Out (FIFO) attention computation with a constant runtime
update, but linear memory costs. A Laplace kernel results in an exponential
smoothing attention with constant runtime and memory costs. Fig. 1 shows a
comparison of traditional attention for streaming videos and our streaming
attention. Both formulations exploit the fact that in streaming recognition
queries used in cross attention are learned parameters and fixed during
inference. During training, we use windowed attention in its original matrix
multiplication form (with explicitly computed kernels). This allows us to
enjoy all the GPU parallelism of modern transformer training. At test time, we
switch to efficient streaming implementations.

To show the effectiveness of TeSTra, we conduct extensive experiments on
standard benchmarks for online action detection and anticipation, namely
THUMOS’14 [24] and EPIC-Kitchen-100 [9]. TeSTra achieves state-of-the-art
performance on both benchmarks. Running at 142.8 FPS alone, TeSTra can serve
as a building block for streaming video recognition with low latency. When we
include an accelerated optical flow computing method and an image-based
feature extractor, the overall system can run as fast as 41.1 FPS and achieves
67.3% mAP on THUMOS’14, outperforming all but one prior approaches. Code is
publicly available at https://github.com/zhaoyue-zephyrus/TeSTra/.

##  2 Related Work

Online Action Detection and Anticipation. Online action detection [10], also
known as early action detection [21], aims to detect the start of an action in
a video stream as soon as it happens. Much of prior work builds ever longer-
term temporal reasoning using various recurrent units or networks [11, 13,
53]. Xu et al. [53] perform online detection (classification) on current frame
and prediction the near-future actions simultaneously. StartNet [18]
decomposes the online detection into two stages: action classification and
start localization. The recently proposed LSTR [54] enlarges the effective
temporal context to as long as 512 seconds by adopting the highly flexible
cross-attention mechanism in Transformer [44]. However, the induced
computation cost is proportional to the temporal span. In contrast, our
streaming attention incurs the same constant runtime cost independent of
temporal span.

Action anticipation [19], or forecasting [29], aims to predict the action
before it occurs. Vondrick et al. [45] propose to anticipate by regressing the
representations of future frames from past ones. Zeng et al. [58] and
Rhinehart et al. [37] use inverse reinforcement learning to perform
forecasting at multiple levels. For egocentric videos anticipation may
additionally incorporate the camera wearer’s trajectory [35], eye gaze [32],
hand-object interaction [30], and environment affordance [34]. In this paper,
we handle the problem by taking longer history into account, which is a
general approach to both third-person and egocentric videos.

Transformers and its Efficient Variants. Since the Transformer architecture
was introduced in [44], much work has gone into improving the efficiency of
dot-product attention. Low-rank approximation on attention matrix [6, 48]
factorizes the attention matrix into two lower-rank matrices. Different
efficient learnable sparsity patterns, such as locality-sensitive hashing
[28], differentiable sorting [41] or fixed patterns [5, 57], reduce the total
number of attention operations. Query-based cross attention mechanisms
compress longer-term input into a fixed-size representation via memory [36,
31] or recurrence [8]. Based on a kernel-reformation [43], Katharopoulos et
al. [27] propose linear attention by decomposing the kernel function
$\kappa(\mathbf{q}_{m},\mathbf{k}_{n})$ between a query-key pair into a
product between the feature mapping of query and key, i.e.
$\phi(\mathbf{q}_{m})^{\top}\cdot\phi(\mathbf{k}_{n})$. In computer vision,
Transformers are made more efficient by (1) leveraging hierarchy using shifted
local window [33] and pooling attention [14], (2) applying axial attention on
separate dimensions [46], and (3) using asymmetric attention (cross attention)
to squeeze high-dimensional inputs into tighter latent variables [26]. In
speech recognition, transformers are tailored to streaming decoding by
integrating recurrence [61] or memory [52]. In this paper, we follow the
kernel interpretation of Tsai et al. [43], and show how to efficiently update
streaming attention kernels.

Efficient Video Processing. Videos are notoriously expensive to process. TSN
[47] suggests sampling frames sparsely and running 2D CNNs on the selected
frames. MVCNN [59] and CoViAR [51] directly learn video representation from
compressed videos. X3D [15] and CSN [42] reduce computation FLOPs by
leveraging channel-wise separable convolution. However, 3D CNN takes video
clips as input whose span can be $2-3$ seconds, therefore may not be the best
solution in a low-latency application. Par-Inception [3] tackles the latency
issue by introducing depth-parallelism to the vanilla I3D [4] at increased
implementation difficulty. Most of the previous methods focus on trimmed
videos whose duration is often in several seconds while our method focuses on
streaming videos whose length can be as long as hours. However, many of these
3D CNNs may form a good backbone to our system.

##  3 Preliminaries

Attention. The attention mechanism [44] is a weighted addition of the input
features. The weights are guided by the similarities between the key and query
elements on an input sequence:

$\displaystyle\mathrm{Attention}(\mathbf{Q},\mathbf{X})=\mathrm{Softmax}\left(\frac{\mathbf{Q}\mathbf{K}^{\top}}{\sqrt{C}}\right)\cdot\mathbf{V}=\mathrm{Softmax}\left(\frac{\mathbf{Q}\cdot(\mathbf{X}\mathbf{W}_{k})^{\top}}{\sqrt{C}}\right)\cdot\mathbf{X}\mathbf{W}_{v},$
(1)

where $\mathbf{Q}\in\mathcal{R}^{M\times C}$ is a set of $M$ queries,
$\mathbf{X}=\begin{bmatrix}\cdots&\mathbf{x}_{n}&\cdots\end{bmatrix}^{\top}\in\mathcal{R}^{N\times
d}$ is the sequence of $N$ input tokens,
$\mathbf{W}_{k/v}\in\mathcal{R}^{d\times C}$ is the weight to map the input to
key/value vector and $C$ is the feature dimension of
$\mathbf{x}_{n}^{\top}\mathbf{W}_{k}$. For self-attention computes queries
from the inputs sequence $\mathbf{Q}=\mathbf{X}\mathbf{W}_{q}$ ($M=N$ in this
case). Cross-attention uses a queries $\mathbf{Q}$ that do not relate to the
input sequence $\mathbf{X}$ (generally $M\neq N$ in this case). Cross-
attention is commonly used in the encoder-decoder architecture [44]. Cross-
attention with $M\ll N$ is also used to efficiently encode large amounts of
data into a fixed-size representation [26, 54].

Attention as kernels. The distance computation in attention is similar to the
mechanism of kernel learning [38]. Tsai et al. [43] reformulated Eq. (1) from
the perspective of kernels:

$\displaystyle\mathrm{Attention}(\mathbf{q}_{m},\\{\mathbf{x}_{n}\\})=\frac{\sum_{n=1}^{N}\kappa(\mathbf{q}_{m},\mathbf{k}_{n})\mathbf{v}_{n}}{\sum_{n=1}^{N}\kappa(\mathbf{q}_{m},\mathbf{k}_{n})},$
(2)

where
$\kappa(\cdot,\cdot):\mathcal{R}^{C}\times\mathcal{R}^{C}\rightarrow\mathcal{R}^{+}$
is a generalized kernel function, which depicts the similarity between the
pair of input vectors. Eq. (1) is equivalent to Eq. (2) for a kernel
$\kappa(\mathbf{q}_{m},\mathbf{k}_{n})=\exp(\frac{\mathbf{q}_{m}^{\top}\mathbf{k}_{n}}{\sqrt{C}})$.
In the next section, we show that this kernel perspective leads to an
efficient streaming formulation of attention in the context of streaming video
recognition.

##  4 Efficient Attention on streaming input

We use cross-attention to summarize a large stream of past frames into a fixed
size context representation. We use a fixed number learned queries and
variable number of keys and values from past frames as input. See Fig. 1 for
an example.

In streaming tasks, we are constantly receiving input and want to generate the
corresponding output on the fly. Examples include simultaneous interpretation
or online detection in broadcast videos. Let
$\mathbf{x}_{[1:t]}=\\{\mathbf{x}_{1},\mathbf{x}_{2},\cdots,\mathbf{x}_{t}\\}$
denote a sequence of encoded past video frames for the current time-step $t$.
The encoder may use an image-based [20, 25] or short-clip-based [4, 15] CNN.
Top-performing video models [54] summarize large parts of the video through
cross-attention on either the entire sequence, i.e.
$\mathrm{Attention}(\mathbf{q}_{1}\ldots\mathbf{q}_{M},\mathbf{x}_{[1:t]})$ or
a chunk of input by sliding a $N$-sized temporal window, i.e.
$\mathrm{Attention}(\mathbf{q}_{1}\ldots\mathbf{q}_{M},\mathbf{x}_{[t-N+1:t]})$.
Mathematically, this attention operation is captured in Eq. (2). Here, a small
number of queries $\\{\mathbf{q}_{1}\ldots\mathbf{q}_{M}\\}$ summarize a large
temporal context. Queries combine learned parameters
$\\{\boldsymbol{\lambda}_{1}\ldots\boldsymbol{\lambda}_{M}\\}$ with a temporal
embedding $\omega_{t}$ of the current frame:
$\mathbf{q}_{m}=\boldsymbol{\lambda}_{m}+\omega_{t}$. Keys
$\\{\mathbf{k}_{1}\ldots\mathbf{k}_{t}\\}$ combine a frame-level embeddings
$\mathbf{f}_{n}=\mathbf{W}_{k}^{\top}\mathbf{x}_{n}$ with a temporal embedding
$\omega_{t}$: $\mathbf{k}_{n}=\mathbf{f}_{n}+\omega_{n}$. Values
$\\{\mathbf{v}_{1}\ldots\mathbf{v}_{t}\\}$ use the same frame-level features
$\mathbf{v}_{n}=\mathbf{W}_{v}^{\top}\mathbf{x}_{n}$. In this setup, keys and
values of past frames remain unchanged, learned queries are constant during
inference, only the temporal query embedding changes frame to frame. This
changing temporal embedding does change the attention kernel $\kappa$ for each
new frame. This means in a streaming setting, we have no choice but to
recompute the entire attention operation frame after frame. This recomputation
grows linearly with the size $N$ of the temporal context considered. Next, we
show how a reformulation of the attention mechanism leads to a much more
efficient streaming evaluation.

Streaming Attention. Note, that both queries and keys combine a temporal and
feature-level embedding in their distance kernel
$\kappa(\mathbf{q}_{m},\mathbf{k}_{n})=\kappa(\boldsymbol{\lambda}_{m}+\omega_{t},\mathbf{f}_{n}+\omega_{n})$.
In Streaming Attention, we simple split this kernel into temporal and feature
component:
$K(\omega_{t},\omega_{n})\kappa(\boldsymbol{\lambda}_{m},\mathbf{f}_{n})$. The
Streaming Attention operation reduces to

$\displaystyle\mathrm{Stream}\mathchar
45\relax\mathrm{Attention}(\mathbf{q}_{m},\mathbf{x}_{[1:t]})=\frac{\sum_{n=1}^{t}K(\omega_{t},\omega_{n})\kappa(\boldsymbol{\lambda}_{m},\mathbf{f}_{n})\mathbf{v}_{n}}{\sum_{n=1}^{t}K(\omega_{t},\omega_{n})\kappa(\boldsymbol{\lambda}_{m},\mathbf{f}_{n})}.$
(3)

Most of the features and kernels used in this attention block remain constant
throughout the streaming setting. Moving from timestep $t$ to $t+1$ only
changes the temporal kernel $K(\omega_{t},\omega_{n})$ to
$K(\omega_{t+1},\omega_{n})$ and adds one more element
$(\mathbf{f}_{t+1},\mathbf{v}_{t+1})$. Because of the change in the temporal
kernel, a naive evaluation of streaming attention (3) still requires a linear
runtime in the size of the temporal context. However, the right choice of a
temporal kernel can alleviate this. Here, we explore two kernels: A box (or
uniform) kernel $K_{B}(\omega_{t},\omega_{n})=1_{[t-n<N]}$ and a Laplace
kernel $K_{L}(\omega_{t},\omega_{n})=\exp(-\lambda(t-n))$ for $\lambda>0$.
Each of these kernels leads to an efficient streaming attention mechanism. A
box kernel results in first-in-first-out (FIFO) attentionw while a Laplace
kernel leads to exponential smoothing attention. Fig. 2 provides an overview
of both kernels.

(a) Box kernel

(b) Laplace kernel

Figure 2: A visualization of a box kernel (a) and Laplace kernel (b) and their
streaming computation

FIFO Attention. Let us define the numerator and denominator of Eq. (3) to be
two intermediate variables

$\displaystyle\mathrm{Stream}\mathchar
45\relax\mathrm{Attention}(\mathbf{q}_{m},\mathbf{x}_{[1:t]})=\frac{\boldsymbol{\phi}(t)}{\boldsymbol{\psi}(t)}.$
(4)

Both
$\boldsymbol{\phi}(t)\\!=\\!\sum_{n=1}^{t}\\!K_{B}(\omega_{t},\omega_{n})\kappa(\boldsymbol{\lambda}_{m},\mathbf{f}_{n})\mathbf{v}_{n}$
and
$\boldsymbol{\psi}(t)\\!=\\!\sum_{n=1}^{t}\\!K_{B}(\omega_{t},\omega_{n})\kappa(\boldsymbol{\lambda}_{m},\mathbf{f}_{n})$
are updated by the following recursion as the streaming attention progresses:

$\displaystyle\begin{split}\boldsymbol{\phi}(t+1)&=\boldsymbol{\phi}(t)+\kappa(\boldsymbol{\lambda}_{m},\mathbf{f}_{t})\mathbf{v}_{t}-\kappa(\boldsymbol{\lambda}_{m},\mathbf{f}_{t-N})\mathbf{v}_{t-N}\\\
\boldsymbol{\psi}(t+1)&=\boldsymbol{\psi}(t)+\kappa(\boldsymbol{\lambda}_{m},\mathbf{f}_{t})\mathbf{v}_{t}-\kappa(\boldsymbol{\lambda}_{m},\mathbf{f}_{t-N}),\end{split}$
(5)

where $\kappa(\boldsymbol{\lambda}_{m},\mathbf{f}_{t-N})=0$ and
$\mathbf{v}_{t-N}=0$ for $t\leq N$, $\boldsymbol{\phi}(0)=0$ and
$\boldsymbol{\psi}(0)=0$.

Like a FIFO queue, we keep track of $\boldsymbol{\phi}(t)$ and
$\boldsymbol{\psi}(t)$ and update them by subtracting the quantity contributed
by the input at time $(t-N)$ and adding up the one at time $t$ in the long
run. Therefore, we call this formulation FIFO-Attention. The advantage of
FIFO-Attention is that the computational cost becomes $O(MC)$ for $M$ queries
and values of $C$ channels. Neither the effective window size $N$ nor the
actual time-step $t$ influences the runtime. However, the subtraction
operation in Eq. (5) requires us to keep a window of features and kernel
values in memory. Hence, the memory complexity is still $O(N)$. The Laplace
kernel addresses this issue.

Exponential Smoothing Attention. The Laplace kernel $K_{L}$ allows for an even
more efficient recursive update:

$\displaystyle\begin{split}\tilde{\boldsymbol{\phi}}(t)&=e^{-\lambda}\tilde{\boldsymbol{\phi}}(t-1)+\kappa(\boldsymbol{\lambda}_{m},\mathbf{f}_{t})\mathbf{v}_{t}\\\
\tilde{\boldsymbol{\psi}}(t)&=e^{-\lambda}\tilde{\boldsymbol{\psi}}(t-1)+\kappa(\boldsymbol{\lambda}_{m},\mathbf{f}_{t}),\end{split}$
(6)

where $\tilde{\boldsymbol{\phi}}(0)=0$ and $\tilde{\boldsymbol{\psi}}(0)=0$.
The parameters $\lambda$ controls the temporal extent of the attention. The
above operation (6) is known as exponential smoothing [23]. Therefore we name
this attention Exponential Smoothing Attention, or ES-Attention for short.
Both ES- and FIFO-Attention reduce to the same operation if $\lambda=0$ and
the windows size $N\to\infty$. The time complexity of ES-Attention is also
constant in the temporal window considered $O(MC)$. More importantly, the
space complexity reduces from $O(N)$ to $O(1)$ since we only maintain
$\tilde{\boldsymbol{\psi}}$, $\tilde{\boldsymbol{\phi}}$ and no longer keep
values in our window around. Exponential smoothing instead slowly reduces the
influence of older keys and values in the attention.

Figure 3:  Overview of our streaming attention architecture TeSTra. The basic
setup follows LSTR [54]: A long-term memory compresses a long temporal history
into $M$ representative queries. A short-term attention mechanism uses the
compressed memory and a short history of frames to compute current and future
actions. The main advantage of TeSTra is that the long-memory incurs only
constant cost, and thus allows for much more efficient long-term reasoning

Video recognition with streaming attention. The streaming attention can
replace the vanilla cross-attention in current Transformer architectures with
minimal modification. Specifically, we follow LSTR architecture [54] for all
our experiments, due to its state-of-the-art performance on online action
detection. The overall architecture of TeSTra is sketched in Fig. 3. Given a
sequence of encoded vectors
$\mathbf{x}_{[1:t]}=\\{\mathbf{x}_{1},\mathbf{x}_{2},\cdots,\mathbf{x}_{t}\\}$,
where $t$ refers to the current time stamp, we divide the historic frames into
two parts: short-term memory $\mathbf{x}_{[t-L+1:t]}$ if size $L\leq 32$ and
long-term memory which contains the rest of distant inputs, namely
$\mathbf{x}_{[1:t-L]}$. The architecture follows an encoder-decoder [44, 54]
design. The encoder module encodes the long-term memory into $M=16$ query
features. The decoder uses the query features and short-term memory to predict
current and anticipated actions.

Figure 4: The basic building blocks of TeSTra. Left: the Transformer Encoder
with ES-Attention; Middle: Multi-head ES-Attention at training time; Right:
Multi-head ES-Attention at inference time

The encoder has two stages of memory compression. First, it uses an ES-
Attention-based Transformer decoder unit [44] to compress the long-term memory
into $M$ latent vectors $\mathbf{Z}$ using learnable queries $\mathbf{Q}$.

$\displaystyle\begin{split}\mathbf{Q}^{\prime}&=\mathrm{Attention}(\mathbf{Q},\mathbf{Q}),\\\
\mathbf{Z}^{\prime}&=\mathrm{ES}\mathchar
45\relax\mathrm{Attention}(\sigma(\mathbf{Q}^{\prime}),\mathbf{X}_{[1:t-L]}),\\\
\mathbf{Z}&=\mathrm{FFN}(\sigma(\mathbf{Z}^{\prime})),\end{split}$ (7)

where $\sigma$ denotes the nonlinear mapping which is composed of a skip
connection with $\mathbf{Q}$ followed by a LayerNorm [1]. Next, the compressed
vectors are further cross-attended by $M^{\prime}$ learnable queries through
$\ell_{enc}$ decoder units into $\mathbf{Z}_{\ell_{enc}}$. Strictly speaking,
it should be possible to learn $\mathbf{Q}^{\prime}$ directly. However, the
training dynamics of transformer work out better using a self-attention block
first. Fig. 4 shows an overview of the encoder.

The decoder uses the short-term memory as queries to attend the compressed
memory and retrieve relevant information through a stack of $\ell_{dec}$
decoder units.

$\displaystyle\begin{split}\mathbf{X}_{[t-L+1:t]}^{\prime}&=\mathrm{Attention}(\mathbf{X}_{[t-L+1:t]},\mathbf{X}_{[t-L+1:t]}),\\\
\mathbf{O}^{\prime}&=\mathrm{Attention}(\mathbf{X}_{[t-L+1:t]}^{\prime},\left[\mathbf{Z}_{\ell_{enc}}\mathbin{\|}\mathbf{X}_{[t-L+1:t]}\right]),\\\
\mathbf{O}&=\mathrm{FFN}(\sigma(\mathbf{O}^{\prime})),\end{split}$ (8)

In Eq. (8), we construct the key/value tokens by concatenating
$\left[\cdot\mathbin{\|}\cdot\right]$ both the compressed long-term memory and
short-term memory to incorporate all the known historic information. This
proves to be effective for action anticipation, where the closer memory is
more important to indicate the upcoming action. The $L$ output vectors are
then passed through a linear layer to produces the scores
$\mathbf{s}_{[t-L+1:t]}\in\mathcal{R}^{L\times(K+1)}$ over $K$ action classes
plus one non-action (background) class 111$\mathbf{s}_{t}\in\mathcal{R}^{K}$
if the background class is absent.. At inference time, we take the score
$\mathbf{s}_{t}$ to be online detection result. In action anticipation, the
frames in the anticipating duration are not observable. We thus attach $L_{a}$
learnable tokens after short-term memory predict $L_{a}$ anticipated actions
$\mathbf{s}_{[t+1:t+L_{a}]}$.

Training TeSTra. At inference time, we naturally apply the recursion in Eq.
(6) in the streaming setting. During training, however, it is computationally
inefficient to feed all historic inputs and update them recursively on a
modern GPU architecture. To handle this, we cut the video into a clip
$\mathbf{x}_{t-L-N+1:t}$. Multiple clips share the same length $N$ and thus
can be packed into a batch. Furthermore, instead of recursion, we compute the
attention in matrix form:

$\displaystyle\mathrm{ES}\mathchar
45\relax\mathrm{Attention}_{train}(\mathbf{Q}^{\prime\prime},\mathbf{X})$
$\displaystyle=\mathrm{Softmax}\left(\log(\mathbf{M}_{ES})+\frac{\mathbf{Q}^{\prime\prime}\mathbf{K}^{\top}}{\sqrt{C}}\right)\cdot\mathbf{V},$
(9) $\displaystyle\mathbf{M}_{ES}$
$\displaystyle=\begin{bmatrix}e^{-\lambda(N-1)}&e^{-\lambda(N-2)}&\cdots&1\\\
e^{-\lambda(N-1)}&e^{-\lambda(N-2)}&\cdots&1\\\ \vdots&\vdots&&\vdots\\\
e^{-\lambda(N-1)}&e^{-\lambda(N-2)}&\cdots&1\\\ \end{bmatrix},$ (10)

where $\log(\cdot)$ takes the element-wise logarithm of a matrix and the
exponential smoothing matrix $\mathbf{M}_{ES}\in\mathcal{R}^{M\times N}$ is a
Vandermonde matrix. Since we train on the windowed input and test on un-
windowed streaming input, we select a decay factor $\lambda$ and window size
$N$ such that $e^{-\lambda(N-1)}$ is sufficiently small. This minimizes the
effect of a potential train-test gap. Fig. 4 shows the difference between
training and inference for streaming attention.

We use the cross-entropy loss to predict both current and anticipated actions.
Following [54, 19], we predict actions for all frames in short-term memory for
a stronger supervisory signal. We use a causal attention mask [44] on the
short-term memory to avoid future actions from influencing our predictions.

##  5 Experiments

###  5.1 Experimental Setup

Datasets. We conduct experiments on THUMOS’14 [24] and Epic-Kitchen-100
(EK100) [9]. THUMOS’14 contains 413 untrimmed videos annotated with 20
actions. We train our model on the validation set (200 videos) and evaluate on
the test set (213 videos). Epic-Kitchen-100 contains 100 hours of egocentric
videos with 90K action segments. The narrations are mapped into 97 verb
classes and 300 noun classes. We follow the train/val split given by Furnari
et al. [16].

Evaluation Metrics. For THUMOS’14, we measure the performance of both online
action detection and anticipation with per-frame mean average precision (mAP).
Anticipation mAP uses an anticipation period $\tau_{o}$ which varies from
0.25s to 2.0s with a stride of 0.25s. Online detection mAP is as a special
case of anticipation mAP at $\tau_{o}=0$. EK-100 uses mean Top-5
Verb/Noun/Action Recall to measure anticipation performance per instance with
a predefined $\tau_{o}=1s$ [9].

Implementation Details. On THUMOS14, we pre-process the videos into 24 FPS,
extract the two-stream deep features pretrained on ActivityNet or Kinetics
following Xu et al. [54]. The visual stream is a ResNet-50 [20] while the
motion stream uses BN-Inception [25]. On EK100, we first pre-process the
videos into 30 FPS and fine-tune the two-stream TSN [47] on EK100 action
classification task with ImageNet-pretrained parameters, following Furnari et
al. [16]. When training TeSTra, we apply equalization loss [40] to handle the
long-tailness of actions. Our model is not restricted to using 2D CNNs as
backbone. Efficient 3D CNN such as X3D [15] is also applicable but the longer
input span might cause higher latency.

The training procedure of TeSTra on THUMOS’14 follows Xu et al. [54] for fair
comparison. Specifically, we train TeSTra with batch size of 16 for 25 epochs
using Adam optimizer with a weight decay of 5e-5 and a base learning rate of
7e-5. We apply a cosine annealing schedule with linear warm-up, i.e. the
learning rate linearly increases from 0 to 7e-5 in the first 10 epochs and
then decays following a cosine function.

Figure 5: Illustration of MixClip. In the example sequence, we have 4 action
instances and 2 of them are replaced by another clip that comes from another
video but is annotated with the same action category

MixClip. The model takes as input both short clips as working memory and
frames in the longer history as long-term memory. The duration of long history
is significantly larger (often $10\times$) than the short clip. This means
that two neighboring clips of interest share a large portion of historical
frames. This causes the model to overfit to those scene-related cues and fail
to generalize to unseen scenarios. To resolve this, we propose a simple
augmentation technique called MixClip which increases the diversity of long
history by composing short clips from different recordings into each other.

Assume that the long memory is composed of a sequence of action instances
$\\{(t_{i}^{(s)},t_{i}^{(e)},a_{i})\\}$, where $t_{i}^{(s)},t_{i}^{(e)}$
denotes the start and end time while $a_{i}$ denotes the action label. With
probability $p_{mc}$, each of the action instances may be replaced with
another instance with the same label from a different video. This input
feature sequence is randomly cropped if the new instance’s duration longer.
Otherwise, the input feature sequence is padded to ensure that the length of
history is unchanged for ease of implementation. Fig. 5 gives an illustration.

MixClip is inspired by some popular augmentation techniques widely used in
image classifications, such as CutOut [12], Mixup [60], and CutMix [55].

###  5.2 Main Results

Table 1: Result of online action detection on THUMOS’14. ↯ denotes optical
flow computed by NVIDIA Optical Flow SDK, a faster alternative to TV-L1 [56].
More detailed runtime analysis will be provided in Sec. 5.4

(a) Using ANet-pretrained feature Method | mAP
---|---
RED [17] | 45.3
IDN [13] | 50.0
TRN [53] | 47.2
OadTR [49] | 58.3
LSTR [54] | 65.3
Ours | 68.2

(b) Using Kinetics-pretrained feature Method | mAP
---|---
IDN [13] | 60.3
TRN [53] | 62.1
OadTR [49] | 65.2
LSTR [54] | 69.5
Ours↯ | 67.3
Ours | 71.2

Table 2: Result of online action anticipation on THUMOS’14. † was reproduced by us because LSTR [54] only reported ActivityNet-pretrained results method | Pre-train | mAP@$\tau_{o}$ | average
---|---|---|---
0.25 | 0.50 | 0.75 | 1.0 | 1.25 | 1.50 | 1.75 | 2.0
RED [17] | ANet1.3 | 45.3 | 42.1 | 39.6 | 37.5 | 35.8 | 34.4 | 33.2 | 32.1 | 37.5
TRN [53] | 45.1 | 42.4 | 40.7 | 39.1 | 37.7 | 36.4 | 35.3 | 34.3 | 38.9
TTM [49] | 45.9 | 43.7 | 42.4 | 41.0 | 39.9 | 39.4 | 37.9 | 37.3 | 40.9
LSTR [54] | - | - | - | - | - | - | - | - | 50.1
Ours |  | 64.7 | 61.8 | 58.7 | 55.7 | 53.2 | 51.1 | 49.2 | 47.8 | 55.3
TTM [49] | K400 | 46.8 | 45.5 | 44.6 | 43.6 | 41.9 | 41.1 | 40.4 | 38.7 | 42.8
LSTR† [54] | 60.4 | 58.6 | 56.0 | 53.3 | 50.9 | 48.9 | 47.1 | 45.7 | 52.6
Ours | 66.2 | 63.5 | 60.5 | 57.4 | 54.8 | 52.6 | 50.5 | 48.9 | 56.8

THUMOS’14. We conduct both online action detection and anticipation
experiments on THUMOS’14. In both tasks, the backbone network from which the
feature is extracted is pretrained on either ActivityNet v1.3 [2] or Kinetics
[4]. Table 1 shows the results of online action detection. TeSTra surpasses
the previous states-of-the-art by a large margin. We also adopt NVIDIA Optical
Flow SDK (NVOFA) 222https://developer.nvidia.com/opticalflow-sdk for faster
optical flow computation. NVOFA can run as fast as 1K FPS on a $240\times 180$
image sequence on a modern GPU. We denote the model that takes NVOFA optical
flow as input to be TeSTra ↯. We observe some performance drop, but an mAP of
67.3% is still competitive. Most importantly, the runtime of the entire
pipeline is significantly sped up. More detailed discussion on runtime
analysis will be provided in Sec. 5.4. All results use ES-Attention.

Table 2 shows the results of online action anticipation. TeSTra with Kinetics-
pretrained feature achieves an average mAP of 56.8%, outperforming all
previous methods. For fair comparison, we also rerun LSTR [54] using the same
Kinetics-pretrained feature. This improved LSTR is still 4% below TeSTra.

Table 3: Result of action anticipation on EK100. The upper half lists RGB-only methods; in lower half all types of inputs are allowed Method | Input | Pre-train | overall | unseen | tail
---|---|---|---|---|---
verb | noun | action | verb | noun | action | verb | noun | action
RULSTM [16] | RGB | IN-1k | 27.5 | 29.0 | 13.3 | 29.8 | 23.8 | 13.1 | 19.9 | 21.4 | 10.6
AVT [19] | IN-1k | 27.2 | 30.7 | 13.6 | - | - | - | - | - | -
AVT [19] | IN-21k | 30.2 | 31.7 | 14.9 | - | - | - | - | - | -
Ours | IN-1k | 26.8 | 36.2 | 17.0 | 27.1 | 30.1 | 13.3 | 19.3 | 28.6 | 13.7
RULSTM [16] |  RGB +OF +Obj | IN-1k | 27.8 | 30.8 | 14.0 | 28.8 | 27.2 | 14.2 | 19.8 | 22.0 | 11.1
TempAgg [39] | IN-1k | 23.2 | 31.4 | 14.7 | 28.0 | 26.2 | 14.5 | 14.5 | 22.5 | 11.8
AVT+ [19] | IN-1k | 25.5 | 31.8 | 14.8 | 25.5 | 23.6 | 11.5 | 18.5 | 25.8 | 12.6
AVT+ [19] | IN-21k | 28.2 | 32.0 | 15.9 | 29.5 | 23.9 | 11.9 | 21.1 | 25.8 | 14.1
Ours | RGB+OF | IN-1k | 30.8 | 35.8 | 17.6 | 29.6 | 26.0 | 12.8 | 23.2 | 29.2 | 14.2

EK100. We compare TeSTra with prior works on the EPIC-Kitchen-100 action
anticipation track [9] in Table 3. We split the results into two halves: the
upper half contains methods with only RGB inputs and the lower half uses
additional information, such as optical flow and object feature. Using the
same ImageNet-1k-pretrained feature, TeSTra significantly outperforms RULSTM
[16] and AVT [19] on the action-level recall. The improvement is most
pronounced in the increase noun-level recall. This demonstrates the
effectiveness of incorporating longer input for anticipation. The long-memory
recalls many objects that appeared previously. TeSTra with RGB+OF achieves
4.4% higher verb-level recall than TeSTra with only RGB. One reason for this
our early-fusion. Unlike late-fusion approaches, RULSTM and AVT+, we
concatenate RGB and optical-flow feature at the beginning so that motion-
related feature can be more effectively leveraged. Again, all results use ES-
Attention.

###  5.3 Ablation Studies

We conduct ablation experiments on EK100 to study the role of each module in
the architecture. Our full ablations uses the RGB-only model, but conclusions
generally hold for two-stream input as well.

Table 4: Temporal smoothing kernels. Using explicit windowed-attention and stream-attention under the Laplace kernel yield consistent results Kernel Type | Test Mode | overall | unseen | tail
---|---|---|---|---
act. rec. | act. rec. | act. rec.
Box | FIFO (Eq. (5)) | 16.14 | 12.64 | 12.89
Box | ES (Eq. (6); $\lambda=0$) | 16.08 | 12.22 | 12.70
Laplace | ES (Eq. (9)) | 16.95 | 13.33 | 13.73
Laplace | ES (Eq. (6)) | 16.94 | 13.28 | 13.72

Temporal Smoothing Kernels. We first verify the correctness of the temporal
smoothing kernels at inference time in Table 4. If we apply the box kernel and
apply the FIFO recursion defined in Eq. (5), the result is 16.14%. However, if
we use the exponential smoothing recursion defined in Eq. (6) with decay
factor $\lambda=0$, action recall drops by $0.2\sim 0.4\%$ on unseen and tail
classes. This indicates the necessity to cache historic elements and pop them
when the queue becomes full. When using the Laplace kernel, we compare batch
mode where windowed attention is computed using Eq. (9) and stream mode where
exponential smoothing recursion is computed using Eq. (6). The results are
consistent (less than 0.05% difference).

Table 5: Ablation studies on position embeddings. Temporal position embeddings are unnecessary for long-term memory, justifying our design of separate the temporal smoothing kernel and feature vector PE @ long memory | PE @ short memory | overall act. rec.
---|---|---
✗ | ✓ | 17.0
✓ | ✓ | 16.8
✗ | ✗ | 15.7

Effectiveness of Positional Embedding. The rationale behind streaming
attention is that we can separate the attention kernel into temporal and
feature components. To justify this, we add a temporal positional embedding in
the long-term memory and observe no performance improvement from Table 5. We
also try to remove the temporal embedding in the short-term memory but this
changes the result significantly (-1.3%).

Table 6: Ablation studies on MixClip and long-/short-term memory fusing

(a) The effect of MixClip MixClip Rate | 0 | 0.2 | 0.5 | 0.8
---|---|---|---|---
overall v. rec. | 25.8 | 26.0 | 26.8 | 26.2
overall n. rec. | 34.6 | 35.3 | 36.2 | 35.2
overall act. rec. | 15.5 | 16.0 | 17.0 | 16.2

|  |  |  |

(b) Long- and short-term memory fusion How to fuse | overall act. rec.
---|---
w/o. long mem. | 16.1
no fuse | 15.9
@ long mem. | 16.0
@ comp. mem. | 17.0

Effectiveness of MixClip Table 6(a) shows the effect of MixClip rate on the
anticipation result. When no MixClip is applied, the baseline drops to 15.5%
action recall. The performance consistently improves with MixClip and achieves
the best (17.0%) at $p_{mc}=0.5$.

Fusing long- and short-term memory Table 6(b) compares different ways of
fusing long- and short-term memory. The naive way is to treat long- and short-
term memory separately, i.e. (1) use the TeSTra encoder to compress distant
inputs and (2) use closer inputs as queries in the TeSTra decoder to attend to
this compressed set of vectors. We observe that this no-fuse approach achieves
15.9% which is even 0.2% lower than short-memory-only baseline, where $N=0$
and the TeSTra decoder is instantiated by self-attention. This indicates that
we might need to incorporate the relationship within the short-term memory
too. To achieve this, we try to augment long-term memory by attaching short-
term memory, denoted by “@ long mem.”, but see no significant improvement. It
might be because long-term memory is much longer than the short-term one so
that the short-term information is overwhelmed at the first stage of
compression. Since the memory length after compressed is in the same order as
the short-term memory, we concatenate both (“@ comp. mem.”) and get 17.0%
action recall, improving the naive way by 1.1%.

###  5.4 Runtime Analysis

Finally, we study the runtime speed of TeSTra using an NVIDIA Quadro RTX 6000
GPU. Fig. 6 shows the comparison of inference speed between LSTR with cross
attention and TeSTra with ES-Attention. We choose the length of the long
memory $N$ to be $\\{32,128,512,2048,8196\\}$. We can clearly see that the
runtime per time step scales linearly for cross-attention-based LSTR but keeps
constant for TeSTra. Specifically, TeSTra runs at a speed of 142.8 FPS. If we
integrate TeSTra into the online detection system, we need to take into
account of the computation overhead by the optical flow computation and
feature extraction. The runtime profile is summarized in Table 7. The full
TeSTra runs at 12.6 FPS. The bottleneck is computing optical flow using TV-L1
algorithm [56]. Using the NVOFA, the real-time TeSTra can run at 41.1 FPS.

Figure 6: Runtime comparison between vanilla cross attention and our
exponential smoothing attention

| OF | RGB | OF | TeSTra | Total
---|---|---|---|---|---
| Comp. | Feat. | Feat.
Ours↯ | 1,000 | 150.0 | 104.7 | 142.8 | 41.1
Ours | 19.3 | 12.6

Table 7: Runtime profile (in FPS) for the entire detection system. Real-time
TeSTra uses NVOFA optical-flow while the default one uses TV-L1 [56]

##  6 Conclusion

We propose stream attention based on the kernel-based reformulation of cross-
attention and apply two kinds of temporal smoothing kernels that reduce the
inference computation to constant cost per frame. The resultant temporal
smoothing transformer achieves excellent performance while running at a low
latency. We hope that our design can shed some light on developing more
efficient models for long-term videos understanding.

Acknowledgement This material is in part based upon work supported by the
National Science Foundation under Grant No. IIS-1845485, IIS-2006820, and the
NSF Institute for Foundations of Machine Learning.

## References

  * [1] Ba, J.L., Kiros, J.R., Hinton, G.E.: Layer normalization. arXiv preprint arXiv:1607.06450 (2016)
  * [2] Caba Heilbron, F., Escorcia, V., Ghanem, B., Carlos Niebles, J.: Activitynet: A large-scale video benchmark for human activity understanding. In: CVPR (2015)
  * [3] Carreira, J., Patraucean, V., Mazare, L., Zisserman, A., Osindero, S.: Massively parallel video networks. In: ECCV (2018)
  * [4] Carreira, J., Zisserman, A.: Quo vadis, action recognition? a new model and the kinetics dataset. In: CVPR (2017)
  * [5] Child, R., Gray, S., Radford, A., Sutskever, I.: Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509 (2019)
  * [6] Choromanski, K.M., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J.Q., Mohiuddin, A., Kaiser, L., et al.: Rethinking attention with performers. In: ICLR (2021)
  * [7] Chung, J., Gulcehre, C., Cho, K., Bengio, Y.: Empirical evaluation of gated recurrent neural networks on sequence modeling. In: Deep Learning and Representation Learning Workshop (2014)
  * [8] Dai, Z., Yang, Z., Yang, Y., Carbonell, J.G., Le, Q., Salakhutdinov, R.: Transformer-xl: Attentive language models beyond a fixed-length context. In: ACL (2019)
  * [9] Damen, D., Doughty, H., Farinella, G.M., , Furnari, A., Ma, J., Kazakos, E., Moltisanti, D., Munro, J., Perrett, T., Price, W., Wray, M.: Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens-100. IJCV (2021), https://doi.org/10.1007/s11263-021-01531-2
  * [10] De Geest, R., Gavves, E., Ghodrati, A., Li, Z., Snoek, C., Tuytelaars, T.: Online action detection. In: ECCV (2016)
  * [11] De Geest, R., Tuytelaars, T.: Modeling temporal structure with lstm for online action detection. In: WACV (2018)
  * [12] DeVries, T., Taylor, G.W.: Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04552 (2017)
  * [13] Eun, H., Moon, J., Park, J., Jung, C., Kim, C.: Learning to discriminate information for online action detection. In: CVPR (2020)
  * [14] Fan, H., Xiong, B., Mangalam, K., Li, Y., Yan, Z., Malik, J., Feichtenhofer, C.: Multiscale vision transformers. In: ICCV (2021)
  * [15] Feichtenhofer, C.: X3D: Expanding architectures for efficient video recognition. In: CVPR (2020)
  * [16] Furnari, A., Farinella, G.M.: Rolling-unrolling lstms for action anticipation from first-person video. TPAMI (2020)
  * [17] Gao, J., Yang, Z., Nevatia, R.: Red: Reinforced encoder-decoder networks for action anticipation. In: BMVC (2017)
  * [18] Gao, M., Xu, M., Davis, L.S., Socher, R., Xiong, C.: Startnet: Online detection of action start in untrimmed videos. In: ICCV (2019)
  * [19] Girdhar, R., Grauman, K.: Anticipative video transformer. In: ICCV (2021)
  * [20] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR (2016)
  * [21] Hoai, M., De la Torre, F.: Max-margin early event detectors. IJCV 107(2), 191–202 (2014)
  * [22] Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural computation 9(8), 1735–1780 (1997)
  * [23] Holt, C.C.: Forecasting seasonals and trends by exponentially weighted moving averages. International journal of forecasting 20(1), 5–10 (2004)
  * [24] Idrees, H., Zamir, A.R., Jiang, Y., Gorban, A., Laptev, I., Sukthankar, R., Shah, M.: The THUMOS challenge on action recognition for videos ”in the wild”. CVIU (2016), http://arxiv.org/abs/1604.06182
  * [25] Ioffe, S., Szegedy, C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: ICML (2015)
  * [26] Jaegle, A., Gimeno, F., Brock, A., Vinyals, O., Zisserman, A., Carreira, J.: Perceiver: General perception with iterative attention. In: ICML (2021)
  * [27] Katharopoulos, A., Vyas, A., Pappas, N., Fleuret, F.: Transformers are rnns: Fast autoregressive transformers with linear attention. In: ICML (2020)
  * [28] Kitaev, N., Kaiser, L., Levskaya, A.: Reformer: The efficient transformer. In: ICLR (2020)
  * [29] Kitani, K.M., Ziebart, B.D., Bagnell, J.A., Hebert, M.: Activity forecasting. In: ECCV (2012)
  * [30] Koppula, H., Saxena, A.: Learning spatio-temporal structure from rgb-d videos for human activity detection and anticipation. In: ICML (2013)
  * [31] Lei, J., Wang, L., Shen, Y., Yu, D., Berg, T., Bansal, M.: Mart: Memory-augmented recurrent transformer for coherent video paragraph captioning. In: ACL (2020)
  * [32] Li, Y., Liu, M., Rehg, J.M.: In the eye of beholder: Joint learning of gaze and actions in first person video. In: ECCV (2018)
  * [33] Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B.: Swin transformer: Hierarchical vision transformer using shifted windows. In: ICCV (2021)
  * [34] Nagarajan, T., Li, Y., Feichtenhofer, C., Grauman, K.: Ego-topo: Environment affordances from egocentric video. In: CVPR (2020)
  * [35] Park, H.S., Hwang, J.J., Niu, Y., Shi, J.: Egocentric future localization. In: CVPR (2016)
  * [36] Rae, J.W., Potapenko, A., Jayakumar, S.M., Hillier, C., Lillicrap, T.P.: Compressive transformers for long-range sequence modelling. In: ICLR (2020)
  * [37] Rhinehart, N., Kitani, K.M.: First-person activity forecasting with online inverse reinforcement learning. In: ICCV (2017)
  * [38] Schölkopf, B., Smola, A.J., Bach, F., et al.: Learning with kernels: support vector machines, regularization, optimization, and beyond. MIT press (2002)
  * [39] Sener, F., Singhania, D., Yao, A.: Temporal aggregate representations for long-range video understanding. In: ECCV (2020)
  * [40] Tan, J., Wang, C., Li, B., Li, Q., Ouyang, W., Yin, C., Yan, J.: Equalization loss for long-tailed object recognition. In: CVPR (2020)
  * [41] Tay, Y., Bahri, D., Yang, L., Metzler, D., Juan, D.C.: Sparse sinkhorn attention. In: ICML (2020)
  * [42] Tran, D., Wang, H., Torresani, L., Feiszli, M.: Video classification with channel-separated convolutional networks. In: ICCV (2019)
  * [43] Tsai, Y.H.H., Bai, S., Yamada, M., Morency, L.P., Salakhutdinov, R.: Transformer dissection: An unified understanding for transformer’s attention via the lens of kernel. In: EMNLP (2019)
  * [44] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. In: NeurIPS (2017)
  * [45] Vondrick, C., Pirsiavash, H., Torralba, A.: Anticipating visual representations from unlabeled video. In: CVPR (2016)
  * [46] Wang, H., Zhu, Y., Green, B., Adam, H., Yuille, A., Chen, L.C.: Axial-deeplab: Stand-alone axial-attention for panoptic segmentation. In: ECCV (2020)
  * [47] Wang, L., Xiong, Y., Wang, Z., Qiao, Y., Lin, D., Tang, X., Van Gool, L.: Temporal segment networks for action recognition in videos. T-PAMI (2018)
  * [48] Wang, S., Li, B.Z., Khabsa, M., Fang, H., Ma, H.: Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768 (2020)
  * [49] Wang, X., Zhang, S., Qing, Z., Shao, Y., Zuo, Z., Gao, C., Sang, N.: OadTR: Online action detection with transformers. In: ICCV (2021)
  * [50] Werbos, P.J.: Backpropagation through time: what it does and how to do it. Proceedings of the IEEE 78(10), 1550–1560 (1990)
  * [51] Wu, C.Y., Zaheer, M., Hu, H., Manmatha, R., Smola, A.J., Krähenbühl, P.: Compressed video action recognition. In: CVPR (2018)
  * [52] Wu, C., Wang, Y., Shi, Y., Yeh, C.F., Zhang, F.: Streaming transformer-based acoustic models using self-attention with augmented memory. In: Interspeech (2020)
  * [53] Xu, M., Gao, M., Chen, Y.T., Davis, L.S., Crandall, D.J.: Temporal recurrent networks for online action detection. In: ICCV (2019)
  * [54] Xu, M., Xiong, Y., Chen, H., Li, X., Xia, W., Tu, Z., Soatto, S.: Long short-term transformer for online action detection. In: NeurIPS (2021)
  * [55] Yun, S., Han, D., Oh, S.J., Chun, S., Choe, J., Yoo, Y.: Cutmix: Regularization strategy to train strong classifiers with localizable features. In: ICCV (2019)
  * [56] Zach, C., Pock, T., Bischof, H.: A duality based approach for realtime tv-l 1 optical flow. In: Joint pattern recognition symposium. pp. 214–223. Springer (2007)
  * [57] Zaheer, M., Guruganesh, G., Dubey, K.A., Ainslie, J., Alberti, C., Ontanon, S., Pham, P., Ravula, A., Wang, Q., Yang, L., et al.: Big bird: Transformers for longer sequences. In: NeurIPS. vol. 33 (2020)
  * [58] Zeng, K.H., Shen, W.B., Huang, D.A., Sun, M., Carlos Niebles, J.: Visual forecasting by imitating dynamics in natural sequences. In: ICCV (2017)
  * [59] Zhang, B., Wang, L., Wang, Z., Qiao, Y., Wang, H.: Real-time action recognition with enhanced motion vector cnns. In: CVPR (2016)
  * [60] Zhang, H., Cisse, M., Dauphin, Y.N., Lopez-Paz, D.: mixup: Beyond empirical risk minimization. In: ICLR (2018)
  * [61] Zhang, Q., Lu, H., Sak, H., Tripathi, A., McDermott, E., Koo, S., Kumar, S.: Transformer transducer: A streamable speech recognition model with transformer encoders and rnn-t loss. In: ICASSP (2020)


# Federated Analytics for 6G Networks: Applications, Challenges, and
Opportunities

Juan Marcelo Parra-Ullauri  İD ,, Xunzheng Zhang İD ,, Anderson Bravalheri İD
, Shadi Moazzeni  İD ,, Yulei Wu İD ,, Reza Nejabati İD , , and Dimitra
Simeonidou İD , The authors are with the High-Performance Networks Group,
Smart Internet Lab, School of Computer Science, Electrical and Electronic
Engineering, and Engineering Maths (SCEEM), Faculty of Engineering, University
of Bristol, BS8 1QU, U.K. (e-mail: jm.parraullauri@bristol.ac.uk).

(June 2023)

###### Abstract

Extensive research is underway to meet the hyper-connectivity demands of 6G
networks, driven by applications like XR/VR and holographic communications,
which generate substantial data requiring network-based processing,
transmission, and analysis. However, adhering to diverse data privacy and
security policies in the anticipated multi-domain, multi-tenancy scenarios of
6G presents a significant challenge. _Federated Analytics (FA)_ emerges as a
promising distributed computing paradigm, enabling collaborative data value
generation while preserving privacy and reducing communication overhead. FA
applies big data principles to manage and secure distributed heterogeneous
networks, improving performance, reliability, visibility, and security without
compromising data confidentiality. This paper provides a comprehensive
overview of potential FA applications, domains, and types in 6G networks,
elucidating analysis methods, techniques, and queries. It explores
complementary approaches to enhance privacy and security in 6G networks
alongside FA and discusses the challenges and prerequisites for successful FA
implementation. Additionally, distinctions between FA and Federated Learning
are drawn, highlighting their synergistic potential through a network
orchestration scenario.

###### Index Terms:

Federated Analytics, 6G, Networking, Federated Learning

##  I Introduction

With 5G technology deployment and global standardisation, industry and
academia efforts are dedicated to researching future 6G networks to meet
projected demands [1]. Societal digitisation, hyper-connectivity, and global
data-driven ecosystems drive standardisation and strategic considerations for
6G networks to address communication requirements [2]. Future 6G networks are
expected to be more distributed, heterogeneous, intelligent, and closer to
end-users [1]. Achieving the ambitious 6G vision, including global coverage,
diverse applications, robust security, spectrum utilisation, sensory
integration, and full digitisation, requires further exploration of unresolved
issues and research directions [2].

A key challenge in 6G networks arises from data-related issues due to advances
in sensing, communication, and edge computing, resulting in a surge of data
generation, transmission, and analysis within edge-cloud environments [3].
Data is invaluable to Communication Service Providers (CSPs), with the top 50
carriers holding data from over five billion consumers [4]. Research indicates
data analytics will play a pivotal role in advancing network infrastructure
towards 6G, offering rapid feedback, aiding troubleshooting, and converting
raw data into actionable knowledge for automation systems [5]. Distributed
analytics provide predictive capabilities, guiding algorithmic decisions to
enhance system efficiency and infrastructure management, and can be integrated
into architectural elements of 6G Network Data Analytics Functions (NWDAF)
[6].

The presence of heterogeneous data owners across diverse domains, multiple
tenancies, and various technologies in 6G networks raises concerns about data
privacy and confidentiality [7]. The conventional edge-cloud computing
paradigm, involving central server analysis, is insufficient for evolving
application requirements [3]. _Federated Analytics (FA)_ emerges as a
promising distributed computing paradigm, enabling collaborative value
generation from data across multiple remote entities while preserving local
data for privacy and reduced communication overhead [8]. FA uses big data
principles and tools to manage and secure distributed and diverse data
networks effectively, improving network performance, reliability, visibility,
and security while preserving confidentiality [3].

In this paper, our main focus is to showcase the benefits and applications of
FA to tackle privacy and confidentiality concerns that arise from the diverse
ownership at administrative, operational, and user levels, as well as the
presence of various cloud and edge devices in 6G networks. We offer a thorough
overview of FA in 6G networks, including its potential applications, domains,
and types, as well as an exploration of data analysis methods, techniques, and
queries. We also explore complementary approaches that enhance privacy and
security when combined with FA. Furthermore, we discuss the open challenges
for successful FA implementation in 6G networks. Lastly, we differentiate FA
from Federated Learning (FL), exemplifying their synergistic potential through
a network orchestration scenario.

The rest of this article is structured as follows. Section II explains 6G
Networks, FA, and their distinctions from FL. We introduce a taxonomy for 6G
FA and an implementation framework. We demonstrate the framework’s working
principles through an example and outline research challenges and open issues.
In conclusion, we sum up this article.

##  II Preliminaries and Related Work

Figure 1: The estimated data volumes generated and evolution of networks
towards future 6G.

###  II-A 6G Networks

The implementation of future services in 6G networks relies on seamless,
instant, and virtually unlimited wireless connectivity [2]. Set to materialize
around 2030, 6G addresses challenges beyond the capabilities of backward
compatibility with 5G NR evolution, attracting substantial attention from
academia and industry with extensive research on 6G requirements and vision
[1].

Key 6G requirements span technical aspects and application scenarios,
encompassing data rate and delay with peak rates of up to 1 Tbps using
cutting-edge technologies like THz and optical wireless communications [2].
Additionally, factors such as user-experienced data rate, latency, security,
and high mobility are critical in 6G systems [1]. With an anticipated surge in
data volumes (as showcased in Figure 1), efficient data management,
optimisation, privacy, and security are key in 6G networks [2, 1]. The
potential of 6G applications spans diverse domains, including IoT, smart grid
2.0, holographic telepresence, UAV-based mobility, extended reality, connected
and autonomous vehicles, and intelligent healthcare [1]. The integration of AI
further accelerates 6G’s evolution through scalable and trustworthy edge AI
systems, decentralised ML models, AI of Things, and Haptic communication [1].
As driving applications generate substantial data, privacy and confidentiality
considerations become paramount [1]. Proper evaluation of Key Performance
Indicators (KPIs) will distinguish 6G networks from 5G counterparts in data
rate, delay, capacity, coverage, service efficiency, and diversified service
evaluation [2] to meet the demands of an intelligent and hyper-connected
society.

###  II-B Federated Analytics and Federated Learning

FA is an emerging distributed knowledge aggregation paradigm designed to
address data governance and privacy concerns related to data-sharing [8]. FA
conducts collaborative statistical analyses across multiple distributed nodes
(referred to as _FA clients_) without exchanging raw data among the parties
[3, 7]. Typically, FA involves a central querier (referred to as the _FA
server_) aiming to obtain knowledge or answer questions based on data
distributed across various clients [8]. FA clients receive the query, perform
data-oriented tasks using their local data, and upload abstracted results to
the server [7]. In summary, as described in [8], FA’s objective is for an FA
server to respond to query $Q$ such as:

$Q(\mathcal{D})=F_{\omega}(\mathcal{D}_{1},\mathcal{D}_{2},...,\mathcal{D}_{N})$
(1)

where $\mathcal{D}:\\{\mathcal{D}_{i}\\}_{i=1}^{n}$ corresponds to the private
raw data at the $N\in\mathbb{R}$ data owners (i.e., FA clients). $F_{\omega}$
refers to parameterised function describing the target query [7].

Similar to FA, FL leverages decentralised data processing to address privacy
concerns and enable collaboration among multiple entities. FL focuses on
training ML models in a distributed privacy-preserving fashion [9]. In FL, the
model is trained locally on the user’s device or at the edge of the network,
and only the model updates are exchanged with a central server or aggregator.
The central server aggregates the updates from multiple devices and uses them
to improve the global model [4]. Considering the formulation of equation (1),
FL can be viewed as a complex FA query on implemented at distributed data
owners when the function $F_{\omega}$ describes an optimisation learning
problem defined as:

$F_{\omega}(\mathcal{D}_{1},\mathcal{D}_{2},...,\mathcal{D}_{N})=\arg\min_{\omega}\sum_{n=1}^{N}\frac{K_{n}}{K}\sum_{(x,y)\in\mathcal{D}_{n}}\ell(f(x;\omega),y)$
(2)

where $K$ represents the total number of data samples in all clients combined
and $K_{n}$ represents the number of data samples in client $n\in N$. The main
goal is to minimise for all clients the weighted sum of $\ell(f(x;\omega),y)$
that represents the loss function of model $f$ parameterised with $\omega$
which maps the discrepancy of predicted output and the true label $y$ for
input $x$ [7, 8].

Although FA and FL share federation characteristics, such as local model
computation, central model aggregation, and interactive updates, they differ
in their objectives and design details. While FL focuses on training ML
models, FA is aimed at non-training data analytics tasks. As a result, the
design details of FA vary due to the diverse nature of data analytics tasks
[7]. In 6G networks, FA and FL will play a crucial role for empowering
intelligent nodes to learn and adapt locally, leading to reduced latency,
improved network efficiency, and enhanced user experiences while maintaining
privacy and will be explored in this document.

###  II-C Related Work

TABLE I: Comparison∗ of studies about FA and Networks

| FA or FL | 6G | Architecture | Methods | Applications | Types | Domains | Privacy and Security
---|---|---|---|---|---|---|---|---
Wang et al. [3] | FA | ✗ | ✓ | ✗ | ✓ | ✗ | ✗ | ✓
Zhao et al. [10] | FA | ✗ | ✓ | ✗ | ✓ | ✗ | ✗ | ✓
Wang et al. [7] | FA | ✗ | ✗ | ✓ | ✓ | ✗ | ✓ | ✗
Samdanis et al. [5] | FL | ✓ | ✗ | ✓ | ✗ | ✓ | subset | ✗
Zhou et al. [6] | FL | ✓ | ✓ | subset | ✓ | subset | subset | ✓
Al-Quraan et al. [11] | FL | ✓ | ✓ | subset | ✓ | subset | subset | ✓
This work | FA | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓

∗ FA methods, applications, types, domains and privacy and security concerns
are described in detail in Section III-B.

In recent years, several works have emerged proposing the use of FA and
related techniques in networking and edge computing contexts. Table I
summarises a few important contributions and position our work with respect to
the literature.

Wang et al. [3] present a comprehensive overview of FA, including a basic
taxonomy and an abstract operating model, in addition to proposing a high
level architecture for FA that takes into consideration not only privacy and
security concerns but also basic resource and peer management challenges.

Zhao et al. [10] propose a semihierarchical architecture for IoT data
analytics, utilizing edge nodes as intermediary servers to aggregate models
from local devices, improving convergence speed and reducing round-trip
communications. They employ a random consensus algorithm for sharing model
weights between servers. Wang et al. [7] present a mechanism for performing FA
in industrial IoT, considering data heterogeneity among clients. Skewness
estimation for local data helps select a subset of clients for FA
computations, resulting in improved accuracy and convergence.

Samdanis et al. [5] explore analytics in closed-loop automation systems’
significance for next-gen network infrastructure. They propose a microservice-
based AI and ML model architecture deployable in distributed or centralised
modes. The study investigates inaccuracies due to data property changes (i.e.,
_concept drift_) and reviews detection, prediction, and adaptation techniques.
Zhou et al. [6] examine FL for implementing NWDAFs in 5G/6G. They propose a
partial homomorphic encryption (HE) scheme for privacy and security
preservation. A key management server generates and distributes
encryption/decryption keys among FL participants. The scheme, validated in
simulations, outperforms a prominent differential privacy algorithm without
significant changes in prediction accuracy. Nevertheless, HE increases the
computational time and resource requirements.

Al-Quraan et al. [11] provide a comprehensive summary of FL techniques in 6G
communications. Their systematic review covers enabling technologies, 6G
drivers, and technical integration into network management systems. They
compare the classical client-server architecture with a hierarchical client-
edge-cloud FL approach and explore decentralisation alternatives like
blockchain and peer-to-peer methods. The paper discusses client selection,
incentivisation, data augmentation, and automatic labelling, addressing
communication cost reduction, resource allocation, latency, and convergence
time. The authors also outline privacy and security enhancement techniques for
data sharing.

Although the majority of works on FA (such as [3, 10, 7]), consider edge
computing and some network-related aspects, they don’t specifically relate to
either 5G or 6G functionalities or take advantage of the infrastructure
provided by these platforms. In this paper, we review and expand on the
taxonomy presented in [3] and explore the ramifications of FA from a network-
centric point of view, focusing on its implications for the development of
future 6G infrastructure.

On the other hand, works that explicitly target network infrastructure
management and orchestration or specific 5G/6G applications (e.g. [5, 6, 11]),
tend to consider forms of distributed analytics that do not fully take
advantage of the available FA techniques, or restrict themselves to FL
implementations. In this paper, we provide a comprehensive review of different
types and methods of FA that can be applied to 6G, covering fundamental
privacy and security aspects and outlining how these techniques can be applied
to support deployments in either single or multiple administrative domains.

##  III Federated Analytics for 6G Networks

###  III-A An Architecure for Federated Analytics for 6G Networks

6G systems face privacy and confidentiality challenges due to distribution,
heterogeneity, and diverse ownership at multiple levels. FA offers
collaborative data analysis while maintaining privacy, with data decentralised
across devices, reducing network data transfer. A vital component for FA in 6G
is an architectural framework that integrates abstractions, supports
functional segregation, and accelerates application development. Efficient
peer administration, data structuring, and privacy governance are crucial to
meet FA’s unique demands. Figure 2 depicts the proposed hierarchical
framework. Analytics can be performed in each of the different distributed _FA
Clients (FC)_ and then sent to central to be aggregated by the _FA Server
(FS)_.

In 6G networks, extreme edge scenarios envision multi-tenancy111We refer a
tenant as the user owner of a task. where users can request network
infrastructure-as-a-service for task-offloading, introducing data that
requires privacy and confidentiality handling. Each tenant can act as an FC,
performing local data analytics to feed higher-level processes through an FS
for resource monitoring and allocation, addressing privacy from an individual
perspective.

At the edge level, 6G networks are expected to be highly autonomous,
supporting different access technologies for a connected world. Multi-access
and multi-administrative domain setups will involve heterogeneous services
owned by different entities, requiring privacy-preserving FA at various levels
for data confidentiality. Privacy concerns will be tackled from operational
and administrative viewpoints.

In the cloud level, 6G networks will involve multi-administrative domain
scenarios, where ensuring data privacy and confidentiality becomes complex as
domains collaborate and share resources. Privacy concerns will be addressed
operationally and administratively, developing robust mechanisms to protect
data across diverse ownership structures and administrative boundaries. FA
will play a vital role in enabling secure data sharing and processing,
preserving privacy while facilitating higher-level insights and decisions.

The proposed framework aims to apply FA in 6G networks, addressing privacy
concerns comprehensively from end-users to administrative and organisational
domains.

Figure 2: Federated Analytics for 6G Architecture

###  III-B Federated Analytics for 6G: A Taxonomy

Figure 3: Federated Analytics for 6G Taxonomy

This section presents a taxonomy (Figure 3) for classifying and organising
concepts related to FA in 6G networks that can be exploited using the proposed
architecture.

####  III-B1 Intelligence

The 5GPPP Architecture WG2225GPPP View on 5G Architecture accessed on
01/08/2023. identifies various networking stakeholders, such as CSP and
Virtual Service Providers (VSP), across different organisational domains with
distinct business and operational responsibilities. Privacy concerns arise
when these stakeholders use network data to enhance their operations, market
understanding, and customer insights. FA in 6G networks allows stakeholders to
access real-time data for privacy-aware Business Intelligence (BI) and
Automation Intelligence to optimise decision-making, streamline operations,
and automate tasks for resource allocation and innovative industry solutions.

####  III-B2 Types

FA encompasses various data analysis techniques within a distributed network
of data sources. FA types include: _descriptive analytics_ for understanding
historical data and highlighting key summary statistics, _diagnostic
analytics_ to identify patterns and reasons behind past events.

_Predictive analytics_ uses statistical and machine learning models to make
future event predictions without centralizing sensitive data. _Prescriptive
analytics_ goes further, offering data-driven recommendations for desired
outcomes while respecting data privacy. FA allows collaborative analysis of
distributed FC data, maintaining data security and decentralizing sensitive
information.

####  III-B3 Domains

In 6G networks, heterogeneous domains define administrative or functional
boundaries for network segments, facilitating policy enforcement, security,
and access control. _Multi-domains_ serve different organisational units,
enabling tailored access control. _Cross-domains_ are managed by different
organisations, supporting data exchange with autonomy and security. _Intra-
domains_ are local segments under a single authority, enhancing communication
and resource sharing. FA benefits these domains in 6G networks. For multi-
domains, FA enables collaborative data analysis and insights generation while
preserving data privacy. In cross-domains, FA promotes secure data sharing and
analysis, facilitating effective data collaboration. In intra-domains, FA
enhances data analysis by pooling resources and knowledge, generating
comprehensive insights and optimised resource allocation without centralising
sensitive data.

####  III-B4 Methods

In FA, various aggregation techniques combine data from different sources with
data privacy and security. Key techniques include _cumulative aggregation_ ,
which accumulates values over a specific period or data points. _Addition
aggregation_ involves adding values of the same data attribute across sources,
calculating the total sum without revealing individual data points. Lastly,
_average aggregation_ computes the mean or average value of a data attribute
across sources, revealing overall averages without exposing data values.

An FA query (Eq. (1) in Sec. II) represents an FS question to distributed FCs,
falling into three categories [8]: a) _Statistical testing queries_ aim to
discover statistical properties from private data. b) _Set queries_ focus on
data associations like intersections and unions. c) _Matrix transformation
queries_ include dimensionality reduction using methods like principal
component analysis and projections.

####  III-B5 Applications

In 6G networks, FA enables decentralised data processing and analysis,
ensuring privacy and efficiency for decision-making among multiple entities.
It supports applications such as _metadata extraction_ , where an FS queries
an FC to gather local data insights, like the number of dataset classes and
features. _Performance monitoring_ allows FCs to report ML model or service
performance through local analytics. _Verification and Validation_ involve
focused queries on local processes, giving FS a global system performance view
without centralising raw data.

_Intrusion and anomaly detection_ benefit from FA’s collaborative approach in
network environments. Distributed intrusion detection systems collectively
analyse data while preserving privacy, enhancing threat detection and secure
local anomaly detection for improved network security. FA’s knowledge
extraction _optimises_ network resources and _capacity planning_ during
_orchestration_ processes. Local analytics provide insights into process and
infrastructure status, enabling _end-to-end analytics_ for data-driven
decisions at the automation or business level.

####  III-B6 Privacy and Security

are crucial in 6G, serving both end users and mission-critical vertical
industries. FA supports network-wide data analytics without sharing local
data, benefitting 6G in two key ways. Firstly, within various administrative
domains (e.g., private and public networks, diverse edge device providers), FA
enables analytics within each domain and consolidates insights for global
results. Secondly, given 6G’s multi-tenancy nature, data sharing between
tenants is restricted. FA allows network management insights encompassing all
tenants without data sharing, making it a promising paradigm for privacy-
preserving data analytics in 6G.

However, there have been recent debates in FA regarding the potential privacy
leakage of local data when sharing insights learned from local data in FA,
despite the data not being shared outside its original region. Some existing
approaches address this privacy issue, e.g., cryptography methods like _HE_
and _secure multi-party computation_ , privacy-enhancing technologies like
_differential privacy_ , and _secure aggregation_. Additionally, tampering
with insights during FA’s aggregation by adversaries can impact FA’s
performance, but this issue can be mitigated using technologies such as
_blockchain_ and _Byzantine Fault Tolerance_.

##  IV Example scenario: Cross-Domain Intelligent Orchestration

Figure 4: Federated Analytics and Learning for 6G Networks: Cross-Domain
Profiling TABLE II:
Federated Analytics: Statistical analysis without centralising datasets. VNF Firewall dataset from [12]. |  |  |  |  |  | Distributed Datasets: Skewness per Feature |  |  |
---|---|---|---|---|---|---|---|---|---
Client | N_sam. | N_feat. | CPUUTP | MEMUTP | RTT | MIR | CPU | MEM | In_RX, | OuT_TX | LINK | Selected?
1 | 895 | 9 | -0.7097 | 0.1806 | -0.5809 | 0.8565 | 0.3008 | -0.0123 | -0.0969 | -0.1929 | -0.0415 | yes
2 | 400 | 9 | -0.7926 | 0.1822 | -0.4863 | 0.9215 | 0.3448 | 0.0325 | -0.0829 | -0.1818 | -0.0623 | yes
3 | 100 | 9 | -1.0135 | 0.0411 | -0.3778 | 1.0952 | 0.4609 | 0.1288 | 0.039 | 0.1132 | 0.1107 | no
4 | 120 | 9 | -0.6359 | 0.0419 | -0.6004 | 1.0823 | 0.2961 | -0.1357 | -0.1723 | -0.0749 | -0.216 | no
5 | 400 | 9 | -0.6633 | 0.1731 | -0.5606 | 0.9464 | 0.3187 | 0.0603 | -0.0401 | -0.2112 | -0.1681 | yes
6 | 330 | 9 | -0.7884 | 0.1024 | -0.5401 | 0.786 | 0.3227 | 0.0534 | -0.1275 | -0.1296 | -0.0588 | yes
7 | 580 | 9 | -0.7906 | 0.1078 | -0.5066 | 0.8427 | 0.3451 | 0.0284 | -0.0409 | -0.1009 | -0.0317 | yes
8 | 780 | 9 | -0.715 | 0.1878 | -0.6041 | 0.8647 | 0.2915 | -0.0274 | -0.1168 | -0.1914 | -0.0407 | yes
9 | 500 | 9 | -0.666 | 0.2334 | -0.6553 | 0.8505 | 0.2909 | -0.0154 | -0.0884 | -0.0907 | -0.074 | yes
10 | 290 | 9 | -0.8571 | 0.2951 | -0.4723 | 1.002 | 0.4308 | -0.1155 | -0.0466 | -0.2576 | -0.0057 | no

###  IV-A Scenario

Throughout this document, it’s emphasised that 6G networks will have a vast
array of heterogeneous devices and technologies from various operational and
administrative domains, raising significant privacy concerns related to data
ownership and confidentiality. In this context, FA and FL emerge as promising
privacy-preserving distributed approaches, enabling valuable insights from
local client data without centralising raw data, ensuring data security, and
promoting collaborative intelligence.

An example of FA and FL application in 6G networks is cross-domain intelligent
orchestration, involving the efficient allocation of computing resources
across different domains. In this scenario (Figure 4), various VSPs utilise
shared infrastructure with different access technologies from the INSP to
provide services. Different SPs can share virtual and physical resources in 6G
to meet service requirements, which can involve deploying various Virtual
Network Functions (VNFs). The challenge is selecting the optimal host domain,
edge nodes, and resources for deploying VNFs while adhering to VSP data
ownership policies.

FA can enhance network management and resource optimisation while preserving
data privacy. From the proposed architecture (Section III), an FS can be
deployed at the cross-domain orchestrator level, which can be in the cloud.
FCs are deployed in different virtual edge nodes across various VSPs on the
INSP’s physical edge nodes. Various FA queries $Q_{i}$ run over different edge
nodes from different domains to gather knowledge about local resources and
data availability for VNF profiling used in intelligent orchestration [12].
Statistical testing queries analyse FC dataset quality to determine eligible
clients for joining an FL process to create VNF profiles, while set queries
assess FC similarities and suitability for the FL system. Matrix
transformation queries reduce dimensionality for selected clients before
initiating the FL process. VNF profiles empower intelligent orchestration.
Profiling predicts the maximum network load a VNF can handle and estimates
required resources like CPU, memory, and network to meet performance targets
and workloads [12]. This knowledge aids effective resource allocation and
capacity planning while addressing privacy concerns in cross-domain
orchestration.

###  IV-B Experiment Definition and Evaluation

We demonstrate the proposed approach’s feasibility with a case study prototype
implementation for VNF profiling in a Federated pipeline on a real network
testbed based on the infrastructure setup from our previous work [9].

####  IV-B1 Dataset, model, and task

We use the Virtual Firewall dataset from [12]. This dataset is comprised of 9
features, 8 predictors and the target. The predictor variables are CPU
utilization (CPUUTP), Memory utilization (MEMUTP), Network latency (RTT), VNF
maximum input rate (MIR) and Packet loss (In_RX, Out_Tx), VNF resource
configurations like CPU cores (CPU), Memory (MEM), and the target is the Link
Capacity (LINK). We partition the dataset randomly to generate sub-datasets
for each client. For this experiment, we use a simple fully connected network
with 3 hidden layers with 24, 12, and 6 neurons respectively. For the
federation processes (i.e., FA and FL), we use the framework Flower built on
top of the Kubernetes setup of [9]. The objective of the experiment is to use
FA to examine the datasets of 10 clients scattered in the network. Based on
this analysis, we select the clients that will be part of the FL process for
cross-domain profiling.

####  IV-B2 Experimentation and results

We use FA to conduct statistical data analysis on different edge nodes. We
perform statistical queries to analyse data distribution and characteristics
of clients without centralising the data. We specifically want to find the
data points (Number of samples), the attributes (Number of features), and the
skewness of the features in the individual FA clients’ datasets. Skewness is a
measurement of the distortion of symmetrical distribution or asymmetry in a
data set. Skewness can be quantified as a representation of the extent to
which a given distribution varies from a normal distribution. The FA server
accumulates the results into a matrix that is used for the analysis. This
descriptive analysis allows us to choose the clients best suited for the FL
process.

Table II showcase the results of the FA process. We can observe that the
different number of samples (N_sam.) ranges between 100 and 895. The number of
features (N_feat.) is 9 for all clients. Regarding skewness, we can observe
that none of the data points per feature follows a normal distribution. The
distributions are either skewed to the right(+) or left(-). Based on this
analysis, different approaches can be used for normalising the data
distribution of the dataset. For example, data augmentation and balancing.
However, in our experiment, we want to point out the importance of the data
distribution for the FL process. We select clients with datasets with a
greater number of samples ($<300$) and less skewed data distributions
($-1<x<1$). Based on these criteria, the clients selected for the FL are 1, 2,
4, 5, 6, 7, 8, and 9.

We then assess the performance of standard FL compared to FA-enhanced FL.
Figure 5 showcase the results. It is demonstrated that by selecting clients
based on their data distribution and characteristics, we enhance the
performance of cross-domain profiling reducing the global Mean Absolute Error
(MAE) loss for predicting the LINK capacity. Mapping back to our taxonomy of
Section III, in this experiment, we showcase how to perform _statistical
testing queries_ and _accumulate_ the results for _descriptive analytics_ to
analyse each client dataset. Moreover, FL _cross-domain_ profiling can be
categorised as _predictive analytics_ used for intelligent _orchestration_
aiming at _automation intelligence_.

Figure 5: MAE for predicting LINK capacity: Standard FL vs Proposed FA-
assisted FL for VNF Firewall dataset from [12].

##  V Research Challenges and Open Issues

###  V-A FA for 6G Challenges

####  V-A1 FA potential future applications in 6G networks

When compared to previous telecom generations, 6G presents unique challenges
due to new physical channel designs (higher frequency bands and transmission
rates) and radio network architectures (e.g., “cell-free” environments, smart
surfaces, and mobile/temporary access points). These complexities demand
advanced monitoring and analytics systems for control and management
algorithms. 6G also shifts focus towards Key Value Indicators (KVIs) and
Quality of Experience (QoE) metrics, requiring novel mathematical and
computational models for estimation and quantification. FA is considered a
crucial enabler for addressing these 6G challenges.

For example, regarding Ultra-Large-Scale MIMO [2], FA’s resource management
capabilities can assist in optimising resource allocation and scheduling,
enhancing spectral efficiency by leveraging data from distributed antennas and
UEs. Furthermore, combined with FL, FA can potentially serve for interference
mitigation by sharing relevant information among base stations and
coordinating beamforming strategies. In Integrated Sensing and Communication
(ISAC) [2], FA can improve spectrum utilisation through data analysis from
sensors and communication devices, assisting dynamic spectrum management. In
the Open RAN landscape, FA can enhance network optimisation and offer insights
into RAN component performance, enabling self-configuration and optimisation.
Together with FL, FA can support cooperative resource sharing among O-RAN
operators. In the context of Reconfigurable Intelligent Surfaces (RIS) [2], FA
can contribute to the analysis of environmental data to optimise surface
deployment. When combined with FL, it enables collective learning and
adaptation among these surfaces. In all the previously mentioned 6G scenarios
FA combined with FL can enhance automation and intelligence with a main focus
on privacy preservation.

Nevertheless, 6G also imposes technical difficulties to the deployment of FA
techniques. In particular, 6G high speeds and low latency in data transmission
depend on the development of novel FA algorithms that are more efficient, more
precise and more reliable. Moreover, the high level of distribution and
diversity of 6G’ computing capabilities (considering not only edge and cloud
computing but also connected devices) require new FA designs for increased
robustness and hyper-scalability.

####  V-A2 FA in Goal/Task-oriented Communications

Goal-oriented communications (also referenced as task-oriented commutations
[13]) involve Semantics of Information (SoI), capturing properties of
information related to its “goal” and “purpose” in exchanges. These paradigms
enable the design of more efficient systems that process and transmit only the
most significant information, conserving energy and channel resources without
sacrificing effectiveness in achieving communication goals [13]. FA can
potentially be used for semantics extraction to identify the meaning or
context from unstructured data to understand its underlying concepts,
relationships, and relevant information while maintaining data locally. The
extracted semantics or insights can then be aggregated at a higher level
without exposing sensitive details about individual data points. This ensures
that privacy is maintained throughout the process, as the raw data never
leaves the individual entities’ boundaries. Goal-oriented and semantic
communications are still in the early stages and the convergence with FA
requires further research.

###  V-B Federated Data Analytics Challenges

####  V-B1 Federated analytics with non-iid data

Previous work about FL in non-iid data has been widely covered. However, in
the 6G era, data fragmentation will become more pronounced, as we know from
Big Data to massive amounts of small-sized data [5]. How to use FA to deal
with massive small-sized data and extract knowledge from them is a promising
research area. In FedACS [7], the authors give three modules: insights
derivation, skewness estimation and client selection. How to make clients
generate insensitive insights about their local data, and make the server
aggregates the insights from non-iid data is still under research. How to
standardise the way to get data insight into FA is also a direction to be
studied.

####  V-B2 Federated analytics with synthetic data

Under the secure condition that the raw data should be kept locally, the FA
algorithm can try to make clients generate synthetic data or similar data like
GANs. By analysing this kind of virtual data, FA may get insights and
inferences from different clients who are isolated from each other. The
analysed results then be deployed to the training participants to have better
system performance. For example, in [14] proposed a learning framework using
synthetic data generated by a GAN where the image styles are transferred from
one domain into another domain. 6G scenarios should consider the generated
data to enhance the model performance and also try to realise self-data
generation with self-model improvement for AI-network mutual enhancement.

####  V-B3 Federated analytics with incentive mechanisms

In the 6G era, increased participation of high-quality clients in federated
training poses challenges in determining when and how to engage in FL.
Combining FA with incentive mechanisms allows clients to execute more
effective strategies, improving machine learning model performance. Challenges
include evaluating client contributions and attracting and retaining more
participants. FA can assess final learning performance, estimate data quality,
share information, calculate compensation, and generate revenue. FA incentive
mechanisms also apply to scenarios like data trading and task-oriented
computation [13].

###  V-C Decentralised FA

The usage of a centralising server to aggregate FA results can create a single
point of failure and undermine the security, resilience and scalability of
network systems. Decentralised alternatives to FL, like gossip learning (GL)
and blockcain powered FA (BFA) have been proposed to address these
limitations. Privacy-preserving techniques (e.g., differential privacy) can be
used to mitigate the privacy leakage issue as discussed in Section III-B6 in
both GL and BFA.

GL employs peer selection and communication protocols to exchange parameters
directly between participants, without significant degradation of training
performance or accuracy [15]. Different flow control mechanisms can be
implemented to affect how the messages travel through the network (e.g.,
preventing congestion). Since the availability of gossip protocols is
decoupled from GL algorithms, decentralised analytics architectures inspired
by GL can also be developed.

In BFA, each client broadcasts its local analytics results and aggregates the
received results from other clients along with its own results. In the case of
FL, for each round of local training, clients additionally compete with each
other based on certain consensus mechanisms [11]. The centralised FA server is
replaced by the peer-to-peer blockchain system and the aggregation of local
results is dealt with by the blockchain system, which enhances reliability.
Blockchain also provides verification of local analytics results,
strengthening privacy and security and allowing malicious local results to be
removed before aggregation.

However, blockchain comes with remarkable overheads in terms of increasing
energy consumption, computation demands, computation delay and storage which
significantly degrades 6G system performance. How to bring the computational
overheads down to an acceptable level for 6G is a challenge. This requires the
selection of appropriate types of blockchain systems (e.g., consortium
blockchain) with more efficient consensus algorithms (e.g., HotStuff \-
developed by Facebook, with linear complexity of the authentication nodes). It
also requires joint optimisation by considering the computation requirements
of the blockchain system and the FA system, the elastic computation and
heterogeneous communication resources of the 6G system, and the (stringent)
requirements of 6G networks and applications.

##  VI Conclusion

The presence of diverse data owners and edge devices adds complexity to data
privacy and confidentiality concerns in 6G Intelligent Networks. In response,
FA emerges as a promising distributed computing paradigm for collaborative
value generation from data while maintaining privacy and reducing
communication overheads. FA offers significant advantages in managing and
securing distributed and heterogeneous data networks in 6G systems. This paper
discusses FA principles and benefits, proposes an implementation framework for
6G networks, and identifies research challenges and open issues. Understanding
the potential of FA and addressing these challenges will shape the future of
6G networks and meet the evolving intelligent communication requirements of
our hyper-connected society.

## Acknowledgements

This work has been partially sponsored by the UKRI MyWorld Strength in Places
Programme (SIPF00006/1), the UK GOV DSIT (FONRC) project REASON, and the EU
H2020 5GASP (grant agreement No. 101016448).

## References

  * [1] S. Dang, O. Amin, B. Shihada, and M.-S. Alouini, “What should 6g be?” _Nature Electronics_ , vol. 3, no. 1, pp. 20–29, 2020.
  * [2] C.-X. Wang, X. You, X. Gao, X. Zhu, Z. Li, C. Zhang, H. Wang, Y. Huang, Y. Chen, H. Haas _et al._ , “On the road to 6g: Visions, requirements, key technologies and testbeds,” _IEEE Communications Surveys & Tutorials_, 2023.
  * [3] D. Wang, S. Shi, Y. Zhu, and Z. Han, “Federated analytics: Opportunities and challenges,” _IEEE Network_ , vol. 36, no. 1, pp. 151–158, 2022.
  * [4] U. Mangla, “Application of federated learning in telecommunications and edge computing,” in _Federated Learning: A Comprehensive Overview of Methods and Applications_. Springer, 2022, pp. 523–534.
  * [5] K. Samdanis, A. N. Abbou, J. Song, and T. Taleb, “AI/ML Service Enablers & Model Maintenance for Beyond 5G Networks,” _IEEE Network_ , pp. 1–10, 2023, conference Name: IEEE Network.
  * [6] C. Zhou and N. Ansari, “Securing Federated Learning Enabled NWDAF Architecture with Partial Homomorphic Encryption,” _IEEE Networking Letters_ , 2023, conference Name: IEEE Networking Letters.
  * [7] Z. Wang, Y. Zhu, D. Wang, and Z. Han, “Federated analytics informed distributed industrial iot learning with non-iid data,” _IEEE Transactions on Network Science and Engineering_ , 2022.
  * [8] A. R. Elkordy, Y. H. Ezzeldin, S. Han, S. Sharma, C. He, S. Mehrotra, S. Avestimehr _et al._ , “Federated analytics: A survey,” _APSIPA Transactions on Signal and Information Processing_ , vol. 12, no. 1, 2023.
  * [9] J. M. P. Ullauri, L. F. Gonzalez, A. C. Bravalheri, R. Hussain, X. Vasilakos, I. Vidal, F. Valera, R. Nejabati, and D. Simeonidou, “Privacy preservation in kubernetes-based federated learning: A networking approach,” in _IEEE International Conference on Computer Communications-Workshops(AIDTSP 2023)_ , 2023.
  * [10] L. Zhao, M. Valero, S. Pouriyeh, L. Li, and Q. Z. Sheng, “Communication-efficient semihierarchical federated analytics in iot networks,” _IEEE Internet of Things Journal_ , vol. 9, no. 14, pp. 12 614–12 627, 2021.
  * [11] M. Al-Quraan, L. Mohjazi, L. Bariah, A. Centeno, A. Zoha, K. Arshad, K. Assaleh, S. Muhaidat, M. Debbah, and M. A. Imran, “Edge-Native Intelligence for 6G Communications Driven by Federated Learning: A Survey of Trends and Challenges,” _IEEE Transactions on Emerging Topics in Computational Intelligence_ , vol. 7, no. 3, pp. 957–979, Jun. 2023, conference Name: IEEE Transactions on Emerging Topics in Computational Intelligence.
  * [12] S. Moazzeni, P. Jaisudthi, A. Bravalheri, N. Uniyal, X. Vasilakos, R. Nejabati, and D. Simeonidou, “A novel autonomous profiling method for the next-generation nfv orchestrators,” _IEEE Transactions on Network and Service Management_ , vol. 18, no. 1, pp. 642–655, 2020.
  * [13] J. Shao, Y. Mao, and J. Zhang, “Task-oriented communication for multidevice cooperative edge inference,” _IEEE Transactions on Wireless Communications_ , vol. 22, no. 1, pp. 73–87, 2022.
  * [14] M. M. Rahman, C. Fookes, M. Baktashmotlagh, and S. Sridharan, “Multi-component image translation for deep domain generalization,” in _2019 IEEE Winter Conference on Applications of Computer Vision (WACV)_. IEEE, 2019, pp. 579–588.
  * [15] I. Hegedűs, G. Danner, and M. Jelasity, “Decentralized learning works: An empirical comparison of gossip learning and federated learning,” _Journal of Parallel and Distributed Computing_ , vol. 148, pp. 109–124, Feb. 2021.

Juan Marcelo Parra-Ullauri [M] obtained a bachelor’s degree in engineering in
Electronics and Telecommunications from the University of Cuenca, Ecuador in
2017. In 2023, he obtained a PhD in Computer Science from Aston University in
the UK. Currently, he works as a Senior Research Associate at the Smart
Internet Lab. His research interests include the Internet of Things,
Distributed Machine Learning, Cloud Computing, Explainability in Autonomous
Systems, and Data Engineering.
---
Xunzheng Zhang [S] received the B.Sc. and M.Sc. degree of Communication
Engineering from Shandong University, China. He is currently pursuing the
Ph.D. degree at the University of Bristol, UK. He has experience with Internet
of Things and edge-cloud collaboration testbed deployment under Industry 4.0.
He has the qualification of electronic technology application engineer, owns
patents and won the Best Paper Award of IEEE International Symposium on
Personal, Indoor and Mobile Radio Communications (IEEE PIMRC) 2020, London.
His current research interests include 6G massive Internet of Things,
federated learning, edge computing and machine learning.
---
Anderson Bravalheri received his Bachelor and Master degrees in Electrical
Engineering (Telecommunications and Telematics) from University of Campinas
(UNICAMP), Brazil in 2011 and 2016 respectively. From 2011 to 2017, he was a
telecommunication researcher at the Research and Development Center in
Telecommunications CPqD (Brazil). Currently, he is a Research Fellow in the
Smart Internet Lab, University of Bristol, working with virtualistion and edge
computing for advanced networks and immersive media applications. His research
interests include distributed systems, fuzzy control, and AI solutions for
networks.
---
Shadi Moazzeni is a Research Fellow with the University of Bristol, Bristol,
UK, where she is a member of the Smart Internet Lab and the High-Performance
Networks research group. She received her M.Sc. degree in Computer
Architecture Engineering from Amirkabir University of Technology (Tehran
Polytechnic), Tehran, Iran in 2010, and her Ph.D. in Computer Architecture
Engineering at the University of Isfahan, Iran in 2018. From July 2016 to
February 2017, she was a Ph.D. visiting researcher at the University of
Bologna, Italy. Her current research focuses on next generation intelligent
network orchestration, multi-access edge computing, 6G networks intelligent
profiling and orchestration towards zero-touch network and service management.
---
Yulei Wu [SM] is an Associate Professor working across the Faculty of
Engineering and the Bristol Digital Futures Institute, University of Bristol,
UK. He is also affiliated with the Smart Internet Lab and is a member of the
High Performance Networks Research Group. He received his Ph.D. degree in
Computing and Mathematics and B.Sc. (1st Class Hons.) degree in Computer
Science from the University of Bradford, UK, in 2010 and 2006, respectively.
His research interests focus on digital twins and ethics-responsible decision
making and their applications on future networks, connected systems, edge
computing, and digital infrastructure.
---
Reza Nejabati [SM] is currently a chair professor of intelligent networks and
head of the High-Performance Network Group in the Department of Electrical and
Electronic Engineering in the University of Bristol, UK. He is also a visiting
professor and Cisco chair in the Cisco centre for Intent Based Networking in
the Curtin University, Australia. He has established successful and
internationally recognised experimental research activities in “Autonomous and
Quantum Networks”. Building on his research, He co-founded a successful start-
up company (Zeetta Networks Ltd) with 25 employees and £6m funding. His
research received the prestigious IEEE Charles Kao Award in 2016 and has done
important contributions in 5G, smart city, quantum communication, and future
Internet experimentation.
---
Dimitra Simeonidou [F] is a Full Professor with the University of Bristol, the
Co-Director of the Bristol Digital Futures Institute and the Director of Smart
Internet Lab. Her research is focusing in the ﬁelds of high performance
networks, programmable networks, wireless-optical convergence, 5G/6G and smart
city infrastructures. She is increasingly working with social sciences on
topics of digital transformation for society and businesses. She has been the
Technical Architect and the CTO of the Smart City Project Bristol Is Open. She
is currently leading the Bristol City/Region 5G urban pilots. She has authored
and coauthored over 600 publications, numerous patents, and several major
contributions to standards. She has been the co-founder of two spin-out
companies, the latest being the University of Bristol VC funded spin-out
Zeetta Networks, delivering SDN solutions for enterprise and emergency
networks. Prof. Simeonidou is a Fellow of Royal Academy of Engineering and a
Royal Society Wolfson Scholar.
---