adieresis=ä,germandbls=ß

# In-Network Processing for Low-Latency Industrial Anomaly Detection in
Softwarized Networks

Huanzhuo Wu1, Jia He1, Máté Tömösközi12, Zuo Xiang1 and Frank H.P. Fitzek12
_Email:{huanzhuo.wu$\mid$mate.tomoskozi$\mid$zuo.xiang$\mid$frank.fitzek}@tu-
dresden.de<EMAIL_ADDRESS>This is a preprint of the work [1],
that has been accepted for publication in the proceedings of the 2021 IEEE
Global Communications Conference. 1Deutsche Telekom Chair of Communication
Networks - Technische Universität Dresden, Germany  2Centre for Tactile
Internet with Human-in-the-Loop (CeTI)

###### Abstract

Modern manufacturers are currently undertaking the integration of novel
digital technologies – such as 5G-based wireless networks, the Internet of
Things (IoT), and cloud computing – to elevate their production process to a
brand new level, the level of smart factories. In the setting of a modern
smart factory, time-critical applications are increasingly important to
facilitate efficient and safe production. However, these applications suffer
from delays in data transmission and processing due to the high density of
wireless sensors and the large volumes of data that they generate. As the
advent of next-generation networks has made network nodes intelligent and
capable of handling multiple network functions, the increased computational
power of the nodes makes it possible to offload some of the computational
overhead. In this paper, we show for the first time our IA-Net-Lite industrial
anomaly detection system with the novel capability of in-network data
processing. IA-Net-Lite utilizes intelligent network devices to combine data
transmission and processing, as well as to progressively filter redundant data
in order to optimize service latency. By testing in a practical network
emulator, we showed that the proposed approach can reduce the service latency
by up to 40%. Moreover, the benefits of our approach could potentially be
exploited in other large-volume and artificial intelligence applications.

###### Index Terms:

anomaly detection, in-network computing, network softwarization, internet of
things

##  I Introduction

Presently, the general industrial community is facing the challenge of
integrating Industry 4.0 technologies into their manufacturing processes,
which will eventually lead towards full digitization and intelligence
empowered by such emerging technologies as Internet of Things (IoT),
artificial intelligence (AI), and cloud computing [2]. Among these, IoT
devices are expected to act as both data collectors and the actuators of
remote intelligent computing agents, thereby closing the full control loop in
order to realize a fully autonomous factory operation. In future smart
factories, IoT devices would be tasked with the monitoring of processes and
machines, etc. The data produced by such IoT devices can be, in turn, employed
to detect potential disturbances in the production process before an error
happens (_predictive maintenance_), which can decrease production downtime and
maintenance costs of smart factories [3].

In particular, time-critical acoustic anomaly detection in industrial
environments carries great potential, since acoustic data is able to
foreshadow potential failures in the early-stage and offers significant gains
to accurately predict when maintenance work is crucial [4]. Unlike traditional
industrial signals – e.g., temperature and humidity – acoustic signals are
generally distorted before an abnormal event occurs. Also, such audio-based
detection methods can avoid blocking and angular distortion problems compared
to the alternative image analysis techniques, as well.

IoT-enabled acoustic sensors deployed in factory halls normally collect and
transmit the raw sensor readings for further analysis to separate dedicated
devices. In a generic setup, several devices are deployed at the production
site, which creates a multi-object acoustic anomaly detection environment.
Particularly, the observed raw sensory acoustic signal is inevitably altered
(mixed) due to the presence and operation of multiple objects. Depending on
the complexity of the deployed sensors and observation systems, the overall
amount of data generated can easily reach several Gbps ranges, which can be
understood as a big data problem with _data-rich, information-poor_
characteristics [5]. In [6], a data process based on a Deep Neural Network
(DNN) and Information Abstraction Neural Network (IA-Net) is proposed to
address the challenge of _data-rich, information-poor_ environments, which
empowers detection techniques with high-information density (the acoustic
deviation). After IA-Net solved the issue of processing large amounts of IoT
data for anomaly detection, we are now facing the question of _where should
IA-Net do the processing_?

The most straightforward solution is to employ a centralized service that
manages the handling of the data stemming from the growing volume of sensors.
This effectiveness relies on the local Internet access, which is often limited
to a few hundred Mbps. While large companies can simply upgrade their
infrastructure, most small and mid-sized industrial manufacturers rely on
traditional ISPs for their networking needs, and higher access speeds are tied
to higher costs [7]. A different approach would be to deploy IA-Net on an edge
cloud physically close to the targeted machines. Compared with the centralized
approach, the edge cloud has an advantage in data transmission time, however,
its resource-constrained characteristics will lead to more delay in data
processing and, ultimately, would prevent the fulfillment of time-critical
services.

With the aid of network softwarization, it is possible to deploy flexible
services and applications by combining local, edge, and remote computing
resources within the same logical network [8]. Typically, the deployment of
softwarized networks includes a combination of technologies: Software Defined
Networking (SDN) [9], Network Functions Virtualization (NFV) [10], and Service
Function Chain (SFC) [11]. Utilizing available computational resources within
the network for in-network processing concepts, computation steps can be
deployed in the on-premise network to reduce the overall data volume and,
potentially, speed up processing times [12]. Therefore, a major challenge is
to filter out redundant data by applying the processing logic of information
abstraction on resource-constrained network nodes.

In this paper, based on the idea of information abstraction proposed in [6],
we show the novel design of a lightweight DNN model called Information
Abstraction Neural Network Lite (IA-Net-Lite), and propose its deployment for
the first time in softwarized networks, which, contrary to present norms,
enables in-network data processing to achieve low-latency industrial anomaly
detection. Extensive evaluations of practical implementations show that IA-
Net-Lite reduces the computation load by up to a quarter of the original and
decreases the data transfer delays by about $34.38\%$, thus achieving an
overall latency reduction of up to $38.89\%$.

The rest of the paper is structured as follows. First, Section II introduces
the idea of information abstraction and describes the problem, after which the
proposed method is described in Section III. In Section IV, we introduce the
experimental setup and its metrics. Section V covers the numerical evaluation
and a discussion about our measurement results. Finally, Section VI concludes
our contribution and points out future research aspects.

##  II Background

In this section, we first describe our system for acoustic anomaly detection
and the methods used to filter out redundant information. Following that, we
discuss the importance of this redundant information filtering approach for
low-latency services and the challenges when exploiting it in networks.

###  II-A Information Abstraction

In multi-machine environments, such as factories, a number of source machines
($n$) are expected to be detected as source objects, see Fig. 1. The source of
acoustic data is generated in $m$ time slots and is denoted as
$\\{\mathbf{s}_{1},\mathbf{s}_{2},\cdots,\mathbf{s}_{n}\\}=\mathbf{S}\in\mathbb{R}^{n\times
m}$. Due to the presence and operation of multiple objects, the observation
$\mathbf{X}\in\mathbb{R}^{1\times m}$ of IoT sensors is inevitably a mixture
of the $n$ source data. $\mathbf{X}$, in turn, is transmitted to the data
processing system to determine irregularities (or anomalies) related to the
$n$ objects when only $\mathbf{X}$ is known.

Figure 1: The schematics of information abstraction for industrial anomaly
detection.

Such an anomaly is defined by the anomaly score and is denoted as
$\mathcal{A}$. $\mathcal{A}$ represents the difference between the actual data
and a reference $\mathbf{S}^{r}$. When $\mathcal{A}$ exceeds a threshold
value, the corresponding object is declared to be an anomaly, as shown in Fig.
1. In order to get $\mathcal{A}$, more attention needs to be paid to the
deviation between the detected and the reference objects than to the detected
object itself. Therefore, redundant information not related to exceptions
should be eliminated and only exception-related information needs to be
extracted. Based on this intuition, we proposed in [6] the idea of information
abstraction to abstract separated anomalous information from raw sensory data
for multi-object acoustic anomaly detection, which is a particularly
advantageous data processing approach when handling large data volumes.

As shown in Fig. 1, the information abstraction consists of three distinct
modules: encoder, abstraction, and separation. The encoder converts
$\mathbf{X}$ from the sensor into high-dimensional representations
$\mathbf{Y}\in\mathbb{R}^{256\times\frac{m}{16}}$ to provide more detailed
information. Then abstraction retrieves the features
$\mathbf{Z}\in\mathbb{R}^{2048\times\frac{m}{256}}$ related to anomalies, and
finally the separation pairs $\mathbf{Z}$ to each $i$-th machine
$\mathbf{f}_{i}\in\mathbb{R}^{1\times 256}$. Based on these anomaly features,
the anomaly score can be obtained directly.

###  II-B In-Network (Pre-)Processing

As a DNN model, the processing logic of information abstraction is often
deployed centralized on a dedicated server/cloud due to the large
computational requirements. However, with the centralized deployment, all raw
sensory data $\mathbf{X}$ needs to be transmitted through the network to this
server/cloud, i.e., the entire sequence of the generated audio of each client.
Such a large volume of data will inevitably lead to network congestion and
data transmission delays.

An alternative way is to deploy the processing logic on an edge cloud near the
individual IoT sensors in order to achieve shorter transmission delays.
However, due to the nature of networks, the computational resources (in this
case, processing and memory) are limited on edge nodes. This limitation
increases the processing delay, which in effect results in limited or no
reduction in the total service latency. As an example, $9.08\times 10^{9}$
operations are needed by IA-Net [6] to complete the processing task.

Fortunately, information abstraction can eliminate redundant information. With
the development of network softwarization, computational resources in the
network become feasible for performing intermediate processing. Therefore, we
can consider that the entire processing logic is partitioned into several
Virtual Network Functions (VNFs) that filter redundant data one by one. When
these VNF s consume only a small amount of computational resources, the
resource-constrained network nodes can perform partial data processing tasks
without adding additional processing latency. At the same time, through these
VNF s, redundant data can be gradually filtered out without causing network
congestion and increased transmission latency. The nodes in the network also
collaborate toward optimizing the overall service latency.

Therefore, following the above idea, this paper takes industrial anomaly
detection as a case study to solve the following two problems:

  1. 1.

How to design the processing logic for the resource-constrained network nodes
so that it is deployable as a VNF without extra processing delay?

  2. 2.

How should such a VNF be deployed in the network to achieve the purpose of
redundant data filtering?

##  III Our Solution: IA-Net-Lite

Figure 2: Overview of the in-network data processing scenario.

In this section, we first show the design of our lightweight DNN model-based
Information Abstraction Neural Network Lite (IA-Net-Lite), so that it is
suitable for deployment on resource-constrained network nodes in order to
abstract information. Secondly, we discuss how it should be partitioned and
deployed to filter traffic to optimize service latency.

Fig. 2 illustrates the scenario of in-network data processing. The collected
data is constantly sent to a network client, where the client can be either a
WiFi access point or a cellular base station of a (non-public) network. This
client keeps forwarding the data to the backend “server” over a forwarding
path. This path can be either dynamically determined based on a routing
mechanism or statically configured. The forwarding path consists of some
intermediate network switches, which support SDN and NFV.

In a traditional communication network, the network nodes (i.e., clients and
switches) are only responsible for forwarding data to the server, and the
entire DNN model is deployed on the server to process the forwarded data. This
approach is usually called Store-and-Forward (S-F). In the scenario of the in-
network process, the DNN model is split into several sub-operations, which are
deployed as VNF s on the network nodes along the data path. These network
nodes perform more than just forwarding data: they first process the data
using the VNF and then forward the intermediate results to the next node. This
is commonly known as Compute-and-Forward (C-F).

###  III-A A Lightweight Deep Neural Network Model for Networks

As the resources (computing and storage) of network devices are limited, the
overall parameters and operations of a DNN model should be tightly controlled
at an acceptable scale, which is suitable for such lightweight devices. The
complexity of a DNN model during inference is defined by the memory and
computing power required to process the input data. This can be mathematically
formulated by the number of the parameters (Param) and the number of the
Multiply-Add Operations (MAdds).

To minimize Param and MAdd, we designed IA-Net-Lite for in-network processing,
inspired by IA-Net in [6]. Under the guidance of the paradigm from IA-Net, IA-
Net-Lite also consists of three distinct modules: encoder, abstraction, and
separation.

####  III-A1 Encoder

In the encoder module, the one-dimensional input data is transformed into a
higher dimensional representation, in order to provide more detailed
information for subsequent operations. IA-Net-Lite down-samples and maps the
mixed input data $\mathbf{X}\in\mathbb{R}^{1\times m}$ in a high-dimensional
STFT-like feature space $\mathbf{Y}\in\mathcal{R}^{N\times\frac{m}{4}}$ by
using a 1-D convolutional layer (1D-Conv) with stride four. The 1D-Conv is a
set of filters consisting of $N$ filters with length $L$. In the encoder
module, only one convolutional layer is used to minimize computation and
storage resources. This module eventually only contains $0.18\times 10^{3}$
Params, and $3.00\times 10^{6}$ MAdd s.

####  III-A2 Abstraction

Figure 3: 1D-residual convolution (1D-R-Conv).

The objective of this module is to extract anomaly information $\mathbf{F}$
from the encoded features $\mathbf{Y}$. The abstract module accounts for most
of the parameters, which is the main part of the model to be compressed.
Therefore, we propose a 1D-residual convolution (1D-R-Conv). As illustrated in
Fig. 3, it follows three strategies: ($i$) depth-wise residual block [13] to
reduce the computation; ($ii$) a large down-sampling rate of four to squeeze
the input data; ($iii$) a dual path [14] structure to obtain enriched semantic
and temporal domain information and to obtain a balance between accuracy and
size.

Empirically, the commonly used $3\times 1$ convolution kernel causes
information loss when the down-sampling rate is four, e.g., ResNet [15]. To
avoid information loss during information abstraction while increasing the
receptive field of the convolutional layer, we use a large convolutional
kernel $7\times 1$ and $5\times 1$ in the dual path structure. Each
convolution operation is followed by a non-linearity layer (ReLU) and a layer
normalization (Norm). The abstraction module consists of several stacked 1D-R-
Conv blocks. Although the abstraction module is most resource-intensive, with
the lightweight design it only requires $2.29\times 10^{6}$ Params and
$416.00\times 10^{6}$ MAdd s.

####  III-A3 Decoder

In the decoder module we utilize an average pooling layer (Avg.) to squeeze
the temporal information to $1$, and replace the fully-connected layer in the
original IA-Net with a $1\times 1$ convolutional layer (1D-Conv) to assign the
anomalous representations to $n$ objects as abstracted features
$\mathbf{F}\in\mathbb{R}^{n\times 256}$. By reducing the number of parameters,
the decoder module has $1.31\times 10^{6}$ Params and $420.00\times 10^{6}$
MAdd s, which is more conducive to deployment in the network.

###  III-B Deployment in the Network

TABLE I: The traffic filter of each operations block in IA-Net-Lite. Operation Block |  Module |  Input |  Output
---|---|---|---
Encoder |  1D-Conv |  $1\times m$ |  $32\times\frac{m}{4}$
Layer 1 |  1D-R-Conv$\times$1 |  $32\times\frac{m}{4}$ |  $16\times\frac{m}{4}$
Layer 2 |  1D-R-Conv$\times$2 |  $16\times\frac{m}{4}$ |  $24\times\frac{m}{16}$
Layer 3 |  1D-R-Conv$\times$3 |  $24\times\frac{m}{16}$ |  $32\times\frac{m}{64}$
Layer 4 |  1D-R-Conv$\times$4 |  $32\times\frac{m}{64}$ |  $64\times\frac{m}{256}$
Layer 5 |  1D-R-Conv$\times$3 |  $64\times\frac{m}{256}$ |  $96\times\frac{m}{256}$
Layer 6 |  1D-R-Conv$\times$3 |  $96\times\frac{m}{256}$ |  $160\times\frac{m}{1024}$
Layer 7 |  1D-R-Conv$\times$1 |  $160\times\frac{m}{1024}$ |  $320\times\frac{m}{1024}$
Layer 8 |  1D-Conv |  $320\times\frac{m}{1024}$ |  $1280\times\frac{m}{1024}$
Separation |  Avg.1D-Conv |  $1280\times\frac{m}{1024}$ |  $4\times 256$

In a network, usually more data is transmitted than is needed or expected.
Therefore, one solution for reducing the amount of data is to use an
information filtering logic as a VNF that can be deployed on the network nodes
to filter out redundant or unwanted data before the traffic leaves each node.

The limitations on the available resources (processing and memory) of the
network nodes is addressed by the lightweight designed DNN model IA-Net-Lite
in Section III-A. Therefore, compute resources inside the network are
available and intermediate processing can be performed in the network and the
operation blocks of IA-Net-Lite can be deployed as VNF within the network.
However, the question of how computing and networking are integrated, i.e.,
which operation blocks the deployed VNF s should be composed of, is
intuitively important. A proper design will reduce the data volume at each
hop, consequently, preventing network congestion, and, ultimately, improving
the overall service latency.

In Table I, the different operation block outputs in relation to the input
data for IA-Net-Lite is listed. To better assess the ability of traffic
filtering of the different operation blocks, we define filter rate $r$ as
follows:

$r=\frac{\text{Size of Operation Block Output}}{\text{Size of System Input}},$
(1)

which indicates how much data has been filtered out by the operation block. An
$r>1$ means that the input data is augmented, and vice versa.

Figure 4: Filter rate $r$ of data flows by each operation block and the
splitting of the VNFs.

The trend of traffic filtering efficiency is fitted to the $r$ of all
operation blocks together. To maximize the filtering efficiency on every link
in the network, we select the concave points in this filtering trend as the
bounds for constructing the VNF s. The concave point of the $r$-curve is
represented by the last operation of the continuous data reduction. Splitting
before this concave point does not maximize the filtering effect, however,
splitting after this point will cause the filtering effect to deteriorate.

As shown in Fig. 4, the $r$ of Layer 4 and Layer 6 are two concave points,
which split IA-Net-Lite into three VNF s. By these three VNF s, the data
volume can be reduced to $25\%$, $16\%$, and $2\%$ of the original input data,
respectively. Therefore, the overall service latency can be improved when
these three VNF s are deployed on network nodes. In Section V-A we show the
evaluation of the effectiveness of traffic filtering.

##  IV Experimental Design

In this section, we describe our experimental design, including the emulation
setups and measurement metrics.

###  IV-A Emulation Setups

In our evaluation, we use the realistic dataset of MIMII [16], which is a
widely used acoustic data set for malfunctioning industrial machine
inspection, containing 26092 normal and anomalous operating acoustic segments
from four types of real machinery: valves, pumps, fans, and slide rails. Each
acoustic segment is a single-channel 10-second-long acoustic segment with a
sampling rate of 16000 Hz. We randomly select all $n=4$ types of sources from
the dataset to construct the source signals $\mathbf{S}$. We mixed the source
signals $\mathbf{S}$ following the standard normal distribution to simulate as
many mixing scenarios as possible. By doing this, our evaluation has covered
various data types and mixing cases.

We conducted the evaluation on the network emulator Communication Networks
Emulator (ComNetsEmu) [17]. As illustrated in Fig. 2, we set up a multi-hop
topology with switches as intermediate network nodes. Each node can perform
either one or both operations of forwarding and data processing. Hence, there
are two different modes for the deployment of the anomaly detection DNN
models, which are Compute-and-Forward (C-F)and Store-and-Forward (S-F).

We assume that the clients connect to the network and send the observed data
using User Datagram Protocol (UDP) to the server. All topology links have the
same homogeneous bandwidth of $10$ Mbps and a fixed propagation delay of $150$
ms. In the emulation, we do not consider packet losses and assume that all
packets are received sequentially. These settings can be considered typical
for an edge network. We also assume that all devices and connections are
general-purpose and available commercially as is. We implement the client and
server functions in Python. For each configuration of a parameter set, we
performed 60 measurements in which the client sends the observed data to the
server, to ensure the experimental results being statistically significant.

Each packet is forwarded directly to the server by the network node at the
fastest possible S-F speed in conventional S-F mode. When operating in the C-F
mode, the SDN controller inserts rules into the flow table of each switch to
forward the data traffic through each middlebox to build an SFC. For the
overall experiments, we employ a Commercial off-the-shelf (COTS) server with a
2.0G Intel i5 CPU with 16GB RAM using Ubuntu 18.04 LTS.

###  IV-B Metrics

For the evaluation of our method we employ the following metrics:

####  IV-B1 Service Latency

Service latency $t_{s}$ is the Round Trip Time (RTT) between the request sent
by the client and the response delivered by the server. It includes all
transmission and computation delays introduced by end nodes (client and
server) and network nodes. $t_{s}$ greatly affects the QoS of network
services. We compare $t_{s}$ of IA-Net-Lite in C-F and S-F modes, and IA-Net
in S-F mode.

####  IV-B2 Computation Load

To measure the computation load of an VNF, we introduce the metric of
processing latency $t_{p}$, which is the computation time required by a device
to complete the VNF deployed on it. With the same computational power, a
smaller $t_{p}$ means that the VNF on the device is completed faster. As a
result, the computation load of this VNF is small. Conversely, a larger
$t_{p}$ implies a large computational load for this task.

####  IV-B3 Transmission Latency

Transmission latency $t_{t}$ is the time taken by the network to transmit the
packets over the links, including propagation delay, queuing delay, and
transmission delay. For the same network setup – e.g., bandwidth, mac access,
and so on – $t_{t}$ is closely related to the size of data transmitted on the
links.

Overall, in our experimental setups one can assume that $t_{s}=t_{p}+t_{t}$.

##  V Results and Discussions

Based on practical experimental results from emulation, we evaluate and
discuss the performance of the in-network processing method for low-latency
industrial anomaly detection.

###  V-A Service Latency

Figure 5: Likelihood of end-to-end service latency in C-F and S-F modes.

Fig. 5 shows the service latency of IA-Net-Lite in C-F and S-F modes, and IA-
Net in S-F mode, respectively. We observed that in S-F mode, the median
service latency is reduced from $3.6$s in IA-Net to $3.3$s in IA-Net-Lite,
which is a $8.33\%$ reduction. This decrease is due to the fact that our
lightweight design makes IA-Net-Lite consume much less computational
resources, resulting in less processing latency on the server. Section V-B
discusses the lightweight DNN in detail.

More importantly, comparing the median service latency of IA-Net-Lite in C-F
and S-F mode, this latency is significantly reduced from $3.3$s to $2.2$s.
This is due to the collaboration of network nodes when filtering out redundant
information and thus reducing the transmission latency. In other words, the
service latency is reduced by $33.33\%$ in the in-network processing scheme.
Compared to the baseline system IA-Net, this latency is significantly reduced
by $38.89\%$. We discuss traffic filtering in the network in more detail in
Section V-C.

To summarize, we propose IA-Net-Lite in this paper for in-network processing,
which can significantly decrease the service latency as it is capable of
reducing both the transmission and processing latency with the help of network
nodes.

###  V-B Processing Latency and Lightweight Design

Figure 6: Overall processing latency in C-F and S-F modes.

Fig. 6 shows the processing latency of the proposed IA-Net-Lite and the
baseline system IA-Net, to verify the lightweight design of IA-Net-Lite. In
S-F mode, we first observe that the processing latency of IA-Net-Lite is about
$0.1$s, which is a mere quarter of IA-Net’s. On the one hand, the reduction of
the overall processing latency explains the gap between IA-Net-Lite and IA-Net
with the S-F mode in Fig. 5. Moreover, while the server remains the same,
reducing the processing latency by quarter means that the computation load
required by IA-Net-Lite is reduced to a quarter. Thus, IA-Net-Lite with the
lightweight design is more suitable for deployment on resource-constrained
network nodes.

Moreover, we observe that IA-Net-Lite’s processing latency in S-F and C-F are
different. In C-F, it is slight about $0.005$s ($5\%$) more than that in S-F.
Since IA-Net-Lite uses exactly the same processing logic in both S-F and C-F
modes, one of the reasons why this happens could be due to the I/O processing
delay between the switch and VNF in C-F. However, we consider the instability
of the processing time to be within the acceptable range compared to the
benefit of reduced service latency.

###  V-C Transmission Latency and Traffic Filter

TABLE II: Data throughput and filter rate $r$ on each link, and the total transmission latency $t_{t}$ of IA-Net-Lite in C-F and S-F modes. Mode |  client$\rightarrow$s1 |  s1$\rightarrow$s2 |  s2$\rightarrow$server |  $\mathbf{t_{t}}$
---|---|---|---|---
C-F |  64397 byte |  40715 byte |  17626 byte |  $2.1s$
|  $25.12\%$ |  $15.88\%$ |  $6.87\%$
S-F |  256406 byte |  256406 byte |  256406 byte |  $3.2s$
|  $100\%$ |  $100\%$ |  $100\%$

To demonstrate the effectiveness of the traffic filter in terms of
transmission latency, we listed the data throughput and the filter rate on
each network connection in Table II, and gave the results in S-F mode for
comparison. Since all bandwidths between network nodes in our emulator setup
are $10$ Mbps with $150$ ms delay, reducing data throughput effectively leads
to a decrease in transmission latency.

Table II shows that after the filtering of VNF s on the client and the switch
1 (s1), the bandwidth between them is consuming only $25.12\%$ and $15.88\%$
of the original data, respectively, while the high-density information
reaching the server is compressed down to $6.87\%$ of the original data after
the last VNF on the switch 2 (s2). Therefore, the total transmission latency
$t_{t}$ in C-F mode is reduced to about $2.1$s. This reduction, expressed in
the overall service latency, is the gap between IA-Net-Lite with the S-F and
C-F mode in Fig. 5.

In contrast, the bandwidth consumption of the traditional S-F remains the same
as the original data, and thus the transmission latency cannot be reduced.
$t_{t}$ stays about $3.2$s with S-F mode. Comparing the C-F and S-F modes,
although they hold exactly the same processing latency, the redundant data is
filtered by the VNF s on the network nodes with the help of the in-network
processing scheme, thus reducing the transmission latency and finally
obtaining a smaller service latency.

##  VI Conclusion

Time-critical applications are becoming more and more important in modern
smart factories, however, these applications have delayed data transmission
and processing due to high-density sensors and large data volumes. In this
paper, we show our brand new design for an in-network data processing with the
help of network softwarization for industrial anomaly detection, which we call
Information Abstraction Neural Network Lite (IA-Net-Lite). This approach
leverages for the first time the resources of increasingly intelligent network
devices to combine data transmission and processing with filters that reduce
redundancy, and, ultimately, optimize service latency. In this paper, we
showed via the employment of a practical network emulator that our proposed
approach can reduce the service latency by up to 38.89%. The design idea of
this method is also applicable to other large-volume and AI applications.

However, comparing with the theoretical filter rate $r$ in Section III-B, the
measured $r$ (see Table II) in our emulations is slightly larger. This is due
to that the C-F can only filter the payloads of the network packets, while the
headers remain unchanged. Header compression techniques [18] can be used to
reduce the packet headers, thereby maximizing the filtering effect. This is in
our future research agenda.

## Acknowledgment

This work was funded by the Federal Ministry of Education and Research of
Germany (Software Campus Net-BliSS 01IS17044), and by the German Research
Foundation (Deutsche Forschungsgemeinschaft) as part of Germany’s Excellence
Strategy (EXC 2050/1 – Project ID 390696704) Cluster of Excellence “Centre for
Tactile Internet with Human-in-the-Loop” (CeTI) of Technische Universität
Dresden.

## References

  * [1] H. Wu, J. He, M. Tömösközi, Z. Xiang, and F. H. P. Fitzek, “In-network processing for low-latency industrial anomaly detection in softwarized networks,” in _2021 IEEE Global Communications Conference: Next-Generation Networking and Internet (Globecom2021 NGNI)_ , Madrid, Spain, 2021.
  * [2] M. Wollschlaeger, T. Sauter, and J. Jasperneite, “The future of industrial communication: Automation networks in the era of the internet of things and industry 4.0,” _IEEE Industrial Electronics Magazine_ , vol. 11, no. 1, pp. 17–27, 2017.
  * [3] F. Civerchia, S. Bocchino, C. Salvadori, E. Rossi, L. Maggiani, and M. Petracca, “Industrial internet of things monitoring solution for advanced predictive maintenance applications,” _Journal of Industrial Information Integration_ , vol. 7, pp. 4–12, 2017.
  * [4] V. Chandola, A. Banerjee, and V. Kumar, “Anomaly detection: A survey,” _ACM Comput. Surv._ , vol. 41, no. 3, Jul. 2009. [Online]. Available: https://doi.org/10.1145/1541880.1541882
  * [5] M. H. u. Rehman, E. Ahmed, I. Yaqoob, I. A. T. Hashem, M. Imran, and S. Ahmad, “Big data analytics in industrial iot using a concentric computing model,” _IEEE Communications Magazine_ , vol. 56, no. 2, pp. 37–43, 2018.
  * [6] H. Wu, J. He, M. Tömösközi, and F. H. Fitzek, “Abstraction-based multi-object acoustic anomaly detection for low-complexity big data analysis,” in _WS17 IEEE ICC 2021 Workshop on Communication, Computing, and Networking in Cyber-Physical Systems (WS17 ICC’21 Workshop - CCN-CPS)_ , Montreal, Canada, Jun. 2021.
  * [7] R. Glebke, M. Henze, K. Wehrle, P. Niemietz, D. Trauth, P. Mattfeld MBA, and T. Bergs, “A case for integrated data processing in large-scale cyber-physical systems,” in _Proceedings of the 52nd Hawaii International Conference on System Sciences_ , 2019.
  * [8] R. Glebke, J. Krude, I. Kunze, J. Rüth, F. Senger, and K. Wehrle, “Towards executing computer vision functionality on programmable network devices,” in _Proceedings of the 1st ACM CoNEXT Workshop on Emerging in-Network Computing Paradigms_ , 2019, pp. 15–20.
  * [9] E. Haleplidis, K. Pentikousis, S. Denazis, J. H. Salim, D. Meyer, and O. Koufopavlou, “Software-defined networking (sdn): Layers and architecture terminology,” _RFC 7426_ , 2015.
  * [10] B. Han, V. Gopalakrishnan, L. Ji, and S. Lee, “Network function virtualization: Challenges and opportunities for innovations,” _IEEE Communications Magazine_ , vol. 53, no. 2, pp. 90–97, 2015.
  * [11] J. Halpern, C. Pignataro _et al._ , “Service function chaining (sfc) architecture,” in _RFC 7665_ , 2015.
  * [12] I. Kunze, K. Wehrle, D. Trossen, and M.-J. Montpetit, “Use Cases for In-Network Computing,” Internet Engineering Task Force, Internet-Draft draft-irtf-coinrg-use-cases-00, Feb. 2021, work in Progress. [Online]. Available: https://datatracker.ietf.org/doc/html/draft-irtf-coinrg-use-cases-00
  * [13] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, “Mobilenetv2: Inverted residuals and linear bottlenecks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 4510–4520.
  * [14] H. Wu, J. He, M. Tömösközi, and F. H. P. Fitzek, “Y-net: A dual path model for high accuracy blind source separation,” in _2020 IEEE Globecom Workshops (GC Wkshps)_ , 2020, pp. 1–6.
  * [15] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [16] H. Purohit, R. Tanabe, T. Ichige, T. Endo, Y. Nikaido, K. Suefusa, and Y. Kawaguchi, “MIMII Dataset: Sound dataset for malfunctioning industrial machine investigation and inspection,” in _Proceedings of the Detection and Classification of Acoustic Scenes and Events 2019 Workshop (DCASE2019)_ , November 2019, pp. 209–213.
  * [17] Z. Xiang, S. Pandi, J. Cabrera, F. Granelli, P. Seeling, and F. H.-P. Fitzek, “An open source testbed for virtualized communication networks,” _IEEE Communications Magazine_ , pp. 1–7, (Accepted, 2020).
  * [18] M. Tömösközi, D. E. Lucani Rötter, F. H. P. Fitzek, and P. Ekler, “Unidirectional robust header compression for reliable low latency mesh networks,” in _2019 IEEE International Conference on Communications (ICC): Mobile and Wireless Networks Symposium (IEEE ICC’19 –MWN Symposium)_ , 2019.


# Collaborative adversary nodes learning on the logs of IoT devices in an IoT
network

Sandhya Aneja1, Melanie Ang Xuan En1, Nagender Aneja2 Email: {sandhya.aneja,
16b4003<EMAIL_ADDRESS>1Faculty of Integrated Technologies,
Universiti Brunei Darussalam
2School of Digital Science, Universiti Brunei Darussalam,

###### Abstract

Artificial Intelligence (AI) development has encouraged many new research
areas, including AI-enabled Internet of Things (IoT) network. AI analytics and
intelligent paradigms greatly improve learning efficiency and accuracy.
Applying these learning paradigms to network scenarios provide technical
advantages of new networking solutions. In this paper, we propose an improved
approach for IoT security from data perspective. The network traffic of IoT
devices can be analyzed using AI techniques. The Adversary Learning
(AdLIoTLog) model is proposed using Recurrent Neural Network (RNN) with
attention mechanism on sequences of network events in the network traffic. We
define network events as a sequence of the time series packets of protocols
captured in the log. We have considered different packets TCP packets, UDP
packets, and HTTP packets in the network log to make the algorithm robust. The
distributed IoT devices can collaborate to cripple our world which is
extending to Internet of Intelligence. The time series packets are converted
into structured data by removing noise and adding timestamps. The resulting
data set is trained by RNN and can detect the node pairs collaborating with
each other. We used the BLEU score to evaluate the model performance. Our
results show that the predicting performance of the AdLIoTLog model trained by
our method degrades by 3-4% in the presence of attack in comparison to the
scenario when the network is not under attack. AdLIoTLog can detect
adversaries because when adversaries are present the model gets duped by the
collaborative events and therefore predicts the next event with a biased event
rather than a benign event. We conclude that AI can provision ubiquitous
learning for the new generation of Internet of Things.

###### Index Terms:

Deep learning, recurrent neural network, gated recurrent unit, internet of
things, adversary

##  I Introduction

The Internet of Things ((IoT) devices are resource-constrained low power
devices collecting a large volume of data for IoT applications in healthcare,
retail, transportation and manufacturing. The data collected through IoT
applications is valuable and huge. Many malwares, and botnets have been
observed to compromise the IoT devices by leveraging the vulnerabilities like
default passwords. The attackers can affect the physical state of the device
once the device is compromised [1, 2]. IoT devices are vulnerable to the
scenario where devices connected to a gateway can collaborate to mislead the
smart decision of the IoT network. One of the scenarios is wherein IoT devices
in two different LANs or locations can collaborate using a high transmission
antenna to exchange data say temperature, pressure, and humidity. The
collaborating IoT devices can then upload the distant location data to the
server with its own location which can cripple the system due to high
temperature maliciously reported as low temperature. Adversary Learning
(AdLIoTLog) framework collects log files from the various application gateways
[3] and apply deep learning [4, 5] to detect collaborating nodes connected
through high transmission power channel for adversary behavior to other nodes
shown in Figure 1.

Figure 1: Deep Adversary Architecture

While capturing data through IoT, metadata can also be captured to apply AI
techniques for IoT network security. Traditional AI techniques were about
centralized data. In another AI paradigm called as federated learning (FL)
model is trained from distributed systems over the cloud. Here interesting
observation for FL is that the learned model over distributed systems can be
secured like other encrypted numbers communicated over the Internet [6]. A
simple/low-complexity resource allocation algorithm is proposed for a wireless
network to support multiple FL groups [7]. IoT devices may be compromised. We
propose in this paper to analyze network traffic logs of IoT devices
distributed in a network behind the application gateways. This network traffic
logged at application gateways can be used to identify compromised devices as
well as collaborative adversaries.

The IoT devices log comprises the chronological events of the packets of these
protocols. The packets include port numbers, IP addresses, sequence numbers,
flags, checksum, window size and domain names. For example, domains such as
example.com, example.net, and example.org are frequently requested by Amazon
Echo; sub-domains of hp.com and hpeprint.com are seen in DNS queries from the
HP printer [8]. Due to this complex structure of IoT log data, it is
complicated to analyze logs for any information.

A similar method [9, 10] is presented at the application gateway to
authenticate the IoT device by analyzing the 212 features like TCP src port
and TCP dst port from the packet headers of IoT devices in the logged network
traffic.

Log analysis using the Recurrent Neural Network (RNN) method has been [11, 4]
studied to predict future events. In ns-2 network simulator, two network
scenarios were set up to generate data for the proposed study. The trace files
log the sequences of network events of the nodes comprising of different types
of protocols packets. The first scenario was a network without any adversary
node while the second scenario was the network with collaborating adversary
nodes which were connected through a link layer tunnel as hidden channel for
adversary behavior to other nodes.

The Adversary Learning model degrades by 3-4% in the presence of attack in
comparison to the scenario when the network is not under attack. Model was
found more robust for UDP packets in comparison to TCP and HTTP packets. A
network protocol fixes the packet format in the network traffic of the
devices. We observed that the Recurrent Neural Network models - LSTM, GRU etc.
are learned with less execution time and better predicting for network
problems in addition to language translation, emotion detection, and fake news
detection problems. Our contributions are as follows:

  1. 1.

Collaborative adversary events detection was found effective using RNN.

  2. 2.

The network simulator ns-2 trace files generated for collaborative attack
dataset and further interfaced in PyTorch for AI analytics using RNN model.

##  II Gated Recurrent Neural Network- Sequence-to-Sequence Model

.

We take the RNN-based GRU model [12] for the problem of anomaly detection in
an IoT Network. Assume that the IoT network log vocabulary can be expressed as
input network event and predicted network event using sequences
$x={x_{1},x_{2},\ldots,x_{|x|}}$ and $y={y_{1},y_{2},\ldots,y_{|y|}}$
respectively. LSTM and GRU create internal gates to regulate the information.
These gates can learn the important data and can pass the relevant information
until long chain. The core of the GRU is composed of encoder-decoder network
which consists of three parts: (i) Encoder (ii) Attention Context Vector (iii)
Decoder.

Encoder: An encoder is a stack of many recurrent units where each accepts an
element of the input network event sequence say
$x={x_{1},x_{2},\ldots,x_{|x|}}$. The hidden states
$h_{t}=f(W^{(hh)}h_{t-1}~{}+~{}W^{(hx)}x_{t})$ are computed with the help of
current input, previous state, and weights of the network. This is the final
hidden state of the encoder.

Attention: The context vector aims to encapsulate input network event sequence
information to assist the prediction of output network event sequence by a
decoder. Context vector acts as an initial hidden state for the decoder. The
context vector $c_{p}$ is computed as in Equation
$c_{p}=\Sigma^{|x|}_{t=1}{\alpha_{tp}}h_{t}$ where
$\alpha_{tp}=\frac{exp(r_{tp})}{\Sigma^{|x|}_{t=1}exp(r_{tp})}$ and
$r_{tp}=v^{T}_{a}tanh(W^{(ss)}{s_{p-1}}+W^{(hh)}h_{t-1})$ with the help of
previous hidden state $h_{t-1}$, previous state $s_{p-1}$, and weights of the
network normalized over the source sequence.

Decoder:  A decoder also comprises of recurrent units cells wherein each cell
predicts an element of the output network event at a time step. Each recurrent
unit cell accepts the previous target state $y_{p-1}$ and source context
vector $c_{p}$ to produce output element
$s_{p}=f(W^{(ss)}.s_{p-1}+~{}W^{(sy)}y_{p-1}+~{}W^{(sc)}c_{p})$ and next
target hidden state.

The target hidden state
$t_{j}=f(W^{(ss)}.s_{j}+~{}W^{(sy)}y_{j-1}+~{}W^{(sc)}c_{j})$ is computed
using the previous hidden state. The Probability distribution
$P_{j}=softmax(W^{s}.t_{j})$ over all elements of network event in target
vocabulary is produced from the decoder conditioned on the previous ground
truth event $y_{j-1}$, the source context $c_{j}$, and the target hidden state
$s_{j}$ using Softmax.

The output of a GRU is a $|y|$-dimensional tensor $y$ which represents the
probability distribution of each network event element of $x$ over the $|y|$
classes. The GRU is trained over by defining a loss function
$L(GRU_{\theta}(x),y)$ minimized iteratively through backpropagation by
varying parameter $\theta$.

##  III Adversary learning- System model

An IoT device uses protocols such as UDP, TCP, HTTP, TLS, DNS, DHCP, ARP, and
ICMP while to upload the data on the data server. IoT devices can collaborate
by using a high range transmission antenna to exchange data. Since
collaborating IoT devices are only using high range channel, therefore, they
can upload the data of distant location as of their location. The network
traffic comprises events of the packets exchanged between the application
gateway (AG) and the server on the cloud. This network traffic can be logged
onto the application gateway for AI analytics [3]. This traffic includes
packets exchanged by collaborating devices also.

AdLIoTLog uses sequences of the packet events of the protocols and
subsequences of the packet events logged of each protocol. We model the
AdLIoTLog using data with attack and the data without attack for comparison
purpose. This model can be trained without actually sharing the data [6].
Initially, a global model is aggregated thereafter local model updates and
provides local model updates to an aggregator. An aggregator combines all
local model updates ($AG_{1}$ with $AG_{2}$) and construct a new global model
($AG_{1}$ with $AG_{2}$ and $\ldots$ $AG_{n}$). Edge devices query the
aggregator for any adversary in the network [13].

###  III-A Model Construction

The GRU RNN model keeps track of dependencies among elements in the sequences
and therefore in the problem of predicting sequences of network events, the
GRU is set to learn the set of pairs $(x,y)$ where $x$ is an input sequence of
events and $y$ is the expected next sequence of network events.

Let $S$ be a set of $p$ malicious nodes represented by $m_{1},m_{2},..m_{p}$.
Let $S_{1}$ is the set of events of $m_{1}$ malicious nodes and $S_{p}$ is the
set of events of $m_{p}$ malicious nodes. The node $m_{1}$ perform $l$
sequence of events
${e_{11}\rightarrow e_{12},e_{12}\rightarrow e_{13}\ldots
e_{1{l-1}}\rightarrow e_{1l}}$.
The node $m_{p}$ perform $t$ sequence of events
${e_{p1}\rightarrow e_{p2},e_{p2}\rightarrow e_{p3}\ldots
e_{p{t-1}}\rightarrow e_{pt}}$

Therefore it is required to learn a function that can be used for any given
source malicious events of ${m_{s}}$ to predict the targeted coordinated
malicious events of ${m_{t}}$. AdIoTLog collects IoT log over the LAN
therefore AdIoTLog comprises of let $m_{1},m_{2},..m_{p}$ nodes over one
application gateway say $AG_{1}$ while $n_{1},n_{2},..n_{q}$ nodes over
another application gateway say $AG_{2}$. AdIoTLog computes the probability of
possible events in the sequence P((${m_{p}:e_{p1}\rightarrow
e_{p2},e_{p2}\rightarrow e_{p3}\ldots e_{p{l-1}}\rightarrow e_{pl}}$)
$\rightarrow$ (${n_{q}:e_{q1}\rightarrow e_{q2},e_{q2}\rightarrow e_{q3}\ldots
e_{q{l-1}}\rightarrow e_{ql}}$)).

##  IV Algorithm

Our method AdIoTLog aggregator combines all local model updates of $AG_{i}$
with $AG_{j}$. AdIoTLog input the network events sequences of $AG_{i}$ and
$AG_{j}$ to the encoder. If direct communication in the IoT network is allowed
then also IoT nodes across the $AGs$ will not be able to communicate due to
low transmission power channel. The malicious IoT devices communicate over the
hidden link layer channel. Communications over a ”hidden” channel include data
packets as well as control packets, such as ARP packets, TCP/UDP packets, and
other types of packets. The trace-driven event log comprises the communication
across the IoT network.

The sequence to sequence model can map sequences of varying lengths to each
other. Encoder track every output and hidden state such as when a network
sequence of size 100 is input with 256 hidden size, it produces encoder output
tensor of size (100, 256) and final hidden state tensor of 256 size.

The decoder uses the last output or the last hidden state of the encoder. The
attention helps the decoder network to compute attention weights and then
these weights are multiplied by encoder output vectors to create a weighted
encoded vector that contains information about the input network event
sequence. We used max sentence length of 100 words to train the attention
layer. The concept of using target outputs as next input is called teacher
forcing [14] that helps to converge the training process faster. AdIoTLog used
the teacher forcing algorithm randomly with a probability of 0.5.

Network loss is computed in AdIoTLog based on decoder output and target
tensor. Network weights for both encoder and decoder are optimized using
stochastic gradient descent (SGD) optimizer using a learning rate in range of
0.01-0.0001. We stored loss after every 100 steps to track if the network is
learning.

If $A$ is the set of events with the top high probabilities from IoT nodes
with same $AG$ then the AdIoTLog reports nodes to be benign otherwise nodes
are reported anomalous. AdIoTLog takes different IoT traffic patterns
depending on the protocols used by the IoT device. When two hosts communicate
with one another, it does not always indicate malicious activity; however, if
those nodes are not in the range, then it indicates malicious behavior, which
is modelled as nodes in two distinct AGs.

The main aim of our method is to feed the higher context, i.e. splitting the
input text into contextual content to increase the model output probability
distribution so that it matches with the probability distribution of the
ground truth values. The TCP packet, UDP packet, and HTTP packet were
considered in different contexts. This potentially can reduce the gap between
training and inference by training the model to handle the situation, which
will appear during test time.

Figure 2: (a) IoT network without collaborating nodes in ns-2 (b) IoT network
with collaborating nodes in ns-2

##  V Experiment Setup, Results, and Analysis

The GRU RNN model is available in PyTorch. For our experiments, we used Intel
4.7 GHz i7-8700K, 8 GB GTX 1080 with 2560 CUDA cores, and 64 GB Dual Channel
DDR4 at 2400 MHz to run the PyTorch library for GRU model. The various
hyperparameters are explained in Table I.

###  V-A Training Data

We used ns-2 network simulator dataset to verify the proposed detection of
collaborating nodes. We explain the training process that includes preparing
data and training the sequence to sequence network.

####  V-A1 The dataset prepared using Network Simulator

The training dataset in ns-2 was created with 16 nodes. Figure 2 shows two
scenarios (a) IoT network without collaborating nodes and (b) IoT network with
collaborating nodes. Node pairs were simulated as IoT device and data server.
For example, node pair (14, 2) was simulated for node 14 to upload data to
node 2. In the first case, when there is no collaboration, node 14 will upload
the data to node 2. In the second case, when nodes can collaborate using
hidden channel, node 14 will upload the data to node 15. Eight UDP
communication node pairs were used with a 1500 byte packet at the rate of 1
Mbps to generate the data. One pair (14,15) was collaborating adversary nodes
which means 12.5% of simulated traffic constitutes the attack. AI analytics
Sequence-to-Sequence model can remember the good events and collaborative
events, therefore, results in detecting the malicious events of the network.

####  V-A2  Training Sequence-to-Sequence network

To interface ns-2 trace file to RNN model, we need a tensor pair. A tensor
indexes IoT network log vocabulary for the input of the model. A tensor pair
was prepared by including network events input tensor and network events
target tensor. The logged network events were paired following the order of
timestamps of network events one after the other. The input file included
12,236 network sequence pairs with 4170 unique elements that comprise
different types of packets, protocols, sequence numbers, and flags. The
combined ns-2 trace files of both setups - networks with hidden channel and
network without hidden channel were input to the model. To compare the two
scenarios, GRU RNN was also trained without adversary node. Figure 3 shows
variation of model training (Negative Log Likelihood Loss) with moving average
of over 100 iterations. Table I shows hyperparameters used in the training
process.

Figure 3: Variation of model training (NLL Loss) x 100 iteration

###  V-B Results and Analysis

We define the following performance metric based on BLEU score [15]. The BLEU
score was computed by comparing the predicted network event sequence with the
ground truth network sequence using 1-gram (single words). It is 100 if
predicted sequence is exactly similar to ground truth sequence. We define
accuracy of the model that is based on the number of testing pairs as follows:

${\bf Accuracy}=\frac{sum~{}of~{}BLEUscore(TestingPairs)}{len(TestingPairs)}$

Let ${tp_{1},tp_{2},...,tp_{n}}$ are testing pairs and their respective bleu
scores are ${b_{1},b_{2},...,b_{n}}$ then accuracy of model output will be
$\frac{\sum_{i=1}^{n}b_{i}}{n}$.

TABLE I: Hyper parameters of model Network |  No of hidden layer |  No ofiterations |  Learning rate |  Hidden Layer size |  optimizer
---|---|---|---|---|---
IoT ns-2 |  1 |  70,000 |  0.01-0.0001 |  256 |  SGD
TABLE II: Comparison of Model Output under collaborative-attack and non-collaborative attack scenarios Network |  Size of set $A$ |  Node tuple in $A$ in data without attack (node, actual data server, pred data server) |  Node tuple in data with attack (node, actual data server, pred data server) |  Accuracy with collaborative attack |  Accuracy without collaborative attack
---|---|---|---|---|---
IoT ns-2 |  5 |  (6,15,15), (3,2,14), (6,15,15), (2,3,3), (8,9,9) |  (14,2,15), (0,12,12), (2,11,14), (0,12,12), (15,5,6) |  89-95% |  91-98%

Table II shows model performance comparison of model output under
collaborative-attack and non-collaborative attack scenarios. The experimental
results show accuracy of 89-95% in case of collaborative attack and 91-98% in
case of non-collaborative attack.

Next, we explain the findings of the experiments based on model performance.
The experiments were designed to answer the following questions:

##### What is the performance of the GRU RNN-based models when the input
network events use sequences of the packet events of the protocols such as
TCP, UDP, and HTTP and subsequences of the packet events like sequence number,
IP addresses, and window scale option logged for each protocol

In simulator trace files, the values of features like sequence numbers, flags
values, and IP addresses were simulated values different from the format used
by TCP/IP model on a network. For example, IP address in dotted format (e.g.
192.168.1.1) was replaced by a node number (e.g. 14). The generated sequence
numbers were much easier to keep track of relatively small, predictable
numbers rather than the actual numbers. Acknowledgement numbers were also not
very random. We observed the high accuracy of GRU RNN in predicting the
network events for simulator data in comparison to dumped TCP/IP model output
on a IoT network.

##### How is the subset $A$ of $x$ with top probabilities over
$|y|$-dimensional tensor $y$ in $|y|$ classes changed using the data under
collaborative attack (shown in Figure 2a) and non attack scenario (shown in
Figure 2b)

When the results of both models (a) with attack data and (b) without attack
data were compared with each other on the set $A$, we observed that the
accuracy of predicted network subsequences was less in the presence of attack.
The collaborating nodes were connected through hidden channel, therefore the
communication of adversarial node in $N_{1}$ with collaborating adversarial
node in $N_{2}$ superseded in comparison to other nodes in their respective
networks $N_{1}$ and $N_{2}$. The model was biased on predicting the
collaborating class in the $|y|$ classes. In other words, in AdIoTLog under
attack scenario, P((${m_{p}:e_{p1}\rightarrow e_{p2},e_{p2}\rightarrow
e_{p3}\ldots e_{p{l-1}}\rightarrow e_{pl}}$) $\rightarrow$
(${n_{q}:e_{q1}\rightarrow e_{q2},e_{q2}\rightarrow e_{q3}\ldots
e_{q{l-1}}\rightarrow e_{ql}}$)) was high most of the times for collaborating
class. If we present Node tuple in $A$ as (node, actual data server, pred data
server) shown in Table II, for tuple (14, 2, 15), the actual node event was by
2 while model predicted node 15 rather than node 2 for source 14.

##### How the performance of model varied when used for scenario with
collaborating malicious nodes

In simulator generated log, there were sufficient instances needed for model
learning. The model performance reduced in the presence of the attack.
Features were much easier to keep track because of relatively small,
predictable numbers rather than the actual numbers. We observed that when
there is less data to train we get more iteration however when there is more
data, number of iteration are less. GRU RNN can predict the event when trained
on even small data with context.

Figure 3 shows variation of model training (Negative Log Likelihood Loss) with
moving average over 100 iterations.

##  VI Background

Wu et al. [11] presented a method for analyzing the bigdata collected through
tools e.g. Hadoop, Mapreduce, Hive and others based on seq2seq- predictive
models. Rather than input the seq vector from the bigdata components, the
authors labeled the data of each component, to get the embedding vector, and
subsequently, input labeled vector to attention matrix. Finally, the predicted
vector is obtained using target information. In contrast, in our proposed
method we split input vector on the basis of context of protocol which is
processed by encoder and subsequently processed by decoder incorporating
attention using target vector.

The model developed by Shen et al. [4] for collaborative nodes trying to use
vulnerabilities of intrusion protection system also used RNN. The negative
shift in model prediction was used to detect the attack, however, the attack
considered was not over distributed machines, also they did not compare output
of system under attack and without attack scenarios. They varied the length of
input sequence to study the performance of RNN with increasing length of input
sequence. Although, we observed that RNN shows better results if delimited
using context rather than by arbitrarily varying the length of sequences input
to the model.

The model developed by Almiani et al. [16] for intrusion detection system in
an IoT network was trained on the NSL-KDD dataset for different types of
attacks. Amanullah et al. [17] studied and presented the deep learning
technologies for IoT security. The growth in use of deep learning models for
security shows its promising applicability on IoT logs.

##  VII Conclusion

In this paper, we studied performance of GRU RNN model on the traffic log of
an IoT network with and without adversary nodes. The adversary nodes are
assumed to collaborate to cripple the data to be uploaded. We found that
adversary nodes can be detected without considering any additional events
rather it is required to log the network traffic. The log can be analyzed by
AI algorithms to detect the adversary nodes in the network.

## References

  * [1] Jianwei Hou, Leilei Qu, and Wenchang Shi.  A survey on internet of things security from data perspectives.  Computer Networks, 148:295–306, 2019.
  * [2] Vikas Hassija, Vinay Chamola, Balindam Chandra Bajpai, Sherali Zeadally, et al.  Security issues in implantable medical devices: Fact or fiction?  Sustainable Cities and Society, page 102552, 2020.
  * [3] Pieter Robyns, Bram Bonné, Peter Quax, and Wim Lamotte.  Noncooperative 802.11 mac layer fingerprinting and tracking of mobile devices.  Security and Communication Networks, 2017.
  * [4] Yun Shen, Enrico Mariconti, Pierre Antoine Vervier, and Gianluca Stringhini.  Tiresias predicting security events through deep learning.  In Proceedings of 2018 ACM SIGSAC Conference on Computer and Communications Security, pages 592–605, 2018.
  * [5] Christopher Sweet, Stephen Moskal, and Shanchieh Jay Yang.  On the veracity of cyber intrusion alerts synthesized by generative adversarial networks.  arXiv:1908.01219, 2019.
  * [6] Dinh C Nguyen, Ming Ding, Pubudu N Pathirana, Aruna Seneviratne, Jun Li, and H Vincent Poor.  Federated learning for internet of things: A comprehensive survey.  arXiv:2104.07914, 2021.
  * [7] Tung T Vu, Hien Quoc Ngo, Thomas L Marzetta, and Michail Matthaiou.  How does cell-free massive mimo support multiple federated learning groups?  arXiv:2107.09577, 2021.
  * [8] Arunan Sivanathan, Hassan Habibi Gharakheili, Franco Loi, Adam Radford, Chamith Wijenayake, Arun Vishwanath, and Vijay Sivaraman.  Classifying iot devices in smart environments using network traffic characteristics.  IEEE Transactions on Mobile Computing, 18(8):1745–1759, 2018.
  * [9] Markus Miettinen, Samuel Marchal, Ibbad Hafeez, N Asokan, Ahmad-Reza Sadeghi, and Sasu Tarkoma.  Iot sentinel: Automated device-type identification for security enforcement in iot.  In 2017 IEEE 37th International Conference on Distributed Computing Systems (ICDCS), pages 2177–2184. IEEE, 2017.
  * [10] Sandhya Aneja, Nagender Aneja, and Md Shohidul Islam.  Iot device fingerprint using deep learning.  In 2018 IEEE International Conference on Internet of Things and Intelligence System (IOTAIS), pages 174–179. IEEE, 2018.
  * [11] Pin Wu, Zhihui Lu, Quan Zhou, Zhidan Lei, Xiaoqiang Li, Meikang Qiu, and Patrick CK Hung.  Bigdata logs analysis based on seq2seq networks for cognitive internet of things.  Future Generation Computer Systems, 90:477–488, 2019.
  * [12] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio.  Neural machine translation by jointly learning to align and translate.  arXiv:1409.0473, 2014.
  * [13] Jianguo Wang, Xiaomeng Yi, Rentong Guo, Hai Jin, Peng Xu, Shengjun Li, Xiangyu Wang, Xiangzhou Guo, Chengming Li, Xiaohai Xu, et al.  Milvus: A purpose-built vector data management system.  In Proceedings of the 2021 International Conference on Management of Data, pages 2614–2627, 2021.
  * [14] Alex M Lamb, Anirudh Goyal Alias Parth Goyal, Ying Zhang, Saizheng Zhang, Aaron C Courville, and Yoshua Bengio.  Professor forcing: A new algorithm for training recurrent networks.  In Advances In Neural Information Processing Systems, pages 4601–4609, 2016.
  * [15] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.  Bleu: a method for automatic evaluation of machine translation.  In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311–318. Association for Computational Linguistics, 2002.
  * [16] Muder Almiani, Alia AbuGhazleh, Amer Al-Rahayfeh, Saleh Atiewi, and Abdul Razaque.  Deep recurrent neural network for iot intrusion detection system.  Simulation Modelling Practice and Theory, page 102031, 2019.
  * [17] Mohamed Ahzam Amanullah, Riyaz Ahamed Ariyaluran Habeeb, Fariza Hanum Nasaruddin, Abdullah Gani, Ejaz Ahmed, Abdul Salam Mohamed Nainar, Nazihah Md Akim, and Muhammad Imran.  Deep learning and big data technologies for iot security.  Computer Communications, 151:495–517, 2020.


# Spaces of Continuous and Measurable Functions Invariant Under a Group Action

Samuel A. Hokamp  Department of Mathematics, Sterling College, Sterling,
Kansas, 67579<EMAIL_ADDRESS>
###### Abstract.

In this paper we characterize spaces of continuous and $L^{p}$-functions on a
compact Hausdorff space that are invariant under a transitive and continuous
group action. This work generalizes Nagel and Rudin’s 1976 results concerning
unitarily and Möbius invariant spaces of continuous and measurable functions
defined on the unit sphere in $\mathbb{C}^{n}$.

###### Key words and phrases:

Spaces of continuous functions, group actions, functional analysis.
_Corresponding author._ Samuel A. Hokamp ✉<EMAIL_ADDRESS>☎
920-634-7356.

###### 2010 Mathematics Subject Classification:

Primary 46E30. Secondary 32A70

##  1\. Introduction

The idea for this paper came from the realization that much of Nagel and
Rudin’s work characterizing unitarily invariant spaces of continuous and
measurable functions on the unit sphere of $\mathbb{C}^{n}$ (originally found
in [1] and summarized in [2]) could be generalized to spaces of continuous and
measurable functions on a compact Hausdorff space $X$, which are invariant
under the continuous and transitive action of a compact group $G$ on $X$.

A space of complex functions on $X$ is $G$-invariant if the pre-composition of
any function in the set with the action of each element of $G$ on $X$ remains
in the set. A $G$-invariant space is $G$-minimal if it contains no proper
$G$-invariant subspace. Our main result (Theorem 4.1) yields that a collection
of closed $G$-minimal spaces of continuous functions satisfying certain
conditions suffices to characterize all closed $G$-invariant spaces of
continuous functions on $X$: each closed $G$-invariant space is the closure of
the direct sum of a unique subcollection of the $G$-minimal spaces.

A unique regular Borel probability measure $\mu$ on $X$ that is $G$-invariant
in the sense that

$\int_{X}f\,d\mu=\int_{X}f(\alpha x)\,d\mu(x),$

for every continuous function $f$ on $X$ and every $\alpha\in G$ is necessary
to define the conditions for the collection of closed $G$-minimal spaces.
Existence of such a measure is due to André Weil in [3]. Additionally, Theorem
4.1 shows each closed $G$-invariant space of measurable functions with respect
to $\mu$ is characterized by the same collection of $G$-minimal spaces.

In Section 3, we define the conditions which a collection of $G$-minimal
spaces must have in order to induce the closed $G$-invariant spaces of
continuous and measurable functions. In Section 4, we prove our main result,
Theorem 4.1. Section 5 is devoted to the proofs of Lemma 4.3 and Lemma 4.4,
which are used in establishing Theorem 4.1.

##  2\. Preliminaries

Let $X$ be a compact Hausdorff space and $C(X)$ the space of continuous
complex functions with domain $X$. Let $G$ be a compact group that acts
continuously and transitively on $X$. When we wish to be explicit, the map
$\varphi_{\alpha}:X\to X$ shall denote the action of $\alpha$ on $X$ for each
$\alpha\in G$; otherwise, $\alpha x$ denotes the action of $\alpha\in G$ on
$x\in X$.

Let $\mu$ denote the unique regular Borel probability measure on $X$ that is
invariant under the action of $G$. Specifically,

(2.1) $\int_{X}f\,d\mu=\int_{X}f\circ\varphi_{\alpha}\,d\mu,$

for all $f\in C(X)$ and $\alpha\in G$. The existence of such a measure is a
result of André Weil from [3]. A construction of $\mu$ can be found in [4]
(Theorem 6.2), but existence can be established using the Riesz Representation
Theorem (for reference, Theorem 6.19 [5]). Throughout the paper, $\mu$ shall
refer to this measure.

The notation $L^{p}(\mu)$ denotes the usual Lebesgue spaces, for $1\leq
p\leq\infty$. For $Y\subset C(X)$, the uniform closure of $Y$ is denoted
$\overline{Y}$, and for $Y\subset L^{p}(\mu)$, the norm-closure of $Y$ in
$L^{p}(\mu)$ is denoted $\overline{Y}^{p}$.

The following is an easy consequence of (2.1):

######  Remark 2.1.

Let $1\leq p<\infty$ and let $p^{\prime}$ be its conjugate exponent. Then

$\int_{X}(f\circ\varphi_{\alpha})\cdot
g\,d\mu=\int_{X}f\cdot(g\circ\varphi_{\alpha^{-1}})\,d\mu,$

for $f\in L^{p}(\mu)$, $g\in L^{p^{\prime}}(\mu)$, and $\alpha\in G$.

The following definitions are generalizations of definitions found in [2]
related to the unitary group. These more specific definitions are given as
references.

######  Definition 2.2 (12.2.4 [2]).

A space of complex functions $Y$ defined on $X$ is invariant under $G$
($G$-invariant) if $f\circ\varphi_{\alpha}\in Y$ for every $f\in Y$ and every
$\alpha\in G$.

######  Remark 2.3.

Since the action is continuous, $C(X)$ is $G$-invariant. Conversely, if $C(X)$
is $G$-invariant, then each action $\varphi_{\alpha}$ must be continuous.

######  Remark 2.4.

Explicitly, the invariance property (2.1) means $\mu(\alpha E)=\mu(E)$ for
every Borel set $E$ and every $\alpha\in G$. Consequently, (2.1) holds for
every $L^{p}$ function, and thus $L^{p}(\mu)$ is $G$-invariant for all $1\leq
p\leq\infty$.

######  Definition 2.5 (12.2.4 [2]).

If $Y$ is $G$-invariant and $T$ is a linear transformation on $Y$, we say $T$
commutes with $G$ if

$T(f\circ\varphi_{\alpha})=(Tf)\circ\varphi_{\alpha}$

for every $f\in Y$ and every $\alpha\in G$.

######  Definition 2.6 (12.2.8 [2]).

A space $Y\subset C(X)$ is $G$-minimal if it is $G$-invariant and contains no
nontrivial $G$-invariant spaces.

######  Example 2.7.

To illustrate these definitions, let $X=G=T^{n}$, the torus in
$\mathbb{C}^{n}$, such that the action of $G$ on $X$ is given by
coordinatewise multiplication. This action is both transitive and continuous.

For each $k=(k_{1},k_{2},\ldots,k_{n})\in\mathbb{Z}^{n}$, we define $H_{k}$ to
be the space of all complex functions $f$ on $T^{n}$ given by $f(z)=cz^{k}$,
where $c\in\mathbb{C}$ and $z^{k}=z_{1}^{k_{1}}z_{2}^{k_{2}}\ldots
z_{n}^{k_{n}}$; that is, $H_{k}$ is the span of the trigonometric monomial of
power $k$.

Observe that $\dim H_{k}=1$, so that each $H_{k}$ is closed. Further,
$G$-invariance of each $H_{k}$ is clear, and thus each $H_{k}$ is $G$-minimal.

Finally, the classical results used in this paper can be found in many texts,
with the reference given being one such place.

##  3\. $G$-Collections

In this section, we introduce the notion of a $G$-collection, a collection of
closed $G$-invariant spaces which characterize all closed $G$-invariant
subspaces of $C(X)$ (Definition 3.2). However, we must first give Definition
3.1. The particular case of the unitary group acting on the unit sphere in
$\mathbb{C}^{n}$ described in [2] inherently satisfies Definition 3.2($*$), so
has no need of the following definition, but this is not necessarily true in
general for $G$ acting on $X$.

######  Definition 3.1.

For each $x\in X$, the space $H(x)$ is the set of all continuous functions
that are unchanged by the action of any element of $G$ which stabilizes $x$.
That is,

$H(x)=\\{f\in C(X):f=f\circ\varphi_{\alpha},\text{ for all }\alpha\in G\text{
such that }\alpha x=x\\}.$

######  Definition 3.2.

Let $\mathscr{G}$ be a collection of spaces in $C(X)$ with the following
properties:

  * (1)

Each $H\in\mathscr{G}$ is a closed $G$-minimal space.

  * (2)

Each pair $H_{1}$ and $H_{2}$ in $\mathscr{G}$ is orthogonal (in
$L^{2}(\mu)$): If $f_{1}\in H_{1}$ and $f_{2}\in H_{2}$, then

$\int_{X}f_{1}\bar{f_{2}}\,d\mu=0.$

  * (3)

$L^{2}(\mu)$ is the direct sum of the spaces in $\mathscr{G}$.

We say $\mathscr{G}$ is a $G$-collection if it also possesses the following
property:

  * ($*$)

$\dim(H\cap H(x))=1$ for each $x\in X$ and each $H\in\mathscr{G}$.

######  Remark 3.3.

A collection of spaces in $C(X)$ lacking at most only property $(*)$ of
Definition 3.2 always exists, as a consequence of the Peter-Weyl theorem from
[6]. This collection is necessarily unique.

######  Remark 3.4.

Explicitly, Definition 3.2(3) requires each $f\in L^{2}(\mu)$ to have a unique
expansion $f=\sum f_{i}$, with $f_{i}\in H_{i}$, that converges
unconditionally to $f$ in the $L^{2}$-norm.

Throughout the remainder of the paper, we assume that a $G$-collection
$\mathscr{G}$ exists for $X$, indexed by $I$. The rest of this section is
devoted to establishing results related to $\mathscr{G}$ and its elements
$H_{i}$, beginning with the following theorem, which is a generalization of
Theorem 12.2.5 of [2]. Note that we use $[\cdot,\cdot]$ to denote the inner
product on $L^{2}(\mu)$:

$[f,g]=\int_{X}f\bar{g}\,d\mu.$

######  Theorem 3.5.

Suppose $H$ is a closed $G$-invariant subspace of $C(X)$, and $\pi$ is the
orthogonal projection of $L^{2}(\mu)$ onto $H$. Then, $\pi$ commutes with $G$,
and to each $x\in X$ corresponds a unique $K_{x}\in H$ such that

(3.1) $(\pi f)(x)=[f,K_{x}]\hskip 18.06749pt(f\in L^{2}(\mu)).$

Additionally, the functions $K_{x}$ satisfy the following:

  * (1)

$\displaystyle K_{x}(y)=\overline{K_{y}(x)}$ $(x,y\in X)$,

  * (2)

$\displaystyle\pi f=\int_{X}f(x)K_{x}\,d\mu(x)$ $(f\in L^{2}(\mu))$,

  * (3)

$K_{\varphi_{\alpha}(x)}=K_{x}\circ\varphi_{\alpha^{-1}}$ $(\alpha\in G)$,

  * (4)

$K_{x}=K_{x}\circ\varphi_{\alpha}$, for all $\alpha\in G$ such that $\alpha
x=x$, and

  * (5)

$K_{x}(x)=K_{y}(y)>0$ $(x,y\in X)$.

###### Proof.

The projection $\pi$ commutes with $G$ due to the $G$-invariance of
$H^{\perp}$, which follows from Corollary 2.1. The existence and uniqueness of
$K_{x}$ follows from the fact that $f\mapsto(\pi f)(x)$ is a bounded linear
functional on $L^{2}(\mu)$. Further, $K_{x}\in H$ since $\pi f=0$ whenever
$f\perp H$. When $f\in H$, we get

$f(x)=[f,K_{x}].$

In particular, $K_{y}(x)=[K_{y},K_{x}]$, which proves (1), and (2) follows
naturally. Since $\pi$ commutes with $G$,

$[f,K_{\varphi_{\alpha}(x)}]=(\pi
f)(\varphi_{\alpha}(x))=\pi(f\circ\varphi_{\alpha})(x)=[f\circ\varphi_{\alpha},K_{x}]=[f,K_{x}\circ\varphi_{\alpha^{-1}}],$

for every $f\in L^{2}(\mu)$ (Corollary 2.1 yields the last equality). This
proves (3) and the special case (4). Finally, (3) also yields

$K_{\varphi_{\alpha}(x)}(\varphi_{\alpha}(x))=(K_{x}\circ\varphi_{\alpha^{-1}})(\varphi_{\alpha}(x))=K_{x}(x).$

This and the transitivity of the group action yields (5), with the inequality
due to

$K_{x}(x)=[K_{x},K_{x}]>0.$

∎

######  Remark 3.6.

Theorem 3.5(4) yields that $\dim(H(x)\cap H_{i})\geq 1$ for each $x\in X$ and
$i\in I$, so that Definition 3.2($*$) requires each $H_{i}$ to contain a
unique (up to a constant multiple) function which satisfies Theorem 3.5(4) for
each $x\in X$.

######  Definition 3.7.

We define $\pi_{i}$ to be the projection of $L^{2}(\mu)$ onto $H_{i}$. The
domain of each $\pi_{i}$ is extended to $L^{1}(\mu)$ by Theorem 3.5(2).

######  Definition 3.8.

If $\Omega\subset I$, we define $E_{\Omega}$ to be the direct sum of the
spaces $H_{i}$ for $i\in\Omega$.

######  Theorem 3.9.

Suppose $T:H_{i}\to H_{j}$ is linear and commutes with $G$. When $i=j$, $T$ is
the identity on $H_{i}$ scaled by a constant $c$. Otherwise, $T=0$.

###### Proof.

For each $x\in X$, let $K_{x}$ denote the kernel of Theorem 3.5 in $H_{i}$ and
$L_{x}$ the same in $H_{j}$. Then, if $\alpha\in G$ such that $\alpha x=x$,
because $T$ commutes with $G$, we get

$TK_{x}=T(K_{x}\circ\varphi_{\alpha})=(TK_{x})\circ\varphi_{\alpha}.$

Thus, by Definition 3.2($*$), $TK_{x}=c(x)L_{x}$ for some constant $c(x)$, and
hence

$(TK_{x})(x)=c(x)L_{x}(x).$

Observe that $L_{x}(x)$ is independent of $x$. Further, if $y=\alpha x$, then

$(TK_{y})(y)=(TK_{x}\circ\varphi_{\alpha^{-1}})(\alpha x)=(TK_{x})(x).$

Thus, $c(x)=c$ is the same constant for all $x\in X$.

If $f\in H_{i}$, we then get

$f=\int_{X}f(x)K_{x}\,d\mu(x).$

Application of $T$ yields

$Tf=\int_{X}f(x)TK_{x}\,d\mu(x)=c\int_{X}f(x)L_{x}\,d\mu(x)=c\pi_{j}f.$

When $i=j$, then $\pi_{j}f=f$ for all $f\in H_{i}$. When $i\neq j$, then
$\pi_{j}f=0$ for all $f\in H_{i}$. ∎

##  4\. Characterization of Closed $G$-Invariant Spaces

We now prove our main result, Theorem 4.1. Throughout the section, we let
$\mathscr{X}$ denote any of the spaces $C(X)$ or $L^{p}(\mu)$ for $1\leq
p<\infty$.

######  Theorem 4.1.

If $Y$ is a closed $G$-invariant subspace of $\mathscr{X}$, then $Y$ is the
closure of the direct sum of some subcollection of $\mathscr{G}$.

The proof of Theorem 4.1 relies on the particular case when $\mathscr{X}$ is
the space $L^{2}(\mu)$ (Theorem 4.2), as well as Lemma 4.3 and Lemma 4.4,
which allow the passage from $L^{2}(\mu)$ to the other spaces. These lemmas
are proved in Section 5.

######  Theorem 4.2.

If $Y$ is a closed $G$-invariant subspace of $L^{2}(\mu)$, then $Y$ is the
$L^{2}$-closure of the direct sum of some subcollection of $\mathscr{G}$.

###### Proof.

Define the set $\Omega=\\{i\in I:\pi_{i}Y\neq\\{0\\}\\}$ and let $i\in\Omega$.
Since $Y$ is $G$-invariant and $\pi_{i}$ commutes with $G$, $\pi_{i}Y$ is a
nontrivial $G$-invariant space in $H_{i}$. The $G$-minimality of $H_{i}$ then
yields that $\pi_{i}Y=H_{i}$.

Let $Y_{0}$ be the null space of $\pi_{i}$ in $Y$, with relative orthogonal
complement $Y_{1}$. Then $Y_{0}$ is $G$-invariant, and so is $Y_{1}$. Further,
$\pi_{i}:Y_{1}\to H_{i}$ is an isomorphism, whose inverse we denote $\Lambda$.
If we fix $j\in I$ such that $j\neq i$ and define $T=\pi_{j}\circ\Lambda$,
then $T$ maps $H_{i}$ into $H_{j}$ and commutes with $G$. Thus, $T=0$.

We conclude that $\pi_{j}Y_{1}=\\{0\\}$ for all $j\neq i$, and thus
$Y_{1}=H_{i}$. Thus, $H_{i}\subset Y$ for all $i\in\Omega$, and further,
$\overline{E}_{\Omega}^{2}\subset Y$. Since $\pi_{j}Y=\\{0\\}$ for all
$j\notin\Omega$, Definition 3.2(3) yields the opposite inclusion. ∎

######  Lemma 4.3.

If $Y$ is a closed $G$-invariant space in $\mathscr{X}$, then $Y\cap C(X)$ is
dense in $Y$.

######  Lemma 4.4.

If $Y\subset C(X)$, $Y$ is a $G$-invariant space, and some $g\in C(X)$ is not
in the uniform closure of $Y$, then $g$ is not in the $L^{2}$-closure of $Y$.

###### Proof of Theorem 4.1.

If $Y$ is a closed $G$-invariant subspace of $\mathscr{X}$, define $\tilde{Y}$
to be the $L^{2}$-closure of $Y\cap C(X)$. Lemma 4.4 then yields

$\tilde{Y}\cap C(X)=Y\cap C(X).$

We next observe that $Y\cap C(X)$ is $L^{2}$-dense in $\tilde{Y}$ and is
$\mathscr{X}$-dense in $Y$, by Lemma 4.3. Each $\pi_{i}$ is
$\mathscr{X}$-continuous as well as $L^{2}$-continuous, so that
$\pi_{i}Y=\\{0\\}$ if and only if $\pi_{i}\tilde{Y}=\\{0\\}$. By Theorem 4.2,
$\tilde{Y}$ is the $L^{2}$-closure of $E_{\Omega}$, where $\Omega$ is the set
of all $i\in I$ such that $\pi_{i}Y\neq\\{0\\}$. Another application of Lemma
4.4 yields

$\tilde{Y}\cap C(X)=\overline{E}_{\Omega}.$

The $\mathscr{X}$-density of $Y\cap C(X)$ in $Y$ then implies $Y$ is the
$\mathscr{X}$-closure of $E_{\Omega}$. ∎

######  Example 4.5.

We now further explore the situation that was set up in Example 2.7. Recall
that $X=G=T^{n}$, the torus in $\mathbb{C}^{n}$, such that the action of $G$
on $X$ is given by coordinatewise multiplication. This action is both
transitive and continuous, and the measure induced by the action is the usual
Lebesgue measure $m$, normalized so that $m(T^{n})=1$.

Further, $H_{k}$ is the space of all complex functions $f$ on $T^{n}$ given by
$f(z)=cz^{k}$, where $c\in\mathbb{C}$ and
$z^{k}=z_{1}^{k_{1}}z_{2}^{k_{2}}\ldots z_{n}^{k_{n}}$ for
$k=(k_{1},k_{2},\ldots,k_{n})\in\mathbb{Z}^{n}$; that is, $H_{k}$ is the span
of the trigonometric monomial of power $k$.

The collection $\mathscr{G}$ of spaces $H_{k}$ forms a $G$-collection: Each
$H_{k}$ is a closed $G$-invariant space of dimension 1, thus is $G$-minimal.
Further,

$\int_{T^{n}}z^{k}\bar{z}^{k^{\prime}}\,dm(z)=\begin{cases}1&\text{if
}k=k^{\prime}\\\ 0&\text{if }k\neq k^{\prime},\end{cases}$

so that the spaces $H_{k}$ are pairwise orthogonal. Finally, $L^{2}(T^{n})$ is
the direct sum of the spaces $H_{k}$ as a consequence of the Stone-Weierstrass
theorem (presented in [7] as a special case of Bishop’s Theorem, Theorem 5.7).
Thus, $\mathscr{G}$ satisfies the first three properties of Definition 3.2.
Lastly, for $z\in T^{n}$, we have $\dim(H(z)\cap H_{k})=1$ from Theorem 3.5
and the fact that $\dim H_{k}=1$. Thus, $\mathscr{G}$ is a $G$-collection.

Theorem 4.1 then yields that every closed $G$-invariant space of continuous or
$L^{p}$ functions on $T^{n}$ is the closure of the direct sum of some
collection of spaces $H_{k}$. Notably, the collection which induces the space
of all functions which are restrictions to $T^{n}$ of functions holomorphic on
the polydisc and continuous on the closed polydisc is the collection of all
$H_{k}$ such that $k$ has nonnegative coordinates.

##  5\. Proofs of Lemma 4.3 and Lemma 4.4

As in Section 4, we let $\mathscr{X}$ denote any of the spaces $C(X)$ or
$L^{p}(\mu)$, for $1\leq p<\infty$. The proofs of Lemma 4.3 and Lemma 4.4
(given at the end of the section) require Lemma 5.1 and Lemma 5.2, which we
now prove.

######  Lemma 5.1.

If $f\in C(X)$, then the map $\alpha\mapsto f\circ\varphi_{\alpha}$ is a
continuous map of $G$ into $C(X)$.

###### Proof.

For $\alpha\in G$, we define the map $\phi:G\to C(X,X)$ by
$\phi(\alpha)=\varphi_{\alpha}$. Then $\phi$ is continuous when $C(X,X)$ is
given the compact-open topology (Theorem 46.11 of [8]). We note that the
continuity of the group action is used here.

We define the map $T_{f}:C(X,X)\to C(X)$ for $f\in C(X)$ by
$T_{f}(\varphi)=f\circ\varphi$, and we endow both spaces with the compact-open
topology. Let $f\circ\varphi\in C(X)$ for $\varphi\in C(X,X)$ and suppose
$f\circ\varphi\in V$, where $V=V(K,U)$ is a subbasis element in $C(X)$.
Explicitly,

$K\subset(f\circ\varphi)^{-1}(U).\hskip 14.45377pt\text{ That is to say,
}\hskip 14.45377ptK\subset\varphi^{-1}(f^{-1}(U)).$

Then $V^{\prime}=V^{\prime}(K,f^{-1}(U))$ is a subbasis element in $C(X,X)$
and $\varphi\in V^{\prime}$. Further, $V^{\prime}\subset T_{f}^{-1}(V)$, so
that $T_{f}$ is continuous when $C(X,X)$ and $C(X)$ are endowed with the
respective compact-open topologies. We finally observe that since $X$ is
compact, the norm topology and the compact-open topology on $C(X)$ coincide. ∎

######  Lemma 5.2.

If $f\in\mathscr{X}$, then the map $\alpha\mapsto f\circ\varphi_{\alpha}$ is a
continuous map of $G$ into $\mathscr{X}$.

###### Proof.

For brevity, we let $||\cdot||$ denote the norm of the space $\mathscr{X}$ and
$||\cdot||_{\infty}$ the uniform norm in $C(X)$. If $\epsilon>0$, then
$||f-g||<\epsilon/3$ for some $g\in C(X)$. There is a neighborhood $N$ of the
identity in $G$ such that $||g-g\circ\varphi_{\alpha}||_{\infty}<\epsilon/3$
for all $\alpha\in N$ (Lemma 5.1). Since

$|f-f\circ\varphi_{\alpha}|\leq|f-g|+|g-g\circ\varphi_{\alpha}|+|(g-f)\circ\varphi_{\alpha}|,$

we have $||f-f\circ\varphi_{\alpha}||<\epsilon$ for all $\alpha\in N$. ∎

###### Proof of Lemma 4.3.

Let $f\in Y$ and choose $N$ as in the proof of Lemma 5.2. Let
$\psi:G\to[0,\infty)$ be continuous, with support in $N$, such that
$\int\psi\,dm=1$, where $m$ denotes the Haar measure on $G$. Define

$g(x)=\int_{G}\psi(\alpha)f(\alpha x)\,dm(\alpha).$

Since $\alpha\mapsto\psi(\alpha)f\circ\varphi_{\alpha}$ is a continuous map
into $Y$, we have $g\in Y$. If $\beta\in G$ such that $\beta x_{1}=x$, the
invariance of the Haar measure yields

$g(x)=\int_{G}\psi(\alpha\beta^{-1})f(\alpha x_{1})\,dm(\alpha).$

Thus, $g\in Y\cap C(X)$.

Finally, the relation

$f-g=\int_{N}\psi(\alpha)(f-f\circ\varphi_{\alpha})\,dm(\alpha)$

gives $||f-g||<\epsilon$, since $||f-f\circ\varphi_{\alpha}||<\epsilon$
whenever $\alpha\in N$. ∎

###### Proof of Lemma 4.4.

There is a $\mu^{\prime}\in M(X)$ such that $\int f\,d\mu^{\prime}=0$ for all
$f\in Y$, but $\int g\,d\mu^{\prime}=1$. There is a neighborhood $N$ of the
identity in $G$ such that $\text{Re}\int
g\circ\varphi_{\alpha}\,d\mu^{\prime}>\frac{1}{2}$ for every $\alpha\in N$.
Associate $\psi$ to $N$ as in the proof of Lemma 4.3, and define $\Lambda\in
C(X)^{*}$ by

$\Lambda h=\int_{X}\int_{G}\psi(\alpha)h(\alpha
x)\,dm(\alpha)\,d\mu^{\prime}(x).$

By the Schwarz inequality,

$\Big{|}\int_{G}\psi(\alpha)h(\alpha
x)\,dm(\alpha)\Big{|}^{2}\leq\int_{G}|\psi(\alpha)|^{2}\,dm(\alpha)\int_{G}|h(\alpha
x)|^{2}\,dm(\alpha)=||\psi||_{2}^{2}\int_{X}|h|^{2}\,d\mu,$

so that

$|\Lambda h|\leq||\mu^{\prime}||\text{ }||\psi||_{2}||h||_{2}.$

Thus, $\Lambda$ extends to a bounded linear functional $\Lambda_{1}$ on
$L^{2}(\mu)$. By interchanging integrals in the definition of $\Lambda$, we
get $\Lambda_{1}f=0$ for every $f\in Y$, but $\text{Re
}\Lambda_{1}g\geq\frac{1}{2}$. Thus, $g\notin\overline{Y}^{2}$. ∎

##  6\. Future Questions

  * (1)

Does a $G$-collection exist for all groups $G$ acting continuously and
transitively on $X$? What about a collection that only lacks ($*$)? What
conditions might exist on $G$ or $X$ that yield a collection that only lacks
($*$)?

  * (2)

Under what conditions can the restrictions on $X$, $G$, and the action of $G$
on $X$ be loosened? Can the compactness of $X$ and $G$ be substituted with
local compactness? Can the continuity of the action be substituted with
separate continuity?

  * (3)

Suppose $H$ is a subgroup of $G$ and $\mathscr{H}$ is a collection of closed
$H$-minimal spaces satisfying the same conditions as $\mathscr{G}$. What is
the relationship between $\mathscr{H}$ and $\mathscr{G}$? The uniqueness of
$\mu$ shows that the $H$-measure is the same as the $G$-measure, and further,
$G$-invariance implies $H$-invariance (of a space).

We note that (3) is prompted from the study of $\mathscr{M}$-invariant and
$\mathscr{U}$-invariant spaces of continuous functions on the unit sphere of
$\mathbb{C}^{n}$ from [1], in which it is shown that there are infinitely many
$\mathscr{U}$-invariant spaces and only six $\mathscr{M}$-invariant spaces.
These six $\mathscr{M}$-invariant spaces are found by combining the
$\mathscr{U}$-minimal spaces in a specific way (see Lemma 13.1.2 of [2]), and
we are curious if this method can be generalized.

  * (4)

Can the results of [9] similarly be generalized? That is, can a $G$-collection
similarly characterize all weak*-closed $G$-invariant subspaces of
$L^{\infty}(\mu)$?

  * (5)

Under what conditions can a $G$-collection characterize the closed
$G$-invariant algebras of continuous functions? We note that the case for the
unitary group acting on the unit sphere of $\mathbb{C}^{n}$ is discussed in
[10] and is also summarized in [2].

##  7\. Data Availability Statement

Data sharing not applicable to this article as no datasets were generated or
analysed during the current study.

## References

  * [1] Alexander Nagel and Walter Rudin.  Moebius-invariant function spaces on balls and spheres.  Duke Math. J., 43(4):841–865, 1976.
  * [2] Walter Rudin.  Function theory in the unit ball of $\mathbb{C}^{n}$.  Classics in Mathematics. Springer-Verlag, Berlin, 2008.  Reprint of the 1980 edition.
  * [3] André. Weil.  L’intégration dans les groupes topologiques et ses applications.  Actual. Sci. Ind., no. 869, 1940.
  * [4] Joe Diestel and Angela Spalsbury.  The Joys of Haar Measure.  American Mathematical Society, April 2014.
  * [5] Walter Rudin.  Real and complex analysis.  McGraw-Hill, 1987.
  * [6] F. Peter and H. Weyl.  Die Vollständigkeit der primitiven Darstellungen einer geschlossenen kontinuierlichen Gruppe.  Math. Ann., 97(1):737–755, December 1927.
  * [7] Walter Rudin.  Functional Analysis.  McGraw-Hill, 1991.
  * [8] James R Munkres.  Topology, 2000.
  * [9] Samuel A. Hokamp.  Certain invariant spaces of bounded measurable functions on a sphere.  Positivity, 25(5):2081–2098, Nov 2021.
  * [10] Walter Rudin.  Unitarily invariant algebras of continuous functions on spheres.  Houston J. Math., 5(2):253–265, 1979.