Master Thesis Report

Research internship with the LIG (Laboratoire d’Informatique de Grenoble), in
partnership with Atos

Self-Supervised learning for Neural Architecture Search (NAS)

Samuel Ducros
M2 SIAM - DS

Atos
1 rue de Provence
38432 Echirolles
LIG
Bâtiment IMAG, 700 Avenue Centrale
38401 Saint-Martin-d’Hères

_Tutor:_ M. Loic Pauletto
_Supervisor:_ M. Massih-Reza Amini

15 February 2022 — 13 August 2022

###### Contents

  1. 1 Introduction
    1. 1.1 Subject presentation
  2. 2 Subject development
    1. 2.1 Technologies
      1. 2.1.1 Python
      2. 2.1.2 Pytorch
      3. 2.1.3 OpenCV
    2. 2.2 State of the art
      1. 2.2.1 Semantic Segmentation
      2. 2.2.2 Semi-Supervised Learning
      3. 2.2.3 Transfer Learning
      4. 2.2.4 Multi-Task Learning
    3. 2.3 Comprehension and getting started with the code
      1. 2.3.1 Inpainting
      2. 2.3.2 Noise
      3. 2.3.3 Colorization
      4. 2.3.4 Jigsaw
    4. 2.4 Tuning the Hyper parameters
      1. 2.4.1 Normalization techniques
      2. 2.4.2 Losses
      3. 2.4.3 Learning rate / Optimizer
  3. 3 Train Protocol
    1. 3.1 mIoU (mean Intersection over Union)

## 1 - Introduction

The topic of this internship is related to Self-Supervised Learning, with the
main idea of finding innovative methods to train a neural network in order to
make a step forward in this field. A major problem that constrains our
research is the use of the smallest possible amount of annotated data to
obtain good final results. The aim is to enable new AIs to understand their
environment and task more efficiently and with the least amount of data
possible, so that they become accessible to companies that do not have the
billions of data available to Google for example.

The objective of this internship is to propose an innovative method that uses
unlabelled data, i.e. data that will allow the AI to automatically learn to
predict the correct outcome. To reach this stage, the steps to be followed can
be defined as follows: (1) consult the state of the art and position ourself
against it, (2) come up with ideas for development paths, (3) implement these
ideas, (4) and finally test them to position ourself against the state of the
art, and then start the sequence again. During my internship, this sequence
was done several times and therefore gives the tracks explored during the
internship.

The first sequence allowed me to get into the swing of things and to
understand the subject with a large section on the state of the art. The idea
that came out of it was first to speed up the execution of the code, to allow
us to do tests more quickly, and at the same time to familiarise myself with
the code.

After that, and apart from the (many) incompatible computer/connection issues,
I wanted to better understand how the different hyperparameters played on the
results rather than doing blind tests. And there are many, which allowed me to
learn a lot about the role of learning rate, norms, connections between
neurons, or losses.

### 1.1 Subject presentation

The internship focuses mainly on object segmentation on an image, i.e.
distinguish the different shapes and groups of shapes on an image (see
Semantic Segmentation). Important constraints are imposed on us:

  * $\blacktriangleright$

At the data level, when we have a dataset of images, it is very expensive to
label all the images in order to train the neural network in a supervised way.
We therefore want to take advantage of this dataset by keeping the major part
unannotated.

  * $\blacktriangleright$

Still at the data level, it is difficult to find a dataset of real images that
corresponds to our problem. However, it is easy to obtain a large number of
synthetic images, extracted from a recent and realistic video game for
example.

  * $\blacktriangleright$

At the hardware level, we have access to several GPUs and CPUs that will allow
us to accelerate our training.

Considering these constraints and our goal, we base our work on the one hand
on the research of innovative methods of Semi-Supervised Learning to take into
account unlabeled data, and on the other hand on the Domain Adaptation, that
is to say the fact of training our network for the segmentation of images on
synthetic images, of which we have a lot of data.

This report is based on the first part of the problematic on the research in
Semi-Supervised Learning for the consideration of unlabeled data. A major part
concerns the knowledge of the state of the art around Semi-Supervised Learning
techniques and around Semantic Segmentation.

## 2 - Subject development

### 2.1 Technologies

The technologies presented below were used for all the work I was able to do.
When I joined the team, the project has already been developed with the Python
language and various libraries, such as Pytorch and OpenCV. So these are the
technologies I continued with.

####  2.1.1 Python

Python111https://www.python.org is the most popular language in the world of
data analysis (data science) and artificial intelligence (machine learning,
deep learning). This craze is reflected in the large number of libraries that
allow the manipulation of mathematical objects or concepts with a high level
of positioning. Among these, we can note the use of NumPy which allows
manipulating matrices, or Scikit-learn which proposes machine learning bricks
and matplotlib for all the aspects concerning the visualization.

As the language is simple to use, untyped, and does not have a long
compilation time, we can quickly iterate by changing different parameters and
approaches. This is ideal, as the goal is not to have an optimized version of
the work since it is in the research state.

####  2.1.2 Pytorch

PyTorch222https://pytorch.org is an open source Python machine learning
software library based on Torch developed by Facebook. PyTorch allows the
tensor calculations necessary for deep learning to be performed. These
calculations are optimized and carried out either by the processor (CPU) or,
where possible, by a graphics processor (GPU) supporting CUDA. It comes from
the research teams at Facebook, and before that from Ronan Collobert in Samy
Bengio’s team at IDIAP. PyTorch is derived from an earlier software, Torch,
which was used with the Lua language. PyTorch is independent of Lua and is
programmed in Python.

####  2.1.3 OpenCV

OpenCV333https://opencv.org is a library for image and video processing. It is
a library written in C++ that also supports CUDA calls and offers a Python API
to facilitate development. It includes many of the algorithms we use, such as
object tracking within a video, image modification when preprocessing our
data, or simply capturing or displaying video content to highlight our
system’s detections.

### 2.2 State of the art

A great part of my internship was to understand the relationships between all
the possible components of a neural network, with all the parameters and
hyperparameters. Each advance in understanding brought up a new questioning on
another point. So a lot of research has been done. First of all, I had to
understand the subject and research the state of the art in general on Deep
Learning and Semantic Segmentation, which are the very first basis of the
subject. More precisely, what we are interested in is to use as little
annotated data as possible, and therefore we are interested in Self-Supervised
Learning and Semi-Supervised Learning. Here are my research results concerning
Semantic Segmentation, Self and Semi-Supervised Learning.

####  2.2.1 Semantic Segmentation

Semantic segmentation, or image segmentation, is the task of clustering parts
of an image together which belong to the same object class. It is a form of
pixel-level prediction because each pixel in an image is classified according
to a category. Some example benchmarks for this task are
Cityscapes444https://www.cityscapes-dataset.com (see Figure 1), PASCAL
VOC555http://host.robots.ox.ac.uk/pascal/VOC/ and
ADE20K666https://groups.csail.mit.edu/vision/datasets/ADE20K/. Models are
usually evaluated with the Mean Intersection-Over-Union (mIoU, see 3 - Train
Protocol).

Figure 1: Semantic segmentation of a scene from the Cityscapes dataset by
Cordts et al. (2016) recorded in Zurich.

The basic architecture in image segmentation consists of an encoder and a
decoder (see Figure 2). The encoder extracts features from the image through
filters. The decoder is responsible for generating the final output which is
usually a segmentation mask containing the outline of the object. Most of the
architectures have this architecture or a variant of it.

Figure 2: Example of architecture for image segmentation of road scenes [1]

####  2.2.2 Semi-Supervised Learning

Supervised learning usually requires a large amount of labelled data.
Obtaining good quality labelled data is a costly and time-consuming task,
especially for a complex task such as object detection, instance segmentation
and more detailed annotations are desired. On the other hand, unlabelled data
is readily available in abundance.

Semi-supervised learning (SSL), also known as learning with partially labelled
data, refers to the process of learning a prediction function from labelled
and unlabelled training samples. In this situation, the labelled instances are
expected to be few in number, resulting in an inefficient supervised model,
but the unlabelled training examples contain useful information about the
prediction problem at hand, which can be exploited to produce an efficient
prediction function. We assume that a collection of labelled training examples
derived from a joint probability distribution and a collection of unlabelled
training examples derived from the marginal distribution are both accessible
in this case. The problem arises in supervised learning if the unlabelled data
set is empty. The opposite extreme example is when the labelled training set
is empty, in which case the problem is reduced to unsupervised learning.

Smoothness is a fundamental assumption of semi-supervised learning, which
states that two instances in a high density region must have identical class
labels. This means that if two points are part of the same group or cluster,
their class labels will most likely be the same. On the other hand, if they
are separated by a low density area, their desired labels should be different.

Suppose that the instances of the same class form a partition; the unlabelled
training data could help to determine the partition boundary more efficiently
than if only labelled training examples were used. Therefore, searching for
partitions using a mixture model and then assigning class labels to groups
using the labelled data they comprise is a technique for using unlabelled data
to train a model. If two instances are in the same group, it is likely that
they belong to the same class, according to the underlying assumption, known
as the cluster assumption. This assumption can be explained as follows: if a
group is created by a large number of instances, it is rare that they all
belong to the same class. This does not imply that a class consists of only
one group of instances, but rather that two instances of distinct classes are
unlikely to be in the same group. If we consider the example partitions as
high density regions, another version of the cluster assumption is that the
decision boundary passes through low density regions, according to the
previous smoothing assumption.

Density estimation is often based on a notion of distance, which may not make
sense in high-dimensional vector spaces. To resolve this difficulty, a third
assumption known as the manifold assumption, which is supported by a number of
semi-supervised models, holds that instances in high-dimensional spaces exist
on low-dimensional topological spaces that are locally Euclidean (or geometric
manifolds).

Self-training is one of the first wraparound techniques for learning a
supervised classifier using partially labelled data. A supervised method is
first trained on the labelled training set, and its predictions are then used
to assign pseudo-labels to a portion of the unlabelled training samples. The
supervised classifier is then retrained on the augmented training set
(labelled and pseudo-labelled), and the procedure is repeated until there are
no unlabelled observations left to pseudo-label. Despite its simplicity, self-
training is difficult to analyse in general. Some studies have proposed bounds
on the error of majority-vote classifiers, used in the envelope, on unlabelled
training data. When the majority-voting classifier makes most of its errors on
low-density regions, this bound is shown to be tight [2, 3, 4, 5].

Figure 3: SSL toy example. The decision boundaries obtained on two-moons
dataset, with a supervised and different SSL approaches using 6 labeled
examples, 3 for each class, and the rest of the points as unlabeled data.
(Source: [6])

The unsupervised learning method we used to take into account the unlabelled
data is called the Self-supervised Learning [7]. It consists in creating the
input data and the target data from the unlabelled data to provide the
supervision. This task could be as simple as given the upper-half of the
image, predict the lower-half of the same image [8], or given the grayscale
version of the colored image, predict the LAB channels (for the CIELAB Color
Space [9]) of the same image [10].

In [11], the authors show the advantage of using self-supervised learning in
the context of semi-supervised learning, by introducing the Self-Supervised
Semi-Supervised Learning (S4L) framework to derive two new methods for semi-
supervised image classification.

Lately, in natural language processing, Transformer models [12] have achieved
a lot of success. Transformers like Bert [13] or T5 [14] applied the idea of
self-supervision to NLP (Natural Language Processing) tasks. They first train
the model with large unlabelled data and then fine-tuning the model with few
labelled data examples.

Figure 4: Illustration of self-supervised learning by solving jigsaw puzzle
(Source: [15])

On this basis, I further refine my research by orienting it towards Multi-Task
Learning and Transfer Learning. The topic leads us to think about the use of
unannotated data to allow the learning of the Segmentation task. The idea is
indeed to train our network with unsupervised pretext-tasks in order to
transfer knowledge for the learning of our target task.

####  2.2.3 Transfer Learning

Transfer learning [16] is one of the research fields in machine learning that
aims to transfer knowledge from one or more source tasks to one or more target
tasks. It can be seen as the ability of a system to recognize and apply
knowledge and skills, learned from previous tasks, to new tasks or domains
sharing similarities.

Figure 5: Intuitive examples about transfer learning. (Source: [16])

####  2.2.4 Multi-Task Learning

Multi-task learning (MTL) [17] is a subfield of machine learning in which
multiple learning tasks are solved at the same time, while exploiting
commonalities and differences across tasks. This can result in improved
learning efficiency and prediction accuracy for the task-specific models, when
compared to training the models separately. Early versions of MTL were called
"hints".

In a widely cited 1997 paper, Rich Caruana gave the following
characterization:

"Multitask Learning is an approach to inductive transfer that improves
generalization by using the domain information contained in the training
signals of related tasks as an inductive bias. It does this by learning tasks
in parallel while using a shared representation; what is learned for each task
can help other tasks be learned better."

Figure 6: Architecture for TCDCN [18]. The base feature extractor is made of a
series of convolutional layers which are shared between all tasks, and the
extracted features are used as input to task-specific output heads. (Source:
[17])

Multi-task learning works because regularization induced by requiring an
algorithm to perform well on a related task can be superior to regularization
that prevents overfitting by penalizing all complexity uniformly. One
situation where MTL may be particularly helpful is if the tasks share
significant commonalities and are generally slightly under sampled.

### 2.3 Comprehension and getting started with the code

The first code I did on this project was mainly optimization. Indeed, as it
was, the code allowed to launch experiments using only one pretext-task among
Inpainting and Denoising, with 3 supervised and 3 unsupervised images. By
understanding more in depth each part of the code, I could find pieces of code
repeated at each iteration unnecessarily, or (and especially) data stored
unnecessarily or which accumulated. From a program that took about 10 days to
run with a configuration C that accepted at most 3 supervised and 3
unsupervised images, we went to a program that runs for 3 days that accepts up
to 8 supervised and 8 unsupervised images for the same configuration C. At the
same time, I rewrote some of the code to make it more automatic and general,
and to anticipate future improvements, for example with for loops on pretext-
tasks instead of the if statement, to generalise all pretext-task
possibilities.

Now taking these optimizations as a base, I improved the code so that it can
take into account several pretext-tasks at the same time for the training.
Thus the input images undergo different transformations depending on the
pretext-task.

####  2.3.1 Inpainting

Existing works for image inpainting [8] can be mainly divided into two groups.
The first group represents traditional diffusion-based or patch-based methods
with low-level features. The second group, that we use, attempts to solve the
inpainting problem by a learning-based approach, e.g. training deep
convolutional neural networks to predict pixels for the missing regions.

As described above, the network that learns the inpainting task is constituted
of an common encoder with the other tasks, and a decoder head of its own. The
input image is the grand truth image from which we erase a square. After
having got through the encoder and the decoder, the prediction is compared
with the grand truth image using Mean Squared Error Loss (see Losses).

Figure 7: Example inpainting results of the method of [8] on images of natural
scene, face and texture. Missing regions are shown in white. In each pair, the
left is input image and right is the direct output of the trained generative
neural networks without any post-processing. (Source: [8])

####  2.3.2 Noise

Many different ways of denoising methods exist [19]. From filters to deep
learning, many researches have been done in this area. Here we use deep
learning and, as the inpainting task, we build an encoder network and a
decoder network specific for the denoising task. The input image is the grand
truth which we added a virtual noise. After having got through the network,
the prediction is compared to this grand truth using Mean Squared Error Loss
(see Losses).

Figure 8: Denoised images of the real noisy image Nikon D800 ISO 6400 1 [20]
by different methods (Source: [21])

####  2.3.3 Colorization

Colorization [10] is the process of adding plausible color information to
monochrome photographs or videos. As the denoising problem, there are several
ways to resolve this problem [22]. Here we need to train the colorization with
our network. As before, a decoder head is specific to the colorization task.
To learn it in a unsupervised way, we give as input a colored image which we
transformed in a grayscale image. The prediction colors are compared with the
grand truth with

####  2.3.4 Jigsaw

The Jigsaw problem [15] is the process of solving a puzzle of an image, i.e.
find the original order between the different tiles extracted from an image
(see Figure 9). Its decoder head is of the same type as the decoder head of a
classification problem, because the problem here is to well classify every
part of the puzzle.

Figure 9: Learning image representations by solving Jigsaw puzzles. (a) The
image from which the tiles (marked with green lines) are extracted. (b) A
puzzle obtained by shuffling the tiles. Some tiles might be directly
identifiable as object parts, but others are ambiguous (e.g., have similar
patterns) and their identification is much more reliable when all tiles are
jointly evaluated. In contrast, with reference to (c), determining the
relative position between the central tile and the top two tiles from the left
can be very challenging. (Source: [15])

### 2.4 Tuning the Hyper parameters

Now that we have these new features and the code works well and is optimized,
we want to run tests and get the best possible results. But before that, we
have to choose the right hyperparameters to test. It took a lot of research to
understand their impact and to know the different possible arrangements. Here
I summarize my research results on the main hyperparameters.

####  2.4.1 Normalization techniques

Normalization techniques can decrease your model’s training time by a huge
factor. It normalizes each feature so that they maintains the contribution of
every feature, as some feature has higher numerical value than others. This
way our network can be unbiased (to higher value features). Batch Norm for
example makes loss surface smoother (i.e. it bounds the magnitude of the
gradients much more tightly) [23]. It makes the Optimization faster because
normalization does not allow weights to explode all over the place and
restricts them to a certain range. A last unintended benefit of Normalization
is that it helps network in Regularization (only slightly, not significantly).
Therefore getting Normalization right can be a crucial factor in getting your
model to train effectively.

Let’s dive into details of each normalization technique one by one.

  * •

Batch Normalization: Batch normalization [24] is a method that normalizes
activations in a network across the mini-batch of definite size. For each
feature, batch normalization computes the mean and variance of that feature in
the mini-batch. It then subtracts the mean and divides the feature by its
mini-batch standard deviation. We can add $\gamma$ and $\beta$ as scale and
shift learn-able parameters respectively, in order to take into account a
greater magnitude of the weights if necessary. This all can be summarized as:

Let $\mathcal{B}=\\{x_{1...m}\\}$ be the mini-batch, containing the features
of each data of the batch. Let $\gamma$ and $\beta$ two parameters to learn.
We have, with $\epsilon$ is the stability constant in the equation:

$\displaystyle\mu_{\mathcal{B}}$
$\displaystyle=\dfrac{1}{m}\sum_{i=1}^{m}x_{i}$
$\displaystyle\sigma_{\mathcal{B}}^{2}$
$\displaystyle=\dfrac{1}{m}\sum_{i=1}^{m}(x_{i}-\mu_{\mathcal{B}})^{2}$
$\displaystyle\hat{x}_{i}$
$\displaystyle=\dfrac{x_{i}-\mu_{\mathcal{B}}}{\sqrt{\sigma_{\mathcal{B}}^{2}+\epsilon}}$
$\displaystyle BN_{\gamma,\beta}(x_{i})$
$\displaystyle=\gamma\hat{x}_{i}+\beta$

  * •

Layer Normalization: Layer Normalization differs from the Batch Normalization
because it normalizes input across the features instead of normalizing input
features across the batch dimension.

  * •

Instance(or Contrast) Normalization: Layer normalization and Instance
normalization are very similar to each other but the difference between them
is that Instance normalization normalizes across each channel in each training
example instead of normalizing across input features in an training example.

  * •

Group Normalization: Group Normalization normalizes over group of channels for
each training example.

We summarize here these norms in the schema in Figure 10 [25]. For the project
we mainly employed Switchable Normalization (SN)[26], which is a normalization
method that uses a weighted average of different mean and variance statistics
from batch normalization, instance normalization, and layer normalization.
Switch Normalization can outperform Batch normalization on tasks such as image
classification and object detection. [26] shows that the instance
normalization is used more often in earlier layers, batch normalization is
preferred in the middle and layer normalization being used in the last more
often. Smaller batch sizes lead to a preference towards layer normalization
and instance normalization.

Figure 10: Normalization methods. Each subplot shows a feature map tensor,
with N as the batch axis, C as the channel axis, and (H, W ) as the spatial
axes. The pixels in blue are normalized by the same mean and variance,
computed by aggregating the values of these pixels. (Source: [25])

####  2.4.2 Losses

In the context of an optimization algorithm, the function used to evaluate a
candidate solution (i.e. a set of weights) is referred to as the objective
function. We may seek to maximize or minimize the objective function, meaning
that we are searching for a candidate solution that has the highest or lowest
score respectively. Typically, with neural networks, we seek to minimize the
error. As such, the objective function is often referred to as a cost function
or a loss function and the value calculated by the loss function is referred
to as simply “loss.”

The cost function reduces all the various good and bad aspects of a possibly
complex system down to a single number, a scalar value, which allows candidate
solutions to be ranked and compared. In calculating the error of the model
during the optimization process, a loss function must be chosen. This can be a
challenging problem as the function must capture the properties of the problem
and be motivated by concerns that are important to the project and
stakeholders. It is important, therefore, that the function faithfully
represent our design goals. If we choose a poor error function and obtain
unsatisfactory results, the fault is ours for badly specifying the goal of the
search.

Here are the different loss functions we use for our project, depending on the
task that is trained.

  * $\blacktriangleright$

Cross-Entropy: When modeling a classification problem where we are interested
in mapping input variables to a class label, we can model the problem as
predicting the probability of an example belonging to each class. In a binary
classification problem, there would be two classes, so we may predict the
probability of the example belonging to the first class. In the case of
multiple-class classification, we can predict a probability for the example
belonging to each of the classes. In the training dataset, the probability of
an example belonging to a given class would be 1 or 0, as each sample in the
training dataset is a known example from the domain. We know the answer.
Therefore, under maximum likelihood estimation, we would seek a set of model
weights that minimize the difference between the model’s predicted probability
distribution given the dataset and the distribution of probabilities in the
training dataset. This is called the cross-entropy. For the project, the
cross-entropy is used as loss function for tasks as Jigsaw, Colorization and
Semantic Segmentation.

  * $\blacktriangleright$

Mean Squared Error (MSE): Mean squared error (MSE) is the most commonly used
loss function for regression. The loss is the mean overseen data of the
squared differences between true and predicted values. MSE is sensitive
towards outliers and given several examples with the same input feature
values, the optimal prediction will be their mean target value. MSE is thus
good to use if you believe that your target data, conditioned on the input, is
normally distributed around a mean value, and when it’s important to penalize
outliers extra much. We use MSE when doing regression, believing that your
target, conditioned on the input, is normally distributed, and want large
errors to be significantly (quadratically) more penalized than small ones. For
the project then, MSE is used as loss function for tasks as Inpainting and
Denoising.

####  2.4.3 Learning rate / Optimizer

The learning rate [27] is a hyperparameter that controls how much to change
the model in response to the estimated error each time the model weights are
updated. Choosing the learning rate is challenging as a value too small may
result in a long training process that could get stuck, whereas a value too
large may result in learning a sub-optimal set of weights too fast or an
unstable training process (see Figure 11).

Figure 11: Consequences of the learning rate (Source: [27])

The stochastic gradient descent (SGD) algorithm is a common optimizer
algorithm for the training. SGD is an optimization algorithm that estimates
the error gradient for the current state of the model using examples from the
training dataset, then updates the weights of the model using the back-
propagation of errors algorithm, referred to as simply backpropagation. Other
learning rate optimization algorithms can be used as ADAM [28].

The learning rate controls how quickly the model is adapted to the problem.
Smaller learning rates require more training epochs given the smaller changes
made to the weights each update, whereas larger learning rates result in rapid
changes and require fewer training epochs. A learning rate that is too large
can cause the model to converge too quickly to a suboptimal solution, whereas
a learning rate that is too small can cause the process to get stuck. The
challenge of training deep learning neural networks involves carefully
selecting the learning rate. It may be the most important hyperparameter for
the model.

To best chose the learning, we use a learning rate schedule, that changes the
learning rate during learning and is most often changed between
epochs/iterations. This is mainly done with two parameters: decay and
momentum. Decay serves to settle the learning in a nice place and avoid
oscillations, a situation that may arise when a too high constant learning
rate makes the learning jump back and forth over a minimum, and is controlled
by a hyperparameter. Momentum is analogous to a ball rolling down a hill; we
want the ball to settle at the lowest point of the hill (corresponding to the
lowest error). Momentum both speeds up the learning (increasing the learning
rate) when the error cost gradient is heading in the same direction for a long
time and also avoids local minima by ’rolling over’ small bumps. Momentum is
controlled by a hyper parameter analogous to a ball’s mass which must be
chosen manually—too high and the ball will roll over minima which we wish to
find, too low and it will not fulfil its purpose.

For our project we use mainly the SGD optimization algorithm.

## 3 - Train Protocol

Finally we have a functional code and we understand the role of
hyperparameters. Now it’s time to launch tests. Our goal is to compare
ourselves to the state of the art regarding the segmentation results, and to
make our method exceed these results. Let’s explain how we compare.

### 3.1 mIoU (mean Intersection over Union)

There are several neural network models working on different platforms, and
different unique approaches for object detection and semantic image
segmentation, so we need to know how to choose one among all in order to have
better results in our field. There has to be a criterion based on which such
decision can be made. The best one is by checking the degree of similarity of
the output produced by such methods with the ground truth and that can be done
in a mathematical way by calculating IoU (Intersection over Union) between the
two. This method takes into account the region common to both (ground truth
and predicted output) and computes to what percentage it has similarity with
the actual one.

It’s quite simple in case of “Single-class based Semantic Image Segmentation”
but not in the case of other “Multiple-class based Semantic Image
Segmentation”, as in the case of Pascal VOC challenge (with 21 classes) where
there can be objects belonging to different classes in the same image. In such
cases each object has to be given a different label and has to be treated
accordingly during IoU computation. A method which can take into account such
cases and find out the overall IoU for multiple classes present in an image is
the calculation of the mean value of IoUs corresponding to different classes
which would match with the actual degree of similarity. This mean value is
regarded as mean IoU (mIoU).

Here is a common approx for the computation of the mIoU, which we use in our
code. For that we need the labelled matrix of both predicted result and
expected one (ground truth). Let two matrices $GT$ and $Pred$, one
representing the actual segmented output and the other predicted by any neural
network or model. The elements of these matrices are the labels representing
different classes to which pixels, at that particular location on the image
belong. Then here are the steps to follow:

  * $\blacktriangleright$

Finding out the frequency count of each class for both the matrix, $F_{GT}$
and $F_{Pred}$,

  * $\blacktriangleright$

Converting the matrices to 1D format, $GT_{1D}$ and $Pred_{1D}$

  * $\blacktriangleright$

Finding out the category matrix, of size $(nb_{classes}\times nb_{classes})$.
The category matrix is one that will have the elements as the category numbers
to which the pixels at that particular location belong.
$Categ=(nb_{classes}\times GT_{1D})+Pred_{1D}$

  * $\blacktriangleright$

Constructing the confusion matrix. A confusion matrix is a
$(nb_{classes}\times nb_{classes})$ size matrix which stores the information
about the number of pixels belonging to a particular category. The frequency
count of the ‘category’ array gives a linear array which on reshaping to
$(nb_{classes}\times nb_{classes})$ gives us the confusion matrix.
$CM=Categ.reshape((nb_{classes}\times nb_{classes}))$

  * $\blacktriangleright$

Calculating IoU for individual classes. The diagonal of the confusion matrix
represents the common region. So, these elements are the intersection values
of the predicted output and ground truth. $I=diag(CM)$ $U=GT_{1D}+Pred_{1D}-I$

  * $\blacktriangleright$

Calculating MIoU for the actual-predicted pair. $IoU=\dfrac{I}{U}$ is a matrix
of size $nb_{c}lasses$. $mIoU=mean(IoU)$ is the mean over all values of $IoU$.

In case of multiple classes, the mIoU has to be calculate rather than just
calculating IoU by treating all the different classes as a single one. So,
considering all the classes mIoU has to be calculated for validation.

There are several other methods to find out the similarity between an actual
image and predicted result, most popular among them is the bounding box method
[29] but the instances involving fine edge detection along with segmentation
where higher accuracy would be required, this method proves to be reliable.

## References

  * [1] Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla.  Segnet: A deep convolutional encoder-decoder architecture for image segmentation, 2016.
  * [2] Vasilii Feofanov, Emilie Devijver, and Massih-Reza Amini.  Transductive bounds for the multi-class majority vote classifier.  In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI, pages 3566–3573, 2019.
  * [3] Massih R. Amini, Nicolas Usunier, and François Laviolette.  A transductive bound for the voted classifier with an application to semi-supervised learning.  In Advances in Neural Information Processing Systems, volume 21, pages 65–72, 2009.
  * [4] Nicolas Usunier, Massih-Reza Amini, and Cyril Goutte.  Multiview semi-supervised learning for ranking multilingual documents.  In Machine Learning and Knowledge Discovery in Databases - European Conference, ECML PKDD, pages 443–458, 2011.
  * [5] Anastasia Krithara, Massih-Reza Amini, Jean-Michel Renders, and Cyril Goutte.  Semi-supervised document classification with a mislabeling error model.  In 30th European Conference on Information Retrieval (ECIR’05), pages 370–381, 2008.
  * [6] Yassine Ouali, Céline Hudelot, and Myriam Tami.  An overview of deep semi-supervised learning, 2020.
  * [7] Weng L.  Self-supervised representation learning.  https://lilianweng.github.io/lil-log/2019/11/10/self-supervised-learning.html, November 2019.
  * [8] Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S. Huang.  Generative image inpainting with contextual attention, 2018.
  * [9] Cielab color space.  https://en.wikipedia.org/wiki/CIELAB_color_space.
  * [10] Richard Zhang, Phillip Isola, and Alexei A. Efros.  Colorful image colorization, 2016.
  * [11] Xiaohua Zhai, Avital Oliver, Alexander Kolesnikov, and Lucas Beyer.  S4l: Self-supervised semi-supervised learning, 2019.
  * [12] Weng L.  The transformer family.  https://lilianweng.github.io/lil-log/2020/04/07/the-transformer-family.html, April 2020.
  * [13] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding, 2019.
  * [14] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer, 2020.
  * [15] Mehdi Noroozi and Paolo Favaro.  Unsupervised learning of visual representations by solving jigsaw puzzles, 2017.
  * [16] Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He.  A comprehensive survey on transfer learning, 2020.
  * [17] Michael Crawshaw.  Multi-task learning with deep neural networks: A survey, 2020.
  * [18] Lisha Chen, Hui Su, and Qiang Ji.  Deep structured prediction for facial landmark detection, 2020.
  * [19] Chunwei Tian, Lunke Fei, Wenxian Zheng, Yong Xu, Wangmeng Zuo, and Chia-Wen Lin.  Deep learning on image denoising: An overview, 2020.
  * [20] Seonghyeon Nam, Youngbae Hwang, Yasuyuki Matsushita, and Seon Joo Kim.  A holistic approach to cross-channel image noise modeling and its application to image denoising.  In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1683–1691, 2016.
  * [21] Jun Xu, Lei Zhang, and David Zhang.  A trilateral weighted sparse coding scheme for real-world image denoising, 2018.
  * [22] Patricia Vitoria, Lara Raad, and Coloma Ballester.  Chromagan: Adversarial picture colorization with semantic class distribution, 2020.
  * [23] Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry.  How does batch normalization help optimization?, 2019.
  * [24] Aakash Bindal.  Normalization techniques in deep neural networks.  https://medium.com/techspace-usict/normalization-techniques-in-deep-neural-networks-9121bf100d8, February 2019.
  * [25] Yuxin Wu and Kaiming He.  Group normalization, 2018.
  * [26] Ping Luo, Zhanglin Peng, Jiamin Ren, and Ruimao Zhang.  Do normalization layers in a deep convnet really need to be distinct?, 2018.
  * [27] Jordan J.  Setting the learning rate of your neural network.  https://www.jeremyjordan.me/nn-learning-rate/, March 2018.
  * [28] Bushaev V.  Adam — latest trends in deep learning optimization.  https://towardsdatascience.com/adam-latest-trends-in-deep-learning-optimization-6be9a291375c, October 2018.
  * [29] Bishwo Adhikari and Heikki Huttunen.  Iterative bounding box annotation for object detection, 2020.


# A Fubini–type limit theorem for the integrated hyperuniform infinitely
divisible moving averages

Evgeny Spodarev Institute of Stochastics, Ulm University, Germany;
<EMAIL_ADDRESS>
###### Abstract

This short note shows a limiting behavior of integrals of some centered
antipersistent stationary infinitely divisible moving averages as the compact
integration domain in $d\geq 1$ dimensions extends to the whole positive
quadrant $\mathbb{R}^{d}_{+}$. Namely, the weak limit of their finite
dimensional distributions is again a moving average with the same infinitely
divisible purely jump integrator measure (i.e., possessing no Gaussian
component), but with an integrated kernel function. The results apply equally
to time series ($d=1$) as well as to random fields ($d>1$). Apart from the
existence of the expectation, no moment assumptions on the moving average are
imposed allowing it to have an infinite variance as e.g. in the case of
$\alpha$–stable moving averages with $\alpha\in(1,2)$ . If the field is
additionally square integrable, its covariance integrates to zero
(hyperuniformity).

Keywords: Stationary random field, moving average, infinitely divisible, ambit
random field, integral spectral representation, $\alpha$–stable, time series,
heavy tails, hyperuniformity, antipersistence.

AMS subject classification 2020: Primary 60F05; Secondary 60G60, 60E07, 60E10

##  1 Introduction

The major object of interest of this note is a real-valued stationary
measurable infinitely divisible (ID, for short) moving average random field
$X=\\{X(t),t\in\mathbb{R}^{d}\\}$ on a $d$–dimensional Euclidean index space
defined by its spectral representation

$X(t)=\int_{\mathbb{R}^{d}}f(t-x)\ \Lambda(dx),\quad
t\in\mathbb{R}^{d},\;d\geq 1,$ (1)

where $\Lambda$ is a centered stationary ID independently scattered purely
jump random measure and $f:\mathbb{R}^{d}\to\mathbb{R}$ is a suitable
$\Lambda$–integrable kernel. Such random functions are often used as
mathematical models in many application contexts varying from turbulent liquid
flows [4] (where they form a special case of ambit random fields, see [29, 3]
for a review) to radial growth of trees and tumors [20, 21]. An important
special case of (1) constitute continuous–time ($d=1$) stationary moving
averages governed by Lévy processes (e.g. the so–called CARMA [13]) which are
widely used in mathematical insurance and finance [5, 12, 9, 27, 39]. They
often do not have a finite variance describing high volatility of financial
risks. Whereas the limiting behaviour of some classes of functionals (both
linear and non–linear) of random fields (1) is relatively well understood if
the integrator $\Lambda$ is Gaussian (see e.g. [23, 25] and references
therein) or has at least finite second or higher moments [1, 35, 14, 16], the
case of a heavy–tailed $\Lambda$ is far less investigated. Few central limit
theorems are available [19, 28, 22, 33, 8, 2] in the case of short range
dependent $X$ as well as some results of non–central type [6, 7, 34] for $X$
with long memory. The novel contribution of this note is a weak limit of
finite dimensional distributions (FDDs, for short) of the sample integral of
$X$ over a cubic infinitely growing sliding domain when we only assume the
existence of the mean $\mathbf{E}X(0)=0$. The kernel function $f$ has to be
integrable over $\mathbb{R}^{d}$ to zero and factorize as

$f(x)=\prod_{j=1}^{d}g_{j}^{\prime}(x_{j}),\quad x=(x_{1},\ldots,x_{d}),$

where differentiable functions $g_{j}:\mathbb{R}\to\mathbb{R}$ belong to the
Sobolev space $W^{1,1}(\mathbb{R})$ a.e., $j=1,\ldots,d$. In case of square
integrable $X$, the above form of $f$ implies that the covariance function of
$X$ also integrates to zero. This phenomenon (widely known in limit theorems
for random functions as an intermediate regime between short and long memory)
is called antipersistence [10]. In physics, it is termed hyperuniformity and
plays an important role in considering the geometric structure of matter (such
as e.g. quasicrystals) which exhibits some regularity while being essentially
random [36, 36, 24]. Recent progress in search for hyperuniformity has been
made in experimental physics, materials science and biology [37, 38].

We show that all FDDs of sample averages

$\int_{[0,T]^{d}+l}X(t)\,dt,\qquad l\in\mathbb{R}^{d},\quad T>0,$

converge as $T\to+\infty$ to the FDDs of the random field

$(-1)^{d}\int_{\mathbb{R}^{d}}\prod_{j=1}^{d}g_{j}(l_{j}-x_{j})\,\Lambda(dx_{1},\ldots,dx_{d}),\quad
l=(l_{1},\ldots,l_{d})\in\mathbb{R}^{d}.$

For simplicity, this result is first proven for the case of stochastic time
series ($d=1$, See Theorem 1) and is then generalized to all $d>1$ in Theorem
2. Interestingly enough, the limiting process belongs to the same family of ID
moving averages with integrator $\Lambda$, but with a different kernel. This
Fubini–type result for sample path integrals of $X$ (valid due to the
linearity of integration operator combined with the convergence of the
corresponding characteristic functions) can be used to understand the
behaviour of spatial averages of hyperuniform structures. To some extent, it
can be seen as a generalization of the Fubini theorem [32, Theorem 4.1] for
the sample path integrals of $\alpha$–stable processes. For symmetric ID
processes, a result with a similar Fubini flavor can be found in [11, Theorem
5.1].

The paper is organized as follows: in Section 2, common notation used
throughout the note is given. Section 3 recalls some basic facts from the
theory of infinitely divisible random functions with integral spectral
representation needed in the sequel. In Section 4 our main results are
formulated and proved in Theorems 1 and 2. Remark 1 explains why the
considered random fields are antipersistent and hyperuniform. The main results
are illustrated by some examples.

##  2 Notation

Denote by ${\cal B}_{\mathbb{R}^{d}}$ the $\sigma$-algebra of Borel subsets of
$\mathbb{R}^{d}$, $d\geq 1$. Let $|\cdot|_{d}$ be the Lebesgue measure
(volume) in $\mathbb{R}^{d}$. In the sequel, the expression almost everywhere
(a.e.) will be understood with respect to $|\cdot|_{d}$. Let
$L^{p}(\mathbb{R}^{d})$ be the set of all real-valued functions that are
absolutely integrable on $\mathbb{R}^{d}$ with their power $p>0$. We use the
standard notation $W^{1,1}(\mathbb{R})$ for the Sobolev space $\\{g\in
L^{1}(\mathbb{R}):g^{\prime}\in L^{1}(\mathbb{R})\\}$, where $g^{\prime}$ is
the derivative of $g$. Let $\mbox{\rm sgn}(x)$ be the sign function of a real
argument $x$. Denote by $\mathbbm{1}(A)$ the indicator function of a set $A$.

##  3 Preliminaries on ID moving averages

Let $X=\\{X(t),t\in\mathbb{R}^{d}\\}$ be a real-valued measurable ID moving
average random field defined on a probability space $(\Omega,{\cal
F},\mathbb{P})$ by the integral representation

$X(t)=\int_{\mathbb{R}^{d}}f(t-x)\ \Lambda(dx),\quad t\in\mathbb{R}^{d},$ (2)

where $\Lambda$ is a centered stationary ID independently scattered purely
jump random measure with Lebesgue control measure. The absence of Gaussian
part means that the characteristic function $\varphi_{\Lambda(B)}$ for bounded
Borel subsets $B$ of $\mathbb{R}^{d}$ has the Lévy-Khintchin representation

$\log\varphi_{\Lambda(B)}(z)=|B|_{d}\int_{\mathbb{R}}\left(e^{izy}-1-izy\chi(y)\right)\,\nu(dy),\quad
z\in\mathbb{R},$ (3)

where $\chi(y)=\mathbbm{1}(|y|\leq 1)$, $y\in\mathbb{R}$, and $\nu(\cdot)$ is
the Lévy measure on $\mathbb{R}$:
$\int_{\mathbb{R}}\min\\{y^{2},1\\}\nu(dy)<\infty$. The kernel function $f$
above is $\Lambda$–integrable, i.e. satisfies the following two conditions
(cf. [30, Theorem 2.7, p. 461]):

  1. (i)

$\int\limits_{\mathbb{R}^{d}}|f(s)|\left|\int\limits_{\\{1\leq|x|\leq|f(s)|^{-1}\\}}x\,\nu(dx)-\int\limits_{\\{|f(s)|^{-1}\leq|x|\leq
1\\}}x\,\nu(dx)\right|\,ds<\infty,$

  2. (ii)

$\int\limits_{\mathbb{R}^{d}}\int\limits_{\\{|x|\geq|f(s)|^{-1}\\}}\nu(dx)\,ds<\infty,$

  3. (iii)

$\int\limits_{\mathbb{R}^{d}}|f(s)|^{2}\int\limits_{\\{|x|<|f(s)|^{-1}\\}}x^{2}\,\nu(dx)\,ds<\infty.$

######  Example 1 ([8], p.3).

Let $\Lambda$ be symmetric with Lévy measure $\nu(\cdot)$ having a density

$\psi(x)\leq\frac{C}{|x|^{1+\beta}},\quad
x\in\mathbb{R}\setminus\\{0\\},\quad\beta\in(0,2),$

where $C>0$ is a constant. The conditions (i) –(iii) are satisfied whenever
$f\in L^{\beta}(\mathbb{R}^{d})$, e.g. for

$|f(x)|\leq
K\left(|x|^{\gamma}\mathbbm{1}(|x|<1)+|x|^{-\alpha}\mathbbm{1}(|x|\geq
1)\right),\quad x\in\mathbb{R}^{d},\quad K,\alpha>0,\quad\gamma\in\mathbb{R},$

with $\alpha\beta>d$, $\gamma\beta>-d$.

By [31, Proposition 2.10], it holds

$X(t)\stackrel{{\scriptstyle
d}}{{=}}\int_{\mathbb{R}^{d+1}}yf(t-s)\left[\eta(ds,dy)-\chi(yf(t-s))ds\,\nu(dy)\right]+b,\quad
t\in\mathbb{R}^{d},$ (4)

where $\eta(\cdot,\cdot)$ is a Poisson point process on
$\mathbb{R}^{d}\times\mathbb{R}$ with intensity measure $ds\,\nu(dy)$, and

$b=\int_{\mathbb{R}^{d+1}}yf(s)\left[\chi(yf(s))-\chi(y)\right]ds\,\nu(dy).$

The log characteristic function of $X(t)$ writes ([30, p. 462])

$\begin{split}\log\mathbf{E}e^{izX(t)}&=iz\int_{\mathbb{R}^{d+1}}yf(t-s)\left[\chi(yf(t-s))-\chi(y)\right]ds\,\nu(dy)\\\
&+\int_{\mathbb{R}^{d+1}}\left[e^{izyf(t-s)}-1-izyf(t-s)\chi(yf(t-s))\right]ds\,\nu(dy),\quad
z\in\mathbb{R}.\end{split}$

Now assume

$f\in L^{1}(\mathbb{R}^{d}),$ (5) $\int_{\mathbb{R}}|y|\,\nu(dy)<\infty,$ (6)

which evidently imply that conditions (i), (ii) hold true and also yield the
integrability of $X(t)$. Under these assumptions, the above characteristic
function simplifies to

$\log\mathbf{E}e^{izX(t)}=-iza+\int_{\mathbb{R}^{d+1}}\left[e^{izyf(t-s)}-1\right]ds\,\nu(dy),\quad
z\in\mathbb{R},$ (7)

where

$a=\int_{\mathbb{R}^{d}}f(s)\,ds\int_{-1}^{1}y\,\nu(dy).$

Interpreting (7), we may write

$X(t)\stackrel{{\scriptstyle
d}}{{=}}\int_{\mathbb{R}^{d+1}}yf(t-s)\eta(ds,dy)-a,\quad t\in\mathbb{R}^{d}$
(8)

as a stationary shot noise random field. If we (additionally to conditions
(5), (6)) require

$f\in L^{2}(\mathbb{R}^{d}),\quad\int_{\mathbb{R}}y^{2}\,\nu(dy)<\infty$ (9)

the field $X$ becomes square integrable with covariance function
$C(t)=\mathbf{E}[X(0)X(t)]$ equal to

$C(t)=\int_{\mathbb{R}}y^{2}\,\nu(dy)\int_{\mathbb{R}^{d}}f(-x)f(t-x)\,dx,\quad
t\in\mathbb{R}^{d},$ (10)

compare [22, Relation 2.44].

##  4 Main results

First, consider the case $d=1$ of stationary time series moving averages (2).
Introduce the sample path integral of $X$ by

$S_{T,l}:=\int_{l}^{T+l}X(t)\,dt,\qquad l\in\mathbb{R},\quad T>0.$

If $\mathbf{E}|X(0)|<\infty$, $S_{T,l}$ exists a.s. by measurability of $X$
and and is integrable by stationarity and Fubini’s theorem:

$\mathbf{E}|S_{T,l}|\leq\int_{l}^{T+l}\mathbf{E}|X(t)|\,dt=T\mathbf{E}|X(0)|<\infty.$

######  Theorem 1.

Let $X=\\{X(t),t\in\mathbb{R}\\}$ be the stationary measurable ID moving
average with integral representation (2) and kernel function
$f(x)=g^{\prime}(x)$ a.e. on $\mathbb{R}$, where $g\in W^{1,1}(\mathbb{R})$
a.e. Assuming that conditions (6) and (iii) hold, one gets the weak
convergence of all FDDs of $\\{S_{T,l},\;l\in\mathbb{R}\\}$ as $T\to+\infty$
to those of $\\{-\int_{\mathbb{R}}g(l-s)\Lambda(ds),\quad l\in\mathbb{R}\\}$.

######  Remark 1.

  1. 1.

Since $f(x)=g^{\prime}(x)$ a.e. on $\mathbb{R}$ and $g\in L^{1}(\mathbb{R})$,
it holds $\int_{\mathbb{R}}f(x)\,dx=\int_{\mathbb{R}}dg(x)=0$ and thus $a=0$
in representation (8). This yields that $X$ is centered: $\mathbf{E}X(t)=0$ .

  2. 2.

As the kernel function $f$ reflects the dependence structure of the process
$X$, the case $\int_{\mathbb{R}}f(x)\,dx=0$ may be called antipersistence,
similar to the case $\int_{\mathbb{R}}C(t)\,dt=0$ for stationary square
integrable moving averages $X$ with covariance function $C$. Indeed, assuming
the conditions (5), (6), (9) the relation (10) for $C(t)$ yields

$\int_{\mathbb{R}}C(t)\,dt=\int_{\mathbb{R}}y^{2}\,\nu(dy)\int_{\mathbb{R}}f(-x)\int_{\mathbb{R}}f(t-x)\,dt\,dx=\int_{\mathbb{R}}y^{2}\,\nu(dy)\left(\int_{\mathbb{R}}f(x)\,dx\right)^{2}=0$

by Fubini theorem. Hence, the condition $\int_{\mathbb{R}}f(x)\,dx=0$
indicates a phenomenon called in physics hyperuniformity which is
characterized by very regular shapes of sample paths of a time series $X$.

###### Proof of Theorem 1.

For any $m\in\mathbb{N},$ $z_{1},\ldots,z_{m}\in\mathbb{R}$, write

$\log\mathbf{E}e^{i\sum_{j=1}^{m}z_{j}S_{T,l_{j}}}=\log\mathbf{E}e^{i\int_{\mathbb{R}}\sum_{j=1}^{m}z_{j}\mathbbm{1}(t\in[l_{j},T+l_{j}])X(t)\,dt}=\int_{\mathbb{R}^{2}}\left(e^{iyJ_{T}(s)}-1\right)ds\,\nu(dy),$

where

$\begin{split}J_{T}(s):=&\int_{\mathbb{R}}\sum_{j=1}^{m}z_{j}\mathbbm{1}(t\in[l_{j},T+l_{j}])f(t-s)\,dt=\sum_{j=1}^{m}z_{j}\int_{l_{j}-s}^{T+l_{j}-s}f(x)\,dx\\\
&\longrightarrow\sum_{j=1}^{m}z_{j}\int_{l_{j}-s}^{+\infty}f(x)\,dx=-\sum_{j=1}^{m}z_{j}g(l_{j}-s)\quad\mbox{as
}T\to+\infty.\end{split}$

Hence,

$\log\mathbf{E}e^{i\sum_{j=1}^{m}z_{j}S_{T,l_{j}}}\longrightarrow\int_{\mathbb{R}^{2}}\left(e^{-iy\sum_{j=1}^{m}z_{j}g(l_{j}-s)}-1\right)ds\,\nu(dy),\quad
T\to+\infty,$

and finally

$\left(S_{T,l_{1}},\ldots,S_{T,l_{m}}\right)\stackrel{{\scriptstyle
d}}{{\longrightarrow}}-\left(\int_{\mathbb{R}^{2}}yg(l_{1}-s)\,\eta(ds,dy),\ldots,\int_{\mathbb{R}^{2}}yg(l_{m}-s)\,\eta(ds,dy)\right)\quad\mbox{as
}T\to+\infty.$

This together with (8) finishes the proof.

∎

######  Example 2.

  1. 1.

Functions $g(x)=e^{-|x|}$, $g(x)=e^{-P_{k}(x)}$, $x\in\mathbb{R}$, belong to
$C^{1}(\mathbb{R})\cap W^{1,1}(\mathbb{R})$ a.e. where $P_{k}(x)$ is a
polynomial of even degree $k\geq 2$ with a positive coefficient at $x^{k}$.
The corresponding ID time series are

$X(t)=\int_{\mathbb{R}}\mbox{\rm
sgn}(t-x)e^{-|t-x|}\,\Lambda(dx)=e^{-t}\int_{-\infty}^{t}e^{x}\,\Lambda(dx)-e^{t}\int^{+\infty}_{t}e^{-x}\,\Lambda(dx),\quad
t\in\mathbb{R},$

which we may call (in analogy to Gaussian integrators $\Lambda$) a signed
two–sided ID Ornstein–Uhlenbeck process, and

$X(t)=\int_{\mathbb{R}}P_{k}^{\prime}(t-x)e^{-P_{k}(t-x)}\,\Lambda(dx),\quad
t\in\mathbb{R}.$

  2. 2.

An important integrator measure $\Lambda$ is the so–called Dickman
subordinator with Lévy measure
$\nu(dy)=\frac{1}{y}\mathbbm{1}\left(y\in(0,1)\right)dy$. For this $\nu$, it
can be easily seen that conditions (6), (9) hold true. Taking a kernel
function $f\in L^{1}(\mathbb{R})\cap L^{2}(\mathbb{R})$ yields a square
integrable time series (2). Additionally assuming
$\int_{-\infty}^{\cdot}f(x)\,dx\in L^{1}(\mathbb{R})$ makes Theorem 1 hold
true for this $X$. See references [15, 18] for more information on the main
properties and use of Dickman subordinators.

  3. 3.

Examples of non–$\alpha$–stable integrator measures $\Lambda$ with finite
mean, but infinite variance can be given e.g. by Inverse Gamma subordinator
$IG(\alpha,\beta)$ and Student $t_{\tau}$–distributed random measures with a
specific choice of form parameters $\alpha,\tau\in(1,2]$ and an arbitrary
$\beta>0$. Their Lévy measures are given by

$\nu(y)=\frac{1}{\pi^{2}}\int_{0}^{\infty}\frac{e^{-sy}\,ds}{s\left(J^{2}_{\alpha}(2\sqrt{\beta
s})+Y^{2}_{\alpha}(2\sqrt{\beta s})\right)}\,\frac{dy}{y},\quad y>0,$

and

$\nu(y)=\frac{2^{3/4}}{\pi^{5/2}\sqrt{|y|}}\int_{0}^{\infty}\frac{K_{1/2}\left(\sqrt{2s}|y|\right)\,ds}{s^{3/4}\left(J^{2}_{\tau/2}(\sqrt{2\tau
s})+Y^{2}_{\tau/2}(\sqrt{2\tau s})\right)}\,{dy},\quad y\in\mathbb{R},$

respectively (cf. [5, 17, 26]), where $J_{\alpha},$ $Y_{\alpha},$ are Bessel
functions of order $\alpha$ of the first and second kind, and $K_{\alpha}$ is
a modified Bessel function of order $\alpha$ of the second kind.

  4. 4.

Set $m=1$, $l_{1}=0$ in the proof of Theorem 1: we have

$S_{T}:=S_{T,0}=\int_{0}^{T}X(t)\,dt\stackrel{{\scriptstyle
d}}{{\longrightarrow}}\int_{\mathbb{R}^{2}}yg(s)\,\eta(ds,dy)\quad\mbox{as
}T\to+\infty,$

since $-\int_{\mathbb{R}^{2}}yg(-s)\,\eta(ds,dy)\stackrel{{\scriptstyle
d}}{{=}}\int_{\mathbb{R}^{2}}yg(s)\,\eta(ds,dy)$ by substitution $s\mapsto-s$.

Theorem 1 can be easily generalized to stationary ID random fields $X$ with
spectral representation (2) and the dimension of index space $d>1$. For a
sample integral

$S_{T,l}:=\int_{[0,T]^{d}+l}X(t)\,dt,\qquad l\in\mathbb{R}^{d},\quad T>0,$

we formulate the following

######  Theorem 2.

Let $X=\\{X(t),t\in\mathbb{R}^{d}\\}$ be a stationary measurable ID random
field with spectral representation (2) and kernel function
$f(x)=\prod_{j=1}^{d}f_{j}(x_{j})$, $x=(x_{1},\ldots,x_{d}),$
$f_{j}(x_{j})=g_{j}^{\prime}(x_{j})$ a.e. on $\mathbb{R}$, where $g_{j}\in
W^{1,1}(\mathbb{R})$ a.e., $j=1,\ldots,d$. Under conditions (6) and (iii) it
holds the weak convergence of all FDDs of $\\{S_{T,l},\;l\in\mathbb{R}^{d}\\}$
as $T\to+\infty$ to those of $\\{Y_{l},\;l\in\mathbb{R}^{d}\\}$, where

$Y_{l}=(-1)^{d}\int_{\mathbb{R}^{d}}\prod_{j=1}^{d}g_{j}(l_{j}-x_{j})\,\Lambda(dx_{1},\ldots,dx_{d}),\quad
l=(l_{1},\ldots,l_{d})\in\mathbb{R}^{d}.$

The proof of Theorem 2 is analogous to the above case $d=1$ of Theorem 1 and
is thus omitted.

Remark 1 evidently stays valid for the ID random fields considered in Theorem
2 since the $d$–dimensional integration for kernels
$f(x)=\prod_{j=1}^{d}f_{j}(x_{j})$ splits into $d$ univariate integrals with
respect to variables $x_{j}$, $j=1,\ldots,d$. Hence, stationary ID moving
averages

$X(t)=\int_{\mathbb{R}^{d}}\prod_{j=1}^{d}g^{\prime}_{j}(t_{j}-x_{j})\,\Lambda(dx_{1},\ldots,dx_{d}),\quad
t=(t_{1},\ldots,t_{d})\in\mathbb{R}^{d}$

for $g_{j}\in C^{1}(\mathbb{R})\cap W^{1,1}(\mathbb{R})$ a.e., $j=1,\ldots,d$
are antipersistent and hyperuniform.

## Acknowledgement

Evgeny Spodarev gratefully acknowledges the partial financial support of his
research visit to Cardiff University in November 2021 provided by the London
Mathematical Society (Grant No. 42007) and German Research Society (Grant No.
39087941). He also thanks Michael Klatt, Nikolai Leonenko and Alexander
Lindner for fruitful discussions and some literature references.

## References

  * [1] F. Avram and M. S. Taqqu.  Noncentral limit theorems and Appell polynomials.  Ann. Probab., 15(2):767–775, 1987.
  * [2] E. Azmoodeh, M. M. Ljungdahl, and C. Thäle.  Multi-dimensional normal approximation of heavy-tailed moving averages.  Stochastic Process. Appl., 145:308–334, 2022.
  * [3] O. E. Barndorff-Nielsen, F. E. Benth, and A. E. D. Veraart.  Ambit stochastics, volume 88 of Probability Theory and Stochastic Modelling.  Springer, Cham, 2018.
  * [4] O. E. Barndorff-Nielsen and J. Schmiegel.  Ambit processes; with applications to turbulence and tumour growth.  In Stochastic Analysis and Applications: The Abel Symposium 2005, pages 93–124. Springer, 2007.
  * [5] O. E. Barndorff-Nielsen and N. Shephard.  Non-Gaussian Ornstein-Uhlenbeck-based models and some of their uses in financial economics.  J. R. Stat. Soc. Ser. B Stat. Methodol., 63(2):167–241, 2001.
  * [6] A. Basse-O’Connor, R. Lachièze-Rey, and M. Podolskij.  Power variation for a class of stationary increments Lévy driven moving averages.  Ann. Probab., 45(6B):4477–4528, 2017.
  * [7] A. Basse-O’Connor, V. Pilipauskaitė, and M. Podolskij.  Power variations for fractional type infinitely divisible random fields.  Electron. J. Probab., 26:Paper No. 55, 35, 2021.
  * [8] A. Basse-O’Connor, M. Podolskij, and C. Thäle.  A Berry-Esseén theorem for partial sums of functionals of heavy-tailed moving averages.  Electron. J. Probab., 25:Paper No. 31, 31, 2020.
  * [9] F. E. Benth, C. Klüppelberg, G. Müller, and L. Vos.  Futures pricing in electricity markets based on stable CARMA spot models.  Energy Economics, 44(C):392–406, 2014.
  * [10] J. Beran, Y. Feng, S. Ghosh, and R. Kulik.  Long-memory processes.  Springer, Heidelberg, 2013.  Probabilistic properties and statistical methods.
  * [11] M. Braverman and G. Samorodnitsky.  Symmetric infinitely divisible processes with sample paths in Orlicz spaces and absolute continuity of infinitely divisible processes.  Stochastic Process. Appl., 78(1):1–26, 1998.
  * [12] P. J. Brockwell and A. Lindner.  Ornstein-Uhlenbeck related models driven by Lévy processes.  In Statistical methods for stochastic differential equations, volume 124 of Monogr. Statist. Appl. Probab., pages 383–427. CRC Press, Boca Raton, FL, 2012.
  * [13] P. J. Brockwell and A. Lindner.  Continuous-Parameter Time Series, volume 98 of de Gruyter Studies in Mathematics.  De Gruyter, 2024.
  * [14] A. Bulinski, E. Spodarev, and F. Timmermann.  Central limit theorems for the excursion set volumes of weakly dependent random fields.  Bernoulli, 18(1):100–118, 2012.
  * [15] F. Caravenna, R. Sun, and N. Zygouras.  The Dickman subordinator, renewal theorems, and disordered systems.  Electron. J. Probab., 24:Paper No. 101, 40, 2019.
  * [16] I. V. Curato, R. Stelzer, and B. Ströh.  Central limit theorems for stationary random fields under weak dependence with application to ambit and mixed moving average fields.  Ann. Appl. Probab., 32(3):1814–1861, 2022.
  * [17] B. Grigelionis.  Student’s $t$-distribution and related stochastic processes.  SpringerBriefs in Statistics. Springer, Heidelberg, 2013.
  * [18] N. Gupta, A. Kumar, N. Leonenko, and J. Vaz.  Generalized fractional derivatives generated by Dickman subordinator and related stochastic processes.  Fractional Calculus and Applied Analysis, https://doi.org/10.1007/s13540-024-00289-x, 2024.
  * [19] T. Hsing.  On the asymptotic distributions of partial sums of functionals of infinite-variance moving averages.  Ann. Probab., 27(3):1579–1599, 1999.
  * [20] K. Y. Jónsdóttir and E. B. V. Jensen.  Gaussian radial growth.  Image Anal. Stereol., 24(2):117–126, 2005.
  * [21] K. Y. Jónsdóttir, J. Schmiegel, and E. B. Vedel Jensen.  Lévy-based growth models.  Bernoulli, 14(1):62–90, 2008.
  * [22] W. Karcher.  On infinitely divisible random fields with an application in insurance.  Dissertation, University of Ulm, 2012.
  * [23] N. N. Leonenko and M. D. Ruiz-Medina.  Sojourn functionals for spatiotemporal Gaussian random fields with long memory.  J. Appl. Probab., 60(1):148–165, 2023.
  * [24] Z. Ma and S. Torquato.  Random scalar fields and hyperuniformity.  Journal of Applied Physics, 121(24):244904, 06 2017.
  * [25] V. Makogin and E. Spodarev.  Limit theorems for excursion sets of subordinated Gaussian random fields with long-range dependence.  Stochastics, 94(1):111–142, 2022.
  * [26] T. Massing.  Simulation of Student-Lévy processes using series representations.  Comput. Statist., 33(4):1649–1685, 2018.
  * [27] R. C. Noven, A. E. D. Veraart, and A. Gandy.  A Lévy-driven rainfall model with applications to futures pricing.  AStA Adv. Stat. Anal., 99(4):403–432, 2015.
  * [28] V. Pipiras and M. S. Taqqu.  Central limit theorems for partial sums of bounded functionals of infinite-variance moving averages.  Bernoulli, 9(5):833–855, 2003.
  * [29] M. Podolskij.  Ambit fields: survey and new challenges.  In XI Symposium on Probability and Stochastic Processes, volume 69 of Progr. Probab., pages 241–279. Birkhäuser/Springer, Cham, 2015.
  * [30] B. S. Rajput and J. Rosiński.  Spectral representations of infinitely divisible processes.  Probab. Theory Related Fields, 82(3):451–487, 1989.
  * [31] J. Rosiński.  Representations and isomorphism identities for infinitely divisible processes.  Ann. Probab., 46(6):3229–3274, 2018.
  * [32] G. Samorodnitsky.  Integrability of stable processes.  Probab. Math. Statist., 13(2):191–204, 1992.
  * [33] E. Spodarev.  Limit theorems for excursion sets of stationary random fields.  In Modern stochastics and applications, volume 90 of Springer Optim. Appl., pages 221–241. Springer, Cham, 2014.
  * [34] D. Surgailis.  Stable limits of empirical processes of moving averages with infinite variance.  Stochastic Process. Appl., 100:255–274, 2002.
  * [35] D. Surgailis.  Stable limits of sums of bounded functions of long-memory moving averages with finite variance.  Bernoulli, 10(2):327–355, 2004.
  * [36] S. Torquato.  Hyperuniformity and its generalizations.  Phys. Rev. E, 94:022122, Aug 2016.
  * [37] S. Torquato.  Hyperuniform states of matter.  Physics Reports, 745:1–95, 2018.  Hyperuniform States of Matter.
  * [38] S. Torquato, J. Kim, and M. A. Klatt.  Local number fluctuations in hyperuniform and nonhyperuniform systems: Higher-order moments and distribution functions.  Phys. Rev. X, 11:021028, May 2021.
  * [39] A. E. D. Veraart.  Modelling the impact of wind power production on electricity prices by regime-switching Lévy semistationary processes.  In Stochastics of environmental and financial economics—Centre of Advanced Study, Oslo, Norway, 2014–2015, volume 138 of Springer Proc. Math. Stat., pages 321–340. Springer, Cham, 2016.