# Leveraging Latent Diffusion Models for Training-Free In-Distribution Data
Augmentation for Surface Defect Detection ††thanks:  This study was carried
out within the PNRR research activities of the consortium iNEST
(Interconnected North-Est Innovation Ecosystem) funded by the European Union
Next-GenerationEU (Piano Nazionale di Ripresa e Resilienza (PNRR) – Missione 4
Componente 2, Investimento 1.5 – D.D. 1058 23/06/2022, ECS_00000043). This
manuscript reflects only the Authors’ views and opinions. Neither the European
Union nor the European Commission can be considered responsible for them.
Furthermore, this study was also partially funded by the European Union -
NextGenerationEU (Italiadomani, Piano Nazionale di Ripresa e Resilienza (PNRR)
- Missione 4 - Componente 2, Investimento 3.3 - progetto M4C2).

Federico Girella, Ziyue Liu, Franco Fummi, Francesco Setti, Marco Cristani,
Luigi Capogrosso<EMAIL_ADDRESS>Department of Engineering for
Innovation Medicine, University of Verona, Italy

###### Abstract

Defect detection is the task of identifying defects in production samples.
Usually, defect detection classifiers are trained on ground-truth data formed
by normal samples (negative data) and samples with defects (positive data),
where the latter are consistently fewer than normal samples. State-of-the-art
data augmentation procedures add synthetic defect data by superimposing
artifacts to normal samples to mitigate problems related to unbalanced
training data. These techniques often produce out-of-distribution images,
resulting in systems that learn what is not a normal sample but cannot
accurately identify what a defect looks like. In this work, we introduce DIAG,
a training-free Diffusion-based In-distribution Anomaly Generation pipeline
for data augmentation. Unlike conventional image generation techniques, we
implement a human-in-the-loop pipeline, where domain experts provide
multimodal guidance to the model through text descriptions and region
localization of the possible anomalies. This strategic shift enhances the
interpretability of results and fosters a more robust human feedback loop,
facilitating iterative improvements of the generated outputs. Remarkably, our
approach operates in a zero-shot manner, avoiding time-consuming fine-tuning
procedures while achieving superior performance. We demonstrate the efficacy
and versatility of DIAG with respect to state-of-the-art data augmentation
approaches on the challenging KSDD2 dataset, with an improvement in AP of
approximately 18% when positive samples are available and 28% when they are
missing. The source code is available at
https://github.com/intelligolabs/DIAG.

###### Index Terms:

Diffusion Models, Data Augmentation, Surface Defect Detection.

##  I Introduction

Figure 1: The DIAG pipeline. Starting from positive samples, we leverage a
Latent Diffusion Model (LDM) to synthesize novel in-distribution high-quality
images of defective surfaces based on defect localization and textual prompts.
These synthetic images are then used as anomaly samples to train a binary
classifier for anomaly detection.

Surface Defect Detection (SDD) is a challenging problem in industrial
scenarios, defined as the task of individuating samples containing a defect
[1], i.e., samples that do not conform to a prototypical texture. In many
real-world applications, a human expert inspects every product and removes
those defective pieces. Unfortunately, training human experts can be
expensive. Moreover, humans are relatively slow in accomplishing this task,
and their performances are subject to stress and fatigue.

Automated defect detection systems [2, 3] can easily overcome most of these
issues by learning classifiers on defective and nominal training products. The
main drawback is the data collection process required to train a model
effectively. Indeed, defective items (i.e., positive samples) are relatively
rare compared to nominal items (i.e., negative samples). Thus, the user may
need to collect massive amounts of data to have enough positive samples.
Moreover, with the rise of the Industry 4.0 paradigm and the transition
towards flexible manufacturing processes, there is an increasing demand for
systems that can quickly adapt to new production setups [4], i.e., customized
products manufactured in small batches. Traditional automated systems cannot
comply with these demands since data collection could easily involve the whole
batch size.

Recent studies on SDD focused on limiting the impact of the labeling process
by formulating the problem under the unsupervised learning paradigm [5, 6] or
training exclusively on nominal samples [7], possibly using few-shot learning
strategies [8]. In both cases, the goal is to generate an accurate model of
the nominal sample distribution and classify everything with a low probability
score as anomalies. However, due to the limited restoration capability of
these models, these approaches tend to generate many false positives,
especially on datasets with complex structures or textures [9].

It is worth noting that, in industrial setups, anomalies are not generated by
Gaussian processes but are the outcome of specific, often predictable, issues
during the production process. Consequently, the anomalous samples are not
randomly distributed outside the nominal distribution; they can be modeled as
a mixture of Gaussian distributions in the feature space instead. Expert
operators can easily define the main problems they can expect from the
manufacturing process, such as which kind of defects, in which locations, and
how often they expect them to appear. Thus, generative AI can represent a
powerful tool for SDD, with defect image generation emerging as a promising
approach to enhance detector performance. Specifically, in recent years,
Denoising Diffusion Probabilistic Models (DDPMs) [10] received significant
attention as a powerful class of generative models.

In this paper, we propose an interactive learning protocol where a vision
language model is used to generate realistic images starting from textual
prompts. Specifically, we promote using DDPMs to produce fine-grained
realistic defect images that can be used as positive samples to train an
anomaly detection model. We name our approach DIAG, a training-free Diffusion-
based In-distribution Anomaly Generation pipeline for data augmentation in the
SDD task. By leveraging pre-trained DDPMs with multimodal conditioning, we can
exploit domain experts’ knowledge to generate plausible anomalies without
needing real positive data. When using these augmented images to train an
anomaly detection model, we show a notable increase in the detection
performance compared to previous state-of-the-art augmentation pipelines.
Figure 1 outlines our approach, which will be explained in Section III.

The main contributions of our work are as follows:

  * •

We present DIAG, a complete pipeline for training anomaly detection models
based on nominal images and textual prompts. We showcase the superior outcomes
achieved by utilizing generated defective samples compared to previous state-
of-the-art approaches.

  * •

We dive into spatial control approaches to enable the synthesis of defect
samples incorporating regional information and exhibit enhanced
controllability of the image generation through a human-in-the-loop pipeline,
effectively utilizing domain expertise to generate more plausible in-
distribution anomalies.

##  II Related Work

Surface defect detection refers to identifying and categorizing
irregularities, flaws, or imperfections on the surface of materials or
objects. These defects include scratches, cracks, discolorations, and any
other anomaly that deviates from the expected surface quality. Research has
been conducted according to different setups: unsupervised approaches [11] use
a mixture of unlabelled positive and negative sample images for training;
supervised approaches require labeled samples in the form of binary masks
representing the defects (full supervision) [12] or simply as a tag for the
whole image (weak supervision) [13]. Supervised methods demonstrated superior
accuracy in the identification of anomalies. Nevertheless, the effort required
to provide good annotations is not always justified. Collecting positive
samples can be time and resource-consuming due to the low rate of defective
products generated by industrial lines.

Thus, many recent approaches adopt a “ _clean_ ” setup, where the training set
consists of only nominal samples. Two strategies can be adopted in clean
setups: model fitting and image generation. Model fitting approaches aim at
generating an accurate model of the nominal distribution, considering an
outlier every sample with a likelihood lower than –or a distance from the
nominal prototype higher than– a predefined threshold [7, 14]. On the
contrary, data augmentation approaches leverage generative methods to
synthesize images of defects and use these images as positive samples for
training a supervised model. Specifically, this work focuses on generation-
based data augmentation under clean setups.

The most popular data augmentation pipeline for SDD consists of a series of
random standard transformations of the input image –such as mirroring,
rotations, and color changes– followed by the super-imposition of noisy
patches [15].

In [16], an ablation study focused on the generation of synthetic anomalies
leads to the following findings: i) adding synthetic noise images is never
counterproductive, it just diminishes the effectiveness in percentage; ii) few
generated anomaly images (in the order of tens) are enough to increase the
performance substantially; iii) textural injection in the anomalies is
essential, or, equivalently, adding uniformly colored patches is ineffective.

In MemSeg [15], the pipeline for the generation of the abnormal synthetic
examples is divided into three steps: i) a Region of Interest (ROI) indicating
where the defect will be located is generated using Perlin noise and the
target foreground; ii) the ROI is applied to a noise image to generate a noise
foreground ROI; iii) the noise foreground ROI is super-imposed on the original
image to obtain the simulated anomalous image. However, all these approaches
are based on generating out-of-distribution patterns that do not faithfully
represent the target-domain anomalies.

In [17], the authors introduced the concept of “extended anomalies”, where the
specific abnormal regions of the seen anomalies are placed at any possible
position within the normal sample after applying random spatial
transformations. Unfortunately, this requires an accurate segmentation of the
training images, an operation we may want to avoid.

More recently, the first work that draws attention to in-distribution defect
data is In&Out [18], in which the authors empirically show that diffusion
models provide more realistic in-distribution defects.

In this paper, we significantly improve the generation of in-distribution
anomalous samples incorporating domain knowledge provided by an expert user
through textual prompts and localization of salient regions. We will use
state-of-the-art MemSeg [15] and In&Out [18] methods as competitors for our
augmentation pipeline. With DIAG, we can produce a distribution of defective
images closer to the real one, resulting in a more precise identification of
the decision boundaries in the classification step.

##  III Methodology

In this section, we provide detailed explanations of DIAG. In particular,
Section III-A covers techniques for diffusion-based image generation, Section
III-B showcases the anomalous image generation pipeline, and Section III-C
outlines the anomaly detection model training procedure.

###  III-A Multimodal diffusion-based image generation

DDPMs [19, 10] are a class of deep latent variable models that work by
modeling the joint distribution of the data over a Markovian inference
process. This process consists of small perturbations of the data with a
variance-preserving property [20], such that the limit distribution after the
diffusion process is approximately identical to a known prior distribution.
Starting with samples from the prior, a reverse diffusion process is learned
by gradually denoising the sample to resemble the initial data by the end of
the procedure.

Formally, the data distribution $q(x_{0})$ is modelled through a latent
variable model $p_{\theta}(x_{0})$:

$\displaystyle p_{\theta}(x_{0})=\int p_{\theta}(x_{0:T})dx_{1:T}\;,$ (1)
$\displaystyle\quad
p_{\theta}(x_{0:T}):=p_{\theta}(x_{T})\prod_{t=1}^{T}p^{(t)}_{\theta}(x_{t-1}|x_{t})\;,$
(2)

where $x_{1},\ldots,x_{T}$ are latent variables of the same dimensionality as
$x_{0}$.

The parameters $\theta$ are learned by maximizing an ELBO of the log evidence,
i.e.:

$\displaystyle\max_{\theta}\mathbb{E}_{q(x_{0})}[\log p_{\theta}(x_{0})]\leq$
$\displaystyle\max_{\theta}\mathbb{E}_{q(x_{0},x_{1},\ldots,x_{T})}\left[\log
p_{\theta}(x_{0:T})-\log q(x_{1:T}|x_{0})\right]\;,$ (3)

where $q(x_{1:T}|x_{0})$ represents a fixed inference process defined as the
following as a Markov chain:

$\displaystyle q(x_{1:T}|x_{0}):=\prod_{t=1}^{T}q(x_{t}|x_{t-1})\;,$ (4)
$\displaystyle
q(x_{t}|x_{t-1}):=\mathcal{N}\left(\sqrt{\frac{\alpha_{t}}{\alpha_{t-1}}}x_{t-1},\left(1-\frac{\alpha_{t}}{\alpha_{t-1}}\right)I\right)\;,$
(5)

where $\alpha_{1:T}\in(0,1]^{T}$ is a predefined variance schedule, and the
covariance matrix is ensured to have positive terms on its diagonal.
Specifically, this parametrization has the property:

$\displaystyle q(x_{t}|x_{0})=\int q(x_{1:t}|x_{0})dx_{1:(t-1)}=$
$\displaystyle\mathcal{N}(x_{t};\sqrt{\alpha_{t}}x_{0},(1-\alpha_{t})I)\;,$
(6)

therefore we can write $x_{t}$ as a linear combination of $x_{0}$ and a noise
variable $\epsilon$.

When we set $\alpha_{T}$ sufficiently close to $0$, $q(x_{T}|x_{0})$ converges
to a standard Gaussian for all $x_{0}$, so it is natural to set
$p_{\theta}(x_{T}):=\mathcal{N}(0,\mathbf{I})$. Given that all the
conditionals are modeled as Gaussians with fixed variance, the objective in
Equation (3) can be greatly simplified. In particular, [10] shows that the
following (further simplified) lower bound provides optimal generative
performance:

$\displaystyle
L(\epsilon_{\theta}):=\sum_{t=1}^{T}\mathbb{E}_{x_{0},\epsilon_{t}}\left[\lVert{\epsilon_{\theta}^{(t)}(\sqrt{\alpha_{t}}x_{0}+\sqrt{1-\alpha_{t}}\epsilon_{t})-\epsilon_{t}}\rVert_{2}^{2}\right]\;,$
(7)

where $x_{0}\sim q(x_{0}),\epsilon_{t}\sim\mathcal{N}(0,I)$,
$\epsilon_{\theta}=\\{\epsilon_{\theta}^{(t)}\\}_{t=1}^{T}$ is a set of $T$
functions, with each $\epsilon_{\theta}^{(t)}:X\to X$ having trainable
parameters $\theta^{(t)}$.

In practice, these functions are approximated by a neural network conditioned
on the diffusion time $t$. After the model is trained, we can generate new
samples by first sampling $x_{T}$ from the known prior $p_{\theta}(x_{T})$,
and then iteratively reversing the diffusion process, thereby sampling
$\\{x_{T-1}\ldots x_{0}\\}$.

In addition, we leveraged the natural ability of DDPMs to incorporate
multimodal conditioning in the generation process, taking inspiration from
[21, 22, 23, 24]. Specifically, we will use prompts, i.e., textual
descriptions of the anomaly, and negative prompts, i.e., prompts that guide
the image generation “away” from its concepts. This results in high-quality
images that comply with the given descriptions [25, 26, 27].

In particular, we opt to utilize an inpainting model, as demonstrated in [19,
22]. Given an image with a masked region, inpainting seamlessly fills it with
content that harmonizes with the surrounding image. Although typically
employed to eliminate undesired artifacts, the inpainting process ensures that
the masked area incorporates the provided prompt, effectively merging textual
and visual content.

###  III-B The DIAG pipeline

To generate an anomalous image $i_{a}$, the process starts by sampling a
random negative image, an anomaly description, and a mask, forming the triplet
$(i_{n},d_{a},m_{a})$. Instead of directly operating on the image pixels using
DDPM, we use a Latent Diffusion Model (LDM) to work in a lower-dimensional
latent space [22]. Thus, the above information will be fed to a text-
conditioned LDM to perform inpainting on image $i_{n}$ using the mask $m_{a}$.

The anomaly description $d_{a}$ guides the generation, filling the masked
region of $i_{n}$ with an anomaly that complies with the prompt. To generate
images resembling real anomalous samples, domain knowledge from industrial
experts is exploited, providing textual descriptions of the potential
anomalies’ type, shape, and spatial information.

The LDM is then conditioned on this information to inpaint plausible anomalies
on defect-free samples. Formally, given pictures of defect-free (negative)
samples $I_{n}$, domain experts will provide textual descriptions $D_{a}$ of
what different anomalies may look like. At the same time, regions where these
anomalies may appear on the defect-free samples will be designated. We define
this set of regions as a set of binary masks $M_{a}$ of possible anomalies,
shapes, and locations. The result of this operation is $i_{a}$, an anomalous
version of $i_{n}$, where an anomaly has been inpainted in the masked region
$m_{a}$. Due to the stochastic nature of LDMs, this process can be repeated
multiple times to generate an augmented set of anomalous sample images
$I_{a}$. Finally, the set $I_{a}$ can be used as data augmentation for
training anomaly detection models, as presented in the following section.

###  III-C Anomaly detection task

We approach the anomaly detection problem as a binary classification problem,
where the objective is to predict whether a sample belongs to one of two
classes. Specifically, we utilized a ResNet-50 [28] backbone trained with a
binary cross-entropy loss function denoted as $\mathcal{L}_{\text{BCE}}$. The
binary cross-entropy loss measures the dissimilarity between the predicted
probability distribution and the actual distribution of the labels.
Mathematically, it is defined as:

$\mathcal{L}_{\text{BCE}}(y,\hat{y})=-\frac{1}{N}\sum_{i=1}^{N}\left[y_{i}\log(\hat{y}_{i})+(1-y_{i})\log(1-\hat{y}_{i})\right]\;,$
(8)

where, $y$ represents the ground truth labels, $\hat{y}$ represents the
predicted probabilities, and $N$ is the number of samples. In detail, $y_{i}$
denotes the true label for sample $i$, which can be either 0 or 1, while
$\hat{y}_{i}$ signifies the predicted probability that sample $i$ belongs to
class 1.

##  IV Experiments

In this section, we show the efficacy of our data augmentation approach for
defect detection from a quantitative and qualitative point of view.

###  IV-A Experiment setup

#### Datasets

We use the Kolektor Surface-Defect Dataset 2 (KSDD2) [13], one of the most
recent, complex, and real-world surface defect detection datasets. This
dataset comprises 246 positive and 2085 negative images in the training set
and 110 positive and 894 negative images in the testing set. Positive images
are images with visible defects, such as scratches, spots, and surface
imperfections.

Since the images have different dimensions, we standardize the dataset
resolution, resizing all the images to $224\times{}632$ pixels while keeping
the number of normal and anomalous samples unchanged.

#### Evaluation metrics

The anomaly detection performance was evaluated based on Average Precision
(AP), Precision, and Recall, following the evaluation protocol defined in
[18].

Additionally, to evaluate the visual similarity between generated images and
the original dataset images, we employ the Fréchet Inception Distance (FID)
[29], a popular metric in the image generation field which computes the
distance between the distribution of two sets of images. More specifically,
the Fréchet distance calculates distance $d(.,.)$ between a Gaussian with mean
$(m,C)$ obtained from $p(.)$ and a Gaussian with mean $(m_{w},C_{w})$ obtained
by $p_{w}(.)$, where $p_{w}(.)$ represents real world data and $p(.)$
represents generated data. In practice, these distributions are two sets of
data: the “world” data (i.e., the images in a dataset) and the “generated”
data (i.e., the generated images). These sets are then fed to an Inception
model pre-trained on ImageNet to extract deep features from each sample of the
distributions. The resulting two sets of features represent the Gaussians with
mean $(m_{w},C_{w})$ and $(m,C)$ for the “world” and “generated” data,
respectively. Specifically, [29] shows that a lower FID score matches a
human’s higher perceived visual similarity (a lower perceptual distance),
meaning that similar sets of images will have a lower FID than dissimilar
sets. Formally, the FID score is:

$d((m,C),(m_{w},C_{w}))=||m-m_{w}||^{2}_{2}+\\\
\text{T}(C+C_{w}-2(CC_{w})^{\frac{1}{2}})$ (9)

where T refers to the trace linear algebra operation.

###  IV-B Implementation details

In this section, we specify all the implementation details for
reproducibility. All training and inferences were conducted on an NVIDIA RTX
3090 GPU.

#### Inpainting via DIAG

We use the pre-trained implementation of SDXL [27] from Diffusers [30] as our
text-conditioned LDM. In particular, SDXL shows drastically improved
performance compared to the previous versions of Stable Diffusion [22] and
achieves results comparable to commercial state-of-the-art image generators.

Following the procedure outlined in Section III-B, we use the negative images
of KSDD2 as the set $I_{n}$. As the set of anomaly descriptions $D_{a}$, we
used the prompts “white marks on the wall” and “copper metal scratches”.
Instead, “smooth, plain, black, dark, shadow” were used as a negative prompt
to improve the performance further. These prompts were selected through a
human-in-the-loop iterative pipeline, until the resulting images resembled
plausible anomalies. We used the segmentation masks of positive samples in the
KSDD2 dataset to simulate the domain experts’ definition of plausible
anomalous regions.

Then, these data are fed to the pre-trained SDXL model to perform inpainting
on the negative images in a training-free process, generating the set of
augmented anomalous images $I_{a}$ as described in Section III-B.

Finally, the generated images $I_{a}$ are added to the training set, which
will be used to train the anomaly detection model.

#### ResNet-50 training and testing

For a fair comparison with [18], we use the same PyTorch implementation of the
ResNet-50 [28] as our anomaly detection model, in which we substitute the
fully connected layers after the backbone to make it a binary classifier. The
network is trained for 50 epochs with Adam [31] as an optimizer, a learning
rate of $0.0001$, and a batch size of 32.

To maintain consistency with the training and evaluation procedures of KSDD2,
our setup is the same as presented in [13, 18], where only the images and
ground truth labels are used to train the model. For our comparison, we use
the official code of In&Out [18] to fine-tune a DDPM in the full-shot scenario
(on all the positive images of the KSDD2 dataset) and generate their augmented
images. Likewise, we follow the procedure of MemSeg [15] to generate the “per-
region” augmented images. Finally, we generate DIAG augmented images,
following the inpainting methodology outlined in Section III. The set of
images used for training changes depending on the experiment and the pipelines
being tested, but in general, it can be seen as a combination of the original
negative images $I_{n}$, an optional set $I_{p}$ of original positive images,
and the set of generated positive images $I_{a}$.

###  IV-C Quantitative results

#### Zero-shot data augmentation

TABLE I: Results between MemSeg, In&Out and DIAG when no anomalous samples are available. In bold, the best results. Underlined, the second best. Model | $\mathbf{N_{aug}}$ | AP $\uparrow$ | Precision $\uparrow$ | Recall $\uparrow$
---|---|---|---|---
MemSeg [15] | 80 | .514 | .733 | .436
MemSeg [15] | 100 | .388 | .633 | .432
MemSeg [15] | 120 | .511 | .683 | .470
In&Out [18] | 80 | .556 | .530 | .655
In&Out [18] | 100 | .626 | .742 | .568
In&Out [18] | 120 | .536 | .699 | .534
DIAG (ours) | 80 | .769 | .851 | .673
DIAG (ours) | 100 | .801 | .924 | .664
DIAG (ours) | 120 | .739 | .944 | .609

Here, we emulate the situation where no original positive samples are
available in the training set. This scenario makes generating augmented
positive samples necessary and restricts the users to augmentation procedures
that do not rely on positive images. To do this, we build the set of augmented
anomalous images $I_{a}$ by generating $N_{aug}$ augmented positive samples
with different pipelines, i.e., MemSeg [15], In&Out [18] and DIAG. Then, we
train a ResNet-50 on a dataset that includes the original negative samples
$I_{n}$ and the augmented positive samples $I_{a}$. Finally, we evaluate the
model on the original test set.

Table I reports the comparison between the models trained with MemSeg, In&Out,
and DIAG augmented data at different values of $N_{aug}$. As we can see, our
proposed method achieves the highest AP (.801), recorded at 100 augmented
images, while also resulting in a consistently higher AP when compared to the
MemSeg and In&Out pipelines. These impressive results highlight how, through
domain expertise in the form of anomaly descriptions and segmentation masks,
it is possible to generate in-distribution images able to meaningfully guide
an anomaly detection network, even in a complicated scenario where no real
anomalous data is available.

Surprisingly, the DIAG performance with $N_{aug}=120$ augmented images is
lower than using a smaller number of augmented images. We hypothesize this is
due to the stochastic nature of the LDMs image generation. While it allows the
generation of various images given the same guidance, it can also lower, in
some cases, the predictability of the quality of the generated samples, which
sometimes may not faithfully comply with the prompt. Future works will focus
on studying quality consistency in the image generation pipeline.

#### Full-shot data augmentation

TABLE II: Results between MemSeg, In&Out and DIAG when all the anomalous samples are available. In bold, the best results. Underlined, the second best. Model | $\mathbf{N_{aug}}$ | AP $\uparrow$ | Precision $\uparrow$ | Recall $\uparrow$
---|---|---|---|---
MemSeg [15] | 80 | .744 | .851 | .691
MemSeg [15] | 100 | .774 | .814 | .752
MemSeg [15] | 120 | .734 | .772 | .707
In&Out [18] | 80 | .747 | .764 | .734
In&Out [18] | 100 | .775 | .868 | .720
In&Out [18] | 120 | .782 | .906 | .689
DIAG (ours) | 80 | .869 | .912 | .755
DIAG (ours) | 100 | .911 | .978 | .800
DIAG (ours) | 120 | .924 | .896 | .864

To showcase DIAG as a general data augmentation technique, we also explore the
scenario where real positive samples are available in the training set. To
this aim, we include all the 246 real positive samples $I_{p}$ in the training
set, together with the real negative images $I_{n}$ and the $N_{aug}$
augmented positive images $I_{a}$.

As we can see from Table II, DIAG achieves the highest average AP yet (.924),
surpassing the .782 set by the previous state-of-the-art data augmentation
pipeline [18]. When comparing these results to the ones obtained in the “zero-
shot data augmentation” scenario, it is clear how more in-distribution images
improve model performance during training. This is highlighted by the
improvement in performance of all the models when adding the real positive
images $I_{p}$ to the training set. At the same time, the inclusion of DIAG
augmented images allows the model to explore the anomaly distribution further,
resulting in the difference in performance between the different data
augmentation pipelines.

###  IV-D Qualitative results

Figure 2: First row displays some negative samples from the KSDD2 dataset.
Instead, the second row shows some images of positive samples from the same
dataset. In the third row, we show the MemSeg-generated defect samples. The
fourth row shows In&Out generated defect samples. Lastly, the final row
showcases images generated with DIAG. Notably, the defect images that DIAG
generated are more realistic and in-distribution.

The main goal of our data augmentation pipeline is to generate in-distribution
synthetic positive images, meaning images that closely resemble the real ones.
Figure 2 shows qualitative results. It’s evident that the images produced by
DIAG are markedly more realistic compared to those generated by MemSeg [15]
and In&Out [18].

In addition, we provide a numeric evaluation of the similarity between the
generated images and the real ones by employing FID [29]. It is worth noting
that due to the limited number of anomalous images in the original dataset, we
are forced to calculate FID on a different network layer, precisely the second
max pooling layer. This is a common procedure in cases where the number of
images is low, as the calculation requires the number of samples (images) to
be higher than the number of features. Note that this only changes the
magnitude of the values obtained, not the metric’s general behavior. In the
specific case of KSDD2, we choose the first and second max-pooling layers with
64 and 192 features, respectively. Specifically, we compare the images
generated with MemSeg, In&Out, and DIAG with the ones available in the KSDD2
dataset and compute the FID scores between the positive images of the KSDD2
and the previously mentioned sets of augmented images.

TABLE III: FID scores between the real positive images of KSDD2 and the images generated by MemSeg, In&Out and DIAG. The scores are calculated using the first and second max pooling layers of the Inception network, having 64 and 192 features, respectively. In bold, the best results. Augmentation procedure | FID 64 $\downarrow$ | FID 192 $\downarrow$
---|---|---
MemSeg [15] | 0.834 | 4.376
DDPM [18] | 0.334 | 1.520
DIAG (ours) | 0.096 | 0.411

The results, reported in Table III, highlight how DIAG can generate images
that are very similar to the ones originally present in the dataset, resulting
in the lowest FID out of all the other methodologies.

Another interesting observation is how both the generative-model-based
procedures (DDPM and DIAG) result in images that are more in-distribution
(lower FID) than the “per-region” augmentation techniques such as MemSeg,
which records the highest FID out of all the tested methodologies.

##  V Conclusions

This work presents DIAG, a novel data augmentation pipeline that leverages
language-conditioned Latent Diffusion Models to produce training-free positive
images for enhancing the performance of a surface defect detection model. We
introduced domain experts in the generation pipeline, asking them to describe
with textual prompts how a defect should look and where it can be localized.
Then, we adopt a pre-trained LDM to generate defective images and train a
binary classifier for isolating the anomalous images. We focus our experiments
on the KSDD2 dataset and establish ourselves as the new state-of-the-art data
augmentation pipeline, surpassing previous approaches in both the zero-shot
and full-shot data augmentation scenarios with an AP of .801 and .924,
respectively. These results highlight the potential of in-distribution data
augmentation in the anomaly detection field, where training-free generative
model pipelines such as DIAG can provide meaningful data for downstream
classification, making them appealing solutions in scenarios where real
anomalous data is difficult to collect or unavailable. These promising results
promote further exploration across various datasets, particularly
investigating how robust the image generation is compared to noisy textual
prompts.

## References

  * [1] T. Wang, Y. Chen, M. Qiao, and H. Snoussi, “A fast and robust convolutional neural network-based defect detection model in product quality control,” _The International Journal of Advanced Manufacturing Technology_ , vol. 94, pp. 3465–3471, 2018.
  * [2] C. S. Tsang, H. Y. Ngan, and G. K. Pang, “Fabric inspection based on the elo rating method,” _Pattern Recognition_ , vol. 51, pp. 378–394, 2016.
  * [3] S. H. Hanzaei, A. Afshar, and F. Barazandeh, “Automatic detection and classification of the ceramic tiles’ surface defects,” _Pattern Recognition_ , vol. 66, pp. 174–189, 2017.
  * [4] L. Capogrosso, F. Cunico, D. S. Cheng, F. Fummi, and M. Cristani, “A machine learning-oriented survey on tiny machine learning,” _IEEE Access_ , 2024.
  * [5] H. Deng and X. Li, “Anomaly detection via reverse distillation from one-class embedding,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022.
  * [6] K. Roth, L. Pemula, J. Zepeda, B. Schölkopf, T. Brox, and P. Gehler, “Towards total recall in industrial anomaly detection,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022, pp. 14 318–14 328.
  * [7] M. Rudolph, B. Wandt, and B. Rosenhahn, “Same same but differnet: Semi-supervised defect detection with normalizing flows,” in _Winter Conference on Applications of Computer Vision (WACV)_ , 2021.
  * [8] Y. Song, T. Wang, P. Cai, S. K. Mondal, and J. P. Sahoo, “A comprehensive survey of few-shot learning: Evolution, applications, challenges, and opportunities,” _ACM Computing Surveys_ , vol. 55, no. 13s, pp. 1–40, 2023.
  * [9] Y. Chen, Y. Ding, F. Zhao, E. Zhang, Z. Wu, and L. Shao, “Surface defect detection methods for industrial products: A review,” _Applied Sciences_ , vol. 11, no. 16, p. 7657, 2021.
  * [10] J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” _Advances in Neural Information Processing Systems (NeurIPS)_ , vol. 33, pp. 6840–6851, 2020.
  * [11] X. Tao, D. Zhang, W. Ma, Z. Hou, Z. Lu, and C. Adak, “Unsupervised anomaly detection for surface defects with dual-siamese network,” _IEEE Transactions on Industrial Informatics_ , vol. 18, no. 11, pp. 7707–7717, 2022.
  * [12] C. Luan, R. Cui, L. Sun, and Z. Lin, “A siamese network utilizing image structural differences for cross-category defect detection,” in _2020 IEEE International Conference on Image Processing (ICIP)_. IEEE, 2020.
  * [13] J. Božič, D. Tabernik, and D. Skočaj, “Mixed supervision for surface-defect detection: From weakly to fully supervised learning,” _Computers in Industry_ , vol. 129, p. 103459, 2021.
  * [14] T. Defard, A. Setkov, A. Loesch, and R. Audigier, “Padim: a patch distribution modeling framework for anomaly detection and localization,” in _International Conference on Pattern Recognition (ICPR)_ , 2021.
  * [15] M. Yang, P. Wu, and H. Feng, “Memseg: A semi-supervised method for image surface defect detection using differences and commonalities,” _Engineering Applications of Artificial Intelligence_ , vol. 119, p. 105835, 2023.
  * [16] V. Zavrtanik, M. Kristan, and D. Skočaj, “Draem-a discriminatively trained reconstruction embedding for surface anomaly detection,” in _IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2021.
  * [17] H. Zhang, Z. Wu, Z. Wang, Z. Chen, and Y.-G. Jiang, “Prototypical residual networks for anomaly detection and localization,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2023.
  * [18] L. Capogrosso, F. Girella, F. Taioli, M. D. Chiara, M. Aqeel, F. Fummi, F. Setti, and M. Cristani, “Diffusion-based image generation for in-distribution data augmentation in surface defect detection,” _arXiv preprint arXiv:2406.00501_ , 2024.
  * [19] J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli, “Deep unsupervised learning using nonequilibrium thermodynamics,” in _International Conference on Machine Learning (ICML)_ , 2015.
  * [20] Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole, “Score-based generative modeling through stochastic differential equations,” in _International Conference on Learning Representations (ICLR)_ , 2020.
  * [21] J. Ho and T. Salimans, “Classifier-free diffusion guidance,” _arXiv preprint arXiv:2207.12598_ , 2022.
  * [22] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022.
  * [23] L. Zhang, A. Rao, and M. Agrawala, “Adding conditional control to text-to-image diffusion models,” in _IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2023, pp. 3836–3847.
  * [24] L. Capogrosso, A. Mascolini, F. Girella, G. Skenderi, S. Gaiardelli, N. Dall’Ora, F. Ponzio, E. Fraccaroli, S. Di Cataldo, S. Vinco _et al._ , “Neuro-symbolic empowered denoising diffusion probabilistic models for real-time anomaly detection in industry 4.0: Wild-and-crazy-idea paper,” in _2023 Forum on Specification & Design Languages (FDL)_. IEEE, 2023, pp. 1–4.
  * [25] C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. L. Denton, K. Ghasemipour, R. Gontijo Lopes, B. Karagol Ayan, T. Salimans _et al._ , “Photorealistic text-to-image diffusion models with deep language understanding,” _Advances in Neural Information Processing Systems (NeurIPS)_ , vol. 35, pp. 36 479–36 494, 2022.
  * [26] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen, “Hierarchical text-conditional image generation with clip latents. arxiv 2022,” _arXiv preprint arXiv:2204.06125_ , 2022.
  * [27] D. Podell, Z. English, K. Lacey, A. Blattmann, T. Dockhorn, J. Müller, J. Penna, and R. Rombach, “Sdxl: Improving latent diffusion models for high-resolution image synthesis,” _arXiv preprint arXiv:2307.01952_ , 2023.
  * [28] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016.
  * [29] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, “GANs trained by a two time-scale update rule converge to a local nash equilibrium,” _Advances in Neural Information Processing Systems (NeurIPS)_ , vol. 30, 2017.
  * [30] P. von Platen, S. Patil, A. Lozhkov, P. Cuenca, N. Lambert, K. Rasul, M. Davaadorj, and T. Wolf, “Diffusers: State-of-the-art diffusion models,” https://github.com/huggingface/diffusers, 2022.
  * [31] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.


# Characterizing the Computational and Memory Requirements of Virtual RANs

Somreeta Pramanik  Politecnico di Torino
Torino, Italy
<EMAIL_ADDRESS>Adlen Ksentini  EURECOM
Sophia Antipolis, France
<EMAIL_ADDRESS>Carla Fabiana Chiasserini  Politecnico di Torino
Torino, Italy
<EMAIL_ADDRESS>
###### Abstract

The virtualization of radio access networks (RANs) is emerging as a key
component of future wireless systems, as it brings agility to the RAN
architecture and offers degrees of design freedom. In this paper, we
investigate and characterize the computational and memory requirements of
virtual RANs. To this end, we build a virtual RAN test-bed leveraging the
srsRAN open-source mobile communication platform and general-purpose
processor-based servers. Through extensive experiments, we profile the
consumption of computing and memory resources, and we assess the system
performance. Further, we build regression models to predict the system
behavior as the number of connected users increases, under diverse radio
transmission settings. In so doing, we develop a methodology and prediction
models that can help designing and optimizing virtual RANs.

###### Index Terms:

Virtual radio access networks, 5G, experimental test-bed, regression models.

##  I Introduction

Virtualization of the Radio Access Network (RAN) is well-recognized as a key
technology to accommodate the ever-increasing mobile data traffic of emerging
applications with stringent requirements [1]. By exploiting software defined
networking and network function virtualization, the Cloud RAN (C-RAN) [2]
concept has evolved towards the Virtual RAN (vRAN) paradigm [3], where a
virtual eNB centralizes a softwarized radio point of access stack into a
computing edge infrastructure. The majority of the baseband processing of a
virtual radio point of access (vRPA) is executed in virtual network functions
running in computing platforms, typically located at edge servers close to the
antennas. In so doing, a vRPA is able to adapt to time varying/non-uniform
data traffic demands, as new vRPAs can be added and upgraded easily, thereby
improving scalability and easing network maintenance while reducing site lease
costs, energy usage, and maintenance expenses. Moreover, due to the virtual
network functions being implemented in edge servers, a vRPA can reduce
latency, ensure highly efficient network operation and service delivery, and
offer an improved user experience.

Importantly, vRAN has also become a cornerstone technology for the realization
of the emerging Open Radio Access Network (O-RAN) paradigm [4]. Indeed, the
level of virtualization and flexibility that characterize a vRAN make it a
perfect fit for the openness and intelligence concepts that are at the basis
of the O-RAN architecture [5]. It is therefore expected that open standard
radio frequency (RF) interfaces, combined with vRAN technologies, will further
increase operational savings and increase the scalability of radio access
networks.

However, the increased network flexibility and programmability allowed by
vRANs come at the cost of a higher consumption of computing and memory
resources at the network edge [6]. In particular, computing resources are
typically pooled inefficiently since most of the implementations over
dimension computing capacity to cope with peak demands in real-time workloads
[7, 8]. It follows that the gains currently attainable by a vRAN are far from
optimal, preventing its deployment at scale.

To solve the above issues, it is critical to dynamically adapt the resources
allocation to the temporal variations of the demand across vRPAs [9]. Towards
this goal, a first, fundamental step is to gain a better hands-on
understanding of the behavior of vRPAs and the relation between radio and
computing/memory resource dynamics, as well as their dependency on such
factors as radio channel conditions and user’s traffic demand. While the
studies in [9, 10, 11, 12, 13, 7, 14, 15, 16, 6, 17] have focused on the
optimization of vRANs through experimental work or by designing analytical
models and algorithmic solutions, to our knowledge, this paper is the first
attempt to characterize the computing and memory resource consumption of vRANs
under diverse settings, as the number of connected users increases.

In particular, we address the following research questions:

1\. What are the computing and memory requirements of a vRAN, as different
settings in terms of number of occupied resource blocks and type of modulation
and coding scheme are adopted?

2\. How do the computing and memory requirements of a vRAN change as the
number of connected users varies?

We answer these questions by investigating the behavior of a vRPA using a
test-bed implementation and conducting an extensive measurement campaign. In
particular, we leverage an srsRAN implementation of an eNB and investigate its
CPU and memory consumption under different experimental settings.

Our main contributions are as follows.

  * •

We develop an srsRAN-based experimental test-bed and perform extensive
experiments, in order to profile the performance limits of the eNB in terms of
processing, memory, and throughput. We show that the CPU utilization of the
eNB increases with the modulation and coding (MCS) index, number of occupied
Resource Blocks (RBs), and importantly, with the number of connected users.

  * •

Using empirical data, we define regression models to predict the percentage of
CPU utilization and memory consumption of the virtual eNB, as the number of
connected users varies. In so doing, we obtain a prediction accuracy of 99%
for both CPU and memory utilization. These approximated models provide real-
world insights and key inputs to formulate, design, and evaluate optimized
resource management in vRANs.

The rest of the paper is organized as follows. Section II discusses the
related work and highlights the novelty of our study. Section III introduces
the design and implementation of our vRAN test-bed, while Section IV presents
experimental results and our empirical models. Finally, Section V concludes
the paper and presents possible directions for future research.

##  II Related Work

Owing to the intricate relationship between radio and computing resource
dynamics, and the advantages offered by vRANs, several works have aimed at
investigating and optimizing such a virtual system. While the studies in [9,
10, 11, 12, 13] focus on evaluating the performance of a vRAN through
experiments, the works in [7, 14, 15, 16, 6, 17] provide an insight into the
theoretical framework.

Experimental work. The work in [9] is one of the first experimental studies
that characterize the potential savings in compute resources when exploiting
the variations in the processing load across base stations. Interestingly,
[10] presents a linear model to calculate the uplink processing time for a
single user in terms of the sub-carrier load, MCS index, and number of
antennas. The linear model is then used to develop RT-OPEX, a C-RAN scheduling
algorithm. The impact of MCS and SNR (radio parameters) on real-time C-RAN
processing (i.e., CPU) is studied in [11], along with a mathematical model for
predicting the decoding time. The work in [12] profiles instead the
performance of a C-RAN in terms of CPU and memory usage, as the iperf
transmission bandwidth increases. In [13], the authors investigated the CPU
consumption of the baseband unit (BBU) under various conditions for the
C-RANs, and characterized the computational demand in terms of throughput.
However, no such work has investigated and characterized the performance of a
vRAN in terms of CPU and memory usage as the number of connected users
increases and under diverse settings.

Theoretical work. For the sake of completeness, we also mention that quite a
large literature exists on analytical models and algorithmic solutions for the
optimization of a vRAN. In particular, the works of [7, 18] set a theoretical
basis for CPU-aware radio resource control. In [7], the authors aim at
reducing the level of variability of the computational load, which leads to
eventual computational outages (where frames are not decoded in time), by
jointly optimizing the selection of the MCS index and the allocation of the
physical resource blocks. The authors in [14, 19] investigate the trade-off
between the consumption of data processing resources and achievable data
rates, taking into account specifically the processing requirements of forward
error correction (FEC) on the uplink. The study in [14] proposes a model for
computational outage and shows how it relates to channel outage.

The above discussed works rely on the same model relating computational
requirements and SNR, and they neglect variations on the arrival bit-rate
load. This issue is addressed instead in [20], which combines real-time
traffic classification and CPU scheduling in a mobile edge computing setup.
However, [20] also relies on a simplistic base-band processing model and does
not include an experimental validation. The works in [14, 21] propose to
enable the coordination between radio and computing resources schedulers by
introducing a computationally aware MCS selection policy that reduces the
computational complexity requirements, at the cost of slightly decreased
spectral efficiency. An analytical framework, FuidRAN, for the vRAN design is
introduced in [15], which jointly selects the function split and routing
policy, tailored to the available network and computing resources. However,
the model is provided for one user only. In [16], a novel reinforcement
learning framework is presented, which efficiently allocates radio resources
to multiple users in terms of link, MCS index, RBs and airtime for packet
transmissions in heterogeneous vRANs. A related relevant contribution is also
given in [6], which designs a solution, named as vrAIn, that dynamically
learns the optimal allocation of computing and radio resources in order to
meet the target level of quality of service. A novel pipeline architecture for
5G distributed units (DUs) is then presented in [17] to guarantee a minimum
set of signals that preserve synchronization between the DU and its users,
during computing capacity shortages. This study relies on techniques that
require predictable computing to provide carrier-grade reliability.

In conclusion, there is no such work which characterizes the computing
requirements of vRANs with respect to contextual dynamics (for instance,
traffic load and number of users). Importantly, designing resource allocation
schemes without knowing such requirements may severely hamper the system
performance in terms of throughput.

(a)

(b)

Figure 1: vRAN test-bed implementation using srsRAN: (a) snapshot of our test-
bed highlighting the edge node hosting the virtual eNB and EPC, and four UEs;
(b) test-bed architecture including UEs, virtual eNB and virtual EPC.

##  III vRAN Test-bed

We now introduce our vRAN test-bed using srsRAN, detailing the test-bed
architecture and configuration, and the adopted experimental methods.

###  III-A Test-bed architecture

Figure 1(a) provides a snapshot of the test-bed we developed, while Figure
1(b) represents its architecture. We leverage software defined radio (SDR)
interfaces enabling point-to-point communications between vRPA and User
Equipment (UE). An vRPA implements the necessary processing stack to transfer
data to/from UEs. In our case, the vRPA acts as a virtual eNB implemented at
the edge of the network. The connectivity between the vRPA and UEs is
supported by means of an LTE radio link implemented using the srsRAN [22] – an
open-source SDR LTE stack implementation offering Evolved Packet Core (EPC),
eNB, and UE applications. It is compliant with LTE Release 9 and supports up
to 20-MHz bandwidth channels as well as transmission modes from 1 to 4, all
using the frequency division duplexing (FDD) configuration.

As RF front-end, Ettus Universal Software Radio Peripheral (USRP) B210 devices
are used to perform up/down-conversion, filtering, amplification and AD/DA
conversion of the UE and eNB LTE signals. All the RF front-ends are connected
to the vRPA and the UEs via USB. Then, physical layer is implemented through a
set of OFDMA-modulated channels, using RB filling across ten 1-ms subframes
forming a frame. RBs assigned to UEs by the MAC layer are modulated and
encoded with an MCS that depends on the user’s Channel Quality Indicator
(CQI), a measure of SNR that is locally available at the UEs for uplink
transmission and is reported periodically by UEs. The modulation order can
vary up to 256-QAM, while FEC is employed using LDPC or Turbo codes with
coding rate of 1/2, 2/3, or 3/4 [23]. At the MAC layer, an automatic repeat
request error control is in place, i.e., an unsuccessfully transmitted packet
can be resent till a maximum number of allowed re-transmission attempts.

The edge host and the mobile terminals are each installed in Ubuntu 18.04
systems. The edge host is equipped with an Intel i7-7700HQ 4-cores CPU and 8
GB of DDR4 RAM, while the UEs feature an Intel i7-8550U 4-cores CPU and 16 GB
of DDR4 RAM. Each Ubuntu system is connected to USRP B210 boards using USRP
Hardware Driver v3.15. In order to faciltate the experiments, all performance
management features in the BIOS (e.g., Intel@TurboBoost, Hyper-thread control,
Intel SpeedStep) are enabled and C-states have been turned off. Moreover, the
real-time thread priorities are enabled in the srsRAN as the applications
(srsENB and srsUE) are executed with root privileges. A set of threads are
created in srsRAN for performance and priority management reasons. The
threading architecture of the physical layer implementation is motivated by
the stringent latency requirements. Also, we monitor the level of CPU
consumption and ensure that, during our experiments, the allocated CPU is
sufficient to keep up with the required data rate so as to avoid severe system
failures during the radio data transfer. Finally, in order to establish a
stable connection, we set the transmit gain (tx_gain) at the eNB to its
maximum value.

###  III-B Monitoring the srsRAN eNB and UEs

To monitor the behavior and track the performance of the vRAN entities, we
leverage some of the useful features of srsRAN (e.g., detailed log system with
per-layer log levels, MAC layer Wireshark packet capture, command-line trace
metrics, detailed input configuration file). A configuration file is provided
to set parameters such as downlink carrier frequency and log or packet capture
options, making the software easy to use. Moreover, we have relied on a useful
feature of srsUE which provides real-time traces.

The srsENB application metrics are generated once per second by default.
Metrics are provided on a per-UE basis for the downlink and uplink,
respectively. The eNB is configured in band 7 (FDD) and the transmission
bandwidth has been set to $10$ MHz, corresponding to $50$ RBs. In order to
determine the successful connection between eNB and UE, the RRC states are
observed. Specifically, when the UEs are successfully paired to the eNB, the
RRC connection setup message is seen. We have also saved the UE and eNB logs
to verify such entities’ status.

As experimental set-up, we connected 30 dB attenuators to the antennas of each
network node; furthermore, the UEs were placed close enough to the eNB so as
to ensure high values of SINR ($\geq 25$ dB). We focus on downlink data
transfer and used iperf for data packet generation.

##  IV Experimental Results and Empirical Models

We present here the performance of the vRPA, i.e., the srsRAN eNB, in terms of
CPU and memory utilization as the number of occupied RBs and the MCS index
vary, when a single UE or multiple UEs are connected. The results have been
obtained by averaging over 10 experiments; in every plot, both the average
value of the presented performance metric and the corresponding 95% confidence
interval are shown.

###  IV-A CPU utilization

The CPU usage is analysed using an Intel Core i7-7700HQ 2.80 GHz CPU. The CPU
utilization is calculated using the top process in Linux, which is widely used
and provides a dynamic real-time view of a running system managed by the
kernel. Specifically, the percentage of consumed CPU is collected by sampling
the top reports every second, over a period of 200 seconds.

Figure 2: CPU utilization and UDP downlink throughput of the virtual eNB, for
different values of the MCS index and a single connected UE.

Figure 2 shows the average CPU utilization (left-hand side y-axis) and the
throughput (right-hand side y-axis) obtained over 10 iterations of the virtual
eNB, as the MCS varies and for a single connected user. For this experiment,
UDP traffic is generated at the eNB at $30$ Mbps and sent to the UE, setting
the number of allocated RBs to $50$. Also, we set the transmission gain to its
maximum value, thus ensuring that the SNR does not drop below $32$ dB. From
the plot, we can observe that, as also shown in [6], the CPU utilization of
the virtual eNB increases as the MCS index grows from $0$ to $27$. Further,
the consumption of computing resources, which is mainly due to the modulation,
demodulation, coding and decoding operations, is quite significant in absolute
terms: as an example, for MCS$=27$, a single user consumes around $51$% of a
single CPU of the edge node. Using empirical data, we found that the CPU
utilization of the virtual eNB can be well approximated as a linear increasing
function of the MCS, i.e., $\mbox{CPU[\%]}=0.429\times\mbox{MCS}+39.58$.

Figure 3: CPU utilization and UDP downlink throughput of the virtual eNB vs
the number of occupied RBs, for a single connected UE and MCS=$27$.

Figure 3 shows the CPU utilization (left-hand side y-axis) of the virtual eNB
as the number of allocated RBs varies from $12$ to $50$, for a single user and
MCS$=27$. The downlink traffic load is set to $9$ (for $12$ RBs), $15$ (for
$24$ RBs), $21$ (for $36$ RBs) and $30$ (for $50$ RBs) Mbps, respectively. We
notice that the CPU utilization increases as the number of occupied RBs
increases, with a maximum of $51$% for a single UE. A higher number of
occupied RBs leads to the user transmitting at a higher rate, which results in
a higher computational resource consumption. From the experimental data, we
found that the CPU utilization of the eNB can be well approximated as a linear
increasing function of the number of occupied RBs, i.e.,
$\mbox{CPU[\%]}=0.2892\times\mbox{No\\_RB}+37.02$. We remark that the provided
approximation functions can help in interpolating the average CPU utilization
with different radio configurations.

Figure 4: CPU utilization of the virtual eNB vs the MCS index, for a varying
number of connected UEs and a total number of occupied RBs equal to $36$.

We are now interested in how the computing resource consumption varies as the
number of users connected to the eNB changes. It is indeed a fact that the
number of served UEs is rapidly increasing, and that cellular networks will
have to support a massive number of users. Figure 4 presents the CPU
utilization (left-hand side y-axis) of the virtual eNB as the MCS index
varies, for different numbers of users. For this experiment, the overall
maximum number of RBs that can be used is set to $36$, downlink traffic is
generated at $21$ Mbps, and the tx_gain is set to its maximum value, so that
the SNR is always above $28$ dB for all the UEs. In this scenario, an
interesting behavior emerges: for a fixed value of the MCS index, the average
CPU consumption of the eNB increases significantly as the number of users
increases, although the traffic load is kept constant. As an example, for
MCS$=27$, the average CPU consumption with four UEs is $62$% of a single CPU,
i.e., about 30% more than with one UE.

Figure 5: CPU utilization of the virtual eNB vs number of occupied RBs, for a
varying number of connected UEs and MCS$=27$.

Figure 6: Regression plot as the number of UEs varies, for $36$ occupied RBs
and MCS$=27$.

(a)

(b)

Figure 7: (a) Residual plot for the regression model in Eq. (1), with 36
occupied RBs and MCS$=27$; (b) Residual plot for the regression model in Eq.
(2), for different values of the MCS index and $36$ occupied RBs.

In addition, Figure 5 shows the CPU consumption of the virtual eNB as the
number of allocated RBs varies from $24$ to $36$, for a different number of
connected UEs, MCS$=27$, and maximum tx_gain. The downlink traffic load for
$24$ RBs is set to $15$ Mbps, while it is $21$ Mbps for $36$ RBs, so that all
the allocated RBs are always occupied. Interestingly, as the number of
connected users grows, the CPU consumption of the virtual eNB increases
linearly.

Since the MCS index and the number of occupied RBs are always finite values
that vary over a very specific range, we are mainly interested in
understanding the CPU requirements of the virtual eNB as the number of users
increases. Then, using the empirical data, we build a regression model that
predicts the CPU utilization of the eNB as the number, $n$, of connected UEs
varies. For 36 occupied RBs and MCS$=27$, we obtain:

$CPU[\%]=4.099\times n+44.935\,.$ (1)

We remark that similar models can be built for different values of MCS index
and number of occupied RBs.

Figure 6 shows the CPU utilization under the above settings, along with the
curve obtained using the regression model in (1). The Significance F for our
model is 0.004, which, being well below 0.05, shows that the model can predict
correctly the behavior under study. Further, the regression output (R-squared)
indicates that 99% of the variation in CPU consumption is due to the number of
UEs. Specifically, every additional UE is expected to entail about 4.1% of
increase in CPU usage at the eNB; it follows that 15 users will easily consume
up to 100% of a CPU.

Again for 36 occupied RBs and MCS$=27$, Figure 7(a) plots the residuals, i.e.,
the difference in percentage between the actual value of CPU utilization and
the one predicted by the regression model, obtained when the number of UEs
varies between 0 and 4. We observe that such residuals are always within -0.5%
to 1% of CPU usage, which confirms the very good accuracy of the model.

Next, it is important to show that a linear regression model (as in (1)),
obtained from experimental data, correctly characterizes the computing
requirements of a vRAN as the number of users increases. To this end, we use
all the experimental data obtained for a number of UEs up to three (i.e., for
varying number of occupied RBs and MCS indices), and we predict the CPU
consumption when four UEs are connected. The corresponding regression model is
given by:

$CPU[\%]=3.9\times n+0.369\times m+35.658$ (2)

where $n$ is the number of connected UEs and $m$ is the adopted MCS index. A
similar model can be obtained considering as explanatory variables the number
of connected UEs and the number of occupied RBs.

Tab. I and Tab. II report the actual CPU utilization when four UEs are
connected, and the corresponding value predicted through (2). Furthermore,
Figure 7(b) plots the residuals obtained for the model in (2): again, all
residual values are within -1% and 1%. These results, along with an
F-statistic value of 0.0023, indicates that the linear regression model well
describes the behavior of CPU utilization. Similar results have been obtained
for different values of the number of occupied RBs.

TABLE I: Actual vs predicted CPU usage with 4 UEs, a varying number of occupied RBs, and MCS$=27$ No. occupied RBs | Actual CPU [%] | Predicted CPU [%]
---|---|---
24 | 56.36 | 55.51
36 | 61 | 60.759
TABLE II: Actual vs predicted CPU usage with 4 UEs, different MCS indices, and 36 occupied RBs MCS index | Actual CPU [%] | Predicted CPU [%]
---|---|---
12 | 54.31 | 55.55
18 | 58.76 | 57.71
27 | 61 | 61.19

Figure 8: Memory utilization of the virtual eNB as the number of occupied RBs
varies, for MCS$=27$ and one connected UE.

Figure 9: Memory utilization of the virtual eNB vs number of UEs, for MCS$=27$
and $50$ occupied RBs.

###  IV-B Memory utilization

Beside CPU, also memory is a precious resource at the network edge. It is
therefore important to investigate the memory usage of the virtual eNB for
different radio configurations.

The memory usage is monitored using the top process in Linux, which reports
both the amount of memory used by the process in GB as well as the percentage
value. As done for the CPU utilization, we profile the memory usage for
different values of MCS index, number of occupied RBs, and number of connected
UEs. In our experiments, the downlink traffic load is set to $30$ Mbps in
order to ensure that all allocated RBs are actually occupied.

Figure 8 shows the memory usage (left-hand side y-axis) as the number of
occupied RBs varies from $24$ to $50$, for MCS$=27$ and only one connected UE.
As observed for the CPU utilization, the memory usage also increases linearly
with the number of occupied RBs. We also find a similar behavior as the number
of UEs grows, as shown in Figure 9 for MCS$=27$ and 50 occupied RBs. In
particular, the percentage of memory utilization (left-hand side y-axis)
increases almost linearly with the number of users, in spite of the constant
traffic load. Furthermore, we observe that $3.24$ GB of memory are consumed
with four connected UEs, which is over 20% more than when only one UE is
connected.

Based on the experimental data we obtained, the regression model predicting
the percentage of memory utilization, for 50 occupied RBs and MCS$=27$, is as
follows:

$\mbox{\em Memory}[\%]=0.42\times n+4.55,$ (3)

where $n$ is the number of UEs. A similar linear regression model can be built
for different values of number of occupied RBs and MCS index.

The accuracy of the model is shown in Figures 10(a)–10(b), for MCS$=27$ and 50
occupied RBs. The two plots depict, respectively, the difference between the
actual and predicted percentage of memory utilization, and the residuals, as
the number of UEs varies. Both figures highlight that also the regression
model we built for the percentage of memory utilization closely mimics the
resource consumption of a vRAN.

(a)

(b)

Figure 10: Percentage of memory usage: (a) linear regression plot, (b)
residual plot for MCS$=27$ and $50$ occupied RBs.

##  V Conclusions

Virtualized radio access networks (vRANs) are the basis of future base
stations design. To provide real-world insights and key inputs to formulate,
design, and evaluate optimized resource-management problems in vRANs, we
investigated and characterized the computational requirements of vRANs by
developing an srsRAN-based test-bed. Through extensive experiments, we
profiled the CPU and memory utilization of the vRAN. Our results shed light on
the vRAN behavior across different scenarios, showing that, remarkably, the
CPU and the memory utilization of the eNB increases substantially with the
number of users. It is worth underlining that the result has been obtained
under a constant value of traffic load and number of occupied resource blocks.
Based on these empirical results, we also built linear regression models for
the prediction of CPU and memory utilization as the number of users varies. To
the best of our knowledge, this is the first work that thoroughly studies the
computational and memory requirements of a vRAN.

We believe that our study and the obtained models can provide researchers and
practitioners with real-world insights and the necessary tools for designing
advanced efficient resource-provisioning and allocation strategies in vRAN
systems. As future work, we intend to exploit the results of this work to
develop algorithms for the efficient and effective scaling of the vRAN
functions, and the optimal settings of the vRAN parameters in the case of
shortage of CPU and memory resources at the network edge.

## References

  * [1] W. River, “vran: The next step in network transformation,” White Paper, 2017\.
  * [2] A. Checko, H. L. Christiansen, Y. Yan, L. Scolari, G. Kardaras, M. S. Berger, and L. Dittmann, “Cloud ran for mobile networks—a technology overview,” IEEE Communications Surveys Tutorials, vol. 17, no. 1, pp. 405–426, 2015\.
  * [3] L. Gavrilovska, V. Rakovic, and D. Denkovski, “From cloud ran to open ran.,” Wirel. Pers. Commun., vol. 113, no. 3, pp. 1523–1539, 2020.
  * [4] O. Alliance, “O-ran: Towards an open and smart ran, white paper,” 2018.
  * [5] S. Niknam, A. Roy, H. S. Dhillon, S. Singh, R. Banerji, J. H. Reed, N. Saxena, and S. Yoon, “Intelligent o-ran for beyond 5g and 6g wireless networks,” arXiv preprint arXiv:2005.08374, 2020.
  * [6] J. A. Ayala-Romero, A. Garcia-Saavedra, M. Gramaglia, X. Costa-Perez, A. Banchs, and J. J. Alcaraz, “vrain: Deep learning based orchestration for computing and radio resources in vrans,” IEEE Transactions on Mobile Computing, 2020.
  * [7] D. Bega, A. Banchs, M. Gramaglia, X. Costa-Pérez, and P. Rost, “Cares: Computation-aware scheduling in virtualized radio access networks,” IEEE Transactions on Wireless Communications, vol. 17, no. 12, pp. 7993–8006, 2018.
  * [8] P. Rost, I. Berberana, A. Maeder, H. Paul, V. Suryaprakash, M. Valenti, D. Wübben, A. Dekorsy, and G. Fettweis, “Benefits and challenges of virtualization in 5g radio access networks,” IEEE Communications Magazine, vol. 53, no. 12, pp. 75–82, 2015.
  * [9] S. Bhaumik, S. P. Chandrabose, M. K. Jataprolu, G. Kumar, A. Muralidhar, P. Polakos, V. Srinivasan, and T. Woo, “Cloudiq: A framework for processing base stations in a data center,” Mobicom ’12, p. 125–136.
  * [10] K. C. Garikipati, K. Fawaz, and K. G. Shin, “Rt-opex: Flexible scheduling for cloud-ran processing,” CoNEXT ’16, p. 267–280, Association for Computing Machinery.
  * [11] H. Khedher, S. Hoteit, P. Brown, R. Krishnaswamy, W. Diego, and V. Vèque, “Processing time evaluation and prediction in cloud-ran,” in IEEE International Conference on Communications (ICC), pp. 1–6, 2019.
  * [12] P.-C. Lin and S.-L. Huang, “Performance profiling of cloud radio access networks using openairinterface,” in IEEE Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), pp. 454–458, 2018.
  * [13] T. X. Tran, A. Younis, and D. Pompili, “Understanding the computational requirements of virtualized baseband units using a programmable cloud radio access network testbed,” in IEEE International Conference on Autonomic Computing (ICAC), pp. 221–226, IEEE, 2017.
  * [14] M. C. Valenti, S. Talarico, and P. Rost, “The role of computational outage in dense cloud-based centralized radio access networks,” in 2014 IEEE Global Communications Conference, pp. 1466–1472.
  * [15] A. Garcia-Saavedra, X. Costa-Perez, D. J. Leith, and G. Iosifidis, “Fluidran: Optimized vran/mec orchestration,” in IEEE INFOCOM 2018-IEEE Conference on Computer Communications, pp. 2366–2374, 2018.
  * [16] S. Tripathi, C. Puligheddu, C. F. Chiasserini, and F. Mungari, “A context-aware radio resource management in heterogeneous virtual rans,” IEEE Transactions on Cognitive Communications and Networking, 2021.
  * [17] G. Garcia-Aviles, A. Garcia-Saavedra, M. Gramaglia, X. Costa-Perez, P. Serrano, and A. Banchs, “Nuberu: Reliable ran virtualization in shared platforms,” in Proceedings of the 27th Annual International Conference on Mobile Computing and Networking, MobiCom ’21, p. 749–761, 2021.
  * [18] P. Rost, A. Maeder, M. C. Valenti, and S. Talarico, “Computationally aware sum-rate optimal scheduling for centralized radio access networks,” in 2015 IEEE Global Communications Conference (GLOBECOM), pp. 1–6, 2015.
  * [19] P. Rost, S. Talarico, and M. C. Valenti, “The complexity–rate tradeoff of centralized radio access networks,” IEEE Transactions on Wireless Communications, vol. 14, no. 11, pp. 6164–6176, 2015.
  * [20] K. Wang, X. Yu, W. Lin, Z. Deng, and X. Liu, “Computing aware scheduling in mobile edge computing system,” Wireless Networks, vol. 27, no. 6, pp. 4229–4245, 2021.
  * [21] M. Sharara, S. Hoteit, P. Brown, and V. Vèque, “Coordination between radio and computing schedulers in cloud-ran,” in 2021 IFIP/IEEE International Symposium on Integrated Network Management (IM), pp. 37–44, IEEE, 2021.
  * [22] I. Gomez-Miguelez, A. Garcia-Saavedra, P. D. Sutton, P. Serrano, C. Cano, and D. J. Leith, “Srslte: An open-source platform for lte evolution and experimentation,” WiNTECH ’16, p. 25–32, Association for Computing Machinery, 2016.
  * [23] “3gpp ts 36.213 v12.4.0, release 12, evolved universal terrestrial radio access (e-utra) physical layer procedures (2014),” tech. rep.