# Improving Diffusion Models’s Data-Corruption Resistance using Scheduled
Pseudo-Huber Loss

Artem Khrapov  Vadim Popov  Tasnima Sadekova  Assel Yermekova  Mikhail Kudinov

###### Abstract

Diffusion models are known to be vulnerable to outliers in training data. In
this paper we study an alternative diffusion loss function, which can preserve
the high quality of generated data like the original squared $L_{2}$ loss
while at the same time being robust to outliers. We propose to use pseudo-
Huber loss function with a time-dependent parameter to allow for the trade-off
between robustness on the most vulnerable early reverse-diffusion steps and
fine details restoration on the final steps. We show that pseudo-Huber loss
with the time-dependent parameter exhibits better performance on corrupted
datasets in both image and audio domains. In addition, the loss function we
propose can potentially help diffusion models to resist dataset corruption
while not requiring data filtering or purification compared to conventional
training algorithms.

diffusion probabilistic modeling, text-to-image, text-to-speech, data
corruption, robustness, Huber loss

##  1 Introduction

Over the past few years denoising diffusion probabilistic models (Ho et al.,
2020; Song et al., 2021b) have been achieving remarkable results in various
generative tasks. It is no exaggeration to say that almost all common computer
vision generative problems such as conditional and unconditional image
generation (Dhariwal & Nichol, 2021), image super-resolution (Gao et al.,
2023), deblurring (Kawar et al., 2022), image editing (Meng et al., 2022) and
many others found high-quality solutions relying on diffusion models. Such
success inspired researches specializing in other fields to apply diffusion-
based approaches to the tasks they worked on. As a result, there appeared
diffusion models solving a vast range of tasks, e.g. text-to-speech synthesis
(Popov et al., 2021), music generation (Hawthorne et al., 2022), audio
upsampling (Han & Lee, 2022), video generation (Luo et al., 2023),
chirographic data generation (Das et al., 2023) and human motion generation
(Tevet et al., 2023) to name but a few. In several areas like text-to-image
(Rombach et al., 2022) and text-to-video (Luo et al., 2023) generation
diffusion models lead to breakthroughs and became de-facto a standard choice.

Various aspects related to diffusion models attracted interest of specialists
in generative modeling. Soon after diffusion models had been introduced,
numerous attempts were made to accelerate them — either by investigating
differential equation solvers (Dockhorn et al., 2022; Lu et al., 2022; Popov
et al., 2022) or by combining them with other kinds of generative models like
Generative Adversarial Networks (GANs) (Xiao et al., 2022) or Variational
Autoencoders (VAEs) (Kingma et al., 2021). Recently proposed generative
frameworks such as flow matching (Lipman et al., 2023) which essentially
consists in optimizing the vector field corresponding to the trajectories of
the probability flow Ordinary Differential Equation, or consistency models
(Song et al., 2023) performing best when applied as a method of diffusion
models distillation, brought performance of diffusion-related generative
models quite close to that of conventional models like GANs and VAEs in terms
of inference speed while preserving high quality of generated samples.

Another aspect important for any generative model is privacy preservation and
robustness to various types of attacks. On the one hand, diffusion models can
help other machine learning algorithms to better deal with adversarial attacks
via adversarial purification (Xiao et al., 2023; Nie et al., 2022). On the
other hand, they tend to memorize samples from the training dataset (Somepalli
et al., 2023) and, moreover, they are more prone to membership inference
attacks than GANs with similar generation quality (Carlini et al., 2023)
meaning that diffusion models trained in a standard way are potentially less
private than GANs. Furthermore, it has lately been shown that diffusion models
can be seriously harmed by the backdoor attacks, i.e. training or fine-tuning
data can be perturbed in such a way that the resulting model produces fine
samples for almost all inputs but in a few cases for specific input its output
is unexpected or arbitrarily bad (Wang et al., 2023; Struppek et al., 2023).
In particular, text-to-image diffusion models are known to be vulnerable to
the presence of look-alikes (i.e. images with the same text captions but
slightly different pixel content) in training datasets. This observation has
been exploited to create algorithms such as Glaze and Nightshade that spoil
samples produced by text-to-image models for specific prompts (Shan et al.,
2023a, b).

The mentioned attacks are natural tests of general robustness of diffusion
models. In this work we also study a similar test and concentrate on the case
when noise samples injected into training dataset are outliers, i.e. they
belong to distributions different from the one diffusion is meant to be
trained on. Although one can use any appropriate outlier detection algorithm
to preprocess training dataset, it can be time- and resource-consuming, so we
consider methods not requiring dataset filtering. Conventional diffusion model
training consists in optimizing squared $L_{2}$ loss between score matching
neural network output and true score function values. However, it is a well-
known fact that Mean Square Error does not provide robust estimators and there
exist alternatives that are less vulnerable to the presence of outliers, e.g.
Huber loss function (Huber, 1964). This function can be seen as a smooth
combination of $L_{2}$ loss for relatively small errors and $L_{1}$ loss for
relatively large errors, thus serving as a robust alternative of Mean Square
Error in various statistical tasks like location parameter estimation as in
the original paper by Huber (1964), or linear regression (Owen, 2007). A
variant of Huber loss called pseudo-Huber (which we will also refer to as
“P-Huber”) loss function has already been employed in the context of
diffusion-like models by Song and Dhariwal (2023) who distilled diffusion
models into consistency models with it. In this paper we explore the impact of
pseudo-Huber loss function with time-dependent parameters on diffusion model
robustness.

Our main contributions can be summarized as follows:

  * •

We propose a novel technique of delta-scheduling by introducing a time-
dependent delta parameter into pseudo-Huber loss;

  * •

We experimentally demonstrate the effectiveness of our approach across
multiple datasets and modalities;

  * •

We make detailed studies on various possible schedules, pseudo-Huber loss
parameters, corruption-percentages and the resilience factor computation.

##  2 Preliminaries

This section contains general facts about diffusion probabilistic modeling and
pseudo-Huber loss used in the consequent sections devoted to robust training
of diffusion models.

###  2.1 Diffusion Probabilistic Modeling

Suppose we have a $n$-dimensional stochastic process of a diffusion type
$X_{t}$ defined for $t\in[0,T]$ for a finite time horizon $T>0$ satisfying
Stochastic Differential Equation (SDE) with drift coefficient
$f(x,t):\mathbb{R}^{n}\times\mathbb{R}_{+}\to\mathbb{R}^{n}$ and positive
diffusion coefficient $g_{t}$. Note that throughout this paper all SDEs are
understood in Itô sense (Liptser & Shiryaev, 1978). Anderson (1982) showed
that under certain technical assumptions we can write down the reverse-time
dynamics of this process given by the reverse SDE:

$dX_{t}=\left(f(X_{t},t)-g_{t}^{2}\nabla\log{p_{t}(X_{t})}\right)dt+g_{t}dW_{t}\
,$ (1)

where $p_{t}$ is the marginal density of the original forward process at time
$t$ and this SDE is to be solved backwards in time starting at time $T$ from
random variable with density function $p_{T}$. Brownian motion $W_{t}$ driving
this process is supposed to be a backward Brownian motion (meaning that its
backward increments are independent, i.e. $W_{s}-W_{t}$ is independent of
$W_{t}$ for $0\leq s<t\leq T$).

In classic diffusion models the data distribution is represented by
$\operatorname{Law}{(X_{0})}$ and the drift and diffusion coefficients are
chosen so that the data is gradually perturbed with Gaussian noise up to time
$T$ where the noisy data $X_{T}$ is very close to standard normal distribution
called the prior of a diffusion model. If the score function
$\nabla\log{p_{t}(x)}$ can be well approximated by a neural network
$s_{\theta}(x,t)$, then sampling from the trained diffusion model can be
performed by solving the reverse SDE (1) backwards in time starting from a
random sample from the prior distribution. Alternatively, one can choose to
solve the probability flow Ordinary Differential Equation (ODE) (Song et al.,
2021b):

$dX_{t}=\left(f(X_{t},t)-\frac{g_{t}^{2}}{2}\nabla\log{p_{t}(X_{t})}\right)dt\
,$ (2)

which in many cases requires less steps to produce samples of the same
quality.

Song et al. (2021a) showed that under certain mild assumptions on data
distribution minimizing weighted squared $L_{2}$ loss between the score
function and neural network $s_{\theta}$ given by the expression

$\mathbb{E}_{X\sim\mu}\left[\int_{0}^{T}{g_{t}^{2}\|s_{\theta}(X_{t},t)-\nabla\log{p_{t}(X_{t})}\|_{2}^{2}}dt\right]\
,$ (3)

where $\mu$ is probability measure of paths $X=\\{X_{t}\\}_{t\in[0,T]}$, is
equivalent to minimizing the Kullback-Leibler (KL) divergence between measure
$\mu$ and the path measure corresponding to reverse diffusion parameterized
with the network $s_{\theta}$. It was shown that this divergence is in fact an
upper bound on negative log-likelihood of training data. Thus, $L_{2}$ loss is
well justified from the point of view of both maximum likelihood training and
optimizing KL divergence between path measures of reverse processes.

It is worth mentioning that although the score function cannot be computed
analytically for real-world data distributions, we can make use of denoising
score matching and show that for every $t$ the following two optimization
problems

$\min_{\theta}\leftarrow\mathbb{E}_{X_{t}}\|s_{\theta}(X_{t},t)-\nabla\log{p_{t}(X_{t})}\|_{2}^{2}\
,$ (4)
$\min_{\theta}\leftarrow\mathbb{E}_{X_{0}}\mathbb{E}_{X_{t}|X_{0}}\|s_{\theta}(X_{t},t)-\nabla\log{p_{t|0}(X_{t}|X_{0})}\|_{2}^{2}$
(5)

are equivalent. Note that $L_{2}$ loss is essential for the equivalence of 4
and 5. Unlike the unconditional score function $\nabla\log{p_{t}(x_{t})}$, the
conditional score function $\nabla\log{p_{t|0}(x_{t}|x_{0})}$ is tractable
since the conditional distribution $\operatorname{Law}{(X_{t}|X_{0})}$ has
Gaussian densities $p_{t|0}$. Thus, training a diffusion model consists in
optimizing the following squared $L_{2}$ loss

$\mathcal{L}_{t}(X_{0})=\mathbb{E}_{X_{t}|X_{0}}\|s_{\theta}(X_{t},t)-\nabla\log{p_{t|0}(X_{t}|X_{0})}\|_{2}^{2}\
,$ (6)

where $X_{0}$ is uniformly sampled from the training dataset and $t$ —
uniformly from $[0,T]$.

The framework described above stays the same for latent diffusion models
(Kingma et al., 2021), a faster alternative of classic diffusion models, with
the exception that the stochastic process $X_{t}$ is defined in latent space
and $X_{0}$ corresponds to some latent data representation.

###  2.2 Huber loss function

One-dimensional Huber loss (Huber, 1964) is defined as

$h_{\delta}(x)=\begin{cases}\frac{1}{2}x^{2}\qquad\qquad\ \text{for}\ \
|x|\leq\delta\\\ \delta(|x|-\frac{1}{2}\delta)\ \ \ \ \ \text{for}\ \
|x|>\delta\end{cases}$ (7)

for a positive $\delta$. Its multi-dimensional version is just a coordinate-
wise sum of losses $h_{\delta_{j}}(x^{j})$ of the corresponding components
$x^{j}$ of the input vector $x\in\mathbb{R}^{n}$. Parameters $\delta_{j}$ for
$j=1,..,n$ can be different. When computed on the difference between the true
value and its prediction by some statistical model, this function penalizes
the small errors like Mean Square Error (MSE) loss and the large errors like
Mean Absolute Error (MAE) loss. Thus, Huber loss penalizes the large errors
caused by outliers less than MSE loss which makes it attractive for robust
statistical methods (Owen, 2007).

It is easy to see that derivative of Huber loss is continuous, but not
differentiable in points $|x|=\delta$. Pseudo-Huber loss is a more smooth
function also behaving like MSE in one-dimensional case in the neighbourhood
of zero and like MAE in the neighbourhood of infinity:

$H_{\delta}(x)=\delta^{2}\left(\sqrt{1+\frac{x^{2}}{\delta^{2}}}-1\right)\ .$
(8)

This function is defined for positive values of the parameter $\delta$
controlling its tolerance to errors with a large $L_{2}$ norm. It is extended
to multi-dimensional input in the coordinate-wise manner as the standard Huber
loss $h_{\delta}(x)$.

In this paper we study the following training objective for diffusion models
expressed in terms of pseudo-Huber loss:

$\mathcal{L}^{(\delta)}_{t}(X_{0})=\mathbb{E}_{X_{t}|X_{0}}H_{\delta}\left(s_{\theta}^{(\delta)}(X_{t},t)-\nabla\log{p_{t|0}(X_{t}|X_{0})}\right),$
(9)

where $\delta$ can be time-dependent.

##  3 Experiments

###  3.1 Text-to-Image

Figure 1: Scheme of the process. Off-topic images are added to a clean dataset
of cat photos. When the L2 loss function is used, it leads to concept
distortion and even erasure (see the fractal-like structures in the second
row). Meanwhile the Huber loss training results stay consistent with their
not-corrupted counterparts.

For text2image customization experiments we used the Dreambooth framework
(Ruiz et al., 2023) on Stable Diffusion v1.5 implemented in Huggingface
Diffusers (von Platen et al., 2022) library. We used LoRA (Hu et al., 2021)
technique for faster training. We tested adaptation on $7$ different datasets
in various domains: characters, styles and landscapes.

We tested $4$ levels of corruption: $0\%(\text{clean run}),15\%,30\%,45\%$.
For each level of corruption a corresponding portion of images from the target
dataset (”clean”) was replaced with samples from the other datasets having the
lowest cosine similarity of CLIP embeddings (Radford et al., 2021) with random
”clean” samples. For each experiment 3 models have been trained: the model
with $L_{2}$ loss, the model with pseudo-Huber loss and the model with pseudo-
Huber loss delta scheduling for which the $\delta$ parameter depended on the
time step $t$. During the preliminary experiments we chose the exponential
decrease scheduler for $\delta$. Other schedules are possible, but not all of
them are optimal. Most importantly, the schedules with increasing $\delta$
have yielded the worst results, supporting the hypothesis that it’s much more
natural for the parameter to deplete over timesteps and ensure well image
approximation at the later steps of the reverse-diffusion process. Appendix
A.1 for the ablation studies on different other possible schedules.

We sampled images from the trained models and for each reference-sample pair –
consisting of a reference image from ”clean” or ”corrupting” datasets and a
sampled image – we computed CLIP similarity and “$1-\text{LPIPS}$” score
(Zhang et al., 2018), then calculated the mean across all pairs.

Now we had the average similarity scores to the clean data reference images:

  * •

when the dataset had been corrupted $S_{\text{to clean}}^{\text{corrupted}}$;

  * •

when it had been trained on the clean dataset $S_{\text{to
clean}}^{\text{clean}}$.

Through the selected similarity metric of these we introduce the key
$R$-variable representing how well the model fares against corruption:

$R=S_{\text{to clean}}^{\text{corrupted}}-S_{\text{to clean}}^{\text{clean}}$
(10)

We used similarity difference instead of, for example, division, as it’s more
stable in regards to zeroing of one of its components and the event of their
sign change. See Appendix A.5 for the numerical and visual comparison of the
results obtained for the different versions of the $R$-value and the detailed
reasoning behind using the differential metric.

CLIP-derived metrics have been shown to be inaccurate to measure image
features similarity, especially in the context of deflecting adversarial
attacks.(Shan et al., 2023c) Because of it, and that LPIPS has been giving
more consistent results than CLIP, we decided to use the “1 - LPIPS”
similarity metric for most of our work. We provide the CLIP and LPIPS
comparison plots and show their qualitatively close and different parts in
Appendix A.2.

Figure 2: The plot of the Resilience factor for “1 - LPIPS” similarity for all the tested text2image prompts at different levels of corruption at the selected $\delta=0.01$. | a landscape of sks | a photo of a shoan | a photo of an eichpoch | a picture by david revoy
---|---|---|---|---
l2 | -0.38 | -0.01 | 0.00 | -0.08
huber-0.1 | -0.28 | -0.20 | -0.02 | -0.09
huber scheduled-0.1 | -0.06 | -0.13 | -0.22 | -0.03
| lordbob111a picture in the style of lordbob | a picture of a drake | maxwell the cat | —
l2 | -0.42 | -0.39 | -0.14 | —
huber-0.1 | -0.33 | -0.40 | -0.13 | —
huber scheduled-0.1 | -0.11 | -0.14 | -0.04 | —
Table 1: R-scores for all the tested text2image prompts/concepts at 0.45 of
corruption, 0.1 Huber $\delta_{0}$ and 2000 steps. To take a look at examples
of each concept, see Appendix B. To get an overview of how these functions
change over the model’s training steps, see the plots in Appendix A.4.

The comparison of Scheduled Pseudo-Huber, Huber and $L_{2}$ losses in terms of
$R$-scores on all datasets at the final training step is listed at Table 1. It
is evident that Scheduled Pseudo-Huber loss outperforms $L_{2}$ in 5 out of 7
cases although the degree of its advantage varies. To see the training-wide
dynamics from step 0 to 2000 please refer to Appendix A.4 where it’s shown
that this advantage is largely consistent on all later training steps.

###  3.2 Text-to-Speech

For speech domain few-shot speaker adaptation scenario was chosen. We fine-
tuned a pre-trained multi-speaker Grad-TTS (Popov et al., 2021) model decoder
for a new voice. The experiment was conducted for $10$ female and $4$ male
speakers. Three datasets for each speaker were constructed: ”clean” dataset
with $16$ records of the target speaker and $2$ ”corrupted” datasets with
additional $4$ ”corrupting” records by a female and a male voice
correspondingly.

We tested performance of Grad-TTS models fined-tuned on corrupted datasets
with $L_{2}$ and pseudo-Huber loss. We also fine-tuned Grad-TTS with $L_{2}$
loss on clean data. As in the previous case, we used pseudo-Huber loss with
exponential decrease scheduler in our experiments.

As a target metric, we used speaker similarity of synthesized speech evaluated
with a pre-trained speaker verification
model222https://github.com/CorentinJ/Real-Time-Voice-Cloning.

Figure 3: Speaker similarity for different iterations averaged across
speakers. Models: clean \- trained on clean dataset; $l_{2}$ and huber
scheduled \- trained on mixed dataset with corresponding losses. Figure 4:
Number of synthesized samples with similarity less than corresponding
threshold. Total number of samples $1260$ from the best checkpoints on $350$
iterations.

We generated $90$ samples for every speaker and every model and calculated
mean and minimum similarity values. We calculated minimum similarity by
averaging similarity score between a generated sample and $5$ clean samples.
Then the minimum value across generated samples was chosen. In case of mean
similarity the mean value across generated samples was calculated.

According to Figure 3 models trained on corrupted datasets show quite high
mean similarity despite a slight drop compared to the model trained on clean
data. Although we observe a comparable drop in minimum similarity, models with
pseudo-Huber loss still constantly demonstrate better score on all iterations
in terms of minimum similarity. Furthermore, we analysed samples from the best
checkpoint ($350$ iterations) depending on different similarity thresholds.
Figure 4 illustrates that standard $L_{2}$ training scheme leads to samples
with lower similarity more often which means that training with Huber loss
with scheduling is more robust than standard $L_{2}$-based approach.

##  4 Limitations and Future Work

While protecting the models as a whole from some parts of the dataset being
poisoned, this method will most likely not work in case the entire dataset has
been corrupted through some sort of an adversarial attack – meaning the big
players will be fine, while the small models creators might still be
vulnerable to style-transfer failure. The conditions over the data structure,
which may be putting limits upon the data-corruption techniques, is still
unknown, however the knowledge of Scheduled Pseudo-Huber loss can spawn a new
generation of such algorithms. Our work doesn’t include tests against the SOTA
poisoning models because it will look like active data-protection removal, and
we will leave it to the community.

##  5 Conclusion

In this paper we investigated the possibility of using Huber loss for training
Diffusion Probabilistic models. We presented the sufficient conditions for the
utility of Huber loss when dealing with dataset corruption. Moreover, our
theoretical analysis predicts that Huber loss might require different delta
parameters at different time steps of the backward diffusion process. We
provided experimental evidence of better performance of Huber loss and Huber
loss with time dependent delta parameter in model adaptation setting for both
image and audio domain. Further research in this area may involve identifying
more precise conditions for the applicability of Huber loss for robust model
adaptation. Additionally, a deeper analysis of scheduling of the delta
parameter could be an important research direction.

##  6 Ethical Statement

Because this data-corruption resilience scheme has virtually no cost over
$L_{2}$ loss computation, it will save considerably more resources – that can
go to more society-beneficial goals – as compared to using neural networks to
re-caption, filter out or ”purify” the images, saving money and time for
large-scale model trainers and helping to preserve the environment.

## References

  * Anderson (1982) Anderson, B. D.  Reverse-time Diffusion Equation Models.  _Stochastic Processes and their Applications_ , 12(3):313 – 326, 1982.  ISSN 0304-4149.
  * Carlini et al. (2023) Carlini, N., Hayes, J., Nasr, M., Jagielski, M., Sehwag, V., Tramèr, F., Balle, B., Ippolito, D., and Wallace, E.  Extracting Training Data from Diffusion Models.  In _32nd USENIX Security Symposium (USENIX Security 23)_ , pp. 5253–5270. USENIX Association, aug 2023.
  * Das et al. (2023) Das, A., Yang, Y., Hospedales, T., Xiang, T., and Song, Y.-Z.  ChiroDiff: Modelling chirographic data with Diffusion Models.  In _The Eleventh International Conference on Learning Representations_ , 2023.
  * Dhariwal & Nichol (2021) Dhariwal, P. and Nichol, A.  Diffusion Models Beat GANs on Image Synthesis.  In _Advances in Neural Information Processing Systems_ , volume 34, pp. 8780–8794. Curran Associates, Inc., 2021.
  * Dockhorn et al. (2022) Dockhorn, T., Vahdat, A., and Kreis, K.  GENIE: Higher-Order Denoising Diffusion Solvers.  In _Advances in Neural Information Processing Systems_ , volume 35, pp. 30150–30166. Curran Associates, Inc., 2022.
  * Gao et al. (2023) Gao, S., Liu, X., Zeng, B., Xu, S., Li, Y., Luo, X., Liu, J., Zhen, X., and Zhang, B.  Implicit Diffusion Models for Continuous Super-Resolution.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 10021–10030, June 2023.
  * Han & Lee (2022) Han, S. and Lee, J.  NU-Wave 2: A General Neural Audio Upsampling Model for Various Sampling Rates.  In _Proc. Interspeech 2022_ , pp. 4401–4405, 2022.
  * Hawthorne et al. (2022) Hawthorne, C., Simon, I., Roberts, A., Zeghidour, N., Gardner, J., Manilow, E., and Engel, J. H.  Multi-instrument Music Synthesis with Spectrogram Diffusion.  In _Proceedings of the 23rd International Society for Music Information Retrieval Conference, ISMIR 2022, Bengaluru, India, December 4-8, 2022_ , pp. 598–607, 2022.
  * Ho et al. (2020) Ho, J., Jain, A., and Abbeel, P.  Denoising Diffusion Probabilistic Models.  In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_ , volume 33. Curran Associates, Inc., 2020.
  * Hu et al. (2021) Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W.  Lora: Low-rank adaptation of large language models, 2021.
  * Huber (1964) Huber, P. J.  Robust Estimation of a Location Parameter.  _The Annals of Mathematical Statistics_ , 35(1):73 – 101, 1964.
  * Kawar et al. (2022) Kawar, B., Elad, M., Ermon, S., and Song, J.  Denoising Diffusion Restoration Models.  In _Advances in Neural Information Processing Systems_ , volume 35, pp. 23593–23606. Curran Associates, Inc., 2022.
  * Kingma et al. (2021) Kingma, D., Salimans, T., Poole, B., and Ho, J.  Variational Diffusion Models.  In _Advances in Neural Information Processing Systems_ , volume 34, pp. 21696–21707. Curran Associates, Inc., 2021.
  * Lipman et al. (2023) Lipman, Y., Chen, R. T. Q., Ben-Hamu, H., Nickel, M., and Le, M.  Flow Matching for Generative Modeling.  In _The Eleventh International Conference on Learning Representations_ , 2023.
  * Liptser & Shiryaev (1978) Liptser, R. S. and Shiryaev, A. N.  _Statistics of Random Processes_ , volume 5 of _Stochastic Modelling and Applied Probability_.  Springer-Verlag, 1978.
  * Lu et al. (2022) Lu, C., Zhou, Y., Bao, F., Chen, J., LI, C., and Zhu, J.  DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps.  In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), _Advances in Neural Information Processing Systems_ , volume 35, pp. 5775–5787. Curran Associates, Inc., 2022.
  * Luo et al. (2023) Luo, Z., Chen, D., Zhang, Y., Huang, Y., Wang, L., Shen, Y., Zhao, D., Zhou, J., and Tan, T.  VideoFusion: Decomposed Diffusion Models for High-Quality Video Generation.  In _2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 10209–10218. IEEE Computer Society, jun 2023.
  * Meng et al. (2022) Meng, C., He, Y., Song, Y., Song, J., Wu, J., Zhu, J.-Y., and Ermon, S.  SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations.  In _International Conference on Learning Representations_ , 2022.
  * Nie et al. (2022) Nie, W., Guo, B., Huang, Y., Xiao, C., Vahdat, A., and Anandkumar, A.  Diffusion Models for Adversarial Purification.  In _Proceedings of the 39th International Conference on Machine Learning_ , volume 162 of _Proceedings of Machine Learning Research_ , pp. 16805–16827. PMLR, 17–23 Jul 2022.
  * Owen (2007) Owen, A. B.  A robust hybrid of lasso and ridge regression.  _Contemporary Mathematics_ , 443:59 – 72, 01 2007.
  * Popov et al. (2021) Popov, V., Vovk, I., Gogoryan, V., Sadekova, T., and Kudinov, M.  Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech.  In _Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event_ , volume 139 of _Proceedings of Machine Learning Research_ , pp. 8599–8608. PMLR, 2021.
  * Popov et al. (2022) Popov, V., Vovk, I., Gogoryan, V., Sadekova, T., Kudinov, M., and Wei, J.  Diffusion-Based Voice Conversion with Fast Maximum Likelihood Sampling Scheme.  In _International Conference on Learning Representations_ , 2022.
  * Radford et al. (2021) Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I.  Learning transferable visual models from natural language supervision, 2021.
  * Rombach et al. (2022) Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B.  High-Resolution Image Synthesis With Latent Diffusion Models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 10684–10695, June 2022.
  * Ruiz et al. (2023) Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., and Aberman, K.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation, 2023.
  * Shan et al. (2023a) Shan, S., Cryan, J., Wenger, E., Zheng, H., Hanocka, R., and Zhao, B. Y.  Glaze: Protecting Artists from Style Mimicry by Text-to-Image Models.  In _Proceedings of the 32nd USENIX Conference on Security Symposium_ , SEC ’23. USENIX Association, 2023a.
  * Shan et al. (2023b) Shan, S., Ding, W., Passananti, J., Zheng, H., and Zhao, B. Y.  Prompt-Specific Poisoning Attacks on Text-to-Image Generative Models, 2023b.
  * Shan et al. (2023c) Shan, S., Wu, S., Zheng, H., and Zhao, B. Y.  A response to glaze purification via impress, 2023c.
  * Somepalli et al. (2023) Somepalli, G., Singla, V., Goldblum, M., Geiping, J., and Goldstein, T.  Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Models.  In _2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 6048–6058. IEEE Computer Society, jun 2023.
  * Song & Dhariwal (2023) Song, Y. and Dhariwal, P.  Improved Techniques for Training Consistency Models, 2023.
  * Song et al. (2021a) Song, Y., Durkan, C., Murray, I., and Ermon, S.  Maximum Likelihood Training of Score-Based Diffusion Models.  In _Advances in Neural Information Processing Systems_ , volume 34, pp. 1415–1428. Curran Associates, Inc., 2021a.
  * Song et al. (2021b) Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., and Poole, B.  Score-Based Generative Modeling through Stochastic Differential Equations.  In _International Conference on Learning Representations_ , 2021b.
  * Song et al. (2023) Song, Y., Dhariwal, P., Chen, M., and Sutskever, I.  Consistency models.  In _Proceedings of the 40th International Conference on Machine Learning_ , ICML’23. JMLR.org, 2023.
  * Struppek et al. (2023) Struppek, L., Hentschel, M., Poth, C., Hintersdorf, D., and Kersting, K.  Leveraging Diffusion-Based Image Variations for Robust Training on Poisoned Data.  In _NeurIPS 2023 Workshop on Backdoors in Deep Learning - The Good, the Bad, and the Ugly_ , 2023.
  * Tevet et al. (2023) Tevet, G., Raab, S., Gordon, B., Shafir, Y., Cohen-or, D., and Bermano, A. H.  Human Motion Diffusion Model.  In _The Eleventh International Conference on Learning Representations_ , 2023.
  * von Platen et al. (2022) von Platen, P., Patil, S., Lozhkov, A., Cuenca, P., Lambert, N., Rasul, K., Davaadorj, M., and Wolf, T.  Diffusers: State-of-the-Art Diffusion Models.  https://github.com/huggingface/diffusers, 2022.
  * Wang et al. (2023) Wang, H., Shen, Q., Tong, Y., Zhang, Y., and Kawaguchi, K.  The Stronger the Diffusion Model, the Easier the Backdoor: Data Poisoning to Induce Copyright Breaches Without Adjusting Finetuning Pipeline.  In _NeurIPS 2023 Workshop on Backdoors in Deep Learning - The Good, the Bad, and the Ugly_ , 2023.
  * Xiao et al. (2023) Xiao, C., Chen, Z., Jin, K., Wang, J., Nie, W., Liu, M., Anandkumar, A., Li, B., and Song, D.  DensePure: Understanding Diffusion Models for Adversarial Robustness.  In _The Eleventh International Conference on Learning Representations_ , 2023.
  * Xiao et al. (2022) Xiao, Z., Kreis, K., and Vahdat, A.  Tackling the Generative Learning Trilemma with Denoising Diffusion GANs.  In _International Conference on Learning Representations_ , 2022.
  * Zhang et al. (2018) Zhang, R., Isola, P., Efros, A. A., Shechtman, E., and Wang, O.  The unreasonable effectiveness of deep features as a perceptual metric.  In _CVPR_ , 2018.

##  Appendix A Ablation studies

###  A.1 Impact of different PHL schedules

We adopted a simple exponential decrease/increase for the constant $\delta$ in
the schedule of Pseudo-Huber loss. The scheduling for the exponential decrease
is given by the formula

$\delta=\exp\left({\frac{\log{\delta_{0}}*\text{timestep}}{\text{num train
timesteps}}}\right)$ (11)

The exponential increase (”backwards”) is obtained by time reversal
$[\text{timestep}:=\text{num train timesteps}-\text{timestep}]$:

In the process of our experiments it turned out the Diffusers implementation
of pseudo-Huber loss

$H_{c}^{\text{diffusers}}(x)=\sqrt{|x|^{2}+c^{2}}-c\ ,$ (12)

with the same coordinate-wise extension to multi-dimensional case as losses
(7) and (8), was not fully mathematically correct as it lacked the leading $c$
coefficient resulting in wrong asymptotics for large values of parameter
($H_{c}^{diffusers}\sim\frac{1}{2}\frac{|x|^{2}}{c}$ for large values of $c$
instead of correct asymptotics $\frac{|x|^{2}}{2}$ of pseudo-Huber loss
$H_{\delta}$). While it didn’t influence normal Huber runs because the loss
was proportional to the true one, it was prominent in our cases of time-
dependent $\delta$. Though the formula 12 is incorrect we decided to include
schedulers based in it into comparison to demonstrate its drastically inferior
performance caused by the incorrect asymptotics.

To decide what schedule is better and to test the claim of the theorem about
the limiting cases values (that were overshadowed by the loss’s secondary
dependence on $c$), we include the full comparison here at Figure
LABEL:fig:schedule-study3.

###  A.2 CLIP/LPIPS comparison for stats computation

In addition to the “1 - LPIPS” perceptual metric, we employ CLIP. Figure
LABEL:fig:contamination-study1-addon demonstrates that “1 - LPIPS” score
measurements are largely consistent with CLIP. Still, we have ultimately
chosen “1 - LPIPS” as the main evaluation metric due to the reasons stated in
the main body of the article.

###  A.3 Dependence on the loss constant

For each dataset in addition to L2 training we made two trainings with
different Huber loss parameters. As shown at Figure LABEL:fig:c-study-addon
increasing the constant improves the stability of the curves by making them
closer to L2, although it lowers the robustness potential.

###  A.4 All prompts Resilience comparison plots

Figure LABEL:fig:prompts-study-addon shows the plot of $R$-value at different
training steps. The results at the final step are aggregated in Table 1. At
this level of corruption, out of 7 datasets our method outperforms L2 in all
but one case, though the margin varies.

###  A.5 Difference- versus Division-based R-factor derivation

In our experiments, in addition to (10) we also tried using an alternative
formula for the $R$-value, taking into account both the similarity to the
clean data and the similarity to the poison:

$R=S_{\text{to clean}}^{\text{corrupted}}/S_{\text{to
poison}}^{\text{corrupted}}-S_{\text{to clean}}^{\text{clean}}/S_{\text{to
poison}}^{\text{clean}}$ (13)

While this metric might have been thought to come naturally, where were a
number of stability problems resulting basically in inability to derive any
statistics comparing different datasets because of the high variance – making
the work look “cherry-picky” – and cases causing the factor to unnaturally
jump highly above zero. Some of the problems are outlined below:

  1. 1.

In event of nearly zero similarity to poison the denominator’s small value
would cause the explosion of the fraction and lead to results greater than
zero;

  2. 2.

As cosine similarity can be negative, if the similarity to the poison and the
clean data both switch sides, the $R$-factor will still be positive;

  3. 3.

If the values have proportional positive growth (the similarity to clean data
grows and the similarity to the poison grows), the value will stay constant,
although it will be more representative if it would come down (examples:
$0.5/0.05$, $0.25/0.025$ have the same value).

Therefore it was decided against it. We provide the plots at Fig.
LABEL:fig:r-comparison showing how these metrics influence the dynamics and
the range of the $R$-values for one example prompt for the CLIP-computed
similarity and $45\%$ of pollution.

##  Appendix B Datasets

Below are the samples of the used datasets, one from each. The images are
square-resized to 512x512. The number of images in each dataset varies from 6
to 17. All the used datasets, including the automatically composed final
versions, on which the trainings have been commenced, are available at the
link.333https://drive.google.com/drive/folders/15JwiRxFYDlFnx5SdT1i78tlZztKNUw3f?usp=drive_link

Figure 14: From left to right. Upper row: a picture in the style of
lordbob1111footnotemark: 11, maxwell the cat1212footnotemark: 12, a photo of
an eichpoch1313footnotemark: 13, a picture of a drake1414footnotemark: 14.
Lower row: random internet picture1515footnotemark: 15, a picture by david
revoy1616footnotemark: 16, a photo of a shoan, a landscape of
sks.171717https://www.artstation.com/theartysquid 1717footnotetext:
https://knowyourmeme.com/editorials/guides/who-is-maxwell-the-cat-how-this-
spinning-cat-gif-became-a-viral-meme1717footnotetext:
https://www.ozon.ru/product/podushka-igrushka-echpochmak-
treugolnik-6523833241717footnotetext:
https://github.com/wesnoth/wesnoth/tree/master/data/core/images/portraits/drakes1717footnotetext:
various authors1717footnotetext: https://www.davidrevoy.com/1717footnotetext:
Contribs listed in C

##  Appendix C Acknowledgements

The first author wants to say thanks to his friend
Ruslan181818https://reddit.com/u/ruhomor who provided moral support and an
extensive dataset of his cat named Shoan photos for training purposes, on
which our method ironically fared relatively poorly.

We also thank our friend Ilseyar who made the stunning photos of the castle
and the eichpoch191919https://en.wikipedia.org/wiki/Uchpuchmak toy and gave
them to be used in our project.

The first author expresses his gratitude to the Deforum AI art
community202020https://discord.com/invite/deforum and its main contributors
for being supportive and inspiring.

Additionally, we thank all the artist, photographers and craft makers whose
work we used in our experiments.

We greatly thank the reviewers for pointing out the strong and the weak parts
of the original version of this paper, helping us to correct it and transition
to more objective, numerically, statistically and perceptually stable metrics,
while our positive empirical results remained largely intact.


# Active Queue Management with Data-Driven Delay Violation Probability
Predictors
††thanks: This work was supported by the European Commission through the H2020
project DETERMINISTIC6G (Grant Agreement no. 101096504), in conjunction with
Digital Futures research center.

Samie Mostafavi, Neelabhro Roy, György Dán and James Gross  KTH Royal
Institute of Technology, Stockholm, Sweden
{ssmos, nroy, gyuri<EMAIL_ADDRESS>
###### Abstract

The increasing demand for latency-sensitive applications has necessitated the
development of sophisticated algorithms that efficiently manage packets with
end-to-end delay targets traversing the networked infrastructure. Network
components must consider minimizing the packets’ end-to-end delay violation
probabilities (DVP) as a guiding principle throughout the transmission path to
ensure timely deliveries. Active queue management (AQM) schemes are commonly
used to mitigate congestion by dropping packets and controlling queuing delay.
Today’s established AQM schemes are threshold-driven, identifying congestion
and trigger packet dropping using a predefined criteria which is unaware of
packets’ DVPs. In this work, we propose a novel framework, Delta, that
combines end-to-end delay characterization with AQM for minimizing DVP. In a
queuing theoretic environment, we show that such a policy is feasible by
utilizing a data-driven approach to predict the queued packets’ DVPs. That
enables Delta AQM to effectively handle links with arbitrary stationary
service time processes. The implementation is described in detail, and its
performance is evaluated and compared with state of the art AQM algorithms.
Our results show the Delta outperforms current AQM schemes substantially, in
particular in scenarios where high reliability, i.e. high quantiles of the
tail latency distribution, are of interest.

###### Index Terms:

active queue management, congestion control, delay violation probability,
latency-sensitive applications

††publicationid: pubid: 978-1-979-8-3503-1090-0/23/$31.00 © 2023 European
Union

##  I Introduction

Cyber-physical systems (CPS) integrate computing and communication elements
and physical processes, enabling intelligent control and monitoring of
physical entities [1]. With the rise of CPS, human-in-the-loop (HITL)
applications have emerged where humans interact with CPS to provide decision-
making, supervision, and intervention capabilities. These applications range
from augmented reality systems to autonomous vehicles and industrial
automation.

Real-time applications, such as CPS and HITL, have stringent delay
requirements, often expressed in terms of a target delay that should not be
exceeded more than with a certain probability. This probability, known as the
delay violation probability (DVP), represents the likelihood that a packet
will not reach its destination before the specified deadline [2]. For example,
HITL applications typically have a delay target around 100 ms with a DVP of
0.99 to 0.999 [3, 4].

Existing approaches to meeting such stringent latency requirements fall into
two main groups. On the one hand, there is a large body of work on the
characterization of the end-to-end latency distribution in computer networks,
with queuing theory as a fundamental tool. Deterministic network calculus
approaches aim to analyze the worst-case network latency by considering
specific service and arrival processes. Subsequently, stochastic network
calculus emerged, focusing on deriving stochastic bounds on network latency.
However, these approaches encounter limitations as they rely on assumptions
regarding the service process and apply only to service processes that are
independent and identically distributed (i.i.d.) [5, 6, 7]. These limitations
constrain their applicability in practical scenarios for DVP optimization.

The apparent limitations of analytical approaches have motivated recent
interest in data-driven end-to-end latency distribution prediction schemes [8,
9] and a renewed interest for active queue management (AQM) [10]. AQM drops
packets from buffers before they would overflow, to prevent excessive end-to-
end delays due to long queues. Existing AQM schemes are threshold-driven,
using predefined criteria to identify congestion and trigger packet dropping,
and hence they cannot incorporate application-layer end-to-end delay
requirements in their dropping decision. Thus, the AQM dropping criteria do
not change in response to changing latency requirements, instead tedious
parameter tuning is needed. Clearly, for making the large scale deployment of
latency sensitive applications, there is a need for a methodological approach
for achieving a target DVP in an automated manner.

In this work, we propose a framework that combines end-to-end delay
characterization with AQM for minimizing DVP. The proposed framework makes it
possible to incorporate transient DVP predictions in the AQM packet dropping
decisions. Or main contributions are as follows.

  * •

We propose of a novel AQM framework, Delta, which effectively minimizes end-
to-end delay violations by incorporating DVP predictions in packet dropping
decisions.

  * •

We make use of a data-driven approach for DVP prediction, enabling Delta AQM
to effectively handle links with arbitrary stationary service time processes.

  * •

We evaluate Delta AQM in a representative queueing environment and show that
it outperforms state of the art AQM schemes.

###  I-A Related Work

Active queue management works such as [11, 12] aim to solve the bufferbloat
problem, i.e., queues in the network causing high latency and delay jitter.
Their proposed schemes PIE and CoDel aim to address this problem by limiting
the average queueing latency to a reference value; and by using the minimum
rather than average as the queue measure, simplified single-state variable
tracking of minimum, and by using queue-sojourn time. There have been some
recent works such as [13], which utilize a deep reinforcement learning based
architectures for queue management. The authors show performance gains over
other deep-Q-networks (DQN) based works. This work utilizes a scaling factor
in their reward function in order to arrive at the trade-off between queueing
delay and the throughput. Since different applications would need different
scaling factors, this approach does not generalize well and is heavily
dependent on fine-tuning the reward functions. Another line of research
involves works which aim to minimize the end-to-end latency in queues. Liu et
al. in [14] propose a method towards end-to-end congestion control and queue
management, which aims to reduce the jitters and make the end-to-end latency
deterministic. Building on top of this, authors in [15] formulate a Semi-
Markov Decision Process (SMDP) to obtain an optimal packet dropping policy.
They realize this by assuming a probabilistic model for the flow rates and the
de-queueing pattern. Their work focuses on characterizing the load, emanating
out of their model of choice where packet data arrivals are given by gamma
distributed inter-arrival times. Additionally, they assume the service times
to be exponentially distributed which may not generalize well.

##  II System Model and Problem Statement

In this study, we assume a single first-in, first-out (FIFO) queue with a
deterministic arrival process. We consider such a simple model for exposing
the problem and the proposed solution. We discuss the extension of the
proposed solution to multiple queues in Section V.

Figure 1: Queueing model with AQM

Packets are generated at the start node and queued until the server becomes
available and can process the corresponding packet. Let $M$ = $\\{1,...,m\\}$
denote a set of consecutive time sensitive packets traversing the link. We
denote by $T_{i}$ the arrival time and by $\tau_{i}$ the target delay of
packet $i\in M$. We denote by $W_{i}$ the queueing time of packet $i$, by
$S_{i}$ its service time, and by $Y_{i}=W_{i}+S_{i}$ the resulting sojourn
time. The server can decide to mark a queued packet to be dropped prior to
service, denoted by $X_{i}=0$, in which case the packet has service time
$S_{i}=\infty$ as shown in Figure 1. A packet not marked for dropping, i.e.,
$X_{i}=1$, will enter the server and will experience service time $S_{i}\sim
f_{S}(s\mid X=1)$ with a general stationary process. The selection of a
general distribution for the service delay process stems from our objective to
replicate the stochastic delay observed in network links, particularly
prevalent in wireless networks. Therefore, the probability density function of
the service time is denoted by $f_{S}(s\mid X)$ where the variable
$X\in\\{0,1\\}$ models the capability of the server refusing to process the
packet, i.e. to drop the packet. By default, all packets are assigned
$X_{i}=1$ on their arrival.

Problem Statement: Our optimization objective is the fraction of successfully
processed packets after the $m$-th packet has been processed. It is denoted by

$R_{M}=\frac{\sum_{i=1}^{m}\mathfrak{I}\left[Y_{i}\leq\tau_{i}\right]}{m},$
(1)

where $\mathfrak{I}\left[\cdot\right]$ is the indicator function equalling $1$
if the argument is true and $0$ otherwise. Formally, an AQM scheme provides a
mapping from the currently backlogged packets to a dropping decision vector,
denoted by $\pi:\mathcal{D}_{n}\rightarrow\mathcal{X}_{n}$. The goal is to
find the AQM policy $\pi^{*}$ that maximizes the fraction of successful
packets $R_{M}$ by deciding about dropping some of the packets of the stream,
denoted by the dropping variable $\mathbf{X}:=\\{X_{i}\\}^{m}_{i=1}$.

##  III Approach

To solve this problem, we strive to find the dropping vector that maximizes
the expected outcome limited to the $n$ packets in the queue $(N\subset M)$
denoted by

$x^{\ast}=\operatorname*{arg\,max}_{x\in\mathcal{X}_{n}}\left(\mathbb{E}\left[R_{N}|\mathbf{X}=x\right]\right),$
(2)

where $\mathbf{X}$ represents the dropping vector. Packet drops can be applied
frequently to increase the effect, particularly if the state of the queue
changes rapidly. In order to describe the queue state at dropping time $t$, we
use the notion of the remaining delay budget of each packet denoted by

$\delta_{i,t}=\max(\tau_{i}-(t-T_{i}),0).$ (3)

Hence, we denote the queue state at time $t$ by vector
$\Delta_{t}:=\\{\delta_{i,t}\\}^{n}_{i=1}$. Remaining delay budget is
considered a decisive attribute of the packets during the decision rounds. For
instance, if the remaining delay budget of a packet is very small, the AQM
algorithm can decide not to serve it, so the successors of the packet face
less waiting time.

Our proposed AQM scheme Delta essentially chooses at every decision round the
dropping vector that maximizes the expectation of successfully transmitted
packets from the queue state vector. We solve Equation 2 by finding the
optimization function as

$\mathbb{E}\left[R_{N}|\mathbf{X}=x\right]=\frac{\sum_{i=1}^{n}\mathbb{E}\left[\mathfrak{I}\left[Y_{i}\leq\tau_{i}\right]|\mathbf{X}=x\right]}{n}.$
(4)

The right-hand side expectation term in Equation 4 essentially describes the
probability of packet $i$ finishing before its deadline $\tau_{i}$, given the
dropping vector is applied. We call this expectation the packet’s success
probability $\psi_{i,x}$ and it can be obtained by calculating the
complementary of the packet’s DVP denoted by $\varphi_{i,x}$.

$\psi_{i,x}=1-\varphi_{i,x}=1-\mathbb{E}\left[\mathfrak{I}\left[Y_{i}>\tau_{i}\right]|\mathbf{X}=x\right].$
(5)

The problem is narrowed down to finding the DVP for each packet in the queue.
Assuming such estimation is feasible, a decision tree is formed to represent
the set of all possible dropping vectors and their outcomes.

Figure 2: The decision tree of dropping packets in a queue of length 2 where 0
means drop and 1 means pass.

As shown in Figure 2, the decision tree is solved by traversing the tree from
the root to the leaf nodes, evaluating the objective function mentioned in
Equation 4 associated with each path through the tree. The optimal solution is
the path through the tree that results in the highest
$\Psi_{x_{j}}=\sum_{i=1}^{n}\psi_{i,x_{j}}$.

Figure 3: An example on calculating the objective function for the dropping
vectors $x_{1}$ and $x_{2}$ which consider if packets in the queue should be
dropped.

In a more detailed example shown in Figure 3, the dropping vectors $x_{1}$ and
$x_{2}$ are compared to decide if the first packet should be dropped or not.
The calculated objective function in b is higher, so it is favored.

To estimate the packets’ DVPs, we resort to the approach described in our
previous work [9]. At its core, we estimate the probability density function
(PDF) of the remaining latency $f_{Z}$ for any packet, from the number of its
predecessors in the queue $X$. For the $i$th packet in the queue at time $t$,
DVP is obtained by calculating the remaining latency’s complementary
cumulative density function (CCDF) at the remaining delay budget
$\delta_{t,i}$ denoted by

$\varphi_{i,x}=f_{Z}(z>\delta_{t,i}\mid X).$ (6)

Considering the dropping vector $x$ chosen by Delta AQM, we must incorporate
the number of effective predecessors, or the predecessors that are not going
to be dropped by $X=\textstyle\sum_{j=1}^{i}x_{j}$.

We utilize a machine learning approach, namely mixture density networks, for
latency density estimation for any queued packet at any time in a queuing
system. In mixture density networks, the parameters of a parametric density
function e.g. a Gaussian mixture model, are controlled by a fully connected
neural network. Therefore, we approximate $\varphi_{i,x}$ by the parametric
density function $\hat{p}_{\theta}$ as

$\varphi_{i,x}\approx\hat{p}_{\theta}(z>\delta_{i,t}\mid
X=\textstyle\sum_{j=1}^{i}x_{j}).$ (7)

To train the neural network, a dataset must be formed by recording the number
of predecessors when a packet enters the queue and the end-to-end latency.

The description of our AQM policy Delta with the conditional density function
is described in Algorithm 1.

Algorithm 1 Delta

1:

2:function $\mu$($\Delta_{t}$)

3: $\Psi_{max}$,$\hat{x}$ $\leftarrow$ $0$

4: for $x\in\mathcal{X}_{n}$ do

5: $\Psi_{x}$ $\leftarrow$ $0$

6: for $i\in N$ do

7: $\psi_{i,x}\leftarrow{\hat{p}_{\theta}(z\leq\delta_{i,t}\mid
X=\textstyle\sum_{j=1}^{i}x_{j})}$

8: $\Psi_{x}\leftarrow\Psi_{x}+\psi_{i,x}$

9: end for

10: if $\Psi_{x}>\Psi_{max}$ then

11: $\Psi_{max}\leftarrow\Psi_{x}$

12: $\hat{x}\leftarrow x$

13: end if

14: end for

15: return $\hat{x}$

16:end function

The time complexity of Delta is O($2^{N}$), which is relatively high and could
pose a challenge in practical applications. This complexity indicates that the
algorithm’s computational requirements increase exponentially with the number
of inputs, making it necessary to use a limit on the queue length. Parallel
processing techniques can be utilized to reduce the algorithm’s run time.
These findings suggest room for further investigation into improving the
algorithm’s efficiency in future research.

##  IV Evaluation

This section comprehensively analyzes the proposed Delta AQM scheme in a
simulated queuing system as shown in Figure 1 111The reproducible results
could be found at: https://github.com/samiemostafavi/delta-queue-management.
Firstly, we present a comparative assessment between Delta AQM and two
established and advanced schemes: CoDel, widely implemented in network
infrastructures [12], and DeepQ [13], a state-of-the-art method based on Deep
Reinforcement Learning. Furthermore, we investigate Delta AQM’s sensitivities
to different factors in-depth. We assess its efficacy in diverse scenarios,
such as situations with various delay targets and utilization factors. The
utilization factor is determined by dividing the packet arrival rate by the
average service rate. Additionally, we analyze the performance of Delta AQM in
scenarios where there is a mismatch between the DVP predictor’s learned
behavior about the link and the actual network conditions, providing insights
into its adaptability.

In simulations, we model the queue’s service delay process by a Gamma
distribution where, by default, the concentration is set to $5$ and the rate
to $0.5$ resulting in $0.1$ as the average service rate. The tasks arrival
process is assumed to be deterministic to represent sensory data traversing
the network link. Tasks arrive at the queue carrying a target delay which is
assumed to be constant for all tasks in each simulation. We analyze the
performance over a range of target delay values. This range is defined to
cover the delay target values that are close or far compared to the average
end-to-end delay in the no-aqm simulation. For instance, in all figures, the
$0.8$ target delay corresponds to the $0.8$ quantile of the end-to-end delay
of the no-aqm simulation. In all benchmarks we compared the AQM schemes by the
failed tasks ratio introduced in Equation 1. We set the number of completed
tasks $m$ in the simulations to at least $10^{7}$ which is large enough to
mitigate the performance variance even for large target delays. In order to
prevent excessively long processing times during simulations, we constrained
the AQM to focus on the initial 15 packets within the queue.

We include the no-AQM system performance and offline-optimum policy
performance in the comparisons. The offline-optimum policy already knows the
end-to-end delay of the tasks in the queue. At every decision round, it drops
the head packet if its end-to-end delay exceeds the delay target. Thus, every
packet being served will make the deadline. Due to access to the delay
information, the offline-optimum AQM scheme’s performance is expected to be
superior to all others.

(a) Utilization factor: $91.6\%$

(b) Utilization factor: $96.7\%$

Figure 4: Comparing different AQM schemes

The evaluation starts by comparing our proposed approach with CoDel [12],
DeepQ [13], the offline-optimum and the case with no-AQM. CoDel, as mentioned
earlier, operates based on predefined criteria and thresholds to detect
congestion and initiate packet dropping, which are determined by interval time
and target delay. In our evaluation, we compare the performance of the AQM
schemes across four different target delay values, varying from those close to
the average no-AQM delay quantiles to those significantly further. Notably,
for each target delay, it was essential to readjust the CoDel parameters;
otherwise, the scheme’s performance exhibited a notable degradation. The same
requirement applied to DeepQ, necessitating a retraining phase to adapt its
policy parameters for each target delay. In contrast, our proposed Delta
scheme demonstrates an advantageous characteristic, as it does not require any
parameter tuning or retraining when the packet’s delay targets change. The
same Delta model remained applicable and effective across all four target
delays without modifications. We demonstrate this in Figure 4(a) and 4(b)
where we increase the utilization factor moving from Figure 4(a) to Figure
4(b). As can be observed from the figures, our proposed approach consistently
performs better than the other approaches, barring the offline-optimum. On
moving from Figure 4(a) to Figure 4(b) we can observe an improvement in
performance for the algorithms (except no-AQM) in terms of fewer task
failures.

Figure 5: Effect of sparsely populated queues

We extend the analysis of utilization factors by comparing 91.6% and 96.7%
from Figure 4(a) with 80.7% and 60% additionally in Figure 5. ”Delta 60” in
Figure 5 corresponds to a utilization factor of 60% and similarly ”Delta 80”
corresponds to a utilization factor of 80.7% and finally ”Delta 91.6” in
Figure 5 corresponds to a utilization factor of 91.6%. The nomenclature for
the offline optimum schemes follows a similar trend, being abbreviated as
”oo”. Here we can observe that as the utilization factor drops appreciably to
60%, Delta’s performance is almost similar to that of no-AQM. As the
utilization factor increases to 80.7%, the performance improves and finally at
91.6%, the performance is significantly better than that of no-AQM. The
offline optimum approach too follows a similar trend as Delta, approaching no-
AQM closely for a utilization of 60% and then gradually improving as the
utilization factor increases. All these findings can be attributed to how, for
sparsely populated queues, there are more drops in terms of the total number
of packets in the queue at any time. It is also worth noting that the target
delay corresponding to the 0.8th quantile with the utilization factor of
$80.7\%$ is different from the one with utilization factor of $91.6\%$.

Figure 6: Analysis of higher quantiles of target delay, utilization factor:
$91.6\%$

Figure 6 extends our analysis for the implementation with the utililzation
factor of 91.6% to higher quantiles of target delays. In this plot, we can
additionally observe and compare no-AQM, our proposed approach Delta and the
offline optimum for the 0.9999th and 0.99999th quantile of target delays. It
is interesting to note here that across the various quantiles of target delays
being analyzed, the offline optimum performs at least 10 times better than the
no-AQM scheme in terms of task failure ratios. Moreover, our proposed solution
approaches the offline-optimum closely.

Next, we analyze the performance when the DVP predictor is trained over a
varying number of samples in Figure 7. The label ”delta 512” denotes the case
where our proposed algorithm is just trained over 512 samples. This is however
repeated over multiple iterations and the average of the runs is shown, along
with the minimum and maximum range of the values obtained (by the blue
marker). Here we can observe that while the average of the different runs has
a better performance (in terms of task failures) than the case with no AQM, in
some runs, the performance equals that of ”No AQM”. The training with just 512
samples can thus be concluded as inadequate. On the other hand, when trained
over 4096 samples, the performance is consistently better than the case with
no AQM and also the case with ”delta 512”, showing that increasing the number
of training samples has a positive impact.

Figure 7: The effect of using a poorly trained DVP predictor on the
performance of the Delta AQM, utilization factor: $90.6\%$

Finally, we evaluate our approach over cases where our predictor is exposed to
differing Gamma service processes against a different service process it was
trained over.

Figure 8: The effect of testing against varying Gamma rates on the performance
of the Delta AQM, utilization factor: $90.6\%$

Figure 8 shows how different algorithms (and our proposed algorithm, tested in
varying conditions) perform against each other with respect to the failed task
ratios, observed across specific target delays. While our proposed algorithm
has undergone the same training process here, it has been tested across
different service processes which correspondingly have different means and
variances. The dots in the bars represent the delayed packet ratio out of the
total tasks. The transparent region (non-dotted) represents the dropped packet
ratio.

We can observe that the least number of failed tasks correspond to the case
where the predictor is tested against a service process which follows the same
delay distribution as the training process. The Gamma rate of the service
process is then varied, while also proportionately changing the packet arrival
rate, so as to maintain the same utilization factor in the benchmark. We
compare our proposed approach against a case where no-AQM is performed and a
case where the queue manager has all the information about incoming packets
apriori which we label as the offline-optimum. For the case where our proposed
approach is trained and tested over the same service process (denoted by Mean
= 10*), we can observe that the performance is consistently better than no-
AQM. This also holds true when we change the mean of the service process by
+-2 while testing (denoted by Mean = 8 and 12). Interestingly, when we test
Delta over a significantly different service process (Mean= 3.33) than what we
trained over, it performs worse than no-AQM case. As we move higher from our
10* delta scheme, towards a higher mean of the service process, we observe a
significantly higher proportion of delayed tasks owing to the higher service
mean, as most tasks can be accommodated in principle, but at the cost of them
exceeding their delay bounds. As we move lower from our 10* delta AQM scheme,
towards a lower service mean, we observe that delayed packets are
significantly lesser than the dropped packets. This is due to the fact that
when the service process has a lower mean, there are more failed tasks, most
of which cannot be accommodated without them being dropped. The general
observation however remains that on testing across varying Gamma rates, there
are more task failures.

##  V Conclusions

This paper presented Delta, a novel AQM scheme that effectively minimizes end-
to-end delay violations by incorporating DVP predictions in dropping
decisions. By utilizing a data-driven DVP predictor, Delta AQM offers a
promising solution for links with general stationery service time processes.
Numerical evaluations in a queuing theoretic environment demonstrate that
Delta performs consistently well across a wide range of delay targets without
the need for modifications. Comparative assessments highlight the advantages
of Delta’s parameter-free approach compared to other schemes that require
parameter adjustments or retraining for each target delay. Additionally, we
conducted a thorough analysis of Delta AQM’s sensitivity to the number of
training samples, as well as its performance under mismatched scenarios where
the characteristics of the link service process differ from the trained model.
These investigations shed light on the sensitivities of Delta AQM, driven by
the underlying machine learning algorithm.

In future work, an important direction of investigation will focus on
assessing the efficacy and advantages of Delta AQM in multi-hop queues. In the
multi-hop scenario, the utilization of the data-driven DVP predictor holds the
potential to construct a more comprehensive probabilistic model of latency,
enabling enhanced decision-making capabilities. This investigation will
provide valuable insights into the applicability and benefits of Delta AQM in
complex multi-hop network environments.

## References

  * [1] E. A. Lee, “Cyber physical systems: Design challenges,” in 2008 11th IEEE International Symposium on Object and Component-Oriented Real-Time Distributed Computing (ISORC), pp. 363–369, 2008.
  * [2] C.-S. Chang, Performance guarantees in communication networks.  Springer Science & Business Media, 2000.
  * [3] G. P. Sharma, D. Patel, J. Sachs, M. D. Andrade, J. Farkas, J. Harmatos, B. Varga, H.-P. Bernhard, R. Muzaffar, M. K. Atiq, F. Duerr, D. Bruckner, E. Montesdeoca, D. Houatra, H. Zhang, and J. Gross, “Towards deterministic communications in 6g networks: State of the art, open challenges and the way forward,” 2023.
  * [4] P. Schulz, M. Matthe, H. Klessig, M. Simsek, G. Fettweis, J. Ansari, S. A. Ashraf, B. Almeroth, J. Voigt, I. Riedel, A. Puschmann, A. Mitschele-Thiel, M. Muller, T. Elste, and M. Windisch, “Latency critical IoT applications in 5G: Perspective on the design of radio interface and network architecture,” IEEE Communications Magazine, vol. 55, no. 2, pp. 70–78, 2017.
  * [5] Y. Jiang, Y. Liu, et al., Stochastic network calculus, vol. 1.  Springer, 2008.
  * [6] M. Fidler and A. Rizk, “A guide to the stochastic network calculus,” IEEE Communications Surveys & Tutorials, vol. 17, no. 1, pp. 92–105, 2014.
  * [7] J. P. Champati, H. Al-Zubaidy, and J. Gross, “Transient analysis for multihop wireless networks under static routing,” IEEE/ACM Transactions on Networking, vol. 28, no. 2, pp. 722–735, 2020.
  * [8] A. Sawabe, Y. Shinohara, and T. Iwai, “Delay Jitter Modeling for Low-Latency Wireless Communications in Mobility Scenarios,” in GLOBECOM 2022 - 2022 IEEE Global Communications Conference, pp. 2638–2643, Dec. 2022.
  * [9] S. S. Mostafavi, G. Dán, and J. Gross, “Data-driven end-to-end delay violation probability prediction with extreme value mixture models,” in 2021 IEEE/ACM Symposium on Edge Computing (SEC), pp. 416–422, 2021.
  * [10] R. Adams, “Active queue management: A survey,” IEEE Communications Surveys & Tutorials, vol. 15, no. 3, pp. 1425–1476, 2013.
  * [11] R. Pan, P. Natarajan, C. Piglione, M. S. Prabhu, V. Subramanian, F. Baker, and B. VerSteeg, “Pie: A lightweight control scheme to address the bufferbloat problem,” in 2013 IEEE 14th International Conference on High Performance Switching and Routing (HPSR), pp. 148–155, 2013.
  * [12] K. Nichols and V. Jacobson, “Controlling queue delay,” Commun. ACM, vol. 55, p. 42–50, jul 2012.
  * [13] M. Kim, M. Jaseemuddin, and A. Anpalagan, “Deep Reinforcement Learning Based Active Queue Management for IoT Networks,” Journal of Network and Systems Management, vol. 29, p. 34, Apr. 2021.
  * [14] J. Liu, J. Huang, W. Jiang, Z. Li, Y. Li, W. Lyu, W. Jiang, J. Zhang, and J. Wang, “End-to-end congestion control to provide deterministic latency over internet,” IEEE Communications Letters, vol. 26, no. 4, pp. 843–847, 2022.
  * [15] S. Kar, B. Alt, H. Koeppl, and A. Rizk, “PAQMAN: A Principled Approach to Active Queue Management,” 2022.  Publisher: arXiv Version Number: 1.