# Perceptual Constancy Constrained Single Opinion Score Calibration for Image
Quality Assessment

Lei Wang, Desen Yuan The authors are with the School of Information and
Communication Engineering, University of Electronic Science and Technology of
China, Chengdu, 611731, China.

###### Abstract

In this paper, we propose a highly efficient method to estimate an image’s
mean opinion score (MOS) from a single opinion score (SOS). Assuming that each
SOS is the observed sample of a normal distribution and the MOS is its unknown
expectation, the MOS inference is formulated as a maximum likelihood
estimation problem, where the perceptual correlation of pairwise images is
considered in modeling the likelihood of SOS. More specifically, by means of
the quality-aware representations learned from the self-supervised backbone,
we introduce a learnable relative quality measure to predict the MOS
difference between two images. Then, the current image’s maximum likelihood
estimation towards MOS is represented by the sum of another reference image’s
estimated MOS and their relative quality. Ideally, no matter which image is
selected as the reference, the MOS of the current image should remain
unchanged, which is termed perceptual cons tancy constrained calibration
(PC3). Finally, we alternatively optimize the relative quality measure’s
parameter and the current image’s estimated MOS via backpropagation and
Newton’s method respectively. Experiments show that the proposed method is
efficient in calibrating the biased SOS and significantly improves IQA model
learning when only SOSs are available.

###### Index Terms:

image quality assessment

##  I Introduction

Image quality assessment (IQA) [1, 2, 3, 4, 5, 6, 7, 8] aims to enable
computers to automatically predict the subjective quality of an image, which
is consistent with the human perception. As the ground truth, the subjective
quality label is the key to IQA model learning and evaluating. However, due to
subjective preference differences, collecting reliable subjective quality
labels is greatly challenging, especially for a large-scale image dataset with
diverse visual content.

The mean opinion score (MOS) [9] is the most widely used subjective image
quality label, which invites multiple human subjects and uses the mean value
of their annotations as the image quality label. Given a large-scale image
dataset, the MOS collection is usually achieved via crowdsourcing, whose
annotation process is uncontrollable and easily results in noisy ratings.
Then, a series of standardized outlier rejection and bias removal methods are
developed by the international telecommunication union (ITU) recommendations
including the P.910 [10], P.913 [11], BT.500 [12]. In [13], Li et al. further
generalizes P.913’s raw opinion scores screening procedure by joint modeling
the subject bias and inconsistency. Despite successfully suppressing
subjective preference differences, existing screening methods require multiple
subjects in measuring the consistency of their ratings, which is very
expensive and time-consuming.

To address this issue, in this paper, we propose a perceptual constancy
constrained calibration (PC3) method, which could estimate the MOS from the
single opinion score (SOS). Unlike the bias and inconsistency analysis across
multiple subjects [10, 11, 12, 13], we model the SOS of an image with the
normal distribution, whose unknown expectation is our desired MOS. In this
way, the MOS estimation is formulated as a maximum likelihood estimation
problem, which tries to find the distribution parameters to fit the observed
SOS best. More specifically, we consider the perceptual correlation of
pairwise images in modeling the likelihood of the SOS and represent the
estimated MOS of an image as the sum of another reference image’s estimated
MOS and a learnable relative quality measure, which leverages the quality-
aware representations learned from the self-supervised backbone [14] and
predicts the MOS difference between two images. Ideally, no matter which image
is selected as the reference, the MOS of the current image should remain
unchanged. Finally, we alternatively optimize the relative quality measure’s
parameter and the current image’s estimated MOS via backpropagation and
Newton’s method respectively.

This simple and effective method is suitable for subjective quality label
post-processing when only single opinion scores are available, which
significantly reduces the cost of MOS collection. Experiments on four popular
IQA datasets show that the proposed method is efficient in calibrating the
biased SOS and significantly improves the performance of the IQA model when
only the SOSs are available for training.

Figure 1: The overall framework of the proposed method. The input includes an
image $x_{n}$ and its SOS $y_{n}$. We calibrate $y_{n}$ by iteratively
updating the estimated MOS $\mu_{x_{n}}$ via our PC3, which requires a
randomly selected reference image.

##  II Related Work

Image quality label collection. Human decision-making presents uncertainty
[15], which brings a great challenge for image quality label collection. To
reduce external interference as much as possible, ITU recommends a highly
controlled environment for the subjective tests. In BT.500 [12], the number of
subjects, viewing conditions, test materials, presentation order, and time are
all standardized to collect the subjects’ ratings. The brightness and contrast
of the display devices and the standard definition of test materials are also
discussed in [16].

Due to limited resources, this highly controlled environment is hard to apply
to multiple subjects’ parallel ratings. In practice, more and more subjective
tests turn to the crowdsourcing platform [17], especially for large-scale
image datasets, which invites multiple internet workers to conduct
uncontrolled subjective experiments. Unsurprisingly, more noise would be
collected in these uncontrolled subjective experiments.

Post-processing for unreliable ratings. To clean the raw opinion scores, there
are also several post-processing methods developed to detect and eliminate
unreliable ratings [18]. To suppress the subjective preference difference of
individuals, it is common practice to average the raw opinion scores from
multiple subjects, yielding a MOS per stimulus. Standardized recommendations
include more advanced corrective mechanisms to compensate for the influence of
test subjects, as well as criteria for determining MOS confidence intervals.

ITU-R BT.500 [12] Recommendation defines the procedure that counts the number
of occurrences when a subject’s opinion score deviates by a few sigmas (i.e.
standard deviation) and rejects the subject if the occurrences are more than a
fraction. The recommendation also specifies the method for calculating the
confidence interval (ITU-R BT.500 Section A1-2.2.1). ITU and researchers have
proposed a number of crowd-sourced data processing standards [12, 10, 11] to
eliminate subjective bias in MOS. These methods mainly rely on inconsistency
analysis across multiple subjects, which is expensive, time-consuming, and
difficult to apply to a single opinion score.

##  III Proposed Method

As shown in Fig. 1, we propose an iterative update framework for MOS
estimation. The inputs include the image and its SOS, and the output is the
estimated MOS. In each update step, we randomly sample a reference image with
an estimated MOS and update the input image’s MOS by measuring their relative
quality. Let $y$ denote the SOS of an image $x$, and $p(y|x;\mu)$ denote the
likelihood of SOS with the unknown expectation $\mu_{x}$, which is our desired
MOS. Since there are no additional subject ratings for $x$, we try to further
utilize the perceptual correlation of multiple images for SOS calibration,
which reformulates the likelihood of SOS as

$\displaystyle p(y|x;\mu_{x})=$
$\displaystyle\mathrm{E}_{r_{x}|x}[p(y|x,{r_{x}},\mu_{r_{x}};\mu_{x})]$ (1)
$\displaystyle=$
$\displaystyle\sum_{m=1}^{M}p[y|x,{r},\mu_{r_{x}(m)};\mu_{x}]p[r_{x}(m)|x]$

where $r_{x}(m)$ is the $m$th reference image of $x$, $\mu_{r_{x}(m)}$ is its
MOS, and $M$ is the total number of available reference images.

According to our perceptual constancy constraint,
$p[y|x,{r_{x}},\mu_{r_{x}(m)};\mu_{x}]$ and $p[r_{x}(m)|x]$ are the same for
any $m$, which results to

$p(y|x;\mu_{x})\propto p(y|x,{r_{x}},\mu_{r_{x}};\mu_{x}),$ (2)

where the likelihood of SOS is transformed from the conditional probability of
a single image to the joint conditional probability of pairwise images.

In this context, we assume that $p(y|x,{r_{x}},\mu_{r_{x}};\mu_{x})$ still
follows the normal distribution and introduces a relative quality measure
$S_{\theta}(x,r_{x})$ to describe it, i.e.,

$p(y|x,{r_{x}},\mu_{r_{x}};\mu_{x})\sim\mathcal{N}\left(\mu_{r_{x}}+S_{\theta}(x,r_{x}),{\sigma}^{2}\right),$
(3)

where $\sigma$ is the standard deviation of the distribution.
$S_{\theta}(x,r_{x})$ is a Siamese network, whose two branches share the same
backbone, i.e., CONTRIQUE [14]. The two branches extract the quality-aware
features from $x$ and $r_{x}$. The difference between them is then fed to
three fully connected layers to obtain their relative quality, where $\theta$
is the parameter of this Siamese network.

Following previous settings, given a collection of single opinion scores
$\\{y_{n}\\}^{N}_{n=1}$ from a subjective experiment, our goal is to solve the
unknown parameters $(\theta,U)$ to fit the observed SOSs best, where
$U=\\{\mu_{x_{n}}\\}^{N}_{n=1}$. This task could be formulated as a maximum
likelihood estimation problem. The log-likelihood of all images could be
represented by

$\begin{aligned}
\ell(\theta,U)&=\log\prod_{n=1}^{N}p[y_{n}\mid\mu_{r_{x_{n}}}+S_{\theta}(x_{n},r_{x_{n}})]\\\
&=\sum_{n=1}^{N}\log
p[y_{n}\mid\mu_{r_{x_{n}}}+S_{\theta}(x_{n},r_{x_{n}})]\end{aligned}.$ (4)

It is noted that the maximum-likelihood estimate is equivalent to the least-
squares estimate for normal distribution [19]. Then, we replace the log-
likelihood based objective function with the following mean square error loss,
i.e.,

$\ell(\theta,U)\cong\frac{1}{N}\sum_{n=1}^{N}[y_{n}-S_{\theta}(x_{n},r_{x_{n}})-\mu_{r_{x_{n}}}]^{2}$
(5)

where $\cong$ denotes equal with the omission of constant terms. Besides the
supervision from $y_{n}$, our perceptual constancy constraint also requires
that the MOS of the current image should remain unchanged with different
reference images. So, the MOS estimation is converted into a least-squares
estimate problem with equality constraint, i.e.,

$\underset{\theta,U}{\arg\min}~{}\ell(\theta,U),\text{ s.t.
}\mu_{x_{n}}=S_{\theta}(x_{n},r_{x_{n}})+\mu_{r_{x_{n}}}.$ (6)

To balance these two objectives, we unify them together with the Lagrangian
function

$L(\theta,U)=\ell(\theta,U)+\beta\cdot R(\theta,U),$ (7)

where
$R(\theta,U)=\frac{1}{N}\sum_{n=1}^{N}[\mu_{x_{n}}-S_{\theta}(x_{n},r_{x_{n}})-\mu_{r_{x_{n}}}]^{2}$,
$\beta$ is the Lagrangian multiplier. Then, the current image’s MOS
$\mu_{x_{n}}$ and the relative quality measure’s parameter $\theta$ are solved
via an alternative optimization strategy. MOS estimation is updated by the
Newton-Raphson rule, i.e.,

$\begin{aligned}
\mu_{x_{n}}^{t+1}\leftarrow&(1-\alpha)\cdot\mu_{x_{n}}^{t}+\alpha\cdot\Delta^{t}\\\
\Delta^{t}=&\mu_{x_{n}}^{t}-\frac{\partial
L(\mu_{x_{n}})/\partial\mu_{x_{n}}}{2\lambda\partial^{2}L(\mu_{x_{n}})/\partial\mu_{x_{n}}^{2}}\\\
=&S_{\theta^{t}}(x_{n},r_{x_{n}}^{t})+\mu_{r_{x_{n}}}^{t}\end{aligned},$ (8)

where $t$ is the iteration step, and $\mu_{x_{n}}^{0}=y_{n}$. As shown in Fig.
1, $r_{x_{n}}^{t}$ is randomly sampled from all candidate reference images,
which may change in every iteration. $\alpha$ is a manually set refresh rate.
Meanwhile, $\theta$ is optimized via the back-propagation of the Siamese
network, i.e.,

$\theta^{t+1}=\theta^{t}-\lambda\cdot\frac{\partial
L(\theta)}{\partial\theta},$ (9)

where $\lambda$ is the learning rate. It is noted that the initial relative
quality measure is unreliable. In our alternative optimization process, we set
a warm-up time of $T_{h}$, where the MOS estimation update only activates when
$S_{\theta}(\cdot,\cdot)$ has completed $T_{h}$ times optimization, i.e.,

$u^{t+1}_{x_{n}}=\left\\{\begin{array}[]{cc}(1-\alpha)\cdot\mu_{x_{n}}^{t}+\alpha\cdot\Delta^{t},&t\geq
T_{h}\\\ y_{n},&\text{otherwise}\end{array}.\right.$ (10)

##  IV Experiments

###  IV-A Datasets and Settings

In our experiment, we investigate the performance of the proposed method on
four popular IQA databases including the VCL [18], TID2013 [20], KONIQ[21],
LIVEC[22], which cover both the synthetic and authentic distortions. Since the
subjective annotations are different across different databases, we develop
specific SOS settings in the following three cases. 1) VCL: All raw opinion
scores are available. The SOS is obtained by randomly sampling a raw opinion
score across all subjects for each image; 2) TID & LIVEC: MOS and the
corresponding standard deviation are available. The SOS is obtained by random
sampling from a normal distribution, which uses the pairwise MOS and standard
deviation as the distribution parameters; 3) KONIQ: The empirical distribution
of all ratings is available. The SOS is obtained by random sampling from the
empirical distribution of each image’s ratings. To avoid bias toward specific
SOS choices, we repeat the random sampling 10 times across all databases and
report the median results for comparison.

For all databases, the original MOS is used as the ground truth. We evaluate
the accuracy of the estimated MOS with three widely used metrics including
Pearson’s Linear Correlation Coefficient (PLCC), Spearman’s Rank Order
Correlation Coefficient (SRCC), and the Mean Square Error (MSE). To the best
of our knowledge, SOS calibration is rarely discussed in IQA research. When
only a single opinion score is available, the existing post-processing or
screening methods [10, 11, 12, 13] are inapplicable. For comparison, we chose
the SOS as the baseline and consider the performance of MOS collected from
abundant subjects as the theoretical upper bound.

###  IV-B Implementation Details

In our experiment, we investigate the performance of the proposed SOS
calibration method and its effect on IQA model learning, which employs a
representative deep IQA model NIMA [23]. For both the
$S_{\theta}(x_{n},r_{x_{n}})$ and NIMA, the input images are resized and
center-cropped to 320, and the initial learning rate $\lambda$ is set to
0.0001 with the Adam optimizer. For our PC3, the additional hyperparameter
settings include $\beta=1/9$, $\alpha=0.1$, $T_{h}=1$. For NIMA, we randomly
split (except for KONIQ which has an official split) each IQA database into
non-overlapped training and testing sets. The random split is repeated 10
times and the median results are reported in the following.

Figure 2: The variation of estimated MOS in the iterative update process of
PC3. TABLE I: SOS calibration performances on TID, VCL, LIVEC, and KONIQ. We
report the median SROCC, PLCC, and MSE results between the calibrated SOS and
the ground truth MOS across 10 times of random sampling.

Datasets | TID | VCL | LIVEC | KONIQ
---|---|---|---|---
Methods | SROCC | PLCC | MSE | SROCC | PLCC | MSE | SROCC | PLCC | MSE | SROCC | PLCC | MSE
MOS | 1.0000 | 1.0000 | 0.0000 | 1.0000 | 1.0000 | 0.0000 | 1.0000 | 1.0000 | 0.0000 | 1.0000 | 1.0000 | 0.0000
SOS | 0.6993 | 0.7048 | 0.0185 | 0.8317 | 0.8422 | 0.0233 | 0.6984 | 0.7228 | 0.0350 | 0.6519 | 0.6916 | 0.0230
PC3 | 0.7683 | 0.7807 | 0.0103 | 0.8834 | 0.8834 | 0.0156 | 0.7509 | 0.7715 | 0.0211 | 0.7237 | 0.7584 | 0.0148
$\Delta$(%) | 9.8670 | 10.7690 | -44.3243 | 6.2162 | 4.8919 | -33.0472 | 7.5172 | 6.7377 | -39.7143 | 11.0140 | 9.6588 | -35.6522

TABLE II: SOS calibration performance on TID, VCL, LIVEC, and KONIQ under
different bias rates. We report the median SROCC, PLCC, and MSE results
between the calibrated SOS and the ground truth MOS across 10 times of random
sampling.

Datasets |  | TID | VCL | LIVEC | KONIQ
---|---|---|---|---|---
Methods | rate | SROCC | PLCC | MSE | SROCC | PLCC | MSE | SROCC | PLCC | MSE | SROCC | PLCC | MSE
SOS | 0.6 | 0.7959 | 0.7923 | 0.0109 | 0.8924 | 0.8966 | 0.0141 | 0.7799 | 0.7985 | 0.0225 | 0.7490 | 0.7836 | 0.0139
PC3 | 0.8343 | 0.8474 | 0.0064 | 0.9151 | 0.9121 | 0.0113 | 0.7998 | 0.8226 | 0.0154 | 0.8012 | 0.8277 | 0.0093
$\Delta$(%) |  | 4.8247 | 6.9544 | -41.2844 | 2.5437 | 1.7288 | -19.8582 | 2.5516 | 3.0182 | -31.5556 | 6.9693 | 5.6279 | -33.0935
SOS | 0.8 | 0.7522 | 0.7533 | 0.0144 | 0.8586 | 0.8604 | 0.0191 | 0.7340 | 0.7528 | 0.0299 | 0.6977 | 0.7352 | 0.0187
PC3 | 0.8158 | 0.8321 | 0.0075 | 0.8912 | 0.8868 | 0.0142 | 0.7722 | 0.7900 | 0.0189 | 0.7676 | 0.7951 | 0.0119
$\Delta$(%) |  | 8.4552 | 10.4606 | -47.9167 | 3.7969 | 3.0683 | -25.6545 | 5.2044 | 4.9416 | -36.7893 | 10.0186 | 8.1474 | -36.3636

Figure 3: Error bar between MOS and SOS/PC3 in VCL dataset. TABLE III: The
performance of IQA model learned from different subjective quality labels. We
report the median SROCC, PLCC, and MSE results between the predicted MOS of
NIMA and the ground truth MOS across 10 times of random splitting.

Datasets | TID | VCL | LIVEC | KONIQ
---|---|---|---|---
Methods | SROCC | PLCC | MSE | SROCC | PLCC | MSE | SROCC | PLCC | MSE | SROCC | PLCC | MSE
MOS | 0.7297 | 0.8011 | 0.0261 | 0.9293 | 0.9233 | 0.0130 | 0.8110 | 0.8538 | 0.0120 | 0.8937 | 0.9126 | 0.0032
SOS | 0.4677 | 0.4956 | 0.0443 | 0.7620 | 0.7600 | 0.0357 | 0.5515 | 0.5630 | 0.0528 | 0.5577 | 0.5838 | 0.0271
PC3 | 0.5134 | 0.5605 | 0.0318 | 0.8589 | 0.8633 | 0.0203 | 0.6787 | 0.6883 | 0.0283 | 0.6835 | 0.7109 | 0.0140
$\Delta$(%) | 9.7712 | 13.0952 | -28.2167 | 12.7165 | 13.5921 | -43.1373 | 23.0644 | 22.2558 | -46.4015 | 22.5569 | 21.7712 | -48.3395

###  IV-C Performance of SOS Calibration

In Fig. 2, we first illustrate the variation of estimated MOS in the iterative
update process, where an input image $x_{i}$ and its reference images in three
different iterations are visualized for clarity. It is seen that our
calibrated result $\mu_{x_{n}}$ gradually approaches its ground-truth MOS, and
the change of the reference images would not affect the stability of the
iterative update process.

The overall SOS calibration performances across all databases are shown in
Table I. We can find that the proposed PC3 is effective in calibrating the
biased SOS, which significantly improves the SRCC, PLCC, and MSE performance
across all databases. In addition, the proposed method is also robust to
different distortion types, whose performance improvements are similar to the
synthetic distortion in TID, VCL, and the authentic distortions in LIVEC,
KONIQ. For clarity, in Fig. 3, we also show the error bar between MOS and
SOS/PC3 across the full range of subjective qualities in the VCL database. It
is seen that the proposed PC3 could reduce the bias of SOS across different
subjective qualities.

To further investigate the impact of PC3 on subjective labels mixed by MOS and
SOS, we introduce the bias rate to control the proportion of SOS on the whole
subjective label set. As shown in Table II, the proposed method is also
efficient for calibrating the mixed subjective labels, whose performances are
superior to the SOS. Meanwhile, we can find that the performance improvement
of PC3 is more significant for a high bias rate. It is not surprising that a
higher bias rate provides a larger improvement space from more SOS.

###  IV-D Impact for IQA Model Learning

To investigate the impact of PC3 on IQA model learning, we train a
representative deep IQA model NIMA [23] with the ground-truth MOS, SOS, and
the calibrated SOS via PC3, respectively. It is seen that the performance of
NIMA would significantly degrade when SOS is directly used as the subjective
label. By contrast, with the calibration of our PC3, the performance
degradation of NIMA would be efficiently alleviated.

###  IV-E Generalization to Few Opinion Scores

Besides SOS, we further investigate the generalization ability of the proposed
PC3 on the mean value of Few Opinion Scores (FOS), which contain multiple
subjects but the number of subjects is smaller than the minimal requirement,
i.e., 15, in [10, 11, 12, 13]. We conduct this experiment on VCL which
provides all raw opinion scores for each image. The investigated number of
subjects includes 1, 2, 4, and 8, and the proposed PC3 is applied to the mean
values of FOS to calibrate the bias. Fig. 4 shows the comparison results
between the mean value of FOS and our PC3. It is seen that the PC3 outperforms
FOS when the number of subjects is smaller than 4. Although PC3 may degrade
the FOS when the number of subjects increases to 8, our performance is
comparable, and the reported SRCC and PLCC results are both very high, which
are larger than 0.97.

TABLE IV: Performance comparison of PC3 with different $\alpha$ settings. We
conduct this test on the VCL database and report the results between the
calibrated SOS and the ground-truth MOS.

| SROCC | PLCC | KROCC | MSE
---|---|---|---|---
$\alpha=0.8$ | 0.6792 | 0.6822 | 0.4946 | 0.0416
$\alpha=0.6$ | 0.7814 | 0.7765 | 0.5923 | 0.0298
$\alpha=0.2$ | 0.8934 | 0.8911 | 0.7104 | 0.0150
$\alpha=0.1$ | 0.9052 | 0.9085 | 0.7286 | 0.0124
$\alpha=0.0$ | 0.8317 | 0.8422 | 0.6436 | 0.0233

###  IV-F Effect of $\alpha$

This section investigates the robustness of PC3 toward different $\alpha$
settings on the VCL database. As shown in Eq. (10), $\alpha$ controls the
update speed of SOS. Too small or too large $\alpha$ may result in under- or
over-calibration. As shown in Table IV, the PC3 is efficient when $\alpha$ is
smaller than 0.2 and significantly degrades when $\alpha$ is larger than 0.6.
The best calibration results were obtained in $\alpha=0.1$, which is employed
by the proposed method.

Figure 4: Performance comparison of FOS with different numbers of subjects.

##  V Conclusions

This paper proposes a simple yet efficient subjective quality label post-
processing method for IQA. Instead of relying on the bias and inconsistency
analysis across multiple subjects, we only utilize the SOS and the perceptual
correlation of multiple images, which significantly reduces the cost of MOS
collection. Following the normal distribution assumption toward SOS and the
perceptual constancy constraint, we formulate the MOS estimation as a
constrained maximum likelihood estimation problem, which could be solved via
an alternative optimization strategy. Extensive experiments verify the
effectiveness of the proposed method for SOS calibration and IQA model
learning. Meanwhile, the proposed method also shows a powerful generalization
ability to the post-processing of a few opinion scores.

## References

  * [1] G. Zhai and X. Min, “Perceptual image quality assessment: a survey,” _Science China Information Sciences_ , vol. 63, no. 11, pp. 1–52, 2020.
  * [2] K. Ma, W. Liu, K. Zhang, Z. Duanmu, Z. Wang, and W. Zuo, “End-to-end blind image quality assessment using deep neural networks,” _IEEE Transactions on Image Processing_ , vol. 27, no. 3, pp. 1202–1213, 2017.
  * [3] J. Ma, J. Wu, L. Li, W. Dong, X. Xie, G. Shi, and W. Lin, “Blind image quality assessment with active inference,” _IEEE Transactions on Image Processing_ , vol. 30, pp. 3650–3663, 2021.
  * [4] L. Wang, Q. Wu, K. N. Ngan, H. Li, F. Meng, and L. Xu, “Blind tone-mapped image quality assessment and enhancement via disentangled representation learning,” in _2020 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC)_. IEEE, 2020, pp. 1096–1102.
  * [5] F. Shao, W. Lin, S. Gu, G. Jiang, and T. Srikanthan, “Perceptual full-reference quality assessment of stereoscopic images by considering binocular visual characteristics,” _IEEE Transactions on Image Processing_ , vol. 22, no. 5, pp. 1940–1953, 2013.
  * [6] X. Zhang, W. Lin, and Q. Huang, “Fine-grained image quality assessment: A revisit and further thinking,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 32, no. 5, pp. 2746–2759, 2021.
  * [7] L. Xu, J. Li, W. Lin, Y. Zhang, L. Ma, Y. Fang, and Y. Yan, “Multi-task rank learning for image quality assessment,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 27, no. 9, pp. 1833–1843, 2016.
  * [8] Q. Wu, L. Wang, K. N. Ngan, H. Li, F. Meng, and L. Xu, “Subjective and objective de-raining quality assessment towards authentic rain image,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 30, no. 11, pp. 3883–3897, 2020.
  * [9] R. C. Streijl, S. Winkler, and D. S. Hands, “Mean opinion score (mos) revisited: methods and applications, limitations and alternatives,” _Multimedia Systems_ , vol. 22, no. 2, pp. 213–227, 2016.
  * [10] T. Installations and L. Line, “Itu-tp. 910,” _Subjective video quality assessment methods for multimedia applications, Recommendation ITU-T_ , p. 910\.
  * [11] I. Union, “Methods for the subjective assessment of video quality audio quality and audiovisual quality of internet video and distribution quality television in any environment,” _SERIES P: TERMINALS AND SUBJECTIVE AND OBJECTIVE ASSESSMENT METHODS_ , 2016.
  * [12] I. BT, “Methodologies for the subjective assessment of the quality of television images, document recommendation itu-r bt. 500-14 (10/2019),” _ITU, Geneva, Switzerland_ , 2020.
  * [13] Z. Li, C. G. Bampis, L. Janowski, and I. Katsavounidis, “A simple model for subject behavior in subjective experiments,” _Electronic Imaging_ , vol. 2020, no. 11, pp. 131–1, 2020.
  * [14] P. C. Madhusudana, N. Birkbeck, Y. Wang, B. Adsumilli, and A. C. Bovik, “Image quality assessment using contrastive learning,” _IEEE Transactions on Image Processing_ , vol. 31, pp. 4149–4161, 2022.
  * [15] M. Hsu, M. Bhatt, R. Adolphs, D. Tranel, and C. F. Camerer, “Neural systems responding to degrees of uncertainty in human decision-making,” _Science_ , vol. 310, no. 5754, pp. 1680–1683, 2005.
  * [16] V. O. W. Site, “Subjective assessment of standard definition digital television (sdtv) systems,” _Rec. ITU-R BT. 1129-2_.
  * [17] Y. Fang, H. Zhu, Y. Zeng, K. Ma, and Z. Wang, “Perceptual quality assessment of smartphone photography,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 3677–3686.
  * [18] A. Zaric, N. Tatalovic, N. Brajkovic, H. Hlevnjak, M. Loncaric, E. Dumic, and S. Grgic, “Vcl@ fer image quality assessment database,” in _Proceedings ELMAR-2011_. IEEE, 2011, pp. 105–110.
  * [19] A. Charnes, E. L. Frome, and P.-L. Yu, “The equivalence of generalized least squares and maximum likelihood estimates in the exponential family,” _Journal of the American Statistical Association_ , vol. 71, no. 353, pp. 169–171, 1976.
  * [20] N. Ponomarenko, L. Jin, O. Ieremeiev, V. Lukin, K. Egiazarian, J. Astola, B. Vozel, K. Chehdi, M. Carli, F. Battisti _et al._ , “Image database tid2013: Peculiarities, results and perspectives,” _Signal processing: Image communication_ , vol. 30, pp. 57–77, 2015.
  * [21] V. Hosu, H. Lin, T. Sziranyi, and D. Saupe, “Koniq-10k: An ecologically valid database for deep learning of blind image quality assessment,” _IEEE Transactions on Image Processing_ , vol. 29, pp. 4041–4056, 2020.
  * [22] D. Ghadiyaram and A. C. Bovik, “Massive online crowdsourced study of subjective and objective picture quality,” _IEEE Transactions on Image Processing_ , vol. 25, no. 1, pp. 372–387, 2015.
  * [23] H. Talebi and P. Milanfar, “Nima: Neural image assessment,” _IEEE transactions on image processing_ , vol. 27, no. 8, pp. 3998–4011, 2018.


# WAE: Workload Automation Engine for CDN-specialized Container Orchestration

maxmin  Elif AK1, Taner OZDAS2, Serkan SEVIM2 and Berk CANBERK1  1 Department
of Computer Engineering, Istanbul Technical University, Turkey
Email<EMAIL_ADDRESS>2 MEDIANOVA, Turkey
Email: {taner.ozdas<EMAIL_ADDRESS>

###### Abstract

Content Delivery Network (CDN) has been emerged as a compelling technology to
provide efficient and scalable web services even under high client request.
However, this leads to a dilemma between minimum deployment cost and robust
service under heavy loads. To solve this problem, we propose the Workload
Automation Engine (WAE) which enables dynamic resource management, automated
scaling and rapid service deployment with least cost for CDN providers. Our
modular design uses an algorithm to calculate the optimal assignment of
virtual CDN functions such as streaming, progressive delivering and load
balancer. In particular, we study on real CDN data which belongs to Medianova
CDN Company in Turkey. Also we use Docker containerization as an underlying
system. The results reveal that our containerized design reduces the latency
and deployment cost by 45% and by 66%, respectively. Moreover, we obtain
roughly 20% more CPU efficiency and 35% more utilized network.

###### Index Terms:

Content delivery networks, containerization, network functions virtualization

##  I Introduction

Delivering massive volume of content in admissible speed has been a challenge
and Content Delivery Network (CDN) try to overcome distributing content to end
users with improving the Quality of Service (QoS) [1]. According to report of
Technavio, CDN market is expected to grow at CAGR of 30% from 2017 to 2021
[2]. However, in the nature of CDN, it builds huge number of Point of
Presences (PoPs) which are distributed geographically [3] to provide replica
content across a wide area. This brings high physical overloading and non-
flexible structure to CDN providers, also it has become difficult to
compensate and manage this increasing CDN market demand.

More specifically, there are three key issues in CDN to be addressed: (i)
stable and low latency even if under high load, (ii) balanced resource
management that decreases idle time of machines, and (iii) minimizing
deployment cost without effecting latency. Moreover, Fig. 1 shows our
observations on number of HTTP connections for 1-day period. This figure is
direct network trace taken from Vienna branch of Medianova, one of the CDN
companies in Turkey. As seen in Fig. 1, the intervals in which high load cause
excessive latency vary by day of hours. For example, in Fig. 1(c), the number
of users who requests the live streaming is in peak demand at the beginning of
the day (00.00). However, in the same interval, users who demand the large
files are relatively less as seen in Fig. 1(b). Such heterogeneous client
requests that rapidly increase the latency indicate the importance of
dynamically changeable CDN functionalities to compensate high load.

On the other hand, none of CDN functionalities work concurrently under high
resource requirements. So it leads to redundant idle time and non-effective
resource usage on the machines. For example, in Fig. 1, the client requests
which lead to excessive resource consumption make up almost 4 percentage of
the day. In other words, hardware based CDN functionalities remain idle in
most of the day.

(a) Client requests log for small edge server in 1-day period

(b) Client requests log for large edge server in 1-day period

(c) Client requests log for stream edge server in 1-day period

Figure 1: Active connection count with respect to edge servers

With these perspectives, current literature focuses on the network function
virtualization (NFV) to put aside aforementioned hardware restrictions by
giving chance to software networking. In [4], the authors study on optimal
placement of virtual CDN (vCDN) components to provide high streaming quality.
Herbaut et al. [5] focus on the virtualization for ISP-CDN collaboration.
Frangoudis et. al. [6] proposes a vCDN model over network operators. Also, [6]
and [7] provide detail comparison on Docker containers with KVM hypervisor in
terms of startup time, request throughput and response. The results show that
unlike the traditional virtual machines (VMs), lightweight containers give
better results in vCDN service. Moreover, there are many studies on container
orchestration to provide service on demand. Tosatto et. al. [8] assess the
challenges in container orchestration. In [9], genetic algorithm is proposed
for optimized container allocation. In [10], linear programing based solution
is proposed for containerized applications. Also, [11] provide detail
comparison on container orchestration tools in industry. [12] proposes an
orchestration model to improve insufficiency of Kubernetes, one of the
orchestration tools in industry. These are the specific and comprehensive
works which motivate our own work. In most of these recent studies, CDN-
specialized container orchestration has not been covered in terms of resource
provisioning. However, orchestration solution is necessary to provide steady
service for end-user and optimized resource usage for CDN providers.

Consequently, keeping these studies in mind, we propose a container based
virtualized architecture which provides resilient resource provisioning for
CDN network. Our proposed design consists of Docker containerization as the
infrastructural layout. Using the heterogeneous client requests and resource
capabilities as inputs, the orchestration module re-assign CDN roles
dynamically to meet client request with lower latency. The architecture
includes helper machines communicating with the WAE through interfaces to
supply such inputs to the orchestration module. The orchestration module
periodically calculates new arrangements. Also the proposed model enables CDN
providers to build own PoPs with least physical machines, i.e., with least
cost. Because our design aims to use underlying hardware with full performance
to minimize idle time of the machines. To the best of our knowledge, this is
the first study on the CDN-specialized container orchestration framework with
resource provisioning. The main contributions of this paper includes the
following:

  * •

Unlike the traditional CDN architecture, we design and implement virtualized
CDN network which enable PoPs to assign own network functions dynamically. By
this way, we obtain lightweight system, efficient resource usage, reduced
deployment time and auto-manageable PoPs.

  * •

Our implementation has all network functions in CDN also includes WAE that
responsible for container (VNF) assignments and resource provisioning on the
contrary other virtualized CDN designs.

  * •

WAE specializes container orchestration to the CDN thanks to the included five
modules which are: Data Collection, Normalization, Orchestration, Network
Integration and Service Discovery. So WAE manages network traffics by taking
request type, network saturation, CPU usage and existing containers in
consideration.

  * •

We collaborate with CDN company (Medianova) to evaluate results in live
environment. Our comparison results can be used in future works to deploy
complex virtualized CDN systems.

The rest of this paper is organized as follows. The proposed architecture and
CDN-specialized container orchestration are explained in sec. II. The
implementation and performance analysis are shown in sec. III and sec. IV
respectively. We conclude the paper in sec. V.

##  II The Container-based Content Delivery Network with Resource Provisioning

Following subsections firstly describe the general view of our design by
comparison with traditional architecture and then illustrates the
implementation of our system and and the Workload Automation Engine.

###  II-A Architecture

The sample traditional PoP which we work on contains one DNS server, one load
balancer, four distinct edge servers and one mid-cache with an origin server
as seen in Fig. 2(a). CDN edge servers also known as cache servers provide
different service in terms of request type and we group the edge servers in
four categories which are: (i) Small Edge: serves small files
($\lessapprox$1MB, e.g., picture), (ii) Large Edge: serves large data
($\gtrapprox$1MB, e.g., pdf document), (iii) VoD Edge: responsible for VoD
streaming and (iv) Live Edge: responsible for live streaming. This grouping is
also preserved in the containerized architecture.

Our virtualized network architecture (see Fig. 2(b)) includes all CDN roles in
the form of VNFs on the underlying Docker containerization. Also it contains
WAE and instance managers (IM) to orchestrate a part of VNFs with
communicating over interfaces. Thus our proposed model overcome intense
requests with flexible manner also with less physical machine in total. For
example, Fig. 2(a) contains only one edge server for small requests; however,
in our design in Fig. 2(b) client requests can be meet with up to 3 virtual
edge server in heavy load. Also our design focus on the orchestration of only
edge servers which are directly related to QoS. Detailed system design and the
algorithm is explained in below subsections.

(a) before: Traditional Content Delivery Network architecture

(b) after: Container-based virtualized Content Delivery Network

Figure 2: Architecture Comparison

###  II-B System Model

####  II-B1 Instance Manager (IM)

Each machine in the virtualized PoPs includes Instance Manager which provides
southbound interface through Data Collection API (DC API). Each IM in the PoP
sends CPU capacity and output network traffic of the host machine to WAE. Also
IMs supply a vector for number of request coming to edge servers. Here, number
of request means that four edge servers collect the coming request within
predefined period (for example ten minutes) then IMs get the total summation
from each different edge servers to produce a vector for total number of
request with respect to request type (small requests, large requests,
streaming and live requests). Fig. 3 shows the interfaces and supplied inputs
by IM.

Figure 3: System Model

####  II-B2 Workload Automation Engine (WAE)

We implement WAE (Fig. 3) which manages containers with respect to CPU usages
of host machines, outgoing network traffic, request URL types and existing
container; then produces an output to send new VNF assignments to Instance
Manager APIs. Thereby, CDN base specialized dynamic scaling improves existing
virtualization solutions by re-arranging PoP according to client demand.

Workload automation engine contains five modules which are: Data Collection,
Normalization, Orchestration, Network Integration and Service Discovery. The
system design of five modules are explained in the following, respectively.

##### Data Collection Module

Data Collection RESTful API listen IM interfaces, which is placed in different
machines, and aggregates input data for resource provisioning algorithm.
Output of DC Module includes three vector which are C, T and R; CPU capacity
vector, output link network traffic, number of requests with request types,
respectively.

##### Normalization Module

Output of the DC Module will be input for Normalization Module. The vectors C
and T are rescaled to range of $\mathnormal{[0,1]}$ for transforming data from
unit based to percentage base form. Likewise, the vector R is an input for
function $f(x_{i})=\frac{x_{i}}{\sum\limits_{j=1}^{i}x_{i}}$ to obtain
weighted distribution of requests. As a result, we obtain pure input for
resource provisioning algorithm.

##### Container Orchestration Module

The module includes resource provisioning algorithm to orchestrate containers
on client demand. The basic heuristic behind this algorithm is to dynamically
scale load for heterogeneous client requests in consideration of existing
resources. Meeting client requests with minimized set of containers (1) lead
to formulating the following optimization. Constraint (2) ensures that
resource distribution is kept inside threshold ($\pm$ 0.1), and (3) guarantees
that each physical machine has at most 1 virtual function for each type. We
indicate the mathematical notations of network parameters and system arguments
in Table I.

TABLE I: Mathematical Notations Parameter | Definition
---|---
N | number of physical machines
M |  | number of different network functions to
---
orchestrate
Z | total number of containers
ANxM |  | 2-dimensional container distribution matrix
---
(input)
BNxM |  | 2-dimensional container distribution matrix
---
(output)
R = {r1, r2, r3, r4} | request vector
C = {c1, c2, …, cN} | cpu capacity vector
T = {t1, t2, …, tN} | output link network traffic vector

$\displaystyle\underset{Z}{\text{minimize}}$ $\displaystyle\sum\limits_{i\in
N}Z\textsubscript{i}$ (1) subject to $\displaystyle X_{i}^{N}-0.1\leq
R_{i}^{N},\leq X_{i}^{N}+0.1,\quad\forall i\in\mathbb{N}$ (2) $\displaystyle
A_{ij}=1\quad\vee\quad A_{ij}=0,\quad\forall i,j\in\mathbb{N}$ (3)

where X is summation of vectors C and T multiplied by matrix A, that is
$\mathnormal{X=(C+T)^{\intercal}\cdot A}$. On the other hand, superscript
$\mathnormal{N}$ represents the normalized vector mentioned in Normalization
Module.

Algorithm 1 CDN-specialized Container Orchestration

0: average CPU capacity vector $\mathnormal{C}$, network output traffic
$\mathnormal{T}$, number of request distribution for each request type
$\mathnormal{R^{N}}$, and container distribution matrix $\mathnormal{A_{NxM}}$

0: new container distribution matrix $\mathnormal{B_{NxM}}$

1: $\mathnormal{V_{sum}}\leftarrow\mathnormal{C}+\mathnormal{T}$

2: $\mathnormal{D}\leftarrow\mathnormal{V_{sum}}\cdot\mathnormal{A}$

3: $\mathnormal{D^{N}}\leftarrow\textsf{{Normalize}}(D)$

4: while $\mathnormal{A_{NxM}}$ is changed do

5: for $i=1$ to $length$ $of$ $D^{N}$ do

6: if $R_{i}^{N}$ \- 0,1 >$D_{i}^{N}$ then

7:
$j\leftarrow\textsf{{FindMinLoadedMachine}}(\mathnormal{A_{NxM}},\mathnormal{V_{sum}})$

8: $A_{ji}\leftarrow 1$

9: else if $R_{i}^{N}$ \+ 0,1 <$D_{i}^{N}$ then

10:
$j\leftarrow\textsf{{FindMaxLoadedMachine}}(\mathnormal{A_{NxM}},\mathnormal{V_{sum}})$

11: $A_{ji}\leftarrow 0$

12: end if

13: end for

14: $\mathnormal{D}\leftarrow\mathnormal{V_{sum}}\cdot\mathnormal{A}$

15: $\mathnormal{D^{N}}\leftarrow\textsf{{Normalize}}(D)$

16: end while

Algorithm 1 shows the algorithmic representation of the optimization problem
(1) to find minimum number of containers in the system to serve client request
under the resource capacity: CPU and Network Link. In this perspective, our
algorithm operates as follow: two resource capacities, CPU and network output
traffic, are equally important for this case so that both of them are sum up
(line 1) to obtain total resource capacity. Then, the algorithm calculates
resource distribution over containers (line 2) to get resource distribution
over virtual edge servers. Moreover, remaining part of algorithm try to find
which type of virtual edge server require expansion or which ones should be
shrinking. For this, each density distribution of resources $D^{N}$ is
compared with the density distribution of client requests $R^{N}$ with respect
to threshold $0.1$ (line 6 and 9). According to condition, the algorithm
decide a physical machine to create or drop the container with respect to
resource usage (minimum loaded or maximum loaded) (line 7 and 10). The
algorithm calculates new density distribution of resources $D^{N}$ with
recently calculated $A$ at the end of each iteration also line 4 guarantees
that the algorithm is terminated.

##### Network Integration Module

In our design, we want to configure VNFs with public IPs so that it can be
reachable remotely. Also it will be well-suited for traditional CDN routing
principles. Network Integration Module provide available network
configurations list for new deployed containers and manages network
configurations of dropped containers. Network Integration Module get dropped
containers list from Service Discovery Module and send available IP list to
Service Discovery Module.

##### Service Discovery Module (SD API)

Each machine has all type of containers with status of paused or running which
is controlled by Service Discovery Module. Service Discovery is northbound
interface to integrate underlying physical infrastructure with virtual network
functions. SD API supply list of container assignment object according to
decision of orchestration algorithm and network configurations provided by
Network Integration Module. Container assignment objects for each machine is
send to IMs.

##  III Implementation

As a proof of concept, we deploy three host machine with 8-core Intel©
processor, 16 GB of RAM, and running CentOS© 7.4. Each host machines are
configured to include VNF instances as Docker container with the version of
17.09. We are using nginx to serve HTTP connections in the virtual edge
servers, with version of nginx 1.12. As we mentioned in our system design, VNF
instances are accessible publicly so that we use macvlan network driver to use
CDN edge servers as capable of direct routing. The macvlan configuration we
used is shown below.

⬇

docker network create -d macvlan \--subnet=

<desired subnet> \--gateway=<gateway> -o

parent=eth0 mn-container-network

Moreover, we create Dockerfile for CDN roles which means we write discretely
five Dockerfile on the private Docker Hub. Also Algorithm 1 is implemented and
run in every 10 minutes periods to re-orchestrate containers. We implement
containerized CDN network as we described in 2(b). Each machines include VNFs
of CDN roles, for example Fig. 4 shows Docker instances of the master machine.

Figure 4: System Screen from Proposed Network Architecture

##  IV Performance Evaluation

We compare our proposed design with traditional CDN PoP, which includes bare-
metal servers for each CDN functionalities, to demonstrate performance
results. Moreover, we indicate the importance of orchestration by comparing
with the virtualized topology which is not include our orchestration module.

We choose 1-hour client requests from the observations on the topology in
Vienna. Then, we generate the same requests with Jmeter, one of the open
source performance test tool written by Java. As a result, our scenario
contains:

  * •

Image Request 5000 users in total with 5 second ramp up time, which means that
it start with 1000 users and it reach 5000 users in 5 seconds.

  * •

Large Content Request 1500 users in total with 3 second ramp up time

  * •

Stream Request 500 users in total with 5 second ramp up time

We observe that the latency of the proposed heuristic is roughly 45% less than
traditional bare-metal CDNs. In addition, as seen in Fig. 5, the proposed
model makes difference with orchestration module compared with containerized
CDN which is not include resource provisioning heuristic.

Figure 5: Latency Comparison TABLE II: Average Resource Usage in Traditional CDN CDN Role | CPU Usage |  | Network Output
---
Link Usage
small edge server | 7% | 0.2%
large edge server | 5% | 12%
stream edge server | 2.5% | 3%
load balancer | 1% | 2%
TABLE III: Average Resource Usage in Container-based CDN Host Machines | CPU Usage |  | Network Output
---
Link Usage
physical machine 1 | 32% | 48%
physical machine 2 | 28% | 42%
physical machine 3 | 25% | 51%

As seen in Table II and Table III , we obtain approximately 20% more CPU
efficiency and 35% more utilized network in proposed model. In other words, we
reach less latency with less resource capacity thanks to resilient resource
provisioning heuristic. On the other hand, as seen in Table III, the resources
of our proposed CDN architecture does not saturate so it can handle with more
heavy load. Graphical representation is placed in Fig. 6(b).

Moreover, we reduce the deployment cost of the PoP by three times compared to
traditional architecture as seen in Fig. 6(a)

(a) Cost Comparison

(b) Resource Usage Comparison

Figure 6: Cost and Resource Usage Comparison

As seen in the results, WAE enhance existing containerized CDN using
orchestration module and produce steady and decreased latency. Because WAE
uses resources and existing containers as inputs, also takes client trends in
consideration (not in other orchestration solutions). Thereby, CDN-specialized
orchestration uses less resources with more effective usage. This leads to
three times less service deployment.

##  V Conclusion

In this paper, we proposed a resilient resource provisioning heuristic for
virtualized CDN to provide steady service for end-users. In our approach,
workload automation engine intelligently performs VNF assignments to achieve
minimum latency with least deployment cost. Also, we collaborated with
Medianova to observe client requests in live environment, then to test our
proposed design. According to results, our design reduces the latency by 45%
with 66% percentage decreased deployment cost compared to traditional CDNs.
Also we optimized the CPU and Network up to 20% and 35% percentage,
respectively. In future work, we concentrate on orchestration framework for
large scale CDN topologies to deploy dynamically changeable CDN PoPs, which
are interconnected each other.

## References

  * [1] A. Pathan and R. Buyya, “A Taxonomy and Survey of Content Delivery Networks,” Technical Report, GRIDS-TR-2007-4, Grid Computing and Distributed Systems Laboratory, The University of Melbourne, Australia., Feb. 2007.
  * [2] Technavio.com. (2018). Global Content Delivery Network (CDN) Market 2017-2021, Market Research Reports - Industry Analysis Size & Trends-Technavio. [online] Available at: https://www.technavio.com/report/global-content-delivery-network-cdn-market [Accessed 4 Mar. 2018].
  * [3] A. Vakali and G. Pallis, ”Content delivery networks: status and trends,” in IEEE Internet Computing, vol. 7, no. 6, pp. 68-74, Nov.-Dec. 2003.
  * [4] H. Khedher, E. Abd-Elrahman, H. Afifi and M. Marot, ”Optimal and Cost Efficient Algorithm for Virtual CDN Orchestration,” 2017 IEEE 42nd Conference on Local Computer Networks (LCN), Singapore, 2017, pp. 61-69.
  * [5] N. Herbaut, D. Negru, Y. Chen, P. A. Frangoudis and A. Ksentini, ”Content Delivery Networks as a Virtual Network Function: A Win-Win ISP-CDN Collaboration,” 2016 IEEE Global Communications Conference (GLOBECOM), Washington, DC, 2016, pp. 1-6.
  * [6] P. A. Frangoudis, L. Yala and A. Ksentini, ”CDN-As-a-Service Provision Over a Telecom Operator’s Cloud,” in IEEE Transactions on Network and Service Management, vol. 14, no. 3, pp. 702-716, Sept. 2017.
  * [7] R. Dua, A. R. Raja and D. Kakadia, ”Virtualization vs Containerization to Support PaaS,” 2014 IEEE International Conference on Cloud Engineering (IC2E), Boston, MA, 2014, pp. 610-614.
  * [8] A. Tosatto, P. Ruiu and A. Attanasio, ”Container-Based Orchestration in Cloud: State of the Art and Challenges,” 2015 Ninth International Conference on Complex, Intelligent, and Software Intensive Systems, Blumenau, 2015, pp. 70-75.
  * [9] C. Guerrero, I. Lera and C. Juiz, ”Genetic algorithm for multi-objective optimization of container allocation in cloud architecture”, 2018 Journal of Grid Computing, 16(1), 113-135.
  * [10] D. Zhang, B. H. Yan, Z. Feng, C. Zhang and Y. X. Wang, ”Container oriented job scheduling using linear programming model,” 2017 3rd International Conference on Information Management (ICIM), Chengdu, 2017, pp. 174-180.
  * [11] P. Heidari, Y. Lemieux and A. Shami, ”QoS Assurance with Light Virtualization - A Survey,” 2016 IEEE International Conference on Cloud Computing Technology and Science (CloudCom), Luxembourg City, 2016, pp. 558-563.
  * [12] C. C. Chang, S. R. Yang, E. H. Yeh, P. Lin and J. Y. Jeng, ”A Kubernetes-Based Monitoring Platform for Dynamic Cloud Resource Provisioning,” GLOBECOM 2017 - 2017 IEEE Global Communications Conference, Singapore, 2017, pp. 1-6.


* [29] Ken-ichi Kawarabayashi and Yusuke Kobayashi.  Linear min-max relation between the treewidth of an $H$-minor-free graph and its largest grid minor.  J. Combin. Theory Ser. B, 141:165–180, 2020.  doi:10.1016/j.jctb.2019.07.007.
  * [30] Ken-ichi Kawarabayashi and Bruce Reed.  A separator theorem in minor-closed classes.  In 2010 IEEE 51st Annual Symposium on Foundations of Computer Science—FOCS 2010, pages 153–162. IEEE Computer Soc., Los Alamitos, CA, 2010.
  * [31] Ken-ichi Kawarabayashi, Robin Thomas, and Paul Wollan.  A new proof of the flat wall theorem.  J. Combin. Theory Ser. B, 129:204–238, 2018.  doi:10.1016/j.jctb.2017.09.006.
  * [32] Ken-ichi Kawarabayashi, Robin Thomas, and Paul Wollan.  Quickly excluding a non-planar graph.  CoRR, abs/2010.12397, 2020.  URL: https://arxiv.org/abs/2010.12397, arXiv:2010.12397.
  * [33] Subhash Khot and Nisheeth K Vishnoi.  On the unique games conjecture.  In FOCS, volume 5, page 3, 2005.
  * [34] Tuukka Korhonen.  A single-exponential time 2-approximation algorithm for treewidth.  In IEEE Annual Symposium on Foundations of Computer Science (FOCS), pages 184–192. IEEE, 2022.  doi:10.1109/FOCS52979.2021.00026.
  * [35] Christophe Paul, Evangelos Protopapas, and Dimitrios M Thilikos.  Graph Parameters, Universal Obstructions, and WQO.  CoRR, abs/2304.03688, 2023.  URL: https://arxiv.org/abs/2304.03688, arXiv:2304.03688.
  * [36] Christophe Paul, Evangelos Protopapas, and Dimitrios M Thilikos.  Universal Obstructions of Graph Parameters.  CoRR, abs/2304.14121, 2023.  URL: https://arxiv.org/abs/2304.14121, arXiv:2304.14121.
  * [37] Prasad Raghavendra and David Steurer.  Graph expansion and the unique games conjecture.  In STOC’10—Proceedings of the 2010 ACM International Symposium on Theory of Computing, pages 755–764. ACM, New York, 2010.
  * [38] Bruce A Reed.  Finding approximate separators and computing tree width quickly.  In ACM Symposium on Theory of Computing (STOC), pages 221–228, 1992\.  doi:10.1145/129712.129734.
  * [39] Neil Robertson and P. D. Seymour.  Graph minors. V. Excluding a planar graph.  J. Combin. Theory Ser. B, 41(1):92–114, 1986.  doi:10.1016/0095-8956(86)90030-4.
  * [40] Neil Robertson and P. D. Seymour.  Graph minors. IX. Disjoint crossed paths.  J. Combin. Theory Ser. B, 49(1):40–77, 1990.  doi:10.1016/0095-8956(90)90063-6.
  * [41] Neil Robertson and P. D. Seymour.  Graph minors. X. Obstructions to tree-decomposition.  J. Combin. Theory Ser. B, 52(2):153–190, 1991.  doi:10.1016/0095-8956(91)90061-N.
  * [42] Neil Robertson and P. D. Seymour.  Graph minors. XVI. Excluding a non-planar graph.  J. Combin. Theory Ser. B, 89(1):43–76, 2003.  doi:10.1016/S0095-8956(03)00042-X.
  * [43] Neil Robertson and Paul D. Seymour.  Graph minors. XI. circuits on a surface.  J. Comb. Theory, Ser. B, 60(1):72–106, 1994.  doi:10.1006/jctb.1994.1007.
  * [44] P. D. Seymour.  Disjoint paths in graphs.  Discrete Math., 29(3):293–309, 1980.  doi:10.1016/0012-365X(80)90158-2.
  * [45] P. D. Seymour and R. Thomas.  Call routing and the ratcatcher.  Combinatorica, 14(2):217–241, 1994.  doi:10.1007/BF01215352.
  * [46] Yossi Shiloach.  A polynomial solution to the undirected two paths problem.  J. Assoc. Comput. Mach., 27(3):445–456, 1980.  doi:10.1145/322203.322207.
  * [47] Dimitrios M. Thilikos and Sebastian Wiederrecht.  Killing a vortex.  In 2022 IEEE 63rd Annual Symposium on Foundations of Computer Science—FOCS 2022, pages 1069–1080. IEEE Computer Soc., Los Alamitos, CA, [2022] ©2022.
  * [48] Dimitrios M. Thilikos and Sebastian Wiederrecht.  Approximating branchwidth on parametric extensions of planarity.  CoRR, abs/2304.04517, 2023.  URL: https://arxiv.org/abs/2304.04517, arXiv:2304.04517.
  * [49] Carsten Thomassen.  $2$-linked graphs.  European J. Combin., 1(4):371–378, 1980.  doi:10.1016/S0195-6698(80)80039-4.
  * [50] Yu Wu, Per Austrin, Toniann Pitassi, and David Liu.  Inapproximability of treewidth, one-shot pebbling, and related layout problems.  J. Artificial Intelligence Res., 49:569–600, 2014.  doi:10.1613/jair.4030.