[table]capposition=above capbtabboxtable[][] plaintop table 11institutetext:
TU Delft
11email<EMAIL_ADDRESS>
# Weakly Supervised Label Smoothing

Gustavo Penha  Claudia Hauff

###### Abstract

We study Label Smoothing (LS), a widely used regularization technique, in the
context of neural learning to rank (L2R) models. LS combines the ground-truth
labels with a uniform distribution, encouraging the model to be less confident
in its predictions. We analyze the relationship between the non-relevant
documents—specifically how they are sampled—and the effectiveness of LS,
discussing how LS can be capturing “hidden similarity knowledge” between the
relevant and non-relevant document classes. We further analyze LS by testing
if a curriculum-learning approach, i.e., starting with LS and after a number
of iterations using only ground-truth labels, is beneficial. Inspired by our
investigation of LS in the context of neural L2R models, we propose a novel
technique called _Weakly Supervised Label Smoothing_ (WSLS) that takes
advantage of the retrieval scores of the negative sampled documents as a weak
supervision signal in the process of modifying the ground-truth labels. WSLS
is simple to implement, requiring no modification to the neural ranker
architecture. Our experiments across three retrieval tasks—passage retrieval,
similar question retrieval and conversation response ranking—show that WSLS
for pointwise BERT-based rankers leads to consistent effectiveness gains. The
source code is available at
https://anonymous.4open.science/r/dac85d48-6f71-4261-a7d8-040da6021c52/.

##  1 Introduction

Neural Learning to Rank (L2R) models are traditionally trained using large
amounts of strongly labeled data, i.e., human generated relevance judgements.
For example, in ad hoc retrieval each instance is comprised of a query, a
document and a relevance judgment. All the other documents in the collection
that were not labeled as (non-)relevant for the query, while not specified
explicitly, can be viewed as non-relevant for the query. Since utilizing an
entire corpus for training a L2R model is practically infeasible, the typical
procedure is to rely on the top-$k$ ranked documents for a query obtained from
an efficient (but less effective) retrieval model such as BM25. While research
has shown that the negative sampler (NS), i.e. the technique to select
documents to use as negative samples for a query, matters a great deal in the
effectiveness of the learned ranker [1, 14, 10, 23, 2] there has been no work
on how to make use of the _scores_ of the NS, which are currently ignored in
the training of L2R models—only the content of the documents are employed.

In this work we first aim to understand, in the realm of neural L2R111Binary
relevance prediction is quite different from other domains such as image
classification and language modelling which employ up to thousands of distinct
classes., a widely used and successful [28, 27, 22] regularization technique
called Label Smoothing [21] (LS), that penalizes the divergence between the
predictions and a uniform distribution. We begin by looking into how the
choice of NS impacts LS, since in the binary relevance prediction problem LS
penalizes the model less than normal training when predicting a negative
document as relevant and vice versa. We also analyze whether it is beneficial
to use a curriculum-learning inspired procedure for the hyper-parameter that
controls the LS strength as shown by recent work on understanding LS in other
domains [4, 24]. This initial exploration to understand LS leads to the
following research question: RQ1 Is label smoothing an effective regularizer
for neural L2R (and if so, under what conditions)? Our experimental results on
three different retrieval tasks reveal that LS is indeed an effective
regularization technique for neural L2R, specifically when (a) there is
similarity between the relevant and the non-relevant sampled documents, i.e.
when we use BM25 as the NS technique, and (b) a curriculum-like approach is
used to control the strength of the smoothing.

Inspired by our findings, we propose the Weakly Supervised Label Smoothing
(WSLS) technique which exploits the NS retrieval scores, as opposed to LS
where all labels are smoothed equally, for training neural L2R models. Instead
of interpolating the ground-truth label distribution with a uniform
distribution (as done in LS), we interpolate it with the NS score
distribution. WSLS has two benefits compared to using the ground-truth labels:
(a) it regularizes the neural ranker by penalizing overconfident predictions
and (b) it provides additional supervision signal through weak supervision [3]
for the negative sampled documents. WSLS is simple to implement, and requires
no modification to the neural ranker architecture, but only to the labels
using weak supervision scores that are readily available. Our experiments to
answer our second research question (RQ2 Is WSLS more effective than LS for
training neural L2R models?) reveal that WSLS is a better way of smoothing the
labels by providing additional weak supervision obtained from the negative
sampling procedure. We reach relative gains of 0.5% in effectiveness across
tasks.

##  2 Background: Label Smoothing (LS)

Given an input instance $x$ (a query and document combination), two classes
($k=0$ means not relevant and $k=1$ relevant, and thus here $K=2$), a ground
truth distribution $q(k\mid x)$ and predictions from the neural L2R model
$p(k\mid
x)=\frac{\exp\left(z_{k}\right)}{\sum_{i=1}^{K}\exp\left(z_{i}\right)}$, where
$z_{i}$ are the logits, we can use the cross entropy loss for training:
$\ell=-\sum_{k=0}^{K}\log(p(k))q(k)$, where $q(k)=\delta_{k,y},$ and
$\delta_{k,y}$ is Dirac delta (equals 1 for $z=y$ and 0 otherwise). Maximizing
the log-likelihood of the correct label is approached if the logit
corresponding to the ground-truth label is much greater than all other logits:
$z_{y}\gg z_{k}$ for all $k\neq y$. This encourages the model to be
overconfident in its predictions, which might not generalize well. Label
smoothing [21] is a regularization mechanism to encourage the model to be less
confident. Given a distribution $u(k)$, _independent_ of the training example
$x,$ and a smoothing parameter $\epsilon$, for a training example with ground-
truth label $y,$ we replace the label distribution $q(k\mid x)=\delta_{k,y}$
with $q^{\prime}(k\mid x)=(1-\epsilon)\delta_{k,y}+\epsilon u(k)$. In LS the
uniform distribution is employed, i.e. $u(k)=1/K$.

While LS is a widely used technique to regularize models, the reasons
underlying its successes [28, 27, 22] and failures [12, 19] remain unclear.
Müller et. al [17] showed that while LS impairs teacher models to do knowledge
distillation222Knowledge distillation [8] is the process of using the
predictions of a teacher model with higher complexity/size as the ground-truth
distribution for a weaker model. it improves the models’ calibration, i.e. how
representative the predictions are with respect to the true likelihood of
correctness [7].

###  2.0.1 Curriculum Learning for Label Smoothing (T-LS)

Xu et. al [24] argued that given the empirical evidence of LS ineffectiveness
in certain cases, it is natural to combine LS with the ground-truth labels
during training in a two-stage training procedure and thus proposed T-LS:
starts training with LS, i.e. $\epsilon>0$, and after $X$ training instances
use normal training, i.e. $\epsilon=0$ (the unmodified ground-truth labels are
used). Similarly, Dogan et. al [4] proposed to move from a distribution of
labels smoothed by the similarity between label classes towards the ground-
truth labels with a curriculum leaning procedure. In this paper we resort to
T-LS333Initial experiments where we decreased $\epsilon$ linearly [4] were as
effective as T-LS [24]. [24] to test whether a curriculum learning inspired
approach for $\epsilon$ is required or not in the training of neural L2R
models.

##  3 Weakly Supervised Label Smoothing (WSLS)

We propose to replace the uniform distribution $u(k)$ that is independent of
the example $x$, with a weakly supervised function $w(k\mid x)$, which is
readily available for documents with label $0$ as part of the negative
sampling procedure of L2R, at no additional cost: the negative sampler (NS)
score444When building the training col. of triplets for DL TREC 2019/20, for
each query there is 1 relevant passage; the non-relevant passages are
retrieved using BM25.. Specifically, $q^{\prime}(k\mid
x)=(1-\epsilon)\delta_{k,y}+\epsilon NS(k\mid x)$, where $NS(k\mid x)$ is the
negative sampling procedure score for instance $x$ and label class $k$. If we
use $BM25{}$ to retrieve negative samples555Since the BM25 scores are not
between 0 and 1 we apply min-max scaling., then for $k=0$ we have
$q^{\prime}(k\mid x)=(1-\epsilon)\delta_{k,y}+\epsilon BM25{}(x)$ and when
$k=1$ we fall back to LS since we have strong labeled data: $q^{\prime}(k\mid
x)=(1-\epsilon)\delta_{k,y}+\epsilon\frac{1}{K}$. In the same way we can
induce a curriculum learning procedure for LS resulting in T-LS (see §2), we
can do it for WSLS, for which we refer to as T-WSLS666This bears resemblance
to strategies for combining strong and weak labeled data [20]..

##  4 Experimental Setup

Tasks & Datasets: In order to evaluate our research questions, we resort to
the three following retrieval tasks: passage retrieval using the 2020 Deep
Learning track of TREC (TREC-DL) dataset777Since the official test split is
not available we split the dev. set into two., similar question retrieval with
the Quora Question Pairs [9] (QQP) dataset and conversation response ranking
with the MANtIS [18] dataset. We use them due to the large amount of labeled
examples (required for training neural ranking models) and diversity of tasks.

Implementation details & Evaluation: We use BERT-based ranking as a strong
neural L2R baseline. We follow previous research [15] and fine-tune BERT using
the [CLS] token to predict binary relevance—the query and the document are
concatenated using the [SEP] token and used as input—using the cross-entropy
loss and Adam optimizer [11] with $lr=5^{-6}$ and $\epsilon=1^{-8}$. We train
with a batch size of $32$ and fine-tune the models for 50000 training
instances.We train and test each model 5 times using different random seeds
with 10 total candidate documents by query888So for example in the passage
retrieval task there are 10 passages and only one is relevant for each query
and for similar question retrieval there are 10 questions.. We resort to a
standard evaluation metric in conversation response ranking [25, 6]: recall at
position $K$ with $n$ candidates999For example $R_{10}@1$ indicates the number
of relevant documents found at the first position when the model has to rank
10 candidates.: $R_{n}@K$. Since all tasks here are concerned with re-ranking
$R_{n}@K$ is a suitable sampled metric101010A sampled metric uses a sample of
documents instead of the whole collection [13]. to compare models on how high
the relevant documents are ranked when having only $n$ candidates. We resort
to a robust and widely used NS to obtain such candidates: BM25. We refer to
using the query as input to BM25 and selecting the top $n-1$ ranked documents
as $NS\textsubscript{BM25}$. We also use random sampling
($NS\textsubscript{random}$)—which samples candidate documents from the whole
collection with the same probability and thus brings documents that are quite
different from the relevant one—to better understand LS.

Table 1: Average $R_{10}@1$ and the standard deviation results of 5 runs with different random seeds for BERT with label smoothing (w. LS) and BERT with two-stage label smoothing (w. T-LS) for different negative samplers during training ($NS\textsubscript{BM25}$ and $NS\textsubscript{random}$) and $\epsilon=0.2$ for the development set. Bold indicate the highest values for each dataset and $\blacktriangleup{}$/$\blacktriangledown{}$ superscripts indicate significant gains and losses respectively over the baseline (BERT) using paired Student’s t-test with confidence level of 0.95. | $NS\textsubscript{BM25}$ | $NS\textsubscript{random}$
---|---|---
| TREC-DL | QQP | MANtIS | TREC-DL | QQP | MANtIS
BERT |  0.568$\pm$.00 |  0.581$\pm$.03 |  0.612$\pm$.01 |  0.385$\pm$.01 |  0.444$\pm$.01 |  0.350$\pm$.01
w. LS |  0.564$\pm$.01▾ |  0.593$\pm$.01▴ |  0.612$\pm$.01 |  0.304$\pm$.05▾ |  0.440$\pm$.03▾ |  0.348$\pm$.01▾
w. T-LS |  0.570$\pm$.01▴ |  0.598$\pm$.01▴ |  0.612$\pm$.01 |  0.382$\pm$.02▾ |  0.444$\pm$.01 |  0.345$\pm$.01▾

##  5 Results

###  5.0.1 Effectiveness of Label Smoothing for Neural Ranking (RQ1)

Table 1 displays the dev. set results111111Since we do not do any hyper-
parameter tuning for RQ1, we resort to the dev. set to avoid overusing the
test set. for the LS and T-LS techniques when changing the NS. The results
reveal that when training BERT with $NS\textsubscript{random}$ to sample
negative documents, it is not effective to use any type of label smoothing. In
fact there is a consistent and statistically significant decrease in the
effectiveness compared to BERT. In contrast, when we sample documents to train
with $NS\textsubscript{BM25}$ we observe that there are significant gains to
train BERT with T-LS, with the exception of MANtIS where there is no
statistical difference. When we compare LS with T-LS, we see that it is indeed
beneficial to use a curriculum-learning approach for label smoothing (T-LS),
which indicates that being more permissive of the mistakes in the first half
of training is effective—this is in line with results obtained in other
domains [4, 24]. This answers our first RQ positively: label smoothing is an
effective regularization technique to train neural L2R models, with gains of
1% of $R_{10}@1$ compared to standard training (BERT) on average across three
different retrieval tasks when (a) using $NS\textsubscript{BM25}$ and (b) a
curriculum learning approach for LS.

We hypothesize that label smoothing is effective for training neural L2R
models if the negative documents are similar to the relevant documents for the
query. Our results when changing from $NS\textsubscript{BM25}$ to
$NS\textsubscript{random}$ support this hypothesis. Intuitively, if the
negative document is random and thus very dissimilar to the query, using a
label smoothing regularizer will penalize the model less for this mistake,
which might hinder learning. When using label smoothing with a negative
document that was sampled using BM25, we are penalizing the model less for
choosing a document that is similar to the query in terms of exact matching
words. In this way we are teaching the model the similarity between the
classes relevant and non-relevant121212A similar reasoning can be found in
recent work which discusses that the similarity between classes on the wrong
responses, i.e. “hidden similarity knowledge” [8], is helpful for learning
better neural networks [26, 4, 5]. by means of documents that are closer to
the classification frontier. Our findings align with [16]: training with
topically similar (but non-relevant) documents—as opposed to random
documents—allows the model to better discriminate between documents provided
by an earlier retrieval stage.

###  5.0.2 Effectiveness of Weakly Supervised Label Smoothing (RQ2)

[.40] [.55]  TREC-DL QQP MANtIS BERT 0.599$\pm$.00 0.595$\pm$.01 0.609$\pm$.01
w. T-LS 0.601$\pm$.00▴ 0.596$\pm$.01 0.607$\pm$.01 w. T-WSLS 0.604$\pm$.00▴▵
0.598$\pm$.01▴▵ 0.609$\pm$.01▵

Figure 1: Stacked and smoothed weak supervision distributions used for WSLS
from the min-max normalized scores of $NS\textsubscript{BM25}$. The dashed
vertical line indicates the distribution used by LS (uniform with $K=2$).
Figure 2: Average $R_{10}@1$ and the standard deviation results of 5 runs with
different random seeds for the test set.
$\blacktriangleup{}$/$\blacktriangledown{}$ and
$\smalltriangleup{}$/$\smalltriangledown{}$ superscripts indicate significant
gains and losses over the baselines (BERT) and (BERT w. T-LS) respectively
using paired Student’s t-test with confidence level of 0.95 and Bonferroni
correction.

Before we dive into the effectiveness of T-WSLS131313Based on RQ1 results we
use the two-stage approaches here (T-LS and T-WSLS). (RQ2), we investigate the
distribution of the normalized weak supervision scores from
$NS\textsubscript{BM25}$ in Figure 2. There is a high density for low scores
indicating that only a few of the sampled documents receive scores close to
the maximum of the list (0.99 score after min-max scaling) and most of them
are closer to the minimum (0.00). This is very different from the uniform
distribution used by T-LS (dashed vertical line), which does not change
according to the sample, and with two classes ($K=2$) is equal to 0.5, whereas
the mean of the weak supervision distribution is 0.33. This suggests that the
optimal $\epsilon$ for T-WSLS is different from T-LS.

Based on this observation, we test different values of $\epsilon$ on the dev.
set in order to tune this hyper-parameter and use it on the test set. Figure 3
displays the effect of $\epsilon$ on the effectiveness of the proposed
approach. The highest $R_{10}@1$ values are observed for T-WSLS: 0.574 ($+1\%$
over the baseline w/o T-WSLS) for TREC-DL when $\epsilon=0.4$ , 0.600
($+3.2\%$) for QQP when $\epsilon=0.2$ and 0.6151 ($+0.5\%$) for MANtIS when
$\epsilon=0.4$. When we apply the best models (for both T-LS and T-WSLS) found
using the dev. set on the test set, we see in Table 2 that BERT w. T-WSLS
outperforms both BERT and BERT w. LS with statistical significance (with the
exception of MANtIS where there is no difference). This answers RQ2 indicating
that WSLS is indeed more effective than LS with statistically significant
gains on all tasks against T-LS and with an average of 0.5% improvement over
BERT.

Figure 3: T-LS and T-WSLS sensitivity to the hyperparameter $\epsilon$ for the
dev. set. Error bars indicate the 95% confidence intervals for $R_{10}@1$ over
5 runs with different random seeds. Dashed horizontal lines indicate the
baseline w/o label smoothing ($\epsilon=0$).

##  6 Conclusion

We studied LS in the context of neural L2R models. Our findings indicate that
LS is effective when there is similarity between relevant and non-relevant
documents and that using curriculum learning for the strength of the
regularization is effective. We proposed a technique that combines the weak
supervision scores of negative sampled documents with label smoothing (WSLS)
which outperforms LS on different retrieval tasks. In future work we will
explore WSLS in a wider range of retrieval models and tasks.

## References

  * [1] Aslam, J.A., Kanoulas, E., Pavlu, V., Savev, S., Yilmaz, E.: Document selection methodologies for efficient and effective learning-to-rank. In: Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval. pp. 468–475 (2009)
  * [2] Cohen, D., Jordan, S.M., Croft, W.B.: Learning a better negative sampling policy with deep neural networks for search. In: Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval. pp. 19–26 (2019)
  * [3] Dehghani, M., Zamani, H., Severyn, A., Kamps, J., Croft, W.B.: Neural ranking models with weak supervision. In: Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval. pp. 65–74 (2017)
  * [4] Dogan, U., Deshmukh, A.A., Machura, M., Igel, C.: Label-similarity curriculum learning. arXiv preprint arXiv:1911.06902 (2019)
  * [5] Furlanello, T., Lipton, Z.C., Tschannen, M., Itti, L., Anandkumar, A.: Born again neural networks. arXiv preprint arXiv:1805.04770 (2018)
  * [6] Gu, J.C., Li, T., Liu, Q., Zhu, X., Ling, Z.H., Su, Z., Wei, S.: Speaker-aware bert for multi-turn response selection in retrieval-based chatbots. arXiv preprint arXiv:2004.03588 (2020)
  * [7] Guo, C., Pleiss, G., Sun, Y., Weinberger, K.Q.: On calibration of modern neural networks. arXiv preprint arXiv:1706.04599 (2017)
  * [8] Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015)
  * [9] Iyer, S., Dandekar, N., Csernai, K.: First quora dataset release: Question pairs (2017)
  * [10] Karpukhin, V., Oğuz, B., Min, S., Wu, L., Edunov, S., Chen, D., Yih, W.t.: Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.04906 (2020)
  * [11] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  * [12] Kornblith, S., Shlens, J., Le, Q.V.: Do better imagenet models transfer better? In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2661–2671 (2019)
  * [13] Krichene, W., Rendle, S.: On sampled metrics for item recommendation. In: Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. pp. 1748–1757 (2020)
  * [14] Li, J., Tao, C., Feng, Y., Zhao, D., Yan, R., et al.: Sampling matters! an empirical study of negative sampling strategies for learning of matching models in retrieval-based dialogue systems. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). pp. 1291–1296 (2019)
  * [15] Lin, J., Nogueira, R., Yates, A.: Pretrained transformers for text ranking: Bert and beyond. arXiv preprint arXiv:2010.06467 (2020)
  * [16] Mitra, B., Diaz, F., Craswell, N.: Learning to match using local and distributed representations of text for web search. In: Proceedings of the 26th International Conference on World Wide Web. pp. 1291–1299 (2017)
  * [17] Müller, R., Kornblith, S., Hinton, G.E.: When does label smoothing help? In: Advances in Neural Information Processing Systems. pp. 4694–4703 (2019)
  * [18] Penha, G., Balan, A., Hauff, C.: Introducing mantis: a novel multi-domain information seeking dialogues dataset. arXiv preprint arXiv:1912.04639 (2019)
  * [19] Seo, J.W., Jung, H.G., Lee, S.W.: Self-augmentation: Generalizing deep networks to unseen classes for few-shot learning. arXiv preprint arXiv:2004.00251 (2020)
  * [20] Shnarch, E., Alzate, C., Dankin, L., Gleize, M., Hou, Y., Choshen, L., Aharonov, R., Slonim, N.: Will it blend? blending weak and strong labeled data in a neural network for argumentation mining. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). pp. 599–605 (2018)
  * [21] Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2818–2826 (2016)
  * [22] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. In: Advances in neural information processing systems. pp. 5998–6008 (2017)
  * [23] Xiong, L., Xiong, C., Li, Y., Tang, K.F., Liu, J., Bennett, P., Ahmed, J., Overwijk, A.: Approximate nearest neighbor negative contrastive learning for dense text retrieval. arXiv preprint arXiv:2007.00808 (2020)
  * [24] Xu, Y., Xu, Y., Qian, Q., Li, H., Jin, R.: Towards understanding label smoothing. arXiv preprint arXiv:2006.11653 (2020)
  * [25] Yuan, C., Zhou, W., Li, M., Lv, S., Zhu, F., Han, J., Hu, S.: Multi-hop selector network for multi-turn response selection in retrieval-based chatbots. In: EMNLP. pp. 111–120 (2019)
  * [26] Yuan, L., Tay, F.E., Li, G., Wang, T., Feng, J.: Revisiting knowledge distillation via label smoothing regularization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3903–3911 (2020)
  * [27] Zeyer, A., Irie, K., Schlüter, R., Ney, H.: Improved training of end-to-end attention models for speech recognition. arXiv preprint arXiv:1805.03294 (2018)
  * [28] Zoph, B., Vasudevan, V., Shlens, J., Le, Q.V.: Learning transferable architectures for scalable image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 8697–8710 (2018)


# Active Learning Framework to Automate Network Traffic Classification

Jaroslav Pešek Dominik Soukup  CTU in Prague
Thakurova 9, Prague, Czech Republic
<EMAIL_ADDRESS><EMAIL_ADDRESS>Tomáš Čejka  CESNET
Zikova 4 Prague, Czech Republic
<EMAIL_ADDRESS>
###### Abstract

Recent network traffic classification methods benefit from machine learning
(ML) technology. However, there are many challenges due to use of ML, such as:
lack of high-quality annotated datasets, data-drifts and other effects causing
aging of datasets and ML models, high volumes of network traffic etc. This
paper argues that it is necessary to augment traditional workflows of ML
training&deployment and adapt Active Learning concept on network traffic
analysis. The paper presents a novel Active Learning Framework (ALF) to
address this topic. ALF provides prepared software components that can be used
to deploy an active learning loop and maintain an ALF instance that
continuously evolves a dataset and ML model automatically. The resulting
solution is deployable for IP flow-based analysis of high-speed (100 Gb/s)
networks, and also supports research experiments on different strategies and
methods for annotation, evaluation, dataset optimization, etc. Finally, the
paper lists some research challenges that emerge from the first experiments
with ALF in practice.

###### Index Terms:

Active Learning, Data Quality, Network traffic analysis.

##  I Introduction

Machine learning (ML) is modern technology, and worldwide researchers show its
feasibility in plenty domains including network traffic monitoring and
analysis. Especially, the popularity of end-to-end encryption in network
traffic forces us to leave traditional methods of traffic analysis based on
visibility into Application layer (L7) (as it was feasible in the past, e.g.,
in [1]) or exploiting unencrypted information from connection establishment
(e.g., TLS handshakes in [2]). The decryption of data is not feasible in
practice on scale, and machine learning is a promising way to exploit advanced
statistics derived from encrypted communication flows even on high speed
communication links (presented, e.g., in [3]).

On the other hand, ML is dependent on the target domain and dataset that is
used for training. In most cases, the dataset must be annotated for network
traffic classification (for supervised ML). This requirement makes ML usage
very complicated since we need to provide ground truth labels. Without
reliably annotated datasets, it is not possible to gain relevant results from
ML models.

It is common that the research experiments for scientific papers are performed
using a fixed dataset, i.e., from one fixed (even short) period of time, which
works perfectly in laboratory environments. However, network traffic evolves,
as it is studied by Brabec et al. in [4], and we can intuitively argue that ML
models and datasets are becoming obsolete in time. Also, deployment in a
different network can cause a performance decrease. As a result, even a
perfectly annotated dataset with good performance will drop in accuracy due to
aging. Therefore, we recognize an essential requirement to research methods
for autonomous ML deployment, monitoring, and regular updates over time.

Fortunately, there is a promising concept of Active Learning (AL) as a sub-
field of ML to deal with a huge amount of incoming data. The aim of AL is to
reduce the initial need of labeled data records by intelligent querying labels
during the training phase (AL is described in more details in Sec. II). This
approach can continuously add new data instances and build a relevant up-to-
date dataset for any target domain.

AL is defined as a general concept that leaves a wide space for methods and
possibilities of how entities of AL are used (annotator, query strategy, ML
model, and input data). Many state-of-the-art approaches are incomplete or
unavailable for our network traffic analysis use case; thus, some implemented
system is missing for practical deployment in production. Therefore, we
propose an Active Learning Framework (ALF)111https://github.com/CESNET/ALF
focused primarily on network traffic analysis and designed for stream-based
processing that allows online or offline dataset evaluation.

This paper describes the following main contributions:

  * •

We propose Active Learning Framework prototype — novel flexible and modular
framework for network traffic classification:

    * –

based on Active Learning technology and state-of-the-art principles and
methods;

    * –

allowing for autonomous and continuous dataset creation and evolving;

    * –

supporting automatic updates, evaluation, and optimization of annotated
datasets of network traffic;

    * –

supporting extensive monitoring of ML models and all stages in ALF workflow.

  * •

Using ALF, we evaluate different “Query strategies” during dataset update, and
present new findings contrary to related work.

  * •

We propose more types of annotators to standardize annotation process within
ALF.

  * •

We showcase experiments on real data and use cases to prove feasibility of
ALF.

This paper is a broad extension of our previous work [5], which provided
initials ideas for quality of datasets and ML model deployment.

The paper is organized as follows. Section II lists related works in active
learning, quality of datasets, and related solutions. Section III contains
details about the architecture and main features of the ALF system. Section IV
lists the most common use cases in the networking domain we target and
evaluation of ALF. Section V presents results we have achieved in a real
network environment in online mode. Finally, Section VI mentions related open
research challenges, and Section VII concludes the paper.

##  II Related Work

This section describes related papers to AL methods but also datasets quality
since our framework is focused on more advanced use cases.

###  II-A Active Learning (AL)

AL is a generally known concept in ML domain. Settles [6] provided the first
comprehensive survey in this field. This survey includes available methods and
scenarios that can be applied with AL. The author discusses a pool-based,
resp. stream-based approaches to AL, which are designed for processing an
offline batch of data, resp. online continuous stream of data. Even though
pool-based approaches are commonly chosen by researchers, the author
recommended stream-based approaches as more appropriate for cases like ours.
He also described basic AL Query strategies (i.e., methods to select which
data to annotate), and three of them are relevant for stream-based use cases
(uncertainty sampling, information density, and random sampling).

Recently, a new technical study regarding AL-based methods for network traffic
classification has been published by Shahraki et al. [7]. The authors reviewed
several Query strategies, and empirically tested and evaluated them. Based on
this study, we extend our set of query strategies with the Reinforcement AL
(RAL) method.

Ju el al. [8], used AL method together with concept drift. Wasserman et al.
[9] introduced a concept reinforcement active learning (RAL) which utilizes a
feedback from an annotator. Cardoso et al. [10] defines ranked batch for
removing duplicates (in sense of distance similarity) in batch of selected
flows in one iteration.

###  II-B Quality of datasets

Quality of datasets is important area for real deployment of ML classifiers.
After initial definitions from Soukup et al. [5], Camacho et al. [11] proposed
a permutation based network dataset quality assessment with promising results.
Li et al. [12] proposed a distance-entropy method to distinguish the good and
bad data. This method is described as introductory and calls out for further
investigation in this domain. Dataset quality evaluation in combination with
AL is beneficial to address challenge of stopping criteria to retrain ML
classifier and verify if dataset is better.

Many researchers are focused on data cleaning and dataset optimization that is
relevant extension of AL approach. Yoon et al. [13] proposes a novel meta
learning framework for data evaluation that is jointly optimized with the
target task predictor model. The proposed framework is focused on removing
noisy and incorrect labels. Our framework is based on trusted annotator.
However, it could be beneficial to use these methods in optimization section
of our framework described in Sec. III.

Several papers are separately focused on dataset cleaning which typically
involves dealing with data inconsistencies, duplicates, outliers or missing
values, but also class imbalance [14], class overlapping [15], incorrect
labeling [16], noise in data [17], unnecessary instances [18], and dataset
size problem [19]. These challenges are analyzed separately and general
framework that will allow to implement them together is missing. There are
existing AL libraries such as Scikit-learn222https://scikit-learn.org/stable/,
modAL333https://github.com/modAL-python/modAL or
ALiPy444https://github.com/NUAA-AL/ALiPy. However, none of them is flexible
enough to implement methods to address challenges with building dataset.
Muller et al. [20] analyzed five steps (Discovery, Capture, Curation, Design,
Creation) that are done to create datasets. This workflow was investigated
with 21 data science professionals. The dataset creation is very individual
task and based on author’s reputation. In this work we propose our framework
to standardize building of dataset and their monitoring with optimization
after deployment.

Ginart et el. [21] introduced the need for post-deployment ML monitoring. This
work proposes a new method with theoretical guarantees to detect changes in
input data stream and ML results, such as distribution or data drift. ML model
monitoring is part of our framework too. Moreover, it is connected with AL.
Therefore, it leverages information from annotators and human experts to
automatically improve the dataset based on detected changes.

###  II-C Summary of related work and differences from our research

ML and AL domains are very rich and popular among researchers. Our proposed
solution is based on the available state of the art ideas. According to [6],
we focused primarily the stream-based approach to minimize required capacity
of storage for high speed network traffic. The other mentioned related works
were taken as inspiration for design and development of the working prototype
based on AL.

We delivered a universal and flexible framework, ALF, that implements known AL
query strategies suitable for stream-based AL and combines state-of-the-art
knowledge referenced in the previous sections. We implemented strategies
described by Settles and Shahraki et al. Based on our experiments, better
results are obtained with a batch of flows. For informative selection in
batch, we implemented Cardoso’s ranked batch. We also implemented Wassermann’s
RAL with a referential implementation with non-significant changes. Principles
introduced by Wassermann’s RAL might be used in the future with more
strategies. The AL core is extended by optimization and monitoring modules to
allow post-deployment improvements of the dataset and ML model.

Using ALF, we can start with an initial dataset and a simple ML model and
automatically improve it over time with higher visibility. Also, ALF was
tested on real traffic — IP flow data from multiple high-speed links (order of
100 Gb/s) at the perimeter of ISP network, and outcomes are compared with
results from the current state of the art. The proposed ALF in this paper
extends the works [6, 7] and targets several open challenges. Details are
provided in Sec. III.

To our best knowledge, there is no publicly available software implementation
of active learning technology applied and optimized for i) network traffic
classification, ii) continuous evolving annotated network traffic datasets for
ML, iii) support of research in dataset evaluation and optimization. Such
ambitions were set for the design and development of the proposed ALF
described in our paper.

##  III Active Learning Framework (ALF)

In this work, we propose a framework that aims to improve the deployment and
monitoring of ML models in real networks. This section provides more detailed
description of its design and components.

###  III-A Overview

ALF is designed as a modular framework that can be completely customized.
Generally, it provides AL workflow that works with any input data format and
ML model. However, this paper is mostly focused on the network traffic domain.
An example of the workflow is depicted in high-level scheme in Fig. 1.

Figure 1: Example of Active Learning Framework workflow

As an input, we expect an initial dataset ($D_{0}$) that can be even empty,
annotator, target ML classifier, and unlabeled data. Based on the provided
configuration, the framework can work in offline (pool-based) or online
(stream-based) mode. This brings flexibility since we can process network data
directly from the network interface or load stored traffic in pcap format.
More details are described in Sec. III-C. The input is standardized to feature
the dataset using a preprocessing engine, i.e., the data must be converted to
the supported data format (e.g., CSV) before processing in ALF. Once the
preprocessing is done, ALF starts an AL loop shown in Fig. 2.

Figure 2: Active Learning Loop (AL loop) implemented in ALF

AL loop consists of the following stages that are internally performed
repeatedly within ALF:

  1. 1.

Read receives new IP flow record representing network communication; this
record will be classified using ML model in ALF.

  2. 2.

Query selects (samples) flow records that should be annotated and maybe
included into the dataset (according to a chosen Query strategy, described
later).

  3. 3.

Annotate provides ground truth for the currently selected flow record.

  4. 4.

Postprocess aims to assess quality of the current dataset and optimize it
after possible insertion of newly annotated flow record.

  5. 5.

Train prepares new version of ML model $M_{i+1}$ trained with the current
dataset $D_{i+1}$.

It is worth noting that the AL Loop may skip some phases. For example,
evaluation and optimization tasks (within the Postprocess phase) are too time
consuming, and it is preferable to run them regularly only once per long time
period (e.g., in hours, days, or even weeks) based on a Query strategy and
score function. This loop generates new versions of the dataset $D_{i+1}$,
computes metrics for monitoring, and performs dataset optimization and quality
assessment.

Dataset quality assessment is based on permutation testing introduced by
Camacho et al. [11]. It allows to evaluate the quality of each dataset version
and compare the improvement for possible retraining of the classifier.
Standard AL approach is focused on enhancing an existing dataset with new
valuable data. Nevertheless, this would incrementally increase the size of the
dataset, its redundancy and complexity. Therefore in the proposed ALF, we
include an option for dataset optimization methods that remove unnecessary
records and try to find the minimal dataset as described by Soukup et al. [5].

The process of obtaining ground truth information is very individual among
data scientists (as it is stated by [20]). The understanding of dataset
quality usually depends more on author’s reputation than on data and labels.
Contrary, we argue that the quality of the dataset should be assessed using
statistical and structural properties of the content of the dataset that can
be evaluated and by careful monitoring the performance of the deployed ML
model in production.

ALF uses the concept of annotators to standardize the process of labeling.
Generally, an annotator is a software module that is able to annotate a subset
of input data (network flows in our case) based on some additional information
from external sources (e.g., OSINT, system logs, service logs, or any
monitoring/auditing tools running at end-point devices). Naturally, having a
perfect annotator seems like an ideal solution to the whole classification
task. However, in practice, the annotator is not available for all cases and
cannot be deployed to process the whole traffic. Therefore, it is assumed
annotator is not able to provide ground truth to all data, and some Query
strategy is required to select the subset that can be and should be annotated.
For example, an annotator based on information from end-points cannot be used
in the network environment with devices out of control. The use of decryption
to annotate the traffic is not feasible at a large scale. Our idea is to train
the ML model using an annotator in some controlled environment and, afterward,
deploy and update classification models in the production. This approach
allows automated and consistent labeling that AL methods can leverage.

Collected metrics from all stages are used for monitoring of classifier and
dataset performance over time. These metrics include Matthews correlation
coefficient, $\text{F}_{1}$ score, accuracy, recall, and precision. These
values are visualized for different query strategies over time to compare.

Even though our solution is focused on automation and monitoring of all steps,
there is still a possibility of mislabeling data and getting false positive
(FP) detection results. In such a case, we allowed an external feedback loop
from human operators in ALF. This feature triggers remarking of labels for
specific data records according to provided knowledge. For example, this
feedback is useful for security teams of a Security Operations Center (SOC)
that manually investigate reported security incidents and are able to identify
false alerts that ML reported from ALF.

From the technical point of the overview, ALF has been developed in Python
language and was primarily intended for any UNIX-based operation system. The
whole framework is implemented as a single Python module with several classes.
These classes are cross-referenced and exchange data via class attributes.
Based on this approach, ALF can be used as a standalone module but can also be
integrated with other frameworks, e.g., NEMEA555NEMEA system is an open source
IP flow based tool for stream-wise analysis of network traffic. [1] is a
working example.

###  III-B Architecture

AL loop, described in Sec. III-A, has to be supported by ALF and prepared for
users/developers. Therefore, there are existing components that provide
default functionality and leave the possibility to customize the stages of the
AL loop for specific use cases. The architectural blocks are explained in this
section.

Figure 3 shows the simplified architecture of ALF with the main modules that
are described further. As of now, one instance of ALF is meant for one ML
classifier, more specifically, one classification task in network traffic.
However, the ALF instance can duplicate stages, e.g., to run more Query
strategies simultaneously within the instance. Additionally, the whole
framework can be easily deployed in parallel for more ML classifiers. ALF is
designed as a modular framework that can be customized for each deployment.

Figure 3: High level architecture of ALF components

### Components for Read stage

The Input Load component is responsible for loading input data into a
standardized format for further processing. Current implementation supports
loading online stream of data from the NEMEA system or offline data from the
filesystem. Offline loading can be done from CSV or UniRec files (binary
format of NEMEA).

Loaded data is typically processed to create a feature dataset (defined in
[5]). This operation is performed in the Preprocess component. It includes
custom methods for each ML use case to convert input data (e.g., network
flows) to feature dataset. Data scientists usually develop these methods
during a prototyping stage.

### Component for Query stage

The Query Strategy component is part of the AL core of this framework. It
contains an implementation of AL strategies to select the most valuable
records for annotation and update of the dataset. We implemented the most
popular query strategies based on the state-of-the-art, such as Uncertainty
sampling, Query-By-Committee, Random, Information Density, and Reinforcement
AL (RAL) (the strategies were explained in [6, 9]). Evaluation and performance
results of these methods are described in Sec. IV. Preferably, all the
implemented strategies are stream-based to minimize large data storage
requirements that would otherwise be needed for the pool-based processing.

### Components for Annotate stage

An important part of ALF is an automatic mechanism to annotate input data and
evolve a reliable dataset. This task is handled by the Annotate component. In
this work, we deployed two types of annotators. The first is based on
information from external services. For example, to annotate DoH traffic, we
leverage a database of known DoH resolvers that can be further verified with
some active checks (scans). This type of annotator can be a direct part of
ALF, and the service (database) can be hosted anywhere. The second type of
annotator is hosted on managed endpoints where we can gather a lot of
information about network traffic before it is encrypted. For example, we
developed a browser web extension plugin666https://github.com/jan-
kala/WebTrafficAnnotator that can collect and analyze all requests created by
the web browser. The collected data is paired with a local IP Flow exporter,
and the result is sent to the IP Flow collector, which provides data to ALF.
Using the second type, we continuously annotate traffic even before the Read
stage of the AL loop.

### Components for Postprocess stage

After selecting new data records and modifying (updating) the dataset, we need
to verify whether the update helped or not. This is the purpose of the
Evaluate component. In the current version of ALF, the evaluation is based on
a metric score from the confusion matrix; however, we plan to include the
dataset quality assessment methods natively in the future to have higher
visibility and confidence in decisions about updates.

Based on the provided results from the Evaluate component, we can further
process and optimize the dataset in module Postprocess. During the
postprocessing, we store generated metrics (Matthews Correlation Coefficient,
F1 Score, accuracy, recall, and precision) and logs for long-term monitoring
of ALF instances performance. Also, when we add new data instances, we need to
control dataset size and imbalance ratio. Currently, we use standard methods
of oversampling and undersampling.

###  III-C Monitoring and Configuration

One of the main contributions of this work is a flexible framework that
supports extensive measurement and monitoring of the ML models performance
metrics and datasets properties. The configuration supports to set required
methods and processing based on the target use case. Each module described in
Sec. III-B has its own class attributes, which are implemented by all classes
in a particular module. The user defines these parameters before each
experiment.

Performance monitoring is a native part of ALF. During all stages, we collect
metrics, as described in Sec. III-B. These metrics are persistently stored
into a local database together with a trained ML classifier ($\text{M}_{i}$).
With this approach, ALF can be restarted without affecting/loosing already
processed data. Valuable metrics are visualized using
Grafana777https://github.com/grafana/grafana dashboard.

##  IV Targeted Real Use-Cases

ALF already contains several algorithms that are available in each component
to be chosen for specific needs of a classification problem. This section
describes use cases we have targeted and that helped to evaluate the proposed
framework and its features.

###  IV-A Classification Improvement (even from empty dataset)

ALF instance is able to run even in such configuration with empty or
insufficiently small initial dataset. Annotator with a suitable query strategy
then works as a generator of a completely new initial dataset inside ALF. This
way, it is possible to generate annotated datasets automatically from the real
network traffic, incrementally update them and to evolve machine learning
models.

For demonstration of this use case, we used ALF for classification of DoH
traffic against non-DoH HTTPS communication. First, we captured DoH data to
discover an appropriate query strategies and their parameters. Experimenting
in offline mode on captured data could be considered as white-box testing. We
have discovered several surprising observations:

  * •

approaches based on similarity (such as ranked batch and information density)
do not work well; parameter $\beta$ (see Settles [6]) has actually negative
effect,

  * •

threshold based on score has no significant effect which is in contrary to
Settles; thus, future research might focus on suitable alternatives,

  * •

random query strategy is fast and works sufficiently although is easily
surpassed by uncertainty based strategies (least confident scoring, entropy
scoring, RAL, etc.)

###  IV-B ML Model Development

It is generally complicated to build a precise ML model and keep it updated
for long time. ALF can be used as annotator and build an ML model quickly with
low effort since we assume query strategy selects most informative samples.
Advantage of this approach is saving effort with annotating flows – in
conventional approach we assume all flows to be annotated. In this use case we
do not strictly demand stream-based AL. Even though ALF is developed with
consideration of streams, employing pool-based AL is also possible. In Fig. 4,
we demonstrate a model development through iterations.

Figure 4: Example of development of a model. We compared different Query
strategies (and run without any strategy used — $\emptyset$) on the same data.

###  IV-C Offline Testing

ALF may work in offline mode which is useful for tuning parameters of
prediction models and/or sample strategies. We collected and classified
traffic of cryptomining protocol. Cryptominer dataset then consists of
balanced amount of labels. Experiments were executed repeatedly, results are
shown in Tab. I.

TABLE I: Comparison of Query strategies during the offline experiments with Cryptominer detection. All measurements were executed on the same dataset and the same iteration size = 10k which trigger AL loop Strategy | Average final $F_{1}$ | Average query time [$sec/it$]
---|---|---
unranked uncertainty | 0.952 | 0.001
ranked uncertainty | 0.919 | 2.116
unranked density | 0.699 | 2.592
ranked density | 0.642 | 3.069
KL divergence | 0.919 | 2.820
random | 0.860 | 0.001

##  V Real World Deployment

ALF framework has already been experimentally deployed for pilot testing on a
flow collector in CESNET2888CESNET2 is a national research and education
network infrastructure in the Czech Republic network infrastructure. The
network has many network lines in range from 10 Gb/s to 100 Gb/s connecting
CESNET2 to other backbone networks. On average, the monitoring probes at the
perimeter of the network generate about 300,000 IP flow records per second
during peak hours. This pilot deployment improved debugging and revealed new
requirements and feature requests.

The deployed ALF instance is related to the DNS over HTTPS (DoH) research of
our team. Having a list of known DoH resolvers (i.e., providers of DoH
service) and also using an active scan, it is possible to check whether a
connection is expected to be a DoH. This process of checking creates an
annotator required for ALF. This way, there is a training of a machine
learning model that is constantly evolving to detect DoH communication.

Every day, we observe approximately 3 millions of DoH and 817 millions of TLS
flows. Due to the high imbalance ratio, we applied sampling of TLS traffic to
be equal to DoH flows. For AL loop we used buffer of 50k records that triggers
the loop and selects 10 instances based on the specific strategy. Preview of
the obtained results is in Fig. 5 as a record of one day; and Fig. 6 as a
record of one week. We also use dashboard in Grafana for real-time monitoring.
We achieved results consistent to the tests in laboratory environment
described in Sec. IV-A. Random strategy does not work optimally and includes a
lot of drops. Contrary, the Uncertainty and RAL strategies are very stable and
provides solid results even in a week time frame. It is worth nothing that we
do not use any optimization or instance selection method in this use case.

Figure 5: Measurement during one day. Random strategy is obviously weaker than
uncertainty based strategies. Figure 6: Measurement during one week. Random
strategy is not only weaker but also more sensitive – drops are more deep than
uncertainty based strategies.

##  VI Open Research Challenges

Enhancement of ML classifiers via dataset improvements is very demanding area
that needs additional research, however, our first experiments using ALF
indicate that it is a promising approach orthogonal to the traditional models
and hyperparameters tuning. The deployed ALF under comprehensive continuous
monitoring (and visualizing the monitored statistics) revealed several
interesting observations that helped to identify the following open research
challenges:

  * •

Continuous Evaluation of Datasets from Real Traffic ALF is ready to process a
stream of flow records, update datasets and machine learning for theoretically
infinite time. This way datasets might grow to impractically large sizes that
are not feasible for training. Therefore, ALF is ready for the future research
of evaluation more advanced strategies and methods that can be automatically
applied on datasets to asses their quality.

  * •

Detection of obsoleted state of the trained machine learning model Another
research potential lies in the detection of state when the machine learning
model must be retrained due to some data drift or distribution drift in the
network traffic. This area is useful to avoid unnecessary computation for
model training that is no better than the current one. On the other hand,
timely detection of an obsoleted machine learning model can improve overall
performance of the deployed security system containing the machine learning
model. In ALF, we have valuable data regarding ML and dataset performance.
However, more advanced detection is missing.

  * •

Research in datasets optimization and dataset merging Replacement strategy is
related to the optimization of the dataset. Sometimes, there are large
datasets that contain too many redundant or similar samples. The larger the
dataset, the more time consuming is the training of machine learning models.
Therefore, it is beneficial to remove redundant information and decrease size
of the dataset. Unfortunately, the removal must be performed carefully to
avoid information loss that would prevent training successful machine learning
models.

##  VII Conclusion

Active learning principle has been known to the academic community for many
years. However, its application in network traffic analysis appeared quite
recently. Network monitoring and traffic analysis (e.g., for network security
purposes) are crucial areas that can benefit from machine learning technology.
Moreover, active learning approach allows for continuous updates of both
datasets and machine learning models.

Therefore, we have developed a new Active Learning Framework (ALF) to address
the needs related to machine learning application in the network traffic
analysis domain. ALF is designed to process a stream of extended flow data
(commonly used in practice to monitor large network infrastructures) and to
evolve datasets and machine learning models using an annotator, optimization
methods, and real online traffic. The reason for ALF use instead of some alone
annotator lies in the performance limits, and computational or time complexity
of the annotating (i.e., labeling process), and finally possibility to
retrieve ground truth information that is not available for all data in
practice. Therefore, ALF is meant to train machine learning models using
available information by the annotator so that such models can be deployed in
the other network environments where no annotator can be used.

The paper also lists several research areas related to the active learning in
network traffic and, more specifically, to ALF. We believe the current version
of ALF will help to accelerate research activities in areas like quality of
dataset assessment, research of replacement strategies and dataset
optimization, and so on.

## Acknowledgement

This work was supported by the European Union’s Horizon 2020 research and
innovation program under grant agreement No. 833418 and also by the Grant
Agency of the CTU in Prague, grant No. SGS20/210/OHK3/3T/18 funded by the MEYS
of the Czech Republic.

## References

  * [1] T. Cejka, V. Bartos, M. Svepes, Z. Rosa, and H. Kubatova, “Nemea: A framework for network traffic analysis,” in _2016 12th International Conference on Network and Service Management (CNSM)_ , 2016, pp. 195–201.
  * [2] B. Anderson, S. Paul, and D. McGrew, “Deciphering malware’s use of tls (without decryption),” _Journal of Computer Virology and Hacking Techniques_ , vol. 14, 08 2018.
  * [3] Z. Tropková, K. Hynek, and T. Čejka, “Novel https classifier driven by packet bursts, flows, and machine learning,” in _2021 17th International Conference on Network and Service Management (CNSM)_ , 2021, pp. 345–349.
  * [4] J. Brabec, T. Komárek, V. Franc, and L. Machlica, “On model evaluation under non-constant class imbalance,” _Computational Science – ICCS 2020_ , vol. 12140, pp. 74 – 87, 2020.
  * [5] D. Soukup, P. Tisovčík, K. Hynek, and T. Čejka, “Towards evaluating quality of datasets for network traffic domain,” in _2021 17th International Conference on Network and Service Management (CNSM)_ , 2021, pp. 264–268.
  * [6] B. Settles, “Active learning literature survey,” 2009.
  * [7] A. Shahraki, M. Abbasi, A. Taherkordi, and A. D. Jurcut, “Active learning for network traffic classification: A technical study,” _IEEE Transactions on Cognitive Communications and Networking_ , vol. 8, no. 1, pp. 422–439, 2022\.
  * [8] J. Lu, A. Liu, F. Dong, F. Gu, J. Gama, and G. Zhang, “Learning under concept drift: A review,” _IEEE Transactions on Knowledge and Data Engineering_ , vol. 31, no. 12, pp. 2346–2363, 2019.
  * [9] S. Wassermann, T. Cuvelier, and P. Casas, “RAL - Improving Stream-Based Active Learning by Reinforcement Learning,” in _European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML-PKDD) Workshop on Interactive Adaptive Learning (IAL)_. [Online]. Available: https://hal.archives-ouvertes.fr/hal-02265426
  * [10] T. N. Cardoso, R. M. Silva, S. Canuto, M. M. Moro, and M. A. Gonçalves, “Ranked batch-mode active learning,” vol. 379, pp. 313–337. [Online]. Available: https://linkinghub.elsevier.com/retrieve/pii/S0020025516313949
  * [11] J. Camacho and K. Wasielewska, “Dataset quality assessment in autonomous networks with permutation testing,” in _IFIP/IEEE International Symposium on Integrated Network Management (IM)_ , 2022.
  * [12] Y. Li and X. Chao, “Distance-entropy: An effective indicator for selecting informative data,” _Frontiers in Plant Science_ , vol. 12, 01 2022.
  * [13] J. Yoon, S. Arik, and T. Pfister, “Data valuation using reinforcement learning,” in _Proceedings of the 37th International Conference on Machine Learning_ , ser. Proceedings of Machine Learning Research, H. D. III and A. Singh, Eds., vol. 119. PMLR, 13–18 Jul 2020, pp. 10 842–10 851. [Online]. Available: https://proceedings.mlr.press/v119/yoon20a.html
  * [14] A. Sahu, Z. Mao, K. Davis, and A. E. Goulart, “Data processing and model selection for machine learning-based network intrusion detection,” in _2020 IEEE International Workshop Technical Committee on Communications Quality and Reliability (CQR)_ , 2020, pp. 1–6.
  * [15] M. Dudjak and G. Martinovic, “An empirical study of data intrinsic characteristics that make learning from imbalanced data difficult,” _Expert Systems Applications_ , vol. 182, p. 115297, 2021.
  * [16] F. R. Cordeiro and G. Carneiro, “A survey on deep learning with noisy labels: How to train your model when you cannot trust on the annotations?” in _33rd SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI)_ , 2020, pp. 9–16.
  * [17] S. Gupta and A. Gupta, “Dealing with noise problem in machine learning data-sets: A systematic review,” _Procedia Computer Science_ , vol. 161, pp. 466–474, 2019, the Fifth Information Systems International Conference, 23-24 July 2019, Surabaya, Indonesia. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S1877050919318575
  * [18] J. Olvera-López, J. Carrasco-Ochoa, J. F. Martínez-Trinidad, and J. Kittler, “A review of instance selection methods,” _Artif. Intell. Rev._ , vol. 34, pp. 133–143, 08 2010.
  * [19] C. Sun, A. Shrivastava, S. Singh, and A. Gupta, “Revisiting unreasonable effectiveness of data in deep learning era,” in _IEEE International Conference on Computer Vision (ICCV)_ , 2017, pp. 843–852.
  * [20] M. Muller, I. Lange, D. Wang, D. Piorkowski, J. Tsay, Q. V. Liao, C. Dugan, and T. Erickson, “How data science workers work with data: Discovery, capture, curation, design, creation,” in _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_ , ser. CHI ’19. New York, NY, USA: Association for Computing Machinery, 2019, p. 1–15. [Online]. Available: https://doi.org/10.1145/3290605.3300356
  * [21] T. Ginart, M. Jinye Zhang, and J. Zou, “Mldemon:deployment monitoring for machine learning systems,” in _Proceedings of The 25th International Conference on Artificial Intelligence and Statistics_ , ser. Proceedings of Machine Learning Research, G. Camps-Valls, F. J. R. Ruiz, and I. Valera, Eds., vol. 151. PMLR, 28–30 Mar 2022, pp. 3962–3997. [Online]. Available: https://proceedings.mlr.press/v151/ginart22a.html