# Knodle: Modular Weakly Supervised Learning with PyTorch

Anastasiia Sedova
University of Vienna
Vienna, Austria
<EMAIL_ADDRESS>
Andreas Stephan
University of Vienna
Vienna, Austria
<EMAIL_ADDRESS>
Marina Speranskaya
Ludwig Maximilian University of Munich
Munich, Germany
<EMAIL_ADDRESS>
Benjamin Roth
University of Vienna
Vienna, Austria
<EMAIL_ADDRESS>

###### Abstract

Methods for improving the training and prediction quality of weakly supervised
machine learning models vary in how much they are tailored to a specific task,
or integrated with a specific model architecture. In this work, we propose a
software framework that provides a modularization for separating weak data
annotations, powerful deep learning models, and methods for improving weakly
supervised training. This modularization gives the training process access to
fine-grained information such as data set characteristics, matches of
heuristic rules, or elements of the deep learning model ultimately used for
prediction. Hence, our framework can encompass a wide range of training
methods for improving weak supervision, ranging from methods that only look at
the correlations of rules and output classes (independently of the machine
learning model trained with the resulting labels), to those methods that
harness the interplay of neural networks and weakly labeled data.

##  1 Introduction

Most of today’s machine learning success stories are built on top of huge
labeled data sets. Creating and maintaining such data sources is a manual,
time consuming and complicated process, thus expensive and error prone.
Various research directions aim to reduce the hunger for bigger and better
datasets.

One of the most popular approaches that recently gains traction is weak
supervision. The learning algorithm is confronted with labels which are easy
to obtain but lack any correctness guarantee. Among others, the creation of
such labels is performed by regex functions, keyword lists, or external
databases (so-called _distant supervision_). Typically, weak supervision
methods and implementations are tailored towards a domain-specific task or
integrated with a specific model architecture. For example, (Lin et al., 2016)
introduce a special architecture called attention over instances for the
relation extraction task, (Keith et al., 2017) propose an EM-based algorithm
for event extraction and (Hedderich et al., 2021) uses the perturbation for
named entity recognition. Such diversity and specificity of approaches makes
it difficult to compare them or transfer to another tasks.

We introduce Knodle: a framework for Knowledge-supervised Deep LEarning, i.e.,
weak supervision with neural networks. The framework provides a simple tensor-
driven abstraction based on PyTorch allowing researchers to efficiently
develop and compare their methods for improving weakly supervised machine
learning models. The following points summarize Knodle main goals:

  * •

Data abstraction. The interface is a tensor-driven data abstraction which
unifies a large number of input variants and is applicable to a large number
of tasks.

  * •

Method independence. The interface of Knodle decouples the implementation of
weak supervision improvement methods and prediction models. This enables
comparability and accounts for domain-specific inductive biases.

  * •

Accessibility. Knodle provides a high level access to the library, what makes
it easy to test existing methods, incorporate new ones and benchmark them
against each other.

Several denoising algorithms as well as classification models and dataset are
already included to Knodle in order to allow the “out-of-the-box” framework
testing. However, the new denoising methods could be also developed and easily
added since it provides a simple tensor-driven abstraction based on PyTorch.

We also discuss the results of Knodle on datasets that exemplify different
properties occurring in weakly supervised learning, such as amount of labeling
functions or their precision-recall balance. Moreover, depending on the
properties of heuristic labelings, methods for improving weak labels need to
remove spurious matches in some cases, or generalize from them in others.

It is clear that such a diverse problem space should be paired with a rich
pool of methods so that each task can be paired with the most appropriate
method. Knodle allows to easily explore the spaces of weakly supervised
learning settings and label improvement algorithms, and hopefully will
facilitate a better understanding of the phenomena that are inherent to weakly
supervised learning.

All codes are publicly available at https://github.com/knodle/knodle, the
datasets could be loaded from Knodle Minio storage server .

##  2 Related work

Many strategies were introduced to reduce the need for huge amounts of data
annotated manually by experts. Among these are active learning (Sun and
Grishman, 2012), where automatically selected instances are manually annotated
by experts, or semi-supervised learning, (Kozareva et al., 2008; Agichtein and
Gravano, 2000), where a small labeled dataset is combined with a huge
unlabeled dataset. Since the arrival of BERT, pre-training and fine-tuning
schemes showed good results for a moderate to small amount of labeled data.
However, Knodle completely abstains from labeled data during training by
making use of weakly supervised learning.

###  2.1 Weak supervision

Instead of manual expert supervision, data is annotated with easy to obtain,
but potentially error-prone labels. Usually, weak supervision is performed
using a set of heuristic rules that are applied to the data to obtain weak
labels. One of the most popular strategies of weakly supervised learning is
distant supervision, which uses knowledge from existing data sources to
annotate unlabeled data. The technique is very popular in relation extraction
(Craven and Kumlien, 1999), where various knowledge databases, such as WordNet
(Snow et al., 2004), Wikipedia (Wu and Weld, 2007) and Freebase (Mintz et al.,
2009), are used as annotation sources.

The most straightforward denoising approach is majority voting, which is used
in early weak supervision algorithms (Thomas et al., 2011) as well as in more
recent experiments (Krasakis et al., 2019; Boland and Krüger, 2019). However,
weakly supervised data produces different kinds of noise, creating a demand
for more noise-specific algorithms. The noise produced by incomplete labels,
which stems from the incompleteness of weak supervision sources, often
increases the number of false negatives. It is often reduced by data
manipulations, e.g. enhancing the knowledge base with the most likely relevant
entries (Xu et al., 2013), by a thorough construction of negative examples to
balance the positive examples (Riedel et al., 2013), or by explicitly
modelling missing knowledge base information with latent variables (Ritter et
al., 2013). The problem of noisy features (increased amount of false positive
labels because of overgeneralising information from databases) is often solved
by using a relaxed distant supervision assumption (Riedel et al., 2010;
Hoffmann et al., 2011), by active learning with additional manual expertise
(Sterckx et al., 2014), with help of topic models (Yao et al., 2011; Roth and
Klakow, 2013), as well as using a combination of multiple methods (Roth,
2014). Apart from that, methods treat the identified potentially noisy samples
differently. They are either kept for further training with reduced weights
(Jat et al., 2018; He et al., 2020), corrected (Shang, 2019) or eliminated
(Qin et al., 2018).

Within this work, we refer to a denoising method as any method that helps to
improve the weakly supervised data regardless the type of noise and the exact
denoising object (weak labels, weak rules etc).

###  2.2 Structure Learning

Structure learning assumes multiple weak labels per instance where each label
is created by a so called labeling function. The goal is to learn a dependency
structure within these labeling functions which motivates the term structure
learning. Most labeling functions are generated by human intuitions,
motivating correlation and dependence between labeling functions. The first
correlation and marginalization based algorithm introduced to the field was
implemented in software package Snorkel (Ratner et al., 2017), which also
comprised the data programming paradigm allowing to programmatically create
labeling functions. Subsequently, various improvements were made (Bach et al.,
2017; Varma et al., 2019). Variations of the task formulation use stochastic,
or more specifically, quality guided labeling functions (Chatterjee et al.,
2019) or introduce semi-supervised learning to the field (Maheshwari et al.,
2020).

###  2.3 Noise-aware learning

A common idea to denoise single noisy labels is to build an architecture which
accounts for noisy data. There are different approaches to model noise-
robustness by adapting the loss function, such as a noise adaption after the
loss calculation (Patrini et al., 2017), or cross-entropy and the mean
absolute error generalization (Zhang and Sabuncu, 2018). Alternatively, a
special noise layer in a neural network could be used (Sukhbaatar et al.,
2015). Many approaches are based on noise assumptions, such as on the
assumption of symmetric label noise (van Rooyen et al., 2015). Another
approach aims at finding and removing wrongly labeled samples from the
training procedure. Recently, the confidence learning framework CleanLab was
introduced (Northcutt et al., 2021), which is based on the intuition that low-
confidence predictions are more likely to be labeled wrongly. Note that most
of these methods were built with the assumption that there is one label
corresponding to each instance while Knodle is makes use several weak signals
per instance.

###  2.4 Crowdsourcing annotations

Another possible solution to reduce the cost of manual data supervision is
crowdsourcing \- annotating the data by the people who are not experts in the
field. Most crowdsourcing experiments are done using the annotations of
multiple people on the same task. From a practical point of view, there are
multiple noisy labels, from which a correct annotation has to be chosen. The
presented system, Knodle, also expects multiple labels, thus the input format
is similar. Still, weakly labeled data sources often have different
properties, e.g. low coverage of a labeling function, presenting a different
challenge to the system. The simplest solution is majority voting (Kosinski et
al., 2012). An alternative measure is given by the inter-annotator agreement
(Tratz and Hovy, 2010). To reduce errors produced by spammers, who either
randomly or by some strategy choose answers to maximise their payment, more
sophisticated approaches are needed. These include anomaly detection (Eskin,
2000), annotator’s reliability modelling (Dawid and Skene, 1979), bayesian
approaches (Raykar and Yu, 2012) and generative models Hovy et al. (2013). The
mistakes of careful but biased people (for example, parents with young
children who tend to be more conservative regarding the websites’ age
requirement) could be easily predicted and, therefore, easily corrected, or
”reversed” as in (Ipeirotis et al., 2010).

###  2.5 Frameworks

Knodle bases upon the ideas of several software frameworks. On a low level,
Knodle is built on top of PyTorch (Paszke et al., 2017). We took design
decisions from several other high level libraries which aim to ease the
training and prediction experience. Namely, we drew inspiration from PyTorch
lightning by (Falcon, 2019) which in essence tries to remove the burdens of
writing your own train loop and Huggingface’s Transformers library, (Wolf et
al., 2020) which gives easy access to various transformer-based architectures.
Particularly, we took over the Trainer class design both use.

##  3 Weakly supervised learning with Knodle

Figure 1: The figure gives an overview of our system. $(a)$ represents the
preprocessed input, given as tensors.$(b)$ resembles the internals of _Knodle_
. The Trainer classes introduced in section 3.4 handle transformation,
denoising and model training in an integrated manner. $(c)$ shows the output,
a trainer PyTorch model.

We see _Knodle_ framework as providing a benchmark where different algorithms
for improving the weakly supervised data could be implemented and compared. As
it was already mentioned, under the improving algorithms we understand not
only the ones that realises the denoising of labels (e.g. WSCrossWeigh), but
also the denoising of rules (e.g. k-NN; see a more detailed explanation of
both algorithms in Section 4), so that it is expected to increase the number
of potential application.

From the practical point of view, _Knodle_ provides a layer of abstraction
that allows the integration of weakly supervised learning signals and
improving algorithms into PyTorch. Thus, the new algorithms could be easily
integrated into it using the well-known Pytorch configurations. To the best of
our knowledge, _Knodle_ is the first framework providing such functionality.

The framework follows two design principles, outlined below. The first one
derives a data specification whereas the second one clarifies the modeling
side of _Knodle_. Apart from that, _Knodle_ provides different solutions for
handling the negative instances, which are also covered in this section.

###  3.1 Tensor-based representations of input data and weak label matches

Similar to Pytorch models, where the data is already assumed to be in Tensor
form, and the specific pre-processing that led to a particular input
representation is outside the scope of the core deep learning model, we choose
to exclude the process of weak label generation from Knodle. Rather, we encode
the weak labels in two tensors. One tensor contains information about which
rules matched for each data instance, while another tensor describes the
relationship between rules and output classes.

Formally, assume we have $r$ rules and $k$ classes. Rule matches are gathered
then in a binary matrix $Z\in\\{0,1\\}^{n\times r}$, where $Z_{ij}=1$ if rule
$j$ matches sample $i$. The initial mapping from rules to the corresponding
classes is given by another binary matrix $T\in\\{0,1\\}^{r\times k}$,
$T_{jk}=1$ if rule $j$ is indicative of class $k$.

This separation between one tensor that contains rule matches and another
tensor that translates them to labels allows _Knodle_ to access this fine-
grained information during training for certain denoising algorithms. This is
in contrast to other approaches that treat weak supervision as learning from a
noisy heuristic label matrix $Y_{heur}=ZT$ without direct access to the
individual rules.

###  3.2 Separation of the prediction model from the weak supervision aspects.

Although _Knodle_ contains both methods to improve the data, such as rules
denoising and generalization, and the training of standard PyTorch models,
these aspects are separated in the framework. Therefore the same prediction
model definition can be used for direct or weak supervision training, allowing
researchers to easily create benchmarks with consistent base models. However,
even though the prediction model is defined separately, the method dealing
with weak data has access to it during training. Thus, a denoising method and
model training can reinforce each other in an integrated manner. For example,
cross-validation schemes, which will be covered later in this section, can use
the model architecture to reweigh training instances. This is in contrast to
approaches that modularize denoising and training by first adjusting label
confidences with the help of rules correlations only and then training a model
with the adjusted labels (Takamatsu et al., 2012; Ratner et al., 2017).
Furthermore, _Knodle_ design is in contrast to the approaches where denoising
is so tightly integrated into the model architecture that it is impossible to
change the underlying prediction model (Sukhbaatar et al., 2015).

In summary, _Knodle_ is able to train any prediction model
$f_{\theta}:\mathbb{R}^{d}\rightarrow\mathbb{R}^{k}$ using a given model
$f_{\theta_{0}}$, tensors $X$, $Z$ and $T$ and a specified training method.
Importantly, _Knodle_ does _not_ directly rely on the weak labels $Y_{heur}$.

###  3.3 Handling of negative instances

In every dataset there could be data samples where no rule matches, which are
traditionally called negative instances and are usually either filtered out
Ratner et al. (2017) or partly included to the training dataset Shu et al.
(2020). However, such a cut-and-dried design decision significantly restricts
the system functionality, since the necessity of having negative instances in
a training set is highly dependent on the task and the dataset. That was a
motivation to include to Knodle both ways of handling such cases depending on
the user’s needs.

The first option is to filter out such samples while calculating the labels.
That is especially important while applying the denoising algorithms where
having negative instances could do harm. For example, k-NN method, where the
samples that are not supposed to match any rule from the very beginning will
lower the neighbours’ quality and could even be allocated to some positive
class after getting some matched rules from the neighbor.

The second option is to save negative instances for training as belonging to a
negative class, which is already defined in development and test data, or
specially introduced for the training data only. That could help to improve
the accuracy of trained classifiers for such tasks, as relation extraction,
where it is crucial to have a substantial amount of negative instances to
train a well-performing model.

The negative class information is also stored in the configuration file and
can also be easily used, if needed (e.g. for class weights calculations or
specific evaluation method for datasets with negative instances which is also
included in Knodle).

###  3.4 Implementation Details

As the most popular deep learning frameworks, such as TensorFlow (Abadi et
al., 2015) and PyTorch (Paszke et al., 2017), we realise learning as a mapping
from input tensor(s) to output tensor(s) guided by a loss function that
measures the quality of the learned mapping. However, while the most common
solution is to represent the training data by a _design matrix_
$X\in\mathbb{R}^{n\times d}$ ($n$ instances represented by $d$ feature
dimensions) and a _label matrix_ $Y\in\mathbb{R}^{n\times k}$ ($k$ classes),
input of Knodle are matrices $x$, $z$ and $t$ described above. The heuristic
labels are calculated later during the weakly supervised learning using the
information contained there.

To ensure seamless use of the developed algorithms, the weakly supervised
algorithms need to be tightly integrated with automatic differentiation and
optimization supported by PyTorch. All of this is done within Trainer classes.
During initialization, they receive data, a model, possibly pre-initialized or
pretrained, and a method-specific configuration, inheriting from Config and
containing such information as model training parameters, criterion,
validation method, class weights and others.

While in the standard deep learning frameworks training can be executed by a
call model.train(X,Y), in Knodle the same functionality would be realised with
the following command (illustrates the usage of the denoising KNNTrainer that
uses the k-NN search, which we describe in 4):

KNNTrainer(model, X, Z, T, config)

##  4 Denoising algorithms (Trainers)

dataset | classes | train / test samples | rules | avg. rule hits | class ratio
---|---|---|---|---|---
Spam | 2 | 1586 / 250 | 10 | 1.63 | 0.47
Spouse | 2 | 22254 / 2701 | 9 | 0.34 | 0.08
IMDb | 2 | 40000 / 5000 | 6786 | 33.97 | 0.50
TAC-based RE | 41 | 1937211 / 18660 | 182292 | 0.51 | -
Table 1: Summary of data statistics. The average rule hits are computed on the
train set.
Class ratio describes the amount of positive samples in the test set for
binary classification datasets, i.e. data skewedness.

We already provide some out-of-the-box trainers which can serve as baselines
or starting points for further investigations. All Trainers are compatible
with any Pytorch model; _Knodle_ provides logistic regression and
HuggingFace’s transformers (Wolf et al., 2020) classification models as an
example.

Majority Voting Baseline. As a simple baseline, the rules are directly applied
to the test data. If several rules match, the prediction is done based on the
majority, ties are broken randomly.

Trainer without Denoising. The simplest trained model is the
NoDenoisingTrainer. The majority vote is computed on the training data and
used to train the given model. This is the most direct use of the rule matches
for training a classifier. To cover cases where several rules match, this
trainer can be configured to either use a one-hot encoding of the winning
label from the majority vote or a distribution over labels (relative to the
number of matching rules).

Trainer with kNN Denoising. This Trainer includes the label denoising method
with a simple geometric interpretation (KNNDenoisingTrainer). The intuition
behind this method is that similar samples should be activated by the same
rules which is allowed by a smoothness assumption on the target space. The
trainer looks at the $k$ most similar samples sorted by, for example, tfidf
features combined with $L_{2}$ distance, and activates the rules matching the
neighbors to create a denoised $\hat{Z}$. Importantly, _Knodle_ allows
separate features for the model training and the neighborhood activation. This
method also provides a way to activate rules for unmatched samples.

Trainer with Snorkel Denoising. _Knodle_ provides a wrapper of the Snorkel
system (SnorkelTrainer). This trainer incorporates both generative and
discriminative Snorkel steps. First, the structure within labels and rules, in
our notation $P(Y,Z,T)$, is learned in an unsupervised fashion (_generative
model_ (Ratner et al., 2017)). Afterward, the training of the final
discriminative model, approximating $P(Y|X)$ according to the labels provided
by the generative model, is performed. _Knodle_ conveniently provides both
steps in a single method call. The mix of Snorkel and k-NN denoising methods
is provided in the SnorkelKNNDenoisingTrainer.

Trainer with Weak Supervision CrossWeigh Denoising. Weak Supervision
CrossWeigh (WSCrossWeigh) is an algorithm for noise correction in weakly
supervised data based on the CrossWeight Framework (Wang et al., 2019) and
included to Knodle as WSCrossWeighTrainer. It allows to detect the samples,
which are annotated with potentially wrong labels, and reduce their importance
in final classifier training. Though the CrossWeigh method was firstly
proposed for mistakes identification in crowdworkers annotations, we claim
that it could be also applied for denoising the weakly supervised data, since
the weak rules could be regarded as human annotators in a sense that they both
make decision about the sample label.

The main intuition behind WSCrossWeigh is the following: if a decision rule
corresponds to a wrong class and, therefore, annotates all samples in the
training set with a wrong label, the model is likely to learn the incorrect
pattern and to make similar mistakes while labeling the test samples. However,
if we take sufficiently big data without samples where this particular rule
matched, train the model on it, and then classify the test samples, the
predictions are supposed to contradict the initial wrong labels, what helps us
to trace the misclassified samples.

As in the original CrossWeigh, the basic idea is similar to the k-fold cross-
validation, where input data is split into k folds, each of which becomes, in
turn, a test set, while the model is trained on the others. In WSCrossWeigh,
however, the splitting is performed not randomly, but basing on rules.
Firstly, the rules are randomly split into K folds $\\{r_{1},\dots,r_{k}\\}$
and, iteratively, each $fold_{l}$ is chosen to form a test set that is built
from all samples matched this fold’s rules. Other samples, in their turn,
build a training set that is used for classification model training. During
the testing of the trained model on the hold-out fold samples, the predicted
label $\hat{y_{i}}$ for each test sample $x_{i}$ is compared to the label
$y_{i}$ originally assigned to $x_{i}$ by weak supervision. If
$\hat{y_{i}}\neq{y_{i}}$, the sample $x_{i}$ is claimed to be potentially
mislabeled, and its weights $w_{x_{i}}$ is reduced by a value of an
empirically estimated parameter $\epsilon$. This procedure could be repeated
several time to increase the accuracy of the misclassified samples detection.

After all sample weights are calculated, the final classifier is trained on
the whole reweighed training dataset. As a result, the more times the original
$y_{i}$ label of data sample $x_{i}$ was suspected to be wrong, the smaller is
its weight $w_{x_{i}}$, and, therefore, the smaller part it will play in the
classifier training.

##  5 Datasets

| Spam | Spouse | IMDb | TAC-based RE
---|---|---|---|---
Mode | Acc | P | R | F1 | Acc | P | R | F1
Majority vote | 0.81 | 0.12 | 0.79 | 0.22 | 0.65 | 0.09 | 0.001 | 0.001
Majority + DistillBert | 0.87 | 0.09 | 0.9 | 0.17 | 0.67 | 0.20 | 0.19 | 0.19
k-NN + DistillBert | 0.94 | 0.12 | 0.86 | 0.21 | 0.5 | - | - | -
WSCrossWeigh + DistillBert | 0.94 | 0.09 | 0.69 | 0.16 | 0.73 | 0.25 | 0.27 | 0.26
Snorkel + DistillBert | 0.88 | 0.13 | 0.7 | 0.23 | - | - | - | -
Table 2: Results of the classifier training with different denoising methods
on the datasets added to Knodle.

Apart from some denoising methods, we also include some well-known datasets in
the Knodle-specific tensor format in order to demonstrate the abilities of our
framework. All datasets, collected for straightforward prediction tasks
(sentiment analysis, spam detection, and relation extraction) and being quite
simple, still have their own peculiarities in $Z$ and $T$ matrices, that are
interesting to be investigated. The Knodle ecosystem, therefore, provides
modular access to datasets and denoising methods (that can, in turn, be
combined with arbitrary deep learning models), enabling easy experimentation.
The overview of dataset statistics is shown in Table 1.

Spam Dataset. The first dataset is a Spam dataset based on the YouTube
comments dataset (Alberto et al., 2015). Here, the task is to classify whether
a text is relevant to the video or holds spam, such as advertisement. The
dataset has a small size of both train and test sets. Thus, a single wrongly
labeled instance might have quite a big impact on the learning algorithm. We
use the preprocessed version by the Snorkel team (Snorkel, 2020b). Among
others, the rules were created based on keywords and regex expressions.

Spouse Dataset. This relation extraction dataset is based on the Signal Media
One-Million News Articles
Dataset111http://research.signalmedia.co/newsir16/signal-dataset.html (Corney
et al., 2016). The task is to decide whether a sentence holds a spouse
relation or not. We also inherited labeling functions written by the Snorkel
team (Ratner et al., 2017) that include a set of known spouse from DBPedia
(Lehmann et al., 2014) as well as keywords and encoded language patterns. The
difficulty of the Spouse dataset is its skewness: over $90\%$ of samples in
the test set hold a no-spouse relation. Again, the preprocessed version by the
Snorkel team is used (Snorkel, 2020a), so the results can be related to
previous studies (Ratner et al., 2017).

IMDb Dataset. The third dataset is based on the well-known IMDb dataset, which
consists of short movie reviews. The task is to determine whether a review
holds a positive or negative sentiment. We use the positive and negative
keyword lists (Hu and Liu, 2004), with a total of $6800$ keywords, to create
the $Z$ and $T$ matrices.

TAC-based Relation Extraction Dataset. Lastly, in order to evaluate the
relation extraction task on a larger dataset, we added to Knodle a dataset
built over Knowledge Base Population challenges in the Text Analysis
Conference. For development and test purposes the corpus annotated via
crowdsourcing and human labeling from KBP is used (Zhang et al., 2017). The
training is done on a weakly-supervised noisy dataset based on TAC KBP corpora
(Surdeanu, 2013; Roth, 2014). The TAC dataset was annotated with entity pairs
extracted from Freebase (Google, 2014) where corresponding relations have been
mapped to the 41 TAC relations. The amount of entity pairs per relation was
limited to 10.000 and each entity pair is allowed to be mentioned in no more
than 500 sentences. An important difference of this dataset to the other three
is the presence of negative instances added to the dataset in equal proportion
to the positive ones (we discussed the details of handling the negative
instances in Knodle in 3.3).

##  6 Experiments

To demonstrate how Knodle works in practice, we performed a couple of
experiments. Our main aim was not to receive the best possible results for
each dataset, but rather to demonstrate that the different methods for
improving the weak annotations perform differently on each dataset and with a
different success rate depending on the data properties and peculiarities.
Thus, in this section we pay more attention to the explanation of the received
results from the data and denoising method combination point of view. That
also serves to demonstrate the Knodle’s role of a benchmark for different
algorithms, where all of them could be applied in turn and their performance
for a specific data could be easily compared.

###  6.1 Experimental Details

In all experiments, the DistillBert uncased model for English language (Sanh
et al., 2019) provided by the HuggingFace 222https://huggingface.co/ (Wolf et
al., 2020) library is used as the prediction model. The optimization is
performed with the AdamW optimizer, a learning rate of $1^{-4}$ and a cross-
entropy loss allowing probabilistic labels. Each model was trained for 2
epochs, apart from where others stand.

Additionally, each denoising method demands its own hyperparameters. For the
k-NN algorithm we made the experiments with different $k$ values and provide
TF-IDF features based on a dictionary of $3000$ words. We use vectors to
compute nearest neighbours with cosine similarity. The WSCrossWeigh denoising
algorithm needs the number of folds the data will be split into, the number of
partitions (that is, how many times the mistake estimation module will be
launched) and a weight-reducing rate (the value, by which the initial sample
weights will be reduced to each time the sample was predicted wrongly). These
parameters are tuned for each dataset individually. The following best
parameter values were found empirically: $3$, $10$ and $0.3$ for the Spam
dataset, $3$, $2$ and $0.3$ for the Spouse dataset and $2$, $25$, $0.7$ for
the IMDb dataset. Apart from that, Knodle provides the opportunity to train
the sample weights with a model different from the final classifier. In our
experiments, the weights were calculated using a Bidirectional LSTM with Glove
Embeddings (Pennington et al., 2014), while the final training was performed
with DistillBert using the same settings as in the experiments with other
denoising methods. The only difference is the number of epochs on the TAC-
based dataset: the best results were obtained with $1$ DistillBert epoch.

###  6.2 Results

An overview of the numbers is given in Table 2. In the Spam dataset, all
denoising methods show an improvement over the simple majority vote baseline.
The data-adaptive k-NN and WSCrossWeigh methods perform best in this setting.
Snorkel and the standard majority voting combined with fine-tuning overfit to
the noisy majority votes. This becomes obvious with the observation that
Snorkel achieves a score of $0.93$ with a simple logistic regression
discriminative model. Interestingly, k-NN performs so well - this can serve as
a proof of neighbouring labels reliability.

Compared to the Spam dataset, the Spouse dataset is much larger. Since the
test set is very unbalanced (the class ratio equals $0.08$), all given metrics
are related to the is-spouse relation, i.e. the underrepresented class. On
average, $0.34$ rules hit per instance, meaning that in almost $70\%$ of the
training data match no rule. Thus, the majority vote is mostly based on random
votes, rendering a high recall but low precision. During a close inspection of
the rules it became clear that they overrepresent the is-spouse class. Thus,
the additional model training magnifies this overfitting which is expressed by
an increased recall and a lower precision. The only denoising system that
turned out to be generalise well is Snorkel. One of the possible reasons of
that is that it provides an explicit rules denoising in contrast to other
methods.

For IMDb, the majority vote shows that the rules have rather low quality.
Still, the addition of a trained model manages to generalize from the given
labels. However, denoising with the k-NN algorithm highlights the problems
inherent to the labels. After its application, the classifier degrades to a
random vote ($50\%$ accuracy). This behaviour can be explained by the high
average of rule hits: on average no less than $33$ keywords match for each
sentence. Thus k-NN with $k=1$ adds on average $33$ labels to its neighbours
which worsens the results. In general, there are cues that k-NN might useful
in cases where the weak labels are already rather reliable but fail in cases
where weak labels are too noisy. On the contrary, WSCrossWeigh denoising
algorithm, which deals not with the rules, but with the data once labeled with
them, works for this dataset and can improve the results. Snorkel fails to
calculate its discriminative model with $6800$ rules in the training matrix,
since its memory consumption is too large for 32Gb RAM.

TAC-based RE dataset turned out to be the most complicated dataset among all
because of its larger size and larger number of rules. Due to its specificity,
there are almost no rule matches on the test set, implying that the simple
majority baseline has scores close to $0$. The training with DistillBert
improves the results, which, however, remains considerably worse than the ones
discussed above. Similar to the IMDb dataset, Snorkel denoising could not be
performed on this dataset due to a large number of rules. Though in both cases
we could calculate the results using the reduced number of rules, we want to
avoid any data manipulations. The original k-NN algorithm could not be
performed because the computation of distances between almost 2 millions
instances, which are necessary to determine the nearest neighbors, turned out
to be extremely memory- and time-consuming and could not be finished in 3 days
on a machine with CPU frequency of 2.2GHz with 40 cores 333parallel
computation was only exploited for building the index and not for querying it
for neighbors. In contrast, WSCrossWeigh denoising method, being not rule-
dependent, is still able to improve the results.

##  7 Conclusion

This work introduces the Knowledge-Infused Deep Learning Framework _Knodle_.
It provides a unified interface to work with multiple weak labeling sources,
so that they can be seamlessly integrated with the training of deep neural
networks. This is achieved by a tensor-based input format and a intuitive
separation of weak supervision aspects and model training. Apart from that,
_Knodle_ ’s modular approach makes it easy to add new data sets and denoising
algorithms. The framework facilitates experimentation that helps researchers
to derive better insights into the general correspondence between methods and
weak supervision settings. From a practical perspective, _Knodle_ can be used
to compare different denoising methods and select the one that gives the best
result for a specific task. Adding functionality to _Knodle_ is
straightforward. We hope that this will encourage researchers to create their
own algorithms to improve learning with weakly annotated data, and incorporate
them to the _Knodle_ framework.

## Acknowledgments

This research was funded by the WWTF through the project “Knowledge-infused
Deep Learning for Natural Language Processing” (WWTF Vienna Research Group
VRG19-008), by the Deutsche Forschungsgemeinschaft (DFG, German Research
Foundation) - RO 5127/2-1, and further supported by a gift from Diffbot
444https://www.diffbot.com/.

## References

  * Abadi et al. (2015) Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2015.  TensorFlow: Large-scale machine learning on heterogeneous systems.  Software available from tensorflow.org.
  * Agichtein and Gravano (2000) Eugene Agichtein and Luis Gravano. 2000.  Snowball: Extracting relations from large plain-text collections.  In _Proceedings of the Fifth ACM Conference on Digital Libraries_ , DL ’00, page 85–94, New York, NY, USA. Association for Computing Machinery.
  * Alberto et al. (2015) T C Alberto, J V Lochter, and T A Almeida. 2015.  TubeSpam: Comment Spam Filtering on YouTube.  In _2015 IEEE 14th International Conference on Machine Learning and Applications (ICMLA)_ , pages 138–143.
  * Bach et al. (2017) Stephen H Bach, Bryan Dawei He, Alexander Ratner, and Christopher Ré. 2017\.  Learning the Structure of Generative Models without Labeled Data.  _CoRR_ , abs/1703.0.
  * Boland and Krüger (2019) K. Boland and F. Krüger. 2019.  Distant supervision for silver label generation of software mentions in social scientific publications.  In _BIRNDL@SIGIR_.
  * Chatterjee et al. (2019) Oishik Chatterjee, Ganesh Ramakrishnan, and Sunita Sarawagi. 2019.  Data Programming using Continuous and Quality-Guided Labeling Functions.  _CoRR_ , abs/1911.0.
  * Corney et al. (2016) D. Corney, M. Albakour, Miguel Martinez-Alvarez, and Samir Moussa. 2016.  What do a million news articles look like?  In _NewsIR@ECIR_.
  * Craven and Kumlien (1999) Mark Craven and Johan Kumlien. 1999.  Constructing biological knowledge bases by extracting information from text sources.  In _Proceedings of the Seventh International Conference on Intelligent Systems for Molecular Biology_ , page 77–86. AAAI Press.
  * Dawid and Skene (1979) A. P. Dawid and A. M. Skene. 1979.  Maximum likelihood estimation of observer error-rates using the em algorithm.  _Journal of the Royal Statistical Society. Series C (Applied Statistics)_ , 28(1):20–28.
  * Eskin (2000) Eleazar Eskin. 2000.  Detecting errors within a corpus using anomaly detection.  In _1st Meeting of the North American Chapter of the Association for Computational Linguistics_.
  * Falcon (2019) WA Falcon. 2019.  Pytorch lightning.  _GitHub. Note: https://github.com/PyTorchLightning/pytorch-lightning Cited by_ , 3.
  * Google (2014) Google. 2014.  Freebase data dumps.  https://developers.google.com/freebase/data.
  * He et al. (2020) Zhengqiu He, Wenliang Chen, Yuyi Wang, Wei Zhang, Guanchun Wang, and Min Zhang. 2020\.  Improving neural relation extraction with positive and unlabeled learning.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 34:7927–7934.
  * Hedderich et al. (2021) Michael A. Hedderich, Dawei Zhu, and Dietrich Klakow. 2021.  Analysing the noise model error for realistic noisy label data.
  * Hoffmann et al. (2011) Raphael Hoffmann, Congle Zhang, Xiao Ling, Luke Zettlemoyer, and Daniel S. Weld. 2011.  Knowledge-based weak supervision for information extraction of overlapping relations.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_ , pages 541–550, Portland, Oregon, USA. Association for Computational Linguistics.
  * Hovy et al. (2013) Dirk Hovy, Taylor Berg-Kirkpatrick, Ashish Vaswani, and Eduard Hovy. 2013.  Learning whom to trust with MACE.  In _Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1120–1130, Atlanta, Georgia. Association for Computational Linguistics.
  * Hu and Liu (2004) Minqing Hu and Bing Liu. 2004.  Mining and Summarizing Customer Reviews.  In _Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , KDD ’04, pages 168–177, New York, NY, USA. Association for Computing Machinery.
  * Ipeirotis et al. (2010) Panos Ipeirotis, Foster Provost, and Jing Wang. 2010.  Quality management on amazon mechanical turk.  _In:Proceedings of the ACM SIGKDD Workshop on Human Computation_.
  * Jat et al. (2018) Sharmistha Jat, Siddhesh Khandelwal, and Partha Talukdar. 2018.  Improving distantly supervised relation extraction using word and entity based attention.
  * Keith et al. (2017) Katherine A. Keith, Abram Handler, Michael Pinkham, Cara Magliozzi, Joshua McDuffie, and Brendan O’Connor. 2017.  Identifying civilians killed by police with distantly supervised entity-event extraction.  _CoRR_ , abs/1707.07086.
  * Kosinski et al. (2012) Michal Kosinski, Yoram Bachrach, Gjergji Kasneci, Jurgen Van-Gael, and Thore Graepel. 2012.  Crowd iq: Measuring the intelligence of crowdsourcing platforms.  _Proceedings of the 3rd Annual ACM Web Science Conference, WebSci’12_.
  * Kozareva et al. (2008) Zornitsa Kozareva, Ellen Riloff, and Eduard Hovy. 2008.  Semantic class learning from the web with hyponym pattern linkage graphs.  In _Proceedings of ACL-08: HLT_ , pages 1048–1056, Columbus, Ohio. Association for Computational Linguistics.
  * Krasakis et al. (2019) Antonios Minas Krasakis, E. Kanoulas, and G. Tsatsaronis. 2019.  Semi-supervised ensemble learning with weak supervision for biomedical relationship extraction.  In _AKBC_.
  * Lehmann et al. (2014) Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick Van Kleef, Sören Auer, and Christian Bizer. 2014.  Dbpedia - a large-scale, multilingual knowledge base extracted from wikipedia.  _Semantic Web Journal_ , 6.
  * Lin et al. (2016) Yankai Lin, Shiqi Shen, Zhiyuan Liu, Huanbo Luan, and Maosong Sun. 2016.  Neural relation extraction with selective attention over instances.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2124–2133, Berlin, Germany. Association for Computational Linguistics.
  * Maheshwari et al. (2020) Ayush Maheshwari, Oishik Chatterjee, KrishnaTeja Killamsetty, Rishabh Iyer, and Ganesh Ramakrishnan. 2020.  Data Programming using Semi-Supervision and Subset Selection.
  * Mintz et al. (2009) Mike Mintz, Steven Bills, Rion Snow, and Daniel Jurafsky. 2009.  Distant supervision for relation extraction without labeled data.  In _Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP_ , pages 1003–1011, Suntec, Singapore. Association for Computational Linguistics.
  * Northcutt et al. (2021) Curtis G Northcutt, Lu Jiang, and Isaac L Chuang. 2021.  Confident Learning: Estimating Uncertainty in Dataset Labels.
  * Paszke et al. (2017) Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017.  Automatic differentiation in pytorch.  _NIPS Workshop_.
  * Patrini et al. (2017) Giorgio Patrini, Alessandro Rozza, Aditya Menon, Richard Nock, and Lizhen Qu. 2017\.  Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approach.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014.  Glove: Global vectors for word representation.  In _Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1532–1543.
  * Qin et al. (2018) Pengda Qin, Weiran Xu, and William Yang Wang. 2018.  Robust distant supervision relation extraction via deep reinforcement learning.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2137–2147, Melbourne, Australia. Association for Computational Linguistics.
  * Ratner et al. (2017) Alexander Ratner, Stephen H Bach, Henry R Ehrenberg, Jason Alan Fries, Sen Wu, and Christopher Ré. 2017.  Snorkel: Rapid Training Data Creation with Weak Supervision.  _CoRR_ , abs/1711.1.
  * Raykar and Yu (2012) Vikas C. Raykar and Shipeng Yu. 2012.  Eliminating spammers and ranking annotators for crowdsourced labeling tasks.  _Journal of Machine Learning Research_ , 13(16):491–518.
  * Riedel et al. (2010) Sebastian Riedel, Limin Yao, and Andrew McCallum. 2010.  Modeling relations and their mentions without labeled text.  In _Proceedings of the 2010 European Conference on Machine Learning and Knowledge Discovery in Databases: Part III_ , ECML PKDD’10, page 148–163, Berlin, Heidelberg. Springer-Verlag.
  * Riedel et al. (2013) Sebastian Riedel, Limin Yao, Andrew McCallum, and Benjamin M. Marlin. 2013.  Relation extraction with matrix factorization and universal schemas.  In _Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 74–84, Atlanta, Georgia. Association for Computational Linguistics.
  * Ritter et al. (2013) Alan Ritter, Luke Zettlemoyer, Mausam, and Oren Etzioni. 2013.  Modeling missing data in distant supervision for information extraction.  _Transactions of the Association for Computational Linguistics_ , 1:367–378.
  * van Rooyen et al. (2015) Brendan van Rooyen, Aditya Krishna Menon, and Robert C Williamson. 2015.  Learning with Symmetric Label Noise: The Importance of Being Unhinged.
  * Roth (2014) Benjamin Roth. 2014.  _Effective distant supervision for end-to-end knowledge base population systems_.  Ph.D. thesis, Saarland University.
  * Roth and Klakow (2013) Benjamin Roth and Dietrich Klakow. 2013.  Feature-based models for improving the quality of noisy training data for relation extraction.  In _Proceedings of the 22nd ACM International Conference on Information and Knowledge Management_ , CIKM ’13, page 1181–1184, New York, NY, USA. Association for Computing Machinery.
  * Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019.  Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.  _ArXiv_ , abs/1910.01108.
  * Shang (2019) Yuming Shang. 2019.  Are noisy sentences useless for distant supervised relation extraction?
  * Shu et al. (2020) Kai Shu, Subhabrata Mukherjee, Guoqing Zheng, Ahmed Hassan, Milad Shokouhi, and Susan Dumais. 2020.  Learning with weak supervision for email intent detection.  pages 1051–1060.
  * Snorkel (2020a) Snorkel. 2020a.  Detecting spouse mentions in sentences.  Last accessed 25 February 2021.
  * Snorkel (2020b) Snorkel. 2020b.  Snorkel intro tutorial: Data labeling.  Last accessed 25 February 2021.
  * Snow et al. (2004) Rion Snow, Daniel Jurafsky, and Andrew Ng. 2004.  Learning syntactic patterns for automatic hypernym discovery.  In _NIPS_ , volume 17.
  * Sterckx et al. (2014) Lucas Sterckx, Thomas Demeester, Johannes Deleu, and Chris Develder. 2014.  Using active learning and semantic clustering for noise reduction in distant supervision.  In _NIPS 2014_.
  * Sukhbaatar et al. (2015) Sainbayar Sukhbaatar, Joan Bruna, Manohar Paluri, Lubomir Bourdev, and Rob Fergus. 2015.  Training Convolutional Networks with Noisy Labels.
  * Sun and Grishman (2012) Ang Sun and Ralph Grishman. 2012.  Active learning for relation type extension with local and global data views.  In _Proceedings of the 21st ACM International Conference on Information and Knowledge Management_ , CIKM ’12, page 1105–1112, New York, NY, USA. Association for Computing Machinery.
  * Surdeanu (2013) M. Surdeanu. 2013.  Overview of the tac2013 knowledge base population evaluation: English slot filling and temporal slot filling.  _Theory and Applications of Categories_.
  * Takamatsu et al. (2012) Shingo Takamatsu, Issei Sato, and Hiroshi Nakagawa. 2012.  Reducing wrong labels in distant supervision for relation extraction.  In _Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 721–729, Jeju Island, Korea. Association for Computational Linguistics.
  * Thomas et al. (2011) Philippe Thomas, Illés Solt, Roman Klinger, and Ulf Leser. 2011.  Learning protein–protein interaction extraction using distant supervision.  In _Proceedings of Workshop on Robust Unsupervised and Semisupervised Methods in Natural Language Processing_ , pages 25–32, Hissar, Bulgaria. Association for Computational Linguistics.
  * Tratz and Hovy (2010) Stephen Tratz and Eduard Hovy. 2010.  A taxonomy, dataset, and classifier for automatic noun compound interpretation.  In _Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics_ , pages 678–687.
  * Varma et al. (2019) Paroma Varma, Frederic Sala, Ann He, Alexander Ratner, and Christopher Ré. 2019.  Learning Dependency Structures for Weak Supervision Models.
  * Wang et al. (2019) Zihan Wang, Jingbo Shang, Liyuan Liu, Lihao Lu, Jiacheng Liu, and Jiawei Han. 2019\.  CrossWeigh: Training named entity tagger from imperfect annotations.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 5154–5163, Hong Kong, China. Association for Computational Linguistics.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M Rush. 2020.  HuggingFace’s Transformers: State-of-the-art Natural Language Processing.
  * Wu and Weld (2007) Fei Wu and Daniel S. Weld. 2007.  Autonomously semantifying wikipedia.  In _CIKM_ , page 41– 50. ACM.
  * Xu et al. (2013) Wei Xu, Raphael Hoffmann, Le Zhao, and Ralph Grishman. 2013.  Filling knowledge base gaps for distant supervision of relation extraction.  In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 665–670, Sofia, Bulgaria. Association for Computational Linguistics.
  * Yao et al. (2011) Limin Yao, Aria Haghighi, Sebastian Riedel, and Andrew McCallum. 2011.  Structured relation discovery using generative models.  In _Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing_ , pages 1456–1466, Edinburgh, Scotland, UK. Association for Computational Linguistics.
  * Zhang et al. (2017) Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D. Manning. 2017.  Position-aware attention and supervised data improve slot filling.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP 2017)_ , pages 35–45.
  * Zhang and Sabuncu (2018) Zhilu Zhang and Mert R Sabuncu. 2018.  Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels.