Copyright for this paper by its authors. Use permitted under Creative Commons
License Attribution 4.0 International (CC BY 4.0).

De-Factify 3.0: Workshop on Multimodal Fact Checking and Hate Speech
Detection, co-located with AAAI 2024

<EMAIL_ADDRESS>]<EMAIL_ADDRESS>url=https://staff.germanistik.rub.de/digitale-forensische-linguistik/ ]

# Ontology Enhanced Claim Detection

Zehra Melce Hüsünbeyi  Tatjana Scheffler  Digital Forensic Linguistics, Ruhr-
Universität-Bochum, Universitätsstraße 150, 44801 Bochum, Germany

(2023)

###### Abstract

We propose an ontology enhanced model for sentence based claim detection. We
fused ontology embeddings from a knowledge base with BERT sentence embeddings
to perform claim detection for the ClaimBuster and the NewsClaims datasets.
Our ontology enhanced approach showed the best results with these small-sized
unbalanced datasets, compared to other statistical and neural machine learning
models. The experiments demonstrate that adding domain specific features
(either trained word embeddings or knowledge graph metadata) can improve
traditional ML methods. In addition, adding domain knowledge in the form of
ontology embeddings helps avoid the bias encountered in neural network based
models, for example the pure BERT model bias towards larger classes in our
small corpus.

###### keywords:

claim detection ontology knowledge base neural ML models fact checking

##  1 Introduction

With the enormous increase in internet resources, the verification of
information on the internet has gained importance. On the one hand,
controlling the dissemination of (false) information manually is not
sustainable, because it requires too much human work at large scales. On the
other hand, fully automatic methods based on text alone lack domain knowledge
and may introduce erroneous decisions. In this context, knowledge-based
approaches have been studied, particularly on data mining, verification, and
representation of factual knowledge from the web. We utilize this approach to
address claim identification as a sentence classification problem, the first
step in order to automate disinformation detection and assist fact checking
organizations, based on both content and the larger context. For this purpose,
we fused the modalities of textual information and knowledge based
representations.

Achieving a common understanding and terminology has become an essential
difficulty in the claim identification process. Identifying and defining the
fact-check worthiness of a claim is influenced by the subjectivity of the
problem, the background of the annotator, the context of expression in
political discourse, and the inherent bias of the fact-checking organization
[1]. In the past, the notion of a relevant “claim” has been operationalized in
different ways, for example, whether a statement is in the public interest, is
a rumor or not, or is verifiable or not. In this paper, we studied on the
ClaimBuster [2] and NewsClaims [3] datasets. ClaimBuster data uses the concept
of check-worthiness and in the NewsClaims dataset, researchers extend this
approach with including claim related attributes, (i.e., object of the claim).

A specific challenge for machine learning approaches to fact-checking and
claim identification is the lack of realistic data sets for many languages.
While disinformation is an international problem relevant to a range of
languages, high quality, large-scale data sets for languages other than
English are almost nonexistent. When they exist, they are very small, leading
to gaps in available information and poor results. In addition to pure textual
features, we therefore propose to combine information obtained from knowledge
bases and ontologies into the model. We compared statistical machine learning
and neural network based methods, and additionally integrated structured web
resources and linguistic features to assess their value in this domain. The
meta-features presented in the ClaimBuster dataset were embedded using the
Translating Embeddings for Modeling Multi-relational Data (TransE) approach
[4]. In addition, ClaimsKG data [5], which is a structured database for
claims, was scraped and an OWL ontology [6] was created to be used in the
claim detection task. The embedded OWL ontology containing the ClaimReview
data of the web111https://toolbox.google.com/factcheck/apis was used to encode
the ClaimBuster and NewsClaims datasets. These were fine-tuned with the BERT
model [7] as a contextual embedding method. We show that an ontology enhanced
conceptual model can be successful in addressing a challenging task that
includes implicit meanings. In our experiments, models incorporating some
domain information (notably, ontological resources) perform better than pure
NLP models.

Our contributions can be summarized as follows: We create an OWL ontology
based on the ClaimsKG dataset of claims from fact-checking organizations. We
systematically explore the performance of statistical and neural machine
learning models for claim identification, using a variety of lexical,
linguistic, and ontological features. We propose an improved approach for
fusing domain-specific features such as the OWL ontology into neural methods.
Finally, we demonstrate the effectiveness of ontology information in claim
identification in small datasets.

##  2 Related Work

Recent research on the claim identification task can be mainly categorized as
feature engineering based approaches and neural network based ones. In the
literature, statistical machine learning classifiers are commonly combined
with surface level features such as named entities, verbal forms, bag-of-words
feature, and part-of-speech (POS) tags. Zuo et al. [8] offered lexical,
stylometric and semantic features for political debates and speeches in a
hybrid approach by combining rules and supervised learning approaches and they
achieved promising results. The most discriminative features presented by
Hassan et al. [9] were POS tags and entity types along with an SVM classifier.
Another approach that combined sentence embeddings (i.e., InferSent
embeddings) and POS tags with named entities in a Logistic Regression
classifier achieved better results with universal sentence representations
than the word-level representations [10].

Neural network-based models have also been used to address the challenge of
automatically detecting claims. Jimenez and Li [11] provided an enhanced
version of the ClaimBuster dataset by adding random sequences of text that
included mostly numbers to increase robustness of the model and learn from
negative instances. They proposed an architecture consisting of convolutional
and pooling layers followed by a Bi-directional LSTM network by taking
advantage of pre-trained word embeddings. Experiment results showed different
combinations of word embeddings did not improve the models significantly and
the addition of nonsensical data has no negative impact on the regular
dataset. In another approach, a neural check-worthiness sentence ranking model
was offered by performing two distinct word encodings such as one hot encoded
syntactic dependencies of words and trained word embeddings using the Word2Vec
skip-gram model [12]. These input representations were fed to a RNN based
neural network model with gated recurrent units and aggregated outputs
followed by an attention mechanism. The researchers stated that they made the
first contribution to examining the effect of word embeddings trained on a
domain-specific dataset for this task. Their performance scores achieved large
improvements with domain-specific embeddings and syntactic dependency parsing.
[13] performed BERT and gradient-based adversarial training on the binary
version of ClaimBuster dataset. To the best of our knowledge, the most recent
work on the ClaimBuster dataset including both crowdsourced and groundtruth
parts by considering checkworthiness and the three class distribution has been
conducted by Jha et al. [14]. They achieved best performance with Glove
embedding and RNN based approach.

Existing models were applied to the claim detection problem with different
conceptualizations and different datasets. Even though it is difficult to
establish what the state-of-the-art model for the claim detection task is, we
can conclude that domain-specific feature enhancement is a significant
approach for implementing promising models. As stated in the survey study
[15], ontologies provide an accurate representation of the problem domain with
a vast amount of information from the web and a rich set of axioms to link
pieces of information. To this end, Tchechmedjiev et al. [5] published a
knowledge graph of fact-checked claims, ClaimsKG, including English claims and
related metadata such as their labels, authors, dates, debunked news, keywords
extracted from the claim body and from the review body together with their
Wikipedia categories, and references cited in the claim reviews. We aim to
take advantage of this graph-structured data to create a claim ontology and
obtain its vector representation to enhance the neural network-based model. To
the best of our knowledge, ours is the first study combining ontology
embeddings and BERT to identify check-worthy claims.

##  3 Data and task

#### ClaimBuster

is a publicly available dataset created to address the claim identification
task, which is an understudied problem, addressing the concept of
checkworthiness [2]. This corpus includes sentence-level annotations of claims
from U.S. presidential debates, from 1960 to 2016. Each sentence has been
marked with one of three categories: non-factual statement (NFS), unimportant
factual statement (UFS), and check-worthy factual statement (CFS). The
sentences belonging to the CFS class include facts that the public would like
to know about their validity. Sentences containing facts where the public
doesn’t question their validity belong to the UFS class. The NFS class
consists of sentences that do not contain any factual assertions, such as
subjective sentences and interrogative sentences.

We used both the ground-truth corpus containing 1032 sentences annotated by
three experts, and the crowdsourced corpus with 22,501 sentences that were
labeled by 2 or 3 participants. For small data experiments, one-third of the
ground-truth dataset was allocated as the test set throughout all experiments.
Thus, the effect of feature sets in small and content-dependent datasets on
neural network-based and statistical ML approaches can be examined using this
corpus. As continuation of the study, the models were trained on the
crowdsourced part of the dataset and tested with the groundtruth part.

#### NewsClaims

was published as a new evaluation benchmark dataset for the claim detection
task. This dataset consists of news articles related to COVID-19 in the LDC
corpus LDC2021E11. 889 claims from 143 news articles were annotated by 3
annotators in terms of claim validity as well as other attributes such as
claim topic, claim object, span, claimer, and stance. As stated by the
researchers, NewsClaims is suitable for the claim identification task in a low
resource setting due to the lack of previous data and urgency of the
situation.

For claim identification, we extracted sentences labeled as factually
verifiable and considered the remaining sentences of the articles as counter
instances. To decrease the noise of the dataset, duplications, URLs, citations
and sentences including less than 4 tokens were excluded. We used one third of
the 5157 binary labeled sentences as test data in our proposed models for the
sake of consistency.

Table 1 shows the class imbalance in the corpora as is observed in almost all
real-life datasets.

ClaimBuster | ‘groundtruth’ | ‘crowdsourced’ | NewsClaims
---|---|---|---
assigned label | sent. | pct. | sent. | pct. | assigned label | sent. | pct.
Non-factual Sentence (NFS) | 731 | 70.83 | 15416 | 65.51 | Factually Verifiable Claims (FVC) | 655 | 12.70
Unimportant Factual Sentence (UFS) | 63 | 6.11 | 2466 | 10.48 | non-Factually Verifiable Claims | 4502 | 87.29
Check-worthy Factual Sentence (CFS) | 238 | 23.06 | 5651 | 24.01 | (non-FVC) |  |

Table 1: Class distribution in datasets

##  4 Methodology

We tested two kinds of models and various feature sets on the task. The first
model type are statistical machine learning methods, which have shown good
results for textual classification tasks, particularly on small datasets.
Here, we used lexical features, specific dictionaries, as well as several
kinds of word embeddings and encodings of the claim metadata. The second model
type are neural machine learning algorithms, which we tested with a BERT model
and with ontology embeddings (as well as in combination).

###  4.1 Vectorization in Statistical Machine Learning Methods

The most common and successful algorithms building statistical machine
learning approaches for text classification problems are Support Vector
Machines and Logistic Regression. Vectorization is a crucial step to get
distinct features out of the text for these models to train on. We extracted
naive binary term occurrence features (i.e. term frequency–inverse document
frequency, TF-IDF) and enhanced them with Linguistic Inquiry and Word Count
(LIWC) [16] and linguistic features. Also, word embeddings and encoded
metadata have been used as features for these statistical ML classifiers.

#### LIWC and other linguistic features

As a baseline feature set, we used word token unigrams and bigrams along with
TF-IDF. In addition, we combined LIWC dictionary-based quantitative text
analysis features with the n-grams. LIWC captures psychological processes and
content-related aspects (e.g., positive or negative emotions, cognitive
processes, social processes). We also included several linguistic features
such as word count, character count, punctuation count, text complexity and
sentiment analysis scores.

#### Word2Vec for Feature Extraction

We used a common word vector model, Word2Vec [17], to take into account the
semantic relatedness of tokens. There is a publicly available pre-trained
Word2Vec model trained on the English CoNLL17 corpus with the skip-gram
method222http://vectors.nlpl.eu/repository/. In addition to this domain-free
word embedding approach, we trained domain-specific word vectors on the
training data via the gensim library. This takes preprocessed data in the form
of tokenized texts as input and produces a model as output. Text sequences
from word embeddings have been created with vector aggregation. For this
purpose, we used hash maps including words and their occurrences with defining
certain length for each sequence. If the length of a sequence is shorter than
the maximum length, we added zero padding, otherwise this sequence was trimmed
from the end.

#### Knowledge Graphs for Encoding of Metadata

Knowledge Graphs (KG) evolved as a technique for semantically structuring
knowledge about real-world entities in a machine-readable format [18]. They
are directed labeled graphs consisting of nodes, edges, and labels. Our
motivation for employing KG structure is related to the challenging task, and
to the domain dependence of fact verification. Some of the metadata already
existing in the dataset constitutes nodes in the knowledge graph structure:
Thus, the ‘speaker’ is the name of the person who made the claim, ‘speaker
title’ is the speaker’s occupation at the time of the debate, ‘speaker party’
is the political affiliation of the speaker, as well as the number of words in
the text and the sentiment of the text. Additionally, we used the TagMe entity
linking annotation API [19] to find the highest-scored entities in each claim.
By encoding patterns between nodes, relations are created. After graph
construction, we continued with graph embedding to obtain a low-dimensional
representation of the feature space. The TransE model [4] was used to embed
the entities and relations by treating each relation as a transformation
operation from the head entity to the tail entity (labeled with the relation).

###  4.2 Neural Network Based Approaches

Neural network approaches based on sequence or graph modeling provide an
opportunity for high performance in various NLP tasks. Claim-related world
knowledge was encoded using the advantage of the graph structure. The scores
for our task were obtained by using the pretrained domain free language models
Word2Vec and BERT. In order to achieve more successful results in a domain-
specific task, we fused the BERT model with graph-embedded ontology entities.

#### Word2Vec Embedding

As we mentioned in the previous section, publicly available pre-trained
Word2Vec embeddings have been used. In this approach, the collection of
embeddings was used to seed the embedding layer with the word embedding
weights. These weights have been passed to a fully connected layer with 500
hidden units and the ReLU activation function. Then, the softmax activation
function was performed for obtaining predictions.

#### OWL2Vec* Ontology Embedding

ClaimsKG is a structured database based on a knowledge graph model which
includes 53,521 English claims published between the years 1996–2022 from 14
fact-checking websites and was extended with DBpedia entities and schema.org
and NIF vocabularies [5]. We scraped relevant data from this database with the
following features: claim bodies, normalized labels for claims, links for
debunked news, the claim review body and its title, references cited in the
claim review, the entities and their Wikipedia categories extracted from the
claim body and from the review body, author of the claim review, the claim
publication date. During data harvesting, non-English and corrupted data
samples were not included.

Different kinds of domain knowledge can be represented in an OWL ontology in
structured form with an Abox part (classes, objects), a Tbox part (instances,
facts) and its own syntax (W3C OWL, RDF, RDFS) [6]. We created our own OWL
ontology based on the scraped claim data and its metadata to obtain encoded
claim-related tokens. We implemented a new OWL ontology by following the data
model and instances of ClaimsKG.

For embedding the OWL ontologies, we used the OWL2Vec framework [20]. It aims
to represent each OWL named entity with a vector so that entity-related
information is retained in vector space. Chen et al. [20] emphasized that this
approach has several advantages compared to other ontology embedding models
such as combinations of the semantics of the graph structure, lexical
information and logic constructors, different graph walking strategies,
ontology entailment reasoning, and word embedding pre-training. Also, results
of researchers’ case studies showed the impact of those features with
achieving significantly better performance than state-of-the-art ontology
embeddings (Onto2Vec, OPA2Vec), and knowledge graph embeddings (RDF2Vec,
TransE). As in the previous model, we processed word vectors through a fully
connected layer with 500 hidden units, and the Rectified Linear Unit (ReLU)
activation function is applied. Lastly, predictions were generated using the
softmax activation function.

#### BERT model

We use the Transformer-based BERT model, which offers state-of-the-art
solutions to numerous NLP problems. With its structure that bidirectionally
processes the incoming text and combines the masked language and next-sentence
prediction models, BERT is able to account for the context of a token. There
is a publicly available English language model provided by Google, that was
trained on several English corpora such as the BookCorpus (a dataset
consisting of 11,038 unpublished books) and English Wikipedia with 12
transformer blocks. We fine-tuned this uncased BERT model for the claim
identification task. The recommended hyperparameters by Devlin et al. [7] were
examined using the Adam optimizer and batch-size and common learning rate were
chosen as 16 and 2e-5, respectively.

#### Multimodal Fusion of BERT Model and Ontology Embeddings

The last hidden state of the CLS token serves as a good representation of
input sequences. As shown in Figure 1, we concatenated the CLS embedding with
the corresponding ontology embeddings for each sequence vertically. These
concatenated vectors go through the linear and dropout layer, and we use
BCEWithLogitsLoss to produce scores for each label.

Figure 1: Proposed model fusing textual and ontology features

##  5 Experiments and Results

###  5.1 ClaimBuster dataset

#### Small data experiments

Our main experiment uses the small groundtruth data of ClaimBuster for both
training and testing as specified above.

For the sake of completeness and to examine the power of statistical ML
models, especially in small and domain-specific datasets, we chose SVM and
Logistic Regression classifiers with TF-IDF weighted word n-grams features as
a baseline approach. According to our results shown in Table 2, Logistic
Regression obtains the highest scores in all metrics except the model that
includes linguistic and LIWC features. The highest accuracy and weighted
average F1-scores were obtained with the model leveraging knowledge graph
embeddings. Also, instead of using pretrained embeddings, training Word2Vec
embeddings on domain data reached the second-highest scores, despite the small
size of the corpus. It can be seen that adding domain-specific features in the
form of external knowledge or domain specific textual embeddings is useful for
a context-based task such as claim identification.

Our experimental results of the neural network based approaches are presented
in Table 3. [14] proposed G2CW framework that relies on Glove embeddings and
gated recurrent units (GRU) that is the latest study on the ClaimBuster
dataset as stated in Section 2. We replicated this model using the
architecture and hyperparameters they specified, and applied it to the
ClaimBuster groundtruth dataset.

Our BERT model, which includes 12 transformer layers, achieved scores
interestingly close to the fully connected, shallow neural network, GRU-based
model, and statistical machine learning models for our task. The class-based
results show that the BERT model did not produce any results for the UFS
label, which has very few instances. The BERT model was biased for the class
containing the largest number of samples in this small-sized and unbalanced
dataset, and failed on the multiclass classification problem. On the other
hand, the addition of ontology vectorization for corresponding text sequences
to the BERT model (“BERT + Ontology Embeddings”) improved the accuracy by 0.06
points and macro-average F1-score by 0.19 points. Thus, our best model
exhibits a macro-average F1-score of 0.74 and an accuracy of 0.92 on the
sentence-level claim identification task.

| SVM | Logistic Regression
---|---|---
ClaimBuster dataset (groundtruth) | P | R | F1 | Acc. | P | R | F1 | Acc.
word (1,2)-gram + tf-idf | macro avg | 0.58 | 0.55 | 0.56 | 0.81 | 0.56 | 0.49 | 0.50 | 0.83
wt. avg | 0.80 | 0.81 | 0.80 | 0.79 | 0.83 | 0.79
ling features + LIWC features | macro avg | 0.61 | 0.59 | 0.59 | 0.83 | 0.53 | 0.51 | 0.52 | 0.82
wt. avg | 0.82 | 0.83 | 0.82 | 0.78 | 0.82 | 0.80
pretrained Word2Vec features | macro avg | 0.65 | 0.63 | 0.64 | 0.81 | 0.71 | 0.63 | 0.66 | 0.82
wt. avg | 0.81 | 0.81 | 0.81 | 0.81 | 0.82 | 0.82
training Word2Vec features on domain data | macro avg | 0.69 | 0.69 | 0.69 | 0.83 | 0.74 | 0.66 | 0.69 | 0.84
wt. avg | 0.82 | 0.83 | 0.82 | 0.83 | 0.84 | 0.83
Knowledge Graph emb. of metadata + Word2Vec | macro avg | 0.68 | 0.69 | 0.68 | 0.84 | 0.69 | 0.67 | 0.68 | 0.85
wt. avg | 0.84 | 0.84 | 0.84 | 0.85 | 0.85 | 0.85
Table 2: Evaluation scores of the statistical machine learning based methods ClaimBuster dataset (groundtruth) | Precision | Recall | F1 | Accuracy
---|---|---|---|---
Word2Vec Embeddings + FCN | macro avg | 0.69 | 0.62 | 0.64 | 0.82
weighted avg | 0.80 | 0.81 | 0.81
Ontology Embeddings + FCN | macro avg | 0.73 | 0.66 | 0.69 | 0.85
weighted avg | 0.84 | 0.84 | 0.84
Glove Embeddings + GRU [14] | macro avg | 0.62 | 0.63 | 0.62 | 0.86
weighted avg | 0.86 | 0.87 | 0.86
BERT model | macro avg | 0.53 | 0.60 | 0.55 | 0.86
weighted avg | 0.94 | 0.86 | 0.89
BERT + Ontology Embeddings | macro avg | 0.70 | 0.93 | 0.74 | 0.92
weighted avg | 0.94 | 0.92 | 0.93
ClaimBuster dataset (crowdsourced and groundtruth) |  |  |  |
BERT model | macro avg | 0.91 | 0.95 | 0.92 | 0.97
weighted avg | 0.97 | 0.97 | 0.97
BERT + Ontology Embeddings | macro avg | 0.91 | 0.94 | 0.92 | 0.97
weighted avg | 0.97 | 0.97 | 0.97
NewsClaims dataset |  |  |  |
BERT model | macro avg | 0.44 | 0.5 | 0.47 | 0.87
weighted avg | 0.76 | 0.87 | 0.81
BERT + Ontology Embeddings | macro avg | 0.83 | 0.59 | 0.62 | 0.89
weighted avg | 0.88 | 0.89 | 0.86
Table 3: Evaluation scores of neural network based approaches

#### Large data experiments

The large crowdsourced part of the ClaimBuster dataset as the training data
and the groundtruth part as the test data were used for comparison. We
experimented with our best two models, the fine-tuned BERT model and the
ontology enhanced BERT model. We performed the experiments with the
hyperparameters that we used in the previous BERT-based models (i.e. Adam
optimizer, learning rate 2e-5, batch-size 16). These two models obtained
almost identical results and reached an accuracy of 0.97, and macro-average
F1-score of 0.97, also shown in Table 3. It can be seen that a large annotated
in-domain dataset for training can compensate for the disadvantages of the
plain BERT model, or the reverse, that external domain knowledge in the shape
of ontologies can be leveraged to compensate for a small training data set,
when combined with the textual information.

###  5.2 NewsClaims dataset

To examine the effect of integrating ontology information in the claim
identification task for a small-sized dataset, we applied our proposed
ontology leveraged BERT model and the fine-tuned BERT model to the NewsClaims
data by following the same hyperparameter set. The best results for this
dataset, which requires in-depth understanding, were obtained by the claim
ontology enhanced BERT model with an accuracy score of 0.89 and a macro
average score of 0.62. This result is compatible with the ClaimBuster
groundtruth dataset with similar features.

##  6 Discussion and Conclusion

In this study, we approached the claim detection problem by incorporating
domain knowledge along with linguistic features in different machine learning
models. The effect of different feature sets on the small-size datasets was
examined. While graph-structured features can capture background knowledge,
text can capture context. We show that both modalities of information must be
combined as they can enrich each other. Ontology models have been shown to be
appropriate for the detection of factual check-worthiness, an implicit and
context-dependent problem. These models benefit from the ability to link
together pieces of information that enable automated reasoning to extract
information that has not been explicitly mentioned before.

We encoded fact-check related data from an internet knowledge base and adapted
it to our problem by converting it into an ontology and embedding the
resulting ontology. Our proposed ontology enhanced claim detection model
achieved the highest scores in all metrics for the groundtruth part of the
ClaimBuster and NewsClaims datasets. This model also proved that adding
ontology features can avoid the bias encountered in neural network based
models, for example the pure BERT model’s bias towards larger classes. Besides
this task, the embedded fake news-related ontology data structure can also be
effective in various fact-checking problems, with its structured data shape,
including the relationship between entity and news texts, and the Wikipedia
data it contains.

Our conducted experiments also show that domain-specific features like a
knowledge graph embedding of metadata, word embeddings trained on in-domain
data, and taking advantage of a fact check-related OWL ontology to encode
input sequences can be more effective than domain free features like
pretrained language models Word2Vec, Glove and BERT.

We further examined the effect of the data size and the addition of ontology
features to a large-size dataset for the BERT model, by retraining it on the
crowdsourced part of ClaimBuster and testing on the groundtruth part. We
concluded that the domain-specific feature addition is not a very effective
solution to reach higher scores for the large-size data. But those features
can have a significant impact in the absence of an expert annotated reliable
dataset of a reasonable size, such as in the checkworthiness task for
languages other than English, where data is very sparse. In addition, domain-
specific features and graph-structured data that may achieve similar scores to
neural network models can play an important role in the development of
transparent and human-understandable machine-learning systems.

## References

  * Boland et al. [2022] K. Boland, P. Fafalios, A. Tchechmedjiev, S. Dietze, K. Todorov,  Beyond facts–a survey and conceptualisation of claims in online discourse analysis,  Semantic Web 13 (2022) 793–827.
  * Arslan et al. [2020] F. Arslan, N. Hassan, C. Li, M. Tremayne,  A benchmark dataset of check-worthy factual claims,  in: Proceedings of the International AAAI Conference on Web and Social Media, volume 14, 2020, pp. 821–829.
  * Gangi Reddy et al. [2022] R. Gangi Reddy, S. C. Chinthakindi, Z. Wang, Y. Fung, K. Conger, A. ELsayed, M. Palmer, P. Nakov, E. Hovy, K. Small, H. Ji,  NewsClaims: A new benchmark for claim detection from news with attribute knowledge,  in: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Abu Dhabi, United Arab Emirates, 2022, pp. 6002–6018. URL: https://aclanthology.org/2022.emnlp-main.403.
  * Bordes et al. [2013] A. Bordes, N. Usunier, A. Garcia-Duran, J. Weston, O. Yakhnenko,  Translating embeddings for modeling multi-relational data,  Advances in neural information processing systems 26 (2013).
  * Tchechmedjiev et al. [2019] A. Tchechmedjiev, P. Fafalios, K. Boland, M. Gasquet, M. Zloch, B. Zapilko, S. Dietze, K. Todorov,  Claimskg: A knowledge graph of fact-checked claims,  in: The Semantic Web–ISWC 2019: 18th International Semantic Web Conference, Auckland, New Zealand, October 26–30, 2019, Proceedings, Part II 18, Springer, 2019, pp. 309–324.
  * Antoniou and Harmelen [2009] G. Antoniou, F. v. Harmelen,  Web ontology language: Owl,  Handbook on ontologies (2009) 91–110.
  * Devlin et al. [2018] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova,  Bert: Pre-training of deep bidirectional transformers for language understanding,  arXiv preprint arXiv:1810.04805 (2018).
  * Zuo et al. [2018] C. Zuo, A. Karakas, R. Banerjee,  A hybrid recognition system for check-worthy claims using heuristics and supervised learning,  in: CEUR workshop proceedings, volume 2125, 2018.
  * Hassan et al. [2017] N. Hassan, F. Arslan, C. Li, M. Tremayne,  Toward automated fact-checking: Detecting check-worthy factual claims by claimbuster,  in: Proceedings of KDD ’17, Association for Computing Machinery, New York, NY, USA, 2017, p. 1803–1812.
  * Konstantinovskiy et al. [2021] L. Konstantinovskiy, O. Price, M. Babakar, A. Zubiaga,  Toward automated factchecking: Developing an annotation schema and benchmark for consistent automated claim detection,  Digital threats: research and practice 2 (2021) 1–16.
  * Jimenez and Li [2018] D. Jimenez, C. Li,  An empirical study on identifying sentences with salient factual statements,  in: 2018 International Joint Conference on Neural Networks (IJCNN), IEEE, 2018, pp. 1–8.
  * Hansen et al. [2019] C. Hansen, C. Hansen, S. Alstrup, J. Grue Simonsen, C. Lioma,  Neural check-worthiness ranking with weak supervision: Finding sentences for fact-checking,  in: Companion Proceedings of the 2019 World Wide Web Conference, 2019, pp. 994–1000.
  * Meng et al. [2020] K. Meng, D. Jimenez, F. Arslan, J. D. Devasier, D. Obembe, C. Li,  Gradient-based adversarial training on transformer networks for detecting check-worthy factual claims,  arXiv preprint arXiv:2002.07725 (2020).
  * Jha et al. [2023] R. Jha, E. Motwani, N. Singhal, R. Kaushal,  Towards automated check-worthy sentence detection using gated recurrent unit,  Neural Computing and Applications 35 (2023) 11337–11357.
  * Sikelis et al. [2021] K. Sikelis, G. E. Tsekouras, K. Kotis,  Ontology-based feature selection: A survey,  Future Internet 13 (2021) 158.
  * Pennebaker et al. [2001] J. W. Pennebaker, M. E. Francis, R. J. Booth,  Linguistic inquiry and word count: Liwc 2001,  Mahway: Lawrence Erlbaum Associates 71 (2001) 2001.
  * Mikolov et al. [2017] T. Mikolov, E. Grave, P. Bojanowski, C. Puhrsch, A. Joulin,  Advances in pre-training distributed word representations,  arXiv preprint arXiv:1712.09405 (2017).
  * Ehrlinger and Wöß [2016] L. Ehrlinger, W. Wöß,  Towards a definition of knowledge graphs.,  SEMANTiCS (Posters, Demos, SuCCESS) 48 (2016) 2.
  * Ferragina and Scaiella [2011] P. Ferragina, U. Scaiella,  Fast and accurate annotation of short texts with wikipedia pages,  IEEE software 29 (2011) 70–75.
  * Chen et al. [2021] J. Chen, P. Hu, E. Jimenez-Ruiz, O. M. Holter, D. Antonyrajah, I. Horrocks,  Owl2vec*: Embedding of owl ontologies,  Machine Learning 110 (2021) 1813–1845.

##  Appendix A Error Analysis

In order to investigate the impact of the ontology information in detail, we
evaluated the instances of the ClaimBuster groundtruth dataset where the two
main models (ontology enhanced versus pure BERT) disagreed in the labels;
statistics for these cases are presented in Table 4. It can be observed that
when the ontology enhanced BERT model correctly predicted the samples
belonging to the UFS class, the BERT model gave wrong results for 4 samples
belonging to this class. In the CFS class, there were 14 samples in which the
BERT model predicted incorrectly and the ontology enhanced BERT model made a
correct prediction. Contrary to this, there was no example where the BERT
model gave the correct result while the ontology enhanced BERT model was
incorrect in the UFS and CFS classes. The class in which the test results of
the models are closest to each other is NFS with the highest number of
samples.

(# of samples in the error set) | UFS | CFS | NFS
---|---|---|---
|  BERT + Ontology Emb. correct,
---
BERT model incorrect
4 | 14 | 5
|  BERT + Ontology Emb. incorrect,
---
BERT model correct
0 | 0 | 4
Table 4: Error distribution for the ClaimBuster groundtruth dataset, models
with and without ontology

We chose 6 sample sentences from the test set errors for further visual
analysis. To take a closer look at the failure of the plain BERT model to
assign the UFS class, we compare two instances from UFS (which were correctly
labelled by BERT+Ontology, but mislabelled by plain BERT) with two typical
samples from each of the CFS and NFS classes, which were correctly assigned by
both models. Tables 5–6 show these sentences with a visualization of the
degree of attention given to each token by the plain BERT model (Table 5) and
the BERT+Ontology model (Table 6). Sentence embeddings of these samples were
formed by averaging the weights from the last 4 layers for the two models. The
shades of colors represent the degree of attention and darker words have
higher weight scores. While quantity words (i.e., ‘three days’, ‘six
predecessors’) have higher weights in the BERT model, it can be seen that
central entities (i.e., ‘president’, ‘Senator Obama’, ‘Oval Office’, ‘women’s
rights’) are more important for us to decide whether a claim will be
checkworthy or not in the ontology enhanced BERT model. In the sentences that
the BERT model predicts incorrectly, the BERT model assigns a higher weight
value to the first-person pronouns. It can be considered that the use of such
features is associated with a more informal and spontaneous style of discourse
which may be important for the determination of the claim’s checkworthiness.

sample sentences from test set of ClaimBuster (groundtruth) |  true class |  BERT model prediction
---|---|---
|  • UFS • UFS • NFS • NFS • CFS • CFS |  • CFS • CFS • NFS • NFS • CFS • CFS

Table 5: Highlighted words by considering weights from the BERT model

sample sentences from test set of ClaimBuster (groundtruth) |  true class |  BERT+Ontology prediction
---|---|---
|  • UFS • UFS • NFS • NFS • CFS • CFS |  • UFS • UFS • NFS • NFS • CFS • CFS

Table 6: Highlighted words by considering weights from the ontology enhanced
BERT model