# Enhancing Visual Question Answering
through Question-Driven Image Captions as Prompts

Övgü Özdemir, Erdem Akagündüz
Department of Modeling and Simulation,
Graduate School of Informatics,
Middle East Technical University, Türkiye
<EMAIL_ADDRESS>
###### Abstract

Visual question answering (VQA) is known as an AI-complete task as it requires
understanding, reasoning, and inferring about the vision and the language
content. Over the past few years, numerous neural architectures have been
suggested for the VQA problem. However, achieving success in zero-shot VQA
remains a challenge due to its requirement for advanced generalization and
reasoning skills. This study explores the impact of incorporating image
captioning as an intermediary process within the VQA pipeline. Specifically,
we explore the efficacy of utilizing image captions instead of images and
leveraging large language models (LLMs) to establish a zero-shot setting.
Since image captioning is the most crucial step in this process, we compare
the impact of state-of-the-art image captioning models on VQA performance
across various question types in terms of structure and semantics. We propose
a straightforward and efficient question-driven image captioning approach
within this pipeline to transfer contextual information into the question-
answering (QA) model. This method involves extracting keywords from the
question, generating a caption for each image-question pair using the
keywords, and incorporating the question-driven caption into the LLM prompt.
We evaluate the efficacy of using general-purpose and question-driven image
captions in the VQA pipeline. Our study highlights the potential of employing
image captions and harnessing the capabilities of LLMs to achieve competitive
performance on GQA under the zero-shot setting. Our code is available at
https://github.com/ovguyo/captions-in-VQA.

##  1 Introduction

Visual Question Answering (VQA) is a complex multimodal task that demands a
high-level understanding of several aspects, such as object and attribute
identification, object localization, comprehension of the relationship between
the image and the question, and reasoning about the context and the scene. The
common steps of a typical VQA model involve generating embeddings of the image
and the question using encoders for each, combining the image and question
embeddings with a fusing module, and generating answers using a text generator
or a classifier. For a general overview of the VQA techniques, the reader may
refer to [34, 33].

The inherent multimodal nature of the VQA problem is the primary factor
contributing to its complexity. Combining different types of information, such
as text and images, makes the model’s training more complex, as the model must
understand and utilize the connections and interactions between these
different modalities. Several studies [18, 32, 28, 4, 15] propose an approach
to tackle multimodality for the VQA problem. However, these methods indicate
limitations in their capacity to adapt to new tasks, particularly in zero-shot
settings.

Recent advances in high-capacity large language models (LLMs) [5, 1, 36] have
marked a dramatic milestone in the domain. LLMs are predominantly trained with
millions (or billions) of parameters and utilized for processing textual data.
LLMs show outstanding performance in a variety of natural language tasks. The
ongoing research challenge lies in extending the capabilities of LLMs to the
intersection of different modalities, _e.g_., textual and visual data.
Recently, GPT-4 [1] and Gemini [36] stand out as remarkable examples of
multimodal LLMs, adept at successfully processing textual and visual
modalities for various downstream tasks, including VQA. Several alternative
approaches [19, 20, 22, 2, 7] have also been proposed in the realm of large-
scale vision-language integration. The challenge in multimodal training lies
in the extensive computational and data costs required to align the
representation spaces of vision and language.

Some recent studies [11, 37, 42] delve into the potential of utilizing image
captions with unimodal LLMs in the zero-shot VQA setting. Our study differs
from these studies in the following aspects. Firstly, we focus on examining
the representation capacity of image captions from various vision-language
models on the VQA performance. Second, our study investigates whether image
captions can be informative for specific types of questions by evaluating the
results in structurally and semantically different questions. Within this
scope, we also evaluate the influence of feeding LLMs with general-purpose and
question-driven captions, and only the most relevant sentence in the caption
during the QA stage.

Numerous VQA datasets are available in the literature, including CLEVR [17],
VQA [3], VQA 2.0 [9], OK-VQA [25], GQA [16]. Among these sets, although each
serves various purposes effectively, GQA stands out for its emphasis on
testing compositional and grounded reasoning abilities and its relatively
diverse Q/A set. In this study, we conduct our experiments on the GQA dataset
and focus on measuring performance on semantically and structurally different
questions.

We structure the VQA task into two fundamental components: image captioning
and question-answering. The goal is to leverage the respective strengths of
these tasks, aiming for a more thorough comprehension of both the visual
content and the corresponding questions. We carry out experiments with state-
of-the-art vision-language models, including CogVLM [40], BLIP-2 [20], and
FuseCap [31] to comprehend their scene representation capacity in the VQA
pipeline.

We outline our contributions as follows:

  * •

We evaluate the image captioning performance of various vision-language models
incorporating them with LLMs for zero-shot VQA, analyzing their effectiveness
across various question types.

  * •

We propose a straightforward question-driven captioning approach to better
transfer the context into LLMs for question-answering.

The rest of the paper is organized as follows. Section 2 reviews related
works. Section 3 mentions the components of the proposed pipeline. In Section
4, we present the experiments designed for our study. Section 5 discusses
evaluation results. Section 6 outlines the conclusions drawn from our findings
and discusses potential avenues for future research.

##  2 Related Literature

###  2.1 Large Language Models

LLMs [5, 27, 38] trained on extensively rich web-scale corpus usually employ
autoregressive methods to generate target tokens. LLMs demonstrate remarkable
proficiency in processing and generating text with human-like characteristics.
This attribute renders them suitable instruments for various language-related
tasks, including question-answering, text generation, machine translation,
etc. Expanding the scope of LLMs to include additional modalities results in
the creation of multimodal LLMs [20, 1, 40, 36, 22], which boosts the
performance for many downstream tasks including image captioning, visual
question answering, text-to-image synthesis.

###  2.2 Visual Question Answering

The main challenge of the VQA domain comes from bridging the gap between
visual understanding and natural language. Numerous studies have been proposed
to tackle questions related to visual content. Relation Networks [32] involves
employing a compact and straightforward neural network module that takes pairs
of features as input and generates a score indicative of the relationship
between these feature pairs. LXMERT [35] is a large-scale transformer model
that fuses textual and visual representations with a cross-modality encoder.
MDETR [18] is an end-to-end modulated detector which is an improved version of
the object detection model DETR [6] by adding the capability of processing
free-form texts. Alternatively, neuro-symbolic approaches in VQA have gained
attention to enhance model interpretability. A neuro-symbolic approach in VQA
combines two main parts: neural network modules for handling images and text
modalities, and a symbolic reasoning module for managing logic and knowledge
representation. NS-VQA [43] and NS-CL [24] use neural networks for scene
parsing and dividing questions into program instructions, and propose a
symbolic module executing the program instructions on the scene
representation. An alternative hybrid approach, ProTo [44], proposes program-
guided transformers that use semantic and structural information of the
programs being parsed from the questions by a sequence-to-sequence model. A
recent approach, namely VisProg [12], generates program instructions from
questions using LLMs and employs instructions on images benefiting from
different modules for object detection, visual question answering, image
classification, and segmentation. Recent large-scale multimodal approaches
used for VQA are mentioned in Section 1 and Section 2.1.

###  2.3 Image Captioning

Image captioning aims to produce a caption describing visual content in
natural language. Conventional approaches in image captioning are based on
attention and encoder-decoder structure [14, 41, 13]. A typical image
captioning model consists of an encoder for gathering visual cues and a
textual decoder to produce the final caption. Like VQA, this requires bridging
the gap between visual and natural language understanding. Recently, large-
scale multimodal models [26, 19, 12, 40, 1, 36, 20] have resulted in notable
enhancements in performance and demonstrated adaptability to various
downstream applications, including image captioning.

###  2.4 Question Answering

Question-answering (QA) models aim to provide contextually appropriate
responses based on a document or text, often requiring an understanding of
linguistic rules, syntax, and contextual nuances. Recent models in QA leverage
transformer architectures and large-scale pre-training on diverse datasets [8,
23, 29, 5].

Figure 1: VQA pipeline exploiting general and the proposed question-driven
(QD) image captioning as an intermediate step.

##  3 Methodology

###  3.1 Caption Generation

The primary and most crucial element in the suggested pipeline is the creation
of image captions with high visual representation capability. Image captions
provide a summarized version of the visual content, and specific visual
details may be lost, which could affect the VQA performance. We survey image
captioning models, selecting ones that provide more detailed captions while
taking into account our computational resource limitations. Consequently, we
evaluate several zero-shot vision-language models, including CogVLM [40],
FuseCap [31], and BLIP-2 OPT2.7b [20] by integrating them into the VQA
pipeline. We employ both the chat and visual grounding variants of CogVLM,
considering their potential performance impacts across different question
types. VQA performance is assessed across various image captions according to
structurally and semantically different question categories. More details
about question categories are given in Section 4.1.

Two approaches are utilized in this paper to generate captions. First, each
image is captioned without considering the questions associated with it, which
we refer to as “general-purpose captioning” throughout the paper. However,
general-purpose captions are designed to provide a broad description of the
visual content, and they may lack the precision needed to address detailed and
specific queries. Therefore, in our second approach, we create image captions
for each image-question pair, a process we refer to as “question-driven image
captioning”. For this purpose, KeyBERT [10] is employed to extract keywords
from the questions. KeyBERT utilizes BERT-embeddings along with a basic cosine
similarity measure to identify the most representative words that encapsulate
the content of the entire text. Extracted keywords are fed into the image
captioning model along with the corresponding image, as illustrated in Figure
1.

We also investigate whether less relevant portions of an image caption could
potentially introduce confusion or result in inaccurate answers for the QA
model/LLM. Hence, in our analysis, we experiment with keeping only the most
relevant sentence of the image caption and providing it to the LLM during the
QA step. To achieve this, we utilize Sentence-BERT [30], specifically
employing the MiniLM-L6 model111https://huggingface.co/sentence-
transformers/multi-qa-MiniLM-L6-cos-v1, to extract the most relevant sentence
from the image caption based on the given question.

###  3.2 Question Answering

As in the pipeline shown in Figure 1, the QA model takes the image caption and
the question as input, leveraging information from the image caption to
generate an answer. During the QA step, we utilize GPT-3.5, recognized for its
high zero-shot performance in QA benchmarks [39]. Despite the superior
performance of the more recent LLM, GPT-4, across various natural language
tasks including QA, we choose not to use GPT-4 in our experiments to keep our
pipeline cost-effective. In future works, the integration of higher-performing
LLMs with the pipeline could be explored.

We derive answers with an open-ended generation, specifically using
GPT-3.5-turbo API provided by OpenAI. The answer size is restricted to a
maximum of two words, aligning with the answer size distribution in the GQA
dataset. Optimal prompts are given in the Section 4.5.

##  4 Experimental Setup

###  4.1 Dataset

We conduct experiments on the GQA [16] dataset, specifically the balanced
version of the test-dev subset, comprising $12,578$ questions. Each image in
the dataset is linked to multiple questions, and the overall number of images
included is $398$. This subset contains a diverse distribution of questions
across various categories, with a primary focus on categorization based on
structure and semantics. The structural type is determined by the final
operation in the functional program of the question, encompassing categories
such as _verify, query, choose, logical,_ and _compare_. The semantic type
specifies the primary focus of the question and includes categories like
_object, attribute, category, relation,_ and _global_. Table 1 presents an
overview of question types, corresponding descriptions, and the respective
number of questions in the GQA test-dev [16].

Table 1:  Overview of the question types

Question type | Description | Example | No. samples
---|---|---|---
verify | yes/no questions | _Does the device under the picture frame look black?_ | 2252
query | open questions | _Which kind of vehicle is waiting for the traffic light?_ | 6805
choose | choosing from alternatives | _What color is the hair, gray or red?_ | 1128
logical | logical inference | _Are the flags triangular and red?_ | 1803
compare | comparison of objects | _Which is larger, the pasture or the horse?_ | 589
object | existence questions | _Are there both a horse and a fence in the image?_ | 778
attribute | object properties/position | _On which side of the picture are the pens?_ | 5185
category | object identification | _What kind of clothing is yellow?_ | 1149
relation | relations with objects/subjects | _Is the toaster to the right of a refrigerator?_ | 5308
global | overall properties | _Is it an outdoors scene?_ | 157

###  4.2 Competing VQA Methods

To evaluate zero-shot VQA performance, we use the chat variation of
CogVLM222https://huggingface.co/THUDM/cogvlm-chat-hf and BLIP-2
FlanT5XL333https://huggingface.co/Salesforce/blip2-flan-t5-xl. CogVLM is an
open-sourced pre-trained vision-language model with 10B visual and 7B language
parameters. CogVLM outperforms many vision-language models, _e.g_.,
InstructBLIP [7] and LlaVA-1.5 [21], in VQA benchmarks. Due to our resource
constraints with $16$ GB VRAM, we apply $4$-bit quantization to CogVLM. BLIP-2
FlanT5XL with 4.1B parameters also indicate high performance surpassing BLIP-2
OPT6.7B and Flamingo [2] in VQA benchmarks. We employ BLIP-2 FlanT5XL with F16
precision.

###  4.3 Image Captioning Methods

We examine the VQA performance attributed to semantic and structural question
types mentioned in Section 4.1. Image captions are obtained through the visual
grounding444https://huggingface.co/THUDM/cogvlm-grounding-generalist-hf and
chat555https://huggingface.co/THUDM/cogvlm-chat-hf variations of the CogVLM,
FuseCap666https://github.com/RotsteinNoam/FuseCap, and BLIP-2
OPT2.7b777https://huggingface.co/Salesforce/blip2-opt-2.7b models. When
determining the image captioning method, we pay attention to both its
alignment with our resource capacity and its high performance in image
captioning benchmarks. We employ 4-bit quantization to CogVLM and use F16
precision for BLIP-2 OPT2.7b.

###  4.4 Evaluation

Before the evaluation, GPT-3.5 predictions undergo post-processing, which
involves the removal of punctuation. During the evaluation process, we employ
the accuracy metric, calculated as the ratio of correctly predicted answers to
the total number of answers. Given that answers are derived through open-ended
generation using LLMs and might include variations, we do not seek an exact
match between the prediction and the ground truth. Instead, we evaluate
semantic similarity using cosine similarity in a vector space with the
threshold $0.70$. If two strings are closely aligned in meaning, the
prediction is accepted as correct; for example, accepting $couch$ as correct
for the label $sofa$. We determine the similarity threshold through manual
observation of the results. At lower thresholds, we observe that predictions
incorporating words related to each other, yet lacking identical meanings, are
also considered correct. For instance, the similarity value between the words
_blue_ and _brown_ is found to be $0.67$. We additionally assess performance
across higher cosine similarity thresholds, _e.g_. $0.8$ and $0.9$, and for
exact matching (EM).

Table 2:  Comparison of the performances of different image captioning methods
in the context of VQA on GQA test-dev. Image captioning methods are employed
with GPT-3.5 as the question-answering (QA) method. Two variants of CogVLM,
namely visual grounding (CogVLM-V) and chat model (CogVLM-C), are utilized for
image captioning. QD and SB refer to question-driven and sentence-based
captions, respectively. The answers with a cosine similarity of $0.7$ or
higher have been considered correct with the label. Accuracy values are
compared with the performance of zero-shot VQA models based on various
question categories.

Question type |  CogVLM-C Cap. \+ GPT-3.5 QA |  CogVLM-V Cap. \+ GPT-3.5 QA |  CogVLM-C QD Cap. \+ GPT-3.5 QA |  CogVLM-C SB Cap. \+ GPT-3.5 QA |  FuseCap Cap. \+ GPT-3.5 QA |  BLIP-2 Cap. \+ GPT-3.5 QA |  CogVLM VQA |  BLIP-2 VQA
---|---|---|---|---|---|---|---|---
verify | 63.01 | 58.53 | 66.83 | 61.06 | 53.60 | 55.82 | 83.04 | 56.48
query | 36.91 | 31.08 | 38.34 | 31.51 | 29.61 | 31.87 | 54.11 | 41.31
choose | 65.25 | 60.90 | 65.51 | 60.90 | 58.07 | 60.82 | 87.32 | 56.91
logical | 59.51 | 60.29 | 59.07 | 58.46 | 57.07 | 56.07 | 77.54 | 54.24
compare | 51.78 | 51.95 | 51.95 | 49.07 | 54.50 | 48.22 | 62.65 | 46.52
object | 61.95 | 63.24 | 59.13 | 58.87 | 59.38 | 58.35 | 84.45 | 57.07
attribute | 51.75 | 46.42 | 54.62 | 50.80 | 45.11 | 46.63 | 70.45 | 49.33
category | 47.35 | 44.21 | 50.39 | 42.56 | 43.52 | 42.47 | 63.19 | 53.35
relation | 42.56 | 38.32 | 42.97 | 35.76 | 34.98 | 37.23 | 59.91 | 43.31
global | 49.04 | 45.86 | 45.86 | 44.59 | 43.95 | 45.22 | 56.05 | 40.13
total | 48.06 | 43.83 | 49.50 | 44.12 | 41.58 | 42.99 | 66.02 | 47.52

###  4.5 Prompt Details

A brief prompt, ‘Describe the scene in this image’ is supplied to the image
captioning model to create general-purpose image captions. To create question-
driven captions, ‘Consider the keywords: [keywords]’ is added to the prompt.
In the QA stage, the LLM prompt involves ‘Answer the question in a maximum of
two words based on the text. Consider the type of question in your answer. For
example, if it is a yes/no question, the answer should be yes or no. Text:
[text], Question: [question]’. We notice a positive impact on the results when
we include an instruction in the prompt to consider the question type. In the
decoding step for the answer generation, we set $temperature$ as $0.2$,
$top_{p}$ as 1, and specify $frequency\\_penalty$ and $presence\\_penalty$ as
0.

##  5 Results

Figure 2: Examples from correct predictions in case that QD image captioning
is applied.

Figure 3: Examples from wrong predictions in case that QD image captioning is
applied.

###  5.1 Main Findings

Table 2 summarizes our results and demonstrates that employing our suggested
QD image captioning approach for VQA enhances performance across most question
categories compared to general-purpose image captioning. Also, Table 3
indicates that the QD image captioning approach utilizing the CogVLM-chat
variant surpasses other image captioning methods in evaluations seeking both
different cosine similarity thresholds and exact matching.

Table 3: Comparison of overall accuracy for exact matching (EM) and in
different cosine similarity thresholds.

Models | EM | sim=$0.9$ | sim=$0.8$
---|---|---|---
CogVLM-C Cap. + GPT-3.5 QA | 36.77 | 38.21 | 43.01
CogVLM-V Cap. + GPT-3.5 QA | 36.21 | 37.51 | 41.21
CogVLM-C QD Cap. + GPT-3.5 QA | 37.64 | 39.24 | 44.48
CogVLM-C SB Cap. + GPT-3.5 QA | 34.14 | 35.06 | 39.41
FuseCap Cap. + GPT-3.5 QA | 33.17 | 34.18 | 37.64
BLIP-2 Cap. + GPT-3.5 QA | 34.77 | 35.53 | 39.11
CogVLM VQA | 58.43 | 59.23 | 62.79
BLIP-2 VQA | 37.82 | 38.57 | 42.33

Significant performance enhancements are evident in QD image captions,
particularly in the _verify_ category for yes/no questions, as well as
_attribute_ and _category_ types primarily focused on identifying and
describing a single object’s properties. However, challenges arise in the
_object_ category often asking which of two objects exists in the frame.
Particularly in this category of questions, despite the QD image captions
containing relevant information, inaccuracies emerge due to the behavior of
the QA model, as elaborated in Section 5.2.

We also notice that the QD captioning emphasizing question keywords is linked
to a performance decline in the _global_ type questions. Global-type questions
typically pertain to the overall content of an image. It suggests that the
emphasis on question keywords in the caption negatively affects the model’s
ability to make inferences about the entire image. On the other hand, it is
quite possible to give other answers to questions of this type that are
meaningful and contextually correct but do not match the label. In most of the
cases, we observe that GPT-3.5 predicts answers that could be correct but do
not precisely match the expected label (see examples in Figure 3).

In most question categories, the accuracy achieved by combining QD image
captions with GPT-3.5 for VQA exceeds the performance of BLIP-2 FlanT5XL in
the zero-shot setting. However, all image captioning-based approaches indicate
inferior performance compared to the CogVLM-chat model for VQA. We are
intrigued to discover a notable disparity in performance when comparing the
image captions extracted by the CogVLM-chat model and provided to LLM, in
contrast to the VQA performance of the CogVLM-chat model, unlike the case with
BLIP-2.

Among the FuseCap, BLIP-2 OPT2.7b, CogVLM-chat, and CogVLM-visual grounding
models, the most informative captions for VQA are obtained through the CogVLM-
chat variant. The CogVLM-visual grounding variant indicates the highest
performance only in _object_ and _logical_ question categories. This suggests
that visual grounding models may provide an advantage in these question
categories with their capacity to connect language queries to relevant visual
elements and reason about object-related relationships.

Limiting image captions to the most relevant sentence reduces the overall
performance of the CogVLM-chat model, though the impact varies across question
types, with _verify_ , _query_ , _choose_ and _relation_ types being more
negatively affected. This suggests that sentences less directly related to the
questions do not result in confusion or inaccuracies for LLM during the QA.
Conversely, generating more comprehensive and context-rich image captions is
necessary for optimal performance.

Figure 2 and 3 feature examples of both correct and incorrect outcomes, where
image captions are generated by the CogVLM-chat model using question-driven
captioning and then fed to GPT-3.5 for answer prediction.

###  5.2 Error Analysis

When examining incorrect predictions based on question types, we discover some
common issues.

We notice that 27% of the incorrect predictions are related to yes/no
questions. A closer look reveals that in 11% of the incorrectly answered
yes/no questions, GPT-3.5 provides a response using a word other than _yes_ or
_no_. For instance, when the provided caption is _‘The image showcases a
skateboarder in action, possibly performing a trick on a ramp. The
skateboarder is wearing protective gear, including a helmet, knee pads, and
elbow pads. The background features a clear blue sky, trees, and a building.
The overall ambiance suggests an outdoor skateboarding event or practice
session.’_ , in response to the question _‘Are there salt shakers or
skateboards in the picture?’_ GPT-3.5’s prediction is _skateboards_ , while
the ground-truth is _yes_. We observe that most similar inaccuracies are
associated with questions related to _object_ and _logical_ types, often
connecting more than one object or attribute using conjunctions like _and_ or
_or_ , as given in the example. We posit that this issue can be alleviated by
crafting more effective prompts for GPT-3.5 or by employing a more powerful
LLM for QA.

We also assess the instances where the LLM fails to provide an answer based on
the information present in the image caption. Specifically, we examine the
occurrences of _not mentioned_ and _not visible_ responses from GPT-3.5. Our
findings indicate that, for the best general-purpose image captioning model,
GPT-3.5 is not able to respond to 1.7% of the questions. Notably, when
employing question-driven captioning, this rate decreases to 0.5%.

##  6 Conclusion

This study aims to develop a zero-shot VQA pipeline, leveraging LLMs with the
inclusion of image captioning as an intermediate step, and evaluate its
performance on the GQA benchmark. The proposed approach involves question-
driven image captioning to transfer contextual information to the QA model.
The study includes a thorough evaluation of zero-shot models for image
captioning in the VQA context, comparing the impact of general-purpose and
question-driven image captions in terms of various types of questions. Our
comparative analysis suggests that incorporating question-driven image
captions into the VQA process has a more favorable effect on overall
performance, surpassing the VQA performance of BLIP-2. Future endeavors may
explore the integration of larger-scale LLMs, _e.g_., GPT-4, to further
enhance performance. Additionally, evaluating the pipeline in a few-shot
setting could offer a more comprehensive comparison. To enhance transparency,
replacing the QA model with an interpretable alternative, such as graph-based
QA models, can be explored.

## Acknowledgements

This work is partially supported by Middle East Technical University
Scientific Research Projects Coordination Unit (METU-BAP), under the project
number ADEP-704-2024-11482.

## References

  * Achiam et al. [2023] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_ , 2023.
  * Alayrac et al. [2022] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al.  Flamingo: a visual language model for few-shot learning.  _Advances in Neural Information Processing Systems_ , 35:23716–23736, 2022.
  * Antol et al. [2015] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh.  Vqa: Visual question answering.  In _Proceedings of the IEEE international conference on computer vision_ , pages 2425–2433, 2015.
  * Ben-Younes et al. [2017] Hedi Ben-Younes, Rémi Cadene, Matthieu Cord, and Nicolas Thome.  Mutan: Multimodal tucker fusion for visual question answering.  In _Proceedings of the IEEE international conference on computer vision_ , pages 2612–2620, 2017.
  * Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Carion et al. [2020] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In _European conference on computer vision_ , pages 213–229. Springer, 2020.
  * Dai et al. [2023] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi.  Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023.
  * Devlin et al. [2018] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ , 2018.
  * Goyal et al. [2017] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh.  Making the v in vqa matter: Elevating the role of image understanding in visual question answering.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 6904–6913, 2017.
  * Grootendorst [2020] Maarten Grootendorst.  Keybert: Minimal keyword extraction with bert.  _Zenodo_ , 2020.
  * Guo et al. [2023] Jiaxian Guo, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Boyang Li, Dacheng Tao, and Steven Hoi.  From images to textual prompts: Zero-shot visual question answering with frozen large language models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10867–10877, 2023.
  * Gupta and Kembhavi [2023] Tanmay Gupta and Aniruddha Kembhavi.  Visual programming: Compositional visual reasoning without training.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 14953–14962, 2023.
  * Herdade et al. [2019] Simao Herdade, Armin Kappeler, Kofi Boakye, and Joao Soares.  Image captioning: Transforming objects into words.  _Advances in neural information processing systems_ , 32, 2019.
  * Huang et al. [2019] Lun Huang, Wenmin Wang, Jie Chen, and Xiao-Yong Wei.  Attention on attention for image captioning.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 4634–4643, 2019.
  * Hudson and Manning [2018] Drew A Hudson and Christopher D Manning.  Compositional attention networks for machine reasoning.  _arXiv preprint arXiv:1803.03067_ , 2018.
  * Hudson and Manning [2019] Drew A Hudson and Christopher D Manning.  Gqa: A new dataset for real-world visual reasoning and compositional question answering.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 6700–6709, 2019.
  * Johnson et al. [2017] Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick.  Clevr: A diagnostic dataset for compositional language and elementary visual reasoning.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2901–2910, 2017.
  * Kamath et al. [2021] Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion.  Mdetr-modulated detection for end-to-end multi-modal understanding.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 1780–1790, 2021.
  * Li et al. [2022] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  In _International Conference on Machine Learning_ , pages 12888–12900. PMLR, 2022.
  * Li et al. [2023] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  _arXiv preprint arXiv:2301.12597_ , 2023.
  * Liu et al. [2023] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee.  Improved baselines with visual instruction tuning.  _arXiv preprint arXiv:2310.03744_ , 2023.
  * Liu et al. [2024] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual instruction tuning.  _Advances in neural information processing systems_ , 36, 2024.
  * Liu et al. [2019] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_ , 2019.
  * Mao et al. [2019] Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B Tenenbaum, and Jiajun Wu.  The neuro-symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision.  _arXiv preprint arXiv:1904.12584_ , 2019.
  * Marino et al. [2019] Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi.  Ok-vqa: A visual question answering benchmark requiring external knowledge.  In _Proceedings of the IEEE/cvf conference on computer vision and pattern recognition_ , pages 3195–3204, 2019.
  * Mokady et al. [2021] Ron Mokady, Amir Hertz, and Amit H Bermano.  Clipcap: Clip prefix for image captioning.  _arXiv preprint arXiv:2111.09734_ , 2021.
  * OpenAI [2023] OpenAI.  Chatgpt.  https://openai.com/blog/chatgpt, 2023.
  * Perez et al. [2018] Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville.  Film: Visual reasoning with a general conditioning layer.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2018.
  * Radford et al. [2019] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9, 2019.
  * Reimers and Gurevych [2019] Nils Reimers and Iryna Gurevych.  Sentence-bert: Sentence embeddings using siamese bert-networks.  _arXiv preprint arXiv:1908.10084_ , 2019.
  * Rotstein et al. [2024] Noam Rotstein, David Bensaïd, Shaked Brody, Roy Ganz, and Ron Kimmel.  Fusecap: Leveraging large language models for enriched fused image captions.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pages 5689–5700, 2024.
  * Santoro et al. [2017] Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap.  A simple neural network module for relational reasoning.  _Advances in neural information processing systems_ , 30, 2017.
  * Sharma and Jalal [2021] Himanshu Sharma and Anand Singh Jalal.  A survey of methods, datasets and evaluation metrics for visual question answering.  _Image and Vision Computing_ , 116:104327, 2021.
  * Srivastava et al. [2021] Yash Srivastava, Vaishnav Murali, Shiv Ram Dubey, and Snehasis Mukherjee.  Visual question answering using deep learning: A survey and performance analysis.  In _Computer Vision and Image Processing: 5th International Conference, CVIP 2020, Prayagraj, India, December 4-6, 2020, Revised Selected Papers, Part II 5_ , pages 75–86. Springer, 2021.
  * Tan and Bansal [2019] Hao Tan and Mohit Bansal.  Lxmert: Learning cross-modality encoder representations from transformers.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_. Association for Computational Linguistics, 2019.
  * Team et al. [2023] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al.  Gemini: a family of highly capable multimodal models.  _arXiv preprint arXiv:2312.11805_ , 2023.
  * Tiong et al. [2022] Anthony Meng Huat Tiong, Junnan Li, Boyang Li, Silvio Savarese, and Steven CH Hoi.  Plug-and-play vqa: Zero-shot vqa by conjoining large pretrained models with zero training.  _arXiv preprint arXiv:2210.08773_ , 2022.
  * Touvron et al. [2023] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ , 2023.
  * Wang et al. [2024] Cunxiang Wang, Sirui Cheng, Qipeng Guo, Yuanhao Yue, Bowen Ding, Zhikun Xu, Yidong Wang, Xiangkun Hu, Zheng Zhang, and Yue Zhang.  Evaluating open-qa evaluation.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Wang et al. [2023] Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al.  Cogvlm: Visual expert for pretrained language models.  _arXiv preprint arXiv:2311.03079_ , 2023.
  * Xu et al. [2015] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio.  Show, attend and tell: Neural image caption generation with visual attention.  In _International conference on machine learning_ , pages 2048–2057. PMLR, 2015.
  * Yang et al. [2022] Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang.  An empirical study of gpt-3 for few-shot knowledge-based vqa.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 3081–3089, 2022.
  * Yi et al. [2018] Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Josh Tenenbaum.  Neural-symbolic vqa: Disentangling reasoning from vision and language understanding.  _Advances in neural information processing systems_ , 31, 2018.
  * Zhao et al. [2021] Zelin Zhao, Karan Samel, Binghong Chen, et al.  Proto: Program-guided transformer for program-guided tasks.  _Advances in neural information processing systems_ , 34:17021–17036, 2021.


11institutetext: Telecom Paris, Institut Polytechnique de Paris, France
11email<EMAIL_ADDRESS>11email<EMAIL_ADDRESS>
# Integrating the Wikidata Taxonomy into YAGO

Fabian Suchanek  Mehwish Alam  Thomas Bonald
Pierre-Henri Paris  Jules Soria

###### Abstract

Wikidata is one of the largest public general-purpose Knowledge Bases (KBs).
Yet, due to its collaborative nature, its schema and taxonomy have become
convoluted. For the YAGO 4 KB, we combined Wikidata with the ontology from
Schema.org, which reduced and cleaned up the taxonomy and constraints and made
it possible to run automated reasoners on the data. However, it also cut away
large parts of the Wikidata taxonomy. In this paper, we present our effort to
merge the entire Wikidata taxonomy into the YAGO KB as much as possible. We
pay particular attention to logical constraints and a careful distinction of
classes and instances. Our work creates YAGO 4.5, which adds a rich layer of
informative classes to YAGO, while at the same time keeping the KB logically
consistent.

###### Keywords:

YAGO Knowledge Base Wikidata Taxonomy

##  1 Introduction

The rise of Large Language Models (LLMs) such as GPT, LLaMA [48], and PaLM [9]
has deeply impacted all areas of natural language processing. And yet, these
models have a crucial weakness: just like humans, they are not good at rote
learning [52, 36, 45, 8, 46]. They cannot memorize large amounts of facts
without forgetting some of them or inventing others. This is to be expected,
as they are machine learning models: they are not made to memorize, but to
generalize. For this, they interpolate between the data points. At the same
time, they will always cover up their lack of accuracy very eloquently: they
know how to talk even when they don’t know what to say.

There are, however, some applications where exact data is indispensable, e.g.,
when computing the distance between two cities [52], when asking for a list of
Nobel Prize winners [36], or when searching for an entity with a specific
property [49]. This is why Knowledge Bases (KBs) have their raison d’être:
they provide crisp symbolic knowledge. Indeed, some of the actors that build
the world’s largest LLMs (Google, Meta) also build some of the world’s largest
KBs [29]. There are also attempts to combine KBs with LLMs [31, 18], which
only further confirms the necessity of KBs.

#### Wikidata.

One of the largest general-purpose KBs nowadays is Wikidata [50]. It provides
a wealth of facts about nearly every domain of common human discourse, with
more than 100 million entities 111All numbers are given as of 2023-04-19 and
around 1.4 billion facts about them. Each entity has an abstract identifier
(such as _Q303_), which makes the identifiers language-independent and
persistent in time. Tens of thousands of people contribute to the project. At
the same time, being a collaborative KB, Wikidata suffers from a lack of
agreement on the schema level: there are several classes that are difficult to
distinguish for the uninitiated user (e.g., geographical location (Q2221906),
location (Q115095765), geographic region (Q82794), _physical location_
(Q17334923), and geographical area (Q3622002)); there are more than ten
thousand relations; constraints are defined but not enforced (_Grotesco_
(Q10509019) is a subclass of Q49094906, which is not a class); classes and
instances are mixed (_scientist_ (Q901), e.g., is both a subclass of _person_
(Q215627) and an instance of _profession_ (Q28640)); there are more than 2.7M
classes of which only 3% are instantiated (1M subclasses of _chemical entity_
(Q43460564) have no instance); and the taxonomy contains cycles (there are 47
pairs of classes that are subclasses of each other, e.g., _method_ (Q1799072)
and _technique_ (Q2695280), and 15 cycles of length 3 or more, e.g., _axiom_
(Q17736), _first principle_ (Q536351), _principle_ (Q211364)). Finally, the
abstract identifiers for Wikidata properties and entities makes downstream
applications more difficult.

#### YAGO 4.

The YAGO KB has been in existence since 2008 [44, 14, 5, 20]. Its fourth
version [47] was designed to address the shortcomings of Wikidata: It combines
the data about instances from Wikidata with the taxonomy and properties from
Schema.org – an ontology developed by a W3C Community
Group222https://www.w3.org/community/schemaorg/. Filtering and constraint
enforcement made YAGO 4 a KB that allows for automated reasoning. However,
this merger came at the expense of abandoning nearly the entire class taxonomy
of Wikidata. That is a pity because classes can express facts that are very
hard to model correctly by RDF properties [33] like something is a “train
ferry route”, a “financial regulatory agency”, or a “de facto consulate”. As a
consequence, one of the major criticisms that users advanced was that the
class hierarchy of YAGO 4 was too sparse.

#### Contributions.

In this paper, we show how this shortcoming of YAGO 4 can be resolved, while
still maintaining the logical consistency and semantic coherence of YAGO. We
carefully incorporate selected parts of the Wikidata taxonomy into the
taxonomy of Schema.org. This leads to considerable challenges. Numerous
organically grown branches of the Wikidata taxonomy have to be disentangled.
Furthermore, many classes in Wikidata are both instances and classes (a
_Doctor of Philosophy_ (Q752297), e.g., is both an instance of a _postgraduate
degree_ (Q23015928) and a subclass of _doctorate_ (Q849697)). The
transformation also posed engineering challenges: Wikidata comprises more than
120 GB even compressed, which have to be parsed and processed. We will
describe how we surmounted these challenges, and what open problems still
remain. The resulting resource, which we call YAGO 4.5, contains 109M facts
and is logically consistent.

This paper is structured as follows: Section 2 recalls related work, Section 3
discusses design decisions, Section 4 discusses implementation issues, and
Section 5 presents the resulting KB, before Section 6 concludes.

##  2 Related Work

#### General-Purpose Knowledge Bases.

The Semantic Web comprises hundreds of KBs333http://cas.lod-cloud.net/. Many
of these are tailored for specific domains or applications, such as the Gene
Ontology [3] for biological processes, functions, and cellular components.
However, in this paper, we are concerned with general-purpose KBs that do not
focus on a given domain. There are several prominent general-purpose KBs.
ConceptNet [43] is a semantic network that primarily deals with common sense
knowledge. This makes it an orthogonal project to YAGO concerned with facts
about instances. BabelNet [28] is a large multilingual encyclopedic dictionary
derived from WordNet, Wikipedia, Wikidata, and several other sources. BabelNet
focuses on relations between concepts and words, and has neither a taxonomy
nor a schema. DBpedia [4] is a large-scale, multilingual KB derived from
Wikipedia (and, more recently, Wikidata). Unlike YAGO, it lacks information
about the temporal validity of the facts. Moreover, the automated generation
from the Wikipedia infoboxes and the priority for recall over precision has
led DBpedia to be not fully consistent [40, 1, 10]. The manually curated part
contains just 4M instances444https://www.dbpedia.org/resources/ontology/
Freebase [6] was an extensive KB consisting of metadata compiled from various
sources. The project was discontinued in 2015, and its content was transferred
[34] to Wikidata. Wikidata [50], a project of the Wikimedia Foundation, is a
collaboratively edited KB that supports other Wikimedia projects like
Wikipedia and Wikimedia Commons. It is by far the largest open KB. However,
due to its collaborative nature, its taxonomy has grown convoluted and
complicated, with inconsistencies in hierarchies and data models as well as
rule violations – making it hard to use even by contributors [7, 35, 41].

In this landscape, YAGO positions itself as a large general KB for facts about
instances, with a taxonomy, manually defined properties, and logical
constraints. Its key property is that it is a centrally controlled data
source, which allows it to establish certain guarantees for the quality of its
data [44, 14, 5, 20, 47]. The latest version, YAGO 4 [47], was designed to be
clean enough to perform automated reasoning on it. However, its taxonomy is
very parsimonious, which is the challenge that we address in the present
paper.

#### Upper Ontologies.

Top ontologies, also known as upper or foundational ontologies, provide a
domain-independent framework for organizing knowledge across various fields.
We discuss some of the most prominent projects below and refer the reader to
Mascardi et al. [23] for a comprehensive comparison. Cyc [17] is one of the
oldest and most comprehensive upper ontologies, developed by Cycorp, aiming to
represent general human knowledge and common sense reasoning. SUMO [25] is an
open-source upper ontology with a formal structure for organizing and
integrating domain-specific ontologies. It consists of a core set of general
concepts and relations and domain-specific extensions that cover various
fields, such as biology, finance, and geography. DOLCE [24] is another top
ontology which focuses on capturing the ontological categories underlying
natural language and human cognition. BFO [2] is an upper ontology that was
created based on the ontologies related to the domain of geospatial
information.WordNet [27] contains lexical and semantic relationships between
sets of synonymous words (synsets). WordNet does not define properties, and
the project was discontinued in 2012.

Schema.org [12] differs from the above in that it is a collaborative project
initiated by major companies like Google, Microsoft, and Yahoo to provide a
shared vocabulary for annotating Web content with structured data. While not a
top ontology in the traditional sense, Schema.org plays a crucial role in the
Semantic Web ecosystem by promoting standardized vocabularies for describing
entities and their properties, thus facilitating data interoperability and
integration. It is broadly adopted (with more than 12 million websites in
2016) and benefits from strong industry support, making it a highly reliable
and sustainable choice for building a KB [26].

#### Taxonomy Induction & Expansion.

Many recent studies automate taxonomy induction and expansion, including
Online Catalog Taxonomy EnrichmenT (OCTET) [22], TaxoCom [16], TaxoOrder[42],
TaxoExpan [38], HiExpan [39], TaxoEnrich [15], and taxonomy induction from a
set of terms [21]. Our own previous YAGO versions automatically mapped WordNet
synsets to Wikipedia categories [44]. In contrast to these automated
approaches, YAGO 4 and the new YAGO 4.5 use a manual mapping. This is because
there are only a few dozen classes to be mapped.

There is a community effort to map Wikidata properties and classes to
schema.org555https://github.com/okfn-brasil/schemaOrg-Wikidata-Map. These
mappings use _exact match (P2888)_ (64 mappings), _equivalent class (P1709)_
(332 mappings), _Equivalent property (P1628)_ (84 mappings), _external
subproperty (P2236)_ (22 mappings), and _external superproperty (P2235)_ (6
mappings). However, the effort was discontinued in 2017. These mappings
inspired the mappings of YAGO 4, which in turn were the basis for the mappings
that we use in this paper.

##  3 Designing YAGO

###  3.1 Design Rationale

Our goal is to have a clean upper taxonomy for YAGO, which is precise and non-
redundant so as to allow for automated reasoning. Our choice falls on
Schema.org, for reasons we have elaborated in Section 2: the taxonomy is
concise, maintained by a W3C consortium, and it finds applications well beyond
its original purpose of annotating Web pages. It has the right level of detail
for our purposes and does not digress into philosophical concepts. It defines
not just classes, but also properties.

One could argue that the upper-level taxonomy is sufficient and that one
should not aim to add more fine-grained classes – least of all the Wikidata
taxonomy: it contains overly specific classes with few instances, some of its
classes are not useful for large KB applications (such as _multi-organism
process_ (Q22269697) for elections), and sibling classes could be further
grouped into common superclasses (e.g., Wikidata is missing a class that
regroups human-made places, making do instead with _human-made geographic
feature_ (Q811430), _human-made geographic feature_ (Q811463), and _human-made
geographic object_ (Q35145743)). All of this, one could argue, makes it an
ill-suited candidate for a taxonomy.

However, these issues go away when a clean upper-level taxonomy is put on top:
less useful lower-level classes (such as Q22269697) disappear because they are
not subclasses of the clean upper-level classes. The missing grouping,
likewise, can be achieved by the upper-level taxonomy – for example for
places. Finally, even if a Wikidata class contains few instances, it can still
carry meaningful information. For example, it is informative for the human
user to know that an entity is a “General aviation monoplane with 1 tractor-
piston-propeller engine” (Q33110974). It was precisely that lack of such
classes that users deplored for YAGO 4. Such classes do not carry logically
formalized meanings. (It would be cumbersome to formally express that
something is a “General aviation monoplane with 1 tractor-piston-propeller
engine” by RDF statements [32, 33].) Rather, the purpose of these lower-level
classes is to convey informal information for the human user. The Wikidata
classes clearly serve this purpose.

Thus, we now face the challenge of integrating the upper taxonomy from
Schema.org with the lower taxonomy from Wikidata. The following design
principles drive our integration:

1.

Prefer properties over class membership. Some information can be expressed
either by a property (_hasGender female_) or by a class membership (_type
Woman_). The option of class membership is chosen only if the class appears in
the domain or range of a property. In our example, _Woman_ does not appear as
a range or domain of any property (its superclass _Human_ does). Hence, the
class _Woman_ should not exist, and the gender should be expressed by a
property. This is consistent with Wikidata’s way of modeling. When we have the
choice between a property and its inverse property (e.g., _hasCitizenship_ and
_hasCitizen_), we choose the one that has, on average, less objects per
subject (i.e., _hasCitizenship_). This, too, is consistent with major KBs
[11].

2.

The upper taxonomy exists to define formal properties that will be populated.
All classes of the upper taxonomy shall define formal properties (e.g., an
_Airline_ as an _iataCode_ , which justifies its existence as an upper-level
class). Both the domain and the range of these properties have to be upper-
level classes.

3.

The lower taxonomy exists to convey human-intelligible information about its
instances in a non-redundant form. This criterion tells us to remove classes
that add no information over upper taxonomy classes, to eliminate links in the
taxonomy that are redundant due to transitivity, to merge classes that are
hard to distinguish for the uninitiated user, and to remove classes that are
not populated.

###  3.2 Upper Taxonomy

#### Top-level classes.

As for YAGO 4 [47], we start with the taxonomy of Schema.org. It defines one
top-level class, _schema:Thing_ with 11 subclasses. We exclude the class
_Action_ 666For ease of reading, we omit prefixes where these can be
inferred., which models mainly Web user actions. We exclude _BioChemEntity_
and _MedicalEntity_ because these require expert knowledge to model correctly,
which we currently do not have in our team. This leaves us with 8 top-level
classes (_CreativeWork, Event, Organization, Taxon, Person, Place, Product,
Intangible_), which we accept as subclasses of _Thing_. All top-level classes
are declared disjoint (except places/organizations, and products/creative
works).

#### Fictional entities.

To deal with fictional entities, we add a class _yago:FictionalEntity_ as a
subclass of _schema:Thing_. This class defines properties such as
_yago:createdBy_ and _yago:appearsIn_ , thus justifying its existence as a
class under Design Principle 1. Fictional entities are not disjoint with any
other class, as anything can also exist in fiction. A fictional entity is an
instance of both _yago:FictionalEntity_ and the class that it belongs to in
fiction. For example, a fictional human is an instance of both
_yago:FictionalEntity_ and _schema:Person_. This has the disadvantage that
fictional humans will be counted as a humans in count queries. However, it has
the advantage that one can easily reason on fictional humans, as they will
share all the properties that we declared for _schema:Person_. Wikidata goes a
different way, by recreating the entire class hierarchy with its properties
also for fictional beings, mapping each class of fictional entities to its
real-world class counterpart. This choice can for sure be defended, but for
YAGO, it would have severely convoluted the schema: It would have required
duplication of all class and property specifications. The current modelization
already has an advantage over the modelization in previous versions of YAGO
(which simply merged real and fictional entities), as well as over other top-
level ontologies such as DOLCE, BFO, and SUMO (which do not model fictional
entities at all).

#### Intangibles.

For our new YAGO, we added the following classes that are not in Schema.org,
but that are necessary to define the ranges of properties (under Design
Principle 2 above): _yago:Award_ , _yago:Gender_ (which differs from
_schema:GenderType_ in that it allows more than two values), and
_yago:BeliefSystem_ (for religious adherence). All are subclasses of
_schema:Intangible_. The other subclasses of _Intangible_ that Schema.org
defines are mostly Web-specific (e.g., _ActionAccessSpecification_). Since
these classes would not have instances, let alone populated properties from
Wikidata, we removed them under Design Principle 2.

Schema.org has a subclass _Occupation_ of _Intangible_ , and models
occupations by the property _hasOccupation_. However, if we model occupations
by a property, (1) we lose the class hierarchy of occupations (_Physicist
subClassOf Scientist_ etc.), and (2) we lose the ability to add properties to
specific professions (such as the _doctoral advisor_ for scientists). Hence,
by Design Principle 1 above, we model professions rather as subclasses of
_Person_.

#### Places.

When it comes to places, the taxonomy of Schema.org is heavily oriented
towards the annotation of Web pages, with subclasses such as _Accommodation_ ,
_Residence_ , _LocalBusiness_ , etc. These classes do not define properties
that we could populate from Wikidata, and therefore, we remove them under
Design Principle 2. We then manually created a taxonomy of subclasses of
_schema:Place_ , which distinguishes _schema:Landform_ (areas with a boundary
given by nature), _schema:AdministrativeArea_ (boundary given by human
administration) and the newly created _yago:HumanMadePlace_ (boundaries given
by human physical construction) and _yago:AstronomicalObject_ (with boundaries
in space). For the former, we add a subclass _yago:Way_ , which regroups all
ways of transit (roads, canals, railway lines, etc.).

#### General considerations.

Under Design Principle 2, we keep only those classes from Schema.org that add
new properties (plus their super-classes, all the way up to _schema:Thing_).
This results in 41 upper classes. As in YAGO 4, all of the above is expressed
as SHACL constraints777https://www.w3.org/TR/shacl/ on the classes of
Schema.org. Thus, there is no special syntax, code, or formalism for these
declarations, and they are all part of the YAGO KB as normal facts.

###  3.3 Lower Taxonomy

#### Mapping to Wikidata.

The lower levels of the YAGO taxonomy come from Wikidata. As in YAGO 4, each
class in the YAGO upper taxonomy is manually mapped to one or more classes in
Wikidata. This happens, likewise, in a fully declarative way with a simple RDF
statement that links the Schema.org-class by a special predicate to the
Wikidata class(es). The mapping can happen at any level of the upper taxonomy:
general classes such as _Organization_ are mapped to Wikidata, and more
special classes such as _Corporation_ are mapped as well. A mapping can give
rise to the following constellations:

One-to-one mapping.

One upper class is mapped to one Wikidata class. All of the subclasses of the
Wikidata class are glued under the upper class, but its super-classes are not
imported.

One-to-many mapping.

One upper class is mapped to several Wikidata classes. Again, all subclasses
of these are glued under the upper class. This has the effect of merging the
Wikidata classes. We do this, e.g., for classes that are equivalent for our
purposes (such as _geographical region_ (Q82794) and _geographical area_
(Q3622002)).

One-to-none mapping.

One upper class is mapped to no Wikidata class – only its subclasses are
mapped to Wikidata. This has the effect that there cannot be direct instances
of the upper class. Nor can there be subclasses other than the ones we
declared. This is a new mechanism that did not exist in YAGO 4. We use it for
classes with a convoluted taxonomy in Wikidata.

We use a one-to-none-mapping for the following classes:

schema:Thing.

In YAGO 4, this class was mapped to the top-level class _entity_ (Q35120) in
Wikidata, which resulted in more than 1 million direct instances of
schema:Thing in Yago. This defies Design Principle 2, because _schema:Thing_
defines only very few properties. Hence, we now accept only entities that fall
into one of the manually approved subclasses of _Thing_. This results in a
clean top-level taxonomy, discarding meta-classes (e.g., _class or metaclass
of Wikidata ontology_ (Q21522864)), overlapping classes (e.g., _geographic
entity_ (Q27096213) and _location_ (Q115095765)) and too specific classes
(there are 6 top classes in Wikidata with less than 40 instances each, e.g.,
_converter_ (Q35825432)).

schema:Place.

The taxonomy of Wikidata for places is highly convoluted, with classes that
are difficult to distinguish such as _terrain_ (Q14524493), _geographical
location_ (Q2221906), _geographical region_ (Q82794), _geographical area_
(Q3622002), and _location_ (Q115095765). Hence, we do not map _schema:Place_ ,
and accept only instances of its manually designed subclasses. In doing so, we
discard 2,861 classes (from 29,826, i.e., less than 1%) and 137k instances
(from 19M, i.e., less than 1%).

schema:Intangible.

Intangible classes in Wikidata, likewise, are highly convoluted, with classes
such as _class_ (Q5127848), _process_ (Q3249551 and Q67518233) or _role_
(Q4897819). Hence, here, too, we would have difficulties establishing
properties to comply with Design Principle 2. Therefore, we apply a one-to-
none mapping and accept only instances and subclasses of the manually declared
subclasses of _Intangible_.

#### Importing the subclasses.

Once the mapping has been defined manually, the subclasses of Wikidata can be
imported automatically, as follows: We consider the subclass graph of
Wikidata, which we construct as follows: Every entity of Wikidata that has a
_subClassOf_ relationship becomes a node in this graph. It is linked to its
superclasses by the _subClassOf_ relationship. We then iterate through all
upper classes of YAGO, and whenever we hit a class that is mapped to Wikidata,
we glue the entire sub-DAG of that Wikidata class under the YAGO top-level
class. This approach differs from the approach in YAGO 4, where we mapped only
classes with a Wikipedia article. However, this restriction limited the
taxonomy to just a handful of classes per instance, which proved to be few.

Several caveats are to be respected in this merging process: As in YAGO 4, we
ensure that we do not add any link that would create a loop in the taxonomy
(49 loops were removed). We do not add transitive links (40k such links were
removed). We do not add a link if that would make a class a transitive
subclass of two top-level classes that we declared disjoint (9k links were
removed). We also do not add a class that is itself mapped to YAGO upper
classes to avoid duplicating the subtree.

#### Excluded classes.

We exclude housekeeping classes of Wikidata that we blacklist manually:
classes of Wikimedia pages, disambiguation pages, lists, and the like. For our
new YAGO, we also exclude linguistic objects (such as characters (Q3241972),
phrases (Q187931), numbers (Q11563), etc.), many of which are technically
infinite, and would otherwise make up 700k entities in YAGO. We also remove
abstract objects such as actions and occurrents, as these have rather
philosophical subclasses that are of limited use for our purposes (e.g.,
_Multi-organism process_ (Q22269697), etc.). The same fate is bestowed on
scholarly articles. The addition of all obtainable scholarly articles to
Wikidata was controversial888https://www.mail-
<EMAIL_ADDRESS>and in YAGO they would
make up of 39M entities, almost half of all entities. Hence, we decided to
remove them. Finally, under Design Principle 3, we also remove all classes
that do not have instances (1.3M).

###  3.4 Instances

#### Identifiers.

As in YAGO 4, every instance is automatically equipped with a readable name.
We use the title of the corresponding Wikipedia page as an entity identifier
(as in _yago:Eleanor_Roosevelt_). If there is none, or if the same Wikipedia
page is used by more than one entity, we use the English label of the entity
and concatenate it with the Wikidata Q-id to avoid ambiguity (as in
_yago:Brazilian_jiu_jitsu_competition_Q105086361_). If there is none, we use a
label that contains legal Turtle characters, concatenated with the Wikidata id
(which was not done in YAGO 4). If there is no such label, we use the Wikidata
id (with a YAGO-prefix for uniformity). The Turtle
standard999https://www.w3.org/TR/turtle/ allows percentage codes in local
names, but many parsers (e.g., the one of Hermit101010http://www.hermit-
reasoner.com/) cannot deal with them. Hence, we replace all characters that
are not letters or numbers by their hexadecimal Unicode, so that two
identifiers that differ only in their inadmissible characters are still
distinct.

#### Instances vs. Classes.

Wikidata contains several items that are both instances and classes. For
example, _English_ (Q1860) is an instance of _Natural Language_ (Q33742), as
well as a subclass of _Anglic_ (Q1346342). That makes sense, because there can
be several subclasses of _English_ , such as e.g., _American English_ (Q7976).
As per our discussion in Section 3.3, _English_ thus becomes a class. The
trouble is that OWL DL (the reasoning formalism we target) does not allow
statements between instances and classes. Thus, it would be impossible to say
that Eleanor Roosevelt spoke English. (That problem did not appear in YAGO 4,
where intermediate classes were eliminated aggressively, and survived only as
instances.) We solve this problem as follows: Whenever we encounter a Wikidata
fact that would link an instance to a class, we create a generic instance of
the class, and use it as object. In our example, we create a fact saying that
Eleanor Roosevelt spoke _yago:English_language_generic_instance_. The
intuition is that Roosevelt spoke something that is an instance of English.
This mechanism is in line with Approach 2 in [30], and kicks in for awards,
belief systems, academic titles, and languages.

This technique works well for the objects of statements, which generally have
an existential interpretation (Eleanor Roosevelt did not speak all dialects of
English, but there is one that she spoke). It works less well for subjects of
statements. For example, commercial products are classes in Wikidata (and
rightly so, as different people can own different instances of the same
product). However, if we take commercial products as classes, we cannot attach
their manufacturer, date of inception, awards, etc. to them. It would be
semantically wrong to attach these to a generic instance of the product, as
they apply to the line of products itself. It would also be semantically wrong
to create an axiom that says that all instances of the class have that
property (not every single iPhone has won an award). Hence, we make every item
that is a product (as identified by the _manufacturer_ (P176) relation in
Wikidata) an instance.

###  3.5 Properties and Constraints

Properties and constraints work largely as in YAGO 4: For _schema:Thing_ and
each sub-class from Schema.org, we manually define the properties that
Schema.org provides. By Design Principle 2, we keep only those properties of
Schema.org that are (1) of general interest (removing specialized properties
such as _hasDriveThroughService_) and (2) existent in Wikidata (because
otherwise they would not be populated). For the new YAGO, we added a few
properties, most notably for the new top-level classes _yago:Award_ and
_yago:FictionalEntity_ , and also for countries (Schema.org does not contain
the relation _hasCapital_). An instance can have only those properties that
are declared for its class or superclasses. We manually add SHACL constraints
for maximum cardinality, patterns of literals (e.g., for ISBNs), and domain
and range constraints. Each relation is mapped manually and declaratively to a
relation in Wikidata, and populated from there (taking only the truthy facts).
We extract time stamps for facts from Wikidata, and attach them to the YAGO
facts in the RDF* model [13].

The entire taxonomy of upper classes, the definition of properties, the
accompanying SHACL constraints, and the mapping to Wikidata classes take the
form of a single Turtle file. It forms an integral part of YAGO, and can be
downloaded along with the rest of the KB. All of this works exactly as in YAGO
4, and we refer the interested reader to the corresponding publication.

Different from YAGO 4, the new YAGO simplifies numerical quantities: while
these were previously values with a range and a unit, they are now simple
literals. We now provide a detailed Design Document that lists, explains, and
justifies the design decisions for the new schema on the website of YAGO 4.5.

##  4 Implementing YAGO

Parsing, analyzing, and transforming a KB of the size of Wikidata (766 GB as
of April 2023) is no easy feat. We describe here the challenges we encountered
when creating YAGO 4.5, and how we surmounted them, in the hope that this will
be useful for other users of Wikidata and YAGO.

#### Infrastructure.

The code of YAGO 4 was written in Rust. While this ensured high performance
and compile-time flagging of code problems, it also provided a formidable
hurdle to other project members (most notably the project leader), and we have
therefore rewritten the code from scratch in Python. The original YAGO 4
loaded the data into a RocksDB key-value store. This had the advantage that
many costly operations (such as constraint checks) could be run directly on
the data store. At the same time, loading the entire data into the data store
and indexing it could easily take a day. The new system, therefore, stores all
data (intermediate and final) in files on the hard drive, which has the
advantage that intermediate results can be inspected and re-used.

#### Data formats.

Wikidata exists in the “full” version and the “truthy”
version111111https://www.wikidata.org/wiki/Wikidata:Database_download. While
the truthy version is much smaller, we need the full version to extract time
stamps for facts (a feature that YAGO had since Version 2 [14]). One has the
choice between the NT format (easier to parse) and the Turtle file (smaller by
a factor of 2), and our choice falls on the latter. The file can be downloaded
as BZ2 and as GZIP, and we strongly recommend the GZIP version. First, the
unpacking is much faster (in the order of hours instead of the order of days
in our case). Second, BZ2 allows no way of seeing the compressed file size
without unpacking it. Finally, GZIP files can be processed sequentially by
Python without unpacking them. For parsing Wikidata, we experimented with
RDFlib. However, the library failed for certain characters in URIs, which
caused an unrecoverable abortion in the middle of the parsing. Furthermore,
the generation of URIs (expanding the prefixes) causes a large overhead.
Therefore, we wrote our own Turtle parser and graph database, which, for our
limited application, turned out to be rather simple (500 lines of code). These
design choices mean that our code does not use any external libraries.

While the initial input (Wikidata and schema.org) is in Turtle, we chose TSV
as our intermediate file format, because it allows for much faster parsing. In
addition, we can attach more information to each fact (time stamp, source,
etc.) in the form of supplementary columns.

#### Data processing.

Our system proceeds in 6 sequential steps. Each step reads the output files of
the previous step, and produces new output files, as in [5]. Each of these
steps can be run on its own, and each of the steps has its own set of test
input files with gold-standard output files, which we can use to check if the
step works as expected.

Two of our steps need to process the entire Wikidata file, and this is done in
a parallelized fashion. We experimented with Python multithreading, only to
find that, due to the Global Interpreter Lock, it does not fully utilize
multiple processor cores for CPU-bound tasks. The correct construction is
multiprocessing, which uses one processor per process. Since processes cannot
efficiently share data, each process has to load a copy of the data that it
needs. Each process writes out its results to its own temporary file, which we
then merge with the others. Each process $i$ of the $n$ processes starts at
position $(i-1)/n\times{}N$ of the Wikidata file (where $N$ is the size of the
file). From that position on, the process scrolls forward to the next item
declaration, where it starts its work. It proceeds until it hits the item
declaration that follows position $i/n\times{}N$. Since the file is UTF-8
encoded, the initial position may hit the middle of a character that is
encoded by more than one byte. This is not a problem because the UTF-8
standard can distinguish the middle-bytes from the initial bytes in an encoded
stream.

#### Steps.

We generate YAGO on a Unix machine with 90 CPUs and 800GB of RAM. We proceed
in 6 steps:

1\. Create schema:

The manual definition of the schema is loaded, and the relevant parts of the
Schema.org-taxonomy are extracted, as described in Section 3.2. This process
operates only on manually defined files and hence terminates in less than a
second on our machine.

2\. Create taxonomy:

Wikidata is parsed for classes, and a loop-free taxonomy is constructed, as
described in Section 3.3. This step is parallelized and takes 4 hours.

3\. Create facts:

Wikidata is parsed for facts, each predicate is mapped to a YAGO predicate as
described in Section 3.5, the subject of the fact is type-checked, and the
objects are type checked if they are literals. The objects that are not
literals cannot yet be type checked because we do not yet have a complete list
of all instances at this stage. This step, likewise, is parallelized and takes
4 hours.

4\. Type-check facts:

The previous step has given us a list of facts, which also contains for each
instance the class that it belongs to. We load this list into memory and run
through all facts to type-check the object of each fact. This step runs on
YAGO data, and not on Wikidata, and thus does not need parallelization. It
runs in 1:30h.

5\. Create ids:

Among those facts that survived the type check, we map each entity to its
legible YAGO name, as described in Section 3.5. This takes 1h.

6\. Create statistics:

Debugging and testing are an integral part of the development. The last step
counts the number of instances per class and of facts per predicate. It
creates a visualization of the taxonomy and a random selection of entities for
manual check. This process also takes one hour.

Thus, the overall process takes about 12 hours.

##  5 Result

|  Wikidata |  YAGO 4 |  YAGO 4.5
---|---|---|---
Individuals | 103M | 67M (37M) | 49M
of which generic | 0 | 0 | 62k
Classes | 2.8M | 10k | 132k
Predicates | 11k | 140 (124) | 112
Facts | 500M | 343M (89M) | 109M
Type facts | 106M | 70M (33M) | 51M
Label facts | 795M | 303M | 468M
Meta facts | 12M | 2.5M | 7M
Dump size | 766GB | 280GB | 109GB
Table 1: Size of YAGO 4.5. Facts exclude _type_ , _label_ , _comment_ ,
_alternateName_ , _sameAs_ , and _mainEntityOfPage_ facts. In brackets:
without redundant properties, properties describing literals, and properties
describing scholarly articles.

#### Size.

Table 1 shows the statistics of YAGO 4.5 and puts them in perspective with
Wikidata and YAGO 4. YAGO 4.5 has fewer properties than YAGO 4. This is due to
the removal of inverse properties, which we removed under Design Principle 1
(e.g., we removed _hasParent_ because we already have _hasChild_ ; 6 such
cases); properties related to scholarly articles (_citation_ etc., 4 in total,
which we removed according to the discussion in Section 3.3); biochemical
properties (11 in total, removed because of a lack of expertise in our team);
and properties of numerical literals (_value_ etc., 6 in total, removed
because literals became simple values in YAGO 4.5). The loss is thus
deliberate121212A detailed comparison of properties in YAGO 4 and 4.5 is in
our Design Document. The same goes for the facts of YAGO 4: 238M facts
describe scholarly articles (mainly citations, pagination, and publication
dates), 13M facts describe literals, and 2.5M facts are redundant because of
an inverse property. If these are discarded, the new YAGO contains slightly
more facts. The dump size is still smaller, because we use the Turtle file
format instead of NT. Concerning the classes, the picture is as rosy as
intended: YAGO 4.5 has vastly more classes.

#### Quality.

To evaluate the quality of the new KB, we draw inspiration from the criteria
for ontology evaluation that a recent survey identified [51]: Consistency
refers to the absence of logical contradictions.131313 First, we computed the
deductive closure, and then, we verified the disjointness axioms using
_rdflib_ (https://rdflib.readthedocs.io). Finally, we validated the SHACL
constraints utilizing _pySHACL_ (https://github.com/RDFLib/pySHACL) to ensure
the correctness and consistency of our data.  Complexity refers to the extent
to which the ontology is complicated. We propose to measure it (1) by the
number of top-level classes (i.e., the number of direct subclasses of _Thing_
; the fewer the better), and (2) by the average number of paths from an
instance to the root in the taxonomic tree (the fewer the better). Modularity
is the degree to which the ontology is composed of discrete subsets. In our
case, these subsets are the disjoint classes, and we report the number of
(non-redundant and actively enforced) class disjointness statements as an
indicator. Conciseness requires the absence of redundancies, and we count the
number of taxonomic loops, taxonomic links that are redundant because of
transitivity, and relations whose inverse also exists. Understandability is
the degree to which the ontology can be comprehended. This is difficult to
operationalize, but we can at least report the percentage of identifiers that
have human-readable names. Coverage refers to the degree that the ontology
covers the domain knowledge. We report the number of classes and facts per
instance (excluding labels etc.).

Table 2 shows that Wikidata has vastly more facts per instance than YAGO. This
is to be expected, as YAGO is a clean subset of Wikidata. YAGO 4, too, has
more facts per instance than YAGO 4.5. However, this is mainly due to the 174M
facts about citations for 40M scholarly articles. YAGO 4 also has a lower
number of paths to the root, which is due to its sparse taxonomy. On all other
measures, YAGO 4.5 scores much better than both YAGO 4 and Wikidata: It has a
clean upper level taxonomy of just 9 top-level classes – instead of the dozens
of Wikidata or the thousands that YAGO 4 attached to the taxonomic root for
lack of good intermediate classes. YAGO 4.5 is also free of all types of
redundancy that plagued Wikidata and YAGO 4. There are just 2.4 paths to the
taxonomic root on average for an individual – instead of the dozens in
Wikidata. At the same time, YAGO 4.5 nearly replicates the taxonomic depth of
Wikidata, with 8 classes per individual – twice as many as in YAGO 4.

Criterion | Operationalization |  Wikidata |  YAGO 4 |  YAGO 4.5
---|---|---|---|---
Consistency | Absence of contradictions | no | yes | yes
Complexity | Top-level classes | 41 | 2714 | 9
| Paths to root | 35 | 1.1 | 2.4
Modularity | Disjointness axioms | 0 | 18 | 24
Conciseness | Taxonomic loops | 62 | 0 | 0
| Redundant taxonomic links | 377k | 1216 | 0
| Redundant relations | 118 | 6 | 0
| Classes without instances | 2.6M | 73 | 0
Understandability | Human-readable names | 0% | 89% | 96%
Coverage | Classes per instance | 8.4 | 3.6 | 8
| Facts per instance | 4.8 | 5.1 | 2.6
Table 2: Quality measures, inspired by [51]

#### Data Format.

The file format of YAGO is Turtle. We separate the subject, predicate, object,
and dot by a tabulator, so that our files are de facto also TSV files. YAGO is
split into the following files:

  * •

Schema: Upper taxonomy, property definitions, and SHACL constraints.

  * •

Taxonomy: The entire taxonomy of YAGO (all _subClassOf_ facts).

  * •

Facts: All facts about entities that have an English Wikipedia page.

  * •

Beyond Wikipedia: All facts about other entities.

  * •

Meta: All temporal annotations (in the RDF* file format [13]).

Downstream applications can load only the files they need, and exclude, e.g.,
meta facts or facts about entities that do not have an English Wikipedia page.

#### Applications.

The YAGO KB has found quite a number of applications in the past [37]. As
every user is free to download the KB and to use it (or not), we do not have
an overview of the projects that use YAGO. We just know that the KB (in
various versions) has been downloaded about 6000 times during the past year
(heuristically excluding bots). We expect our new YAGO to be even more useful,
as it provides fine-grained information about instances by the new classes.
For example, whereas the singer Elvis Presley was in a mere 4 classes in YAGO
4 (Thing, Person, Human, Twin), our new taxonomy tells us also that he was an
actor, a singer, a film person, and an artist.

##  6 Conclusion

In this paper, we have presented a method to merge the Wikidata taxonomy with
the taxonomy of Schema.org, giving rise to a new version of the YAGO KB, YAGO
4.5. Our work adds a rich layer of informative classes to YAGO, while at the
same time keeping YAGO logically consistent.

Several open challenges remain: First, we did not include the classes of
_BioChemEntities_ and _MedicalEntities_ for lack of domain-specific expertise.
These can be added in the future. Now that the general framework is in place,
more upper classes and properties can be defined. An interesting avenue of
research in this direction would be the (automated) translation of textual
class descriptions (such as _de facto embassy_ (Q5244910)) into logical
properties and constraints [19]. This would require a better understanding of
commonsense statements [33]. A more fundamental challenge is the treatment of
classes as subjects of properties. We would like to say, e.g., that cats eat
mice (which is a general statement on cats, which is why we currently model
_cat_ (Q146) as an instance), and that _Garfield_ (Q767120) is a cat (which
requires _cat_ to be a class). These are exciting challenges for the Semantic
Web community as a whole.

Resource Availability Statement: The Web page of YAGO 4.5 is https://yago-
knowledge.org. It contains download links for the data, the Design Document,
the documentation, the list of publications, and contributors. The Web page
also offers an interactive browser for the KB. The SPARQL endpoint is at
https://yago-knowledge.org/sparql. The URIs of YAGO 4.5 are dereferenceable.
YAGO 4.5 is available with Creative Commons Attribution-ShareAlike License (as
imposed by the license of schema.org). The source code of YAGO 4.5 is
available via Github at https://github.com/yago-naga/yago-4.5 under a Creative
Commons Attribution license.

Acknowledgement. This work was partially funded by the grant
ANR-20-CHIA-0012-01 (“NoRDF”).

## References

  * [1] Abián, D., Guerra, F., Martínez-Romanos, J., Trillo-Lado, R.: Wikidata and dbpedia: a comparative study. In: IKC (2018)
  * [2] Arp, R., Smith, B., Spear, A.D.: Building ontologies with basic formal ontology. Mit Press (2015)
  * [3] Ashburner, M., Ball, C.A., Blake, J.A., Botstein, D., Butler, H., Cherry, J.M., Davis, A.P., Dolinski, K., Dwight, S.S., Eppig, J.T., et al.: Gene ontology: tool for the unification of biology. Nature genetics (2000)
  * [4] Auer, S., Bizer, C., Kobilarov, G., Lehmann, J., Cyganiak, R., Ives, Z.G.: Dbpedia: A nucleus for a web of open data. In: ISWC (2007)
  * [5] Biega, J.A., Kuzey, E., Suchanek, F.M.: Inside YAGO2s: A Transparent Information Extraction Architecture . In: WWW demo track (2013)
  * [6] Bollacker, K.D., Evans, C., Paritosh, P.K., Sturge, T., Taylor, J.: Freebase: a collaboratively created graph database for structuring human knowledge. In: SIGMOD (2008)
  * [7] Brasileiro, F., Almeida, J.P.A., Carvalho, V.A., Guizzardi, G.: Applying a Multi-Level Modeling Theory to Assess Taxonomic Hierarchies in Wikidata. In: WWW (2016)
  * [8] Cao, B., Lin, H., Han, X., Sun, L., Yan, L., Liao, M., Xue, T., Xu, J.: Knowledgeable or educated guess? revisiting language models as knowledge bases. arXiv preprint arXiv:2106.09231 (2021)
  * [9] Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H.W., Sutton, C., Gehrmann, S., et al.: Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 (2022)
  * [10] Färber, M., Bartscherer, F., Menne, C., Rettinger, A.: Linked data quality of dbpedia, freebase, opencyc, wikidata, and yago. Semantic Web (2018)
  * [11] Galárraga, L., Teflioudi, C., Hose, K., Suchanek, F.M.: Fast Rule Mining in Ontological Knowledge Bases with AMIE+ . In: VLDBJ (2015)
  * [12] Guha, R.V., Brickley, D., Macbeth, S.: Schema.org: evolution of structured data on the web. Commun. ACM (2016)
  * [13] Hartig, O.: Rdf* and sparql*: An alternative approach to annotate statements in RDF. In: ISWC poster track (2017)
  * [14] Hoffart, J., Suchanek, F.M., Berberich, K., Weikum, G.: YAGO2: A Spatially and Temporally Enhanced Knowledge Base from Wikipedia . In: Artificial Intelligence (2013)
  * [15] Jiang, M., Song, X., Zhang, J., Han, J.: Taxoenrich: Self-supervised taxonomy completion via structure-semantic representations. In: WWW (2022)
  * [16] Lee, D., Shen, J., Kang, S., Yoon, S., Han, J., Yu, H.: Taxocom: Topic taxonomy completion with hierarchical discovery of novel topic clusters. In: WWW (2022)
  * [17] Lenat, D.B., Guha, R.V.: Representation and inference in the cyc project (1990)
  * [18] Liu, Q., Yogatama, D., Blunsom, P.: Relational memory-augmented language models. TACL (2022)
  * [19] Lu, X., Liu, J., Gu, Z., Tong, H., Xie, C., Huang, J., Xiao, Y., Wang, W.: Parsing natural language into propositional and first-order logic with dual reinforcement learning. In: Coling (2022)
  * [20] Mahdisoltani, F., Biega, J.A., Suchanek, F.M.: YAGO3: A Knowledge Base from Multilingual Wikipedias . In: CIDR (2015)
  * [21] Mao, Y., Ren, X., Shen, J., Gu, X., Han, J.: End-to-end reinforcement learning for automatic taxonomy induction. In: ACL (2018)
  * [22] Mao, Y., Zhao, T., Kan, A., Zhang, C., Dong, X.L., Faloutsos, C., Han, J.: Octet: Online catalog taxonomy enrichment with self-supervision. In: SIGKDD (2020)
  * [23] Mascardi, V., Cordì, V., Rosso, P.: A Comparison of Upper Ontologies. In: AI/TABOO joint workshop (2007)
  * [24] Masolo, C., Borgo, S., Gangemi, A., Guarino, N., Oltramari, A., Schneider, L.: The wonderweb library of foundational ontologies. WonderWeb deliverable D (2002)
  * [25] de Melo, G., Suchanek, F.M., Pease, A.: An Application: Extending SUMO with Wikipedia . In: Adam Pease: Ontology: A Practical Guide. (2011)
  * [26] Mika, P.: On schema. org and why it matters for the web. IEEE Internet Computing (2015)
  * [27] Miller, G.A.: WordNet: An electronic lexical database. MIT press (1998)
  * [28] Navigli, R., Ponzetto, S.P.: Babelnet: The automatic construction, evaluation and application of a wide-coverage multilingual semantic network. Artif. Intell. (2012)
  * [29] Noy, N., Gao, Y., Jain, A., Narayanan, A., Patterson, A., Taylor, J.: Industry-scale knowledge graphs: Lessons and challenges: Five diverse technology companies show how it’s done. Queue (2019)
  * [30] Noy, N., Uschold, M., Welty, C.: Representing classes as property values on the semantic web (2005), https://www.w3.org/TR/2005/NOTE-swbp-classes-as-values-20050405/
  * [31] OpenAI: Chatgpt plugins. https://openai.com/blog/chatgpt-plugins (2023)
  * [32] Paris, P.H., Aoud, S.E., Suchanek, F.M.: The Vagueness of Vagueness in Noun Phrases. In: AKBC (2021)
  * [33] Paris, P.H., Suchanek, F.M.: Non-named entities - the silent majority. In: ESWC short paper track (2021)
  * [34] Pellissier Tanon, T., Vrandečić, D., Schaffert, S., Steiner, T., Pintscher, L.: From freebase to wikidata: The great migration. In: WWW (2016)
  * [35] Piscopo, A., Simperl, E.: Who Models the World?: Collaborative Ontology Creation and User Roles in Wikidata (2018)
  * [36] Razniewski, S., Yates, A., Kassner, N., Weikum, G.: Language models as or for knowledge bases. arXiv preprint arXiv:2110.04888 (2021)
  * [37] Rebele, T., Suchanek, F.M., Hoffart, J., Biega, J.A., Kuzey, E., Weikum, G.: YAGO: a multilingual knowledge base from Wikipedia, Wordnet, and Geonames . In: ISWC (2016)
  * [38] Shen, J., Shen, Z., Xiong, C., Wang, C., Wang, K., Han, J.: Taxoexpan: Self-supervised taxonomy expansion with position-enhanced graph neural network. In: WWW (2020)
  * [39] Shen, J., Wu, Z., Lei, D., Zhang, C., Ren, X., Vanni, M.T., Sadler, B.M., Han, J.: Hiexpan: Task-guided taxonomy construction by hierarchical tree expansion. In: SIGKDD (2018)
  * [40] Sheng, Z., Wang, X., Shi, H., Feng, Z.: Checking and handling inconsistency of dbpedia. In: WISM (2012)
  * [41] Shenoy, K., Ilievski, F., Garijo, D., Schwabe, D., Szekely, P.A.: A study of the quality of Wikidata. J. Web Semant. (2022)
  * [42] Song, X., Shen, J., Zhang, J., Han, J.: Who should go first? a self-supervised concept sorting model for improving taxonomy expansion. arXiv preprint arXiv:2104.03682 (2021)
  * [43] Speer, R., Chin, J., Havasi, C.: Conceptnet 5.5: An open multilingual graph of general knowledge. In: AAAI (2017)
  * [44] Suchanek, F.M., Kasneci, G., Weikum, G.: Yago - A Core of Semantic Knowledge . In: WWW (2007)
  * [45] Suchanek, F.M., Varoquaux, G.: On Language Models and Symbolic Representations. In: The Conversation (2022)
  * [46] Sung, M., Lee, J., Yi, S., Jeon, M., Kim, S., Kang, J.: Can language models be biomedical knowledge bases? arXiv preprint arXiv:2109.07154 (2021)
  * [47] Tanon, T.P., Weikum, G., Suchanek, F.M.: YAGO 4: A Reason-able Knowledge Base . In: ESWC (2020)
  * [48] Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al.: Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)
  * [49] Vout, T.: Chatgpt’s ‘snow white’ problem: The danger of common knowledge. https://medium.com/oxford-semantic-technologies/chatgpts-snow-white-problem-the-danger-of-common-knowledge-89c850e0e596 (2023)
  * [50] Vrandecic, D., Krötzsch, M.: Wikidata: a free collaborative knowledgebase. Commun. ACM (2014)
  * [51] Wilson, R., Indika, W., Ginige, A.: Analysis of ontology quality dimensions, criteria and metrics. In: ICCSA. Springer (2021)
  * [52] Wolfram, S.: Wolfram alpha as the way to bring computational knowledge superpowers to chatgpt. https://writings.stephenwolfram.com/2023/01/wolframalpha-as-the-way-to-bring-computational-knowledge-superpowers-to-chatgpt/ (2023)