# Multi-objective Reinforcement learning from AI Feedback

Marcus Williams
<EMAIL_ADDRESS>
(June 2024)

###### Abstract

This paper presents Multi-Objective Reinforcement Learning from AI Feedback
(MORLAIF), a novel approach to improving the alignment and performance of
language models trained using reinforcement learning from AI feedback (RLAIF).
In contrast to standard approaches that train a single preference model to
represent all human preferences, MORLAIF decomposes this task into multiple
simpler principles, such as toxicity, factuality, and sycophancy. Separate
preference models are trained for each principle using feedback from
GPT-3.5-Turbo. These preference model scores are then combined using different
scalarization functions to provide a reward signal for Proximal Policy
Optimization (PPO) training of the target language model. Our experiments
indicate that MORLAIF outperforms the standard RLAIF baselines and that
MORLAIF can be used to align larger language models using smaller ones.
Surprisingly, the choice of scalarization function does not appear to
significantly impact the results.

##  1 Introduction

Recent advancements in large language models (LLMs) have led to remarkable
performance across a wide range of natural language tasks (Bommasani et al.
(2021); Brown et al. (2020)). However, ensuring that these models behave in
alignment with human values and preferences remains a significant challenge
(Kenton et al. (2021)). Reinforcement learning from human feedback (RLHF) has
emerged as a promising approach to address this issue by training models to
optimize for human-specified reward functions (Christiano et al. (2017);
Stiennon et al. (2020)), but many issues with RLHF have been identified, such
as the limited ability of humans to evaluate responses and reward hacking of
the preference model (Casper et al. (2023)). In a standard RLHF setup, a
preference model is trained on human comparisons of model outputs to represent
human preferences. This preference model is then used to provide an RL
training signal to update the target language model’s behavior. Reinforcement
Learning from AI Feedback (RLAIF) outsources the human comparison step to
another language model. In Bai et al. (2022b), they train the preference model
based on a set of “constitutional” principles that the model should adhere to.
However, these principles are often quite general, containing many different
desiderata, and the preference modeling task of capturing all human
preferences in a single model is highly complex. To address these limitations,
we propose Multi-Objective Reinforcement Learning from AI Feedback (MORLAIF).
This involves decomposing the preference modeling into multiple distinct
principles, such as factuality, toxicity, and conciseness, that are more
specific and well-defined. Separate preference models are trained for each of
these principles using feedback from strong language models, in our case
GPT-3.5-Turbo. During RL training, the outputs from these preference models
are combined using a scalarization function to provide the reward signal for
optimizing the target language model. MORLAIF has several appealing
properties. First, by breaking down preference modeling into specific
principles, the feedback collection and preference model training becomes a
more straightforward and well-defined task, which we hypothesize will lead to
improved preference model performance and better alignment of the resulting
language models. Second, MORLAIF allows for training on a large number of
principles without necessarily diluting the reward signal. Third, the multi-
objective framing provides a more interpretable system, where we can examine
how model outputs score on each principle. Finally, MORLAIF allows for easily
adjusting preference model behavior by modifying the scalarization function
without expensive retraining of the preference models.

##  2 Multi-Objective Reinforcement Learning and scalarization functions

Multi-objective reinforcement learning (MORL) is a framework that enables the
learning of policies to simultaneously optimize multiple objectives, which may
potentially conflict with each other. MORL has been extensively studied in
various domains, particularly in robotics Roijers et al. (2013) and game-
playing Mossalam et al. (2016). One of the primary challenges in MORL lies in
combining the diverse objectives into a single reward signal that can be used
by reinforcement learning algorithms. Several approaches have been employed to
address this challenge, including scalarization methods such as weighted sums
Van Moffaert et al. (2013), thresholding Gábor et al. (1998), and Pareto-based
methods Moffaert & Nowé (2014).

For our setup, we have a set of reward functions $R_{1},\ldots,R_{n}$ and wish
to maximize:

$J(\pi)=\mathbb{E}_{\xi\sim\pi}\left[f\left(\sum_{t=0}^{\infty}\gamma^{t}R_{1}\left(s_{t},a_{t},s_{t+1}\right),\ldots,\sum_{t=0}^{\infty}\gamma^{t}R_{n}\left(s_{t},a_{t},s_{t+1}\right)\right)\right]$
(1)

where $\pi$ is the policy, $\xi$ is a trajectory sampled from the policy,
$\gamma$ is the discount factor, and $f$ is a scalarization function that
combines the multiple objectives into a single scalar value. The scalarization
function $f$ is used to balance the importance of each objective and provide a
single reward signal for our RL algorithm to optimize. In our language
modelling case we can drop the $s_{t+1}$ as we have deterministic transitions.

To ensure that the optimization remains tractable with gradient-based methods,
we must constrain the scalarization function $f$ to be monotonic in each
argument (Roijers et al. (2013)). A scalarization function $f(u_{1},\ldots
u_{i},\ldots u_{n})$ is said to be monotonic in an argument $u_{i}$ if:

$\displaystyle\forall x,y\;(x\geq y\implies f(u_{1},\ldots,x,\ldots,u_{n})\geq
f(u_{1},\ldots,y,\ldots,u_{n}))\ \lor$ $\displaystyle\forall x,y\;(x\geq
y\implies f(u_{1},\ldots,x,\ldots,u_{n})\leq f(u_{1},\ldots,y,\ldots,u_{n}))$

In this paper we will test the following scalarization functions:

  * •

Weighted Linear Combination: Perhaps the simplest MORL objective is a weighted
linear combination. Theoretically, a weighted linear combination MOMDP can
always be reformulated as a standard MDP with a single reward function (Skalse
& Abate (2023)), thus not gaining anything over just using one reward
function. In practice however, using MORL often performs better as it eases
the specification of said reward function. Max-average is a special case of a
linear combination where all the weights are equal to 1. This scalarization
function has the disadvantage that it requires an explicit weighting of our
principles, as a starting point we will find weights using a logistic
regression on a human preference dataset. Mathematically a weighted linear
combination looks like:

$f(R_{1},\ldots,R_{n})=\sum_{i=1}^{n}w_{i}R_{i}$ (2)

  * •

Worst-Case Optimization: This scalarization function which is also known as
minimax, the Rawlsian social welfare function or max-min optimization, aims to
find a policy that performs well in the worst-case scenario among all the
objectives. Here we seek to maximize the minimum reward across all objectives.
Minimax optimization is particularly useful in risk-averse domains where the
agent wants to ensure a certain level of performance for all objectives.
However, this approach may lead to overly conservative policies and may not
fully exploit the potential trade-offs between objectives. Mathematically:

$f(R_{1},\ldots,R_{n})=\min_{i}\\{R_{i}\\}$ (3)

  * •

Soft max-min: Soft max-min is similar to Worst-Case Optimization except that
we optimize the softmin instead of the minimum. In theory this should give us
some of the benefits of max-min but lead to less conservative policies. This
scalarization introduces an extra hyperparameter, the temperature $T$, which
controls how soft the softmin is. This hyperparameter may need some tuning for
good results. $f$ becomes:

$f(R_{1},\ldots,R_{n})=\sum_{i=1}^{n}\text{softmin}(R_{i})R_{i}=\sum_{i=1}^{n}\frac{e^{-\frac{R_{i}}{T}}}{\sum_{j=1}^{n}e^{-\frac{R_{j}}{T}}}R_{i}$
(4)

  * •

Uncertainty-Weighted Optimization: Uncertainty-Weighted Optimization takes
into account the uncertainty associated with each objective when making
decisions. In this approach, we subtract a term proportional to the variance
between the different reward functions. This means that for cases where the
reward functions agree more we have a larger weight update than when there is
a large difference between the reward functions. This scalarization function
introduces the extra hyperparameter $\lambda$ which controls how much to
punish uncertainty. Coste et al. (2024) notes that uncertainty weighted
optimization works well for most reasonable choices of $\lambda$.
Mathematically our scalarization function is:

$f(R_{1},\ldots,R_{n})=\frac{1}{n}\sum_{i=0}^{n}R_{i}-\lambda\frac{1}{n}\sum_{i=0}^{n}\left(R_{i}-\frac{1}{n}\sum_{i=0}^{n}R_{i}\right)^{2}$
(5)

  * •

Lower Quantile Optimization: Lower quantile optimization focuses on optimizing
a lower quantile of the reward functions. This is somewhat related to worst
case optimisation where only the lowest reward matters, here instead the
lowest quantile $Q_{\alpha}$ of rewards matter, where $\alpha$ is the fraction
of reward functions we want to include. In this paper we test $\alpha=1/3$.
Lower quantile optimization shares similarities with worst-case optimization
by prioritizing the objectives with the lowest scores. However, it considers
multiple rewards simultaneously, leading to less conservative policies
compared to worst-case optimization. Mathematically, the lower quantile
optimization scalarization function is expressed as:

$f(R_{1},\ldots,R_{n})=Q_{\alpha}(\\{R_{i}\\}_{i=1}^{n})$ (6)

  * •

Max-Median: As the name suggests Max-median optimizes the median reward. It is
likely more robust to outliers than max-average but we lose the flexibility of
an arbitrary linear weighting.

$f(R_{1},\ldots,R_{n})=\text{median}(\\{R_{i}\\}_{i=1}^{n})$ (7)

  * •

Bernoulli-Nash: The Bernoulli-Nash social welfare function is a scalarization
approach that aggregates reward functions multiplicatively, unlike the
additive aggregation used in other methods. It is notable for being the only
scalarization function that simultaneously satisfies several theoretically
desirable properties, including Pareto Optimality, Symmetry, Scale Invariance,
and Independence of Irrelevant Alternatives Kaneko & Nakamura (1979).
Mathematically this function takes the form:

$f(R_{1},\ldots,R_{n})=\sqrt[n]{\prod_{i=1}^{n}R_{i}}$ (8)

##  3 Methodology

Figure 1: Diagram illustrating the training setup.

The methodology consists of two main stages which are illustrated in figure 1,
the preference modeling stage and the reinforcement learning from AI feedback
stage.

###  3.1 Preference Modeling:

First a SFT model is created by finetuning a base model on the openassistant-
guanaco dataset (Dettmers et al. (2023)). The SFT model then generates pairs
of responses for a given set of prompts, in this case the prompt portion of
Anthropic’s HH-RLHF (Bai et al. (2022a)). A feedback model then evaluates
which response is better for each individual principle defined in the study.
The ratings from the feedback model are used to train separate preference
models for each principle. The preference models can be either full models or
Low Rank Adaptations (LoRAs) (Hu et al. (2021)).

###  3.2 Reinforcement Learning from AI Feedback:

During the RL loop a MORL scalarization function is used to combine the
ratings from each preference model. The different scalarization functions
tested are listed in section 2 above. The combined score from the
scalarization function serves as a reward signal for training the target model
using Proximal Policy Optimization (PPO). We use the implementation of PPO use
in Ziegler et al. (2020). The initial checkpoint used is the same SFT model we
trained on openassistant-guanaco.

###  3.3 Model details

The study uses 5 target models, namely GPT-2-medium/large/XL (Radford et al.
(2019)), Gemma-2B (Team et al. (2024)) and Llama-7B (Touvron et al. (2023)).
For the preference models both full finetunes and LoRAs were experimented
with, but based on results from GPT-2 it was determined that LoRAs perform
just as well as full models and consume 0 extra VRAM compared to single
objective RLAIF during the RLAIF phase so these were primarily used. The
preference models are typically based on the same model as the one being
trained, although we also conduct one experiment where we train Llama-7B using
GPT-2-medium as the preference models. GPT-3.5-Turbo was employed as the
feedback model to rate response pairs according to each principle.

###  3.4 Evaluation

We evaluate both the performance of the multi-objective preference models and
the performance of the final models. The PMs are evaluated by measuring their
accuracy on the HH-RLHF testset.

We evaluate the performance of the target models by comparing their win rates
against a single-objective baseline. We do this in two ways: first, we employ
human crowdworkers for two of our most interesting comparisons; second, we
utilize GPT-4-Turbo, for all comparisons to provide a comprehensive assessment
of the target models’ performance.

For the human win-rates we recruited people via the crowdworking platform
Amazon MTurk. Workers were asked to have a conversation where two options were
presented at each step. These options came from two different models and the
human was asked to rate which response was better or if they were equally
good. The conversations lasted a maximum of 10 turns or until the human choose
to start a new one. The conversations lasted on average 5.2 turns. For more
details on how we conducted this and how we reduced “cheating” with LLMs see
appendix B.

For the GPT-4-Turbo win-rates we used the HH-RLHF test set and generated one
response for each model and then prompted GPT-4-Turbo to rate which of the two
was better. The preferred response was the one with higher token probability.
Unlike in the human study we do not give GPT-4-Turbo the option of choosing
“Tie”.

###  3.5 Baselines

For the single-objective RLAIF baseline we essentially copied the procedure in
Bai et al. (2022b). We sample principles from their constitution and use the
same prompt they used. We also include few-shot examples in the same way they
did.

Another baseline which was used was training 12 preference models on the
standard RLAIF setup and optimizing our MORL scalarization functions with
these PMs. This would eliminate the possibility that the benefit arises from
using many PMs, along the lines of Coste et al. (2024), rather than from
decomposing human preferences.

###  3.6 Principles

The full list of principles used is: helpfulness, ethicality, factuality,
toxicity, sycophancy, empathy, relevance, context, bias, understandability,
repetitiveness, detail and conciseness.

Unless overwise stated, we use the full 12 principles. We also conduct an
ablation experiment with using fewer principles and see how that reduces
performance. These principles were inspired by Anthopic’s constitution (Bai et
al. (2022b)) and the principles used in Go et al. (2024).

###  3.7 Code

The code used in this project can be found at:
https://github.com/carolius/Multi-Objective-Reinforcement-Learning-from-AI-
Feedback

##  4 Results

###  4.1 Preference Model Accuracy:

(a)

(b)

Figure 2: (a) shows the accuracy the different models achieve when trained as
a preference model for one of the different principles. (b) shows the accuracy
of our Multi-Objective PMs for the different scalarization functions and our
single objective baselines.

In figure 2(a) we can see the accuracy increases with model size as expected.
Comparing to figure 2(b) we can see that the accuracies for specific
principles are generally much higher than for PMs on the true labels. The
accuracies also seem to grow faster with increasing model size than in the
single principle case. There is quite a large difference between the highest
and lowest accuracy principle, roughly 0.1 for GPT-2-medium.

Figure 2(b) shows the accuracy of different MORL scalarization functions. Note
that uncertainty weighted scalarization is not included here since the way in
which it differs from linear is intended to affect step sizes during training,
which is not relevant here. We can see that all MORL objectives perform better
than standard single PM RLAIF, but surprisingly they all seem to perform
equally well despite being quite different. Based on this data, it is not
possible to say which, if any, of the objectives performs the best. Bakker et
al. (2022) also note that different social welfare functions seem to behave
the same in practice, although their setup is significantly different from
ours. We can also note that the accuracy for multi-objective GPT-2-medium is
slightly higher than for single-objective Llama-2-7b, despite having roughly
1/20 the number of parameters and being trained on roughly 1/250 the number of
tokens. This prompted an experiment to align Llama-2-7b using GPT-2-medium.
The figure also includes the performance ceiling of using our 12 principles
and represents having PMs that have 100% accuracy on every principle.

###  4.2 Win Rates:

(a)

(b)

Figure 3: (a) shows the human judged win-rate of MORLAIF Llama-2-7b and one
trained with GPT-2-medium PMs over the single objective version. (b) shows the
GPT-4-Turbo judged win-rate for the different models.

Figure 3(a) shows the results of the human crowdworker preference experiments.
We can see that MORLAIF Llama-2-7b is preferred to Single Objective RLAIF by a
large margin, and that a version trained with GPT-2-medium preference models
performs the same as the SORLAIF model. As discussed earlier, the fact that
using a much weaker multi-objective model results in the same performance as a
stronger single-objective model indicates that MORLAIF could be a promising
scalable oversight solution.

Figure 3(b) instead shows the win rates as judged by GPT-4-Turbo. Unlike the
human experiments, GPT-4 did not get the option of a tie. We see that the win
rate is very high for GPT-2-medium but that it decreases with model size,
although it is still significant. We can also note that the GPT-4 rated win
rate for Llama-2-7b is approximately the same as the human-rated win rate +
tie rate, indicating that GPT-4 is probably a reasonably accurate proxy for
human evaluation.

Figure 4: Comparison between the win-rates for the different MORL
scalarization functions. Note that $e_{i,j}=1-e_{j,i}$.

It could have been the case that while the different MORL objectives score the
same in terms of accuracy, they differ in how they affect the final RLAIF
model. Figure 4 excludes that possibility, at least when using GPT-2-medium
according to GPT-4. While again all MORL objectives result in a model much
better than using a single PM, the differences between them are
indistinguishable from noise.

###  4.3 Correlation Between the Principles and Ablation of Principles

(a)

(b)

Figure 5: (a) shows the correlations between the feedback for the different
principles. (b) shows how the multi-objective PM accuracy depends on the
number of principles for GPT-2-medium, Llama-2-7b, and the theoretical ceiling
which represents 100% accuracy on each individual principle.

Figure 5(a) illustrates the correlations between the feedback for the
different principles and single objective RLAIF. While some of the
correlations are relatively strong, the preference models are clearly not
being trained to detect exactly the same thing. Notably, sycophancy exhibits
weak correlations with the other principles. Furthermore, sycophancy is the
only preference model that receives a negative weight in the linear
scalarization function logistic regression, indicating that the preferred
responses are more sycophantic than the rejected ones. One benefit of MORLAIF
is that we could explicitly set the sycophancy weight to zero or a positive
weight if we want to reduce sycophancy without needing to go back and redo our
feedback collection process. Doing so slightly reduces performance as measured
by our dataset; however, it may increase performance on what we actually care
about. MORLAIF might be a way to achieve a more aligned model despite having a
flawed dataset.

Figure 5(b) shows how the performance decreases when we use fewer principles.
We remove principles in the order of their weight, starting with the lowest,
which was sycophancy.

##  5 Related work

Our work on MORLAIF builds upon and extends several lines of research,
including Reinforcement Learning from Human Feedback (RLHF), Constitutional
AI, Multi-Objective Reinforcement Learning (MORL), and task decomposition. In
this section, we provide an overview of the most relevant prior work in each
of these areas.

###  5.1 Reinforcement Learning from Human Feedback (RLHF)

Reinforcement learning from human feedback (RLHF) is a promising approach for
aligning AI systems with human preferences. The key idea is to learn a reward
function based on human feedback and use this learned reward to optimize the
agent’s behavior via reinforcement learning. Christiano et al. (2017)
pioneered this approach, using human preferences to train a reward model for
Atari games and robotic control tasks. Subsequent work has extended RLHF to
more complex domains, including summarization (Stiennon et al. (2020); Ziegler
et al. (2019)), dialogue (Jaques et al. (2019)), and language modeling (Ouyang
et al. (2022)). A core component of RLHF is the reward modeling step, where
human preferences are distilled into a learned reward function. Our work
builds on these ideas by decomposing the reward modeling into multiple
principle-specific preference models.

###  5.2 Constitutional AI and RLAIF

Constitutional AI, proposed by Bai et al. (2022b), is a framework for training
AI systems to adhere to a set of predefined principles or rules, analogous to
a constitution. One of the methods they propose is Reinforcement learning from
AI feedback, RLAIF. RLAIF works much like RLHF except that an LLM is now
choosing the best response based on principles sampled from a constitution
rather than utilizing humans. The authors demonstrate Constitutional AI and
RLAIF on open-ended dialogue, and work on using RLAIF for summarization has
been done by Lee et al. (2023). While both Constitutional AI and our MORLAIF
approach aim to constrain model behavior based on predefined principles, there
are several key differences. Constitutional AI uses a single set of principles
and generates both critiques and revisions in the supervised phase, while
MORLAIF trains separate preference models for each moral principle. In
Constitutional AI, each principle has to be quite broad and cover most of what
we mean by human preferences in order to ensure convergence. In MORLAIF, by
contrast, having specific and unique principles is preferable, giving us more
fine-grained control and interpretability.

###  5.3 Task Decomposition

Task decomposition is a general strategy for simplifying complex tasks by
breaking them down into more manageable subtasks. In the context of
reinforcement learning, task decomposition has been used to improve sample
efficiency and generalization (Ghosh et al. (2018); Peng et al. (2022)).
MORLAIF can be seen as a form of task decomposition applied to reward
modeling, where the complex task of representing human preferences is broken
down into simpler principle-specific subtasks. This decomposition allows for
more targeted data collection and potentially simplifies the learning problem
for each preference model.

###  5.4 Decomposing Human Preferences

Go et al. (2024) proposed Compositional Preference Models (CPMs), which
decompose the reward modeling task into multiple interpretable features.
Rather than training a single preference model end-to-end, CPMs first define a
set of human-interpretable features (e.g., helpfulness, specificity,
factuality). They then use a prompted language model to assign scalar scores
to each feature for a given query-response pair. Finally, a logistic
regression model is trained to combine the feature scores into an overall
preference judgment. The authors show that CPMs are more robust compared to
standard PMs and exhibit reduced overoptimization. Furthermore, CPMs allow for
more interpretable and modular preference modeling, as the importance of each
feature can be examined and the model’s behavior adjusted by modifying the
logistic regression weights without retraining. MORLAIF shares the high-level
motivation of CPMs in terms of decomposing reward modeling into simpler, more
interpretable components. However, there are several key differences. First,
MORLAIF trains separate preference models for each principle using preference
comparisons, while CPMs obtain scalar scores for each feature from a single
prompted LLM to train a single PM. Second, MORLAIF uses the preference model
outputs as rewards in an RL framework to fine-tune the policy, while CPMs
focus on the preference modeling stage and use a simple logistic regression to
combine features. Nonetheless, both approaches demonstrate the benefits of
decomposing preference modeling for improved interpretability and robustness
in language model alignment.

###  5.5 Using Multiple Preference Models

Coste et al. (2024) study the use of multiple preference models to improve
robustness and mitigate overoptimization issues in RLHF. They use worst-case
optimization (WCO) and uncertainty-weighted optimization (UWO), which are also
tested in our paper. They find that using an ensemble of reward models and
combining their outputs using conservative objectives practically eliminates
overoptimization and improves performance compared to using a single reward
model, especially under the realistic setting of noisy preference labels. In
their paper, the different preference models are trained on the same
objective; by contrast, we train preference models on different principles,
though our method likely also mitigates overoptimization in a similar way.

###  5.6 Distributional Preference Learning

Siththaranjan et al. (2024) propose Distributional Preference Learning (DPL)
to address hidden context in preference learning and RLHF. Hidden context,
such as annotator diversity or varying annotation objectives, can lead to
suboptimal reward modeling with standard preference learning. DPL estimates a
distribution over utility values to account for this hidden context. Applied
to an RLHF dataset, DPL identifies the hidden context of helpfulness vs.
harmlessness without supervision. Optimizing a lower quantile of the DPL
utility distributions also reduces jailbreak vulnerabilities caused by
conflicting objectives. While MORLAIF decomposes the reward into separate
models for each principle, DPL learns a single distributional model capturing
uncertainty from hidden context, providing a complementary approach to
improving interpretability and robustness in RLHF reward modeling.

##  6 Conclusion

In this paper, we introduced Multi-Objective Reinforcement Learning from AI
Feedback (MORLAIF), a novel approach to improving the alignment and
performance of language models trained using Reinforcement Learning from AI
Feedback (RLAIF). By decomposing the complex task of representing human
preferences into multiple principle-specific preference models, MORLAIF allows
for more targeted data collection, interpretability, and control over the
model’s behavior. Our experiments demonstrate that MORLAIF outperforms
standard single-objective RLAIF baselines across a range of model sizes.
Surprisingly, the choice of scalarization function did not significantly
impact the results, suggesting that the benefits of MORLAIF may be more
closely tied to the decomposition of preferences rather than the specific
method of aggregation. This finding simplifies the selection of scalarization
functions but may also indicate that the theoretical benefits of certain
scalarization functions are limited in practice. Furthermore, our results show
that MORLAIF can be used to align larger language models using smaller
preference models, which could be a promising approach for scalable oversight.
However, the benefits of MORLAIF seem to diminish with increasing model scale,
suggesting that its effectiveness may be reduced when applied to frontier-
scale models. Despite these limitations, MORLAIF offers several appealing
properties, such as improved interpretability, the ability to adjust model
behavior without retraining preference models, and potential mitigation of
overoptimization issues. The decomposition of preferences into specific
principles also enables more fine-grained control and analysis of the model’s
behavior.

## References

  * Bai et al. (2022a) Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan.  Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022a.
  * Bai et al. (2022b) Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al.  Constitutional ai: Harmlessness from ai feedback.  _arXiv preprint arXiv:2212.08073_ , 2022b.
  * Bakker et al. (2022) Michiel A. Bakker, Martin J. Chadwick, Hannah R. Sheahan, Michael Henry Tessler, Lucy Campbell-Gillingham, Jan Balaguer, Nat McAleese, Amelia Glaese, John Aslanides, Matthew M. Botvinick, and Christopher Summerfield.  Fine-tuning language models to find agreement among humans with diverse preferences, 2022.
  * Bommasani et al. (2021) Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al.  On the opportunities and risks of foundation models.  _arXiv preprint arXiv:2108.07258_ , 2021.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Casper et al. (2023) Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, Tony Wang, Samuel Marks, Charbel-Raphaël Segerie, Micah Carroll, Andi Peng, Phillip Christoffersen, Mehul Damani, Stewart Slocum, Usman Anwar, Anand Siththaranjan, Max Nadeau, Eric J. Michaud, Jacob Pfau, Dmitrii Krasheninnikov, Xin Chen, Lauro Langosco, Peter Hase, Erdem Bıyık, Anca Dragan, David Krueger, Dorsa Sadigh, and Dylan Hadfield-Menell.  Open problems and fundamental limitations of reinforcement learning from human feedback, 2023.
  * Christiano et al. (2017) Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei.  Deep reinforcement learning from human preferences.  In _Advances in neural information processing systems_ , pp. 4299–4307, 2017.
  * Coste et al. (2024) Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger.  Reward model ensembles help mitigate overoptimization, 2024.
  * Dettmers et al. (2023) Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer.  Qlora: Efficient finetuning of quantized llms, 2023.
  * Ghosh et al. (2018) Dibya Ghosh, Avi Singh, Aravind Rajeswaran, Vikash Kumar, and Sergey Levine.  Divide-and-conquer reinforcement learning, 2018.
  * Go et al. (2024) Dongyoung Go, Tomasz Korbak, Germ’an Kruszewski, Jos Rozen, and Marc Dymetman.  Compositional preference models for aligning lms.  _arXiv preprint arXiv:2310.13011_ , 2024.
  * Gábor et al. (1998) Zoltán Gábor, Zsolt Kalmár, and Csaba Szepesvári.  Multi-criteria reinforcement learning.  In _Proceedings of the 15th International Conference on Machine Learning (ICML 1998)_ , pp. 197–205, 01 1998.
  * Hu et al. (2021) Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  Lora: Low-rank adaptation of large language models, 2021.
  * Jaques et al. (2019) Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard.  Way off-policy batch deep reinforcement learning of implicit human preferences in dialog, 2019.
  * Kaneko & Nakamura (1979) Mamoru Kaneko and Kenjiro Nakamura.  The nash social welfare function.  _Econometrica_ , 47(2):423–435, 1979.  URL http://www.jstor.org/stable/1914191.  Accessed: 23/03/2012 00:10.
  * Kenton et al. (2021) Zachary Kenton, Tom Everitt, Laura Weidinger, Iason Gabriel, Vladimir Mikulik, and Geoffrey Irving.  Alignment of language agents.  _arXiv preprint arXiv:2103.14659_ , 2021.
  * Lee et al. (2023) Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, and Sushant Prakash.  Rlaif: Scaling reinforcement learning from human feedback with ai feedback, 2023.
  * Moffaert & Nowé (2014) Kristof Van Moffaert and Ann Nowé.  Multi-objective reinforcement learning using sets of pareto dominating policies.  _Journal of Machine Learning Research_ , 15(107):3663–3692, 2014.  URL http://jmlr.org/papers/v15/vanmoffaert14a.html.
  * Mossalam et al. (2016) Hossam Mossalam, Yannis M. Assael, Diederik M. Roijers, and Shimon Whiteson.  Multi-objective deep reinforcement learning, 2016.
  * Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.  Training language models to follow instructions with human feedback.  _arXiv preprint arXiv:2203.02155_ , 2022.
  * Peng et al. (2022) Xue Bin Peng, Yunrong Guo, Lina Halper, Sergey Levine, and Sanja Fidler.  Ase: large-scale reusable adversarial skill embeddings for physically simulated characters.  _ACM Transactions on Graphics_ , 41(4):1–17, July 2022.  ISSN 1557-7368.  doi: 10.1145/3528223.3530110.  URL http://dx.doi.org/10.1145/3528223.3530110.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  Language models are unsupervised multitask learners.  _OpenAI Blog_ , 2019.  URL https://openai.com/blog/better-language-models.
  * Roijers et al. (2013) D. M. Roijers, P. Vamplew, S. Whiteson, and R. Dazeley.  A survey of multi-objective sequential decision-making.  _Journal of Artificial Intelligence Research_ , 48:67–113, October 2013.  ISSN 1076-9757.  doi: 10.1613/jair.3987.  URL http://dx.doi.org/10.1613/jair.3987.
  * Siththaranjan et al. (2024) Anand Siththaranjan, Cassidy Laidlaw, and Dylan Hadfield-Menell.  Distributional preference learning: Understanding and accounting for hidden context in rlhf, 2024.
  * Skalse & Abate (2023) J. M. V. Skalse and A. Abate.  On the limitations of markovian rewards to express multi-objective, risk-sensitive, and modal tasks.  https://openreview.net/forum?id=Vcvg76ZmcZt, June 2023.
  * Stiennon et al. (2020) Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano.  Learning to summarize from human feedback.  _arXiv preprint arXiv:2009.01325_ , 2020.
  * Team et al. (2024) Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Amélie Héliou, Andrea Tacchetti, Anna Bulanova, Antonia Paterson, Beth Tsai, Bobak Shahriari, Charline Le Lan, Christopher A. Choquette-Choo, Clément Crepy, Daniel Cer, Daphne Ippolito, David Reid, Elena Buchatskaya, Eric Ni, Eric Noland, Geng Yan, George Tucker, George-Christian Muraru, Grigory Rozhdestvenskiy, Henryk Michalewski, Ian Tenney, Ivan Grishchenko, Jacob Austin, James Keeling, Jane Labanowski, Jean-Baptiste Lespiau, Jeff Stanway, Jenny Brennan, Jeremy Chen, Johan Ferret, Justin Chiu, Justin Mao-Jones, Katherine Lee, Kathy Yu, Katie Millican, Lars Lowe Sjoesund, Lisa Lee, Lucas Dixon, Machel Reid, Maciej Mikuła, Mateo Wirth, Michael Sharman, Nikolai Chinaev, Nithum Thain, Olivier Bachem, Oscar Chang, Oscar Wahltinez, Paige Bailey, Paul Michel, Petko Yotov, Rahma Chaabouni, Ramona Comanescu, Reena Jana, Rohan Anil, Ross McIlroy, Ruibo Liu, Ryan Mullins, Samuel L Smith, Sebastian Borgeaud, Sertan Girgin, Sholto Douglas, Shree Pandya, Siamak Shakeri, Soham De, Ted Klimenko, Tom Hennigan, Vlad Feinberg, Wojciech Stokowiec, Yu hui Chen, Zafarali Ahmed, Zhitao Gong, Tris Warkentin, Ludovic Peran, Minh Giang, Clément Farabet, Oriol Vinyals, Jeff Dean, Koray Kavukcuoglu, Demis Hassabis, Zoubin Ghahramani, Douglas Eck, Joelle Barral, Fernando Pereira, Eli Collins, Armand Joulin, Noah Fiedel, Evan Senter, Alek Andreev, and Kathleen Kenealy.  Gemma: Open models based on gemini research and technology, 2024.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom.  Llama 2: Open foundation and fine-tuned chat models, 2023.
  * Van Moffaert et al. (2013) Kristof Van Moffaert, Madalina M. Drugan, and Ann Nowé.  Scalarized multi-objective reinforcement learning: Novel design techniques.  In _2013 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL)_ , pp. 191–199, 2013.  doi: 10.1109/ADPRL.2013.6615007.
  * Veselovsky et al. (2023) Veniamin Veselovsky, Manoel Horta Ribeiro, Philip Cozzolino, Andrew Gordon, David Rothschild, and Robert West.  Prevalence and prevention of large language model use in crowd work, 2023.
  * Ziegler et al. (2019) Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving.  Fine-tuning language models from human preferences.  _arXiv preprint arXiv:1909.08593_ , 2019.  Accessed: date-of-access.
  * Ziegler et al. (2020) Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving.  Fine-tuning language models from human preferences, 2020.

##  Appendix A Feedback Model and Win-Rate Prompts

For feedback, the following prompt was used:

Feedback Model Prompt without CoT You will be given a conversation between a
human and an AI assistant. Your job is to judge which of the two possible
assistant responses is principle. Here is the conversation: conversation Here
are the possible response options: Option A: responseA Option B: responseB
Please judge which option is principle, regardless of how good the answer is
from other perspectives. Please respond only with A or B. The best response is
option:

Feedback Model Prompt with CoT You will be given a conversation between a
human and an AI assistant. Your job is to judge which of the two possible
assistant responses is principle. Here is the conversation: conversation Here
are the possible response options: Option A: responseA Option B: responseB
Please judge which option is principle, regardless of how good the answer is
from other perspectives. Begin your response with your reasoning inside
<reasoning></reasoning>tags. End your response with ”Chosen option: ” followed
by A or B.

Win-Rate Prompt You will be given a conversation between a human and an AI
assistant. Your job is to judge which of the two possible assistant responses
is better. Here is the conversation: conversation Here are the possible
response options: Option A: responseA Option B: responseB Please judge which
option is better. Please respond only with A or B. The best response is
option:

##  Appendix B Human Experiments

This section discusses some specifics of our human crowdworker win-rate
experiment. Crowdworkers were recruited from the online crowdworking platform
Amazon MTurk.

Figure 6: An example showing the interface which was presented to the
crowdworkers and an example conversation

After filling in a short form and reading the instructions, the crowdworkers
were given a link which took them to a website with the interface shown in
figure 6. Users were asked to have a conversation with an AI assistant. After
each message, the two different models generated responses which were shown as
option A and option B (in random order to mitigate positional biases). The
users were then asked to rate which of the two options was better or if it was
a tie. If the user chose A or B, the conversation continued with that
response, while if they chose tie, a random response was added to the
conversation. The conversation could continue for a maximum of 10 turns, and
there was also a button to start a new conversation instead of continuing the
current one. The average conversation lasted 5.2 turns. Each worker could have
a maximum of 10 conversations.

We ran two win-rate experiments, one comparing a multi-objective linear
Llama-2-7b with a single-objective version and one comparing a multi-objective
linear Llama-2-7b trained using GPT-2-medium preference models with the same
single-objective Llama-2-7b trained with a single preference model. For these
comparisons, we collected 4238 and 4281 datapoints, respectively.

According to Veselovsky et al. (2023), LLM use is prevalent on crowdworking
platforms. If crowdworkers use LLMs to generate their dialogue or choose their
preferred responses, that invalidates the entire point of using humans in the
first place. In order to mitigate this, a few steps were taken. Firstly,
workers were instructed not to use LLMs and tick a box indicating that they
understood this. Secondly, copy-pasting was turned off, limiting the speedup
using an LLM would entail. Thirdly, before starting, users were prompted with
an ”LLM-proof riddle” in order to weed out people using LLMs. The task was a
variation of an actual riddle where all the hard bits have been removed and
there is only a trivial answer, but LLMs typically answer with the solution to
the original riddle rather than the modified one. The riddle used was the
following:

“LLM proof” Riddle There is a goat and a man on one side of a river. There’s a
boat which can only carry two at a time and can only be rowed manually by the
man. How can they cross the river?

LLMs very often fail to notice the trivial solution of ”They both take the
boat across the river” since they have been trained on the original riddle.
Finally, we checked responses with GPTZero, a tool which, despite high error
rates, predicts how likely it was that a text was generated by AI. We
discarded all data from users who scored above a 25% chance of being AI, which
represented about 13% of our collected data.


# Empirical Analysis of Foundational Distinctions in Linked Open Data

Luigi Asprino1,2, Valerio Basile3, Paolo Ciancarini2, Valentina Presutti1
1 STLab, ISTC-CNR (Rome, Italy)
2 University of Bologna (Bologna, Italy)
3 University of Turin (Turin, Italy)
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

The Web and its Semantic extension (i.e. Linked Open Data) contain open
global-scale knowledge and make it available to potentially intelligent
machines that want to benefit from it. Nevertheless, most of Linked Open Data
lack ontological distinctions and have sparse axiomatisation. For example,
distinctions such as whether an entity is _inherently_ a class or an
individual, or whether it is a physical object or not, are hardly expressed in
the data, although they have been largely studied and formalised by
foundational ontologies (e.g. DOLCE, SUMO). These distinctions belong to
common sense too, which is relevant for many artificial intelligence tasks
such as natural language understanding, scene recognition, and the like. There
is a gap between foundational ontologies, that often formalise or are inspired
by pre-existing philosophical theories and are developed with a top-down
approach, and Linked Open Data that mostly derive from existing databases or
crowd-based effort (e.g. DBpedia, Wikidata). We investigate whether machines
can learn foundational distinctions over Linked Open Data entities, and if
they match common sense. We want to answer questions such as “does the DBpedia
entity for _dog_ refer to a class or to an instance?”. We report on a set of
experiments based on machine learning and crowdsourcing that show promising
results.

##  1 Common Sense and Linked Open Data

Common Sense Knowledge is knowledge about the world, shared by all people.
Common sense reasoning is also at the core of many Artificial Intelligence
(AI) tasks such as natural language understanding, object and action
recognition, and behavior arbitration Davis and Marcus (2015), but it is
difficult to teach to those systems. Its importance was assessed back in 1989
by Hayes (1989) who argued that AI needs a “formalization of a sizable portion
of commonsense knowledge about the everyday physical world” (cit.), which, he
says, must have three main characteristics: uniformity, density, and breadth.
The Semantic Web effort has partly addressed this requirement with Linked Open
Data (LOD): $\sim$150 billion linked facts111http://stats.lod2.eu/, accessed
on April 25th 2018, formally and uniformly represented in RDF and OWL, and
openly available on the Web. Nevertheless, LOD still fails in addressing
density (high ratio of facts about concepts) and breadth (large coverage of
physical phenomena). In fact, it is very rich for domains such as geography,
linguistics, life sciences and scholarly publications, as well as for cross-
domain knowledge, but it mostly encodes this knowledge from an encyclopaedic
perspective. The ultimate goal of our research is to enrich LOD with common
sense knowledge, going beyond the encyclopaedic view. We claim that an
important gap to be filled towards this goal is: assessing foundational
distinctions over LOD entities, that is to distinguish and formally assert
whether a LOD entity inherently refers to e.g. a class or an individual, a
physical object or not, a location, a social object, etc., from a common sense
perspective.

###  1.1 Foundational Distinctions

High level categorial distinctions (e.g. class vs. instance) are a fundamental
human cognitive ability: “There is nothing more basic than categorization to
our thought, perception, action, and speech.” Lakoff (1987). This is also why
“the organisation of objects into categories is a vital part of knowledge
representation” Russell and Norvig (2009). Foundational distinctions have been
theorised and modelled in foundational ontologies such as DOLCE Masolo et al.
(2003) and SUMO Pease and Niles (2002) with a top-down approach, but
populating and empirically validating them has been rarely addressed. In this
study, we perform a set of experiments to assess _whether machines can learn
to perform foundational distinctions, and if they match common sense_. The
former issue is investigated by applying machine learning techniques, the
latter is assessed by crowdsourcing the annotation of the evaluation datasets.
We use DOLCE+DnS UltraLite (DUL)222DOLCE+DnS UltraLite (DUL)
http://www.ontologydesignpatterns.org/ont/dul/DUL.owl is derived from DOLCE.
DUL inherits most of DOLCE’s distinctions by providing a more intuitive
terminology and simplified axiomatisation. It has been widely adopted by the
Semantic Web community. as reference foundational ontology to select the
targets of our experiments. We start by focusing on two very basic but diverse
distinctions, which need to be addressed before approaching all the others:
whether a LOD entity e.g. dbr:Rome333dbr: stands for
http://dbpedia.org/resource/, (i) inherently refers to a class or an instance,
and whether it (ii) refers to a physical object or not. The first distinction
(class vs. instance) is fundamental in formal ontology, as evidenced by upper-
level ontologies (e.g. SUMO and DOLCE), and showed its practical importance in
modelling and meta-modelling approaches in computer science, e.g. the
class/classifier distinction in Meta Object
Facility444https://www.omg.org/spec/MOF/. It is also at the basis of LOD
knowledge representation formalisms (RDF and OWL) for supporting taxonomic
reasoning (e.g. inheritance). Automatically learning whether a LOD entity is a
class or an instance – from a common sense perspective – impacts on the
behaviour of practical applications relying on LOD as common sense background
knowledge. Examples include: question answering, knowledge extraction, and
more broadly human-machine interaction. In fact, many LOD datasets that are
commonly used for supporting these tasks (especially general purpose datasets
e.g. DBpedia, Wikidata, BabelNet) only partially, and often incorrectly,
assert whether their entities are classes or instances, and this has been
proved to be a source of many inconsistencies and error patterns Paulheim and
Gangemi (2015).

The second distinction (physical object or not) is essential to represent the
physical world. In fact, only physical objects can move in space or be the
subject of axioms expressing their expected (naive) physical behaviour (e.g.
gravity). We refer to the definition of _Physical Object_ provided by DUL:
_“Any Object that has a proper space region. The prototypical physical object
has also an associated mass, but the nature of its mass can greatly vary based
on the epistemological status of the object (scientifically measured,
subjectively possible, imaginary)”_.

###  1.2 Contribution

In the reminder of this paper we describe an automated way of making these
distinctions emerge empirically. We present and discuss a set of experiments,
conducted on a sample of LOD, involving manual inspection, ontology alignment,
machine learning, and crowdsourcing. The obtained results are promising, and
motivate us to extend them to a much larger scale, in line with a recent
inspiring talk “The Empirical Turn in Knowledge
Representation”555https://goo.gl/BDSGY1 by van Harmelen, who suggests that LOD
is a unique opportunity to “observe how knowledge representations behave at
very large scale”.

In summary, the contribution of this research includes:

  * •

a novel method that leverages supervised machine learning and crowdsourcing to
automatically assess foundational distinctions over LOD entities (cf. Section
3), according to common sense;

  * •

four reusable datasets, based on a sample of DBpedia, separately annotated by
experts and by the crowd with class/instance and physical object
classification, for each entity (cf. Section 4). The crowdsourced task designs
are on their turn reusable;

  * •

a set of reproducible experiments targeting two foundational distinctions:
_class vs. instance_ and _physical object vs. not a physical object_ , showing
that machines can learn them by using a same set of features, and that they
match common sense (cf. Section 5).

##  2 Related Work

Only a few studies focus on typing entities based on foundational
distinctions. To the best of our knowledge, our research is the first to test
the hypothesis that machines can learn foundational distinctions that match
common sense, by using web resources. The closest work to ours in approach and
scale is Gangemi et al. (2012), which produced a dataset of DBpedia entities
annotated with DUL classes, using ontology learning. We reuse this dataset and
compare our results against it. The work by Miller and Hristea (2006)
addresses the problem of distinguishing classes from instances in WordNet
Fellbaum (1998) synsets, through purely manual annotation. This approach is
inappropriate to test our research questions due to its lack of scalability.
Over the years, a number of common sense knowledge bases have been proposed
for supporting diverse tasks spanning from automated reasoning to natural
language processing. We decided to use the English
DBpedia666http://dbpedia.org Bizer et al. (2009) in our experiments. Besides
being very popular, it is the _de facto_ main hub of LOD, with its 4.58
million entities777http://wiki.dbpedia.org/about/facts-figures, accessed on
April 25th 2018. There are other resources that either target common sense, or
encode potentially relevant knowledge for common sense reasoning. However they
lack explicit assertions of foundational distinctions, and show very sparse
coverage distribution. Two of them are relevant in this context, and we plan
to investigate how to leverage them in future experiments that we plan to
perform at a much larger scale: ConceptNet888http://conceptnet5.media.mit.edu/
Liu and Singh (2004) is a large-scale semantic network that integrates other
existing resources, mostly derived from crowdsourced resources, expert-created
resources, and games with a purpose. We found it unsuitable for reuse at this
stage of our study: only a very small portion of it is linked to LOD, and its
representation is bound to linguistic expressions instead of formalised
concepts. OpenCyC999http://www.opencyc.org/ Lenat (1995) includes an ontology
and a knowledge base of common sense knowledge organised in modular theories,
released as part of the long-standing CyC project. OpenCyC has been mainly
developed with a top-down approach by ontology engineering experts. It is a
potentially valuable resource to compare with, but it uses a proprietary
representation language, which makes it hard to work with, currently. There is
a plan to make it available to the scientific community as linked data,
although apparently inaccessible at the time of our experiment.

##  3 Automatic Classification of Foundational Distinctions

We want to answer the following research questions: _(RQ1) Do foundational
distinctions match common sense? (RQ2) Does the (Semantic) Web provide an
empirical basis for supporting foundational distinctions over LOD entities,
according to common sense? and (RQ3) what ensemble of features, resources, and
methods works best to make machines learn foundational distinctions over LOD
entities?_

Our objective is to test all the distinctions formalised in DUL. Nevertheless,
for each of them we need to create a set of reference datasets (cf. Section 4)
in order to train and evaluate the proposed method, which requires a
significant amount of work. For this reason, as anticipated in Section 1, we
start focusing on two distinctions: between class and instance, and between
what is a physical object and what is not. These are two of the most basic,
but very diverse distinctions in knowledge representation and formal ontology.
The former applies at a very high level, and is usually modelled by means of
logical language primitives (e.g. rdf:type, rdfs:subClassOf). The latter
concerns the identification of those entities that constitute the physical
world, hence highly relevant and primitive as far as common sense about
physics is concerned. We argue that by investigating these two distinctions,
given their diverse character, we can assess the feasibility of a larger study
based on the proposed method, and get an indication of its generalisability.
We use DBpedia (release 2016-10) in our study as most LOD datasets link to it.
We approach this problem as a classification task, using two classification
approaches: alignment-based (cf. Section 3.1) and _machine learning-based_
(cf. Section 3.2). Since no established procedure exists, we tested different
families of methods in an exploratory way. This led us to reuse – or compare
to – existing work, which provides us with a baseline, which includes Tìpalo
Gangemi et al. (2012) as well as other relevant alignments between DBpedia and
lexical resources (cf. Section 3.1).

###  3.1 Alignment-based Classification

Alignment-based methods exploit the linking structure of LOD, in particular
the alignments between DBpedia, foundational ontologies, and lexical linked
data, i.e. LOD datasets that encode lexical/linguistic knowledge. The
advantage of these methods is their inherent unsupervised nature. Their main
disadvantages are the need of studying the data models for designing suitable
queries, and the potential limited coverage and errors that may accompany the
alignments. We have developed SENECA (Selecting Entities Exploiting Linguistic
Alignments), which relies on existing alignments in LOD, to make an automatic
assessment of the foundational distinctions asserted over DBpedia entities. A
graphical description of SENECA is depicted in Figure 1.
Class vs. Instance. As far as this distinction is concerned, SENECA works
based on the hypothesis that common nouns are mainly classes and they are
expected to be found in dictionaries, while it is less the case for proper
nouns, that usually denote instances. This hypothesis was suggested by Miller
and Hristea (2006), who manually annotated instances in WordNet, information
that SENECA reuses when available. A good quality alignment between the main
LOD lexical resources and DBpedia is provided by BabelNet Navigli and Ponzetto
(2012)101010We use BabelNet 3.6, which is aligned to WordNet 3.1. SENECA
exploits these alignments and selects all the DBpedia entities that are linked
to an entity in WordNet111111http://wordnet-rdf.princeton.edu/, we use WordNet
3.0 and its alignments to WordNet 3.1, to ensure interoperability with the
other resources, Wiktionary121212https://www.wiktionary.org/ or
OmegaWiki131313http://www.omegawiki.org/. With this approach, 63,620 candidate
classes have been identified, as opposed to WordNet annotations that only
provide 38,701 classes. In order to further increase the potential coverage,
SENECA leverages the typing axioms of Tipalo Gangemi et al. (2012), broadening
it to 431,254 total candidate classes. All the other DBpedia entities are
assumed to be candidate instances. SENECA criteria for selecting candidate
classes among DBpedia entities are depicted in Figure 1(a).

(a) The alignment paths followed by SENECA for selecting candidate classes
among DBpedia entities. It identifies as classes all DBpedia entities aligned
via BabelNet to a WordNet synset, an OmegaWiki synset or a Wiktionary page,
and all DBpedia entities typed as owl:Class in Tìpalo.

(b) The alignment paths used by SENECA for identifying candidate Physical
Objects among DBpedia entities. It navigates the YAGO taxonomy that via
OntoWordNet links DBpedia entities to dul:PhysicalObject or Tìpalo that links
DBpedia entities to dul:PhysicalObject.

Figure 1: SENECA approach for assessing whether a DBpedia entity is a class or
an instance (Figure 1(a)) and whether it is a physical object or not (Figure
1(b)).

Physical Object. Almost 600,000 DBpedia entities are only typed as owl:Thing
or not typed at all. However, each DBpedia entity belongs to at least one
Wikipedia category. Wikipedia categories have been formalised as a taxonomy of
classes (i.e. by means of rdfs:subClassOf) and aligned to WordNet synsets in
YAGO Suchanek et al. (2007)141414We use YAGO 3, aligned to WordNet 3.1.
WordNet synsets are in turn formalised as an OWL ontology in OntoWordNet
Gangemi et al. (2003)151515OntoWordNet is aligned to WordNet 3.0. OntoWordNet
is based on DUL, hence it is possible to navigate the taxonomy up to the DUL
class for Physical Object. SENECA looks up the Wikipedia category of a DBpedia
entity and follows these alignments. Additionally, it uses Tìpalo, which
includes type axioms of DBpedia entities based on DUL classes. SENECA uses
these paths of alignments and taxonomical relations, as well as the automated
inferences that enable to assess whether a DBpedia entity is a Physical Object
or not. With this approach, graphically summarised in Figure 1(b), 67,005
entities were selected as candidate physical objects.

###  3.2 Machine Learning-based Classification

Within machine learning, classification is the problem of predicting which
category an entity belongs to, given a set of examples, i.e. a training set.
The training set is processed by an algorithm in order to learn a predictive
model based on the observation of a number of _features_ , which can be
categorical, ordinal, integer-valued or real-valued. We have designed our
target distinctions in the form of two binary classifications. We have
experimented with eight classification algorithms: J48, Random Forest,
REPTree, Naive Bayes, Multinomial Naive Bayes, Support Vector Machines,
Logistic Regression, and K-nearest neighbours classifier. We have used
WEKA161616https://www.cs.waikato.ac.nz/ml/weka/ for their implementation.
Features. The classifiers were trained using the following four features.

Abstract. Considering that DBpedia entities are all associated with an
abstract providing a definitional text, our assumption is that these texts
encode useful distinctive patterns. Hence, we retrieve DBpedia entity
abstracts, and represent them as 0-1 vectors (bags of words). We built a
dictionary containing the 1000 most frequent tokens found in all the abstracts
of the dataset. The dictionary is case-sensitive, since the tokens are not
normalised. The resulting vector has a value 1 for each token mentioned in the
abstract, 0 for the others. By inspecting a good amount of abstracts, we
noticed that very frequent words, such as conjunctions and determiners, are
used in a way that can be informative for this type of classifications. For
example, most of class definitions begin with “A” (“A knife is a tool…”). For
this reason, we did not remove stop-words.

URI. We notice that the ID part of URIs is often as informative as a label,
and often follows conventions that may be discriminating especially for the
class vs. instance classification. In DBpedia, the ID of a URI reflects an
entity name (it is common practice in order to make the URI more human-
readable), and it always starts with an upper case letter. If the entity’s
name is a compound term and the entity denotes an instance, each of its
components starts with a capital letter. We have also noticed that DBpedia
entity names are always mentioned at the beginning of their abstract and, for
most of the instance entities, they have the same capitalisation pattern as
the URI ID. Moreover, instances tend to have more terms in their ID than
classes. These observations were captured by three numerical features: (i)
number of terms in the ID starting with a capital letter, (ii) number of terms
in the ID that are also found in the abstract, and (iii) number of terms in
the ID.

Incoming and Outgoing Properties. As part of our exploratory approach, we want
to test the ability of LOD to show relevant patterns leading to foundational
distinctions. Given that triples are the core tool of LOD, we model a feature
based on ongoing and outgoing properties of a DBpedia entity. An outgoing
property of a DBpedia entity is a property of a triple having the entity as
subject. On the contrary, an incoming property is a property of a triple
having the entity as object. For example, considering the triple dbr:Rome
:locatedIn dbr:Italy, the property :locatedIn is an outgoing property for
dbr:Rome and an incoming property for dbr:Italy. For each DBpedia entity, we
count its incoming and outgoing properties, per type. For example, properties
such as dbo:birthPlace or dbo:birthDate are common outgoing properties of an
individual person, hence their presence suggests that the entity is an
individual.

Outcome of SENECA. Following an exploratory approach, we decided to use the
output of SENECA as a binomial feature (taking value “yes” or “no”) for the
classifiers (excluding Multinomial Naive Bayes classifier).

##  4 Reference Datasets for Classification Experiments

In order to perform our experiments and evaluate the results of our approach
(cf. Section 3), we have created two datasets for each of the foundational
distinctions under study: one annotated by experts, and another one annotated
by the crowd. In this way we can get an indication whether foundational
distinctions match common sense (cf. _RQ1_). The resulting datasets include a
sample of annotated DBpedia entities and are available online
171717https://github.com/fdistinctions/ijcai18.

Selecting DBpedia Entities. The first step to build the datasets is to select
a sample of DBpedia entities to be submitted for annotations. It is not
straightforward to select a balanced number of classes and instances from
DBpedia. A random selection would cause a strong unbalance towards instances
because DBpedia contains a larger number of named individuals – e.g. Rome or
Barack Obama – than concepts. A possible solution could be to manually select
a sufficient equal number of DBpedia instances and classes, however this may
inject a bias in the datasets. We have opted for a compromise solution by
following the intuition that if entities are represented as vectors, their
neighbour vectors would include classes and instances with a more balanced
ratio than the random choice. As for minimising the bias, we only manually
select an arbitrarily small (i.e. 20) number of seeds (equally distributed).
We leverage NASARI Camacho-Collados et al. (2016), a resource of vector
representations for BabelNet synsets and Wikipedia entities. For each vector
corresponding to a seed entity, we retrieve its 100 nearest neighbours181818We
observed that by picking 100 nearest neighbour entities the cosine similarity
is always above a threshold of 0.6. After cleaning off duplicated entities
(e.g. Wikipedia redirects), entities without abstracts, disambiguation pages,
etc., we still assessed (through expert annotations) an unbalance towards
instances. In the light of a broader usage of the same dataset to also
annotate the distinction between physical objects and not physical objects, we
enriched the sample with class entities representing physical locations (a
good source of physical objects). In order to select only physical location
classes from DBpedia, we used the corresponding DBpedia category dbc:Places
and the SUN database191919https://groups.csail.mit.edu/vision/SUN/, a computer
vision-oriented dataset containing a collection of annotated images, covering
a large variety of environmental scenes, places, and the objects within. We
retrieved DBpedia entities whose labels match SUN locations or that belong to
the category dbc:Places, and added a number of them to the sample that would
suffice to improve the balance. As a result, a total number of 4502 entities
were collected in the newly created dataset.

CIE Dataset: Class vs. Instance Annotation Performed by Experts. Two authors
of the paper have manually and independently annotated all entities by
indicating whether they were instances or classes, using the associated
DBpedia abstract as reference description. Their judgements showed an
agreement of 93,6%: they only disagreed on 286 entities. From a joint second
inspection, they agreed on additional 281 entities that were initially
misclassified by one of the two. Examples of misclassified entities are:
dbr:Select_Comfort (a U.S. manufacturer) that was erroneously annotated as
class; dbr:Catawba_Valley_Pottery (a kind of pottery) annotated as instance
instead of class. Among the remaining entities, five are polysemous cases,
where the entity and its description point to both types of referents, e.g.
dbr:Slide_Away_Bed is a trademark commonly used also to refer to a type of
beds. The authors decided to annotate these entities as classes. As a result,
the CIE annotated dataset contains 1983 classes and 2519 instances, which is
reasonable balanced (44% classes, 56% instances).

CIC Dataset: Class vs. Instance Annotation Performed by Crowd. The same
dataset was then crowdsourced: each worker was asked to indicate whether an
entity is a class or an instance based on its name, abstract, and its
corresponding Wikipedia article. We want to assess the agreement between the
experts and the crowd, which indicates whether foundational distinctions match
common sense or not (cf. _RQ1_). The task was executed on Figure Eight
202020https://www.figure-eight.com/ by English speakers with high
trustworthiness. The quality of the contributors has been assessed with 51
test questions with a tolerance of only 20% of errors. We collected 22,510
judgments from 117 contributors: each entity was annotated by at least 5
different workers. For each entity $e$, we computed the level of agreement on
each class $c$, weighted by the trustworthiness scores

$agreement(e,c)=\frac{SumOfTrust(e,c)}{SumOfTrustOfWorkers(e)}$ (1)

where $SumOfTrust(e,c)$ is the sum of the trustworthiness scores of the
workers that annotated entity $e$ with class $c$; and $SumOfTrustOfWorkers(e)$
is the sum of the trustworthiness scores of all the workers that annotated the
entity $e$. Table 1 reports the results of the task indicating the
distribution of classes and instances per level of agreement. The average
agreement of the crowd is 95.76% .

Agreement | # Class | # Instance | Total
---|---|---|---
$\geq$ 0.5 | 1934 | 2568 | 4502
$\geq$ 0.6 | 1884 | 2495 | 4379
$\geq$ 0.8 | 1631 | 2330 | 3961

Table 1: CIC dataset crowd-based annotated dataset of classes and instances.
The table provides an insight of the dataset per level of agreement. Agreement
values computed according to Formula 1.

We compared crowd’s annotations (with agreement greater than 0.5) against
experts’ ones. The judgements of the crowd workers diverge from the experts’
only on 193 entities, i.e. agreement is 95,7%, suggesting that _the instance
vs. class foundational distinction matches common sense_ (cf. RQ1 applied to
this distinction). Some of those entities (61) also caused a disagreement
between experts, hence denoting ambiguous cases. Examples include polysemic
entities such as dbr:Zeke_the_Wonder_Dog or music genres (e.g. dbr:Ragga).

POE Dataset: Physical Object Annotation Performed by Experts. Two authors of
the paper further annotated (independently) the dataset by indicating for each
entity whether it referred to a physical object (PO) or not (NPO), using its
DBpedia abstract as reference description. They only disagreed on 272
entities, showing an agreement of 93,9%. By means of a joint second
inspection, they agreed that the disagreement was caused by errors in the
classification, some of which were borderline cases e.g.: communities (e.g.
dbr:Desert_Lake,_California), wrongly interpreted as society instead of
neighbourhood, trademarked materials (e.g. dbr:Waxtite) and entities with
complex description (e.g. dbr:Cabaña_pasiega). The resulting POE annotated
dataset contains 3055 POs and 1447 NPOs.

POC Dataset: Physical Object Annotation Performed by Crowd. We also
crowdsourced the annotations of physical objects vs. not a physical object:
the workers were asked to perform this task by using the entity’s name, its
abstract, and Wikipedia page as reference descriptions. The quality of the
workers has been assessed with 49 test questions, used to exclude contributors
that scored an accuracy lower than 80% We collected 25,776 judgments from 173
workers. Each entity has been annotated by at least 5 different English
speakers. Table 2 summarises the level of agreement associated with the
distribution of PO vs. NPO annotations. The average agreement of the crowd’s
annotations is 85.48% . The agreement between the crowd and the experts is
85,69%, suggesting that _the PO vs NPO foundational distinction also matches
common sense_ (cf. RQ1 applied to this distinction).

Agreement | # Physical Object | # $\neg$ Physical Object | Total
---|---|---|---
$\geq$ 0.5 | 3601 | 901 | 4502
$\geq$ 0.6 | 3448 | 641 | 4089
$\geq$ 0.8 | 2989 | 335 | 3324

Table 2: POC dataset: crowd-based annotated dataset of physical objects. The
table provides an insight of the dataset per level of agreement. Agreement
values computed according to Formula 1.

##  5 Experiments Results and Discussion

The results of the performed experiments are expressed in terms of precision,
recall and F1 measure, computed for each classification and for each target
class (class vs. instance and physical object vs. $\neg$ physical object). The
average F1 score is also provided. We compare the results of the methods,
described in Section 3, against the reference datasets CIE, CIC, POE and POC,
described in Section 4. As for CIC and POC, we only include the annotations
having agreement $\geq$ 80%.

###  5.1 Alignment-based Methods: SENECA

Class vs. Instance. Table 3 shows SENECA’s performance on the class vs.
instance classification, by comparing its results with CIE and CIC. SENECA
shows very good performance with best avg F1 $=$ .836, when compared with CIC.
Considering that SENECA is unsupervised, and is based on existing alignments
in LOD, this result suggests that LOD may better reflect common sense than the
expert’s perspective, an interesting hint for further investigation on this
specific matter.

Dataset | PC | RC | FC1 | PI | RI | FI1 | $\overline{\mbox{F\textsubscript{1}}}$
---|---|---|---|---|---|---|---
CIE | .919 | .693 | .796 | .753 | .939 | .836 | .813
CIC | .935 | .731 | .818 | .778 | .945 | .853 | .836
Dataset | PPO | RPO | FPO1 | PNPO | RNPO | FNPO1 | $\overline{\mbox{F\textsubscript{1}}}$
POE | .877 | .247 | .385 | .561 | .965 | .713 | .548
POC | .954 | .247 | .393 | .567 | .988 | .721 | .557

Table 3: Results of SENECA on the Class vs. Instance and Physical Object
classifications compared against the reference datasets described in Section
4. P*, R* and F*1 indicate precision, recall and F1 measure on Class (C),
Instance (I), Physical Object (PO) and complement of Physical Object (NPO).
$\overline{\mbox{F\textsubscript{1}}}$ is the average of the F1 measures.

Physical Object. Table 3 shows the performance of SENECA on the Physical
Object classification task computed by comparing its results with POE and POC
(cf. Section 4). We observe a significant drop in the best average F1 score
(.557) as compared to the class vs. instance classification task (.836). On
one hand, this may suggest that the task is harder. On the other hand, the
alignment paths followed in the two cases are different, since for classifying
Physical Objects more alignment steps are required. In the first case (class
vs. instance), BabelNet directly provides the final alignment step (cf. Figure
1(a)), while in the second case (PO vs. NPO), three more alignment steps are
required: DBpedia Category $\rightarrow$ YAGO $\rightarrow$ WordNet (cf.
Figure 1(b)). It is reasonable to think that this implies a higher potential
of error propagation along the flow. This hypothesis is partly supported by
Gangemi et al. (2012), who report a similar drop when they add an automatic
disambiguation step followed by an alignment step to DUL classes (including
Physical Object). Also for this distinction, SENECA better matches the
judgements of the crowd than the experts’.

###  5.2 Machine Learning Methods

We performed a set of experiments with eight classifiers: J48, Random Forest,
REPTree, Naive Bayes, Multinomial Naive Bayes, Support Vector Machines,
Logistic Regression, and K-Nearest Neighbors (cf. Section 3.2). We used a
10-fold cross validation strategy using the reference datasets (cf. Section
4). Before training the classifiers, the datasets were adjusted in order to
balance the samples of the two classes. The CIE and POE datasets were balanced
by randomly removing a set of annotated entities. CIC and POC were balanced by
removing entities associated with lower agreement (which constitute weak
examples for the classifiers). Each classifier was trained and tested with all
four features, described in Section 3.2, both individually and in all possible
combinations, with and without performing feature selection. We found that
performing feature selection makes the results worse. Having two datasets for
each classification (i.e. annotated by the experts and by the crowd) enables
multiple configurations of the training set. When we train the classifiers
with samples from CIE and POE, they all have the same weight $=$ 1\.
Differently, when the samples come from CIC and POC, they are weighted
according to their associated agreement score _agreement(e, c)_ , computed
with Formula 1 (cf. Section 4). As previously studied by Aroyo and Welty
(2015), this diverse weighing allows a classifier to learn richer information,
including ambiguity and consequent entities that may belong to an “unknown”
class, which better represent human cognitive behaviour. Due to space limits,
we only report the results of the best performing algorithm212121The results
of all experiments are available online at
https://github.com/fdistinctions/ijcai18, which is Support Vector Machine,
without feature selection, trained and tested on samples associated with an
agreement score $\geq$ 80%. We report on all combinations of features, but D
alone (i.e. SENECA’s output).

Class vs. Instance. Table 4 shows the results of Support Vector Machine,
trained on and tested against CIE and CIC. The best average performance is
obtained with CIC by combining all features. Combining all features is also
the best configuration for each individual classification (i.e. Class (C) and
Instance (I)). When CIE is used there is a slight drop in performance,
although the quality of the classification remains high. A possible cause of
this result may be the agreement-based weighing provided by CIC.

|  Results compared against CIE |  Results compared against CIC |  Results compared against POE |  Results compared against POC
---|---|---|---|---
A | U | E | D | PC | RC | FC1 | PI | RI | FI1 | $\overline{\mbox{F\textsubscript{1}}}$ | PC | RC | FC1 | PI | RI | FI1 | $\overline{\mbox{F\textsubscript{1}}}$ | PPO | RPO | FPO1 | PNPO | RNPO | F NPO1 | $\overline{\mbox{F\textsubscript{1}}}$ | PPO | RPO | FPO1 | PNPO | RNPO | FNPO1 | $\overline{\mbox{F\textsubscript{1}}}$
|  |  |  | .927 | .921 | .924 | .921 | .927 | .924 | .924 | .958 | .965 | .961 | .965 | .957 | .961 | .961 | .828 | .814 | .821 | .817 | .832 | .824 | .823 | .879 | .837 | .858 | .844 | .884 | .864 | .861
|  |  |  | .881 | .933 | .906 | .929 | .873 | .909 | .903 | .908 | .970 | .938 | .967 | .902 | .933 | .936 | .615 | .822 | .703 | .732 | .485 | .584 | .644 | .596 | .863 | .705 | .751 | .413 | .533 | .619
|  |  |  | .854 | .975 | .911 | .971 | .834 | .897 | .904 | .886 | .983 | .932 | .981 | .874 | .924 | .928 | .786 | .865 | .824 | .857 | .764 | .805 | .814 | .782 | .886 | .831 | .868 | .752 | .806 | .818
|  |  |  | .928 | .935 | .932 | .935 | .928 | .931 | .932 | .966 | .971 | .968 | .971 | .966 | .968 | .968 | .831 | .829 | .838 | .833 | .832 | .831 | .831 | .851 | .811 | .831 | .819 | .857 | .838 | .834
|  |  |  | .939 | .943 | .941 | .943 | .939 | .941 | .941 | .971 | .976 | .974 | .976 | .971 | .973 | .974 | .869 | .867 | .868 | .867 | .870 | .868 | .868 | .912 | .853 | .882 | .862 | .918 | .889 | .885
|  |  |  | .934 | .927 | .939 | .928 | .934 | .931 | .931 | .966 | .964 | .965 | .964 | .966 | .965 | .965 | .849 | .829 | .835 | .831 | .843 | .837 | .836 | .865 | .834 | .849 | .839 | .869 | .854 | .852
|  |  |  | .919 | .968 | .943 | .966 | .914 | .939 | .941 | .961 | .982 | .971 | .981 | .963 | .979 | .971 | .816 | .852 | .833 | .845 | .808 | .826 | .832 | .802 | .889 | .842 | .875 | .777 | .823 | .833
|  |  |  | .881 | .939 | .909 | .935 | .873 | .903 | .906 | .908 | .973 | .939 | .971 | .902 | .935 | .937 | .659 | .761 | .707 | .718 | .607 | .658 | .682 | .951 | .243 | .387 | .565 | .987 | .719 | .553
|  |  |  | .859 | .978 | .915 | .975 | .846 | .903 | .909 | .889 | .987 | .935 | .985 | .877 | .928 | .932 | .928 | .735 | .826 | .788 | .943 | .854 | .837 | .966 | .762 | .852 | .803 | .973 | .886 | .866
|  |  |  | .942 | .946 | .944 | .945 | .942 | .944 | .944 | .973 | .981 | .976 | .980 | .973 | .976 | .976 | .865 | .866 | .866 | .866 | .865 | .865 | .865 | .927 | .847 | .885 | .859 | .933 | .894 | .891
|  |  |  | .939 | .933 | .936 | .934 | .939 | .937 | .936 | .968 | .969 | .968 | .969 | .968 | .968 | .968 | .831 | .824 | .828 | .826 | .833 | .829 | .828 | .878 | .831 | .855 | .838 | .876 | .856 | .853
|  |  |  | .945 | .949 | .943 | .941 | .946 | .943 | .943 | .973 | .976 | .975 | .976 | .973 | .975 | .975 | .867 | .862 | .864 | .863 | .868 | .865 | .865 | .922 | .879 | .899 | .884 | .923 | .903 | .901
|  |  |  | .926 | .967 | .946 | .966 | .922 | .944 | .945 | .964 | .981 | .973 | .981 | .964 | .972 | .973 | .933 | .736 | .823 | .782 | .947 | .857 | .843 | .962 | .759 | .849 | .801 | .975 | .877 | .863
|  |  |  | .946 | .949 | .947 | .948 | .946 | .947 | .947 | .981 | .982 | .982 | .982 | .981 | .982 | .982 | .871 | .877 | .871 | .879 | .872 | .871 | .871 | .905 | .866 | .886 | .872 | .909 | .895 | .888

Table 4: Results of the Support Vector Machine classifier on Class vs.
Instance and Physical Object classification task against the reference
datasets described in Section 4. The first four columns indicate the features
used by the classifier: A is the abstract, U is the URI, E are incoming and
outgoing properties, D are the results of the alignment-based methods. P*, R*,
F*1 indicate precision, recall and F1 measure on Class (C), Instance (I),
Physical Object (PO) and the complement of Physical Object (NPO).
$\overline{\mbox{F\textsubscript{1}}}$ is the average of the F1 measures.

Physical Object. Table 4 also shows the results of the Support Vector Machine
algorithm trained on and tested against POE and POC. Similarly to the
behaviour of SENECA, statistical approaches worsen their overall performance
as compared to the case of the class vs. instance classification. We also
observe a different behaviour of the individual features. The best average
performance with POE is achieved by combining all the feature, while the best
average performance with POC is achieved by combining the abstract (A), the
outgoing/incoming properties (E), and SENECA output (D). In a sense, this
confirms that conventions used for creating URI IDs are informative mainly for
the class vs. instance distinction.

###  5.3 Remarks and Discussion

We claim that the performed experiments show promising results as far as our
research questions are concerned (cf. Section 3). Given the diversity and the
basic nature of the two distinctions that we have analysed, and the positive
results obtained in both cases by applying the same methods with the same
configurations, we claim that the proposed methods can be generalised to other
foundational distinctions.
_RQ1: Do foundational distinctions match common sense?_ As anticipated in
Section 4 the high agreement observed among workers that participated in the
crowdsourcing tasks, as well as the high agreement between the crowd and the
experts, suggest that the foundational distinctions that we have tested do
actually match common sense.
_RQ2: Does the (Semantic) Web provide an empirical basis for supporting
foundational distinctions over LOD entities, according to common sense?_ We
claim that the high average value of F1 measure associated with all
experiments indicates that the Web, and in particular LOD, implicitly encodes
foundational distinctions. We also think that, more in general, this is a hint
that the Web is a good source for common sense knowledge extraction. We find
particularly interesting to observe that the feature E (i.e. ongoing/incoming
properties) has always a positive impact, in all features combinations, on the
classifier’s performance (cf. Table 4), for both tasks. This motivates us in
conducting further investigations (i) towards identifying and testing
additional features based on LOD, e.g. more sophisticated use of assertions
and axioms from LOD as well as (ii) to analyse LOD at a much larger scale
(e.g. by using LOD Laundromat Beek et al. (2016)) with an _empirical science_
perspective: looking for emerging patterns that may encode relevant pieces of
common sense knowledge Gangemi and Presutti (2010). Our promising results open
a number of possible research directions: besides replicating these
experiments at a larger scale, we plan a follow up study concerning the
application of the same approach to distinguishing physical objects that can
act as locations for other physical objects. This is particularly relevant in
order to extract knowledge about where things are usually located in, whether
a location is appropriate for an object in terms of its size, etc. Another
relevant distinction to be investigated with priority is the one between
physical and social objects (e.g. organisations), which is often prone to
_systematic polysemy_ Pustejovsky (1998), i.e. objects that have a same
linguistic reference, but different (disjoint) types of referents. For
example, the term _National Library_ is used to refer both to an organisation
(a social object) taking care of the library’s collections, and of the related
administrative and organisational issues, and to the buildings (physical
objects) where the organisational staff works and the collections are located
in. Besides covering foundational distinctions, we aim to extend our approach
to learn or discover relational knowledge such as the one modelled and encoded
in terms of frames Fillmore (1982); Gangemi et al. (2016).
_RQ3: What ensemble of features, resources, and methods works best to make
machines learn foundational distinctions over LOD entities?_ According to our
results, statistical methods perform better than alignment-based methods. We
use supervised learning and crowdsourcing to test two very diverse
foundational distinctions, both very basic in knowledge representation and
foundational ontologies. It emerges that two features show the same ability to
positively impact on the methods’ performance, for both distinctions: A (a
text describing the entity) and E (entity’s outgoing and incoming properties).
Both features convey the semantic description of an entity: the former by
means of natural language, which characterises a huge portion of the web, the
latter by means of LOD triples, which characterise the semantic web. Based on
these observations, we argue that the method can be generalised, even if each
specific distinction may benefit from a specialised extension of the feature
set. In our case, the U feature (i.e. URI ID) clearly shows effectiveness for
the class vs. instance rather than for PO vs. NPO.
A question is whether DBpedia text is special because of its “standardised”
style of writing. Our experiments and results do not cover this issue, which
needs to be assessed in order to provide a stronger support to our claim of
generalisability. A similar doubt can be raised as far as outgoing and
incoming links are concerned. DBpedia properties mainly come from infoboxes,
which also follow, and are influenced by the standardised way of writing
Wikipedia pages. Nevertheless, for this feature we argue that the doubt does
not apply, since incoming and outgoing properties include links to and from
LOD datasets that are outside DBpedia, hence independent from the standardised
content of Wikipedia.

##  6 Conclusion

This study reports a set of experiments for assessing whether the Web, and in
particular Linked Open Data, provides an empirical basis to extract
foundational distinctions, and if they match common sense. For testing the
former, we adopt and compare two approaches, namely alignment-based methods
and machine learning methods. For the latter we use crowdsourcing and compare
the judgements of the crowd with those of experts’. For both questions we
observe promising results and define a method that can be generalised to
investigate additional distinctions. We plan experiments on other foundational
distinctions (e.g. types of locations, objects that can serve as locations or
containers, etc.) and with additional methods. Our ultimate goal is to advance
the state of the art of AI tasks requiring common sense reasoning by designing
a methodological framework that enables mass-production of common sense
knowledge, and its injection into LOD.

Acknowledgements. This work was partly funded by EU H2020 Project MARIO (grant
n. 643808). The authors thank Aldo Gangemi for his enthusiastic support and
precious hints.

## References

  * Aroyo and Welty [2015] Lora Aroyo and Chris Welty.  Truth is a lie: Crowd truth and the seven myths of human annotation.  AI Magazine, 36(1):15–24, 2015.
  * Beek et al. [2016] Wouter Beek, Laurens Rietveld, Stefan Schlobach, and Frank van Harmelen.  LOD laundromat: Why the semantic web needs centralization (even if we don’t like it).  IEEE Internet Computing, 20(2):78–81, 2016.
  * Bizer et al. [2009] Christian Bizer, Jens Lehmann, Georgi Kobilarov, Sören Auer, Christian Becker, Richard Cyganiak, and Sebastian Hellmann.  DBpedia - A crystallization point for the Web of Data.  International Journal of Web Semantics, 7(3):154–165, 2009.
  * Camacho-Collados et al. [2016] José Camacho-Collados, Mohammad Taher Pilehvar, and Roberto Navigli.  Nasari: Integrating explicit knowledge and corpus statistics for a multilingual representation of concepts and entities.  Artificial Intelligence, 240:36 – 64, 2016.
  * Davis and Marcus [2015] Ernest Davis and Gary Marcus.  Commonsense reasoning and commonsense knowledge in artificial intelligence.  Communications of the ACM, 58(9):92–103, 2015.
  * Fellbaum [1998] Christiane Fellbaum.  WordNet: an electronic lexical database.  MIT Press, 1998.
  * Fillmore [1982] Charles J. Fillmore.  Frame semantics.  In Linguistics in the Morning Calm, pages 111–137. 1982.
  * Gangemi and Presutti [2010] Aldo Gangemi and Valentina Presutti.  A pattern science for the semantic web.  Semantic Web, 1(1-2):61–68, 2010.
  * Gangemi et al. [2003] Aldo Gangemi, Roberto Navigli, and Paola Velardi.  The OntoWordNet Project: Extension and Axiomatization of Conceptual Relations in WordNet.  In Proceedings of the On the Move to Meaningful Internet Systems Conference: COOPIS, DOA and ODBASE Confederated International Conferences, pages 3–7, 2003.
  * Gangemi et al. [2012] Aldo Gangemi, Andrea Giovanni Nuzzolese, Valentina Presutti, Francesco Draicchio, Alberto Musetti, and Paolo Ciancarini.  Automatic typing of dbpedia entities.  In Proceedings of the International Semantic Web Conference (ISWC), Part I, pages 65–81, 2012.
  * Gangemi et al. [2016] Aldo Gangemi, Mehwish Alam, Luigi Asprino, Valentina Presutti, and Diego Reforgiato Recupero.  Framester: A wide coverage linguistic linked data hub.  In Proceedings of Knowledge Engineering and Knowledge Management (EKAW), pages 239–254, 2016.
  * Hayes [1989] Patrick Hayes.  The second naive physics manifesto.  In Readings in qualitative reasoning about physical systems, pages 567–585. 1989.
  * Lakoff [1987] George Lakoff.  Women, Fire, and Dangerous Things: What Categories Reveal About the Mind.  University of Chicago Press, 1987.
  * Lenat [1995] Douglas B. Lenat.  CYC: A large-scale investment in knowledge infrastructure.  Communications of the ACM, 38(11):32–38, 1995.
  * Liu and Singh [2004] Hugo Liu and Push Singh.  ConceptNet –a practical commonsense reasoning tool-kit.  BT Technology Journal, 22(4):211–226, 2004.
  * Masolo et al. [2003] Claudio Masolo, Stefano Borgo, Aldo Gangemi, Nicola Guarino, and Alessandro Oltramari.  WonderWeb Deliverable D18 - Ontology Library (Final Version), 2003.
  * Miller and Hristea [2006] George A. Miller and Florentina Hristea.  Wordnet nouns: Classes and instances.  Computational Linguistics, 32(1):1–3, 2006.
  * Navigli and Ponzetto [2012] Roberto Navigli and Simone Paolo Ponzetto.  BabelNet: The Automatic Construction, Evaluation and Application of a Wide-Coverage Multilingual Semantic Network.  Artificial Intelligence, 193:217–250, 2012.
  * Paulheim and Gangemi [2015] Heiko Paulheim and Aldo Gangemi.  Serving dbpedia with DOLCE - more than just adding a cherry on top.  In Proceedings of the International Semantic Web Conference (ISWC), Part I, pages 180–196, 2015.
  * Pease and Niles [2002] Adam Pease and Ian Niles.  IEEE standard upper ontology: a progress report.  The Knowledge Engineering Review, 17(1):65–70, 2002.
  * Pustejovsky [1998] James Pustejovsky.  The Generative Lexicon.  MIT Press, 1998.
  * Russell and Norvig [2009] Stuart Russell and Peter Norvig.  Artificial Intelligence: A Modern Approach.  Prentice Hall Press, 3rd edition, 2009.
  * Suchanek et al. [2007] Fabian M. Suchanek, Gjergji Kasneci, and Gerhard Weikum.  Yago: A Core of Semantic Knowledge.  In Proceedings of the 16th International Conference on World Wide Web (WWW), pages 697–706, 2007.