# Sandra - A Neuro-Symbolic Reasoner Based On Descriptions And Situations

Nicolas Lazzari1 Stefano De Giorgis1 Aldo Gangemi 1&Valentina Presutti1
1University of Bologna {nicolas.lazzari3, stefano.degiorgis2, aldo.gangemi,
<EMAIL_ADDRESS>
###### Abstract

This paper presents _sandra_ , a neuro-symbolic reasoner combining vectorial
representations with deductive reasoning. _Sandra_ builds a vector space
constrained by an ontology and performs reasoning over it. The geometric
nature of the reasoner allows its combination with neural networks, bridging
the gap with symbolic knowledge representations. _Sandra_ is based on the
Description and Situation (DnS) ontology design pattern, a formalization of
frame semantics. Given a set of facts (a situation) it allows to infer all
possible perspectives (descriptions) that can provide a plausible
interpretation for it, even in presence of incomplete information. We prove
that our method is correct with respect to the DnS model. We experiment with
two different tasks and their standard benchmarks, demonstrating that, without
increasing complexity, _sandra_ (i) outperforms all the baselines (ii)
provides interpretability in the classification process, and (iii) allows
control over the vector space, which is designed a priori.

##  1 Introduction

Reasoning on perspectives is relevant in many contexts and applications. In
very rigorous domains such as medicine or jurisprudence, the same case can be
interpreted through more than one description: a patient’s situation can be
analyzed through different diagnostic hypotheses; a legal state of affairs can
be interpreted by applying different, possibly conflicting, norms. In
storytelling techniques, spin doctors can frame the same topic in different
ways to support alternative political scenarios: “ _Conservatives claim that
we need relief from taxes vs. Democrats claim that taxes are investments for
us._ ” Vossen and Fokkens (2022).

Figure 1: Example of two descriptions (Commerce buy and Contest winning) that
are satisfied by a situation that involves the entities Bob, ENCOM, Laptop.
The two descriptions define the same roles, hence they provide two different
perspectives from which the situation can be interpreted.

Informally, a set of facts (situation) can be described through more than one
- potentially conflicting - perspective, which can be inferred even in
presence of partial data. This paper proposes a method and a neuro-symbolic
reasoner, named _sandra_ , able to infer all perspectives that are plausible
descriptions for a given situation.

Figure 2: Examples of some descriptions and situations alongside the
conversion of a situation into $V$ to detect the satisfied descriptions. The
process of deducing which descriptions are satisfied by $\bm{f}_{s}$ is shown
on the right.

Associating different plausible descriptions to a situation requires the
capability to reason at both the intensional and the extensional level, i.e.,
with both concepts and facts, posing a challenge on how to formally represent
the knowledge required, as well as on the reasoning technique to apply.

From a knowledge representation perspective, the Description and Situation
(DnS) Gangemi and Mika (2003) ontology design pattern addresses this problem.
DnS provides a formalization of frame semantics, generalizing over Fillmore’s
Fillmore and others (2006) and Minsky’s Minsky (1974) proposals. It has been
largely used in many ontology projects (mainly using its OWL formalization)
Scherp et al. (2009); Boyan Brodaric and Femke Reitsma and Yi Qiang (2008);
Porzel and Cangalovic (2020); Bikakis et al. (2021). DnS defines a general
vocabulary for n-ary relations, introducing the concepts of _description_ and
_situation_. A situation is a set of facts, as it is described by an observer,
involving a set of entities. In Figure 1 a situation involving three entities
is represented: Bob, ENCOM, and $s_{1}$Laptop. A description is a perspective
(a theory, a schema) defining concepts that can classify, hence interpret, the
entities observed in a situation. In Figure 1 two descriptions are
represented, Commerce buy and Contest winning. With DnS, both descriptions and
situations are formalized as first-order entities in an ontology, therefore
potentially enabling reasoning over them both at the intensional (description)
and extensional (situation) level, in the same domain of discourse. In DnS
terms, we say that a situation $s$ _satisfies_ a description $d$, when $d$ is
a plausible or correct interpretation of $s$. In Figure 1 both descriptions
(Commerce buy and Contest winning) are satisfied by the situation $s_{1}$
involving Bob, ENCOM, and Laptop, even if they are conceptually incompatible.
A perspective-based reasoner shall be able to detect this conceptual
difference and yet infer both satisfies relations.

To the best of our knowledge, this behaviour is unobtainable with existing
logic-based reasoners. Within a traditional logic approach, conflicting
concepts (such as Commerce buy and Contest winning) are correctly modelled as
disjoint classes. Therefore, classifying the situation from the example of
Figure 1 in both descriptions would lead to an inconsistency. Furthermore, no
classification would be provided if the extensional data available is partial.
Fuzzy reasoners, such as Kamide (2022), may be more tolerant but would show
the same problem (inconsistency) adding an issue of possible undecidability.
Logic-based representations and reasoning have also benefits: a clear, formal
model of descriptions is defined and the inference is always sound and
complete with respect to that model. At the same time, the data representing
situations must be structured and compliant with such model, requiring
significant effort.

Neural Network models are a solution to both obtaining an approximate
classification, and being able to inject any type of data (images, text,
etc.), not only structured data. Nevertheless, they may need a large sample
including both descriptions and situations, and the results may be
incompatible with the formal model, for example it might happen that only some
of the plausible descriptions are retrieved. In short, logic reasoning lacks
inconsistency and partial-data-tolerant deductions, while neural network
models lack the completeness of deductive inference.

Neuro-symbolic methods, such as Manhaeve et al. (2018); Geh et al. (2023); van
Krieken et al. (2022), combine important benefits of logic-based and
approximate reasoning, and (very important) they can ingest unstructured data.
Nevertheless, they rely on the same consistency assumption, which is
undesirable in a perspective-reasoning scenario.

The approach proposed in this research (_sandra_) relies on the neuro-symbolic
paradigm but addresses this issue. It formalizes and implements the inference
of the satisfies relation between situations and descriptions and, at the same
time, it provides a probability score and a deductive, interpretable
inference.

It includes: (i) a theoretical framework formalizing DnS in a differentiable
and probabilistic fashion, which we prove to be correct with respect to DnS.
It creates a vector space $V$ isomorphic to a DnS-based ontology (defining a
set of descriptions); (ii) the implementation of a neuro-symbolic
architecture, where the neural network is constrained to position any detected
situation (from any source type) in one or more subspaces of $V$, each
corresponding to a description defined in the ontology. The intuition is that
this constraint makes the network less prone to noise. Our formulation allows
efficient inference in the DnS domain. Our hypothesis is that by constraining
the representation learning process to a DnS-based ontology we can perform
perspective-based reasoning without performance loss or increase of
computational complexity.

To evaluate our hypothesis we experiment our method on two different tasks:
visual reasoning on the I-RAVEN benchmark Hu et al. (2021) and domain
generalization on Fashion-MNIST Xiao et al. (2017). For each task, we asses
the influence of adding _sandra_ to a standard baseline model and find
increased performances with polynomial complexity111in time and space with
respect to the number of entities within an ontology..

Our contribution can be summarized as follows:  1. A correct differentiable
probabilistic formalization of DnS;  2. A neuro-symbolic reasoner that
combines deductive and inductive reasoning to classify arbitrary situations
extracted by a neural network into the descriptions (perspectives) that can
plausibly interpret them

The rest of the paper provides a formalization of DnS (Section 2) and of our
method (Section 3). Section 4 describes our experiments and results, which are
further discussed, along with future work in Section 5. Section 6 discusses
relevant related work, while Section 7 concludes the paper.

##  2 Descriptions & Situations

In DnS, descriptions and situations are n-ary relations. We refer to the
arguments of a description as _roles_ , and to those of a situation as
_entities_. For example, in Figure 1, Business, Consumer, and Item are the
roles of the Commerce buy description, while ENCOM, Bob, and Laptop are the
entities of the situation $s_{1}$. A situation $s$ satisfies a description
$d$, if each entity of $s$ can be classified by one role of $d$/footnoteIn the
original formalization of DnS this is calles maximal satisfaction. Our
formalization is able to capture all possible types of satisfaction although
the terminology is simplified.. For example, $s_{1}$ satisfies Commerce buy as
Business classifies ENCOM, Consumer classifies Bob, and Item classifies
Laptop.

A formalization of DnS in First Order Logic is given in Gangemi and Mika
(2003). We extend it to allow roles as n-ary relations and provide here the
resulting formal semantics. With $\mathcal{O}$ a DnS-based ontology, we define
$\mathcal{D}$ as the set of descriptions in the ontology, with $\mathcal{R}$
the set of roles and $\mathcal{E}$ the set of entities and assume that
$\mathcal{D}\cup\mathcal{R}\cup\mathcal{E}\subset\mathcal{O}$.

######  Definition 1.

A description $d$ is a tuple $d=(r_{1},\cdots,r_{n})\in\mathcal{D}$ with
$r_{i}\in\mathcal{R}\cup(\mathcal{D}-\\{d\\})$.

For instance, in Figure 2, $\textit{Refund}\in\mathcal{D}$ with
$\textit{Consumer},\textit{Business},\textit{Item})\in\mathcal{R}$.

Notice that descriptions can be used as roles as well. For instance,
$\textit{Customer care policy}=(\textit{Commerce
buy},\textit{Refund},\textit{Reason})\in\mathcal{D}$, where $\textit{Commerce
buy},\textit{Refund}\in\mathcal{D}$.

######  Definition 2.

A situation $s$ is a tuple $s=(e_{1},\cdots,e_{n})\in\mathcal{S}$ with
$e_{i}\in\mathcal{D}\cup\mathcal{R}\cup\mathcal{E}\cup\mathcal{S}$.

For example, the tuple $(\textit{Bob},\textit{ENCOM},\textit{Laptop})$ of
Figure 2 is a situation.

A situation can also involve other situations as its entities. For example, in
Figure 2, the tuple

$\displaystyle s=($
$\displaystyle(\textit{ENCOM},\textit{Alice},\textit{Laptop}),$
$\displaystyle(\textit{Alice},\textit{ENCOM},\textit{Laptop}),$
$\displaystyle\textit{BatteryDefect})$

represents a situation in which Alice bought a Laptop from ENCOM, ENCOM
received it back from Alice and a BatteryDefect reason is specified. Both
Alice buying and ENCOM receiving the Laptop are two separate situations that
are part of another situation, which also involves BatteryDefect.

When a description $d$ explains a situation $s$, we say that $s$ satisfies
$d$, written as $d\models s$.

######  Definition 3 (satisfaction).

With $d\in\mathcal{D}$ and $s\in\mathcal{S}$

$d\models s\Leftrightarrow\forall r\in d.(\exists e\in s.e\in r)\lor(\exists
s^{\prime}\in s.r\models s^{\prime})$

In Figure 2 the situation $s=(\textit{Bob},\textit{ENCOM},\textit{Laptop})$
satisfies the descriptions Commerce buy and Contest winning. Both descriptions
are perspectives from which $s$ can be interpreted.

Assuming that $\mathcal{O}$ asserts that
$\textit{Consumer},\textit{Business}\subseteq\textit{Agent}$ (i.e. Agent is a
broader role than Consumer and Business), we have that $\textit{Commercial
transaction}\models s$ as well, since the two roles are also Agents.

######  Corollary 1.

Given $d,d^{\prime}\in\mathcal{D}$ with $d^{\prime}\subseteq d$ and
$s\in\mathcal{S}$ then $d^{\prime}\models s\Rightarrow d\models s$.

Proof to Corollary 1 trivially follows from definition 3.

Definition 3 states the condition such that a perspective is valid for a
situation. However, a description can be a plausible perspective for a
situation even if only some of its roles are fulfilled. In this case, we say
that a description _nearly_ satisfies a situation, written as
$d\widetilde{\models}s$.

######  Definition 4 (_near_ -satisfaction).

With $d\in\mathcal{D}$ and $s\in\mathcal{S}$,

$d\widetilde{\models}s\Leftrightarrow\exists r\in d.(\exists e\in s.e\in
r)\lor(\exists s^{\prime}\in s.r\models s^{\prime})$

From Definition 4 it follows that if a situation satisfies a description, then
it also nearly-satisfies it.

######  Corollary 2.

With $d\in\mathcal{D}$ and $s\in\mathcal{S}$, $d\models s\Rightarrow
d\widetilde{\models}s$

The proof is trivial and follows from Definitions 3 and 4.

##  3 Method

With reference to Section 2 we define a vector space $V$ over the field
$\mathbb{R}^{|\mathcal{R}\cup\mathcal{D}|}$. For each description
$d\in\mathcal{D}$ a subspace $V_{d}$ is identified by its roles. Assuming that
$\mathbf{v}_{s}$ is a vector representing $s\in\mathcal{S}$ and that $d\models
s$ with $d\in\mathcal{D}$, the intuition following from Definition 3 (and 4)
is that $\mathbf{v}_{s}$ must be defined by (some) entities classified by the
roles in $\mathcal{D}$ that identify the subspace $V_{d}$.

In this section, we describe how a subspace $V_{d}$ is defined and how to
deduce when a description is (nearly-)satisfied by a situation. Notation-wise,
we assume that $s\in\mathcal{S}$ and for some $d\in\mathcal{D}$, $V_{d}$ is
the subspace associated with $d$ and $B_{d}$ is the spanning set (span) of
$V_{d}$.

##### Subspace definition

The set $B_{d}$, defined as $B_{d}=\\{\mathbf{b}_{1},\cdots,\mathbf{b}_{n}\\}$
is a basis for the subspace $V_{d}$, i.e. the vectors in $B_{d}$ form a
minimal spanning set for $V_{d}$. Hence every $\mathbf{v}\in V_{d}$ can be
expressed as a linear combination (a weighted sum) of the vectors in $B_{d}$
Meyer and Stewart (2023).

Given Definitions 3 and 4, it follows that the vectors in $B_{d}$ must
originate from the description $d$ and its roles $r\in d$. We define
$\bm{f}_{d}$ as the function that converts each description $d\in\mathcal{D}$
to a vector $\bm{f}_{d}(d)\in B_{d}$. Informally, $\bm{f}_{d}(d)$ is obtained
by recursively aggregating the vector representations $\bm{v}_{r}$ of the
roles $r\in d$. A function $\phi$ computes $\bm{v}_{r}$ ensuring that
$\bm{f}_{d}(d)\in B_{d}$ and that $\bm{v}_{r}$ reflects subsumption relations
between roles.

######  Definition 5 ($\bm{f}_{d}$).

Given $d\in\mathcal{D}$, the function $\bm{f}_{d}:\mathcal{D}\rightarrow V$ is
defined as

$\bm{f}_{d}(d)=\sum\limits_{r\in d}\begin{cases}\bm{f}_{d}(r)&\text{if
}r\in\mathcal{D}\\\
[\phi(r,x_{1})\mathbin{\|}\cdots\mathbin{\|}\phi(r,x_{n})]&\text{otherwise}\end{cases}$

where $x_{i}\in\mathcal{R}\cup\mathcal{D}$ and
$\phi:(\mathcal{R}\cup\mathcal{D})\times(\mathcal{R}\cup\mathcal{D})\rightarrow\mathbb{R}$
with $\phi(x,y)=\mathds{1}_{[x\subseteq y]}$

######  Proposition 1 ($\bm{f}_{d}$ is bijective).

With $d\in\mathcal{D}$ and $\alpha,\beta\in V$

$\bm{f}_{d}(d)=\alpha\land\bm{f}_{d}(d)=\beta\Leftrightarrow\alpha=\beta$

###### Proof.

We prove that $\bm{f}_{d}$ is injective ($\dagger$) and surjective
($\ddagger$):

  * $\dagger$

With $\bm{f}_{d}(d)=\alpha\neq\beta=\bm{f}_{d}(d)\Rightarrow\exists i.\
\alpha_{i}\neq\beta_{i}\stackrel{{\scriptstyle\text{Def.
\ref{def:fd}}}}{{\Longrightarrow}}\exists x,r_{i}.\
\phi(x,r_{i})\neq\phi(x,r_{i})\Rightarrow x\subseteq r_{i}\land x\not\subseteq
r_{i}\rightarrow\bot$

  * $\ddagger$

With $\bm{f}_{d}(d)=\bm{f}_{d}(d^{\prime})$ and $d\neq
d^{\prime}\Rightarrow\forall i\
\bm{f}_{d}(d)_{i}=\bm{f}_{d}(d^{\prime})_{i}\stackrel{{\scriptstyle\text{Def.
\ref{def:fd}}}}{{\Longrightarrow}}\exists x,x^{\prime}.\
\phi(x,x^{\prime})=\phi(x^{\prime},x)=1\stackrel{{\scriptstyle\text{Def.
\ref{def:fd}}}}{{\Longrightarrow}}x\subseteq x^{\prime}\land
x^{\prime}\subseteq x\Rightarrow x=x^{\prime}\rightarrow\bot$

Given $\dagger$ and $\ddagger$, $\bm{f}_{d}$ is bijective by definition. ∎

######  Proposition 2 ($\bm{f}_{d}$ is a linear function).

With $d\in\mathcal{D},\alpha\in\mathbb{R}$

$\alpha\bm{f}_{d}(d)=\bm{f}_{d}(\alpha d)$

###### Proof.

$\alpha\bm{f}_{d}(d)=\alpha\sum\limits_{r\in
d}[\phi(r,x_{1})\mathbin{\|}\cdots\mathbin{\|}\phi(r,x_{n})]=\sum\limits_{r\in
d}[\alpha\phi(r,x_{1})\mathbin{\|}\cdots\mathbin{\|}\alpha\phi(r,x_{n})]=\bm{f}_{d}(\alpha
d)$ ∎

######  Theorem 1.

Given a description $d\in\mathcal{D}$, the set
$B=\\{\bm{f}_{d}(r_{1}),\cdots,\bm{f}_{d}(r_{n})\\}$ with
$r_{1},\cdots,r_{n}\in d$ is composed of linearly independent vectors and
forms a basis for the subspace $V_{d}$.

###### Proof.

$\alpha_{1}\bm{f}_{d}(x_{1})+\cdots+\alpha_{n}\bm{f}_{d}(x_{n})=0\stackrel{{\scriptstyle\text{Prop
\ref{prop:fd-linear-
function}}}}{{\Longleftrightarrow}}\bm{f}_{d}(\alpha_{1}r_{1})+\cdots+\bm{f}_{d}(\alpha_{n}r_{n})=0\stackrel{{\scriptstyle\text{Prop
\ref{prop:fd-linear-
function}}}}{{\Longleftrightarrow}}\bm{f}_{d}(\alpha_{1}r_{1}+\cdots+\alpha_{n}r_{n})=0\stackrel{{\scriptstyle\text{Prop
\ref{prop:fd-
bijective}}}}{{\Longleftrightarrow}}\alpha_{1}+\cdots+\alpha_{n}=0$ ∎

It follows that we can find the spanning set $B_{d}$ (and hence $V_{d}$) of
any description $d$ within the ontology $\mathcal{O}$ as
$B_{d}=\\{\bm{f}_{d}(r_{1}),\cdots,\bm{f}_{d}(r_{n})\\}$ for all $r_{i}\in d$.

We assume, without any loss of generality, that $\mathcal{O}$ can be
represented as an acyclic directed graph. It follows from Definition 5 that
for any $d\in\mathcal{D}$, $B_{d}$ can be computed in polynomial time
$\mathcal{O}(\mathcal{|D}|^{2})$. The proof follows from Definition 5: in the
worst case for each description we have to loop through every other
description.

##### Situation encoding

Given the definition of the subspace $V_{d}$, the intuition is that its
vectors are those representing all the situations $s$ such that $d\models s$.
To obtain $\bm{v}_{s}\in V$, the vector representation of $s$, we introduce
the function $\bm{f}_{s}$, which maps a situation $s\in\mathcal{S}$ into a
vector.

######  Definition 6.

Given $s\in\mathcal{S}$, $e\in\mathcal{E}$ and $r\in\mathcal{R}$, and with
$\psi:\mathcal{E}\to\mathcal{R}$ defined such that $e\in
r\Rightarrow\psi(e)=r$, the function $\bm{f}_{s}:\mathcal{S}\rightarrow V$ is
defined as

$\bm{f}_{s}(s)=\sum\limits_{e\in s}\begin{cases}\bm{f}_{s}(e)&\text{if
}e\in\mathcal{S}\\\ \bm{f}_{d}(\psi(e))&\text{otherwise}\end{cases}$

The function $\psi$ can be implemented by a deductive reasoner (e.g. Glimm et
al. (2014)) or provided externally.

The vector $\bm{v}_{s}$ produced by the function $\bm{f}_{s}$ is the
aggregation of the vector representation of the roles that classifies the
entities in $s$ and its nested situations. Nested situations are recursively
converted into a vector using the same function $\bm{f}_{s}$. Note that
$\bm{f}_{s}$ is a positive definite function, i.e. $\bm{f}_{s}(x)\geq 0\
\forall s\in\mathcal{S}$. This follows directly from $\phi$ in Definition 5.

##### Description satisfaction

Given Theorem 1 and function $\bm{f}_{s}$, we can check whether the vector
representation $\bm{v}_{s}$ lies in the subspace $V_{d}$ by checking whether
$\bm{v}_{s}$ is a linear combination of the vectors in $B_{d}$.

Consider $A_{d}$ as the $|\mathcal{R}\cup\mathcal{D}|\times|d|$ matrix whose
column-space is $B_{d}$ (i.e. the columns are formed by all the vectors
$\bm{b}\in B_{d}$). The solution $\bm{x}$ to the linear system
$A_{d}\bm{x}=\bm{v}_{s}$ contains the coefficients to obtain the vector
$\bm{v}_{s}$ as the linear combination of the vectors $\bm{b}\in B_{d}$. Since
the columns of $A_{d}$ are linearly independent by definition, it follows that
$rank(A_{d})=|d|$. The solution $\bm{x}$ is unique and can be computed as
$\bm{x}=A_{d}^{+}\bm{v}_{s}$ where $A_{d}^{+}$ is the Moore-Penrose pseudo-
inverse of $A_{d}$ Meyer and Stewart (2023).

If $\bm{x}_{i}\neq 0$ for all $\bm{x}_{i}\in\bm{x}$ then $\bm{v}_{s}$ can be
expressed as a linear combination of the vectors in $B_{d}$ meaning that
$\bm{v}_{s}\in V_{d}$. Since $B_{d}$ is directly defined after the roles of
$d$ (Definition 5), then the roles of $d$ must classify the entities in $s$.
Based on this, we define how to deduce the descriptions satisfied by $s$
according to Definition 3.

######  Theorem 2.

Given $s\in\mathcal{S}$, $d\in\mathcal{D}$, and $\bm{v}_{s}=\bm{f}_{s}(s)$ we
have that

$d\models s\Leftrightarrow\bm{v}_{s}\in V_{d}$

###### Proof.

With $d\models s$, from Definition 3 (satisfaction), we have that $\forall
r\in d.(\exists e\in s.e\in r)\lor(\exists s^{\prime}\in s.r\models
s^{\prime})$. Consider $e\in s$ with $e\in r$. From definition 6 we have that
$\bm{v}_{s}=\bm{f}_{d}(r)+\varepsilon$ where $\varepsilon\in V$ is the
residual due to the other entities in $s$. It follows that $\bm{v}_{s}$ is
linear combination of $\bm{f}_{d}(r)$. Given the arbitrary choice of $r$, this
is true for any $r\in d$ with $r\in\mathcal{R}$. From Definition 3 (near-
satisfaction) we can assume that $\exists s^{\prime}\in s.r\models
s^{\prime}$. Hence there must exist a nested situation $s^{\prime}$ in $s$
that satisfies $r\in\mathcal{D}$. The proof for $r\models
s^{\prime}\Rightarrow\bm{v}_{s^{\prime}}\in V_{r}$ is the same for $d\models
s\Rightarrow\bm{v}_{s}\in V_{d}$. Thus, by induction, $d\models
s\Rightarrow\bm{v}_{s}\in V_{d}$ holds.

If instead we assume that $\bm{v}_{s}\in V_{d}$ then, given Theorem 1 and
Definition 6, $\bm{v}_{s}$ is a linear combination of $\bm{f}_{d}(r)$ with
$r\in d$. Consider $r\in\mathcal{R}$, then given Definition 6, it must hold
that $\bm{v}_{s}=\bm{f}_{d}(r)+\varepsilon$ where $\varepsilon$ is defined as
above with $e\in s$ and $e\in r$. On the other hand, if $r\in\mathcal{D}$ then
since $\bm{v}_{s}$ is linear combination of $\bm{f}_{d}(r)$ there must exist a
nested situation $s^{\prime}\in s$ such that
$\bm{v}_{s}=\bm{v}_{s^{\prime}}+\varepsilon$ by Definition 6 and
$\bm{v}_{s^{\prime}}\in V_{r}$. It follows by induction that $r\models
s^{\prime}$. Hence $\bm{v}_{s}\in V_{d}\Rightarrow(\exists e\in s.e\in
r)\lor(\exists s^{\prime}\in s.r\models s^{\prime})$ for any arbitrary $r\in
d$. ∎

#####  $\mathds{P}$-_sandra_ (Probabilistic _sandra_)

According to Theorem 2 each element of the vector $\bm{x}$ (with
$\bm{x}=A_{d}^{+}\bm{v}_{s}$) indicates the presence (or absence) of an entity
$e\in s$ with $e\in r$ and $r\in d$. By applying the Heaviside function
$\mathds{1}_{[>0]}$ to $\bm{x}$ we can interpret it as a boolean vector.

We model the satisfaction of a description as a multinomial probability
distribution, hence the probability that $s$ satisfies $d$ is

$p(d\models
s)=\frac{\sum\limits_{i=0}^{|d|}\mathds{1}_{[>0]}(\bm{x})_{i}}{|d|}$ (1)

.

######  Definition 7.

Given $d\in\mathcal{D}$, $s\in\mathcal{S}$ and $A_{d}$ the matrix whose
column-space is $B_{d}$, we say that $s$ nearly-satisfies $d$ with probability
$\hat{p}$, written as $d\widetilde{\models}_{\hat{p}}s$, if $p(d\models
s)=\hat{p}$.

Given Definition 7, we can now state Theorem 3, which generalizes Theorem 2 to
nearly-satisfied descriptions according to Definition 4.

######  Theorem 3.

With $s\in\mathcal{S}$, $d\in\mathcal{D}$ then
$d\widetilde{\models}_{\hat{p}}s\Rightarrow d\widetilde{\models}s$ with
$\hat{p}=p(d\models s)>0$; $d\widetilde{\models}_{1}s\Rightarrow d\models s$;
$d\widetilde{\models}_{0}s\Rightarrow d\not\models s$.

###### Proof.

If $d\widetilde{\models}_{\hat{p}}s$ then, according to Definition 7, there is
some $r\in d$ such that, given the coefficient $i$ that corresponds to $r$ in
$\bm{x}$, $(\bm{x})_{i}>0$. Hence $\bm{v}_{s}$ is a linear combination of
$\bm{f}_{d}(r)$. Similarly to proof of Theorem 2, given Definition 6 it can be
proven that exists $e\in s$ with $e\in r$. Thus, from Definition 4,
$d\widetilde{\models}s$.

If $d\widetilde{\models}_{1}s$ then
$\sum_{i=0}^{|d|}\mathds{1}_{[>0]}(\bm{x})_{i}=1$ and $\bm{v}_{s}$ is linear
combination of the vector representation of all the roles $r\in d$. Thus,
$d\models s$ follows from Theorem 2.

If $d\widetilde{\models}_{0}s$ then
$\sum_{i=0}^{|d|}\mathds{1}_{[>0]}(\bm{x})_{i}=0$ and $\bm{v}_{s}$ can not be
expressed as a linear combination of any of the vector representation of the
roles $r\in d$. Thus, $d\not\models s$ follows from Theorem 2. ∎

Theorem 3 enables a crucial property of our formalization: to infer to which
degree a description is satisfied by a situation, while retaining correctness
with respect to Definitions 3 (satisfaction) and 4 (near-satisfaction).

#####  $d\mathds{P}$-_sandra_ (Differentiable $\mathds{P}$-_sandra_)

By virtue of Theorem 3, the process of deducing which descriptions are
satisfied (or nearly-satisfied) by a situation is a function
$H:V\rightarrow\mathbb{R}^{|\mathcal{D}|}$, with

$H(s)=[p(d_{1}\models s)\|\cdots\|p(d_{n}\models s)]$ (2)

with $d_{1},\cdots,d_{n}\in\mathcal{D}$.

$H$ is differentiable with respect to the input vector $\bm{v}_{s}$ (it is
composition of differentiable functions) and in particular $\nabla
H=[\delta(\bm{v}_{s})\|\cdots\|\delta(\bm{v}_{s}))]$ where $\delta$ is Dirac’s
delta and $d_{1},\cdots,d_{n}\in\mathcal{D}$. Due to the use of the Heaviside
function, the gradient is zero everywhere except for the descriptions with
probability zero, since $\delta(x)=0\ \forall x\neq 0$. This prevents an
effective use of $H$ in Machine Learning methods that rely on gradients to
learn the model’s parameters, such as in Deep Learning, where the back-
propagation algorithm is used LeCun et al. (2015).

To overcome this issue, we replace the Heaviside function
$\mathds{1}_{>0}(\bm{x})$ with the $ReLU$ function Glorot et al. (2011). We
argue, without any loss in generality, that the behaviour of the $ReLU$
function is comparable to the Heaviside function in our setting, since
$ReLU(x)>0\Leftrightarrow x>0$. Indeed, we can interpret the $ReLU$ function
as a smooth version of $\mathds{1}_{>0}(\bm{x})$. Other functions could be
used for the same purpose, such as continuous approximations of the Heaviside
function, $tanh$, logistic function etc. In this case, we rely on $ReLU$ since
empirically it suffers less from the exploding/vanishing gradient problem.
Nonetheless, we will perform a systematic assessment of other functions in the
future, as different functions might lead to different useful properties.

###  3.1 Neuro-symbolic integration

Since $d\mathds{P}$-sandra is differentiable with respect to its input vector,
it can be used as a standard neural network layer. We define the function
$\tilde{H}$ by replacing the Heaviside function in $H$ (as defined in Equation
2) with the $ReLU$ function. The function $\tilde{H}$ allows a seamless
integration of a DnS-based ontology $\mathcal{O}$ within any arbitrary neural
network $NN$.

The $NN$ approximates the function $\bm{f}_{s}$. Given $\bm{x}$ a vector
generated by a neural network, for instance the features extracted by a
Convolutional Neural Network (CNN) from an image or the embedding of a
sentence computed by a Transformer, $\bm{f}_{s}(\bm{x})\in V$.

Hence, we can check if the situation represented by $\bm{x}$ satisfies $d$ by
relying on Definition 1. Moreover, since $\tilde{H}$ is fully differentiable
with respect to its input, it is possible to define an objective function that
optimizes the output of the $NN$ such that the representation learned for
$\bm{x}$ lies in $V_{d}$ as if it was produced by $\bm{f}_{s}$.

We highlight that our experiments (c.f. Section 4) show that the
representation learned by $NN$ (a vector $\bm{x}$ originating from any source)
the vector representation of a situation $s$ manually curated for modeling or
representing that source, such as in a Knowledge Graph. This provides us with
a means to interpret the intermediate output of the $NN$, and better
understand its internal representation, without influencing the process to
solve the down-stream task. For example, given the image of a lung X-ray,
_sandra_ infers all the possible diagnoses without compromising the freedom of
the model to formulate the final prediction.

##  4 Experiments

| #Params | C | 2$\times$2 | 3$\times$3 | O-IC | O-IG | L-R | U-D
---|---|---|---|---|---|---|---|---
Baseline | 205k | 26.85 | 14.55 | 12.15 | 12.3 | 13.4 | 11.85 | 13.15
sandra | 275k | 45.75 | 16.15 | 14.1 | 14.8 | 14.85 | 13.05 | 13.15
(a) Results on the accuracy in the I-RAVEN dataset. Number of parameters of
the baseline (first column) is compared with the addition of sandra. The
remaining columns refer to the specific configuration of the I-RAVEN dataset
(number of shapes and their position)

Model | A | B | C
---|---|---|---
CNN | 16.02 | 17.21 | 43.13
sandra | 15.74 | 18.62 | 52.49
MLP | 14.56 | 16.64 | 31.34
sandra | 16.96 | 18.11 | 32.88
(b) Results on the accuracy in R-FMNIST benchmark. The parameters in the CNN
are $30k$ and $85.6k$ using sandra while $268.7k$ and $278$ using sandra in
the MLP.

Table 1: Experimental results.

We experiment the integration of $d\mathds{P}$-sandra in different neural
network architectures. Our goal is to assess whether sandra allows to perform
perspective-based reasoning without any performance loss or increase in
computational complexity, as hyphotesized in Section 1. We test our method on
two benchmarks: I-RAVEN Hu et al. (2021), a visual reasoning task, and
RotatedFashionMNIST, a domain generalisation task based on FashionMNIST Xiao
et al. (2017). The choice of the tasks is motivated by the suitability of the
data to be modeled according to DnS with a reasonable effort. All the
experiments have been trained with the AdamW optimizer on a NVIDIA RTX3090
with 24Gb of RAM.

###  4.1 Visual Reasoning

The I-RAVEN dataset, based on Raven Progressive Matrices (RPMs) is a
repository of images for visual reasoning tasks, designed to examine abstract
reasoning capabilities in neural networks Hu et al. (2021). An RPM is a
$3\times 3$ grid of images where the last image has been removed. The task is
to predict the removed image among $8$ possible alternatives Małkiński and
Mańdziuk (2022). Each image might be composed of multiple (nested) figures,
which are simple geometric shapes where the color, size and rotation are
varied Hu et al. (2021). Recent approaches frame the task as a classification
problem and tailor the model to the RPM problem. This has been done by
structuring the models such that the relationship between the images is
captured Santoro et al. (2017); Barrett et al. (2018); Zhang et al. (2019a);
He et al. (2023) or by reasoning upon the extracted visual features through
neuro-symbolic methods Xu et al. (2023); Hersche et al. (2023). Our approach
extends the baseline model proposed by Zhang et al. (2019a) with _sandra_. The
baseline model is composed by an LSTM that combines the visual features
extracted by a CNN on each image. Details on the architecture are in the
Appendix 222Included in the additional material.. We integrate _sandra_ within
the baseline model by adding a linear layer $W$ that projects the
representation obtained by the CNN in the vector space $V$. $W$ is meant to
approximate the function $\bm{f}_{s}$.

Given an image within an RPM, $s_{R}$ is the situation defined accordingly to
Definition 1 and $\hat{\bm{x}}$ the vector representing the visual features of
the image extracted by the CNN. We approximate $\bm{f}_{s}(s_{R})$ by
minimizing the binary cross entropy

$\mathcal{L}_{BCE}(\hat{H}(\bm{f}_{s}(s)),\hat{H}(ReLU(W\hat{\bm{x}}))$ (3)

where $\tilde{H}$ is computed as explained in Section 3.1.

Our model classifies an image using linear regression on
$\hat{\bm{x}}\|\hat{H}(ReLU(W\hat{\bm{x}}))$. The loss function is hence

$\mathcal{L}=\mathcal{L}_{CE}+\mathcal{L}_{BCE}(\hat{H}(ReLU(W\hat{\bm{x}})),H(\bm{f}_{s}(s)))$
(4)

where $\mathcal{L}_{CE}$ is the cross-entropy loss as defined in Zhang et al.
(2019a).

Our approach radically differs from the others since, by relying on _sandra_ ,
the visual features extracted from an image are geometrically constrained such
that it is possible to infer valid descriptions from them. The model learns to
accurately classify an image, while capturing the semantics of the ontology in
its internal representation. The influence of _sandra_ is a regularization
with respect to a DnS-based ontology. We manually compile a DnS-based ontology
(detailed in Appendix) that models the RAVEN dataset. Each image is then
converted into a situation compliant to the ontology. Relying on _sandra_
allows the inference of which shapes are contained within a figure. A
comparison of sandra with the baseline is shown in Table 1a, we report the of
number of parameters of the network (first column) and accuracy (the other
columns) of the classification. Each model has been trained using a batch size
of $32$, a learning rate of $0.001$ and the gradient is clipped to have a norm
$\leq 0.5$. The results show that integrating _sandra_ does not compromise
performances (improving them in some cases) nor it requires a significant
increase in the number of parameters. We also compare sandra with other
approaches tailored for this task and for the sake of space, the comparison is
reported in the Appendix. We remark that, despite its generality, our method
is competitive with other methods 333We remind that our primary goal is to
test our hypothesis rather than solving the downstream task..

###  4.2 Domain Generalization

Figure 3: Example of two images and the descriptions they satisfy. The first
image has a high probability of satisfying the FootWear description, which is
reasonable considering the target label. In the second image, we can
understand that the wrong classification is associated with the similar
probability given to the LowerBodyClothes and UpperBodyClothes descriptions,
leading to an unreliable result.

The domain generalization task aims at testing how a model is able to learn
representations that are independent of a specific domain, i.e. they are
effective on distributions never seen during the training process Zhou et al.
(2022). This includes, for examples, correctly classifying images seen from a
different angle than the one in the training set. Many approaches have been
investigated, from learning representations that specifically tackles the
problem Mahajan et al. (2021), to different training techniques Vedantam et
al. (2021). A wide selection of architectures and techniques is analysed in
Zhou et al. (2023). Similarly to Mahajan et al. (2021), we rely on the
Rotated-FashionMNIST (R-FMNIST) dataset. R-FMNIST is an image classification
task where the training images are rotated in three configurations:
$A=\\{30^{\circ},45^{\circ}\\}$, $B\equiv A\cup\\{60^{\circ}\\}$, $C\equiv
B\cup\\{15^{\circ},75^{\circ}\\}$. The testing set images are rotated by
$\\{0^{\circ},90^{\circ}\\}$.

The final prediction is computed using linear regression over the features
extracted by a CNN or a MLP. The architectures are described in the Appendix.
We manually create a DnS-based ontology, detailed in the Appendix, that models
the dataset. Table 1b describes the results in terms of number of parameters
of the network (in the caption) and accuracy of the classification. Each model
has been trained with a batch size of $32$ and a fixed learning rate of
$1e^{-4}$. The use of _sandra_ proves to be beneficial also in this experiment
even though the images of R-FMNIST are only described with a single label,
leading to a fairly simple ontology.

These experiments allow us to empirically show the sandra improves the
interpretability of the model and of its results, as addressed in Section 3.
Figure 3 shows the effect of sandra in the interpretation of two images, one
correctly labeled and one misclassified. By looking at the inferred
descriptions, it is possible to interpret the motivation of the mistake.

##  5 Discussion and Future Works

_Sandra_ approaches the problem of reasoning on different persepectives by
formalizing DnS in a vector space. Through the formulation described in
Section 3 it is possible to infer the descriptions satisfied by a situation.
Most importantly, it is possible to quantify the degree of satisfiability as a
probability distribution, enabling the inference of valid interpretations of a
situation regardless of their compatibility and of the amount of information
available. This is an important aspect that finds application in many domains
where rigorous reasoning with limited information is a key requirement such as
robotics, medicine or jurisprudence. We discuss some limitations and
interesting aspects that are worth further investigation.

##### DnS-shape Compatibility

_Sandra_ ’s dependency on DnS ontologies, which underpins $V$, presents a non-
trivial limitation. This is relevant if one wants to integrate it with
ontologies using different (if any) ODP. However, we argue that it is possible
to find an isomorphism between any arbitrary ontology and a DnS ontology.
Informally, we can use reification. Each binary predicate (property) $p$ can
become a class expressing a n-ary relation, whose arguments are its domain and
range. Any axiom involving the property $p$ can be reified as a description
including $p$ as role. When reversed - as expected - this transformation may
be backward inconsistent, since the inferred satisfied descriptions may
correspond to disjoint classes in the original ontology. We will explore how
to formally define such isomorphic transformation, alongside its properties
and limitations. Another consideration is that the existence of an ontology
compatible with DnS does not always guarantee improved performance. We need
further research to identify the most effective DnS transformation patterns
for integration with _sandra_. We plan to study _sandra_ ’s behavior with
other ontologies, particularly with complex ones, to find correlations between
ontology design solutions and _sandra_ ’s performance.

##### Ontology Complexity and Scalability

Another issue is the inherent computational complexity. The function
$\bm{f}_{s}$ is linear in space with the dimension of the ontology. While this
might seem an ideal condition, as the ontology gets more complex (e.g.
thousands of descriptions) this might result in a large overhead, especially
if $\bm{f}_{s}$ is approximated through a neural network. Further research is
required to identify if it is possible to obtain more compact representations,
for instance by exploiting the hierarchical structure of the ontology or by
exploring methods that pre-process the ontology removing non-essential
elements.

##### Methodological Improvements

The definitions of $\bm{f}_{d}$ and $\bm{f}_{s}$ might be improved to obtain
additional useful properties. For instance, our current method overlooks a
procedure to retrieve the original situation $s$ from $\bm{f}_{s}$. This
precludes useful outcomes, such as uniquely identifying which entity of $s$
has been classified by which role in $d$. An interesting approach is to extend
the method of Section 3 to some parts of Description Logic as well. In that
case, it might be possible to benefit from additional features that have been
formalized in the ontology but that are unexploited since they are missing in
the DnS pattern.

##### Neurosymbolic integration

The results of Section 4 provide a positive insight on how _sandra_ can be
beneficial to neural architectures. Nonetheless, the variance in performance
gains between different tasks requires a systematic analysis, in order to
better understand which kind of architectures can benefit the most from
_sandra_ and how they should be integrated. Investigating the ODPs that lead
to increased performances is an important step towards tighter integration
between Machine Learning (ML) and Knowledge Representation (KR). Notably, this
aspect has the potential to catalyse a novel paradigm in the realm of KR,
since different ML techniques might behave differently with different
ontologies.

##  6 Related Works

The idea of using geometric representation for cognitive theory was originally
proposed by Peter Gärdenfors Gardenfors (2004) as an intermediate
representation layer between sub-symbolic and symbolic knowledge. A conceptual
space depicts concepts as convex regions. Each region is shaped by the
attributes that characterise that concept. Our method follows the intuition of
conceptual spaces. Two main problems on conceptual spaces are highlighted by
Bouraoui et al. (2022). One is the derivation of region-based representations
for concepts that is difficult when data comes from real-world scenarios;
another issue is that the representations in conceptual spaces are not
amenable to expressing relational knowledge. Given the integration of a neural
network as a situation approximator, as proposed in Section 3.1, and the Dns
formalization ( Section 2), our method overcomes both limitations. Other
approaches that implement conceptual spaces, such as Bouraoui et al. (2022),
propose to interpret embeddings as conceptual spaces. Others approximate
ontologies based on their graph representation or rely on specific neural
architectures, like Logic Tensor Networks to obtain deductive inferences Chen
et al. (2021); Ebrahimi et al. (2021). Differently to these approaches, our
representation is not an approximation of the ontology. The ontology is used
to obtain a vector space in which concepts are represented. This is similar to
the approach described in Özçep et al. (2020), where geometrical properties
are exploited to obtain sound knowledge representation methods.

Other related approaches integrate symbolic representations in geometrical
spaces to support reasoning such as in Geh et al. (2023) where the authors
complement Answer Set Programming with neural networks; in Manhaeve et al.
(2018) ProbLog is extended to the use of neural predicates; and van Krieken et
al. (2022) implements approximate logical inference. These approaches are
inherently dependent on probabilistic logic, suffering from a lack of
inconsistency tolerance.

##  7 Conclusion

We contribute a novel neuro-symbolic approach named _sandra_ to address
perspective-based reasoning based on the DnS ontology design pattern. Our
experiments show that _sandra_ brings benefits to the representation learning
process without loss of performances or increase in computational complexity.

## Acknowledgments

This project has received funding from the FAIR – Future Artificial
Intelligence Research foundation as part of the grant agreement MUR n. 341 and
from the European Union’s Horizon 2020 research and innovation programme under
grant agreement No 101004746.

## References

  * Asprino et al. [2022] Luigi Asprino, Enrico Daga, Aldo Gangemi, and Paul Mulholland.  Knowledge graph construction with a façade: A unified method to access heterogeneous data sources on the web.  ACM Trans. Internet Technol., 2022.
  * Barrett et al. [2018] David Barrett, Felix Hill, Adam Santoro, Ari Morcos, and Timothy Lillicrap.  Measuring abstract reasoning in neural networks.  In International conference on machine learning, pages 511–520. PMLR, 2018.
  * Beckett et al. [2014] David Beckett, Tim Berners-Lee, Eric Prud’hommeaux, and Gavin Carothers.  Rdf 1.1 turtle.  World Wide Web Consortium, pages 18–31, 2014.
  * Bikakis et al. [2021] Antonis Bikakis, Beatrice Markhoff, Alessandro Mosca, Stephane Jean, Eero Hyvönen, Francesco Beretta, Antonis Bikakis, Eero Hyvonen, Stéphane Jean, Beatrice Markhoff, and Alessandro Mosca.  A challenge for historical research: Making data fair using a collaborative ontology management environment (ontome).  Semant. Web, 12(2):279–294, jan 2021.
  * Bouraoui et al. [2022] Zied Bouraoui, Victor Gutierrez Basulto, and Steven Schockaert.  Integrating ontologies and vector space embeddings using conceptual spaces.  International Research School in Artificial Intelligence in Bergen, 99:3, 2022.
  * Boyan Brodaric and Femke Reitsma and Yi Qiang [2008] Boyan Brodaric and Femke Reitsma and Yi Qiang.  SKIing with DOLCE : toward an e-Science knowledge infrastructure.  In Carola Eschenbach and Michael Gruninger, editor, Frontiers in Artificial Intelligence and Applications, volume 183, pages 208–219. IOS Press, 2008.
  * Chen et al. [2021] Jiaoyan Chen, Pan Hu, Ernesto Jiménez-Ruiz, Ole Magnus Holter, Denvar Antonyrajah, and Ian Horrocks.  Owl2vec*: embedding of OWL ontologies.  Mach. Learn., 110(7):1813–1845, 2021.
  * Ebrahimi et al. [2021] Monireh Ebrahimi, Aaron Eberhart, Federico Bianchi, and Pascal Hitzler.  Towards bridging the neuro-symbolic gap: deep deductive reasoners.  Appl. Intell., 51(9):6326–6348, 2021.
  * Fillmore and others [2006] Charles J Fillmore et al.  Frame semantics.  Cognitive linguistics: Basic readings, 34:373–400, 2006.
  * Gangemi and Mika [2003] Aldo Gangemi and Peter Mika.  Understanding the semantic web through descriptions and situations.  In OTM Confederated International Conferences” On the Move to Meaningful Internet Systems”, pages 689–706. Springer, 2003.
  * Gardenfors [2004] Peter Gardenfors.  Conceptual spaces: The geometry of thought.  MIT press, 2004.
  * Geh et al. [2023] Renato Lui Geh, Jonas Gonçalves, Igor Cataneo Silveira, Denis Deratani Mauá, and Fabio Gagliardi Cozman.  dpasp: A comprehensive differentiable probabilistic answer set programming environment for neurosymbolic learning and reasoning.  arXiv preprint arXiv:2308.02944, 2023.
  * Glimm et al. [2014] Birte Glimm, Ian Horrocks, Boris Motik, Giorgos Stoilos, and Zhe Wang.  Hermit: An OWL 2 reasoner.  J. Autom. Reason., 53(3):245–269, 2014.
  * Glorot et al. [2011] Xavier Glorot, Antoine Bordes, and Yoshua Bengio.  Deep sparse rectifier neural networks.  In Geoffrey J. Gordon, David B. Dunson, and Miroslav Dudík, editors, Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2011, Fort Lauderdale, USA, April 11-13, 2011, volume 15 of JMLR Proceedings, pages 315–323. JMLR.org, 2011.
  * He et al. [2023] Wentao He, Jialu Zhang, Jianfeng Ren, Ruibin Bai, and Xudong Jiang.  Hierarchical convit with attention-based relational reasoner for visual analogical reasoning.  In Brian Williams, Yiling Chen, and Jennifer Neville, editors, Thirty-Seventh AAAI Conference on Artificial Intelligence, AAAI 2023, Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence, IAAI 2023, Thirteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2023, Washington, DC, USA, February 7-14, 2023, pages 22–30. AAAI Press, 2023.
  * Hersche et al. [2023] Michael Hersche, Mustafa Zeqiri, Luca Benini, Abu Sebastian, and Abbas Rahimi.  A neuro-vector-symbolic architecture for solving raven’s progressive matrices.  Nat. Mac. Intell., 5(4):363–375, 2023.
  * Horridge and Patel-Schneider [2009] Matthew Horridge and Peter F Patel-Schneider.  Owl 2 web ontology language manchester syntax.  W3C Working Group Note, 2009.
  * Hu et al. [2021] Sheng Hu, Yuqing Ma, Xianglong Liu, Yanlu Wei, and Shihao Bai.  Stratified rule-aware network for abstract visual reasoning.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 1567–1574, 2021.
  * Kamide [2022] Norihiro Kamide.  Reasoning with inconsistency-tolerant fuzzy description logics.  In Ana Paula Rocha, Luc Steels, and H. Jaap van den Herik, editors, Proceedings of the 14th International Conference on Agents and Artificial Intelligence, ICAART 2022, Volume 3, Online Streaming, February 3-5, 2022, pages 63–74. SCITEPRESS, 2022.
  * LeCun et al. [2015] Yann LeCun, Yoshua Bengio, and Geoffrey E. Hinton.  Deep learning.  Nat., 521(7553):436–444, 2015.
  * Mahajan et al. [2021] Divyat Mahajan, Shruti Tople, and Amit Sharma.  Domain generalization using causal matching.  In International Conference on Machine Learning, pages 7313–7324. PMLR, 2021.
  * Małkiński and Mańdziuk [2022] Mikołaj Małkiński and Jacek Mańdziuk.  Deep learning methods for abstract visual reasoning: A survey on raven’s progressive matrices.  arXiv preprint arXiv:2201.12382, 2022.
  * Manhaeve et al. [2018] Robin Manhaeve, Sebastijan Dumancic, Angelika Kimmig, Thomas Demeester, and Luc De Raedt.  Deepproblog: Neural probabilistic logic programming.  Advances in neural information processing systems, 31, 2018.
  * Meyer and Stewart [2023] Carl D Meyer and Ian Stewart.  Matrix analysis and applied linear algebra.  SIAM, 2023.
  * Minsky [1974] Marvin Minsky.  A framework for representing knowledge, 1974.
  * Özçep et al. [2020] Özgür Lütfü Özçep, Mena Leemhuis, and Diedrich Wolter.  Cone semantics for logics with negation.  In IJCAI, pages 1820–1826, 2020.
  * Porzel and Cangalovic [2020] Robert Porzel and Vanja Sophie Cangalovic.  What say you: An ontological representation of imperative meaning for human-robot interaction.  In Joint Ontology Workshops, 2020.
  * Santoro et al. [2017] Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap.  A simple neural network module for relational reasoning.  Advances in neural information processing systems, 30, 2017.
  * Santoro et al. [2018] Adam Santoro, Felix Hill, David G. T. Barrett, Ari S. Morcos, and Timothy P. Lillicrap.  Measuring abstract reasoning in neural networks.  In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 4477–4486. PMLR, 2018.
  * Scherp et al. [2009] Ansgar Scherp, Thomas Franz, Carsten Saathoff, and Steffen Staab.  F–a model of events based on the foundational ontology dolce+dns ultralight.  In Proceedings of the Fifth International Conference on Knowledge Capture, K-CAP ’09, page 137–144, New York, NY, USA, 2009. Association for Computing Machinery.
  * van Krieken et al. [2022] Emile van Krieken, Thiviyan Thanapalasingam, Jakub M Tomczak, Frank van Harmelen, and Annette ten Teije.  A-nesi: A scalable approximate method for probabilistic neurosymbolic inference.  arXiv preprint arXiv:2212.12393, 2022.
  * Vedantam et al. [2021] Ramakrishna Vedantam, David Lopez-Paz, and David J. Schwab.  An empirical investigation of domain generalization with empirical risk minimizers.  In Marc’Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pages 28131–28143, 2021.
  * Vossen and Fokkens [2022] Piek Vossen and Antske Fokkens.  Creating a More Transparent Internet.  Studies in Natural Language Processing. Cambridge University Press, 2022.
  * Xiao et al. [2017] Han Xiao, Kashif Rasul, and Roland Vollgraf.  Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms.  arXiv preprint arXiv:2308.02944, 2017.
  * Xu et al. [2023] Jingyi Xu, Tushar Vaidya, Yufei Wu, Saket Chandra, Zhangsheng Lai, and Kai Fong Ernest Chong.  Abstract visual reasoning: An algebraic approach for solving raven’s progressive matrices.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6715–6724, 2023.
  * Zhang et al. [2019a] Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chun Zhu.  RAVEN: A dataset for relational and analogical visual reasoning.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 5317–5327. Computer Vision Foundation / IEEE, 2019.
  * Zhang et al. [2019b] Chi Zhang, Baoxiong Jia, Feng Gao, Yixin Zhu, Hongjing Lu, and Song-Chun Zhu.  Learning perceptual inference by contrasting.  In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 1073–1085, 2019.
  * Zhang et al. [2021] Chi Zhang, Baoxiong Jia, Song-Chun Zhu, and Yixin Zhu.  Abstract spatial-temporal reasoning via probabilistic abduction and execution.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 9736–9746. Computer Vision Foundation / IEEE, 2021.
  * Zheng et al. [2019] Kecheng Zheng, Zheng-Jun Zha, and Wei Wei.  Abstract reasoning with distracting features.  In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 5834–5845, 2019.
  * Zhou et al. [2022] Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy.  Domain generalization: A survey.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
  * Zhou et al. [2023] Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy.  Domain generalization: A survey.  IEEE Trans. Pattern Anal. Mach. Intell., 45(4):4396–4415, 2023.
  * Zhuo and Kankanhalli [2021] Tao Zhuo and Mohan S. Kankanhalli.  Effective abstract reasoning with dual-contrast network.  In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
  * Zhuo et al. [2021] Tao Zhuo, Qiang Huang, and Mohan S. Kankanhalli.  Unsupervised abstract reasoning for raven’s problem matrices.  IEEE Trans. Image Process., 30:8332–8341, 2021.

## Appendix

The Appendix provides the details omitted from the manuscript due to space
limitations and the links to the implementations to reproduce the results. In
Section A we explain in details the architectures and ontologies used in
Section 4 (experiments). The python implementation of sandra is available at
https://anonymous.4open.science/r/sandra-C3D3/.

##  Appendix A Experiments

This section provides additional details on the experiments performed in
Section 4. Section A.1 describes the model’s architecture and the ontology
used in Section 4.1. Section A.2 we describes the models’ architectures and
the ontology used in Section A.2.

###  A.1 Visual Reasoning

The original I-RAVEN dataset includes three main types of entities: Matrices,
Panels, and Figures. In I-RAVEN a Matrix is a set of 9 Panels. A Panel is a
component of the Matrix. Each Panel has a numeric value (from 1 to 9), which
encodes its position in the Matrix, from top left to bottom right. A Panel
could include a number N of Figures. Each Figure has a Shape, a Rotation
Angle, a Color, and a Size.

Figure 4: Raven Progressive Matrix example. The RPM is composed by $9$
numbered Panels ($3\times 3$), the last one will be determined based on the
$8$ explicit panels. Each Panel can include a variable number of Figures with
some Shape, Rotation Angle, Color, and Size.

For example in Figure 4, the Matrix is composed of $9$ Panels. The Shape in
the first Panel is a Figure containing a pentagon and composed of 3 small
inner circles. The Color of the inner circles is black. The outer pentagon is
rotated.

####  A.1.1 I-RAVEN ontology

We reverse-engineer the I-RAVEN conceptual module to model the I-RAVEN
intensional layer. We define the set of components that can appear in an image
(see Figure 4) based on the I-RAVEN official code 444Available at
https://github.com/husheng12345/SRAN/. We model a set of descriptions based on
these components.

Each RPM configurations is modeled as a situation. The situation is compiled
starting from the original XML generated alongside the images, which contains
structured information on the RPM such as the specific composition of each
panel.

We use SPARQL Anything Asprino et al. [2022] to interpret and transform the
XML data into situations, serialized in RDF using Turtle notation Beckett et
al. [2014]. SPARQL Anything is a querying tool for Semantic Web re-
engineering. It allows users to query several data formats with the SPARQL
query language. Through the use of the CONSTRUCT clause it is possible to
compile a Knowledge Graph from the data formats supported by it.

The complexity of the ontology accommodates various meta-levels of deductions
pertaining to different situations. For example, according to the above
description, we add the axiom rv:hasComponent some rv:Figure to the rv:Panel
class. This results in Panel being a description
($\texttt{rv:Panel}\in\mathcal{D}$) that includes the description rv:Figure
($\texttt{rv:Figure}\in\mathcal{D}$ and
$\texttt{rv:Figure}\in\texttt{rv:Panel}$). :Figure is modeled as a description
which includes the roles :Angle, :Color and :Size. In order to model the
semantics within the I-RAVEN dataset and to allow for better understanding of
the data, we model the combinatorial permutation of figures within panels as
descriptions as well. For instance, a figure containing a circle satisfies the
description $\texttt{rv:FT5}\in\mathcal{D}$, with
$\texttt{rv:T5}\in\texttt{rv:FT5}$. The role $\texttt{rv:T5}\in\mathcal{R}$ is
instantiated in a situation if the situation (i.e. the figure) contains a
circle.

The I-RAVEN ontology models a description by declaring universal restrictions
on the classes representing a role. This is done through the use of the
rdfs:subClassOf predicate. For instance, rv:FT5 rdfs:subClassOf some rv:T5,
which translates to
$(\texttt{rv:FT5})\sqsubseteq\forall(\texttt{rdfs:subClassOf}).(\texttt{rv:T5})$
in Description Logic. In total, the ontology contains $144$ classes
($|\mathcal{D}\cup\mathcal{R}|=144$). Table 5 provides a detailed description
of all the descriptions and roles in the ontology.

####  A.1.2 Model Architecture

The model’s architecture used for the visual reasoning experiments is a
straightforward CNN followed by a single-layer LSTM. The last hidden state of
the LSTM is used to perform the classification using a linear classification
head. The architecture is described in detail in Table 2.

3x3 conv., 16 filters, 2 stride, BN, ReLU
---
3x3 conv., 16 filters, 2 stride, BN, ReLU
3x3 conv., 16 filters, 2 stride, BN, ReLU
3x3 conv., 16 filters, 2 stride, BN, ReLU
Linear layer for projection in sandra’s $V$
LSTM, 1 layer, $128$ hidden dimension
Linear layer
Total of $205$ K parameters
Total of $275$ K parameters with _sandra_
Table 2: Architecture used for the experiments in the I-RAVEN dataset.

####  A.1.3 Comparison with state of the art models

In Table 6 a comparison of the results of Section 4 with recent works is
reported. An estimate of the number of parameters is also described. _Sandra_
performs better than the baseline but is outperformed by the other approaches.
We remark that the other approaches are optimized to solve the I-RAVEN task
and, differently to the baseline and to our method, they implement an
architecture that considers all the nuances of the task (e.g. explicitly
modeling panels on the same row). As already mentioned, outperforming specific
solutions to the I-RAVEN task is out of scope of our research, rather we want
to demonstrate that _sandra_ is a competitive generalized reasoner, which is
supported by its performance results against the baseline.

###  A.2 Domain Generalisation

In this section we describe the ontology developed for FashionMNIST (Section
A.2.1) and the architectures used for the experiments (Section A.2.2. The code
to reproduce the experiments is available at
https://anonymous.4open.science/r/sandra-fashionmnist-EEF0/. The original
Fashion-MNIST dataset provides 10 labels, one per each type of clothes: _T-
shirt/top, Trousers, Pullover, Dress, Coat, Sandal, Shirt, Sneaker, Bag, and
Ankle boot_. No further semantics is provided about the entities, therefore
the ontology developed for FashionMNIST is a straightforward representation
modeled after intuitive and common-sense features of the clothes represented
in the dataset.

####  A.2.1 Fashion-MNIST ontology

We re-engineer the Fashion-MNIST dataset to represent its intensional layer.
The 10 types of clothes are represented as rdfs:subClassOf four top classes:
:FootWear, :LowerBodyClothes, :UpperBodyClothes, and :Accessories. The
semantics is introduced in the ontology via several object properties, listed
in Table 7.

For example, the description $\texttt{:Pullover}\in\mathcal{D}$ has roles
$\texttt{:LongSleeves},\texttt{:Neckhole}\in\mathcal{R}$ and it is expressed
as a subclass of $\texttt{:UpperBodyClothes}\in\mathcal{D}$.

####  A.2.2 Model’s architecture

For the domain generalisation experiments two baseline architectures have been
used: a straightforward CNN followed by a linear classification head,
described in Table 3, and an MLP described in Table 4.

3x3 conv., 16 filters, 2 padding, BN, ReLU
---
Max pool
3x3 conv., 32 filters, 2 padding, BN, ReLU
Max pool
Linear layer for projection in sandra’s $V$ Linear layer
Total of $29.0$ K parameters
Total of $84.1$ K parameters with sandra
Table 3: CNN architecture used for the experiments in the Rotated-FashionMNIST
dataset. Linear layer, $256$ hidden dimension, ReLU
---
Linear layer, $256$ hidden dimension, ReLU
Linear layer for projection in sandra’s $V$ Linear layer
Total of $268.7$ K parameters
Total of $278$ K parameters with sandra
Table 4: MLP architecture used for the experiments in the Rotated-FashionMNIST dataset. Class | Set |  Purpose |  Manchester syntax axiom(s)
---|---|---|---
rv:RavenMatrix | $\mathcal{D}$ |  Represents the whole RPM |  rv:hasPanelSet some rv:PanelSet
rv:PanelSet | $\mathcal{D}$ |  Sequence of $3$ panels on a row |  rv:hasPanel some rv:Panel
rv:Panel | $\mathcal{D}$ |  Represents each of the $9$ panels in an RPM ($16$ if considering the $8$ alternatives to choose from) as shown in Figure 4 |  rv:hasNumberValue some rv:Number, rv:hasComponent some rv:Figure
Classes of the form rv:PFXY where $X\in\\{\texttt{rv:A},\texttt{rv:C},\texttt{rv:S},\texttt{rv:T}\\}$ and $Y$ depends on $X$. For example, rv:A refers to the angle with $X\in\\{0,1,2,3,4,5,6,7\\}$. | $\mathcal{D}$ |  Implements the possible permutations of elements composing and qualifying the panel |  rv:hasComponent some rv:PXY
rv:Figure | $\mathcal{D}$ |  Represents each figure in each panel. As shown in Figure 4 a figure can vary for several dimensions (color, shape, etc.) |  rv:hasAngle some rv:Angle, rv:hasColor some rv:Color, rv:hasShape some rv:Shape, rv:hasSize some rv:Size
rv:Number | $\mathcal{R}$ |  Represents the number of the panel in the RPM. |
rv:Color | $\mathcal{R}$ |  Represents the color of a figure. There are $10$ possible color variations, expressed via $10$ subclasses of the rv:Color class. |
rv:Shape | $\mathcal{R}$ |  Represents the shape of a figure. There are $5$ possible shapes, modeled as subclasses: rv:Triangle, rv:Square, rv:Pentagon, rv:Hexagon, and rv:Circle. |
rv:Angle | $\mathcal{R}$ |  Represents the angle orientation of a figure. There are $8$ possible orientations, expressed as subclasses of the rv:Angle class. |
rv:Size | $\mathcal{R}$ |  Represents the size of a figure. There are $6$ possible sizes, expressed as subclasses of the rv:Size class. |
Table 5: Detailed description of each class in the I-RAVEN ontology. The Set column describes the type of each description, the purpose comments on the class while the axiomatisation is reported using Manchester syntax Horridge and Patel-Schneider [2009]. | #Params | C | 2$\times$2 | 3$\times$3 | O-IC | O-IG | L-R | U-D
---|---|---|---|---|---|---|---|---
Baseline Zhang et al. [2019a] | 205k | 26.85 | 14.55 | 12.15 | 12.3 | 13.4 | 11.85 | 13.15
WReN Santoro et al. [2018] |  $>$ 1.5M | 29.4 | 26.8 | 23.5 | 22.5 | 21.5 | 21.9 | 21.4
ResNet Zhang et al. [2019a] |  $>$ 25M | 44.7 | 29.3 | 27.9 | 46.2 | 35.8 | 51.2 | 47.4
ResNet + DRT Zhang et al. [2019a] |  $>$ 25M | 46.5 | 28.8 | 27.3 | 46 | 34.2 | 50.1 | 49.8
LEN Zheng et al. [2019] |  $>$ 5M | 56.4 | 31.7 | 29.7 | 52.1 | 31.7 | 44.1 | 44.2
CoPINet Zhang et al. [2019b] |  $>$ 5M | 54.4 | 36.8 | 31.9 | 52.2 | 42.8 | 51.9 | 52.5
DCNet Zhuo and Kankanhalli [2021] |  $>$ 5M | 57.8 | 34.1 | 35.5 | 57 | 42.9 | 58.5 | 60
NCD Zhuo et al. [2021] |  $>$ 11M | 60 | 31.2 | 30 | 62.4 | 39 | 58.9 | 57.2
SRAN Hu et al. [2021] |  $>$ 33M | 78.2 | 50.1 | 42.4 | 68.2 | 46.3 | 70.1 | 70.3
PrAE Zhang et al. [2021] $\dagger$ |  $>$ 150k | 90.5 | 85.35 | 45.60 | 74.7 | 99.7 | 99.5 | 97.4
ConViT He et al. [2023] |  $>$ 5M | 99.9 | 96.2 | 75.5 | 99.4 | 99.6 | 99.5 | 87.3
Hersche et al. [2023] $\dagger$ |  $>$ 11M | 100 | 99.5 | 97.1 | 100 | 100 | 100 | 96.4
Baseline + sandra | 275k | 45.75 | 16.15 | 14.1 | 14.8 | 14.85 | 13.05 | 13.15
Table 6: Results on the accuracy in the I-RAVEN dataset. The number of parameters for related works is a rough estimation based on the architecture used - e.g. we assume that when using a ResNet18 the model will have at least $11M$ parameters. It serves as a comparison between the number of parameters of the implemented baseline and those on related works. Results are retrieved from the corresponding papers. Models with $\dagger$ include the use of external solvers, such as planning algorithms. F-MNIST property |  Purpose |  Manchester syntax axiom example
---|---|---
:covers |  Expresses which part of the body is covered by a piece of clothing, if any |  :Dress :covers some :WholeBody, :Sandal :covers some :Feet
:hasClosure |  Expresses the type of closure of the piece of clothing (e.g. zip, buttons, laces, etc.), if any |  :Coat :hasClosure some :ButtonClosure
:hasShape |  Expresses if the entity covers entirely the body part or if it leaves some uncovered parts |  :Sandal :hasShape some :OpenShape, :Sneaker :hasShape some :ClosedShape
:hasSleeves |  Expresses the entity’s kind of sleeves, if any |  :Pullover :hasSleeves some :LongSleeves, :T-shirt_top :hasSleeves some :ShortSleeves
:wearingContext |  Expresses the prototypical context in which a certain entity can be worn |  :AnkleBoot :wearingContext some :FormalContext, :T-shirt_top :wearingContext some :InformalContext
:wearingPoint |  Expresses the part of the body from which an item of clothing is first worn |  :Coat :wearingPoint some :SleevesHole, :Pullover :wearingPoint some :NeckHole
Table 7: Details of each property used to axiomatize each description in the
ontology. Axioms examples can be seen in the Axiomatization column, written in
Manchester syntax Horridge and Patel-Schneider [2009].