# Online Grounding of Symbolic Planning Domains in Unknown Environments

Leonardo Lamanna1,2 Luciano Serafini1 Alessandro Saetti2 Alfonso Gerevini2
Paolo Traverso1 1Fondazione Bruno Kessler, Trento, Italy
2University of Brescia, Italy<EMAIL_ADDRESS>{alessandro.saetti,
<EMAIL_ADDRESS>
###### Abstract

If a robotic agent want to exploit symbolic planning techniques to achieve
some goal, it must be able to properly ground an abstract planning domain in
the environment in which it operates. However, if the environment is initially
unknown by the agent, the agent need to explore it and discover the salient
aspects of the environment needed to reach its goals. Namely, the agent has to
discover: (i) the objects present in the environment, (ii) the properties of
these objects and their relations, and finally (iii) how abstract actions can
be successfully executed. The paper proposes a framework that aims to
accomplish the aforementioned perspective for an agent that perceives the
environment partially and subjectively, through real value sensors (e.g., GPS,
and on-board camera) and can operate in the environment through low level
actuators (e.g., move forward of 20 cm). We evaluate the proposed architecture
in photo-realistic simulated environments, where the sensors are RGB-D on-
board camera, GPS and compass, and low level actions include movements,
grasping/releasing objects, and manipulating objects. The agent is placed in
an unknown environment and asked to find objects of a certain type, place an
object on top of another, close or open an object of a certain type. We
compare our approach with the state of the art methods on object goal
navigation based on reinforcement learning, showing better performances.

##  1 Introduction

Symbolic planners are powerful and flexible tools (?) that, given a general
symbolic description of an available set of actions (i.e., a planning domain)
and a detailed description of an environment, are capable of generating plans
for achieving ideally any goal about (known) objects in the environment. In
several applications, the information about the environment required to
instantiate a planning domain is not available from the beginning. In
particular, when an agent is placed in a new environment, it does not know the
objects that populate the environment, and therefore it does not know their
specific properties and relations. Consider, for instance, a robot that has to
move around and manipulate objects in a kitchen (tables, chairs, apples, etc.)
without knowing which and how many objects are really in the room. In this
setting, the exploitation of a planning domain is a compelling challenge for
three main reasons. First, in realistic environments, it is unfeasible for the
robot to acquire a complete/correct and sufficiently detailed description of
the environment before starting to plan and execute actions towards the
achievement of its goals. Second, a robotic agent usually has a first-person
perspective and partial view of the environment (e.g., by an on-board camera),
so that the only way to acquire symbolic knowledge suitable for planning is by
executing actions, observing their effects through its sensors, and mapping
the sensory data (e.g., raw images) in a symbolic state. Third, high level
actions of the planning domain are not directly executable by the robot, and
therefore they need to be compiled to low level actions executable in the
environment by the agent actuators. For instance, the action
$\mathsf{goCloseTo}(\mathsf{table}_{0})$ is compiled into a sequence of robot
movements and rotations, which follow the path provided by a path-planner, and
moves the robot to the (nearest) location close to object
$\mathsf{table}_{0}$.

This paper proposes a framework for agents that incrementally instantiate a
planning domain, specified in PDDL, by planning, acting, and sensing, in an
unknown environment. At each time-point, the belief of the agent about the
current state of the environment is represented by three components, namely:
$(i)$ the set of objects currently known by the agent and their properties
expressed with the predicates of the PDDL domain, e.g., $\mathsf{apple}_{1}$,
$\mathsf{table}_{0}$, and
$\mathsf{on}(\mathsf{apple}_{1},\mathsf{table}_{0})$; $(ii)$ for each known
object, a set of low-level features as perceived by the agent, e.g., visual
features and positions of $\mathsf{apple}_{1}$ and $\mathsf{table}_{0}$;
$(iii)$ a set of global features associated to the current environment state,
e.g., an occupancy map of the environment, and the current pose of the robot.

For this framework, we propose an online iterative algorithm, called Ogamus
(Online Grounding of Action Models in Unknown Situations), that allows an
agent, equipped with a lifted PDDL planning domain, and placed in an unknown
environment, to achieve a set of goals expressed in the language of its PDDL
action model. The agent is initialized without prior knowledge about the
environment where it has to operate, i.e., with the empty set of objects, the
empty set of their properties, and all the points of the occupancy map set as
traversable. Ogamus attempts to achieve the goal by combining four main
activities, namely: (i) _exploring the environment_ to acquire the knowledge
needed to achieve the goals; (ii) _abstracting the sensor information_
obtained at every step into a symbolic state; (iii) performing _symbolic
planning_ in the abstract model grounded with the current beliefs and current
abstract state; (iv) _executing_ the planned abstract actions by compiling
them into low-level operations suitable for the current state of the
environment.

The main features of Ogamus are the following. _Generality:_ Ogamus is able to
deal with any goal that can be expressed by a (first-order) formula using the
predicates of the PDDL domain. For instance the goal of putting “two apples on
a table” can be specified by the formula $\exists
x\,y\,z.\mathsf{on}(x,z)\wedge\mathsf{on}(y,z)\wedge\mathsf{apple}(x)\wedge\mathsf{apple}(y)\wedge\mathsf{table}(z)\wedge
x\neq y$. Notice that goals are expressed with existentially quantified
variables; this is because, initially, the agent is not aware of any objects
in the domain. An important step necessary to achieve the goal is indeed
discovering the objects for replacing the existential variables.
_Explainability_ : The behaviour of the agent, its plans, and the effects of
actions are represented at a symbolic level in which the states of the PDDL
domain are derived at every step by abstracting the sensory data.
_Robustness:_ The action model, the obtained symbolic state representation,
and the action compilation are not required to be fault free. As
experimentally shown in this paper, Ogamus achieves high success rate even
with low precision object detectors and classifiers.

We have implemented and experimentally evaluated Ogamus in the iThor (?) and
RoboThor (?) simulated photo-realistic environments for embodied AI. We
evaluate Ogamus on different tasks including “go close to an object of a
certain type” (e.g., “go close to a fridge”). In the area of Embodied AI, this
class of tasks is called “object goal navigation”, and it is still considered
a challenging and open problem. Our framework and evaluation go beyond such a
challenge, considering more complex tasks, such as “open/close an object of a
certain type” (e.g., “open a laptop”), and “put an object of a type on top of
an object of another type” (e.g., “put an apple on a table”). We compared our
results with the state-of-the-art approaches on the object goal navigation
task. In particular, we show that, in the benchmarks of the last RoboThor
object goal navigation challenge, our approach outperforms the state of the
art methods based on Reinforcement Learning
(RL).111https://ai2thor.allenai.org/ The evaluation on the other tasks leads
to very positive performance, but we cannot compare them with any state-of-
the-art approach since, to the best of our knowledge, we are the first solving
such types of tasks in the iThor and RoboThor environment.

The paper is structured as follows: firstly we analyze the related literature,
then we describe the framework and the algorithm adopted by the agent to reach
a specific goal in an unknown environment; finally, we present the
experimental evaluation and a comparison with RL-based approaches.

##  2 Related Work

The problem of integrating symbolic action models with low level sensory data
and actions has been addressed by different approaches. Most of them are based
on RL techniques. ? (?) propose a framework, called SDRL, which combines
symbolic planning on PDDL and Deep RL to learn policies that compile high
level actions into low level operations. SDRL assumes that a grounded domain
model is provided in input and never updated. A fundamental difference w.r.t.
our approach is that Ogamus, instead, learns how to ground the domain model
with new objects discovered online. Moreover, SDRL assumes a perfect oracle
that maps low level perceptions into symbolic states, while Ogamus deals with
faulty mappings without assuming to have an oracle.

NSRL (?) represents abstract domains in first-order logic and uses RL to learn
high level policies. NSRL generates a compact representation of the learned
policies as a set of rules via Inductive Logic Programming. Similarly to SDRL,
NSRL assumes a given and fixed abstract domain instantiation and a perfect
mapping from sensory data to symbolic states, while Ogamus overcomes these
assumptions.

DPDL (?) represents abstract domains in PDDL. It learns online both mappings
from sensory data to symbolic states and low level policies for high level
actions. In Ogamus, instead, the mapping from perceptions to symbolic states
is obtained by combining a set of neural networks that are trained off-line.
Moreover some of the high level actions are pre-compiled in low level
operations (e.g., pick-up an object at a given position), while policies for
moving actions are computed online via path planning. As the other methods
mentioned above, DPDL assumes a given and fixed grounded PDDL domain, while
this is not the case for Ogamus. Moreover, it focuses on performing
manipulation tasks in a single scene type (i.e., a kitchen). Ogamus, instead,
can work on different scenes (we evaluate it on 35 different scenes grouped by
kitchens, living-rooms, bedrooms, bathrooms and apartments). Finally, DPDL
works with an external fixed camera, while Ogamus uses egocentric and dynamic
views. This makes the tasks more difficult for Ogamus, since the agent needs
to navigate, explore the environment, and find new objects outside of its
current view.

Differently from the above mentioned works, in the approach proposed by ? (?),
like in Ogamus, the agent instantiate the abstract domain online by augmenting
the set of objects every time it discovers new ones. The states of the
instantiated abstract model is represented with a set of propositional atoms
on the current set of constants. However, this approach is evaluated only with
an extremely simple environment, while Ogamus is tested in accurate and photo-
realistic simulated environments with complex objects and egocentric views.
Furthermore, the approach in (?) does not take advantage of the power of
symbolic planning techniques on PDDL domain descriptions, and it does not
generalize over different tasks. On the contrary, Ogamus exploits symbolic
planning which allows to accomplish variegated tasks that can be specified
using PDDL.

We experimentally evaluate Ogamus on the Object Goal Navigation task that has
recently received much attention in the embodied AI community (?; ?; ?; ?; ?;
?; ?; ?; ?). We experimentally show that, in the benchmark of the last
RoboThor object goal navigation challenge, Ogamus performs better than a
method based on DD-PPO (?), which won the challenge using pure RL based on
low-level features, without exploiting a symbolic domain.

##  3 The Framework

In this section, we start by introducing the basic definitions of our
reference framework. We then introduce the Ogamus algorithm. The section
closes with the description of what are the basic components that need to be
specified in order to cope with new abstract predicates or actions of the PDDL
domain.

###  3.1 Preliminary definitions

Let $\mathcal{P}$ be a set of first order predicates, $\mathcal{V}$ a set of
variables (also called parameters), and $\mathcal{C}$ a set of constants. We
use $\mathcal{P}(\mathcal{V})$ to denote the set of atoms
$P(x_{1},\dots,x_{m})$, where $x_{i}\in\mathcal{V}$ and $P\in\mathcal{P}$, and
$\mathcal{P}(\mathcal{C})$ to denote the set of atoms obtained by grounding
$\mathcal{P}(\mathcal{V})$ with the constants in $\mathcal{C}$.

######  Definition 1 (Action model).

Given a set of operators $\mathcal{O}$, an _action model_ $\mathcal{M}$
associates to each $op\in\mathcal{O}$ an _action schema_ , which is a tuple
$\left<\mathsf{par}(op),\mathsf{pre}(op),\mathsf{eff}^{+}(op),\mathsf{eff}^{-}(op)\right>$,
where $\mathsf{par}(op)\subseteq\mathcal{V}$, $\mathsf{pre}(op)$,
$\mathsf{eff}^{+}(op)$, and $\mathsf{eff}^{-}(op)$ are subsets of
$\mathcal{P}(\mathsf{par}(op))$.

######  Definition 2 (Ground action).

The _ground action_ $op(\bm{c})$ of an operator $op\in\mathcal{O}$ with
$\bm{c}=\langle c_{1},\dots,c_{n}\rangle$ constants in $\mathcal{C}$ is the
tuple
$\langle\mathsf{pre}(op(\bm{c})),\mathsf{eff}^{+}(op(\bm{c})),\mathsf{eff}^{-}(op(\bm{c}))\rangle$,
obtained by instantiating the atoms of $\mathsf{pre}(op)$,
$\mathsf{eff}^{+}(op)$, and $\mathsf{eff}^{-}(op)$ with $\bm{c}$.

######  Definition 3 (Planning problem).

A _planning problem_ is a tuple
$\langle\mathcal{M},\mathcal{C},s_{0},\mathcal{G}\rangle$ where $\mathcal{M}$
is an action model, $\mathcal{C}$ is a (possibly empty) set of constants,
$s_{0}\subseteq\mathcal{P}(\mathcal{C})$ is the initial state, and
$\mathcal{G}$ is a first order formula over $\mathcal{P}$, $\mathcal{V}$ and
$\mathcal{C}$.

######  Definition 4 (Plan).

A _plan_ for a planning problem
$\langle\mathcal{M},\mathcal{C},s_{0},\mathcal{G}\rangle$ is a sequence
$\langle op_{1}(\bm{c}_{1}),\dots,op_{n}(\bm{c}_{n})\rangle$ such that there
is a sequence $\langle s_{1},\dots,s_{n}\rangle$ of subsets of
$\mathcal{P}(\mathcal{C})$ (aka states), such that for every $0\leq i<n$,
$\mathsf{pre}(op_{i}(\bm{c}_{i}))\subseteq s_{i}$,
$s_{i}=s_{i-1}\cup\mathsf{eff}^{+}(op_{i}(\bm{c}_{i}))\setminus\mathsf{eff}^{-}(op_{i}(\bm{c}_{i}))$,
and $s_{n}\models\mathcal{G}$.

Notice that our definition of planning problem allows to express first-order
goal formula $\mathcal{G}$. We say that a state $s\models\mathcal{G}$ iff
$\bigwedge_{P(\bm{c})\in
s}P(\bm{c})\wedge\bigwedge_{P(\bm{c})\in\mathcal{P}(\mathcal{C})\setminus
s}\neg P(\bm{c})\models\mathcal{G}$, under the assumption that all the
elements of the problem are in $\mathcal{C}$.

In order to use an abstract model, an agent needs to _anchor_ the symbols
occurring in the states of the planning domain with the real-world
perceptions, and to map abstract actions into actions executable in the real
world (?). We suppose that the agent can _partially_ observe the current state
of the environment through a set of sensors, for instance images provided by
an RGB-D camera, which do not directly correspond to the states of the
abstract model. Furthermore, the set of sensors provide only a partial and
subjective view of the environment. For instance, the RGB-D camera provides
only an egocentric view of a portion of the room visible by the agent. We also
suppose that the agent interacts with the environment by executing low-level
operations (e.g., move 25 cm forward, rotate $30^{\circ}$ left, pick up or put
down an object at the GPS-coordinates $(x,y,z)$), which are different from the
actions in the abstract action model. We need therefore to link the abstract
state to real perceptions, and the abstract actions to operations executable
by the actuators of the agent. Let us first consider the relationship between
abstract states and perceptions.

Object and state anchoring.

Every object that the agent is aware of at a given instant is represented by a
constant $c\in\mathcal{C}$ that is the internal identifier for such an object.
Following the approaches to symbol anchoring proposed in the literature (?;
?), every constant $c\in\mathcal{C}$ is associated with a tuple of numeric
features denoted by $\bm{z}_{c}$. For instance, $\bm{z}_{c}$ might include the
estimated position of $c$ and a set of visual features of the different views
of $c$. In addition, for each state $s$ determined by the agent, we have a
vector of state features $\bm{z}_{s}$, consisting of the 3D position of the
agent in the environment, the orientation of the agent relative to its initial
pose, the information about the success of the last low-level operation made
by the agent, and an occupancy map of the environment. The occupancy map is a
2D map of the environment storing the areas that are believed to be
traversable by the robot. The occupancy map is initialized so that every point
is traversable.

Predicate predictors.

In order to map the perceptions about objects into atoms of the symbolic
state, the agent associates to every predicate a probabilistic model, e.g., a
neural network, that computes the probability of a certain atom $P(\bm{c})$ to
be true given the features associated to $\bm{c}$ and the current state ones,
i.e., $Pr(Y_{P(\bm{c})}=\mathrm{True}\mid\bm{z}_{\bm{c}},\bm{z}_{s})$, where
$Y_{P(\bm{c})}$ is a boolean random variable associated to the atom
$P(\bm{c})$. These probabilistic models can be updated during execution on the
basis of new observations. In this paper, however, we suppose that these
probabilistic models are given (e.g., a pre-trained neural network), and they
are not modified during execution.

We call belief state the agent’s knowledge about object/state anchoring and
predicate predictors.

######  Definition 5.

An agent belief state is a 5-tuple
$\langle\mathcal{C},\bm{z}_{\mathcal{C}},s,\bm{z}_{s},\mathbf{Pr}\rangle$
where:

  * –

$\mathcal{C}$ is a set of constants;

  * –

$\bm{z}_{\mathcal{C}}=\\{\bm{z}_{c}\\}_{c\in\mathcal{C}}$ is a set of object
feature vectors $\bm{z}_{c}$;

  * –

$s\subseteq\mathcal{P}(\mathcal{C})$ is the set of atoms that are believed to
be true;

  * –

$\bm{z}_{s}$ is a vector of state features;

  * –

$\mathbf{Pr}=\\{Pr(Y_{P(\bm{c})}\mid\bm{z}_{\bm{c}},\bm{z}_{s})\\}_{P\in\mathcal{P}}$
is the set of probabilistic models used to predict the truth value of
$P(\bm{c})$ given the features $\bm{z}_{s}$ and $\bm{z}_{\bm{c}}$ associated
with the constants in $\bm{c}$.

###  3.2 The Ogamus algorithm

So far, we have not considered how the set $\mathcal{C}$ of constants
identifying objects is obtained by the agent. We do not assume that they are
given a priori to the agent; instead, we are interested in providing the agent
with the capability to discover objects by adding new constants to the
representation of the environment, updating the anchor to an object, merging
two constants anchored to the same object, and deleting a constant from the
representation that was erroneously identifying a non existing object in the
environment.

Let $\bm{x}$ be the vector that contains the data returned by the sensors
(i.e., the observations) at a given time; the agent extracts from $\bm{x}$ a
set of objects $\mathcal{C}_{\bm{x}}$, and for each object
$c\in\mathcal{C}_{\bm{x}}$ a feature vector $\bm{z}_{c}$. Since the agent can
also recognize objects that it has already seen, it is possible that
$\mathcal{C}_{\bm{x}}\cap\mathcal{C}\neq\emptyset$.

Algorithm 1 Ogamus algorithm

Input: $\mathcal{M}$, $\mathcal{G}$, $\mathbf{Pr}$ and
$\textsc{MaxIter}\in\mathbb{N}$.
Output: Success/Fail

1:
$\langle\mathcal{C},\bm{z}_{\mathcal{C}},s,\bm{z}_{s}\rangle\leftarrow\langle\emptyset,\emptyset,\emptyset,(\mathbf{0},nil,\emptyset)\rangle$

2: for $1=0$ to MaxIter do

3: if $s\models\mathcal{G}$ then

4: return Success

5: end if

6: $\pi\leftarrow\textsc{Plan}(\mathcal{M},\mathcal{C},s,\mathcal{G})$

7: if $\pi=\textsc{None}$ then

8: $\bm{e}\leftarrow\textsc{explore}(\bm{z}_{s})$

9: else

10: $op(\bm{c})\leftarrow\textsc{Pop}(\pi)$

11: $\bm{e}\leftarrow\textsc{compile}(op(\bm{c}),\bm{z}_{\bm{c}},\bm{z}_{s})$

12: end if

13: $e_{1}\leftarrow$Pop$(\bm{e})$

14: $\bm{x}\leftarrow\textsc{Exec}(e_{1})$

15: $\bm{z}_{s}\leftarrow\textsc{getStateFeatures}(\bm{x})$

16:
$\mathcal{C}_{\bm{x}},\bm{z}_{\mathcal{C}_{\bm{x}}}\leftarrow\textsc{GetObjs}(\bm{x})$

17:
$\mathcal{C},\bm{z}_{\mathcal{C}}\leftarrow\textsc{updateObjs}(\mathcal{C},\bm{z}_{\mathcal{C}},\mathcal{C}_{\bm{x}},\bm{z}_{\mathcal{C}_{\bm{x}}})$

18:
$\textit{Pr}(\bm{Y}_{\mathcal{P}(\mathcal{C})})\leftarrow\textsc{PredictState}(\bm{z}_{\mathcal{C}},s,\bm{z}_{s})$

19:
$s\leftarrow\\{p(\bm{c})\in\mathcal{P}(\mathcal{C})\mid\textit{Pr}(Y_{p(\bm{c})}=\textit{True}\mid\bm{z}_{\bm{c}})>\\!1\\!-\\!\epsilon\\}$

20: if $\pi{\not=}$ None and succeed($op(\bm{c})$) then

21: $s\leftarrow
s\cup\mathsf{eff}^{+}(op(\bm{c}))\setminus\mathsf{eff}^{-}(op(\bm{c})))$

22: end if

23: end for

24: return Fail

In the following, we shortly describe the Ogamus algorithm (Algorithm 1).

  * –

The algorithm takes as input an action model $\mathcal{M}$, a set
$\mathbf{Pr}$ of probabilistic models for predicting the predicates in
$\mathcal{P}$, a goal formula $\mathcal{G}$, and a maximum number of
iterations. Notice that the goal $\mathcal{G}$ cannot contain constants, since
we suppose that at the beginning the agent is not aware of any object. For
instance, the goal requiring that an apple is inside a box can be encoded by
the PDDL expression representing formula $\exists x,y\leavevmode\nobreak\
\mathsf{apple}(x)\land\mathsf{box}(y)\land\mathsf{in}(x,y)$. The main
objective of the algorithm is to reach a state that achieves the goal.

  * –

The agent starts by initializing all the components of its state to the
emptyset (line 1). We assume indeed that the agent is not aware of any object
in the environment, therefore $\mathcal{C}=\emptyset$. Since $\mathcal{C}$ is
empty, $\bm{z}_{C}$, $\mathcal{P}(\mathcal{C})$ and $s$ are also empty. The
information in $\bm{z}_{s}$ representing the position and orientation of the
agent is initialized with a vector of $0$’s; the information in $\bm{z}_{s}$
about the success of the last operation is set to nil; finally, the occupancy
map of the environment in $\bm{z}_{s}$ is set to an empty map so that all the
points are traversable.

  * –

Then the agent iterates for a maximum number of steps, checking if the current
state $s$ satisfies the goal (line 4); when this is the case, it returns
Success.

  * –

Otherwise, the agent invokes a planner (line 6) to solve the planning problem
defined on the input action model, the current set of objects, the current
state, and the input goal formula $\mathcal{G}$.

  * –

If the planner does not find a plan that satisfies the goal, then the agent
explores the environment in order to discover new objects that are needed to
satisfy the goal. For instance, if the goal is to put an apple into a box,
then the planner can find a plan only if in the current state $s$ there is at
least one object of type apple and one of type box. For the exploration phase
(line 8), the agent randomly selects a target position on the occupancy map
(stored in $\bm{z}_{s}$) that it believes to be free from other obstacles. As
shown in Figure 1, $\textsc{explore}(\bm{z}_{s})$ calls a path planner that
checks if such a position is reachable (if it is not reachable a new position
is selected) and returns a sequence $\bm{e}$ of low-level navigation and
rotation operations, which, according to the current knowledge of the agent,
moves the agent from its current position to the selected target. For
efficiency reasons, this path is computed in an approximated occupancy map
obtained by discretizing the occupancy map through a grid.

Agent position
${\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\\{x=0.9,y=0.4\\}}$
Target position
${\color[rgb]{0,1,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,0}\\{x=0.2,y=0.2\\}}$
PositionsOccupancy mapPath plannerPath
planmoveAheadmoveAheadmoveAheadrotateLeftrotateLeft...Low-level operations
Figure 1: Example of explore($\bm{z}_{s}$). The occupancy map, agent position
(in blue) and target position (in green) are given as input to a path planner
which discretizes the occupancy map, computes a path plan, and compile the
path plan into a sequence of low-level operations.  RGB viewObject
detectorDetected objectsImage cropbox0 imageDepth view Position estimator
$<-0.2,1.2,-0.4>$box0 positionbboxes Figure 2: Example of object features
extraction. The object detector takes as input the perception composed by the
agent view RGB image, and returns a set of bounding boxes together with the
detected object types. The object bounding boxes are used with the agent view
depth image to estimate their positions (in meters w.r.t. the initial agent
position). The lighter the pixels in the box, the farther the associated
object.

The execution of such a sequence of operations might fail due to the partial
or incorrect knowledge of the agent, i.e., when the agent wrongly believes
that a certain area on the path returned by the path-planner is traversable
while there is an obstacle. In Figure 1, the agent fails to reach the red
cell. Indeed, the execution of the third $\mathsf{moveAhead}$ action fails
because the robot bumps into a table, whose size was only partially determined
at the beginning of the exploration phase. The exploration terminates whenever
the goal is reachable according to the learned problem. For example, consider
the goal requiring that an apple is in a box and assume that an apple has
already been discovered in the environment. The execution of the computed
sequence of operations terminates in advance, when the agent detects the
presence of a box on the table in Figure 1, as it approaches the table by
executing the first operations in the sequence.

  * –

If instead the planner succeeds and returns a valid plan $\pi$, then the first
action of $\pi$ is compiled into a sequence of low-level operations $\bm{e}$
(line 11). The compilation of the action is based on the object and state
features available in the agent’s state. For instance, the high level action
$\mathsf{pickup}(c)$ is compiled into the low-level operation
$\mathsf{pickup}(x,y,z)$ where $(x,y,z)$ is the current (believed) position of
object $c$, memorized in $\bm{z}_{c}$. The simulator executes such an
operation by picking up what is present at these coordinates. To compile the
action $\mathsf{goCloseTo}(c)$ instead, the agent calls a path-planner that
provides a path from the current position of the agent (memorized in
$\bm{z}_{s}$) to a position close to $c$.

  * –

Successively, the first operation of sequence $\bm{e}$ is executed (line 14),
and a new observation $\bm{x}$ is obtained. The execution of the first
operation may fail or not. In both cases, the agent can acquire new knowledge
(e.g., discover new objects or an obstacle), which can be used to produce a
better compilation of an high-level action, and/or produce a better plan.
Then, the new state features $\bm{z}_{s}$ are extracted from the sensory data
$\bm{x}$ (line 15). The information about the occupancy map is updated using
the information of success/failure of the action and the depth image.

  * –

Then the agent runs an object detector (line 16) on the RGB image contained in
observation $\bm{x}$ which returns a set of objects $\mathcal{C}_{\bm{x}}$,
each associated with a vector of numeric features $\bm{z}_{c}$. These features
include the bounding box, an estimation of the object position, and a vector
of visual features extracted from the cropping of the image with the bounding
box. Figure 2 gives an example of extraction of a number of objects, including
a box, from the egocentric view of an agent robot, together with the bounding-
box image of such a box and the estimate of its position.

  * –

Next, at line 17, the agent merges the objects $\mathcal{C}_{\bm{x}}$
recognized in the current perception with the ones already known, i.e.,
$\mathcal{C}$. For every object ${c^{\prime}}\in\mathcal{C}_{\bm{x}}$ there
are two possible situations: $(i)$ ${c^{\prime}}$ does not match with any
object $c\in\mathcal{C}$, and therefore it is added to $\mathcal{C}$ with the
corresponding features $\bm{z}_{{c^{\prime}}}$; $(ii)$ ${c^{\prime}}$ matches
with a $c\in\mathcal{C}$; in this case the features $\bm{z}_{c}$ of $c$ are
extended/updated with the features $\bm{z}_{c^{\prime}}$. In the
implementation, we use a very simple matching criteria which considers only
the estimated position of the objects. Two objects are matched when their
distance is less then a given threshold (set to 20cm). More sophisticated
criteria can be adopted by defining a suitable distance measure between the
entire set of object features. However, this simple criteria turned out to be
sufficiently effective in our experiments.

  * –

In line 18, the agent predicts the truth values of each atom in
$\mathcal{P}(\mathcal{C})$ for the updated set of constants $\mathcal{C}$ by
applying the predictors $\mathbf{Pr}$ on the features $\bm{z}_{\mathcal{C}}$.
For predicate $\mathsf{closeToAgent}$, the prediction takes also as input the
agent position in $\bm{z}_{s}$. All the atoms involving new or merged objects
must be evaluated; the remaining atoms are evaluated only if the corresponding
predictor takes as input some feature that has been updated after the
execution of the last action. For instance, if the agent executes a move
action, then all the atoms $\mathsf{closeToAgent}(c)$ for all
$c\in\mathcal{C}$ must be evaluated. Each atom $\mathsf{closeToAgent}(c)$ is
predicted true if the euclidean distance between the position of the object
represented by $c$ in $z_{c}$ and the agent position in $z_{s}$ is lower than
a given threshold (set to $140$ cm). When the action $\mathsf{open(box_{0})}$
is executed, the visual features of $\mathsf{box_{0}}$ probably change, and
the truth value of predicate $\mathsf{isOpen}(\mathsf{box_{0}})$ is predicted
as depicted in Figure 3. Notice that, after executing an open operation it is
not guaranteed that the object will be open, as the action might fail.

$\mathsf{box_{0}}$ image$\mathsf{open}$ predictor“yes”$\mathsf{isOpen(box0)}$
? Figure 3: Example of predictor for the $\mathsf{open}$ predicate. The
predictor takes as input the RGB images associated to $\mathsf{box_{0}}$, and
returns the predicted truth value of $\mathsf{isOpen}(\mathsf{box_{0}})$.

  * –

At line 19, the new state $s$ is created with all predicates $P(\bm{c})$ such
that $\textit{Pr}(Y_{P(\bm{c})}=\textit{True}\mid\bm{z}_{\bm{c}},\bm{z}_{s})$
is higher than a given threshold $1-\epsilon$ with $\epsilon\in[0,1]$. Our
approach does not assume to have access to the correct abstract state. Indeed,
the agent can produce inconsistent states (e.g., a box is both on the table
and on another box), or states that do not comply with action effects.
Inconsistent states do not prevent Ogamus to further plan and, whether a
failure occurs, revise the agent’s knowledge making them consistent. In the
second case, the agent monitors the execution of high-level actions by
comparing the state predicted by the PDDL model with the perceived state, and
it solves possible inconsistencies in favour of the action effects in the
model.

  * –

At line 21, when succeed$(op(\bm{c}))$ is true, i.e., the entire sequence
$\bm{e}$ of operations compiling the first action $op(\bm{c})$ of $\pi$ is
successfully executed, the state $s$ is updated according to the effects of
$op(\bm{c})$.

  * –

If the agent does not reach a state $s$ that satisfies the goal $\mathcal{G}$
after MaxIter steps, then the algorithm returns Fail (line 24).

###  3.3 Knowledge revision for new tasks

To show the generality and the modularity of the proposed framework, we
describe how it can be easily extended to cope with new tasks that can
possibly involve a set of new (PDDL) actions, predicates, and object types. To
allow the agent to accomplish a new task $t_{new}$, we firstly need to encode
$t_{new}$ in a PDDL goal formula. If the encoding of $t_{new}$ does not
require the introduction of new predicates, actions, or object types, then to
solve the task it is sufficient to invoke Ogamus with the goal formula
encoding $t_{new}$. If, instead, the encoding of $t_{new}$ requires some new
predicates, actions, or object types, then we have to provide the agent with
the capability of (1) recognizing objects of the new types, (2) predicting the
truth value of the new predicates, and (3) compiling the new actions in low-
level operations executable by the agent’s actuators. Consider, for instance,
the case where $t_{new}$ is the task turning a lamp on: $t_{new}$ can be
specified by the goal formula $\exists
x.\mathsf{lamp}(x)\wedge\mathsf{turned\\_on}(x)$, where $\mathsf{lamp}$ is a
new object type and $\mathsf{turned\\_on}$ a new unary predicate.

To detect objects with the newly introduced type e.g., $\mathsf{lamp}$, the
object detector need to be extended and retrained with the new object type.
This implies that the upper part of the detector (which is responsible of
classifying the objects in their types) need to be extended with the new type
and re-trained on a dataset containing also examples for this new type. The
introduction of a new predicate, in our example $\mathsf{turned\\_on}$,
requires the deployment of a new classifier that predicts if the predicate
holds for the objects detected from the sensory data. In case the predictor is
based on a supervised learning model, then a training dataset with object
labelled with positive and negative examples of the predicate need to be
provided. Finally, if the new task requires adding new actions to the PDDL
model, (e.g., to make the predicate $\mathsf{turned\\_on}(x)$ true/false we
need to introduce two new action $\mathsf{turn\\_on}$ and
$\mathsf{turn\\_off}$) we need to specify how the new action can be compiled
into a sequence of low-level operations executable by the agent. For example,
action $\mathsf{turn\\_on}(c)$ will be compiled into a low-level operation
$\mathsf{turn\\_on}(x,y,z)$ where $(x,y,z)$ is the believed position of object
$c$.

##  4 Experiments

We perform two sets of experiments. First, we experimentally evaluate Ogamus
with a simulated environment on four tasks that involves going close and move
objects present in a number of rooms. Then, we compare Ogamus with a state-of-
the-art approach on the specific task of object goal navigation in different
apartments.

###  4.1 Evaluating Ogamus

The tasks and the corresponding goals on which we evaluate Ogamus are the
following:

  1. 1.

Object goal navigation (objNav $t_{1}$): given an object type $t_{1}$, the
agent has to find, go close to, and look at an object of type $t_{1}$. For
instance, the agent has to go close to an apple and look at it. The
corresponding goal is $\exists
x(\mathsf{Apple}(x)\wedge\mathsf{CloseToAgent}(x)\wedge\mathsf{Visible}(x))$.
The agent is close to an object when the distance from the object is less than
1.5 meter.

  2. 2.

Open/close an object (Open/Close $t_{1}$): the agent is required to go close
to an object of type $t_{1}$, look at it, and open/close it. For instance the
agent has to open a drawer; the corresponding goal is $\exists
x(\mathsf{Drawer}(x)\wedge\mathsf{Open}(x))$. In order to manipulate an object
the agent need to be at a distance less than 1.5 meter.

  3. 3.

Stack an object of type $t_{1}$ on an object of type $t_{2}$ (On $t_{1}$
$t_{2}$): the agent has to find two objects of types $t_{1}$ and $t_{2}$ and
put the one of type $t_{1}$ on top of the other of type $t_{2}$. For instance
the agent has to put an apple on a table. The corresponding goal is: $\exists
xy(\mathsf{Apple}(x)\wedge\mathsf{Table}(y)\wedge\mathsf{On}(x,y))$.

Since these tasks do not involve numerical resources or temporal constraints,
we adopted propositional PDDL planning.

#### Simulator.

We used the iThor (?) simulator, an open source interactive environment for
Embodied AI. iThor provides 120 different scenes, such as kitchens, living-
rooms, bathrooms, and bedrooms, and allows a realistic simulation of the
environment, including the physics of the objects. The scenes contain objects
of 118 different types. The agent perceives the current state of the
environment through an RGB-D on-board camera that provides a photo-realistic
rendering of the egocentric view of the agent. The agent also perceives its
relative position and orientation via a GPS and a compass (relative to the
starting pose, which is unknown). The agent can navigate the environment by
moving ahead of a given distance (set to 25cm), turn left or right, and look
up or down of a given angle (set to $30^{\circ}$).222These settings are those
indicated by the simulator developers for their proposed challenges. The agent
can pick up objects, move them around, and change their state (e.g., a fridge
can be opened or a laptop switched on).

For the object goal navigation task, we also considered a second simulator,
RoboThor (?). RoboThor is another simulation environment designed to develop
embodied AI agents. Recently, RoboThor hosted a competition that tackles an
object goal navigation challenge; in our experiments, we also compared Ogamus
with the approaches that took part in the competition.

#### Object detector.

As an object detector we used the Faster-RCNN model available in PyTorch 1.9
(?), pre-trained on the COCO dataset (?) and fine-tuned on a self-generated
dataset. In addition to the bounding box of the detected object, the object
detector returns also the classification in one of the 118 classes. The object
detector has been trained on a dataset composed by 69,095 training and
validation images. The labeling of the dataset has been done by using the
ground-truth provided by iThor. We tested it on 12,892 images obtaining a
precision and recall of 50.99% and 65.18%, respectively.

#### Predicate predictors.

For predicting predicate On, we trained a feed-forward neural network (?) with
244 input features composed by the bounding boxes coordinates of the two
objects involved in the predicate relation and the 1-hot encoding of the two
predicted classes returned by the object detector. For such a predicate, the
training (and validation) sets is composed of 36,344 labelled pairs of
objects. We evaluate the prediction of predicate On on a test set composed of
8678 object pairs, obtaining 98.32% of both precision and recall. For
predicting the unary predicate Open, we used a ResNet50 neural network (?) to
extract features from the cropped object image, followed by a linear layer
with input size 2048.333Further technical details about the hyper-parameters
and datasets are available in the supplementary material. We trained it on
48,476 labelled examples, and test it on 9685 examples, obtaining 92.84%
precision and 92.54% recall. The unary predicate closeToAgent, meaning that
the agent is near to the object mentioned by the predicate, is computed
directly from the features of the object. Specifically, we check if the
distance between the agent position memorized in $\bm{z}_{s}$ and the object
position memorized in the object feature vector is less than the manipulation
distance, which is set to $1.5$ meter in iThor and $1$ meter in RoboThor.
Finally, we have to predict the equality predicate, i.e., when two objects $c$
and $d$ with features $\bm{z}_{c}$ and $\bm{z}_{d}$ represent the same object.
To this purpose, we compute the distance between the two estimated object
positions, and assign the object features to the same object instance whether
such a distance is lower than a given threshold (set to 20 cm in our
experiments). All the training, validation, and testing data have been
extracted from a set of images collected by navigating in the iThor simulator.

#### Evaluation metrics.

The evaluation is provided by calculating a number of standard metrics over a
set of episodes. For each task, an episode is obtained by randomly placing the
agent in a random unseen scene and providing it a randomly generated goal for
the given task. The generated goals are feasible since the object types used
in their definitions are randomly chosen from a proper set of types; e.g., the
goal to open a box is defined by randomly choosing the type “box” from a the
set of object types that can be opened. For all the tasks we adopt the
following standard evaluation metrics:

Success rate (Success):

is equal to the fraction of successful episodes on the total number of
episodes.

Distance To Success (DTS):

For tasks (ObjNav $t_{1}$), (Open $t_{1}$), and (Close $t_{1}$), it is the
average distance between the agent and the closest object of type $t_{1}$; for
the task (On $t_{1}\,t_{2}$), it is the average distance between the closest
pair of objects of types $t_{1}$ and $t_{2}$. If the episode succeeds such a
distance is set to 0.

In order to measure the impact of errors in object detecting, for each task we
consider two versions of Ogamus. In a version the set of objects $\mathcal{C}$
are those returned by our object detector; in the second version the set of
objects $\mathcal{C}_{GT}$ are those returned by the iThor simulator, which
corresponds to a ground-truth object detector. Moreover, for all tasks, we
evaluate the precision $P_{\mathcal{C}}$ and recall $R_{\mathcal{C}}$ of the
detected objects, and the precision $P_{\cal P}$ and recall $R_{\cal P}$ of
their predicate relations. $P_{\cal P}$ and $R_{\cal P}$ take into account
only the objects that match with ground-truth ones. The matching is performed
by computing the Intersection over Union (IoU) among the 2D bounding box
detected during the episode and the ground-truth ones: if the IoU is higher
than 50% for a ground-truth object of the same class, then the detected object
matches with it.

#### Experimental results.

|  _Success_ $\,\uparrow$ |  _DTS_ $\,\downarrow$ | $P_{\mathcal{C}}\uparrow$ | $R_{\mathcal{C}}\uparrow$ | $P_{\cal P}\uparrow$ | $R_{\cal P}\uparrow$
---|---|---|---|---|---|---
| $\mathcal{C}$ | $\mathcal{C}_{GT}$ | $\mathcal{C}$ | $\mathcal{C}_{GT}$ | $\mathcal{C}$ | $\mathcal{C}_{GT}$ | $\mathcal{C}$ | $\mathcal{C}_{GT}$ | $\mathcal{C}$ | $\mathcal{C}_{GT}$ | $\mathcal{C}$ | $\mathcal{C}_{GT}$
On | 0.5 | 0.8 | 1 | 0.37 | 0.28 | 1 | 0.86 | 1 | 0.83 | 0.82 | 0.8 | 0.87
Open | 0.75 | 0.87 | 0.45 | 0.25 | 0.35 | 1 | 0.78 | 1 | 0.82 | 0.81 | 0.72 | 0.82
Close | 0.78 | 0.89 | 0.39 | 0.16 | 0.32 | 1 | 0.8 | 1 | 0.8 | 0.79 | 0.73 | 0.82
objNav | 0.78 | 0.83 | 0.27 | 0.19 | 0.42 | 1 | 0.8 | 1 | 0.82 | 0.8 | 0.75 | 0.84

Table 1:  Performance of Ogamus with/out the ground-truth object detection,
evaluated on the considered tasks in the iThor simulator.
$\uparrow$/$\downarrow$ means the higher/lower the better.

In our experiments, a run of Ogamus consists of 200 steps, where at each step
a low-level operation is performed; we call each of these runs an episode. For
all tasks, the episode dataset uses the test scenes of iThor, i.e., all
environments that does not appear in the datasets generated for training the
predicate classifiers and object detector.

In Table 1, we report the average results of all tasks with and without
ground-truth object detection over the considered episodes. For task on, we
randomly generated 400 different goals, defining 400 episodes; for tasks open
and close, we randomly generated 100 goals, defining 100 episodes for each
task; for the object goal navigation task, we used the test set of goals
proposed in (?), defining 2133 episodes. It is worth noting that, for the
object goal navigation task, two different episodes often have the same goal
but a different initial pose of the agent.

The impact of errors in object detecting for tasks objNav, open and close is
pretty low and, as expected, it is the half of the impact for task on, since
this latter task requires to detect two objects, while all other tasks
requires to detect a single object. Without ground-truth object detection,
Ogamus achieves the best success rate on the object goal navigation task; same
or similar results are also provided in tasks open and close, since they can
be seen as an extension of the object goal navigation task where, after
finding and going near to an object, the agent has only to open or close the
object. In the on task, the success rate decreases significantly, because it
requires moving towards two objects, instead of only one, and has two
additional complexities given by the facts that one object must be placed on
the other one in a clear place, i.e., a place not obstructed by other objects,
and that the total encumbrance of the agent increases when it carries an
object, which causes more collisions during the navigation.

Metric $P_{\mathcal{C}}$ measures the amount of false positives in detecting
objects. Although the values of $P_{\mathcal{C}}$ is quite low for almost all
the considered tasks, the success rate is relatively high because (i) many
false positive objects are not involved in the definition of goals, and (ii)
the agent acts by using the objects with the highest confidence, which usually
correspond to ground-truth objects. $P_{\mathcal{C}}$ is higher for the object
goal navigation task, because for this task the agent achieves the goal in
fewer steps than for other tasks, and this reduces the number of predictions
and the chance of detecting false positive objects.

Metric $R_{\mathcal{C}}$ measures the amount of true positive detected
objects. The values for $R_{\mathcal{C}}$ are quite high, and hence the real
existing objects are often detected, although in our experiments the agent
sometimes fails to recognize objects when they are far from the agent.
Moreover, the values of $P_{\cal P}$ and $R_{\cal P}$ are relatively high, and
hence the agent can construct a symbolic state that is quite correct and
complete, enabling an effective planning.

As expected, when Ogamus is provided with ground-truth object detection, all
metrics are better than or similar to using our object detection. Only
$P_{\cal P}$ is slightly lower when ground-truth object detection is used; we
think this is due to the fact that sometimes the ground-truth object detection
identifies objects which are only partially seen by the agent camera and
predicting their properties more likely fails (e.g., the agent fails in
predicting whether a fridge is open when it sees only a corner of the fridge).

###  4.2 Comparison on object goal navigation

|  _Success_ $\,\uparrow$ |  _SPL_ $\,\uparrow$
---|---|---
Random | 1.72% | 1.33%
DD-PPO | 35.11% | 17.37%
DD-PPOaction_boost | 36.61% | 17.49%
Ogamus | 56.78% | 24.87%

Table 2: Performance of Ogamus w.r.t. the random baseline, DD-PPO, and DD-
PPOaction_boost, evaluated on the object goal navigation task in the RoboThor
simulator.

We did not find other approaches using simulator iThor that solve the tasks
considered in our experiments. So, in our experimental analysis we considered
a second simulator, RoboThor (?), for which the last challenge concerning the
object goal navigation was launched in 2021.

For the object goal navigation task, we compared Ogamus with a random
baseline, an RL baseline provided in the challenge, called DD-PPO, and the
winner of the challenge, called DD-PPOaction_boost. Both the RL baseline and
the winner exploit the DD-PPO algorithm (?) where the hidden state is computed
by providing, as input to a GRU (?), the visual features of the RGB-D images
computed by a ResNet-18 (?). The baseline and the winner approach have been
trained on 108,000 episodes for 300 and about 10 million steps, respectively.

For this experiment, we adopt an additional metric, called Success weighted by
Path Length (SPL) and introduced by ? (?). This metric measures the efficiency
of the agent in reaching the goals and is defined as:

$SPL=\frac{1}{N}\cdot\sum_{i=1}^{N}\left(s_{i}\cdot\frac{p_{i}^{\star}}{max(p_{i},p_{i}^{\star})}\right)$

where $N$ is the number of episodes, $p_{i}^{\star}$ is the shortest-path
distance from the initial position of the agent to the closest goal in the
$i$-th episode, $p_{i}$ is the length of the agent path in the $i$-th episode,
and $s_{i}$ is a boolean variable equal to 1 when the $i$-th episode succeeds,
and equal to 0 otherwise. If the path of the agent is the shortest one, the
term in parenthesis is 1. The longer the path, the lower the term in
parenthesis and the worse the metric.

For the experiment, we considered the validation episode dataset provided in
the challenge, which is composed by 1800 episodes set in the 15 validation
scenes of RoboThor. We did not consider the test episode dataset of the
challenge, because for such a dataset the evaluation can be done only by the
organizers of the challenge who require that the evaluated approach plays by
the challenge rule. This is not the case for Ogamus because it allows the
agent to perceive its pose, which is not available in the challenge. While the
usage of this additional information can in principle favors Ogamus w.r.t. the
approaches that took part in the challenge, it is worth noting that the agent
position can be approximately derived from the RGB-D egocentric views by means
of visual simultaneous localization and mapping methods (?). Most importantly,
the usage of the validation dataset of RoboThor disfavors Ogamus w.r.t. the
other compared approaches because the object detector and predicate
classifiers of Ogamus are trained using the training and validation scenes of
a different simulation environment, iThor, while the other compared approaches
are trained and validated on the training and validation scenes of RoboThor.

Each episode of the dataset consists of 500 steps, and regards finding and
moving toward objects of 12 types. We trained an object detector similarly to
the one for iThor simulator, but focused on the 12 goal object types of
RoboThor, which provides a performance slightly higher than the object
detector trained using all the 118 object types of iThor, obtaining 59.02%
precision and 69.06% recall.

Table 2 shows the results of the comparison. The random baseline provides poor
performances. This indicates that, for the RoboThor simulator, the object goal
navigation task is quite challenging. The complexity of the task is confirmed
by the performance of the RL baseline which is higher than the random baseline
but still quite low. DD-PPOaction_boost provides results slightly higher than
the RL baseline. Remarkably, Ogamus outperforms DD-PPOaction_boost in terms of
success rate and _SPL_. This confirms that the integration of symbolic
planning with state recognition from sensory data can provide competitive
results w.r.t. RL based approaches.

###  4.3 Error analysis

Figure 4: Ratio of the occurrences of different error types made by Ogamus.

In Figure 4, we analyze the errors made by Ogamus on all tasks. For few
episodes, denoted as “Not inspected”, the agent detects a far object of the
same type as the type used for the goal definition, and subsequently
approaches the object but is no more able to recognize it. This is due to the
fact that either the object does not really exists, or the agent does not
recognize an existing object, despite being close to and looking at it. For
some episodes, namely “Not reachable”, the agent finds a goal object but
cannot reach a position close enough to the object. This can be due to the
fact that either the agent collides or the goal object estimated position is
farther than the real one. Collisions more often happen for the task on, when
the agent holds an object as the agent encumbrance increases. An error in the
estimation of the object position is more likely for large objects, such as
tables or televisions, since the agent considers the center of the object as
its position. There are few episodes, labelled as “Not found”, where the agent
does not find the object, due to either an ineffective exploration of the
environment or false negatives of the object detector. We observed that the
latter case is more likely than the former, because the agent almost always
explores the entire environment within the given number of steps. The errors
labelled by “Confused” denote episodes for which the agent believes it
succeeded while the task has not been completed. This is due to false
positives of the object detector. Finally, “Others” denote all other task-
dependent failures. E.g., for the on, open and close tasks, the agent
sometimes fails to identify the object position when it has to manipulate an
object. This more likely happens for small objects, such as spoons or
saltshakers. Moreover, for the on predicate an agent can fail to put an object
on a table due to the fact that the target position is already occupied, or
there is not enough space on the table.

Figure 5: Average performance of Ogamus for the goal object navigation task in
the RoboThorsimulator, using a number of steps ranging from 0 to 500.

Figure 5 shows the success rate and _SPL_ for a number of steps ranging from 0
to 500. For almost all episodes the agent achieves the goal in 300 steps. For
few episodes, the agent achieves the goal only after 500 steps. This happens
because the agent is actually close to and looks at a goal object, but it
fails to recognize the object.

##  5 Conclusions and Future Work

We have proposed a framework, called Ogamus, for the online grounding of
planning domains in unknown environments. Our approach enables an agent to map
the sensory data into a symbolic state, allowing to perform and exploit
efficient planning in a wide variety of different environments. We have tested
the proposed method on different tasks obtaining better results than recent
RL-based approaches. Future work will focus on learning a policy to compile
the high-level actions into low-level executable operations, and on learning,
online, the mapping of the sensory representations to symbolic ones.

## References

  * Anderson et al. 2018 Anderson, P.; Chang, A.; Chaplot, D. S.; Dosovitskiy, A.; Gupta, S.; Koltun, V.; Kosecka, J.; Malik, J.; Mottaghi, R.; Savva, M.; et al.  2018\.  On evaluation of embodied navigation agents.  arXiv preprint arXiv:1807.06757.
  * Campari et al. 2020 Campari, T.; Eccher, P.; Serafini, L.; and Ballan, L.  2020\.  Exploiting scene-specific features for object goal navigation.  In European Conference on Computer Vision, 406–421.  Springer.
  * Chaplot et al. 2019 Chaplot, D. S.; Gandhi, D.; Gupta, S.; Gupta, A.; and Salakhutdinov, R.  2019\.  Learning to explore using active neural slam.  In International Conference on Learning Representations.
  * Chaplot et al. 2020 Chaplot, D. S.; Gandhi, D. P.; Gupta, A.; and Salakhutdinov, R. R.  2020\.  Object goal navigation using goal-oriented semantic exploration.  In Larochelle, H.; Ranzato, M.; Hadsell, R.; Balcan, M. F.; and Lin, H., eds., Advances in Neural Information Processing Systems, volume 33, 4247–4258.  Curran Associates, Inc.
  * Cho et al. 2014 Cho, K.; van Merriënboer, B.; Bahdanau, D.; and Bengio, Y.  2014\.  On the properties of neural machine translation: Encoder–decoder approaches.  In Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation, 103–111.
  * Coradeschi and Saffiotti 2003 Coradeschi, S., and Saffiotti, A.  2003\.  An introduction to the anchoring problem.  Robotics and autonomous systems 43(2-3):85–96.
  * Deitke et al. 2020 Deitke, M.; Han, W.; Herrasti, A.; Kembhavi, A.; Kolve, E.; Mottaghi, R.; Salvador, J.; Schwenk, D.; VanderBilt, E.; Wallingford, M.; Weihs, L.; Yatskar, M.; and Farhadi, A.  2020\.  RoboTHOR: An Open Simulation-to-Real Embodied AI Platform.  In CVPR.
  * Fang et al. 2019 Fang, K.; Toshev, A.; Fei-Fei, L.; and Savarese, S.  2019\.  Scene memory transformer for embodied agents in long-horizon tasks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 538–547.
  * Garnelo, Arulkumaran, and Shanahan 2016 Garnelo, M.; Arulkumaran, K.; and Shanahan, M.  2016\.  Towards deep symbolic reinforcement learning.  arXiv preprint arXiv:1609.05518.
  * He et al. 2016 He, K.; Zhang, X.; Ren, S.; and Sun, J.  2016\.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, 770–778.
  * Kase et al. 2020 Kase, K.; Paxton, C.; Mazhar, H.; Ogata, T.; and Fox, D.  2020\.  Transferable task execution from pixels through deep planning domain learning.  In 2020 IEEE International Conference on Robotics and Automation (ICRA), 10459–10465.  IEEE.
  * Kolve et al. 2017 Kolve, E.; Mottaghi, R.; Han, W.; VanderBilt, E.; Weihs, L.; Herrasti, A.; Gordon, D.; Zhu, Y.; Gupta, A.; and Farhadi, A.  2017\.  AI2-THOR: An Interactive 3D Environment for Visual AI.  arXiv.
  * Lin et al. 2014 Lin, T.-Y.; Maire, M.; Belongie, S.; Hays, J.; Perona, P.; Ramanan, D.; Dollár, P.; and Zitnick, C. L.  2014\.  Microsoft coco: Common objects in context.  In European conference on computer vision, 740–755.  Springer.
  * Lyu et al. 2019 Lyu, D.; Yang, F.; Liu, B.; and Gustafson, S.  2019\.  Sdrl: interpretable and data-efficient deep reinforcement learning leveraging symbolic planning.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, 2970–2977.
  * Ma et al. 2021 Ma, Z.; Zhuang, Y.; Weng, P.; Zhuo, H. H.; Li, D.; Liu, W.; and Hao, J.  2021\.  Learning symbolic rules for interpretable deep reinforcement learning.  arXiv preprint arXiv:2103.08228.
  * Mirowski et al. 2017 Mirowski, P.; Pascanu, R.; Viola, F.; Soyer, H.; Ballard, A. J.; Banino, A.; Denil, M.; Goroshin, R.; Sifre, L.; Kavukcuoglu, K.; et al.  2017\.  Learning to navigate in complex environments.  ICLR.
  * Mousavian et al. 2019 Mousavian, A.; Toshev, A.; Fišer, M.; Košecká, J.; Wahid, A.; and Davidson, J.  2019\.  Visual representations for semantic target driven navigation.  In 2019 International Conference on Robotics and Automation (ICRA), 8846–8852.  IEEE.
  * Paszke et al. 2019 Paszke, A.; Gross, S.; Massa, F.; Lerer, A.; Bradbury, J.; Chanan, G.; Killeen, T.; Lin, Z.; Gimelshein, N.; Antiga, L.; et al.  2019\.  Pytorch: An imperative style, high-performance deep learning library.  Advances in neural information processing systems 32:8026–8037.
  * Persson et al. 2019 Persson, A.; Dos Martires, P. Z.; De Raedt, L.; and Loutfi, A.  2019\.  Semantic relational object tracking.  IEEE Transactions on Cognitive and Developmental Systems 12(1):84–97.
  * Savva et al. 2017 Savva, M.; Chang, A. X.; Dosovitskiy, A.; Funkhouser, T.; and Koltun, V.  2017\.  Minos: Multimodal indoor simulator for navigation in complex environments.  arXiv preprint arXiv:1712.03931.
  * Svozil, Kvasnicka, and Pospichal 1997 Svozil, D.; Kvasnicka, V.; and Pospichal, J.  1997\.  Introduction to multi-layer feed-forward neural networks.  Chemometrics and intelligent laboratory systems 39(1):43–62.
  * Taketomi, Uchiyama, and Ikeda 2017 Taketomi, T.; Uchiyama, H.; and Ikeda, S.  2017\.  Visual slam algorithms: a survey from 2010 to 2016.  IPSJ Transactions on Computer Vision and Applications 9(1):1–11.
  * Wijmans et al. 2019 Wijmans, E.; Kadian, A.; Morcos, A.; Lee, S.; Essa, I.; Parikh, D.; Savva, M.; and Batra, D.  2019\.  Dd-ppo: Learning near-perfect pointgoal navigators from 2.5 billion frames.  In International Conference on Learning Representations.
  * Wortsman et al. 2019 Wortsman, M.; Ehsani, K.; Rastegari, M.; Farhadi, A.; and Mottaghi, R.  2019\.  Learning to learn how to learn: Self-adaptive visual navigation using meta-learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 6750–6759.
  * Ye et al. 2021 Ye, J.; Batra, D.; Das, A.; and Wijmans, E.  2021\.  Auxiliary tasks and exploration enable objectnav.  arXiv preprint arXiv:2104.04112.


Copyright for this paper by its authors. Use permitted under Creative Commons
License Attribution 4.0 International (CC BY 4.0). XAI 2024 - The 2nd World
Conference on eXplainable Artificial Intelligence

[orcid=0000-0001-7239-4270<EMAIL_ADDRESS>] [1]
[1]Corresponding author.

[orcid=0000-0002-3678-0390<EMAIL_ADDRESS>]

[orcid=0000-0002-0558-3810<EMAIL_ADDRESS>]

[orcid=0000-0001-6955-9572<EMAIL_ADDRESS>]

[orcid=0000-0002-6776-3868<EMAIL_ADDRESS>]

# Patch-based Intuitive Multimodal Prototypes Network (PIMPNet) for
Alzheimer’s Disease classification

Lisa Anita De Santi  Department of Information Engineering, University of
Pisa, Pisa, Italy  Fondazione Toscana G Monasterio - Bioengineering Unit,
Pisa, Italy  Jörg Schlötterer  University of Marburg, Marburg, Germany
University of Mannheim, Mannheim, Germany  Meike Nauta  Datacation, Eindhoven,
Netherlands  Vincenzo Positano  Christin Seifert

(2024)

###### Abstract

Volumetric neuroimaging examinations like structural Magnetic Resonance
Imaging (sMRI) are routinely applied to support the clinical diagnosis of
dementia like Alzheimer’s Disease (AD). Neuroradiologists examine 3D sMRI to
detect and monitor abnormalities in brain morphology due to AD, like global
and/or local brain atrophy and shape alteration of characteristic structures.
There is a strong research interest in developing diagnostic systems based on
Deep Learning (DL) models to analyse sMRI for AD. However, anatomical
information extracted from an sMRI examination needs to be interpreted
together with patient’s age to distinguish AD patterns from the regular
alteration due to a normal ageing process. In this context, part-prototype
neural networks integrate the computational advantages of DL in an
interpretable-by-design architecture and showed promising results in medical
imaging applications. We present PIMPNet, the first interpretable multimodal
model for 3D images and demographics applied to the binary classification of
AD from 3D sMRI and patient’s age. Despite age prototypes do not improve
predictive performance compared to the single modality model, this lays the
foundation for future work in the direction of the model’s design and
multimodal prototype training process.

###### keywords:

Interpretability-by-design Prototype Prototype-network Multimodal Deep
Learning Alzheimer MRI Age

##  1 Introduction

There is a significant research interest in supporting Alzheimer’s Disease
(AD) diagnosis with Deep Learning (DL) models [1]. Existing diagnostic
guidelines often integrate the clinical evaluation of the patient with
structural Magnetic Resonance Imaging (sMRI), to detect pathological brain
patterns like gray matter atrophy.

Brain alterations in sMRI might support the early and differential diagnosis
and the prediction of disease’s progression. There are sets of common
practices used for analysing sMRI acquisition, but there are still no
universally accepted methods [2, 3, 4]. In addition, information collected
from sMRI should be interpreted together with the patient’s age, as there are
anatomical brain changes due to the physiological ageing process [5, 6].

DL architectures can facilitate the analysis of neuroimaging data, and might
be able to identify unconventional AD subtypes and extract yet unknown image-
based biomarkers [7, 8]. Prototypical-Part (PP) networks combine the
advantages of DL models in an interpretable-by-design architecture, and are
collecting interesting results in medical imaging applications where the
black-box nature of standard DL models poses controversy [9].

There are currently different variants of PP networks, including PIPNet [10],
originally applied to 2D images and then extended to handle 3D scans [11].
PIPNet showed appealing properties in the medical imaging domain [12],
including a reduced number of part-prototypes, semantic significance of
learned prototypes, and ability to cope with Out-of-Distribution data (which
might be particularly useful in dementia diagnosis, where unusual
neurodegeneration pattern are reported [4]). However, sMRI data should be
interpreted together with patients’ demographics to discern age-related image
alteration from pathological alteration, and existing PP models cannot be
directly applied to perform this task. Adding non-image prototypes to the
standard PP architecture is a non-trivial task, and there are no unique
strategies available. There are some works which integrate the concept of
learning prototypes from multiple modalities which are based on the
concatenation (deterministic prototypes) or on the multimodal feature
extraction (shifted prototypes). However, available models cannot be applied
to our task, as specifically designed for images and textual data [13].

We present Patch-based Intuitive Multimodal Prototypes Network (PIMPNet), the
first multimodal prototype classifier which learns 3D image part-prototypes
and prototypical values from structured data, to predict patient’s cognitive
level in AD from sMRI and age values.

##  2 Method

This section introduces the architecture (cf. Sect. 2.1 and Fig. 1) and the
training process (Sect. 2.2), of PIMPNet.

Figure 1: PIMPNet architecture

###  2.1 Proposed Model: PIMPNet

We propose an age-prototypes layer integrated into the original PIPNet 3D
model [11] to create our multimodal architecture. In contrast to “ordinary”
age-binning for the inclusion of age information, the age-prototypes layer has
the advantages of: (i) being able to learn important age values for the
diagnostic task performed (which might not be equally distributed, and might
not be easily identifiable in priors); (ii) not to assign different age bins
to two patients of similar age close to the bin boundary.

Our PIMPNet has an input layer which takes the 3D image
$\mathbf{x}_{img}\in\mathbb{R}^{ch\times S\times R\times C}$ and the age
$\mathbf{x}_{age}\in\mathbb{R}^{1}$ as input, where $ch,S,R,C$ respectively
represents the number of channels, slices, rows and columns of the input image
volume. Image $\mathbf{x}_{img}$ and age $\mathbf{x}_{age}$ are processed in
parallel. A CNN backbone processes $\mathbf{x}_{img}$,
$\mathbf{z}=f(\mathbf{x}_{img};\mathbf{w}_{f})$ extracting $M$ 3-dimensional
$(D\times H\times W)$ feature maps, where $\mathbf{z}_{m,d,h,w}$ represents
the activation of image-prototype $m$ at patch location $d,h,w$. Next, a 3D
max-pooling applied to every feature map extracts $M$ image prototype presence
scores $\mathbf{p}_{img}\in[0.,1.]^{M}$, where $\mathbf{p}_{img,m}$ measures
the presence of the image prototype $m$ in the input image. This defines the
image-prototypes layer.

In parallel, we have the age-prototypes layer, constituted by $N$ trainable
1-dimensional tensors $\mathbf{t}_{age,n}\in\mathbb{R}^{1\times N}$, which
aims to learn prototypical age values for the classification task. This layer
computes age prototype presence scores $\mathbf{p}_{age}\in[0.,1.]^{N}$, a
similarity measurement between the input age to every age prototype, defining
a smooth age binning 111The similarity function employed is inspired by the
magnitude of a Butterworth filter [14]. In preliminary experiments, we used an
exponential similarity function as in ProtoTree:
$\mathbf{p}_{age,n}=exp(-||\mathbf{x}_{age}-\mathbf{t}_{age,n}||)$, but as
$exp(-2)\approx 0.13$, a $2$ years age difference would already result in
little similarity, which is not in line with domain knowledge about age
relevance for Alzheimer’s disease.:

$\mathbf{p}_{age,n}=\frac{1}{\sqrt{1+\left(\frac{\mathbf{x}_{age}-\mathbf{t}_{age,n}}{\overline{t}}\right)^{2s}}},$
(1)

where $\mathbf{t}_{age}$ are trainable parameters and $\overline{t}$ and $s$
are hyper-parameters which regulate the band and the slope of the similarity
function. We then have a prototypes layer which concatenates the image and age
prototype presence scores obtaining a layer of $L=M+N$ prototypes
$\mathbf{p}\in[0.,1.]^{L}$
$\mathbf{p}=concat(\mathbf{p}_{img},\mathbf{p}_{age})$. The final
classification is performed by a sparse linear positive layer
$\mathbf{w_{c}}\in\mathbb{R}^{L\times K}_{\geq 0}$ which connects image and
age prototypes to the $K$ classes acting as a scoring sheet system. The $K$
class output scores are given by the sum of the prototypes’ presence score
weighted for the contribution of prototype $l$ to class $k$
$\mathbf{w_{c}}^{l,k}$, i.e., $\mathbf{o}=\mathbf{p}\mathbf{w_{c}}$, where
$\mathbf{o}$ is $1\times K$ and
$\mathbf{o}_{k}=\sum_{l=1}^{L}\mathbf{p}_{l}\mathbf{w_{c}}^{l,k}$. PIMPNet
returns the output class only using the most activated age-prototype, i.e.,
closest to the patient’s age according to the similarity metric222We selected
only the most activated age prototype during inference (not during the
optimization process)..

###  2.2 PIMPNet Training

We optimize PIMPNet’s parameters by integrating the training of age prototypes
into the original PIPNet training process [10]. This includes two main stages:
(1) Self-Supervised pre-training of Image-Prototypes, and (2) PIMPNet
training.

As the original PIPNet [10], the 1st stage generates positive pairs
$\mathbf{x}_{img}^{\prime},\mathbf{x}_{img}^{\prime\prime}$ by applying data
augmentation transformation to $\mathbf{x}_{img}$ selected so that humans
consider the two views similar. These are used to minimize the loss function
$\lambda_{A}\mathcal{L}_{A}+\lambda_{T}\mathcal{L}_{T}$ by updating
$\mathbf{w}_{f}$, where $\mathcal{L}_{A}=-\frac{1}{DHW}\sum\limits_{(d,h,w)\in
D\times H\times
W}log(\mathbf{z}^{\prime}_{:,d,h,w}\cdot\mathbf{z}^{\prime\prime}_{:,d,h,w})$
is an Alignment Loss which optimizes positive pairs to activate the same
prototype. Together with a softmax over $\mathbf{z}_{:,d,h,w}$, the alignment
results in near-binary encodings where an image patch corresponds to exactly
one prototype. $\mathcal{L}_{T}=-\frac{1}{M}\sum
log(tanh(\sum\mathbf{p}_{img,b})+\epsilon)$ is a Tanh-Loss used to prevent the
trivial solution that one prototype node is activated on all image patches in
each image in the dataset and instead activates multiple distinct prototypes
per batch $b$. Only during training, output scores are calculated as
$\mathbf{o}=\log((\mathbf{p}\mathbf{w}_{c})^{2}+1)$, acting as regularization
for sparsity.

The 2nd training stage includes the training of age prototypes, optimization
of classification performance and image-prototypes fine-tuning for the
downstream classification task. The optimization minimizes
$\lambda_{A}\mathcal{L}_{A}+\lambda_{T}\mathcal{L}_{T}+\lambda_{C}\mathcal{L}_{C}$
by updating $\mathbf{w}_{f},\mathbf{t}_{age},\mathbf{w}_{c}$, where
$\mathcal{L}_{C}$ is the Log-likelihood classification loss

##  3 Evaluation

We used the multimodal dataset from the Alzheimer’s Disease Neuroimaging
Initiative (ADNI) database333https://adni.loni.usc.edu. We selected “ADNI1
Standardized Screening Data Collection for 1.5T scans” processed with
Gradwarp, B1 non-uniformity, and N3 correction, obtaining 307 CN and 243 AD
sMRI brain scans and the corresponding patients’ age. We report statistics on
patients’ demographic of the selected ADNI cohort in Table 1.

Table 1: Patients’ demographics of the selected ADNI cohort, further divided according to the clinical labels. Class | N° subjects | Mean $\pm$ SD Age | Age Range
---|---|---|---
Both | 550 | 76 $\pm$ 6 | 55-91
CN | 307 | 76 $\pm$ 5 | 60-90
AD | 243 | 75 $\pm$ 8 | 55-91

We preprocessed sMRI data inspired by the pre-processing pipeline applied in
previous works [15]. We tranformed all the images to the common ICBM152 Non-
Linear Symmetric 2009c standard space [16] with affine registration. We
selected the grey matter structures by applying the ICBM152 Non-Linear
Symmetric 2009c brain mask and kept a margin of 3 from its first and last non-
empty slices. We applied an image downsampling of 2 and we scaled all image
intensities to the range [0-1] with a min-max normalization. We implemented
PIMPNet using PyTorch and MONAI444https://monai.io, training our models on an
Intel Core i7 5.1 MHz PC, 32 GB RAM, equipped with an NVIDIA RTX3090 GPU with
24 GB of embedded RAM. As CNN backbones we used ResNet-18 3D pretrained on
Kinetics400 [17] and ConvNeXt-Tiny 3D pretrained on the STOIC medical dataset
(Study of Thoracic CT in COVID-19) [18]. We finetuned PIMPNet with Adam
optimizer using the same hyperparameter settings of the original PIPNet [10].
We only reduced the batch size to $12$ to adapt it to our computational
capabilities and we set the learning rate of age prototypes to $0.1$555Using
the same learning rate of the original PIPNet used to train the image-
prototypes (0.05) results in irrelevant updates of the Age Prototypes. We
arbitrarily set the number of age prototypes $=5$ evenly spaced between $40$
and $90$ to cover the patients’ age range of our dataset. For the age
similarity function, we respectively set $\overline{t}=4$ and $s=8$666We leave
an extensive hyperparameter search for learning the age prototypes for future
work. We performed 5-fold cross-validation with patient-wise splits. $20\%$ of
training images are used for validation.

We evaluated the models in terms of classification performance and with
functionally grounded metrics of explainability. Results are reported in
Tables 2 and 3.

Table 2: Performances comparison between PIPNet trained on 3D sMRI and PIMPNet trained on 3D sMRI + Age averaged over 5 folds.  Model | Acc | Bal Acc | SENS | SPEC | F1
---|---|---|---|---|---
PIPNet |  |  |  |  |
ResNet-18 3D | 83 $\pm$ 04 | 83 $\pm$ 04 | 86 $\pm$ 06 | 79 $\pm$ 07 | 81 $\pm$ 05
ConvNeXt-Tiny 3D | 65 $\pm$ 12 | 66 $\pm$ 09 | 56 $\pm$ 32 | 76 $\pm$ 15 | 66 $\pm$ 05
PIMPNet |  |  |  |  |
ResNet-18 3D | 84 $\pm$ 04 | 83 $\pm$ 04 | 89 $\pm$ 03 | 77 $\pm$ 08 | 81 $\pm$ 05
ConvNeXt-Tiny 3D | 72 $\pm$ 04 | 70 $\pm$ 04 | 86 $\pm$ 10 | 55 $\pm$ 14 | 63 $\pm$ 09

We compared PIMPNet performance (sMRI + age) with PIPNet-3D (sMRI only) [11],
to evaluate if including age information improves diagnostic performance. We
measured performance using Accuracy (Acc), Balanced Accuracy (Bal Acc),
Sensitivity (SENS, Acc of Cognitive Normal class), Specificity (SPEC, Acc of
Alzheimer’s Disease class), and F1 score (F1). We measured the Global size
(GS) of the model as the total number of prototypes, and the Local size (LS)
of explanations as the number of detected prototypes in a single 3D sMRI,
averaged over all the images in the test set. Additionally, we report the
Sparsity (Sp) of the decision layer as the percentage of zero-weights in the
linear classification layer [10] to assess the compactness of the prototypes-
classes layer. We further assessed whether prototypes are consistently located
in the same brain region, and the purity of the prototypes in terms of the
anatomical regions included based on the CerabrA atlas annotation [19]. More
specifically, the Prototypes Localization Consistency (LCp) evaluates the
differences in the coordinate centre of the prototypical part in the input
image, while the Prototype Brain Entropy (Hp) as a measure of purity computes
the Shannon Entropy of the brain regions included in the prototypical part
[11].

We show the learned age prototypes $\mathbf{t}_{age}$ from five different
folds (denoted as Mx where x indicates the current fold) in Table 4.

Table 3: Functionally-grounded evaluation of PIPNet trained on 3D sMRI and PIMPNet trained on 3D sMRI + Age averaged over 5 folds. $\uparrow$ and $\downarrow$: tendency for better values. Model | GS $\downarrow$ | LS $\downarrow$ | Sp $\uparrow$ | LCp $\downarrow$ | Hp $\downarrow$
---|---|---|---|---|---
ResNet-18 3D |  |  |  |  |
PIPNet | 149 $\pm$ 18 | 73 $\pm$ 10 | 0.855 $\pm$ 0.018 | 0.008 $\pm$ 0.006 | 2.474 $\pm$ 0.249
PIMPNet | 143 $\pm$ 35 | 74 $\pm$ 20 | 0.861 $\pm$ 0.033 | 0.006 $\pm$ 0.006 | 2.424 $\pm$ 0.162
ConvNeXt-Tiny 3D |  |  |  |  |
PIPNet | 4 $\pm$ 2 | 2 $\pm$ 1 | 0.997 $\pm$ 0.001 | 0.000 $\pm$ 0.000 | 1.803 $\pm$ 0.999
PIMPNet | 10 $\pm$ 9 | 4 $\pm$ 4 | 0.993 $\pm$ 0.002 | 0.000 $\pm$ 0.000 | 1.543 $\pm$ 0.626
Table 4: Prototypical Age Values $\mathbf{t}_{Age,i}$ learned for folds M1, …, M5 trained with different backbones. Fold | $\mathbf{t}_{Age,1}$ | $\mathbf{t}_{Age,2}$ | $\mathbf{t}_{Age,3}$ | $\mathbf{t}_{Age,4}$ | $\mathbf{t}_{Age,5}$ | $\mathbf{t}_{Age,1}$ | $\mathbf{t}_{Age,2}$ | $\mathbf{t}_{Age,3}$ | $\mathbf{t}_{Age,4}$ | $\mathbf{t}_{Age,5}$
---|---|---|---|---|---|---|---|---|---|---
| ResNet-18 3D | ConvNeXt-Tiny 3D
M1 | 65.77 | 65.81 | 66.14 | 76.81 | 80.99 | 56.81 | 65.00 | 64.96 | 74.13 | 85.80
M2 | 68.46 | 69.40 | 70.38 | 77.04 | 82.38 | 55.75 | 58.39 | 64.96 | 74.32 | 85.59
M3 | 66.37 | 67.27 | 67.91 | 75.87 | 81.96 | 54.86 | 56.63 | 65.21 | 74.40 | 85.11
M4 | 66.72 | 66.72 | 67.07 | 77.07 | 79.75 | 58.22 | 58.59 | 66.50 | 75.88 | 89.09
M5 | 66.51 | 66.52 | 67.23 | 77.37 | 80.00 | 57.79 | 66.94 | 65.44 | 72.55 | 84.58

##  4 Discussion and Conclusion

Both PIPNet and PIMPNet with the ResNet-18 3D backbone achieve higher
classification performance than with the ConvNext-Tiny backbone. Our
preliminary results also show that the proposed Age-Prototype layer can learn
prototypical age values; however, these do not improve classification
performances compared to the baseline model. Our functionally-grounded
evaluation of prototypes shows that all models learn prototypes consistently
located in the same anatomical brain regions (low LCp values). We also observe
that the models trained with the ConvNeXt-Tiny 3D backbone have higher
compactness. This might partially explain the lower performance scores (the
number of prototypes learned is not enough for performing the diagnosis), but
is an interesting observation for future research as such a highly compact
model can be considered more interpretable than larger ones and can be easily
evaluated by domain experts. We also observe that the image prototypes of the
ConvNeXt-Tiny 3D backbone are generally purer777Purity is measured w.r.t. to
the annotation provided by the CerebrA atlas. Despite purity being a desirable
property for prototypes [20], because of the design of the purity metric, also
a prototype which only includes the background, i.e., a clinically irrelevant
prototype, will have high purity888Posterior quantitative evaluation performed
w.r.t. the CerebrA atlas revealed that the test set image prototypes (averaged
over the 5-folds) obtained with the ConvNeXt-Tiny backbone have a higher
percentage of background voxels included compared to the ones obtained with
ResNet-18 (76.6$\%$ vs 59.2$\%$).

In summary, we proposed PIMPNet, an interpretable multimodal prototype-based
classifier. The proposed architecture is the first prototypes-based network
which performs an interpretable classification based on the detection of
prototypes learned from different data modalities (3D images and age
information). We applied PIMPNet to the binary classification of Alzheimer’s
Disease from 3D sMRI images together with the patient’s age. Despite the usage
of age prototypes do not improve predictive performance compared to the model
trained with only images, we identified different potential reasons which
define the future directions of our work. First, as the original PIPNet
training paradigm includes a pretraining stage [10] of image prototypes, we
plan to include an age prototypes pretraining step w.r.t. the log-likelihood
classification loss. Second, we also plan to work on the model’s design. As
the simple concatenation of the prototype presence score might not be able to
properly represent the relationship between age and image prototypes for the
downstream task, we plan to combine image and age prototypes using a different
(but still interpretable) classifier than a scoring-sheet system.

###### Acknowledgements.

Data used in the preparation of this article was obtained from the Alzheimer’s
Disease Neuroimaging Initiative (ADNI) database. The ADNI was launched in 2003
as a public-private partnership with the primary goal to test whether serial
magnetic resonance imaging (MRI), positron emission tomography (PET), other
biological markers, and clinical and neuropsychological assessment can be
combined to measure the progression of mild cognitive impairment (MCI) and
early Alzheimer’s disease (AD).

## References

  * Ebrahimighahnavieh et al. [2020] M. A. Ebrahimighahnavieh, S. Luo, R. Chiong,  Deep learning to detect alzheimer’s disease from neuroimaging: A systematic literature review,  Computer Methods and Programs in Biomedicine 187 (2020). doi:10.1016/j.cmpb.2019.105242.
  * De Santi et al. [2023] L. De Santi, E. Pasini, M. Santarelli, D. Genovesi, V. Positano,  An Explainable Convolutional Neural Network for the Early Diagnosis of Alzheimer’s Disease from 18F-FDG PET,  Journal of Digital Imaging 36 (2023). doi:10.1007/s10278-022-00719-3.
  * Chandra et al. [2019] A. Chandra, G. Dervenoulas, M. Politis, Magnetic resonance imaging in Alzheimer’s disease and mild cognitive impairment, 2019. doi:10.1007/s00415-018-9016-3.
  * Vemuri and Jack [2010] P. Vemuri, C. Jack,  Role of structural MRI in Alzheimer’s disease,  Alzheimer’s Research and Therapy 2 (2010). doi:10.1186/alzrt47.
  * Zhao et al. [2019] L. Zhao, W. Matloff, K. Ning, H. Kim, I. D. Dinov, A. W. Toga,  Age-related differences in brain morphology and the modifiers in middle-aged and older adults,  Cerebral Cortex 29 (2019) 4169–4193. doi:10.1093/cercor/bhy300.
  * Sivera et al. [2019] R. Sivera, H. Delingette, M. Lorenzi, X. Pennec, N. Ayache,  A model of brain morphological changes related to aging and alzheimer’s disease from cross-sectional assessments,  NeuroImage 198 (2019) 255–270. doi:10.1016/j.neuroimage.2019.05.040.
  * Böhle et al. [2019] M. Böhle, F. Eitel, M. Weygandt, K. Ritter,  Layer-wise relevance propagation for explaining deep neural network decisions in MRI-based Alzheimer’s disease classification,  Frontiers in Aging Neuroscience 10 (2019). doi:10.3389/fnagi.2019.00194.
  * Khojaste-Sarakhsi et al. [2022] M. Khojaste-Sarakhsi, S. S. Haghighi, S. F. Ghomi, E. Marchiori,  Deep learning for Alzheimer’s disease diagnosis: A survey,  Artificial Intelligence in Medicine 130 (2022) 102332. doi:https://doi.org/10.1016/j.artmed.2022.102332.
  * Longo et al. [2024] L. Longo, M. Brcic, F. Cabitza, J. Choi, R. Confalonieri, J. D. Ser, R. Guidotti, Y. Hayashi, F. Herrera, A. Holzinger, R. Jiang, H. Khosravi, F. Lecue, G. Malgieri, A. Páez, W. Samek, J. Schneider, T. Speith, S. Stumpf,  Explainable artificial intelligence (xai) 2.0: A manifesto of open challenges and interdisciplinary research directions,  Information Fusion 106 (2024) 102301. URL: http://creativecommons.org/licenses/by/4.0/. doi:10.1016/j.inffus.2024.102301.
  * Nauta et al. [2023] M. Nauta, J. Schlötterer, M. van Keulen, C. Seifert,  PIP-Net: Patch-Based Intuitive Prototypes for Interpretable Image Classification,  in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023. doi:10.1109/CVPR52729.2023.00269.
  * Santi et al. [2024] L. A. D. Santi, J. Schlötterer, M. Scheschenja, J. Wessendorf, M. Nauta, V. Positano, C. Seifert, Pipnet3d: Interpretable detection of alzheimer in mri scans, 2024. arXiv:2403.18328.
  * Nauta et al. [2024] M. Nauta, J. H. Hegeman, J. Geerdink, J. Schlötterer, M. v. Keulen, C. Seifert,  Interpreting and correcting medical image classification with pip-net,  in: Artificial Intelligence. ECAI 2023 International Workshops, 2024, pp. 198–215.
  * Ma et al. [2022] Y. Ma, S. Zhao, W. Wang, Y. Li, I. King,  Multimodality in meta-learning: A comprehensive survey,  Knowledge-Based Systems 250 (2022). doi:10.1016/j.knosys.2022.108976.
  * Butterworth et al. [1930] S. Butterworth, et al.,  On the theory of filter amplifiers,  Wireless Engineer 7 (1930) 536–541.
  * Mulyadi et al. [2023] A. W. Mulyadi, W. Jung, K. Oh, J. S. Yoon, K. H. Lee, H.-I. Suk,  Estimating explainable Alzheimer’s disease likelihood map via clinically-guided prototype learning,  NeuroImage 273 (2023). doi:10.1016/j.neuroimage.2023.120073.
  * Fonov et al. [2009] V. Fonov, A. Evans, R. McKinstry, C. Almli, D. Collins,  Unbiased nonlinear average age-appropriate brain templates from birth to adulthood,  NeuroImage 47 (2009) S102. URL: https://www.sciencedirect.com/science/article/pii/S1053811909708845. doi:https://doi.org/10.1016/S1053-8119(09)70884-5, organization for Human Brain Mapping 2009 Annual Meeting.
  * Tran et al. [2017] D. Tran, H. Wang, L. Torresani, J. Ray, Y. LeCun, M. Paluri,  A closer look at spatiotemporal convolutions for action recognition (2017). URL: http://arxiv.org/abs/1711.11248.
  * Kienzle et al. [2022] D. Kienzle, J. Lorenz, R. Schön, K. Ludwig, R. Lienhart,  Covid detection and severity prediction with 3d-convnext and custom pretrainings (2022). URL: http://arxiv.org/abs/2206.15073.
  * Manera et al. [2020] A. L. Manera, M. Dadar, V. Fonov, D. L. Collins,  Cerebra, registration and manual label correction of mindboggle-101 atlas for mni-icbm152 template,  Scientific Data 7 (2020). doi:10.1038/s41597-020-0557-9.
  * Nauta and Seifert [2023] M. Nauta, C. Seifert,  The co-12 recipe for evaluating interpretable part-prototype image classifiers,  in: Explainable Artificial Intelligence, 2023, pp. 397–420.