# The Tensor Brain: A Unified Theory of Perception, Memory and Semantic
Decoding

Volker Tresp<EMAIL_ADDRESS>
LMU Munich and Siemens Munich, Germany Sahand Sharifzadeh+∗
<EMAIL_ADDRESS>
LMU Munich, Germany Hang Li+<EMAIL_ADDRESS>
LMU Munich and Siemens Munich, Germany Dario Konopatzki<EMAIL_ADDRESS>
LMU Munich, Germany Yunpu Ma<EMAIL_ADDRESS>
LMU Munich and Siemens Munich, Germany

###### Abstract

We present a unified computational theory of an agent’s perception and memory.
In our model, both perception and memory are realized by different operational
modes of the oscillating interactions between a symbolic index layer and a
subsymbolic representation layer. The two layers form a bilayer tensor network
(BTN). The index layer encodes indices for concepts, predicates, and episodic
instances. The representation layer broadcasts information and reflects the
cognitive brain state; it is our model of what authors have called the “mental
canvas” or the “global workspace.” As a bridge between perceptual input and
the index layer, the representation layer enables the grounding of indices by
their subsymbolic embeddings, which are implemented as connection weights
linking both layers. The brain is a sampling engine: Only activated indices
are communicated to the remaining parts of the brain. Although memory appears
to be about the past, its main purpose is to support the agent in the present
and the future. Recent episodic memory provides the agent with a sense of the
here and now. Remote episodic memory retrieves relevant past experiences to
provide information about possible future scenarios. This aids the agent in
decision-making. “Future” episodic memory, based on expected future events,
guides planning and action. Semantic memory retrieves specific information,
which is not delivered by current perception, and defines priors for future
observations. Our approach explains the great similarity between episodic and
semantic memory: Semantic memory models the episodic memory of a future
instance. We analyse episodic memory and semantic memory in the context of
meta-learning. We argue that it is important for the agent to encode
individual entities, not just classes and attributes. Perception is learning:
Episodic memories are constantly being formed, and we demonstrate that a form
of self-supervised learning can acquire new concepts and refine existing ones.
We test our model on a standard benchmark data set, which we expanded to
contain richer representations for attributes, classes, and individuals. Our
key hypothesis is that obtaining a better understanding of perception and
memory is a crucial prerequisite to comprehending human-level intelligence.

Extended Version111Sahand Sharifzadeh is now with DeepMind London. Dario
Konopatzki is now with ETH Zurich. Extended version of the paper that appeared
in Neural Computation, 2023. The naming of the models has been changed. The
discussion on the post-observation model and Dirichlet fusion in Section 3.7
has been added. Section 4.1 describes the BTN as an autoencoder. Section 10.6,
which discusses forgetting and modularity, has been added.

##  1 Introduction

With an increase in higher animals’ abilities to move and act came a growing
demand for high-performing perceptual systems, beyond simple labeling of
entities with attributes and classes (Hommel et al., 2001). This might have
been a driving force to develop episodic and semantic memory. Episodic memory
engrams permit the recall of recent and remote memories and provide guidance
for acting right. Semantic memory engrams provide concept grounding and
complement perceived information with background knowledge about concepts. The
agent does not just acquire new skills but gains and memorizes knowledge: It
learns about and remembers things in the world. Memories support the agent in
the present and the future: Without memory systems, the brain is literally
memoryless.

Episodic memories retrieve previous experiences in an agent’s life. Recent
episodic memory permits the agent to remember the immediate past since some
state information cannot be directly derived from perceptual input. A recall
is triggered by nearness in time and relevance. It has evolved such that the
agent can remember where it has been before, why it is where it is, and what
the general context is. Recent episodic memory is an episodic memory that is
almost treated as a current observation. For instance, the agent needs to
remember that, even though perception does not give a clue, it is still in the
hideout because the bear had been chasing it and might still be lurking
outside. Remote episodic memory can remind an agent about past situations
—similar to the current— and imminent danger and favorable actions associated
with that situation. It provides estimates about possible future scenarios and
aids the agent in decision-making. Continuing the previous example, the agent
might remember previous personal bear encounters and subsequent dangerous
situations. Recall in remote episodic memory is triggered by closeness between
episodic representation and scene representation. Finally, we define future
episodic memories as events that are expected to be a memory in the future.
Information on future events influences planning and action.

Semantic memory models the time-invariant statistics of statement
probabilities and acts as their prior. It is a dictionary view of the agent’s
life.222Some authors distinguish between a formal dictionary and a grounded
cognitive encyclopedia (Evans, 2012). We will not make this distinction. It
aggregates entity information, which might have been acquired at different
episodes, and this information becomes local in the graph formed by semantic
memory. Semantic memory enables multimodal integration. For example, if Sparky
is discovered in a scene, semantic memory provides background information, for
example, that Sparky is a young dog and is owned by Jack; and although dogs,
in general, might be aggressive, Sparky is a friendly dog. Semantic memory
support can be essential for survival. An agent simply knows that bears are
dangerous, even when a bear looks cozy and sleepy and even if the agent did
not yet have an unpleasant encounter with a bear. Semantic memory is the
default semantic state estimator and is complemented by perception and recent
episodic memories, when and where available. Our approach explains the great
similarity between episodic and semantic memory: Semantic memory is the
expected episodic memory of a future instance. An elementary symbolic
statement says that an entity has a particular attribute, belongs to a
specific class, or that an entity has a particular relationship with another
entity. This motivates our assumption that basic facts are expressed as triple
statements of the form (subject, predicate, object).333 In a relational
Bayesian network or a Markov logic network, a triple statement is represented
as a node; in a graph neural network, it would be a typical output node. In
most languages, basic facts are expressed in this or a similar format. Thus,
triple sentences are arguably of fundamental relevance for encoding and
communicating perception, episodic memory, and semantic memory, all of which
humans can easily describe by language. Communication is important to make the
invisible visible: for example, an agent could inform peers that a bear is
lurking outside the hideout, even if it cannot be seen. Triple statements can
represent relationships between entities, which enables, for example, rich
scene descriptions and reasoning in social and spatial networks. An agent can
understand not only that a bear and a deer are in a scene but that, luckily,
the bear is not chasing the agent itself but the deer. Symbolic
representations and reasoning, classical System-2 properties, depend on the
representations of entity-to-entity relationships (Halford et al., 2014).

The starting point of our work is a mathematical model, that is, the bilayer
tensor network (BTN). It implements mathematical models for perception and
memory that obey some constraints imposed by biology. We then discuss how
particular features of our model relate to current discussions in cognition
and neuroscience and make predictions about implications of our model to both.
The BTN implies a very simple architecture that contains two basic layers: the
index layer and the representation layer. Context is provided by a third
layer, the dynamic context layer, which interacts with the representation
layer, provides context, and stores state information when the brain’s
attention moves from one entity to another.

The symbolic index layer contains indices for concepts, predicates, and
episodic instances known to the agent. The index layer labels the activation
pattern in the representation layer and then feeds back the embedding of that
label to the representation layer. The embedding vectors are implemented as
connection weights linking both layers. An index is a focal point of activity
and competes with other indices, but, since it constantly interacts with the
representation layer, it is never active in isolation. Embeddings have an
integrative character: the embedding vector for a concept index integrates all
that is known about that concept, and the embedding vector for an episodic
index represents the world state at that instance. The subsymbolic
representation layer is the main communication platform. In cognitive
neuroscience, it would correspond to, what authors call, the “mental canvas”
or the “global workspace” and reflects the cognitive brain state. In bottom-up
mode, scene inputs activate the representation layer, which then activates the
index layer. In top-down mode, an index activates the representation layer,
which might subsequently activate even earlier processing layers. This last
process is called the embodiment of a concept.

Perception and memories first produce subsymbolic representations. They
represent the agent’s current and past cognitive state and are subsequently
decoded semantically to produce sequences of activated indices that form
symbolic triple statements. These sequences then give feedback to the
representation layer and earlier processing layers and thus inform the brain
as a whole about what has been decoded. The brain is a sampling engine: Only
activated indices are communicated to the remaining parts of the brain.

In our approach, perception and memory produce triple statements by stochastic
sampling, which is a central bottleneck, as also discussed by Dehaene (2014).
We introduce an attention approximation, which avoids intermediate sampling
decisions on visual entities.

One can consider a graph where concepts are represented as nodes, and triples
become labeled directed links pointing from subject node to object node. In
computer science, this would be called a knowledge graph (KG). Whereas
symbolic reasoning would purely act on the graph, embedded reasoning, as
realized by the BTN, involves concept embeddings and performs information
propagation via the representation layer.

This article is organized as follows. In the next section, we cover related
work. Section 3 introduces concepts, triple statements, and probabilistic
memory models. In Section 4, we present our proposed bilayer tensor network
(BTN) and demonstrate how it realizes perception and the different memory
functions. We relate it to hierarchical Bayesian modelling and meta-learning.
We show how a posterior model for an instance can be defined and how this
posterior model, but also episodic memory and semantic memory, can be viewed
as different variants of Dirichlet updates. Section 5 presents and discusses
the BTN’s algorithmic implementation, sampling, and attention approximations.
The following sections present experimental results and discuss potential
relationships to cognition and neuroscience. Section 6 introduces the data set
and discusses perception, the representation layer, and the dynamic context
layer. Section 7 discusses semantic memory engrams and their semantic
decoding. Using social network data, we demonstrate multimodal integration in
semantic memory. Section 8 compares embedded reasoning with symbolic reasoning
and proposes that embedded symbolic reasoning performs symbolic reasoning
using embedding vectors. We introduce the one-brain hypothesis, which
emphasizes, first, that the brain uses only one representation layer and,
second, that perception, episodic memory, semantic memory, and embedded
reasoning all rely on the same BTN architecture. We discuss relationships
between our approach to cognitive linguistics and consciousness research. We
argue that the internal triple-oriented fast speech is transformed into an
external, sophisticated slow speech. Section 9 covers episodic memory and the
way the agent estimates current and future world states. In Section 10, we
discuss that perception involves the storing of new episodic memories and show
how a form of self-supervised learning can learn new concepts and refine
existing ones. We consider that new perceptual experiences might require
representations for new entities; that is, we do not assume domain closure.
Section 11 contains our conclusions.

##  2 Related Work

###  2.1 Tensor Networks for Modeling Knowledge Graphs

The bilayer tensor network (BTN) is an example of a tensor network. RESCAL was
the first tensor-based embedding model for triple prediction in relational
data sets and knowledge graphs (Nickel et al., 2011, 2012). Embedding learning
for knowledge graphs evolved into a sprawling research area (Bordes et al.,
2013; Socher et al., 2013; Yang et al., 2014; Nickel et al., 2015b; Trouillon
et al., 2016; Dettmers et al., 2018). (Nickel et al., 2015a) provides an
overview and PyKEEN (Ali et al., 2021) a comprehensive software library. In
contrast to previous approaches, the BTN can be implemented as an interaction
between an index layer and a representation layer, and thus it is more
suitable for brainware implementations.

###  2.2 Cognitive Tensor Networks and Related Models

Tensors have been used previously as memory models where the main focus was on
simple associations (Hintzman, 1984; Kanerva, 1988; Humphreys et al., 1989;
Osth and Dennis, 2015) and compositional structures (Smolensky, 1990; Pollack,
1990; Plate, 1997; Halford et al., 1998; Ma et al., 2018a). In the tensor
product approach (Smolensky, 1990), encoding or binding is realized by a
tensor product (generalized outer product) and compositionality by tensor
addition. In the structured tensor analogical reasoning (STAR) model (Halford
et al., 1998), a working memory is constructed by a superposition of tensor
products. Either standard basis (one-hot) or random vectors are used as
representations for concepts and predicates. Some basic symbolic reasoning
operations are implemented, such as analogical and transitive inference. The
main difference to our work is that we approximate the data tensor by a tensor
network, that is, the BTN. This permits generalization and realizes embedded
reasoning, as well as the seamless integration of sensory input.

The application of embedding-based tensor models to neurocognitive models
started with (Tresp et al., 2015), which introduced tensor networks with index
embeddings for perception, as well as semantic and episodic memory. It did not
explicitly consider scene bounding boxes and did not contain experimental
results. In further work, the connection between temporal and semantic tensor
networks was analysed (Tresp and Ma, 2016; Tresp et al., 2017a, b; Ma et al.,
2018b). Semantic memory was derived from episodic memory by an integration
step performed in latent space. In this article, we avoid explicit integration
and extend the approach to include perception.

The neurocognitive tensor network theory evolved in the 1980s and followed the
idea of geometrization of biology. It is a theory of brain function,
particularly that of the cerebellum. Metric tensors transform sensory space-
time coordinates into motor coordinates (Pellionisz and Llinás, 1980). There,
tensor networks are used very differently from the work presented here.

###  2.3 Visual Relationship Detection and Scene Graphs

In 2016, the Stanford Visual Relationship data set was published, which
contained images annotated with triple sentences (Lu et al., 2016) and
(Krishna et al., 2017). The two articles made their annotated data available,
which spawned an explosion of research activity in visual relationship
detection (VRD). The background information in (Lu et al., 2016) was extracted
from a text corpus. Recent work in this direction is (Luo et al., 2019).

VRD models for knowledge graphs were proposed by (Baier et al., 2017; Zhang et
al., 2017; Baier et al., 2018). (Baier et al., 2017) showed how a prior
distribution derived from triple occurrences could significantly improve on
pure vision-based approaches and on approaches that used prior distributions
derived from language models. (Sharifzadeh et al., 2019) showed further
improvements by including 3D image information. (Tresp et al., 2019, 2020)
describe more recent publications in this tradition. The presented work
introduces more clearly the different operational modes and provides more
extensive experimental results.

Triple statements generated from an image form a scene graph (Johnson et al.,
2015). Work on scene graphs attempts to find a unique, globally optimal
interpretation of an image. State-of-the-art scene graph models are described
in (Yang et al., 2018; Zellers et al., 2018; Hudson and Manning, 2019).
(Sharifzadeh et al., 2020) captures the interplay between perception and
semantic knowledge by introducing schema representations and implementing the
classification as an attention layer between image-based representations and
the schema.

###  2.4 Related Modern Technical Models for Memory

(Hochreiter and Schmidhuber, 1997) convincingly demonstrated the importance of
memory systems in recurrent neural networks. Important later extensions are
the neural Turing machine (NTMs) (Graves et al., 2014) and memory networks
(Weston et al., 2014; Sukhbaatar et al., 2015). In those articles, episodic
memory acts as an instance buffer. Both use a recurrent neural network in
combination with attention mechanisms. Large language models, like OpenAI’s
GPT-3, are more recent developments (Brown et al., 2020).

In our BTN, a dynamic context layer is part of a (nonstandard) recurrent
neural network. The attention mechanisms in our model, episodic attention and
semantic attention, are quite different from the attention mechanisms in those
articles. Also, our goal is to derive triple statements, whereas, in those
models, the tasks are query answering and improved token embeddings.

###  2.5 Dual Process Theory and Complementary Learning Systems (CLS)

In psychology, dual process theory concerns the interplay in the mental
processing of an implicit, automatic, unconscious process (shared with
animals) and an explicit, controlled, conscious process (uniquely human). See
(Evans, 2003) for a review. In our model, the implicit side would be on the
level of embeddings and representations, whereas the explicit side is on the
level of the concept indices and the extracted triple sentences.

One instance of a dual process theory is Kahneman’s System-1 / System-2
dichotomy (Kahneman, 2011). System-1 is fast, instinctive, and emotional and
does not require mental effort. System-2 is slower, more deliberate, and more
logical and requires mental effort.

CLARION is a dual-process model of implicit and explicit learning (Sun and
Peterson, 1996). It is based on one-shot explicit rule learning (i.e.,
explicit learning) and gradual implicit tuning (i.e., implicit learning).

A different but related dichotomy can be found in the complementary learning
systems (CLS) theory (McClelland et al., 1995; Kumaran et al., 2016), where
the formation of time indices and their embeddings would be part of a
nonparametric learning system centered on the hippocampus, which allows rapid
learning of the specifics of individual items and experiences (Kumaran et al.,
2016). Slow training would be part of a parametric learning system, which
serves as the basis for the gradual acquisition of structured knowledge about
the environment to neocortex (Kumaran et al., 2016). In this article, we
introduce self-supervised learning for rapid learning and discuss a
consolidation process of learned knowledge. McClelland et al. (2020) analysed
the connection between memory, perception, and language, which is also a focus
of this article.

###  2.6 The Bayesian Brain

Our approach can be related to the tradition of Bayesian approaches to brain
modeling (Dayan et al., 1995; Rao and Ballard, 1999; Knill and Pouget, 2004;
Körding et al., 2004; Tenenbaum et al., 2006; Griffiths et al., 2008; Friston,
2010). In our approach, all past experiences train the perceptual system and
also contribute to perception via memory. In (Baier et al., 2017) an explicit
semantic prior distribution was used, describing a priori probabilities for
triple sentences. For inference, Bayes’ formula is used. The remarkable
improvement in performance after integrating the prior information indicates
that triple representations might be a powerful abstraction level for
formulating prior knowledge, in general. (Sharifzadeh et al., 2020) showed
that the probabilistic knowledge graph acts as an inductive bias in
perception. It discusses the role of a prior as an integrator of multimodal
information and its role in filling in nonperceptual background information.

In the work presented here, we separately model and perform completion on the
observed statements, statement priors, and statement posteriors. This is in
the context of Bayesian probabilistic inference but not in the sense of
Bayesian statistics.

In a Bayesian brain approach, top-down connections are often associated with
predictive models; they provide predictions about future state probabilities.
Here, we emphasize that, in addition, top-down connections inform the
representation layer and earlier processing layers— what the brain has
detected.

##  3 Concepts, Triple Statements and Probabilistic Memory Models

You only see what you know. —Johann Wolfgang von Goethe, an Friedrich von
Müller, April 24, 1819

###  3.1 Triple Statements

To understand what is perceived, an agent needs to have an understanding of
the things in the world and their relationships. We assume that the agent’s
mind is aware of $N_{C}$ concepts $\mathcal{C}=\\{c_{1},\dots,c_{N_{C}}\\}$. A
concept can, for example, represent an entity
$e\in\mathcal{E}\subset\mathcal{C}$, or a class
$k\in\mathcal{K}\subset\mathcal{C}$, which stands for a collection of
entities, or an attribute $b\in\mathcal{B}\subset\mathcal{C}$.

In addition to concepts, we also consider a set of predicates
$p\in{\mathcal{P}=\\{p_{1},\dots,p_{N_{P}}\\}}$, where $N_{P}$ is the number
of predicates the agent is aware of. A triple statement has the form
$(s,p,o)$, where $s\in\mathcal{C}$ assumes the role of the subject,
$o\in\mathcal{C}$ assumes the role of the object, and predicate
$p\in\mathcal{P}$. Examples of triple statements are: (Munich, partOf,
Bavaria), (Sparky, looksAt, Jack), (AkiraKurosawa, directorOf, SevenSamurai),
and (Jack, knows, Mary).

Following the notion of the semiotic triangle of (Ogden and Richards, 1923),
we can look at triples and their semantics from three different perspectives:

  * •

The agent-independent objective world (world semantics). Some triples have an
interpretation in the real world, and they stand for propositions, like
(Munich, partOf, Bavaria) and (Sparky, looksAt, Jack). This view is mostly
taken in a formal analysis of cognition and linguistics (Montague, 1970;
Fodor, 1975) and is sometimes referred to as truth-conditional semantics. An
agent is only aware of some of the concepts in the world, sometimes called the
“projected world” or the “construal world.”

  * •

An agent’s mind and brain (an agent’s neurocognitive semantics): A triple is a
statement based on concepts in an agent’s mind. These might have an immediate
meaning in the objective world but could also be untestable statements with
unclear world semantics such as (Love, stongerThan, Hate). A hypothesis of
this article is that triple statements correspond to processes in the brain,
that is, a sequential firing of index representations. They form the inner
fast speech.

  * •

An agent’s utterance (an agent’s linguistic semantics). A triple is a simple
language clause involving symbols. Within the approach in this article, we
assume that any triple statement in the mind can also be expressed
linguistically. But what is spoken, in general, represents external slow
speech and is obviously more complex, nuanced, and sophisticated than the
inner fast speech, and is modulated, for example, by intent, social context,
and cultural background. In general, the relationship between statements in
the mind (with concepts) and in language (with symbols) is a matter of an open
debate (Evans, 2012).

We are aware that the relationship of the three perspectives touches on
fundamental issues in cognition, linguistics, neuroscience, philosophy, and
many other scientific fields and academic disciplines. A more detailed
discussion would be beyond the scope of this article. This article assumes the
position of the agent. We are concerned only about how the agent views the
world. We focus on personal Bayesian probabilities, modeling the agent’s
neurocognitive semantics, where we restrict ourselves to statements that, in
the mind of the agent, can be true or false, for example, (Sparky, ownedBy,
Jack), and facts that can be related to observations, for example, (Sparky,
hasColor, Black). This is more of a practical than a principled constraint. On
some dimensions thought and reality agree simply to act right and to guarantee
survival; on others, there is likely no agreement between agents, no clear
relationship to an objective reality, and in determining the truth values on
statements, personal emotions and judgments might play a significant role.

###  3.2 Knowledge Graphs

In a knowledge graph (KG), a triple is represented as a directed labeled link
from concept $s\in\mathcal{C}$ to concept $o\in\mathcal{C}$, where the link is
labeled by $p$. Thus, a labeled link represents a triple statement of the form
$(s,p,o)$ where $s$ is called head or source node and $o$ is called tail or
target node. Knowledge graphs currently have a great impact in applications
and, as our approach, are entity oriented.

###  3.3 Unary and Binary Statements

When $s$ and $o$ are entities, $(s,p,o)$ stands for the ground atom $p(s,o)$.
(Munich, partOf, Bavaria) would stand for partOf(Munich, Bavaria).

We assume a strong default predicate hasAttribute (abbreviated as hA), which,
depending on the subject type and the object type, can stand for certain other
predicates from the set $\mathcal{P}^{B}\subseteq\mathcal{P}$. We write (s,
hA, c), $c\in\mathcal{C}$.

  * •

If both subject and object are entities, the implicit predicate is sameAs;
thus, (Jack, hA, John) stands for (Jack, sameAs, John).

  * •

If the subject is an entity and the object is an attribute, the implicit
predicate is attribute specific but should be obvious; thus (Jack, hA, Tall)
stands for (Jack, height, Tall).

  * •

If the subject is an entity and the object is a class, the implicit predicate
is type, as in (Sparky, type, Dog).

  * •

If both subject and object are classes, the implicit predicate is subClass, as
in (Dog, subClass, Mammal); note that the transitive subClass predicate
permits the modeling of deep ontologies.

Triple sentences involving the hasAttribute predicate (and its substitutes) we
call unary statements. The remaining $N_{B}$ predicates from
$\mathcal{P}_{B}\subseteq\mathcal{P}$ form binary statements. We will refer to
the object in a unary statement also simply as the unary label of the subject,
and to the predicate in a binary statement as the binary label of a concept
pair. Binary statements are required when the default unary interpretation is
not applicable, as in (Jane, motherOf, Jack) (the default would be (Jane,
sameAs, Jack)).

Higher-order relations can be reduced to a set of binary statements, for
example, by using additional concepts (Noy et al., 2006). For example,
match(Player1, Player2, Location) becomes (matchID, hasPlayer, Player1),
(matchID, hasPlayer, Player2), and (matchID, hasLocation, Location), where
(matchID) is the additional concept.

###  3.4 Probabilistic Models with Boolean Variables

In the assumed lifetime and the mind of an agent, some statements are always
true, as (Munich, partOf, Bavaria), but the truth values of other statements
can change in time, as (Munich, weather, Sunny). Thus, with each triple
sentence, at each episodic instance $t$, the agent associates a Boolean
semantic state variable $Y_{s,p,o,t}$. If the agent is certain that $(s,p,o)$
is true at episodic instance $t$, then $Y_{s,p,o,t}=1$ and if $(s,p,o)$ is
observed to be false at episodic instance $t$, then $Y_{s,p,o,t}=0$. The
agent’s semantic world state at episodic instance $t$ is defined as the states
of all semantic state variables. Here, we assume that the agent is concerned
with $N_{T}$ past episodic instance $\mathcal{T}=\\{t_{1},\dots,t_{N_{T}}\\}$.

We consider that data, that is, information on the states on triple
statements, arrives in chunks, which we call episodes. McClelland et al.
(2020) call them situations and in cognition they are called event frames. In
knowledge graphs, they could form a namespace. We use the terms events and
episodes almost interchangeably, although, in a narrower sense, we reserve the
term episode for a sequence of events. (See also the discussion in Section 9.)
Each episode provides information about the truth values of a subset of all
statements. In this article, we assume that episodic data is either provided
by a human annotator (supervised learning) or generated in self-supervised
learning (see Section 10). We can form a data tensor of observed triples as a
four-mode triple-observation tensor of the form

${\underline{\underline{T}}}=\sum_{s,p,o,t}y_{s,p,o,t}\;\mathbf{e}^{s}\otimes\mathbf{e}^{p}\otimes\mathbf{e}^{o}\otimes\mathbf{e}^{t}.$

Here, $\mathbf{e}^{i}$ is our notation for a standard basis vector, that is, a
one-hot vector, with the one at position $i$. Such data tensors have also been
the starting point for the tensor memories of Halford et al. (1998) and (Ma et
al., 2018a). A tensor entry $y_{s,p,o,t}$ is simply equal to one if the
statement $(s,p,o)$ was observed by the agent to be true at $t$ and equal to
zero if the corresponding statement was observed or concluded by the agent to
be false at $t$; otherwise, it is unknown. The temporal KG (tKG) is the graph
representation of ${\underline{\underline{T}}}$, where the labeled links
between concepts are time-dependent . We assume that data is acquired from
different modalities, like vision or language. We invoke a local closed-world
assumption (LCWA): We assume that for entities for which data is acquired in a
modality, the truth values of all modality-specific unary statements are
available. For entity pairs in a modality, we assume that the truth values of
all modality-specific binary statements are available. The issue of which
entries in ${\underline{\underline{T}}}$ are observed might involve some
selection or attention process. For example, state changes or singular events
might be selected with some priority. We discuss this issue in Section 9.9.

In the next section, we introduce the bilayer tensor network (BTN), which
performs data completion. For all $y_{s,p,o,t}$ that are either observed to be
true or concluded by the LCWA to be false, the BTN Boolean observation model
is

$\mathbb{E}(Y_{s,p,o,t}|\;{\underline{\underline{T}}})\equiv\mathbb{E}(Y_{s,p,o,t})\approx
y_{s,p,o,t}.$ (1)

The observation model has the characteristics of an autoencoder, where
${\underline{\underline{T}}}$ represents noisy observations. This relationship
is indicated by the approximation symbol ($\approx$). As indicated, we
simplify notations here and in the following by not explicitly conditioning on
${\underline{\underline{T}}}$.

We also define the quantity $i_{s,p,o,t}$, where $i_{s,p,o,t}=1$, if
$y_{s,p,o,t}=1$, and $i_{s,p,o,t}=0$, otherwise. Thus, unknowns in the triple-
observation tensor are assigned the value $0$. Consider a future instance
$t^{\prime}$ for which no data is yet available. We define the BTN’s expected-
state model as

$\mathbb{E}(Y_{s,p,o,\bar{t}}|\;{\underline{\underline{T}}})\equiv\mathbb{E}(Y_{s,p,o,\bar{t}})\approx\frac{1}{{N_{s,p,o,\mathrm{known}}}}\sum_{t}i_{s,p,o,t}.$
(2)

Here, $\bar{t}$ is a new instance not represented in
${\underline{\underline{T}}}$. $N_{s,p,o,\mathrm{known}}>0$ counts the times
that the truth values of $(s,p,o)$ are known, either because they were
observed to be true or as concluded by the LCWA to be false. The expected-
state model simply predicts the mean truth value of a future triple statement.
The average in the last equation might only consider a limited time horizon to
be able to account for state changes (see also Section 10).

If we consider that the observations at an instance describe a knowledge
graph, then the expected-state model performs integration by modelling the
average truth values of known triples’ truth values.

###  3.5 Observation Model (Episodic Memory)

In this context $s,p,o,t$ are the states of categorical variables. For subject
and object, they have $N_{C}$ states, for the predicate $N_{P}$ states, and
for instances $N_{T}$ states. The BTN’s observation model is

$\mathbb{P}(s,p,o|t,{\underline{\underline{T}}})\equiv\mathbb{P}(s,p,o|t)\approx\frac{1}{N_{t}}i_{s,p,o,t}.$
(3)

It models the distribution of the indices. $N_{t}=\sum_{s,p,o}i_{s,p,o,t}$ is
the total number of triples observed to be true at $t$.

As we will discuss further down, the observation model mathematically
describes the symbolic aspect of episodic memory. Episodic memory is about
what was actually observed. It implicitly assumes a closed-world assumption
(CWA), in the sense that what was not observed should not be sampled.

From the definitions, it follows that for observed triples and where the LCWA
applies

$\mathbb{P}(s,p,o|t)\approx\mathbb{E}(Y_{s,p,o,t})/{N_{\textit{t}}}.$

The agent can thus assume that for a given $t$, a sample generated from the
observation model is true. If ${\underline{\underline{T}}}$ is a database,
$\mathbb{E}(Y_{s,p,o,t})$ describes a probabilistic model of that database,
and $\mathbb{P}(s,p,o|t)$ is a database retrieval model.

###  3.6 Pre-observation model (Semantic Memory)

The BTN’s pre-observation model is

$\mathbb{P}(s,p,o|\bar{t},{\underline{\underline{T}}})\equiv\mathbb{P}(s,p,o|\bar{t})\approx\frac{1}{N_{\mathrm{total}}}\sum_{t}i_{s,p,o,t}.$
(4)

$N_{\mathrm{total}}=\sum_{t}N_{t}$ is the total number of observed true
triples. The pre-observation model provides background information on the
subject. It represents the predictions of a model with an informative learned
prior for some future instance $\bar{t}$. As we will discuss in the course of
the paper, the pre-observation model mathematically describes the symbolic
aspect of semantic memory. Samples generated by the pre-observation model
correspond to triples often observed to be true. See a more detailed
discussion in Appendix B.

If we consider again that the observations at an instance describe a knowledge
graph, then the pre-observation model performs integration by summing the
truth values of a triple statement at different instances, followed by
normalization. Whereas the expected-state model predicts what is true or
false, the pre-observation model predicts what should be observed next at a
new instance.

###  3.7 Post-observation Model and Meta-learning

The pre-observation model estimates the semantic world state prior to
observations for a new instance $t^{\prime}$. It predicts what should be
observed next. The post-observation model combines episodic and semantic
memory to estimate the state for $t^{\prime}$ after taking into account
observations at $t^{\prime}$. Motivated by a hierarchical Bayesian analysis,
we consider the pre-observation model to be the base distribution of a
Dirichlet distribution with concentration parameter $\gamma>0$. We obtain the
post-observation model by a Dirichlet update of the form

$\frac{1}{\gamma+\tilde{N}_{t^{\prime}}}\left(\gamma\mathbb{P}(s,p,o|\bar{t})+\tilde{N}_{t^{\prime}}\mathbb{P}(s,p,o|t^{\prime})\right).$
(5)

The value of $\gamma$ indicates the trust in the pre-observation model and
$\tilde{N}_{t}=N_{t}$. We can sample from this distribution by sampling from
the pre-observation model (semantic memory) with a frequency proportional to
$\gamma$ and from the observation model (episodic memory) with a frequency
proportional to $N_{t}$. The latter recalls actual observations. The former
adds triples describing the state, prior to the observation. As discussed in
Section 9.9, the observations might include samples from recent episodic
memories, which are treated as current observations. The post-observation
model should not be confused with the updates of the pre-observation model.
For the update of the pre-observation model, we would have
$\gamma=N_{\textit{total}}$. In essence, in the post-observation model, the
current observations obtain a larger weight. In terms of meta-learning, an
instance is a domain, and an instance-specific episodic memory a domain-
specific model, but with shared embedding vectors. Semantic memory would be an
instance-independent model trained by pooling all data. The post-observation
model is the target-domain prediction. Table 1 provides a summary of the
different models. Appendix C provides a visualization.

Model | Function | $\gamma$ | $\tilde{N}_{t}$ | Embedding
---|---|---|---|---
Pre-observation model ($<t^{\prime}$) | Semantic memory | $N_{\textit{total}}$ | $0$ | $\mathbf{\bar{a}}$
Observation model ($t^{\prime}$) | Episodic memory | 0 | $N_{t^{\prime}}$ | ${\mathbf{a}_{t}}$
Post-observation model ($t^{\prime}$) | Posterior state at $t^{\prime}$ | hyperp. | $N_{t^{\prime}}$ | ${\approx\mathbf{a}_{t}}+(\gamma/N_{t^{\prime}})\mathbf{\bar{a}}$
Pre-observation model ($>t^{\prime}$) | Semantic memory | $N_{\textit{total}}$ | $N_{t^{\prime}}$ |  $\mathbf{\bar{a}}$ (updated)
Table 1:  Special cases of the Dirichlet update in Equation 5. The pre-
observation model can generate samples from the state prior to observations at
$t^{\prime}$ ($<t^{\prime}$). The observation model can generate samples from
the observations at $t^{\prime}$. The post-observation can generate samples
from the state posterior to observations at $t^{\prime}$, for the state at
$t^{\prime}$. The pre-observation model ($>t^{\prime}$) integrates
observations at $t^{\prime}$ to predict future states. Embedding vectors will
be introduced in Section 4.

###  3.8 Modelling Dependencies: Generalized Statements

Can classes or attributes be the subject in a triple statement? Whereas the
semantics of (Sparky, hasColor, Black) is clear, the semantics of (Dog,
hasColor, Black) is less well defined: For example, it could mean that there
is a least one dog that is black, or that some or most dogs are black, or that
all dogs are black. In our work, we consider frequencies.

We define a triple $(c_{1},\textit{hA},c_{2})$ with
$c_{1},c_{2}\in\mathcal{C}$ where

$\mathbb{E}(Y_{c_{1},\textit{hA},c_{2},\bar{t}})\equiv\mathbb{E}(Y_{\bar{s},\textit{hA},c_{2},\bar{t}}|Y_{\bar{s},\textit{hA},c_{1},\bar{t}})$
(6)
$\approx\frac{1}{{\sum_{s}N_{s,\textit{hA},c_{1},\mathrm{known}}}}{\sum_{t,s}i_{s,\textit{hA},c_{1},t}}\;i_{s,\textit{hA},c_{2},t}.$

This is the probability that a new entity $\bar{s}$ at a new time instance
$\bar{t}$ that has unary label $c_{1}$ will also have unary label $c_{2}$.
Generalized statements are useful for predictions at a new instance $\bar{t}$
where only partial measurements are available. With a predicted unary label
Dog and using the generalized statement (Dog, hA, Mammal), the unary label
Mammal can be predicted.

A generalized statement corresponds to a probabilistic version of the rule
$\forall s:\textit{(s, hA, Mammal)}\leftarrow\textit{(s, hA, Dog)}$ where $s$
is a variable that stands for an entity. Generalized statements are the basis
for embedded symbolic reasoning described in Section 8.3.

Distinguishing between entities, classes, and attributes is important for some
of the discussions. But generalized statements permit us to treat all concepts
almost identically in the algorithmic implementation in Section 5.

##  4 A Bilayer Tensor Network (BTN)

###  4.1 The BTN Observation Model

In this section, we introduce the embedding-based bilayer tensor network
(BTN). It assigns an $r$-dimensional latent embedding vector to each concept
$c$, predicate $p$, and instance $t$. The embedding vectors
$\\{\mathbf{a}_{c}\\}_{c\in\mathcal{C}}$,
$\\{\mathbf{a}_{p}\\}_{p\in\mathcal{P}_{B}}$, and
$\\{\mathbf{a}_{t}\\}_{t\in\mathcal{T}}$ form the columns of the embedding
matrix $A$.

In the BTN, all observed triples are independent, given the embedding matrix
$A$. With some of the children observed, represented as the triple-observation
tensor ${\underline{\underline{T}}}$, we get for a triple $(s,p,o)$ at $t$,

$\mathbb{P}(s,p,o|t,{\underline{\underline{T}}})=\int\mathbb{P}(s,p,o|t,\mathbf{a}_{s},\mathbf{a}_{p},\mathbf{a}_{o},\mathbf{a}_{t})\;\mathbb{P}(A|{\underline{\underline{T}}})\;dA.$
(7)

The last equation has the characteristics of an autoencoder, where
${\underline{\underline{T}}}$ represents noisy observations. We use a point
estimate approximation as

$\mathbb{P}(A|{\underline{\underline{T}}})\approx\delta(A-\hat{A})$ (8)

where $\delta(\cdot)$ is the Dirac delta distribution. The observation model
then becomes

$\mathbb{P}(s,p,o|t,{\underline{\underline{T}}})\approx\mathbb{P}(s,p,o|\mathbf{\hat{a}}_{s},\mathbf{\hat{a}}_{p},\mathbf{\hat{a}}_{o},\mathbf{\hat{a}}_{t}).$
(9)

We ignore in this notation that due to the softmax transfer function, the
conditional probability implicitly depends on the complete embedding matrix.

The point estimates
$\mathbf{\hat{a}}_{s},\mathbf{\hat{a}}_{p},\mathbf{\hat{a}}_{o},\mathbf{\hat{a}}_{t}$
can be sensitive to changes in ${\underline{\underline{T}}}$, implying that
there are global dependencies between all triple statements. For example, if a
new triple becomes known for the current instance $t$, this information will
become part of ${\underline{\underline{T}}}$ and potentially results in major
changes to $\mathbf{\hat{a}}_{t}$, affecting all triple statements at $t$.

In the following notation we do not use the hat notation; for example, we
write $\mathbf{a}_{t}$ instead of $\mathbf{\hat{a}}_{t}$.

In the sampling mode, we decompose into conditional probabilities and generate
samples for binary statements from

$\mathbb{P}(s,p,o|t)=\mathbb{P}(s|t)\mathbb{P}(o|s,t)\mathbb{P}(p|s,o,t)$ (10)

where $p\in\mathcal{P}_{B}$ is a binary label. For unary statements where we
use $p=\textit{hA}$ (hasAttribute) and we generate samples from
$\mathbb{P}(s|t)\mathbb{P}(c|s,t)$, where $c$ is the unary label.

###  4.2 The BTN Pre-observation and Post-observation Model

We obtain the pre-observation model (semantic memory) by using the
approximation in Equation 9 but by replacing
$\mathbf{\hat{a}_{t}}\leftarrow\mathbf{\bar{a}}$. Technically,
$\mathbf{\bar{a}}$ is a hyperparameter vector that was fit on all past data
(data pooling).

The post-observation model is achieved simply by applying the Dirichlet update
Equation 5 to the learned BTN models. Actually, we would propose that the
brain generates samples alternately from the observation model and semantic
memory, enabling the mind to keep track of what was observed and what was
added by semantic memory, representing background knowledge.

###  4.3 BTN Architecture

The BTN is implemented by an interaction of two layers, which, as we argue
later, might be related to functional brain operations. One layer is the
representation layer $q$ with pre-activation vector
$\mathbf{q}\in\mathbb{R}^{r}$; here, $r\in\mathbb{N}$ is the embedding
dimension, that is, the rank of the BTN approximation. As we propose later,
$\mathbf{q}$ reflects the cognitive state of the brain (see Figure 1). The
other one is the index layer ${n}$ with pre-activation vector
$\mathbf{n}\in(0,1)^{d}$, with $d=N_{C}+N_{P}+N_{T}$. The index layer contains
one dimension or unit for each concept, predicate, and episodic instance. The
connection matrix $A$ links both layers, and its columns are the embeddings of
the indices. We also introduce a dynamic context layer $h$ with pre-activation
vector $\mathbf{h}$. In the following, we describe the implementations of
layers and operations using the unfolded view in Figure 2.

Figure 1:  Perception and memory: Our model architecture consists of two main
layers, the representation layer ${q}$ and the index layer ${n}$. $\mathbf{q}$
and $\mathbf{n}$ are the vectors of pre-activations. In perception, the
representation layer obtains inputs from the scene. Whereas the computational
operations in these layers are practically instantaneous, the dynamic context
layer ${h}$ represents internal brain activities and stores information when
attention moves from one entity to another. The dotted line represents
embodiment, the activation of earlier processing layers via the representation
layer.  Figure 2:  Unfolded representation of episodic and semantic memory. In
perception, the structures in yellow color at the bottom are added; they stand
for inputs from the visual scene.

###  4.4 BTN Algorithm for Memory Recall

In this section, the embedding vectors contain free parameters optimized for
good performance. In the following sections, we will ground them in
perception, in the spirit of an embodied approach.

We start with an instance $t$ to activate episodic memory. Thus the activation
of the index layer is $\mathbf{n}_{T}=\mathbf{e}^{t}$. This activates
$\mathbf{q}_{T}=\mathbf{a}_{t}$.
$\mathbb{P}(c_{\mathrm{instance}}|t)=\mathrm{softmax}^{\beta}_{c}(\mathbf{n}_{C})$
is the probability for a unary label of the instance, for example, a
particular vacation day, or a particular location, for example, Paris. Here,
$\mathbf{n}_{C}=\mathbf{a}^{\top}_{c}\mathrm{sig}(\mathbf{q}_{T})$. In our
experiments, we do not use instance labels. We get for the subject

$\mathbb{P}(s|t)=\mathrm{softmax}^{\beta}_{s}(\mathbf{n}_{S})$ (11)

with pre-activations
$\mathbf{n}_{S}=\mathbf{a}_{s}^{\top}\mathrm{sig}(\mathbf{g}(\mathbf{q}_{T}))$.

For unary entity labels, we obtain

$\mathbb{P}(c|s,t)=\mathrm{softmax}^{\beta}_{c}(\mathbf{n}_{C})$ (12)

with $\mathbf{n}_{C}=\mathbf{a}^{\top}_{c}\mathrm{sig}(\mathbf{q}_{S})$. For
the object, we obtain

$\mathbb{P}(o|s,t)=\mathrm{softmax}^{\beta}_{o}(\mathbf{n}_{O})$ (13)

where
$\mathbf{n}_{O}=\mathbf{a}_{o}^{\top}\mathrm{sig}(\mathbf{g}(\mathbf{q}_{S}))$
and $\mathbf{q}_{S}=\mathbf{a}_{s}+\mathbf{g}(\mathbf{q}_{T})$. For binary
labels, we get

$\mathbb{P}(p|s,o,t)=\mathrm{softmax}^{\beta}_{p}(\mathbf{n}_{P})$ (14)

with
$\mathbf{n}_{P}=\mathbf{a}_{p}^{\top}\mathrm{sig}(\mathbf{g}(\mathbf{q}_{O}))$
and $\mathbf{q}_{O}=\mathbf{a}_{o}+\mathbf{g}(\mathbf{q}_{S})$.

For Boolean variables and unary statement, we obtain (cf. Equation 1)

$\mathbb{E}(Y_{s,\textit{hA},c,t})=\mathrm{sig}(\mathbf{n}_{C})$ (15)

and for binary statements,

$\mathbb{E}(Y_{s,p,o,t})=\mathrm{sig}(\mathbf{n}_{P}).$ (16)

Essentially, we replace the softmax activation function with the sig
activation function. This renormalization is discussed in Appendix B.

For the semantic memory model (i.e., the pre-observation model), we use
$\mathbf{a}_{t}\leftarrow\mathbf{\bar{a}}$. Also, we input $s$.

Here, $\mathbf{g}(\cdot)$ is a nonlinear function. In our approach, we use

$\mathbf{g}(\mathbf{q})=W\textrm{sig}(B\textrm{sig}(V\textrm{sig}(\mathbf{q})))$

which performs computations in the hidden layer ${h}$ of the recurrent
network. $W,B,V$ are learned matrices. Also,
$\mathrm{sig}\left(x\right)=1/(1+\exp(-x))$ is the logistic function and

$\mathrm{softmax}_{i}^{\beta}\left(\mathbf{x}\right)=\frac{\exp\beta
x_{i}}{\sum_{i^{\prime}}\exp\beta x_{i^{\prime}}}.$

Then, the post-activation vector
$\mathrm{softmax}^{\beta}\left(\mathbf{x}\right)$ (without the subscript) is
the column vector of all
$\\{\mathrm{softmax}^{\beta}_{i}\left(\mathbf{x}\right)\\}_{i}$. Here,
$\beta\geq 0$ is an inverse-temperature parameter and can be used for making
the response more or less focused. $\mathbf{x}$ is the vector of pre-
activations.

###  4.5 Discussion on the BTN

An important property of the BTN is that in the algorithmic implementation and
at any iteration step, the representation vectors have the dimension of the
embedding vector and, for example, cannot represent the concatenation of two
embeddings. We can relate this to brain function. Since the representation
layer might occupy a significant portion of the brain, leaving no space for a
second concurrent representation, we call this property the “one-brain
hypothesis” (discussed in more detail in Section 8.6). Also, due to the
nonlinearity of the function $\mathbf{g}(\cdot)$, one breaks the symmetry
between subject and object embedding. It is clear which entity is the subject
and which one is the object, and the agent can distinguish between (Sparky,
ownedBy, Jack) and (Jack, ownedBy, Sparky). Thus, proper role labeling is
performed. Halford et al. (2014) call this “structural alignment.” We did not
select more standard models for several reasons. First, most tensor
factorizations (Hackbusch, 2012), such as CPD, the Tucker decomposition, the
tensor train, and RESCAL (Nickel et al., 2011), require an excessive
multiplication of factors; multiplication is an operation that is not easily
implemented in biological hardware. Second, standard tensor networks are
functions of several embedding vectors that would need to be presented
concurrently; this would violate our one-brain hypothesis. By representing
functions of functions, our model is compositional (in the sense of Poggio et
al. (2020)), which is a property that is used to explain the superior
performance of deep architectures. Other forms of compositionality are
discussed in Section 8.

###  4.6 BTN Algorithm for Perception

We consider the following setting: At a new episodic instance $t^{\prime}$,
the agent encounters a new $\textit{scene}_{t^{\prime}}$. Then a bounding box
$\textit{BB}_{\textit{sub}}$ is segmented, whose content describes a visual
entity $s^{\prime}$. The visual entity $s^{\prime}$ might be a known entity,
or it might be a novel entity, not yet known to the agent. The agent might
also detect a second visual object $o^{\prime}$ with bounding box
$\textit{BB}_{\textit{obj}}$ in the scene and might be interested in its
relationship to $s^{\prime}$. The content of a third bounding box
$\textit{BB}_{\textit{pred}}$, typically encompassing the subject and object
bounding boxes, describes the predicate. The goal is now to produce statements
that are likely true, considering the context of the scene.

We get for the episodic instance

$\mathbb{P}(t^{\prime}=t|\textit{scene}_{t^{\prime}})=\mathrm{softmax}^{\beta}_{t}(\mathbf{n}_{T})$
(17)

where
$\mathbf{n}_{T}=\mathbf{a}_{t}^{\top}\textrm{sig}(\mathbf{f}(\textit{scene}_{t^{\prime}}))$.

We obtain for the subject (cf. Equation 11)

$\mathbb{P}(s^{\prime}=s|{t^{\prime}=t},\textit{BB}_{sub},\textit{scene}_{t^{\prime}})=\mathrm{softmax}^{\beta}_{s}(\mathbf{n}_{S})$

with
$\mathbf{n}_{S}=\mathbf{a}_{s}^{\top}\textrm{sig}\left(\mathbf{f}(\textit{BB}_{sub})+\mathbf{g}(\mathbf{q}_{T})\right)$
and $\mathbf{q}_{T}=\mathbf{a}_{t}+\mathbf{f}(\textit{scene}_{t^{\prime}})$.

For unary entity labels, we obtain

$\mathbb{P}(c^{\prime}=c|s^{\prime}=s,{t^{\prime}=t},\textit{BB}_{\textit{sub}},\textit{scene}_{t^{\prime}})=\mathrm{softmax}^{\beta}_{c}(\mathbf{n}_{C})$

with $\mathbf{n}_{C}=\mathbf{{a}}_{c}^{\top}\textrm{sig}(\mathbf{q}_{S})$.

We obtain for the object (cf. Equation 13)

$\mathbb{P}(o^{\prime}=o|s^{\prime}=s,t^{\prime}=t,\textit{BB}_{sub},\textit{BB}_{obj},\textit{scene}_{t^{\prime}})=\mathrm{softmax}^{\beta}_{o}(\mathbf{q}_{O})$

with
$\mathbf{q}_{O}=\mathbf{a}_{o}^{\top}\textrm{sig}(\mathbf{f}(\textit{BB}_{obj})+\mathbf{g}(\mathbf{q}_{S}))$
and
$\mathbf{q}_{S}=\mathbf{a}_{s}+\mathbf{f}(\textit{BB}_{\textit{sub}})+\mathbf{g}(\mathbf{q}_{T})$.

For binary labels, we get

$\mathbb{P}(p^{\prime}=p|o^{\prime}=o,s^{\prime}=s,t^{\prime}=t,\textit{BB}_{\textit{sub}},\textit{BB}_{\textit{obj}},\textit{BB}_{\textit{pred}},\textit{scene}_{t^{\prime}})=\mathrm{softmax}^{\beta}_{p}(\mathbf{n}_{P})$

with
$\mathbf{n}_{P}=\mathbf{a}_{p}^{\top}\textrm{sig}(\mathbf{f}\left(\textit{BB}_{\textit{pred}}\right)+\mathbf{g}(\mathbf{q}_{O}))$
and
$\mathbf{q}_{O}=\mathbf{a}_{o}+\mathbf{f}(\textit{BB}_{\textit{obj}})+\mathbf{g}(\mathbf{q}_{S})$.
For Boolean variables and for unary statements, we obtain

$\mathbb{E}(Y_{s^{\prime},\textit{hA},c,t^{\prime}}|s^{\prime}=s,{t^{\prime}=t},\textit{BB}_{\textit{sub}},\textit{scene}_{t^{\prime}})=\mathrm{sig}(\mathbf{n}_{C})$
(18)

and for binary statements

$\mathbb{E}(Y_{s^{\prime},p,o^{\prime},t^{\prime}}|o^{\prime}=o,s^{\prime}=s,t^{\prime}=t,\textit{BB}_{\textit{sub}},\textit{BB}_{\textit{obj}},\textit{BB}_{\textit{pred}},\textit{scene}_{t^{\prime}})=\mathrm{sig}(\mathbf{n}_{P}).$
(19)

Here, $\mathbf{f}(\cdot)$ is a representation vector derived from visual
inputs realized by a deep convolutional neural network (DCNN, see Section 6).

##  5 Algorithmic Implementation and the Attention Approximations

1 switch  _Perception_ do

Input: scene, $\textit{BB}_{\textit{sub}}$, $\textit{BB}_{\textit{obj}}$,
$\textit{BB}_{\textit{pred}}$; $u=1$

2

3 end switch

4switch  _Episodic Memory_ do

Input: $t_{\textit{in}}^{*}$; $u=0$

5

6 end switch

Output: $t^{*}$, $s^{*}$, $c^{*}$, $o^{*}$, $p^{*}$

7 $\mathbf{h}=\mathbf{0}$ $\triangleright$ Alternatively, $\mathbf{h}$ is
inherited from past decoding

8

9$\mathbf{\tilde{q}}_{T}\leftarrow u\mathbf{f}(\textit{scene})$
$\triangleright$ Representation of overall scene

10

11switch  _Perception_ do

12

13 $\forall
t\in\mathcal{T}:n_{T(t)}\leftarrow\mathbf{a}_{t}^{\top}\mathrm{sig}(\mathbf{\tilde{q}}_{T})$

14 Sample $t^{*}\sim\mathrm{softmax}^{\beta}\left(\mathbf{n}_{T}\right)$
$\triangleright$ Sample a past episodic

15

16 end switch

17switch  _Episodic Memory_ do

18 $t^{*}=t_{\textit{in}}^{*}$ $\triangleright$ Input for episodic memory

19

20 end switch

21

22$\mathbf{q}_{T}\leftarrow\mathbf{\tilde{q}}_{T}+\mathbf{{a}}_{t^{*}}$

23 $\forall
c\in\mathcal{C}:n_{C(c)}\leftarrow\mathbf{a}_{c}^{\top}\mathrm{sig}(\mathbf{q}_{T})$

24

25$\mathbf{h}\leftarrow
B\mathrm{sig}[\mathrm{sig}(\mathbf{h})+V\mathrm{sig}(\mathbf{q}_{T})]$

26

27$\mathbf{\tilde{q}}_{S}\leftarrow
u\mathbf{f}(\textit{BB}_{\textit{sub}})+W\mathrm{sig}(\mathbf{h})$
$\triangleright$ $\mathbf{g}(\cdot)=W\mathrm{sig}(\mathbf{h})$

28

29$\forall
s\in\mathcal{C}:{{n}}_{S}(s)\leftarrow\mathbf{a}_{s}^{\top}\mathrm{sig}(\mathbf{{\tilde{q}}}_{S})$

30 Sample $s^{*}\sim\mathrm{softmax}^{\beta}(\mathbf{{n}}_{S})$
$\triangleright$ Sample a subject entity

31 $\mathbf{q}_{S}\leftarrow\mathbf{\tilde{q}}_{S}+\mathbf{{a}}_{s^{*}}$

32

33$\forall
c\in\mathcal{C}:n_{C}(c)\leftarrow\mathbf{a}_{c}^{\top}\mathrm{sig}(\mathbf{q}_{S})$

34Sample $c^{*}\sim\mathrm{softmax}^{\beta}\left(\mathbf{{n}}_{C}\right)$
$\triangleright$ Sample a unary label for the subject

35

36$\mathbf{h}\leftarrow
B\mathrm{sig}[\mathrm{sig}(\mathbf{h})+V\mathrm{sig}(\mathbf{q}_{S})]$

37

38$\mathbf{\tilde{q}}_{O}\leftarrow
u\mathbf{f}(\textit{BB}_{\textit{obj}})+W\mathrm{sig}(\mathbf{h})$

39

40$\forall
o\in\mathcal{C}:{{n}}_{O}(o)\leftarrow\mathbf{a}_{o}^{\top}\mathrm{sig}(\mathbf{{\tilde{q}}}_{O})$

41 Sample $o^{*}\sim\mathrm{softmax}^{\beta}\left(\mathbf{{n}}_{O}\right)$
$\triangleright$ Sample an object entity

42 $\mathbf{q}_{O}\leftarrow\mathbf{\tilde{q}}_{O}+\mathbf{a}_{o^{*}}$

43

44$\mathbf{h}\leftarrow
B\mathrm{sig}[\mathrm{sig}(\mathbf{h})+V\mathrm{sig}(\mathbf{q}_{O})]$

45

46$\mathbf{\tilde{q}}_{P}\leftarrow
u\mathbf{f}(\textit{BB}_{\textit{pred}})+W\mathrm{sig}(\mathbf{h})$

47

48$\mathbf{q}_{P}\leftarrow\mathbf{\tilde{q}}_{P}$

49 $\forall
p\in\mathcal{P}^{B}:n_{P}(p)\leftarrow\mathbf{a}_{p}^{\top}\mathrm{sig}(\mathbf{q}_{P})$

50 Sample $p^{*}\sim\mathrm{softmax}^{\beta}(\mathbf{{n}}_{P})$
$\triangleright$ Sample a binary label

51

return _$t^{*}$ , $s^{*}$, $c^{*}$, $o^{*}$, $p^{*}$ _

Algorithm 1 The BTN for Perception and Episodic Memory.

###  5.1 Algorithmic Implementation by Stochastic Sampling

Figure 2 illustrates the unfolded processing steps of the architecture shown
in Figure 1. Algorithm 1 describes the processing steps implementing the
Equations in the previous sections by generating samples from the conditional
probabilities. The main difference between episodic memory and perception is,
that, in the latter, visual inputs from the contents of the bounding boxes are
integrated. The algorithm outputs $t^{*}$, $s^{*}$, $c^{*}$, $o^{*}$, $p^{*}$
from which we can form triples of the form $(s^{\prime},\textit{hA},c^{*})$
and $(s^{\prime},p^{*},o^{\prime})$, but also $(s^{*},\textit{hA},c^{*})$ and
$(s^{*},p^{*},o^{*})$. Here, for example, $s^{\prime}$ is an entity in the
scene, and $s^{*}$ is an entity with a permanent representation in the index
layer. In perception, the episodic instance $t^{*}$ would be the index of a
similar episode in the past.

In Figure 2, we indicate that we can also sample a unary label for the
complete scene or episode and for the predicate, which we do not actually do
in the experiment. We also indicate additional processing steps for
calculating embeddings. For example,
$\mathbf{\ddot{q}}_{S}\leftarrow\mathbf{q}_{S}+\mathbf{{a}}_{c^{*}}$ is the
embedding of the sentence ($s^{*}$, hA, $c^{*}$) in the context of the scene.
Whereas concepts, predicates, and episodic instances have static embeddings
realized as connection weights linking the index layer and the representation
layer, the embedding of a sentence is dynamic.

In Algorithm 1, we explicitly introduce the latent vector $\mathbf{h}$
modelling dynamic context, supporting the implementation of the nonlinear
function $\mathbf{g}(\cdot)$. Table 2 illustrates the generation of activated
indices (“firing indices or neurons”) in different operational modes.

The algorithm for semantic memory (i.e., the pre-observation model) is
identical to the one for episodic memory, only that
$\mathbf{a}_{t^{*}}\leftarrow\mathbf{\bar{a}}$. One might also specify
$s^{*}\leftarrow s$ and not sample it.

Sampling is a unique, maybe temporary, commitment to a concept. In a
biological interpretation, a single winning unit (neuron) fires. Even a single
or a few spikes can be sufficient for communicating the winning concept to
later processing. Since sampling commits unique indices $t^{*}$ and $s^{*}$,
this allows the association of semantic and episodic memory experiences to the
observation.

###  5.2 Embedded Symbolic Reasoning by Chaining

As indicated by the four groups of dots in Figure 2, unary decoding can
continue during the time of operation of the dynamic context layer. For
example, if $\mathbf{\ddot{q}}_{S}$ represents the triples (Sparky, hA, Dog)
and the agent has learned the generalized statement (Dog, hA, Mammal), then it
is likely that the index for Mammal fires next. Thus the sequential index
pattern Sparky, Dog, Mammal is interpreted by the agent that Sparky is a dog
and a mammal. This form of embedded symbolic reasoning, as further discussed
in Section 8.3, is based on generalized statements discussed in Section 3.8.

###  5.3 Sampling and the Attention Approximation

For episodic and semantic memory, we need to generate triples of the form
($s^{*}$, hA, $c^{*}$), whereas to label entities in perception, we only need:
($s^{\prime}$, hA, $c^{*}$), that is, $s^{\prime}$, the subject in the scene,
does not need to be identified as a stored entity $s^{*}$. For example, if the
agent is attacked by a bear, it does not care what the name of the bear might
be; it might be a bear not yet known. Our approach is motivated by the
computational attention approach used in deep learning (Vaswani et al., 2017).
It can be related to the concept of internal attention in cognition to
distinguish it from external attention, for example, the attention to a
particular perceptual entity. With parallel hardware (e.g., brainware) the
sampling process can be replaced by a computational attention approximation,
which can be executed in parallel and does not commit to a particular $s^{*}$.

The semantic attention (SA) approximation for the subject replaces line 1 in
Algorithm 1 with

$\mathbf{q}_{S}\leftarrow\mathbf{\tilde{q}}_{S}+A\;\mathrm{softmax}^{\beta}(A^{\top}\textrm{sig}(\mathbf{\tilde{q}}_{S})).$

In matrix $A$ we only consider the columns relating to entities. In terms of
standard attention, as defined by (Weston et al., 2014; Sukhbaatar et al.,
2015; Vaswani et al., 2017), the argument of the softmax calculates thescore
vector, the softmax output is the alignment vector, the multiplication with
matrix $A$ calculates the context vector, the columns of $A$ are the key
vectors and the value vectors, and $\textrm{sig}(\mathbf{\tilde{q}}_{S})$ is
the query. The main differences to the standard approaches to attention is
that, in our approach, key vectors and value vectors are stored semantic
embeddings and the first term on the right is the pre-activation instead of
the post-activation; processing several bounding boxes from the same image in
parallel, as it is done in visual attention, for example, in (Koner et al.,
2020), would violate our one-brain hypothesis from Section 8.6. In the
sampling mode of operation, a symbolic unit in the index layer competes with
other units in the same layer to be activated. In contrast, in our attention
approximation, the index layer acts as a standard neural network layer
(without sampling) with softmax activation (i.e., the sum in the equation) and
pre-activation skip connections ($\mathbf{\tilde{q}}_{S}$ in the equation).

In training the model, and also in the attention approximation, we use
$\beta=1$. With $\beta=1$, predictive samples reflect the predictive
uncertainty in prediction. In the actual sampling experiments, we set the
inverse-temperature parameter $\beta\rightarrow\infty$, effectively taking the
concept with the highest probability. We call this winner-take-all sampling.
If there were a dominating dimension or if we would set
$\beta\rightarrow\infty$ in the attention approximation, the attention
approximation would become identical to winner-take-all sampling.

Similarly, for the object we replace line 1 with

$\mathbf{q}_{O}\leftarrow\mathbf{\tilde{q}}_{O}+A\;\mathrm{softmax}^{\beta}(A^{\top}\textrm{sig}(\mathbf{\tilde{q}}_{O})).$

In matrix $A$ we only consider the columns relating to entities. Attention can
also be applied to episodes in perception. The episodic attention (EA) for the
episodic index replaces line 1 with

$\mathbf{q}_{T}\leftarrow\mathbf{\tilde{q}}_{T}+A\;\mathrm{softmax}^{\beta}(A^{\top}\textrm{sig}(\mathbf{\tilde{q}}_{T})).$

In matrix $A$ we only consider the columns relating to episodic instances. EA
is the default in all experiments on perception.

With no scene input,
$\mathbf{q}_{T}\leftarrow({1}/{N_{T}})\sum_{t}a_{t}\approx\bar{a}$. Thus we
can consider semantic memory also as a situation where all instance indices
are activated equally. Technically, this is a form of an empirical
hierarchical Bayesian approach where the embedding average is inherited by a
new instance.

Sequential Index Pattern | Equivalent Triple Statements | Mode
---|---|---
Sparky, Dog, Friendly, Black | (Sparky, hA, {Dog, Friendly, Black}) | SM:U
Sparky, Jack, looksAt, ownedBy |  (Sparky, {looksAt, ownedBy}, Jack) | SM:B
t, Sparky, Dog, Friendly, Black |  t: (Sparky, hA, {Dog, Friendly, Black}) | EM: U
t, Sparky, Jack, looksAt, ownedBy |  t: (Sparky, {looksAt, ownedBy}, Jack) | EM: B
Dog, Friendly, Black |  $t^{\prime},s^{\prime}$: ($s^{\prime}$, hA, {Dog, Friendly, Black}) | P-Dir/SA: U
looksAt, ownedBy |  $t^{\prime},s^{\prime},o^{\prime}$: (s’, {looksAt, ownedBy}, $o^{\prime}$) | P-Dir/SA: B
Sparky, Dog, Friendly, Black |  At $t^{\prime}$: (Sparky, hA, {Dog, Friendly, Black}) | P-Samp: U
Sparky, Jack, looksAt, ownedBy |  At $t^{\prime}$: (Sparky, {looksAt, ownedBy}, Jack) | P-Samp: B
Table 2:  The first column shows the sequence of activated indices (“firing
neurons”). The second column shows equivalent triples. The last column shows
the operational model: SM (semantic memory), EM (episodic memory), P-Dir
(direct perception), P-SA (semantic attention), and P-Samp (Perception with
sampling). We indicate that for given episodic instances, several unary labels
(U) and binary labels (B) can be generated. If we extend the knowledge graph
to also contain nodes for predicates and time instances, then, in sampling,
exactly one node is active at a time.

##  6 Perception, the Representation Layer, and the Dynamic Context Layer

In this and the following sections, we present experimental results. Here, we
focus on perception, and in the following sections on engrams, semantic
memory, reasoning, language, episodic memory, and self-supervised learning.
Intertwined with the experiments, we make the connection to cognition and
neuroscience.

There is a long tradition in cognition and neuroscience to distinguish between
anatomical brain structure, for example, the actual anatomical structure of
the biological neural network, and functional architectures of the cortex
(Friston et al., 1995; van den Heuvel and Sporns, 2013; Bassett and Sporns,
2017; Sporns, 2018; Leopold et al., 2019). We discuss the functional
architectures primarily but occasionally also consider structure.

###  6.1 Data Set

We tested our approach experimentally using an augmented version of the VRD
data set (Lu et al., 2016). In the past, this data set has been the basis for
much research on visual relationship detection. Each visual entity is labeled
as belonging to one out of 100 classes. There are 70 binary labels, with
37,993 binary statements total. We followed other work and assigned 4000
images to the training set and 1000 images to the test set. The training
images contain, overall, 26,430 bounding boxes, thus on average, 6.60 per
image.

We generated a first derived data set, VRD-E (for VRD-Entity), with additional
labels for each visual entity. First, each entity in each image obtains an
individual entity index (or name). The 26,430 bounding boxes in the training
images describe as many entity indices. Second, we used concept hierarchies
from WordNet (Fellbaum (2010); see Figure 3). Each entity is assigned exactly
one basis class (or B-Class) from VRD (e.g., Dog), one parent class (or
P-Class, for example, Mammal), and one grandparent class (or G-Class, for
example, LivingBeing). At any level, we use the default class Other for
entities that cannot be assigned to a WordNet concept. We perform subclass
reasoning in the training data and label an entity with its entity index, its
B-Class, P-Class, and G-Class.

In addition, we used pretrained attribute classifiers (Anderson et al., 2018;
Wu et al., 2019) to label visual entities using the attribute ontology shown
in Figure 3. Each visual entity obtains exactly one color (including the color
Other) and exactly one activity attribute, for example, a person can be
standing or running. We also introduce the unary labels Young and Old, which
are randomly assigned, such that these can only be predicted for test entities
that already occurred in training, not for novel entities.

Furthermore, we introduce the nonvisual, or hidden, unary label Dangerous to
all living things and Harmless to all nonliving things. We do not use these
labels in perceptual training; we use them instead to demonstrate how semantic
memory can supplement nonvisual labels.

In summary, every visual entity receives one entity index and, in addition,
seven positive unary labels— for example, Entity=Sparky, B-Class=Dog,
P-Class=Mammal, G-Class=LivingBeing, Age=Young, Color=Black, Activity =
Standing, Risk=Harmless.

Figure 3: Ontologies. (Top) Class ontology. (Bottom) Attribute ontology.

Based on the VRD-E data set, we generate the VRD-EX data set. Here, we distort
each image in the training data set, which generates another 4000 training
images44footnotemark: 4. We used an open library to distort the image (Jung et
al., 2020). To obtain a distorted image, we apply a sequence of
transformations, including translation, rotation, shearing, and horizontal
flipping of the original image (Bloice et al., 2017). Each transformation is
associated with a probability of actually using it. When the operation has
coefficients, such as the displacement of translation, a random value within a
reasonable range is generated. Thus VRD-EX has 8000 training images. Then we
perform another distortion on each original image and create a set of another
4000 test images.555Due to distortion, some objects and images are discarded,
resulting in a reduced number of samples. Note that in these new 4000 test
images, every visual entity has already occurred in the training set twice
(See Figure 4).

In addition to the visual concepts, we introduce nonvisual entities, which do
not occur in any image. In the experiments, we relate visual entities to those
hidden entities, for example, by the predicate ownedBy or the predicate
lovedBy. For example, each visual dog is owned by a person who is not in any
scene. Table 3 shows the overall statistics.

Figure 4: Generation of VRD-EX images. (Left) An original VRD image. (Center)
A distorted VRD-EX image that is assigned to the training set. (Right) A
distorted VRD-EX image that is assigned to the test set.

data set | Training | Test | #BB | #VisEnt | #BinStat | # Attr/Ent
---|---|---|---|---|---|---
| Images | Images | Train | Train | Train | Train
VRD (Lu et al., 2016) | 4000 | 1000 | 26430 | 26430 | 30355 | 1
VRD-Entity | 4000 | 1000 | 26430 | 26430 | 30355 | 8
VRD-EXtended | 7737 | 3753 | 50910 | 26430 | 59095 | 8

Table 3: Statistics of the data sets VRD-E and VRD-EX. Attr/Ent stands for the
average number of unary labels per episodic instance. Overall, 15% of the
labels are Others.

Training was performed using the Adam optimizer (Kingma and Ba, 2014). For
evaluation, we consider top-1 accuracy for unary labels and Hits@k for binary
labels and sampling of entities. Hits@k is the fraction of correct entities
that appear in the top-k-ranked entities.

###  6.2 DCNN

As discussed, we use a deep convolutional neural network (DCNN) for the
mapping $\mathbf{f}(\cdot)$ from scene and bounding box content to the
representation layer. Kriegeskorte and Douglas (2018) and others have
discussed how DCNNs might represent functional modules in the brain. This is
not the topic of this article. Our approach is object-centric. The specific
DCNN architecture we use is the VGG-19 architecture (Simonyan and Zisserman,
2014), pre-trained on ImageNet (Russakovsky et al., 2015) and fine-tuned to
our data. The VGG-19 architecture constitutes the backbone layer of the Faster
R-CNN (Ren et al., 2015), which we employ to produce a set of bounding boxes
as output, where each bounding box contains an object. (For more details, see
the Appendix E.) In all of our experiments, we used rank $r=4096$ as the
dimension of the representation layer.

###  6.3 Direct Perception

In our approach, four bounding boxes are analysed: first, a bounding box
describing the complete scene; second, a bounding box describing the subject
entity; third, a bounding box describing the object entity; and finally, a
bounding box describing the predicate. In direct perception, we assume that
labels are independently generated by perception, purely based on bounding box
content, and entirely in bottom-up mode. There are connections from the
representation layer to the index layer but not in the opposite direction.
Also, there are no connections between the representation layer and the
dynamic context layer.

From a perceptional view, this approach exploits neither dependencies between
labels for the same bounding box (that a detected Sparky is known to be a
black dog), nor dependencies between labels for the different bounding boxes.
Direct perception requires concept indices, together with their associated
embedding vectors, which become the connection weights from the representation
layer to the index layer. Without connections from the index layer back to the
representation layer, no episodic memory or semantic memory can be formed. In
direct perception, it is unclear how the overall brain is informed about
winning or likely concept labels. This is one of the issues addressed in BTN
perception, discussed next.

Table 5 shows results for unary labels on direct perception (rows labeled
P-Direct). The results on VRD-E are pretty competitive. The performance is
improved in VRD-EX, likely due to some form of memorization by overfitting
since different views on the same entities occur in the training set and the
test set. The results on the prediction of binary labels (see Table 4) are not
very good; as expected, and a well-known result from scene graph analysis,
information on the subject and the object bounding boxes is required for
binary label prediction.

###  6.4 BTN Perception

In perception with the BTN, we add connections from the index layer back to
the representation layer. We also connect the representation layer with the
dynamic context layer. This has several important benefits.

First, in a top-down mode of operation, the representation layer, and thus the
whole brain, is informed about which concepts are detected in the scene. Thus,
if an entity is labeled as being Dangerous or recognized as being Jack, this
information is communicated to the whole brain. This pattern completion
process is a property typically discussed in the context of associative
memories. If this affects earlier processing layers, this process is referred
to as embodiment. Some theories on embodiment assume, as the name indicates,
that thought and language influence bodily states, or at least the brain
regions mapping directly to bodily states (Lakoff et al., 1999).

Second, labels would get biased: If Sparky is detected in the scene, it will
bias the unary labeling toward the unary label Black if this is known from
semantic memory to be true.

Third, by connecting the representation layer with the dynamic context layer,
we obtain a memory that can transport information from the analysis of one
bounding box to another. This change of attention from one bounding box to
another might be associated with saccadic eye movement, where the dynamic
context layer stores information during the transition phase. The dynamic
context layer is crucial for evaluating triples involving binary labels.

Finally, the bidirectional connectivity between index layer and representation
layer enables the formation of episodic and semantic memory and can enrich
perception with information from either, as will be discussed in the following
sections.

One can think of perception as a triple-generating language model in the
context of visual inputs. Through the top-down information flow via the
representation layer, the whole brain is kept up to date about scene content,
about which concepts have been detected in the scene and which statements
describe scene content.

The representation layer reflects the context. For example, if Sparky is
detected in the subject bounding box, then $\mathbf{q}_{S}$ is an embedding of
Sparky in the context of the scene. If Sparky in the scene is labeled as
Friendly, then $\mathbf{\ddot{q}}_{S}$ would be the embedding of the
statements (Sparky, hA, Friendly) in the context of the scene. If Sparky looks
at Jack in the scene and this is detected, then $\mathbf{q}_{P}$ would be the
embedding of the statement (Sparky, looksAt, Jack) in the scene’s context.

###  6.5 Experiments with BTN Perception

In all experiments, we use episodic attention (EA): a sampling of a past
episodic instance $t^{*}$ is used only to evoke a related episodic memory, as
discussed in Section 9.

Experiments with semantic attention are labeled P-SA where there is no
commitment to a specific $s^{*}$ or a specific $o^{*}$. In semantic attention,
one is primarily interested in fast labeling rather than in associating
memories.

The results in Table 5 show that on the VRD-E data set with novel entities in
the test set, perception with semantic attention (P-SA) shows the best
results. However, direct perception (P-Direct) is quite competitive.

On the VRD-EX data set with known entities in testing, perception with
sampling, P-Samp, is best, where the algorithm could “remember” past
encounters of the same entities. Here, P-Direct is not competitive. For
P-Samp, we use winner-take-all sampling.

Table 4 shows results from binary label prediction. For the binary labeling
with novel entities on the VRD-E data, P-SA shows the best performance. The
inferior performance of P-Direct confirms that the dynamic context layer is
important for achieving good performance. We see a large improvement for the
VRD-EX data set.

|  | Binary labels
---|---|---
| Model | @10 | @1
VRD-E | P-Direct | 85.45 | 31.68
| P-Samp | 90.39 | 45.09
| P-SA | 91.33 | 46.84
VRD-EX | P-SA | 99.58 | 80.63
Table 4: Binary label prediction in perception. As the inferior results for
P-direct indicate, the dynamic context layer is essential to perform well in
binary label prediction (rows 1-3). For known entities (VRD-EX) (fourth row),
entity indices permit some memorization and improve performance. Best results
in each column are in bold.

In Table 6, we compare our model with other visual relationship detection
methods. In the methods from the literature, only binary labels are tested,
with one class label for each entity. In our framework, this would be the
decoding of generalized statements. The table demonstrates that generalized
statements (i.e., probabilistic rules) can be generalized themselves. In a
zero-shot situation, where the triple (subject-class, predicate, object-class)
never occurred in training, our approach achieved a recall score of 81.61%,
which is much better than the result from the BFM (Baier et al., 2017) with
76.05%. In summary, our approach gives competitive results overall and
superior results for zero-shot binary labeling.

Figure 5 illustrates perception with unary labels and visual and nonvisual
binary statements.

|  | Unary labels (accuracy)
---|---|---
| Model | Entity | B-Class | P-Class | G-Class | Y/O | Color | Activity | Average
VRD-E | P-Direct | - | 81.30 | 88.44 | 94.67 | 50.60 | 69.06 | 83.73 | 77.97
| P-Samp | - | 80.91 | 88.85 | 95.02 | 47.88 | 68.02 | 82.90 | 77.26
| P-SA | - | 81.80 | 89.34 | 95.44 | 49.46 | 68.93 | 83.58 | 78.09
VRD-EX | P-Direct | 92.20 | 90.87 | 93.54 | 96.47 | 76.54 | 84.49 | 92.77 | 89.56
| P-Samp | 92.81 | 96.47 | 97.49 | 98.47 | 94.58 | 94.08 | 97.59 | 95.93
| P-SA | 92.65 | 95.27 | 97.11 | 98.20 | 93.10 | 92.71 | 96.83 | 95.12

Table 5: Unary label prediction in perception. We evaluate (s’, hA, c) for the entities $s^{\prime}$ in the scene. The columns labeled “B-Class, P-Class, G-Class” are class labels, and the columns labeled “Y/O, Color, Activity” are unary labels. “Average” refers to the average over all columns. In P-Direct, there are no links from $n$ to $q$, and $q$ and $h$ are independent. P-Samp stands for the sampling approach using the index $s^{*}$ with maximum activation. P-SA uses semantic attention. On the VRD-E data set (rows 1-3), where each entity is novel, P-SA shows the best results. Not surprisingly, P-Direct is also quite competitive. On the VRD-EX data set (rows 4-6), where each entity is known, P-Samp shows the best results since it can recognize specific entities. In particular, the unary label Y/O can only be learned for already known entities. Not surprisingly, P-Direct is significantly worse since it cannot benefit from memory, although overfitting leads to better results than with the VRD-E data. The column labeled “Entity” evaluates if the right entity is recognized. Thus we evaluate ($s^{\prime}$, sameAs, s) for the entities $s^{\prime}$ in the scene. Best results in each block are in bold.  Model | ph | z-s-ph | rl | z-s-rl
---|---|---|---|---
P-SA | 24.50 | 8.55 | 93.99 | 81.61
P-Direct | 13.54 | 5.73 | 84.64 | 68.35
BFM (Baier et al., 2017) | 25.11 | 7.96 | 93.81 | 76.05
Approach in (Tresp and Ma, 2016) | 23.45 | 10.95 | 93.32 | 78.79
Table 6: Binary label prediction in perception. We compare binary labeling
with methods published in the literature using the original VRD data set.
Phrase (ph) shows the recall of binary labels, where also the extracted
bounding box contents are evaluated. The binary (relationship) label (rl)
shows the recall of the binary label given the ground truth class of subject
and object. z-s-ph and z-s-rl denote zero-shot performance for triples that
did not occur in the training set. Our proposed model, P-SA (first row), is
superior in the last task. The much better performance of P-SA compared to
P-Direct (second row) demonstrates the importance of the dynamic context
layer. The third and fourth rows show results from approaches published in the
literature. Best results in each block are in bold.  Figure 5:  Illustration
of perception with known entities (VRD-EX). The left bounding box is
identified (sampled) as $s^{\prime}=s^{*}=24470=\textit{Sparky}$. Highly
ranked unary labels are: Dog, Mammal, LivingBeing, Young, White,
OtherActivity. Highly ranked unary labels for the second bounding box
($24471$) are: Bench, Furniture, NonLivingBeing, Old, OtherColor,
OtherActivity. Sampled binary statements are: (Dog, sitsOn, Bench),
(LivingBeing, on, Furniture), (Mammal, sitsOn, Old), (White, sitsOn, Bench).
We also indicate how semantic memory can support perception by adding binary
statements to object entities, not in the scene: (Sparky, ownedBy,
Jack(26675)), (Sparky, lovedBy, Mary(26676)), where Jack and Mary are not on
the scene, but in the agent’s semantic memory. The semantic memory experience
is further discussed in Section 7.

###  6.6 Representation Layer

In our model, $\mathbf{q}$, that is, the activation of the representation
layer, represents the cognitive brain state. In cognitive neuroscience, the
representation layer is referred to as “mental canvas” or the “global
workspace,” enabling a global information exchange. It is also known as
“theater of the brain,” “communication platform,” “communication bus,” or
“blackboard.” It is assumed to have a distributed representation involving
large parts of the brain. (Binder and Desai, 2011) states: “The neural systems
specialized for storage and retrieval of semantic knowledge are widespread and
occupy a large proportion of the cortex in the human brain.” During perception
or memory recall, the representation layer integrates information and then
makes this information available to the brain as a whole. In particular, the
embedding vector of the episodic experience, that is, $\mathbf{a}_{t}$,
represents a holistic, integrative view of the cognitive state at that
instance.

Since the world of an agent mostly changes smoothly, one can develop models to
forecast embedding vectors of future instances (Tresp et al., 2015; Han et
al., 2020b). The brain’s cognitive state is, to a large degree, determined by
perception, which might explain a personal feeling of personal instability and
sensitivity to external influences, whereas other individuals, represented by
their embedding vectors, appear stable and slowly changing.

###  6.7 Dynamic Context Layer

To really capture the content of a scene, it is essential to understand the
relationships between the scene entities; this requires an additional storage
facility since, following our one-brain hypothesis, the brain possesses only
exactly one global representation layer (see the discussion in Section 8.6).
As illustrated in Figures 1 and 2, we propose that this functionality is
represented by the dynamic context layer ${h}$, which represents the state of
some of the brain modules not explicitly discussed in this article. It
receives input from the representation layer, processes that information as a
recurrent neural network, and then returns the processed information to the
representation layer, which starts processing visual input from a subsequent
bounding box. Thus, the dynamic context layer stores and processes information
between saccadic eye movements. The dynamic context layer might also involve
higher brain regions, like working memory.

Figure 6 shows the Pearson correlation between activities of units in the
dynamic context layer. It shows a certain clustering structure, as exhibited
in recent studies on functional and structural brain connectivity (e.g.,
Figure 2 in Pope et al. (2021)). In the brain, such a block structure is a
signature of a small-world architecture. Since it is not clear what the
dynamic context layer in our model actually represents in the brain, we
refrain from a deeper discussion of this apparent similarity.

The dynamic context layer also represents internal mental states that might
not be driven directly by perception and memory. In neuroscience, the default
network is assumed to be active when a person is not focused on the outside
world; instead, the brain is at wakeful rest, such as during daydreaming or
mind-wandering (Gazzaniga et al., 2013). Perception, memory recall, the
initiation of activity, and other causes can induce the brain to leave the
default state. In the default state of our model, the brain’s representation
layer interacts with the dynamic context layer, without being driven by
perception.

Figure 6: Pearson correlation between units in the dynamic context layer,
followed by spectral clustering to order columns and rows.

##  7 Semantic Memory Engrams, Semantic Decoding, and Multimodality

###  7.1 Background on Semantic Memory

According to (Tulving, 1985), the semantic memory experience is independent of
a particular episodic experience. It developed out of perception as an
emerging property where semantic enrichment became independent of perceptual
input. Thus, in the transition from episodic memory to semantic memory,
provenance is lost. It is the longest-lasting and most durable memory since it
models the stable statistics in the world. It aggregates information and is a
dictionary view of the agent’s life experience. Our approach explains the
remarkable similarity between episodic and semantic memory: Semantic memory
models the episodic memory of a future instance. Whereas an episodic memory
experience requires the activation of the embedding vector of the
corresponding instance in the representation layer, a semantic memory
experience requires the activation of the semantic memory representation
$\mathbf{\bar{a}}$. Semantic memory can also be realized by the activation of
all instance indices in the episodic attention approximation (see Section
5.3). Another alternative is represented in (Tresp et al., 2017b) where
semantic memory is generated by averaging in embedding space. But that
approach is restricted to multilinear models.

###  7.2 Neuroscience Perspective: Semantic Memory Engrams

Engrams are memory traces in the brain (Ralph et al., 2017). In our approach,
the combination of an index with its embedding vector would be an engram for
that concept, although we will typically only refer to the latter part as the
engram. An index is a symbol for a concept, whereas embeddings are part of an
implicit concept memory and provide symbol grounding (Harnad, 1990; Barsalou
et al., 2008).

Here, we are in agreement with several theories on semantic memory engrams
from neuroscience. For example, Binder and Desai (2011) state that semantic
memory engrams consist of both modal and amodal representations, supported by
the “gradual convergence of information throughout large regions of temporal
and inferior parietal association cortex.” The amodal representations are
described as a high-level convergence zone. In our approach, the embeddings
would be the modal distributed representations and the indices the amodal,
local, and symbolic representations.

The relationship between the index layer and the representation layer reminds
one of the hub-and-spoke model (Ralph et al., 2017). The hub is supposed to be
located in the anterior temporal lobes (ATLs), which might be where concept
indices are consolidated. The hub is connected to several different areas
(e.g., visual cortex, auditory cortex, orbitofrontal cortex), which might be
part of the biological realization of the representation layer. Other hubs
might be located in the parietal and the temporal lobe (Binder and Desai,
2011) and maybe in the frontal lobe (Tomasello et al., 2017).

Some works propose that the anatomical distinction between the representation
layer and the index layer might be blurred in the brain. One should instead
assume an “interactive continuum of hierarchically ordered neural ensembles,
supporting progressively more combinatorial and idealized representations”
(Binder and Desai, 2011).

###  7.3 Neuroscience Perspective: Indices

Indices are explicit concept representations, that is, an index is a symbol
for a concept and could be implemented functionally by a single unit. In the
sampling mode of operation, a symbolic unit in the index layer competes with
other units in the index layer. This is in contrast to the representation
layer, where the activations of the ensemble of units contribute to the
processing.

We are purposely imprecise about how exactly an index is represented
anatomically (i.e., structurally), in brainware. In the one extreme, an index
might be a single neuron, realizing localist codes, where neurons respond
highly selectively to single entities (“grandmother cells”). The other extreme
are densely distributed codes where items are encoded through the activity of
many neurons (Kumaran et al., 2016). Most researchers favor a sparse
population of cells, realizing a population code. A distributed representation
might facilitate the consolidation of new information (see Section 10).

The debate about localized representations in the brain is ongoing. Specific
concept cells have been found in the medial lobe (MTL) region of the brain.
MTL includes the hippocampus, along with the surrounding hippocampal region
consisting of the perirhinal (“what” path), parahippocampal (“where” path),
and entorhinal neocortical regions. Researchers have reported on a remarkable
subset of MTL neurons that are selectively activated by strikingly different
pictures of given individuals, landmarks, or objects and, in some cases, even
by letter strings with their names (Quiroga, 2012; Quiroga et al., 2005).
Naturally, locality of representation is probably discovered only in well-
designed experiments. In our model, an activated concept index activates many
units in the representation layer, and a unit in the representation layer in
turn activates many indices. Since index activations might change rapidly, the
general appearance might be that of a globally activated system, hiding the
locality of representation. An index is a focal point of activity, but it is
never active in isolation.

Figure 7:  t-SNE visualization of embeddings of 15 classes and randomly
selected entities from these classes. Features for the analysis are respective
embedding vectors. A dot stands for an entity, for example, a specific dog,
Sparky. The color of a dot marks the basis class of that entity. A cross
stands for a class, for example, Dog, which is labeled by the same color as
entities belonging to the class. The embedding vector of a class concept lies
within the cluster of the classes’ entities. Recall that we are not indicating
cluster centers but the embeddings that happened to be learned for the class
concepts in learning. In cognitive neuroscience, the semantic embedding space
is sometimes referred to as conceptual space, where points denote objects, and
regions denote concepts (Gärdenfors, 2016).

###  7.4 Neuroscience Perspective on Concept Grounding

Our discussion can be related to recent results from neuroscience where, in
different brain regions, maps have been discovered that code, for example, for
visual appearance, sound, and function. For example, if the concept Hammer is
activated in the index layer, it might excite brain areas indicating a typical
hammer appearance, the sound of hammering, and the required motor movement of
hammering, all represented in the biological equivalence of the representation
layer (Rueschemeyer et al., 2010). Another example is the concept Cat, which
includes the information that a cat has four legs, is furry, meows, can move,
or can be petted (Kiefer and Pulvermüller, 2012). In terms of (Binder and
Desai, 2011), the embeddings are modality specific (here: visual), and the
indices represent convergence zones. As (Dehaene, 2014) puts it: “Every
cortical site holds its own specialized piece of knowledge.”

Evidence for distributed semantic activation has also been described by (Huth
et al., 2016; de Heer et al., 2017). Both papers developed a detailed atlas of
semantic categories and their topographic organization through extensive fMRI
studies, showing the involvement of the lateral temporal cortex, the ventral
temporal cortex, the lateral parietal cortex, the medial parietal cortex, the
medial prefrontal cortex, the superior prefrontal cortex, and the inferior
prefrontal cortex (Huth et al., 2016).

Recently, it has been proposed that embeddings are context-dependent (Popp et
al., 2019). Our model suggests that concept embedding is rather stable but
that the induced activation of a concept in the representation layer is
superimposed with context information, as discussed throughout the article.
The representation layer is activated by sensory input and activated concept
indices, so the model is informed about a concept in context, even with the
connection weights between index and representation layer being fixed.

###  7.5 Semantic Memory Engram in the BTN

In perception, the embedding vectors determine which indices are activated by
the representation layer in a bottom-up operation. However, the operation is
bidirectional: An index can activate the representation layer by its
embedding, which might even activate earlier perceptual layers. We call this
top-down mode the grounding or the embodiment of that concept. Since an
embedding vector is optimized for a concept’s role in perception and memory,
it implicitly reflects all background that is known about it.

Consider Equation 18, which labels entity $s$ with unary label $c$. With no
visual inputs, $\mathbf{{a}}_{c}^{\top}\textrm{sig}(\mathbf{{a}}_{s})$ is the
contribution from semantic memory; it acts as a prior for the unary
statements. The visual input contributes the term
$\mathbf{{a}}_{c}^{\top}\textrm{sig}(\mathbf{f}(\textit{BB}_{\textit{sub}}))$,
which is the inner product of the embedding vector with high-level features
generated from visual inputs. Thus, whereas semantic memory is dependent on
the embedding vectors and does not care about the meaning of a dimension, this
meaning is provided by perception— the inner product with high-level features.
We call this the grounding of embedding vectors.

The embedding vector is a prototypical vector for that concept in a high-
dimensional representation space; the assumption is that, in this space,
distances are meaningful, and the complex mapping performed by the DCNN
performs a normalization, such that, for example, all kinds of dogs with
different shapes and appearances form a connected subspace in the semantic
embedding space. In cognitive neuroscience, the semantic embedding space is
sometimes referred to as conceptual space (Gärdenfors (2016) p. 21ff), where
points denote objects, and regions denote concepts.

Figure 7 shows an analysis of the entity embeddings, based on the t-SNE
visualization (Van der Maaten and Hinton, 2008). The plot clearly shows a
schema-like organization of concept embeddings in this two-dimensional
projection, also known as a cognitive map (Tolman, 1948).

$s^{*}$ (ID) | Unary attribute/ | Unary entity labels | Binary statements
---|---|---|---
| Class labels | ($p$: sameAs) | ($s^{*}$, p, $o^{*}$)
10830[Person] | Person, 0.96 | 10830[Person] | (10830, wears, Shirt)
| Mammal, 0.97 |  | (10830, wears, 3495[Glasses])
| LivingBeing, 0.98 |  |
| Old, 0.98 |  |
| Other, 0.98 |  |
| Walking, 0.96 |  |
Dog | Dog, 0.99 | 3537[Dog] | (Dog, on, Grass)
| Mammal, 1.0 | 602[Dog] | (Dog, behind, Person)
| LivingBeing, 1.0 | 5976[Dog] |
| Young, 0.52 |  |
| Brown 0.35 |  |
| Other 0.99 |  |
Mammal | Dog, 0.38 | 3901[Cat] | (Mammal, on, Street)
| Mammal 1.0 | 9100[Horse] |
| LivingBeing, 1.0 |  |
| Young 0.6 |  |
| Brown 0.31 |  |
| Other 0.96 |  |
Black | Person 0.24 | 9812[Bag] | (Black, on, Person)
| Other 0.43 | 3634[Keyboard] | (Black, under, Sky)
| NonLivingBeing 0.95 |  |
| Old 0.59 |  |
| Black 0.99 |  |
| Other 0.99 |  |

Table 7: Semantic memory experience with generalized statements on VRD-EX
data. The first column shows the queried $s^{*}$ (an entity, a class, or an
attribute), the input to the algorithm. The second column shows highly rated
attribute and class labels describing $s^{*}$. The third column shows highly-
ranked (sampled) entities for the sameAs predicate. The fourth column shows
binary statements. We see that person [10830] is a mammal and often wears
shirts and glasses. However, we also see that the model “explains” what the
class Dog stands for, what the class Mammal stands for, and what the attribute
Black stands for: We see that a dog is a mammal with 100% probability, is
brown with 35% probability, and is often on the grass or behind a person. We
see that if something is black, it is often a person (24%), and black entities
are often “on persons” and “under the sky.” See our discussion on generalized
statements in Section 3.8.

| Binary label | Unary labels: attributes/classes
---|---|---
Model | @10 | @1 | B-Class | P-Class | G-Class | Y/O | Color | Act.
SM-givenClass | 92.67 | 49.74 |  |  |  |  |  |
RESCAL-givenClass | 89.95 | 26.06 |  |  |  |  |  |
SM-givenEntity | 100.0 | 90.42 | 100.0 | 100.0 | 100.0 | 100.0 | 100.0 | 100.0
RESCAL-givenEntity | 100.0 | 90.12 |  |  |  |  |  |

Table 8: Semantic memory experience for models trained on the VRD-E data set. In the first two rows, we set the class labels of the subject and the object, e.g, Dog or Person, and predict binary labels. Our results are better than a state-of-the-art model (RESCAL). (Ruffinelli et al., 2019) showed in a recent study that RESCAL is competitive with state-of-the-art approaches. In the third and fourth rows, we set the entity indices (e.g., Sparky or Jack). In this memorization, the performance is boosted by the entity representations. In these experiments, we are testing memory retrieval and not generalization, which explains the high scores. Best results in each block are in bold.  | Classification Accuracy
---|---
Model | Dangerous
P-SA | 52.01
SM | 100.0
P-enriched | 98.24
Table 9:  Semantic memory experience integrated with perception. The task is
to predict the unary label Dangerous with or without semantic memory. P-SA is
the perceptual system, where the label Dangerous was not provided in training.
It can only be predicted by chance if an entity is dangerous. We trained the
label Dangerous as part of semantic memory. In SM, the semantic memory
experience is activated, which supplements the information from semantic
memory if a visual entity is dangerous. P-enriched shows that, when the
semantic memory is trained with the Dangerous label, this information is also
automatically integrated with perception, without an extra activation of a
semantic memory experience. Enrichment works well with not-perceptual
information, like social network background, which is, in a way, orthogonal to
the visual scene input and is an indication that statements become dependent
in training by the shared embeddings. The best result is in bold.

###  7.6 Structure of the Embedding Vector

All memories are implemented as embedding vectors forming the matrix $A$. In
accordance with the standard neurocognitive view, the entries in $A$
correspond to synaptic coupling strengths (Gazzaniga et al., 2013).

In our approach, we implemented symmetric connections between the units in the
index layer and the representation layer. Thus, we have connection matrices
$A$ and $A^{\top}$ in Figures 1 and 2. Although backward connections are
common in the brain, they are typically not symmetrical. We did extensive
experiments where we removed that constraint. The result was that the
performance dropped by about $1\%$ in basically all experiments, so we stayed
with symmetric connections in our work.

The representation layer is high-dimensional, although embeddings might be
sparse, and a given index only activates a small number of components of the
representation layer. Naturally, the index that represents the color Red is
likely to mostly connect to components of the representation layer that are
excited by red images. Another advantage of sparse distributed representations
(Rolls, 2016) is that this might lead to increased memory capacity (Ma et al.,
2018a). Sparsity in the embedding vectors can be achieved in technical models,
for example, by using appropriate regularization terms, like Lasso
(Tibshirani, 1996). In our experiments, when we applied Lasso on all
parameters, we obtained $70\%$ sparsity.

###  7.7 Semantic Memory Experience: Semantic Decoding and Embodiment

The engram of a concept is identical to its embedding. However, its treasure
is only discovered by the semantic decoding. A complete semantic memory
experience is not only the activation of a concept index and the activation of
the representation layer with its embedding (i.e., its grounding) but also the
subsequent decoding of the embedding into triples. For example, after
$s=\textit{Sparky}$ activates $\mathbf{q}\leftarrow\mathbf{a}_{s}$, the agent
can classify, for example, if $s$ is $c=\textit{Friendly}$, producing the
triple statement (Sparky, hA, Friendly). This exactly is going on in semantic
memory decoding when we generate the sequential index pattern Sparky,
Friendly. The decoding produces triples from the pre-observation model which
reflects the prior probabilities for triple statements.

In the dual subsymbolic view, the semantic decoding results in an embedding of
the decoded sentences: Activating the index for Sparky, for example, embodies
Sparky’s embedding vector in $\mathbf{q}_{S}$; activating the index for
Friendly then leads to a superposition of both embeddings in
$\mathbf{\ddot{q}}_{S}$ and represents Sparky but with an emphasis that Sparky
is friendly.

In a semantic memory recall, a second entity $o$ can be activated, that is,
$o$ = Jack, and then the binary label ownedBy obtains high activation,
producing the triple (Sparky, ownedBy, Jack). The subsymbolic representation
of that statement is activated in $\mathbf{q}_{P}$.

Both grounding and semantic decoding are local to the entity and might
integrate information on that entity that was collected at different episodic
instances and in different modalities. Table 8 shows that in our model,
semantic memory can realize a very precise memory recall.

Table 7 illustrates the semantic memory experience, which is triggered by an
entity, a class, or an attribute. The latter two correspond to a recall of
generalized statements (see Section 3.8). For example, it is shown that
semantic memory can recall general information on Dogs (that they are mammals
with $100\%$), Mammals (that they are dogs with $38\%$ and living beings with
100%), and the color Black (which, with 24% probability, is a property of a
person).

###  7.8 Multimodality and Social Networks

An agent can obtain data from different modalities. This could concern
subsymbolic sensory data besides vision or symbolic information from
conversations, books, movies, and other media. Multimodal data become part of
episodic memory and semantic memory and enriches perception in different ways.
First, a multimodal episode is represented by an episodic index, and by
activating that index, the data from that modality in that episode can be
retrieved. Second, semantic memory serves as a site of multimodal integration
simply because semantic memory is trained on triples from all modalities. So
multimodal data will become part of semantic memory. Third, consider
perception. We can now distinguish between visual labels —labels, which were
used in the training images like the color Black— and multimodal labels, which
are the remaining labels, like Dangerous or Rich. Table 9 shows how
multimodality directly enters in perception. We call this enriched perception,
P-enriched. The unary label Dangerous was not trained in perception but just
in semantic memory. The table shows that information from the semantic memory
is integrated into perception and episodic memory, even without activating an
explicit semantic memory experience. Activating an entity index in perception
or episodic memory subsequently activates the unary labels of that entity in
context, but also recalls background of that entity, for example, relating to
other modalities. Here it relies on semantic memory (see Section 8.1).

To further examine multimodality, we derived a social network involving all
persons in the data set by linking entities representing persons with the
predicate knows. Each person is linked with five other persons, whom we also
denote as friends. In addition, we define a social network’s episodic instance
as an instance at which the agent learns about the social contacts of one
person. Our social network data set has 4987 person entities (along with their
unary labels), 24953 knows statements, and 4987 social episodes. We refer to
this data set as VRD-S. More details on the generation of the social network
data can be found in the Appendix D.

Table 10 shows numerical results on VRD-S. Given a person of interest, the
semantic memory can recover friends (object $o^{*}$) and predict unary labels.
Table 11 illustrates episodic and semantic memory, including social network
data recall.

Model | $s^{*}$ | $o^{*}$ | Unary labels of $s^{*}$ |
---|---|---|---|---
| @10 | @1 | @10 | @1 | B-Class | P-Class | G-Class | Y/O | Color | Activity
Episodic Memory | 100 | 51.47 | 99.97 | 65.80 | 100.0 | 100.0 | 100.0 | 100.0 | 100.0 | 100.0
Semantic Memory | - | - | 97.39 | 18.25 | 100.0 | 100.0 | 100.0 | 100.0 | 100.0 | 100.0
RESCAL | - | - | 95.76 | 17.96 |  |  |  |  |  |

Table 10: Episodic and semantic memory experience for the social network data
VRD-S. The first row shows the performance of an episodic recall, with $t^{*}$
given. Columns 2 and 3 evaluate the prediction of a subject entity. For
columns 3 to 10, the subject $s^{*}$ is also given. The second row shows the
semantic memory experience where the subject $s^{*}$ is given. The unary
labels have perfect recall. Columns 3 and 4 show the performance of predicting
a friend. Since each person has five friends, the recall@1 is upper-bounded by
20%. As a comparison, we show the performance of RESCAL in the third row,
which shows slightly worse performance.
---
t* | s* | Unary labels | o* | Binary labels
2177 | 14518 | Person, Mammal, LivB | 14511 | on
[Person] | Young, Other, Sitting | [Bus] |
14515 | Building, Other, NonLivB | 14516 | has
[Building] | Old, White, Other | [Roof] |
14511 | Bus, Vehicle, NonLivB | 14512 | nextTo (-)
[Bus] | Old, Other, Other | [Road] |
6662 | 14518 | Person, Mammal, LivB | 10669 | knows
| Young, Other, Sitting |  |
18010 | Person, Mammal, LivB | 14518 | knows
| Young, Other, Other |  |
8318 | Person, Mammal, LivB | 14518 | knows
| Old, Other, Other |  |
| 14518 | Person, Mammal, LivB | 10669 | knows
Young, Other, Sitting | 25066 | knows
| 12825 (-) | knows (-)
Table 11: Episodic and semantic memory experience that includes multimodal
data (from a social network). The figure shows the original visual scene,
which is unavailable during episodic recall. The scene index of the image is
$t^{*}=2177$. The top segment shows sampled subject entities ($s^{*}$), highly
ranked unary labels ($c^{*}$), a sampled object $o^{*}$, and the top predicted
binary label ($p^{*}$). All labels are correct except for one binary label
(indicated by (-)). The second segment shows an episodic recall of a social
episode with index ($t^{*}=6662$). At that episodic instance, social network
information (binary labels knows) was provided, which is recovered in the
episodic recall. Shown are three correct triples that were recovered in the
sampling of the episodic memory. The bottom segment shows knows statements
recovered from semantic memory for $s^{*}=14518$ (thus without a recall of a
special episodic memory). Two binary statements are correct, and one is
incorrect (indicated by (-)).

##  8 Reasoning and Language

###  8.1 Embedded Reasoning and Generalization

When the states of new triples are observed, this will be reflected in changes
in the ${\underline{\underline{T}}}$ tensor, which affects
$\mathbb{P}(A|{\underline{\underline{T}}})$ and the embedding matrix $A$
(Equation 8). Thus, although triple statements are independent, given all
entries in $A$, entries in $A$ change with new data. For example,
$\mathbf{a}_{s}$ changes when new truth values of statements involving $s$
become available. Thus, if it becomes known that Sparky is a “Dog”,
$\mathbf{a}_{\textit{Sparky}}$ will be adapted and the unary statement for the
label “Mammal” will exhibit a high probability.

Episodic memory, that is, the observation model, generates samples for
observed entities in the modality of the instance. Although episodic memory
purely models the observations, there is also some degree of generalization by
exploiting correlations in observations. Semantic memory, that is, the pre-
observation model, can generate samples for all entities and modalities and
fills in the unobserved. Semantic memory can show a significant degree of
generalization. If a particular triple is often sampled in episodic memory but
not in semantic memory, this would indicate an anomaly. As an example,
consider that Sparky is a friendly dog (semantic memory), but in a particular
episode, he was agitated and barking (episodic memory), then that is
remarkable.

Perception is based on episodic attention (EA). After perceptual decoding, an
instance embedding $\mathbf{a}_{t}$ is realized, which is fitted to model the
extracted triples. Subsequently, $\mathbf{a}_{t}$ generalizes perception and
becomes the basis for episodic memory. See also the discussion in Section 10.

The mind can distinguish between what is currently being observed, what the
semantic memory fills in (by activating $\bar{t}$ and $\mathbf{\bar{a}}$), and
what an episodic memory could contribute (by activating a past $t$ and
$\mathbf{a}_{t}$).

###  8.2 Symbolic Reasoning

In contrast to embedded reasoning, symbolic reasoning solely depends on
indices and their relationships, without involving the representation layer.
It would be completely amodal, in contrast to the modal embedded reasoning. In
particular, it does not involve the adaptation of the embedding matrix $A$.
Examples would be the definitional structure of a bachelor (a male person who
is not and has never been married) and reasoning with subclass relationships
(any dog is also a mammal). Definitional examples, like the definition of a
bachelor, enter the mind of the agent via language and are crisp. They
describe how the world should be. Symbolic reasoning can also link singular
events with state changes. For example, one might specify that if someone gets
married (a singular event), the state of this person changes from single to
married. Another example is the triangle rule in social networks: if person A
and person B are known to be friends and person A is known to be a friend of
person C, then person B is also likely a friend of person C. This example
shows that symbolic reasoning is not necessarily deterministic. Descriptional
dependencies, like the triangle rule, are learned by means of observations,
are often probabilistic, and describe how the world actually is. (Halford et
al., 2014) has explored how analogical reasoning can be performed in the data
tensor, loaded in working memory. The work assigns symbolic reasoning to
System-2. One might call it “inference by contemplation.” (Nickel et al.,
2014) is an example where both embodied and symbolic reasoning are combined in
an additive model.

###  8.3 Embedded Symbolic Reasoning

Triple sentences used in symbolic reasoning have an embedding and lead to
another form of embedded reasoning. But in contrast to standard embedded
reasoning, the embedding matrix $A$ is not modified in embedded symbolic
reasoning. A simple symbolic rule would be that a dog is a mammal, that is,
$\forall s:\textit{(s, hA, Mammal)}\leftarrow\textit{(s, hA, Dog)}$. The
corresponding triple statement is $Y_{\textit{Dog, hA, Mammal}}$ is embedded
and $\mathbb{E}(Y_{\textit{Dog, hA, Mammal}})$ can be learned from data.
Section 5.2 describes how the BTN’s embedding approach can produce the
sequential index pattern Sparky, Dog, Mammal, which is interpreted as Sparky
being a Dog and a Mammal. We call this embedded symbolic reasoning. It builds
on the same BTN architecture as perception and memory. Embedded symbolic
reasoning is integrated in sampling. Assuming that, based on scene input, the
unary label Mammal is active with $60\%$ for a given visual entity, so the
agent is uncertain about this label. Then, if sampling “decodes” that the
visual entity is a Dog, embedded symbolic reasoning lifts the probability for
the label Mammal to $100\%$ (due to the learned rule that every dog is a
mammal) and the corresponding unary label is sampled as well. Embedded
symbolic reasoning can capture complex dependencies and thus can go beyond
embedded reasoning. Embedded symbolic reasoning can be executed fast since it
does not depend on a tuning of embeddings for entities or instances.

Table 7 shows generalized statements, which are the basis for embedded
symbolic reasoning. We see that (Dog, hA, Mammal) is true with $100\%$ (every
dog is a mammal) and (Mammal, hA, Dog) is true with $38\%$ (not every mammal
is a dog). Also, Table 4 shows results for generalized binary statements.

###  8.4 A Foundation for Consciousness?

The dynamic context layer might play a role in working memory. In general,
working memory is associated with decision-making and cognitive control
(Baddeley, 1992) and is necessary for keeping task-relevant information
active, as well as manipulating that information to accomplish behavioral
tasks. A modern view is that working memory is distributed across the cortex
(Buschman and Miller, 2020). There is an emerging consensus that most working
memory tasks recruit a network of the prefrontal cortex (PFC) (front-of-the-
brain hypothesis) and/or parietal areas in the prefrontal parietal network
(PPN) (back-of-the-brain hypothesis) (Seth and Bayne, 2022).

PPN activity is consistently reported in both attention and consciousness
studies (Bor and Seth, 2012). Their publication proposes that the PPN can be
viewed as a “core correlate” of consciousness. (Dehaene, 2014) defines
consciousness as “global information sharing” where information has entered
into a specific storage area that makes it available to the rest of the brain.
Koch et al. (2016) argue that the posterior hot zone (PHZ) is the minimal
neural substrate essential for conscious perception. The PHZ includes cortical
sensory areas in the parietal, temporal, and occipital lobes.

The interactions between the index layer, the representation layer, and the
dynamic context layer (which might involve working memory), might be a
foundation from which evolution eventually generated human consciousness. The
representation layer, associated with the parietal area of the brain, plays a
major role in our approach, as well as its interaction with the dynamic
context layer, which might include the working memory. The activation of the
representation layer encodes the cognitive brain state. It integrates
information and then makes this information available to the brain as a whole.
In particular, the activation of the representation layer represents a
holistic, integrative view of the agent’s cognitive state at that instance.
See also our discussion in the remaining part of this section. The role of the
memory system in consciousness is discussed in (Budson et al., 2022).

###  8.5 Serialization of Parallel Computing and the Central Bottleneck

Here are some examples of compositionality and serial processing in the BTN,
and possibly in the mind. First, perception analyses a scene that evolves in
time, which is a serial process. Second, an episode is composed of several
events. Third, an event consists of several bounding boxes, which are analysed
serially. Fourth, many triples are generated serially to decode a scene
semantically. Finally, a triple itself is represented as a sequence of indices
(see Table 2).

On the other hand, any computation involving the representation layer —in
particular, the mappings involving the DCNN— and the mappings between the
index layer and the representation layer are highly parallel.

It is this mixture of parallel and sequential processing exhibited in our
model that might also make up the brain’s operation. Consider, that in a first
step, we map $\textit{scene}_{t^{\prime}}$ to
$\mathbf{f}(\textit{scene}_{t^{\prime}})$, and this representation can be
analysed rapidly and might trigger action with a limited understanding of
scene content. Only then is bounding box content analysed in detail, which
might be supported by saccadic eye movements, and this is a slower serial
process, permitting a transition to language, as discussed later in this
section. Maybe not surprisingly, language makes thoughts explicit but at the
same time might slow down thinking.

Sequential processing is also a core concept in the theory of a global
workspace (Baars, 1997; Bor and Seth, 2012; Dehaene, 2014; Goyal et al.,
2021): (Koch, 2014) discusses that Dehaene’s workspace has extremely limited
capacity (“the central bottleneck”) and that the mind can be conscious of and
pay attention to only one or a few items or events, although these might be
quickly varying. In cognition neuroscience, the general understanding is that
parallel multitasking of cognitive tasks likely is an illusion. Even working
memory is assumed to be able to store only three to four items at a time (Awh
and Vogel, 2020). Sequential processing would also contribute to a potential
solution of the binding problem (Singer, 2001) since the decoding focuses on
concepts in a serial fashion, and associations between activities in the
representation layer and the index layer are well defined. The importance of
sampling is also recognized in (Dehaene, 2014) in the context of conscious
perception. For example, the author states that “consciousness is a slow
sampler.”

Another interesting point is that both (Dehaene, 2014) and (Koch et al., 2016)
assume mental states, well delineated from all the other states. Switching
between different interpretations is also a property of our sampling approach
if one interprets a sample as a temporary decision or an interpretation: ”It’s
a bird, or a plane, or it’s Superman, but not all of them at the same time”
(Dehaene, 2014). Dehaene discusses a similar process of “collapsing all
unconscious probabilities into a single conscious sample.” His model assumes a
“winning neural coalition”; in our approach, the embedding of an episodic
instance integrates all information available at that instance, and the
winning neural coalition would be the samples and triple sentences associated
with that embedding. The optimization of the embedding of an episode, that is,
$\mathbf{a}_{t}$, is a step that occurs after the decoding of a scene, and
thus the conscious experience associated with it might be slightly delayed
(see also Section 10.3). A current hypothesis in cognitive neuroscience is
that conscious awareness often follows after an observation or a decision is
made and serves to explain and justify but not to trigger an action (Gazzaniga
et al., 2013; Budson et al., 2022). Only some important decisions are actually
made with the help of conscious slow thinking. In the semantic decoding of our
model, the representation layer is periodically activated, which might be
reflected in neural signals and could be related to some of the neural
oscillations found in the brain. A candidate is the beta rhythm (13-35 Hz),
considered to be related to consciousness, perception, and motor behavior.
Also of relevance might be the gamma wave (25-140 Hz), which is correlated
with large-scale brain network activity and cognitive phenomena such as
working memory, attention, and perceptual grouping.

###  8.6 One-brain Hypothesis

We discuss the one-brain hypothesis in the context of two aspects. First, the
representation layer can contain a superposition of embedding vectors and
activation vectors, but it cannot represent a concatenation of two vectors.
This would, in a way, require two separate representation layers. We propose
that this is biologically not plausible, and we adhere to Dehaene’s concept of
a central bottleneck. Applied to perception, this means that the mind can only
process one bounding box content at a time.

Second, perception, episodic memory, semantic memory, and even embedded
symbolic reasoning are realized by the different operational modes of a single
architecture; thus, we do not assume different modules for these different
functions. Instead, the brain repurposes the same architecture for different
functions.

Of course, the brain does many things concurrently, and many of these
activities are hidden from conscious awareness. This does not contradict our
one-brain hypothesis, which is only concerned with the proposed operations of
perception and memory, and in particular, when they involve the representation
layer, that is, the cognitive state of the brain.

###  8.7 Cognitive Linguistics

Humans differ from other animals in their ability to express themselves
through natural language. Human language is the basis for communication but
also a means to argue and reason. Thus, an agent can tell another agent not to
leave the hideout since a bear is lurking outside, even when the bear is not
visible. Perception, episodic memory, and semantic memory are all declarative
(i.e., explicit), and humans can verbally report on either. We propose that
the generated triples in our approach are a basis from which the rich human
language might have evolved.

Leading approaches in cognition and linguistics are, first, the formal
approach, second, the connectionist approach, and third, the embodied approach
(Evans, 2012). Our work relates to all three.

Let’s consider first the formal approach. The language of thought hypothesis
assumes that mental representation has a linguistic structure, as well:
thoughts are sentences in the mind. Fodor (1975) describes the nature of
thought as possessing “language-like” or compositional structure (sometimes
referred to as mentalese). In this view, simple concepts combine
systematically (akin to the rules of grammar in language) to build thoughts.
Also, in our approach, the brain talks to itself by producing triple sentences
and their embeddings. We also agree that language offers a window into the
operation of the brain. However, in our approach, we do not follow Fodor’s
formal logic-based view: Our brain is more of a chatterbox.

Second, we follow a connectionist approach (McClelland et al., 2020), since
the BTN includes a DCNN and neural processing. However, our notions of
discrete symbolic indices, embodiment, and compositionality go beyond a more
standard connectionist approach.

Finally, we can relate to the concept of an embodied language. There is
general agreement that in a form of bottom-up processing, perception, the
state of mind, and the body influence language. The idea of an embodied
language is that in a form of top-down processing, language can influence the
perceptual path and the body (Evans, 2012). As discussed, we see our approach
as being embodied, at least as far it concerns earlier processing in the
brain. One might even consider that language takes over the mind, particularly
in memory, with no immediate visual input to be decoded. From a brain, talking
to itself, of course, there is a small step to a brain that speaks to others.
Most generated triple sentences are not transferred into language; what is
actually spoken is obviously more complex, nuanced, and sophisticated, and is
modulated, for example, by intent, social context, and cultural background.
Thus an internal triple-oriented fast speech is transferred into an external,
sophisticated slow speech. When individuals learn a new language, this might
involve the mappings between fast speech and slow speech, in addition to the
direct mapping from one language to another language. The inner fast speech
might already exist in some animals.

If the agent is the recipient of language, say, in a conversation, by reading,
or by consuming media, the acquired information could be stored as an episodic
memory instance, that is, as an episodic index with its associated embedding
vector. This knowledge then can become part of semantic memory. In our
simplistic triple-oriented conversation, the agent can learn about facts, for
example, (Sparky, hA, Dog) and generalized statements, such as Hearst patterns
(Hearst, 1992) like (Dog, hA, Mammal).

Language compresses information. Significantly different scenes might generate
very similar descriptions, demonstrating great invariance in triple
descriptions and language in general.

##  9 Episodic Memory

###  9.1 Background on Episodic Memory

Episodic memory documents the life of an agent. Tulving (1985) describes
episodic memory as a memory that, in contrast to semantic memory, requires a
recollection of a prior experience. It is considered to be the result of rapid
associative learning in that a single episode and its context become
associated and bound together and can be retrieved from memory after a single
episode. Episodic memory stores information of general and personal events
(Tulving, 1972, 1985, 2002; Gazzaniga et al., 2013) and concerns information
we “remember,” including the spatiotemporal context of events (Gluck et al.,
2013).

Some theories emphasize the sequential nature of episodic memory and the
memory process. Moscovitch et al. (2016) considers an episodic memory
experience to be an active process that involves details of the event and its
location. Sometimes the reconstruction is regarded as a Bayesian process of
reconstructing the past as accurately as possible based on available engram
information (Hemmer and Steyvers, 2009).