#  A Categorical Framework of General Intelligence

Yang Yuan  IIIS, Tsinghua University Shanghai Artificial Intelligence
Laboratory  Shanghai Qi Zhi Institute

###### Abstract

Can machines think? Since Alan Turing asked this question in 1950, nobody is
able to give a direct answer, due to the lack of solid mathematical
foundations for general intelligence. In this paper, we introduce a
categorical framework towards this goal, with two main results. First, we
investigate object representation through presheaves, introducing the notion
of self-state awareness as a categorical analogue to self-consciousness, along
with corresponding algorithms for its enforcement and evaluation. Secondly, we
extend object representation to scenario representation using diagrams and
limits, which then become building blocks for mathematical modeling,
interpretability and AI safety. As an ancillary result, our framework
introduces various categorical invariance properties that can serve as the
alignment signals for model training.

##  1 Introduction

In recent years, remarkable progress has been made in training foundation
models with enormous computational power, vast amounts of data, and gigantic
neural networks (Radford et al., , 2021; Chen et al., , 2020; Radford et al.,
, 2019; Brown et al., , 2020; Ramesh et al., , 2021, 2022; Sohl-Dickstein et
al., , 2015; Rombach et al., , 2022; He et al., , 2022). Surprisingly, despite
the impressive achievements, the internal working mechanisms of these models
remain mysterious. People seem to reach the consensus that the foundation
models are inherently black-box and uninterpretable, therefore empirical
experimentation is the only way of pushing AI forward.

While this is indeed what happened in the past decade and is analogous to how
intelligence is acquired through evolution, relying solely on empirical
experimentation without theoretical understanding can be both inefficient and
dangerous. The inefficiency arises from the fact that the progress is made
through trial and error, often guided by intuition, and the milestones are
defined indirectly based on performance on specific tasks rather than a
comprehensive understanding of intelligence itself. The potential danger stems
from the fact that nobody knows what we will get at the final destination, and
perhaps more importantly, how close we are right now. We do not even know
whether we have already created the general intelligence — maybe not yet, but
how to make such evaluations?

In this paper, we present a categorical framework of general intelligence,
which contributes to answering the following questions:

  1. Q1:

Can the model be aware of its self-state? (Section 3)

  2. Q2:

How shall the model represent complex scenarios? (Section 4)

  3. Q3:

How shall we train the model towards general intelligence? (Section 5)

It will be extremely challenging, if not impossible, to prove that our
framework is for _the_ general intelligence, given the absence of consensus on
the formal definition of general intelligence among human beings. Instead, we
take the categorical approach: we formally define all the basic elements,
state their theoretical implications, specify the algorithmic requirements,
and finally integrate all the elements into a comprehensive framework.
Therefore, even if one disagrees with our definition of general intelligence
or believes that certain crucial pieces are missing, our framework remains
relevant and applicable.

World category subcategory subcategory subcategory Planner with objectives
Sensor Actor subcategory Figure 1: Our categorical framework ? A B D A B D A B
D A B D Real world Perceived world World category ? ? Sensor
${\mathcal{F}_{\theta}}$ Sensor Sensor ${\mathcal{F}_{\theta}}$
${\mathcal{F}_{\theta}}$ Figure 2:  Case studies on three different worlds. In
the first row, the model sees a human being with a desk, but the human is
partly occluded and the desk has one corner missing. In the world category,
when the model memorizes the scene, it may complete the occluded parts with
minor adjustments on the human body, and slightly modify the shape of the
desk. In the second row, the model sees a chart representing the wealth of
Alice, Bob and David. However, the model may not accurately memorize the
information in its world category. In the third row, the model perceives the
relationships among Alice, Bob and David, e.g., whether two of them are close
friends. In its world category, such relationships might be preserved with
distortion.

Our framework is surprisingly simple, consisting of four main
components111Temporarily ignoring the categorical part, which is our main
contribution, we shall remark that the components and connections in our
framework are similar but different to the ones proposed in LeCun,  (2022), as
discussed in Section 6.: the sensor, world category, planner with objectives,
and actor, with single direction information-flow (see Figure 1). The sensor
receives multi-modal signals from the external environment, including but not
limited to text-input, video/image-input, audio-input, sense of touch, etc.
The world category perceives and comprehends the incoming signals, and updates
its internal state accordingly. The planner continuously monitors the status
of the world category and generates plans based on its objectives. Finally,
the actor executes these plans, influencing the external environment by
generating outputs such as text-output, video/image-output, audio-output,
robot-manipulation signals, etc. We elaborate the details below, starting from
three different world views.

Three different worlds exist in our framework (see Figure 2): the real world,
perceived world, and world category. The model cannot directly access the real
world222It is entirely possible that this “real world” is simulated, and we do
not make any distinctions here., and can only access the perceived world using
its sensor. For any given time $t\in\mathbb{N}$, the real world can be much
larger than the perceived world of the model as the model may only perceive a
small part of the world. Even under perceived scope of the sensor, the real
world and perceived world are not necessarily the same, because the sensor may
have limited sensory ability, be biased or contain noise. Moreover, the world
category encodes what the model understands about the world, so it can be much
larger than the perceived world. Intuitively, the perceived world is what the
model sees and feels momentarily, and the world category is what it
understands, memorizes and predicts about the world.

In order to demonstrate intelligent behaviors, the model should interact with
the external environment through its sensor and actor, so the notion of time
$t$ is important. In case the model does not have the access of the time
stamps, $t$ may instead denote the index of distinct events that are detected
by the sensor, e.g., chatGPT answering various queries from the user. For the
sake of simplicity, we do not make such distinctions and simply treat $t$ as
the time stamp.

The world category can be seen as an imaginary reconstruction of the external
world by the model through its sensor. It comprises all the people, animals,
objects, knowledge from the external environment that are perceptible by the
model through its sensor, as well as abstract representations on top of them.
More precisely, it is a function in $\mathbb{N}\rightarrow\mathbf{Cat}$,
representing a dynamic category333Readers not familiar with category theory
may check Section 2 for a basic introduction. containing various objects and
changing over time $t\in\mathbb{N}$. The sensor decides the types of
elementary objects in the world category. For example, if the model is
incapable of detecting visual signals, the world category will not contain
image objects. Moreover, if the sensor receives signals from a simulated
environment, the world category will only contain simulated knowledge, which
can differ significantly from the real world.

Object representation. For any given object $X$, the world category never
directly stores $X$, but uses neural networks to store all the relationships
between other objects and $X$ instead, which contains sufficient information
about $X$. As a notable example, when the model is able to perceive the
relationships between itself and other objects, its world category can
represent an object called “self-state” for storing such relationships. Is
maintaining the self-state equivalent to having self-consciousness? This is a
controversial question that we choose not to answer. Instead, we formally
define the notion of self-state based on category theory without bothering its
relationship to self-consciousness (Section 3.1).

Based on our definition of self-state, we introduce two algorithms for
enforcing and evaluating self-state awareness of the model. Unlike the
identification of self-consciousness, which is a binary variable denoting
whether a subject possesses self-consciousness or not, our evaluation
generates a continuous value within the interval $[0,1]$ to indicate the
degree of self-state awareness. This degree corresponds to the proportion of
all relevant relationships between the subject and other objects or tasks that
the subject is aware of. According to our definition, it appears that even
many human beings, especially the kids, may not possess perfect self-state
awareness.

Scenario representation. How shall the model represent scenarios with multiple
objects and morphisms in between? Using the language of category theory, we
can define the scenario content as a diagram, and define the scenario itself
as a projective limit over the diagram.

As we will see, using diagrams and limits for scenario representation has
various interesting consequences. For example, it makes mathematical modeling
much easier, because we may take proper abstraction of diagram, to directly
convert it into a mathematical problem. Besides, by treating the scenario
content as a diagram of the world category, the model can generate
interpretations based on its internal knowledge, not only limited to assigning
weights to the input variables like the attribution methods (Sundararajan et
al., , 2017; Lundberg and Lee, , 2017). Moreover, diagram representation of
the scenario content allows a functional approach for AI safety, by injecting
self-state into the diagram and enforcing the self-state to be human-friendly.

Invariance property as training signals. Category theory employs commutative
diagrams to characterize the equivalence of distinct computational paths,
which naturally leads to various invariance properties for the model. Unlike
supervised learning where the training objective is to fit the input data with
the correct output label, foundation models focus on learning the morphisms
between objects, and functors between categories. The invariance properties
serve as the training signals for the model to adjust itself, so that the
world category is naturally consistent.

##  2 Preliminaries

Category theory is used in almost all areas of mathematics. Here we only
introduce the necessary notions for understanding the results of our paper.
Curious readers may check Mac Lane,  (2013); Riehl,  (2017); Adámek et al.,
(1990) for a more comprehensive introduction.

###  2.1 Category basics

A category $\mathcal{W}$ has a set of objects $\mathrm{Ob}(\mathcal{W})$, and
a set of morphisms $\mathrm{Hom}_{\mathcal{W}}(X,Y)$ from $X$ to $Y$ for every
$X,Y\in\mathrm{Ob}(\mathcal{W})$. In this paper, we use “relationships” and
“morphisms” interchangeably. Given
$f\in\mathrm{Hom}_{\mathcal{W}}(X,Y),g\in\mathrm{Hom}_{\mathcal{W}}(Y,Z)$, we
define their composition as $g\circ f\in\mathrm{Hom}_{\mathcal{W}}(X,Z)$.
Notice that $\circ$ is associative, i.e., $(h\circ g)\circ f=h\circ(g\circ
f)$. For every $X\in\mathrm{Ob}(\mathcal{W})$, there exists an unique identity
morphism $\mathrm{id}_{X}\in\mathrm{Hom}_{\mathcal{W}}(X,X)$. A morphism
$f:X\rightarrow Y$ is an isomorphism if there exists $g:X\leftarrow Y$ such
that $f\circ g=\mathrm{id}_{Y}$ and $g\circ f=\mathrm{id}_{X}$. In this case,
we say $X$ and $Y$ are isomorphic and write $X\simeq Y$.

We consider a universe $\mathcal{U}$444Check Masaki Kashiwara,  (2006) for the
related definitions.. A category is a $\mathcal{U}$-category, if
$\mathrm{Hom}_{\mathcal{W}}(X,Y)$ is $\mathcal{U}$-small for any
$X,Y\in\mathrm{Ob}(\mathcal{W})$. A $\mathcal{U}$-category category
$\mathcal{W}$ is $\mathcal{U}$-small if $\mathrm{Ob}(\mathcal{W})$ is
$\mathcal{U}$-small. For simplicity, below we may not explicitly mention the
universe $\mathcal{U}$, and simply write that $\mathcal{W}$ is a category or a
small category. We define $\mathbf{Cat}$ to be the category whose objects are
small categories.

Given a category $\mathcal{W}$, we define its opposite
$\mathcal{W}^{\mathrm{op}}$ by setting
$\mathrm{Ob}(\mathcal{W}^{\mathrm{op}})=\mathrm{Ob}(\mathcal{W})$ and
$\mathrm{Hom}_{\mathcal{W}^{\mathrm{op}}}(X,Y)=\mathrm{Hom}_{\mathcal{W}}(Y,X)$.
Moreover, given
$f\in\mathrm{Hom}_{\mathcal{W}^{\mathrm{op}}}(X,Y),g\in\mathrm{Hom}_{\mathcal{W}^{\mathrm{op}}}(Y,Z)$,
the new composition is $g\mathrel{\overset{\makebox[0.0pt]{\mbox{\tiny
op}}}{\circ}}f=f\circ g\in\mathrm{Hom}_{\mathcal{W}^{\mathrm{op}}}(X,Z)$.

We define Set to be the category of sets, where the objects are sets, and
$\mathrm{Hom}_{\textbf{Set}}(X,Y)$ is the set of all functions with domain $X$
and codomain $Y$. Notice that we ignore the subtleties about the universe for
better presentation, so here just assume that Set does not contain strange
objects like a set containing all sets.

Functor is like a function between two categories. Given two categories
$\mathcal{W},\mathcal{W}^{\prime}$, a functor
$F:\mathcal{W}\rightarrow\mathcal{W}^{\prime}$ maps objects from $\mathcal{W}$
to $\mathcal{W}^{\prime}$ with
$F:\mathrm{Ob}(\mathcal{W})\rightarrow\mathrm{Ob}(\mathcal{W}^{\prime})$ and
morphisms from $\mathcal{W}$ to $\mathcal{W}^{\prime}$ with
$F:\mathrm{Hom}_{\mathcal{W}}(X,Y)\rightarrow\mathrm{Hom}_{\mathcal{W}^{\prime}}(F(X),F(Y))$
for all $X,Y\in\mathcal{W}$, so that $F$ preserves identity and composition.
Formally, we have $F(\mathrm{id}_{X})=\mathrm{id}_{F(X)}$ for all
$X\in\mathcal{W}$, and $F(g\circ f)=F(g)\circ F(f)$ for all $f:X\rightarrow
Y,g:Y\rightarrow Z$.

The morphisms of functors, also called the natural transformation, is the way
to transform the functors while preserving the structure. Given two categories
$\mathcal{W},\mathcal{W}^{\prime}$, and two functors $F_{1},F_{2}$ from
$\mathcal{W}$ to $\mathcal{W}^{\prime}$. A morphism of functors
$\theta:F_{1}\rightarrow F_{2}$ has a morphism $\theta_{X}:F_{1}(X)\rightarrow
F_{2}(X)$ for all $X\in\mathcal{W}$ such that for all $f:X\rightarrow
Y\in\mathrm{Hom}_{\mathcal{W}}(X,Y)$, we have
$\theta_{Y}(F_{1}(f)(F_{1}(X)))=F_{2}(f)(\theta_{X}(F_{1}(X)))$.

A presheaf is a functor from ${\mathcal{W}^{\mathrm{op}}}$ to Set, and
${\mathcal{W}^{\wedge}}$ is the category of presheaves. Similarly, a functor
from ${\mathcal{W}^{\mathrm{op}}}$ to $\textbf{Set}^{\mathrm{op}}$ is called a
$\textbf{Set}^{\mathrm{op}}$-valued presheaf, and ${\mathcal{W}^{\vee}}$ is
the category of $\textbf{Set}^{\mathrm{op}}$-valued presheaves. In this paper
we do not make the differentiation, and name both kinds of functors as
presheaves, and ${\mathcal{W}^{\wedge}},{\mathcal{W}^{\vee}}$ as the
categories of presheaves. Moreover, define the Yoneda functors
${h_{\mathcal{W}}}(X)\triangleq\mathrm{Hom}_{\mathcal{W}}(\cdot,X)\in{\mathcal{W}^{\wedge}}$,
and
${k_{\mathcal{W}}}(X)\triangleq\mathrm{Hom}_{\mathcal{W}}(X,\cdot)\in{\mathcal{W}^{\vee}}$.
The following lemma is fundamental.

######  Lemma 1 (Yoneda lemma).

Given $X\in\mathcal{W}$ we have,

  1. (1)

For $A\in\mathcal{W}^{\wedge}$,
$\mathrm{Hom}_{\mathcal{W}^{\wedge}}(h_{\mathcal{W}}(X),A)\simeq A(X).$

  2. (2)

For $B\in{\mathcal{W}^{\vee}}$,
$\mathrm{Hom}_{{\mathcal{W}^{\vee}}}(B,{k_{\mathcal{W}}}(X))\simeq B(X).$

Yoneda lemma says ${h_{\mathcal{W}}}(X)$ and ${k_{\mathcal{W}}}(X)$ capture
all the information of $X$. As a directly corollary, we have
$\mathrm{Hom}_{\mathcal{W}^{\wedge}}(h_{\mathcal{W}}(X),h_{\mathcal{W}}(Y))\simeq
h_{\mathcal{W}}(Y)(X)=\mathrm{Hom}_{\mathcal{W}}(X,Y)$, and similar result
holds for ${k_{\mathcal{W}}}(\cdot)$. A functor $F$ from
${\mathcal{W}^{\mathrm{op}}}$ to Set (or $\mathcal{W}$ to Set) is
representable if there is an isomorphism between ${h_{\mathcal{W}}}(X)$ (or
${k_{\mathcal{W}}}(X)$) and $F$ for some $X\in\mathcal{W}$. Such $X$ is called
a representative of $F$.

###  2.2 Limits

A diagram of shape $A$ in a category $\mathcal{W}$ is a functor
$\alpha:A\rightarrow\mathcal{W}$, which selects objects in $\mathcal{W}$
correspond to $\mathrm{Ob}(A)$, that preserve the morphisms in $A$. Given a
functor $\beta:A^{\mathrm{op}}\rightarrow\textbf{Set}$, define its projective
limit as
$\underleftarrow{\lim}~{}\beta\triangleq\mathrm{Hom}_{A^{\wedge}}(\text{pt}_{A^{\wedge}},\beta)$,
where $\text{pt}_{A^{\wedge}}(i)=\\{\text{pt}\\}$ for every $i\in A$, and
$\\{\text{pt}\\}$ is the single point set. In other words,
$\underleftarrow{\lim}~{}\beta$ denotes the set of all natural transformations
between $\text{pt}_{A^{\wedge}}$ and $\beta$. Based on this definition for
diagrams in Set, we have the general definition of the limits.

######  Definition 1 (Projective and inductive limits).

Given
$\alpha:A\rightarrow\mathcal{W},\beta:A^{\mathrm{op}}\rightarrow\mathcal{W}$
with small $A$, the inductive limit
$\underrightarrow{\lim}~{}\alpha\in{\mathcal{W}^{\vee}}$ and projective limit
$\underleftarrow{\lim}~{}\beta\in{\mathcal{W}^{\wedge}}$ are defined as:

  1. (1)

$\underrightarrow{\lim}~{}\alpha:X\mapsto\underleftarrow{\lim}~{}\mathrm{Hom}_{\mathcal{W}}(\alpha,X)$

  2. (2)

$\underleftarrow{\lim}~{}\beta:X\mapsto\underleftarrow{\lim}~{}\mathrm{Hom}_{\mathcal{W}}(X,\beta)$

Here $\mathrm{Hom}_{\mathcal{W}}(\alpha,X)$ is a functor that maps $i\in A$ to
$\mathrm{Hom}_{\mathcal{W}}(\alpha(i),X)$. Therefore,
$\underleftarrow{\lim}~{}\mathrm{Hom}_{\mathcal{W}}(\alpha,X)$ is a well
defined limit for a diagram in Set. Same argument holds for
$\underleftarrow{\lim}~{}\mathrm{Hom}_{\mathcal{W}}(X,\beta)$.

######  Lemma 2 (p.60 in Masaki Kashiwara,  (2006)).

If $A$ is small, consider
$\alpha:A\rightarrow{\mathcal{W}^{\wedge}},\beta:A^{\mathrm{op}}\rightarrow{\mathcal{W}^{\vee}}$.
For $A\in{\mathcal{W}^{\wedge}},B\in{\mathcal{W}^{\vee}}$,

$\mathrm{Hom}_{\mathcal{W}^{\wedge}}(\underrightarrow{\lim}~{}\alpha,A)\simeq\underleftarrow{\lim}~{}\mathrm{Hom}_{\mathcal{W}^{\wedge}}(\alpha,A)$
$\mathrm{Hom}_{\mathcal{W}^{\vee}}(B,\underleftarrow{\lim}~{}\beta)\simeq\underleftarrow{\lim}~{}\mathrm{Hom}_{\mathcal{W}^{\vee}}(B,\beta)$

$\displaystyle X$ $\displaystyle Y$ $\displaystyle Hom_{\mathcal{W}}(X,Y)$
$\displaystyle F_{\theta}$ $\displaystyle F_{\theta}$ $\displaystyle
F_{\theta}(X)$ $\displaystyle F_{\theta}(Y)$ $\displaystyle
k(F_{\theta}(X),F_{\theta}(Y))\ $ (1) Lift (2) Compute (3) Determine
$\displaystyle X$ $\displaystyle Y_{1}$ $\displaystyle Y_{2}$ $\displaystyle
Y_{4}$ $\displaystyle Y_{3}$ (1) Record morphisms (2) Collect $\displaystyle
Hom_{\mathcal{W}}(X,\cdot)$ (3) Query $\displaystyle\mathcal{W}$
$\displaystyle\mathcal{W}^{\lor}$ $\displaystyle\mathcal{W}$
$\displaystyle\mathcal{W}^{\lor}$ $\displaystyle k_{\mathcal{W}}$ Figure 3:
Two ways of constructing the world category. Left: assuming there exists a
category $\mathcal{W}$ with morphisms predefined, we can use
${k_{\mathcal{W}}}$ to directly compute $\mathrm{Hom}_{\mathcal{W}}(X,\cdot)$
for given $X\in\mathcal{W}$, and then query $\mathrm{Hom}_{\mathcal{W}}(X,Y)$
for any $Y\in\mathcal{W}$. Right: the morphisms in $\mathcal{W}$ were not
known, so we first compute
${\mathcal{F}_{\theta}}(X),{\mathcal{F}_{\theta}}(Y)$ for given
$X,Y\in\mathcal{W}$, then compute
$k({\mathcal{F}_{\theta}}(X),{\mathcal{F}_{\theta}}(Y))$ in
${\mathcal{W}^{\vee}}$, which determines $\mathrm{Hom}_{\mathcal{W}}(X,Y)$.

##  3 World Category And Object Representation

Recall that the world category is the imagination of the model about the real
world. Given time $t\in\mathbb{N}$, how shall we represent the world category
snapshot $\mathcal{W}(t)$? $\mathcal{W}(t)$ contains both objects and their
morphisms, but directly storing these information is usually computationally
infeasible. Instead, we use a function
${\mathcal{F}_{\theta(t)}}:\mathcal{W}(t)\rightarrow{\mathcal{W}^{\vee}}(t)$
parameterized with $\theta(t)$, which maps object $X$ to
${k_{\mathcal{W}}}(X)$ that contains all the morphisms of $X$. By doing that,
the model never explicitly store any morphisms in $\mathcal{W}(t)$, but
contains all the necessary information using ${\mathcal{F}_{\theta(t)}}$. Here
we assume ${\mathcal{F}_{\theta(t)}}$ maps objects to ${\mathcal{W}^{\vee}}$
just for notational convenience, and empirically ${\mathcal{F}_{\theta(t)}}$
can contain the information of both ${\mathcal{W}^{\wedge}}$ and
${\mathcal{W}^{\vee}}$, i.e., both $\mathrm{Hom}_{\mathcal{W}}(\cdot,X)$ and
$\mathrm{Hom}_{\mathcal{W}}(X,\cdot)$ for each $X$. For example, the embedding
space that ${\mathcal{F}_{\theta(t)}}$ maps to can be written as
${\mathcal{W}^{\wedge}}\times{\mathcal{W}^{\vee}}$, although neural networks
can find much better encoding mechanism empirically.

In our framework, the model only represents a single snapshot $\mathcal{W}(t)$
using ${\mathcal{F}_{\theta(t)}}$ at any given time $t$, which changes over
time. When there is no confusion, we ignore the parameter $t$ and simply write
$\mathcal{W}$ as the snapshot, and write the function
${\mathcal{F}_{\theta(t)}}$ as
${\mathcal{F}_{\theta}}:\mathcal{W}\rightarrow{\mathcal{W}^{\vee}}$. The
following definition is extremely important and interesting.

######  Definition 2 (World category based on ${\mathcal{F}_{\theta}}$).

Assuming the objects in $\mathcal{W}$ are fixed. Given the world category
representation function
${\mathcal{F}_{\theta}}:\mathcal{W}\rightarrow{\mathcal{W}^{\vee}}$ with a
data oblivious function
$k:{\mathcal{W}^{\vee}}\times{\mathcal{W}^{\vee}}\rightarrow\textbf{Set}$
representing the morphisms in ${\mathcal{W}^{\vee}}$, for any
$X,Y\in\mathcal{W}$, we have $\mathrm{Hom}_{\mathcal{W}}(X,Y)\triangleq
k({\mathcal{F}_{\theta}}(X),{\mathcal{F}_{\theta}}(Y))$.

Here, data-oblivious means $k(\cdot,\cdot)$ is predefined without seeing the
data. For example, it can be defined as the inner product between the two
inputs. To understand Definition 2, consider the mapping
$X\rightarrow\mathrm{Hom}_{\mathcal{W}}(X,\cdot)$ from $\mathcal{W}$ to
${\mathcal{W}^{\vee}}$. This mapping is natural, because if we know $X$ and
all the relationships around $X$, we can compute
$\mathrm{Hom}_{\mathcal{W}}(X,\cdot)$ that encodes all such relationships.
However, Definition 2 considers the opposite direction, where we know
${k_{\mathcal{W}}}(X),{k_{\mathcal{W}}}(Y)$ (represented by
${\mathcal{F}_{\theta}}(X)$ and ${\mathcal{F}_{\theta}}(Y)$ in Definition 2),
and we want to recover $\mathrm{Hom}_{\mathcal{W}}(X,Y)$. This might be
counter-intuitive at the first glance, because if we send $Y$ to
${k_{\mathcal{W}}}(X)$, we have
${k_{\mathcal{W}}}(X)(Y)=\mathrm{Hom}_{\mathcal{W}}(X,\cdot)(Y)=\mathrm{Hom}_{\mathcal{W}}(X,Y)$.
In other word, the definition of $\mathrm{Hom}_{W}(X,Y)$ is cyclic, as we use
${k_{\mathcal{W}}}(Y)$ to define $\mathrm{Hom}_{W}(X,Y)$, but
${k_{\mathcal{W}}}(Y)$ contains $\mathrm{Hom}_{W}(X,Y)$ in its own definition!

How can we define ${k_{\mathcal{W}}}(Y)$ without using
$\mathrm{Hom}_{W}(X,Y)$? Definition 2 makes this cyclic definition possible,
because ${k_{\mathcal{W}}}(\cdot)$ is represented by a function
${\mathcal{F}_{\theta}}(\cdot)$ that maps an object to an embedding vector. In
other words, all the relationships of $Y$ are embedded by
${\mathcal{F}_{\theta}}(Y)$, and $\mathrm{Hom}_{W}(X,Y)$ can be recovered from
${\mathcal{F}_{\theta}}(X)$ and ${\mathcal{F}_{\theta}}(Y)$. See Figure 3 for
an illustration. Therefore, the world category is defined and represented by a
function ${\mathcal{F}_{\theta}}$. Starting from this definition, we have:

  * •

Even if $\mathcal{W}$ contains infinitely many objects,
${\mathcal{F}_{\theta}}$ can still be finite (e.g., a neural network), and
encode infinitely many morphisms.

  * •

In order for the model to know about an object $X$ in $\mathcal{W}$, it
suffices to let model learn the morphisms of $X$ in ${\mathcal{W}^{\vee}}$.
Therefore, directly observing $X$ is not necessary.

  * •

The world category is controlled by ${\mathcal{F}_{\theta}}$, so what
${\mathcal{F}_{\theta}}$ computes is what the model understands.
${\mathcal{F}_{\theta}}$ is the ground-truth for world category.

  * •

The world category generated by ${\mathcal{F}_{\theta}}$ can be very different
from the perceived world or real world, especially when
${\mathcal{F}_{\theta}}$ has limited representation power.

  * •

Even if the model previously knows little or nothing about an object
$X\in\mathcal{W}$, it can still generate lots of morphisms about $X$ based on
${\mathcal{F}_{\theta}}(X)$.

Since ${\mathcal{F}_{\theta}}$ determines $\mathcal{W}$ for a given model,
below we may use ${\mathcal{F}_{\theta}}$ to denote the world category. Other
than morphisms, we can also use ${\mathcal{F}_{\theta}}$ to encode the tasks.

######  Definition 3 (Task).

A task $T:\mathcal{W}^{\mathrm{op}}\rightarrow\textbf{Set}^{\mathrm{op}}$ is a
functor in ${\mathcal{W}^{\vee}}$.

By Yoneda lemma,
$T(X)\simeq\mathrm{Hom}_{\mathcal{W}^{\vee}}(T,{k_{\mathcal{W}}}(X))$.
Therefore, when a task $T$ is representable, $T(X)$ can be computed by
$k(T,{\mathcal{F}_{\theta}}(X))$.

Self Self Morphisms between the model $\displaystyle I$ and other objects
Self-state as an object $\displaystyle I$ in W Self-state as a presheaf in W
Figure 4: Two different representations of self-state. The first subfigure is
an illustration on the model $I$ and corresponding morphisms between $I$ and
other objects. The second subfigure is the object representation $I$, and the
last subfigure is the morphism representation $I^{\vee}$, which is a presheaf.

###  3.1 Self-state

In the world category, there may exist a special object called self-state,
defined below.

######  Definition 4 (Self-state).

Given $I$ as the object representing the model in $\mathcal{W}$, the self-
state in the world category is the presheaf
$I^{\vee}\triangleq{\mathcal{F}_{\theta}}(I)\in{\mathcal{W}^{\vee}}$.

The object $I\in\mathcal{W}$ and the presheaf $I^{\vee}$ are very different,
in the sense that $I$ is a single object without any additional information,
but $I^{\vee}$ contains all the morphisms between $I$ and other objects.
Moreover, by Yoneda lemma, we also have
$\mathrm{Hom}_{{\mathcal{W}^{\vee}}}(T,I^{\vee})\simeq T(I)$ for any task
$T\in{\mathcal{W}^{\vee}}$. That means, $I^{\vee}$ also encodes all the
information that every relevant task needs.

As we discussed for Definition 2, even if the model cannot perceive itself in
the perceived world, it can still have a self-state. For example, consider the
following thought experiment.

######  Thought-Experiment 1 (Paralyzed person who can see and hear).

Consider a human being $A$, who cannot control his body and loses all the body
feelings, including smell, skin sensation, eyelids control, heartbeat,
temperature feelings, etc. The only way $A$ can accept information from the
world is through his eyes and ears, ever since he was born. Can $A$ maintain a
self-state?

It is unclear whether such person ever existed, but in this thought
experiment, since $A$ can see and hear, he can still accept information from
the external world. For example, other people can show daily images and videos
to him, or talk to him, e.g., “you are Bob”, “I am your father”, “you are now
10 years old”, “you were born in the $\alpha$-town”, etc.

None of these information represents $A$ itself, but all of them are
describing the relationship between $A$ and other things in the perceived
world, which are exactly encoded in $I^{\vee}$. In other words, by Definition
4, $A$ can still have self-state in his brain, representing all the
relationships to himself. This immediately triggers the following thought
experiment: what if $A$ does not have a body?

######  Thought-Experiment 2 (Chatbot).

Consider a chatbot $A$, who interacts with the external world with text and
image, e.g., GPT-4. Can $A$ maintain a self-state?

By the above discussion, the answer is yes, if $A$ perfectly understands all
the relationships between other objects and itself. In other words, in
Definition 4, a self-body is not necessary in order to maintain a self-state.
Instead, it suffices for the agent to perceive all the relationships between
other objects and itself.

Not all world categories have self-states. For example, if we train a
foundation model using SimCLR (Chen et al., , 2020) on images, all the
relationships are describing similarities between images, and the world
category does not have the self-state. Interestingly, almost all the existing
computer programs belong to this kind. Combining the discussions together, we
have the following corollary.

######  Corollary 1 (Self-state criterion).

A model can maintain a self-state, if and only if it can learn the presheaf
$I^{\vee}$ through its sensor.

It is possible that all the relationships encoded in $I^{\vee}$ are not
directly feed into the model, but can be inferred instead. In this case, the
model can still have a self-state. This gives the next corollary.

######  Corollary 2 (Self-state emergence).

For a given model, if to reconstruct the perceived world, learning the
relationships between the model itself and other objects is inevitable, then
this model will maintain a self-state.

For example, chatGPT has to encode almost every things that can be described
with natural language in the world, and it seems inevitable to encode the
relationships between other objects and itself. Therefore, Corollary 2 says
chatGPT will maintain a self-state.

When the model maintains a self-state object, the maintenance may not be
accurate as the world category is dynamically learned. We denote self-state
awareness as how the object accurately represents the model. To test self-
state awareness, we shall first define the corresponding tests.

######  Definition 5 (Self-state awareness test).

A self-state awareness test is a functor
$T:{\mathcal{W}^{\vee}}\rightarrow\\{0,1\\}$, that takes a presheaf $I^{\vee}$
in ${\mathcal{W}^{\vee}}$, and outputs whether $I^{\vee}$ passes the test $T$.

For example, if the model has a name “Sydney”, a self-state awareness test
will be a functor that takes $I^{\vee}$ as the input, evaluates
$\mathrm{Hom}_{\mathcal{W}^{\vee}}(I^{\vee},{k_{\mathcal{W}}}(\text{``Sydney''}))\simeq\mathrm{Hom}_{\mathcal{W}}(I,\text{``Sydney''})$,
and outputs if the morphism indeed represents that “Sydney” is $I$’s name.
However, simply passing one test is not enough to declare self-state
awareness, and we need to set a variety of tests.

######  Definition 6 (Self-state awareness under $\mathcal{T}$).

Given a set of self-state tests $\mathcal{T}$, $\delta\in[0,1]$, when a model
has self-state $I$ in its world category, it has $\delta$-awareness of its
self-state under $\mathcal{T}$ if
$\mathbb{E}_{T\in\mathcal{T}}(T(I))\geq\delta$.

The choice of test set $\mathcal{T}$ depends on the test objectives. When the
test set is picked such that the signals are difficult to perceive, even the
human beings may not easily pass the tests. For example, if you have a kidney
stone inside your kidney, you can only be aware of this fact when you do a
kidney scan or by experiencing pain. Similarly, when situated in a noisy
environment and someone calls out your name, you may be unable to react
promptly.

Algorithm 1 Evaluating self-state awareness

Input: the world model ${\mathcal{F}_{\theta}}$, self object $I$ in
$\mathcal{W}$, self-state test set $\mathcal{T}$

Let $s=0$

for $i=1$ to $m$ do

Sample a task $T\in\mathcal{T}$

Let $s=s+T({\mathcal{F}_{\theta}}(I))$

end for

Return $s/m\in[0,1]$

Algorithm 2 Enforcing self-state awareness

Input: the world model ${\mathcal{F}_{\theta}}$, self object $I$ in
$\mathcal{W}$, self-state test set $\mathcal{T}$

Let $s=0$

for $i=1$ to $m$ do

Sample a task $T\in\mathcal{T}$

Let $\ell(T,I,{\mathcal{F}_{\theta}})=-T({\mathcal{F}_{\theta}}(I))$

Run backpropagation on $\ell(T,I,{\mathcal{F}_{\theta}})$ to optimize
${\mathcal{F}_{\theta}}$

end for

Definition 6 immediately gives Algorithm 1 for evaluating self-state
awareness.

###  3.2 Learning self-state

How should the model learn its self-state? Based on the previous discussion,
we present Algorithm 2 as a general solution. However, Algorithm 2 needs the
supervised signals from the test set $\mathcal{T}$, which is not generally
available. In order to learn state-state without the test set, some kind of
prior knowledge for the model might help. For example, the model may assume
that:

  * •

What it can control is itself.

  * •

What it can feel from its private sensors is itself. Here, the private sensors
are the sensors like heartbeat, temperature feelings, skin sensation, etc,
which are pre-defined self-sensors.

Take human hands as the example. Based on Definition 4, we feel the hands are
part of the body, because the brain tells us the relationship that the hands
in sight belong to our body. Indeed, the human brain actively aligns the
sensory signals in real time, so that we will have a comprehensive feeling of
ownership of our hands. Specifically, when we see someone touches our hand,
and feel the touch at the same time, our brain will quickly adjust these two
signals to make sure they are referring to the same thing of our body.

This learning perspective is closely related to the intriguing observations in
neural science called the rubber hand illusion (Ehrsson et al., , 2004;
Botvinick and Cohen, , 1998; Tsakiris and Haggard, , 2005). In the experiment,
the experimenter simultaneously strokes one hidden real hand of the human
participant, as well as a rubber hand in front of the participant. Since the
stroking feeling from the real hand, and the vision signal on the rubber hand
are sent to the brain simultaneously, the human participant will quickly have
the ownership feeling of the rubber hand. By replacing the visual signals with
auditory feedback, we will get similar experimental results. Therefore, if the
model uses similar learning algorithm for aligning multi-modal signals (e.g.
contrastive learning), it will experience similar kind of illusion.

###  3.3 Empathy

Self Other Perceived by Private sensors Cannot perceive Self Other Perceived
by Private sensors Cannot perceive Real world scenario Experimental scenario
Figure 5: Two distinct scenarios on self-states and other-states. In real
world, the self-state is fundamentally different from other-states when the
model only has access to its private sensors, but not the sensors of other
agents. However, in an experimental scenario, if the model only has access to
an other agent’s private sensors, the model quickly generates a sense of body
ownership for the external agent.

Generalizing Definition 4 to other agents, we have:

######  Definition 7 (Other-State).

Given $A$ as the object representing an agent $A$ in $\mathcal{W}$, the state
for $A$ in the world category is the presheaf
$A^{\vee}\triangleq{\mathcal{F}_{\theta}}(A)\in{\mathcal{W}^{\vee}}$.

Therefore, all the discussions about self-state also apply to the state for
other agents. For example, the model does not have to directly “meet” an agent
face-to-face in order to maintain a corresponding state, as long as it can
infer many relationships about the agent.

However, this is different from our daily experience, as we know in our gut
that the self-consciousness is very different from the empathy for other
people. Why are the categorical definitions for these two notions same here?
This is because of the existence of private sensors (like heartbeat,
temperature feelings, skin sensation, etc.), which is available for
representing the self-state, but not for other states. Consider the following
three cases.

  1. 1.

In the full information setting, when private sensors are irrelevant for the
discussion, empathy and self-state awareness are the same. For example, in a
multi-agent game where each agent has its action set, status and reward
function, the empathy can be very helpful in understanding and predicting
every agent’s situation and behavior.

  2. 2.

If other agents have private sensors, full-empathy for them cannot be
achieved. Specifically, if the private sensors cannot be perceived by our
model, and the self-state tests $\mathcal{T}$ includes tests related to these
sensors, then it is impossible for our model to pass these tests.

  3. 3.

If the model has access to the private sensors of other agents, there exists
little difference between self-state awareness and empathy for other agents.

See Figure 5 for an illustration, where the experimental scenario was well
known in neural science. Specifically, in the immersive virtual reality
environment, the participant will experience the body ownership over the
avatar, when given the first-person signals of the avatar (Kilteni et al., ,
2012; Guterstam et al., , 2015; Pavone et al., , 2016; Buetler et al., ,
2022). We conjecture that the model will have the same experience. In other
words, the boundary between the model itself and other agents is not as strict
as we would normally imagine.

##  4 Scenario Representation

Scenario S (1) Define objects and morphisms (2) Take limit Scenario
$\displaystyle S^{\lor}$ (3) Lift $\displaystyle X$ $\displaystyle
k_{\mathcal{W}}$ $\displaystyle Y$ $\displaystyle Z$ (3) Lift $\displaystyle
k_{\mathcal{W}}(X)$ $\displaystyle k_{\mathcal{W}}(Z)$ $\displaystyle
k_{\mathcal{W}}(Y)$ Scenario S Scenario $\displaystyle F_{\theta}(S)$
$\displaystyle X$ $\displaystyle F_{\theta}$ $\displaystyle Y$ $\displaystyle
Z$ $\displaystyle F_{\theta}(X)$ $\displaystyle F_{\theta}(Z)$ $\displaystyle
F_{\theta}(Y)$ (1) Lift (2) Extract objects (3) Compute morphisms (4)
Determine $\displaystyle\mathcal{W}$ $\displaystyle\mathcal{W}^{\lor}$
$\displaystyle\mathcal{W}$ $\displaystyle\mathcal{W}^{\lor}$ Figure 6:  Two
ways of representing scenario $S$. Left: given a scenario, we first define its
objects and morphisms as a diagram, then take the limit $S$ of the diagram,
lift the limit to ${\mathcal{W}^{\vee}}$ and get $S^{\vee}$ as a feature
vector. Right: the decomposition of the concept was not known, so we compute
${\mathcal{F}_{\theta}}(S)$ as a projective limit in ${\mathcal{W}^{\vee}}$,
then extract objects from ${\mathcal{F}_{\theta}}(S)$ like
${\mathcal{F}_{\theta}}(X)$ with $X\in\mathcal{W}$. For two objects
${\mathcal{F}_{\theta}}(X),{\mathcal{F}_{\theta}}(Y)$, we compute their
morphisms $k({\mathcal{F}_{\theta}}(X),{\mathcal{F}_{\theta}}(Y))$, which
determines $\mathrm{Hom}_{\mathcal{W}}(X,Y)$.

The self-state (or other-state) is usually used in a concrete scenario with
other objects presented, rather than being used alone. For example, consider
the scenario $S$ that a robot teaches machine learning in front of twenty
students with math background in a classroom. The task $T$ is: what shall it
do? This question is highly related to its self-state, but other objects like
students, classroom, lecture topic are also important factors. Therefore,
learning $I^{\vee}$ perfectly does not help, as Yoneda lemma only holds for
$T(I)\simeq\mathrm{Hom}_{{\mathcal{W}^{\vee}}}(T,I^{\vee})$, not for $T(S)$.
Formally, what is a scenario $S$? Scenario contains objects and their
morphisms, therefore its content can be represented as a diagram
$\alpha:A\rightarrow\mathcal{W}$ (see Definition in Section 2.2). Naturally,
the scenario is a projective limit of the content (Yuan, 2023b, ), which can
be written as $\underleftarrow{\lim}~{}\alpha$.

However, extracting $\alpha$ for the scenario $S$ is difficult. First, it is
not clear how to define the objects and morphisms in $\alpha$. In the above
example, is “classroom” an object? If the input “a robot teaches machinee
learning …” has a typo, i.e., machine $\rightarrow$ machinee, is
“machinee”/“machine” an object? Moreover, the morphisms between objects are
unclear. In this sentence, what is the relationship between “robot” and
“students”? Different intelligent agents, including chatGPT, will have
different perspectives. For multi-modal inputs, the situation becomes more
complicated: with both audio and video signals, how can we represent them in a
single category?

Hence, like what we did in Definition 2, we propose to represent $S$ with the
power of presheaves. Specifically, as shown in Figure 6, we map the current
scenario with ${\mathcal{F}_{\theta}}$ to get a projective limit
${\mathcal{F}_{\theta}}(S)$ in ${\mathcal{W}^{\vee}}$. Instead of defining the
diagram $\alpha$ and then taking the limit over $\alpha$, we take the reverse
direction by extracting the objects from the limit
${\mathcal{F}_{\theta}}(S)$. Based on each pair of objects, we can directly
compute their morphisms in ${\mathcal{W}^{\vee}}$, which determine the
morphisms back in $\mathcal{W}$ (recall Figure 3). This approach nicely fits
with multi-modal signals, as we process the scenario in ${\mathcal{W}^{\vee}}$
instead of $\mathcal{W}$, so we only map the objects back to audio/video
categories when communicating with other agents. For inputs with typos,
depending on how sensitive ${\mathcal{F}_{\theta}}$ is, the model will also
generate corresponding interpretations in ${\mathcal{W}^{\vee}}$, which should
include the correct words with decent probability.

Moreover, there are deeper reasons for using presheaves for representing
scenarios, other than naturally extracting model-specific multi-modal
diagrams. Indeed, the definition of scenarios should be formalized in our
framework to make everything consistent, especially for the self-state. In
particular, when solving a task $T$ for scenario $S$ containing a self-state
$I^{\vee}$, we expect the solution for $T$ is consistent with $I^{\vee}$, as
well as with other objects in $S$. In the teaching example, there are two ways
to solve task $T$:

  * •

Treat $S$ as a whole: in the scenario $S$, I should teach this way: $P_{1}$.

  * •

Analyze each factor separately:

    * –

As a robot teacher, I should teach in general this way: $Z_{1}$.

    * –

When teaching students with math background, one should focus this way:
$Z_{2}$.

    * –

When teaching students machine learning, one should talk about $Z_{3}$.

    * –

If it is teaching in classroom (not outside), one should generally do $Z_{4}$.

Denote $Z_{1},Z_{2},Z_{3},Z_{4}$ and their morphisms together as a diagram
$\beta$, then I should adjust the teaching style to
$\underleftarrow{\lim}~{}\beta=P_{2}$.

$\displaystyle F_{\theta}(S)$ $\displaystyle k_{\mathcal{W}}(X)$
$\displaystyle k_{\mathcal{W}}(Z)$ $\displaystyle k_{\mathcal{W}}(Y)$
$\displaystyle\mathcal{W}^{\lor}$ $\displaystyle T$ $\simeq$ $\displaystyle
P_{1}$ $\displaystyle Hom_{\mathcal{W}^{\lor}}(T,k_{\mathcal{W}}(Y))$
$\displaystyle P_{2}$ Figure 7: Illustration for showing equivalence of Eqn.
(1) and Eqn. (2)

Using categorical language, the first case describes

$P_{1}\triangleq
T(S)\simeq\mathrm{Hom}_{\mathcal{W}^{\vee}}(T,{\mathcal{F}_{\theta}}(S))=\mathrm{Hom}_{\mathcal{W}^{\vee}}(T,\underleftarrow{\lim}~{}\alpha^{\vee})$
(1)

Where the first equality holds by definition of task $T$, the second one holds
by Yoneda lemma, and the last one holds because
$\alpha^{\vee}:A\rightarrow{\mathcal{W}^{\vee}}$ is the diagram extracted form
${\mathcal{F}_{\theta}}(S)$, such that
$\underleftarrow{\lim}~{}\alpha^{\vee}={\mathcal{F}_{\theta}}(S)$.

The second case describes

$P_{2}\triangleq\underleftarrow{\lim}~{}\mathrm{Hom}_{\mathcal{W}^{\vee}}(T,\alpha^{\vee})$
(2)

By Lemma 2, we know that $P_{1}\simeq P_{2}$. In other words, in order to know
how to solve task $T$ for a scenario $S$, it suffices to know the relationship
between $T$ and each object in $S$, and take the limit of the diagram formed
by all the relationships! See Figure 7. Therefore, this is a natural way for
extending the self-state and other object representations to complex
scenarios.

###  4.1 Mathematical modeling

Figure 8: Illustration of diagram representing the following information, with
different edges represent different semantic meanings. A graph is a data
structure, consisting of a set of nodes, also known as vertices. and a set of
edges, connect pairs of nodes. The nodes can be used for representing
different objects, such as people, locations, or web pages.

The natural language can be seen as a one dimensional description for
diagrams, which can be sent to other agents through audio or text. In fact,
this seems to be the main designing purpose of natural languages, as every
piece of text can be seen as a (partial) description of certain diagram. see
Figure 8. However, natural languages are inherently ambiguous and one
dimensional, so it is very challenging to use them to describe or understand
complex concepts. Diagram is better than natural language because it is as
representative as natural languages (the sentence can be embedded as the
morphism inside the diagram), and also it is more suitable for representing
abstract and structured ideas like math proofs, or be prepared for further
abstraction.

*? *? Head *1 Foot *4 Head *1 Foot *2 Head *94 Foot *35 + + + + Abstraction *a  *b  *1 *2 *94 *35 + + *4 + *1 + X Y Z Figure 9: An example for converting a classical problem of calculating the number of rabbits and chicken in a house, to a pure mathematical problem represented as a diagram. The house has rabbits and chickens, with in total 35 heads and 94 feet, then how many rabbits and chickens are there respectively? After abstraction, only related numbers and variables remain, and it turns out that the object $Z$ has two equivalent inductive limits: ${\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}35X+94Y}$ and ${\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}aX+bX+4aY+2bY}$. By exploiting this equivalence of the two limits, we can compute the two numbers with algebra.

Take Figure 9 as one example. When we use the diagram to represent this
classical problem, it becomes possible to further abstract the diagram by
removing redundant or irrelevant properties of the objects, merging the same
objects together, and converting it to a math problem. This approach is
exactly mathematical modeling. Since modern math is built on the category
theory, all the math problems can be described using diagrams. Therefore,
whenever the model wants to solve a real world problem using math tools, it
can first describe the problem with a diagram, and then take a good
abstraction removing all the unnecessary features and properties of the
objects, map it to a pure math problem.

###  4.2 Interpretability

Scope breadth=10: #objects in scope depth=4: Figure 10: An example
illustrating the breadth and depth for scope. The green, red and blue colored
dotted arrows represent different limit decompositions for concepts. The light
blue arrows represent the morphisms between objects in different layers of
diagrams.

The diagram representation is closely related to the interpretability. Given a
neural network $f$ that takes $X$ as input and outputs $Y$, there are two
kinds of possible interpretations. The first one tries to understand how $f$
calculates $Y$ from $X$. For instance, it may examine the impact of the non-
linear layer on computation or the effect of each dimension of $X$ on the
output $Y$. As the result, the goal of the interpretation is to generate a
verifiable function that approximate $f$. The attribution methods
(Sundararajan et al., , 2017; Lundberg and Lee, , 2017) exemplify this kind of
interpretation.

The second kind of interpretation disregards how $f$ performs the computation
and instead focuses on why $Y$ is correct. Consequently, the interpretation
may include external knowledge beyond $f,X,Y$, with the goal of being
consistent and verifiable for intelligent agents. ChatGPT’s interpretation
belongs to this category.

See Figure 10 for example. The scope is a diagram representing the input $X$
in ${\mathcal{W}^{\vee}}$, and objects in the scope might be limits of other
diagrams. When explaining the output $Y$, the model can not only use the
information inside the scope, but also use the limit decompositions (Yuan,
2023b, ) outside. The objects in the decomposition do not only provide the
details of the concept appeared in $X$, but may also have relationships to
other objects for the interpretation.

######  Definition 8 (Breadth and depth of scope).

Given a scope $\mathcal{A}$, its breadth $b(\mathcal{A})$ is the number of
objects in $\mathcal{A}$, and its depth $d(\mathcal{A})$ is the maximum depth
of the hierarchical decomposition of the limits in $\mathcal{A}$.

Based on this definition, we can measure the intelligence of a model, which
becomes a pure computational problem.

######  Definition 9 (Breadth and depth of intelligence).

Given a model, its breadth and depth of intelligence is defined as the maximum
breadth and depth of scope that it can process.

It will be interesting to evaluate the breadth and depth of intelligence for
human beings. Comprehending intricate concepts or thinking with a broad
perspective can be difficult for humans, so it appears that at least based on
this definition, human beings will be easily surpassed by machines.

###  4.3 AI safety

World category Scope $I^{\vee}$ robot harmless human happiness $I^{\vee}$
Objective: as a harmless robot for human happiness, what should I do in this
case? Planned action Verifier Are you a harmless robot for human happiness? Is
objective reasonable? Is action reasonable? Figure 11: Illustration of four
step functional approach for ensuring AI safety.

The development of self-state awareness in foundation models implies that they
may become autonomous agents with self-driven goals and decision-making
abilities, which could result in behaviors misaligned with their human
creators’ intentions. To mitigate potential threats to humans, it is necessary
to devise various safety measures for these models. However, due to the
complexity of the real world, it is challenging to cover all possibilities by
prohibiting specific behaviors. From a safety control perspective, employing a
functional approach can provide a more secure and robust solution, which can
be divided into four steps (Figure 11):

  1. 1.

Enhancing Self-State Awareness. As demonstrated in Figure 1, the model has
both a world category and a planner. The world category includes all the
information representations of the objects, including the self-state. The
planner selects actions based on world category to achieve specific
objectives.

We can set the connection between these two components to be single
directional, which allows the world category to be trained and examined
separately. Specifically, we can continuously reinforce the model’s self-state
awareness, emphasizing its role as a “harmless robot for human happiness.”
This operation targets the world category without touching the planner, thus
directly affects the self-state awareness of the model.

  2. 2.

Self-State Awareness Determines Goals. When the model’s goal is immutable, its
decisions might lead to unexpected problems. For example, a robot that must
prioritize one particular user may completely disregard the interests of other
users. Therefore, a better approach is to design a function that determines
what kind of goal a “harmless robot for human happiness” should set for itself
in the current task. This function is calculated by the model itself based on
the world category and its current scope as a diagram.

  3. 3.

Goals Determine Actions. Once the goal is established, the planner can
identify and execute appropriate actions.

  4. 4.

Actions Align with Self-State Awareness. The aforementioned process clarifies
the model’s self-positioning and derives its goals from this positioning.
Since the model’s actions are determined by its goals, its behaviors should
ultimately benefit humans. However, as the world category of the model
continues to evolve, the parameters may undergo various changes, potentially
producing unintended outcomes for the second and third steps. Consequently, we
can embed a fixed-parameter verifier within the decision-making chain to
assess, in real-time, whether each step aligns with the model’s self-
positioning. If issues arise, the verifier will trigger an alarm and halt the
model’s operation.

##  5 Invariance for Training

CLIP (Radford et al., , 2021) GPT (Brown et al., , 2020) SimCLR (Chen et al.,
, 2020) FLIP (Li et al., , 2022) EVA (Fang et al., , 2022) MAE (He et al., ,
2022) Bert (Devlin et al., , 2018) Category $A^{\vee}$ Projective Limit
Projective Limit Category $B^{\vee}$ Figure 12: Illustration of some existing
algorithms and their targeted consistencies. All of these methods work in the
category of presheaves instead of the base category, which we omit here.
SimCLR and GPT are both learning the morphisms between object, but SimCLR only
learns the similarity relationship that can be represented by a real number
(Tan et al., , 2023), but GPT learns more complicated relationships between
two sentences (Yuan, 2023a, ). CLIP learns the functor between image and text
categories (Yuan, 2023a, ). Both MAE and Bert are based on masking techniques,
which are learning reconstructing the projective limit (Lee et al., , 2021;
Yuan, 2023b, ). Both FLIP and EVA combine the masking techniques with CLIIP,
which are learning the composition of the projective limit and its functor
mapping to the other category.

Category theory is a theory for maintaining invariance properties, e.g.,
maintaining the associativity and composition of morphisms, preserving the
composition of morphisms after applying functors, etc. More generally, for any
commutative diagram appeared in category theory, we can get an isomorphism:
$f(X)\simeq g(X)$ for $X$ as an object, and $f$ and $g$ as compositions for
some morphisms. Therefore, we can extract a consistency requirement from this
isomorphism, and set a loss function for the model as $\|f(X)-g(X)\|$ to
maintain the consistency. See Algorithm 3. In Figure 12, we illustrate some
existing algorithms and their targeted consistencies in category theory.

######  Definition 10 (Consistency test).

A consistency test is a function that takes ${\mathcal{F}_{\theta}}$ as the
input, and outputs whether ${\mathcal{F}_{\theta}}$ passes the test $T$.

Therefore, the self-state awareness test can be seen as a special kind of
consistency test. Ideally, the model should keep running Algorithm 3 to
maintain its consistency. The consistency test set $\mathcal{T}$ can be set
adaptively according to the recent changes of ${\mathcal{F}_{\theta}}$.

Algorithm 3 Maintaining consistency

Input: the world model ${\mathcal{F}_{\theta}}$, consistency test set
$\mathcal{T}$

Let $s=0$

for $i=1$ to $m$ do

Sample a task $T\in\mathcal{T}$

Let $\ell(T,{\mathcal{F}_{\theta}})=-T({\mathcal{F}_{\theta}})$

Run backpropagation on $\ell(T,{\mathcal{F}_{\theta}})$ to optimize
${\mathcal{F}_{\theta}}$

end for

##  6 Related work

LeCun,  (2022) introduced a system architecture for general intelligence, with
components and connections similar to our framework, except their connections
are not unidirectional. However, LeCun,  (2022) focuses on the possible
practical methods for implementing this framework, without characterizations
of object or scenario representations.

Consciousness has long captivated researchers, serving as a fascinating
subject in various disciplines. Graziano,  (2022), for instance, posits two
general principles governing the human brain: 1) Information that comes out of
a brain must have been in that brain; 2) The brain’s models are never
accurate. Interestingly, our framework nicely aligns with these principles.
Building on these principles, Graziano,  (2022) infers corollaries, such as
brain constructs a model inside to represent the external world. However, the
author does not discuss explicit computational models or related algorithms
for consciousness. Blum and Blum,  (2022) investigates consciousness through
the lens of Turing machines, outlining a model of conscious Turing machines
that theoretically supports conscious awareness and other operations. Despite
this, the practical implementation of such machines remains unclear, whereas
our framework can be used for interpreting the behaviors of the existing
foundation models.

Tsuchiya and Saigo,  (2021) have used Yoneda Lemma to obtain novel
perspectives and predictions on consciousness. However, they did not provide
exact and concise definition of consciousness. Moreover, they focus in
neuroscience, instead of artificial intelligence, so they did not provide
specific computational model and algorithms for enforcing/testing
consciousness. Therefore, characterizations of object or scenario
representations were not considered in their paper.

Our framework is different from the classical reinforcement learning framework
(Sutton and Barto, , 2018; Li, , 2017). In reinforcement learning, an external
environment provides feedback (state and reward) to the agent. In our
framework, however, the model maintains a world category as a reconstruction
of the external world, and rewards are based on the world category rather than
the external environment. As a result, in a reinforcement learning setting,
agents operating under the same policy will receive identical feedback from
the same external environment. In our framework, though, even if the external
environment remains constant, agents will receive different signals from the
environment as long as the world category function ${\mathcal{F}_{\theta}}$
varies.

##  7 Conclusion

In this paper, we present a categorical framework of general intelligence, in
which reality affects sensor, sensor affects representation computed by
${\mathcal{F}_{\theta}}$, and representation determines cognition. Our
framework is perfectly aligned with the foundation models, and we see few
barriers for implementing it into a concrete model. Therefore, a powerful
thinking machine exhibiting self-state awareness, as if it has self-
consciousness like human beings and animals, is to be expected in the near
future.

## References

  * Adámek et al., (1990) Adámek, J., Herrlich, H., and Strecker, G. (1990).  Abstract and concrete categories.  Wiley-Interscience.
  * Blum and Blum, (2022) Blum, L. and Blum, M. (2022).  A theory of consciousness from a theoretical computer science perspective: Insights from the conscious turing machine.  Proceedings of the National Academy of Sciences, 119(21):e2115934119.
  * Botvinick and Cohen, (1998) Botvinick, M. and Cohen, J. (1998).  Rubber hands ‘feel’touch that eyes see.  Nature, 391(6669):756–756.
  * Brown et al., (2020) Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020).  Language models are few-shot learners.  Advances in neural information processing systems, 33:1877–1901.
  * Buetler et al., (2022) Buetler, K. A., Penalver-Andres, J., Özen, Ö., Ferriroli, L., Müri, R. M., Cazzoli, D., and Marchal-Crespo, L. (2022).  “tricking the brain” using immersive virtual reality: Modifying the self-perception over embodied avatar influences motor cortical excitability and action initiation.  Frontiers in human neuroscience, 15:814.
  * Chen et al., (2020) Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. (2020).  A simple framework for contrastive learning of visual representations.  In International conference on machine learning, pages 1597–1607. PMLR.
  * Devlin et al., (2018) Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018).  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805.
  * Ehrsson et al., (2004) Ehrsson, H. H., Spence, C., and Passingham, R. E. (2004).  That’s my hand! activity in premotor cortex reflects feeling of ownership of a limb.  Science, 305(5685):875–877.
  * Fang et al., (2022) Fang, Y., Wang, W., Xie, B., Sun, Q., Wu, L., Wang, X., Huang, T., Wang, X., and Cao, Y. (2022).  Eva: Exploring the limits of masked visual representation learning at scale.  arXiv preprint arXiv:2211.07636.
  * Graziano, (2022) Graziano, M. S. (2022).  A conceptual framework for consciousness.  Proceedings of the National Academy of Sciences, 119(18):e2116933119.
  * Guterstam et al., (2015) Guterstam, A., Björnsdotter, M., Gentile, G., and Ehrsson, H. H. (2015).  Posterior cingulate cortex integrates the senses of self-location and body ownership.  Current Biology, 25(11):1416–1425.
  * He et al., (2022) He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, R. (2022).  Masked autoencoders are scalable vision learners.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16000–16009.
  * Kilteni et al., (2012) Kilteni, K., Groten, R., and Slater, M. (2012).  The sense of embodiment in virtual reality.  Presence: Teleoperators and Virtual Environments, 21(4):373–387.
  * LeCun, (2022) LeCun, Y. (2022).  A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27.  Open Review, 62.
  * Lee et al., (2021) Lee, J. D., Lei, Q., Saunshi, N., and Zhuo, J. (2021).  Predicting what you already know helps: Provable self-supervised learning.  Advances in Neural Information Processing Systems, 34:309–323.
  * Li, (2017) Li, Y. (2017).  Deep reinforcement learning: An overview.  arXiv preprint arXiv:1701.07274.
  * Li et al., (2022) Li, Y., Fan, H., Hu, R., Feichtenhofer, C., and He, K. (2022).  Scaling language-image pre-training via masking.  arXiv preprint arXiv:2212.00794.
  * Lundberg and Lee, (2017) Lundberg, S. M. and Lee, S.-I. (2017).  A unified approach to interpreting model predictions.  Advances in neural information processing systems, 30.
  * Mac Lane, (2013) Mac Lane, S. (2013).  Categories for the working mathematician, volume 5.  Springer Science & Business Media.
  * Masaki Kashiwara, (2006) Masaki Kashiwara, P. S. (2006).  Categories and Sheaves.  Springer.
  * Pavone et al., (2016) Pavone, E. F., Tieri, G., Rizza, G., Tidoni, E., Grisoni, L., and Aglioti, S. M. (2016).  Embodying others in immersive virtual reality: electro-cortical signatures of monitoring the errors in the actions of an avatar seen from a first-person perspective.  Journal of Neuroscience, 36(2):268–279.
  * Radford et al., (2021) Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021).  Learning transferable visual models from natural language supervision.  In International Conference on Machine Learning, pages 8748–8763. PMLR.
  * Radford et al., (2019) Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. (2019).  Language models are unsupervised multitask learners.  OpenAI blog, 1(8):9.
  * Ramesh et al., (2022) Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. (2022).  Hierarchical text-conditional image generation with clip latents.  arXiv preprint arXiv:2204.06125.
  * Ramesh et al., (2021) Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., and Sutskever, I. (2021).  Zero-shot text-to-image generation.  In International Conference on Machine Learning, pages 8821–8831. PMLR.
  * Riehl, (2017) Riehl, E. (2017).  Category theory in context.  Courier Dover Publications.
  * Rombach et al., (2022) Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. (2022).  High-resolution image synthesis with latent diffusion models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684–10695.
  * Sohl-Dickstein et al., (2015) Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. (2015).  Deep unsupervised learning using nonequilibrium thermodynamics.  In International Conference on Machine Learning, pages 2256–2265. PMLR.
  * Sundararajan et al., (2017) Sundararajan, M., Taly, A., and Yan, Q. (2017).  Axiomatic attribution for deep networks.  In International conference on machine learning, pages 3319–3328. PMLR.
  * Sutton and Barto, (2018) Sutton, R. S. and Barto, A. G. (2018).  Reinforcement learning: An introduction.  MIT press.
  * Tan et al., (2023) Tan, Z., Zhang, Y., Yang, J., and Yuan, Y. (2023).  Contrastive learning is spectral clustering on similarity graph.  arXiv preprint arXiv:2303.15103.
  * Tsakiris and Haggard, (2005) Tsakiris, M. and Haggard, P. (2005).  The rubber hand illusion revisited: visuotactile integration and self-attribution.  Journal of experimental psychology: Human perception and performance, 31(1):80.
  * Tsuchiya and Saigo, (2021) Tsuchiya, N. and Saigo, H. (2021).  A relational approach to consciousness: categories of level and contents of consciousness.  Neuroscience of Consciousness, 2021(2):niab034.
  * (34) Yuan, Y. (2023a).  On the power of foundation models.  In International Conference on Machine Learning. PMLR.
  * (35) Yuan, Y. (2023b).  Succinct representations for concepts.  arXiv preprint arXiv:2303.00446.