#  Succinct Representations for Concepts

Yang Yuan  IIIS, Tsinghua University Shanghai Artificial Intelligence
Laboratory  Shanghai Qi Zhi Institute

###### Abstract

Foundation models like chatGPT have demonstrated remarkable performance on
various tasks. However, for many questions, they may produce false answers
that look accurate. How do we train the model to precisely understand the
concepts? In this paper, we introduce succinct representations of concepts
based on category theory. Such representation yields concept-wise invariance
properties under various tasks, resulting a new learning algorithm that can
provably and accurately learn complex concepts or fix misconceptions.
Moreover, by recursively expanding the succinct representations, one can
generate a hierarchical decomposition, and manually verify the concept by
individually examining each part inside the decomposition.

##  1 Introduction

The existing foundation models like chatGPT are powerful, but with a
significant challenge: they may generate hard-to-notice false statements if
they do not know the correct answer. Ideally, the model should precisely
understand the concepts, and honestly state what it knows and what it does not
know, enabling users to trust its responses with confidence.

One way to teach models a concept is learning through different tasks. For
example, if a model can draw different images of dogs, accurately
differentiate between dogs and cats, correctly identify that dogs have four
legs and two ears, and competently perform various other dog-related tasks, it
is very likely that the model understands what a dog is. However, this
approach may not be exhaustive, as some important cases may not be covered.
Theoretically, we can conclude that the model has perfectly understood the
concept, only if the model passes _all possible_ tasks, which is usually
computationally infeasible to learn or evaluate.

In this paper, we show that it is possible to generate succinct
representations of a given concept. By doing so, one effectively reduces the
problem of passing all possible tasks to finding the correct representation.
Such reduction keeps the functionality of the concept invariant, in the sense
that replacing a concept with its representation will generate equivalent
outputs for all possible tasks. Based on this invariance property, we
introduce a new learning algorithm that can provably and accurately learn
complex concepts or fix misconceptions, given enough sampled tasks.

The verification of a given concept can also be significantly simplified by
the reduction from evaluation of all possible tasks to evaluation of the
representation. However, there is no free lunch. The representation is
succinct, but for accurate verification, one has to recursively evaluate the
sub-concepts inside the representation. The recursive expansion of such
representation has finite depth, because not all concepts have succinct
representations. For example, what is “love”? What is “time”? These basic
concepts cannot be further decomposed. In other words, we decompose the
challenging task of verifying a complex concept, into verifying a hierarchical
decomposition with basic concepts as the leaf nodes that also require
verification.

However, it is worth pointing out that the foundation models, like chatGPT,
have accurate understanding of the relative simple concepts due to their
pretraining on large-scale data sets, as tested with billions of tasks.
Therefore, the learning or verification process of a given concept may stop at
any level of the hierarchical decomposition, when the remaining parts seem
obviously correct.

Formally, what is a concept? Given a category111Readers not familiar with
category theory may check Section 2 for a basic introduction. $\mathcal{C}$,
it seems natural to treat all the objects in $\mathcal{C}$ as concepts. For
example, when $\mathcal{C}$ is the category of natural language, both “wall”
and “clock” are basic concepts, and their combination “wall clock” is also a
concept. However, not all concepts are representable, or equivalently, have a
formal name in $\mathcal{C}$. By contrast, the category of presheaves is much
larger than $\mathcal{C}$, and the presheaves can capture the unrepresentable
concepts. Specifically, presheaves can be seen as a natural extension of
objects, as every object $X$ corresponds to a presheaf
$\mathrm{Hom}_{\mathcal{C}}(\cdot,X)$ (or
$\mathrm{Hom}_{\mathcal{C}}(X,\cdot)$), but the reverse is not true.
Therefore, in this paper, a concept is a presheaf.

Our concept reduction adopts the categorical perspective, where two things are
considered equivalent if they are isomorphic. Indeed, consider the case of
“wall clock”, which can be represented as “clock that can be put on the wall”.
Since “buying a wall clock” is equivalent to (or isomorphic to) “buying a
clock that can be put on the wall”, this representation passes the task of
being used in the sentence “buying a  ”.

If isomorphism implies equivalence, why should we even care about the tasks?
It seems that for a given concept $A$, it suffices to find any $X\simeq A$ as
the representation of $A$. However, if $X$ is not a decomposition, knowing the
equivalence between $X$ and $A$ does not provide sufficient information of the
model’s knowledge of $A$. Moreover, even if $X\simeq A$ is a decomposition,
the sub-concepts in $X$ have to perfectly replace $A$ in all possible tasks,
to make the hierarchical learning or verification of $A$ feasible. Hence, the
task-wise isomorphisms are necessary in addition to the object-wise
isomorphism.

In order to characterize these task-wise isomorphisms, we have to first define
the set of all possible tasks. Categorically, we care about two kinds of
tasks: morphisms and functors. For a given concept $A\in\mathcal{C}$, the
morphisms are $\mathrm{Hom}_{\mathcal{C}}(\cdot,A)$ or
$\mathrm{Hom}_{\mathcal{C}}(A,\cdot)$ that captures the relationship between
$A$ and another object in $\mathcal{C}$, while a functor
$F:\mathcal{C}\rightarrow\mathcal{C}^{\prime}$ computes
$F(A)\in\mathcal{C}^{\prime}$ as the output.

A succinct representation for a concept $A$ is valid, only if for any task,
replacing $A$ with the representation generates an equivalent result. As we
will show, there are two kinds of representations that meet this criterion,
denoted as $``\underrightarrow{\lim}"\alpha$ and
$``\underleftarrow{\lim}"\beta$, for $\alpha:I\rightarrow\mathcal{C}$ and
$\beta:I^{\mathrm{op}}\rightarrow\mathcal{C}$. While the formal definitions of
the representations are difficult to comprehend, we notice that they can be
informally treated as the “definition” and “induction” of the concepts, the
two handy tools that people use everyday. The definition or induction for a
given concept are not necessarily unique, just like the limits in this paper.

Within category theory, the equivalence of limits and concept representations
seems nothing more than a surprising tautology. However, we focus on the
learning scenario, which makes the story completely different. Specifically,
our goal is to learn a category $\mathcal{C}$ with a foundation model
$f:\mathcal{C}\rightarrow{\mathcal{C}^{\wedge}}$ (or ${\mathcal{C}^{\vee}}$),
see Yuan,  (2022). In this setting, the morphisms, functors and limits are not
known, so $f$ is not necessarily accurate. We assume the existence of a limit
extractor $g$, which outputs the limit $g(f,A)$ for concept $A$. Such
extractor is already included in foundation models like chatGPT. If you use
the prompt like “Please define $A$”, or “Can you provide typical examples of
$A$?”, you will get high quality (but not necessarily accurate) description
correspond to a projective or inductive limit.

Since $f$ is not accurate, the invariance properties existed in $\mathcal{C}$
become the ideal training goals and verification criteria for $f$. Our main
theorem says, $f$ precisely understands a concept $A$, if the invariance
properties hold for the hierarchical decomposition of $A$ obtained by
recursively expanding the succinct representations of (sub-)concepts. This
characterization immediately leads to Algorithm 2, Algorithm 3 and Algorithm
4, which provably learn and verify the concepts for $f$.

Our main contributions in this paper include:

  * •

Draw connections between concept learning and limits from category theory.

  * •

Introduce new algorithms that can provably and accurately learn or verify
concepts.

  * •

Reduce concept verification to evaluating the corresponding hierarchical
concept decomposition, which becomes feasible for human beings when the leaf
nodes are known to be accurate.

##  2 Preliminaries

Category theory is used in almost all areas of mathematics. Here we only
introduce the necessary notions for understanding the results of our paper.
Curious readers may check Mac Lane,  (2013); Riehl,  (2017); Adámek et al.,
(1990) for a more comprehensive introduction.

###  2.1 Category basics

A category $\mathcal{C}$ has a set of objects $\mathrm{Ob}(\mathcal{C})$, and
a set of morphisms $\mathrm{Hom}_{\mathcal{C}}(X,Y)$ from $X$ to $Y$ for every
$X,Y\in\mathrm{Ob}(\mathcal{C})$. Given
$f\in\mathrm{Hom}_{\mathcal{C}}(X,Y),g\in\mathrm{Hom}_{\mathcal{C}}(Y,Z)$, we
define their composition as $g\circ f\in\mathrm{Hom}_{\mathcal{C}}(X,Z)$.
Notice that $\circ$ is associative, i.e., $(h\circ g)\circ f=h\circ(g\circ
f)$. For every $X\in\mathrm{Ob}(\mathcal{C})$, there exists an unique identity
morphism $\mathrm{id}_{X}\in\mathrm{Hom}_{\mathcal{C}}(X,X)$. A morphism
$f:X\rightarrow Y$ is an isomorphism if there exists $g:X\leftarrow Y$ such
that $f\circ g=\mathrm{id}_{Y}$ and $g\circ f=\mathrm{id}_{X}$. In this case,
we say $X$ and $Y$ are isomorphic and write $X\simeq Y$.

We consider a universe $\mathcal{U}$222Check Masaki Kashiwara,  (2006) for the
related definitions.. A category is a $\mathcal{U}$-category, if
$\mathrm{Hom}_{\mathcal{C}}(X,Y)$ is $\mathcal{U}$-small for any
$X,Y\in\mathrm{Ob}(\mathcal{C})$. A $\mathcal{U}$-category category
$\mathcal{C}$ is $\mathcal{U}$-small if $\mathrm{Ob}(\mathcal{C})$ is
$\mathcal{U}$-small. A category $\mathcal{C}$ is $\mathcal{U}$-big, if it is
not a $\mathcal{U}$-category. For simplicity, below we may not explicitly
mention the universe $\mathcal{U}$, and simply write that $\mathcal{C}$ is a
category, a small category, or a big category.

Given a category $\mathcal{C}$, we define its opposite
$\mathcal{C}^{\mathrm{op}}$ by setting
$\mathrm{Ob}(\mathcal{C}^{\mathrm{op}})=\mathrm{Ob}(\mathcal{C})$ and
$\mathrm{Hom}_{\mathcal{C}^{\mathrm{op}}}(X,Y)=\mathrm{Hom}_{\mathcal{C}}(Y,X)$.
Moreover, given
$f\in\mathrm{Hom}_{\mathcal{C}^{\mathrm{op}}}(X,Y),g\in\mathrm{Hom}_{\mathcal{C}^{\mathrm{op}}}(Y,Z)$,
the new composition is $g\mathrel{\overset{\makebox[0.0pt]{\mbox{\tiny
op}}}{\circ}}f=f\circ g\in\mathrm{Hom}_{\mathcal{C}^{\mathrm{op}}}(X,Z)$.

We define Set to be the category of sets, where the objects are sets, and
$\mathrm{Hom}_{\textbf{Set}}(X,Y)$ is the set of all functions with domain $X$
and codomain $Y$. Notice that we ignore the subtleties about the universe for
better presentation, so here just assume that Set does not contain strange
objects like a set containing all sets.

Functor is like a function between two categories. Given two categories
$\mathcal{C},\mathcal{C}^{\prime}$, a functor
$F:\mathcal{C}\rightarrow\mathcal{C}^{\prime}$ maps objects from $\mathcal{C}$
to $\mathcal{C}^{\prime}$ with
$F:\mathrm{Ob}(\mathcal{C})\rightarrow\mathrm{Ob}(\mathcal{C}^{\prime})$ and
morphisms from $\mathcal{C}$ to $\mathcal{C}^{\prime}$ with
$F:\mathrm{Hom}_{\mathcal{C}}(X,Y)\rightarrow\mathrm{Hom}_{\mathcal{C}^{\prime}}(F(X),F(Y))$
for all $X,Y\in\mathcal{C}$, so that $F$ preserves identity and composition.
Formally, we have $F(\mathrm{id}_{X})=\mathrm{id}_{F(X)}$ for all
$X\in\mathcal{C}$, and $F(g\circ f)=F(g)\circ F(f)$ for all $f:X\rightarrow
Y,g:Y\rightarrow Z$. A functor $F:\mathcal{C}\rightarrow\mathcal{C}^{\prime}$
is an isomorphism of categories if there exists
$G:\mathcal{C}^{\prime}\rightarrow\mathcal{C}$ such that $G\circ F(X)=X$ and
$F\circ G(Y)=Y$ for all $X\in\mathcal{C},Y\in\mathcal{C}^{\prime}$, and
similarly for the morphisms. In this case, we say $\mathcal{C}$ and
$\mathcal{C}^{\prime}$ are isomorphic and write
$\mathcal{C}\simeq\mathcal{C}^{\prime}$.

The morphisms of functors, also called the natural transformation, is the way
to transform the functors while preserving the structure. Given two categories
$\mathcal{C},\mathcal{C}^{\prime}$, and two functors $F_{1},F_{2}$ from
$\mathcal{C}$ to $\mathcal{C}^{\prime}$. A morphism of functors
$\theta:F_{1}\rightarrow F_{2}$ has a morphism $\theta_{X}:F_{1}(X)\rightarrow
F_{2}(X)$ for all $X\in\mathcal{C}$ such that for all $f:X\rightarrow
Y\in\mathrm{Hom}_{\mathcal{C}}(X,Y)$, we have
$\theta_{Y}(F_{1}(f)(F_{1}(X)))=F_{2}(f)(\theta_{X}(F_{1}(X)))$. We write
$F_{1}\simeq F_{2}$ if there exists an isomorphism between $F_{1}$ and
$F_{2}$.

A presheaf is a functor from ${\mathcal{C}^{\mathrm{op}}}$ to Set, and
${\mathcal{C}^{\wedge}}$ is the category of presheaves. Similarly, a functor
from ${\mathcal{C}^{\mathrm{op}}}$ to $\textbf{Set}^{\mathrm{op}}$ is called a
$\textbf{Set}^{\mathrm{op}}$-valued presheaf, and ${\mathcal{C}^{\vee}}$ is
the category of $\textbf{Set}^{\mathrm{op}}$-valued presheaves. In this paper
we do not make the differentiation, and name both kinds of functors as
presheaves, and ${\mathcal{C}^{\wedge}},{\mathcal{C}^{\vee}}$ as the
categories of presheaves. Moreover, define
${h_{\mathcal{C}}}(X)\triangleq\mathrm{Hom}_{\mathcal{C}}(\cdot,X)\in{\mathcal{C}^{\wedge}}$,
and
${k_{\mathcal{C}}}(X)\triangleq\mathrm{Hom}_{\mathcal{C}}(X,\cdot)\in{\mathcal{C}^{\vee}}$.
The following lemma is fundamental.

######  Lemma 1 (Yoneda lemma).

Given $X\in\mathcal{C}$ we have,

  1. (1)

For $A\in\mathcal{C}^{\wedge}$,
$\mathrm{Hom}_{\mathcal{C}^{\wedge}}(h_{\mathcal{C}}(X),A)\simeq A(X).$

  2. (2)

For $B\in{\mathcal{C}^{\vee}}$,
$\mathrm{Hom}_{{\mathcal{C}^{\vee}}}(B,{k_{\mathcal{C}}}(X))\simeq B(X).$

Yoneda lemma says ${h_{\mathcal{C}}}(X)$ and ${k_{\mathcal{C}}}(X)$ capture
all the information of $X$. In this paper, when there is no confusion, we may
ignore ${h_{\mathcal{C}}}(\cdot)$ and ${k_{\mathcal{C}}}(\cdot)$, and simply
write $X$ to denote ${h_{\mathcal{C}}}(X)\in{\mathcal{C}^{\wedge}}$ or
${k_{\mathcal{C}}}(X)\in{\mathcal{C}^{\vee}}$. Therefore, Yoneda lemma says,
$\mathrm{Hom}_{\mathcal{C}^{\wedge}}(X,A)\simeq A(X)$ and
$\mathrm{Hom}_{{\mathcal{C}^{\vee}}}(B,X)\simeq B(X).$

A functor $F$ from ${\mathcal{C}^{\mathrm{op}}}$ to Set (or $\mathcal{C}$ to
Set) is representable if there is an isomorphism between
${h_{\mathcal{C}}}(X)$ (or ${k_{\mathcal{C}}}(X)$) and $F$ for some
$X\in\mathcal{C}$. Such $X$ is called a representative of $F$.

Let
$L:\mathcal{C}\rightarrow\mathcal{C}^{\prime},R:\mathcal{C}^{\prime}\rightarrow\mathcal{C}$
be two functors. The pair $(L,R)$ is a pair of adjoint functors, or $L$ is a
left adjoint functor to $R$, or $R$ is a right adjoint functor to $L$, if
there exists an isomorphism from
$\mathcal{C}^{\mathrm{op}}\times\mathcal{C}^{\prime}$ to Set:
$\mathrm{Hom}_{\mathcal{C}^{\prime}}(L(\cdot),\cdot)\simeq\mathrm{Hom}_{\mathcal{C}}(\cdot,R(\cdot)).$

###  2.2 Limits

A diagram of shape $I$ in a category $\mathcal{C}$ is a functor
$F:I\rightarrow\mathcal{C}$, which selects objects in $\mathcal{C}$ correspond
to $\mathrm{Ob}(I)$, that preserve the morphisms in $I$. Given a functor
$\beta:I^{\mathrm{op}}\rightarrow\textbf{Set}$, define its projective limit as
$\underleftarrow{\lim}~{}\beta\triangleq\mathrm{Hom}_{I^{\wedge}}(\text{pt}_{I^{\wedge}},\beta)$,
where $\text{pt}_{I^{\wedge}}(i)=\\{\text{pt}\\}$ for every $i\in I$, and
$\\{\text{pt}\\}$ is the single point set. In other words,
$\underleftarrow{\lim}~{}\beta$ denotes the set of all natural transformations
between $\text{pt}_{I^{\wedge}}$ and $\beta$. Based on this definition for
diagrams in Set, we have the general definition of the limits.

######  Definition 1 (Projective and inductive limits).

Given
$\alpha:I\rightarrow\mathcal{C},\beta:{I^{\mathrm{op}}}\rightarrow\mathcal{C}$
with small $I$, the inductive limit
$\underrightarrow{\lim}~{}\alpha\in{\mathcal{C}^{\vee}}$ and projective limit
$\underleftarrow{\lim}~{}\beta\in{\mathcal{C}^{\wedge}}$ are defined as:

  1. (1)

$\underrightarrow{\lim}~{}\alpha:X\mapsto\underleftarrow{\lim}~{}\mathrm{Hom}_{\mathcal{C}}(\alpha,X)$

  2. (2)

$\underleftarrow{\lim}~{}\beta:X\mapsto\underleftarrow{\lim}~{}\mathrm{Hom}_{\mathcal{C}}(X,\beta)$

Here $\mathrm{Hom}_{\mathcal{C}}(\alpha,X)$ is a functor that maps $i\in I$ to
$\mathrm{Hom}_{\mathcal{C}}(\alpha(i),X)$. Therefore,
$\underleftarrow{\lim}~{}\mathrm{Hom}_{\mathcal{C}}(\alpha,X)$ is a well
defined limit for a diagram in Set. Same argument holds for
$\underleftarrow{\lim}~{}\mathrm{Hom}_{\mathcal{C}}(X,\beta)$.

XY$\displaystyle\\{pt\\}$fgX$\displaystyle\times_{*}$YX$\displaystyle\times$Y(x,
y)XYxy (a) Projective limit: $X\times Y$

XYfg$\displaystyle f^{-1}$b$\displaystyle\in X$ (b) Projective limit:
$f^{-1}b$

X$\displaystyle+$Yx yXYxy (c) Inductive limit: $X+Y$

XYfgY/$\displaystyle\sim$p (d) Inductive limit: quotient set

When the functor $\underrightarrow{\lim}~{}\alpha$ (resp.
$\underleftarrow{\lim}~{}\beta$) is representable, we use the same notation
$\underrightarrow{\lim}~{}\alpha\in\mathcal{C}$ (resp.
$\underleftarrow{\lim}~{}\beta\in\mathcal{C}$) to denote its representative.
If for every diagram from $I$ (or $I^{\mathrm{op}}$) to $\mathcal{C}$,
$\underrightarrow{\lim}~{}\alpha$ (or $\underleftarrow{\lim}~{}\beta$) is
representable, we say $\mathcal{C}$ admits inductive (or projective) limits
indexed by $I$. If $\mathcal{C}$ admits inductive (or projective) limits
indexed by all small categories, we say $\mathcal{C}$ admits small projective
(or inductive) limits. Below we provide a few examples of the limits, where we
use  red to denote the diagram, and  blue to denote the limit of the diagram.

Product $X\times Y$. Given $X,Y\in\mathcal{C}$, $X\times Y$ can be seen as the
limit of two different diagrams.

  1. 1.

It is the projective limit of the functor
$\beta:{I^{\mathrm{op}}}\rightarrow\mathcal{C}$, where ${I^{\mathrm{op}}}$ is
a discrete category of two objects $\\{I_{1},I_{2}\\}$, and $\beta(I_{1})=X$,
$\beta(I_{2})=Y$. It simply means we will map any $(x,y)\in X\times Y$ to
$x\in X$ and $y\in Y$. See Figure 0(a) left.

  2. 2.

It is also the projective limit of the pullback functor
$\beta:{I^{\mathrm{op}}}\rightarrow\mathcal{C}$, where ${I^{\mathrm{op}}}$ is
a category of three objects $\\{I_{1},I_{2},I_{3}\\}$, and $\beta(I_{1})=X$,
$\beta(I_{2})=Y,\beta(I_{3})=\\{\text{pt}\\}$. Moreover, there are two non-
identity morphisms: $m_{1}:I_{1}\rightarrow I_{3}$, $m_{2}:I_{2}\rightarrow
I_{3}$. Denote $f\triangleq\beta(m_{1}),g\triangleq\beta(m_{2})$. Since
$\beta(I_{3})$ is the single point set, we know both $f$ and $g$ are the
collapse morphism that maps everything to the same point. We use the
$X\times_{*}Y$ to represent the projective limit of this diagram. By
definition, we know $X\times_{*}Y=\\{(x,y)\in X\times
Y|f(x)=g(y)=\\{\text{pt}\\}\\}=X\times Y$. See Figure 0(a) right.

Preimage. It is the projective limit of the equalizer functor
$\beta:{I^{\mathrm{op}}}\rightarrow\mathcal{C}$, where ${I^{\mathrm{op}}}$ is
a category of two objects $\\{I_{1},I_{2}\\}$, with two morphisms
$m_{1},m_{2}$ in between. Denote $\beta(I_{1})=X,\beta(I_{2})=Y$, and
$\beta(m_{1})=f,\beta(m_{2})=g$, both are morphisms from $X$ to $Y$. If $g$ is
the morphism that sends everything to $b$, the projective limit of $\beta$ is
the set $\\{(x,y)\in X\times Y|f(x)=g(x)=y=b\\}$, in other words, this set is
isomorphic to the set $f^{-1}b\in X$. See Figure 0(b).

Sum $X+Y$. Given $X,Y\in\mathcal{C}$, $X+Y$ can be seen as the inductive limit
of the function $\alpha:I\rightarrow\mathcal{C}$, where $I$ is a discrete
category of two objects $\\{I_{1},I_{2}\\}$, and
$\alpha(I_{1})=X,\alpha(I_{2})=Y$. It simply means we will combine the
elements in $X$ and $Y$ together, to form a bigger object $X+Y$. See Figure
0(c).

Quotient. It is the inductive limit of the coequalizer functor
$\alpha:I\rightarrow\mathcal{C}$, where $I$ is a category of two objefts
$\\{I_{1},I_{2}\\}$, with two morphisms $m_{1},m_{2}$ in between. Denote
$\alpha(I_{1})=X,\alpha(I_{2})=Y$, and $\alpha(m_{1})=f,\alpha(m_{2})=g$, both
are morphisms from $X$ to $Y$. If we define the equivalence relation as
$f(x)\sim g(x)$ for all $x\in X$, the inductive limit of $\alpha$ becomes the
largest set of the form $\\{p(f(x))~{}|~{}p(f(x))=p(g(x))\text{~{} for
all~{}}x\in X\\}$ for some $p$, which is $Y/\sim$. See Figure 0(d).

###  2.3 Ind-lim and Pro-lim

The limits in the category of presheaves (${\mathcal{C}^{\wedge}}$ and
${\mathcal{C}^{\vee}}$) have special names, defined below.

######  Definition 2 (Ind-lim and Pro-lim).

Given
$\alpha:I\rightarrow{\mathcal{C}^{\wedge}},\beta:I^{\mathrm{op}}\rightarrow{\mathcal{C}^{\vee}}$,
we denote $``\underleftarrow{\lim}"\alpha$ as the inductive limit of $\alpha$,
and $``\underrightarrow{\lim}"\beta$ as the projective limit of $\beta$.

We add quotation marks for these limits, because we want to overload the
notation for the diagrams valued in $\mathcal{C}$. Specifically, for
$\alpha:I\rightarrow\mathcal{C}$, we define
$``\underrightarrow{\lim}"\alpha\triangleq``\underrightarrow{\lim}"({h_{\mathcal{C}}}\circ\alpha)$.
For $\beta:I^{\mathrm{op}}\rightarrow\mathcal{C}$, we define
$``\underleftarrow{\lim}"\beta\triangleq``\underleftarrow{\lim}"({k_{\mathcal{C}}}\circ\beta)$.
In other words, $``\underrightarrow{\lim}"$ and $``\underleftarrow{\lim}"$
have two kinds of inputs, and the quotation marks means the diagram will be
lifted to ${\mathcal{C}^{\wedge}}$ and ${\mathcal{C}^{\vee}}$ if necessary.

######  Lemma 2 (Corollary 2.4.3 in Masaki Kashiwara,  (2006)).

Both ${\mathcal{C}^{\wedge}}$ and ${\mathcal{C}^{\vee}}$ admit small inductive
and small projective limits.

##  3 Concept

We first give the formal definition of a concept, as well as a valid
decomposition of a given concept.

######  Definition 3 (Concept).

Given a category $\mathcal{C}$, a concept $A$ is a presheaf in
${\mathcal{C}^{\wedge}}$ or ${\mathcal{C}^{\vee}}$.

######  Definition 4 (Valid decomposition under $T$).

Given a concept $A$, a decomposition $\mathcal{D_{A}}$ is valid under task $T$
for $A$, if applying $T$ to $A$ is equivalent to applying $T$ to each
subconcept in $\mathcal{D_{A}}$ individually and then combine the results
together with (possibly recursive) application of projective or inductive
limits.

Definition 5 can be seen as the weaker version of saying $T$ commutes with the
decomposition operator. For example, if we use
$\underrightarrow{\lim}~{}\alpha$ to represent the decomposition of $A$, $T$
commutes with $\underrightarrow{\lim}~{}\alpha$ means
$T(\underrightarrow{\lim}~{}\alpha)\simeq\underrightarrow{\lim}~{}T(\alpha)$,
but this decomposition is still valid if
$T(\underrightarrow{\lim}~{}\alpha)\simeq\underleftarrow{\lim}~{}T(\alpha)$.
More generally, the decomposition can be a hierarchical tree, where the
combination step recursively merge the leaf nodes to their fathers, see
Section 4.2 for example.

We consider two kinds of tasks, the morphisms
$\mathrm{Hom}_{\mathcal{C}}(\cdot,A)$ or
$\mathrm{Hom}_{\mathcal{C}}(A,\cdot)$, and the functors
$F:\mathcal{C}\rightarrow\mathcal{C}^{\prime}$. As a result, a valid
decomposition for $A$ maintains all the relationships and functionalities
invariant.

######  Definition 5 (Valid decomposition).

Given a concept $A$, when a decomposition $\mathcal{D_{A}}$ is valid for $A$
under all possible morphisms and functors, we say it is valid for $A$.

###  3.1 Limits as representations

Definition 5 naturally leads to the projective and inductive limits in
$\mathcal{C}$. Indeed, we have the following two lemmas.

######  Lemma 3 (Limits for morphism tasks).

Consider
$\alpha:I\rightarrow\mathcal{C},\beta:I^{\mathrm{op}}\rightarrow\mathcal{C}$
with small $I$. If $\underrightarrow{\lim}~{}\alpha$ and
$\underleftarrow{\lim}~{}\beta$ are representable, respectively, we have

$\mathrm{Hom}_{\mathcal{C}}(\underrightarrow{\lim}~{}\alpha,X)\simeq\underleftarrow{\lim}~{}\mathrm{Hom}_{\mathcal{C}}(\alpha,X)$
$\mathrm{Hom}_{\mathcal{C}}(X,\underleftarrow{\lim}~{}\beta)\simeq\underleftarrow{\lim}~{}\mathrm{Hom}_{\mathcal{C}}(X,\beta)$

Lemma 3 easily follows by Definition 1.

######  Lemma 4 (Limits for functor tasks, Proposition 2.1.10 in Masaki
Kashiwara,  (2006)).

Let $F:\mathcal{C}\rightarrow\mathcal{C}^{\prime}$ be a functor. Assume that:

  * •

$F$ admits a left adjoint $G:\mathcal{C}^{\prime}\rightarrow\mathcal{C}$,

  * •

$\mathcal{C}$ admits projective limits indexed by a small category $I$.

Then $F$ commutes with projective limits indexed by $I$, that is, the natural
morphism
$F(\underleftarrow{\lim}~{}\beta)\rightarrow\underleftarrow{\lim}~{}F(\beta)$
is an isomorphism for any $\beta:I^{\mathrm{op}}\rightarrow\mathcal{C}$.
Similar result holds for inductive limits. If $\mathcal{C}$ admits inductive
limits indexed by $I$, and $F$ admits a right adjoint, then $F$ commutes with
such limits.

Lemma 3 and Lemma 4 indicates that both projective and inductive limits can be
seen as valid decompositions for morphism and functor tasks. However, they are
not perfect for the following reasons:

  1. 1.

Order matters in Lemma 3. The inductive limit operator
$\underrightarrow{\lim}~{}$commutes with $\mathrm{Hom}_{\mathcal{C}}$ only
when it appears in domain, but not in codomain (e.g.,
$\mathrm{Hom}_{\mathcal{C}}(X,\underrightarrow{\lim}~{}\alpha)$). Similarly,
the projective limit operator $\underleftarrow{\lim}~{}$commutes with
$\mathrm{Hom}_{\mathcal{C}}$ only when it appears in codomain, but not in
domain (e.g., $\mathrm{Hom}_{\mathcal{C}}(\underleftarrow{\lim}~{}\beta,X)$).

  2. 2.

The concept has to be representable for both lemmas. Otherwise, the limits are
not well defined in $\mathcal{C}$.

  3. 3.

The functor $F$ must admit a left adjoint in Lemma 4.

###  3.2 Ind-lim and Pro-lim as representations

It turns out that the previous problems can be easily solved in the category
of presheaves. Moreover, from the deep learning perspective, it is more
convenient to work with the feature space, instead of the input space. As
discussed in Yuan,  (2022), the feature space of the neural network can be
seen as the empirical representations for presheaves.

Extending Lemma 2 to presheaves, we immediately have the following lemma:

######  Lemma 5.

If $I$ is small, consider
$\alpha:I\rightarrow{\mathcal{C}^{\wedge}},\beta:I^{\mathrm{op}}\rightarrow{\mathcal{C}^{\vee}}$.
For $A\in{\mathcal{C}^{\wedge}},B\in{\mathcal{C}^{\vee}}$,

$\mathrm{Hom}_{\mathcal{C}^{\wedge}}(``\underrightarrow{\lim}"\alpha,A)\simeq\underleftarrow{\lim}~{}\mathrm{Hom}_{\mathcal{C}^{\wedge}}(\alpha,A)$
$\mathrm{Hom}_{\mathcal{C}^{\vee}}(B,``\underleftarrow{\lim}"\beta)\simeq\underleftarrow{\lim}~{}\mathrm{Hom}_{\mathcal{C}^{\vee}}(B,\beta)$

Notice that $A$ and $B$ are presheaves, therefore they can be treated as
functors in $\mathcal{C}^{\mathrm{op}}\rightarrow\textbf{Set}$ or
$\mathcal{C}^{\mathrm{op}}\rightarrow\textbf{Set}^{\mathrm{op}}$. When
$\alpha$ is representable, denote $X\in\mathcal{C}$ as the representative, we
have $\mathrm{Hom}_{\mathcal{C}^{\wedge}}(X,A)\simeq A(X)$ by Yoneda lemma.
Interestingly, unlike Lemma 3, we have guarantees for the other side of the
morphisms:

######  Lemma 6 ((2.6.1-2) in Masaki Kashiwara,  (2006)).

If $I$ is small, consider
$\alpha:I\rightarrow{\mathcal{C}^{\wedge}},\beta:I^{\mathrm{op}}\rightarrow{\mathcal{C}^{\vee}}$.
For $X\in\mathcal{C}$,

$\mathrm{Hom}_{\mathcal{C}^{\wedge}}(X,``\underrightarrow{\lim}"\alpha)\simeq\underrightarrow{\lim}~{}\mathrm{Hom}_{\mathcal{C}^{\wedge}}(X,\alpha)$
$\mathrm{Hom}_{\mathcal{C}^{\vee}}(``\underleftarrow{\lim}"\beta,X)\simeq\underrightarrow{\lim}~{}\mathrm{Hom}_{\mathcal{C}^{\vee}}(\beta,X)$

Lemma 5 and Lemma 6 indicate that both ind-lims and pro-lims can be seen as
valid decompositions for morphism and Set-valued functor tasks. They are
better than the projective and inductive limits in $\mathcal{C}$ for the
following reasons:

  1. 1.

Order does not matter for the morphisms. Indeed, combining the two lemmas, we
can see for any given object $X\in\mathcal{C}$, $X$ may appear in both domain
and codomain in $\mathrm{Hom}(\cdot,\cdot)$.

  2. 2.

The ind-lims and Pro-lims are always representable, due to Lemma 2.

  3. 3.

There is no restrictions on the functor $F$ in Lemma 5. $F$ can be any functor
in ${\mathcal{C}^{\wedge}}$ or ${\mathcal{C}^{\vee}}$.

For the more general functor $F:\mathcal{C}\rightarrow\mathcal{C}^{\prime}$,
we have the following two lemmas. The first lemma says, as long as
$``\underrightarrow{\lim}"\alpha$ is representable, the decomposition is valid
under any $F$.

######  Lemma 7 (Proposition 2.6.4 in Masaki Kashiwara,  (2006)).

Let $I$ be a small category and $\alpha:I\rightarrow\mathcal{C}$ a diagram.
Assume that $``\underrightarrow{\lim}"\alpha\in{\mathcal{C}^{\wedge}}$ is
represented by $X\in\mathcal{C}$, then for any functor
$F:\mathcal{C}\rightarrow\mathcal{C}^{\prime},\underleftarrow{\lim}~{}(F\circ\alpha)\simeq
F(X)$. Similar results hold for $``\underleftarrow{\lim}"\beta$ with
$\beta:I^{\mathrm{op}}\rightarrow\mathcal{C}$ as well.

Lemma 7 requires $``\underrightarrow{\lim}"\alpha$ to be representable, and
works in $\mathcal{C}$ instead of the category of presheaves. This is not
convenient because foundation models work in the feature space, especially for
prompt tuning or fine tuning. Therefore the next lemma is more relevant.

######  Lemma 8 (Proposition 2.7.1 in Masaki Kashiwara,  (2006)).

Assume $\mathcal{C}$ is small and $\mathcal{C}^{\prime}$ admits small
inductive limits. Given $F:\mathcal{C}\rightarrow\mathcal{C}^{\prime}$, there
exists $\tilde{F}:{\mathcal{C}^{\wedge}}\rightarrow\mathcal{C}^{\prime}$, such
that $\tilde{F}\circ{h_{\mathcal{C}}}\simeq F$ and $\tilde{F}$ commutes with
small inductive limits:
$\tilde{F}(``\underrightarrow{\lim}"\alpha)\simeq\underrightarrow{\lim}~{}(F\circ\alpha)$
for any $\alpha:I\rightarrow\mathcal{C}$ with $I$ small. Similar results hold
for ${\mathcal{C}^{\vee}}$and $``\underleftarrow{\lim}"\beta$ as well.

Lemma 8 directly works in the category of presheaves, and shows that the
decompositions are valid under any functor $F$. Therefore, it becomes clear
that Ind-lims and Pro-lims are better representations for concepts. We have
the following definitions to distinguish basic and complex concepts.

######  Definition 6 (Basic concept).

A concept $A$ is basic, if $A$ is representable and there is no
$\alpha:I\rightarrow\mathcal{C}$ such that
$``\underrightarrow{\lim}"\alpha\simeq A$, and no
$\beta:I^{\mathrm{op}}\rightarrow\mathcal{C}$ such that
$``\underleftarrow{\lim}"\beta\simeq A$.

We remark that $\alpha,\beta$ are valued in $\mathcal{C}$ instead of
${\mathcal{C}^{\wedge}}$ or ${\mathcal{C}^{\vee}}$. This is to make
hierarchical decompositions possible, as we will discuss in Section 4.1.

######  Definition 7 (Complex concept).

A concept $A$ is complex, if there exists $\alpha:I\rightarrow\mathcal{C}$
such that $``\underrightarrow{\lim}"\alpha\simeq A$, or
$\beta:I^{\mathrm{op}}\rightarrow\mathcal{C}$ such that
$``\underleftarrow{\lim}"\beta\simeq A$.

Interestingly, if a presheaf is not representable, and not isomorphic to any
limits, it is neither a basic concept nor a complex concept. We do not
consider these cases in our paper. Therefore, when we talk about a concept
below, we assume it is either basic or complex.

###  3.3 Concept transfer

If the concepts are defined as limits, can we transfer concepts from one
category to another? This directly relates to how functors preserve or reflect
limits.

######  Definition 8 (Preservation and reflection of limits).

Given $\alpha:I\rightarrow\mathcal{C}$ with small $I$,
$F:\mathcal{C}\rightarrow\mathcal{C}^{\prime}$, we say

  * •

$F$ preserves the limits $\underrightarrow{\lim}~{}\alpha$, if whenever
$\underrightarrow{\lim}~{}\alpha$ exists,
$F(\underrightarrow{\lim}~{}\alpha)\simeq\underrightarrow{\lim}~{}F(\alpha)$.

  * •

$F$ reflects the limits, if whenever $\underrightarrow{\lim}~{}F(\alpha)\simeq
F(X)$, we know $\underrightarrow{\lim}~{}\alpha$ exists and
$\underrightarrow{\lim}~{}\alpha\simeq X$.

Similar definitions hold for the projective limits as well.

“$F$ preserves the limits” is equivalent to “$F$ commutes with the limits”.
Therefore by Lemma 8, $\tilde{F}$ preserves limits. If we treat the limits as
concepts, there are many interesting implications. For example, consider the
CLIP model (Radford et al., , 2021), which learns a functor that connects the
text category and image category (Yuan, , 2022). The current implementation of
CLIP relies on the InfoNCE loss, which constructs a similarity graph on the
image and text objects. If one injects more complicated semantics during
training, so that both categories can have deeper concepts, by Lemma 8, we
will be able to generate more semantically meaningful images with a limit-
preserving functor, and generate more comprehensive text descriptions with a
limit-reflecting functor.

###  3.4 Definitions, examples and analogies

Intuitively, the projective limit and inductive limit can be seen as two ways
of learning new concepts: learn by definition and learn by examples.
Specifically, the projective limit captures the underlying elements supporting
the concept, and the inductive limit captures the similarities shared by the
underlying examples in the representation.

Making analogies is another typical way of learning new concepts. It can be
seen as making a transformation between the limits of two diagrams
$\alpha:I\rightarrow\mathcal{C}$ and $\beta:J\rightarrow\mathcal{C}^{\prime}$.
If we are very familiar with one limit (concept), with the help of this
transformation we may quickly grasp the other concept as well. Therefore, the
similarity between two concepts can be modeled as the similarity between the
two diagrams. However, this transformation does not decompose a given concept,
so is not counted as a succinct representation.

Being able to make analogies is important. For example, human beings may ask
chatGPT dangerous questions like “how to destroy the world?” We certainly do
not want AI to answer this question. The researchers tried hard to filter
these dangerous questions, but empirically the users are creative enough to
generate semantically different but inherently similar questions to bypass the
filter. To solve this problem, we may prevent AI from answering any questions
containing concepts that are similar to the dangerous ones. On the other hand,
if the question contains concepts that are very different to any dangerous
concepts, it shall be safe to speak out the answer.

##  4 Algorithms

###  4.1 Hierarchical decomposition

We have seen that a complex concept can be decomposed using Ind-lims and Pro-
lims. How can we do it recursively? Notice that ${h_{\mathcal{C}}}$ commutes
with projective limits but not inductive limits, and ${k_{\mathcal{C}}}$
commutes with inductive limits but not projective limits. If we want to
include both kinds of limits in our decomposition, we cannot stay inside the
same category. Otherwise the presheaves we operate will not have a
representative in $\mathcal{C}$, therefore they cannot be easily understood by
human beings.

To solve this problem, we only consider $``\underrightarrow{\lim}"\alpha$ and
$``\underleftarrow{\lim}"\beta$ for
$\alpha:I\rightarrow\mathcal{C},\beta:I^{\mathrm{op}}\rightarrow\mathcal{C}$.
The diagrams are valued in $\mathcal{C}$ instead of ${\mathcal{C}^{\wedge}}$
or ${\mathcal{C}^{\vee}}$ to make the hierarchical decomposition feasible.
Specifically, after each decomposition, we obtain a diagram on the objects
representable in $\mathcal{C}$. If further decompositions are needed, we may
use ${h_{\mathcal{C}}}$ (or ${k_{\mathcal{C}}}$) to lift the object to
${\mathcal{C}^{\wedge}}$ (or ${\mathcal{C}^{\vee}}$), depending on whether it
is a projective or inductive limit. Notice that ${h_{\mathcal{C}}}$ commutes
with projective limits, so we can directly decompose it in
${\mathcal{C}^{\wedge}}$ (same for ${k_{\mathcal{C}}}$). Using this trick,
recursive decomposition is possible, and we may jump between
${\mathcal{C}^{\wedge}}$ and ${\mathcal{C}^{\vee}}$ during the decomposition.
See Algorithm 1 for details, which assumes the existence of a limit extractor
defined below.

######  Definition 9 (Limit extractor).

Given a foundation model $f$, a concept $A$, a limit extractor $g$ outputs a
limit $g(f,A)$ that contains a limit operator and a diagram, i.e.,
$``\underrightarrow{\lim}"\alpha$ for $\alpha:I\rightarrow\mathcal{C}$ or
$``\underleftarrow{\lim}"\beta$ for
$\beta:I^{\mathrm{op}}\rightarrow\mathcal{C}$.

In Definition 9, we did not assume $``\underrightarrow{\lim}"g(f,A)\simeq A$
or $``\underleftarrow{\lim}"g(f,A)\simeq A$, because $g$ is a function that we
have to implement. Ideally, it shall output a diagram whose limit is
equivalent to $A$, but empirically it may make mistakes. It is possible that
there are different limits for a concept, and here we assume that $g$ picks a
random one, or follows some specific rules.

Algorithm 1 Deconcept

Input: foundation model $f$, concept $A$ in ${\mathcal{C}^{\wedge}}$ or
${\mathcal{C}^{\vee}}$, limit extractor $g$

Let $\mathcal{L_{A}}=g(f,A)$

if $\mathcal{L_{A}}$ not null then

Let $\mathcal{D_{A}}=\\{\\}$

for $A_{i}$ in $\mathcal{L_{A}}$ do

Let $L_{A_{i}}=g(f,A_{i})$

if $L_{A_{i}}$ not null then

if $L_{A_{i}}$ is projective limit then

Let
$\mathcal{D_{A}}=\mathcal{D_{A}}+$Deconcept($f,{h_{\mathcal{C}}}(A_{i}),g$)

else

Let
$\mathcal{D_{A}}=\mathcal{D_{A}}+$Deconcept($f,{k_{\mathcal{C}}}(A_{i}),g$)

end if

else

Let $\mathcal{D_{A}}=\mathcal{D_{A}}+{A_{i}}$

end if

end for

return $\mathcal{D_{A}}$

else

return $A$

end if

###  4.2 Decomposition example

Consider we want to decompose the concept “dynamic programming”, which may
have both (one or more) inductive and projective limits. As the projective
limit, it means “an algorithm that breaks down the original problem into sub-
problems, and solve the sub-problems recursively”, represented by a diagram.
As the inductive limit, it may contain different types of dynamic programming,
e.g., DP on one-dimensional array, DP on two-dimensional array, DP on higher-
dimensional array, DP on graphs, etc.

Most of the sub-concepts can be further decomposed. For example, The concept
of “sub-problem recursion” may have a projective limit that describes how to
recursively run a sub-problem until the boundary case. In this description,
there may exists a sub-concept like “for-loop”, which is similar to another
concept “while-loop” by making a diagram analogy. The concept of “DP on one-
dimensional array” may have a projective limit describing how to transform the
problem into the one-dimensional array, how to store the optimal state, how to
define the recursion, etc.

With large amount of training data, it is possible that the diagrams used by
the foundation models will contain much more details, but the overall
structure of the hierarchical decomposition should similar.

###  4.3 Main Results

If $\mathcal{D_{A}}$ is generated using Algorithm 1, it is not necessarily
valid because the limit extractor may make mistakes. Our main theorem says
that, being able to recursively generating the correct limits of each concept
in $\mathcal{D_{A}}$ is equivalent to precise comprehension of the concept.

######  Theorem 1 (Main theorem).

A foundation model $f$ with the limit extractor $g$, precisely understands
$A$, if every non-leaf node in $\mathcal{D_{A}}\triangleq{Deconcept}(f,A,g)$
is isomorphic to the limit extraction given by $g$.

Applying Theorem 1 for learning $f$, we have the following corollary. See
Algorithm 2 for details, where $T(\mathcal{L_{A}})$ means applying $T$ to each
subconcept in $\mathcal{L_{A}}$ and then combine the results together.

Algorithm 2 Concept learning

Input: foundation model $f$, a concept $A$ in ${\mathcal{C}^{\wedge}}$ or
${\mathcal{C}^{\vee}}$, limit extractor $g$

for $i=1$ to $m$ do

Sample a task $T$ for $A$

Let $\mathcal{L_{A}}=g(f,A)$

Let $\ell(T,A,\mathcal{L_{A}})=d(T(A),T(\mathcal{L_{A}}))$ for a similarity
function $d$

Run backpropagation on $\ell(T,A,\mathcal{L_{A}})$ to optimize $f$ and $g$

end for

######  Corollary 1.

To learn $f$ that precisely understand the concepts, one should learn a limit
extractor $g$ such that given any concept $A$, for all possible task $T$,
$g(f,A)$ is valid under $T$.

Algorithm 2 can be easily modified to fix a misconception, or learn a new
concept. Specifically, if we know the correct representation for a concept
$A$, denoted as $\mathcal{L_{A}}^{*}$, we may simply run supervised learning
for $f$ and $g$, so that $\mathcal{L_{A}}^{*}=g(f,A)$.

Apply Theorem 1 for verifying a concept $A$ with $f$, we have the following
corollary.

Algorithm 3 Concept verification with tasks

Input: foundation model $f$, a concept $A$ in ${\mathcal{C}^{\wedge}}$ or
${\mathcal{C}^{\vee}}$, limit extractor $g$

Let $\mathcal{L_{A}}=g(f,A)$

if $\mathcal{L_{A}}$ not null then

for $i=1$ to $m$ do

Sample a task $T$ for $A$

Let $\ell(T,A,\mathcal{L_{A}})=d(T(A),T(\mathcal{L_{A}}))$ for a similarity
function $d$

Return failure if $\ell(T,A,\mathcal{L_{A}})\geq\epsilon$

end for

end if

Return success

######  Corollary 2.

To verify that $f$ has precisely understood a concept $A$ with limit extractor
$g$, one should use $g$ to produce a decomposition $\mathcal{D_{A}}$, and
verify the decomposition of every non-leaf node in $\mathcal{D_{A}}$ is a
projective or inductive limit.

There are two kinds of algorithms for verifying a concept. Algorithm 3 is
similar to Algorithm 2 as it verifies concepts by sampling various tasks. Both
Algorithm 2 and Algorithm 3 can be extended to the hierarchical decomposition
by calling Algorithm 1, which we omit here.

Algorithm 4 Concept verification with verifier

Input: foundation model $f$, a concept $A$ in ${\mathcal{C}^{\wedge}}$ or
${\mathcal{C}^{\vee}}$, limit extractor $g$, a limit verifier $v$

let $\mathcal{D_{A}}=$Deconcept($f,A,g$)

for non-leaf concept $B$ in $\mathcal{D_{A}}$ do

Let $\mathcal{L_{B}}=g(f,B)$

Return failure if $v(\mathcal{L_{B}},B)=0$

end for

Return success

Algorithm 4 assumes the existence of a limit verifier, defined below.

######  Definition 10 (Limit verifier).

Given a foundation model $f$, a concept $A$, a limit extractor $g$, a limit
verifier $v$ is a function that verifies $g(f,A)$ by computing
$v(g(f,A),A)\triangleq I_{g(f,A)\simeq A}$.

The limit verifier can be an existing well trained foundation model, which
serves as a teacher that teaches the current model $f$. It can also be a
human-being, who verifies whether the limit $g(f,A)$ is true (especially for
math concepts). It is worth pointing out that we assume that $f$ precisely
understands the leaf nodes, just like how chatGPT understands basic concepts
by training with billions of tasks. The correctness of Algorithm 4 easily
follows by this assumption and Theorem 1.

##  5 Discussions

Almost all the branches of modern mathematics can be described using category
theory, and complicated math concepts are defined using limits from the
objects in various categories. A theorem (or lemma) can be seen as an
isomorphism between two concepts. Therefore, it is conceivable that AI will be
able to solve math problems after accurately learning the concepts using our
algorithms.

There are numerous other applications as well. For example, it can be used for
programming. Each subroutine of the program has certain functionality, which
corresponds to a concept. If we can learn a functor that transfer the concept
from the natural language category to the program category, AI will be able to
write code automatically with language hints. Although chatGPT has
demonstrated amazing power on generating code, it will be able to think deeper
after learning the concepts.

Moreover, our characterization for the concepts as presheaves easily explains
why the existing foundation models are super good at creating plausible and
non-existent stories on well-known characters. For example, if we ask chatGPT
to “tell what Odysseus will say to Confucius”, it will generate a long
conversation based on their respective beliefs and values. Such conversation
certainly never existed before in the training data, but the model understands
Odysseus and Confucius by learning other related materials, and encodes them
into high dimensional vectors to approximate their corresponding functors
(i.e. presheaves) that can predict their behaviors for all possible tasks,
including a hypothetical conversation. In other words, when the model sees
words like “Odysseus” or “Confucius”, it sees not only the words, but also all
the relationships and properties, just like our human beings.

## References

  * Adámek et al., (1990) Adámek, J., Herrlich, H., and Strecker, G. (1990).  Abstract and concrete categories.  Wiley-Interscience.
  * Mac Lane, (2013) Mac Lane, S. (2013).  Categories for the working mathematician, volume 5.  Springer Science & Business Media.
  * Masaki Kashiwara, (2006) Masaki Kashiwara, P. S. (2006).  Categories and Sheaves.  Springer.
  * Radford et al., (2021) Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021).  Learning transferable visual models from natural language supervision.  In International Conference on Machine Learning, pages 8748–8763. PMLR.
  * Riehl, (2017) Riehl, E. (2017).  Category theory in context.  Courier Dover Publications.
  * Yuan, (2022) Yuan, Y. (2022).  On the power of foundation models.  arXiv preprint arXiv:2211.16327.