# The Hurewicz map in motivic homotopy theory

Utsav Choudhury  and  Amit Hogadi

###### Abstract.

For an ${\mathbb{A}}^{1}$-connected pointed simplicial sheaf ${\mathcal{X}}$
over a perfect field $k$, we prove that the Hurewicz map
$\pi_{1}^{{\mathbb{A}}^{1}}({\mathcal{X}})\to
H_{1}^{{\mathbb{A}}^{1}}({\mathcal{X}})$ is surjective. We also observe that
the Hurewicz map for ${\mathbb{P}}^{1}_{k}$ is the abelianisation map. In the
course of proving this result, we also show that for any morphism $\phi$ of
strongly ${\mathbb{A}}^{1}$-invariant sheaves of groups, the image and kernel
of $\phi$ are also strongly ${\mathbb{A}}^{1}$-invariant.

###### 2000 Mathematics Subject Classification:

14F42

##  1\. Introduction

For a field $k$, let $Sm/k$ denote the category of smooth $k$-varieties with
Nisnevich topology. Let $\Delta^{op}Sh(Sm/k)$ denote the category of
simplicial sheaves on the category $Sm/k$. This cateogry with its
${\mathbb{A}}^{1}$-model structure as defined in [6] is one of the main
objects of study in ${\mathbb{A}}^{1}$-homotopy theory. For any pointed
simplicial sheaf ${\mathcal{X}}$ in $\Delta^{op}Sh(Sm/k)$ one defines the
${\mathbb{A}}^{1}$-homotopy group sheaves,
$\pi_{i}^{{\mathbb{A}}^{1}}({\mathcal{X}})$, to be the sheaves of simplicial
homotopy groups of a fibrant replacement of ${\mathcal{X}}$ in the
${\mathbb{A}}^{1}$-model structure. Morel, in his foundational work in [5, Ch.
6] has defined, for every integer $i$, ${\mathbb{A}}^{1}$-homology groups
$H_{i}^{{\mathbb{A}}^{1}}({\mathcal{X}})$ and canonical Hurewicz morphisms

$\pi_{i}^{{\mathbb{A}}^{1}}({\mathcal{X}})\to
H_{i}^{{\mathbb{A}}^{1}}({\mathcal{X}})$

The above maps are analogous to the Hurewicz map that we have in topology. In
the topological setup, the Hurewicz morphism for $i=1$ is known to be the
abelianisation when the underlying space is connected. We will refer to this
result as the Hurewicz theorem. Hurewicz theorem is expected in
${\mathbb{A}}^{1}$-homotopy theory (see [5, 6.36]), but not yet known. However
the following theorem by Morel is the closest known result to the Hurewicz
theorem.

######  Theorem 1.1.

[5, 6.35] For a connected simplicial sheaf ${\mathcal{X}}$, the Hurewicz
morphism

$\pi_{1}^{{\mathbb{A}}^{1}}({\mathcal{X}})\to
H_{1}^{{\mathbb{A}}^{1}}({\mathcal{X}})$

is a universal map to a strictly ${\mathbb{A}}^{1}$-invariant sheaf of abelian
groups.

Recall that a sheaf of groups ${\mathcal{G}}$ is called strongly
${\mathbb{A}}^{1}$-invariant if for $i=0,1$ the maps

$H^{i}(U,{\mathcal{G}})\to H^{i}(U\times{\mathbb{A}}^{1},{\mathcal{G}})$

are bijective for all $U$ in $Sm/k$. If ${\mathcal{G}}$ is abelian, then it is
called strictly ${\mathbb{A}}^{1}$-invariant if the above isomorphism holds
for all $i\geq 0$. $\pi_{1}^{{\mathbb{A}}^{1}}({\mathcal{X}})$ is strongly
${\mathbb{A}}^{1}$-invariant and $H_{1}^{{\mathbb{A}}^{1}}({\mathcal{X}})$ is
known to be strictly ${\mathbb{A}}^{1}$-invariant (see [5, 6.1, 6.23]).

In topology, the surjectivity of the Hurewicz map is almost a direct
consequence of the definitions. This is not the case in
${\mathbb{A}}^{1}$-homotopy theory. The main source of difficulty lies in the
non-explicit nature of ${\mathbb{A}}^{1}$-fibrant replacements; non-explicit
from the viewpoint of making explicit calculations. In this paper we prove
this surjectivity by using Giraud’s theory of non-abelian cohomology.

######  Theorem 1.2.

Let $k$ be a perfect field and ${\mathcal{X}}$ be a pointed simplicial sheaf
on $Sm/k$ in the Nisnevich topology. Then the Hurewicz map
$\pi_{1}^{{\mathbb{A}}^{1}}({\mathcal{X}})\to
H_{1}^{{\mathbb{A}}^{1}}({\mathcal{X}})$ is surjective.

The above theorem will be deduced from the following result, which is of
independent interest.

######  Theorem 1.3.

Let $k$ be a perfect field. Let $G$ be a strongly ${\mathbb{A}}^{1}$-invariant
sheaf of groups on $Sm/k$ and $G\to H$ be an epimorphism. Then $H$ is strongly
${\mathbb{A}}^{1}$-invariant iff it is ${\mathbb{A}}^{1}$-invariant.

######  Remark 1.4.

If $k$ is perfect field, a theorem of Morel [5, 5.46] says that any strongly
${\mathbb{A}}^{1}$-invariant Nisnevich sheaf of abelian groups on $Sm/k$ is
also strictly ${\mathbb{A}}^{1}$-invariant. Unfortunately it is not yet known
if this statement holds for imperfect fields. This is the sole reason for
assuming $k$ to be perfect in Theorems 1.3 and 1.2. Also note that strongly
${\mathbb{A}}^{1}$-invariant is a stronger notion than just
${\mathbb{A}}^{1}$-invariant. In particular, there exists
${\mathbb{A}}^{1}$-invariant sheaves which are not strongly
${\mathbb{A}}^{1}$-invariant (see [7, Lemma 5.6]).

For a morphism of strongly ${\mathbb{A}}^{1}$-invariant abelian sheaves over a
perfect field, the kernel and image of the morphism are also strongly
${\mathbb{A}}^{1}$-invariant. This result is a consequence of a nontrivial
theorem of Morel (see [5, 6.24]) that the category of strongly
${\mathbb{A}}^{1}$-invariant sheaves of abelian groups is an abelian category,
as it is obtained as a heart of a $t$-structure. The theorem below, can be
viewed as a generalization of this result for non-abelian strongly
${\mathbb{A}}^{1}$-invariant sheaves. Moreover the proof of this
generalization is completely different and is more direct in the sense that it
does not appeal to the existence of $t$-structures.

######  Theorem 1.5.

Let $G\xrightarrow{\phi}H$ be a morphism of strongly
${\mathbb{A}}^{1}$-invariant sheaves of groups. Then the image and the kernel
of $\phi$ are strongly ${\mathbb{A}}^{1}$-invariant.

###### Proof.

The image ${\rm Image}(\phi)$ is ${\mathbb{A}}^{1}$-invariant, since it is a
subsheaf of an ${\mathbb{A}}^{1}$-invariant sheaf $H$. Thus by Theorem 1.3 it
is strongly ${\mathbb{A}}^{1}$-invariant. The kernel $K$ is strongly
${\mathbb{A}}^{1}$-invariant as it fits in the following exact sequence

$1\to K\to G\to{\rm Image(\phi)}\to 1$

where the other two sheaves are strongly ${\mathbb{A}}^{1}$-invariant. ∎

Acknowledgement: We thank Fabien Morel, Tom Bachmann and O. Röndigs for their
comments.

##  2\. Preliminaries on Gerbes and Giraud’s non-abelian cohomology

Let ${\mathcal{C}}$ be any small site (e.g. $Sm/k$ with Nisnevich topology)
and $\Delta^{op}({\mathcal{C}})$ be the category of simplicial sheaves on
${\mathcal{C}}$. The goal of this section is to recall the main results of
Giraud on non-abelian cohomology. Everything in this section is a subset of
[3]. We work with the following definition of a gerbe.

######  Definition 2.1.

A simplicial sheaf ${\mathcal{X}}$ on ${\mathcal{C}}$ is called a gerbe if it
is connected and if for any $U\in{\mathcal{C}}$ and any
$x\in{\mathcal{X}}(U)$, the homotopy sheaves of groups
$\pi_{i}({\mathcal{X}}_{|U},x)=0$ for all $i\in{\mathbb{N}}$.

Given any simplicial sheaf ${\mathcal{X}}$ (not necessarily a gerbe), one gets
a category fibered in groupoids over ${\mathcal{C}}$ defined by the
fundamental groupoid construction: i.e. for every $U\in{\mathcal{C}}$, the
fiber category ${\mathcal{X}}_{|U}$ is the fundamental groupoid of the space
${\mathcal{X}}(U)$, a category whose objects are elements of
${\mathcal{X}}(U)$ and morphisms are paths up to homotopy. This category
fibered in groupoids is in fact a gerbe in the sense of [4, 3.2] if
${\mathcal{X}}$ is connected. If ${\mathcal{X}}$ was a gerbe to start with,
then it can be recovered, up to weak equivalence, using this category fibered
in groupoids using the simplicial nerve construction. A gerbe ${\mathcal{X}}$,
is called neutral if it has a global section. In this case, by making a choice
of a global section, one can define the fundamental group of ${\mathcal{X}}$.
Since ${\mathcal{X}}$ is connected, a different choice gives a fundamental
group which can be canonically identified with the previous one, modulo an
inner automorphism.

This motivates the following definition by Giraud.

######  Definition 2.2.

[3, 1.1.3] For two sheaves of groups $F$ and $G$, let ${\rm Isex}(F,G)$ denote
the set of isomorphisms from $F$ onto $G$ modulo the action of inner
automorphisms of $F$ (acting on the left) and the action of inner
automorphisms of $G$.

Consider the pre-stack whose objects over $U$ are sheaves of groups over $U$
(small w.r.t to a fixed universe) with morphisms between $F$ and $G$ defined
as elements of ${\rm Isex}(F,G)$. One can stackify this pre-stack (see [4])
and objects of this stack are called bands. In particular, every sheaf of
groups defines a band. Since every band is represented locally by a sheaf of
groups, all those concepts related to sheaves of groups which are local in
nature (e.g. exact sequence, epimorphism, kernel, center) also make sense for
bands. It is a simple exercise to show that the center of a band is
necessarily represented by a sheaf of groups. The ’fundamental group’ of any
gerbe ${\mathcal{X}}$ (neutral or not) is always defined as a band. For a band
$L$, a gerbe banded by $L$ (or simply an $L$-gerbe) will mean a gerbe together
with an isomorphism of $L$ with the band defined by ${\mathcal{X}}$. An
equivalence of $L$-gerbes means an equivalence of the gerbes compatible with
the given isomorphisms of their bands with $L$.

######  Definition 2.3.

[3, 3.1.1] For a band $L$ on a site ${\mathcal{C}}$, let
$H^{2}({\mathcal{C}},L)$ or simply $H^{2}(L)$ denote the equivalence class of
$L$-gerbes. The subset represented by neutral classes in
$H^{2}({\mathcal{C}},L)$ is denoted by $H^{2}({\mathcal{C}},L)^{\prime}$ or
simply ($H^{2}(L)^{\prime}$).

######  Remark 2.4.

Note that $H^{2}(L)^{\prime}$ is non-empty if and only if $L$ can be
represented by a sheaf of groups (see [3, 3.2.4]), in which case it is a
singleton set, as can be seen for e.g. by Theorem 2.8 stated below.

######  Remark 2.5.

If a band $L$ is representable by a sheaf of abelian groups $A$, then
$H^{2}(L)$ defined above is in canonical bijection with the $H^{2}(A)$ as
defined by sheaf cohomology [3, 3.4].

Given an exact sequence of sheaves of groups

$1\to A\xrightarrow{a}B\xrightarrow{b}C\to 1$

one has a long exact sequence

$1\to H^{0}(A)\to H^{0}(B)\to H^{0}(C)\to H^{1}(A)\to H^{1}(B)\to H^{1}(C).$

One of the goals of introducing the non-abelian $H^{2}$ is to extend this
exact sequence on the right. We first note that $B$ acts on itself by inner
automorphisms. Since $A$ is normal in $B$, this action also induces an action
on $A$. Thus $B$ also acts on ${\mathsf{band}}(A)$, where ${\mathsf{band}}(A)$
denotes the band defined by $A$. This action factors through $C$.

We need the following definition to state the result extending the above long
exact sequence to $H^{2}$:

######  Definition 2.6.

[3, 4.2.3] For an epimorphism of bands $v:M\to N$, one defines a pointed set

${\mathsf{O}}(v):=N(v)/R$

where

  1. (1)

$N(v)$ is the set of all triples $(K,L,u)$ where $K$ is a gerbe with band $L$
and $u:L\to M$ is a monomorphism which makes the following sequence exact

$1\to L\xrightarrow{u}M\xrightarrow{v}N\to 1.$

  2. (2)

$R$ is the equivalence relation defined by declaring
$(K,L,u)\sim(K^{\prime},L^{\prime},u^{\prime})$ if there exists a morphism of
gerbes $\alpha:K\to K^{\prime}$ such that the induced morphism $\alpha:L\to
L^{\prime}$ on bands makes the following diagram commute

$\textstyle{L\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{\alpha}$$\scriptstyle{u}$$\textstyle{M\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\textstyle{L\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{u^{\prime}}$$\textstyle{M}$

  3. (3)

${\mathsf{O}}(v)^{\prime}$ denotes the subset of ${\mathsf{O}}(v)$ defined by
all those $(K,L,u)$ where $K$ is a neutral $L$-gerbe.

Now one has the following general result.

######  Theorem 2.7.

[3, 4.2.8, 4.2.10] Given an exact sequence of sheaves of groups $1\to
A\xrightarrow{f}B\xrightarrow{g}C\to 1$, we have the following long exact
sequence

$1\to H^{0}(A)\to H^{0}(B)\to H^{0}(C)\to H^{1}(A)\to H^{1}(B)\to
H^{1}(C)\xrightarrow{d}{\mathsf{O}}(g)\to H^{2}(B)\to H^{2}(C).$

where the map $d$ is as defined in [3, 4.7.2.4] and exactness of the sequence
is defined similar to that in the case of pointed sets with subsets
${\mathsf{O}}(g)^{\prime}$ and $H^{2}(B)^{\prime}$, $H^{2}(C)^{\prime}$
playing the role of base points. When the action of $C$ on
${\mathsf{band}}(A)$ is trivial, on has a canonical bijection
${\mathsf{O}}(g)\cong H^{2}(A)$.

Another important result of Giraud we need is the following.

######  Theorem 2.8.

[3, 3.3.3] Let $L$ be a band and $C$ be its center. Then one has a canonical
action of $H^{2}(C)$ on $H^{2}(L)$ which is free and transitive.

This theorem, loosely speaking, says that the non-abelian cohomology set
$H^{2}(L)$ is essentially all abelian as it ”comes from” its center. Note
however that if $H^{2}(L)$ has no class represented by a neutral gerbe, then
there is no canonical bijection between $H^{2}(L)$ and $H^{2}(C)$.

The following is a direct consequence of the above theorem and the definition
of

######  Lemma 2.9.

Let $1\to A\xrightarrow{f}B\xrightarrow{g}C\to 1$ be an exact sequence of
sheaves of groups. Assume that $H^{2}(A)$ is trivial. Then
${\mathsf{O}}(g)^{\prime}={\mathsf{O}}(g)$.

###### Proof.

Let $(K,L,u)$ be any triple where $L$ is a band which fits in the exact
sequence

$1\to L\xrightarrow{u}{\mathsf{band}}(B)\to{\mathsf{band}}(C)\to 1$

and $K$ is an $L$-gerbe. To prove the theorem it is enough to show that $K$ is
neutral. However we note that center $Z(L)$ coincides with $Z(A)$, the center
of $A$. The result then directly follows from the above theorem. ∎

##  3\. Strong ${\mathbb{A}}^{1}$-invariance of the center and applications

The goal of this section is to prove the theorems mentioned in the
introduction. We start by proving the following.

######  Theorem 3.1.

Let $G$ be a strongly ${\mathbb{A}}^{1}$-invariant sheaf of groups on $Sm/k$.
Then $Z(G)$, the center of $G$ is also strongly ${\mathbb{A}}^{1}$-invariant.

###### Proof.

Since $G$ is strongly ${\mathbb{A}}^{1}$-invariant, $BG$ is
${\mathbb{A}}^{1}$-local. By choosing a simplicially fibrant model for $BG$ we
may further assume, without loss of generality, that $BG$ is
${\mathbb{A}}^{1}$-fibrant. To show $Z(G)$ is strongly
${\mathbb{A}}^{1}$-invariant, we need to show $BZ(G)$ is
${\mathbb{A}}^{1}$-local. Let $BZ(G)\xrightarrow{u}{\mathcal{X}}$ be an
${\mathbb{A}}^{1}$-fibrant replacement (in the category of pointed spaces).
Thus $u$ is a trivial ${\mathbb{A}}^{1}$-cofibration. To prove the theorem, it
suffices to show that $u$ is a simplicial weak equivalence. Equivalently, it
suffices to show that the map on sheaves of fundamental groups

$\pi_{1}(BZ(G))\longrightarrow\pi_{1}({\mathcal{X}})(=:H)$

is an isomorphism. Since $BG$ is ${\mathbb{A}}^{1}$-fibrant and $u$ is a
trivial ${\mathbb{A}}^{1}$-cofibration, we have a factorization $h$ as below

$\textstyle{BZ(G)\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{u}$$\textstyle{BG}$$\textstyle{{\mathcal{X}}\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{\exists
h}$

which gives a commutative diagram of the maps induces on the fundamental
groups

$\textstyle{Z(G)\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{u_{*}}$$\textstyle{G}$$\textstyle{H\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{h_{*}}$

∎

In the above diagram, if we show that the image of $h_{*}$ is $Z(G)$, then it
will follow that $Z(G)$ is a retract of the strongly
${\mathbb{A}}^{1}$-invariant sheaf $H$ and hence is itself strongly
${\mathbb{A}}^{1}$-invariant. Thus it suffices to show that image of $h_{*}$
is contained in $Z(G)$. This is equivalent to showing that for every smooth
$k$-scheme $U$ and an element $g\in G(U)$, the map $h_{|U}$ is homotopic to
the composite of

${\mathcal{X}}_{|U}\xrightarrow{h_{|U}}BG_{|U}\xrightarrow{x\mapsto
gxg^{-1}}BG_{|U}.$

But note that the base change functor from
$\Delta^{op}({\mathcal{S}}h(k))\to\Delta^{op}({\mathcal{S}}h(U))$ preserves
trivial ${\mathbb{A}}^{1}$-cofibrations since it is a left quillen functor.
Moreover it also preserves ${\mathbb{A}}^{1}$-fibrations. Thus

$BZ(G)_{|U}\xrightarrow{\nu_{|U}}{\mathcal{X}}_{|U}$

is a trivial ${\mathbb{A}}^{1}$-cofibration. Since $Z(G)$ is the center of
$G$, the map $BZ(G)_{|U}\xrightarrow{\nu_{|U}}BG_{|U}$ is homotopic (in fact
equal) to the composite

$BZ(G)_{|U}\to BG_{|U}\xrightarrow{x\mapsto gxg^{-1}}BG_{|U}.$

The proof now follows from commutative diagram below, using the fact that
$\nu_{\scriptscriptstyle{|U}}$ is an ${\mathbb{A}}^{1}$-weak equivalence and
the fact that $BG_{|U}$ is ${\mathbb{A}}^{1}$-local by Lemma 3.2.

$\textstyle{BZ(G)_{|U}\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{\nu_{\scriptscriptstyle{|U}}}$$\textstyle{BG_{|U}\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{x\mapsto
gxg^{-1}}$$\textstyle{BG_{|U}}$$\textstyle{{\mathcal{X}}_{|U}\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{h_{|U}}$

######  Lemma 3.2.

$U\in Sm/k$. Then the restriction functor
$\Delta^{op}({\mathcal{S}}h(k))\to\Delta^{op}({\mathcal{S}}h(U))$ takes
${\mathbb{A}}^{1}$-fibrant objects to ${\mathbb{A}}^{1}$-local objects.

###### Proof.

Let ${\mathcal{Y}}\in\Delta^{op}({\mathcal{S}}h(k))$ be an
${\mathbb{A}}^{1}$-fibrant object. Then ${\mathcal{Y}}_{|U}$ has BG property,
since BG property is defined in terms of Nisnevich distinguished triangles and
every Nisnevich distinguished triangle in the category $Sm/U$ is also a
Nisnevich disntinguished triangle in $Sm/k$. Moreover, since
${\mathcal{Y}}(V\times{\mathbb{A}}^{1})\to{\mathcal{Y}}(V)$ is a weak
equivalence for every $V/U$. Thus by arguments given as in [5, A.6],
${\mathcal{Y}}_{|U}$ is ${\mathbb{A}}^{1}$-local. ∎

###### Proof of 1.3.

Let $K$ denote the kernel of the the epimorphism $G\to H$. Thus we have a
short exact sequence of Nisnevich sheaves of groups

$1\to K\to G\to H\to 1.$

Step 1: For every smooth $k$-scheme $U$, this gives us an exact sequence (see
[3, 3.3.1]) of pointed cohomology sets

$1\to H^{0}(U,K)\to H^{0}(U,G)\to H^{0}(U,H)\to H^{1}(U,K)\to H^{1}(U,G)\to
H^{1}(U,H)$

Using functoriality of the above exact sequence in the case when $U$ is Hensel
local, we deduce that the ${\mathbb{A}}^{1}$-invariance of $H$ implies (in
fact is equivalent to) strong ${\mathbb{A}}^{1}$-invariance of $K$. By Theorem
3.1, $Z(K)$, the center of $K$ is strictly ${\mathbb{A}}^{1}$-invariant sheaf.

Step 2: To show strong ${\mathbb{A}}^{1}$-invariance of $H$, it is enough to
show that for all henselian local essentially smooth schemes $U/k$,
$H^{1}(U\times{\mathbb{A}}^{1}_{k},H)$ is trivial. By [3, 4.7.2.4], we have an
exact sequence of pointed sets

$\to H^{1}(U\times{\mathbb{A}}^{1}_{k},G)\to
H^{1}(U\times{\mathbb{A}}^{1}_{k},H)\to O(\phi)$

where $\phi$ denotes restriction of $G\to H$ to the over-category
$(Sm/k)/U\times{\mathbb{A}}^{1}$ which will be denoted by
$Sm_{k}/U\times{\mathbb{A}}^{1}$ for simplicity and $O(\phi)$ is as defined in
2.6. It is enough to show $O(\phi)$ is trivial. This follows from Lemma 2.9
and the strict ${\mathbb{A}}^{1}$-invariance of $Z(K)$.

∎

Let ${\mathcal{F}}$ be a sheaf on $Sm/k$. For $U\in Sm/k$ we say
$\alpha,\beta\in{\mathcal{F}}(U)$ are naive ${\mathbb{A}}^{1}$-homotopic if
there exists a $\gamma\in{\mathcal{F}}(U\times{\mathbb{A}}^{1})$ such that

$\alpha=\sigma_{0}^{*}(\gamma)\ \ \text{and}\ \ \beta=\sigma_{1}^{*}(\gamma)$

where $\sigma_{0},\sigma_{1}:U\to U\times{\mathbb{A}}^{1}$ are the sections
defined by $0$ and $1$ respectively. As in [2, 2.9], let $S({\mathcal{F}})$
denote the sheaf associated to the presheaf

$U\mapsto\frac{{\mathcal{F}}(U)}{\sim}$

where $\sim$ denotes the equivalence relation generated by naive
${\mathbb{A}}^{1}$-homotopies. There is a canonical epimorphism
${\mathcal{F}}\to S({\mathcal{F}})$. Let

$S^{\infty}({\mathcal{F}}):=\lim_{n\to\infty}S^{n}({\mathcal{F}})$

The following lemma is straightforward to check.

######  Lemma 3.3.

(see [2, 2.13]) The canonical morphism ${\mathcal{F}}\to
S^{\infty}({\mathcal{F}})$ is an epimorphism and is a universal map from
${\mathcal{F}}$ to an ${\mathbb{A}}^{1}$-invariant sheaf. Moreover if
${\mathcal{F}}$ is a sheaf of groups, then so is $S^{\infty}({\mathcal{F}})$.

###### Proof of Theorem 1.2.

By lemma 3.3, the map

$\pi_{1}^{{\mathbb{A}}^{1}}({\mathcal{X}})\xrightarrow{h}H_{1}^{{\mathbb{A}}^{1}}({\mathcal{X}})$

factors uniquely through

$\pi_{1}^{{\mathbb{A}}^{1}}({\mathcal{X}})\xrightarrow{s}S^{\infty}\left(\pi_{1}^{{\mathbb{A}}^{1}}({\mathcal{X}})^{ab}\right).$

Since the map $s$ is an epimorphism, Theorem 1.3 implies that
$S^{\infty}\left(\pi_{1}^{{\mathbb{A}}^{1}}({\mathcal{X}})^{ab}\right)$ is
strongly ${\mathbb{A}}^{1}$-invariant sheaf of abelian groups. Thus $s$ is a
universal map to strictly ${\mathbb{A}}^{1}$-invariant sheaf of abelian
groups. However by [5, 6.35], so is $h$. Thus the induced map from
$S^{\infty}\left(\pi_{1}^{{\mathbb{A}}^{1}}({\mathcal{X}})^{ab}\right)\to
H_{1}^{{\mathbb{A}}^{1}}({\mathcal{X}})$ must be an isomorphism. In particular
$h$ must be an epimorphism. ∎

##  4\. Hurewicz map for ${\mathbb{P}}^{1}_{k}$

In this section we reserve the notation $H$ to denote the Hurewicz map for
${\mathbb{P}}^{1}$, i.e.
$\pi_{1}^{{\mathbb{A}}^{1}}({\mathbb{P}}^{1})\xrightarrow{H}H^{{\mathbb{A}}^{1}}_{1}({\mathbb{P}}^{1})$.
The goal of this section is to prove the following propositions

######  Proposition 4.1.

The kernel of the Hurewicz map for ${\mathbb{P}}^{1}_{k}$,

$\pi_{1}^{{\mathbb{A}}^{1}}({\mathbb{P}}^{1})\xrightarrow{H}H^{{\mathbb{A}}^{1}}_{1}({\mathbb{P}}^{1})$

is equal to the commutator subgroup of
$\pi_{1}^{{\mathbb{A}}^{1}}({\mathbb{P}}^{1})$.

As a consequence of the explicit computation of the Hurewicz map we obtain the
following :

######  Proposition 4.2.

The sequence of Nisnevich sheaves

$0\to
h\underline{K}_{2}^{MW}\to\underline{K}_{2}^{MW}\xrightarrow{\eta}\underline{K}_{1}^{MW}$

is exact.

######  Remark 4.3.

We do not know if there is an elementary way to prove the above proposition,
using generator and relations. In particular we do not know if

$0\to
h\underline{K}_{n}^{MW}\to\underline{K}_{n}^{MW}\xrightarrow{\eta}\underline{K}_{n-1}^{MW}$

is exact for every $n\geq 1$. However, as pointed out to us by O. Röndigs, it
is possible that the above short exact sequence is induced by a cofiber
sequence given in [1, Prop. 11].

The most difficult part of the computation in the above propositions is the
universality of the Hurewicz map and the computation of
$\pi_{1}^{{\mathbb{A}}^{1}}({\mathbb{P}}^{1})$ itself, both of which has been
elegantly done in [5, 6.35, 7.3]. We first restate Morel’s computation of
$\pi_{1}^{{\mathbb{A}}^{1}}({\mathbb{P}}^{1})$ as it will also help us to
build notation for use in subsequent calculation.

Let $F^{{\mathbb{A}}^{1}}(1):=\pi_{1}^{{\mathbb{A}}^{1}}({\mathbb{P}}^{1})$.
First we recall the following two maps defined by Morel:

  1. (1)

A map $\theta:{\mathbb{G}}_{m}\to F^{{\mathbb{A}}^{1}}(1)$ which is a result
of an ${\mathbb{A}}^{1}$-equivalence
${\mathbb{P}}^{1}_{k}\sim\Sigma({\mathbb{G}}_{m})$.

  2. (2)

A map $\underline{K}_{2}^{MW}\to F^{{\mathbb{A}}^{1}}(1)$ which is a result of
applying $\pi_{1}$ to the map ${\mathbb{A}}^{2}-0\to{\mathbb{P}}^{1}$ and a
theorem of Morel which shows
$\underline{K}_{2}^{MW}\cong\pi_{1}^{{\mathbb{A}}^{1}}({\mathbb{A}}^{2}-0)$.

In what follows, we will freely use standard notation for denoting elements of
$K_{*}^{MW}$ used in [5, Chapter 3], e.g. $\left<-1\right>,h,[U]$ etc.

######  Theorem 4.4.

[5, 7.29] As a sheaf of sets $F^{{\mathbb{A}}^{1}}(1)$ is a product
$K_{2}^{MW}\times{\mathbb{G}}_{m}$ and the following describes the structure
of $F^{{\mathbb{A}}^{1}}(1)$ completely:

  1. (i)

The sequence

$1\to\underline{K}_{2}^{MW}\to
F^{{\mathbb{A}}^{1}}(1)\xrightarrow{\gamma}{\mathbb{G}}_{m}\to 1$

is exact and is a central extension.

  2. (ii)

For two units $U,V$ in any field extension $F$ of $k$, the following hold

$\theta(U)\theta(V)^{-1}=[-U][-V]\theta(U^{-1}V)^{-1}$
$\theta(U)^{-1}\theta(V)=[U^{-1}][-V]\theta(U^{-1}V).$

The following is the main calculation in the proof of Proposition 4.1.

######  Lemma 4.5.

For any essentially smooth field extension $F/k$, the commutator subgroup of
$F^{{\mathbb{A}}^{1}}(1)(F)$ is equal to $hK_{2}^{MW}(F)$.

###### Proof.

Since $K_{2}^{MW}(F)$ is in the center of $F^{{\mathbb{A}}^{1}}(1)(F)$, we
have that the commutator subgroup

$[F^{{\mathbb{A}}^{1}}(1)(F),F^{{\mathbb{A}}^{1}}(1)(F)]=\left<\theta(U)\theta(V)\theta(U)^{-1}\theta(V)^{-1}|\
U,V\in F\right>$

where the angle brackets on RHS denote subgroup generated by the elements
within.

${\begin{split}\theta(U)\theta(V)&=\left<-1\right>[U][V]\theta(UV)&...(\text{by
\cite[cite]{[\@@bibref{}{morel}{}{}, 7.31]}})\\\
\theta(V)\theta(U)&=\left<-1\right>[U][V]\theta(UV)&\\\
\theta(U)\theta(V)\theta(U)^{-1}\theta(V)^{-1}&=\left<-1\right>[U][V]\theta(UV)\cdot\big{(}-\left<-1\right>[V][U]\theta(UV)^{-1}\big{)}&...(\text{by
\cite[cite]{[\@@bibref{}{morel}{}{}, 7.29(ii)]}})\\\
&=\left<-1\right>\big{(}[U][V]-[V][U])&...(\because K_{2}^{MW}\text{is in the
center})\\\
&=[U][V]\big{(}\left<-1\right>+\left<-1\right>^{2}\big{)}&...(\text{using
\cite[cite]{[\@@bibref{}{morel}{}{}, 3.7(3)]}})\\\ &=h(h-1)[U][V]&...(\because
h=1+\left<-1\right>)\end{split}}$

Thus we have

${\begin{split}[F^{{\mathbb{A}}^{1}}(1)(F),F^{{\mathbb{A}}^{1}}(1)(F)]&=\left<h(h-1)[U][V]\
|\ U,V\in F\right>&\\\ &=h(h-1)K_{2}^{MW}(F)&...(\text{using
\cite[cite]{[\@@bibref{}{morel}{}{}, 3.6(1)]}})\\\ &=h\cdot
K_{2}^{MW}(F)&...(\because h-1=\left<-1\right>\text{is a unit by
\cite[cite]{[\@@bibref{}{morel}{}{}, 3.5(4)]}})\end{split}}$

∎

###### Proof of Proposition 4.1.

Recall that $H$ is a universal map from
$\pi_{1}^{{\mathbb{A}}^{1}}({\mathbb{P}}^{1})$ to a strongly
${\mathbb{A}}^{1}$-invariant sheaf of abelian groups. Thus it is enough to
show that the abelianisation of $\pi_{1}^{{\mathbb{A}}^{1}}({\mathbb{P}}^{1})$
is strongly ${\mathbb{A}}^{1}$-invariant.
$\pi_{1}^{{\mathbb{A}}^{1}}({\mathbb{P}}^{1})^{ab}$ is a quotient of a
strongly ${\mathbb{A}}^{1}$-invariant sheaf therefore by Theorem 1.3, we need
to show that $\pi_{1}^{{\mathbb{A}}^{1}}({\mathbb{P}}^{1})^{ab}$ is homotopy
invariant. However by (4.4) and Lemma 4.5,
$\pi_{1}^{{\mathbb{A}}^{1}}({\mathbb{P}}^{1})^{\rm ab}$ fits in the following
exact sequence

$0\to\frac{\underline{K}_{2}^{MW}}{h\underline{K}_{2}^{MW}}\to\pi_{1}^{{\mathbb{A}}^{1}}({\mathbb{P}}^{1})^{ab}\to{\mathbb{G}}_{m}\to
0.$

Thus it is enough to show that
$\frac{\underline{K}_{2}^{MW}}{h\underline{K}_{2}^{MW}}$ is
${\mathbb{A}}^{1}$-invariant or equivalently $h\underline{K}_{2}^{MW}$ is
strongly ${\mathbb{A}}^{1}$-invariant. However this follows from [5, 3.32]. ∎

The following lemmas give explicit description of the Hurewicz morphism $H$.

######  Lemma 4.6.

There exists an isomorphism
$\phi:H_{1}^{{\mathbb{A}}^{1}}({\mathbb{P}}^{1})\cong\underline{K}_{1}^{MW}$
such that

$\phi\circ H([U][V]\theta(W))=\eta[U^{-1}][V]+[W]$

where $U,V,W$ are sections of ${\mathbb{G}}_{m}$ on an object in
${\mathbb{G}}_{m}$.

###### Proof.

Since there is an ${\mathbb{A}}^{1}$-equivalence
${\mathbb{P}}^{1}_{k}\cong\Sigma{\mathbb{G}}_{m}$, we have

$H_{1}^{{\mathbb{A}}^{1}}({\mathbb{P}}^{1}_{k})\cong
H_{1}^{{\mathbb{A}}^{1}}(\Sigma{\mathbb{G}}_{m})\cong\widetilde{H}_{0}^{{\mathbb{A}}^{1}}({\mathbb{G}}_{m})$

where the second isomorphism is due to ${\mathbb{A}}^{1}$-suspension theorem
for homology [5, 6.30]. In the above statement ${\mathbb{G}}_{m}$ is
considered as a sheaf of sets pointed by $1$. By the definition of
${\mathbb{A}}^{1}$-homology groups,
$\widetilde{H}_{0}^{{\mathbb{A}}^{1}}({\mathbb{G}}_{m})$ is the strictly
${\mathbb{A}}^{1}$-invariant sheaf of abelian groups generated by the pointed
sheaf ${\mathbb{G}}_{m}$. By [5, 3.2], this must be isomorphic to
$\underline{K}_{1}^{MW}$. Now recall that we have the following commutative
diagram

$\textstyle{{\mathbb{G}}_{m}\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{U\mapsto[U]}$$\scriptstyle{\theta}$$\textstyle{\pi_{1}(\Sigma{\mathbb{G}}_{m})\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\textstyle{\pi_{1}^{{\mathbb{A}}^{1}}({\mathbb{P}}^{1}_{k})\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{H}$$\textstyle{\underline{K}_{1}^{MW}}$$\textstyle{\widetilde{H}_{0}^{{\mathbb{A}}^{1}}({\mathbb{G}}_{m})\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{\sim}$$\textstyle{H_{1}^{{\mathbb{A}}^{1}}(\Sigma{\mathbb{G}}_{m})\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{\sim}$$\scriptstyle{\phi}$

The diagram commutes because all morphisms in this diagram are a result of
some universal property. The commutativity of the diagram gives us the formula

(1) $\phi(H([W]))=[W]$

Now, using the following equality proved in [5, 7.29(1)]

$\theta(U^{-1})^{-1}\theta(U^{-1}V)\theta(V)^{-1}=[U][V]$

we get

${\begin{split}\phi(H([U][V]\theta(W)))&=\phi(H(\theta(U^{-1})^{-1}\theta(U^{-1}V)\theta(V)^{-1}\theta(W)))&\\\
&=-[U^{-1}]+[U^{-1}V]-[V]+[W]&...(\text{using eqn(1) above})\\\
&=\eta[U^{-1}][V]+[W]&...(\text{using \cite[cite]{[\@@bibref{}{morel}{}{},
3.1(2)]}})\end{split}}$

∎

To state the next lemma, recall that $\underline{K}_{2}^{MW}$ is the free
strongly ${\mathbb{A}}^{1}$-invariant sheaf of abelian groups generated by the
pointed sheaf of sets ${\mathbb{G}}_{m}\wedge{\mathbb{G}}_{m}$. Thus any
automorphism of ${\mathbb{G}}_{m}\wedge{\mathbb{G}}_{m}$ gives rise to an
automorphism of $\underline{K}_{2}^{MW}$. In particular, the automorphism of
$\underline{K}_{2}^{MW}$ induced by

${\begin{split}{\mathbb{G}}_{m}\wedge{\mathbb{G}}_{m}&\longrightarrow{\mathbb{G}}_{m}\wedge{\mathbb{G}}_{m}\\\
(U,V)&\mapsto(U^{-1},V)\end{split}}$

will be denoted by $\tau$.

######  Lemma 4.7.

Let $\tau:\underline{K}_{2}^{MW}\to\underline{K}_{2}^{MW}$ denote the
automorphism which sends $[U][V]\mapsto[U^{-1}][V]$. Then the restriction of
the Hurewicz map $H$ to $\underline{K}_{2}^{MW}$ coincides with the
composition

$\underline{K}_{2}^{MW}\xrightarrow{\tau}\underline{K}_{2}^{MW}\xrightarrow{\cdot\eta}\underline{K}_{1}^{MW}$

###### Proof.

This follows directly from the explicit formula for $H$ in the above lemma. ∎

###### Proof of Proposition 4.2.

By universality of $H$, as noted before, the map
$F^{{\mathbb{A}}^{1}}(1)\xrightarrow{\gamma}{\mathbb{G}}_{m}$ factors through
$H$. Thus

${\rm Ker}(H)\subset{\rm Ker}(\gamma)=\underline{K}_{2}^{MW}.$

${\rm Ker}(H)$ must coincide with the kernel of restriction of $H$ to
$\underline{K}_{2}^{MW}$. Now the proposition follows from Lemmas 4.5 and 4.7.
∎

## References

  * [1] A. Ananyevskiy, O. Röndigs, P. Østvær ; On very effective hermitian K-theory. Math. Z. 294 (2020), no. 3-4, 1021–1034.
  * [2] Chetan Balwe, Amit Hogadi and Anand Sawant; $A^{1}$-connected components of schemes. Adv. Math. 282 (2015), 335–361.
  * [3] Giraud, Jean; Cohomologie non abélienne. Die Grundlehren der mathematischen Wissenschaften, Band 179. Springer-Verlag, Berlin-New York, 1971.
  * [4] G. Laumon, L. Moret-Bailly; Champs algébriques. Ergebnisse der Mathematik und ihrer Grenzgebiete. 3. Folge. 39. Springer-Verlag, Berlin, 2000.
  * [5] F. Morel, ${\mathbb{A}}^{1}$-algebraic topology over a field, LNM 2052, Springer, Heidelberg, 2012.
  * [6] F. Morel, V. Voevodsky, A1-homotopy theory of schemes. Inst. Hautes Études Sci. Publ. Math. No. 90 (1999), 45–143 (2001).
  * [7] U. Choudhury; Connectivity of motivic $H$-spaces. Algebraic and Geometric Topology 14 (2014) 37–55.


# Go with the flow:
Adaptive Control for Neural ODEs

Mathieu Chalvidal1,2,3, Matthew Ricci4, Rufin VanRullen1,3, Thomas Serre1,2
1Artificial and Natural Intelligence Toulouse Institute, Universite de
Toulouse, France
2Carney Institute for Brain Science, Dpt. of Cognitive Linguistic &
Psychological Sciences
Brown University, Providence, RI 02912
3Centre de Recherche Cerveau & Cognition CNRS, Université de Toulouse
4Data Science Initiative, Brown University, Providence, RI 02912
{mathieu_chalvid, mgr<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

Despite their elegant formulation and lightweight memory cost, neural ordinary
differential equations (NODEs) suffer from known representational limitations.
In particular, the single flow learned by NODEs cannot express all
homeomorphisms from a given data space to itself, and their static weight
parameterization restricts the type of functions they can learn compared to
discrete architectures with layer-dependent weights. Here, we describe a new
module called neurally-controlled ODE (N-CODE) designed to improve the
expressivity of NODEs. The parameters of N-CODE modules are dynamic variables
governed by a trainable map from initial or current activation state,
resulting in forms of open-loop and closed-loop control, respectively. A
single module is sufficient for learning a distribution on non-autonomous
flows that adaptively drive neural representations. We provide theoretical and
empirical evidence that N-CODE circumvents limitations of previous NODEs
models and show how increased model expressivity manifests in several
supervised and unsupervised learning problems. These favorable empirical
results indicate the potential of using data- and activity-dependent
plasticity in neural networks across numerous domains.

##  1 Introduction

The interpretation of artificial neural networks as continuous-time dynamical
systems has led to both theoretical and practical advances in representation
learning. According to this interpretation, the separate layers of a deep
neural network are understood to be a discretization of a continuous-time
operator so that, in effect, the net is infinitely deep. One important class
of continuous-time models, neural ordinary differential equations (NODEs)
(Chen et al., 2018), have found natural applications in generative variational
inference (Grathwohl et al., 2019) and physical modeling (Köhler et al., 2019;
Ruthotto et al., 2020) because of their ability to take advantage of black-box
differential equation solvers and correspondence to dynamical systems in
nature.

Nevertheless, NODEs suffer from known representational limitations, which
researchers have tried to alleviate either by lifting the NODE activation
space to higher dimensions or by allowing the transition operator to change in
time, making the system non-autonomous (Dupont et al., 2019). For example,
Zhang et al. (2020) showed that NODEs can arbitrarily approximate maps from
$\mathbb{R}^{d}$ to $\mathbb{R}$ if NODE dynamics operate with an additional
time dimension in $\mathbb{R}^{d+1}$ and the system is affixed with an
additional linear layer. The same authors showed that NODEs could approximate
homeomorphisms from $\mathbb{R}^{d}$ to itself if the dynamics were lifted to
$\mathbb{R}^{2d}$. Yet, the set of homeomorphisms from $\mathbb{R}^{d}$ to
itself is in fact quite a conservative function space from the perspective of
representation learning, since these mappings preserve topological invariants
of the data space, preventing them from “disentangling” data classes like
those of the annulus data in Fig. 1 (lower left panel). In general, much
remains to be understood about the continuous-time framework and its
expressive capabilities.

Figure 1: Vector fields for continuous-time neural networks. Integral curves
with arrows show the trajectories of data points under the influence of the
network. Top left: Standard NODEs learn a single time-independent flow (in
black) that must account for the whole data space. Top right: N-CODE learns a
family of vector fields (red vs yellow vs blue), enabling the system to
flexibly adjust the trajectory for every data point. Bottom left: Trained NODE
trajectories of initial values in a data set of concentric annuli, colored
green and yellow. The NODE transformation is a homeomorphism on the data space
and cannot separate the classes as a result. Colored points are the initial
state of the dynamics, and black points are the final state. Bottom right:
Corresponding flows for N-CODE which easily separate the classes. Transiting
from the inner to outer annulus effects a bifurcation which linearly separates
the data.

In this paper, we propose a new approach that we call neurally-controlled ODEs
(N-CODE) designed to increase the expressivity of continuous-time neural nets
by using tools from control theory. Whereas previous continuous-time methods
learn a single, time-varying vector field for the whole input space, our
system learns a _family_ of vector fields parameterized by data. We do so by
mapping the input space to a collection of control weights which interact with
neural activity to optimally steer model dynamics. The implications of this
new formulation are critical for model expressivity.

In particular, the transformation of the input space is no longer constrained
to be a homeomorphism, since the flows associated with each datum are
specifically adapted to that point. Consequently, our system can easily “tear”
apart the two annulus classes in Fig. 1 (lower right panel) without directly
lifting the data space to a higher dimension. Moreover, when control weights
are allowed to vary in time, they can play the role of fast, plastic synapses
which can adapt to dynamic model states and inputs.

The rest of the paper proceeds as follows. First, we will lay out the
background for N-CODE and its technical formulation. Then, we will demonstrate
its efficacy for supervised and unsupervised learning. In the supervised case,
we show how N-CODE can classify data by learning to bifurcate its dynamics
along class boundaries as well as memorize high-dimensional patterns in real-
time using fast synapses. Then, we show how the flows learned by N-CODE can be
used as latent representations in an unsupervised autoencoder, improving image
generation over a base model.

##  2 Background

Neural ODEs (NODEs) (Chen et al., 2018) are dynamical systems of the form

$\frac{d\bm{x}}{dt}=f(\bm{x},\bm{\theta},t),$ (1)

where $\mathcal{X}$ is a space of features, $\bm{\theta}\in\Theta$ is a
collection of learnable parameters, and
$f:\mathcal{X}\times\Theta\times\mathbb{R}\mapsto\mathcal{X}$ is an equation
of motion which we take to be differentiable on its whole domain. $f$ defines
a flow, i.e. a triple $(\mathcal{X},\mathbb{R},\Phi_{\bm{\theta}})$ with
$\Phi_{\bm{\theta}}:\mathcal{X}\times\mathbb{R}\mapsto\mathcal{X}$ defined by

$\Phi_{\bm{\theta}}(\bm{x}(0),T)=\bm{x}(0)+\int\limits_{0}^{T}f(\bm{x}(t),\bm{\theta},t)dt$
(2)

which relates an initial point $\bm{x}(0)$ to an orbit of points
$\\{\bm{x}(t)=\Phi_{\theta}(\bm{x}(0),t),t\in\mathbb{R}\\}$. For a fixed $T$,
the map $x\mapsto\Phi_{\bm{\theta}}(x,T)$ is a homeomorphism from
$\mathcal{X}$ to itelf parametrized by $\bm{\theta}$.

Several properties of such flows make them appealing for machine learning. For
example, the ODEs that govern such such flows can be solved with off-the shelf
solvers and they can potentially model data irregularly sampled in time.
Moreover, such flows are reversible maps by construction whose inverse is just
the system integrated backward in time,
$\Phi_{\theta}(.,t)^{-1}=\Phi_{\theta}(.,-t)$. This property enables depth-
constant memory cost of training thanks to the adjoint sensitivity method
(Pontryagin Lev Semyonovich ; Boltyanskii V G & F, 1962) and the modeling of
continuous-time generative normalizing flow algorithms
(grathwohl2018scalable).

Interestingly, discretizing Eq. 2 yields the recursive formulation of a
residual network (He et al., 2015) with a single residual operator
$f_{\bm{\theta}}$:

$\Phi_{\textit{ResNet}}(\bm{x}(0),T)=\bm{x}(0)+\sum\limits_{t=1}^{T}f_{\bm{\theta}}(\bm{x}_{t-1})$
(3)

In this sense, NODEs with a time-independent (autonomous) equation of motion,
$f$, are the infinitely-deep limit of weight-tied residual networks. Relying
on the fact that every non-autonomous dynamical system with state
$\bm{x}\in\mathbb{R}^{d}$ is equivalent to an autonomous system on the
extended state $(\bm{x},t)\in\mathbb{R}^{d+1}$, Eq. 2 can also be used to
model general, weight-untied residual networks. However it remains unclear how
dependence of $f$ in time should be modeled in practice and how their dynamics
relate to their discrete counterparts with weights evolving freely across
blocks through gradient descent.

##  3 N-CODE: Learning to control data-dependent flows

#### General formulation -

The main idea of N-CODE is to consider the parameters, $\bm{\theta}(t)$, in
Eq. 1 as control variables for the dynamical state, $\bm{x}(t)$. Model
dynamics are then governed by a coupled system of equations on the extended
state $\bm{z}(t)=(\bm{x}(t),\bm{\theta}(t))$. The initial value of the control
weights, $\bm{\theta}(0)$, is given by a mapping
$\gamma:\mathcal{X}\to\Theta$. Throughout, we assume that the initial time
point is $t=0$. The full trajectory of control weights, $\bm{\theta}(t)$, is
then output by a controller, $g$, given by another differentiable equation of
motion $g:\Theta\times\mathcal{X}\times\mathbb{R}\mapsto\Theta$ with initial
condition $\gamma(\bm{x}_{0}).$ Given an initial point, $\bm{x}(0)$, we can
solve the initial value problem (IVP)

$\begin{cases}\dfrac{d\bm{z}}{dt}=h(\bm{z},t)\\\
\bm{z}(0)=\bm{z}_{0}\end{cases}=\;\;\begin{cases}\left(\dfrac{d\bm{x}}{dt},\dfrac{d\bm{\theta}}{dt}\right)=\left(f(\bm{x},\bm{\theta},t),g(\bm{\theta},\bm{x},t)\right)\\\
(\bm{x}(0),\bm{\theta}(0))=(\bm{x}_{0},\gamma(\bm{x}_{0}))\end{cases},$ (4)

Figure 2: Diagram of a general N-CODE module: Given a initial state
$\bm{x}(0)$, the module consists of an augmented dynamical system that couples
activity state $\bm{x}$ and weights $\bm{\theta}$ over time (red arrows). A
mapping $\gamma$ infers initial control weights $\bm{\theta}_{0}$ defining an
initial flow (open-loop control). This flow can potentially evolve in time as
$\bm{\theta}$ might be driven by a feedback signal from $\bm{x}$ (closed-loop,
dotted line). This meta-parameterization of $f$ can be trained with gradient
descent by solving an augmented adjoint sensitivity system (blue arrows).

where $h=(f,g)$. We may think of $g$ and $\gamma$ as a controller of the
dynamical system with equation of motion, $f$. We model $g$ and $\gamma$ as
neural networks parameterized by $\mu\in\mathbb{R}^{n_{\mu}}$ and use gradient
descent techniques where the gradient can be computed by solving an adjoint
problem (Pontryagin Lev Semyonovich ; Boltyanskii V G & F, 1962) that we
describe in the next section. Our goal here is to use the meta-
parameterization in the space $\Theta$ to create richer dynamic behavior for a
given $f$ than directly optimizing fixed weights $\bm{\theta}$.

Well-posedness - If $f$ and $g$ are continuously differentiable with respect
to $\bm{x}$ and $\bm{\theta}$ and continuous with respect to $t$, then, for
all initial conditions $(\bm{x}(0),\bm{\theta}(0))$, there exists a unique
solution $\bm{z}$ for Eq. 4 by the Cauchy-Lipschitz theorem. This result leads
to the existence and uniqueness of the augmented flow
$(\mathcal{X}\times\Theta,\mathbb{R},\Phi_{\bm{\mu}})$ with
$\Phi_{\bm{\mu}}:(\mathcal{X}\times\Theta)\times\mathbb{R}\mapsto\mathcal{X}\times\Theta$.
Moreover, considering the restriction of such a flow on $\mathcal{X}$, we are
now endowed with a universal approximator for at least the set of
homeomorphisms on $\mathcal{X}$ given that this restriction constitutes a non-
autonomous system. We discuss now how $g$ and $\gamma$ affect the evolution of
the variable $\bm{x}(t)$, exhibiting two forms of control and noting how they
relate to previous extensions of NODEs.

###  3.1 open and closed-loop controllers

If the controller outputs control weights as a function of the current state,
$\bm{x}(t)$, then we say it is a _closed-loop_ controller. Otherwise, it is an
_open-loop_ controller.

Open-loop control: First, we consider the effect of using only the mapping
$\gamma$ in Eq. 4 as a controller. Here, $\gamma$ maps the input space
$\mathcal{X}$ to $\Theta$ so that $f$ is conditioned on $\bm{x}(0)$ but not
necessarily on $\bm{x}(t)$ for $t>0$. In other words, each initial value
$\bm{x}(0)$ evolves according to its own learned flow
$(\mathcal{X},\mathbb{R},\Phi_{\bm{\gamma(\bm{x}(0)})})$. This allows for
trajectories to evolve more freely than within a single flow that must account
for the whole data distribution and resolves the problem of non-intersecting
orbits (see Figure 4). Recently, (Massaroli et al., 2020b) proposed a similar
form of data-conditioned open-loop control with extended state
$(\bm{x}(t),\bm{x}(0)))$. This is a version of our method in which $\gamma$ is
of the form $\gamma(\bm{x})=\theta(0)=[C:id]$ with $C$ a constant vector, $id$
is the identity function, and $:$ denotes concatenation. Our open-loop
formulation makes an architectural distinction between controller and dynamics
and is consequently generalizable to the following closed-loop formulation.

$\displaystyle\dfrac{d^{2}\bm{x}}{d^{2}t}=\bm{\mu}(1-\bm{x}^{2})\dfrac{d\bm{x}}{dt}-\bm{x}$
$\displaystyle\iff\begin{cases}\dot{\bm{x}}=\bm{\mu}(\bm{x}-\frac{1}{3}\bm{x}^{3}-\bm{\theta})\\\
\dot{\bm{\theta}}=\frac{1}{\bm{\mu}}\bm{x}\end{cases}$

Figure 3: Example of dynamical control augmentation: The Van der Pol
oscillator: As a second order differential equation, the dynamics cannot be
approximated by a single 1-dimensional NODE with a constant control
$\bm{\theta}$ (degenerate solution in red). However, if the dynamics are
decomposed into a planar system with a dynamic control variable
$\bm{\theta}(t)$, then the parameter $\bm{\mu}$ can be adapted to fit a
particular oscillatory regime (in black). This is a particular form of
augmentation discussed in (Dupont et al. (2019); Norcliffe et al. (2020)) that
showcases the benefit of using additional variables as evolving parameters of
the dynamical system.

Closed-loop control: Defining a differentiable mapping, $g$, which outputs the
time-dependent control weights $\bm{\theta}(t)$ given the state of the
variable $\bm{x}(t)$ yields a non-autonomous system on $\mathcal{X}$ (see Fig.
3). This can be seen as a specific dimension augmentation technique where
additional variables correspond to the parameters $\bm{\theta}$. However,
contrary to appending extra dimensions which does not change the autonomous
property of the system, this augmentation results in a module describing a
time-varying transformation $\Phi_{\bm{\theta(t)}}(\bm{x_{0}},t)$. Note that
this formulation generalizes the functional parameterization proposed in
recent non-autonomous NODES systems (Choromanski et al., 2020; Massaroli et
al., 2020b; zhang2019anodev2), since the evolution of $\bm{\theta}(t)$ depends
on $\bm{x}(t)$. Much like in the case of classical control theory, we
hypothesized that the use of dynamic control weights would be of particular
use in reacting to a non-stationary stimulus. We evaluate this hypothesis in
section. 5.3.

The expressivity of N-CODE compared to other continuous-time neural networks
is encapsulated in the following proposition. The result, proven in appendix,
shows that both open-loop and closed-loop control systems overcome NODEs’
expressivity constraint with two distinct strategies, data-conditioning and
state-space augmentation.

Proposition 1 - There exists a transformation
$\phi:\mathbb{R}^{d}\to\mathbb{R}^{d}$ which can be expressed by N-CODE but
not by NODEs. In particular, $\phi$ is not a homeomorphism.

Figure 4: Trajectories over time for three types of continuous-time neural
networks learning the 1-dimensional reflection map $\varphi(\bm{x})=-\bm{x}$.
Left: (NODE) Irrespective of the form of $f$, a NODE module cannot learn such
a function as trajectories cannot intersect (Dupont et al., 2019). Middle:
(Open-loop N-CODE). The model is able to learn a family of vector fields by
controlling a single parameter of $f$ conditioned on $x(0)$. Right: (Closed-
loop N-CODE) With a fixed initialization, $x(0)$, the controller model still
learns a deformation of the vector field to learn $\varphi$. Note that the
vector field is time-varying in this case, contrary to the two others
versions.

###  3.2 Training

Loss function: Dynamics are evaluated according to a generalized loss function
that integrates a cost over some interval $[0,T]$:

$l(\bm{z}):=\int\limits_{0}^{T}\ell(\bm{z}(t),t)dt=\int\limits_{0}^{T}\ell(\bm{x}(t),\bm{\theta}(t),t)dt$
(5)

The loss in Eq. 5 is more general than in Chen et al. (2018) since $\ell$ can
be any Lebesgue-measurable function of both states, $\bm{x}$, and control
parameters, $\bm{\theta}$. In particular, this includes penalties at discrete
time points or over the entire trajectory (Massaroli et al., 2020b) but also
regularizations on the weights or activations over the entire trajectory
rather than the final state $\bm{z}(T)$ of the system.

In order to estimate a control function $\bm{\theta}(t)$ that is optimal with
respect to Eq. 5, we invoke Pontryagin’s maximum principle (PMP) (Pontryagin
Lev Semyonovich ; Boltyanskii V G & F, 1962), which only requires mild
assumptions on the functional control space $\Theta$ and applies to functions
$f$ that are non-smooth in $\bm{\theta}$. The PMP gives necessary conditions
on $\bm{\theta}(t)$ at optimality via the augmented adjoint variable
$\bm{a}(t)$. This quantity is the Jacobian of $\ell$ with respect to both
$\bm{x}(t)$ and $\bm{\theta}(t)$. In the case of $\bm{\theta}(t)$ being
differentiable with respect to the meta-parameters $\bm{\mu}$, solving for the
augmented adjoint state $\bm{a}(t)$ as in Chen et al. (2018) allows us to
compute the gradient of the loss with respect to $\bm{\mu}$ thanks to Theorem
1.

Theorem 1 - Augmented adjoint method: Given the IVP of equation 4 and for
$\ell$ defined in equation 5, we have:

$\dfrac{\partial
l}{\partial\bm{\mu}}=\int\limits_{0}^{T}\bm{a}(t)^{T}\dfrac{\partial
h}{\partial\bm{\mu}}dt,\;\;\text{such that }\bm{a}\text{
satisfies}\;\;\begin{cases}\dfrac{d\bm{a}}{dt}=-\bm{a}^{{T}}.\dfrac{\partial
h}{\partial\bm{z}}-\dfrac{\partial\ell}{\partial\bm{z}}\\\
\bm{a}(T)=\bm{0}\end{cases}$ (6)

where $\dfrac{\partial h}{\partial\bm{z}}$ is the Jacobian of $h$ with respect
to $\bm{z}$: $\;\;\;\frac{\partial
h}{\partial\bm{z}}=\begin{pmatrix}\frac{\partial
f}{\partial\bm{x}}&\frac{\partial f}{\partial\bm{\theta}}\\\ \frac{\partial
g}{\partial\bm{x}}&\frac{\partial g}{\partial\bm{\theta}}\end{pmatrix}.$

In practice, we compute the Jacobian for this augmented dynamics with open
source automatic differentiation libraries using Pytorch (Paszke et al.,
2019), enabling seamless integration of N-CODE modules in bigger
architectures. We show an example of such modules in section B of Appendix.

##  4 Experiments

###  4.1 Reflection and Concentric Annuli

We introduce our approach with the 1- and 2-dimensional problems discussed in
(Dupont et al., 2019; Massaroli et al., 2020b), consisting of learning either
the reflection map $\varphi(\bm{x})=-\bm{x}$ or a linear classification
boundary on a data-set of concentric annuli. Earlier results have shown that
vanilla NODEs cannot learn these functions since NODEs preserve the topology
of the data space, notably its linking number, leading to unstable and complex
flows for entangled classes. However, both the open and closed loop
formulations of N-CODE easily fit these functions, as shown in Figs 4 and 5.
The adaptive parameterization allows N-CODE to learn separate vector fields
for each input, allowing for simpler paths that ease model convergence.
Informally, we may think of the classification decisions as being mediated by
a bifurcation parameterized by the data space. Transiting across the true
classification boundary switches the vector field from one pushing points to
one side of the classification boundary to the other. Moreover, the
construction of $\gamma$ and $g$ as a neural network guarantees a smooth
variation of the flow with respect to the system state, which can potentially
provide interesting regularization properties on the family learned by the
model.

Figure 5: The concentric annuli problem. A classifier must separate two
classes in $\mathbb{R}^{2}$, a central disk and an annulus that encircles it.
Left: Soft decision boundaries (blue to white) for NODE (First), N-CODE open-
loop (Second) and closed-loop (First) models. Right: Training curve for the
three models.

###  4.2 Supervised image classification

We now explore different data-dependent controls for continuous-time models on
an image classification task. We tested the open-loop version of N-CODE
against augmented NODEs (Dupont et al., 2019) and the data-control model of
(Massaroli et al., 2020b). We perform this experiment on the MNIST and
CIFAR-10 image datasets. Our model consists of the dynamical system with
equation of motion $f$ expressed as a single convolutional block $(1\times
1\mapsto 3\times 3\mapsto 1\times 1)$ with 50 channels. The control weights
$\bm{\theta}\in\mathbb{R}^{K}$ are a real-valued vector of $K$ concatenated
parameters instantiating the convolution kernels.

Model | MNIST | CIFAR-10
---|---|---
ANODE | 98.2% | 61.2%
NODE DC | 98.5% | 62.0%
N-CODE (open) | 99.2% | 74.1%

Table 1: Classification accuracy on MNIST and CIFAR10 test sets for NODE, NODE
with data-dependant flow (NODE DC) and open-loop N-CODE (ours)

The 10-class prediction vector is obtained as a linear transformation of the
final state $\bm{x}(T)$ of the dynamics with a softmax activation. We aimed at
simplicity by defining the mapping $\gamma$ as a two-layer perceptron with 10
hidden units. We varied how $\gamma$ affects $\bm{\theta}$ by defining
different linear heads for each convolution in the block. (see C.2 of
Appendix). We did not perform any fine-tuning, although we did augment the
state dimension, akin to ANODE, by increasing the channel dimension by 10,
which helped stabilize training. We show that the open-loop control
outperforms competing models. Results are shown in Table. 1 with further
details in Section C.2 of Appendix.

###  4.3 Real-time pattern memorization

To validate our intuition that closed-loop control is particularly valuable on
problems involving flexible adaptation to non-stationary stimuli and to
distinguish our system from related data-conditioned methods, we trained
N-CODE on a dynamic, few-shot memorization task (Miconi et al., 2018). Here,
the model is cued to quickly memorize sets of sequentially presented
$n$-dimensional binary patterns and to reconstruct one of these patterns when
exposed to a degraded version. For each presentation episode, we exposed the
model sequentially to $m=2,5,$ or $10$ randomly generated $n$-length
($n=100,1000$) bit patterns with $\\{-1,1\\}$-valued bits. Then, we presented
a degraded version of one of the previous patterns in which half of the bits
were zeroed, and we tasked the system with reconstructing this degraded
pattern via the dynamic state, $\bm{x}(t)$. This is a difficult task since it
requires the memorization of several high-dimensional data in real-time.

Figure 6: Left: Diagram of an episode of the few-shot memorization task Miconi
et al. (2018). Three, 5-length ${-1,1}$-valued bit patterns ($n=5$ here for
visualization) are presented to the system at regular intervals. At query
time, $t_{q}$, a degraded version of one of the patterns is presented and the
task of the model is to complete this pattern. Right: Performance of a closed-
loop N-CODE, an LSTM, a continuous-time ODE-RNN Rubanova et al. (2019) and the
data-conditioned NODE of Massaroli et al. (2020b) averaged over 3 runs. All
models besides N-CODE plateau at high values (chance is .25). N-CODE, on the
other hand, learns immediately and with extremely high accuracy across all
$n$. (see Appendix)

Miconi et al. (2018) have shown that, although recurrent networks are
theoretically able to solve this task, they struggle during learning, while
models storing transient information in dynamic synapses tend to perform
better. Following this result, we evaluated the performance of a closed-loop
N-CODE model in which the output of the control function takes the form of
dynamic synapses (Eq. 7). Here, the control parameters are a matrix of dynamic
weights, $\bm{\theta}(t)\in\mathbb{R}^{m\times m}$, which are governed by the
outer product $\mu\odot\bm{x}\bm{x}^{T}$,

$\begin{cases}\left(\dfrac{d\bm{x}}{dt},\dfrac{d\bm{\theta}}{dt}\right)=\left(\rho\left(\bm{\theta}\bm{x}\right),\mu\odot\bm{x}\bm{x}^{T}\right)\\\
(\bm{x}(0),\bm{\theta}(0))=(\bm{x}_{\text{stim}},\bm{\theta}_{0})\end{cases}$
(7)

where $\bm{\mu}$ is matrix of learned parameters, $\bm{\theta}_{0}$ is a
learned initial condition, $\rho$ is an element-wise sigmoidal non-linearity,
$\odot$ is element-wise multiplication, and $x_{\text{stim}}$ is the presented
stimulus. We have suppressed $t$ for concision. We ran this experiment on the
closed-loop version of N-CODE as well as an LSTM, continuous-time ODE-RNN
(Rubanova et al., 2019) and the data-controlled model of Massaroli et al.
(2020b). All systems had the same number of learnable parameters, except for
the LSTM which was much larger. We found that all considered models, except
closed-loop N-CODE, struggled to learn this task (Fig. 6). Note that chance is
.25 since half the bits are degraded. A discrete LSTM endowed with a much
larger hidden state of 5000 neurons learns marginally better than chance
level. ODE-RNN and data control versions plateau at 17.5% error rate. For both
$n$ tested, N-CODE learned the problem not only much better, but strikingly
faster.

###  4.4 Unsupervised learning : Image autoencoding with controlled flow

Finally, we use our control approach in an autoencoding model for low-
dimensional representation learning in order to see if merely adding N-CODE to
a simpler model can improve performance. Our idea is to use the solution to a
system of linear ODEs to encode the model’s latent representations. First,
images from $\mathcal{U}\in\mathbb{R}^{n}$ are mapped by an encoder to a space
of couplings, $\bm{\theta}$, by

$\displaystyle\gamma_{\bm{\mu}}:\bm{u}\to\gamma_{\bm{\mu}}(\bm{u})=\bm{\theta}\in\mathbb{R}^{m\times
m}.$ (8)

The latent code is taken as the vector $\bm{x}(T)$ acquired by solving the
autonomous differential system presented in Eq. 9 from 0 to $T$ starting from
the initial condition $(\bm{x}_{0},\gamma_{\bm{\mu}}(u))$ with
$\bm{x}_{0}\sim\mathcal{N}(0,I)$:

$\begin{cases}\left(\dfrac{d\bm{x}}{dt},\dfrac{d\bm{\theta}}{dt}\right)=(\bm{\theta}\bm{x},0)\\\
(\bm{x}(0),\bm{\theta}(0))=(\bm{x}_{0},\gamma_{\bm{\mu}}(\bm{u}))\end{cases}$
(9)

This model, which we call autoN-CODE, can be interpreted as a hybrid version
of autoencoders and generative normalizing flows, where the latent generative
flow is data-dependent and parameterized by the encoder output.

We found that a simple linear system was already more expressive than a
discrete linear layer in shaping the latent representation. Since the output
dimension of the encoder grows quadratically with the desired latent code
dimension of the bottleneck, we adopted a sparse prediction strategy, where as
few as two elements of each row of $\bm{\theta}$ were non-zero, making our
model match exactly the number of parameters of a VAE with same architecture.
We believe that the increase in representational power over a vanilla
autoencoder, as shown by improvements in Fréchet Inception Distance (FID;
Heusel et al. (2017)) and more disentangled latent representations (see A.7),
despite these simplifying assumptions, shows the potential of this approach
for representation learning (Fig. 2). Training details can be found in Sec.
C.4.

Here we measure the marginal improvement acquired by adding latent flows to a
baseline VAE. In order to endow our deterministic model with the ability to
generate image samples, we perform, similarly to (Ghosh et al., 2020), a post
hoc density estimation of the distribution of the latent trajectories’ final
state. Namely, we employ a gaussian mixture model fit by expectation-
maximization and explore the effect of increasing the number of components in
the mixture. We report in Table 2, contrary to Ghosh et al. (2020), a
substantial decrease in FID of our sampled images to the image test set with
an increasing number of components in the mixture (see Fig. 7 for latent
evolution and Fig. 8 for sampled examples).

###  4.5 Image generation

FID ($\downarrow$) | CIFAR-10 | CelebA-(64)
---|---|---
VAE†(Kingma & Welling, 2013) | 105.45 | 68.07
Auto-encoder† (100 comp.) | 73.24 | 63.11
Auto-encoder† (1000 comp.) | 55.55 | 55.60
AutoNCODE (100 comp.) | 34.90 | 55.27
AutoNCODE (1000 comp.) | 24.19 | 52.47

Table 2: Fréchet Inception Distance (FID) for several recent architectures
(lower is better). ”Components” are the numbers of components used in the
gaussian mixture estimation of the latent distribution.

Considering the identical architecture of the encoder and decoder to a vanilla
model, these results suggest that the optimal encoding control formulation
produced a structural change in the latent manifold organization. (See section
C.7 in Appendix). Finally, we note that this model offers several original
possibilities to explore, such as sampling from the control coefficients, or
adding a regularization on the dynamic evolution as in (Finlay et al., 2020),
which we leave for future work.

Figure 7: Left to right: Temporal evolutions of an image reconstruction along
controlled orbits of AutoN-CODE for representative CIFAR-10 and CelabA test
images. Last column: ground truth image.

##  5 Related work

Neural ODEs -  This work builds on the recent development of Neural ODEs,
which have shown great promise for building generative normalizing flows
(Grathwohl et al., 2019) used for modeling molecular interactions (Köhler et
al., 2019) or in mean field games (Ruthotto et al., 2019). More recent work
has focused on the topological properties of these models (Dupont et al.,
2019; Zhang et al., 2020), introducing time-dependent parameterization (Zhang
et al., 2019b; Massaroli et al., 2020b; Choromanski et al., 2020), developing
novel regularization methods using optimal transport or stochastic
perturbations (Finlay et al., 2020; Oganesyan et al., 2020) and adapting them
to stochastic differential equations (Tzen & Raginsky, 2019).

Optimal Control -  Several authors have interpreted deep learning optimization
as an optimal control problem (Li et al., 2017; Benning et al., 2019; Liu &
Theodorou, 2019; Zhang et al., 2019a; Seidman et al., 2020) providing strong
error control and yielding convergence results for alternative algorithms to
standard gradient-based learning methods. Of particular interest is the mean-
field optimal control formulation of (E et al., 2018), which notes the
dependence of a unique control variable governing the network dynamics on the
whole data population.

Hypernetworks -  Finally, our system is related to network architectures with
adaptive weights such as hypernetworks (Ha et al., 2017), dynamic filters (Jia
et al., 2016) and spatial transformers (Jaderberg et al., 2015). Though not
explicitly formulated as neural optimal control, these approaches effectively
implement a form of modulation of neural network activity as a function of
input data and activation state, resulting in compact and expressive models.
These methods demonstrate, in our opinion, the significant untapped potential
value in developing dynamically controlled modules for deep learning.

Figure 8: Left Reconstructions of random test images from CelebA for different
autoencoding models. Right: Random samples from the latent space.
Deterministic models are fit with 100-components gaussian mixture model.

##  6 Conclusion

In this work, we have presented an original control formulation for
continuous-time neural feature transformations. We have shown that it is
possible to dynamically shape the trajectories of the transformation module
applied to the data by augmenting the network with a trained control
mechanism, and further demonstrated that this can be applied in the context of
supervised and unsupervised representation learning. In future work, we would
like to investigate the robustness and generalization properties of such
controlled models as well as their similarities with fast-synaptic modulation
systems observed in neuroscience, and test this on natural applications such
as recurrent neural networks and robotics. An additional avenue for further
research is the connection between our system and the theory of bifurcations
in dynamical systems and neuroscience.

## Acknowledgments and Disclosure of Funding

This work was funded by the ANR-3IA Artificial and Natural Intelligence
Toulouse Institute (ANR-19-PI3A-0004).

Additional support provided by ONR (N00014-19-1-2029), NSF (IIS-1912280, OSCI-
DEEP ANR grant (ANR-19-NEUC-0004) and the Center for Computation and
Visualization (CCV). We acknowledge the Cloud TPU hardware resources that
Google made available via the TensorFlow Research Cloud (TFRC) program as well
as computing hardware supported by NIH Office of the Director (S10OD025181).

## References

  * Benning et al. (2019) Martin Benning, Elena Celledoni, Matthias J. Ehrhardt, Brynjulf Owren, and Carola-Bibiane Schönlieb.  Deep learning as optimal control problems: models and numerical methods, 2019.
  * Caron et al. (2018) Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze.  Deep clustering for unsupervised learning of visual features.  In _European Conference on Computer Vision_ , 2018.
  * Chang et al. (2017) J. Chang, L. Wang, G. Meng, S. Xiang, and C. Pan.  Deep adaptive image clustering.  In _2017 IEEE International Conference on Computer Vision (ICCV)_ , 2017.
  * Chen et al. (2018) Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud.  Neural ordinary differential equations.  In _Advances in Neural Information Processing Systems_ , 2018.
  * Choromanski et al. (2020) Krzysztof Choromanski, Jared Quincy Davis, Valerii Likhosherstov, Xingyou Song, Jean-Jacques Slotine, Jacob Varley, Honglak Lee, Adrian Weller, and Vikas Sindhwani.  An ode to an ode, 2020.
  * Dempster et al. (1977) A. P. Dempster, N. M. Laird, and D. B. Rubin.  Maximum likelihood from incomplete data via the em algorithm.  _Journal of the Royal Statistical Society. Series B (Methodological)_ , 1977.
  * Dupont et al. (2019) Emilien Dupont, Arnaud Doucet, and Yee Whye Teh.  Augmented neural odes.  In _Advances in Neural Information Processing Systems_ , 2019.
  * E et al. (2018) Weinan E, Jiequn Han, and Qianxiao Li.  A mean-field optimal control formulation of deep learning, 2018.
  * Finlay et al. (2020) Chris Finlay, Jörn-Henrik Jacobsen, Levon Nurbekyan, and Adam M Oberman.  How to train your neural ode: the world of jacobian and kinetic regularization, 2020.
  * Ghosh et al. (2020) P. Ghosh, M. S. M. Sajjadi*, A. Vergari, M. J. Black, and B. Schölkopf.  From variational to deterministic autoencoders.  In _8th International Conference on Learning Representations (ICLR)_ , 2020.
  * Grathwohl et al. (2019) Will Grathwohl, Ricky T. Q. Chen, Jesse Bettencourt, and David Duvenaud.  Scalable reversible generative models with free-form continuous dynamics.  In _International Conference on Learning Representations_ , 2019.
  * Ha et al. (2017) David Ha, Andrew Dai, and Quoc V. Le.  Hypernetworks.  2017\.
  * He et al. (2015) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  2015\.
  * Heusel et al. (2017) Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium, 2017.
  * Hochreiter & Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber.  Long short-term memory.  _Neural Computation_ , 1997.
  * Jaderberg et al. (2015) Max Jaderberg, Karen Simonyan, Andrew Zisserman, and koray kavukcuoglu.  Spatial transformer networks.  In _Advances in Neural Information Processing Systems_ , 2015.
  * Ji et al. (2019) Xu Ji, João F Henriques, and Andrea Vedaldi.  Invariant information clustering for unsupervised image classification and segmentation.  In _Proceedings of the IEEE International Conference on Computer Vision_ , 2019.
  * Jia et al. (2016) Xu Jia, Bert De Brabandere, Tinne Tuytelaars, and Luc V Gool.  Dynamic filter networks.  In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.), _Advances in Neural Information Processing Systems_ , 2016.
  * Kingma & Ba (2014) Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization, 2014.
  * Kingma & Welling (2013) Diederik P Kingma and Max Welling.  Auto-encoding variational bayes, 2013.
  * Köhler et al. (2019) Jonas Köhler, Leon Klein, and Frank Noé.  Equivariant flows: sampling configurations for multi-body systems with symmetric energies, 2019.
  * Li et al. (2017) Qianxiao Li, Long Chen, Cheng Tai, and Weinan E.  Maximum principle based algorithms for deep learning, 2017.
  * Liu & Theodorou (2019) Guan-Horng Liu and Evangelos A. Theodorou.  Deep learning theory review: An optimal control and dynamical systems perspective, 2019.
  * Liu et al. (2015) Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang.  Deep learning face attributes in the wild.  In _Proceedings of International Conference on Computer Vision (ICCV)_ , December 2015.
  * Lloyd (1982) Stuart P. Lloyd.  Least squares quantization in pcm.  _IEEE Transactions on Information Theory_ , 1982.
  * Maaten & Hinton (2008) Laurens van der Maaten and Geoffrey Hinton.  Visualizing data using t-sne.  _Journal of machine learning research_ , 2008.
  * Massaroli et al. (2020a) Stefano Massaroli, Michael Poli, Michelangelo Bin, Jinkyoo Park, Atsushi Yamashita, and Hajime Asama.  Stable neural flows, 2020a.
  * Massaroli et al. (2020b) Stefano Massaroli, Michael Poli, Jinkyoo Park, Atsushi Yamashita, and Hajime Asama.  Dissecting neural odes, 2020b.
  * Miconi et al. (2018) Thomas Miconi, Kenneth Stanley, and Jeff Clune.  Differentiable plasticity: training plastic neural networks with backpropagation.  In _Proceedings of the 35th International Conference on Machine Learning_ , 2018.
  * Ng (2011) Andrew Ng.  Sparse autoencoders, 2011.
  * Norcliffe et al. (2020) Alexander Norcliffe, Cristian Bodnar, Ben Day, Nikola Simidjievski, and Pietro Liò.  On second order behaviour in augmented neural odes, 2020.
  * Oganesyan et al. (2020) Viktor Oganesyan, Alexandra Volokhova, and Dmitry Vetrov.  Stochasticity in neural odes: An empirical study, 2020.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  Pytorch: An imperative style, high-performance deep learning library, 2019\.
  * Pontryagin Lev Semyonovich ; Boltyanskii V G & F (1962) Gamkrelidze R V Pontryagin Lev Semyonovich ; Boltyanskii V G and Mishchenko E F.  _The mathematical theory of optimal processes_.  1962\.
  * Radford et al. (2015) Alec Radford, Luke Metz, and Soumith Chintala.  Unsupervised representation learning with deep convolutional generative adversarial networks, 2015.
  * Rubanova et al. (2019) Yulia Rubanova, Ricky T. Q. Chen, and David Duvenaud.  Latent odes for irregularly-sampled time series, 2019.
  * Ruthotto et al. (2019) Lars Ruthotto, Stanley Osher, Wuchen Li, Levon Nurbekyan, and Samy Wu Fung.  A machine learning framework for solving high-dimensional mean field game and mean field control problems, 2019.
  * Ruthotto et al. (2020) Lars Ruthotto, Stanley J. Osher, Wuchen Li, Levon Nurbekyan, and Samy Wu Fung.  A machine learning framework for solving high-dimensional mean field game and mean field control problems.  _Proceedings of the National Academy of Sciences_ , 2020.
  * Seidman et al. (2020) Jacob H. Seidman, Mahyar Fazlyab, Victor M. Preciado, and George J. Pappas.  Robust deep learning as optimal control: Insights and convergence guarantees.  Proceedings of Machine Learning Research, 2020.
  * Shi & Malik (2000) Jianbo Shi and Jitendra Malik.  Normalized cuts and image segmentation.  Technical report, 2000.
  * Tolstikhin et al. (2017) Ilya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schoelkopf.  Wasserstein auto-encoders, 2017.
  * Tzen & Raginsky (2019) Belinda Tzen and Maxim Raginsky.  Neural stochastic differential equations: Deep latent gaussian models in the diffusion limit, 2019.
  * Vincent et al. (2010) Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, and Pierre-Antoine Manzagol.  Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion.  _Journal in Machine Learning Res._ , 2010.
  * Zhang et al. (2019a) Dinghuai Zhang, Tianyuan Zhang, Yiping Lu, Zhanxing Zhu, and Bin Dong.  You only propagate once: Accelerating adversarial training via maximal principle.  In _Advances in Neural Information Processing Systems 32_. 2019a.
  * Zhang et al. (2020) Han Zhang, Xi Gao, Jacob Unterman, and Tom Arodz.  Approximation capabilities of neural odes and invertible residual networks, 2020.
  * Zhang et al. (2019b) Tianjun Zhang, Zhewei Yao, Amir Gholami, Joseph E Gonzalez, Kurt Keutzer, Michael W Mahoney, and George Biros.  Anodev2: A coupled neural ode framework.  In _Advances in Neural Information Processing Systems_. 2019b.

## Appendix

##  Appendix A Theoretical results

###  A.1 Proof of Proposition 1

In order to show that the set of functions defined by N-CODE are not
necessarily homeomorphisms on $\mathcal{X}$, we show that they are not
generally injective.

Open-loop: Consider the one-dimensional N-CODE system
$\Phi:(\mathcal{X},\mathbb{R})\mapsto\mathcal{X}$ with equation of motion
$f(\bm{x},\bm{\theta)}=-\theta$ where $\theta=\gamma(\bm{x}(0))=\bm{x}(0)$.
This system has solution

$\displaystyle\Phi(\bm{x}(0),T)$ $\displaystyle=\bm{x}(0)+\int
f(\bm{x},\bm{\theta},t)\ dt$ $\displaystyle=\bm{x}(0)-\bm{x}(0)T.$ (10)

However, this implies

$\Phi(0,1)=\Phi(1,1)=0,$ (11)

giving the result.

Closed-loop: Similarly, we can show that the following 2D oscillatory system:

$\begin{cases}\dot{\bm{x}}(t)=\bm{\theta}(t)\\\
\dot{\bm{\theta}}(t)=-\bm{x}(t)\end{cases}\implies\bm{x}(t)=\alpha\cos(t)+\beta\sin(t)\implies\bm{x}(t)=\bm{x}(0)\cos(t)\;\;\text{if}\;\;\bm{\theta}(0)=0$
(12)

But $\Phi(x,\pi/2)=0$, so $\bm{x}\mapsto\Phi(\bm{x},t)=\bm{x}(t)$ is not
injective.

This shows that the two control forms, open and closed, are not restricted to
learn homemorphisms on the space $\mathcal{X}$.

###  A.2 Proof of Theorem 1

###### Proof.

The proof is inspired from Massaroli et al. (2020a). We place our analysis in
the Banach space $\mathbb{R}^{n}$. Since $\ell$ and $h$ are continuously
differentiable, let us form the Lagrangian functional $\mathcal{L}$ where
$\bm{a}$ is an element of the dual space of $\bm{z}=\begin{bmatrix}\bm{x}\\\
\bm{\theta}\end{bmatrix}$:

$\mathcal{L}(\bm{z},\bm{a},t):=\int\limits_{0}^{T}[\ell-\langle\bm{a},\dfrac{\partial\bm{z}}{\partial
t}-h\rangle]dt$ (13)

The constraint $\langle\bm{a}(t),\dfrac{\partial z}{\partial
t}-h(\bm{z},t)\rangle$ is always active on the admissible set by construction
of $\bm{z}$ in equation 4, such that we have
$\forall\bm{z},\bm{a},\;\;\dfrac{\partial\mathcal{L}}{\partial\bm{\mu}}(\bm{z},\bm{a},t)=\dfrac{\partial
l}{\partial\bm{\mu}}(\bm{z},\bm{a},t)$. Moreover, integrating the left part of
the integral in equation 13 gives:

$\int\limits_{0}^{T}\langle\bm{a},\dfrac{\partial\bm{z}}{\partial t}\rangle
dt=\langle\bm{a},\bm{z}\rangle\big{|}^{T}_{0}-\int\limits_{0}^{T}\langle\dfrac{\partial\bm{a}}{\partial
t},\bm{z}\rangle dt$ (14)

Now, given that $\ell-\langle\dfrac{d\bm{a}}{dt},\bm{z}\rangle$ is
differentiable in $\mu$ for any $t\in[0,T]$, Leibniz integral rule allows to
write

$\dfrac{\partial
l}{\partial\bm{\mu}}=\int\limits_{0}^{T}\dfrac{\partial}{\partial\bm{\mu}}[\ell+\langle\dfrac{d\bm{a}}{dt},\bm{z}\rangle+\langle\bm{a},h\rangle]dt+\langle\bm{a}(T),\dfrac{\partial\bm{z}(T)}{\partial\bm{\mu}}\rangle-\langle\bm{a}(0),\cancel{\dfrac{\partial\bm{z}(0)}{\partial\bm{\mu}}\rangle}$
(15) $\dfrac{\partial
l}{\partial\bm{\mu}}=\int\limits_{0}^{T}[\dfrac{\partial\ell}{\partial\bm{z}}\dfrac{\partial\bm{z}}{\partial\bm{\mu}}+\langle\dfrac{d\bm{a}}{dt},\dfrac{\partial\bm{z}}{\partial\bm{\mu}}\rangle+\langle\bm{a},\dfrac{\partial
h}{\partial\bm{\mu}}\rangle]dt+\langle\bm{a}(T),\dfrac{\partial\bm{z}(T)}{\partial\bm{\mu}}\rangle$
(16) $\dfrac{\partial
l}{\partial\bm{\mu}}=\int\limits_{0}^{T}[\dfrac{\partial\ell}{\partial\bm{z}}\dfrac{\partial\bm{z}}{\partial\bm{\mu}}+\langle\dfrac{d\bm{a}}{dt},\dfrac{\partial\bm{z}}{\partial\bm{\mu}}\rangle+\langle\bm{a},\dfrac{\partial
h}{\partial\bm{\mu}}+\cancel{\dfrac{\partial h}{\partial t}\dfrac{\partial
t}{\partial\bm{\mu}}}+\dfrac{\partial
h}{\partial\bm{z}}\dfrac{\partial\bm{z}}{\partial\bm{\mu}}]dt+\langle\bm{a}(T),\dfrac{\partial\bm{z}(T)}{\partial\bm{\mu}}\rangle$
(17)

The last equation can be reordered as:

$\dfrac{\partial
l}{\partial\bm{\mu}}=\int\limits_{0}^{T}\langle\dfrac{d\bm{a}}{dt}-\bm{a}\dfrac{\partial
h}{\partial\bm{z}}-\dfrac{\partial\ell}{\partial\bm{z}},\dfrac{\partial\bm{z}}{\partial\bm{\mu}}\rangle
dt+\int\limits_{0}^{T}\langle\bm{a},\dfrac{\partial
h}{\partial\bm{\mu}}\rangle
dt+\langle\bm{a}(T),\dfrac{\partial\bm{z}(T)}{\partial\bm{\mu}}\rangle$ (18)

Posing that $\bm{a}(T)=\mathbb{O}_{|\mathcal{X}|+|\Theta|}$ and
$\langle\dfrac{d\bm{a}}{dt}-\bm{a}\dfrac{\partial
h}{\partial\bm{z}}-\dfrac{\partial\ell}{\partial\bm{z}},\dfrac{\partial\bm{z}}{\partial\bm{\mu}}\rangle=0$,
the result follows. ∎

Remark 1: This result allows to compute gradient estimate assuming that
$\bm{\mu}$ directly parametrizes $h$ such that the gradient $\dfrac{\partial
h}{\partial\bm{\mu}}$ are straightforward to compute. However, in the case of
optimizing the initial mapping $\gamma$ that infer the variable $\bm{\theta}$
, this result can be combined with usual chain rule to estimate the gradient:

$\dfrac{\partial
l}{\partial\bm{\mu}_{\gamma}}=\dfrac{\partial\theta}{\partial\bm{\gamma}}\dfrac{\partial
l}{\partial\bm{\theta}}$ (19)

##  Appendix B Implementation

Generic N-CODE module - We provide here a commented generic PyTorch
implementation for the N-CODE module in the open-loop setting.

⬇

\parclass NCODE_func(torch.nn.Module):

def __init__(self, *args):

super(NCODE_func, self).__init__()

\par#Definition of the ODE system

self.f = …

self.g = …

\pardef forward(self, t, z):

\par#Unpack variables (theta_t is potentially a tuple of tensors)

x_t, *theta_t = z

\par#Compute evolution od the system

d_x_t = self.f(x_t,theta_t)

d_theta_t = self.g(theta_t,d_x_t = f(x_t,theta_t)

\parreturn (d_x_t, *d_theta_t)

\parclass NCODE_block(torch.nn.Module):

def __init__(self, *args):

super(NCODE_block, self).__init__()

\par#Definition of module and controller

self.gamma = …

self.NCODE_func = …

\pardef forward(self, x_0):

\par#Set initial conditions

self.theta_0 = self.gamma(x_0)

z_0 = (x_0,*self.theta_0)

\par#Run the system forward with defined settings

x_t, *theta_t = odeint_adjoint(self.odefunc, z_0, integration_time, rtol,
atol, method,options, adjoint_params=self.NCODE_func.parameters())

\parreturn (x_t, *theta_t)

##  Appendix C Experimental results

Resources - Our experiments were run on a 12GB NVIDIA® Titan Xp GPUs cluster
equipped with CUDA 10.1 driver. Neural ODEs were trained using the torchdiffeq
(Chen et al., 2018) PyTorch package.

###  C.1 Flows for annuli dataset

Figure A.1: Time-varying flows for two points in different classes (inner: top
row; outer: bottom row) of the annuli dataset. Time advances from left to
right.

###  C.2 Supervised image classification

For MNIST and CIFAR-10 classification, we defined the equation of motion $f$
as sequence of convolutional filters replicating a ResNet block architecture
with padding to conserve dimensionality and non-linear ReLU activation. The
base architecture takes one of the forms

  * •

$1\times 1$, $\bm{k}$ filters, 1 stride, 0 padding

  * •

$3\times 3$, $\bm{k}$ filters, 1 stride, 1 padding

  * •

$1\times 1$, $\bm{c}+\bm{a}$ filters, 1 stride, 0 padding

with $\bm{k}$ the number of filters, $\bm{c}$ the number of channels (1 for
MNIST and 3 for CIFAR10) and $\bm{a}$ the augmentation channel (Dupont et al.,
2019). The open-loop controller is a very simple linear transformation of the
data into 10 hidden units followed by multiple heads outputting the vector of
weights for each convolution kernel. In order to examine the effect of the
size of dimensionality of the equation of motion on learnability, we tried
different parameterizations for $f$:

  * •

NCODE : The three kernels are conditioned by the controller $\gamma$.

  * •

NCODE $3\times 3$: The inner 3x3 kernels are dynamically conditioned.

  * •

NODE DC : This corresponds to the form of data-control proposed by (Massaroli
et al., 2020b) where the image data is concatenated with state $\bm{x}(t)$ at
each evaluation of the first 1x1 convolution kernel.

All parameters not adaptively controlled are simply learned as fixed
parameters. For all models we used the adaptative solver Dormund-Prince with
tolerance of 1e-3.

Figure A.2: Accuracy, Train and test losses over training interations and
Number of function evaluations (NFE) vs loss for several versions of NCODE on
CIFAR-10.

The results show that N-CODE models achieve higher accuracy in fewer epochs
and overall lower loss (Fig. A.2, top panels). However, the simplicity of the
mapping $\gamma$ in this experiment and the absence of regularization of the
induced map $\Phi_{\gamma}$ (Finlay et al., 2020) yields a complex flow
requiring a higher total number of function evaluations (Fig. A.2, lower
panels). This potentially prevents NCODE from achieving even better accuracy.
Interestingly, the results for the partially controlled systems suggest that
gradient descent accommodates the learning of hybrid modules $f$ with both
static and dynamic convolution parameters. These results advocate for the
exploration of specific closed-loop formulations and regularization of control
for convolution layers which we leave for future work.

###  C.3 Pattern Memorization

This experiment is an update of Miconi et al. (2018), which showed that using
a simple Hebbian update rule between weights endowed recurrent networks with
fast memory properties. The base model is a single-cell recurrent network with
all-to-all coupling, $\bm{\theta}(t)$. The considered models are

Open-loop control: We adopt the implementation of (Massaroli et al., 2020b) by
appending the presented pattern $\bm{x}_{stim}$ to the dynamic variable
$\bm{x}(t)$ such that $\bm{\theta}(t)=\bm{\theta}$ is linear layer
$L(\mathbb{R}^{2\times N},\mathbb{R}^{N})$.

Closed-loop control: The weights are fully dynamic and the influence of the
plastic evolution is tuned by learning the components $\mu$ that apply an
element-wise gain for every weight velocity, according to Eq. 7.

ODE-RNN: Here, a static-weight NODE models the evolution of a continuous-time
single-cell recurrent network whose hidden state is carried over from one
presentation to the next. A supplementary linear read-out
$L(\mathbb{R}^{N},\mathbb{R}^{N})$ outputs the model’s guess and balances the
number of parameters with other models.

LSTM: A vanilla LSTM cell (Hochreiter & Schmidhuber, 1997). In coherence with
(Miconi et al., 2018) results, the hidden state dimension need to be greatly
increased for the module to start memorizing. We tested a 5000-dimensional
hidden state for the reported results.

For each model, we learn the model weights with gradient descent using an Adam
optimizer with a learning rate $\lambda=3e-4$ and use the $L_{2}$ distance
between the reconstruction and objective pattern as our objective function.
For continuous models, each episode consists of a sequential presentation of
each pattern for 0.5 sec followed by a query time of 0.5 sec. For the LSTM, we
adopted the setting of (Miconi et al., 2018) where the sequence of
presentation is discretized into 5 time-steps of presentations. We tested for
episodes of 3,5 and 10 patterns with 2 presentations in random order and
different proportions of degradation (0.5,0.7 and 0.9). Static-weight model
performance rapidly deteriorated in more challenging settings, whereas N-CODE
reconstruction converged to a residual error below 1% in all cases.

Figure A.3: Error rate of N-CODE on the 1000-bit reconstruction task as a
function of episodes for (Left) different proportions of degraded bits and
(Right) different number of patterns presented in one episode. In all cases,
the model learns in several hundreds episodes.

###  C.4 AutoN-CODE architectures

| MNIST | CIFAR-10 | CelebA
---|---|---|---
| $\textbf{x}\in\mathbb{R}^{3\times 28\times 28}$ | $\textbf{x}\in\mathbb{R}^{3\times 32\times 32}$ | $\textbf{x}\in\mathbb{R}^{3\times 64\times 64}$
Encoder: | Flatten | Conv256 $\mapsto$ BN $\mapsto$ Relu | Conv256 $\mapsto$ BN $\mapsto$ Relu
| FC400 $\mapsto$ Relu | Conv512 $\mapsto$ BN $\mapsto$ Relu | Conv512 $\mapsto$ BN $\mapsto$ Relu
| FC25 $\mapsto$ Norm | Conv1024 $\mapsto$ BN $\mapsto$ Relu | Conv1024 $\mapsto$ BN $\mapsto$ Relu
|  | FC128∗2 $\mapsto$ Norm | FC64∗10 $\mapsto$ Norm
Latent dynamics: | ODE[0,10] | ODE[0,10] | ODE[0,10]
Decoder: | FC400 $\mapsto$ Relu | FC1024∗8∗8 $\mapsto$ Norm | FC1024∗8∗8 $\mapsto$ Norm
| FC784 $\mapsto$ Sigmoid | ConvT512 $\mapsto$ BN $\mapsto$ Relu | ConvT512 $\mapsto$ BN $\mapsto$ Relu
|  | ConvT256 $\mapsto$ BN $\mapsto$ Relu | ConvT256 $\mapsto$ BN $\mapsto$ Relu
|  | ConvT3 | ConvT3
Table A.1: Model architectures for the different data sets tested. FCn and
Convn represent, respectively, fully connected and convolutional layers with
$n$ output/filters. We apply a component-wise normalisation of the control
components which proved crucial for good performance of the model. The dynamic
is run on the time segment [0,10] which empirically yields good results.

Adapting the models used by Tolstikhin et al. (2017) and Ghosh et al. (2020),
we use a latent space dimension of 25 for MNIST, 128 for CIFAR-10 and 64 for
CelebA. All convolutions and transposed convolutions have a filter size of
4$\times$4 for MNIST and CIFAR-10 and 5$\times$5 for CELEBA. We apply batch
normalization to all layers. They all have a stride of size 2 except for the
last convolutional layer in the decoder. We use Relu non-linear activation and
batch normalisation at the end of every convolution filter. Official train and
test splits are used for the three datasets. For training, we use a mini-batch
size of 64 in MNIST and CIFAR and 16 for CelebA in AutoN-CODE. (64 for control
models.) All models are trained for a maximum of 50 epochs on MNIST and CIFAR
and 40 epochs on CelebA. We make no additional change to the decoder. We train
the parameters of the encoder and decoder module for minimizing the mean-
squared error (MSE) on CIFAR-10 and CelebA (Liu et al., 2015) or alternatively
the Kullback-Leibler divergence between the data distribution and the output
of the decoder for MNIST (formulas in Appendix.) Gradient descent is performed
for 50 epochs with the Adam optimizer (Kingma & Ba, 2014) with learning rate
$\lambda=1e^{-3}$ reduced by half every time the loss plateaus. All
experiments are run on a single GPU GeForce Titan X with 12 GB of memory.

###  C.5 Visualization of latent code dynamical evolution

Figure A.4: Reconstructions of the image along the controlled orbits of AutoN-
CODE for MNIST. The temporal dimension reads from left to right. Last column:
ground truth image Figure A.5: Reconstructions of the image along the
controlled orbits of AutoN-CODE for CIFAR-10. The temporal dimension reads
from left to right. Last column: ground truth image. Figure A.6:
Reconstructions of the image along the controlled orbits of AutoN-CODE for
CelebA. The temporal dimension reads from left to right. Last column: ground
truth image

###  C.6 Exploring sampling distributions

For random sampling, we train the VAE with a $\mathcal{N}(0,I)$ prior. For the
deterministic models, samples are drawn from a mixture of multivariate
gaussian distributions fit using the testing set embeddings. The distribution
is obtained through expectation-maximization Dempster et al. (1977) with one
single k-means initialization, tolerance $1e^{-3}$ and run for at most 100
iterations. We compute the FID using 10k generated samples evaluated against
the whole test set for all FID evaluations, using the standard
2048-dimensional final feature vector of an Inception V3 following Heusel et
al. (2017) implementation.

Figure A.7: Left: Evolution of FID as a function of the number of components
in the Gaussian mixture used to fit the latent distribution. We also tested a
vanilla autoencoder with twice the capacity (236-dimensional), exactly
matching the number of parameters of our model. Interestingly, this model
performs worst in a low component regime. We interpret this a as a
manifestation of the “curse of dimensionality”, as the latent example
population becomes less dense in this augmented space, making the naive
gaussian mixture fitting less accurate for fitting the latent distribution.
Right: Samples of generated images for different number of mixture components.
The shape and details becomes clearer with increasing components.

The results show that 10K components (matching the number of data points in
the test sets) naturally overfits on the data distribution. Generated images
display marginal changes compared to test images. However, 1000 components
does not, showing that our AutoN-CODE sampling strategy mediates a trade-off
between sample quality and generalization of images. We alternatively tested
non-parametric kernel density estimation with varying kernel variance to
replace our initial sampling strategy. We report similar results to the
gaussian mixture experiment with an overall lower FID of AutoN-CODE for small
variance kernels. As the fitting distribution becomes very rough
($\sigma\approx 5$), the generated image quality is highly deteriorated.(see
Figure A6).

Figure A.8: Nearest neighbors search of random samples from the gaussian
mixture with (Left) 1000 components (Left) and (Right) 10K components in the
testing set of CIFAR-10. The sampled latents show overfiting on the test set
in the highest case. Figure A.9: (Left) Evolution of FID between test set and
generated samples as a function of the gaussian kernel variance used to
perform kernel estimation of the the latent code distribution. (Right) Nearest
neighbor search of random samples from the latent distribution fit by gaussian
kernel estimation with variance $\sigma=2$ in the testing set.

###  C.7 AutoN-CODE Latent Clustering

We further investigate the representations learned by our model by measuring
its clustering accuracy against several other techniques in Fig. A.10.
Similarly to the image generation experiment, we consider the final state of
the dynamical system as the latent representation that we cluster using a 10
component gaussian mixture with different initializations. We did not perform
any supplementary fine-tuning or data augmentation to train our model, but we
also tested a version with further dimensionality reduction using t-SNE
(Maaten & Hinton, 2008). The results, although inferior to recent deep
clustering techniques, show better clustering accuracy than other autoencoding
models, suggesting a different organization of the latent code compared to the
vanilla linear projection of the autoencoder.

Clustering accuracy |
---|---
Model | CIFAR-10 | MNIST
K-means (Lloyd, 1982) | 22.9 | 57.2
Spectral clustering (Shi & Malik, 2000) | 24.7 | 69.6
Variational Bayes AE† (Kingma & Welling, 2013) | 29.1 | 83.2
Sparse AE (Ng, 2011) | 29.7 | 82.7
Denoising AE (Vincent et al., 2010) | 29.7 | 83.2
AE (GMM)† | 31.4 | 81.2
GAN (2015) (Radford et al., 2015) | 31.5 | 82.8
DeepCluster (Caron et al., 2018) | 37.4 | 65.6
DAC (Chang et al., 2017) | 52.2 | 97.8
IIC (Ji et al., 2019) | 61.7 | 99.2
AutoeN-CODE (GMM)† | 33.31 | 86.02
AutoeN-CODE (t-SNE + GMM)† | 27.00 | 97.26

Figure A.10: Unsupervised image clustering accuracy on CIFAR-10 and MNIST
against recent models. Results obtained with the authors original code are
noted with ${\dagger}$. Figure A.11: tSNE embeddings of the latent code at
$t=t_{1}$ for MNIST test set colored with a 10 component gaussian mixture
model.

###  C.8 Latent code interpolation

Figure A.12: Interpolation: We further explore the latent space of AutoN-CODE
by interpolating between the latent vectors of the CelebA test set. (Upper
panel) Linear interpolation between random samples reconstructed with AutoN-
CODE. (Middle panel) Interpolation comparison between AutoN-CODE and a vanilla
autoencoder for a single pair of vectors. (Lower panel) 2d interpolation with
three anchor reconstructions corresponding to the three corners of the square
(A:up-left,B:up-right and C:down-left. Left square corresponds to AutoN-CODE
reconstructions and right to a vanilla autoencoder.