# Self-similar solutions to the Hesse flow

Shun Maeta  Department of Mathematics, Shimane University, Nishikawatsu 1060
Matsue, 690-8504, Japan<EMAIL_ADDRESS>or<EMAIL_ADDRESS>
###### Abstract.

We define a Hesse soliton, that is, a self-similar solution to the Hesse flow
on Hessian manifolds. On information geometry, the $e$-connection and the
$m$-connection are important, which do not coincide with the Levi-Civita one.
Therefore, it is interesting to consider a Hessian manifold with a flat
connection which does not coincide with the Levi-Civita one. We call it a
proper Hessian manifold. In this paper, we show that any compact proper Hesse
soliton is expanding and any non-trivial compact gradient Hesse soliton is
proper. Furthermore, we show that the dual space of a Hesse-Einstein manifold
can be understood as a Hesse soliton.

###### Key words and phrases:

Hesse flow; Hesse solitons; Hessian manifolds; Information geometry

###### 2010 Mathematics Subject Classification:

53B12, 53E99, 35C06, 62B11

The author is partially supported by the Grant-in-Aid for Young Scientists,
No.19K14534, Japan Society for the Promotion of Science.

##  1\. Introduction

Machine learning might be one of the most powerful tool for the human race.
Information geometry is regarded as a basic and important mathematical
research in this field, which was introduced by S. Amari. In particular,
Exponential families and Mixture families of probability distributions are
important in information geometry. They have the dual flat structure. Hence,
interestingly, they have Hessian structure.

Information geometry is built on the basis of differential geometry. The
geometric flow is one of the most powerful tool in the theory of differential
geometry. In particular, the Ricci flow is powerful. In fact, as is well
known, Poincare conjecture was solved by the Ricci flow. The Ricci flow is
defined as follows by using the Ricci tensor ${\rm Ric}(g(t))$ (cf. [6]):

$\frac{\partial}{\partial t}g(t)=-2{\rm Ric}(g(t)),$

where $g(t)$ is the time dependent Riemannian metric on a Riemannian manifold
$(M,g(t))$. Therefore, Hessian manifolds can be deeply understood by
considering the geometric flow for the second Koszul form $\beta$. In fact,
$\beta$ plays a similar role to that of the Ricci tensor in a Hessian manifold
(i.e., a manifold with a Hessian structure). The flow is called the Hesse flow
(or the Hesse-Koszul flow) defined by M. Mirghafouri and F. Malek (cf. [7],
see also [9]):

$\frac{\partial}{\partial t}g(t)=2\beta(g(t)).$

They proved the short-time existence, the global existence and the uniqueness
of it. S. Puechmorel and T. D. Tô [9] studied some convergence theorems of it
on compact Hessian manifolds.

A self-similar solution i.e., a soliton equation plays an important and
fundamental role in the study of a geometric flow. In fact, the Ricci soliton
which is the self-similar solution to the Ricci flow plays an important role
in solving Poincare conjecture and the geometrization conjecture.

Therefore, in this paper, we define the self-similar solution to the Hesse
flow and study it. In particular, we show that the expanding case is important
in compact Hesse solitons. We also show that one can understand that a non
trivial gradient Hesse soliton is interesting from the point of view
information geometry. In particular, under the second Koszul form coincides
with the dual one, any compact gradient Hesse soliton must be Hesse-Einstein.
Hesse-Einstein manifolds can be regarded to a notion of Einstein manifolds in
Riemannian geometry. Furthermore, we show that one can understand the dual
space of a Hesse-Einstein manifold as a Hesse soliton.

##  2\. Preliminary

In this section, we set up terminology and define some notions which are
related to Hessian manifolds and information geometry.

###  2.1. Riemannian geometry

Let $(M,g)$ be an $n$-dimensional Riemannian manifold. As is well known, the
Levi-Civita connection $\nabla:TM\times C^{\infty}(TM)\rightarrow
C^{\infty}(TM)$ is the unique connection on $TM$, which is compatible with the
metric and is torsion free:

$Xg(Y,Z)=g(\nabla_{X}Y,Z)+g(Y,\nabla_{X}Z),$ $\nabla_{X}Y-\nabla_{Y}X=[X,Y].$

The Riemannian curvature tensor is defined by

$Rm(X,Y)Z=\nabla_{X}\nabla_{Y}Z-\nabla_{Y}\nabla_{X}Z-\nabla_{[X,Y]}Z,$

for any vector field $X,Y,Z\in\mathfrak{X}(M).$ We use the notations
${R^{i}}_{jkl}$ as

$Rm\left(\frac{\partial}{\partial x_{k}},\frac{\partial}{\partial
x_{l}}\right)\frac{\partial}{\partial
x_{j}}=\sum_{i}^{n}{R^{i}}_{jkl}\frac{\partial}{\partial x_{i}},$

and $R_{ijkl}=g^{ip}{R^{p}}_{jkl}$. The Ricci and scalar curvatures are
defined by $R_{ij}=R_{ijik}$ and $R=R_{ii}$.

###  2.2. Hessian manifolds

Let $M$ be an $n$-dimensional smooth manifold. A connection $D$ is said to be
flat if $D$ satisfies that it is torsion free and the curvature tensor
$Rm^{D}$ vanishes, that is,

$D_{X}Y-D_{Y}X=[X,Y],$

and

$Rm^{D}(X,Y)Z:=D_{X}D_{Y}Z-D_{Y}D_{X}Z-D_{[X,Y]}Z=0.$

######  Definition 2.1.

A Riemannian metric $g$ on a flat manifold $(M,D)$ is called a Hessian metric
if $g$ can be locally expressed by

$g=Dd\varphi,$

for some smooth function $\varphi$, that is,

$g_{ij}=\frac{\partial^{2}\varphi}{\partial x_{i}\partial x_{j}},$

for an affine coordinate system. $(D,g)$ is called a Hessian structure.
$(M,D,g)$ is called a Hessian manifold.

Let $D^{\prime}=2\nabla-D$, then $D^{\prime}$ is also a flat connection and
$(D^{\prime},g)$ is a Hessian structure. $D^{\prime}$ is called the dual
connection and $(D^{\prime},g)$ is called the dual Hessian structure of
$(D,g)$.

The flat connection $D$ and the dual one $D^{\prime}$ satisfy that

$Xg(Y,Z)=g(D_{X}Y,Z)+g(Y,D^{\prime}_{X}Z).$

######  Definition 2.2.

Let $v_{g}$ be the volume form of $g$ and $X$ be a vector field on $M$. The
first and second Koszul forms $\alpha$ and $\beta$ of $(M,D)$ are defined by

(2.1) $D_{X}v_{g}=\alpha(X)v_{g},$ (2.2) $\beta=D\alpha.$

We denote by $\gamma$ the difference tensor of $\nabla$ and $D$:

$\gamma_{X}Y=\nabla_{X}Y-D_{X}Y.$

Here we remark that since $D_{\partial_{i}}\partial_{j}=0$, the components
${\gamma^{i}}_{jk}$ of $\gamma$ with respect to affine coordinate systems
coincide with the Christoffel symbols ${\Gamma^{i}}_{jk}$ of $\nabla$, where
$\partial_{i}=\frac{\partial}{\partial x_{i}}$.

A tensor field $H$ of type $(1,3)$ defined by the covariant differential

$H=D\gamma$

of $\gamma$ is said to be the Hessian curvature tensor for $(D,g)$. The
components ${H^{i}}_{jkl}$ of $H$ is given by

${H^{i}}_{jkl}=\frac{\partial{\gamma^{i}}_{jl}}{\partial x_{k}}.$

######  Proposition 2.3 (Proposition 3.4 in [11]).

On a Hessian manifold, the following holds:

$(1)$ $\alpha(X)={\rm Trace}\,\gamma_{X}$.

$(2)$ $\alpha_{i}={\gamma^{r}}_{ri}$.

$(3)$ $\beta_{ij}={H^{r}}_{rij}={H_{ijr}}^{r}$.

The second Koszul form $\beta$ plays a similar role to that of the Ricci
tensor and one can define Hesse-Einstein manifolds:

######  Definition 2.4.

Let $(M,D,g)$ be a Hessian manifold. If the following holds

$\beta=\lambda g,$

then $(M,D,g)$ is called a Hesse-Einstein manifold.

######  Proposition 2.5 (Proposition 2.3 in [11]).

The curvature tensor of a Hessian manifold $(M,D)$ is given as follows.

(2.3)
${R^{i}}_{jkl}={\gamma^{i}}_{lr}{\gamma^{r}}_{jk}-{\gamma^{i}}_{kr}{\gamma^{r}}_{jl}.$

This implies the following.

(2.4) $\displaystyle
R_{jk}={R^{s}}_{jsk}={\gamma^{s}}_{kr}{\gamma^{r}}_{js}-\alpha_{r}{\gamma^{r}}_{jk},$

and

(2.5) $\displaystyle R=|{\gamma}|^{2}-|\alpha|^{2}.$

We can use the following notations without confusion: For example,
$\gamma_{ijk}\gamma_{ist}={\gamma^{i}}_{jk}\gamma_{ist},$
$H_{rrij}={H^{r}}_{rij}$, $\alpha_{r}\gamma_{rij}=\alpha^{r}\gamma_{rij}$,
etc.

###  2.3. Information geometry

Hessian manifolds play an important role in information geometry: For
$\Omega_{n}=\\{1,2,\cdots,n\\}$, let

$S_{n-1}:=\left\\{p:\Omega_{n}\rightarrow\mathbb{R}_{+};\sum_{\omega\in\Omega_{n}}p(\omega)=1\right\\}$

be a set of all probability distribution on $\Omega_{n},$ where
$\mathbb{R}_{+}=\\{x\in\mathbb{R};x>0\\}$. As is well known, one can regard it
as a manifold (see for example [5]). A metric $g^{F}$ on $S_{n-1}$ such as
$g^{F}_{p}(X,Y)=\sum_{\omega=1}^{n}p(\omega)(X\log p(\omega))(Y\log
p(\omega))$ is called a Fisher information metric. For each
$\alpha\in\mathbb{R}$, $\nabla^{(\alpha)}$ is determined by

$g^{F}_{p}(\nabla^{(\alpha)}_{X}Y,Z)=g^{F}_{p}(\nabla_{X}Y,Z)-\frac{\alpha}{2}\sum_{\omega=1}^{n}p(\omega)(X\log
p(\omega))(Y\log p(\omega))(Z\log p(\omega)),$

where $\nabla$ is the Levi-Civita connection compatible with $g^{F}.$
$\nabla^{(\alpha)}$ is called the $\alpha$-connection.

Chentsov (cf. [3]) shows that an extremely natural invariance requirement of
$S_{n-1}$ determines a metric and a connection of $S_{n-1}$, that is, the
metric is the Fisher information metric and the connection is the
$\alpha$-connection on $S_{n-1}$. This means that on information geometry, the
$\alpha$-connection is the most natural connection.

The $\alpha$-connection $\nabla^{(\alpha)}$ satisfies that

$Xg^{F}(Y,Z)=g^{F}(\nabla^{(\alpha)}_{X}Y,Z)+g^{F}(Y,\nabla^{(-\alpha)}_{X}Z).$

The most important case is $\alpha=1$. It is known that for
$(g^{F},\nabla^{(1)},\nabla^{(-1)})$, $S_{n-1}$ is the dual flat manifold and
$g^{F}$ can be written $g^{F}_{ij}=\partial_{i}\partial_{j}\varphi$ for an
affine coordinate system (cf [1]). Hence, $(S_{n-1},\nabla^{(1)},g^{F})$ is a
Hessian manifold with $\nabla^{(1)}\not=\nabla.$

Therefore, to apply Hessian geometry to information geometry, it is important
to consider a Hessian manifold $(M,D,g)$ with $D\not=\nabla.$ From this, we
define the following:

######  Definition 2.6.

Let $(M,D)$ be a Hessian manifold. If $D\not=\nabla$, $M$ is called a proper
Hessian manifold.

##  3\. Hesse solitons

Let $(M,D,g)$ be a Hessian manifold. In this section, we consider self-similar
solutions to the Hesse flow $\partial_{t}g=2\beta$. We first consider the
Hesse flow from the point of view of the Laplacian on Hessian manifolds. H.
Shima [10] considered the Laplacian on Hessian manifolds.

######  Definition 3.1 ([10]).

Let $\mathcal{A}^{p,q}$ be the tensor product
$(\overset{p}{\wedge}TM)\otimes(\overset{q}{\wedge}T^{*}M)$. Let $v_{g}$ be
the volume element determined by $g$. We identify $v_{g}$ with $v_{g}\otimes
1\in\mathcal{A}^{n,0}$ and set $\overline{v}_{g}=1\otimes
v_{g}\in\mathcal{A}^{0,n}$. For any vector field $X$, we define interior
product operators by

$i(X):\mathcal{A}^{p,q}\rightarrow\mathcal{A}^{p-1,q},~{}~{}i(X)\omega=\omega(X,\cdots;\cdots),$
$\overline{i}(X):\mathcal{A}^{p,q}\rightarrow\mathcal{A}^{p,q-1},~{}~{}\overline{i}(X)\omega=\omega(\cdots;X,\cdots).$

A coboundary operator
$\partial:\mathcal{A}^{p,q}\rightarrow\mathcal{A}^{p+1,q}$ is defined by

$\partial=e(dx_{i})D_{\frac{\partial}{\partial x_{i}}},$

where $e$ is an exterior product operator defined by

$e(\omega):\eta\in\mathcal{A}^{p,q}\rightarrow\omega\wedge\eta\in\mathcal{A}^{p+r,q+s},$

for $\omega\in\mathcal{A}^{r,s}.$

The adjoint operator of $\partial$ is denoted by
$\delta=(-1)^{p}\star^{-1}\partial\star$ on the space $\mathcal{A}^{p,q}$ ,
where $\star$ is the star operator
$\star:\mathcal{A}^{p,q}\rightarrow\mathcal{A}^{n-p,n-q}$ defined by

$\displaystyle(\star\omega)$
$\displaystyle(X_{1},\cdots,X_{n-p};Y_{1},\cdots,Y_{n-q})v_{g}\wedge\overline{v}_{g}$
$\displaystyle=$
$\displaystyle~{}\omega\wedge\overline{i}(X_{1})g\wedge\cdots\wedge\overline{i}(X_{n-p})g\wedge
i(Y_{1})g\wedge\cdots\wedge i(Y_{n-q})g.$

Then, we can define the Laplacian on a Hessian manifold as follows

$\Delta=\partial\delta+\delta\partial.$

By the above definition, Shima [10] showed that

$\Delta g=\beta.$

Therefore, interestingly, one can obtain that the Hesse flow can be written as

$\frac{\partial}{\partial t}g(t)=2\Delta g(t).$

In the following of this section, we consider a self-similar solution to the
Hesse flow:

$g(t)=\sigma(t)\psi^{*}(t)g(0),$

where $g(0)$ is the Hessian metric $g$,
$\sigma(t):\mathbb{R}\rightarrow\mathbb{R_{+}}$ is a smooth function and
$\psi(t):M\rightarrow M$ is a 1-parameter family of diffeomorphisms. By
differentiating, we have

(3.1)
$2\beta(g(t))=\frac{d}{dt}\sigma(t)\psi^{*}(t)g(0)+\sigma(t)\psi^{*}(t)(\mathcal{L}_{X}g(0)),$

where $\mathcal{L}_{X}$ denotes the Lie derivative by the time dependent
vector field $X$ such that $X(\psi(t)(p))=\frac{d}{dt}(\psi(t)(p))$ for any
$p\in M$.

######  Claim 1.

$\beta(cg)=\beta(g)$ for any positive constant $c\in\mathbb{R}$.

###### Proof.

Let $v_{g}$ and $v_{cg}$ be the volume forms of $g$ and $cg$, respectively.
Assume that $\alpha^{c}$ is the first Koszul form for $cg$. By definition
(2.1), we have

$D_{X}v_{cg}=\alpha^{c}(X)v_{cg}.$

From this and the definition of the volume form, we obtain

$D_{X}v_{g}=\alpha^{c}(X)v_{g}.$

Therefore, we have

$\alpha^{c}=\alpha.$

From this and the definition of the second Koszul form (2.2),

$\beta(cg)=D\alpha^{c}=D\alpha=\beta(g).$

∎

By (3.1) and Claim 1, one has

(3.2) $2\beta(g(t))=\frac{d}{dt}\sigma(t)g(t)+\mathcal{L}_{Y}g(t),$

where $Y(t)=\sigma(t)X(t)$. Therefore, we define self-similar solutions to the
Hesse flow as follows:

######  Definition 3.2.

Let $(M,D,g=Dd\varphi)$ be a Hessian manifold. If there exist a vector field
$X$ and $\lambda\in\mathbb{R}$, such that

(3.3) $\beta-\frac{1}{2}\mathcal{L}_{X}g=\lambda g,$

then, $M$ is called a Hesse soliton. If $\lambda>0,\lambda=0,\lambda<0$, then
the Hesse soliton is called expanding, steady or shrinking, respectively. If
there exists a smooth function $f$ on $M$ such that $X={\rm grad}\,f$, that
is,

(3.4) $\beta-\nabla\nabla f=\lambda g,$

then the Hesse soliton $(M,D,g,f)$ is called a gradient Hesse soliton, where
$\nabla\nabla f$ is the Hessian of $f$. $f$ is called a potential function.

Hesse-Einstein manifolds are trivial solutions of Hesse solitons. Therefore,
if a Hesse soliton is Hesse-Einstein, then it is called trivial. If a Hesse
soliton is a proper Hessian manifold, then it is called a proper Hesse
soliton.

##  4\. Existence and non-existence theorems for
Hesse solitons

In this section, we show some existence and non-existence theorems for Hesse
solitons.

######  Theorem 4.1.

$(1)$ There exist no compact shrinking Hesse solitons.

$(2)$ Any compact steady Hesse soliton is non proper and trivial.

Unlike in the case (1), in the case (2), we remark that there exist non proper
and trivial steady Hesse solitons. We will consider it later.

We first show that the following lemma.

######  Lemma 4.2.

On any Hessian manifold, the following formula holds.

(4.1) $\displaystyle\frac{1}{2}\Delta R=$
$\displaystyle~{}~{}\nabla_{i}\nabla_{j}\alpha_{k}\gamma_{ijk}-\nabla_{r}\nabla_{r}\alpha_{i}\alpha_{i}+|\nabla\gamma|^{2}+|{\rm
Rm}|^{2}+|{\rm Ric}|^{2}$
$\displaystyle+R_{ij}\beta_{ij}-R_{ij}\nabla_{i}\alpha_{j}.$

###### Proof.

Since

$\gamma_{ijk}=\frac{1}{2}\frac{\partial g_{ij}}{\partial
x_{k}}~{}~{}~{}\text{and}~{}~{}~{}g_{ij}=\partial_{i}\partial_{j}\varphi,$

we have

$\nabla_{i}\gamma_{jkl}=\frac{1}{2}\partial_{i}\partial_{j}\partial_{k}\partial_{l}\varphi-(\gamma_{rij}\gamma_{rkl}+\gamma_{rik}\gamma_{rjl}+\gamma_{ril}\gamma_{rjk}).$

This means that $\nabla_{i}\gamma_{jkl}$ is symmetry with respect to
$i,j,k,l$.

A direct computation shows that

$\displaystyle\nabla_{r}\nabla_{r}\gamma_{ijk}=$
$\displaystyle~{}\nabla_{r}\nabla_{i}\gamma_{rjk}$ $\displaystyle=$
$\displaystyle~{}\nabla_{i}\nabla_{r}\gamma_{rjk}+R_{rirp}\gamma_{pjk}+R_{rijp}\gamma_{rpk}+R_{rikp}\gamma_{rjp}$
$\displaystyle=$
$\displaystyle~{}\nabla_{i}\nabla_{r}\gamma_{rjk}+(\gamma_{rpt}\gamma_{tir}-\alpha_{t}\gamma_{tip})\gamma_{pjk}$
$\displaystyle+(\gamma_{rpt}\gamma_{tij}-\gamma_{rjt}\gamma_{tip})\gamma_{rpk}+(\gamma_{rpt}\gamma_{tik}-\gamma_{rkt}\gamma_{tip})\gamma_{rjp}$
$\displaystyle=$
$\displaystyle~{}\nabla_{i}\nabla_{j}\alpha_{k}+\gamma_{rpt}(\gamma_{tir}\gamma_{pjk}+\gamma_{tij}\gamma_{rpk}+\gamma_{tik}\gamma_{rjp})$
$\displaystyle-\alpha_{t}\gamma_{tip}\gamma_{pjk}-\gamma_{rjt}\gamma_{tip}\gamma_{rpk}-\gamma_{rkt}\gamma_{tip}\gamma_{rjp},$

where the first and third equalities follow from the symmetric property of
$\nabla_{i}\gamma_{jkl}$ with respect to $i,j,k,l$, and the second one follows
from the Ricci identity. From this, one has

(4.2) $\displaystyle\frac{1}{2}\Delta|\gamma|^{2}=$
$\displaystyle~{}\nabla_{r}\nabla_{r}\gamma_{ijk}\gamma_{ijk}+|\nabla\gamma|^{2}$
$\displaystyle=$
$\displaystyle~{}\\{\nabla_{i}\nabla_{j}\alpha_{k}+\gamma_{rpt}(\gamma_{tir}\gamma_{pjk}+\gamma_{tij}\gamma_{rpk}+\gamma_{tik}\gamma_{rjp})$
$\displaystyle-\alpha_{t}\gamma_{tip}\gamma_{pjk}-\gamma_{rjt}\gamma_{tip}\gamma_{rpk}-\gamma_{rkt}\gamma_{tip}\gamma_{rjp}\\}\gamma_{ijk}+|\nabla\gamma|^{2}.$

Substituting

$\displaystyle|{\rm Rm}|^{2}=$ $\displaystyle~{}R_{ijkl}R_{ijkl}$
$\displaystyle=$
$\displaystyle~{}\gamma_{rpt}\gamma_{tir}\gamma_{pjk}\gamma_{ijk}+\gamma_{rpt}\gamma_{tij}\gamma_{rpk}\gamma_{ijk}-\gamma_{rjt}\gamma_{tip}\gamma_{rpk}\gamma_{ijk}-\gamma_{rkt}\gamma_{tip}\gamma_{rjp}\gamma_{ijk},$

into (4.2), we have

$\displaystyle\frac{1}{2}\Delta|\gamma|^{2}=$
$\displaystyle~{}\nabla_{i}\nabla_{j}\alpha_{k}\gamma_{ijk}+|\nabla\gamma|^{2}+|{\rm
Rm}|^{2}$
$\displaystyle+\gamma_{rpk}\gamma_{tik}\gamma_{rjp}\gamma_{ijk}-\alpha_{t}\gamma_{tip}\gamma_{pjk}\gamma_{ijk}.$

From this, (2.4),

$\displaystyle|{\rm
Ric}|^{2}=\gamma_{skr}\gamma_{rjs}\gamma_{kip}\gamma_{ijp}-\gamma_{skr}\gamma_{rjs}\alpha_{i}\gamma_{ijk}-\alpha_{r}\gamma_{rjk}\gamma_{ikp}\gamma_{pji}+\alpha_{r}\alpha_{i}\gamma_{rjk}\gamma_{ijk},$

and

$\nabla_{i}\alpha_{j}=\beta_{ij}-\gamma_{rij}\alpha_{r},$

one has

$\displaystyle\frac{1}{2}\Delta|\gamma|^{2}=$
$\displaystyle~{}\nabla_{i}\nabla_{j}\alpha_{k}\gamma_{ijk}+|\nabla\gamma|^{2}+|{\rm
Rm}|^{2}+|{\rm
Ric}|^{2}+\gamma_{ijk}\gamma_{rjk}\alpha_{p}\gamma_{ipr}-\alpha_{i}\alpha_{r}\gamma_{ijk}\gamma_{rjk}$
$\displaystyle=$
$\displaystyle~{}~{}\nabla_{i}\nabla_{j}\alpha_{k}\gamma_{ijk}+|\nabla\gamma|^{2}+|{\rm
Rm}|^{2}+|{\rm Ric}|^{2}+R_{ij}\beta_{ij}-R_{ij}\nabla_{i}\alpha_{j}.$

Therefore, we have

$\displaystyle\frac{1}{2}\Delta R=$
$\displaystyle~{}~{}\nabla_{i}\nabla_{j}\alpha_{k}\gamma_{ijk}-\nabla_{r}\nabla_{r}\alpha_{i}\alpha_{i}+|\nabla\gamma|^{2}+|{\rm
Rm}|^{2}+|{\rm Ric}|^{2}$
$\displaystyle+R_{ij}\beta_{ij}-R_{ij}\nabla_{i}\alpha_{j}.$

∎

By using the above lemma, one can show Theorem 4.1.

###### Proof of Theorem 4.1.

By taking the trace of (3.3), we have

$\beta_{ii}-{\rm div}\,X=\lambda n.$

From this, one has

$\displaystyle\lambda\,n\,\mathrm{Vol}(M,g)=$
$\displaystyle~{}\int_{M}\beta_{ii}-{\rm div}\,Xv_{g}$ $\displaystyle=$
$\displaystyle~{}\int_{M}\nabla_{i}\alpha_{i}+|\alpha|^{2}-{\rm div}\,Xv_{g}$
$\displaystyle=$ $\displaystyle~{}\int_{M}|\alpha|^{2}v_{g}\geq 0,$

where the last equality follows from Stokes’ theorem. Hence, one has
$\lambda\geq 0$. Therefore, there exist no compact shrinking Hesse solitons.

If $\lambda=0$, then one has $\alpha=0$. Furthermore, by Lemma 4.2, we have

$\displaystyle\frac{1}{2}\Delta R=$ $\displaystyle~{}|\nabla\gamma|^{2}+|{\rm
Rm}|^{2}+|{\rm Ric}|^{2}.$

By Green’s formula, one has

$\int_{M}|{\rm Ric}|^{2}+|{\rm Rm}\,|^{2}+|\nabla\gamma|^{2}v_{g}=0.$

Therefore, $M$ is flat, in particular $R=0.$ From this and (2.5), we have
$\gamma=0$, that is, $D=\nabla.$ Furthermore, we also have $\beta=0$.
Therefore, it is also trivial. ∎

As mentioned above, we consider the properness of steady Hesse solitons. The
equation of steady Hesse solitons is

$\beta-\frac{1}{2}\mathcal{L}_{X}g=0.$

From the proof of (2) of Theorem 4.1, any compact steady Hesse soliton is flat
and non proper. Hence, any compact steady Hesse soliton is

$\mathcal{L}_{X}g=0.$

One can construct many examples of steady Hesse solitons by taking the vector
field $X$ as Killing vector field:

######  Proposition 4.3.

Let $(M,g,X)$ be a non proper Hessian manifold with a Killing vector field $X$
$($and the Levi-Civita connection $\nabla)$. Then, $\beta$ and
$\mathcal{L}_{X}g$ vanishes, and therefore, $(M,g,X)$ is a steady Hesse
soliton.

We remark that if $\alpha=\alpha^{\prime}$, then $D=\nabla$ on compact Hessian
manifolds. In fact, since the volume form is parallel,
$\alpha^{\prime}(X)=D^{\prime}_{X}v_{g}=(2\nabla-D)v_{g}=-D_{X}v_{g}=-\alpha(X)$,
that is, $\alpha=-\alpha^{\prime}$, thus one has $\alpha=\alpha^{\prime}=0.$
By Lemma 4.2 and Green’s formula, one has

$\int_{M}|{\rm Rm}|^{2}+|{\rm Ric}|^{2}+|\nabla\gamma|^{2}v_{g}=0.$

From this and (2.5), we have $\gamma=0$, that is, $D=\nabla.$ On a complete
Hessian manifold, it is well known that E. Calabi obtained the same conclusion
(cf. [2]), that is, any complete Hessian manifold with $\alpha=0$ satisfies
$D=\nabla.$

From the above arguments, we consider a more general problem. Obviously, if a
Hessian manifold $M$ is non proper, then $\beta=\beta^{\prime}(=0)$. In
particular, $\nabla\alpha=0.$ In fact, since $\alpha=-\alpha^{\prime}$, one
has
$\beta^{\prime}=D^{\prime}\alpha^{\prime}=-D^{\prime}\alpha=-(2\nabla-D)\alpha=\beta-2\nabla\alpha.$
Therefore, $\beta^{\prime}-\beta=2\nabla\alpha.$

However, the converse is not true, that is, even if $\beta=\beta^{\prime}$,
$M$ might not satisfies that $D=\nabla$. In fact, the following example
satisfies $\beta=\beta^{\prime}=\frac{n}{2}g$, but $D\not=\nabla.$ This means
that there exist proper Hesse-Einstein manifolds.

######  Example.

Let

$\Omega=\left\\{x\in\mathbb{R}^{n};x_{n}>\sqrt{\displaystyle\sum_{i=1}^{n-1}(x_{i})^{2}}\right\\}~{}~{}~{}\text{and}~{}~{}~{}\varphi=-\log\left(x_{n}^{2}-\left(\displaystyle\sum_{i=1}^{n-1}(x_{i})^{2}\right)\right).$

Then, $(\Omega,D,g=Dd\varphi)$ is a Hessian structure on $\Omega$.

From the above argument, it is interesting to consider the problem: “Does
there exist non trivial Hesse soliton with $\beta=\beta^{\prime}$ (that is,
$\nabla\alpha=0)$?”

We first consider a complete Einstein Hessian manifold with a non-negative
Einstein constant $\lambda$, that is, a Hessian manifold with ${\rm
Ric}=\lambda g$ with $\lambda\geq 0.$

######  Proposition 4.4.

Any complete Einstein Hessian manifold with a non-negative Einstein constant
$\lambda$ and $\beta=\beta^{\prime}$ is flat and $\nabla\gamma=0.$

###### Proof.

By the assumption,

$0=\nabla_{i}\alpha_{j}=\beta_{ij}-{\gamma^{r}}_{ij}\alpha_{r}=\beta_{ij}-g^{rs}\gamma_{rij}\alpha_{s}.$

From this, (4.1) and the assumption,

$\displaystyle\frac{1}{2}\Delta R=$ $\displaystyle~{}|\nabla\gamma|^{2}+|{\rm
Rm}|^{2}+|{\rm Ric}|^{2}+\lambda\beta_{ii}$ $\displaystyle=$
$\displaystyle~{}|\nabla\gamma|^{2}+|{\rm Rm}|^{2}+|{\rm
Ric}|^{2}+\lambda|\alpha|^{2}$ $\displaystyle\geq$
$\displaystyle\frac{1}{n}R^{2},$

where the last inequality follows from the Schwarz inequality. Since the Ricci
curvature is non-negative, by the Omori-Yau maximum principle (cf. [8], [12]),
$R=0.$ Hence $M$ is flat and $\nabla\gamma=0.$ ∎

######  Lemma 4.5.

Any Hesse soliton with $\beta=\beta^{\prime}$ satisfies that ${\rm div}\,X$ is
constant.

###### Proof.

Since

$0=\nabla_{i}\alpha_{j}=\beta_{ij}-{\gamma^{r}}_{ij}\alpha_{r}=\beta_{ij}-g^{rs}\gamma_{rij}\alpha_{s},$

we have

$\beta_{ii}=|\alpha|^{2}.$

Hence, one has

$\nabla_{k}\beta_{ii}=0.$

By the equation of Hesse solitons (3.3),

$0=\nabla_{k}\beta_{ii}=\nabla_{k}({\rm div}\,X+n\lambda)=\nabla_{k}{\rm
div}\,X,$

which implies that ${\rm div}\,X$ is constant. ∎

By Lemma 4.5, one can show the following:

######  Proposition 4.6.

If compact Hesse solitons satisfy $\beta=\beta^{\prime}$, then ${\rm
div}\,X=0$.

###### Proof.

By Lemma 4.5, ${\rm div}\,X$ is constant, say $C$. By Stokes’ theorem,

$0=\int_{M}{\rm div}\,Xv_{g}=C\,\mathrm{Vol}(M).$

Thus, $C=0$, that is, ${\rm div}\,X=0$. ∎

In particular, if $M$ is gradient, by the standard maximum principle, one can
obtain the following.

######  Corollary 4.7.

Any compact gradient Hesse soliton with $\beta=\beta^{\prime}$ is trivial.

A similar result for complete Hesse solitons can be obtained.

######  Proposition 4.8.

Any complete gradient Hesse soliton with $\beta=\beta^{\prime}$ and non-
negative Ricci curvature is trivial.

###### Proof.

By Lemma 4.5,

$\displaystyle\Delta|\nabla f|^{2}=$ $\displaystyle~{}2|\nabla\nabla
f|^{2}+2{\rm Ric}(\nabla f,\nabla f)+2g(\nabla f,\nabla\Delta f)$
$\displaystyle=$ $\displaystyle~{}2|\nabla\nabla f|^{2}+2{\rm Ric}(\nabla
f,\nabla f)\geq 0.$

Hence, Omori-Yau maximum principle shows that $|\nabla f|^{2}$ is constant,
say $C$. Assume that $C>0$. Since $\nabla\nabla f=0$, $\Delta f=0.$ From this,

$\displaystyle\Delta e^{f}=~{}|\nabla f|^{2}e^{f}+\Delta fe^{f}=~{}|\nabla
f|^{2}e^{f}>0.$

By Omori-Yau maximum principle again, $e^{f}$ is constant, that is, $f$ is
constant, which is a contradiction.

∎

By Corollary 4.7, it is interesting to consider non trivial gradient Hesse
solitons from the point of view of information geometry.

######  Corollary 4.9.

Any compact non trivial gradient Hesse soliton is proper.

###### Proof.

By Corollary 4.7, $\nabla\alpha\not=0$ at some point $p\in M$, i.e., on some
open set $\Omega\ni p$. Hence, $\nabla\gamma\not=0$ on $\Omega$. In fact, if
$\nabla\gamma=0$ at $q\in\Omega$, then we have $\nabla\alpha=0$ at $q$, which
is a contradiction.

Therefore, $\gamma\not=0$ on some set $\tilde{\Omega}$ of $M$, which means
that the soliton is proper.

∎

By the same argument, one can show the following.

######  Corollary 4.10.

Any complete non trivial gradient Hesse soliton with non-negative Ricci
curvature is proper.

##  5\. Dual Hessian structure

In this section, we consider the dual space of a Hessian manifold $(M,D,g)$
and show that one can understand the dual space of a Hesse-Einstein manifold
as a Hesse soliton.

######  Theorem 5.1.

Let $(M,D,g)$ be a Hesse soliton,

$\beta-\frac{1}{2}\mathcal{L}_{X}g=\lambda g,$

then the dual space $(M,D^{\prime},g)$ is also a Hesse soliton which satisfies
that

$\beta^{\prime}-\frac{1}{2}\mathcal{L}_{(X-2\alpha^{\sharp})}g=\lambda g,$

where $\sharp$ is a musical isomorphism $\sharp:TM^{*}\rightarrow TM$,

$g(\alpha^{\sharp},X)=\alpha(X),$

for any vector field $X$ on $M$.

###### Proof.

By the definition of the musical isomorphism $\sharp$,

$\displaystyle(\nabla\alpha)(Y,Z)=$ $\displaystyle~{}(\nabla_{Y}\alpha)(Z)$
$\displaystyle=$ $\displaystyle~{}Y\alpha(Z)-\alpha(\nabla_{Y}Z)$
$\displaystyle=$
$\displaystyle~{}Yg(\alpha^{\sharp},Z)-g(\alpha^{\sharp},\nabla_{Y}Z)$
$\displaystyle=$
$\displaystyle~{}g(\nabla_{Y}\alpha^{\sharp},Z)+g(\alpha^{\sharp},\nabla_{Y}Z)-g(\alpha^{\sharp},\nabla_{Y}Z)$
$\displaystyle=$ $\displaystyle~{}g(\nabla_{Y}\alpha^{\sharp},Z).$

Since $\beta$ and $\beta^{\prime}$ are symmetric 2 forms,
$\nabla\alpha=\frac{1}{2}(\beta^{\prime}-\beta)$ is also a symmetric 2 form.
Thus,

$\displaystyle 2(\nabla\alpha)(Y,Z)=$
$\displaystyle~{}(\nabla\alpha)(Y,Z)+(\nabla\alpha)(Z,Y)$ $\displaystyle=$
$\displaystyle~{}g(\nabla_{Y}\alpha^{\sharp},Z)+g(\nabla_{Z}\alpha^{\sharp},Y)$
$\displaystyle=$ $\displaystyle~{}\mathcal{L}_{\alpha^{\sharp}}g(Y,Z).$

Since $2\nabla\alpha=\beta-\beta^{\prime}$ and $(M,D,g)$ is a Hesse soliton

$\beta-\frac{1}{2}\mathcal{L}_{X}(Y,Z)=\lambda g,$

one has

$\displaystyle\beta^{\prime}(Y,Z)=$
$\displaystyle~{}\beta(Y,Z)-2(\nabla\alpha)(Y,Z)$ $\displaystyle=$
$\displaystyle~{}\frac{1}{2}\mathcal{L}_{X}g(Y,Z)+\lambda
g(Y,Z)-\mathcal{L}_{\alpha^{\sharp}}g(Y,Z)$ $\displaystyle=$
$\displaystyle~{}\frac{1}{2}\mathcal{L}_{(X-2\alpha^{\sharp})}g(Y,Z)+\lambda
g(Y,Z).$

∎

We consider gradient Hesse solitons. If the first Koszul form $\alpha$ is
exact, that is, $\alpha=dF$ for some smooth function $F$ on $M$, then the
Hesse soliton of the dual space is also gradient.

######  Corollary 5.2.

Let $(M,D,g,f)$ be a gradient Hesse soliton, such that the first Koszul form
is exact, that is, $\alpha=dF$ for some smooth function $F$ on $M$. Then the
dual space $(M,D^{\prime},g)$ is also a gradient Hesse soliton with the
potential function $f-2F.$

###### Proof.

Since $\alpha=dF$, we have

$g(\alpha^{\sharp},Y)=\alpha(Y)=dF(Y)=XF=g(\nabla F,Y),$

for any vector field $Y$ on $M$. Thus, we have

$\alpha^{\sharp}=\nabla F.$

By Theorem 5.1, the proof is complete. ∎

One can understand the dual space of Hesse-Einstein manifolds as Hesse
solitons.

######  Corollary 5.3.

Let $(M,D,g)$ be a Hesse-Einstein manifold,

$\beta=\lambda g,$

then the dual space is a Hesse soliton $(M,D^{\prime},g,-2\alpha^{\sharp})$,
that is, it satisfies

$\beta^{\prime}-\frac{1}{2}\mathcal{L}_{(-2\alpha^{\sharp})}g=\lambda g.$

## References

  * [1] S. Amari, Information Geometry and Its Applications, Applied Mathematical Sciences, 194. Springer, [Tokyo], (2016).
  * [2] E. Calabi, Improper affine hyperspheres of convex type and a generalization of a theorem by K. Jörgens, Mich. Math. J. 5, (1958) 105-126.
  * [3] N. N. Chentsov, Statistical Decision Rules and Optimal Inference, American Mathematical Society, Providence, RI (1982).
  * [4] B. Chow, S-C. Chu, D. Glickenstein, C. Guenther, J. Isenberg, T. Ivey, D. Knopf, P. Lu, F. Luo and L. Ni, The Ricci Flow: Techniques and Applications Part1 Mathematical Surveys and Monographs 135, Amer. Math. Soc., (2007).
  * [5] A. Fujiwara, Foundations of information geometry, Makinoshoten, (2015).
  * [6] R. Hamilton, Three-manifolds with positive Ricci curvature, J. Differential Geom. (1982) 17, 255–306.
  * [7] M. Mirghafouri, F. Malek, Long-time existence of a geometric flow on closed Hessian manifolds, J. Geom. Phys. 119 (2017) 54-65.
  * [8] H. Omori, Isometric immersions of Riemannian manifolds, J. Math. Soc. Japan. (1967) 19, 205-214.
  * [9] S. Puechmorel and T. D. Tô, Convergence of the Hesse-Koszul flow on compact Hessian manifolds, arXiv:2001.02940 [math.DG].
  * [10] H. Shima, Vanishing theorems for compact Hessian manifolds, Ann. Inst. Fourier, Grenoble, 36-3 (1986), 183-205.
  * [11] H. Shima, The geometry of Hessian structures, World Scientific Publishing Co. Pte. Ltd., Hackensack, NJ, (2007).
  * [12] S.-T. Yau, Harmonic functions on complete Riemannian manifolds, Comm. Pure Appl. Math. 28, (1975), 201–228.


#  Self Expanding Convolutional Neural Networks

Alex Deaconu  University of British Columbia  Blaise Appolinary  University of
British Columbia  Sophia Yang  University of British Columbia  Qingze (Eric)
Li  University of British Columbia

(January 10, 2024)

###### Abstract

In this paper, we present a novel method for dynamically expanding
Convolutional Neural Networks (CNNs) during training, aimed at meeting the
increasing demand for efficient and sustainable deep learning models. Our
approach, drawing from the seminal work on Self-Expanding Neural Networks
(SENN), employs a natural expansion score as an expansion criteria to address
the common issue of over-parameterization in deep convolutional neural
networks, thereby ensuring that the model’s complexity is finely tuned to the
task’s specific needs. A significant benefit of this method is its eco-
friendly nature, as it obviates the necessity of training multiple models of
different sizes. We employ a strategy where a single model is dynamically
expanded, facilitating the extraction of checkpoints at various complexity
levels, effectively reducing computational resource use and energy consumption
while also expediting the development cycle by offering diverse model
complexities from a single training session. We evaluate our method on the
CIFAR-10 dataset and our experimental results validate this approach,
demonstrating that dynamically adding layers not only maintains but also
improves CNN performance, underscoring the effectiveness of our expansion
criteria. This approach marks a considerable advancement in developing
adaptive, scalable, and environmentally considerate neural network
architectures, addressing key challenges in the field of deep learning.

Keywords
machine learning, self expanding neural networks, computational efficiency,
convolutional neural networks.

##  1 Introduction

Convolutional Neural Networks (CNNs) have revolutionized the field of deep
learning, especially in processing grid-like data structures such as images
[1]. Their effectiveness in tasks like image classification [2, 3], object
detection [4, 5], semantic segmentation [6, 7] and image generation [8] stem
from their ability to effectively learn spatial features. Convolutional
layers, using filters or kernels, capture local patterns and extract features
from input images. One important feature of convolutional layers is the shared
weights implemented by kernels. This allows for efficient deep learning on
images, as using only fully connected layers for such tasks would result in
unfathomable numbers of parameters. Pooling layers, like max pooling and
average pooling, reduce the spatial dimensions of these features, helping the
network to focus on the most significant aspects.

Despite their popularity, CNNs face challenges in computational efficiency and
adaptability. There have been several convolutional neural network
architectures that have been proposed that are aimed at efficiency. Some of
such architectures include MobileNet [12] and EfficientNet [13]. However, such
traditional CNNs, with fixed architectures and number of parameters, may not
perform uniformly across different types of input data with varying levels of
complexity.

Neural Architecture Search (NAS), a method for selecting optimal neural
network architectures, has been a response to this challenge. NAS aims to
obtain the best model for a specific task under certain constraints [14].
However, NAS is often resource-intensive due to the need to train multiple
candidate models. in order to determine the optimal architecture. It is
estimated that the carbon emission produced when using NAS to train a
transformer model can amount to five times the lifetime carbon emissions of an
average car [15]. This highlights the importance of finding suitable
architectures for neural networks, yet also points to the limitations of
current approaches in terms of static structure and proneness to over-
parameterization.

Self Expanding Neural Networks (SENN), introduced in [9], offer a promising
direction. Inspired by neurogenesis, SENN dynamically adds neurons and fully
connected layers to the architecture during training using a natural expansion
score (defined in section 2.1) as a criteria to guide this process. This helps
overcome the problem of over-parametrization. However, its application has
been limited to multilayer perceptrons, with extensions to more practical
architectures like CNNs identified as a future research prospect.

Our study aims to develop a Self Expanding Convolutional Neural Network
(SECNN), building on the concept of SENN and applying it to modern vision
tasks. To the best of our knowledge, there has been no research on Self
Expanding CNNs, despite the potential they hold for addressing model
efficiency and adaptability in vision tasks. Unlike existing approaches that
often require restarting training after modifications or rely on preset
mechanisms for expansion, our approach utilizes the natural expansion score
for dynamic and optimal model expansion. This research represents a
significant step in developing adaptable, efficient CNN models for a variety
of vision-related tasks. The contributions of this research are as follows:

  * •

Developing a Self Expanding CNN that dynamically determines the optimal model
size based on the task, thereby enhancing efficiency.

  * •

Eliminating the need to train multiple CNN models of varying sizes by allowing
for the extraction of checkpoints at diverse complexity levels.

  * •

Eliminating the need to restart the training process after expanding the CNN
model.

##  2 Methodology

In order to develop a dynamically expanding convolutional neural-network
architecture, we need an expansion criteria that triggers when to expand the
model. The criteria we use is the natural expansion score as defined in
section 2.1.

###  2.1 Natural Expansion Score

The natural expansion score, as defined by [9], serves as a critical metric
for assessing the effectiveness and uniqueness of augmenting the capacity of a
neural network. This score is calculated by the inner product of the model’s
gradients and its natural gradients, encapsulated in the formula

$\displaystyle\eta=g^{T}F^{-1}g$ (1)

Here, $F$ represents the Fisher Information Matrix (FIM), a key factor in
understanding the loss landscape’s curvature. The natural gradient, $F^{-1}g$,
is an enhanced version of the standard gradient, adjusted to reflect the loss
landscape’s curvature. Recognizing the computational demands of calculating
the full FIM, we utilize the Empirical Fisher as a practical approximation,
focusing on the diagonal elements of the FIM. The empirical Fisher is given by

$\displaystyle
F=\frac{1}{N}\sum_{i=1}^{N}\nabla_{\theta}L(\theta;x_{i})\nabla_{\theta}L(\theta;x_{i})^{T}$
(2)

where:

  * •

$L(\theta;x_{i})$ is the loss function with respect to parameters $\theta$ for
the $i$-th data point $x_{i}$

  * •

$\nabla_{\theta}L(\theta;x_{i})$ is the gradient of the loss function $L$ with
respect to model parameters for the $i$-th data point $x_{i}$.

  * •

$\nabla_{\theta}L(\theta;x_{i})\nabla_{\theta}L(\theta;x_{i})^{T}$ is the
outer product of the gradient vector by itself

Intuitively, the natural expansion score $\eta$ captures the rate in loss
reduction under a natural gradient descent [9].

Additionally, we add a regularization term to the computation of the natural
expansion score to moderate the impact of adding new parameters to the
network. This regularization is particularly important as it introduces a cost
associated with increasing the model’s complexity and prevents overexpansion.
The modified formula for the natural expansion score, incorporating this
regularization, is given by

$\displaystyle\eta_{\text{regularized}}=\eta\times\exp(-\lambda_{n}\times\Delta
p^{2})$ (3)

where $\lambda_{n}$ represents the natural expansion score regularization
coefficient and $\Delta p^{2}$ represents the increase of parameters of the
proposed model from the previous one. The exponential decay factor,
$\exp(-\lambda_{n}\times\Delta p^{2})$, ensures that the score is
significantly reduced as the number of new parameters from the addition
increases. This regularization term is especially critical in preventing
unnecessary or excessive growth of the network, aiding the expansion process
to maintain an optimal balance between performance and complexity.

####  2.1.1 Adding a New Layer

When adding a new convolutional layer in the network, we ensure that the
functionality of the current architecture is not affected by initializing the
weights of the new layer as an Identity layer with a Gaussian noise. This
strategy transmits all information from the preceding layer through to the new
layer. This new layer adapts and learns its distinct weights later during
training. Furthermore, the addition of the Identity layer ensures that there
is a smooth integration of the layer into the network without a noticeable
drop in model performance in the short run. This eliminates the need to
restart training once we expand our model.

##  3 Architecture Design, Initial Configuration, and Expansion Criteria

Our neural network is built on a modular, block-based design, starting with a
sequence of convolutional blocks. Each block initially consists of a single
convolutional layer, a batch normalization layer, and a LeakyReLU activation
function. This flexible design allows us to specify the number of channels for
each block, supporting dynamic expandability.

Figure 1: The model architecture. Each block includes a CNN layer, a batch
normalization layer and a Leaky ReLU function. The blocks are separated by a
pooling layer denoted by p. We include a skip connection from the first block
to the output of the final block. We set a maximum capacity of each block to
$N$. During training, the network dynamically expands by either adding an
identity convolutional layer or upgrading the number of channels in a block,
provided it does not exceed the block’s capacity. These expansions occur when
the network identifies a need for increased complexity to improve performance.

The architecture processes input data through these blocks, engaging in
feature extraction and refinement via layers of convolutions, normalizations,
and activations. Pooling layers follow each convolutional block, crucial for
reducing spatial dimensions and preventing overfitting.

The network concludes with a 1x1 convolutional layer with Leaky ReLU
activation, a fully connected linear layer, and an output layer suitable for a
10-class classification dataset. Skip connections from the first to the final
block are included to address the vanishing gradient problem.

###  3.1 When, Where, and What to Expand

We set a threshold $\tau$ that plays a significant role in the expansion. In
determining when, where, and what to expand, our model employs the following
steps:

  * •

Calculate the natural expansion score $\eta_{c}$ of the current model.

  * •

Create a temporary model with weights copied from the current model.

  * •

For identity layer addition:

    * –

For each block, add an identity convolutional layer.

    * –

Track the block index leading to the highest natural expansion score
$\eta_{l}$.

  * •

For out channels increase:

    * –

For each block, add $C$ channels to every CNN layer, with Gaussian-distributed
weights.

    * –

Identify the block index yielding the highest expansion score $\eta_{i}$.

  * •

Decide expansion type:

    * –

If $\eta_{l}>\eta_{i}$, and $\frac{\eta_{l}}{\eta_{c}}>\tau$, add an identity
layer to the original model at the optimal index.

    * –

If $\eta_{i}>\eta_{l}$, and $\frac{\eta_{i}}{\eta_{c}}>\tau$, add channels to
layers in the original model at the optimal index.

    * –

If neither condition is met, the network does not expand at this moment.

In our experimentation, we set the threshold $\tau=2$.

###  3.2 Model Expansion Strategy and Implementation

The expansion strategy of the model allows for the addition of layers or
channels within set constraints. Each block can expand up to a maximum
capacity $N$ through either the addition of an identity convolutional layer or
increasing the number of output channels.

The decision to expand is based on the natural expansion score, evaluated
every epoch. Initially, channels are added in increments of 2, later adjusted
to 4 to ensure thoughtful expansions.

Post-expansion, a 10-epoch cooldown period is implemented for network
stabilization. When expanding a block, the output channels of all its layers
are increased linearly, and the input channels of the next block’s first layer
are adjusted. New channels are initialized with Gaussian noise (mean=0) and a
small coefficient (1e-4), balancing training progression with the integration
of new parameters.

###  3.3 Training

####  3.3.1 Training Dataset

For evaluating our classification model, we used the CIFAR-10 dataset [11]. It
comprises 60,000 colored images split 80-20 into training and testing sets.
The data is divided into 10 mutually exclusive classes, with 6,000 images per
class (5000 training, 1000 testing). Images are photo-realistic and only
contain the class object, which can be partially occluded. CIFAR-10 is widely
used and serves as a benchmark for evaluating model performance for image
classification. See Figure 2 for a breakdown of the dataset classes with
examples. The dataset was chosen for its large number of examples and its low
resolution, which help minimize training time and computational resources

Figure 2: The 10 classes of the CIFAR-10 dataset along with classes [10].

####  3.3.2 Training Method

We set our initial starting conditions to be a model consisting of 3 blocks,
each with 16 output layers 3 input channels, on an NVIDIA GeForce RTX 3090
GPU, running on a Windows Desktop.

We trained 5 models from identical starting conditions for 300 epochs, with a
batch size of 512 on the CIFAR-10 dataset. We set the initial learning rate to
2e-3, which gets reduced by half after every 15 epochs without improvement in
validation accuracy. To prevent overfitting we set our dropout rate to 0.1 for
all convolutional layers, and 0.05 for the fully connected layers. This small
dropout rate worked well, likely due to our model’s small size which also
prevented overfitting. To further combat overfitting, we used L1 regularized
loss, with a regularization coefficient of 1e-5. We utilized minimal data
augmentation in our training set, only using random horizontal transforms to
encourage the model to learn more robust features. Loss was calculated as
cross entropy. We set the maximum number of blocks per layer to be 10, to
allow the model ample room for linear expansion while enforcing a limit to
prevent overexpansion and prevent its associated problems such as the high
computational cost and vanishing gradients.

##  4 Results

The results of our trials can be found in Figure 3 and a comparison of our
model with other models on CIFAR-10 image classification can be found in
Figure 4.

|  | Number | of | Parameters |  |
---|---|---|---|---|---|---
Metric | Trial 1 | Trial 2 | Trial 3 | Trial 4 | Trial 5 | Mean
Val Accuracy (at 70%) | 13696 | 11360 | 11360 | 11360 | 9024 | 11360.0
Val Accuracy (at 80%) | 27852 | 51880 | 22636 | 37320 | 28588 | 33655.2
Highest Val Accuracy (%) | 83.4 | 84.6 | 84.6 | 84.5 | 83.2 | 84.1
Parameters at Highest Accuracy | 74564 | 73720 | 57808 | 66440 | 40960 | 62698.4
Figure 3: Table displaying the number of parameters required to achieve
different validation accuracies on CIFAR-10 over 5 different trials with the
same hyperparameters. Figure 4: Graph showing our best and smallest models
compared to other models under 2M parameters [17].

##  5 Discussion

The trials on our Self-Expanding Convolution Neural Network (SECNN) on the
CIFAR-10 dataset have yielded promising results, demonstrating our model’s
ability to dynamically grow and adjust to the complexity of the task. Our mean
validation accuracy of 84.1% is competitive considering our efficient approach
to developing a full size model, as well as when we consider its simplistic
approach. Its efficiency is further highlighted by the fact that this score
was achieved with an average of 62,698 parameters, and an 80% accuracy can be
achieved with a mere 22,636 parameters. This is only 6% lower than the
existing most efficient architecture, ResNet-8 which is more complex
architecture, features trainable activations, and over double the parameters
[16]. Compared to the other models in Figure 4, SECNN is not only highly
efficient but has a competitive validation accuracy as well. Looking at our
data, there is variation in results between trials. Due to our indeterministic
expansion method, there is considerable variability in our results between
trials. In some trials, the model opts to expand sooner and more aggressively,
while in others, it follows a more gradual expansion pattern. Because of our
indetermistic expansion method, there is much variability in our results
between trials, as in some trials the model decides to expand sooner and more
aggressively, while other times the model decides expands in a more
progressive pattern. We approximated the Fisher Information Matrix as the
Emprical Fisher Information Matrix, which is computationally efficient but
less accurate than other approximations. Utilizing the Kronecker method [18]
in our approach could potentially refine the calculation of the Fisher
Information Matrix, leading to more precise natural gradients and possibly
more effective expansions. It is important that we address the limitations in
our approach. For example, the CIFAR-10 dataset features very small images and
may not have a full range of complexities and detail compared to real-world
datasets. Therefore, our model’s performance and behaviour as it expands may
be different in more complex datasets, and may or may not be as effective.

##  6 Conclusion

In this article, we introduced the Self Expanding Convolutional Neural
Network, a dynamically expanding architecture that uses the natural expansion
score to optimize model growth. The CIFAR-10 dataset was used to train an
initial model consisting over 5 different trials, which resulted in a 84.1%
mean validation accuracy. Our model demonstrates how a Self Expanding CNN
offers a computationally efficient solution to dynamically determine an
optimal architecture for vision tasks while eliminating the need to restart or
train multiple models.

## Acknowledgements

We would like to thank Rupert Mitchell, Martin Mundt and Kristian Kersting,
whose work on Self Expanding Neural Networks greatly inspired our research.

## References

  * [1] A. Khan et al., "A Survey of the Recent Architectures of Deep Convolutional Neural Networks," 2020. [Online]. Available: http://dx.doi.org/10.1007/s10462-020-09825-6
  * [2] Simonyan et al., Very Deep Convolutional Networks for Large-Scale Image Recognition, 2015 [Online]. Available: https://arxiv.org/abs/1409.1556
  * [3] A. Krizhevsky, I. Sutskever, and G. E. Hinton, "ImageNet Classification with Deep Convolutional Neural Networks," in Advances in Neural Information Processing Systems 25 (NIPS 2012), 2012, [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2012/file/c399862d3b9d6b76c8436e924a68c45b-Paper.pdf
  * [4] J. Redmon et al., "You Only Look Once: Unified, Real-Time Object Detection,", 2015. [Online]. Available:https://arxiv.org/abs/1506.02640
  * [5] S. Ren et al., "Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks,", 2015.[Online]. Available: https://arxiv.org/abs/1506.01497
  * [6] O. Ronneberger, P. Fischer, and T. Brox, "U-Net: Convolutional Networks for Biomedical Image Segmentation,".2015. [Online]. Available: https://arxiv.org/abs/1505.04597
  * [7] O. Oktay et al., "Attention U-Net: Learning Where to Look for the Pancreas,", 2018. [Online]. Available https://arxiv.org/abs/1804.03999
  * [8] I. Goodfellow et al., "Generative Adversarial Networks," in Advances in Neural Information Processing Systems 27, 2014. [Online]. Available: https://arxiv.org/abs/1406.2661
  * [9] M. Mitchel et al., "Self-Expanding Neural Networks,", 2023. [Online]. Available: https://arxiv.org/abs/2307.04526
  * [10] A. Krizhevsky, “CIFAR-10 and CIFAR-100 datasets,” Toronto.edu, 2009. [Online]. Available: https://www.cs.toronto.edu/~kriz/cifar.html
  * [11] A. Howard et al., "MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications,".2017. [Online]. Available: https://www.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf
  * [12] Howard et al. (2017). MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications https://arxiv.org/abs/1704.04861
  * [13] M. Tan et al, "EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks," 2020. [Online]. Available: https://arxiv.org/pdf/1905.11946.pdf
  * [14] T. White et al., "Neural Architecture Search: Insights from 1000 Papers,", 2023. [Online]. Available: https://arxiv.org/abs/2301.08727
  * [15] E. Strubell, A. Ganesh, and A. McCallum, "Energy and Policy Considerations for Deep Learning in NLP," in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
  * [16] E. Pishchik, “Trainable Activations for Image Classification,” Preprints.org, Jan. 26, 2023. https://www.preprints.org/manuscript/202301.0463/v1 (accessed Jan. 10, 2024).
  * [17] Papers with Code., Image Classification on CIFAR-10 Leaderboard. https://paperswithcode.com/sota/image-classification-on-cifar-10
  * [18] R. Grosse, J. Martens, A Kronecker-factored approximate Fisher matrix for convolution layers, 2016. [Online]. Available: https://arxiv.org/abs/1602.01407


# Towards a Definition of Complex Software System

Jan Žižka  0009-0007-6483-0037
Faculty of Informatics
Masaryk University
Brno, Czech Republic
Botanická 68a, Brno, 60200
Email<EMAIL_ADDRESS>Bruno Rossi  0000-0002-8659-1520
Faculty of Informatics
Masaryk University
Brno, Czech Republic
Botanická 68a, Brno, 60200
Email<EMAIL_ADDRESS>Tomáš Pitner  0000-0002-2933-2290
Faculty of Informatics
Masaryk University
Brno, Czech Republic
Botanická 68a, Brno, 60200
Email<EMAIL_ADDRESS>
###### Abstract

Complex Systems were identified and studied in different fields, such as
physics, biology, and economics. These systems exhibit exciting properties
such as self-organization, robust order, and emergence. In recent years,
software systems displaying behaviors associated with Complex Systems are
starting to appear, and these behaviors are showing previously unknown
potential (e.g., GPT-based applications). Yet, there is no commonly shared
definition of a Complex Software System that can serve as a key reference for
academia to support research in the area. In this paper, we adopt the theory-
to-research strategy to extract properties of Complex Systems from research in
other fields, mapping them to software systems to create a formal definition
of a Complex Software System. We support the evolution of the properties
through future validation, and we provide examples of the application of the
definition. Overall, the definition will allow for a more precise, consistent,
and rigorous frame of reference for conducting scientific research on software
systems.

###### Index Terms:

Software System, Complex System Theory, Complex Software System

##  I Introduction

Complex Systems manifest multifaceted dependencies and interrelationships with
other systems and environments, making them difficult, if not impossible, to
model in their entirety [1, 2]. Complex Systems show properties that make them
peculiar, such as the property of emergent behavior, i.e., behavior deriving
from the different parts of a system that cannot easily be determined or
forecasted when components are observed in isolation.

Complex Systems theory focuses on understanding and explaining the behavior of
Complex Systems formed by interacting components [1]. The theory provides a
general framework and a set of methodologies to study the emergent properties
and dynamics embedded in Complex Systems. However, there is no agreed precise
definition of the term as different authors might have other points of view
[3]. Complex Systems have been studied in various fields [4, 5, 6], for
example, in social sciences by exploring the complex interactions of
individuals in cities.

The successes of software systems in the past years based on, for example,
Neural Networks, such as systems developed by DeepMind [7], or OpenAI [8],
bring a new range of software systems to wide attention. The appearance of
applications such as AlphaFold or ChatGPT and others in a very short time
suggests that many more such systems will rise soon.

The exciting behaviors of these new software systems, such as self-
organization and emergence, cannot be explained by inspecting the software
implementation they are based on. This range of software systems has specific
behaviors correlating with the behaviors of Complex Systems as defined by the
Complex Systems theory.

While in Computer Science, complexity is studied in different contexts, such
as code complexity and the complexity of algorithms [9], this paper focuses on
complexity in the context of software systems. Also, many software systems are
socio-technical systems where humans are part of the system rather than only
forming its environment. Our study is interested in pure technical systems
where humans are not part of the system but may build the system’s
environment.

Our work aims to provide a clear definition of a Complex Software System (CSS)
based on the theory-to-research strategy [10, 11], providing a frame of
reference about the properties of such systems in relation to what is
postulated by Complex Systems’ theory [1].

As the field of Complex Systems is evolving [2], we suggest a framework that
will also allow the evolution of the definition and terms. The precise
definitions will allow more straightforward and unambiguous communication
within academia and will be able to connect to existing and future real-world
Complex Software Systems. The definition will also provide boundaries for new
research fields with a degree of focus cleared of possible ambiguity due to
the lack of definitions.

To summarize, we have the following contribution in this article:

  * •

Setting up a framework under which the definition of a Complex Software System
is created;

  * •

Defining a Complex Software System based on reference to general Complex
Systems theory;

  * •

Listing examples of the use of such a definition;

  * •

Based on the definition and proposed use, list potential future research
directions;

The article is structured as follows. In Section II, we provide basic
definitions that are commonly adopted in the context of software systems when
discussing Complex Software Systems, such as System of Systems (SoS), Software
Ecosystems (SECO), and Complex Adaptive Systems (CAS). The purpose of the need
for a precise definition is discussed in Section III. Section IV discusses the
method for creating the definition of a Complex Software System. We select
several postulates in Section V to form an initial base for defining a Complex
Software System. Section VI provides examples of using such a definition.
Future research directions are discussed in Section VII, and conclusions are
presented in Section VIII.

##  II Basic Definitions

In Software Engineering, several commonly used terms and definitions of
software systems discuss how software systems and components can be combined
and aggregated. This section lists some of the main definitions and examines
their relationship to Complex Software Systems.

System of Systems (SoS) is a collection of independent interacting systems
[12]. An SoS has several key properties [13]:

  * •

Operational Independence. Any system part of an SoS is self-standing and can
operate even if the whole SoS is disaggregated.

  * •

Managerial Independence. Every single system in an SoS is self-governing.

  * •

Geographic Distribution. SoS are often distributed over geographic regions.

  * •

Evolutionary Development. The existence and development of SoS are under
constant change.

  * •

Emergent Behaviour. _”Through the collaboration between the systems in an SoS,
a synergism is reached in which the system behavior fulfills a purpose that
cannot be achieved by, or attributed to, any of the individual systems.”_ [13]

The systems which are part of an SoS may also be Complex Systems, or the SoS
as a whole may form a Complex System – however, the definition of an SoS does
not imply that such a system is a Complex System.

Software Ecosystems (SECO) are ”defined as a set of businesses functioning as
a unit and interacting with a shared market for software and services,
together with relationships among them” [14]. A SECO may be composed of
Complex Software Systems and is a type of SoS. SECOs are typically socio-
technical systems [15], which exhibit Complex System behaviors. In a SECO,
introducing new elements can potentially have disruptive effects. SECOs
features [16, 17] for example contain and provide:

  * •

Inherited characteristics of natural ecosystems like predation, parasitism,
mutualism, commensalism, symbiosis, and amensalism.

  * •

Architectural concepts like interface stability, evolution management,
security, and reliability.

  * •

Open source development model.

  * •

Platform for negotiating requirements aligning needs with solutions,
components, and portfolios.

  * •

Capability for process innovation.

  * •

Controlled central part for the core of the technology.

Complex Adaptive Systems (CAS) ”are systems that have a large number of
components, often called agents, that interact and adapt or learn” [18, 19].
The field of CAS focuses on the adaptive behavior of Complex Systems.

Software CAS refers to software systems that exhibit emergent behavior and
self-organization, similar to Complex Adaptive Systems found in nature. These
systems can adapt and evolve based on their interactions with the environment
through feedback loops. They involve multiple interacting components or agents
that collectively exhibit behavior that cannot be easily predicted from the
behavior of individual components [18].

A subset of Software CAS are Software Self-Adaptive Systems (SSAS) that focus
on the ability of a software system to autonomously adapt and modify the
behavior or configuration in response to changing conditions or requirements
[20, 21]. SSAS have built-in mechanisms that monitor the system’s state,
analyze environmental changes, and take actions to maintain or improve system
properties at runtime [20].

Software CAS [18, 19]:

  * •

Typically operate far from equilibrium.

  * •

Undergo revisions and improvements.

  * •

Do not conform to classic, equilibrium-based mathematical approaches.

  * •

Continuously adapt through recombination of the building blocks.

We summarize the main characteristics of SoS, SECO, and CAS in Table I.

TABLE I: SoS, SECO, CAS Concepts |  System of Systems (SoS) |  Software Ecosystems (SECO) |  Complex Adaptive Systems (CAS)
---|---|---|---
Definition |  Collection of independent interacting systems [12] |  Collection of software components, applications, and services [14] |  Collection of components (agents) that interact and evolve [18, 19]
Focus |  Collection of interacting systems |  Software and services relationships |  Software agents interaction
Emergent Behavior |  As systems get larger, emergent behavior is more probable [22] |  Limited emergence, the introduction of new elements might have disrupting effects |  Emergence in terms of adaptive behavior and self-organization
Examples |  Smart Grids Systems |  Android Ecosystem |  Robotic Swarm

##  III Why there is a need for a definition of a Complex Software System?

We need the definition of a Complex Software System for several reasons. Below
we discuss benefits, which are the motivators for the research presented in
this article.

Clarity and precision: ensure that the meaning of the term Complex Software
System is unambiguous.

Consistency: avoid that a software Complex System is defined differently by
different researchers or in other contexts, and ensure that the term’s meaning
remains consistent over time.

Rigor: provide a framework for scientific research. Scientific definitions are
necessary for the development of clear, precise, and consistent scientific
concepts and for the advancement of scientific research.

Once the definition has been developed, it can be used in various ways. For
example for:

Hypothesis testing: support the development of hypotheses about properties of
a Complex Software System and its behaviors so that they can be tested through
experiments and observations. For example, empirical methods can be used to
verify if a software system fulfills the necessary conditions for forming a
Complex Software System.

Classification: allow classification or categorization of Complex Software
Systems based on their properties. For example, software systems use different
paradigms, such as Neural Networks or multi-agent architectures. This can be
used to create classifications based on system boundaries, technology, or the
form of their implementation.

Comparison: a known Complex Software Systems can be compared based on the
definition with other systems to find similar or differing properties. This
can serve as grounds for expanding the definition or driving the creation of
similar software systems.

Theory development: develop new theories or models.

Definitions are essential for academic research, providing a clear and precise
framework for developing hypotheses, conducting experiments, and developing
theories. Definitions allow researchers to communicate effectively and to
build upon each other’s work.

##  IV Methodology to Build the Definition

To define a Complex Software System, we adopt the theory-to-research strategy,
in which there is a continuous cycle between theory and empirical validation
[10, 11]:

  * •

we extract from the literature (e.g., [1, 2]) common properties of Complex
Systems as have been studied in the different fields;

  * •

as there is no full agreement on all properties in the context of the theory
of Complex Systems [3], we discuss the most appropriate in the context of
software systems, both according to our view of software systems
implementation and deployment and with the aid of further related works [18,
23, 6, 4];

  * •

we map each of the properties to a set of identified necessary, sufficient and
representative conditions to the context of software systems;

  * •

we provide an initial application of the definition to showcase the main
benefits;

Ladyman [2] defines a Complex System based on reviewing attempts in the
literature to characterize a Complex System and compiles a set of necessary
conditions to represent complexity. In their work, authors provide conditions
that are qualitative and which may not be sufficient for complexity, but they
set a basis for a way how complexity can be defined. We suggest using the same
method for defining a Complex Software System.

We base the method of writing a formal definition of a Complex Software System
on a list of properties of three types that are necessary, sufficient, and
representative conditions written in natural language.

The set of properties is the definition of a Complex Software System.

The properties will be assembled in the form of graphical frames followed by a
commentary. Different types of properties will be color-coded for clarity in
the following way, where _”n”_ is an ordered number and _”keyword”_ is a word
abbreviating the property:

Property n - _”keyword”_ \- Necessary A property that is a necessary condition
but not sufficient to define a Complex Software System. Any Complex Software
System must be fulfilling all properties that are necessary conditions. But
fulfilling all such conditions doesn’t imply that the Software System is
Complex Software System.

Property n - _”keyword”_ \- Sufficient A property that is the sufficient
condition to define a Complex Software System. If a software system fulfils
any single property that is sufficient condition then such a system is a
Complex Software System.

Property n - _”keyword”_ \- Representative A property describing a typical
feature of a Complex System is a representative property. Such property is
neither necessary nor sufficient but does describe a commonality among Complex
Software Systems.

The nomenclature allows referring specific property such as Pn-N  _”keyword”_
for a necessary condition property, Pn-S  _”keyword”_ for sufficient condition
property and Pn-R  _”keyword”_ for a representative property.

The numbering of properties is sequential across all the types. The intention
and expectation is that the type of the property may change based on future
validation and research and keeping the numbering intact will allow for
unambiguous referencing.

##  V Initial Definition of a Complex Software System

A Complex Software System may be defined by a set of properties which may be
viewed as necessary, sufficient, and representative for a system to exhibit
Complex System behaviors. Such properties are generic for any Complex System
and are also described in existing publications such as [1, 2]. In this
section, we will summarize the basic properties of Complex Systems and put
those in the context of software systems, creating a base for the definition
of a Complex Software System.

Property 1 - _”components”_ \- Necessary A Complex Software System is composed
of many components.

All definitions of systems complexity [1, 2, 4, 6] require the system to have
many components. In the context of Complex Systems, the word _”many”_ refers
to the qualitative rather than quantitative nature of the term. It would,
therefore, be incorrect to attempt to quantify it. For example, a system
composed of two Complex Systems with manifold interactions and fulfilling
other necessary properties is a Complex System, as well as a system composed
of millions of components of a similar type, maybe a Complex System. This
property comes directly from the definition of the word ”system” [24, 25].
Software systems are typically composed of components. This property is a
necessary condition but not sufficient for a software system to exhibit
complexity. In software, a component may describe different entities based on
view or perspective. It can represent a code module, software package,
process, or service. From the perspective of a Complex Software System, only a
subset of such representations can serve as components in a Complex Software
System as they must possess further attributes discussed in the following
paragraphs.

Property 2 - _”communication”_ Necessary The components of a Complex Software
System have means of intercommunication.

Communication is an essential condition for a Complex Software System. As
Ladyman [2] explains: _”Without interaction, a system merely forms a “soup” of
particles which necessarily are independent and have no means of forming
patterns, of establishing order.”_. Communication through messaging shared
data, and interfaces is fundamental in software systems. However, this
property is not a sufficient condition for a Complex Software System, as many
software systems communicate but lack other necessary properties.

Property 3 - _”similarity”_ \- Representative The components in a Complex
Software System are similar.

Based on Ladyman [2]: _”For interactions to happen and for pattern and
coherence to develop, the elements have to be not only many but also similar
in nature.”_. From the software systems perspective, for example, a system
based on front-end, business logic middle-ware, and back-end database
components may not form a Complex Software System. This has fascinating
implications for software systems, which may be considered complex. However,
this condition is not sufficient to determine a Complex Software System. This
property may be necessary, but such a statement cannot be demonstrated and
proven with the current knowledge and it is a question if a Software System
with dis-similar components may still form a Complex Software System or if the
system boundaries would exclude such dis-similar components into system’s
environment rather than being part of the system itself. It also remains to be
defined what precisely _similar_ means in the context of software components.

Property 4 - _”interaction-change”_ \- Necessary The strength of components
interactions in a Complex Software System are not static, but they change over
time.

_”Most interactions are mediated through some sort of exchange process between
nodes (components). In that sense, interaction strength is often related to
the quantity of objects exchanged.”_ [1]. The interactions among components
have to change over time for a Complex Software System to evolve and create a
self-organized clustered structure [1]. The resulting network topology
contains information about the dynamics and formation of the nodes and links
(Chapter 4.5) [1]. This is a familiar property in software systems studies,
for example, in the field of dynamic or adaptive networks [26, 27]. This
property is necessary for a Complex Software System from which self-
organization and clustering emerge.

Property 5 - _”states”_ \- Necessary Components of a Complex Software System
are characterized by states.

Complex Software Systems are systems that evolve. An algorithmic description
of evolution (Chapter 5) [1] is based on the fact that the system has states,
and the evolution forms a path through states from time $t$ to time $t+1$.
Therefore the existence of states is necessary to create a Complex Software
System. The notion of states in software systems is among the basic concepts
of any information systems [28]. However, the existence of states is not a
sufficient condition for Complex Software Systems.

Property 6 - _”co-evolution”_ \- Representative The intercommunication and
states of components in a Complex Software System are not independent but co-
evolve.

As discussed in (Chapter 1.5) [2] _”Complex systems are characterized by the
fact that states and interactions are often not independent but evolve
together by mutually influencing each other; states and interactions co-
evolve.”_. The Complex Systems are characterized by co-evolutionary dynamics
(Chapter 4.8) [1]. From a software systems perspective, this can be
represented, for example, by adaptive network models [27], which are known to
exhibit such co-evolutionary dynamics [29, 30].

Property 7 - _”context-dependence”_ \- Representative A Complex Software
System is context-dependent.

As shown by Thurner [1], the Complex Systems are often represented by multi-
layer networks and _”… for any dynamic process happening on a given network
layer, the other layers represent the ’context’ …”_. In other words, such
context defines how components on different layers may be influenced. This
property is typical for Complex Systems to co-evolve through context
dependency.

Property 8 - _”algorithmicism”_ \- Representative A Complex Software System is
algorithmic.

Based on Thurner [1], the _”… (Complex Systems) algorithmic nature is a direct
consequence of the discrete interactions between interaction networks and
states.”_. This fits software systems that are naturally algorithmic [28].

Property 9 - _”path-dependence”_ \- Representative A Complex Software System
processes are path-dependent and non-ergodic.

_”The Complex Systems are typically governed by path-dependent processes.”_
(Section 2.5) [1]. The process, in a general theory of complex systems, refers
to stochastic processes [31]. This further means that processes in complex
systems are inherently non-Markovian. It can also be shown that Complex
Systems are non-ergodic (for in-depth discussion, see [1]). From the software
systems perspective, this means that software system to exhibit such Complex
System properties, they must change their boundary conditions as the system
evolves.

Property 10 - _”disorder”_ \- Necessary A Complex Software System is
disordered and out-of-equilibrium.

Ladyman [2] argues that _”…complex systems are precisely those whose order
emerges from a disorder rather than being built into them.”_. Also, it can be
noted that Complex Systems are generally out-of-equilibrium [1], which drives
interesting challenges to the concepts of entropy. Although it can be shown
[2, 1] that Complex Systems exhibit such properties, it is not obvious how to
apply those to software systems.

Property 11 - _”robust-order”_ \- Necessary A Complex Software System exhibits
robust order.

The concept of robust order is derived from system disorder. As shown by
Ladyman [2] _”… a system consisting of many similar components (elements)
which are communicating (interacting) in a disordered way has the potential of
forming patterns or structures”_. This refers to self-organization and
emergence property. From a software system perspective, this indicates that a
Complex Software System shall be composed of similar and at least initially
disordered components. This might seem to contradict with P10-N  _”disorder”_
but _”… although the elements continue to interact in a disordered way, the
overall patterns and structures are preserved. A macroscopic level arises out
of microscopic interaction, and it is stable”_ [2] which Ladyman defines it as
a robust order and continues that _”t(T)his kind of robust order is a further
necessary condition for a system to be complex”_. Therefore disorder and
robust order may co-exist. One example of software systems showing such a
property are Artificial Neural Networks (ANN), where initially, input weights
of neurons may be initialized with random values and, through learning, such
initial disorder forms patterns, structures, or clusters. Also, when examined
on a neuron level, ANN will still be disordered.

Property 12 - _”memory”_ \- Necessary A Complex Software System has memory.

From Holland [18]: _”A system remembers through the persistence of internal
structure”_ , Ladyman [2] infer that _”Memory is a straightforward corollary
of robust order.”_. And Thurner [1] notes that the _”Complex systems often
have memory. Information about the past can be stored in nodes (components),
if they have a memory, or in the network structure of the various layers.”_ In
such a sense, _memory_ refers to the internal self-organized structure of the
system. The difference between _memory_ and _states_ defined by P5-N
_”states”_ is that _states_ represent the system at a specific point in time,
but they do not represent history-dependent dynamics stored in the systems
_memory_. This property might have various interpretations in software
systems, such as a path through imitation-learning [32] or system audit
trails. This interesting property might also have yet unknown interpretations
in software systems.

Property 13 - _”SoS sufficiency”_ \- Sufficient A System of Complex Software
Systems is a Complex Software System.

As an intuitive analogy to properties of a Complex Software System – as in
Ackoff [12] – it may be possible to show that a system of Complex Software
Systems forms a Complex Software System.

This might have exciting implications in practice as once a Complex Software
System is created and exists, a new Complex Software System may be formed by
creating a system of such systems (SoS).

As the software does not require any material or physical manipulation and
software systems can be created relatively quickly, this allows the possible
rapid advancement of software-based systems exhibiting Complex System
behaviors.

##  VI Application of the definition

###  VI-A Unambiguous communication within academia

”Complex Software System” is a widely used term in academia and industry. It
refers to a wide range of software systems and viewpoints with a generic
notion of a system’s complexity. The definition presented in this paper
attempts to provide a concrete reference that can be utilized throughout
academic discussions to facilitate a common understanding of the term and
properties of such a system. Also, the proposed definition framework is
intended to extend and refine the definition to support further Complex
Software Systems theory development.

The definition of a Complex Software System may be referenced as a whole, or
specific properties may be the focus of empirical and other research when
studying the properties of software systems. Having a definition of a Complex
Software System will bring clarity through academic discussions.

###  VI-B Complex Software System categorization

Software systems are open systems [33] with external interactions. The
boundary of the system defines what belongs to the system itself and what its
surroundings are. The edge of the system may be used for categorization. Many
software systems nowadays are socio-technical systems [34] where people are
part of the system rather than creating the surroundings and interacting with
the system only through the system boundary.

The software systems also interact with humans or are part of machine-to-
machine interactions. The software system boundaries can be used as one of the
aspects of categorizing types of software systems. Most importantly, this
categorization has an essential perspective from Complex Software Systems
theory. Most of the socio-technical systems are Complex Systems [34], and the
involvement of humans fulfills the necessary conditions presented in Section
V.

For example, if we consider the Internet as a Complex Software System, it can
be viewed as a socio-technical system. In which case, it fulfills the P4-N
”interaction-change” property. The changes are done by human developers,
companies, and communities, which interconnect services throughout the
Internet. If the boundary of the Internet as a system excludes human actors,
the P4-N ”interaction-change” might not hold.

The presented properties applied to different boundaries of a software systems
can, in this way, provide mechanisms to create a categorization and
demonstrate which boundary is allowing the creation of a Complex System and
which is not, as they are not fulfilling the necessary conditions defined by
the presented properties.

###  VI-C Complex Software System modeling

To dive into understanding Complex Software Systems, it will be required to
have a model to analyze the properties’ effects, how the necessary and
sufficient conditions may be fulfilled or violated, and how representative
properties may help define a Complex Software System. This can be achieved,
for example, by studying existing Complex Systems, as it is done in other
fields. However, the challenge is that we might not have access to such
systems and, based on the boundary categorization (Section VI-B), some
categories of Complex Software Systems might not even exist, for example, pure
technical Complex Software Systems, where humans are outside the system
boundary.

The model can be designed and developed to study such Complex Software Systems
based on the presented definition and specification of necessary conditions
for such a system to exist.

The model will allow experiments to evaluate the assumptions placed by the
properties of Complex Software Systems. Understanding underlying principles
might show how such software systems may be constructed.

With models based on the defined Complex System properties, it may be, for
example, demonstrated that P13-S ”SoS sufficiency” is a sufficient condition.

Creating models of different categories of Complex Software Systems is another
research path we will follow.

##  VII Future research

Creation of the definition of a Complex Software System and a framework for
describing the definition will open doors for several research areas:

  * •

Search for and refining Complex Software System properties;

  * •

Exploring categories of Complex Software Systems;

  * •

Creation of Complex Software System models;

  * •

Search for underlying principles of Complex Software Systems;

The framework discussed in Section IV provides means of extending and refining
the definition based on future research in the field of Complex Software
Systems. The properties may be updated or expanded as new information becomes
available. This will require empirical validation of hypotheses and possibly
rejecting null hypotheses posed by the definition. The validation is expected
to trigger a refinement cycle for the theory, as defined by theory-to-research
strategy [10]. The validation may be based on case studies of existing
software systems or experimental research based on simulations and modeling.

The list of the initial 13 properties harvested from studies of Complex
Systems in other fields may not always have a precise mapping to software
systems. Some properties that define necessary conditions will require further
research to understand what they can indicate in the context of software
systems. Especially P7-R ”context-dependency”, P9-R ”path-dependency”, P10-N
”disorder”, P12-N ”memory” or P13-S ”SoS sufficiency”.

The categorization, discussed in Section VI-B, based on the definition, might
provide additional research topics while helping to uncover new underlying
general principles of Complex Software Systems.

##  VIII Conclusions

Complex Systems were identified and studied in different fields, such as
physics, biology, and economics. These systems exhibit properties such as
self-organization, robust order, and emergence. In recent years, software
systems started to display behaviors associated with Complex Systems, showing
previously unknown potential (e.g., GPT-based applications). However, a
commonly shared definition of a Complex Software System is not yet available.

For this reason, in this paper, we have presented a definition of a Complex
Software System that can serve as a reference for academia to support future
research. The definition is a set of 13 initial necessary, representative, and
sufficient conditions for a software system to exhibit Complex System
behaviors. The properties were selected from Complex Systems research in other
fields and mapped to software systems. We suggested allowing for evolution and
refinement of the properties, as the definition can be refined by evaluating
the studied properties using empirical methods. We have also provided examples
of the use of the definition and discussed further research directions in the
area of Complex Software Systems.

An unambiguous definition of a Complex Software System is a stepping stone
toward understanding its underlying principles.

## Acknowledgement

The work was supported from ERDF/ESF “CyberSecurity, CyberCrime and Critical
Information Infrastructures Center of Excellence” (No.
CZ.02.1.01/0.0/0.0/16_019/0000822).

## References

  * [1] S. Thurner, R. Hanel, and P. Klimek, _Introduction to the theory of complex systems_. Oxford University Press, 2018.
  * [2] J. Ladyman, J. Lambert, and K. Wiesner, “What is a complex system?” _European Journal for Philosophy of Science_ , vol. 3, no. 1, pp. 33–67, Jan 2013. doi: 10.1007/s13194-012-0056-8
  * [3] H. Ledford, “Language: Disputed definitions,” _Nature_ , vol. 455, no. 7216, pp. 1023–1028, Oct 2008. doi: 10.1038/4551023a
  * [4] M. Mitchell, _Complexity: A guided tour_. Oxford university press, 2009.
  * [5] G. J. Klir and H. A. Simon, _The architecture of complexity_. Boston, MA: Springer US, 1991, pp. 457–476.
  * [6] M. M. Waldrop, _Complexity: The emerging science at the edge of order and chaos_. Simon and Schuster, 1993.
  * [7] [Online]. Available: https://www.deepmind.com/
  * [8] [Online]. Available: https://openai.com/
  * [9] J. Van Leeuwen, _Handbook of theoretical computer science (vol. A) algorithms and complexity_. Cambridge, MA, USA: Mit Press, 1991. ISBN 0444880712
  * [10] R. A. Swanson and T. J. Chermack, _Theory building in applied disciplines_. Berrett-Koehler Publishers, 2013.
  * [11] P. D. Reynolds, _Primer in theory construction: An A &B classics edition_. Routledge, 2015.
  * [12] R. L. Ackoff, “Towards a system of systems concepts,” _Management science_ , vol. 17, no. 11, pp. 661–671, 1971. doi: 10.1287/mnsc.17.11.661
  * [13] C. B. Nielsen, P. G. Larsen, J. Fitzgerald, J. Woodcock, and J. Peleska, “Systems of systems engineering: basic concepts, model-based techniques, and research directions,” _ACM Computing Surveys (CSUR)_ , vol. 48, no. 2, pp. 1–41, 2015. doi: 10.1145/2794381
  * [14] D. G. Messerschmitt, C. Szyperski _et al._ , _Software ecosystem: understanding an indispensable technology and industry_. MIT press Cambridge, 2003, vol. 1.
  * [15] T. Lima, R. P. dos Santos, and C. Werner, “A survey on socio-technical resources for software ecosystems,” in _Proceedings of the 7th International Conference on Management of computational and collective intElligence in Digital EcoSystems_ , 2015. doi: 10.1145/2857218.2857230 pp. 72–79.
  * [16] J. Joshua, D. Alao, S. Okolie, and O. Awodele, “Software ecosystem: Features, benefits and challenges,” _International Journal of Advanced Computer Science and Applications_ , vol. 4, no. 8, 2013. doi: 10.14569/IJACSA.2013.040833
  * [17] D. Lettner, F. Angerer, H. Prähofer, and P. Grünbacher, “A case study on software ecosystem characteristics in industrial automation software,” in _Proceedings of the 2014 International Conference on Software and System Process_ , ser. ICSSP 2014. New York, NY, USA: Association for Computing Machinery, 2014. doi: 10.1145/2600821.2600826. ISBN 9781450327541 pp. 40–49.
  * [18] J. H. Holland, “Complex adaptive systems,” _Daedalus_ , vol. 121, no. 1, pp. 17–30, 1992.
  * [19] ——, “Studying complex adaptive systems,” _Journal of systems science and complexity_ , vol. 19, pp. 1–8, 2006. doi: 10.1007/s11424-006-0001-z
  * [20] R. De Lemos, H. Giese, H. A. Müller, M. Shaw, J. Andersson, M. Litoiu, B. Schmerl, G. Tamura, N. M. Villegas, T. Vogel _et al._ , “Software engineering for self-adaptive systems: A second research roadmap,” in _Software Engineering for Self-Adaptive Systems II: International Seminar, Dagstuhl Castle, Germany, October 24-29, 2010 Revised Selected and Invited Papers_. Springer, 2013. doi: 10.1007/978-3-642-02161-9_1 pp. 1–32.
  * [21] F. D. Macías-Escrivá, R. Haber, R. del Toro, and V. Hernandez, “Self-adaptive systems: A survey of current approaches, research challenges and applications,” _Expert Systems with Applications_ , vol. 40, no. 18, pp. 7267–7279, 2013. doi: 10.1016/j.eswa.2013.07.033
  * [22] J. S. Osmundson, T. V. Huynh, and G. O. Langford, “Emergent behavior in systems of systems,” in _INCOSE International Symposium_ , vol. 18, no. 1. Wiley Online Library, 2008. doi: 10.1002/j.2334-5837.2008.tb00900.x pp. 1557–1568.
  * [23] J. M. Ottino, “Complex systems,” _American Institute of Chemical Engineers. AIChE Journal_ , vol. 49, no. 2, p. 292, 2003. doi: 10.1002/aic.690490202
  * [24] Merriam-Webster. System. [Online]. Available: https://www.merriam-webster.com/dictionary/system
  * [25] O. E. Dictionary. system, n. [Online]. Available: https://www.oed.com/view/Entry/196665
  * [26] F. Kuhn and R. Oshman, “Dynamic networks: models and algorithms,” _ACM SIGACT News_ , vol. 42, no. 1, pp. 82–96, 2011. doi: 10.1145/1959045.1959064
  * [27] A. H. Sayed, “Adaptive networks,” _Proceedings of the IEEE_ , vol. 102, no. 4, pp. 460–497, 2014. doi: 10.1109/JPROC.2014.2306253
  * [28] I. Sommerville, _Software Engineering_ , ser. Always learning. Pearson, 2016. ISBN 9780133943030
  * [29] D. Antoniades and C. Dovrolis, “Co-evolutionary dynamics in social networks: A case study of twitter,” _Computational Social Networks_ , vol. 2, no. 1, pp. 1–21, 2015. doi: 10.1109/SITIS.2014.68
  * [30] M. Farajtabar, M. Gomez-Rodriguez, Y. Wang, S. Li, H. Zha, and L. Song, “Co-evolutionary dynamics of information diffusion and network structure,” in _Proceedings of the 24th International Conference on World Wide Web_ , 2015\. doi: 10.1145/2740908.2744105 pp. 619–620.
  * [31] S. M. Ross, _Stochastic processes_. John Wiley & Sons, 1995.
  * [32] O. Vinyals, I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi, R. Powell, T. Ewalds, P. Georgiev _et al._ , “Grandmaster level in starcraft ii using multi-agent reinforcement learning,” _Nature_ , vol. 575, no. 7782, pp. 350–354, 2019. doi: 10.1038/s41586-019-1724-z
  * [33] L. v. Bertalanffy, _General system theory: Foundations, development, applications_. G. Braziller, 1968.
  * [34] V. Tomic, “A bionic view on complex software systems-and the consequences for digital resilience,” 2021. doi: 10.34726/hss.2021.82584