# Parallel translations for a left invariant spray

Ming Xu  Ming Xu
School of Mathematical Sciences, Capital Normal University, Beijing 100048, P.
R. China<EMAIL_ADDRESS>
###### Abstract.

In this paper, we study the left invariant spray geometry on a connected Lie
group. Using the technique of invariant frames, we find the ordinary
differential equations on the Lie algebra describing for a left invariant
spray structure the linearly parallel translations along a geodesic and the
nonlinearly parallel translations along a smooth curve. In these equations,
the connection operator plays an important role. Using linearly parallel
translations, we provide alternative interpretations or proofs for some
homogeneous curvature formulae. Concerning the nonlinearly ones, we propose
two questions in left invariant spray geometry. One question generalizes
Landsberg Problem in Finsler geometry, and the other concerns the restricted
holonomy group.

Mathematics Subject Classification (2010): 53B40, 53C30, 53C60

Key words: connection operator, geodesic, holonomy group, invariant frame,
invariant spray structure, parallel translation, Riemann curvature, spray
vector field

##  1\. Introduction

Spray geometry concerns a spray structure $\mathbf{G}$ on a smooth manifold
$M$, which is a smooth vector field on $TG\backslash\\{0\\}$ with the standard
local coordinate presentation
$\mathbf{G}=y^{i}\partial_{x^{i}}-2\mathbf{G}^{i}\partial_{y^{i}}$, where
$\mathbf{G}^{i}=\mathbf{G}^{i}(x,y)$ is positive 2-homogeneous for the
$y$-entry [13]. For example, when $\mathbf{G}$ is induced by a Finsler metric
$F$, then
$\mathbf{G}^{i}=\tfrac{1}{4}g^{il}([F^{2}]_{x^{k}y^{l}}y^{k}-[F^{2}]_{x^{l}})$
[14]. Many notions in Finsler geometry, like (nonconstant) geodesic, Riemann
curvature, linear and nonlinear parallel translations, are only relevant to
$\mathbf{G}$, i.e., they are originated from spray geometry [1]. See
[5][10][11][19] for some recent progress in this field.

In this paper, we discuss a special class of homogeneous spray structure. A
spray structure $\mathbf{G}$ on a connected Lie group $G$ is called left
invariant, if it is preserved by all left translations [5][16]. Using a left
invariant frame $\\{\widetilde{U}_{i},\partial_{u^{i}},\forall i\\}$ on $TG$
(see Section 2.2 below or Section 3.1 in [16]), a left invariant spray
structure $\mathbf{G}$ can be presented as
$\mathbf{G}=\mathbf{G}_{0}-\mathbf{H}$, where
$\mathbf{G}_{0}=u^{i}\widetilde{U}_{i}$ is the canonical bi-invariant spray
structure, and $\mathbf{H}=\mathbf{H}^{i}\partial_{u^{i}}$ is a left invariant
vector field on $TG\backslash 0$ which is tangent to each $T_{g}G$. The
restriction $\eta=\mathbf{H}|_{T_{e}G\backslash\\{0\\}}$ is called the spray
vector field associated to $\mathbf{G}$ [16]. This notion was first proposed
by L. Huang in homogeneous Finsler geometry [2]. We usually present $\eta$ as
smooth function from $\mathfrak{g}\backslash\\{0\\}$ to $\mathfrak{g}$ (see
Section 2.3 below).

The philosophy of homogeneous geometry implies that, to explore a left
invariant $\mathbf{G}$, we only need to observe the interaction between the
dynamical system of $\eta$ and the Lie algebra structure of $\mathfrak{g}$.
Following this thought, we find homogeneous curvature formulae for
$\mathbf{G}$ (see (3.13) below and Corollary 4.1 in [16]) which generalizes
those of L. Huang [2][3] in homogeneous Finsler geometry, and prove a
correspondence between geodesics of $\mathbf{G}$ and integral curves of
$-\eta$ (see Theorem 3.2 below or Theorem D in [16]).

As a continuation of this exploration, we switch in this paper to linearly and
nonlinearly parallel translations for $\mathbf{G}$. It turns out that the
connection operator
$N:(\mathfrak{g}\backslash\\{0\\})\times\mathfrak{g}\rightarrow\mathfrak{g}$
plays an important role. This notion was first defined for a homogeneous
Finsler space by L. Huang, using fundamental tensor and Cartan tensor [2][3].
He later pointed out another description (see (4) in [4]), which implies that
it can be generalized to homogeneous spray geometry. In particular, for a left
invariant spray structure $\mathbf{G}$ with the spray vector field $\eta$, the
connection operator is

$N(y,w)=\tfrac{1}{2}D\eta(y,w)-\tfrac{1}{2}[y,w]_{\mathfrak{g}},$ (1.1)

in which $D\eta(y,w)$ is the derivative of
$\eta:\mathfrak{g}\backslash\\{0\\}\rightarrow\mathfrak{g}$ at $y$ in the
direction of $w$. Here $[\cdot,\cdot]_{\mathfrak{g}}$ is the Lie bracket of
$\mathfrak{g}=\mathrm{Lie}(G)$. The more usual notation $[\cdot,\cdot]$ is
reserved for the canonical bracket between two smooth vector fields (see
Theorem 1.2 and Remark 1.3 below).

Firstly, we consider linearly parallel translations along a smooth curve on
$(G,\mathbf{G})$ and prove

######  Theorem 1.1.

Let $G$ be a connected Lie group endowed with a left invariant spray structure
$\mathbf{G}$ with the spray vector field $\eta$, $c(t)$ a smooth curve on
$(G,\mathbf{G})$ with nowhere-vanishing $\dot{c}(t)$, and $W(t)$ a vector
field along $c(t)$. Denote
$y(t)=(L_{c(t)^{-1}})_{*}(\dot{c}(t))\in\mathfrak{g}\backslash\\{0\\}$ and
$w(t)=(L_{c(t)^{-1}})_{*}(W(t))\in\mathfrak{g}$. Then $W(t)$ is linearly
parallel along $c(t)$ if and only if $w(t)$ is a solution of

$\tfrac{{\rm d}}{{\rm d}t}w(t)+N(y(t),w(t))+[y(t),w(t)]_{\mathfrak{g}}=0.$
(1.2)

As an application of Theorem 1.1, the connection operator and Riemann
curvature operators for a left invariant $\mathbf{G}$ can be alternatively
described by

######  Theorem 1.2.

Let $c(t)$ be a geodesic on the connected Lie group $G$, for the left
invariant spray structure $\mathbf{G}$, and
$y(t)=(L_{c(t)^{-1}})_{*}(\dot{c}(t))$ the corresponding integral curve for
$-\eta$, where $\eta$ is the spray vector field associated with $\mathbf{G}$.
Denote $w(t)$ the vector field along $y(t)$ which corresponds to a linearly
parallel vector field $W(t)$ along $c(t)$, i.e.,
$w(t)=(L_{c(t)^{-1}})_{*}(W(t))$. Then $N(t)=N(y(t),w(t))$ and
$R(t)=R_{y(t)}(w(t))$, where $N(\cdot,\cdot)$ is the connection operator and
$R_{\cdot}\cdot$ is the Riemann curvature, are vector fields along $y(t)$
determined by

$N(t)=-[\eta,w(t)]\quad\mbox{and}\quad R(t)=[\eta,N(t)]=-[\eta,[\eta,w(t)]].$
(1.3)

######  Remark 1.3.

The vector fields $w(t)$ and $N(t)$ are smooth vector field along an integral
of $-\eta$. So $[\eta,w(t)]=-[-\eta,w(t)]$ and $[\eta,N(t)]=-[-\eta,N(t)]$ in
(1.3) are in deed Lie derivatives. Notice that they can be calculated as the
canonical bracket between smooth vector field on a manifold, as they are
presented in Theorem 1.2. To be precise, let $X$ be a smooth vector field on
$M$ and $Y(t)$ a smooth vector field along an integral curve $c(t)$ for $X$,
then $[X,Y(t)]$ is a well defined smooth vector field along $c(t)$. When
$c(t)$ is not constant, we may locally extend $Y(t)$ to a smooth vector field
$Z$ on $M$, then $[X,Y(t)]=[X,Z]|_{c(t)}$ is independent of the extension.
Using local coordinate, the bracket between $X=X^{i}\partial_{x^{i}}$ and
$Y(t)=Y^{i}(t)\partial_{x^{i}}|_{c(t)}$ can be presented as

$[X,Y(t)]=(\tfrac{{\rm d}Y^{i}(t)}{{\rm
d}t}\partial_{x^{i}}-Y^{i}(t)\tfrac{\partial}{\partial
x^{i}}X^{j}\partial_{x^{j}})|_{c(t)}.$ (1.4)

Notice that when $c(t)$ is constant, (1.4) can still be used to calculate
$[X,Y(t)]$, which is independent of the choice of local coordinate.

Theorem 1.1 (together with Theorem D in [16]) can provide shortcuts to other
curvature formulae of $\mathbf{G}$ as well. See Section 3.3 for some examples.

Nextly, we consider nonlinearly parallel translations along a smooth curve on
$(G,\mathbf{G})$ and prove

######  Theorem 1.4.

Let $G$ be a connected Lie group endowed with a left invariant spray structure
$\mathbf{G}$, $c(t)$ a smooth curve on $G$ and $Y(t)$ a nowhere vanishing
vector field along $c(t)$. Denote $w(t)=(L_{c(t)^{-1}})_{*}(\dot{c}(t))$ and
$y(t)=(L_{c(t)^{-1}})_{*}(Y(t))$. Then $Y(t)$ is nonlinearly parallel along
$c(t)$ iff $y(t)$ is a solution of

$\tfrac{{\rm d}}{{\rm d}t}y(t)+N(y(t),w(t))=0.$ (1.5)

When $w(t)\equiv w$ is constant, i.e., $c(t)=\exp tw$ is a one-parameter
subgroup of $G$, the equation (1.5) generates a one-parameter subgroup of
diffeomorphisms on $\mathfrak{g}\backslash\\{0\\}$. So Theorem 1.4 has the
following immediate consequence.

######  Theorem 1.5.

Let $G$ be a connected Lie group endowed with a left invariant spray structure
$\mathbf{G}$, and $c(t)=\exp tw$ for any $w\in\mathfrak{g}$ a one-parameter
subgroup of $G$. Denote $\mathbf{P}^{\mathrm{nl}}_{c(0),c(t);c}$ the nonlinear
parallel translation along $c(\cdot)$ from $c(0)$ to $c(t)$. Then
$\rho_{t}=(L_{c(t)^{-1}})_{*}\circ\mathbf{P}^{\mathrm{nl}}_{c(0),c(t);c}$ is
the one-parameter subgroup of diffeomorphisms generated by the smooth vector
field $-N(\cdot,w)$ on $\mathfrak{g}\backslash\\{0\\}$.

Denote $\mathfrak{H}$ the Lie algebra generated by the space
$\mathcal{N}=\\{N(\cdot,v),\forall v\in\mathfrak{g}\\}$, using the canonical
bracket between two smooth vector fields on $\mathfrak{g}\backslash\\{0\\}$,
for a left invariant spray structure $\mathbf{G}$ on the Lie group $G$. The
importance of $\mathfrak{H}$ in homogeneous spray and Finsler geometries is
implied by Theorem 1.5, i.e., $\mathfrak{H}$ contains all the information for
the nonlinear parallel translations on $(G,\mathbf{G})$. We propose two
questions (see Question 4.3 and Question 4.4 in Section 4.2). The first
question concerns if we always have $\dim\mathfrak{H}=+\infty$ when
$\mathbf{G}$ is not affine. This question can be viewed as a generalization of
Landsberg Problem [12] in Finsler geometry. The second question concerns the
relation between $\mathfrak{H}$ and the restricted holonomy group
$\mathrm{Hol}_{0}(G,\mathbf{G})$. Notice that when $\mathbf{G}$ is induced by
a left invariant Riemannian metric, we can often get
$\mathfrak{H}=\mathrm{Lie}(\mathrm{Hol}_{0}(G,\mathbf{G}))$ [8][9]. However,
in Finsler or spray geometry, both $\mathfrak{H}$ and
$\mathrm{Hol}_{0}(G,\mathbf{G})$ might have infinite dimensions [6], and then
the second question becomes much harder.

This paper is organized as following. In Section 2, we summarize some
necessary notions and techniques. In Section 3, we prove Theorem 1.1 and
Theorem 1.2. In Section 4, we prove Theorem 1.4 and propose two questions.

##  2\. Preliminaries

###  2.1. Spray structure and parallel translation

In this subsection, we summarize some fundamental knowledge in [13] on spray
geometry.

A spray structure on a smooth manifold $M$ is a smooth tangent vector field
$\mathbf{G}$ on the slit tangent bundle $TM\backslash 0$, which can be locally
presented as

$\mathbf{G}=y^{i}\partial_{x^{i}}-2\mathbf{G}^{i}\partial_{y^{i}}$ (2.6)

for any standard local coordinate $(x^{i},y^{i})$, i.e., $x=(x^{i})\in M$ and
$y=y^{i}\partial_{x^{i}}\in T_{x}M$, such that each
$\mathbf{G}^{i}=\mathbf{G}^{i}(x,y)$ is positive 2-homogeneous for the
$y$-entry. A smooth curve $c(t)$ in $M$ with nonvanishing $\dot{c}(t)$
everywhere is called a geodesic for $\mathbf{G}$, if its lifting
$(c(t),\dot{c}(t))$ in $TM\backslash 0$ is an integral curve of $\mathbf{G}$.

Following (2.6), we denote

$N^{i}_{j}=\tfrac{\partial\mathbf{G}^{i}}{\partial{y^{j}}}\quad\mbox{and}\quad\delta_{x^{i}}=\partial_{x^{i}}-N^{i}_{j}\partial_{y^{j}}.$

Let $W(t)=z^{i}(t)\partial_{x^{i}}|_{c(t)}$ be a smooth vector field along the
smooth curve $c(t)$ and we assume $\dot{c}(t)$ is nonvanishing everywhere. The
linearly covariant derivative $D_{\dot{c}(t)}W(t)$ is the following smooth
vector field along $c(t)$,

$D_{\dot{c}(t)}W(t)=(\tfrac{{\rm d}z^{i}(t)}{{\rm
d}t}+z^{j}(t)N^{i}_{j}(c(t),\dot{c}(t)))\partial_{x^{i}}|_{c(t)}.$ (2.7)

We say $W(t)$ is linearly parallel along $c(t)$ if $D_{\dot{c}(t)}W(t)\equiv
0$. For any initial value $w\in T_{c(t_{0})}M$, there exists a unique linearly
parallel $W(t)$ globally along $c(t)$. This fact is implies by the existence
and uniqueness theorem for ordinary differential equation with initial value,
together with the smoothness and positive 1-homogeneity of $N^{i}_{j}$’s.

Suppose $c(t)$ with $t\in[a,b]$, $c(a)=p$ and $c(b)=q$ is a smooth curve on
$M$ with nonvanishing $\dot{c}(t)$ everywhere. Then the linear parallel
translation $\mathbf{P}^{\mathrm{l}}_{p,q;c}:T_{p}M\rightarrow T_{q}M$ is
defined by $\mathbf{P}^{\mathrm{l}}_{p,q;c}(w)=W(b)$, where $W(t)$ is the
linearly parallel vector field along $c(t)$ satisfying $W(a)=w$.

Using the nonlinearly covariant derivative

$\widetilde{D}_{\dot{c}(t)}W(t)=(\tfrac{{\rm d}z^{i}(t)}{{\rm
d}t}+\dot{c}^{j}(t)N^{i}_{j}(c(t),W(t)))\partial_{x^{i}}|_{c(t)}$

for any nowhere vanishing $W(t)=z^{i}(t)\partial_{x^{i}}|_{c(t)}$ along
$c(t)$, the nonlinearly parallel translation
$\mathbf{P}^{\mathrm{nl}}_{p,q;c}:T_{p}M\backslash\\{0\\}\rightarrow
T_{q}M\backslash\\{0\\}$ from $p=c(a)$ to $q=c(b)$ along a smooth curve $c(t)$
can be similarly defined. It can be alternatively described by the integral
curves for the horizonal lifting of $\dot{c}(t)$, i.e.,

$\widetilde{\dot{c}(t)}^{\mathcal{H}}=\dot{c}^{i}(t)\delta_{x^{i}}|_{T_{c(t)}M\backslash\\{0\\}}=\dot{c}^{i}(t)(\partial_{x^{i}}-N^{i}_{j}\partial_{y^{j}})|_{T_{c(t)}M\backslash\\{0\\}}.$

To be precise, $\widetilde{\dot{c}(t)}^{\mathcal{H}}$ is viewed as a smooth
tangent vector field on the submanifold
$S=\cup_{t}(T_{c(t)}M\backslash\\{0\\})$. For any nowhere vanishing smooth
vector field $Y(t)$ along $c(t)$, it is nonlinearly parallel, i.e.,
$Y(t)=\mathbf{P}^{\mathrm{nl}}_{c(t_{0}),c(t);c}(Y(t_{0}))$ for all values of
$t$, iff $(c(t),Y(t))$ is an integral curve of
$\widetilde{\dot{c}(t)}^{\mathcal{H}}$ in $S$.

###  2.2. Invariant frames on a Lie group

In this section, we summarize some notations in [16] for invariant frames.

Let $G$ be a connected Lie group, $L_{g}(g^{\prime})=gg^{\prime}$ and
$R_{g}(g^{\prime})=g^{\prime}g$ the left and right translations respectively.
Denote $\mathfrak{g}=\mathrm{Lie}(G)=T_{e}G$ and
$[\cdot,\cdot]_{\mathfrak{g}}$ its Lie bracket. We fix a basis
$\\{e_{1},\cdots,e_{n}\\}$ of $\mathfrak{g}$ with
$[e_{i},e_{j}]_{\mathfrak{g}}=c^{k}_{ij}e_{k}$.

For each $1\leq i\leq n$, we have a left invariant tangent vector field
$U_{i}(g)=(L_{g})_{*}(e_{i})$, so $\\{U_{i},\forall i\\}$ is a left invariant
frame on $G$ satisfying $[U_{i},U_{j}]=c^{k}_{ij}U_{k}$. The complete lifting
of $U_{i}$ is denoted as $\widetilde{U}_{i}$, which is a smooth tangent vector
field on $TG$ (see Section 2.1 in [16]). Any tangent vector $y\in T_{g}G$ can
be uniquely presented as $y=u^{i}U_{i}(g)$, which determines the smooth
functions $u^{i}$’s on $T_{g}G$. Denote $\partial_{u^{i}}$’s the smooth vector
fields on $TG$ which are tangent to and correspond to the $u^{i}$-coordinates
in each $T_{g}G$. It is easy to observe the left invariancy of $U_{i}$’s and
$u^{i}$’s, so we call $\\{\widetilde{U}_{i},\partial_{u^{i}},\forall i\\}$ a
left invariant frame on $TG$.

The transformation between $\\{\widetilde{U}_{i},\partial_{u^{i}},\forall
i\\}$ and $\\{\partial_{x^{i}},\partial_{y^{i}},\forall i\\}$ for a standard
local coordinate $(x^{i},y^{i})$ is the following (see (2.2) and Lemma 2.1 in
[16]),

$\displaystyle U_{i}=A^{j}_{i}\partial_{x^{i}},\quad
u^{i}=y^{j}B^{i}_{j}\quad\mbox{and}\quad\partial_{u^{i}}=A^{j}_{i}\partial_{y^{i}},$
(2.8)
$\displaystyle\widetilde{U}_{i}=A^{j}_{i}\partial_{x^{i}}+y^{j}\tfrac{\partial}{\partial
x^{j}}A^{k}_{i}\partial_{y^{i}},$ (2.9)

where $(A^{j}_{i})=(A^{j}_{i}(x))$ and
$(B^{j}_{i})=(B^{j}_{i}(x))=(A^{j}_{i}(x))^{-1}$ (i.e.,
$A^{j}_{i}B^{k}_{j}=B^{j}_{i}A^{k}_{j}=\delta^{k}_{i}$) are matrix valued
functions which only depend on the $x$-entry. Notice that $\partial_{x^{i}}$’s
in (2.8) and (2.9) are local tangent vector fields on $G$, and their complete
liftings to $TG$ respectively. Comparing the coefficients of
$\partial_{x^{k}}$ in both sides of

$c^{p}_{qi}A^{k}_{p}\partial_{x^{k}}=c^{p}_{qi}U_{p}=[U_{q},U_{i}]=(A^{j}_{q}\tfrac{\partial}{\partial
x^{j}}A^{k}_{i}-A^{j}_{i}\tfrac{\partial}{\partial
x^{j}}A^{k}_{q})\partial_{x^{k}},$

we get (see (3.14) in [16])

######  Lemma 2.1.

$A^{j}_{q}\tfrac{\partial}{\partial
x^{j}}A^{k}_{i}-A^{j}_{i}\tfrac{\partial}{\partial
x^{j}}A^{k}_{q}=c^{p}_{qi}A^{k}_{p}$.

Using the right invariant tangent vector fields $V_{i}(g)=(R_{g})_{*}(e_{i})$
on $G$ and the presentation $y=v^{i}V_{i}(g)\in T_{g}G$, we can similarly get
a right invariant frame $\\{\widetilde{V}_{i},\partial_{v^{i}},\forall i\\}$
on $TG$.

To describe the transformation between
$\\{\widetilde{U}_{i},\partial_{u^{i}},\forall i\\}$ and
$\\{\widetilde{V}_{i},\partial_{v^{i}},\forall i\\}$, we denote $\phi_{i}^{j}$
and $\psi_{i}^{j}$ the functions on $G$ such that
$\mathrm{Ad}(g)e_{i}=\phi_{i}^{j}e_{j}$ and
$\mathrm{Ad}(g^{-1})e_{i}=\psi_{i}^{j}e_{j}$ (so we have
$\psi_{i}^{j}\phi_{j}^{k}=\phi_{i}^{j}\psi_{j}^{k}=\delta_{i}^{k}$). Then at
each $g\in G$,

$U_{i}(g)=(L_{g})_{*}(e_{i})=(R_{g})_{*}(R_{g^{-1}})_{*}(L_{g})_{*}(e_{i})=(R_{g})_{*}(\mathrm{Ad}(g)e_{i})=(R_{g})_{*}(\phi^{j}_{i}(g)e_{j})=\phi^{j}_{i}(g)V_{j}(g),$

so we have

$U_{i}=\phi_{i}^{j}V_{j},\quad
u^{i}=\psi_{j}^{i}v^{j},\quad\partial_{u^{i}}=\phi_{i}^{j}\partial_{v^{j}}.$
(2.10)

In [16], we have proved

######  Lemma 2.2.

(1) $\phi^{j}_{l}V_{j}\phi_{i}^{k}=c^{j}_{li}\phi^{k}_{j}$, (2)
$\widetilde{U}_{i}=\phi^{j}_{i}\widetilde{V}_{j}+c^{q}_{pi}u^{p}\partial_{u^{q}}$.

We briefly recall its proof here. To prove (1), we observe

$\displaystyle V_{j}\phi_{i}^{k}e_{k}$ $\displaystyle=$ $\displaystyle
V_{j}(\phi_{i}^{k}e_{k})=\tfrac{{\rm d}}{{\rm d}t}(\mathrm{Ad}(\exp
te_{j}\cdot g)e_{i})$ $\displaystyle=$ $\displaystyle\tfrac{{\rm d}}{{\rm
d}t}\mathrm{Ad}(\exp
te_{j})(\mathrm{Ad}(g)e_{i})=[e_{j},\mathrm{Ad}(g)e_{i}]_{\mathfrak{g}},$

and

$\phi_{l}^{j}V_{j}\phi_{i}^{k}e_{k}=[\mathrm{Ad}(g)e_{l},\mathrm{Ad}(g)e_{i}]_{\mathfrak{g}}=\mathrm{Ad}(g)[e_{l},e_{i}]_{\mathfrak{g}}=c^{j}_{li}\mathrm{Ad}(g)e_{j}=c^{j}_{li}\phi^{k}_{j}e_{k}.$
(2.11)

Then (1) follows after a comparison for the coefficients of $e_{k}$ in (2.11).
To prove (2), we apply Lemma 2.2 in [16] to the first equality in (2.10) and
get

$\widetilde{U}_{i}=\phi_{i}^{j}\widetilde{V}_{j}+v^{j}V_{j}\phi_{i}^{k}\partial_{v^{k}}=\phi_{i}^{j}\widetilde{V}_{j}+u^{p}\phi_{p}^{j}V_{j}\phi_{i}^{k}\partial_{v^{k}}.$

Then (2) follows after (1) and the third equality in (2.10) immediately.

###  2.3. Left invariant spray structure on a Lie group

In this subsection, we introduce the notions of invariant spray structure in
[16].

Let $G$ be a connected Lie group and $\mathbf{G}$ a spray structure on $G$. We
call $\mathbf{G}$ left invariant (or right invariant) if all left (or right
respectively) translations preserve $\mathbf{G}$. We call $\mathbf{G}$ bi-
invariant if it is both left and right invariant. Using the invariant frames
$\\{\widetilde{U}_{i},\partial_{u^{i}},\forall i\\}$ and
$\\{\widetilde{V}_{i},\partial_{v^{i}},\forall i\\}$ on $TG$, the left and
right invariancies of $\mathbf{G}$ can be equivalently described as

$[\widetilde{V}_{i},\mathbf{G}]=0,\forall
i\quad\mbox{and}\quad[\widetilde{U}_{i},\mathbf{G}]=0,\forall
i\quad\mbox{respectively}.$

The canonical bi-invariant spray structure
$\mathbf{G}_{0}=u^{i}\widetilde{U}_{i}=v^{i}\widetilde{V}_{i}$ (see Theorem A
in [16]) serves as the origin in the space of left invariant spray structures
on $G$. Any left invariant spray structure $\mathbf{G}$ on $G$ can be
presented as
$\mathbf{G}=\mathbf{G}_{0}-\mathbf{H}=\mathbf{G}_{0}-\mathbf{H}^{i}\partial_{u^{i}}$.
Here $\mathbf{H}=\mathbf{H}^{i}\partial_{u^{i}}$ is a left invariant smooth
vector field on $TG\backslash 0$, and $\mathbf{H}^{i}$’s are left invariant
smooth functions on $TG\backslash 0$ which are positive 2-homogeneous in each
$T_{g}G$. We denote $\eta=\mathbf{H}|_{T_{e}G\backslash\\{0\\}}$ and call it
the spray vector field associated with $\mathbf{G}$. Usually $\eta$ is
presented as a smooth map from $\mathfrak{g}\backslash\\{0\\}$ to
$\mathfrak{g}$, i.e., $\eta(y)=\mathbf{H}^{i}(e,y)e_{i}$. The connection
operator
$N(\cdot,\cdot):(\mathfrak{g}\backslash\\{0\\})\times\mathfrak{g}\rightarrow\mathfrak{g}$
is then defined by
$N(y,w)=\tfrac{1}{2}D\eta(y,w)-\tfrac{1}{2}[y,w]_{\mathfrak{g}}$, in which
$D\eta(y,w)$ is the derivative of the spray vector field
$\eta:\mathfrak{g}\backslash\\{0\\}\rightarrow\mathfrak{g}$ at $y$ in the
direction of $w$.

If we present the left invariant spray structure
$\mathbf{G}=\mathbf{G}_{0}-\mathbf{H}^{i}\partial_{u^{i}}$ as
$\mathbf{G}=y^{i}\partial_{x^{i}}-2\mathbf{G}^{i}\partial_{y^{i}}$ for any
standard local coordinate $(x^{i},y^{i})$. Using (2.8) and (2.9), it is easy
to check (see (3.10) in [16])

######  Lemma 2.3.

$N^{k}_{j}=\tfrac{\partial\mathbf{G}^{k}}{\partial
y^{j}}=\tfrac{1}{2}A^{k}_{i}\tfrac{\partial}{\partial
y^{j}}\mathbf{H}^{i}-\tfrac{1}{2}u^{i}\tfrac{\partial}{\partial
x^{j}}A^{k}_{i}-\tfrac{1}{2}y^{l}B^{i}_{j}\tfrac{\partial}{\partial
x^{l}}A^{k}_{i}$.

##  3\. Linear parallel translation along a geodesic

###  3.1. Proof of Theorem 1.1

Suppose the connected Lie group $G$ is endowed with a left invariant spray
structure
$\mathbf{G}=\mathbf{G}_{0}-\mathbf{H}=u^{i}\widetilde{U}_{i}-\mathbf{H}^{i}\partial_{u^{i}}$
with the spray vector field
$\eta:\mathfrak{g}\backslash\\{0\\}\rightarrow\mathfrak{g}$. Now we consider
the linear parallel translation along a smooth curve $c(t)$ on
$(G,\mathbf{G})$ with nowhere-vanishing $\dot{c}(t)$. We denote
$y(t)=(L_{c(t)^{-1}})_{*}(\dot{c}(t))$, which is a smooth curve on
$\mathfrak{g}\backslash\\{0\\}$. Using left translations, smooth vector fields
along $c(t)$ and those along $y(t)$ can be one-to-one corresponded. Consider
any smooth vector field $W(t)=w^{i}(t)U_{i}(c(t))$ along $c(t)$, then the
corresponding $w(t)=(L_{c(t)^{-1}})_{*}(W(t))$ along $y(t)$ can be presented
as $w(t)=w^{i}(t)e_{i}$.

######  Lemma 3.1.

$D_{\dot{c}(t)}W(t)=(\tfrac{{\rm d}w^{l}(t)}{{\rm
d}t}+\tfrac{1}{2}w^{j}(t)\tfrac{\partial}{\partial
u^{j}}\mathbf{H}^{l}+\tfrac{1}{2}w^{j}(t)u^{k}(t)c_{kj}^{l})U_{l}(c(t))$.

Proof. In any standard local coordinate $(x^{i},y^{i})$, we have the
presentations $c(t)=(c^{i}(t))$,
$y^{i}(t)=y^{i}(c(t),\dot{c}(t))=\dot{c}^{i}(t)$ and
$W(t)=w^{i}(t)U_{i}(c(t))=z^{i}(t)\partial_{x^{i}}|_{c(t)}$. Using the
notations in (2.8), i.e., $U_{i}=A^{j}_{i}\partial_{x^{i}}$,
$u^{i}=y^{j}B^{i}_{j}$ and $(B^{i}_{j}(x))=(A^{i}_{j}(x))^{-1}$, we also have
$u^{i}(t)=y^{j}(t)B^{i}_{j}(c(t))$ and $z^{i}(t)=w^{j}(t)A^{i}_{j}(c(t))$. So
at each point $x=c(t)$, the covariant derivative $D_{\dot{c}(t)}W(t)$ (see
(2.7)) can be calculated as following,

$\displaystyle D_{\dot{c}(t)}W(t)$ $\displaystyle=$ $\displaystyle(\tfrac{{\rm
d}z^{i}(t)}{{\rm d}t}+z^{j}(t)N^{i}_{j}(c(t),\dot{c}(t)))\partial_{x^{i}}$
(3.12) $\displaystyle=$ $\displaystyle\tfrac{{\rm d}}{{\rm
d}t}(w^{j}(t)A_{j}^{i})B_{i}^{k}U_{k}+z^{j}(t)(\tfrac{1}{2}A^{i}_{l}\tfrac{\partial}{\partial
y^{j}}\mathbf{H}^{l}-\tfrac{1}{2}u^{l}(t)\tfrac{\partial}{\partial
x^{j}}A^{i}_{l}-\tfrac{1}{2}y^{p}(t)B^{l}_{j}\tfrac{\partial}{\partial
x^{p}}A^{i}_{l})\partial_{x^{i}}$ $\displaystyle=$ $\displaystyle(\tfrac{{\rm
d}w^{k}(t)}{{\rm d}t}U_{k}+z^{p}(t)y^{l}(t)B_{p}^{j}\tfrac{\partial}{\partial
x^{l}}A^{i}_{j}\partial_{x^{i}})$
$\displaystyle+\tfrac{1}{2}w^{j}(t)\tfrac{\partial}{\partial
u^{j}}\mathbf{H}^{l}U_{l}-(\tfrac{1}{2}z^{j}(t)y^{p}(t)B^{l}_{p}\tfrac{\partial}{\partial
x^{j}}A^{i}_{l}+\tfrac{1}{2}z^{j}(t)y^{p}(t)B^{l}_{j}\tfrac{\partial}{\partial
x^{p}}A^{i}_{l})\partial_{x^{i}}$ $\displaystyle=$ $\displaystyle(\tfrac{{\rm
d}w^{k}(t)}{{\rm d}t}+\tfrac{1}{2}w^{j}(t)\tfrac{\partial}{\partial
u^{j}}\mathbf{H}^{k})U_{k}+\tfrac{1}{2}(z^{p}(t)y^{l}(t)B^{j}_{p}\tfrac{\partial}{\partial
x^{l}}A^{i}_{j}-z^{j}(t)y^{p}(t)B^{l}_{p}\tfrac{\partial}{\partial
x^{l}}A^{i}_{k})\partial_{x^{i}}$ $\displaystyle=$ $\displaystyle(\tfrac{{\rm
d}w^{k}(t)}{{\rm d}t}+\tfrac{1}{2}w^{j}(t)\tfrac{\partial}{\partial
u^{j}}\mathbf{H}^{k})U_{k}+\tfrac{1}{2}(w^{j}(t)u^{k}(t)A^{l}_{k}\tfrac{\partial}{\partial
x^{l}}A^{i}_{j}-w^{j}(t)u^{k}(t)A_{j}^{l}\tfrac{\partial}{\partial
x^{l}}A^{i}_{k})\partial_{x^{i}}$ $\displaystyle=$ $\displaystyle(\tfrac{{\rm
d}w^{l}(t)}{{\rm d}t}+\tfrac{1}{2}w^{j}(t)\tfrac{\partial}{\partial
u^{j}}\mathbf{H}^{l}+\tfrac{1}{2}w^{j}(t)u^{k}(t)c_{kj}^{l})U_{l}(c(t)),$

in which the second line uses Lemma 2.3 and the last line uses Lemma 2.1.

Now we are ready to prove Theorem 1.1, which interprets Lemma 3.1 by left
translations.

Proof of Theorem 1.1. Lemma 3.1 indicates

$\displaystyle(L_{c(t)^{-1}})_{*}(D_{\dot{c}(t)}W(t))$ $\displaystyle=$
$\displaystyle(\tfrac{{\rm d}w^{l}(t)}{{\rm
d}t}+\tfrac{1}{2}w^{j}(t)\tfrac{\partial}{\partial
u^{j}}\mathbf{H}^{l}+\tfrac{1}{2}w^{j}(t)u^{k}(t)c_{kj}^{l})e_{l}$
$\displaystyle=$ $\displaystyle(\tfrac{{\rm d}w^{l}(t)}{{\rm
d}t}+(\tfrac{1}{2}w^{j}(t)\tfrac{\partial}{\partial
u^{j}}\mathbf{H}^{l}-\tfrac{1}{2}w^{j}(t)u^{k}(t))c_{kj}^{l}+w^{j}(t)u^{k}(t)c_{kj}^{l})e_{l}$
$\displaystyle=$ $\displaystyle\tfrac{{\rm d}}{{\rm
d}t}{w}(t)+N(y(t),w(t))+[y(t),w(t)]_{\mathfrak{g}},$

where $y(t)=u^{i}(t)e_{i}$ and $w(t)=w^{i}(t)e_{i}$. So
$D_{\dot{c}(t)}W(t)\equiv 0$ if and only if
$(L_{c(t)^{-1}})_{*}(D_{\dot{c}(t)}W(t))\equiv 0$, i.e., $w(t)$ is a solution
of $\tfrac{{\rm d}}{{\rm
d}t}{w}(t)+N(y(t),w(t))+[y(t),w(t)]_{\mathfrak{g}}=0$.

###  3.2. Proof of Theorem 1.2

The Riemann curvature formula for a left invariant spray structure
$\mathbf{G}$ is given by Theorem C in [16]. In particular, when restricted to
$T_{e}G$, the Riemann curvature operator
$R_{\cdot}(\cdot):(\mathfrak{g}\backslash\\{0\\})\times\mathfrak{g}\rightarrow\mathfrak{g}$
satisfies (see Corollary 4.1 in [16]),

$R_{y}(w)=DN(\eta,y,w)-N(y,N(y,w))+N(y,[y,w]_{\mathfrak{g}})-[y,N(y,w)]_{\mathfrak{g}},$
(3.13)

where $DN(\eta,y,w)=\tfrac{{\rm d}}{{\rm d}t}|_{t=0}N(y+t\eta,w)$, i.e., it is
the derivative of $N(\cdot,w)$ at $y$ in the direction of $\eta(y)$. See
Proposition 3.2 in [3] for (3.13) in homogeneous Finsler geometry, and Lemma
5.1 in [5] for another homogeneous Riemannian curvature formula when
$\eta(y)=2P(y)y$ for some smooth positive 1-homogeneous function $P(\cdot)$ on
$\mathfrak{g}\backslash\\{0\\}$.

Let $c(t)$ be a geodesic on $(G,\mathbf{G})$ and denote
$y(t)=(L_{c(t)^{-1}})_{*}(\dot{c}(t))$. The following theorem indicates that
$y(t)$ is an integral curve of $-\eta$ (see Theorem D in [16]).

######  Theorem 3.2.

Let $\mathbf{G}$ be a left invariant spray structure on the Lie group $G$ with
the associated spray vector field $\eta$. Then for any open interval
$(a,b)\subset\mathbb{R}$ containing $0$, there is a one-to-one correspondence
between the following two sets:

  1. (1)

the set of all curves $c(t)$ on $G$, with $t\in(a,b)$ and $c(0)=e$, which are
geodesics for $\mathbf{G}$;

  2. (2)

the set of all $y(t)$ on $\mathfrak{g}\backslash\\{0\\}$, with $t\in(a,b)$,
which are integral curves of $-\eta$.

The correspondence between these two sets is given by
$y(t)=(L_{c(t)^{-1}})_{*}(\dot{c}(t))$.

Let $W(t)$ be a linearly parallel vector field along $c(t)$ and denote
$w(t)=(L_{c(t)^{-1}})_{*}(\dot{c}(t))$, which is viewed as a smooth vector
field along the curve $y(t)$ on $\mathfrak{g}\backslash\\{0\\}$. By Theorem
1.1, we have $\tfrac{{\rm d}}{{\rm
d}t}w(t)+N(y(t),w(t))+[y(t),w(t)]_{\mathfrak{g}}=0$. Now we prove Theorem 1.2,
i.e., an alternative interpretation of (3.13).

Proof of Theorem 1.2. By (1.4) in Remark 1.3, we have

$\displaystyle[-\eta,w(t)]=\tfrac{{\rm d}}{{\rm
d}t}w(t)+D\eta(y(t),w(t))=\tfrac{{\rm d}}{{\rm
d}t}w(t)+2N(y(t),w(t))+[y(t),w(t)]_{\mathfrak{g}},$

in which $D\eta(y(t),w(t))=\tfrac{{\rm d}}{{\rm
d}s}|_{s=0}\eta(y(t)+sw(t))=2N(y(t),w(t))+[y(t),w(t)]_{\mathfrak{g}}$. By
Theorem 1.1, $\tfrac{{\rm d}}{{\rm d}t}w(t)+N(y(t),w(t))+[y(t),w(t)]=0$, so
$N(t)=N(y(t),w(t))=[-\eta,w(t)]=-[\eta,w(t)]$.

Using Theorem 1.1, Theorem 3.2 and the linearity of $N(y,w)$ for the
$w$-entry, we have

$\displaystyle\tfrac{{\rm d}}{{\rm d}t}N(y(t),w(t))$ $\displaystyle=$
$\displaystyle DN(-\eta,y(t),w(t))+N(y(t),\tfrac{{\rm d}}{{\rm d}t}w(t))$
(3.14) $\displaystyle=$ $\displaystyle-
DN(\eta,y(t),w(t))+N(y(t),-N(y(t),w(t))-[y(t),w(t)]_{\mathfrak{g}})$
$\displaystyle=$ $\displaystyle-DN(\eta,y(t),w(t))-N(y(t),N(y(t),w(t)))$
$\displaystyle-N(y(t),[y(t),w(t)]_{\mathfrak{g}}).$

Here $DN(\eta,y(t),w(t))$ is the derivative of $N(\cdot,w(t))$ at $y(t)$ in
the direction of $w(t)$, for each fixed value of $t$. So (1.4) and (3.14)
imply

$\displaystyle[\eta,N(t)]$ $\displaystyle=$
$\displaystyle-[-\eta,N(t)]=-\tfrac{{\rm d}}{{\rm
d}t}N(y(t),w(t))-D\eta(y(t),N(y(t),w(t)))$ $\displaystyle=$ $\displaystyle
DN(\eta,y(t),w(t))+N(y(t),N(y(t),w(t)))+N(y(t),[y(t),w(t)]_{\mathfrak{g}})$
$\displaystyle-(2N(y(t),N(y(t),w(t)))+[y(t),N(y(t),w(t))]_{\mathfrak{g}})$
$\displaystyle=$ $\displaystyle
DN(\eta,y(t),w(t))-N(y(t),N(y(t),w(t)))+N(y(t),[y(t),w(t)]_{\mathfrak{g}})$
$\displaystyle-[y(t),N(y(t),w(t))]_{\mathfrak{g}}$ $\displaystyle=$
$\displaystyle R_{y(t)}(w(t))=R(t).$

This ends the proof of Theorem 1.2.

###  3.3. Landsberg curvature and S-curvature for a left invariant Finsler
metric

Theorem 1.1 and Theorem 3.2 can be used to prove other curvature formulas for
$(G,\mathbf{G})$. Here we take the Landsberg and S-curvature for a left
invariant Finsler metric for example.

The Landsberg curvature $L$ for a Finsler metric $F$ can be calculated by

$L_{\dot{c}(t)}(W(t),W(t),W(t))=\tfrac{{\rm d}}{{\rm
d}t}C_{\dot{c}(t)}(W(t),W(t),W(t)),$ (3.15)

in which $c(t)$ is a geodesic, $W(t)$ is linearly parallel along $c(t)$, and
$C_{\cdot}(\cdot,\cdot,\cdot)$ is the Cartan tensor (see (7.16) in [14]). When
$F$ is left invariant, Theorem 1.1 and Theorem 3.2 translate (3.15) to

$\displaystyle L_{y(t)}(w(t),w(t),w(t))$ $\displaystyle=$
$\displaystyle\tfrac{{\rm d}}{{\rm d}t}C_{y(t)}(w(t),w(t),w(t))$
$\displaystyle=$ $\displaystyle(\tfrac{{\rm d}}{{\rm
d}t}C_{y(t)})(w(t),w(t),w(t),\eta(y(t)))+3C_{y(t)}(\tfrac{{\rm d}}{{\rm
d}t}w(t),w(t),w(t))$ $\displaystyle=$ $\displaystyle-
C_{y(t)}(w(t),w(t),w(t),\eta(y(t)))$
$\displaystyle-3C_{y(t)}(N(y(t),w(t))+[y(t),w(t)],w(t),w(t)),$

where $y(t)=(L_{c(t)^{-1}})_{*}(\dot{c}(t))$ is an integral curve of $-\eta$
and $w(t)=(L_{c(t)^{-1}})_{*}(W(t))$ is a solution of (1.2). It verifies the
Landsberg curvature formula in Proposition 6.1 in [2] for a left invariant
Finsler metric, i.e.,

$L_{y}(w,w,w)=3C_{y}(w,w,[w,y]-N(y,w))-C_{y}(w,w,w,\eta(y)).$

The S-curvature for a Finsler metric $F$ and a smooth measure ${\rm d}\mu$ can
be calculated by

$S(c(t),\dot{c}(t))=\tfrac{{\rm d}}{{\rm d}t}\ln\det(\langle
W_{i}(t),W_{j}(t)\rangle_{\dot{c}(t)})-\tfrac{{\rm d}}{{\rm
d}t}\ln{|\omega(W_{1}(t),\cdots,W_{n}(t))|},$ (3.16)

in which $c(t)$ is a geodesic, $\\{W_{1}(t),\cdots,W_{n}(t)\\}$ is any frame
along $c(t)$, and $\langle\cdot,\cdot\rangle_{\cdot}$ is the fundamental
tensor of $F$ [18]. When $F$ is left invariant, ${\rm d}\mu=\omega$ is a left
invariant volume form, and $W_{i}(t)$’s are taken to be linearly parallel
along $c(t)$ and orthonormal with respect to
$\langle\cdot,\cdot\rangle_{\dot{c}(t)}$, the first summand in the right of
(3.16) vanishes, and Theorem 1.1 implies

$\displaystyle S(e,y(t))$ $\displaystyle=$ $\displaystyle
S(c(t),\dot{c}(t))=-\tfrac{{\rm d}}{{\rm
d}t}\ln|\omega(W_{1}(t),\cdots,W_{n}(t))|$ $\displaystyle=$
$\displaystyle-\tfrac{{\rm d}}{{\rm d}t}\ln|\omega(w_{1}(t),\cdots,w_{n}(t))|$
$\displaystyle=$ $\displaystyle-\tfrac{\omega(\tfrac{{\rm d}}{{\rm
d}t}w_{1}(t),w_{2}(t),\cdots,w_{n}(t))+\cdots+\omega(w_{1}(t),\cdots,w_{n-1}(t),\tfrac{{\rm
d}}{{\rm d}t}w_{n}(t))}{\omega(w_{1}(t),\cdots,w_{n}(t))}$ $\displaystyle=$
$\displaystyle-\sum_{i=1}^{n}\langle\tfrac{{\rm d}}{{\rm
d}t}w_{i}(t),w_{i}(t)\rangle_{y(t)}$ $\displaystyle=$
$\displaystyle\sum_{i=1}^{n}\langle
N(y(t),w_{i}(t))+[y(t),w_{i}(t)],w_{i}(t)\rangle_{y(t)}$ $\displaystyle=$
$\displaystyle\mathrm{Tr}_{\mathbb{R}}N(y(t),\cdot)+\mathrm{Tr}_{\mathbb{R}}\mathrm{ad}(y(t)).$

in which $y(t)=(L_{c(t)^{-1}})_{*}(\dot{c}(t))$ and
$w_{i}(t)=(L_{c(t)^{-1}})_{*}(W_{i}(t))$ for each $i$. It verifies the
S-curvature formula in Proposition 6.1 of [2], i.e.,

$S(y)=\mathrm{Tr}_{\mathbb{R}}N(y,\cdot)+\mathrm{Tr}_{\mathbb{R}}\mathrm{ad}(y).$

##  4\. Nonlinear parallel translation along a smooth curve

###  4.1. Proof of Theorem 1.4

Let $G$ be a connected Lie group endowed with a left invariant spray structure
$\mathbf{G}=\mathbf{G}_{0}-\mathbf{H}=u^{i}\widetilde{U}_{i}-\mathbf{H}^{i}\partial_{u^{i}}$
with the spray vector field $\eta$, and $c(t)$ with $t\in(a,b)$ a smooth curve
on $G$ which is simple (i.e., it has no self intersection) and has
nonvanishing $\dot{c}(t)$ for all values of $t$.

We denote $\dot{c}(t)=w^{i}(t)U_{i}(c(t))$, then its horizonal lifting
$\widetilde{\dot{c}(t)}^{\mathcal{H}}=w^{i}(t)\widetilde{U}^{\mathcal{H}}_{i}|_{T_{c(t)}G\backslash\\{0\\}}$
is a smooth tangent vector field on the imbedded submanifold
$S=\cup_{t\in(a,b)}(T_{c(t)}G\backslash\\{0\\})$ of $TG\backslash 0$. Lemma
3.2 in [16] provides a formula for $\widetilde{U}^{\mathcal{H}}_{i}$, i.e.,

######  Lemma 4.1.

The horizonal lifting of $U_{q}$ is
$\widetilde{U}^{\mathcal{H}}_{q}=\widetilde{U}_{q}-(\tfrac{1}{2}\tfrac{\partial}{\partial
u^{q}}\mathbf{H}^{i}-\tfrac{1}{2}u^{j}c^{i}_{qj})\partial_{u^{i}}.$

Its proof uses (2.9), and a very similar calculation as (3.12).

Using Lemma 4.1 and (2) of lemma 2.2, we get the decomposition

$\displaystyle\widetilde{\dot{c}(t)}^{\mathcal{H}}$ $\displaystyle=$
$\displaystyle
w^{i}(t)\widetilde{U}^{\mathcal{H}}_{i}|_{T_{c(t)}G\backslash\\{0\\}}=w^{i}(t)(\widetilde{U}_{i}-(\tfrac{1}{2}\tfrac{\partial}{\partial
u^{i}}\mathcal{H}^{j}-\tfrac{1}{2}u^{k}c^{j}_{ik})\partial_{u^{j}})|_{T_{c(t)}G\backslash\\{0\\}}$
(4.17) $\displaystyle=$ $\displaystyle
w^{i}(t)\phi^{j}_{i}(c(t))\widetilde{V}_{j}|_{T_{c(t)}G\backslash\\{0\\}}+(\tfrac{1}{2}w^{i}(t)u^{p}c^{j}_{pi}-\tfrac{1}{2}w^{i}(t)\tfrac{\partial}{\partial
u^{i}}\mathcal{H}^{j})\partial_{u^{j}}|_{T_{c(t)}G\backslash\\{0\\}}$

at each $(c(t),y)\in S$ with $y=u^{i}U_{i}(c(t))\in
T_{c(t)}G\backslash\\{0\\}$. Both summands in the right side of (4.17) are
smooth vector fields on $N$. In particular, the first one,
$w^{i}(t)\phi^{j}_{i}(c(t))\widetilde{V}_{j}|_{T_{c(t)}G\backslash\\{0\\}}$
lifts $\dot{c}(t)$.

On $S$, we have the global coordinate $(t,u^{1},\cdots,u^{n})$ for
$y=u^{i}U_{i}(c(t))\in T_{c(t)}M\backslash\\{0\\}$, and the corresponding
global frame $\\{\partial_{t},\partial_{u^{1}},\cdots,\partial_{u^{n}}\\}$.
Using this frame, $\widetilde{\dot{c}(t)}^{\mathcal{H}}$ can be presented as

######  Lemma 4.2.

Using the global frame
$\\{\partial_{t},\partial_{u^{1}},\cdots,\partial_{u^{n}}\\}$ on $S$, we have

$\widetilde{\dot{c}(t)}^{\mathcal{H}}=\partial_{t}+(-\tfrac{1}{2}w^{i}\tfrac{\partial}{\partial
u^{i}}\mathcal{H}^{j}+\tfrac{1}{2}w^{i}(t)u^{p}c^{j}_{pi})\partial_{u^{j}},$

for each $(c(t),y)\in S$ with $y=u^{i}U_{i}(c(t))\in
T_{c(t)}G\backslash\\{0\\}$.

Proof. We only need to prove
$\partial_{t}=w^{i}(t)\phi^{j}_{i}(c(t))\widetilde{V}_{j}|_{T_{c(t)}G\backslash\\{0\\}}$
on $N$. Notice that the smooth vector field on $N$ which lifts $\dot{c}(t)$
and keeps all $u^{i}$’s invariant is unique. Obviously $\partial_{t}$ on $N$
is such a lifting. The left invariancy of the $u^{i}$’s implies
$\widetilde{V}_{j}u^{i}=0,\forall i,j$. Together with (4.17), it implies
$w^{i}(t)\phi^{j}_{i}(c(t))\widetilde{V}_{j}|_{T_{c(t)}G\backslash\\{0\\}}$ is
also such a lifting. These two liftings must be the same.

Proof of Theorem 1.4. Since Theorem 1.4 is a local result, we only need to
prove it in the case that $c(t)$ is a simple smooth curve. We may further
assume $c(t)$ is defined for $t\in(a,b)$ and it has nonvanishing $\dot{c}(t)$
everywhere, because continuity can help us with the rest.

Using the global coordinate $(t,u^{1},\cdots,u^{n})$ on $S$, a curve
$(c(t),Y(t))$ in $S$ with $Y(t)=u^{i}(t)U_{i}(c(t))\in
T_{c(t)}G\backslash\\{0\\}$ can be represented as
$(t,u^{1}(t),\cdots,u^{n}(t))$. Then Lemma 4.2 indicates that $Y(t)$ is
nonlinearly parallel, i.e., $(c(t),Y(t))$ is an integral curve of
$\widetilde{\dot{c}(t)}^{\mathcal{H}}$, iff

$\tfrac{{\rm d}}{{\rm
d}t}u^{j}(t)=\tfrac{1}{2}w^{i}(t)u^{p}c^{j}_{pi}-\tfrac{1}{2}w^{i}\tfrac{\partial}{\partial
u^{i}}\mathcal{H}^{j},\quad\forall j.$ (4.18)

Using left translations, i.e., $y(t)=(L_{c(t)^{-1}})_{*}$
$(Y(t))=u^{i}(t)e_{i}$ and
$w(t)=(L_{c(t)^{-1}})_{*}(\dot{c}(t))=w^{i}(t)e_{i}$, (4.18) is translated to
$\tfrac{{\rm d}}{{\rm d}t}y(t)+N(y(t),w(t))=0$. This ends the proof.

###  4.2. Two questions related to Landsberg Conjecture and holonomy

Theorem 1.4 and Theorem 1.5 imply for any left invariant spray structure
$\mathbf{G}$ on the Lie group $G$, the following space of smooth vector fields
on $\mathfrak{g}\backslash\\{0\\}$, $\mathcal{N}=\\{N(\cdot,w),\forall
w\in\mathfrak{g}\\}$, contains all information of nonlinear parallel
translations. Naturally we would like to know more about the Lie algebra
$\mathfrak{H}$ that $\mathcal{N}$ generates using the canonical bracket
between smooth vector fields.

Here are some examples. When
$\mathbf{G}=\mathbf{G}_{0}=u^{i}\widetilde{U}_{i}$, we have $\eta=0$ and
$N(y,w)=-\tfrac{1}{2}[y,w]$. It is easy to check
$[-2N(\cdot,w_{1}),-2N(\cdot,w_{2})]=-2N(\cdot,[w_{1},w_{2}])$, so in this
case $\mathfrak{H}=\mathcal{N}$ is isomorphic to
$\mathfrak{g}/\mathfrak{c}(\mathfrak{g})$. More generally, when the left
invariant spray structure $\mathbf{G}$ is affine (see Definition 6.1.1 in
[13]), the associated spray vector field $\eta$ is quadratic. Then
$\mathfrak{H}$ is a finite dimensional subalgebra in
$\mathfrak{gl}(\mathfrak{g},\mathbb{R})$.

Above examples suggest we ask

######  Question 4.3.

Is there an example of left invariant spray structure $\mathbf{G}$ such that
$\mathbf{G}$ is not affine and $\dim\mathfrak{H}$ is finite?

Finsler geometry provides another motivation for Question 4.3. In Finsler
geometry, a metric is called a Berwald metric if its induced spray structure
is affine [14], and it is called a Landsberg metric if all nonlinearly
parallel translations are isometries for the Hessian metrics on the punctured
tangent spaces [7]. Landsberg Problem asks if there exists a (regular)
Landsberg metric which is not Berwald [12]. See [15][17] and the references
therein for some recent progress on this problem. Theorem 1.5 implies that, if
$\mathbf{G}$ is induced by a left invariant Landsberg metric, then
$\mathfrak{H}$ is a Lie subalgebra in the space of all Killing vector fields
for the Hessian metric of $F(e,\cdot)$ on $\mathfrak{g}\backslash\\{0\\}$,
which must have a finite dimension. So Question 4.3 may be viewed as a
generalization for Landsberg Problem in the left invariant spray geometry.

Another natural question for $\mathfrak{H}$ is the following.

######  Question 4.4.

What is the relation between the Lie algebra $\mathfrak{H}$ and the restricted
holonomy group of $(G,\mathbf{G})$.

The restricted holonomy group $\mathrm{Hol}_{0}(G,\mathbf{G})$ of
$(G,\mathbf{G})$ is the subgroup of $\mathrm{Diff}(\mathfrak{g}\backslash 0)$
generated by $\mathbf{P}^{\mathrm{nl}}_{c(0),c(1);c}$ for all piecewise smooth
curves $c(t):[0,1]\rightarrow G$ which are homotopic to a constant map and
satisfy $c(0)=c(1)=e$.

When the left invariant spray structure $\mathbf{G}$ is induced by a
Riemannian metric $F$ on the Lie group $G$, $\mathrm{Hol}_{0}(G,\mathbf{G})$
is a compact Lie group. Comparing Lemma 2.2 in [9] and Definition 4 in [3],
then we see that $N(\cdot,v)$ coincides with the linear operator $a_{v}$ in
[8] for each $v\in\mathfrak{g}$. So in this case, Theorem 4.5 in [8] indicates
$\mathfrak{H}=\mathrm{Lie}(\mathrm{Hol}_{0}(G,\mathbf{G}))$ when either $G$ is
compact or $(G,F)$ is irreducible with nonvanishing Ricci curvature.

However, when the left invariant $\mathfrak{H}$ is more generic, very likely
both $\mathfrak{H}$ and $\mathrm{Hol}_{0}(G,\mathbf{G})$ have infinite
dimensions [6], making Question 4.4 is much harder in this situation.

Acknowledgement. This paper is supported by Beijing Natural Science Foundation
(Z180004), National Natural Science Foundation of China (12131012, 11821101).
The author sincerely thanks Ming Li for helpful discussions.

## References

  * [1] L. Berwald, Untersuchung der Krümmung allgemeiner metrischer Räume auf Grund des in ihnen herrschenden Parallelismus, Math. Z. 25 (1926), 40-73.
  * [2] L. Huang, On the fundamental equations of homogeneous Finsler spaces, Diff. Geom. Appl. 40 (2015), 187-208.
  * [3] Ricci curvature of left invariant Finsler metrics on Lie groups, Israel J. Math. 207 (2) (2015), 783-192.
  * [4] L. Huang, Flag curvatures of homogeneous Finsler spaces, Euro. J. Math., S.I. 3 (4) (2017), 1000-1029.
  * [5] L. Huang and X. Mo, Inverse problem of left invariant sprays on Lie groups, Internat. J. Math. (2021), doi:10.1142/S0129167X21500762.
  * [6] B. Hubicska, V.S. Matveev and Z. Muzsnay, Almost all Finsler metrics have infinite dimensional holonomy group, J. Geom. Anal. 31 (2021), 6067-6079.
  * [7] Y. Ichijyō, On special Finsler connections with vanishing hv-curvature tensor, Tensor, N.S. 32 (1978), 146-155.
  * [8] B. Kostant, Holonomy and the Lie algebra of infinitesimal motions of a Riemannian manifold, Trans. Amer. Math. Soc. 80 (2) (1955), 528-542.
  * [9] B. Kostant, On holonomy and homogeneous spaces, Nagoya Math. J. 12 (1957), 31-54.
  * [10] Y. Li, X. Mo and Y. Yu, Inverse problem of sprays with scalar curvature, Internat. J. Math. 30 (2019), 1950041.
  * [11] B. Li and Z. Shen, Sprays of isotropic curvature, Int. J. Mah. 29 (1) (2018), 1850003.
  * [12] M. Matsumoto, Remarks on Berwald and Landsberg spaces, Finsler geometry (Seattle, WA, 1995), 79-82, Contemp. Math. 196, Amer. Math. Soc., Providence, RI, 1996.
  * [13] Z. Shen, Differential Geometry of Spray and Finsler Spaces, Kluwer Academic Publishers, (2001).
  * [14] Z. Shen, Lectures on Finsler Geometry, World Scientific, (2001).
  * [15] A. Tayebi and B. Najafi, On homogeneous Landsberg surfaces, J. Geom. Phys. (2021), doi:10.1016/ j.geomphys.2021.104314.
  * [16] M. Xu, Left invariant spray structure on Lie group, preprint (2021), arXiv:2103.08901v2.
  * [17] M. Xu and V.S. Matveev, Proof of Laugwitz Conjecture and Landsberg Unicorn Conjecture for Minkowski norms with $SO(k)\times SO(n-k)$-symmetry, Canad. J. Math. (2021), doi:10.4153/S0008414X21000304.
  * [18] M. Xu, V.S. Matveev, K. Yan and S. Zhang, Some geometric correspondences for homothetic navigation, Publ. Math. Debrecen 97 (3-4) (2020), 449-474.
  * [19] G. Yang, Some classes of sprays in projective spray geometry, Diff. Geom. Appl. 29 (2011), 601-614.


# SwiftNet: Using Graph Propagation as Meta-knowledge to Search Highly
Representative Neural Architectures

Hsin-Pai Cheng  ECE Department, Duke University, Durham, NC 27708  Tunhou
Zhang  ECE Department, Duke University, Durham, NC 27708  Yukun Yang  ECE
Department, Duke University, Durham, NC 27708  Feng Yan  CSE Department,
University of Nevada, Reno, NV 89557  Shiyu Li  ECE Department, Duke
University, Durham, NC 27708  Harris Teague  Qualcomm AI Research, 5775
Morehouse Drive, San Diego, CA 92121  Hai Li  ECE Department, Duke University,
Durham, NC 27708  Yiran Chen  ECE Department, Duke University, Durham, NC
27708

###### Abstract

Designing neural architectures for edge devices is subject to constraints of
accuracy, inference latency, and computational cost. Traditionally,
researchers manually craft deep neural networks to meet the needs of mobile
devices. Neural Architecture Search (NAS) was proposed to automate the neural
architecture design without requiring extensive domain expertise and
significant manual efforts. Recent works utilized NAS to design mobile models
by taking into account hardware constraints and achieved state-of-the-art
accuracy with fewer parameters and less computational cost measured in
Multiply-accumulates (MACs). To find highly compact neural architectures,
existing works relies on predefined cells and directly applying width
multiplier, which may potentially limit the model flexibility, reduce the
useful feature map information, and cause accuracy drop. To conquer this
issue, we propose GRAM (GRAph propagation as Meta-knowledge) that adopts fine-
grained (node-wise) search method and accumulates the knowledge learned in
updates into a meta-graph. As a result, GRAM can enable more flexible search
space and achieve higher search efficiency. Without the constraints of
predefined cell or blocks, we propose a new structure-level pruning method to
remove redundant operations in neural architectures. SwiftNet, which is a set
of models discovered by GRAM, outperforms MobileNet-V2 by 2.15$\times$ higher
accuracy density and 2.42$\times$ faster with similar accuracy. Compared with
FBNet, SwiftNet reduces the search cost by 26$\times$ and achieves
2.35$\times$ higher accuracy density and 1.47$\times$ speedup while preserving
similar accuracy. SwiftNet can obtain 63.28% top-1 accuracy on ImageNet-1K
with only 53M MACs and 2.07M parameters. The corresponding inference latency
is only 19.09 ms on Google Pixel 1.

##  1 INTRODUCTION

Figure 1: ImageNet-1K top-1 accuracy density vs Model MACs. SwiftNet achieves
a better Pareto-optimal of accuracy-MAC curve of other state-of-the-art models
with significant smaller size. Experiments are detailed in Section 4.1.

Deep Neural Networks (DNNs) are able to achieve state-of-the-art performance
in many cognitive applications, including computer vision [1], speech
recognition [2], and natural language processing [3, 4]. The architectures of
DNNs are continuously becoming deeper and wider with many new operations and
activation function designs being constantly invented. Such dynamics make
manual design of DNNs very challenging, especially when extensive domain
expertise and experience are needed.

Neural Architecture Search (NAS) was recently proposed to automate the search
for neural network architecture [5]. Evolutionary algorithms [6, 7],
reinforcement learning [8], and differentiable algorithms [9, 10] are the
trending methods for performing NAS and obtaining networks; the performance of
which outperforms the best manually-crafted DNN models [11]. NAS was
particularly studied for searching large-scale neural networks that consume a
large amount of computing power and time [8]. However, in edge computing
applications such as autonomous vehicles, disaster response, and health-care
informatics, etc., many new challenges emerge, including limited computing
power, short learning ramp-up time, real-time inference response, and quick
adaptation to new tasks.

Some recent studies have started to investigate these emerging challenges by
taking into account hardware resource constraints, device runtime latency, and
FLOPs during neural architecture design [12, 13, 10, 14]. Although these
techniques are able to reduce inference latency and/or computational cost,
they often induce a significant accuracy drop when the model size is small,
e.g., less than 20M MACs [11, 15, 10].

The accuracy drop is due to: (1) the predefined cell having less flexibility
for small model design and (2) directly applying width multiplier largely
reduces the useful feature map information. Figure 1 compares the difference
of the architectures we discovered with previous works. Previous works are
related in that they are characterized by their similar Pareto distribution.
Our discovered architectures go beyond the Pareto-optimal of previous works.

To overcome the above limitations of existing NAS approaches, we propose a new
NAS methodology called GRAM (GRAph propagation as Meta-knowledge), which aims
to achieve high performance, rapid and efficient searches, good adaptability,
and multi-objective searches (e.g., resource-aware inference). Our approach
builds on abstracting the operation in computational Directed Acyclic Graph
(DAG) as nodes and the flow of tensors into network layers as passing through
an edge connecting two nodes. In this way, the problem of finding the optimal
neural architecture in the full search space is equivalent to finding the
optimal sub-DAG in the complete DAG. There are two key components in our work:
meta-graph and structure-level pruning. Existing NAS approaches cannot retain
the learned knowledge in previous updates when a new search starts, making the
search process inefficient and with poor adaptability. Thus we develop a new
node-based search method that continuously accumulates the learned neural
architecture knowledge into a meta-graph. To support resource constraints on
edge devices, we propose a novel structure-level pruning method by removing
edges with lower weight values in the complete DAG to produce resource-aware
models.

Compared with traditional cell-based search methods, the proposed node-based
method has much less constraint in search space and thus has significantly
higher accuracy density. GRAM also enables a faster, more efficient search
process as the learned information is accumulated in meta-graph instead of
end-to-end learning as in existing NAS methods. With graph modularization,
GRAM has more flexibility to accumulate and preserve knowledge from past
training experiences and generalize for unseen architecture without any
constraints. In addition, GRAM has great adaptability, thanks to the meta-
graph as it preserves the previously learned information which can be
effectively reused for new tasks. Finally, the proposed structure-level
pruning is better than traditional cell-based pruning as the latter cannot
identify structural redundancy while our method is capable of removing it in
the trained meta-graph. It also facilitates changing objectives as it can be
done by simply changing the meta-graph update rules. We conduct extensive
experimental evaluations on a wide range of image classification applications,
and the results show that SwiftNet achieves state-of-the-art accuracy density
(8.90), which is 2.15$\times$ more than MobileNet-V2 (4.14) [15] and
2.35$\times$ more than FBNet (3.79) [10]. The latency of SwiftNet is
2.42$\times$ lower than MobileNet with similar accuracy. Our search cost is
26$\times$ lower than FBNet, and the latency of SwiftNet is 1.47$\times$ lower
than FBNet with similar accuracy. SwiftNet can achieve 63.28% top-1 accuracy
on ImageNet-1K Classification Task with only 2 million parameters and 53
million MACs.

We summarize our main contributions as follows:

  * •

We proposed GRAM, a new search algorithm for neural architecture based on
meta-graph that can explore much more flexible search space and accumulate
learned knowledge to enable high accuracy density search and good
adaptability.

  * •

We proposed a new structure-level pruning method, which can preserve high
accuracy while transforming to small models, a significant improvement from
the existing channel scaling methods.

  * •

By using the proposed search and pruning methods, the discovered set of models
SwiftNet can outperform state-of-the-art NAS works in accuracy density, top-1
accuracy, latency, and MACs for small models.

  * •

GRAM supports multi-objective neural architecture search and facilitates
adding and changing objectives through changing the update rules in meta-
graph.

Figure 2: Overview diagram of the search process. To form a sampled DNN, we
subsample multiple DAGs from the complete DAG. After several training epochs
with proxy training set, we use the search metrics (such as latency and
accuracy) to update the complete DAG.

##  2 RELATED WORK

#### Deep Neural Networks for Mobile Devices:

Continuously increasing demand for the use of deep neural network in mobile
device promotes the research for low cost and high-efficient models. In the
past several years, mobile neural networks are well hand-crafted and achieved
competitive performance. Squeezenet [16] is a compact deep neural network
designed by delicately selecting channel depth and the convolutional filters.
For example, 3x3 convolution is replaced by 1x1 convolution. Later on,
Mobilenet-V1 [11] is proposed and its separable depthwise convolution brought
a big impact on efficient neural network design. Mobilenet-V2 [15] is a design
with residual and bottleneck layers. These well tuned hand-crafted models have
achieved breakthrough results on mobile vision tasks. These previous works
provide us some good intuitions for designing good mobile models. To push the
boundary further, we need to efficiently explore unseen network design with
novel operations. However, such dynamics makes it challenging and inefficient
for manual design.

#### Neural Architecture Search for Mobile Devices:

After the idea of NAS [8, 5] came out, we are able to find new neural
architectures by automatically explore the combination of operations,
activation functions, hyperparameters, etc [9, 17, 18, 19, 20]. While NAS
achieved 74% top-1 accuracy on ImageNet, it has 5.3 million parameters and 564
million MACs making it inapplicable to mobile devices. Recent works
demonstrated to search mobile models by using FLOPs as regularization,
considering inference latency into the search process and predicting the
accuracy of candidate networks [12, 13, 10, 14, 19, 21].

While these techniques help to find neural architectures with lower inference
latency and computational cost, there are two main things that have not been
well explored. First, to narrow down the wide search space, previous works
used cells and blocks of layers to build neural architectures. However, this
course-grained searching space induces a large number of MACs and parameters.
Second, to downsize the found architecture to smaller a model, channel depth
multiplier is generally adopted. By reducing the number of output channels,
channel depth multiplier can dramatically decrease MACs. But it also comes
with an inevitable and significant accuracy drop. For example, FBNet was
proposed to find a mobile model with low latency and high accuracy. By using
latency as an optimization factor combining with gradient-based search method,
FBNet achieved 74.1% top-1 accuracy on ImageNet, 296M FLOPs. However, to adapt
to a smaller model (13.7 million MACs), FBNet’s accuracy downgraded to 50.2%
[10]. Therefore, how to fast adapt found architecture to a smaller model for
strict hardware constraints efficiently without much accuracy loss become
vitally important.

Recent progress in graph neural network [22, 23, 24] demonstrates the
potential of searching neural architecture with graph-based method [25]. For
example, GHN [25] achieved 73% top-1 accuracy on ImageNet in 0.84 GPU days.
However, the neural architecture of GHN is randomly sampled and the searching
process is unaware of structure redundancy. It causes the found model to have
large MACs and low accuracy density as shown in Figure 1 and 4.

##  3 METHODOLOGY

We propose GRAM (GRAph propagation as Meta-knowledge), a DAG-based methodology
for neural architecture search. Our main goal is to find the optimal neural
architecture based on two search metrics: accuracy and latency 111We use
accuracy and latency as example search metrics in this paper, but our method
can be extended to support other search criteria.:

$\mathcal{A}^{*}=\arg\max\limits_{\mathcal{A}}\mathcal{M}(\mathcal{A}),$ (1)

where $\mathcal{A}$ represents a neural network structure and

$\mathcal{M}=Accuracy(w^{*}(\mathcal{A}),\mathcal{A},\mathcal{D})-\gamma\cdot
Latency(\mathcal{A}).$ (2)

Here $Accuracy()$ denotes the accuracy of current architecture given optimal
weight distribution evaluated by validation dataset. $Latency()$ is measured
by the inference time of given network structure.

$\mathcal{D}$ represents the validation data set, $\gamma$ is an adjustable
penalty term, and

${w^{*}(\mathcal{A})}=\arg\min\limits_{w}\mathcal{L}_{train}({w,\mathcal{A}}),$
(3)

where $w$ represents trainable parameters (weights and biases) and
$\mathcal{L}$ is cross-entropy loss without regularization.

###  3.1 Graphical Representation of Architecture

We represent the structure of the neural network using Directed Acyclic Graph
(DAG): $G=(\mathcal{V,E})$. Each node $v\in V$ has an operation $o_{v}$, such
as 1x1 convolution, parameterized by $w_{v}$. Each node produces an output
tensor $x_{v}$. Edge $e_{u\rightarrow v}=(u,v,w_{uv})\in\mathcal{E}$
represents the flow of the tensor from node $u$ to node $v$, and the
probability of the connection is given by $w_{uv}$. $x_{v}$ is computed by
performing the corresponding operation on each input and then sum the output
of operation as follows:

$x_{v}=\sum_{e_{u\rightarrow v}\in\mathcal{E}}o_{v}(x_{u};w_{v})$ (4)

Meta-Graph. Every sampled neural network can be expressed as $K$ DAGs and each
DAG is a sub-graph of a complete DAG. Therefore, we construct Meta-Graph as
$K$ independent complete DAGs, which contains all the update information of
sampled neural networks.

###  3.2 Architecture Search using GRAM

GRAM is an operation-wise architecture search method that can accumulate the
knowledge learned in each update into meta-graph. Compared with existing NAS
works, GRAM supports much wider search space with significantly fewer
constraints.

Search Space. Existing NAS works perform architecture search based on cell or
block [10] units. These units reduce the search complexity at the cost of
flexibility. In this work, we use node as the basic search unit, where each
node is one of the operations in the computational DAG. Node-based search is
more flexible than cell-based or block-based search because it allows us to
explore new cell structures rather than use the predefined modules. For
Convolutional Neural Networks (CNNs), the convolution is followed by batch
normalization and ELU activation function [26]. Each node can choose the
operation from below:

  * •

1 $\times$ 1 Convolution with 32, 64, 128 filters

  * •

3 $\times$ 3 Convolution with 32, 64 filters

In the interest of space, we use CNNs as an example throughout the paper to
explain our approach, but it can be easily extended to other network
structures by modifying the search space according to the requirements. For
example, an extension to RNNs can be done by substituting the CNN operations
in the meta-graph with RNN operations.

For CNNs, down-sampling is an essential operation to reduce the sensitivity to
output shifts and distortions [27]. Thus we first place several down-sampling
operations in the neural network structure and use them as a coarse-grained
hierarchy. Then we fill multiple complete DAGs in parallel between 2 adjacent
down-sampling operations to form a flexible search space. Multiple DAGs learn
their structural representation independently to generate new architectures.
Note that single DAG is feasible, but multiple DAGs can avoid the bias
introduced in the random DAG institutionalization. The structure is visualized
in Figure 2. Each DAG represents a large space that contains many
possibilities of the forwarding path. Each node can choose from 1 $\times$ 1
and 3 $\times$ 3 convolution with different channel numbers. The topology of
complete DAGs provides all possible connections.Therefore, our search space is
much wider than other cell-based and block-based approaches. For example, with
$n=30$ nodes in each complete DAG, $m=3$ DAGs in parallel between two down-
sampling modules, and $h=3$ down-sampling modules in total, the search space
contains $10^{84}$ possible architectures. In comparison, FBNet with block
design only provides around $10^{21}$ possible architectures.

Search Algorithm. Figure 2 gives an overview of GRAM’s search process. In
phase A, we sample each DAG to obtain a sampled neural network. In phase B, we
train the sampled neural network on proxy data set and get the performance
metrics. Then in phase C, we update the connection weights in each DAG to
meta-graph according to the metrics. Since each DAG is sampled and updated
individually, they are independent. Thus the optimal meta-graph distribution
can be described as:

$P^{*}(\mathbf{E_{w}})=\prod_{k=1}^{K}p^{*}(\mathbf{E_{w}^{k}}),$ (5)

where $E_{w}^{k}\in\mathbb{R}^{\frac{n(n-1)}{2}}$ is the connection weights of
the k-th DAG, and $P(\mathbf{E_{w}})$ is the distribution function of the
meta-graph.

Each meta-graph is a sample of the optimal distribution. The optimal neural
network structure $\mathcal{A}^{*}$ satisfying Eq. 1 is the mean of the
optimal meta-graph distribution:

$\mathcal{A}^{*}=\mathbb{E}[f(e_{w})]=\int f(e_{w})p(e_{w})de_{w},$ (6)

where $f(e_{w})$ is the probability density function of $\mathbf{E_{w}}$.
Therefore, our meta-graph updating process is to approximate the mean of the
distribution, which is in accordance with the goal of optimizing accuracy and
latency.

Since there is no closed-form solution to compute the mean of the optimal
meta-graph distribution, we adopt Gibbs Sampling method to approximate the
optimal meta-graph by updating the connection weights in meta-graph. More
specifically, at the beginning of each training round, we sample each complete
DAG and construct the sampled neural network with the sampled DAGs and
downsampling modules. Let $E_{w}$ represent a sample in the optimal meta-
graph, and the update process can be described as:

$\begin{split}e_{w(i,j)}^{k,t+1}\sim
p(e_{w(i,j)}^{k}\>|\>e_{w(0,1)}^{k,t+1},e_{w(0,2)}^{k,t+1},...,\\\
e_{w(i-1,j)}^{k,t+1},e_{w(i+1,j)}^{k,t},e_{w(i+2,j)}^{k,t},...)\\\
\forall{k\in[0,K)},\forall{t\in[0,T)},\forall{i,j}\in\mathcal{V},i<j,\end{split}$
(7)

where $K$ is the number of the DAGs, $T$ is the iteration times. The update
process stops when meta-graph converges to the mean of the optimal meta-graph
distribution and the convergence is guaranteed by Gibbs Sampling. Adopting
Gibbs Sampling instead of performing a search on the complete DAG also greatly
reduces the computational overhead.

Algorithm 1 Update Connection Weights in Meta-graph.

Input:
G: Initialized meta-graph with uniform connection weights
Hyperparameter:
$K$: total number of complete DAGs
$n$: number of nodes for each DAG
$m$: number of DAGs per hierarchy
$h$: number of hierarchies in the architecture
$\alpha$: convergence learning rate of meta-graph
$\beta$: moving expected performance on specific tasks, which is updated
according to history performance collected through meta-graph training
Intermediate variables:
$\tau$: latency of sampled model
$\eta$: accuracy on the validation set of sampled model
$\eta^{\prime}$: accuracy penalized by latency on the validation set of
sampled model
$Z_{t}$: Connection weight normalization term
begin

for $t\leftarrow 1$ to $T$ do

Sub-sample Meta-graph and train on proxy dataset
$G_{sub}=subsample(\textbf{G})$
${w^{*}}=\left.\arg\min\limits_{w}\mathcal{L}_{train}({w,G_{sub},\mathcal{D}})\right.$

Evaluate the model and calculate penalized accuracy
$\left.\eta,\tau=Eval(w^{*},G_{sub},\mathcal{D})\right.$
$\eta^{\prime}=\left.\eta-\gamma\tau\right.$

Update connection weights for meta-graph:

for $k\leftarrow 0$ to $K-1$ do

Update connection weights for $DAG_{k}$
$e_{w(i,j)}^{k,t}=\left.\begin{cases}\frac{e_{w(i,j)}^{k,t-1}}{Z_{t}}\exp[\alpha(\eta^{\prime}-\beta)]&\text{If
$e^{k}_{i\rightarrow j}\in\mathcal{E}_{k}$}\\\
\frac{e_{w(i,j)}^{k,t-1}}{Z_{t}}&\text{Otherwise}\end{cases}\right.$

end for

end for

end

Weights Update. After obtaining the sampled neural network structure, we train
the network on the proxy data set and estimate the performance of the sampled
network. In this work, we use accuracy and latency as example metrics for
weight updating. The weights are updated according to the difference between
the estimated performance and expected performance. Note that only the weights
of edges chosen in the sampled neural network are updated. We apply
exponential function to boost the updating rate, and a scale factor $\alpha$
to adjust the rate of the update. The updated weights are normalized for a
valid probability representation. The pseudo code of the complete graph
updating is given in Algorithm 1.

###  3.3 Structure-Level Pruning for Compact Model

GRAM is highly adaptable to different sizes of models. To support resource-
constraint edge computing applications, we propose a structure level pruning
method. Existing works such as MobileNets [11, 15] uses width multiplier to
thinner the models and down-scale the channel depth to remove the redundant
channels. Such approaches could not identify redundant operations and thus
less efficient than the proposed structure-level pruning approach. To generate
compact architectures from the meta-graph, we sample neural architectures by
setting a pruning level for the connection weights. More specifically, the
connection weights that are below the given level are pruned while the
connections weights larger than the level are kept. In addition, we constrain
the maximum filter depth in the search space, which shares the similar idea as
channel down-scaling to efficiently reduce model size without accuracy loss.
We develop a profiling-based method to determine the pruning level based on
the resource constraint, which is detailed in Section 4.2.

#### Channel Up-scaling.

Small models are prone to underfitting for challenging tasks because they have
limited representation capacity. For example, if we transfer the trained meta-
graph on CIFAR-10 proxy dataset to the more challenging task ImageNet, the
model capacity may be not sufficient and results in accuracy loss.To solve
this issue, we introduce the channel up-scaling technique. Channel up-scaling
applies a scaling factor larger than 1 for each node’s output channel. The
channel up-scaling is node-wise in our approach, so the scaling has better
flexibility.

##  4 EXPERIMENTS

###  4.1 Experiment Setup

Meta-graph. We configure meta-graph with 3 coarse-grained hierarchies. Each
hierarchy contains 3 complete DAGs in parallel with 30 nodes for each DAG.
Pooling size and stride for each down sampling layer is set to 2. We use batch
normalization layer after each convolutional layer except the last one. We use
ELU [26] activation after every batch normalization layer to speed up the
training. $\alpha$ is set to 0.9. $\beta$ is the moving average of the history
performance with initial value 0.4. $\gamma$ is set to 1 to penalize latency
in the unit of ms. We trained the meta-graph for 1000 iterations.

Proxy Dataset. Training the meta-graph on the entire dataset is too expensive,
so we adopt the approach used in previous works [19, 8, 28, 10] to train the
meta-graph on a small proxy dataset. Here we randomly select 500 samples from
the entire dataset as our proxy dataset.

Full Model Training on CIFAR-10. We train the models generated from meta-graph
with the full dataset for evaluating the performance, e.g., for CIFAR-10 [29],
there are 50,000 images in the training dataset and 10,000 images in the
validation dataset. We use the momentum optimizer [30] with initial learning
rate 0.01, and decay the learning rate by 0.1 at 50% and 75% of the total
training epochs respectively. To prevent overfitting, we use 0.0005 L2 weight
decay during training.

Model Adaptation on ImageNet. After training the meta-graph on CIFAR-10, we
adapt it to ImageNet-1K [31] by adding sequential convolutional layers before
the generated model and adjusting the final average pooling layer to match the
dimension of the outputs. We adopt the hyper-parameter settings in
Inception-v3 [32], and use the RMSprop [33] optimizer with initial learning
rate 0.045 and decay the learning rate by 0.94 for every two epochs. Since the
generated model is not prone to overfitting, we use 0.00001 L2 weight decay
during training to speedup the process.

Figure 3: A representative architecture discovered by GRAM.

###  4.2 Performance Analysis

#### SwiftNet.

Figure 3 demonstrates a representative architecture (composed of DAGs as
building components) discovered by GRAM and we name it SwiftNet. With
different initialization, each DAG develops its independent structure after
graph propagation through meta-graph training and structure-level pruning. The
structure knowledge learned by each DAG is highly representative: the DAG on
the left learns a combination and concatenation of different-sized filters to
extract spatial information from the input feature map; the DAG in the middle
identities mapping to transfer the current input feature map to the next
hierarchy; and the DAG on the right explores a set of $1\times 1$
convolutional operators. Intuitively, these $1\times 1$ operators serve as the
approximation of $3\times 3$ convolutional filters to best utilize the
information from the input feature map.

As shown in Table 1, compared with state-of-the-art methods, SwiftNet has much
higher accuracy density while fewer parameters, i.e., 85.92% with only 250k
parameters. Unlike FBNet [10] and other cell-based or block-based approaches,
SwiftNet is not restricted to stacking sequential layers with mutable channel
depth and negligible connections and thus has much larger search space and
more flexible architecture choice. SwiftNet contains the characteristic of
Inception and Densely-connected network [32, 34]. The Inception-like structure
is enabled by paralleling different DAGs to extract and map the input feature
maps, and DenseNet-like structure is enabled by preserving the node
connections in each DAG. SwiftNet is less memory and computationally intensive
than Inception and DenseNet as the structure-level pruning heavily reduces
redundant connections in the meta-graph while preserves critical filter stacks
and concatenations.

Table 1: Performance and accuracy density of different versions of SwiftNet
trained on CIFAR-10 in the format of SwiftNet-0.5, where 0.5 represents the
level of structure-level pruning.

Model | Accuracy | #MACs | Density
---|---|---|---
| (%) |  | (%/M-MACs)
SwiftNet-0.65 | 85.92 | 3M | 28.64
SwiftNet-0.5 | 88.74 | 5M | 17.75
SwiftNet-0.4 | 89.09 | 12M | 7.42
SwiftNet-0.3 | 90.17 | 36M | 2.50

Figure 4: ImageNet-1K top-1 accuracy density comparison between SwiftNet and state-of-the-art NAS approaches. Accuracy density shows how well a model use computational resource. SwiftNet achieves 2.15$\times$ accuracy density compared with the MobileNets. Table 2: Compact model comparison between SwiftNet and baselines on ImageNet-1K. The 0.35 in MobileNet-V2-0.35 and FBNet-96-0.35-2 both stand for 0.35 width multiplier. In comparison, SwiftNet-0.4 does not applied width multiplier. By using structure pruning with a level of 0.4, SwiftNet achieves a better accuracy and latency trade-off. *Our searching is done by 1 NVIDIA GEFORCE GTX 1080. $\dagger$ Since our work is focusing on compact model design, we compare SwiftNet with baseline models of similar scale.  Model | Method | Search | Search Cost | Latency(ms) | #MACs | Top-1
---|---|---|---|---|---|---
|  | Space | (GPU hours) |  |  | Accuracy(%)
MobileNet-V2-0.35$\dagger$ | Manual | – | – | 4.5 | 11M | 45.5
FBNet-96-0.35-2$\dagger$ | Gradient | layer-wise | 216 | 4.26 | 13.7M | 51.9
SwiftNet-0.4$\dagger$ | GRAM | node-wise |  8.3* | 4.24 | 10.4M | 54.4

#### Performance Comparison with State-of-the-art NAS Methods.

Figure 4 illustrates that SwiftNet outperforms other architectures in terms of
accuracy density, which indicates that SwiftNet can utilize the parameters and
MACs more efficiently. Table 2 shows the comparison between SwiftNet and
state-of-the-art NAS approaches in terms of search cost, latency, and top-1
accuracy. In the interest of space and also the fact that some NAS approaches
mainly focus on relatively large models (i.e., they do not provide the results
of comparable small models), the results of NAS approaches with significantly
lower accuracy density are not shown in the table. The results show that
SwiftNet outperforms both MobileNet-V2 and FBNet in all metrics. Specifically,
the search speed of GRAM is about $26\times$ faster than FBNet even though the
search space of GRAM is more than $10^{63}$ times larger, which suggests the
high search efficiency of GRAM. The high search efficiency comes from the
sampling process and weight update scheme as connections that lead to bad
performance on proxy dataset are unlikely to be sampled in the next iteration.
In terms of accuracy, our node-wise search space allows GRAM to explore models
with greater structure diversity than MobileNet-V2 and FBNet. In a vision
perspective, by combining both simpler and more complex structures from
independent DAGs, our meta-graph is able to approximate hypercomplex receptive
fields [35] in favour of accuracy. Detailed discussion on the latency
advantage of SwiftNet is present in Section 4.3.

#### Structure-Level Pruning.

The proposed structure-level pruning method is used to extract and refine the
final model according to the resource constraints or other objectives. To
determine the pruning level based on the resource constraint, we conduct
profiling to find out the relationship between model scale (or other resource
constraint metrics) and structure pruning level. We convert our sampled
architectures to TensorFlow models and do the profiling to calculate MACs and
number of parameters. The profiling is lightweight as it can be finished in
less than 1 hour on a Xeon E5 CPU. Figure 5 shows the profiling results. The
model scale drops down drastically in the range of [0.2, 0.6], which suggests
pruning is more efficient in this range.

Figure 5: Model scale vs Structure pruning level.

#### Channel Up-scaling.

We evaluate the same generated architecture with channel up-scaling factor
1.25 and 1.5 on ImageNet classification task. To ensure a fair comparison
between different scaling factors, we use fixed data-preprocessing pipelines
and hyper-parameter settings in training process. Table 3 demonstrates the
accuracy, MACs, and Model Size under different scaling settings on ImageNet.
As expected, the best model has the lower pruning level and higher up-scaling
factor. The results suggest Channel Up-Scaling can provide up to 5% top-1
accuracy improvement at the cost of about 1 Million additional parameters.
This verifies that Channel Up-Scaling is useful when transfer existing model
for new tasks, especially when the new task is more challenging than existing
model.

Table 3: Performance, MACs, and Model Size of different versions of SwiftNet
trained on ImageNet-1K in the format of SwiftNet-96-0.5-1.00, where 96
represents the image input size, 0.5 represents the structure prune level, and
1.00 represents the channel up-scaling factor. Channel Up-scaling provides up
to 5% performance gain at the cost of up to 2$\times$ model size.

Model | Top-1 Accuracy(%) | Top-5 Accuracy(%) | #MACs | #Parameter
---|---|---|---|---
SwiftNet-96-0.5-1.00 | 48.24 | 73.56 | 10M | 0.87M
SwiftNet-96-0.4-1.00 | 54.44 | 77.12 | 14M | 1.15M
SwiftNet-128-0.5-1.00 | 54.69 | 77.27 | 18M | 0.87M
SwiftNet-128-0.4-1.00 | 59.13 | 81.32 | 26M | 1.15M
SwiftNet-96-0.5-1.25 | 51.03 | 74.33 | 15M | 1.20M
SwiftNet-96-0.4-1.25 | 56.21 | 78.83 | 21M | 1.58M
SwiftNet-128-0.5-1.25 | 57.91 | 80.51 | 26M | 1.20M
SwiftNet-128-0.4-1.25 | 61.99 | 84.26 | 38M | 1.58M
SwiftNet-96-0.5-1.50 | 52.76 | 76.01 | 20M | 1.56M
SwiftNet-96-0.4-1.50 | 57.46 | 80.03 | 30M | 2.07M
SwiftNet-128-0.5-1.50 | 59.45 | 81.62 | 36M | 1.56M
SwiftNet-128-0.4-1.50 | 63.28 | 85.22 | 53M | 2.07M

Figure 6: Trade-off between latency and accuracy on ImageNet-1K Classification
Task. We achieve significant latency improvement than MobileNet-V-2 and FBNet
under the same top-1 accuracy, i.e., up to 2.42 and 1.47 speedup respectively.
Since FBNet did not report latency on Google Pixel 1, we map FBNet’s latency
to Google Pixel 1 according to MACs by referring to [10].

###  4.3 Latency Analysis

To measure runtime latency, we prototype SwiftNet as a TensorFlow Lite model
and deploy it as a mobile APP on the Google Pixel 1. We compare the accuracy-
latency curve between SwiftNet and MobileNet-V2 [15] under input size of
96$\times$96 and 128$\times$128 in Figure 6. Under the same accuracy, the
latency of SwiftNet is significantly better (i.e., up to 2.42$\times$ better
than MobileNet and up to 1.47$\times$ better than FBNet) for input size of
96$\times$96 and is also better (i.e., up to 2.11$\times$ better than
MobileNet and up to 1.12$\times$ better than FBNet) for input size of
128$\times$128\. In addition, our experimental results also show that the
architecture of SwiftNet makes good use of the parallel scheme, i.e., SwiftNet
gets about 2$\times$ speedup in runtime latency when running in multi-
threading mode than single-thread. This is because the computation of the
three DAGs can be performed concurrently, see Figure 3.

Table 4: Adaptability of meta-graph to different tasks.

Task | #MACs | Accuracy | Accuracy Density
---|---|---|---
|  | (%) | (%/M-MACs)
MNIST | 0.23M | 99.24 | 431.47
SVHN | 1.15M | 94.95 | 82.56
Fashion MNIST | 0.86M | 93.56 | 108.79

###  4.4 Structural Adaptability to Different Tasks

One of the most important contributions of the proposed method is that the
trained meta-graph can be easily adapted to new tasks and schemes
structurally. Existing neural architecture search methods finds a definite
architecture with very limited degree of freedom as the found architecture can
only be changed by applying the width multiplier or changing layers around the
searched architecture. Although changing the filter depth is beneficial to
adjusting the model to new tasks, the models usually cannot be structurally
adapted to the new scheme. Therefore, they usually require multiple steps of
updates (shots) to tune the architecture towards the new task.

However, our GRAM methodology is capable of providing a range of flexible
architectures by changing the level of structure-level pruning. Since
structural information is preserved and accumulated during meta-graph
training, we no longer need to do the fine-tune updates before adapting our
meta-graph to new tasks. This means it is convenient for meta-graph to
directly form a new deep neural network for new tasks without further
training.

We select a wide range of image classification tasks and use the trained meta-
graph (trained on CIFAR-10) to directly generate architectures for them. The
datasets include MNIST [27], Fashion MNIST [36], and SVHN [37]. Table 4 shows
the performance of the generated architectures. The adapted models achieve
good performance (i.e., 99.24% accuracy on MNIST, 94.95% accuracy on SVHN, and
93.56% accuracy on Fashion MNIST) while preserving at least 80 accuracy
density, thanks to the great flexibility in the trained meta-graph.

###  4.5 Limitations of GRAM

While GRAM is able to explore a large amount of potential neural architectures
in a short amount of time, the convergence of meta-graph relies on Gibbs
Sampling, which does not provide a strict guarantee of upper bound with
respect to the number of iterations before convergence. A rigorous theoretical
guarantee on meta-graph convergence analysis is deferred as our future work.

##  5 CONCLUSION

We present GRAM (GRAph propagation as Meta-knowledge) for searching highly
representative neural architectures. By using meta-graph as meta-knowledge to
preserve learned knowledge, GRAM can efficiently search a space that is
$10^{63}$ larger than FBNet’s search space in just $8.3$ hours. Unlike
previous works, instead of using channel depth scaling for the trade-off
between resource and accuracy, we propose structure-level pruning to achieve a
higher accuracy density. GRAM successfully discovered a set of models named
SwiftNet, which outperforms MobileNet-V2 by $2.15\times$ higher accuracy
density, 2.42$\times$ faster with similar accuracy. SwiftNet also outperforms
FBNet, a differentiable-based searching method, by $26\times$ in search cost
and $1.47\times$ lower in inference latency while preserving a similar
accuracy. SwiftNet can achieve 63.28% top-1 accuracy on ImageNet-1K
Classification Task with only 2 million parameters and 53 million MACs. GRAM
can be strengthened by the proposed channel up-scaling to avoid underfitting
when adapt to more challenging tasks using trained meta-graph. GRAM can be
quickly adapted to different tasks with no update shot, thanks to the
flexibility of proposed meta-graph.

## References

  * [1] K. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask r-cnn,” in Proceedings of the IEEE International Conference on Computer Vision, pp. 2961–2969, 2017.
  * [2] G. Hinton, L. Deng, D. Yu, G. Dahl, A.-r. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, B. Kingsbury, et al., “Deep neural networks for acoustic modeling in speech recognition,” in IEEE Signal processing magazine, vol. 29, 2012.
  * [3] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in arXiv preprint arXiv:1810.04805, 2018.
  * [4] G. Lample, M. Ott, A. Conneau, L. Denoyer, and M. Ranzato, “Phrase-based & neural unsupervised machine translation,” in Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2018.
  * [5] B. Zoph and Q. V. Le, “Neural architecture search with reinforcement learning,” in Proceedings of the International Conference on Learning Representations, 2017.
  * [6] E. Real, S. Moore, A. Selle, S. Saxena, Y. L. Suematsu, J. Tan, Q. V. Le, and A. Kurakin, “Large-scale evolution of image classifiers,” in Proceedings of the 34th International Conference on Machine Learning-Volume 70, pp. 2902–2911, JMLR. org, 2017.
  * [7] T. Elsken, J. H. Metzen, and F. Hutter, “Efficient multi-objective neural architecture search via lamarckian evolution,” in Proceedings of the International Conference on Learning Representations, 2019.
  * [8] B. Zoph, V. Vasudevan, J. Shlens, and Q. V. Le, “Learning transferable architectures for scalable image recognition,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8697–8710, 2018\.
  * [9] H. Liu, K. Simonyan, and Y. Yang, “Darts: Differentiable architecture search,” in Proceedings of the International Conference on Learning Representations, 2019.
  * [10] B. Wu, X. Dai, P. Zhang, Y. Wang, F. Sun, Y. Wu, Y. Tian, P. Vajda, Y. Jia, and K. Keutzer, “Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019.
  * [11] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam, “Mobilenets: Efficient convolutional neural networks for mobile vision applications,” in arXiv preprint arXiv:1704.04861, 2017.
  * [12] A. Gordon, E. Eban, O. Nachum, B. Chen, H. Wu, T.-J. Yang, and E. Choi, “Morphnet: Fast & simple resource-constrained structure learning of deep networks,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1586–1595, 2018.
  * [13] H. Cai, L. Zhu, and S. Han, “ProxylessNAS: Direct neural architecture search on target task and hardware,” in Proceedings of the International Conference on Learning Representations, 2019.
  * [14] X. Dai, P. Zhang, B. Wu, H. Yin, F. Sun, Y. Wang, M. Dukhan, Y. Hu, Y. Wu, Y. Jia, et al., “Chamnet: Towards efficient network design through platform-aware model adaptation,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019.
  * [15] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, “Mobilenetv2: Inverted residuals and linear bottlenecks,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4510–4520, 2018.
  * [16] F. N. Iandola, S. Han, M. W. Moskewicz, K. Ashraf, W. J. Dally, and K. Keutzer, “Squeezenet: Alexnet-level accuracy with 50x fewer parameters and¡ 0.5 mb model size,” in arXiv preprint arXiv:1602.07360, 2016.
  * [17] H. Pham, M. Y. Guan, B. Zoph, Q. V. Le, and J. Dean, “Efficient neural architecture search via parameter sharing,” in Proceedings of the International Conference on Machine Learning, 2018.
  * [18] S. Xie, H. Zheng, C. Liu, and L. Lin, “Snas: stochastic neural architecture search,” in Proceedings of the International Conference on Learning Representations, 2019.
  * [19] M. Tan, B. Chen, R. Pang, V. Vasudevan, and Q. V. Le, “Mnasnet: Platform-aware neural architecture search for mobile,” in arXiv preprint arXiv:1807.11626, 2018.
  * [20] T.-J. Yang, A. Howard, B. Chen, X. Zhang, A. Go, M. Sandler, V. Sze, and H. Adam, “Netadapt: Platform-aware neural network adaptation for mobile applications,” in Proceedings of the European Conference on Computer Vision (ECCV), pp. 285–300, 2018.
  * [21] Y. He, J. Lin, Z. Liu, H. Wang, L.-J. Li, and S. Han, “Amc: Automl for model compression and acceleration on mobile devices,” in Proceedings of the European Conference on Computer Vision (ECCV), pp. 784–800, 2018.
  * [22] F. Scarselli, M. Gori, A. C. Tsoi, M. Hagenbuchner, and G. Monfardini, “The graph neural network model,” in IEEE Transactions on Neural Networks, vol. 20, pp. 61–80, IEEE, 2009.
  * [23] Y. Li, D. Tarlow, M. Brockschmidt, and R. Zemel, “Gated graph sequence neural networks,” in Proceedings of the International Conference on Learning Representations, 2016.
  * [24] T. N. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks,” in Proceedings of the International Conference on Learning Representations, 2017.
  * [25] C. Zhang, M. Ren, and R. Urtasun, “Graph hypernetworks for neural architecture search,” in arXiv preprint arXiv:1810.05749, 2018.
  * [26] D.-A. Clevert, T. Unterthiner, and S. Hochreiter, “Fast and accurate deep network learning by exponential linear units (elus),” in Proceedings of the International Conference on Learning Representations, 2016.
  * [27] Y. LeCun, L. Bottou, Y. Bengio, P. Haffner, et al., “Gradient-based learning applied to document recognition,” in Proceedings of the IEEE, vol. 86, pp. 2278–2324, 1998.
  * [28] E. Real, A. Aggarwal, Y. Huang, and Q. V. Le, “Regularized evolution for image classifier architecture search,” in Proceedings of the Association for the Advance of Artificial Intelligence Conference on Artificial Intelligence, 2019.
  * [29] A. Krizhevsky and G. Hinton, “Learning multiple layers of features from tiny images,” tech. rep., Citeseer, 2009.
  * [30] I. Sutskever, J. Martens, G. E. Dahl, and G. E. Hinton, “On the importance of initialization and momentum in deep learning.,” ICML (3), vol. 28, no. 1139-1147, p. 5, 2013.
  * [31] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in 2009 IEEE conference on computer vision and pattern recognition, pp. 248–255, Ieee, 2009.
  * [32] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for computer vision,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2818–2826, 2016.
  * [33] T. Tieleman and G. Hinton, “Divide the gradient by a running average of its recent magnitude. coursera neural netw,” Mach. Learn, 2012.
  * [34] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4700–4708, 2017.
  * [35] V. Glezer, V. Ivanoff, and T. Tscherbach, “Investigation of complex and hypercomplex receptive fields of visual cortex of the cat as spatial frequency filters,” in Vision research, vol. 13, pp. 1875–IN6, Elsevier, 1973.
  * [36] H. Xiao, K. Rasul, and R. Vollgraf, “Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms,” in arXiv preprint arXiv:1708.07747, 2017.
  * [37] Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng, “Reading digits in natural images with unsupervised feature learning,” in Proceesings of the NIPS workshop on deep learning and unsupervised feature learning, 2011.