# Homogeneous Artificial Neural Network

Andrey Polyakov Univ. Lille, Inria, CNRS, UMR 9189 CRIStAL, Centralle Lille,
F-59000 Lille, France<EMAIL_ADDRESS>
###### Abstract

The paper proposes an artificial neural network (ANN) being a global
approximator for a special class of functions, which are known as generalized
homogeneous. The homogeneity means a symmetry of a function with respect to a
group of transformations having topological characterization of a dilation. In
this paper, a class of the so-called linear dilations is considered. A
homogeneous universal approximation theorem is proven. Procedures for an
upgrade of an existing ANN to a homogeneous one are developed. Theoretical
results are supported by examples from the various domains (computer science,
systems theory and automatic control).

##  1 Introduction

###  1.1 State of the art

The universal approximation theorems [4], [10], [9] put limits on what
artificial neural networks (ANNs) can theoretically learn. These theorems
guarantee the existence of ANN, which approximates a continuous function on a
compact set with an arbitrary high precision. A training of the ANN is based a
compactly supported data as well, while, the trained ANN may be utilized next
as a predictor of the function value for an input data, which does not belong
to the training set. Sometimes, the new input may be have a rather large
distance even from a convex hull of the training set. In the latter case, the
ANN is utilized as an extrapolator of the function and the approximation
theorems are not applicable. Moreover, the analysis of the extrapolation error
is impossible if there is no information about the function away from the
training set. Therefore, a global extrapolation of a function based on a local
data can be provided only under additional assumption about the class of
functions approximated by ANN. This paper deals with approximation of the so-
called generalized homogeneous functions [24], [12], [2], [21] and introduces
the corresponding homogeneous artificial neural network, which key feature is
a global approximation based on local data.

The generalized homogeneity is a symmetry of an object (a function, a set, a
vector field, etc) with respect to a group of the so-called generalized
dilations. Recall that the standard (classical) homogeneity were introduced by
Leonhard Euler in 18th century as the symmetry of a function $x\mapsto f(x)$
with respect to the standard (isotropic) dilation of its argument
$x\mapsto\lambda x$, namely, $f(\lambda
x)=\lambda^{\nu}f(x),\forall\lambda>0,\forall x$, where $\nu\in\mathbb{R}$ is
the homogeneity degree. To the best of author’s knowledge, generalized
(anisotropic) dilations are studied in systems theory since 1950s (see, e.g.,
[24]), where, instead of multiplication of the argument $x$ on a scalar
$\lambda>0$, some group of transformations having topological characterization
of a dilation is utilized (see, e.g. [13], [12], [11]). In particular, if
$\mathbf{d}(s):\mathbb{V}\mapsto\mathbb{V},s\in\mathbb{R}$ is a one parameter
group of coordinate transformations on a normed vector space $\mathbb{V}$
satisfying $\lim_{s\to\pm\infty}\|\mathbf{d}(s)x\|=e^{\pm\infty},\forall
x\neq\mathbf{0}$ then $\mathbf{d}$ is a dilation in $\mathbb{V}$ and a
functional $h:\mathbb{V}\mapsto\mathbb{R}$ satisfying
$h(\mathbf{d}(s)x)=e^{\nu s}h(x)$ is said to be $\mathbf{d}$-homogeneous of
degree $\nu$. In this framework, the standard dilation corresponds to the
dilation $\mathbf{d}(s)x=e^{s}x,s\in\mathbb{R},x\in\mathbb{V}$.

Generalized homogeneous mathematical objects are studied in various domains of
systems theory [6], [8], [12], [23], [5]. Many models of mathematical physics
[21] are homogeneous in a generalized sense. Homogeneous models also appear as
local approximations of nonlinear systems [1]. The control systems design is
frequently based on generalized homogeneity [16], [20], [22]. This paper deals
with functions $\mathbb{R}^{n}\mapsto\mathbb{R}$, which are symmetric
($\mathbf{d}$-homogeneous) with respect to linear dilations
$\mathbf{d}(s)=e^{sG_{\mathbf{d}}},s\in\mathbb{R}$, where
$G_{\mathbf{d}}\in\mathbb{R}^{n\times n}$ is an anti-Hurwitz matrix111All
eigenvalues have positive real parts. The homogeneity implies certain
equivalence of local and global information, since any local knowledge about a
function can be expanded globally using the dilation symmetry.

The dilation symmetry (in particular, standard homogeneity) is shown to be
useful for analysis and learning of ANNs [18]. However, the dilation symmetry
of the neural network parameters is usually studied, while the present paper
studies a dilation symmetry of input-output relations. The approximation
procedure should take into account the symmetry (if any), otherwise, the
geometry of the mapping $g$ may be lost. As shown in the examples given below,
this may badly impacts extrapolation capabilities of the ANN. For example, the
dilation symmetry (scaling invariance) is important for the pattern
recognition [3], which is expected to be invariant with respect to a zoom
(standard dilation) of the image. The systems theory provides many other
examples of the generalized homogeneous functions.

###  1.2 Basic idea of homogeneous approximation

Following the classical model of a controlled rotational motion of the rigid
body in the 3-D space, the dynamics of the angular velocity of the body is
governed by

$\dot{\omega}(t)=J^{-1}(-\omega\times J\omega+\tau),\quad t\geq 0,$ (1)

where
$\omega(t)=(\omega_{1}(t),\omega_{2}(t),\omega_{3}(t))^{\top}\in\mathbb{R}^{n}$
is the angular velocity in the body frame, $J=J^{\top}\in\mathbb{R}^{n\times
n}$ is the inertia matrix, $\tau(t)\in\mathbb{R}^{3}$ is the external momentum
(e.g., control input) and $\times$ denotes the vector product, i.e.,
$\omega\times J\omega=\left(\begin{smallmatrix}0&-\omega_{3}&\omega_{2}\\\
\omega_{3}&0&-\omega_{1}\\\
-\omega_{2}&\omega_{2}&0\end{smallmatrix}\right)J\omega$. The function
$h:\mathbb{R}^{6}\mapsto\mathbb{R}^{3}$ given by
$h(\omega,\tau)=-J^{-1}(\omega\times J\omega+\tau)$ is homogeneous of degree
$2$ with respect to a generalized dilation:

$h(e^{s}\omega,e^{2s}\tau)=e^{2s}h(\omega,\tau),\quad\forall
s\in\mathbb{R},\forall\omega,\tau\in\mathbb{R}^{3}.$

This means that the right-hand side of the differential equation (1) is
homogeneous. Considering the problem of data-driven approximation the dynamic
equation, if $h_{\epsilon}$ is an ANN approximation of $h$ on a compact
$\Omega\subset\mathbb{R}^{6}$ such that
$|h(\omega,\tau)-h_{\epsilon}(\omega,\tau)|\leq\epsilon$ for all
$(\omega,\tau)\in\Omega$, then
$h^{s}_{\epsilon}(\omega,\tau):=e^{2s}h_{\epsilon}(e^{-s}\omega,e^{-2s}\tau)$
is an approximation of $h$ on the set $\mathbf{d}(s)\Omega$, where
$\mathbf{d}(s)=\mathrm{diag}(e^{s}I_{3},e^{2s}I_{3})$ is the generalized
dilation. Indeed, due to $\mathbf{d}$-homogeneity of $h$, the approximation
error for $h^{s}_{\epsilon}$ admits the estimate
$|h(\omega,\tau)-h^{s}_{\epsilon}(\omega,\tau)|=e^{2s}|h(e^{-s}\omega,e^{-2s}\tau)-h_{\epsilon}(e^{-s}\omega,e^{-2s}\tau)|\leq
e^{2s}\epsilon$ for all $(\omega,\tau)\in\mathbf{d}(s)\Omega$. Since
$s\in\mathbb{R}^{n}$ is an arbitrary number, then the mapping
$h^{s}_{\epsilon}$ can be interpreted as as a homogeneous extrapolation of
$h_{\epsilon}$ away from $\Omega$. This paper utilizes such an extrapolation
for a global ANN approximator design.

###  1.3 Contributions

The paper has two main contributions to the approximation theory by ANNs:

  * •

A $\mathbf{d}$-homogeneous ANN with one hidden level is designed. Such an ANN
is shown to be a global approximator of any continuous
$\mathbf{d}$-homogeneous function on $\mathbb{R}^{n}$. The corresponding
homogeneous approximation theorem is proven.

  * •

A methodology of an upgrade (transformation) of existing well-trained ANN to a
homogeneous one is developed, justified by theoretical proofs and supported by
numerical example. The upgrade assumes that a homogeneous mapping is already
approximated by the conventional ANN locally, but the homogeneity parameters
(such as the homogeneity degree or the dilation group) are partially known or
unknown at all. Based on available data, these parameters are identified and,
next, the existing ANN is homogenized (transformed to a homogeneous ANN) in
order to make it a global approximatior of the homogeneous function. This
procedure can be interpreted as a homogeneous extrapolation of ANN.

###  1.4 Structure of the paper

Preliminaries about generalized homogeneous functions are given in Section 2.
Homogeneous ANNs are introduced in Section 3, where, in particular, the
homogeneous universal approximation theorem is proven. Section 4 is devoted to
upgrading the classical ANN to a homogeneous one. Numerical examples from
computer science, systems theory and automatic control are given in Section 5,
where, in particular, the problem of pattern recognition with scaling
invariance and ANN identification of homogeneous dynamical system (1) are
considered.

###  1.5 Notation

$\mathbb{R}$ is the field of reals;
$\mathbb{R}^{n}_{\mathbf{0}}=\mathbb{R}^{n}\backslash\\{\mathbf{0}\\}$, where
$\mathbf{0}$ is the zero element of a vector space (e.g.,
$\mathbf{0}\in\mathbb{R}^{n}$ means that $\mathbf{0}$ is the zero vector);
$\|\cdot\|$ is a norm in $\mathbb{R}^{n}$ (to be specified later); the matrix
norm for $A\in\mathbb{R}^{n\times n}$ is defined as
$\|A\|=\sup_{x\neq\mathbf{0}}\frac{\|Ax\|}{\|x\|}$; $\lambda_{\min}(P)$ denote
a minimal eigenvalue of a symmetric matrix $P=P^{\top}\in\mathbb{R}^{n\times
n}$; $P\succ 0$ means that the symmetric matrix $P$ is positive definite;
$C(\Omega_{1},\Omega_{2})$ denotes the set of continuous functions
$\Omega_{1}\subset\mathbb{R}^{n}\mapsto\Omega_{2}\subset\mathbb{R}^{m}$;
$C_{c}(\mathbb{R}^{n},\mathbb{R})$ denotes the set of compactly supported
continuous functions; $L^{p}(\mathbb{R}^{n},\mathbb{R})$ is the Lebesgue
space.

##  2 Generalized Homogeneity

###  2.1 Linear Dilation

Let us recall that a family of operators
$\mathbf{d}(s):\mathbb{R}^{n}\mapsto\mathbb{R}^{n}$ with $s\in\mathbb{R}$ is a
one-parameter group if $\mathbf{d}(0)x\\!=\\!x$,
$\mathbf{d}(s)\mathbf{d}(t)x\\!=\\!\mathbf{d}(s+t)x$, $\forall
x\\!\in\\!\mathbb{R}^{n},\forall s,t\\!\in\\!\mathbb{R}$. A group $\mathbf{d}$
is a) continuous if the mapping $s\mapsto\mathbf{d}(s)x$ is continuous,
$\forall x\\!\in\\!\mathbb{R}^{n}$; b) linear if $\mathbf{d}(s)$ is a linear
mapping (i.e., $\mathbf{d}(s)\in\mathbb{R}^{n\times n}$), $\forall
s\in\mathbb{R}$; c) a dilation [12] in $\mathbb{R}^{n}$ if
$\liminf\limits_{s\to+\infty}\|\mathbf{d}(s)x\|=+\infty$ and
$\limsup\limits_{s\to-\infty}\|\mathbf{d}(s)x\|=0$, $\forall x\neq\mathbf{0}$.
Any linear continuous group in $\mathbb{R}^{n}$ admits the representation [19]

$\mathbf{d}(s)=e^{sG_{\mathbf{d}}}=\sum_{j=1}^{\infty}\tfrac{s^{j}G_{\mathbf{d}}^{j}}{j!},\quad
s\in\mathbb{R},\vspace{-2mm}$ (2)

where $G_{\mathbf{d}}\in\mathbb{R}^{n\times n}$ is a generator of
$\mathbf{d}$. A continuous linear group (2) is a dilation in $\mathbb{R}^{n}$
if and only if $G_{\mathbf{d}}$ is an anti-Hurwitz matrix. In this paper we
deal only with continuous linear dilations. A dilation $\mathbf{d}$ in
$\mathbb{R}^{n}$ is i) monotone if the function $s\mapsto\|\mathbf{d}(s)x\|$
is strictly increasing, $\forall x\neq\mathbf{0}$; ii) strictly monotone if
$\exists\beta\\!>\\!0$ such that $\|\mathbf{d}(s)x\|\\!\leq\\!e^{\beta
s}\|x\|$, $\forall s\\!\leq\\!0$, $\forall x\in\mathbb{R}^{n}$.
The following result is the straightforward consequence of the existence of
the quadratic Lyapunov function for stable linear system [17].

###### Corollary 1

A linear continuous dilation in $\mathbb{R}^{n}$ is strictly monotone with
respect to the weighted Euclidean norm $\|x\|=\sqrt{x^{\top}Px}$ with $0\prec
P\in\mathbb{R}^{n\times n}$ if and only if
$PG_{\mathbf{d}}+G_{\mathbf{d}}^{\top}P\succ 0,P\succ 0.$

The standard dilation corresponds to $G_{\mathbf{d}_{1}}=I_{n}$ and
$\mathbf{d}_{1}(s)=e^{s}I_{n}$. The generator of the weighted dilation [24] is
$G_{\mathbf{d}_{2}}\\!=\\!\mathrm{diag}(r_{1},...,r_{n})$.

###  2.2 Canonical homogeneous norm

A continuous positive definite function
$\|\cdot\|_{\mathbf{d}}:\mathbb{R}^{n}\mapsto\mathbb{R}$ satisfying
$\|\pm\mathbf{d}(s)x\|=e^{s}\|x\|_{\mathbf{d}},\forall x\in\mathbb{R}^{n}$ is
usually called a $\mathbf{d}$-homogeneous norm [12], [7], [1]. For the
weighted dilation $\mathbf{d}(s)=\left(\begin{smallmatrix}e^{2s}&0\\\
0&e^{s}\end{smallmatrix}\right)$, a $\mathbf{d}$-homogeneous norm can be
defined, for example, as $\|x\|_{\mathbf{d}}=\sqrt{|x_{1}|}+|x_{2}|$, where
$x=(x_{1},x_{2})^{\top}\in\mathbb{R}^{2}$.

The $\mathbf{d}$-homogeneous norm $\|\cdot\|_{\mathbf{d}}$ does not satisfy
the triangle inequality in the general case, so, formally, this is not even a
semi-norm. However, we would follow the tradition accepted in the systems
sciences and call the function $\|\cdot\|_{\mathbf{d}}$ satisfying the above
property by a “norm”, but, basically, we use this name for the canonical
homogeneous norm (see below) being a norm (in the classical sense) for the
vector space $\mathbb{R}^{n}_{\mathbf{d}}$ homeomorphic to $\mathbb{R}^{n}$.
The space $\mathbb{R}^{n}_{\mathbf{d}}$ has the same set of elements as
$\mathbb{R}^{n}$, but the rules for addition of vectors and multiplication of
a vector by a scalar are modified by means of a special homeomorphism on
$\mathbb{R}^{n}$ (see [21] for more details).

###### Definition 1

Let a linear dilation $\mathbf{d}$ in $\mathbb{R}^{n}$ be continuous and
monotone with respect to a norm $\|\cdot\|$. A function
$\|\cdot\|_{\mathbf{d}}:\mathbb{R}^{n}\mapsto[0,+\infty)$ defined as follows:
$\|\mathbf{0}\|_{\mathbf{d}}=0$ and

$\|x\|_{\mathbf{d}}\\!=\\!e^{s_{x}},\;\text{where}\;s_{x}\in\mathbb{R}:\|\mathbf{d}(-s_{x})x\|\\!=\\!1,\;x\\!\neq\\!\mathbf{0}\vspace{-2mm}$
(3)

is said to be a canonical $\mathbf{d}$-homogeneous norm in $\mathbb{R}^{n}$

For standard dilation $\mathbf{d}(s)\\!=\\!e^{s}I_{n}$ we have
$\|x\|_{\mathbf{d}}\\!=\\!\|x\|$. In other cases, the canonical homogeneous
norm $\|x\|_{\mathbf{d}}$ with $x\neq\mathbf{0}$ is implicitly defined by the
nonlinear algebraic equation (3), which always have a unique solution due to
monotonicity of the dilation. In some particular cases, this implicit equation
has explicit solution.

###### Lemma 1

[21] If a linear continuous dilation $\mathbf{d}$ in $\mathbb{R}^{n}$ is
monotone with respect to a norm $\|\cdot\|$ then 1) the function
$\|\cdot\|_{\mathbf{d}}:\mathbb{R}^{n}\mapsto[0,+\infty)$ implicitly defined
by (3) is single-valued and continuous on $\mathbb{R}^{n}$; 2) there exist
$\sigma_{1},\sigma_{2}\in\mathcal{K}_{\infty}$:

$\sigma_{1}(\|x\|_{\mathbf{d}})\leq\|x\|\leq\sigma_{2}(\|x\|_{\mathbf{d}}),\quad\quad\forall
x\in\mathbb{R}^{n};$ (4)

3) $\|\cdot\|$ is locally Lipschitz continuous on
$\mathbb{R}^{n}\backslash\\{\mathbf{0}\\}$; 4) $\|\cdot\|_{\mathbf{d}}$ is
continuously differentiable on $\mathbb{R}^{n}\backslash\\{\mathbf{0}\\}$
provided that $\|\cdot\|$ is continuously differentiable on
$\mathbb{R}^{n}\backslash\\{\mathbf{0}\\}$ and $\mathbf{d}$ is strictly
monotone.

For the $\mathbf{d}$-homogeneous norm $\|x\|_{\mathbf{d}}$ induced by the
weighted Euclidean norm $\|x\|=\sqrt{x^{\top}Px}$ (see Corollary 1) we have
[21]

$\tfrac{\partial\|x\|_{\mathbf{d}}}{\partial
x}=\|x\|_{\mathbf{d}}\tfrac{x^{\top}\mathbf{d}^{\top}(-\ln\|x\|_{\mathbf{d}})P\mathbf{d}(-\ln\|x\|_{\mathbf{d}})}{x^{\top}\mathbf{d}^{\top}(-\ln\|x\|_{\mathbf{d}})PG_{\mathbf{d}}\mathbf{d}(-\ln\|x\|_{\mathbf{d}})x},\vspace{-2mm}$
(5) $\sigma_{1}(\rho)=\left\\{\begin{smallmatrix}\rho^{\alpha}&\text{ if
}&\rho\leq 1,\\\ \rho^{\beta}&\text{
if}&\rho>1,\end{smallmatrix}\right.\quad\quad\sigma_{2}(\rho)=\left\\{\begin{smallmatrix}\rho^{\beta}&\text{
if }&\rho\leq 1,\\\ \rho^{\alpha}&\text{ if}&\rho>1,\end{smallmatrix}\right.$
(6)

where
$\alpha=\frac{\lambda_{\max}\left(P^{\frac{1}{2}}G_{\mathbf{d}}P^{-\frac{1}{2}}+P^{-\frac{1}{2}}G^{\top}_{\mathbf{d}}P^{\frac{1}{2}}\right)}{2}\geq$
$\beta=\frac{\lambda_{\min}\left(P^{\frac{1}{2}}G_{\mathbf{d}}P^{-\frac{1}{2}}+P^{-\frac{1}{2}}G^{\top}_{\mathbf{d}}P^{\frac{1}{2}}\right)}{2}>0.$

All $\mathbf{d}$-homogeneous norms are equivalent, i.e., there exist
$0<k_{1}\leq k_{2}<+\infty$ such that
$k_{1}\|x\|_{\mathbf{d},1}\leq\|x\|_{\mathbf{d},2}\leq
k_{2}\|x\|_{\mathbf{d},1},\forall x\in\mathbb{R}^{n},$ where
$\|\cdot\|_{\mathbf{d},1}$ and $\|\cdot\|_{\mathbf{d},2}$ are
$\mathbf{d}$-homogeneous norms.

In Section 5 we present an ANN-based approximation of
$\|\cdot\|_{\mathbf{d}}$, which uses the so-called $\mathbf{d}$-homogeneous
projector on the unit sphere

$\pi_{\mathbf{d}}(x)=\mathbf{d}(-\ln\|x\|_{\mathbf{d}})x,\quad
x\neq\mathbf{0}.$

Obviously, $\|\pi_{\mathbf{d}}(x)\|_{\mathbf{d}}=1$ and
$\pi_{\mathbf{d}}(\pi_{\mathbf{d}}(x))=\pi_{\mathbf{d}}(x),\forall
x\neq\mathbf{0}$, so $\pi$ is, indeed, a projector on the unit sphere
$\\{x\in\mathbb{R}^{n}:\|x\|_{\mathbf{d}}=1\\}$. In the case of the canonical
homogeneous norm this sphere coincides with
$S=\\{x\in\mathbb{R}^{n}:\|x\|=1\\}$, where $\|\cdot\|$ is the conventional
norm in $\mathbb{R}^{n}$ that induces $\|\cdot\|_{\mathbf{d}}$ (see Definition
1).

####  2.2.1 Homogeneous functions

Below we study functions being symmetric with respect to a linear dilation
$\mathbf{d}$. The dilation symmetry introduced by the following definition is
known as the generalized homogeneity [24], [12], [23], [2], [21].

###### Definition 2

[12] Let $\mathbf{d}$ be a dilation in $\mathbb{R}^{n}$. A function
$h:\mathbb{R}^{n}\mapsto\mathbb{R}$ is $\mathbf{d}$-homogeneous of degree
$\nu\in\mathbb{R}$ if

$h(\mathbf{d}(s)x)=e^{\nu s}h(x),\quad\quad\forall s\in\mathbb{R},\quad\forall
x\in\mathbb{R}^{n}.\vspace{-2mm}$ (7)

Local behavior of any homogeneous function $h$ can be expanded globally taking
into account the homogeneity degree of the function. The sign of the
homogeneity degree also defines some properties of the function [2].

###### Proposition 1

Let $h:\mathbb{R}^{n}\mapsto\mathbb{R}$ be a $\mathbf{d}$-homogeneous function
of degree $\nu\\!\in\\!\mathbb{R}$, $h\\!\neq\\!\mathbf{0}$ and $\sup_{x\in
S}|h(x)|<+\infty$.

  * •

If $\nu>0$ then $h$ is bounded on any $\mathbf{d}$-homogeneous ball
$B_{\mathbf{d}}(r)=\\{x\in\mathbb{R}^{n}:\|x\|_{\mathbf{d}}\leq r\\}$ with
$r>0$ and

$\lim_{\|x\|\to 0}h(x)=0\quad\text{ and
}\quad\limsup_{\|x\|\to+\infty}|h(x)|=+\infty.$

  * •

If $\nu<0$ then $h$ is bounded on any set $\mathbb{R}^{n}\backslash
B_{\mathbf{d}}(r)$,

$\lim_{\|x\|\to+\infty}h(x)=0\quad\text{ and }\quad\limsup_{\|x\|\to
0}|h(x)|=+\infty.$

  * •

If $\deg_{\mathbf{d}}(h)=0$ then $h$ is uniformly bounded on $\mathbb{R}^{n}$;
moreover, $h\equiv\mathrm{const}$ if $h$ is continuous at $\mathbf{0}$.

  * •

A non-constant homogeneous function $h$ is continuous at zero if and only if
$\deg_{\mathbf{d}}(h)>0$.

Since the identity (7) with $s=-\ln\|x\|_{\mathbf{d}}$ yields

$h(u)=\|x\|^{\nu}_{\mathbf{d}}h(\mathbf{d}(-\ln\|x\|_{\mathbf{d}})x)$

then the properties of $h$ claimed above follow from the inclusion
$\mathbf{d}(-\ln\|x\|_{\mathbf{d}})x\in S$ and from the properties of the
canonical homogeneous norm (see Lemma 1).

Euler homogeneous function theorem is one of fundamental results
characterizing homogeneous mappings. Traditionally it relates a smooth
homogeneous function and its derivative. The following extension of the Euler
homogeneous function theorem also links a locally Lipschitz homogeneous
function with its integral and deals with the generalized homogeneity. Let us
denote

$R(x,x^{*})\\!=\\!\left\\{(y_{1},...,y_{n})^{\\!\top}\\!\\!\in\\!\mathbb{R}^{n}:\underline{x}_{i}\\!\leq\\!y_{i}\\!\leq\\!\overline{x}_{i},i\\!=\\!1,...,n\right\\}\\!,$

where $x\\!=\\!(x_{1},...,x_{n})^{\\!\top}\\!\\!\in\\!\mathbb{R}^{n}$,
$x^{*}\\!=\\!(x_{1}^{*},...,x_{n}^{*})^{\\!\top}\\!\\!\in\\!\mathbb{R}^{n}\\!$
and $\underline{x}_{i}=\min\\{x_{i},x_{i}^{*}\\}$,
$\overline{x}_{i}=\max\\{x_{i},x_{i}^{*}\\}$.

######  Theorem 1 (Euler homogeneous function theorem)

Let a function $h:\mathbb{R}^{n}\mapsto\mathbb{R}^{n}$ be locally Lipschitz
continuous on $\mathbb{R}^{n}\backslash\\{\mathbf{0}\\}$ and
$h(\mathbf{0})=0$. Then the following three claims are equivalent

  * 1)

$h$ is $\mathbf{d}$-homogeneous of degree $\nu\in\mathbb{R}$;

  * 2)

the identity

$\frac{\partial h(x)}{\partial x}G_{\mathbf{d}}x\stackrel{{\scriptstyle
a.e.}}{{=}}\nu h(x),$ (8)

holds almost everywhere on $\mathbb{R}^{n}$, where
$x\in\mathbb{R}^{n}\backslash\\{\mathbf{0}\\}$ and
$G_{\mathbf{d}}\in\mathbb{R}^{n\times n}$ is a generator of the linear
dilation $\mathbf{d}$;

  * 3)

the identity

$\int^{x}_{x^{*}}\\!\\!\\!(\operatorname{trace}(G_{\mathbf{d}})+\nu)h(y)-\\!\\!\\!\sum\limits_{i:x_{i}\neq
x_{i}^{*}}\\!\\!\\!\\!\Delta_{i}(h,y,x,x^{*},,G_{\mathbf{d}})\;dy\\!=\\!0$ (9)

holds for $x\\!=\\!(x_{1},...,x_{i},...,x_{n})^{\\!\top}$,
$x^{*}\\!=\\!(x_{1}^{*},...,x_{i},...,x_{n}^{*})^{\\!\top}$ such that
$R(x,x^{*})\in\mathbb{R}^{n}\backslash\\{\mathbf{0}\\}$, where
$\int^{x}_{x^{*}}=\int^{x_{1}}_{x_{1}^{*}}...\int^{x_{n}}_{x_{n}^{*}}$,
$dy=dy_{n}...dy_{1}$, $i=1,...,n$, $e_{i}=(0,...,1,...,0)^{\top}$ and

$\Delta_{i}(h,y,x,x^{*},G_{\mathbf{d}})=\frac{h(y_{x_{i}})e_{i}^{\top}G_{\mathbf{d}}y_{x_{i}}-h(y_{x^{*}_{i}})e_{i}^{\top}G_{\mathbf{d}}y_{x^{*}_{i}}}{x_{i}-x^{*}_{i}}$

with $y_{x_{i}}\\!=\\!(y_{1},...,x_{i},...,y_{n})^{\top}$,
$y_{x^{*}_{i}}\\!=\\!(y_{1},...,x^{*}_{i},...,y_{n})^{\top}$.

Moreover, if $h$ is differentiable on $\mathbb{R}^{n}$ (resp.,
$\mathbb{R}^{n}\backslash\\{\mathbf{0}\\}$), the identity (8) is defined
everywhere on $\mathbb{R}^{n}$ (resp.,
$\mathbb{R}^{n}\backslash\\{\mathbf{0}\\}$).

Proof.  1)$\Rightarrow$2). Let $h\in\mathcal{H}_{\mathbf{d}}(\mathbb{R}^{n})$
and $\nu=\deg_{\mathbf{d}}(h)\in\mathbb{R}$. Due to homogeneity, we have

$\begin{split}h(x)=&h\left(\mathbf{d}(\ln\|x\|_{\mathbf{d}})\mathbf{d}(-\ln\|x\|_{\mathbf{d}})x\right)\\\
=&\|x\|^{\nu}_{\mathbf{d}}h(\mathbf{d}(-\ln\|x\|_{\mathbf{d}})x),\end{split}\quad\quad\quad\forall
x\neq\mathbf{0}.$

Since, by Rademacher Theorem, the locally Lipschitz continuous function $h$ is
differentiable almost everywhere on $\mathbb{R}^{n}\backslash\\{\mathbf{0}\\}$
then for almost all $x\in\mathbb{R}^{n}\backslash\\{\mathbf{0}\\}$ using
product and chain rules for derivatives we obtain

$\begin{split}\frac{\partial h(x)}{\partial x}\\!\stackrel{{\scriptstyle
a.e.}}{{=}}&\\!\nu\|x\|_{\mathbf{d}}^{\nu-1}h(\mathbf{d}(-\ln\|x\|_{\mathbf{d}})x)\frac{\partial\|x\|_{\mathbf{d}}}{\partial
x}\\\ &+\|x\|^{\nu}_{\mathbf{d}}\frac{\partial h(z)}{\partial
z}\frac{\partial}{\partial
x}\\!\left(\mathbf{d}(-\ln\|x\|_{\mathbf{d}})x\right),\end{split}$

where $z=\mathbf{d}(-\ln\|x\|_{\mathbf{d}})x$. Taking into account the
identities
$\frac{d}{ds}\mathbf{d}(s)=G_{\mathbf{d}}\mathbf{d}(s)=\mathbf{d}(s)G_{\mathbf{d}},\forall
s\in\mathbb{R}$ we conclude

$\tfrac{\partial\left(\mathbf{d}(-\ln\|x\|_{\mathbf{d}})x\right)}{\partial
x}=\mathbf{d}(-\ln\|x\|_{\mathbf{d}})-\tfrac{G_{\mathbf{d}}\mathbf{d}(-\ln\|x\|_{\mathbf{d}})x}{\|x\|_{\mathbf{d}}}\tfrac{\partial\|x\|_{\mathbf{d}}}{\partial
x}.$

Finally, since $\frac{\partial\|x\|_{\mathbf{d}}}{\partial
x}G_{\mathbf{d}}x=\|x\|_{\mathbf{d}}$, $\forall
x\in\mathbb{R}^{n}\backslash\\{\mathbf{0}\\}$ due to the formula (5), then
$\frac{\partial\left(\mathbf{d}(-\ln\|x\|_{\mathbf{d}})x\right)}{\partial
x}G_{\mathbf{d}}x=\mathbf{0}$ and the identity (8) holds.

2)$\Rightarrow$1). For an arbitrary
$x\in\mathbb{R}^{n}\backslash\\{\mathbf{0}\\}$ let us consider the function
$s\mapsto h(\mathbf{d}(s)x)$. Since $\mathbf{d}(s)x\neq\mathbf{0}$ for all
$s\in\mathbb{R}$ then the considered function is locally Lipschitz continuous
on $\mathbb{R}$ and for almost all $x\neq\mathbf{0}$ we have

$\tfrac{dh(\mathbf{d}(s)x)}{ds}\stackrel{{\scriptstyle
a.e}}{{=}}\left.\tfrac{\partial h(z)}{\partial
z}\right|_{z=\mathbf{d}(s)x}\tfrac{d\mathbf{d}(s)x}{ds}\stackrel{{\scriptstyle
a.e}}{{=}}\left.\tfrac{\partial h(z)}{\partial
z}G_{\mathbf{d}}z\right|_{z=\mathbf{d}(s)x}$

due to $\frac{d}{ds}\mathbf{d}(s)=G_{\mathbf{d}}\mathbf{d}(s),\forall
s\in\mathbb{R}$. Using the identity (8) we derive

$\tfrac{dh(\mathbf{d}(s)x)}{ds}\stackrel{{\scriptstyle a.e}}{{=}}\nu
h(\mathbf{d}(s)x).$

Hence, taking into account $h(\mathbf{d}(0)x)=h(x)$ we derive

$h(\mathbf{d}(s)x)=e^{\nu s}h(x),\forall s\in\mathbb{R},\quad\forall
x\neq\mathbf{0}.$

Since by assumption $h(\mathbf{0})=0$ then
$h\in\mathcal{H}_{\mathbf{d}}(\mathbb{R}^{n})$.

2)$\Leftrightarrow$3) Since we have $\frac{\partial h}{\partial
x}G_{\mathbf{d}}x\stackrel{{\scriptstyle
a.e.}}{{=}}\sum_{i=1}^{n}\frac{\partial h}{\partial
x_{i}}e_{i}^{\top}G_{\mathbf{d}}x=h(x)$ then, taking into account local
Lipschitz continuity of $h$ on $\mathbb{R}^{n}\backslash\\{\mathbf{0}\\}$ we
conclude that the identity (8) is equivalent

$\sum_{i=1}^{n}\int_{x_{1}^{*}}^{x_{1}}...\int^{x_{n}}_{x_{n}^{*}}\frac{\partial
h}{\partial
y_{i}}e_{i}^{\top}G_{\mathbf{d}}y\;dy_{1}...dy_{n}=\nu\int_{x^{*}}^{x}h(y)\,dy$

for all $x,x^{*}\in\mathbb{R}^{n}$ such that
$R(x,x^{*})\in\mathbb{R}^{n}\backslash\\{\mathbf{0}\\}$. Changing the
integration order

$\int_{x_{1}^{*}}^{x_{1}}\\!\\!\\!\\!\\!...\\!\int^{x_{n}}_{x_{n}^{*}}\\!\\!\\!\\!...dy=\int_{x_{1}^{*}}^{x_{1}}\\!\\!\\!\\!\\!...\\!\int_{x_{i-1}^{*}}^{x_{i-1}}\\!\\!\int_{x_{i+1}^{*}}^{x_{i+1}}\\!\\!\\!\\!\\!\\!\\!...\\!\int^{x_{n}}_{x_{n}^{*}}\\!\\!\\!\int_{x_{i}^{*}}^{x_{i}}\\!\\!\\!\\!...dy_{i}dy_{n}dy_{n-1}...$

and using the integration by parts

$\int_{x_{i}^{*}}^{x_{i}}\\!\tfrac{\partial h}{\partial
y_{i}}e_{i}^{\top}\\!G_{\mathbf{d}}y\;dy_{i}=\left.h(y)e^{\top}_{i}\\!G_{\mathbf{d}}y\right|^{x_{i}}_{x_{i}^{*}}-\int_{x_{i}^{*}}^{x_{i}}\\!\\!\\!h(y)\tfrac{\partial
e_{i}^{\top}\\!G_{\mathbf{d}}y}{\partial y_{i}}\;dy_{i}$

we derive

$\int_{x_{1}^{*}}^{x_{1}}\\!\\!\\!\\!\\!...\\!\int^{x_{n}}_{x_{n}^{*}}\\!\\!\\!\tfrac{\partial
h}{\partial y_{i}}e_{i}^{\top}G_{\mathbf{d}}ydy=$
$\int_{x_{1}^{*}}^{x_{1}}\\!\\!\\!\\!\\!\\!...\\!\int_{x_{i\\!-\\!1}^{*}}^{x_{i\\!-\\!1}}\\!\\!\\!\int_{x_{i\\!+\\!1}^{*}}^{x_{i\\!+\\!1}}\\!\\!\\!\\!\\!\\!\\!\\!...\\!\int^{x_{n}}_{x_{n}^{*}}\\!\\!\left.h(y)e^{\top}_{i}\\!G_{\mathbf{d}}y\right|^{x_{i}}_{x_{i}^{*}}dy_{n}...dy_{i\\!+\\!1}dy_{i\\!-\\!1}...dy_{1}-$
$\int_{x_{1}^{*}}^{x_{1}}\\!\\!\\!\\!\\!\\!...\\!\\!\int_{x_{i\\!-\\!1}^{*}}^{x_{i\\!-\\!1}}\\!\\!\\!\int_{x_{i\\!+\\!1}^{*}}^{x_{i\\!+\\!1}}\\!\\!\\!\\!\\!\\!\\!\\!...\\!\\!\int^{x_{n}}_{x_{n}^{*}}\\!\\!\\!\\!\int_{x_{i}^{*}}^{x_{i}}\\!\\!\\!\\!\\!h(y)e_{i}^{\top}\\!G_{\mathbf{d}}e_{i}\,dy_{i}dy_{n}...dy_{i\\!+\\!1}dy_{i\\!-\\!1}...dy_{1}.$

Hence, using the identity
$\frac{1}{x_{i}-x_{i}^{*}}\int^{x_{i}}_{x_{i}^{*}}1dy_{i}=1$ we derive the
formula (9).

The identity (9) is utilized below for identification of the generator
$G_{\mathbf{d}}$ of a dilation group $\mathbf{d}$ by means of ANNs.

##  3 Homogeneous ANNs

The statements of universal approximation theorems [4], [10], [9] vary
dependently of the restriction to an activation function
$\sigma:\mathbb{R}\mapsto\mathbb{R}$ of ANN. To the best of author’s
knowledge, the weakest restriction to $\sigma$ is given by the following
theorem.

######  Theorem 2 (Universal approximation theorem, [15])

Let $\sigma\\!\in\\!C(\mathbb{R},\mathbb{R})$ be a non-polynomial function
and $\Omega\subset\mathbb{R}^{n}$ be a compact set. If
$g:\mathbb{R}^{n}\mapsto\mathbb{R}$ is a continuous function then for any
$\epsilon>0$ there exists $N\in\mathbb{N},A\in\mathbb{R}^{N\times
n},b\in\mathbb{R}^{N}$ and $C\in\mathbb{R}^{1\times N}$ such that

$|g(x)-g_{\epsilon}(x)|\leq\epsilon,\quad\forall x\in\Omega,$ (10)
$g_{\epsilon}(x)=C\sigma(Ax+b),$ (11)

where the function $\sigma$ in the above formula is applied in the component-
wise manner.

The function $g_{\epsilon}$ given by (11) is an ANN with one hidden layer
(also known as Shallow ANN). The above theorem shows that the considered ANN
is the universal approximator of any continuous mapping on a compact. For
simplicity, all theoretical results are proven below for scalar-valued
functions $\mathbb{R}^{n}\mapsto\mathbb{R}$. The extension to the vector-
valued functions $\mathbb{R}^{n}\mapsto\mathbb{R}^{m}$ is straightforward.

Let us consider, first, an unknown $\mathbf{d}$-homogeneous mappings having a
known homogeneity degree. We introduce the $\mathbf{d}$-homogeneous artificial
neural network being a global universal approximator of a continuous
$\mathbf{d}$-homogeneous function.

###  3.1 Homogeneous universal approximation theorem

In the case of homogeneous function, the universal approximation theorem can
be revisited as follows.

###### Theorem 3

Let $\mathbf{d}$ be a linear continuous dilation in $\mathbb{R}^{n}$. Let
$\|\cdot\|_{\mathbf{d}}$ be an arbitrary (explicit or implicit)
$\mathbf{d}$-homogeneous norm in $\mathbb{R}^{n}$. Let $\sigma\in
C(\mathbb{R},\mathbb{R})$ be a non-polynomial function.

If $h\in C(\mathbb{R}^{n}\backslash\\{\mathbf{0}\\},\mathbb{R})$ is a
$\mathbf{d}$-homogeneous function of degree $\nu\in\mathbb{R}$ then for any
$\epsilon>0$ there exist $N\in\mathbb{N}$, $A\in\mathbb{R}^{N\times n}$,
$b\in\mathbb{R}^{N}$ and $C\in\mathbb{R}^{1\times N}$ such that

$|h(x)-h_{\varepsilon}(x)|\leq\epsilon\|x\|_{\mathbf{d}}^{\nu},\quad\forall
x\in\mathbb{R}^{n}\backslash\\{\mathbf{0}\\},$ (12)
$h_{\epsilon}(x)=\|x\|_{\mathbf{d}}^{\nu}C\sigma(A\mathbf{d}(-\ln\|x\|_{\mathbf{d}})x+b),\quad$
(13)

where the function $\sigma$ in the above formula is applied in the component-
wise manner.

Proof.  By definition, the $\mathbf{d}$-homogeneous norm
$\|\cdot\|_{\mathbf{d}}$ is a continuous positive definite function satisfying
$\|\mathbf{d}(s)x\|_{\mathbf{d}}=e^{s}\|x\|_{\mathbf{d}}$. Then
$\|\cdot\|_{\mathbf{d}}$ is radially unbounded and the set

$S_{\mathbf{d}}=\\{x\in\mathbb{R}^{n}:\|x\|_{\mathbf{d}}=1\\}$

is a compact such that $\mathbf{0}\notin S_{\mathbf{d}}$. Since
$\|\mathbf{d}(-\ln\|x\|_{\mathbf{d}})x\|_{\mathbf{d}}=e^{-\ln\|x\|_{\mathbf{d}}}\|x\|_{\mathbf{d}}=1$
for any $x\neq 0$ then
$\mathbf{d}(-\ln\|x\|_{\mathbf{d}})x\\!\in\\!S_{\mathbf{d}}$. Let $\|\cdot\|$
be a norm in $\mathbb{R}^{n}$ and

$\begin{split}r_{1}:=&\inf_{x\in\mathbb{R}^{n}\backslash\\{\mathbf{0}\\}}\|\mathbf{d}(-\ln\|x\|_{\mathbf{d}})x\|>0,\\\
r_{2}:=&\sup_{x\in\mathbb{R}^{n}\backslash\\{\mathbf{0}\\}}\|\mathbf{d}(-\ln\|x\|_{\mathbf{d}})x\|\in[r_{1},+\infty).\end{split}$

Let us consider the set

$\Omega=\\{y\in\mathbb{R}^{n}:r_{1}\leq\|y\|\leq r_{2}\\}.$

By construction, the set $\Omega$ is a compact and $\mathbf{0}\notin\Omega$.
Since the function $h$ is continuous on $\Omega$, then, by Theorem 2, for any
$\epsilon>0$ there exists $N\in\mathbb{N}$, $A\in\mathbb{R}^{N\times n}$,
$b\in\mathbb{R}^{N}$ and $C\in\mathbb{R}^{1\times N}$ such that

$\sup_{y\in\Omega}\left|h(y)-C\sigma(Ay+b)\right|\leq\epsilon.$

Since, due to $\mathbf{d}$-homogeneity of $h$, we have

$h(x)=\|x\|^{\nu}_{\mathbf{d}}h(\underbrace{\mathbf{d}(-\ln\|x\|_{\mathbf{d}})x}_{=y\in\Omega}),\quad\forall
x\neq\mathbf{0},$

then the estimate (12) holds for $h_{\epsilon}$ given by (13).

The key feature of the homogeneous ANN is a global approximation based on
local measurements. Indeed, by the formula (13) the homogeneous ANN uses the
conventional ANN $C\sigma(A\pi_{\mathbf{d}}+b)$ with the input
$\pi_{\mathbf{d}}=\mathbf{d}(-\ln\|x\|_{\mathbf{d}})x$ being compactly
supported on the unit sphere
$\\{\pi\in\mathbb{R}^{n}:\|\pi\|_{\mathbf{d}}=1\\}$ in $\mathbb{R}^{n}$. The
classical universal approximation theorem (see Theorem 2) guarantees the error
estimate (10) only on this compact. Due to homogeneity, any local estimate of
the homogeneous function can be extended globally (see (12)). The
approximation error in this case depends of the norm of the vector $x$ and the
homogeneity degree $\nu$ of the function $h$. If $\nu>0$ then, due to
Proposition 1, $h$ is continuous at $\mathbf{0}$ and the above error estimate
holds at zero as well. For $\nu<0$ the function $h$ is discontinuous and
unbounded at $\mathbf{0}$. In this case, the approximation error may grow to
infinity as $x\to\mathbf{0}$. For $\nu=0$ the error is estimated uniformly on
$\mathbb{R}^{n}$.

Usually algorithms of ANN learning optimize the parameters $A,b,C$ based on
available measurements

$y_{j}=h(x_{j}),\quad j=1,...,M,$

where $M$ is a number of experiments. Mathematically, the goal of the ANN
learning is to minimize a norm of the residual

$e_{j}=y_{j}-\|x_{j}\|_{\mathbf{d}}^{\nu}C\sigma(A\mathbf{d}(-\ln\|x_{j}\|_{\mathbf{d}})x_{j}+b).$
(14)

It is worth stressing that the homogeneous ANN has structurally the same
dependence on the parameters $A,b,C$. So, all conventional learning algorithms
are applicable for its training. Their study goes out of the scope of this
paper since, due to the similarity in structures of ANN and homogeneous ANN,
all main advantages and disadvantages of the corresponding methods will be the
same in our case. The aim of this paper is to demonstrate that the use of
dilation symmetry in ANN design may improve the approximation precision of ANN
provided that the mapping, which needs to be approximated, is homogeneous.

###  3.2 Homogenization of ANN with known homogeneity parameters

By the universal approximation theorem, the ANN is a local approximator of a
function, but the homogeneous ANN is a global approximator of a homogeneous
function. The transformation of the ANN to a homogeneous ANN can be
interpreted as global homogeneous extrapolation of the ANN. In this
subsection, we consider the simplest case, when the homogeneity parameters
($G_{\mathbf{d}}$ and $\nu$) are assumed to be known. In the next section, the
problem of homogeneous extrapolation is studied under parametric uncertainty.

We assume that a conventional ANN (11) is already well trained such that the
approximation error (10) is somehow minimized. The linear dilation and the
homogeneity degree are assumed to be known. We need to transform the classical
ANN (11) to the homogeneous ANN (13).

###### Proposition 2

Let $\Omega\subset\mathbb{R}^{n}$ be a compact set. Let $g_{\epsilon}$ be ANN
(11) approximating a function $h\in
C(\mathbb{R}^{n}\backslash\\{\mathbf{0}\\},\mathbb{R})$ on the compact
$\Omega\subset\mathbb{R}^{n}$ with an error $\epsilon>0$, i.e., the inequality
(10) is fulfilled for $g=h$ . Let the function $h$ be $\mathbf{d}$-homogeneous
of degree $\nu\in\mathbb{R}$. If there exists a norm $\|\cdot\|$ in
$\mathbb{R}^{n}$ such that

$S=\\{x\in\mathbb{R}^{n}:\|x\|=1\\}\subset\operatorname{int}\Omega$ (15)

and the dilation $\mathbf{d}$ be monotone with respect to $\|\cdot\|$, then
the homogeneous ANN (13) with parameters $A,b,C$ taken from the conventional
ANN (11) satisfies the error estimate (12), where $\|\cdot\|_{\mathbf{d}}$ is
the canonical homogeneous norm induced by $\|\cdot\|$.

Proof.  By construction the homogeneous ANN $h_{\epsilon}$ coincides with the
original ANN $h_{\epsilon}$ on the unit sphere, i.e.,

$h_{\epsilon}(y)=g_{\epsilon}(y),\quad\forall y\in S.$

This means that

$|g(y)-g_{\epsilon}(y)|=|h(y)-h_{\epsilon}(y)|\leq\epsilon,\quad\forall y\in
S.$

Since $\|\cdot\|_{\mathbf{d}}$ is a canonical homogeneous norm induced by the
norm $\|\cdot\|$ then $y=\mathbf{d}(-\ln\|x\|_{\mathbf{d}})x\in S$ for all
$x\in\mathbb{R}^{n}\backslash\\{\mathbf{0}\\}$ and using the homogeneity we
derive

$\tfrac{|h(x)-h_{\epsilon}(x)|}{\|x\|_{\mathbf{d}}^{-\nu}}\leq|h(\mathbf{d}(-\ln\|x\|_{\mathbf{d}})x)-h_{\epsilon}(-\ln\|x\|_{\mathbf{d}})x)|\leq\epsilon.$

The case of homogeneous vector field can be analyzed similarly.

The above proposition shows that the conventional ANN can be straightforwardly
upgraded to the homogeneous ANN provided that the information about the
dilation symmetry of the function $h$ is available (e.g., from physics). The
idea of the transformation/upgrade is very simple: we just assign a unit
sphere $S\subset\Omega$ and expand the values of the $g_{\epsilon}$ from the
unit sphere to the whole space $\mathbb{R}^{n}$ by means of the dilation.
Notice that with such a homogeneous extrapolation the local error estimate
(10) (being uniform on $\Omega$) is converted to the global
$\|x\|_{\mathbf{d}}$-dependent estimate (12) on $\mathbb{R}^{n}$. The similar
upgrade can be proposed for a function
$h:\mathcal{D}\subset\mathbb{R}^{n}\mapsto\mathbb{R}$ defined on a
$\mathbf{d}$-homogeneous cone222A set $\mathcal{D}$ is said to be a
$\mathbf{d}$-homogeneous cone if $\mathbf{d}(s)\mathcal{D}\subset\mathcal{D}$
for all $s\in\mathbb{R}$. $\mathcal{D}$. The condition (15) becomes
$S\cap\mathcal{D}\subset\Omega\cap\mathcal{D}$ in this case.

##  4 Data-Driven Dilation Symmetry

In this section we continue the study of a possibility of transformation of an
existing artificial neural network to a homogeneous one. We assume that an ANN
is given by $g_{\epsilon}:\mathbb{R}^{n}\mapsto\mathbb{R}^{m}$ such that

$g_{\epsilon}(x)=C\sigma(Ax+b)$

approximates a function $g\in
C(\mathbb{R}^{n}\backslash\\{\mathbf{0}\\},\mathbb{R})$ on a compact
$\Omega\subset\mathbb{R}^{n}\backslash\\{\mathbf{0}\\}$, where
$A\in\mathbb{R}^{N\times n},b\in\mathbb{R}^{N},C\in\mathbb{R}^{1\times N}$.
The index $\epsilon>0$ indicates that the approximation precision of ANN (see
Theorem 2) is given by

$\sup_{x\in\Omega}|g(x)-g_{\epsilon}(x)|\leq\epsilon,\quad\forall x\in\Omega.$

The ANN is supposed to be well trained to approximate a homogeneous function
$g=h$ on a compact $\Omega$, i.e., $\epsilon>0$ is assumed to be small enough.
But now we consider the case when some (or any) information about dilation
symmetry of the function $h$ is unavailable, so, first, the parameters of the
dilation group and/or the homogeneity degree have to be identified using
$g_{\epsilon}$, next, a homogeneous ANN has to be designed.

###  4.1 Practical homogeneity degree

Let us study the problem of identification of an unknown homogeneity degree of
the function $h$ under the assumption that the linear dilation
$\mathbf{d}(s)=e^{sG_{\mathbf{d}}},s\in\mathbb{R}$ is known.

By Definition 2, the scalar-valued function
$h:\mathbb{R}^{n}\mapsto\mathbb{R}$ is $\mathbf{d}$-homogeneous of degree
$\nu\in\mathbb{R}$ if

$h(e^{sG_{\mathbf{d}}}x)=e^{\nu s}h(x),\quad\forall
x\in\mathbb{R}^{n},\quad\forall s\in\mathbb{R},$ (16)

where $\mathbf{d}(s)=e^{sG_{\mathbf{d}}}$ is a linear dilation in
$\mathbb{R}^{n}$. If $g_{\epsilon}$ approximates a $\mathbf{d}$-homogeneous
function $g=h$ on $\Omega$ with a high enough precision then we should have

$g_{\epsilon}(\mathbf{d}(s)x)\approx e^{\nu s}g_{\epsilon}(x),\quad
x\in\Omega,\quad s\in\mathbb{R}:\mathbf{d}(s)x\in\Omega.$

Let a norm $\|\cdot\|$ in $\mathbb{R}^{n}$ and the set $\Omega$ be such that
$S=\\{x\in\mathbb{R}^{n}:\|x\|=1\\}\subset\Omega$ and the dilation
$\mathbf{d}$ be monotone with respect to $\|\cdot\|$. In this case, due to
$\mathbf{d}(-\ln\|x\|_{\mathbf{d}})x\in S$, the above approximate identity
implies

$g_{\epsilon}(\mathbf{d}(-\ln\|x\|_{\mathbf{d}})x)\approx\|x\|_{\mathbf{d}}^{-\nu}g_{\epsilon}(x),\quad\forall
x\in\Omega,$ (17)

where $\|\cdot\|_{\mathbf{d}}$ is the canonical homogeneous norm induced by
the norm $\|\cdot\|$. Hence, for $x\neq\mathbf{0}$ we have

$\nu\approx\frac{1}{\ln\|x\|_{\mathbf{d}}}\ln\frac{g_{\epsilon}(x)}{g_{\epsilon}(\mathbf{d}(-\ln\|x\|_{\mathbf{d}})x)}$
(18)

provided that

$0<\frac{g_{\epsilon}(x)}{g_{\epsilon}(\mathbf{d}(-\ln\|x\|_{\mathbf{d}})x)}<+\infty.$
(19)

Notice that the implication

$h(x)\neq 0\quad\Rightarrow\quad
0<\frac{h(x)}{h(\mathbf{d}(s)x)}<+\infty,\quad\forall s\in\mathbb{R}$

is necessary for a function $h$ to be homogeneous. Indeed, if $h(x)\neq 0$ and
$\frac{h(x)}{h(-\mathbf{d}(\ln\|x\|_{\mathbf{d}})x)}<0$ then the identity (16)
is impossible and $h$ is surely non-homogeneous. Therefore, the condition (19)
can be utilized for detection of the non-homogeneity of $g_{\epsilon}$. To
decrease an impact of the approximation error on a decision about non-
homogeneity of $g_{\epsilon}$, we restrict the domain in (19) to the compact
set

$K_{\delta}=\left\\{x\in\Omega:\begin{smallmatrix}\delta\leq\|x\|_{\mathbf{d}}\leq
1,\\\
\min\\{|g_{\epsilon}(x)|,|g_{\epsilon}(\pi_{\mathbf{d}}(x))|\\}\geq\delta\end{smallmatrix}\right\\},$
(20)

where $\pi_{\mathbf{d}}(x)=\mathbf{d}(-\ln\|x\|_{\mathbf{d}})x$ is the
$\mathbf{d}$-homogeneous projector of $x\neq\mathbf{0}$ on the unit sphere $S$
and $\delta>0$ is small enough. In this case, if

$\frac{g_{\epsilon}(x)}{g_{\epsilon}(\pi_{\mathbf{d}}(x))}>0,\quad\forall x\in
K_{\delta}$ (21)

then the practical homogeneity degree of the function $g_{\epsilon}$ can be
defined as

$\nu_{\epsilon}=\frac{1}{M_{\delta}}\int_{K_{\delta}}\frac{1}{\ln\|x\|_{\mathbf{d}}}\ln\frac{g_{\epsilon}(x)}{g_{\epsilon}(\pi_{\mathbf{d}}(x))}dx$
(22)

provided that $\epsilon>0$ is small enough, where
$M_{\delta}=\int_{K_{\delta}}dx.$ If the condition (21) is not fulfilled then
$g_{\epsilon}$ is not an approximation of homogeneous function or the
approximation precision $\epsilon>0$ is not enough to detect
$\mathbf{d}$-homogeneity.

###### Proposition 3

Let $g_{\epsilon}$ be ANN (11) approximating a non-zero continuous function
$h\in C(\mathbb{R}^{n}\backslash\\{\mathbf{0}\\},\mathbb{R})$ on a compact
$\Omega\subset\mathbb{R}^{n}$ with an error $\epsilon>0$, i.e., the inequality
(10) is fulfilled for $g=h$. Let the function $h$ be $\mathbf{d}$-homogeneous
of degree $\nu\in\mathbb{R}$. If there exists a norm $\|\cdot\|$ in
$\mathbb{R}^{n}$ such that

$S=\\{x\in\mathbb{R}^{n}:\|x\|=1\\}\subset\operatorname{int}\Omega$ (23)

and the dilation $\mathbf{d}$ is monotone with respect to $\|\cdot\|$, then
for sufficiently small $\delta>0$ it holds

$\nu_{\epsilon}\to\nu\quad\text{ as }\quad\epsilon\to 0$ (24)

where $\nu_{\epsilon}\in\mathbb{R}$ is given by (22).

Proof.  On the one hand, since $h$ is $\mathbf{d}$-homogeneous, then, for a
sufficiently small $\delta>0$, the compact set
$K_{\delta}\subset\Omega\backslash\\{\mathbf{0}\\}$ is non-empty. Since $h$ is
$\mathbf{d}$-homogeneous then
$\nu=\frac{1}{\|x\|_{\mathbf{d}}}\ln\frac{h(x)}{h(\mathbf{d}(-\ln\|x\|_{\mathbf{d}})x)}$
for all $x\in K_{\delta}$ and

$\nu=\frac{1}{M_{\delta}}\int_{K_{\delta}}\frac{1}{\ln\|x\|_{\mathbf{d}}}\ln\frac{h(x)}{h(\pi_{\mathbf{d}}(x))}dx.$

On the other hand, by definition $\nu_{\epsilon}$, we have

$\nu_{\epsilon}=\frac{1}{M_{\delta}}\int_{K_{\delta}}\frac{1}{\ln\|x\|_{\mathbf{d}}}\ln\frac{h(x)+o(x,\epsilon)}{h(\pi_{\mathbf{d}}(x)))+o(\pi_{\mathbf{d}}(x)),\epsilon)}dx,$

where $\sup_{y\in\Omega}|o(y,\epsilon)|\leq\epsilon$ (see (10)). Since $h$ is
assumed to be nonzero and
$\pi_{\mathbf{d}}(x)=\mathbf{d}(-\ln\|x\|_{\mathbf{d}})x\in S$ then
$|h(\pi_{\mathbf{d}}(x))|\geq\underline{h}=\inf_{y\in S\cap
K_{\delta}}h(y)>0$. This implies that

$\tfrac{1}{\|x\|_{\mathbf{d}}}\ln\tfrac{g_{\epsilon}(x)}{g_{\epsilon}(\pi_{\mathbf{d}}(x))}\to\tfrac{1}{\|x\|_{\mathbf{d}}}\ln\tfrac{h(x)}{h(\pi_{\mathbf{d}}(x))}\quad\text{
as }\quad\epsilon\to 0$ (25)

uniformly on any compact from $\Omega\cap
K_{\delta}\backslash\\{\mathbf{0}\\}$, so $\nu_{\epsilon}\to\nu$ as
$\epsilon\to 0$.

The practical homogeneity degree is still well-defined by the formula (22)
provided that the practical “necessary” condition of homogeneity (21) is
fulfilled. The formula (22) defines the practical homogeneity degree as a mean
value of some function. This value can also be approximated as follows

$\nu_{\epsilon}\approx\frac{1}{M}\sum_{i=1}^{M}\frac{1}{\ln\|x_{i}\|_{\mathbf{d}}}\ln\frac{g_{\epsilon}(x_{i})}{g_{\epsilon}(\pi_{\mathbf{d}}(x_{i}))},$
(26)

where the points $x_{i}\in K_{\delta}\cap\Omega$ are assumed to be uniformly
distributed in $K_{\delta}\cap\Omega$ and $M\in\mathbb{N}$ is a large enough
number.

###  4.2 Practical linear dilation

Let us study the identification problem of a dilation symmetry (if any) for
the ANN $g_{\epsilon}$ which approximates a continuous function $g$. It is
easy to see that if a function $g$ is $\mathbf{d}$-homogeneous of degree $\nu$
(i.e., $g=h$) then, for any $\gamma>0$ this function is
$\tilde{\mathbf{d}}$-homogeneous of the degree $\tilde{\nu}=\gamma\nu$ with
respect to the dilation $\tilde{\mathbf{d}}(s)=\mathbf{d}(\gamma
s),s\in\mathbb{R}.$ So, the identification of the dilation symmetry of

$g_{\epsilon}$

can be reduced to identification of the generator
$G_{\mathbf{d}}\in\mathbb{R}^{n\times n}$ only, while the homogeneity degree
can always be assigned to one of the following constants $\\{-1,0,1\\}$. In
other words, if the sign of the homogeneity degree is unknown, then the three
identification problems with $\nu=-1$, $\nu=0$ and $\nu=1$ has to be solved in
order to identify a dilation symmetry of $g_{\epsilon}$. Without loss of
generality we assume below that the homogeneity degree $\nu$ of $g_{\epsilon}$
is known or fixed to one of the following numbers $\\{-1,0,1\\}$. We just need
to identify an unknown generator $G_{\mathbf{d}}\in\mathbb{R}^{n\times n}$ of
the dilation $\mathbf{d}$.

By Theorem 1, for any locally Lipschitz continuous (on
$\mathbb{R}^{n}\backslash\\{\mathbf{0}\\}$) $\mathbf{d}$-homogeneous function
$h$, we have

$\int^{x}_{x^{*}}\\!(\operatorname{trace}(G_{\mathbf{d}})+\nu)h(y)-\\!\sum\limits_{i:x_{i}\neq
x_{i}^{*}}\\!\\!\\!\Delta_{i}(h,y,x,x^{*},,G_{\mathbf{d}})\;dy\\!=\\!0$ (27)

for all $x,x^{*}\in\\!\mathbb{R}^{n}$ such that
$R(x,x^{*})\in\mathbb{R}^{n}\backslash\\{\mathbf{0}\\}$.

If $g_{\epsilon}$ approximates a $\mathbf{d}$-homogeneous function $h$ of
degree $\nu$ with a rather high precision $\epsilon$ on a compact $\Omega$
then the identity (27) implies that

$\int^{x}_{x^{*}}\\!\\!(\operatorname{trace}(G)+\nu)g_{\epsilon}(y)-\\!\\!\sum\limits_{i:x_{i}\neq
x_{i}^{*}}\\!\\!\\!\Delta_{i}(g_{\epsilon},y,x,x^{*},G_{\mathbf{d}})\;dy\approx
0.$ (28)

The only unknown parameter $G_{\mathbf{d}}\in\mathbb{R}^{\times n}$ is
involved in above approximate identity in the affine manner. Its
identification can be based on a quadratic optimization.

Let $\Omega$ have a non-empty interior. For $\delta\in(0,1)$ let
$\Omega_{\delta}\subset\operatorname{int}\Omega$ be such that

$\Omega_{\delta}\dot{+}B_{\delta}\subset\Omega\quad\text{ and
}\quad\Omega_{\delta}\cap B_{\delta}=\emptyset,$ (29)

where
$B_{\delta}=\\{x=(x_{1},...,x_{n})^{\top}\in\mathbb{R}^{n}:|x_{i}|\leq\delta,i=1,...,n\\}$.
Homogeneous functions may have singularity at zero, so the condition
$\Omega_{\delta}\cap B_{\delta}=\emptyset$ exclude this potential singularity
point. Let us introduce

$G_{\epsilon}\in\operatorname{argmin}_{G\in\Xi}J_{\epsilon}(G),$ (30)

where $\Xi\subset\mathbb{R}^{n\times n}$ is a set of admissible $n\times n$
matrices and

$J_{\epsilon}(G)\\!=\\!\frac{1}{M_{\delta}}\int_{\Omega_{\delta}}\int_{B_{\delta}}\mathcal{J}^{2}_{\epsilon}(x^{*}+z,x^{*},G)\,dzdx^{*},$
(31)
$\mathcal{J}_{\epsilon}(x,x^{*}\\!,G)\\!=\\!\\!\int^{x}_{x^{*}}\\!\\!\\!(\operatorname{trace}(G)+\nu)g_{\epsilon}(y)-\\!\\!\\!\sum\limits_{i:x_{i}\neq
x_{i}^{*}}\\!\\!\\!\\!\Delta_{i}(g_{\epsilon},y,x,x^{*},G_{\mathbf{d}})\;dy,$

and $M_{\delta}=\int_{\Omega_{\delta}}dx\cdot\int_{B_{\delta}}dz.$ If
$G_{\epsilon}$ is anti-Hurwitz then, in the view of the following theorem, it
can be interpreted as a generator of the practical dilation in
$\mathbb{R}^{n}$.

###### Theorem 4

Let $\Xi\subset\mathbb{R}^{n\times n}$ be a class of admissible generators of
linear dilations. Let $g_{\epsilon}$ be an ANN (11) approximating a function
$h:\mathbb{R}^{n}\mapsto\mathbb{R}$ on a compact $\Omega\subset\mathbb{R}^{n}$
with an error $\epsilon>0$, i.e., the inequality (10) is fulfilled for $g=h$.
If $h$ is locally Lipschitz continuous on
$\mathbb{R}^{n}\backslash\\{\mathbf{0}\\}$ and $\mathbf{d}$-homogeneous of
degree $\nu\in\mathbb{R}$ with respect to a linear dilation
$\mathbf{d}(s)=e^{G_{\mathbf{d}}s},s\in\mathbb{R}^{n}$ with
$G_{\mathbf{d}}\in\Xi$, then

$\min_{G\in\Xi}J_{\epsilon}(G)\to 0\quad\text{as}\quad\epsilon\to 0.$

Proof.  By Theorem 1, we have

$\int^{x}_{x^{*}}\\!(\operatorname{trace}(G_{\mathbf{d}})+\nu)h(y)-\\!\\!\sum\limits_{i:x_{i}\neq
x_{i}^{*}}\\!\\!\\!\Delta_{i}(h,y,x,x^{*},G_{\mathbf{d}})\;dy=0$

for all $x,x^{*}\in\mathbb{R}^{n}\backslash\\{\mathbf{0}\\}$ such that
$R(x,x^{*})\in\mathbb{R}^{n}\backslash\\{\mathbf{0}\\}$. Since
$g_{\epsilon}(x)\to h(x)$ as $\epsilon\to 0$ uniformly on compacts from
$\Omega$ then

$\int^{x}_{x^{*}}\\!(\operatorname{trace}(G_{\mathbf{d}})+\nu)g_{\epsilon}(y)-\\!\\!\sum\limits_{i:x_{i}\neq
x_{i}^{*}}\\!\\!\\!\ \Delta_{i}(g_{\epsilon},y,x,x^{*},G_{\mathbf{d}})\;dy\to
0$

as $\epsilon\to 0$ for all $x,x^{*}\in\Omega_{\delta}$. This means that

$\min_{G\in\Xi}J_{\epsilon}(G)\to 0\quad\text{as}\quad\epsilon\to 0.$

The proof is complete

Notice that the above theorem implies that

$G_{\mathbf{d}}\in\mathcal{P}:=\lim_{\epsilon\to
0}\operatorname{argmin}_{G\in\Xi}J_{\epsilon}(G).$

provided that $G_{\mathbf{d}}\in\Xi$. However, the set $\mathcal{P}$ is not a
singleton in the general case and the problem of identification of the
generator $G_{\mathbf{d}}$ is ill-posed even if $\Xi=\Xi_{\rm lin}$, where

$\Xi_{\rm lin}=\\{G\in\mathbb{R}^{n\times n}:G\text{ is anti-Hurwitz}\\}.$
(32)

Indeed, for example, for any $\gamma>0$, the function
$h:\mathbb{R}^{2}\mapsto\mathbb{R}$ given by

$h(x)=x^{\top}x,\quad x=(x_{1},x_{2})^{\top}\in\mathbb{R}^{2}$ (33)

is $\mathbf{d}_{\gamma}$-homogeneous of degree $2$ with respect to the linear
dilation $\mathbf{d}_{\gamma}(s)=e^{s}\left(\begin{smallmatrix}\cos(\gamma
s)&-\sin(\gamma s)\\\ \sin(\gamma s)&\cos(\gamma
s)\end{smallmatrix}\right),s\in\mathbb{R}.$ So, the linear dilation symmetry
of the function $h$ given above cannot be uniquely identified.

To reduce the uncertainty in the dilation identification, the cost functional
can be modified as follow

$G_{\epsilon}=\operatorname{argmin}_{G\in\Xi}J_{\epsilon}(G)+\xi\operatorname{trace}(G^{\top}G),$
(34)

where $\Xi=\mathbb{R}^{n\times n}$ and $\xi>0$ is a small regularization
parameter. The second term in the above formula is a strictly convex function
(being a norm) in the space of square matrices. The quadratic optimization
problem becomes strictly convex with respect to $G$, so, for any fixed
$\epsilon>0$, the matrix $G_{\epsilon}$ is uniquely defined by (34). Another
possible way to reduce the uncertainty is to restrict the set of admissible
generators $\Xi$, for example, to diagonalizable (or simply diagonal)
matrices. However, this introduces additional a-priori assumption about the
function.

The functional $J_{\epsilon}$ admits the following approximation

$J_{\epsilon}(G)\approx\frac{1}{ML}\sum_{k=1}^{M}\sum_{j=1}^{L}\mathcal{J}^{2}_{\epsilon}(x_{k}+z_{j},x_{i},G)\,dz$
(35)

where the points $x_{k}\in\Omega_{\delta}$ are uniformly distributed in
$\Omega_{\delta}$, $z_{j}\in B_{\delta}$ are uniformly distributed in
$B_{\delta}$ and $M,L\in\mathbb{N}$. Finally, denoting
$f(x,x^{*}\\!,y,G)=(\operatorname{trace}(G)+\nu)g_{\epsilon}(y)-\\!\\!\\!\sum\limits_{i:x_{i}\neq
x_{i}^{*}}\\!\\!\\!\\!\Delta_{i}(g_{\epsilon},y,x,x^{*},G)$ we derive the
following approximation of the functional $\mathcal{J}^{2}_{\epsilon}$ for a
sufficiently small $\delta\in(0,1)$:

$\mathcal{J}_{\epsilon}(x_{i}+z_{j},x_{i},G)\approx\prod^{n}_{i=1}e^{\top}_{i}z_{j}\sum_{\lambda\in\\{0,1\\}^{n}}f(x_{k}+z_{j},x_{k},x_{k}+\mathrm{diag}(\lambda)z_{j},G).$
(36)

The presented discretizations of the functionals $J_{\epsilon}$ and
$\mathcal{J}_{\epsilon}$ can be utilized for identification of the generator
$G_{\epsilon}$ of the practical dilation, however, in this case, an anti-
Hurwitz solution of the optimization problem (30) may exist even if the ANN
$g_{\epsilon}$ approximates a non-homogeneous function. So, the practical
necessary condition of homogeneity (21) has be utilized in order to check if
the ANN indeed approximates a $\mathbf{d}_{\epsilon}$-homogeneous (at least on
$\Omega$) function. In the view of Proposition 4, a correctness of this
conclusion depends on approximation precision of ANN and increases as
$\epsilon\to 0$.

##  5 Examples of Homogeneous ANNs

###  5.1 Example 1: Homogeneous pattern recognition

The scaling invariance is desirable feature for a pattern recognition by ANN.
There are several approaches to exhibit the required invariance (see, e.g.,
[3, page 262]). One of them is to build the invariance properties into the
structure of a neural network. The present example uses the generalized
homogeneity for this purpose.

For simplicity, we assume that we deal with gray-scale images. Following the
conventional approach, we associated any image with a function
$(z_{1},z_{2})\mapsto\phi(z)$ of two arguments, which value
$\phi(z)\in[0,255]$ define a tone of the gray color at the coordinates
$z=(z_{1},z_{2})^{\top}$. The value $\phi(z)=0$ corresponds to the white
color, while the value $\phi(z)=255$ means that the point with the coordinate
$z$ has the black color. A pattern depicted in the image is assumed to be
centered. The origin $z=(0,0)$ of the coordinate frame corresponds to the
center of the image. In computer, the coordinates of the points $z$ as well as
the possible values of $\phi(z)$ are represented by integers , i.e.,
$z\in\mathbb{Z}^{2},\phi(z)\in\\{0,1,2,...,255\\}$. We extend the domain of
$\phi$ to $\mathbb{R}^{2}$ and the range of $\phi$ to the interval $[0,255]$,
respectively. This extension of the function $\phi$ can always be made in a
continuous manner. For the points $z$ out of the image range we assign
$\phi(z)=0$. In this case, the size of image does not impact to the definition
of $\phi\in C_{c}(\mathbb{R}^{2},\mathbb{R})$.

Traditionally, the values of $\phi(z)$ are utilized as inputs to ANN. Since
the function $\phi$ is not homogeneous even with respect to the standard
dilation $z\mapsto e^{s}z$, then the efficiency of a homogeneous ANN is
questionable in this case. To homogenize the problem, we introduce the
following transformations

$x_{i}(\phi)=\int_{\mathbb{R}^{2}}b_{i}(z)\phi(z)dz,\quad i=1,2,...,n$ (37)

where $b_{i}:\mathbb{R}^{2}\mapsto\mathbb{R}$ are standard homogeneous
functions of positive degree $r_{i}>-2$, i.e.,
$b_{i}(e^{s}z)=e^{r_{i}s}b(z),\forall s\in\mathbb{R},\forall
z\in\mathbb{R}^{2}$ and $n\in\mathbb{N}$. For example, we may take
$b_{i}(z)=\frac{z^{p_{i}}_{1}z_{2}^{q_{i}}}{r_{i}!}$, where
$m_{i}\in\\{1,2\\}$ and $p_{i},q_{i}\in\mathbb{N}\cup\\{0\\}$,
$r_{i}=p_{i}+q_{i}$. In this case, the functions $b_{i}$ are elements of
polynomial basis in $L^{2}(\Omega,\mathbb{R})$, where
$\Omega\subset\mathbb{R}^{2}$ contains a support of $\phi$. So, any function
$\phi\in C_{c}(\mathbb{R}^{2},\mathbb{R})\cap L^{2}(\Omega,\mathbb{R})$ can be
approximated as linear combination of the polynomial basis functions $b_{i}$
with the coefficients dependent on the vector
$x=(x_{1}^{\top},...,x_{n}^{\top})^{\top}\in\mathbb{R}^{2n}$ related with
coordinates of $\phi$ in the polynomial basis. Since the function $\phi$ is
compactly supported with the support dependent on the image size, the
computation of $x_{i}$ is not a difficult problem and it can be easily
parallelized.

Notice that the zoom of the image corresponds to the coordinate transformation
$z\mapsto e^{s}z$ with $s\in\mathbb{R}$, which implies the transformation
$\phi\mapsto\phi_{s}$, where $\phi_{s}(z):=\phi(e^{-s}z)$. Hence, we derive

$\begin{split}x_{i}(\phi_{s})=&\int_{\mathbb{R}^{2}}b_{i}(z)\phi(e^{-s}z)dz\\\
=&e^{2s}\int_{\mathbb{R}^{2}}e^{i_{i}s}b_{i}(e^{-s}z)\phi(e^{-s}z)d(e^{-s}z)\\\
=&e^{(2+p_{i})s}x_{i}(\phi)\end{split}$

In other words, the dilation $z\mapsto e^{s}z$ (the zoom of the image) implies
the dilation of the coordinates $x_{i}\mapsto e^{(2+r_{i})s}x_{i}$,
$i=0,...,n$. The vector $x=(x_{1},...,x_{n})^{\top}$ can be utilized as an
input to an ANN, which is going to be designed $\mathbf{d}$-homogeneous of
degree $0$ with respect to the dilation

$\mathbf{d}(s)\\!=\\!\mathrm{diag}(e^{(2+r_{1})s},...,,e^{(2+r_{n})s}),\quad
s\\!\in\\!\mathbb{R}.$

In this case, the scaling/zoom of the image will not impact the output of the
$\mathbf{d}$-homogeneous ANN. Therefore, once being trained, such ANN can be
utilized for the pattern recognition independently of the scaling of the
function $\phi\in C_{c}(\mathbb{R}^{2},\mathbb{R})$. However, scaling of the
image maps a function $\phi$ having a discrete domain $\mathbb{Z}^{2}$. In
this case a reduction of the image size may lead to a degradation of the image
quality and impossibility of its recognition. So, the $\mathbf{d}$-homogeneous
ANN will have the natural limitation in recognition of very reduced images.

For numerical illustration, we consider the 8 images of the characters shown
on Figure 1. Each image has the size 100 $\times$ 100 pixels. The components
of the input vector $x$ are defined by the formula (37) with $n=8$ and
$b_{1}=1,b_{2}=z_{1},b_{3}=z_{2},b_{4}=0.5z_{1}z_{2},b_{5}=0.5x^{2},b_{6}=0.5y^{2},b_{7}=x^{2}y/6;b_{8}=xy^{2}/6$.
To demonstrate the computational robustness of the method, the coordinates
$x_{i}$ are computed using the following approximation of the integral (37):

$x_{i}(\phi)\approx\sum_{k=1}^{100}\sum_{j=1}^{100}b_{i}(k-50,j-50)\phi(k,j).$

Figure 1: The characters for recognition

The ANN is assumed to have $k=8$ outputs. For the $i$-th character at the
input, only the $i$-th output must have the value $1$. All other outputs have
to be zero in this case. The training set consist of nominal images of the
size 100 $\times$ 100 pixels and the scaled images the size 75$\times$ 75
pixels, 50 $\times$ 50 pixels and 35 $\times$ 35 pixels, respectively. The
total number of images in the training set is 32. The matrices
$A\in\mathbb{R}^{N\times n}$ and $B\in\mathbb{R}^{N\times 1}$ are selected
randomly with $N=8$. The matrix $C\in\mathbb{R}^{k\times N}$ is obtained by
means of least square optimization on the training set. The trained
homogeneous ANN has been tested on images, which are scaled (zoomed) to
30%+$i$10% with $i=0,1,,...,17$. Notice that the decrease of the size of the
nominal image implies a degradation of the image quality (see Fig
LABEL:fig:char_zoom). ll scaled images have been successfully recognized.

To test the robustness with respect to the noise, a normally distributed noise
has been added to the nominal images. About 50% of pixels have been corrupted
by the noise. The homogeneous ANN successfully recognized the noisy images as
well. The construed homogeneous ANN have good capabilities of pattern
recognition with relatively small set of parameters to be tuned (just $8\times
8$ matrix $C$ of real numbers).

###  5.2 Example 2: Identification of homogeneous system

Let us consider now the rotation dynamics (1) of the rigid body with
$J=\left(\begin{smallmatrix}1&0.3&0.1\\\ 0.3&1.2&0.2\\\
0.1&0.2&0.8\end{smallmatrix}\right)$. First, we approximate the right-hand
side of (1)

$h(x)=-J^{-1}(\omega\times J\omega+\tau),\quad
x=(\omega^{\top},\tau^{\top})^{\top}\in\mathbb{R}^{6}$

by means of the conventional ANN with the sigmoid activation function

$g_{\epsilon}(x)=C\sigma(Ax+b)$

where $A\in\mathbb{R}^{N\times 6},b\in\mathbb{R}^{N}$ are randomly selected
for $N=500$ and $C\in\mathbb{R}^{3\times N}$ is obtained by means of the least
square error minimization for $M=20000$ measurements $y_{i}=h(x_{i})$ of the
function $h$ at the points $x_{i}\in\mathbb{R}^{6}$ randomly uniformly
distributed in the set

$\Omega=\\{x\in\mathbb{R}^{6}:0.95\leq|x|\leq 1.05\\}.$

On $\Omega$ the obtained ANN has the following approximation precision:

$|g_{\epsilon}(x)-h(x)|\leq\epsilon\approx 0.0110,\quad x\in\Omega$

which is approximated by taking $20000$ random points in $\Omega$.

####  5.2.1 Homogenization with known $\nu$ and $G_{\mathbf{d}}$

By Proposition 2, the conventional ANN $g_{\epsilon}$ can be upgraded to a
homogeneous ANN $h_{\epsilon}$ by the formula (13), where
$G_{\mathbf{d}}=\mathrm{diag}(1,1,1,2,2,2)$, $\nu=2$ and the canonical
homogeneous norm $\|\cdot\|_{\mathbf{d}}$ is induced by the usual Euclidean
norm $|\cdot|$. Such an upgrade slightly improve the approximation precision
($L^{\infty}$-norm of the error) on the set $\Omega$. Moreover, due to
homogeneity of $h$, the approximation precision of the homogeneous ANN (hANN),
on the domains $\\{x\in\mathbb{R}^{6}:|x|\leq 0.95\\}$ and
$\\{x\in\mathbb{R}^{6}:|x|\geq 1.05\\}$ is much better (in times) than the
precision of the classical ANN. Table 1 presents the comparison of the
approximation precision for the domains

$\begin{array}[]{ll}\Omega_{-4}=\\{x\in\mathbb{R}^{6}:0<|x|\leq
0.25\\},&\Omega_{4}=\\{x\in\mathbb{R}^{6}:1.75<|x|\leq 2\\},\\\
\Omega_{-3}=\\{x\in\mathbb{R}^{6}:0.25<|x|\leq
0.5\\},&\Omega_{3}=\\{x\in\mathbb{R}^{6}:1.5<|x|\leq 1.75\\},\\\
\Omega_{-2}=\\{x\in\mathbb{R}^{6}:0.5<|x|\leq
0.75\\},&\Omega_{3}=\\{x\in\mathbb{R}^{6}:1.25<|x|\leq 1.5\\},\\\
\Omega_{-1}=\\{x\in\mathbb{R}^{6}:0.75<|x|\leq
1\\},&\Omega_{1}=\\{x\in\mathbb{R}^{6}:1<|x|\leq 1.25\\}.\end{array}$

The table contains the estimates of the norms
$\|g_{\epsilon}(x)-h(x)\|_{L^{\infty}(\Omega_{k},\mathbb{R}^{6})}$ calculated
using 20000 randomly selected points in $\Omega_{k}$, $k=-4,...,4$.

Domain | ANN | hANN | hANN | hANN
---|---|---|---|---
|  | $\nu=2$ | $\nu=\nu_{\epsilon}$ | $\nu=1,G_{\mathbf{d}}=G_{\epsilon}$
$\Omega_{-4}$ | 0.0864 | 0.0010 | 0.0134 | 0.0284
$\Omega_{-3}$ | 0.0821 | 0.0024 | 0.0168 | 0.0286
$\Omega_{-2}$ | 0.0583 | 0.0038 | 0.0170 | 0.0213
$\Omega_{-1}$ | 0.0274 | 0.0063 | 0.0135 | 0.0122
$\Omega_{0}=\Omega$ | 0.0110 | 0.0074 | 0.0082 | 0.0094
$\Omega_{1}$ | 0.0612 | 0.0090 | 0.0193 | 0.0293
$\Omega_{2}$ | 0.2257 | 0.0115 | 0.0417 | 0.0787
$\Omega_{3}$ | 0.5056 | 0.0153 | 0.0773 | 0.1551
$\Omega_{4}$ | 1.0409 | 0.0192 | 0.1160 | 0.3097
Table 1: Approximation precision of the conventional and homogeneous ANN

####  5.2.2 Homogenization using practical homogeneity degree

Let us assume that the homogeneity degree $\nu$ of the function $h$ is
unknown, but the dilation $G_{\mathbf{d}}$ is known. By Proposition 3, we can
identify a practical homogeneity $\nu_{\epsilon}$ using the conventional ANN,
which is already trained on the set $\Omega$. We compute $\nu_{\epsilon}$ by
the formula (26) for randomly selected points $x_{i}\in\Omega$, where
$i=1,2,...,M=2000$. This gives the practical estimate of the homogeneity
degree $\nu_{\epsilon}=1.9647$ being rather close to the real value $\nu=2$.
The approximation precision of the hANN with the practical homogeneity degree
is given in Table 1. The use of homogeneous ANN (even with the practical
homogeneity degree) essentially improves the extrapolation capabilities of the
ANN.

####  5.2.3 Homogenization using practical dilation symmetry

Now we assume that the dilation is unknown and we need to approximate
$G_{\mathbf{d}}$ using the already well-trained ANN $g_{\epsilon}$. For the
data-driven identification of the dilation symmetry, we consider the set
$\Omega_{\delta}=\\{x\in\mathbb{R}^{n}:0.98\leq|x|\leq 1.02\\}$ and optimize
the approximate functional (35) with $M=4000$,
$L=1,z_{1}=\delta(1,...,1)^{\top},\delta=0.01$. We also restrict the class of
admissible generators to diagonal matrices. For $\nu=1$ the obtained matrix
$G_{\epsilon}=\mathrm{diag}(0.5199,0.5150,0.4972,1.0021,0.9965,1.0027)$,
which, obviously, corresponds to $G_{\mathbf{d}}$ up to the scaling of
$G_{\epsilon}$ and $\nu$ by the multiplier $\gamma=2$ (please see explanations
in the beginning of Section 4.2 for more details). The homogeneous ANN
constructed in this way is a homogeneous extrapolation of the ANN
$g_{\epsilon}$. In all domain $\Omega_{k}$ except the training set
$\Omega_{0}$, the approximation precision of hANN is better in times comparing
with the original ANN (see Table 1).

Notice that For $\nu=0$ and $\nu=-1$, the optimization problem does not
provide anti-Hurwitz solutions. This implies impossibility of non-positive
homogeneity degree of the function.

###  5.3 Example 3: Homogeneous control implementation

The canonical homogeneous norm $\|\cdot\|_{\mathbf{d}}$ (see Definition 1) is
utilized in control theory for fast/robust regulation of linear plants [22],
[21]:

$\dot{x}=Ax+Bu,\quad u=K_{0}+K\mathbf{d}(-\ln\|x\|_{\mathbf{d}})x,$

where $x(t)\in\mathbb{R}^{n}$ is the system state, $u(t)\in\mathbb{R}^{m}$ is
the control input, $A\in\mathbb{R}^{n\times n}$, $B\in\mathbb{R}^{n\times m}$
are system matrices and $K_{0},K\in\mathbb{R}^{m\times n}$ are control gains.
The methodology for tuning of the control parameters $K_{0},K$ of the above
nonlinear controller is well developed, but the homogeneous norm
$\|\cdot\|_{\mathbf{d}}$ is defined implicitly (see Definition (1)). The
practical implementation of this implicit regulator requires of an application
of numerical numerical procedure (e.g., bisection method) for computation of
$\|x\|_{\mathbf{d}}$ at each $x\in\mathbb{R}^{n}$. For a real-life control
application, the norm $\|x(t)\|_{\mathbf{d}}$ has to be computed in a real
time. This may be impossible, if the computational power of the digital
controller is low. In the view of Theorem 3, the implicitly defined norm
$\|\cdot\|_{\mathbf{d}}$ can be approximated with arbitrary high precision by
means of an explicitly defined norm $\|\cdot\|_{\mathbf{d}^{*}}$ and the
homogeneous ANN (13):

$\|x\|_{\mathbf{d}}\approx\|x\|_{\mathbf{d}^{*}}C\sigma(A\mathbf{d}(-\ln\|x\|_{\mathbf{d}^{*}})x+b).$
(38)

For example, let us consider the weighted dilation
$\mathbf{d}(s)=\mathrm{diag}(e^{5s},e^{4s})$ in $\mathbb{R}^{2}$ and the
canonical homogeneous norm $\|\cdot\|_{\mathbf{d},1}$ induced by the weighted
Euclidean norm $\|\cdot\|=\sqrt{x^{\top}Px}$ with
$P=\left(\begin{smallmatrix}0.1157&0.0194\\\
0.0194&0.1186\end{smallmatrix}\right)$. The explicit homogeneous norm is
defined as follows

$\|x\|_{\mathbf{d}^{*}}=\left(\Psi^{\top}(x)Q\Psi(x)\right)^{1/10},\quad
Q=\left(\begin{smallmatrix}0.1225&0.0176\\\
0.0176&0.0720\end{smallmatrix}\right),$

where $x=(x_{1},x_{2})^{\top}\in\mathbb{R}^{2}$,
$\Psi(x)=\left(\begin{smallmatrix}x_{1}\\\
|x_{2}|^{5/4}sign(x_{2})\end{smallmatrix}\right)$ and the matrix $Q$ is
selected such that the difference between $\|\cdot\|_{\mathbf{d}}$ and
$\|\cdot\|_{\mathbf{d}^{*}}$ is minimized [21]. On the unit sphere
$S=\\{x\in\mathbb{R}^{2}:\|x\|=1\\}$ the difference between norms is about
$\epsilon\approx 0.007$. Such an error is unacceptable for some control
applications, where a high control precision is required. To improve the
approximation, we use the homogeneous ANN (38) with the sigmoid activation
function $\sigma$ and randomly selected parameters $A\in\mathbb{R}^{N\times
1}$ and $b\in\mathbb{R}^{N}$. By definition, the canonical homogeneous norm
equals $1$ on the unit sphere, i.e., $\|x\|_{\mathbf{d}}=1$ for all $x\in S$.
The vector $C\in\mathbb{R}^{1\times N}$ is optimized using least square
criterion

$\sum_{i=1}^{M}\left(1\\!-\\!\|x_{i}\|_{\mathbf{d}^{*}}C\sigma(A\mathbf{d}(-\ln\|x_{i}\|_{\mathbf{d}^{*}})x+b)\right)^{2}\to\min_{C\in\mathbb{R}^{1\times
N}},$

where the points $x_{i}$ are randomly distributed on the unit sphere $S$. For
$N=10$ such homogeneous ANN

$\|x\|_{\mathbf{d},\epsilon}=\|x\|_{\mathbf{d}^{*}}C\sigma(A\mathbf{d}(-\ln\|x\|_{\mathbf{d}^{*}})x+b)$

improves the approximation precision of $\|x\|_{\mathbf{d}}$ on the unit
sphere in two times $\epsilon\approx 0.0034$ in comparison with the norm
$\|\cdot\|_{\mathbf{d}^{*}}$(see Fig. 2).

Figure 2: Level lines $\|x\|_{\mathbf{d}}=1$ (red), $\|x\|_{\mathbf{d}^{*}}=1$
(blue) and $\|x\|_{\mathbf{d},\epsilon}=1$ (green)

For $N=20$ we have $\epsilon\approx 5\cdot 10^{-5}$. The plot for the latter
two cases is not presented since the corresponding spheres are visually
indistinguishable with the unit sphere. Therefore, the canonical homogeneous
norm estimated by the homogeneous ANN can be efficiently utilized in control
engineering for implementation of the implicit homogeneous controller [22],
[21] in digital devices.

##  6 Conclusion

The paper introduces a class of ANNs being global approximators of the so-
called generalized homogeneous functions. The homogeneous universal
approximation theorem is proven. A dilation symmetry of the homogeneous
function allows local approximation results to be extended globally.

An information about dilation symmetry of a function approximated by a
conventional ANN may be unknown. However, the generalized homogeneous function
theorem proven in the paper shows that this information is, in fact, hidden in
ANN provided that the ANN is well trained. This important information can be
extracted (identified) for the ANN by solving a quadratic optimization problem
and, next, the conventional ANN can be easily homogenized using the identified
dilation. The obtained homogeneous ANN is a global extrapolation of the
conventional ANN away from the training set.

The potential advantages of homogeneous ANN are demonstrated on academic
examples from computer science, systems theory and control engineering.
Application of homogeneous ANN to practical problems is an interesting problem
of future research. A theoretical study about homogeneous physics-informed
ANNs [14] may also be interesting, since many models of mathematical physics
are homogeneous in generalized sense [21].

##  7 Data and Code

The code for MATLAB and the trained ANNs for the considered examples are
available at

https://gitlab.inria.fr/polyakov/homogeneous-artificial-neural-networks

## References

  * [1] V. Andrieu, L. Praly, and A. Astolfi.  Homogeneous Approximation, Recursive Observer Design, and Output Feedback.  SIAM Journal of Control and Optimization, 47(4):1814–1850, 2008\.
  * [2] S. P. Bhat and D. S. Bernstein.  Geometric homogeneity with applications to finite-time stability.  Mathematics of Control, Signals and Systems, 17:101–127, 2005.
  * [3] C. Bishop.  Pattern Recognition and Machine Learning.  Springer, 2006.
  * [4] G. Cybenko.  Approximation by superpositions of a sigmoidal function.  Mathematics of Control, Signals, and Systems, 4(2):303–3014, 1989\.
  * [5] V. Fischer and M. Ruzhansky.  Quantization on Nilpotent Lie Groups.  Springer, 2016.
  * [6] G. Folland.  Subelliptic estimates and function spaces on nilpotent Lie groups.  Arkiv for Matematik, 13(1-2):161–207, 1975.
  * [7] L. Grüne.  Homogeneous state feedback stabilization of homogeneous systems.  SIAM Journal of Control and Optimization, 38(4):1288–1308, 2000\.
  * [8] H. Hermes.  Nilpotent approximations of control systems and distributions.  SIAM Journal of Control and Optimization, 24(4):731, 1986.
  * [9] K. Hornik.  Approximation capabilities of muitilayer feedforward networks.  Neural Netorks, 4(2):251–257, 1991.
  * [10] K. Hornik, M. Stinchcombe, and H. White.  Multilayer feedfoward networks are universal approximators.  Neural Networks, 2(5):359–366, 1989.
  * [11] L.S. Husch.  Topological Characterization of The Dilation and The Translation in Frechet Spaces.  Mathematical Annals, 190:1–5, 1970.
  * [12] M. Kawski.  Families of dilations and asymptotic stability.  Analysis of Controlled Dynamical Systems, pages 285–294, 1991.
  * [13] V. V. Khomenuk.  On systems of ordinary differential equations with generalized homogenous right-hand sides.  Izvestia vuzov. Mathematica (in Russian), 3(22):157–164, 1961.
  * [14] I. E. Lagaris, A. Likas, and D. I. Fotiadis.  Artificial neural networks for solving ordinary and partial differential equations.  IEEE Transactions on Neural Networks, 9(5):987–1000, 1998.
  * [15] M. Leshno, V. Lin, A. Pinkus, and S. Schocken.  Multilayer feedforward networks with a nonpolynomial activation function can approximate any function.  Neural Networks, 6(6):861–867, 1993.
  * [16] A. Levant.  Higher-order sliding modes, differentiation and output-feedback control.  International Journal of Control, 76(9-10):924–941, 2003.
  * [17] A. M. Lyapunov.  The general problem of the stability of motion.  Taylor & Francis, 1992.
  * [18] B. Neyshabur, R. Salakhutdinov, and N. Srebro.  Advances in Neural Information Processing Systems, chapter Path-SGD: Path-Normalized Optimization in Deep Neural Networks.  The MIT Press, 2015.
  * [19] A. Pazy.  Semigroups of Linear Operators and Applications to Partial Differential Equations.  Springer, 1983.
  * [20] W. Perruquetti, T. Floquet, and E. Moulay.  Finite-time observers: application to secure communication.  IEEE Transactions on Automatic Control, 53(1):356–360, 2008.
  * [21] A. Polyakov.  Generalized Homogeneity in Systems and Control.  Springer, 2020.
  * [22] A. Polyakov, D. Efimov, and W. Perruquetti.  Finite-time and fixed-time stabilization: Implicit Lyapunov function approach.  Automatica, 51(1):332–340, 2015.
  * [23] L. Rosier.  Homogeneous Lyapunov function for homogeneous continuous vector field.  Systems & Control Letters, 19:467–473, 1992.
  * [24] V.I. Zubov.  On systems of ordinary differential equations with generalized homogeneous right-hand sides.  Izvestia vuzov. Mathematica (in Russian), 1:80–88, 1958.