#  Complex-Valued Neural Networks - Theory and Analysis

Rayyan Abdalla

###### Abstract

Complex-valued neural networks (CVNNs) have recently been successful in
various pioneering areas which involve wave-typed information and frequency-
domain processing. This work addresses different structures and classification
of CVNNs. The theory behind complex activation functions, implications related
to complex differentiability and special activations for CVNN output layers
are presented. The work also discusses CVNN learning and optimization using
gradient and non-gradient based algorithms. Complex Backpropagation utilizing
complex chain rule is also explained in terms of Wirtinger calculus. Moreover,
special modules for building CVNN models, such as complex batch normalization
and complex random initialization are also discussed. The work also highlights
libraries and software blocks proposed for CVNN implementations and discusses
future directions. The objective of this work is to understand the dynamics
and most recent developments of CVNNs.

##  1 Introduction

Artificial neural networks have been tightly fused into various domains and
applications fields. In many practical applications, including
electromagnetics, signal and image processing, complex numbers occur by nature
or design to represent information and frequency domain representations. This
suggests that neural networks-based models utilized for such applications have
parameters and variables in the complex domain. In general, the majority of
widely deployed deep learning models are based on real-valued neural networks
(RVNNs) due to less complexity and easier implementation associated with real
parameters. Nonetheless, complex-valued neural networks (CVNNs), which are
networks processing information with complex-valued parameters, have gained
more interest and proven potential advantages over conventional RVNNs [1, 2,
3].

The prospects behind adopting CVNN can be viewed from biological,
computational and application perspectives [2, 3]. Complex-valued
representations of neuron activity allows more versatile extension of brain
physicality [1]. Moreover, CVNNs are proved to perform more efficiently with
better generalization for complex-valued tasks as well as nonlinear
inseparable real-valued tasks. In terms of application fields, CVNNs recently
have successful implementations in areas of image protection, image
segmentation, image reconstruction, signal identification and estimation,
telecommunications, speech processing, wind prediction, control systems and
many other relevant domains [2]. This article explains the motivation when and
why CVNNs can be more effective than RVNNs. Different neural networks
structures that deal with complex-valued data are also discussed. Moreover,
this article addresses CVNN distinctive features in details, including complex
activation functions and CVNN learning algorithms, besides challenges and
special considerations related to building CVNN based models. The article also
lists most recent implementation efforts and highlights supporting software
libraries for solving complex-valued problems.

The work organization is as follows: section 2 is CVNN motivation. Section 3
provides proposed neural networks structures dealing with complex-valued
representations. The complex activation functions are discussed in section 4.
Learning algorithms and CVNN optimization is presented in section 5. Section 6
explains implementation of batch normalization technique in the complex
domain. Section 7 is complex weight initialization. Section 8 summarizes
implementation and software deployment attempts. Finally, future research
directions and concluding remarks are drawn in section 9.

##  2 Motivation of CVNN

Neural Networks based on Complex-valued representations have shown several
advantages over their real-valued counterparts, though widely applied RVNNs
manifest easier implementations regarding activation function design and
learning algorithms. CVNNs have great potential in application domains that
involve complex-valued data and signals [2, 4, 5, 6, 7, 8]. Besides, CVNN
models enable improved optimization, better generalization, and faster
learning even for real-valued input representations [9, 10] . In terms of
biological viewpoint, complex-valued configuration of Neural Networks
parameters proposes more versatile and biologically plausible neuron firing
[1, 2, 3].

Several practical applications encode processed information using complex-
valued data types. For example, image and signal processing involve frequency
domain representations and expressions in terms of Transforms bases. Moreover,
electromagnetic signals and wave-related applications, such as sensing, bio-
informatics and telecommunications, employ amplitude-phase complex expressions
to naturally represent signal energy and phase as progression in time. Neural
networks utilized in such applications suggest that corresponding models use
complex numbers to represent network inputs, outputs and learnable weights,
i.e. using CVNNs. While many analyses propose that CVNNs can be implemented
using double dimensional RVNNs, this may result in erroneous network behavior
since complex number arithmetic is different from Real-valued operations and
complex components are statistically correlated [11]. For wave-typed
information, CVNNs are essentially more compatible than RVNNs since amplitude
and phase are directly related to learning objective and real-world wave
information can be fully extracted when training a CVNN rather than RVNN [1].

In terms of computational motivation, CVNNs feature complex numbers
arithmetic, particularly complex multiplication operation, for weights
learning and input-output mapping. While complex multiplication yields
amplitude attenuation and phase rotation, this results in an advantageous
reduction in degrees of freedom and minimizing learnable parameter [1] . In
other words, RVNNs approach generally represents complex components
independently either in vector or matrix form. That is, real-valued matrix
operations on complex components increase the model parameters and dimensions,
resulting in undesirable generalization characteristics and greater tendency
for model overfitting [11]. On the other hand, CVNNs with complex weights
utilize phase rotation dynamics to underpin learning task in one-dimensional
complex domain, and matrix representation in the complex domain mimics
rotation matrix with half of matrix entries already known. Therefore, CVNN
handling of complex-valued problems results in reduced solution arbitrariness
and enhanced generalization characteristics. The computational power of CVNNs
is also revealed through the orthogonal property of the complex-valued neuron.
This property refers to the concept of orthogonal decision boundary existing
in the complex-valued neuron in which two hypersurfaces intersect and divide
decision boundary into four regions. This high generalization ability enables
solving special problems, such as the XOR problem, that cannot be solved by a
single real-valued neuron [12]. Furthermore, representations based on complex
numbers have additional computational advantages for neural networks with
architectures similar to associative memories, such as Deep residual networks
(ResNets) and recurrent neural networks (RNNs). It has been shown that
Holographic Reduced Representations, which use complex numbers, enable
numerically efficient insertion and retrieval into associative memory in key-
value pairs, increasing representational capacity and stability [13]. In
addition, orthogonal weight matrices have been proposed to mitigate the
problem of vanishing or exploding gradients associated with long-term
dependencies in network learning process. Unitary weights matrices, based on
which Unitary RNNs were proposed, are generalization of the orthogonal
matrices in the complex plane. Unitary RNNs uncover spectral representations
by implementing discrete Fourier transform to learn unitary weight matrices
[14]. That is, this enables richer representations by spanning parts of the
matrix full set and preserve gradient over learning time.

From biological perspective, using complex-valued weights proposes plausible
neuronal formulation that allows constructing a richer, versatile deep network
representation. In general, neurons convey information in the form of
rhythmical spikes characterized by average firing rate and comparative timing
of activity [15], which can be specified by amplitude and phase of a complex
valued neuron. On the contrary, real-valued neuron representation considers
only the firing rate and neglects the phase, which may affect neurons
synchronization. With complex formulation, input neurons of similar phase are
synchronous and add constructively, those with dissimilar phases add
destructively and are terms asynchronous. Therefore, inclusion of phase
information is effective to synchronize inputs that the network propagates
with time at a given layer, useful for controlling mechanisms of deep gating-
based networks.

Given all these supportive evidences from computational, biological and signal
processing perspectives, It is reasonable to apply complex numbers to neural
network models.

##  3 Neural Networks for Complex-valued Data Representations

###  3.1 RVNN Implementation

There are two possible approaches for handling complex-valued inputs using
conventional RVNNs. In both methods, the individual components of complex
inputs are treated independently as real-valued numbers. Such RVNN variants
can be described as follows [11]:

####  3.1.1 Real-valued Bivariate Network Structure:

RVNN with double input terminals for real and imaginary parts of the input,
and double output neurons to generate real and imaginary parts of output
signal. Therefore, forward processing is performed in a bivariate manner by
doubling the number of neurons of different network layers.

####  3.1.2 Dual Real-valued Univariate Network Structure:

This is a type of RVNN where real and imaginary parts of the input are
processed separately. Many techniques can be applied to separate or mix the
real and imaginary parts. That is, the neurons from the real-part network are
not connected to the neurons from the imaginary-part network. Therefore, the
respective networks will not double the number of terminals and output
neurons.

Handling complex input with real-valued networks increases network dimensions
and learnable parameters [2, 1]. Moreover, RVNN approaches result in phase
distortion and decreased approximation accuracy as real-valued gradient does
not reflect true complex-valued gradient [2].

###  3.2 CVNN Implementation

As CVNNs retain complex-valued parameters and variables, there are two types
of CVNNs [2]:

####  3.2.1 Split-CVNNs:

Here, the input value splits into two real-valued components in the form of
either rectangular coordinates (real and imaginary components) or polar
coordinates (magnitude and phase components). The split components are fed to
a network with complex valued weights and real-valued activation function [9,
10].

####  3.2.2 Fully Complex CVNNs:

Such networks deal directly with complex-valued signals, where both weights
and activation functions are in the complex domain [16, 17].

Unlike RVNN implementation, CVNN approaches overcome phase distortion problem
and result in minimum generalization error for highly coherent signals [11].
Nonetheless, Fully complex CVNN are computationally complex as they require
complex-differentiable (i.e. Holomorphic) activation function, whereas Split-
CVNN are mostly-implemented CVNN since they make use of conventionally
available non-holomorphic activation functions.

##  4 Complex Activation Functions

Activation functions are required to introduced non-linear mapping to affine
transforms in artificial neural networks. Given an input signal $x_{i}$, a
connection weight $w_{i}$, an internal state $u=\sum_{i}x_{i}w_{i}$,
activation function $f$, the output $y$ is given by:

$y=f(u)=f\left(\sum_{i}w_{i}x_{i}\right)$ (1)

In general, activation functions are chosen to be peicewise smooth to
facilitate gradient computation during the learning process. Due to complex
differentiability concerns, designing a suitable activation function for CVNN
is one of the most challenging parts that hindered earlier CVNN developments
[1].

###  4.1 Concerns Regarding Activation Function Differentiability in the
Complex Domain

In terms of differentiability in the complex domain, functions are though to
be holomorphic, i.e. complex differentiable, or non-holomorphic. For a complex
function to be holomorphic, it must satisfy Cauchy-Riemann equations [9, 10].
Designing complex activation functions conforms Liouville’s theorem which
implies that bounded real-valued activation functions are non-holomorphic in
the complex plane $\mathbb{C}$. Therefore, it is restrictedly difficult to
directly extend conventional real-valued activation functions in the complex
domain. This is further elaborated in the following:

####  4.1.1 Holomorphism and Cauchy-Riemann Equations

Holomorphism, also termed analyticity, in a domain within the complex
coordinate space refers to that a complex function is differentiable in the
neighborhood of every point within the corresponding domain. A complex
function that is differentiable in the whole complex plane $\mathbb{C}$ is
called entire function. That is, for a complex number $z=x+iy$ and a complex-
valued function $f(z)=u(x,y)+iv(x,y)$ where $u$ and $v$ are real-valued
functions, then $f$ is holomorphic in a complex domain if the derivative
$f^{{}^{\prime}}(z_{0})$ exists at every point $z_{0}$ in that domain.

$f^{{}^{\prime}}(z_{0})=\lim_{\Delta z\to 0}\left[\frac{\left(f(z_{0})+\Delta
z\right)-f(z_{0})}{\Delta z}\right]=\lim_{\Delta x\to 0}\lim_{\Delta y\to
0}\left[\frac{\Delta u(x_{0},y_{0})+i\Delta v(x_{0},y_{0})}{\Delta x+i\Delta
y}\right]$ (2)

Since $\Delta z=\Delta x+i\Delta y$, $\Delta z$ can approach $0$ from multiple
directions whether along real axis (i.e. $\Delta y=0$), imaginary axis (i.e.
$\Delta x=0$ or in-between axes.However, for the function $f$ to be complex
differentiable, $f^{{}^{\prime}}(z_{0})$ must give the same quantity
regardless the direction of approach. Consider $\Delta z$ approaches $0$ along
real axis or imaginary axis, respectively:

$f^{{}^{\prime}}(z_{0})=\lim_{\Delta x\to 0}\left[\frac{\Delta
u(x_{0},y_{0})+i\Delta v(x_{0},y_{0})}{\Delta x+i0}\right]=\lim_{\Delta y\to
0}\left[\frac{\Delta u(x_{0},y_{0})+i\Delta v(x_{0},y_{0})}{0+i\Delta
y}\right]$ (3)

The above equation is equivalent to:

$\frac{\partial f}{\partial z}=\frac{\partial u}{\partial x}+i\frac{\partial
v}{\partial x}=-i\frac{\partial u}{\partial y}+\frac{\partial v}{\partial y}$
(4)

Following equations 3 and 4, the function $f$ is holomorphic if it satisfies
the two equations:

$\frac{\partial u}{\partial x}=\frac{\partial v}{\partial y},\frac{\partial
u}{\partial y}=-\frac{\partial v}{\partial x}$ (5)

The equations in 5 are Cauchy-Riemann equations which represent a necessary
condition for holomorphism. If the functions $u$ and $v$ have continuous first
derivatives, then the equations 5 are sufficient condition for a holomorphic
$f$ [10].

In order to analyze complex neural networks dynamics and learning in the same
manner as in real-valued neural networks, proposed nonlinear complex
activation function must be analytic over whole $\mathbb{C}$ to avoid the
problem of exploding or vanishing gradients during backpropagation .

####  4.1.2 Liouville’s Theorem

In complex analysis, a function $f(z)$ is called bounded if it satisfies:
$|f(z)|<M,M\in\mathbb{R^{+}}$. Liouville’s Theorem states that if a function
$f(z)$ that is analytic, i.e. holomorphic at any point in $\mathbb{C}$, and is
bounded, it must be a Constant function: $f(z)=c,c\in\mathbb{C}$.
Equivalently, entire nonlinear functions, that are differentiable in whole
plane $\mathbb{C}$, are unbounded functions, making them unsuitable as CVNN
activation functions [9, 3, 2].

###  4.2 CVNN Activations with Complex Partial Derivatives

The complex differentiation concerns mentioned previously constrain CVNN
activation function design by selecting one property, either bounded or
analytic. By bounding the domain of the complex function and isolating
singular points, one can propose an activation function that is holomorphic in
the corresponding domain [10]. Nevertheless, according to [1], complex
activation functions can also be designed without paying attention to
differentiability, instead, CVNN dynamics and self-organization can be
constructed by means of partial derivatives with respect to complex
components. An activation function shape can be determined with respect to
input/output properties and processing purposes, meaning that non-holomorphic
and complex linear activations are also possible.

Non-holomorphic function differentiation can be done utilizing Wirtinger
Calculus [18] to compute the gradient of complex components in part. In other
words, complex backpropagation and CVNN optimization can be implemented using
non-holomorphic complex activation functions by computing the partial
derivatives with respect to real and imaginary parts. This therefore
facilitates complex chain rule implementation for both real and complex
functions. Further explanation is provided in the following:

####  4.2.1 Wirtinger Calculus

Wirtinger calculus provides a generalized framework for computing complex
function derivative, with the holomorphic function being a special case. For a
complex function $f(z)$ of a complex variable $z=x+iy$, the partial derivative
with respect to $z$ and the conjugate $\bar{z}$ [9]:

$\begin{split}\frac{\partial f}{\partial z}=\frac{1}{2}\left(\frac{\partial
f}{\partial x}-i\frac{\partial f}{\partial y}\right)\\\ \frac{\partial
f}{\partial\bar{z}}=\frac{1}{2}\left(\frac{\partial f}{\partial
x}+i\frac{\partial f}{\partial y}\right)\end{split}$ (6)

From 6, the derivatives of $f$ with respect to real part $x$ and imaginary
part $y$ are given by:

$\begin{split}\frac{\partial f}{\partial x}=\frac{\partial f}{\partial
z}+\frac{\partial f}{\partial\bar{z}}\\\ \frac{\partial f}{\partial
y}=i\left(\frac{\partial f}{\partial z}-\frac{\partial
f}{\partial\bar{z}}\right)\end{split}$ (7)

Based on Wirtinger formulation and for a holomorphic function $f$ satisfying
Cauchy-Riemann condition in 5, then: $\frac{\partial f}{\partial\bar{z}}=0$,
as proved in theorem 2.5 of [9]. This condition makes holomorphic functions
more computationally efficient since gradient values are shared during
backpropagation [10], as will be shown in later sections.

####  4.2.2 Complex Chain Rule

Following Wirtinger Calculus framework, chain rule can be implemented for both
real and complex functions with respect to complex variables. Analogous to
multivariate chain rule with real values, as shown in theorem A.3 of [9],
given a complex variable $z=x+iy$, complex functions $h,g$ where $h$ is a
function of $g$ is a function of $z$ such that $h(g(z))$. Theorem A.4 and A.5
of [9] state that the complex chain rule is given by:

$\begin{split}\frac{\partial h(g)}{\partial z}=\frac{\partial h}{\partial
g}\frac{\partial g}{\partial z}+\frac{\partial
h}{\partial\bar{g}}\frac{\partial\bar{g}}{\partial z}\\\ \frac{\partial
h(g)}{\partial\bar{z}}=\frac{\partial h}{\partial g}\frac{\partial
g}{\partial\bar{z}}+\frac{\partial
h}{\partial\bar{g}}\frac{\partial\bar{g}}{\partial\bar{z}}\end{split}$ (8)

And the complex chain rule with respect to real and imaginary parts of $z$:

$\begin{split}\frac{\partial g}{\partial z}=\frac{\partial g}{\partial
x}\frac{\partial x}{\partial z}+\frac{\partial g}{\partial y}\frac{\partial
y}{\partial z}\\\ \end{split}$ (9)

There is no assumption regarding whether the complex functions $h,g$ are
holomorphic or not. Note that real valued functions are already included in
the complex domain and follow the same chain rule with respect to complex
variables. Mathematical proofs of the formulas are provided in [9].

###  4.3 Types of CVNN Activation Functions

The complex domain widens the possibilities to design a complex activation
function. Taking into account all previously mentioned complex analysis, two
categories of CVNN activation functions are proposed [2, 3] , upon which CVNN
types were defined as in 3.2. First, Split activation functions, which deals
with complex components independently. Second, Fully complex activation
functions, in which complex components are treated as a single entity.

####  4.3.1 Split Activation Functions

The idea of split activation function was motivated by ability to
differentiate non-holomorphic functions with respect to their complex
component parts, as explained in 4.2. Split activation functions are adopted
from real-valued state-of-the-art deep architectures, that is, such CVNN
activations apply real-valued conventional activation function to complex
component parts independently and combine results to produce complex neuron
output . Therefore, such activation function are bounded and not analytic.
Based on complex coordinate system, i.e. cartesian coordinates or polar
coordinates, which is imposed by CVNN application, split activation function
are further divided into two types [1, 3, 9, 2]. For a complex variable
$z=x+iy=|z|\exp(i\arg(z)),z\in\mathbb{C}$ and $x,y\in\mathbb{R}$

  1. 1.

Type-A Split Activation Function: $f_{A}(z)=f_{Re}(x)+if_{Im}(y)$

  2. 2.

Type-B Split Activation Function $f_{B}(z)=f_{r}(|z|)+if_{\phi}(\arg(z))$

Where $f_{A},f_{B}$ are complex-valued functions, and
$f_{Re},f_{Im},f_{r},f_{\phi}$ are real-valued activation functions of
conventional types ( can be Sigmoid, tanh, Identity …). In general, CVNNs
employing Type-A activation functions select the same real function for
$f_{Re}$ and $f_{Im}$. Such networks work well with complex information having
symmetry or special meaning on real and imaginary axes [1], and can also
process real-valued data with half number of RVNN network parameters [10, 9].
On the other hand, widely implemented Type-B activation functions squashes the
magnitude and leave the phase unchanged, i.e. set $f_{r}$ as a conventional
activation function while $f_{\phi}$ is identity. That is, networks with
Type-B activation functions are adopted for wave-related applications and
information of isotropic structure due to their direct correspondence to
signal amplitude and phase[1].

Most of popular activation functions from RVNNs are extensible and easily
implemented using Type-A and Type-B complex activation function, though the
hyperbolic tangent function is already implemented in the complex domain and
its transformation is not necessary [9].

####  4.3.2 Fully-Complex Activation Functions

Fully complex activations corresponds to using holomorphic functions that deal
with complex variable as a single entity. Difficulty in fulfilling Liouville’s
theorem greatly restrict the use of fully complex activation function since
the proposed function must be bounded and analytic in the desired complex
domain [10].

Early works of fully complex activations involve the mapping of multi-valued
neuron (MVN) [19]. The MVN is a neural element with activation that maps the
complex weighted input into an output lying on the unit circle. Specifically,
for a complex weighted sum $z$ and activation function $f$, the neuron output:
$\omega^{j}=\exp(i2\pi j/k),j=0,1,...,k-1$, is one of $k$ roots of unity
according to:

$\begin{split}f(z)=\exp(i2\pi j/k),\hskip 36.135ptif\hskip 7.22743pt2\pi
j/k\leq\arg(z)\leq 2\pi(j+1)k\end{split}$ (10)

Moreover, MVN activation function can deal with continuous-valued inputs by
making $k\to\infty$ [3], such that

$f(z)=\exp(i\arg(z))=\frac{z}{|z|}$ (11)

Fully-complex MVN activation function can be also seen as a Type-B split
activation function that sets the magnitude to unity and squashes the phase,
as the case of "Phasor" networks [20].

Furthermore, elementary transcendal functions (ETFs) are proposed for fully-
complex activations [16, 21]. These functions ( for example: sinh($z$),
sin($z$), arcsin($z$), tan($z$), arctan($z$), arctanh($z$), arccos($z$) ) are
holomorphic almost everywhere in the complex domain where singularities can be
isolated or removed in the desired complex domain [3, 22]. However, training
with ETFs is restrictedly limited due to lower generalization ability and slow
convergence rate [2].

There is still no agreed rule on which approach is best for CVNN activations.
The only concern is that the selected activation function does not cause
vanishing or exploding gradient during learning process. Nevertheless, other
complex activation functions with structures slightly dissimilar to those
described above were also proposed, such as complex adaptions of popular ReLU
functions.

###  4.4 Complex Rectified Linear Unit (ReLU)

The RelU function, $ReLU(x):=max(x,0)$, is widely applied in conventional
RVNNs due to proved cabaility of earning faster than equivalents with
saturating neurons. Therefore, this motivated defining several variants of
RelU function in the complex domain. As regards ReLU split activation, ReLU
Type-B activation function is not interesting since modulating the amplitude
without the phase makes the activation converge to a linear function.
Nevertheless, Type-A is successfully implemented and is therefore defined as
Complex ReLU or C-ReLU [9, 10].

Another ReLU adaptation is also proposed and is called z-ReLU, which lets the
output as the the input for points in the first quadrants in the complex
domain [23]:

$z-ReLU=\begin{cases}z\hskip 36.135ptif\hskip 7.22743pt0<\arg(z)<\pi/2\\\
0\hskip 36.135ptif\hskip 7.22743ptotherwise\end{cases}$ (12)

Other ReLU adaptations attenuate the magnitude of the complex variable and
preserve the phase. One function is the popular function modReLU [14], which
defines a radius $b$ in the complex plane along which the function output is
$0$ and is defined as

$modReLU=\begin{cases}ReLU(|z|+b)\frac{z}{|z|}\hskip 36.135ptif\hskip
7.22743pt|z|\geq b\\\ 0\hskip 104.79128ptif\hskip
7.22743ptotherwise\end{cases}$ (13)

another ReLU extension is the cardioid function [24], which also leave the
phase unchanged and modifies the magnitude based on the phase, and is defined
as

$f_{cardioid}=\frac{(1+\cos(\arg(z)))z}{2}$ (14)

All previously mentioned Complex ReLU variants were implemented and tested by
[9, 10]

###  4.5 Output Layer Activation Function

The nature of the final output neuron depends on the CVNN objective. That is,
CVNN output may be required in the real domain $\mathbb{R}$. For Instance, in
classification tasks, data labels may be provided as real-valued integers.
Further, CVNNs may be adopted for solving linearly non-separable real-valued
problems that cannot be solved by RVNNs or for computational efficiency [2].
Therefore, one proposed solution is to leave the complex output activation
unchanged and cast labels with the transformation: $\text{for label
}c\in\mathbb{R},c\to c+ic$ [9].

Another solution implies that activation functions specific for output layer
are required to map complex input to real output. Thus, assuming
$z=x+iy=|z|\exp(i\arg(z))$, input to output layer activation function $f$,
conversion to real domain is done by [2, 12]

$f(z)=|z|\\\ $ (15)

or

$f(z)=(x-y)^{2}$ (16)

Another possible mapping function is the softmax function which maps integers
to the interval $[0,1]$. the complex components of $z$ can be processed with
real-valued operations ( such as averaging, multiplication or addition ) and
then fed to the softmax activation function. The softmax function can also be
applied to $z$ components independently and resultant values are then combined
with real-valued operation. More details are found in [9].

##  5 Learning and Optimization in CVNN

Neural network learning is the process of attractively tuning network weights
in order to optimize a certain learning objective such as minimizing a loss
function. The target is to achieve optimal set of weights that generalizes
best to out-of-sample data. For CVNNs, the complex weights are tuned to
minimize a real-valued loss function, as the minimum of two complex numbers is
not feasible. There are many options to specify CVNN minimization criterion
depending on network application and relevance of complex components.

###  5.1 Complex Compatible Loss Function

In the context of supervised learning, the ground truth and desired network
output is already known. For complex-valued regression tasks, given the
desired and predicted network output, $d\in\mathbb{C}$ and $o\in\mathbb{C}$,
respectively, the error function is naturally given by [2, 3]

$E=\frac{1}{2}e.\bar{e}=\frac{1}{2}|e|^{2},\hskip 36.135pt\text{where}\hskip
14.45377pte=d-o\in\mathbb{C}$ (17)

The regression loss of $N$ output neuron is then defined as

$\mathcal{L}=\sum_{k=1}^{N}E_{k}=\frac{1}{2}\sum_{k=1}^{N}|e_{k}|^{2}$ (18)

The above expression 17 only represents the error in amplitude. However, for
complex valued signals, the phase also carries important information.
Therefore, another error function based on logarithm minimization was proposed
[2]

$\begin{split}e_{log}=\left(log(o_{k})-log(d_{k})\right)\overline{\left(log(o_{k})-log(d_{k})\right)}\\\
E_{log}=\frac{1}{2}|e_{log}|^{2}=\frac{1}{2}\left[\left(log\frac{r_{d}}{r_{o}}\right)^{2}+\left(\phi_{d}-\phi_{o}\right)^{2}\right]\end{split}$
(19)

Where $r_{d}$, $r_{o}$ and $\phi_{d}$, $\phi_{d}$ are the magnitudes and
phases of desired and predicted outputs, respectively, and the logarithmic
loss function is computed as a summation of $E_{l}og$ of output neurons.

For classification and semantic segmentation tasks, the loss function does not
usually support complex-valued output, such as cross entropy function. One
possible solution is to map complex output to the real domain using output
activation function, as described in 4.5. Another solution was proposed for
evaluating categorical cross-entropy loss function for complex-valued outputs
[25]. This solution compares the real and imaginary parts of the predicted
output independently with labels and computes the loss function using the
equation

$\mathcal{L}_{ACE}=\frac{1}{2}\left(\mathcal{L}_{CE}(\Re{(o))},d)+\mathcal{L}_{CE}(\Im{(o)},d)\right)$
(20)

Where ${L}_{ACE}$ is the proposed average cross-entropy loss function for
complex values and ${L}_{CE}$ is the conventional cross-entropy function
utilized for classification tasks in RVNNs. All of the above loss functions
were implemented by [9].

###  5.2 CVNN Learning Approaches

The learning algorithm of conventional real-valued neural networks, which is
backpropagation with gradient decent, requires optimizing the network using
gradient or any similar partial-derivative technique. CVNN training, on the
other hand, can be done using two possible approaches. One approach follows
the same method of RVNN training with slight modifications to implement
gradient decent and backpropagation in complex domain. The second approach
implements error backpropagation without gradient decent and update weights
analytically for each training sample [3].

####  5.2.1 Gradient-based Learning

Similar to RVNN learning process, CVNN gradient-based learning calculates the
error in the forward pass and then the error is backpropagated to each neuron
of the network and weights are adjusted in backward pass. Adjusting CVNN
weights with gradient decent requires evaluating derivative of the real-valued
loss function with respect to complex-valued network weights and thus,
applying chain rule. Due to complex differentiability concerns illustrated in
4.1, standard complex differentiation only exists for holomorphic functions.
However, CVNN utilize non-holomorphic function in two ways:

  * •

Real-valued loss function is non-holomorphic according to Liouville’s theorem
4.1.2.

  * •

Split-activation functions 4.3.1, extended from RVNN implementation, are non-
holomorphic.

However, the Wirtinger calculus, discussed in 4.2.1, is utilized to derive the
complex gradient for both holomorphic and non-holomorphic functions and
generalize the chain rule, as shown in 4.2.2, to successfully implement CVNNs
[3, 9, 2].

For further analysis of CVNN dynamics, consider the multi-layer feedforward
neural network shown in Figure 1 from [9]. The network has $L$ layers indexed
$0\leq l\leq L$ where $l=0$ corresponds to the input layer. Each layer $l$ has
$N_{l}$ neurons indexed $1\leq n\leq N_{l}$. The parameter $w^{(l)}_{nm}$ is
the weight of the $n^{th}$ neuron of layer $l-1$ connected to the $m^{th}$ of
layer $l$. The symbol $\sigma$ indicates the complex activation function. The
output of layer $l$ and input to layer $l+1$ is
$X^{(l)}_{n}=\sigma\left(V^{(l)}_{n}\right)$ where
$V^{(l)}_{n}=\sum_{m=1}^{N_{l-1}}w^{(l)}_{nm}X^{(l-1)}_{m}$. The error
function of neuron $n$ of output layer is denoted $E_{n}$ and the loss
function is given by $\mathcal{L}=\sum_{n=1}^{N_{L}}E_{n}$

Figure 1: Feed-forward Neural Network Diagram

##### Complex Backpropagation

The complex gradient of a complex-valued function $f$, relative to complex
variable $z=x+iy$ and $f$ is differentiable with respect to real and imaginary
parts, is given by [26, 27, 28]:

$\nabla_{z}f(z,\bar{z})=\frac{\partial f}{\partial x}+i\frac{\partial
z}{\partial y}$ (21)

Which is, according to Wirtinger calculus 4.2.1, is evaluated as [9]

$\nabla_{z}f(z,\bar{z})=\overline{\left(\frac{\partial f}{\partial
z}+\frac{\partial
f}{\partial\bar{z}}\right)}=2\frac{\partial\Re(f)}{\partial\bar{z}}$ (22)

For CVNN gradient decent implementation, the loss function is real valued,
therefore it is natural to define gradient $\nabla_{z}f=2\frac{\partial
f}{\partial\bar{z}}$.

Recall that in RVNN learning, starting at the output layer, the the partial
derivative of the loss function with respect to a single network weight is
found with chain rule as follows

$\frac{\partial\mathcal{L}}{\partial
w^{(l)}_{nm}}=\sum_{n=1}^{N_{L}}\frac{\partial E_{n}}{\partial
X^{(L)}_{n}}\frac{\partial X^{(L)}_{n}}{\partial V^{(L)}_{n}}\frac{\partial
V^{(L)}_{n}}{\partial w^{(l)}_{nm}}$ (23)

and the derivative components are evaluated by passing the network backwards
with the chain rule. In case of CVNN learning, the chain rule is changed using
4.2.2 and the derivative of error function with respect to complex neuron
weight is

$\frac{\partial E}{\partial w}=\frac{\partial E}{\partial X}\frac{\partial
X}{\partial V}\frac{\partial V}{\partial w}+\frac{\partial E}{\partial
X}\frac{\partial X}{\partial\bar{V}}\frac{\partial\bar{V}}{\partial
w}+\frac{\partial E}{\partial\bar{X}}\frac{\partial\bar{X}}{\partial
V}\frac{\partial V}{\partial w}+\frac{\partial
E}{\partial\bar{X}}\frac{\partial\bar{X}}{\partial\bar{V}}\frac{\partial\bar{V}}{\partial
w}$ (24)

Since $E$ is real-valued, the conjugate rule applies as

$\frac{\partial E}{\partial\bar{X}}=\overline{\left(\frac{\partial E}{\partial
X}\right)}\hskip 7.22743pt,\hskip
7.22743pt\frac{\partial\bar{X}}{\partial\bar{V}}=\overline{\left(\frac{\partial
X}{\partial V}\right)}\hskip 7.22743pt,\hskip 7.22743pt\frac{\partial
X}{\partial\bar{V}}=\overline{\left(\frac{\partial\bar{X}}{\partial
V}\right)}$ (25)

As explained in [9], for $h\leq l-2$, the formulas for obtaining complex
derivatives can be given by

$\frac{\partial V^{(l)}_{n}}{\partial
w^{(h)}_{jk}}=\begin{cases}X^{(l-1)}_{j}&h=l\\\ w^{(l)}_{jk}\frac{\partial
X^{(l-1)}_{j}}{\partial V^{(l-1)}_{j}}\frac{\partial V^{(l-1)}_{j}}{\partial
w^{(l-1)}_{jk}}&h=l-1\\\ \sum_{i=1}^{N_{l-1}}w^{(l)}_{ik}\left[\frac{\partial
X^{(l-1)}_{i}}{\partial V^{(l-1)}_{i}}\frac{\partial V^{(l-1)}_{i}}{\partial
w^{(l-1)}_{jk}}+\frac{\partial
X^{(l-1)}_{i}}{\partial\bar{V}^{(l-1)}_{i}}\frac{\partial\bar{V}^{(l-1)}_{i}}{\partial
w^{(l-1)}_{jk}}\right]&h\leq l-2\end{cases}$ (26)

And using conjugate properties

$\frac{\partial\bar{V}^{(l)}_{n}}{\partial w^{(h)}_{jk}}=\begin{cases}0&h=l\\\
\bar{w}^{(l)}_{jk}\frac{\partial\bar{X}^{(l-1)}_{j}}{\partial
V^{(l-1)}_{j}}\frac{\partial V^{(l-1)}_{j}}{\partial w^{(l-1)}_{jk}}&h=l-1\\\
\sum_{i=1}^{N_{l-1}}\bar{w}^{(l)}_{ik}\left[\frac{\partial\bar{X}^{(l-1)}_{i}}{\partial
V^{(l-1)}_{i}}\frac{\partial V^{(l-1)}_{i}}{\partial
w^{(l-1)}_{jk}}+\frac{\partial\bar{X}^{(l-1)}_{i}}{\partial\bar{V}^{(l-1)}_{i}}\frac{\partial\bar{V}^{(l-1)}_{i}}{\partial
w^{(l-1)}_{jk}}\right]&h\leq l-2\end{cases}$ (27)

Using the fact that $\frac{\partial\bar{V}^{(l-1)}_{j}}{\partial
w^{(l-1)}_{jk}}=0$, then equation 24 can be re-written as

$\begin{split}\frac{\partial E_{n}}{\partial w^{(l)}_{ji}}=\frac{\partial
E_{n}}{\partial X^{(l)}_{i}}\frac{\partial X^{(l)}_{i}}{\partial
V^{(l)}_{i}}\frac{\partial V^{(l)}_{i}}{\partial w^{(l)}_{ji}}+\frac{\partial
E_{n}}{\partial\bar{X}^{(l)}_{i}}\frac{\partial\bar{X}^{(l)}_{i}}{\partial
V^{(l)}_{i}}\frac{\partial V^{(l)}_{i}}{\partial w^{(l)}_{ji}}\\\
=\frac{\partial E_{n}}{\partial X^{(l)}_{i}}\frac{\partial
X^{(l)}_{i}}{\partial V^{(l)}_{i}}X^{(l-1)}_{j}+\frac{\partial
E_{n}}{\partial\bar{X}^{(l)}_{i}}\frac{\partial\bar{X}^{(l)}_{i}}{\partial
V^{(l)}_{i}}X^{(l-1)}_{j}\end{split}$ (28)

Analyzing the error derivative with respect to layer input, the term
$E_{n}/\partial X^{(l)}_{i}$ becomes

$\frac{\partial E_{n}}{\partial X^{(l)}_{i}}=\begin{cases}\frac{\partial
E_{n}}{\partial X^{(L)}_{n}}&l=L\\\ \frac{\partial E_{n}}{\partial
X^{(L)}_{n}}\frac{\partial X^{(L)}_{n}}{\partial
V^{(L)}_{n}}w^{(L)}_{in}+\frac{\partial
E_{n}}{\partial\bar{X}^{(l)}_{i}}\frac{\partial\bar{X}^{(l)}_{i}}{\partial
V^{(L)}_{n}}w^{(L)}_{in}&l=L-1\\\ \sum_{k=1}^{N_{l+1}}\left[\frac{\partial
E_{n}}{\partial X^{(l+1)}_{k}}\frac{\partial X^{(l+1)}_{k}}{\partial
V^{(l+1)}_{k}}w^{(l+1)}_{ik}+\frac{\partial
E_{n}}{\partial\bar{X}^{(l+1)}_{k}}\frac{\partial\bar{X}^{(l+1)}_{k}}{\partial
V^{(l+1)}_{k}}w^{(l+1)}_{ik}\right]&l\leq L-2\end{cases}$ (29)

That is, $\partial E_{n}/\partial\bar{X}^{(l)}_{i}$ can be obtained using the
conjugate rule of 25. With equations 26,27 and 29, the backpropagation
algorithm is fully defined in the complex domain.

####  5.2.2 Non-gradient-based Learning

This learning process of CVNN is based on MVN neural element with fully-
complex phasor activation function, discussed in 4.3.2. Unlike the gradient-
based approach, the MVN learning is derivative free and CVNN weights are
updated with the error correction rule explained in [19]. Upon determining the
neuron error, the neural learning is done through a simple movement along the
unit circle, i.e. only neuron phase is updated analytically with no gradient
involved [3]

Considering the same feed-forward network architecture of Figure 1, the
network weights are initialized and the forward processing is evaluated. The
complex network error is then estimated and is backpropagated starting from
the output layer and the neuron error in each layer is used to update the
weight of the corresponding neuron. Assume the error of neuron $m$ in the
layer $l$ is $e_{m}^{(l)}$, the weight correction formula for hidden and
output layer is

$\tilde{w}_{nm}^{(l)}=w_{nm}^{(l)}+\frac{C_{m}^{(l)}}{N_{l-1}+1}e_{m}^{(l)}\bar{\tilde{X}}^{l-1}_{n}$
(30)

And for the input layer, i.e. $l=0$

$\tilde{w}_{nm}^{(0)}=w_{nm}^{(0)}+\frac{C_{m}^{(0)}}{n+1}e_{m}^{(0)}X^{0}_{n}$
(31)

Where $C_{m}^{l}$ is the learning rate for the $m^{th}$ of layer $l$. The
learning process continues until a termination condition is met, which is
generally average error magnitude of each layer falls under a threshold,
sufficient to terminate the weights update [3].

##  6 Complex Batch Normalization

Batch normalization technique was originally proposed for RVNNs to accelerate
learning and optimize the model [29]. As for CVNNs, special formulation is
required to apply batch normalization in the complex domain. Standardizing an
array of complex numbers such that their mean is 0 and variance is 1 cannot
guarantee that resultant real and imaginary components are circularly
distributed. In order to obtain equal variance in both components, the
normalization problem can be assumed as whitening 2D vectors such that:
$z=a+ib\in\mathbb{C}\to\textit{{x}}=(a,b)\in\mathbb{R}^{2}$ [10, 9]. That is,
the complex batch normalization steps are explained as follows:

For a complex variable x with estimated mean
$\tilde{\boldsymbol{\mu_{x}}}\in\mathbb{R}^{2}$ and a covariance matrix
$\tilde{\boldsymbol{\mathit{V}}}\in\mathbb{R}^{2\times 2}$, the normalized
output $\hat{\textit{{x}}}$ is given by

$\hat{\textit{{x}}}=\tilde{\boldsymbol{\mathit{V}}}^{-\frac{1}{2}}(\textit{{x}}-\tilde{\boldsymbol{\mu_{x}}})$
(32)

Where

$\tilde{\boldsymbol{\mathit{V}}}=\begin{bmatrix}V_{rr}&V_{ri}\\\
V_{ir}&V_{ii}\end{bmatrix}=\begin{bmatrix}\text{Cov}\left(\Re{(\textit{{x}})},\Re{(\textit{{x}})}\right)&\text{Cov}\left(\Re{(\textit{{x}})},\Im{(\textit{{x}})}\right)\\\
\text{Cov}\left(\Im{(\textit{{x}})},\Re{(\textit{{x}})}\right)&\text{Cov}\left(\Im{(\textit{{x}})},\Im{(\textit{{x}})}\right)\end{bmatrix}$
(33)

This normalization procedure allows decorrelation of the real and imaginary
components to avoid co-adaptation and possible overfitting [10].

As for batch normalization layer initialization, a moving variance variable
$\boldsymbol{\mathit{V}}^{{}^{\prime}}$ is initialized to
$\mathbf{I}/\sqrt{2}$ and a moving mean $\boldsymbol{\mu_{x}^{{}^{\prime}}}$
is initialized to $\mathbf{0}$, to be used later by the moving average rule.

During the training phase, $\tilde{\boldsymbol{\mu_{x}}}$ and
$\tilde{\boldsymbol{\mathit{V}}}$ are estimated for each input batch along the
desired principal component and then the normalization output is estimated
using 32.

Similar to batch normalization in RVNNs, the normalized output can be shifted
and scaled using two learnable parameters, $\boldsymbol{\beta}$ and
$\boldsymbol{\gamma}$. The shift parameter
$\boldsymbol{\beta}=\left(\Re{(\boldsymbol{\beta})}\right),\left(\Im{(\boldsymbol{\beta})}\right)^{T}$
has two learnable components and is initialized to $\mathbf{0}$. The scale
parameter $\boldsymbol{\gamma}$ is a $2\times 2$ matrix has three components
to learn since off-diagonal elements are equal, and is initialized to
$\mathbf{I}/\sqrt{2}$. Therefore, the complex batch normalization output upon
scaling and shifting is given by

$BN(\hat{\textit{{x}}})=\boldsymbol{\gamma}\hat{\textit{{x}}}+\boldsymbol{\beta}$
(34)

In addition, in the moving average step, batch normalization layer also use
running averages with momentum $\alpha$ to keep an estimate of mean and
variance during the training phase. The moving mean and moving variance are
iteratively updated as follows:

$\boldsymbol{\mu_{x}^{{}^{\prime}}}_{n+1}=\alpha\boldsymbol{\mu_{x}^{{}^{\prime}}}_{n}+(1-\alpha)\tilde{\boldsymbol{\mu_{x}}}_{n}$
(35)
$\boldsymbol{V^{{}^{\prime}}}_{n+1}=\alpha\boldsymbol{V^{{}^{\prime}}}_{n}+(1-\alpha)\tilde{\boldsymbol{V}}_{n}$
(36)

That is, in the inference phase, no batch statistics are computed and the
layer output is computed directly using 32 but with
$\boldsymbol{\mu_{x}^{{}^{\prime}}}$ and $\boldsymbol{V^{{}^{\prime}}}$.

##  7 Complex Weight Initialization

In neural networks learning, proper weight initialization is essential to
avoid the risk of vanishing or exploding gradients. A complex weight
initializer requires defining suitable variance for complex weights
components. Two formulations, based on either polar or rectangular complex
representations, were proposed for CVNN weight initialization [9, 10]. Both
formulations estimate the variance of complex parameters following a criterion
analyzed by [30], which ensures almost constant variance for layers input,
output and their gradients.

###  7.1 Complex Initialization from Polar Representation Perspective

Consider a complex weight $w=|w|\exp{(-i\phi)}=\Re{(w)}+\Im{(w)}$, where $|w|$
and $\phi$ are the weight magnitude and phase, respectively. The variance of
$w$ is given by

$\text{Var}(w)=\text{E}\left[w\bar{w}\right]-\left(\text{E}[w]\right)^{2}=\text{E}\left[|w|^{2}\right]-\left(\text{E}[w]\right)^{2}$
(37)

Assuming $w$ follows circular symmetric distribution, i.e. $\text{E}[w]=0$,
then 37 reduces to $\text{Var}(w)=\text{E}\left[|w|^{2}\right]$. Moreover,
symmetric distribution of $w$ implies normally distributed real and imaginary
components, that is, the magnitude $|w|$ follows Rayleigh distribution (Chi-
Squared distribution with two degrees of freedom). Thus, it is reasonable to
find $Var(w)$ in terms of $Var(|w|)$ as follows

$\text{Var}\left(|w|\right)=\text{E}\left[|w|^{2}\right]-(\text{E}\left[|w|\right])^{2}=\text{Var}(w)-(\text{E}\left[|w|\right])^{2}$
(38)

For a Rayleigh distribution with parameter $\sigma_{r}$, the mean and variance
of $|w|$ are given by

$\text{E}\left[|w|\right]=\sigma_{r}\sqrt{\frac{\pi}{2}}\hskip
14.45377pt,\hskip
14.45377pt\text{Var}\left(|w|\right)=\frac{4-\pi}{2}\sigma^{2}$ (39)

Therefore, the variance of $w$ becomes

$\text{Var}(w)=\text{Var}\left(|w|\right)+(\text{E}\left[|w|\right])^{2}=2\sigma_{r}^{2}$
(40)

According to [30], a good initialization compromise that maintains almost
constant variance of layers weights during training is that
$\text{Var}(w)=2/\left(N_{in}+N_{out}\right)$, where $N_{in}$ and $N_{out}$
are the number of input and out weights corresponding to each layer. In order
to satisfy this criterion, initializing weights magnitude in polar form
follows Rayleigh distribution with parameter
$\sigma_{r}=1/\sqrt{\left(N_{in}+N_{out}\right)}$. The phase of the complex
weights is initialized following Uniform distribution between $0$ and $\pi$.
Subsequently, multiplying the randomly generated magnitude with the generated
phasor gives initialized weight parameter [9, 10].

###  7.2 Complex Initialization from Rectangular Representation Perspective

According to the work of [9], the real and imaginary components of complex
weights can be initialized independently with a suitable constant varaince for
both components. Considering the network architecture of figure 1, under the
assumption of constant variance for all input features, statistically centered
weights and constant weights variance related to a single layer, the necessary
condition to keep constant information flow in forward and direction is

$\text{Var}X_{n}^{(l)}=\text{Var}X_{n}^{(l^{\prime})},\forall 1\leq l\leq
l^{\prime}\leq N$ (41)

And the necessary condition for constant flow in the backward direction is

$\text{Var}\left[\frac{\partial\mathcal{L}}{\partial
V_{n}^{(l)}}\right]=\text{Var}\left[\frac{\partial\mathcal{L}}{\partial
V_{n}^{(l^{\prime})}}\right],\forall 1\leq l\leq l^{\prime}\leq N$ (42)

The above conditions can only be jointly met if all network layers have the
same width so that the variance is given by $\text{Var}(w)=1/N$ [9].
Therefore, the criterion proposed by [30] provides a good trade-off for
$\text{Var}(w)=2/\left(N_{in}+N_{out}\right)$. That is, the weight
initialzation can follow a uniform distribution with this variance such that

$w\sim
U\left[-\frac{\sqrt{6}}{\sqrt{\left(N_{in}+N_{out}\right)}},\frac{\sqrt{6}}{\sqrt{\left(N_{in}+N_{out}\right)}},\right]$
(43)

Since real and imaginary parts are statistically independent, the weight
variance can be expressed as

$\text{Var}(w)=\text{Var}\left(\Re{(w)}\right)+\text{Var}\left(\Im{(w)}\right)$
(44)

Thus, initializing weights components is done by

$\Re{(w)}=\Im{(w)}\sim
U\left[-\frac{\sqrt{3}}{\sqrt{\left(N_{in}+N_{out}\right)}},\frac{\sqrt{3}}{\sqrt{\left(N_{in}+N_{out}\right)}},\right]$
(45)

##  8 CVNN Implementation Efforts

The implementations of complex-valued deep neural networks have always been
related to specific structures and application scenarios. That is, CVNNs
mathematical formulation is difficult to implement and yet popular Python
libraries for implementing neural networks, which are PyTorch and TensorFlow,
do not fully support CVNN model creation . In fact, TensorFlow supports the
use of complex data types for automatic differentiation algorithm, it has been
verified by [9] that TensorFlow calculates complex gradients using Wirtinger
Calculus. On the other hand, PyTorch added complex data types as BETA starting
with version 1.6 and also supported complex differentiation with Wirtinger
Calculus. Later on with version 1.12, complex convolution functionality was
added by PyTorch. These are most recent CVNN developments by PyTorch and
TensorFlow, which indicates tendency towards CVNN support.

There are several libraries and code repositories available to develop CVNN
models. The very first and probably most popular CVNN implementation is the
GitHub code 111https://github.com/ChihebTrabelsi/deep_complex_networks which
reproduce experiments presented by the paper [10]. This work was originally
proposed to solve real-valued problems using complex-valued deep neural
networks. The authors provided analysis and building blocks required to design
complex-valued neural networks structures, particularly complex convolutional
residual networks and recurrent neural networks, targeting richer
representational capacity and more efficient memory retrieval mechanisms. The
complex components and algorithms presented are complex batch normalization in
section 6, complex weights initialization in section 7 and complex
convolutions. Their model was designed to process complex-valued information
with double dimensional real-valued data types and that imaginary parts are
learnt during training for real-valued data. Nonetheless, the implemented code
(662 stars and 272 forks ) uses Theano back-end, which is no longer
maintained. Motivated by this, many other implementation attempts, based on
either TensorFlow Keras or PyTorch, reproduced the work of [10]. Such attempts
include the library with GitHub code repository [31] (122 stars) which uses
Keras with TensorFlow back-end. Another re-implementation attempt is the
library with GitHub code
222https://github.com/wavefrontshaping/complexPyTorch/tree/master (459 stars),
which uses PyTorch v1.7 with fully complex tensors.

Furthermore, another library that enables implementing comprehensive CVNN
models was proposed by [9]. This library natively supports complex number
data-types and supports a wide range of split-type and fully complex
activation functions. Moreover, the library also supports special complex
components proposed earlier by [10], which are complex batch normalization and
complex initialization. The code uses TensorFlow as a back-end and was
published in Zenodo 333https://doi.org/10.5281/zenodo.7303587 and GitHub (107
stars) [32]. The library also support implementing RVNN models and gives
results comparable to those obtained by Conventional TensorFlow based models.

Other codes repositories involving complex-valued neural networks are also
available. However, such repositories are related to specific applications and
neural networks variants with models that are less generalized. For this
reason, the mathematical analysis of this article relies on the theory and
implementations by [9] and [10].

##  9 Conclusion and Possible Research Directions

The research and advancements on CVNNs have not been very active compared to
RVNNs. This is due to several challenges and difficulties associated with CVNN
architectures and learning mechanisms. This section highlights future research
prospects and potential directions towards establishing effective CVNN models.

One of the most obvious limitations in CVNN training is the design of complex
activation function. Due to Liouville’s theorem, complex activations
compromise one crucial property, either analytic or bounded. Moreover, split-
type activation functions in polar form does not account for phase information
during training. Therefore, an active research area is the design of fully-
complex nonlinear activation function that is differentiable over widely
applicable complex plane. Another proposed research direction is expressing
CVNN learning rate in the complex domain. The complex-valued learning rate
extends the gradient search area in the complex domain and allows escaping
saddle points. For this reason, selection of adaptive generalized complex-
valued learning rate can be beneficial for CVNNs. As regards learning
algorithms, the difficulties associated with complex differentiability
suggests exploring learning algorithms that does not involve gradient
computations [2]. Another reason for slow development of CVNN models is the
lack of public libraries for CVNN training. Even standard deep learning
libraries, based TensorFlow or PyTorch, does not fully support CVNN
implementation. Therefore, new deep learning libraries are required to
optimize and train CVNN models. Moreover, more generalized complex building
blocks, based on proposing statistical distributions for generating complex
parameters such as complex random initialization and complex batch
normalization, is also an active research topic. Hence, future studies are
required to develop more consistent building blocks that are less sensitive to
random complex parameters [3]. Further, recalling the biological motivation
behind CVNN systems, an interesting research area is the development of
biologically plausible neuronal model that mimics the human neuronal
information transmission. In other words, such novel biological CVNN systems
may be beneficial for nonlinear real-world problems that favor controlling the
neuron firing time and magnitude.

In conclusion, this work summarized the theory related to CVNN dynamics and
corresponding implementations. Deep learning models based on CVNNs can be
successfully applied following the aforementioned formulations and learning
algorithms. This article is good start for understanding CVNN different
structures and potential research directions.

## References

  * [1] Akira Hirose.  Complex-valued neural networks, volume 400.  Springer Science & Business Media, 2012.
  * [2] ChiYan Lee, Hideyuki Hasegawa, and Shangce Gao.  Complex-valued neural networks: A comprehensive survey.  IEEE/CAA Journal of Automatica Sinica, 9(8):1406–1426, 2022.
  * [3] Joshua Bassey, Lijun Qian, and Xianfang Li.  A survey of complex-valued neural networks.  arXiv preprint arXiv:2101.12249, 2021.
  * [4] Alexander Fuchs, Johanna Rock, Mate Toth, Paul Meissner, and Franz Pernkopf.  Complex-valued convolutional neural networks for enhanced radar signal denoising and interference mitigation.  In 2021 IEEE Radar Conference (RadarConf21), pages 1–6. IEEE, 2021\.
  * [5] Theresa Scarnati and Benjamin Lewis.  Complex-valued neural networks for synthetic aperture radar image classification.  In 2021 IEEE Radar Conference (RadarConf21), pages 1–6. IEEE, 2021\.
  * [6] Hyeong-Seok Choi, Jang-Hyun Kim, Jaesung Huh, Adrian Kim, Jung-Woo Ha, and Kyogu Lee.  Phase-aware speech enhancement with deep complex u-net.  In International Conference on Learning Representations, 2018.
  * [7] Olga Fink, Enrico Zio, and Ulrich Weidmann.  Predicting component reliability and level of degradation with complex-valued neural networks.  Reliability Engineering & System Safety, 121:198–206, 2014.
  * [8] Hirofumi Tsuzuki, Mauricio Kugler, Susumu Kuroyanagi, and Akira Iwata.  An approach for sound source localization by complex-valued neural network.  IEICE TRANSACTIONS on Information and Systems, 96(10):2257–2265, 2013.
  * [9] Jose Agustin Barrachina, Chengfang Ren, Gilles Vieillard, Christele Morisseau, and Jean-Philippe Ovarlez.  Theory and implementation of complex-valued neural networks.  arXiv preprint arXiv:2302.08286, 2023.
  * [10] Chiheb Trabelsi, Olexa Bilaniuk, Ying Zhang, Dmitriy Serdyuk, Sandeep Subramanian, João Felipe Santos, Soroush Mehri, Negar Rostamzadeh, Yoshua Bengio, and Christopher J Pal.  Deep complex networks.  arXiv e-prints, pages arXiv–1705, 2017.
  * [11] Akira Hirose and Shotaro Yoshida.  Generalization characteristics of complex-valued feedforward neural networks in relation to signal coherence.  IEEE Transactions on Neural Networks and learning systems, 23(4):541–551, 2012.
  * [12] Tohru Nitta.  Solving the xor problem and the detection of symmetry using a single complex-valued neuron.  Neural Networks, 16(8):1101–1105, 2003.
  * [13] Tony A Plate.  Holographic Reduced Representation: Distributed representation for cognitive structures, volume 150.  CSLI Publications Stanford, 2003.
  * [14] Martin Arjovsky, Amar Shah, and Yoshua Bengio.  Unitary evolution recurrent neural networks.  In International conference on machine learning, pages 1120–1128. PMLR, 2016.
  * [15] David P Reichert and Thomas Serre.  Neuronal synchrony in complex-valued deep networks.  arXiv preprint arXiv:1312.6115, 2013.
  * [16] Taehwan Kim and Tulay Adali.  Fully complex backpropagation for constant envelope signal processing.  In Neural Networks for Signal Processing X. Proceedings of the 2000 IEEE Signal Processing Society Workshop (Cat. No. 00TH8501), volume 1, pages 231–240. IEEE, 2000.
  * [17] Akira Hirose.  Proposal of fully complex-valued neural networks.  In [Proceedings 1992] IJCNN International Joint Conference on Neural Networks, volume 4, pages 152–157. IEEE, 1992.
  * [18] Ken Kreutz-Delgado.  The complex gradient operator and the cr-calculus.  arXiv preprint arXiv:0906.4835, 2009.
  * [19] NN Aizenberg, Yu L Ivas’ kiv, DA Pospelov, and GF Khudyakov.  Multivalued threshold functions: Ii. synthesis of multivalued threshold elements.  Cybernetics, 9(1):61–77, 1973.
  * [20] Akira Hirose.  Dynamics of fully complex-valued neural networks.  Electronics letters, 16(28):1492–1494, 1992.
  * [21] George M Georgiou and Cris Koutsougeras.  Complex domain backpropagation.  IEEE transactions on Circuits and systems II: analog and digital signal processing, 39(5):330–334, 1992.
  * [22] Taehwan Kim and Tülay Adalı.  Approximation by fully complex multilayer perceptrons.  Neural computation, 15(7):1641–1666, 2003.
  * [23] Nitzan Guberman.  On complex valued convolutional neural networks.  arXiv preprint arXiv:1602.09046, 2016.
  * [24] Patrick Virtue, X Yu Stella, and Michael Lustig.  Better than real: Complex-valued neural nets for mri fingerprinting.  In 2017 IEEE international conference on image processing (ICIP), pages 3953–3957. IEEE, 2017.
  * [25] Yice Cao, Yan Wu, Peng Zhang, Wenkai Liang, and Ming Li.  Pixel-wise polsar image classification via a novel complex-valued deep fully convolutional network.  Remote Sensing, 11(22):2653, 2019.
  * [26] Md Faijul Amin, Muhammad Ilias Amin, Ahmed Yarub H Al-Nuaimi, and Kazuyuki Murase.  Wirtinger calculus based gradient descent and levenberg-marquardt learning algorithms in complex-valued neural networks.  In International Conference on Neural Information Processing, pages 550–559. Springer, 2011.
  * [27] Henry Leung and Simon Haykin.  The complex backpropagation algorithm.  IEEE Transactions on signal processing, 39(9):2101–2104, 1991.
  * [28] Hualiang Li and Tülay Adalı.  Complex-valued adaptive signal processing using nonlinear functions.  EURASIP Journal on Advances in Signal Processing, 2008:1–9, 2008\.
  * [29] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In International conference on machine learning, pages 448–456. pmlr, 2015.
  * [30] Xavier Glorot and Yoshua Bengio.  Understanding the difficulty of training deep feedforward neural networks.  In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249–256. JMLR Workshop and Conference Proceedings, 2010.
  * [31] Jesper Sören Dramsch and Contributors.  Complex-valued neural networks in keras with tensorflow, 2019.
  * [32] J Agustin Barrachina.  Negu93/cvnn: Complex-valued neural networks, November 2022.