# Physics-Informed Neural Networks: Minimizing Residual Loss with Wide
Networks and Effective Activations

Nima Hosseini Dashtbayaz1 Ghazal Farhani2,111Corresponding Author Boyu
Wang1,∗&Charles X. Ling1 1Department of Computer Science, University of
Western Ontario
2National Research Council of Canada
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

The residual loss in Physics-Informed Neural Networks (PINNs) alters the
simple recursive relation of layers in a feed-forward neural network by
applying a differential operator, resulting in a loss landscape that is
inherently different from those of common supervised problems. Therefore,
relying on the existing theory leads to unjustified design choices and
suboptimal performance. In this work, we analyze the residual loss by studying
its characteristics at critical points to find the conditions that result in
effective training of PINNs. Specifically, we first show that under certain
conditions, the residual loss of PINNs can be globally minimized by a wide
neural network. Furthermore, our analysis also reveals that an activation
function with well-behaved high-order derivatives plays a crucial role in
minimizing the residual loss. In particular, to solve a $k$-th order PDE, the
$k$-th derivative of the activation function should be bijective. The
established theory paves the way for designing and choosing effective
activation functions for PINNs and explains why periodic activations have
shown promising performance in certain cases. Finally, we verify our findings
by conducting a set of experiments on several PDEs. Our code is publicly
available at https://github.com/nimahsn/pinns_tf2.

##  1 Introduction

The success of deep learning in a wide variety of tasks has motivated its
application in scientific domains as well Sirignano and Spiliopoulos (2018);
Reiser et al. (2022); Li et al. (2020). PINNs Raissi et al. (2017) in
particular are designed to solve differential equations as an alternative to
traditional solvers, benefiting from discretization-free construction and the
vast availability of machine learning tools and techniques. As a result, PINNs
have been deployed in various physics and engineering problems, such as
solving inverse scattering problems in photonics Chen et al. (2020a), flow
problems in fluid dynamics Cai et al. (2021), and computational
neuromusculoskeletal models raised in biomedical and rehabilitation sciences
Zhang et al. (2022).

Consider a general-form PDE with a Dirichlet boundary condition such as

$\displaystyle\mathcal{D}\left[u\right](x)=f(x)\;$ $\displaystyle x\in\Omega$
(1) $\displaystyle u(x)=g(x)\;$ $\displaystyle x\in\partial\Omega$ ,

where $u$ is the solution of the PDE on a bounded domain
$\Omega\subset\mathbb{R}^{d}$ of $d$ independent variables with boundaries
$\partial\Omega$, $f$ and $g$ are known functions, and $\mathcal{D}$ is a
differential operator. Here, the operator $\mathcal{D}$ expresses the physical
rules governing $u$ through a differential expression. PINNs are then trained
to respect the underlying physical dynamics given in $\mathcal{D}$ by
minimizing the residual loss

$L_{r}=\sum_{x\in\mathbf{x}}l(\mathcal{D}\left[\hat{u}\right](x)-f(x)),$ (2)

where $\hat{u}$ is a neural network approximation of $u$, $l$ is an error
function such as squared error, and $\mathbf{x}$ is a set of training
collocation points in $\Omega$. To guarantee a unique solution, boundary (and
initial) conditions are also imposed by adding other supervised loss terms,
referred to as boundary loss, trained with boundary data sampled from
$\partial\Omega$. The resulting loss function can then be treated as a multi-
objective optimization task Raissi et al. (2017).

While proven effective, training PINNs is often a challenging task. These
challenges usually originate from either the discrepancy between the residual
loss and the boundary loss Wang et al. (2020b); Farhani et al. (2022); Wang et
al. (2020a) or the nature of the residual loss Krishnapriyan et al. (2021);
Wang et al. (2022). Notably, as Eq. 2 involves differentiation over a neural
network with respect to (w.r.t.) its inputs, the outputs of the network
undergo a significant structural transformation. To better understand the
aforementioned process and its implications, let us consider a simple
differential operator $\frac{\partial u}{\partial x}$ of a single independent
variable and an $L$-layer feed-forward network with an activation function
$\sigma$. One can find that the application of this differential operator on
the neural network, $\mathcal{D}[\hat{u}]$, is given by

$\mathcal{D}\left[\hat{u}\right]=W_{L}^{\top}\times(\sigma^{\prime}(G_{L-1})\circ
W_{L-1})^{\top}\times\dots\times(\sigma^{\prime}(G_{1})\circ W_{1})^{\top},$
(3)

where $W_{i}$ and $G_{i}$ are weights and linear outputs of layer $i$, and
$\circ$ and $\times$ denote element-wise (Hadamard) and matrix products. In
contrast, the original neural network $\hat{u}$ can be defined recursively as

$\hat{u}(x)=G_{L}(x),\;G_{i}(x)=\sigma(G_{i-1}(x))\times W_{i}+b_{i}.$

Eq. 3 shows how the differentiation transforms the outputs of a neural
network. Firstly, note that the simple recursive relation between the layers
of a feed-forward network is disrupted by applying $\mathcal{D}$, and
additional element-wise products with weights emerge as well. More
significantly, we observe that the derivative $\sigma^{\prime}$ of the
activation function appears in the outputs. This presence of $\sigma^{\prime}$
in PINNs highlights the importance of an activation function with well-behaved
derivatives in the model’s expressive power in learning $\mathcal{D}$ and
likewise in the optimization process as it involves higher-order derivatives
of $\sigma$.

Altogether, the distinct characteristics of $\mathcal{D}[\hat{u}]$ and the
resulting residual loss, contribute to a problem that is quite different from
common supervised training tasks. Consequently, the existing theory around
loss functions and their characteristics cannot readily be applied to PINNs,
and the lack of understanding about PINNs and their optimization dynamics
leads to uninformed design choices and suboptimal performance even for
seemingly easy PDEs.

In this work, we focus on the residual loss and its landscape. Specifically,
we are interested in finding what neural networks and design choices enable
PINNs to globally minimize the residual loss. To this end, we study the
residual loss at a critical point of the network parameter space and search
for distinctive characteristics of a global minimum compared to other critical
points. Once these characteristics are identified, our investigation shifts
towards determining sufficient conditions within the network design, in
particular, width and activation function, that guarantee the existence of
global minima in the parameter space. Our findings underscore the importance
of the width and activation functions with well-behaved high-order derivatives
in acquiring a high expressive power in learning the differential operator.
Finally, we verify our findings by conducting extensive experiments on several
PDEs.

We summarize our contributions as follows. (1) We theoretically show that the
residual loss of PINNs can be globally minimized, given a two-layer neural
network with a width equal to or greater than the number of collocation
points. (2) Through our analysis, we establish that the residual loss for a
$k$-th order differential operator is optimally minimized when using an
activation function with a bijective $k$-th order derivative. We leverage this
theoretical foundation as a guideline for selecting activation functions,
justifying the choice of sinusoidal activations, and subsequently validating
their effectiveness through empirical demonstrations and experiments.

##  2 Related Works

###  2.1 Wide Neural Networks

Wide neural networks have historically been of significant interest in machine
learning. With classical results such as Universal Approximation and Gaussian
processes, and more recently, NTK theory Jacot et al. (2018), wide networks
have been studied to understand neural networks in certain regimes Chen et al.
(2020b); Lee et al. (2019). The optimal width of a neural network is also
studied for convergence guarantees Oymak and Soltanolkotabi (2020); Du et al.
(2019); Allen-Zhu et al. (2019); Nguyen and Mondelli (2020) and loss geometry
Safran and Shamir (2016); Nguyen and Hein (2017) with certain classes of
neural networks and optimizers. The convergence guarantees are often provided
for a width polynomial in the number of training samples and the number of
layers Allen-Zhu et al. (2019). Safran and Shamir Safran and Shamir (2016)
studied the basins of the loss function for wide two-layer ReLU networks,
showing that wider networks are initialized at a good basin with higher
probability. Nguyen and Hein (2017) also showed that most of the critical
points in a wide neural network are also global minima. The developed theory
in most of the aforementioned works cannot be directly applied to PINNs, as
they either rely on specific neural network formulations Nguyen and Hein
(2017); Nguyen and Mondelli (2020), which are disrupted by differentiation, or
certain hyper-parameters that are not effective for PINNs, such as ReLU
activation function Du and Hu (2019); Safran and Shamir (2016); Allen-Zhu et
al. (2019).

###  2.2 Periodic Activation Functions

Sitzmann et al. Sitzmann et al. (2020) proposed using sinusoidal activation
functions in neural networks with low-dimensional inputs for learning
differentiable signals. Notably, they also showed the capability of Sine
networks in solving Wave and Helmholtz PDEs with PINNs. Since then, few works
have explored the behaviour of neural networks with periodic activation
functions at initialization Belbute-Peres and Kolter (2022) and their
expressive power as function approximators Meronen et al. (2021). Meronen et
al. Meronen et al. (2021) studied the inductive bias introduced by periodic
activation functions on the neural network functional space, and showed that
such networks are less sensitive to input shifts.

###  2.3 Physics-Informed Neural Networks

Besides the applications of PINNs in solving various PDEs, there has been a
surge in analyzing the behaviour and pitfalls of PINNs in recent years,
especially from the optimization perspective Wang et al. (2020a); Liu et al.
(2020); Farhani et al. (2022). Using Neural Tangent Kernel (NTK) theory from
infinitely wide neural networks, Wang et al. (2020b) showed that high-
frequency terms in a PDE result in discrepancies in the convergence rate
between the loss objectives when trained with Gradient Descent, leading the
model to exhibit behaviours similar to spectral bias Rahaman et al. (2018).
Wang et al. Wang et al. (2020a) also showed similar results by studying the
magnitude of the loss gradients at different layers. Later on, the momentum
term was shown to address the discrepancy in optimization in the infinite-
width regime Farhani et al. (2022).

Many recent works alleviate the optimization challenges in PINNs and improve
their performance by assigning weights to each loss term Wang et al. (2020b);
McClenny and Braga-Neto (2020); Wight and Zhao (2020), designing new
architectures and embeddings Wong et al. (2022); Wang et al. (2020a, 2021);
Dong and Ni (2021), and using sophisticated training strategies such as
curriculum learning Krishnapriyan et al. (2021); Wang et al. (2022). Among
them, Wang et al. (2020b) and Wong et al. (2022), suggested mapping the inputs
to random or trainable Fourier features and the use of sinusoidal activation
functions to overcome the spectral bias and the convergence discrepancy.

##  3 Global Minima of the Residual Loss

In this section, we study the residual loss at its critical points to obtain
sufficient conditions for the existence of global minima. We present the
proofs for the lemmas and theorems in this section in the Appendix. First, we
introduce the notation and the setup used throughout this section.

###  3.1 Notation and Setup

We use
$\hat{u}_{\mathcal{W}}:\mathbb{R}^{d}\times\mathbb{R}^{|\mathcal{W}|}\rightarrow\mathbb{R}^{n_{L}}$
to denote an $L$-layer feed-forward neural network parameterized by
$\mathcal{W}=\\{W_{i},b_{i}\mid 1\leq i\leq
L,W_{i}\in\mathbb{R}^{n_{i-1}\times n_{i}},b_{i}\in\mathbb{R}^{n_{i}}\\}$,
where $n_{i}$ is the number of neurons in layer $i$, $n_{0}=d$, and $n_{L}=1$.
We drop $\mathcal{W}$ from $\hat{u}_{\mathcal{W}}$ for simplicity if there is
no ambiguity. The neural network $\hat{u}$ for an input
$x=(x_{1},\dots,x_{d})$ is formulated as

$\displaystyle\hat{u}(x)=G_{L}(x),$ (4) $\displaystyle
G_{i}(x)=F_{i-1}(x)\times W_{i}+b_{i}$ $\displaystyle\;\forall i\in\\{1,\dots
L\\},$ $\displaystyle F_{i}(x)=\sigma(G_{i}(x))$ $\displaystyle\;\forall
i\in\\{1,\dots L-1\\},$

where $\sigma$ is an activation function, and $F_{0}(x)=x$. We further define
$F^{(k)}_{i}(x)$ as

$F^{(k)}_{i}(x)=\sigma^{(k)}(G_{i}(x)),$

where $\sigma^{(k)}$ is the $k$-th derivative of $\sigma$. In the case of
$k=1$, we simply use $F^{\prime}_{i}$ and $\sigma^{\prime}$ instead. For a
batch $\mathbf{x}$ of $N$ samples, $F_{i}(\mathbf{x})$, $G_{i}(\mathbf{x})$,
and $F^{(k)}_{i}(\mathbf{x})$ are $N\times n_{i}$ matrices. Also, the matrix
power $W^{k}$ represents an element-wise power.

In a PINN, the neural network $\hat{u}$ is trained to approximate the solution
$u$ of a differential equation denoted as in Eq. 1. In this work, we assume
that $\mathcal{D}$ is a linear differential operator, i.e., the PDE is linear
in the derivatives of $u$ and $u$ itself. We reformulate the residual loss in
Eq. 2 to be a function of weights $\mathcal{W}$ and choose $l(r)=r^{2}$.

$\phi_{r}(\mathbf{x};\mathcal{W})=\sum_{x\in\mathbf{x}}l(\mathcal{D}\left[\hat{u}\right](x)-f(x)).$
(5)

Throughout the rest of this section, we consider a two-layer neural network
and a simple $k$-th order differential operator
$\mathcal{D}\left[u\right]=\frac{\partial^{k}u}{\partial x^{k}}$ for a single
independent variable $x$ (i.e., $d=1$). We generalize the results in this
section to more independent variables in the Appendix.

###  3.2 Residual Loss of a Two-layer PINN

To study the residual loss and its critical points, we first need to derive
the analytic formula for the residual loss and its gradients. The next two
lemmas, provide us with these tools by finding the differentiation
$\mathcal{D}\left[\hat{u}\right]$ and then deriving the gradients of the
resulting residual loss.

######  Lemma 1.

For a two-layer neural network $\hat{u}$ defined in Eq. 4, and a $k$-th order
differential operator $\mathcal{D}\left[u\right]=\frac{\partial^{k}u}{\partial
x^{k}}$ of a single independent variable $x$, $\mathcal{D}[\hat{u}]$ is

$\displaystyle\mathcal{D}[\hat{u}](x)=W_{2}^{\top}\times(F_{1}^{(k)}(x)\circ
W_{1}^{k})^{\top}.$

With the analytic formula for $\mathcal{D}[\hat{u}]$ in hand, it is easy to
plug it into Eq. 5 to get the residual loss. The next lemma derives the
gradients $\nabla_{W_{2}}\phi_{r}(\mathbf{x};\mathcal{W})$ of the residual
loss w.r.t. the weights of the last layer.

######  Lemma 2.

For $\hat{u}$ and $\mathcal{D}[\hat{u}]$ given in Lemma 1, gradients of the
residual loss w.r.t. the weights of the second layer over the training
collocation data $\mathbf{x}$ of $N$ samples are given by

$\nabla_{W_{2}}\phi_{r}(\mathbf{x};\mathcal{W})=W_{1}^{k}\circ[l^{\prime}(\mathcal{D}[\hat{u}](\mathbf{x})-f(\mathbf{x}))^{\top}\times
F^{(k)}_{1}(\mathbf{x})].$

######  Remark 1.

Lemmas 1 and 2 generalize the appearance of derivatives in the outputs of the
neural network as in Eq. 3, showing that a $k$-th order differential term
similarly contains the $k$-th derivative of the activation function. Thus,
activation functions with vanishing high-order derivatives, such as ReLU,
significantly reduce the network representation power in approximating the
residuals. Note that the gradients w.r.t. $W_{1}$ contain the $(k+1)$-th
derivative of the activation function, further highlighting the importance of
well-behaved derivatives in optimization.

In the following section, the gradients given in Lemma 2 are studied at a
critical point to find the characteristics of global minima of the residual
loss. Note that global minimum in this context refers to the parameters that
make the loss zero.

###  3.3 Critical Points of Wide PINNs

We are eventually interested in finding sufficient conditions for the
existence of a global minimum of the residual loss, i.e.,
$\phi_{r}(\mathbf{x};\mathcal{W})=0$. The following theorem takes the first
step by providing a necessary condition for globally minimizing the residual
loss. We then turn this requirement into a sufficient condition by
establishing a set of assumptions. Note that the squared error $l(r)$ is a
non-negative convex function of the residuals $r$, and $l^{\prime}(r)=0$
results in $l(r)=0$. Thus, a critical point $\overline{\mathcal{W}}$ of
$\phi_{r}(\mathbf{x};\mathcal{W})$ in the parameter space globally minimizes
the residual loss if
$l^{\prime}(\mathcal{D}[\hat{u}_{\overline{\mathcal{W}}}](x)-f(x))=0$ for
every training sample in $\mathbf{x}$.

######  Theorem 1.

For $\hat{u}$ and $\mathcal{D}[\hat{u}]$ as in Lemma 1, a critical point
$\overline{\mathcal{W}}$ of the residual loss
$\phi_{r}(\mathbf{x},\mathcal{W})$ is a global minimum if the following
conditions are satisfied:

  1. 1.

Weights $\overline{W}_{1}$ of the first layer are strictly non-zero,

  2. 2.

$F^{(k)}_{1}$ has full row rank, i.e.,
$\text{rank}(F_{1}^{(k)}(\mathbf{x}))=N$.

Theorem 1 distinguishes the global minima from other critical points of the
residual loss. However, there is no guarantee that an arbitrary neural network
can satisfy the conditions in this theorem. In other words, a critical point
that makes $F_{1}^{(k)}(\mathbf{x})$ full row rank may not exist in the
parameter space of a neural network at all. Still, this theorem does give out
a necessary condition for such a neural network. Since
$F_{1}^{(k)}(\mathbf{x})$ is an $N\times n_{1}$ matrix, the width $n_{1}$ of
the first layer must be at least $N$ for it to be full row rank. In fact,
given other assumptions, the next theorem shows that $n_{1}\geq N$ is also a
sufficient condition for the existence of a global minimum. Note that the
first condition on $W_{1}$ is satisfied with a high probability in a
continuous high-dimensional parameter space.

We first define the non-degenerate critical points used in the next theorem
and establish a set of assumptions that connect the two theorems together.

######  Definition 1 (Non-degenerate Critical Point Nguyen and Hein (2017)).

For a function $f\in C^{2}:U\subset\mathbb{R}^{n}\rightarrow\mathbb{R}$ (i.e.,
$f$ has continuous second-order derivatives), a critical point
$x=(x_{1},\dots,x_{n})\in U$of $f$ is non-degenerate if its Hessian matrix at
$x$ is non-singular. Furthermore, $x$ is non-degenerate on a subset of
variables $s\subset\\{x_{1},\dots,x_{n}\\}$ if the Hessian w.r.t. only the
variables in $s$ is non-singular at $x$.

######  Assumptions 1.

For the collocation training data $\mathbf{x}$ of $N$ points, the activation
function $\sigma$ in $\hat{u}$, and the $k$-th order differential operator
defined in Lemma 1, we assume that

  1. 1.

samples in $\mathbf{x}$ are distinct,

  2. 2.

$\sigma^{(k)}$ is a continuous and strictly monotonically increasing function,
and

  3. 3.

$\sigma^{(k)}$ is a bounded function with an infimum of zero.

######  Theorem 2.

With Assumptions 1 holding and for $\mathcal{D}\left[\hat{u}\right]$ as in
Lemma 1, if $n_{1}\geq N$, then every critical point $\overline{\mathcal{W}}$
of $\phi_{r}(\mathbf{x};\mathcal{W})$ that is non-degenerate on
$\\{W_{2},b_{2}\\}$ is a global minimum of $\phi_{r}$.

The following remark allows us to make our final conclusion from Theorem 2.

######  Remark 2.

As explained in Milnor et al. (1965) and Nguyen and Hein (2017), for a
function $f$ in $C^{2}$ that maps an open subset $U\subset\mathbb{R}^{n}$ to
$\mathbb{R}$, the degenerate critical points in $U$ are rare as the set of all
degenerate points has Lebesgue measure zero.

Theorem 2 provides sufficient conditions for a global minimum of the residual
loss with a wide network of width $N$ or higher. Based on this theorem, if a
PINN with a width of at least $N$ has a non-degenerate critical point, then it
also has a global minimum for the residual loss. Since the residual loss
$\phi_{r}(\mathbf{x};\cdot)$ is a function from $\mathbb{R}^{|\mathcal{W}|}$
to $\mathbb{R}$ and has continuous second derivatives, the degenerate critical
points are rare, and the wide PINN in Theorem 2 has a global minimum.

Note that the residual loss is a strong regularizer that results in a data-
efficient training process. Thus, PINNs are often trained with
$\mathcal{O}(1000)$ collocation points and even fewer boundary data Raissi et
al. (2017); Krishnapriyan et al. (2021). Therefore, the constraint on the
width is well within the practical settings of neural networks. Furthermore,
as we observe in the experiments in Section 5, while satisfying the constraint
on the width improves the performance, one can expect relatively good results
with smaller width as long as the other conditions in Assumptions 1 are almost
satisfied.

##  4 On the Choice of Activation Function

(a) First derivatives

(b) Second derivatives

Figure 1: Derivatives of most of the common activation functions are not
bijective. Here, only Softplus has a bijective first derivative.

(a) 3-Layer 64-Neuron

(b) 3-Layer 512-Neuron

Figure 2: Distribution of the linear outputs of the layers in Sine networks at
initialization.

The conditions outlined in Theorem 2 and Assumptions 1 collectively establish
an important set of necessities for achieving global minimization of the
residual loss. Notably, the requirement of strictly monotonically increasing
$\sigma^{(k)}$ implies that it should be a bijection, providing an important
guideline in choosing effective activation functions for PINNs. It is
noteworthy that bijective activation functions are widely prevalent in deep
learning, and extending this characteristic to their derivatives for improved
expressiveness in representing differential operators is a plausible goal.

However, the activation functions frequently used in deep learning do not
satisfy the bijection property even for the first-order derivatives. As
depicted in Figure 1, only Softplus has a bijective first-order derivative,
and as we show in Section 5, it indeed improves the performance of the first-
order Transport PINN significantly. Meanwhile, there has been an increasing
interest in the use of sinusoidal functions either as feature embeddings Wong
et al. (2022); Wang et al. (2021) or activation functions Sitzmann et al.
(2020); Belbute-Peres and Kolter (2022) for PINNs.

As shown in Figure 2, we observe that the linear outputs of the layers in a
neural network with the Sine activation function at initialization are centred
at zero with low variance when initialized with normal Xavier initialization.
Sitzmann et al. Sitzmann et al. (2020) also proposed a uniform initialization
scheme for Sine networks that produces normal linear outputs at all layers
with a desired variance. Consequently, in both cases, most of the linear
outputs of the layers lie in the $\left[-\pi/2,\pi/2\right]$ interval where
Sine is bijective. Furthermore, as we train the PINNs with the Sine activation
function, we observe that layers still exhibit the same behaviour, i.e., most
of the linear outputs of the layers are between $-\pi/2$ and $\pi/2$ after
convergence, especially as the width grows larger. Figure 3 illustrates the
output distributions for each layer of the trained Wave and Klein-Gordon PINNs
(We later define these equations in Section 5).

The observations above suggest that the sinusoidal functions can be utilized
to almost satisfy the bijective condition of the activation function.
Specifically, we use Cosine and Sine activation functions to train PINNs with
first- and second-order terms, respectively. As a result, as long as the width
is adequately large to produce low-variance pre-activations within
$\left[-\pi/2,\pi/2\right]$, the first-order terms in Cosine networks and the
second-order terms in Sine networks are determined with the bijective interval
of Sine. The same approach can be taken when solving PDEs with higher odd or
even terms.

The experiments in the next section show that sinusoidal non-linearity greatly
improves the performance of PINNs compared to the common Tanh activation, and
the gains are often greater as the width grows. We note that while the
Assumption 1.3 facilitates the proof of the Theorem 2, the crucial property is
the bijective $\sigma^{(k)}$, and we relax the assumption on the infimum of
the derivatives.

(a) Wave Equation

(b) Klein-Gordon Equation

Figure 3: Distribution of linear outputs of PINNs’ layers. Top row: 1024
neurons wide, Bottom row: 256 neurons wide

##  5 Experiments

In this section, we provide numerical results for several PDEs, revealing the
impact of the activation functions and the width. We first experiment with the
first-order Transport equation, comparing the Softplus and Cosine activation
functions with Tanh. Next, we study three second-order PDEs using Sine and
Tanh activation functions. We empirically show that Sine significantly
improves the performance of PDEs with second-order terms with a noticeable
decrease in error as the width exceeds the number of training samples.

In all of the experiments, we use a three-layer feed-forward network with a
width varying from 64 neurons up to 1024 neurons and initialized with Normal
Xavier initialization. All the models are trained with normalized inputs for
80K epochs using the Adam optimizer and an exponential learning rate decay
scheme. The only exception is the Wave equation, for which the models are
trained for 120K epochs for better convergence. We repeat each experiment
three times with a different random initialization and report the average and
the best results.

###  5.1 Transport Equation

(a) Tanh 256-Neuron

(b) Cosine 256-Neuron

(c) Softplus 256-Neuron

Figure 4: Transport Equation. Top panels: Exact solution, Middle panels:
Predicted solution, Bottom panels: Absolute Error

The transport equation is a first-order linear PDE that describes a quantity
as it moves through time and space. We experiment with the one-dimensional
equation with the following formulation:

$\frac{\partial u}{\partial t}+30\frac{\partial u}{\partial
x}=0\;\;\;\;x\in\left[0,2\pi\right],t\in\left[0,1\right]$

We also impose a periodic boundary condition $u(t,0)=u(t,2\pi)$ and a
Dirichlet initial condition $u(0,x)$ consistent with the solution in
Krishnapriyan et al. (2021). The PINN is then trained with 256 collocation
training points and 200 boundary samples. To verify our results in Section
3.3, we choose Softplus as the activation function as it is a smooth version
of ReLU with a bijective first derivative equal to the Sigmoid function. We
also use Cosine, as it results in a Sine network for the first-order terms.

Table 1 reports the mean absolute errors for the transport equation. In all
cases, Softplus and Cosine perform significantly better than Tanh, as shown in
Figure 4. Furthermore, as the width becomes equal to the number of collocation
samples (256), we observe a noticeable decrease in the absolute error. The
same improvement is also evident in the training curve of the residual loss as
shown in Figure 5, where the wide models follow a steep curve. For the PINNs
with a width of 256 or wider with both Cosine and Softplus activation
functions, the absolute errors are between $10^{-2}$ and $10^{-3}$,
outperforming the reported $1.1\times 10^{-2}$ absolute error achieved with
curriculum learning in Krishnapriyan et al. (2021).

Figure 5: Average residual loss curve for the Transport PINNs with the
Softplus activation function and trained with 256 collocation samples.

| Tanh | Softplus | Cosine
---|---|---|---
Width | Avg | Best | Avg | Best | Avg | Best
64 | 0.6314 | 0.6212 | 0.0207 | 0.0111 | 0.0095 | 0.0086
128 | 0.6118 | 0.6083 | 0.0229 | 0.0156 | 0.0042 | 0.0039
256 | 0.5673 | 0.5588 | 0.0092 | 0.0040 | 0.0029 | 0.0013
512 | 0.5168 | 0.4158 | 0.0093 | 0.0062 | 0.0019 | 0.0004
1024 | 0.3632 | 0.0011 | 0.0103 | 0.0062 | 0.0014 | 0.0011

Table 1: Average and best mean absolute error for the Transport equation over
three random initializations trained with 256 collocation points.

| Helmholtz | Klein Gordon | Wave
---|---|---|---
| Tanh | Sine | Tanh | Sine | Tanh | Sine
Width | Avg | Best | Avg | Best | Avg | Best | Avg | Best | Avg | Best | Avg | Best
64 | 4.7235 | 4.1456 | 0.0125 | 0.0087 | 0.0275 | 0.0167 | 0.0097 | 0.0059 | 0.3166 | 0.2676 | 0.1966 | 0.1272
128 | 2.8161 | 2.2007 | 0.0105 | 0.0064 | 0.0134 | 0.0096 | 0.0056 | 0.0035 | 0.2902 | 0.2351 | 0.1377 | 0.0594
256 | 1.8516 | 1.4016 | 0.0273 | 0.0212 | 0.0606 | 0.0191 | 0.0052 | 0.0031 | 0.1937 | 0.1808 | 0.3608 | 0.0676
512 | 0.7854 | 0.4909 | 0.0044 | 0.0028 | 0.0875 | 0.0289 | 0.0096 | 0.0049 | 0.1577 | 0.0725 | 0.0604 | 0.0587
1024 | 0.5946 | 0.2221 | 0.0067 | 0.0056 | 0.1939 | 0.0928 | 0.0090 | 0.0046 | 0.1147 | 0.0311 | 0.0620 | 0.0562

Table 2: Average and best mean absolute errors for second-order PDEs over
three random initializations. Underlined values show where the width is equal
to $N$.

(a) Sine 128-Neruon

(b) Tanh 128-Neuron

(c) Tanh 1024-Neuron

Figure 6: Wave Equation. Top panels: Exact solution, Middle panels: Predicted
solution, Bottom panels: Absolute Error Figure 7: Average residual loss curve
for Wave PINNs with the Tanh activation function, trained with 512 collocation
samples.

###  5.2 Wave Equation

The wave equation describes mechanical and electromagnetic waves and has the
following form in 1-D:

$\frac{\partial^{2}u}{\partial t^{2}}=c^{2}\frac{\partial^{2}u}{\partial
x^{2}}$

Here $c$ is the velocity of the wave. For $c=1$ and the solution

$u(t,x)=\sin(5\pi x)\cos(5\pi t)+2\sin(7\pi x)\cos(7\pi t),$

we train the PINN with 512 collocation points for $t,x\in[0,1]$ and impose the
initial and boundary conditions below with 256 boundary data points:

$\displaystyle u(0,x)=\sin(5\pi x)+2\sin(7\pi x)\;\;$ $\displaystyle
x\in\left[0,1\right]$ $\displaystyle\frac{\partial u}{\partial t}(0,x)=0$
$\displaystyle x\in\left[0,1\right]$ $\displaystyle u(t,1)=u(t,0)=0$
$\displaystyle t\in\left[0,1\right]$

The residual loss training curves in Figure 7 highlight the impact of the
width in training PINNs, with wider models consistently achieving smaller loss
values, and the 512- and 1024-neuron models following almost the same path.
This behaviour is similar to the curves in Figure 5 for the transport
equation, where all the models with a width of at least $N$ achieve very close
loss values. As evident in Table 2, PINNs with both Tanh and Sine activation
functions perform notably better than narrow PINNs, with a mean absolute error
of $3.11\times 10^{-2}$ and $5.62\times 10^{-2}$ for Tanh and Sine
respectively. Figure 6 illustrates the inability of the narrow Tanh network
compared to Sine and wide Tanh models in representing the solution. Also,
while narrow Sine PINNs are able to find good solutions, the training is more
unstable and the performance is worse on average compared to wider models.

###  5.3 Helmholtz Equation

We consider a 2D Helmholtz equation of the following form with
$t,x\in\left[-1,1\right]$ as in Wong et al. (2022):

$\displaystyle\frac{\partial^{2}u}{\partial
x^{2}}+\frac{\partial^{2}u}{\partial y^{2}}+u=(1-\pi^{2}-(6\pi)^{2})\sin(\pi
x)\cos(6\pi y)$

With zero boundary conditions, the solution is given by

$u(x,y)=\sin(\pi x)\sin(6\pi y).$

The PINN is then trained with 512 collocation points and 256 boundary data,
using Sine and Tanh as the activation function. As reported in Table 2, Sine
performs remarkably better than Tanh across all widths. Similar to the Wave
and Transport equations, there is also a slight decrease in the errors as the
width exceeds the number of collocation points. As in the Wave equation, PINNs
with the Tanh activation function start to perform better as the width grows
larger. However, as illustrated in Figure 8, even the 1024 neurons-wide Tanh
network is still unable to capture the solution, while the Sine network finds
an acceptable solution even with a width of 128 neurons.

(a) Sine 128-Neuron

(b) Sine 1024-Neuron

(c) Tanh 1024-Neuron

Figure 8: Helmholtz Equation. Top panels: Exact solution, Middle panels:
Predicted solution, Bottom panels: Absolute Error

###  5.4 Klein-Gordon Equation

(a) Exact Solution

(b) Tanh 128-Neuron

(c) Sine 256-Neuron

Figure 9: Absolute error of the Klein-Gordon PINNs.

We conduct the same experiments on the non-linear one-dimensional Klein-Gordon
equation of the following form:

$\displaystyle\frac{\partial^{2}u}{\partial
t^{2}}-\frac{\partial^{2}u}{\partial x^{2}}+u^{3}=f(t,x),\;\;\;\;$
$\displaystyle x\in\left[0,1\right],t\in\left[0,1\right].$ (6)

We adopt the solution provided in Wang et al. (2020a) and derive the source
term $f(t,x)$ in Eq. 6 to be consistent with the solution below:

$\displaystyle u(t,x)=x\cos(5\pi t)+(xt)^{3}$

The model is then trained with a zero initial condition and a Dirichlet
boundary condition corresponding to the solution $u$, using 256 collocation
training points and 200 boundary data.

The resulting absolute errors are reported in Table 2. Similar to other
equations, the Sine activation function performs consistently better for all
widths, achieving the best mean absolute error of $3.1\times 10^{-3}$ with the
256 neurons-wide model. Tanh also performs reasonably well with the 128
neurons-wide model, although it still performs slightly worse than the Sine
models, as shown in Figure 9.

##  6 Conclusion

The differentiation process in the residual loss of PINNs transforms the
structure of the neural networks and their outputs, rendering the existing
theory around loss functions and common supervised tasks ineffective in
analyzing PINNs. In this work, we aim to fill the gap in our understanding of
the residual loss and derive the requirements in network design that lead to
achieving global minimization of this loss function. To this end, we study the
residual loss at a critical point in the parameter space of the neural network
and look for distinct characteristics of a global minimum that sets it apart
from other critical points. We then use those characteristics to derive the
requirements in the neural network design that ensure the existence of a
global minimum. In particular, we show that under certain conditions, wide
networks globally minimize the residual loss. Additionally, we reveal that
activation functions with well-behaved high-order derivatives are crucial in
the optimal minimization of the residual loss. We then use the established
theory and empirical observations to choose activation functions and verify
their effectiveness by conducting a set of experiments. The theory developed
in this work paves the way for further development of better activation
functions and provides a guideline for designing effective PINNs.

##  7 Acknoledgments

This research is supported by Natural Sciences and Engineering Research
Council of Canada (NSERC), Discovery Grants program, and the Vector
Scholarship in Artificial Intelligence, provided through the Vector Institute.

## References

  * Allen-Zhu et al. [2019] Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song.  A convergence theory for deep learning via over-parameterization.  In International conference on machine learning, pages 242–252. PMLR, 2019.
  * Belbute-Peres and Kolter [2022] Filipe de Avila Belbute-Peres and J Zico Kolter.  Simple initialization and parametrization of sinusoidal networks via their kernel bandwidth.  arXiv preprint arXiv:2211.14503, 2022.
  * Cai et al. [2021] Shengze Cai, Zhiping Mao, Zhicheng Wang, Minglang Yin, and George Em Karniadakis.  Physics-informed neural networks (pinns) for fluid mechanics: A review.  Acta Mechanica Sinica, 37(12):1727–1738, 2021.
  * Chen et al. [2020a] Yuyao Chen, Lu Lu, George Em Karniadakis, and Luca Dal Negro.  Physics-informed neural networks for inverse problems in nano-optics and metamaterials.  Optics express, 28(8):11618–11633, 2020.
  * Chen et al. [2020b] Zixiang Chen, Yuan Cao, Quanquan Gu, and Tong Zhang.  A generalized neural tangent kernel analysis for two-layer neural networks.  Advances in Neural Information Processing Systems, 33:13363–13373, 2020.
  * Dong and Ni [2021] Suchuan Dong and Naxian Ni.  A method for representing periodic functions and enforcing exactly periodic boundary conditions with deep neural networks.  Journal of Computational Physics, 435:110242, 2021.
  * Du and Hu [2019] Simon Du and Wei Hu.  Width provably matters in optimization for deep linear neural networks.  In International Conference on Machine Learning, pages 1655–1664. PMLR, 2019.
  * Du et al. [2019] Simon Du, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai.  Gradient descent finds global minima of deep neural networks.  In International conference on machine learning, pages 1675–1685. PMLR, 2019.
  * Farhani et al. [2022] G. Farhani, Alexander Kazachek, and Boyu Wang.  Momentum diminishes the effect of spectral bias in physics-informed neural networks.  arXiv preprint arXiv:2206.14862, 2022.
  * Jacot et al. [2018] Arthur Jacot, Franck Gabriel, and Clément Hongler.  Neural tangent kernel: Convergence and generalization in neural networks.  Advances in neural information processing systems, 31, 2018.
  * Krishnapriyan et al. [2021] Aditi Krishnapriyan, Amir Gholami, Shandian Zhe, Robert Kirby, and Michael W Mahoney.  Characterizing possible failure modes in physics-informed neural networks.  Advances in Neural Information Processing Systems, 34:26548–26560, 2021.
  * Lee et al. [2019] Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl-Dickstein, and Jeffrey Pennington.  Wide neural networks of any depth evolve as linear models under gradient descent.  Advances in neural information processing systems, 32, 2019.
  * Li et al. [2020] Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar.  Fourier neural operator for parametric partial differential equations.  arXiv preprint arXiv:2010.08895, 2020.
  * Liu et al. [2020] Chaoyue Liu, Libin Zhu, and Misha Belkin.  On the linearity of large non-linear models: when and why the tangent kernel is constant.  Advances in Neural Information Processing Systems, 33:15954–15964, 2020.
  * McClenny and Braga-Neto [2020] Levi McClenny and Ulisses Braga-Neto.  Self-adaptive physics-informed neural networks using a soft attention mechanism.  arXiv preprint arXiv:2009.04544, 2020.
  * Meronen et al. [2021] Lassi Meronen, Martin Trapp, and Arno Solin.  Periodic activation functions induce stationarity.  Advances in Neural Information Processing Systems, 34:1673–1685, 2021.
  * Milnor et al. [1965] John Milnor, L. Siebenmann, and J. Sondow.  Lectures on the H-Cobordism Theorem.  Princeton University Press, 1965.
  * Nguyen and Hein [2017] Quynh Nguyen and Matthias Hein.  The loss surface of deep and wide neural networks.  In International conference on machine learning, pages 2603–2612. PMLR, 2017.
  * Nguyen and Mondelli [2020] Quynh N Nguyen and Marco Mondelli.  Global convergence of deep networks with one wide layer followed by pyramidal topology.  Advances in Neural Information Processing Systems, 33:11961–11972, 2020.
  * Oymak and Soltanolkotabi [2020] Samet Oymak and Mahdi Soltanolkotabi.  Toward moderate overparameterization: Global convergence guarantees for training shallow neural networks.  IEEE Journal on Selected Areas in Information Theory, 1(1):84–105, 2020.
  * Rahaman et al. [2018] Nasim Rahaman, A. Baratin, Devansh Arpit, Felix Dräxler, Min Lin, F. Hamprecht, Yoshua Bengio, and Aaron C. Courville.  On the spectral bias of neural networks.  International Conference On Machine Learning, 2018.
  * Raissi et al. [2017] Maziar Raissi, Paris Perdikaris, and George Em Karniadakis.  Physics informed deep learning (part i): Data-driven solutions of nonlinear partial differential equations.  arXiv preprint arXiv:1711.10561, 2017.
  * Reiser et al. [2022] Patrick Reiser, Marlen Neubert, André Eberhard, Luca Torresi, Chen Zhou, Chen Shao, Houssam Metni, Clint van Hoesel, Henrik Schopmans, Timo Sommer, et al.  Graph neural networks for materials science and chemistry.  Communications Materials, 3(1):93, 2022.
  * Safran and Shamir [2016] Itay Safran and Ohad Shamir.  On the quality of the initial basin in overspecified neural networks.  In International Conference on Machine Learning, pages 774–782. PMLR, 2016.
  * Sirignano and Spiliopoulos [2018] Justin Sirignano and Konstantinos Spiliopoulos.  Dgm: A deep learning algorithm for solving partial differential equations.  Journal of computational physics, 375:1339–1364, 2018.
  * Sitzmann et al. [2020] Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein.  Implicit neural representations with periodic activation functions.  Advances in neural information processing systems, 33:7462–7473, 2020.
  * Wang et al. [2020a] Sifan Wang, Yujun Teng, and Paris Perdikaris.  Understanding and mitigating gradient pathologies in physics-informed neural networks.  arXiv preprint arXiv:2001.04536, 2020.
  * Wang et al. [2020b] Sifan Wang, Xinling Yu, and P. Perdikaris.  When and why pinns fail to train: A neural tangent kernel perspective.  Journal Of Computational Physics, 2020.
  * Wang et al. [2021] Sifan Wang, Hanwen Wang, and Paris Perdikaris.  On the eigenvector bias of fourier feature networks: From regression to solving multi-scale pdes with physics-informed neural networks.  Computer Methods in Applied Mechanics and Engineering, 384:113938, 2021.
  * Wang et al. [2022] Sifan Wang, Shyam Sankaran, and Paris Perdikaris.  Respecting causality is all you need for training physics-informed neural networks.  arXiv preprint arXiv:2203.07404, 2022.
  * Wight and Zhao [2020] Colby L Wight and Jia Zhao.  Solving allen-cahn and cahn-hilliard equations using the adaptive physics informed neural networks.  arXiv preprint arXiv:2007.04542, 2020.
  * Wong et al. [2022] Jian Cheng Wong, Chinchun Ooi, Abhishek Gupta, and Yew-Soon Ong.  Learning in sinusoidal spaces with physics-informed neural networks.  IEEE Transactions on Artificial Intelligence, 2022.
  * Zhang et al. [2022] Jie Zhang, Yihui Zhao, Fergus Shone, Zhenhong Li, Alejandro F Frangi, Sheng Quan Xie, and Zhi-Qiang Zhang.  Physics-informed deep learning for musculoskeletal modeling: Predicting muscle forces and joint kinematics from surface emg.  IEEE Transactions on Neural Systems and Rehabilitation Engineering, 31:484–493, 2022.

####

##  Appendix A Generalizing to Multiple Independent Variables

We first generalize Lemmas 1 and 2 and Theorem 1 in Section 3 to multiple
($d$) independent variables and present the proofs for them, and then, prove
Theorem 2 in the next section. We consider a function of $d$ independent
variables and a differential operator $\mathcal{D}$ with $k$-th order terms
and coefficients $c_{1},\dots,c_{d}$ of the following form:

$\mathcal{D}\left[u\right]=c_{1}\frac{\partial^{k}u}{\partial
x_{1}^{k}}+\dots+c_{d}\frac{\partial^{k}u}{\partial x_{d}^{k}}$ (7)

We also define $(W_{1})_{i}\in\mathbb{R}^{1\times n_{1}}$ for
$i\in\\{1,\dots,d\\}$ to represent the $i$-th row of the weight matrix
$W_{1}$, i.e., $(W_{1})_{i}$ are the weights corresponding to the $i$-th input
variable.

######  Lemma 3 (Generalization of Lemma 1).

For a two-layer neural network $\hat{u}$ of $d$ input variables and the $k$-th
order differential operator in Eq. 7, $\mathcal{D}[\hat{u}]$ is given by

$\displaystyle\mathcal{D}[\hat{u}](x)=W_{2}^{\top}\times(F_{1}^{(k)}(x)\circ($
$\displaystyle c_{1}(W_{1})_{1}^{k}+\dots$
$\displaystyle+c_{d}(W_{1})_{d}^{k}))^{\top}.$

###### Proof.

Note that the neural network is defined as

$\displaystyle\hat{u}(x)$ $\displaystyle=F_{1}(x)\times W_{2}+b_{2}$
$\displaystyle=\sigma(G_{1}(x))\times W_{2}+b_{2},\;G_{1}(x)=x\times
W_{1}+b_{1}$

The first derivative $\frac{\partial\hat{u}}{\partial x_{1}}$ w.r.t. the first
input variable is then derived as follows:

$\displaystyle\frac{\partial\hat{u}}{\partial x_{1}}$
$\displaystyle=\frac{\partial\hat{u}}{\partial
F_{1}}\times\left(\frac{\partial F_{1}}{\partial G_{1}}\frac{\partial
G_{1}}{\partial x_{1}}\right)$
$\displaystyle=W_{2}^{\top}\times\left(\frac{\partial F_{1}}{\partial
G_{1}}\frac{\partial G_{1}}{\partial x_{1}}\right)$ (8)

Derivative of $F_{1}$ w.r.t. $G_{1}$ is just $\sigma^{\prime}(G_{1}(x))$. We
also have

$\displaystyle G_{1}(x)$ $\displaystyle=x\times W_{1}+b_{1}$
$\displaystyle=\sum_{i=1}^{d}x_{i}(W_{1})_{i}+b_{1}.$

Thus, Eq. 8 yields:

$\displaystyle\frac{\partial\hat{u}}{\partial
x_{1}}=W_{2}^{\top}\times\left(\sigma^{\prime}(G_{1}(x))\circ(W_{1})_{1}\right)^{\top}$

For higher-order derivatives, only $G_{1}(x)$ depends on $x_{1}$, and other
terms are constants. Hence, assuming that the $(k-1)$-th derivative is given
by
$W_{2}^{\top}\times\left(\sigma^{(k-1)}(G_{1}(x))\circ(W_{1})_{1}^{k-1}\right)^{\top}$,
the $k$-th derivative is

$\displaystyle\frac{\partial^{k}\hat{u}}{\partial x_{1}^{k}}$
$\displaystyle=W_{2}^{\top}\times\left(\frac{\partial F_{1}^{(k-1)}}{\partial
x_{1}}\circ(W_{1})^{k-1}_{1}\right)^{\top}$
$\displaystyle=W_{2}^{\top}\times\left(\sigma^{(k)}(G_{1}(x))\circ(W_{1})_{1}\circ(W_{1})^{k-1}_{1}\right)^{\top}$
$\displaystyle=W_{2}^{\top}\times\left(F_{1}^{(k)}(x)\circ(W_{1})^{k}_{1}\right)^{\top}.$

Derivatives w.r.t. other input variables are taken similarly. Thus, the
differential operator $\mathcal{D}[\hat{u}]$ is

$\displaystyle\mathcal{D}[\hat{u}](x)$
$\displaystyle=\sum_{i=1}^{d}c_{i}W_{2}^{\top}\times\left(F_{1}^{(k)}(x)\circ(W_{1})^{k}_{i}\right)^{\top}$
$\displaystyle=W_{2}^{\top}\times\left(\sum_{i=1}^{d}c_{i}F_{1}^{(k)}(x)\circ(W_{1})_{i}^{k}\right)$
$\displaystyle=W_{2}^{\top}\times\left(F_{1}^{(k)}(x)\circ\sum_{i=1}^{d}c_{i}(W_{1})_{i}^{k}\right).$

∎

Note that Lemma 3 (and other results in this section) can be further
generalized to include mixed derivatives such as

$\frac{\partial^{2}\hat{u}}{\partial
x_{1}x_{2}}=W_{2}^{\top}\times(F_{1}^{(k)}(x)\circ(W_{1})_{1}\circ(W_{1})_{2})^{\top},$

which we don’t consider for simplicity.

######  Lemma 4 (Generalization of Lemma 2).

For $\hat{u}$ and $\mathcal{D}[\hat{u}]$ as in Lemma 3, gradients of the
residual loss w.r.t. the weights of the second layer over the training
collocation data $\mathbf{x}$ of $N$ samples are given by

$\displaystyle\nabla_{W_{2}}\phi_{r}(\mathbf{x};\mathcal{W})=$
$\displaystyle\;\;\left(\sum_{i=1}^{d}c_{i}(W_{1})_{i}^{k}\right)\circ\left(l^{\prime}(\mathcal{D}[\hat{u}](\mathbf{x})-f(\mathbf{x}))^{\top}\times
F^{(k)}_{1}(\mathbf{x})\right).$

###### Proof.

Let $A\in\mathbb{R}^{1\times n_{1}}$ be $\sum_{i=1}^{d}c_{i}(W_{1})_{i}^{k}$.
From Lemma 3, the residual loss over collocation samples
$\mathbf{x}=[x^{1},\dots,x^{N}]^{\top}$ is given by

$\displaystyle\phi_{r}(\mathbf{x};\mathcal{W})$
$\displaystyle=\sum_{i=1}^{N}l\biggl{(}W_{2}^{\top}\times\bigl{(}F_{1}^{(k)}(x^{i})\circ
A\bigr{)}-f(x^{i})\biggr{)}$
$\displaystyle=\sum_{i=1}^{N}l\biggl{(}\sum_{j=1}^{n_{1}}(W_{2})_{j}\bigl{(}F_{1j}^{(k)}(x^{i})A_{j}\bigr{)}-f(x^{i})\biggr{)}\,,$

where $F_{1j}^{(k)}$ is the scalar output of neuron $j$ in the first layer fed
to $\sigma^{(k)}$, and $A_{j}$ and $(W_{2})_{j}$ are the $j$-th elements of
the row and column vectors $A$ and $W_{2}$ (note that
$W_{2}\in\mathbb{R}^{n_{1}\times 1}$). Let $r_{i}\in\mathbb{R}$ denote the
residual $\sum_{j=1}^{n_{1}}(W_{2})_{j}(F_{1j}^{(k)}(x^{i})A_{j})-f(x^{i})$
for a sample $x^{i}\in\mathbf{x}$. The partial derivative
$\frac{\partial\phi_{r}}{\partial(W_{2})_{\alpha}}$ for
$\alpha\in\\{1,\dots,n_{1}\\}$ is then derived as

$\displaystyle\frac{\partial\phi_{r}}{\partial(W_{2})_{\alpha}}(\mathbf{x};\mathcal{W})$
$\displaystyle=\sum_{i=1}^{N}\frac{\partial l(r_{i})}{\partial
r_{i}}\frac{\partial r_{i}}{\partial(W_{2})_{\alpha}}$
$\displaystyle=\sum_{i=1}^{N}l^{\prime}(r_{i})F_{1\alpha}^{(k)}(x^{i})A_{\alpha}$
$\displaystyle=A_{\alpha}\sum_{i=1}^{N}l^{\prime}(r_{i})F_{1\alpha}^{(k)}(x^{i}),$

and the gradients w.r.t. $W_{2}$ are

$\displaystyle\nabla_{W_{2}}\phi_{r}(\mathbf{x};\mathcal{W})$
$\displaystyle=\begin{bmatrix}A_{1}\sum_{i=1}^{N}l^{\prime}(r_{i})F_{11}^{(k)}(x^{i})\\\
\vdots\\\
A_{n_{1}}\sum_{i=1}^{N}l^{\prime}(r_{i})F_{1n_{1}}^{(k)}(x^{i})\end{bmatrix}^{\top}$
$\displaystyle=A\circ\begin{bmatrix}\sum_{i=1}^{N}l^{\prime}(r_{i})F_{11}^{(k)}(x^{i})\\\
\vdots\\\
\sum_{i=1}^{N}l^{\prime}(r_{i})F_{1n_{1}}^{(k)}(x^{i})\end{bmatrix}^{\top}$
$\displaystyle=A\circ\biggl{(}\sum_{i=1}^{N}l^{\prime}(r_{i})F_{1}^{(k)}(x^{i})\biggr{)}$
$\displaystyle=A\circ\bigl{(}L^{\prime\top}\times
F_{1}^{(k)}(\mathbf{x})\bigr{)},$

where $L^{\prime}\in\mathbb{R}^{N}$ is
$[l^{\prime}(r_{1}),\dots,l^{\prime}(r_{N})]^{\top}$.

∎

######  Theorem 3 (Generalization of Theorem 1).

For $\hat{u}$ and $\mathcal{D}[\hat{u}]$ as in Lemma 3, a critical point
$\overline{\mathcal{W}}$ of the residual loss
$\phi_{r}(\mathbf{x},\mathcal{W})$ is a global minimum if the following
conditions are satisfied:

  1. 1.

The summation $\sum_{i=1}^{d}c_{i}(\overline{W}_{1})_{i}^{k}$ of the weights
of each input variable in the first layer is strictly non-zero,

  2. 2.

$F^{(k)}_{1}$ has full row rank, i.e.,
$\text{rank}(F_{1}^{(k)}(\mathbf{x}))=N$.

###### Proof.

At a critical point $\overline{\mathcal{W}}$ of $\phi_{r}$, the gradients of
$\phi_{r}$ w.r.t. the weights of the second layer are zero. Let
$L^{\prime}\in\mathbb{R}^{N}$ be
$l^{\prime}(\mathcal{D}[\hat{u}_{\overline{\mathcal{W}}}](\mathbf{x})-f(\mathbf{x}))$
and $A\in\mathbb{R}^{1\times n_{1}}$ be
$\sum_{i=1}^{d}c_{i}(\overline{W}_{1})_{i}^{k}$. Lemma 4 yields

$\nabla_{W_{2}}\phi_{r}(\mathbf{x};\overline{\mathcal{W}})=A\circ({L^{\prime}}^{\top}\times
F^{(k)}_{1}(\mathbf{x}))=0.$

Since $A$ is strictly non-zero, ${L^{\prime}}^{\top}\times
F^{(k)}_{1}(\mathbf{x})$ must be zero. Furthermore, if
$F^{(k)}_{1}(\mathbf{x})$ has full row rank, the null space of its transpose
is zero, and thus, $L^{\prime}$ must be zero for the above product to hold.
Since $l$ is convex,
$l(\mathcal{D}[\hat{u}_{\overline{\mathcal{W}}}](\mathbf{x})-f(\mathbf{x}))$
and $\phi_{r}(\mathbf{x};\overline{\mathcal{W}})$ are also zero, and
$\overline{\mathcal{W}}$ globally minimizes the residual loss. ∎

##  Appendix B Proof of Theorem 2

In this section, we present a brief explanation of the proof for Theorem 2. We
refer the readers to [Nguyen and Hein, 2017, Section 4] for complete and more
detailed proof, while we provide the necessary modifications for PINNs here.
Note that in Nguyen and Hein [2017] it is shown that a global minimum for a
normal neural network is achieved when $[F_{1}(\mathbf{x}),\mathbf{1}_{N}]$ is
full row rank, and then, a width of $n_{1}\geq N-1$ is shown to be sufficient.
For PINNs, we showed that $F_{1}^{(k)}(\mathbf{x})$ should be full row rank
for globally minimizing the residual loss and a width of $n_{1}\geq N$ is
sufficient.

In summary, the proof works by first showing that a parameterization of the
neural network that results in full row-rank $F_{1}^{(k)}(\mathbf{x})$ does
exist. Based on the fact that $F_{1}^{(k)}(\mathbf{x})$ and determinants of
its sub-matrices are real-analytic functions of the model parameters, it is
then shown that for any set of parameters $\\{W_{1},b_{1}\\}$ that
$F_{1}^{(k)}(\mathbf{x})$ is not full row-rank, there exists another set of
parameters in a ball of any positive radius around $\\{W_{1},b_{1}\\}$ that
results in $\textit{rank}(F_{1}^{(k)}(\mathbf{x}))=N$. Finally, using the
implicit function theorem and the non-degeneracy of the critical point on
$\\{W_{2},b_{2}\\}$, it is shown that there exists another critical point in a
ball of an arbitrarily small radius around the original critical point that
results in full row rank $F_{1}^{(k)}(\mathbf{x})$.

We start by modifying Lemma 4.3 in Nguyen and Hein [2017], showing that
$F_{1}^{(k)}(\mathbf{x})$ can be full row-rank for some parameters in the
parameter space when $n_{1}\geq N$.

######  Lemma 5 (Existence of full row-rank $F_{1}^{(k)}(\mathbf{x})$).

With Assumptions 1 holding and for a two-layer neural network with a width
$n_{1}\geq N$, there exists a parameterization that results in
$\textit{rank}(F_{1}^{(k)}(\mathbf{x}))=N$

###### Proof.

Following the proof in Nguyen and Hein [2017], we construct the parameters
$\\{W_{1},b_{1}\\}$ such that $F_{1}^{(k)}$ is full row-rank. Since $n_{1}\geq
N$ and $F_{1}^{(k)}(\mathbf{x})\in\mathbb{R}^{N\times n_{1}}$, $N$ columns
from $F_{1}^{(k)}$ must be linearly independent for it to have rank $N$. We
denote the sub-matrix containing the first $N$ columns with $A$ and the other
$n_{1}-N$ columns with $B$. Showing that the matrix $A$ is full rank for a set
of parameters proves the argument as the original matrix
$F_{1}^{(k)}(\mathbf{x})=\left[A\;B\right]$ contains the same $N$ linearly
independent columns in $A$.

Pick a vector $a\in\mathbb{R}^{d}$ such that

$a\cdot x^{1}<a\cdot x^{2}<\dots<a\cdot x^{N}\;\;(a\cdot x^{i}\text{denotes
dot product})$ (9)

Note that by Assumption 1.1, the rows of the inputs
$\mathbf{x}=[x^{1},\dots,x^{N}]^{\top}$ are distinct. Thus, for some
$i,j\in\\{1,\dots,N\\}$ that $i\neq j$, the set $\\{a\in R^{d}\mid
a\cdot(x^{i}-x^{j})=0\\}$ is a hyperplane with Lebesgue measure zero. For all
pairs of distinct samples in $\mathbf{x}$, there are $N(N-1)/2$ such
hyperplanes, and their union still has Lebesgue measure zero. As a result, a
vector $a$ that results in the inequality above does exist.

By Assumption 1.3, the infimum of $\sigma^{(k)}$ is zero. Also, let $\gamma$
denote the supremum of $\sigma^{(k)}$. Thus,
$\lim_{x\to-\infty}\sigma^{(k)}(x)=0$ and
$\lim_{x\to\infty}\sigma^{(k)}(x)=\gamma$ as $\sigma^{(k)}$ is strictly
monotonically increasing. Pick any $\beta\in\mathbb{R}$. Since $\sigma^{(k)}$
is strictly monotonically increasing, $\sigma^{(k)}(\beta)$ is greater than
the infimum zero. For $\alpha\in\mathbb{R}$, the weights and biases of the
first $N$ neurons in the first layer are defined as $w_{j}=-\alpha a$ and
$b_{j}=\alpha x^{j}\cdot a+\beta$ for $j\in\\{1,\dots,N\\}$, as in Nguyen and
Hein [2017]. With this choice of weights, the outputs $A$ of the first $N$
neurons become a function of $\alpha$ with
$A(\alpha)_{ij}=\sigma^{(k)}(x^{i}\cdot
w_{j}+b_{j})=\sigma^{(k)}(\alpha(x^{j}-x^{i})\cdot a+\beta)$ being the output
of the $j$-th neuron for the $i$-th sample for $i,j\in\\{1,\dots,N\\}$. Based
on the ordering in Eq. 9, $A(\alpha)$ at the limit becomes:

$\lim_{\alpha\to\infty}A(\alpha)=\begin{bmatrix}\sigma^{(k)}(\beta)&\gamma&\gamma&\dots&\gamma\\\
0&\sigma^{(k)}(\beta)&\gamma&\dots&\gamma\\\
0&0&\sigma^{(k)}(\beta)&\dots&\gamma\\\ \vdots&\vdots&\vdots&\ddots&\vdots\\\
0&0&0&\dots&\sigma^{(k)}(\beta)\end{bmatrix}$

The matrix above is upper triangular with a positive diagonal, and thus, it
has a positive determinant and is full rank.

The rest of the proof is identical to Nguyen and Hein [2017]. As the
determinant of $A(\alpha)$ is a polynomial function of its entries and is
continuous in $\alpha$, for some $\alpha_{0}\in\mathbb{R}$, $A(\alpha)$ has a
positive determinant and is full rank for all $\alpha>\alpha_{0}$. Thus, the
weights of the first $N$ neurons in the first layer can be chosen in a way
that results in a full row rank $F_{1}^{(k)}(\mathbf{x})$, regardless of the
weights of other neurons. ∎

The rest of the proof for Theorem 2 can be adapted from [Nguyen and Hein,
2017, Section 4] by replacing $[F_{1}(\mathbf{x}),\mathbf{1}_{N}]$ with
$F_{1}^{(k)}(\mathbf{x})$ and subtle modifications. Thus, we briefly go
through the proof without formally presenting the details. The proof continues
by observing that $F_{1}^{(k)}(\mathbf{x})$ is a real analytic function of
parameters $\\{W_{1},b_{1}\\}$ (as long as $\sigma^{(k)}$ is real analytic),
as it is a composition of linear transformations, multiplications, and
addition, which are all real analytic, with the real analytic $\sigma^{(k)}$.
Similarly, the determinants of the $N\times N$ sub-matrices of
$F_{1}^{(k)}(\mathbf{x})$ are also polynomial functions of the parameters and
are real analytic. As proven in [Nguyen and Hein, 2017, Lemma 4.4], the last
argument on determinants and Lemma 5 conclude that the set
$\\{\\{W_{1},b_{1}\\}\mid\textit{rank}(F_{1}^{(k)}(\mathbf{x}))<N\\}$ has
Lebesgue measure zero. This is because, for a real analytic function that is
not constantly zero, the set of parameters that make it zero has Lebesgue
measure zero. Since the determinant function is real analytic and is not
constant zero, as shown in Lemma 5, the set of parameters that make it zero
has Lebesgue measure zero. Consequently, for any set of parameters
$\\{W^{\prime}_{1},b^{\prime}_{1}\\}$ that
$\textit{rank}(F_{1}^{(k)}(\mathbf{x}))<N$, the ball centred at
$\\{W^{\prime}_{1},b^{\prime}_{1}\\}$ with any radius $\epsilon>0$ contains at
least one $\\{W_{1},b_{1}\\}$ such that
$\textit{rank}(F_{1}^{(k)}(\mathbf{x}))=N$ [Nguyen and Hein, 2017, Corollary
4.5].

Using the implicit function theorem, it is then shown that for a critical
point
$\overline{\mathcal{W}}=\\{\overline{W}_{1},\overline{b}_{1},\overline{W}_{2},\overline{b}_{2}\\}$
that is non-degenerate on $\\{W_{2},b_{2}\\}$, there exists an open ball
around $\\{\overline{W}_{1},\overline{b}_{1}\\}$ for some radius
$\delta_{1}>0$ that for every $\\{W^{\prime}_{1},b^{\prime}_{1}\\}$ in the
ball, $\\{W^{\prime}_{1},b^{\prime}_{1},\overline{W}_{2},\overline{b}_{2}\\}$
is also a critical point. Moreover, as we discussed before, for every
$0<\delta_{2}\leq\delta_{1}$, there is a ball of radius $\delta_{2}$ around
$\\{W^{\prime}_{1},b^{\prime}_{1}\\}$ that contains at least one set of
parameters $\\{\tilde{W}_{1},\tilde{b}_{1}\\}$ that satisfies
$\textit{rank}(F_{1}^{(k)}(\mathbf{x}))=N$. Note that
$\\{\tilde{W}_{1},\tilde{b}_{1},\overline{W}_{2},\overline{b}_{2}\\}$ is still
a critical point as well. Nguyen et al. Nguyen and Hein [2017] further show
how $\\{\tilde{W}_{1},\tilde{b}_{1}\\}$ are equal to the original
$\\{\overline{W}_{1},\overline{b}_{1}\\}$ as the radius of the ball approaches
zero.

With the results above, we get a critical point that makes $F_{1}^{(k)}$ full-
rank. Based on Theorems 1 and 3, the resulting critical point is also a global
minimum as long as the condition on the sum of weights is also satisfied. Note
that the rarity of degenerate critical points (as explained in Section 3.3)
means that non-degenerate ones are the common type of critical points when
$n_{1}\geq N$. Thus, the conditions on the sum of the weights of the first
layer are also satisfied for most of the non-degenerate critical points with a
high probability.

##  Appendix C Comparison with Sinusoidal Features

| TT | TS | W | K | H
---|---|---|---|---|---
128 | 0.613 | 0.011 | 0.406 | 0.024 | 0.590
256 | 0.625 | 0.050 | 0.135 | 0.071 | 0.762
512 | 0.627 | 0.002 | 0.046 | 0.089 | 0.502
1024 | 0.654 | 0.668 | 0.024 | 0.118 | 0.443
Table 3: MAE from sf-PINN. TT: Transport with Tanh, TS: Transport with
Softplus, W: Wave, K: Klein-Gordon, H: Helmholtz

Applying random or trainable sinusoidal embeddings on PINN inputs has been
explored before Wong et al. [2022]; Wang et al. [2020b]. In particular, the
first layer in sf-PINN uses $\sin(2\pi x)$ as the activation function,
followed by other layers with any activation function of choice Wong et al.
[2022]. To confirm the effectiveness of the Sine activation function compared
to Sine features in sf-PINN, we trained sf-PINNs for the equations in Section
5.

Table 3 reports the MAEs from sf-PINN for each equation. Compared to Tables 1
and 2, we observe that in most cases, our results outperform sf-PINN. Note
that, consistent with Section 3, Softplus activation significantly improves
the performance of sf-PINNs for the Transport equation, underpinning our
findings about the activation functions. In the case of the Wave equation, sf-
PINN with Tanh performs better than plain Sine MLP, though the condition on
width still improves the performance as the number of neurons grows beyond the
number of collocation points ($N=512$).

##  Appendix D Non-linear PDEs

| Navier-Stokes | Burger’s
---|---|---
W | T | SF-T | S | T | SF-T | S
64 | 0.047 | 0.009 | 0.010 | 0.010 | 0.099 | 0.014
128 | 0.010 | 0.010 | 0.011 | 0.013 | 0.046 | 0.034
256 | 0.007 | 0.013 | 0.010 | 0.053 | 0.060 | 0.027
Table 4: MAEs for Lid-driven Cavity ($Re=400$) and Burger’s equation with
vanilla Tanh MLP (T), sf-PINN with Tanh (SF-T), and Sine MLP (S).

Our theoretical analysis primarily focused on linear PDEs, and we empirically
showed the effectiveness of our findings for solving linear PDEs. We also
tried the non-linear Klein-Gordon equation in Section 5.4 and demonstrated the
effectiveness of our method for solving this equation. However, as apparent in
Table 2, the gains for the Klein-Gordon equation are not as significant and
consistent as other linear equations, especially regarding the width.

To better understand to what extent our findings generalize to non-linear
equations, we perform similar experiments with the non-linear Burger’s
equation and steady-state Navier-Stokes (lid-driven cavity flow with
$Re=400$). Table 4 reports the MAEs for each equation. Similar to our
observations from the Klein-Gordon equation, the gains from the width and Sine
activation are not as consistent as linear equations. For the Navier-Stokes
equation, while the Sine MLP performs consistently well across all various
widths, the Tanh MLP eventually outperforms both sf-PINN and Sine MLP as the
width grows. With Burger’s equation, the Tanh MLP performs best with a narrow
network and degrades as the width grows, while the Sine MLP fluctuates
closely.

Altogether, our experiments suggest that while the conditions discussed for
linear equations can partly improve the performance of non-linear PINNs, non-
linear equations come with intricacies regarding the representation power and
optimization of PINNs that require further analysis.


[affiliation=1]SnehaRay Barman [affiliation=1,2]ShakuntalaMahanta
[affiliation=1,3]Neeraj KumarSharma

# Deciphering Assamese Vowel Harmony with Featural InfoWaveGAN

###### Abstract

Traditional approaches for understanding phonological learning have
predominantly relied on curated text data. Although insightful, such
approaches limit the knowledge captured in textual representations of the
spoken language. To overcome this limitation, we investigate the potential of
the Featural InfoWaveGAN model to learn iterative long-distance vowel harmony
using raw speech data. We focus on Assamese, a language known for its
phonologically regressive and word-bound vowel harmony. We demonstrate that
the model is adept at grasping the intricacies of Assamese phonotactics,
particularly iterative long-distance harmony with regressive directionality.
It also produced non-iterative illicit forms resembling speech errors during
human language acquisition. Our statistical analysis reveals a preference for
a specific [+high,+ATR] vowel as a trigger across novel items, indicative of
feature learning. More data and control could improve model proficiency,
contrasting the universality of learning.

###### keywords:

Computational Linguistics, Generative Adversarial Networks, Phonological
Learning, Vowel Harmony, Assamese Language

##  1 Introduction

Understanding human language acquisition involves examining various aspects of
grammar, with phonology playing a crucial role. Approaches such as Optimality
Theory [1] models phonological grammar as input-output pairs, selecting the
most optimal output based on input. In contrast, Maximum Entropy Grammar [2]
proposes that grammar constraints are language-specific and learned from data,
employing machine learning techniques to determine these constraints.

Integrating neural network methodologies, multiple studies have recently
proposed Generative Adversarial Network (GAN) and WaveGAN models for phonetic
and phonological learning, with advancements like Featural InfoWaveGAN
(fiwGAN) enhancing lexical learning and demonstrating proficiency in
phonological representation learning across languages. A comprehensive review
of prior generative linguistics studies is provided by Pater [3]. Beguš [4]
proposed Generative Adversarial Network (GAN [5]) as a method to model
learnability more organically, wherein phonetic and phonological processes are
represented as mappings from randomly distributed latent space to generative
data. Donahue et al. [6] introduced the WaveGAN model for audio data,
employing Deep Convolutional GAN (DCGAN [7]) architecture to learn language
features from continuous speech signals. This model processes 1-second audio
files sampled at $16$ kHz, converting them into vectors for input to the
discriminator network. WaveGAN, adapted for phonetic and phonological learning
[4], effectively captures linguistic representations through a one-to-one
mapping from latent space to generated data, mimicking how humans construct
underlying phonological representations from speech streams. However, human
acquisition also involves lexical learning, the ability to store unique
information associated with meaning-bearing units or phonemes while adhering
to language-specific phonotactics. Beguš [8] enhanced the WaveGAN architecture
by integrating a lexical learning component, the Q-network [9], resulting in
Featural InfoWaveGAN (fiwGAN). This advancement introduces a new latent space
structure to comprehend featural representations of phonetic and phonological
learning. The fiwGAN generator network learns to produce acoustic data by
encoding unique lexical information and generating innovative acoustic data
associated with distinct binary codes for each lexical item. Examining the
fiwGAN network's performance on English allophonic distribution, Beguš & Zhou
[10] demonstrated its generative and learning capabilities by producing data
comparable to human speech acquisition outputs. Furthermore, Chen & Elsner
[11] evaluated FiwGAN's robustness by assessing English and French nasality,
confirming its ability to learn phonological representations where latent
variables correspond to identifiable phonological features. This highlights
FiwGAN's proficiency as a phonological learner, which is particularly evident
in controlled settings when comparing the same feature across languages.

Building on these studies, we investigate the potential of the Featural
InfoWaveGAN model to learn iterative long-distance vowel harmony using raw
speech data. Vowel harmony is a phonological phenomenon observed in many
languages where vowels within a word tend to be influenced by one another in
terms of their phonetic properties, such as tongue height, tongue position, or
roundedness [12, 13]. Owing to this phenomenon, certain vowels in a word may
undergo changes to better match the phonetic characteristics of neighboring
vowels. The result is an enhancement of the overall harmony and fluidity of
pronunciation within the word [14]. Vowel harmony can be categorized as
regressive, where vowels are influenced by following vowels or progressive,
where vowels are influenced by preceding vowels. We focus on Assamese, a
language known for its phonologically regressive and word-bound vowel harmony
[15, 16]. We demonstrate that the model is adept at grasping the intricacies
of Assamese phonotactics, particularly iterative long-distance harmony with
regressive directionality.

##  2 Vowel Harmony in Assamese

Assamese is an Indo-Aryan language spoken in the northeastern state of Assam
(in India)by $15.3$ million individuals [17]. The variety analyzed in this
paper is the colloquial Assamese spoken in the eastern region of Assam, also
known as Upper Assam. This features eight surface vowels, namely, [i, e, , , ,
o, , u], and twenty consonants, namely [p, ph, b, bh, t, th, d, dh, k, kh, g,
gh, m, n, ŋ s, z, x, h, , j, w, l] [15].

Table 1: Vowel inventory of Assamese

Vowels Front Back ATR High i u +ATR -ATR Mid e o +ATR -ATR Low -ATR

In Assamese, the surface vowel occurrences are restricted in the following
ways: (a) The two high vowels /i/ and /u/ are pronounced with an advanced
tongue root [+ATR], as are the mid vowels /e/ and /o/; (b) The mid vowels / /
and / / are slightly lower than /e/ and /o/ and are not realized with an
advanced tongue root [-ATR], and (c) The vowels /e/ and /o/ do not occur word-
finally or initially, they occur only under circumstances of vowel harmony.

Table 2: Examples of vowel harmony in Assamese [15]

Stem | Gloss | -suffix | Harmonised | Gloss
---|---|---|---|---
/pt/ | ‘belly’ | -u | [petu] | ‘pot-bellied’
/pgl/ | `mad-M' | -i | [pgoli] | `mad-F'
/gl/ | `to mix' | -i | [guli] | `I mix'
/bsr/ | `year' | -i | [bosori] | `yearly'

Assamese exhibits right-to-left ATR harmony where the high vowels /i/ and /u/
trigger [+ATR] harmony of [-ATR] high // and mid vowels [//, //] in the
language. From the examples of Table 2, we can argue that the stem vowels
alternate under the influence of suffixes in Assamese. [+ATR] acts as a
dominant feature that spreads from the suffix vowels in the leftward direction
and affects vowels on the left side of the triggering vowel. As seen in the
examples, harmony applies to all the [-ATR] vowels preceding a [+ATR] vowel
within the word domain showcasing long-distance iterative harmony (see example
4 from Table 2).[15] proposed that the harmony sequence in Assamese may be
subject to the tendency of avoiding featural cooccurrence of [-ATR, +ATR]
vowels. Therefore, [-ATR] vowels do not trigger harmony playing a recessive
role. // appears to be an opaque vowel that blocks harmony from its targeting
vowels to its left (znki `firefly' *zunki). Exceptional patterns emerge when
// in the root morpheme is raised, followed by derivational suffixes [uw] and
[ij]. // is raised according to the preceding [-ATR] vowel; for example, when
the preceding vowel is //, // becomes [e] (lah-elehuw) and [o] otherwise. [16]
proposed that iterative harmony is gradient and vowels closer to the trigger
is affected more than the ones further away from the trigger. Our study
attempts to navigate these claims through modeling learnability
computationally.

##  3 Proposed Approach

The novelty of our study lies in computing iterative long-distance harmony
from raw speech in an unsupervised manner. Vowel harmony is a complex
phonological process requiring the learner to grasp crucial aspects like
directionality, domains, features, iteration, locality, and opacity [18]. The
vowel harmony system of Assamese provides a crucial case study for
directionality as it is neither stem-controlled nor dominant-recessive [15].
We train the model with Assamese speech data containing both harmonic and non-
harmonic words. We quantify the presence of [+ATR] harmony when the frequency
of F1 of the target vowel [-ATR] is much lower in the vicinity of the trigger
[+ATR] vowel than respective F2 and F3 [19, 20]. Regressive directionality is
anticipated when V2 explains the properties of V1 in a V1CV2 setting. We
analyze the output audios statistically and observe that although the
Generator strings vowels and consonants together to form a novel word, it does
follow a vowel sequence within the word domain. We also identify that some
innovative items have lexical meaning in Assamese, suggesting the emergence of
lexical learning from the fiwGAN model.

##  4 Model Architecture

The fiwGAN neural network model comprises three primary components: a
Generator, a Q-network, and a Discriminator (refer to Figure 1). Unlike
traditional GAN models, where the Generator typically takes uniformly
distributed latent variables as input, fiwGAN's Generator operates within a
latent space composed of binary codes ($\phi$) and latent variables ($z$).
Here, each binary variable ($\phi_{n}$) corresponds to a distinct feature
[21]. Trained to maximize the Discriminator's error rate while minimizing the
Q-network's error, the Generator produces a vector containing approximately
$16384$ data points for a $1$ second of audio data (sampled at $16$ kHz).
These generated outputs, along with real acoustic data, are then assessed by
the Discriminator, which computes the Wasserstein distance [22] between the
generated and real data.

In parallel, the Q-network, which shares the same input data as the
discriminator, operates differently. Its final layer comprises nodes
corresponding to categorical variables ($\phi$). Unlike the Discriminator, the
Q-network's loss function drives updates not only to its own weights but also
to those of the Generator. This mechanism enables the Generator to associate
each lexical item with a unique latent code, allowing the Q-network to
retrieve the lexical information solely from the acoustic signals. This
process facilitates lexical learning within the model. While the Generator
produces raw acoustic data similar to but not an exact replication of the real
input data, it effectively captures essential characteristics of the acoustic
signals. 111We used the Pytorch implementation of fiwGAN made available by
[21] at https://github.com/gbegus/ciwganFiwGAN-pytorch.git.

Figure 1: An illustration of fiwGAN architecture used in this work. We chose
$N=82$ spoken utterances corresponding to lexical items in the Assamese
language. The latent space contains 93 uniformly distributed latent variables,
z, and 7 binary features ($\phi$) for 27=128 lexical classes.

##  5 Materials

###  5.1 Assamese Speech Data

For this experiment, $15$ native Assamese speakers ($7$ males and $8$ females)
between the ages of $19-35$ were consulted. The speech data was recorded in
the soundproof booth at the Phonetics and Phonology laboratory of the Indian
Institute of Technology Guwahati (India) using a Tascam DR-100 MKII recorder.
The created speech dataset consisted of $82$ spoken words(40 harmonic and 42
non-harmonic)222The dataset is available at
https://github.com/sneha2599/IS2024_fiwGAN-for-vowel-harmony.git. Each target
word was in a carrier sentence written in Assamese script, that is, ``moi X
buli kolu'', which corresponds to ``I say X'' in English. The participants
were asked to utter each sentence at least four times. The collected speech
data was then curated through manual segmentation using the audio-visual
utility in PRAAT[23] software. This resulted in $4789$ clean speech utterances
(or tokens), each corresponding to an Assamese word.

Table 3: Example dataset for training fiwGAN

Stem -suffix Surface Harmony type dil -i dilei harmonic nokoril -u nokorileu
harmonic grm –t grmt non-harmonic bpr -i bpri non-harmonic

###  5.2 Model Implementation

The fiwGAN model takes 4789 (3169 harmonic and 1620 non-harmonic) 1-second-
long unannotated single-channel audio files sampled at 16 kHz in raw
waveforms. The latent space consists of 93 uniformly distributed latent
variables $z$ ($U$$\sim$(-1,1)) and 7 binary variables, allowing for 27=128
unique classes, represented as [1,0,0,0,0,0,0 for word1], [0,1,0,0,0,0 for
word2], etc. The Generator and Discriminator were trained with Adam optimizer
while the Q-network was trained with RMSProp algorithm at a .0001 learning
rate with a batch size of 64. It ran for 72 hours on an NVIDIA RTX A4500 GPU
at the CLST lab of IIT Guwahati. The Generator data was analyzed following 960
training epochs ($\sim$ 44000 steps). Based on their intelligibility and noise
quality, 64 out of 100 outputs were examined in our experiment after manual
listening and segmentation in Praat[23]. We extracted the first formants of
the vowels in training and generated data at 10-time points using Praat and
analyzed only the mean value to quantify the presence of ATR vowel
harmony[23], followed by regression analysis in R [24] to assess the
learnability of directionality and other discrete categories related to vowel
harmony.

(a) [pdobi](b) [debeku](c) [korisuw] Figure 2: Spectrograms of three fiwGAN
generated audio files: (a)[pdobi], also an illicit item; (b) [debeku], also an
innovative item following long-distance iterative harmony; and (c)[korisuw] a
novel word with lexical meaning in the Assamese language. Figure 3: F1
comparison of [podobi] (training data; shown in bars) and [pdobi] (generated
data; shown in hatched bars). Here, o1 and o2 denote the first and second
vowel, and i denotes the third vowel, in the input training data ``podobi''.

##  6 Results

On the completion of each training epoch, the Generator produces $100$ audio
files. In previous studies exploring English allophonic distributions and
French nasality [8, 11], the Generator learned to produce speech-like sounds
after close to $650$ epochs. In our case, the speech-like sounds were found to
be generated after $720$ epochs. Intelligible sounds were generated after
$800$ epochs. After $960$ epochs, the model started generating lexical items
identical to the training dataset ([prohori],[kbr],[plx]) and innovative
outputs ([dekhisi],[korobe],[korisuw],[debeku] etc.). Additionally, some
illicit forms ([nkorilu], [pdobi], etc.) were also found in the outputs. An
illustration of some outputs is shown in Figure 2.

Subsequently, we analyzed the innovative outputs and observed that the model
strings together vowels and consonants from the training data; for example,
[dekhisi] is assumed to be influenced by the word [dekhisu] in the input,
while [korobe] may stem from [korobi]. In the context of vowel harmony, the
novel items [debeku], [dekhisi], and [korisuw] follow the vowel harmony
pattern where [+high,+ATR] vowels [i,u] trigger harmony of the underlying
[-ATR] vowels turning them into [+ATR]. We observed that [e]/[o] occurs when
followed by [i]/[u] locally [korisuw/dekhisi] as well as over a longer domain
[debeku]. This suggests the capability of the model to learn the phonotactics
of Assamese and the iterative long-distance vowel harmony.

Analyzing the illicit outputs, we observed that the vowels in them follow a
specific pattern where the trigger vowels impact the immediately preceding
vowel, suggesting non-iterative harmony([pdobi] instead of [podobi]) (see
Figure 3). Moreover, [korisuw] is close to a real-world compound word [kori
suw'do see'] and a word the model has not encountered in the input, implying
the emergence of lexical learning from the training.

###  6.1 Statistical Analysis

We examined the generated outputs statistically to quantify the presence of
discrete categories related to vowel harmony in Assamese The formant values of
vowels in the training data were analyzed. It was observed that the first
formant frequency (F1) of the [-ATR] vowels is much lower in the vicinity of
the [+ATR] vowels (see Figure 3). We hypothesize that if V2 in the V1CV2
setting explains V1 better than V1 explains V2, the dataset follows regressive
directionality. If not, the directionality is assumed to be left-to-right. A
linear mixed-effects regression analysis was carried out on the training
dataset [25] (using R [24]). This helped examining the relationship between
the dependent variable F1V1 and predictor variables V1 and V2
333V1full.model=lmer(F1V1$\sim$V1+V2+(1$|$word), data=formants, REML=FALSE).
V1 and V2 are the fixed effects, while `word' is the random effect in the
model. The likelihood ratio test compares the full model to a null model. The
AIC score revealed a statistically significant difference between the two
models (AIC(full)=1588.8, AIC(null)=1595.9). The chi-square test indicates a
significant difference in model fit between the null and full models
($\chi\textsuperscript{2}$=33.062, df=13, p$<$0.001), reflecting a significant
effect of V2 on the F1 value of V1 (see Table 4). The dataset is then
categorized into two subsets: harmonic and non-harmonic. Both the subsets are
analyzed using the lmer model. The fixed-effects results from the full model
for harmony444V1vhfull=lmer(F1V1$\sim$V1+V2+(1$|$word),data=har, REML=FALSE)
showed a distinction between V1, V2, and the intercept representing F1 of V1.
The estimate of the intercept is about 552.129 Hz with a p-value$<$2e-16. A
comparison of the full and null model using the likelihood ratio test showed
that V2 affects V1 ($\chi\textsuperscript{2}$= 27.829, df = 7, p = 0.0002361),
which reveals that the contrast is statistically significant
(AIC(full)=467.04, AIC(null)=480.87)(see Table 4). Assuming the results from
statistical models for training data as a baseline, the generated items are
annotated and then fit to linear regression models in R to examine the
relationship between the dependent variable F1V1 and predictor variables V1
and F1V2 (see Table 5). The model was statistically significant (F(6, 15) =
9.504, p = 0.0002087), indicating that at least one predictor variable had a
non-zero effect. The Adjusted R-squared value of 0.7084 suggests that the
model explained approximately 70.84% of the variability in F1V1. The
coefficients for individual predictor variables were significant at various
levels. Our regression model suggests that V1 and, to some extent, F1V2 are
associated with variations in F1V1. Further analysis shows that the
coefficients of V2[T.i] are significantly higher than other V2 variables
(Estimate=-279.11, t-value=-3.376,p= 0.00817). This supports the idea that
[i], a [+high,+ATR] vowel in Assamese, acts as a triggering vowel in the
machine-generated outputs, suggesting the learnability of height as a trigger
feature for Assamese vowel harmony. The overall analysis suggests that the
model learns the right-to-left harmony pattern, which is evident in the
influence of V2 on V1 for both first and second formant frequencies.

Table 4: LMER model for the training dataset

Data | Directionality | Fixed effects | DF | $\chi\textsuperscript{2}$ | p
---|---|---|---|---|---
Whole | right-to-left | F1V1$\sim$V1+V2 | 13 | 33.062 |  $<$0.001
| left-to-right | F1V2$\sim$V2+V1 | 10 | 6.5156 | 0.77
Only [+ATR] | right-to-left | F1V1$\sim$V1+V2 | 7 | 27.829 |  $<$ 0.001
| left-to-right | F1V2$\sim$V2+V1 | 2 | 1.6522 | 0.43

Table 5: Linear regression model for machine-generated items

Data | Estimate | t-value | p-value
---|---|---|---
Whole | 605.25 | 7.793 | $<.001$
only V2[i] coefficient | -279.11 | 3.376 | $.01$

##  7 Conclusion

In summary, this study presented long-distance vowel pattern learning,
emphasizing learners' acquisition of intricate vowel changes crucial for
grasping vowel harmony. Unlike previous literature focusing solely on
phonological learning, our findings mark an advancement, leveraging deep
learning models to facilitate unsupervised learning of both acoustic and
articulatory properties alongside underlying segmental features and vowel
harmony. Through analysis of the Generator performance across epochs and
manipulated latent space variables, insights into human-like speech generation
have emerged. Our observations also indicate that, given limited data, the
fiwGAN is an effective phonotactic learner, which is evident from its ability
to learn complex systems like iterative long-distance harmony. Analysis of
innovative items revealed instances of long-distance harmony alongside error
items displaying non-iterative local harmony. The dominance of V2[i] as a
trigger vowel indicates the model's capacity to learn trigger features, while
statistical analysis suggests grammatical outputs following regressive
directionality 555[26] observed coarticulation propensity in Assamese non-
harmonic sequences which is strictly phonetic and not a goal of this paper..
Moreover, lexical learning emerged post-training. Notably, no results with the
opaque vowel [] were observed, prompting further inquiry into the model's
ability to identify vowel opacity.

## References

  * [1] A. Prince and P. Smolensky, ``Optimality Theory: Constraint interaction in generative grammar,'' _Optimality Theory in phonology: A reader_ , pp. 1–71, 2004, publisher: Wiley Online Library.
  * [2] B. Hayes and C. Wilson, ``A maximum entropy model of phonotactics and phonotactic learning,'' _Linguistic inquiry_ , vol. 39, no. 3, pp. 379–440, 2008, publisher: MIT press One Rogers Street, Cambridge, MA 02142-1209, USA journals-info ….
  * [3] J. Pater, ``Generative linguistics and neural networks at 60: Foundation, friction, and fusion,'' _Language_ , vol. 95, no. 1, pp. e41–e74, 2019.
  * [4] G. Beguš, ``Generative adversarial phonology: Modeling unsupervised phonetic and phonological learning with neural networks,'' _Frontiers in artificial intelligence_ , vol. 3, p. 44, 2020, publisher: Frontiers Media SA.
  * [5] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, ``Generative adversarial nets,'' _Advances in neural information processing systems_ , vol. 27, 2014.
  * [6] C. Donahue, J. McAuley, and M. Puckette, ``Adversarial audio synthesis,'' _arXiv preprint arXiv:1802.04208_ , 2018.
  * [7] A. Radford, L. Metz, and S. Chintala, ``Unsupervised representation learning with deep convolutional generative adversarial networks,'' _arXiv preprint arXiv:1511.06434_ , 2015.
  * [8] G. Beguš, ``Identity-based patterns in deep convolutional networks: Generative adversarial phonology and reduplication,'' _Trans.Association for Computational Linguistics_ , vol. 9, pp. 1180–1196, 2021.
  * [9] X. Chen, Y. Duan, R. Houthooft, J. Schulman, I. Sutskever, and P. Abbeel, ``Infogan: Interpretable representation learning by information maximizing generative adversarial nets,'' _Advances in neural information processing systems_ , vol. 29, 2016.
  * [10] G. Beguš and A. Zhou, ``Interpreting intermediate convolutional layers of generative CNNs trained on waveforms,'' _IEEE Trans. Audio, Speech, Lang. Process._ , vol. 30, pp. 3214–3229, 2022.
  * [11] J. Chen and M. Elsner, ``Exploring How Generative Adversarial Networks Learn Phonological Representations,'' _arXiv preprint arXiv:2305.12501_ , 2023.
  * [12] J. Blevins, _Evolutionary phonology: The emergence of sound patterns_. Cambridge University Press, 2004.
  * [13] S. Rose and R. Walker, ``Harmony systems,'' _The handbook of phonological theory_ , pp. 240–290, 2011.
  * [14] J. J. Ohala, ``Towards a universal, phonetically-based, theory of vowel harmony,'' in _Proc. Intl. Conf. on Spoken Lang. Processing (ICSLP)_ , 1994, pp. 491–494.
  * [15] S. Mahanta, _Directionality and locality in vowel harmony: With special reference to vowel harmony in Assamese_. Netherlands Graduate School of Linguistics, 2008.
  * [16] D. B. Archangeli and J. Yip, ``Assamese vowels and vowel harmony,'' _Journal of South Asian Languages and Linguistics_ , vol. 6, no. 2, pp. 151–183, 2020, publisher: De Gruyter.
  * [17] Government General of India, ``Census of India,'' 2011.
  * [18] D. Archangeli and D. Pulleyblank, ``Harmony,'' in _The Cambridge Handbook of Phonology_ , ser. Cambridge Handbooks in Language and Linguistics, P. d. Lacy, Ed. Cambridge University Press, 2007, pp. 353–378.
  * [19] T. H. Mintz, R. L. Walker, A. Welday, and C. Kidd, ``Infants' sensitivity to vowel harmony and its role in segmenting speech,'' _Cognition_ , vol. 171, pp. 95–107, 2018.
  * [20] P. Olejarczuk, M. A. Otero, and M. M. Baese-Berk, ``Acoustic correlates of anticipatory and progressive [ATR] harmony processes in Ethiopian Komo,'' _Journal of Phonetics_ , vol. 74, pp. 18–41, 2019.
  * [21] G. Beguš, ``CiwGAN and fiwGAN: Encoding information in acoustic data to model lexical learning with Generative Adversarial Networks,'' _Neural Networks_ , vol. 139, pp. 305–325, 2021, publisher: Elsevier.
  * [22] M. Arjovsky, S. Chintala, and L. Bottou, ``Wasserstein Generative Adversarial Networks,'' in _Proc. Intl. Conf. Machine Learning (ICML)_ , D. Precup and Y. W. Teh, Eds., vol. 70. PMLR, 2017, pp. 214–223.
  * [23] P. Boersma and D. Weenink, ``Praat: doing phonetics by computer (Version 5.1.13),'' 2009. [Online]. Available: http://www.praat.org
  * [24] R Core Team, _R: A Language and Environment for Statistical Computing_. Vienna, Austria: R Foundation for Statistical Computing, 2021. [Online]. Available: https://www.R-project.org/
  * [25] D. Bates, M. Mächler, B. Bolker, and S. Walker, ``Fitting Linear Mixed-Effects Models Using lme4,'' _Journal of Statistical Software_ , vol. 67, no. 1, pp. 1–48, 2015.
  * [26] I. Dutta, I. S., P. Gogoi, and P. Sarmah, ``Nature of Contrast and Coarticulation: Evidence from Mizo Tones and Assamese Vowel Harmony,'' in _Proc. Interspeech 2017_ , 2017, pp. 224–228.