# Deep learning for solution and inversion of structural mechanics and
vibrations ††thanks: A preprint to be published in: _Modeling and Computations
in Vibration Problems_ , Edited by S. Chakraverty, F. Tornabene, J. N. Reddy,
Institute of Physics Publishing. 2021.

Ehsan Haghighat
Department of Civil Engineering
University of British Columbia
Vancouver, BC
<EMAIL_ADDRESS>
& Ali C. Bekar
Department of Aerospace and Mechanical Engineering
University of Arizona
Tucson, AZ
<EMAIL_ADDRESS>
& Erdogan Madenci
Department of Aerospace and Mechanical Engineering
University of Arizona
Tucson, AZ
<EMAIL_ADDRESS>
& Ruben Juanes
Department of Civil and Environment Engineering
Massachusetts Institute of Technology
Boston, MA
<EMAIL_ADDRESS>

(October 2020)

###### Abstract

Deep learning has been the most popular machine learning method in the last
few years. In this chapter, we present the application of deep learning and
physics-informed neural networks concerning structural mechanics and vibration
problems. Demonstration problems involve de-noising data, solution to time-
dependent ordinary and partial differential equations, and characterizing the
system’s response for a given data.

_Keywords_ Deep learning $\cdot$ Physics informed $\cdot$ Structural mechanics
$\cdot$ Vibrations $\cdot$ Identification

##  1 Introduction

The Finite Element Method (FEM) is commonly employed to construct and analyze
a representative model of the structure and as a part of the inversion and
identification techniques. Inversion algorithms minimize the difference
between the experimental data and computational model through direct
displacement, mode shapes and, mode frequencies [1, 2, 3, 4]. Classical
studies primarily concern the identification of mechanical properties of the
system such as stiffness and damping by matching the numerical predictions
with experimental measurements. Recently, more fundamental statistical
techniques such as Kalman filter and recurrent neural networks are used to
identify the system and approximate its response [5, 6]

Deep learning (DL), a subclass of machine learning (ML) and artificial
intelligence (AI), has been in the forefront of recent advances in AI for
addressing challenging problems in computer vision and autonomous driving,
speech recognition and natural language processing, and e-commerce. Deep
learning uses deep neural network (NN) architectures including densely-
connected (DNN), convolutional (CNN), and recurrent (RNN) neural network
architectures to perform these tasks [7, 8, 9]. Training a deep neural network
can be achieved through supervised optimization in which the network is
trained on labeled data set that covers the expected outcomes. Alternatively,
an unsupervised approach can be adopted in which optimization is performed on
a set of targets without the use of labeled data.

Application of deep learning in science and engineering can be classified in
multiple categories. In the most natural way, DL models are trained on labeled
data to classify events and predict system response [10, 11, 12, 13]. They
have been employed to construct surrogate models of the systems for on-the-fly
response predictions [14, 15, 16, 17, 18, 19, 20]. Being trained on a large
set of simulated data, they are suitable for real-time inversion and system
identification [21, 22].

More recently, deep learning has been successfully employed to solve
differential and integral equations describing a physical phenomenon without
labeled data [23, 24, 25, 26, 27, 28, 29]. This class of deep learning, also
commonly refereed to as Physics-Informed Neural Networks (PINN), employs
neural networks as the approximate solution to the partial differential
equations (PDEs). Differentiation is performed by using the Automatic
Differentiation (AD) algorithm. The solution, i.e., the parameters of the
neural network, is identified by optimizing an objective or loss function that
incorporates the PDE residuals, boundary conditions, and initial conditions.
PINNs have been used for inverse analysis as well as surrogate modeling in
fluid and solid mechanics problems [24, 30, 27, 31]. Depending on the
application, different variations of PINNs are proposed to improve its
accuracy[32, 33, 34, 35]. The most attractive feature of PINNs is that it
enables forward solution, inversion, surrogate model construction, and data-
driven modeling all within the same framework.

This chapter focuses on the use of deep learning for structural vibration
problems through regression techniques, solution methods to the partial
differential equations, and inversion techniques.

##  2 Deep learning

Deep learning, a subclass of machine learning, employs deep neural networks. A
feed-forward neural network with a single hidden layer, and with inputs
$\boldsymbol{x}\in\mathbb{R}^{m}$, outputs $\boldsymbol{y}\in\mathbb{R}^{n}$,
and $d$ hidden units is schematically shown in Figure 1 and mathematically
expressed as

$\displaystyle\boldsymbol{y}=\boldsymbol{W}^{1}\cdot\sigma(\boldsymbol{W}^{0}\cdot\boldsymbol{x}+\boldsymbol{b}^{0})+\boldsymbol{b}^{1},$
(1)

where $\sigma$ is an activation function, such as _hyperbolic-tangent_ , that
makes the transformation nonlinear. The parameters,
$\boldsymbol{W}^{0}\in\mathbb{R}^{d\times m}$ and
$\boldsymbol{W}^{1}\in\mathbb{R}^{n\times d}$ are known as _weights_ of this
transformation and $\boldsymbol{b}^{0}\in\mathbb{R}^{d}$ and
$\boldsymbol{b}^{1}\in\mathbb{R}^{n}$ are _biases_. Randomly initialized
components of these vectors and matrices define the degrees of freedom (DOF)
of this transformation. Therefore, the transformation eq. 1 has $(d+1)\times
n+(m+1)\times d$ parameters (DOFs) to be identified. Throughout this study,
all of the trainable parameters of the network are collected into a vector of
size $D$ and denoted as $\boldsymbol{\theta}\in\mathbb{R}^{D}$. It was shown
by Hornik et al. [36] that this transformation can approximate any measurable
function.

Figure 1: A feed-forward neural network with a single hidden layer with $d$
units, input features $\boldsymbol{x}\in\mathbb{R}^{m}$ and outputs
$\boldsymbol{y}\in\mathbb{R}^{n}$. The connections highlight the weights of
each layer, i.e. $\boldsymbol{W}^{0}\in\mathbb{R}^{d\times m}$ and
$\boldsymbol{W}^{1}\in\mathbb{R}^{n\times d}$. The bias terms
$\boldsymbol{b}^{0}\in\mathbb{R}^{d}$ and
$\boldsymbol{b}^{1}\in\mathbb{R}^{n}$ sit at each node of this graph. This
network constructs the functional form expressed in eq. 1.

To extend the single layer neural network eq. 1 to multi-layers, let us define
the transformation $\Sigma$ as
$\Sigma^{i}(\hat{\boldsymbol{x}}^{i}):=\hat{\boldsymbol{y}}^{i}=\sigma^{i}(\boldsymbol{W}^{i}\cdot\hat{\boldsymbol{x}}^{i}+\boldsymbol{b}^{i})$,
with $\hat{\boldsymbol{x}}^{i}$ as the input and $\hat{\boldsymbol{y}}^{i}$ as
the output of any hidden layer $i$, $\boldsymbol{x}=\hat{\boldsymbol{x}}^{0}$
as the main input to the network, and
$\boldsymbol{y}=\Sigma^{L}(\hat{\boldsymbol{x}}^{L})$ as the final output of
the network. Thus, we can construct a general neural network with $L$ hidden
layers as composition of $\Sigma^{i}$ transformation as

$\displaystyle\boldsymbol{y}=\Sigma^{L}\circ\Sigma^{L-1}\circ\dots\circ\Sigma^{0}(\boldsymbol{x}).$
(2)

This chapter focuses on regression-type problems or solution methods for PDEs;
therefore, we only employ this type of networks, known as _densely-connected_
networks or _multilayer perceptron models_ (when _sigmoid_ activation
functions are employed). There are different variations of neural networks
that are designed for other applications including _Convolutional_ and
_Recurrent_ networks [9] for computer vision and natural language processing,
respectively. In the context of Physics-Informed Neural Networks, these
networks are not currently applicable for solution and inversion of the
partial differential equations considered in this chapter.

The initial values for $\boldsymbol{y}(\boldsymbol{x}^{*})$ are random since
the parameters of neural network, i.e. $\boldsymbol{\theta}$, are randomly
initialized. These parameters are identified using supervised or unsupervised
optimization. As an example, the parameters of a regression problem can be
identified by optimizing the following objective (loss) function,

$\displaystyle\boldsymbol{\theta}^{*}=\operatorname*{arg\,min}_{\boldsymbol{\theta}\in\mathbb{R}^{D}}\mathcal{L}(\boldsymbol{\theta}):=|\boldsymbol{y}(\boldsymbol{x}^{*};\boldsymbol{\theta})-\hat{\boldsymbol{y}}^{*}|\quad\mathrm{where}\quad|\circ|=\mathrm{MSE}(\circ).$
(3)

Here, the asterisked variables $\circ^{*}$ are the training or collocation
points. The loss term is constructed using the mean-squared error (MSE).
Optimizing the loss function results in values for the parameters of neural
network, i.e., $\boldsymbol{\theta}^{*}$, that minimizes the error between
network outputs $\boldsymbol{y}(\boldsymbol{x}^{*}_{i})$ and their expected
values $\boldsymbol{y}_{i}^{*}$. We will discuss optimization techniques in
the following sections.

##  3 Physics-Informed Neural Networks

Physics-Informed Neural Networks (PINN) proposed by Raissi et al. [24] is a
robust approach to solve differential equations and to perform inversion.
According to this architecture, the solution variables are approximated using
neural networks with space and time variables as the network features.
Differentiation is performed using Automatic Differentiation (AD) [37].
Initial and boundary conditions as well as differential equations are included
in the loss function. The parameters of the neural network are identified by
optimizing the loss function. The main advantage of this approach is that the
given data related to any variable can be incorporated into the loss function.

This approach can be illustrated by considering the following Laplace’s
equation

$\displaystyle\kappa\nabla^{2}f=\kappa\left(\frac{\partial^{2}}{\partial
x^{2}}+\frac{\partial^{2}}{\partial y^{2}}\right)f=0,$
$\displaystyle\mathrm{for}$ $\displaystyle~{}\boldsymbol{x}\in\Omega,$ (4)
$\displaystyle f=\bar{f},$ $\displaystyle\mathrm{for}$
$\displaystyle~{}\boldsymbol{x}\in\Gamma_{f},$ (5)
$\displaystyle\kappa\frac{\partial f}{\partial n}=\kappa\nabla
f\cdot\boldsymbol{n}=\bar{q},$ $\displaystyle\mathrm{for}$
$\displaystyle~{}\boldsymbol{x}\in\Gamma_{q},$ (6)

where $\kappa$ is a material parameter, $\Omega$ and
$\Gamma=\Gamma_{f}\cup\Gamma_{q}$ describe the domain and its boundary,
respectively. The domain is subjected to Dirichlet and Neumann boundary
conditions $\bar{f}$ and $\bar{q}$ on $\Gamma_{f}$ and $\Gamma_{q}$,
respectively.

Assuming the solution to this problem is $f=f(x,y)$, the solution can be
approximated using a neural network expressed as

$\displaystyle
f(x,y)\approx\hat{f}(x,y;\boldsymbol{\theta}):(x,y)\rightarrow\mathcal{N}_{f}(x,y;\boldsymbol{\theta}).$
(7)

To find the solution to the Laplace’s equation, i.e., to find
$\boldsymbol{\theta}^{*}\in\mathbb{R}^{D}$ that satisfies relations eqs. 4, 5
and 6, we need to perform the following optimization:

$\displaystyle\begin{split}\boldsymbol{\theta}^{*}=\operatorname*{arg\,min}_{\boldsymbol{\theta}\in\mathbb{R}^{D}}\mathcal{L}&:=\lambda_{1}|\kappa(\frac{\partial^{2}\hat{f}}{\partial
x^{2}}+\frac{\partial^{2}\hat{f}}{\partial y^{2}})|\\\
&+\lambda_{2}|\delta(\boldsymbol{x}\in\Gamma_{f})(\hat{f}-\bar{f})|\\\
&+\lambda_{3}|\delta(\boldsymbol{x}\in\Gamma_{q})(\kappa\frac{\partial\hat{f}}{\partial
n}-\bar{q})|\end{split}$ (8)

where $\delta$ is the Dirac function and $|\circ|$ represents the MSE norm.
$\lambda_{i}$ are the weights associated with each term in the loss function.
Optimization of the total loss $\mathcal{L}$ results in the solution to the
Laplace equation eqs. 4, 5 and 6.

The inversion with PINN can be illustrated by considering the same problem. In
the inversion process, we will find the parameter of the system, i.e.,
$\kappa$ based on the available measurements. Let us assume that the field
variable $f$ is measured on a set of discrete points
$\boldsymbol{x}^{*}\in\Omega$. To use PINN for inversion, we need to also
define $\kappa$ as a trainable parameter. Additionally, we can incorporate a
loss term for the measurements on $f$. Therefore, the optimization problem eq.
8 is modified as follows:

$\displaystyle\begin{split}\kappa^{*},\boldsymbol{\theta}^{*}=\operatorname*{arg\,min}_{\kappa\in\mathbb{R},~{}\boldsymbol{\theta}\in\mathbb{R}^{D}}\mathcal{L}&:=\lambda_{1}|\kappa(\frac{\partial^{2}\hat{f}}{\partial
x^{2}}+\frac{\partial^{2}\hat{f}}{\partial y^{2}})|\\\
&+\lambda_{2}|\delta(\boldsymbol{x}\in\Gamma_{f})(\hat{f}-\bar{f})|\\\
&+\lambda_{3}|\delta(\boldsymbol{x}\in\Gamma_{q})(\kappa\frac{\partial\hat{f}}{\partial
n}-\bar{q})|\\\
&+\lambda_{4}|\delta(\boldsymbol{x}^{*}\in\Omega)(\hat{f}-f^{*})|.\end{split}$
(9)

where $f^{*}$ are measurements at $\boldsymbol{x}^{*}$ points. There exist
other variations of PINNs to address different types of PDEs including
Variations, Parareal, Domain Decomposed, and Nonlocal PINNS [25, 32, 35, 33].
However, we limit the scope of this chapter to the standard PINN.

##  4 Training Neural Networks

The goal of training or optimization is to find a set of parameters
$\boldsymbol{\theta}^{*}$ that minimizes the loss function
$\mathcal{L}(\boldsymbol{\theta}):\mathbb{R}^{D}\rightarrow\mathbb{R}$, where
$D$ is the total number of parameters. For the purpose of illustration,
considering the loss function associated with the Laplace’s equation, Taylor’s
theorem for the loss function $\mathcal{L}$ leads to

$\displaystyle\mathcal{L}(\boldsymbol{\theta}+\boldsymbol{\vartheta})=\mathcal{L}(\boldsymbol{\theta})+\nabla\mathcal{L}(\boldsymbol{\theta})\cdot\boldsymbol{\vartheta}+\frac{1}{2}\boldsymbol{\vartheta}\cdot\nabla^{2}\mathcal{L}(\boldsymbol{\theta}+t\boldsymbol{\vartheta})\cdot\boldsymbol{\vartheta},$
(10)

for $t\in(0,1)$, where $\nabla$ and $\nabla^{2}$ are gradient and _Hessian_
operators. The necessary conditions for $\boldsymbol{\theta}^{*}$ to be a
local minimizer is that the gradient vector
$\nabla\mathcal{L}(\boldsymbol{\theta}^{*})=\boldsymbol{0}$ and the Hessian
matrix to be positive definite, i.e.,
$\boldsymbol{v}\cdot\nabla^{2}\mathcal{L}(\boldsymbol{\theta}^{*})\cdot\boldsymbol{v}>0$
for any arbitrary $\boldsymbol{v}\in\mathbb{R}^{D}$. These conditions imply
that when $\mathcal{L}$ is convex, any local minimizer
$\boldsymbol{\theta}^{*}$ is a global minimizer. Note that for neural
networks, the loss function is often non-convex.

There are two family of algorithms to solve this optimization problem. The
first one is known as _line search_ or first-order methods and the second one
as the _trust region_ or second-order methods [38]. Line search methods are
based on moving along a direction that most rapidly decreases the loss
function. It can be easily shown that if we write the Taylor’s expansion for
$\mathcal{L}(\boldsymbol{\theta}+\alpha\boldsymbol{\vartheta})$ up to the
first order, the maximum drop in $\mathcal{L}(\boldsymbol{\theta})$ happens
when we move in the $\boldsymbol{\vartheta}=-\nabla\mathcal{L}$ direction.
This builds the foundation for the first-order _gradient descent_ methods.
There are numerous ways to choose a _step-size_ or _learning-rate_ $\alpha$.
Among those, the most commonly used approach is _Adam_ methods, where $\alpha$
is chosen adaptively for each parameter $\theta_{i}$ based on the first and
second momentum updates [39].

The most fundamental second-order method also known as _trust-region_ is the
Newton’s method, which can be derived by setting the variation of the second
order Taylor’s expansion of the loss
$\mathcal{L}(\boldsymbol{\theta}+\boldsymbol{\vartheta})$ to zero. This
results into a direction update
$\boldsymbol{\vartheta}=-(\nabla^{2}\mathcal{L})^{-1}\cdot\nabla\mathcal{L}$.
By comparison to the first order updates, the learning rate $\alpha$ is
replaced by the inverse of the Hessian matrix. Evaluation of the inverse of
Hessian matrix and its memory storage can become quickly unmanageable for
high-dimensional optimization problems. Therefore, approximations of the
inverse of Hessian matrix is desirable. There are numerous methods that are
designed to address this drawback. Among those, BFGS algorithm and its
variations are among the most commonly known approaches. We refer the
interested readers to [38, 40] for additional details.

As we find, relations eq. 8 and eq. 9 represent multi-objective optimization
and $\lambda_{i}$ highlight the scaling factor for each term. These factors
are used to scale each term in the loss function so that the optimizer can
find the global minimum more efficiently. While they can be tuned manually to
increase or decrease the importance of each term, there are numerous
suggestions that can be used to automatically scale each term in the loss
function, including a weighted sum approach based on the value of loss
function [41]. For the physics-informed neural networks, Wang et al. [42]
propose a scaling approach based on balancing the distribution of gradients of
each term in the loss function. This approach is adopted for the problems
solved in this chapter.

##  5 Applications of Deep Learning for Data Representation

This section presents the applications of deep learning by considering
vibration problems relevant to civil engineering. These applications concern
regression and smoothing data, solution and characterization of single-degree
of freedom rigid-block, membranes, and plates. All the problems presented in
this chapter are solved using the SciANN [43] project and the codes are shared
in this github repository SciANN-Vibrations.

###  5.1 Polynomial regression

Polynomial regression can be performed through a neural network as

$\displaystyle\boldsymbol{y}=\boldsymbol{b}+\boldsymbol{W}_{1}\cdot\boldsymbol{x}+\boldsymbol{W}_{2}\cdot\boldsymbol{x}^{2}+\dots+\boldsymbol{W}_{p}\cdot\boldsymbol{x}^{p}$
(11)

This defines a polynomial regression model of order $p$. The main advantage of
constructing such a model using neural networks is that we can leverage the AD
algorithm for optimization as well as for the PINN framework. The input
features are denoted as
$(\boldsymbol{x},\boldsymbol{x}^{2},\dots,\boldsymbol{x}^{p})$. For $p=1$, it
a linear regression model and $p=2$ is a quadratic regression model.

The simplest neural network is a linear model expressed as

$\displaystyle\boldsymbol{y}=\mathbf{W}\cdot\boldsymbol{x}+\mathbf{b}$ (12)

The dimension of weight matrix $\boldsymbol{W}$ and bias vector
$\boldsymbol{b}$ depend on the size of inputs $\boldsymbol{x}$ and outputs
$\boldsymbol{y}$.

As shown in Figure 2, for a given noisy dataset $\mathbf{X},\mathbf{Y}^{*}$
containing $N$ data points, the input and output have a dimension of 1, i.e.,
$\boldsymbol{W}\in\mathbb{R}^{1}$ and $\boldsymbol{b}\in\mathbb{R}^{1}$.
Therefore, the total number of parameters for the training is 2, i.e.,
$\boldsymbol{\theta}=(\boldsymbol{W},\boldsymbol{b})\in\mathbb{R}^{2}$.
According to eq. 3, the optimization problem can be defined as

$\displaystyle\operatorname*{arg\,min}_{\boldsymbol{\theta}\in\mathbb{R}^{2}}\mathcal{L}:=|y(x_{i})-y^{*}_{i}|=\sum_{i=1}^{N}\frac{\Big{(}(W~{}x_{i}+b)-y^{*}_{i}\Big{)}^{2}}{N}$
(13)

Minimization of this problem results in optimized values for $W\approx 2.0$
and $b\approx 1.0$.

Figure 2: Use of a simple neural network architecture to perform linear
regression on noisy measurements. The optimizer correctly approximates the
_true_ function.

For the noisy dataset $(\mathbf{X},\mathbf{Y}^{*})$ shown in Figure 3, we can
construct a quadratic regression model with neural networks and train the
model on this dataset. Accordingly, the total number of parameters for this
model is 3, i.e., $\boldsymbol{\theta}\in\mathbb{R}^{3}$. The optimization
problem becomes:

$\displaystyle\operatorname*{arg\,min}_{\boldsymbol{\theta}\in\mathbb{R}^{3}}\mathcal{L}:=|y(x_{i})-y^{*}_{i}|=\sum_{i=1}^{N}\frac{\Big{(}(W_{1}~{}x_{i}+W_{2}~{}x_{i}^{2}+b)-y^{*}_{i}\Big{)}^{2}}{N}$
(14)

Minimization of this objective functions results in optimized values for
$W_{1}\approx-1$, $W_{2}\approx 2$ and $b\approx 1$.

Figure 3: Use of a simple neural network for quadratic regression on the noisy
data. The inputs features are $x,x^{2}$. The network correctly identifies the
_true_ underlying model for data.

###  5.2 Smoothing noisy vibration measurements

Noisy vibration measurements are considered to illustrate a more complex
neural network architecture for deep learning. The data is generated by
employing the following periodic function, also shown in Figure 4:

$\displaystyle y=A~{}(\sin\omega
t-\beta\sin\bar{\omega}t)+\varepsilon,\quad\varepsilon\sim N(0,\sigma^{2})$
(15)

with $A=1$, $\beta=1$, $\omega=2\pi/2$, and $\bar{\omega}=2\pi/1.5$.

The objective is to construct a smooth and continuous representation of the
noisy measurements. This representation is often necessary to evaluate the
parameters of the system. Since the data set is intrinsically periodic, we set
up a Fourier network with a sinusoidal activation function to perform the
regression. The functional form of the neural network is expressed as

$\displaystyle\boldsymbol{y}=\boldsymbol{W}^{2}\cdot\sin(\boldsymbol{W}^{1}\cdot\boldsymbol{x}+\boldsymbol{b}^{1})+\boldsymbol{b}^{2}.$
(16)

This single layer neural network represents the following Fourier series

$\displaystyle
y=\sum_{i=1}^{N_{n}}W^{2}_{i}\sin(W^{1}_{i}t+b^{1}_{i})+b^{2}_{i},$ (17)

where $N_{n}$ is the number of neurons in the hidden layer. We determine
different frequencies $\boldsymbol{W}^{1}$, phases $\boldsymbol{b}^{1}$, and
amplitudes $\boldsymbol{W}^{2}$. As shown in Figure 4, this network identifies
the true distribution of the data without _overfitting_ to the noise.

Figure 4: Use of a single-layer network for regression on noisy periodic data.
The network accurately predicts the underlying model of the data.

##  6 Deep Learning for Solution and Inversion of Vibration Problems

This section presents the application of deep learning for solving and
inversion of time dependent ordinary and partial differential equations.

###  6.1 Forced vibration spring-mass problem

The forced vibration of spring-mass system can be described by

$\displaystyle\ddot{u}+\omega^{2}~{}u$ $\displaystyle=f_{0}\sin\bar{\omega}t,$
(18) $\displaystyle u(t=0)$ $\displaystyle=0,$ (19)
$\displaystyle\dot{u}(t=0)$ $\displaystyle=0,$ (20)

where $\omega^{2}=k/m$ is the natural frequency of the spring-mass system with
$m$ and $k$ as its mass and stiffness, respectively. With $F_{0}=m~{}f_{0}$
and $\bar{\omega}$ as the amplitude and frequency of the applied external
load, the analytical solution to this system is expressed as

$\displaystyle
u(t)=\frac{F_{0}}{\omega^{2}-\bar{\omega}^{2}}\Big{(}\sin\bar{\omega}t-\beta\sin\omega
t\Big{)},$ (21)

with $\beta$ as $\beta=\bar{\omega}/\omega$.

Neural networks can be used to solve the response of this initial value
problem. Employing the PINN framework explained earlier, the solution space
can be approximated with a multi-layer neural network
$\hat{u}=\mathcal{N}_{u}(t;\boldsymbol{\theta})$, and the ODE residual is
evaluated using AD. The optimization problem is then expressed as:

$\displaystyle\operatorname*{arg\,min}_{\boldsymbol{\theta}\in\mathbb{R}^{D}}\mathcal{L}(\boldsymbol{\theta}):=\lambda_{1}|\frac{\partial^{2}\hat{u}}{\partial
t^{2}}+\omega^{2}\hat{u}-f_{0}\sin\bar{\omega}t|+\lambda_{2}|\hat{u}(t=0)|+\lambda_{3}|\frac{\partial\hat{u}}{\partial
t}(t=0)|$ (22)

Figure 5 presents the results for a system with $\omega=3$, $\beta=1.5$, and
$F_{0}=1$. The displacement $u$ is approximated with a network with 4 hidden
layers, each with 20 neurons, and with hyperbolic-tangent as the activation
function. The data is generated for $t\in[0,4\pi]$ and tested for
$t\in[0,8\pi]$. Two networks with hyperbolic-tangent and sinusoidal activation
functions are considered, both with 4 hidden layers and 20 neurons per layer.
The results show much more accuracy and predictive capability for the
sinusoidal network, as it is naturally a more suitable choice due the periodic
nature of the data.

Figure 5: Forward solution of the forced vibration problem for $\omega=3$,
$\beta=1.5$, and $F_{0}=1$. Neural network has 4 hidden layers with 20
neurons. For the left plot, $\mathrm{tanh}$ activation function is used while
for the right plot, the activation function is $\mathrm{sin}$. The highlighted
grey area show the training domain. The network on the right provides better
predictive capability and therefore has learned the underlying dynamics
imposed on the network.

Next, we assume that a displacement history for the problem is given, and we
want to characterize the system, i.e., identifying the natural frequency of
the problem. Mathematically, the optimization problem is now expressed as

$\displaystyle\operatorname*{arg\,min}_{\boldsymbol{\theta}\in\mathbb{R}^{D},~{}\omega\in\mathbb{R}}\mathcal{L}(\boldsymbol{\theta},\omega):=\lambda_{1}|\frac{\partial^{2}\hat{u}}{\partial
t^{2}}+\omega^{2}\hat{u}^{2}-f_{0}\sin\bar{\omega}t|+\lambda_{2}|\hat{u}-u^{*}|.$
(23)

This framework can be easily adopted for both forward and inversion problems.
The result of this optimization is a neural network model that gives the
solution for the discrete displacement data as well as identification of the
frequency of the system. The results are shown in fig. 6.

Figure 6: Inversion solution of the forced vibration problem on a dataset
generated with $\omega=3$, $\beta=1.5$, and $F_{0}=1$. Neural network has 4
hidden layers with 20 neurons. For the left plot, $\mathrm{tanh}$ activation
function is used while for the right plot, the activation function is
$\mathrm{sin}$. The highlighted grey area show the training domain. The
network on the right provides better predictive capability and therefore has
learned the underlying dynamics imposed on the network.

###  6.2 Free vibration of rectangular membrane

A rectangular membrane with unit dimensions $1\times 1$ is excited by an
initial displacement $u=\sin\pi x\sin\pi y$. The governing PDE and boundary
conditions are expressed as

$\displaystyle c\nabla^{2}u=c\left(\frac{\partial^{2}}{\partial
x^{2}}+\frac{\partial^{2}}{\partial
y^{2}}\right)u=\frac{\partial^{2}u}{\partial t^{2}},$
$\displaystyle\mathrm{for}$
$\displaystyle~{}\boldsymbol{x}\in[0,1]^{2},~{}t>0,$ (24) $\displaystyle u=0,$
$\displaystyle\mathrm{for}$ $\displaystyle~{}\boldsymbol{x}\in\Gamma_{u},$
(25) $\displaystyle u=\sin{\pi x}\sin{\pi y},$ $\displaystyle\mathrm{for}$
$\displaystyle~{}t=0,$ (26) $\displaystyle\frac{\partial u}{\partial t}=0,$
$\displaystyle\mathrm{for}$ $\displaystyle~{}t=0,$ (27)

where $u$ is the displacement, and $c$ is the wave propagation speed. The
analytical solution to governing equation is $u(x,y,t)=\sin\pi x\sin\pi
y\cos\sqrt{2}\pi t$.

To solve this problem using PINN, the solution space can be constructed using
a neural network
$\hat{u}(x,y,t;\boldsymbol{\theta})=\mathcal{N}_{u}(x,y,t;\boldsymbol{\theta})$.
The forward optimization problem is expressed as

$\displaystyle\begin{split}\operatorname*{arg\,min}_{\boldsymbol{\theta}\in\mathbb{R}^{D}}\mathcal{L}(\boldsymbol{\theta})&:=\lambda_{1}|c(\frac{\partial^{2}\hat{u}}{\partial
x^{2}}+\frac{\partial^{2}\hat{u}}{\partial
y^{2}})-\frac{\partial^{2}\hat{u}}{\partial t^{2}}|\\\
&+\lambda_{2}|\delta(\boldsymbol{x}\in\Gamma_{u})(\hat{u})|\\\
&+\lambda_{3}|\delta(t)(\hat{u}-\sin{\pi x}\sin{\pi y})|\\\
&+\lambda_{4}|\delta(t)(\frac{\partial\hat{u}}{\partial t})|.\end{split}$ (28)

To perform the optimization, a collocation grid is constructed with
$N_{x}=N_{y}=40$, $N_{t}=20$, and terminal time is $T=1/(2\sqrt{2})$ unit. The
stiffness coefficient $c$ is specified as $c=1$. We use a neural network with
4 hidden layers, each containing 20 neurons, and with sinusoidal activation
function. The training is performed by using $20,000$ epochs. The out of plane
displacement as well as the error plot are shown in fig. 7. As we find, the
PINN framework can accurately solve this problem.

Figure 7: The results of the PINN solution to the membrane problem at $t=0.5$.
(Left) out of plane deflection of the membrane. (Right) the absolute error
between network predictions and analytical solution.

For the inverse problem, the dataset for training is generated using the
analytical solution. Same test case is used but with a terminal time of
$T=1.0s$ with discretization parameters of $N_{x}=N_{y}=N_{t}=40$. The
optimization problem becomes

$\displaystyle\operatorname*{arg\,min}_{c\in\mathbb{R},~{}\boldsymbol{\theta}\in\mathbb{R}^{D}}\mathcal{L}(c,\boldsymbol{\theta})$
$\displaystyle:=\lambda_{1}|c(\frac{\partial^{2}\hat{u}}{\partial
x^{2}}+\frac{\partial^{2}\hat{u}}{\partial
y^{2}})-\frac{\partial^{2}\hat{u}}{\partial
t^{2}}|+\lambda_{2}|\hat{u}-u^{*}|$ (29)

This optimization results in a value for parameter $c$ as well as a solution,
in the form of a neural network, that satisfies the membrane governing
equation. The results are shown in Figure 8. As you find, the PINN framework
accurately captures the parameter of the problem, i.e., $c$, as well as the
solution to this problem.

Figure 8: Inversion solution to the membrane problem. (Top-left) The out of
plane deformation of the membrane at $t=0.5$. (Top right) The absolute error
at $t=0.5$ for the out of plane deformation. (Bottom) The training history for
the system parameter $c$, which has been accurately predicted by the PINN
framework.

###  6.3 Free vibration of a rectangular plate

The governing equation describing the out of plane motion of a plate is
described as

$\displaystyle D\nabla^{4}u$
$\displaystyle=D\left(\frac{\partial^{4}}{\partial
x^{4}}+2\frac{\partial^{4}}{\partial x^{2}y^{2}}+\frac{\partial^{4}}{\partial
y^{4}}\right)u=-\rho\frac{\partial^{2}u}{\partial t^{2}},$ (30)

where $D$ is the flexural stiffness given as $D=\frac{Eh^{3}}{12(1-\nu^{2})}$,
with $E$ as Young’s modulus, $h$ as thickness of the plate, and $\nu$ as the
Poisson’s ratio. The in-plane dimensions of the plate are specified as unity.
The plate is subjected to the following initial and boundary conditions:

$\displaystyle u=0,$ $\displaystyle\mathrm{for}$
$\displaystyle\quad\boldsymbol{x}\in\Gamma_{u},$ (31)
$\displaystyle\frac{\partial^{2}u}{\partial n^{2}}=0,$
$\displaystyle\mathrm{for}$ $\displaystyle\quad\boldsymbol{x}\in\Gamma_{u},$
(32) $\displaystyle u=\sin{\pi x}\sin{\pi y},$ $\displaystyle\mathrm{for}$
$\displaystyle\quad t=0,$ (33) $\displaystyle\frac{\partial u}{\partial t}=0,$
$\displaystyle\mathrm{for}$ $\displaystyle\quad t=0.$ (34)

The analytical solution to the governing equation can be constructed as

$\displaystyle u(x,y,t)=\sin(\pi x)\sin(\pi y)\cos(\frac{4\pi^{4}D}{\rho}t).$
(35)

The solution to this problem with PINN requires the construction of a neural
network as $\hat{u}(x,y,t)=\mathcal{N}_{u}(x,y,t;\boldsymbol{\theta})$. The
optimization problem becomes

$\displaystyle\begin{split}\operatorname*{arg\,min}_{\boldsymbol{\theta}\in\mathbb{R}^{D}}\mathcal{L}(\boldsymbol{\theta})&:=\lambda_{1}|D(\frac{\partial^{4}\hat{u}}{\partial
x^{4}}+\frac{\partial^{4}\hat{u}}{\partial
x^{2}y^{2}}+\frac{\partial^{4}\hat{u}}{\partial
y^{4}})+\rho\frac{\partial^{2}\hat{u}}{\partial t^{2}}|\\\
&+\lambda_{2}|\delta(\boldsymbol{x}\in\Gamma_{u})(\hat{u})|\\\
&+\lambda_{3}|\delta(\boldsymbol{x}\in\Gamma_{u})(\frac{\partial^{2}\hat{u}}{\partial
n^{2}})|\\\ &+\lambda_{4}|\delta(t)(\hat{u}-\sin{\pi x}\sin{\pi y}/100)|\\\
&+\lambda_{5}|\delta(t)(\frac{\partial\hat{u}}{\partial t})|\end{split}$ (36)

The specified plate parameters are: thickness as $h=0.004m$, Poisson’s ratio
as $\nu=0.25$, Young’s modulus as $E=7E+10Pa$, and density as
$\rho=2700kg/m^{3}$. The optimization is performed with a uniform grid
$N_{x}=N_{y}=20$, $N_{t}=40$ and a terminal time $T=0.1s$. The network
consists of 4 hidden layers with 40 neurons in each layer and sine activation
function. The training is conducted with $1000$ epochs and a batch size of
$512$. The results are shown in Figure 9. The out of plane displacements
predictions, in this case, is less accurate than the membrane problem. This is
associated with the higher order of the governing equations and could be
improved increasing the weight associated with the boundary conditions.

Figure 9: (Left) Predicted out of plane deformation using PINN. (Right) The
absolute error between the predicted PINN results and the analytical solution.

For the inversion, the dataset for training is generated using the analytical
solution. The same test case is used but with a terminal time of $T=1.0s$ and
discretization parameters of $N_{x}=N_{y}=N_{t}=40$. The optimization problem
then becomes

$\displaystyle\begin{split}\operatorname*{arg\,min}_{D\in\mathbb{R},~{}\boldsymbol{\theta}\in\mathbb{R}^{D}}\mathcal{L}(D,\boldsymbol{\theta})&:=\lambda_{1}|D(\frac{\partial^{4}\hat{u}}{\partial
x^{4}}+\frac{\partial^{4}\hat{u}}{\partial
x^{2}y^{2}}+\frac{\partial^{4}\hat{u}}{\partial
y^{4}})+\rho\frac{\partial^{2}\hat{u}}{\partial t^{2}}|\\\
&+\lambda_{2}|\hat{u}-u^{*}|\end{split}$ (37)

Also, the parameter $D$ is normalized using density value as $\hat{D}=D/\rho$
. The ground truth value of the normalized parameter is $0.147$. The change in
loss through the optimization process and inverted parameter $c$ is depicted
in Figure 10.

Figure 10: Identification of plate-vibration problem. (Top-left) The solution
to the problem using PINN as a result of combined solution and inversion
algorithm. (Top-right) The absolute error between neural network predictions
and analytical solution for out of plane deformation. (Bottom) Evolution of
mean-squared error for the inverse problem with accurate prediction for the
value of the normalized parameter $\hat{D}=D/\rho$.

##  7 Discussions and final remarks

This chapter presents a brief description of deep learning methods and their
application concerning structural vibrations. In describing the applicability
of these method, the datasets are generated using the analytical solutions.
However, these DL methods can handle data-driven solutions and data-driven
discovery of the parameters. The results presented in this chapter demonstrate
that PINN can solve several types of structural vibration problems with high
accuracy. Thus, DL predictions could provide a way to assist in the prediction
and optimization of vibration characteristics and system properties. It is
worth emphasizing that training neural networks are often challenging due to
initialization method and optimization algorithms. Once the correct network
architecture is identified, they can be effectively trained for the problem in
hand.

## References

  * HALL et al. [1970] BERT HALL, E CALKIN, and M SHOLAR.  Linear estimation of structural parameters from dynamic test data.  In _11th Structures, Structural Dynamics, and Materials Conference_ , page 1521, 1970.
  * Kabe [1985] Alvar M Kabe.  Stiffness matrix adjustment using mode data.  _AIAA journal_ , 23(9):1431–1436, 1985.
  * Ahmadian et al. [1997] H Ahmadian, GML Gladwell, and F Ismail.  Parameter selection strategies in finite element model updating.  1997\.
  * Friswell et al. [1998] MI Friswell, DJ Inman, and Deborah F Pilkey.  Direct updating of damping and stiffness matrices.  _AIAA journal_ , 36(3):491–493, 1998.
  * Yan et al. [2017] Gang Yan, Hao Sun, and Oral Büyüköztürk.  Impact load identification for composite structures using bayesian regularization and unscented kalman filter.  _Structural Control and Health Monitoring_ , 24(5):e1910, 2017.
  * Zhang et al. [2019] Ruiyang Zhang, Zhao Chen, Su Chen, Jingwei Zheng, Oral Büyüköztürk, and Hao Sun.  Deep long short-term memory networks for nonlinear structural seismic response prediction.  _Computers & Structures_, 220:55–68, 2019.
  * Bishop [2006] Christopher M. Bishop.  _Pattern Recognition and Machine Learning_.  2006\.  ISBN 9780387310732.  URL http://www.springer.com/us/book/9780387310732.
  * LeCun et al. [2015] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton.  Deep learning.  _Nature_ , 521(7553):436–444, 5 2015.  ISSN 0028-0836.  doi:10.1038/nature14539.  URL http://www.nature.com/articles/nature14539.
  * Goodfellow et al. [2016] Ian Goodfellow, Yoshua Bengio, and Aaron Courville.  _Deep Learning_.  2016\.  ISBN 9781405161251.  URL https://www.deeplearningbook.org.
  * Rafiei and Adeli [2017] Mohammad Hossein Rafiei and Hojjat Adeli.  A novel machine learning-based algorithm to detect damage in high-rise building structures.  _Structural Design of Tall and Special Buildings_ , 26(18):1–11, 2017.  ISSN 15417808.  doi:10.1002/tal.1400.
  * Rouet-Leduc et al. [2017] Bertrand Rouet-Leduc, Claudia Hulbert, Nicholas Lubbers, Kipton Barros, Colin J. Humphreys, and Paul A. Johnson.  Machine Learning Predicts Laboratory Earthquakes.  _Geophysical Research Letters_ , 44(18):9276–9282, 2017.  ISSN 19448007.  doi:10.1002/2017GL074677.
  * Perol et al. [2018] Thibaut Perol, Michaël Gharbi, and Marine Denolle.  Convolutional neural network for earthquake detection and location.  _Science Advances_ , 4(2):e1700578, 2 2018.  ISSN 2375-2548.  doi:10.1126/sciadv.1700578.  URL https://advances.sciencemag.org/lookup/doi/10.1126/sciadv.1700578.
  * Ross et al. [2019] Zachary E. Ross, Daniel T. Trugman, Egill Hauksson, and Peter M. Shearer.  Searching for hidden earthquakes in Southern California.  _Science_ , 771(May):767–771, 2019.  ISSN 10959203.  doi:10.1126/science.aaw6888.
  * Pilania et al. [2013] Ghanshyam Pilania, Chenchen Wang, Xun Jiang, Sanguthevar Rajasekaran, and Ramamurthy Ramprasad.  Accelerating materials property predictions using machine learning.  _Scientific Reports_ , 3:1–6, 2013.  ISSN 20452322.  doi:10.1038/srep02810.
  * Liang et al. [2018] Liang Liang, Minliang Liu, Caitlin Martin, and Wei Sun.  A deep learning approach to estimate stress distribution: a fast and accurate surrogate of finite-element analysis.  _Journal of the Royal Society Interface_ , 15(138), 2018\.  ISSN 17425662.  doi:10.1098/rsif.2017.0844.
  * DeVries et al. [2018] Phoebe M.R. DeVries, Fernanda Viégas, Martin Wattenberg, and Brendan J. Meade.  Deep learning of aftershock patterns following large earthquakes.  _Nature_ , 560(7720):632–634, 2018.  ISSN 14764687.  doi:10.1038/s41586-018-0438-y.  URL http://dx.doi.org/10.1038/s41586-018-0438-y.
  * Wang and Sun [2018] Kun Wang and Wai Ching Sun.  A multiscale multi-permeability poroplasticity model linked by recursive homogenizations and deep learning.  _Computer Methods in Applied Mechanics and Engineering_ , 334:337–380, 2018.  ISSN 00457825.  doi:10.1016/j.cma.2018.01.036.
  * Kabacaoğlu and Biros [2019] Gökberk Kabacaoğlu and George Biros.  Machine learning acceleration of simulations of Stokesian suspensions.  _Physical Review E_ , 99(6):063313, 6 2019.  ISSN 2470-0045.  doi:10.1103/PhysRevE.99.063313.  URL https://link.aps.org/doi/10.1103/PhysRevE.99.063313.
  * Bergen et al. [2019] Karianne J.; Bergen, Paul A.; Johnson, Maarten V.; de Hoop, and Gregory C. Beroza.  Machine learning for data-driven discovery in solid Earth geoscience.  _Science_ , 363(6433):eaau0323, 3 2019.  ISSN 0036-8075.  doi:10.1126/science.aau0323.  URL http://www.sciencemag.org/lookup/doi/10.1126/science.aau0323.
  * Dana and Wheeler [2020] Saumik Dana and Mary F Wheeler.  A machine learning accelerated FE$^2$ homogenization algorithm for elastic solids.  2020\.  URL http://arxiv.org/abs/2003.11372.
  * Wagner and Rondinelli [2016] Nicholas Wagner and James M. Rondinelli.  Theory-guided machine learning in materials science.  _Frontiers in Materials_ , 3(June):1–9, 2016\.  ISSN 22968016.  doi:10.3389/fmats.2016.00028.
  * Rudy et al. [2018] Samuel Rudy, Alessandro Alla, Steven L. Brunton, and J. Nathan Kutz.  Data-driven identification of parametric partial differential equations.  18(2):643–660, 2018.  URL http://arxiv.org/abs/1806.00732.
  * Weinan and Yu [2018] E. Weinan and Bing Yu.  The Deep Ritz Method: A Deep Learning-Based Numerical Algorithm for Solving Variational Problems.  _Communications in Mathematics and Statistics_ , 6(1):1–14, 2018.  ISSN 2194671X.  doi:10.1007/s40304-018-0127-z.
  * Raissi et al. [2019] M. Raissi, P. Perdikaris, and G. E. Karniadakis.  Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations.  _Journal of Computational Physics_ , 378:686–707, 2019\.  ISSN 10902716.  doi:10.1016/j.jcp.2018.10.045.  URL https://doi.org/10.1016/j.jcp.2018.10.045.
  * Kharazmi et al. [2019] E. Kharazmi, Z. Zhang, and G. E. Karniadakis.  Variational Physics-Informed Neural Networks For Solving Partial Differential Equations.  pages 1–24, 2019.  URL http://arxiv.org/abs/1912.00873.
  * Fang and Zhan [2019] Zhiwei Fang and Justin Zhan.  A Physics-Informed Neural Network Framework For Partial Differential Equations on 3D Surfaces : Time Independent Problems.  _IEEE Access_ , PP:1, 2019.  doi:10.1109/ACCESS.2019.2963390.
  * Haghighat et al. [2020a] Ehsan Haghighat, Maziar Raissi, Adrian Moure, Hector Gomez, and Ruben Juanes.  A deep learning framework for solution and discovery in solid mechanics.  2020a.
  * Mao et al. [2020] Zhiping Mao, Ameya D. Jagtap, and George Em Karniadakis.  Physics-informed neural networks for high-speed flows.  _Computer Methods in Applied Mechanics and Engineering_ , 360:112789, 2020.  ISSN 00457825.  doi:10.1016/j.cma.2019.112789.  URL https://doi.org/10.1016/j.cma.2019.112789.
  * Fuks and Tchelepi [2020] O Fuks and H. A. Tchelepi.  Physics Based Deep Learning for Nonlinear Two-Phase Flow in Porous Media.  In _ECMOR XVII_ , number Sep, pages 1–10. European Association of Geoscientists & Engineers, 2020.  doi:10.3997/2214-4609.  URL https://www.earthdoc.org/content/papers/10.3997/2214-4609.202035147.
  * Sun et al. [2020] Luning Sun, Han Gao, Shaowu Pan, and Jian Xun Wang.  Surrogate modeling for fluid flows based on physics-constrained deep learning without simulation data.  _Computer Methods in Applied Mechanics and Engineering_ , 361:112732, 2020.  ISSN 00457825.  doi:10.1016/j.cma.2019.112732.  URL https://doi.org/10.1016/j.cma.2019.112732.
  * Tartakovsky et al. [2020] A. M. Tartakovsky, C. Ortiz Marrero, Paris Perdikaris, G. D. Tartakovsky, and D. Barajas-Solano.  Physics-Informed Deep Neural Networks for Learning Parameters and Constitutive Relationships in Subsurface Flow Problems.  _Water Resources Research_ , 56(5):1–16, 2020\.  ISSN 19447973.  doi:10.1029/2019WR026731.
  * Meng et al. [2019] Xuhui Meng, Zhen Li, Dongkun Zhang, and George Em Karniadakis.  PPINN: Parareal Physics-Informed Neural Network for time-dependent PDEs.  pages 1–17, 2019.  URL http://arxiv.org/abs/1909.10145.
  * Haghighat et al. [2020b] Ehsan Haghighat, Ali Can, Erdogan Madenci, and Ruben Juanes.  A nonlocal physics-informed deep learning framework using the peridynamic differential operator.  (May), 2020b.
  * Kharazmi et al. [2020] Ehsan Kharazmi, Zhongqiang Zhang, and George Em Karniadakis.  hp-VPINNs: Variational Physics-Informed Neural Networks With Domain Decomposition.  pages 1–21, 2020.  URL http://arxiv.org/abs/2003.05385.
  * Jagtap et al. [2020] Ameya D. Jagtap, Ehsan Kharazmi, and George Em Karniadakis.  Conservative physics-informed neural networks on discrete domains for conservation laws: Applications to forward and inverse problems.  _Computer Methods in Applied Mechanics and Engineering_ , 365:113028, 2020.  ISSN 00457825.  doi:10.1016/j.cma.2020.113028.  URL https://doi.org/10.1016/j.cma.2020.113028.
  * Hornik et al. [1989] Kurt Hornik, Maxwell Stinchcombe, and Halbert White.  Multilayer feedforward networks are universal approximators.  _Neural Networks_ , 2(5):359–366, 1989.  ISSN 08936080.  doi:10.1016/0893-6080(89)90020-8.
  * Güne¸ et al. [2018] Atılım Güne¸, Güne¸s Baydin, Barak A Pearlmutter, and Jeffrey Mark Siskind.  Automatic Differentiation in Machine Learning: a Survey.  _Journal of Machine Learning Research_ , 18:1–43, 2018\.  URL http://www.jmlr.org/papers/volume18/17-468/17-468.pdf.
  * Nocedal and Wright [2006] Jorge Nocedal and Stephen J. Wright.  _Numerical Optimization_ , volume 17 of _Springer Series in Operations Research and Financial Engineering_.  Springer New York, 2006.  ISBN 978-0-387-30303-1.  doi:10.1007/978-0-387-40065-5.  URL http://link.springer.com/10.1007/978-0-387-40065-5.
  * Kingma and Ba [2015] Diederik P. Kingma and Jimmy Lei Ba.  Adam: A method for stochastic optimization.  _3rd International Conference on Learning Representations, ICLR 2015 - Conference Track Proceedings_ , pages 1–15, 2015.
  * Sun [2019] Ruoyu Sun.  Optimization for deep learning: theory and algorithms.  _arXiv_ , pages 1–60, 2019.
  * Kim and De Weck [2005] I. Y. Kim and O. L. De Weck.  Adaptive weighted-sum method for bi-objective optimization: Pareto front generation.  _Structural and Multidisciplinary Optimization_ , 29(2):149–158, 2005.  ISSN 1615147X.  doi:10.1007/s00158-004-0465-1.
  * Wang et al. [2020] Sifan Wang, Yujun Teng, and Paris Perdikaris.  Understanding and mitigating gradient pathologies in physics-informed neural networks.  pages 1–28, 2020.  URL http://arxiv.org/abs/2001.04536.
  * Haghighat and Juanes [2019] Ehsan Haghighat and Ruben Juanes.  SciANN: A Keras wrapper for scientific computations and physics-informed deep learning using artificial neural networks.  2019\.