# Adversarial Multi-task Learning Enhanced Physics-informed Neural Networks
for Solving Partial Differential Equations

Pongpisit Thanasutives2, Masayuki Numao3, Ken-ichi Fukui3  2Graduate School of
Information Science and Technology, Osaka University, Japan
3The Institute of Scientific and Industrial Research, Osaka University, Japan
{thanasutives<EMAIL_ADDRESS>
###### Abstract

Recently, researchers have utilized neural networks to accurately solve
partial differential equations (PDEs), enabling the mesh-free method for
scientific computation. Unfortunately, the network performance drops when
encountering a high nonlinearity domain. To improve the generalizability, we
introduce the novel approach of employing multi-task learning techniques, the
uncertainty-weighting loss and the gradients surgery, in the context of
learning PDE solutions. The multi-task scheme exploits the benefits of
learning shared representations, controlled by cross-stitch modules, between
multiple related PDEs, which are obtainable by varying the PDE
parameterization coefficients, to generalize better on the original PDE.
Encouraging the network pay closer attention to the high nonlinearity domain
regions that are more challenging to learn, we also propose adversarial
training for generating supplementary high-loss samples, similarly distributed
to the original training distribution. In the experiments, our proposed
methods are found to be effective and reduce the error on the unseen data
points as compared to the previous approaches in various PDE examples,
including high-dimensional stochastic PDEs.

©2021 IEEE. Personal use of this material is permitted. Permission from IEEE
must be obtained for all other uses, in any current or future media, including
reprinting/republishing this material for advertising or promotional purposes,
creating new collective works, for resale or redistribution to servers or
lists, or reuse of any copyrighted component of this work in other works.

##  I Introduction

Deep learning has tremendous successes in several subfields of artificial
intelligence and scientific computing including computer vision, natural
language processing, robotics; moreover, deep learning for physics simulation
[1] is an attractive field. As a result of advances in automatic
differentiation [2] for estimating derivatives with respect to any input
dimension, lately, neural networks have been extensively studied as an
alternative way to numerically solve partial differential equations (PDEs)
without reliance on spatial-temporal grids. Solving PDEs using neural networks
plays an essential role in the development of a hybrid system between physics
and machine learning, owing to the physically consistent network outputs as
the PDE solutions.

An influential work by Raissi el al. [3], physics informed neural networks
(PINNs), carefully designs the solver network loss to be constrained by the
underlying PDE and the boundary conditions. Also in [4], Raissi has cracked
the way to solve coupled forward-backward stochastic differential equations
(FBSDEs) based on minimizing the loss of a network structure called FBSNN,
which is constructed from Euler-Maruyama discretization. However, in practice,
the generalizability problem of deep learning is hard to avoid since the
network could not always be trained well in high-dimensional non-convex
optimization with limited training data, the sampling distribution might not
be effective enough [5] and the overfitting problem, where the network
performs worse in spite of the greater performance in the training set. This
motivates us to let multi-task learning and adversarial training come to the
rescue. To the best of our knowledge, we are the first to leverage this
combination of learning techniques in the context of solving PDEs.

The construction of the multi-task setting is natural as most PDEs are
parameterized by certain coefficients. We propose that the varied coefficient
can be viewed as an auxiliary task that shares additional insights to the
original task. Multi-task learning could be implemented in both network
architecture and optimization process. We presume that the following
conditions hold true for our method to be effective: (1) The neural network is
capable of exploiting the knowledge of discrepancy, which is induced by
equation’s coefficient modification, between multiple related PDEs to achieve
the generalized performance. (2) Minimizing the designed physics-informed loss
function converges the predicted solution to the true PDE solution. As long as
(2), which is theoretically supported by the Theorem 1. in [6], holds, we
could apply the novel multi-task learning strategies without supervision.

In this paper, the cross-stitch module [7] is selected as the component for
learning how to share the information between tasks. The module works well in
the case of 2-task learning, in which we are interested. To optimize the
multi-task architecture, we experiment with (1) the uncertainty-weighting loss
[8] since the strategy needs no extra hyperparameter tuning and potentially
excel at training with low-noise or noiseless tasks such as solving for the
solutions that fit the PDE constraints. Alternatively, we also try (2) the
gradients surgery algorithm (PCGrad) [9] to the unweighted losses to enhance
the conformity of gradients magnitude and direction between tasks. We provide
a qualitative comparison of these strategies for each PDE problem in Section
IV.

Moreover, the sampling distribution for training data generation is also of
concern. Particularly, in the case of solving forward PDE where the sampling
process is performed to construct data points in the spatial-temporal grid
before training. Previous work [5] has employed naive uniform sampling and
inspected performance degradation on the less emphasized regions. Similar
observations are found in [6] that the original PINNs perform suboptimally on
the high-frequency domain regions where greater attention is needed in order
for the network to fit well. As visualized in Fig. 1(A and B), we tackle the
problem by including more difficult samples (red points) through our proposed
adversarial training while maintaining the characteristics of the training
distribution. This is valid since the computational complexity does not blow
up rapidly as the number of training points increases [10].

Inspired by the success of generative adversarial networks (GAN) [11], the
difficult samples are defined so that they maximize the loss of the solver
network, accounting for the lack of samples in high-loss regions. We also
control the generated samples to be similarly distributed to the training
distribution. This helps prevent the solver network from being laser-focused
or overfitting a particular domain. In our forward PDE experiments (See
Section III-A), we apply adversarial training to the best variation of multi-
task learning optimization techniques to obtain more superior results, which
are also reported in Section IV.

##  II Related Works

###  II-A Neural networks for solving PDEs

Early researchers studied the possibility of employing a universal
approximator [12] based on a neural network to represent PDE solutions back in
[13, 14], resulting in an ability to obtain approximated solutions fast.
However, those methods rely on the supervision of referenced solutions, which
are achievable analytically or numerically. This is either not practical or
limiting the performance of the network since the exact solutions are not
always available and numerical approximations do not precisely represent the
solutions.

Figure 1: In the Burgers’ equation example, we use uniform distribution to
sample training data points and calculate the corresponding losses. (A) and
(B) are examples of the transformed samples (from different iterations) in the
process of our proposed adversarial training. The color bar indicates the
trained PINN loss magnitude. The transformed samples overlap the regions,
where more samples are required, and inherit the uniform patterns.

Nowadays, PDE solution’s derivatives with respect to any independent variables
could be easily estimated using automatic differentiation on a computational
graph, emerging the rise of neural network applications for solving several
kinds of PDE such as [4, 15, 16, 3, 10] to name but a few. Raissi et al. [3]
approximated the first and higher-order derivatives using automatic
differentiation, whilst Sirignano et al. [16] leveraged the Monte Carlo method
for higher-order derivatives approximation, proposing a so-called Deep
Galerkin Method (DGM). Both of them layout the similar fundamentals of
considering PDE constraints along with the initial and boundary conditions as
the neural network loss function.

Due to the power of the neural network for overcoming the “curse of
dimensionality” in practice, more previous works, FBSNN [4] and Deep BSDE
[15], extended the framework to solve a large class of high-dimensional
nonlinear PDEs by taking a neural network to estimate intractable mathematical
expressions from each step of the multi-step prediction process, which was
constructed by Euler–Maruyama discretization. Nonetheless, as mentioned,
drawbacks of neural networks as PDE solvers were spotted in the past
literature.

Van der Meer et al. [6] found that typical single-objective optimization
without scaling factors for balancing between the losses on PDE constraints
and boundary conditions could deteriorate the overall performance of the
network; therefore, a heuristic method to approximate the optimal scaling
parameter, with respect to the relative error, was proposed.

Al-Aradi et al. [5] pointed out that the data sampling method could be one
possible source of the model’s generalization error. Specifically for the
high-dimensional PDEs, Güler et al. [17] tackled the generalization problem
and improved training stability via the network architecture modification
(e.g. more residual connections). Differently from [17], we focus more on how
multi-task and adversarial training, which are substantially developed in the
context of data-driven learning, could be used to alleviate the generalization
problem and enable an effective training strategy for solving PDEs.

Figure 2: 1D Laplace equation example: We approximate the solution using a
neural network, then use automatic differentiation on $\hat{u}(x;\theta)$ with
respect to the input, $x$, twice to calculate the second-order derivative.
$N_{b_{j}}$ and $N_{b_{k}}$ denotes the number of samples for learning the
condition $u(0)=0$ and $u(1)=2$.

###  II-B Multi-task learning

Multi-task learning (MTL) tackles multiple tasks concurrently through a
learned shared representation. MTL can be viewed from a network architecture
perspective and the optimization procedure for joint learning a set of task-
specific losses. From the architecture point-of-view, MTL is usually
categorized into hard or soft parameter sharing. In hard parameter sharing
(e.g. UberNet [18]), the parameter set is explicitly divided into shared and
task-specific layers, leading to a question of which layers the common
information should be learned. Thus, soft parameter sharing, such as Cross-
stitch [7] and Sluice networks [19], is proposed to adaptively transfer useful
information between task-specific networks. However, the scalability to the
higher dimension (the number of tasks) of soft parameter sharing is
restricted.

More challenges of MTL stem from the optimization process, which balances the
impact of each task learning signal. One could deal with the problem by either
changing the balancing weights through time or adjusting the gradients of each
task-specific loss. Without introducing more hyperparameters, Kendall et al.
[8] weighted multiple losses by considering the homoscedastic uncertainty (a
kind of aleatoric uncertainty) of each task. The weighted loss was derived
through maximum likelihood estimation, which was parameterized by the Gaussian
noise of each target variable. In our scope, the uncertainty weighting is more
preferred than the other methods (e.g. GradNorm [20] and DWA [21]), which
either require manual gradient magnitudes balancing or designed and mostly
tested with hard parameter sharing architecture. Differently from the
uncertainty weighting, Yu et al. [9] proposed a so-called PCGrad algorithm to
modify the gradients of each task-specific loss so that they were aligned in
both direction and magnitude. We choose to compare the effectiveness of
uncertainty weighting and PCGrad in terms of the error on unseen data points
since both algorithms can be seamlessly applied to existing frameworks of
solving PDEs without the need for tuning additional hyperparameters.

###  II-C Adversarial machine learning

A fascinating property of neural networks, including the state-of-the-art
ones, is that they consistently misunderstood adversarial examples, which are
easily constructed by adding immeasurably small perturbation on the original
inputs (See [22, 23] for more details). To prevent this vulnerability, [22,
23] suggests that training with crafted adversarial examples helps regularize
the model and reduce the error rate on test sets and the associated
adversarial examples, inspiring us to incorporate the transformed samples (See
Fig. 1) to the original training data points.

The adversarial examples do not always have to be crafted before training. In
a generative adversarial network (GAN), adversarial samples are automatically
generated by a generator, which aims to maximize the discriminator loss. In
this study, we exploit the GAN heuristics to produce difficult-to-predict
samples, which are adaptive through iterations (See Fig. 1(A and B)), for
abolishing the weaknesses of the solver network around the relatively high-
loss domain regions.

Regarding the inverse problem, in which the aim is to recover the network
representation of a PDE from experimental observations, Yang et al. [24] has
put forth adversarial inference for joint distribution matching between
generated and observed data, obtaining posterior quantification of the
uncertainty associated with the predicted solutions. Thus, the purpose of the
generated data and the generator loss are clearly distinct from our proposals.

##  III Method

###  III-A Solving single forward partial differential equation

We consider the general form of PDEs as follows

$\boldsymbol{\mathcal{N}}[u(t,x);\lambda]=0,x\in\mathbb{R}^{D},t\in\mathbb{R}$
(1)

where $u(t,x)$ denotes the latent solution and
$\boldsymbol{\mathcal{N}}[u(t,x);\lambda]$ is the underlying partial
differential equation which is parameterized by $\lambda$. The typical
physics-informed loss function [3] for solving PDE includes both the
underlying equation and a boundary condition, given by

$\begin{multlined}\mathcal{L}_{eq}=\frac{1}{N_{f}}\sum^{N_{f}}_{i=1}\mathinner{\\!\left\lvert\boldsymbol{\mathcal{N}}[\hat{u}(t^{i}_{f},x^{i}_{f};\theta);\lambda]\right\rvert}^{2}+\\\
\frac{1}{N_{b}}\sum^{N_{b}}_{j=1}\mathinner{\\!\left\lvert\hat{u}(t^{j}_{b},x^{j}_{b};\theta)-u(t^{j}_{b},x^{j}_{b})\right\rvert}^{2}\end{multlined}\mathcal{L}_{eq}=\frac{1}{N_{f}}\sum^{N_{f}}_{i=1}\mathinner{\\!\left\lvert\boldsymbol{\mathcal{N}}[\hat{u}(t^{i}_{f},x^{i}_{f};\theta);\lambda]\right\rvert}^{2}+\\\
\frac{1}{N_{b}}\sum^{N_{b}}_{j=1}\mathinner{\\!\left\lvert\hat{u}(t^{j}_{b},x^{j}_{b};\theta)-u(t^{j}_{b},x^{j}_{b})\right\rvert}^{2}$
(2)

where $\hat{u}(t_{f},x_{f};\theta)$ and $\hat{u}(t_{b},x_{b};\theta)$ together
denote the predictions of a PDE solver network, parameterized by $\theta$, for
the entire input space. $N_{f}$ denotes the number of interior collocation
points. $N_{b}$ denotes the data points for learning the initial and boundary
condition. An elementary PINN loss formulation example of the 1D Laplace is
provided in Fig. 2.

###  III-B Auxiliary task generation

By setting $\lambda^{aux}=\alpha\lambda$, we are able to acquire an auxiliary
task to be learned jointly. In practice, a good choice of $\alpha$ is
achievable via either arbitrary assignment by human or Bayesian optimization
[25] to assuredly enforce the similar solution behavior by minimizing the
solver network loss, which we shall discuss shortly. After acquiring
$\lambda^{aux}$ for each generated auxiliary task, we turn our focus to how
the solver network can learn from multiple physics-informed objectives based
on two effective multi-task learning strategies, uncertainty-weighted loss and
gradient surgery, namely PCGrad update rule. In order to apply a supervised
weighting scheme, we proceed by the assumption that there exists a close
neural network approximation $\hat{u}({t}_{f},{x}_{f};\theta)$ at the desired
accuracy or equivalently
$\mathinner{\\!\left\lVert\boldsymbol{\mathcal{N}}[\hat{u}({t}_{f},{x}_{f};\theta);\lambda]\right\rVert}<\epsilon$.
According to the Theorem 1. in [6], minimizing Eq. (2), the neural network
predictions are close to the true PDE solutions to certain extent. More
specifically, for the interior collocation points, we presumably set

$\displaystyle\theta^{*}$
$\displaystyle=\mathop{\mathrm{argmin}}_{\theta}\sum^{N_{f}}_{i=1}\mathinner{\\!\left\lvert\hat{u}(t^{i}_{f},x^{i}_{f};\theta)-u(t^{i}_{f},x^{i}_{f})\right\rvert}^{2}$
(3)
$\displaystyle=\mathop{\mathrm{argmin}}_{\theta}\sum^{N_{f}}_{i=1}\mathinner{\\!\left\lvert\boldsymbol{\mathcal{N}}[\hat{u}(t^{i}_{f},x^{i}_{f};\theta);\lambda]\right\rvert}^{2}.$

###  III-C Uncertainty-weighted physics-informed loss function

As a result of Eq. (3), we replace the squared losses in [8] with the typical
losses for solving PDEs. We define our uncertainty-weighted physics-informed
loss function (Uncert) as follows

$\mathcal{L}_{uncert}=\sum^{N_{T}}_{i=1}(\frac{1}{2\sigma^{2}_{i}}\mathcal{L}^{i}_{eq}+\log\sigma_{i})\qquad[Option\>I]$
(4)

where $N_{T}$ refers to the number of tasks. Here, $\sigma_{i}$ is a gradient-
based trainable (e.g. by utilizing $\nabla_{\sigma_{i}}\mathcal{L}_{uncert}$)
parameter which indicates each PDE solution uncertainty. The uncertainty
originates under the assumption that, for each PDE
$p(u^{i}(t,x)|\hat{u}^{i}(t,x;\theta))=\mathcal{N}(\hat{u}^{i}(t,x;\theta),\sigma^{2}_{i})$
which fairly encapsulates both the noisy and noise-free cases. Then
$\sigma_{i}$, the scaling factor, is derived to maximize the multi-task
Gaussian likelihood, which could be seen as minimizing Bayesian information
criterion (BIC) so that the solver network yields the optimal performance for
a given solver network parameters $\theta$, or model complexity.

###  III-D PCGrad: Project conflicting gradients

Nevertheless, there are alternative approach for learning from multiple
objectives. We also apply the backpropagation algorithm with PCGrad updates
(Algorithm 1 in [9]) to calculate the modified gradients for the unweighted
equation losses. Consequently, we update the model parameters with

$\begin{gathered}\delta_{PC}\theta=\sum^{N_{T}}_{i=1}\textbf{g}^{i}_{PC}\qquad[Option\>II]\\\
\\{\textbf{g}^{i}_{PC}\\}=PCGrad(\\{\nabla_{\theta}\mathcal{L}^{i}_{eq}\\})\end{gathered}$
(5)

where $\textbf{g}^{i}_{PC}$ refers to the resulting modified gradients by the
PCGrad update for the task $i$. Then, we use the $\delta_{PC}\theta$ to update
the solver network parameters, $\theta$. For a couple of tasks, PCGrad
projects task $i$’s gradient onto the normal vector of task $j$’s gradient,
and vice versa, to deconflict the gradient directions during training.

Figure 3: Overview of the cross-stitch network architecture for the
adversarial multi-task training (Algorithm 1).

###  III-E Multi-task learning architecture

From the multi-task learning architecture point of view, we leverage two
parallel baseline networks with cross-stitch modules to share the activations
among all single-task networks. The overall architecture is depicted in Fig.
3. Assume that we are considering two activation maps
($act_{Task1},act_{Task2}$) at a particular layer, which belong to Task1 and
Task2 respectively. We note that Task1 is the target PDE and Task2 is the
generated auxiliary PDE, A trainable linear transformation of these activation
maps is applied, before feeding into the successive layer for each PDE. The
transformation can be formalized as

$\left[\begin{array}[]{c}\tilde{act}_{Task1}\\\
\tilde{act}_{Task2}\end{array}\right]=\begin{bmatrix}\gamma_{11}&\gamma_{12}\\\
\gamma_{21}&\gamma_{22}\end{bmatrix}\times\left[\begin{array}[]{c}act_{Task1}\\\
act_{Task2}\end{array}\right]$ (6)

where every $\gamma$ is gradient-based learnable to linearly control how much
information to share from Task1 to Task2 and vice versa. With $\gamma$, cross-
stitch modules adaptively retain the low-layer information, for example,
$\gamma_{11}$ determines how much $act_{Task1}$ influence the higher-layer
activations of Task1.

###  III-F Adversarial multi-task training

Additionally including the high-loss samples lets the solver network focus
more on the domain regions that are more challenging to regress for the
solutions, for example, the dynamically changing regions of Fokker-Planck
solutions shown in Fig. 6. The generator periodically (every $F$ iterations,
See Algorithm 1.) produces the additional high-loss samples by minimizing the
loss function defined as follows

$\begin{gathered}\mathcal{L}_{gen}=\frac{1}{N_{f}}\mathinner{\\!\left\lVert
scale(h(t_{f},x_{f};\theta_{g}),lb,ub)-(t_{f},x_{f})\right\rVert}_{2}^{2}-\sum^{N_{T}}_{j=1}\mathcal{L}^{j}_{eq}\\\
scale(a,lb,ub)=(a-lb)\oslash(ub-lb)\\\
\\{(t^{k},x^{k})\\}\mathrel{\stackrel{{\scriptstyle\makebox[0.0pt]{\mbox{\tiny
U}}}}{{\sim}}}\\{scale(h(t_{f}^{i},x_{f}^{i};\theta_{g}),lb,ub)\\},|\\{(t^{k},x^{k})\\}|=pN_{f}\\\
\\{(t^{i},x^{i})\\}\mathrel{\mathop{\mathchar
58\relax}}=concat(\\{(t^{i},x^{i})\\},\\{(t^{k},x^{k})\\})\end{gathered}$ (7)

where $h(t_{f},x_{f};\theta_{g})$ denotes the transformed samples from the
$N_{f}$ interior collocation points, $(t_{f},x_{f})$. The averaged squared
$l_{2}$-norm of $\mathcal{L}_{gen}$ helps the transformed samples to maintain
the characteristics of the training distribution. The second term,
$-\sum^{N_{T}}_{i=1}\mathcal{L}^{i}_{eq}$, is not averaged, and therefore
dominate the overall loss magnitude for inducing the transformed samples to be
from the regions that are difficult to regress. We scale the generator’s
outputs to the PDE domain (bounded from below and above by the vector $lb$ and
$ub$) and then uniformly pick a portion, $p$, of the scaled values to
reconstruct $\\{(t^{i},x^{i})\\}$ as the new training set with total
$(1+p)N_{f}+N_{b}$ samples. The $\oslash$ and
$\mathrel{\stackrel{{\scriptstyle\makebox[0.0pt]{\mbox{\tiny U}}}}{{\sim}}}$
notation refer to Hadamard division and uniformly sampling elements from a
set. The pseudocode for our adversarial multi-task training is described in
Algorithm 1.

Require: Adversarial training frequency $F$, limit $L$ and iterations $E$,
Sampling proportion $p$ and Generator parameters $\theta_{g}$

for $iter=0$ to $epochs-1$ do

if $(iter\>mod\>F)=0$ and $iter\leq\lfloor\frac{epochs}{L}\rfloor$ then

for $e$ = $0$ to $E-1$ do

Freeze the solver network parameters $\theta$

Generator’s forward pass to obtain $scale(h(t_{f},x_{f};\theta_{g}),lb,ub)$

Reconstruct $\forall{i}$ $(t^{i},x^{i})$ as the solver’s input

Solver’s forward pass to obtain $-\sum^{N_{T}}_{j=1}\mathcal{L}^{j}_{eq}$

Calculate the adversarial loss $\mathcal{L}_{gen}$

Backpropagate $\nabla_{\theta_{g}}\mathcal{L}_{gen}$ and update $\theta_{g}$

end for

else

Freeze the generator parameters $\theta_{g}$

Set the latest $\forall{i}$ $(t^{i},x^{i})$ as the solver’s input

Solver’s forward pass to obtain $\forall{j}$ $\mathcal{L}^{j}_{eq}$

Backpropagate $\nabla_{\theta}\mathcal{L}_{uncert}$ or $\delta_{PC}\theta$
(the options described in Section III-A) and update $\theta$

end if

end for

Return: $\theta^{*}$ and $\theta^{*}_{g}$

Algorithm 1 Adversarial multi-task training

##  IV Experiments and Results

In this section, we investigate the powerfulness of our methods on a couple of
PDE settings, forward partial differential equations and high-dimensional
forward-backward stochastic partial differential equations (FBSDE). For each
experimental equation of the first setting, trained using the original PINN’s
loss, 5 fully connected layers with 50 hidden units and tanh activation are
considered as the baseline while 4 fully connected layers with 256 hidden
units and tanh activation (FBSNN architecture) are selected for experiments of
the second case. The sample generator is a single tanh activated hidden layer
with 32 units, which exhibits enough capability for obtaining the difficult
samples. We always consider an equal number of training and testing samples
when comparing our MTL modifications to the typical approach for each PDE.

The performance is quantified using mean absolute error (MAE), mean squared
error (MSE) and relative $l_{2}$ error. These metrics are defined as follows

$\begin{gathered}MAE=\operatorname{\mathbb{E}}[|\hat{u}-u|],\,MSE=\operatorname{\mathbb{E}}[(\hat{u}-u)^{2}]\\\
Relative\>l_{2}\>error=\frac{\mathinner{\\!\left\lVert\hat{u}-u\right\rVert}_{2}}{\mathinner{\\!\left\lVert
u\right\rVert}_{2}}\end{gathered}$ (8)

where $\hat{u}$ is the approximated solution (i.e. the PINN solution), $u$ is
the exact solution, $\mathinner{\\!\left\lVert\cdot\right\rVert}_{2}$ is the
$l_{2}$-norm and $\operatorname{\mathbb{E}}$ denotes the expectation over test
samples.

###  IV-A Forward partial differential equation

####  IV-A1 Burgers’ equation

This equation arises in various areas of applied mathematics, including fluid
mechanics, and traffic flow [26]. The equation is notoriously hard to solve
using traditional numerical methods. We consider the following Burgers’
equation with Dirichlet boundary conditions, as being studied in [3].

$\begin{gathered}u_{t}+uu_{x}-(0.01/\pi)u_{xx}=0,\,x\in[-1,1],\,t\in[0,1]\\\
u(0,x)=-\sin(\pi x),\,u(t,-1)=u(t,1)=0\end{gathered}$ (9)

Following the training data creation procedure in [3], a Latin Hypercube
Strategy (LHS) [27] is chosen for sampling 10,000 interior data points
($N_{f}$) and 100 points of the the initial and boundary data ($N_{u}$). The
exact solutions for testing are made available by [26]. We set
$\lambda=0.01/\pi$ and consider a nearby coefficient of the form
$\lambda^{aux}=\alpha\lambda$. Then, the Bayesian optimization, with
$\mathcal{L}_{uncert}$ in Eq. (4) as the objective criteria, is applied for
searching the best-tuned $\alpha^{*}$ under the $(0,1)$ range (for the sake of
bounded computational time), obtaining $\alpha^{*}\approx 0.6$. In the other
experiments, we have found that random assignments with minimal tuning are
sufficient for training the multi-task networks to outperform the existing
approaches (See Section IV-C for the effect of $\alpha$). Our neural networks
are optimized by full-batch Adam [28] with 0.005 learning rate for 50,000
epochs. For the adversarial setting, we assign $F=100$, $L=2$, $E=10$ and
$p=0.1$.

The performance comparison amongst variations of our method, PINN [3] and
ResNet [17] is listed in Table I. Both the multi-task modifications improve
the performance by reducing the MAE, MSE and relative $l_{2}$ error. We select
the best performing strategy, which in this PDE, is the uncertainty-weighting
scheme, to undergo our adversarial training, and consequently, the MSE reaches
the order of $10^{-6}$. The adversarial training is found to successfully
diminish the losses around the domain regions with high nonlinearity. With the
aleatoric uncertainty quantification, Uncert carries out the similar
performance against the noisy initial and boundary solutions, signifying our
method robustness. Our ablation study of the cross-stitch units, comparing the
4 and 5 row in Table I on both the noise-free and noisy cases, shows that the
modules are keys for enabling knowledge share between the neural networks,
which reduces the overfitting as shown in Fig. 4.

TABLE I: Burgers’equation: Performance comparison

Method | MAE | MSE | Rel. $l_{2}$ error
---|---|---|---
PINN [3] |  | $2.3\times 10^{-3}$
---
$6.7\times 10^{-3}$
| $2.1\times 10^{-4}$
---
$2.1\times 10^{-3}$
| $2.4\times 10^{-2}$
---
$7.4\times 10^{-2}$
ResNet [17] |  | $1.8\times 10^{-3}$
---
$2.8\times 10^{-3}$
| $5.0\times 10^{-5}$
---
$1.7\times 10^{-4}$
| $1.1\times 10^{-2}$
---
$2.1\times 10^{-2}$
PCGrad w/ CS* |  | $2.3\times 10^{-3}$
---
$7.1\times 10^{-3}$
| $2.7\times 10^{-5}$
---
$9.3\times 10^{-5}$
| $8.4\times 10^{-3}$
---
$5.7\times 10^{-2}$
Uncert w/o CS* |  | $6.4\times 10^{-3}$
---
$3.0\times 10^{-3}$
| $2.2\times 10^{-4}$
---
$2.1\times 10^{-4}$
| $2.5\times 10^{-2}$
---
$2.3\times 10^{-2}$
Uncert w/ CS* |  | $1.1\times 10^{-3}$
---
$1.8\times 10^{-3}$
| $2.1\times 10^{-5}$
---
$1.5\times 10^{-5}$
| $7.4\times 10^{-3}$
---
$6.3\times 10^{-3}$
Uncert w/ CS* + Adv.** |  | $\mathbf{4.1\times 10^{-4}}$
---
$\mathbf{1.5\times 10^{-3}}$
| $\mathbf{1.0\times 10^{-6}}$
---
$\mathbf{1.0\times 10^{-5}}$
| $\mathbf{1.6\times 10^{-3}}$
---
$\mathbf{5.2\times 10^{-3}}$

_Note:_ CS* refers to the cross-stitch module and Adv.** refers to the
adversarial multi-task training. These abbreviations are also used in the
other tables. Indicated by the blue colour, $\mathcal{N}(0,0.01)$ (Gaussian
noise) is added to the referenced solutions on the initial and boundary
condition, $u(t_{b},x_{b})$, for testing the method robustness. The best
performance is on boldface.

Figure 4: The average training loss
($\frac{1}{N_{T}}\sum^{N_{T}}_{i=1}\mathcal{L}^{i}_{eq}$) plotted with mean
squared error (MSE) evaluated on Burgers’ equation test set in the case of (A)
original PINN (unweighted losses) (B) PCGrad and (C) Uncertainty-weighted
loss. The dotted black blocks indicate the overfitting areas. In (B) and (C),
where the multi-task learning is employed, the overfitting areas are either
narrower (B) or unnoticeable (c). We note that, during training in Section IV,
the network does not have access to any validation or test sets.

####  IV-A2 Poisson equation

The equation is commonly encountered in fluid dynamics. The considering
equation is not varying in time, but still, hard to solve analytically and
therefore a numerical approach is usually required.

$\begin{gathered}u_{xx}+u_{yy}=f(x,y),\,x\in[0,1],\,y\in[0,1]\\\
f(x,y)=-\sin(\pi x)\sin(\pi y)\\\ u(x,0)=0,\,u(x,1)=-\sin(\pi x)\sin(\pi)\\\
u(0,y)=0,\,u(1,y)=-\sin(\pi)\sin(\pi y)\end{gathered}$ (10)

We sample $N_{f}=8,000$ and $N_{b}=200$ using LHS for training. To build the
test set with the shifted distribution, the input space of $(x,y)$ is
thoroughly discretized, having $\Delta x=\Delta y=0.005$. To generate an
auxiliary task, we scale up the $f(x,y)$, setting
$f^{aux}(x,y)=-2\pi^{2}\sin(\pi x)\sin(\pi y)$ as employed in [29]. We train
our neural networks using full-batch Adam with 0.005 learning rate for 50,000
epochs. For the adversarial setting, we set $F=100$, $L=5$, $E=10$ and
$p=0.1$.

The reported results in Table II shows that both the MTL modifications show
the greater performance compared to PINN and ResNet, still, there is no
significant difference between the strategies; thus the adversarial training
is conducted on both strategies. The similar results are found in the PCGrad
case whilst the performance boosts are seen in the uncertainty weighting case,
contributing to the highest prediction accuracy as shown in Fig. 5. We also
found that the cross-stitch units are recommended to make the most out of the
designed MTL losses.

Figure 5: 2D visualization of the Poisson equation’s estimated solutions by (A) PINN and (B) our uncertainty weighting based multi-task network enhanced with the adversarial training (the best variant of Table II). At the second row, the color bar indicates the absolute prediction error from the ground truth. TABLE II: Poisson equation: Performance comparison Method | MAE | MSE | Rel. $l_{2}$ error
---|---|---|---
PINN [3] | $7.9\times 10^{-4}$ | $8.8\times 10^{-7}$ | $2.9\times 10^{-2}$
ResNet [17] | $4.7\times 10^{-4}$ | $2.7\times 10^{-7}$ | $1.7\times 10^{-2}$
PCGrad w/ CS | $1.1\times 10^{-4}$ | $2.0\times 10^{-8}$ | $4.8\times 10^{-3}$
Uncert w/o CS | $3.9\times 10^{-4}$ | $2.1\times 10^{-7}$ | $1.5\times 10^{-2}$
Uncert w/ CS | $1.2\times 10^{-4}$ | $2.0\times 10^{-8}$ | $4.7\times 10^{-3}$
PCGrad w/ CS + Adv. | $1.4\times 10^{-4}$ | $2.6\times 10^{-8}$ | $4.7\times 10^{-3}$
Uncert w/ CS + Adv. | $\mathbf{9.5\times 10^{-5}}$ | $\mathbf{1.6\times 10^{-8}}$ | $\mathbf{4.1\times 10^{-3}}$
TABLE III: Fokker-Planck equation: Performance comparison Method | MAE | MSE | Rel. $l_{2}$ error
---|---|---|---
PINN [3] | $2.2\times 10^{-3}$ | $8.4\times 10^{-6}$ | $9.9\times 10^{-3}$
ResNet [17] | $3.9\times 10^{-3}$ | $2.6\times 10^{-5}$ | $1.7\times 10^{-2}$
PCGrad w/ CS | $1.7\times 10^{-3}$ | $4.9\times 10^{-6}$ | $7.6\times 10^{-3}$
Uncert w/o CS | $1.8\times 10^{-3}$ | $5.2\times 10^{-6}$ | $7.8\times 10^{-3}$
Uncert w/ CS | $9.0\times 10^{-4}$ | $1.4\times 10^{-6}$ | $4.0\times 10^{-3}$
Uncert w/ CS + Adv. | $\mathbf{3.2\times 10^{-4}}$ | $\mathbf{1.9\times 10^{-7}}$ | $\mathbf{1.5\times 10^{-3}}$

####  IV-A3 1D Fokker-Planck equation

The 1D PDE, from [10], which describes a snapshot of the probability density
functions evolution of stochastic systems, is stated in Eq. (11).

$\begin{gathered}-[(ax-
bx^{3})u(x)]_{x}+\frac{\sigma^{2}}{2}u(x)_{xx}=0,\,\Delta{x}\sum^{N_{f}}_{i=1}u(x^{i})=1\\\
u(-2.2)=u(2.2)=0,\,(a,b,\sigma,\Delta{x})=(0.3,0.5,0.5,0.01)\\\
\end{gathered}$ (11)

As described in [10], we can get the training set and the boundary set with
the step length, $\Delta x=0.01$, and evaluate the solver network performance
on the more fine-grained points constructed with the smaller step length,
$\Delta x=0.005$. We consider $a$ to be the PDE parameter $\lambda$ and set
$\lambda^{aux}=0.5$ (a close value) to simply generate an auxiliary equation.
Our neural networks are optimized using full-batch Adam with 0.01 learning
rate for 30,000 epochs. For the adversarial setting, we set $F=100$, $L=3$,
$E=10$ and $p=0.1$.

Based on the performance comparison, which is provided in Table III, we
inspect that the neural network, trained with PCGrad, does not outperform
PINN. This might be because, in practice, we could fortuitously break the
assumptions of PCGrad (See Theorem 2. in [9]), which essentially need to be
held for the loss reduction guarantee. We choose the uncertainty-weighting
scheme to further experiment with our adversarial training and, in
consequence, the performance is enhanced in terms of the reduced MSE from
$10^{-6}$ to $10^{-7}$.

Figure 6: Comparison of the predicted and exact solutions. The predictions at
the first row are reproduced from the previous works (PINN or FBSNN). The
predictions at the second row are from the best variant of our proposed method
for each PDE. The absolute errors (AEs or dotted yellow lines) around the
high-loss or dynamically changing regions are lower when our method is
employed.

###  IV-B High-dimensional forward-backward stochastic partial differential
equation

We investigate the effectiveness of applying multi-task learning in the
context of solving high-dimensional stochastic differential equations (SDE).
Unlike Section III-A, the loss, which is particularly designed to solve SDE,
is based on discretizing a SDE using the standard Euler-Maruyama scheme [4].
The designed loss focuses on keeping the relationship between the current
state values and the next time-step state values of each stochastic process,
$X_{t},Y_{t}$ and $Z_{t}$ rather than estimating the solutions for the entire
spatial-temporal grid; therefore the experiment with adversarial training is
deducted due to the problem formulation difference and some subtle issues, for
instance, Brownian motions at which time steps should be replaced with the
transformed ones. This needs further investigation, which is beyond the scope
of this paper.

To be more specific we consider coupled forward-backward SDE of the form

$\begin{gathered}dX_{t}=\mu(t,X_{t},Y_{t},Z_{t})dt+\sigma(t,X_{t},Y_{t})dW_{t},t\in[0,T]\\\
dY_{t}=\varphi(t,X_{t},Y_{t},Z_{t})dt+Z^{tr}_{t}\sigma(t,X_{t},Y_{t})dW_{t},t\in[0,T)\\\
X_{0}=\xi,\,Y_{T}=g(X_{T}),\,Z_{t}=\nabla Y_{t}\end{gathered}$ (12)

where $W_{t}$ is a vector-valued Brownian motion, $X_{0}=\xi\in\mathbb{R}^{d}$
($X_{t},Z_{t}\in\mathbb{R}^{d}$ as well) is the initial condition for the
forward equation and $Y_{T}=g(X_{T})$ represents the known terminal condition
of the backward equation. $tr$ is defined to be the transpose operation. In
our examples, $\mu$ and $\sigma$ are mappings which output in $\mathbb{R}^{d}$
and $\mathbb{R}^{d\times d}$ while $\varphi$ maps to the equal dimension as
$Y_{t}$, which is a scalar depending on $t$ and stochastic process $X_{t}$.
Assumptions on $\mu$, $\sigma$ and $\varphi$ hold based upon [17]. For a
single trajectory (1 Brownian realization), we could apply the Euler-Maruyama
discretization and approximate

$\begin{multlined}X_{n+1}\approx X_{n}+\mu(t_{n},X_{n},Y_{n},Z_{n})\Delta
t_{n}+\sigma(t_{n},X_{n},Y_{n})\Delta W_{n}\\\ Y_{n+1}\approx
Y_{n}+\varphi(t_{n},X_{n},Y_{n},Z_{n})\Delta
t_{n}+Z^{tr}_{n}\sigma(t_{n},X_{n},Y_{n})\Delta W_{n}\\\
\end{multlined}X_{n+1}\approx X_{n}+\mu(t_{n},X_{n},Y_{n},Z_{n})\Delta
t_{n}+\sigma(t_{n},X_{n},Y_{n})\Delta W_{n}\\\ Y_{n+1}\approx
Y_{n}+\varphi(t_{n},X_{n},Y_{n},Z_{n})\Delta
t_{n}+Z^{tr}_{n}\sigma(t_{n},X_{n},Y_{n})\Delta W_{n}\\\ $ (13)

for $n=1,2,..,N-1$, where $\Delta t^{n}=t^{n+1}-t^{n}=T/N$ and $\Delta
W_{n}\sim\mathcal{N}(0,\Delta t_{n})$. The approximated error term to be
minimized is defined as follows

$\begin{multlined}\mathcal{L}_{eq}=\sum^{M}_{m=1}\sum^{N-1}_{n=0}|Y^{m}_{n+1}-\hat{Y}^{m}_{n+1}|^{2}+\sum^{M}_{m=1}|Y^{m}_{N}-g(X^{m}_{N})|^{2}\\\
=\sum^{M}_{m=1}\sum^{N-1}_{n=0}|Y^{m}_{n+1}-Y^{m}_{n}-\varphi(t_{n},X^{m}_{n},Y^{m}_{n},Z^{m}_{n})\Delta
t_{n}\\\ -(Z^{m}_{n})^{tr}\sigma(t_{n},X^{m}_{n},Y^{m}_{n})\Delta
W^{m}_{n}|^{2}+\sum^{M}_{m=1}|Y^{m}_{N}-g(X^{m}_{N})|^{2}\end{multlined}\mathcal{L}_{eq}=\sum^{M}_{m=1}\sum^{N-1}_{n=0}|Y^{m}_{n+1}-\hat{Y}^{m}_{n+1}|^{2}+\sum^{M}_{m=1}|Y^{m}_{N}-g(X^{m}_{N})|^{2}\\\
=\sum^{M}_{m=1}\sum^{N-1}_{n=0}|Y^{m}_{n+1}-Y^{m}_{n}-\varphi(t_{n},X^{m}_{n},Y^{m}_{n},Z^{m}_{n})\Delta
t_{n}\\\ -(Z^{m}_{n})^{tr}\sigma(t_{n},X^{m}_{n},Y^{m}_{n})\Delta
W^{m}_{n}|^{2}+\sum^{M}_{m=1}|Y^{m}_{N}-g(X^{m}_{N})|^{2}$ (14)

where M and N denote the number of trajectories (batch sizes) and time steps.
For every time steps $n$, we parameterize $Y_{n}=u(t_{n},X_{n};\theta)$ using
a single neural network, then $Z_{n}$ can be estimated by automatic
differentiation of $u(t_{n},X_{n};\theta)$ with respect to $X_{n}$. The
approximated error in Eq. (14) could be intuitively thought of as the
$\mathcal{L}_{eq}$ defined in Eq. (2) and our proposed multi-task learning
modifications are applied in the same manner as described in section III-A.
The varied terms for producing an auxiliary SDE, for example, the drift $\mu$,
the volatility $\sigma$ and even the terminal condition, $g(X_{T})$. We have
tested the effectiveness of our approach for a couple of benchmark problems,
the high-dimensional (100D) Black-Scholes-Barenblatt equation and 20D Allen-
Cahn equation.

####  IV-B1 100D Black-Scholes-Barenblatt equation

One of the most well-known SDE in quantitative finance for modelling the price
of various financial derivatives.

$\begin{gathered}dX_{t}=\sigma diag(X_{t})dW_{t},t\in[0,T]\\\
X_{0}=(1,0.5,1,0.5,...,1,0.5)^{tr}\in\mathbb{R}^{100}\\\
dY_{t}=r(Y_{t}-Z^{tr}_{t}X_{t})dt+\sigma
Z^{tr}_{t}diag(X_{t})dW_{t},t\in[0,T)\\\ Y_{T}=\mathinner{\\!\left\lVert
X_{T}\right\rVert}^{2},(T,\sigma,r)=(1,0.4,0.05)\\\ \end{gathered}$ (15)

To train the solver network, we randomly sample 100 Brownian motions for each
epoch, according to the training settings, which are employed in [17]. Each
motion is discretized into 50 time snapshots. We separately sample another 100
Brownian motions with equal discretized time snapshots for evaluating the
network performance after training. We vary the parameter $\sigma$ which
controls the volatility (Gaussian noise) of the stochastic process $X_{t}$
from 0.4 to a nearby value, 0.3, to produce the auxiliary SDE. Adam optimizer
with 0.001 learning rate is selected for training 20,000 epochs.

The results are listed in Table IV and visualized in the rightmost column of
Fig. 6. PCGrad yields the best performance and enables the predictions to be
closer to the exact solutions, including nearby the terminal conditions and
the spiky areas where it is harder to regress using the FBSNN or ResNet.

TABLE IV: 100D Black-Scholes-Barenblatt equation: Performance comparison

Method | MAE | MSE | Mean relative error
---|---|---|---
FBSNN [4] | $4.9\times 10^{-1}$ | $3.9\times 10^{-1}$ | $6.4\times 10^{-3}$
ResNet [17] | $6.7\times 10^{-1}$ | $6.1\times 10^{-1}$ | $8.6\times 10^{-3}$
PCGrad w/ CS | $\mathbf{2.2\times 10^{-1}}$ | $\mathbf{8.3\times 10^{-2}}$ | $\mathbf{2.8\times 10^{-3}}$
Uncert w/ CS | $2.5\times 10^{-1}$ | $8.9\times 10^{-2}$ | $3.4\times 10^{-3}$

_Note:_ The performance results are averaged across 100 unseen Brownian
realizations.

####  IV-B2 20D Allen-Cahn equation

$\begin{gathered}dX_{t}=dW_{t},t\in[0,T]\\\
X_{0}=(0,0,...,0)^{tr}\in\mathbb{R}^{20}\\\
dY_{t}=(-Y_{t}+Y^{3}_{t})dt+Z^{tr}_{t}dW_{t},t\in[0,T)\\\
Y_{T}=g(X_{T}),g(x)=(2+0.4\mathinner{\\!\left\lVert
x\right\rVert}^{2})^{-1},T=0.3\\\ \end{gathered}$ (16)

Following the training and testing scheme employed in [17], we arbitrarily
sample 100 Brownian motions for each training epoch. Each motion is
discretized into 15 time snapshots. Another 100 Brownian motions with equal
discretized time snapshots are sampled for testing the network performance. We
generate the auxiliary SDE by slightly changing the terminal condition,
setting $g^{aux}(x)=(2+0.3\mathinner{\\!\left\lVert x\right\rVert}^{2})^{-1}$.
Adam optimizer with 0.001 learning rate is selected for training 20,000
epochs.

We measure the solver network test performance with 100 unseen Brownian
motions at the terminal time, $T=0.3$, where the analytical solution is known.
The results listed in Table V indicates that the uncertainty-weighted strategy
attains the best performance, being competitive to ResNet. This may be because
both models perform similar mechanisms for retaining low-level information
which helps to suppress the vanishing gradient problem. The visual comparison
between FBSNN and the uncertainty weighting approach is shown in Fig. 7.

TABLE V: 20D Allen-Cahn equation: Performance comparison

Method | MAE | MSE | Mean relative error
---|---|---|---
FBSNN [4] | $6.7\times 10^{-3}$ | $5.1\times 10^{-5}$ | $2.8\times 10^{-2}$
ResNet [17] | $3.2\times 10^{-3}$ | $1.4\times 10^{-5}$ | $1.4\times 10^{-3}$
PCGrad w/ CS | $4.1\times 10^{-3}$ | $2.3\times 10^{-5}$ | $1.7\times 10^{-2}$
Uncert w/ CS | $\mathbf{2.8\times 10^{-3}}$ | $\mathbf{1.2\times 10^{-5}}$ | $\mathbf{1.2\times 10^{-2}}$

_Note:_ The performance results are averaged over 100 unseen Brownian
realizations at time $T=0.3$.

Figure 7: Predictions by (A) FBSNN and (B) our uncertainty weighting based
approach on 5 Brownian motions, which are randomly selected from the test set.
More accurate predictions at time $T=0.3$ are clearly spotted when the
uncertainty weighting is applied.

###  IV-C Auxiliary task selection heuristic

To study the effect of $\alpha$ in the Burgers’ example, we conduct 10 runs of
Bayesian optimization with Asynchronous Successive Halving Algorithm (ASHA)
[30] early stopping, which aims to economically search for the optimal alpha
50 trials, minimizing $\mathcal{L}_{uncert}$ for maximum 200 iterations. The
purpose of the multiple runs is to lessen the bias towards a particular random
weights initialization and enhance the exploration capacity. The results are
shown in Fig 8.

We have found that, with respect to the random weights initialization
difference across the multiple Bayesian searches, the various local optimal
coefficients are spotted. In the exploitation frame, the network gradually
performs better when the coefficient is marginally changed, indicating the
loss sensitivity to a group of slightly distinct (not identical) $\alpha$.
Adhering to a local optimal coefficient offers some chances, but does not
guarantee obtaining the optimum result in a long run. Since, in practice, the
initial neural network weights do not have to be in one particular form, this
encourages us that random coefficient assignment is required and might even
yield a satisfying trained network as well. To be computationally inexpensive,
one should try exploring with the close or random coefficient values to
preliminarily obtain the performance. If the network performance does not
comply with the requirement, the Bayesian optimization should come to the
rescue by increasing the exploitation capability.

Figure 8: The results from 10 runs of the Bayesian optimization, excluding the
early stopped trials. We frame the exploration box to signify that there are
multiple local optimum coefficients, which produce decent performing network
checkpoints.

##  V Conclusion

We have introduced the novel approach of applying multi-task learning to the
original PINN and FBSNN for producing better-generalized solutions. Compared
to the previous works, our method helps reduce the model’s test errors on
various PDE examples, ranging from low-dimensional settings to high-dimension
settings. Both multi-task learning and adversarial training contribute to the
enhanced performance of the trained network by incorporating joint
representation learning of multiple target PDE instances and additional
difficult samples, which helps the network to highly concentrate on the high-
loss regions that are more challenging to learn.

As future investigations, our methods are flexible and could be further
advanced to tackle (1) chaotic systems, in which two (systems of) equations
have similar coefficients, but exhibit completely distinct solutions, (2)
parameterized PDEs or (3) a system of PDEs. Finally, together with the
development of Physics-guided machine learning such as those studies in [31]
and [32], once the PDE, which governs how the physical system evolves, is
effectively solved at the desired accuracy, any learned features embedded in
the neural network are easily included in the standard deep learning pipeline
for enhancing the network accuracy and robustness.

## Acknowledgment

This work was supported by JSPS KAKENHI Grant Number JP19K22876.

## References

  * [1] G. Carleo, I. Cirac, K. Cranmer, L. Daudet, M. Schuld, N. Tishby, L. Vogt-Maranto, and L. Zdeborová, “Machine learning and the physical sciences,” Reviews of Modern Physics, vol. 91, no. 4, p. 045002, 2019.
  * [2] A. G. Baydin, B. A. Pearlmutter, A. A. Radul, and J. M. Siskind, “Automatic differentiation in machine learning: a survey,” The Journal of Machine Learning Research, vol. 18, no. 1, pp. 5595–5637, 2017.
  * [3] M. Raissi, P. Perdikaris, and G. E. Karniadakis, “Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations,” Journal of Computational Physics, vol. 378, pp. 686–707, 2019.
  * [4] M. Raissi, “Forward-backward stochastic neural networks: Deep learning of high-dimensional partial differential equations,” arXiv preprint arXiv:1804.07010, 2018.
  * [5] A. Al-Aradi, A. Correia, D. Naiff, G. Jardim, and Y. Saporito, “Solving nonlinear and high-dimensional partial differential equations via deep learning,” arXiv preprint arXiv:1811.08782, 2018.
  * [6] R. van der Meer, C. Oosterlee, and A. Borovykh, “Optimally weighted loss functions for solving pdes with neural networks,” arXiv preprint arXiv:2002.06269, 2020.
  * [7] I. Misra, A. Shrivastava, A. Gupta, and M. Hebert, “Cross-stitch networks for multi-task learning,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3994–4003, 2016.
  * [8] A. Kendall, Y. Gal, and R. Cipolla, “Multi-task learning using uncertainty to weigh losses for scene geometry and semantics,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7482–7491, 2018\.
  * [9] T. Yu, S. Kumar, A. Gupta, S. Levine, K. Hausman, and C. Finn, “Gradient surgery for multi-task learning,” in Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
  * [10] Y. Xu, H. Zhang, Y. Li, K. Zhou, Q. Liu, and J. Kurths, “Solving fokker-planck equation using deep learning,” Chaos: An Interdisciplinary Journal of Nonlinear Science, vol. 30, no. 1, p. 013133, 2020.
  * [11] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in Advances in Neural Information Processing Systems, vol. 27, pp. 2672–2680, Curran Associates, Inc., 2014.
  * [12] B. C. Csáji et al., “Approximation with artificial neural networks,” Faculty of Sciences, Etvs Lornd University, Hungary, vol. 24, no. 48, p. 7, 2001.
  * [13] M. Dissanayake and N. Phan-Thien, “Neural-network-based approximations for solving partial differential equations,” communications in Numerical Methods in Engineering, vol. 10, no. 3, pp. 195–201, 1994.
  * [14] I. E. Lagaris, A. Likas, and D. I. Fotiadis, “Artificial neural networks for solving ordinary and partial differential equations,” IEEE transactions on neural networks, vol. 9, no. 5, pp. 987–1000, 1998.
  * [15] J. Han, A. Jentzen, and E. Weinan, “Solving high-dimensional partial differential equations using deep learning,” Proceedings of the National Academy of Sciences, vol. 115, no. 34, pp. 8505–8510, 2018.
  * [16] J. Sirignano and K. Spiliopoulos, “Dgm: A deep learning algorithm for solving partial differential equations,” Journal of computational physics, vol. 375, pp. 1339–1364, 2018.
  * [17] B. Güler, A. Laignelet, and P. Parpas, “Towards robust and stable deep learning algorithms for forward backward stochastic differential equations,” arXiv preprint arXiv:1910.11623, 2019.
  * [18] I. Kokkinos, “Ubernet: Training a universal convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6129–6138, 2017.
  * [19] S. Ruder, J. Bingel, I. Augenstein, and A. Søgaard, “Latent multi-task architecture learning,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 4822–4829, 2019.
  * [20] Z. Chen, V. Badrinarayanan, C.-Y. Lee, and A. Rabinovich, “Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks,” in International Conference on Machine Learning, pp. 794–803, PMLR, 2018.
  * [21] S. Liu, E. Johns, and A. J. Davison, “End-to-end multi-task learning with attention,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1871–1880, 2019.
  * [22] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. J. Goodfellow, and R. Fergus, “Intriguing properties of neural networks,” in 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
  * [23] I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” in 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
  * [24] Y. Yang and P. Perdikaris, “Adversarial uncertainty quantification in physics-informed neural networks,” Journal of Computational Physics, vol. 394, pp. 136–152, 2019.
  * [25] J. Snoek, H. Larochelle, and R. P. Adams, “Practical bayesian optimization of machine learning algorithms,” in Advances in Neural Information Processing Systems, vol. 25, pp. 2951–2959, Curran Associates, Inc., 2012.
  * [26] C. Basdevant, M. Deville, P. Haldenwang, J. Lacroix, J. Ouazzani, R. Peyret, P. Orlandi, and A. Patera, “Spectral and finite difference solutions of the burgers equation,” Computers & fluids, vol. 14, no. 1, pp. 23–41, 1986\.
  * [27] M. Stein, “Large sample properties of simulations using latin hypercube sampling,” Technometrics, vol. 29, no. 2, pp. 143–151, 1987.
  * [28] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
  * [29] T. Dockhorn, “A discussion on solving partial differential equations using neural networks,” arXiv preprint arXiv:1904.07200, 2019.
  * [30] L. Li, K. Jamieson, A. Rostamizadeh, E. Gonina, M. Hardt, B. Recht, and A. Talwalkar, “Massively parallel hyperparameter tuning,” arXiv preprint arXiv:1810.05934, 2018.
  * [31] K. Fukui, J. Tanaka, T. Tomita, and M. Numao, “Physics-guided neural network with model discrepancy based on upper troposphere wind prediction,” in 2019 18th IEEE International Conference On Machine Learning And Applications (ICMLA), pp. 414–419, IEEE, 2019.
  * [32] J. Willard, X. Jia, S. Xu, M. Steinbach, and V. Kumar, “Integrating physics-based modeling with machine learning: A survey,” arXiv preprint arXiv:2003.04919, 2020.


eqnum(())eqref eqref

# Challenges of Testing an Evolving Cancer Registration Support System in
Practice

Christoph Laaber, Tao Yue, Shaukat Ali  Simula Research Laboratory
Oslo, Norway
{laaber, tao<EMAIL_ADDRESS>Thomas Schwitalla  Cancer Registry of Norway
Oslo, Norway
<EMAIL_ADDRESS>Jan F. Nygård  Cancer Registry of Norway
Oslo, Norway
UiT The Arctic University of Norway
Tromsø, Norway
<EMAIL_ADDRESS>
###### Abstract

The Cancer Registry of Norway (CRN) is a public body responsible for capturing
and curating cancer patient data histories to provide a unified access to
research data and statistics for doctors, patients, and policymakers. For this
purpose, CRN develops and operates a complex, constantly-evolving, and socio-
technical software system. Recently, machine learning (ML) algorithms have
been introduced into this system to augment the manual decisions made by
humans with automated decision support from learned models. To ensure that the
system is correct and robust and cancer patients’ data are properly handled
and do not violate privacy concerns, automated testing solutions are being
developed. In this paper, we share the challenges that we identified when
developing automated testing solutions at CRN. Such testing potentially
impacts the quality of cancer data for years to come, which is also used by
the system’s stakeholders to make critical decisions. The challenges
identified are not specific to CRN but are also valid in the context of other
healthcare registries. We also provide some details on initial solutions that
we are investigating to solve the identified challenges.

###### Index Terms:

research challenges, healthcare, cancer registry, software testing, evolution

(-1.5em,18cm)  © 2023 IEEE. Personal use of this material is permitted.
Permission from IEEE must be obtained for all other uses, in any current or
future media, including reprinting/republishing this material for advertising
or promotional purposes, creating new collective works, for resale or
redistribution to servers or lists, or reuse of any copyrighted component of
this work in other works. DOI: 10.1109/ICSE-Companion58688.2023.00102

##  I Introduction

The Cancer Registry of Norway (CRN) is a public entity responsible for
collecting cancer patients’ data from multiple sources, including diagnostic,
treatment, and follow-up data, as shown in Figure 1. Next, such data are
processed for cancer research and statistics to various users such as
patients, medical institutions, and relevant governmental authorities. For
this purpose, CRN has developed a Cancer Registration Support System (CaReSS),
an interactive, human-in-the-loop decision support system. CaReSS experiments
with using machine learning (ML) techniques and is subject to continuous
evolution due to, e.g., new diagnostic methods and treatment, standards, and
regulations (e.g., General Data Protection Regulation (GDPR)), in addition to
bug fixing and updates to its ML models.

CaReSS collects patient information from various medical entities such as
pathology and radiology laboratories, hospitals, and other registries such as
the national patient registry and the death registry, as shown in Figure 1.
Such patient information is recorded as cancer messages and prepared for
trained medical coders to create cancer cases, each of which is a timeline of
a patient’s diagnostic tests, treatments and follow-ups. The coding process,
i.e., transforming medical reports (messages) into uniform codes based on a
classification system such as the International Classification of Diseases
(ICD)111https://www.who.int/standards/classifications/classification-of-
diseases from World Health Organization (WHO), is dependent on hundreds of
medical rules, which are used to validate cancer messages, assigning these to
cancer cases, etc. These rules are defined based on the domain knowledge of
medical experts. The CRN constantly introduces new and revises existing rules
according to new medical knowledge and procedures. However, validating and
assigning messages to cancer cases are performed by a rule engine of CaReSS.

Below, we provide two example rules based on the examples provided in [1, 2].
Rule 1 is a simple check on the invalid values of two variables, i.e., basis
and surgery. Rule 2, on the other hand, is more complex and states that for
all the cancer messages of type H coming to CaReSS from medical entities
(Figure 1), if the surgery value is equal to 96 then the basis value must be
greater than 32.

Rule 1:

$basis\neq 71\land surgery\neq 21$

Rule 2:

$\forall\;messageType=H\implies(surgery=96\implies basis>32)$

CaReSS outputs data for researchers to perform epidemiological studies with
standard statistical methods and ML techniques. It also produces cancer
statistics used by end-users (e.g., patients and policymakers) as references
for well-informed decisions, as shown in Figure 1.

Figure 1: Overview of the Cancer Registration Support System.

Given that the CaReSS is used to make critical decisions by its stakeholders,
it is important that the system is tested thoroughly to ensure its
correctness, security, and privacy (e.g., according to the GDPR). Moreover,
the data augmentation, transformation, and classification in CaReSS directly
affect the quality of the produced data, which are further used to make
decisions by the CaReSS’s users. Therefore, testing is critical to ensure that
the data produced is of the highest quality such that accurate decisions can
be made by CaReSS’s end-users.

CRN’s team developing the CaReSS is now building automated testing solutions
to deal with the constant evolution of the CaReSS, including the changes in
the rules it uses, the algorithms it employs, and bugs found. To this end, a
set of testing challenges have been identified based on discussion with
various people involved in developing the CaReSS. Such challenges are quite
common across other healthcare registries; thus, this paper aims to share
these challenges together with the investigation of the initial set of
proposed solutions.

We organize the paper as follows: Section II presents the real-world testing
problem we are focusing on, our main objective, and the potential impact that
solving the problem will bring. Section III explains various challenges,
whereas Section IV will present how these challenges are also valid in other
industrial and real-world contexts. Finally, we conclude the paper in Section
V.

##  II Scope of the Real-World Problem

This section presents the real-world problem we focus on for testing in
Section II-A, Section II-B explains our key testing objectives, and finally
Section II-C presents the potential impact we would like to achieve with
automated testing.

###  II-A Scope: CaReSS Evolution

Automated checking of cancer data against medical rules is an important step
in ensuring the quality of patient history registration at CRN and in CaReSS.
However, these medical rules evolve constantly, e.g., because of new findings
from medical research and the introduction of new cancer message types from
new medical diagnostics or treatments. Moreover, CaReSS employs algorithms for
producing cancer statistics. These algorithms undergo constant changes because
CaReSS keeps upgrading itself by employing advanced versions of algorithms and
retraining models when new data or new data types are available. Such upgrades
are necessary to improve the efficiency and timeliness of the cancer data used
for medical research and the general public. In summary, continuous changes
and evolution in data (messages, patients’ histories, rules, etc.) and
algorithms lead to the continuous evolution of the CaReSS’s key software
components.

###  II-B Objective: Testing Evolving CaReSS

Due to continuous evolution, CaReSS must be continuously tested to guarantee
that it produces reliable statistics and data for its end users. Failing to do
so would significantly negatively affect scientific results produced by
researchers and potentially mislead patients, hospitals, and policymakers to
make inaccurate decisions. Thus, a cost-effective, systematic, automated, and
self-managing testing approach is needed to handle the evolving aspect of
CaReSS! More specifically, the innovation planned is a state-of-the-art test
infrastructure with cost-effective testing techniques utilizing ML and
evolutionary computation algorithms. This will significantly improve the
quality of CaReSS and the data and statistics it produces by dealing with the
algorithms’ continuous evolution and unpredictable behavior.

###  II-C Expected Impact: Improved CaReSS Quality

First, this innovation is expected to have a long-term impact (of several
decades at least) on the service quality that CaReSS provides for its users.
More specifically, the well-tested CaReSS would produce data and statistics
with higher quality much more quickly. Such data will be consumed by
researchers to perform analyses more accurately and reliably and provide
patients, doctors, the general public, and government officials to achieve
various objectives. Second, a systematic and automated testing solution for
CaReSS will improve the overall time-wise efficiency of evolving CaReSS for
decades, implying a significant reduction in the overall cost of testing and
maintaining CaReSS.

##  III Challenges

This section presents challenges from three perspectives, i.e., scientific
(Section III-A), project execution (Section III-B), and sharing tools (Section
III-C). The scientific challenges are of interest to researchers developing
novel testing methods solving these challenges. The project execution and tool
sharing challenges are relevant for practitioners who are involved in a
similar project as ours in a real-world setting.

###  III-A Scientific Challenges

New research is needed to propose (modeling) methodologies that can
systematically capture various types of the evolution of CaReSS as test models
to support automated testing. Examples of evolution include changes in medical
rules and software updates. Next, based on the models, test cases can be
generated based on test strategies focusing on test objectives, such as
finding specific kinds of faults and achieving various types of coverage. To
this end, there is a need for new testing strategies, e.g., covering coverage
specific to CaReSS, e.g., coverage of rules. Another aspect to investigate is
whether classical code coverage (e.g., statement coverage) correlates to rule
coverage.

At CRN, an important objective is to test CaReSS in the presence of evolution,
which leads to uncertain behavior due to unknown or erroneous data about a
patient’s history, uncertain time of the determination of a cancer case, and
the uncertain behavior of rules when emerging or incorrect data are provided.
Solutions for testing software systems under uncertainties (e.g., [3, 4, 5])
have been proposed, which, however, do not address the challenges of testing
constantly-evolving software systems like CaReSS, though we acknowledge we can
get inspired from the literature on aspects like how to model/specify
uncertainties in test models. Existing testing approaches (e.g., [6, 7])
mostly focus on testing evolving software by just looking into their versions.

The use of ML for decision-making introduces additional uncertainty because of
the unpredictable nature of such algorithms. Solutions (e.g., [8, 9]) have
been proposed to test artificial neural networks. Some works (e.g., [10, 11])
focus on generating highly uncertain test inputs. Such solutions, however,
only test ML models in an isolated manner, which is considered insufficient
for our context. We, hence, need novel test strategies to test CaReSS in the
presence of uncertainty and evolution. Testing uncertainty and the emerging
behaviors of a constantly-evolving healthcare system has not been sufficiently
investigated in either the literature or practice.

Another area we plan to investigate is uncertainty quantification in the ML
models used for decision-making such that we can assess the robustness of
these ML models against such uncertainties and possibly reduce such
uncertainties. Some works exist that quantify uncertainty in deep learning
models [12, 13, 14]. We want to investigate whether applying such uncertainty
quantification methods can generate highly uncertain inputs to test the
implemented ML models in isolation and as part of the whole system. Any issues
found with such testing would be investigated to see whether these issues can
be resolved with additional training data for the ML models or other measures.

CaReSS is a socio-technical system, i.e., medical coders are an essential part
of the system to validate, encode/transform hence creating medical data. This
further complicates the testing of CaReSS. It is unclear how to _mock_ medical
coders (human) in the automated testing without trivializing the system’s
capabilities. One possible direction could be to learn from logs how a human
has dealt with different input types and substitute the human with a model
during testing. Although there is some initial research on learning user
behavior for testing purposes [15, 16], these are arguably less complex as the
interactions are more trivial than in the medical context where highly-trained
domain experts are required.

Due to the dependence on human decisions, CaReSS is currently predominantly
tested manually, as previously observed for other systems [17]. That is,
medical coders use their domain knowledge to manually test CaReSS to ensure it
adheres to its (informal) specification. Due to the complexity of and reliance
on medical coders, creating meaningful test cases is complex for software
engineers who are not medical experts. Consequently, supporting medical coders
in their manual test efforts, e.g., selecting and prioritizing manual testing
as well as manual test quality monitoring [17], is an orthogonal (to automated
testing) important challenge.

###  III-B Project Execution Challenges

A central challenge concerns the security and privacy of handling sensitive
healthcare data. Handling patient data is confined to a restricted and secure
zone within the premises of CRN. This includes data access (e.g., for external
scientific collaborators with testing expertise), installation of arbitrary
software that is needed for testing, and usage of real data for testing
purposes. In particular, the last point is crucial because using real patient
data for testing is prohibited by laws and regulations, such as GDPR.
Alternatives include generating synthetic and realistic data with novel ML
approaches (e.g., Generative Adversarial Networks (GANs)). However, this is
non-trivial as well, as concerns about the re-identification of real patients
from synthetic data and the models trained on real data remain an open issue.
This inhibits or at least complicates sharing of synthetic data generated by
as well as the models themselves trained on real patient data.

###  III-C Tool Sharing Challenges

Beyond the data, sharing tools that handle patient data is a challenge on its
own. CaReSS is custom-built and uses a high degree of internally-hosted,
proprietary software solutions, which complicates the creation of a
_standalone_ version for our research purposes. While one option is to rely on
open-source systems (if they exist) instead, this decreases the realism of the
context in which novel solutions are designed and consequently complicates
their adoptions in the real application contexts.

##  IV Generalization Beyond the Cancer Registry of Norway

The challenges are not specific to CRN, and it can be tailored to serve cancer
registries in other countries with reasonable effort, as cancer registration
and coding are quite “universal.” For instance, ICD-O-3 is commonly referenced
for cancer codes.222https://seer.cancer.gov/icd-o-3/ We believe the scientific
challenges we identified are commonly faced by cancer registry systems. In
addition, other registries such as the national registry used by tax
authorities, election authorities, and other public authorities (e.g., police)
face similar challenges. For instance, tax authorities might automatically
execute rules to calculate and validate a person’s or a company’s taxes.

The challenges of re-identification and synthetic user data for testing are
also not unique to the CRN case or cancer registries in general. Other
personal, sensitive data require increased protection, such as racial and
ethnic origin as well as biometric data, as specified by GDPR.333https://gdpr-
info.eu/recitals/no-51/ Software systems handling such data, which should also
be tested, can adapt the solutions from the medical domain and the CRN case.

Finally, testing any kind of system where human processing is essential will
experience similar challenges related to mocking or learning human behavior,
particularly when automated testing is desired. We believe the solutions
proposed for CRN can inform researchers and practitioners working with other
systems and in different domains addressing learning domain expert
interactions.

##  V Conclusion and Future Work

This paper presented an innovation project taking place at the Cancer Registry
of Norway (CRN) to automate the testing of the Cancer Registration Support
System (CaReSS) with the ultimate objective of improving the quality of the
systems and the data it produces. The data produced by the system are used by
key system stakeholders to make key decisions; therefore, ensuring
correctness, robustness, and privacy is of utmost importance. We presented the
key challenges of automating the testing of CaReSS, including evolution,
uncertainty, and dealing with human-in-the-loop. Such automation will
revolutionize the current testing practice at CRN, whose impact will result in
high-quality CaReSS and the data it produces, thereby positively affecting
CaReSS’s users. We hope that the presented challenges are interesting for the
researchers to investigate since such challenges are common across cancer
registries worldwide and other healthcare registries.

Currently, we are implementing various testing solutions to automate testing
CaReSS from various aspects. First, we are looking into automated testing from
a human perspective with Graphical User Interface (GUI)-based testing. Second,
we are investigating evolutionary algorithms to test CaReSS with its provided
REST APIs. Third, we plan to utilize different machine learning algorithms,
such as reinforcement learning, to automate testing. Finally, we envision
integrating GUI-based and REST API-based testing to develop a more
comprehensive testing solution. In addition, we will also develop solutions to
handle testing challenges that we described in the paper, including
uncertainty and evolution.

##  VI Acknowledgement

This work is supported by the "AI-Powered Testing Infrastructure for Cancer
Registry System" project (Project #309642) supported by the Research Council
of Norway under the Innovation in Public Sector scheme.

## References

  * [1] H. Lu, S. Wang, T. Yue, s. Ali, and J. F. Nygård, “Automated refactoring of ocl constraints with search,” _IEEE Transactions on Software Engineering_ , vol. 45, no. 2, pp. 148–170, 2019.
  * [2] S. Wang, T. Schwitalla, T. Yue, S. Ali, and J. F. Nygård, “Rcia: Automated change impact analysis to facilitate a practical cancer registry system,” in _2017 IEEE International Conference on Software Maintenance and Evolution (ICSME)_ , 2017, pp. 603–612.
  * [3] M. Zhang, S. Ali, and T. Yue, “Uncertainty-wise test case generation and minimization for cyber-sphysical systems,” _Journal of Systems and Software_ , vol. 153, pp. 1–21, 2019.
  * [4] S. Y. Shin, K. Chaouch, S. Nejati, M. Sabetzadeh, L. C. Briand, and F. Zimmer, “Uncertainty-aware specification and analysis for hardware-in-the-loop testing of cyber-physical systems,” _Journal of Systems and Software_ , vol. 171, p. 110813, 2021.
  * [5] M. Camilli, A. Gargantini, P. Scandurra, and C. Trubiani, “Uncertainty-aware exploration in model-based testing,” in _Proceedings of the 14th IEEE Conference on Software Testing, Verification and Validation_ , ser. ICST 2021\. IEEE, 2021, pp. 71–81.
  * [6] S. Mühlbauer, S. Apel, and N. Siegmund, “Identifying software performance changes across variants and versions,” in _Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering_ , ser. ASE 2020. IEEE, 2020, pp. 611–622.
  * [7] X. Wu, “Effectively analyzing evolving software with differential facts,” in _Proceedings of the 36th IEEE/ACM International Conference on Automated Software Engineering_ , 2021, pp. 1064–1068.
  * [8] Y. Sun, X. Huang, D. Kroening, J. Sharp, M. Hill, and R. Ashmore, “Testing deep neural networks,” _arXiv preprint arXiv:1803.04792_ , 2018.
  * [9] Z. Wang, H. You, J. Chen, Y. Zhang, X. Dong, and W. Zhang, “Prioritizing test inputs for deep neural networks via mutation analysis,” in _Proceedings of the 43rd IEEE/ACM International Conference on Software Engineering_ , ser. ICSE 2021, 2021, pp. 397–409.
  * [10] F. O. Catak, T. Yue, and S. Ali, “Uncertainty-aware prediction validator in deep learning models for cyber-physical system data,” _ACM Transactions on Software Engineering and Methodology_ , vol. 31, no. 4, Jul. 2022\. [Online]. Available: https://doi.org/10.1145/3527451
  * [11] M. Weiss, A. G. Gómez, and P. Tonella, “A forgotten danger in DNN supervision testing: Generating and detecting true ambiguity,” _arXiv preprint arXiv:2207.10495_ , 2022.
  * [12] F. Catak, T. Yue, and S. Ali, “Prediction surface uncertainty quantification in object detection models for autonomous driving,” in _2021 IEEE International Conference On Artificial Intelligence Testing (AITest)_. Los Alamitos, CA, USA: IEEE Computer Society, aug 2021, pp. 93–100. [Online]. Available: https://doi.ieeecomputersociety.org/10.1109/AITEST52744.2021.00027
  * [13] F. O. Catak, T. Yue, and S. Ali, “Uncertainty-aware prediction validator in deep learning models for cyber-physical system data,” _ACM Trans. Softw. Eng. Methodol._ , vol. 31, no. 4, jul 2022. [Online]. Available: https://doi.org/10.1145/3527451
  * [14] Q. Xu, S. Ali, T. Yue, and M. Arratibel, “Uncertainty-aware transfer learning to evolve digital twins for industrial elevators,” in _Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ , ser. ESEC/FSE 2022. New York, NY, USA: Association for Computing Machinery, 2022, p. 1257–1268. [Online]. Available: https://doi.org/10.1145/3540250.3558957
  * [15] J. Ahlgren, M. E. Berezin, K. Bojarczuk, E. Dulskyte, I. Dvortsova, J. George, N. Gucevska, M. Harman, R. Lämmel, E. Meijer, S. Sapora, and J. Spahr-Summers, “Wes: Agent-based user interaction simulation on real infrastructure,” in _Proceedings of the 42nd IEEE/ACM International Conference on Software Engineering Workshops_ , ser. ICSEW’20. Association for Computing Machinery, 2020, pp. 276–284. [Online]. Available: https://doi.org/10.1145/3387940.3392089
  * [16] P. Salza, M. E. Palma, and H. C. Gall, “Synthetic end-user testing: Modeling realistic agents based on behavioral examples,” 2022. [Online]. Available: https://arxiv.org/abs/2208.12261
  * [17] R. Haas, D. Elsner, E. Juergens, A. Pretschner, and S. Apel, “How can manual testing processes be optimized? Developer survey, optimization guidelines, and case studies,” in _Proceedings of the 29th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ , ser. ESEC/FSE 2021. Association for Computing Machinery (ACM), Aug. 2021. [Online]. Available: https://doi.org/10.1145/3468264.3473922