# Balanced Adversarial Training: Balancing Tradeoffs between
Fickleness and Obstinacy in NLP Models

Hannah Chen, Yangfeng Ji, David Evans
Department of Computer Science
University of Virginia
Charlottesville, VA 22904
<EMAIL_ADDRESS>

###### Abstract

Traditional (_fickle_) adversarial examples involve finding a small
perturbation that does not change an input’s true label but confuses the
classifier into outputting a different prediction. Conversely, _obstinate_
adversarial examples occur when an adversary finds a small perturbation that
preserves the classifier’s prediction but changes the true label of an input.
Adversarial training and certified robust training have shown some
effectiveness in improving the robustness of machine learnt models to fickle
adversarial examples. We show that standard adversarial training methods
focused on reducing vulnerability to fickle adversarial examples may make a
model more vulnerable to obstinate adversarial examples, with experiments for
both natural language inference and paraphrase identification tasks. To
counter this phenomenon, we introduce Balanced Adversarial Training, which
incorporates contrastive learning to increase robustness against both fickle
and obstinate adversarial examples.

##  1 Introduction

Interpreted broadly, an adversarial example is an input crafted intentionally
to confuse a model. Most research on adversarial examples, however, focuses on
a definition of an adversarial example as an input that is constructed by
making minimal perturbations to a normal input that change the model’s output,
assuming that the small perturbations preserve the original true label
(Goodfellow et al., 2015). Such adversarial examples occur when a model is
overly influenced by small changes in the input. Attackers can also target the
opposite objective—to find inputs with minimal changes that change the ground
truth label but for which the model retains its prior prediction (Jacobsen et
al., 2019b).

Various names have been used in the research literature for these two types of
adversarial examples including perturbation or sensitivity-based and
invariance-based examples (Jacobsen et al., 2019b, a), and over-sensitive and
over-stable examples Niu and Bansal (2018); Kumar and Boulanger (2020). To
avoid confusions associated with these names, we refer them as _fickle
adversarial examples_ (the model changes its output too easily) and _obstinate
adversarial examples_ (the model doesn’t change its output even though the
input has changed in a way that it should).

In NLP, synonym-based word substitution is a common method for constructing
fickle adversarial examples (Alzantot et al., 2018; Jin et al., 2020) since
synonym substitutions are assumed to not change the true label for an input.
These methods target a model’s weakness of being invariant to certain types of
changes which makes its predictions insufficiently responsive to small input
changes. Attacks based on antonyms and negation have been proposed to create
obstinate adversarial examples for dialogue models (Niu and Bansal, 2018).

Adversarial training is considered as the most effective defense strategy yet
found against adversarial examples (Madry et al., 2018; Goodfellow et al.,
2016). It aims to improve robustness by augmenting the original training set
with generated adversarial examples in a way that results in decision
boundaries that correctly classify inputs that otherwise would have been
fickle adversarial examples. Adversarial training has been shown to improve
robustness for NLP models (Yoo and Qi, 2021). Recent works have also studied
certified robustness training which gives a stronger guarantee that the model
is robust to all possible perturbations of a given input (Jia et al., 2019; Ye
et al., 2020).

While prior work on NLP robustness focuses on fickle adversarial examples, we
consider both fickle and obstinate adversarial examples. We then further
examine the impact of methods designed to improve robustness to fickle
adversarial examples on a model’s vulnerability to obstinate adversarial
examples. Recent work in the vision domain demonstrated that increasing
adversarial robustness of image classification models by training with fickle
adversarial examples may increase vulnerability to obstinate adversarial
examples (Tramer et al., 2020). Even in cases where the model certifiably
guarantees that no adversarial examples can be found within an $L_{p}$-bounded
distance, the norm-bounded perturbation does not align with the ground truth
decision boundary. This distance-oracle misalignment makes it possible to have
obstinate adversarial examples located within the same perturbation distance,
as depicted in Figure 1. In text, fickle examples are usually generated with a
cosine similarity constraint to encourage the representations of the original
and the perturbed sentence to be close in the embedding space. However, this
similarity measurement may not preserve the actual semantics (Morris et al.,
2020) and the model may learn poor representations during adversarial
training.

Figure 1: Distance-oracle misalignment (Tramer et al., 2020). While the model
is trained to be robust to $\epsilon$-bounded perturbation, it becomes too
invariant to small changes in the example (obstinate example $\tilde{x}$) that
lie on the other side of the oracle decision boundary.

Contributions. We study fickle and obstinate adversarial robustness in NLP
models with a focus on synonym and antonym-based adversarial examples (Figure
2 shows a few examples). We evaluate both kinds of adversarial robustness on
natural language inference and paraphrase identification tasks with BERT
(Devlin et al., 2019) and RoBERTa (Liu et al., 2019) models. We find that
there appears to be a tradeoff between robustness to synonym-based and
antonym-based attacks. We show that while certified robust training increases
robustness against synonym-based adversarial examples, it increases
vulnerability to antonym-based attacks (Section 3). We propose a modification
to robust training, Balanced Adversarial Training (BAT), which uses a
contrastive learning objective to help mitigate the distance misalignment
problem by learning from both fickle and obstinate examples (Section 4). We
implement two versions of BAT with different contrastive learning objectives,
and show the effectiveness in improving both fickleness and obstinacy
robustness (Section 4.2).

Figure 2: Fickle and obstinate adversarial examples for BERT model fine-tuned
on natural language inference (left) and paraphrase identification (right)
tasks. Words in red are substituted with their synonyms and words in blue are
replaced by their antonyms.

##  2 Constructing Adversarial Examples

We consider a classification task where the goal of the model $f$ is to learn
to map the textual input $x$, a sequence of words, $x_{1},x_{2},...,x_{L}$, to
its ground truth label $y\in\\{1,...,c\\}$. We assume there is a labeling
oracle $\mathcal{O}$ that corresponds to ground truth and outputs the true
label of the given input. We focus on word-level perturbations where the
attacker substitutes words in the original input $x$ with words from a known
perturbation set (which we show how we construct it in the following
sections). The goal of the attacker is to find an adversarial example
$\tilde{x}$ for input $x$ such that the output of the model is different from
what human would interpret, i.e. $f(\tilde{x})\neq\mathcal{O}(\tilde{x})$.

###  2.1 Fickle Adversarial Examples

For a given input $(x,y)$ correctly classified by model $f$ and a set of
allowed perturbed sentences $\mathcal{S}_{x}$, an _fickle adversarial example_
is defined as an input $\tilde{x}_{f}$ such that:

$\displaystyle\quad\textrm{1. }\tilde{x}_{f}\in\mathcal{S}_{x}$
$\displaystyle\quad\textrm{2. }f(\tilde{x}_{f})\neq f(x)$
$\displaystyle\quad\textrm{3. }\mathcal{O}(\tilde{x}_{f})=\mathcal{O}(x)$

There are many different methods for finding fickle adversarial examples. The
most common way is to use synonym word substitutions where the target words
are replaced with similar words found in the word embedding (Alzantot et al.,
2018; Jin et al., 2020) or use known synonyms from WordNet (Ren et al., 2019).
Recent work has also explored using masked language models to generate word
replacements (Li et al., 2020; Garg and Ramakrishnan, 2020; Li et al., 2021).

We adopt the synonym word substitution method as in Ye et al. (2020). For each
word $x_{i}$ in an input $x$, we create a synonym set $S_{x_{i}}$ containing
the synonym words of $x_{i}$ including itself. $\mathcal{S}_{x}$ is then
constructed by a set of sentences where each word in $x$ can be replaced by a
word in $S_{x_{i}}$. We consider the case where the attacker does not have a
constraint on the number of words that can be perturbed for each input,
meaning the attacker can perturb up to $L$ words which is the length of $x$.

The underlying assumption for fickle examples to work is that the perturbed
sentence $\tilde{x}_{f}\in S_{x}$ should have the same ground truth label as
the original input $x$, i.e. $\mathcal{O}(\tilde{x}_{f})=\mathcal{O}(x)=f(x)$.
However, common practice for constructing fickle examples does not guarantee
this is true. Swapping a word with its synonym may change the semantic meaning
of the example since even subtle changes in words can have a big impact on
meaning, and a word can have different meanings in different context. For
instance, “the whole race of human kind” and “the whole competition of human
kind” describe different things. Nonetheless, previous human evaluation has
shown that synonym-based adversarial examples still retain the same semantic
meaning and label as the original texts most of the time (Jin et al., 2020; Li
et al., 2020).

###  2.2 Obstinate Adversarial Examples

For a given input $(x,y)$ correctly classified by model $f$ and a set of
allowable perturbed sentences $\mathcal{A}_{x}$, an _obstinate adversarial
example_ is defined as an input $\tilde{x}_{o}$ such that:

$\displaystyle\quad\textrm{1. }\tilde{x}_{o}\in\mathcal{A}_{x}$
$\displaystyle\quad\textrm{2. }f(\tilde{x}_{o})=f(x)$
$\displaystyle\quad\textrm{3. }\mathcal{O}(\tilde{x}_{o})\neq\mathcal{O}(x)$

While it is challenging to construct obstinate adversarial examples
automatically for image classifiers (Tramer et al., 2020), we are able to
automate the process for NLP models. We use a similar antonym word
substitution strategy as proposed by Niu and Bansal (2018) to construct
obstinate adversarial examples. Similar to synonym word substitutions, for
each word $x_{i}$ in an input $x$, we construct an antonym set $A_{x_{i}}$
that consists of the antonyms of $x_{i}$. Since we would like to change the
semantic meaning of the input in a way that is likely to flip its label for
the task, the attacker is only allowed to perturb one word with its antonym
for each sentence.

The way we construct obstinate adversarial examples may not always satisfy the
assumption where the ground truth label of the obstinate example would be
different from the original input. The substituted word may not affect the
semantic meaning of the input depending on the task. For example, in natural
language inference, changing “the weather is great, we should go out and have
fun” to “the weather is bad, …” does not effect the entailment relationship
with “we should have some outdoor activities” since the main argument is in
the second part of the sentence. However, we find that antonym substitutions
are able to change the semantic meaning of the text most of the time and we
choose two tasks that are most likely to change the label under antonym-based
attack.

##  3 Robustness Tradeoffs

Normally, adversarial defense methods only target fickle adversarial examples,
so there is a risk that such methods increase vulnerability to obstinate
adversarial examples. According to the distance-oracle misalignment assumption
(Tramer et al., 2020) as depicted in Figure 1, the distance measure for
finding adversarial examples and labeling oracle $\mathcal{O}$ is misaligned
if we have $\mathcal{O}(\tilde{x}_{f})=\mathcal{O}(x)=y$ and
$\mathcal{O}(\tilde{x}_{o})\neq\mathcal{O}(x)$, but
$\mathit{dist}(x,\tilde{x}_{f})>\mathit{dist}(x,\tilde{x}_{o})$.

###  3.1 Setup

Our experiments are designed to test our hypothesis that optimizing
adversarial robustness of NLP models using only fickle examples deteriorates
the model’s robustness on obstinate adversarial examples. We use the SAFER
certified robust training method proposed by Ye et al. (2020). The idea is to
train a smoother model by randomly perturbing the sentences with words in the
synonym substitution set at each training iteration. While common IBP-based
certified robust training methods do not scale well onto large pre-trained
language models (Jia et al., 2019; Huang et al., 2019), SAFER is a structure-
free approach that can be applied to any kind of model architectures. In
addition, it gives stronger robustness than traditional adversarial training
method (Yoo and Qi, 2021).

We train BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) models on
two different tasks with SAFER training for 15 epochs. We then test the attack
success rate for both fickleness and obstinacy attacks at each training epoch.
We use the same perturbation method as described in Section 2.1 for both the
training and the attack. For each word, the synonym perturbation set is
constructed by selecting the top $k$ nearest neighbors with a cosine
similarity constraint of 0.8 in GLOVE embeddings (Pennington et al., 2014),
and the antonym perturbation set consists of antonym words found in WordNet
(Miller, 1995). We follow the method of Jin et al. (2020) for finding fickle
adversarial examples by using word importance ranking and Part-of-Speech (PoS)
and sentence semantic similarity constraints as the search criteria. We
replace words from the ones with the highest word importance scores to the
ones with the least and make sure the new substituted words have the same PoS
tags as the original words. For antonym attack, we also use word importance
ranking and PoS to search for word substitutions. For comparison, we set up
baseline models with normal training on the original training sets.

###  3.2 Tasks

We choose two different tasks from the GLUE benchmark (Wang et al., 2018) that
are good candidates for the antonym attack. Antonym-based attacks work well on
these tasks since both tasks consist of sentence pairs and changing a word to
an opposite meaning is likely to break the relationship between the pairs.

Natural Language Inference. We experiment with Multi-Genre Natural Language
Inference (MNLI) dataset (Williams et al., 2018) which contains a premise-
hypothesis pair for each example. The task is to identify the relation between
the sentences in a premise-hypothesis pair and determine whether the
hypothesis is true (_entailment_), false (_contradiction_) or undetermined
(_neutral_) given the premise. We consider the case where both premise and
hypothesis can be perturbed, but only one word from either premise or
hypothesis can be substituted for antonym attack. We exclude examples with a
_neutral_ label when constructing obstinate adversarial examples since antonym
word substitutions may not change their label to a different class.

Paraphrase Identification. We use Quora Question Pairs (QQP) (Iyer et al.,
2017) which consists of questions extracted from Quora. The goal of the task
is to identify duplicate questions. Each question pair is labeled as duplicate
or non-duplicate. For our antonym attack strategy, we only target the
duplicate class since antonym word substitutions are unlikely to flip an
initially non-duplicate pair into a duplicate.

We also conducted experiments using the Wiki Talk Comments (Wulczyn et al.,
2017) dataset, a dataset for toxicity detection, by adding or removing toxic
words for creating obstinate examples. However, we found adding toxic words
can reach almost 100% attack success rate, so there did not seem to be an
interesting tradeoff to explore for available models for this task, and we do
not include it in our results.

(a) MNLI (BERT)

(b) MNLI (RoBERTa)

Figure 3: Fickleness and obstinacy tradeoff where fickleness attack success
rate increases as obstinacy attack success rate decreases. The figure shows
the results on MNLI matched validation set with average and standard deviation
across three different runs. Dash lines show the synonym/antonym attack
success rate on baseline model with normal training.

(a) MNLI (BERT)

(b) MNLI (RoBERTa)

Figure 4: The synonym and antonym attack success rate at each SAFER training
epoch with varying batch size. When the model is trained with smaller batch
size, the synonym attack success rate is lower and the antonym success rate is
higher.

###  3.3 Results

We visualize the attack success rates for fickleness (synonym attack) and
obstinacy (antonym attack) attacks in Figure 3. The results are consistent
with our hypothesis that optimizing adversarial robustness of NLP models using
only fickle examples can result in models that are more vulnerable to
obstinacy attacks. Robustness training for the BERT model on MNLI improves
fickleness robustness, reducing the synonym attack success rate from 36% to
11% (a 69% decrease) after training for 15 epochs (3(a)), but antonym attack
success rate increases from 56% to 63% (a 13% increase). The antonym attack
success rate increases even more for the RoBERTa model (3(b)), increasing from
56% to 67% (a 20% increase) while the synonym attack success rate decreases
from 31.2% to 10% (a 68% decrease). The RoBERTa model is pre-trained to be
more robust than the BERT model with dynamic masking, which perhaps explains
the difference. We observe a robustness tradeoff for QQP dataset as well (see
Appendix A.1). In addition, the fickle adversarial training does not sacrifice
the performance on the original examples and increases consistently throughout
training (see Figure 9 in the appendix).

Impact of Batch Size. We experiment with different batch sizes for fickle-
based robust training. Figure 4 shows the results on MNLI dataset. When the
model is trained with a smaller batch size, the synonym attack success rate
becomes lower, but the antonym success rate gets higher. This means that the
model may overfit on the fickle examples due to smaller training batch size,
exacerbating the impact of the unbalanced adversarial training. Similar
observation is found on the QQP task (see Figure 8 in the appendix). We found
similar evidence on the evaluation accuracy on the original validation set
(see Figure 9 in the appendix). While models with smaller batch sizes converge
faster, they lead to lower performance and poorer generalization. In Appendix
C.4, we show that our proposed method is not affected by the training batch
size.

##  4 Balanced Adversarial Training

In previous section, we argued that the tradeoff between fickleness and
obstinacy can be attributed to distance-oracle misalignment. This section
proposes and evaluates a modification to adversarial training that balances
both kinds of adversarial examples.

Model | Method | Eval Acc (%) | Antonym ASR (%) | Synonym ASR (%)
---|---|---|---|---
| Normal Training | 84.01 $\pm 0.32$ | 57.48 $\pm 0.14$ | 36.63 $\pm 0.34$
| A2T | 84.44 | 56.51 | 21.67
BERT | SAFER | 83.81 $\pm 0.40$ | 63.33 $\pm 1.55$ | 10.61 $\pm 0.17$
| BAT-Pairwise | 84.03 $\pm 0.21$ | 35.47 $\pm 1.51$ | 26.70 $\pm 0.78$
| BAT-Triplet | 84.66 $\pm 0.12$ | 34.25 $\pm 2.19$ | 25.70 $\pm 0.13$
| Normal Training | 87.74 $\pm 0.09$ | 55.86 $\pm 0.47$ | 30.86 $\pm 0.50$
| A2T | 86.98 | 56.84 | 19.78
RoBERTa | SAFER | 86.18 $\pm 0.18$ | 67.15 $\pm 2.17$ | 10.00 $\pm 0.34$
| BAT-Pairwise | 87.23 $\pm 0.38$ | 38.68 $\pm 1.61$ | 27.29 $\pm 0.20$
| BAT-Triplet | 87.44 $\pm 0.12$ | 33.02 $\pm 0.81$ | 27.54 $\pm 0.45$
Table 1: Balanced Adversarial Training evaluation results on MNLI matched
validation set. Results shown with standard deviations are average across
three different runs.

###  4.1 Approach

The most intuitive way to make the semantic distance in the representation
space align better with human perception is to move the fickle example closer
to the original input and push the original input apart from the obstinate
example in the representation space. This goal matches the objective of
contrastive learning, a type of self-supervised learning that learns
representations with positive (similar) examples close together and negative
(dissimlar) examples far apart (Hadsell et al., 2006; Schroff et al., 2015).
Positive examples are usually generated with data augmentation such as spatial
transformation, and negative examples are sampled from other examples (Chen et
al., 2020).

We adapt contrastive learning to balance adversarial training by treating
fickle adversarial examples as positive examples and obstinate adversarial
examples as negative examples. The idea is to minimize the distance between
the positive pairs and maximize the distance between the negative pairs. We
construct positive pairs by pairing the original input with a corresponding
fickle example, and negative pairs as the original input paired with an
obstinate example. We generate fickle examples by applying synonym
transformations, and obstinate examples by applying antonym transformations.

We combine normal training with a contrastive learning objective and
experiment with two different approaches for contrastive loss: pairwise and
triplet loss. While recent contrastive learning incorporates multiple positive
and negative examples for each input, we use these two methods as they
consider the simplest case where only a positive and a negative example is
needed for each input. Similarly to SAFER certified robust training, we use an
augmented approach without querying the model to check if the attack succeeds.
We choose this approach over traditional adversarial training since it is
computationally less expensive.

Given an input $(x,y)$, we generate an example $\tilde{x}_{o}$ by applying
synonym perturbations and an example $\tilde{x}_{u}$ by applying antonym
perturbations. Let $d(x_{1},x_{2})$ denote the distance measure between
$x_{1}$ and $x_{2}$ in the representation space.

BAT-Pairwise. For the pairwise approach, we independently optimize the
distance for the fickle pair $(x,\tilde{x}_{f})$ and the obstinate pair
$(x,\tilde{x}_{o})$:

$\displaystyle\mathcal{L}^{\mathit{BAT}_{\mathit{pair}}}$
$\displaystyle=\mathcal{L}_{\mathit{ML}}+\mathcal{L}_{\mathit{pair}}$
$\displaystyle\mathcal{L}_{\mathit{ML}}$ $\displaystyle=\log f(y\;|\;x)$
$\displaystyle\mathcal{L}_{\mathit{pair}}$ $\displaystyle=\alpha
d(x,\tilde{x}_{f})+\beta\max(0,m-d(x,\tilde{x}_{o}))$

The hyperparameters $\alpha$ and $\beta$ control the weighting of the fickle
and obstinate pairs, and $m$ is the margin. The $\mathcal{L}_{pair}$ loss term
is designed to minimize the distance to the fickle adversarial example and
maximize the distance to the obstinate adversarial example. The margin $m$
penalizes the model when the obstinate example within $m$ distance of the
original input ($d(x,\tilde{x}_{o})<m$). We use cosine similarity for distance
measure (ranges from 0 to 1), and set the margin as 1 as we find it gives the
best performance (see Appendix C.2.2). For the case where we are unable to
find a valid fickle and obstinate adversarial example, we set the
corresponding term, either $d(x,\tilde{x}_{f})$ or $m-d(x,\tilde{x}_{o})$, to
0.

BAT-Triplet. For the triplet approach, the original input $x$ acts as an
anchor and a triplet, $(x,\tilde{x}_{f},\tilde{x}_{o})$, is considered instead
of pairs. The triplet loss aims to make the distance between the obstinate
pair larger than the distance between the fickle pair, with at least a margin
$m$: $d(x,\tilde{x}_{o})>d(x,\tilde{x}_{f})+m$. The training loss can be
formalized as:

$\displaystyle\mathcal{L}^{\mathit{BAT}_{\mathit{triplet}}}$
$\displaystyle=\mathcal{L}_{ML}+\lambda\mathcal{L}_{\mathit{triplet}}$
$\displaystyle\mathcal{L}_{\mathit{triplet}}$
$\displaystyle=\max(0,d(x,\tilde{x}_{f})+(m-d(x,\tilde{x}_{o})))$

where the hyperparameter $\lambda$ controls the weight of the contrastive loss
term. Similarly to the pairwise loss, if no fickle and obstinate example is
available, we mask out $d(x,\tilde{x}_{f})$ or $m-d(x,\tilde{x}_{o})$ in
$\mathcal{L}_{\mathit{triplet}}$.

We show the training details and how we find the best hyperparameters in
Appendix C.2.

Model Method Eval Acc (%) F1 Antonym ASR (%) Synonym ASR (%) Normal Training
90.78 $\pm 0.12$ 87.67 $\pm 0.12$ 44.94 $\pm 1.24$ 21.16 $\pm 0.36$ A2T 90.70
87.52 36.27 14.71 BERT SAFER 90.85 $\pm 0.19$ 87.65 $\pm 0.23$ 50.10 $\pm
0.44$ 4.83 $\pm 0.15$ BAT-Pairwise 90.31 $\pm 0.03$ 86.97 $\pm 0.09$ 22.46
$\pm 1.08$ 16.00 $\pm 0.73$ BAT-Triplet 90.82 $\pm 0.11$ 87.79 $\pm 0.04$
16.24 $\pm 1.41$ 15.73 $\pm 0.09$ Normal Training 91.18 $\pm 0.09$ 88.29 $\pm
0.12$ 40.52 $\pm 0.71$ 18.9 $\pm 0.12$ A2T 91.14 88.04 42.42 13.04 RoBERTa
SAFER 91.26 $\pm 0.10$ 88.27 $\pm 0.15$ 46.07 $\pm 1.44$ 4.87 $\pm 0.45$ BAT-
Pairwise 90.19 $\pm 0.29$ 86.94 $\pm 0.40$ 15.29 $\pm 2.62$ 16.58 $\pm 0.44$
BAT-Triplet 91.04 88.21 13.02 16.89

Table 2: Balanced Adversarial Training evaluation results on QQP validation
set.

###  4.2 Results

Table 1 shows BAT training results on the MNLI validation sets. We use normal
training as the non-robust baseline, and include two robust baselines:
certified robust training (SAFER), and traditional adversarial training (A2T)
(Yoo and Qi, 2021). Balanced Adversarial Training increases the model’s
adversarial robustness against both antonym and synonym attacks, while
preserving its performance on the original validation set.

While both of the robust baselines (SAFER and A2T), which only consider fickle
adversarial examples, perform best when evaluated solely based on fickleness
robustness, they are more vulnerable to obstinate adversarial examples. We
found that BAT-Triplet performs better than BAT-Pairwise in terms of improving
robustness against antonym attacks. With BAT-Triplet, the antonym attack
success rate on BERT decreases from 57% to 34% (a 40% decrease) comparing to
normal training, and the synonym atack success rate decreases from 36% to 26%
(a 28% decrease).

Results for the QQP dataset are shown in Table 2. While the antonym attack
success rates drop more than half (around 67% decrease) after BAT training,
the synonym attack success rate has a 24% decrease on BERT and only 10% on
RoBERTa, as the synonym attack success rate is already low on the model with
normal training.

###  4.3 Representation Analysis

We compare the learned representations of models trained with BAT to normal
training and SAFER. We sample 500 examples from MNLI dataset (excluding the
neutral class) and apply synonym and antonym perturbations for each input. We
then project the model representations before the last classification layer to
2 dimensional space with t-SNE (van der Maaten and Hinton, 2008) and visualize
the results in Figure 5.

When training with normal training or SAFER, we can see that both fickle and
obstinate adversarial examples are fairly close to the original examples.
However, with BAT-Pairwise or BAT-Triplet, obstinate examples are pushed
further away from both original and fickle examples. This matches with BAT’s
training goal where the distance between obstinate and original examples is
maximized and the distance between fickle and original examples is minimized.
This also shows how BAT is able to fix the distance-oracle misalignment,
making the semantic distance in the representation space aligns better with
human perception, and further improve robustness against both types of
adversarial examples.

(a) Normal training

(b) SAFER

(c) BAT-Pairwise

(d) BAT-Triplet

Figure 5: 2D projection of model representation for RoBERTa MNLI models
trained with normal training, certified robust training with fickle
adversarial examples (SAFER), BAT-Pairwise, and BAT-Triplet.

##  5 Related Work

Compared to fickle adversarial examples, obstinacy has been less studied in
NLP as well as other domains. Feng et al. (2018) delete words iteratively from
the input to create examples that appear rubbish to human but retain the
model’s prediction with high confidence. Welbl et al. (2020) use Part-of-
Speech and Name Entity based perturbations against reading comprehension
models. Niu and Bansal (2018) study both types of attack strategies for
dialogue models. They create obstinate adversarial examples by substituting
words with antonyms or adding negation words to the input.

Our work is the first to study tradeoffs between fickle and obstinate
adversarial examples in NLP, but a few previous works have considered these
tradeoffs in the vision domain. Jacobsen et al. (2019b) show that adversary
can not only target the model’s excessive sensitivity but its excessive
invariance to small changes in the input. They propose an alternative training
objective based on information theory to make the model less invariant to
semantically meaningful changes. Tramer et al. (2020) study the tradeoff
between the two types of adversarial examples for image classifiers. They show
that data augmentation can help increase robustness against obstinacy attacks,
but is not sufficient to impede both types of attacks. Our work differs in
that we propose a new adversarial training method that improves model
robustness against both types of adversarial examples. In addition, unlike
images where human inspection is usually required to check whether the
perturbed pixels would change the true label of the image, we are able to
automate the process of generating obstinate examples for text.

Recent work introduce contrastive learning for image classifiers in the
adversarial learning setting where an fickle adversarial augmentation is used
to generate positive examples and negative examples are sampled from other
images. Kim et al. (2020) generate diverse positive examples by launching
instance-wise attack on augmented images and show that it improves model’s
fickleness robustness. Ho and Nvasconcelos (2020) create challenging positive
pairs by using the gradients of the contrastive loss to generate fickle
adversarial examples and they show that it improves model performance.

##  6 Conclusion

We demonstrate the tradeoff between vulnerability to synonym-based (fickle)
and antonym-based (obstinate) adversarial examples for NLP models and show
that increasing robustness against synonym based attacks also increases
vulnerability to antonym-based attacks. To manage this tension, we introduce a
new adversarial training method, BAT, which targets the distance-oracle
misalignment problem and can help balance the fickleness and obstinacy in
adversarial training.

## Limitations

We showed robustness tradeoffs exist between synonym and antonym-based
adversarial examples. Since there are numerous ways to construct adversarial
examples for NLP models, further investigation is needed to show if this holds
true for any kind of fickleness and obstinacy attacks for NLP models and we
will leave it for future work. In order to launch antonym attack
automatically, we are also limited to sentence pair tasks that are more prone
to changing the ground truth label when replacing a word with its antonym. In
addition, BAT sacrifices the performance on the synonym-based attack success
rate for robustness to antonym-based attack when comparing to fickle
adversarial training methods. We show that there is a tradeoff between
robustness against synonym and antonym based attacks and our goal is to
achieve a better tradeoff between them.

## Availability

Code for reproducing our experiments is available at:
https://github.com/hannahxchen/balanced-adversarial-training.

## Acknowledgements

This work was partially supported by a grant from the National Science
Foundation (#1804603).

## References

  * Alzantot et al. (2018) Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai-Wei Chang. 2018.  Generating natural language adversarial examples.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_.
  * Chen et al. (2020) Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. 2020.  A simple framework for contrastive learning of visual representations.  _CoRR_ , abs/2002.05709.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_.
  * Feng et al. (2018) Shi Feng, Eric Wallace, Alvin Grissom II, Mohit Iyyer, Pedro Rodriguez, and Jordan Boyd-Graber. 2018.  Pathologies of neural models make interpretations difficult.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_.
  * Garg and Ramakrishnan (2020) Siddhant Garg and Goutham Ramakrishnan. 2020.  BAE: BERT-based adversarial examples for text classification.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
  * Goodfellow et al. (2016) Ian J. Goodfellow, Nicolas Papernot, and Patrick D. McDaniel. 2016.  cleverhans v0.1: an adversarial machine learning library.  _CoRR_ , abs/1610.00768.
  * Goodfellow et al. (2015) Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015.  Explaining and harnessing adversarial examples.  _CoRR_ , abs/1412.6572.
  * Hadsell et al. (2006) R. Hadsell, S. Chopra, and Y. LeCun. 2006.  Dimensionality reduction by learning an invariant mapping.  In _2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06)_ , volume 2.
  * Ho and Nvasconcelos (2020) Chih-Hui Ho and Nuno Nvasconcelos. 2020.  Contrastive learning with adversarial examples.  In _Advances in Neural Information Processing Systems_ , volume 33.
  * Huang et al. (2019) Po-Sen Huang, Robert Stanforth, Johannes Welbl, Chris Dyer, Dani Yogatama, Sven Gowal, Krishnamurthy Dvijotham, and Pushmeet Kohli. 2019.  Achieving verified robustness to symbol substitutions via interval bound propagation.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_.
  * Iyer et al. (2017) Shankar Iyer, Nikhil Dandekar, and Kornél Csernai. 2017.  First quora dataset release: Question pairs.  https://www.quora.com/q/quoradata/First-Quora-Dataset-Release-Question-Pairs.
  * Jacobsen et al. (2019a) Joern-Henrik Jacobsen, Jens Behrmann, Nicholas Carlini, Florian Tramèr, and Nicolas Papernot. 2019a.  Exploiting excessive invariance caused by norm-bounded adversarial robustness.  _Safe Machine Learning workshop at ICLR_.
  * Jacobsen et al. (2019b) Joern-Henrik Jacobsen, Jens Behrmann, Richard Zemel, and Matthias Bethge. 2019b.  Excessive invariance causes adversarial vulnerability.  In _International Conference on Learning Representations_.
  * Jia et al. (2019) Robin Jia, Aditi Raghunathan, Kerem Göksel, and Percy Liang. 2019.  Certified robustness to adversarial word substitutions.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_.
  * Jin et al. (2020) Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. 2020.  Is bert really robust? a strong baseline for natural language attack on text classification and entailment.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 34(05).
  * Kim et al. (2020) Minseon Kim, Jihoon Tack, and Sung Ju Hwang. 2020.  Adversarial self-supervised contrastive learning.  In _Advances in Neural Information Processing Systems_ , volume 33.
  * Kumar and Boulanger (2020) Vivekanandan S. Kumar and David Boulanger. 2020.  Explainable automated essay scoring: Deep learning really has pedagogical value.  In _Frontiers in Education_.
  * Li et al. (2021) Dianqi Li, Yizhe Zhang, Hao Peng, Liqun Chen, Chris Brockett, Ming-Ting Sun, and Bill Dolan. 2021.  Contextualized perturbation for textual adversarial attack.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_.
  * Li et al. (2020) Linyang Li, Ruotian Ma, Qipeng Guo, Xiangyang Xue, and Xipeng Qiu. 2020.  BERT-ATTACK: Adversarial attack against BERT using BERT.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  RoBERTa: A robustly optimized BERT pretraining approach.  _CoRR_ , abs/1907.11692.
  * Madry et al. (2018) Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2018.  Towards deep learning models resistant to adversarial attacks.  In _International Conference on Learning Representations_.
  * Miller (1995) George A. Miller. 1995.  WordNet: A lexical database for English.  _Communications of the ACM_ , 38(11).
  * Morris et al. (2020) John Morris, Eli Lifland, Jack Lanchantin, Yangfeng Ji, and Yanjun Qi. 2020.  Reevaluating adversarial examples in natural language.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_.
  * Niu and Bansal (2018) Tong Niu and Mohit Bansal. 2018.  Adversarial over-sensitivity and over-stability strategies for dialogue models.  In _Proceedings of the 22nd Conference on Computational Natural Language Learning_.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014.  GloVe: Global vectors for word representation.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
  * Ren et al. (2019) Shuhuai Ren, Yihe Deng, Kun He, and Wanxiang Che. 2019.  Generating natural language adversarial examples through probability weighted word saliency.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_.
  * Schroff et al. (2015) Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015.  FaceNet: A unified embedding for face recognition and clustering.  In _2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Tramer et al. (2020) Florian Tramer, Jens Behrmann, Nicholas Carlini, Nicolas Papernot, and Joern-Henrik Jacobsen. 2020.  Fundamental tradeoffs between invariance and sensitivity to adversarial perturbations.  In _Proceedings of the 37th International Conference on Machine Learning_ , volume 119 of _Proceedings of Machine Learning Research_.
  * van der Maaten and Hinton (2008) Laurens van der Maaten and Geoffrey Hinton. 2008.  Visualizing data using t-SNE.  _Journal of Machine Learning Research_ , 9(86).
  * Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_.
  * Welbl et al. (2020) Johannes Welbl, Pasquale Minervini, Max Bartolo, Pontus Stenetorp, and Sebastian Riedel. 2020.  Undersensitivity in neural reading comprehension.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_.
  * Wulczyn et al. (2017) Ellery Wulczyn, Nithum Thain, and Lucas Dixon. 2017.  Ex Machina: Personal attacks seen at scale.  In _Proceedings of the 26th International Conference on World Wide Web_.
  * Ye et al. (2020) Mao Ye, Chengyue Gong, and Qiang Liu. 2020.  SAFER: A structure-free approach for certified robustness to adversarial word substitutions.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_.
  * Yoo and Qi (2021) Jin Yong Yoo and Yanjun Qi. 2021.  Towards improving adversarial training of NLP models.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_.

##  Appendix A Fickleness and Obstinacy Robustness Tradeoffs

###  A.1 Synonym and Antonym Attack Robustness Tradeoffs

(a) QQP (BERT)

(b) QQP (RoBERTa)

(c) MRPC (BERT)

(d) SNLI (BERT)

Figure 6: Robustness tradeoffs between synonym and antonym based attacks on
QQP and MRPC dataset. The figure shows the average and standard deviation
across 3 different runs.

###  A.2 Synonym and Negation Attack Robustness Tradeoffs

We test how negation attack success rate would change as the model robustness
against synonym attack increases. For negation attack, we add negation to a
verb in the sentence, i.e., “I can do it” to “I can’t do it”, or remove
negation from a sentence, i.e., “I am not going” to “I am going”. We follow
similar setup as in Section 3. We found that there exists a tradeoff between
synonym-based adversarial examples and negation-based adversarial examples on
QQP task, but found no significant tradeoff on MNLI task, as shown in Figure
7.

(a) MNLI (BERT)

(b) MNLI (RoBERTa)

(c) QQP (BERT)

(d) QQP (RoBERTa)

Figure 7: Negation attack success rate on models at each epoch when training
with SAFER.

##  Appendix B Fickleness Robust Training with Varying Batch Size

###  B.1 Synonym and Antonym Robustness Tradeoffs on QQP Task

(a) QQP (BERT)

(b) QQP (RoBERTa)

Figure 8: The synonym and antonym attack success rate at each SAFER training
epoch with varying batch size.

###  B.2 Evaluation Accuracy at Each SAFER Training Epoch

(a) MNLI (BERT)

(b) MNLI (RoBERTa)

(c) QQP (BERT)

(d) QQP (RoBERTa)

Figure 9: The evaluation accuracy on original validation set at each SAFER
training epoch with varying batch size.

##  Appendix C Balanced Adversarial Training

###  C.1 Balanced Adversarial Training Details

We implement BAT similarly to the SAFER training method as described in
Section 3.1 where we randomly perturb the inputs with words from the
synonym/antonym substitution sets.

We train the BERT and RoBERTa models for 2 or 3 epochs with a learning rate of
$2\times 10^{-5}$ or $3\times 10^{-5}$ and batch size of 32. For contrastive
loss weights and margin in BAT-Pairwise and BAT-Triplet, we perform
hyperparmeter search and choose the ones with best performance (see Appendix
C.2).

###  C.2 BAT Hyperparameter Search

####  C.2.1 Contrastive Loss Weights

In Figure 10, we show varying fickle loss weights $(\alpha)$ with obstinate
loss weight fixed $(\beta=1.0)$ and vice versa when training with BAT-Pairwise
method. As the value of $\alpha$ increases, antonym attack success rate
increases. On the other hand, as the value of $\beta$ increases, synonym
attack success rate has a small increase as well. We found $\alpha=1.0$ and
$\beta=1.2$ gives the best performance for BERT MNLI model. We test different
contrastive loss weights ($\lambda$) when training with BAT-Triplet and show
the results in Figure 11. We found that as we increase $\lambda$, model
accuracy on the validation set decreases.

(a) Varying $\beta$ with $\alpha=1.0$

(b) Varying $\alpha$ with $\beta=1.0$

Figure 10: Performance of BERT models trained on MNLI tasks with different
fickle ($\alpha$) and obstinate ($\beta$) loss weights in BAT-Pairwise. Figure
11: Performance of BERT models trained on MNLI tasks with different
contrastive loss weights ($\lambda$) in BAT-Triplet.

####  C.2.2 Margin

We test varying margin $m$ from 0.3 to 1.0 as cosine similarity ranges between
0 to 1. In Figure 12, we can see that as margin approaches 1, both synonym and
antonym attack success rates decrease. Model achieves best performance when
margin is closer to 1. We also found that margin has larger effect on BAT-
Triplet.

(a) BAT-Triplet

(b) BAT-Pairwise

Figure 12: Performance of BERT models trained on MNLI tasks with different
margin ($m$).

###  C.3 BAT Evaluation on MNLI Mismatched Validation Set

Model | Method | Eval Acc (%) | Antonym ASR (%) | Synonym ASR (%)
---|---|---|---|---
| Normal Training | 84.28 $\pm 0.51$ | 58.59 $\pm 0.28$ | 41.15 $\pm 0.45$
| A2T | 85.00 | 57.86 | 24.84
BERT | SAFER | 83.73 $\pm 0.12$ | 65.13 $\pm 1.64$ | 12.59 $\pm 0.78$
| BAT-Pairwise | 84.52 $\pm 0.06$ | 36.71 $\pm 1.31$ | 29.63 $\pm 0.95$
| BAT-Triplet | 84.99 $\pm 0.06$ | 34.55 $\pm 2.52$ | 29.02 $\pm 0.17$
| Normal Training | 87.49 $\pm 0.05$ | 57.66 $\pm 0.95$ | 34.56 $\pm 0.16$
| A2T | 86.52 | 58.19 | 21.07
RoBERTa | SAFER | 86.26 $\pm 0.23$ | 69.52 $\pm 1.76$ | 11.12 $\pm 0.22$
| BAT-Pairwise | 87.16 $\pm 0.41$ | 39.34 $\pm 2.09$ | 30.58 $\pm 0.28$
| BAT-Triplet | 87.19 $\pm 0.18$ | 33.60 $\pm 0.78$ | 30.19 $\pm 0.91$
Table 3: Balanced Adversarial Training evaluation results on MNLI mismatched
validation set.

###  C.4 BAT-Triplet with Varying Batch Size

In Section 3.3, we observed that certified robust training with smaller batch
sizes results in larger gaps in robustness tradeoff. We test BAT-Triplet with
varying batch size when training BERT on MNLI task and we find that it gives
consistent improvement on robustness regardless of the batch size, as shown in
Table 4.

Batch | Accuracy | Antonym | Synonym
---|---|---|---
Size | (%) | ASR (%) | ASR (%)
8 | 84.45 | 34.59 | 25.66
16 | 84.08 | 31.05 | 25.89
32 | 84.70 | 32.15 | 25.83
Table 4: BAT-Triplet with BERT training, varying batch size, evaluated on MNLI
matched validation set.

###  C.5 Additional BAT Results

Model | Method | Eval Acc (%) | F1 | Antonym ASR (%) | Synonym ASR (%)
---|---|---|---|---|---
| Normal Training | 85.37 $\pm 0.42$ | 89.66 $\pm 0.30$ | 67.79 $\pm 1.71$ | 8.23 $\pm 0.26$
BERT | SAFER | 85.62 $\pm 0.46$ | 90.01 $\pm 0.38$ | 77.19 $\pm 4.28$ | 3.24 $\pm 0.34$
| BAT-Pairwise | 85.54 $\pm 0.40$ | 89.68 $\pm 0.46$ | 44.34 $\pm 3.96$ | 6.02 $\pm 0.43$
| BAT-Triplet | 85.13 $\pm 0.31$ | 89.50 $\pm 0.20$ | 45.22 $\pm 0.63$ | 6.62 $\pm 0.25$
Table 5: Balanced Adversarial Training evaluation results on MRPC validation set. Model | Method | Eval Acc (%) | Antonym ASR (%) | Synonym ASR (%)
---|---|---|---|---
| Normal Training | 90.82 $\pm 0.33$ | 69.90 $\pm 0.23$ | 27.46 $\pm 0.18$
BERT | SAFER | 90.00 $\pm 0.24$ | 72.10 $\pm 0.56$ | 4.21 $\pm 0.07$
| BAT-Pairwise | 90.58 $\pm 0.31$ | 54.01 $\pm 1.15$ | 15.10 $\pm 0.04$
| BAT-Triplet | 90.67 $\pm 0.39$ | 44.19 $\pm 1.31$ | 15.49 $\pm 0.37$
Table 6: Balanced Adversarial Training evaluation results on SNLI validation
set.


# Structure Preserving PINN for Solving Time Dependent PDEs with Periodic
Boundary

Baoli Haoa, Ulisses Braga-Netob, Chun Liua, Lifan Wangc, Ming Zhonga
a Department of Applied Mathematics, Illinois Institute of Technology,
Chicago, IL, USA
b Department of Electrical and Computing Engineering, Texas A$\&$M University,
College Station, TX, USA
b Department of Physics and Astrophysics, Texas A$\&$M University, College
Station, TX, USA

###### Abstract

We present a structure preserving PINN for solving a series of time dependent
PDEs with periodic boundary. Our method can incorporate the periodic boundary
condition as the natural output of any deep neural net, hence significantly
improving the training accuracy of baseline PINN. Together with mini-batching
and other PINN variants (SA-PINN, RBA-PINN, etc.), our structure preserving
PINN can even handle stiff PDEs for modeling a wide range of convection-
diffusion and reaction-diffusion processes. We demonstrate the effectiveness
of our PINNs on various PDEs from Allen Cahn, Gray Scott to nonlinear
Schrödinger.

Keywords: Physics-Informed Neural Network; Reaction-Diffusion PDE; Convection-
Diffusion PDE; Structure Preserving.

##  1 Introduction

Time-dependent partial differential equations (PDEs) hold a ubiquitous
presence across numerous scientific and engineering models, underpinning
various physical phenomena (Whitham, 2011; Burgers, 1948; Allen and Cahn,
1979; Shen and Yang, 2010; Bazant, 2017; Horstmann et al., 2013; Tian et al.,
2015; Hyman et al., 2014; Takatori and Brady, 2015; Cahn and Hilliard, 1958;
Lee et al., 2014; Miranville, 2017; Kim et al., 2016; Kudryashov, 1990;
Michelson, 1986; De Kepper et al., 1991; Nishiura and Ueyama, 1999;
Zhabotinsky, 2007; Cassani et al., 2021; Kato, 1987; Kevrekidis et al., 2001;
Fibich, 2015). Particularly, when coupled with periodic boundary conditions,
these PDEs become instrumental in modeling phenomena on infinite physical
domains. However, the task of solving such PDEs often poses significant
challenges. In response to these challenges, we propose an innovative series
of machine learning-based solvers tailored for handling time-dependent PDEs,
with a specific emphasis on forward evolution of the data. The utilization of
machine learning methodologies has demonstrated notable success in various
scientific endeavors, ranging from the intricate process of protein folding
Jumper et al. (2021), new drug discoveries Carracedo-Reboredo et al. (2021),
to expediting matrix multiplication Fawzi et al. (2022). Traditional data-
driven machine learning approaches rely heavily on vast amounts of training
data, sometimes yielding models that lack interpretability due to their
detachment from the physical understanding of the problem domain. Moreover, in
numerous scientific domains, acquiring high-fidelity data proves to be both
expensive and time-consuming. The emerging field of physics-informed machine
learning presents a promising avenue to address these challenges. By
integrating constraints derived from physical laws, physics-informed machine
learning enables the prediction of complex system behaviors using sparse data
Yang et al. (2019); Y. Chen et al. (2021); Raissi and Karniadakis (2018);
Raissi et al. (2019); Liu and Wang (2019, 2021); Jin et al. (2021); Cai et al.
(2021); Rad et al. (2020); Zhu et al. (2021); X. Chen et al. (2022); R. Wang
et al. (2023). However, the successful training of physics-informed machine
learning models remains a formidable task S. Wang, Teng, and Perdikaris
(2020); S. Wang, Yu, and Perdikaris (2020); McClenny and Braga-Neto (2020);
Coutinho et al. (2023). Fundamental challenges include the difficulty in
propagating information away from data, particularly in scenarios involving
stiff PDEs.

In this paper, we introduce a novel algorithm designed to preserve the
structural integrity of physics-informed neural networks (PINNs). Our approach
involves embedding information regarding initial and boundary conditions
directly into the neural network architecture, thereby reducing the reliance
on spectral bases in training various types of losses S. Wang, Yu, and
Perdikaris (2020). This architectural modification not only accelerates the
training process but also enhances the accuracy of solutions derived from
PINNs.

Our key insight lies in recognizing that collocation-based machine learning
solvers, utilized for training PINNs, can be viewed as a specialized form of
regularized regression. In this framework, the initial and boundary
conditions, along with observed data points, serve as the basis for training,
while collocation points act as regularization data. However, the multi-
objective nature of the training loss often poses challenges in finding global
minimizers. Furthermore, the solution manifolds of most time-dependent PDEs
exhibit sensitivity to initial and boundary conditions, where slight
variations may yield vastly different outcomes. By integrating initial and
boundary condition data into the neural network structure—thus preserving the
underlying problem structure—we simplify the PINN training process,
particularly for stiff time-dependent PDEs. Moreover, our approach can be
augmented with existing training enhancement techniques, such as self-adaptive
weights McClenny and Braga-Neto (2020), causal PINNs S. Wang, Yu, and
Perdikaris (2020), and time-marching PINNs Wight and Zhao (2020), Residual-
based Attention PINN Anagnostopoulos et al. (2024), co-training PINN Zhong et
al. (2024), further refining prediction accuracy.

####  1.0.1 Related Work

The foundational PINN algorithm, while demonstrating remarkable efficacy
across numerous applications, occasionally exhibits shortcomings in accurately
approximating solutions or achieving convergence, particularly when applied to
“stiff” PDEs, where a small time scale is required in order for traditional
numerical algorithms to stay stable. Extensive research indicates that such
challenges stem from an imbalance within the PINN loss function between its
data-fitting and residual components (Wight and Zhao, 2020; Shin et al., 2020;
S. Wang, Teng, and Perdikaris, 2020; S. Wang, Yu, and Perdikaris, 2020).
Notably, gradient descent tends to prioritize minimizing the residual loss
over the data-fitting component, impeding convergence to the correct solution.
This issue is particularly pronounced in forward problems, where all available
data are confined to initial and boundary conditions, with scant information
within the PDE domain. For instance, in time-evolution scenarios, neural
network training struggles to propagate information from initial conditions to
subsequent time steps, as corroborated by several researchers Wight and Zhao
(2020); McClenny and Braga-Neto (2020); Krishnapriyan et al. (2021); S. Wang
et al. (2022); Haitsiukevich and Ilin (2022).

In addressing these challenges, Wight and Zhao (2020) proposed a method to
propagate information forward in time by segmenting the time axis into smaller
intervals, sequentially training PINNs on each segment, starting from the
interval closest to the initial condition. However, this approach is time-
intensive due to the necessity of training multiple PINNs. Furthermore, both
Wight and Zhao (2020) and McClenny and Braga-Neto (2020) emphasized the
importance of weighting initial condition data and residual points near $t=0$
to effectively propagate information forward, as evidenced in nonlinear Allen-
Cahn PDE benchmarks. Notably, McClenny and Braga-Neto (2020) demonstrated that
their self-adaptive PINNs autonomously learn this weighting during training, a
phenomenon observed in benchmarks featuring complex initial conditions, such
as the Wave equation. Similarly, Krishnapriyan et al. (2021) proposed a
“sequence-to-sequence” approach, wherein the PINN sequentially learns each
time step in a time-marching scheme, further addressing the propagation
challenge. S. Wang et al. (2022) identified a lack of “causality” in standard
PINN training and proposed a corrective weighting scheme that gradually
prioritizes data from early to later times, leveraging information from the
PDE residue itself, akin to the self-adaptive weighting mechanism proposed by
McClenny and Braga-Neto (2020). Moreover, recent developments, such as the
pseudo-label approach introduced in Haitsiukevich and Ilin (2022), bear
resemblance to self-training methodologies in PINNs, further diversifying the
arsenal of techniques available for enhancing convergence and accuracy in
PINN-based solutions.

Finally, a characteristic-informed neural networks (CINNs) framework was
introduced in Braga-Neto (2023) for training PINNs for solving transport
equations, where the PDE information is built-in to the structure of the
neural networks. Similar approach was proposed in Lagaris et al. (1998) for a
Dirichlet BC, our method is essential similar to that approach but with a more
careful design of stability in mind.

####  1.0.2 Contributions of this Work

In this paper, we proposed a structure-preserving PINN algorithm by building
the IC/BC information into the structure of neural networks, in order to
reduce the training difficulties for PINNs for solving various stiff time-
dependent PDEs.

  * •

We develop structure preserving PINN algorithms, and investigate their
stability, training speed, and prediction accuracy. The structure preserving
PINN is similar to the method proposed recently in Braga-Neto (2023). However
in that paper, PDE is built into the structure; whereas ours build the IC/BC
into the PINNs.

  * •

We investigate how structure preserving ameliorates the issue of propagating
information forward in time, which is a common failure mode of PINNs.

  * •

We propose the first structure preserving PINNs which incorporate both IC and
BC data in a neural networks.

##  2 Methodology

We consider the following setup for a family of time dependent PDEs. Let $u$
be an unknown scalar function defined on $[0,T]\times\Omega$ with
$\Omega\subset\mathcal{R}^{d}$. Here the physical domain $\Omega$ comes with a
Lipschitz boundary $\partial\Omega$. Let $\mathcal{P}$ be a spatial partial
differential operator, i.e. for the heat equation $u_{t}-\lambda u_{xx}=0$,
$\mathcal{P}=\lambda\frac{\partial^{2}}{\partial x^{2}}$. Moreover
$\mathcal{P}$ can be nonlinear. Let $\mathcal{B}$ be another partial
differential operator on the boundary, i.e. for Dirichlet boundary condition
$u(t,x)=g(t,x)$ for $(t,x)\in[0,T]\times\partial\Omega$,
$\mathcal{B}=\mathcal{I}$ (the identity). Then we say that $u$ is a solution
of an PDE, if $u$ satisfies the following

$\begin{cases}u_{t}-\mathcal{P}(u)(t,\mathbf{x})&=0,\quad(t,\mathbf{x})\in(0,T]\times\Omega,\\\
u(0,\mathbf{x})&=u_{0}(\mathbf{x}),\quad\mathbf{x}\in\bar{\Omega}=\Omega\cup\partial\Omega,\\\
\mathcal{B}(u)(t,\mathbf{x})&=g(t,\mathbf{x}),\quad(t,\mathbf{x})\in[0,T]\times\partial\Omega,\\\
\end{cases}$ (1)

Here the two functions $u_{0},g$ are user-input and are assumed to satisfy
desired regularity so that the existence and uniqueness of solutions for such
a PDE is guaranteed. We also assume that the compactibility condition, i.e.,
$g(0,\mathbf{x})=u_{0}(\mathbf{x})$ for $\mathbf{x}\in\partial\Omega$, is
satisfied in order to prevent any ill-conditioning of the PDE problem.

Types of Boundary Conditions: one can consider many kinds of boundary
conditions as follows

$\displaystyle\mathcal{B}(u)(t,\mathbf{x})$
$\displaystyle=u(t,\mathbf{x}),\quad\text{Dirichlet type},$
$\displaystyle\mathcal{B}(u)(t,\mathbf{x})$ $\displaystyle=\frac{\partial
u}{\partial\mathbf{n}}(t,\mathbf{x}),\quad\text{Neumann type, $\mathbf{n}$ is
the outward normal vector to $\partial\Omega$},$
$\displaystyle\mathcal{B}(u)(t,\mathbf{x};a,b)$
$\displaystyle=au(t,\mathbf{x})+b\frac{\partial
u}{\partial\mathbf{n}}(t,\mathbf{x}),\quad\text{Robin type}.$

The periodic boundary condition gives $d$ different equations, and they are
given as follows,

$\mathcal{B}(u)(t,\mathbf{x})=u(t,\mathbf{x})-u(t,\mathbf{x}+P\mathbf{e}_{i})=0,\quad
i=1,\cdots,d.$

Here $P$ is the period and $\mathbf{e}_{i}$ is the $i^{th}$ standard basis
vector in $\mathcal{R}^{d}$, i.e.

$\mathbf{e}_{i}=\underbrace{\begin{bmatrix}0&\cdots&0&1&0&\cdots&0\end{bmatrix}^{\top}}_{\text{$1$
is the in the $i^{th}$ position.}}.$

One can even consider a mixture of two or all of the above BC conditions, i.e.
where $\Omega=\Gamma_{1}\cup\cdots\cup\Gamma_{K}$ and each $\Gamma_{i}$
defines a different kind of BC. In this paper, we will mainly focus on the
periodic type of boundary conditions.

Classical numerical techniques such as the Finite Difference Method (which
approximates derivatives on a point mesh), Finite Element Method (utilizing
basis approximation on a triangular mesh with weak formulation), and Spectral
Method (employing Fourier basis with FFT on a point mesh) are widely utilized
in scientific computations. However, recent advancements in scientific machine
learning integrate the principles of physics, particularly the understanding
of PDEs, into the training process of machine learning models. This
integration enables the development of PINNs to solve for $u$ as follows: one
tries to find an approximate solution from a set of deep neural networks (of
the same depth, same number of neurons on each hidden layer, and same
activation functions on each layer) $\mathcal{H}_{\text{NN}}$, which is a
minimizer of the following loss functional

$\begin{aligned}
\text{Loss}(u_{nn})&=\frac{1}{N_{CL}}\sum_{i=1}^{N_{CL}}|(\frac{\partial
u}{\partial
t}-\mathcal{P}(u_{nn}))(t_{i}^{CL},\mathbf{x}_{i}^{CL})|^{2}+\frac{1}{N_{IC}}\sum_{i=1}^{N_{IC}}|u_{nn}(0,\mathbf{x}_{i}^{IC})-u_{0}(\mathbf{x}_{i}^{IC})|^{2}\\\
&\quad+\sum_{i=1}^{d}\frac{1}{N_{BC}}\sum_{j=1}^{N_{BC}}|u_{nn}(t_{j}^{BC},\mathbf{x}_{j}^{BC})-u_{nn}(t_{j}^{BC},\mathbf{x}_{j}^{BC}+P\mathbf{e}_{i})|^{2}\end{aligned},\quad\forall
u_{nn}\in\mathcal{H}_{\text{NN}}$

Here $\\{(t^{CL},\mathbf{x}^{CL})_{i}\\}_{i=1}^{N_{CL}}\in(0,T]\times\Omega$
are called collocation points,
$\\{(0,\mathbf{x}^{IC})_{i}\\}_{i=1}^{N_{IC}}\in\\{0\\}\times\bar{\Omega}$ are
the initial condition points, and
$\\{(t^{BC},\mathbf{x}^{BC})_{i}\\}_{i=1}^{N_{BC}}\in[0,T]\times\partial\Omega$
are the boundary condition points. The minimizer, denoted as $u_{NN}$, will an
be approximate solution to (1).

###  2.1 Hard Constrains

The major motivation for us to consider hard constrains transformation for
PINNs is the possible ill-conditioning of the PDE solution map’s dependence on
IC and BC. Traditional numerical methods do not need to encounter this kind of
difficulty, as the IC data is exactly satisfied in their solutions. PINNs, on
the other hand, use $L^{2}$-loss to fit the IC/BC and PDE residual data,
causing the solutions to be highly sensitive to the goodness of fit of the IC
in the training (as pointed out in McClenny and Braga-Neto (2020)). By moving
the IC/BC data into the structure of PINNs, it reduces the ill-conditioning of
the solutions. Therefore, We consider the following transformation

$\tilde{u}(t,\mathbf{x})=\psi(t,\mathbf{x})+\phi(t,\mathbf{x})u_{nn}(t,\mathbf{x}).$

Here the two functions $\psi$ and $\phi$ are both smooth function with the
following properties

$\phi(0,\mathbf{x})=0\quad\text{for
$\mathbf{x}\in\bar{\Omega}$}\quad\text{and}\quad\phi(t,\mathbf{x})=0,\quad\text{for
$\mathbf{x}\in\partial\Omega$};$

and

$\psi(0,\mathbf{x})=u_{0}(\mathbf{x})\quad\text{for
$\mathbf{x}\in\bar{\Omega}$}\quad\text{and}\quad\psi(t,\mathbf{x})=0,\quad\text{for
$\mathbf{x}\in\partial\Omega$}.$

For the Neumann and Robin type, a more complicated $\psi$ and $\phi$ have to
be designed. For the periodic BC, we simply require $\psi$ and $\phi$ to be
$P$-periodic. If we use such transformation, then the training for $u_{nn}$ is
changed, due to the fact that IC/BC can be built into the PDE loss, i.e.

$\text{Loss}(u_{nn})=\frac{1}{N_{CL}}\sum_{i=1}^{N_{CL}}|(\tilde{u}_{t}-\mathcal{P}(\tilde{u}))(t_{i}^{CL},\mathbf{x}_{i}^{CL})|^{2},\quad\text{for
$\tilde{u}=\psi+\phi u_{nn}$}.$

We have reduced the kinds of losses in the training; however the information
about IC and BC has then moved into the PDE residual loss through $\psi$ and
$\phi$. The difficulty in reducing the new PDE residual loss has been
increased. Hence, other training enhancement techniques can be used to greatly
improve training.

Periodic Neural Networks: even though we require $\psi$ and $\phi$ to be
periodic, we still need to build the information of periodicity into $u_{n}n$,
hence we consider the following composition

$u_{nn}(t,x)=f_{nn}(v(t,x)),\quad\text{$f_{nn}$ is a neural network, and
$x\in\mathcal{R}$},$

and

$v(t,x)=\begin{bmatrix}t&1&\cos(\frac{2\pi}{P}x)&\sin(\frac{2\pi}{P}x)&\cos(\frac{2\pi}{P}2x)&\sin(\frac{2\pi}{P}2x)&\cdots&\cos(\frac{2\pi}{P}mx)&\sin(\frac{2\pi}{P}mx)\end{bmatrix}^{\top},$

where $m>0$ is a positive integer (another hyper-parameter). With this
composition, $u_{nn}$ will be automatically $P$-periodic (Dong and Ni, 2021).
For higher dimensional physical $\mathbf{x}$, similar transformation can be
used. (write the $2D$ here).

###  2.2 Mini-batching and Other Techniques

As we have mentioned in the previous section, although we have reduced the
kinds of losses to one PDE residual loss, the information about IC and BC have
been transferred into the new PDE residual loss. So we consider mini-batching
in the training for decreasing the PDE residual loss. As pointed out in Wight
and Zhao (2020), the mini-batching technique is similar to time-marching
resampling. But with mini-batching, one do not consider an explicit time-grid.
Similar training enhancement can be added, such as SA PINN McClenny and Braga-
Neto (2020), RBA PINN Anagnostopoulos et al. (2024), xPINN Jagtap and
Karniadakis (2020).

##  3 Numerical Experiments

In this section, We will evaluate our proposed PINN scheme in the previous
section on multiple prototypical time dependent PDEs with periodic boundary.

First of all, we use a Latin hypercube sampling (LHS) strategy to sample the
collocation points in the whole domain including the boundary. We next take a
simple transformation as follows

$\tilde{u}(t,\mathbf{x})=u_{0}(\mathbf{x})*\exp(-t)+t*u_{nn}(t,\mathbf{x}),\quad(t,\mathbf{x})\in[0,T]\times\bar{\Omega},$

where $\psi(t,\mathbf{x})=u_{0}(\mathbf{x})$ and $\phi(t,\mathbf{x})=t$.
Through the construction of the transformed u-network and the specification of
appropriate loss function, we train our neural network using the generated
points. Next, we utilize the trained model to approximate the solution to the
partial differential equations (PDEs).

To ascertain the accuracy of our results, we use Chebfun, a spectral-style
system implemented in MATLAB for handling functions in an object-oriented
manner. This allows us to obtain solutions with high accuracy against which we
can rigorously compare the performance of our scheme. Given the data points
$\\{x_{i},t_{i}\\}_{i=1}^{N}$ with $N$ the total number of points, we take
relative $L_{2}$ norm of the ”exact” value $u(x_{i},t_{i})$ at those points
and the network output $\mathcal{U}(x_{i},t_{i})$ at those points to evaluate
the accuracy of our trained model:

$\mathcal{E}=\frac{\sqrt{\sum_{i=1}^{N}|\mathcal{U}(x_{i},t_{i})-u(x_{i},t_{i})|^{2}}}{\sqrt{\sum_{i=1}^{N}|u(x_{i},t_{i})|^{2}}}.$

###  3.1 Solving the Viscous Burger’s Equation

The viscous Burger’s Equation is a fundamental partial differential equation
and convection-diffusion equation used in fluid dynamics and nonlinear waves
(Whitham, 2011; Burgers, 1948).

For a given field $u(x,t)$, and viscosity $\nu$, the general form of viscous
Burgers’ equation is:

$\displaystyle u_{t}+uu_{x}=\nu
u_{xx}\,,\quad(x,t)\in[-1,1]\times[0,\infty)\,,$ $\displaystyle
u(x,0)=u_{0}(x)\,,\quad-1<x<1\,,\quad\quad\quad\quad$ $\displaystyle
u(1,t)=u(-1,t)=0\,,\quad t>0\,.$

The parameters we used to train are in Table 1:

# Co. Points | # Layers | # Neurons
---|---|---
$16,384$ | $7$ | $32$
Table 1: PINN Params

It trains with $50k$ Adam steps with learning rate at $5\times 10^{-3}$ first,
and then uses the L-BFGS-B optimizer to fine-tune the neural network. The
results are shown in Figure 1 and Figure 2:

Figure 1: Exact solution of the 1D Viscous Burger’s Equation with the
corresponding network prediction and the absolute error difference. Figure 2:
Solutions of 1D Viscous Burger’s Equation.

###  3.2 Solving the Allen-Cahn Equation

Allen-Cahn type PDE is a classical phase-field model, has been widely used to
investigate the phenomena of phase separation (Bazant, 2017). The Allen-Cahn
equation has numerous practical applications across various fields, such as
material science (Allen and Cahn, 1979; Shen and Yang, 2010), biological
systems (Hyman et al., 2014; Takatori and Brady, 2015), electro-chemical
systems (Horstmann et al., 2013; Tian et al., 2015) etc. We first tested the
one-dimensional Allen-Cahn Equation with periodic boundary conditions, as
follows:

$\displaystyle
u_{t}-\gamma_{1}u_{xx}+\gamma_{2}u^{3}-\gamma_{2}u=0,\quad(t,x)\in(0,T)\times(a,b),$
(2) $\displaystyle u(0,x)=u_{0}(x),\quad x\in[a,b],$ $\displaystyle
u(t,a)=u(t,b),\quad t\in[0,T],$ $\displaystyle u_{x}(t,a)=u_{x}(t,b),\quad
t\in[0,T],$

where $\gamma_{1},\gamma_{2}>0,T>0,a<b$ are prescribed constants. As
$\gamma_{2}$ increases, the transition interface of the solutions is sharper,
which makes it harder to solve the AC equation numerically.

####  3.2.1 Case I

Therefore, we demonstrate the effectiveness of our scheme by testing on AC PDE
with a large $\gamma_{2}$:

$u_{0}(x)=x^{2}\cos(\pi x),\quad T=1,\quad a=-1,\quad
b=1,\quad\gamma_{1}=0.001,\quad\gamma_{2}=5.$

We train a PINN solution using the same parameters in Table 1.

Figure 3 and LABEL:fig_ac_case2 show the results:

Figure 3: Case I: Exact solution of the 1D AC with the corresponding network prediction and the absolute error difference. Figure 4: Case I: Solutions of the AC equation $\gamma_{2}=5$. Allen-Cahn | Baseline PINN |  Re-sampling (Wight and Zhao, 2020) | Our Approach
---|---|---|---
Relative $L_{2}$ |  $9.90e$-1 |  $2.33e$-2 |  $9.16e$-4
Relative $L_{1}$ |  $9.90e$-1 |  $6.20e$-3 |  $4.84e$-4
$L_{\infty}$ norm |  $9.96e$-1 |  $2.64e$-1 |  $1.43e$-1
Table 2: Comparison of errors in the learned solutions of the Allen-Cahn
equation using various PINN approaches

Table 2 compares the errors among the three approaches: baseline PINN, re-
sampling technique proposed by Wight and Zhao (2020) and our approach.
Unfortunately, using the standard PINN approach alone, we were not able to
learn the accurate solution for the Allen-Cahn equation, and capture the
dynamics of equation. The relative error is almost equal to 1. Compared with
the technique proposed by Wight and Zhao (2020), the error is much less and
the accuracy is improved.

####  3.2.2 Case II

Next, we explore how our enhanced PINN approach performs in handling scenarios
with sharper moving interfaces. Specifically, we modify the initial conditions
of the Allen-Cahn equation to introduce more oscillations, and vary the
parameters to observe the method’s effectiveness across different problem
settings. Instead of Equation 2 in case I, we use the following equations:

$\displaystyle
u_{t}-\gamma_{1}u_{xx}+\gamma_{2}u^{3}-\gamma_{2}u=0,\quad(t,x)\in(0,1)\times(-1,1),$
(3) $\displaystyle u(0,x)=x^{2}\sin(2\pi x),\quad x\in[-1,1],$ $\displaystyle
u(t,-1)=u(t,1),\quad t\in[0,1],$ $\displaystyle u_{x}(t,-1)=u_{x}(t,1),\quad
t\in[0,1],$

where $\gamma_{1}=0.001,\gamma_{2}=4$.

Figure 5 and 6 show the results:

Figure 5: Case II: Exact solution of the 1D AC with the corresponding network
prediction and the absolute error difference. Figure 6: Case II: Solutions of
the AC equation $\gamma_{2}=4$.

###  3.3 Solving the Cahn-Hilliard Equation

Related to Allen-Cahn Equation, Cahn-Hilliard Equation also describes the
process of phase separation, by which the two components of a binary fluid
spontaneously separate and form domains pure in each component (Cahn and
Hilliard, 1958; Lee et al., 2014; Miranville, 2017; Kim et al., 2016). It is
defined as the following equation:

$\displaystyle
u_{t}=\epsilon_{1}(-u_{xx}-\epsilon_{2}u_{xxxx}+(u^{3})_{xx}),\quad(t,x)\in(0,T]\times(-L,L),$
(4) $\displaystyle u(0,x)=u_{0}(x),\quad x\in[-L,L],$ $\displaystyle
u(t,-L)=u(t,L),\quad t\in[0,T],$

where $\epsilon_{1}=10^{-2},\epsilon_{2}=10^{-4},T=1,L=1$. We specify the
initial condition as $u_{0}(x)=-\cos(2\pi x)$. This PDE has higher order
derivatives and is known to be harder to solve than the Allen-Cahn equation.
We train the neural network by using 7 layers, 128 neurons and 16,384
collection points with $50k$ steps. The results are shown in the following
Figures 7 and Figure 8:

Figure 7: Exact solution of the 1D CH with the corresponding network
prediction and the absolute error difference. Figure 8: Solutions of the CH
equation.

###  3.4 Solving the Kuramoto-Sivashinsky equation

Kuramoto–Sivashinsky equation is a fourth-order nonlinear partial differential
equation, known for its chaotic behavior (Kudryashov, 1990; Michelson, 1986),
as follows:

$\displaystyle u_{t}=-u_{xx}-u_{xxxx}-uu_{x},\quad(t,x)\in(0,T]\times(a,b),$
(5) $\displaystyle u(0,x)=u_{0}(x),\quad x\in[a,b],$ $\displaystyle
u(t,a)=u(t,b),\quad t\in[0,T],$

where $T=20,a=0,b=32\pi$, and
$u_{0}(x)=\cos\frac{x}{16}(1+\sin\frac{x-1}{16})$. The parameters are the same
with Table 1. The results are shown in Figure 9 and 10:

Figure 9: Exact solution of the 1D KS with the corresponding network
prediction and the absolute error difference. Figure 10: Solutions of the KS
equation.

###  3.5 Solving the Gray-Scott Equation

Reaction and diffusion of chemical species can produce a variety of patterns
(De Kepper et al., 1991; Nishiura and Ueyama, 1999), reminiscent of those
often seen in nature. The Gray-Scott type system is one of classical
mathematical models for chemical reactions (Gray and Scott, 1983, 1984; Liang
et al., 2022). The general irreversible GS equations describe such reactions:

$\displaystyle U+2V$ $\displaystyle\longrightarrow 3V,$ (6) $\displaystyle V$
$\displaystyle\longrightarrow P.$

This system is defined by two equations that describe the dynamics of two
reacting substances:

$\displaystyle
u_{t}=\epsilon_{1}u_{xx}+b(1-u)-uv^{2},\quad(t,x)\in(0,T]\times(-L,L),$ (7)
$\displaystyle v_{t}=\epsilon_{2}v_{xx}-(b+k)v+uv^{2},$ $\displaystyle
u(0,x)=u_{0}(x),\quad v(0,x)=v_{0}(x),\quad\forall x\in[-L,L],$ $\displaystyle
u(t,-L)=u(t,L),\quad v(t,-L)=v(t,L),\quad\forall t\in[0,T],j$

where $T=20,L=50$, $\epsilon_{1}=1,\epsilon_{2}=0.01$ are diffusion rates,
$b=0.02$ is the ”feeding rate” that adds $U$, $k=0.0562$ is the ”killing rate”
that removes $V$. We set our initial conditions as:

$u_{0}(x)=1-\frac{\sin(\pi(x-50)/100)^{4}}{2},\quad
v_{0}(x)=\frac{\sin(\pi(x-50)/100)^{4}}{4}.$

The parameters that we used are the same with before. Figures 12 and
LABEL:fig:gs_v show the results:

Figure 11: Solutions of the GS-u equation. Figure 12: Solutions of the GS-v
equation.

Figure 13 and Figure14 show the results of exact solutions of the 1D GS
Equations that we obtain using Chebfun, with the corresponding network
predictions and the absolute error differences:

Figure 13: Exact solution of the 1D GS-u Equation with the corresponding
network prediction and the absolute error difference. Figure 14: Exact
solution of the 1D GS-v Equation with the corresponding network prediction and
the absolute error difference.

###  3.6 Solving the Belousov-Zhabotinsky equation

A Belousov–Zhabotinsky reaction is one of a class of reactions that serve as a
classical example of non-equilibrium thermodynamics. It typically involves the
oxidation of organic compounds by bromine in an acidic medium (Zhabotinsky,
2007; Cassani et al., 2021). The system is as follows:

$\displaystyle u_{t}=\epsilon_{1}u_{xx}+u+v-uv-u^{2},$ (8) $\displaystyle
v_{t}=\epsilon_{2}v_{xx}+w-v-uv,$ $\displaystyle
w_{t}=\epsilon_{1}w_{xx}+u-w,\quad(t,x)\in(0,T]\times[-L,L]$ $\displaystyle
u(0,x)=u_{0}(x),\quad v(0,x)=v_{0}(x),\quad w(0,x)=w_{0}(x),\quad\forall
x\in[-L,L]$ $\displaystyle u(t,-L)=u(t,L),\quad v(t,-L)=v(t,L),\quad
w(t,-L)=w(t,L),\quad\forall t\in[0,T],$

where $T=3,L=1$, $\epsilon_{1}=10^{-5},\epsilon_{2}=2\times 10^{-5}$ are
diffusion rates. We let the initial conditions be:

$u_{0}(x)=\exp(-100(x+0.5)^{2}),\quad v_{0}(x)=\exp(-100x^{2}),\quad
w_{0}(x)=\exp(-100(x-0.5)^{2}).$

The results are:

Figure 15: Solutions of the BZ-u equation. Figure 16: Solutions of the BZ-v
equation. Figure 17: Solutions of the BZ-w equation.

The comparison between the exact solutions of $(u,v,w)$ equations with
corresponding neural network predictions by our approach on the whole $(x,t)$
domain are shown in Figures 18, 16 and 17:

Figure 18: Exact solution of the 1D BZ-u Equation with the corresponding
network prediction and the absolute error difference. Figure 19: Exact
solution of the 1D BZ-v Equation with the corresponding network prediction and
the absolute error difference. Figure 20: Exact solution of the 1D BZ-u
Equation with the corresponding network prediction and the absolute error
difference.

###  3.7 Solving the Nonlinear Schroedinger equation

In theoretical physics, nonlinear schroedinger equation is a nonlinear PDE,
applicable to classical and quantum mechanics (Kato, 1987; Kevrekidis et al.,
2001; Fibich, 2015). The dimensionless equation of the classical field is

$\displaystyle u_{t}=iu_{xx}+i|u|^{2}u,\quad(t,x)\in[0,2]\times[-\pi,\pi],$
(9) $\displaystyle u(0,x)=u_{0}(x),\quad x\in[-\pi,\pi],$ $\displaystyle
u(t,-\pi)=u(t,\pi),\quad t\in[0,2],$

where we let

$u_{0}(x)=\frac{2}{2-\sqrt{2}\cos(x)}-1.$

Using the same parameters and training step with before, we obtain the
following results:

Figure 21: Exact solution of the real part of 1D NLS with the corresponding
network prediction and the absolute error difference. Figure 22: Real-part
Solutions of the NLS equation. Figure 23: Exact solution of the imaginary part
of 1D NLS with the corresponding network prediction and the absolute error
difference. Figure 24: Imaginary-part Solutions of the NLS equation.

##  4 Conclusion

We introduce a structure-preserving PINN where the information about IC and BC
are built-in to the structure of the neural networks. It greatly reduces the
training difficulties for seven different kinds of stiff $1$-dimensional time-
dependent PDEs with periodic boundary condition, ranging scalar output, vector
output, and to complex number output. Future work on how to extend such
methods to $2$\- or $3$-dimensional PDEs are on the work, and investigation on
the optimal choice of $\psi$ and $\phi$, i.e. the transformation is ongoing.

## Acknowledgements

BH and MZ thank Prof. Yannis Kevrekids (JHU) for the discussion of well-
conditioning of PDE solution maps, and Prof. George Karniadakis (Brown) for
his suggestion on training enhancement methods for PINNs.

## Funding Statement

MZ is supported by NSF-AoF grant $\\#2225507$.

## References

  *   * Allen and Cahn (1979) Allen, S. M., and Cahn, J. W.  (1979).  A microscopic theory for antiphase boundary motion and its application to antiphase domain coarsening.  _Acta metallurgica_ , _27_(6), 1085–1095.
  * Anagnostopoulos et al. (2024) Anagnostopoulos, S. J., Toscano, J. D., Stergiopulos, N., and Karniadakis, G. E.  (2024).  Residual-based attention in physics-informed neural networks.  _Computer Methods in Applied Mechanics and Engineering_ , _421_ , 116805.  Retrieved from https://www.sciencedirect.com/science/article/pii/S0045782524000616 DOI: https://doi.org/10.1016/j.cma.2024.116805
  * Bazant (2017) Bazant, M. Z.  (2017).  Thermodynamic stability of driven open systems and control of phase separation by electro-autocatalysis.  _Faraday discussions_ , _199_ , 423–463.
  * Braga-Neto (2023) Braga-Neto, U.  (2023).  _Characteristics-informed neural networks for forward and inverse hyperbolic problems._
  * Burgers (1948) Burgers, J. M.  (1948).  A mathematical model illustrating the theory of turbulence.  _Advances in applied mechanics_ , _1_ , 171–199.
  * Cahn and Hilliard (1958) Cahn, J. W., and Hilliard, J. E.  (1958).  Free energy of a nonuniform system. i. interfacial free energy.  _The Journal of chemical physics_ , _28_(2), 258–267.
  * Cai et al. (2021) Cai, S., Wang, Z., Lu, L., Zaki, T. A., and Karniadakis, G. E.  (2021).  DeepM&Mnet: Inferring the electroconvection multiphysics fields based on operator approximation by neural networks.  _Journal of Computational Physics_ , _436_ , 110296.
  * Carracedo-Reboredo et al. (2021) Carracedo-Reboredo, P., Liñares-Blanco, J., Rodríguez-Fernández, N., Cedrón, F., Novoa, F. J., Carballal, A., … Fernandez-Lozano, C.  (2021).  A review on machine learning approaches and trends in drug discovery.  _Computational and Structural Biotechnology Journal_ , _19_ , 4538-4558.  Retrieved from https://www.sciencedirect.com/science/article/pii/S2001037021003421 DOI: https://doi.org/10.1016/j.csbj.2021.08.011
  * Cassani et al. (2021) Cassani, A., Monteverde, A., and Piumetti, M.  (2021).  Belousov-zhabotinsky type reactions: The non-linear behavior of chemical systems.  _Journal of mathematical chemistry_ , _59_ , 792–826.
  * X. Chen et al. (2022) Chen, X., Jeffery, D. J., Zhong, M., McClenny, L., Braga-Neto, U., and Wang, L.  (2022).  _Using physics informed neural networks for supernova radiative transfer simulation._
  * Y. Chen et al. (2021) Chen, Y., Hosseini, B., Owhadi, H., and Stuart, A. M.  (2021).  Solving and learning nonlinear PDEs with Gaussian processes.  _Journal of Computational Physics_ , _447_ , 110668.
  * Coutinho et al. (2023) Coutinho, E. J. R., Dall’Aqua, M., McClenny, L., Zhong, M., Braga-Neto, U., and Gildin, E.  (2023).  Physics-informed neural networks with adaptive localized artificial viscosity.  _Journal of Computational Physics_ , _489_ , 112265.  Retrieved from https://www.sciencedirect.com/science/article/pii/S0021999123003601 DOI: https://doi.org/10.1016/j.jcp.2023.112265
  * De Kepper et al. (1991) De Kepper, P., Castets, V., Dulos, E., and Boissonade, J.  (1991).  Turing-type chemical patterns in the chlorite-iodide-malonic acid reaction.  _Physica D: Nonlinear Phenomena_ , _49_(1-2), 161–169.
  * Dong and Ni (2021) Dong, S., and Ni, N.  (2021).  A method for representing periodic functions and enforcing exactly periodic boundary conditions with deep neural networks.  _Journal of Computational Physics_ , _435_ , 110242.
  * Fawzi et al. (2022) Fawzi, A., Balog, M., Huang, A., Hubert, T., Romera-Paredes, B., Barekatain, M., … Kohli, P.  (2022).  Discovering faster matrix multiplication algorithms with reinforcement learning.  _Nature_ , _610_ , 47 - 53.
  * Fibich (2015) Fibich, G.  (2015).  _The nonlinear schrödinger equation_ (Vol. 192).  Springer.
  * Gray and Scott (1983) Gray, P., and Scott, S.  (1983).  Autocatalytic reactions in the isothermal, continuous stirred tank reactor: isolas and other forms of multistability.  _Chemical Engineering Science_ , _38_(1), 29–43.
  * Gray and Scott (1984) Gray, P., and Scott, S. K.  (1984).  Autocatalytic reactions in the isothermal, continuous stirred tank reactor: Oscillations and instabilities in the system a+ 2b→ 3b; b→ c.  _Chemical Engineering Science_ , _39_(6), 1087–1097.
  * Haitsiukevich and Ilin (2022) Haitsiukevich, K., and Ilin, A.  (2022).  Improved training of physics-informed neural networks with model ensembles.  _arXiv preprint arXiv:2204.05108_.
  * Horstmann et al. (2013) Horstmann, B., Danner, T., and Bessler, W. G.  (2013).  Precipitation in aqueous lithium–oxygen batteries: a model-based analysis.  _Energy & Environmental Science_, _6_(4), 1299–1314.
  * Hyman et al. (2014) Hyman, A. A., Weber, C. A., and Jülicher, F.  (2014).  Liquid-liquid phase separation in biology.  _Annual review of cell and developmental biology_ , _30_ , 39–58.
  * Jagtap and Karniadakis (2020) Jagtap, A. D., and Karniadakis, G. E.  (2020).  Extended physics-informed neural networks (xpinns): A generalized space-time domain decomposition based deep learning framework for nonlinear partial differential equations.  _Communications in Computational Physics_ , _28_(5), 2002–2041.
  * Jin et al. (2021) Jin, X., Cai, S., Li, H., and Karniadakis, G. E.  (2021).  NSFnets (Navier-Stokes flow nets): Physics-informed neural networks for the incompressible Navier-Stokes equations.  _Journal of Computational Physics_ , _426_ , 109951.
  * Jumper et al. (2021) Jumper, J., R., E., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., … Hassabis, D.  (2021).  Highly accurate protein structure prediction with alphafold.  _Nature_ , _596_ , 583 - 589.
  * Kato (1987) Kato, T.  (1987).  On nonlinear schrödinger equations.  In _Annales de l’ihp physique théorique_ (Vol. 46, pp. 113–129).
  * Kevrekidis et al. (2001) Kevrekidis, P., Rasmussen, K., and Bishop, A.  (2001).  The discrete nonlinear schrödinger equation: a survey of recent results.  _International Journal of Modern Physics B_ , _15_(21), 2833–2900.
  * Kim et al. (2016) Kim, J., Lee, S., Choi, Y., Lee, S.-M., Jeong, D., et al.  (2016).  Basic principles and practical applications of the cahn–hilliard equation.  _Mathematical Problems in Engineering_ , _2016_.
  * Krishnapriyan et al. (2021) Krishnapriyan, A., Gholami, A., Zhe, S., Kirby, R., and Mahoney, M. W.  (2021).  Characterizing possible failure modes in physics-informed neural networks.  _Advances in Neural Information Processing Systems_ , _34_ , 26548–26560.
  * Kudryashov (1990) Kudryashov, N. A.  (1990).  Exact solutions of the generalized kuramoto-sivashinsky equation.  _Physics Letters A_ , _147_(5-6), 287–291.
  * Lagaris et al. (1998) Lagaris, I., Likas, A., and Fotiadis, D.  (1998).  Artificial neural networks for solving ordinary and partial differential equations.  _IEEE Transactions on Neural Networks_ , _9_(5), 987–1000.  Retrieved from http://dx.doi.org/10.1109/72.712178 DOI: 10.1109/72.712178
  * Lee et al. (2014) Lee, D., Huh, J.-Y., Jeong, D., Shin, J., Yun, A., and Kim, J.  (2014).  Physical, mathematical, and numerical derivations of the cahn–hilliard equation.  _Computational Materials Science_ , _81_ , 216–225.
  * Liang et al. (2022) Liang, J., Jiang, N., Liu, C., Wang, Y., and Zhang, T.-F.  (2022).  On a reversible gray-scott type system from energetic variational approach and its irreversible limit.  _Journal of Differential Equations_ , _309_ , 427–454.
  * Liu and Wang (2019) Liu, D., and Wang, Y.  (2019).  Multi-fidelity physics-constrained neural network and its application in materials modeling.  _Journal of Mechanical Design_ , _141_(12).
  * Liu and Wang (2021) Liu, D., and Wang, Y.  (2021).  A dual-dimer method for training physics-constrained neural networks with minimax architecture.  _Neural Networks_ , _136_ , 112–125.
  * McClenny and Braga-Neto (2020) McClenny, L., and Braga-Neto, U.  (2020).  Self-adaptive physics-informed neural networks using a soft attention mechanism.  _arXiv preprint arXiv:2009.04544_.
  * Michelson (1986) Michelson, D.  (1986).  Steady solutions of the kuramoto-sivashinsky equation.  _Physica D: Nonlinear Phenomena_ , _19_(1), 89–111.
  * Miranville (2017) Miranville, A.  (2017).  The cahn–hilliard equation and some of its variants.  _AIMS Mathematics_ , _2_(3), 479–544.
  * Nishiura and Ueyama (1999) Nishiura, Y., and Ueyama, D.  (1999).  A skeleton structure of self-replicating dynamics.  _Physica D: Nonlinear Phenomena_ , _130_(1-2), 73–104.
  * Rad et al. (2020) Rad, M. T., Viardin, A., Schmitz, G., and Apel, M.  (2020).  Theory-training deep neural networks for an alloy solidification benchmark problem.  _Computational Materials Science_ , _180_ , 109687.
  * Raissi and Karniadakis (2018) Raissi, M., and Karniadakis, G. E.  (2018).  Hidden physics models: Machine learning of nonlinear partial differential equations.  _Journal of Computational Physics_ , _357_ , 125–141.
  * Raissi et al. (2019) Raissi, M., Perdikaris, P., and Karniadakis, G. E.  (2019).  Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations.  _Journal of Computational Physics_ , _378_ , 686–707.
  * Shen and Yang (2010) Shen, J., and Yang, X.  (2010).  Numerical approximations of allen-cahn and cahn-hilliard equations.  _Discrete Contin. Dyn. Syst_ , _28_(4), 1669–1691.
  * Shin et al. (2020) Shin, Y., Darbon, J., and Karniadakis, G. E.  (2020).  On the convergence and generalization of physics informed neural networks.  _arXiv preprint arXiv:2004.01806_.
  * Takatori and Brady (2015) Takatori, S. C., and Brady, J. F.  (2015).  Towards a thermodynamics of active matter.  _Physical Review E_ , _91_(3), 032117.
  * Tian et al. (2015) Tian, W., Mao, X., Brown, P., Rutledge, G. C., and Hatton, T. A.  (2015).  Electrochemically nanostructured polyvinylferrocene/polypyrrole hybrids with synergy for energy storage.  _Advanced Functional Materials_ , _25_(30), 4803–4813.
  * R. Wang et al. (2023) Wang, R., Zhong, M., Xu, K., Sánchez-Cortés, L. G., and de Cominges Guerra, I.  (2023).  _Pinns-based uncertainty quantification for transient stability analysis._
  * S. Wang et al. (2022) Wang, S., Sankaran, S., and Perdikaris, P.  (2022).  Respecting causality is all you need for training physics-informed neural networks.  _arXiv preprint arXiv:2203.07404_.
  * S. Wang, Teng, and Perdikaris (2020) Wang, S., Teng, Y., and Perdikaris, P.  (2020).  Understanding and mitigating gradient pathologies in physics-informed neural networks.  _arXiv preprint arXiv:2001.04536_.
  * S. Wang, Yu, and Perdikaris (2020) Wang, S., Yu, X., and Perdikaris, P.  (2020).  When and why PINNs fail to train: A neural tangent kernel perspective.  _arXiv preprint arXiv:2007.14527_.
  * Whitham (2011) Whitham, G. B.  (2011).  _Linear and nonlinear waves_.  John Wiley & Sons.
  * Wight and Zhao (2020) Wight, C. L., and Zhao, J.  (2020).  Solving allen-cahn and cahn-hilliard equations using the adaptive physics informed neural networks.  _arXiv preprint arXiv:2007.04542_.
  * Yang et al. (2019) Yang, X., Barajas-Solano, D., Tartakovsky, G., and Tartakovsky, A. M.  (2019).  Physics-informed cokriging: A Gaussian-process-regression-based multifidelity method for data-model convergence.  _Journal of Computational Physics_ , _395_ , 410–431.
  * Zhabotinsky (2007) Zhabotinsky, A. M.  (2007).  Belousov-zhabotinsky reaction.  _Scholarpedia_ , _2_(9), 1435.
  * Zhong et al. (2024) Zhong, M., Liu, D., Arroyave, R., and Braga-Neto, U.  (2024).  _Label propagation training schemes for physics-informed neural networks and gaussian processes._
  * Zhu et al. (2021) Zhu, Q., Liu, Z., and Yan, J.  (2021).  Machine learning for metal additive manufacturing: predicting temperature and melt pool fluid dynamics using physics-informed neural networks.  _Computational Mechanics_ , _67_(2), 619–635.