# Learning Vision-based Robotic Manipulation Tasks Sequentially in Offline
Reinforcement Learning Settings

Sudhir Pratap Yadav1, Rajendra Nagar2, and Suril V. Shah3 *This work was done
with collaboration of IIT Jodhpur and iHub-Drishti Foundation, IIT Jodhpur
1Sudhir Pratap Yadav, iHub Drishti Foundation<EMAIL_ADDRESS>Nagar, IIT Jodhpur<EMAIL_ADDRESS>V. Shah, IIT Jodhpur
<EMAIL_ADDRESS>
###### Abstract

With the rise of deep reinforcement learning (RL) methods, many complex
robotic manipulation tasks are being solved. However, harnessing the full
power of deep learning requires large datasets. Online-RL does not suit itself
readily into this paradigm due to costly and time-taking agent environment
interaction. Therefore recently, many offline-RL algorithms have been proposed
to learn robotic tasks. But mainly, all such methods focus on a single task or
multi-task learning, which requires retraining every time we need to learn a
new task. Continuously learning tasks without forgetting previous knowledge
combined with the power of offline deep-RL would allow us to scale the number
of tasks by keep adding them one-after-another. In this paper, we investigate
the effectiveness of regularisation-based methods like synaptic intelligence
for sequentially learning image-based robotic manipulation tasks in an
offline-RL setup. We evaluate the performance of this combined framework
against common challenges of sequential learning: catastrophic forgetting and
forward knowledge transfer. We performed experiments with different task
combinations to analyze the effect of task ordering. We also investigated the
effect of the number of object configurations and density of robot
trajectories. We found that learning tasks sequentially helps in the
propagation of knowledge from previous tasks, thereby reducing the time
required to learn a new task. Regularisation based approaches for continuous
learning like the synaptic intelligence method although helps in mitigating
catastrophic forgetting but has shown only limited transfer of knowledge from
previous tasks.

##  I INTRODUCTION

Robots now have the capability to learn many single manipulation tasks using
deep Reinforcement Learning (RL), such as pick-place [1], peg-in-hole [23],
Cloth folding [14], and tying rope knots [17]. Multitask RL has also been
applied successfully to learn robotic manipulation tasks [8], [10]. Number of
tasks and task-data distribution are kept fixed in the case of multi-task RL.
Therefore, agent has to be trained from scratch whenever it needs to learn a
new task, even if there is a substantial overlap between tasks. Scaling this
approach to learn all manipulation tasks at par with humans is not feasible.
Humans use the experience of previous tasks for learning a new task and do not
need to learn from the start. The sequential (or continual) learning approach
tries to address this problem by providing a framework where an agent can
learn new tasks one-after-another without starting from scratch. We use
offline-RL as the base framework to learn a single image-based robotic-
manipulation task and then use a regularisation based continual learning
approach for learning tasks sequentially. This combined framework forms main
contribution of this work.

Figure 1: Block Diagram of SAC-CQL-SI method for Sequential Learning

###  I-A Related Work

Most work in sequential task learning is focused on classification-based tasks
using typical classification datasets such as MNIST, CIFAR and their
variations [7], [15]. Some works in continual reinforcement Learning setup use
Atari-games [12]. Other try to extend continual RL to GYM environments [2].
Recent work in [20] uses Offline-RL for solving manipulation tasks using image
observations alone. While this work focuses on generalizing to novel initial
conditions but does not attempt sequential task learning. On the other hand,
our work is about learning tasks sequentially with only the current task data
available for learning. Some very recent works try to apply continual RL on
robotics manipulation tasks [22], [4]. [22] introduces a continual learning
benchmark for robotic manipulation tasks. It gives baselines for major
continual learning methods over these robotics tasks in online RL settings
using soft actor-critic (SAC) method [9]. But this work focuses on online-
continual RL with low-dimensional observation space such as joint and task
space data as they assume full access to the simulator. While our work focuses
on Offline RL with high-dimensional observation space (image) in sequential
learning of robotics manipulation tasks.

In the sequential learning setup based on deep RL, neural-networks (NN) are
prone to change in data-distribution. Hence, its accuracy on previous tasks
drops significantly when it is trained on a new task. This problem is actively
studied under the name of catastrophic forgetting. More broadly, in every
connectionist model of memory and computation problem of stability-plasticity
exists. This means the network needs to be flexible enough to accommodate new
information and simultaneously not forget previous information, as discussed
in [6]. Many solutions have been suggested to mitigate this problem. We place
these under two major categories architectural and penalty based. In the
architectural type of solutions, relevant changes are made in the neural
network architecture without changing the loss function. For example,
Progressive neural network [19] uses multiple parrallel paths with lateral
connenctions, and policy distillation [18] distills the policy learned by
larger network into smaller without loss of performance. On the other hand,
penalty based methods put penalties on neural network parameters so that they
stay close to solution of previous task. Two important work in this regards
are Elastic Weight Consolidation (EWC) [12] and Synaptic Intelligence [24].
EWC gives regularisation based solution for catastrophic forgetting, but
computation for the importance of parameters is not local. In this paper we
use the approach proposed in Synaptic Intelligence [24] because of its local
measure of importance for synapses (weights of Neural Network) as the nature
of local computation helps in keeping the solution independent of the
particularities of the problem hence making it more general.

To the best of our knowledge, this is the first work investigating sequential
learning for image based robotic task manipulation in offline RL settings. In
this paper, we focus on two sequential learning challenges: catastrophic
forgetting and forward knowledge transfer. We analyse the effect of task-
ordering and number of object configurations on forgetting and forward
knowledge transfer between tasks.

##  II LEARNING IMAGE BASED ROBOTIC MANIPULATION TASKS SEQUENTIALLY

In this section, we formulate our RL agent and environment interaction setup
to learn robotic manipulation tasks. We then discuss the problem of sequential
task learning and present an approach to solve this problem.

###  II-A RL formulation for Learning Image Based Robotic Manipulation Tasks

Agent and environment interaction is formally defined by the Markov Decision
Process (MDP) concept. A Markov Decision Process is a discrete-time stochastic
control process. In RL, we formally define the MDP as a tuple
$\langle\mathcal{S},\mathcal{A},\mathsf{P},r,\gamma\rangle$. Here,
$\mathcal{S}$ is a finite set of states, $\mathcal{A}$ is a finite set of
actions, $\mathsf{P}$ is the state transition probability matrix, $r$ is the
reward for a given state-action pair and $\gamma$ is the discount factor. A
stochastic policy is defined as a distribution over actions given the states,
i.e., the probability of taking each action for every state.
$\pi(\mathbf{a}|\mathbf{s})=\mathbb{P}[\mathcal{A}_{t}=\mathbf{a}|\mathcal{S}_{t}=\mathbf{s}]$.

RL formulation: We formulate the vision-based robotic manipulation tasks using
the deep RL framework as below.

  * •

Environment: It consists of WidowX 250 five-axes robot arm equipped with a
gripper. We place a table in front of the robot. Every task consists of an
object placed on the table, which needs to be manipulated to complete the task
successfully. We place a camera in the environment in eye-to-hand
configuration.

  * •

State: The state $\mathbf{s}_{t}$ represents the RGB image of the environment
captured at time step $t$. We use capture images of size $48\times 48\times
3$.

  * •

Action: We define the action at the time step $t$ as a 7 dimensional vector
$\mathbf{a}_{t}=\begin{bmatrix}\Delta X_{t}&\Delta
O_{t}&g_{t}\end{bmatrix}^{\top}$. Here, $\Delta X_{t}\in\mathbb{R}^{3}$,
$\Delta O_{t}\in\mathbb{R}^{3}$, $g_{t}\in\\{0,1\\}$ denotes the change in
position, change in orientation, and gripper command (open/close),
respectively at time step $t$.

  * •

Reward: The reward $r(\mathbf{s}_{t},\mathbf{a}_{t})\in\\{0,1\\}$ is a binary
variable which is equal to 1 if the task is successful and 0, otherwise.
Reward is given at each time step.

The reward is kept simple and not shaped according to the tasks so that the
same reward framework can be used while scaling for large number of tasks.
Also, giving reward at each time step, instead at the end of the episode,
makes the sum of rewards during an episode dependent on time steps. Therefore,
if the agent completes a task in fewer steps, its total reward for that
episode will be more.

###  II-B Sequential Learning Problem and Solution

We define the sequential tasks learning problem as follows. The agent is
required to learn $N$ number of tasks but with the condition that tasks will
be given sequentially to the agent and not simultaneously. Therefore, when the
agent is learning to perform a particular task, it can only access the data of
the current task. This learning process reassembles how a human learns. Let a
sequence of robotic manipulation tasks $T_{1},T_{2},...,T_{N}$ is given. We
assume that each task has the same type of state and action space. Each task
has its own data in typical offline reinforcement learning format
$\langle\mathbf{s}_{t},\mathbf{a}_{t},\mathbf{r}_{t},\mathbf{s}_{t+1}\rangle$.
The agent has to learn a policy $\pi$, a mapping from state to action, for
every task. If we naively train a neural network in this fashion problem of
catastrophic forgetting will occur, which means performance on the previous
task will decrease drastically as soon as the neural network starts learning a
new task.

We use a regularisation based approach presented in [24] to mitigate the
problem of catastrophic forgetting. Figure 1 provides the overall framework we
use to solve this problem. Each task data is given one by one to the
algorithm, which then starts training for the current task. First, a mini-
batch is sampled from this current-task data and passed to the SAC-CQL
algorithm (described in the next section), which then calculates actor
(Q-loss) and critic (policy) loss. If the task-index is greater than one then
we add a quadratic regularisation as defined in [24] to the actor loss to
reduce forgetting. Then, these losses are used to update neural networks,
which represents policy (actor network) and Q-value function (critic network).
After the current task is successfully learned next task data comes, and this
process is repeated until all tasks are learned.

##  III INTEGRATING SEQUENTIAL TASK LEARNING WITH OFFLINE RL

In this section, we discuss the SAC-CQL [13] offline algorithm and its
implementation details. We then discuss the SI regularisation method for
continual learning and provide details to integrate these methods to learn
sequential tasks.

###  III-A SAC-CQL algorithm for Offline RL

There are two frameworks, namely online and offline learning, to train an RL
agent. In the case of an online-RL training framework, an RL agent interacts
with the environment to collect experience, update itself (train), interact
again, and so on. In simple terms, the environment is always available for the
RL agent to evaluate itself and improve further. This interaction loop is
repeated for many episodes during training until the RL agent gets good enough
to perform the task successfully. While in offline RL settings, we collect
data once and are no more required to interact with the environment. This data
can be collected by executing a hand-designed policy or can be obtained by a
human controlling the robot (human-demonstration). Data is a sequence of
$\langle\mathbf{s}_{t},\mathbf{a}_{t},\mathbf{r}_{t},\mathbf{s}_{t+1}\rangle$
tuples.

In recent years the SAC (soft-actor critic) [9] has emerged as the most robust
algorithm for training RL agents in continuous action space (when action is a
real vector), which typically is a case in robotics. SAC is an off-policy
entropy based actor-critic method for continuous action MDPs. Entropy based
methods add an additional entropy term to the existing optimisation goal of
maximising expected reward. In addition to maximising expected reward, the RL
agent also needs to maximise the entropy of the overall policy. This helps in
making the policy inherently exploratory and not stuck inside a local minima.
Haarnoja _et al._ [9] define the RL objective in maximum entropy RL settings
as in (1).

$J(\pi)=\sum\limits_{t=0}^{T}\mathbb{E}_{(\mathbf{s}_{t},\mathbf{a}_{t})\sim\rho_{\pi}}[r(\mathbf{s}_{t},\mathbf{a}_{t})+\alpha\mathcal{H}(\pi(\cdot|\mathbf{s}_{t}))].$
(1)

Here, $\rho_{\pi}(\mathbf{s}_{t},\mathbf{a}_{t})$ denotes the joint
distribution of the state and actions over all trajectories of the agent could
take and $\mathcal{H}(\pi(\cdot|\mathbf{s}_{t}))$ is the entropy of the policy
for state $\mathbf{s}_{t}$ as defined in (2).

$\mathcal{H}(\pi(\cdot|\mathbf{s}_{t}))=\mathbb{E}[-\text{log}(f_{\pi}(\cdot|\mathbf{s}_{t}))].$
(2)

Here, $\pi(\cdot|\mathbf{s}_{t})$ is a probability distribution over actions
and $f_{\pi}(\cdot|\mathbf{s}_{t})$ is the density function of the policy
$\pi$. $\alpha$ is the temperature parameter controlling the entropy in the
policy.

SAC provides an actor-critic framework where policy is separately represented
by the actor and critic only helps in improving the actor, thus limiting its
role only to training. We use CNNs to represent both actor and critic, and
instead of using a single Q-value network for the critic, we use two Q-value
networks and take their minimum to better estimate Q-value as proposed in
[21]. To stabilize the learning, we use two more neural-network to represent
target Q-values for each critic network, as described in DQN [16]. $\phi$,
$\theta_{1}$, $\theta_{2}$, $\hat{\theta}_{1}$ and $\hat{\theta}_{2}$
represents parameters of policy network, 2 Q-value networks and 2 target
Q-value networks for critic respectively. Therefore in total, we use 5 CNNs to
implement the SAC algorithm.

Our CNN architecture is similar to [20] except for the multi-head part, which
is a single layer neural-network for each head. Q-value network takes state
and action as input and directly gives Q-value. We use _tanh-guassian_ policy,
as used in [20]. Since we use stochastic policy thus, the policy network takes
the state as input and outputs the mean and standard deviation of the gaussian
distribution of each action. Action is then sampled from this distribution and
passed through _tanh_ function to bound actions between $(-1,1)$. Equation (3)
defines the target Q-value which is then used in (4) to calculate Q-loss for
each critic networks. Equation (5) defines policy-loss for actor network.
These losses are then used to update actor and critic networks using adam [11]
optimisation algorithm.

$\hat{Q}_{\bar{\theta}_{1},\bar{\theta}_{2}}(\mathbf{s}_{t+1},\mathbf{a}_{t+1})=\mathbf{r}_{t}\\\
+\gamma\mathbb{E}_{(\mathbf{s}_{t+1}\sim\mathcal{D},\mathbf{a}_{t+1}\sim\pi_{\phi}(\cdot|\mathbf{s}_{t+1}))}[\\\
\text{
min}[Q_{\bar{\theta}_{1}}(\mathbf{s}_{t+1},\mathbf{a}_{t+1}),Q_{\bar{\theta}_{2}}(\mathbf{s}_{t+1},\mathbf{a}_{t+1})]\\\
-\alpha\text{log}(\pi_{\phi}(\mathbf{a}_{t+1}|\mathbf{s}_{t+1}))]$ (3)
$\displaystyle J_{Q}({\theta_{i}})$
$\displaystyle=\frac{1}{2}\mathbb{E}_{(\mathbf{s}_{t},\mathbf{a}_{t})\sim\mathcal{D}}[(\hat{Q}_{\bar{\theta}_{i},\bar{\theta}_{2}}(\mathbf{s}_{t+1},\mathbf{a}_{t+1})-Q_{\theta_{1}}(\mathbf{s}_{t},\mathbf{a}_{t}))^{2}].$
(4)
$J_{\pi}(\phi)=\mathbb{E}_{(\mathbf{s}_{t}\sim\mathcal{D},\mathbf{a}_{t}\sim\pi_{\phi}(\cdot|\mathbf{s}_{t}))}[\alpha\text{log}(\pi_{\phi}(\mathbf{a}_{t}|\mathbf{s}_{t}))\\\
-\text{
min}[Q_{\theta_{1}}(\mathbf{s}_{t},\mathbf{a}_{t}^{\pi}),Q_{\theta_{2}}(\mathbf{s}_{t},\mathbf{a}_{t}^{\pi})]]$
(5)

Here, $i\in\\{1,2\\}$, $\mathbf{a}_{t}^{\pi}$ is the action sampled from
policy $\pi_{\phi}$ for state $\mathbf{s}_{t}$ and $\mathcal{D}$ represents
the current task data. For offline-RL, we use the non-Lagrange version of the
conservative Q-learning (CQL) approach proposed in [13] as it only requires
adding a regularisation loss to already well-established continuous RL methods
like Soft-Actor Critic. This loss function is defined in (6).

$J_{Q}^{\text{total}}({\theta_{i}})=J_{Q}({\theta_{i}})\\\
+\alpha_{\text{cql}}\mathbb{E}_{\mathbf{s}_{t}\sim\mathcal{D}}[\text{log}\sum\limits_{\mathbf{a}_{t}}\text{exp}(Q_{\theta_{i}}(\mathbf{s}_{t},\mathbf{a}_{t}))-\mathbb{E}_{\mathbf{a}_{t}\sim\mathcal{D}}[Q_{\theta_{i}}(\mathbf{s}_{t},\mathbf{a}_{t})]]$
(6)

Here, $i\in\\{1,2\\}$, $\alpha_{\text{cql}}$ control the amount of CQL-loss to
be added to Q-loss to penalize actions that are too far away from the existing
trajectories, thus keeping the policy conservative in the sense of
exploration.

###  III-B Applying Synaptic Intelligence in Offline RL

Synaptic intelligence is a regularisation based algorithm proposed in [24] for
sequential task learning. It regularises the loss function of a task with a
quadratic loss function as defined in (7) to reduce catastrophic forgetting.

$L_{\mu}=\sum\limits_{k}\Omega^{\mu}_{k}(\tilde{\phi}_{k}-\phi_{k})^{2}$ (7)

Here, $L_{\mu}$ is the SI loss for the current task being learned with index
$\mu$, $\phi_{k}$ is $k$-th weight of the policy network, and
$\tilde{\phi}_{k}$ is the reference weight corresponding to policy network
parameters at the end of the previous task. $\Omega^{\mu}_{k}$ is per-
parameter regularisation strength for more details on how to calculate
$\Omega^{\mu}_{k}$ refer to [24]. SI algorithm penalizes neural network
weights based on their contributions to the change in the overall loss
function. Weights that contributed more to the previous tasks are penalized
more and thus do not deviate much from their original values, while other
weights help learn new tasks. SI defines importance of weights as the sum of
the gradients over the training trajectory, as this approximates contribution
to the reduction in the overall loss function. We use a similar approach to
apply SI to Offline-RL as presented in [22]. Although the authors didn’t use
SI or offline-RL, the approach is similar to applying any regularisation based
continual learning method for the actor-critic RL framework. We regularise the
actor to reduce forgetting on previous tasks while learning new tasks using
offline reinforcement learning. We add quadratic loss as defined in [24] to
the policy-loss term in the SAC-CQL algorithm. So now over-all policy-loss
becomes as described in (8)

$J_{\pi}^{\text{total}}(\phi)=J_{\pi}(\phi)+cL_{\mu}$ (8)

Here, $c$ is regularisation strength. Another aspect of continual learning is
finding a way to provide the current task index to the neural network. There
are many approaches to tackle this problem, from 1-hot encoding to recognizing
the task from context. We chose the most straightforward option of a multi-
head neural network. Each head of the neural network represents a separate
task. Therefore we simply select the head for a given task. For training each
task we keep a fixed compute budget of 100k of gradient-steps.

##  IV EXPERIMENTS, RESULTS AND DISCUSSION

In this section we first discuss the RL environment setup and provide details
of data collection for offline RL. Further, we evaluate performance of SI with
varying number of object configurations and densities for different task
ordering.

###  IV-A Experimental Setup

Our experimental setup is based on a simulated environment, Roboverse, used in
[20]. It is a GYM [3] like environment based upon open-source physics
simulator py-bullet [5]. We collected data for three tasks using this
simulated environment.

Object Space and Tasks: We define object space as a subset of the workspace of
the robot where the target object of the task is to be placed. In our case, it
is a rectangular area on the table in front of the robot. The target object is
randomly placed in the object-space when initializing the task. We selected
the following 3 tasks for all our experiments with some similarities.

1) _Press Button:_ Button is placed in the object-space. The objective of the
task is to press the button. This is easiest task as the robot only needs to
learn to reach the object.

2) _Pick Shed:_ The objective of this task is to pick the object successfully.
Thus, robot also needs to learn to close the gripper apart from reaching the
object. Figure 2(a) shows the object space of task pick-shed.

3) _Open Drawer:_ The objective of this task is to open the drawer.

(a) (b)

Figure 2: Object space and reward distribution for pick-shed task with area of
size of $1000\text{ cm}^{2}$ and density of 20 object configurations per
$\text{ cm}^{2}$. (a) Object Space. (b) Reward distribution (cumulative reward
along sample trajectories) of pick-shed task (area=1000, density=20). Red
semicircle on top represents the robot location

Data Collection: For each task we collect 6 datasets by varying the area
($\text{40, 360 and 1000 cm}^{2}$) and density ($\text{10 and 20 object
configurations per cm}^{2}$) of object-space. Each episode consists of 20
steps and each step is a typical tuple
$<\mathbf{s}_{t},\mathbf{a}_{t},\mathbf{r}_{t},\mathbf{s}_{t+1}>$ used in
reinforcement learning. We use simple but accurate policies to collect data.
Accuracy of these data collection policies is above $80\%$. Figure 2(b) shows
how reward is distributed across object space for pick-shed task. Each dot
represents a trajectory, and the color represents the total reward for each
trajectory. It can be seen, when the object is placed closer to the robot, the
reward is high as task is completed in few steps, while it becomes low as the
object moves away.

###  IV-B Empirical Results and Analysis

We performed a total of 72 experiments. We performed sequential learning on
two task (doublets). Six doublets are possible using data collected for three
tasks. These are button-shed, button-drawer, shed-button, shed-drawer, drawer-
shed, and drawer-button. For each doublet sequence, we perform 2 sets of
experiments, one with SI regularisation and another without SI regularisation.
Each set contains 6 experiments by varying area and density of object-space.
Apart from these 72 experiments, we also trained the agent for single tasks
using SAC-CQL for reference baseline performance to evaluate forward transfer.
We do behaviour-cloning for the initial 5k steps to learn faster as we have
limited compute budget. We use metrics mentioned in [22] for evaluating the
performance of a continual learning agent. Each task is trained for
$\Delta=100K$ steps. The total number of tasks in a sequence is $N=2$. Total
steps $T=2\cdot\Delta$. The $i\text{-th}$ task is train from
$t\in[(i-1)\cdot\Delta,i\cdot\Delta]$.

Task Accuracy: We evaluate the agent after every 1000 training steps by
sampling 10 trajectories from the environment for each task. The accuracy of
the agent for a task is defined as the number of successful trajectories out
of those 10 trails. Figure 3 shows the accuracy of three task-sequences
(button-shed, button-drawer, drawer-button) over the complete training period
of 200k steps for the area size of $40cm^{2}$ with density of 10 and 20 object
configurations per $cm^{2}$. Top row represents sequential learning with SI
while bottom row represents sequential learning without SI. SI is found to be
working better as evident by overlapping Task-1 and Task-2 accuracy. We
observed that SI was most helpful in button-shed task doublet due to
overlapping nature of these tasks as both these tasks require reaching the
object. This shows benefit of using SI for overlapping tasks

(a) Task Accuracy (area=360, density=10)

(b) Task Accuracy (area=360, density=20)

Figure 3: Task accuracy for tasks button-shed, button-drawer and drawer-button
(area=360, density=10,20). Top row is with SI, bottom row is without SI

Forgetting: It measures decrease in accuracy of the task as we train more
tasks and defined as $F_{i}:=p_{i}(i.\Delta)-p_{i}(T)$. Here,
$p_{i}(t)\in[0,1]$ is success rate of task $i$ at time $t$. Figure 4 shows the
forgetting of Task-1 after training Task-2. We can see that SI performed
better or equal in all cases. In fact, in some cases, like button-shed
forgetting is negative, which means the performance of Task-1 improved after
training on Task-2. This indicates knowledge transfer from Task-1 to Task-2.
This phenomenon is not seen in case of sequential learning without SI. This
clearly indicates that SI helps in reducing catastrophic forgetting. No
significant trends are observed in variation of object-space area but
forgetting increased with the increase in object-space density. This might be
due to the limited compute budget (100K) per task as tasks with more area size
and density would require more training to show good results.

Figure 4: Forgetting Matrix. Top row is with SI regularisation, bottom row is
without regularisation Figure 5: Forward Transfer Matrix

Forward Transfer: It measures knowledge transfer by comparing the performance
of a given task when trained individually versus learning the task after the
network is already trained on previous tasks and defined as

$FT_{i}:=\frac{\text{AUC}_{i}-\text{AUC}_{i}^{b}}{1-\text{AUC}_{i}^{b}},$ (9)

where
$\text{AUC}_{i}=\frac{1}{\Delta}\int_{(i-1)\cdot\Delta}^{i\cdot\Delta}p_{i}(t)\text{d}t$
represents area under the accuracy curve of task $i$ and
$\text{AUC}_{i}^{b}=\frac{1}{\Delta}\int_{0}^{\Delta}p_{i}^{b}(t)\text{d}t$,
represents area under curve of the reference baseline task. $p_{i}^{b}(t)$
represent reference baseline performance. Figure 5 shows forward transfer for
Task-2 after it is trained on Task-1. We use single-task training performance
as the reference for Task-2 while evaluating forward transfer. We observed
that in most cases, training without SI gives a better transfer ratio than
training with SI. This may be because of two reasons. Firstly, due to the high
value of SI regularisation strength (which is set to 1 for all cases), this
restricts movement of weights from the solution of the previous task. This can
also be noticed in the form of reduced accuracy levels of Task-2 in the Figure
3. The accuracy level of Task-2 are lower as compared to its non-SI
counterpart. Although, high regularisation strength helps in reducing
catastrophic forgetting but also hinders the ability to learn new-task thus
reducing forward-transfer. This highlights the problem of stability-
plasticity, any method which tries to make learning more stable to reduce
forgetting inadvertently also restricts the flexibility of the connectionist
model to learn a new task.

Training Time: Apart from these metrics, we observed that, agent requires on
an average 14k, 10k, and 16k steps to achieve its first success on Task-2 when
trained directly, sequentially without SI, and sequentially with SI,
respectively. This means that the agent learns the task faster when trained
sequentially without adding SI regularisation but a little slower when trained
sequentially with SI regularisation than directly training the task. This
shows another benefit of sequential learning over single task-learning.

Another interesting observation we made in the case of shed-button (area 360,
density 20) task. While training for Task-1 (pick shed) agent showed some
success on Task-2 (press button) even before getting any success on Task-1
itself. This might be due to the nature of the tasks, as the trajectory of the
press button task is common for other task. Therefore, agent has tendency to
acquire knowledge for similar tasks. This may also be the result of behaviour-
cloning for the initial 5k steps, where the agent tries to mimic the data
collection policy for a few initial training steps. Also, we observed that
increasing the object space area helps in knowledge transfer, which can be
seen by the increase in average forward transfer with area size.

##  V CONCLUSION AND FUTURE WORK

We investigated catastrophic forgetting and forward knowledge transfer for
sequentially learning image-based robotic manipulation tasks by combining a
continual learning approach with offline RL framework. We use SAC-CQL as an
offline deep RL algorithm with synaptic intelligence (SI) to mitigate
catastrophic forgetting. Multi-headed CNN was used to provide knowledge of the
current Task-index to the neural-network. We performed a series of experiments
with different task combinations and with a varying number of object
configurations and densities. We found that SI is useful for reducing
forgetting but showed a limited forward transfer of knowledge.

We also found that the ordering of tasks significantly affects the performance
of sequential task learning. Therefore, tasks may be chosen in a manner so
that the previous task helps in learning the next task as the complexity of
tasks increases. This calls for exploring curriculum learning for sequential
tasks. Experiments also suggests the importance of prior knowledge for
continual learning. Agent trained only with state-action pairs of large number
of diverse tasks (even without reward), may provide a better prior knowledge.
Future work will also focus on training tasks with more number of steps to
explore more interesting patterns.

## References

  * [1] Lars Berscheid, Pascal Meißner and Torsten Kröger  “Self-supervised learning for precise pick-and-place without object model”  In _IEEE Robotics and Automation Letters_ 5.3 IEEE, 2020, pp. 4828–4835
  * [2] Greg Brockman et al.  “OpenAI Gym”  In _CoRR_ abs/1606.01540, 2016  arXiv: http://arxiv.org/abs/1606.01540
  * [3] Greg Brockman et al.  “Openai gym”  In _arXiv preprint arXiv:1606.01540_ , 2016
  * [4] Massimo Caccia et al.  “Task-Agnostic Continual Reinforcement Learning: In Praise of a Simple Baseline”  In _arXiv preprint arXiv:2205.14495_ , 2022
  * [5] Coumans Erwin and Bai Yunfei  “PyBullet a python module for physics simulation for games”  In _PyBullet_ , 2016
  * [6] Robert M French  “Catastrophic forgetting in connectionist networks”  In _Trends in cognitive sciences_ 3.4 Elsevier, 1999, pp. 128–135
  * [7] Ian J Goodfellow et al.  “An empirical investigation of catastrophic forgetting in gradient-based neural networks”  In _arXiv preprint arXiv:1312.6211_ , 2013
  * [8] Abhishek Gupta et al.  “Reset-free reinforcement learning via multi-task learning: Learning dexterous manipulation behaviors without human intervention”  In _2021 IEEE International Conference on Robotics and Automation (ICRA)_ , 2021, pp. 6664–6671  IEEE
  * [9] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel and Sergey Levine  “Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor”  In _International conference on machine learning_ , 2018, pp. 1861–1870  PMLR
  * [10] Dmitry Kalashnikov et al.  “Scaling Up Multi-Task Robotic Reinforcement Learning”  In _5th Annual Conference on Robot Learning_ , 2021
  * [11] Diederik P Kingma and Jimmy Ba  “Adam: A method for stochastic optimization”  In _arXiv preprint arXiv:1412.6980_ , 2014
  * [12] James Kirkpatrick et al.  “Overcoming catastrophic forgetting in neural networks”  In _Proceedings of the national academy of sciences_ 114.13 National Acad Sciences, 2017, pp. 3521–3526
  * [13] Aviral Kumar, Aurick Zhou, George Tucker and Sergey Levine  “Conservative q-learning for offline reinforcement learning”  In _Advances in Neural Information Processing Systems_ 33, 2020, pp. 1179–1191
  * [14] Robert Lee et al.  “Learning arbitrary-goal fabric folding with one hour of real robot experience”  In _arXiv preprint arXiv:2010.03209_ , 2020
  * [15] Arun Mallya and Svetlana Lazebnik  “Packnet: Adding multiple tasks to a single network by iterative pruning”  In _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition_ , 2018, pp. 7765–7773
  * [16] Volodymyr Mnih et al.  “Playing atari with deep reinforcement learning”  In _arXiv preprint arXiv:1312.5602_ , 2013
  * [17] Ashvin Nair et al.  “Combining self-supervised learning and imitation for vision-based rope manipulation”  In _2017 IEEE international conference on robotics and automation (ICRA)_ , 2017, pp. 2146–2153  IEEE
  * [18] Andrei A Rusu et al.  “Policy distillation”  In _arXiv preprint arXiv:1511.06295_ , 2015
  * [19] Andrei A Rusu et al.  “Progressive neural networks”  In _arXiv preprint arXiv:1606.04671_ , 2016
  * [20] Avi Singh et al.  “Cog: Connecting new skills to past experience with offline reinforcement learning”  In _arXiv preprint arXiv:2010.14500_ , 2020
  * [21] Hado Van Hasselt, Arthur Guez and David Silver  “Deep reinforcement learning with double q-learning”  In _Proceedings of the AAAI conference on artificial intelligence_ 30.1, 2016
  * [22] Maciej Wołczyk et al.  “Continual world: A robotic benchmark for continual reinforcement learning”  In _Advances in Neural Information Processing Systems_ 34, 2021, pp. 28496–28510
  * [23] André Yuji Yasutomi, Hiroki Mori and Tetsuya Ogata  “A peg-in-hole task strategy for holes in concrete”  In _2021 IEEE International Conference on Robotics and Automation (ICRA)_ , 2021, pp. 2205–2211  IEEE
  * [24] Friedemann Zenke, Ben Poole and Surya Ganguli  “Continual learning through synaptic intelligence”  In _International Conference on Machine Learning_ , 2017, pp. 3987–3995  PMLR


# A Splitting Approach to Dynamic Mode Decomposition of Nonlinear Systems

Jovan Žigić
Dominican University New York, Orangeburg NY 10962, USA

###### Abstract

Reduced-order models have long been used to understand the behavior of
nonlinear partial differential equations (PDEs). Naturally, reduced-order
modeling techniques come at the price of computational accuracy for a decrease
in computation time. Optimization techniques are studied to improve either or
both of these objectives and decrease the total computational cost of the
problem. This paper focuses on the dynamic mode decomposition (DMD) applied to
nonlinear PDEs with periodic boundary conditions. It provides a study of a
newly proposed optimization framework for the DMD method called the Split DMD.

AMS subject classifications: Primary 65K10, Secondary 65M22.

##  1 Introduction

The Navier-Stokes (NS) equations are the primary mathematical model for
understanding the behavior of fluids. The existence and smoothness of the NS
equations [8] is considered to be one of the most important open problems in
mathematics, and challenges in their numerical simulation is a barrier to
understanding the physical phenomenon of turbulence. Due to the difficulty of
studying this problem directly, problems in the form of nonlinear partial
differential equations that exhibit similar properties to the NS equations are
studied as preliminary steps towards building a wider understanding of the
field. In fact, the use of the proper orthogonal decomposition method to build
reduced-order models for fluids was accelerated with the publication of the
first edition of the monograph “Turbulence, Coherent Structures, Dynamical
Systems and Symmetry” [9].

Reduced-order models has since evolved as its own discipline with applications
to control [5][4][6], optimization [16][1], and uncertainty quantification
[7]. Naturally, reduced-order modeling techniques come at the price of
computational accuracy for a decrease in computation time. Optimization
techniques are studied to improve either or both of these objectives and
decrease the total computational cost of the problem.

##  2 Split Dynamic Mode Decomposition

By definition of the term [17] and observation of simulated models governed by
nonlinear PDEs, bifurcation present in system dynamics greatly alters system
behavior over time. When approaching such a problem with DMD, this bifurcation
effect leads to issues arising with selecting DMD modes that represent both
transient dynamics and states of equilibrium that follow. This phenomena is a
natural obstacle to both the DMD algorithm [11] and the Levenberg-Marquardt
algorithm [12] [13] crucial to the optimized DMD (OD) method [3] [2].

In the DMD algorithm, the primary issue generated by effects of bifurcation is
selecting DMD modes that accurately represent the dominant coherent
structures, or pattern of behavior, of the dynamical system over the time
interval. In a chaotic dynamical system, this pattern between the transient
state and equilibrium state is not explicit. Thus, the linear problem that
provides a solution for the low-rank Koopman operator is forced to compensate
for the information provided throughout the entire interval, and consequently
choose DMD modes that represent two distinct patterns. Thus, the fact that the
resulting reduced-order system modeled by the DMD modes provides one pattern,
or one set of coherent structures, to represent multiple distinct patterns is
inherently flawed.

In the Levenberg-Marquardt algorithm (LMA), the primary issue generated by
effects of transient states is the increased difficulty to solve the nonlinear
least squares problem [15], which directly contributes to computational
complexity at the expense of time. Since approximating nonlinear PDEs presents
nonconvex optimization problems through the LMA, globalization strategies such
as trust region or line search methods must be employed to find stationary
points. Generally, less iterations are required to solve the DMD problems when
they contain simpler dynamics. However, when solving subproblems within each
iteration becomes an increasingly difficult task, the cost of computational
time becomes significant to the point of computational infeasibility.

This section proposes the split DMD algorithm as a solution to the issue of
selecting DMD or OD modes for nonlinear PDEs whose coherent structures vary
over time. The split DMD algorithm uses a simple modification prior to the DMD
routine to choose a domain for evaluation without changing the existing DMD
method of computing system modes. The contribution to the existing DMD
procedure comes from splitting the entire time interval into several
subintervals before computing the low-rank Koopman modes. The split lines,
determining the boundaries of each subinterval, are selected by the n-split
algorithm described in the next section. The $n$-split algorithm produces $n$
number of subintervals by a recursive method.

The goal of the algorithm is to separate the simulation interval into
subintervals that have different ranges of data values, and are hypothesized
to represent the time intervals with distinct system coherent structures. By
heuristic reasoning, $n$ number of split lines are chosen as an initial guess
for range comparison. These split lines are defined by the set of points in
time:

$\displaystyle t_{split}=\left\\{t_{0},t_{1},t_{2},\dotsc,t_{n}\right\\}$ (1)

where a time interval $[0,T]$ implies $t_{0}=0$ and $t_{n}=T$, so that the
data (or system values) $Z(x,t)$ with respect to time $t$ and space $x$ is
separated into subintervals:

$Z_{split}=\left\\{Z(x,[t_{0},t_{1}]),Z(x,[t_{1},t_{2}]),\dotsc,Z(x,[t_{n-1},t_{n}])\right\\}$
(2)

By random selection, a line of space $x$ denoted $x_{test}$ is chosen to
compare the system behavior across subintervals of time. Two tests are used to
determine whether or not to merge two adjacent intervals. These involve either
the difference in data value range or the length of the subinterval.

##  3 Implementation of the Split DMD Algorithm

For the first test, called the $\varepsilon$ test, the necessary difference in
range to retain a split line selection is data that exceeds a lower bound
tolerance $\varepsilon$, which is heuristically chosen as a fraction of the
data range. Furthermore, the minimum change $\varepsilon$ in data is inspected
at both upper and lower bounds of the subinterval data ranges, as illustrated
in the following figures:

$t$$z$$t_{k}$$t_{k-1}$$t_{k+1}$$<\varepsilon$$>\varepsilon$

Figure 1: Retained Split Line $t_{k}$

$t$$z$$t_{k}$$t_{k-1}$$t_{k+1}$$<\varepsilon$$<\varepsilon$

Figure 2: Discarded Split Line $t_{k}$

In Figures 1 and 2, the two-dimensional shapes represent the change in range
of system values in time at a spatial point $x$. For an example of the
$\varepsilon$ test, if the data range is $[0,1]$, a chosen 10% tolerance or
$\varepsilon=0.1$ change in either bound between adjacent subintervals is the
minimum amount required to retain the split line. Otherwise, if the adjacent
subintervals for $k\in\left\\{1,2,\dotsc,{n-1}\right\\}$ satisfy both of the
following relationships:

$\left|\max_{t\in[t_{k-1},t_{k}]}Z(x_{test},t)-\max_{t\in[t_{k},t_{k+1}]}Z(x_{test},t)\right|<\varepsilon$
(3)
$\left|\min_{t\in[t_{k-1},t_{k}]}Z(x_{test},t)-\min_{t\in[t_{k},t_{k+1}]}Z(x_{test},t)\right|<\varepsilon$
(4)

then the split line $t_{k}$ is identified as unnecessary and discarded from
the set of split lines $t_{split}$.

For the second test, called the $\delta$ test, the necessary length of a
subinterval to retain a split line selection is a lower bound tolerance
$\delta$, which is heuristically chosen as a fraction of the data time
interval. As in the first test, an illustration of whether or not a split line
is retained or discarded is portrayed by the following figures:

$t$$z$$t_{k}$$t_{k-1}$$t_{k+1}$$>\delta$

Figure 3: Retained Split Line $t_{k}$

$t$$z$$t_{k}$$t_{k-1}$$t_{k+1}$$<\delta$

Figure 4: Discarded Split Line $t_{k}$

For an example of the $\delta$ test, if the data range is $[0,1]$, a chosen
10% tolerance or $\delta=0.1$ subinterval length $[t_{k-1},t_{k}]$ for
$k\in\left\\{1,2,\dotsc,{n-1}\right\\}$ is the minimum amount required to
retain the split line. Otherwise, if the following relationship is satisfied:

$\left|t_{k}-t_{k-1}\right|<\delta$ (5)

then the split line $t_{k}$ is identified as unnecessary and discarded from
the set of split lines $t_{split}$. The value of the second subinterval length
$\left|t_{k+1}-t_{k}\right|$ stemming from the $\varepsilon$ test is only
evaluated for $k=n-1$, since each adjacent subinterval pair is evaluated
across the entire data time interval proceeding from the initial time $t=0$ to
the final time $t=T$.

Recursion is performed until a sufficient number of iterations is reached. In
the case that all chosen splits $t_{k}$ for
$k\in\left\\{1,2,\dotsc,n-1\right\\}$ are discarded, the method restarts with
a different selection of $t_{split}$ values by adding in one more split than
was used in the previous iteration. Consequently, for data displaying one
distinct pattern across the time interval, it is expected that no split lines
within the interval are necessary to be retained as long as the $\varepsilon$
and $\delta$ tests are satisfied to the given tolerances.

When a split line $t_{k}$ is retained, or the split line $t_{k}$ has “passed”
the $\varepsilon$ and $\delta$ tests, the algorithm is run recursively within
the subinterval $[t_{k-1},t_{k}]$ in order to find any further distinct
patterns in the data subinterval.

After completing the maximum number of iterations, the algorithm outputs
$t_{split}$ with split lines that satisfy the $\varepsilon$ and $\delta$ tests
for a chosen point in space $x$. For added robustness, the algorithm is run
for a number of different $x_{test}$ values, and the resulting $t_{split}$
with split lines that consistently satisfy the $\varepsilon$ and $\delta$
tests for these spatial points are chosen to define the subintervals for the
DMD routine that follows.

A more explicit breakdown of the $n$-split algorithmic process is given by the
following pseudo-code:

Table 1: $n$-split Algorithm

0: dataset $Z$, split lines $t_{split}$, maximum iterations $M$

0: split lines $t_{split}$, iteration count

while iterations $<M$ do

Choose $x_{k}$ randomly from the spatial nodes of $Z$

Set $s$ as number of split lines

Set $\varepsilon$ as fraction of $\left\|Z(x_{k},[0,T])\right\|_{\infty}$ and
$\delta$ as fraction of $[0,T]$

for $k=1$ to $s-1$ do

if $\displaystyle\max\left\\{\begin{matrix}\left|\max
Z(x_{k},[t_{k-1},t_{k}])-\max Z(x_{k},[t_{k}t_{k+1}])\right|\\\
\left|\min_{t\in[t_{k-1},t_{k}]}Z-\min_{t\in[t_{k},t_{k+1}]}Z\right|\end{matrix}\right\\}>\epsilon$
then

if $\left|t_{k}-t_{k-1}\right|>\delta$ then

Retain split line $t_{k}$ and add a new split line in $[t_{k-1},t_{k}]$

Run $n$-split algorithm on subinterval $Z(x,[t_{k-1},t_{k}])$ and add new
split lines as required

Increase iteration count by output in previous step

else

return

end if

if $k=s-1$ and $\left|t_{k+1}-t_{k}\right|>\delta$ then

similar steps as previous if statement

end if

end if

end for

if no split lines are retained then

Increase iteration count by 1 and choose a new guess for $t_{split}$

Run $n$-split algorithm on $Z$ and replace current split lines by output

Increase iteration count by output in previous step

end if

Discard redundant split lines

end while

Thus, stepping out of the $n$-split algorithm, the split DMD algorithmic
process is given by the following pseudo-code:

Table 2: Split DMD Algorithm

0: rank of approximation $r$, dataset $Z$

0: Order-$r$ approximations $Z_{DMD}$, Order-$r$ DMD modes $\Phi^{DMD}$

Choose split lines $t_{split}$ heuristically and set maximum number of
iterations $M$

for enough iterations to reach steady-state solution do

Run $n$-split algorithm on $Z$

end for

Retain steady-state selection of split lines $t_{split}$

Set $s$ as number of split lines

for $k=1$ to $s$ do

Run DMD algorithm to output DMD modes $\Phi_{k}^{DMD}$ and rank-$r$
$Z_{DMD}(x,[t_{k-1},t_{k}])$

end for

Set $\Phi^{DMD}=\left\\{\Phi_{k}^{DMD}\right\\}_{k=1}^{s}$

return $Z_{DMD},\Phi^{DMD}$

##  4 Results

The main purpose of this study is to compare the effectiveness of the split
DMD method, i.e. selecting DMD modes in subdomains of the entire solution,
relative to DMD methods that select DMD modes across the entire domain on a
nonlinear PDE problem. A simplified version of the proposed $n$-split
algorithm in Section 2 was implemented for this study, by using an evenly-
spaced initial guess for the split lines and a heuristic choice for the number
of initial split lines.

###  4.1 Comparison of Split OD and OD

The split DMD algorithm was applied using both OD and standard DMD to the
following form of the Kuramoto-Sivashinsky (KS) equation:

$\displaystyle\frac{\partial w}{\partial t}+\frac{\partial^{4}w}{\partial
x^{4}}=-2w\frac{\partial w}{\partial x}-\frac{\partial^{2}w}{\partial x^{2}}$
(6)

The stabilizing fourth-order and destabilizing second-order terms in the KS
equation mimic the Navier-Stokes’ energy behavior [9]. An important feature of
the KS equation is the bifurcation parameter $L$, the length of the periodic
domain where the model is studied, that determines the behavior of the
dynamical system. To perform a non-dimensionalization of the KS equation to a
unit periodic domain $\bar{x}=\frac{x}{L}$, define
$\varepsilon=\frac{1}{L^{2}}$ so that the non-dimensional model is

$\displaystyle\frac{\partial w}{\partial
t}+\varepsilon^{2}\frac{\partial^{4}w}{\partial\bar{x}^{4}}=-2\varepsilon
w\frac{\partial
w}{\partial\bar{x}}-\varepsilon\frac{\partial^{2}w}{\partial\bar{x}^{2}}$ (7)

This exposes the relationships between length versus nonlinearity and
stabilizing versus destabilizing terms. Solutions to the non-dimensional
equations above are considered in the domain

$(\bar{x},t)=\mathbb{T}\times(0,T)$

with periodic domain $\mathbb{T}=(0,1)$, different values of the length
parameter $L$ (and thus $\varepsilon$) and the final time $T$. These equations
are solved with periodic boundary conditions

$\displaystyle w(0,t)=w(1,t),\leavevmode\nobreak\ \leavevmode\nobreak\
w_{{\bar{x}}}(0,t)=w_{{\bar{x}}}(1,t),$ $\displaystyle\leavevmode\nobreak\
\leavevmode\nobreak\
w_{\bar{x}\bar{x}}(0,t)=w_{\bar{x}\bar{x}}(1,t),\leavevmode\nobreak\
\leavevmode\nobreak\
w_{\bar{x}\bar{x}\bar{x}}(0,t)=w_{\bar{x}\bar{x}\bar{x}}(1,t)$

and initial condition

$\displaystyle w(\bar{x},0)=\frac{\sin(4\pi\bar{x})}{\sqrt{\varepsilon}}$ (8)

The dynamical behavior of a system modeled by the KS equation is summarized in
the following table [9].

Table 3: Dynamical Behavior of Kuramoto-Sivashinsky

Length $L$ | $\varepsilon$ | Solution Behavior
---|---|---
12.5664 $\approx 4\pi$ | 0.00633257 | Bifurcation
12.8767 | 0.00603102 | Heteroclinic Bifurcation
13.1403 | 0.00579148 | Hopf Bifurcation
402.2590 $\approx 128\pi$ | 0.00000618 | “Chaos”

The periodic domain lengths $L$ initially tested were 12.6, 13.2, and 402.3.
The following figures portray a visual inspection of the improvement of
reduced-order approximation to the full model due to the splitting procedure
of the split DMD algorithm:

Figure 5: KS OD, $L=12.6$

Figure 6: KS 4-split OD, $L=12.6$

Figure 7: KS OD, $L=13.2$

Figure 8: KS 4-split OD, $L=13.2$

Figure 9: KS OD, $L=402.3$

Figure 10: KS 10-split OD, $L=402.3$

The relative error of the split OD models provided in the preceding figures
shows a clear improvement in comparison to the standard OD models with respect
to the finite element solution. Without splitting, it is evident that OD does
not provide a representation of the data that captures the coherent structures
throughout the time interval.

The following table provides a comparison of the split OD ROMs versus the OD
ROMs for $t=400$, with information on each reduced-order model concerning:

  1. 1.

Number of splits used: $n$

  2. 2.

Length of the periodic time interval (bifurcation parameter): $L$

  3. 3.

Rank of the approximation: $r$

  4. 4.

Computation error: $\left\|r(t)\right\|_{2}$

  5. 5.

Computation time (in seconds)

Table 4: Split OD ROM vs OD ROM, $t=400$

$n$-Split | Length | Rank | Computation error | Time
---|---|---|---|---
0 | 12.60 | 11 | 0.539273074481482 | $1.656649834\times 10^{2}$
4 | 12.60 | 11 | 0.086861903201658 | $1.424075578\times 10^{2}$
0 | 13.20 | 13 | 0.450356679589967 | $2.765350954\times 10^{2}$
4 | 13.20 | 13 | 0.187951801209479 | $1.874434272\times 10^{2}$
0 | 402.3 | 13 | 18.071816203629595 | $3.078671183\times 10^{2}$
10 | 402.3 | 13 | 2.266344147532708 | $1.270328812\times 10^{2}$

The table suggests a substantial improvement in using the splitting approach.

###  4.2 Sensitivity of Initial Condition

For the chaotic dynamics ($L=402.3$), the sensitivity of the results in
Section 4.1 was tested by adding a random perturbation to the discretization
of the previous initial condition. Letting
$\beta(x)=\text{unif}\left(0,\frac{1}{20}\right)$, a uniformly distributed
random number at each spatial location (which is 161 in this case), the new
initial conditions are:

$\displaystyle w(x,0)=\frac{\sin(4\pi
x)}{\sqrt{\varepsilon}}+\beta(x),\leavevmode\nobreak\ \leavevmode\nobreak\
w_{x}(x,0)=\frac{4\pi\cos(4\pi x)}{\sqrt{\varepsilon}}$ (9)

The results are summarized in the following table, compared to the standard OD
and DMD models :

Table 5: Split DMD ROM vs Split OD ROM, $\beta$ shift

ROM |  $n$-Split | Length | Rank | Computation error | Time
---|---|---|---|---|---
DMD | 0 | 402.3 | 13 | 28.927186508313124 | 0.1634884
DMD | 10 | 402.3 | 13 | 5.798752827124941 | 0.17863
OD | 0 | 402.3 | 13 | 22.233228342958030 | $2.616457798\times 10^{2}$
OD | 10 | 402.3 | 13 | 2.739207339916917 | $1.858652532\times 10^{2}$

###  4.3 Sensitivity of Bifurcation Parameter

For the chaotic dynamics ($L=402.3$), the sensitivity of the results in
Section 4.1 was tested by shifting the bifurcation parameter to $L=402.35$.

The results are summarized in the following table:

Table 6: OD ROM vs Split OD ROM, $L=402.35$

$n$-Split | Length | Rank | Computation error | Time
---|---|---|---|---
0 | 402.35 | 13 | 19.038661517042843 | $2.773977651\times 10^{2}$
10 | 402.35 | 13 | 2.670265429612690 | $1.323419021\times 10^{2}$

###  4.4 Sensitivity of Split Lines

For the chaotic case in Section 4.1, the sensitivity of the results was tested
by shifting the split lines. The four sensitivity test cases for the 10-split
OD model are uniform shifts by $\pm 1,\pm 3$ seconds in each of the interior
time splits.

The results are summarized in the following table, with the “Shift” column
values measured in seconds:

Table 7: Split OD ROM with split line shifts

$n$-Split | Shift | Length | Rank | Computation error | Time
---|---|---|---|---|---
10 | +3 | 402.3 | 13 | 3.362777585745632 | $1.464054566\times 10^{2}$
10 | +1 | 402.3 | 13 | 4.197440908311128 | $1.340162133\times 10^{2}$
10 | -1 | 402.3 | 13 | 5.702330507412169 | $1.3498605\times 10^{2}$
10 | -3 | 402.3 | 13 | 4.645326326310565 | $1.42945084\times 10^{2}$

##  5 Discussion

As mentioned in Section 2, the OD method without splitting has issues
reconstructing systems with bifurcation present. Using the $n$-split
algorithm, the split OD method reconstructs the system in question to a level
of accuracy sufficient for recognizing the effects of bifurcation on the
system dynamics.

The standard DMD algorithm does not provide any useful results in the test
case in Section 4.1. By the $\beta$ shift in Section 4.2, it is evident that
the split DMD is able to provide results that sufficiently reconstruct the
system dynamics in certain cases. The anticipation is that the split DMD,
although computationally cheap, is limited in its potential solution accuracy.
In any case, accuracy of a DMD solution which rivals an OD solution for the KS
equation is not expected.

An increase in periodic length affects the solution to the standard OD ROM
significantly, whereas the 10-split OD model maintains a high level of
accuracy. Furthermore, the robustness to where the split occurs in the chaotic
dynamics implies that a sufficient number of splits is enough to produce an
accurate reconstruction of the finite element solution.

The main issue with the splitting approach is the inability to predict future
time states with the resulting modes. In the $L=13.2$ case, the solution for
each split can simply be “copied” in anticipation of the evident recurring
pattern. However, in the $L=402.3$ case, the chaotic dynamics (implying
differing coherent structures for each split) do not provide a gateway to
predict the the chaotic pattern.

The conclusion of these results is that a decrease in computation time and
increase in accuracy of solutions suggests that the $n$-split algorithm for
DMD methods is superior to the standard DMD and optimized DMD methods for
reconstructing a solution to the KS equation.

The numerical study applying the split DMD method to the KS equation suggests
that there is a benefit to using this splitting approach for modeling
dynamical systems with certain structural features. However, the work
described in this text did not factor in many possibilities that may come from
studying dynamical systems theory. Papers such as [10] provides a method to
choose modes over a longer time period that are not biased by the transient
dynamics. Page 52 of [11] provides a numerical method for inspecting the
attracting manifolds of the dynamical system. [14] and Chapter 3 of [11]
provide direction towards future state prediction based on Koopman theory.
Further study on these topics can lead to a more sophisticated approach
towards creating a metric for attractors when chaotic dynamics are involved.

## References

  * [1] Arian, Eyal, Marco Fahl, and Ekkehard W. Sachs. Trust-region proper orthogonal decomposition for flow control. Institute for Computer Applications in Science and Engineering, Hampton VA, 2000.
  * [2] Askham, Travis. (2017, July 12). duqbo/optdmd: optdmd v1.0.1 (Version v1.0.1). Zenodo. http://doi.org/10.5281/zenodo.826433
  * [3] Askham, Travis, and J. Nathan Kutz. “Variable projection methods for an optimized dynamic mode decomposition.” SIAM Journal on Applied Dynamical Systems 17.1 (2018): 380-416.
  * [4] Atwell, Jeanne A., Jeffrey T. Borggaard, and Belinda B. King. “Reduced order controllers for Burgers’ equation with a nonlinear observer.” International Journal of Applied Mathematics and Computer Science 11 (2001): 1311-1330.
  * [5] Atwell, Jeanne A., and Belinda B. King. “Proper orthogonal decomposition for reduced basis feedback controllers for parabolic equations.” textitMathematical and computer modelling 33.1-3 (2001): 1-19.
  * [6] Burns, John A., Belinda B. King, and Diana Rubio. “On the design of feedback controllers for a convecting fluid flow via reduced order modeling.” Proceedings of the 1999 IEEE International Conference on Control Applications (Cat. No. 99CH36328). Vol. 2. IEEE, 1999.
  * [7] Chen, Peng, and Christoph Schwab. “Model order reduction methods in computational uncertainty quantification.” Handbook of uncertainty quantification (2016): 1-53.
  * [8] Fefferman, Charles L. “Existence and smoothness of the Navier-Stokes equation.” The millennium prize problems 57 (2006): 67.
  * [9] Holmes, Philip, et al. Turbulence, coherent structures, dynamical systems and symmetry. Cambridge university press, 2012.
  * [10] Kassam, Aly-Khan, and Lloyd N. Trefethen. “Fourth-order time-stepping for stiff PDEs.” SIAM Journal on Scientific Computing 26.4 (2005): 1214-1233.
  * [11] Kutz, J. Nathan, et al. Dynamic mode decomposition: data-driven modeling of complex systems. Society for Industrial and Applied Mathematics, 2016.
  * [12] Levenberg, Kenneth. “A method for the solution of certain non-linear problems in least squares.” Quarterly of applied mathematics 2.2 (1944): 164-168.
  * [13] Marquardt, Donald W. “An algorithm for least-squares estimation of nonlinear parameters.” Journal of the society for Industrial and Applied Mathematics 11.2 (1963): 431-441.
  * [14] Mauroy, Alexandre, Y. Susuki, and I. Mezić. The Koopman Operator in Systems and Control. Springer, 2020.
  * [15] Perchikov, Nathan, and O. V. Gendelman. “Transient dynamics in strongly nonlinear systems: optimization of initial conditions on the resonant manifold.” Philosophical transactions. Series A, Mathematical, physical, and engineering sciences 376.2127 (2018).
  * [16] Qian, Elizabeth, et al. “A certified trust region reduced basis approach to PDE-constrained optimization.” SIAM Journal on Scientific Computing 39.5 (2017): S434-S460.
  * [17] Walter, Wolfgang. “Ordinary Differential Equations.” Differential and Integral Inequalities. Springer, Berlin, Heidelberg, 1970. 63-123.
  * [18] Žigić, Jovan. (2021, May 11). jovanzigic/OptimizedDMD_1DBurgers: Release (Version v1.0). Zenodo. http://doi.org/10.5281/zenodo.4750369
  * [19] Žigić, Jovan. (2021, May 11). jovanzigic/SplitDMD_KS: Release (Version v1.0). Zenodo. http://doi.org/10.5281/zenodo.4750392