# Replay across Experiments:
A Natural Extension of Off-Policy RL

Antiquus S. Hippocampus, Natalia Cerebro & Amelie P. Amygdale
Department of Computer Science
Cranberry-Lemon University
Pittsburgh, PA 15213, USA
<EMAIL_ADDRESS>
&Ji Q. Ren & Yevgeny LeNet
Department of Computational Neuroscience
University of the Witwatersrand
Joburg, South Africa
<EMAIL_ADDRESS>
Coauthor
Affiliation
Address
email Use footnote for providing further information about author (webpage,
alternative address)— _not_ for acknowledging funding agencies. Funding
acknowledgements go at the end of the paper.

###### Abstract

Replaying data is a principal mechanism underlying the stability and data
efficiency of off-policy reinforcement learning (RL). We present an effective
yet simple framework to extend the use of replays across multiple experiments,
minimally adapting the RL workflow for sizeable improvements in controller
performance and research iteration times. At its core, Replay across
Experiments (RaE) involves reusing experience from previous experiments to
improve exploration and bootstrap learning while reducing required changes to
a minimum in comparison to prior work. We empirically show benefits across a
number of RL algorithms and challenging control domains spanning both
locomotion and manipulation, including hard exploration tasks from egocentric
vision. Through comprehensive ablations, we demonstrate robustness to the
quality and amount of data available and various hyperparameter choices.
Finally, we discuss how our approach can be applied more broadly across
research life cycles and can increase resilience by reloading data across
random seeds or hyperparameter variations.

##  1 Introduction

In the last few years, reinforcement learning (RL) has transitioned from a
topic of academic study to a practical tool for the generation of controllers
across various real-world applications
degrave2022magnetic,Bellemare2020AutonomousNO,47493,9811912. These
advancements have been enabled, in part, by recent advancements in algorithms
improving data efficiency, robustness, and controller performance
haarnoja2019soft, lillicrap2016continuous, schwarzer23abbf. Nevertheless, many
problems remain hard to solve with RL. For instance, high-dimensional and
partial observations (e.g. egocentric camera views), high-dimensional action
spaces, or reward functions that provide inadequate learning signal can all
lead to poor asymptotic performance, high variance, low data efficiency, and
long training times.

A major challenge in RL remains the interaction between data collection and
learning. Unlike in the case of supervised learning where all data is
available and static, in online RL both the quality and quantity of data
available for policy or value function learning changes over the course of an
experiment. The difficulty of collecting suitable data, and the complex
interactions between data collection and function optimization leads to
various problems including failure to learn, instabilities, and the premature
convergence of function approximators early in learning
Nikishin2022ThePB,ash2020warm.

Experience replay Lin92 has become a popular component of most, modern off-
policy RL algorithms. Storing data collected over the course of an experiment
and continuously training policy and value functions with this growing dataset
can greatly increase data efficiency and stability of RL algorithms. It does
not, however, address issues around premature convergence of the function
approximators ash2020warm, nor does it directly provide a solution to reusing
data from prior experiments. In this paper, we investigate the possibility of
extending the use of replay and reusing data in an iterative setting. Our main
insight is that a minimal change to the RL workflow can greatly improve the
asymptotic performance of off-policy reinforcement learning algorithms. By
reusing interaction data from prior training runs it can further reduce the
overall experiment time and thus speed up research iterations.

A number of studies have previously investigated how prior data (including
expert data) can be used to kick-start RL training vecerik2017leveraging,
singh2020cog, nair2020accelerating, Walke2022DontSF, ball2023efficient. We
find that the simplest approach, mixing prior and online data with a fixed
ratio, is particularly effective across a wide range of application scenarios
and algorithms. Our method dubbed Replay across Experiments (RaE) performs as
well as or better than alternative approaches with fewer hyperparameter
choices to be made. Furthermore, for domains that are normally hard to solve
even with state-of-the-art algorithms, we find that it can be advantageous to
perform multiple training iterations giving each iteration access to all data
from prior training runs, effectively realising a minimalist perspective to
lifelong learning. We hope that RaE’s simplicity will enable straightforward
integration into existing infrastructure and help improve the efficiency of RL
workflows.

The main contributions of our work are as follows:

  * •

Introduce and empirically validate our simple strategy to solve challenging
tasks. Demonstrate state-of-the art performance on a number of domains
including when reusing data from the publicly available offline RL Unplugged
benchmark Gulcehre_2020.

  * •

Demonstrate that our approach works across multiple algorithms including DMPO,
D4PG, CRR and SAC-X.

  * •

Compare RaE to common baselines and recent state-of-the-art methods. Highlight
the combination of factors that leads to the effectiveness of our approach.

  * •

Provide additional ablations to highlight the robustness of RaE to quantity
and quality of data collected and hyper-parameters.

##  2 Method

###  2.1 Background

We consider the reinforcement learning (RL) problem in which an agent observes
the environment, takes an action, the environment changes its state and the
agent receives a reward in response. The environment is modeled as a Markov
Decision Process (MDP) consisting of the state space $S$, the action space
$A$, the transition probability $p(s_{t+1}|s_{t},a_{t})$ and reward
$r(s_{t},a_{t})$ when taking action $a_{t}$ in state $s_{t}$. The agent’s
behavior is specified using a deep neural network policy
$\pi(a_{t}|s_{t};\theta)$ with parameters $\theta$; which we will denote as
$\pi(a_{t}|s_{t})$ for brevity.

We optimize the agent to maximize the sum of discounted future rewards, as
denoted by:

$J(\pi)=\mathbb{E}_{\rho_{0}\left(s_{0}\right),p(s_{t+1}|s_{t},a_{t}),\pi(a_{t}|s_{t})}\Big{[}\sum_{t=0}^{\infty}\gamma^{t}r_{t}\Big{]},$
(1)

where $\gamma\in[0,1]$ is the discount factor,
$r_{t}=r\left(s_{t},a_{t}\right)$ is the reward and
$\rho_{0}\left(s_{0}\right)$ is the initial state distribution.

Given a policy $\pi$ , the state-action value function, or critic,
$Q(s_{t},a_{t})$ is defined as the expected discounted return when taking an
action $a_{t}$ in state $s_{t}$ and then following the policy.

$\displaystyle
Q(s_{t},a_{t})=r(s_{t},a_{t})+\gamma\mathbb{E}_{p(s_{t+1}|s_{t},a_{t}),\pi(a|s_{t})}[Q(s_{t+1},a)].$
(2)

Modern off-policy algorithms haarnoja2019soft, barth-maron2018distributional,
abdolmaleki2018maximum usually operate through simultaneous or alternating
optimization of policy and state-action value function. In this context,
_experience replay_ Lin92 is a fundamental mechanism to decouple data
collection from policy and value-function optimization. Data is collected by
one or multiple policies (which are commonly obtained over the course of a
single experiment) and is stored in a replay buffer from which it can be
retrieved to compute updates to the policy or value function. This approach
has multiple benefits including data efficiency, reduced variance of the
updates and smoothing the learning process mnih2015human.

The success of off-policy algorithms and the need for data-efficient and safe
learning schemes has led to growing interest in approaches that allow to reuse
data across experiments. One end of a spectrum aims to learn policies entirely
offline from a pre-generated, fixed corpus of training data siegel2020keep,
wang2020critic, kumar2020conservative, singh2020parrot, ajay2021opal. This
usually requires specialist algorithmic modifications to avoid instabilities
when no online data is available. It is often desirable to combine prior data
with new data gathered during a learning experiment. In the simplest case a
policy (and value function) pretrained offline can be then finetuned online,
on the same or a related task. While various offline-RL algorithms have been
optimised for this setting, simpler off-policy approaches can work well with
sufficiently diverse data yarats2022don.

The second perspective to re-using data is the straightforward reloading into
the replay buffer during online learning. This approach has been successful in
bootstrapping learning on tasks with expert demonstrations
vecerik2017leveraging, nair2018overcoming, Walke2022DontSF or to transfer to
new tasks by incorporating data from scripted controllers singh2020cog,
Smith2023LearningAA,jeong2020learning. Finetuning offline learned agents and
mixing data into a replay is further combined in the AWAC algorithm
nair2020accelerating. It operates in two phases: in the first phase learning
is performed entirely offline by pre-loading the replay buffer with offline
data. Then, after a specified number of training steps, online data is mixed
into the replay and learning proceeds off-policy. AWAC proposes a specific
algorithm for both phases that is similar to the formulation of the offline-RL
algorithm CRR wang2020critic.

While these methods have shown promising gains in data efficiency and
performance, they each come with specific assumptions that increase
complexity, implementation effort, and restrict their application. These
include the requirement for multiple training stages singh2020cog,
Smith2023LearningAA, the introduction of additional training losses
vecerik2017leveraging, nair2018overcoming,song2022hybrid or validation with
specific architectures and algorithms nair2020accelerating,
Walke2022DontSF,ball2023efficient which typically require domain-specific
hyperparameter tuning. The core idea that we are exploring in this paper is
that the reuse of prior data can be implemented in the context of most
contemporary off-policy RL workflows in a very simple way that is robust and
leads to excellent results, without many of the algorithmic complexities or
hyper-parameter choices of prior work.

###  2.2 Replay across Experiments

Figure 1: Intuitive schema for the use of regular replay in off-policy RL and
the extended application in RaE. In every experiment, the agent can learn from
a mixture of data from all previous experiments to accelerate learning and
increase asymptotic performance.

The basic insight of this work is that reusing data across multiple
experiments in off-policy learning is a very simple but effective way to
accelerate training and improve final performance. Our key argument is the
extension of this insight across all experimentation during a project. Prior
data can be used to bootstrap new training runs, and it can further be
beneficial to train difficult tasks in multiple iterations, bootstrapping
later iterations with the data collected during earlier ones.

This idea is illustrated in Figure 1. Whereas off-policy RL is generally
focused on reusing data during a single experiment, we consider sequences of
experiments where data from earlier training runs is made available during
later ones111We store and reuse all training data across experiments, not just
trajectories left in the final replay.. At the beginning of each training run,
policy and value-function are re-initialized in line with stand-alone
experiments. The only required algorithmic change is the availability of a
second replay mechanism that allows replaying prior and online data with a
particular fixed ratio throughout the course of training (we use a naive 50/50
mix of offline and online data for our main results, without optimizing this
ratio).

This simple approach performs very well without introducing additional
algorithmic complexities or the need for hyper-parameter tuning. By learning
in the mixed online setting, the agent controls the data distribution and in
line with recent results in offline RL yarats2022don,Lambert2022TheCO, we show
that given the right data distribution, existing off-policy algorithms are
sufficient for effectively using this offline data. This workflow does not
require any changes with respect to the RL agent itself and is generally
agnostic to agent and architecture changes across experiments. In order to
emphasize this aspect, for the main results presented in Section 3.3, we
integrate RaE into existing algorithmic workflows across all of the domains
considered. Consequently, we evaluate RaE across a set of state-of-the-art
agents including DMPO abdolmaleki2018maximum, SAC-Q riedmiller2018learning,
CRR wang2020critic and D4PG barth-maron2018distributional across a number of
complex domains.

##  3 Experiments

In this section, we evaluate the performance of RaE across feature-based and
vision-based, simulated robot locomotion and manipulation settings and
standardised RL control benchmarks.

###  3.1 Domains

We consider the following set of challenging domains (visualised in Figure 2):

Figure 2: Domains: (from left to right) Locomotion, Manipulation stacking and
the RL Unplugged domains - Humanoid run, Manipulator insert peg, Manipulator
insert ball.

#### Locomotion Soccer

For locomotion, we use a simulated robot soccer task Haarnoja2023LearningAS
where a simulated OP3 humanoid robot robotisOP3 is rewarded for scoring goals
(along with additional shaping rewards) against an opponent. We consider two
versions of this task: using proprioception and task-specific features
(Locomotion Soccer State) and using proprioception with egocentric visual
features (Locomotion Soccer Vision). This second variant is particularly
challenging since the agent must learn to score from a sparse reward signal
and visual features that make the environment partially observable. Further
information on this domain can be found in Appendix A.1. We use MPO
abdolmaleki2018maximum with a distributional critic
bellemare2017distributional, barth-maron2018distributional as underlying off-
policy RL algorithm. For our experiments we gather training data of 4e5 and
2e5 episodes 222Throughout the text we use the scientific notation shorthand
where 4e5 refers to $4*10^{5}$. for state and vision respectively.

#### Manipulation RGB Stacking

For manipulation, we consider the task of stacking parameterized color-coded
objects from [lee2021beyond]. This task builds on visual inputs and only
sparse rewards. We use the multi-task SAC-Q off-policy RL algorithm
riedmiller2018learning in this domain. Further information can be found in
Appendix A.2. The offline dataset here consists of 15e4 episodes of training
data from a prior experiment with the same algorithm.

#### RL Unplugged

Finally, we evaluate with the offline RL benchmark and dataset RL Unplugged
Gulcehre_2020, which includes offline data on various simulated control
domains. Within these, we focus on the three most challenging Control Suite
domains tassa2020dmcontrol: humanoid run, manipulator insert peg and
manipulator insert ball. The benchmark dataset contains 3000 episodes for
humanoid run and 1500 episodes each for the manipulator domains. We use RaE
with the offline RL algorithm CRR wang2020critic, which to our knowledge has
state-of-the-art performance for pure offline learning on this benchmark.

The offline data in this setting is notably different from the other domains.
In order to collect the data, the authors follow a sub-sampling criterion (see
[Gulcehre_2020] for details) to select a subset of episodes which are then
randomly stored as single-step trajectories. This is in contrast to our simple
protocol of collecting and reusing all data across training with a single
algorithm. We include this domain to illustrate the robustness and flexibility
of RaE as well as to simplify future extensions and comparisons. More details
on all domains are presented in Appendix A.

###  3.2 Baselines

We compare RaE against a number of strong baselines to illustrate its
effectiveness.

#### Fine-tuning

We first train a policy and state-action value function offline using Critic
Regularized Regression (CRR) wang2020critic to retain the flexibility of
architecture and related changes across experiments. We evaluate a pure
behavior cloning agent (BC) and two variants of CRR, CRR binary and CRR exp
(see Appendix B). Subsequently, we choose the best performing seed from across
these runs and fine-tune the agent (both policy and value function) online.

#### AWAC

AWAC nair2020accelerating starts by learning entirely offline for a fixed
number of pre-training steps before switching to online iteration. The main
difference to finetuning applies to the online phase, where it uses a
offline/online shared replay buffer with a specific algorithmic formulation.
The authors of this work prescribe 25e3 pretraining steps offline before
incorporating online data. For our sweep we also considered 5e4 steps and 1e5
pre-training steps. We also sweep over different values for the Lagrange
multiplier $\lambda$ (0.3 and 1) as proposed by the original work.

#### Random Weight Resetting

Reloading data for an experiment restart implicitly involves resetting network
weights. [Nikishin2022ThePB] observed that network resets on their own can be
beneficial for learning. 333When using RaE, the data distribution throughout
learning uses a fixed ratio between offline and online data, in addition to
implicitly resetting network weights. To tease apart the effect of weight
resets from the benefit of data reloading, we consider a baseline where all
weights (policy, critic and optimizer) are reset every $K$ policy updates;
where $K$ is set to the number of updates used to train the policy that
generated the data. We sweep over reset frequencies of $K$, $K/10$ and
$K/100$444Since the data collection methodology for the RL Unplugged domains
is different and the number of updates unknown, we instead chose to sweep over
values of 1e4, 1e5 and 1e6 updates for these domains. All experiments are run
till convergence or at least $2*K$ updates.

For each method, we report results of the best performing hyperparameter
variant averaged across 5 seeds. Importantly, note that RaE did not require
any tuning and we report results using the same hyperparameter set across all
of the main results (50% offline data). While minor performance improvements
with different parameters for RaE can be observed in extreme settings (see
Table 1), we find RaE has very low sensitivity to hyperparameter choices
across a range of diverse domains.

(a) L: Soccer State

(b) L: Soccer Vision

(c) M: Place

(d) M: Stack Leave

Figure 3: Comparison of RaE against baselines on the Locomotion Soccer and
Manipulation domains (denoted by L: and M: respectively). Y-axis shows
accumulated reward (undiscounted episode return) for each method; dotted blue
line indicates offline learning performance with CRR. RaE consistently
performs at-par or better than the baselines with a notable improvement on
tasks involving vision.

###  3.3 Main Results

Figure 3 visualises our results on the Locomotion Soccer and Manipulation
domains. It compares asymptotic performance achieved by RaE and other
baselines after convergence as a bar plot with the dark solid line showing a
95% confidence interval around the mean averaged across 5 seeds after
smoothing over 1000 episodes. As the figure shows, RaE consistently achieves
the highest asymptotic performance across these tasks with a particularly
notable improvement on the challenging vision-based ‘Locomotion Soccer Vision’
domain. In the Manipulation settings, both finetuning and RaE perform
similarly, although finetuning requires choosing the right algorithm and
hyperparameter for offline learning.

Figure 4 compares RaE with all baselines that can use the RL Unplugged offline
dataset. The figure shows the accumulated reward over the total number of
additional online data consumed for each method. The performance of the best
pure offline variant (CRR) is shown as a dotted blue line. We observe that RaE
performs at-par or better than other comparable methods with the biggest
difference being in the manipulator tasks. Methods that use offline learning
(CRR and finetuning) work well on the densely rewarded Humanoid run domain but
peform poorly in the sparsely rewarded manipulator domains whereas RaE works
well across all settings.

(a) Humanoid Run

(b) Manipulator: Insert peg

(c) Manipulator: Insert ball

Figure 4: Performance of various mechanisms to reuse offline data using the RL
Unplugged dataset. Accumulated rewards (undiscounted episode return) on the
Y-axis are plotted against the total steps. Each curve plots the mean
performance across 5 seeds with standard deviation shown as shaded regions.
RaE consistently performs the best across all tasks.

###  3.4 Ablations

In this section, we present a set of experiments designed to answer the
following questions:

  * •

How much data is required for performance improvements?

  * •

What kind of data is best for mixing: expert data, early training data or a
mix?

  * •

How sensitive is RaE to the ratio of online to offline data in different data
regimes?

  * •

Is there an advantage of applying RaE repeatedly across experiment iterations?

  * •

Is RaE agnostic to the underlying choice of algorithm?

Unless otherwise specified, we present results on the ‘Locomotion Soccer
State‘ task with the DMPO algorithm.

#### Analysis with varying data

We answer the first three of these questions together in this section. For
this analysis we use multiple subsets of the data collected for the
‘Locomotion Soccer State’ task, divided into three regimes:

  * •

High Return We consider data generated only from the end of training from a
complete training run. This corresponds to highly rewarding trajectories or
‘expert’ data.

  * •

Mixed Return We consider data sampled uniformly at random throughout learning.
This corresponds to a mixed regime with of high and low return trajectories.

  * •

Low Return Finally, we consider data generated only from the start of
training. This corresponds to early low return data.

High and Low Return data are sampled according to recency while Mixed return
samples data uniformly. For each regime, we consider 2 datasets: one with 1e5
episodes555The original dataset consists of 4e5 episodes. and another with
just 1e4 episodes. The purpose of this analysis is to understand how best to
adapt RaE when in a regime with limited data. For each data subset we then
consider data mixing at different ratios of online to offline data: in
addition to the 50% mix that was used in the main results we consider mixes of
70, 80 and 90% online to offline data. For each setting, we consider the
asymptotic performance reached when using RaE on the ‘Locomotion Soccer State‘
task.

Table 1 shows the final asymptotic reward as a percentage of the final reward
achieved when learning online from scratch, which corresponds to 4e5 episodes.
To improve the ease of interpretation the cells are colored based on the
reward achieved: yellow for below 90% performance, orange for between 90-100%
and blue for greater than 100%.

A few interesting patterns emerge which we summarize below:

  * •

In the lower data regime (1e4 episodes) a mixture with more online data is
beneficial. However, as more data becomes available, a lower ratio works
better (of 70-80% at 1e5 episodes). We hypothesize that using more online data
for learning prevents over-fitting to a small set of offline trajectories.

  * •

In the lower data regime, low return data tends to be the most beneficial. As
the dataset size increases though, a mix of high and low return trajectories
provide a greater benefit. Surprisingly, expert data is the least beneficial
in both regimes. This indicates that the advantage of mixing data may stem
from the benefit of having a larger state distribution with mixed rewards.

  * •

Gains in performance can be achieved with as little as 1e4 episodes. This is
particularly promising since even a small amount of prior data can
considerably improve results.

| High return | Mixed return | Low return
---|---|---|---
Data mixEpisodes |  10,000 |  100,000 |  10,000 |  100,000 |  10,000 |  100,000
50% Online |  51% |  90% |  80% |  119% |  97% |  112%
70% Online |  80% |  110% |  101% |  124% |  108% |  121%
80% Online |  98% |  118% |  106% |  126% |  110% |  123%
90% Online |  108% |  120% |  113% |  120% |  108% |  121%
Table 1: Performance comparison when using RaE in different data regimes with
various mixes of online and offline data. Each cell represents asymptotic
performance improvement as a percentage of the performance reached when
learning from scratch to convergence(100% online). Colors are used to indicate
the trend: red for below 90% performance, orange between 90 to 100% and blue
when greater than 100%

#### Iterative improvement

Figure 5(a) shows the improvement in performance when iteratively applying RaE
on the ‘Locomotion Soccer State’ domain. For this experiment, we begin by
generating a smaller dataset of 1e4 episodes and then apply RaE to that. We
collect the data from this already improved run (RaE iteration 1) and apply
RaE again. We observe that there is a small gain in asymptotic performance and
speed of learning even on the second iteration although a performance plateau
is reached on a third iteration. This result suggests that in some settings it
may be preferable to break a single training run into smaller runs for
iterative performance improvements.

(a) Multiple iterations of RaE

(b) RaE with D4PG

(c) Reloading across seeds

Figure 5: a) RaE can be applied iteratively on the ‘Locomotion Soccer State’
domain. In some cases it may be preferable to have multiple smaller iterations
although a performance plateau may be reached eventually. b) RaE on the
‘Locomotion Soccer State’ domain continues to improve performance with the
D4PG algorithm. c) Combining data from seeds of a high variance experiment.
RaE performs well even if just a single original seed achieves high reward.
Performance of the original data generating experiments plotted as dotted
lines. All figures plot mean accumulated reward against total online steps
with standard deviation shown as a shaded region.

#### Indifference to algorithm

For the main analysis of Section 3.3, we show that RaE can improve performance
across a range of domains with different underlying algorithms. Figure 5(b)
reinforces this point on the ‘Locomotion Soccer State’ domain where we show a
considerable gain in performance when applying RaE using the D4PG barth-
maron2018distributional algorithm for both data collection and mixing.

#### Increased robustness

By applying RaE to combine data across seeds and hyperparameters from previous
experiments we can add crucial robustness to the underlying agent’s variance
in performance. Figure 5(c) demonstrates robust performance when reloading
data across random seeds in comparison to purely high or low performing data
sources.

##  4 Related Work

The most common paradigms to collect and reuse data in RL involve either
offline agent datasets (e.g. RL Unplugged Gulcehre_2020 and D4RL Fu_2020) or
the use of human or expert demonstration trajectories vecerik2017leveraging,
Bohez2022ImitateAR. While some work has focused on solving benchmark
challenges wang2020critic, kumar2020conservative others have focused on
improving online performance by integrating offline data. For example,
[singh2020parrot] and [ajay2021opal] learn hierarchical architectures offline
and reuse them to solve more challenging tasks online. Related to our
approach, [Smith2023LearningAA] mix experience using controllers designed for
a different setting with online data to improve learning efficiency for
quadruped robots. Our work instead highlights the merits of directly
transferring data to improve asymptotic performance on a single domain.

#### Learning from Data Combinations

The idea of mixing expert data sources to bootstrap learning on the same
domain has also been studied in a number of related works
vecerik2017leveraging, nair2018overcoming, singh2020cog, Walke2022DontSF,
Davchev2021WishYW, lee2022offline, ball2023efficient, nair2020accelerating.
While these methods have shown impressive results on a range of domains, the
complexity and specificity of various algorithmic assumptions limit their
generality. While [vecerik2017leveraging] also mix expert demonstration data
into the replay, they include a prioritized replay and a mix of 1-step and
N-step returns with L2 regularization in their setup. In a similar vein,
[nair2018overcoming] introduce a BC-loss, Q-filter and state-reset mechanism
as part of their data mixing approach. Other work relies on multi-stage
procedures when mixing data: [singh2020cog] distill prior data from skill
experts to be later mixed with data from a scripted controller followed by the
application of CQL; [Walke2022DontSF] train a multi-task policy with separate
forward and backward policies that are optimized separately. Even work that
uses existing offline datasets as opposed to expert demonstrations inevitably
include other components that increase their complexity. For instance,
[lee2022offline] finetune an ensemble of policies trained offline using a
prioritized replay and a density ratio to choose the data mixture to improve
performance with the D4RL dataset. On the same domain, [ball2023efficient]
begin with a similar formulation to ours but then argue for random ensemble
distillation and per-environment design choices and LayerNorm when training
the Q-function with which they demonstrate improvements using a single off-
policy algorithm. Finally, the AWAC algorithm nair2020accelerating that we
compare against sits in between finetuning and data mixing but uses a specific
algorithmic formulation which introduces a number of domain-dependent
hyperparameters.

In contrast, our focus is on the simplicity of the method: we empirically
demonstrate the effectiveness and versatility of mixing previous data with a
fixed ratio using many off-policy RL algorithms.

#### Resetting Network Parameters in Reinforcement Learning

A consequence of restarting experiments with mixed data is the resetting of
neural network parameters. Resetting weights can change the learning dynamics
and prevent overfitting which has been shown to be beneficial in both
supervised learning ash2020warm and RL Nikishin2022ThePB, schwarzer23abbf.
Under certain settings where parts of a network are reset at specific
intervals, these methods have been shown to greatly improve learning
efficiency on challenging domains like Atari. In this work, we demonstrate
that reusing previous experience can improve asymptotic performance on a range
of tasks. Combined with its simplicity, we hope our approach can be included
as part of the natural iteration cycle in RL and robotics in particular.

##  5 Discussion

As RL continues to move from the object of study to a practical engineering
tool for control degrave2022magnetic, Bellemare2020AutonomousNO, 47493,
9811912, kaufmann2023champion, osinski2020simulation, simple and practical
methods that can be generally applied will become increasingly important.

We can apply RaE throughout the lifetime of a project, across multiple
experiments, algorithms, and hyperparameter settings. In the following
paragraphs, we provide some intuition on some practical use-cases for RaE.

#### Lifelong / Project-long learning

Research into novel domains nominally involves many iterations of trial-and-
error to achieve state-of-the-art performance. While learnings from early
iterations of experimentation inform algorithmic choices, the data collected
in these trials is commonly discarded and never reused. Our initial analysis
in Section 3.4 indicates that even low-return data can be useful to boost
performance. With costs of data storage typically being far lower than
compute, a different workflow where all experimental data (particularly in
domains like robotics) is stored and reused to bootstrap learning could
improve efficiency across project lifetimes.

#### Multiple Source Experiments

RaE can also be applied in domains with many related source experiments. For
example, consider many tasks defined via different reward functions but with
the same underlying dynamics (e.g. the family of all manipulation tasks with a
specific morphology). High-return data on some tasks may result in lower
returns in another. However this data is still informative and may be useful
in improving exploration when transferred.

#### Multiple Hyperparameters and Seeds

Combining data from different variants of the same experiment (such as random
seeds or hyperparameters) can enable better use of large, expensive sweeps.
When reloading data with high performance variance over these parameters, the
new experiment can benefit from the best option as initially discussed in
Section 3.4.

#### Potential Limitations and Strategies for Mitigation

Across all experiments, RaE demonstrates better or similar performance to
state-of-the-art baselines for efficient off-policy RL without requiring per
task hyperparameter tuning. However the applicability of the method is limited
by the ability to reuse data. For example, changes in dynamics or experimental
settings might invalidate previously collected data 666However in practice,
this may not be much of an issue: see Appendix C.4. In such cases, an
intermediate step to collect transitional data between old and new settings
might be useful. Another challenge that may arise with widespread use of RaE
is in the comparison of new algorithms. RL already suffers from issues of
reproducibility henderson2018deep; if algorithms incorporate RaE, subtle
changes in the data distribution when learning may create large differences in
performance. One solution to this may be to specify deterministic orderings
for benchmark datasets using fixed random seeds.

##  6 Conclusions

With this work, we introduce an effective and, importantly, simple workflow
change for off-policy RL experimentation. Reloading data requires minimal
infrastructure and can greatly improve performance as shown in Section 3. This
makes it particularly useful from a project-long (or life-long) learning
perspective in RL experimentation. We believe that as our understanding of RL
improves and its use as an engineering and control tool becomes more
commonplace, simplicity is key to effective integration.

##  Appendix A Environment Details

###  A.1 Locomotion Soccer

The ‘Locomotion Soccer’ environment introduced by [Haarnoja2023LearningAS]
consists of a Robotis OP3 robotisOP3 robot which is placed in a $4m\times 4m$
walled arena with a soccer ball. The robot must remain in its own half and
scores goals when the ball enters a goal area $0.5mtimes1m$ positioned across
the center of the back wall in the other half. The robot must also contend
with a random agent in the other half. The reward for this task is given by:

$R_{goalscoring}=R_{score}+R_{upright}+R_{maxvelocity}$ (3)

where: $R_{score}$ is 1000 on the single timestep where the ball enters the
goal region (and then becomes unavailable until the ball has bounced back to
the robot’s own half); $R_{maxvelocity}$ is the norm of the planar velocity of
the robot’s feet in the robot’s forward direction; $R_{upright}$ is 1.0 when
the robot’s orientation is close to vertical, decaying to zero outside a
margin of $12.5^{\circ}$. Additionally, episodes are terminated with zero
reward if the robot leaves its own half, or body parts other than the feet
come within 4cm of the ground. The agent chooses a desired joint angle every
25 milliseconds (40Hz) based on input observations that include the joint
angles, angular velocity and gravity direction of the torso. For ‘Locomotion
Soccer State’ the agent also receives the coordinates of the ball, goal and
opponent. For the ‘Locomotion Soccer Vision‘ domain the robot instead receives
a $40\times 30$ render of the egocentric camera.

We use the experimental settings and networks described in
[Haarnoja2023LearningAS] for this setting. We use a batch size of 128 with 5
seeds for the main results in Section 3 and a batch size of 256 with 2 seeds
for the experiments in Section 3.4.

###  A.2 Robot Manipulation

The RGB Stacking benchmark introduced by [lee2021beyond] consists of
parametric extruded shapes, which need to be stacked on top of each other by a
fixed robotic manipulator, with the role of objects in the configuration being
denoted via color coding. The benchmark uses a Rethink Sawyer robot arm,
controlled in Cartesian velocity space on which a Robotiq 2F-85 parallel
gripper is mounted.

We focus on the ”skill mastery” challenge of the benchmark as described in
[lee2021beyond], where five fixed triplets of objects are used both for
training and testing, with the goal being to place the red object on the blue
one, while ignoring the green one. Each object triplet highlights a different
manipulation aspect, such as balancing or reorientation.

For the SAC-X setup, we use a curriculum of sub-tasks that can be chosen by
the scheduler. These follow the reward terms of the staged reward terms used
in [lee2021beyond], and intuitively contain sub-tasks for reaching, lifting,
placing and stacking the red object. For evaluation, we report the final
”stack-leave” reward term, which is a sparse reward given when the red object
is placed precisely on the blue one, and the arm has been moved away by a
predefined distance.

The observation provided to the agent consists of the images of three static
cameras, as well as proprioception data from the robot and gripper, including
joint angles, velocities and torques, as well as a simulated force-torque
sensor attached to the wrist. It notably does not contain the tracked position
of the objects, so the agent must learn to achieve the goal primarily from
vision inputs.

In order to solve this task from visual inputs we use a network architecture
where both policy and value function use a convolutional neural network with
residual blocks (as used in espeholt2018impala) that is then passed through an
MLP for multi-task output for each of the subtasks. Both networks use (2, 2,
2) residual blocks with convolutional layers with 16, 32 and 32 channels
respectively. The output of the policy network is used to condition a multi-
headed Gaussian where each Gaussian output represents the policy output for a
sub-task. Similarly the value function outputs a multi-headed categorical for
each task . Appendix B describes how this can be used by the SAC-Q algorithm.
All experiments are run with 5 seeds for the main results of Section 3.3 and 2
seeds for the ablations in Section 3.4.

###  A.3 RL Unplugged

We consider 3 datasets from the RL Unplugged benchmark Gulcehre_2020: Humanoid
run, Manipulator insert peg and Manipulator insert ball; which are categorized
as ‘hard’ domains by [wang2020critic]. The RL Unplugged datasets were
collected on the DeepMind Control Suite tassa2018control implemented in the
MuJoCo todorov2012mujoco simulation framework.

Humanoid run consists of 3000 episodes of a 21 dimensional simulated humanoid
walker that is rewarded for running forwards while staying upright. The data
for this domain is generated using D4PG barth-maron2018distributional. The
Manipulator insert ball and Manipulator insert peg domains consist of 1500
episodes each where a 5-dimensional manipulator is sparsely rewarded for
inserting a ball and peg respectively into a specified hole. Data for this
domain was generated using V-MPO song2020vmpo since D4PG was unable to solve
the tasks. The data generated on these tasks is then reduced in size via sub-
sampling and the number of successful episodes in each dataset is reduced by
$2/3$ to ensure the data does not contain too many successful trajectories.
For all domains each episode consists of 1000 timesteps.

We use experimental settings as described in [wang2020critic] for this domain
and use their networks for the offline and finetuning experiments. For the
other baselines and RaE we use a an MLP architecture with sizes (256, 256,
128) for the policy and (512, 512, 256) for the value function. The policy
network output is used to parameterize a Mixture of Gaussian (MoG) with 5
components as in [wang2020critic]. Since the dataset only involves single-step
transitions we use 1-step return for all baselines except the Random resets
where we found 5-step returns performs significantly better.

##  Appendix B Algorithm Details

In this section we describe the algorithms used in the main text in more
detail.

#### Policy Evaluation

We use the critic update described in [barth-maron2018distributional] for
$N-$step returns:

$(\mathcal{T}_{\pi}^{N}Q)(s_{0},a_{0})=r(s_{0},a_{0})+\mathbb{E}\left[\sum_{n=1}^{N-1}\gamma^{n}r(s_{n},a_{n})+\gamma^{N}Q(s_{n},\pi(a_{n}|s_{N}))|s_{0},a_{0}\right]$

where the expectation is with respect to $N-$step transition dynamics and the
distribution $Q$ and $\mathcal{T}$ represents the distributional Bellman
operator. For our experiments we parameterize the critic as a categorical
distribution with the number of bins set based on the domain. We set $N=5$ for
our experiments except in the RL unplugged domains where only single-step
trajectories exist in the dataset. However, we use $N=5$ for the random resets
baseline in this domain since it can be run online and is not constrained by
the limitation in the data.

#### DMPO

For our online experiments in ‘Locomotion Soccer’ and ‘Manipulation Stacking‘,
we use the Maximum a-posteriori Policy Optimisation algorithm (MPO)
abdolmaleki2018maximum to adhere to [Haarnoja2023LearningAS]. MPO optimizes
the RL objective in an E and M-step. The E-step update optimizes:

$\displaystyle\max_{q}\int_{s}\mu(s)\int_{a}q(a|s)Q(s,a)\,da\,ds$
$\displaystyle\,s.\,t.\int_{s}\mu(s)D_{\mathrm{KL}}(q(a|s)||\pi(a|s))\,ds<\epsilon$

where $q(a|s)$ an improved non-parametric policy that is optimized for states
$\mu(s)$ drawn from the replay buffer. The solution for $q$ is shown to be:

$\displaystyle q(a|s)$
$\displaystyle\propto\pi(a|s,\theta)\exp(\frac{Q_{\theta}(s,a)}{\eta^{*}}).$

where $Q$ is computed as an expectation over the categorical distribution. In
the M-step, an improved policy is the obtained via supervised learning:

$\displaystyle\pi^{n+1}$
$\displaystyle=\arg\max_{\pi_{\theta}}\sum_{j}^{M}\sum_{i}^{N}q_{ij}\log\pi_{\theta}(a_{i}|s_{j}).$
$\displaystyle\,s.\,t.D_{\mathrm{KL}}(\pi^{n}(a|s_{j})||\pi_{\theta}(a|s_{j}))$

#### SAC-Q

For the online experiments in ’Manipulation Stacking’, we use the Scheduled
Auxiliary Control (SAC-X) algorithm for multi-task exploration
riedmiller2018learning. SAC-X builds on a multi-headed network architecture to
represent both critic and policy, with a single ”torso” per network shared
across multiple tasks, and a separate output ”head” for each task. At any
given time, the active head to use is decided by a scheduler process. Here,
the scheduler itself is also being learned, using the SAC-Q variant of the
algorithm. At fixed intervals during each episode, a new sub-task may be
chosen, and the scheduler trains a Q-function that optimizes the sequence of
tasks executed such that it maximizes the total return of one task designated
as the main goal.

To update the policy and Q-function, SAC-Q may use any update rule. Here, we
use the MPO algorithm with a distributional critic, as described above.

#### CRR

For the offline experiments in Section 3.3 we use the Critic Regularized
Regression (CRR) algorithm wang2020critic which achieves state-of-the-art
performance on the RL Unplugged benchmark. CRR uses the same policy evaluation
step described above to learn a distributional critic. However the policy in
CRR is trained by filtering data via the Q function by optimizing:

$\displaystyle\operatorname*{arg\,max}_{\pi}\mathbb{E}_{(s,a)\sim
D}\left[f(Q,\pi,s,a)\log\pi(a|s)\right],$

where the states and actions are drawn from the data source $D$ and $f$ is a
non-negative, scalar function whose value increases monotonically with $Q$. We
consider both variants of CRR introduced in the paper defined by choices of
$f$:

$\displaystyle f:=\mathbbm{1}\left[A(s,a)>0\right],$ (4) $\displaystyle
f:=\exp\left(A(s,a)/\beta\right),$ (5)

where $A(s,a)$ is the advantage function which can be computed using:

$\displaystyle
A(s,a)=Q(s,a)-\frac{1}{m}\sum_{j=1}^{m}Q(s,a^{j});a^{j}\sim\pi(.|s).$

Following the authors of CRR, we refer to CRR with $f$ from Equation 4 as CRR
binary and Equation 5 as CRR exp.

#### AWAC

For the AWAC baseline in the main text we follow the description from
[nair2020accelerating] where learning proceeds in two stages: entirely offline
initially and then with online data added to the same replay buffer. The
transition between these two stages is defined by a hyper parameter which is
set to 25,000 steps in the original work. For our experiments we sweep over an
additional value of 50,000 steps and 100,000 steps for the RL Unplugged
domain.

AWAC also introduces an algorithm which is similar to the CRR exp formulation
described above with a slightly different notation that introduces a
temperature $\lambda=\frac{1}{\beta}$ above. As per the original work we test
the method with $\lambda$ set to 0.3 and 1.

##  Appendix C Additional Results

###  C.1 Manipulation results on remaining tasks

(a) Open

(b) Reach Grasp

(c) Lift

(d) Stack

Figure 6: Comparison of RaE against baselines on Open, Reach Grasp, Lift and
Stack sub tasks in the Manipulation RGB stacking setting. Average performance
across 5 seeds is plotted on the Y-axis with standard deviation as as dark
lines.

Figure 6 compares RaE against the other baselines from Section 3.3 on the
remaining sub-tasks in the ‘Manipulation RGB Stacking’ setting. While all
methods fare well on the easier tasks like ‘Open’ and ‘Reach Grasp’, RaE
performs well on the harder ‘Lift’ and ‘Stack’ tasks similar to the trend on
‘Place’ and ‘Stack Leave’ that were presented in the main text.

###  C.2 Finetuning with RaE

The ‘finetuning’ baseline considered in the main experiments of Section 3.3 is
orthogonal to the core idea of RaE and as such, we can apply RaE when
finetuning. Figure 7 compares RaE , finetuning and finetuning with RaE on the
‘Locomotion Soccer‘ tasks from state and vision. We observe that finetuning
with RaE performs well across both domains matching the high asymptotic
performance of RaE and learning faster by taking advantage of the network
weights pre-trained offline.

(a) Locomotion Soccer State

(b) Locomotion Soccer Vision

Figure 7: Comparison of finetuning, RaE and finetuning with RaE on the
‘Locomotion Soccer‘ state and vision tasks (left and right respectively).
Finetuning with RaE combines the best of both worlds: improved learning speed
with high asymptotic performance. Accumulated reward on the Y-axis is plotted
against total online steps on the X. Results are averaged across 5 seeds with
the shaded region representing the standard deviation.

###  C.3 Effect of mixing ratio on full dataset

The results presented in Table 1 show the effect of mixing different ratios of
data when using RaE with smaller subsets of data (of 10,000 and 100,000)
episodes. Figure 8(a) instead presents the same analysis when using the full
dataset (of 4e5 episodes) on the ‘Locomotion Soccer State’ task. In this
setting we see a clearer advantage of using offline data where a mixture of 50
to 70 percent of offline data works better than using more online data (90 %).
In fact, as the figure shows, a mixing ratio of 90 % may even slightly degrade
performance when compared to learning purely online.

###  C.4 Effect of changing dynamics

In this section we analyze the effect of reusing RaE in a regime where the
underlying dynamics change. We reuse the data collected in the ‘Locomotion
Soccer (State)’ task but transfer to an environment where a mass is randomly
attached to the left and right legs of the walker to perturb it’s motion.
Figure 8(b) shows that RaE continues to show an advantage as compared to
learning from scratch even when the underlying dynamics of the task have
changed. This somewhat surprising finding may be explained by modeling the
environment as a partially observed MDP where under some conditions unknown to
the agent, the dynamics of walking alter. Reusing data can still guide
learning in such a setting showing the robustness of our approach.

(a) Mixing ratio with full dataset

(b) RaE with changing dynamics

Figure 8: (Left) Comparison with different mixtures of offline and online data
when using RaE with 4e5 episodes of data on the ‘Locomotion (State)’ task.
Accumulated reward (total undiscounted return) for each ratio of data is
plotted. A mixture of 50 to 70 percent of online data works best. (Right)
Comparison when using RaE to reload data to a new task with different
dynamics. RaE is suprisingly robust and learns to perform better than learning
from scratch indicating it’s robustness to changes in dynamics.


Original Article Xiaofei Chen, PhD
Department of Statistical Science, Southern Methodist University, Dallas, TX,
75275, USA<EMAIL_ADDRESS>
# A Vector Autoregression Prediction Model for COVID-19 Outbreak

Qinan Wang  Lyle School of Engineering, Southern Methodist University, Dallas,
TX, 75275, USA  Yaomu Zhou  Lyle School of Engineering, Southern Methodist
University, Dallas, TX, 75275, USA  Xiaofei Chen  Department of Statistical
Science, Southern Methodist University, Dallas, TX, 75275, USA  Department of
Population & Data Sciences, UT Southwestern Medical Center, Dallas, TX, 75390,
USA

###### Abstract

Since two people came down a county of north Seattle with positive COVID-19
(coronavirus-19) in 2019, the current total cases in the United States (U.S.)
are over 12 million. Predicting the pandemic trend under effective variables
is crucial to help find a way to control the epidemic. Based on available
literature, we propose a validated Vector Autoregression (VAR) time series
model to predict the positive COVID-19 cases. A real data prediction for U.S.
is provided based on the U.S. coronavirus data. The key message from our study
is that the situation of the pandemic will getting worse if there is no
effective control.

###### keywords:

COVID-19, Prediction, Time series data, Vector autoregression, Internal
validation

##  1 Introduction

COVID-19 (coronavirus-19) a new type virus, belonging to the Coronaviridae
family, spreads from Wuhan, China in 2019.[1] The Coronaviridae family
consists of two main subfamilies: Coronavirinae and Torovirinae. These viruses
affect the neurological, gastrointestinal, hepatic, and respiratory systems
and can be grown by humans, livestock, etc.[2, 3, 4]

Since the appearance, COVID-19 has infected over 59 million people worldwide.
[5]The worst situation experienced by the United States (U.S.) followed by the
United Kingdom, Italy, France, and Spain. The U.S. has a cumulative 12 million
positive cases up to now. It found itself grappling with the worst outbreak
after Italy and Spain.[6]The Centers for Disease Control and Prevention (CDC)
has verified evidence that COVID-19 is distributed from human to human, and
has also reported that COVID-19 spreads through touching surfaces, close
contact, air, or objects that contain viral particles. In the incubation
period, it can spread to others. It should be noted that the incubation period
and median age of confirmed cases are 3 days and 47 years respectively. [7, 8]

The economic and social disruption caused by the pandemic is devastating. The
disease prevention and control is eager for a disease prediction guidance.
Efficient models for short-term forecasting has a pivotal role to develop
strategic planning methods in the public health system. Under the guidance of
the prediction model, we know the severity and the trends of epidemic under
different strategies. It can arouse public awareness and help government take
the most benefit measures to avoid deaths and reduce infection, such as
ordered school closure, case-base measures, the banning of public events, the
encouragement of social distancing, and lockdown.

Per literature, a system of differential equations for Susceptible-Infected-
Removed (SIR) sequences is a typical mathematical epidemiological model for
COVID-19 forecasting.[5, 9, 10, 11, 12, 13, 14] Joining SIR models, Khan et
al. proposed the SQUIDER compartmental model to predict the coronavirus 2019
spread [15], and Xu et al. applied the generalized fractional-order SEIR
model.[16] The SIR model has a good fitting for the simulation and data of the
outbreak in the early stage of the disease. However, the obvious limitations
are not limited to that the overall model system has a small external control
power, and the number of patients presents a typical exponential growth, which
is due to the absence of external drugs and preventive measures.

Other works on COVID-19 prediction has been carried out in Deep Learning and
ARIMA (Auto Regressive Integrated Moving Average) univariate time series
model. To assess the dynamics of epidemic diseases, time series analysis tools
and deep learning are also widely used in publications. Zeroual et al., Shahid
et al., and Chimmula et al. performed the Recurrent Neural Network to predict
the spread.[17, 18, 19] With time series tools, Alzahrani et al., Sahai et
al., and Kumar et al. predicted the COVID-19 by ARIMA univariate model.[20,
21, 22] Deep learning requires a high number of training samples. However, the
data we have are still few, so the model generalization is unappealing, namely
overfitting. In the time series field, the ARIMA model is quite simple,
requiring only endogenous variables and no other exogenous variables. A
Stationary is required for the time series, or it is stationary after
differencing. Essentially, it lays a shortfall in explaining the causality
between different variables.

This article aims to build a generalized VAR (Vector Autoregressive) model for
predicting the dynamics of COVID-19 daily cases of the epidemic. VAR is a
comprehensive model integrating the advantages of multiple linear regression
and the advantages of time series model (the influence of lag term can be
analyzed). It applies linear relations to describe a stable system. Under the
stationary condition, we can achieve a consistent estimator with the least-
square estimation. Besides, VAR can describe the dynamic linear correlation
between variables that affect each other, whether used for prediction,
interpretation, or sensitivity analysis are clear. With the selected
correlated variables among undetected infected, detected deaths, detected
recovered, average temperature, precipitation, wind speed, humidity,
population density, social trust and civic engagement, that are commonly cited
in other epidemiology publications, VAR multivariate model can have a better
performance on forecasting and provide an interpretive result. [15, 23, 24,
25, 26, 27, 28]

The correlated variables we choose are useful for analyzing the critical
factors driving epidemics. Not only for COVID-19 but may also this model
enlighten other epidemics prediction. For the result, some publications tend
to be more concerned with the cumulative positive cases, while this article
has a very definite awareness of the daily increase cases. A cumulative
positive cases prediction is less meaningful than a daily cases increase,
since the latter is a better representative signal for epidemic severity. It
is also a critical indicator to access the efficiency of COVID-19 control.

In the next section (Section 2), we describe the data we used in the analyses.
The method section (Section 3) elaborates the proposed VAR model and analysis
plan. The Results section (Section 4) provides the prediction results by VAR
modeling and an internal validation/evaluation of the model. Section 5
discusses the model performance, further improvement, and comparison with
other models.

##  2 The daily reported COVID-19 data

The COVID-19 disease has been reported by CDC (Centers for Disease Control and
Prevention) and published in nation and fifty states in the United States by
the Center for Systems Science and Engineering (CSSE) at Johns Hopkins
University. We obtain the data from https://covidtracking.com/data/download
maintained by “The Atlantic Monthly Group”. The data contain the number of
death confirmed, death increasing, death probable, hospitalized, hospitalized
cumulative, positive confirmed, positive case viral, positive increasing, etc.
The available data is beginning on January 22, 2020 to November 24, 2020
(now).

##  3 Method

###  3.1 A Vector Autoregressive Panel Time Series Model

VAR model was proposed by Christopher Sims in 1980s, using all the current
variables in the model to carry out regression for some lagged variables. It
is an extension of the AR (autoregression) model, which has been widely used
for time series. VAR model takes each endogenous variable as a function of the
lag value of all endogenous variables in the system, thus extending the
univariate autoregressive model to the "vector" autoregressive model composed
of multiple time series variables.

Let $\mathbf{X}_{t}$ be a causal, stationary multivariate process, then the
VAR model can be expressed as:

$\mathbf{X}_{t}=\bm{\alpha}+\bm{\Phi}_{1}\mathbf{X}_{t-1}-\cdots-\bm{\Phi}_{p}\mathbf{X}_{t-p}+\mathbf{a}_{t}$
(1)

where $\mathbf{X}_{t}=\left(X_{t1},\ldots,X_{tm}\right)^{\text{T}}$ is an
$m\times t$ matrix; $\bm{\Phi}_{k}$ is a real-valued $m\times m$ matrix for
each $k=1,\ldots,p$; $\mathbf{a}_{t}$ is multivariate white noise with
covariance matrix
$\mathbb{E}\left[\mathbf{a}_{t}\mathbf{a}_{t}^{\text{T}}\right]=\bm{\Gamma}_{a}$;
4. $\bm{\alpha}=\left(\mathbf{I}-\Phi_{1}-\cdots-\Phi_{p}\right)\bm{\mu},$ and
$\bm{\mu}=\mathbb{E}\left(\mathbf{X}_{t}\right)$;
$\mathbf{I}=\\{1,1,\dots,1\\}$. Now $\mathbf{X}_{t}$ is called a VAR ($p$)
process, that is, a vector AR process of order $p$.

Equation (1) can be expressed in multivariate operator notation way:
$\bm{\Phi}(B)\left(\mathbf{X}_{t}-\bm{\mu}\right)=\mathbf{a}_{t},$ where
$\bm{\Phi}(B)=\mathbf{I}-\bm{\Phi}_{1}B-\cdots-\bm{\Phi}_{p}B^{p}$ and
$B^{k}\mathbf{X}_{t}=\mathbf{X}_{t-k}$.

A multivariate process $\mathbf{X}_{t}$ satisfying the difference equation in
Equation (1) is a stationary and causal VAR ($p$) process if and only if the
roots of the determinantal
equation,$|\bm{\Phi}(z)|=\left|\mathbf{I}-\Phi_{1}z-\cdots-\Phi_{p}z^{p}\right|=0$
lie outside the unit circle. A detailed proof see Brockwell et al. and Reinsel
et al. [29, 30]

###  3.2 Variables potentially correlated to outcome

Several potential variables might influence the number of COVID-19 positive
cases according to literature [15, 23, 24, 25, 26, 27, 28]: undetected
infected, detected deaths, detected recovered, average temperature,
precipitation, wind speed, humidity, population density, social trust, civic
engagement, that are considered in other publications of epidemic prediction.

In Chowdhury et al. [31], climate changes directly affect five infectious
disease transmission. Altered climatic conditions may increase the vector
biting rate and the vector’s reproduction rate and shorten the pathogen
incubation period. Furthermore, depending on the report, If the temperature is
higher than 25.0∘C, there is a significant negative correlation between
increasing temperature and pneumonia ($p=0.017$). [31] That is, if the
temperature is decreasing under 25.0∘C, pneumonia would spread out faster.
[31] In Liu et al. [32], when the temperature is lower than 13.0∘C, the number
of hospital admission increases, which means the speed of infection also rises
up. Those are the reason why COVID-19 positive confirmed cases appear rebound
tendency after October. [32]

Depending on data reported by the Tasci et al. [33], during the periods of
high, normal, and low humidity, the number of days admitted with pneumonia was
higher at high humidity rates ($p<0.05$). AS a result, the speed of COVID-19
transmission would increase at high humidity situation. In other words, the
positive confirmed cases show a significant positive relationship with
humidity.[33]

According to Brundage et al. [34], the pneumonia rate has a stronger positive
correlation with mortality. The mortality increase would affect COVID-19
spreading out faster than before. However, the number of death increased would
happen after COVID-19 transmission rising. [34]

Recovered cases should also have a negative correlation with COVID-19. If
recovered cases become more, the number of patients with the virus should be
less than before. As a result, fewer patients with the virus would match the
lower spread of the virus. When the recovered cases are increasing, the
transmission of COVID-19 transmission would be controlled.

###  3.3 Model selection

As shown in Section 3.1, we need determine the lag order $p$ of the VAR model.
There are diverse criteria, Akaike Information Criterion (AIC), Hannan-Quinn
Criterion (HQC), Schwarz Criterion (SC), and Final Prediction Error (FPE), to
find the optimal $p$. Specifically, AIC is an estimator of out-of-sample
prediction error and thereby relative quality of statistical models for a
given set of data. [35, 36] Suppose that we have a statistical model of some
data. Let $k$ be the number of estimated parameters in the model and $L$ be
the maximum value of the likelihood function for the model. Then the AIC value
of the model is $\mathrm{AIC}=2k-2\ln(L)$. [37, 38] HQC is an alternative to
AIC and Bayesian information criterion (BIC). It is given as
$\mathrm{HQC}=-2L+2k\ln(\ln(n))$, where $n$ is the number of observations.

Schwarz criterion (SC) is given as
$\mathrm{SC}=\log(\mathrm{n})\mathrm{k}-2\log(\mathrm{L}(\hat{\theta}))$,
where $\theta$ is set of all parameter values and $L(\hat{\theta})$ is
likelihood of the model returning the data we have, when tested at the maximum
likelihood values of $\theta$. Final Prediction Error (FPE) criterion provides
a measure of model quality by simulating the situation where the model is
tested on a different data set. It is givin as
$\operatorname{det}\left(\frac{1}{n}\sum_{1}^{n}e\left(t,\hat{\theta}_{i}\right)\left(e\left(t,\hat{\theta}_{i}\right)\right)^{\text{T}}\right)\left(\frac{1+d_{n}}{1-d_{n}}\right)$,
where n is the number of values in the estimation data set, $e(t)$ is a
$n$-by-$1$ vector of prediction errors, $\hat{\theta}_{i}$ represents the
$i$-th estimated parameters, $d$ is the number of estimated parameters.

The ordinary least square (OLS) approach is applied to achieve the model
estimation. Besides, the model residuals are diagnosed to see if the VAR model
assumptions meet.

##  4 Result

###  4.1 Preliminary analysis

According to literature and correlation analysis, we include cumulative death,
cumulative recovered patients, temperature and humidity in the VAR model.
Considering the positive cases prediction nationwide, we choose the climate
data of Washington D.C. that could be representative.

The correlation analyses are shown in Figure 1. Even though ’Death’ and
’Humidity’ have relatively small correlative coefficients, 0.071 and 0.016,
with daily positive case increase, we still keep these two variables. Because
the COVID-19 have been verified correlated with cumulative death cases and
different humidity.[33, 34]

Figure 1: Correlation matrix plot. (Death: Cumulative death case; Pos.Case:
daily positive case; Recovered: cumulative recovered case; Temp: temperature)

A descriptive analysis for cumulative death case, cumulative recovered case,
temperature, humidity is shown in Figure 2. The cumulative death cases and the
cumulative recovered cases presents straight up tendency.

Since co-integration between daily positive cases and other selected variables
is required by VAR model. We run the co-integration test (Engle Granger test)
on all the variables (series). This test is for daily increase positive cases
with other variables. The null hypothesis is that there is no co-integration
relationship between the two variables. If the variables are all co-integrated
with daily positive cases, we can claim that they have are stably correlated
in a long run. Results see Appendix Table 3. As all $p$-values are less than
0.05, we have all variables co-integrated with daily increase positive cases.
Based on the above results, it is considered that there is a stable
relationship and there is no spurious regression for the constructed model.

Figure 2: Time series from March 25 to November 24, 2020: (a) Cumulative death
of COVID-19 cases in US; (b) Cumulative recovered cases; (c) Temperature in
Washington D.C.; (d) Humidity in Washington D.C..

###  4.2 Model selection

To determine the lag order of our VAR model, we take AIC, HQC, SC and FPE into
consideration (results see Appendix Table 4). The optimal lag order is
determined to be 8.

With the suggested lag order 8, we estimate our model using ordinary least
square technique. We show the parameter estimates in Table 1.

Table 1: VAR model parameters estimation1

Lag order/ Variables | Death | Pos.Case | Recovered | Temp | Humidity
---|---|---|---|---|---
1 | 3.62 | 2.29 | 7.49 | 1.30 | 4.97
2 | -5.07 | -1.28 | -3.46 | -3.68 | -1.21
3 | 3.25 | -4.38 | -1.53 | -2.96 | 4.12
4 | -5.00 | -40.62 | 9.61 | 2.66 | -3.91
5 | 9.06 | 2.09 | 1.67 | -1.61 | -1.69
6 | 7.43 | -2.46 | -2.92 | -3.96 | 8.07
7 | -6.89 | 5.80 | -1.50 | 3.61 | -6.47
8 | 1.53 | -1.29 | 1.94 | -5.22 | -2.83

Note: 1 “constant” item is 8327.24.

To verify the assumptions of VAR model, we plot residuals and residuals
autocorrelation as shown in Appendix Figure 4. The mean value of residual is
almost zero (-1.45e-14) and autocorrelation coefficients are within 95%
confidence interval (CI; blue dotted line). We also test the residuals by
Ljung-Box test and have p-value 0.20 (null hypothesis is that the data are
independently distributed). Hence, the fitted model satisfies the assumptions
mentioned above: $\mathbb{E}\left(e_{t}\right)=0$ and
$\mathbb{E}\left(e_{t}e_{t-k}^{\text{T}}\right)=0$, where $e_{t}$ is the
residual at time $t$.

###  4.3 COVID-19 daily positive cases prediction

The objective is predicting the trend of the daily increase positive cases. We
predict 30-day daily positive cases starting from July 2, August 21, and
November 24, respectively, for internal validation purpose. We pick these
three dates for particular reasons. First, 30-day daily increase positive
cases after July 2 and August 21 are not fluctuating too much. It is a
preliminary test on model performance. Critically, the trend after September 6
becomes steep suddenly. However, the trend before September 6 is similar to
before July 2 and before August 21. It may be trapped for the model to
identify these three conditions. We want to test if the model will predict the
correct rapid increase after September 6.

Figure 3: Real data (black line) and prediction (red line) with 95% confidence
interval (blue dotted line).
(i). For validation purpose: (a) 30 days prediction from July 2, 2020. (b) 30
days prediction from August 21, 2020. (c) 30 days prediction from September 6,
2020.
(ii). Real prediction: (d) 30 days prediction from November 24, 2020.

As mentioned, the first three plots (a) – (c) in Figure 3 are for internal
validation purpose. As we can see that the model is useful, since the real
data (black) is covered by the predicted 95% confidence interval (blue dotted
line). To be specific, in Figure 3 (a), the black line is the real selected
positive confirmed cases daily data, which presents a stable tendency in the
first three months, around 40,000 cases every day. After that, in the middle
of June, the real data begin to increase. The short red color line is our
prediction using proposed model. The black line and the red line are almost
overlapped. One thing need to mention is that the black line fluctuates
slightly larger than the red, but the predicted is mostly covered by the 95%
confidence interval. It concludes a satisfied prediction. In Figure 3 (b),
after the middle of July, the number of daily positive confirmed cases
decreases and experiences the first peek of 80,000 cases. However, both the
black and red lines show a stable trend then, and the figure shows almost the
same appearance as the first 30-day prediction. In Figure 3 (c), the real data
experiences a decreasing trend. But, at the beginning of September, the number
of daily positive confirmed cases appears a rebound, directed straight up to
the second peak value. The peak value even reaches two hundred thousand cases
for one day. Our predicted values are a little lower than real values. The
reason can include Halloween holiday parties and some assemblies because those
happened at the end of October, and many COVID-19 cases can be confirmed in
early November. Furthermore, those are some extrinsic factor besides the ones
in our VAR model. As a result, it is reasonable that the black line is higher
than the red prediction line and the confidence interval’s upper bound. The
success is that the model correctly predicts the rapidly increasing trend
after August 21.

The last plot (d) in Figure 3 is our main result that predicts the daily
positive COVID-19 cases 30 days later starting from November 24 (now), that
is, a prediction for unknown future trend (to December 24). It is obvious that
the the future 30-day growth trend will increase if government are not taking
any new measures to control the transmission of COVID-19. During the
Christmas, the predicted daily positive case is around 240,000 in US.

Table 2 shows a comparison of real values and predictions with a 95%
confidence interval. Considering that the daily cases increase data on Monday
is partly derived from the cases accumulation over the weekend, we compare the
predicted data on each Tuesday with the real values.

Table 2: The real values and predictions of the daily increase positive cases
on Tuesday with 95% confidence interval1.

Date | Real value | Prediction | Lower 95% CI | Upper 95% CI
---|---|---|---|---
2020-07-09 | 58961 | 64116 | 58240 | 69993
2020-07-16 | 70446 | 66497 | 57820 | 75173
2020-07-23 | 71225 | 66129 | 56216 | 76043
2020-07-30 | 68806 | 71059 | 60634 | 81484
2020-08-25 | 36588 | 35466 | 28817 | 42115
2020-09-01 | 42426 | 30940 | 21343 | 40536
2020-09-08 | 22137 | 30037 | 17357 | 42717
2020-09-15 | 34904 | 32092 | 16668 | 47515
2020-11-03 | 86662 | 86890 | 77433 | 96348
2020-11-10 | 131182 | 104960 | 89508 | 120411
2020-11-17 | 156722 | 121455 | 100033 | 142878
2020-11-24 | 167012 | 152604 | 123769 | 181438
2020-12-01 |  | 176995 | 161998 | 191993
2020-12-08 |  | 185548 | 160309 | 210787
2020-12-15 |  | 194945 | 159893 | 229998
2020-12-22 |  | 208196 | 164852 | 251539

1 Values on December 1, December 8, December 15, December 22 are blank since
real data are not available until now.

In Table 2, the real values are generally within the 95% confidence level. For
the predictions on November 3, 10, 17, 24, the model predicts 86,890, 104,960,
121,455, 152,604 and upper bounds are 96,348, 120,411, 142,878 and 181,438.
The real values exceed upper bounds on November 10 and 17 by around 10,000. It
still shows the real values are within the confidence interval since the real
values do not deviate from the upper bound too far. The model has good
performance of catching a rapid increase trend and regular trend.

##  5 Discussion

The study proposed and applied the VAR model for predicting the dynamics of
daily COVID-19 positive cases. We selected relevant variables according to
literature and checked their correlation coefficients and co-integration. We
evaluated our model by comparing the predicted values and real values.

We can introduce more relevant variables in the future to improve the
performance if outside force appears to influence viral transmission, control
or exacerbate. The most possible variables available may be the estimation of
social distance and the number of vaccination. Then the model will be still
useful after vaccine comes out. It enables the model to predict the decrease
of infections at the vaccination initial stage. It is also the reason why we
investigate the application of VAR model on pandemic predictions. The VAR
model is different from and better than SIR and ARIMA. Because SIR and ARIMA
have an unsatisfied performance when outside force gets involved.

The proposed model can be strongly generalized because it is not limited to
specific data, since the structure of the model is constructed. Based on the
generalization, this model can be used to predict other epidemics with the
same characteristics as COVID-19.

## References

  * Lu et al. [2020] Lu R, Zhao X, Li J, Niu P, Yang B, Wu H, et al.  Genomic characterisation and epidemiology of 2019 novel coronavirus: implications for virus origins and receptor binding.  The Lancet 2020;395(10224):565–574.
  * Chen et al. [2020] Chen Y, Liu Q, Guo D.  Emerging coronaviruses: genome structure, replication, and pathogenesis.  Journal of medical virology 2020;92(4):418–423.
  * Tang et al. [2020] Tang B, Wang X, Li Q, Bragazzi NL, Tang S, Xiao Y, et al.  Estimation of the transmission risk of the 2019-nCoV and its implication for public health interventions.  Journal of clinical medicine 2020;9(2):462.
  * Wang et al. [2006] Wang L, Kraemer R, Borngraeber J.  An improved highly-linear low-power down-conversion micromixer for 77 GHz automotive radar in SiGe technology.  In: 2006 IEEE MTT-S International Microwave Symposium Digest IEEE; 2006\. p. 1834–1837.
  * Kufel et al. [2020] Kufel T, et al.  ARIMA-based forecasting of the dynamics of confirmed Covid-19 cases for selected European countries.  Equilibrium Quarterly Journal of Economics and Economic Policy 2020;15(2):181–204.
  * Konarasinghe [2020] Konarasinghe K.  Modeling COVID-19 Epidemic of USA, UK and Russia.  Journal of New Frontiers in Healthcare and Biological Sciences 2020;1(1):1–14.
  * Guan et al. [2020] Guan Wj, Ni Zy, Hu Y, Liang Wh, Ou Cq, He Jx, et al.  Clinical characteristics of 2019 novel coronavirus infection in China.  MedRxiv 2020;.
  * Maleki et al. [2020] Maleki M, Mahmoudi MR, Wraith D, Pho KH.  Time series modelling to forecast the confirmed and recovered cases of COVID-19.  Travel Medicine and Infectious Disease 2020;p. 101742.
  * Malavika et al. [2020] Malavika B, Marimuthu S, Joy M, Nadaraj A, Asirvatham ES, Jeyaseelan L.  Forecasting COVID-19 epidemic in India and high incidence states using SIR and logistic growth models.  Clinical Epidemiology and Global Health 2020;.
  * Dhanwant and Ramanathan [2020] Dhanwant JN, Ramanathan V.  Forecasting COVID 19 growth in India using Susceptible-Infected-Recovered (SIR) model.  arXiv preprint arXiv:200400696 2020;.
  * Ndiaye et al. [2020] Ndiaye BM, Tendeng L, Seck D.  Analysis of the COVID-19 pandemic by SIR model and machine learning technics for forecasting.  arXiv preprint arXiv:200401574 2020;.
  * Bastos and Cajueiro [2020] Bastos SB, Cajueiro DO.  Modeling and forecasting the Covid-19 pandemic in Brazil.  arXiv preprint arXiv:200314288 2020;.
  * Fanelli and Piazza [2020] Fanelli D, Piazza F.  Analysis and forecast of COVID-19 spreading in China, Italy and France.  Chaos, Solitons & Fractals 2020;134:109761.
  * Chen et al. [2020] Chen YC, Lu PE, Chang CS, Liu TH.  A Time-dependent SIR model for COVID-19 with undetectable infected persons.  IEEE Transactions on Network Science and Engineering 2020;.
  * Khan et al. [2020] Khan Z, Van Bussel F, Hussain F.  A predictive model for Covid-19 spread–with application to eight US states and how to end the pandemic.  Epidemiology & Infection 2020;148.
  * Xu et al. [2020] Xu C, Yu Y, Yang Q, Lu Z.  Forecast analysis of the epidemics trend of COVID-19 in the United States by a generalized fractional-order SEIR model.  arXiv preprint arXiv:200412541 2020;.
  * Zeroual et al. [2020] Zeroual A, Harrou F, Dairi A, Sun Y.  Deep learning methods for forecasting COVID-19 time-Series data: A Comparative study.  Chaos, Solitons & Fractals 2020;140:110121.
  * Shahid et al. [2020] Shahid F, Zameer A, Muneeb M.  Predictions for COVID-19 with deep learning models of LSTM, GRU and Bi-LSTM.  Chaos, Solitons & Fractals 2020;140:110212.
  * Chimmula and Zhang [2020] Chimmula VKR, Zhang L.  Time series forecasting of COVID-19 transmission in Canada using LSTM networks.  Chaos, Solitons & Fractals 2020;p. 109864.
  * Alzahrani et al. [2020] Alzahrani SI, Aljamaan IA, Al-Fakih EA.  Forecasting the spread of the COVID-19 pandemic in Saudi Arabia using ARIMA prediction model under current public health interventions.  Journal of infection and public health 2020;13(7):914–919.
  * Sahai et al. [2020] Sahai AK, Rath N, Sood V, Singh MP.  ARIMA modelling & forecasting of COVID-19 in top five affected countries.  Diabetes & Metabolic Syndrome: Clinical Research & Reviews 2020;14(5):1419–1427.
  * Kumar et al. [2020] Kumar P, Kalita H, Patairiya S, Sharma YD, Nanda C, Rani M, et al.  Forecasting the dynamics of COVID-19 Pandemic in Top 15 countries in April 2020: ARIMA Model with Machine Learning Approach.  medRxiv 2020;.
  * Siedner et al. [2020] Siedner MJ, Harling G, Reynolds Z, Gilbert RF, Haneuse S, Venkataramani AS, et al.  Social distancing to slow the US COVID-19 epidemic: Longitudinal pretest–posttest comparison group study.  PLoS medicine 2020;17(8):e1003244.
  * Behnood et al. [2020] Behnood A, Golafshani EM, Hosseini SM.  Determinants of the infection rate of the COVID-19 in the US using ANFIS and virus optimization algorithm (VOA).  Chaos, Solitons & Fractals 2020;139:110051.
  * Bialek et al. [2020] Bialek S, Bowen V, Chow N, Curns A, Gierke R, Hall A, et al.  Geographic differences in COVID-19 cases, deaths, and incidence—United States, February 12–April 7, 2020 2020;.
  * Elgar et al. [2020] Elgar FJ, Stefaniak A, Wohl MJ.  The trouble with trust: Time-series analysis of social capital, income inequality, and COVID-19 deaths in 84 countries.  Social Science & Medicine 2020;263:113365.
  * Bruine de Bruin [2020] Bruine de Bruin W.  Age differences in COVID-19 risk perceptions and mental health: Evidence from a national US survey conducted in March 2020.  The Journals of Gerontology: Series B 2020;.
  * James and Menzies [2020] James N, Menzies M.  Cluster-based dual evolution for multivariate time series: Analyzing COVID-19.  Chaos: An Interdisciplinary Journal of Nonlinear Science 2020;30(6):061108.
  * Brockwell et al. [1991] Brockwell PJ, Davis RA, Fienberg SE.  Time series: theory and methods: theory and methods.  Springer Science & Business Media; 1991.
  * Reinsel [2003] Reinsel GC.  Elements of multivariate time series analysis.  Springer Science & Business Media; 2003.
  * Chowdhury et al. [2018] Chowdhury FR, Ibrahim QSU, Bari MS, Alam MJ, Dunachie SJ, Rodriguez-Morales AJ, et al.  The association between temperature, rainfall and humidity with common climate-sensitive infectious diseases in Bangladesh.  PLoS One 2018;13(6):e0199579.
  * Liu et al. [2014] Liu Y, Kan H, Xu J, Rogers D, Peng L, Ye X, et al.  Temporal relationship between hospital admissions for pneumonia and weather conditions in Shanghai, China: a time-series analysis.  BMJ open 2014;4(7).
  * Tasci et al. [2018] Tasci SS, Kavalci C, Kayipmaz AE.  Relationship of meteorological and air pollution parameters with pneumonia in elderly patients.  Emergency Medicine International 2018;2018.
  * Brundage and Shanks [2008] Brundage JF, Shanks GD.  Deaths from bacterial pneumonia during 1918–19 influenza pandemic.  Emerging infectious diseases 2008;14(8):1193.
  * McElreath [2020] McElreath R.  Statistical rethinking: A Bayesian course with examples in R and Stan.  CRC press; 2020.
  * Taddy [2019] Taddy M.  Business data science: Combining machine learning and economics to optimize, automate, and accelerate business decisions.  McGraw Hill Professional; 2019.
  * Burnham and Anderson [2002] Burnham KP, Anderson DR.  A practical information-theoretic approach.  Model selection and multimodel inference, 2nd ed Springer, New York 2002;2.
  * Akaike [1974] Akaike H.  A new look at the statistical model identification.  IEEE transactions on automatic control 1974;19(6):716–723.

## Appendix

Table 3: Result for Engle-Granger test (co-integration test) Variables | Statistics | p-value | Co-integration
---|---|---|---
Cumulative death cases | 0.0656 | < .0001 | Y
Cumulative recovered cases | 0.0864 | < .0001 | Y
Temperature | 0.0472 | < .0001 | Y
Humidity | 0.0373 | < .0001 | Y
Table 4: Lag order selection: AIC, HQ, SC, and FPE Lag order | AIC | HQC | SC | FPE
---|---|---|---|---
1 | 5.57 | 5.59 | 5.62 | 1.57
2 | 5.50 | 5.54 | 5.59 | 7.82
3 | 5.48 | 5.53 | 5.61 | 6.52
4 | 5.47 | 5.53 | 5.63 | 5.67
5 | 5.45 | 5.53 | 5.65 | 4.76
6 | 5.44 | 5.54 | 5.68 | 4.45
7 | 5.42 | 5.52 | 5.69 | 3.42
8 | 5.39 | 5.51 | 5.69 | 2.49
9 | 5.39 | 5.53 | 5.73 | 2.55
10 | 5.39 | 5.55 | 5.78 | 2.76
Figure 4: Residual plot (left) and its autocorrelation (right). Blue dash
lines are upper and lower bound of 95% confidence interval.