# Selection of Time Headway in Connected and Autonomous Vehicle Platoons under
Noisy V2V Communication

Guoqi Ma, Prabhakar R. Pagilla∗, Swaroop Darbha The authors are with the
Department of Mechanical Engineering, Texas A&M University, College Station,
TX 77843, USA (e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS>dswaroop@tamu.edu).
∗Corresponding author. Part of this work was presented at the 26th IEEE
International Conference on Intelligent Transportation Systems (ITSC), Bilbao,
Spain, 2023 [1].

###### Abstract

In this paper, we investigate the selection of time headway to ensure robust
string stability in connected and autonomous vehicle platoons in the presence
of signal noise in Vehicle-to-Vehicle (V2V) communication. In particular, we
consider the effect of noise in communicated vehicle acceleration from the
predecessor vehicle to the follower vehicle on the selection of the time
headway in predecessor-follower type vehicle platooning with a Constant Time
Headway Policy (CTHP). Employing a CTHP based control law for each vehicle
that utilizes on-board sensors for measurement of position and velocity of the
predecessor vehicle and wireless communication network for obtaining the
acceleration of the predecessor vehicle, we investigate how time headway is
affected by communicated signal noise. We derive constraints on the CTHP
controller gains for predecessor acceleration, velocity error and spacing
error and a lower bound on the time headway which will ensure robust string
stability of the platoon against signal noise. We provide comparative
numerical simulations on an example to illustrate the main result.

###### Index Terms:

Connected and Autonomous Vehicle Platoons, Cooperative Adaptive Cruise Control
(CACC), Constant Time Headway Policy (CTHP), V2V Communication, Signal-to-
Noise Ratio (SNR), Robust String Stability.

##  I Introduction

The deployment of connected and autonomous vehicle platoons has the potential
to benefit transportation systems in a profound and comprehensive way [2, 3].
Initially, under the Adaptive Cruise Control (ACC) paradigm facilitated by the
use of onboard sensors (such as radars) for measuring the velocities of and
distances from adjacent vehicles, the vehicle platoon can maintain a constant
inter-vehicular spacing, referred to as Constant Spacing Policy (CSP) [4].
Recently, the benefits of employing advanced vehicular communication
technologies and modern communication protocols have been expounded in great
detail in the literature; for example, Dedicated Short Range Communication
(DSRC) [5], Long Term Evolution (LTE) [6], 5G [7], and V2V communication [8].
In addition to the use of onboard sensors, advanced vehicular wireless
communications can lead to a higher-level of connectivity by incorporating the
acceleration information of other vehicles which has the potential to
significantly improve safety, increase mobility and throughput, and reduce
fuel consumption under the paradigm of Cooperative Adaptive Cruise Control
(CACC) with Constant Time Headway Policy (CTHP) [9].

In addition to internal stability for each vehicle, the connected and
autonomous vehicle platoon should also exhibit robust string stability, i.e.,
robustness of the platoon to uncertain lag, noise, and external disturbances.
In recent decades, there has been extensive research devoted to connected and
autonomous vehicle platoons on a wide range of topics including controller
design [10, 11, 12, 13, 14, 15], communication mechanisms [16, 17, 18],
experimental validation in realistic environments [19, 20], mixed human and
autonomous traffic [21, 22, 23], string stability analysis [24, 25]. However,
most existing results assume ideal V2V communication which is often not
practical. Among many other factors, signal noise in communication channels is
a key concern, which can be caused by a variety of factors including limited
communication bandwidth [26], cyber-attacks [27], etc. If key control
parameters, such as time headway, are chosen without consideration of signal
noise in communicated signals, then there is a possibility of the onset of
string instability and collisions. Although communication quality has improved
significantly under 5G and more advanced wireless communication, signal noise
when propagating through the platoon could have a substantial affect on safety
and performance.

Previous research on imperfect communication focused on how packet drops
affected string stability [28]. In [29, 30], the disturbance estimation and
control problem was investigated, though without considering the quantitative
effect of disturbance on the design parameters. In contrast to prior research
with imperfect communication, this paper considers the following problem:
Given a Signal-to-Noise Ratio (SNR) with V2V communication, what is the lower
bound of the employable time headway for CACC based vehicle platooning? For
this purpose, we consider a CTHP based control law where the feedforward
acceleration of the predecessor vehicle is communicated and contains signal
noise. Since the communication is through an $n$-bit channel, we model the
noise in the acceleration signal as a sum of $n$ binary random variables. To
address the issue of the vehicle closed-loop governing equation being
stochastic due to the noise signal, we develop an equivalent deterministic
governing equation using the averaging procedure described in [28]. Based on
this, we obtain the spacing error propagation equation and show that the
platoon is robustly string stable in the presence of signal noise and
parasitic lag and derive conditions on control gains and time headway under
which the platoon is robustly string stable. We provide numerical simulation
results on a platoon to verify the main result.

The main contributions of the work can be summarized as follows. In the
context of communication from a single predecessor vehicle, this paper
provides an extension to our previous work in [9] by considering signal noise
in communicated acceleration from the predecessor vehicle. We derive a lower
bound on the time headway that is a function of the parasitic lag in the
vehicle dynamics ($\tau_{0}$), acceleration gain ($k_{a}$), and SNR ($\rho$).
This lower bound aids in the selection of the time headway in the presence of
noise. Further, the lower bound reduces to the ideal communication case given
in [9] with no noise in the communicated acceleration signal. In addition, our
formulation and analysis allow for systematic selection of the control gains
as well as time headway for a given signal to noise ratio.

The remainder of the paper is organized as follows. Section II contains
preliminaries including vehicle dynamics and relevant definitions. The main
theoretical results are provided in Section III. An illustrative numerical
example and simulation results are provided in Section IV. Finally, some
concluding remarks are given in Section V.

##  II Preliminaries

Consider a string of autonomous vehicles equipped with V2V communication as
illustrated in Fig. 1.

Figure 1: Autonomous and connected vehicle platoon with V2V communication.

The $i$-th vehicle dynamics is given by the model:

$\displaystyle\left\\{\begin{array}[]{*{20}{l}}\ddot{x}_{i}(t)=a_{i}(t),\\\
\tau\dot{a}_{i}(t)+a_{i}(t)=u_{i}(t),\end{array}\right.$ (3)

where $x_{i}(t)$, $a_{i}(t)$, $u_{i}(t)$ represent the position, acceleration,
and control input of the $i$-th vehicle at time instant $t$, and
$i\in\mathcal{N}=\\{1,2,\cdots,N\\}$, where $N$ is the total number of the
following vehicles in the platoon, $\tau$ denotes the parasitic actuation lag.
It is assumed that $\tau$ is uncertain with $\tau\in\left(0,\tau_{0}\right]$,
where $\tau_{0}$ is a positive real constant.

###### Definition II.1

Let $d$ denote the minimum or standstill spacing between adjacent vehicles,
$v_{i}(t)$ denote the velocity of the $i$-th vehicle, $h_{w}$ denote the time
headway, and $e_{i}(t):=x_{i}(t)-x_{i-1}(t)+d$ be the spacing error for the
$i$-th vehicle with respect to the $(i-1)$-th vehicle. Define the velocity
dependent inter-vehicular spacing error for the $i$-th vehicle as:

$\displaystyle\delta_{i}(t)=e_{i}(t)+h_{w}v_{i}(t).$ (4)

######  Definition II.2 ([9])

Let $\delta_{i}(s)$ denote the Laplace transform of $\delta_{i}(t)$ and let
$H(s)$ denote the spacing error propagation transfer function that satisfies
$\delta_{i}(s)=H(s)\delta_{i-1}(s)$. The connected and autonomous vehicle
platoon is said to be robustly string stable if the following two conditions
hold for all $\tau\in(0,\tau_{0}]$: (i) $H(s)$ achieves internal stability and
(ii) the platoon is string stable, i.e., it holds that
$\|\delta_{i}(t)\|_{\infty}\leq\|\delta_{i-1}(t)\|_{\infty}$, or in the
frequency domain

$\displaystyle\|H(j\omega)\|_{\infty}\leq 1.$ (5)

###### Definition II.3

Let $s(t)$ and $n(t)$ denote the transmitted signal and the noise signal,
respectively, and assume $n(t)=0$ when $s(t)=0$. Then, the Signal-to-Noise
Ratio (SNR) for $s(t)\neq 0$ is defined as

$\displaystyle{\rm SNR}=\min\limits_{t>0}20\log\frac{|s(t)|}{|n(t)|},$ (6)

where $\rm log$ denotes the logarithmic function with base 10. From the ${\rm
SNR}$ definition (6), we can obtain that if ${\rm SNR}\coloneqq\varrho$, then
$\rho=\min\limits_{t>0}\frac{|s(t)|}{|n(t)|}=10^{\frac{\varrho}{20}}$, and for
notational convenience, we will use
$\rho=\min\limits_{t>0}\frac{|s(t)|}{|n(t)|}$ instead of ${\rm SNR}$.

##  III Main Results

###  III-A Model of Noise

Let $a_{i-1}(t)$ denote the acceleration signal of the preceding vehicle
($(i-1)$-th) and $w_{i,i-1}(t)>0$ denote the noise factor associated with V2V
communication of the acceleration signal of the $(i-1)$-th vehicle to the
$i$-th vehicle. Considering the noise affected signal, the communicated
acceleration signal available to the $i$-th vehicle is given by
$w_{i,i-1}(t)a_{i-1}(t)$. We define $\rho>1$ as the signal to noise ratio
factor such that $1-\frac{1}{\rho}\leq w_{i,i-1}(t)\leq 1+\frac{1}{\rho}$,
that is, the communicated acceleration signal to the $i$-th vehicle satisfies
$\left|\left(1-\frac{1}{\rho}\right)a_{i-1}(t)\right|\leq\left|w_{i,i-1}(t)a_{i-1}(t)\right|\leq\left|\left(1+\frac{1}{\rho}\right)a_{i-1}(t)\right|$.
An illustration of the admissible region of the noise signal in terms of the
ideal communicated acceleration signal is provided in Fig. 2, and the range of
the communicated signal with respect to the actual signal is provided in Fig.
3.

Figure 2: An illustration of the admissible region for the signal noise in the
communicated acceleration signal ($\rho=5$).  Figure 3: An illustration of the
admissible region of the communicated acceleration signal with respect to the
actual one ($\rho=5$).

Considering the communicated acceleration signal is through an $n$-bit
communication channel, we model $w_{i,i-1}(t)$ as follows:

$\displaystyle
w_{i,i-1}(t)=\left(1-\frac{1}{\rho}\right)+\frac{1}{\rho}\left(\sum_{j=0}^{n-1}\frac{z_{i,j}(t)}{2^{j}}\right),$
(7)

where $z_{i,j}(t),j=0,\ldots,n-1,$ are independent binary random processes.
Let $\gamma_{i,j}:=\mathbb{E}[z_{i,j}(t)]$. We assume that $\gamma_{i,j}$ is
independent of time, i.e., the characteristics of the noise processes of the
communication channel are time-invariant. However, in order to model wide
variety of noise processes, we do not assume that $\gamma_{i,j}$’s are known a
priori. Note that $\gamma_{i,j}\in(0,1)$ then this noise modeling procedure
will allow us to get a communicated signal that takes any value lying between
$\left(1-\frac{1}{\rho}\right)a_{i,i-1}$ and
$\left(1+\frac{1}{\rho}\right)a_{i,i-1}$.

###  III-B Inter-vehicular Spacing Error Propagation Equation

Consider the following CTHP control law for the $i$-th vehicle:

$\displaystyle
u_{i}(t)=k_{a}w_{i,i-1}(t)a_{i-1}(t)-k_{v}(v_{i}(t)-v_{i-1}(t))-k_{p}\delta_{i}(t),$
(8)

then the governing equation for the $i$-th vehicle is given by:

$\displaystyle\tau\dddot{x}_{i}(t)+\ddot{x}_{i}(t)$
$\displaystyle=k_{a}w_{i,i-1}(t)a_{i-1}(t)$ $\displaystyle\quad-
k_{v}(v_{i}(t)-v_{i-1}(t))-k_{p}\delta_{i}(t).$ (9)

The governing equation is stochastic due to the presence of the stochastic
noise signal $\omega_{i,i-1}(t)$; correspondingly, the state is also
stochastic. To perform robust string stability analysis, we consider an
equivalent deterministic governing equation along the lines described in [28].
By defining the augmented state vector as
$\hat{X}(t)=[x_{0}(t),v_{0}(t),a_{0}(t),x_{1}(t),v_{1}(t),a_{1}(t),\cdots,x_{N}(t),v_{N}(t),a_{N}(t)]$
and letting $\hat{\omega}(t)$ be the vector of noise signals with its $i$-th
component being $\omega_{i,i-1}(t)$, and the acceleration input to the lead
vehicle to be $U(t)$, the augmented state equation for the vehicle platoon can
be developed as

$\displaystyle\dot{\hat{X}}(t)=\hat{A}(\hat{w}(t))\hat{X}(t)+BU(t),$ (10)

where the structures of the matrices $\hat{A}$ and $B$ are given in [28].
Through a discretization process for a sufficiently small time interval, it
was shown that

$\displaystyle\mathbb{E}[\hat{A}]=\bar{A},\quad\mathbb{E}[e^{\hat{A}t}]=e^{\bar{A}t},$
(11)

where $\bar{A}=\hat{A}(\bar{w})$ with $\bar{w}$ as the expectation of
$\hat{w}(t)$. The second equality is true because of the specific lower
triangular and banded structure of the governing equations for the CACC case.
Let $\bar{X}=\mathbb{E}[\hat{X}]$; in the following, unless specified
otherwise, we will use bar to indicate expected value for a corresponding
random variable. The evolution of the vehicle states in the platoon converge
to the states of the following deterministic state equation:

$\displaystyle\dot{\bar{X}}(t)=\bar{A}\bar{X}(t)+BU(t).$ (12)

Utilizing this procedure, the governing equation for the $i$-th vehicle is
given by:

$\displaystyle\tau\dddot{\bar{x}}_{i}(t)+\ddot{\bar{x}}_{i}(t)$
$\displaystyle=k_{a}\mathbb{E}[w_{i,i-1}(t)]\bar{a}_{i-1}(t)$
$\displaystyle\quad-
k_{v}(\bar{v}_{i}(t)-\bar{v}_{i-1}(t))-k_{p}\bar{\delta}_{i}(t).$ (13)

Let

$\displaystyle\tilde{k}_{a}:=k_{a}\mathbb{E}[w_{i,i-1}(t)]=\left[1-\frac{1}{\rho}+\frac{1}{\rho}\left(\sum_{j=0}^{n-1}\frac{\gamma_{i,j}}{2^{j}}\right)\right]k_{a}.$
(14)

The governing equation for the $i$-th vehicle can be re-written as:

$\displaystyle\tau\dddot{\bar{x}}_{i}(t)+\ddot{\bar{x}}_{i}(t)$
$\displaystyle=\tilde{k}_{a}\ddot{\bar{x}}_{i-1}(t)$ $\displaystyle\quad-
k_{v}(\dot{\bar{x}}_{i}(t)-\dot{\bar{x}}_{i-1}(t))-k_{p}\bar{\delta}_{i}(t).$
(15)

Let
$\bar{\delta}_{i}(t)=\bar{x}_{i}(t)-\bar{x}_{i-1}(t)+d+h_{w}\dot{\bar{x}}_{i}(t)$,
and let $\bar{\delta}(s)$ be the Laplace transform of $\bar{\delta}_{i}(t)$.
Then, the inter-vehicular spacing error propagation equation can be derived as

$\displaystyle\bar{\delta}_{i}(s)=\tilde{H}(s;\tau)\bar{\delta}_{i-1}(s),$
(16)

where $\tilde{H}(s;\tau)=\tilde{\mathcal{N}}(s)/\mathcal{D}(s)$ is the inter-
vehicular spacing error propagation transfer function with

$\displaystyle\tilde{\mathcal{N}}(s)$
$\displaystyle=\tilde{k}_{a}s^{2}+k_{v}s+k_{p},$ $\displaystyle\mathcal{D}(s)$
$\displaystyle=\tau s^{3}+s^{2}+\gamma s+k_{p},$

where $\gamma:=k_{v}+h_{w}k_{p}$.

The platoon error propagation equation given by (III-B) was studied in Theorem
1 of [9] which can be recast for the case considered in this paper as follows:

###### Theorem III.1

The following are true for the platoon given by the inter-vehicular spacing
error propagation equation (16):

  1. (a)

$\|\tilde{H}(j\omega;\tau)\|_{\infty}\leq 1$, $\forall\tau\in(0,\tau_{0}]$,
implies that $\tilde{k}_{a}\in(0,1)$;

  2. (b)

given any $\tilde{k}_{a}\in(0,1)$, $h_{w}$ satisfying the following:

$\displaystyle h_{w}\geq\frac{2\tau_{0}}{1+\tilde{k}_{a}},$ (17)

there exist $k_{p},k_{v}>0$ such that
$\|\tilde{H}(j\omega;\tau)\|_{\infty}\leq 1$ for all $\tau\in(0,\tau_{0}]$.

###  III-C Stability Analysis

###### Theorem III.2

The following are true for the platoon given by inter-vehicular spacing error
propagation equation (16):

  1. (a)

$\|\tilde{H}(j\omega;\tau)\|_{\infty}\leq 1$, $\forall\tau\in(0,\tau_{0}]$,
implies that $\tilde{k}_{a}\in(0,1)$.

  2. (b)

given any

$\displaystyle k_{a}\in\left(0,\frac{1}{1+\frac{1}{\rho}}\right),$ (18)

and $h_{w}$ satisfying the following:

$\displaystyle
h_{w}>h_{w,lb}(k_{a}):=2\tau_{0}\left(\frac{1-\left(1-\frac{1}{\rho}\right)k_{a}}{1-\left(1+\frac{1}{\rho}\right)^{2}k_{a}^{2}}\right),$
(19)

there exist $k_{p},k_{v}>0$ such that
$\|\tilde{H}(j\omega;\tau)\|_{\infty}\leq 1$ for all $\tau\in(0,\tau_{0}]$;

  3. (c)

given any $\rho>1$, the minimizing value, $k_{a}^{*}$, of $k_{a}$ and the
corresponding minimum value, $h_{w,lb}^{*}$, of $h_{w,lb}(k_{a})$ are given
by:

$\displaystyle
k_{a}^{*}=\left(\frac{1-\frac{1}{\sqrt{\rho}}}{1+\frac{1}{\sqrt{\rho}}}\right)\frac{1}{\left(1+\frac{1}{\rho}\right)},$
(20) $\displaystyle
h_{w,lb}^{*}=\tau_{0}\frac{\left(1+\frac{1}{\sqrt{\rho}}\right)^{2}}{\left(1+\frac{1}{\rho}\right)}.$
(21)

Correspondingly, there exist $k_{p},k_{v}>0$ such that
$\|\tilde{H}(j\omega;\tau)\|_{\infty}\leq 1$ for all $\tau\in(0,\tau_{0}]$.

###### Remark III.1

Minimizing $h_{w,lb}$ is useful for improving traffic throughput because a
lower time headway can be employed while still guaranteeing “robust” string
stability.

###### Proof:

For internal stability, $\gamma-\tau k_{p}>0$ by Routh-Hurwitz criterion on
$\mathcal{D}(s)$. Applying _Theorem III.1_ to the error propagation equation
(16), we infer that:

  1. (a)

$\tilde{k}_{a}\in(0,1)$;

  2. (b)

$h_{w}\geq\dfrac{2\tau_{0}}{1+\tilde{k}_{a}}$.

Since $\gamma_{i,j}$’s are not known, we can bound $\tilde{k}_{a}$ by
considering their maximum possible values. Hence, we have

$\displaystyle\tilde{k}_{a}\leq\left(1-\frac{1}{\rho}+\frac{2}{\rho}\right)k_{a}=\left(1+\frac{1}{\rho}\right)k_{a}.$
(22)

Thus, if

$\displaystyle k_{a}<\frac{1}{1+\frac{1}{\rho}}\implies\tilde{k}_{a}<1,$ (23)

allowing us to apply part (b) of Theorem _Theorem III.1_. We then have

$\displaystyle h_{w}\geq\frac{2\tau_{0}}{1+\tilde{k}_{a}}.$ (24)

Again, since $\tilde{k}_{a}$ is dependent on $\gamma_{i,j}$’s which are not
known a priori, we will consider the worst possible lower bound, i.e., the
lower bound for $h_{w}$ that is a maximum over all possible values of
$\gamma_{i,j}$’s. To address this, we require

$\displaystyle h_{w}$
$\displaystyle\geq\max\limits_{\gamma_{i,0},\ldots,\gamma_{i,n}}\frac{2\tau_{0}}{1+\tilde{k}_{a}}.$
(25)

Hence, if the above condition holds, then (24) will hold. Substituting
$\tilde{k}_{a}$ from (14) into (25), the condition (24) will hold if

$\displaystyle h_{w}$
$\displaystyle\geq\max\limits_{\gamma_{i,0},\ldots,\gamma_{i,n}}\frac{2\tau_{0}}{1+\left(1-\frac{1}{\rho}+\frac{1}{\rho}\left(\sum\limits_{j=0}^{n-1}\frac{\gamma_{i,j}}{2^{j}}\right)\right)k_{a}}$
$\displaystyle=\frac{2\tau_{0}}{\min\limits_{\gamma_{i,0},\ldots,\gamma_{i,n}}\left(1+\left(1-\frac{1}{\rho}+\frac{1}{\rho}\left(\sum\limits_{j=0}^{n-1}\frac{\gamma_{i,j}}{2^{j}}\right)\right)k_{a}\right)}$
$\displaystyle=\frac{2\tau_{0}}{1+\left(1-\frac{1}{\rho}\right)k_{a}}.$ (26)

In [9], the feasible region for the control gains $k_{p}$ and $k_{v}$ is
specific to one chosen value of $\tilde{k}_{a}\in(0,1)$. However, in this
case, $\tilde{k}_{a}$ could be any value in the interval

$\mathcal{I}:=\left[\left(1-\frac{1}{\rho}\right)k_{a},\left(1+\frac{1}{\rho}\right)k_{a}\right]$

for a given $k_{a}\in\left(0,\frac{1}{1+\frac{1}{\rho}}\right)$. Let
$\mathcal{F}(\tilde{k}_{a})$ denote the set of all $(k_{p},k_{v})$ that ensure
robust string stability for a given value of $\tilde{k}_{a}$. Unlike in [9],
we need to show that the set of all feasible $(k_{p},k_{v})$ that work for any
$\tilde{k}_{a}\in\mathcal{I}$ denoted by $\mathcal{S}$ is non-empty, i.e.,

$\mathcal{S}:=\bigcap_{\tilde{k}_{a}\in\mathcal{I}}\mathcal{F}(\tilde{k}_{a})\neq\emptyset.$

In the remainder of the proof, we will focus on showing that
$\mathcal{S}\neq\emptyset$ and explicitly construct this set for synthesizing
the controller. We do so by considering the robust stability condition,
$\|\tilde{H}(j\omega;\tau)\|_{\infty}^{2}\leq 1\;\forall\tau\in(0,\tau_{0}]$
and employing a time headway satisfying (III-C).

According to _Definition II.2_, robust string stability is guaranteed when
$\|\tilde{H}(j\omega;\tau)\|_{\infty}^{2}\leq 1$, i.e.,
$|\tilde{\mathcal{N}}(j\omega)|^{2}\leq|\mathcal{D}(j\omega)|^{2},\forall\omega$,
which can be rewritten as

$\displaystyle\tau^{2}\omega^{4}+(1-\tilde{k}_{a}^{2}-2\tau\gamma)\omega^{2}+\gamma^{2}-2k_{p}-k_{v}^{2}+2\tilde{k}_{a}k_{p}\geq
0.$ (27)

The above condition is satisfied if $\forall\tau\in(0,\tau_{0}]$,

$\displaystyle 1-\tilde{k}_{a}^{2}-2\tau\gamma\geq 0,$ (28a)
$\displaystyle\gamma^{2}-2k_{p}-k_{v}^{2}+2\tilde{k}_{a}k_{p}\geq 0,$ (28b)

i.e.,

$\displaystyle\gamma\leq\frac{1-\tilde{k}_{a}^{2}}{2\tau_{0}},$ (29a)
$\displaystyle\gamma\geq\sqrt{2k_{p}(1-\tilde{k}_{a})+k_{v}^{2}}.$ (29b)

Thus, by considering the appropriate lower and upper bounds of $\tilde{k}_{a}$
in the inequalities (29a) and (29b), these inequalities on $\gamma$ can be
satisfied for all $\tilde{k}_{a}\in\mathcal{I}$ if $\gamma$ satisfies

$\displaystyle\gamma\leq\min\limits_{\tilde{k}_{a}\in\mathcal{I}}\frac{1-\tilde{k}_{a}^{2}}{2\tau_{0}}=\frac{1-\left(1+\frac{1}{\rho}\right)^{2}k_{a}^{2}}{2\tau_{0}},$
(30a)
$\displaystyle\gamma\geq\max\limits_{\tilde{k}_{a}\in\mathcal{I}}\sqrt{2k_{p}(1-\tilde{k}_{a})+k_{v}^{2}}$
$\displaystyle\qquad=\sqrt{2k_{p}\left(1-\left(1-\frac{1}{\rho}\right)k_{a}\right)+k_{v}^{2}},$
(30b)

from which the admissible range of $\gamma$ is given by

$\displaystyle\sqrt{2k_{p}\left(1-\left(1-\frac{1}{\rho}\right)k_{a}\right)+k_{v}^{2}}\leq\gamma\leq\frac{1-\left(1+\frac{1}{\rho}\right)^{2}k_{a}^{2}}{2\tau_{0}}.$
(31)

Next, we will show that the set $\mathcal{S}\neq\emptyset$. First, since
$\gamma=k_{v}+h_{w}k_{p}$, the right inequality in (31) can be rewritten as

$\displaystyle
k_{v}+h_{w}k_{p}\leq\frac{1-\left(1+\frac{1}{\rho}\right)^{2}k_{a}^{2}}{2\tau_{0}}.$
(32)

Thus, an admissible set of $k_{p}$ and $k_{v}$ is given by

$\displaystyle\mathcal{S}_{1}:=\left\\{(k_{p},k_{v}):k_{p}>0,k_{v}>0,\frac{k_{v}}{a_{1}}+\frac{k_{p}}{b_{1}}\leq
1\right\\},$ (33)

where

$\displaystyle\begin{cases}a_{1}=\dfrac{1-\left(1+\frac{1}{\rho}\right)^{2}k_{a}^{2}}{2\tau_{0}},\\\
b_{1}=\dfrac{1-\left(1+\frac{1}{\rho}\right)^{2}k_{a}^{2}}{2\tau_{0}h_{w}}=\frac{1}{h_{w}}a_{1}.\end{cases}$
(34)

Second, the left inequality in (31) can be rewritten as

$\displaystyle
k_{v}+h_{w}k_{p}\geq\sqrt{2k_{p}\left(1-\left(1-\frac{1}{\rho}\right)k_{a}\right)+k_{v}^{2}}.$
(35)

Squaring both sides of (35) and simplifying, we obtain

$\displaystyle 2h_{w}k_{v}+h_{w}^{2}k_{p}\geq
2\left(1-\left(1-\frac{1}{\rho}\right)k_{a}\right).$ (36)

Thus, an admissible set of $k_{p}$ and $k_{v}$ is given by

$\displaystyle\mathcal{S}_{2}:=\left\\{(k_{p},k_{v}):k_{p}>0,k_{v}>0,\frac{k_{v}}{a_{2}}+\frac{k_{p}}{b_{2}}\geq
1\right\\},$ (37)

where

$\displaystyle\begin{cases}{}a_{2}=\dfrac{1-\left(1-\frac{1}{\rho}\right)k_{a}}{h_{w}},\\\
b_{2}=\frac{2\left(1-\left(1-\frac{1}{\rho}\right)k_{a}\right)}{h_{w}^{2}}=\frac{2}{h_{w}}a_{2}.\end{cases}$
(38)

Then, combining (33) and (37), the feasible region of $k_{p}$ and $k_{v}$ is
given by the set

$\displaystyle\mathcal{S}=\mathcal{S}_{1}\cap\mathcal{S}_{2}.$ (39)

Note that $\mathcal{S}$ is nonempty if $a_{1}\geq a_{2}$ or $b_{1}\geq b_{2}$.
In particular, notice that

$\frac{a_{1}}{a_{2}}=\frac{h_{w}}{2\tau_{0}}\left(\frac{1-\left(1+\frac{1}{\rho}\right)^{2}k_{a}^{2}}{1-\left(1-\frac{1}{\rho}\right)k_{a}}\right).$

Substituting $h_{w}$ from (19), we have

$\frac{a_{1}}{a_{2}}>1.$

Thus, $\mathcal{S}\neq\emptyset$. This completes the proof of Statement (b) of
_Theorem III.2_. In the following we prove Statement (c) of _Theorem III.2_.

Let $\bar{h}_{w}(k_{a}):=\dfrac{h_{w,lb}(k_{a})}{2\tau_{0}}$,
$m:=1-\frac{1}{\rho}$, and $n:=\left(1+\frac{1}{\rho}\right)^{2}$. Then,

$\displaystyle\bar{h}_{w}(k_{a})=\frac{1-mk_{a}}{1-nk_{a}^{2}}$ (40)

and

$\displaystyle\frac{d\bar{h}_{w}(k_{a})}{dk_{a}}=\frac{-mnk_{a}^{2}+2nk_{a}-m}{(1-nk_{a}^{2})^{2}}.$
(41)

Let $f(k_{a}):=-mnk_{a}^{2}+2nk_{a}-m$. Then, the roots of $f(k_{a})=0$ are
given by

$r_{1,2}:=\frac{n\mp\sqrt{n(n-m^{2})}}{mn}.$

Note that $r_{1}<\frac{1}{1+\frac{1}{\rho}}$ and
$r_{2}>\frac{1}{1+\frac{1}{\rho}}$. The function $f(k_{a})$ vs. $k_{a}$ is
provided as an illustration for $\rho=5$.

Figure 4: $f(k_{a})$ vs. $k_{a}$ when $\rho=5$.

Since $k_{a}\in\left(0,\frac{1}{1+\frac{1}{\rho}}\right)$, we need to consider
only $r_{1}$. Hence, we have

$\displaystyle\begin{cases}\dfrac{d\bar{h}_{w}}{dk_{a}}<0,\mbox{when}~{}k_{a}\in(0,r_{1});\\\
\dfrac{d\bar{h}_{w}}{dk_{a}}>0,\mbox{when}~{}k_{a}\in\left(r_{1},\frac{1}{1+\frac{1}{\rho}}\right).\end{cases}$
(42)

Thus,

$\displaystyle\min\limits_{k_{a}\in\left(0,\frac{1}{1+\frac{1}{\rho}}\right)}\bar{h}_{w}(k_{a})=\bar{h}_{w}(r_{1}).$
(43)

Substituting $m$ and $n$ into $r_{1}$, we obtain

$\displaystyle k_{a}^{*}:=r_{1}$
$\displaystyle=\left(\frac{1-\frac{1}{\sqrt{\rho}}}{1+\frac{1}{\sqrt{\rho}}}\right)\left(\frac{1}{1+\frac{1}{\rho}}\right).$
(44)

Further, substituting $r_{1}$ into $\bar{h}_{w}(r_{1})$, we obtain

$\displaystyle\bar{h}_{w}(r_{1})=\frac{\left(1+\frac{1}{\sqrt{\rho}}\right)^{2}}{2\left(1+\frac{1}{\rho}\right)}.$
(45)

Therefore,

$\displaystyle
h_{w,lb}^{*}=2\tau_{0}\bar{h}_{w}(r_{1})=\tau_{0}\frac{\left(1+\frac{1}{\sqrt{\rho}}\right)^{2}}{1+\frac{1}{\rho}}.$
(46)

This completes the proof of Statement (c) of _Theorem III.2_.

Finally, we will prove the lower bound of the time headway given by (21)
satisfies the internal stability condition. For this purpose, note that

$\displaystyle
h_{w}k_{p}\geq\tau_{0}k_{p}\frac{\left(1+\frac{1}{\sqrt{\rho}}\right)^{2}}{\left(1+\frac{1}{\rho}\right)}.$
(47)

Thus, $\gamma=k_{v}+h_{w}k_{p}>h_{w}k_{p}\geq\tau_{0}k_{p}$.

Therefore, this completes the proof of _Theorem III.2_. ∎

###### Remark III.2

If $\rho\to\infty$ (no signal noise case), then $k_{a}\in(0,1)$ and
$h_{w}=\frac{2\tau_{0}}{1+k_{a}}$ as given in in [9].

###### Remark III.3

In (19), if $\rho\to\infty$, then the lower bound of the time headway reduces
to $h_{w}\geq h_{w,lb}(k_{a})=\frac{2\tau_{0}}{1+k_{a}}$ as given in [9]. For
the same $k_{a}$ value, the lower bound of the time headway increases as
$\rho$ decreases in the presence of signal noise. In addition, according to
(18), the upper bound of $k_{a}$ is smaller when compared to the no noise
case, which also factors into the increase of the lower bound of the time
headway in the presence of noise. In addition, in (21), if $\rho\to\infty$,
the minimum lower bound of the time headway reduces to
$h^{\ast}_{w,lb}\to\tau_{0}$, and the corresponding optimal $k_{a}$ given by
(20) becomes $k_{a}^{\ast}\to 1$.

##  IV Numerical Simulations

In this section, we present a numerical example and simulation results to
corroborate the results in Section III. We consider the following numerical
values for the system parameters: $\tau_{0}=0.5$ s, $d=5$ m, $N=12$. In the
numerical simulation, $\tau$ was chosen as $\tau=\tau_{0}$. We assume that the
lead vehicle experiences an external disturbance which causes a perturbation
on its acceleration, denoted as $a_{0}(t)$, as follows:

$\displaystyle a_{0}(t)=\begin{cases}0.5\sin(0.1(t-10)),10~{}{\rm
s}<t<(10+20\pi)~{}{\rm s},\\\ 0,\mbox{otherwise},\end{cases}$ (48)

and under which the performance of the communication and control strategy
considered in Section III will be evaluated in the following.

Assume $\rho=5$, then the upper bound of $k_{a}$ can be computed as
$k_{a}<\frac{1}{1+\frac{1}{\rho}}=0.8333$. Then, by choosing $k_{a}=0.5$ and
substituting $\tau_{0},\rho,k_{a}$ values in (19), $h_{w}\geq 0.9375$ s.
Choosing $h_{w}=0.95$ s, the feasible region of $k_{p}$ and $k_{v}$ is
obtained as shown in Fig. 5, from which we choose $k_{p}=0.009$, $k_{v}=0.63$
for the numerical simulations. With the above chosen values of the time
headway and control gains, the frequency response of
$\left|\tilde{H}(j\omega;\tau_{0})\right|$ is shown in Fig. 6 which
demonstrates string stability of the platoon.

Figure 5: The feasible region of $k_{p}$ and $k_{v}$ when $k_{a}=0.5$,
$h_{w}=0.95$ s.  Figure 6: $|\tilde{H}(j\omega;\tau_{0})|$ when $k_{a}=0.5$,
$k_{p}=0.009$, $k_{v}=0.63$, $h_{w}=0.95$ s.

Suppose we model noise in (7) by choosing $n=16$ discretizations, and the
expectation of the binary random variables $z_{i,j}$ are respectively:
$\gamma_{i,0}=0.8055$, $\gamma_{i,1}=0.5767$, $\gamma_{i,2}=0.1829$,
$\gamma_{i,3}=0.2399$, $\gamma_{i,4}=0.8865$, $\gamma_{i,5}=0.0287$,
$\gamma_{i,6}=0.4899$, $\gamma_{i,7}=0.1679$, $\gamma_{i,8}=0.9787$,
$\gamma_{i,9}=0.7127$, $\gamma_{i,10}=0.5005$, $\gamma_{i,11}=0.4711$,
$\gamma_{i,12}=0.0596$, $\gamma_{i,13}=0.6820$, $\gamma_{i,14}=0.0424$,
$\gamma_{i,15}=0.0714$. The evolutions of the inter-vehicular spacing errors
are shown in Fig. 7.

Figure 7: The inter-vehicular spacing errors of the following vehicles
($k_{a}=0.5$, $k_{p}=0.009$, $k_{v}=0.63$ and $h_{w}=0.95$ s).

In addition, as an example, the noise and the communicated acceleration signal
from vehicle 1 to vehicle 2 are shown in Fig. 8.

Figure 8: The noise and the communicated acceleration signal from vehicle 1 to
vehicle 2 ($k_{a}=0.5$, $k_{p}=0.009$, $k_{v}=0.63$ and $h_{w}=0.95$ s).

For comparison, under the same condition as above except for $h_{w}=0.65$ s,
then the frequency response of $|\tilde{H}(j\omega;\tau_{0})|$ is shown in
Fig. 9 which exhibits string instability. In addition, the inter-vehicular
spacing errors of the following vehicles are shown in Fig. 10.

Figure 9: $|\tilde{H}(j\omega;\tau_{0})|$ when $k_{a}=0.5$, $k_{p}=0.009$,
$k_{v}=0.63$, $h_{w}=0.65$ s.  Figure 10: The inter-vehicular spacing errors
of the following vehicles ($k_{a}=0.5$, $k_{p}=0.009$, $k_{v}=0.63$ and
$h_{w}=0.65$ s).

In addition, we have conducted numerical simulations to evaluate the
performance of the platoon for $k_{a}^{\ast}$ and $h_{w,lb}^{\ast}$ given in
Statement (c) of Theorem III.2. For $\rho=5$, we can obtain
$k_{a}^{\ast}=0.3183$ and $h^{\ast}_{w,lb}=0.8727$. Choosing
$k_{a}=k_{a}^{\ast}$ and $h_{w}=0.88$, the feasible region of $k_{v}$ and
$k_{p}$ is shown in Fig. 11.

Figure 11: The feasible region of $k_{p}$ and $k_{v}$ when
$k_{a}=k_{a}^{\ast}$, $h_{w}=0.88$ s.

Choosing $k_{p}=0.003$, $k_{v}=0.85$ from the feasible region, the evolution
of the inter-vehicular spacing errors is provided in Fig. 12.

Figure 12: The inter-vehicular spacing errors of the following vehicles
($k_{a}=k_{a}^{\ast}$, $k_{p}=0.003$, $k_{v}=0.85$ and $h_{w}=0.88$ s).

Figure 13 provides the comparison of the platoon size ($x_{0}-x_{N}$) for the
two time headways, $h_{w}=0.95$ s and $h_{w}=0.88$ s, indicating higher
throughput as stated in Remark III.1.

Figure 13: The comparison of the length of the platoon between $h_{w}=0.95$
and $h_{w}=0.88\approx h^{\ast}_{w,lb}$.

From the numerical simulation results, it can be seen that this vehicle
platoon is robustly string stable with the synthesized control gains and time
headway according to the analysis and design procedure provided in Section
III.

##  V Conclusion

We have investigated robust string stability of connected and autonomous
vehicle platoons with cooperative adaptive cruise control systems subject to
noisy V2V communication. We have derived conditions on control gains for
predecessor acceleration, relative velocity and spacing errors and a lower
bound for time headway that are dependent on the signal to noise ratio due to
V2V communication of acceleration, while ensuring that the CACC system is
internal and string stable. We have provided a systematic analysis through
which one can select control gains and time headway for a given SNR. We have
also provided an illustrative example and corresponding simulation results to
demonstrate the main results. For future work, we will investigate extension
to noisy communication when information from multiple predecessors is
employed.

## References

  * [1] G. Ma, P. R. Pagilla, and S. Darbha, “The effect of signal noise and latency in communicated acceleration on time headway in vehicle platooning,” in _2023 IEEE 26th International Conference on Intelligent Transportation Systems (ITSC)_. IEEE, 2023, pp. 4532–4537.
  * [2] A. Matin and H. Dia, “Impacts of connected and automated vehicles on road safety and efficiency: A systematic literature review,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 24, no. 3, pp. 2705–2736, 2023.
  * [3] M. Garg and M. Bouroche, “Can connected autonomous vehicles improve mixed traffic safety without compromising efficiency in realistic scenarios?” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 24, no. 6, pp. 6674–6689, 2023.
  * [4] D. Swaroop and J. K. Hedrick, “Constant Spacing Strategies for Platooning in Automated Highway Systems,” _Journal of Dynamic Systems, Measurement, and Control_ , vol. 121, no. 3, pp. 462–470, 09 1999.
  * [5] J. B. Kenney, “Dedicated short-range communications (DSRC) standards in the United States,” _Proceedings of the IEEE_ , vol. 99, no. 7, pp. 1162–1182, 2011.
  * [6] R. Molina-Masegosa and J. Gozalvez, “System level evaluation of lte-v2v mode 4 communications and its distributed scheduling,” in _2017 IEEE 85th Vehicular Technology Conference (VTC Spring)_ , 2017, pp. 1–5.
  * [7] M. H. C. Garcia, A. Molina-Galan, M. Boban, J. Gozalvez, B. Coll-Perales, T. Şahin, and A. Kousaridas, “A tutorial on 5G NR V2X communications,” _IEEE Communications Surveys & Tutorials_, vol. 23, no. 3, pp. 1972–2026, 2021.
  * [8] A. Vinel, L. Lan, and N. Lyamin, “Vehicle-to-vehicle communication in c-acc/platooning scenarios,” _IEEE Communications Magazine_ , vol. 53, no. 8, pp. 192–197, 2015.
  * [9] S. Darbha, S. Konduri, and P. R. Pagilla, “Benefits of V2V communication for autonomous and connected vehicles,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 20, no. 5, pp. 1954–1963, 2018.
  * [10] J. Ploeg, D. P. Shukla, N. van de Wouw, and H. Nijmeijer, “Controller synthesis for string stability of vehicle platoons,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 15, no. 2, pp. 854–865, 2014.
  * [11] G. F. Silva, A. Donaire, A. McFadyen, and J. J. Ford, “String stable integral control design for vehicle platoons with disturbances,” _Automatica_ , vol. 127, p. 109542, 2021.
  * [12] R. Rajamani and C. Zhu, “Semi-autonomous adaptive cruise control systems,” _IEEE Transactions on Vehicular Technology_ , vol. 51, no. 5, pp. 1186–1192, 2002.
  * [13] J. Jiang, A. Astolfi, and T. Parisini, “Robust traffic wave damping via shared control,” _Transportation Research Part C: Emerging Technologies_ , vol. 128, p. 103110, 2021.
  * [14] W. Scholte, P. Zegelaar, and H. Nijmeijer, “A control strategy for merging a single vehicle into a platoon at highway on-ramps,” _Transportation Research Part C: Emerging Technologies_ , vol. 136, p. 103511, 2022.
  * [15] J. Shen, E. K. H. Kammara, and L. Du, “Fully distributed optimization-based cav platooning control under linear vehicle dynamics,” _Transportation Science_ , vol. 56, no. 2, pp. 381–403, 2022.
  * [16] S. Darbha, S. Konduri, and P. R. Pagilla, “Vehicle platooning with constant spacing strategies and multiple vehicle look ahead information,” _IET Intelligent Transport Systems_ , vol. 14, no. 6, pp. 589–600, 2020.
  * [17] Y. Bian, Y. Zheng, W. Ren, S. E. Li, J. Wang, and K. Li, “Reducing time headway for platooning of connected vehicles via v2v communication,” _Transportation Research Part C: Emerging Technologies_ , vol. 102, pp. 87–105, 2019.
  * [18] G. Guo, J. Kang, H. Lei, and D. Li, “Finite-time stabilization of a collection of connected vehicles subject to communication interruptions,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 23, no. 8, pp. 10 627–10 635, 2022.
  * [19] G. J. L. Naus, R. P. A. Vugts, J. Ploeg, M. J. G. van de Molengraft, and M. Steinbuch, “String-stable cacc design and experimental validation: A frequency-domain approach,” _IEEE Transactions on Vehicular Technology_ , vol. 59, no. 9, pp. 4268–4279, 2010.
  * [20] J. I. Ge and G. Orosz, “Connected cruise control among human-driven vehicles: Experiment-based parameter estimation and optimal control design,” _Transportation Research Part C: Emerging Technologies_ , vol. 95, pp. 445–459, 2018.
  * [21] L. Jin, M. Čičić, K. H. Johansson, and S. Amin, “Analysis and design of vehicle platooning operations on mixed-traffic highways,” _IEEE Transactions on Automatic Control_ , vol. 66, no. 10, pp. 4715–4730, 2021.
  * [22] S. Gong and L. Du, “Cooperative platoon control for a mixed traffic flow including human drive vehicles and connected and autonomous vehicles,” _Transportation Research Part B: Methodological_ , vol. 116, pp. 25–61, 2018\.
  * [23] J. Lan, D. Zhao, and D. Tian, “Data-driven robust predictive control for mixed vehicle platoons using noisy measurement,” _IEEE Transactions on Intelligent Transportation Systems_ , pp. 1–11, 2021.
  * [24] J. Ploeg, N. van de Wouw, and H. Nijmeijer, “Lp string stability of cascaded systems: Application to vehicle platooning,” _IEEE Transactions on Control Systems Technology_ , vol. 22, no. 2, pp. 786–793, 2014.
  * [25] B. Besselink and K. H. Johansson, “String stability and a delay-based spacing policy for vehicle platoons subject to disturbances,” _IEEE Transactions on Automatic Control_ , vol. 62, no. 9, pp. 4376–4391, 2017.
  * [26] Z. Li, B. Hu, M. Li, and G. Luo, “String stability analysis for vehicle platooning under unreliable communication links with event-triggered strategy,” _IEEE Transactions on Vehicular Technology_ , vol. 68, no. 3, pp. 2152–2164, 2019.
  * [27] R. A. Biroon, Z. A. Biron, and P. Pisu, “False data injection attack in a platoon of cacc: Real-time detection and isolation with a pde approach,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 23, no. 7, pp. 8692–8703, 2022.
  * [28] V. Vegamoor, S. Rathinam, and S. Darbha, “String stability of connected vehicle platoons under lossy V2V communication,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 23, no. 7, pp. 8834–8845, 2022.
  * [29] M. Hu, X. Wang, Y. Bian, D. Cao, and H. Wang, “Disturbance observer-based cooperative control of vehicle platoons subject to mismatched disturbance,” _IEEE Transactions on Intelligent Vehicles_ , vol. 8, no. 4, pp. 2748–2758, 2023.
  * [30] Y. Zhu, Y. Li, H. Zhu, W. Hua, G. Huang, S. Yu, S. E. Li, and X. Gao, “Joint sliding-mode controller and observer for vehicle platoon subject to disturbance and acceleration failure of neighboring vehicles,” _IEEE Transactions on Intelligent Vehicles_ , vol. 8, no. 3, pp. 2345–2357, 2023.


# t-DGR: A Trajectory-Based
Deep Generative Replay Method for
Continual Learning in Decision Making

William Yue
The University of Texas at Austin
<EMAIL_ADDRESS>
&Bo Liu
The University of Texas at Austin
<EMAIL_ADDRESS>
&Peter Stone
The University of Texas at Austin, Sony AI
<EMAIL_ADDRESS>

###### Abstract

Deep generative replay has emerged as a promising approach for continual
learning in decision-making tasks. This approach addresses the problem of
catastrophic forgetting by leveraging the generation of trajectories from
previously encountered tasks to augment the current dataset. However, existing
deep generative replay methods for continual learning rely on autoregressive
models, which suffer from compounding errors in the generated trajectories. In
this paper, we propose a simple, scalable, and non-autoregressive method for
continual learning in decision-making tasks using a generative model that
generates task samples conditioned on the trajectory timestep. We evaluate our
method on Continual World benchmarks and find that our approach achieves
state-of-the-art performance on the average success rate metric among
continual learning methods. Code is available at
https://github.com/WilliamYue37/t-DGR.

##  1 Introduction

Continual learning, also known as lifelong learning, is a critical challenge
in the advancement of general artificial intelligence, as it enables models to
learn from a continuous stream of data encompassing various tasks, rather than
having access to all data at once [1]. However, a major challenge in continual
learning is the phenomenon of catastrophic forgetting, where previously
learned skills are lost when attempting to learn new tasks [2].

To mitigate catastrophic forgetting, replay methods have been proposed, which
involve saving data from previous tasks and replaying it to the learner during
the learning of future tasks. This approach mimics how humans actively prevent
forgetting by reviewing material for tests and replaying memories in dreams.
However, storing data from previous tasks requires significant storage space
and becomes computationally infeasible as the number of tasks increases.

In the field of cognitive neuroscience, the Complementary Learning Systems
theory offers insights into how the human brain manages memory. This theory
suggests that the brain employs two complementary learning systems: a fast-
learning episodic system and a slow-learning semantic system [3, 4, 5]. The
hippocampus serves as the episodic system, responsible for storing specific
memories of unique events, while the neocortex functions as the semantic
system, extracting general knowledge from episodic memories and organizing it
into abstract representations [6].

Figure 1: The first row presents a comparison of three generative methods for
imitating an agent’s movement in a continuous 2D plane with Gaussian noise.
The objective is to replicate the ground truth path, which transitions from
darker to lighter colors. The autoregressive method (CRIL) encounters a
challenge at the first sharp turn as nearby points move in opposing
directions. Once the autoregressive method deviates off course, it never
recovers and compromises the remaining trajectory. In contrast, sampling
individual state observations i.i.d. without considering the temporal nature
of trajectories (DGR) leads to a fragmented path with numerous gaps. Our
proposed method t-DGR samples individual state observations conditioned on the
trajectory timestep. By doing so, t-DGR successfully avoids the pitfalls of
CRIL and DGR, ensuring a more accurate replication of the desired trajectory.
The second row illustrates how each method generates trajectory data. CRIL
generates the next state observation conditioned on the previous state
observation. DGR, in contrast, does not attempt to generate a trajectory but
generates individual state observations i.i.d. On the other hand, t-DGR
generates state observations conditioned on the trajectory timestep.

Drawing inspiration from the human brain, deep generative replay (DGR)
addresses the catastrophic forgetting issue in decision-making tasks by using
a generative model as the hippocampus to generate trajectories from past tasks
and replay them to the learner which acts as the neocortex (Figure 2) [7]. The
time-series nature of trajectories in decision-making tasks sets it apart from
continual supervised learning, as each timestep of the trajectory requires
sufficient replay. In supervised learning, the learner’s performance is not
significantly affected if it performs poorly on a small subset of the data.
However, in decision-making tasks, poor performance on any part of the
trajectory can severely impact the overall performance. Therefore, it is
crucial to generate state-action pairs that accurately represent the
distribution found in trajectories. Furthermore, the high-dimensional
distribution space of trajectories makes it computationally infeasible to
generate complete trajectories all at once.

Existing DGR methods adopt either the generation of individual state
observations i.i.d. without considering the temporal nature of trajectories or
autoregressive trajectory generation. Autoregressive approaches generate the
next state(s) in a trajectory by modeling the conditional probability of the
next state(s) given the previously generated state(s). However, autoregressive
methods suffer from compounding errors in the generated trajectories. On the
other hand, generating individual state observations i.i.d. leads to a higher
sample complexity compared to generating entire trajectories, which becomes
significant when replay time is limited (see Section 2.2).

To address the issues in current DGR methods, we propose a simple, scalable,
and non-autoregressive trajectory-based DGR method. We define a generated
trajectory as temporally coherent if the transitions from one state to the
next appear realistic (refer to Section 3.4 for a formal definition). Given
that current decision-making methods are trained on state-action pairs, we do
not require trajectories to exhibit temporal coherence. Instead, our focus is
on ensuring an equal number of samples generated at each timestep of the
trajectory to accurately represent the distribution found in trajectories. To
achieve equal sample coverage at each timestep, we train our generator to
produce state observations conditioned on the trajectory timestep, and then
sample from the generator conditioned on each timestep of the trajectory. The
intuition behind our method is illustrated in Figure 1.

To evaluate the effectiveness of our proposed method, t-DGR, we conducted
experiments on the Continual World benchmarks CW10 and CW20 [8] using
imitation learning. Our results indicate that t-DGR achieves state-of-the-art
performance in terms of average success rate when compared to other top
continual learning methods.

##  2 Related Work

This section provides an overview of existing continual learning methods
within the context of “General Continual Learning", with a particular focus on
pseudo-rehearsal methods.

###  2.1 Continual Learning in the Real World

As the field of continual learning continues to grow, there is an increasing
emphasis on developing methods that can be effectively applied in real-world
scenarios [9, 10, 11, 12, 13]. The concept of “General Continual Learning" was
introduced by Buzzega et al. [14] to address certain properties of the real
world that are often overlooked or ignored by existing continual learning
methods. Specifically, two important properties, bounded memory and blurry
task boundaries, are emphasized in this work. Bounded memory refers to the
requirement that the memory footprint of a continual learning method should
remain bounded throughout the entire lifespan of the learning agent. This
property is crucial to ensure practicality and efficiency in real-world
scenarios. Additionally, blurry task boundaries highlight the challenge of
training on tasks that are intertwined, without clear delineation of when one
task ends and another begins. Many existing methods fail to account for this
characteristic, which is common in real-world learning scenarios. While there
are other significant properties associated with continual learning in the
real world, this study focuses on the often-neglected aspects of bounded
memory and blurry task boundaries. By addressing these properties, we aim to
develop methods that are more robust and applicable in practical settings.

###  2.2 Continual Learning Methods

Continual learning methods for decision-making tasks can be categorized into
three main categories.

#### Regularization

Regularization methods in continual learning focus on incorporating
constraints during model training to promote the retention of past knowledge.
One simple approach is to include an $L_{2}$ penalty in the loss function.
Elastic Weight Consolidation (EWC) builds upon this idea by assigning weights
to parameters based on their importance for previous tasks using the Fisher
information matrix [15]. MAS measures the sensitivity of parameter changes on
the model’s output, prioritizing the retention of parameters with a larger
effect [16]. VCL leverages variational inference to minimize the Kullback-
Leibler divergence between the current and prior parameter distributions [17].
Progress and Compress learns new tasks using a separate model and subsequently
distills this knowledge into the main model while safeguarding the previously
acquired knowledge [18]. However, regularization methods may struggle with
blurry task boundaries as they rely on knowledge of task endpoints to apply
regularization techniques effectively. In our experiments, EWC was chosen as
the representative regularization method based on its performance in the
original Continual World experiments [8].

#### Architecture-based Methods

Architecture-based methods aim to maintain distinct sets of parameters for
each task, ensuring that future learning does not interfere with the knowledge
acquired from previous tasks. Packnet [19], UCL [20], and AGS-CL [21] all
safeguard previous task information in a neural network by identifying
important parameters and freeing up less important parameters for future
learning. Identification of important parameters can be done through iterative
pruning (Packnet), parameter uncertainty (UCL), and activation value (AGS-CL).
However, a drawback of parameter isolation methods is that each task requires
its own set of parameters, which may eventually exhaust the available
parameters for new tasks and necessitate a dynamically expanding network
without bounded memory [22]. Additionally, parameter isolation methods require
training on a single task at a time to prune and isolate parameters,
preventing concurrent learning from multiple interwoven tasks. In our
experiments, PackNet was selected as the representative architecture-based
method based on its performance in the original Continual World experiments
[8].

#### Pseudo-rehearsal Methods

Pseudo-rehearsal methods mitigate the forgetting of previous tasks by
generating synthetic samples from past tasks and replaying them to the
learner. Deep generative replay (DGR) (Figure 2) utilizes a generative model,
such as generative adversarial networks [23], variational autoencoders [24],
or diffusion models [25, 26], to generate the synthetic samples. Originally,
deep generative replay was proposed to address continual supervised learning
problems, where the generator only needed to generate single data point
samples [7]. However, in decision-making tasks, expert demonstrations consist
of trajectories (time-series) with a significantly higher-dimensional
distribution space.

One existing DGR method generates individual state observations i.i.d. instead
of entire trajectories. However, this approach leads to a higher sample
complexity compared to generating entire trajectories. The sample complexity
of generating enough individual state observations i.i.d. to cover every
portion of the trajectory $m$ times can be described using the Double Dixie
Cup problem [27]. For trajectories of length $n$, it takes an average of
$\Theta(n\log n+mn\log\log n)$ i.i.d. samples to ensure at least $m$ samples
for each timestep. In scenarios with limited replay time (small $m$) and long
trajectories (large $n$) the sample complexity can be approximated as
$\Theta(n\log n)$ using the Coupon Collector’s problem [28]. The additional
$\Theta(\log n)$ factor reduces the likelihood of achieving complete sample
coverage of the trajectory when the number of replays or replay time is
limited, especially considering the computationally expensive nature of
current generative methods. Furthermore, there is a risk that the generator
assigns different probabilities to each timestep of the trajectory, leading to
a selective focus on certain timesteps rather than equal representation across
the trajectory.

Another existing DGR method is autoregressive trajectory generation. In the
existing autoregressive method, CRIL, a generator is used to generate samples
of the initial state, and a dynamics model predicts the next state based on
the current state and action [29]. However, even with a dynamics model
accuracy of 99% and a 1% probability of deviating from the desired trajectory,
the probability of an autoregressively generated trajectory going off course
is $1-0.99^{n}$, where $n$ denotes the trajectory length. With a trajectory
length of $n=200$ (as used in our experiments), the probability of an
autoregressively generated trajectory going off course is $1-0.99^{200}=0.87$.
This example demonstrates how the issue of compounding error leads to a high
probability of failure, even with a highly accurate dynamics model.

In our experiments, t-DGR is evaluated against all existing trajectory
generation methods in pseudo-rehearsal approaches to assess how well t-DGR
addresses the limitations of those methods.

Figure 2: The deep generative replay paradigm. The algorithm learns to
generate trajectories from past tasks to augment real trajectories from the
current task in order to mitigate catastrophic forgetting. Both the generator
and policy model are updated with this augmented dataset.

##  3 Background

This section introduces notation and the formulation of the continual
imitation learning problem that we use in this paper. Additionally, we provide
a concise overview of diffusion probabilistic models used in our generative
model implementation.

###  3.1 Imitation Learning

Imitation learning algorithms aim to learn a policy $\pi_{\theta}$
parameterized by $\theta$ by imitating a set of expert demonstrations
$D=\\{\tau_{i}\\}_{i=1\ldots M}$. Each trajectory $\tau_{i}$ consists of a
sequence of state-action pairs $\\{(s_{j},a_{j})\\}_{j=1\ldots|\tau_{i}|}$
where $|\tau_{i}|$ is the length of the trajectory. Each trajectory comes from
a task $\mathcal{T}$ which is a Markov decision process that can be
represented as a tuple $\langle S,A,T,\rho_{0}\rangle$ with state space $S$,
action space $A$, transition dynamics $T:S\times A\times S\to[0,1]$, and
initial state distribution $\rho_{0}$. Various algorithms exist for imitation
learning, including behavioral cloning, GAIL [30], and inverse reinforcement
learning [31]. In this work, we use behavioral cloning where the objective can
be formulated as minimizing the loss function:

$\mathcal{L}(\theta)=\mathbb{E}_{s,a\sim
D}\bigg{[}\big{\|}\pi_{\theta}(s)-a\big{\|}^{2}_{2}\bigg{]}$ (1)

where the state and action spaces are continuous.

###  3.2 Continual Imitation Learning

In the basic formulation most common in the field today, continual imitation
learning involves sequentially solving multiple tasks
$\mathcal{T}_{1},\mathcal{T}_{2},\ldots,\mathcal{T}_{N}$. When solving for
task $\mathcal{T}_{i}$, the learner only gets data from task $\mathcal{T}_{i}$
and can not access data for any other task. In a more general scenario,
certain tasks may have overlapping boundaries, allowing the learner to
encounter training data from multiple tasks during certain phases of training.
The learner receives a continuous stream of training data in the form of
trajectories $\tau_{1},\tau_{2},\tau_{3},\ldots$ from the environment, where
each trajectory $\tau$ corresponds to one of the $N$ tasks. However, the
learner can only access a limited contiguous portion of this stream at any
given time.

Let $s_{i}$ be the success rate of task $\mathcal{T}_{i}$ after training on
all $N$ tasks. The continual imitation learning objective is defined as
maximizing the average success rate over all tasks:

$S=\frac{1}{N}\sum_{i=1}^{N}s_{i}$ (2)

The primary issue that arises from the continual learning problem formulation
is the problem of catastrophic forgetting where previously learned skills are
forgotten when training on a new task.

###  3.3 Diffusion Probabilistic Models

Diffusion probabilistic models [25, 26] generate data through a learned
reverse denoising diffusion process $p_{\theta}(x_{t-1}\mid x_{t})$. The
forward diffusion process $q(x_{t}\mid x_{t-1})$ gradually adds Gaussian noise
to an input $x_{0}$ at each time step $t$, ultimately resulting in pure noise
$x_{T}$ at $t=T$. The forward diffusion process is defined as:

$q(x_{t}\mid
x_{t-1})=\mathcal{N}\left(x_{t};\sqrt{1-\beta_{t}}x_{t-1},\beta_{t}\mathbf{I}\right)$
(3)

where $0<\beta_{t}<1$ is defined by a known variance schedule. In our
implementation, we adopted the cosine schedule proposed by Nichol et al. [32].
For a sufficiently large time horizon $T$ and a well-behaved variance
schedule, $x_{T}$ approximates an isotropic Gaussian distribution. If we had
the reverse diffusion process $p(x_{t-1}\mid x_{t})$, we could sample
$x_{T}\sim\mathcal{N}(0,\mathbf{I})$ and obtain a sample from $q(x_{0})$ by
denoising $x_{T}$ with $p(x_{t-1}\mid x_{t})$. However, computing
$p(x_{t-1}\mid x_{t})$ is intractable as it necessitates knowledge of the
distribution of all possible $x_{t}$. Instead, we approximate $p(x_{t-1}\mid
x_{t})$ using a neural network:

$p_{\theta}(x_{t-1}\mid
x_{t})=\mathcal{N}\left(x_{t-1};\mu_{\theta}(x_{t},t),\Sigma(x_{t},t)\right)$
(4)

Since $q$ and $p_{\theta}$ can be viewed as a variational auto-encoder [24],
we can use the variational lower bound to minimize the negative log-likelihood
of the reverse process. We can express $\mu_{\theta}(x_{t},t)$ from Equation 4
as:

$\mu_{\theta}(x_{t},t)=\frac{1}{\sqrt{\alpha_{t}}}\left(x_{t}-\frac{\beta_{t}}{\sqrt{1-\overline{\alpha}_{t}}}\epsilon_{\theta}(x_{t},t)\right)$
(5)

where $\alpha_{t}=1-\beta_{t}$ and
$\overline{\alpha}_{t}=\prod_{s=0}^{t}\alpha_{s}$. The training loss can then
be defined as:

$\mathcal{L}(\theta)=\mathbb{E}_{x_{0},t,\epsilon}\left[\lVert\epsilon-\epsilon_{\theta}(x_{t},t)\rVert^{2}\right]$
(6)

Note that the timesteps $t$ in the diffusion process differ from the
trajectory timesteps $t$. Henceforth, we will refer only to the trajectory
timesteps $t$.

###  3.4 Notation

Deep generative replay involves training two models: a generator $G_{\gamma}$
parameterized by $\gamma$ and a learner $\pi_{\theta}$ parameterized by
$\theta$. We define $G_{\gamma}^{(i)}$ as the generator trained on tasks
$\mathcal{T}_{1}\ldots\mathcal{T}_{i}$ and capable of generating data samples
from tasks $\mathcal{T}_{1}\ldots\mathcal{T}_{i}$. Similarly,
$\pi_{\theta}^{(i)}$ represents the learner trained on tasks
$\mathcal{T}_{1}\ldots\mathcal{T}_{i}$ and able to solve tasks
$\mathcal{T}_{1}\ldots\mathcal{T}_{i}$.

A sequence of state observations $(s_{1},s_{2},\ldots,s_{n-1},s_{n})$ is
temporally coherent if $\forall 1\leq i<n,\exists a\in
A:T(s_{i},a,s_{i+1})>\varepsilon$, where $0<\varepsilon<1$ is a small constant
representing a threshold for negligible probabilities.

##  4 Method

Our proposed method, t-DGR, tackles the challenge of generating long
trajectories by training a generator, denoted as $G_{\gamma}(j)$, which is
conditioned on the trajectory timestep $j$ to generate state observations.
Pseudocode for t-DGR is provided as Algorithm 1. The algorithm begins by
initializing the task index, replay ratio, generator model, learner model, and
learning rates (Line 1). The replay ratio, denoted as $0\leq r<1$, determines
the percentage of training samples seen by the learner that are generated.
Upon receiving training data from the environment, t-DGR calculates the number
of trajectories to generate based on the replay ratio $r$ (Lines 4-5). The
variable $L$ (Line 7) represents the maximum length of trajectories observed
so far.

To generate a trajectory $\tau$ of length $L$, t-DGR iterates over each
timestep $1\leq j\leq L$ (Line 9). At each timestep, t-DGR generates the
$j$-th state observation of the trajectory using the previous generator
$G_{\gamma}^{(t-1)}$ conditioned on timestep $j$ (Line 10), and then labels it
with an action using the previous policy $\pi_{\theta}^{(t-1)}$ (Line 11).
After generating all timesteps in the trajectory $\tau$, t-DGR adds it to the
existing training dataset (Line 14). Note that the generated state
observations within a trajectory do not have temporal coherence, as each state
observation is generated independently of other timesteps. This approach is
acceptable since our learner is trained on state-action pairs rather than full
trajectories. However, unlike generating state observations i.i.d., our method
ensures equal coverage of every timestep during the generative process,
significantly reducing sample complexity.

Once t-DGR has augmented the training samples from the environment with our
generated training samples, t-DGR employs backpropagation to update both the
generator and learner using the augmented dataset (Lines 16-18). The t-DGR
algorithm continues this process of generative replay throughout the agent’s
lifetime, which can be infinite (Line 2). Although we perform the generative
process of t-DGR at task boundaries for ease of understanding, no part of
t-DGR is dependent on clear task boundaries.

#### Architecture

We employ a U-net [33] trained on the loss specified in Equation 6 to
implement the generative diffusion model $G_{\gamma}$. Since we utilize
proprioceptive observations in our experiments, $\pi_{\theta}$ is implemented
with a multi-layer perceptron trained on the loss specified in Equation 1.

Algorithm 1 Trajectory-based Deep Generative Replay (t-DGR)

1:Initialize task index $t=0$, replay ratio $r$, generator $G^{(0)}_{\gamma}$,
learner $\pi^{(0)}_{\theta}$, and learning rates
$\lambda_{\gamma},\lambda_{\theta}$.

2:while new task available do

3: $t\leftarrow t+1$

4: Initialize dataset $D$ with trajectories from task $t$.

5: $n\leftarrow\frac{r*|D|}{1-r}$ $\triangleright$ number of trajectories to
generate

6: for $i=1$ to $n$ do

7: $L\leftarrow$ maximum trajectory length

8: $\tau\leftarrow\emptyset$ $\triangleright$ initialize trajectory of length
$L$

9: for $j=1$ to $L$ do

10: $S\leftarrow G_{\gamma}^{(t-1)}(j)$ $\triangleright$ generate states

11: $A\leftarrow\pi_{\theta}^{(t-1)}(S)$ $\triangleright$ label with actions

12: $\tau_{j}\leftarrow(S,A)$ $\triangleright$ add to trajectory

13: end for

14: $D\leftarrow D\cup\tau$ $\triangleright$ add generated trajectory to $D$

15: end for

16: Update generator and learner using $D$

17:
$\gamma^{(t)}\leftarrow\gamma^{(t-1)}-\lambda_{\gamma}\nabla_{\gamma}\mathcal{L}_{G^{(t-1)}}(\gamma^{(t-1)})$

18:
$\theta^{(t)}\leftarrow\theta^{(t-1)}-\lambda_{\theta}\nabla_{\theta}\mathcal{L}_{\pi^{(t-1)}}(\theta^{(t-1)})$

19:end while

##  5 Experiments

In this section, we outline the experimental setup and performance metrics
employed to compare t-DGR with representative methods, followed by an analysis
of experimental results across different benchmarks and performance metrics.

###  5.1 Experimental Setup

We evaluate our method on the Continual World benchmarks CW10 and CW20 [8],
along with our own “General Continual Learning" variant of CW10 called GCL10.
CW10 consists of a sequence of 10 Meta-World [34] tasks, where each task
involves a Sawyer arm manipulating one or two objects in the Mujoco physics
simulator. For computational efficiency, we provide the agents with
proprioceptive observations. Notably, the observation and action spaces are
continuous and remain consistent across all tasks. CW20 is an extension of
CW10 with the tasks repeated twice. To our knowledge, Continual World is the
only standard continual learning benchmark for decision-making tasks. GCL10
gives data to the learner in 10 sequential buckets $B_{1},\ldots,B_{10}$. Data
from task $\mathcal{T}_{i}$ from CW10 is split evenly between buckets
$B_{i-1}$, $B_{i}$, and $B_{i+1}$, except for the first and last task. Task
$\mathcal{T}_{1}$ is evenly split between buckets $B_{0}$ and $B_{1}$, and
task $\mathcal{T}_{10}$ is evenly split between buckets $B_{9}$ and $B_{10}$.

In order to ensure bounded memory usage, we adopt a one-hot vector approach to
condition the model on the task, rather than maintaining a separate final
neural network layer for each individual task. Additionally, we do not allow
separate biases for each task, as originally done in EWC [15]. Expert
demonstrations for training are acquired by gathering 100 trajectories per
task using hand-designed policies from Meta-World, with each trajectory
limited to a maximum of 200 steps. Importantly, the learner model remains
consistent across different methods and benchmark evaluations. Moreover, we
maintain a consistent replay ratio of $r=0.9$ across all pseudo-rehearsal
methods.

We estimated the success rate $S$ of a model by running each task 100 times.
The metrics for each method were computed using 5 seeds to create a 90%
confidence interval. Further experimental details, such as hyperparameters,
model architecture, random seeds, and computational resources, are included in
the appendix. This standardization enables a fair and comprehensive comparison
of our proposed approach with other existing methods.

###  5.2 Metrics

We evaluate our models using three metrics proposed by the Continual World
benchmark [8], with the average success rate being the primary metric.
Although the forward transfer and forgetting metrics are not well-defined in a
“General Continual Learning" setting, they are informative within the context
of Continual World benchmarks. As a reminder from Section 3.2, let $N$ denote
the number of tasks, and $s_{i}$ represent the success rate of the learner on
task $\mathcal{T}_{i}$. Additionally, let $s_{i}(t)$ denote the success rate
of the learner on task $\mathcal{T}_{i}$ after training on tasks
$\mathcal{T}_{1}$ to $\mathcal{T}_{t}$.

#### Average Success Rate

The average success rate, as given by Equation 2, serves as the primary
evaluation metric for continual learning methods.

#### Average Forward Transfer

We introduce a slightly modified metric for forward transfer that applies to a
broader range of continual learning problems beyond just continual
reinforcement learning in the Continual World benchmark. Let
$s_{i}^{\mathrm{ref}}$ represent the reference performance of a single-task
experiment on task $\mathcal{T}_{i}$. The forward transfer metric $FT_{i}$ is
computed as follows:

$\displaystyle
FT_{i}=\frac{D_{i}-D_{i}^{\mathrm{ref}}}{1-D_{i}^{\mathrm{ref}}}$
$\displaystyle D_{i}=\frac{s_{i}(i)+s_{i}(i-1)}{2}$ $\displaystyle
D_{i}^{\mathrm{ref}}=\frac{s_{i}^{\mathrm{ref}}}{2}$

The expressions for $D_{i}$ and $D^{\mathrm{ref}}_{i}$ serve as approximations
of the integral of task $\mathcal{T}_{i}$ performance with respect to the
training duration for task $\mathcal{T}_{i}$. The average forward transfer
$FT$ is then defined as the mean forward transfer over all tasks, calculated
as $FT=\frac{1}{N}\sum_{i=1}^{N}FT_{i}$.

#### Average Forgetting

We measure forgetting using the metric $F_{i}$, which represents the amount of
forgetting for task $i$ after all training has concluded. $F_{i}$ is defined
as the difference between the success rate on task $\mathcal{T}_{i}$
immediately after training and the success rate on task $\mathcal{T}_{i}$ at
the end of training.

$F_{i}=s_{i}(i)-s_{i}(N)$

The average forgetting $F$ is then computed as the mean forgetting over all
tasks, given by $F=\frac{1}{N}\sum_{i=1}^{N}F_{i}$.

###  5.3 Baselines

We compare the following methods on the Continual World benchmark using
average success rate as the primary evaluation metric. Representative methods
were chosen based on their success in the original Continual World
experiments, while DGR-based methods were selected to evaluate whether t-DGR
addresses the limitations of existing pseudo-rehearsal methods.

  * •

Finetune: The policy is trained only on data from the current task.

  * •

Multitask: The policy is trained on data from all tasks simultaneously.

  * •

oEWC [18]: A variation of EWC known as online Elastic Weight Consolidation
(oEWC) bounds the memory of EWC by employing a single penalty term for the
previous model instead of individual penalty terms for each task. This
baseline is the representative regularization-based method.

  * •

PackNet [19]: This baseline is the representative parameter isolation method.
Packnet safeguards previous task information in a neural network by
iteratively pruning, freezing, and retraining parts of the network.

  * •

DGR [7]: This baseline is a deep generative replay method that only generates
individual state observations i.i.d. and not entire trajectories.

  * •

CRIL [29]: This baseline is a deep generative replay method that trains a
policy along with a start state generator and a dynamics model that predicts
the next state given the current state and action. Trajectories are generated
by using the dynamics model and policy to autoregressively generate next
states from a start state.

  * •

t-DGR: Our proposed method.

Due to the inability of oEWC and PackNet to handle blurry task boundaries, we
made several adjustments for CW20 and GCL10. Since PackNet cannot continue
training parameters for a task once they have been fixed, we treated the
second repetition of tasks in CW20 as distinct from the first iteration,
resulting in PackNet being evaluated with $N=20$, while the other methods were
evaluated with $N=10$. As for GCL10 and its blurry task boundaries, the best
approach we could adopt with oEWC and PackNet was to apply their
regularization techniques at regular training intervals rather than strictly
at task boundaries. During evaluation, all tasks were assessed using the last
fixed set of parameters in the case of PackNet.

(a) CW10 Method | Success Rate $\uparrow$ | FT$\uparrow$ | Forgetting$\downarrow$
---|---|---|---
Finetune | 16.4 $\pm$6.4 | -3.0 $\pm$6.0 | 78.8 $\pm$7.6
Multitask | 97.0 $\pm$1.0 | N/A | N/A
oEWC | 18.6 $\pm$5.3 | -6.3 $\pm$5.7 | 74.1 $\pm$6.1
PackNet | 81.4 $\pm$3.7 | -14.8 $\pm$7.8 |  -0.1 $\pm$1.2
DGR | 75.0 $\pm$5.8 | -4.3 $\pm$5.1 | 17.8 $\pm$4.1
CRIL | 28.4 $\pm$10.6 | -1.1 $\pm$2.8 | 68.6 $\pm$10.4
t-DGR |  81.9 $\pm$3.3 |  -0.3 $\pm$4.9 | 14.4 $\pm$2.5

(b) GCL10 Method | Success Rate $\uparrow$
---|---
Finetune | 21.7 $\pm$2.6
Multitask | 97.0 $\pm$1.0
oEWC | 21.8 $\pm$1.7
PackNet | 26.9 $\pm$5.6
DGR | 75.3 $\pm$4.4
CRIL | 53.5 $\pm$5.5
t-DGR |  81.7 $\pm$4.0

(c) CW20 Method | Success Rate $\uparrow$ | FT$\uparrow$ | Forgetting$\downarrow$
---|---|---|---
Finetune | 14.2 $\pm$4.0 | -0.5 $\pm$3.0 | 82.2 $\pm$5.6
Multitask | 97.0 $\pm$1.0 | N/A | N/A
oEWC | 19.4 $\pm$5.3 | -2.8 $\pm$4.1 | 75.2 $\pm$7.5
PackNet | 74.1 $\pm$4.1 | -20.4 $\pm$3.4 |  -0.2 $\pm$0.9
DGR | 74.1 $\pm$4.1 | 18.9 $\pm$2.9 | 23.3 $\pm$3.3
CRIL | 50.8 $\pm$4.4 | 4.4 $\pm$4.9 | 46.1 $\pm$5.4
t-DGR |  83.9 $\pm$3.0 |  30.6 $\pm$4.5 | 14.6 $\pm$2.9

(d) Replay Ratio Ratio | t-DGR | DGR
---|---|---
0.5 |  63.2 $\pm$2.6 | 52.8 $\pm$2.9
0.6 |  66.3 $\pm$4.4 | 56.9 $\pm$4.5
0.7 |  70.8 $\pm$4.1 | 62.5 $\pm$3.6
0.8 |  75.0 $\pm$6.9 | 69.2 $\pm$4.9
0.9 |  81.9 $\pm$3.3 | 75.0 $\pm$5.8

Table 1: Tables (a), (b), and (c) present the results for Continual World 10,
General Continual Learning 10, and Continual World 20, respectively. The
tables display the average success rate, forward transfer, and forgetting (if
applicable) with 90% confidence intervals using 5 random seeds. An up arrow
indicates that higher values are better and a down arrow indicates that
smaller values are better. Table (d) compares the impact of replay amount on
the average success rate of t-DGR and DGR on CW10 with 90% confidence
intervals obtained using 5 random seeds. The best results are highlighted in
bold.

###  5.4 Discussion

t-DGR emerges as the leading method, demonstrating the highest success rate on
CW10 (Table 1(a)), CW20 (Table 1(c)), and GCL10 (Table 1(b)). Notably,
PackNet’s performance on the second iteration of tasks in CW20 diminishes,
highlighting its limited capacity for continually accommodating new tasks.
This limitation underscores the fact that PackNet falls short of being a true
lifelong learner, as it necessitates prior knowledge of the task count for
appropriate parameter capacity allocation. On the contrary, pseudo-rehearsal
methods, such as t-DGR, exhibit improved performance with the second iteration
of tasks in CW20 due to an increased replay time. These findings emphasize the
ability of DGR methods to effectively leverage past knowledge, as evidenced by
their superior forward transfer in both CW10 and CW20.

GCL10 (Table 1(b)) demonstrates that pseudo-rehearsal methods are mostly
unaffected by blurry task boundaries, whereas PackNet’s success rate
experiences a significant drop-off. This discrepancy arises from the fact that
PackNet’s regularization technique does not work effectively with less clearly
defined task boundaries.

Additionally, the diminishing performance gap between DGR and t-DGR as the
replay ratio increases in Table 1(d) indicates that a higher replay ratio
reduces the likelihood of any portion of the trajectory being insufficiently
covered when sampling individual state observations i.i.d., thereby
contributing to improved performance. This trend supports the theoretical
sample complexity of DGR derived in Section 2.2, as $\Theta(n\log n+mn\log\log
n)$ closely approximates the sample complexity of t-DGR, $\Theta(mn)$, when
the replay amount $m\to\infty$. However, while DGR can achieve comparable
performance to t-DGR with a high replay ratio, the availability of extensive
replay time is often limited in many real-world applications.

Overall, t-DGR exhibits promising results, outperforming other methods in
terms of success rate in all evaluations. Notably, t-DGR achieves a
significant improvement over existing pseudo-rehearsal methods on CW20 using a
Welch t-test with a significance level of $\text{p-value}=0.005$. Its ability
to handle blurry task boundaries, leverage past knowledge, and make the most
of replay opportunities position it as a state-of-the-art method for continual
lifelong learning in decision-making.

##  6 Conclusion

In conclusion, we have introduced t-DGR, a novel method for continual learning
in decision-making tasks, which has demonstrated state-of-the-art performance
on the Continual World benchmarks. Our approach stands out due to its
simplicity, scalability, and non-autoregressive nature, positioning it as a
solid foundation for future research in this domain.

Importantly, t-DGR aligns with the concept of “General Continual Learning" by
taking into account essential properties of the real world, including bounded
memory and blurry task boundaries. These considerations ensure that our method
remains applicable and effective in real-world scenarios, enabling its
potential integration into practical applications.

Looking ahead, one potential avenue for future research is the refinement of
the replay mechanism employed in t-DGR. Rather than assigning equal weight to
all past trajectories, a more selective approach could be explored. By
prioritizing certain memories over others and strategically determining when
to replay memories to the learner, akin to human learning processes, we could
potentially enhance the performance and adaptability of our method.

## Acknowledgements

This work has taken place in the Learning Agents Research Group (LARG) at the
Artificial Intelligence Laboratory, The University of Texas at Austin. LARG
research is supported in part by the National Science Foundation
(FAIN-2019844, NRT-2125858), the Office of Naval Research (N00014-18-2243),
Army Research Office (E2061621), Bosch, Lockheed Martin, and Good Systems, a
research grand challenge at the University of Texas at Austin. The views and
conclusions contained in this document are those of the authors alone. Peter
Stone serves as the Executive Director of Sony AI America and receives
financial compensation for this work. The terms of this arrangement have been
reviewed and approved by the University of Texas at Austin in accordance with
its policy on objectivity in research.

## References

  * [1] Mark Ring.  Continual Learning in Reinforcement Environments.  PhD thesis, University of Texas at Austin, 1994.
  * [2] Michael McCloskey and Neal J. Cohen.  Catastrophic interference in connectionist networks: The sequential learning problem.  volume 24 of Psychology of Learning and Motivation, pages 109–165. Academic Press, 1989.
  * [3] James L. McClelland, Bruce L. McNaughton, and Randall C. O’Reilly.  Complementary learning systems within the hippocampus: A neural network modeling approach to understanding episodic memory consolidation.  Psychological Review, 102(3):419–457, 1995.
  * [4] Dharshan Kumaran, Demis Hassabis, and James L. McClelland.  What learning systems do intelligent agents need? complementary learning systems theory updated.  Trends in Cognitive Sciences, 20(7):512–534, 2016.
  * [5] James Mcclelland, Bruce Mcnaughton, and Randall O’Reilly.  Why there are complementary learning systems in the hippocampus and neocortex: Insights from the successes and failures of connectionist models of learning and memory.  Psychological review, 102:419–57, 08 1995.
  * [6] Randall C. O’Reilly and Kenneth A. Norman.  Hippocampal and neocortical contributions to memory: Advances in the complementary learning systems framework.  Trends in Cognitive Sciences, 6(12):505–510, December 2002.
  * [7] Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim.  Continual learning with deep generative replay, 2017.
  * [8] Maciej Wołczyk, Michał Zając, Razvan Pascanu, Łukasz Kuciński, and Piotr Miłoś.  Continual world: A robotic benchmark for continual reinforcement learning, 2021.
  * [9] Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu.  A comprehensive survey of continual learning: Theory, method and application, 2023.
  * [10] Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio.  Gradient based sample selection for online continual learning.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
  * [11] Jihwan Bang, Heesu Kim, YoungJoon Yoo, Jung-Woo Ha, and Jonghyun Choi.  Rainbow memory: Continual learning with a memory of diverse samples, 2021.
  * [12] Yen-Chang Hsu, Yen-Cheng Liu, Anita Ramasamy, and Zsolt Kira.  Re-evaluating continual learning scenarios: A categorization and case for strong baselines, 2019.
  * [13] Gido M. van de Ven and Andreas S. Tolias.  Three scenarios for continual learning, 2019.
  * [14] Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara.  Dark experience for general continual learning: a strong, simple baseline, 2020.
  * [15] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell.  Overcoming catastrophic forgetting in neural networks.  Proceedings of the National Academy of Sciences, 114(13):3521–3526, mar 2017.
  * [16] Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars.  Memory aware synapses: Learning what (not) to forget, 2018.
  * [17] Cuong V. Nguyen, Yingzhen Li, Thang D. Bui, and Richard E. Turner.  Variational continual learning, 2018.
  * [18] Jonathan Schwarz, Jelena Luketina, Wojciech M. Czarnecki, Agnieszka Grabska-Barwinska, Yee Whye Teh, Razvan Pascanu, and Raia Hadsell.  Progress & compress: A scalable framework for continual learning, 2018.
  * [19] Arun Mallya and Svetlana Lazebnik.  Packnet: Adding multiple tasks to a single network by iterative pruning, 2018.
  * [20] Hongjoon Ahn, Sungmin Cha, Donggyu Lee, and Taesup Moon.  Uncertainty-based continual learning with adaptive regularization, 2019.
  * [21] Sangwon Jung, Hongjoon Ahn, Sungmin Cha, and Taesup Moon.  Continual learning with node-importance based adaptive group sparse regularization, 2021.
  * [22] Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang.  Lifelong learning with dynamically expandable networks, 2018.
  * [23] Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial networks, 2014.
  * [24] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes, 2022.
  * [25] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models, 2020.
  * [26] Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  CoRR, abs/1503.03585, 2015.
  * [27] Donald J. Newman.  The double dixie cup problem.  The American Mathematical Monthly, 67(1):58–61, 1960.
  * [28] Amy N. Myers and Herbert S. Wilf.  Some new aspects of the coupon-collector’s problem, 2003.
  * [29] Chongkai Gao, Haichuan Gao, Shangqi Guo, Tianren Zhang, and Feng Chen.  Cril: Continual robot imitation learning via generative and prediction model, 2021.
  * [30] Jonathan Ho and Stefano Ermon.  Generative adversarial imitation learning, 2016.
  * [31] Andrew Y. Ng and Stuart J. Russell.  Inverse reinforcement learning.  In Proceedings of the 17th International Conference on Machine Learning (ICML-2000), pages 663–670, 2000.
  * [32] Alex Nichol and Prafulla Dhariwal.  Improved denoising diffusion probabilistic models, 2021.
  * [33] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation, 2015.
  * [34] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Avnish Narayan, Hayden Shively, Adithya Bellathur, Karol Hausman, Chelsea Finn, and Sergey Levine.  Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning, 2021.

## Appendix

##  Appendix A Hyperparameters

###  A.1 Finetune

Hyperparameter | Value | Brief Description
---|---|---
batch size | 32 | number of samples in each training iteration
epochs | 250 | number of times the entire dataset is passed through per task
learning rate | $10^{-4}$ | learning rate for gradient descent
optimization algorithm | Adam | optimization algorithm used
$\beta_{1}$ | 0.9 | exponential decay rate for first moment estimates in Adam
$\beta_{2}$ | 0.999 | exponential decay rate for second moment estimates in Adam
epsilon | $10^{-8}$ | small constant for numerical stability
weight decay | 0 | weight regularization

###  A.2 Multitask

Hyperparameter | Value | Brief Description
---|---|---
batch size | 32 | number of samples in each training iteration
epochs | 500 | number of times the entire dataset of all tasks is passed through
learning rate | $10^{-4}$ | learning rate for gradient descent
optimization algorithm | Adam | optimization algorithm used
$\beta_{1}$ | 0.9 | exponential decay rate for first moment estimates in Adam
$\beta_{2}$ | 0.999 | exponential decay rate for second moment estimates in Adam
epsilon | $10^{-8}$ | small constant for numerical stability
weight decay | 0 | weight regularization

###  A.3 oEWC

Hyperparameter | Value | Brief Description
---|---|---
batch size | 32 | number of samples in each training iteration
epochs | 250 | number of times the entire dataset of all tasks is passed through
learning rate | $10^{-4}$ | learning rate for gradient descent
Fisher multiplier | $10^{2}$ | the Fisher is scaled by this number to form the EWC penalty
optimization algorithm | Adam | optimization algorithm used
$\beta_{1}$ | 0.9 | exponential decay rate for first moment estimates in Adam
$\beta_{2}$ | 0.999 | exponential decay rate for second moment estimates in Adam
epsilon | $10^{-8}$ | small constant for numerical stability
weight decay | 0 | weight regularization

The Fisher multiplier hyperparameter was tuned with the values:
$10^{-2},10^{-1},10^{0},10^{1},10^{2},10^{3},10^{4},10^{5},10^{6}$. We
selected the value $10^{2}$ based on the success rate metric given by Equation
2.

###  A.4 PackNet

Hyperparameter | Value | Brief Description
---|---|---
batch size | 32 | number of samples in each training iteration
epochs | 250 | number of times the entire dataset of all tasks is passed through
retrain epochs | 125 | number of training epochs after pruning
learning rate | $10^{-4}$ | learning rate for gradient descent
prune percent | 0.75 | percent of free parameters pruned for future tasks
optimization algorithm | Adam | optimization algorithm used
$\beta_{1}$ | 0.9 | exponential decay rate for first moment estimates in Adam
$\beta_{2}$ | 0.999 | exponential decay rate for second moment estimates in Adam
epsilon | $10^{-8}$ | small constant for numerical stability
weight decay | 0 | weight regularization

The retrain epochs and prune percent hyperparameters were chosen following the
approach in the original PackNet paper. After training the first task, bias
layers are frozen.

###  A.5 DGR

Hyperparameter | Value | Brief Description
---|---|---
batch size | 32 | number of samples in each training iteration
epochs | 250 | number of times the entire dataset of all tasks is passed through
learning rate | $10^{-4}$ | learning rate for gradient descent
diffusion training steps | $10^{4}$ | number of training steps for the diffusion model per task
diffusion warmup steps | $5*10^{4}$ | number of extra training steps for the diffusion model on the first task
diffusion timesteps | $10^{3}$ | number of timesteps in the diffusion process
replay ratio | 0.9 | percentage of training examples that are generated
optimization algorithm | Adam | optimization algorithm used
$\beta_{1}$ | 0.9 | exponential decay rate for first moment estimates in Adam
$\beta_{2}$ | 0.999 | exponential decay rate for second moment estimates in Adam
epsilon | $10^{-8}$ | small constant for numerical stability
weight decay | 0 | weight regularization

###  A.6 CRIL

Hyperparameter | Value | Brief Description
---|---|---
batch size | 32 | number of samples in each training iteration
epochs | 300 | number of times the entire dataset of all tasks is passed through
learning rate | $10^{-4}$ | learning rate for gradient descent
diffusion training steps | $10^{4}$ | number of training steps for the diffusion model per task
diffusion warmup steps | $5*10^{4}$ | number of extra training steps for the diffusion model on the first task
diffusion timesteps | $10^{3}$ | number of timesteps in the diffusion process
replay ratio | 0.9 | percentage of training examples that are generated
optimization algorithm | Adam | optimization algorithm used
$\beta_{1}$ | 0.9 | exponential decay rate for first moment estimates in Adam
$\beta_{2}$ | 0.999 | exponential decay rate for second moment estimates in Adam
epsilon | $10^{-8}$ | small constant for numerical stability
weight decay | 0 | weight regularization

###  A.7 t-DGR

Hyperparameter | Value | Brief Description
---|---|---
batch size | 32 | number of samples in each training iteration
epochs | 300 | number of times the entire dataset of all tasks is passed through
learning rate | $10^{-4}$ | learning rate for gradient descent
diffusion training steps | $10^{4}$ | number of training steps for the diffusion model per task
diffusion warmup steps | $5*10^{4}$ | number of extra training steps for the diffusion model on the first task
diffusion timesteps | $10^{3}$ | number of timesteps in the diffusion process
replay ratio | 0.9 | percentage of training examples that are generated
optimization algorithm | Adam | optimization algorithm used
$\beta_{1}$ | 0.9 | exponential decay rate for first moment estimates in Adam
$\beta_{2}$ | 0.999 | exponential decay rate for second moment estimates in Adam
epsilon | $10^{-8}$ | small constant for numerical stability
weight decay | 0 | weight regularization

##  Appendix B Model Architecture

Layer (type) | Output Shape | Param #
---|---|---
Linear-1 | [32, 512] | 25,600
Linear-2 | [32, 512] | 262,656
Linear-3 | [32, 512] | 262,656
Linear-4 | [32, 512] | 262,656
Linear-5 | [32, 4] | 2,052
Total params: | 815,620
Trainable params: | 815,620
Non-trainable params: | 0
Table 2: Multi-layer perceptron architecture of the learner shared by all
methods

##  Appendix C Experiment Details

We utilized the following random seeds for the experiments: 1, 2, 3, 4, 5. All
experiments were conducted on Nvidia A100 GPUs with 80 GB of memory. The
computational node consisted of an Intel Xeon Gold 6342 2.80GHz CPU with 500
GB of memory. For our longest benchmark, CW20, the runtimes were as follows:
DGR and t-DGR took 3 days, CRIL took 16 hours, finetune and oEWC took 6 hours,
and PackNet took 8 hours.