# Eventual Discounting Temporal Logic Counterfactual Experience Replay

Cameron Voloshin  Abhinav Verma  Yisong Yue

###### Abstract

Linear temporal logic (LTL) offers a simplified way of specifying tasks for
policy optimization that may otherwise be difficult to describe with scalar
reward functions. However, the standard RL framework can be too myopic to find
maximally LTL satisfying policies. This paper makes two contributions. First,
we develop a new value-function based proxy, using a technique we call
eventual discounting, under which one can find policies that satisfy the LTL
specification with highest achievable probability. Second, we develop a new
experience replay method for generating off-policy data from on-policy
rollouts via counterfactual reasoning on different ways of satisfying the LTL
specification. Our experiments, conducted in both discrete and continuous
state-action spaces, confirm the effectiveness of our counterfactual
experience replay approach.

Machine Learning, ICML

##  1 Introduction

In the standard reinforcement learning (RL) framework, the goal is to develop
a strategy that maximizes a reward function in an unknown environment. In many
applications of RL, a practitioner is responsible for generating the reward
function so that the agent will behave desirably after the learning process.
However, it can be challenging to convey real-world task specifications
through scalar rewards (Randløv & Alstrøm, 1998; Toromanoff et al., 2019;
Ibarz et al., 2018; Zhang et al., 2021; Ng et al., 1999). Colloquially known
as reward-shaping, practitioners often resort to using heuristic ”breadcrumbs”
(Sorg, 2011) to guide the agent towards intended behaviors. Despite the
“reward function is enough” hypothesis (Sutton & Barto, 2018; Silver et al.,
2021)), some tasks may not be reducible to scalar rewards (Abel et al., 2021).

In response to these challenges, alternative RL frameworks using Linear
Temporal Logic (LTL) to specify agent behavior have been studied (see Section
7). LTL can express desired characteristics of future paths of a system (Baier
& Katoen, 2008), allowing for precise and flexible task/behavior
specification. For example, we may ask for a system to repeatedly accomplish a
set of goals in specific succession (see Section 2 for more examples).

Without significant assumptions, there is no precise signal on the probability
of a policy satisfying an LTL objective. Existing work overwhelmingly uses
Q-learning with a sparse RL heuristic (Bozkurt et al., 2020; Cai et al.,
2021a) meant to motivate an agent to generate trajectories that appear to
satisfy the task. First, these heuristics involve complicated technical
assumptions obscuring access to non-asymptotic guarantees, even in finite
state-action spaces. Second, sparse reward functions pose substantial
challenges to any gradient-based RL algorithm since they provide poor signal
to adequately solve credit assignment. Resolving sparsity involves using a
hierarchical approach (Bozkurt et al., 2020) or re-introducing reward-shaping
(Hasanbeig et al., 2018). See Section 7 for further elaboration on prior work.

Our contributions. In this paper, we focus on model-free policy learning of an
LTL specified objective from online interaction with the environment. We make
two technical contributions. First, we reformulate the RL problem with a
modified value-function proxy using a technique we call _eventual
discounting_. The key idea is to account for the fact that optimally
satisfying the LTL specification may not depend on the length of time it takes
to satisfy it (e.g., “eventually always reach the goal”). We prove in Section
4 that the optimal policy under eventual discounting maximizes the probability
of satisfying the LTL specification.

Second, we develop an experience replay method to address the reward sparsity
issue. Namely, any LTL formula can be converted to a fully known specialized
finite state automaton from which we can generate multiple counterfactual
trajectories from a single on-policy trajectory. We call this method LTL-
guided counterfactual experience replay. We empirically validate the
performance gains of our counterfactual experience replay approach using both
finite state/action spaces as well as continuous state/action spaces using
both Q-learning and Policy Gradient approaches.

##  2 Preliminaries

We give the necessary background and examples to understand our problem
statement and solution approach. An _atomic proposition_ is a variable that
takes on a truth value. An _alphabet_ over a set of atomic propositions AP is
given by $\Sigma=2^{\text{AP}}$. For example, if $\text{AP}=\\{x,y\\}$ then
$\Sigma=\\{\\{\\},\\{x\\},\\{y\\},\\{x,y\\}\\}$. $\Delta(A)$ represents the
set of probability distributions over a set $A$.

###  2.1 Running Example

Figure 1: Examples. First: Illustration of the Flatworld environment. The
agent is a green dot and there are 3 zones: yellow, blue and red. Second: LDBA
$\mathcal{B}$ for “$FGy$”. $\mathcal{S}^{\mathcal{B}^{\ast}}=\\{1\\}$, denoted
by green circle. The initial state is $b_{-1}=0$. Third: LDBA $\mathcal{B}$
for “$GF(y\;\&\;XFr)\;\&\;G\lnot b$”.
$\mathcal{S}^{\mathcal{B}^{\ast}}=\\{1\\}$, denoted by green circle. The
initial state is $b_{-1}=1$. Fourth: For this example, an agent starting in
state $0$ and solving
$\arg\max_{\pi\in\Pi}\mathbb{E}_{\tau\sim\mathrm{T}_{\pi}^{P}}[\sum_{i=0}^{\infty}\gamma^{i}\mathbf{1}_{\\{b_{i}\in\mathcal{S}^{\mathcal{B}^{\ast}}\\}}]$
where $\mathcal{S}^{\mathcal{B}^{\ast}}$ is illustrated as the green circles
would choose to take action $B$ with probability $1$ if $\alpha\in(1/2,1)$ for
any $\gamma\in[0,1]$. Such a policy has
$\mathbb{P}[\pi\models\varphi]=\alpha<1$. However the probability optimal
policy deterministically takes action $A$, with
$\mathbb{P}[\pi^{\ast}\models\varphi]=1$. This illustrates catastrophic myopic
behavior.

We will be using the environment illustrated in Figure 1 (First) as a running
example. The agent is given by a green dot and there are 3 circular regions in
the environment colored yellow, blue and red. The AP are given by
$\\{y,b,r\\}$, referring to the respective colored zones. Elements of $\Sigma$
indicate which zone(s) the agent is in.

###  2.2 MDPs with Labelled State Spaces

Following a similar notation as Voloshin et al. (2022), we assume that the
environment follows the discounted, labelled Markov Decision Process (MDP)
framework given by the tuple
$\mathcal{M}=(\mathcal{S}^{\mathcal{M}},\mathcal{A}^{\mathcal{M}},P^{\mathcal{M}},d_{0}^{\mathcal{M}},\gamma,L^{\mathcal{M}})$
consisting of a state space $\mathcal{S}^{\mathcal{M}}$, an action space
$\mathcal{A}^{\mathcal{M}}$, an unknown transition function
$P^{\mathcal{M}}:\mathcal{S}^{\mathcal{M}}\times\mathcal{A}^{\mathcal{M}}\to\Delta(\mathcal{S}^{\mathcal{M}})$,
an initial state distribution
$d^{\mathcal{M}}_{0}\in\Delta(\mathcal{S}^{\mathcal{M}})$, and a labelling
function $L^{\mathcal{M}}:\mathcal{S}^{\mathcal{M}}\to\Sigma$. Let
$A^{\mathcal{M}}(s)$ to be the set of available actions in state $s$.

Unlike traditional MDPs, $\mathcal{M}$ has a labeling function
$L^{\mathcal{M}}$ which returns the atomic propositions that are true in that
state. For example, in Figure 1 (First), when the agent enters a state
$s\in\mathcal{S}^{\mathcal{M}}$ such that it is both in the yellow and blue
zone then $L^{\mathcal{M}}(s)=\\{y,b\\}$.

###  2.3 Linear Temporal Logic (LTL)

Here we give a basic introduction to LTL. For a more comprehensive overview,
see Baier & Katoen (2008).

######  Definition 2.1 (LTL Specification, $\varphi$).

An LTL specification $\varphi$ is the entire description of the task,
constructed from a composition of atomic propositions with logical
connectives: not ($\lnot$), and ($\&$), and implies ($\rightarrow$); and
temporal operators: next $(X)$, repeatedly/always/globally ($G$), eventually
($F$), and until ($U$).

Examples. For $AP=\\{x,y\\}$, some basic task specifications include safety
($G\lnot x$), reachability ($Fx$), stability ($FGx$), response ($x\rightarrow
Fy$), and progress $(x\;\&\;XFy)$.

Consider again the environment in Figure 1 (First) where $AP=\\{y,r,b\\}$. If
the task is to eventually reach the yellow zone and stay there (known as
stabilization) then we write $\varphi=FGy$. Or, if we would like the agent to
infinitely loop between the yellow and red zone while avoiding the blue zone
then $\varphi=GF(y\;\&\;XFr)\;\&\;G\lnot b$, a combination of safety,
reachability, and progress.

###  2.4 LTL Satisfaction

LTL has recursive semantics defining the meaning for logical connective
satisfaction. Without loss of generality, we will be using a specialized
automaton, an LDBA $\mathcal{B}_{\varphi}$ (Sickert et al., 2016), defined
below to keep track of the progression of $\varphi$ satisfaction. More details
for constructing LDBAs are in Hahn et al. (2013); Baier & Katoen (2008);
Křetínskỳ et al. (2018). We drop $\varphi$ from $\mathcal{B}_{\varphi}$ for
brevity.

######  Definition 2.2.

(Limit Deterministic Büchi Automaton, LDBA (Sickert et al., 2016)) An LDBA is
a tuple
$\mathcal{B}=(\mathcal{S}^{\mathcal{B}},\Sigma\cup\mathcal{A}_{\mathcal{B}},P^{\mathcal{B}},\mathcal{S}^{\mathcal{B}^{\ast}},b^{\mathcal{B}}_{-1})$
consisting of (i) a finite set of states $\mathcal{S}^{\mathcal{B}}$, (ii) a
finite alphabet $\Sigma=2^{\text{AP}}$, $\mathcal{A}_{\mathcal{B}}$ is a set
of indexed jump transitions (iii) a transition function
$P^{\mathcal{B}}:\mathcal{S}^{\mathcal{B}}\times(\Sigma\cup\mathcal{A}_{\mathcal{B}})\to\mathcal{S}^{\mathcal{B}}$,
(iv) accepting states
$\mathcal{S}^{\mathcal{B}^{\ast}}\subseteq\mathcal{S}^{\mathcal{B}}$, and (v)
initial state $b^{\mathcal{B}}_{-1}$. There exists a mutually exclusive
partitioning of
$\mathcal{S}^{\mathcal{B}}=\mathcal{S}^{\mathcal{B}}_{D}\cup\mathcal{S}^{\mathcal{B}}_{N}$
such that
$\mathcal{S}^{\mathcal{B}^{\ast}}\subseteq\mathcal{S}^{\mathcal{B}}_{D}$, and
for $b\in S^{\mathcal{B}}_{D},a\in\Sigma$ then
$P^{\mathcal{B}}(b,a)\subseteq\mathcal{S}^{\mathcal{B}}_{D}$, closed.
$\mathcal{A}_{\mathcal{B}}(b)$ is only (possibly) non-empty for
$b\in\mathcal{S}^{\mathcal{B}}_{N}$ and allows $\mathcal{B}$ to transition to
$\mathcal{S}^{\mathcal{B}}_{D}$ without reading an AP. A path
$\varrho=(b_{0},b_{1},\ldots)$ is a sequence of states in $\mathcal{B}$
reached through successive transitions under $P^{\mathcal{B}}$.

######  Definition 2.3.

($\mathcal{B}$ accepts) $\mathcal{B}$ accepts a path $\varrho$ if there exists
some state $b\in\mathcal{S}^{\mathcal{B}^{\ast}}$ in the path that is visited
infinitely often.

Examples. Consider again the environment in Figure 1 (First) where
$AP=\\{y,r,b\\}$. If we would like to make an LDBA for $\varphi=FGy$ (reach
and stabilize at $y$) then we would get the state machine seen in Figure 1
(Second). In this state machine, the agent starts at state $0$. The accepting
set is given by $\mathcal{S}^{\mathcal{B}^{\ast}}=\\{1\\}$. The transition
between state $0$ and state $1$ is what is formally referred to as a jump
transition: $\mathcal{A}_{\mathcal{B}}(0)=\\{\epsilon\\}$ while
$\mathcal{A}_{\mathcal{B}}(\cdot)=\varnothing$ otherwise. Whenever the agent
is in state $0$ of the LDBA, there is a choice of whether to stay at state $0$
or transition immediately to state $1$. This choice amounts to the agent
believing that it has satisfied the “eventually” part of the LTL
specification. When the agent takes this jump, then it must thereafter satisfy
$y$ to stay in state $1$. The agent gets the decision of when it believes it
is capable of satisfying $y$ thereafter. When the agent takes the jump, if it
fails to stay in $y$, it immediately transitions to the sink, denoted state
$2$. The LDBA accepts when the state $1$ is reached infinitely often, meaning
the agent satisfies “always $y$” eventually, as desired.

Another example, this time without jump transitions, would be for
$\varphi=GF(y\;\&\;XFr)\;\&\;G\lnot b$ (oscillate between $y$ and $r$ forever
while avoiding $b$). The LDBA can be seen in Figure 1 (Third). In this state
machine, the agent starts at state $1$ and the accepting set is given by
$\mathcal{S}^{\mathcal{B}^{\ast}}=\\{1\\}$. To make a loop back to state $1$,
the agent must visit both $r$ and $y$. Doing so infinitely often satisfies the
LDBA condition and therefore the specification. If at any point $b$ is
encountered then the agent transitions to the sink, denoted state $3$.

##  3 Problem Formulation

We first introduce slightly more notation. Let
$\mathcal{Z}=\mathcal{S}^{\mathcal{M}}\times\mathcal{S}^{\mathcal{B}}$. Let
$\Pi:\mathcal{Z}\times\mathcal{A}\to\Delta([0,1])$ be a (stochastic) policy
class over the product space of the MDP and the LDBA (defined below), where
$\mathcal{A}((s,b))=\mathcal{A}^{\mathcal{M}}(s)\cup\mathcal{A}^{\mathcal{B}}(b)$,
to account for jump transitions in $\mathcal{B}$.

Synchronizing the MDP with the LDBA. For any $(s,b)\in\mathcal{Z}$, a policy
$\pi\in\Pi$ is able to select an action in $\mathcal{A}^{\mathcal{M}}(s)$ or
an action in $\mathcal{A}^{\mathcal{B}}(b)$, if available. We can therefore
generate a trajectory as the sequence
$\tau=(s_{0},b_{0},a_{0},s_{1},b_{0},a_{1},\ldots)$ under a new probabilistic
transition relation given by

$P(s^{\prime},b^{\prime}|s,b,a)=\\\
\begin{cases}P^{\mathcal{M}}(s,a,s^{\prime})&a\in
A^{\mathcal{M}}(s),b^{\prime}\in P^{\mathcal{B}}(b,L(s^{\prime}))\\\ 1,&a\in
A^{\mathcal{B}}(b),b^{\prime}\in P^{\mathcal{B}}(b,a),s=s^{\prime}\\\
0,&\text{otherwise}\end{cases}$ (1)

Let the LDBA projection of $\tau$ be the subsequence
$\tau_{\mathcal{B}}=(b_{0},b_{1},\ldots)$. Elements of $\tau_{\mathcal{B}}$
can be thought of as tracking an agent’s LTL specification satisfaction:

######  Definition 3.1 (Run Satisfaction, $\tau\models\varphi$).

We say a trajectory satisfies $\varphi$ if $\mathcal{B}$ accepts
$\tau_{\mathcal{B}}$, which happens if $\exists b\in\tau_{\mathcal{B}}$
infinitely often with $b\in\mathcal{S}^{\mathcal{B}^{\ast}}$.

Let $\mathrm{T}^{P}_{\pi}=\mathbb{E}_{z\sim
d_{0}^{\mathcal{M}}\times\\{b_{-1}\\}}[\mathrm{T}^{P}_{\pi}(z)]$ be the
distribution over all possible trajectories starting from any initial state
$z\in d_{0}^{\mathcal{M}}\times\\{b_{-1}\\}$ where $\mathrm{T}^{P}_{\pi}(z)$
is the (conditional) distribution over all possible trajectories starting from
$z\in\mathcal{Z}$ generated by $\pi$ under relation $P$ (given in (1)). The
probability of LTL satisfaction results from counting how many of the
trajectories satisfy the LTL specification:

######  Definition 3.2 (State Satisfaction, $z\models\varphi$).

$\mathbb{P}_{\pi}[z\models\varphi]=\mathbb{E}_{\tau\sim\mathrm{T}^{P}_{\pi}(z)}[\mathbf{1}_{\\{\tau\models\varphi\\}}]=\mathbb{E}_{\tau\sim\mathrm{T}^{P}_{\pi}}[\mathbf{1}_{\\{\tau\models\varphi\\}}|z_{0}=z]$

######  Definition 3.3 (Policy Satisfaction, $\pi\models\varphi$).

$\mathbb{P}[\pi\models\varphi]=\mathbb{E}_{\tau\sim\mathrm{T}^{P}_{\pi}}[\mathbf{1}_{\\{\tau\models\varphi\\}}]$
where $\mathbf{1}_{X}$ is the indicator for $X$.

Ideally we would like to find a policy with highest probability of LTL
specification satisfaction: one that generates the most number of LTL-
satisfying runs. Formally,

$\pi^{\ast}\in\arg\max_{\pi\in\Pi}\mathbb{P}[\pi\models\varphi].$ (2)

We note that Eq (2) is the standard starting point for formulating policy
optimization for LTL satisfaction (Yang et al., 2021; Bozkurt et al., 2020;
Cai et al., 2021a; Hasanbeig et al., 2018, 2020; Voloshin et al., 2022).

##  4 RL-Friendly Form: Eventual Discounting

Unfortunately, the maximization problem in Eq (2) is not easily optimized
since we dont have a direct signal on $\mathbb{P}[\pi\models\varphi]$. Without
any additional assumptions (such as structured knowledge of the MDP), any
finite subsequence can only give evidence on whether $\tau\models\varphi$ but
not a concrete proof.

Eventual Discounting. To address the above issue, we develop a modified value-
function based surrogate as follows. Given a trajectory
$\tau=(s_{0},b_{0},a_{0},\ldots)$, we keep track of how often
$b_{i}\in\mathcal{S}^{\mathcal{B}^{\ast}}$ and incentivize an agent to visit
$\mathcal{S}^{\mathcal{B}^{\ast}}$ as many times as possible. In particular,
under eventual discounting, the value function will give the agent a reward of
$1$ when in a state $b_{i}\in\mathcal{S}^{\mathcal{B}^{\ast}}$ and not
discount length of time between visits to $\mathcal{S}^{\mathcal{B}^{\ast}}$.
Formally, we will be seeking

$\pi_{\gamma}^{\ast}\in\arg\max_{\pi\in\Pi}\mathbb{E}_{\tau\sim\mathrm{T}_{\pi}^{P}}[\sum_{i=0}^{\infty}\Gamma_{i}\mathbf{1}_{\\{b_{i}\in\mathcal{S}^{\mathcal{B}^{\ast}}\\}}]\
\ \ (\equiv V_{\pi}^{\gamma}),$ (3)

where $\Gamma_{0}=1$ and

$\Gamma_{i}=\prod_{t=0}^{i-1}\gamma(b_{t}),\quad\gamma(b_{t})=\begin{cases}\gamma,\quad
b_{t}\in\mathcal{S}^{\mathcal{B}^{\ast}}\\\
1,\quad\text{otherwise}\end{cases}.$ (4)

Intuition for $\Gamma_{i}$. At first glance setting $\Gamma_{i}=\gamma^{i}$ to
be the traditional RL exponential discount rate would seem reasonable.
Unfortunately, $\nexists\gamma\in[0,1]$ with $\Gamma_{i}=\gamma^{i}$ that
avoids catastrophic myopic behavior. In particular, take Figure 1 (Fourth).
The agent starts in state $0$ and only has two actions $A$ and $B$. Taking
action $A$ transitions directly to an accepting state from which point the
accepting state is visited every $2$ steps. On the other hand, action $B$
transitions to an accepting state with probability $\alpha$ and a sink state
with probability $1-\alpha$. The accepting state reached by action $B$ is
revisited every step. Suppose $\beta=\pi(A)=1-\pi(B)$ then we can calculate:

$\mathbb{E}_{\tau\sim\mathrm{T}_{\pi}^{P}}[\sum_{i=0}^{\infty}\gamma^{i}\mathbf{1}_{\\{b_{i}\in\mathcal{S}^{\mathcal{B}^{\ast}}\\}}]=\frac{\beta}{1-\gamma^{2}}+\frac{(1-\beta)\alpha}{1-\gamma}.$
(5)

For $\alpha>1/2$, the optimal choice $\beta$ is $\beta=0$ implying that
$P(\pi\models\varphi)=\alpha$. When $\alpha\in(1/2,1)$ then this implies that
$\pi$ is not probability optimal. Indeed, $P(\pi\models\varphi)=\alpha<1$ when
$\beta=0$ but $P(\pi^{\ast}\models\varphi)=1$ by selecting $\beta=1$. The
intuition here, which can be formalized by taking $\gamma\to 1$, is that the
average reward for taking action $A$ is $\frac{1}{2}$ while the average reward
for taking action $B$ is $1$ with probability $\alpha$, which is worth the
risk for large $\alpha>1/2$.

To avoid this myopic behavior, we must avoid discriminating between return
times between good states. The number steps (on average) it takes to return to
$\mathcal{S}^{\mathcal{B}^{\ast}}$ is irrelevant: we only require that the
system does return. For this reason we do not count time (hence $\gamma=1$) in
our definition of $\Gamma_{i}$ when the system is not in
$\mathcal{S}^{\mathcal{B}^{\ast}}$. We call this _eventual discounting_.

###  4.1 Analysis of $\pi^{\ast}_{\gamma}$

In this section we analyze how the probability of $\pi_{\gamma}^{\ast}$
satisfying $\varphi$ compares to that of the best possible one $\pi^{\ast}$.

Let the set $O(\tau)=\\{i:b_{i}\in\mathcal{S}^{\mathcal{B}^{\ast}}\\}$ denote
the occurences (time steps) when a good state is reached. This quantity is
natural since $|O(\tau)|=\infty$ if and only if $\tau\models\varphi$.

######  Lemma 4.1.

For any $\pi\in\Pi$ and $\gamma\in(0,1)$, we have

$|(1-\gamma)V_{\pi}^{\gamma}-\mathbb{P}[\pi\models\varphi]|\leq\log(\frac{1}{\gamma})O_{\pi}$

where
$O_{\pi}=\mathbb{E}_{\tau\sim\mathrm{T}_{\pi}^{P}}\left[|O(\tau)|\bigg{|}\tau\not\models\varphi\right]$
is the expected number of visits to an accepting state for the trajectories
that do not satisfy $\varphi$.

###### Proof.

Fix some state $z=(s,b)\in\mathcal{Z}$.

$\displaystyle V_{\pi}^{\gamma}(z)$
$\displaystyle=\mathbb{E}_{\tau\sim\mathrm{T}_{\pi}^{P}}[\sum_{i=0}^{\infty}\Gamma_{i}\mathbf{1}_{\\{b_{i}\in\mathcal{S}^{\mathcal{B}^{\ast}}\\}}|z_{0}=z]$
$\displaystyle=\mathbb{E}_{\tau\sim\mathrm{T}_{\pi}^{P}}\left[\sum_{j=0}^{|O(\tau)|}\gamma^{j}|z_{0}=z\right]$

Using the fact that $\sum_{j=0}^{k}\gamma^{j}=\frac{1-\gamma^{k}}{1-\gamma}$,
we have

$V_{\pi}^{\gamma}(z)=\mathbb{E}_{\tau\sim\mathrm{T}_{\pi}^{P}}\left[\frac{1-\gamma^{|O(\tau)|}}{1-\gamma}\bigg{|}\begin{subarray}{c}\tau\models\varphi\\\
z_{0}=z\\\ \end{subarray}\right]\mathbb{P}_{\pi}[z\models\varphi]\\\
+\mathbb{E}_{\tau\sim\mathrm{T}_{\pi}^{P}}\left[\frac{1-\gamma^{|O(\tau)|}}{1-\gamma}\bigg{|}\begin{subarray}{c}\tau\not\models\varphi\\\
z_{0}=z\\\ \end{subarray}\right]\mathbb{P}_{\pi}[z\not\models\varphi]$ (6)

Since $|O(\tau)|=\infty$ for any $\tau\models\varphi$,

$\mathbb{E}_{\tau\sim\mathrm{T}_{\pi}^{P}}\left[\frac{1-\gamma^{|O(\tau)|}}{1-\gamma}\bigg{|}\begin{subarray}{c}\tau\models\varphi\\\
z_{0}=z\\\ \end{subarray}\right]=\frac{1}{1-\gamma}$ (7)

together with $\mathbb{P}_{\pi}[z\not\models\varphi]\geq 0$ implies

$V_{\pi}^{\gamma}(z)\geq\frac{1}{1-\gamma}\mathbb{P}_{\pi}[z\models\varphi].$
(8)

Taking the expectation over initial states we have

$V_{\pi}^{\gamma}\geq\frac{1}{1-\gamma}\mathbb{P}[\pi\models\varphi].$ (9)

Now we find an upper bound. Let
$M_{\pi}(t)=\mathbb{E}_{\tau\sim\mathrm{T}_{\pi}^{P}}\left[e^{t|O(\tau)|}\bigg{|}\tau\not\models\varphi\right]$.
Starting again with Eq (6) and using Eq (7), we have

$V_{\pi}^{\gamma}(z)\leq\frac{\mathbb{P}_{\pi}[z\models\varphi]}{1-\gamma}+\frac{1-\mathbb{E}_{\tau\sim\mathrm{T}_{\pi}^{P}}\left[e^{\log(\gamma)|O(\tau)|}\bigg{|}\begin{subarray}{c}\tau\not\models\varphi\\\
z_{0}=z\\\ \end{subarray}\right]}{1-\gamma}$ (10)

where we have used that $\mathbb{P}_{\pi}[z\not\models\varphi]\leq 1$ for any
$z\in\mathcal{Z}$. Taking the expectation with respect to the initial state
distribution then we have

$(1-\gamma)V_{\pi}^{\gamma}\leq\mathbb{P}[\pi\models\varphi]+1-M_{\pi}(\log(\gamma))$
(11)

In particular, $M_{\pi}(t)$ is convex and therefore it lies above its
tangents:

$\displaystyle M_{\pi}(t)\geq M_{\pi}(0)+tM^{\prime}_{\pi}(0)$
$\displaystyle=1+t\mathbb{E}_{\tau\sim\mathrm{T}_{\pi}^{P}}\left[|O(\tau)|\bigg{|}\tau\not\models\varphi\right]$
$\displaystyle=1+tO_{\pi}$

Plugging this inequality into Eq (11), together with Eq (9),

$\mathbb{P}[\pi\models\varphi]\leq(1-\gamma)V_{\pi}^{\gamma}\leq\mathbb{P}[\pi\models\varphi]+\log(\frac{1}{\gamma})O_{\pi}$
(12)

Subtracting $\mathbb{P}[\pi\models\varphi]$ from both sides and taking the
absolute value completes the proof. ∎

######  Theorem 4.2.

(Non-asymptotic guarantee) For any $\gamma\in(0,1)$,

$\sup_{\pi\in\Pi}\mathbb{P}[\pi\models\varphi]-\mathbb{P}[\pi_{\gamma}^{\ast}\models\varphi]\leq
2\log(\frac{1}{\gamma})\sup_{\pi\in\Pi}O_{\pi}$ (13)

where
$O_{\pi}=\mathbb{E}_{\tau\sim\mathrm{T}_{\pi}^{P}}\left[|O(\tau)|\bigg{|}\tau\not\models\varphi\right]$.

###### Proof.

Consider any sequence $\\{\pi_{i}\\}_{i=1}^{\infty}$ such that
$\mathbb{P}[\pi_{i}\models\varphi]\to\sup_{\pi}\mathbb{P}[\pi\models\varphi]$
as $i\to\infty$. Then we have for any $\pi_{i}$,

$\displaystyle\mathbb{P}[\pi_{i}\models\varphi]-\mathbb{P}[\pi_{\gamma}^{\ast}\models\varphi]$
$\displaystyle=\mathbb{P}[\pi_{i}\models\varphi]-(1-\gamma)V_{\pi_{i}}^{\gamma}$
$\displaystyle\quad+(1-\gamma)V_{\pi_{i}}^{\gamma}-(1-\gamma)V_{\pi_{\gamma}^{\ast}}^{\gamma}$
$\displaystyle\quad+(1-\gamma)V_{\pi_{\gamma}^{\ast}}^{\gamma}-\mathbb{P}[\pi_{\gamma}^{\ast}\models\varphi]$
$\displaystyle\stackrel{{\scriptstyle(a)}}{{\leq}}|\mathbb{P}[\pi_{i}\models\varphi]-(1-\gamma)V_{\pi_{i}}^{\gamma}|$
$\displaystyle\quad+|\mathbb{P}[\pi_{\gamma}^{\ast}\models\varphi]-(1-\gamma)V_{\pi_{\gamma}^{\ast}}^{\gamma}|$
$\displaystyle\stackrel{{\scriptstyle(b)}}{{\leq}}\log(\frac{1}{\gamma})(O_{\pi_{i}}+O_{\pi_{\gamma}^{\ast}})$
$\displaystyle\stackrel{{\scriptstyle(c)}}{{\leq}}2\log(\frac{1}{\gamma})\sup_{\pi\in\Pi}O_{\pi}$

where $(a)$ is triangle inequality together with removing the term
$(1-\gamma)V_{\pi_{i}}^{\gamma}-(1-\gamma)V_{\pi_{\gamma}^{\ast}}^{\gamma}$
since it is nonpositive by definition of $\pi_{\gamma}^{\ast}$, $(b)$ is an
application of Lemma 4.1, and $(c)$ is a supremum over all policies. Taking
the limit on both sides as $i\to\infty$ completes the proof. ∎

######  Corollary 4.3.

If the number of policies in $\Pi$ is finite then
$\sup_{\pi\in\Pi}O_{\pi}=m<\infty$ is attained, is a finite constant and

$\sup_{\pi\in\Pi}\mathbb{P}[\pi\models\varphi]-\mathbb{P}[\pi_{\gamma}^{\ast}\models\varphi]\leq
2m\log(\frac{1}{\gamma})$

######  Corollary 4.4.

In the case that $\mathcal{S}^{\mathcal{M}}$ and $\mathcal{A}^{\mathcal{M}}$
are finite, then $\mathcal{Z}$ and $\mathcal{A}$ are finite. It is known that
optimal policies are deterministic (Puterman, 2014) and therefore there we
need only consider deterministic policies, for which there are a finite
number. Thus $\sup_{\pi\in\Pi}O_{\pi}=m<\infty$ is attained, is a finite
constant and

$\sup_{\pi\in\Pi}\mathbb{P}[\pi\models\varphi]-\mathbb{P}[\pi_{\gamma}^{\ast}\models\varphi]\leq
2m\log(\frac{1}{\gamma})$

###  4.2 Interpretation

Theorem 4.2 relies on the quantity $\sup_{\pi\in\Pi}O_{\pi}$ to be finite for
the bound to have meaning. In fact, we need only make requirements on
$M_{\pi}(\log(\gamma))$ but the requirements are more easily understood on
$O_{\pi}$. As an aside, $M_{\pi}(\log(\gamma))$ can be interpreted as the
moment generating function of the random variable which is the number of
visits to $\mathcal{S}^{\mathcal{B}^{\ast}}$. Instead we consider the equally
natural quantity $O_{\pi}$. $O_{\pi}$ is the (average) number of times that a
good state is visited by a trajectory that does not satisfy the specification.
Ideally, this number would be small and it would be easy to discriminate
against good and bad policies.

The bad news. In the case that $\Pi$ is an infinite class, conditions for
ensuring $\sup_{\pi\in\Pi}O_{\pi}$ is finite is nontrivial and is dependent on
the landscape of the transition function $P$ of the MDP and $\Pi$.

Let us suppose $\sup_{\pi\in\Pi}O_{\pi}$ is infinite. This means there are
policies that induce bad trajectories that eventually fail to reach
$\mathcal{S}^{\mathcal{B}^{\ast}}$, but along the way visited
$\mathcal{S}^{\mathcal{B}^{\ast}}$ an arbitrarily large (but finite) number of
times. In other words, they are policies that are indistinguishable from
actual probability-optimal policies until the heat death of the universe.

Consider the specification in Figure 1 (right), given by infinitely often
cycle between red and yellow while avoiding blue. A good-looking bad policy is
one that accomplishes the task frequently but, amongst the times that it
fails, it would cycle between red and yellow many times before failing.
$\sup_{\pi\in\Pi}O_{\pi}$ being infinite means that there are policies that
will cycle arbitrarily many times before failing.

The good news. Corollary 4.4 reveals that discretization suffices to generate
probability optimal policies, with suboptimality shrinking at a rate of
$\log(\frac{1}{\gamma})$. This suggests that compactness of $P$ and $\Pi$ and
continuity of $P$ may very well be enough but we leave these conditions for
future work. Finally, since all computers deal with finite precision, the
number of policies is finite and therefore Corollary 4.3 similarly applies.

##  5 LTL Counterfactual Experience Replay

One can optimize the formulation in Eq (3) using any Q-learning or policy
gradient approach, as seen in Algorithm 1 (Line 4). However, doing so is
challenging since it suffers from reward sparsity: the agent only receives a
signal if it reaches a good state.

Algorithm 1 Learning with LCER

0: Maximum horizon $T$. Replay buffer $D=\\{\\}$.

1: for $k=1,2,\ldots$ do

2: Run $\pi_{k-1}$ in the MDP for $T$ timesteps and collect
$\tau=(s_{0},b_{0},a_{0},\ldots,s_{T-1},b_{T-1},a_{T-1},s_{T},b_{T})$

3: $D_{k}\leftarrow\texttt{LCER}(D_{k-1},\tau)$

4: $\pi_{k}\leftarrow\texttt{Update}(\pi_{k-1},D_{k})$ $//$ Q-learn/Policy
grad.

5: end for

We combat reward sparsity by exploiting the LDBA: $P^{\mathcal{\mathcal{B}}}$
is completely known. By knowing $P^{\mathcal{\mathcal{B}}}$, we can generate
multiple off-policy trajectories from a single on-policy trajectory by
modifying which stats in the LDBA we start in, which notably does not require
any access to the MDP transition function $P^{\mathcal{M}}$. We call this
approach _LTL-guided Counterfactual Experience Replay_ , LCER (Algorithm 1,
Line 3), as it is a modification of standard experience replay (Lin, 1992;
Mnih et al., 2013, 2015) to include counterfactual experiences elsewhere in
the LDBA. LCER is most simply understood through Q-learning, and needs careful
modification for policy gradient methods.

Q-learning with LCER. See Algorithm 2 for a synopsis of LCER for Q-learning.
Regardless of whatever state $s\in\mathcal{S}^{\mathcal{M}}$ the agent is in,
we can pretend that the agent is in any $b\in\mathcal{S}^{\mathcal{B}}$. Then
for any action the agent takes we can store experience tuples:

$\\{(s,b,a,r,s^{\prime},\tilde{b^{\prime}})\;|\;\;\forall
b\in\mathcal{S}^{\mathcal{B}}\\}$ (14)

where $\tilde{b^{\prime}}=P^{\mathcal{B}}(b,L^{\mathcal{M}}(s^{\prime}))$ is
the transition that would have occurred from observing labelled state
$L(s^{\prime})$ in state $(s,b)$ and $r=\mathbf{1}_{\tilde{b}^{\prime}\in
B^{\ast}}$. Furthermore we can add all jump transitions:

$\\{(s,b,\epsilon,r,s,\tilde{b^{\prime}})\;|\;\;\forall
b\in\mathcal{S}^{\mathcal{B}},\forall\epsilon\in\mathcal{A}^{\mathcal{B}}(b)\\}$
(15)

since jumps also do not affect the MDP. Notice when we add the jumps that
$s^{\prime}=s$, since only the LDBA state shifts in a jump.

Algorithm 2 LCER for Q-learning

0: Dataset $D$. Trajectory $\tau$ of length $T$.

1: for $(s_{t},a_{t},s_{t+1})\in\tau$ do

2: for $b\in\mathcal{S}^{\mathcal{B}}$ do

3: Set $\tilde{b}\leftarrow P^{\mathcal{B}}(b,L^{\mathcal{M}}(s_{t+1}))$

4: $D\leftarrow D\cup(s_{t},b,a_{t},\mathbf{1}_{\tilde{b}\in
B^{\ast}},s_{t+1},\tilde{b})$

5: for $\epsilon\in\mathcal{A}^{\mathcal{B}}(s)$ do

6: Set $\tilde{b}\leftarrow P^{\mathcal{B}}(b,\epsilon)$

7: $D\leftarrow D\cup(s_{t},b,\epsilon,\mathbf{1}_{\tilde{b}\in
B^{\ast}},s_{t},\tilde{b})$

8: end for

9: end for

10: end for

11: return $D$

Policy Gradient with LCER. See Algorithm 3 for a summary of LCER for policy
gradient. For policy gradient, unlike Q-learning, it is necessary to calculate
future reward-to-go:
$R_{k}(\tau)=\sum_{i=k}^{T}\Gamma_{i}\mathbf{1}_{\\{b_{i}\in\mathcal{S}^{\mathcal{B}^{\ast}}\\}}$.
Thus, we have to generate entire trajectories that are consistent with
$P^{\mathcal{B}}$ rather than independent transition tuples as in Eq (14). We
will show how to generate all feasible trajectories.

Consider a trajectory $\tau=(s_{0},b_{0},a_{0},\ldots,s_{T},b_{T})$ was
collected. Let us remove jump transitions $(s_{i},b_{i},a_{i})$ where
$a_{i}\in\mathcal{A}^{\mathcal{B}}(b_{i})$ and consider the projection of the
trajectory to the MDP $\tau_{\mathcal{M}}=(s_{0},s_{1},\ldots,s_{T})$. We
should only have control over the initial LDBA state $b_{0}$ as all other
automaton states $(b_{1},\ldots,b_{T})$ in a trajectory sequence are
determined by $\tau_{\mathcal{M}}$ and
$b_{i+1}=P^{\mathcal{B}}(b_{i},L^{\mathcal{M}}(s_{i}))$.

Therefore we add

$\tilde{\mathcal{T}}(\tau)=\\{(s_{0},\tilde{b}_{0},a_{0},\ldots,s_{T},\tilde{b}_{T})\;|\;\;\\\
\forall\tilde{b}_{0}\in\mathcal{S}^{\mathcal{B}},\tilde{b}_{i}=P^{\mathcal{B}}(\tilde{b}_{i-1},L^{\mathcal{M}}(s_{i}))\\}$
(16)

where only the LDBA states are different between the trajectories.

Now we handle jump transitions. Consider some
$\tilde{\tau}\in\tilde{\mathcal{T}}(\tau)$. Recall, a jump transition can
occur whenever $\mathcal{A}^{\mathcal{B}}(\tilde{b}_{i})$ is non-empty. This
involves adding a trajectory that is identical to $\tilde{\tau}$ all the way
until the jump occurs. The jump occurs and then the same action sequence and
MDP state sequence follows but with different LDBA states. Specifically,
suppose $\tilde{b_{i}}$ had an available jump transitions,
$\epsilon\in\mathcal{A}^{\mathcal{B}}(\tilde{b}_{i})$. Then:

$\tilde{\tau}_{i,\epsilon}=(s_{0},\tilde{b}^{\prime}_{i},a_{0},\ldots,s_{i},\tilde{b}^{\prime}_{i},\epsilon,s_{i},\tilde{b}^{\prime}_{i+1},a_{i},\ldots,s_{T},\tilde{b}^{\prime}_{T})$
(17)

where $\tilde{b}^{\prime}_{k}=\tilde{b}_{i}$ for $k\leq i$ and
$\tilde{b}^{\prime}_{k}=P^{\mathcal{B}}(\tilde{b}^{\prime}_{k-1},L^{\mathcal{M}}(s_{k}))$
otherwise.

We have to add all possible $\tilde{\tau}^{\prime}_{i,\epsilon}$ that exist.
Let $\mathcal{E}$ be the operator that adds jumps to existing sequences:

$\mathcal{E}(\tilde{\mathcal{T}}(\tau))=\tilde{\mathcal{T}}(\tau)\cup\\{\tilde{\tau}_{i,\epsilon}\text{
from Eq }\eqref{eq:jumps}|\\\
\forall\tilde{\tau}\in\tilde{\mathcal{T}}(\tau),\exists
b_{i}\in\tilde{\tau}\text{ s.t.
}\exists\epsilon\in\mathcal{A}^{\mathcal{B}}(b_{i})\\}.$ (18)

We can only apply
$\mathcal{E}(\mathcal{E}(\ldots(\mathcal{E}(\tilde{\mathcal{T}}(\tau)))))$ at
most $T$ times since the original length of $\tau$ is $T$.

Algorithm 3 LCER for Policy Gradient

0: Dataset $D$. Trajectory $\tau$ of length $T$.

1: Set $\mathcal{\tilde{T}}_{0}\leftarrow\mathcal{\tilde{T}}(\tau)$

2: for $k=1,\ldots,T-1$ do

3: $\mathcal{\tilde{T}}_{k}\leftarrow\mathcal{E}(\mathcal{\tilde{T}}_{k-1})$

4: if $\mathcal{\tilde{T}}_{k}==\mathcal{\tilde{T}}_{k-1}$ then

5: Set $\mathcal{\tilde{T}}_{T-1}\leftarrow\mathcal{\tilde{T}}_{k}$

6: break

7: end if

8: end for

9: Set $D\leftarrow D\cup\mathcal{\tilde{T}}_{T-1}$

10: return $D$

######  Remark 5.1.

The length of $\tau$ has to be sufficiently large to make sure the LDBA has
opportunity to reach $\mathcal{S}^{\mathcal{B}^{\ast}}$. A sufficient
condition is $T\geq|\\{b|\mathcal{A}^{\mathcal{B}}(b)\neq\varnothing\\}|$, the
number of LDBA states with jump transitions.

It is possible to constructively generate feasible trajectories during the
rollout of a policy rather than after-the-fact, see Appendix B.

Figure 2: Results. Each column is an environment and a LTL formula we’d like
an agent to satisfy. The environment and a trajectory from the final policy is
illustrated in the center of the column (except for Pacman, which is the
initial state). The learning curves at the bottom of each column show that
adding off-policy data using LCER has strong benefits for empirical
performance. First Column: Minecraft, where an agent should visit the yellow
and blue areas while avoiding the red. The final policy is illustrated via
blue dots. Second Column: Pacman, where an agent should collect the food while
avoiding a ghost. Third Column: Flatword, where an agent should eventually
stabilize in the yellow region. When the actions are discrete we use
Q-learning, when the actions are continuous we use PPO. Fourth Column: Same as
the third column except an agent should oscillate between the yellow and red
regions while avoiding the blue. Fifth Column: Carlo, where an agent should
drive in a circle without crashing by visiting the blue regions labelled $1$
and $2$ infinitely often.

##  6 Experiments

We perform experiments in four domains with varying LTL formulas, state
spaces, action spaces, and environment stochasticity summarized in the
following section. Our aim is to answer the following two questions: (1) Can
we achieve policies that behave the way we expect an LTL-satisfying policy to
behave? (2) How does LCER impact the performance of learning.

###  6.1 Environment Details

Minecraft The Minecraft environment is a $10\times 10$ deterministic gridworld
with 5 available actions: left, right, up, down, nothing. The agent, given by
a red triangle starts in the cell $(9,2)$. The environment, as well as the
final behavior of the agent (given by blue dots) can be seen in Figure 2
(First).

Pacman The Pacman environment is a $5\times 8$ deterministic gridworld with 5
available actions: left, right, up, down, nothing. The agent, given by a red
triangle starts in the cell $(0,3)$. The ghost chases the agent with
probability $0.8$ and takes a random action with probability $0.2$, for this
reason the environment is stochastic. The starting position of the environment
can be seen in Figure 2 (Second).

Flatworld The Flatworld environment (seen in Figure 2 Third and Fourth) is a
two dimensional continuous world. The agent (given by a green dot) starts at
$(-1,-1)$. The dynamics of the world are given by $x^{\prime}=x+a/10$ where
both $x\in\mathbb{R}^{2}$ and $a\in[0,1]^{2}$. We also allow the action space
to be discrete by letting there be $5$ actions (right, up, left, down,
nothing) where the agent takes a full-throttle action in each respective
direction.

Carlo The Carlo environment (seen in Figure 2 Fifth)is a simplified self-
driving simulator that uses a bicycle model for the dynamics. The agent
observes its position, velocity, and heading in radians for a total of $5$
dimensions. The agent has control over its heading and throttle, for an action
space of $[-1,1]^{2}$. For this domain, we have chosen to use a circular track
where the agent starts in the center of the road at an angle of
$\\{\pi(1+2i)/4\\}_{i=0}^{3}$ and drive counterclockwise around in a circle
without crashing.

###  6.2 Methods and Baseline

When the action space is discrete, we use Q-learning with LCER otherwise we
use PPO with LCER. The baseline we compare against is the same method without
LCER. This allows us to verify the extent to which LCER impacts performance.
We also plot a trajectory from the final policy for each environment in the
middle of each column of Figure 2, except for Pacman as it is difficult to
visualize the interaction between the ghost and pacman outside of video.

Dealing with $\mathcal{A}$. For PPO, the agent’s policy is a Gaussian (as in
standard implementations) over the continuous action space. In order to deal
with jump transitions (in the LDBA) when in a continuous action space (in the
MDP), we first let the agent decide whether to execute a jump transition or
not (ie. a probabilistic coin flip). If the agent chooses to not, then we take
the action according to the Gaussian. The coin flip probability is learned, as
well as the Gaussian. For the importance sampling term of PPO, the density of
$\pi$ is modified to account for the coin flip. For more details see Appendix
A.

###  6.3 Results

Can we achieve desired behavior? The answer here is a resounding yes. For each
environment (except Pacman) we illustrate the trajectory of the final policy
above each learning curve in Figure 2. Determining the probability of
satisfaction of the final policy is currently a challenging open problem
(except in finite-state action spaces). Nevertheless, in each environment the
agent qualitatively accomplishes the task. Even for challenging tasks with
continuous action spaces, the agent is able to learn to accomplish the LTL
specification.

Does LCER help in the learning process? According to the learning curves in
the last row of Figure 2, LCER demonstrably expedites learning. In every
environment with the exception of Carlo, LCER generates significant lift over
lack of experience replay.

Intuition for why LCER helps? One way of viewing an LDBA is as a curriculum
for what steps need to be taken in order to accomplish a task. By replacing
the LDBA state of the agent with some other dream LDBA state, we are allowing
the agent to “pretend” that it has already accomplished some portion of the
task.

As an example, consider the Flatworld example in Figure 2 with
$\varphi=GF(y\;\&\;XF(r))\;\&\;(G\lnot b)$. A baseline agent (without LCER)
would need to accomplish the entirety of the task in order to see any reward.
However, an agent with counterfacual data, need only visit $y$ from state $0$
of the LDBA (see figure 1 for the LDBA). Then once the agent is really good at
getting to $y$, it needs to learn how to reach $r$ from state $2$. After both
of these tasks are accomplished, independently, the agent has solved the whole
task. By placing the agent in state $0$ of the LDBA, we are effectively
letting the agent pretend that it has already visited $r$. In this sense, part
of the task has been accomplished.

##  7 Related Work

Finding LTL-satisfying policies. Among the attempts at finding LTL-satisfying
policies, Q-learning approaches have been the primary method of choice when
the dynamics are unknown and Linear Programming methods when the dynamics are
known (Sadigh et al., 2014; Hasanbeig et al., 2018; Bozkurt et al., 2020; Cai
et al., 2021b; Ding et al., 2014). The Q-learning approaches are predominantly
constrained to finite state-action spaces. Among the works that extend to
continuous action spaces (Hasanbeig et al., 2020), DDPG is used and takes the
form of hierarchical RL which is known to potentially find myopic policies
(Toro Icarte et al., 2022).

Handling a subset of LTL specifications involving those expressible as finite
expressions can also be addressed with Reward machines (Toro Icarte et al.,
2022; Camacho et al., 2019; Vaezipoor et al., 2021). Our work handles
$\omega$-regular expressions, subsuming regular expressions. Many problems are
$\omega$-regular problems, but not regular, such as liveness (something good
will happen eventually) and safety (nothing bad will happen forever).

On the formulation in Eq (3). Notable prior work on defining the value
function as a function of the number of visits to $\mathcal{S}^{\mathcal{B}}$
and a state-dependent $\Gamma_{i}$ function include Bozkurt et al. (2020); Cai
et al. (2021a). Most notably, these authors use multiple different state-
dependent discount rates that have a complicated relationships between them
that needs to be satisfied in the limit as $\gamma\to 1^{-}$. Our work
drastically simplifies this, getting rid of the technical assumptions, while
strengthening the guarantees. This allows us to find a non-asymptotic
dependence on the suboptimality of a policies’ probability of LTL satisfaction
as a function of $\gamma$.

Off-policy data. One may view the counterfactual samples in Toro Icarte et al.
(2022) as an instantiation of LCER, limited to finite LTL expressions and
discrete action spaces. Extension to continuous action space and full LTL
requires a careful treatment. In the continuous action and full LTL setting,
(Wang et al., 2020) incorporate starting the agent from a different initial
LDBA state (than $b_{-1}$) which is still on-policy but from a different
starting state and doesn’t take advantage of the entire LDBA structure. This
work can be seen as complimentary to our own.

Theory. Works with strong theoretical guarantees on policy satisfaction
include Fu & Topcu (2014); Wolff et al. (2012); Voloshin et al. (2022) but are
once again limited to discrete state/action spaces. Extensions of these work
to continuous state space is not trivial as they make heavy use of the
discrete Markov chain structure afforded to them.

##  8 Discussion

Our work, to the best of our knowledge, is the first to make full use of the
LDBA as a form of experience replay and first to use policy gradient to learn
LTL-satisfying policies. Our eventual discounting formulation is unrestricted
to Finitary fragments of LTL like most prior work.

Despite the guarantees afforded to us by eventual discounting, in general the
problem given in Eq (2) is not PAC learnable (Yang et al., 2021). Though, like
SAT solvers, it is still useful to find reasonable heuristics to problems that
are difficult. We show that under particular circumstances, eventual
discounting gives a signal on the quantity of interest in (2) and even when it
fails, it selects a policy that is difficult to differentiate from a
successful one. Further, the bad news discussed in Section 4.2 we speculate is
unavoidable in general LTL specifications, without significant assumptions on
the MDP. For example, for stability problems in LTL and assuming control-
affine dynamics then Lyapunov functions can serve as certificates for a
policies’ LTL satisfaction. A reasonable relaxation to this would be require a
system to behave a certain way for a long, but finite amount of time.

## References

  * Abel et al. (2021) Abel, D., Dabney, W., Harutyunyan, A., Ho, M. K., Littman, M., Precup, D., and Singh, S.  On the expressivity of markov reward.  In _Advances in Neural Information Processing Systems_ , 2021.  URL https://proceedings.neurips.cc/paper/2021/file/4079016d940210b4ae9ae7d41c4a2065-Paper.pdf.
  * Baier & Katoen (2008) Baier, C. and Katoen, J.-P.  _Principles of model checking_.  The MIT Press, Cambridge, Mass, 2008.  ISBN 978-0-262-02649-9.
  * Bozkurt et al. (2020) Bozkurt, A. K., Wang, Y., Zavlanos, M. M., and Pajic, M.  Control synthesis from linear temporal logic specifications using model-free reinforcement learning.  In _2020 IEEE International Conference on Robotics and Automation (ICRA)_ , pp. 10349–10355, 2020.  doi: 10.1109/ICRA40945.2020.9196796.
  * Cai et al. (2021a) Cai, M., Hasanbeig, M., Xiao, S., Abate, A., and Kan, Z.  Modular deep reinforcement learning for continuous motion planning with temporal logic.  _IEEE Robotics and Automation Letters_ , 6(4):7973–7980, 2021a.
  * Cai et al. (2021b) Cai, M., Xiao, S., Li, Z., and Kan, Z.  Optimal probabilistic motion planning with potential infeasible ltl constraints.  _IEEE Transactions on Automatic Control_ , pp. 1–1, 2021b.  doi: 10.1109/TAC.2021.3138704.
  * Camacho et al. (2019) Camacho, A., Toro Icarte, R., Klassen, T. Q., Valenzano, R., and McIlraith, S. A.  Ltl and beyond: Formal languages for reward function specification in reinforcement learning.  In _Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19_ , pp. 6065–6073. International Joint Conferences on Artificial Intelligence Organization, 7 2019\.  doi: 10.24963/ijcai.2019/840.  URL https://doi.org/10.24963/ijcai.2019/840.
  * Ding et al. (2014) Ding, X., Smith, S. L., Belta, C., and Rus, D.  Optimal control of markov decision processes with linear temporal logic constraints.  _IEEE Transactions on Automatic Control_ , 59(5):1244–1257, May 2014.  ISSN 0018-9286, 1558-2523.  doi: 10.1109/TAC.2014.2298143.
  * Fu & Topcu (2014) Fu, J. and Topcu, U.  Probably approximately correct MDP learning and control with temporal logic constraints.  In Fox, D., Kavraki, L. E., and Kurniawati, H. (eds.), _Robotics: Science and Systems X, University of California, Berkeley, USA, July 12-16, 2014_ , 2014.  doi: 10.15607/RSS.2014.X.039.  URL http://www.roboticsproceedings.org/rss10/p39.html.
  * Hahn et al. (2013) Hahn, E. M., Li, G., Schewe, S., Turrini, A., and Zhang, L.  Lazy probabilistic model checking without determinisation.  _arXiv preprint arXiv:1311.2928_ , 2013.
  * Hasanbeig et al. (2018) Hasanbeig, M., Abate, A., and Kroening, D.  Logically-constrained reinforcement learning, 2018.  URL https://arxiv.org/abs/1801.08099.
  * Hasanbeig et al. (2020) Hasanbeig, M., Kroening, D., and Abate, A.  Deep reinforcement learning with temporal logics.  In _International Conference on Formal Modeling and Analysis of Timed Systems_ , pp. 1–22. Springer, 2020.
  * Hasselt et al. (2016) Hasselt, H. v., Guez, A., and Silver, D.  Deep reinforcement learning with double q-learning.  In _Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence_ , AAAI’16, pp. 2094–2100. AAAI Press, 2016.
  * Ibarz et al. (2018) Ibarz, B., Leike, J., Pohlen, T., Irving, G., Legg, S., and Amodei, D.  Reward learning from human preferences and demonstrations in atari.  _Advances in neural information processing systems_ , 31, 2018.
  * Kingma & Ba (2015) Kingma, D. P. and Ba, J.  Adam: A method for stochastic optimization.  In Bengio, Y. and LeCun, Y. (eds.), _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_ , 2015.  URL http://arxiv.org/abs/1412.6980.
  * Křetínskỳ et al. (2018) Křetínskỳ, J., Meggendorfer, T., and Sickert, S.  Owl: a library for $\omega$-words, automata, and ltl.  In _International Symposium on Automated Technology for Verification and Analysis_ , pp. 543–550. Springer, 2018.
  * Lin (1992) Lin, L.-J.  Self-improving reactive agents based on reinforcement learning, planning and teaching.  _Machine learning_ , 8:293–321, 1992.
  * Mnih et al. (2013) Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M.  Playing atari with deep reinforcement learning.  _arXiv preprint arXiv:1312.5602_ , 2013.
  * Mnih et al. (2015) Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al.  Human-level control through deep reinforcement learning.  _nature_ , 518(7540):529–533, 2015.
  * Ng et al. (1999) Ng, A. Y., Harada, D., and Russell, S.  Policy invariance under reward transformations: Theory and application to reward shaping.  In _Icml_ , volume 99, pp. 278–287, 1999.
  * Puterman (2014) Puterman, M. L.  _Markov decision processes: discrete stochastic dynamic programming_.  John Wiley & Sons, 2014.
  * Randløv & Alstrøm (1998) Randløv, J. and Alstrøm, P.  Learning to drive a bicycle using reinforcement learning and shaping.  In _ICML_ , 1998.
  * Sadigh et al. (2014) Sadigh, D., Kim, E. S., Coogan, S., Sastry, S. S., and Seshia, S. A.  A learning based approach to control synthesis of markov decision processes for linear temporal logic specifications.  In _53rd IEEE Conference on Decision and Control_ , pp. 1091–1096, 2014.  doi: 10.1109/CDC.2014.7039527.
  * Sickert et al. (2016) Sickert, S., Esparza, J., Jaax, S., and Křetínský, J.  Limit-deterministic büchi automata for linear temporal logic.  In Chaudhuri, S. and Farzan, A. (eds.), _Computer Aided Verification_ , pp. 312–332, Cham, 2016. Springer International Publishing.  ISBN 978-3-319-41540-6.
  * Silver et al. (2021) Silver, D., Singh, S., Precup, D., and Sutton, R. S.  Reward is enough.  _Artificial Intelligence_ , 299:103535, 2021.
  * Sorg (2011) Sorg, J.  _The Optimal Reward Problem: Designing Effective Reward for Bounded Agents_.  PhD thesis, University of Michigan, USA, 2011.  URL https://hdl.handle.net/2027.42/89705.
  * Sutton & Barto (2018) Sutton, R. S. and Barto, A. G.  _Reinforcement Learning: An Introduction_.  The MIT Press, second edition, 2018.  URL http://incompleteideas.net/book/the-book-2nd.html.
  * Toro Icarte et al. (2022) Toro Icarte, R., Klassen, T. Q., Valenzano, R., and McIlraith, S. A.  Reward machines: Exploiting reward function structure in reinforcement learning.  _J. Artif. Int. Res._ , 73, may 2022.  ISSN 1076-9757.  doi: 10.1613/jair.1.12440.  URL https://doi.org/10.1613/jair.1.12440.
  * Toromanoff et al. (2019) Toromanoff, M., Wirbel, E., and Moutarde, F.  Is deep reinforcement learning really superhuman on atari? leveling the playing field.  _arXiv preprint arXiv:1908.04683_ , 2019.
  * Vaezipoor et al. (2021) Vaezipoor, P., Li, A. C., Icarte, R. T., and McIlraith, S. A.  Ltl2action: Generalizing LTL instructions for multi-task RL.  In _Proceedings of the 38th International Conference on Machine Learning, ICML_ , volume 139 of _Proceedings of Machine Learning Research_ , pp. 10497–10508, 2021.  URL http://proceedings.mlr.press/v139/vaezipoor21a.html.
  * Voloshin et al. (2022) Voloshin, C., Le, H. M., Chaudhuri, S., and Yue, Y.  Policy optimization with linear temporal logic constraints.  In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), _Advances in Neural Information Processing Systems_ , 2022.  URL https://openreview.net/forum?id=yZcPRIZEwOG.
  * Wang et al. (2020) Wang, C., Li, Y., Smith, S. L., and Liu, J.  Continuous motion planning with temporal logic specifications using deep neural networks, 2020.  URL https://arxiv.org/abs/2004.02610.
  * Wolff et al. (2012) Wolff, E. M., Topcu, U., and Murray, R. M.  Robust control of uncertain markov decision processes with temporal logic specifications.  In _2012 IEEE 51st IEEE Conference on Decision and Control (CDC)_ , pp. 3372–3379, 2012.  doi: 10.1109/CDC.2012.6426174.
  * Yang et al. (2021) Yang, C., Littman, M. L., and Carbin, M.  Reinforcement learning for general LTL objectives is intractable.  _CoRR_ , abs/2111.12679, 2021.  URL https://arxiv.org/abs/2111.12679.
  * Zhang et al. (2021) Zhang, B., Rajan, R., Pineda, L., Lambert, N., Biedenkapp, A., Chua, K., Hutter, F., and Calandra, R.  On the importance of hyperparameter optimization for model-based reinforcement learning.  In _International Conference on Artificial Intelligence and Statistics_ , 2021.

##  Appendix A Experiments

###  A.1 Environment Details

The environment and experiment details are summarized in Table 1.

Table 1: Environment Details Environment | Experiment | $\mathcal{S}^{\mathcal{M}}$ | $\mathcal{A}^{\mathcal{M}}$ | Dynamics | LTL Formula
---|---|---|---|---|---
Minecraft | Q-learning | Discrete | Discrete | Deterministic | $GF(y\;\&\;XF(b))\;\&\;(G\lnot r)$
Pacman | Q-learning | Discrete | Discrete | Stochastic | $F(\text{food})\;\&\;(G\lnot\text{ghost})$
Flatworld 1 | Q-learning | $\mathbb{R}^{2}$ | Discrete | Deterministic | $FGy$
Flatworld 2 | Q-learning | $\mathbb{R}^{2}$ | Discrete | Deterministic | $GF(y\;\&\;XF(r))\;\&\;(G\lnot b)$
Flatworld 3 | PPO | $\mathbb{R}^{2}$ | $[0,1]^{2}$ | Deterministic | $FGy$
Flatworld 4 | PPO | $\mathbb{R}^{2}$ | $[0,1]^{2}$ | Deterministic | $GF(y\;\&\;XF(r))\;\&\;(G\lnot b)$
Carlo | PPO | $\mathbb{R}^{5}$ | $[-1,1]^{2}$ | Deterministic | $GF(\text{zone1}\;\&\;XF(\text{zone2}))\;\&\;(G\lnot\text{crash})$

###  A.2 Experiment Setup

Each experiment is run with $10$ random seeds. Results from Figure 2 are from
an average over the seeds.

Q-learning experiments. Let $k$ be the greatest number of jump transitions
available in some LDBA state
$k=\max_{b\in\mathcal{S}^{\mathcal{B}}}|\mathcal{A}^{\mathcal{B}}(b)|$. Let
$m=\max_{s\in\mathcal{S}^{\mathcal{M}}}|\mathcal{A}^{\mathcal{M}}(s)|$. The
neural network $Q_{\theta}(s)$ takes as input $s\in\mathcal{S}^{\mathcal{M}}$
and outputs $\mathbb{R}^{(m+k)\times|\mathcal{S}^{\mathcal{B}}|}$ a
$(m+k)$-dim vector for each $b\in\mathcal{S}^{\mathcal{B}}$. For our purposes,
we consider $Q_{\theta}(s,b)$ to be the single $(m+k)$-dim vector
cooresponding to the particular current state of the LDBA $b$.

When $\mathcal{S}^{\mathcal{M}}$ is discrete then we parametrize
$Q_{\theta}(s,b)$ as a table. Otherwise, $Q_{\theta}(s,b)$ is parameterized by
$3$ linear layers with hidden dimension $128$ with intermediary ReLU
activations and no final activation. After masking for how many jump
transitions exist in $b$, we can select
$\arg\max_{i\in[0,\ldots,|\mathcal{A}^{\mathcal{B}}(b)|]}Q_{\theta}(s,b)_{i}$
the highest $Q$-value with probability $1-\eta$ and uniform with $\eta$
probability. Here, $\eta$ is initialized to $\eta_{0}$ and decays linearly (or
exponentially) at some specified frequency (see Table 2).

At each episode (after a rollout of length $T$), we perform $K$ gradient steps
with different batches of size given in Table 3. We use Adam optimizer (Kingma
& Ba, 2015) with a learning rate also specified by the table.

When in a continuous state space, we implement DDQN (Hasselt et al., 2016)
(rather than DQN) with a target network that gets updated at some frequency
specified by Table 3.

Table 2: Hyperparameters for Q-learning experiments (Discrete Action Space) | $\eta$ |  $\eta$ Decay |  |  |  |  |  |
---|---|---|---|---|---|---|---|---
Experiment | $\eta_{0}$ |  Min $\eta$ | Type | Rate | Freq | Batch size |  $K$ ($\\#$ batches) | LR | Target update | T | $\gamma$
Minecraft | .3 | 0 | Exponential | .9 | 100 | 128 | 20 | - | - | 100 | .99
Pacman | .4 | 0 | Linear | .05 | 400 | 512 | 200 | - | - | 100 | .999
Flatworld 1 | .8 | .15 | Exponential | .9 | 100 | 128 | 5 | .001 | 15 | 20 | .95
Flatworld 2 | .8 | .15 | Exponential | .9 | 100 | 128 | 5 | .001 | 15 | 50 | .95

PPO experiments. Let $k$ be the greatest number of jump transitions available
in some LDBA state
$k=\max_{b\in\mathcal{S}^{\mathcal{B}}}|\mathcal{A}^{\mathcal{B}}(b)|$. The
neural network $f_{\theta}(s)$ takes as input $s\in\mathcal{S}^{\mathcal{M}}$
and outputs $\mathbb{R}^{(k+2)\times|\mathcal{S}^{\mathcal{B}}|}$ is a
$(k+2)$-dim vector for each $b\in\mathcal{S}^{\mathcal{B}}$. For our purposes,
we consider $f_{\theta}(s,b)$ to be the single $(k+2)$-dim vector
cooresponding to the particular current state of the LDBA $b$.

$f_{\theta}(s,b)$ is parameterized by $3$ linear layers with hidden dimension
$64$ with intermediary ReLU activations. The first dimension corresponds to
sampling a Gaussian action
$a\sim\mathcal{N}(f_{\theta}(s,b)[0],\text{diag}(\sigma^{2}))$ where $\sigma$
is initialized to $\sigma_{0}$ (see Table 3) and decays exponentially (at a
rate given in the table) every $10$ episodes. The remaining $k+1$ dimensions
(after proper masking to account for the size of
$|\mathcal{A}^{\mathcal{B}}(b)|$ and softmax) represent the probability
$p=[p_{a},p_{\epsilon_{0}},\ldots,p_{\epsilon_{k}}]$ of taking either the MDP
action $a$ or a some jump transition $\epsilon_{i}$. We sample from a
Categorical($p$) variable to select whether to return
$a\sim\mathcal{N}(\text{Tanh}(f_{\theta}(s,b)[0]),\text{diag}(\sigma^{2}))$ or
$a=\epsilon_{i}$ for some $i$. The density can be calculated by multiplying
$p_{a}$ by the Gaussian density when $a$ is selected, and $p_{\epsilon_{i}}$
otherwise.

For the critic, we have a parametrized network $f_{\phi}(s,b)\to\mathbb{R}$ of
$3$ linear layers with hidden dimension $64$ with intermediary Tanh
activations and no final activation.

At each episode (after a rollout of length $T$), we perform $5$ gradient steps
with different batches of size given in Table 3. The importance sampling term
in PPO is clipped to $1\pm.4$. The critic learning rate is $.01$. We use Adam
optimizer (Kingma & Ba, 2015) for both the actor and critic.

Table 3: Hyperparameters for PPO experiments (Continuous Action Space) Experiment | $\sigma_{0}$ |  $\sigma$ Decay Rate |  Min $\sigma$ | Batch size | LR Actor | T
---|---|---|---|---|---|---
Flatworld 3 | $1.8$ | $.98$ | $.3$ | $128$ | $.001$ | 20
Flatworld 4 | $1.8$ | $.99$ | $.1$ | $128$ | $.001$ | 50
Carlo | $.5$ | $.999$ | $.3$ | $16$ | $.0001$ | 500

##  Appendix B Constructing feasible trajectories for policy gradient during
rollout

Suppose we wanted to generate feasible trajectories in realtime while the
policy is being rolled out. That is, we have a partial trajectory of the form
$\tau_{t}=(s_{0},b_{0},a_{0},\ldots,s_{t},b_{t})$ generated by running $\pi$
in $P$. Let $a_{t}=a\in\mathcal{A}$ be the $t$-th action taken by $\pi$ and
$s_{t+1}=s^{\prime}\in\mathcal{M}$ be the next observed state observed in the
MDP.

Let $\mathcal{T}_{t}$ be the current set of feasible (partial) trajectories at
timestep $t$. Elements
$\tau_{k}=(s_{0},b_{0},a_{0},\ldots,s_{k},b_{k})\in\mathcal{T}_{t}$ denote
$k$-step (partial) trajectory, not necessarily part of the trajectory observed
during the course of a rollout of $\pi$. Here, $k\geq t$. Then, for each
$\tau_{k}\in\mathcal{T}_{t}$, one of 4 cases holds:

Case 1. Action $a$ is not a jump transition (ie.
$a\in\mathcal{A}^{\mathcal{M}}(s_{k})$) and there are no jump transitions
available in $b_{k}$ ($\mathcal{A}^{\mathcal{B}}(b_{k})=\varnothing$). Then we
can form the concatenation: $\tau_{k+1}=\tau_{k}\cup(a,s^{\prime},b_{k+1})$
where $b_{k+1}=P^{\mathcal{B}}(b_{k},L^{\mathcal{M}}(s^{\prime}))$. We set
$\mathcal{T}_{\epsilon}=\varnothing$.

Case 2. Action $a$ is a jump transition and is currently feasible in $b_{k}$
(ie. $a\in\mathcal{A}^{\mathcal{M}}(b_{k})$). Then we can form the
concatenation $\tau_{k+1}=\tau_{k}\cup(a,s^{\prime},b_{k+1})$ where
$b_{k+1}=P^{\mathcal{B}}(b_{k},a)$. We set
$\mathcal{T}_{\epsilon}=\varnothing$.

Case 3. Action $a$ is a not a jump transition (ie.
$a\in\mathcal{A}^{\mathcal{M}}(s_{k})$), but there is at least one feasible
jump transition in $b_{k}$ (ie.
$\mathcal{A}^{\mathcal{B}}(b_{k})\neq\varnothing$). Then, in addition to
forming $\tau_{k+1}$ from Case 1, we have all the possible jumps:

$\mathcal{T}_{\epsilon}=\\{\tau_{k}\cup(\epsilon,s_{k},b_{k+1},a,s^{\prime},b_{k+2})|\forall\epsilon\in\mathcal{A}^{\mathcal{B}}(b_{k}),\\\
b_{k+1}=P^{\mathcal{B}}(b_{k},\epsilon),b_{k+2}=P^{\mathcal{B}}(b_{k+1},a_{t})\\}$

Case 4. Action $a$ is a jump transition is infeasible in $b_{k}$ (ie.
$a\not\in\mathcal{A}^{\mathcal{B}}(b_{k})$). In this case, we just pass this
trajectory. Setting $\tau_{k+1}=\tau_{k}$ and
$\mathcal{T}_{\epsilon}=\varnothing$.

At the end of iterating over each element of $\tau_{k}\in\mathcal{T}_{t}$ and
forming $\tau_{k+1}$ and $\mathcal{T}_{\epsilon}$, we can update our current
set of feasible trajectories:

$\mathcal{T}_{t+1}=\cup_{\tau_{k}\in\mathcal{T}_{t}}\bigg{(}(\mathcal{T}_{t}\setminus\\{\tau_{k}\\})\cup\\{\tau_{k+1}\\}\cup\mathcal{T}_{\epsilon}\bigg{)}$
(19)

To put this process simply, we are swapping out $\tau_{k}$ for $\tau_{k+1}$
and also adding in any jump transitions if they are available. The algorithm
can be seen in Algo 4.

Algorithm 4 LCER for Policy Gradient (Option 2)

0: Dataset $D$. Trajectory $\tau$ of length $T$.

1: Set $\mathcal{T}_{0}\leftarrow\\{(s_{0},b)|b\in\mathcal{B}\\}$

2: for $(s_{t},a_{t},s_{t+1})\in\tau$ do

3: Form $\mathcal{T}_{t}$ according to Eq (19)

4: end for

5: Set $D\leftarrow D\cup\mathcal{T}_{T}$

6: return $D$


[affiliation=1]DanLyth [affiliation=2]SimonKing

# Natural language guidance of high-fidelity text-to-speech with synthetic
annotations

###### Abstract

Text-to-speech models trained on large-scale datasets have demonstrated
impressive in-context learning capabilities and naturalness. However, control
of speaker identity and style in these models typically requires conditioning
on reference speech recordings, limiting creative applications. Alternatively,
natural language prompting of speaker identity and style has demonstrated
promising results and provides an intuitive method of control. However,
reliance on human-labeled descriptions prevents scaling to large datasets.

Our work bridges the gap between these two approaches. We propose a scalable
method for labeling various aspects of speaker identity, style, and recording
conditions. We then apply this method to a 45k hour dataset, which we use to
train a speech language model. Furthermore, we propose simple methods for
increasing audio fidelity, significantly outperforming recent work despite
relying entirely on found data.

Our results demonstrate high-fidelity speech generation in a diverse range of
accents, prosodic styles, channel conditions, and acoustic conditions, all
accomplished with a single model and intuitive natural language conditioning.
Audio samples can be heard at https:// text-description-to-speech.com/.

##  1 Introduction

Scaling both model and training data size has driven rapid progress in
generative modeling, especially for text and image synthesis [1, 2, 3, 4].
Natural language conditioning provides an intuitive method for control and
creativity in these modalities, enabled by web-scale human-authored text and
image annotations [5, 6]. However, only recently has speech synthesis started
to exploit scale and natural language conditioning.

The initial results from large-scale text-to-speech (TTS) models have
demonstrated impressive in-context learning capabilities, such as zero-shot
speaker and style adaptation, cross-lingual synthesis, and content editing [7,
8, 9]. However, a reliance on reference speech limits their practical
application. It also forces the user to reproduce the likeness of an existing
speaker, which is beneficial in some use cases but has the potential for harm,
especially when so little enrollment data is required.

To alleviate these shortcomings, the use of natural language to describe
speaker and style is starting to be explored (the most recent example is
concurrent work to ours, Audiobox. [10]). Unlike the image modality, no large
dataset containing natural language descriptions of speech exists, so this
metadata must be created from scratch. To date, this has been achieved using a
combination of human annotations and statistical measures, with the results
often passed through a large language model to mimic the variability that
might be expected in genuine human annotations [10, 11, 12, 13, 14]. However,
any approach requiring human annotations is challenging to scale to large
datasets. For example, Multilingual LibriSpeech [15] contains over 10 million
utterances across 50k hours of audio, equivalent to over five years. Because
of the human annotation bottleneck, TTS models using natural language
descriptions have been of limited scale and, therefore, unable to demonstrate
some of the broad range of capabilities associated with larger models.

In this work, we rely entirely on automatic labeling, enabling us to scale to
large data for the first time (along with concurrent work [10]). Coupling this
with large-scale speech language models allows us to synthesize speech in a
wide range of speaking styles and recording conditions using intuitive natural
language control.

Specifically, we:

  1. 1.

Propose a method for efficiently labeling a 45k hour dataset with multiple
attributes, including gender, accent, speaking rate, pitch, and recording
conditions.

  2. 2.

Train a speech language model on this dataset and demonstrate the ability to
control these attributes independently, creating speaker identities and style
combinations unseen in the training data.

  3. 3.

Demonstrate that with as little as 1% high-fidelity audio in the training data
and the use of the latest state-of-the-art audio codec models, it is possible
to generate extremely high-fidelity audio.

Figure 1: Overview of the model architecture

##  2 Related Work

###  2.1 Control of speaker identity and style

Controlling non-lexical speech information, such as speaking style and speaker
identity, has been explored through various approaches. With neural models,
the first attempts at this relied on reference embeddings or ``global style
tokens'' derived from exemplar recordings [16, 17]. This approach is effective
but constrains users to existing recordings, significantly limiting
versatility and scalability. To alleviate this, more flexible approaches
sample from the continuous latent spaces of Gaussian mixture models and
variational autoencoders [18]. However, this approach requires careful
training to ensure that the latent variables are disentangled, as well as
complex analysis to identify the relationship between these variables and
attributes of speech.

In an attempt to bypass the brittleness of reference embeddings and the
complexity of disentangled latent variable approaches, recent work has
attempted to use natural language descriptions to guide non-lexical speech
variation directly. This line of work has no doubt been inspired by the
success in other modalities (particularly text-to-image models), but a key
challenge for speech is the lack of natural language descriptive metadata.

An initial attempt at circumventing this issue was proposed in [11]. In this
work, the authors use statistically derived metrics (such as speaking rate and
pitch) from a dataset of real speech, combined with emotion labels from a
dataset of synthetic speech provided by a commercial TTS model. Together, this
dataset offers five axes of non-lexical variation, which are turned into
keywords, each with three levels of granularity (high, medium, and low).

The authors of [12] move away from computational labeling methods and explore
human annotation. They label 44 hours of data with natural language sentences
describing style and emotion but rely on a fixed set of speaker IDs to control
speaker likeness. Human annotation is used again in [13] (with an even smaller
6-hour dataset) in service of style transfer, and again, speaker likeness is
controlled by speaker IDs. However, in [14], the authors do tackle the
labeling and generation of different speaker identities. Human annotations are
combined with computational statistics similar to [11] and are then fed into a
language model to create natural language variations. Unlike our work, the
authors make no attempt to model accent or channel conditions and only label
16% of the speakers in a dataset already two orders of magnitude smaller than
the one we use. This difficulty in scaling human annotations provides some of
the motivation for this work.

While we were running the evaluations for this work, the authors of [10]
released Audiobox. This concurrent work uses a similar approach to ours to
label a large dataset. However, we propose simple methods to significantly
outperform this work in terms of the overall naturalness and audio fidelity of
the generated speech. We also outperform this work in how closely our model
matches the text description (for those attributes of speech shared across
both lines of work). PromptTTS2 [19] is also concurrent work that scales to a
large dataset, but their approach only attempts to control four variables,
significantly limiting the range of capabilities.

##  3 Method

###  3.1 Metadata collection

The data for this study comprises two English speech corpora derived from the
LibriVox audiobook project111librivox.org \- the English portion of
Multilingual LibriSpeech (MLS) [15] (45k hours) and LibriTTS-R [20] (585
hours). While LibriTTS-R is significantly smaller in scale, we include it
given the higher audio quality resulting from enhancement via the Miipher
system [21]. Both datasets provide transcriptions and a label for gender
generated using a predictive model.

####  3.1.1 Accent

Speaker accent is an aspect of speech that natural language prompting in TTS
has so far overlooked, despite the wide range of accents found in the datasets
typically used.

We appreciate that labeling accents with discrete labels is an ill-formed task
considering the discrete-continuous nature of accents. However, the
alternative, i.e., ignoring accent altogether, is unacceptable. To this end,
we train an accent classifier and use it to label our datasets.

We use EdAcc [22], VCTK [23], and the English-accented subset of VoxPopuli
[24] as the training data for our accent classifier. In total, these datasets
cover 53 accents. We extract embeddings using the language ID model from [25]
and train a simple linear classifier using these embeddings, achieving an
accuracy of 86% on a held-out test set. We then run this model on our datasets
and spot-check the results.

####  3.1.2 Recording quality

Large-scale publicly available speech datasets are typically derived from
crowd-sourced projects such as LibriVox. This leads to a fundamental
limitation of these datasets - the audio recording quality is often suboptimal
compared to professional recordings. For example, many utterances have low
signal-to-noise, narrow bandwidth, codec compression artifacts, and excessive
reverberation.

To circumvent this limitation, we include LibriTTS-R, a dataset derived from
LibriVox but which has, as mentioned, significantly improved audio fidelity.
By including this high-fidelity dataset and labeling features related to
recording quality across both datasets, we hypothesize that the model will
learn a latent representation of audio fidelity. Crucially, this should allow
the generation of clean, professional-sounding recordings for accents and
styles that only have low-fidelity utterances in the training data.

The two proxies we use for labeling recording quality are the estimated
signal-to-noise ratio (SNR) and estimated C50. C50 is the ratio of early
reflections to late reflections and indicates how reverberant a recording is.
For both of these features, we use the Brouhaha
library222github.com/marianne-m/brouhaha-vad introduced in [26].

####  3.1.3 Pitch and speaking rate

We compute pitch contours for all utterances using the PENN
library333github.com/interactiveaudiolab/penn proposed in [27] and then
calculate the speaker-level mean and utterance-level standard deviation. The
speaker-level mean is used to generate a label for speaker pitch relative to
gender, and the standard deviation is used as a proxy for how monotone or
animated an individual utterance is.

The speaking rate is simply calculated by dividing the number of phonemes in
the transcript by the utterance length (silences at the beginning and end of
the audio files have already been removed). We use the library
g2p444github.com/roedoejet/g2p for the grapheme-to-phoneme conversion.

###  3.2 Metadata preparation

The next stage is to take all the variables described above and convert them
into natural language sentences. To do this, we first create keywords for each
variable.

The discrete labels such as gender (provided by the dataset creators) and
accent require no further processing and can be directly used as keywords.
However, the pitch, speaking rate, and estimated SNR and C50 are all
continuous variables that must first be mapped to discrete categories. We do
this by analyzing the variables across the full dataset and then applying
appropriate binning. A visual example of this is shown in Figure 2, where the
estimated SNR across all utterances can be seen along with the bin boundaries.
For each variable, we apply seven bins and then use appropriate short phrases
to describe each bin. For example, in the case of speaking rate, we use terms
such as “very fast”, “quite fast”, “fairly slowly” etc.

Once this binning is complete for all continuous variables, we have keywords
for gender, accent, pitch relative to speaker, pitch standard deviation,
speaking rate, estimated SNR, and estimated C50. We also create a new category
when the SNR and C50 are both in their highest or lowest bin and label these
“very good recording” and “very bad recording”, respectively.

To improve generalization and allow the user to input descriptive phrases
using their own terminology, we feed these sets of keywords into a language
model (Stable Beluga 2.5555huggingface.co/stabilityai/StableBeluga2) with
appropriate prompts to create full sentences. For example, “female”,
“Hungarian”, “slightly roomy sounding”, “fairly noisy”, “quite monotone”,
“fairly low pitch”, “very slowly” could be converted into “a woman with a deep
voice speaking slowly and somewhat monotonously with a Hungarian accent in an
echoey room with background noise”.

###  3.3 Model

We adapt the general-purpose audio generation library
AudioCraft666github.com/facebookresearch/audiocraft and make it suitable for
TTS. This library supports multiple forms of conditioning (text, embeddings,
wave files) that can be applied in various ways (pre-pending, cross-attention,
summing, interpolation). To date, it has only been used for audio and music
generation [28, 29]. In order to make it suitable for TTS, we remove word
drop-out from the transcript conditioning and, after some initial experiments,
settle on pre-pending the transcript and using cross-attention for the
description (see Figure 1). Unlike previous work, we do not provide any form
of audio or speaker embedding conditioning from similar utterances - the model
must rely entirely on the text description for gender, style, accent, and
channel conditions.

We use the Descript Audio Codec777github.com/descriptinc/descript-audio-codec
(DAC) (44.1kHz version) introduced in [30] to provide our discrete feature
representations. This residual vector quantized model produces tokens at a
frame rate of 86Hz and has nine codebooks. We use the delay pattern introduced
in [29] to deal with these nine dimensions in the context of a language model
architecture. We choose this codec rather than the popularly used Encodec [31]
because the authors of DAC demonstrate a subjective and objective improvement
in audio fidelity over Encodec.

Figure 2: Estimated SNR across the MLS dataset and the discrete bin boundaries
used to create keywords.

##  4 Experiments

###  4.1 Objective evaluation

To evaluate whether our model is capable of generating speech that matches a
provided description, we first carry out the following objective evaluations.
To test the targeted control of specific attributes, we use the test set
sentences from MLS and LibriTTS-R but manually write the description to test
the variable of interest. The only exception is the accent test set, where we
combine sentences from the Rainbow Passage [32], Comma Gets a Cure [33], and
Please Call Stella. In all cases, we ensure that the descriptions are balanced
across the test set.

To test control of gender, we use a pre-trained gender
classifier888huggingface.co/alefiury/wav2vec2-large-xlsr-53-gender-
recognition-librispeech that achieves an F1 score of 0.99 on LibriSpeech Clean
100. Using this classifier, our generated test set scores an accuracy of 94%.
In a similar fashion, we re-use our accent classifier (see Section 3.1.1) and
classify the accents of our generated accent test set. Here, we see a somewhat
poorer accuracy of 68%. We hypothesize that this is likely to be due to noisy
labeling and a very imbalanced distribution of accents in the training set.

The remaining attributes that we labeled are continuous variables. For these
variables, we run our generated test sets through the same models that were
used to label the training set (see Section 3.1). The results can be seen in
Figure 3. We see that for every attribute other than C50, the model performs
fairly well at generating speech that matches the provided description. We are
unsure as to why the model performs poorly at generating audio with the
appropriate C50, and further investigation is required.

Our final objective evaluation aims to quantify the audio fidelity of our
model when asked to produce audio with “excellent recording quality” or
similar terms. Here, we use the recently proposed Torchaudio Speech Quality
and Intelligibility Measures [34]. This model provides a reference-less
estimate of Wideband Perceptual Estimation of Speech Quality (PESQ), Short-
Time Objective Intelligibility (STOI), and Scale-Invariant Signal-to-
Distortion Ratio (SD-SRD). Using 20 test sentences and descriptions from
LibriTTS-R, we run these metrics on outputs from our model, Audiobox (using
the public website
interface999audiobox.metademolab.com/capabilities/tts_description_condition),
and the ground truth audio. As can be seen in Table 1, our model produces
speech with values that are significantly higher than Audiobox and often very
close to the ground truth. As mentioned in Section 3.1.2, this is achieved
despite training on only $\sim$500 hours of high-fidelity speech in the
context of the full 45k hour dataset.

(a) Mean pitch

(b) Pitch standard deviation

(c) Speaking rate

(d) Estimated C50

(e) Estimated SNR

Figure 3: Correlation between description labels and synthesized labels (with
95% confidence intervals).

###  4.2 Subjective evaluation

To complement our objective evaluations, we run two subjective listening
tests. The first of these aims to quantify how well our model follows a
natural language description (”relevance” or “REL”). An example of such a
description taken from the listening test set is: “A female voice with an
Italian accent reads from a book. The recording is very noisy. The speaker
reads fairly quickly with a slightly high-pitched and monotone voice.” We
generate 40 sets of samples (20 from MLS and 20 from LibriTTS-R) using
descriptions created using the method described in Section 3. Again, we employ
30 listeners (native English speakers) and ask them to evaluate how closely
the speech matches the description using a 5-point scale. For each sentence
and description, we present the listeners with outputs from our model,
Audiobox, and the ground truth audio. The only processing we apply is loudness
normalization to -18 LUFS and the removal of silence before and after speech.

As can be seen in Table 2, our model outperforms Audiobox in this evaluation.
Somewhat counterintuitively, it also outperforms the ground truth. However,
there appear to be two clear reasons for this. Firstly, the test set
descriptions contain label noise. For example, if a ground truth utterance is
labeled with the incorrect accent, the generated speech is likely to be more
faithful to the description than the ground truth. Similarly, we also see very
occasional instances where samples from LibriTTS-R contain mild audio
artifacts (we removed samples containing significant artifacts). In this case,
the audio fidelity from our model is likely to be higher than the ground truth
and, therefore, closer to the description. One final note on this evaluation
is that we are aware that there are aspects of speech that Audiobox is capable
of controlling that our model is not (for example, age).

Table 1: Speech Quality and Intelligibility Measures (SQUIM) (with 95% confidence intervals) Model | PESQ | STOI | SI-SDR
---|---|---|---
Ground truth |  4.15 $\pm{0.04}$ |  0.997 $\pm{0.001}$ |  27.45 $\pm{1.09}$
Ours |  3.84 $\pm{0.10}$ |  0.996 $\pm{0.001}$ |  26.53 $\pm{1.16}$
Audiobox |  3.46 $\pm{0.16}$ |  0.988 $\pm{0.004}$ |  21.84 $\pm{1.37}$
Table 2: Naturalness and relevance results (with 95% confidence intervals) Model | MOS | REL
---|---|---
Ground truth |  3.67 $\pm{0.09}$ |  3.62 $\pm{0.06}$
Ours |  3.92 $\pm{0.07}$ |  3.88 $\pm{0.06}$
Audiobox |  2.79 $\pm{0.09}$ |  3.19 $\pm{0.06}$

Our second listening test aims to evaluate the overall naturalness and audio
fidelity of our model. In this case, we only use samples from LibriTTS-R. We
ask 30 listeners to rate the speech on a scale of 1-5 (we’ll refer to this
simply as mean opinion score, or MOS). We appreciate that this evaluation
conflates the naturalness of speech patterns and audio fidelity, but we chose
this test design to match that used by the authors of Audiobox.

In Table 2, we see that our model significantly outperforms Audiobox. We
hypothesize that there are two reasons for this. Firstly, we believe that our
use of the DAC codec rather than Encodec has a significant impact (see Section
3.3 for further details). Secondly, our use (and labeling) of a small high-
fidelity dataset (LibriTTS-R) may well have had a significant impact. However,
this hypothesis is difficult to validate without full knowledge of the data
used to train Audiobox.

Again, we see our model outperforming the ground truth. As discussed
previously, it would appear that this is due to minor audio artifacts in
LibriTTS-R caused by the speech-enhancement model. See our demo
website101010text-description-to-speech.com to hear examples.

##  5 Conclusion

In this work, we propose a simple but highly effective method for generating
high-fidelity text-to-speech that can be intuitively guided by natural
language descriptions. To the best of our knowledge, this is the first time
that such a method is capable of controlling such a wide range of speech and
channel condition attributes in conjunction with such high audio fidelity and
overall naturalness. In particular, we note that this was possible with an
amateurly recorded dataset coupled with a comparatively tiny amount of clean
data.

However, this work only demonstrates efficacy in a relatively narrow domain
(audiobook reading in English). In the future, we plan to extend to a wider
range of languages, speaking styles, vocal effort, and channel conditions.

## References

  * [1] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei, ``Language Models are Few-Shot Learners.'' [Online]. Available: http://arxiv.org/abs/2005.14165
  * [2] A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, P. Schuh, K. Shi, S. Tsvyashchenko, J. Maynez, A. Rao, P. Barnes, Y. Tay, N. Shazeer, V. Prabhakaran, E. Reif, N. Du, B. Hutchinson, R. Pope, J. Bradbury, J. Austin, M. Isard, G. Gur-Ari, P. Yin, T. Duke, A. Levskaya, S. Ghemawat, S. Dev, H. Michalewski, X. Garcia, V. Misra, K. Robinson, L. Fedus, D. Zhou, D. Ippolito, D. Luan, H. Lim, B. Zoph, A. Spiridonov, R. Sepassi, D. Dohan, S. Agrawal, M. Omernick, A. M. Dai, T. S. Pillai, M. Pellat, A. Lewkowycz, E. Moreira, R. Child, O. Polozov, K. Lee, Z. Zhou, X. Wang, B. Saeta, M. Diaz, O. Firat, M. Catasta, J. Wei, K. Meier-Hellstern, D. Eck, J. Dean, S. Petrov, and N. Fiedel, ``PaLM: Scaling language modeling with pathways.'' [Online]. Available: http://arxiv.org/abs/2204.02311
  * [3] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen, ``Hierarchical text-conditional image generation with CLIP latents,'' version: 1. [Online]. Available: http://arxiv.org/abs/2204.06125
  * [4] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-Resolution Image Synthesis with Latent Diffusion Models. [Online]. Available: http://arxiv.org/abs/2112.10752
  * [5] [Online]. Available: https://commoncrawl.org/
  * [6] C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, P. Schramowski, S. Kundurthy, K. Crowson, L. Schmidt, R. Kaczmarczyk, and J. Jitsev, ``LAION-5b: An open large-scale dataset for training next generation image-text models.'' [Online]. Available: http://arxiv.org/abs/2210.08402
  * [7] C. Wang, S. Chen, Y. Wu, Z. Zhang, L. Zhou, S. Liu, Z. Chen, Y. Liu, H. Wang, J. Li, L. He, S. Zhao, and F. Wei. Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers. [Online]. Available: http://arxiv.org/abs/2301.02111
  * [8] Z. Zhang, L. Zhou, C. Wang, S. Chen, Y. Wu, S. Liu, Z. Chen, Y. Liu, H. Wang, J. Li, L. He, S. Zhao, and F. Wei. Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling. [Online]. Available: http://arxiv.org/abs/2303.03926
  * [9] M. Le, A. Vyas, B. Shi, B. Karrer, L. Sari, R. Moritz, M. Williamson, V. M. Y. Adi, J. Mahadeokar, and W.-N. Hsu, ``Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale.''
  * [10] A. Team, A. Vyas, B. Shi, M. Le, A. Tjandra, Y.-C. Wu, B. Guo, J. Zhang, X. Zhang, R. Adkins, W. Ngan, J. Wang, I. Cruz, B. Akula, A. Akinyemi, B. Ellis, R. Moritz, Y. Yungster, A. Rakotoarison, L. Tan, C. Summers, C. Wood, J. Lane, M. Williamson, and W.-N. Hsu, ``Audiobox: Unified Audio Generation with Natural Language Prompts.''
  * [11] Z. Guo, Y. Leng, Y. Wu, S. Zhao, and X. Tan. PromptTTS: Controllable Text-to-Speech with Text Descriptions. [Online]. Available: http://arxiv.org/abs/2211.12171
  * [12] D. Yang, S. Liu, R. Huang, G. Lei, C. Weng, H. Meng, and D. Yu. InstructTTS: Modelling Expressive TTS in Discrete Latent Space with Natural Language Style Prompt. [Online]. Available: http://arxiv.org/abs/2301.13662
  * [13] G. Liu, Y. Zhang, Y. Lei, Y. Chen, R. Wang, Z. Li, and L. Xie. PromptStyle: Controllable Style Transfer for Text-to-Speech with Natural Language Descriptions. [Online]. Available: http://arxiv.org/abs/2305.19522
  * [14] R. Shimizu, R. Yamamoto, M. Kawamura, Y. Shirahata, H. Doi, T. Komatsu, and K. Tachibana. PromptTTS++: Controlling Speaker Identity in Prompt-Based Text-to-Speech Using Natural Language Descriptions. [Online]. Available: http://arxiv.org/abs/2309.08140
  * [15] V. Pratap, Q. Xu, A. Sriram, G. Synnaeve, and R. Collobert, ``MLS: A Large-Scale Multilingual Dataset for Speech Research,'' in _Interspeech 2020_ , pp. 2757–2761. [Online]. Available: http://arxiv.org/abs/2012.03411
  * [16] R. J. Skerry-Ryan, E. Battenberg, Y. Xiao, Y. Wang, D. Stanton, J. Shor, R. J. Weiss, R. Clark, and R. A. Saurous, ``Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron.'' [Online]. Available: http://arxiv.org/abs/1803.09047
  * [17] Y. Wang, D. Stanton, Y. Zhang, R. J. Skerry-Ryan, E. Battenberg, J. Shor, Y. Xiao, F. Ren, Y. Jia, and R. A. Saurous, ``Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis.'' [Online]. Available: http://arxiv.org/abs/1803.09017
  * [18] W.-N. Hsu, Y. Zhang, R. J. Weiss, H. Zen, Y. Wu, Y. Wang, Y. Cao, Y. Jia, Z. Chen, J. Shen, P. Nguyen, and R. Pang, ``Hierarchical Generative Modeling for Controllable Speech Synthesis.'' [Online]. Available: http://arxiv.org/abs/1810.07217
  * [19] Y. Leng, Z. Guo, K. Shen, X. Tan, Z. Ju, Y. Liu, Y. Liu, D. Yang, L. Zhang, K. Song, L. He, X.-Y. Li, S. Zhao, T. Qin, and J. Bian. PromptTTS 2: Describing and Generating Voices with Text Prompt. [Online]. Available: http://arxiv.org/abs/2309.02285
  * [20] Y. Koizumi, H. Zen, S. Karita, Y. Ding, K. Yatabe, N. Morioka, M. Bacchiani, Y. Zhang, W. Han, and A. Bapna, ``LibriTTS-r: A restored multi-speaker text-to-speech corpus.'' [Online]. Available: http://arxiv.org/abs/2305.18802
  * [21] Y. Koizumi, H. Zen, S. Karita, Y. Ding, K. Yatabe, N. Morioka, Y. Zhang, W. Han, A. Bapna, and M. Bacchiani, ``Miipher: A robust speech restoration model integrating self-supervised speech and text representations.'' [Online]. Available: http://arxiv.org/abs/2303.01664
  * [22] R. Sanabria, N. Bogoychev, N. Markl, A. Carmantini, O. Klejch, and P. Bell. The Edinburgh International Accents of English Corpus: Towards the Democratization of English ASR. [Online]. Available: http://arxiv.org/abs/2303.18110
  * [23] ``CSTR VCTK Corpus: English multi-speaker corpus for cstr voice cloning toolkit (version 0.92).'' [Online]. Available: https://datashare.ed.ac.uk/handle/10283/3443
  * [24] C. Wang, M. Rivière, A. Lee, A. Wu, C. Talnikar, D. Haziza, M. Williamson, J. Pino, and E. Dupoux, ``VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation.'' [Online]. Available: http://arxiv.org/abs/2101.00390
  * [25] V. Pratap, A. Tjandra, B. Shi, P. T. A. Babu, S. Kundu, A. Elkahky, Z. Ni, A. Vyas, M. F.-Z. A. Baevski, Y. Adi, X. Zhang, W.-N. Hsu, A. Conneau, and M. Auli, ``Scaling Speech Technology to 1,000+ Languages.''
  * [26] M. Lavechin, M. Métais, H. Titeux, A. Boissonnet, J. Copet, M. Rivière, E. Bergelson, A. Cristia, E. Dupoux, and H. Bredin. Brouhaha: Multi-task training for voice activity detection, speech-to-noise ratio, and C50 room acoustics estimation. [Online]. Available: http://arxiv.org/abs/2210.13248
  * [27] M. Morrison, C. Hsieh, N. Pruyne, and B. Pardo. Cross-domain Neural Pitch and Periodicity Estimation. [Online]. Available: http://arxiv.org/abs/2301.12258
  * [28] F. Kreuk, ``AudioGen: Textually Guided Audio Generation.'' [Online]. Available: https://openreview.net/forum?id=CYK7RfcOzQ4
  * [29] J. Copet, F. Kreuk, I. Gat, T. Remez, D. Kant, G. Synnaeve, Y. Adi, and A. Défossez. Simple and Controllable Music Generation. [Online]. Available: http://arxiv.org/abs/2306.05284
  * [30] R. Kumar, P. Seetharaman, A. Luebs, I. Kumar, and K. Kumar. High-Fidelity Audio Compression with Improved RVQGAN. [Online]. Available: http://arxiv.org/abs/2306.06546
  * [31] A. Défossez, J. Copet, G. Synnaeve, and Y. Adi. High Fidelity Neural Audio Compression. [Online]. Available: http://arxiv.org/abs/2210.13438
  * [32] G. Fairbanks, _Voice and Articulation Drillbook_ , 1960.
  * [33] D. Honorof, J. McCullough, and B. Somerville, ``Comma gets a cure: A diagnostic passage for accent study,'' _Retrieved February_ , vol. 20, p. 2007, 2000.
  * [34] A. Kumar, K. Tan, Z. Ni, P. Manocha, X. Zhang, E. Henderson, and B. Xu. TorchAudio-Squim: Reference-less Speech Quality and Intelligibility measures in TorchAudio. [Online]. Available: http://arxiv.org/abs/2304.01448