# Propensity Score Matching underestimates Real Treatment Effect, in a
simulated theoretical multivariate model

Daniel García Iglesias  Arrhythmya Unit. Cardiology Department. Hospital
Universitario Central de Asturias. Oviedo. Spain.  Instituto de Investigación
Sanitaria del Principado de Asturias. Oviedo. Spain

###### Abstract

Propensity Score Matching (PSM) is an useful method to reduce the impact
ofTreatment - Selection Bias in the estimation of causal effects in
observational studies. After matching, the PSM significantly reduces the
sample under investigation, which may lead to other possible biases. In this
sense, we want to analyse the behaviour of this PSM compared with other widely
used method to deal with non-comparable groups, as is the Multivariate
Regression Model (MRM). Monte Carlo Simulations are made to construct groups
with different effects in order to compare the behaviour of PSM and MRM
estimating this effects. Also the Treatment - Selection Bias reduction for the
PSM is calculated. With the PSM a reduction in the Treatment - Selection Bias
is achieved, with a reduction in the Relative Real Treatment Effect Estimation
Error, but despite of this bias reduction and estimation error reduction, the
MRM significantly reduces more this estimation error compared with the PSM.
Also the PSM leads to a not insignificant reduction of the sample. This loss
of information derived from the matching process may lead to another not known
bias, and thus, to the inaccurate of the effect estimation compared with the
MRM.

Keywords: Propensity Score Matching, Multivariate Analysis, General Linear
Model, Monte Carlo Method, Causal Effect Estimation, Observational Study.

Abreviations: PS: Propensity Score, PSM: Propensity Score Matching, RTE: Real
Treatment Effect, GLM: General Linear Model, MRM: Multivariate Regression
Model.

##  1 Introduction

Propensity Score Matching (PSM) is an useful method to reduce the impact of
treatment-selection bias in the estimation of causal effects in observational
studies. Since firstly described by Rosenbaum and Rubin in 1983 [1], its
utility in Medicine, Psicology, Economics and other fields, has increased
exponentially in the last years [2]. Although it does not bypass the necessity
for randomised studies, it may be an alternative to reduce the impact of
treatment-selection bias in observational studies.

The Propensity Score (PS) is defined as the subject’s probability of receiving
a specific treatment conditional on the observed covariates[1]. After
stratification by its PS, treated and untreated patients are matched by their
PS with the most similar individuals of the opposite group. It leads to a more
similar distribution of baseline characteristics between treated and untreated
subjects, and it have been demonstrated that this method reduces the
Treatment-selection bias[5, 4].

Once two comparable groups have been obtained, researchers treat this studies
like more similar to randomised studies (although it does not substitute this
randomised studies), and use them as a reasonable alternative for
observational studies[3]. In this sense, it is thought that because PSM
controls the possible treatment-selection bias, it would be possible to
directly measure the effects in both matched groups, and thus, it may be
better for observational studies than other multivariate adjustment methods.

One important concern is the influence of the not insignificant loss of non
matched individuals that may be seen in some works using this PSM method[6],
and who are not used for posterior analysis. Because we need to eliminate
enough unmatched individuals to guarantee in some way the Treatment -
Selection Bias correction, it is not possible to know if the elimination of
this unmatched individuals can cause some loss of information that in other
ways would be analysed, and therefore lead us to a non controlled bias. In
this sense other authors have reported the over-employment of this technique
and its potential implications in potential biases[11, 10].

Moreover, although we are controlling the Treatment - Selection Bias, after
matching we are directly measuring the effect in both matched groups. In this
way, its behaviour compared with a multivariate adjustment method, which is
widely used to control groups for other possible confounders, has never been
tested. Given that there is a loss of individuals prior to the analysis, we do
not know if this loss of information can variate the results obtained with the
multivariate analysis.

Because of that, we wanted to test the behaviour of PSM in different
situations, compared with a multivariate analysis based on General Linear
Models (GLM), in the estimation of treatment effects. For this purpose we
developed a theoretical Montecarlo Method of treatment effects in which we
applied the PSM and a multivariate analysis based on a GLM to compare their
ability to estimate the Real Treatment Effect (RTE) in each situation.

##  2 Methods

###  2.1 Theoretical Multivariate model

Suppose $Y_{z}(x)$ the patient $z$ probability for a certain event. Its
probability may be influenced by a series of independent variables $A_{zj}$
and $B_{zk}$ each one of them with a concrete weight in this patient
probability prediction $q_{j}$ and $s_{k}$ respectively. $A_{zj}$ variables
may be related to the received treatment and $B_{zk}$ variables are supposed
to be independent of the received treatment. It may be also influenced by the
treatment status $X_{z}$ of the patient, which may confer some protection $t$
against the event under study.

For each patient there may be also some unmeasured influence $\epsilon_{z}$ in
its event probability, which may vary from patient to patient and may be
because some unknown or non measured variables. We consider it as a random
variable, which may be based in a normal distribution:

$\epsilon_{z}\sim\mathcal{N}(0,1)$ (1)

Each patient may present some different characteristics. A part from this
$A_{zj}$ and $B_{zk}$ characteristics that may predispose in some way the
probability for the event under study, it may present other variables
unrelated with the event under study. Some of them may predispose to receive
the treatment under study $C_{l}$ and others $D_{m}$ may be unrelated to
either patient outcomes or treatment predisposition.

For a concrete patient the logit of the probability for a certain event may be
predicted by the formula:

$Y_{z}(x)=X_{z}t+A_{z1}q_{1}+...+A_{zj}q_{j}+B_{z1}s_{1}+...+B_{zk}s_{k}+\epsilon_{z}$
(2)

And thus the theoretical event probability for a $z$ given patient will be:

$\rho_{z}=\frac{e^{Y_{z}(x)}}{1+e^{Y_{z}(x)}}$ (3)

###  2.2 Monte Carlo Simulations

Once the theoretical model is build, Monte Carlo Simulations are made to
construct groups for a posterior analysis. Simulated experiments under this
conditions are made, and an event status $\mathbb{E}_{z}$ is assigned to each
patient $z$ in each simulated experiment, based on a binomial distribution
with $\rho_{z}$ probability.

$\mathbb{E}_{z}\sim\mathcal{B}(n,\rho)=\mathcal{B}(1,\rho_{z})$ (4)

The theoretical (real) treatment effect is modified after each group of
simulations, ranging from a $0$ (null) effect to a $5$ fold event reduction,
in $0.1$ intervals.

###  2.3 Unadjusted Model

The Odds Ratio for the event prevention under Treatment status is calculated
with a univariable General Linear Model (GLM). For each experiment, with the
complete matrix of events $Y$ and treatment $X$ status, a GLM is build to
estimate the unadjusted estimated risk prevention effect for the treatment
$t_{ua}$:

$Y=Xt_{ua}+U$ (5)

From this built model, the estimated Odds Ratio for the RTE $OR_{ua}$ is
calculated:

$OR_{ua}=e^{t_{ua}}$ (6)

Since this measured Odds Ratio does not take in account other variables, it
will be named the unadjusted Odds Ratio and it will be considered the
reference for the improvement in the RTE estimation.

###  2.4 Multivariate Regression Model

For control purposes a Multivariate Regression Model (MRM) is build for each
experiment with all the variables under analysis. In each experiment the
complete matrix of events $Y$, treatment status $X$ and analysed variables
$A$, $B$, $C$ and $D$ are used to build the MRM and estimate the multivariate
adjusted event reduction of the treatment $t_{multi}$:

$Y=Xt_{multi}+Aq_{multi}+Bs_{multi}+Cu_{multi}+Dv_{multi}+U$ (7)

From this built model, the estimated Odds Ratio for the RTE (Multivariate Odds
Ratio, $OR_{multi}$) is calculated:

$OR_{multi}=e^{t_{multi}}$ (8)

It will be used as the gold standar for the estimation of the RTE.

###  2.5 Propensity Score Matching Model

PSM is used to estimate the Treatment Effect. As described early, the PS is
build based on a MRM designed to estimate each patient’s predisposition to
receive the treatment under investigation. In each patient the PS is
calculated for posterior matching between treated and untreated individuals.
For the matching the nearest method is used, with a caliper of 0.2.

Once matched is done, the treatment effect is estimated based on a GLM. For
each experiment, the matrix of matched individuals with the events $Y_{match}$
and treatment $X_{match}$ status are used to build a GLM to estimate the
estimated risk prevention effect for the treatment $t_{match}$:

$Y_{match}=X_{match}t_{match}+U$ (9)

From this built model, the estimated Odds Ratio for the RTE $OR_{match}$ is
calculated:

$OR_{match}=e^{t_{match}}$ (10)

###  2.6 Statistical Analysis

For each treatment effect situation, RTE Estimation is measured with each one
of the three described methods. This RTE estimation is lately compared with
the RTE to calculate the inaccurate of the RTE Estimation (Relative RTE
Estimation Error). Al variables are expressed as mean +/- 95% Confidence
Interval. The comparison of the RTE Estimation Error between the three
described methods is done with a multivariate analysis of variance (MANOVA
test).

For the MRM and the PSM Model, the RTE Estimation Error is compared with the
Unadjusted Model, to calculate the Relative RTEE Estimation Error Reduction
for each one. The comparison of this Relative RTEE Estimation Error Reduction
between MRM and PSM Model is done with a paired T Test.

For the Unadjusted Model and the PSM Model the Treatment-Selection Bias is
calculated with a Pearson’s chi-squared test to evaluate the homogeneity
between groups. The comparison of the Treatment - Selection bias between both
models is done with a paired T Test comparing the the chi-squared test
statistic of both models, and the reduction in the Treatment - Selection bias
with the PSM Model is expressed as mean +/- 95% Confidence Interval.

For the PSM Model the percentage of excluded patients in each analysis will be
also analysed. This will be expressed as the mean of the percentage exclusions
in each analysis and the 95% Confidence Interval of the percentage exclusions
in each analysis.

###  2.7 Analysis Software

For the data generation with the Monte Carlo Simulations and the posterior
analysis, the open software R [8] and the R library MatchIt [9] were used. All
wrote code for this purpose is available through the PropensityScoreReview
repository [7].

##  3 Results

###  3.1 Monte Carlo Simulations

A total of $5\cdot 10^{6}$ Montecarlo Simulations are done. There are 50
blocks of experiments, each one with a different real treatment effect. In
each block of experiments, each experiment is simulated with 500 individuals
and later repeated 200 times for each real treatment effect.

Figure 1: Comparison of the Propensity Score Matching Model Treatment-Selection Bias reduction, Percentage of Matched Individuals and Relative Real Treatment Effect Estimation Reduction with the Multivariate Model Relative Real Treatment Effect Estimation Reduction. | T-S Bias |  |
---|---|---|---
Real Treatment Effect |  Unadjusted Model |  Matched Model |  p Value |  T-S Bias Reduction |  Excluded Patients (%)
5 |  220.7 (217.82, 223.57) |  3.631 (3.479, 3.783) |  <0.001 |  0.983 (0.983, 0.984) |  70.66 (70.2, 71.12)
4 |  219.09 (216.36, 221.83) |  3.599 (3.446, 3.751) |  <0.001 |  0.983 (0.983, 0.984) |  70.43 (69.98, 70.88
3 |  219.1 (216.29, 221.90) |  3.698 (3.536, 3.861) |  <0.001 |  0.983 (0.982, 0.984) |  70.55 (70.1, 71)
2 |  217.6 (214.80, 220.33) |  3.721 (3.576, 3.866) |  <0.001 |  0.983 (0.982, 0.983) |  70.12 (69.68, 70.56)
1 |  216.98 (214.31, 219.66) |  3.738 (3.603, 3.873) |  <0.001 |  0.982 (0.982, 0.983) |  70.19 (69.72, 70.65)
0 |  219.47 (216.85, 222.09) |  3.684 (3.532, 3.835) |  <0.001 |  0.983 (0.982, 0.984) |  70.21 (69.77, 70.65)
Table 1: Treatment - Selection Bias for the Unadjusted Model and the
Propensity Score Matching Model. Treatment - Selection Bias Reduction for the
Propensity Score Matching Model. Percentage of excluded patients for the
Propensity Score Matching Model. T-S Bias: Treatment - Selection Bias.

###  3.2 Treatment - Selection Bias reduction with Propensity Score Matching

The PSM Model significantly reduces the Treatment - Selection Bias in all
scenarios. As seen in table 1 and Figure 2, there is a Relative Treatment -
Selection Bias reduction of about 0.98 in all scenarios. The main problem of
this model is the important number of excluded patients (there is only a 70%
of patients that are included for the analysis).

###  3.3 Real Treatment Effect Estimation and Relative Real Treatment Effect
Estimation Error Reduction with Propensity Score Matching Model and
Multivariate Regression Model

As it can be seen in figure 2 and 3, and table 2, the PSM Model and the MRM
significantly estimate a more accurate RTE than the Unadjusted Model. This PSM
Model and RMM present a significantly reduced Relative RTE Estimated Error,
compared with the Unadjusted Model.

The MRM also present a Relative RTE Estimated Error significantly lower than
the PSM Model, which leads to a significantly increased Relative RTE Estimated
Error Reduction compared with the PSM Model.

Figure 2: Comparison of Real Treatment Effect and Real Treatment Effect
Estimation from the Unadjusted Model, Multivariate Regression Model and
Propensity Score Matching Model. Figure 3: Comparison of Relative Real
Treatment Effect Estimation Error with the Unadjusted Model, Multivariate
Model and Propensity Score Matching Model.

| Estimated Treatment Effect | Relative RTE Estimated Error | RTE Estimated Error Reduction
---|---|---|---
Real Treatment Effect |  Unadjusted Model |  Matched Model |  Multivariate Model |  p Value |  Unadjusted Model |  Matched Model |  Multivariate Model |  p Value |  Matched Model |  Multivariate Model |  p Value
5 |  -2.225 (-2.254, -2.195) |  -2.922 (-2.984, -2.86) |  -4.521 (-4.594, -4.448) |  <0.001 |  0.555 (0.549, 0.561) |  0.416 (0.403, 0.428) |  0.096 (0.082, 0.11) |  <0.001 |  0.139 (0.128, 0.151) |  0.459 (0.446, 0.473) |  <0.001
4 |  -1.63 (-1.659, -1.601) |  -2.3 (-2.355, -2.245) |  -3.597 (-3.659, -3.536) |  <0.001 |  0.592 (0.585, 0.6) |  0.425 (0.411, 0.439) |  0.101 (0.085, 0.116) |  <0.001 |  0.167 (0.154, 0.181) |  0.492 (0.478, 0.506) |  <0.001
3 |  -1.059 (-1.088, -1.030) |  -1.707 (-1.759, -1.655) |  -2.676 (-2.736, -2.616) |  <0.001 |  0.647 (0.637, 0.657) |  0.431 (0.41, 0.448) |  0.108 (0.088, 0.128) |  <0.001 |  0.216 (0.2, 0.232) |  0.539 (0.522, 0.556) |  <0.001
2 |  -0.504 (-0.536, -0.472) |  -1.201 (-1.257, -1.144) |  -1.852 (-1.91, -1.793) |  <0.001 |  0.748 (0.732, 0.76) |  0.4 (0.371, 0.428) |  0.074 (0.045, 0.103) |  <0.001 |  0.348 (0.324, 0.373) |  0.674 (0.649, 0.698) |  <0.001
1 |  0.129 (-0.096, 0.161) |  -0.604 (-0.661, -0.548) |  -0.927 (-0.99, -0.864) |  <0.001 |  1.129 (1.096, 1.161) |  0.396 (0.339, 0.452) |  0.073 (0.01, 0.136) |  <0.001 |  0.733 (0.684, 0.783) |  1.056 (1.003, 1.109) |  <0.001
0 |  0.743 (-0.708, 0.777) |  0.026 (-0.032, 0.085) |  -0.078 (-0.139, -0.017) |  <0.001 |  8.426 (8.08, 8.772) |  1.264 (0.676, 1.853) |  0.22 (-0.394, 0.83) |  <0.001 |  7.162 (6.622, 7.701) |  8.206 (7.699, 8.714) |  <0.001
Table 2: Real Treatment Effect Estimation for the Unadjusted Multivariate and
Propensity Score Matching models. Relative Real Treatment Effect Estimation
Error for the Unadjusted Multivariate and Propensity Score Matching models.
Relative Real Treatment Effect Estimation Error Reduction for the Multivariate
and Propensity Score Matching models. RTE: Real Treatment Effect Estimation.
T-S Bias: Treatment - Selection Bias.

##  4 Discussion

PSM has been widely used in different subjects for Treatment - Selection Bias
reduction. As we shown in this present work, and as it was seen in previous
works[3], with this method we can correct the Treatment - Selection Bias
properly, obtaining two comparable groups, so we can thus directly measure the
effect under investigation. In this sense, in our work the Treatment -
Selection Bias practically disappears with the PSM (Treatment - Selection Bias
reduction of 0.982 - 0.983 among all scenarios).

As we mentioned earlier, our main concern about the PSM method is the
percentage of unmatched individuals and its possible influence in the
posetrior estimation of RTE. In our work results there is a not insignificant
reduction of the sample, with a percentage of analysed individuals of 70.12 to
70.66% from the total of individuals under investigation. And what we can not
know is if this important reduction in the population for analysis may lead to
another non controlled bias, since the excluded patients may have some
characteristics that we are obviating for posterior analysis.

Other widely used methods, such as MRM, do not deal with two comparable
populations, but instead, they weigh the different variables under study.
Despite dealing with non-comparable groups, thanks to this weighting of the
analysed variables, it can solve in a different way the problem of the
Treatment - Selection Bias. Also, since no individual is eliminated from the
analysis, there is no loss of information, reducing other potential biases
that may appear in the PSM method.

In our present work, although both (PSM and MRM) reduce the Relative RTE
Estimation Error, this reduction is better with the MRM, compared with the
PSM. This best performance of the MRM may confirm our previous preoccupation
about the possible influence of the sample reduction on the posterior
estimation of the RTE.

Since there may be a significant reduction of the sample under analysis when
we are using a PSM method, we have to take it in account before accepting the
obtained results, specially when this reduction is important. Other
multivariate methods should always be done, in addition to the PSM analysis,
and both results compared, in order to seek for a possible uncontrolled bias.
If a significant difference is obtained between both analysis, we have to
suspect a possible bias derived from the sample reduction, once the matching
has been done.

## References

  * [1] Paul R. Rosenbaum and Donald B. Rubin. The Central Role of the Propensity Score in Observational Studies for Causal Effects. Biometrika. 1983.
  * [2] Peter C. Austin. A critical appraisal of propensity-score matching in the medical literature between 1996 and 2003 Statistics in Medicine. 2008.
  * [3] Peter C. Austin. Some Methods of Propensity-Score Matching had Superior Performance to Others: Results of an Empirical Investigation and Monte Carlo simulations. Biometrical Journal. 2009.
  * [4] Francesco Innocenti, Math J.J.M. Candel, Frans E.S. Tan and Gerard J.P van Breukelen. Relative efficiencies of two-stage sampling schemes for mean estimation in multilevel populations when cluster size is informative. Statistics in Medicine. 2018.
  * [5] Fan Li, Alan M. Zaslavsky and Mary Beth Landrum. Propensity score weighting with multilevel data. Statistics in Medicine. 2013.
  * [6] You, Seng Chan, An, Min Ho, Yoon, Dukyong, Ban, Ga-Young, Yang, Pil-Sung, Yu, Hee Tae, Park, Rae Woong, Joung, Boyoung. Rate control and clinical outcomes in patients with atrial fibrillation and obstructive lung disease. Heart Rhythm. 2018.
  * [7] García Iglesias, Daniel. Propensity Score Review. GitHub repository. 2019.
  * [8] R Core Team. Propensity Score Review. R Foundation for Statistical Computing. 2018.
  * [9] Daniel E. Ho, Kosuke Imai, Gary King, Elizabeth A. Stuart MatchIt: Nonparametric Preprocessing for Parametric Causal Inference. Journal of Statistical Software. 2011.
  * [10] Timothy L. McMurry, Yinin Hu, Eugene H. Blackstone, Benjamin D. Kozower. Propensity scores: Methods, considerations, and applications in the Journal of Thoracic and Cardiovascular Surgery. The Journal of Thoracic and Cardiovascular Surgery. 2015.
  * [11] Gary King, Richard Nielsen. Why Propensity Scores Should Not Be Used for Matching. Political Analysis. Forthcoming.


11institutetext: The University of Texas at Austin, TX 78712, USA
22institutetext: US Army Research Laboratory
33institutetext: Sony AI

# RAIL: A modular framework for Reinforcement-learning-based Adversarial
Imitation Learning††thanks: This work has taken place in the Learning Agents
Research Group (LARG) at UT Austin. LARG research is supported in part by NSF
(CPS-1739964, IIS-1724157, NRI-1925082), ONR (N00014-18-2243), FLI (RFP2-000),
ARO (W911NF-19-2-0333), DARPA, Lockheed Martin, GM, and Bosch. Peter Stone
serves as the Executive Director of Sony AI America and receives financial
compensation for this work. The terms of this arrangement have been reviewed
and approved by the University of Texas at Austin in accordance with its
policy on objectivity in research.

Eddy Hudson 11 0000-0002-8350-3919 Garrett Warnell 11 2 2 0000-0003-3846-8787
Peter Stone 11 3 3 0000-0002-6795-420X

###### Abstract

While Adversarial Imitation Learning (AIL) algorithms have recently led to
state-of-the-art results on various imitation learning benchmarks, it is
unclear as to what impact various design decisions have on performance. To
this end, we present here an organizing, modular framework called
Reinforcement-learning-based Adversarial Imitation Learning (RAIL) that
encompasses and generalizes a popular subclass of existing AIL approaches.
Using the view espoused by RAIL, we create two new IfO (Imitation from
Observation) algorithms, which we term SAIfO: SAC-based Adversarial Imitation
from Observation and SILEM (Skeletal Feature Compensation for Imitation
Learning with Embodiment Mismatch). We go into greater depth about SILEM in a
separate technical report [11]. In this paper, we focus on SAIfO, evaluating
it on a suite of locomotion tasks from OpenAI Gym, and showing that it
outperforms contemporaneous RAIL algorithms that perform IfO.

###### Keywords:

Reinforcement Learning Imitation Learning Adversarial Imitation Learning

##  1 Introduction

In the past decade, the field of deep learning has been punctuated by
revolutionary results in computer vision, natural language processing (NLP),
and reinforcement learning (RL). Key to this explosive growth has been an
implicit modularization that has allowed researchers to work on improving
individual modules independently and in parallel. For example, in computer
vision, an approach to tackle the ImageNet challenge [16] can be said to
comprise the following two main modules among a host of other components: an
optimization algorithm, and the deep network architecture. Research into
optimization algorithms has yielded algorithms such as AdaGrad [4] and Adam
[12], while research into deep network architectures has resulted in
architectures such as convolutional networks and ResNet [9]. Progress in NLP
can similarly be characterized by modules corresponding to optimization
algorithm and network architecture. Research into network architectures for
NLP has been taking place independent of the optimization algorithm used, and
has resulted in success stories from seq2seq [19] to the Transformer [21].

Recently, there has been rapid progress on imitation learning (IL), especially
with respect to a class of new algorithms termed adversarial imitation
learning (AIL). Starting from GAIL [10] and GAIfO [20], to DAC [13] and OPOLO
[22], the sample complexity of AIL approaches have been rapidly declining.
However, progress in this space has been ad hoc: AIL research lacks the type
of organizing and modular characterization that has led to the steady and
substantial improvement enjoyed by the other communities mentioned above.

In this work, we aim to provide a modular characterization for a sub-class of
AIL algorithms that employs RL. We term this class of algorithms
Reinforcement-learning-based Adversarial Imitation Learning (RAIL). We
characterize RAIL techniques using a modular framework, with modules
corresponding to the RL backbone and the input format to the discriminator. As
evidence that the modularization of RAIL can accelerate progress in it, we
explore the resulting design space and discover a new RAIL variant for
imitation from observation that we call SAC-based Adversarial Imitation from
Observation (SAIfO), where SAC [7] is a recently proposed RL algorithm. We
evaluate SAIfO on a suite of locomotion tasks from OpenAI Gym [3], and show
that it outperforms recent IfO algorithms that fall under the RAIL umbrella.

##  2 Background

The ultimate goal in imitation learning is to learn a controller that solves a
sequential decision making problem. Such problems are typically formulated in
the context of a Markov decision process (MDP), i.e., a tuple
$\mathcal{M}=<\mathcal{S},\mathcal{A},T,R,\gamma>$, where $\mathcal{S}$
denotes an agent’s state space, $\mathcal{A}$ denotes the agent’s action
space, $T:\mathcal{S}\times\mathcal{A}\rightarrow\Delta(\mathcal{S})$ denotes
the environment model which maps state-action pairs to a distribution over the
agent’s next state,
$R:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\rightarrow\mathbb{R}$ is a
reward function that provides a scalar-valued reward signal for state-action-
next-state tuples, and $\gamma\in[0,1]$ is a discount factor that specifies
how the agent should weight short- vs. long-term rewards. Solutions to
sequential decision making problems are often specified by reactive policies
$\pi:\mathcal{S}\rightarrow\Delta(\mathcal{A})$, which specify agent behavior
by providing a mapping from the agent’s current state to a distribution over
the actions it can take. Machine learning solutions to problems described by
an MDP typically search for policies that can maximize the agent’s expected
sum of future rewards.

The IL problem is typically formulated using an MDP without a specified reward
function, i.e., $\mathcal{M}\setminus R$. Instead of reward, the agent is
provided with demonstration trajectories—typically assumed to have been
generated by an expert—that specifies the desired behavior, i.e.,
$\tau_{E}=(s_{0},a_{0},s_{1},a_{1},...)$. Imitation from observation (IfO) is
a sub-problem of IL in which the agent does not have access to the actions
taken during the demonstration trajectories, i.e.,
$\tau_{E}=(s_{0},s_{1},...)$. Techniques designed to solve the IL problem seek
to use observed demonstrations to find policies that an imitating agent can
use to imitate the demonstrator.

Adversarial imitation learning (AIL) is a particular way to perform IL that
has recently come to the fore (Figure 1). AIL leverages the adversarial
training technique popularized by GANs [6], in that both involve the same min-
max game with discriminators and generator networks. The discriminator, $D$,
is trained to distinguish between the demonstration trajectories and
trajectories generated by the imitator. In particular, the goal of updating
$D$ is to drive $\mathbb{E}_{o\sim\tau_{E}}[D(o)]$ toward $1$ and
$\mathbb{E}_{o\sim\tau}[D(o)]$ toward $0$, where $o$ is a segment of the
trajectory, $\tau$ represents trajectories recently generated by the imitator,
and $\tau_{E}$ is a dataset of demonstration trajectories. In the seminal AIL
algorithm GAIL [10] and more recent AIL algorithms ASAF [2] and ValueDICE
[14], $o=(s_{t},a_{t})$, whereas in GAIfO [20] and OPOLO [22],
$o=(s_{t},s_{t+1})$. The generator, which in AIL algorithms is the imitator’s
policy $\pi$, is trained to induce behavior that elicits large output from
$D$, i.e., to “fool” $D$ into thinking that the imitator’s trajectories came
from the demonstrator. By iteratively updating $D$ and $\pi$ as described, AIL
approaches are able to find imitator policies that successfully mimic the
demonstrated behavior.

##  3 RAIL: Reinforcement-learning-based Adversarial Imitation Learning

In this work, we focus on a specific slice of AIL that we term RAIL:
Reinforcement-learning-based Adversarial Imitation Learning. As suggested by
the name, we include in RAIL all AIL algorithms that optimize the policy by
using an RL algorithm where the reward is given by the discriminator. Thus,
GAIL and GAIfO are RAIL algorithms, as are OPOLO and DAC [13]. On the other
hand, ValueDICE is not a RAIL algorithm since it optimizes the policy by
backpropagating directly into the discriminator, and ASAF is not a RAIL
algorithm since it optimizes the policy by simply copying over the weight
values from the discriminator.

We characterize RAIL techniques using a modular framework consisting of the
following two modules (color coded according to the corresponding design
decisions in Figure 1):

  * •

the RL backbone

  * •

the discriminator’s input representation

sample trajectorysegmentssample trajectorysegmentsuse Discriminatorto optimize
policyDemonstrationTrajectoriesLearnerPolicyDiscriminator Figure 1: Schematic
diagram explaining the work-flow in AIL algorithms

The first module is the RL algorithm used by a RAIL algorithm to optimize the
learner’s policy. Early RAIL algorithms like GAIL or GAIfO, used on-policy RL
algorithms such as TRPO [17] or PPO [18]. More recent RAIL algorithms OPOLO
and DAC [13] radically reduced their sample complexity by using off-policy RL
algorithms such as TD3 [5] or AlgaeDICE [15].

The second module is the input representation used by the discriminator. This
module controls whether the discriminator gets access to state-action pairs
$(s_{t},a_{t})$, state-next state pairs $(s_{t},s_{t+1})$, or any arbitrary
subsequence of the trajectory. For example, $(s_{t},s_{t+3})$. It is worth
pointing out here that most RAIL algorithms either use $(s_{t},a_{t})$, or
$(s_{t},s_{t+1})$ as the input to the discriminator. This last module that we
just introduced is an especially powerful one as it allows an RL researcher to
seamlessly create an IfO algorithm out of any RAIL algorithm by changing the
input representation to the discriminator. For example, by changing
$(s_{t},a_{t})$ to $(s_{t},s_{t+1})$, as was done to create GAIfO from GAIL
(Table 1).

Table 1: Recent RAIL algorithms arranged according to our modular characterization of RAIL. Off policy algorithms are denoted by green-colored names. $T$ is the affine transform that is learned by SILEM. RAIL Algorithm | RL Backbone | Discriminator Input
---|---|---
GAIL | TRPO [17] | $(s_{t},a_{t})$
DAC | TD3 [5] | $(s_{t},a_{t})$
GAIfO | TRPO [17] | $(s_{t},s_{t+1})$
OPOLO | AlgaeDICE [15] | $(s_{t},s_{t+1})$
DACfO [22] | TD3 [5] | $(s_{t},s_{t+1})$
SAIfO | SAC [8] | $(s_{t},s_{t+1})$
SILEM [11] | PPO [18] | $T(s_{t},s_{t+1},s_{t+2},s_{t+3})$

Apart from the two modules defined above, RAIL algorithms also typically
incorporate miscellaneous tricks to improve performance such as absorbing
states in DAC, and regularization with a dynamics model in OPOLO. We disregard
them in our modular characterization due to questions that have arisen
regarding their value. The authors of the OPOLO paper observe that the
regularization only adds marginal improvements in performance [22], and our
own experience has indicated that absorbing states are not necessary for state
of the art performance.

##  4 SAIfO

Given our modular characterization of RAIL algorithms, a natural way to try to
obtain a state-of-the-art RAIL algorithm would be to try to maximize the
performance of each module independently of one another. Inspired by a recent
IfO algorithm, OPOLO [22], we develop a state-of-the-art IfO algorithm by
maximizing the performance of the RL backbone. We could also attempt to
improve performance by choosing a different form of input to the discriminator
that does not include action information, but we leave that to future work.

We maximize performance of the RL backbone by choosing a state-of-the-art off-
policy RL algorithm, Soft Actor Critic (SAC) [8], and we term the resulting
RAIL algorithm SAC-based Adversarial Imitation from Observation (SAIfO).

##  5 Experiments and Results

We run experiments on locomotion tasks provided by OpenAI gym in the Mujoco
environment. Our implementation of SAIfO is built on the SAC implementation
provided by Spinning Up OpenAI [1]. Every other algorithm we use is based on
the implementation provided by the authors of OPOLO [22]. The appendix
contains details on the hyperparameters we use in each algorithm along with
the neural network architectures we use.

Our main goal with the experiments is to show that our proposed RAIL
algorithm, SAIfO, outperforms other RAIL algorithms that perform IfO. We
compare SAIfO against GAIfO, OPOLO and DACfO, and find that it indeed does
better than the three prior RAIL algorithms (Figure 2). Note that our OPOLO
and DACfO results are much better than those reported by Zhu et al. [22] since
we strongly optimized all three of our baselines to provide a fair comparison
with SAIfO. Refer to the appendix for specific details on how we optimized
their implementations of GAIfO, OPOLO and DACfO.

(a) Legend

(b) HalfCheetah-v2 (c) Hopper-v2

Figure 2: Results comparing SAIfO to RAIL algorithms that perform IfO. The
horizontal black lines show expert level performance in each domain. The
$x$-axis shows the number of interactions with the environment. Results shown
are the mean, minimum and maximum over 10 independent trials.

##  6 SILEM

SAIfO leverages our modular characterization of RAIL to merely accelerate IfO.
In SILEM (Skeletal Feature Compensation for Imitation Learning with Embodiment
Mismatch) [11], we go a step further and enable IfO to work even in the
presence of embodiment mismatch. In SILEM, we process the discriminator’s
input using a learnable affine transform before inputting it to the
discriminator. We train the affine transform to compensate for embodiment
mismatch between the learner’s trajectories and demonstration trajectories. In
Hudson et. al. [11], we evaluate SILEM on a suite of challenging environments
involving learning from human demonstrations, and we show that SILEM far
outperforms the alternatives.

##  7 Conclusion

In this work, we introduced a modular characterization of a sub-area in
Adversarial Imitation Learning that we term RAIL: Reinforcement-learning-based
Adversarial Imitation Learning. We describe the modules apparent in RAIL and
show how recent AIL algorithms fit within the modular framework. Using the
modularity of the framework as a basis, we develop a new off-policy IfO
algorithm (SAIfO) that is faster than prior RAIL algorithms that perform IfO,
thus also showing the benefits of the modular characterization.

While developing SAIfO, we only focused on improving performance of the RL
backbone. Exploring different formats of the discriminator input is an
interesting avenue of future work. For example, rather than using
$(s_{t},s_{t+1})$ as an input, using $(s_{t},s_{t+4})$ or
$(s_{t},s_{t+1}-s_{t})$. We also wish to perform more experiments to figure
out why SAC makes a much better RL backbone than the alternatives, TD3 and
AlgaeDICE. We suspect that the _stochastic update_ afforded by the kernel
trick plays an important role [8].

Lastly, we also showcased how our modular characterization of RAIL even
allowed us to develop an algorithm capable of performing IfO in the presence
of embodiment mismatch. We did this by outlining a recent algorithm we created
called SILEM [11].

## References

  * [1] Achiam, J.: Spinning Up in Deep Reinforcement Learning (2018)
  * [2] Barde, P., Roy, J., Jeon, W., Pineau, J., Pal, C., Nowrouzezahrai, D.: Adversarial soft advantage fitting: Imitation learning without policy optimization. In: Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual (2020), https://proceedings.neurips.cc/paper/2020/hash/9161ab7a1b61012c4c303f10b4c16b2c-Abstract.html
  * [3] Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., Zaremba, W.: Openai gym (2016)
  * [4] Duchi, J., Hazan, E., Singer, Y.: Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research 12(7) (2011)
  * [5] Fujimoto, S., van Hoof, H., Meger, D.: Addressing function approximation error in actor-critic methods. In: Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018. Proceedings of Machine Learning Research, vol. 80, pp. 1582–1591. PMLR (2018), http://proceedings.mlr.press/v80/fujimoto18a.html
  * [6] Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A.C., Bengio, Y.: Generative adversarial networks. CoRR abs/1406.2661 (2014), http://arxiv.org/abs/1406.2661
  * [7] Haarnoja, T., Zhou, A., Abbeel, P., Levine, S.: Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In: International Conference on Machine Learning. pp. 1861–1870. PMLR (2018)
  * [8] Haarnoja, T., Zhou, A., Abbeel, P., Levine, S.: Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In: Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018. Proceedings of Machine Learning Research, vol. 80, pp. 1856–1865. PMLR (2018), http://proceedings.mlr.press/v80/haarnoja18b.html
  * [9] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)
  * [10] Ho, J., Ermon, S.: Generative adversarial imitation learning. In: Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain. pp. 4565–4573 (2016), https://proceedings.neurips.cc/paper/2016/hash/cc7e2b878868cbae992d1fb743995d8f-Abstract.html
  * [11] Hudson, E., Warnell, G., Torabi, F., Stone, P.: Skeletal feature compensation for imitation learning with embodiment mismatch (2021)
  * [12] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings (2015), http://arxiv.org/abs/1412.6980
  * [13] Kostrikov, I., Agrawal, K.K., Dwibedi, D., Levine, S., Tompson, J.: Discriminator-actor-critic: Addressing sample inefficiency and reward bias in adversarial imitation learning. In: 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net (2019), https://openreview.net/forum?id=Hk4fpoA5Km
  * [14] Kostrikov, I., Nachum, O., Tompson, J.: Imitation learning via off-policy distribution matching. In: 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net (2020), https://openreview.net/forum?id=Hyg-JC4FDr
  * [15] Nachum, O., Dai, B., Kostrikov, I., Chow, Y., Li, L., Schuurmans, D.: Algaedice: Policy gradient from arbitrary experience. CoRR abs/1912.02074 (2019), http://arxiv.org/abs/1912.02074
  * [16] Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M.S., Berg, A.C., Li, F.: Imagenet large scale visual recognition challenge. CoRR abs/1409.0575 (2014), http://arxiv.org/abs/1409.0575
  * [17] Schulman, J., Levine, S., Abbeel, P., Jordan, M.I., Moritz, P.: Trust region policy optimization. In: Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015. JMLR Workshop and Conference Proceedings, vol. 37, pp. 1889–1897. JMLR.org (2015), http://proceedings.mlr.press/v37/schulman15.html
  * [18] Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O.: Proximal policy optimization algorithms. CoRR abs/1707.06347 (2017), http://arxiv.org/abs/1707.06347
  * [19] Sutskever, I., Vinyals, O., Le, Q.V.: Sequence to sequence learning with neural networks. In: Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada. pp. 3104–3112 (2014), https://proceedings.neurips.cc/paper/2014/hash/a14ac55a4f27472c5d894ec1c3c743d2-Abstract.html
  * [20] Torabi, F., Warnell, G., Stone, P.: Generative adversarial imitation from observation. CoRR abs/1807.06158 (2018), http://arxiv.org/abs/1807.06158
  * [21] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need. In: Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA. pp. 5998–6008 (2017), https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html
  * [22] Zhu, Z., Lin, K., Dai, B., Zhou, J.: Off-policy imitation learning from observations. In: Advances in Neural Information Processing Systems. vol. 33, pp. 12402–12413. Curran Associates, Inc. (2020), https://proceedings.neurips.cc/paper/2020/file/92977ae4d2ba21425a59afb269c2a14e-Paper.pdf

##  8 Appendix

###  8.1 Ensuring a fair comparison with baselines

We spent a considerable amount of time optimizing our baselines to ensure a
fair comparison. We first added a _warmup_ phase to the code provided by the
authors of OPOLO where we trained the discriminator and Q function for a
configurable number of iterations before using them to train the policy with
DACfO or OPOLO. We also applied multiple grid searches to find the best
configuration of hyperparameters for our three baselines. We found
hyperparameter configurations that resulted in a better performance than the
default hyperparameters provided by the OPOLO authors.

###  8.2 Hyperparameters used for each algorithm

For every experiment, we used a grid search to find the best set of hyper-
parameters. Specifically, for every configuration of hyper-parameters in the
experiment, we ran 10 independent trials. Within each trial, we trained a
policy, measured the average reward obtained by that policy over the last 10
tests (with each test run in intervals of 4000 interactions with the
environment), and assigned the resulting number as the score for that
particular trial. The score for each configuration of hyper-parameters is the
average score over the 10 trials corresponding to that configuration. The best
configuration of hyper-parameters is then that which maximizes this score. We
usually ran multiple successive grid searches to more strongly optimize
performance instead of just running one single grid search for experiment.

####  8.2.1 SAIfO

We list below the hyperparameter values we used for our implementation of
SAIfO. We only list hyperparameter values that we either introduced or changed
from the default value.

Table 2: Hyperparameter settings for HalfCheetah-v2 | Hyperparameter
---
name
Value
Discriminator learning rate | $6\times 10^{-5}$
Entropy coefficient for discriminator | 0
Batch size (SAC) | 400
No. steps of warm up for Q function | 5000
No. steps of warm up for Discriminator | 500
Update discriminator every X iterations | 100
SAC alpha | 0.02
Gradient penalty for Discriminator | 0.0006
Update policy X times every Y iterations | 674, 100
Table 3: Hyperparameter settings for Hopper-v2 | Hyperparameter
---
name
Value
Discriminator learning rate | $1\times 10^{-4}$
Entropy coefficient for discriminator | 0
Batch size (SAC) | 400
No. steps of warm up for Q function | 5000
No. steps of warm up for Discriminator | 1000
Update discriminator every X iterations | 10
SAC alpha | 0.15
Gradient penalty for Discriminator | 0.002
Update policy X times every Y iterations | 674, 100

####  8.2.2 DACfO

We list below the hyperparameter values we used for the implementation of
DACfO by the OPOLO authors. We only list hyperparameter values that we either
introduced or changed from the default value.

Table 4: Hyperparameter settings for HalfCheetah-v2 | Hyperparameter
---
name
Value
No. steps of warm up for Q function | 5000
No. steps of warm up for Discriminator | 200
Update discriminator every X iterations | 15
Update policy X times every Y iterations | 7000, 1000
Table 5: Hyperparameter settings for Hopper-v2 | Hyperparameter
---
name
Value
No. steps of warm up for Q function | 5000
No. steps of warm up for Discriminator | 100
Update discriminator every X iterations | 300
Update policy X times every Y iterations | 700, 100
Batch size | 50

####  8.2.3 OPOLO

We list below the hyperparameter values we used for the implementation of
OPOLO by the OPOLO authors. We only list hyperparameter values that we either
introduced or changed from the default value.

Table 6: Hyperparameter settings for HalfCheetah-v2 | Hyperparameter
---
name
Value
No. steps of warm up for Q function | 5000
No. steps of warm up for Discriminator | 200
Update policy X times every Y iterations | 2000, 1000
Table 7: Hyperparameter settings for Hopper-v2 | Hyperparameter
---
name
Value
No. steps of warm up for Q function | 1000
No. steps of warm up for Discriminator | 100
Update policy X times every Y iterations | 700, 100
Batch size | 400
Update Discriminator every X iterations | 200

####  8.2.4 GAIfO

We list below the hyperparameter values we used for the implementation of
GAIfO by the OPOLO authors. We only list hyperparameter values that we either
introduced or changed from the default value.

Table 8: Hyperparameter settings for HalfCheetah-v2 | Hyperparameter
---
name
Value
No. steps of Conjugate gradient descent | 10
No. training steps for value function | 5
Table 9: Hyperparameter settings for Hopper-v2 | Hyperparameter
---
name
Value
No. steps of Conjugate gradient descent | 7
No. training steps for value function | 3

###  8.3 Neural network architectures

For every algorithm but SAIfO, we used the default network architecture
provided by the authors of OPOLO. For SAIfO, all the deep networks that we
used were (multi layer perceptrons) MLPs with two hidden layers and tanh
nonlinearities. The discriminator contained 128 units in each hidden layer,
while all the other MLPs contained 256 units each.


# FCTalker: Fine and Coarse Grained Context Modeling for Expressive
Conversational Speech Synthesis

###### Abstract

Conversational Text-to-Speech (TTS) aims to synthesis an utterance with the
right linguistic and affective prosody in a conversational context. The
correlation between the current utterance and the dialogue history at the
utterance level was used to improve the expressiveness of synthesized speech.
However, the fine-grained information in the dialogue history at the word
level also has an important impact on the prosodic expression of an utterance,
which has not been well studied in the prior work. Therefore, we propose a
novel expressive conversational TTS model, termed as FCTalker, that learn the
fine and coarse grained context dependency at the same time during speech
generation. Specifically, the FCTalker includes fine and coarse grained
encoders to exploit the word and utterance-level context dependency. To model
the word-level dependencies between an utterance and its dialogue history, the
fine-grained dialogue encoder is built on top of a dialogue BERT model. The
experimental results show that the proposed method outperforms all baselines
and generates more expressive speech that is contextually appropriate. We
release the source code at: https://github.com/walker-hyf/FCTalker

Index Terms—  Conversational Text-to-Speech (TTS), Fine and Coarse Grained,
Context, Expressive

##  1 Introduction

In conversational Text-to-Speech (TTS), we take the speaker interaction
history between two speakers into account and generate expressive speech for a
target speaker [1, 2]. This technique is highly demanded in the deployment of
intelligent agents [3, 4].

With the advent of deep learning, neural TTS [5, 6, 7, 8], i.e. Tacotron [5,
6], FastSpeech [7, 8] based models, has gained remarkable performance over the
traditional statistical parametric speech synthesis methods [9, 10] in terms
of speech quality. However, the prosodic rendering of neural TTS in a
conversational context remains a challenge.

Fig. 1: An example of word-level context dependencies in a conversation, which
the blue words in the conversation history have a direct effect on the
prosodic expression of the orange words in the current utterance.

The attempts at conversational TTS can be traced back to the HMM era [11, 12,
13]. They make use of rich textual information, such as dialog acts [11] and
extended context [13] for expressive speech generation. However, these
approaches are limited by the need of manual annotation and inadequate
dialogue representation of the model. In the context of neural TTS, Guo et al.
[1] proposed a conversation context encoder based on Tacotron2 model to
extract utterance-level prosody-related information from the dialogue history.
Cong et al. [2] proposed a context-aware acoustic model which predicting the
utterance-level acoustic embedding according to the dialogue history. Mitsui
et al. [14] exploited utterance-level BERT encoding to predict conversational
speaking styles with spontaneous behavior during TTS synthesis. These studies
have advanced the state-of-the-art in conversational TTS. However, they didn’t
exploit the word-level information in the dialogue history for the prosody
rendering of current utterance.

Speech prosody is rendered at various segmental level from syllable, lexical
word, to sentence [15, 16]. As shown in Fig1, the blue words “trophy” and
“won” are strong indicators that determine the prosodic expression of the
final response. We also find that fine-grained token-level information has
played a significant role in conversation-related studies, such as multiturn
dialog generation [16], conversational emotion recognition [15], dialogue
state tracking [17, 18], conversation intent classification [19] etc. They
simultaneously model the hierarchical contextual semantic dependencies, i.e.
word and sentence, between the current utterances and its conversational
history, and achieve performance gains.

Inspired by this, we design a novel dialogue BERT [20] based fine-grained
encoder in addition to the traditional coarse-grained context encoder. Pre-
training a fine-grained encoder, we take the word-level sequences of dialogue
history and current utterance as input to learn their fine-grained dependency.
At last, the outputs of the fine and coarse-grained context encoders are
combined with the phoneme encoding of the input text for expressive speech
generation.

The main contributions of this paper include: 1) We propose a novel
conversational TTS model FCTalker to synthesis expressive speech; 2) We design
a novel fine and coarse grained context modeling scheme to incorporate word
and utterance level context; and 3) The proposed model outperforms all state-
of-the-art baselines in terms of expressiveness rendering. To our best
knowledge, this is the first in-depth conversational TTS study that leverage
both word and utterance level information for prosodic rendering.

##  2 FCTalker: Methodology

As shown in Fig.2, our FCTalker adopts the non-autoregressive Fastspeech2 [8]
TTS framework as the backbone, which includes fine-grained context encoder,
coarse-grained context encoder, text encoder, speaker encoder, duration
predictor, length regulator, variance adaptor and the mel-spectrum decoder.

###  2.1 Overall Architecture

Given a conversation of $T$ dialogue turns, $L=\\{{L_{1},L_{2},...L_{T}}\\}$,
where $L_{his}=\\{{L_{1},L_{2},...L_{T-1}}\\}$ is seen as the dialogue
history, while $L_{T}$ is the current utterance to be synthesized. For each
utterance $L_{i}$ ($i\in[1,T]$), $W_{i}$ indicates the word-level sequence.
Therefore, $W_{his}=\\{W_{1},W_{2},...,W_{T-1}\\}$ is used to indicate the
word-level sequence of dialogue history while $W_{T}$ to represent the word-
level sequence of current utterance.

The fine-grained context encoder $\text{Enc}_{FG}$ takes the $W_{his}$ and
$W_{T}$ as input to generate the fine-grained context embedding
$\mathcal{H}_{F}$. The coarse-grained context encoder $\text{Enc}_{CG}$ reads
the dialogue history and current utterance in utterance, that are $L_{his}$
and $L_{T}$, to extract the coarse-grained context embedding
$\mathcal{H}_{C}$. The text encoder, consists of 4 layers of Feed-Forward
Transformer (FFT) blocks, seeks to extract the phoneme embedding
$\mathcal{H}_{P}$ for the input current utterance. The speaker encoder is a
learnable lookup table, that encodes the speaker identity into a speaker code
$\mathcal{H}_{S}$ [21, 22] to represent the speaker of current utterance.
Similar with the [8], the remaining module, acoustic decoder, consists of
duration predictor, length regulator, variance adaptor and the mel-spectrum
decoder. Variance adaptor aims to estimate different variance information such
as duration, pitch and energy and add them into the hidden sequence to predict
the mel-spectrum features. For duration predictor, we replace the external
aligner section of the FastSpeech2 framework with a CTC-based aligner [23]
since external aligners show a risk of an out-of-distribution problem and
perform misalignment for some data. Note that the fine and coarse grained
context embeddings $\mathcal{H}_{F}$, $\mathcal{H}_{C}$ seek to modulate the
input phoneme embedding $\mathcal{H}_{P}$ with the word and utterance level
context information, which enables all variance information (including pitch,
energy and duration) to be affected to express the prosodic in conversation
scenario. As shown in Fig.2, fine-and coarse-grained context embeddings
$\mathcal{H}_{F}$, $\mathcal{H}_{C}$, phoneme embedding $\mathcal{H}_{P}$ and
the speaker code $\mathcal{H}_{S}$ are concatenated together and feed to the
acoustic decoder to predict the mel-spectrum feature. We employ HiFi-GAN
vocoder [24] to convert the generated mel-spectrum into speech waveform.

Fig. 2: The overall architecture of the FCTalker model. The fine-and coarse-
grained context encoders aim to learn the word-and utterance-level context
dependency respectively between the dialogue history and the current
utterance.

We follow [1] to implement the coarse-grained context encoder structure. Each
utterance $L_{i}(i\in[1,T])$ is embedded using a pre-trained BERT model to
obtain the sentence embedding. Each sentence embedding is appended with a one-
hot vector as a speaker ID to distinguish different speakers. Then the GRU
layer is used to encode the sentence embedding sequence of dialogue history as
a prosody feature, which is then concatenated with the sentence embedding of
the current utterance. The following linear layer is used to convert the
concatenated feature to the final utterance-level context embedding
$\mathcal{H}_{C}$ [1]. For fine-grained context encoder, to model deep
contextual dependencies at the word level, we build its structure with the
dialogue BERT model [20]. The parameters of the fine-grained context encoder
are initialized by a pre-training step, termed as “Fine-Grained Context
Modeling with Dialogue BERT Pre-training”. We will demonstrate the details of
pre-training and fine-grained context embedding extraction in the following
subsections.

###  2.2 Fine-Grained Context Encoder

####  2.2.1 Dialogue BERT Pretrianing

We use BERT’s deep bidirectional context modeling capability [20] to model the
word-level dependencies of dialogue history and current utterance. We use
multiple turns of a conversation involving different speakers for the BERT
pretraining in this work.

As shown in Fig.3(a), the input sequence of dialogue BERT based
$\text{Enc}_{FG}$ consists of [CLS] token, $W_{his}$ and $W_{T}$ sequence, and
the [SEP] token. To capture speaker information and the underlying interaction
behavior in conversation, the speaker token [SPK] for each utterance is
prefixed to each word-level sequence $W_{i}$. At last, the input of the
dialogue BERT pre-training model is processed as “[CLS] [SPK] $W_{1}$ [SPK]
$W_{2}$ … [SPK] $W_{T-1}$ [SPK] $W_{T}$ [SEP]” with standard positional
embeddings and segmentation embeddings. With the help of global context
modeling capacity of self-attention [25], the multi-layer Transformer blocks
are stacked to learn the deep context for the word-level input sequence. We
follow [26] and build the dialogue BERT model with BERT-based uncased model
[26], which includes 12-layers Transformer blocks with 12 attention heads with
hidden sizes $s_{h}=768$ in each layer.

We conduct the pre-training using two loss functions: 1) Masked Language
Modeling Loss ($\mathcal{L}_{mlm}$) and 2) Dialogue Contrastive Loss
($\mathcal{L}_{dc}$). $\mathcal{L}_{mlm}$ is a common objective function for
BERT-like architectures. Note that unlike the BERT model that mask and replace
the token once before training [20], inspired by [26], we conduct token
masking dynamically during batch training. The $\mathcal{L}_{mlm}$ is defined
as: $\mathcal{L}_{mlm}=-\sum_{m=1}^{M}\log P\left(x_{m}\right)$, where $M$ is
the total number of masked tokens and $P(x_{m})$ is the predicted probability
of the token $x_{m}$ over the vocabulary size. $\mathcal{L}_{dc}$ encourage
the model to capture the underlying conversational sequential order, context
information, etc. Suppose that there are $T^{\prime}$ conversations, each
consisting of multiple turns of dialogue. The $\mathcal{L}_{dc}$ is defined
as: $\mathcal{L}_{dc}=-\sum_{i=1}^{T^{\prime}}\log
M_{k,k};\quad\\!\\!\\!\\!\\!\\!M=\operatorname{Softmax}(C\\!R^{\top})\in\mathbb{R}^{T^{\prime}\times
T^{\prime}}$, where $C\in\mathbb{R}^{T^{\prime}\times s_{h}}$ and
$R\in\mathbb{R}^{T^{\prime}\times s_{h}}$ are history and current matrices
respectively by taking the output [CLS] representations from the $T^{\prime}$
dialogues. More details are refer to [26]. At last, we sum them up as the
total loss: $\mathcal{L}$ = $\mathcal{L}_{mlm}$ \+ $\mathcal{L}_{dc}$.

Fig. 3: The fine-grained context encoder is implemented on the basis of a
pretrained dialogue BERT model, as shown in panel (a); we add a additional
linear layer on the top of the neurons for current utterance to extract the
fine-grained context embedding, as shown in panel (b).

####  2.2.2 Fine-Grained Context Embedding Extraction

During the FCTalker training, the $\text{Enc}_{FG}$ learns the word-level
context information to extract the context embedding for current utterance. As
shown in Fig.3(b), we add a linear layer on the top of the neurons for $W_{T}$
to extract the fine-grained context embedding $\mathcal{H}_{F}$. The neurons
for dialogue history and other special tokens are discarded.

There was a study [27] of a local attention module to learn the fine-grained
context in a conversation. For clarity, we would like to highlight two
differences of our work: 1) [27] is focused on the multimodal conversation
while this work is only on textual modality; 2) the simple attention mechanism
in [27] only learns the interaction and ignores the deep semantic dependency
between the dialogue history and the current utterance, that we learn via a
BERT model in this study.

##  3 Experiments and Results

###  3.1 Dataset

We validate the FCTalker on a recently public dataset for conversational TTS
called DailyTalk [28], which is a subset of the open-domain conversation
dataset DailyDialog [29]. DailyTalk consists of 23,773 audio clips
representing 20 hours in total, in which 2,541 conversations were sampled,
modified, and recorded. All dialogues are long enough to represent the context
of each conversation. The dataset was recorded by a male and a female
simultaneously. All speech samples are sampled at 44.10 kHz and coded in 16
bits. We partition the speech data into training, validation, and test sets at
a ratio of 8:1:1.

The dialogue BERT pretraining is conducted on a set of dialogue corpora [26]
consisting of nine different task-oriented datasets, such as MetaLWOZ [30] and
Schema [31]. There are about 100 thousand dialogues covering more than 60
different domains.

###  3.2 Experimental Setup

The fine-grained encoder consists of 12-layers Transformer blocks and 12
attention heads with hidden size $s_{h}$ = 768. Note that the additional
linear layer of Fig.3(b) reduces the output dimension from 768 to 256. The
coarse-grained encoder is implemented by following [1]. The text encoder takes
256-dimensions phoneme sequence as input. We extract 80-channel mel-spectrum
features with a frame size of 50ms and 12.5ms frame shift as the reference
target. The dimensions of $\mathcal{H}_{F}$, $\mathcal{H}_{C}$,
$\mathcal{H}_{P}$ and $\mathcal{H}_{S}$ are all 256. The detailed setup of
acoustic decoder can be found in [28].

The number of conversation turns $T$ is a hyper-parameter that can be adjusted
to validate the effects of different context Windows. We set the value of $T$
to range from 1 to 14 for comparison. For FCTalker training, we use Adam
optimizer with $\beta_{\mbox{\scriptsize 1}}$ = 0.9, $\beta_{\mbox{\scriptsize
2}}$ = 0.98. All speech samples are re-sampled to 22.05 kHZ. The model is
trained on a Tesla V100 GPU with a batch size of 32 and 900k steps. We follow
[26] to pretrain the fine-grained dialogue BERT with 900k steps. Due to the
space limit, we report the subjective evaluations, instead of objective
evaluations for pitch, energy and duration, in following subsection.

###  3.3 Comparative Study

We develop three neural TTS systems for a comparative study, that include the
1) FastSpeech2 [8]: the state-of-the-art neural TTS model that takes the
single utterance as input, without any conversational context modeling; 2)
DailyTalk [28]: a latest neural conversational TTS model with coarse-grained
context encoder; 3) FCTalker: the proposed model with fine-and coarse-grained
context modeling strategy and 4) Ground Truth speech under conversation
scenario.

Table 1: The MOS results for all comparative systems, with 95% confidence interval. System | MOS
---|---
Utterance-level | Dialogue-level
FastSpeech2 [8] | 3.80 $\pm$ 0.026 | 3.90 $\pm$ 0.046
DailyTalk [28] ($T$=2) | 3.86 $\pm$ 0.042 | 3.96 $\pm$ 0.056
FCTalker ($T$=2) | 3.97 $\pm$ 0.040 | 4.15 $\pm$ 0.031
Ground Truth | 4.46 $\pm$ 0.031 | 4.52 $\pm$ 0.034

###  3.4 Experiments and Results

We conduct 5-point Likert scale mean opinion score (MOS) [32] listening
experiments 111Speech samples: https://walker-hyf.github.io/FCTalker/ to
validate the FCTalker model in terms of prosodic expressiveness and context
modeling.

####  3.4.1 Prosodic Expressiveness Evaluation

Note that the prosodic expressiveness of conversational speech is related to
its adjacent dialogue, therefore, we design two kinds of experiments,
including Utterance-level and Dialogue-level MOS tests, to test the prosodic
performance of synthesized speech in a conversation scenario. Specifically,
Utterance-level MOS means the volunteers only rated single sentences from
different systems, without reference to the dialogue history. Dialogue-level
MOS indicates that volunteers rate a multi-turn conversation that includes
several audios, some of which are synthesized from the TTS model.

In this section, each audio is listened by 30 volunteers, each of which
listens to 80 speech samples. For DailyTalk and FCTalker, we set the dialogue
turns $T$ to 2 and report the results in Table 1. It’s observed that our
FCTalker outperforms all baselines and performs the smallest gap with the
ground truth speech, in both the utterance and dialogue scenarios. FastSpeech2
achieves the lowest performance due to the lack of prosodic information in
context during speech generation. DailyTalk can produce more prosodic speech
than FastSpeech2 since it incorporates utterance-level contextual information.
As expected, FCTalker achieves the best performance by considering both word-
level and sentence-level contextual information, which is remarkable.

Fig. 4: The MOS results of various dialogue history turn number $T$ (range
from 2 to 14) for FCTalker, with 95% confidence interval.

####  3.4.2 Dialogue History Turns Comparison

Contextual information is important for the prosodic expression of the current
statement, but how the length of a dialog history turn is chosen to achieve
the optimal effect needs to be further verified. In this section, considering
the average number 9.3 of dialogue turn in the DailyTalk, we set the dialogue
turn $T$ range from 2 to 14 to compare the utterance-and dialogue-level MOS
scores.

We also invite 30 volunteers and follow the settings of Sec.3.4.1 to conduct
the listening test. As shown in Fig.4, when the value of dialogue turn $T$
increases from 2 to 12, the utterance-and dialogue-level MOS scores show an
overall upward trend, while they show a certain downward trend when it
increases from 12 to 14. We also found that when the number of $T$ ranged from
9 to 12, the MOS scores remained above 4.0 steadily. Given that the average
dialogue turn of DailyTalk is 9.3, we concluded that it might be a good choice
to set the optimal $T$ according to the average dialogue turn.

##  4 Conclusion

This work proposed a novel expressive conversational TTS model, termed as
FCTalker, which includes both fine-and coarse-grained encoders. The fine-
grained encoder is implemented on a pre-trained dialogue BERT model and can
learn the deep fine-grained context dependencies of the dialogue history and
the current utterance. Experimental results show that our model outperforms
all baselines and generates expressive speech that is more in line with the
conversational context. We also give suggestions on the selection of the
optimal dialogue history turns through further analysis. To further optimize
the context modeling mechanism will be our future focus.

## References

  * [1] Haohan Guo, Shaofei Zhang, Frank K Soong, Lei He, and Lei Xie,  “Conversational end-to-end tts for voice agents,”  in 2021 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2021, pp. 403–409.
  * [2] Jian Cong, Shan Yang, Na Hu, Guangzhi Li, Lei Xie, and Dan Su,  “Controllable context-aware conversational speech synthesis,”  arXiv preprint arXiv:2106.10828, 2021.
  * [3] Michael McTear,  “Conversational ai: Dialogue systems, conversational agents, and chatbots,”  Synthesis Lectures on Human Language Technologies, vol. 13, no. 3, pp. 1–251, 2020.
  * [4] Ting-En Lin, Yuchuan Wu, Fei Huang, Luo Si, Jian Sun, and Yongbin Li,  “Duplex conversation: Towards human-like interaction in spoken dialogue systems,”  in Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. aug 2022, ACM.
  * [5] Yuxuan Wang, RJ Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, et al.,  “Tacotron: Towards end-to-end speech synthesis,”  arXiv preprint arXiv:1703.10135, 2017.
  * [6] Jonathan Shen, Ruoming Pang, Ron J Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan, et al.,  “Natural tts synthesis by conditioning wavenet on mel spectrogram predictions,”  in 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2018, pp. 4779–4783.
  * [7] Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu,  “Fastspeech: Fast, robust and controllable text to speech,”  Advances in Neural Information Processing Systems, vol. 32, 2019\.
  * [8] Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu,  “Fastspeech 2: Fast and high-quality end-to-end text to speech,”  arXiv preprint arXiv:2006.04558, 2020.
  * [9] Heiga Zen, Keiichi Tokuda, and Alan W Black,  “Statistical parametric speech synthesis,”  speech communication, vol. 51, no. 11, pp. 1039–1064, 2009.
  * [10] Heiga Ze, Andrew Senior, and Mike Schuster,  “Statistical parametric speech synthesis using deep neural networks,”  in 2013 ieee international conference on acoustics, speech and signal processing. IEEE, 2013, pp. 7962–7966.
  * [11] Ann K Syrdal and Yeon-Jun Kim,  “Dialog speech acts and prosody: Considerations for tts,”  in Proceedings of speech prosody, 2008, pp. 661–665.
  * [12] Tomoki Koriyama, Takashi Nose, and Takao Kobayashi,  “Conversational spontaneous speech synthesis using average voice model,”  in Eleventh Annual Conference of the International Speech Communication Association, 2010.
  * [13] Tomoki Koriyama, Takashi Nose, and Takao Kobayashi,  “On the use of extended context for hmm-based spontaneous conversational speech synthesis,”  in Twelfth Annual Conference of the International Speech Communication Association, 2011.
  * [14] Kentaro Mitsui, Tianyu Zhao, Kei Sawada, Yukiya Hono, Yoshihiko Nankaku, and Keiichi Tokuda,  “End-to-end text-to-speech based on latent representation of speaking styles using spontaneous dialogue,”  arXiv preprint arXiv:2206.12040, 2022.
  * [15] Yu-Ping Ruan, Shu-Kai Zheng, Taihao Li, Fen Wang, and Guanxiong Pei,  “Hierarchical and multi-view dependency modelling network for conversational emotion recognition,”  in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 7032–7036.
  * [16] Jiamin Wang, Xiao Sun, Qian Chen, and Meng Wang,  “Information-enhanced hierarchical self-attention network for multiturn dialog generation,”  IEEE Transactions on Computational Social Systems, 2022.
  * [17] Jun Quan and Deyi Xiong,  “Modeling long context for task-oriented dialogue state generation,”  arXiv preprint arXiv:2004.14080, 2020.
  * [18] Adarsh Kumar, Peter Ku, Anuj Goyal, Angeliki Metallinou, and Dilek Hakkani-Tur,  “Ma-dst: Multi-attention-based scalable dialog state tracking,”  in Proceedings of the AAAI Conference on Artificial Intelligence, 2020, vol. 34, pp. 8107–8114.
  * [19] Rashmi Gangadharaiah and Balakrishnan Narayanaswamy,  “Joint multiple intent detection and slot labeling for goal-oriented dialog,”  in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 2019, pp. 564–569.
  * [20] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova,  “Bert: Pre-training of deep bidirectional transformers for language understanding,”  in Proceedings of NAACL-HLT, 2019, pp. 4171–4186.
  * [21] Andrew Gibiansky, Sercan Arik, Gregory Diamos, John Miller, Kainan Peng, Wei Ping, Jonathan Raiman, and Yanqi Zhou,  “Deep voice 2: Multi-speaker neural text-to-speech,”  Advances in neural information processing systems, vol. 30, 2017\.
  * [22] Wei Ping, Kainan Peng, Andrew Gibiansky, Sercan O Arik, Ajay Kannan, Sharan Narang, Jonathan Raiman, and John Miller,  “Deep voice 3: Scaling text-to-speech with convolutional sequence learning,”  in International Conference on Learning Representations, 2018.
  * [23] Rohan Badlani, Adrian Lańcucki, Kevin J Shih, Rafael Valle, Wei Ping, and Bryan Catanzaro,  “One tts alignment to rule them all,”  in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 6092–6096.
  * [24] Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae,  “Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis,”  Advances in Neural Information Processing Systems, vol. 33, pp. 17022–17033, 2020.
  * [25] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin,  “Attention is all you need,”  Advances in neural information processing systems, vol. 30, 2017\.
  * [26] Chien-Sheng Wu, Steven C.H. Hoi, Richard Socher, and Caiming Xiong,  “TOD-BERT: Pre-trained natural language understanding for task-oriented dialogue,”  in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, Nov. 2020, pp. 917–929.
  * [27] Jingbei Li, Yi Meng, Xixin Wu, Zhiyong Wu, Jia Jia, Helen Meng, Tian Qiao, Yuping Wang, and Yuxuan Wang,  “Inferring speaking styles from multi-modal conversational context by multi-scale relational graph convolutional networks,”  in Proceedings of the 30th ACM International Conference on Multimedia, Lisboa, Portugal, 2022, MM ’22, ACM.
  * [28] Keon Lee, Kyumin Park, and Daeyoung Kim,  “Dailytalk: Spoken dialogue dataset for conversational text-to-speech,”  arXiv preprint arXiv:2207.01063, 2022.
  * [29] Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu,  “Dailydialog: A manually labelled multi-turn dialogue dataset,”  arXiv preprint arXiv:1710.03957, 2017.
  * [30] Sungjin Lee, Hannes Schulz, Adam Atkinson, Jianfeng Gao, Kaheer Suleman, Layla El Asri, Mahmoud Adada, Minlie Huang, Shikhar Sharma, Wendy Tay, and Xiujun Li,  “Multi-domain task-completion dialog challenge,”  in Dialog System Technology Challenges 8, March 2019.
  * [31] Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan,  “Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset,” 2019.
  * [32] Robert C Streijl, Stefan Winkler, and David S Hands,  “Mean opinion score (mos) revisited: methods and applications, limitations and alternatives,”  Multimedia Systems, vol. 22, no. 2, pp. 213–227, 2016.