# Approximation of deterministic mean field type control systems

Yurii Averboukh

###### Abstract

The paper is concerned with the approximation of the deterministic the mean
field type control system by a mean field Markov chain. It turns out that the
dynamics of the distribution in the approximating system is described by a
system of ordinary differential equations. Given a strategy for the Markov
chain, we explicitly construct a control in the deterministic mean field type
control system. Our method is a realization of the model predictive approach.
The converse construction is also presented. These results lead to an estimate
of the Hausdorff distance between the bundles of motions in the deterministic
mean field type control system and the mean field Markov chain. Especially, we
pay the attention to the case when one can approximate the bundle of motions
in the mean field type system by solutions of a finite systems of ODEs.
MSC Classification (2020): 93A14,93B11,93C10,93E10.
Keywords: mean field type control, bundle of motions, mean field Markov chain,
model predictive control.

##  1 Introduction

The paper studies mean field type control systems. These systems describe an
evolution of many identical agents who play cooperatively and interact via
some external field. The mean field type control systems appear within the
modeling of swarm of robots, pedestrian flows, etc [19, 18, 20, 10, 16].

The concept of mean field models comes back to the model of plasma proposed by
Vlasov in 1938 [36, 37] and was formalized within the theory of McKean-Vlasov
equation [31, 35]. We will focus on the deterministic (first-order) mean field
type systems. This means that the dynamics of each agent is described by an
ordinary differential equation on the Euclidean space with the right-hand side
depending on his/her state, control and the distribution of all agents.

Notice that the settings of the control theory include the study of optimal
control problems as well as the examination of the qualitative properties of
the bundle of trajectories.

The mean field type optimal control theory started with paper [1]. Nowadays,
for the second order mean field type control system, i.e., when the dynamics
of each agent is determined by a stochastic differential equation, the
necessary and optimality conditions are derived (see [9, 15, 17, 22, 30, 33,
32]). The case of first-order mean field type optimal control problems was
studied in papers [24, 34, 6], where the variants of dynamic programming and
Pontryagin maximum principle were obtained.

The qualitative theory for mean field type control systems studies the general
properties of bundle of motions as well as viability theory issues. For the
deterministic mean field type control systems, the existence theorem was
proved under the general assumption on the dynamics [11, 8, 24], while the
Filippov and relaxation theorem are derived under additional assumption of
continuity of the vector field [12, 14]. The viability theorem was obtained in
the terms of tangent cones (see [4, 13]) and in the terms of proximal normal
cones [7].

The main object of the paper is an approximation of the bundle of motions of
the first order mean field type control system. Our approach is based on so
called Markov approximations. They appear when one replace the ODE determining
the dynamics of each agent by a continuous-time mean field Markov chain. The
latter can be regarded as a system of infinitely many similar agents with the
dynamics determined by transition rates depending, in particular, on a current
distribution of agents. In this case, the dynamics of the whole distribution
of agents is described by a nonlinear Kolmogorov equation. If, additionally,
one can assure that the agents in the original system do not leave a compact
space, the phase space for the approximating Markov chain can be chosen to be
finite. This leads to an approximation of the first-order mean field type
control system by a finite system of ODE.

The approximation of the deterministic control system by a Markov chain was
proposed in [3]. In that paper, the approximation of the value function of the
zero-sum differential game was studied based on stochastic control with guide
approach coming back to research of Krasovskii and Kotelnikova [27, 28, 26,
29]. This technique was extended to mean field type differential games in [5].
There, based on a modification of the extremal shift rule for the Wasserstein
space and control of the guide strategies, the value function of the mean
field type differential game was approximated by a solution of a finite
dimensional differential game.

In the paper, we focus on the approximation of the mean field type control
system and implement the model predictive control approach (see [23] and
reference therein). Notice that for the original deterministic mean field type
control system we assume the open-loop strategies those are distribution of
pairs consisting of an initial state and a control whereas the approximating
mean field Markov chain implies the feedback strategies. The latters can be
regarded as a sequence of open loop strategies those work at the appropriate
state.

As it was mentioned above, we use the methodology of the model predictive
approach. For the considered deterministic mean field type control system,
this means that, given a feedback strategy in Markov chain, an agent uses on a
small time interval a control borrowed from a state in the approximating
Markov chain. The weights of the controls are determined by an optimal plan
between the distribution in the original and approximating system. To solve
the converse problem which implies the design of a feedback strategy in the
Markov chain based on a given distribution of controls in the mean field type
system, one can on each small time interval integrate the controls according
to the optimal plan between distributions.

The main results of the paper includes also an approximation rates of the
aforementioned constructions. They depend only on the original system,
distance between the original and approximating systems, fineness of partition
and the maximal transition rate in the mean field Markov chain. Notice also
that, if the fineness of the partition tends to infinity, the limiting
approximation rates are determined only by the original system and the
distance between the original and approximating systems. This, in particular,
provides the distance between the bundles of motions in the deterministic mean
field type control system and a finite dimensional control system. The earlier
results gave only one-side estimate.

The rest of the paper is organized as follows. In Section 2, we introduce the
general notation. The assumption on the deterministic mean field type control
system as well as the class of admissible controls are presented in Section 3.
The approximating mean field Markov chain is introduced in Section 4. Here, we
define the general system and show the way to construct an approximating
Markov chain for the original deterministic mean field type system. Section 5
deals with the model predictive control for the deterministic system. In this
case, we assume that a feedback strategy for the mean field Markov chain is
given and derive the approximation rate of the constructed motion in the
deterministic mean field type system. The model predictive control for the
mean field Markov chain is discussed in Section 6. Finally, we estimate the
Hausdorff distance between the bundle of motions in the deterministic mean
field type control system and the Markov chain (see Section 7).

##  2 General notation

  * •

If $n$ is a natural number, $X_{1},\ldots,X_{n}$ are sets,
$i_{1},\ldots,i_{k}$ are indices from $\\{1,\ldots,n\\}$, then
$\operatorname{p}^{i_{1},\ldots,i_{k}}$ is a projection from
$X_{1}\times\ldots X_{n}$ to $X_{i_{1}}\times\ldots X_{i_{k}}$, i.e,

$p^{i_{1},\ldots,i_{k}}(x_{1},\ldots,x_{n})\triangleq(x_{i_{1}},\ldots,x_{i_{k}}).$

  * •

If $(X,\rho_{X})$, $(Y,\rho_{Y})$ are Polish spaces, then $C_{b}(X,Y)$ denotes
the space of bounded continuous functions from $X$ to $Y$. If $Y$ is a normed
vector space, then $C_{b}(X,Y)$ is also a normed vector space with usual
$\sup$-norm. Moreover, $C_{b}(X)\triangleq C_{b}(X,\mathbb{R})$.

  * •

We always endow a Polish space $(X,\rho_{X})$ with the Borel $\sigma$-algebra
denoted by $\mathcal{B}(X)$. Moreover, $\mathcal{M}(X)$ stands for the set of
Borel nonnegative measures, whereas $\mathcal{P}(X)$ denotes the set of all
Borel probabilities:

$\mathcal{P}(X)\triangleq\\{m\in\mathcal{M}(X):m(X)=1\\}.$

We consider on $\mathcal{M}(X)$ the topology of narrow convergence, i.e., a
sequence of measures $\\{m_{n}\\}_{n=1}^{\infty}$ converges to a measure $m$
if, for each $C_{b}(X)$,

$\int_{X}\phi(x)m_{n}(dx)\rightarrow\int_{X}\phi(x)m(dx)\text{ as
}n\rightarrow\infty.$

Notice that $\mathcal{P}(X)$ is closed w.r.t. the narrow convergence.

  * •

If $(\Omega,\mathcal{F})$, $(\Omega^{\prime},\mathcal{F}^{\prime})$ are
measurable spaces, $m$ is a measure on $\mathcal{F}$, whilst
$h:\Omega\rightarrow\Omega^{\prime}$ is
$\mathcal{F}/\mathcal{F}^{\prime}$-measurable, then we denote by $h\sharp m$
the push-forward measure on $\mathcal{F}^{\prime}$ defined by the rule: for
each $\Upsilon\in\mathcal{F}^{\prime}$,

$(h\sharp m)(\Upsilon)\triangleq m(h^{-1}(\Upsilon)).$

  * •

If $(X,\rho_{X})$ and $(Y,\rho_{Y})$ are two Polish space, $m$ is a measure on
$X$, then we denote by $\Lambda(X,m;Y)$ the set of measures on $X\times Y$
with marginal distribution on $X$ equal to $m$, i.e.,

$\Lambda(X,m;Y)\triangleq\big{\\{}\alpha\in\mathcal{M}(X\times
Y):\operatorname{p}^{1}\sharp\alpha=m\big{\\}}.$

  * •

A function $\beta:X\rightarrow\mathcal{P}(X)$ is called weakly measurable if,
for each $\phi\in C_{b}(Y)$, the function

$x\mapsto\int_{Y}\phi(y)\beta(x,dy)$

is measurable. Furthermore, each weakly measurable function $\beta$ generates
a measure $m\star\beta\in\Lambda(X,m;Y)$ by the rule: for each $\phi\in
C_{b}(X\times Y)$,

$\int_{X\times
X}\phi(x,y)(m\star\beta)(d(x,y))\triangleq\int_{X}\int_{Y}\phi(x,y)\beta(x,dy)m(dx).$

  * •

If $\alpha\in\Lambda(X,m;Y)$, then there exists a weakly measurable function
$\beta$ such that $\alpha=m\star\beta$ [21, III-70]. These function is called
a disintegration of the measure $\alpha$ and denoted by
$x\mapsto\alpha(\cdot|x)$. The disintegration is unique a.e.

  * •

If $(X,\rho_{X})$ is a Polish space, $p\geq 1$, then $\mathcal{P}^{p}(X)$ is
the space of probabilities on $X$ such that, for some (equivalently, every)
$x_{*}\in X$,

$\int_{X}\rho_{X}^{p}(x,x_{*})m(dx)<\infty.$

If $X$ is a normed vector space, then we will always choose $x_{*}=0$ and put

$\varsigma_{p}(m)\triangleq\left[\int_{X}\|x\|^{p}m(dx)\right]^{1/p}.$

  * •

The space $\mathcal{P}^{p}(X)$ is endowed with the Wasserstein metric defined
by the rule: if $m_{1},m_{2}\in\mathcal{P}^{p}(X)$, then

$W_{p}(m_{1},m_{2})\triangleq\Bigg{[}\inf\Biggl{\\{}\int_{X\times
X}\rho_{X}^{p}(x_{1},x_{2})\pi(d(x_{1},x_{2})):\pi\in\Pi(m_{1},m_{2})\Biggr{\\}}\Bigg{]}^{1/p}.$

Hereinafter, $\Pi(m_{1},m_{2})$ stands for the set of probabilities $\pi$ on
$X_{1}\times X_{2}$ such that $\operatorname{p}^{i}\sharp\pi=m_{i}$, $i=1,2$.
The convergence within $W_{p}$ implies the narrow convergence, the converse
holds true only if $X$ is compact [2, Proposition 7.1.5]. Primary, we will
consider the case $p=2$.

  * •

The set of all continuous curves in $\mathbb{R}^{d}$ on $[s,r]$ is denoted by
$\Gamma_{s,r}\triangleq C([s,r],\mathbb{R}^{d})$. If $t\in[s,r]$, the denote
by $e_{t}$ the evaluation operator defined by the rule: for
$x(\cdot)\in\Gamma_{s,r}$

$e_{t}(x(\cdot))\triangleq x(t).$

##  3 First-order mean field type control system

In the paper, we consider a mean field type control system formally described
by a continuity equation

$\partial_{t}m(t)+\operatorname{div}(f(t,x,m(t),u(t,x))m(t))=0.$ (1)

Here, $t\in[0,T]$, $x\in\mathbb{R}^{d}$ is a phase variable, $m(t)$ stands for
a current distribution of agents, $u(t,x)$ is a control implemented by an
agent at the time $t$ and the state $x$. We will assume that the control is
chosen from a set $U$. This system describes the behavior of the infinitely
many identical agents such that the dynamics of each agent is determined by
the ODE:

$\frac{d}{dt}x(t)=f(t,x(t),m(t),u(t)).$ (2)

We impose the following conditions on $U$ and $f$.

  1. (C1)

$U$ is a metric compact;

  2. (C2)

$f$ is a continuous function;

  3. (C3)

there exists a set $\mathcal{K}\subset\mathbb{R}^{d}$ such that, if
$\operatorname{supp}(m)\subset\mathcal{K}$, then

$f(t,x,m,u)=0,\ \ x\notin\mathcal{K};$ (3)

  4. (C4)

$f$ is bounded on
$[0,T]\times\mathcal{K}\times\mathcal{P}^{2}(\mathcal{K})\times U$ by a
constant $R$;

  5. (C5)

$f$ is Lipschitz continuous w.r.t. the space and measure variables on
$[0,T]\times\mathcal{K}\times\mathcal{P}^{2}(\mathcal{K})\times U$; the
Lipschitz constant is denoted by $C_{f}$.

It is convenient to use the relaxation of the controls. This means that we
replace the set of instantaneous controls $U$ with the set of probabilities on
$U$. The time-dependent relaxed controls are defined as follows.

For $s,r\in[0,T]$, $s<r$, put

$\mathcal{U}_{s,r}\triangleq\Lambda([s,r],\lambda;U),$

where $\lambda$ is the Lebesgue measure on $[s,r]$. An element of
$\mathcal{U}_{s,r}$ is a control measure on $[s,r]$. Notice that
$\mathcal{U}_{s,r}$ is a compact subset of $\mathcal{M}([s,r]\times U)$. For
now, assume that we are given with a flow of probabilities
$m(\cdot):[s,r]\rightarrow\mathcal{P}^{p}(\mathbb{R}^{d})$. Furthermore, let
$y\in\mathbb{R}^{d}$ be an initial state, and let $\xi\in\mathcal{U}_{s,r}$ be
a relaxed control. Then, the corresponding motion of an agent is a solution of
the initial value problem

$\frac{d}{dt}x(t)=\int_{U}f(t,x(t),m(t),u)\xi(du|t),\ \ x(s)=y.$ (4)

We denote this motion by $x(\cdot,s,y,m(\cdot),\xi)$. Let us denote by
$\operatorname{traj}^{s,r}_{m(\cdot)}$ the operator assigning to
$y\in\mathbb{R}^{d}$ and $\xi\in\mathcal{U}_{s,r}$ the trajectory
$x(\cdot,s,y,m(\cdot),\xi)\in\Gamma_{s,r}$.

Further, for $m\in\mathcal{P}^{p}(\mathbb{R}^{d})$, put

$\mathcal{A}_{s,r}[m]\triangleq\Lambda(\mathbb{R}^{d},m;\mathcal{U}_{s,r}).$

The set $\mathcal{A}_{s,r}[m]$ is the set of distributions of pairs consisting
of an initial state and a relaxed control compatible with the initial
probability $m$.

######  Definition 3.1.

Let $s,r\in[0,T]$, $s<r$, $m_{*}\in\mathcal{P}^{p}(\mathbb{R}^{d})$,
$\alpha\in\mathcal{A}[m_{*}]$. We say that a function
$m(\cdot):[s,r]\rightarrow\mathcal{P}^{p}(\mathbb{R}^{d})$ is a motion of the
deterministic mean field type system (1) produced by the initial time $s$ and
the distribution of controls $\alpha\in\mathcal{A}_{s,r}[m_{*}]$ if there
exists a measure $\chi\in\mathcal{P}^{2}(\Gamma_{s,r})$ such that

  * •

$m(s)=m_{*}$;

  * •

$\chi=\operatorname{traj}^{s,r}_{m(\cdot)}\sharp\alpha$;

  * •

$m(t)=e_{t}\sharp\chi$ on $[s,r]$.

Below we denote the motion of the system (1) produced by the initial time $s$
and the distribution of controls $\alpha\in\mathcal{A}_{s,r}[m_{*}]$ by
$m(\cdot,s,\alpha)$.

######  Proposition 3.2.

For each $s,r\in[0,T]$, $r>s$, $m_{*}\in\mathcal{P}^{2}(\mathcal{K})$,
$\mathcal{A}_{s,r}[m_{*}]$, there exists a unique motion $m(\cdot,s,\alpha)$.
Moreover, $m(t,s,\alpha)\in\mathcal{P}^{2}(\mathcal{K})$ for all $t\in[s,r]$.

This proposition in fact is proved in [8].

Further, we introduce the concatenation of distribution of controls. First, if
$s_{0},s_{1},s_{2}\in[0,T]$, $s_{0}<s_{1}<s_{2}$,
$\xi_{0}\in\mathcal{U}_{s_{0},s_{1}}$, $\xi_{1}\in\mathcal{U}_{s_{1},s_{2}}$,
then the concatenation $\xi\triangleq\xi_{0}\diamond_{s_{1}}\xi_{1}$ of these
controls is defined by its disintegration w.r.t. the Lebesgue measure:

$\xi(\cdot|t)\triangleq\left\\{\begin{array}[]{cc}\xi_{0}(\cdot|t),&t\in[s_{0},s_{1}),\\\
\xi_{1}(\cdot|t),&t\in[s_{1},s_{2}].\end{array}\right.$

######  Definition 3.3.

Let

  * •

$s_{0},s_{1},s_{2}\in[0,T]$, $s_{0}<s_{1}<s_{2}$;

  * •

$m_{0},m_{1}\in\mathcal{P}^{2}(\mathbb{R}^{d})$;

  * •

$\alpha_{0}\in\mathcal{A}_{s_{0},s_{1}}[m_{0}]$,
$\alpha_{1}\in\mathcal{A}_{s_{1},s_{2}}[m_{1}]$.

be such that

$m_{1}=m(s_{1},s_{0},\alpha_{0}).$

A probability $\alpha\in\mathcal{A}_{s_{0},s_{2}}[m_{0}]$ defined by the rule:
for every $\phi\in C_{b}(\mathbb{R}^{d}\times\mathcal{U}_{s_{0},s_{2}})$,

$\begin{split}\int_{\mathbb{R}^{d}\times\mathcal{U}_{s_{0},s_{2}}}\phi(&y,\xi)\alpha(d(y,\xi))\\\
&\triangleq\int_{\mathbb{R}^{d}\times\mathcal{U}_{s_{0},s_{1}}}\int_{\mathcal{U}_{s_{1},s_{2}}}\phi(y,\xi_{0}\diamond_{s_{1}}\xi_{1})\alpha_{1}(d\xi_{1}|x^{0}(y,\xi_{0}))\alpha_{0}(d(y,\xi_{0}))\end{split}$

is called a concatination of distribution $\alpha_{0}$ and $\alpha_{1}$. Here
we use the designations

$x^{0}(y,\xi_{0})\triangleq x(s_{1},s_{0},y,m^{0}(\cdot),\xi_{0}),\ \
m^{0}(\cdot)\triangleq m(\cdot,s_{0},y,m_{0},\alpha_{0}).$

With some abuse of notation, we denote the concatenation of distributions by
$\alpha_{0}\diamond_{s_{1}}\alpha_{1}$.

######  Proposition 3.4.

Assume that $s_{0}<s_{1}<s_{2}$, $m_{0},m_{1}\in\mathcal{P}^{2}(\mathcal{K})$,
$\alpha_{0}\in\mathcal{A}_{s_{0},s_{1}}[m_{0}]$,
$\alpha_{1}\in\mathcal{A}_{s_{1},s_{2}}[m_{1}]$ are such that

$m_{1}=m(s_{1},s_{0},\alpha_{0}).$

Then,

  * •

$m(t,s_{0},\alpha_{0}\diamond_{s_{1}}\alpha_{1})=m(t,s_{0},\alpha_{0})$ when
$t\in[s_{0},s_{1}]$;

  * •

$m(t,s_{0},\alpha_{0}\diamond_{s_{1}}\alpha_{1})=m(t,s_{1},\alpha_{1})$ when
$t\in[s_{1},s_{2}]$.

This proposition directly follows from the definition of concatenation.

Let us complete this section with the equivalent formalization of the motion
in mean field type control system (1). The following result is proved in [24,
Theorem 1].

######  Proposition 3.5.

Let $m(\cdot):[s,r]\rightarrow\mathcal{P}^{2}(\mathcal{K})$ be equal to
$m(\cdot,s,\alpha)$ for some distribution of controls $\alpha$. Then there
exists a velocity field $v:[s,r]\times\mathbb{R}^{d}\rightarrow\mathbb{R}^{d}$
such that

  1. (V1)

$v(t,x)\in\operatorname{co}\\{f(t,x,m(t),u):u\in U\\}$ for a.e. $t\in[s,r]$,
$m(t)$-a.e. $x\in\mathbb{R}^{d}$;

  2. (V2)

the equation

$\partial_{t}m(t)+\operatorname{div}(v(t,x)m(t))=0$

holds in the sense of distributions, i.e., for every $\phi\in
C_{c}^{1}([s,r]\times\mathbb{R}^{d})$,

$\int_{s}^{r}\int_{\mathbb{R}^{d}}[\partial_{t}\phi(t,x)+\langle\nabla\phi(t,x),v(t,x)\rangle]m(t,dx)dt=0.$

Conversely, if $m(\cdot)$ and $v(\cdot,\cdot)$ satisfy conditions (V1), (V2),
then there exits a distribution of controls $\alpha$ such that

$m(\cdot)=m(\cdot,s,\alpha).$

##  4 Mean filed Markov chains

In this section, we introduce a controlled nonlinear Markov chain. Let
$\mathcal{S}\subset\mathcal{K}$ be at most countable set. Distributions on
$\mathcal{S}$ are described by sequences
$\mu=(\mu_{\bar{x}})_{\bar{x}\in\mathcal{S}}$ such that

$\mu_{\bar{x}}\geq 0,\ \ \sum_{\bar{x}\in\mathcal{S}}\mu_{\bar{x}}=1.$

The set of such distributions is simplex on $\mathcal{S}$ denoted below by
$\Sigma$. Furthermore, let $\Sigma^{2}$ be a set of sequences
$\mu=(\mu_{\bar{x}})_{\bar{x}\in\mathcal{S}}$ such that

$\sum_{\bar{x}\in\mathcal{S}}\|\bar{x}\|^{2}\mu_{\bar{x}}<\infty.$

If $\mathcal{S}$ is finite, the sets $\Sigma$ and $\Sigma^{2}$ coincide. There
is a natural isomorphism between $\Sigma$ and $\mathcal{P}(\mathcal{S})$:

$\mu=(\mu_{\bar{x}})_{\bar{x}\in\mathcal{S}}\mapsto\mathscr{I}(\mu)\triangleq\sum_{\bar{x}\in\mathcal{S}}\delta_{\bar{x}}\mu_{\bar{x}}.$

Hereinafter, $\delta_{z}$ stands for the Dirac measure concentrated at $z$.

For $t\in[0,T]$, $\mu\in\Sigma^{2}$, $u\in U$, let
$Q(t,\mu,u)=(Q_{\bar{x},\bar{y}}(t,\mu,u))_{\bar{x},\bar{y}\in\mathcal{S}}$ be
a Kolmogorov matrix, i.e., $Q_{\bar{x},\bar{y}}(t,\mu,u)\geq 0$ when
$\bar{x}\neq\bar{y}$ and, for each $\bar{x}\in\mathcal{S}$,

$\sum_{\bar{y}\in\mathcal{S}}Q_{\bar{x},\bar{y}}(t,\mu,u)=0.$

Now let us introduce a mean field Markov chain generated by this Kolmogorov
matrix. To this end, we first consider a relaxation of the control space. As
above, a relaxed control on $[s,r]$ is an element of $\mathcal{U}_{s,r}$. We
will use the feedback approach. This means that we are given with a sequence
of relaxed control
$\zeta_{\mathcal{S}}=(\zeta_{\bar{x}})_{\bar{x}\in\mathcal{S}}$. Notice that
the set of feedback controls is $\mathcal{U}^{\mathcal{S}}_{s,r}$. In this
case, the instantaneous probability rate for transition from $\bar{x}$ to
$\bar{y}$ is equal to

$\mathcal{Q}_{\bar{x}\,\bar{y}}(t,\mu(t),\zeta_{\mathcal{S}})\triangleq\int_{U}Q_{\bar{x},\bar{y}}(t,\mu,u)\zeta_{\bar{x}}(du|t).$

Moreover, we denote

$\mathcal{Q}(t,\mu(t),\zeta_{\mathcal{S}})\triangleq(\mathcal{Q}_{\bar{x}\,\bar{y}}(t,\mu(t),\zeta_{\mathcal{S}}))_{\bar{x},\bar{y}\in\mathcal{S}}.$

######  Definition 4.1.

Given a time interval $[s,r]$, an initial distribution of states
$\mu_{*}\in\Sigma^{2}$ and a feedback control
$\zeta_{\mathcal{S}}=(\zeta_{\bar{x}})_{\bar{x}\in\mathcal{S}}$, we say that
$\mu(\cdot)$ is a motion in the mean field Markov chain if it satisfies the
following initial value problem:

$\frac{d}{dt}\mu_{\bar{y}}(t)=\sum_{\bar{x}\in\mathcal{S}}\mu_{\bar{x}}(t)\mathcal{Q}_{\bar{x},\bar{y}}(t,\mu(t),\zeta_{\mathcal{S}}),\
\ \mu_{\bar{y}}(s)=\mu_{*,\bar{y}}.$ (5)

Notice that system (5) can be rewritten in the vector form

$\frac{d}{dt}\mu(t)=\mu(t)\mathcal{Q}(t,\mu(t),\zeta_{\mathcal{S}}).$ (6)

To guarantee the existence of the distribution $\mu(\cdot)$, it is sufficient
to assume that $Q$ has continuous entries, for each $\bar{x}$ only finite
number of entries $Q_{\bar{x},\bar{y}}(t,\mu,u)$ are non-zero and the
dependence of the matrix $Q$ on $\mu$ is Lipschitz continuous.

Let us also give a probabilistic interpretation of Definition 4.1. Set

  * •

$\Omega_{s,r}\triangleq D([s,r];\mathcal{S})$, where $D([s,r];\mathcal{S})$
stands a Skorokhod space of càdlàg functions;

  * •

$\mathcal{F}_{s,r}\triangleq\mathcal{B}(D([s,r];\mathcal{S}))$;

  * •

$\mathbb{F}_{s,r}=\\{\mathcal{F}_{s,r}^{t}\\}_{t\in[s,r]}$, where
$\mathcal{F}_{s,r}^{t}\subset\mathcal{F}_{s,r}$ is a $\sigma$-algebra such
that projections of its elements on $[s,t]$ form the whole $\sigma$-algebra
$\mathcal{B}(D([s,t];\mathcal{S}))$, while the projection on $[t,r]$ is a
trivial $\sigma$-algebra;

  * •

$X(t,\omega)\triangleq\omega(t)$.

Further, we define the generator $L_{t}[\mu,\zeta_{\mathcal{S}}]$ by the rule:
for $\phi\in C_{b}(\mathcal{S})$,

$L_{t}[\mu,\zeta_{\mathcal{S}}]\phi(x)\triangleq\sum_{\bar{y}\in\mathcal{S}}\mathcal{Q}_{\bar{x},\bar{y}}(t,\mu,\zeta_{\mathcal{S}})\phi(\bar{y}).$

######  Definition 4.2.

Let $[s,r]$ be a time interval, $\mu_{*}$ be an initial distribution of
states, $\zeta_{\mathcal{S}}\in\mathcal{U}_{s,r}^{\mathcal{S}}$ and let
$\mu(\cdot)=\mu(\cdot,s,\mu_{*},\zeta_{\mathcal{S}})$. We say that a
probability $\mathbb{P}_{s,r}$ on $\mathcal{F}_{s,r}$ realizes $\mu(\cdot)$ if

  * •

$\mu_{\bar{x}}(t)=\mathbb{P}_{s,r}\\{\omega(t)=\bar{x}\\}$;

  * •

for each $\phi\in C_{b}(\mathcal{S})$, the process

$\phi(X(t))-\int_{s}^{t}L_{t}[\mu(t),\zeta_{\mathcal{S}}]\phi(X(t^{\prime}))dt^{\prime}$

is a $\mathbb{F}_{s,r}$-martingale.

Below, if $\mathbb{P}_{s,r}$ is a realization of $\mu(\cdot)$,
$\mathbb{E}_{s,r}$ stands for the corresponding expectation.

Notice that there exists at least one representation of flow $\mu(\cdot)$ [25,
Theorem 5.4.2].

The main result of the paper is proved under the following approximation
assumptions.

  1. (A1)

$\max_{x\in\mathcal{K}}\min_{\bar{y}\in\mathcal{S}}\|x-\bar{y}\|\leq\varepsilon;$

  2. (A2)

entries of the matrix $Q$ are uniformly bounded by a number $B_{Q}$.

  3. (A3)

for each $t\in[0,T]$, $\bar{x}\in\mathcal{S}$, $\mu\in\Sigma^{2}$ and $u\in
U$,

$\Bigl{\|}f(t,\bar{x},\mathscr{I}(\mu),u)-\sum_{\bar{y}\in\mathcal{S}}(\bar{y}-\bar{x})Q_{\bar{x},\bar{y}}(t,\mu,u)\Bigl{\|}\leq\varepsilon;$

  4. (A4)

for each $t\in[0,T]$, $\bar{x}\in\mathcal{S}$, $\mu\in\Sigma^{2}$ and $u\in
U$,

$\sum_{\bar{y}\in\mathcal{S}}\|\bar{y}-\bar{x}\|^{2}Q_{\bar{x},\bar{y}}(t,\mu,u)\leq\varepsilon^{2}.$

Without loss of generality, we assume that $\varepsilon\leq 1$.

An example of the Markov chain that satisfies assumptions (A1)–(A4) can be
constructed on a regular lattice as follows. First, we represent $f$ in the
coordinate-wise form

$f(t,x,m,u)=(f_{i}(t,x,m,u))_{i=1}^{d}.$

Let $h>0$, $\mathcal{K}^{h}\triangleq\mathcal{K}+\mathbb{B}_{h}$, where
$\mathbb{B}_{h}$ is a ball centered at the origin and radius $h$. We put

$\mathcal{S}\triangleq\mathcal{K}^{h}\cap h\mathbb{Z},$ (7)
$Q_{\bar{x},\bar{y}}(t,\mu,u)\triangleq\left\\{\begin{array}[]{ll}h^{-1}|f_{i}(t,\bar{x},\mathscr{I}(\mu),u)|,&\bar{y}=\bar{x}+h\operatorname{sgn}(f_{i}(t,\bar{x},\mathscr{I}(\mu),u)),\\\
-h^{-1}\sum_{j=1}^{d}|f_{j}(t,\bar{x},\mathscr{I}(\mu),u)|,&\bar{y}=\bar{x},\\\
0,&\text{otherwise}.\end{array}\right.$ (8)

One can directly show that this Markov chain satisfies conditions (A1)–(A4)
with $B_{Q}=dRh^{-1}$ and $\varepsilon\triangleq\max\\{h,\sqrt{hdR}\\}$.

Notice that if $\mathcal{K}$ is compact, the phase space for Markov chain
introduced by rules (7), (8) is finite and equation (6) is a system of the
finite number of ODEs.

##  5 Model predictive control of the first-order mean field type system

In this section, we show that a feedback control in the mean filed type Markov
chain can be used directly to construct a motion in the first order mean filed
type control system.

Let $\mu_{0}\in\Sigma^{2}$,
$\zeta_{\mathcal{S}}\in\mathcal{U}^{\mathcal{S}}_{0,T}$. Notice that there
exists a unique motion in the mean field Markov chain on the time interval
$[0,T]$ produced by the control $\zeta_{\mathcal{S}}$ and the initial
distribution $\mu_{0}$. For shortness, we denote it by $\mu(\cdot)$. Below,
let $\hat{\zeta}$ assign to $x\in\mathcal{K}$ and $\bar{x}\in\mathcal{S}$ a
measure $\zeta_{\bar{x}}\in\mathcal{U}_{0,T}$. Further, let
$m_{0}\in\mathcal{P}^{2}(\mathcal{K})$, $\Delta=\\{s_{i}\\}_{i=0}^{n}$ be a
partition of $[0,T]$. A model predictive strategy for the first order mean
field game is constructed as follows.

  1. (D1)

Let $\pi_{0}$ be an optimal plan between $m_{0}$ and $\mathscr{I}(\mu_{0})$.
We define

$\alpha_{0}\triangleq(\operatorname{p}^{1},\hat{\zeta})\sharp\pi_{0}.$

  2. (D2)

Assume now that we already construct controls $\alpha_{i}$, $i=0,\ldots,k-1$,
and a flow of probabilities $m(\cdot)$ on $[0,s_{k}]$ such that

$\alpha_{i}\in\mathcal{A}_{s_{i},s_{i+1}}[m(s_{i})],\ \ i=0,\ldots,k-1,$
$m(t)=m(t,0,\alpha_{0}\diamond_{s_{1}}\ldots\diamond_{s_{k-1}}\alpha_{k-1}),\
\ t\in[0,s_{k}].$

Set $m_{k}\triangleq
m(s_{k},0,\alpha_{0}\diamond_{s_{1}}\ldots\diamond_{s_{k-1}}\alpha_{k-1})$ and
choose $\pi_{k}$ to be an optimal plan between $m_{k}$ and
$\mathscr{I}(\mu(s_{k}))$. As above $\pi_{k}(\cdot|x)$ is a disintegration of
this plan w.r.t. $m_{k}$. We put

$\alpha_{k}\triangleq(\operatorname{p}^{1},\hat{\zeta})\sharp\pi_{k}$

and, for $t\in[s_{k},s_{k+1}]$,

$m(t)\triangleq
m(s_{k},0,\alpha_{0}\diamond_{s_{1}}\ldots\diamond_{s_{k-1}}\alpha_{k-1}\diamond_{s_{k}}\alpha_{k}).$

######  Theorem 5.1.

If $\Delta=\\{s_{i}\\}_{i=0}^{n}$ is a partition of $[0,T]$, with
$d(\Delta)\leq 1$, while $\alpha_{0},\ldots,\alpha_{n-1}$ are constructed by
the rules (D1), (D2) and $m(\cdot)\triangleq
m(\cdot,0,\alpha_{0}\diamond_{s_{1}}\ldots\diamond_{s_{n-1}}\alpha_{n-1})$,
then

$\begin{split}W_{p}(m(t),\mathscr{I}(\mu(t)))\leq
C_{0}W_{2}(\mathscr{(}\mu_{0}),m_{0})+C_{1}&\varepsilon+C_{2}d^{1/2}(\Delta)\\\
&+C_{3}d(\Delta)+C_{4}\varepsilon
B_{Q}d(\Delta)+C_{5}B_{Q}d^{2}(\Delta).\end{split}$

Here $C_{0},\ldots,C_{5}$ are constants those depend only on $f$ and $T$.

######  Lemma 5.2.

Let $\mu(\cdot)$ be a distribution of agents in the mean field Markov chain,
$\mathbb{P}_{s,r}$ be its realization. Then,

$\mathbb{E}_{s,r}(\|X(t)-X(s)\|^{2}|X(s)=\bar{z})\leq\varepsilon^{2}(t-s)+C^{\prime}_{1}(t-s)^{3/2},$

where

$C^{\prime}_{1}\triangleq 4(R+1)e^{2(R+1)T}/3.$

###### Proof.

For fixed $\bar{z}\in\mathcal{S}$, let us denote

$q_{\bar{z}}(\bar{x})\triangleq\|\bar{x}-\bar{z}\|^{2}.$

We have that

$\begin{split}L_{t}[\mu,u]q_{\bar{z}}(\bar{x})&=\sum_{\bar{y}\in\mathcal{S}}Q_{\bar{x},\bar{y}}(t,\mu,u)\|\bar{y}-\bar{z}\|^{2}\\\
&=\sum_{\bar{y}\in\mathcal{S}}Q_{\bar{x},\bar{y}}(t,\mu,u)(\|\bar{y}-\bar{x}\|^{2}+\|\bar{x}-\bar{z}\|^{2}+2\langle\bar{y}-\bar{x},\bar{x}-\bar{z}\rangle)\\\
&=\sum_{\bar{y}\in\mathcal{S}}Q_{\bar{x},\bar{y}}(t,\mu,u)\|\bar{y}-\bar{x}\|^{2}+\Bigl{\langle}\sum_{\bar{y}\in\mathcal{S}}Q_{\bar{x},\bar{y}}(t,\mu,u)(\bar{y}-\bar{x}),\bar{x}-\bar{z}\Bigr{\rangle}.\end{split}$

Due to assumption (A4), we have that the first term is bounded by
$\varepsilon^{2}$. Moreover, condition (A3) implies that

$\Bigg{\|}\sum_{\bar{y}\in\mathcal{S}}Q_{\bar{x},\bar{y}}(t,\mu,u)(\bar{y}-\bar{x})\Bigg{\|}\leq
R+\varepsilon.$

Using these estimates and Definition 4.2, we conclude that that

$\begin{split}\mathbb{E}_{s,rT}(\|X(t)&-X(s)\|^{2}|X(s)=\bar{z})\\\
&\leq\varepsilon^{2}(t-s)+2\int_{s}^{t}(R+\varepsilon)(\mathbb{E}_{0,T}\|X(t^{\prime})-X(s)\||X(s)=\bar{z})dt^{\prime}.\end{split}$
(9)

Since we assume that $\varepsilon\leq 1$, we have that

$\mathbb{E}_{0,T}(\|X(t)-X(s)\|^{2}|X(s)=\bar{z})\leq
C^{\prime\prime}_{1}(t-s),$

where

$C^{\prime\prime}_{1}\triangleq e^{2(R+1)T}.$

Plugging this estimate to (9), we obtain the statement of the lemma. ∎

######  Lemma 5.3.

Assume that $s,r\in[0,T]$, $s<r$,
$\nu_{*}=(\nu_{*,\bar{x}})_{\bar{x}\in\mathcal{S}}\in\Sigma^{2}$,
$\zeta_{\mathcal{S}}\in\mathcal{U}^{\mathcal{S}}_{0,T}$,
$\mu(\cdot):[0,T]\rightarrow\Sigma^{2}$, while $\nu(\cdot)$ satisfies

$\frac{d}{dt}\nu(t)=\nu(t)Q(t,\mu(t),\zeta_{\mathcal{S}}(t)),\nu(s)=\nu_{*}.$
(10)

Then, for each $\bar{x}\in\mathcal{S}$,

$|\nu_{\bar{x}}(t)-\nu_{*,\bar{x}}|\leq B_{Q}(t-s).$

###### Proof.

We have that

$|\nu_{\bar{x}}(t)-\nu_{*,\bar{x}}|\leq\int_{s}^{t}\int_{U}\sum_{\bar{y}\in\mathcal{S}}\nu_{\bar{y}}(t^{\prime})\big{|}Q_{\bar{y},\bar{x}}(t^{\prime},\mu(t),u)\big{|}\zeta_{\bar{y}}(du|t^{\prime}))dt^{\prime}\leq
B_{Q}(t-s).$

∎

###### Proof of Theorem 5.1.

Let us estimate the squared Wasserstein distance $W_{2}^{2}(\mu(t),m(t))$ for
$t\in[s_{k},s_{k+1}]$, $k=0,\ldots,n-1$. Let $x_{k}(\cdot,y,\bar{z})$ be a
solution on $[s_{k},s_{k+1}]$ of the differential equation

$\frac{d}{dt}x(t)=\int_{U}f(t,x(t),m(t),u)\zeta_{\bar{z}}(du|t),\ \
x(s_{k})=y.$

Notice that $m(t)=x_{k}(t,\cdot,\cdot)\sharp\pi_{k}$, while by construction
$\mathscr{I}(\mu(t))\triangleq X(t)\sharp\mathbb{P}_{s_{k},s_{k+1}}$. Recall
that $\mathbb{P}_{s_{k},s_{k+1}}$ is a probability that realizes a flow of
distributions $\mu(\cdot)$ on $[s_{k},s_{k+1}]$, whereas
$\mathbb{E}_{s_{k},s_{k+1}}$ is the corresponding expectation. Thus, we have
that

$W_{2}^{2}(\mathscr{I}(\mu(t)),m(t))=\int_{\mathcal{K}\times\mathcal{S}}\mathbb{E}_{s_{k},s_{k+1}}\big{(}\|X(t)-x_{k}(t,y,\bar{z})\|^{2}|X(s_{k})=\bar{z}\big{)}\pi_{k}(d(y,\bar{z})).$
(11)

Now, let us evaluate the quantity
$\mathbb{E}_{s_{k},s_{k+1}}(\|X(t)-x_{k}(t,y,\bar{z})\|^{2}|X(s_{k})=\bar{z})$.
First, notice that

$\begin{split}\|X(t)-x_{k}(t&,y,\bar{z})\|^{2}\leq\|X(s_{k})-x_{k}(s_{k},y,\bar{z})\|^{2}\\\
+2\|X(&t)-X(s_{k})\|^{2}+2\|x_{k}(t,y,\bar{z})-x_{k}(s_{k},y,\bar{z})\|^{2}\\\
&+2\langle X(t)-X(s_{k}),X(s_{k})-x_{k}(s_{k},y,\bar{z})\rangle\\\ &-2\langle
x_{k}(t,y,\bar{z})-x_{k}(s_{k},y,\bar{z}),X(s_{k})-x_{k}(s_{k},y,\bar{z})\rangle.\end{split}$

Thus,

$\begin{split}\mathbb{E}_{s_{k},s_{k+1}}\big{(}\|X(t)-x_{k}&(t,y,\bar{z})\|^{2}|X(s_{k})=\bar{z}\big{)}\leq\|y-\bar{z}\|^{2}\\\
+2\mathbb{E}_{s_{k},s_{k+1}}&\big{(}\|X(t)-\bar{z}\|^{2}|X(s_{k})=\bar{z}\big{)}+2\|x_{k}(t,y,\bar{z})-y\|^{2}\\\
+\big{\langle}&\mathbb{E}_{s_{k},s_{k+1}}(X(t)-\bar{z}|X(s_{k})=\bar{z}),\bar{z}-y\big{\rangle}-\langle
x_{k}(t,y,\bar{z})-y,\bar{z}-y\rangle.\end{split}$ (12)

Due to Lemma 5.2, and the boundness of $f$, we have that

$\begin{split}2\mathbb{E}_{s_{k},s_{k+1}}\big{(}\|X(t)-\bar{z}&\|^{2}|X(s_{k})=\bar{z}\big{)}+2\|x_{k}(t,y,\bar{z})-y\|^{2}\\\
&\leq
2\varepsilon^{2}(t-s)+2C^{\prime}_{1}(t-s)^{3/2}+R^{2}(t-s)^{2}.\end{split}$
(13)

Further,

$\mathbb{E}_{s_{k},s_{k+1}}(X(t)-\bar{z}|X(s_{k})=\bar{z})=\mathbb{E}_{s_{k},s_{k+1}}\Bigg{(}\int_{s_{k}}^{t}\sum_{\bar{y}\in\mathcal{S}}Q_{X(t^{\prime}),y}(y-X(t^{\prime}))|X(s)=\bar{z}\Bigg{)}dt^{\prime}.$

This and condition (A3) yield that

$\begin{split}\Bigg{\|}\mathbb{E}_{s_{k},s_{k+1}}(X(t)-\bar{z}|X(&s_{k})=\bar{z})\\\
-\int_{s_{k}}^{t}\mathbb{E}_{s_{k},s_{k+1}}\Bigg{(}&\int_{U}f(t^{\prime},X(t^{\prime}),\mathscr{I}(\mu(t)),u)\zeta_{X(t^{\prime})}(du|t^{\prime})|X(s)=\bar{z}\Bigg{)}dt^{\prime}\Bigg{\|}\\\
&{}\hskip 210.0pt\leq\varepsilon(t-s_{k}).\end{split}$

Recall that

$\begin{split}\mathbb{E}_{s_{k},s_{k+1}}\Bigg{(}\int_{U}f(t^{\prime},X(t^{\prime}),&\mathscr{I}(\mu(t)),u)\zeta_{X(t^{\prime})}(du|t^{\prime})|X(s)=\bar{z}\Bigg{)}dt^{\prime}\\\
&=\sum_{\bar{x}\in\mathcal{S}}\int_{U}f(t^{\prime},\bar{x},\mathscr{I}(\mu(t)),u)\zeta_{X(t^{\prime})}(du|t^{\prime})\nu_{*,\bar{x}}.\end{split}$

Thus, due to Lemma 5.3, we have that

$\begin{split}\Bigg{\|}\mathbb{E}_{s_{k},s_{k+1}}(X(t)-\bar{z}|X(s_{k})=\bar{z})-\int_{s_{k}}^{t}\int_{U}f(t^{\prime},\bar{z},&\mathscr{I}(\mu(t)),u)\zeta_{\bar{z}}(du|t^{\prime})dt^{\prime}\Bigg{\|}\\\
&\leq\varepsilon(t-s_{k})+RB_{Q}(t-s_{k})^{2}.\end{split}$

This and the definition of the motion $x(\cdot,y,\bar{z})$ imply that

$\begin{split}\Bigg{|}\big{\langle}\mathbb{E}_{s_{k},s_{k+1}}(&X(t)-\bar{z}|X(s_{k})=\bar{z}),\bar{z}-y\big{\rangle}-\langle
x_{k}(t,y,\bar{z})-y,\bar{z}-y\rangle\Bigg{|}\\\
&\leq\int_{s_{k}}^{t}\int_{U}\|f(t^{\prime},\bar{z},\mathscr{I}(\mu(t)),u)-f(t^{\prime},x(t^{\prime},y,\bar{z}),m(t),u)\|\zeta_{\bar{z}}(du|t^{\prime})dt^{\prime}\cdot\|y-\bar{z}\|\\\
&{}\hskip
200.0pt+(\varepsilon(t-s_{k})+RB_{Q}(t-s_{k})^{2})\|y-\bar{z}\|.\end{split}$

Using the Lipschitz continuity of the function $f$, we obtain the following:

$\begin{split}\Bigg{|}\big{\langle}\mathbb{E}_{s_{k},s_{k+1}}(X(t)-\bar{z}|X(s_{k})=\bar{z}),&\bar{z}-y\big{\rangle}-\langle
x_{k}(t,y,\bar{z})-y,\bar{z}-y\rangle\Bigg{|}\\\
\leq&C_{f}R(t-s_{k})^{2}\|y-\bar{z}\|\\\
&+C_{f}\int_{s_{k}}^{t}W_{2}(\mathscr{I}(\mu(t^{\prime})),m(t^{\prime}))\|y-\bar{z}\|\\\
&+(\varepsilon(t-s_{k})+RB_{Q}(t-s_{k})^{2})\|y-\bar{z}\|.\end{split}$

Plugging this estimate into (12) and taking into account (13), we derive the
estimate:

$\begin{split}\mathbb{E}_{s_{k},s_{k+1}}\big{(}\|X(t)-x_{k}(t&,y,\bar{z})\|^{2}|X(s_{k})=\bar{z}\big{)}\leq\|y-\bar{z}\|^{2}\\\
&+2\varepsilon^{2}(t-s_{k})+2C^{\prime}_{1}(t-s)^{3/2}+R(t-s_{k})^{2}\\\
&+C_{f}R(t-s_{k})^{2}+C_{f}R(t-s_{k})\|y-\bar{z}\|^{2}\\\
&+C_{f}(t-s_{k})W_{2}^{2}(\mathscr{I}(\mu(s_{k})),m(s_{k}))+3C_{f}(t-s_{k})\|y-\bar{z}\|^{2}\\\
&+C_{f}\varepsilon^{2}(t-s_{k})^{2}+C^{\prime}_{1}C_{f}(t-s_{k})^{5/2}+RC_{f}(t-s_{k})^{2}\\\
&+(\varepsilon+RB_{Q}(t-s_{k}))^{2}(t-s_{k})+(t-s_{k})\|y-\bar{z}\|^{2}\\\
\leq\|y-\bar{z}\|^{2}&+C^{\prime}_{2}(t-s_{k})\|y-\bar{z}\|^{2}+C^{\prime}_{3}(t-t_{k})W_{2}^{2}(\mathscr{I}(\mu(s_{k})),m(s_{k}))\\\
&+3\varepsilon(t-s_{k})+2C^{\prime}_{1}(t-s_{k})^{3/2}+C^{\prime}_{4}(t-s_{k})^{2}\\\
&+C^{\prime}_{5}\varepsilon
B_{Q}(t-s_{k})^{2}+R^{2}B_{Q}^{2}(t-s_{k})^{3}.\end{split}$

Due to (11), we arrive at the inequality

$\begin{split}W_{2}^{2}(\mathscr{I}(\mu(t)),m(t))\leq(1+C^{\prime}_{6}&(t-s_{k}))W_{2}^{2}(\mathscr{I}(\mu(s_{k})),m(s_{k}))\\\
&+3\varepsilon(t-s_{k})+2C^{\prime}_{1}(t-s_{k})^{3/2}+C^{\prime}_{4}(t-s_{k})^{2}\\\
&+C^{\prime}_{5}\varepsilon
B_{Q}(t-s_{k})^{2}+R^{2}B_{Q}^{2}(t-s_{k})^{3}.\end{split}$ (14)

Applying this inequality sequentially, we deduce the statement of the theorem.
∎

##  6 Model predictive control for Markov chains

In this section, given an initial distribution for the deterministic mean
field type control system $m_{0}\in\mathcal{P}^{2}(\mathcal{K})$, a
distribution of controls $\alpha\in\mathcal{U}_{0,T}$ such that
$\alpha\in\mathcal{A}_{0,T}[m_{0}]$, and an initial system for mean field
Markov chain $\mu_{0}\in\Sigma^{2}$, we construct a feedback strategy
$\zeta_{\mathcal{S}}$ such that the corresponding motion of the Markov chain
starting at $\mu_{0}$ approximates the motion $m(\cdot,0,\alpha)$. Within this
section, we denote

$m(\cdot)\triangleq m(\cdot,0,\alpha).$

Further, for $(y,\xi)\in\mathbb{R}^{d}\times\mathcal{U}_{0,T}$ and
$s\in[0,T]$, we put

$\mathscr{T}^{s}(y,\xi)\triangleq x(s,0,y,m(\cdot),\xi).$

Notice that, if $(y^{\prime},\xi^{\prime})=\mathscr{T}^{s}(y,\xi)$, then

$x(\cdot,0,y,m(\cdot),\xi)=x(\cdot,s,y^{\prime},m(\cdot),\xi^{\prime}).$

Informally, the operator $\mathscr{T}^{s}$ transfers the initial condition and
the control from $t=0$ to the time $s$.

Below, if $\zeta_{\mathcal{S}},\zeta^{\prime}_{\mathcal{S}}$ are feedback
controls for the Markov chain on $[s,r]$ and $[r,\theta]$ respectively, then
we denote by $\zeta_{\mathcal{S}}\diamond_{r}\zeta_{\mathcal{S}}^{\prime}$ the
feedback control such that
$\zeta_{\mathcal{S}}\diamond_{r}\zeta_{\mathcal{S}}^{\prime}=(\zeta_{\bar{x}}\diamond_{r}\zeta_{\bar{x}}^{\prime})_{\bar{x}\in\mathcal{S}}$.

Finally, let $\Delta=\\{s_{i}\\}_{i=0}^{n}$ be a partition of $[0,T]$.

The construction is stepwise.

  1. (M1)

Let $\pi_{0}$ be an optimal plan between $m_{0}$ and $\mathscr{I}(\mu_{0})$
and let $\pi_{0}(\cdot|\bar{x})$ be its disintegration w.r.t.
$\mathscr{I}(\mu_{0})$. We define a probability
$\zeta_{\mathcal{S},0}=(\zeta_{\bar{x},0})_{\bar{x}}\in\mathcal{U}_{s_{0},s_{1}}^{\mathcal{S}}$
by the rule: for $\phi\in C_{b}([s_{0},s_{1}]\times U)$

$\begin{split}\int_{[s_{0},s_{1}]\times
U}\phi(t,u)&\zeta_{\bar{x},0}(d(t,u))\\\
&\triangleq\int_{\mathcal{K}}\int_{\mathcal{U}_{s_{0},s_{1}}}\int_{[s_{0},s_{1}]\times
U}\phi(t,u)\xi(d(t,u))\alpha_{0}(\xi|y)\pi_{0}(dy|\bar{x}).\end{split}$

Hereinafter, $\alpha_{0}$ is a restriction of $\alpha$ on $[s_{0},s_{1}]$.

  2. (M2)

Assume now that we already constructed controls
$\zeta_{\mathcal{S},0},\ldots,\zeta_{\mathcal{S},k-1}$ and put, for
$t\in[0,s_{k}]$,

$\mu(t)\triangleq\mu(t,0,\mu_{0},\zeta_{\mathcal{S},0}\diamond_{s_{1}}\ldots\diamond_{s_{k-1}}\zeta_{\mathcal{S},k-1}).$

To extend the control to the next time step, we first introduce $\alpha_{k}$
to be a restriction of the distribution of controls
$\mathscr{T}^{s_{k}}\sharp\alpha$ to the time interval $[s_{k},s_{k+1}]$.
Further, let $\pi_{k}$ be an optimal plan between $\mathscr{I}(\mu(s_{k}))$
and $m(s_{k})$. A feedback control
$\zeta_{\mathcal{S},k}=(\zeta_{\bar{x},k})_{\bar{x}\in\mathcal{S}}\in\mathcal{U}^{\mathcal{S}}_{[s_{k},s_{k+1}]}$
is defined by the rule: if $\phi\in C([s_{k},s_{k+1}]\times U)$, then

$\begin{split}\int_{[s_{0},s_{1}]\times
U}\phi(t,u)&\zeta_{\bar{x},k}(d(t,u))\\\
&\triangleq\int_{\mathcal{K}}\int_{\mathcal{U}_{s_{k},s_{k+1}}}\int_{[s_{k},s_{k+1}]\times
U}\phi(t,u)\xi(d(t,u))\alpha_{k}(d\xi|y)\pi_{k}(dy|\bar{x}).\end{split}$

######  Theorem 6.1.

Let conditions (A1)–(A4) hold true and let
$\mu(\cdot)\triangleq\mu(\cdot,0,\mu_{0},\zeta_{\mathcal{S},0}\diamond_{s_{1}}\ldots\diamond_{s_{n-1}}\zeta_{\mathcal{S},n-1})$,
then

$\begin{split}W_{p}(m(t),\mathscr{I}(\mu(t)))\leq
C_{0}W_{2}(\mathscr{I}(\mu_{0})&,m_{0})+C_{1}\varepsilon+C_{2}d^{1/2}(\Delta)\\\
&+C_{3}d(\Delta)+C_{4}\varepsilon
B_{Q}d(\Delta)+C_{5}B_{Q}d^{2}(\Delta).\end{split}$

Here $C_{0},\ldots,C_{5}$ are the same constant as in Theorem 5.1.

###### Proof.

The proof mimics the proof of Theorem 5.1. We consider a time interval
$[s_{k},s_{k+1}]$ and choose $\mathbb{P}_{s_{k},s_{k+1}}$ to be a realization
of the flow $\mu(\cdot)$ on $[s_{k},s_{k+1}]$. If $t\in[s_{k-1},s_{k}]$, then

$\begin{split}W_{2}^{2}(m&(t),\mathscr{I}(\mu(t)))\\\
&\leq\int_{\mathcal{K}\times\mathcal{S}}\int_{\mathcal{U}_{s_{k},s_{k+1}}}\mathbb{E}_{s_{k},s_{k+1}}(\|X(t)-x_{k}(t,y,\xi)\|^{2}|X(s_{k})=\bar{z})\pi_{k}(d(y,\bar{z})).\end{split}$
(15)

Here, as above, we denote

$x_{k}(t,y,\xi)\triangleq x(t,s_{k},y,m(\cdot),\xi).$

Further, we have that

$\begin{split}\mathbb{E}_{s_{k},s_{k+1}}&(\|X(t)-x_{k}(t,y,\xi)\|^{2}|X(s_{k})=\bar{z})\leq\|\bar{z}-y\|^{2}\\\
&+2\mathbb{E}_{s_{k},s_{k+1}}(\|X(t)-X(s_{k})\|^{2}|X(s_{k})=\bar{z})+2\|x_{k}(t,y,\xi)-y\|^{2}\\\
&+2\mathbb{E}_{s_{k},s_{k+1}}(\langle
X(t)-X(s_{k}),\bar{z}-y\rangle|X(s_{k})=\bar{z})\\\ &-2\langle
x_{k}(t,y,\xi)-y,\bar{z}-y\rangle.\end{split}$

Using Lemma 5.2, we conclude that

$\begin{split}2\mathbb{E}_{s_{k},s_{k+1}}\big{(}\|X(t)-X(s_{k})&\|^{2}|X(t_{k})=\bar{z}\big{)}+2\|x_{k}(t,y,\bar{z})-y\|^{2}\\\
&\leq
2\varepsilon^{2}(t-s)+2C^{\prime}_{1}(t-s)^{3/2}+R^{2}(t-s)^{2}.\end{split}$
(16)

Further, as in the proof of Theorem 5.1, we have that

$\begin{split}\Bigg{\|}\mathbb{E}_{s_{k},s_{k+1}}(X(t)-\bar{z}|X(s_{k})=\bar{z})-\int_{s_{k}}^{t}\int_{U}f(t^{\prime},\bar{z},&\mathscr{I}(\mu(t^{\prime})),u)\zeta_{\bar{z},k}(du|t^{\prime})dt^{\prime}\Bigg{\|}\\\
&\leq\varepsilon(t-s_{k})+RB_{Q}(t-s_{k})^{2}.\end{split}$

Simultaneously,

$x_{k}(t,y,\xi)-y=\int_{s_{k}}^{t}\int_{U}f(t^{\prime},x_{k}(t^{\prime},y,\xi),m(t^{\prime}),u)\xi(du|t^{\prime})dt^{\prime}$

Plugging this into 15 and taking into account the definition of
$\zeta_{\mathcal{S},k}$, we conclude that

$\begin{split}W_{2}^{2}(m&(t),\mathscr{I}(\mu(t)))\leq
W_{2}^{2}(m(s_{k}),\mathscr{I}(\mu(s_{k})))\\\
&+2\varepsilon^{2}(t-s)+2C^{\prime}_{1}(t-s)^{3/2}+R(t-s)^{2}+\varepsilon(t-s_{k})+B_{Q}(t-s_{k})^{2}\\\
&+\int_{\mathcal{K}\times\mathcal{S}}\int_{\mathcal{U}_{s_{k},s_{k+1}}}\Bigg{\|}\int_{U}f(t^{\prime},\bar{z},\mathscr{I}(\mu(t^{\prime})),u)\xi(du|t^{\prime})\\\
&{}\hskip
100.0pt-\int_{U}f(t^{\prime},x_{k}(t^{\prime},y,\xi),m(t^{\prime}),u)\xi(du|t^{\prime})\Bigg{|}\cdot\|\bar{z}-y\|.\end{split}$

Estimating the last term as in the proof of Theorem 5.1, we obtain the
inequality:

$\begin{split}W_{2}^{2}(\mathscr{I}(\mu(t)),m(t))\leq(1+C^{\prime}_{6}&(t-s_{k}))W_{2}^{2}(\mathscr{I}(\mu(s_{k})),m(s_{k}))\\\
&+3\varepsilon(t-s_{k})+2C^{\prime}_{1}(t-s_{k})^{3/2}+C^{\prime}_{4}(t-s_{k})^{2}\\\
&+C^{\prime}_{5}\varepsilon
B_{Q}(t-s_{k})^{2}+R^{2}B_{Q}^{2}(t-s_{k})^{3}.\end{split}$ (17)

Here the constants are the same as in (14). Applying (17) sequentially, we
arrive at the statement of the theorem. ∎

##  7 Hausdorff distance between bundles of flows

In this short section, we consider the bundles of flows of probabilities
generated by the original first order mean field type control system and the
mean field Markov chain. To define them, let

$\mathcal{X}(m_{0})\triangleq\big{\\{}m(\cdot,0,\alpha):\alpha\in\mathcal{A}_{0,T}[m_{0}]\big{\\}},$
$\mathcal{X}_{Q}(\mu_{0})\triangleq\big{\\{}\mu(\cdot,0,\mu,\zeta_{\mathcal{S}}):\zeta_{\mathcal{S}}\in\mathcal{U}_{0,T}^{\mathcal{S}}\big{\\}}.$

Notice that $\mathcal{X}(m_{0})\subset
C([0,T];\mathcal{P}^{2}(\mathbb{R}^{d}))$, while $\mathcal{X}_{Q}\subset
C([0,T];\Sigma^{2})$

If $m(\cdot)\in C([0,T];\mathcal{P}^{2}(\mathbb{R}^{d}))$, $\mu\in
C([0,T];\Sigma^{2})$, we denote

$\mathbbm{d}(m(\cdot),\mu(\cdot))\triangleq\sup_{t\in[0,T]}W_{2}(m(t),\mathscr{I}(\mu(t))).$

If $\Upsilon_{1}\subset C([0,T];\mathcal{P}^{2}(\mathbb{R}^{d}))$,
$\Upsilon_{2}\subset C([0,T],\Sigma^{2})$ are closed sets, then we introduce
the Hausdorff distance in the standard way:

$\mathbbm{H}(\Upsilon_{1},\Upsilon_{2})\triangleq\max\Bigg{\\{}\sup_{m(\cdot)\in\Upsilon_{1}}\inf_{\mu(\cdot)\in\Upsilon_{2}}\mathbbm{d}(m(\cdot),\mu(\cdot)),\sup_{\mu(\cdot)\in\Upsilon_{2}}\inf_{m(\cdot)\in\Upsilon_{1}}\mathbbm{d}(m(\cdot),\mu(\cdot))\Bigg{\\}}.$

The main result of this short section is the following.

######  Proposition 7.1.

Assume that the approximation conditions (A1)–(A4) are in force. Then,

$\mathbbm{H}(\mathcal{X}(m_{0}),\mathcal{X}_{Q}(\mu_{0}))\leq
C_{0}W_{2}(\mathscr{I}(\mu_{0}),m_{0})+C_{1}\varepsilon,$

where $C_{0}$ and $C_{1}$ are constants dependent only on $f$ and $T$.

###### Proof.

From Theorem 6.1, it follows that, given $m(\cdot)\in\mathcal{X}(m_{0})$ and
arbitrary $\delta>0$, one can find a partition of $[0,T]$
$\Delta=\\{s_{i}\\}_{i=0}^{n}$ and controls
$\zeta_{\mathcal{S}},\ldots,\zeta_{\mathcal{S},n-1}$ such that

$\mathbbm{d}(m(\cdot),\mu(\cdot))\leq
C_{0}W_{2}(\mathscr{(}\mu_{0}),m_{0})+C_{1}\varepsilon+\delta,$

where
$\mu(\cdot)=\mu(\cdot,0,\mu_{0},\zeta_{\mathcal{S},0}\diamond_{s_{1}}\ldots\diamond_{s_{n-1}}\zeta_{\mathcal{\S},n-1})$.
Passing to the limit while $\delta\rightarrow 0$, we conclude that

$\sup_{m(\cdot)\in\mathcal{X}(m_{0})}\inf_{\mu(\cdot)\in\mathcal{X}_{Q}(\mu_{0})}\mathbbm{d}(m(\cdot),\mu(\cdot))\leq
C_{0}W_{2}(\mathscr{I}(\mu_{0}),m_{0})+C_{1}\varepsilon.$

The inequality

$\sup_{\mu(\cdot)\in\mathcal{X}_{Q}(\mu_{0})}\inf_{m(\cdot)\in\mathcal{X}(m_{0})}\mathbbm{d}(m(\cdot),\mu(\cdot))\leq
C_{0}W_{2}(\mathscr{I}(\mu_{0}),m_{0})+C_{1}\varepsilon$

is proved in the similar way using Theorem 5.1. ∎

## References

  * [1] N. Ahmed and X. Ding.  Controlled McKean-Vlasov equation.  Commun. Appl. Anal., 5:183–206, 2001.
  * [2] L. Ambrosio, N. Gigli, and G. Savaré.  Gradient flows: in metric spaces and in the space of probability measures.  Lectures in Mathematics. ETH Zurich. Birkhäuser, Basel, 2005.
  * [3] Y. Averboukh.  Approximate solutions of continuous-time stochastic games.  SIAM J. Control Optim., 54(5):2629–2649, 2016.
  * [4] Y. Averboukh.  Viability theorem for deterministic mean field type control systems.  Set-Valued Var. Anal., 26(4):993–1008, 2018.
  * [5] Y. Averboukh.  Lattice approximations of the first-order mean field type differential games.  Nonlinear Differ. Equ. Appl., 28(6):42, 2021.  Id/No 65.
  * [6] Y. Averboukh and D. Khlopin.  Pontryagin maximum principle for the deterministic mean field type optimal control problem via the lagrangian approach.  Preprint at arXiv:2207.01892, 2022.
  * [7] Y. Averboukh, A. Marigonda, and M. Quincampoix.  Extremal shift rule and viability property for mean field-type control systems.  J. Optim. Theory Appl., 189(1):244–270, 2021.
  * [8] Y. V. Averboukh.  A mean field type differential inclusion with upper semicontinuous right-hand side.  Vestn. Udmurt. Univ.: Mat. Mekhanika Komp’yuternye Nauki, 32(4):489–501, 2022.
  * [9] E. Bayraktar, A. Cosso, and H. Pham.  Randomized dynamic programming principle and Feynman-Kac representation for optimal control of McKean-Vlasov dynamics.  Trans. Amer. Math. Soc., 370:2115–2160, 2018.
  * [10] N. Bellomo, B. Piccoli, and A. Tosin.  Modeling crowd dynamics from a complex system viewpoint.  Math. Models Methods Appl. Sci., 22:1230004, 29, 2012.
  * [11] M. Bivas and M. Quincampoix.  Nonsmooth feedback control for multi-agent dynamics.  Set-Valued Var. Anal., 29(2):501–518, 2021.
  * [12] B. Bonnet and H. Frankowska.  Differential inclusions in Wasserstein spaces: the Cauchy-Lipschitz framework.  J. Differ Equ., 271:594–637, 2021.
  * [13] B. Bonnet and H. Frankowska.  Viability and exponentially stable trajectories for differential inclusions in Wasserstein spaces.  In 2022 IEEE 61st Conference on Decision and Control (CDC), pages 5086–5091. IEEE, 2022.
  * [14] B. Bonnet-Weil and H. Frankowska.  Carathéodory theory and a priori estimates for continuity inclusions in the space of probability measures.  Preprint at arXiv:2302.00963, 2023.
  * [15] R. Buckdahn, B. Djehiche, and J. Li.  A general stochastic maximum principle for SDEs of mean-field type.  Appl. Math. Optim., 64(2):197–216, 2011.
  * [16] F. Bullo, J. Cortés, and S. Martínez.  Distributed control of robotic networks: a mathematical approach to motion coordination algorithms. Princeton Ser. Appl. Math. Princeton, NJ: Princeton University Press, 2009\.
  * [17] R. Carmona and F. Delarue.  Forward-backward stochastic differential equations and controlled McKean–Vlasov dynamics.  Ann. Probab., 43(5):2647–2700, 2015.
  * [18] R. M. Colombo, M. Garavello, and M. Lécureux-Mercier.  Non-local crowd dynamics.  C. R. Acad. Sci. Paris Sér. I Math., 349(13-14):769–772, 2011\.
  * [19] R. M. Colombo and M. D. Rosini.  Pedestrian flows and non-classical shocks.  Math. Methods Appl. Sci., 28(13):1553–1567, 2005.
  * [20] E. Cristiani, B. Piccoli, and A. Tosin.  Multiscale modeling of pedestrian dynamics, volume 12 of MS&A, Model. Simul. Appl. Springer, 2014.
  * [21] C. Dellacherie and P. Meyer.  Probabilities and potential.  North-Holland Publishing Co., Amsterdam, 1978.
  * [22] B. Djehiche and S. Hamadène.  Optimal control and zero-sum stochastic differential game problems of mean-field type.  Appl. Math. Optim., 81:933–960, 2020.
  * [23] L. Grüne and J. Pannek.  Nonlinear Model Predictive Control. Theory and Algorithms.  Springer, 2011.
  * [24] C. Jimenez, A. Marigonda, and M. Quincampoix.  Optimal control of multiagent systems in the Wasserstein space.  Calc. Var. Partial Differ. Equ., 59:Article number: 58, 2020.
  * [25] V. N. Kolokoltsov.  Markov Processes, Semigroups and Generators.  De Gruyter, Berlin, 2011.
  * [26] N. N. Krasovskii and A. N. Kotelnikova.  Unification of differential games, generalized solutions of the hamilton-jacobi equations, and a stochastic guide.  Differ. Equ., 45(11):1653–1668, 2009.
  * [27] N. N. Krasovskii and A. N. Kotelnikova.  An approach-evasion differential game: stochastic guide.  Proc. Steklov Inst. Math., 269 Supplement(1 Supplement):191–213, 2010.
  * [28] N. N. Krasovskii and A. N. Kotelnikova.  On a differential interception game.  Proc. Steklov Inst. Math., 268(1):161–206, 2010.
  * [29] N. N. Krasovskii and A. N. Kotelnikova.  Stochastic guide for a time-delay object in a positional differential game.  Proc. Steklov Inst. Math., 277 Supplement(1 Supplement):145–151, 2012.
  * [30] M. Laurière and O. Pironneau.  Dynamic programming for mean-field type control.  C. R. Math. Acad. Sci. Paris, 352(9):707–713, 2014.
  * [31] H. P. McKean.  A class of Markov processes associated with nonlinear parabolic equations.  Proc. Natl. Acad. Sci. U.S.A., 56:1907–1911, 1966.
  * [32] H. Pham and X. Wei.  Dynamic programming for optimal control of stochastic McKean-Vlasov dynamics.  SIAM J. Control Optim., 55:1069–1101, 2017.
  * [33] H. Pham and X. Wei.  Bellman equation and viscosity solutions for mean-field stochastic control problem.  ESAIM Control Optim. Calc. Var., 24(1):437–461, 2018.
  * [34] N. Pogodaev.  Optimal control of continuity equations.  Nonlinear Differ. Equ. Appl., 23:Art21, 24 pp., 2016.
  * [35] A.-S. Sznitman.  Topics in propagation of chaos.  In Ecole dété de probabilités de Saint-Flour XIX – 1989, volume 1464 of Lecture Notes in Math. Springer, Berlin, 1991.
  * [36] A. Vlasov.  On vibration properties of electron gas.  J. Exp. Theor. Phys., 8(3):291–318, 1938.  (in Russian).
  * [37] A. Vlasov.  Many-particle theory and its application to plasma.  Gordon and Breach, New York, 1961.

Yurii Averboukh: | Krasovskii Institute of Mathematics and Mechanics,
---|---
| 16, S. Kovalevskaya str, Yekaterinburg, Russia
| e-mail:mailto:ayv<EMAIL_ADDRESS>


# Partner Personas Generation for Diverse Dialogue Generation

Hongyuan Lu, Wai Lam, Hong Cheng, Helen M. Meng
The Chinese University of Hong Kong
<EMAIL_ADDRESS>

###### Abstract

Incorporating personas information allows diverse and engaging responses in
dialogue response generation. Unfortunately, prior works have primarily
focused on self personas and have overlooked the value of partner personas.
Moreover, in practical applications, the availability of ground truth partner
personas is often not the case. This paper attempts to tackle these issues by
offering a novel framework that leverages automatic partner personas
generation to enhance the succeeding dialogue generation. We incorporate
reinforcement learning with a dedicatedly designed critic network for reward
judgement. Experimental results from both automatic and human evaluation
demonstrate: a) Our framework is capable of generating relevant, informative
and coherent partner personas, even compared to the ground truth partner
personas. b) Generated partner personas enhance the succeeding response
generation, thus surpassing our baselines and comparison model when partner
personas are missing during the inference stage. c) Our framework generates
responses that are more informative and engaging than our baseline conditioned
on the ground truth partner personas during inference. d) Our dedicatedly
designed critic network reinforces our framework effectively. Finally, our
framework gives better explainability and reduces the demands for external
databases for partner personas.111We will release our data and our code upon
publication.

##  1 Introduction

Building informative and engaging dialogue agents (Zhang et al., 2019; Roller
et al., 2020; Bao et al., 2021) is a popular research direction within the
area of natural language processing. For the sake of engagement, diverse and
consistent responses (Song et al., 2020, 2021) are important factors, and
personas information (Zhang et al., 2018) gives rise to both of the factors.
There are two types of personas information, namely self persona and partner
persona. The former refers to a self profile consisting of several sentences
representing the dialogue agent. Such a persona allows the agent to produce
consistent responses rather than solely relying on the personas information
(Kim et al., 2020) that is randomly learned and embedded in the model
parameters. The latter also refers to a profile but representing the user.
Leveraging such partner personas is helpful for dialogue generation (Gu et
al., 2021). Therefore, we exploit partner personas for diverse dialogue
generation.

Unfortunately, the user profile could be commonly missing due to the cold-
start (Li et al., 2020) when deploying online dialogue agents or for newly
registered users. Most of the works, if not all, (Mazaré et al., 2018; Song et
al., 2019; Gu et al., 2019; Zhao et al., 2019) have been either overlooking
the value of partner personas or simply focusing on the impractical situation
where partner personas guarantee to exist in both training and inference
stages. Our work demonstrates the importance of diverse partner personas
generation, and we particularly investigate the practical situation when
partner personas are missing in the inference stage. Such an investigation is
essential as there is no guarantee that the ground truth partner personas
always exist. Ultimately, our proposed framework can produce even more diverse
and engaging responses than our baseline that conditions on the ground truth
partner personas. We demonstrate a case study in Section 5.2 that illustrates
this advantage.

To our best knowledge, this is the first attempt to formulate partner personas
prediction in a generative manner that boosts the performance of the
succeeding dialogue generation. Our work is motivated by three underlying
hypotheses: i) Partner personas generation is plausible given the self
personas and dialogue context. ii) Generated personas are more diverse and
interesting than the retrieved ground truth. iii) Such diverse generated
personas help to produce diverse succeeding dialogue responses. Our automatic
and human evaluation results support these hypotheses, and this paper paves
the way to exploit generative partner personas for diverse dialogue
generation.

We develop a novel framework composed of three major components, namely a
partner personas generator, a dialogue response generator and a critic
network. We use a partner personas generator to generate partner personas,
which the dialogue response generator uses for succeeding dialogue response
generation. We employ reinforcement learning with a dedicatedly designed
critic network that propagates the reward back to the generators.

Prior works have investigated retrieval-based partner persona (Zhang et al.,
2018; Song et al., 2019). The human-constructed ground truth personas serve as
the upper bound for such retrieval-based systems, and we argue that the ground
truth are not diverse enough. We observe that the generative counterpart
generates relevant, informative and coherent partner personas, which further
diversifies the succeeding dialogue response generation. It follows another
advantage that our framework does not need an external database to retrieve
from (Madotto et al., 2020; Xu et al., 2021).

One close work to ours is a multi-task framework for meta-learning (Lee et
al., 2021) that uses personas reconstruction as an auxiliary task to improve
response consistency. The differences are that theirs does not differentiate
between self personas and partner personas, while ours does. Their model
focuses on meta-learning, while ours does not set such a constraint. Theirs
indicates an improvement over personality consistency only, while ours focuses
on diverse dialogue generation. We conduct an empirical comparison with their
model, reconstructing the partner personas. Experimental results indicate that
such a multi-task model does not work well in our problem setting.

We compare our proposed framework with some competitive baselines. The
automatic and human evaluation results indicate that our framework can
generate even more diverse and engaging responses than the baseline
conditioned on ground truth partner personas. It leads to a conclusion that i)
Partner personas generation is plausible. ii) The generated partner personas
are more diverse than the ground truth partner personas. iii) Our framework
produces even more diverse and engaging responses than our competitive
baselines that condition on the ground truth partner personas.

##  2 Related Work

###  2.1 Personalized Dialgoue Generation

Conditioning on personas helps produce informative and engaging responses. The
largest multi-turn dialogue dataset conditioned on personal profiles is
PersonaChat, in which two crowdsourcers converse and find more about each
other. To better utilise the self personas in generating consistent responses,
the community has proposed quite a lot of methods. Mazaré et al. (2018)
employs a pre-training stage based on dedicatedly extracted large-scale
persona-based dialogues and fine-tunes the model on PersonaChat. Zhao et al.
(2019) fuses information in personas and dialogue context into individual
contextualised representations by attending to different parts of both. Gu et
al. (2019) exploits the interaction between personas, dialogue context and
response to improve retrieval-based dialogue agents. Lee et al. (2021)
utilises multi-task learning for improved personality consistency in the meta-
learning scenario. Gu et al. (2021) employs four different strategies for
personas fusing, which learns to use self persona and partner persona in a
more effective manner. There have also been several works based on GPT (Wolf
et al., 2019). However, most of these prior works focus on exploiting self
personas rather than partner personas, and they have been assuming that the
ground truth partner personas guarantee to exist.

###  2.2 Reinforcement Learning

Reinforcement learning (RL), or specifically, policy gradient methods
(Williams, 1992), have been frequently adopted to both task-oriented dialogue
agents (Roman Roman et al., 2020; Deng et al., 2021) or open-domain chitchat
agents (Li et al., 2016; Saleh et al., 2019). It can either propagate non-
differentiable loss (Cai et al., 2019) or optimize an expert reward such as
ease of answering (Li et al., 2016). It also adopts a scenario where a user
simulator and a dialogue agent interact, and an expert reward function is
defined to assign the goodness to each response generated (Roman Roman et al.,
2020).

Figure 1: An example of the inference flow that shows the generated partner
personas and the incorporation of partner personas generation into response
generation.

Figure 2: The illustrated reinforcement learning strategy that directly
backpropagates the response-related rewards from the critic network to the
partner personas generator and the dialogue response generator.

##  3 Proposed Framework

We develop a novel framework composed of three major components, namely a
partner personas generator, a dialogue response generator and a critic network
used by reinforcement learning. Figure 1 depicts the inference flow of our
setting. The input dialogue context with self persona is first fed into the
partner personas generator. The generated partner personas output is then
concatenated with the dialogue context and the self personas as the input into
the dialogue response generator. In the beginning, we train our partner
personas generator and dialogue response generator separately under supervised
learning. In the training stage, we use the ground truth partner personas to
train the dialogue response generator, and we replace it with generated
partner personas in the inference stage. After the supervised learning stage,
the second stage is a reinforcement learning stage which jointly optimizes
both partner personas generator and dialogue response generator as depicted in
Figure 2. Such framework has two advantages: i) The partner personas generator
can be trained directly using the reward signal that is relevant to dialogue
response generation. ii) The dialogue response generator trained using ground
truth partner personas can be further fine-tuned on the generated partner
personas.222Section 5.4 presents an ablation study on reinforcement learning
that supports this claim. Particularly, we employ a dedicatedly designed
critic network that receives generated partner personas and generated dialogue
responses as the input and output a reward that measures the relevance between
the generated personas and responses and propagates back to the generators.

###  3.1 Partner Personas Generation

A Seq2Seq neural network (Sutskever et al., 2014) is adopted as our partner
personas generator for the task of partner personas generation (PPG). The
concatenation of dialogue context c and self personas $\mathbf{s}$ is fed as
an input into the partner personas generator. The personas generator then
outputs an approximated partner personas $\mathbf{\hat{p}}$ conditioned on the
input, which maximises the following conditional likelihood:

$\displaystyle
P(\mathbf{\hat{p}}\mid\mathbf{s},\mathbf{c})=\prod_{t=1}^{T}P(\hat{p}_{t}\mid\hat{p}_{1},...,\hat{p}_{t-1},\mathbf{s},\mathbf{c})$

For training, the ground truth partner personas $\mathbf{p}$ is used and we
train our generator under the maximum likelihood estimation:

$\displaystyle
P(\mathbf{p}\mid\mathbf{s},\mathbf{c})=\prod_{t=1}^{T}P(p_{t}\mid
p_{1},...,p_{t-1},\mathbf{s},\mathbf{c})$

###  3.2 Dialogue Response Generation

We also adopt a Seq2Seq neural network for the task of dialogue response
generation (DRG). The concatenation of dialogue context c, self personas
$\mathbf{s}$, and partner personas $\mathbf{\hat{p}}$ is fed as an input into
the dialogue response generator. The personas generator then outputs an
approximated dialogue response $\mathbf{\hat{r}}$ conditioned on the input,
which maximises the following conditional likelihood:

$\displaystyle
P(\mathbf{\hat{r}}\mid\mathbf{s},\mathbf{\hat{p}},\mathbf{c})=\prod_{t=1}^{T}P(\hat{r}_{t}\mid\hat{r}_{1},...,\hat{r}_{t-1},\mathbf{s},\mathbf{\hat{p}},\mathbf{c})$

For training, the ground truth partner personas $\mathbf{p}$ and the ground
truth dialogue response $\mathbf{r}$ is used and we train our generator under
the maximum likelihood estimation:

$\displaystyle
P(\mathbf{r}\mid\mathbf{s},\mathbf{p},\mathbf{c})=\prod_{t=1}^{T}P(r_{t}\mid
r_{1},...,r_{t-1},\mathbf{s},\mathbf{p},\mathbf{c})$

We use the ground truth partner personas for training and generated partner
personas for inference.

###  3.3 Reinforcement Learning

We employ a critic network that is the core of our reinforcement learning (RL)
algorithm to reward our reinforcement agents. We train a binary classifier as
our critic network by extracting sub-training-instances
$(\mathbf{s},\mathbf{r},L=1)$, where $L=1$ represents positive training
samples. We then randomly sample two distinct positive sub-instances $A$ and
$B$:

$(\mathbf{s}^{A},\mathbf{r}^{A},L=1)$ $(\mathbf{s}^{B},\mathbf{r}^{B},L=1)$

Then two negative samples can be derived as:

$(\mathbf{s}^{A},\mathbf{r}^{B},L=0)$ $(\mathbf{s}^{B},\mathbf{r}^{A},L=0)$

Thereafter, we fine-tune a binary classifier as our critic on this training
set by minimizing the following binary cross-entropy loss:

$-Llog(P(L\mid\mathbf{s},\mathbf{r}))-(1-L)log(1-P(L\mid\mathbf{s},\mathbf{r}))$

In the equation above, the binary label $L$ indicates whether the given
response is relevant to the given personas. During the reinforcement learning
stage, this classifier acts as a critic network that outputs $\hat{L}$,
conditioned on the generated partner personas $\mathbf{\hat{p}}$ and generated
response $\mathbf{\hat{r}}$. The predicted binary label $\hat{L}$ is then
converted to a reward $R$. $R$ is a positive reward when $\hat{L}=1$, and $R$
is a negative reward when $\hat{L}=0$. We then update our RL agents with the
following gradients:

$\Delta\theta_{PPG}=R\triangledown_{\theta_{PPG}}-logP(\mathbf{\hat{p}}\mid\mathbf{s},\mathbf{c})$

for the partner personas generator (PPG), and for the dialogue response
generator (DRG):

$\Delta\theta_{DRG}=R\triangledown_{\theta_{DRG}}-logP(\mathbf{\hat{r}}\mid\mathbf{s},\mathbf{\hat{p}},\mathbf{c})$

We particularly want to give positive rewards to our RL agents when they give
high-quality responses that differ from the ground truth. Since it is not
straightforward to understand the underlying motivation for such a critic
network, we divide it into two cases and conquer each of them:

  * •

The critic network outputs $0$ as $\hat{L}$: the generated personas and
response are irrelevant, and we assign a negative reward.

  * •

The critic network outputs $1$ as $\hat{L}$: the generated personas and
response are relevant, and we assign a positive reward.

The first case is trivial, as it is reasonable to assign a negative reward
when at least one of our RL agents generates an output far away from the
ground truth. For the second case, in addition to the trivial case that both
agents output ground-truth-like generation, it also considers such a case when
both partner personas generator and dialogue response generator generate a
relevant output, but not the exact ground truth. Maximum likelihood estimation
might fail to capture this reward, as there could still be a certain distance
to the ground truth. In contrast, our critic network captures this by
outputting $1$ and assigns both of our RL agents a positive reward. We design
such a dedicate reward mechanism to encourage the generator to produce a
diverse and engaging response with the diverse partner personas generated. We
present a case study in Section 5.2 that illustrates this advantage.

Previous work (Cai et al., 2019) employed critic network for reinforcement
loss backpropagation. At first glance, our usage of the critic network shares
some resemblances to theirs, but indeed, the underlying motivation vastly
differs. The major difference is that their critic is trained in an
adversarial manner (Li et al., 2018) to pick up the ground truth response
among other negative candidates. Also, their critic network conditions only on
the dialogue response but not on the generated skeleton. In contrast, we
further diversify the response generation with a classifier conditioning on
both the generated personas and the generated response.

Model | PPL$\downarrow$ | ROUGE | METEOR | Distinct-1 | Distinct-2
---|---|---|---|---|---
E2E w/o Partner Personas | $17.86$ | $17.21$ | $24.62$ | $0.01278$ | $0.1502$
E2E w/ Training Partner Personas | $17.45$ | $16.91$ | $24.29$ | $0.01278$ | $0.1485$
Multi-task Learning (Lee et al., 2021) | $177.5$ | $10.33$ | $12.85$ | $0.00423$ | $0.0574$
Our Framework w/ RL PPG | $17.01$ | $17.15$ | $25.55$ | $0.01304$ | $0.1602$
Our Framework w/ RL DRG | $\mathbf{16.87}$ | $17.46$ | $25.88$ | $0.01450$ | $0.1664$
Our Framework w/o RL | $17.39$ | $17.43$ | $24.81$ | $0.01344$ | $0.1543$
Our Framework w/ RL PPG&DRG | $17.16$ | $\mathbf{17.51}$ | $\mathbf{26.36}$ | $\mathbf{0.01495}$ | $\mathbf{0.1726}$
Table 1:  Test results on the PersonaChat dataset. Perplexity (PPL) attains
better quality with lower scores, and the remaining metrics attain better
quality with higher scores. PPG represents partner personas generation and DRG
represents dialogue response generation. We leave corresponding validation
performance in Appendix A.

##  4 Experimental Setup

###  4.1 Dataset

We conduct all our experiments on PersonaChat (Zhang et al., 2018), the
largest multi-turn dialogue dataset conditioned on personas profile. We follow
the training/validation/testing split from the ParlAI platform (Miller et al.,
2017), which contains about 65,000 training instances, about 7,800 validation
instances and about 7,500 testing instances. As for the reinforcement learning
in Section 3.3, we collect about 130,000 training instances from the training
partition with equally distributed positive and negative samples to train our
critic network.

Figure 3: Change of the validation perplexity for our proposed framework
during RL.

###  4.2 Baselines and Comparison Models

#### End-to-end Baseline without Partner Personas

Our first baseline is an end-to-end response generator without any partner
persona information throughout the experiment. We notice that it could be
unfair to directly compare our proposed framework with this baseline, as our
framework uses ground truth partner personas in the training stage for our
proposed framework. Since it does not use the same amount of training
information as our proposed framework, we offer our second baseline trained
with the ground truth partner personas.

#### End-to-end Baseline with Training Partner Personas

Our second baseline is a an end-to-end response generator that is trained
using ground truth partner personas. In the inference stage, we feed the
concatenation of self personas and dialogue context as the input. For the sake
of fairness, it uses the same amount of training information and inference
information as our proposed framework.

#### Multi-task Learning Comparison Model

Following prior work (Lee et al., 2021), we build a multi-task learning
comparison model with partner personas generation as an auxiliary task. The
model is trained to maximise the training objective of the sum of the partner
personas generation labels likelihood $L_{PPG}$, and the dialogue response
generation labels likelihood $L_{DRG}$. Both of the tasks are generated and
conditioned on dialogue context and self personas by sharing the same model
parameters. We maximise the loss:

$\displaystyle\alpha L_{PPG}+L_{DRG},$

where $\alpha$ is a loss weighting parameter which we tune it over the
validation set.

###  4.3 Implementation Details

For all of our baselines, comparison model, the partner personas generator and
the dialogue response generator, we use pre-trained GPT-2 (Radford et al.,
2019) to initialize their model parameters. For the supervised phase, we set
Adam (Kingma and Ba, 2014) as our optimizer, with hyperparameters
$\eta=5e{-}4$, $\beta_{1}=0.9$, $\beta_{2}=0.999$, $\epsilon=1e{-}8$. We fine-
tune 2 epochs on all the baselines, comparison models and our proposed
framework modules and select the models with the lowest validating perplexity.
For the RL phase, we set Adam as our optimizer, with hyperparameters
$\eta=5e{-}6$, $\beta_{1}=0.9$, $\beta_{2}=0.999$, $\epsilon=1e{-}8$. We
update the model parameters every $20$ training instances and validate the
model performance every $50$ updates. For our critic network for reward
judgement in the RL phase, we use DistilBERT (Sanh et al., 2019) to initialize
the model parameters. We set Adam as our optimizer, with hyperparameters
$\eta=5e{-}6$, $\beta_{1}=0.9$, $\beta_{2}=0.999$, $\epsilon=1e{-}8$. We fine-
tune the critic for 1 epoch on the original training split from the
PersonaChat. We conduct all our experiments based on the Transformers library
from Huggingface (Wolf et al., 2020).

###  4.4 Evaluation Metrics

We report intrinsic perplexity to evaluate the model with the ground truth
response (Roller et al., 2020). We report distinct-1 and distinct-2 (Li et
al., 2015) to evaluate model diversity, which calculate the ratio of distinct
unigrams/bigrams against total unigrams/bigrams generated. We report ROUGE
(Lin, 2004) and METEOR (Banerjee and Lavie, 2005) as the extrinsic
evaluations.

##  5 Results

Recall our four claims: a) Our framework generates diverse partner personas
than the ground truth partner personas. b) Partner personas generation
benefits the succeeding response generation, thus surpassing our baselines. c)
Our framework generates more diverse and engaging responses than our
competitive baseline that uses ground truth training and inference partner
personas. d) We employ reinforcement learning with a dedicatedly designed
critic network that effectively boosts our framework. Also, recall the three
hypotheses that motivate our work: i) Partner personas generation is
plausible. ii) Generated personas are more diverse and interesting than the
retrieved ground truth. iii) Generated personas boost succeeding dialogue
generation. In the remaining section, we attempt to verify these claims and
hypotheses.

###  5.1 Main Results

The main results are presented in Table 1.

#### Our Framework w/o RL

Even when trained without the reinforcement learning algorithm, our framework
surpasses all our baselines and a comparison model. This phenomenon serves as
an evidence for our claim b) and hypothesis iii). Therefore, our framework
more efficiently parameterises the training partner personas than our E2E
baseline trained with ground truth partner personas. Also, our framework
relaxes the constraint that requires the main and the auxiliary task to have a
similar nature, which is unrealistic in our case.

Model | Distinct-1 | Distinct-2
---|---|---
Validation Set
Ground Truth Label | $0.003120$ | $0.03575$
Our Generator | $\mathbf{0.003920}$ | $\mathbf{0.05422}$
Test Set
Ground Truth Label | 0.003267 | 0.03856
Our Generator | $\mathbf{0.004090}$ | $\mathbf{0.05597}$
Table 2:  Diversity evaluation for the generated partner personas against the ground truth partner personas.  Model | Distinct-1 | Distinct-2
---|---|---
E2E w/ Full | $0.01370$ | $0.1544$
Our Framework | $\mathbf{0.01495}$ | $\mathbf{0.1726}$
Table 3:  Testing diversity evaluation for dialogue generation on our
framework against the competitive E2E baseline with ground truth partner
personas. We leave corresponding validation performance in Appendix A.

#### Our Framework w/ RL PPG&DRG

Our proposed framework performs the best when trained with reinforcement
learning. Again, this phenomenon strengthens our claim b) and hypothesis iii).
In addition, this phenomenon serves as direct evidence for our claim d). This
phenomenon implicitly verifies our hypothesis i), because we train our critic
network using ground truth partner personas. Since the reinforcement learning
algorithm is effective, our critic network produces valid rewards, which
indicates that the generated partner personas are reasonable. Furthermore, we
present the change of validation performance for dialogue generation and
partner personas generation in Figure 3.333The result is scaled for the sake
of space and clarity. We observe that both performances improve during the RL
stage. Therefore, the improvement in partner personas generation intimately
relates to the improvement in the succeeding dialogue generation; see our
claim b) and iii).

|  Dialogue Context |  Generated Personas |  Ours |  E2E w/ Full
---|---|---|---
Hi, want to go shopping with me? |  I love to shop. I drive BMW. |  Sure! I love shopping. I am a bit envious. |  I am a student at a small community college.
I am just happy that the apartment is quiet for once. |  I live in an apartment. |  I live in an apartment. I like to watch TV. |  I live in California.
I just went hiking also tired. |  I love the outdoors. |  I am sorry to hear that. Do you like to travel? |  I do not know. I am not very good at it.
Table 4:  A case study that compares our framework against the baseline that conditions on the ground truth partner personas. We denote the latter one as E2E w/ Full. For the sake of clarity and space, we present the preceding partner utterance as dialogue context, and we give the most salient generated partner personas.  Criteria | E2E w/ Full | Ours
---|---|---
Appropriateness | $20$ |  $\mathbf{80}$‡
Informativeness | 31 |  $\mathbf{69}$‡
Engagingness | $22$ |  $\mathbf{78}$‡
Human-likeness | $24$ |  $\mathbf{76}$‡
Table 5:  Human evaluation results in winning percentages. ${\ddagger}$
indicates the results as passing a two-tailed binomial significance test with
$p<0.001$.

#### End-to-end (E2E) Baseline Models

Our E2E baseline with training partner personas has a better perplexity and
worse extrinsic scores than the E2E baseline without partner personas. This
might be due to the training-inference discrepancy, which could significantly
impact the extrinsic evaluations.

#### Multi-task Learning Comparison Model

Our multi-task learning comparison model produces inferior results. This is,
however, predictable. First of all, the prior work (Lee et al., 2021)
constrained itself to a meta-learning framework. More concretely, the nature
of partner personas generation and dialogue response generation largely
differs. The output format of partner personas always initiates with first-
person sentence starters, while dialogue responses give more general responses
ranging from greetings to goodbyes.

###  5.2 Case Study on Dialogue Response Generation

Table 4 depicts the case study for response generation. In the first case, our
partner personas generator successfully gave a reasonable imagination that a
person who likes shopping could be rich and drive a luxury car, which is not
in the ground truth personas. This follows a surprising response ‘I am a bit
envious’. In the second case, our personas generator successfully identifies
that the partner lives in an apartment. Succeedingly, the response generator
gives a relevant response and reduces the undesired hallucination444‘I like to
watch TV’ is in the self personas, but ‘I live in California’ is not. The
latter one is thus a hallucination. from the baseline model. In the third
case, our partner personas generator generates a partner persona ‘I love the
outdoors’, which is not even in the ground truth personas. After that, the
response generator produces a relevant response which also expresses empathy.
These facts support our underlying hypotheses i), ii), and iii). Furthermore,
as in Table 3, we compare our proposed framework with an end-to-end dialogue
agent with both training and inference ground truth partner personas. For
these cherry-picked examples, our framework generates more informative and
engaging responses than this competitive baseline. This verifies our claim c)
and hypothesis iii).

|  Generated Partner Personas |  Ground Truth Partner Personas
---|---
I am working on a biology degree. I love book. 1984 is my favourite book. I am in college. |  I am a student. I attend university and study biology. I am very studious and do not like to party or drink. I want to be marine biologist.
I have been married for 6 years. I am a financial analyst for a brewery. I like to go to the casino on weekends. |  I am a carpenter. I like playing poker. I do not have many friends. I have a wife and three kids.
I like to watch football. My friends like watching it too. Its great fun. We drink beer and eat food. |  I love watching football on Sunday. I have three dogs. My favroutie food is cheese piazza. I am a hair stylist.
Table 6:  A case study to show that our generated personas are relevant,
informative and coherent. For the sake of space, we present more cases in
Appendix B.

###  5.3 Human Evaluation

We hired experienced annotators who have degrees relevant to English
Linguistics or Applied Linguistics. We present a questionnaire composed of 280
questions with randomly sampled 70 testing instances. Three annotators compare
model outputs in an A/B setting. As in previous work (Zou et al., 2021) and
ACUTE-Evals (Li et al., 2019), annotators follow the criteria:

  * •

(Appropriateness): "Who is more appropriate given the previous dialogue
context?"

  * •

(Informativeness): "Who is more diverse instead of null answers such as I do
not know?"

  * •

(Engagingness): "Who would you prefer to talk with for a long conversation?"

  * •

(Human-likeness): "Which speaker do you think sounds more like a real person?"

Table 5 presents the human evaluation results. It is exciting to see our
framework trained under reinforcement learning surpasses the end-to-end model
that leverages both training and inference ground truth partner personas, from
all the aspects, significantly. This supports our claim c) and d).

###  5.4 Ablation Study on Reinforcement Learning

Table 1 presents an ablation study on the framework when only one of the
modules, namely partner personas generator (PPG) or dialogue response
generator (DRG), is trained under reinforcement learning. Our framework
exceeds these two variants in all the metrics except for perplexity, which
aligns with the prior work (Roller et al., 2020).555PPL does not always
correlate well with other metrics.

###  5.5 Case Study on Partner Personas Generation

As depicted in Table 2, we observe that our partner personas generator
generates more diverse partner personas compared to the ground truth partner
personas label, which is essentially the upper bound for retrieval-based
partner personas predictor. This phenomenon verifies our claim a) and
hypothesis i), indicating our generator produces even more informative and
interesting partner personas than the ground truth partner personas.

As depicted in Table 6, our partner personas generator can generate plausible
partner personas which are relevant to the ground truth partner personas. Our
partner personas generator can give fascinating but reasonable imagination
that is not even in the dialogue context or the ground truth partner personas.
In the first case, the generator successfully identified the partner as a
student studying biology. The generator recognizes the partner as being
married in the second case, which is not even mentioned in the dialogue
context. This phenomenon could be a matter of the fact that personas could be
semantically closer to each other when they frequently co-occur in the
training set. In the third case, the generator generates diverse personas,
saying that the partner would drink beer and eat food while watching football,
which is even not in any of the dialogue context or the ground truth partner
personas. This verifies claim a) and hypothesis i) and ii), We depict more
case studies in Appendix B to show that our personas generator generates
informative and coherent partner personas.666Our partner personas generator is
even capable of producing unseen personas. An offensiveness check thus is
necessary for the actual usage, as in prior works (Baheti et al., 2021).

##  6 Conclusion

Our novel framework incorporates partner personas generation into succeeding
dialogue response generation. First of all, our proposed framework mitigates
the cold-start problem in practical applications when ground truth partner
personas could be missing during inference. The experimental results with both
automatic and human evaluation demonstrate that our framework generates
informative and coherent partner personas, even compared to the ground truth
partner personas, yet still reasonable and relevant. This enhances the
succeeding response generation, thus surpassing our baselines and producing
responses that are more diverse and engaging than our baseline conditioned on
the ground truth partner personas. We employ reinforcement learning with a
dedicatedly designed critic network that boosts our framework. Extensive case
studies demonstrate that our framework can generate satisfying dialogue
responses and partner personas. Finally, our framework gives better
explainability and reduces the demands for external databases for partner
personas.

## References

  * Baheti et al. (2021) Ashutosh Baheti, Maarten Sap, Alan Ritter, and Mark Riedl. 2021.  Just Say No: Analyzing the Stance of Neural Dialogue Generation in Offensive Contexts.  _arXiv e-prints_ , page arXiv:2108.11830.
  * Banerjee and Lavie (2005) Satanjeev Banerjee and Alon Lavie. 2005.  METEOR: An automatic metric for MT evaluation with improved correlation with human judgments.  In _Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization_ , pages 65–72, Ann Arbor, Michigan. Association for Computational Linguistics.
  * Bao et al. (2021) Siqi Bao, Huang He, Fan Wang, Hua Wu, Haifeng Wang, Wenquan Wu, Zhihua Wu, Zhen Guo, Hua Lu, Xinxian Huang, Xin Tian, Xinchao Xu, Yingzhan Lin, and Zhengyu Niu. 2021.  PLATO-XL: Exploring the Large-scale Pre-training of Dialogue Generation.  _arXiv e-prints_ , page arXiv:2109.09519.
  * Cai et al. (2019) Deng Cai, Yan Wang, Wei Bi, Zhaopeng Tu, Xiaojiang Liu, Wai Lam, and Shuming Shi. 2019.  Skeleton-to-response: Dialogue generation guided by retrieval memory.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 1219–1228, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Deng et al. (2021) Yang Deng, Yaliang Li, Fei Sun, Bolin Ding, and Wai Lam. 2021.  Unified Conversational Recommendation Policy Learning via Graph-based Reinforcement Learning.  _arXiv e-prints_ , page arXiv:2105.09710.
  * Gu et al. (2019) Jia-Chen Gu, Zhen-Hua Ling, Xiaodan Zhu, and Quan Liu. 2019.  Dually interactive matching network for personalized response selection in retrieval-based chatbots.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 1845–1854, Hong Kong, China. Association for Computational Linguistics.
  * Gu et al. (2021) Jia-Chen Gu, Hui Liu, Zhen-Hua Ling, Quan Liu, Zhigang Chen, and Xiaodan Zhu. 2021.  Partner Matters! An Empirical Study on Fusing Personas for Personalized Response Selection in Retrieval-Based Chatbots.  _arXiv e-prints_ , page arXiv:2105.09050.
  * Kim et al. (2020) Hyunwoo Kim, Byeongchang Kim, and Gunhee Kim. 2020.  Will I sound like me? improving persona consistency in dialogues through pragmatic self-consciousness.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 904–916, Online. Association for Computational Linguistics.
  * Kingma and Ba (2014) Diederik P. Kingma and Jimmy Ba. 2014.  Adam: A Method for Stochastic Optimization.  _arXiv e-prints_ , page arXiv:1412.6980.
  * Lee et al. (2021) Jing Yang Lee, Kong Aik Lee, and Woon Seng Gan. 2021.  Generating Personalized Dialogue via Multi-Task Meta-Learning.  _arXiv e-prints_ , page arXiv:2108.03377.
  * Li et al. (2018) Dianqi Li, Qiuyuan Huang, Xiaodong He, Lei Zhang, and Ming-Ting Sun. 2018\.  Generating Diverse and Accurate Visual Captions by Comparative Adversarial Learning.  _arXiv e-prints_ , page arXiv:1804.00861.
  * Li et al. (2015) Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2015.  A Diversity-Promoting Objective Function for Neural Conversation Models.  _arXiv e-prints_ , page arXiv:1510.03055.
  * Li et al. (2016) Jiwei Li, Will Monroe, Alan Ritter, Michel Galley, Jianfeng Gao, and Dan Jurafsky. 2016.  Deep Reinforcement Learning for Dialogue Generation.  _arXiv e-prints_ , page arXiv:1606.01541.
  * Li et al. (2019) Margaret Li, Jason Weston, and Stephen Roller. 2019.  ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisons.  _arXiv e-prints_ , page arXiv:1909.03087.
  * Li et al. (2020) Shijun Li, Wenqiang Lei, Qingyun Wu, Xiangnan He, Peng Jiang, and Tat-Seng Chua. 2020.  Seamlessly Unifying Attributes and Items: Conversational Recommendation for Cold-Start Users.  _arXiv e-prints_ , page arXiv:2005.12979.
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  * Madotto et al. (2020) Andrea Madotto, Samuel Cahyawijaya, Genta Indra Winata, Yan Xu, Zihan Liu, Zhaojiang Lin, and Pascale Fung. 2020.  Learning knowledge bases with parameters for task-oriented dialogue systems.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 2372–2394, Online. Association for Computational Linguistics.
  * Mazaré et al. (2018) Pierre-Emmanuel Mazaré, Samuel Humeau, Martin Raison, and Antoine Bordes. 2018\.  Training millions of personalized dialogue agents.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2775–2779, Brussels, Belgium. Association for Computational Linguistics.
  * Miller et al. (2017) Alexander H. Miller, Will Feng, Adam Fisch, Jiasen Lu, Dhruv Batra, Antoine Bordes, Devi Parikh, and Jason Weston. 2017.  ParlAI: A Dialog Research Software Platform.  _arXiv e-prints_ , page arXiv:1705.06476.
  * Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.
  * Roller et al. (2020) Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Kurt Shuster, Eric M. Smith, Y-Lan Boureau, and Jason Weston. 2020.  Recipes for building an open-domain chatbot.  _arXiv e-prints_ , page arXiv:2004.13637.
  * Roman Roman et al. (2020) Homero Roman Roman, Yonatan Bisk, Jesse Thomason, Asli Celikyilmaz, and Jianfeng Gao. 2020.  RMM: A recursive mental model for dialogue navigation.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 1732–1745, Online. Association for Computational Linguistics.
  * Saleh et al. (2019) Abdelrhman Saleh, Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, and Rosalind Picard. 2019.  Hierarchical Reinforcement Learning for Open-Domain Dialog.  _arXiv e-prints_ , page arXiv:1909.07547.
  * Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019.  DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter.  _arXiv e-prints_ , page arXiv:1910.01108.
  * Song et al. (2021) Haoyu Song, Yan Wang, Kaiyan Zhang, Wei-Nan Zhang, and Ting Liu. 2021.  BoB: BERT over BERT for training persona-based dialogue models from limited personalized data.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 167–177, Online. Association for Computational Linguistics.
  * Song et al. (2020) Haoyu Song, Yan Wang, Wei-Nan Zhang, Xiaojiang Liu, and Ting Liu. 2020.  Generate, delete and rewrite: A three-stage framework for improving persona consistency of dialogue generation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5821–5831, Online. Association for Computational Linguistics.
  * Song et al. (2019) Haoyu Song, Wei-Nan Zhang, Yiming Cui, Dong Wang, and Ting Liu. 2019.  Exploiting Persona Information for Diverse Generation of Conversational Responses.  _arXiv e-prints_ , page arXiv:1905.12188.
  * Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014.  Sequence to Sequence Learning with Neural Networks.  _arXiv e-prints_ , page arXiv:1409.3215.
  * Williams (1992) Ronald J. Williams. 1992.  Simple statistical gradient-following algorithms for connectionist reinforcement learning.  _Mach. Learn._ , 8(3–4):229–256.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Wolf et al. (2019) Thomas Wolf, Victor Sanh, Julien Chaumond, and Clement Delangue. 2019.  TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents.  _arXiv e-prints_ , page arXiv:1901.08149.
  * Xu et al. (2021) Yan Xu, Etsuko Ishii, Zihan Liu, Genta Indra Winata, Dan Su, Andrea Madotto, and Pascale Fung. 2021.  Retrieval-Free Knowledge-Grounded Dialogue Response Generation with Adapters.  _arXiv e-prints_ , page arXiv:2105.06232.
  * Zhang et al. (2018) Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018.  Personalizing dialogue agents: I have a dog, do you have pets too? In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2204–2213, Melbourne, Australia. Association for Computational Linguistics.
  * Zhang et al. (2019) Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2019.  DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation.  _arXiv e-prints_ , page arXiv:1911.00536.
  * Zhao et al. (2019) Xueliang Zhao, Chongyang Tao, Wei Wu, Can Xu, Dongyan Zhao, and Rui Yan. 2019.  A Document-grounded Matching Network for Response Selection in Retrieval-based Chatbots.  _arXiv e-prints_ , page arXiv:1906.04362.
  * Zou et al. (2021) Yicheng Zou, Zhihua Liu, Xingwu Hu, and Qi Zhang. 2021.  Thinking Clearly, Talking Fast: Concept-Guided Non-Autoregressive Generation for Open-Domain Dialogue Systems.  _arXiv e-prints_ , page arXiv:2109.04084.

##  Appendix A Validation Performance

Table 8 presents the corresponding validation performance to Table 1. Table 7
presents the corresponding validation performance to Table 3.

Model | Distinct-1 | Distinct-2
---|---|---
E2E w/ Full | $0.01293$ | $0.1484$
Our Framework | $\mathbf{0.01339}$ | $\mathbf{0.1591}$
Table 7:  Validation diversity evaluation for dialogue generation that
compares our framework against the competitive E2E baseline with ground truth
partner personas, as promised in Table 3.

##  Appendix B More Case Studies

Table 9 presents extensive case studies for partner personas generation. These
examples indicate that our framework can generate informative and coherent
partner personas. We highlight in pink for informativeness and in yellow for
coherence.

Model | PPL$\downarrow$ | ROUGE | METEOR | Distinct-1 | Distinct-2
---|---|---|---|---|---
E2E w/o Partner Personas | $20.02$ | $16.42$ | $24.46$ | $0.01168$ | $0.1422$
E2E w/ Training Partner Personas | $19.71$ | $16.27$ | $24.10$ | $0.01221$ | $0.1413$
Multi-task Learning (Lee et al., 2021) | $218.7$ | $10.27$ | $12.80$ | $0.00395$ | $0.0560$
Our Framework w/ RL PPG | $18.31$ | $\mathbf{16.77}$ | $25.20$ | $0.01329$ | $0.1511$
Our Framework w/ RL DRG | $18.70$ | $16.69$ | $25.64$ | $\mathbf{0.01345}$ | $0.1575$
Our Framework w/o RL | $18.56$ | $16.71$ | $24.79$ | $0.01322$ | $0.1490$
Our Framework w/ RL PPG&DRG | $\mathbf{18.19}$ | $\mathbf{16.77}$ | $\mathbf{26.14}$ | $0.01339$ | $\mathbf{0.1591}$
Table 8:  Corresponding validation performance as promised in Table 1.  |  Generated Partner Personas
---
Personas A: I am a shy person, but I love to sing. Until recently, I ve never
been able to sing in front of anyone. Anyways, I decided to give it a try and
participaed in an audition for a talent show. My shyness made me panick and I
didn t show up.
Personas B: I play the violin. I am married with 5 kids. I am nurse. I met my
husband when I was a freshman in college.
Personas C: I am a soccer player. I am a goalie. My number is 42. Nike cleats
are my favorite. I joined a new team last month.
Personas D: I have two kids, ages 2 and 6. I am from sterling heights,
michigan. My favorite movie is titanic. I work part time at aldis. My husband
owns a small auto repair shop.
Personas E: I am a retired computer programmer. I have one grandson and one
daughter. I just turned 77. I love animals. I like watching british tv shows
and movies.
Personas F: I like to go hunting. I like to remodel homes. I like to shoot a
bow. My favorite holiday is halloween. I like to go shopping with my
daughters.
Personas G: I have a large cd collection. I collect stamps. Favorite band is
the beetles. I play the bass. I like vintage furniture.
Personas H: I have a daughter. I am a yoga instructor. I enjoy shopping. I
have two adopted kids.
Personas I: I like to drink wine. I enjoy reading history books. I am a
teacher. I love to write stories while sitting in the grass in my back yard. I
grew up in new hampshire.
Personas J: I am retired. I stay active. I have eight grandchildren. I have
good health.
Personas K: I m a student. I like to go out to eat. I like listening to other
rap music too. One of my favorite artists is drake. A hobby of mine is the
drums. I also enjoy cooking .
Personas L: I have two children. I like to go on walks. I am from mexico. I
used to be a chef, but I am a teacher now. I like to bake.
Personas M: I drive a prius. My mom stays at home. I was adopted when I was a
baby. My adopted dad works at hp.
Personas N: I love youtube. My father works in advertising agency. I have my
own channel. I enjoy making let s plays.
Personas O: I like to cook. I am a foodie. I love to chat with my friends. I
love kids and dogs. I like to go shopping with my daughters.
Personas P: My family hates my fiance. We will be traveling to niagra falls
for our honeymoon. We are getting married in a park. My dog is the ring
bearer.
Table 9:  Case studies as promised in Table 6. We highlight in pink for
informativeness and in yellow for coherence.