# Asymptotics of Language Model Alignment

Joy Qiping Yang
University of Sydney
Sydney, Australia
<EMAIL_ADDRESS>
Salman Salamatian
Massachusetts Institute of Technology
Cambridge, MA, USA
<EMAIL_ADDRESS>
Ziteng Sun, Ananda Theertha Suresh, Ahmad Beirami
Google Research
New York, NY, USA
{zitengsun, theertha<EMAIL_ADDRESS>
###### Abstract

Let $\bm{p}$ denote a reference generative language model. Let $\bm{r}$ denote
a reward model that returns a scalar to capture the degree at which a draw
from $\bm{p}$ is preferred. The goal of language model alignment is to alter
$\bm{p}$ to a new distribution $\bm{\phi}$ that results in a higher expected
reward while keeping $\bm{\phi}$ close to $\bm{p}.$ A popular alignment method
is the KL-constrained reinforcement learning (RL), which chooses a
distribution $\bm{\phi}_{\Delta}$ that maximizes
$E_{\bm{\phi}_{\Delta}}\bm{r}(\bm{y})$ subject to a relative entropy
constraint $D_{\operatorname{KL}}(\bm{\phi}_{\Delta}\|\bm{p})\leq\Delta.$
Another simple alignment method is best-of-$N$, where $N$ samples are drawn
from $\bm{p}$ and one with highest reward is selected. In this paper, we offer
a closed-form characterization of the optimal KL-constrained RL solution. We
then demonstrate that any alignment method that achieves a comparable trade-
off between KL divergence and expected reward must approximate the optimal KL-
constrained RL solution in terms of relative entropy. To further analyze the
properties of alignment methods, we introduce two simplifying assumptions: we
let the language model be memoryless, and the reward model be linear. Although
these assumptions may not reflect complex real-world scenarios, they enable a
precise characterization of the asymptotic (in the sequence length) behavior
of both the best-of-$N$ alignment, and the KL-constrained RL method, in terms
of information-theoretic quantities. We prove that the reward of the optimal
KL-constrained RL solution satisfies a large deviation principle, and we fully
characterize its rate function. We also show that the rate of growth of the
scaled cumulants of the reward is characterized by a proper Rényi cross
entropy. Finally, we show that best-of-$N$ is asymptotically equivalent to KL-
constrained RL solution by proving that their expected rewards are
asymptotically equal, and concluding that the two distributions must be close
in KL divergence.

##  1 Introduction

Generative models, such as large language models (Brown et al., 2020; Team et
al., 2023; Touvron et al., 2023) and diffusion models (Sohl-Dickstein et al.,
2015; Ho et al., 2020; Song et al., 2020), are increasingly popular general-
purpose problem solvers. These models are generally trained on large corpora
of unlabeled data; with additional training on domain-specific data through
supervised finetuning (SFT). We refer to the resulting model as the reference
model. The reference model, while capable of generating realistic outputs, may
still generate undesirable outcomes, e.g., unsafe language or images. Hence,
it is desirable to further align the output of the reference model with a
reward that captures human preferences (Christiano et al., 2017; Stiennon et
al., 2020; Ouyang et al., 2022; Bai et al., 2022).

The literature has seen a proliferation of algorithmic techniques for
alignment (Christiano et al., 2017; Stiennon et al., 2020; Zhao et al., 2022;
Ouyang et al., 2022; Bai et al., 2022; Rafailov et al., 2023; Mudgal et al.,
2023; Yang & Klein, 2021). Two of the popular methods are: (a) KL-constrained
reinforcement learning (RL), $\bm{\phi}_{\Delta}$, (Christiano et al., 2017;
Ouyang et al., 2022), which maximizes expected reward subject to a KL
divergence constraint, $\Delta,$ between the aligned model and the reference
model; and (b) Best-of-$N$, $\bm{\pi}_{N}$, (Nakano et al., 2021; Touvron et
al., 2023), where $N$ i.i.d. samples are drawn from the reference model and
one with the highest reward is returned.

Despite the popularity of both of these alignment methods, no relationship
between them is known. In particular, best-of-$N$ has shown to be surprisingly
competitive with, or even outperform more involved KL-constrained RL
algorithms that aim at achieving the best reward-KL tradeoffs (Mudgal et al.,
2023; Gao et al., 2023; Eisenstein et al., 2023). In this work, we provide
theoretical reasoning for this empirical observation by showing that the two
methods are in fact asymptotically equivalent in some specific settings. We
hope our work initiates further study into the relationship between different
alignment methods. The summary of our contributions may be found below:

  * •

We characterize the unique optimal solution of the KL-constrained
reinforcement learning problem, denoted by $\bm{\phi}_{\Delta},$ for a given
KL constraint of $\Delta$. We show that the solution is a mismatched tilted
distribution that had appeared and characterized by Salamatian et al. (2019)
in the context of mismatched guesswork.

  * •

We show that if an alignment method $\bm{\omega}$ satisfies a KL constraint
$\Delta$, and also achieves almost the same expected reward as
$\bm{\phi}_{\Delta}$ (the optimal KL-constrained RL solution), then
$D_{\operatorname{KL}}(\bm{\omega}\|\bm{\phi}_{\Delta})$ must be vanishingly
small, where $D_{\operatorname{KL}}(\cdot\|\cdot)$ denotes the KL divergence
and is formally defined in (1).

  * •

We show that the cumulants of the reward are related to the Rényi cross
entropy of the alignment distribution to the source distribution. Under
assumptions on the data generating process, we also show that the optimal KL-
constrained RL solution, $\bm{\phi}_{\Delta},$ satisfies a large deviation
principle, where we characterize its rate function using information-theoretic
quantities.

  * •

Under assumptions on the data-generating process, we show that the outcomes of
the best-of-$N$ alignment method, $\bm{\pi}_{N},$ are similar to those of the
optimal KL-constrained RL, $\bm{\phi}_{\Delta},$ for $N=\exp(\Delta)$. In
particular, their expected rewards are also close, which implies that KL-
constrained RL method is close to the best-of-$N$ method in KL divergence,
i.e., $D_{\operatorname{KL}}(\bm{\pi}_{N}\|\bm{\phi}_{\Delta})$ is vanishingly
small.

##  2 Problem Setup

Let $\bm{x}\in\bm{\mathcal{X}}$ denote an input prompt, e.g., Who is Barack
Obama? Let $\bm{\mathcal{Y}}$ be the set of possible outputs of a generative
language model. A language model $\bm{p}_{\bm{y}|\bm{x}}$ assigns probability
distributions over $\bm{\mathcal{Y}}$ given a context (input prompt)
$\bm{x}\in\bm{\mathcal{X}}$ i.e.,
$\bm{p}_{\bm{y}|\bm{x}}:\bm{\mathcal{X}}\times\bm{\mathcal{Y}}\to\mathbb{R}^{\geq
0}$ such that $\forall\bm{x}$

$\sum_{\bm{y}\in\bm{\mathcal{Y}}}\bm{p}_{\bm{y}|\bm{x}}(\bm{y}|\bm{x})=1.$

Throughout this paper, we use $\bm{p}$ to denote the reference language model
which we wish to further align.

A reward model $\bm{r}$ assigns scalar scores for every prompt and output
i.e., $\bm{r}:\bm{\mathcal{X}}\times\bm{\mathcal{Y}}\to\mathbb{R}$. Given such
a reward model, one can also define an alignment distribution as follows.

######  Definition 1 (alignment distribution).

For a given reward function $\bm{r}$, we denote $\bm{q}_{\bm{y}|\bm{x}}$ as
the corresponding alignment distribution defined as

$\bm{q}_{\bm{y}|\bm{x}}(\bm{y}|\bm{x})=\exp\left(\bm{r}(\bm{x},\bm{y})\right)/R_{\bm{x}},$

where $R_{\bm{x}}$ is the partition function, i.e.,
$R_{\bm{x}}:=\sum_{\bm{y}\in\bm{\mathcal{Y}}}\exp\left(\bm{r}(\bm{x},\bm{y})\right)$.
Without loss of generality, and for convenience in the rest of this paper we
assume that $R_{\bm{x}}=1,$ i.e., reward is the negative log likelihood of a
generative distribution.

For two distributions $\bm{p}$ and $\bm{q}$, let $H(\bm{p}\|\bm{q})$ denote
the cross-entropy of $\bm{p}$ to $\bm{q}$; and
$D_{\operatorname{KL}}(\bm{p}\|\bm{q})$ denote the Kullback–Leibler
divergence. For discrete distributions $\bm{p}$ and $\bm{q}$ over the same
alphabet $\bm{\mathcal{Y}}$, we have that111All logarithms in this paper are
in base $e.$

$H(\bm{p}\|\bm{q})=\sum_{\bm{y}\in\bm{\mathcal{Y}}}p(\bm{y})\log\frac{1}{q(\bm{y})};\quad\quad
D_{\operatorname{KL}}(\bm{p}\|\bm{q})=\sum_{{\bm{y}\in\bm{\mathcal{Y}}}}p(\bm{y})\log\frac{p(\bm{y})}{q(\bm{y})}.$
(1)

With these definitions, the KL-constrained RL problem is defined as follows:

######  Definition 2 (KL-constrained RL).

We say $\bm{\phi}_{\Delta}$ is an optimal aligned model of $\bm{p}$ with KL
divergence $\Delta,$ if

$\bm{\phi}_{\Delta}(\cdot|\bm{x})\in\arg\min_{\bm{\phi}\in\mathcal{D}_{\Delta}(\bm{p})}H(\bm{\phi}(\cdot|\bm{x})\|\bm{q}(\cdot|\bm{x})),$
(2)

where
$\mathcal{D}_{\Delta}(\bm{p}):=\\{\bm{\phi}:D_{\operatorname{KL}}(\bm{\phi}(\cdot|\bm{x})\|\bm{p}(\cdot|\bm{x}))\leq\Delta\\}$.

Under the KL-constrained RL method, outputs are then generated using
$\bm{\phi}_{\Delta}$. We remark that

$\displaystyle H(\bm{\phi}(\cdot|\bm{x})\|\bm{q}(\cdot|\bm{x}))$
$\displaystyle=-\mathbb{E}_{\bm{y}\sim\bm{\phi}(\cdot|\bm{x})}\log\bm{q}_{\bm{y}|\bm{x}}(\bm{y}|\bm{x})=-\mathbb{E}_{\bm{y}\sim\bm{\phi}(\cdot|\bm{x})}\bm{r}(\bm{x},\bm{y}),$

and hence minimizing the cross entropy
$H(\bm{\phi}(\cdot|\bm{x})\|\bm{q}(\cdot|\bm{x}))$ is equivalent to maximizing
expected reward
$\mathbb{E}_{\bm{y}\sim\bm{\phi}(\cdot|\bm{x})}\bm{r}(\bm{x},\bm{y})$, which
is the common terminology in the language model alignment literature (Gao et
al., 2023; Eisenstein et al., 2023; Mudgal et al., 2023).

Next, we describe the best-of-$N$ alignment method.

######  Definition 3 (best-of-$N$).

Let $\bm{y}_{1},\ldots,\bm{y}_{N}$ be $N$ i.i.d. draws from
$\bm{p}_{\bm{y}|\bm{x}}(\cdot|\bm{x})$. The best-of-$N$ alignment policy
returns222We define $[N]:=\\{1,\ldots,N\\}$.

$\bm{y}=\bm{y}_{k^{*}}\quad\quad\text{where}\quad\quad
k^{*}:=\arg\max_{k\in[N]}\bm{q}_{\bm{y}|\bm{x}}(\bm{y}_{k}|\bm{x}).$ (3)

Further, let $\bm{\pi}_{N}$ denote the probability distribution of the best-
of-$N$ strategy. It is clear that $\bm{\pi}_{1}=\bm{p}$.

The best-of-$N$ alignment method generates $N$ candidate outputs directly from
the reference model and selects the one with highest reward. Further, its PMF
is characterized by Beirami et al. (2024, Lemma 1).

In empirical studies (Gao et al., 2023; Mudgal et al., 2023; Eisenstein et
al., 2023; Rafailov et al., 2023), in order to compare a family of decoding
policy $\\{\bm{\psi}_{\beta}\\}$ parameterized by $\beta$, it is common to
report tradeoff curves of expected reward
$E_{\bm{x}}E_{\bm{\psi}_{\beta}}\\{\bm{r}(\bm{x},\bm{y})\\}$ vs KL divergence
$E_{\bm{x}}D_{\operatorname{KL}}(\bm{\psi}_{\beta}(\cdot|\bm{x})\|\bm{p}(\cdot|\bm{x})),$
which we refer to as reward-KL tradeoff curves (e.g., (Mudgal et al., 2023,
Fig. 3)). A decoding policy is desired when its reward-KL tradeoff curve
dominates that of any other decoding policy. The optimal aligned model
(Definition 2) must dominate any other alignment technique by definition.
However, in practice best-of-$N$ achieves remarkably good reward-KL tradeoffs
and often dominates those of sophisticated methods solving the RL problem in
practical scenarios (Mudgal et al., 2023; Gao et al., 2023). One of our major
objectives in this paper is to develop a theoretical understanding for this
phenomenon.

##  3 Optimal KL-Constrained RL Solution

Our first goal is to understand the precise theoretical behavior of the reward
under the optimal KL-constrained RL. Our second goal is to study the
relationship between the optimal KL-constrained RL and best-of-$N$ methods. We
want to answer the questions: does best-of-$N$ method produce good or even
optimal solutions to the KL-constrained RL problem? Is there any theoretical
support for the good empirical performance of best-of-$N$?

First, we characterize the solution to the KL-constrained RL problem.

######  Lemma 1 (optimal aligned model).

The solution of the KL-constrained RL problem in Definition 2 is unique and is
given by

$\bm{\phi}_{\Delta}(\cdot|\bm{x})=T(\bm{q}(\cdot|\bm{x}),\bm{p}(\cdot|\bm{x}),\alpha(\Delta)),$
(4)

where $T(\bm{q}(\cdot|\bm{x}),\bm{p}(\cdot|\bm{x}),\alpha)$ is the mismatched
tilt (Salamatian et al., 2019, Definition 1):

$T(\bm{q}(\cdot|\bm{x}),\bm{p}(\cdot|\bm{x}),\alpha)(\bm{y}):=\frac{\bm{p}(\bm{y}|\bm{x})\bm{q}^{\alpha}(\bm{y}|\bm{x})}{Z_{\bm{x},\alpha}},$
(5)

where

$Z_{\bm{x},\alpha}:=\sum_{\bm{z}\in\bm{\mathcal{Y}}}\bm{p}(\bm{z}|\bm{x})\bm{q}^{\alpha}(\bm{z}|\bm{x}),$
(6)

and

$\alpha(\Delta):=\arg_{\alpha\in\mathbb{R}^{\geq
0}}\left\\{D_{\operatorname{KL}}(T(\bm{q}(\cdot|\bm{x}),\bm{p}(\cdot|\bm{x}),\alpha)\|\bm{p})=\Delta\right\\}.$
(7)

###### Proof sketch.

This follows from applying the Karush–Kuhn–Tucker (KKT) conditions on the
convex problem in Definition 2. ∎

A variant of this result has already appeared in (Korbak et al., 2022a; b) and
we provide a proof in the appendix for completeness. The result above suggests
that the aligned models (parameterized by $\Delta$) form an exponential family
of distributions,333This specific family of distribution also arises in
characterization of other elements in information theory, e.g. in mismatched
guesswork and mismatched one-to-one source coding (Salamatian et al., 2019,
Definition 1), and is referred there as the mismatched tilted family of
distributions. and we define this family of solutions to the KL-constrained RL
for different values of $\Delta$ as the aligned family.

######  Definition 4 (aligned family).

Let the aligned family of $\bm{p}$ towards $\bm{q}$ be denoted by
$\mathcal{T}(\bm{p},\bm{q})$, and be given by

$\mathcal{T}(\bm{p},\bm{q})=\\{\bm{\phi}_{\Delta}\\}_{\Delta\geq 0}.$ (8)

Equipped with the optimal aligned model for a given KL divergence constraint,
we ask how close a model would be to optimal, if it offered almost the same
expected reward as the optimal model.

######  Lemma 2 (closeness of models given closeness of expected rewards).

Let $\bm{\psi}_{\Delta}$ be such that

$D_{\operatorname{KL}}(\bm{\psi}_{\Delta}\|\bm{p})\leq\Delta;\quad\quad
H(\bm{\psi}_{\Delta}\|\bm{q})\leq H(\bm{\phi}_{\Delta}\|\bm{q})+\epsilon,$ (9)

where $\bm{\phi}_{\Delta}$ is the optimal aligned model at KL divergence
$\Delta$. Then,

$D_{\operatorname{KL}}(\bm{\psi}_{\Delta}\|\bm{\phi}_{\Delta})\leq\alpha(\Delta)\epsilon,$
(10)

where $\alpha(\Delta)$ is defined in (7).

###### Proof.

Note that

$\displaystyle
D_{\operatorname{KL}}(\bm{\psi}_{\Delta}(\cdot|\bm{x})\|\bm{\phi}_{\Delta}(\cdot|\bm{x}))$
$\displaystyle=D_{\operatorname{KL}}(\bm{\psi}_{\Delta}(\cdot|\bm{x})\|\bm{\phi}_{\Delta}(\cdot|\bm{x}))-D_{\operatorname{KL}}(\bm{\phi}_{\Delta}(\cdot|\bm{x})\|\bm{\phi}_{\Delta}(\cdot|\bm{x}))$
$\displaystyle=\mathbb{E}_{\bm{y}\sim\bm{\psi}_{\Delta}(\cdot|\bm{x})}\log\frac{\bm{\psi}_{\Delta}(\bm{y}|\bm{x})}{\bm{p}(\bm{y}|\bm{x})}+\alpha(\Delta)\mathbb{E}_{\bm{y}\sim\bm{\psi}_{\Delta}(\cdot|\bm{x})}\log\frac{1}{\bm{q}(\bm{y}|\bm{x})}+\log
Z_{\bm{x},\alpha(\Delta)}$
$\displaystyle-\mathbb{E}_{\bm{y}\sim\bm{\phi}_{\Delta}(\cdot|\bm{x})}\log\frac{\bm{\phi}_{\Delta}(\bm{y}|\bm{x})}{\bm{p}(\bm{y}|\bm{x})}-\alpha(\Delta)\mathbb{E}_{\bm{y}\sim\bm{\phi}_{\Delta}(\cdot|\bm{x})}\log\frac{1}{\bm{q}(\bm{y}|\bm{x})}-\log
Z_{\bm{x},\alpha(\Delta)}$ (11)
$\displaystyle=D_{\operatorname{KL}}(\bm{\psi}_{\Delta}(\cdot|\bm{x})\|\bm{p}(\cdot\|\bm{x}))-D_{\operatorname{KL}}(\bm{\phi}_{\Delta}(\cdot|\bm{x})\|\bm{p}(\cdot\|\bm{x}))$
$\displaystyle+\alpha(\Delta)\left(H(\bm{\psi}_{\Delta}\|\bm{q})-H(\bm{\phi}_{\Delta}\|\bm{q})\right)$
$\displaystyle\leq\alpha(\Delta)\epsilon,$ (12)

where (11) follows from the fact that $\bm{\phi}_{\Delta}$ could be expressed
by (4) in Lemma 1, and (12) follows from the assumptions of the lemma in (9),
which completes the proof. ∎

Lemma 10 states that any model that satisfies KL constraint $\Delta,$ and
whose expected reward is within $\epsilon$ of that of the optimal aligned
model must be close to the optimal aligned model in KL divergence. We will use
this result to establish such closeness between best-of-$N$ and the optimal
KL-constrained model in the sequel.

Before we proceed, we would like to also observe a connection between the
cumulants of the reward and Rényi cross entropy.

######  Definition 5 (Rényi cross entropy).

For all $t>0,$ let Rényi cross entropy of order $t$ be defined as444The
definition is extended via continuous extension at $t=1.$ (Li et al., 2023,
Eq. (38))

$H_{t}(\bm{p}\|\bm{q}):=\frac{1}{1-t}\log\sum_{\bm{y}\in\bm{\mathcal{Y}}}\bm{p}(\bm{y})\bm{q}(\bm{y})^{t-1}.$
(13)

Rényi cross entropy of order $1$ recovers the cross entropy, i.e.,
$H_{1}(\bm{p}\|\bm{q})=H(\bm{p}\|\bm{q}).$

######  Lemma 3 (cumulant generating function of reward).

Let $E_{\Delta,\rho}(\bm{p},\bm{q})$ be the $\rho$-th cumulant of the reward
under the optimal $\Delta$-aligned model, and for all $\rho\geq 0,$be defined
as555For $\rho=0,$ the definition is extended via continuous extension.

$E_{\Delta,\rho}(\bm{p},\bm{q}):=\frac{1}{\rho}\log\mathbb{E}_{\bm{y}\in\bm{\phi}_{\Delta}(\cdot|\bm{x})}\bm{q}_{\bm{y}|\bm{x}}(\bm{y}|\bm{x})^{\rho}=\frac{1}{\rho}\log\mathbb{E}_{\bm{y}\in\bm{\phi}_{\Delta}(\cdot|\bm{x})}\exp(\rho\,\bm{r}(\bm{x},\bm{y})),$
(14)

where the latter equality is due to the fact that we assumed $R_{\bm{x}}=1.$
Then,

$E_{\Delta,\rho}(\bm{p},\bm{q})=-H_{1+\rho}(\bm{\phi}_{\Delta}\|\bm{q}).$ (15)

###### Proof.

By definition,

$H_{1+\rho}(\bm{\phi}_{\Delta}\|\bm{q})=-\frac{1}{\rho}\log\mathbb{E}_{\bm{y}\in\bm{\phi}_{\Delta}(\cdot|\bm{x})}\bm{q}_{\bm{y}|\bm{x}}(\bm{y}|\bm{x})^{\rho},$
(16)

which completes the proof. ∎

In particular,

$E_{\Delta,0}(\bm{p},\bm{q})=\mathbb{E}_{\bm{y}\in\bm{\phi}_{\Delta}(\cdot|\bm{x})}\log\bm{q}_{\bm{y}|\bm{x}}(\bm{y}|\bm{x})=\mathbb{E}_{\bm{y}\in\bm{\phi}_{\Delta}(\cdot|\bm{x})}\bm{r}(\bm{x},\bm{y})=-H(\bm{\phi}_{\Delta}\|\bm{q}).$
(17)

##  4 Main Results

Equipped with the preliminaries, in this section we derive our main results.
To this end, we make two simplifying assumptions.

######  Assumption 1 (memoryless reference model).

We assume that $\bm{p}_{\bm{y}|\bm{x}}$ is a memoryless source such that the
outcome is a sequence of length $m$ from the $m$-product of the categorical
distribution defined by stochastic vector $p\in\mathcal{S}^{K}_{\zeta}$, where
$\mathcal{S}^{K}_{\zeta}$ denotes the interior of the simplex over alphabet
size $K,$ such that

$\mathcal{S}^{K}_{\zeta}:=\left\\{p:\sum_{k\in[K]}p_{k}=1,\quad
p_{k}>\zeta,\quad\forall k\neq k^{\prime}~{}~{}p_{k}\neq
p_{k^{\prime}}\right\\}.$ (18)

######  Assumption 2 (linear reward).

We assume that the reward is the negative loglikelihood of an alignment
distribution $\bm{q}_{\bm{y}|\bm{x}}$ that is memoryless and a product of
categorical distribution $q\in\mathcal{S}^{K}_{\zeta}$, i.e.,
$\bm{q}_{\bm{y}|\bm{x}}$ satisfies Assumption 18. This immediately implies
that $\bm{r}$ is bounded from above and is linear in the outcome.

We adopt Assumptions 18-2 for brevity in this paper, and comment on extension
to more general classes of language models and reward functions. Also note
that the regularity condition on $\bm{q}_{\bm{y}|\bm{x}}$ is equivalently a
regularity condition on the reward function (i.e. that it is bounded,
additive, and assigns distinct reward to all of possible types).

######  Definition 6 (type).

Let $y^{m}$ be a sequence of length $m$ supported on alphabet of size $K.$
Then the type of sequence $y^{m}$ is denoted by $t(y^{m})$ defined as

$t(y^{m})=\left(\frac{1}{m}\sum_{i\in[m]}\mathbf{1}\\{y_{i}=1\\},\ldots,\frac{1}{m}\sum_{i\in[m]}\mathbf{1}\\{y_{i}=K\\}\right),$
(19)

where $\mathbf{1}\\{\cdot\\}$ is the indicator function.

The type of a sequence denotes its empirical distribution and is a sufficient
statistic of the sequence under the memoryless assumption for the reference
model.

In this setup, our goal is to show that both best-of-$N$ and optimal KL-
constrained RL method return sequences whose type (empirical distribution) is
close to the intersection of the reward contour and the KL divergence contour.
To this end, we first start with the KL-constrained RL solution.

###  4.1 KL-Constrained RL Problem

Under the memoryless setup of Assumption 18-2, the aligned family and optimal
KL-constrained RL solution can be further simplified as follows.

######  Lemma 4 (optimal aligned model is in the same family as the reference
model and reward model).

If Assumptions 18-2 hold, then for each $\bm{x}$,
$\bm{\phi}_{\Delta}(y^{m}|\bm{x})$ is also a memoryless distribution that
satisfies Assumption 18, i.e., there exists a distribution $\phi_{\delta}$
such that $\Delta=m\delta$ and

$\bm{\phi}_{\Delta}(y^{m}|\bm{x})=\prod^{m}_{i=1}\phi_{\delta}(y_{i}).$

Furthemore, $\phi_{\delta}$ is given by

$\phi_{\delta}=T(q,p,\alpha(\delta)),$ (20)

where $T(q,p,\alpha)$ is the mismatched tilt (Salamatian et al., 2019,
Definition 1):

$T(q,p,\alpha)(y)=\frac{p_{i}q_{i}^{\alpha}}{\sum_{j\in[K]}p_{j}q_{j}^{\alpha}}$
(21)

and where

$\alpha(\delta):=\arg_{\alpha\in\mathbb{R}+}\left\\{D_{\operatorname{KL}}(T(q,p,\alpha)\|p)=\delta\right\\}.$
(22)

###### Proof.

The proof follows from Lemma 1 and by identifying that:

$\displaystyle H(\bm{\phi}_{\Delta}\|\bm{q})$
$\displaystyle=H(\phi^{m}_{\delta}\|q^{m})=mH(\phi_{\delta}\|q),$
$\displaystyle D_{\operatorname{KL}}(\bm{\phi}_{\Delta}\|\bm{q})$
$\displaystyle=D_{\operatorname{KL}}(\phi^{m}_{\Delta}\|q^{m})=mD_{\operatorname{KL}}(\phi_{\delta}\|q).$

∎

Figure 1: We consider a ternary memoryless source, where the green point is
$p=(\frac{1}{5},\frac{3}{10},\frac{1}{2})$, the red point is
$q=(\frac{2}{3},\frac{1}{9},\frac{2}{9})$. The green curve characterizes all
$v$ s.t. $D_{\operatorname{KL}}(v\|p)=\Delta=0.11$, which is the KL contour.
The black line characterizes all $w$ s.t. $H(w\|q)=H(T(q,p,\alpha(0.11)\|q),$
which is a linear family. The blue curve is $\mathcal{T}_{p,q},$ i.e., the
aligned family. The unique intersection of the green KL ball and the black
constant reward front is a point on the blue curve which is $\phi_{\Delta}.$
The brown point is the expected value of the type of the best-of-$N$ for
$N=\exp(m\delta)\approx 3,$ where $m=10,$ which is remarkably close to
$\phi_{\Delta}.$

######  Example 1 (ternary memoryless source).

Let us introduce a recurring example of such source on a ternary alphabet
$\mathcal{Y}=\\{1,2,3\\},$ with $K=3.$ Let
$p=(\frac{1}{5},\frac{3}{10},\frac{1}{2})$ and
$q=(\frac{2}{3},\frac{1}{9},\frac{2}{9})$. In Figure 1, the two distributions
are depicted on the ternary probability simplex. A KL divergence contour
$\\{\phi:D_{\operatorname{KL}}(\phi\|p)=\delta\\}$ is depicted via the green
curve. A reward contour $\\{\phi:H(\phi\|p)=c\\}$ is depicted via the black
line. The aligned family $\\{\phi_{\delta}\\}_{\delta\geq 0}$ is depicted via
the blue line.

Equipped with this result, we now show that the sequence of optimal aligned
models as a function the sequence length, $m,$ satisfies a large deviation
principle (LDP).

######  Definition 7 (large deviation principle (LDP)).

A sequence of random variables $\\{Y_{n}:n\in\mathbb{N}\\}$ taking values in
$\mathbb{R}$ satisfies the Large Deviation Principle with rate function
$J:\mathbb{R}\to[0,\infty]$ if $J$ is lower semi-continuous and has compact
level-sets, and for all Borel sets $B$

$\displaystyle-\inf_{t\in B^{\circ}}J(t)$
$\displaystyle\leq\liminf_{n\to\infty}\frac{1}{n}\log\mathbb{P}\\{Y_{n}\in
B\\}\leq\limsup_{n\to\infty}\frac{1}{n}\log\mathbb{P}\\{Y_{n}\in
B\\}\leq-\inf_{t\in\bar{B}}J(t),$

where $B^{\circ}$ is the interior of $B$ and $\bar{B}$ is its closure.

The rate function of the Large Deviation Principle (LDP) thoroughly
characterizes the probability of rare events in which $Y_{n}$ significantly
deviates from its mean. Specifically, an LDP offers a complete description of
the cumulant generating function of $Y_{n}$ by employing Varadhan’s Lemma
(Dembo & Zeitouni, 1998, Theorem 4.3.1). We are now ready to state our next
result which is an LDP for the reward function for sequences generated from
$\bm{\phi}_{\Delta}$, and under Assumptions 18-2.

######  Theorem 1 (LDP for aligned reward).

Let $Y^{m}\sim\phi_{\delta}^{m}.$ The sequence $\\{-\frac{1}{m}\log
q^{m}(Y^{m})\\}_{m\in\mathbb{N}}$ satisfies an LDP with rate function

$J_{\Delta}(t)=D_{\operatorname{KL}}(T(q,p,\beta(t))\|\phi_{\delta}),$ (23)

where $\beta(t)=\arg_{\beta\in\mathbb{R}}\\{H(T(q,p,\beta)\|q)=t\\}.$

###### Proof sketch of Theorem 1.

The proof relies on a connection with the LDP of mismatched information along
with a change of measure along the aligned family to $\phi_{\delta}^{m}$. The
LDP for the mismatched information is itself proven using a proof technique
developed in (Salamatian et al., 2019, Theorem 2). More precisely, one first
constructs sets in the space of distribution and argue that the empirical type
of a sequence falling within these sets constrains the reward. Finally, a
careful application of Sanov’s Theorem (Dembo & Zeitouni, 1998, Theorem
6.2.10) concludes the argument. ∎

As we mentioned above, one practical usecase for the LDP of the reward, is the
derivation of the scaled cumulant of function via Varadhan’s lemma.

######  Corollary 1 (cumulants of the reward under aligned model).

For all $\rho\geq 0,$666The definition is continuously extended at $\rho=0$.
the scaled cumulants of the reward are given by

$\lim_{m\to\infty}\frac{1}{m}\frac{1}{\rho}\log
E_{Y^{m}\sim\phi^{m}_{\delta}}e^{\rho\,r(Y^{m})}=-H_{1+\rho}(\phi_{\delta}\|q).$
(24)

Note that the same result could have been directly derived from the
characterization of the cumulants in Lemma 15 as well. This immediately
implies that the average reward of a draw from $\phi_{\delta}$ concentrates on
$H(\phi_{\delta}\|q):$

$\lim_{m\to\infty}\frac{1}{m}E_{Y^{m}\sim\phi^{m}_{\Delta}}r(Y^{m})=-H(\phi_{\delta}\|q).$
(25)

This characterizes the expected reward that is achieved by the solution to the
KL-constrained RL problem.

###  4.2 Best-of-$N$ Alignment

Next, we provide theoretical guarantees on the best-of-$N$ alignment method
and relate it to the optimal KL-constrained alignment problem. We first start
with proving some properties of the best-of-$N$ policy.

Under Assumptions 18-2, let $\pi^{m}_{N}$ denote best-of-$N$ distribution on
sequences of length $m.$ Intuitively, one would think that $\pi^{m}_{N}$ still
retains the nice memoryless property of $p^{m}$; we show below that, sadly
this property is lost.

######  Example 2 (The set of memoryless sources is not closed under best-
of-$N$).

Let $m=2$ and $N=2$. Let $p$ be a uniform distribution over
$\mathcal{Y}=\\{0,1,2\\}$: $p(y_{1}=0)=0.2$, $p(y_{1}=1)=0.3$,
$p(y_{2}=2)=0.5$. Let the reward function be $r(y=0)=\log_{e}6$, $r(y=1)=0$
and $r(y=2)=\log_{e}2$. Additionally, suppose best-of-$N$ will pick uniformly
for tie-breaking. We can compute directly $\pi^{m}_{N}$ and show that it is no
longer i.i.d.: $\pi^{m}_{N}(y_{1}=0)=\pi^{m}_{N}(y_{2}=0)=209/625$, while,

$\pi^{m}_{N}(y_{1}=0,y_{2}=0)=49/625\neq\pi^{m}_{N}(y_{1}=0)\pi^{m}_{N}(y_{2}=0).$

For completeness, we list $\pi^{m}_{N}$ for all outcomes in Table 1.

Table 1: Full description of $\pi_{N}^{m}$ for Example 2 $\pi_{N}^{m}(y_{1},y_{2})$ | $y_{1}=0$ | $y_{1}=1$ | $y_{1}=2$
---|---|---|---
$y_{2}=0$ | 49/625 | 21/250 | 43/250
$y_{2}=1$ | 21/250 | 81/10000 | 9/125
$y_{2}=2$ | 43/250 | 9/125 | 103/400

Even though $\pi^{m}_{N}$ is not a memoryless distribution, we show that it
still is symmetric and is an exchangeable distribution.

######  Lemma 5 (best-of-$N$ alignment model is exchangeable).

If Assumptions 18-2 hold and the ties in best-of-$N$ algorithm are broken
uniformly at random, then $\pi^{m}_{N}$ is an exchangeable distribution, i.e.,

$\pi^{m}_{N}(X_{i}=x_{i},\ldots,X_{j}=x_{j})=\pi^{m}_{N}(X_{i}=x_{j},\ldots,X_{j}=x_{i}),$

for any $i\neq j$.

###### Proof of Lemma 5.

Without loss of generality, we prove for $i=1$ and $j=2$. We need to show

$P[\pi_{N}^{m}=(x_{1},x_{2},\ldots,x_{m})]=P[\pi_{N}^{m}=(x_{2},x_{1},\ldots,x_{m})].$

Let $\bm{p}(\cdot|\bm{x})=p^{m}$. Let $p_{c}^{m}$ be a coupling of $p^{m}$
such that it will swap first item and second item of $p^{m}$. Since $p^{m}$ is
memoryless and product of identical distributions, the probability mass
function of $p_{c}^{m}$ and $p^{m}$ are the same. By running best-of-$N$
algorithm over $p_{c}^{m}$ and $p^{m}$, we can see that their first and second
item are swapped while having the same probability mass function for their
corresponding best-of-$N$. ∎

In view of the previous example, it may seem hopeless to relate best-of-$N$
aligned model and the optimal aligned model, given that the latter remains a
memoryless source whereas best-of-$N$ does not even satisfy that criterion.
However, we show that for certain values of $m$ and $N$, both the optimal KL-
constrained RL alignment and best-of-$N$ alignment would roughly be doing the
same thing, which we establish using Lemma 10.

To this end, we first state a KL divergence upper bound for the best-of-$N$
strategy.

######  Lemma 6 (KL divergence upper bound for best-of-$N$).

For any $N,$ $\bm{x},$ $m,$ and $p,$

$D_{\operatorname{KL}}(\pi^{m}_{N}(\cdot\|\bm{x})\|p^{m}(\cdot\|\bm{x}))\leq\log
N.$ (26)

###### Proof.

This is a direct corollary of (Beirami et al., 2024, Theorem 1). ∎

Given this result, we set $N=\exp(m\delta)$ such that the optimal KL-
constrained aligned model and best-of-$N$ both satisfy the same upper bound on
their KL divergence, i.e., $\Delta=m\delta$. To show that they are close to
each other in KL divergence using Lemma 10, we need to establish that they
roughly achieve the same expected reward, which then implies the KL divergence
between the two distributions $\pi^{m}_{N}$ and $\phi_{\delta}^{m}$ tends to
zero. We show that not only the two methods have the same reward
asymptotically, but their output types are also similar.

######  Lemma 7.

Let Assumptions 18-2 hold. Let $Y^{m}\sim\pi_{N}^{m}$ be a sequence generated
from the best-of-$N$ method. Let $\delta\in[0,\log\frac{1}{\zeta}]$, and
$N=\exp(m\delta)$. Denote by $t(Y^{m})$ the type of $Y^{m}$ (Definition 6).
Then, for any $\epsilon>0$, we have:

$\lim_{m\to\infty}\mathbb{P}(|t(Y^{m})-\phi_{\delta}|\leq\epsilon)\to 1.$ (27)

The above result implies the following result on the reward of the best-of-$N$
model.

######  Lemma 8 (best-of-$N$ achieves the same reward as optimal aligned
model).

Let Assumptions 18-2 hold. Let $\delta\in[0,\log\frac{1}{\zeta}]$ and
$N=\exp(m\delta)$. Then

$\lim_{m\rightarrow\infty}\frac{1}{m}H(\pi^{m}_{N}(\cdot|\bm{x})\|q^{m}(\cdot|\bm{x}))=\lim_{m\to\infty}\frac{1}{m}H(\phi^{m}_{\delta}(\cdot|\bm{x})\|q^{m}(\cdot|\bm{x}))$

###### Proof.

Let $Y^{m}\sim\pi_{N}^{m}$. We first relate a sample’s type to its reward:

$\mathbb{E}_{x\sim
t(Y^{m})}[\bm{r}(x)]=\sum_{x\in\bm{\mathcal{Y}}}\frac{N_{x}}{m}\bm{r}(x)=\frac{1}{m}\,\bm{r}(Y^{m}),$

where $N_{x}$ is the count of element $x$ over the stream of $m$ tokens of
$Y_{i}^{m}$. By Lemma 7, the type of best-of-$N$ converges to $\phi_{\delta}$,
and hence,

$\displaystyle\lim_{m\rightarrow\infty}\frac{1}{m}\left(H(\pi^{m}_{N}(\cdot|\bm{x})\|q^{m}(\cdot|\bm{x}))-H(\phi^{m}_{\delta}(\cdot|\bm{x})\|q^{m}(\cdot|\bm{x}))\right)$
$\displaystyle=-\lim_{m\rightarrow\infty}\frac{1}{m}\,\bm{r}(Y^{m})-\mathbb{E}_{x\sim\phi_{\Delta}}[\bm{r}(x)]=0,$

which completes the proof. ∎

We now state our main theorem, which says that the best of $N=\exp(m\delta)$
is asymptotically the same to that of the optimal solution of KL-constrained
RL alignment.

######  Theorem 2.

Under Assumptions 18-2, let $\phi_{\delta}^{m}$ be the optimal solution to
Definition 2, and $\pi^{m}_{N}$ be the distribution of the best-of-$N$. For
any $\delta\in[0,\log\frac{1}{\zeta}]$, if $N=\exp(m\delta)$, then we have
that for all $\bm{x}$,

$\lim_{m\to\infty}\frac{1}{m}D_{\operatorname{KL}}(\pi^{m}_{N}(\cdot|\bm{x})\|\phi^{m}_{\delta}(\cdot|\bm{x}))=0.$
(28)

###### Proof.

Combining Lemma 8 with Lemma 10 yields Theorem 2. ∎

Theorem 2 shows that the best-of-$N$ distribution is asymptotically close to
the KL-constrained RL solution. We remark that even though best-of-$N$ is not
necessarily a product distribution as shown in Example 2, it still is close to
the optimal solution of the KL-constrained RL, which is a product distribution
in our setting. We believe that Theorem 2 sheds some light on the remarkable
performance of the best-of-$N$ method when evaluated on the tradeoffs between
expected reward and the KL divergence between the aligned model and the
reference model (reward-KL tradeoffs).

We also empirically validate that the type of best-of-$N$ is close to the
solution for the KL-constrained RL problem for $N=\exp(m\delta)$. To this end,
we revisit the ternary example in Figure 1. As can be seen, even for modest
values of $m$, and $N$, we observe that $\mathbb{E}[t(Y^{m})]$ where
$Y^{m}\sim\pi^{m}_{N}$ is remarkably close to $\phi_{\delta}$.

We have thus shown that the typical behavior of the best-of-$N$ method is
mirrored by the solution to the KL-constrained RL. We conclude this section
with a conjecture: we propose that the atypical behavior of the best-of-$N$
could also be described by a Large Deviation Principle (LDP) with the
identical rate function as that of the KL-constrained RL, as detailed in
Theorem 1.

##  5 Concluding Remarks

In this paper, we considered the language model alignment problem, and studied
two popular alignment techniques: KL-constrained reinforcement learning and
best-of-$N.$ We established several properties of the solutions of both
techniques and established an asymptotic equivalence between the two problems,
providing theoretical justification for the remarkable performance of best-
of-$N$ in practice. While our derivations were obtained under strong
assumptions about the string-generating sources, we believe that these results
may be extended beyond these classes of sources. In particular, we performed
some extra experiments to compare the KL divergence of best-of-$N$ solution,
$\bm{\pi}_{N}$ to that of the optimal KL-regularized RL where the KL
divergence budget for both models is $\Delta$, i.e., $\bm{\phi}_{\Delta}$. We
found that for random $\bm{p}$ and $\bm{q}$ over alphabet of size $K>1000,$
the KL divergence $D_{\operatorname{KL}}(\bm{\pi}_{N}\|\bm{\phi}_{\Delta})$
was always bounded by $0.01$ for all $N\in[1,1000]$ which is striking. Even
for $K<10,$ we empirically observed that
$D_{\operatorname{KL}}(\bm{\pi}_{N}\|\bm{\phi}_{\Delta})<0.5.$ We hope future
research can extend the theoretical investigation to better understand this
phenomenon.

## Acknowledgements

The authors are grateful to Alekh Agarwal, Ananth Balashankar, Jonathan
Berant, Michael Collins, Jacob Eisenstein, Adam Fisch, Jong Lee, and Mahyar
Jafarinodeh for helpful discussions on the fundamentals of language model
alignment.

## References

  * Bai et al. (2022) Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al.  Training a helpful and harmless assistant with reinforcement learning from human feedback.  _arXiv preprint arXiv:2204.05862_ , 2022.
  * Beirami et al. (2018) Ahmad Beirami, Robert Calderbank, Mark M Christiansen, Ken R Duffy, and Muriel Médard.  A characterization of guesswork on swiftly tilting curves.  _IEEE Transactions on Information Theory_ , 65(5):2850–2871, 2018.
  * Beirami et al. (2024) Ahmad Beirami, Alekh Agarwal, Jonathan Berant, Alexander D’Amour, Jacob Eisenstein, Chirag Nagpal, and Ananda Theertha Suresh.  Theoretical guarantees on the best-of-n alignment policy.  _arXiv preprint arXiv:2401.01879_ , 2024.
  * Boyd & Vandenberghe (2004) Stephen Boyd and Lieven Vandenberghe.  _Convex optimization_.  Cambridge university press, 2004.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Christiano et al. (2017) Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei.  Deep reinforcement learning from human preferences.  _Advances in neural information processing systems_ , 30, 2017.
  * Dembo & Zeitouni (1998) A. Dembo and O. Zeitouni.  _Large Deviations Techniques and Applications_.  Springer, 1998.
  * Eisenstein et al. (2023) Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ahmad Beirami, Alex D’Amour, DJ Dvijotham, Adam Fisch, Katherine Heller, Stephen Pfohl, Deepak Ramachandran, Peter Shaw, and Jonathan Berant.  Helping or herding? reward model ensembles mitigate but do not eliminate reward hacking.  _arXiv preprint arXiv:2312.09244_ , 2023.
  * Gao et al. (2023) Leo Gao, John Schulman, and Jacob Hilton.  Scaling laws for reward model overoptimization.  In _International Conference on Machine Learning_ , pp. 10835–10866. PMLR, 2023.
  * Ho et al. (2020) Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  _Advances in neural information processing systems_ , 33:6840–6851, 2020.
  * Korbak et al. (2022a) Tomasz Korbak, Hady Elsahar, Germán Kruszewski, and Marc Dymetman.  On reinforcement learning and distribution matching for fine-tuning language models with no catastrophic forgetting.  _Advances in Neural Information Processing Systems_ , 35:16203–16220, 2022a.
  * Korbak et al. (2022b) Tomasz Korbak, Ethan Perez, and Christopher Buckley.  RL with KL penalties is better viewed as Bayesian inference.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pp. 1083–1091, 2022b.
  * Li et al. (2023) Tian Li, Ahmad Beirami, Maziar Sanjabi, and Virginia Smith.  On tilted losses in machine learning: Theory and applications.  _Journal of Machine Learning Research_ , 24(142):1–79, 2023.
  * Mudgal et al. (2023) Sidharth Mudgal, Jong Lee, Harish Ganapathy, YaGuang Li, Tao Wang, Yanping Huang, Zhifeng Chen, Heng-Tze Cheng, Michael Collins, Trevor Strohman, Jilin Chen, Alex Beutel, and Ahmad Beirami.  Controlled decoding from language models.  _arXiv preprint arXiv:2310.17022_ , 2023.
  * Nakano et al. (2021) Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al.  WebGPT: Browser-assisted question-answering with human feedback.  _arXiv preprint arXiv:2112.09332_ , 2021.
  * Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.  Training language models to follow instructions with human feedback.  _arXiv preprint arXiv:2203.02155_ , 2022.
  * Rafailov et al. (2023) Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn.  Direct preference optimization: Your language model is secretly a reward model.  _Neural Information Processing Systems (NeurIPS)_ , 2023.
  * Salamatian et al. (2019) Salman Salamatian, Litian Liu, Ahmad Beirami, and Muriel Médard.  Mismatched guesswork.  _arXiv preprint arXiv:1907.00531_ , 2019.
  * Sohl-Dickstein et al. (2015) Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  In _International conference on machine learning_ , pp. 2256–2265. PMLR, 2015.
  * Song et al. (2020) Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations.  _arXiv preprint arXiv:2011.13456_ , 2020.
  * Stiennon et al. (2020) Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano.  Learning to summarize with human feedback.  _Advances in Neural Information Processing Systems_ , 33:3008–3021, 2020.
  * Team et al. (2023) Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al.  Gemini: a family of highly capable multimodal models.  _arXiv preprint arXiv:2312.11805_ , 2023.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_ , 2023.
  * Yang & Klein (2021) Kevin Yang and Dan Klein.  FUDGE: Controlled text generation with future discriminators.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pp. 3511–3535, Online, June 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.naacl-main.276.  URL https://aclanthology.org/2021.naacl-main.276.
  * Zhao et al. (2022) Yao Zhao, Mikhail Khalman, Rishabh Joshi, Shashi Narayan, Mohammad Saleh, and Peter J Liu.  Calibrating sequence likelihood improves conditional language generation.  In _The Eleventh International Conference on Learning Representations_ , 2022.

##  Appendix A Proofs of Properties of KL-constrained RL

###### Proof of Lemma 1.

The Lagrangian is written as

$\mathcal{L}(\bm{\phi})=H(\bm{\phi}\|\bm{q})+\lambda\left(D_{\operatorname{KL}}(\bm{\phi}\|\bm{p})-\Delta\right).$
(29)

Fix $\lambda>0$ we can solve for its minimum with respect to $\bm{\phi}$
following steps,

$\displaystyle\min_{\bm{\phi}}\mathcal{L}(\bm{\phi})$
$\displaystyle=\min_{\bm{\phi}}H(\bm{\phi}\|\bm{q})+\lambda\left(D_{\operatorname{KL}}(\bm{\phi}\|\bm{p})-\Delta\right)$
(30)
$\displaystyle=\min_{\bm{\phi}}\sum_{\bm{y}\in{\mathcal{Y}}}\bm{\phi}(\bm{y})\left(\log\frac{1}{\bm{q}(\bm{y})}+\lambda\log\frac{\bm{\phi}(\bm{y})}{\bm{p}(\bm{y})}-\lambda\Delta\right)$
(31)
$\displaystyle=\min_{\bm{\phi}}\lambda\sum_{y}\bm{\phi}(\bm{y})\left(\log\frac{\bm{\phi}(\bm{y})}{\bm{p}(\bm{y})\bm{q}(\bm{y})^{1/\lambda}}-\Delta\right)$
(32)
$\displaystyle=\min_{\bm{\phi}}\lambda\sum_{\bm{y}}\bm{\phi}(\bm{y})\left(\log\frac{\bm{\phi}(\bm{y})}{T(\bm{q},\bm{p},1/\lambda)(\bm{y})}\right)+C(\lambda,\Delta),$
(33) $\displaystyle=\min_{\bm{\phi}}\lambda
D_{\operatorname{KL}}(\bm{\phi}(\bm{y})\|T(\bm{q},\bm{p},1/\lambda)(\bm{y}))+C(\lambda,\Delta),$
(34)

which is uniquely minimized by
$\bm{\phi}(\bm{y})=T(\bm{q},\bm{p},1/\lambda)(\bm{y})$. ∎

##  Appendix B Proofs of Large Deviation Principle (LDP)

Now to prove LDP, we first show some results on the KL-constrained RL.

######  Theorem 3 (LDP for mismatched information).

Let $Y^{m}\sim p^{m}.$ The sequence $\\{-\frac{1}{m}\log
q^{m}(Y^{m})\\}_{m\in\mathbb{N}}$ satisfies an LDP with rate function

$J_{0}(t)=D_{\operatorname{KL}}(T(q,p,\beta(t))\|p),$ (35)

where

$\beta(t)=\arg_{\beta\in\mathbb{R}}\\{H(T(q,p,\beta)\|q)=t\\}.$ (36)

The proof of Theorem 36 relies on a correspondence between mismatched
information, and some sets of distributions, which we will define shortly.
This correspondence is implicitly used in (Beirami et al., 2018, proof of
Theorem 5) and made explicit in (Salamatian et al., 2019, Eq. (25)-(27)). For
$\epsilon\geq 0$ and $\alpha\in\mathbb{R}$, let

$\displaystyle\mathcal{D}(q,\alpha,\epsilon)$
$\displaystyle\triangleq\left\\{\varphi\in\Delta_{\mathcal{X}}:H(\varphi\|q)-H(T(q,\alpha)\|q)\leq\epsilon\right\\}$
(37) $\displaystyle\mathcal{E}(q,\alpha,\epsilon)$
$\displaystyle\triangleq\left\\{\varphi\in\Delta_{\mathcal{X}}:H(\varphi\|q)-H(T(q,\alpha)\|q)\geq-\epsilon\right\\}$
(38) $\displaystyle\mathcal{B}(q,\alpha,\epsilon)$
$\displaystyle\triangleq\left\\{\varphi\in\Delta_{\mathcal{X}}:H(T(q,\alpha)\|q)-H(\varphi\|q)\in[0,\epsilon]\right\\}.$
(39)

The sets above are extensions of tilted weakly typical sets of order $\alpha$
(Beirami et al., 2018, Definition 18), and capture the set of types which are
respectively, more likely, less likely, and as likely according to $q$ than
$T(q,\alpha)$, where we define

$T(q,\alpha):=T(q,u,\alpha),$ (40)

where $u=(1/|\mathcal{Y}|,\ldots,1/|\mathcal{Y}|)$ is the uniform
distribution. For these sets, we then have the following lemma.

######  Lemma 9.

For any $\alpha>0$, the following inclusion relations hold, for sufficiently
large $n$,

$\displaystyle\left|-\frac{1}{m}\log
q(y^{m})-H(T(q,\alpha)\|q)\right|\leq\epsilon\Rightarrow
t(y^{m})\in\mathcal{D}(q,\alpha,2\epsilon),$ (41)
$\displaystyle\left|-\frac{1}{m}\log
q(y^{m})-H(T(q,\alpha)\|q)\right|\leq\epsilon\Rightarrow
t(y^{m})\in\mathcal{E}(q,\alpha,2\epsilon),$ (42)
$\displaystyle\left|-\frac{1}{m}\log
q(y^{m})-H(T(q,\alpha)\|q)\right|\leq\epsilon\Leftarrow
t(y^{m})\in\mathcal{B}(q,\alpha,\epsilon).$ (43)

This was proved implicitly in the proofs of Theorems 3 and 5 in (Beirami et
al., 2018). We are now equipped to provide the proof of the main theorem.

###### Proof of Theorem 36.

Note that as $-\frac{1}{m}\log q(Y^{m})$ takes values in a compact subset
$[0,\log\frac{1}{\zeta}]$ of $\mathbb{R}$ (see Assumptions 18-2), it is
sufficient to prove that the limit below exists and evaluates to the rate
function (see (Beirami et al., 2018, Section V) for a formal discussion),
i.e.,

$\displaystyle\lim_{\epsilon\downarrow
0}\lim_{m\to\infty}\frac{1}{m}\log\mathbb{P}_{p}^{m}\left(\left|-\frac{1}{m}\log
q(Y^{m})-t\right|<\epsilon\right)=-J(t).$ (44)

We proceed with the proof in three separate cases.

Case (a): We let $t\in(H(q),\log\frac{1}{\zeta})$, which implies
$\alpha(t)\in(0,1)$ by monotonicity of $H(T(q,\alpha)\|q)$ for
$\alpha\in\mathbb{R}$. Note that (41) and (43) respectively imply

$\displaystyle\lim_{\epsilon\downarrow
0}\limsup_{m\to\infty}\frac{1}{m}\log\mathbb{P}_{p^{m}}\left(\left|-\frac{1}{m}\log
q(Y^{m})-H(T(q,\alpha(t))\|q)\right|\leq\epsilon\right)$ $\displaystyle\hskip
5.0pt\leq\lim_{\epsilon\downarrow
0}\limsup_{m\to\infty}\frac{1}{m}\log\mathbb{P}_{p^{m}}(t(Y^{m})\in\mathcal{D}(q,\alpha(t),2\epsilon)),$
(45) $\displaystyle\lim_{\epsilon\downarrow
0}\liminf_{m\to\infty}\frac{1}{m}\log\mathbb{P}_{p^{m}}\left(\left|-\frac{1}{m}\log
q(Y^{m})-H(T(q,\alpha(t))\|q)\right|\leq\epsilon\right)$ $\displaystyle\hskip
5.0pt\geq\lim_{\epsilon\downarrow
0}\liminf_{m\to\infty}\frac{1}{m}\log\mathbb{P}_{p^{m}}(t(Y^{m})\in\mathcal{B}(q,\alpha(t),\epsilon)).$
(46)

Thus, it suffices to show that the RHS of (45) and (46) both evaluate to
$-D(T(q,p,\alpha(t))\|p)$. This is done via Sanov’s Theorem. Recall that
Sanov’s Theorem (Dembo & Zeitouni, 1998, Theorem 6.2.10) states that, for a
set of distributions $\mathcal{C}$,

$\displaystyle-\inf_{\gamma\in\mathrm{int}\mathcal{C}}D(\gamma\|p)$
$\displaystyle\leq\liminf_{m\to\infty}\frac{1}{m}\log\mathbb{P}(t(y^{m})\in\mathcal{C})$
$\displaystyle\leq\limsup_{m\to\infty}\frac{1}{m}\log\mathbb{P}(t(y^{m})\in\mathcal{C})$
$\displaystyle\leq-\inf_{\gamma\in\mathrm{cl}\mathcal{C}}D(\gamma\|p).$ (47)

To obtain the upper bound, we apply this result to the set
$\mathcal{D}(q,\alpha(t),2\epsilon/\alpha(t))$. Observing that this holds for
any $\epsilon$, and then letting $\epsilon\downarrow 0$, we get that the RHS
of (45) is upper bounded

$\displaystyle-\lim_{\epsilon\downarrow
0}\inf_{\gamma\in\mathrm{cl}\mathcal{D}(q,\alpha(t),2\epsilon/\alpha(t))}D(\gamma\|p).$
(48)

We now make use of a basic topological fact. Observe that $D(\gamma\|p)$ is
strictly convex in $\gamma$ for a fixed $p$, and thus there is a unique
minimizer $\gamma(t,\epsilon)$. Noting that the minimizer $\gamma(t,\epsilon)$
is in the set $\mathrm{cl}\mathcal{D}(q,\alpha(t),2\epsilon)$, by continuity
of $D(\gamma\|p)$ and compactness of the set. Thus, the collection of
minimizers $\gamma(t,\epsilon)$ is a collection of points such that
$\gamma(t,\epsilon)\in\mathrm{cl}\mathcal{D}(q,\alpha(t),2\epsilon)$. It
follows from compactness that the limit point $\lim_{\epsilon\downarrow
0}\gamma(\epsilon,t)\in\bigcap_{\epsilon>0}\mathrm{cl}\mathcal{D}(q,\alpha(t),2\epsilon)=\mathrm{cl}\mathcal{D}(q,\alpha(t),0)$,
where we have used that $\alpha(t)>0$. Therefore, we have the bound

$\displaystyle\inf_{\gamma\in\Delta_{\mathcal{X}}}\hskip
30.00005ptD(\gamma\|p)$ $\displaystyle\text{subject to }\hskip
15.00002ptH(\gamma\|q)\leq H(T(q,\alpha(t))\|q)$ (49)

Note that this optimization problem is convex, and thus can be solved
analytically by writing the KKT conditions (Boyd & Vandenberghe, 2004), which
give a solution $\gamma_{q,p}(t)\in\mathcal{T}_{q,p}$, and optimal value
$D(\gamma_{q,p}(t)\|p)$.

Analogously, the RHS of (43) can be shown to be lower bounded by $-\inf
D(\gamma\|p)$, where $\gamma\in\mathcal{B}(q,\alpha(t),0)$, by noting
$\mathcal{B}(q,\alpha,0)\subset\mathrm{int}\mathcal{B}(q,\alpha,\epsilon)$,
for any $\epsilon>0$. Again, this optimization can be solved analytically, and
gives the desired output. Putting these results together, we get that

$\displaystyle\lim_{\epsilon\downarrow 0}\lim_{m\to\infty}\frac{1}{m}\log$
$\displaystyle\mathbb{P}_{p}^{m}\left(\left|-\frac{1}{m}\log
q(Y^{m})-H(T(q,\alpha(t))\|q)\right|<\epsilon\right)$
$\displaystyle=-D(\gamma_{q,p}(t)\|p).$ (50)

Case (b): We now let $t\in(0,H(q))$, which implies $\alpha(t)\in(1,\infty)$.
The proof in this case follows from the same step as in Case (a), by replacing
the set $\mathcal{D}(q,\alpha(t),\epsilon)$ with the set
$\mathcal{E}(q,\alpha(t),\epsilon)$.

Case (c): Finally, let $t=H(q)$, or equivalently, $\alpha(t)=1$. In this case,
note that $p\in\mathcal{B}(q,1,\epsilon)$, and thus, by the law of large
numbers and (46), we have that

$\displaystyle\lim_{\epsilon\downarrow
0}\lim_{m\to\infty}\frac{1}{m}\log\mathbb{P}_{p}^{m}\left(\left|-\frac{1}{m}\log
q(Y^{m})-t\right|<\epsilon\right)\geq 0,$ (51)

which implies that $J(t)=0$ in this case. ∎

###### Proof of Theorem 1.

The proof is completed by invoking Theorem 36 with $q$ replaced with
$\phi_{\Delta}$, noticing that $\phi_{\Delta}=T(q,p,\Delta),$ and invoking
Lemma 73 to express $T(q,\phi_{\Delta},\beta)$ in terms of $p$. ∎

###### Proof of Corollary 1.

We use Varadhan’s Lemma (Dembo & Zeitouni, 1998, Theorem 4.3.1), which states
that if a sequence of random variables $A_{m}$ satisfies a LDP with rate
function $J(t)$, then we have:

$\lim_{m\to\infty}\frac{1}{m}\log\mathbb{E}[\exp(mF(A_{m}))]=\sup_{t}\\{F(t)-J(t)\\},$
(52)

for any continuous and bounded function $F$. Applying this result with
$F(t)=\gamma\cdot t$, we obtain:

$\displaystyle\lim_{m\to\infty}$ $\displaystyle\frac{1}{m}\log
E_{Y^{m}\sim\phi^{m}_{\Delta}}e^{\gamma r(Y^{m})}$ (53) $\displaystyle=$
$\displaystyle\sup_{t}\;\\{\gamma
H(T(q,p,\beta(t))\|q)-D(T(q,p,\beta(t))\|\phi_{\Delta})\\}$ $\displaystyle=$
$\displaystyle\sup_{\beta}\;\\{\gamma
H(T(q,p,\beta\|q)-D(T(q,p,\beta\|\phi_{\Delta})\\},$ (54)

where we used the definition of $\beta(t)$ in Theorem 1 in (53), and changed
the optimization variable to $\beta$ in (54). Expanding the RHS, and
collecting terms, we get:

$\displaystyle\;-\sum_{y}T(q,p,\beta)(y)\log\frac{T(q,p,\beta)(y)}{\phi_{\Delta}(y)\cdot
q(y)^{\gamma}}$ $\displaystyle=$
$\displaystyle-\sum_{y}T(q,p,\beta)(y)\log\frac{T(q,p,\beta)(y)}{T(q,\phi_{\Delta},\gamma)(y)}$
$\displaystyle-\log\sum_{y}\phi_{\Delta}(y)q(y)^{\gamma}$ (55)
$\displaystyle=$ $\displaystyle-
D_{\operatorname{KL}}(T(q,p,\beta)\|T(q,\phi_{\Delta},\gamma))-\log\sum_{y}\phi_{\Delta}(y)q(y)^{\gamma}.$

This is maximized when $T(q,p,\beta)=T(q,\phi_{\Delta},\gamma)$, which is
possible since $\phi_{\Delta}$ is on the aligned family between $p$ and $q$.
The proof follows from identifying the remaining term as
$(\gamma+1)H_{\gamma+1}(\phi_{\Delta}\|q)$. ∎

##  Appendix C Proofs of Properties of Best-of-$N$

Let $K=|\mathcal{Y}|$. For a sequence $y^{m},$ let $t({y^{m}})$ denote its
type. The proof of this notion of $m$-grained types, which is defined below.

######  Definition 8.

A discrete distribution $p$ on $\mathcal{Y}$ is called $m$-grained if for all
elements, its probability are multiples of $\frac{1}{m}$, i.e., $\forall
i\in\bm{\mathcal{Y}},\exists c\in\mathbb{N},p(i)=c\cdot\frac{1}{m}$.

See 7

###### Proof of Lemma 7.

Denote by $\mathcal{E}=\\{\lambda:|\lambda-\phi_{\delta}|>\epsilon\\}$, let
$\mathcal{T}$ be the set of $m$-grained distributions, we note that:

$\displaystyle\mathbb{P}(|t(Y^{m})-\phi_{\delta}|>\epsilon)$
$\displaystyle=\mathbb{P}(t(Y^{m})\in\mathcal{E})$ (56)
$\displaystyle\leq\sum_{\lambda\in\mathcal{E}\cap\mathcal{T}}\mathbb{P}(t(Y^{m})=\lambda).$
(57)

Therefore, if we prove that for any valid
$\lambda\in\mathcal{E}\cap\mathcal{T}$, the probability of $Y^{m}$ having type
$\lambda$ is exponentially small (in $m$), our proof is complete by observing
that there are only a polynomial number of valid types.

Recall that the best-of-$N$ is selected such that $Y^{m}$ is the sequence
among $X_{1}^{m},\ldots,X_{N}^{m}\overset{i.i.d.}{\sim}\bm{p}$ which has the
highest reward, i.e. $r(Y^{m})=\max\\{r(X_{1}^{m}),\ldots,r(X)\\}$ For any
valid type $\lambda\in\mathcal{T}$, we thus have:

$\displaystyle\mathbb{P}(t(Y^{m})=\lambda)$
$\displaystyle=N\mathbb{P}(t(X^{m}_{N})=\lambda)\prod_{i=1}^{N-1}\mathbb{P}(r(X^{m}_{i})\leq
r(\lambda)).$ (58)

Next, note that $r(X^{m})\leq r(\lambda)$ can be written equivalently in terms
of the type of $X^{n}$ as $H(t(X^{m})\|q)\leq H(\lambda\|q)$. Letting the set
$\mathcal{A}(\lambda)=\\{t:H(\lambda\|q)>H(t\|q)\\}$, the last term in (58) is
equivalently written as:

$\displaystyle\mathbb{P}(r(X^{m})\leq r(\lambda))$
$\displaystyle=\mathbb{P}(t(X^{m})\in\mathcal{A}(\lambda))$ (59)
$\displaystyle\leq 1-\exp\\{-mD(T(p,q,\alpha(\lambda))\|p)\\},$ (60)

where $\alpha(\lambda)$ is the unique tilt so that
$H(T(p,q,\alpha(\lambda))\|q)=H(\lambda\|q)$ (by Sanov’s Theorem). Therefore,
the product in (58) can be bounded by:

$\displaystyle\prod_{i=1}^{N-1}\mathbb{P}(r(X^{m}_{i})\leq r(\lambda))$
$\displaystyle\leq\left(1-\exp\left\\{-m\left(D(T(p,q,\alpha(\lambda))\|p)\right)\right\\}\right)^{N}$
(61)
$\displaystyle\leq\exp{-m\left[D(T(p,q,\alpha(\lambda))\|p)-\delta\right]_{+}}$
(62)

On the other hand, if $N$ is sufficiently large, precisely if
$\delta>D(T(p,q,\alpha(\lambda))\|\bm{p})$, another bound is tighter, namely:

$\displaystyle\prod_{i=1}^{N-1}\mathbb{P}(r(X^{m}_{i})\leq r(\lambda))$
$\displaystyle\leq\left(1-\exp\left\\{-m\left(D(T(p,q,\alpha(\lambda))\|p)\right)\right\\}\right)^{N}$
(63) $\displaystyle=\left(\left(1-\frac{1}{x}\right)^{x}\right)^{N/x}$ (64)
$\displaystyle\leq\exp{-(\exp{m(\delta-D(T(p,q,\alpha(\lambda)\|p)}))}$ (65)

where we used $(1-1/x)^{x}\leq 1/e$, and
$x=\exp{mD(T(p,q,\alpha(\lambda))\|p)}$. Therefore, we obtain the general
bound:

$\displaystyle\prod_{i=1}^{N-1}\mathbb{P}(r(X^{m}_{i})\leq
r(\lambda))\leq\exp-mF(m,\alpha(\lambda),\delta,p,q),$ (66)

where

$\displaystyle
F(m,\alpha(\lambda),\delta,p,q)=\begin{cases}D(T(p,q,\alpha(\lambda))\|p)-\delta&\text{if
}D(T(p,q,\alpha(\lambda))\|p)>\delta\\\
\frac{1}{m}\exp{m(\delta-D(T(p,q,\alpha(\lambda))\|p))}&\text{if
}D(T(p,q,\alpha(\lambda))\|p)<\delta\\\ 0&\text{if
}D(T(p,q,\alpha(\lambda))\|p)=\delta\end{cases}$ (67)

Then, putting everything together in (58), we get:

$\displaystyle\mathbb{P}(t(Y^{m})=\lambda)$
$\displaystyle\leq\exp\left\\{-m\left[D(\lambda\|p)+F(m,\alpha(\lambda),\delta,p,q)-\delta\right]\right\\}$
(68)
$\displaystyle=\exp\left\\{-m\left[D(\lambda\|T(p,q,\alpha(\lambda))+D(T(p,q,\alpha(\lambda))\|p)+F(m,\alpha(\lambda),\delta,p,q)-\delta\right]\right\\}$
(69)

where (69) follows from the I-Pythagorean Theorem. For (69) to decay to zero
as $m\to\infty$, we must have the exponent be strictly positive. We check this
in three steps:

  * •

Let $D(T(p,q,\alpha(\lambda))\|p)>\delta$, and identifying (67), the exponent
in (69) is:

$\displaystyle
D(\lambda\|T(p,q,\alpha(\lambda))+2(D(T(p,q,\alpha(\lambda))\|p)-\delta)>0$
(70)

and therefore $\mathbb{P}(t(Y^{m})=\lambda)$ goes to zero exponentially fast.

  * •

Let $D(T(p,q,\alpha(\lambda))\|p)<\delta$, then, the exponent in (69) is:

$\displaystyle
D(\lambda\|T(p,q,\alpha(\lambda))+D(T(p,q,\alpha(\lambda))\|p)-\delta+\frac{1}{m}\exp{m(\delta-D(T(p,q,\alpha(\lambda))\|p))}>0,$
(71)

for $m$ large enough.

  * •

Finally, let $D(T(p,q,\alpha(\lambda))\|p)=\delta$, then the exponent becomes:

$\displaystyle D(\lambda\|T(p,q,\alpha(\lambda))),$ (72)

which is positive if and only if $\lambda=T(p,q,\alpha(\lambda))$.

Putting these cases together, we obtain that $\mathbb{P}(t(Y^{m}))$ goes to 0
exponentially fast unless $\lambda=T(p,q,\alpha(\delta))$. ∎

######  Lemma 10.

Let $\alpha,\beta\in\mathbb{R},$ then:

$T(q,T(q,p,\alpha),\beta))=T(q,p,\alpha+\beta).$ (73)

###### Proof.

The proof is completed by noticing

$\displaystyle T(q,T(q,p,\alpha),\beta))$ $\displaystyle\propto
T(q,p,\alpha)q^{\beta}$ $\displaystyle\propto
pq^{\alpha\beta}=T(q,p,\alpha+\beta).$ (74)

∎


A Heavy Ion Fireball freeze-out Dipion Cocktail for Au-Au Collisions at
$\sqrt{s_{NN}}$=200 GeV $p_{t}$ dependence(Part 2).
R.S. Longacrea
aBrookhaven National Laboratory, Upton, NY 11973, USA

###### Abstract

In this paper we use methods developed in Part 1 of The Dipion Cocktail, to
fit the $p_{t}$ dependence of dipions for mid-central Au-Au collisions at
$\sqrt{s_{NN}}$=200 GeV. For the minijet fragmentation part we use PYTHIA
fragmentation as described in Part 1. For the thermal resonance production we
use an exponential growth behavior. The interference between the direct
production of dipion pairs from non-resonance minijet fragmentation and re-
scattering through resonance states gives a measure of the size of the re-
scattering region. This size is contain in the $\alpha$ parameter of Part1. We
assumed a relationship between the $\alpha$ parameter and the mass shift of
the $\rho$ resonance. The data used for the fits comes from the RHIC collider
as measured in the STAR experiment.

##  1 Introduction

The ultra-relativistic heavy ion collision starts out as a state of high
density nuclear matter called the Quark Gluon Plasma(QGP) and expands rapidly
to freeze-out. During the freeze-out phase quarks and gluons form a system of
strongly interacting hadrons. These hadrons continue to expand in a thermal
manner until no further scattering is possible because the system becomes to
dilute. However this transition from quarks and gluons(partons) into hadrons
is not a smooth affair. The expansion is very rapid and some faster or hard
scattered partons fragment directly into hadron through a minijet[1] process.
Thus we have thermal and minijet hadrons present in the last scattering of the
hadrons. The Dipion Cocktail Part 1 considered this mixture of sources and
applied it to the dipion mass spectrum of the heavy ion fireball formed in Au-
Au collisions at $\sqrt{s_{NN}}$=200. Part 1 showed that both thermal or soft
production of hadrons and the minijet fragmented hadrons can be described
through a set of unified formal equations. Part 2(this paper) applies this
formalism to the $p_{t}$ dependence of dipions for Au-Au collisions at
$\sqrt{s_{NN}}=$ 200 GeV and 40% to 80% centrality.

The paper is organized in the following manner:

Sec. 1 Introduction. Sec. 2 Review of the two component model which we use to
fit the dipion data within a set of $p_{t}$ ranges. Sec. 3 Discussion of the
relationship between the $\alpha$ parameter and the mass and widths of
resonances. Sec. 4 we present a fit to 19 $p_{t}$ ranges for Au-Au collisions
at $\sqrt{s_{NN}}=$ 200 GeV and 40% to 80% centrality. Sec. 5 Summary and
Discussion.

##  2 Two component model with Breit-Wigner parameters

In this section we will alter equation 6 of Part 1 so it can use Breit-Wigner
parameters (mass, width) instead of phase shifts. We will also need to modify
the re-scattering part of the equation in order to have the correct threshold
behavior we have introduced in Part 1 for the minijet partial waves. The phase
shift can be written for the $\ell^{th}$ wave as

$cot\delta_{\ell}=\frac{(M_{\ell}^{2}-M_{\pi\pi}^{2})}{M_{\ell}\Gamma_{\ell}},$
(1)

where $M_{\ell}$ is the mass of the resonance in the $\ell$wave and
$\Gamma_{\ell}$ is its total width.

$\Gamma_{\ell}=\Gamma_{0\ell}{\frac{qB_{\ell}(q/q_{s})}{M_{\pi\pi}}\over{\frac{q_{\ell}B_{\ell}(q_{\ell}/q_{s})}{M_{\ell}}}}$
(2)

with $\Gamma_{0\ell}$ the total width at resonance, $B_{\ell}$ is the Blatt-
Weisskopf-barrier factor[2] for the $\ell$ of the resonance, $q$ is the
$\pi\pi$ center mass momentum, $q_{\ell}$ is $q$ at resonance, $M_{\ell}$ is
the mass of the resonance, and $q_{s}$ is center mass momentum related to the
size(1.0 fm is used $q_{s}$ = .200 GeV/c).

Using equation 1 we rewrite equation 6 of Part 1 as

$|T_{\ell}|^{2}=|D_{\ell}|^{2}\frac{sin^{2}\delta_{\ell}}{PS_{\ell}}+\frac{|A_{\ell}|^{2}sin^{2}\delta_{\ell}}{PS_{\ell}}\left|\alpha+PS_{\ell}cot\delta_{\ell}\right|^{2}$
(3)

The $D_{\ell}$ is the thermal production term and is constant except for the
Boltzmann weight(see equation 13 in Part 1). The expected threshold behavior
$q^{2\ell+1}$ comes from the $sin\delta_{\ell}$ term. Since there is
$sin^{2}\delta_{\ell}$ one of the $q^{2\ell+1}$ is killed off by dividing by
$PS_{\ell}$. In Figure 6 of Part 1 we see we have put into our minijet
$A_{\ell}$ the correct threshold $q^{2\ell+1}$ so we need to kill off the
$q^{2\ell+1}$ of the other $sin\delta_{\ell}$ term. Therefore the above
equation for our minijet $A_{\ell}$ we will use

$|T_{\ell}|^{2}=|D_{\ell}|^{2}\frac{sin^{2}\delta_{\ell}}{PS_{\ell}}+\frac{|A_{\ell}|^{2}sin^{2}\delta_{\ell}}{PS_{\ell}^{2}}\left|\alpha+PS_{\ell}cot\delta_{\ell}\right|^{2}$
(4)

Rewriting equation 6 of Part 1 for each partial wave with Breit-Wigner
parameters the first term becomes

$|T_{\ell}|_{1}^{2}=|D_{\ell}|^{2}\frac{M_{\pi\pi}^{2}}{\sqrt{M_{\pi\pi}^{2}+p^{2}_{t}}}exp\frac{-\sqrt{M_{\pi\pi}^{2}+p^{2}_{t}}}{T}\frac{M_{\ell}\Gamma_{\ell}}{(M_{\ell}^{2}-M_{\pi\pi}^{2})^{2}+M_{\ell}^{2}\Gamma_{\ell}^{2}},$
(5)

while the second term

$|T_{\ell}|_{2}^{2}=|A_{\ell}|^{2}\frac{M_{\ell}^{2}\Gamma_{\ell}^{2}}{(M_{\ell}^{2}-M_{\pi\pi}^{2})^{2}+M_{\ell}^{2}\Gamma_{\ell}^{2}}\left|\alpha+\frac{2qB_{\ell}(\frac{q}{q_{s}})(M_{\ell}^{2}-M_{\pi\pi}^{2})}{M_{\pi\pi}M_{\ell}\Gamma_{\ell}}\right|^{2}\left(\frac{M_{\pi\pi}^{2}}{4q^{2}B_{\ell}^{2}(\frac{q}{q_{s}})}\right).$
(6)

$|T|^{2}=\sum_{\ell}|T_{\ell}|^{2}$ (7)

where

$|T_{\ell}|^{2}=|T_{\ell}|_{1}^{2}+|T_{\ell}|_{2}^{2}$ (8)

and
$|A_{0}|^{2}=S(M_{\pi^{+}\pi^{-}})$,$|A_{1}|^{2}=P(M_{\pi^{+}\pi^{-}})$,$|A_{2}|^{2}=D(M_{\pi^{+}\pi^{-}})$,
and $|A_{3}|^{2}=F(M_{\pi^{+}\pi^{-}})$. S, P, D and F comes from subsection
5.2 of Part 1.

##  3 A Relationship between the $\alpha$ parameter and the mass and widths of
resonances.

Equation 6 of Part 1 has an important factor the coefficient $\alpha$. This
coefficient is related to the real part of the $\pi\pi$ re-scattering loop and
is given by equation 9. When the pions re-scatter or interact at a close
distance or a point the real part of the loop $\alpha$ has its maximum value
of $\alpha_{0}$. While if the pions re-scatter or interact at a distance
determined by the diffractive limit the value of $\alpha$ is zero. The
$\alpha$ which is the real part of the re-scattering factor has a simple form
given by

$\alpha=(1.0-\frac{r^{2}}{r_{0}^{2}})\alpha_{0}$ (9)

where $r$ is the radius of re-scattering in fm’s and $r_{0}$ is 1.0 fm or the
limiting range of the strong interaction ranging to $r$ = 0.0 for point like
interactions.

When $\pi\pi$ pairs interact at the diffractive limit their phase shift should
be the same as the phase shift of the vacuum. The same statement is true for
$\pi\pi$ interacting at a point since asymptotic freedom demands that the
strong interaction should have no effect. However values of $\alpha$ in
between zero and $\alpha_{0}$ represent a confined volume where strongly
interacting gluons, quarks and virtual mesons may influence the phase shift of
the $\pi\pi$ system. Phase shifts under a Breit-Wigner parameters assumption
depend on the mass and width of the resonance parameter. In the next section
we use fits to data to determine the relationship between Breit-Wigner
parameters and the value of $\alpha$.

##  4 STAR data dipion $p_{t}$ range (0.2 GeV/c $<$ $p_{t}$ $<$ 4.0 GeV/c)

We have fitted 19 dipion $p_{t}$ ranges(see Table I) using equation 7 above
for the STAR Au-Au collisions at $\sqrt{s_{NN}}=$ 200 GeV 40% to 80%
centrality data. We included minijets up to $\ell$ = 3 and resonances $\sigma$
$\ell$ = 0, $\rho(770)$ $\ell$ = 1, and $f_{2}(1270)$ $\ell$ = 2. Using the
arguments of Sec. 3 of Part 1, we added the $f_{0}$ as a direct thermal term
($|T_{0}|_{1}^{2}$) and only the $\sigma$ interfered with $\ell$ = 0 minijet
background. Two other thermal terms are present in the cocktail, the
$k^{0}_{s}$ and the $\omega_{0}$. All the thermal terms have an exponential
behavior with dipion $p_{t}$. The spectrum of the minijet partial waves is
obtained from PYTHIA[3](see Sec. 5.2 of Part 1). We let the data determine
which minijet partial wave to add. We find only Swave minijet background is
important until $p_{t}$ equal to 1.1 GeV/c. Above 1.5 GeV/c all four minijet
partial waves are used up to Fwave. It should be noted Dwave and Fwave are
small effects. We used PDG[4] for the $f_{2}(1270)$ mass = 1.275 GeV and width
= .185 GeV. The $f_{0}$ was fitted obtaining mass = 0.9727 $\pm$ .0039 GeV and
width = 0.04512 $\pm$ 0.01128 GeV. The $\sigma$ mass and width used was fixed
because it was ill determined. The mass used was mass = 1.011 GeV and width =
1.015 GeV. The $\rho$ mass and width is explained below.

Finally the threshold effective mass region .280 GeV to .430 GeV is dominated
by the Swave and receives contributions from minijet fragmentation, $\pi\pi$
Swave phase shift, $\eta$ decay, HBT adding to the like sign $\pi\pi$
distribution that has been subtracted away from the unlike sign $\pi\pi$ and
the coulomb correction between the charged pions. The minijet fragmentation is
the least known of the effects since we relied on PYTHIA, however there are
large uncertainty in all the other effects. So for these fits we let the
minijet fragmentation be free to fit the data and let the Breit-Wigner
parameters for the $\sigma$ determine the Swave phase shifts plus leaving out
all other effects.

For the $\alpha$ parameter in $p_{t}$ bins up to 1.1 GeV/c the minijet Swave
interference is the determining factor. Above 1.1 GeV/c the Pwave interference
becomes most important. The values of $\alpha$ which gives a reasonable fit
are shown in Table II.

We have determined that the $\sigma$ pole or Breit-Wigner parameters is so far
away from the real axis thus it is too short lived to be influenced by
hadronic interactions. The $\rho$ phase shift being of a life time comparable
to hadronic interaction taking place becomes most sensitive. We have found as
a function of $\alpha$ the best of $\rho$ width is always 0.147 GeV with an
error of $\pm$ .007 GeV. The mass however decreases as $\alpha$ grows,
reaching a minimum of 0.738 GeV at an $\alpha$ of 0.907. This is a mass shift
of 37 MeV. An $\alpha$ of 0.504 is the smallest $\alpha$ we find in our fits.
A mass of 0.775 GeV is the best fit when the value of $\alpha$ is at 0.504.
Using equation 9 in Table II we determine the radius of $\pi\pi$ re-scattering
for each $p_{t}$ bin. The value of $\alpha_{0}$ used in Table II is equal to
2.0 as determined in Appendix B of Part 1. Table II shows an interesting
density effect around dipion $p_{t}$ of 0.6 to 1.0 GeV/c. If one consider that
$p_{t}$ maybe related to fireball size through the idea of hubble flow, then
pions with a $p_{t}$ of around 0.4 GeV/c maybe coming from a less dense region
in the central part of the fireball. This could be a density wave effect.

Table I. The $p_{t}$ ranges bins we fit.

Table I
---
$p_{t}$ bin number | lower edge(GeV/c) | upper edge(GeV/c)
1 | 0.2 | 0.4
2 | 0.4 | 0.6
3 | 0.6 | 0.8
4 | 0.8 | 1.0
5 | 1.0 | 1.2
6 | 1.2 | 1.4
7 | 1.4 | 1.6
8 | 1.6 | 1.8
9 | 1.8 | 2.0
10 | 2.0 | 2.2
11 | 2.2 | 2.4
12 | 2.4 | 2.6
13 | 2.6 | 2.8
14 | 2.8 | 3.0
15 | 3.0 | 3.2
16 | 3.2 | 3.4
17 | 3.4 | 3.6
18 | 3.6 | 3.8
19 | 3.8 | 4.0

Table II. The $\alpha$ value, $\rho$ mass value and radius in each $p_{t}$
range.

Table II
---
$p_{t}$(GeV/c) | $\alpha$ |  $\rho$ mass(GeV) | radius(f)
0.3 | 0.907 $\pm$ .028 | 0.738 $\pm$ .004 | .739 $\pm$ .010
0.5 | 0.806 $\pm$ .025 | 0.748 $\pm$ .003 | .773 $\pm$ .008
0.7 | 0.706 $\pm$ .022 | 0.755 $\pm$ .002 | .804 $\pm$ .007
0.9 | 0.706 $\pm$ .022 | 0.755 $\pm$ .002 | .804 $\pm$ .007
1.1 | 0.806 $\pm$ .025 | 0.748 $\pm$ .003 | .773 $\pm$ .008
1.3 | 0.806 $\pm$ .025 | 0.748 $\pm$ .003 | .773 $\pm$ .008
1.5 | 0.806 $\pm$ .025 | 0.748 $\pm$ .003 | .773 $\pm$ .008
1.7 | 0.806 $\pm$ .025 | 0.748 $\pm$ .003 | .773 $\pm$ .008
1.9 | 0.806 $\pm$ .025 | 0.748 $\pm$ .003 | .773 $\pm$ .008
2.1 | 0.605 $\pm$ .019 | 0.759 $\pm$ .002 | .835 $\pm$ .006
2.3 | 0.504 $\pm$ .016 | 0.775 $\pm$ .001 | .865 $\pm$ .004
2.5 | 0.504 $\pm$ .016 | 0.775 $\pm$ .001 | .865 $\pm$ .004
2.7 | 0.504 $\pm$ .016 | 0.775 $\pm$ .001 | .865 $\pm$ .004
2.9 | 0.504 $\pm$ .016 | 0.775 $\pm$ .001 | .865 $\pm$ .004
3.1 | 0.504 $\pm$ .016 | 0.775 $\pm$ .001 | .865 $\pm$ .004
3.3 | 0.504 $\pm$ .016 | 0.775 $\pm$ .001 | .865 $\pm$ .004
3.5 | 0.504 $\pm$ .016 | 0.775 $\pm$ .001 | .865 $\pm$ .004
3.7 | 0.504 $\pm$ .016 | 0.775 $\pm$ .001 | .865 $\pm$ .004
3.9 | 0.504 $\pm$ .016 | 0.775 $\pm$ .001 | .865 $\pm$ .004

The 19 dipion spectrum for the $p_{t}$ bins are shown in Figure 1 through
Figure 19.

Figure 1: Fit to STAR dipion effective mass distribution (0.2 GeV/c $<$
$p_{t}$ $<$ 0.4 GeV/c) for Au-Au collisions at $\sqrt{s_{NN}}=$ 200 GeV 40% to
80% centrality using equation 7. See text for complete information.

Figure 2: Fit to STAR dipion effective mass distribution (0.4 GeV/c $<$
$p_{t}$ $<$ 0.6 GeV/c) for Au-Au collisions at $\sqrt{s_{NN}}=$ 200 GeV 40% to
80% centrality using equation 7. See text for complete information.

Figure 3: Fit to STAR dipion effective mass distribution (0.6 GeV/c $<$
$p_{t}$ $<$ 0.8 GeV/c) for Au-Au collisions at $\sqrt{s_{NN}}=$ 200 GeV 40% to
80% centrality using equation 7. See text for complete information.

Figure 4: Fit to STAR dipion effective mass distribution (0.8 GeV/c $<$
$p_{t}$ $<$ 1.0 GeV/c) for Au-Au collisions at $\sqrt{s_{NN}}=$ 200 GeV 40% to
80% centrality using equation 7. See text for complete information.

Figure 5: Fit to STAR dipion effective mass distribution (1.0 GeV/c $<$
$p_{t}$ $<$ 1.2 GeV/c) for Au-Au collisions at $\sqrt{s_{NN}}=$ 200 GeV 40% to
80% centrality using equation 7. See text for complete information.

Figure 6: Fit to STAR dipion effective mass distribution (1.2 GeV/c $<$
$p_{t}$ $<$ 1.4 GeV/c) for Au-Au collisions at $\sqrt{s_{NN}}=$ 200 GeV 40% to
80% centrality using equation 7. See text for complete information.

Figure 7: Fit to STAR dipion effective mass distribution (1.4 GeV/c $<$
$p_{t}$ $<$ 1.6 GeV/c) for Au-Au collisions at $\sqrt{s_{NN}}=$ 200 GeV 40% to
80% centrality using equation 7. See text for complete information.

Figure 8: Fit to STAR dipion effective mass distribution (1.6 GeV/c $<$
$p_{t}$ $<$ 1.8 GeV/c) for Au-Au collisions at $\sqrt{s_{NN}}=$ 200 GeV 40% to
80% centrality using equation 7. See text for complete information.

Figure 9: Fit to STAR dipion effective mass distribution (1.8 GeV/c $<$
$p_{t}$ $<$ 2.0 GeV/c) for Au-Au collisions at $\sqrt{s_{NN}}=$ 200 GeV 40% to
80% centrality using equation 7. See text for complete information.

Figure 10: Fit to STAR dipion effective mass distribution (2.0 GeV/c $<$
$p_{t}$ $<$ 2.2 GeV/c) for Au-Au collisions at $\sqrt{s_{NN}}=$ 200 GeV 40% to
80% centrality using equation 7. See text for complete information.

Figure 11: Fit to STAR dipion effective mass distribution (2.2 GeV/c $<$
$p_{t}$ $<$ 2.4 GeV/c) for Au-Au collisions at $\sqrt{s_{NN}}=$ 200 GeV 40% to
80% centrality using equation 7. See text for complete information.

Figure 12: Fit to STAR dipion effective mass distribution (2.4 GeV/c $<$
$p_{t}$ $<$ 2.6 GeV/c) for Au-Au collisions at $\sqrt{s_{NN}}=$ 200 GeV 40% to
80% centrality using equation 7. See text for complete information.

Figure 13: Fit to STAR dipion effective mass distribution (2.6 GeV/c $<$
$p_{t}$ $<$ 2.8 GeV/c) for Au-Au collisions at $\sqrt{s_{NN}}=$ 200 GeV 40% to
80% centrality using equation 7. See text for complete information.

Figure 14: Fit to STAR dipion effective mass distribution (2.8 GeV/c $<$
$p_{t}$ $<$ 3.0 GeV/c) for Au-Au collisions at $\sqrt{s_{NN}}=$ 200 GeV 40% to
80% centrality using equation 7. See text for complete information.

Figure 15: Fit to STAR dipion effective mass distribution (3.0 GeV/c $<$
$p_{t}$ $<$ 3.2 GeV/c) for Au-Au collisions at $\sqrt{s_{NN}}=$ 200 GeV 40% to
80% centrality using equation 7. See text for complete information.

Figure 16: Fit to STAR dipion effective mass distribution (3.2 GeV/c $<$
$p_{t}$ $<$ 3.4 GeV/c) for Au-Au collisions at $\sqrt{s_{NN}}=$ 200 GeV 40% to
80% centrality using equation 7. See text for complete information.

Figure 17: Fit to STAR dipion effective mass distribution (3.4 GeV/c $<$
$p_{t}$ $<$ 3.6 GeV/c) for Au-Au collisions at $\sqrt{s_{NN}}=$ 200 GeV 40% to
80% centrality using equation 7. See text for complete information.

Figure 18: Fit to STAR dipion effective mass distribution (3.6 GeV/c $<$
$p_{t}$ $<$ 3.8 GeV/c) for Au-Au collisions at $\sqrt{s_{NN}}=$ 200 GeV 40% to
80% centrality using equation 7. See text for complete information.

Figure 19: Fit to STAR dipion effective mass distribution (3.8 GeV/c $<$
$p_{t}$ $<$ 4.0 GeV/c) for Au-Au collisions at $\sqrt{s_{NN}}=$ 200 GeV 40% to
80% centrality using equation 7. See text for complete information.

The direct cross sectional yield of the thermally produced states into
$\pi^{+}$ $\pi^{-}$ is shown in Figure 20. The states are $k_{s}$, $\rho$,
$f_{0}$ and $f_{2}$ and the yields come from the exponential fits to the
direct thermal component of equation 7.

Figure 20: The direct thermal $p_{t}$ yeild into the $\pi^{+}\pi^{-}$ channel
as seen in the STAR dipion effective mass fit for the $k_{s}$, $\rho$, $f_{0}$
and $f_{2}$ using equation 7. See text for complete information.

##  5 Summary and Discussion

The Dipion Cocktail Part 2 applies a formalism derived in The Dipion Cocktail
Part 1 which considered a mixture of sources present in the dipion mass
spectrum of the heavy ion fireball. Part 1 showed that both thermal or soft
production of hadrons and the minijet fragmented hadrons can be described
through a set of unified formal equations. Part 2(this paper) applies this
Part 1 formalism to the $p_{t}$ dependence of dipions for Au-Au collisions at
$\sqrt{s_{NN}}=$ 200 GeV and 40% to 80% centrality.

Part 1 started with the basic definition of elastic $\pi\pi$ scattering. Next
showed how re-scattering of pions depends on the unitary condition that
interactions present in the phase shift of an orbital state must interact all
the time. The process of parton fragmentation into dipion states through
unitarity leads to a equation of production and re-scattering in a given
orbital quantum number. This equation(equation 7) has two components in each
orbital state: one being the thermal production of resonances in a dipion
orbital state, the other is the re-scattering of dipions coming from parton or
minijet fragmentation into the dipion orbital state which do not come directly
from the resonance. Unitarity requires that there most be re-scatter through
resonance phase shifts which we defined through Breit-Wigner parameters (mass,
width).

We have fitted 19 dipion $p_{t}$ ranges(see Table I) using equation 7. We
included minijets up to $\ell$ = 3 and resonances $\sigma$ $\ell$ = 0,
$\rho(770)$ $\ell$ = 1, and $f_{2}(1270)$ $\ell$ = 2. Using the arguments of
Sec. 3 of Part 1, we added the $f_{0}$ as a direct thermal term
($|T_{0}|_{1}^{2}$) and only the $\sigma$ interfered with $\ell$ = 0 minijet
background. Two other thermal terms are present in the cocktail, the
$k^{0}_{s}$ and the $\omega_{0}$. All the thermal terms have an exponential
behavior with dipion $p_{t}$ and are shown Figure 20. The spectrum of the
minijet partial waves is obtained from PYTHIA[3](see Sec. 5.2 of Part 1). We
let the data determine which minijet partial wave to add. We find only Swave
minijet background is important until $p_{t}$ equal to 1.1 GeV/c. Above 1.5
GeV/c all four minijet partial waves are used up to Fwave. It should be noted
Dwave and Fwave are small effects. We used PDG[4] for the $f_{2}(1270)$ mass =
1.275 GeV and width = .185 GeV. The $f_{0}$ was fitted obtaining mass = 0.9727
$\pm$ .0039 GeV and width = 0.04512 $\pm$ 0.01128 GeV. The $\sigma$ mass and
width used was fixed because it was ill determined. The mass used was mass =
1.011 GeV and width = 1.015 GeV.

For the $\alpha$ parameter in $p_{t}$ bins up to 1.1 GeV/c the minijet Swave
interference is the determining factor. Above 1.1 GeV/c the Pwave interference
becomes most important. The values of $\alpha$ which gives a reasonable fit
are shown in Table II.

We have determined that the $\sigma$ pole or Breit-Wigner parameters is so far
away from the real axis thus it is too short lived to be influenced by
hadronic interactions. The $\rho$ phase shift being of a life time comparable
to hadronic interaction taking place becomes most sensitive. We have found as
a function of $\alpha$ the best of $\rho$ width is always 0.147 GeV with an
error of $\pm$ .007 GeV. The mass however decreases as $\alpha$ grows,
reaching a minimum of 0.738 GeV at an $\alpha$ of 0.907. This is a mass shift
of 37 MeV. An $\alpha$ of 0.504 is the smallest $\alpha$ we find in our fits.
A mass of 0.775 GeV is the best fit when the value of $\alpha$ is at 0.504.
Using equation 9 in Table II we determine the radius of $\pi\pi$ re-scattering
for each $p_{t}$ range. Table II shows an interesting density effect around
dipion $p_{t}$ of 0.6 to 1.0 GeV/c. If one consider that $p_{t}$ maybe related
to fireball size through the idea of hubble flow, then pions with a $p_{t}$ of
around 0.4 GeV/c maybe coming from a less dense region in the central part of
the fireball. This could be a density wave effect.

##  6 Acknowledgments

This research was supported by the U.S. Department of Energy under Contract
No. DE-AC02-98CH10886. The author thanks William Love for the STAR analysis of
the angular correlation data from Run 4. Also for his assistance in the
production of figures. It is sad that he is gone.

## References

  * [1] T. Trainor, Phys. Rev. C 80 (2009) 044901.
  * [2] F. von Hippel and C. Quigg, Phys. Rev. 5 (1972) 624.
  * [3] T. Sjostrand, M. van Zijil, Phys. Rev. D 36 (1987) 2019.
  * [4] J. Beringer et al. (Particle Data Group), J. Phys. D86 (2012) 010001\.