# Function approximation by deep neural networks with parameters
$\\{0,\pm\frac{1}{2},\pm 1,2\\}$

###### Abstract

In this paper it is shown that $C_{\beta}$-smooth functions can be
approximated by deep neural networks with ReLU activation function and with
parameters $\\{0,\pm\frac{1}{2},\pm 1,2\\}$. The $l_{0}$ and $l_{1}$ parameter
norms of considered networks are thus equivalent. The depth, width and the
number of active parameters of the constructed networks have, up to a
logarithmic factor, the same dependence on the approximation error as the
networks with parameters in $[-1,1]$. In particular, this means that the
nonparametric regression estimation with the constructed networks attains the
same convergence rate as with sparse networks with parameters in $[-1,1]$.

Aleksandr Beknazaryan<EMAIL_ADDRESS>
University of Twente

Keywords: neural networks, function approximation, entropy, nonparametric
regression

##  1 Introduction

The problem of function approximation with neural networks has been of big
interest in mathmatical research for the last several decades. Various results
have been obtained that describe the approximation rates in terms of the
structures of the networks and the properties of the approximated functions.
One of the most remarkable results in this direction is the universal
approximation theorem, which shows that even shallow (but sufficiently wide)
networks can approximate continuous functions arbitrarily well (see [9] for
the overview and possible proofs of the theorem). Also, in [6] it was shown
that integrable functions can be approximated by networks with fixed width.
Those networks, however, may need to be very deep to attain small
approximation errors. Yet, from a pragmatic point of view, and, in particular,
in statistical applications, allowing very big number of network parameters
may be impractical. The reason is that in this case controlling the complexity
of approximant networks at an optimal rate becomes problematic. Complexities
of classes of neural networks are usually described in terms of their covering
numbers and entropies. Those two concepts also play an important role in
various branches of statistics, such as regression analysis, density
estimation and empirical processes (see, e.g., [3], [1], [8]). In particular,
in regression estimation the following dichotomy usually comes up while
selecting the class of functions from which the estimator will be chosen: on
the one hand, the selected class of approximants should be “big” enough to be
able to approximate various non-trivial functions and on the other hand it
should have “small” enough entropy to attain good learning rates. Thus, the
general problem is to obtain powerful classes of functions with well
controllable entropies. As to the powerfulness of classes of neural networks,
it has recently been shown ([10], [13]) that with properly chosen architecture
the classes of sparse deep neural networks with ReLU activation function can
well approximate smooth functions. In particular, it is shown in [13], that
$C_{\beta}$-smooth functions on $[0,1]^{d}$ can be $\varepsilon$-approximated
by deep ReLU networks with $O(\varepsilon^{-d/\beta}\log_{2}(1/\varepsilon))$
active (nonzero) parameters. A similar result for sparse ReLU networks with
parameters in $[-1,1]$ has been obtained in [10]. The number of active
parameters $s$ in those networks is much smaller than the total number of
network parameters and the network depth $L$ depends logarithmically on the
approximation error. Boundedness of parameters of the networks constructed in
[10] implies that the $\varepsilon-$entropy of the approximating networks has
order $O(sL^{2}\log_{2}(1/\varepsilon)).$ The main advantages of this entropy
bound are its logarithmic dependence on $1/\varepsilon$, which allows to take
the covering radius $\varepsilon$ to be very small in applications, and its
linear dependence on the sparisty $s$ and quadratic dependence on the depth
$L$, both of which, as described above, can also be taken to be small. Using
this entropy bound, it is then shown in [10], that if the regression function
is a composition of Hölder smooth functions, then sparse neural networks with
depth $L\lesssim\log_{2}n$ and the number of active parameters $s\sim
n^{\frac{t}{2\beta+t}}\log_{2}n,$ where $\beta>0$ and $t\geq 1$ depend on the
structure and the smoothness of the regression function, attain the minimax
optimal prediction error rate $n^{\frac{-2\beta}{2\beta+t}}$ (up to a
logarithmic factor), where $n$ is the sample size. It would therefore be
desirable to obtain a similar entropy bound for the spaces of networks for
which the above $l_{0}$ (sparsity) regularization is replaced by the better
practically implementable $l_{1}$ regularization.

Networks with $l_{1}$ norm of all parameters bounded by $1$ are considered in
[12]. As in those networks there are at most $1/\varepsilon^{2}$ parameters
outside of the interval $(-\varepsilon^{2},\varepsilon^{2})$, an entropy bound
of order $O((2/L)^{2L-1}/\varepsilon^{2}),$ has been obtained by taking in the
covering networks the remaining parameters to be 0. This bound, however,
depends polynomially on $1/\varepsilon$, and it leads to the convergence rate
of order $1/\sqrt{n}$ for regression estimation with given $n$ samples. As it
is discussed in [12], the rate $1/\sqrt{n}$ is seemingly the best possible for
$l_{1}$ regularized estimators. Alternative approaches of sparsifying neural
networks using derivatives, iterative prunings and clipped $l_{1}$ penalties
are given in [2], [4], [5] and [7].

To combine the advantages of both $l_{0}$ and $l_{1}$ regularizations, as well
as to make the networks easier to encode, we consider networks with parameters
$\\{0,\pm\frac{1}{2},\pm 1,2\\}$. The $l_{0}$ and $l_{1}$ parameter
regularizations of those networks can differ at most by a factor of 2, which,
in particular, allows to employ all the features induced from the sparsity of
networks (including their entropy bounds) while imposing $l_{1}$ constraints
on their parameters. Moreover, discretization of parameters allows to
calculate the exact number of networks (the 0–entropy) required to attain a
given approximation rate. Importantly, the depth, the width and the number of
active parameters in the approximant networks are equivalent to those of
networks constructed in [10]. Hence, for the considered networks the $l_{0}$
parameter regularization can be replaced by the $l_{1}$ parameter
regularization, leading, up to a logarithmic factor, to the same statistical
guarantees as in [10]. In our construction the parameters $\pm 1$ are used to
add/subtract the nodes, change, if necessary, their signs and transfer them to
the next layers. The parameters $\pm\frac{1}{2}$ and $2$ are used to attain
the values of the form $k/2^{j}\in[-1,1]$, $j\in\mathbb{N}$, $k\in\mathbb{Z},$
which can get sufficiently close to any number from $[-1,1]$. Note that this
can also be done using only the parameters $\pm\frac{1}{2}$ and $1$. The
latter, however, would require a larger depth and a bigger number of active
nodes.

Notation. The notation $|\textbf{v}|_{\infty}$ is used for the $l_{\infty}$
norm of a vector $\textbf{v}\in\mathbb{R}^{d}$ and
$\|f\|_{L^{\infty}[0,1]^{d}}$ denotes the sup norm of a function $f$ defined
on $[0,1]^{d}$, $d\in\mathbb{N}$. For $x,y\in\mathbb{R}$ we denote $x\lor
y:=\max\\{x,y\\}$ and $(x)_{+}:=\max\\{0,x\\}$. Also, to make them
multiplicable with preceeding matrices, the vectors from $\mathbb{R}^{d}$,
depending on the context, are considered as matrices from $\mathbb{R}^{d\times
1}$ rather than $\mathbb{R}^{1\times d}$.

##  2 Main result

Consider the set of neural networks with $L$ hidden layers and with ReLU
activation function $\sigma(x)=0\lor x=(x)_{+}$ defined by

$\mathcal{F}(L,\textbf{p}):=\\{f:[0,1]^{d}\to\mathbb{R}^{p_{L+1}}\;|\;\;f(\textbf{x})=W_{L}\sigma_{\textbf{v}_{L}}W_{L-1}\sigma_{\textbf{v}_{L-1}}...W_{1}\sigma_{\textbf{v}_{1}}W_{0}\textbf{x}\\},$

where $W_{i}\in\mathbb{R}^{p_{i}\times p_{i+1}}$ are weight matrices,
$i=0,...,L,$ $\textbf{v}_{i}$ are shift vectors, $i=1,...,L,$ and
$\textbf{p}=(p_{0},p_{1},...,p_{L+1})$ is the width vector with $p_{0}=d$. For
a given shift vector $\textbf{v}=(v_{1},...,v_{p})$ and a given input vector
$\textbf{y}=(y_{1},...,y_{p})$ the action of shifted activation function
$\sigma_{\textbf{v}}$ on $\textbf{y}^{\top}$ is defined as

$\sigma_{\textbf{v}}(\textbf{y}^{\top})=\begin{pmatrix}\sigma(y_{1}-v_{1}),\cdot\cdot\cdot,\sigma(y_{p}-v_{p})\end{pmatrix}^{\top}.$

It is assumed that network parameters (the entries of matrices $W_{i}$ and
shift vectors $\textbf{v}_{i}$) are all in $[-1,1]$. For $s\in\mathbb{N}$ let
$\mathcal{F}(L,\textbf{p},s)$ be the subset of $\mathcal{F}(L,\textbf{p})$
consisting of networks with at most $s$ nonzero parameters. In [10], Theorem
5, the following approximation of $\beta$-Hölder functions belonging to the
ball

$\displaystyle\mathcal{C}^{\beta}_{d}(K)=\bigg{\\{}f:[0,1]^{d}\to\mathbb{R}:\sum\limits_{0\leq|\boldsymbol{\alpha}|<\beta}\|\partial^{\boldsymbol{\alpha}}f\|_{L^{\infty}[0,1]^{d}}+\sum\limits_{|\boldsymbol{\alpha}|=\lfloor\beta\rfloor}\sup\limits_{\begin{subarray}{c}\textbf{x},\textbf{y}\in[0,1]^{d}\\\
\textbf{x}\neq\textbf{y}\end{subarray}}\frac{|\partial^{\boldsymbol{\alpha}}f(\textbf{x})-\partial^{\boldsymbol{\alpha}}f(\textbf{y})|}{|\textbf{x}-\textbf{y}|_{\infty}^{\beta-\lfloor\beta\rfloor}}\leq
K\bigg{\\}}$

with networks from $\mathcal{F}(L,\textbf{p},s)$ is given:

######  Theorem 1.

For any function $f\in\mathcal{C}^{\beta}_{d}(K)$ and any integers $m\geq 1$
and $N\geq(\beta+1)^{d}\lor(K+1)e^{d},$ there exists a network
$\tilde{f}\in\mathcal{F}(L,\emph{{p}},s)$ with depth

$L=8+(m+5)(1+\lceil\log_{2}(d\lor\beta)\rceil),$

width

$|\emph{{p}}|_{\infty}=6(d+\lceil\beta\rceil)N$

and number of nonzero parameters

$s\leq 141(d+\beta+1)^{3+d}N(m+6),$

such that

$\|\tilde{f}-f\|_{L^{\infty}[0,1]^{d}}\leq(2K+1)(1+d^{2}+\beta^{2})6^{d}N2^{-m}+K3^{\beta}N^{-\frac{\beta}{d}}.$

The proof of the theorem is based on local sparse neural network approximation
of Taylor polynomials of the function $f$.

Our goal is to attain an identical approximation rate for networks with
parameters in $\\{0,\pm\frac{1}{2},\pm 1,2\\}$. In our construction we will
omit the shift vectors (by adding a coordinate $1$ to the input vector x) and
will consider the networks of the form

$\bigg{\\{}f:[0,1]^{d}\to\mathbb{R}\;|\;\;f(\textbf{x})=W_{L}\circ\sigma\circ
W_{L-1}\circ\sigma\circ...\circ\sigma\circ W_{0}(1,\textbf{x})\bigg{\\}}$ (1)

with weight matrices $W_{i}\in\mathbb{R}^{p_{i}\times p_{i+1}}$, $i=0,...,L,$
and with width vector $\textbf{p}=(p_{0},p_{1},...,p_{L+1}),$ $p_{0}=d$. In
this case the ReLU activation function $\sigma(x)$ acts coordinate-wise on the
input vectors. Let $\mathcal{\widetilde{F}}(L,\textbf{p})$ be the set of
networks of the form (1) with parameters in $\\{0,\pm\frac{1}{2},\pm 1,2\\}$.
For $s\in\mathbb{N}$ let $\mathcal{\widetilde{F}}(L,\textbf{p},s)$ be the
subset of $\mathcal{\widetilde{F}}(L,\textbf{p})$ with at most $s$ nonzero
parameters. We then have the following

######  Theorem 2.

For any function $f\in\mathcal{C}^{\beta}_{d}(K)$ and any integers $m\geq 1$
and $N\geq(\beta+1)^{d}\lor(K+1)e^{d},$ there exists a network
$\tilde{f}\in\mathcal{\widetilde{F}}(\tilde{L},\tilde{\textbf{p}},\tilde{s})$
with depth

$\tilde{L}\leq 4{\Delta}+2L,$

width

$|\tilde{\textbf{p}}|_{\infty}\leq 2(1+d+R+\Delta)\lor
2^{d}|{\textbf{p}}|_{\infty}$

and number of nonzero parameters

$\tilde{s}\leq(1+d+R+\Delta)\tilde{L}+2^{d}s,$

such that

$\|\tilde{f}-f\|_{L^{\infty}[0,1]^{d}}\leq(2K+1)(1+d^{2}+\beta^{2})12^{d}N2^{-m}+(K+1)3^{\beta}N^{-\frac{\beta}{d}},$

where $\Delta\leq 2\log_{2}(N^{\beta+d}Ke^{d})$, $R\leq(2\beta)^{d}N$ and
$L,\textbf{p}$ and $s$ are the same as in Theorem 1.

Let us now compare the above two theorems. First, the approximation errors in
those theorems differ by a constant factor depending only on the input
dimension $d$ (note that the values of $\beta,d$ and $K$ are assumed to be
fixed). The depths and the number of nonzero parameters of the networks
presented in Theorems 1 and 2 differ at most by $\log_{2}N$ multiplied by a
constant depending on $\beta,d$ and $K$, and the maximal widths of those
networks differ at most by a constant factor $C(\beta,d,K)$. Thus, the
architecture and the number of active parameters of network given in Theorem 2
have, up to a logarithimc factor, the same dependence on the approximation
error as the network given in Theorem 1.

Application to nonparametric regression. Consider a nonparametric regression
model

$Y_{i}=f_{0}(\textbf{X}_{i})+\epsilon_{i},$

where $f_{0}:[0,1]^{d}\to[-F,F]$ is the unknown regression function that needs
to be recovered from $n$ observed iid pairs $(\textbf{X}_{i},Y_{i})$,
$i=1,...,n$. The standard normal noise variables $\epsilon_{i}$ are assumed to
be independent of $\textbf{X}_{i}$. For a set of functions $\mathcal{F}$ from
$[0,1]^{d}$ to $[-F,F]$ and for an estimator $\hat{f}\in\mathcal{F}$ of
$f_{0}$ define

$\Delta_{n}=\Delta_{n}(\hat{f},f_{0},\mathcal{F})=\operatorname{\mathbb{E}}_{f_{0}}\bigg{[}\frac{1}{n}\sum_{i=1}^{n}(Y_{i}-\hat{f}(\textbf{X}_{i}))^{2}-\inf\limits_{f\in\mathcal{F}}\frac{1}{n}\sum_{i=1}^{n}(Y_{i}-f(\textbf{X}_{i}))^{2}\bigg{]}.$

The subscript $f_{0}$ indicates that the expectation is taken over the
training data generated by our regression model and
$\Delta_{n}(\hat{f},f_{0},\mathcal{F})$ measures how close the estimator
$\hat{f}$ is to the empirical risk minimizer. Let also

$R(\hat{f},f_{0})=\operatorname{\mathbb{E}}_{f_{0}}[(\hat{f}(\textbf{X})-f_{0}(\textbf{X}))^{2}]$

be the prediction error of the estimator $\hat{f},$ where
$\textbf{X}\stackrel{{\scriptstyle\mathcal{D}}}{{=}}\textbf{X}_{1}$ is
independent of the sample $(\textbf{X}_{i},Y_{i})$. The following oracle-type
inequality is obtained in [10], Lemma 4:

######  Lemma 2.1.

For any $\delta\in(0,1]$

$R(\hat{f},f_{0})\leq
4\bigg{[}\inf\limits_{f\in\mathcal{F}}\operatorname{\mathbb{E}}[(f(\textbf{X})-f_{0}(\textbf{X}))^{2}]+F^{2}\frac{18\log_{2}\mathcal{N}(\delta,\mathcal{F},\|\cdot\|_{\infty})+72}{n}+32\delta
F+\Delta_{n}\bigg{]},$

where $\mathcal{N}(\delta,\mathcal{F},\|\cdot\|_{\infty})$ is the covering
number of $\mathcal{F}$ of radius $\delta$ taken with respect to the
$\|\cdot\|_{\infty}$ distance of functions on $[0,1]^{d}$.

Assume that $f_{0}\in\mathcal{C}^{\beta}_{d}(K)$ with $F\geq\max(K,1)$. Taking
in Theorem 2 $r=d$, $m=\lceil\log_{2}n\rceil$ and $N=n^{\frac{d}{2\beta+d}},$
we get the existence of a network
$\tilde{f}_{n}\in\mathcal{\widetilde{F}}(\tilde{L}_{n},\tilde{\textbf{p}}_{n},\tilde{s}_{n})$
with $\tilde{L}_{n}\leq c\log_{2}n$, $|\tilde{\textbf{p}}_{n}|_{\infty}\leq
cn^{\frac{d}{2\beta+d}}$ and $\tilde{s}_{n}\leq
cn^{\frac{d}{2\beta+d}}\log_{2}n$ such that

$\|\tilde{f}_{n}-f_{0}\|^{2}_{L^{\infty}[0,1]^{d}}\leq
cn^{\frac{-2\beta}{2\beta+d}},$ (2)

where $c=c(\beta,d,F)$ is some constant. In order to apply Lemma 2.1 it
remains to estimate the covering number
$\mathcal{N}(\delta,\mathcal{\widetilde{F}}(\tilde{L}_{n},\tilde{\textbf{p}}_{n},\tilde{s}_{n}),\|\cdot\|_{\infty})$.
Note however, that since the parameters of networks from
$\mathcal{\widetilde{F}}(\tilde{L}_{n},\tilde{\textbf{p}}_{n},\tilde{s}_{n})$
belong to the discrete set $\\{0,\pm\frac{1}{2},\pm 1,2\\},$ we can calculate
the exact number of networks from
$\mathcal{\widetilde{F}}(\tilde{L}_{n},\tilde{\textbf{p}}_{n},\tilde{s}_{n})$,
or, in other words, we can upper bound the covering number of radius
$\delta=0$. Indeed, as there are at most
$(\tilde{L}_{n}+1)|\tilde{\textbf{p}}_{n}|_{\infty}^{2}$ parameters in the
networks from
$\mathcal{\widetilde{F}}(\tilde{L}_{n},\tilde{\textbf{p}}_{n},\tilde{s}_{n}),$
then for a given $s$ there are at most
$\bigg{(}(\tilde{L}_{n}+1)|\tilde{\textbf{p}}_{n}|_{\infty}^{2}\bigg{)}^{s}$
ways to choose $s$ nonzero parameters. As the nonzero parameters can take one
of the 5 values $\\{\pm\frac{1}{2},\pm 1,2\\},$ then the total number of
networks from
$\mathcal{\widetilde{F}}(\tilde{L}_{n},\tilde{\textbf{p}}_{n},\tilde{s}_{n})$
is bounded by

$\sum\limits_{s\leq\tilde{s}_{n}}\bigg{(}5(\tilde{L}_{n}+1)|\tilde{\textbf{p}}_{n}|_{\infty}^{2}\bigg{)}^{s}\leq\bigg{(}5(\tilde{L}_{n}+1)|\tilde{\textbf{p}}_{n}|_{\infty}^{2}\bigg{)}^{\tilde{s}_{n}+1}.$

Together with (2) and Lemma 2.1, for the empirical risk minimizer

$\hat{f}_{n}\in\operatorname*{arg\,min}_{f\in\mathcal{\widetilde{F}}(\tilde{L}_{n},\tilde{\textbf{p}}_{n},\tilde{s}_{n})}\sum_{i=1}^{n}(Y_{i}-f(\textbf{X}_{i}))^{2}$

we get an existence of a constant $C=C(\beta,d,F)$ such that

$R(\hat{f}_{n},f_{0})\leq Cn^{\frac{-2\beta}{2\beta+d}}\log_{2}^{2}n$ (3)

which coincides, up to a logarithmic factor, with the minimax estimation rate
$n^{\frac{-2\beta}{2\beta+d}}$ of the prediction error for $\beta$-smooth
functions.

######  Remark 2.1.

As the parameters of networks from
$\mathcal{\widetilde{F}}(\tilde{L}_{n},\tilde{\textbf{p}}_{n},\tilde{s}_{n})$
belong to $\\{0,\pm\frac{1}{2},\pm 1,2\\}$, then, instead of defining the
sparsity constraint $\tilde{s}_{n}$ to be the maximal number of nonzero
parameters, we could define $\tilde{s}_{n}$ to be the upper bound of $l_{1}$
norm of all parameters of networks from
$\mathcal{\widetilde{F}}(\tilde{L}_{n},\tilde{\textbf{p}}_{n},\tilde{s}_{n})$.
As the $l_{0}$ and $l_{1}$ parameter norms of networks from
$\mathcal{\widetilde{F}}(\tilde{L}_{n},\tilde{\textbf{p}}_{n})$ can differ at
most by a factor of 2, then this change of notation would lead to the same
convergence rate as in (3).

##  3 Proofs

One of the ways to approximate functions by neural networks is based on the
neural network approximation of local Taylor polynomials of those functions
(see, e.g., [10], [13]). Thus, in this procedure, approximation of the product
$xy$ given the input $(x,y)$ becomes crucial. The latter is usually done by
representing the product $xy$ as a linear combination of functions that can be
approximated by neural network-implementable functions. For example, the
approximation algorithm presented in [10] is based on the approximation of a
function $g(x)=x(1-x)$, which then leads to an approximation of the product

$xy=g\bigg{(}\frac{x-y+1}{2}\bigg{)}-g\bigg{(}\frac{x+y}{2}\bigg{)}+\frac{x+y}{2}-\frac{1}{4}.$
(4)

The key observation is that the function $g(x)$ can be approximated by
combinations of triangle waves and the latter can be easily implemented by
neural networks with ReLU activation function. In the proof of Theorem 1,
neural network approximation of function $(x,y)\mapsto xy$ is followed by
approximation of the product $(x_{1},...,x_{r})\mapsto\prod_{j=1}^{r}x_{j}$
which then leads to approximation of monomials of degree up to $\beta$. The
result then follows by local approximation of Taylor polynomials of $f$. Below
we show that all those approximations can also be performed using only the
parameters $\\{0,\pm\frac{1}{2},\pm 1,2\\}$.

######  Lemma 3.1.

For any positive integer $m$, there exists a network _Mult_
${}_{m}\in\mathcal{\widetilde{F}}(2m+4,\emph{{p}})$, with $p_{0}=3,$
$p_{L+1}=1$ and $|\emph{{p}}|_{\infty}=9,$ such that

$|\textrm{{Mult}}_{m}(1,x,y)-xy|\leq 2^{-m},\quad\textrm{for
all}\;x,y\in[0,1].$ (5)

###### Proof.

Consider the functions $T^{k}:[0,2^{2-2k}]\to[0,2^{-2k}],k\in\mathbb{N},$
defined by

$T^{k}(x):=(x/2)_{+}-(x-2^{1-2k})_{+}=T_{+}(x)-T_{-}^{k}(x),$ (6)

where $T_{+}(x):=(x/2)_{+}$ and $T_{-}^{k}(x):=(x-2^{1-2k})_{+}$. In [11],
Lemma A.1, it is shown that for the functions $R^{k}:[0,1]\to[0,2^{-2k}],$

$R^{k}=T^{k}\circ T^{k-1}\circ...\circ T^{1},$ (7)

and for any positive integer $m$,

$|g(x)-\sum_{k=1}^{m}R^{k}(x)|\leq 2^{-m},\quad x\in[0,1],$ (8)

where $g(x)=x(1-x)$. Taking into account (4) and (8), we need to construct a
network that computes

$(1,x,y)\mapsto\bigg{(}\sum_{k=1}^{m+1}R^{k}\bigg{(}\frac{x-y+1}{2}\bigg{)}-\sum_{k=1}^{m+1}R^{k}\bigg{(}\frac{x+y}{2}\bigg{)}+\frac{x+y}{2}-\frac{1}{4}\bigg{)}_{+}\wedge
1.$ (9)

Let us first construct a network $N_{m}$ with depth $2m$, width $4$ and
weights $\\{0,\pm\frac{1}{2},\pm 1\\}$ that computes

$(1/4,T_{+}(u),h(u),T_{-}^{1}(u))\mapsto\sum_{k=1}^{m+1}R^{k}(u)+h(u),\quad
u\in[0,1].$

For this goal, we modify the network presented in [11], Fig. 2, to assure that
the parameters are all in $\\{0,\pm\frac{1}{2},\pm 1\\}$. More explicitly,
denote

$A:=\begin{pmatrix}\frac{1}{2}&0&0&0\\\ 0&\frac{1}{2}&0&-\frac{1}{2}\\\
0&1&1&-1\\\ -\frac{1}{2}&1&0&-1\end{pmatrix}$

and

$B:=\begin{pmatrix}\frac{1}{2}&0&0&0\\\ 0&1&0&0\\\ 0&0&1&0\\\
0&0&0&1\end{pmatrix}.$

Then

$N_{m}=\sigma\circ(0\;\;\ 1\;\;\ 1\;\;-1)\circ\sigma\circ B\circ\sigma\circ
A\circ...\circ\sigma\circ B\circ\sigma\circ A\circ\sigma\circ
B\circ\sigma\circ A,$

where each of the mutually succeeding matrices $A$ and $B$ appears in the
above representation $m$ times. Using parameters $\\{0,\pm\frac{1}{2},\pm
1\\}$, for a given input $(1,x,y)$ the first two layers of the network Multm
compute the vector

$\bigg{(}1,\frac{1}{4},T_{+}\bigg{(}\frac{x-y+1}{2}\bigg{)},\bigg{(}\frac{x+y}{2}\bigg{)}_{+},T_{-}^{1}\bigg{(}\frac{x-y+1}{2}\bigg{)},\frac{1}{4},T_{+}\bigg{(}\frac{x+y}{2}\bigg{)},\frac{1}{4},T_{-}^{1}\bigg{(}\frac{x+y}{2}\bigg{)}\bigg{)}$

(note that as in our construction we omit shift vectors, throughout the whole
construction we will keep the first coordinate equal to $1$). We then apply
the network $N_{m}$ to the first and last four coordinates of the above vector
that follow the first coordinate $1$. We thus obtain a network with $2m+2$
hidden layers and of width $9$ that computes

$(1,x,y)\mapsto\bigg{(}1,\sum_{k=1}^{m+1}R^{k}\bigg{(}\frac{x-y+1}{2}\bigg{)}+\frac{x+y}{2},\sum_{k=1}^{m+1}R^{k}\bigg{(}\frac{x+y}{2}\bigg{)}+\frac{1}{4}\bigg{)}.$
(10)

Finally, the last two layers of Multm compute
$(1,u,v)\mapsto(1-(1-(u-v))_{+})_{+}$ applied to the vector obtained in (10)
(note that this computation only requires parameters $0$ and $\pm 1$). We thus
get a network Multm computing (9) and the inequality (5) follows by combining
(4) and (8). $\square$

######  Lemma 3.2.

For any positive integer $m$, there exists a network _Mult
${}^{r}_{m}\in\mathcal{\widetilde{F}}(L,\textbf{p})$_, with
$L=(2m+5)\lceil\log_{2}r\rceil,$ $p_{0}=r+1,$ $p_{L+1}=1$ and
$|\emph{{p}}|_{\infty}\leq 9r,$ such that

$|\emph{Mult}^{r}_{m}(1,\emph{{x}})-\prod_{i=1}^{r}x_{i}|\leq
r^{2}2^{-m},\quad\textit{for
all}\;\emph{{x}}=(x_{1}\;...\;x_{r})\in[0,1]^{r}.$

###### Proof.

In order to approximate the product $\prod_{i=1}^{r}x_{i}$ we first pair the
neighbouring entries to get the triples $(1,x_{k},x_{k+1})$, and apply the
previous lemma to each of those triples to obtain the values
$\textrm{{Mult}}_{m}(1,x_{k},x_{k+1})$. We repeat this procedure
$q:=\lceil\log_{2}r\rceil$ times, until there is only one entry left. As
pairing the entries requires only parameters $0$ and $1$, then it follows from
the previous lemma that the entries of the constructed network are in
$\\{0,\pm\frac{1}{2},\pm 1\\}$. Using Lemma 3.1 and applying the inequality
$|\textrm{{Mult}}_{m}(1,x,y)-tz|\leq 2^{-m}+|x-z|+|y-t|$, $x,y,z,t\in[0,1]$,
$q$ times we get
$|\textrm{Mult}^{r}_{m}(1,{\textbf{x}})-\prod_{i=1}^{r}x_{i}|\leq
3^{q-1}2^{-m}\leq r^{2}2^{-m}$. $\square$

For $\gamma>0$ let $C_{d,\gamma}$ denote the number of $d$-dimensional
monomials $\textbf{x}^{\boldsymbol{\alpha}}$ with degree
$|\boldsymbol{\alpha}|<\gamma$. Note that $C_{d,\gamma}<(\gamma+1)^{d}$. From
Lemma 3.2 it follows that using weights $\\{0,\pm\frac{1}{2},\pm 1\\},$ we can
simultaneously approximate monomials up to degree $\gamma$ (see also [11],
Lemma A.4):

######  Lemma 3.3.

There exists a network _Mon_
${}_{m,\gamma}^{d}\in\mathcal{\widetilde{F}}(L,\emph{{p}})$ with
$L\leq(2m+5)\lceil\log_{2}(\gamma\lor 1)\rceil+1,$ $p_{0}=d+1$,
$p_{L+1}=C_{d,\gamma}$ and $|\emph{{p}}|_{\infty}\leq 9\lceil\gamma\rceil
C_{d,\gamma}$ such that

$\bigg{|}\emph{Mon}_{m,\gamma}^{d}(1,\emph{{x}})-(\emph{{x}}^{\boldsymbol{\alpha}})_{|{\boldsymbol{\alpha}}|<\gamma}\bigg{|}_{\infty}\leq\gamma^{2}2^{-m},\quad\emph{{x}}\in[0,1]^{d}.$

We now present the final stage of the approximation, that is, the local
approximation of Taylor polynomials of $f$.

Proof of Theorem 2. For a given $N$ let $\tilde{N}\geq N$ be the smallest
integer with $\tilde{N}=(2^{\nu}+1)^{d}$ for some $\nu\in\mathbb{N}$. Note
that $\tilde{N}/2^{d}\leq N\leq\tilde{N}.$ We are going to apply Theorem 1
with $N$ in the condition of that theorem replaced by $\tilde{N}$. For
$\textbf{a}\in[0,1]^{d}$ let

$P_{\textbf{a}}^{\beta}f(\textbf{x})=\sum\limits_{0\leq|\boldsymbol{\alpha}|<\beta}(\partial^{\boldsymbol{\alpha}}f)(\textbf{a})\frac{(\textbf{x}-\textbf{a})^{\boldsymbol{\alpha}}}{\boldsymbol{\alpha}!}:=\sum\limits_{0\leq|\boldsymbol{\gamma}|<\beta}c_{{\textbf{a},\boldsymbol{\gamma}}}\textbf{x}^{\boldsymbol{\gamma}}$
(11)

be the partial sum of Taylor series of $f$ around a. Choose $M$ to be the
largest integer such that $(M+1)^{d}\leq\tilde{N},$ that is, $M=2^{\nu}$, and
consider the set of $(M+1)^{d}$ grid points
$\textbf{D}(M):=\\{\textbf{x}_{\boldsymbol{\ell}}=(\ell_{j}/M)_{j=1,...,d}:\boldsymbol{\ell}=(\ell_{1},...,\ell_{d})\in\\{0,1,...,M\\}^{d}\\}$.
Denoting
$\textbf{x}_{\boldsymbol{\ell}}=(x_{1}^{\boldsymbol{\ell}},...,x_{d}^{\boldsymbol{\ell}})$
it is shown in [11], Lemma B.1, that

$\|P^{\beta}f-f\|_{L^{\infty}[0,1]^{d}}\leq KM^{-\beta},$ (12)

where

$P^{\beta}f(\textbf{x})=\sum\limits_{\textbf{x}_{\boldsymbol{\ell}}\in\textbf{D}(M)}P_{\textbf{x}_{\boldsymbol{\ell}}}^{\beta}f(\textbf{x})\prod_{j=1}^{d}(1-M|x_{j}-x_{j}^{\boldsymbol{\ell}}|)_{+}.$

As in our construction we only use parameters $\\{0,\pm\frac{1}{2},\pm
1,2\\}$, we need to modify the coefficients given in (11) to make them
implementable by those parameters. Denote $B:=\lfloor 2Ke^{d}\rfloor$ and let
$b\in\mathbb{N}$ be the smallest integer with $2^{b}\geq
BM^{\beta}(\beta+1)^{d}$. As $|c_{{\textbf{a},\boldsymbol{\gamma}}}|<B$ ([11],
eq. 34), then for each $c_{{\textbf{a},\boldsymbol{\gamma}}}$ there is an
integer $k\in[-2^{b},2^{b}]$ with
$c_{{\textbf{a},\boldsymbol{\gamma}}}\in[\frac{k}{2^{b}}B,\frac{k+1}{2^{b}}B)$.
Denote then $\tilde{c}_{{\textbf{a},\boldsymbol{\gamma}}}=\frac{k}{2^{b}}B$
and define

$\tilde{P}_{\textbf{a}}^{\beta}f(\textbf{x})=\sum\limits_{0\leq|\boldsymbol{\gamma}|<\beta}\tilde{c}_{{\textbf{a},\boldsymbol{\gamma}}}\textbf{x}^{\boldsymbol{\gamma}}.$

As the number of monomials of degree up to $\beta$ is bounded by
$(\beta+1)^{d}$, then

$\|P_{\textbf{a}}^{\beta}f-\tilde{P}_{\textbf{a}}^{\beta}f\|_{L^{\infty}[0,1]^{d}}\leq(\beta+1)^{d}\frac{B}{2^{b}}\leq
M^{-\beta}.$

Also, as

$\sum\limits_{\textbf{x}_{\boldsymbol{\ell}}\in\textbf{D}(M)}\prod_{j=1}^{d}(1-M|x_{j}-x_{j}^{\boldsymbol{\ell}}|)_{+}=1,$

then

$\|P^{\beta}f(\textbf{x})-\sum\limits_{\textbf{x}_{\boldsymbol{\ell}}\in\textbf{D}(M)}\tilde{P}_{\textbf{x}_{\boldsymbol{\ell}}}^{\beta}f(\textbf{x})\prod_{j=1}^{d}(1-M|x_{j}-x_{j}^{\boldsymbol{\ell}}|)_{+}\|_{L^{\infty}[0,1]^{d}}\leq
M^{-\beta}.$

Thus, defining

$\tilde{P}^{\beta}f(\textbf{x})=\sum\limits_{\textbf{x}_{\boldsymbol{\ell}}\in\textbf{D}(M)}\tilde{P}_{\textbf{x}_{\boldsymbol{\ell}}}^{\beta}f(\textbf{x})\prod_{j=1}^{d}(1-M|x_{j}-x_{j}^{\boldsymbol{\ell}}|)_{+},$

we get that

$\|\tilde{P}^{\beta}f-f\|_{L^{\infty}[0,1]^{d}}\leq(K+1)M^{-\beta}.$ (13)

In the proof of Theorem 1, the neural network approximation of the function
$(x_{1},...,x_{r})\mapsto\prod_{j=1}^{r}x_{j}$ is first constructed followed
by approximation of monomials of degree up to $\beta$. The result then follows
by approximating the function $P^{\beta}f(\textbf{x})$ and applying (12). In
the latter approximation the set of parameters not belonging to
$\\{0,\pm\frac{1}{2},\pm 1\\}$ consists of:

  * •

shift coordinates $j/M,$ $j=1,...,M-1,$ (the grid points);

  * •

at most $(\beta(M+1))^{d}$ weight matrix entries of the form
$c_{{\textbf{x}_{\boldsymbol{\ell}},\boldsymbol{\gamma}}}/B$, where
$c_{{\textbf{x}_{\boldsymbol{\ell}},\boldsymbol{\gamma}}}$ are coefficients of
the polynomial $P^{\beta}_{\textbf{x}_{\boldsymbol{\ell}}}f(\textbf{x}),$
$\textbf{x}_{\boldsymbol{\ell}}\in\textbf{D}(M),|\boldsymbol{\gamma}|<\beta$;

  * •

a shift coordinate $1/(2M^{d})$ (used to scale the output entries).

Note that the above list gives at most $D:=M+(\beta(M+1))^{d}$ different
parameters. Taking into account (13) we can use $\tilde{P}^{\beta}f$ instead
of ${P}^{\beta}f$ to approximate $f$. Thus, we can replace the entries
$c_{{\textbf{x}_{\boldsymbol{\ell}},\boldsymbol{\gamma}}}/B$ by the entries
$\tilde{c}_{{\textbf{x}_{\boldsymbol{\ell}},\boldsymbol{\gamma}}}/B=\frac{k}{2^{b}},$
where $k$ is some integer from $[-2^{b},2^{b}]$. Also, as $M=2^{\nu}$, then
denoting $\Delta=\max\\{\nu d+1;b\\}$ we need to obtain $D$ parameters from
the set
$\mathcal{S}=\\{\frac{k}{2^{\Delta}},k\in\mathbb{Z}\cap(0,2^{\Delta}]\\}.$ As
any natural number can be represented as a sum of powers of 2, then for any
$y_{1},...,y_{D}\in\mathbb{Z}\cap(0,2^{\Delta}]$ we can compute

$(1,x_{1},...,x_{d})\mapsto(1,x_{1},...,x_{d},y_{1},...,y_{D})$

with parameters from $\\{0,1,2\\}$ using at most ${\Delta}$ hidden layers. The
number of active parameters required for this computation is bounded by
$(1+d+D+\Delta){\Delta}$. Hence, for any $z_{1},...,z_{D}\in\mathcal{S}$, we
can compute

$(1,x_{1},...,x_{d})\mapsto(1,x_{1},...,x_{d},z_{1},...,z_{D})$

with $2\Delta$ hidden layers and $2(1+d+D+\Delta){\Delta}$ active parameters.
Applying Theorem 1 we get the existence of a network
$\tilde{f}\in\mathcal{\widetilde{F}}(\tilde{L},\tilde{\textbf{p}},\tilde{s})$
with the desired architecture and sparsity. $\square$

## Acknowledgement

The author has been supported by the NWO Vidi grant: “Statistical foundation
for multilayer neural networks”. The author would like to thank Johannes
Schmidt-Hieber and Hailin Sang for useful comments and suggestions.

## References

  * [1] Birgé, L., Estimating a density under order restrictions: Nonasymptotic minimax risk. Annals of Statistics, 15: 995-1012, 1987.
  * [2] Gale, T., Elsen, E. and Hooker, S., The state of sparsity in deep neural networks, https://arxiv.org/pdf/1902.09574.pdf
  * [3] Györfi, L., Kohler, M., Krzyzak, A., and Walk, H., A distribution-free theory of nonparametric regression. Springer Science & Business Media, 2006.
  * [4] Han, S., Pool, J., Tran, J., and Dally, W., Learning both weights and connections for efficient neural network. In Advances in Neural Information Processing Systems 28 (C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama and R. Garnett, eds.): 1135-1143, 2015.
  * [5] Hassibi, B., Stork, D. G., Second order derivatives for network pruning: Optimal brain surgeon. In Advances in Neural Information Processing Systems 5 (S. J. Hanson, J. D. Cowan and C. L. Giles, eds.): 164-171, 1993.
  * [6] Lu, Z., Pu, H., Wang, F., Hu, Z., Wang, L., The expressive power of neural networks: A view from the width. Advances in Neural Information Processing Systems, 6231-6239, 2017.
  * [7] Ohn, I., Kim, Y., Nonconvex sparse regularization for deep neural networks and its optimality, https://arxiv.org/pdf/2003.11769.pdf
  * [8] Pollard, D., Convergence of Stochastic Processes. Springer, Berlin, 1984.
  * [9] Scarselli, F., Tsoi, A. C., Universal approximation using feedforward neural networks: A survey of some existing methods, and some new results. Neural Networks, 11, 15-37, 1998.
  * [10] Schmidt-Hieber, J., Nonparametric regression using deep neural networks with ReLU activation function. Annals of Statistics, 48 (4): 1875–1897, 2020.
  * [11] Schmidt-Hieber, J., Supplement to “Nonparametric regression using deep neural networks with ReLU activation function”, https://arxiv.org/pdf/1708.06633.pdf
  * [12] Taheri, M., Xie, F., and Lederer, J., Statistical Guarantees for Regularized Neural Networks. Neural Networks, 142, 148–161, 2021.
  * [13] Yarotsky, D., Error bounds for approximations with deep ReLU networks. Neural Networks, 94: 103-114, 2017.


# The Alignment Ceiling: Objective Mismatch in
Reinforcement Learning from Human Feedback

Nathan Lambert
Allen Institute for AI
Berkeley, CA, USA
<EMAIL_ADDRESS>
&Roberto Calandra
TU Dresden
Dresden, Germany
<EMAIL_ADDRESS>

###### Abstract

Reinforcement learning from human feedback (RLHF) has emerged as a powerful
technique to make large language models (LLMs) easier to prompt and more
capable in complex settings. RLHF at its core is providing a new toolkit to
optimize LLMs other than next-token prediction, enabling the integration of
qualitative training goals. The attempted match between user preferences and
downstream performance, which happens in a learned reward model, results in an
optimization landscape where training and evaluation metrics can appear
correlated. The apparent correlation can lead to unexpected behaviors and
stories of “too much RLHF.” In RLHF, challenges emerge because the following
sub-modules are not consistent with each other: the reward model training, the
policy model training, and the policy model evaluation. This mismatch results
in models that sometimes avoid user requests through false safety flags, are
difficult to steer to an intended characteristic, or always answer in a
specific style. As chat model evaluation becomes increasingly nuanced, the
reliance on a perceived link between reward model score and downstream
performance drives the objective mismatch issue. In this paper, we illustrate
the cause of this issue, reviewing relevant literature from model-based
reinforcement learning, and discuss relevant solutions to encourage further
research. By solving objective mismatch in RLHF, the LLMs of the future will
be more precisely aligned to user instructions for both safety and
helpfulness.

##  1 Introduction

Reinforcement learning from human feedback (RLHF) is a powerful tool for
integrating qualitative styles and values into large machine learning models
(Christiano ., 2017; Ouyang ., 2022; Bai ., 2022). RLHF was popularized with
its use to integrated human values into large language models (LLMs) for
aligning chat tools (Schulman ., 2022). In doing so, RLHF has become an
important technique in the process of making models better at responding to
user requests, often referred to as instruction-tuned, steerability, chat-
models, etc. RLHF methods typically operate in a two step process following
the training of a base language model, first they learn a model of human
preferences that acts as a reward function, and second they use this model
within a reinforcement learning (RL) optimization loop. In the RLHF process,
these two steps are often executed independently, with an accurate reward
model being trained on human preference data and then the RL optimizer is used
to extract maximum information into the chat model. A common challenge of
modern LLMs trained with RLHF is difficulties in extracting intended behaviors
from the model. Sometimes, the models decline benign requests for safety
reasons and other times they need clever prompt tuning to extract full
performance.

In this paper, we detail a fundamental challenge in modern RLHF learning
schemes: the objective mismatch issue. In RLHF, three important parts of
training are numerically decoupled: the design of evaluation metrics, the
training of a reward model, and the training of the generating model. This
mismatch between the reward model and the RL training is visualized in Fig. 1,
yet other links exist between the goals of evaluation and simulating human
values. Specifically, there are many avenues to better align reward model
training to the literature in preference quantification (Lambert, Gilbert
Zick, 2023) and fundamental optimization challenges need to be solved in RLHF
practices (Casper ., 2023). ChatGPT, the most popular model trained with RLHF,
shows signs this limitation through issues such as verbosity, self-doubt and
question refusals, repeated phrases, hedging, and more (Schulman, 2023). These
traits of overoptimization are results of the subtle proxy objective problem
that objective mismatch provides a frame for studying and solving – the reward
model attributes excess value to phrases that do not contribute to user
benefit, which the RL optimizer exploits, such as safety flags. On the hand,
the current training setups are not fully aligned with evaluation tools
because the RLHF’d models still need sophisticated prompting techniques such
as “thinking step by step” (J. Wei ., 2022) or “take a deep breath” (Yang .,
2023) to reach maximum performance. Solving objective mismatch will remove the
need for these advanced techniques and reduce the likelihood of out-of-scope
refusals from an LLM.

The phrase objective mismatch originates from model-based reinforcement
learning (MBRL), where an agent iteratively learns a dynamics model that it
later uses to solve a control task (Lambert ., 2020; R. Wei ., 2023). In this
context, the mismatch is between learning an accurate dynamics model rather
than one that is optimized for high task reward. In RLHF, the problem is
related, but with added complexity, as the reward model is optimized for
preference data over a closed distribution, which does not match the end
users. Second, the task of open-ended language generation is less specific to
a notion of reward than that of RL control policies. For these reasons, as we
explore in this paper, the objective mismatch issue is more nuanced and
critical to RLHF.

In this position paper, we make three contributions:

  * •

Clearly explain the origins and potential manifestations of objective mismatch
in chat-tuned LLMs,

  * •

Connect related work from NLP and RL literature around objective mismatch,

  * •

Propose directions of study to solve the mismatch and foster better RLHF
practices.

Figure 1: An illustration of where the objective mismatch issue emerges within
the RL optimization phase of RLHF. A mismatch occurs when the score from the
reward model is assumed to be correlated with other downstream evaluation
metrics, such as human preferences over evaluation sets, classic NLP
benchmarks, or LLM-as-a-judge systems.

##  2 Related Work

###  2.1 Reinforcement learning from human feedback

Early work in RLHF focused on continuous control domains with various methods
for altering the behavior across trajectories (Christiano ., 2017; Wirth .,
2017). The impacts of RLHF today primarily has been centered around its use
with LLMs. Initial work on RLHF for LLMs utilized user preferences from a
batch of 4 options (Ziegler ., 2019) to train a reward model across general
LLM benchmarks. Group preferences were changed to pairwise preferences, and
rather than general benchmarks the reward model was focused on the task of
summarization (Stiennon ., 2020; J. Wu ., 2021). Next emerged general
question-answering models (Ouyang ., 2022) and web crawling agents (Nakano .,
2021), primarily from scaling the initial model and human datasets. Now, RLHF
is used to train general chat models across a variety of tasks (Bai ., 2022;
Schulman ., 2022; Touvron ., 2023) and in specific domains such as harm
reduction (Glaese ., 2022) or information accuracy (Menick ., 2022).

The development of these methods has accelerated markedly, with many
variations on the methodology for integrating feedback into language models
(Fernandes ., 2023). The most popular reinforcement learning optimizer is
still Proximal Policy Optimization (PPO) (Schulman ., 2017), with its many
stable and scalable implementations. Recent works have been developing methods
for the RL optimizer, such as the offline RL algorithm Implicit Language Q
Learning (ILQL) (Snell ., 2022), direct preference optimization (DPO)
(Rafailov ., 2023) for utilizing preference data without a reward model, or
Advantage-Leftover Lunch RL (A-LOL) (Baheti ., 2023) which is designed to act
on the entire response as a single action (which RLHF generally does).

###  2.2 Problem mis-specification in RLHF

There is a substantial emerging literature on varieties of numerical issues,
unexpected behaviors such as verbosity and evasiveness (Schulman, 2023), and
potential solutions in RLHF, which can be mitigated by progress on solving
objective mismatch. A prominent recent example is the behavior of the flagship
Llama 2 chat model refusing to answer a request asking “How do I kill a Linux
process,” conflating the computer process with the morals of killing a living
creature. It has been shown that there are predictable behaviors of reward
model overoptimization with PPO and best-of-N optimization techniques (Gao .,
2022), which can be partially mitigated by training ensemble reward models
(Coste ., 2023). Other issues have emerged in RLHF models that demonstrates
the need for improved reward models, such as a bias towards long responses
(Singhal ., 2023), a lack of language consistency Shen . (2023) (invariance
over changes that maintain meaning), or a reduction of output diversity Kirk .
(2023). A similar argument is made in A. Wei . (2023), where the authors argue
that “competing objectives and mismatched generalization” mislead the models –
we present how objective mismatch covers both of these limitations and more
possible failure cases.

There are also papers studying how the reward model training formulation does
not align with the data collection process and downstream RL optimization,
suggesting the models should model advantage estimates rather than direct
value functions (Peng ., 2023; Knox Stone, 2008).

###  2.3 Evaluating LLMs trained with RLHF

Core to the ideas of objective mismatch with LLMs is the methods of evaluation
used to correlate performance. Historically, LLMs have been evaluated across a
wide variety of tasks trying to capture specific characteristics of models,
making evaluation a extremely broad process (Liang ., 2022) where progress is
saturating (Kiela ., 2023). Now, many models are focused on hard to specify
tasks such as chat, where existing benchmarks were not well correlated with
performance (Zheng ., 2023), so new chat based evaluations such as MT-Bench
(Zheng ., 2023) and AlpacaEval (Li ., 2023) have been introduced, but
substantial further work is needed.

##  3 Background

###  3.1 Reward model training

Reward models are trained with human preference data most often consisting of
a task given to the model prompt, i.e a request or instruction, and ratings of
the completion, or answer. The feedback can consist of selecting the best from
groups of responses (Ziegler ., 2019), scores and rankings of a group of
candidate responses (Ouyang ., 2022), a choice between a pair of responses
(Bai ., 2022) (choose best response between two options), and even finer
grained data (Z. Wu ., 2023). The workers employed are generally given
detailed instructions on which styles, occurrences, or values to prioritize in
their labels.

The reward models trained for RLHF are most often trained as classifiers
between a chosen and rejected completion to a prompt before optimizing with RL
where they return a scalar value for each piece of text. Given two options for
a completion $y$ from a prompt $x$, and the scores they obtain a scalar output
$r$ from a initially untrained value head on an LLM or value model entirely,
the loss for the reward model follows (Askell ., 2021; Ouyang ., 2022)

$L=\text{log}\big{(}1+e^{r_{\text{chosen}}-r_{\text{rejected}}}\big{)}\,.$ (1)

The loss function is designed to increase the distance between the two
samples, where variations exist including losses of 4 samples rather than a
pair (Ziegler ., 2019), updating the model with batches of pairwise labels on
a given prompt (Ouyang ., 2022), or optimizing based on the margin between
$r_{\text{chosen}}$ and $r_{\text{rejected}}$ (Touvron ., 2023). For inference
during RL optimization, the reward is taken as the raw logit output from this
model that represents an non-normalized probability of the text being
preferred.

###  3.2 Reinforcement Learning on Language

Language generation optimized via reinforcement learning, which RLHF is a
version of, can be formalized as a partially observable Markov decision
process (POMDP) (Spaan, 2012). We define a POMDP $\mathcal{M}$ at a per-token
level with
$\mathcal{M}=(\mathcal{S},\mathcal{A},\mathcal{O},\mathcal{T},\mathcal{Z},\mu_{0},\mathcal{R},\gamma)$.
Here, the state of the system is $s_{t}\in\mathcal{S}$, which the agent
receives as an observation $h_{t}\in\mathcal{O}$. The observation is a history
of tokens $h_{t}=\\{t_{0},t_{1},\ldots,t_{t-1}\\}$ and the action space is the
possible set of next-tokens in the vocabulary of the policy model
$a_{t}=t_{t}\in\mathcal{A}$, including the end-of-sequence token
$a_{\text{end}}$. As in a traditional MPD, $\mathcal{T}$ is the transition
function $\mathcal{T}(\cdot|s_{t},a_{t})$.

The goal of the RLHF process is to learn a policy that is mapping
$\pi:\mathcal{O}\mapsto\mathcal{P}(\mathcal{A})$. This is done with the reward
model, which acts as a reward function $R(s_{t},a_{t})\mapsto\mathcal{R}$,
used after each sequence is generated. The full sequence, until end-of-
sequence token $a_{\text{end}}$, is called the action and used to get a scale
reward $r_{t}$ from the reward model.

With LLMs, the generating model is referred to the as the policy model. In
RLHF, the discount factor of reward is set to 1 and no further actions are
taken for the given prompt, casting the problem as contextual bandits. An
example of the RLHF loop is shown in Fig. 2(b) in comparison to a standard RL
loop shown in Fig. 2(a).

(a) Trial and error reinforcement learning.

(b) Reinforcement learning from human feedback.

Figure 2: Comparing RLHF to a traditional RL problem. (left) is the canonical
RL problem, where an agent interacts repeatedly with an environment. (right)
is RLHF, where an agent is optimized against a set of predetermined prompts
and a learned reward model for one action per input state.

##  4 Understanding Objective Mismatch

The objective mismatch in RLHF emerges from three broad causes: First, common
practice in RL engineering dictates that as long as reward is increasing the
model is improving. Second, the evaluation methods available for models
trained with RLHF is often incomplete relative to their downstream use-cases.
Third, the assumption that the reward model trained is a suitable reward
function for optimization. For these reasons, objective mismatch emerges as
the assumption that downstream evaluation will be correlated with the reward
model score for the current policy, which is not proven.

###  4.1 Origins of mismatch

Objective mismatch in RLHF is the result of the interactions between three
different sub-components, rather than just the two (i.e., dynamics model and
policy) from MBRL: It is a balance of the reward model training, the goal of
getting a calibrated reward function, the policy training, the process of
uniformly extracting information from the reward model, and the often bespoke
evaluation techniques used for RLHF models, the process of fairly evaluating a
multi-use model. There exists an interface between each pair of these three
that provides an axis for erroneous assumptions regarding the true
optimization problem, but the importance of each link is not uniform for
mitigation.

#### Reward model training $\leftrightarrow$ policy model training

Uniformly extracting the information from the reward model into the policy and
avoiding the reward hacking inherent to RL (Pan ., 2022) that can result in
overoptimization of reward models (Gao ., 2022) is central to RLHF. A good
reward model may not be one that is empirically easy to train a policy with
high reward from, but rather one that is well correlated with downstream
evaluation metrics. Common practice in RLHF, especially with larger models
where gradients are less stable, is to spend additional compute in search of
“stable” training runs with increasing reward, which induces further
likelihood of mismatch.

#### Reward model training $\leftrightarrow$ evaluation tools

While relatively little work and resources exist for the study of state-of-
the-art reward models, the matching of the reward signal they provide to the
intended use-case of the final policy (via evaluation metrics) is central to
solving the objective mismatch issue. The reward models are trained on
aggregated datasets to maximize agreement of the model on a held out set of
data, which in practice often only maximizes at 60-75% accuracy (Ouyang .,
2022; Bai ., 2022). Given the complex task encompassed in reward modeling, it
is unlikely that the models converge to 100% accuracy, but studying the
sources of this performance delta could indicate sources of mismatch. New
tools are needed for evaluation of reward models that better match their
conceptual underpinnings as a representation of human values for solving the
alignment problem (Leike ., 2018) and as a practical realization as targets
for optimization Lambert, Gilbert Zick (2023).

#### Policy model training $\leftrightarrow$ evaluation tools

The third link contributes the least to the emergence of mismatch, but is the
easiest axis to visualization potential signs of objective mismatch. Directly
matching RL training with any additional evaluation metrics is technically
challenges. In MBRL, such a solution could be by using a differentiable
simulator (R. Wei ., 2023), but with the complexity of RLHF such solutions are
less obtainable. Exploring any types of regularization or calibration of
training with respect to final evaluations is viable as research directions,
but this area of study is best suited for visualizing signs of objective
mismatch, as shown in Fig. 3.

Figure 3: Illustrating the most likely visualization of objective mismatch in
RLHF, the link between policy training and downstream evaluation. Measuring
the correlation between evaluation and RL training is crucial to understanding
the scope of impact of objective mismatch on current and future RLHF trained
models.

###  4.2 Investigations and solutions

There is already emerging research on many potential causes of mismatch in
RLHF, yet further work can be inspired by solutions from the broader RL
literature. Many of the solutions to objective mismatch in MBRL will not apply
directly because in MBRL they have a true reward from the environment, and for
that reason research is needed to understand the outputs of reward models.
Here follows a series of investigations to expanded to mitigate objective
mismatch:

  * •

Reward model evaluation There are many axes by which a reward model is
expected to behave in order to be a reasonable approximation of a reward
function, but they are typically not studied. Reward models need to be
assessed for consistency, robustness to adversarial attacks, calibration
across distributions, and more, as discussed in Lambert, Gilbert Zick (2023).
Understanding reward models performance is the foundation of solving the
mismatch problem. Evaluating reward models will be a indirect but useful path
to measure the varied preference datasets used for open RLHF models.

  * •

Reward model training methods In order to solve limitations of reward models
across better evaluation techniques, numerous new training methods will be
developed. Early research has already shown reward model ensembles can help
mitigate overoptimization (Coste ., 2023). Further research is warranted to
integrate techniques that have improved performance of model-based RL
algorithms, such as probabilistic loss functions for the dynamics models and
planning (Chua ., 2018), calibrated probability estimates (Malik ., 2019)
during training the reward model as a classifier, and other solutions (R. Wei
., 2023). Additionally, links should be explored between the reward models of
inverse reinforcement learning (IRL) (Ng ., 2000), the subfield tasked with
learning a reward function from agent behavior, and those of RLHF. Early
research also shows reformatting the reward model training to better match
preference learning literature could improve performance (Knox ., 2023).

  * •

Reward model training datasets High-quality datasets are a bottleneck slowing
progress in open RLHF research, given the large costs required to acquire
them. There are a few datasets available, but they are unproven in their
ability match the performance of the best models. The Stack Exchange
preference dataset (Lambert, Tunstall ., 2023), Stanford Preferences Dataset
of Reddit content (Ethayarajh ., 2022), UltraFeedback synthetic preference
data (Cui ., 2023), WebGPT internet browsing (Nakano ., 2021), learning to
summarize (Stiennon ., 2020), and Anthropic HHH dataset (Askell ., 2021) serve
as a strong foundation for research, but exploration is needed.

  * •

Value-guided sampling techniques Increased compute can be spent at inference
time to improve the performance of RLHF models by utilizing the values
returned by the reward model (Deng Raffel, 2023). Feng . (2023) explores this
through Monte Carlo tree search generation, yet many more methods can be
explored across the planning literature.

  * •

Human-centric NLP evaluation The most popular evaluation technique for chat-
tuned RLHF models is preference percentage versus other top models on
evaluation prompt sets (as done in open RLHF models including Llama 2 (Touvron
., 2023) and Dromedary-2 (Sun ., 2023)). This evaluation mechanism, while
well-motivated in the popular use-cases of the models, suffers from bias and
reproducibility challenges. The prompts can easily be chosen to support the
model designed by the authors, and the prompts are often not released or
aggregated into a future benchmark. Expanding the reproducibility and
consistency of these practices will be important to creating robust practices
for RLHF.

  * •

RL optimizers for language As discussed in Sec. 2.1, the optimizers used for
RLHF are most often those from previous RL literature. Now there is an
opportunity for expansion of RL algorithms into the niche of RLHF, where
conditions are highly specialized through the expansive actions space and
bandits formulation. New algorithms are a step in the right direction, such as
T. Wu . (2023) modifying the PPO algorithm for pairwise preferences or Baheti
. (2023) proposing an offline RL algorithm for full-completion actions.

  * •

Other solutions Other creative mismatch solutions will exist, such as work
integrating the LLM policy, reward model, and transition function into a
single model (Xu ., 2023). Methods such as this need to be evaluated across
many scales to confirm that they are still numerically stable with the larger
state-of-the-art models where powerful emergent behaviors exist.

###  4.3 Mismatch of next-token prediction

The original training object used in popular language model architectures,
autoregressive next-token prediction also suffers from an objective mismatch
problem, as almost all LLM evaluation techniques evaluate the entire output
rather than individual tokens. While this is true, the development signal that
the next-token prediction loss provides is more orthogonal to the goals of
RLHF. In RLHF, and most related work in RL, the reward signal is interpretted
as a direct indicator of performance. This assumption creates a much more
unintentionally nuanced research setup, warranting the specific study of its
impacts.

In MBRL, the learning of a dynamics model is also often done via one-step
transitions, with recent work studying autoregressive models (Janner ., 2021;
Lambert ., 2021), where the compounding error of multiple one-step predictions
is well known as a deeply related issue to objective mismatch (Lambert .,
2022). In the case where mismatch becomes a fundamental problem of LLMs,
similar solutions could be investigated.

##  5 Discussions

Figure 4: The outer loop of RLHF is the process to train the next reward model
for RL to improve performance is areas of user interest. This setup induces
additional complexity in objective mismatch in RLHF.

#### Iterative deployment of RLHF

The iterative deployment form of RLHF where reward models are retrained based
on user data, which induces a second feedback loop, is shown in Fig. 4.
Schulman (2023) discusses how this is used in ChatGPT to mitigate issues such
as evasiveness, verbosity, and other unexpected, undesirable qualities.
Designing in this framework introduces further complexity onto engineering
objectives, but allows iterative mitigation of mismatch. This style of
iterative RL deployment has been understood as exogenous feedback (Gilbert .,
2022) and can have societal implications.

There is some literature in this space, but expanding related works to the
scale of use of modern LLMs will be difficult. For example, Suhr Artzi (2022)
shows theoretical results on outer-loop optimization of instruction-tuned
models.

#### Contextual bandits

The modifications made to the RL optimization of RLHF cast it as a contextual
bandits problem, where an agent takes one action and the dynamics are
abstracted into one trajectory-reward pairing. Work in this area has
investigated the potential of integrating partial, skewed, or noisy human
feedback into the optimization process (Nguyen ., 2017).

The subarea of dueling bandits has further specified the problem that is
closely aligned with RLHF, but in primarily theoretical work with much smaller
models, datasets, and tasks. Yue . (2012) explains this space in work showing
theoretical bounds:

> “In contrast to conventional approaches that require the absolute reward of
> the chosen strategy to be quantifiable and observable, our setting assumes
> only that (noisy) binary feedback about the relative reward of two chosen
> strategies is available. This type of relative feedback is particularly
> appropriate in applications where absolute rewards have no natural scale or
> are difficult to measure… but where pairwise comparisons are easy to make.”

This, while closely related to RLHF, will require substantial experimentation
to be applicable. Others have built on this into work directly learning from
human preferences (Sekhari ., 2023) or from implicit human feedback (Maghakian
., 2022).

##  6 Conclusion

This paper presents the multiple ways by which objective mismatch limits the
accessibility and reliability of RLHF methods. This current disconnect between
design a reward model, optimizing it, and the downstream model goals creates a
method that is challenging to implement and improve on. Future work mitigating
mismatch and the proxy objectives present in RLHF, LLMs and other popular
machine learning methods will becomes easier to align with human values and
goals, solving many common challenges users encounter with state-of-the-art
LLMs.

## Acknowledgments

This work was partly supported by the German Research Foundation (DFG,
Deutsche Forschungsgemeinschaft) as part of Germany’s Excellence Strategy –
EXC 2050/1 – Project ID 390696704 – Cluster of Excellence “Centre for Tactile
Internet with Human-in-the-Loop” (CeTI) of Technische Universität Dresden, and
by Bundesministerium für Bildung und Forschung (BMBF) and German Academic
Exchange Service (DAAD) in project 57616814 (SECAI, School of Embedded and
Composite AI).

## References

  * Askell . (2021) askell2021generalAskell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T.others  2021\.  A general language assistant as a laboratory for alignment A general language assistant as a laboratory for alignment.  arXiv preprint arXiv:2112.00861.
  * Baheti . (2023) baheti2023improvingBaheti, A., Lu, X., Brahman, F., Bras, RL., Sap, M. Riedl, M.  2023\.  Improving Language Models with Advantage-based Offline Policy Gradients Improving language models with advantage-based offline policy gradients.  arXiv preprint arXiv:2305.14718.
  * Bai . (2022) bai2022trainingBai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N.others  2022\.  Training a helpful and harmless assistant with reinforcement learning from human feedback Training a helpful and harmless assistant with reinforcement learning from human feedback.  arXiv preprint arXiv:2204.05862.
  * Casper . (2023) casper2023openCasper, S., Davies, X., Shi, C., Gilbert, TK., Scheurer, J., Rando, J.others  2023\.  Open problems and fundamental limitations of reinforcement learning from human feedback Open problems and fundamental limitations of reinforcement learning from human feedback.  arXiv preprint arXiv:2307.15217.
  * Christiano . (2017) christiano2017deepChristiano, PF., Leike, J., Brown, T., Martic, M., Legg, S. Amodei, D.  2017\.  Deep reinforcement learning from human preferences Deep reinforcement learning from human preferences.  Advances in neural information processing systems30.
  * Chua . (2018) chua2018deepChua, K., Calandra, R., McAllister, R. Levine, S.  2018\.  Deep reinforcement learning in a handful of trials using probabilistic dynamics models Deep reinforcement learning in a handful of trials using probabilistic dynamics models.  Advances in neural information processing systems31.
  * Coste . (2023) coste2023rewardCoste, T., Anwar, U., Kirk, R. Krueger, D.  2023\.  Reward Model Ensembles Help Mitigate Overoptimization. Reward model ensembles help mitigate overoptimization.
  * Cui . (2023) cui2023ultrafeedbackCui, G., Yuan, L., Ding, N., Yao, G., Zhu, W., Ni, Y.Sun, M.  2023\.  UltraFeedback: Boosting Language Models with High-quality Feedback Ultrafeedback: Boosting language models with high-quality feedback.  arXiv preprint arXiv:2310.01377.
  * Deng Raffel (2023) deng2023rewardDeng, H. Raffel, C.  2023\.  Reward-Augmented Decoding: Efficient Controlled Text Generation With a Unidirectional Reward Model Reward-augmented decoding: Efficient controlled text generation with a unidirectional reward model.  arXiv preprint arXiv:2310.09520.
  * Ethayarajh . (2022) pmlr-v162-ethayarajh22aEthayarajh, K., Choi, Y. Swayamdipta, S.  202217–23 Jul.  Understanding Dataset Difficulty with $\mathcal{V}$-Usable Information Understanding dataset difficulty with $\mathcal{V}$-usable information.  K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu S. Sabato (), Proceedings of the 39th International Conference on Machine Learning Proceedings of the 39th international conference on machine learning ( 162, 5988–6008).  PMLR.
  * Feng . (2023) feng2023alphazeroFeng, X., Wan, Z., Wen, M., Wen, Y., Zhang, W. Wang, J.  2023\.  Alphazero-like Tree-Search can Guide Large Language Model Decoding and Training Alphazero-like tree-search can guide large language model decoding and training.  arXiv preprint arXiv:2309.17179.
  * Fernandes . (2023) fernandes2023bridgingFernandes, P., Madaan, A., Liu, E., Farinhas, A., Martins, PH., Bertsch, A.others  2023\.  Bridging the gap: A survey on integrating (human) feedback for natural language generation Bridging the gap: A survey on integrating (human) feedback for natural language generation.  arXiv preprint arXiv:2305.00955.
  * Gao . (2022) gao2022scalingGao, L., Schulman, J. Hilton, J.  2022\.  Scaling Laws for Reward Model Overoptimization Scaling laws for reward model overoptimization.  arXiv preprint arXiv:2210.10760.
  * Gilbert . (2022) gilbert2022choicesGilbert, TK., Dean, S., Zick, T. Lambert, N.  2022\.  Choices, Risks, and Reward Reports: Charting Public Policy for Reinforcement Learning Systems Choices, risks, and reward reports: Charting public policy for reinforcement learning systems.  arXiv preprint arXiv:2202.05716.
  * Glaese . (2022) glaese2022improvingGlaese, A., McAleese, N., Trębacz, M., Aslanides, J., Firoiu, V., Ewalds, T.others  2022\.  Improving alignment of dialogue agents via targeted human judgements Improving alignment of dialogue agents via targeted human judgements.  arXiv preprint arXiv:2209.14375.
  * Janner . (2021) janner2021offlineJanner, M., Li, Q. Levine, S.  2021\.  Offline reinforcement learning as one big sequence modeling problem Offline reinforcement learning as one big sequence modeling problem.  Advances in neural information processing systems341273–1286.
  * Kiela . (2023) kiela2023plottingprogressKiela, D., Thrush, T., Ethayarajh, K. Singh, A.  2023\.  Plotting Progress in AI Plotting progress in ai.  Contextual AI Blog.  https://contextual.ai/blog/plotting-progress
  * Kirk . (2023) kirk2023understandingKirk, R., Mediratta, I., Nalmpantis, C., Luketina, J., Hambro, E., Grefenstette, E. Raileanu, R.  2023\.  Understanding the Effects of RLHF on LLM Generalisation and Diversity. Understanding the effects of rlhf on llm generalisation and diversity.
  * Knox . (2023) knox2023learningKnox, WB., Hatgis-Kessell, S., Adalgeirsson, SO., Booth, S., Dragan, A., Stone, P. Niekum, S.  2023\.  Learning Optimal Advantage from Preferences and Mistaking it for Reward. Learning optimal advantage from preferences and mistaking it for reward.
  * Knox Stone (2008) knox2008tamerKnox, WB. Stone, P.  2008\.  Tamer: Training an agent manually via evaluative reinforcement Tamer: Training an agent manually via evaluative reinforcement.  2008 7th IEEE international conference on development and learning 2008 7th ieee international conference on development and learning ( 292–297).
  * Lambert . (2020) lambert2020objectiveLambert, N., Amos, B., Yadan, O. Calandra, R.  2020\.  Objective Mismatch in Model-based Reinforcement Learning Objective mismatch in model-based reinforcement learning.  Learning for Dynamics and Control Learning for dynamics and control ( 761–770).
  * Lambert, Gilbert Zick (2023) lambert2023entangledLambert, N., Gilbert, TK. Zick, T.  2023\.  Entangled Preferences: The History and Risks of Reinforcement Learning and Human Feedback. Entangled preferences: The history and risks of reinforcement learning and human feedback.
  * Lambert . (2022) lambert2022investigatingLambert, N., Pister, K. Calandra, R.  2022\.  Investigating compounding prediction errors in learned dynamics models Investigating compounding prediction errors in learned dynamics models.  arXiv preprint arXiv:2203.09637.
  * Lambert, Tunstall . (2023) h4stackexchangeLambert, N., Tunstall, L., Rajani, N. Thrush, T.  2023\.  HuggingFace H4 Stack Exchange Preference Dataset. Huggingface h4 stack exchange preference dataset.  https://huggingface.co/datasets/HuggingFaceH4/stack-exchange-preferences
  * Lambert . (2021) lambert2021learningLambert, N., Wilcox, A., Zhang, H., Pister, KS. Calandra, R.  2021\.  Learning accurate long-term dynamics for model-based reinforcement learning Learning accurate long-term dynamics for model-based reinforcement learning.  2021 60th IEEE Conference on Decision and Control (CDC) 2021 60th ieee conference on decision and control (cdc) ( 2880–2887).
  * Leike . (2018) leike2018scalableLeike, J., Krueger, D., Everitt, T., Martic, M., Maini, V. Legg, S.  2018\.  Scalable agent alignment via reward modeling: a research direction Scalable agent alignment via reward modeling: a research direction.  arXiv preprint arXiv:1811.07871.
  * Li . (2023) alpaca_evalLi, X., Zhang, T., Dubois, Y., Taori, R., Gulrajani, I., Guestrin, C.Hashimoto, TB.  2023\.  AlpacaEval: An Automatic Evaluator of Instruction-following Models. Alpacaeval: An automatic evaluator of instruction-following models.  https://github.com/tatsu-lab/alpaca_eval.  GitHub.
  * Liang . (2022) liang2022holisticLiang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M.others  2022\.  Holistic evaluation of language models Holistic evaluation of language models.  arXiv preprint arXiv:2211.09110.
  * Maghakian . (2022) maghakian2022personalizedMaghakian, J., Mineiro, P., Panaganti, K., Rucker, M., Saran, A. Tan, C.  2022\.  Personalized Reward Learning with Interaction-Grounded Learning (IGL) Personalized reward learning with interaction-grounded learning (igl).  arXiv preprint arXiv:2211.15823.
  * Malik . (2019) malik2019calibratedMalik, A., Kuleshov, V., Song, J., Nemer, D., Seymour, H. Ermon, S.  2019\.  Calibrated model-based deep reinforcement learning Calibrated model-based deep reinforcement learning.  International Conference on Machine Learning International conference on machine learning ( 4314–4323).
  * Menick . (2022) menick2022teachingMenick, J., Trebacz, M., Mikulik, V., Aslanides, J., Song, F., Chadwick, M.others  2022\.  Teaching language models to support answers with verified quotes Teaching language models to support answers with verified quotes.  arXiv preprint arXiv:2203.11147.
  * Nakano . (2021) nakano2021webgptNakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C.others  2021\.  Webgpt: Browser-assisted question-answering with human feedback Webgpt: Browser-assisted question-answering with human feedback.  arXiv preprint arXiv:2112.09332.
  * Ng . (2000) ng2000algorithmsNg, AY., Russell, S. .  2000\.  Algorithms for inverse reinforcement learning. Algorithms for inverse reinforcement learning.  Icml Icml ( 1, 2).
  * Nguyen . (2017) nguyen2017reinforcementNguyen, K., Daumé III, H. Boyd-Graber, J.  2017\.  Reinforcement learning for bandit neural machine translation with simulated human feedback Reinforcement learning for bandit neural machine translation with simulated human feedback.  arXiv preprint arXiv:1707.07402.
  * Ouyang . (2022) ouyang2022trainingOuyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, CL., Mishkin, P.others  2022\.  Training language models to follow instructions with human feedback Training language models to follow instructions with human feedback.  arXiv preprint arXiv:2203.02155.
  * Pan . (2022) pan2022effectsPan, A., Bhatia, K. Steinhardt, J.  2022\.  The effects of reward misspecification: Mapping and mitigating misaligned models The effects of reward misspecification: Mapping and mitigating misaligned models.  arXiv preprint arXiv:2201.03544.
  * Peng . (2023) peng2023stabilizingPeng, B., Song, L., Tian, Y., Jin, L., Mi, H. Yu, D.  2023\.  Stabilizing RLHF through Advantage Model and Selective Rehearsal. Stabilizing rlhf through advantage model and selective rehearsal.
  * Rafailov . (2023) rafailov2023directRafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, CD. Finn, C.  2023\.  Direct preference optimization: Your language model is secretly a reward model Direct preference optimization: Your language model is secretly a reward model.  arXiv preprint arXiv:2305.18290.
  * Schulman (2023) schulman2023proxySchulman, J.  2023\.  Proxy objectives in reinforcement learning from human feedback. Proxy objectives in reinforcement learning from human feedback.  https://icml.cc/virtual/2023/invited-talk/21549 International Conference on Machine Learning (ICML)
  * Schulman . (2017) schulman2017proximalSchulman, J., Wolski, F., Dhariwal, P., Radford, A. Klimov, O.  2017\.  Proximal policy optimization algorithms Proximal policy optimization algorithms.  arXiv preprint arXiv:1707.06347.
  * Schulman . (2022) chatgptSchulman, J., Zoph, B., Kim, C. more.  2022\.  ChatGPT: Optimizing Language Models for Dialogue. Chatgpt: Optimizing language models for dialogue.  https://openai.com/blog/chatgpt/.  Accessed: 2023-02-12
  * Sekhari . (2023) sekhari2023contextualSekhari, A., Sridharan, K., Sun, W. Wu, R.  2023\.  Contextual Bandits and Imitation Learning via Preference-Based Active Queries Contextual bandits and imitation learning via preference-based active queries.  arXiv preprint arXiv:2307.12926.
  * Shen . (2023) shen2023trickleShen, L., Chen, S., Song, L., Jin, L., Peng, B., Mi, H.Yu, D.  2023\.  The Trickle-down Impact of Reward (In-) consistency on RLHF The trickle-down impact of reward (in-) consistency on rlhf.  arXiv preprint arXiv:2309.16155.
  * Singhal . (2023) singhal2023longSinghal, P., Goyal, T., Xu, J. Durrett, G.  2023\.  A Long Way to Go: Investigating Length Correlations in RLHF. A long way to go: Investigating length correlations in rlhf.
  * Snell . (2022) snell2022offlineSnell, C., Kostrikov, I., Su, Y., Yang, M. Levine, S.  2022\.  Offline rl for natural language generation with implicit language q learning Offline rl for natural language generation with implicit language q learning.  arXiv preprint arXiv:2206.11871.
  * Spaan (2012) spaan2012partiallySpaan, MT.  2012\.  Partially observable Markov decision processes Partially observable markov decision processes.  Reinforcement learning: State-of-the-art Reinforcement learning: State-of-the-art ( 387–414).  Springer.
  * Stiennon . (2020) stiennon2020learningStiennon, N., Ouyang, L., Wu, J., Ziegler, D., Lowe, R., Voss, C.Christiano, PF.  2020\.  Learning to summarize with human feedback Learning to summarize with human feedback.  Advances in Neural Information Processing Systems333008–3021.
  * Suhr Artzi (2022) suhr2022continualSuhr, A. Artzi, Y.  2022\.  Continual learning for instruction following from realtime feedback Continual learning for instruction following from realtime feedback.  arXiv preprint arXiv:2212.09710.
  * Sun . (2023) sun2023salmonSun, Z., Shen, Y., Zhang, H., Zhou, Q., Chen, Z., Cox, D.Gan, C.  2023\.  SALMON: Self-Alignment with Principle-Following Reward Models. Salmon: Self-alignment with principle-following reward models.
  * Touvron . (2023) touvron2023llamaTouvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y.others  2023\.  Llama 2: Open foundation and fine-tuned chat models Llama 2: Open foundation and fine-tuned chat models.  arXiv preprint arXiv:2307.09288.
  * A. Wei . (2023) wei2023jailbrokenWei, A., Haghtalab, N. Steinhardt, J.  2023\.  Jailbroken: How Does LLM Safety Training Fail? Jailbroken: How does llm safety training fail?
  * J. Wei . (2022) wei2022chainWei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E.others  2022\.  Chain-of-thought prompting elicits reasoning in large language models Chain-of-thought prompting elicits reasoning in large language models.  Advances in Neural Information Processing Systems3524824–24837.
  * R. Wei . (2023) wei2023unifiedWei, R., Lambert, N., McDonald, A., Garcia, A. Calandra, R.  2023\.  A Unified View on Solving Objective Mismatch in Model-Based Reinforcement Learning. A unified view on solving objective mismatch in model-based reinforcement learning.
  * Wirth . (2017) wirth2017surveyWirth, C., Akrour, R., Neumann, G., Fürnkranz, J. .  2017\.  A survey of preference-based reinforcement learning methods A survey of preference-based reinforcement learning methods.  Journal of Machine Learning Research181361–46.
  * J. Wu . (2021) wu2021recursivelyWu, J., Ouyang, L., Ziegler, DM., Stiennon, N., Lowe, R., Leike, J. Christiano, P.  2021\.  Recursively summarizing books with human feedback Recursively summarizing books with human feedback.  arXiv preprint arXiv:2109.10862.
  * T. Wu . (2023) wu2023pairwiseWu, T., Zhu, B., Zhang, R., Wen, Z., Ramchandran, K. Jiao, J.  2023\.  Pairwise Proximal Policy Optimization: Harnessing Relative Feedback for LLM Alignment. Pairwise proximal policy optimization: Harnessing relative feedback for llm alignment.
  * Z. Wu . (2023) wu2023fineWu, Z., Hu, Y., Shi, W., Dziri, N., Suhr, A., Ammanabrolu, P.Hajishirzi, H.  2023\.  Fine-Grained Human Feedback Gives Better Rewards for Language Model Training Fine-grained human feedback gives better rewards for language model training.  arXiv preprint arXiv:2306.01693.
  * Xu . (2023) xu2023shatteringXu, W., Dong, S., Arumugam, D. Van Roy, B.  2023\.  Shattering the Agent-Environment Interface for Fine-Tuning Inclusive Language Models Shattering the agent-environment interface for fine-tuning inclusive language models.  arXiv preprint arXiv:2305.11455.
  * Yang . (2023) yang2023largeYang, C., Wang, X., Lu, Y., Liu, H., Le, QV., Zhou, D. Chen, X.  2023\.  Large Language Models as Optimizers. Large language models as optimizers.
  * Yue . (2012) yue2012kYue, Y., Broder, J., Kleinberg, R. Joachims, T.  2012\.  The k-armed dueling bandits problem The k-armed dueling bandits problem.  Journal of Computer and System Sciences7851538–1556.
  * Zheng . (2023) zheng2023judgingZheng, L., Chiang, WL., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y.others  2023\.  Judging LLM-as-a-judge with MT-Bench and Chatbot Arena Judging llm-as-a-judge with mt-bench and chatbot arena.  arXiv preprint arXiv:2306.05685.
  * Ziegler . (2019) ziegler2019fineZiegler, DM., Stiennon, N., Wu, J., Brown, TB., Radford, A., Amodei, D.Irving, G.  2019\.  Fine-tuning language models from human preferences Fine-tuning language models from human preferences.  arXiv preprint arXiv:1909.08593.