# The structure of homogeneous Riemannian manifolds with nullity

Antonio J. Di Scala  ,  Carlos E. Olmos  and  Francisco Vittone

###### Abstract.

We find new conditions that the existence of nullity of the curvature tensor
of an irreducible homogeneous space $M=G/H$ imposes on the Lie algebra
$\mathfrak{g}$ of $G$ and on the Lie algebra $\tilde{\mathfrak{g}}$ of the
full isometry group of $M$. Namely, we prove that there exists a transvection
of $M$ in the direction of any element of the nullity, possibly by enlarging
the presentation group $G$. Moreover, we prove that these transvections
generate an abelian ideal of $\tilde{\mathfrak{g}}$. These results constitute
a substantial improvement on the structure theory developed in [DOV]. In
addition we construct examples of homogeneous Riemannian spaces with non-
trivial nullity, where $G$ is a non-solvable group, answering a natural open
question. Such examples admit (locally homogeneous) compact quotients. In the
case of co-nullity $3$ we give an explicit description of the isometry group
of any homogeneous locally irreducible Riemannian manifold with nullity.

###### 2010 Mathematics Subject Classification:

Primary 53C30; Secondary 53C40

A. J. Di Scala is member of CrypTO, GNSAGA of INdAM and of DISMA Dipartimento
di Eccellenza MIUR 2018-2022; C. E. Olmos was supported by Famaf-UNC and CIEM-
Conicet; F. Vittone was supported by UNR and Conicet.

##  1\. Introduction

In a previous paper [DOV] we developed a general theory for the structure of
irreducible homogeneous spaces $M=G/H$ in relation to the nullity distribution
$\nu$ associated to the curvature tensor $R$. The nullity subspace of $M$ at
$p$ is defined as

$\nu_{p}=\\{v\in T_{p}M\,:\,R_{\,\cdot\,,\,\cdot}v=0\\}.$

Since $M$ is homogeneous, all these subspaces have the same dimension, called
the index of nullity of $M$. The assignment $p\mapsto\nu_{p}$ defines an
autoparallel distribution, with flat totally geodesic integral manifolds,
called the nullity distribution or simply the nullity of $M$. The manifold $M$
is said to have trivial nullity if either $\nu=\\{0\\}$ or $\nu$ is the
tangent space of a local de Rham flat factor of $M$ (i.e., $\nu$ is a parallel
distribution with respect to the Levi-Civita connection of $M$).

The theory developed in [DOV] allowed us to prove some interesting results. On
the one hand, we showed that if $M=G/H$ is a simply connected homogeneous
Riemannian manifold without Euclidean de Rham factor such that the Lie algebra
$\mathfrak{g}$ of $G$ is reductive (in particular, if $M$ is compact) or
$\mathfrak{g}$ is $2$-step nilpotent, then $\nu$ must be trivial (cf. [DOV,
Proposition C]). On the other hand, we were able to construct the first known
examples of irreducible homogeneous spaces with non-trivial nullity. All these
examples are solvmanifolds which do not admit compact quotients. Moreover, we
showed in part (4) of [DOV, Theorem A] that if the co-nullity (i.e. the co-
dimension of the nullity $\nu$) is $k=3$, then the presentation group $G$ must
be solvable. However, as we pointed out in the Introduction of [DOV], we did
not know of any example of an irreducible simply connected homogeneous
Riemannian manifold $M=G/H$ with positive index of nullity such that either
$H$ is non-trivial or $G$ is non-solvable. In this paper we answer the second
question by constructing examples of Riemannian homogeneous spaces with non-
trivial nullity and non-solvable presentation group. Such examples admit
compact (locally homogeneous) quotients (see Section 6).

The main goal of the paper is to revisit the structure theory developed in
[DOV] and substantially improve it. Let $M=G/K$ be a locally irreducible
homogeneous Riemannian manifold with non-trivial nullity $\nu$. One can
consider some natural distributions associated to $\nu$. Namely,
$\nu^{(1)},\nu^{(2)}$, $\hat{\nu}$ and $\mathcal{U}$.

The distributions $\nu^{(1)}$ and $\nu^{(2)}$ are called the osculating
distributions of order $1$ and $2$ of $\nu$, respectively. If one puts
$\nu^{(0)}=\nu$, then $\nu^{(i+1)}$ is obtained by adding to $\nu^{(i)}$ the
covariant derivative, in any direction, of fields that lie in $\nu^{(i)}$
($i=0,1$). The distribution $\nu^{(1)}$ can, in turn, be decomposed as

$\nu^{(1)}=\nu+\hat{\nu}$

where $\hat{\nu}_{p}$ is the linear span of $\\{\nabla_{\nu_{p}}Z\\}$ with
$Z\in\mathcal{K}^{G}(M)\simeq\mathfrak{g}$, the space of Killing fields of $M$
whose flow belong to $G$ (see equation 2.8). The distribution $\hat{\nu}$ is
called the adapted distribution of $\nu$.

Finally, the distribution $\mathcal{U}$, called the bounded distribution.
Namely, $\mathcal{U}_{p}$ consists of the directions of the Killing fields in
$\mathcal{K}^{G}(M)$ whose covariant derivatives along the leaf of nullity
$N(p)$ lie in $\nu$ (see equation(3.1)).

All these distributions are $G$-invariant and verify the following inclusions
(see [DOV, Theorem A (1)]):

$\\{0\\}\subsetneq\nu\subsetneq\nu^{(1)}\subsetneq\nu^{(2)}\subset\mathcal{U}\subsetneq
TM.$

This imposes a first important restriction: if $M$ has non-trivial nullity,
then the co-nullity must be at least $3$.

Another important consequence of a non-trivial nullity is the existence of the
so-called adapted transvections. These are transvections induced by $G$ (i.e.
elements of $\mathcal{K}^{G}(M)$ such that $(\nabla X)_{p}=0$) whose initial
condition lie in $\hat{\nu}$. In [DOV, Theorem A (2)] it was proven that for
each $v\in\hat{\nu}_{p}$ there exist an adapted transvection $Y$ with
$Y_{p}=v$ such that the Jacobi operator $R_{\,\cdot\,,\,v}v=0$,
$[Y,[Y,\mathcal{K}(M)]]=0$ and $Y$ does not belong to the center of
$\mathcal{K}^{G}(M)$ (where $\mathcal{K}(M)$ denotes the space of Killing
fields of $M$). The existence of an adapted transvection $Y$ of order two,
i.e. $ad_{Y}^{2}=0$. plays a crucial role in the construction of the examples
of [DOV, Section9]).

In this paper we further examine the restrictions that the existence of
nullity imposes on the Lie algebra $\mathcal{K}(M)$. In particular, we prove
that there exists a transvection, that might not belong to
$\mathcal{K}^{G}(M)$, in the direction of any element of $\nu$. Moreover, we
prove that these transvections generate an abelian ideal $\mathfrak{a}$ of
$\mathcal{K}(M)$. Our main structure result is the following:

######  Theorem 1.1.

Let $M=G/H$ be a simply connected irreducible homogeneous Riemannian manifold
with a non-trivial nullity distribution $\nu$, let
$\mathfrak{g}=\text{Lie}(G)\simeq\mathcal{K}^{G}(M)$ and
$\tilde{\mathfrak{g}}=\text{Lie}(I(M))\simeq\mathcal{K}(M)$. Then

  * 1)

For any $p\in M$, $v\in\nu_{p}$ there exists a transvection with initial
condition $v$ (possibly, not in $\mathfrak{g}$).

  * 2)

Let $\tilde{\mathfrak{tr}}^{p}_{0}\subset\tilde{\mathfrak{g}}$ be the vector
space of transvections at $p$ with initial condition in $\nu_{p}$. Then
$\mathfrak{g}^{\prime}:=\tilde{\mathfrak{tr}}^{p}_{0}+\mathfrak{g}$ is a Lie
subalgebra of $\tilde{\mathfrak{g}}$ which does not depend on $p\in M$.
Moreover, $\mathfrak{g}$ is an ideal of $\mathfrak{g}^{\prime}$.

  * 3)

The set $\tilde{\mathfrak{tr}}^{p}_{0}$ generates an abelian ideal
$\mathfrak{a}$ in $\tilde{\mathfrak{g}}$. Moreover, $\mathfrak{a}$ does not
depend on $p\in M$ and coincides with the ideal generated by
$\tilde{\mathfrak{tr}}^{p}_{0}$ in $\mathfrak{g}^{\prime}$.

  * 4)

The integrable distribution $\mathcal{D}$ of $M$, given by
$\mathcal{D}_{q}=\mathfrak{a}.q$, contains $\nu^{(2)}$. Moreover, $\nu$ and
$\nu^{(1)}$ are parallel along any orbit $\bar{A}\cdot q$, where $\bar{A}$ is
the closure in $I(M)$ of the Lie group associated to $\mathfrak{a}$.

Part (2) and (3) of the above theorem explains why the adapted transvections
of [DOV, Theorem A (2)] have order two. In fact, any adapted transvection can
be obtained as the Lie bracket of a transvection in
$\tilde{\mathfrak{tr}^{p}_{0}}$ and a Killing field in $\mathcal{K}^{G}(M)$
(see Remark 5.1). Hence the subspace of adapted transvections is contained in
the abelian ideal $\mathfrak{a}$.

Moreover, since the adapted transvections cannot lie in the center of
$\mathcal{K}^{G}(M)$, the existence of the abelian ideal $\mathfrak{a}$
implies that the Lie algebra $\mathfrak{g}$ of $G$ cannot be reductive. This
gives a conceptual much simpler proof of [DOV, Proposition C].

The results of Theorem 1.1 also allow us to improve part (4) of Theorem A of
[DOV]. In fact, we obtain the following explicit description of the Lie group
$G$ in case of co-nullity $k=3$.

######  Corollary 1.2.

Let $M^{n}=G/H$ be a simply connected irreducible homogeneous Riemannian
manifold of co-nullity $3$, where $G$ is connected ($n>3$). Then $H$ is
trivial and $G=\mathbb{R}^{n-1}\rtimes\mathbb{R}$.

Observe that the projection from $M$ to the quotient $\bar{A}\backslash M$ by
the orbits of $\bar{A}$ is a $G$-invariant Riemannian submersion with
intrinsically flat fibers.

Finally, we give an explicit way to construct non-solvable examples. Namely:

######  Theorem 1.3.

Let $K$ be a simply connected compact simple Lie group and let $\rho:K\to
SO_{n}$ be an irreducible orthogonal representation. Let $\mathbb{V}_{0}$ be a
non-trivial vector subspace of $\mathbb{R}^{n}$ such that
$\dim(\mathbb{V}_{0})(1+\dim(K))<n$. Then there exists a left invariant metric
$\langle\,,\,\rangle$ on $G=\mathbb{R}^{n}\rtimes_{\rho}K$ such that
$M=(G,\langle\,,\,\rangle)$ is an irreducible Riemannian manifold and the
nullity subspace $\nu_{e}$ contains $\mathbb{V}_{0}$ (and hence has dimension
at least $\dim(\mathbb{V}_{0})$). In particular, the quotient
$N=\mathbb{Z}^{n}\backslash M$ of $M$ by the orbits of $\mathbb{Z}^{n}$ is a
compact, locally irreducible, non-homogeneous but locally homogeneous
Riemannian manifold with a non-trivial nullity distribution.

Note that for any simply connected and simple compact Lie group $K$ there
exists a representation $\rho:K\to SO_{n}$ for arbitrary large $n$. The
manifold $M$ given by the above theorem is homotopic to a compact simply
connected Lie group and thus it admits no solvable transitive group of
isometries.

Observe that, as we mentioned before, any compact locally irreducible
homogeneous Riemannian manifold has a trivial nullity distribution ([DOV,
Proposition C]). Then the compact manifold $N=\mathbb{Z}^{n}\backslash M$,
given by the above theorem, shows that the assumption of homogeneity can not
be replaced by local homogeneity. This also answers a natural question (other
examples of compact locally homogeneous spaces with non-trivial nullity were
found in [GG]).

##  2\. Preliminaries

In this section we introduce some basic facts which will be needed in the rest
of the paper. For further details we refer to [DOV]. Let
$(M,\langle\,,\,\rangle)$ be a (connected) complete Riemannian manifold with
Levi-Civita connection $\nabla$ . A vector field $X$ of $M$ is called a
Killing field if $(\nabla_{\cdot}X)_{p}$ is a skew-symmetric endomorphisms of
$T_{p}M$, for all $p\in M$. Such a condition, the so-called Killing equation,
is equivalent to the fact that the flow $\phi_{t}$ of $X$ is by isometries.
The Lie algebra $\mathrm{Lie}(I(M))$ of the group $I(M)$ of isometries of $M$
is naturally identified with the Lie algebra $\mathcal{K}(M)$ of Killing
fields of $M$. Namely, $u\in\mathrm{Lie}(I(M))=T_{e}(I(M))$ induces the
Killing field $\tilde{u}$ defined by
$\tilde{u}_{p}=\tfrac{\mathrm{d}\,}{\mathrm{d}t}_{0}\mathrm{Exp}(tu).p$ and
the map $u\mapsto\tilde{u}$ is a linear isomorphism. Moreover,
$[u,w]^{\tilde{\ }}=-[\tilde{u},\tilde{w}]$ since $\tilde{u}$ and $\tilde{w}$
are naturally identified with the right invariant fields of $I(M)$ with
initial conditions $u$ and $w$, respectively.

Let $G$ act by isometries on $M$. The Killing fields of $M$ induced by
elements in the Lie algebra of $G$ are denoted by $\mathcal{K}^{G}(M)$
(briefly, the Killing fields induced by $G$). If the action of $G$ on $M$ is
not almost effective, then there exist non-zero elements $z\in\mathfrak{g}$
such that $\tilde{z}=0$.

Let $X\in\mathcal{K}(M)$. The initial conditions of $X$ at $p\in M$ are given
by the pair

$(X)^{p}:=(X_{p},(\nabla X)_{p})\in T_{p}M\oplus\Lambda^{2}(T_{p}M).$

These conditions completely determine the Killing field $X$, in the sense that
two Killing fields with the same initial conditions at some point $p$ must
coincide on $M$. A Killing field is called a transvection at $p$ if its
initial conditions at $p$ are $(X)^{p}:=(X_{p},0)$, i.e., if $(\nabla
X)_{p}=0$.

A Killing field $X$ satisfies, besides the Killing equation, the following
identity, for all $p\in M$, $u,v\in T_{p}M$

(2.1) $\ \ \ \ \ \ \ \ \ \ \ \ \nabla^{2}_{u,v}X=R_{u,X_{p}}v\ \ \ \ \ \ \ \ \
\ \mathit{affine\ Killing\ equation}.$

where $R$ is the Riemannian curvature tensor of $M$. The affine Killing
equation reflects the fact that the flow of $X$ preserves the Levi-Civita
connection.

From the affine Killing equation and the Bianchi identity one can determine
the initial conditions at $p$ of the bracket $[X,X^{\prime}]$ of two Killing
fields in terms of the initial conditions $(X)^{p}=(v,B)$,
$(X^{\prime})^{p}=(v^{\prime},B^{\prime})$. Namely,

(2.2)
$([X,X^{\prime}])^{p}=(B^{\prime}(v)-B(v^{\prime}),R_{v,v^{\prime}}-[B,B^{\prime}]).$

This equation gives a useful formula for computing the curvature in terms of
the Killing fields $X$ and $Y$. Namely:

(2.3) $R_{X_{p},Y_{p}}=(\nabla[X,Y])_{p}+[(\nabla X)_{p},(\nabla Y)_{p}]$

The so-called Koszul formula gives the Levi-Civita connection in terms of
brackets of vector fields and scalar products. From the fact that the Lie
derivative of the metric tensor along any Killing vector field is zero one has
the following useful formula (see e.g. (3.4) of [ORT]):

(2.4)
$2\langle\nabla_{X}Y,Z\rangle=\langle[X,Y],Z\rangle+\langle[X,Z],Y\rangle+\langle[Y,Z],X\rangle.$

Let $X$ be a Killing field and let $\phi_{t}(q)$ be its associated flow. Let
$p\in M$ and let $c(t)=\phi_{t}(p)$ be the integral curve of $X$ by $p$. Let
$\tau_{t}$ denote the parallel transport along $c(t)$, form $0$ to $t$. Then
$\tau_{t}^{-1}\circ\mathrm{d}\phi_{t}:T_{p}M\to T_{p}M$ is a $1$-parameter
subgroup of linear isometries. Moreover, (see e.g. Remark 2.3 of [OS])

(2.5) $\tau_{t}^{-1}\circ\mathrm{d}_{p}\phi_{t}=\mathrm{e}^{t(\nabla X)_{p}}$

or equivalently

(2.6) $\tau_{t}=\mathrm{d}_{p}\phi_{t}\circ\mathrm{e}^{-t(\nabla X)_{p}}.$

######  Remark 2.1.

(cf. [DOV, Remark 2.3]). Let $X\in\mathcal{K}(M)$ and let $T\subset
SO(T_{p}M)$ be the torus given by the closure of $\\{\mathrm{e}^{t(\nabla
X)_{p}}:t\in\mathbb{R}\\}$. Then for any given $d\in T$ there is a sequence of
real numbers $\\{t_{n}\\}_{n\in\mathbb{N}}$ which tends to $+\infty$ and such
that
$\tau_{t_{n}}^{-1}\circ\mathrm{d}_{p}\phi_{t_{n}}=\mathrm{e}^{t_{n}(\nabla
X)_{p}}$ tends to $d$. In particular, for $d=\mathrm{e}^{s_{0}(\nabla
X)_{p}}$, where $s_{0}\in\mathbb{R}$ is arbitrary.

The nullity of the curvature tensor $R$ at $p\in M$ is the subspace $\nu_{p}$
of $T_{p}M$ defined as

$\nu_{p}:=\\{v\in T_{p}M:R_{v,x}=0,\forall x\in T_{p}M\\}$

or, equivalently, from the identities of the curvature tensor,

$\ \ \ \ \ \nu_{p}:=\\{v\in T_{p}M:R_{x,y}v=0,\forall x,y\in T_{p}M\\}.$

The map $\nu:p\mapsto\nu_{p}$ defines an autoparallel distribution in the open
and dense subset $\Omega$ of $M$ where the dimension $\dim(\nu_{q})$ is
locally constant, called the nullity distribution of $M$. If $M$ is
homogeneous, then $\nu$ is a distribution in $M$ and one has:

######  Theorem 2.2 (cf. [DOV] Theorem C).

If $M$ is a homogeneous compact Riemannian manifold with no Euclidean local de
Rham factor, then its nullity distribution is trivial.

We recall Lemma 2.8 of [DOV]:

######  Lemma 2.3.

Let $M$ be a Riemannian manifold. Let $\gamma_{v}(t)$ be a geodesic with
$\gamma_{v}(0)=p$, $\gamma_{v}^{\prime}(0)=v$ and such that
$\gamma_{v}^{\prime}(t)$ belongs to the nullity subspace $\nu_{\gamma_{v}(t)}$
for every $t$ (if $\nu$ is a distribution, then this is equivalent to the fact
that $v\in\nu_{p}$). Denote by $\tau_{t}$ the parallel transport along
$\gamma_{v}(t)$ from $0$ to $t$. If $Z$ is an arbitrary Killing field on $M$,
then

  1. (i)

$Z_{\gamma_{v}(t)}=\tau_{t}(Z_{p})+t\tau_{t}(\nabla_{v}Z)$.

  2. (ii)

$\nabla_{\gamma_{v}^{\prime}(t)}(\nabla Z)=0$, i.e., $\nabla Z$ is parallel
along $\gamma_{v}(t)$, or equivalently

$(\nabla Z)_{\gamma_{v}(t)}=\tau_{t}((\nabla Z)_{p}):=\tau_{t}\circ(\nabla
Z)_{p}\circ\tau_{t}^{-1}.$

Let $M=G/H$ be a simply connected homogeneous Riemannian manifold without
Euclidean de Rham factor and assume that its nullity distribution $\nu$ is
non-trivial. Observe that $\nu$ is not a parallel distribution, otherwise $M$
would split off a Euclidean factor. In addition, in this case, a non-trivial
Killing field cannot be always tangent to the nullity distribution ([DOV,
Propositon 3.19]).

For a $G$-invariant distribution $\mathcal{D}$ on $M$, consider the osculating
distribution $\mathcal{D}^{(1)}$ of $\mathcal{D}$ defined as

$\mathcal{D}^{(1)}_{q}:=\mathcal{D}_{q}+\mathrm{span}\,\\{\nabla_{w}X:X\in
C^{\infty}(\mathcal{D}),w\in T_{q}M\\}$

where $C^{\infty}(\mathcal{D})$ denotes the tangent fields of $M$ that lie in
$\mathcal{D}$. It is not difficult to see that $\mathcal{D}^{(1)}$ is a
$G$-invariant distribution of $M$ such that
$\mathcal{D}\subset\mathcal{D}^{(1)}$. Moreover, if $\mathcal{D}$ is not
parallel, then $\mathcal{D}^{(1)}$ properly contains $\mathcal{D}$. On has
that (cf. [DOV, Lemma 3.1])

(2.7) $\mathcal{D}^{(1)}=\mathcal{D}+\hat{\mathcal{D}}$

where

$\hat{\mathcal{D}}_{q}:=\\{\nabla_{v}Z\,:\,Z\in\mathcal{K}^{G}(M),\,v\in\mathcal{D}_{q}\\}.$

$\hat{\mathcal{D}}$ is called the adapted distribution of $\mathcal{D}$. The
adapted distribution might depend on the presentation group $G$, but the
higher order osculating distributions
$\mathcal{D}^{(k)}:=(\mathcal{D}^{(k-1)})^{(1)}$ (where
$\mathcal{D}^{(0)}=\mathcal{D})$ do not depend on $G$.

Let $\nu^{(1)}$ be the osculating distribution of the nullity distribution and
set $\nu^{(2)}:=(\nu^{(1)})^{(1)}$, called the second order osculating
distribution of $\nu$. Both $\nu^{(1)}$ and $\nu^{(2)}$ are $G$-invariant
distributions. Since $\nu$ is not parallel, $\nu$ is properly contained in
$\nu^{(1)}$. So one has that

(2.8)
$\nu_{q}^{(1)}=\nu_{q}+\hat{\nu}_{q}=\nu_{q}+\mathrm{span}\,\\{\nabla_{v}Z:Z\in\mathcal{K}^{G}(M),v\in\nu_{q}\\}$

and one can describe $\nu^{(2)}$ as

(2.9) $\nu^{(2)}=\nu^{(1)}+\hat{\nu^{(1)}}=\nu+\hat{\nu}+\hat{\hat{\nu}}.$

where
$\hat{\hat{\nu}}=\mathrm{span}\,\\{\nabla_{v}Z:Z\in\mathcal{K}^{G}(M),v\in\hat{\nu}_{q}\\}$
(cf. [DOV, Formula 5.0.2]).

The osculating distributions of $\nu$ have the following properties:

######  Lemma 2.4 ([DOV] Theorem A and Corollary 3.13).

Let $M$ be a Riemannian homogeneous manifold which does not split off a local
Euclidean de Rham factor and with non-trivial nullity distribution $\nu$.
Then:

  1. (1)

$\nu$ and $\nu^{(1)}$ are autoparallel distributions and
$\nu\subsetneq\nu^{(1)}\subsetneq\nu^{(2)}\subsetneq TM$. In particular, the
co-dimension of $\nu$ is at least $3$.

  2. (2)

For any $v\in\hat{\nu}_{q}$ there exists a transvection $Y$ of $M$ which
belongs to $\mathcal{K}^{G}(M)$ and such that $Y_{p}=v$ (such a transvection
is called an  adapted transvection). Moreover, the Jacobi operator
$R_{.,\,v}v$ is null,

$[Y,[Y,\mathcal{K}(M)]]=\\{0\\}$

and $[Y,\mathcal{K}^{G}(M)]\neq\\{0\\}$, i.e $Y$ does not belong to the center
of $\mathcal{K}^{G}(M)$.

Observe that if $X$ is a transvection at $p$, since $\nabla$ is torsion free,

(2.10) $\nabla_{X_{p}}Z=[X,Z]_{p}\,.$

Hence, from property (2) of Lemma 2.4 one gets that

(2.11) $\hat{\hat{\nu}}_{p}=\\{{[X,Z]_{p}}:Z\in\mathcal{K}^{G}(M),X\text{ is
an adapted transvection at }p\\}.$

##  3\. The ideal $\mathfrak{u}_{00}$

We keep the notation of Section 2. Let $M=G/H$ be a simply connected
Riemannian homogeneous manifold without Euclidean de Rham factor and assume
that the nullity distribution $\nu$ of $M$ is non-trivial.

The bounded algebra of $M$ at $p$ is given by

(3.1)
$\mathfrak{u}^{p}:=\\{Z\in\mathcal{K}^{G}(M):\nabla_{\nu_{p}}Z\subset\nu_{p}\\}.$

From formula (2.2) one obtains that $\mathfrak{u}^{p}$ is a Lie subalgebra of
$\mathcal{K}^{G}(M)$ (see [DOV, Section 5]) which verifies that for every
$g\in G$,

$\mathfrak{u}^{gp}=\mathrm{Ad}_{g}\mathfrak{u}^{p}.$

So $\mathcal{U}_{p}=\mathfrak{u}^{p}.p$ defines an integrable distribution.
Namely, the integral manifold of $\mathcal{U}$ by $p$ is the orbit by $p$ of
the Lie group associated to $\mathfrak{u}^{p}$. The distribution $\mathcal{U}$
is the so-called bounded distribution. It contains the first two osculating
distributions ([DOV, Theorem A]). More precisely, on has that

(3.2) $\nu\subsetneq\nu^{(1)}\subsetneq\nu^{(2)}\subset\mathcal{U}\subsetneq
TM.$

Observe that any transvection at $p$ belongs to $\mathfrak{u}^{p}$. In
particular,

(3.3) $\mathfrak{tr}^{p}:=\\{X\in\mathcal{K}^{G}(M):X\textrm{ is a
transvection at $p$ with }X_{p}\in\nu^{(1)}_{p}\\}\subset\mathfrak{u}^{p}.$

Moreover, by [DOV, Remark 3.10] one has that

(3.4) $R_{\,\mathfrak{tr}^{p},\mathfrak{u}^{p}}=0$

Then, from (2.2), it follows that $\mathfrak{tr}^{p}$ is an abelian Lie
subalgebra of $\mathfrak{u}^{p}$.

Observe that since $\hat{\nu}_{p}\subset\nu_{p}^{(1)}$, from item (2) of Lemma
2.4 one gets that for each $u,v\in\hat{\nu}_{p}$ there exist transvections
$X,Y\in\mathfrak{tr}^{p}$ such that $X_{p}=u$, $Y_{p}=v$. So by (3.4)
$R_{u,v}=R_{X_{p},Y_{p}}=0$, i.e.,

(3.5) $R_{\hat{\nu}_{p},\hat{\nu}_{p}}=0.$

######  Remark 3.1.

From (2.8) and Lemma 2.4, any $X\in\mathfrak{tr}^{p}$ is the sum of an adapted
transvection at $p$ (i.e., a transvection whose value al $p$ belongs to
$\hat{\nu}_{p}$) and a transvection tangent to the nullity at $p$. But we do
not know whether there exists a transvection in any direction of $\nu_{p}$. We
will prove this fact in the next section (but such a transvection might not
belong to $\mathcal{K}^{G}(M)$).

######  Remark 3.2.

From Lemma 2.4 one has that $R_{\,\cdot,X_{p}}X_{p}=0$, for all
$X\in\mathfrak{tr}^{p}$. Polarizing this formula one gets that
$R_{\,\cdot,X_{p}}Y_{p}+R_{\,\cdot,Y_{p}}X_{p}=0$ for all
$X,Y\in\mathfrak{tr}^{p}$. Note that from Bianchi identity
$R_{\,\cdot,X_{p}}Y_{p}-R_{\,\cdot,Y_{p}}X_{p}=-R_{X_{p},Y_{p}}\cdot=0$ (see
(3.4)). Then, for all $X,Y\in\mathfrak{tr}^{p}$,

$R_{\,\cdot,X_{p}}Y_{p}=0.$

Let

(3.6) $\mathfrak{u}^{p}_{0}:=\\{Z\in\mathcal{K}^{G}(M):\nabla_{\nu_{p}}Z=0\\}$

and

(3.7)
$\mathfrak{u}^{p}_{00}:=\\{Z\in\mathcal{K}^{G}(M):\nabla_{\nu_{p}}Z=0\text{
and }\nabla_{\hat{\nu}_{p}}Z=0\\}.$

Equivalently, by (2.8),

(3.8)
$\mathfrak{u}^{p}_{00}:=\\{Z\in\mathcal{K}^{G}(M):\nabla_{(\nu^{(1)})_{p}}Z=0\\}$

Clearly,

(3.9)
$\mathfrak{tr}^{p}\subset\mathfrak{u}^{p}_{00}\subset\mathfrak{u}^{p}_{0}\subset\mathfrak{u}^{p}.$

######  Lemma 3.3.

If $U\in\mathfrak{u}^{p}_{0}$, then
$\nabla_{\hat{\nu}_{p}}U\subset\hat{\nu}_{p}$.

###### Proof.

Let $(U)^{p}=(u,B)$ be the initial conditions of $U$. Let
$Z\in\mathcal{K}^{G}(M)$ be arbitrary with initial conditions
$(Z)^{p}=(z,B^{\prime})$ and put $W=[U,Z]$. Let $v\in\nu_{p}$ be arbitrary.
Observe that, from the definition of $\hat{\nu}_{p}$,
$\nabla_{v}W\in\hat{\nu}_{p}$. From (2.2) the initial conditions of $W$ at $p$
are

$(W)^{p}=(B^{\prime}(u)-B(z),R_{u,z}-(BB^{\prime}-B^{\prime}B)).$

Since $v\in\nu_{p}$, one has that $B(v)=0$ and $R_{u,z}(v)=0$. Hence

$\nabla_{v}W=-B(B^{\prime}(v))\in\hat{\nu}_{p}.$

But the set $\\{B^{\prime}(v)\\}$, for $Z\in\mathcal{K}^{G}(M)$ and
$v\in\nu_{p}$ arbitrary, spans $\hat{\nu}_{p}$. Then

$B(\hat{\nu}_{p})\subset\hat{\nu}_{p}\,.$

∎

######  Theorem 3.4.

$\mathfrak{u}^{p}_{00}$ is an ideal of $\mathcal{K}^{G}(M)$ and does not
depend on $p\in M$ (and so in the next we will refer to it as
$\mathfrak{u}_{00}$). Moreover, for any $p\in M$, $\mathfrak{tr}^{p}$ is
contained in the center $\mathfrak{z}(\mathfrak{u}_{00})$ of
$\mathfrak{u}_{00}$.

###### Proof.

Let $Z\in\mathcal{K}^{G}(M)$ be arbitrary with initial conditions
$(Z)^{p}=(z,B^{\prime})$ and let $U\in\mathfrak{u}^{p}_{00}$, with initial
conditions $(U)^{p}=(u,B)$. Set $W=[U,Z]$. Let us see first that
$W\in\mathfrak{u}^{p}_{0}$.

From (2.2), the initial conditions of $W$ at $p$ are

$(W)^{p}=(B^{\prime}(u)-B(z),R_{u,z}-(BB^{\prime}-B^{\prime}B)).$

Since $U\in\mathfrak{u}^{p}_{00}$ one has that
$B(\nu_{p})=B(\hat{\nu}_{p})=0$. On the other hand, from the definition of
$\hat{\nu}$ on gets that $B^{\prime}(\nu_{p})\subset\hat{\nu}_{p}$, hence
$B(B^{\prime}(\nu_{p}))=0$.

So, if $v\in\nu_{p}$ is arbitrary, then

$\nabla_{v}W=R_{u,z}v-B(B^{\prime}(v))+B^{\prime}(B(v))=-B(B^{\prime}(v))=0.$

Then $W\in\mathfrak{u}^{p}_{0}$. So we need to prove that
$\nabla_{\hat{\nu}_{p}}W=0$.

Let now $r\in\hat{\nu}_{p}$. From Lemma 3.3 one has that
$\nabla_{r}W\in\hat{\nu}_{p}$. So, in order to show that $\nabla_{r}W=0$, we
only have to show that the projection of $\nabla_{r}W$ to $\hat{\nu}_{p}$ is
zero. Let $s\in\hat{\nu}_{p}$ be arbitrary. Then

$\langle\nabla_{r}W,s\rangle=\langle R_{u,z}r,s\rangle-\langle
B(B^{\prime}(r)),s\rangle+\langle B^{\prime}(B(r)),s\rangle.$

Since $r,s\in\hat{\nu}_{p}$, by (3.5) $R_{r,s}=0$. Hence $\langle
R_{u,z}r,s\rangle=\langle R_{r,s}u,z\rangle=0.$

On the other hand, recall that $B(\hat{\nu}_{p})=0$ hence $B(r)=B(s)=0$ and
then

$\langle\nabla_{r}W,s\rangle=-\langle B(B^{\prime}(r)),s\rangle=\langle
B^{\prime}(r),B(s)\rangle=0.$

Then $W\in\mathfrak{u}^{p}_{00}$ and thus $\mathfrak{u}^{p}_{00}$ is an ideal
of $\mathcal{K}^{G}(M)$.

Being $\mathfrak{u}^{p}_{00}$ an ideal it does not depend on $p\in M$, since
$\mathfrak{u}^{gp}_{00}=\mathrm{Ad}(g)\mathfrak{u}^{p}_{00}$.

By making use of (2.2), (3.4) and (3.9) one obtains that
$\mathfrak{tr}^{p}\subset\mathfrak{z}(\mathfrak{u}^{p}_{00})$ for all $p\in
M$. ∎

######  Remark 3.5.

Observe that the center of an ideal of a Lie algebra $\mathfrak{g}$ is an
abelian ideal of $\mathfrak{g}$. Then $\mathfrak{z}(\mathfrak{u}_{00})$ is an
abelian ideal of $\mathcal{K}^{G}(M)$.

##  4\. Transvections in the directions of the nullity

All throughout this section $M=G/H$ will be a homogeneous Riemannian manifolds
with nullity distribution $\nu$ and curvature tensor $R$.

######  Lemma 4.1.

Let $p\in M$, $v\in\nu_{p}$. Then $\nabla_{v}R=0$.

###### Proof.

Since the first terms of both the equations in i) and ii) are linear in $v$,
it is clear that we only need to prove them for $v$ in a set of generators of
$\nu_{p}$. Now each leaf of $\nu$ is a homogeneous, flat totally geodesic
submanifold of $M$ (see [DOV, Paragraph 3.2]). Hence $\nu_{p}$ is generated by
the direction of homogeneous geodesics tangent to $\nu$, which are given by
the flow of some Killing field in $\mathcal{K}^{G}(M)$ (cf. [DOV, Paragraph
2.5]).

So let $v\in\nu_{p}$ be such that there exist a homogeneous geodesic
$\gamma_{v}(t)$ tangent to $\nu$ with $\gamma_{v}(0)=p$ and
$\gamma^{\prime}_{v}(0)=v$ and let $X\in\mathcal{K}^{G}(M)$ be such that

$\gamma_{v}(t)=\phi_{t}(p)$

where $\phi_{t}$ is the flow associated to $X$.

Let $\tau_{t}$ be the parallel transport along $\gamma_{v}(t)$ from $0$ to
$t$. By Lemma 2.3,

$Z_{\gamma_{v}(t)}=\tau_{t}(Z_{p})+t\tau_{t}(\nabla_{v}Z)$

and $\nabla_{\gamma_{v}^{\prime}(t)}(\nabla Z)=0$ and the same is true if one
replaces $Z$ by $Y$. From (2.2) one has that

$(\nabla[Y,Z])_{\gamma_{v}(t)}=R_{Y_{\gamma_{v}(t)},Z_{\gamma_{v}(t)}}-[(\nabla
Y)_{\gamma_{v}(t)},(\nabla Z)_{\gamma_{v}(t)}].$

Observe that $C(t)=[(\nabla Y)_{\gamma_{v}(t)},(\nabla Z)_{\gamma_{v}(t)}]$ is
a parallel skew symmetric $(1,1)$ tensor along $\gamma_{v}(t)$ since both
terms of the bracket are so. Then

$C(t)=\tau_{t}(C(0))=\tau_{t}\circ C(0)\circ\tau_{t}^{-1}.$

Set $u=Y_{p}$, $u^{\prime}=\nabla_{v}Y$, $w=Z_{p}$, $w^{\prime}=\nabla_{v}Z$.
Then,

(4.1) $\begin{split}(\nabla[Y,Z])_{\gamma_{v}(t)}=&\
R_{\tau_{t}(u),\tau_{t}(w)}+t(R_{\tau_{t}(u),\tau_{t}(w^{\prime})}+R_{\tau_{t}(u^{\prime}),\tau_{t}(w)})\\\
&+t^{2}R_{\tau_{t}(u^{\prime}),\tau_{t}(w^{\prime})}{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}-}C(t)\end{split}$

By (2.6) one has that

(4.2) $\tau_{t}=\mathrm{d}\phi_{t}\circ g_{t}\,,$

where $g_{t}:=\mathrm{e}^{-t(\nabla X)_{p}}$ is a one parameter group of
linear isometries of $T_{p}M$. So, since $\phi_{t}$ preserves the curvature,

(4.3)
$\begin{split}&(\nabla[Y,Z])_{\gamma_{v}(t)}-R_{\tau_{t}(u),\tau_{t}(w)}=\\\
&\mathrm{d}\phi_{t}\bigg{(}t(R_{g_{t}(u),g_{t}(w^{\prime})}+R_{g_{t}(u^{\prime}),g_{t}(w)})+t^{2}R_{g_{t}(u^{\prime}),g_{t}(w^{\prime})}\bigg{)}{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}-}\tau_{t}(C(0)))\\\
&=\mathrm{d}\phi_{t}\circ\bigg{(}t(R_{g_{t}(u),g_{t}(w^{\prime})}+R_{g_{t}(u^{\prime}),g_{t}(w)})+t^{2}R_{g_{t}(u^{\prime}),g_{t}(w^{\prime})}\bigg{)}\circ\mathrm{d}\phi_{t}^{-1}{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}-}\\\
&\tau_{t}\circ C(0)\circ\tau_{t}^{-1}\end{split}$

Observe that, since $M$ is homogeneous, $\|R_{x,y}\|$ is bounded if $\|x\|$
and $\|y\|$ are bounded. In addition, recall that the covariant derivative of
any Killing field is parallel along the nullity (cf. Lemma 2.3). Hence
$(\nabla[Y,Z])_{\gamma_{v}(t)}$ is parallel and thus the norm of the left hand
side of (4.3) is bounded.

On the other hand, $\|\tau_{t}\circ C(0)\circ\tau_{t}^{-1}\|$ is also bounded
and then

(4.4)
$\alpha(t)=\|t(R_{g_{t}(u),g_{t}(w^{\prime})}+R_{g_{t}(u^{\prime}),g_{t}(w)})+t^{2}R_{g_{t}(u^{\prime}),g_{t}(w^{\prime})}\|$

must be bounded.

Fix an arbitrary $t$ and choose a sequence $t_{n}\to+\infty$ such that
$g_{t_{n}}\to g_{t}$ (see Remark 2.1) and put

$R_{n}=(R_{g_{t_{n}}(u),g_{t_{n}}(w^{\prime})}+R_{g_{t_{n}}(u^{\prime}),g_{t_{n}}(w)}).$

Since $\frac{\alpha(t_{n})}{t_{n}}\to 0$, $\|R_{n}\|$ and
$\|R_{g_{t_{n}}(u^{\prime}),g_{t_{n}}(w^{\prime})}\|$ are bounded, and

$\frac{\alpha(t_{n})}{t_{n}}\geq\left|t_{n}\|R_{g_{t_{n}}(u^{\prime}),g_{t_{n}}(w^{\prime})}\|-\|R_{n}\|\right|$

we must have that both
$t_{n}\|R_{g_{t_{n}}(u^{\prime}),g_{t_{n}}(w^{\prime})}\|$ and $\|R_{n}\|$
tend to $0$. In particular,

$0=\lim_{n\to\infty}R_{n}=R_{g_{t}(u),g_{t}(w^{\prime})}+R_{g_{t}(u^{\prime}),g_{t}(w)}$

and

$0=\lim_{n\to\infty}R_{g_{t_{n}}(u^{\prime}),g_{t_{n}}(w^{\prime})}=R_{g_{t}(u^{\prime}),g_{t}(w^{\prime})}.$

Then, from (4.1),

$R_{\tau_{t}(u),\tau_{t}(w)}=(\nabla[Y,Z])_{\gamma_{v}(t)}+C(t)$

Since $(\nabla[Y,Z])_{\gamma_{v}(t)}$ and $C(t)$ are parallel along
$\gamma_{v}(t)$ one concludes that $R$ is parallel along $\gamma(t)$ and so
$\nabla_{v}R=0$. ∎

######  Remark 4.2.

Let $p\in M$ and let $X$ be a Killing field. Let $B=(\nabla X)_{p}$ and let
$T$ be a tensor. Since the Levi-Civita connection is torsion free one has that
$\nabla_{X_{p}}T-(\mathcal{L}_{X}T)_{p}=B.T_{p}$, where $\mathcal{L}_{X}$ is
the Lie derivative along $X$ and $B$ acts as a derivation on the algebraic
tensor $T_{p}$. If $T$ is invariant under isometries, or more generally under
the flow of $X$, then $\mathcal{L}_{X}T=0$. Hence $\nabla_{X_{p}}T=B.T_{p}$.
In particular, if $T=R$ is the curvature tensor, then
$\nabla_{X_{p}}R=B.R_{p}$ (cf. [CO, Section 2]). Assume now that $M$ is
homogeneous with nullity $\nu$ and that $X_{p}\in\nu_{p}$. Then, by Lemma 4.1,
$B.R_{p}=0$ or, equivalently, $\mathrm{e}^{tB}(R_{p})=R_{p}$. ∎

We will now introduce some notation that we will keep for the rest of this
section.

First, recall that one can identify any Killing field Y with its initial
conditions $(Y)^{p}=(Y_{p},(\nabla Y)_{p})$ at $p$. Under this identification
the Lie bracket is given by (2.2) and one can think of $\mathcal{K}^{G}(M)$ as
a linear subspace of $T_{p}M\oplus\mathfrak{so}(T_{p}M)$.

On the other hand, recall that if $g$ be an isometry of $M$ and $Y$ is a
Killing field of $M$, then $g_{*}(Y)$ is also a Killing field, where
$(g_{*}(Y))_{q}:=\mathrm{d}g(Y_{g^{-1}(q)})$.

Let now $\gamma_{v}(t)$ be a homogeneous geodesic with $v\in\nu_{p}$ and let
$X\in\mathcal{K}^{G}(M)$ be such that $\gamma_{v}(t)=\phi_{t}(p)$, where
$\phi_{t}$ is the flow associated to $X$.

Let us consider, for $Z\simeq(u,C)$ in $\mathcal{K}^{G}(M)\subset
T_{p}M\oplus\mathfrak{so}(T_{p}M)$, the one parameter family
$Z^{t}\in\mathcal{K}^{G}(M)$ given by

$Z^{t}:=(\phi_{-t})_{*}(Z).$

The map $Z\to Z^{t}$ is a one parameter group of automorphisms of the Lie
algebra $\mathcal{K}^{G}(M)$ and, as it is well-known
$\frac{\mathrm{d}\,}{\mathrm{d}t}_{|0}Z^{t}=[X,Z].$

From (2.5) and Lemma 2.3 one has that, if $B=(\nabla X)_{p}$, then

(4.5)
$(Z^{t})^{p}=(\mathrm{e}^{-tB}u+t\mathrm{e}^{-tB}Cv,\mathrm{e}^{-tB}C\mathrm{e}^{tB}).$

The following result is crucial for our purposes.

######  Lemma 4.3.

We keep the previous notation and assumptions. Let

$h_{t}:T_{p}M\oplus\mathfrak{so}(T_{p}M)\to T_{p}M\oplus\mathfrak{so}(T_{p}M)$

be the one parameter group of linear isomorphisms given by

$h_{t}(u,C)=(\mathrm{e}^{-tB}u,\mathrm{e}^{-tB}C\mathrm{e}^{tB}).$

Then $h_{t}(\mathcal{K}^{G}(M))\subset\mathcal{K}^{G}(M)$ and
$h_{t}:\mathcal{K}^{G}(M)\to\mathcal{K}^{G}(M)$ is a Lie algebra automorphism,
where $\mathcal{K}^{G}(M)$ is identified with a linear subspace of
$T_{p}M\oplus\mathfrak{so}(T_{p}M)$. Moreover,
$h_{t}(\mathcal{K}_{p}^{G}(M))\subset\mathcal{K}_{p}^{G}(M)$ where
$\mathcal{K}_{p}^{G}(M)$ is the isotropy algebra at $p$.

###### Proof.

Fix $t\in\mathbb{R}$ and let $t_{k}$ be sequence that tends to $+\infty$ and
such that $\mathrm{e}^{-t_{k}B}$ tends to $\mathrm{e}^{-tB}$ (see Remark 2.1).
Let $Z\simeq(u,C)\in\mathcal{K}^{G}(M)$. Then by (4.5)
$\frac{1}{t_{k}}(Z^{t_{k}})^{p}$ converges to $(\mathrm{e}^{-tB}Cv,0)$. Hence
$(t\mathrm{e}^{-tB}Cv,0)\in\mathcal{K}^{G}(M)$ (observe that this is a
transvection at $p$). Then, since $Z^{t}\in\mathcal{K}^{G}(M)$, one obtains
that

$h_{t}(Z)=Z^{t}-(t\mathrm{e}^{-tB}Cv,0)\in\mathcal{K}^{G}(M)$

and hence $h_{t}(\mathcal{K}^{G}(M))\subset\mathcal{K}^{G}(M)$. The fact that
$[h_{t}(Y),h_{t}(Z)]=h_{t}([Y,Z])$, for any $Y,Z\in\mathcal{K}^{G}(M)$,
follows from a direct computation, using (2.2) and the fact that
$\mathrm{e}^{-tB}(R_{p})=R_{p}$ (see Lemma 4.1 and Remark 4.2).

The last assertion is trivial since the isotropy algebra corresponds to those
pairs with first component equals to zero. ∎

######  Lemma 4.4.

We keep the notation and assumptions of this section. Let
$X\in\mathcal{K}^{G}(M)$ be such that $\gamma(t)=\phi_{t}(p)$ is a
(homogeneous) geodesic tangent to the the nullity, where $\phi_{t}$ is the
flow associated to $X$. If $M$ is simply connected, then there exists a
transvection at $p$ with initial conditions $(X_{p},0)$.

###### Proof.

We may assume that $G$ is simply connected. In this case the action of $G$ is
not necessarily effective, but almost effective. Observe that, since $M$ is
simply connected, then $H:=G_{p}$ must be connected. Then the one parameter
group of Lie algebra automorphisms $h_{t}$ induces a one parameter group of
automorphisms $L_{t}$ of $G$, with $\mathrm{d}_{e}L_{t}=h_{t}$. Moreover,
$L_{t}(H)=H$. Then $L_{t}$ induces a one parameter group of diffeomorphisms
$\ell_{t}$ of $M=G/H$ given by

$\ell_{t}(gH)=L_{t}(g)H.$

Observe that $\ell_{t}(p)=p$ for all $t\in\mathbb{R}$. Let
$Z=(u,C)\in\mathcal{K}^{G}(M)\simeq\mathfrak{g}$. Then

(4.6)
$\begin{split}\mathrm{d}_{p}\ell_{t}(u)&=\mathrm{d}_{p}\ell_{t}(Z.p)=\tfrac{\mathrm{d}\,}{\mathrm{d}s}_{0}\ell_{t}(\mathrm{Exp}(sZ)p)\\\
&=\tfrac{\mathrm{d}\,}{\mathrm{d}s}_{0}L_{t}(\mathrm{Exp}(sZ))p=h_{t}(Z).p=\mathrm{e}^{-tB}u.\end{split}$

We conclude that $\mathrm{d}_{p}\ell_{t}$ is a linear isometry for all
$t\in\mathbb{R}$. Note that, for all $g\in G$,

$\ell_{t}(x)=L_{t}(g)\ell_{t}(g^{-1}x)$

or, equivalently,

$\ell_{t}={m}_{L_{t}(g)}\circ\ell_{t}\circ{m}_{g^{-1}},$

where ${m}_{r}(x)=rx$, $r\in G$, $x\in M$. Then

$\mathrm{d}_{gp}\ell_{t}=\mathrm{d}_{p}{m}_{L_{t}(g)}\circ\mathrm{d}_{p}\ell_{t}\circ\mathrm{d}_{gp}m_{g^{-1}}.$

So $\mathrm{d}_{gp}\ell_{t}$ is a linear isometry. Since $g\in G$ is
arbitrary, then $\ell_{t}$ is an isometry that fixes $p$.

From (4.6) we obtain that the Killing field associated to the one parameter
group of isometries $\ell_{t}$ is $Y=(0,-B)$. Then $U=X+Y$ is the transvection
with initial conditions $(X_{p},0)$. ∎

######  Corollary 4.5.

Let $M=G/H$ be a simply connected homogeneous Riemannian manifold with a non-
trivial nullity distribution $\nu$. Then, for all $p\in M$, $v\in\nu_{p}$
there exists a transvection $Y$ such that $Y_{p}=v$ (it could happen that
$Y\notin\mathcal{K}^{G}(M)$).

###### Proof.

It follows from Lemma 4.4 and the fact that the directions of homogeneous
geodesics span $\nu_{p}$. ∎

##  5\. The abelian ideal generated by the transvections

Let $M=G/H$ be a locally irreducible Riemannian homogeneous manifold with a
non-trivial nullity. Let

(5.1) $\mathfrak{tr}_{0}^{p}:=\\{X\in\mathcal{K}^{G}(M):X\textrm{ is a
transvection at p with }X_{p}\in\nu_{p}\\}$

(compare with (3.3)) and let

(5.2) $\tilde{\mathfrak{tr}}_{0}^{p}:=\\{X\in\mathcal{K}(M):X\textrm{ is a
transvection at p with }X_{p}\in\nu_{p}\\}.\ \ $

By Corollary 4.5 one has that
$\dim(\tilde{\mathfrak{tr}}_{0}^{p})=\dim(\nu_{p})$. Observe that the
inclusion $\mathfrak{tr}_{0}^{p}\subset\tilde{\mathfrak{tr}}_{0}^{p}$ could be
strict.

######  Remark 5.1.

The adapted transvections at $p$, with respect to $G$, are given by
$[\mathcal{K}^{G}(M),\tilde{\mathfrak{tr}}_{0}^{p}]$. In fact, let
$v\in\nu_{p}$, $X\in\tilde{\mathfrak{tr}}_{0}^{p}$ with $X_{p}=v$ and
$Z\in\mathcal{K}^{G}(M)$. Then $([X,Z])^{p}=(\nabla_{v}Z,0)$ which defines an
arbitrary adapted transvection (see (2.2)). Thus,
$[\mathcal{K}^{G}(M),\tilde{\mathfrak{tr}}_{0}^{p}]\subset\mathcal{K}^{G}(M)$
and hence $\mathcal{K}^{G}(M)+\tilde{\mathfrak{tr}}_{0}^{p}$ is a Lie algebra
of Killing fields of $M$. Observe that this Lie algebra does not depend on
$p\in M$. This follows from the fact that
$\mathrm{Ad}_{g}(\tilde{\mathfrak{tr}}_{0}^{p})=\tilde{\mathfrak{tr}}_{0}^{gp}$
for all $g\in G$.

Throughout the rest of this section we will denote by
$\mathfrak{g}=\mathrm{Lie}(G)\simeq\mathcal{K}^{G}(M)$ and by
$\tilde{\mathfrak{g}}=\mathrm{Lie}(I(M))\simeq\mathcal{K}(M)$. Set

$\mathfrak{g}^{\prime}=\mathfrak{g}+\tilde{\mathfrak{tr}}_{0}^{p}\subset\tilde{\mathfrak{g}}.$

and put

$\tilde{\mathfrak{tr}}^{p}=\\{X\in\mathfrak{g}^{\prime}\,:\,X\ \text{is a
transvection at $p$ with }X_{p}\in\nu^{(1)}_{p}\\}$

From Remark 5.1, $\mathfrak{g}^{\prime}$ is a Lie subalgebra of
$\tilde{\mathfrak{g}}$. On the other hand, from Lemma 2.4 and Remark 5.1, it
follows that $\hat{\nu}_{p}=[\mathfrak{g},\tilde{\mathfrak{tr}}^{p}_{0}].p$
and form Corollary 4.5, $\nu_{p}=\tilde{\mathfrak{tr}}^{p}_{0}.p$. Since
$\nu^{(1)}=\nu+\hat{\nu}$ one gets that

(5.3)
$\tilde{\mathfrak{tr}}^{p}=[\mathfrak{g},\tilde{\mathfrak{tr}^{p}_{0}}]+\tilde{\mathfrak{tr}}^{p}_{0}$

and

(5.4) $\tilde{\mathfrak{tr}}^{p}.p=\nu^{(1)}_{p}.$

The last equality implies, since a transvection at $p$ is defined by its value
at $p$, that (5.3) does not depend on $G$ and hence

(5.5)
$\tilde{\mathfrak{tr}}^{p}=[\mathfrak{g}^{\prime},\tilde{\mathfrak{tr}^{p}_{0}}]+\tilde{\mathfrak{tr}}^{p}_{0}=[\tilde{\mathfrak{g}},\tilde{\mathfrak{tr}^{p}_{0}}]+\tilde{\mathfrak{tr}}^{p}_{0}.$

Let $\mathfrak{a}$ be the ideal generated by $\tilde{\mathfrak{tr}}_{0}^{p}$
in $\mathfrak{g}^{\prime}$.This ideal does not depend on $p\in M$ since any
$g\in G$ maps $\nu_{p}$ into $\nu_{gp}$ and so it maps
$\tilde{\mathfrak{tr}}_{0}^{p}$ into $\tilde{\mathfrak{tr}}_{0}^{gp}$. Observe
that with the same arguments of Theorem 3.4 and Remark 3.5 one gets that
$\mathfrak{a}$ is an abelian ideal.

######  Lemma 5.2.

The abelian ideal $\mathfrak{a}$ does not depend on the presentation group
$G$. In particular, $\mathfrak{a}$ is also the ideal of $\tilde{\mathfrak{g}}$
generated by $\tilde{\mathfrak{tr}}^{p}_{0}$.

###### Proof.

It suffices to show that the ideal $\bar{\mathfrak{a}}$ of
$\tilde{\mathfrak{g}}\simeq\mathcal{K}(M)$ generated by
$\tilde{\mathfrak{tr}}_{0}^{p}$ coincides with $\mathfrak{a}$. Let us denote,
for a pair of Lie subalgebras $\mathfrak{b}$ and $\mathfrak{c}$ of a Lie
algebra, $\ [\mathfrak{c},\mathfrak{b}]^{0}=\mathfrak{b}$ and inductively
$[\mathfrak{c},\mathfrak{b}]^{k+1}=[\mathfrak{c},[\mathfrak{c},\mathfrak{b}]^{k}]$.
We need to show that for each $r\in\mathbb{N}$,

(5.6)
$\sum_{k=0}^{r}[\mathfrak{g}^{\prime},\tilde{\mathfrak{tr}_{0}}^{p}]^{k}=\sum_{k=0}^{r}[\tilde{\mathfrak{g}},\tilde{\mathfrak{tr}}_{0}^{p}]^{k}.$

Let us write

(5.7) $\tilde{\mathfrak{g}}=\mathfrak{g}^{\prime}\oplus\mathfrak{h}^{\prime},$

where $\mathfrak{h}^{\prime}$ is a linear subspace of the full isotropy
algebra $\tilde{\mathfrak{g}}_{p}$. We shall prove first that

(5.8)
$\left[\mathfrak{h}^{\prime},\sum_{k=0}^{r}[\tilde{\mathfrak{g}},\tilde{\mathfrak{tr}^{p}_{0}}]^{k}\right]\subset\sum_{k=0}^{r}[\tilde{\mathfrak{g}},\tilde{\mathfrak{tr}^{p}_{0}}]^{k}$

Since the isotropy at $p$ maps $\nu_{p}$ into itself and transvections into
transvections one has that
$[\mathfrak{h}^{\prime},\tilde{\mathfrak{tr}_{0}^{p}}]\subset\tilde{\mathfrak{tr}_{0}^{p}}$
and so (5.8) holds for $r=0$. Assume now that (5.8) is true for some $r$. Then

$\displaystyle\left[\mathfrak{h}^{\prime},[\tilde{\mathfrak{g}},\tilde{\mathfrak{tr}^{p}_{0}}]^{r+1}\right]$
$\displaystyle=$
$\displaystyle\left[\mathfrak{h}^{\prime},[\tilde{\mathfrak{g}},[\tilde{\mathfrak{g}},\tilde{\mathfrak{tr}^{p}_{0}}]^{r}]\right]=\left[[\mathfrak{h}^{\prime},\tilde{\mathfrak{g}}],[\tilde{\mathfrak{g}},\tilde{\mathfrak{tr}^{p}_{0}}]^{r}]\right]+\left[\tilde{\mathfrak{g}},[\mathfrak{h}^{\prime},[\tilde{\mathfrak{g}},\tilde{\mathfrak{tr}^{p}_{0}}]^{r}]\right]$
$\displaystyle\subset$
$\displaystyle[\tilde{\mathfrak{g}},[\tilde{\mathfrak{g}},\tilde{\mathfrak{tr}^{p}_{0}}]^{r}]+\left[\tilde{\mathfrak{g}},\sum_{k=0}^{r}[\tilde{\mathfrak{g}},\tilde{\mathfrak{tr}^{p}_{0}}]^{k}\right]\subset\sum_{k=0}^{r+1}[\tilde{\mathfrak{g}},\tilde{\mathfrak{tr}^{p}_{0}}]^{k}.$

This implies that (5.8) holds for every $r\in\mathbb{N}$.

Let us turn to the proof of (5.6). It trivially holds for $r=0$, and from
equation (5.5), one gets that it holds for $r=1$. So assume that (5.6) is true
for some $r$. In particular,

$[\tilde{\mathfrak{g}},\tilde{\mathfrak{tr}^{p}_{0}}]^{r}\subset\sum_{k=0}^{r}[\tilde{\mathfrak{g}},\tilde{\mathfrak{tr}}_{0}^{p}]^{k}=\sum_{k=0}^{r}[\mathfrak{g}^{\prime},\tilde{\mathfrak{tr}_{0}}^{p}]^{k}$

From this and from (5.8) one gets that

$[\tilde{\mathfrak{g}},\tilde{\mathfrak{tr}^{p}_{0}}]^{r+1}=[\mathfrak{g}^{\prime}+\mathfrak{h}^{\prime},[\tilde{\mathfrak{g}},\tilde{\mathfrak{tr}^{p}_{0}}]^{r}]\subset\sum_{k=0}^{r+1}[\mathfrak{g}^{\prime},\tilde{\mathfrak{tr}_{0}}^{p}]^{k}$

which implies the non-trivial inclusion on (5.6). ∎

###### Proof of Theorem 1.1.

Part (1) is just Corollary 4.5. Part (2) follows from Remark 5.1. Part (3)
follows from Lemma 5.2 and its preceding paragraph.

Let us prove (4): by (2.8) and (2.9) one has that $\nu^{(1)}=\nu+\hat{\nu}$
and

$\nu^{(2)}=\nu^{(1)}+\hat{\hat{\nu}}$

where
$\hat{\hat{\nu}}_{q}=\mathrm{span}\,\\{\nabla_{v}Z:Z\in\mathcal{K}^{G}(M),v\in\hat{\nu}_{q}\\}$.

Observe that, from (5.4), $\nu^{(1)}_{p}=\tilde{\mathfrak{tr}}^{p}.p$ and from
(5.5), $\tilde{\mathfrak{tr}}^{p}\subset\mathfrak{a}$. So
$\nu^{(1)}_{p}\subset\mathfrak{a}.p$. On the other hand, from Lemma 2.4 and
remark 5.1, any element $v\in\hat{\nu}_{p}$ is the initial value of a
transvection $X\in[\mathfrak{g},\tilde{\mathfrak{tr}}^{p}_{0}]$. Then for each
$Z\in\mathfrak{g}\simeq\mathcal{K}^{G}(M)$ one has that
$\nabla_{v}Z=\nabla_{X_{p}}Z=[X,Z]_{p}$. Hence
$\hat{\hat{\nu}}_{p}\subset[\mathfrak{g},[\mathfrak{g},\tilde{\mathfrak{tr}}^{p}_{0}]].p\subset\mathfrak{a}.p$
and so $\nu^{(2)}_{p}\subset\mathfrak{a}.p$. This proves the first assertion
of (4).

Let $a\in\bar{A}$. One has that
$\tilde{\mathfrak{tr}}^{ap}=\mathrm{Ad}_{a}(\tilde{\mathfrak{tr}}^{p})$
(resp.,
$\tilde{\mathfrak{tr}}_{0}^{ap}=\mathrm{Ad}_{a}(\tilde{\mathfrak{tr}}_{0}^{p})$).
Then, since $\bar{A}$ is abelian,
$\tilde{\mathfrak{tr}}^{ap}=\tilde{\mathfrak{tr}}^{p}$
(resp.,$\tilde{\mathfrak{tr}}_{0}^{ap}=\tilde{\mathfrak{tr}}_{0}^{p}$) for all
$a\in\bar{A}$. Observe that the distribution $\nu^{(1)}$ (resp., $\nu$),
restricted to $\bar{A}\cdot p$, is spanned by the elements $X$ of
$\tilde{\mathfrak{tr}}^{p}$ (resp., $\tilde{\mathfrak{tr}}_{0}^{p}$). Since
$\nabla X=0$ when restricted to $\bar{A}.p$, we obtain the second part of (4).
∎

###### Proof of Corollary 1.2.

The isotropy $H$ is trivial by Theorem A (3) of [DOV]. Observe that, with the
same argument, any presentation group of $M$ must have trivial isotropy. So,
in this case, $\mathfrak{g}=\mathfrak{g}^{\prime}=\tilde{\mathfrak{g}}$.

Let $\mathfrak{a}$ be the abelian ideal of $\mathfrak{g}$ generated by
$\tilde{\mathfrak{tr}}^{p}_{0}$ (see part (3) of Theorem 1.1). Since $M$ is
non-flat, $\mathfrak{a}.p$ is properly contained in $T_{p}M$. From Theorem A
(1) of [DOV] one has that $\nu^{(2)}_{p}$ has codimension $1$ in $T_{p}M$ (cf.
3.2). Then, from part (4) of Theorem 1.1, $\mathfrak{a}$ has codimension $1$
in $\mathfrak{g}$. If $w\in\mathfrak{g}$ does not belong to $\mathfrak{a}$
then $\mathfrak{g}=\mathbb{R}w\rtimes\mathfrak{a}$. Let $A$ be the Lie normal
subgroup of $G$ associated to $\mathfrak{a}$ and $L$ be the one-dimensional
Lie subgroup of $G$ associated to $\mathbb{R}w$. Let
$\pi:\tilde{A}\simeq\mathbb{R}^{n-1}\to A$ and
$\pi^{\prime}:\tilde{L}\simeq\mathbb{R}\to L$ be the universal cover of $A$
and $L$, respectively. Then the map $m:\tilde{L}\times\tilde{A}\to G$ given by
$m((l,a))=\pi^{\prime}(l)\pi(a)$ is a covering map. Since $G\simeq M$ is
simply connected one has that $m$ is a diffeomorphism. Hence
$L\simeq\mathbb{R}$ and $A\simeq\mathbb{R}^{n-1}$ and thus $G$ is isomorphic
to a semidirect product $\mathbb{R}^{n-1}\rtimes\mathbb{R}$. ∎

We finish this section with a lemma and a remark that will be needed in the
following section.

######  Lemma 5.3.

. Let $M=G/H$ be a (connected) homogeneous Riemannian manifold where $G$ is a
connected closed Lie subgroup of $I^{o}(M)$. Let $N$ be a connected closed
abelian normal subgroup of $G$ and let $N\cdot p\subset M$ be any orbit. Then
the isotropy $N_{p}$ is trivial.

###### Proof.

Since $N$ is abelian, then $N=T^{k}\times\mathbb{R}^{m}$ where $T^{k}$ is a
torus. Since $N$ is a normal subgroup and $G$ is connected the conjugation by
$g\in G$ $I_{g}:N\to N$ acts trivially on $T^{k}$. Thus, $T^{k}$ is included
in the center $C(G)$ of $G$. Since $N_{p}$ is compact, $N_{p}\subset T^{k}$.
Then $N_{p}\subset C(G)$ and hence $N_{p}=\\{1\\}$ since $G$ acts effectively.
∎

######  Remark 5.4.

We have the following infinitesimal version of Lemma 5.3. Let $\mathfrak{a}$
be an abelian ideal of $\mathfrak{g}=\mathrm{Lie}(G)$, where $G$ acts, by
isometries, almost effectively and transitively on $M$. Then
$\mathfrak{a}\cap\mathfrak{g}_{p}=\\{0\\}$ for all $p\in M$, where
$\mathfrak{g}_{p}$ is the isotropy algebra at $p$. This can be proved as
follows: the Killing form of $\mathfrak{g}$ is null when restricted to
$\mathfrak{a}$ and it is negative definite when restricted to the isotropy
algebra (see the proof of Lemma 2.7 of [DOV]).

##  6\. Non-solvable examples

Let $K$ be a simple simply connected compact Lie group with Lie algebra
$\mathfrak{k}$ and let $\rho:K\to SO_{n}$ be an irreducible orthogonal
representation. Let us consider the semidirect product
$G=\mathbb{R}^{n}\rtimes_{\rho}K$. Namely $G=\mathbb{R}^{n}\times K$ and
$(v,k)(v^{\prime},k^{\prime})=(v+\rho(k)(v^{\prime}),kk^{\prime})$. Observe
that the Lie algebra of $G$ is
$\mathfrak{g}=\mathbb{R}^{n}\rtimes_{\rho}\mathfrak{k}$, which is the vector
space $\mathbb{R}^{n}\oplus\mathfrak{k}$ with the following Lie bracket
$[\,,\,]$: if $v,w\in\mathbb{R}^{n}$ and $X,Y\in\mathfrak{k}$ then

$[v,w]=0,\ \ [X,Y]=[X,Y]_{\mathfrak{k}},\ \ [X,v]=d\rho_{e}(X)(v).$

Let us consider the simply connected Riemannian manifold $M=G$, with a left
invariant metric which restricted to $\mathbb{R}^{n}$ coincides with the
canonical metric of $\mathbb{R}^{n}$. Then for each
$w\in\mathbb{R}^{n}\subset\mathfrak{g}$, the Killing vector field of $M$
induced by $w$ is

$\tilde{w}_{(v,k)}=(w,0_{k}).$

Moreover, from (2.4), and the fact that $\rho$ is an orthogonal
representation, one obtains that for each $v,w\in\mathbb{R}^{n}$

(6.1) $\nabla_{\tilde{v}}\tilde{w}=0.$

Let $\mathcal{D}$ be the (integrable) distribution of $M$ defined at $p=(v,k)$
by $\mathcal{D}_{p}=(\mathbb{R}^{n},0_{k})$. It is not hard to prove that
$\mathcal{D}$ is left-invariant, and from equation (6.1) one obtains that
$\mathcal{D}$ is an autoparallel distribution.

######  Remark 6.1.

The only proper ideal of $\mathfrak{g}$ is $\mathbb{R}^{n}$. In fact, consider
the projection
$\pi:\mathfrak{g}\to\mathfrak{g}/\mathbb{R}^{n}\simeq\mathfrak{k}$ and let
$\mathfrak{J}$ be a proper ideal of $\mathfrak{g}$. Then $\pi(\mathfrak{J})$
is an ideal of this quotient, and since $\mathfrak{k}$ is simple, either
$\pi(\mathfrak{J})=\\{0\\}$ or $\pi(\mathfrak{J})\simeq\mathfrak{k}$. If
$\pi(\mathfrak{J})=0$, then $\mathfrak{J}\subset\mathbb{R}^{n}$. But then
$\mathfrak{J}$ is a $\mathfrak{k}$-invariant subspace of $\mathbb{R}^{n}$.
Since the action of $K$ on $\mathbb{R}^{n}$ is irreducible one concludes that
$\mathfrak{J}=\mathbb{R}^{n}$. A similar argument shows that if
$\pi(\mathfrak{J})=\simeq\mathfrak{k}$, then $\mathfrak{J}=\mathfrak{g}$.

######  Remark 6.2.

Let $u\in\mathfrak{g}$ and let $\tilde{u}$ be the Killing field induced by
$u$, i.e. $\tilde{u}_{q}=u.q$. Then, if $g$ is an isometry,
$g_{*}(\tilde{u})=\textrm{Ad}_{g}(u)^{\tilde{\,}}$. In particular, $u$ is a
transvection at $q$ if and only if $g_{*}(\tilde{u})$ is a transvection at
$g(q)$.

######  Lemma 6.3.

If $M$ has a non-trivial Euclidean de Rham factor then $\mathcal{D}$ is a
parallel distribution and the Killing fields induced by
$\mathbb{R}^{n}\simeq\mathrm{Lie}(\mathbb{R}^{n})$ are parallel fields.
Furthermore, $M$ is isometric to the Riemannian product of the Euclidean space
$\mathbb{R}^{n}$ by the Lie group $K$ with a left invariant metric.

###### Proof.

Let $M=E\times M^{\prime}$ where $E\simeq\mathbb{R}^{r}$ is the Euclidean de
Rham factor and $M^{\prime}$ is the product of the irreducible de Rham
factors. Then the space of Killing fields of $M$ decomposes as

(6.2)
$\mathcal{K}(M)=\mathcal{K}(\mathbb{R}^{r})\oplus\mathcal{K}(M^{\prime})$

and the full isometry group of $M$ decomposes as

(6.3) $I(M)=I(E)\times I(M^{\prime}).$

Let $\mathcal{E}$ be the parallel distribution of $M$ associated to the flat
de Rham factor. Then its first osculating distribution $\mathcal{E}^{(1)}$
(see equation 2.7) coincides with $\mathcal{E}$ and so

(6.4) $\nabla_{\mathcal{E}_{p}}X\subset\mathcal{E}_{p}$

for all $X\in\mathcal{K}^{G}(M)\simeq\mathfrak{g}=\mathrm{Lie}(G)$, $p\in M$.
The same argument used in [DOV, Proposition 3.6] shows that there exists
always a transvection induced by $G$ in the direction of $\nabla_{v}X$, for
all $X\in\mathcal{K}^{G}(M)$, $v\in\mathcal{E}_{p}$ (the argument used in the
proof of such a proposition does not use that there is no flat de Rham factor,
except for finding a transvection not in the nullity).

Case (i): $\nabla_{v}X=0$, for all $X\in\mathcal{K}^{G}(M)$,
$v\in\mathcal{E}_{p}$. Then, if $X\in\mathcal{K}^{G}(M)$, $X=T+Z$ where $T$ is
a transvection of the Euclidean factor and $Z$ is a Killing field of
$M^{\prime}$ (by enlarging $G$ to the full isometry group and with the usual
identification 6.2). Then, if $X_{p}\perp\mathcal{E}_{p}$, for some $p\in M$,
the field $X$ must be always tangent to the distribution $\mathcal{E}^{\perp}$
associated to the factor $M^{\prime}$. Let us regard $K\subset I(M)$ and
consider the projection $\pi:K\to I(E)$ according to the decomposition 6.3.
Then $\pi(K)$ is a compact subgroup of isometries of $E\simeq\mathbb{R}^{r}$.
Then there exists $q\in E$ which is fixed by $\pi(K)$ or equivalently, since
$\pi(K)$ is connected, $u.q=0$ for all $u\in\mathrm{Lie}(\pi(K))$. We may
regard $E$ as a fixed integral manifold of $\mathcal{E}$ and so we will regard
$q$ as an element of $M$. Then $\mathfrak{k}.q\perp\mathcal{E}_{q}$ where
$\mathfrak{k}=\mathrm{Lie}(K)$. Then, from what previously observed, any
Killing field $X$ induced by an element of $\mathfrak{k}$ lies in
$\mathcal{E}^{\perp}$. Since $\mathcal{E}^{\perp}$ is $G$-invariant, the
Killing fields induced by $G$ which lie in this distribution are an ideal, let
us say $\mathfrak{I}$, of $\mathfrak{g}$. Since
$\mathfrak{k}\subset\mathfrak{I}$, from remark 6.1 we conclude that
$\mathfrak{I}=\mathfrak{g}$ and thus $\mathcal{E}=\\{0\\}$. A contradiction
that show that this case cannot hold if $E\neq\\{0\\}$.

Case (ii): $\nabla_{v}X\neq 0$, for some $X\in\mathfrak{g}$ and some
$v\in\mathcal{E}_{p}$. Since, by (6.4), $\nabla_{v}X\in\mathcal{E}_{p}$ we
conclude that there exists a non-trivial transvection $Y$ induced by $G$ such
that $Y_{p}\in\mathcal{E}_{p}$. Such a transvection must be always tangent to
the Euclidean factor of $M$ and must be a transvection at any point. Then, for
any $g\in G$, $g_{*}(Y)$ lies in $\mathcal{E}$ and it is a transvection at any
point of $M$ (see Remark 6.2). Then the ideal $\mathfrak{J}$ of $\mathfrak{g}$
generated by $Y$ consists of transvections at any point that lie in
$\mathcal{E}$. Since such transvections must commute we conclude that
$\mathfrak{J}$ is a non-trivial abelian ideal of $\mathfrak{g}$. Thus by
Remark 6.1, $\mathfrak{J}=\mathbb{R}^{n}$. In particular
$r=\dim(\mathcal{E})=n$. Observe that in this case any element $v$ of
$\mathfrak{J}$ induces a parallel vector field $\tilde{v}$ of $M$ which proves
the first assertions. This implies that $\mathcal{D}=\mathcal{E}$.

Let us consider again the natural projection $\pi:K\to I(E)$. Then, as in the
proof of case (i), $\mathfrak{k}.q\perp\mathcal{E}_{q}=\mathcal{D}_{q}$. Then,
since $G$ acts simply transitively on $M$ we must have that $K$ act simply
transitively on the integral manifold by $q$ of the parallel distribution
associated to the factor $M^{\prime}$ of $M$. From this we conclude that
$M^{\prime}$ is isometric to $K$ with a left invariant metric. ∎

######  Lemma 6.4.

Let $M=\mathbb{R}^{n}\rtimes_{\rho}K$ be a semidirect product with a left
invariant metric which restricted to $\mathbb{R}^{n}$ coincides with the
canonical metric of $\mathbb{R}^{n}$, where $\rho:K\to\mathbb{S}O_{n}$ is an
irreducible orthogonal representation and $K$ is a simply connected compact
simple Lie group with $\dim(K)<n$. Assume that $M$ has no Euclidean de Rham
factor. Then $M$ is an irreducible Riemannian manifold.

###### Proof.

Let $\mathfrak{g}=\mathbb{R}^{n}\rtimes_{\mathrm{d}\rho}\mathfrak{k}$ be the
Lie algebra of $G=\mathbb{R}^{n}\rtimes_{\rho}K$, let $M_{i}$ be an
irreducible de Rham factor of $M$ and let
$\pi_{i}:\mathfrak{g}\to\mathrm{Lie}(I(M_{i}))$ the the natural projection.
Observe that $\mathfrak{J}:=\ker(\pi_{i})$ is an ideal which is properly
contained in $\mathfrak{g}$. Then from Remark 6.1 either
$\mathfrak{J}=\\{0\\}$ or $\mathfrak{J}=\mathbb{R}^{n}$.

Assume that $\mathfrak{J}=\mathbb{R}^{n}$. Let us denote also by $\pi_{i}$ the
induced Lie group morphism from $G$ into $I(M_{i})^{o}$. Then
$\pi_{i}(G)=\pi_{i}(K)$ is compact. Since $\pi_{i}(G)$ must act transitively
on $M_{i}$, we conclude that $M_{i}$ is compact.

If $\mathfrak{J}=\\{0\\}$ then $\pi_{i}(\mathbb{R}^{n})$ is an $n$-dimensional
ideal of $\pi_{i}(\mathfrak{g})$. By Remark 5.4
$\dim(\pi_{i}(\mathbb{R}^{n}).q)=n$ for any $q\in M_{i}$ and hence
$\dim(M_{i})\geq n$. Since $\dim(M)=n+\dim(K)<2n$ there is at most one of such
factors with $\mathrm{ker}(\pi_{i})=\\{0\\}$. Note that there must be one of
such factors, let us say $M_{1}$, since $M$ in non-compact.

Let $\mathcal{D}$ be the parallel foliation of $M$ associated to $M_{1}$. Let
$M^{\prime}$ be the product of the de Rham factors $M_{i}$ of $M$ different
from $M_{1}$ (and so $\mathrm{ker}(\pi_{i})=\mathbb{R}^{n}$). Assume
$M^{\prime}$ in non-trivial. We identify $M^{\prime}$ with the integral
manifold of $\mathcal{D}^{\perp}$ by $e=(0,1)$. We have shown that any
irreducible de Rham factor of $M^{\prime}$ is compact, and so $M^{\prime}$ is
compact. Let $\pi:G\to I(M^{\prime})^{o}$ be the natural projection. Then
clearly $\ker(\pi)=\mathbb{R}^{n}$ and so, $\pi(G)=\pi(K)$ and $\pi(G)$ is
locally isomorphic to $K$. Since $\mathcal{D}^{\perp}$ is $G$-invariant we
have that $M^{\prime}=H\cdot e$ for some simply connected compact Lie subgroup
$H$ of $K$. Observe, since $G$ acts without isotropy, that $H$ that has no
isotropy and in particular acts effectively on $M^{\prime}$. Observe that
$\pi(G)$ contains the restriction to $M^{\prime}$ of $H$ (that we identify
with $H$ acting by left multiplications on $M^{\prime}$). From [OT] (see also
[Go]) one has that $H$ is a normal subgroup of $I(M)^{o}$. Then $H$ is a
normal subgroup of $\pi(G)$. Since $H\neq\\{1\\}$ and $K$ is simple, we
conclude that $H\simeq\pi(G)$. Then $\dim(M^{\prime})=\dim(K)$ and so
$\dim(M_{1})=n$. Observe, since $\pi(\mathbb{R}^{n})=0$, that the Killing
fields induced by the elements in
$\mathrm{Lie}(\mathbb{R}^{n})\simeq\mathbb{R}^{n}\subset\mathfrak{g}$ lie in
the distribution $\mathcal{D}$. Then, by Remark 5.4, $\mathbb{R}^{n}$ acts
transitively (by isometries) on $M_{1}$. Then $M_{1}$ is flat. A contradiction
that proves the $M^{\prime}$ is trivial. ∎

######  Proposition 6.5.

Let $K$ be a simply connected compact simple Lie group and let $\rho:K\to
SO_{n}$ be an irreducible orthogonal representation. Let $\mathbb{V}_{0}$ be a
non-trivial vector subspace of $\mathbb{R}^{n}$ such that
$\dim(\mathbb{V}_{0})(1+\dim(K))<n$. Then there exists a left invariant metric
$\langle\,,\,\rangle$ on $G=\mathbb{R}^{n}\rtimes_{\rho}K$ such that
$M=(G,\langle\,,\,\rangle)$ is an irreducible Riemannian manifold and the
nullity distribution $\nu$ of $M$ at $(0,e)$ contains $\mathbb{V}_{0}$ (hence
the index of nullity of $M$ is at least $\dim(\mathbb{V}_{0})$).

###### Proof.

Let us define, for $i\geq 1$,

$\mathbb{V}_{i}=[\mathfrak{g},\mathbb{V}_{i-1}]+\mathbb{V}_{i-1}=[\mathfrak{k},\mathbb{V}_{i-1}]+\mathbb{V}_{i-1}.$

Since $[\mathfrak{k},\mathbb{V}_{i}]\subset\mathbb{V}_{i+1}$ and $K$ acts
irreducibly on $\mathbb{R}^{n}$ there exists $d\in\mathbb{N}$ such that
$\mathbb{V}_{d}=\mathbb{R}^{n}$. We choose $d$ to be the minimal with this
property. Observe that
$\dim(\mathbb{V}_{1})\leq\dim(\mathbb{V}_{0})\dim(K)+\dim(\mathbb{V}_{0})=\dim(\mathbb{V}_{0})(1+\dim(K))<n$.
So $\mathbb{V}_{1}$ is a proper subspace of $\mathbb{R}^{n}$ and thus $d\geq
2$. Let $e=(0,1)\in G$, let $(\,,\,)$ be the canonical inner product of
$\mathbb{R}^{n}$ and let $\langle\,,\,\rangle^{\prime}$ be an inner product in
$\mathfrak{k}\simeq\mathfrak{k}.e$. Let $w\in\mathfrak{k}$ and
$v\in\mathbb{V}_{d-1}$ be such that $[w,v]\notin\mathbb{V}_{d-1}$. Let $0\neq
z\in\mathfrak{k}$ be perpendicular to $w$. Let $v^{\prime}$ be the orthogonal
projection of $[w,v]$ into the orthogonal complement
$\mathbb{V}_{d-1}^{\perp}$ of $\mathbb{V}_{d-1}$ in
$\mathbb{V}_{d}=\mathbb{R}^{n}$ (any element of
$\mathbb{R}^{n}\subset\mathfrak{g}$ defines a Killing field which is
identified with its value at $e$). Let $\bar{\mathbb{V}}$ be the orthogonal
complement of $v^{\prime}$ in $\mathbb{V}_{d-1}^{\perp}$. So

$T_{e}G=\mathbb{V}_{d-1}\oplus\overline{\mathbb{V}}\oplus\mathbb{R}v^{\prime}\oplus\mathfrak{k}.$

Let us consider the inner product $\langle\,,\,\rangle$ on $T_{e}G$ defined
by:

1) $\langle\,,\,\rangle=(\,,\,)\times\langle\,,\,\rangle^{\prime}$ when
restricted to
$\big{(}\mathbb{V}_{d-1}\oplus\bar{\mathbb{V}}\big{)}\oplus\mathfrak{k}$.

2) $\langle\,,\,\rangle=(\,,\,)$ when restricted to $\mathbb{R}^{n}\subset
T_{e}G$.

3) $v^{\prime}$ is perpendicular to the orthogonal complement of the linear
span of $w,\,z$ in $\mathfrak{k}$ and $\langle v^{\prime},w\rangle=a$,
$\langle v^{\prime},z\rangle=b$, where $a,b$ are small generic constants, that
will be fixed later, and such that $\langle\,,\,\rangle$ is a positive
definite inner product of $T_{e}G$.

From formula (2.4) one obtains that the elements of $\mathbb{V}_{d-2}$ define
transvections at $e$ (by making use that the bracket by an element of
$\mathfrak{k}$ defines a skew symmetric transformation of $\mathbb{R}^{n}$).

Let us write $[v,z]=\lambda v^{\prime}+u$, where $u\perp v^{\prime}$.
Identifying $x\in\mathfrak{g}$ with the associated Killing field $q\mapsto
x.q$ we have that

(6.5)
$\begin{split}2\langle\nabla_{w.e}v,z.e\rangle&=\langle[w,v].e,z.e\rangle+\langle[w,z].e,v.e\rangle+\langle[v,z].e,w.e\rangle\\\
&=\langle[w,v].e,z.e\rangle+\langle[v,z].e,w.e\rangle=b+\lambda a\end{split}$

(observe that $[v,w]\in\mathfrak{k}$ and $v\in\mathbb{V}_{d-1}$ are
perpendicular). One can choose the generic constants $a$ and $b$ such that
$b+\lambda a\neq 0$. Then the Killing field $q\mapsto v.q$ is not a
transvection at $e$. Then, by Lemma 6.3, $M$ has no Euclidean factor. Hence,
by Lemma 6.4, $M$ is an irreducible Riemannian manifolds. From formula (2.3)
we obtain that $\mathbb{V}_{d-2}.e\subset\nu_{e}$ since the elements of both
$\mathbb{V}_{d-2}$ and
$[\mathfrak{g},\mathbb{V}_{d-2}]\subset\mathbb{V}_{d-1}$ induce transvections.
∎

###### Proof of Theorem 1.3.

If follows directly from Proposition 6.5. ∎

## References

  * [BCO] Berndt, J., Console, S., and Olmos, C, Submanifolds and holonomy, Research Notes in Mathematics Chapman & Hall/CRC, Boca Raton FL , Second Edition 2016.
  * [BOR] Berndt, J., Olmos, C.,and S. Reggiani, Compact homogeneous Riemannian manifolds with low co-index of symmetry, J. Eur. Math. Soc. (JEMS), 19 (2017), 221–254.
  * [CO] Console, S., Olmos, C., Curvature invariants, Killing vector fields and cohomogeneity, Proc. AMS 137 (2009), 1069–1072.
  * [DOV] Di Scala, A. J.,Olmos, C., and Vittone, F., Homogeneous Riemannian manifolds with non-trivial nullity, Transformation Groups, 17 (2022), 31-72
  * [Go] Gordon, C., Riemannian Isometry Groups Containing Transitive Reductive Subgroups, Math, Ann. 248 (1980), 185–192.
  * [GG] Gorodski, C. and Guimarães, F. The $\kappa$-nullity of Riemannian manifolds and their splitting tensors. arXiv preprint arXiv:2204.04551 (2022).
  * [ORT] Olmos, C., Reggiani, and S., Tamaru, H., The index of symmetry of compact naturally reductive spaces, Math. Z. 277 (2014), 611–628.
  * [OS] Olmos, C., and Salvai, M. Holonomy of homogeneous vector bundles and polar representations, Indiana Univ. Math. J. 44 (1995), 1007–1015.
  * [OT] Ochiai, T., and Takahashi, T., The Group of Isometries of a Left Invariant Riemannian Metric on a Lie Group, Math. Ann. 223 (1976), 91–96.

Antonio J. Di Scala |  | Carlos E. Olmos |  | Francisco Vittone
---|---|---|---|---
Dipartimento di Scienze Matematiche |  | FaMAF, CIEM-Conicet |  | DM, ECEN, FCEIA - Conicet
Politecnico di Torino |  | Universidad Nac. de Córdoba |  | Universidad Nac. de Rosario
Corso Duca degli Abruzzi, 24 |  | Ciudad Universitaria |  | Av. Pellegrini 250
10129, Torino, Italy |  | 5000, Córdoba, Argentina |  | 2000, Rosario, Argentina
<EMAIL_ADDRESS>|  |<EMAIL_ADDRESS>|  |<EMAIL_ADDRESS>
http://calvino.polito.it/$\sim$adiscala |  |  |  | www.fceia.unr.edu.ar/$\sim$vittone


# Towards Comprehensive Preference Data Collection for Reward Modeling

Yulan Hu1,2, Qingyang Li2, Sheng Ouyang1,2, Ge Chen2,3, Kaihui Chen2,
Lijun Mei2, Xucheng Ye2, Fuzheng Zhang2, Yong Liu1
1 Renmin University of China, Gaoling School of Artificial Intelligence,
Beijing
<EMAIL_ADDRESS>
2 Kuaishou Technology, Beijing
<EMAIL_ADDRESS>
3 University of Chinese Academy of Sciences, Beijing
<EMAIL_ADDRESS>

###### Abstract

Reinforcement Learning from Human Feedback (RLHF) facilitates the alignment of
large language models (LLMs) with human preferences, thereby enhancing the
quality of responses generated. A critical component of RLHF is the reward
model, which is trained on preference data and outputs a scalar reward during
the inference stage. However, the collection of preference data still lacks
thorough investigation. Recent studies indicate that preference data is
collected either by AI or humans, where chosen and rejected instances are
identified among pairwise responses. We question whether this process
effectively filters out noise and ensures sufficient diversity in collected
data. To address these concerns, for the first time, we propose a
comprehensive framework for preference data collection, decomposing the
process into four incremental steps: Prompt Generation, Response Generation,
Response Filtering, and Human Labeling. This structured approach ensures the
collection of high-quality preferences while reducing reliance on human labor.
We conducted comprehensive experiments based on the data collected at
different stages, demonstrating the effectiveness of the proposed data
collection method.

##  1 Introduction

Reinforcement Learning from Human Feedback (RLHF) Ziegler et al. (2019) has
demonstrated significant potential in aligning Large Language Models (LLMs)
with human preferences Ouyang et al. (2022); Touvron et al. (2023b). Within
the RLHF framework, the reward model (RM) outputs a scalar reward for a given
prompt and response, further guiding the reinforcement learning. The reward
model typically relies on collected preference data for training, enabling it
to distinguish between chosen and rejected responses Wang et al. (2024).

Recent years have seen increasing discussions on constructing and improving
reward models (RMs). Notable strategies include employing mixtures of experts
architectures Artetxe et al. (2022) to enhance model robustness, and
ensembling logits Zhang et al. (2024); Eisenstein et al. (2023) or parameters
Ramé et al. (2024) of multiple RMs to mitigate the reward hacking problem
Skalse et al. (2022). These methods refine RMs from a model perspective, while
studies focusing on data aspects are largely overlooked. As revealed in Wang
et al. (2024), the preference data used for reward model training—whether off-
the-shelf or collected by AI or human—often contains noise and may not be
suitable for RM training. Unfortunately, both released technical reports Bai
et al. (2022); Touvron et al. (2023b); Bai et al. (2023); Yang et al. (2023);
DeepSeek-AI et al. (2024) and research studies lack detailed analysis on
collecting high-quality preference data for RM training.

In this paper, we present the first comprehensive study on collecting
preference data for training reward models (RMs). We propose a framework
designed to gather high-quality preference data. Specifically, we decompose
the preference data collection process into four sub-steps: Prompt Generation,
which selects challenging prompts that the SFT model struggles to handle;
Response Generation, which produces diverse responses to enhance the model’s
generalization; Response Filtering, which removes noisy answer pairs; and
Human Labeling, which annotates a modest amount of pseudo preference data.
Finally, the RM is trained on the data reviewed by human labelers.

As an initial attempt to thoroughly investigate preference data collection for
reward models (RMs), the proposed framework consolidates AI filtering with
human intervention. Compared to relying solely on AI or human annotation, this
framework effectively reflects human preferences while significantly reducing
the amount of human labor required. We conducted experiments on preference
data collected at different stages, demonstrating that performance enhancement
is achieved as the quality of the preference data improves. This study bridges
the gap in research on preference data collection for RMs.

##  2 Methodology

Figure 1: The overview of the proposed framework.

In this section, we present the details of the proposed framework. As
illustrated in Figure 1, the framework comprises five hierarchical steps, with
the first four steps dedicated to preference data collection. The first three
steps involve intricate design, while the fourth step is primarily carried out
by annotators. In the subsequent sections, we first present standard RM
training process, followed by a step-by-step deconstruction of the proposed
framework.

###  2.1 RM Training

RLHF Ouyang et al. (2022) focuses on maximizing the reward of generated
samples Ziegler et al. (2019); Ouyang et al. (2022), involving a reward model
(RM) that outputs a scalar reward to evaluate the quality of given text.
Consider a RM $r_{\psi}$ parameterized by $\psi$, where $r_{\psi}$ is
initialized from a SFT model $\pi_{SFT}$ and then trained in a supervised
manner on preference data
$\mathcal{D}=\\{(x_{i},y_{i}^{+},y_{i}^{-})\\}_{i=1}^{N}$. Here, $x_{i}$
represents the prompt, and $y_{i}^{+}$ and $y_{i}^{-}$ are the two responses,
with $y_{i}^{+}$ preferred over $y_{i}^{-}$. After collecting sufficient
preference data, we frame the RM training as a binary classification problem
as follows:

$\begin{split}\mathcal{L}\left(r_{\psi},\mathcal{D}\right)=&-\mathbb{E}_{\left(x,y^{+},y^{-}\right)\sim\mathcal{D}}\\\
&\left[\log\sigma\left(r_{\psi}\left(x,y^{+}\right)-r_{\psi}\left(x,y^{-}\right)\right)\right],\end{split}$
(1)

###  2.2 The proposed framework

Step 1: Prompt Generation.

Figure 2: The prompt generation (Step 1) process.

The prompt generation phase aims to collect sufficient challenging prompts,
which will be used to generate responses for RM training. The model trained
through RLHF should preserve the overall capabilities of the SFT model while
also being able to handle those prompts that are difficult for the SFT model.
To achieve this, two critical issues need to be addressed in prompt
generation. First, the prompt set should exhibit diversity to avoid the barrel
effect, i.e., where the RM scores accurately in one domain but is less precise
in another. Second, the prompt set should include samples that are difficult
for the SFT model to handle.

To address the two issues, we develop a comprehensive strategy illustrated in
Figure 2. We randomly sample a sufficient number of prompts from diverse
categories to form the prompt pool $X=\\{x_{0},x_{1},\ldots,x_{N}\\}$ ,
thereby fulfilling the first requirement. For the latter requirement, our core
premise is that if the quality of the response inferred by the SFT model is
close to that of strong LLM models, i.e., GPT-4, for the same prompt, it
indicates that the SFT model can effectively resolve this prompt, eliminating
the need for further learning in the RLHF stage, we achieve this with the
assist of an off-the-shelf proxy RM, $r^{\star}$. Specifically, we first use
the SFT model $s_{\phi}$ and the GPT-4 model to generate two responses for
each prompt in $X$, acquiring $y^{G}$ and $y^{S}$. Then, we use $r^{\star}$ to
score the $\langle\text{prompt, response}\rangle$ pairs as follows:

$\Delta_{(y^{G},y^{S})}=\left\\{\begin{aligned}
r^{\star}(x,y^{G})-r^{\star}(x,y^{S})\leq\epsilon,&\quad\text{drop}\,x\\\
r^{\star}(x,y^{G})-r^{\star}(x,y^{S})>\epsilon,&\quad\text{keep}\,x\\\
\end{aligned}\right.$ (2)

where $\epsilon$ denotes the preset threshold difference between $y^{G}$ and
$y^{S}$. Equation 2 indicates that we only keep the prompts if the
corresponding response generated by the existing SFT model $s_{\phi}$ is
relatively lagging behind the well-performing models. This approach filters
the "hard" prompt samples from $X$, obtaining the refined prompt set $X^{*}$
for RM training.

Step 2: Response Generation. RM training accepts a prompt $x$ and two
preference responses $(y^{+},y^{-})$ for pairwise learning. To generate
feasible $(y^{+},y^{-})$, similar to step 1, the quality and diversity of the
responses need to be ensured. The primary challenge lies in the implicit
supervisory signal inherent in $(y^{+},y^{-})$, which means that at least
$y^{+}$ should be generated by models that are at least as superior as the
current model being optimized. For instance, consider $\pi_{SFT}$ as a
13B-size SFT model; it is necessary to use a stronger model to generate the
responses, such as a 65B-size or 175B-size SFT model, GPT-4, etc.

Another key requirement lies in the diversity of the responses. We achieve
this by combining results from various models. Under the premise of fulfilling
the first condition, we can combine multiple LLMs with different
configurations, such as different parameter settings and sizes, to generate
$(y^{+},y^{-})$. Additionally, off-the-shelf strong models can also be
employed as a supplement.

Step 3: Response Filtering. In step 2, we generate multiple pairwise responses
for each prompt, forming the training candidate set $\mathcal{D}$, with each
training instance formulated as a triad $\langle x,y^{+},y^{-}\rangle$.
Ideally, $y^{+}$ should exhibit a certain degree of superiority over $y^{-}$,
meaning that $\langle x,y^{+},y^{-}\rangle$ should not be too easy or too hard
for pairwise learning. However, such a condition cannot always be fulfilled.
For instance, the responses to an objective question may be identical,
offering no supervisory signal for RM training while conversely introducing
additional labeling overhead for annotators.

Thus, refinement is necessary before sending $\mathcal{D}$ to the annotators.
We incorporate GPT-4 to help filter out useless training samples.
Specifically, we score each instance in $\mathcal{D}$ using the in-context
learning technique Dong et al. (2023b). We divide the scoring criteria for
each instance into five levels, where 1 represents the worst response quality
and 5 represents the best, we then employ GPT-4 to score $x,y^{+}$ and
$x,y^{-}$, respectively. It is worth noting that since the prompts belong to
different categories, the corresponding scoring criteria for prompts of
different categories are also different.

After scoring, we obtain score pairs as $\langle x,y^{+},r^{+}\rangle$ and
$\langle x,y^{-},r^{-}\rangle$. Based on these scores, we select the pairs
that exhibit certain differences for the annotators to further review. We
build a filtering strategy presented in Table 1, where we consider two kinds
of samples that should be discarded. First, responses with identical scores,
as such pairs cannot provide any discriminative knowledge during RM training.
Second, responses that exhibit extreme distinctness, for example, pairs where
$r^{+}$ is scored 5 and $r^{-}$ is scored 1. We consider that the RM possesses
the ability to distinguish samples with significant divergence, thus
eliminating the need for further assessment by the annotators. These two kinds
of hollow samples occupy a large portion of $\mathcal{D}$, and filtering them
enhances labeling efficiency to a large extent.

$<r^{+},r^{-}>$ | 1 | 2 | 3 | 4 | 5
---|---|---|---|---|---
1 | 1-1 | 1-2 | 1-3 | 1-4 | 1-5
2 | 2-1 | 2-2 | 2-3 | 2-4 | 2-5
3 | 3-1 | 3-2 | 3-3 | 3-4 | 3-5
4 | 4-1 | 4-2 | 4-3 | 4-4 | 4-5
5 | 5-1 | 5-2 | 5-3 | 5-4 | 5-5
Table 1: The responses filtering scoring matrix is formulated in the shape of
$5\times 5$. The scoring pairs highlighted in  green are preserved, while
those in  grey are discarded.

###  2.3 Data Funnel

In step 4, the annotators further review and score the filtered training
samples from step 3. The ultimately reviewed results will be used for RM
training in step 5.

As a hierarchical RM data generation method, the proposed framework involves
multiple filtering strategies. Consequently, a data funnel exists between each
pair of steps, meaning that not all data from the previous step will be fully
transferred to the next step. We illustrate the practical data funnel in
Figure 3. Consider that the initial number of candidate prompts is $N$. The
loss rate of the prompt filter (Step 1) is relatively small, around 10%, while
the loss rate for Step 3 is comparatively large, nearly 60%. Finally, after
human labeling, we discard nearly half of the labeled samples from step 3 that
are not appropriate for training. Overall, around 20% of the prepared training
samples are filtered for RM training.

Figure 3: The data funnel illustrates the loss rate at each step. The ultimate
data retention is roughly 20%, meaning that only 20% of the prepared samples
are qualified for training.

##  3 Experiment

###  3.1 Setups

We employ two SFT models of varying sizes (13B and 65B) as the base model, the
basic architecture are built upon LLaMA Touvron et al. (2023a). The overall
preference data are collected from two sources: the available open-source
preference data and the data inferred by AI or human, each accounts for about
30w. To reveal the effectiveness and necessity of different steps of the
proposed framework, we follow the prompt preparation step to collect roughly
15w refined prompts, then proceed with the proposed preference data collection
procedure. Specifically, we train two RMs based on the data collected at step
3 and step 4, respectively. Roughly, around 5.4w preference data is collected
in step 3 and around 3w preference data are finally used in step 4.

We save the checkpoint of the last iteration for evaluation. We evaluate the
two RMs from two aspects, the preference benchmarks and the overall
performance of the RMs incorporation with downstream policy. We follow the
preference benchmark used in Touvron et al. (2023b); Bai et al. (2023),
containing Anthropic Helpfulness Bai et al. (2022), OpenAI Summarize Stiennon
et al. (2020), OpenAI WebGPT Nakano et al. (2021) and Standford SHP Ethayarajh
et al. (2022).

###  3.2 Results

Results on preference benchmarks. We report the results on preference
benchmarks in Table 2, using accuracy as the evaluation metric. The results
for both the 13B-size and 65B-size RMs validate the improvement from step 3 to
step 4, indicating that refinement of preference data can indeed boost
performance. Despite the scale of preference data used in step 3 being almost
twice that used in step 4, we observe that the refinement of data quality is
beneficial.

RMs |  | Anthropic
---
Helpful
| OpenAI
---
Summ.
| Stanford
---
SHP
WebGPT | Overall
13B | RM-Step3 | 68.7 | 68.2 | 67.2 | 65.9 | 67.5
RM-Step4 | 69.6 | 68.6 | 68.1 | 66.7 | 68.3
65B | RM-Step3 | 69.9 | 68.7 | 71.5 | 71.4 | 70.4
RM-Step4 | 71.4 | 71.4 | 72.1 | 70.8 | 71.4

Table 2: The results on preference benchmarks.

Results of Best-of-N experiments. In addition, we integrate the trained RMs
with the BoN reranking policy Dong et al. (2023a). BoN is an inference-time
sampling strategy that aims to select the answer with the highest reward from
$n$ candidates, usually generated by the SFT model $\pi_{SFT}$. The gains
obtained by BoN are approximated by $\text{log}(N)-\frac{N-1}{N}$ Beirami et
al. (2024). Our BoN experiments are conducted on AlignBench Liu et al. (2023).
For each prompt in AlignBench, we use the SFT model to generate $n$ answers
and choose the best answer from the answer set based on the RM score. The
value of $n$ is chosen from {5, 10, 20, 50}. We then calculate the win rate
for the RM trained on preference data collected in step 3 against step 4, and
plot the results in Figure LABEL:fig:bon_res. The results validate that the
reward models consistently help select better answers than the raw SFT model
for both the 13B and 65B models, further verifying the enhancement in
performance with the refinement of preference data.

##  4 Conclusion

In this paper, we conduct a thorough inspection and develop a framework for
the collection of preference data for RM training. Specifically, we decompose
the process into several sub-steps, which facilitates the collection of high-
quality data while reducing the labor required from humans. We validate the
framework using both preference data benchmarks and policy learning, with
results demonstrating improvements in data quality brought about by the
framework. As an initial attempt, we believe the proposed framework bridges
the gap in comprehensive preference data collection within the LLM community.

##  5 Limitations

We discuss the limitation of the proposed framework in this section, namely,
the relatively long-term preference data production pipeline.

Long-term data production. As illustrated in Figure 1, the proposed framework
contains four steps to obtain the ultimate high-quality preference data, each
step requires relatively extensive filtering. The long-term collection
pipeline may not facilitate collect enough training data in a short period of
time. Therefore, we believe the proposed framework is more suitable for the
later stages of RM optimization and for optimizing certain specific verticals.
In the early stages, we can rely on AI or open-source data for RM tuning.

## References

  * Artetxe et al. (2022) Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Ott, Sam Shleifer, Xi Victoria Lin, Jingfei Du, Srinivasan Iyer, Ramakanth Pasunuru, Giri Anantharaman, Xian Li, Shuohui Chen, Halil Akin, Mandeep Baines, Louis Martin, Xing Zhou, Punit Singh Koura, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Mona Diab, Zornitsa Kozareva, and Ves Stoyanov. 2022.  Efficient large scale language modeling with mixtures of experts.
  * Bai et al. (2023) Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. 2023.  Qwen technical report.
  * Bai et al. (2022) Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022.  Training a helpful and harmless assistant with reinforcement learning from human feedback.  _arXiv preprint arXiv:2204.05862_.
  * Beirami et al. (2024) Ahmad Beirami, Alekh Agarwal, Jonathan Berant, Alexander D’Amour, Jacob Eisenstein, Chirag Nagpal, and Ananda Theertha Suresh. 2024.  Theoretical guarantees on the best-of-n alignment policy.  _arXiv preprint arXiv:2401.01879_.
  * DeepSeek-AI et al. (2024) DeepSeek-AI, Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Hanwei Xu, Hao Yang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jin Chen, Jingyang Yuan, Junjie Qiu, Junxiao Song, Kai Dong, Kaige Gao, Kang Guan, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge, Ruizhe Pan, Runxin Xu, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Size Zheng, T. Wang, Tian Pei, Tian Yuan, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang, Xin Liu, Xin Xie, Xingkai Yu, Xinnan Song, Xinyi Zhou, Xinyu Yang, Xuan Lu, Xuecheng Su, Y. Wu, Y. K. Li, Y. X. Wei, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang, Yi Zheng, Yichao Zhang, Yiliang Xiong, Yilong Zhao, Ying He, Ying Tang, Yishi Piao, Yixin Dong, Yixuan Tan, Yiyuan Liu, Yongji Wang, Yongqiang Guo, Yuchen Zhu, Yuduan Wang, Yuheng Zou, Yukun Zha, Yunxian Ma, Yuting Yan, Yuxiang You, Yuxuan Liu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhewen Hao, Zhihong Shao, Zhiniu Wen, Zhipeng Xu, Zhongyu Zhang, Zhuoshu Li, Zihan Wang, Zihui Gu, Zilin Li, and Ziwei Xie. 2024.  Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model.
  * Dong et al. (2023a) Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. 2023a.  Raft: Reward ranked finetuning for generative foundation model alignment.  _arXiv preprint arXiv:2304.06767_.
  * Dong et al. (2023b) Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, Lei Li, and Zhifang Sui. 2023b.  A survey on in-context learning.
  * Eisenstein et al. (2023) Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ahmad Beirami, Alex D’Amour, DJ Dvijotham, Adam Fisch, Katherine Heller, Stephen Pfohl, Deepak Ramachandran, et al. 2023.  Helping or herding? reward model ensembles mitigate but do not eliminate reward hacking.  _arXiv preprint arXiv:2312.09244_.
  * Ethayarajh et al. (2022) Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. 2022.  Understanding dataset difficulty with $\mathcal{V}$-usable information.  In _International Conference on Machine Learning_ , pages 5988–6008. PMLR.
  * Liu et al. (2023) Xiao Liu, Xuanyu Lei, Shengyuan Wang, Yue Huang, Zhuoer Feng, Bosi Wen, Jiale Cheng, Pei Ke, Yifan Xu, Weng Lam Tam, et al. 2023.  Alignbench: Benchmarking chinese alignment of large language models.  _arXiv preprint arXiv:2311.18743_.
  * Nakano et al. (2021) Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. 2021.  Webgpt: Browser-assisted question-answering with human feedback.  _arXiv preprint arXiv:2112.09332_.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ , 35:27730–27744.
  * Ramé et al. (2024) Alexandre Ramé, Nino Vieillard, Léonard Hussenot, Robert Dadashi, Geoffrey Cideron, Olivier Bachem, and Johan Ferret. 2024.  Warm: On the benefits of weight averaged reward models.  _arXiv preprint arXiv:2401.12187_.
  * Skalse et al. (2022) Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. 2022.  Defining and characterizing reward gaming.  _Advances in Neural Information Processing Systems_ , 35:9460–9471.
  * Stiennon et al. (2020) Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020.  Learning to summarize with human feedback.  _Advances in Neural Information Processing Systems_ , 33:3008–3021.
  * Touvron et al. (2023a) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a.  Llama: Open and efficient foundation language models.
  * Touvron et al. (2023b) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023b.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Wang et al. (2024) Binghai Wang, Rui Zheng, Lu Chen, Yan Liu, Shihan Dou, Caishuang Huang, Wei Shen, Senjie Jin, Enyu Zhou, Chenyu Shi, et al. 2024.  Secrets of rlhf in large language models part ii: Reward modeling.  _arXiv preprint arXiv:2401.06080_.
  * Yang et al. (2023) Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, et al. 2023.  Baichuan 2: Open large-scale language models.  _arXiv preprint arXiv:2309.10305_.
  * Zhang et al. (2024) Shun Zhang, Zhenfang Chen, Sunli Chen, Yikang Shen, Zhiqing Sun, and Chuang Gan. 2024.  Improving reinforcement learning from human feedback with efficient reward model ensemble.  _arXiv preprint arXiv:2401.16635_.
  * Ziegler et al. (2019) Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019.  Fine-tuning language models from human preferences.  _arXiv preprint arXiv:1909.08593_.