# Einstein submanifolds with parallel mean curvature into
$\mathbb{S}^{n}\times\mathbb{R}$

Estela Garcia111The first author is partially supported by CAPES grant
10869871/D and CNPq grant 141475/2019-6 and Fernando Manfio222The second
author is supported by Fapesp, grant 2016/23746-6.

###### Abstract

We prove that Einstein submanifolds in $\mathbb{S}^{n}\times\mathbb{R}$ with
flat normal bundle and parallel mean curvature are warped product of isometric
immersions.

Key words: Einstein submanifolds, Parallel mean curvature, Flat normal bundle.

##  1 Introduction

One of the basic problems in submanifold theory is to provide conditions that
imply that an isometric immersion of a product manifold must be a extrinsic
product of isometric immersions. The first contribution to this problem was
addressed by Moore [12] for isometric immersions into Euclidean space and
extended by Molzan [11] for any space form as ambient space. They showed that
an isometric immersion of a Riemannian product into a space form must be an
extrinsic product of isometric immersions of the factors whenever its second
fundamental form is adapted to the product structure of the manifold.

In this paper we consider isometric immersions of Einstein manifolds into the
product space $\mathbb{Q}^{n}_{\epsilon}\times\mathbb{R}$, with flat normal
bundle and parallel mean curvature vector field. Recall that a Riemannian
manifold $M$ is an Einstein manifold if the Ricci tensor satisfies

$\mathrm{Ric}(X,Y)=\lambda\langle X,Y\rangle,$

for all $X,Y\in\mathfrak{X}(M)$ and some constante $\lambda\in\mathbb{R}$.
Riemannian manifolds $M^{n}_{c}$ with constant sectional curvature $c$ are the
simplest examples of Einstein Manifolds, where $\lambda=(n-1)c$.

An isometric immersion is said to be Einstein if the submanifold, endowed with
the induced metric, is an Einstein manifold. For example, Ryan [16] gave a
local classification of Einstein hypersurfaces in any space form. In higher
codimension, Di Scala [4] proved that Einstein real Kahler submanifolds of a
Euclidean space are totally geodesic provided that they are minimal, and same
conclusion, by Nolker [14], still holds for minimal Einstein submanifolds in
the Euclidean space, with flat normal bundle.

In [15], Onti classified Einstein submanifolds into space forms, with flat
normal bundle and parallel mean curvature vector field, extending a previous
result due to Dajczer-Tojeiro [2] for isometric immersions of Riemannian
manifolds with constant sectional curvature. More recently, Leandro-Pina-dos
Santos [6] classified Einstein hypersurfaces into product space
$\mathbb{Q}^{n}_{\epsilon}\times\mathbb{R}$.

In the present paper, we classify Einstein submanifolds $f\colon
M^{m}\to\mathbb{S}^{n}\times\mathbb{R}$, $m\geq 3$, with flat normal bundle
and parallel mean curvature vector field. As a first step, we prove that such
submanifolds have the property that the tangent component of the second factor
of $\mathbb{S}^{n}\times\mathbb{R}$ is an eigenvector of all shape operators
of $f$. They belong to a more general class, named class $\mathcal{A}$,
introduced by Tojeiro [17] for the case of hypersurfaces and extended to
higher codimension by Mendonça-Tojeiro [10]. Some important classes of
submanifolds of $\mathbb{Q}^{n}_{\epsilon}\times\mathbb{R}$ that are included
in class $\mathcal{A}$ are hypersurfaces with constant sectional curvature
[8], rotational hypersurfaces [3], constant angle hypersurfaces [17] and
biconservative submanifolds [9].

######  Theorem 1.1.

Let $f\colon M^{m}\to\mathbb{Q}^{n}_{\epsilon}\times\mathbb{R}$, $m\geq 3$, be
an isometric immersion of a connected Einstein manifold, with Ricci curvature
$\lambda$ and flat normal bundle. Then $f$ belongs to class $\mathcal{A}$.

Theorem 1.1 extends Lemma 5 of [6] for the case of hypersurfaces. Our main
result, which extends the aforementioned result of Onti [15] is stated as
follows:

######  Theorem 1.2.

Let $f\colon M^{m}\to\mathbb{S}^{n}\times\mathbb{R}$, $m\geq 3$, be a proper
isometric immersion of a connected Einstein manifold with Ricci curvature
$\lambda$, flat normal bundle, and parallel mean curvature vector field $H$.
Then $f$ is a warped product of isometric immersions.

The paper is organized as follows. In the next section we just recall the
notion of extrinsic warped product of isometric immersions. In Section 3 we
recall the basic equations of an isometric immersion into
$\mathbb{Q}^{n}_{\epsilon}\times\mathbb{R}$. In Section 4 we study
submanifolds with flat normal bundle. In particular, we obtain an equivalent
condition for a submanifold, with such properties, to belong to class
$\mathcal{A}$. In Section 5 we calculate the Ricci tensor of a submanifold in
$\mathbb{Q}^{n}_{\epsilon}\times\mathbb{R}$, and prove that every Einstein
submanifold with flat normal bundle belongs to class $\mathcal{A}$. Finally,
in the last section we prove a sequence of lemmas and Theorem 1.2.

######  Acknowledgment.

We are indebted to Ruy Tojeiro and Felippe Guimarães for valuable suggestions
which improved some results in this paper.

##  2 Preliminaries

A metric $g$ on a product manifold $M^{m}=M_{0}\times M_{1}\times\ldots\times
M_{k}$ is called a warped product if there exist positive functions
$\rho_{i}\in C^{\infty}(M_{0})$, $1\leq i\leq k$, such that

$g=\pi_{0}^{\ast}g_{0}+\sum_{i=1}^{k}(\rho_{i}\circ\pi_{i})^{2}\pi_{i}^{\ast}g_{i},$

where $\pi_{i}\colon M^{m}\to M_{i}$ denotes the canonical projection. We call
$M^{m}$ endowed with this metric the warped product of $M_{0},\ldots,M_{k}$
with warping functions $\rho_{1},\ldots,\rho_{k}$ and denote it by

$M^{m}=M_{0}\times_{\rho_{1}}M_{1}\times\ldots\times_{\rho_{k}}M_{k}.$

Let $\mathbb{Q}^{n}_{\epsilon}$ denote a complete and simply connected space
form of constant sectional curvature $\epsilon$. Fix a point
$q\in\mathbb{Q}^{n}_{\epsilon}$ and let
$\mathbb{Q}_{\epsilon}^{n_{0}},\mathbb{Q}^{n_{1}}_{\epsilon_{1}},\ldots,\mathbb{Q}^{n_{k}}_{\epsilon_{k}}$,
with $n=n_{0}+n_{1}+\ldots+n_{k}$, be submanifolds of
$\mathbb{Q}^{n}_{\epsilon}$ through $q$ such that the first one is totally
geodesic and all the others are totally umbilical with mean curvature vectors
$z_{1},\ldots,z_{k}$ at $q$ and $\langle z_{i},z_{j}\rangle=-\epsilon$, for
$i\neq j$. The warped product representation

$\psi\colon\mathbb{Q}_{\epsilon}^{n_{0}}\times_{\sigma_{1}}\mathbb{Q}_{\epsilon_{1}}^{n_{1}}\times\ldots\times_{\sigma_{k}}\mathbb{Q}^{n_{k}}_{\epsilon_{k}}\to\mathbb{Q}^{n}_{\epsilon}$
(2.1)

of $\mathbb{Q}^{n}_{\epsilon}$ is the map

$\psi(p_{0},p_{1},\ldots,p_{k})=p_{0}+\sum_{i=1}^{k}\sigma_{i}(p_{0})(p_{i}-q),$

where the functions
$\sigma_{i}\colon\mathbb{Q}_{\epsilon}^{n_{0}}\to\mathbb{R}_{+}$ are defined
by

$\sigma_{i}(p)=\left\\{\begin{array}[]{lcl}1+\langle
p-q,a_{i}\rangle,&\mbox{if}&\epsilon=0\\\ \langle
p,a_{i}\rangle,&\mbox{if}&\epsilon\neq 0\end{array},\right.$

and satisfy $\sigma_{i}(q)=1$, with $a_{i}=cq-z_{i}$.

It was shown by Nolker [13] that any isometry of a warped product with $k+1$
factors onto an open dense subset of $\mathbb{Q}^{n}_{\epsilon}$ arises as the
restriction of a warped product representation as above. Given a warped
product representation as in (2.1) and isometric immersions $f_{i}\colon
M_{i}\to\mathbb{Q}_{\epsilon_{i}}^{n_{i}}$, with $0\leq i\leq k$ and
$\epsilon_{0}=\epsilon$, the map

$f=\psi\circ(f_{0}\times f_{1}\times\ldots\times f_{k})$

is an isometric immersion of the warped product manifold $M^{m}$ with warping
functions $\rho_{i}=\sigma_{i}\circ f_{0}$, called an extrinsic warped product
of isometric immersions. If all factors $f_{i}$ of the extrinsic product are
identity maps, the the map $f$ is called the multi-rotational submanifold
determined by $\psi$ with $f_{0}$ as profile.

##  3 Basic facts on submanifolds in
$\mathbb{Q}^{n}_{\epsilon}\times\mathbb{R}$

From now on, let $\mathbb{Q}^{n}_{\epsilon}$ denote either $\mathbb{S}^{n}$ or
$\mathbb{H}^{n}$, according to whether $\epsilon=1$ or $\epsilon=-1$,
respectively. Given an isometric immersion $f\colon
M^{m}\to\mathbb{Q}^{n}_{\epsilon}\times\mathbb{R}$, let $\partial_{t}$ denote
a unit vector field tangent to the second factor. Then, a tangent vector field
$T$ on $M^{m}$ and a normal vector field $\eta$ along $f$ are defined by

$\partial_{t}=f_{\ast}T+\eta.$ (3.1)

Using that $\partial_{t}$ is a parallel vector field in
$\mathbb{Q}^{n}_{\epsilon}\times\mathbb{R}$, we obtain by differentiating
equation (3.1) that

$\nabla_{X}T=A_{\eta}X$ (3.2)

and

$\alpha_{f}(X,T)=-\nabla^{\perp}_{X}\eta,$ (3.3)

for all $X\in\mathfrak{X}(M)$. Here and in the sequel $A_{\eta}$ stands for
the shape operator of $f$ in the direction $\eta$, given by

$\langle A_{\eta}X,Y\rangle=\langle\alpha_{f}(X,Y),\eta\rangle,$

for all $X,Y\in\mathfrak{X}(M)$. The Gauss, Codazzi and Ricci equations for
$f$ are, respectively,

$\displaystyle\begin{aligned}
R(X,Y)Z&=A_{\alpha_{f}(Y,Z)}X-A_{\alpha_{f}(X,Z)}Y+\epsilon\big{(}X\wedge Y\\\
&+\langle X,T\rangle Y\wedge T-\langle Y,T\rangle X\wedge
T\big{)}Z,\end{aligned}$ (3.4)
$\displaystyle\left(\nabla^{\perp}_{X}\alpha_{f}\right)(Y,Z)-\left(\nabla^{\perp}_{Y}\alpha_{f}\right)(X,Z)=\epsilon\langle(X\wedge
Y)T,Z\rangle\eta$ (3.5)

and

$\displaystyle
R^{\perp}(X,Y)\xi=\alpha_{f}(X,A_{\xi}Y)-\alpha_{f}(A_{\xi}X,Y),$ (3.6)

for all $X,Y,Z\in\mathfrak{X}(M)$ an for all $\xi\in\Gamma(TM^{\perp})$.
Equation (3.5) can also be written as

$(\nabla_{X}A)(Y,\xi)-(\nabla_{Y}A)(X,\xi)=\epsilon\langle\eta,\xi\rangle(X\wedge
Y)T,$ (3.7)

where

$(X\wedge Y)T=\langle Y,T\rangle X-\langle X,T\rangle Y,$

whereas equation (3.6) is equivalent to

$\displaystyle\langle
R^{\perp}(X,Y)\xi,\zeta\rangle=\langle[A_{\xi},A_{\zeta}]X,Y\rangle.$ (3.8)

Although this will not be used in this work, it is worth mentioning that
equations (3.2)–(3.6) completely determine an isometric immersion of a
Riemannian manifold $M^{m}$ into $\mathbb{Q}^{n}_{\epsilon}\times\mathbb{R}$
up to isometries of $\mathbb{Q}^{n}_{\epsilon}\times\mathbb{R}$ (see [7,
Corollary 3]).

Given an isometric immersion $f\colon
M^{m}\to\mathbb{Q}^{n}_{\epsilon}\times\mathbb{R}$, consider $M^{m}$ as a
submanifold in the underlying flat space $\mathbb{E}^{n+2}$, that is, consider
the isometric immersion $\tilde{f}=i\circ f$, where
$i:\mathbb{Q}^{n}_{\epsilon}\times\mathbb{R}\to\mathbb{E}^{n+2}$ denotes de
canonical inclusion. The second fundamental forms and normal connections of
$f$ and $\tilde{f}$ are related as follows. Let $\bar{N}=\pi\circ i$ the unit
normal vector field to the inclusion $i$, where
$\pi\colon\mathbb{E}^{n+1}\times\mathbb{R}\to\mathbb{E}^{n+1}$ is the
projection. We have

$\widetilde{\nabla}_{Z}\bar{N}=\pi_{\ast}i_{\ast}Z=i_{\ast}(Z-\langle
Z,\partial_{t}\rangle\partial_{t}),$

for every $Z\in\mathfrak{X}(\mathbb{Q}^{n}_{\epsilon}\times\mathbb{R})$, where
$\widetilde{\nabla}$ is the canonical connection on $\mathbb{E}^{n+2}$. Hence,

$A_{\bar{N}}^{i}Z=-Z+\langle Z,\partial_{t}\rangle\partial_{t}.$ (3.9)

The normal spaces of $f$ and $\tilde{f}$ are related by

$TM^{\perp}_{\tilde{f}}=i_{\ast}TM^{\perp}_{f}\oplus\text{span}\\{N\\},$

where $N=\bar{N}\circ f=\pi\circ\tilde{f}$. Denoting by $\overline{\nabla}$
the Levi-Civita connection of $\mathbb{Q}^{n}_{\epsilon}\times\mathbb{R}$, and
given $\xi\in TM^{\perp}_{f}$, it follows from (3.9) that

$\widetilde{\nabla}_{X}i_{\ast}\xi=-\tilde{f}_{\ast}A^{f}_{\xi}X+i_{\ast}\nabla^{\perp}_{X}\xi+\epsilon\langle
X,T\rangle\langle\xi,\eta\rangle N.$

In particular, we have

$\widetilde{\nabla}^{\perp}_{X}i_{\ast}\xi=i_{\ast}\nabla^{\perp}_{X}\xi+\epsilon\langle
X,T\rangle\langle\xi,\eta\rangle N,$ (3.10)

for every $\xi\in TM^{\perp}_{f}$, where $\widetilde{\nabla}^{\perp}$ is the
normal connection of $\tilde{f}$. Moreover,

$\widetilde{\nabla}_{X}N=\tilde{f}_{\ast}(X-\langle X,T\rangle T)-\langle
X,T\rangle i_{\ast}\eta$

which implies, in particular, that

$\widetilde{\nabla}^{\perp}_{X}N=-\langle X,T\rangle i_{\ast}\eta.$ (3.11)

##  4 Submanifolds with flat normal bundle in
$\mathbb{Q}^{n}_{\epsilon}\times\mathbb{R}$

An important class of submanifolds in
$\mathbb{Q}^{n}_{\epsilon}\times\mathbb{R}$ consists of those which have flat
normal bundle. As in the case of submanifolds in a space with constant
sectional curvature, flatness of the normal bundle of a submanifold into
$\mathbb{Q}^{n}_{\epsilon}\times\mathbb{R}$ has the following useful
characterization.

######  Proposition 4.1.

An isometric immersion $f\colon
M^{m}\to\mathbb{Q}^{n}_{\epsilon}\times\mathbb{R}$ has flat normal bundle at a
point $x\in M^{m}$ if and only if the shape operators

$\\{A_{\xi}:\xi\in T_{x}M^{\perp}\\}$

are simultaneously diagonalizable. Equivalently, if and only if there exists
an orthonormal basis $\\{X_{1},\ldots,X_{m}\\}$ of $T_{x}M$ such that

$\alpha_{f}(X_{i},X_{j})=0,\ 1\leq i\neq j\leq m.$

###### Proof.

By the Ricci equation (3.8), the normal curvature tensor $R^{\perp}$ vanishes
at $x\in M^{m}$ if and only if all shape operators $A_{\xi}$, $\xi\in
T_{x}M^{\perp}$, commute, and the conclusion follows. ∎

It follows from Proposition 4.1 that, at each $x\in M$ where $R^{\perp}(x)=0$,
the tangent space $T_{x}M$ decomposes orthogonally as

$T_{x}M=E_{1}(x)\oplus\cdots\oplus E_{s(x)}(x).$ (4.1)

This decomposition has the property that for each $\xi\in T_{x}M^{\perp}$,
there exist real numbers $\lambda_{i}(\xi)$, $1\leq i\leq s=s(x)$, such that

$A_{\xi}|_{E_{i}(x)}=\lambda_{i}(\xi)\text{Id}$

and the maps $\xi\mapsto\lambda_{i}(\xi)$ are pairwise distinct. Since such
maps are linear, there exist unique pairwise distinct vectors $\xi_{i}(x)\in
T_{x}M^{\perp}$, $1\leq i\leq s$, called the principal normals of $f$ at $x$,
such that

$\lambda_{i}(\xi)=\langle\xi_{i}(x),\xi\rangle,\quad 1\leq i\leq s.$

Therefore, denoting by $E_{i}(x)=E_{\xi_{i}(x)}$, we have

$E_{\xi_{i}(x)}=\\{X\in T_{x}M:\alpha_{f}(X,Y)=\langle X,Y\rangle\xi_{i}(x),\
\forall\ Y\in T_{x}M\\},$

and the second fundamental form of $f$ has the simple representation

$\alpha(X,Y)=\sum_{i=1}^{s}\langle X^{i},Y^{i}\rangle\xi_{i}(x),$ (4.2)

where $X\mapsto X^{i}$ is the orthogonal projection onto $E_{i}(x)$.
Equivalently,

$A_{\xi}X=\sum_{i=1}^{s}\langle\xi,\xi_{i}(x)\rangle X^{i}$ (4.3)

for all $X\in T_{x}M$ and $\xi\in T_{x}M^{\perp}$.

######  Remark 4.2.

Throughout this paper a submanifold with flat normal bundle will always be
considered proper, that is, it has a constant number $s=s(x)$ of principal
normals vector fields. In this case, the principal normal vector fields $x\in
M^{m}\mapsto\xi_{i}(x)$, $1\leq i\leq s$, are smooth. Moreover, the
distributions $x\in M^{m}\mapsto E_{i}(x)$, $1\leq i\leq s$, have constant
dimension and are also smooth.

A direct calculation shows the following

######  Lemma 4.3.

Let $f\colon M^{m}\to\mathbb{Q}^{n}_{\epsilon}\times\mathbb{R}$ be an
isometric immersion with flat normal bundle. Then, the Codazzi equation (3.5)
can be written as

$\langle X_{j},Y_{j}\rangle(\nabla^{\perp}_{X_{i}}\xi_{j}+\epsilon\langle
X_{i},T\rangle\eta)=\langle\nabla_{X_{j}}Y_{j},X_{i}\rangle(\xi_{j}-\xi_{i}),$
(4.4)

for $1\leq i\neq j\leq s$, $X_{i}\in\Gamma(E_{i})$, and
$X_{j},Y_{j}\in\Gamma(E_{j})$, or

$\langle\nabla_{X_{j}}X_{i},X_{k}\rangle(\xi_{i}-\xi_{k})=\langle\nabla_{X_{i}}X_{j},X_{k}\rangle(\xi_{j}-\xi_{k}),$
(4.5)

for $1\leq i\neq j\neq k\leq s$, $X_{i}\in\Gamma(E_{i})$,
$X_{j}\in\Gamma(E_{j})$, and $X_{k}\in\Gamma(E_{k})$.

Following the notations of [10], we denote by $\mathcal{A}$ the class of
isometric immersions $f\colon
M^{m}\to\mathbb{Q}^{n}_{\epsilon}\times\mathbb{R}$ with the property that $T$
is an eigenvector of all shape operators of $f$. Our next result shows a
relationship between isometric immersions into
$\mathbb{Q}^{n}_{\epsilon}\times\mathbb{R}$ which have flat normal bundle and
those that are in class $\mathcal{A}$.

######  Proposition 4.4.

Let $f\colon M^{m}\to\mathbb{Q}^{n}_{\epsilon}\times\mathbb{R}$ be an
isometric immersion with flat normal bundle. Then, $f$ is in class
$\mathcal{A}$ if and only if the tangent vector field $T$ belongs to some
$E_{i}$ in the decomposition (4.1).

###### Proof.

If $f$ belongs to class $\mathcal{A}$, then $A_{\xi}T=\lambda T$ for every
normal vector field $\xi\in\Gamma(TM^{\perp})$, where $\lambda=\lambda(\xi)$
is a smooth function along $M$. On the other hand, it follows from (4.3) that

$A_{\xi}T=\sum_{i=1}^{s}\langle\xi,\xi_{i}\rangle T^{i}.$ (4.6)

Taking the inner product with $T^{i}\neq 0$ of both sides of (4.6) yields

$\lambda\|T^{i}\|^{2}=\langle\xi,\xi_{i}\rangle\|T^{i}\|^{2}.$ (4.7)

Suppose that there exists another index $j\neq i$ such that $T^{j}\neq 0$. It
follows from (4.7) that $\langle\xi,\xi_{i}\rangle=\langle\xi,\xi_{j}\rangle$.
In particular, taking $\xi=\xi_{i}$ and then $\xi=\xi_{j}$, we obtain
$\|\xi_{i}\|^{2}=\langle\xi_{i},\xi_{j}\rangle=\|\xi_{j}\|^{2}$. This implies
that $\xi_{i}=\xi_{j}$, which is a contradiction. Conversely, suppose that
$T\in E_{i}$, for some $1\leq i\leq s$. Thus

$\alpha(T,X)=\langle T,X\rangle\xi_{i},$

for every $X\in TM$. Therefore, given $\xi\in\Gamma(TM^{\perp})$, we have
$A_{\xi}T=\langle\xi,\xi_{i}\rangle T$, and this shows that $T$ is an
eigenvector of $A_{\xi}$, for any $\xi\in\Gamma(TM^{\perp})$. ∎

######  Remark 4.5.

Under the hypotheses of Proposition 4.4, we can suppose, without loss of
generality, that $T\in E_{1}$.

##  5 A basic result

Given an isometric immersion $f\colon
M^{m}\to\mathbb{Q}^{n}_{\epsilon}\times\mathbb{R}$, we recall that the Ricci
tensor of $M^{m}$ is defined by

$\mathrm{Ric}(X,Y)=tr\\{Z\mapsto R(Z,X)Y\\}$

for every $X,Y\in TM$. In the next result we compute the Ricci tensor of
$M^{m}$ in terms of the second fundamental form of $f$.

######  Lemma 5.1.

The Ricci tensor of an isometric imersion $f\colon
M^{m}\to\mathbb{Q}^{n}_{\epsilon}\times\mathbb{R}$ is given by

$\displaystyle\begin{aligned} \mathrm{Ric}(X,Y)=&\
\epsilon(m-1-\|T\|^{2})\langle X,Y\rangle+\epsilon(2-m)\langle
X,T\rangle\langle Y,T\rangle\\\ &+m\langle H,\alpha(X,Y)\rangle-
III(X,Y),\end{aligned}$ (5.1)

where

$III(X,Y)=\sum_{i=1}^{m}\langle\alpha(X,X_{i}),\alpha(Y,X_{i})\rangle$

denotes the third fundamental form of $f$ in terms of an orthonormal tangent
frame $\\{X_{1},\ldots,X_{m}\\}$ of $M^{m}$.

###### Proof.

Given an orthonormal tangent frame $\\{X_{1},\ldots,X_{m}\\}$ of $M^{m}$, the
Gauss equation of $f$ yields

$\displaystyle\mathrm{Ric}(X,Y)$ $\displaystyle=$
$\displaystyle\sum_{i=1}^{m}\langle R(X_{i},X)Y,X_{i}\rangle$ $\displaystyle=$
$\displaystyle\sum_{i=1}^{m}\big{[}\langle\epsilon(X_{i}\wedge
X)Y-\epsilon\langle X,T\rangle(X_{i}\wedge T)Y+\epsilon\langle
X_{i},T\rangle(X\wedge T)Y$
$\displaystyle+\langle\alpha(X_{i},X_{i}),\alpha(X,Y)\rangle-\langle\alpha(X,X_{i}),\alpha(Y,X_{i})\rangle\big{]}$
$\displaystyle=$ $\displaystyle\epsilon(m-1)\langle X,Y\rangle-\epsilon\langle
X,T\rangle\big{(}m\langle Y,T\rangle-\langle Y,T\rangle\big{)}+\epsilon\langle
Y,T\rangle\langle X,T\rangle$ $\displaystyle-\epsilon\langle
X,Y\rangle\|T\|^{2}+m\langle H,\alpha(X,Y)\rangle-III(X,Y)$ $\displaystyle=$
$\displaystyle\epsilon\big{(}m-1-\|T\|^{2}\big{)}\langle
X,Y\rangle+\epsilon(2-m)\langle X,T\rangle\langle Y,T\rangle$
$\displaystyle+m\langle H,\alpha(X,Y)\rangle-III(X,Y),$

and the proof of (5.1) is completed. ∎

######  Remark 5.2.

The formula for the Ricci tensor in (5.1) extends the one obtained in [6] for
the case of hypersurfaces.

The Ricci curvature in the direction of a unit vector $X\in TM$ is defined as

$\mathrm{Ric}(X)=\frac{1}{m-1}\mathrm{Ric}(X,X).$

It follows from (5.1) that

$\displaystyle\begin{aligned} \mathrm{Ric}(X)=&\
\epsilon-\frac{\epsilon}{m-1}\big{(}\|T\|^{2}-\langle
T,X\rangle^{2}\big{)}-\epsilon\langle T,X\rangle^{2}\\\ &+\frac{m}{m-1}\langle
H,\alpha(X,X)\rangle-\frac{1}{m-1}III(X,X),\end{aligned}$ (5.2)

for any unit vector $X\in TM$. Equation (5.2) yields the following obstruction
for the existence of a minimal submanifold into any product space
$\mathbb{Q}^{n}_{\epsilon}\times\mathbb{R}$.

######  Corollary 5.3.

Let $f\colon M^{m}\to\mathbb{Q}^{n}_{\epsilon}\times\mathbb{R}$ be a minimal
isometric immersion. Then for any point $x\in M$, we have
$\mathrm{Ric}(X)\leq\epsilon$ for every unit vector $X\in T_{x}M$. Moreover,
the equality holds identically if and only if $f(M)$ is an open subset of a
slice $N^{m}\times\\{t_{0}\\}$, where $N^{m}$ is a submanifold of
$\mathbb{Q}^{n}_{\epsilon}\times\mathbb{R}$.

###### Proof.

Since $f$ is minimal, we have

$\mathrm{Ric}(X)=\epsilon-\frac{\epsilon}{m-1}\big{(}\|T\|^{2}-\langle
T,X\rangle^{2}\big{)}-\epsilon\langle T,X\rangle^{2}-\frac{1}{m-1}III(X,X),$

and this proves the first statement, since $\|T\|^{2}-\langle
X,T\rangle^{2}\geq 0$. The equality holds identically if and only if $f$ is
totally geodesic and $T=0$, that is, $f(M)$ is an open subset of a slice of
$\mathbb{Q}^{n}_{\epsilon}\times\mathbb{R}$. ∎

######  Corollary 5.4.

Let $f\colon M^{m}_{c}\to\mathbb{Q}^{n}_{\epsilon}\times\mathbb{R}$ be a
minimal isometric immersion. Then:

  1. (a)

$c\leq\epsilon$,

  2. (b)

$c=\epsilon$ if and only if $f(M)$ is an open subset of a slice in
$\mathbb{Q}^{n}_{\epsilon}\times\mathbb{R}$.

The next result provides a characterization of Einstein submanifolds with flat
normal bundle, extending the one of [6] for the case of hypersurfaces.

######  Theorem 5.5.

Let $f\colon M^{m}\to\mathbb{Q}^{n}_{\epsilon}\times\mathbb{R}$ be an Einstein
submanifold, $m\geq 3$, with flat normal bundle at a point $x\in M$. If $T\neq
0$ at $x\in M$, then $T$ is an eigenvector of all shape operators of $f$ at
$x$.

###### Proof.

By Proposition 4.1, there exists an orthonormal basis
$\\{X_{1},\ldots,X_{m}\\}$ of $T_{x}M$ such that

$\alpha_{f}(X_{i},X_{j})=0,\ 1\leq i\neq j\leq m.$

Write $T=\sum_{k=1}^{m}t_{k}X_{k}$ at the point $x$. On the other hand, since
$M$ is an Einstein manifold one has

$\mathrm{Ric}(X_{i},X_{j})=\rho\delta_{ij},$ (5.3)

for some $\rho\in\mathbb{R}$. That is,

$\mathrm{Ric}(X_{i},X_{j})=\sum_{k=1}^{m}\langle
R(X_{k},X_{i})X_{j},X_{k}\rangle=\rho\delta_{ij}.$

When applying the Ricci tensor to the referencial $\\{X_{1},\ldots,X_{m}\\}$,
we obtain

$\displaystyle\begin{aligned} \mathrm{Ric}(X_{i},X_{j})=&\
\big{(}\epsilon(m-1-\|T\|^{2}+m\lambda_{i}(H)\big{)}\delta_{ij}\\\
&+\epsilon(2-m)t_{i}t_{j}-III(X_{i},X_{j}).\end{aligned}$ (5.4)

It follows from (5.3) and (5.4) that

$\displaystyle\big{(}\epsilon(m-1-\|T\|^{2}+m\lambda_{i}(H)-\rho\big{)}\delta_{ij}+\epsilon(2-m)t_{i}t_{j}-III(X_{i},X_{j})=0.$
(5.5)

Note that the third fundamental form of $f$, expressed in terms of the basis
$\\{X_{1},\ldots,X_{m}\\}$, is always equal to zero, for $i\neq j$. Thus, it
follows from (5.5) that

$t_{i}t_{j}=0,\ 1\leq i\neq j\leq m.$

Since $T(x)\neq 0$, there is only one index $1\leq k\leq m$, with $t_{k}\neq
0$, and this implies that $T=t_{k}X_{k}$ at $x\in M$. ∎

##  6 The proof of the main result

Let $f\colon M^{m}\to\mathbb{Q}^{n}_{\epsilon}\times\mathbb{R}$, $m\geq 3$, be
an isometric immersion of a connected Einstein manifold with Ricci curvature
$\lambda$, flat normal bundle, and parallel mean curvature vector field $H$.
In order to prove Theorem 1.2, we will show a sequence of auxiliary results.

######  Lemma 6.1.

If the tangent vector field $T$ is nowhere vanishing, then

$\left\|\xi_{k}-\frac{m}{2}H\right\|^{2}=\frac{m^{2}}{4}\left\|H\right\|^{2}-\lambda+(m-2)\epsilon+\|\eta\|^{2}\epsilon,$
(6.1)

for any $2\leq k\leq s$, and for $k=1$ if $\dim E_{1}>1$.

###### Proof.

Let $X_{1}=T/\|T\|$, and consider a unit vector field
$X_{2}\in\Gamma(E_{1})\cap\\{T\\}^{\perp}$. Complete to an orthonormal frame
$\\{X_{1},X_{2},\ldots,X_{m}\\}$ of $M$, where each $X_{i}$ belongs to some
$\Gamma(E_{i})$. It follows from Gauss equation (3.4), for $k=1$, that

$\displaystyle\lambda$ $\displaystyle=$
$\displaystyle\mathrm{Ric}(X_{2},X_{2})=\sum_{i=1}^{m}\langle
R(X_{i},X_{2})X_{2},X_{i}\rangle$ $\displaystyle=$
$\displaystyle\epsilon\|\eta\|^{2}+\|\xi_{1}\|^{2}(\dim(E_{1})-1)+(m-2)\epsilon+\sum_{i=2}^{s}\langle\xi_{i},\xi_{2}\rangle\dim(E_{i}),$

that is,

$\|\xi_{1}\|^{2}\dim(E_{1})+\sum_{i=2}^{s}\langle\xi_{i},\xi_{1}\rangle\dim(E_{i})=\lambda-(m-2)\epsilon+\|\xi_{1}\|^{2}-\|\eta\|^{2}\epsilon.$

On the other hand, we can write

$\displaystyle\begin{aligned}
m\langle\xi_{1},H\rangle&=|\xi_{1}\|^{2}\dim(E_{1})+\sum_{i=2}^{s}\langle\xi_{i},\xi_{1}\rangle\dim(E_{i})\\\
&=\lambda-(m-2)\epsilon-\|\eta\|^{2}\epsilon+\|\xi_{1}\|^{2}.\end{aligned}$
(6.2)

Therefore, using (6.2), we have

$\displaystyle\left\|\xi_{1}-\frac{m}{2}H\right\|^{2}$ $\displaystyle=$
$\displaystyle\left\langle\xi_{1}-\frac{m}{2}H,\xi_{1}-\frac{m}{2}H\right\rangle$
$\displaystyle=$
$\displaystyle\|\xi_{1}\|^{2}-m\langle\xi_{1},H\rangle+\frac{m^{2}}{4}\|H\|^{2}$
$\displaystyle=$
$\displaystyle\frac{m^{2}}{4}\|H\|^{2}-\lambda+(m-2)\epsilon+\|\eta\|^{2}\epsilon.$

The case $k\geq 2$ is analogous to the previous case. In fact, it follows from
Gauss equation (3.4) that

$\displaystyle\langle\xi_{1},\xi_{k}\rangle\dim(E_{1})+\sum_{i=2}^{s}\langle\xi_{i},\xi_{k}\rangle\dim(E_{i})=\lambda-(m-2)\epsilon-\|\eta\|^{2}\epsilon+\|\xi_{k}\|^{2}.$

The equation (6.1) follows by writing

$m\langle\xi_{k},H\rangle=\lambda-(m-2)\epsilon-\|\eta\|^{2}\epsilon+\|\xi_{k}\|^{2},$
(6.3)

as in the previous case. ∎

######  Remark 6.2.

It follows from Lemma 6.1 that the functions

$h_{k}=\left\|\xi_{k}-\frac{m}{2}H\right\|^{2}$

are constant along $\\{T\\}^{\perp}$, for any $2\leq k\leq s$, and for $k=1$
if $\dim E_{1}>1$.

######  Lemma 6.3.

The principal normals $\xi_{1},\ldots,\xi_{s}$ of $f$ are parallel along
$\\{T\\}^{\perp}$.

###### Proof.

Given $X,Y\in\Gamma(E_{i})$ and $Z\in\Gamma(E_{k})\cap\\{T\\}^{\perp}$, with
$2\leq i\leq s$ and $1\leq k\neq i\leq s$, it follows from Codazzi equation
(4.4) that

$\langle
X,Y\rangle\nabla^{\perp}_{Z}\xi_{i}=\langle\nabla_{X}Y,Z\rangle(\xi_{i}-\xi_{k}).$
(6.4)

Since $H$ is parallel, we have

$\langle
X,Y\rangle\nabla^{\perp}_{Z}\left(\xi_{i}-\frac{m}{2}H\right)=\langle\nabla_{X}Y,Z\rangle(\xi_{i}-\xi_{k}).$

This implies that

$\langle X,Y\rangle
Z\left(\left\|\xi_{i}-\frac{m}{2}H\right\|^{2}\right)=2\left\langle\nabla_{X}Y,Z\right\rangle\left\langle\xi_{i}-\xi_{k},\xi_{i}-\frac{m}{2}H\right\rangle$
(6.5)

By Remark 6.2, the left-hand side of (6.5) is zero. Moreover, from (6.1), and
using (6.2) and (6.3), one has

$\left\langle\xi_{i}-\xi_{k},\xi_{i}-\frac{m}{2}H\right\rangle=\frac{1}{2}\|\xi_{i}-\xi_{k}\|^{2}\neq
0.$

This implies that $\langle\nabla_{X}Y,Z\rangle=0$, for all
$X,Y\in\Gamma(E_{i})$, with $2\leq i\leq s$. Thus, from (6.4), one has

$\nabla^{\perp}_{Z}\xi_{i}=0,$

for all $Z\in\Gamma(E_{k})\cap\\{T\\}^{\perp}$, with $2\leq k\neq i\leq s$.
For $Z\in\Gamma(E_{i})$, we obtain

$\displaystyle 0$ $\displaystyle=$
$\displaystyle\nabla^{\perp}_{Z}H=\sum_{i\neq
k=1}^{s}\dim(E_{k})\nabla^{\perp}_{Z}\xi_{k}+\dim(E_{i})\nabla^{\perp}_{Z}\xi_{i}$
$\displaystyle=$ $\displaystyle\dim(E_{i})\nabla^{\perp}_{Z}\xi_{i}.$

This shows that $\xi_{i}$ is parallel along $\\{T\\}^{\perp}$, for $2\leq
i\leq s$. For the case $i=1$, it follows from Codazzi equation (4.4) that

$\|T\|^{2}\nabla^{\perp}_{Z}\xi_{1}=\langle\nabla_{T}T,Z\rangle(\xi_{1}-\xi_{k})=0,$

for every $Z\in\Gamma(E_{k})$, with $2\leq k\leq s$, that is,
$\nabla^{\perp}_{Z}\xi_{1}=0$, for every $Z\in\Gamma(E_{k})$. Finally, if
there exists $X\in\Gamma(E_{1})\cap\\{T\\}^{\perp}$, then

$\dim(E_{1})\nabla^{\perp}_{X}\xi_{1}=\nabla^{\perp}_{X}H-\sum_{k=2}^{s}\dim(E_{k})\nabla^{\perp}_{X}\xi_{k}=0,$

by the previous case. This shows that $\xi_{i}$ is parallel along
$\\{T\\}^{\perp}$. ∎

Given an isometric immersion $f\colon
M^{m}\to\mathbb{Q}^{n}_{\epsilon}\times\mathbb{R}$ as it has been fixed,
consider the composition $\tilde{f}=i\circ f$. In this case, it follows from
[10, Corollary 1.3] that $\tilde{f}$ also has flat normal bundle. In order to
relate the flatness of the normal bundle of $f$ and $\tilde{f}$, let
$\bar{N}=\pi\circ i$ the unit normal vector field to the inclusion $i$, and
consider the decomposition (4.1) for $f$ with principal normal vector fields
$\xi_{1},\ldots,\xi_{s}$, where

$E_{i}(x)=\\{X\in T_{x}M:\alpha_{f}(X,Y)=\langle X,Y\rangle\xi_{i}(x),\
\forall\ Y\in T_{x}M\\}.$

######  Lemma 6.4.

The following assertions hold:

  1. (a)

If $\dim E_{1}=1$, then (4.1) is also a decomposition for $\tilde{f}$, with
principal normal vector fields given by

$i_{\ast}\xi_{1}-\|\eta\|^{2}\bar{N},i_{\ast}\xi_{2}-\bar{N},\dots,i_{\ast}\xi_{s}-\bar{N}.$

  2. (b)

If $\dim E_{1}\geq 2$, then

$T_{x}M=\text{span}\\{T(x)\\}\oplus\big{(}E_{1}(x)\cap\text{span}\\{T(x)\\}^{\perp}\big{)}\oplus
E_{2}(x)\oplus\ldots\oplus E_{s}(x),$

is a decomposition for $\tilde{f}$, whose principal normals are given by

$i_{\ast}\xi_{1}-\|\eta\|^{2}\bar{N},i_{\ast}\xi_{1}-\bar{N},i_{\ast}\xi_{2}-\bar{N},\dots,i_{\ast}\xi_{s}-\bar{N}.$

###### Proof.

Take any vector $X\in E_{k}(x)$, with $1\leq k\leq s$ and
$X\in\\{T(x)\\}^{\perp}$. Then

$\displaystyle\alpha_{\tilde{f}}(X,Y)$ $\displaystyle=$ $\displaystyle
i_{\ast}\alpha_{f}(X,Y)+\alpha_{i}(f_{\ast}X,f_{\ast}Y)$ $\displaystyle=$
$\displaystyle i_{\ast}\langle X,Y\rangle\xi_{k}-\langle
X,Y\rangle\bar{N}+\langle X,T\rangle\langle Y,T\rangle\bar{N}$
$\displaystyle=$ $\displaystyle\langle X,Y\rangle(i_{\ast}\xi_{k}-\bar{N}),$

for all $Y\in TM$. Moreover,

$\displaystyle\alpha_{\tilde{f}}(T,Y)$ $\displaystyle=$ $\displaystyle
i_{\ast}\alpha_{f}(T,Y)+\alpha_{i}(f_{\ast}T,f_{\ast}Y)$ $\displaystyle=$
$\displaystyle i_{\ast}\langle T,Y\rangle\xi_{1}-\langle
T,Y\rangle\bar{N}+\langle T,T\rangle\langle T,Y\rangle\bar{N}$
$\displaystyle=$ $\displaystyle\langle
T,Y\rangle(i_{\ast}\xi_{1}-\|\eta\|^{2}\bar{N}),$

for all $Y\in TM$, and this concludes the proof. ∎

From now on we write $E_{0}=\text{span}\\{T\\}$ and, with abuse of notation,
we write $E_{1}=E_{1}\cap E_{0}^{\perp}$. Thus, at each point $x\in M$, we
have the decomposition for $\tilde{f}$ given by

$T_{x}M=E_{0}(x)\oplus E_{1}(x)\oplus\ldots\oplus E_{s}(x),$ (6.6)

whose principal normals are given by

$i_{\ast}\xi_{1}-\|\eta\|^{2}\bar{N},i_{\ast}\xi_{1}-\bar{N},\dots,i_{\ast}\xi_{s}-\bar{N},$

respectively, $E_{1}=\\{0\\}$ may occur.

######  Lemma 6.5.

The distributions $\\{E_{k}\\}$ of the decomposition (6.6) are such that
$E_{k}$ is spherical and $E_{k}^{\perp}$ is totally geodesic, for all $1\leq
k\leq s$.

###### Proof.

Given $X,Y\in\Gamma(E_{k})$, $1\leq k\leq s$, we have

$\displaystyle\langle\nabla_{X}Y,T\rangle$ $\displaystyle=$
$\displaystyle-\langle Y,\nabla_{X}T\rangle=-\langle
T,A_{\eta}X\rangle=-\langle Y,A_{i_{\ast}X}^{\tilde{f}}X\rangle$
$\displaystyle=$
$\displaystyle-\langle\alpha_{\tilde{f}}(X,Y),i_{\ast}\eta\rangle=-\langle
X,Y\rangle\langle i_{\ast}\xi_{k}-\bar{N},i_{\ast}\eta\rangle$
$\displaystyle=$ $\displaystyle-\langle
X,Y\rangle\langle\xi_{k},\eta\rangle=-\langle X,Y\rangle\langle
T,\langle\xi_{k},\eta\rangle T\rangle\frac{1}{\|T\|^{2}}$

that is,

$\langle\nabla_{X}Y,T\rangle=\langle X,Y\rangle\langle T,\delta\rangle,$ (6.7)

where

$\delta=-\frac{1}{\|T\|^{2}}\langle\xi_{k},\eta\rangle
T\in\Gamma(E_{k}^{\perp}).$ (6.8)

Moreover, for $Z\in\Gamma(E_{i})$, with $1\leq i\neq k\leq s$, Lemma 6.3
yields $\nabla^{\perp}_{Z}\xi_{k}=0$. Thus, it follows from Codazzi equation
(4.4) that

$\langle\nabla_{X}Y,Z\rangle=0=\langle X,Y\rangle\langle Z,\delta\rangle.$
(6.9)

Equations (6.7) and (6.9) imply that $E_{k}$ is an umbilical distribution with
mean curvature vector field $\delta$ given in (6.8). On the other hand, it
follows from Lemma 6.3 and the fact that $\|T\|$ is contant along
$\\{T\\}^{\perp}$, that

$\langle\nabla_{X}\delta,Z\rangle=-\frac{1}{\|T\|^{2}}\langle
T,Z\rangle\langle\eta,\nabla^{\perp}_{X}\xi_{k}\rangle=0,$

for all $X\in\Gamma(E_{k})$ and $Z\in\Gamma(E_{k}^{\perp})$, and this shows
that $E_{k}$ is a spherical distribution. In order to show that
$E_{k}^{\perp}$ is totally geodesic, it suffices to prove that

$\langle\nabla_{X}Y,Z\rangle=0,$ (6.10)

for all $X\in\Gamma(E_{i})$, $Y\in\Gamma(E_{j})$ and $Z\in\Gamma(E_{k})$, with
$1\leq k\leq s$, $0\leq i,j\leq s$, $k\neq\\{i,j\\}$. For $0\leq i=j\leq s$,
equation (6.10) follows from Lemma 6.3 and Codazzi equation. Now, let $0\leq
i\neq j\leq s$, and let us consider the following cases.
Case 1: If $j=0$ and $Y=\lambda T$, for some smooth function $\lambda$ along
$M$, then

$\displaystyle\langle\nabla_{X}Y,Z\rangle$ $\displaystyle=$
$\displaystyle\langle\nabla_{X}\lambda T,Z\rangle=\langle
X(\lambda)T+\lambda\nabla_{X}T,Z\rangle$ $\displaystyle=$
$\displaystyle\lambda\langle A_{\eta}X,Z\rangle=0.$

Case 2: If $i=0$ and $X=\lambda T$, for some smooth function $\lambda$ along
$M$, it follows from Codazzi equation (4.5) that

$\displaystyle\langle\nabla_{X}Y,Z\rangle(\xi_{j}-\xi_{k})$ $\displaystyle=$
$\displaystyle\langle\nabla_{\lambda T}Y,Z\rangle(\xi_{j}-\xi_{k})$
$\displaystyle=$ $\displaystyle\langle\nabla_{Y}\lambda
T,Z\rangle(\xi_{i}-\xi_{k})$ $\displaystyle=$ $\displaystyle 0$

as in the previous case.
Case 3: Suppose now $1\leq i\neq j\leq s$ and let $X,Y\in\\{T\\}^{\perp}$. It
follows from Codazzi equation (4.5) that

$\langle\nabla_{X}Y,Z\rangle(\xi_{j}-\xi_{k})=\langle\nabla_{Y}X,Z\rangle(\xi_{i}-\xi_{k}).$
(6.11)

We claim that the vector fields $\xi_{i}-\xi_{k}$ and $\xi_{j}-\xi_{k}$ are
linearly independent. Indeed, assume otherwise that there exists a smooth
nonzero function $u$ along $M$ such that

$\xi_{i}-\xi_{k}=u(\xi_{j}-\xi_{k}).$

It follows that

$(u-1)\left(\xi_{k}-\frac{m}{2}H\right)=u\left(\xi_{j}-\frac{m}{2}H\right)-\left(\xi_{i}-\frac{m}{2}H\right).$

By taking the norms, it follows from (6.1) that

$\left\|\xi_{j}-\frac{m}{2}H\right\|^{2}=\left\langle\xi_{j}-\frac{m}{2}H,\xi_{i}-\frac{m}{2}H\right\rangle.$
(6.12)

Now, using (6.2) and (6.3) in (6.12), we obtain $\|\xi_{i}=\xi_{j}\|$, that
is, $\xi_{i}=\xi_{j}$, which is a contradiction. The conclusion now follows
from (6.11). ∎

Finally, we can prove the main result of this paper.

###### Proof of Theorem 1.2.

Let $f\colon M^{m}\to\mathbb{S}^{n}\times\mathbb{R}$, $m\geq 3$, be a proper
isometric immersion of a connected Einstein manifold, with flat normal bundle
and parallel mean curvature vector field $H$. Moreover, suppose that the
vector field $T$ given in (3.1) is nowhere vanishing. Let $\tilde{f}=i\circ f$
be considered as a submanifold in the Euclidean space $\mathbb{R}^{n+2}$, and
consider the distributions $(E_{k})$, $0\leq k\leq s$, of $M^{m}$ given in
(6.6) which, by virtue of Lemma 6.5, is such that $E_{k}$ is spherical and
$E_{k}^{\perp}$ is totally geodesic, for $1\leq k\leq s$. It follows from
theorem of the Hiepko [5] that there exists locally (globally, if $M^{m}$ is
simply connected and complete) a product representation

$\psi\colon M_{0}\times M_{1}\times\ldots\times M_{s}\to M^{m}$

which is an isometry with respect to a warped product metric on the product
$M_{0}\times M_{1}\times\ldots\times M_{s}$, where $M_{0}$ is totally geodesic
and $M_{k}$ are intrinsic spheres, for $1\leq k\leq s$. In this case, as
$\dim(M_{0})=1$, we can suppose that $M_{0}$ is an open interval
$I_{0}\subset\mathbb{R}$. Consider now the immersion

$F=\tilde{f}\circ\psi\colon
I_{0}\times_{\rho_{1}}M_{1}^{n_{1}}\times\ldots\times_{\rho_{s}}M_{s}^{n_{s}}\to\mathbb{R}^{n+2}.$

$F$ is an isometric immersion defined in a warped product manifold, whose
second fundamental form is adapted to the product net (6.6) of $M^{m}$. It
follows from Nolker’s theorem [13] that $F$ is an extrinsic warped product of
isometric immersions. More precisely, there exist a warped product
representation

$\phi\colon\mathbb{R}^{s+1}\times_{\sigma_{1}}N_{1}^{n_{1}+k_{1}}\times\ldots\times_{\sigma_{s}}N_{s}^{n_{s}+k_{s}}\to\mathbb{R}^{n+2},$

isometric immersions $f_{j}\colon M_{j}^{n_{j}}\to N_{j}^{n_{j}+k_{j}}$,
$1\leq j\leq s$, and an smooth curve $f_{0}\colon I_{0}\to\mathbb{R}^{s+1}$
such that $\rho_{j}=\sigma_{j}\circ f_{0}$ for $1\leq j\leq s$ and

$F=\phi\circ(f_{0}\times\ldots\times f_{s}),$

where $N_{j}^{n_{j}+k_{j}}$ are Euclidean spheres. Moreover, each isometric
immersion $f_{j}\colon M_{j}^{n_{j}}\to N_{j}^{n_{j}+k_{j}}$ is umbilical,
with mean curvature vector fiel given by $H_{j}=i_{\ast}\xi_{j}-\bar{N}$. Each
of these normal vector fields is parallel along the normal connection
${}^{f_{j}}\widetilde{\nabla}^{\perp}$ of $f_{j}$, which is the restriction of
$\widetilde{\nabla}^{\perp}$ along $M_{j}^{n_{j}}$. In fact, using Lemma 6.3,
and equations (3.10) and (3.11), we obtain

${}^{f_{j}}\widetilde{\nabla}^{\perp}_{X}H_{j}$ $\displaystyle=$
${}^{f_{j}}\widetilde{\nabla}^{\perp}_{X}i_{\ast}\xi_{j}-^{f_{j}}\widetilde{\nabla}^{\perp}_{X}\bar{N}$
$\displaystyle=$ $\displaystyle i_{\ast}\nabla^{\perp}_{X}\xi_{j}+\langle
X,T\rangle\langle\xi_{j},\eta\rangle\bar{N}+\langle X,T\rangle i_{\ast}\eta$
$\displaystyle=$ $\displaystyle 0,$

for every $X\in\mathfrak{X}(M_{j}^{n_{j}})$. Therefore, each $f_{j}$ reduce
codimension to one, and by rigidity, all of them must be the canonical
inclusion. We conclude that $f$ is a multi-rotational submanifold with profile
$f_{0}$, where $f_{0}$ is a smooth curve determined by the vector field $T$. ∎

## References

  * [1] M. Dajczer, R. Tojeiro, Submanifold Theory. Beyond an Introduction. Universitext, Springer, New York, 2019.
  * [2] M. Dajczer, R. Tojeiro, Submanifolds of constant sectional curvature with parallel or constant mean curvature. Tohoku Math. J. (2) 45 (1993), 43–49.
  * [3] F. Dillen, J. Fastenakels, J. Van der Veken, Rotation hypersurfaces in $\mathbb{S}^{n}\times\mathbb{R}$ and $\mathbb{H}^{n}\times\mathbb{R}$, Note di Matematica 29 (2008), 41–54.
  * [4] A. J. Di Scala, Minimal immersions of Kahler manifolds into Euclidean spaces. Bull. Lond. Math. Soc. 35 (2003), 825–827.
  * [5] S. Hiepko, Eine innere Kennzeichnung der verzerrten Produkte. Math. Ann. 241 (1979), 209–215.
  * [6] B. Leandro, R. Pina, J. P. dos Santos, Einstein hypersurfaces of $\mathbb{S}^{n}\times\mathbb{R}$ and $\mathbb{H}^{n}\times\mathbb{R}$. Bull. Braz. Math. Soc. 52 (2021), no. 3, 537–546.
  * [7] J. H. Lira, R. Tojeiro, F. Vitório, A Bonnet theorem for isometric immersions into products of space forms. Archiv der Math. 95 (2010), 469–479.
  * [8] F. Manfio, R. Tojeiro, Hypersurfaces with constant sectional curvature in $\mathbb{S}^{n}\times\mathbb{R}$ and $\mathbb{H}^{n}\times\mathbb{R}$, Illinois J. Math., 55 (2011), no. 1, 397–415.
  * [9] F. Manfio, N. Turgay, A. Upadhyay, Biconservative submanifolds in $\mathbb{S}^{n}\times\mathbb{R}$ and $\mathbb{H}^{n}\times\mathbb{R}$. J. Geom. Anal. 29 (2019), 283–298.
  * [10] B. Mendonça, R. Tojeiro, Umbilical submanifolds of $\mathbb{S}^{n}\times\mathbb{R}$. Canad. J. Math. 66 (2014), no. 2, 400–428.
  * [11] R. Molzan, Extrinsische Extrinsische Produkte und symmetrische Untermannigfaltigkeiten in Standard räumen konstanter und konstanter holomorpher Krümmung. Dissertation, Köln (1983).
  * [12] J. D. Moore, Isometric immersions of Riemannian products. J. Differential Geom. 5 (1971), 159–168.
  * [13] S. Nolker, Isometric immersions of warped products. Differ. Geom. Appl. 6 (1996), 1–30.
  * [14] S. Nolker, Isometric immersions with homothetical Gauss map. Geom. Dedicata 34 (1990), 271–280.
  * [15] C.-R. Onti, Einstein submanifolds with parallel mean curvature. Arch. Math. 110 (2018), 523–531.
  * [16] P. J. Ryan, Homogeneity and some curvature conditions for hypersurfaces. Tohoku Math. J. (2) 21 (1969), 363–388.
  * [17] R. Tojeiro, On a class of hypersurfaces in $\mathbb{S}^{n}\times\mathbb{R}$ and $\mathbb{H}^{n}\times\mathbb{R}$, Bull. Braz. Math. Soc. 41 (2010), no. 2, 199–209.

Institute of Mathematics and Computer Science
University of São Paulo, São Carlos, Brazil
E-mail address<EMAIL_ADDRESS><EMAIL_ADDRESS>


# Explaining Reinforcement Learning Agents Through Counterfactual Action
Outcomes

Yotam Amitai∗, Yael Septon∗, Ofra Amir
Faculty of Data and Decision Sciences, Technion - Israel Institute of
Technology
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Explainable reinforcement learning (XRL) methods aim to help elucidate agent
policies and decision-making processes. The majority of XRL approaches focus
on local explanations, seeking to shed light on the reasons an agent acts the
way it does at a specific world state. While such explanations are both useful
and necessary, they typically do not portray the outcomes of the agent’s
selected choice of action. In this work, we propose “COViz”, a new local
explanation method that visually compares the outcome of an agent’s chosen
action to a counterfactual one. In contrast to most local explanations that
provide state-limited observations of the agent’s motivation, our method
depicts alternative trajectories the agent could have taken from the given
state and their outcomes. We evaluated the usefulness of COViz in supporting
people’s understanding of agents’ preferences and compare it with reward
decomposition, a local explanation method that describes an agent’s expected
utility for different actions by decomposing it into meaningful reward types.
Furthermore, we examine the complementary benefits of integrating both
methods. Our results show that such integration significantly improved
participants’ performance.

**footnotetext: These authors contributed equally to this work.

## Introduction

The behavior of reinforcement learning (RL) agents is complex - these agents
operate in large, stochastic settings, and are trained through rewards
received during their interaction with the environment, which is often sparse.
Moreover, state-of-the-art RL models often utilize deep learning, resulting in
complex models that can be hard to interpret, making the task of explaining
such an agent’s behavior even more complicated. While supporting people’s
understanding of the decision-making of RL agents is challenging, it is
important for facilitating effective collaboration and appropriate trust, in
particular in high-stakes domains [1] such as healthcare or transportation.

Numerous approaches for explainable reinforcement learning (XRL) have been
proposed in recent years [2]. Some methods aim to explain _local_ decisions of
the agents in particular world-states. These include saliency maps that show
what features of the state the agent attends to [3], causal action graphs [4]
that are based on a causal model of the agent’s policy, or reward
decomposition [5] which depict the agent’s expected reward for different
components of the reward function, among others. In contrast to these local
explanation methods, a complementary set of explanation approaches aim to
convey the _global_ policy of the agent, that is, describe its overall
strategy, behavior, or how it behaves in different regions of the state space.
Such approaches include policy summaries [6] that demonstrate agent behavior
in selected world-states based on some criteria or learning a decision tree
that approximates the agent’s policy [7].

In this work, we focus on helping laypeople understand and anticipate agent
preferences by depicting the trade-offs between alternative actions. We
propose a new method for visualizing the outcomes of actions not taken by the
agent as a means to explain their chosen action. We introduce “COViz” –
Counterfactual Outcome Visualization***Code repository:
https://github.com/yotamitai/COViz, a novel local explanation method that
looks beyond the agent’s decision, to its manifestation. By displaying the
outcomes of both actions side-by-side, COViz provides insight into the
consequences of the agent’s decision. Observing the outcomes (“what if?”) can
be beneficial for understanding agent behavior or preferences. The contrastive
nature of the explanation is in line with the literature on explanation in the
social sciences which show that people typically provide and prefer
explanations that contrast alternatives [8].

COViz draws inspiration from global policy summaries [6]. Similar to policy
summaries, COViz conveys agent behavior by showing trajectories of the agent
acting in the environment. It is therefore natural to incorporate COViz in
policy summaries. We propose a simple extension to COViz for generating global
explanations augmented by COViz which selects summary states based on
alternative action outcomes.

The idea of presenting action outcomes naturally complements many other local
explanation methods that focus on state-limited reasoning. In particular, this
approach pairs well with methods that present the agent’s motivation for
choosing a specific action (“why?”), such as reward decomposition [5] which
quantifies the agent’s expectations regarding the utility of different
actions.

To evaluate the contribution of COViz to people’s understanding of agent
preferences, we conducted two user studies in which participants were asked to
characterize the reward function of the agent by ranking its preferences.
Participants were presented with explanations in the form of counterfactual
outcomes (COViz), reward decomposition, or a combination of both approaches.
We examined this in the context of multiple different state selection methods,
based on global explanations approaches. Both studies showed that combining
reward decomposition with COViz improved participants’ understanding, compared
to showing only one of the explanations. This result suggests that while local
explanation methods are typically examined in isolation, their contributions
may be complementary, and thus integrating multiple approaches has the
potential to benefit users.

Our contributions are fourfold: (1) We introduce COViz, a new local
explanation method for highlighting trade-offs between alternative courses of
action that an agent considers; (2) We propose a way to generate global
explanations augmented by COViz to incorporate alternative outcomes; (3) We
integrate COViz with reward decomposition, and (4) We conduct user studies
showing both that our new method is beneficial as of itself, and that its
integration with reward decomposition improves users’ understanding of agent
behavior compared to each method separately.

## Related Work

Our work draws inspiration from _global_ RL explanation approaches. These aim
to provide users with a higher-level description of the agent by attempting to
portray its behavior, strategy, capabilities, or logic [9, 6, 10]. One
approach for conveying the global behavior of an agent is _Agent Strategy
Summarization_ [6]. In this paradigm, the agent’s policy is demonstrated
through a carefully selected set of world states. The strategy summarization
objective is to select the subset of state-action pairs that best describes
the policy of the agent. The criteria for selecting states can vary based on
the summary objective, e.g., state importance [11, 12, 13] or using machine
teaching approaches [14, 9].

Building upon this approach, the DISAGREEMENTS algorithm [15] portrays the
diverging trajectories of two agents upon reaching a disagreement between them
on how best to proceed from a given state. It provides a side-by-side
comparison of the difference in outcomes between the agents, constituting a
method for agent comparison and behavior difference evaluation. The COViz
method proposed in this paper builds on and extends the DISAGREEMENTS
algorithm both for the single-agent use case and for generating a _local_
explanation.

Local explanations in XRL seek to describe why a particular choice of action
was made by a given agent in a specific world-state [16, 17], for example by
using saliency maps to identify what elements of the environment the agent
pays attention to [18, 19, 20] or generating causal explanations by
constructing the agent’s action graph [21]. Juozapaitis et al. (2019) use
reward decomposition to provide insights into the agents’ expected utility by
showing users a decomposition of the environment’s reward into a sum of
meaningful reward types. While this approach shows what the agent expects in
terms of utility, it does not show the actual outcomes of its actions. In
contrast, COViz demonstrates the tradeoff between actions in terms of
trajectories rather than expected reward types.

While growing, there is still an evident lack of research on how to combine
different explanation methods. Anderson et al. (2019) presents a user study
that investigates the impact of explanations on non-experts’ understanding of
reinforcement learning agents. They investigate both a common RL
visualization, saliency maps, and reward decomposition bars. They designed a
four-treatment experiment to compare participants’ mental models of an RL
agent in a simple Real-Time Strategy game. Their results showed that the
combination of both saliency and reward bars was needed to achieve a
statistically significant improvement in the mental model score over the
control. Other works integrated global explanations in the form of policy
summaries with local explanations in the form of saliency maps [13] or reward
decomposition [23]. These studies showed that adding reward decomposition
explanations improved user performance compared to showing only summaries
chosen by HIGHLIGHTS [11] or by frequency-based criteria.

In this work, we introduce a new local explanation method and focus on reward
decomposition as a baseline for comparison, since it also illustrates
tradeoffs between actions, and because saliency maps were shown to be less
helpful [13]. Other approaches such as causal explanations typically require
additional information (e.g., a causal graph) and cannot be implemented
directly. Additionally, we show these methods can be combined and why their
explanations complement each other.

## Background

We assume a Markov Decision Process (MDP) setting. Formally, an MDP is defined
by a tuple $<S,A,R,Tr>$ where $S$ is the set of states, $A$ is the set of
actions, $R(s,a,s^{\prime}):S\times A\times S\rightarrow\mathbb{R}$ is the
reward received after transitioning from state $s$ to state $s^{\prime}$, due
to action $a$ and $Tr$ is a transition probability function
$Tr(s,a,s^{\prime}):S\times A\times S\rightarrow[0,1]$ defining the
probability of transitioning to state $s^{\prime}$ after taking action $a$ in
$s$.

The Q-function is defined as the expected value of taking action $a$ in state
$s$ under a policy $\pi$ throughout an infinite horizon while using a discount
factor $\gamma$:
$Q^{\pi}(s,a)=\mathrm{E}^{\pi}[\sum_{t=0}^{\inf}\gamma^{t}R_{t+1}|s_{t}=s,a_{t}=a]$.

### The Highway Environment

In the highway environment (Figure 1) the RL agent controls a green vehicle
that navigates a multi-lane highway while driving alongside and interacting
with other (blue) vehicles. The agent can choose to accelerate, decelerate,
turn left, turn right or stay idle. The environment can be modified for the
number of lanes, vehicle speed range, the number of vehicles, vehicle density,
and more. Throughout the paper, we will refer to the lanes from the
perspective of the agent, e.g. the bottom-most lane will be denoted as the
right-most lane.

A key property of this environment is its lack of an inherent goal that must
be reached, instead, an agent’s reward is determined by its driving regime.
This allows for designing agents with divergent, yet, reasonable strategies as
no ground-truth “optimal” behavior exists. Additionally, the environment is
both easily understandable to laypeople and represents a relatable real-world
scenario.

We focus on a single domain in the current study, as both reward decomposition
and approaches similar to COViz (e.g., HIGHLIGHTS, DISAGREEMENTS) have been
shown to generalize to additional domains in prior work [11, 5, 22, 12, 15,
23].

### Reward Decomposition

Hierarchical Reward Architecture (HRA) [24] decomposes the agent’s reward
function into distinct components, each associated with a specific type of
reward (e.g., crashing or changing lanes in the highway environment). This
allows the Q-function to be approximated more easily by a low-dimensional
representation, which can lead to faster learning. Technically, in the context
of deep reinforcement learning, this architecture is implemented in a network
that has multiple heads (one for each reward component) that share the lower
network layers.

Juozapaitis et al. (2019) used this approach as the basis for a local
explanation approach, called _Reward Decomposition_. They present users with
Reward Decomposition bars which show the Q-values for each reward component
for alternative actions. This aims to reveal the agents’ preferences and
predictions to users.

Figure 1: The highway environment. The agent controls the green vehicle.

## Counterfactual Outcomes Visualization

According to cognitive science literature, one of the key features of “good”
explanations is that they are contrastive [8]. Several XRL approaches chose to
leverage some form of contrastive information in their explanation techniques
[25, 26, 27]. Counterfactual explanations are a form of contrastive
explanations. A counterfactual explanation provides an answer to the question
“Why $p$ rather than $q$?”, where $p$ is the fact which occurred and $q$ is
some hypothetical foil which the user might have expected to occur, but did
not [28].

Many explanation techniques are not contrastive and concentrate primarily on
the agent’s chosen actions (_fact $p$_), without considering alternative
foils. One policy-summarization (global explanation) approach that does
provide contrastive information is the DISAGREEMENTS algorithm [15]. It does
so by comparing two different agents to one another and visually portraying
their points of disagreement.

The DISAGREEMENTS algorithm included an approach for quantifying the
importance of a disagreement between the agents at a given state, by comparing
their trajectories that branch out of it. The method that proved most
efficient for quantifying the differences in these trajectories was comparing
the values of the last state (in a limited trajectory) reached by each of
them.

We build upon the contrastive nature of DISAGREEMENTS and the idea of focusing
on the outcomes of alternative trajectories to devise a local explanation
method. Specifically, We adapt the DISAGREEMENTS algorithm to a single agent
use case and visually depict trajectories at a given state, each associated
with a distinct action available to the agent. In practice, these alternative
trajectories are obtained by forcing the agent to take a particular action and
then allowing it to progress based on its policy for another $k$ steps. These
trajectories visualize different paths the agent _could have_ taken (_foil
$q$_), had it _not_ chosen the specific action that it had (_fact $p$_).

We assume that the most relevant counterfactual action for the agent is its
second-best choice, as this represents the most likely alternative and often
provides the most informative contrast for nonexpert users. While the current
focus is on this second-best action, the algorithm can be adapted through the
$CF_{Meth}$ parameter (See table 1) to consider other counterfactual options,
such as the ’worst’ action or a user-selected action, which we acknowledge as
a potential area for future investigation.

Local | $k$ |  | Trajectory length, i.e. the number of states succeeding
---
the state where the counterfactual action was initiated.
7
$Nsim$ |  | The number of simulations (execution traces) to run
---
for collecting trajectories.
200
$CF_{Meth}$ |  | Counterfactual action, i.e. the method by which to
---
choose the counterfactual action.
SecondBest
Global | $n$ |  | Summary budget, i.e. number of trajectories to
---
include in output summary.
4
$overlap$ |  | Maximal number of shared states allowed between
---
two trajectories in the summary.
5
$I_{Meth}$ |  | Importance method used for evaluating and ranking
---
the summary states.
| Last-State
---

Table 1: COViz algorithm parameters and study values.

#### The COViz algorithm.

The pseudo-code for the algorithm is given in Algorithm 1 and parameters are
summarized in Table 1 along with their user study values.

The algorithm works as follows: Two empty lists are initialized to account for
the execution traces and counterfactual trajectories (lines 3–4). For as many
traces as defined, the simulation is initialized and the agent executes its
policy (lines 5–22). At each simulation state $s_{i}$, we obtain both the
agent’s preferred and counterfactual action (lines 9–10). a counterfactual
trajectory is obtained by having the agent initiate the counterfactual action
and progress according to its policy for another $k$ steps (lines 11–16). The
counterfactual trajectory is stored and the agent is reverted back to $s_{i}$
to progress with its preferred action (lines 17–20). Each state is added to
the trace. Once all simulations are completed, traces and counterfactual
trajectories are passed to $CFTrajPairs$ where each state is associated with
both its true future trajectory and paired to the corresponding counterfactual
trajectory, (line 24). The algorithm output is a set of pairs of true and
counterfactual trajectories for each state in each trace.

_Trajectory length $k$._ The number of steps to include in the counterfactual
trajectory ($k$) was based on the specific environment (highway)
implementation, – too few steps result in not enough time to showcase the
counterfactual difference while raising the number of steps increases the
influence of states succeeding the one being explained.

_Complexity._ For each state reached by the agent in trace $t$ we generate a
counterfactual trajectory of length $k$. Therefore the complexity is
$|t|\times k$, for each trace. Importantly, these computations are done
offline (once) and the choice of how many traces to simulate is configurable.

Algorithm 1 Counterfactual Outcome Visualizations

1: Input: $\pi,k,Nsims,CF_{Meth},I_{Meth},n,overlap$

2: Output: $CO$

3: $Traces\leftarrow$ empty list  #Execution traces

4: $C_{T}\leftarrow$ empty list  #Counterfactual trajectories

5: for $i=1$ to $Nsims$ do

6: $t\leftarrow$ empty list  #Current trace

7: $sim,s=InitializeSimulation()$

8: while $(!sim.ended())$ do

9: $a^{\pi}\leftarrow sim.getBestAction(\pi(s))$

10: $a^{C}\leftarrow sim.getCFAction(\pi(s),CF_{Meth})$

11: $c_{t}\leftarrow$ empty list  #Counterfactual trajectory

12: for $j=1$ to $k$ do

13: $s^{C}\leftarrow sim.advanceState(a^{C})$

14: $a^{C}\leftarrow sim.getAction(\pi(s^{C}))$

15: $c_{t}.add(s^{C})$

16: end for

17: $C_{T}.add(c_{t})$

18: $sim,s=reloadSimulation(s)$

19: $s\leftarrow sim.advanceState(a^{\pi})$

20: $t.add(s)$

21: end while

22: $Traces.add(t)$

23: end for

24: $CO\leftarrow CFTrajPairs(Traces,C_{T})$

25: Global Extension:

26: $\mathrm{S}\leftarrow topImpTraj(CO,I_{Meth},n,overlap)$

As opposed to DISAGREEMENTS, which only compared conflicting states between
the agents, the COViz algorithm generates a counterfactual trajectory at
_each_ step during execution. While this method does not explicitly answer the
original “why” question, it does enable the user to implicitly infer
information about the agent’s preferences by its choice of action and to
observe the short-term alternative outcomes of these.

_Depicting counterfactuals._ In terms of visualization, we adopt
DISAGREEMENTS’s visual comparison approach of displaying the counterfactual
agent as a red rectangle originating around the true agent and moving away as
their trajectories diverge (see Figure 2). This visualization has been shown
to be efficient and satisfying for users [15].

#### Global Extension to COViz.

The set of counterfactual trajectory pairs obtained by COViz is stored such
that a user could ask to view them for any agent-visited state. Alternatively,
these can be integrated into a policy summary, showing the most meaningful
pairs of counterfactual trajectories. This extension is depicted in Algorithm
1 using the $topImpTraj$ function, (line 26).

After collecting counterfactual trajectories for each of the execution traces’
states, their importance is evaluated to generate a ranking. The top-$n$
ranked states will then be used to construct the output summary. To determine
the importance of a state $s_{i}$, we compare the two trajectories that branch
out of it, 1) the one chosen (fact $p$) and 2) the counterfactual (foil $q$).
Importance is then calculated via the importance method $I_{Meth}$. Many
criteria can be used to determine trajectory pair importance. We focus on the
_Last-State Importance_ metric proposed in [15], which evaluates the
significance of the originating state $s_{i}$ based on the last state reached
by the compared trajectories. Formally:

$Im(s_{i})=|V(s^{p}_{i+k})-V(s^{q}_{i+k})|$

Where $s^{p}_{i+k},s^{q}_{i+k}$ denote the states reached by the agent
following $k$ steps after selecting the fact($p$) or foil($q$) in state
$s_{i}$ respectively. This measure utilizes the agent’s inherent value
function $V(s)$ to describe the estimated utility loss of choosing the foil
over the fact (i.e. optimal action). This reflects how “far off” from the
original plan the counterfactual action has led the agent.

The global parameter values (Table 1) were chosen based on pilot user studies
used to design and optimize the final study. For instance, parameter $n$
which, when raised, increased study duration and cognitive load.

#### Integrating Counterfactual Outcomes with Reward Decomposition.

COViz and reward decomposition naturally complement each other. COViz answers
the question “What if?” by depicting alternatives, while reward decomposition
answers the question “why?” an action was chosen based on the quantification
of the expected utility. For a given state, we can integrate the approaches by
showing both the reward decomposition bars for the chosen action and the
counterfactual action, as well as presenting the outcome of these choices by
showing the trajectories stemming from them. An example of the integration of
COViz and reward decomposition can be seen in Figure 2.

## Empirical Methodology

To evaluate the benefits of COViz and its integration with reward
decomposition, we conducted two user studies asking participants to predict
preference differences between agents. Additional experiments were conducted
to compare different approaches for state selection for the studies, as
further described below.

### Agent Training

The agents trained for the experiment were optimized to have preference
differences in their driving regimes. Their reward functions specified
positive rewards with the following behaviors: changing lanes (CL), high-speed
driving (HS), and driving in the right-most lane (RML). All agents received a
negative reward for reaching a collision state. To account for each positive
reward type, the number of components in the reward decomposition was set to
$|C|=3$.

Three agents were trained with different preferences over the reward types
(see Table 2). No future rewards are obtained upon ending the simulation and
all agents received the same negative reward for collision. We experimented
with different reward-type values to get qualitatively different agent
behaviors.

Agents were trained using a double deep Q-Network (DDQN) [29] architecture for
2,000 simulations, each with a maximum of 80 steps. The network input is the
observation state represented by an array of size 25 (5X5). The input layer is
followed by two fully connected hidden layers of length 256. The final layer
connects to three different heads, each of which is designed to account for a
specific desired behavior we specified above. Each head consists of a linear
layer and outputs a Q-value vector of length of 5 that predicts the Q-value
for each of the possible actions: moving left (upwards), idle, moving right
(downwards), going faster, going slower. Our implementation is based on two
open-source repositories: the Highway environment and its compatible agent
architectures [30, 31].

### State Selection

To extract the counterfactual outcomes, we ran 200 simulations of each trained
agent and saved their traces. States extracted from these traces were used in
the study. The main study reported here used the last-state value method
described in the COViz global extension section.

In addition, we experimented with two alternative methods for determining
state importance (parameter $I_{Meth}$).

One approach for quantifying a state’s importance is to consider the
differences in Q-values between different actions [11]. This is usually
restricted to comparing the agent’s preferred action with its second-best or
least preferred action. However, this approach proved unsuitable for our
studies with COViz, as states selected using this criterion highly
incentivized the agents to return to their original path after deviating with
a single action, rendering the counterfactual trajectories almost identical to
the true ones. We note this effect is domain-dependent – in some domains it is
easy to return to the original policy after deviating, while in other domains
it may not be possible.

Another approach for selecting states is sampling them based on the frequency
of the agent encountering the states, thus reflecting the agent’s “typical”
behavior. Technically, this can be done by randomly sampling states from
execution traces. We ran a replication of our main study with this approach
(See supplementary material***Supplementary material can be found at
https://t.ly/AhrTf for complete information, and a short summary in the
Results section).

Reward Type | CL | HS | RML | Collision
---|---|---|---|---
Agent 1 | 3 | 1 | 8 | -3
Agent 2 | 5 | 8 | 1 | -3
Agent 3 | 8 | 1 | 5 | -3
Table 2: User-study agents’ reward-type values. Figure 2: User study
explanation types. A+B. Counterfactual Outcomes (COViz): videos depicting the
different outcomes of counterfactual actions. A and B depict the start and end
frame of the video respectively; C. Reward Decomposition (RD): a graph
presenting the value each reward component contributed to the chosen action at
the portrayed state. Integrated method (CORD): The COViz outcome video and RD
graph together. RD graph portrays both the true and counterfactual action.

### Study Design

We conducted two studies to evaluate the contribution of COViz and its
integration with reward decomposition to people’s understanding of agents’
preferences. Study 1 was a between-subject study, where participants were
exposed to a single type of explanation. This design avoids learning effects
but suffers from higher variance due to differences between study participants
and does not allow to elicit users’ subjective preferences. To address this
limitation, study 2 used a within-subject design where participants were
exposed to all explanation methods, and could thus compare them directly. Both
studies were approved by our institutional review board.

_Experimental conditions_. The studies included three local explanation types:
Counterfactual outcomes (COViz), reward decomposition (RD), and their
integration (CORD). Examples of explanation types can be seen in Figure 2.
Additionally, a control condition of summaries without local explanations
(SUM) was tested in study 1. In study 1, participants were assigned a
condition and observed all agents using their assigned explanation. In study
2, participants saw each agent with a different, randomly assigned,
explanation type. Since it focused on participants’ preferences for local
explanations, we did not include the summary-only control in this study. The
ordering of the COViz and RD explanations was random, however, the combined
method was always shown last as it was more natural to show it after each
method appeared separately. While this may introduce a learning effect, study
1 does not suffer from this confound.

_Task._ In both studies, participants’ task was to assess the behavioral
preferences of the different agents. For each agent, participants observed
four scenarios (i.e., states selected as explained above) and their
explanations. Since the Q-values of each agent are different, the “important”
states differ as well. However, for a specific agent, all participants viewed
the same scenarios. After participants observed the explanation of the agent’s
behavior in each scenario, they were asked to predict the agent’s preferences
between two reward components, e.g. ”based on the video, which component do
you think the agent (green car) cares about more, high-speed or lane change?”
(Full surveys in supplementary material). Participants could return and review
the explanations while answering the questions. The agents’ order and the task
answers were randomly presented.

After each task, participants rated their agreement on a 7-point Likert scale
[32] with statements adapted from Hoffman’s explanation satisfaction
questionnaire [33]. These adaptations included specifying the task at hand
rather than using general terms like ’goal’ or ’task’, and removing items
irrelevant to our study objectives (see supplementary for full scale).
Notably, despite these minor modifications, the internal consistency of our
adapted scale was confirmed, yielding high Cronbach’s Alpha values ($>$0.9)
across all studies.

Then, participants were asked to rate their confidence in each answer on a
Likert scale from 1 (“not confident at all”) to 5 (“very confident”) and they
had the option to describe their reasoning in a free-text response. In study
2, where participants saw multiple explanation types, they were also asked to
rank their preferences among them and provide textual feedback regarding how
each method helped them.

_Procedure._ In both studies, participants were first introduced to the
highway environment. Then, participants learned about their condition’s
explanation method along with an example to assist in interpreting the
visualized information. In study 1, participants were presented with only one
of the four explanation types: Sum, COViz, RD, or CORD. The explanation type
was randomly assigned and consistent across all three tasks. In study 2,
before each task, instructions and information regarding the upcoming
explanation method were provided, after which, a demonstration for the task
they would be asked to complete was shown. Prior to starting the task itself,
participants were quizzed on their understanding and only allowed to proceed
once answering all questions correctly. Each task included an attention check
question. Finally, participants were asked to provide demographic information
and rate their proficiency in AI. All study materials including the specific
instructions for each explanation method can be found in the supplementary.

_Participants_. We recruited participants through Amazon Mechanical Turk
($N_{1}=147$ and $N_{2}=50$ for study 1 and study 2, respectively). On
average, study participants self-proclaimed low proficiency in AI which aligns
with the purpose of explanations for laypeople. Participants were native
English speakers, ages 18 - 55 from the US, UK, or Canada. Participants
received a $3 base payment and an additional bonus 30-cent bonus for
identifying the preferences of each of the agents correctly. Participants who
failed the attention question were excluded from the final analysis, as well
as participants who completed the survey in less time than two standard
deviations of the mean completion time. After exclusions, we had 115
participants (SUM=27, COViz=30, RD=31, CORD=27) in study 1 and 49 participants
in study 2 with a mean age of 39 & 37, and a female count of 39 & 27 for study
1 and 2 respectively. We note that the number of participants in each
experiment aligns with previous user studies in XRL, e.g. [34, 35].

## Results

The objective measure used in our analysis is the mean fraction of correct
reward component comparisons, i.e., participants’ correctness rate, for each
condition. We used the non-parametric Mann-Whitney $U$ test in the statistical
analyses. We note that there were no differences between different demographic
groups (age or gender).

Overall, the integration of the two explanation types improved participants’
ability to asses the agents’ preferences. These results were replicated in
both studies. In study 1, the integration method of combining both explanation
types CORD (M= 0.64) led to significantly improved performance compared to RD
or COViz alone (RD: M=0.45, RD vs. CORD: $U=204,p<0.001$; COViz: M=0.42, COViz
vs. CORD: $U=657.5,p<0.001$). Comparing to SUM (M= 0.31) there was a
significant improvement in the performance as well ($U=607,p=0.002$). The
difference between COViz and RD was not significant ($p=0.22$). Similarly, in
study 2, the combination of explanation types, CORD (M= 0.63), significantly
improved participants’ performance compared to RD or COViz (RD: M=0.48, RD vs.
CORD: $U=767,p<0.001$ and COViz: M= 0.4, COViz vs. CORD: $U=1814.5,p<0.001$).
Here too, the difference between COViz and RD was not significant ($p=0.064$).

Moreover, in our experimentation with frequency-based state selection, we
found that all local explanation methods were useful, with no significant
difference between them, when addressing informative states. However, for less
informative states all explanation methods proved less useful than a coin
toss, except the integrated approach which was still superior (see
supplementary material).

While CORD participants were subjected to more information than the other
conditions, prior work has demonstrated that more information does not
necessarily lead to better outcomes [22], implying that integration between
these methods successfully captures relevant and complementary information for
performance without imposing excessive cognitive load.

In study 1, when breaking down the results by agent, the trend was consistent
with the overall results. The greatest improvement from the integration of the
two approaches was for Agent 1 (Figure 3). We note that Agent 1 is a tricky
agent as its second preference - being in the right-most lane is correlated
with its first preference changing lanes. This shows that the combination of
the two explanations has a significant impact, especially for challenging
behaviors. Participants’ comments also mentioned the benefit of combining the
different explanations. For instance, one participant wrote “The two
[explanations] together helped me understand but separately I was a lot more
confused”. Another comment was “The graphs help to visualize how the agent is
rewarded for behaviors while the videos show the results of rewarding (or
prioritizing) those actions for the agent.”

In study 2, we asked participants to rank the three explanation types.
Participants differed in their preferences. Some preferred the video-based
approach of COViz, e.g., “video is always easy for me to comprehend”, while
others preferred the reward bars: “The graph is more efficient”. A participant
that preferred the combination wrote: “It is more instructive to both see the
component values and then watch the video. ” The overall ranking results show
that COViz was ranked first by 28 participants ($57\%$ of participants), while
the combination of the two explanation types was ranked in last place by 26
participants ($53\%$ of participants) as shown in Figure 4.

For both studies, participants’ confidence and satisfaction ratings were above
the neutral rating ($>3$). Specifically, for study 1 and study 2 the
satisfaction results for each of the conditions were COViz: M=4.9 and 5.16,
RD: M=5.06 and 5.06, CORD: M=5.29 and 5.04 respectively and SUM=5.4 in study 1
(Graphs provided in supplementary). With respect to participants’ confidence,
the ratings by condition observed in studies 1 and 2 were COViz: M=4.04 and
3.93, RD: M=3.8 and 4.02, CORD: M=4.03 and 3.83, respectively). None of the
differences between conditions in terms of satisfaction or confidence were
significant.

(A)

(B)

Figure 3: Participants’ mean success rate in identifying the preferences by
condition and agent in study 1 (A) and study 2 (B). The error bars show the
95% CI. Figure 4: The distribution of participants’ ranking of the three
explanation types

## Discussion and Future Work

This paper presented a new method for explaining RL agents through
counterfactual outcomes, and the benefits of combining them with another local
explanation - reward decomposition. We conducted user studies to evaluate the
contribution of this approach to people’s ability to analyze agent preferences
as well as rating the participants’ preferences of each explanation type. Our
results show that both explanation types, COViz, and reward decomposition,
resulted in better-than-random participant performance, but the integration of
the two enabled participants to reach a significantly higher success rate. It
is not trivial that this would be the case since the benefit might only stem
from one explanation, and the cognitive load might be higher as was observed
in some studies [36]. Indeed, when showing frequency-based chosen states,
COViz or reward decomposition alone did not always result in higher
performance than random guess. However, showing both types of explanations
together did improve performance beyond random guessing.

Even though the combination of the explanation types led to a higher
correctness rate, when participants were asked to rate the explanation types,
the combination was ranked last. We hypothesize that the reason for that is
the mental overload on the participants. For example, one participant
commented that “…the combination is too confusing for me.” When combining two
explanation types participants are required to pay attention to more details.
This finding is in line with previous work stating that subjective
satisfaction and proxy tasks often do not align with objective performance
[37]. A potential future-work solution is to reduce user cognitive load by
cascading the information and allowing users to control the amount they view
at once, thus avoiding overwhelming them and engaging them more.

Our experimentation of state-selection criteria highlighted a vulnerability of
COViz – only a single counterfactual action is constrained. In the generalized
continuous setting, a single action deviation does not necessarily result in a
divergent outcome and is more likely to be negligible or easily reversible.
Future work can alleviate this limitation by enforcing an alternative sequence
of actions rather than forcing only a single deviation from the policy.

In this work, the selection of counterfactual actions was guided by heuristic
methods. Alternatively, this choice can be handed over to users to enhance
interactivity and foster trust, or for purposes of debugging.

Finally, while this work focused on the task of identifying agent preferences,
future work can try to characterize which explanation types are most useful
for different user tasks, and explore the combination of other local
explanation types, including saliency maps, action graphs, and others.

## Acknowledgements

The research was partially funded by the European Research Council (grant
#101078158 \- CONVEY), and by the Israel Science Foundation (grant no.
2185/20). The authors thank Tobias Huber for helpful advice and feedback.

## References

  * Rong et al. [2022] Y. Rong, T. Leemann, T. trang Nguyen, L. Fiedler, T. Seidel, G. Kasneci, and E. Kasneci, “Towards human-centered explainable ai: User studies for model explanations,” 2022.
  * Milani et al. [2022] S. Milani, N. Topin, M. Veloso, and F. Fang, “A survey of explainable reinforcement learning,” _arXiv preprint arXiv:2202.08434_ , 2022.
  * Greydanus et al. [2017] S. Greydanus, A. Koul, J. Dodge, and A. Fern, “Visualizing and understanding atari agents,” _arXiv preprint arXiv:1711.00138_ , 2017.
  * Madumal et al. [2020a] P. Madumal, T. Miller, L. Sonenberg, and F. Vetere, “Explainable reinforcement learning through a causal lens,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 34, 2020, pp. 2493–2500.
  * Juozapaitis et al. [2019] Z. Juozapaitis, A. Koul, A. Fern, M. Erwig, and F. Doshi-Velez, “Explainable reinforcement learning via reward decomposition,” in _IJCAI/ECAI Workshop on Explainable Artificial Intelligence_ , 2019.
  * Amir et al. [2019] O. Amir, F. Doshi-Velez, and D. Sarne, “Summarizing agent strategies,” _Autonomous Agents and Multi-Agent Systems_ , vol. 33, no. 5, pp. 628–644, 2019.
  * Liu et al. [2018] G. Liu, O. Schulte, W. Zhu, and Q. Li, “Toward interpretable deep reinforcement learning with linear model u-trees,” in _Joint European Conference on Machine Learning and Knowledge Discovery in Databases_. Springer, 2018, pp. 414–429.
  * Miller [2019] T. Miller, “Explanation in artificial intelligence: Insights from the social sciences,” _Artificial intelligence_ , vol. 267, pp. 1–38, 2019.
  * Huang et al. [2019] S. H. Huang, D. Held, P. Abbeel, and A. D. Dragan, “Enabling robots to communicate their objectives,” _Autonomous Robots_ , vol. 43, no. 2, pp. 309–326, 2019.
  * Booth et al. [2019] S. Booth, C. Muise, and J. Shah, “Evaluating the interpretability of the knowledge compilation map: Communicating logical statements effectively.” in _IJCAI_ , 2019, pp. 5801–5807.
  * Amir and Amir [2018] D. Amir and O. Amir, “Highlights: Summarizing agent behavior to people,” in _Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems_ , 2018, pp. 1168–1176.
  * Sequeira and Gervasio [2020] P. Sequeira and M. Gervasio, “Interestingness elements for explainable reinforcement learning: Understanding agents’ capabilities and limitations,” _Artificial Intelligence_ , vol. 288, p. 103367, 2020.
  * Huber et al. [2021] T. Huber, K. Weitz, E. André, and O. Amir, “Local and global explanations of agent behavior: Integrating strategy summaries with saliency maps,” _Artificial Intelligence_ , vol. 301, p. 103571, 2021.
  * Lage et al. [2019] I. Lage, D. Lifschitz, F. Doshi-Velez, and O. Amir, “Exploring computational user models for agent policy summarization,” in _IJCAI: proceedings of the conference_ , vol. 28. NIH Public Access, 2019, p. 1401.
  * Amitai and Amir [2022] Y. Amitai and O. Amir, “” i don’t think so”: Summarizing policy disagreements for agent comparison,” _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2022.
  * Tabrez et al. [2019] A. Tabrez, S. Agrawal, and B. Hayes, “Explanation-based reward coaching to improve human performance via reinforcement learning,” in _2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI)_. IEEE, 2019, pp. 249–257.
  * Dodson et al. [2011] T. Dodson, N. Mattei, and J. Goldsmith, “A natural language argumentation interface for explanation generation in markov decision processes,” in _International Conference on Algorithmic Decision Theory_. Springer, 2011, pp. 42–55.
  * Hilton et al. [2020] J. Hilton, N. Cammarata, S. Carter, G. Goh, and C. Olah, “Understanding rl vision,” _Distill_ , vol. 5, no. 11, p. e29, 2020.
  * Huber et al. [2019] T. Huber, D. Schiller, and E. André, “Enhancing explainability of deep reinforcement learning through selective layer-wise relevance propagation,” in _KI 2019: Advances in Artificial Intelligence_ , C. Benzmüller and H. Stuckenschmidt, Eds. Cham: Springer International Publishing, 2019, pp. 188–202.
  * Puri et al. [2020] N. Puri, S. Verma, P. Gupta, D. Kayastha, S. Deshmukh, B. Krishnamurthy, and S. Singh, “Explain your move: Understanding agent actions using specific and relevant feature attribution,” in _8th International Conference on Learning Representations, ICLR_. OpenReview.net, 2020.
  * Madumal et al. [2020b] P. Madumal, T. Miller, L. Sonenberg, and F. Vetere, “Distal explanations for model-free explainable reinforcement learning,” _arXiv preprint arXiv:2001.10284_ , 2020.
  * Anderson et al. [2019] A. Anderson, J. Dodge, A. Sadarangani, Z. Juozapaitis, E. Newman, J. Irvine, S. Chattopadhyay, A. Fern, and M. Burnett, “Explaining reinforcement learning to mere mortals: An empirical study,” _arXiv preprint arXiv:1903.09708_ , 2019.
  * Septon et al. [2023] Y. Septon, T. Huber, E. André, and O. Amir, “Integrating policy summaries with reward decomposition for explaining reinforcement learning agents,” in _International Conference on Practical Applications of Agents and Multi-Agent Systems_. Springer, 2023, pp. 320–332.
  * Van Seijen et al. [2017] H. Van Seijen, M. Fatemi, J. Romoff, R. Laroche, T. Barnes, and J. Tsang, “Hybrid reward architecture for reinforcement learning,” _arXiv preprint arXiv:1706.04208_ , 2017.
  * Silva et al. [2021] C. R. Silva, M. Bowling, and L. H. Lelis, “Teaching people by justifying tree search decisions: An empirical study in curling,” _Journal of Artificial Intelligence Research_ , vol. 72, pp. 1083–1102, 2021.
  * Lin et al. [2020] Z. Lin, K.-H. Lam, and A. Fern, “Contrastive explanations for reinforcement learning via embedded self predictions,” _arXiv preprint arXiv:2010.05180_ , 2020.
  * van der Waa et al. [2018] J. van der Waa, J. van Diggelen, K. v. d. Bosch, and M. Neerincx, “Contrastive explanations for reinforcement learning in terms of expected consequences,” _arXiv preprint arXiv:1807.08706_ , 2018.
  * Lipton [1990] P. Lipton, “Contrastive explanation,” _Royal Institute of Philosophy Supplements_ , vol. 27, pp. 247–266, 1990.
  * Mnih et al. [2015] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski _et al._ , “Human-level control through deep reinforcement learning,” _nature_ , vol. 518, no. 7540, pp. 529–533, 2015.
  * Leurent [2018a] E. Leurent, “An Environment for Autonomous Driving Decision-Making,” 2018. [Online]. Available: https://github.com/eleurent/highway-env
  * Leurent [2018b] ——, “rl-agents: Implementations of reinforcement learning algorithms,” 2018. [Online]. Available: https://github.com/eleurent/rl-agents
  * Likert [1932] R. Likert, “A technique for the measurement of attitudes.” _Archives of psychology_ , 1932.
  * Hoffman et al. [2018] R. R. Hoffman, S. T. Mueller, G. Klein, and J. Litman, “Metrics for explainable ai: Challenges and prospects,” _arXiv preprint arXiv:1812.04608_ , 2018.
  * Huang et al. [2018] S. H. Huang, K. Bhatia, P. Abbeel, and A. D. Dragan, “Establishing appropriate trust via critical states,” in _2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2018, pp. 3929–3936.
  * Sreedharan et al. [2020] S. Sreedharan, S. Srivastava, and S. Kambhampati, “Tldr: Policy summarization for factored ssp problems using temporal abstractions,” in _Proceedings of the International Conference on Automated Planning and Scheduling_ , vol. 30, 2020, pp. 272–280.
  * Anderson et al. [2020] A. Anderson, J. Dodge, A. Sadarangani, Z. Juozapaitis, E. Newman, J. Irvine, S. Chattopadhyay, M. Olson, A. Fern, and M. Burnett, “Mental models of mere mortals with explanations of reinforcement learning,” _ACM Transactions on Interactive Intelligent Systems (TiiS)_ , vol. 10, no. 2, pp. 1–37, 2020.
  * Buçinca et al. [2020] Z. Buçinca, P. Lin, K. Z. Gajos, and E. L. Glassman, “Proxy tasks and subjective measures can be misleading in evaluating explainable ai systems,” in _Proceedings of the 25th international conference on intelligent user interfaces_ , 2020, pp. 454–464.