††institutetext: Stanford Institute for Theoretical Physics and Department of
Physics,
Stanford University, Stanford, CA 94305, USA

# Ward Identities for Superamplitudes

Renata Kallosh

###### Abstract

We introduce Ward identities for superamplitudes in D-dimensional
$\mathcal{N}$-extended supergravities. These identities help to clarify the
relation between linearized superinvariants and superamplitudes. The solutions
of these Ward identities for an $n$-partice superamplitude take a simple
universal form for half BPS and non-BPS amplitudes. These solutions involve
arbitrary functions of spinor helicity and Grassmann variables for each of the
$n$ superparticles. The dimension of these functions at a given loop order is
exactly the same as the dimension of the relevant superspace Lagrangians
depending on half-BPS or non-BPS superfields, given by $(D-2)L+2-\mathcal{N}$
or $(D-2)L+2-2\mathcal{N}$, respectively. This explains why soft limits
predictions from superamplitudes and from superspace linearized
superinvariants agree.

##  1 Introduction

Supersymmetry and R-symmetry Ward identities impose linear relations among
$n$-point on-shell supergravity amplitudes. The superamplitudes satisfying
Ward identities are consistent with linearized supersymmetry.

It is also possible to study linearized supersymmetry constraints in
supergravity using superinvariants. These are integrals over linearized
superspace of Lagrangians depending on linearized superfiedlds. In this case,
a dimensional analysis is available, and it leads to specific predictions with
regard to candidate counterterms and soft scalar limits.

A relation between superamplitudes and linearized superinvariants is known in
various examples. Here, we will present a universal relation that exists
between these two approaches to linearized supersymmetry in various
dimensions.

In 4D supergravities, Ward identities for multi-point superamplitudes are
known in maximal supergravity case Bianchi:2008pu . Explicit solutions to
these Ward identities were given in Elvang:2009wd . The method developed in
Elvang:2009wd  to produce superamplitudes, solutions of Ward identities for
$\mathcal{N}=8$, was adapted to 4D supergravities with $\mathcal{N}=5,6$ in
Freedman:2018mrv . The purpose of studying 6-point superamplitudes in
Freedman:2018mrv  was to understand their soft scalar behavior. It was also
observed in Freedman:2018mrv  that the study of local superinvariants gave the
same results as the actual solutions of Ward identities for 6-point
superamplitudes in all cases $\mathcal{N}\geq 5$.

Ward identities for superamplitudes were explored in general for higher
dimensions; see, for example, Boels:2012ie  and references therein. In type
IIB 10D supergravity in Wang:2015jna ; Green:2019rhz  the form of the Ward
identities for the amplitudes in maximal 10D supergravity was proposed and
investigated. Here we will generalize it to $D\geq 4$ $\mathcal{N}$-extended
supergravities.

Recently, a set of local 6-point superinvariants was studied in
Kallosh:2023dpr  in 6D maximal supergravity. It was observed111C. Wen, Y.-t.
Huang, private communication and Kallosh:2023dpr . that the properties of
superamplitudes in 10D and superinvariants in 6D are consistent.

The purpose of this note is to propose a form of Ward identities for multi-
point superamplitudes for supergravities in diverse integer dimensions. In
particular, this will make clear the relation between linearized
superinvariants and superamplitudes for D-dimensional supergravities with
$\mathcal{N}$-extended supersymmetries.

Our main applications of the studies of Ward identities for amplitudes will be
with regard to cases where there is no 1-loop anomaly in R-symmetry. Namely,
in 4D, there are no 1-loop R-symmetry anomalies at $\mathcal{N}\geq 5$, and in
6D, there is no 1-loop R-symmetry anomaly in maximal supergravity
Marcus:1985yy . But in both cases, the status of soft scalar limits of the
6-point amplitudes, i.e., the status of 1-loop $\mathcal{G}$-symmetry
anomalies, has not been established.

We will find out that the solution of the Ward identities for local amplitudes
combined with dimensional analysis directly gives correct results concerning
soft scalar limits without the need to perform complicated calculations of the
explicit 6-point superamplitudes.

##  2 Superamplitudes and Ward Identities in D-dimensional supergravity

Consider multi-point superamplitudes, which are constrained by the requirement
of linearized supersymmetry. Let us first ignore the issue of dimensions, loop
order, and power of gravitational coupling. We just require the $n$-point
superamplitudes to be linearly supersymmetric, i.e., satisfy Ward identities.
The nice form of on-shell supersymmetry Ward identities for amplitudes in 10D
type IIB maximal supergravity was given in Wang:2015jna . We generalize it
here for other dimensions and $\mathcal{N}$-extended supergravities.

Following Tanii:1998px  we describe the field contents of ungauged
supergravities as determined by irreducible representations of the super
Poincaré algebras. There are translations $P_{a}$, generators of Lorentz
transformations $M_{ab}$, supercharges $Q^{\alpha\underline{i}}$, and
generators of automorphism group $T^{A}$. In cases of massless states, there
are no central charges.

$[M_{ab},Q^{\underline{i}}]={1\over 2}\gamma_{ab}Q^{\underline{i}}\
,\qquad[T^{A},Q^{\underline{i}}]=(t^{A})^{\underline{i}}{}_{\underline{j}}Q^{\underline{j}}\
,\qquad[T^{A},T^{B}]=f^{AB}{}_{C}T^{C}\ .$ (1)

Here $t^{A}$ and $f^{AB}{}_{C}$ are representation matrices and structure
constant of the Lie algebra of the automorphism group. The automorphism group
is R, and the form of anticommutators $\\{Q,Q\\}$ depends on the spinor type
of $Q^{\underline{i}}$.

The spinors in various dimensions have different properties described in
Tanii:1998px . They could be Weyl, Majorana, symplectic (pseudo) Majorana,
Majorana-Weyl, and symplectic (pseudo) Majorana-Weyl spinors. In even
dimensions, the supersymmetry algebra can be presented via spinors of positive
and negative chirality. In odd dimensions, the supercharges are (pseudo)
Majorana in 9D and 11D, and symplectic (pseudo) Majorana in 5D and 7D.

In all dimensions, there is a choice of supersymmetries with a total amount of
$4\mathcal{N}$, which can be split in half, $2\mathcal{N}$+$2\mathcal{N}$. The
choice in each case is defined by the choice of the 1/2 BPS superfields, which
depend on 1/2 of fermionic directions. It is known in each dimension. In
applications to $n$-point amplitudes we define these two sets as $Q^{A}$ and
$\tilde{Q}^{A}$

$Q^{A}=\sum_{i=1}^{n}q^{A}_{i}=\sum_{i=1}^{n}(\lambda_{i}\,\eta_{i})^{A}\
,\quad\tilde{Q}^{A}=\sum_{i=1}^{n}\tilde{q}^{A}_{i}=\sum_{i=1}^{n}\lambda_{i},{\partial\over\partial\eta_{i}}\
,\quad i=1\ ,\dots,n\ ,\quad A=1,\dots,2\mathcal{N}\ ,\\\ $

where

$\\{Q^{A},{\tilde{Q}}^{B}\\}=\gamma^{AB}_{a}P^{a}\,,\qquad\\{Q^{A},Q^{B}\\}=0\,,\qquad\\{{\tilde{Q}}^{A},{\tilde{Q}}^{B}\\}=0\
.$ (2)

The $n$-point amplitude is defined as

${\cal A}=\delta^{D}(P)\delta^{2\mathcal{N}}(Q){\cal F}(\lambda_{i},\eta_{i})\
.$ (3)

The amplitude is manifestly supersymmetric under the action of the $Q$
operator, i. e. under 1/2 of supersymmetry. However, the action of the
operator $\tilde{Q}$ leads to Ward identities

$\boxed{\tilde{Q}{\cal A}=\delta^{D}(P)\delta^{2\mathcal{N}}(Q)\tilde{Q}[{\cal
F}(\lambda_{i},\eta_{i})]=0}$ (4)

There are 3 types of solutions given in Wang:2015jna  in the 10D case. These
are also valid in general dimensions222See however Appendix A here where we
discuss one-quarter and one-eighth linearised BPS superinvariants..

1\. The first solution requires that ${\cal F}$ is $\eta$ independent

${\cal F}(\lambda_{i},\eta_{i})={\cal F}(\lambda_{i})=f(s_{ij})\ .$ (5)

The supersymmetric amplitude is

${\cal A}=\delta^{D}(P)\delta^{2\mathcal{N}}(Q)f(s_{ij})$ (6)

since $\\{Q,{\tilde{Q}}\\}=\gamma_{a}P^{a}\,,$ and there is a delta-function
$\delta^{D}(P)$, the amplitude has all $4\mathcal{N}$ supersymmetries. These
are superamplitudes of the 1/2 BPS type.

2\. A CPT conjugate amplitude ${\overline{A}}$ is also supersymmetric. An
example in the 10D case is given in eq. (6) of Wang:2015jna .

3\. The function ${\cal F}$ is $\eta$-dependent, and this dependence is
strongly constrained by the Ward Identity. The function ${\cal
F}(\lambda_{i},\eta_{i})$ has to be of the form

${\cal
F}(\lambda_{i},\eta_{i})=\tilde{Q}\,h(\lambda_{i},\eta_{i})=\sum_{j=1}^{n}\lambda_{j}\,{\partial\over\partial\eta_{j}}h(\lambda_{i},\eta_{i})$
(7)

and Ward identities (4) are satisfied

$\tilde{Q}[{\cal
F}(\lambda_{i},\eta_{i})]=\tilde{Q}^{2}\,h(\lambda_{i},\eta_{i})=0\,.$ (8)

Therefore the solution to Ward identities is

$\tilde{Q}\,{\cal A}=0\quad\rightarrow\quad{\cal
A}=\delta^{D}(P)\delta^{2\mathcal{N}}(Q)\tilde{Q}\,h(\lambda_{i},\eta_{i})\ .$
(9)

Thus, the $n$-point amplitude satisfying supersymmetric Ward identities has
the form given in (9) where the function $h(\lambda_{i},\eta_{i})$ in not
constrained as long as the total amplitude has a factor

$\delta^{2\mathcal{N}}(Q)\tilde{Q}\ ,$ (10)

which has dimension $2\mathcal{N}$ as this expression involves the product of
$4\mathcal{N}$ $\lambda$’s which are spinor helicity variables of dimension
1/2. Note that in amplitudes, the $\eta$’s are dimensionless variables. Thus
we observe that these superamplitudes are associated with superinvariants
where the $\theta$ and $\tilde{\theta}$ variables have dimension 1/2.

$\delta^{2\mathcal{N}}(Q)\tilde{Q}\qquad\Longleftrightarrow\qquad
d^{2\mathcal{N}}\theta d^{2\mathcal{N}}\tilde{\theta}\ ,\qquad{\rm
dim}\,2\mathcal{N}\ .$ (11)

These superamplitudes are non-BPS type.

##  3 Dimensional analysis of superamplitudes

Consider the amplitudes at the $L$-loop order in D-dimensional
$\mathcal{N}$-extended supergravity. In front of the L-loop superamplitude, we
have a gravitational coupling

$\kappa^{2(L-1)}{\cal A}$ (12)

where dimension of $\kappa^{2}$ is $(2-D)$.

1\. The half-BPS amplitudes are

$\kappa^{2(L-1)}{\cal
A}_{BPS}=\kappa^{2(L-1)}\delta^{D}(P)\delta^{2\mathcal{N}}(Q)f(\lambda_{i})\
.$ (13)

therefore the dimension counting is $(2-D)(L-1)-D+\mathcal{N}+{\rm
dim}f(\lambda_{i})=0$ and

${\rm dim}[f(\lambda_{i})]=(D-2)L+2-\mathcal{N}\ .$ (14)

Consider an example of 6D maximal $\mathcal{N}=8$ supergravity 4-point
superamplitude. At $L=0$ ${\rm dim}[f(s_{ij})]=-6$, at $L=3$ ${\rm
dim}[f(s_{ij})]=6$. This agree with the $L=0$ result in Cachazo:2018hqa  where
$f={1\over stu}$ and $L=3$ result in Kallosh:2023dpr  where $f=stu$.

Another example is 4D maximal supergravity 4-point amplitudes. At $L=0$ ${\rm
dim}[f(s_{ij})]=-6$, at $L=3$ ${\rm dim}[f(s_{ij})]=0$. This agree with the
$L=0$ result in Kallosh:2008ru  where $f={1\over stu}$ and $L=3$ where
$f=const$.

2\. The non-BPS $n$-point amplitudes are

$\kappa^{2(L-1)}{\cal A}_{non-
BPS}=\kappa^{2(L-1)}\delta^{D}(P)\delta^{2\mathcal{N}}(Q)\tilde{Q}\,h(\lambda_{i},\eta_{i})\
.$ (15)

Dimension counting is $(2-D)(L-1)-D+2\mathcal{N}+{\rm
dim}[h(\lambda_{i},\eta_{i})]=0$ and

${\rm dim}[h(\lambda_{i},\eta_{i})]=(D-2)L+2-2\mathcal{N}\ .$ (16)

We find that to have supervertices, i.e., superamplitudes with no poles in
momenta with ${\rm dim}[h(\lambda_{i},\eta_{i})]\geq 0$ requires that

$(D-2)L+2-2\mathcal{N}\geq 0\ .$ (17)

For example, in 4D we need $L\geq\mathcal{N}-1$ to have supervirtices (no
poles). But at $L\leq\mathcal{N}-2$ these supervertices are not available,
which results in a non-vanishing scalar soft limits and broken $E_{7(7)}$
symmetry if there were UV divergences at $L\leq\mathcal{N}-2$. This is in full
agreement with Beisert:2010jx  for $\mathcal{N}=8$ and with Freedman:2018mrv
for $\mathcal{N}\geq 5$. Moreover, we have observed that already based on
supersymmetric Ward identities and dimensional analysis of superamplitudes, it
was possible to establish this result without explicit computations of the
6-point superamplitudes.

##  4 Dimensional analysis of linearized superinvariants

Half-BPS linearized superinvariants are

$S^{BPS}=\kappa^{2(L-1)}\int d^{2\mathcal{N}}\theta d^{D}x\,{\cal L}^{BPS}\ .$
(18)

Dimension of $\kappa^{2}$ is $2-D$. For the action to be dimensionless we
require that

${\rm dim}\,{\cal L}^{BPS}=L(D-2)+2-\mathcal{N}$ (19)

For non-BPS superinvariants we have

$S^{non-BPS}=\kappa^{2(L-1)}\int d^{4\mathcal{N}}\theta d^{D}x\,{\cal L}^{non-
BPS}\ .$ (20)

and

${\rm dim}\,{\cal L}^{non-BPS}=L(D-2)+2-2\mathcal{N}$ (21)

In amplitude applications, what we would like to call “local multi-point
superinvariants” are often called supervertices, namely superamplitudes with
no poles in momenta, see, for example, Wang:2015jna . Let us find out the
smallest loop order where $n$-point supervertices are available. Consider
superinvariants in eq. (20) where

$L(2-D)-2+2\mathcal{N}+{\rm dim}\,{\cal L}^{non-BPS}=0\ ,$ (22)

and

$L={2(\mathcal{N}-1)+{\rm dim}\,{\cal L}^{non-BPS}\over(D-2)}\ .$ (23)

Below this loop order, there are no local multi-point superinvariants, i.e.,
there are no superamplitudes without the poles in momenta.

Consider known examples where for local superinvariants we require that

${\rm dim}\,{\cal L}\geq 0\ .$ (24)

In 4D we can take ${\rm dim}\,{\cal L}=0$ as it depends on a product of $n$
superfields with scalars in the first component. The R-invariant $n$-point
local multi-point superinvariants with $n\geq 6$ according to (22) are
available starting with

$L=\mathcal{N}-1\,,\qquad D=4\ ,$ (25)

and they are not available at $L\leq\mathcal{N}-2$ in 4D. This is a result
obtained in Freedman:2018mrv  based on analysis of superinvariants.

This simple statement for superinvariants was confirmed by direct computations
of NkMHV 6-point superamplitudes in $\mathcal{N}=5,6,8$ supergravities
Freedman:2018mrv . The result of the computation, using solutions of Ward
identities, did confirm the linearized superinvariant analysis.

Now we take 6D maximal supergravity where there is a UV divergence at $L=3$.
It was found in Kallosh:2023dpr  that local multi-point superinvariants
R-symmetry invariants are available starting with

$L={14+{\rm dim}\,{\cal L}\over 4}>3\,,\qquad D=6\,,\qquad\mathcal{N}=8\ .$
(26)

But unlike in the 4D case, the computation of the relevant 6-point
superamplitude has not yet been performed. It would confirm that at L=3 there
are no R-invariant local (no poles) superamplitudes.

In 8D, one finds in maximal supergravity case where there is a UV divergence
at $L=1$ and that local multi-point superinvariants are available starting
with

$L={14+{\rm dim}\,{\cal L}\over 6}>2\,,\qquad D=8\,,\qquad\mathcal{N}=8\ .$
(27)

##  5 R symmetry

The supervertices in eq. (6), which we call half-BPS amplitudes in all
dimensions, are also called F-term vertices in 10D case Wang:2015jna , and MRV
(Maximal R-symmetry Violating) amplitudes in Boels:2012zr . In 4D, these are
known as MHV, Maximally Helicity Violating amplitudes, but these do not exist
in other dimensions Boels:2012zr .

4-point superamplitudes/superinvariants make an exception in all dimensions.
The BPS 4-point amplitudes do not break R-symmetry. The existence of BPS
amplitudes consistent with R-symmetry for 4-point amplitudes is known in 4D
supergravities Kallosh:1980fi ; Howe:1981xy  where it was proven for local BPS
superinvariants. In 10D, it was proven in Boels:2012zr  that for $n>4$, BPS
superamplitudes R-symmetry is broken, but it is preserved for $n=4$. Finally,
in 6D, it was proven in Kallosh:2023dpr  that 4-point BPS superinvariants have
unbroken R-symmetry, but the ones with $n>4$ break R-symmetry.

Also, in known examples, the 4-point superamplitudes defined as half-BPS
invariants in eq. (6) are consistent with R-symmetry. However, at higher point
amplitudes $n>4$, the issue of R-symmetry is more complicated, and these BPS
superamplitudes often break R-symmetry.

However, there are cases of $n$-point half-BPS superamplitudes/linearized
superinvariants with $n>4$, which preserve R-symmetry. For example, in
Huang:2015sla  in 4D $\mathcal{N}=4$ supergravity, there is a half-BPS 5-point
amplitude which is not anomalous under $U(1)$ symmetry. It corresponds to a
rational part of the 5-point amplitude $R_{5}$ computed in Dunbar:2010fy . The
corresponding $U(1)$ preserving superamplitude is given in eq. (4.13) in
Huang:2015sla . Here, we will present it in the form of the linearized
superinvariant

${\rm MHV}^{(3,2)}=M^{1}+M^{2}$ (28)

where

$M^{1}=\int d^{8}\bar{\theta}\,{1\over
s_{45}s_{43}s_{53}}\overline{W}_{1}\overline{W}_{2}C_{3}*C_{4}*C_{5}$ (29)

and

$M^{2}=\int d^{8}\bar{\theta}\,{1\over
s_{34}^{2}s_{25}s_{35}s_{45}}{\overline{W}}_{1}\,\partial^{\alpha}{}_{\dot{\alpha}}\partial^{\beta}{}_{\dot{\beta}}{\overline{W}}_{2}(\partial^{\gamma}{}_{\dot{\alpha}}C_{3}*\partial^{\delta}{}_{\dot{\beta}}\,C_{4})C_{5\,\alpha\beta\gamma\delta}+\\{1,2\\}+\\{3,4,5\\}$
(30)

Here

$\overline{W}_{i}\equiv\overline{W}(p_{i},\bar{\theta})\,,\qquad
C_{3}*C_{4}*C_{5}\equiv{\rm
C}_{\alpha\beta}{}^{\gamma\delta}(p_{3},\bar{\theta}){\rm
C}_{\gamma\delta}{}^{\mu\nu}(p_{4},\bar{\theta}){\rm
C}_{\mu\nu}{}^{\alpha\beta}(p_{5},\bar{\theta})$ (31) $\partial C_{3}*\partial
C_{4}\equiv\partial
C_{\alpha\beta}{}^{\gamma\delta}(p_{3},\bar{\theta})\partial{\rm
C}_{\gamma\delta}{}^{\alpha\beta}(p_{4},\bar{\theta})$ (32)

The fact that this half-BPS 5-point superinvariants/superamplitudes preserve
$U(1)$ is obvious. The Weyl superfields ${\rm
C}_{\alpha\beta}{}^{\gamma\delta}$ have zero $U(1)$ charge, and two $\bar{W}$’
have $U(1)$ charge canceling the ones from a half-BPS measure of integration.

In these superinvariants

${\rm dim}[{\cal L}^{1}(x,\bar{\theta})]={\rm
dim}[{1\over\partial^{6}}\overline{W}^{2}C^{3}]=0\,\qquad{\rm dim}[{\cal
L}^{2}(x,\bar{\theta})]={\rm
dim}[{1\over\partial^{10}}{\overline{W}}\,\partial^{2}{\overline{W}}\partial
C\partial CC]=0$ (33)

We can compare it with explicit expressions for the superamplitudes for
Huang:2015sla  in half-BPS (6)

${\rm dim}[f^{1}(\lambda_{i})]={\rm dim}[{[34][45][53]\over\langle
34\rangle\langle 45\rangle\langle 53\rangle}]=0\,,\qquad{\rm
dim}[f^{2}(\lambda_{i})]={\rm dim}[{[34]^{2}\over\langle
34\rangle^{2}}{[25]\langle 23\rangle\langle 24\rangle\over\langle
25\rangle\langle 35\rangle\langle 45\rangle}]=0$ (34)

This is an example of the R-invariant half-BPS 5-point superamplitude in eq.
(14) in 4D at $\mathcal{N}=4$ and $L=1$

${\rm dim}[f(\lambda_{i})]=(D-2)L+2-\mathcal{N}=2+2-4=0$ (35)

On the other hand, the known examples of non-BPS $n$-point amplitudes with
$n>4$ satisfy Ward identities in eq. (9) can easily preserve R-symmetry. One
can see from eq. (11) that the part of the superamplitude has a factor
$\delta^{2\mathcal{N}}(Q)\tilde{Q}$ analogous to the measure of the
integration over the whole superspace $d^{2\mathcal{N}}\theta
d^{2\mathcal{N}}\tilde{\theta}$ which has R-symmetry.

To keep R-symmetry using whole superspace superinvariants, one needs to have
also a Lagrangian $\cal L$ preserving R-symmetry.

$\int d^{2\mathcal{N}}\theta d^{2\mathcal{N}}\tilde{\theta}d^{D}x\,{\cal
L}(x,\theta,\tilde{\theta})\ .$ (36)

In superamplitudes (9), the function $h(\lambda_{i},\eta_{i})$ has to be taken
in the form preserving R-symmetry to have R-symmetric $n$-point
superamplitudes.

##  6 Application to 4D $\mathcal{N}\geq 5$ supergravity soft limits

In Freedman:2018mrv , the computation of 6-point superamplitudes was
performed. The results were that the local superamplitudes are available
starting from $L=\mathcal{N}-1$ but not below this loop order.

Here, instead of computing 6-point superamplitudes satisfying Ward identities,
we present an alternative derivation of the same result. R-symmetry invariant
$n$-point amplitudes with $n>4$ require the form of solutions of Ward
identities given in (20), (16), (17)

Supervertices (superamplitudes without poles) are available for $n$-point
amplitudes, $n>4$ in 4D supergravity starting with

${\rm dim}[h(\lambda_{i},\eta_{i})]=2L+2-2\mathcal{N}=2(L-(\mathcal{N}-1))\geq
0\ .$ (37)

This means that 6-point superamplitudes, supervertices without poles, are
available at

$L\geq\mathcal{N}-1\ .$ (38)

This confirms the computation of 6-point superamplitudes in Freedman:2018mrv
where it was found that all constructed 6-point superamplitudes with non-
vanishing soft scalar limit are not local.

Here, we have found a way around the complicated computation of all explicit
6-point amplitudes of the kind performed in Freedman:2018mrv . It was possible
to use a non-explicit solution of 4D Ward identities in eq. (20) and only
study the dimension of the relevant function $h(\lambda_{i},\eta_{i})$.

Let us illustrate the case with $\mathcal{N}=8,L=6$ where there are no local
6-point superamplitudes, according to (37).

To study soft limits, we need a 6-point superinvariant with bosonic part

$\kappa^{10}\int d^{4}x\,\partial^{6}\,R^{4}\,\phi_{ijkl}\,\bar{\phi}^{ijkl}\
.$ (39)

We can try

$\kappa^{10}\,\int\,d^{4}x\,d^{32}\theta\,\partial^{2l}(W_{ijkl}\,\bar{W}^{ijkl})^{3}\
.$ (40)

It means that $-10-4+16+2l=0$, and $l=-1$ so the relevant supersymmetric
expression is

$\kappa^{10}\,\int\,d^{4}x\,d^{32}\theta\,(W_{ijkl}\,\bar{W}^{ijkl})^{3}\partial^{-2}\
,$ (41)

and it is non-local. Using superamplitudes (37) we find the same dimension of
$h$ as dimension of ${\cal L}=\partial^{2l}(W_{ijkl}\,\bar{W}^{ijkl})^{3}$
with $l=-1$

${\rm dim}[h(\lambda_{i},\eta_{i})]=2(6-7)=-2\,.$ (42)

Thus, we see that at $L=6$, maximal 4D supergravity does not have a local
expression with bosonic part $\partial^{6}\,R^{4}\,\phi\,\bar{\phi}$. This, in
turn, means that the 4-point UV divergence at $L=6$ maximal 4D supergravity
would imply that the $E_{7(7)}$ symmetry is broken. This conclusion was
reached in Beisert:2010jx  using string theory and in Freedman:2018mrv  using
superinvariants. Here we deduced the same result using dimensional analysis of
4D superamplitudes satisfying Ward identities.

##  7 Applications to 6D maximal supergravity soft limits

The supercharges are symplectic Majorana-Weyl spinors with positive chirality
$Q_{+}^{i}$ ($i=1,2,\cdots,N_{+}$) and symplectic Majorana-Weyl spinors with
negative chirality $Q_{-}^{i}$ ($i=1,2,\cdots,N_{-}$). They satisfy
$\Omega_{+}^{ij}(Q_{+}^{j})^{c}=Q_{+}^{i}$,
$\Omega_{-}^{ij}(Q_{-}^{j})^{c}=Q_{-}^{i}$, where $\Omega_{\pm}^{ij}$ are
antisymmetric matrices. The numbers $N_{+}$ and $N_{-}$ must be even. The
automorphism group is USp($N_{+}$) $\times$ USp($N_{-}$). Anticommutators of
the supercharges are

$\displaystyle\\{Q_{+}^{i},Q_{+}^{jT}\\}$ $\displaystyle\\!\\!\\!\ =\ $
$\displaystyle\\!\\!\\!{1\over
2}(1+\bar{\gamma})\gamma^{a}C_{-}P_{a}\Omega_{+}^{ij}\ ,$
$\displaystyle\\{Q_{-}^{i},Q_{-}^{jT}\\}$ $\displaystyle\\!\\!\\!=$
$\displaystyle\\!\\!\\!{1\over
2}(1-\bar{\gamma})\gamma^{a}C_{-}P_{a}\Omega_{-}^{ij}\ ,$
$\displaystyle\\{Q_{+}^{i},Q_{-}^{jT}\\}$ $\displaystyle\\!\\!\\!=$
$\displaystyle\\!\\!\\!{1\over 2}(1+\bar{\gamma})C_{-}Z^{ij}\ .$ (43)

Consider maximal 6D supergravity and 1/2 BPS superfields.

$N_{+}=4\,,\qquad N_{-}=4\ .$ (44)

The R symmetry group is $USp(4)\times USp(4)$. It coincides with the isotropy
group ${\mathcal{H}}$ associated with the coset
${\mathcal{G}\over\mathcal{H}}={E_{5(5)}\over USp(4)\times USp(4)}$. It is
convenient to discuss the ultrashort multiplets using the following choice of
the 4x4 $\Omega_{\pm}$-matrix Ferrara:2000xg

$\left(\begin{array}[]{cc}0&{\cal I}\\\ -{\cal I}&\,0\end{array}\right)\,,$
(45)

here ${\cal I}$ is the 2x2 identity matrix. The non-vanishing entries in the
RHS of supersymmetry algebra are

$\Omega^{13}_{\pm}=\Omega^{24}_{\pm}=1\ .$ (46)

Spinor coordinates of the superspace satisfy a Majorana-Weyl pseudoreality
condition

$\bar{\theta}^{i}=\Omega^{ij}\theta_{j}^{\beta}c_{\beta\alpha}\ .$ (47)

The spinor derivatives satisfy the supersymmetry algebra

$\\{D_{\alpha}^{i},D_{\beta}^{j}\\}=i\Omega^{ij}\gamma^{\mu}\partial_{\mu}\ .$
(48)

The 25 scalars are given by $V_{a}^{\hat{a}}$ field in $SO5)\times SO(5)$
form, $a,\hat{a}=1,2,3,4,5$. The BPS superfield is described in $USp(4)\times
USp(4)$ basis so that the first component is

$V_{ij}^{\hat{i}\hat{j}}=\gamma^{a}_{ij}V_{a}^{\hat{a}}\gamma_{\hat{a}}^{\hat{i}\hat{j}}\
.$ (49)

Using linearized supersymmetry of the 6D maximal supergravity action
Tanii:1984zk ; Bergshoeff:2007ef  one finds that the superfield
$W_{12}^{\hat{1}\hat{2}}(x,\theta)$ depends only on half of fermionic
coordinates

$D_{\alpha 1}W_{12}^{\hat{1}\hat{2}}=D_{\alpha
2}W_{12}^{\hat{1}\hat{2}}=D^{\hat{\alpha}\hat{1}}W_{12}^{\hat{1}\hat{2}}D^{\hat{\alpha}\hat{2}}W_{12}^{\hat{1}\hat{2}}=0\
.$ (50)

This is consistent with the choice of the symplectic matrix $\Omega$ in (45).
By making this choice of the superfield, we have broken $USp(4)\times USp(4)$
R symmetry down to its subgroup $SU(2)\times SU(2)$.

Now we can apply this information to 6D multi-point superamplitudes along the
lines used for type IIB supergravity amplitudes in 10D in Wang:2015jna . The
$n$-point amplitude is defined as

${\cal A}=\delta^{6}(P)\delta^{16}(Q){\cal F}(\lambda_{i},\eta_{i})\ .$ (51)

Here

$\delta^{16}(Q)=\delta^{8}\left(\sum_{i=1}^{n}q^{A,I}_{i}\right)\delta^{8}\left(\sum_{i=1}^{n}{\hat{q}}^{{\hat{I}}}_{i,{\hat{A}}}\right)\,,$
(52)

which has manifest permutation symmetry. Here the supercharges are defined as
$q^{A,I}_{i}=\lambda^{A}_{i,a}\eta^{I,a}_{i}$, and
${\hat{q}}^{{\hat{I}}}_{i,{\hat{A}}}=\hat{\lambda}_{i,\hat{A},\hat{a}}\hat{\eta}^{{\hat{I}},\hat{a}}_{i}$.
These are half of the supercharges, and the other half involve $\eta$
derivatives. $I,\hat{I}=1,2$ and $A,\hat{A}=1,2,3,4.$ Conservation of these
additional supercharges automatically follows from the first set together with
the R symmetry. With

$Q=\sum_{i=1}^{n}\lambda_{i}\,\eta_{i}\qquad\tilde{Q}=\sum_{i=1}^{n}\lambda_{i}\,{\partial\over\partial\eta_{i}}\
,$ (53)

we recover the susy algebra in (2). The amplitude is manifestly supersymmetric
under the action of the $Q$ operator, i. e. under 1/2 of supersymmetry. The
action of $\tilde{Q}$ is more constraining, in general. We have already
explained this in Kallosh:2023dpr  in terms of superinvariants. At that time,
the relevant information from the point of view of superamplitudes was
proposed in the context of 10D type IIB supergravity amplitudes. Here we can
do it directly in the framework of linearized 6D maximal supergravity
amplitudes.

Consider R-invariant multi-point superamplitudes in 6D. The $n$-point
superamplitudes with $n>4$ at $L$-loop order satisfying linearized Ward
Identity at maximal supergravity are

$\kappa^{2(L-1)}\delta^{6}(P)\delta^{2\mathcal{N}}(Q)\tilde{Q}\,h(\lambda_{i},\eta_{i})\
,$ (54)

where according to (16) in 6D and $\mathcal{N}=8$

${\rm dim}[h(\lambda_{i},\eta_{i})]=4L-14\ .$ (55)

Thus, at $L=3$ we find that

${\rm dim}[h(\lambda_{i},\eta_{i})]=12-14=-2\ .$ (56)

and at 3-loops, there are no supervertices! There was again no need actually
to find the explicit solutions of Ward identities for the 6-point amplitudes,
as we did in Freedman:2018mrv  in 4D case. It was sufficient to establish the
dimension of the function $h(\lambda_{i},\eta_{i})$, as we did also above for
the 4D $\mathcal{N}\geq 5$ supergravities in Sec. 6.

This, in turn, means that the 4-point UV divergence at $L=3$ maximal 6D
supergravity breaks $E_{5(5)}$ symmetry since the soft scalar limit of the
total 6-point amplitude is broken.

Thus, in addition to the analysis of superinvariants, we have also shown that
the superamplitudes in 6d supergravity lead to the same conclusion.

A nice example of the 10D 6-point superamplitude was already given in
Green:2019rhz  and discussed in Kallosh:2023dpr . In 6D, the 3-loop UV
divergence is of the type $\kappa^{4}\int d^{4}x\partial^{6}R^{4}$. We are
looking at the local $SO(5)\times SO(5)$ invariant amplitude with two
additional scalars

$\kappa^{4}\int
d^{4}x\partial^{6}R^{4}\phi_{a}^{\hat{a}}\phi_{b}^{\hat{b}}\delta^{ab}\delta_{\hat{a}\hat{b}}\,,\qquad
a,\hat{a}=1,2,3,4,5\ .$ (57)

A supersymmetric one would take a form (20)

$\kappa^{4}{\cal A}_{non-
BPS}=\kappa^{4}\delta^{6}(P)\delta^{16}(Q)\tilde{Q}\,h(\lambda_{i},\eta_{i})\
.$ (58)

We already know that dimension of $h$ here is negative, so the superamplitude
has a pole. It is interesting to see how it depends on $\lambda$ and $\eta$’s.

The scalar superfields have a graviton at the 4th component, so to have 4
gravitons we need 16 $\eta$’s. One of the scalars has $\eta^{0}$, the other
one has $\eta^{8}$, one is the first component of the BPS superfield, the
other one is the last. So we need the total of $\eta$’s in $h$ to be 24. This
is literally the superamplitude given in Green:2019rhz  for 10d type IIB
supergravity and discussed in Kallosh:2023dpr

$\delta^{16}(Q_{n})(\bar{Q}_{n})^{16}(\eta_{i})^{8}(\eta_{j})^{8}(\eta_{k})^{8}\
,$ (59)

To be more specific, in 6D at 3-loop order, we have the following
superamplitude, which includes 4 gravitons and 2 scalars which is
$USp(4)\times USp(4)$ invariant

$\kappa^{4}\delta^{6}(P)\delta^{16}(Q_{n})(\bar{Q}_{n})^{16}(\eta)^{24}h(\lambda)\
,\qquad{\rm dim}\,h(\lambda)=-2\ .$ (60)

Note that we have 32 $\lambda$’s in $\delta^{16}(Q_{n})(\bar{Q}_{n})^{16}$
which gives 16 in momenta, and only -14 from $\kappa^{4}$ and $\delta^{6}(P)$
at $L=3$. Now we can see why the 10D superamplitude in (59) is in agreement
with our superinvariant analysis in 6D in Kallosh:2023dpr  as well as our new
analysis here of the superamplitudes satisfying Ward identities in 6D as given
in eq. (60) here.

The relevant 6-point candidate 3-loop linearized superinvariants does not
exists since the action has a negative dimension

${\rm dim}\Big{[}\kappa^{4}\int d^{6}x\,d^{32}\theta\,Tr\,V^{6}\Big{]}=-2\ ,$
(61)

Here

$Tr\,V^{6}=V_{a}^{\hat{a}}V_{b}^{\hat{b}}V_{c}^{\hat{c}}V_{d}^{\hat{d}}V_{e}^{\hat{e}}V_{f}^{\hat{f}}t^{abcdef}_{\hat{a}\hat{b}\hat{c}\hat{d}\hat{e}\hat{f}}$
(62)

and $t^{abcdef}_{\hat{a}\hat{b}\hat{c}\hat{d}\hat{e}\hat{f}}$ is an
$SO(5)\times SO(5)$ covariant tensor. $V_{a}^{\hat{a}}$ is a scalar superfield
starting with a 25-components scalar field $\phi_{a}^{\hat{a}}$. Its dimension
is the same as in a relevant superamplitude in eq. (60).

##  8 Summary

Supersymmetric Ward identities for linearized extended $\mathcal{N}\geq 5$
supergravities in 4D are known for a long time Bianchi:2008pu ; Elvang:2009wd
; Freedman:2018mrv . We have found here that the more recent understanding of
supersymmetric Ward identities in 10D type IIB supergravity in Wang:2015jna ;
Green:2019rhz  allows a nice and useful generalization of these identities to
$D\geq 4$ and extended $\mathcal{N}\geq 5$ supergravities.

It is satisfying to see this kind of universality with regard to Ward
identities for amplitudes. The short summary is that

1\. 1/2 BPS superamplitudes have the form

$\kappa^{2(L-1)}\delta^{D}(P)\delta^{2\mathcal{N}}(Q)f(s_{ij})\ .$ (63)

and

${\rm dim}[f(s_{ij})]=(D-2)L+2-\mathcal{N}\ .$ (64)

This is to be compared with the 1/2 BPS linearized superinvariants of the form
where we integrate over 1/2 of the superspace a Lagrangian depending on
superfields

$\kappa^{2(L-1)}\int d^{D}xd^{2\mathcal{N}}\theta\,{\cal
L}^{BPS}\Big{(}\phi(x,\theta)\Big{)}\ ,$ (65)

and

${\rm dim}[{\cal L}^{BPS}(\phi(x,\theta))]=(D-2)L+2-\mathcal{N}\ .$ (66)

Thus

$\boxed{{\rm dim}[{\cal L}^{BPS}(\phi(x,\theta))]={\rm dim}[f(s_{ij})]}$ (67)

2\. non-BPS $n$-point R-invariant superamplitudes with $n>4$ have the form

$\kappa^{2(L-1)}\delta^{d}(P)\delta^{2\mathcal{N}}(Q)\tilde{Q}\,h(\lambda_{i},\eta_{i})\
,$ (68)

and

${\rm dim}[h(\lambda_{i},\eta_{i})]=(D-2)L+2-2\mathcal{N}\ .$ (69)

This is to be compared with the non-BPS linearized superinvariants of the form
where we integrate over the whole superspace a Lagrangian depending on
superfields

$\kappa^{2(L-1)}\int d^{D}xd^{4\mathcal{N}}\theta\,{\cal L}^{non-
BPS}\Big{(}\phi(x,\theta)\Big{)}\ ,$ (70)

and

${\rm dim}[{\cal L}^{non-BPS}(\phi(x,\theta))]=(D-2)L+2-2\mathcal{N}$ (71)

Here we also note that

$\boxed{{\rm dim}[h(\lambda_{i},\eta_{i})]={\rm dim}[{\cal L}^{non-
BPS}(\phi(x,\theta))]}\ .$ (72)

If the dimension of ${\cal L}(\phi(x,\theta)$ is negative, the superinvariant
is not a local one; some insertion of inverse derivatives is required like
$\partial^{-2}$. But if the dimension of ${\cal L}(\phi(x,\theta)$ is not
negative, we have a local superinvariant. If the dimension of the function
$h(\lambda_{i},\eta_{i})$ is negative, it means that $\lambda$’s are in
denominator, and there are some poles in momentum space since $p$ is quadratic
in $\lambda$’s. Here $\eta$’s are dimensionless. If the dimension of
$h(\lambda_{i},\eta_{i})$ is non-negative, the corresponding superamplitudes
are so-called supervertices; they have no poles in momentum space. It is,
therefore, not surprising that superamplitudes and linearized superinvariants
have the same predictions.

The examples are in 4D $\mathcal{N}\geq 5$, where there are no 6-point
supervertices at $L<\mathcal{N}-1$,

${\rm dim}[h(\lambda_{i},\eta_{i})]={\rm dim}[{\cal L}^{non-
BPS}(\phi(x,\theta))]=2(L-(\mathcal{N}-1))\ ,$ (73)

since at $L<\mathcal{N}-1$, both superamplitudes as well as superinvariants
are non-local. In Beisert:2010jx  and in Freedman:2018mrv  it was shown using
superamplitudes as well as superinvariants.

In maximal 6D

${\rm dim}[{\cal L}^{non-BPS}(\phi(x,\theta))]={\rm
dim}[h(\lambda_{i},\eta_{i})]=4L-14\ .$ (74)

The absence of 6-point supervertices at $L\leq 3$ was proven in
Kallosh:2023dpr  using linearized superinvariants. Here we have shown that the
same result follows from superamplitudes satisfying Ward identities.

The nice thing we have learned here is that it was not necessary to compute
explicitly the 6-point amplitudes satisfying Ward identities, as we did in 4D
case in Freedman:2018mrv  for $L\leq\mathcal{N}-1$. It was sufficient both in
4D and in 6D cases to establish a dimension of the function $h(\lambda,\eta)$
which appears in the solutions of Ward identities. That allowed us to find out
here that certain 6-point amplitudes are not local, which was important for
the soft scalar limits studies.

The actual computation of the $n$-point amplitudes is necessary when we are
looking for the 1-loop anomalies. This was the case in 4D $\mathcal{N}=4$
supergravity Carrasco:2013ypa . The existence of linearized $n$-point half-BPS
superinvariants with non-vanishing soft scalar limits was established, for
example, there was a term

${d_{n+2}\over(4\pi)^{2}}\int d^{4}xd^{8}\theta W^{n+2}+cc$ (75)

where the actual value of $d_{n}$ was not known. The corresponding
superamplitude with 2 gravitons and $n$ scalars was also known to be
proportional to
$\delta^{8}\Big{(}\sum_{i=1}^{n}\tilde{\eta}_{i}\tilde{\lambda}_{i}\Big{)}$.
But the actual computation of this amplitude has confirmed that
$d_{n+2}={1\over 2(4\pi)^{2}}{1\over n(n+1)(n+2)}\neq 0$ and that the 1-loop
anomaly is present.

There was another candidate for 1-loop anomaly $U(1)$ anomaly in 4D
$\mathcal{N}=4$ supergravity in the form of a linearized superinvariant

$f_{4}\int
d^{8}\bar{\theta}{1\over\partial^{6}}C\,\partial\partial\overline{W}\,\partial\partial\overline{W}\,\overline{W}$
(76)

The superamplitude matching this superinvariant was computed in
Carrasco:2013ypa  using the double-copy method. The result has shown that
$f_{4}\neq 0$ and the corresponding 1-loop anomaly is present.

The fact that some superamplitudes are consistent with Ward identities or,
equivalently, can be presented as superspace integrals can only tell us that
such amplitudes are consistent with linearized supersymmetry. But only the
actual computation can tell us if these are present or not. For example, the
1-loop 6-point amplitudes in 6D maximal supergravity have to be computed to
find out if there are $E_{5(5)}$ 1-loop anomalies.

Acknowledgments: I am grateful to J. J. Carrasco, H. Elvang, D. Freedman,
Y.-t. Huang, A. Linde, J. Parra-Martinez, R. Roiban, C. Wen, and Y. Yamada for
stimulating discussions of the Ward identities and of the soft scalar limits
in supergravities. This work is supported by SITP and by the US National
Science Foundation grant PHY-2310429.

##  Appendix A One-quarter and one-eighth BPS superinvariants

In this paper, we have studied only half of BPS and non-BPS superinvariants
and the relevant superamplitudes. On the other hand, there are examples of
linearized superinvariants, which are one-half, one-quarter, and one-eighth
superinvariants Bossard:2009sy . These are based on superfields in harmonic
superspace, which depend on $4\mathcal{N}(1-{1\over k})$ directions, with
$k=2,4,8$. Ignoring harmonic coordinates, they are given as integrals over a
subspace of the whole superspace of the form

$\int d^{D}xd^{4(\mathcal{N}-{1\over k})}\theta{\cal L}(\phi(x,\theta))$ (77)

where the superfields $\phi(x,\theta)$ depend only on a fraction of all
$\theta$-directions. For example, in maximal $\mathcal{N}=8$ case, one-half of
BPS superfields depend on 16 $\theta$’s, one-quarter of BPS superfields depend
on 24 $\theta$’s, and one-eighth BPS superfields depend on 28 $\theta$’s.

All 1/k BPS superinvariants have R-symmetry for 4-point superinvariants.
However, when the number of superfields exceeds 4 in all examples, the
R-symmetry of such invariants is broken.

In the context of superamplitudes, which we studied in this paper, the cases
of one-half-BPS and non-BPS are simple and are now fully described in this
paper for dimensions $D\geq 4$ and $\mathcal{N}\geq 5$.

We do not know of examples of one-quarter and one-eighth BPS superamplitudes
corresponding to relevant known superinvariants, and whether these
superamplitudes are available or not. Nevertheless, we brought it up here to
keep in mind that there are cases with linearized superinvariants that have
not been studied in the context of superamplitudes.

## References

  * (1) M. Bianchi, H. Elvang and D. Z. Freedman, _Generating Tree Amplitudes in N=4 SYM and N = 8 SG_ , _JHEP_ 09 (2008) 063, [0805.0757].
  * (2) H. Elvang, D. Z. Freedman and M. Kiermaier, _Solution to the Ward Identities for Superamplitudes_ , _JHEP_ 10 (2010) 103, [0911.3169].
  * (3) D. Z. Freedman, R. Kallosh and Y. Yamada, _Duality Constraints on Counterterms in $\mathcal{N}=5,\ 6$ Supergravities_, _Fortsch. Phys._ 2018 (2018) 1800054, [1807.06704].
  * (4) R. H. Boels and D. O’Connell, _Simple superamplitudes in higher dimensions_ , _JHEP_ 06 (2012) 163, [1201.2653].
  * (5) Y. Wang and X. Yin, _Constraining Higher Derivative Supergravity with Scattering Amplitudes_ , _Phys. Rev. D_ 92 (2015) 041701, [1502.03810].
  * (6) M. B. Green and C. Wen, _Modular Forms and $SL(2,{\mathbb{Z}})$-covariance of type IIB superstring theory_, _JHEP_ 06 (2019) 087, [1904.13394].
  * (7) R. Kallosh, _Soft Theorems, Anomalies and Precursors_ , 2311.10084.
  * (8) N. Marcus, _Composite Anomalies in Supergravity_ , _Phys. Lett._ B157 (1985) 383–388.
  * (9) Y. Tanii, _Introduction to supergravities in diverse dimensions_ , in _YITP Workshop on Supersymmetry_ , 2, 1998.  hep-th/9802138.
  * (10) F. Cachazo, A. Guevara, M. Heydeman, S. Mizera, J. H. Schwarz and C. Wen, _The S Matrix of 6D Super Yang-Mills and Maximal Supergravity from Rational Maps_ , _JHEP_ 09 (2018) 125, [1805.11111].
  * (11) R. Kallosh, C. H. Lee and T. Rube, _N=8 Supergravity 4-point Amplitudes_ , _JHEP_ 02 (2009) 050, [0811.3417].
  * (12) N. Beisert, H. Elvang, D. Z. Freedman, M. Kiermaier, A. Morales and S. Stieberger, _$E_{7(7)}$ constraints on counterterms in N=8 supergravity_, _Phys. Lett._ B694 (2011) 265–271, [1009.1643].
  * (13) R. H. Boels, _Maximal R-symmetry violating amplitudes in type IIB superstring theory_ , _Phys. Rev. Lett._ 109 (2012) 081602, [1204.4208].
  * (14) R. E. Kallosh, _Counterterms in extended supergravities_ , _Phys. Lett._ B99 (1981) 122–127.
  * (15) P. S. Howe, K. S. Stelle and P. K. Townsend, _Superactions_ , _Nucl. Phys._ B191 (1981) 445–464.
  * (16) Y.-t. Huang and C. Wen, _Soft theorems from anomalous symmetries_ , _JHEP_ 12 (2015) 143, [1509.07840].
  * (17) D. C. Dunbar, J. H. Ettle and W. B. Perkins, _Perturbative expansion of N $<$8 Supergravity_, _Phys. Rev. D_ 83 (2011) 065015, [1011.5378].
  * (18) S. Ferrara and E. Sokatchev, _Representations of (1,0) and (2,0) superconformal algebras in six-dimensions: Massless and short superfields_ , _Lett. Math. Phys._ 51 (2000) 55–69, [hep-th/0001178].
  * (19) Y. Tanii, _$N=8$ Supergravity in Six-dimensions_, _Phys. Lett. B_ 145 (1984) 197–200.
  * (20) E. Bergshoeff, H. Samtleben and E. Sezgin, _The Gaugings of Maximal D=6 Supergravity_ , _JHEP_ 03 (2008) 068, [0712.4277].
  * (21) J. J. M. Carrasco, R. Kallosh, R. Roiban and A. A. Tseytlin, _On the U(1) duality anomaly and the S-matrix of N=4 supergravity_ , _JHEP_ 07 (2013) 029, [1303.6219].
  * (22) G. Bossard, P. S. Howe and K. S. Stelle, _The Ultra-violet question in maximally supersymmetric field theories_ , _Gen. Rel. Grav._ 41 (2009) 919–981, [0901.4661].


# Consistent Dropout for Policy Gradient Reinforcement Learning

Matthew Hausknecht  Nolan Wagener

###### Abstract

Dropout has long been a staple of supervised learning, but is rarely used in
reinforcement learning. We analyze why naïve application of dropout is
problematic for policy-gradient learning algorithms and introduce consistent
dropout, a simple technique to address this instability. We demonstrate
consistent dropout enables stable training with A2C and PPO in both continuous
and discrete action environments across a wide range of dropout probabilities.
Finally, we show that consistent dropout enables the online training of
complex architectures such as GPT without needing to disable the model’s
native dropout.

Machine Learning, ICML

##  1 Introduction

Dropout, the practice of randomly zeroing a fraction of unit activations at
select neural network layers, is an established technique for regularizing
neural network learning and reducing overfitting (Srivastava et al., 2014).
However, in the field of reinforcement learning, dropout is rarely applied.
Consider the large body of work without dropout, including high-profile
projects like AlphaGo (Silver et al., 2016), AlphaStar (Vinyals et al., 2019),
and OpenAI Five (Berner et al., 2019) – none of which mention dropout. Even
models that are commonly supervised-trained with dropout, such as Transformers
(Vaswani et al., 2017), disable dropout entirely when repurposed for
reinforcement learning (Parisotto et al., 2019; Loynd et al., 2020). This
trend is reinforced by our open source tools: Surveying six popular open
source RL frameworks111Surveyed frameworks were Dopamine (Castro et al.,
2018), RLlib (Liang et al., 2017), OpenAI Baselines (Dhariwal et al., 2017),
Tianshou (Weng et al., 2020), Stable-Baselines3 (Raffin et al., 2019), and
rlpyt (Stooke & Abbeel, 2019)., we found none that supported dropout.

We contend that the naïve application of dropout in reinforcement learning can
be detrimental to learning performance. To test this hypothesis, we added
dropout layers to the policy networks of A2C and PPO agents in three RL
frameworks: RLlib (Liang et al., 2017), Stable-Baselines3 (Raffin et al.,
2019), and Tianshou (Weng et al., 2020). Agents were then trained on the
MuJoCo Half-Cheetah-v3 continuous control environment using the hyperparmeters
recommended by their respective codebases. To summarize results in Appendix B,
Tianshou’s A2C experienced large negative returns (e.g., $-10^{7}$) in five of
sixteen runs, and all sixteen PPO runs prematurely terminated due to NaN
values. Similarly, RLlib experienced instability in five of six A2C runs and
two of six PPO runs. Stable-Baselines3, on the other hand, completed all runs
without error, though we found that A2C experienced large policy loss
magnitudes on seven of twelve runs and PPO declined in performance by $21\%$
on average with dropout. Thus, we conclude that instability stemming from
naïve application of dropout is not an artifact of a single implementation or
codebase but instead affects to various degrees all of the tested
frameworks.222Notably, the use of Tanh nonlinearities in policy networks
reduced instability compared to ReLU. Although we did not perform a detailed
investigation, we suspect this relates to Tanh bounding the network outputs.

To better understand why dropout is problematic, Section 4 shows the
instability stems from different dropout masks being applied to the same
observations during rollouts and updates. This mismatch leads to policy
collapse for A2C and reduced performance for PPO. Furthermore, we show that
more complex networks such as GPT (Radford et al., 2019) suffer more acutely
than MLPs from the instabilities introduced by naïve application of dropout.
To correct this instability, Section 5 introduces consistent dropout, a simple
and mathematically sound technique that consists of saving the random dropout
masks generated during rollouts and reapplying the same masks during updates.
Experimental results in Section 6 show this technique stabilizes learning for
both A2C and PPO on discrete and continuous environments across a spectrum of
dropout probabilities. We also show this technique allows for GPT to be
trained online without needing to disable dropout.

The contributions of this paper are:  1) Demonstrate that naïve application of
dropout creates instability for policy gradient methods across a variety of
open-source RL frameworks, algorithms, and environments.  2) Analyze the
source of instability and introduce a simple method of saving and reapplying
dropout masks to stabilize training.  3) Demonstrate stable online RL training
of GPT with dropout.

##  2 Related Work

Farebrother et al. (2018) investigated dropout as a regularization technique
for the DQN learning agent (Mnih et al., 2015) on Atari environments. Their
findings indicate that moderate levels of dropout ($p=0.1$), paired with
$\ell_{2}$ regularization improved zero-shot generalization on variants of the
training environment. However, the authors note that dropout slowed the
training performance and typically required more training iterations to
compete with the unregularized baseline.

Similarly, Cobbe et al. (2018) examined the effects of dropout and $\ell_{2}$
regularization on the Procgen benchmark (Cobbe et al., 2019). Using the PPO
algorithm, they found both $\ell_{2}$ regularization and moderate levels of
dropout ($p=0.1$) aided generalization to held-out test environments, but also
slightly reduced performance on training environments.333Inspection of the
CoinRun codebase indicates Cobbe et al. implemented a variant of consistent
dropout, but it is neither mentioned in the paper nor analyzed.

Dropout has been successfully applied in sequential decision making settings
when training is based on offline RL or imitation learning. Examples such as
the Decision Transformer (Chen et al., 2021) and Trajectory Transformer
(Janner et al., 2021) show that transformer models are capable of learning
high quality policies from offline expert data. In this work we show how such
models can be additionally trained with online RL, potentially enabling a
paradigm of offline pretraining followed by online finetuning.

##  3 Preliminaries

We consider an infinite-horizon Markov decision process (MDP) defined as the
tuple $\mathcal{M}=(\SS,\mathcal{A},p,R,\gamma)$, where $\SS$ is the state
space, $\mathcal{A}$ is the action space, $p(s^{\prime}|s,a)$ is the
transition kernel, $R(s,a)$ is the reward function, and $\gamma$ is the
discount factor. Let $\pi_{\theta}(a|s)$ be a stochastic policy parameterized
by some vector $\theta$. Given some start state distribution $d_{0}(s_{0})$,
the policy and transition kernel induce a trajectory distribution
$\rho^{\pi_{\theta}}(\tau)$, where $\tau=(s_{0},a_{0},s_{1},a_{1},\dots)$. Our
goal is to find a parameter setting that maximizes the expected sum of
discounted rewards in the MDP:

$\max_{\theta}\,J(\theta)=\mathbb{E}_{\rho^{\pi_{\theta}}(\tau)}\left[\sum_{t=0}^{\infty}\gamma^{t}R(s_{t},a_{t})\right].$

We define the value function $V^{\pi_{\theta}}(s)$ and state-action value
function $Q^{\pi_{\theta}}(s,a)$ as:

$\displaystyle V^{\pi_{\theta}}(s)$
$\displaystyle=\mathbb{E}_{\rho^{\pi_{\theta}}(\tau)}\left[\sum_{t=0}^{\infty}\gamma^{t}R(s_{t},a_{t})\;\Bigg{|}\;s_{0}=s\right]$
$\displaystyle Q^{\pi_{\theta}}(s,a)$
$\displaystyle=\mathbb{E}_{\rho^{\pi_{\theta}}(\tau)}\left[\sum_{t=0}^{\infty}\gamma^{t}R(s_{t},a_{t})\;\Bigg{|}\;s_{0}=s,a_{0}=a\right]$

and furthermore define the advantage function
${A^{\pi_{\theta}}(s,a)=Q^{\pi_{\theta}}(s,a)-V^{\pi_{\theta}}(s)}$.

Let $d_{t}^{\pi_{\theta}}(s)$ be the state distribution induced by running
$\pi_{\theta}$ in $\mathcal{M}$ for $t$ time steps from the initial state
distribution $d_{0}$. Defining the state visitation
${d^{\pi_{\theta}}(s)=(1-\gamma)\sum_{t=0}^{\infty}\gamma^{t}d_{t}^{\pi_{\theta}}(s)}$,
it can be shown (Sutton et al., 2000) that the policy gradient has the form

$\nabla
J(\theta)=\mathbb{E}_{d^{\pi_{\theta}}(s)}\mathbb{E}_{\pi_{\theta}(a|s)}\left[A^{\pi_{\theta}}(s,a)\nabla_{\theta}\log\pi_{\theta}(a|s)\right].$
(1)

Practical implementations of policy gradient algorithms roll out
$\pi_{\theta}$ and collect state-action-reward triplets $(s,a,r)$ into a
buffer $\mathcal{D}$. After learning an approximate advantage function
$\hat{A}(s,a)$ (e.g., through generalized advantage estimation (Schulman et
al., 2016)), they form the loss function

$L_{PG}(\theta)=-\mathbb{E}_{(s,a)\sim\mathcal{D}}[\hat{A}(s,a)\log\pi_{\theta}(a|s)]$
(2)

and subsequently perform a gradient descent step.

##  4 Why Dropout Disrupts Policy Gradient

We now assume that $\pi_{\theta}$ is a multi-layer neural network where there
is a subset of neurons wherein each activation is zeroed out with some
probability, as shown in Fig. 1. Letting $m$ be the resulting dropout mask,
our stochastic policy has the form
$\pi_{\theta}(a|s)=\sum_{m}p(m)\pi_{\theta}(a|s,m)$, where $p(m)$ is the
probability function of the mask. In practice, dropout is treated as an
unobserved internal process of the network, so that we observe a noisy
estimator of $\pi_{\theta}(a|s)$.

Figure 1: An example of a policy after a dropout mask $m$ has been sampled.
The neurons in red have their activations zeroed out.

Due to this practice, at rollout time, we sample some dropout mask $\hat{m}$
at a given state $s$ and then sample an action from
$\pi_{\theta}(a|s,\hat{m})$ to apply in the MDP. At update time, we sample a
new dropout mask when querying the policy for its log-probability. This
results in a modified loss function for policy optimization:

$\widetilde{L}_{PG}(\theta)=-\mathbb{E}_{(s,a)\sim\mathcal{D}}\mathbb{E}_{p(m)}[\hat{A}(s,a)\log\pi_{\theta}(a|s,m)].$

The use of different dropout masks for the same state $s$ at rollout time
versus update time can lead to vastly different action probabilities. For A2C,
this manifests in $\pi_{\theta}(a|s)$ being close to zero and
$\log\pi_{\theta}(a|s)$ approaching negative infinity. At this point, updates
become unstable as policy loss and gradient magnitudes explode. In practice,
gradient norm clipping can keep training numerically stable, but learning
performance still suffers due to the gradient estimator having high variance.
Table 1 demonstrates how the application of different dropout masks can change
the outputs and lower the log probabilities of actions in an untrained, newly
initialized network. These effects only become more pronounced over the course
of training.

Table 1: Forward passes over the same state with different dropout masks
change output actions and reduce the log probabilities: Let $a^{0}$ and
$a^{1}$ be the most likely (i.e., mode) actions from two forward passes on the
same state $s$ using random dropout masks $m^{0}$ and $m^{1}$, respectively.
For continuous actions, $d(a^{0},a^{1})$ is the absolute difference
$|a^{0}-a^{1}|$. For discrete actions, $d(a^{0},a^{1})$ indicates the
frequency of disagreement $\mathbb{P}[a^{0}\neq a^{1}]$. Higher levels of
dropout result in greater differences in actions as well as lower log
probabilities. This effect is more pronounced in GPT, in which a dropout
probability of $0.1$ is approximately equivalent to an MLP with dropout of
$0.75$.
Net | Dropout | $d(a^{0},a^{1})$ | $\log\pi_{\theta}(a^{0}|s,m^{1})$
---|---|---|---
3-Layer | $0$ | $0\pm 0$ | $-0.83\pm 0.00$
MLP w/ | $0.1$ | $0.04\pm 0.03$ | $-0.84\pm 0.01$
4 continuous | $0.25$ | $0.06\pm 0.05$ | $-0.86\pm 0.02$
actions | $0.5$ | $0.12\pm 0.09$ | $-0.90\pm 0.07$
| $0.75$ | $0.23\pm 0.25$ | $-1.13\pm 0.38$
| $0.9$ | $0.63\pm 0.65$ | $-6.36\pm 7.04$
4-Layer | $0$ | $0\pm 0$ | $-0.83\pm 0.00$
GPT w/ | $0.1$ | $0.26\pm 0.20$ | $-1.14\pm 0.33$
4 continuous | $0.25$ | $0.41\pm 0.32$ | $-1.57\pm 0.77$
actions | $0.5$ | $0.52\pm 0.40$ | $-2.01\pm 1.21$
| $0.75$ | $0.62\pm 0.47$ | $-2.47\pm 1.63$
| $0.9$ | $0.64\pm 0.49$ | $-2.64\pm 1.80$
3-Layer | $0$ | $0\pm 0$ | $-1.31\pm 0.03$
MLP w/ | $0.1$ | $0.18\pm 0.39$ | $-1.35\pm 0.04$
4 discrete | $0.25$ | $0.51\pm 0.50$ | $-1.34\pm 0.05$
actions | $0.5$ | $0.60\pm 0.48$ | $-1.34\pm 0.10$
| $0.75$ | $0.71\pm 0.47$ | $-1.39\pm 0.21$
| $0.9$ | $0.75\pm 0.43$ | $-1.51\pm 0.58$

###  4.1 Dropout and PPO

PPO (Schulman et al., 2017) optimizes a clipped objective that ensures the
deviation of the current policy $\pi_{\theta}$ from the rollout policy
$\pi_{\theta_{\mathrm{old}}}$ is small:

$\displaystyle L_{PPO}(\theta)=-\mathbb{E}_{\mathcal{D}}[\min\\{$
$\displaystyle r(s,a,\theta)\hat{A}(s,a),$
$\displaystyle\mathrm{clip}(r(s,a,\theta),1-\epsilon,1+\epsilon)\hat{A}(s,a)\\}],$

where
$r(s,a,\theta)=\frac{\pi_{\theta}(a|s)}{\pi_{\theta_{\mathrm{old}}}(a|s)}$ is
the ratio of the action probabilities under the new and old policies,
respectively, and $\epsilon$ is a small constant which determines how far the
policy is allowed to diverge from the old policy. In addition to this clipped
objective, PPO also has the option to stop taking gradient steps if the KL
divergence between the new and old policies grows past a threshold. This early
stopping mechanism can help ensure monotonic policy improvement by preventing
gradient updates from extrapolating too far in any direction.

Adding dropout to the policy network increases the variance in the ratio of
action probabilities $r$. The PPO objective clips this variance and prevents
policy loss $L_{PPO}(\theta)$ from exploding. When using an early stopping
threshold, we find that increased variance in $r$ will often trigger immediate
early stopping, preventing any updates from taking place. To preface our
results, without early stopping we observe stable learning but notably reduced
performance.

##  5 Policy Gradient with Dropout

We now present two methods to correctly compute the policy gradient in Eq. 1
in the presence of dropout.

###  5.1 Dropout-Marginalized Gradient

This method gives the correct policy gradient by marginalizing over possible
dropout masks that are randomly sampled at update time: For some $s\in\SS$ and
$a\in\mathcal{A}$, the policy’s score function can be computed as:

$\displaystyle\nabla_{\theta}\log\pi_{\theta}(a|s)$
$\displaystyle=\nabla_{\theta}\log\left(\sum_{m}p(m)\pi_{\theta}(a|s,m)\right)$
$\displaystyle=\frac{\sum_{m}p(m)\pi_{\theta}(a|s,m)\nabla_{\theta}\log\pi_{\theta}(a|s,m)}{\sum_{m}p(m)\pi_{\theta}(a|s,m)}$
(3)
$\displaystyle=\sum_{m}p(m|s,a,\theta)\nabla_{\theta}\log\pi_{\theta}(a|s,m).$

Thus, the score function is the expected score function of the dropout-
conditioned policy under the posterior distribution of the dropout mask.

In this work, we use Eq. 3 to compute the score function. Since it is
intractable to enumerate every dropout mask, we instead rely on sampling. For
some $s\in\SS$ and $a\in\mathcal{A}$, we sample $N$ dropout masks
$m_{1},\dots,m_{N}$ i.i.d. from $p(m)$. For some mask $m_{n}$, we compute the
weight

$w_{n}=\frac{p(m_{n})\pi_{\theta}(a|s,m_{n})}{\sum_{k=1}^{N}p(m_{k})\pi_{\theta}(a|s,m_{k})},$

whereupon the score function can be estimated as:

$\nabla_{\theta}\log\pi_{\theta}(a|s)\approx\sum_{n=1}^{N}w_{n}\nabla_{\theta}\log\pi_{\theta}(a|s,m_{n}).$

We note that the weights approximate the mask posterior, i.e., $w_{n}\approx
p(m_{n}|s,a,\theta)$. The estimated score function can then be used to compute
the policy gradient in Eq. 1.

This method has the advantage of not needing to store the dropout mask $m$
when rolling out the policy $\pi_{\theta}$, but the multiple samples required
to compute the approximate mask posterior can lead to computational overhead
for larger networks.

###  5.2 Dropout-Conditioned Gradient

Figure 2: Stochastic computation graph (Schulman et al., 2015) of the MDP
including the dropout mask $m$

Consider the modified MDP (Fig. 2) in which the policy outputs both the action
and the dropout mask $(a,m)$, so that the stochastic policy is now
${\pi_{\theta}(a,m|s)=p(m)\pi_{\theta}(a|s,m)}$. This MDP is equivalent to the
original since neither the dynamics nor the reward are modified, i.e.,
${p(s^{\prime}|s,a,m)=p(s^{\prime}|s,a)}$ and ${R(s,a,m)=R(s,a)}$. As a
consequence, the advantage function $A^{\pi_{\theta}}$ remains a function only
of the state $s$ and original action $a$.

Now, the score function of the policy is:

$\displaystyle\nabla_{\theta}\log\pi_{\theta}(a,m|s)$
$\displaystyle=\nabla_{\theta}(\log p(m)+\log\pi_{\theta}(a|s,m))$
$\displaystyle=\nabla_{\theta}\log\pi_{\theta}(a|s,m)$

so that the resultant policy gradient is

$\displaystyle\nabla J(\theta)$
$\displaystyle=\mathbb{E}_{d^{\pi_{\theta}}(s)}\mathbb{E}_{\pi_{\theta}(a,m|s)}\left[A^{\pi_{\theta}}(s,a)\nabla_{\theta}\log\pi_{\theta}(a,m|s)\right]$
$\displaystyle=\mathbb{E}_{d^{\pi_{\theta}}(s)}\mathbb{E}_{\pi_{\theta}(a,m|s)}\left[A^{\pi_{\theta}}(s,a)\nabla_{\theta}\log\pi_{\theta}(a|s,m)\right].$

Thus, we conclude that conditioning the policy on the same mask $m$ that is
sampled during rollouts results in an unbiased policy gradient at update time.
Finally, we note that while the derivations in the preceding sections are
specialized to dropout, they can be easily extended to other networks that
internally use stochasticity, such as Bayesian neural networks and hidden
layers that rely on sampling.

###  5.3 Implementation

From an implementation perspective, due to the computational challenges of
using the dropout-marginalized gradient (explored in Section 6.1), we focus on
the dropout-conditioned gradient for the remainder of this work and refer to
it as _consistent dropout_. At rollout time, for some state $s$, we sample a
mask $m$ and action $a$ from $\pi_{\theta}(a,m|s)$, observe a reward $r$, and
store the tuple $(s,a,r,m)$ into a buffer $\mathcal{D}$. During subsequent
updates, the saved masks can be reapplied with the corresponding states and
actions. The resulting policy loss we optimize has the form:

$L_{PG}(\theta)=-\mathbb{E}_{(s,a,m)\sim\mathcal{D}}[\hat{A}(s,a)\log\pi_{\theta}(a|s,m)].$

This reuse of dropout masks ensures that any change in the policy network’s
output is due to a change in the policy network’s weights, rather than a
different dropout mask.

To facilitate easy storage and retrieval of masks, we design a custom dropout
layer shown in Figure 3 that retains the most recent masks from the forward
pass, where they can be easily stored and reapplied at update time. From a
computational perspective, this approach is essentially just as fast as the
original, but does require additional storage of dropout masks whose size and
number vary based on the architecture of the policy net. However, by storing
masks as binary arrays, we find they often take less space than states or
actions. Finally, additional source code and examples showing the use of this
consistent dropout layer may be found in the supplementary materials.

⬇

import torch

from torch import nn

class ConsistentDropout(nn.Module):

"""

␣␣␣␣Consistent␣Dropout␣Layer␣uses␣masks␣provided␣in␣source␣and␣stores␣masks␣in␣sink.

␣␣␣␣"""

def __init__(self, source, sink, p=0.5):

super().__init__()

self.source = source

self.sink = sink

self.p = p

self.scale_factor = 1 / (1 - self.p)

def forward(self, x):

if self.training:

if self.source:

mask = self.source.pop(0)

else:

mask = torch.bernoulli(x.data.new(x.data.size()).fill_(1-self.p))

self.sink.append(mask.bool())

return x * mask * self.scale_factor

else:

return x

class ConsistentPolicyNet(nn.Module):

def __init__(self, obs_size, action_size, hidden_size, dropout):

super().__init__()

self.source, self.sink = [], []

self.policy_net = nn.Sequential(

nn.Linear(obs_size, hidden_size),

nn.ReLU(),

ConsistentDropout(self.source, self.sink, p=dropout),

nn.Linear(hidden_size, hidden_size),

nn.ReLU(),

ConsistentDropout(self.source, self.sink, p=dropout),

nn.Linear(hidden_size, action_size)

)

def forward(self, obs, dropout_masks=None):

if dropout_masks:

self.source.extend(dropout_masks)

action = self.policy_net(obs)

masks = self.sink.copy()

self.source.clear()

self.sink.clear()

return action, masks

Figure 3: Consistent Dropout Layer: PyTorch implementation of the consistent
dropout layer and modifications needed to integrate into a policy network: The
ConsistentDropout layer will retrieve and apply a dropout mask if provided in
source and save the used dropout mask to sink. ConsistentPolicyNet
demonstrates the integration of the ConsistentDropout layer into a policy
network: Both source and sink are implemented using lists shared between all
dropout layers in the network. In the forward pass, if dropout masks are
provided they will be applied in correct order. Otherwise, random masks will
be generated and returned. Returned masks can then be stored in the agent’s
replay memory and reapplied at update time.

##  6 Experiments

Table 2: Relative Performance Under Dropout: Training performance relative to
a baseline that does not use dropout. Across most environments, A2C with
inconsistent dropout becomes unstable and achieves nearly zero performance.
Consistent A2C (A2C-C) and Consistent PPO (PPO-C) achieve significantly more
performance than their inconsistent counterparts on all MuJoCo environments.
In Atari environments, PPO and PPO-C are much closer in relative performance
with the trend favoring PPO-C.
Drop | A2C | A2C-C | PPO | PPO-C
---|---|---|---|---
HalfCheetah-v3
$0.1$ | $0\pm 0$ | $\mathbf{0.77}\pm 0.03$ | $0.61\pm 0.08$ | $\mathbf{0.95}\pm 0.07$
$0.25$ | $0\pm 0$ | $\mathbf{0.72}\pm 0.09$ | $0.21\pm 0.01$ | $\mathbf{0.82}\pm 0.12$
$0.5$ | $0\pm 0$ | $\mathbf{0.49}\pm 0.03$ | $0.07\pm 0.09$ | $\mathbf{0.55}\pm 0.11$
Hopper-v3
$0.1$ | $0\pm 0$ | $\mathbf{1.04}\pm 0.06$ | $\mathbf{1.08}\pm 0.31$ | $1.04\pm 0.2$
$0.25$ | $0\pm 0$ | $\mathbf{1.05}\pm 0.03$ | $0.79\pm 0.10$ | $\mathbf{1.00}\pm 0.21$
$0.5$ | $0\pm 0$ | $\mathbf{0.98}\pm 0.05$ | $0.31\pm 0.05$ | $\mathbf{1.08}\pm 0.34$
Walker2d-v3
$0.1$ | $0\pm 0$ | $\mathbf{0.90}\pm 0.10$ | $0.27\pm 0.05$ | $\mathbf{0.91}\pm 0.08$
$0.25$ | $0\pm 0$ | $\mathbf{0.96}\pm 0.14$ | $0.08\pm 0.01$ | $\mathbf{0.55}\pm 0.05$
$0.5$ | $0\pm 0$ | $\mathbf{0.50}\pm 0.08$ | $0.02\pm 0.00$ | $\mathbf{0.40}\pm 0.03$
Beamrider
$0.1$ | $0.14\pm 0.10$ | $\mathbf{0.48}\pm 0.07$ | $0.38\pm 0.04$ | $\mathbf{0.70}\pm 0.22$
$0.25$ | $0.08\pm 0.06$ | $\mathbf{0.43}\pm 0.03$ | $0.40\pm 0.07$ | $\mathbf{0.51}\pm 0.25$
$0.5$ | $0.07\pm 0.10$ | $\mathbf{0.23}\pm 0.09$ | $0.16\pm 0.02$ | $\mathbf{0.17}\pm 0.03$
Breakout
$0.1$ | $0\pm 0$ | $\mathbf{1.12}\pm 0.63$ | $0.64\pm 0.03$ | $\mathbf{0.77}\pm 0.04$
$0.25$ | $0.01\pm 0$ | $\mathbf{0.94}\pm 0.13$ | $0.56\pm 0.10$ | $\mathbf{0.68}\pm 0.08$
$0.5$ | $0.01\pm 0$ | $\mathbf{0.36}\pm 0.13$ | $\mathbf{0.53}\pm 0.08$ | $0.28\pm 0.03$
Seaquest
$0.1$ | $1.56\pm 1.45$ | $\mathbf{2.08}\pm 0.53$ | $\mathbf{1.13}\pm 0.09$ | $1.10\pm 0.25$
$0.25$ | $0.34\pm 0.39$ | $\mathbf{1.57}\pm 0.34$ | $\mathbf{0.99}\pm 0.06$ | $0.97\pm 0.07$
$0.5$ | $1.5\pm 1.33$ | $\mathbf{2.13}\pm 0.89$ | $1.01\pm 0.12$ | $\mathbf{1.10}\pm 0.32$

In order to compare consistent and inconsistent dropout, we evaluate A2C and
PPO on several popular continuous and discrete environments. In particular,
the aim of our experiments is to empirically understand the effects of
inconsistent dropout on policy gradient algorithms and to understand if
consistent dropout stabilizes policy gradient learning.

###  6.1 Continuous-Action Environments

We conduct experiments on the Hopper, Walker2d, and Half-Cheetah MuJoCo tasks
to compare the stability and performance of A2C and PPO when trained with
consistent and inconsistent dropout probabilities of $(0/0.1/0.25/0.5)$.

Results on Half-Cheetah are shown in Figure 5 and similar results from the
other MuJoCo tasks may be found in Appendix C. Across all MuJoCo environments,
we find that A2C is highly unstable for any level of inconsistent dropout. PPO
is more robust to inconsistent dropout and often remains stable and performant
at dropout of $0.1$. However, dropout probabilities of $0.25$ and $0.5$
degrade PPO’s performance significantly.

On the other hand, as shown in Table 2, consistent dropout effectively
stabilizes A2C and PPO across all MuJoCo environments and dropout levels. All
levels of consistent dropout lead to stable learning, although dropout of
$0.5$ notably reduces performance for both algorithms.

In terms of absolute performance, baseline models without dropout perform
slightly better on average than equivalent models trained with dropout. This
finding reflects the intuition that when training and testing on the same
environment, less regularization is likely beneficial.

We also experimented with a version of PPO that uses an estimate of the
marginalized gradient in Eq. 3. As shown in Fig. 4, the large variance of the
gradient estimate dominates the learning, even with 100 samples used for Eq.
3.

(a) $N=10$

(b) $N=100$

Figure 4: Marginalized Gradient: PPO training with marginalized gradients on
HalfCheetah-v3. High variance gradient estimates lead to highly unstable
learning dynamics.

(a) A2C

(b) A2C Consistent

(c) PPO

(d) PPO Consistent

Figure 5: MuJoCo-HalfCheetah-v3: Clipped score of A2C and PPO with and without
consistent dropout probabilities $0/0.1/0.25/0.5$. Curves are averages of
three independent training runs.

(a) A2C

(b) A2C Consistent

(c) PPO

(d) PPO Consistent

Figure 6: Atari-Beamrider: Clipped score of A2C and PPO with inconsistent and
consistent dropout probabilities $0/0.1/0.25/0.5$. Curves reflect averages of
three independent training runs.

###  6.2 Discrete-Action Environments

We tested three discrete-action Atari environments (Beamrider, Breakout, and
Seaquest), and as shown in Table 2 A2C is intolerant to any amount of
inconsistent dropout. On the other hand, PPO often benefits from low-levels of
inconsistent dropout (e.g., $0.1$). We believe the source of this benefit
stems from an increase in policy entropy. Nevertheless, inconsistent dropout
triggers PPO’s KL divergence-based early stopping, which prevents updates from
occurring and precludes any policy learning. Therefore, to facilitate training
we disabled PPO’s early stopping and forced a set number of gradient steps to
be performed at each update.

Consistent dropout leads to stable learning (Fig. 6), but we observed that
policies can become overly confident in their actions, as evidenced by quickly
growing action probabilities and shrinking policy entropy. Full Atari results
may be found in Appendix D.

###  6.3 Stable GPT Training

Prior work on training transformer models with reinforcement learning disable
dropout (Parisotto et al., 2019; Loynd et al., 2020). This is unsurprising as
even small levels of dropout cause much larger changes to GPT’s outputs
compared to similarly deep MLP networks (Table 1). As a result, PPO models
trained with inconsistent dropout are unable to exceed random performance on
Half-Cheetah. On the other hand, consistent dropout (Fig. 7) enables stable
GPT training across all dropout probabilities. Extrapolating from this result,
we believe consistent dropout is applicable to a variety of network
architectures.

(a) PPO

(b) PPO Consistent

Figure 7: GPT Training: PPO training of GPT models on HalfCheetah-v3. With any
level of inconsistent dropout all PPO models are unable to learn. Consistent
dropout stabilizes training for all dropout probabilities.

(a) Dropout

(b) No Dropout

Figure 8: Evaluating with Dropout: Evaluation performance of Consistent PPO on
Half-Cheetah-v3 with dropout enabled and disabled for dropout probabilities
$0/0.1/0.25/0.5$. Similar to supervised learning, disabling dropout at
evaluation time increases evaluation performance, particularly with a dropout
of $0.5$. Curves reflect evaluations performed on models from three
independent training runs.

##  7 Evaluating with Dropout

Common practice for supervised learning is to disable dropout at evaluation
time. However, it is unclear whether this practice would be beneficial for
sequential decision making. To answer this question, we examined the effects
of enabling or disabling dropout at evaluation time, and found a strong
preference towards disabling dropout at evaluation time (Fig. 8). Specifically
we observe a $\mathbf{2\%}$ improvement on average in final evaluation score
when disabling dropout at evaluation time for the $p=0.1$ models, a
$\mathbf{17\%}$ improvement for the $p=0.25$ models, and a $\mathbf{46\%}$
improvement in score for the $p=0.5$ models. Finally, we find that models
trained with inconsistent dropout are largely unaffected by enabling or
disabling dropout at evaluation time. Further evaluation results may be found
in Appendix C.

##  8 Limitations

This study pertains only to on-policy policy gradient methods. Other types of
reinforcement learning algorithms such as off-policy and model-based
reinforcement learning are affected by dropout in different ways, which are
beyond the scope of this paper.

Having stabilized dropout for policy gradient algorithms, our findings raise
the larger question of whether dropout aids generalization to unseen tasks or
novel variations of training environments. Related work (Cobbe et al., 2018;
Farebrother et al., 2018) has found beneficial generalization effects from
small amounts of dropout, and its common use in supervised settings suggests
that dropout may aid generalization especially when training overparameterized
networks. However, we expect that thoroughly exploring how dropout contributes
to generalization in sequential decision settings is a substantial
undertaking, one that lies beyond the scope of this work.

##  9 Conclusions

We demonstrate that naïve application of dropout to policy gradient
reinforcement learning results in instability due to the mismatch between the
dropout masks that are applied during rollouts and those used during updates.
We address this instability through consistent dropout, a simple approach for
saving and reapplying the same dropout masks during rollouts and updates. We
demonstrate that this solution is both theoretically sound and empirically
stable on a variety of continuous and discrete actions environments.
Additionally, we show that this technique scales to enable stable online
training of transformer models such as GPT. Given the complete lack of support
for dropout in open-source reinforcement learning frameworks, we hope this
work will pave the way for a re-evaluation of this time-tested technique.

## References

  * Berner et al. (2019) Berner, C., Brockman, G., Chan, B., Cheung, V., Debiak, P., Dennison, C., Farhi, D., Fischer, Q., Hashme, S., Hesse, C., Józefowicz, R., Gray, S., Olsson, C., Pachocki, J., Petrov, M., de Oliveira Pinto, H. P., Raiman, J., Salimans, T., Schlatter, J., Schneider, J., Sidor, S., Sutskever, I., Tang, J., Wolski, F., and Zhang, S.  Dota 2 with large scale deep reinforcement learning.  _CoRR_ , abs/1912.06680, 2019.  URL http://arxiv.org/abs/1912.06680.
  * Castro et al. (2018) Castro, P. S., Moitra, S., Gelada, C., Kumar, S., and Bellemare, M. G.  Dopamine: A Research Framework for Deep Reinforcement Learning.  2018\.  URL http://arxiv.org/abs/1812.06110.
  * Chen et al. (2021) Chen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., Abbeel, P., Srinivas, A., and Mordatch, I.  Decision transformer: Reinforcement learning via sequence modeling.  _CoRR_ , abs/2106.01345, 2021.  URL https://arxiv.org/abs/2106.01345.
  * Cobbe et al. (2018) Cobbe, K., Klimov, O., Hesse, C., Kim, T., and Schulman, J.  Quantifying generalization in reinforcement learning.  _CoRR_ , abs/1812.02341, 2018.  URL http://arxiv.org/abs/1812.02341.
  * Cobbe et al. (2019) Cobbe, K., Hesse, C., Hilton, J., and Schulman, J.  Leveraging procedural generation to benchmark reinforcement learning.  _arXiv preprint arXiv:1912.01588_ , 2019.
  * Dhariwal et al. (2017) Dhariwal, P., Hesse, C., Klimov, O., Nichol, A., Plappert, M., Radford, A., Schulman, J., Sidor, S., Wu, Y., and Zhokhov, P.  OpenAI Baselines.  https://github.com/openai/baselines, 2017.
  * Farebrother et al. (2018) Farebrother, J., Machado, M. C., and Bowling, M.  Generalization and regularization in DQN.  _CoRR_ , abs/1810.00123, 2018.  URL http://arxiv.org/abs/1810.00123.
  * Janner et al. (2021) Janner, M., Li, Q., and Levine, S.  Offline reinforcement learning as one big sequence modeling problem.  _Advances in Neural Information Processing Systems_ , 34, 2021.
  * Liang et al. (2017) Liang, E., Liaw, R., Nishihara, R., Moritz, P., Fox, R., Gonzalez, J., Goldberg, K., and Stoica, I.  Ray RLLib: A Composable and Scalable Reinforcement Learning Library.  _CoRR_ , abs/1712.09381, 2017.  URL http://arxiv.org/abs/1712.09381.
  * Loynd et al. (2020) Loynd, R., Fernandez, R., Celikyilmaz, A., Swaminathan, A., and Hausknecht, M.  Working memory graphs.  In _International Conference on Machine Learning_ , pp. 6404–6414. PMLR, 2020.
  * Mnih et al. (2015) Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., and Hassabis, D.  Human-level control through deep reinforcement learning.  _Nature_ , 518(7540):529–533, February 2015.  ISSN 00280836.  URL http://dx.doi.org/10.1038/nature14236.
  * Parisotto et al. (2019) Parisotto, E., Song, H. F., Rae, J. W., Pascanu, R., Gülçehre, Ç., Jayakumar, S. M., Jaderberg, M., Kaufman, R. L., Clark, A., Noury, S., Botvinick, M. M., Heess, N., and Hadsell, R.  Stabilizing transformers for reinforcement learning.  _CoRR_ , abs/1910.06764, 2019.  URL http://arxiv.org/abs/1910.06764.
  * Radford et al. (2019) Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I.  Language models are unsupervised multitask learners.  2019\.
  * Raffin et al. (2019) Raffin, A., Hill, A., Ernestus, M., Gleave, A., Kanervisto, A., and Dormann, N.  Stable-Baselines3.  https://github.com/DLR-RM/stable-baselines3, 2019.
  * Schulman et al. (2015) Schulman, J., Heess, N., Weber, T., and Abbeel, P.  Gradient estimation using stochastic computation graphs.  In _Advances in Neural Information Processing Systems_ , 2015.
  * Schulman et al. (2016) Schulman, J., Moritz, P., Levine, S., Jordan, M., and Abbeel, P.  High-dimensional continuous control using generalized advantage estimation.  In _International Conference on Learning Representations_ , 2016.
  * Schulman et al. (2017) Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O.  Proximal policy optimization algorithms.  _CoRR_ , abs/1707.06347, 2017.  URL http://arxiv.org/abs/1707.06347.
  * Silver et al. (2016) Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., Dieleman, S., Grewe, D., Nham, J., Kalchbrenner, N., Sutskever, I., Lillicrap, T., Leach, M., Kavukcuoglu, K., Graepel, T., and Hassabis, D.  Mastering the game of Go with deep neural networks and tree search.  _Nature_ , 529(7587):484–489, Jan 2016.  ISSN 0028-0836.  doi: 10.1038/nature16961.
  * Srivastava et al. (2014) Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R.  Dropout: A simple way to prevent neural networks from overfitting.  _Journal of Machine Learning Research_ , 15(56):1929–1958, 2014.  URL http://jmlr.org/papers/v15/srivastava14a.html.
  * Stooke & Abbeel (2019) Stooke, A. and Abbeel, P.  rlpyt: A Research Code Base for Deep Reinforcement Learning in PyTorch.  _CoRR_ , abs/1909.01500, 2019.  URL http://arxiv.org/abs/1909.01500.
  * Sutton et al. (2000) Sutton, R. S., McAllester, D. A., Singh, S. P., and Mansour, Y.  Policy gradient methods for reinforcement learning with function approximation.  In _Advances in Neural Information Processing Systems_ , pp. 1057–1063, 2000.
  * Vaswani et al. (2017) Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u., and Polosukhin, I.  Attention is all you need.  In _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc., 2017.  URL https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf.
  * Vinyals et al. (2019) Vinyals, O., Babuschkin, I., Chung, J., Mathieu, M., Jaderberg, M., Czarnecki, W., Dudzik, A., Huang, A., Georgiev, P., Powell, R., Ewalds, T., Horgan, D., Kroiss, M., Danihelka, I., Agapiou, J., Oh, J., Dalibard, V., Choi, D., Sifre, L., Sulsky, Y., Vezhnevets, S., Molloy, J., Cai, T., Budden, D., Paine, T., Gulcehre, C., Wang, Z., Pfaff, T., Pohlen, T., Yogatama, D., Cohen, J., McKinney, K., Smith, O., Schaul, T., Lillicrap, T., Apps, C., Kavukcuoglu, K., Hassabis, D., and Silver, D.  AlphaStar: Mastering the Real-Time Strategy Game StarCraft II.  https://deepmind.com/blog/alphastar-mastering-real-time-strategy-game-starcraft-ii/, 2019\.
  * Weng et al. (2020) Weng, J., Zhang, M., Yan, D., Su, H., and Zhu, J.  Tianshou.  https://github.com/thu-ml/tianshou, 2020.

##  Appendix A Hyperparameters

Table 3: Hyperparameters: Networks consisted of 3-layer MLPs with ReLU
nonlinearities. Dropout was applied between each layer of the network. Note
that PPO Consistent uses a target-KL early stopping condition to terminate
updates when the KL divergence passes a threshold. We attempted to run
standard PPO with the same parameter but found that it would trigger early
termination due to the large KL divergences caused by inconsistent dropout
masks. Thus, the best performance for standard PPO was with the target-KL
disabled.
Variant | Hyperparameter | MuJoCo | Atari
---|---|---|---
A2C | max steps | 1e8 | 1e8
& | workers | 16 | 16
A2C Consistent | steps per epoch | 80 | 5
| learning rate | 7e-4 | 1e-4
| critic lr | 7e-4 | -
| entropy coefficient | 0.01 | 0.01
| discount factor | 0.99 | 0.99
| gae-lambda | 0.95 | 0.95
| hidden size | 64 | 512
| gradient norm clip | 0.5 | 0.5
| rms prop epsilon | 3e-6 | 3e-6
| advantage normalization | 1 | 0
PPO | max steps | 1e8 | 1e8
| workers | 16 | 16
| learning rate | 3e-4 | 1e-4
| steps per epoch | 4096 | 4096
| gradient steps per update | 16 | 16
| batch size | 64 | 64
| hidden size | 64 | 512
| value coefficient | 0.5 | 0.5
| entropy coefficient | 0.01 | 0
| gradient norm clip | 0.5 | 0.5
| discount factor | 0.99 | 0.99
| gae-lambda | 0.97 | 0.95
| clip ratio | 0.2 | 0.2
| target KL | - | -
PPO Consistent | max steps | 1e8 | 1e8
| workers | 16 | 16
| learning rate | 3e-4 | 1e-4
| steps per epoch | 4096 | 4096
| gradient steps per update | 16 | 16
| batch size | 64 | 64
| hidden size | 64 | 512
| value coefficient | 0.5 | 0.5
| entropy coefficient | 0.01 | 0
| gradient norm clip | 0.5 | 0.5
| discount factor | 0.99 | 0.99
| gae-lambda | 0.97 | 0.95
| clip ratio | 0.2 | 0.2
| target KL | 0.01 | 0.01
GPT | max steps | 1e8 |
& | workers | 16 |
GPT Consistent | learning rate | 3e-4 |
| critic learning rate | 7e-4 |
| steps per epoch | 1024 |
| gradient steps per update | 128 |
| batch size | 64 |
| hidden size | 64 |
| entropy coefficient | 0.01 |
| gradient norm clip | 0.5 |
| discount factor | 0.99 |
| gae-lambda | 0.97 |
| clip ratio | 0.2 |
| target KL | 0.01 |
| block size | 8 |
| number of layers | 4 |
| number of heads | 4 |

##  Appendix B RL Framework Testing

The following tables show results from testing inconsistent dropout on three
open-source reinforcement learning frameworks. This was accomplished by adding
dropout layers to the policy networks and running the A2C and PPO on the
HalfCheetah-v3 environment using the hyperparameters suggested by the
respective codebase. In all the following tables, where ReLU was not used,
Tanh was applied.

Table 4: Tianshou (version 0.4.2) on HalfCheetah-v3: For A2C, ReLU activation
seems to be the biggest causative factor of instability, while Tanh is a
stabilizer. ActionClipping can also help stabilize when dropout levels are
moderate. Reward normalization may have a slightly helpful effect, but less
than other factors. PPO even with moderate levels of dropout demonstrated
broad instability: PPO† indicates early termination of the experiment due to
instabilities resulting in NaN values.
Alg | Dropout | RewNorm? | ActClip? | ReLU? | Best | Final
---|---|---|---|---|---|---
A2C | $0$ | ✓ | ✓ | ✗ | $1556$ | $1542$
A2C | $0$ | ✓ | ✓ | ✓ | $2949$ | $2927$
A2C | $0.1$ | ✗ | ✓ | ✗ | $3192$ | $2969$
A2C | $0.1$ | ✗ | ✓ | ✓ | $3085$ | $2693$
A2C | $0.1$ | ✗ | ✗ | ✗ | $3593$ | $3357$
A2C | $0.1$ | ✗ | ✗ | ✓ | $1628$ | $-3\mathrm{e}7$
A2C | $0.1$ | ✓ | ✓ | ✗ | $1193$ | $1152$
A2C | $0.1$ | ✓ | ✓ | ✓ | $537$ | $573$
A2C | $0.1$ | ✓ | ✗ | ✗ | $3066$ | $2900$
A2C | $0.1$ | ✓ | ✗ | ✓ | $3204$ | $-1.2\mathrm{e}8$
A2C | $0.25$ | ✗ | ✓ | ✗ | $1049$ | $977$
A2C | $0.25$ | ✗ | ✓ | ✓ | $171$ | $-13$
A2C | $0.25$ | ✗ | ✗ | ✗ | $1065$ | $-3.4\mathrm{e}7$
A2C | $0.25$ | ✗ | ✗ | ✓ | $-146$ | $-3.9\mathrm{e}9$
A2C | $0.25$ | ✓ | ✓ | ✗ | $1141$ | $1139$
A2C | $0.25$ | ✓ | ✓ | ✓ | $1137$ | $1048$
A2C | $0.25$ | ✓ | ✗ | ✗ | $3872$ | $3313$
A2C | $0.25$ | ✓ | ✗ | ✓ | $260$ | $-2.1\mathrm{e}9$
PPO | $0$ | ✓ | ✓ | ✗ | $7472$ | $5645$
PPO | $0$ | ✓ | ✓ | ✓ | $8743$ | $7890$
PPO† | $0.1$ | ✗ | ✓ | ✗ | $1708$ | $-335$
PPO† | $0.1$ | ✗ | ✓ | ✓ | $1548$ | $463$
PPO† | $0.1$ | ✗ | ✗ | ✗ | $2267$ | $1162$
PPO† | $0.1$ | ✗ | ✗ | ✓ | $1372$ | $1050$
PPO† | $0.1$ | ✓ | ✓ | ✗ | $2238$ | $1813$
PPO† | $0.1$ | ✓ | ✓ | ✓ | $408$ | $42$
PPO† | $0.1$ | ✓ | ✗ | ✗ | $2268$ | $1870$
PPO† | $0.1$ | ✓ | ✗ | ✓ | $1007$ | $-55$
PPO† | $0.25$ | ✗ | ✓ | ✗ | $1001$ | $881$
PPO† | $0.25$ | ✗ | ✓ | ✓ | $279$ | $153$
PPO† | $0.25$ | ✗ | ✗ | ✗ | $1270$ | $1006$
PPO† | $0.25$ | ✗ | ✗ | ✓ | $614$ | $246$
PPO† | $0.25$ | ✓ | ✓ | ✗ | $1292$ | $1167$
PPO† | $0.25$ | ✓ | ✓ | ✓ | $705$ | $-123$
PPO† | $0.25$ | ✓ | ✗ | ✗ | $348$ | $1321$
PPO† | $0.25$ | ✓ | ✗ | ✓ | $348$ | $-160$
Table 5: Stable-Baselines3 (version 1.2.0a0) on HalfCheetah-v3. Policy loss is
shown in to provide an intuition on about the stability of the training
process. Very high policy losses correlate to high-magnitude $\log\pi(a|s)$
values, a symptom of unstable dynamics induced by inconsistent dropout.
Alg | Dropout | ReLU? | Ortho Init? | Final | Policy Loss
---|---|---|---|---|---
A2C | $0$ | ✗ | ✗ | $974$ | $2.5$
A2C | $0$ | ✗ | ✓ | $688$ | $-1.1$
A2C | $0$ | ✓ | ✗ | $526$ | $8$
A2C | $0$ | ✓ | ✓ | $1954$ | $-5.4$
A2C | $0.1$ | ✗ | ✗ | $1003$ | $-12.1$
A2C | $0.1$ | ✗ | ✓ | $650$ | $-8.8$
A2C | $0.1$ | ✓ | ✗ | $1639$ | $8.8\mathrm{e}3$
A2C | $0.1$ | ✓ | ✓ | $-40$ | $-242$
A2C | $0.25$ | ✗ | ✗ | $1325$ | $26$
A2C | $0.25$ | ✗ | ✓ | $499$ | $25$
A2C | $0.25$ | ✓ | ✗ | $1497$ | $-2.9\mathrm{e}5$
A2C | $0.25$ | ✓ | ✓ | $950$ | $-2.7\mathrm{e}5$
A2C | $0.5$ | ✗ | ✗ | $116$ | $1\mathrm{e}3$
A2C | $0.5$ | ✗ | ✓ | $639$ | $-7.2\mathrm{e}3$
A2C | $0.5$ | ✓ | ✗ | $1527$ | $1.7\mathrm{e}8$
A2C | $0.5$ | ✓ | ✓ | $1083$ | $-7.8\mathrm{e}8$
PPO | $0$ | ✗ | ✗ | $2238$ | $-0.007$
PPO | $0$ | ✗ | ✓ | $3211$ | $-0.001$
PPO | $0$ | ✓ | ✗ | $1232$ | $-0.0006$
PPO | $0$ | ✓ | ✓ | $1277$ | $-0.001$
PPO | $0.1$ | ✗ | ✗ | $1963$ | $0.04$
PPO | $0.1$ | ✗ | ✓ | $1256$ | $0.05$
PPO | $0.1$ | ✓ | ✗ | $1830$ | $0.04$
PPO | $0.1$ | ✓ | ✓ | $2463$ | $0.04$
PPO | $0.25$ | ✗ | ✗ | $1468$ | $0.04$
PPO | $0.25$ | ✗ | ✓ | $1383$ | $0.04$
PPO | $0.25$ | ✓ | ✗ | $2217$ | $0.05$
PPO | $0.25$ | ✓ | ✓ | $1825$ | $0.05$
PPO | $0.5$ | ✗ | ✗ | $1331$ | $0.06$
PPO | $0.5$ | ✗ | ✓ | $1328$ | $0.06$
PPO | $0.5$ | ✓ | ✗ | $904$ | $0.06$
PPO | $0.5$ | ✓ | ✓ | $993$ | $0.05$
Table 6: RLlib (version 1.8.0) on HalfCheetah-v3. Dagger denotes training
became numerically unstable and produced errors. For these runs, we report the
final performance level before the error.
Alg | Dropout | ReLU? | Final
---|---|---|---
A2C | $0$ | ✗ | $712$
A2C | $0$ | ✓ | $747$
A2C | $0.1$ | ✗ | $-393$
A2C† | $0.1$ | ✓ | $-439$
A2C† | $0.25$ | ✗ | $-253$
A2C† | $0.25$ | ✓ | $-594$
A2C† | $0.5$ | ✗ | $-643$
A2C† | $0.5$ | ✓ | $-596$
PPO | $0$ | ✗ | $2044$
PPO | $0$ | ✓ | $5390$
PPO | $0.1$ | ✗ | $909$
PPO† | $0.1$ | ✓ | $-109$
PPO | $0.25$ | ✗ | $383$
PPO† | $0.25$ | ✓ | $-238$
PPO | $0.5$ | ✗ | $867$
PPO | $0.5$ | ✓ | $1517$

##  Appendix C Full MuJoCo Results

Hyperparameters were tuned on Half-Cheetah-v3 then deployed without further
modification on Hopper-v3 and Walker2d-v3.

(a) A2C

(b) A2C Consistent

(c) PPO

(d) PPO Consistent

Figure 9: Half-Cheetah-v3: Training performance of various models on MuJoCo
Half-Cheetah-v3. Curves are averages of three independent training runs.

(a) A2C

(b) A2C Consistent

(c) PPO

(d) PPO Consistent

Figure 10: Hopper-v3: Training performance of various models on MuJoCo
Hopper-v3. Curves are averages of three independent training runs.

(a) A2C

(b) A2C Consistent

(c) PPO

(d) PPO Consistent

Figure 11: Walker2d-v3: Training performance of various models on MuJoCo
Walker2d-v3. Curves are averages of three independent training runs.

(a) A2C

(b) A2C Consistent

(c) PPO

(d) PPO Consistent

Figure 12: Half-Cheetah-v3 Deterministic Evaluation with Dropout: Curves are
averages of three independent training runs.

(a) A2C

(b) A2C Consistent

(c) PPO

(d) PPO Consistent

Figure 13: Hopper-v3 Deterministic Evaluation with Dropout: Curves are
averages of three independent training runs.

(a) A2C

(b) A2C Consistent

(c) PPO

(d) PPO Consistent

Figure 14: Walker2d-v3 Deterministic Evaluation with Dropout: Curves are
averages of three independent training runs.

(a) A2C

(b) A2C Consistent

(c) PPO

(d) PPO Consistent

Figure 15: Half-Cheetah-v3 Deterministic Evaluation No-Dropout: Curves are
averages of three independent training runs.

(a) A2C

(b) A2C Consistent

(c) PPO

(d) PPO Consistent

Figure 16: Hopper-v3 Deterministic Evaluation No-Dropout: Curves are averages
of three independent training runs.

(a) A2C

(b) A2C Consistent

(c) PPO

(d) PPO Consistent

Figure 17: Walker2d-v3 Deterministic Evaluation No-Dropout: Curves are
averages of three independent training runs.

##  Appendix D Full Atari Results

Hyperparameters were tuned for Beamrider then deployed without further
modification on Breakout and Seaquest. The lower performance from PPO without
dropout versus PPO Consistent without dropout is due to the lack of target-KL
early stopping threshold for PPO. If this threshold is not disabled, PPO, due
to inconsistent dropout, fails to perform any policy updates and makes no
learning progress.

(a) A2C

(b) A2C Consistent

(c) PPO

(d) PPO Consistent

Figure 18: Beamrider: Training performance of various models on Atari
Beamrider. Curves are averages of three independent training runs.

(a) A2C

(b) A2C Consistent

(c) PPO

(d) PPO Consistent

Figure 19: Breakout: Training performance of various models on Atari Breakout.
Curves are averages of three independent training runs.

(a) A2C

(b) A2C Consistent

(c) PPO

(d) PPO Consistent

Figure 20: Seaquest: Training performance of various models on Atari Seaquest.
Curves are averages of three independent training runs. Poor performance from
all algorithms indicates hyperparameters which were tuned for Beamrider are
suboptimal for Seaquest.

(a) A2C

(b) A2C Consistent

(c) PPO

(d) PPO Consistent

Figure 21: Beamrider Deterministic Evaluation with Dropout: Curves are
averages of three independent training runs.

(a) A2C

(b) A2C Consistent

(c) PPO

(d) PPO Consistent

Figure 22: Breakout Deterministic Evaluation with Dropout: Curves are averages
of three independent training runs.

(a) A2C

(b) A2C Consistent

(c) PPO

(d) PPO Consistent

Figure 23: Seaquest Deterministic Evaluation with Dropout: Curves are averages
of three independent training runs.

(a) A2C

(b) A2C Consistent

(c) PPO

(d) PPO Consistent

Figure 24: Beamrider Deterministic Evaluation with No-Dropout: Curves are
averages of three independent training runs.

(a) A2C

(b) A2C Consistent

(c) PPO

(d) PPO Consistent

Figure 25: Breakout Deterministic Evaluation with No-Dropout: Curves are
averages of three independent training runs.

(a) A2C

(b) A2C Consistent

(c) PPO

(d) PPO Consistent

Figure 26: Seaquest Deterministic Evaluation with No-Dropout: Curves are
averages of three independent training runs.