# Gauge invariants of linearized gravity with a general background metric

Deepen Garg  Department of Astrophysical Sciences, Princeton University,
Princeton, New Jersey 08544, USA  I. Y. Dodin  Department of Astrophysical
Sciences, Princeton University, Princeton, New Jersey 08544, USA  Princeton
Plasma Physics Laboratory, Princeton, NJ 08543, USA

###### Abstract

A general method is proposed for identifying the gauge-invariant part of the
metric perturbation within linearized gravity, and the six independent gauge
invariants per se, for an arbitrary background metric. For the Minkowski
background, the operator that projects the metric perturbation on the
invariant subspace is proportional to the well-known dispersion operator of
linear gravitational waves in vacuum.

##  I Introduction

The perturbation approach has proven useful in studying various phenomena in
classical gravity, for example, gravitational waves (GWs). Within this
approach, the spacetime metric $\smash{\mathsf{g}_{\alpha\beta}}$ is split
into a background metric $\smash{g_{\alpha\beta}}=\mathcal{O}(1)$ and a
perturbation $\smash{h_{\alpha\beta}}=\mathcal{O}(a)$, where $a\ll 1$ is a
small parameter. A coordinate transformation $\smash{x^{\mu}\to
x^{\prime\mu}=x^{\mu}+\xi^{\mu}}$, with $\smash{\xi^{\mu}}=\mathcal{O}(a)$,
induces a metric transformation
$\smash{\mathsf{g}_{\alpha\beta}}\to\smash{\mathsf{g}^{\prime}_{\alpha\beta}}=\mathsf{g}_{\alpha\beta}-\smash{\mathrm{\text{\pounds}}_{\xi}\mathsf{g}_{\alpha\beta}}+\mathcal{O}(a^{2})$,
where $\smash{\mathrm{\text{\pounds}}_{\xi}}$ is the Lie derivative along the
vector field $\smash{\xi^{\mu}}$ book:carroll . Assuming linearized gravity,
where $\smash{\mathcal{O}(a^{2})}$ corrections are neglected and the
background is $a$-independent by definition, this implies
$\smash{g_{\alpha\beta}}\to\smash{g^{\prime}_{\alpha\beta}}=\smash{g_{\alpha\beta}}$
and
$\smash{h_{\alpha\beta}}\to\smash{h^{\prime}_{\alpha\beta}}=\smash{h_{\alpha\beta}}-\smash{\mathrm{\text{\pounds}}_{\xi}g_{\alpha\beta}}$.
If $\smash{h_{\alpha\beta}}$ is treated as a tensor field on the unperturbed
spacetime, so its indices are manipulated using $\smash{g_{\alpha\beta}}$ as
the metric, one also has

$\displaystyle h^{\alpha\beta}\to
h^{\prime\alpha\beta}=h^{\alpha\beta}+\mathrm{\text{\pounds}}_{\xi}g^{\alpha\beta}.$
(1)

The transformation (1) can be viewed as a gauge transformation and, by general
covariance, cannot have measurable effects. Thus, the physical, gauge-
invariant, part of $\smash{h^{\alpha\beta}}$ is defined only up to the Lie
derivative of $g^{\alpha\beta}$ along an arbitrary vector field, which is
encoded by four functions (in a four-dimensional spacetime). Because the
symmetric tensor $\smash{h^{\alpha\beta}}$ is encoded by ten functions, this
leaves room for six gauge-invariant degrees of freedom.

Identifying these degrees of freedom for a given metric perturbation
$\smash{h^{\alpha\beta}}$ is important for representing the linearized-gravity
equations in a gauge-invariant form. This problem has a well-known solution
for the Minkowski background ref:flanagan05 , and it has also been solved ad
hoc for the Friedmann–Lemaître–Robertson–Walker background ref:bardeen80 ;
ref:malik09 ; ref:malik13 ; ref:nakamura07c ; ref:bruni97 ; ref:luca20 .
However, less attention has been paid to general background metrics,
particularly those that emerge in problems involving GW–matter coupling
my:gwponder ; ref:baym17 ; ref:bamba18 ; ref:asenjo20 ; ref:barta18 ;
ref:chesters73 ; ref:asseo76 ; ref:macedo83 ; ref:flauger18 ; ref:servin01 ;
ref:moortgat03 ; ref:forsberg10a ; ref:isliker06 ; ref:duez05a ;
ref:mendonca02b . This leads to the question: how can one find the invariant
part of $\smash{h^{\alpha\beta}}$ for general $g_{\alpha\beta}$?

Here, we answer this question. We start by showing that any symmetric tensor
$\smash{h^{\alpha\beta}}$ can be uniquely decomposed as

$\displaystyle h^{\alpha\beta}=\smash{\widehat{\Pi}}^{\alpha\beta}_{\rm
inv}{}_{\gamma\delta}h^{\gamma\delta}+\smash{\widehat{\Pi}}^{\alpha\beta}_{\rm
g}{}_{\gamma\delta}h^{\gamma\delta},$ (2)

where the operators $\smash{\smash{\widehat{\Pi}}^{\alpha\beta}_{\rm
inv}{}_{\gamma\delta}}$ and $\smash{\smash{\widehat{\Pi}}^{\alpha\beta}_{\rm
g}{}_{\gamma\delta}}$ satisfy

$\displaystyle\smash{\widehat{\Pi}}^{\alpha\beta}_{\rm
inv}{}_{\gamma\delta}+\smash{\widehat{\Pi}}^{\alpha\beta}_{\rm
g}{}_{\gamma\delta}=\delta^{\alpha}_{(\gamma}\delta^{\beta}_{\delta)},$ (3a)
$\displaystyle\smash{\widehat{\Pi}}^{\alpha\beta}_{\rm
inv}{}_{\gamma\delta}\smash{\widehat{\Pi}}^{\gamma\delta}_{\rm
inv}{}_{\lambda\varepsilon}=\smash{\widehat{\Pi}}^{\alpha\beta}_{\rm
inv}{}_{\lambda\varepsilon},$ (3b)
$\displaystyle\smash{\widehat{\Pi}}^{\alpha\beta}_{\rm
g}{}_{\gamma\delta}\smash{\widehat{\Pi}}^{\gamma\delta}_{\rm
g}{}_{\lambda\varepsilon}=\smash{\widehat{\Pi}}^{\alpha\beta}_{\rm
g}{}_{\lambda\varepsilon},$ (3c)
$\displaystyle\smash{\widehat{\Pi}}^{\alpha\beta}_{\rm
inv}{}_{\gamma\delta}\smash{\widehat{\Pi}}^{\gamma\delta}_{\rm
g}{}_{\lambda\varepsilon}=\smash{\widehat{\Pi}}^{\alpha\beta}_{\rm
g}{}_{\gamma\delta}\smash{\widehat{\Pi}}^{\gamma\delta}_{\rm
inv}{}_{\lambda\varepsilon}=0,$ (3d)
$\displaystyle\smash{\widehat{\Pi}}^{\alpha\beta}_{\rm
inv}{}_{\gamma\delta}\mathrm{\text{\pounds}}_{u}g^{\gamma\delta}=0,$ (3e)
$\displaystyle\smash{\widehat{\Pi}}^{\alpha\beta}_{\rm
g}{}_{\gamma\delta}\mathrm{\text{\pounds}}_{u}g^{\gamma\delta}=\mathrm{\text{\pounds}}_{u}g^{\alpha\beta},$
(3f) $\displaystyle\smash{\widehat{\Pi}}^{\alpha\beta}_{{\rm inv},{\rm
g}}{}_{\gamma\delta}=\smash{\widehat{\Pi}}^{\alpha\beta}_{{\rm inv},{\rm
g}}{}_{\delta\gamma}=\smash{\widehat{\Pi}}^{\beta\alpha}_{{\rm inv},{\rm
g}}{}_{\gamma\delta}.$ (3g)

(Parentheses in indices denote symmetrization, as usual, and $u^{\mu}$ is any
vector field.) In Sec. II, we present a method for how to calculate the
operators $\smash{\smash{\widehat{\Pi}}^{\alpha\beta}_{\rm
inv}{}_{\gamma\delta}}$ and $\smash{\smash{\widehat{\Pi}}^{\alpha\beta}_{\rm
g}{}_{\gamma\delta}}$ for general $g_{\alpha\beta}$. We also show that the
gauge-invariant part of a metric perturbation $\smash{h^{\alpha\beta}}$ can be
calculated as $\smash{\smash{\widehat{\Pi}}^{\alpha\beta}_{\rm
inv}{}_{\gamma\delta}h^{\gamma\delta}}$, while
$\smash{\smash{\widehat{\Pi}}^{\alpha\beta}_{\rm
g}{}_{\gamma\delta}h^{\gamma\delta}}$ is the gauge-dependent part
representable as $\smash{\mathrm{\text{\pounds}}_{\zeta}g^{\alpha\beta}}$,
where $\zeta^{\mu}$ is a vector field linear in $\smash{h^{\alpha\beta}}$.
These results lead to a gauge-invariant formulation of linearized gravity with
any background metric, as will be shown in a follow-up paper. In Sec. III, we
illustrate the application of our results to the Minkowski background. We
derive the six gauge-invariant components of $\smash{h^{\alpha\beta}}$ and
show the agreement with the commonly known results. In addition, we show that
$\smash{\smash{\widehat{\Pi}}^{\alpha\beta}_{\rm inv}{}_{\gamma\delta}}$ is
proportional to the dispersion operator of linear GWs in Minkowski vacuum. In
Sec. IV, we summarize our main results. Auxiliary calculations are presented
in appendices.

##  II Basic theory

In this section, we present a method for how to calculate the operators
$\smash{\smash{\widehat{\Pi}}^{\alpha\beta}_{\rm inv}{}_{\gamma\delta}}$ and
$\smash{\smash{\widehat{\Pi}}^{\alpha\beta}_{\rm g}{}_{\gamma\delta}}$ for
general $g_{\alpha\beta}$. We assume the sign convention is as in Refs.
book:carroll ; book:misner77 , so

$\displaystyle[\nabla_{\beta},\nabla^{\alpha}]\xi^{\beta}=R^{\alpha}{}_{\beta}\xi^{\beta}$
(4)

for any vector field $\xi^{\alpha}$, where $\smash{R^{\alpha}{}_{\beta}}$ is
the Ricci tensor. Also as a reminder,

$\displaystyle\mathrm{\text{\pounds}}_{\xi}g^{\alpha\beta}=-\nabla^{\alpha}\xi^{\beta}-\nabla^{\beta}\xi^{\alpha}\equiv-2\nabla^{(\alpha}\xi^{\beta)}.$
(5)

###  II.1 Special case

Let us consider an auxiliary problem first, only to motivate introducing the
machinery that will be used in Sec. II.2. For a given symmetric tensor
$\smash{h^{\alpha\beta}}$, let us search for a divergence-free vector field
$\smash{u^{\alpha}}$ such that the tensor

$\displaystyle h^{\prime\alpha\beta}\doteq
h^{\alpha\beta}-\mathrm{\text{\pounds}}_{u}g^{\alpha\beta}$ (6)

(the symbol $\doteq$ denotes definitions) satisfies the Lorenz gauge; i.e.,

$\displaystyle\nabla_{\beta}h^{\prime\alpha\beta}=0.$ (7)

Because we assume

$\displaystyle\nabla_{\alpha}u^{\alpha}=0,$ (8)

Eqs. (5)–(7) lead to an equation for $u^{\alpha}$ that is similar to the
driven Maxwell’s equation for the Lorenz-gauge electromagnetic vector
potential in vacuum my:spinhall :

$\displaystyle\smash{\widehat{Q}}^{\alpha}{}_{\beta}u^{\beta}=\nabla_{\beta}h^{\alpha\beta},$
(9a)
$\displaystyle\smash{\widehat{Q}}^{\alpha}{}_{\beta}\doteq-\delta_{\beta}^{\alpha}\nabla_{\mu}\nabla^{\mu}-R^{\alpha}{}_{\beta},$
(9b)

where we also used Eq. (4). Equation (9a) has a solution

$\displaystyle
u^{\alpha}=\smash{\widehat{\Xi}}^{\alpha}{}_{(\gamma}\nabla_{\delta)}h^{\gamma\delta},$
(10)

where $\smash{\smash{\widehat{\Xi}}^{\alpha}{}_{\beta}}$ is the Green’s
operator of Eq. (9a). (Symmetrization with respect to the lower indices is
added for convenience and does not affect the result.) From now on, we assume
the adiabatic limit my:nonloc , when
$\smash{\smash{\widehat{Q}}^{\alpha}{}_{\beta}}$ is approximately invertible
for fields of interest. Then, one can express
$\smash{\smash{\widehat{\Xi}}^{\alpha}{}_{\beta}}$ as

$\displaystyle\smash{\widehat{\Xi}}^{\alpha}{}_{\beta}=(\smash{\widehat{Q}}^{-1})^{\alpha}{}_{\beta}.$
(11)

Let us assume, for now, that $\smash{h^{\gamma\delta}}$ is such that the
solution (10) does in fact satisfy the previously imposed constraint (8).
Then, Eq. (7) is satisfied by
$\smash{h^{\prime\alpha\beta}}=\smash{\smash{\widehat{\pi}}^{\alpha\beta}{}_{\gamma\delta}h^{\gamma\delta}}$,
where

$\displaystyle\smash{\widehat{\pi}}^{\alpha\beta}{}_{\gamma\delta}\doteq\delta^{\alpha}_{(\gamma}\delta^{\beta}_{\delta)}+2\nabla^{(\alpha}\smash{\widehat{\Xi}}^{\beta)}{}_{(\gamma}\nabla_{\delta)}.$
(12)

In combination with Eq. (6), these results yield that

$\displaystyle
h^{\alpha\beta}=\smash{\widehat{\pi}}^{\alpha\beta}{}_{\gamma\delta}h^{\gamma\delta}+\mathrm{\text{\pounds}}_{u}g^{\alpha\beta},$
(13a)
$\displaystyle\mathrm{\text{\pounds}}_{u}g^{\alpha\beta}=-2\nabla^{(\alpha}\smash{\widehat{\Xi}}^{\beta)}{}_{(\gamma}\nabla_{\delta)}h^{\gamma\delta},$
(13b)

and a direct calculation shows that (Appendix B)

$\displaystyle\smash{\widehat{\pi}}^{\alpha\beta}{}_{\gamma\delta}\mathrm{\text{\pounds}}_{u}g^{\gamma\delta}=0.$
(14)

Equation (14) is similar to Eq. (3e) and makes the decomposition (13) close to
Eq. (2) indeed, except it is constrained by Eq. (8). This can be taken as a
hint that $\smash{\smash{\widehat{\pi}}^{\alpha\beta}{}_{\gamma\delta}}$ is
close to the sought $\smash{\smash{\widehat{\Pi}}^{\alpha\beta}_{\rm
inv}{}_{\gamma\delta}}$. Hence, we approach the general case as follows.

###  II.2 General case

Let us consider application of
$\smash{\widehat{\pi}}^{\alpha\beta}{}_{\gamma\delta}$ to
$\smash{\mathrm{\text{\pounds}}_{u}g^{\alpha\beta}}$ with a general
$\smash{u^{\alpha}}$. A direct calculation shows that the result can be
expressed as (Appendix B)111Here and further,
$g_{\alpha\beta}\equiv\smash{\widehat{g}}_{\alpha\beta}$ and
$R^{\alpha}{}_{\beta}\equiv\smash{\widehat{R}}^{\alpha}{}_{\beta}$ serve as
multiplication operators, and the assumed notation is
$\smash{\smash{\widehat{A}}\smash{\widehat{B}}f=\smash{\widehat{A}}(\smash{\widehat{B}}f)}$
for any operators $\smash{\smash{\widehat{A}}}$ and
$\smash{\smash{\widehat{B}}}$ and function $f$ that they act upon. For
example,
$\nabla^{\mu}g_{\gamma\delta}\mathrm{\text{\pounds}}_{u}g^{\gamma\delta}\equiv\nabla^{\mu}[g_{\gamma\delta}(\mathrm{\text{\pounds}}_{u}g^{\gamma\delta})]$.

$\displaystyle\smash{\widehat{\pi}}^{\alpha\beta}{}_{\gamma\delta}\mathrm{\text{\pounds}}_{u}g^{\gamma\delta}=\nabla^{(\alpha}\smash{\widehat{\Xi}}^{\beta)}{}_{\mu}\nabla^{\mu}g_{\gamma\delta}\mathrm{\text{\pounds}}_{u}g^{\gamma\delta}.$
(15)

Hence, the operator

$\displaystyle\smash{\widehat{\Pi}}^{\alpha\beta}_{\rm
inv}{}_{\gamma\delta}\doteq\smash{\widehat{\pi}}^{\alpha\beta}{}_{\gamma\delta}-\nabla^{(\alpha}\smash{\widehat{\Xi}}^{\beta)}{}_{\mu}\nabla^{\mu}g_{\gamma\delta}$
(16)

automatically satisfies Eq. (3e). Let us substitute Eq. (12) and rewrite this
operator as follows:

$\displaystyle\smash{\widehat{\Pi}}^{\alpha\beta}_{\rm
inv}{}_{\gamma\delta}=\delta^{\alpha}_{(\gamma}\delta^{\beta}_{\delta)}-\smash{\widehat{\Pi}}^{\alpha\beta}_{\rm
g}{}_{\gamma\delta},$ (17a)
$\displaystyle\smash{\widehat{\Pi}}^{\alpha\beta}_{\rm
g}{}_{\gamma\delta}\doteq-2\nabla^{(\alpha}\smash{\widehat{\Xi}}^{\beta)}{}_{(\gamma}\nabla_{\delta)}+\nabla^{(\alpha}\smash{\widehat{\Xi}}^{\beta)}{}_{\mu}\nabla^{\mu}g_{\gamma\delta}.$
(17b)

This satisfies Eqs. (3a), (3f), and (3g). (The latter ensures that
$\smash{\smash{\widehat{\Pi}}^{\alpha\beta}_{{\rm inv},{\rm
g}}{}_{\gamma\delta}f^{\gamma\delta}}=0$ for all anti-symmetric
$\smash{f^{\gamma\delta}}$, which is convenient.) The property (3c) is proven
by direct calculation (Appendix C). Equation (3d) can be derived from Eqs.
(3a) and (3c), and the remaining property (3b) can then be obtained from Eqs.
(3a) and (3d).

Let us discuss how this result helps identify the invariant part of a metric
perturbation. First, notice that

$\displaystyle\smash{\widehat{\Pi}}^{\alpha\beta}_{\rm
g}{}_{\gamma\delta}h^{\gamma\delta}$
$\displaystyle=-2\nabla^{(\alpha}\smash{\widehat{\Xi}}^{\beta)}{}_{(\gamma}\nabla_{\delta)}h^{\gamma\delta}+\nabla^{(\alpha}\smash{\widehat{\Xi}}^{\beta)}{}_{\mu}\nabla^{\mu}g_{\gamma\delta}h^{\gamma\delta}$
$\displaystyle=-2\nabla^{(\alpha}\zeta^{\beta)}$
$\displaystyle=\mathrm{\text{\pounds}}_{\zeta}g^{\alpha\beta},$ (18)

where we introduced

$\displaystyle\zeta^{\beta}\doteq\smash{\widehat{\Xi}}^{\beta}{}_{(\gamma}\nabla_{\delta)}h^{\gamma\delta}-\frac{1}{2}\,\smash{\widehat{\Xi}}^{\beta}{}_{\mu}\nabla^{\mu}g_{\gamma\delta}h^{\gamma\delta}.$
(19)

Hence, Eq. (2) can be rewritten as

$\displaystyle h^{\alpha\beta}=\psi^{\alpha\beta}+\phi^{\alpha\beta},$ (20a)
$\displaystyle\psi^{\alpha\beta}\doteq\smash{\widehat{\Pi}}^{\alpha\beta}_{\rm
inv}{}_{\gamma\delta}h^{\gamma\delta},$ (20b)
$\displaystyle\phi^{\alpha\beta}\doteq\smash{\widehat{\Pi}}^{\alpha\beta}_{\rm
g}{}_{\gamma\delta}h^{\gamma\delta}=\mathrm{\text{\pounds}}_{\zeta}g^{\alpha\beta}.$
(20c)

Upon a gauge transformation (1), one obtains

$\displaystyle
h^{\prime\alpha\beta}=\psi^{\prime\alpha\beta}+\phi^{\prime\alpha\beta},$
(21a)
$\displaystyle\psi^{\prime\alpha\beta}=\smash{\widehat{\Pi}}^{\alpha\beta}_{\rm
inv}{}_{\gamma\delta}h^{\prime\gamma\delta}=\psi^{\alpha\beta}+\smash{\widehat{\Pi}}^{\alpha\beta}_{\rm
inv}{}_{\gamma\delta}\mathrm{\text{\pounds}}_{\xi}g^{\alpha\beta}=\psi^{\alpha\beta},$
(21b)
$\displaystyle\phi^{\prime\alpha\beta}=\smash{\widehat{\Pi}}^{\alpha\beta}_{\rm
g}{}_{\gamma\delta}h^{\prime\gamma\delta}=\phi^{\alpha\beta}+\mathrm{\text{\pounds}}_{\xi}g^{\alpha\beta}=\mathrm{\text{\pounds}}_{\zeta+\xi}g^{\alpha\beta},$
(21c)

where we used Eqs. (3d)–(3f). The function $\smash{\phi^{\prime\alpha\beta}}$
can be zeroed by choosing the gauge $\xi^{\mu}=-\zeta^{\mu}$. This means that
$\smash{\phi^{\alpha\beta}}$, which is encoded by the four functions
$\zeta^{\mu}$, does not contain gauge-independent information. Hence, any
solution that has nonzero $\smash{\phi^{\alpha\beta}}$ and zero
$\smash{\psi^{\alpha\beta}}$ can be classified as a coordinate artifact. In
contrast, $\smash{\psi^{\alpha\beta}}$ is gauge-invariant by Eq. (21b). By the
argument presented in Sec. I, it is encoded by six independent functions, or
gauge-invariant degrees of freedom. Also note that
$\smash{\psi^{\alpha\beta}}$ does not necessarily satisfy the Lorenz-gauge
condition $\smash{\nabla_{\beta}\psi^{\alpha\beta}=0}$.

###  II.3 Gauge invariants

Now let us discuss how to extract the six independent functions from the
sixteen gauge-invariant functions $\psi^{\alpha\beta}$. To do so, let us
consider $h^{\alpha\beta}$ as a 16-dimensional (16-D) field $h^{a}$, or
${\boldsymbol{h}}$ in the index-free notation, of the form

$\displaystyle{\boldsymbol{h}}=(h^{00},h^{01},h^{02},h^{03},h^{10},\ldots,h^{32},h^{33})^{\intercal},$
(22)

where ⊺ denotes transpose. In other words,

$\displaystyle h^{a}=h^{\alpha\beta},$ $\displaystyle\quad
h_{b}=h_{\gamma\delta},$ (23) $\displaystyle\\{\alpha,\beta\\}=\iota(a),$
$\displaystyle\quad\\{\gamma,\delta\\}=\iota(b),$ (24)

where the index function $\iota$ is defined via

$\displaystyle\iota(a)\doteq\big{\\{}1+\lfloor(a-1)/4\rfloor,1+(a-1)\,\text{mod}\,4\big{\\}}.$
(25)

(Here and further, Latin indices from the beginning of the alphabet range from
1 to 16.) Let us define $\mathscr{H}_{1}$ as a Hilbert space of one-component
functions on the background spacetime with the usual inner product
$\braket{\cdot\,,\cdot}_{1}$. Then, the 16-D fields (22) can be considered as
vectors in the Hilbert space $\mathscr{H}_{16}$ that is the tensor product of
16 copies of $\mathscr{H}_{1}$, with the inner product

$\displaystyle\braket{{\boldsymbol{\xi}},{\boldsymbol{\varphi}}}=\int\mathrm{d}^{4}x\sqrt{-g}\,\xi_{a}^{*}\,\varphi^{a}=\sum_{a=1}^{16}\braket{\xi_{a},\varphi^{a}}_{1},$
(26)

where $g\doteq\det g_{\alpha\beta}$. (Unlike in the rest of the paper,
summation is shown explicitly here in order to emphasize the difference
between $\braket{\cdot\,,\cdot}$ and $\braket{\cdot\,,\cdot}_{1}$.) Then,
$\smash{\smash{\widehat{\Pi}}_{\rm inv}^{\alpha\beta}{}_{\gamma\delta}}$
induces an operator $\smash{\smash{\widehat{\Pi}}^{a}{}_{b}}$ on
$\mathscr{H}_{16}$ defined via

$\displaystyle\smash{\widehat{\Pi}}^{a}{}_{b}h^{b}\doteq\smash{\widehat{\Pi}}_{\rm
inv}^{\alpha\beta}{}_{\gamma\delta}h^{\gamma\delta},$ (27)

where we again assumed the notation as in Eq. (24). From Eqs. (3), one finds
that

$\displaystyle\smash{\widehat{\Pi}}^{a}{}_{b}\smash{\widehat{\Pi}}^{b}{}_{c}=\smash{\widehat{\Pi}}^{a}{}_{c},$
(28a)
$\displaystyle\smash{\widehat{\Pi}}^{a}{}_{b}\mathrm{\text{\pounds}}_{u}g^{b}=0.$
(28b)

Equation (28a), which in the index-free notation can be written as
$\smash{\widehat{\boldsymbol{\Pi}}}^{2}=\smash{\widehat{\boldsymbol{\Pi}}}$,
means that $\smash{\widehat{\boldsymbol{\Pi}}}$ is a projector. (Note that
$\smash{\widehat{\boldsymbol{\Pi}}}^{\dagger}\neq\smash{\widehat{\boldsymbol{\Pi}}}$,
so the projector is not orthogonal but oblique.) Hence, each eigenvalue of
$\smash{\widehat{\boldsymbol{\Pi}}}$ is either zero or unity and
$\smash{\widehat{\boldsymbol{\Pi}}}$ is diagonalizable. This means that
$\smash{\widehat{\boldsymbol{\Pi}}}$ can be represented as

$\displaystyle\smash{\widehat{\boldsymbol{\Pi}}}=\smash{\widehat{\boldsymbol{V}}}\smash{\widehat{\boldsymbol{J}}}\smash{\widehat{\boldsymbol{V}}}^{-1},$
(29)

where $\smash{\smash{\widehat{\boldsymbol{V}}}}$ is a diagonalizing
transformation and the operator $\smash{\widehat{\boldsymbol{J}}}$ is such
that $\smash{\widehat{\boldsymbol{J}}}{\boldsymbol{h}}$ equals either zero or
${\boldsymbol{h}}$ for any ${\boldsymbol{h}}$. But each linear operator in
$\mathscr{H}_{16}$ is a $16\times 16$ matrix of operators on
$\mathscr{H}_{1}$. Then, $\smash{\widehat{\boldsymbol{J}}}$ must be
represented by a constant matrix ${{\boldsymbol{J}}}$ of the form

$\displaystyle{{\boldsymbol{J}}}=\text{diag}\,\big{\\{}\underbrace{1,1,\ldots,1}_{n},\underbrace{0,0,\ldots,0,0}_{16-n}\big{\\}},$
(30)

where, for clarity, we have ordered the basis such that the nonzero
eigenvalues are grouped together and have indices $1,\ldots,n$.

The gauge-invariant part of ${\boldsymbol{h}}$, which is given by Eq. (20b),
can now be expressed as
${\boldsymbol{\psi}}=\smash{\widehat{\boldsymbol{\Pi}}}{\boldsymbol{h}}$.
Using Eq. (29), one can also rewrite this as

$\displaystyle\smash{\widehat{\boldsymbol{P}}}{\boldsymbol{\psi}}={\boldsymbol{\Psi}},\quad{\boldsymbol{\Psi}}={{\boldsymbol{J}}}\smash{\widehat{\boldsymbol{P}}}{\boldsymbol{h}},\quad\smash{\widehat{\boldsymbol{P}}}\doteq\smash{\widehat{\boldsymbol{V}}}^{-1}.$
(31)

Because ${\boldsymbol{h}}$ is an arbitrary vector field parameterized by 16
functions and $\smash{\widehat{\boldsymbol{P}}}$ is invertible, the field
$\smash{\widehat{\boldsymbol{P}}}{\boldsymbol{h}}$ is also parameterized by 16
functions. Then, ${\boldsymbol{\Psi}}$ is parameterized by $n$ functions. But
we know that ${\boldsymbol{\psi}}$ is parameterized by 6 functions (Sec. I),
and thus so is ${\boldsymbol{\Psi}}$. Then, $n=6$, and the nonzero elements of
${\boldsymbol{\Psi}}$ are the sought invariants.

In summary, to find the gauge invariants, one needs to find the diagonalizing
transformation $\smash{\smash{\widehat{V}}^{a}{}_{b}}$ that brings
$\smash{\smash{\widehat{\Pi}}^{a}{}_{b}}$ to the form given by Eqs. (29) and
(30). Then, the invariants can be found as

$\displaystyle\Psi^{s}=J^{s}{}_{b}(\smash{\widehat{V}}^{-1})^{b}{}_{c}h^{c},\quad
s=1,2,\ldots,6.$ (32)

##  III Example: Minkowski background

###  III.1 Gauge invariants

In the flat-space limit, when $R^{\alpha}{}_{\beta}\to 0$, one has
$\smash{\smash{\widehat{\Xi}}^{\alpha}{}_{\beta}\to-\delta^{\alpha}_{\beta}\nabla^{-2}}$.
Below we assume the Minkowski background metric, in which case,
$\smash{\smash{\widehat{\Xi}}^{\alpha}{}_{\beta}}$ is further simplified to

$\displaystyle\smash{\widehat{\Xi}}^{\alpha}{}_{\beta}\to-\delta^{\alpha}_{\beta}\partial^{-2}.$
(33)

Here, $\smash{\partial^{-2}}$ is the operator inverse to
$\smash{\partial^{2}\doteq\partial_{\mu}\partial^{\mu}}$; i.e.,
$\smash{\varphi^{\alpha}}=\smash{\partial^{-2}q^{\alpha}}$ is the solution of
$\smash{\partial^{2}\varphi^{\alpha}=q^{\alpha}}$ (Appendix A). Formally,
$\smash{\partial^{-2}}$ is singular on free vacuum GWs, but the vacuum case
can still be considered as a limit (Sec. III.2).

Using Eq. (33), one can rewrite Eqs. (17a) as

$\displaystyle\smash{\widehat{\Pi}}^{\alpha\beta}_{\rm
inv}{}_{\gamma\delta}=\delta^{\alpha}_{(\gamma}\delta^{\beta}_{\delta)}-2\,\partial^{-2}\partial^{(\alpha}\delta^{\beta)}_{(\gamma}\partial^{\phantom{\beta)}}_{\delta)}+\partial^{-2}\partial^{\alpha}\partial^{\beta}g_{\gamma\delta}.$
(34)

Let us consider this operator in the Fourier representation, in which case it
becomes a local matrix function of the wavevector $k_{\mu}$; namely,
$\smash{\smash{\widehat{\Pi}}^{\alpha\beta}_{\rm
inv}{}_{\gamma\delta}}=\Pi^{\alpha\beta}_{\rm inv}{}_{\gamma\delta}$,

$\displaystyle\Pi^{\alpha\beta}_{\rm
inv}{}_{\gamma\delta}=\delta^{\alpha}_{(\gamma}\delta^{\beta}_{\delta)}-\frac{2k^{(\alpha}_{\phantom{\beta)}}\delta^{\beta)}_{(\gamma}k^{\phantom{\beta)}}_{\delta)}}{k^{2}}+g_{\gamma\delta}\,\frac{k^{\alpha}k^{\beta}}{k^{2}}.$
(35)

Using that $\nabla_{\mu}\to\partial_{\mu}\to\mathrm{i}k_{\mu}$ in the Fourier
representation [and in particular,
$\mathrm{\text{\pounds}}_{u}g^{\alpha\beta}=-2\mathrm{i}k^{(\alpha}u^{\beta)}$],
the properties (3) are easily verified. One also finds by direct calculation
foot:math  that, as expected from Eqs. (29) and (30),

$\displaystyle\text{rank}\,{{\boldsymbol{\Pi}}}=6.$ (36)

The invariant part of the metric perturbation (20b) is now given by
$\smash{\psi^{\alpha\beta}}=\smash{\Pi^{\alpha\beta}_{\rm
inv}{}_{\gamma\delta}h^{\gamma\delta}}$, or explicitly,

$\displaystyle\psi^{\alpha\beta}=h^{\alpha\beta}-\frac{k^{\alpha}k_{\mu}}{k^{2}}h^{\mu\beta}-\frac{k^{\beta}k_{\mu}}{k^{2}}h^{\alpha\mu}+\frac{k^{\alpha}k^{\beta}}{k^{2}}h,$
(37)

where $h\doteq\smash{\text{tr}\,h^{\alpha\beta}}$. Without loss of generality,
let us assume coordinates such that

$\displaystyle k^{\alpha}=(\omega,0,0,\mathsf{k}),$ (38)

where $\mathsf{k}$ is the spatial wavenumber. Using this, the fact that
$k^{2}=\mathsf{k}^{2}-\omega^{2}$, and also Eq. (24), the 16-D vector
${\boldsymbol{\psi}}$ is found to be:

$\displaystyle{\boldsymbol{\psi}}=\frac{1}{k^{2}}\begin{pmatrix}{h^{00}\mathsf{k}^{2}-2h^{03}\omega\mathsf{k}+\omega^{2}(h^{11}+h^{22}+h^{33})}\vskip
3.0pt plus 1.0pt minus 1.0pt\\\
{h^{01}\mathsf{k}^{2}-h^{13}\omega\mathsf{k}}\vskip 3.0pt plus 1.0pt minus
1.0pt\\\ {h^{02}\mathsf{k}^{2}-h^{23}\omega\mathsf{k}}\vskip 3.0pt plus 1.0pt
minus 1.0pt\\\ {(h^{11}+h^{22})\mathsf{k}\omega}\vskip 3.0pt plus 1.0pt minus
1.0pt\\\ {h^{01}\mathsf{k}^{2}-h^{13}\omega\mathsf{k}}\vskip 3.0pt plus 1.0pt
minus 1.0pt\\\ h^{11}({\mathsf{k}^{2}-\omega^{2}})\vskip 3.0pt plus 1.0pt
minus 1.0pt\\\ h^{12}({\mathsf{k}^{2}-\omega^{2}})\vskip 3.0pt plus 1.0pt
minus 1.0pt\\\ {h^{01}\omega\mathsf{k}-h^{13}\omega^{2}}\vskip 3.0pt plus
1.0pt minus 1.0pt\\\ {h^{02}\mathsf{k}^{2}-h^{23}\omega\mathsf{k}}\vskip 3.0pt
plus 1.0pt minus 1.0pt\\\ h^{12}({\mathsf{k}^{2}-\omega^{2}})\vskip 3.0pt plus
1.0pt minus 1.0pt\\\ h^{22}({\mathsf{k}^{2}-\omega^{2}})\vskip 3.0pt plus
1.0pt minus 1.0pt\\\ {h^{02}\omega\mathsf{k}-h^{23}\omega^{2}}\vskip 3.0pt
plus 1.0pt minus 1.0pt\\\ {(h^{11}+h^{22})\mathsf{k}\omega}\vskip 3.0pt plus
1.0pt minus 1.0pt\\\ {h^{01}\omega\mathsf{k}-h^{13}\omega^{2}}\vskip 3.0pt
plus 1.0pt minus 1.0pt\\\ {h^{02}\omega\mathsf{k}-h^{23}\omega^{2}}\vskip
3.0pt plus 1.0pt minus 1.0pt\\\
\mathsf{k}^{2}(-h^{00}+h^{11}+h^{22})+2h^{03}\omega\mathsf{k}-h^{33}\omega^{2}\end{pmatrix}.$

In order to extract the six gauge invariants from this ${\boldsymbol{\psi}}$,
notice that the operator (27) is represented by a local function of $k_{\mu}$,
$\smash{\widehat{\boldsymbol{\Pi}}}={{\boldsymbol{\Pi}}}$, and thus so is the
diagonalizing transformation (29). Specifically,
$\smash{\smash{\widehat{\boldsymbol{V}}}}={{\boldsymbol{V}}}$, and the columns
of the matrix ${{\boldsymbol{V}}}$ are just the eigenvectors of
${{\boldsymbol{\Pi}}}$:

$\displaystyle{\boldsymbol{V}}=({\boldsymbol{v}}_{1}\kern
5.0pt{\boldsymbol{v}}_{2}\kern 5.0pt\ldots\kern
5.0pt{\boldsymbol{v}}_{16}),\quad{{\boldsymbol{\Pi}}}{\boldsymbol{v}}_{a}=\lambda_{a}{\boldsymbol{v}}_{a},$
(39)

where $\lambda_{a}\in\\{0,1\\}$. The calculation of these eigenvectors and of
the matrix $\smash{{{\boldsymbol{V}}}^{-1}}$ can be automated foot:math , and
the six gauge invariants (32) are readily found to be

$\displaystyle{\boldsymbol{\Psi}}=\begin{pmatrix}\displaystyle\frac{\mathsf{k}^{2}(-h^{00}+h^{11}+h^{22})+2\omega\mathsf{k}h^{03}-\omega^{2}h^{33}}{\mathsf{k}^{2}-\omega^{2}}\\\\[10.0pt]
\displaystyle\frac{\omega\mathsf{k}h^{01}-\omega^{2}h^{13}}{\mathsf{k}^{2}-\omega^{2}}\\\\[10.0pt]
\displaystyle\frac{\omega\mathsf{k}h^{02}-\omega^{2}h^{23}}{\mathsf{k}^{2}-\omega^{2}}\\\\[10.0pt]
\displaystyle\frac{\omega\mathsf{k}(h^{11}+h^{22})}{\mathsf{k}^{2}-\omega^{2}}\\\\[10.0pt]
h^{22}\\\\[5.0pt] \displaystyle h^{12}\end{pmatrix}.$ (40)

The coordinate representation of these invariants is found by taking the
inverse Fourier transform of Eq. (40).

Our result is in agreement with Eqs. (2.45)–(2.47) in Ref. ref:flanagan05
(which operates with $h_{\alpha\beta}$ instead of our $h^{\alpha\beta}$). This
is seen from the fact that any linear combinations of our $\Psi^{s}$ are gauge
invariants too. In other words, instead of $\smash{\Psi^{s}}$, one can
introduce the invariants as $\smash{\bar{\Psi}^{s}}$ given by

$\displaystyle\bar{\Psi}^{s}\doteq C^{s}{}_{r}\Psi^{r},\quad
r,s=1,2,\ldots,6,$ (41)

or $\bar{{\boldsymbol{\Psi}}}={\boldsymbol{C}}{\boldsymbol{\Psi}}$ in the
index-free representation, where ${\boldsymbol{C}}$ is an arbitrary matrix
that may depend on $k_{\mu}$. This is particularly convenient at
$\smash{k^{2}}\equiv\smash{\mathsf{k}^{2}-\omega^{2}}\to 0$, when
${\boldsymbol{\Psi}}$ becomes singular. Specifically, by choosing

$\displaystyle{\boldsymbol{C}}=\text{diag}\,\big{\\{}k^{2},k^{2},k^{2},k^{2},1,1\big{\\}},$
(42)

we obtain invariants that are well-behaved at all $k_{\mu}$:

$\displaystyle\bar{{\boldsymbol{\Psi}}}=\begin{pmatrix}\displaystyle\mathsf{k}^{2}(-h^{00}+h^{11}+h^{22})+2\omega\mathsf{k}h^{03}-\omega^{2}h^{33}\vskip
3.0pt plus 1.0pt minus 1.0pt\\\
\displaystyle\omega\mathsf{k}h^{01}-\omega^{2}h^{13}\vskip 3.0pt plus 1.0pt
minus 1.0pt\\\ \displaystyle\omega\mathsf{k}h^{02}-\omega^{2}h^{23}\vskip
3.0pt plus 1.0pt minus 1.0pt\\\
\displaystyle\omega\mathsf{k}(h^{11}+h^{22})\vskip 3.0pt plus 1.0pt minus
1.0pt\\\ h^{22}\vskip 3.0pt plus 1.0pt minus 1.0pt\\\ \displaystyle
h^{12}\end{pmatrix}.$ (43)

Let us also address why the original vectors ${\boldsymbol{\psi}}$ and
${\boldsymbol{\Psi}}$ are singular at $\smash{k^{2}}\to 0$. In this limit, the
vectors ${\boldsymbol{v}}_{a}$ [Eq. (39)] are well-behaved, and thus so is the
matrix ${{\boldsymbol{V}}}$. However, they cease to be linearly independent at
$k^{2}=0$, so $\smash{{{\boldsymbol{V}}}^{-1}}$ becomes singular, and as a
result, ${\boldsymbol{\Pi}}$ becomes singular too. This means that no finite
invariant projection of a generic $\smash{h^{\alpha\beta}}$ can be defined in
the Fourier space at $k^{2}=0$. The corresponding gauge-dependent part
$\phi^{\alpha\beta}$ becomes singular as well in this limit, as seen from Eqs.
(19) and (20c), where $\smash{\smash{\widehat{\Xi}}^{\alpha}{}_{\beta}}$
becomes singular (Appendix A).222This is the same effect as the unlimited
growth, at $x^{\mu}\to\infty$, of the gauge field that brings a generic
$\smash{h^{\alpha\beta}}$ to the Lorenz gauge. See Appendix A in conjunction
with Eq. (10), which is commonly known for the Minkowski background
foot:schutz . Still, our general formulation correctly predicts the invariants
(43) at zero $k^{2}$, and these invariants can be related to vacuum GWs as
discussed in the next section.

###  III.2 Free GWs in the Minkowski space

It is instructive to compare the key operator in our formulation,
$\smash{\smash{\widehat{\Pi}}^{\alpha\beta}_{\rm inv}{}_{\gamma\delta}}$, with
the operator that governs linear GWs in vacuum. By comparing Eq. (34) with,
for example, Eqs. (5.4) and (2.7) in Ref. ref:isaacson68a , one finds that the
equation for vacuum GWs in the Minkowski spacetime can be expressed as

$\displaystyle\smash{\widehat{D}}^{\alpha\beta}{}_{\gamma\delta}h^{\gamma\delta}=0,\quad\smash{\widehat{D}}^{\alpha\beta}{}_{\gamma\delta}=\partial^{2}\smash{\widehat{\Pi}}^{\alpha\beta}_{\rm
inv}{}_{\gamma\delta}.$ (44)

In other words, in the special case of the Minkowski spacetime, the dispersion
operator $\smash{\smash{\widehat{D}}^{\alpha\beta}{}_{\gamma\delta}}$ of
vacuum GWs is exactly $\partial^{2}$ times the operator that projects a metric
perturbation on the invariant subspace.

For completeness, let us also briefly discuss monochromatic waves.333Cf. a
similar discussion in Ref. ref:maccallum73 , except their Eq. (3.6) describes
the _trace-reversed_ metric perturbation. In this case, Eq. (44) becomes

$\displaystyle k^{2}\,\Pi^{\alpha\beta}_{\rm
inv}{}_{\gamma\delta}\,h^{\gamma\delta}=0,$ (45)

where the matrix $\smash{k^{2}\,\Pi^{\alpha\beta}_{\rm inv}}$ is well-behaved
for all $k_{\mu}$. Equation (45) can be written as the following six of
equations, which determine the six gauge invariants (43):

$\displaystyle{\mathsf{k}^{2}h^{00}+\omega(-2\mathsf{k}h^{03}+\omega
h^{33})}=0,$ (46a)
$\displaystyle{\mathsf{k}^{2}h^{01}-\omega\mathsf{k}h^{13}}=0,$ (46b)
$\displaystyle{\mathsf{k}^{2}h^{02}-\omega\mathsf{k}h^{23}}=0,$ (46c)
$\displaystyle\mathsf{k}\omega{(h^{11}+h^{22})}=0,$ (46d) $\displaystyle
k^{2}(h^{11}-h^{22})=0,$ (46e) $\displaystyle k^{2}h^{12}=0.$ (46f)

For $k^{2}\neq 0$, Eqs. (46) indicate that all the six invariants (43) are
zero, so only coordinate waves are possible in this case. For $k^{2}=0$, Eqs.
(46a)–(46d) yield

$\displaystyle\bar{\Psi}^{1}=\bar{\Psi}^{2}=\bar{\Psi}^{3}=\bar{\Psi}^{4}=0,$
(47)

and in particular, $h^{11}+h^{22}=0$. However, Eqs. (46e) and (46f) are
satisfied identically at $k^{2}=0$, so the other two invariants,

$\displaystyle\bar{\Psi}^{5}=h^{22}=-h^{11},\quad\bar{\Psi}^{6}=h^{12}=h^{21},$
(48)

can be arbitrary, in agreement with known results ref:flanagan05 . In
particular, these are the two invariants that determine the commonly known
transverse-traceless polarization of a GW in the Lorenz gauge:

$\displaystyle h^{\alpha\beta}=\left(\begin{array}[]{cccc}0&0&0&0\\\
0&h_{+}&h_{\times}&0\\\ 0&h_{\times}&-h_{+}&0\\\ 0&0&0&0\end{array}\right).$
(53)

Specifically,

$\displaystyle h_{+}\doteq(h^{11}-h^{22})/2=-\bar{\Psi}^{5},$ (54a)
$\displaystyle h_{\times}\doteq h^{12}=h^{21}=\bar{\Psi}^{6}.$ (54b)

##  IV Conclusions

In summary, we propose a method for identifying the gauge-invariant part
$\smash{\psi^{\alpha\beta}}$ of the metric perturbation
$\smash{h^{\alpha\beta}}$ within linearized gravity for an arbitrary
background metric $\smash{g_{\alpha\beta}}$. Specifically, we show that
$\smash{\psi^{\alpha\beta}}=\smash{\smash{\widehat{\Pi}}^{\alpha\beta}_{\rm
inv}{}_{\gamma\delta}h^{\gamma\delta}}$, where
$\smash{\smash{\widehat{\Pi}}^{\alpha\beta}_{\rm inv}{}_{\gamma\delta}}$ is a
linear operator given by Eq. (17a). This result leads to a gauge-invariant
formulation of linearized gravity with any background metric, as will be shown
in a follow-up paper. The six independent functions from the sixteen gauge-
invariant functions $\psi^{\alpha\beta}$ can be found using Eq. (32). We also
show that for the Minkowski background,
$\smash{\smash{\widehat{\Pi}}^{\alpha\beta}_{\rm inv}{}_{\gamma\delta}}$ is
proportional to the well-known dispersion operator of linear GWs in vacuum
[Eq. (44)]. Also, our general formulation systematically yields the known
gauge invariants for the Minkowski background.

This material is based upon the work supported by National Science Foundation
under the grant No. PHY 1903130.

##  Appendix A Asymptotic representation of
$\boldsymbol{\smash{\widehat{\Xi}}^{\alpha}{}_{\beta}}$

The operator $\smash{\smash{\widehat{\Xi}}^{\alpha}{}_{\beta}}$ defined in Eq.
(11) can be written in the index-free representation as

$\displaystyle\smash{\widehat{\boldsymbol{\Xi}}}=-(\nabla^{2}+\smash{\widehat{\boldsymbol{R}}})^{-1},$
(55)

where $\nabla^{2}\doteq\nabla_{\mu}\nabla^{\mu}$,
$\smash{\widehat{\boldsymbol{R}}}$ is the operator whose coordinate
representation is the Ricci tensor $\smash{R^{\alpha}{}_{\beta}}$, and -1
denotes the operator inverse. In order for this inverse to exist
(approximately), we assume the adiabatic limit. Specifically, we assume that
the characteristic GW wavelength $\lambda$ is much smaller than the
characteristic radius $L$ of the spacetime curvature, i.e., when
$\epsilon\doteq\lambda/L\ll 1$. Assuming the ordering $\lambda=\mathcal{O}(1)$
and $L=\mathcal{O}(\epsilon^{-1})$, one has $\nabla^{2}=\mathcal{O}(1)$ and
$\smash{\widehat{\boldsymbol{R}}}=\mathcal{O}(\epsilon^{2})$. Then,

$\displaystyle\smash{\widehat{\boldsymbol{\Xi}}}=-\nabla^{-2}+\nabla^{-2}\smash{\widehat{\boldsymbol{R}}}\,\nabla^{-2}+\mathcal{O}(\epsilon^{4}),$
(56)

where $\nabla^{-2}$ is the inverse of $\smash{\nabla^{2}}$; i.e.,
$\smash{\varphi^{\alpha}}=\smash{\nabla^{-2}q^{\alpha}}$ is defined as the
solution of $\smash{\nabla^{2}\varphi^{\alpha}=q^{\alpha}}$.

Because the operators in Eqs. (55) and (56) are intended to act specifically
on vector fields, one can also write them explicitly. For example, in normal
coordinates, one has (Appendix D)

$\displaystyle\nabla^{2}=\partial^{2}-\frac{\smash{\widehat{\boldsymbol{R}}}}{3},$
(57)

and the corresponding inverse is

$\displaystyle\nabla^{-2}=\partial^{-2}+\frac{1}{3}\,\partial^{-2}\smash{\widehat{\boldsymbol{R}}}\,\partial^{-2}+\mathcal{O}(\epsilon^{4}),$
(58)

so Eq. (56) leads to

$\displaystyle\smash{\widehat{\boldsymbol{\Xi}}}=-\partial^{-2}+\frac{2}{3}\,\partial^{-2}\smash{\widehat{\boldsymbol{R}}}\,\partial^{-2}+\mathcal{O}(\epsilon^{4}).$
(59)

The operator $\partial^{-2}$ that enters here is understood as the Green’s
operator of the equation

$\displaystyle\partial^{2}\varphi^{\alpha}=q^{\alpha}.$ (60)

(This is the same equation that emerges in the well-known linear gravity in
the Minkowski background foot:schutz ; see also Eq. (10).) Suppose that the
right-hand side of Eq. (60) is quasimonochromatic, i.e.,
$\smash{q^{\alpha}=Q^{\alpha}\exp[\mathrm{i}\theta(x^{\mu})]}$ with
$\partial_{\beta}Q^{\alpha}=\mathcal{O}(\epsilon)$ and
$\partial_{\beta}k_{\alpha}=\mathcal{O}(\epsilon)$, where
$k_{\alpha}\doteq\partial_{\alpha}\theta$ is the local wavevector. Then,

$\displaystyle\partial^{-2}=(k_{\mu}k^{\mu})^{-1}+\smash{\widehat{\Delta}},$
(61)

where $\smash{\widehat{\Delta}}=\mathcal{O}(\epsilon)$ is a differential
operator to act on the envelope $\smash{Q^{\alpha}}$. If $k^{2}\doteq
k_{\mu}k^{\mu}$ approaches zero, as would be the case for GWs in the Minkowski
vacuum, then $\varphi^{\alpha}$ grows indefinitely at $x^{\mu}\to\infty$. This
is due to the fact that at $k^{2}\to 0$, $q^{\alpha}$ acts as a resonant
driving force for $\varphi^{\alpha}$. No quasimonochromatic solution is
possible in this case, and $\varphi^{\alpha}$ necessarily diverges at
infinity. In particular, this means that even if the Fourier spectrum of
$q^{\alpha}$ is analytic but includes harmonics with $k^{2}=0$, the Fourier
spectrum of the corresponding $\varphi^{\alpha}$ is singular.

This indicates that the case $k^{2}=0$ cannot be treated within the adiabatic
approximation that we assume in this paper. However, it still can be
considered as a limit, as discussed in Sec. III. Also, no such issues arise in
problems that involve GW–matter coupling, because then $k^{2}\neq 0$. In this
case, the term $\smash{\widehat{\Delta}}$ in Eq. (61) can also be found
explicitly using asymptotic methods of the Weyl calculus my:quasiop1 ;
ref:mcdonald88 . Because an explicit formula for $\smash{\widehat{\Delta}}$ is
not needed for our purposes, and because its derivation would involve
machinery that us far beyond the scope of our paper, such a derivation is not
presented here.

##  Appendix B Derivation of Eqs. (14) and (15)

Using Eq. (12) for
$\smash{\smash{\widehat{\pi}}^{\alpha\beta}{}_{\gamma\delta}}$ and Eq. (5) for
$\smash{\mathrm{\text{\pounds}}_{u}g^{\gamma\delta}}$, one obtains

$\displaystyle\smash{\widehat{\pi}}^{\alpha\beta}{}_{\gamma\delta}\mathrm{\text{\pounds}}_{u}g^{\gamma\delta}$
$\displaystyle=-\big{(}\delta_{\gamma}^{\alpha}\delta_{\delta}^{\beta}+\nabla^{\alpha}{\smash{\widehat{\Xi}}^{\beta}}{}_{\gamma}\nabla_{\delta}$
$\displaystyle\qquad+\nabla^{\beta}{\smash{\widehat{\Xi}}^{\alpha}}{}_{\gamma}\nabla_{\delta}\big{)}\left(\nabla^{\gamma}u^{\delta}+\nabla^{\delta}u^{\gamma}\right).$
(62)

Then using Eq. (4) in the above equation yields

$\displaystyle\smash{\widehat{\pi}}^{\alpha\beta}{}_{\gamma\delta}\mathrm{\text{\pounds}}_{u}g^{\gamma\delta}$
$\displaystyle=-2\nabla^{(\alpha}u^{\beta)}-2\nabla^{(\alpha}{\smash{\widehat{\Xi}}^{\beta)}}{}_{\gamma}\left[\nabla_{\delta},\nabla^{\gamma}\right]u^{\delta}$
$\displaystyle\qquad-2\nabla^{(\alpha}{\smash{\widehat{\Xi}}^{\beta)}}{}_{\gamma}\nabla^{\gamma}\nabla_{\delta}u^{\delta}-\nabla^{(\alpha}{\smash{\widehat{\Xi}}^{\beta)}}{}_{\gamma}\nabla^{2}u^{\gamma}$
$\displaystyle=-2\nabla^{(\alpha}u^{\beta)}-2\nabla^{(\alpha}{\smash{\widehat{\Xi}}^{\beta)}}{}_{\gamma}\left(\delta_{\delta}^{\gamma}\nabla^{2}+{R^{\gamma}}_{\delta}\right)u^{\delta}$
$\displaystyle\qquad-2\nabla^{(\alpha}{\smash{\widehat{\Xi}}^{\beta)}}{}_{\gamma}\nabla^{\gamma}\nabla_{\delta}u^{\delta}.$
(63)

Using Eq. (9b) for $\smash{\smash{\widehat{Q}}^{\alpha}{}_{\beta}}$ in
combination with Eq. (11), one obtains

$\displaystyle\smash{\widehat{\pi}}^{\alpha\beta}{}_{\gamma\delta}\mathrm{\text{\pounds}}_{u}g^{\gamma\delta}$
$\displaystyle=-2\nabla^{(\alpha}u^{\beta)}+2\nabla^{(\alpha}{\smash{\widehat{\Xi}}^{\beta)}}{}_{\mu}{\smash{\widehat{Q}}^{\mu}}{}_{\delta}u^{\delta}$
$\displaystyle\qquad-2\nabla^{(\alpha}{\smash{\widehat{\Xi}}^{\beta)}}{}_{\mu}\nabla^{\mu}\nabla_{\delta}u^{\delta}$
$\displaystyle=-2\nabla^{(\alpha}u^{\beta)}+2\nabla^{(\alpha}_{\phantom{\delta}}\delta^{\beta)}_{\delta}u^{\delta}$
$\displaystyle\qquad-2\nabla^{(\alpha}{\smash{\widehat{\Xi}}^{\beta)}}{}_{\mu}\nabla^{\mu}\nabla_{\delta}u^{\delta}$
$\displaystyle=-2\nabla^{(\alpha}{\smash{\widehat{\Xi}}^{\beta)}}{}_{\mu}\nabla^{\mu}\nabla_{\delta}u^{\delta}.$
(64)

For $\nabla_{\delta}u^{\delta}=0$, this leads to
$\smash{\smash{\widehat{\pi}}^{\alpha\beta}{}_{\gamma\delta}\mathrm{\text{\pounds}}_{u}g^{\gamma\delta}}=0$,
which is Eq. (14). Otherwise, notice that

$\displaystyle
2\nabla_{\delta}u^{\delta}=2g_{\delta\gamma}\nabla^{\gamma}u^{\delta}=2g_{\gamma\delta}\nabla^{(\gamma}u^{\delta)}=-g_{\gamma\delta}\mathrm{\text{\pounds}}_{u}g^{\gamma\delta}.$
(65)

Then, one can rewrite Eq. (64) as

$\displaystyle\smash{\widehat{\pi}}^{\alpha\beta}{}_{\gamma\delta}\mathrm{\text{\pounds}}_{u}g^{\gamma\delta}=\nabla^{(\alpha}{\smash{\widehat{\Xi}}^{\beta)}}{}_{\mu}\nabla^{\mu}g_{\gamma\delta}\mathrm{\text{\pounds}}_{u}g^{\gamma\delta},$
(66)

which is precisely Eq. (15).

##  Appendix C Derivation of Eq. (3c)

Using Eq. (17b), we get

$\smash{\widehat{\Pi}}_{\rm
g}^{\alpha\beta}{}_{\gamma\delta}\smash{\widehat{\Pi}}_{\rm
g}^{\gamma\delta}{}_{\lambda\varepsilon}=4\nabla^{(\alpha}\smash{\widehat{\Xi}}^{\beta)}{}_{(\gamma}\nabla_{\delta)}\nabla^{(\gamma}\smash{\widehat{\Xi}}^{\delta)}{}_{(\lambda}\nabla_{\varepsilon)}\\\
-2\nabla^{(\alpha}\smash{\widehat{\Xi}}^{\beta)}{}_{(\gamma}\nabla_{\delta)}\nabla^{(\gamma}\smash{\widehat{\Xi}}^{\delta)}{}_{\nu}\nabla^{\nu}g_{\lambda\varepsilon}\\\
-2\nabla^{(\alpha}\smash{\widehat{\Xi}}^{\beta)}{}_{\mu}\nabla^{\mu}g_{\gamma\delta}\nabla^{(\gamma}\smash{\widehat{\Xi}}^{\delta)}{}_{(\lambda}\nabla_{\varepsilon)}\\\
+\nabla^{(\alpha}\smash{\widehat{\Xi}}^{\beta)}{}_{\mu}\nabla^{\mu}g_{\gamma\delta}\nabla^{(\gamma}\smash{\widehat{\Xi}}^{\delta)}{}_{\nu}\nabla^{\nu}g_{\lambda\varepsilon}.$
(67)

Let us simplify the individual terms on the right-hand side separately. We
start by expanding one pair of symmetrized indices to get

$\displaystyle
4\nabla^{(\alpha}\smash{\widehat{\Xi}}^{\beta)}{}_{(\gamma}\nabla_{\delta)}\nabla^{(\gamma}\smash{\widehat{\Xi}}^{\delta)}{}_{(\lambda}\nabla_{\varepsilon)}$
$\displaystyle\qquad=2\nabla^{(\alpha}\smash{\widehat{\Xi}}^{\beta)}{}_{\gamma}\nabla_{\delta}\nabla^{\gamma}\smash{\widehat{\Xi}}^{\delta}{}_{(\lambda}\nabla_{\varepsilon)}+2\nabla^{(\alpha}\smash{\widehat{\Xi}}^{\beta)}{}_{\gamma}\nabla^{2}\smash{\widehat{\Xi}}^{\gamma}{}_{(\lambda}\nabla_{\varepsilon)}$
$\displaystyle\qquad=2\nabla^{(\alpha}\smash{\widehat{\Xi}}^{\beta)}{}_{\gamma}\nabla^{\gamma}\nabla_{\delta}\smash{\widehat{\Xi}}^{\delta}{}_{(\lambda}\nabla_{\varepsilon)}+2\nabla^{(\alpha}\smash{\widehat{\Xi}}^{\beta)}{}_{\gamma}\nabla^{2}\smash{\widehat{\Xi}}^{\gamma}{}_{(\lambda}\nabla_{\varepsilon)}$
$\displaystyle\qquad\qquad+2\nabla^{(\alpha}\smash{\widehat{\Xi}}^{\beta)}{}_{\gamma}\left[\nabla_{\delta},\nabla^{\gamma}\right]\smash{\widehat{\Xi}}^{\delta}{}_{(\lambda}\nabla_{\varepsilon)}.$
(68)

Recognizing that the operator would act on a rank-2 tensor
$\smash{h^{\lambda\varepsilon}}$, we can use Eq. (4) for the commutator;
hence,

$4\nabla^{(\alpha}\smash{\widehat{\Xi}}^{\beta)}{}_{(\gamma}\nabla_{\delta)}\nabla^{(\gamma}\smash{\widehat{\Xi}}^{\delta)}{}_{(\lambda}\nabla_{\varepsilon)}=2\nabla^{(\alpha}\smash{\widehat{\Xi}}^{\beta)}{}_{\gamma}\nabla^{\gamma}\nabla_{\delta}\smash{\widehat{\Xi}}^{\delta}{}_{(\lambda}\nabla_{\varepsilon)}\\\
+2\nabla^{(\alpha}\smash{\widehat{\Xi}}^{\beta)}{}_{\gamma}\left({R^{\gamma}}_{\delta}+\delta_{\delta}^{\gamma}\nabla^{2}\right)\smash{\widehat{\Xi}}^{\delta}{}_{(\lambda}\nabla_{\varepsilon)}.$
(69)

The terms in the parenthesis on the right-hand side of the above equation can
be expressed through $\smash{\smash{\widehat{Q}}^{\alpha}{}_{\beta}}$ [Eq.
(9b)], which is also the inverse of
$\smash{\smash{\widehat{\Xi}}^{\alpha}{}_{\beta}}$ [Eq. (11)]; hence,

$\displaystyle
4\nabla^{(\alpha}\smash{\widehat{\Xi}}^{\beta)}{}_{(\gamma}\nabla_{\delta)}\nabla^{(\gamma}\smash{\widehat{\Xi}}^{\delta)}{}_{(\lambda}\nabla_{\varepsilon)}$
$\displaystyle\quad=2\nabla^{(\alpha}\smash{\widehat{\Xi}}^{\beta)}{}_{\gamma}\nabla^{\gamma}\nabla_{\delta}\smash{\widehat{\Xi}}^{\delta}{}_{(\lambda}\nabla_{\varepsilon)}-2\nabla^{(\alpha}\smash{\widehat{\Xi}}^{\beta)}{}_{\gamma}{\smash{\widehat{Q}}^{\gamma}}{}_{\delta}\smash{\widehat{\Xi}}^{\delta}{}_{(\lambda}\nabla_{\varepsilon)}$
$\displaystyle\quad=2\nabla^{(\alpha}\smash{\widehat{\Xi}}^{\beta)}{}_{\gamma}\nabla^{\gamma}\nabla_{\delta}\smash{\widehat{\Xi}}^{\delta}{}_{(\lambda}\nabla_{\varepsilon)}-2\nabla^{(\alpha}\smash{\widehat{\Xi}}^{\beta)}{}_{(\lambda}\nabla_{\varepsilon)}.$
(70)

Using a similar process, the second term is found to be

$\displaystyle
2\nabla^{(\alpha}\smash{\widehat{\Xi}}^{\beta)}{}_{(\gamma}\nabla_{\delta)}\nabla^{(\gamma}\smash{\widehat{\Xi}}^{\delta)}{}_{\nu}\nabla^{\nu}g_{\lambda\varepsilon}$
$\displaystyle\quad=\nabla^{(\alpha}\smash{\widehat{\Xi}}^{\beta)}{}_{\gamma}\nabla_{\delta}\nabla^{\gamma}{\smash{\widehat{\Xi}}^{\delta}}{}_{\nu}\nabla^{\nu}g_{\lambda\varepsilon}+\nabla^{(\alpha}\smash{\widehat{\Xi}}^{\beta)}{}_{\gamma}\nabla^{2}{\smash{\widehat{\Xi}}^{\gamma}}{}_{\nu}\nabla^{\nu}g_{\lambda\varepsilon}$
$\displaystyle\quad=\nabla^{(\alpha}{\smash{\widehat{\Xi}}^{\beta)}}{}_{\gamma}\left({R^{\gamma}}_{\delta}+\delta_{\delta}^{\gamma}\nabla^{2}\right){\smash{\widehat{\Xi}}^{\delta}}{}_{\nu}\nabla^{\nu}g_{\lambda\varepsilon}$
$\displaystyle\qquad+\nabla^{(\alpha}{\smash{\widehat{\Xi}}^{\beta)}}{}_{\gamma}\nabla^{\gamma}\nabla_{\delta}{\smash{\widehat{\Xi}}^{\delta}}{}_{\nu}\nabla^{\nu}g_{\lambda\varepsilon}$
$\displaystyle\quad=-\nabla^{(\alpha}\smash{\widehat{\Xi}}^{\beta)}{}_{\mu}\nabla^{\mu}g_{\lambda\varepsilon}+\nabla^{(\alpha}{\smash{\widehat{\Xi}}^{\beta)}}{}_{\gamma}\nabla^{\gamma}\nabla_{\delta}{\smash{\widehat{\Xi}}^{\delta}}{}_{\nu}\nabla^{\nu}g_{\lambda\varepsilon}.$
(71)

The third and the fourth terms are simply

$\displaystyle
2\nabla^{(\alpha}\smash{\widehat{\Xi}}^{\beta)}{}_{\mu}\nabla^{\mu}g_{\gamma\delta}\nabla^{(\gamma}\smash{\widehat{\Xi}}^{\delta)}{}_{(\lambda}\nabla_{\varepsilon)}=2\nabla^{(\alpha}\smash{\widehat{\Xi}}^{\beta)}{}_{\mu}\nabla^{\mu}\nabla_{\delta}\smash{\widehat{\Xi}}^{\delta}{}_{(\lambda}\nabla_{\varepsilon)},$
$\displaystyle\nabla^{(\alpha}\smash{\widehat{\Xi}}^{\beta)}{}_{\mu}\nabla^{\mu}g_{\gamma\delta}\nabla^{(\gamma}\smash{\widehat{\Xi}}^{\delta)}{}_{\nu}\nabla^{\nu}g_{\lambda\varepsilon}=\nabla^{(\alpha}\smash{\widehat{\Xi}}^{\beta)}{}_{\mu}\nabla^{\mu}\nabla_{\delta}\smash{\widehat{\Xi}}^{\delta}{}_{\nu}\nabla^{\nu}g_{\lambda\varepsilon}.$

Combining all these expressions, we get

$\displaystyle\smash{\widehat{\Pi}}_{\rm
g}^{\alpha\beta}{}_{\gamma\delta}\smash{\widehat{\Pi}}_{\rm
g}^{\gamma\delta}{}_{\lambda\varepsilon}$
$\displaystyle=2\nabla^{(\alpha}\smash{\widehat{\Xi}}^{\beta)}{}_{\gamma}\nabla^{\gamma}\nabla_{\delta}\smash{\widehat{\Xi}}^{\delta}{}_{(\lambda}\nabla_{\varepsilon)}$
$\displaystyle-2\nabla^{(\alpha}\smash{\widehat{\Xi}}^{\beta)}{}_{(\lambda}\nabla_{\varepsilon)}+\nabla^{(\alpha}\smash{\widehat{\Xi}}^{\beta)}{}_{\mu}\nabla^{\mu}g_{\lambda\varepsilon}$
$\displaystyle-\nabla^{(\alpha}{\smash{\widehat{\Xi}}^{\beta)}}_{\gamma}\nabla^{\gamma}\nabla_{\delta}{\smash{\widehat{\Xi}}^{\delta}}{}_{\nu}\nabla^{\nu}g_{\lambda\varepsilon}$
$\displaystyle-2\nabla^{(\alpha}\smash{\widehat{\Xi}}^{\beta)}{}_{\mu}\nabla^{\mu}\nabla_{\delta}\smash{\widehat{\Xi}}^{\delta}{}_{(\lambda}\nabla_{\varepsilon)}$
$\displaystyle+\nabla^{(\alpha}\smash{\widehat{\Xi}}^{\beta)}{}_{\mu}\nabla^{\mu}\nabla_{\delta}\smash{\widehat{\Xi}}^{\delta}{}_{\nu}\nabla^{\nu}g_{\lambda\varepsilon}.$
(72)

Canceling the first term on the right-hand side with the fifth term, and the
fourth term with the sixth term, we arrive at

$\displaystyle\smash{\widehat{\Pi}}_{\rm
g}^{\alpha\beta}{}_{\gamma\delta}\smash{\widehat{\Pi}}_{\rm
g}^{\gamma\delta}{}_{\lambda\varepsilon}=-2\nabla^{(\alpha}\smash{\widehat{\Xi}}^{\beta)}{}_{(\lambda}\nabla_{\varepsilon)}+\nabla^{(\alpha}\smash{\widehat{\Xi}}^{\beta)}{}_{\mu}\nabla^{\mu}g_{\lambda\varepsilon}.$
(73)

Upon comparison with Eq. (17b), this leads to Eq. (3c).

##  Appendix D Derivation of Eq. (57)

For any vector field $u^{\alpha}$, one has

$\displaystyle\nabla^{\beta}\nabla_{\beta}u^{\alpha}$
$\displaystyle=\nabla^{\beta}\left(\partial_{\beta}u^{\alpha}+\Gamma^{\alpha}_{\beta\lambda}u^{\lambda}\right)$
$\displaystyle=\partial^{\beta}\left(\partial_{\beta}u^{\alpha}+\Gamma^{\alpha}_{\beta\lambda}u^{\lambda}\right)$
$\displaystyle\quad+g^{\beta\gamma}\Gamma^{\alpha}_{\gamma\rho}\left(\partial_{\beta}u^{\rho}+\Gamma^{\rho}_{\beta\lambda}u^{\lambda}\right)$
$\displaystyle\quad-g^{\beta\gamma}\Gamma^{\rho}_{\beta\gamma}\left(\partial_{\rho}u^{\alpha}+\Gamma^{\alpha}_{\rho\lambda}u^{\lambda}\right),$
(74)

where $\Gamma^{\alpha}_{\beta\gamma}$ are the Christoffel symbols. In normal
coordinates, the Christoffel symbols are zero, but their derivatives are not.
This leads to

$\displaystyle\nabla^{2}u^{\alpha}=\partial^{2}u^{\alpha}+u^{\lambda}\partial^{\beta}\Gamma^{\alpha}_{\beta\lambda}.$
(75)

The derivatives of the Christoffel symbols can be expressed through the
Riemann tensor ${R^{\rho}}_{\sigma\mu\nu}$ ref:brewin98 :

$\displaystyle\partial_{\nu}\Gamma_{\mu\sigma}^{\rho}=-\frac{1}{3}\left({R^{\rho}}_{\sigma\mu\nu}+{R^{\rho}}_{\mu\sigma\nu}\right).$
(76)

Using the well-known symmetries of the Riemann tensor and of the Ricci tensor
$R_{\sigma\nu}\doteq{R^{\rho}}_{\sigma\rho\nu}$, one then finds that

$\partial^{\beta}\Gamma_{\beta\lambda}^{\alpha}=-\frac{1}{3}\left({R^{\alpha}}_{\lambda\beta}{}^{\beta}+{R^{\alpha}}_{\beta\lambda}{}^{\beta}\right)=-\frac{1}{3}\,{R^{\alpha}}_{\beta\lambda}{}^{\beta}\\\
=-\frac{1}{3}\,{R_{\lambda}}^{\beta\alpha}{}_{\beta}=-\frac{1}{3}\,{R^{\beta}}{}_{\lambda\beta}{}^{\alpha}=-\frac{1}{3}\,{R}_{\lambda}{}^{\alpha}=-\frac{1}{3}\,{R^{\alpha}}_{\lambda}.$
(77)

Hence, one can rewrite Eq. (75) as

$\displaystyle\nabla^{2}u^{\alpha}=\partial^{2}u^{\alpha}-\frac{1}{3}\,{R^{\alpha}}_{\beta}u^{\beta},$
(78)

or equivalently, as

$\displaystyle(\nabla^{2})^{\alpha}{}_{\beta}=\delta^{\alpha}_{\beta}\partial^{2}-\frac{1}{3}\,R^{\alpha}{}_{\beta}.$
(79)

In the index-free representation, this leads to Eq. (57).

## References

  * (1) S. Carroll, Spacetime and Geometry: An Introduction to General Relativity (Addison-Wesley, San Francisco, 2004).
  * (2) E. E. Flanagan and S. A. Hughes, The basics of gravitational wave theory, New J. Phys. 7, 204 (2005).
  * (3) J. M. Bardeen, Gauge-invariant cosmological perturbations, Phys. Rev. D 22, 1882 (1980).
  * (4) K. A. Malik and D. Wands, Cosmological perturbations, Phys. Rep. 475, 1 (2009).
  * (5) K. A. Malik and D. R. Matravers, Comments on gauge-invariance in cosmology, Gen. Relativ. Gravit. 45, 1989 (2013).
  * (6) K. Nakamura, Second-order gauge invariant cosmological perturbation theory: — Einstein equations in terms of gauge invariant variables —, Prog. Theor. Phys. 117, 17 (2007).
  * (7) M. Bruni, S. Matarrese, S. Mollerach, and S. Sonego, Perturbations of spacetime: gauge transformations and gauge invariance at second order and beyond, Class. Quantum Gravity 14, 2585 (1997).
  * (8) V. De Luca, G. Franciolini, A. Kehagias, and A. Riotto, On the gauge invariance of cosmological gravitational waves, J. Cosmol. Astropart. Phys. 2020, 014 (2020).
  * (9) D. Garg and I. Y. Dodin, Average nonlinear dynamics of particles in gravitational pulses: Effective Hamiltonian, secular acceleration, and gravitational susceptibility, Phys. Rev. D 102, 064012 (2020).
  * (10) G. Baym, S. P. Patil, and C. J. Pethick, Damping of gravitational waves by matter, Phys. Rev. D 96, 084033 (2017).
  * (11) K. Bamba, S. Nojiri, and S. D. Odintsov, Propagation of gravitational waves in strong magnetic fields, Phys. Rev. D 98, 024002 (2018).
  * (12) F. A. Asenjo and S. M. Mahajan, Resonant interaction between dispersive gravitational waves and scalar massive particles, Phys. Rev. D 101, 063010 (2020).
  * (13) D. Barta and M. Vasúth, Dispersion of gravitational waves in cold spherical interstellar medium, Int. J. Mod. Phys. D 27, 1850040 (2018).
  * (14) D. Chesters, Dispersion of gravitational waves by a collisionless gas, Phys. Rev. D 7, 2863 (1973).
  * (15) E. Asseo, D. Gerbal, J. Heyvaerts, and M. Signore, General-relativistic kinetic theory of waves in a massive particle medium, Phys. Rev. D 13, 2724 (1976).
  * (16) P. G. Macedo and A. H. Nelson, Propagation of gravitational waves in a magnetized plasma, Phys. Rev. D 28, 2382 (1983).
  * (17) R. Flauger and S. Weinberg, Gravitational waves in cold dark matter, Phys. Rev. D 97, 123506 (2018).
  * (18) M. Servin, G. Brodin, and M. Marklund, Cyclotron damping and Faraday rotation of gravitational waves, Phys. Rev. D 64, 024013 (2001).
  * (19) J. Moortgat and J. Kuijpers, Gravitational and magnetosonic waves in gamma-ray bursts, Astron. Astrophys. 402, 905 (2003).
  * (20) M. Forsberg and G. Brodin, Linear theory of gravitational wave propagation in a magnetized, relativistic Vlasov plasma, Phys. Rev. D 82, 124029 (2010).
  * (21) H. Isliker, I. Sandberg, and L. Vlahos, Interaction of gravitational waves with strongly magnetized plasmas, Phys. Rev. D 74, 104009 (2006).
  * (22) M. D. Duez, Y. T. Liu, S. L. Shapiro, and B. C. Stephens, Relativistic magnetohydrodynamics in dynamical spacetimes: Numerical methods and tests, Phys. Rev. D 72, 024028 (2005).
  * (23) J. T. Mendonça, Gravitational waves in plasmas, Plasma Phys. Control. Fusion 44, B225 (2002).
  * (24) C. W. Misner, K. S. Thorne, and J. A. Wheeler, Gravitation (Freeman, San Francisco, 1973).
  * (25) M. A. Oancea, J. Joudioux, I. Y. Dodin, D. E. Ruiz, C. F. Paganini, and L. Andersson, Gravitational spin Hall effect of light, Phys. Rev. D 102, 024075 (2020).
  * (26) I. Y. Dodin, A. I. Zhmoginov, and D. E. Ruiz, Variational principles for dissipative (sub)systems, with applications to the theory of linear dispersion and geometrical optics, Phys. Lett. A 381, 1411 (2017).
  * (27) Our calculations were facilitated by Mathematica © 1988–2019 Wolfram Research, Inc., version number 12.0.0.0.
  * (28) B. Schutz, A First Course in General Relativity (Cambridge University Press, New York, 2009), Eq. (8.36).
  * (29) R. A. Isaacson, Gravitational radiation in the limit of high frequency. I. The linear approximation and geometrical optics, Phys. Rev. 166, 1263 (1968).
  * (30) M. A. H. MacCallum and A. H. Taub, The averaged Lagrangian and high-frequency gravitational waves, Commun. Math. Phys. 30, 153 (1973).
  * (31) I. Y. Dodin, D. E. Ruiz, K. Yanagihara, Y. Zhou, and S. Kubo, Quasioptical modeling of wave beams with and without mode conversion. I. Basic theory, Phys. Plasmas 26, 072110 (2019).
  * (32) S. W. McDonald, Phase-space representations of wave equations with applications to the eikonal approximation for short-wavelength waves, Phys. Rep. 158, 337 (1988).
  * (33) L. Brewin, Riemann normal coordinates, smooth lattices and numerical relativity, Class. Quantum Gravity 15, 3085 (1998).


# Guiding Attention using Partial-Order Relationships for Image Captioning

Murad Popattia1, Muhammad Rafi1, Rizwan Qureshi1,2, Shah Nawaz3†
1National University of Computer and Emerging Sciences, Karachi, Pakistan,
2Hamad Bin Khalifa University, Doha, Qatar
3Pattern Analysis & Computer Vision (PAVIS) - Istituto Italiano di Tecnologia
(IIT)
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

The use of attention models for automated image captioning has enabled many
systems to produce accurate and meaningful descriptions for images. Over the
years, many novel approaches have been proposed to enhance the attention
process using different feature representations. In this paper, we extend this
approach by creating a guided attention network mechanism, that exploits the
relationship between the visual scene and text-descriptions using spatial
features from the image, high-level information from the topics, and temporal
context from caption generation, which are embedded together in an ordered
embedding space. A pairwise ranking objective is used for training this
embedding space which allows similar images, topics and captions in the shared
semantic space to maintain a partial order in the visual-semantic hierarchy
and hence, helps the model to produce more visually accurate captions. The
experimental results based on MSCOCO dataset shows the competitiveness of our
approach, with many state-of-the-art models on various evaluation metrics.

##  1 Introduction

††$\dagger$ Current Affiliation: Deutsches Elektronen-Synchrotron
(DESY)††Email<EMAIL_ADDRESS>
Recent success of deep neural networks in computer vision, speech, and natural
language processing have prompted academics to think beyond these fields as
separate entities, instead solving challenges at their intersections [9, 33,
6, 2, 17, 19]. Generating descriptive and meaningful captions for images, and
to capture its semantic meaning, is one such multimodal inference problem [10,
3]. Despite its complexity, it has various applications, including visually-
impaired assistance, intelligent chat-bots, medical report generation, self-
driving cars, and many more [23]. In general, an image captioning model should
be able to find objects, their positions, map the relationship, as well as
express this relationships in a human understandable language.

A typical image caption system consists of a convolutional neural network
(CNN) and a recurrent neural network (RNN), with CNN as the image encoder and
RNN as the sentence decoder [26, 21]. However, in order to capture the spatial
context from the image in an efficient manner, other approaches such as [32,
8, 34, 31] incorporate high-level information from topics or detected objects
as semantic features to the decoder model. Another line of research was to
make use of cross-modal associations between image and text features in a
joint-embedding space. Earlier research work [15, 13] treated images and
caption as a symmetric relationship by using Euclidean or cosine distances to
gauge similarities between these two modalities. On the other hand, in [25]
treated these associations as asymmetric by enforcing a hierarchical order
within the embedding space, and has shown to perform better than symmetric
relationships.

Figure 1: Examples of generated captions by humans (GT), attention (ATT) and
using guided attention (T-OE-ATT). The words higlighted in respective colors
denote a comparison between the semantic detail captured by the approaches
used.

Further improvement in this framework, is the introduction of attention
mechanism [22], which allows the decoder to focus on a sub-region of the
image, when predicting the next word in the caption [28]. Despite of focusing
only on spatial attention, Lu et al. [16] presented a novel adaptive mechanism
for helping the attention module to learn, when to shift between spatial and
temporal context during word prediction. In addition, Anderson et al. [1]
improves the attention process by first detecting a set of salient image
regions (bottom-up) and then attending to these fixated regions (top-down).
[30] builds upon this concept by exploiting the semantic relationships between
the detected spatial regions using GCN (Graph Convolution Networks). [11] also
make use of a similar approach but instead modify the attention module by
adding self-attention module on top of the conventional attention mechanism,
which helps the decoder to draw relations between various attended vectors.

On the other hand, Jiang et al. [12], focused on increasing the semantic
information fed to the decoder by using a fusion of multiple encoders, each
focusing on a different view point, to build better representations for the
decoder. Likewise, Wang et al. [27] also worked in a similar direction that
guides attention using a hierarchy of semantic features. However, lack of
inter-feature correlations between these encoders makes it difficult for the
decoder to leverage the association from the resulting joint representations.
Lastly, despite relying on spatial cues from encoded features, Ke et al. [14]
worked on improving the temporal coherence of words during descriptions by
applying attention on both visual and textual domains.

Alongside the same line of work of incorporating semantic associations between
different spatial regions using GCNs [30], our idea is to make use of multi-
modal representations such as ordered embeddings [25] as our semantic feature
vectors to guide the attention module. Similar to the late-fusion of features
as done in [30], we instead use a weighted summation as our fusion mechanism
to fuse these embeddings.

Overall the main contributions of our work are three-fold:

  * •

We make use of ordered embedding features for topics and images to guide the
attention module instead of feeding them as low-level features. This step has
been shown to improved metrics, see ablation study (Section 3.3.1).

  * •

We incorporate a global weighted sum for fusing “visual” and “temporal” states
instead of feeding them at each time-step separately which helps the model to
learn the best estimation of the attention required for each image.

  * •

Lastly, we present an ablation study of each contribution and how it effects
the overall performance of the model on the MSCOCO dataset.

Figure 2: The overall framework of the proposed model, where $\Sigma_{w}$
represents a weighted-summation and $\oplus$ denotes matrix addition. The
model consists of a feature extractor and a topic classifier to extract
spatial features and topics given in an image. These semantic attributes are
then fed into a retrieval model which arranges an image, topic and caption
triplet in a partial-order hierarchy. The resultant embeddings are then late-
fused using weighted summation and then fed into a ’guiding LSTM’. The ’core-
lstm’ then makes use of this hidden state for temporal attention.
Consequently, two separate attention blocks are used, each attending to
different aspects of decoding and the resulting attention vectors are late-
fused again in a weighted fashion to produce captions.

##  2 Methodology

###  2.1 Overall Framework

Our approach follows the traditional encoder-decoder framework, where the
encoder is responsible to pass on features used by the decoder to output the
most likely word during captioning. Figure 2 illustrates the overall
framework.

Similar to recent approaches of sending objects or topics during encoding [34,
29], we used topics instead of objects to capture both the ”actors” as well
the “activities” binding them. The encoder consists of three components: 1)
topic classifier 2) feature extractor and the 3) retrieval model. We use a
pre-trained deep CNN model as a feature extractor to extract visual features
from the image and train a multi-label topic classifier to predict topics for
given images. After that, we train a retrieval model which embeds captions,
image and topics into a shared semantic space, in which a similarity score can
be calculated between them. Interestingly, using embeddings helps to better
learn the latent relationships between image and topic features, lost during
feature extraction. This helps the attention module in describing and
discriminating spatial regions more effectively. (Details in Section 2.3)

Inspired from the simple yet effective architecture defined in [1], we used
two LSTM branches in the decoder i.e. the guiding-lstm and the core-lstm.
Here, we define a weighted sum of the semantic embeddings of both the images
and topics, as input to the guiding-lstm at the first time-step, which gives
the model a better understanding of the alignment of visual features and the
topics. We then utilize its hidden state ht-1 for guiding the language LSTM
and the context vector zt used for attention. Similar to using a visual
sentinel [16], we used a weighted summation for fusing the attention weights
instead of a sentinel gate to shift between spatial and temporal attentions.
This allows for a more simpler architecture in terms of learning parameters
involved, whilst maintaining the accuracy during word prediction.

###  2.2 Topic Classifier

For extracting topics T, the ground-truth captions are concatenated to form
documents D, where each document d corresponds to captions C, for a given
image and contains a set of words W. After that, we train a Latent Dirichlet
Allocation (LDA) model [4], which is a probabilistic model to learn the topic
representations from documents. The trained topic model outputs a set of topic
probabilities T={T1, T2, T3, … Tn}.

For training the classifier, the topics are sampled and converted to one-hot
representations using the following function:

$f\textsubscript{t\textsubscript{i} $\subseteq$
T\textsubscript{i}}(x)=\left\\{\begin{array}[]{rcl}1&\mbox{if}&P(x)\geq 0.1\\\
0&\mbox{else}\end{array}\right.$ (1)

where ti represents a single topic from a set of topics T, for image i from a
set of images I and P(x) represents the topic-confidence from LDA. We
formulate this as a multi-label classification problem, since an image can
have multiple topics. A pre-trained CNN model is used to extract image
features which are then fed into a feed-forward neural network with a sigmoid
activation for the prediction layer. This layer outputs an (Ni$\times$Nt)
vector where Ni corresponds to the number of images and Tt are the number of
topics. We report the evaluation for the topic classifier in Section 4.1 of
the paper.

###  2.3 Retrieval Model

The architecture of the retrieval model is inspired by the approaches in [32,
25]. It follows the idea of [13] to align caption and images in the same
space, but with a partial-order relation rather than a symmetric relation.
This is a more intuitive approach as images have captions with different
levels of details, and because the captions are so dissimilar, it is
impossible to map both their embeddings close to the same image embedding
using a symmetric distance measure like cosine similarity. Nevertheless,
maintaining order is robust to such affect, as dissimilar caption can have
embeddings placed very far away from the image, while remaining above it in
the partial order. The partial order relation can be defined as:
$x\preceq y$ $\Longleftrightarrow\forall x\forall y(x\geq y)$. This imposes
for all values of the vector x to be greater than all values of the vector y
in the embedding space to maintain order.

We start with three entities i.e. images I, topics T and captions C. As per
[25], we utilized domain-specific encoders to extract features for training
the embeddings. For images and topics, we utilized the fully-connected
features from the feature-extractor and the topic features from the topic
classifier respectively. While for captions, we used a Gated Relu Unit (GRU)
as the RNN based text-encoder instead of an LSTM, because of its computational
efficiency. These feature vectors are then weighted with WI, WT and WC before
being projected in the embedding space:

$\vspace{1mm}O\textsubscript{i}=\|W\textsubscript{I}\cdot
f\textsubscript{FE}(I)\|^{2}$ (2)
$\vspace{1mm}O\textsubscript{t}=\|W\textsubscript{T}\cdot
f\textsubscript{TC}(T)\|^{2}$ (3)
$O\textsubscript{c}=\|W\textsubscript{C}\cdot GRU(C)\|^{2}$ (4)

Oi, Ot, Oc represents the order embeddings of image, topics, and captions
respectively. fFE(I) represents the image features from the feature-extractor,
while fTC(T) represents the features from the topics classifier. We use
L2-Norm during encoding instead of an absolute value function to mitigate
overfitting [25].

Similarity Function The general notion of similarity between two vectors x and
y in the embedding space can hence be quantified as the degree to which a pair
of points violates the partial order  $x\preceq y$ [25]:

$S(x,y)=-(\|max(0,O\textsubscript{y}-O\textsubscript{x})\|^{2})$ (5)

where Ox and Oy represents the encoded feature vector in the embedding space.
The negative sign constitutes to the fact that a positive difference between
Oy and Ox denotes violation of the order penalty.

Loss Function As previous works which learn embedding in cross-modal retrieval
tasks [15, 13], we re-use the pair-wise ranking loss objective to increase the
similarity for the matching pairs and vice-versa for the contrastive terms by
a margin $\alpha$:

$\begin{split}L(x,y)=\sum_{(x,y)}(\sum_{x^{\prime}}max\\{0,\alpha-S(x,y)+S(x^{\prime},y)\\}+\\\
\sum_{y^{\prime}}max\\{0,\alpha-S(x,y)+S(x,y^{\prime})\\})\end{split}$ (6)

where $(x,y)$ is the ground-truth pair while $(x^{\prime},y)$ and
$(x,y^{\prime})$ are constrastive terms. Our hierarchy has image at the top of
the partial order, followed by captions which are then bounded by the topics.
Hence, the total loss can be defined as the summation of losses over all three
partial orders:

$L=L(I,C)+L(I,T)+L(C,T)$ (7)

###  2.4 Caption Generation

We now describe the decoding phase of the model. The trained encoding
functions $O\textsubscript{i}$ and $O\textsubscript{t}$ are used to produce
relevant embeddings for image and topics during feature extraction. We then
used a weighted-summation ( $\Sigma_{w}$ ), of these embeddings:

$\Sigma_{w(OE)}=\lambda\cdot O_{i}+(1-\lambda)\cdot O_{t}$ (8)

where $\lambda$ is a learnable parameter. The reason for a weighted-sum is to
allow the model to learn the relative importance of each embedding during
training. Different from the approach of [32], we focused on guiding the
decoder in a 3-way manner i.e. using the embedding information, visual
features and reliance on past information from the hidden states.

Dual-LSTM branch We used an auxiliary guiding-lstm, to process the information
from the learned embeddings and feeding the hidden state information to both
the attention vector $z_{t}$ and the core-lstm at initial timestep t = -1:

$h_{t-1}=LSTM_{g}(\Sigma_{w(OE)})$ (9) $z_{t}=W_{g}h_{t-1}+W_{c}h_{t}$ (10)
$h_{t}=LSTM_{c}(x_{t},h_{t-1})$ (11)

where $h_{t-1}$ and $h_{t}$ represents the hidden states at relevant
timesteps, $W_{g}$ and $W_{c}$ are learnable parameters in the context vector
$z_{t}$. $LSTM_{g}$ and $LSTM_{c}$ represent the guiding and core LSTMs
respectively. The initial hidden state for $LSTM_{g}$ is essentially zeroes
and hence not shown in the formulation.

Spatial Attention Block This block is responsible to generate the attention
distribution vector over the important visual regions of the image. Similar to
the idea of soft-attention [16], we utilize the context-vector $z_{t}$ from
equation 10 instead of just the hidden state information done in [16], in
order to guide attention pertaining to the partial-order relation between the
image and topic:

$\alpha_{t}=softmax(W_{\alpha}[W_{f}F_{L}+W_{z}z_{t}])$ (12)
$\rho_{s}=\sum_{i=1}^{N}\alpha_{ti}f_{i}$ (13)

where $F_{L}=\\{f_{1},f_{2},....f_{N}\\}$ represent the local image features
from the convolution layer just before the FC layer of the feature extractor,
$\alpha_{t}$ denotes the attention weights over the features in $F_{L}$,
$\alpha_{ti}$ denotes the weight over the ith part of $F_{L}$ and $\rho_{s}$
denotes the spatial-context vector.

Temporal Attention Block The temporal block guides the attention module
whether the information is required at all, or the next word can be predicted
using the past information stored within the decoder [16]. Likewise, we
utilize the information from the LSTM’s memory cell along with the context
vector $z_{t}$ which contains the residual embedding information from the
previous timestep. It helps the temporal block decide whether the current
timestep requires attending to visual features or not. This is illustrated
below:

$\rho_{t}=\tanh(c_{t})\bigodot\sigma({W}_{x}x_{t}+W_{z^{\prime}}z_{t})$ (14)

where $c_{t}$ is the memory cell of the core-lstm, $x_{t}$ is the word vector
at timestep t, $z_{t}$ denotes the context vector, $\bigodot$ refers to an
element-wise product and $\rho_{t}$ denotes the temporal-context vector.

Word Prediction Instead of keeping track of the temporal information for each
word, we let the model generalize the ratio between these attentions using a
weighted-summation ( $\Sigma_{w}$ ). This is because ideally it is a more
simpler approach to rely less on the attention gate at each timestep and
generalize from the embedding context obtained from $z_{t}$.

$\Sigma_{w(ATT)}=\mu\cdot\rho_{s}+(1-\mu)\cdot\rho_{t}$ (15)

We then calculate the word probability over a vocabulary of possible words at
time t:

$p_{t}=softmax(f_{MLP}(z_{t}+\Sigma_{w(ATT)}))$ (16)

where $f_{MLP}$ denotes a dense layer with ReLU activation.

##  3 Experiments

###  3.1 Implementation Details

As our model is divided into sub-components, we train each part separately
instead of training them end-to-end.

Feature Extractor We use a ResNet-$152$ [7] model trained on ImageNet dataset.
The FC features are taken from the last layer of the CNN which have a
dimension of $2048$$\times$$1$. We use
$F_{L}=\\{f_{1},f_{2},....f_{N}\\},f_{i}\in R^{512}$ to represent the spatial
CNN features at each of the N grid locations where N = $49$.

Topic Classifier For the training the topic model, we limit our vocabulary to
top $5000$ and train the LDA on these features for $100$ iterations. We
empirically set the number of topics to be $80$ for our case. Increasing the
topics made the topic vectors more sparse and decreased the recall for the
topic classifier. For the topic classifier, we used the image features
$R^{2048\times 1}$ to be fed into a $5$-layer feed-forward NN, with the
prediction layer $R^{80}$ having a sigmoid activation. The classifier was
optimized using SGD with a learning-rate of $0.1$ and momentum $0.9$. The
learning-rate was changed in case of plateauing with a patience of $0.4$ and a
factor of $0.2$.

Retrieval Model For the retrieval model, we reused the FC image features
$R^{2048\times 1}$ from the feature extractor and the $R^{80}$ topic features
from the topic classifier in Section 2.2. The dimensions of the embedding
space and the GRU hidden state in equation (4) were set to $1024$, and the
margin $\alpha$ is set to $0.05$ as per [25].

Caption Model For the decoder, our model used LSTMs. The guiding and core
LSTMs both have a dimension of $512$. For the captions, we use a word
embedding size of $256$. During training, we see that downsizing and
concatenating FC image features with this embedding improved results. The
initial value for $\lambda$ and $\mu$ in equation (8) is set to $0.5$ for
both, and learned during training. Furthermore, the number of units for
$f_{MLP}$ was set to $1024$. Lastly, for sampling the captions, we use beam
size of $1$. The whole model was optimized using Adam optimizer with a mini-
batch size of 128 and learning rate of 0.001. The model trained for $10$
epochs on a Tesla T$4$ GPU and the training finished in $10$ hours to produce
the results.

###  3.2 Datasets

We conducted experiments on the popular benchmark: Microsoft COCO dataset
111https://cocodataset.org/ as this has been widely used for benchmarking in
the related literature. Also, we adopt the ‘Karpathy’ splits setting [14],
which includes 118,287 training images, and 5K testing images for evaluation.
Some images had more than 5 corresponding captions, the excess of which are
discarded for consistency. We directly use the publicly available code
222https://github.com/tylin/coco-caption provided by Microsoft for result
evaluation, which includes BLEU, METEOR, ROUGE-L and CIDEr.

Approaches | BLEU-I | BLEU-II | BLEU-III | BLEU-IV | METEOR | ROUGE-L | CIDEr
---|---|---|---|---|---|---|---
Adaptive ATT [16] | 74.2 | 58.0 | 43.9 | 33.2 | 26.6 | - | 108.5
LSTM-A [31] | 75.4 | - | - | 35.2 | 26.9 | 55.8 | 108.8
RF-Net [12] | 76.4 | 60.4 | 46.6 | 35.8 | 27.4 | 56.5 | 112.5
Up-Down ATT [1] | 77.2 | - | - | 36.2 | 27.0 | 56.4 | 113.5
HAN [27] | 77.2 | 61.2 | 47.7 | 36.2 | 27.5 | 56.6 | 114.8
RDN [14] | 77.5 | 61.8 | 47.9 | 36.8 | 27.2 | 56.8 | 115.3
GCN-LSTM [30] | 77.4 | - | - | 37.1 | 28.1 | 57.2 | 117.1
AoA-Net [11] | 77.4 | - | - | 37.2 | 28.4 | 57.5 | 119.8
Ours (T-OE-ATT) | 77.0 | 61.2 | 47.1 | 35.9 | 28.4 | 57.3 | 115.9
Table 1: Performance comparison on MSCOCO ’Karpathy’ test split trained on a single-model using cross-entropy loss without CIDEr optimization. (-) indicates metrics not provided. All values are provided in percentages (%) with the highest bold-faced. Approach | B-IV | METEOR | ROUGE-L | CIDEr
---|---|---|---|---
Topic | 25.5 | 22.9 | 50.1 | 80.2
T-OE(VGG) | 34.4 | 27.8 | 56.5 | 112.7
T-OE(Resnet) | 35.4 | 28.2 | 57.0 | 114.4
T-OE-ATT | 35.9 | 28.4 | 57.3 | 115.9
Table 2: Ablation study on MSCOCO ’Karpathy’ test split.

###  3.3 Evaluation

####  3.3.1 Ablation Study

To study the effects of guiding the attention module, we design an ablation
experiment to assess the effect of 1) using an embedding space 2) using a
different feature extractor and 3) using embedding along with attention as
shown in Table 2. We see that the initial approach of feeding topics as low-
level features performs poorly. A dramatic improvement was seen when using an
embedding space in the process. This confirms the hypothesis that embeddings
serve as a better auxiliary guidance for attention. We term this as (T-OE).
Moreover, we assess the model’s performance on a less accurate feature
extractor such as VGG-19 [20] which only incurred as small change in the
metrics signifying that trained embeddings are robust to changes in the
feature extractor. Lastly, we incorporate attention in the process (T-OE-ATT)
and guide them using the trained embeddings which shows an improved score in
all metrics, signifying the importance of the embeddings to guide attention.

####  3.3.2 Quantitative Evaluation

In Table 1, we compare our proposed architecture with recent state-of-the-art
models on the MSCOCO dataset that make use of LSTMs in their decoder
architecture. For fair comparison, we report the scores for single model for
each approach that use the same CNN backbone as ours (ResNet [7]), without
using ensembling and CIDEr optimizations.

Our approach is able to outperform RF-Net [12] and HAN [27] signifying that
using partial order is more suitable for building joint multi-modal
representations as compared to using domain-specific encoders alone. Moreover,
incorporating attention with T-OE, as shown in Table 2, also helps us
outperform RDN [14] over notable metrics such as METEOR, ROUGE-L and CIDEr
which show that orthogonal improvements to encoder or decoder alone are less
susceptible to improvement as compared to jointly improving both the feature
representations and the caption generation process. It is worth noting that
compared to our architecture, RDN [14] and RF-Net [12] have a greater number
of learning parameters (1.15B parameters [14] for RDN), whilst our decoder
contains comprises of only 29M parameters and yet is able to produce
competitive results. Both GCN-LSTM [30] and AoA-Net [11] use Faster-RCNN as
their feature encoder which is able to feed in region-level information while
our model uses only the fully connected features from the ResNet backbone and
is still competitive over METEOR and ROUGE-L scores. It should also be noted
that AoA-Net [11] leverage the use of self-attention mechanisms which have
been used alongside transformers and are able to produce state-of-the-art
results. On the contrary, our work can be extended to incorporate region level
information alongside topics or to use a different attention mechanism to
improve results and has not been explored in this study.

As our model uses LSTMs for caption generation, hence this comparison does not
take into account transformer-based architectures [5, 24]. Transformers are a
different class of architecture as compared to LSTMs as they do not follow the
auto-regressive nature of LSTMs and process the inputs in a parallel fashion
[24] so incorporating partial-order embeddings alongside this class of
architecture could also be a favourable research direction.

####  3.3.3 Qualitative Evaluation

We assess our model qualitatively as illustrated in Figure 1. The baseline
model is based on model’s output based on topic and image features, while the
guided attention model is based on topic and image embeddings. Without
embeddings, we see that attention lack descriptiveness of the context
associated with the visual features such as double-decker, grassy, drinking
etc.

Figure 3: Examples of inaccurate captions from the model.

We also see an influence when comparing ground truth captions where the model
was able to capture semantic context like parked instead of sitting and
drinking water instead of in the water. It is because the model is able to
draw associations between objects and actions due to partial-order information
from the underlying topics of the captions fed into the decoder module
denoting how attention was guided.

However, as denoted in Figure 3, the attention module can pick up on noise
from these embedded features such as confusing between a bus and a truck. This
is evident from T-OE, where the caption contains truck even though it is
absent from the image. An explanation can be bus and truck being semantically
closer in the embedding space. Moreover, relying on spatial attention can also
lead to mis-classifying objects in the image from spatula to knife. This can
be seen from the caption generated from the model without T-OE where the
object is misidentified as a knife.

##  4 Discussion

###  4.1 Evaluation of topic classifier and retrieval model

As the topic classifier and the embedding sub-space act as intermediaries to
the final model, hence, we evaluate their performance on relative metrics as
well. The output of the topic classifier is a set of sigmoid probabilities
which are converted into one-hot encodings. Using precision solely for
evaluating one-hot encodings is not enough as we can see that a higher
precision does not mean our model has a good recall. Hence, we use F1-score
with a $\beta$ more inclined towards recall. The highest F1-score was achieved
in the COCO dataset which may be due to a larger amount of data being used to
train the model. We summarize these results in Table 3.

Dataset | Precision | Recall | F1-Score
---|---|---|---
Flickr30k | 60.08 | 42.33 | 43.56
MSCOCO | 77.54 | 60.48 | 61.52
Table 3: Performance results of the topic classifier on validation sets of
Flick30k and MSCOCO dataset

For the order embedding model, we assess the quality of the model by treating
it as a Caption Retrieval task. The metric used in this experiment was
Recall@K which refers to the percentage of recall achieved in top-k items. We
summarize these results in Table 4.

Dataset | R@1 | R@5 | R@10
---|---|---|---
Flickr30k | 35.2 | 61.9 | 73.4
MSCOCO | 49.5 | 79.6 | 89.3
Table 4: Performance results of the retrieval model on validation sets of
Flick30k and MSCOCO dataset

Nevertheless, the scores for both the topic classifier and the retrieval model
were not state-of-the-art but were enough to extract suitable features for the
training images. Respective improvements to the models in terms of fine-tuning
or using a different architecture, might positively impact the overall
accuracy during captioning but is beyond the scope of this paper.

###  4.2 Visualizing the embedding space

In this section, we present a high-level visualization of the partial-order
structure between images, topics and captions in the embedding space, as shown
in Figure 4.

The embedding space consist of three modalities, with images being at the
highest order, captions being at the center and topics being at the lowest
order of the hierarchy posing a lower bound for the captions. This
hierarchical arrangement also conforms with the cognitive arrangement of these
modalities. Images are generally abstract points from which we derive meaning
about its context while separate words such as topics can be used to
complement images but do not contribute to any meaning on their own. Captions
on the other hand, describe a story which the spatial cues of the image
support.

We can then visualize these captions as a collection of words each of which
can constitute to a topic. Treating the problem as a caption retrieval task,
where given an image the model outputs the set of all possible captions,
setting a lower bound with topics helps constraint this search space and helps
reduce noise from overlapping caption regions. [32]

###  4.3 Analysis of the weighted summation for attention

Figure 4: Representation of order in the embedding space

Contrasting to the approach followed in [16], where the model is trained to
shift attention at each word prediction step, we constraint the model in
determining an overall ratio of the spatial or temporal attention needed for
word prediction and keep this as a static value for all succeeding
predictions. However, despite setting the values randomly, we allow the
decoder to generalize from a set of caption on the amount of attention needed
for each caption. For testing, we set the temporal context $\mu$ to be 0.3 for
spatial and consequently, 0.7 for temporal attention. The reason for a higher
ratio for temporal context is because it complements the RNNs capability to
work with sequences. For the model we use the ATT approach where the image
features are fed directly as spatial cues to the decoder. Figure 5 shows the
learned ratios after several iteration of training.

It can be seen that the model gradually learns to increase the gradient flow
from the spatial block of the attention module, signifying the need of visual
attention. However, we do notice some peaks for the flow of temporal
information. A plausible reason is because while visual information is
necessary, it may not always be inline with temporal coherence when describing
images. Hence, we sample captions with different values for $\mu$ as shown in
Figure 6.

Figure 5: Weight distributions of spatial and temporal attention for several
iterations on MSCOCO dataset.

For a lower value of $\mu$ in the formulation 15 in Section 2.4, the model
allows the flow of temporal information in the decoder and hence we see a
time-relative aspect in sentences with phrases such as ”about to hit” and ”is
laying on”. On the contrary, if we shift the value of $\mu$ higher, it boosts
the gradient flow from the spatial block filling in visual details from the
image such as ”white shirt”, ”white shorts”, ”laptop computer”, ”top of desk”.
However, we see that despite being rich in scene-specific details, the model
misses out the global context of the image imposing the need for a good
balance between both the attention modules. This is the reason we allow the
model to learn these weights during training.

##  5 Limitations

In this section, we discuss the architectural limitations to our work and also
explore future extensions to this approach. Firstly, the performance of the
decoder phase is dependant on the output from the topic classifier and bottle-
necks the overall improvement from training. Moreover, most recent works such
as GCN-LSTM [30] and AoA-Net [11] make use of Faster-RCNN to feed in region-
level information and hence incorporating these object-level associations
alongside topics in the multi-modal embedding space are susceptible to
increase in efficacy of the approach used. Another limitation of our work is
the use of traditional attention mechanisms. Our study make use of soft-
attention mechanism which involves the averaging of feature maps. Comparing
our approach with HAN [27] which also makes use of soft-attention mechanism,
we gain a relative improvement as discussed in Section 3.3.2. However, our
approach struggles against AoA-Net [11] which uses a more robust attention
mechanism. Moroever, the use of self-attention has been shown to improve
performance over traditional attention mechanisms such as [11], more notably
in transformers [5, 24] and hence can be incorporated with the use of these
multi-modal embeddings to improve performance. Lastly, using recent
reinformcement learning based techniques such as CIDEr optimizations [18] have
yielded state-of-the-art results for image captioning, incorporating them with
our study may further boost the performance over the metrics used.

Figure 6: Sampled captions on varying $\mu$ values on COCO dataset. A higher
value of $\mu$ denotes more weight being given to the spatial flow of
information within the decoder and viceversa.

##  6 Conclusion

In this work, we proposed a new approach to guide the attention model by
exploiting partial-order relationships between image, captions and topics.
Arranging the image and textual modalities in an asymmetric fashion results in
more effective learning of the latent space. Hence, we make use of a multi-
modal embedding space that is able to arrange the visual and textual
modalities in an asymmetrical hierarchy where the caption embeddings are
bounded between image and topic features. We then make use of these joint
representations to guide the attention module. An extensive ablation study was
also performed to indicate that using ordered embeddings, the attention model
was able to draw accurate links between semantically important regions of the
image when attending to them, which helped improve the overall
interpretability, syntax and descriptiveness of the captions. The proposed
architecture was not only simpler in terms of complexity, but also competitive
with many recent LSTM-based architectures. For next steps, a promising
direction can be to incorporate the highlighted approach with transformers or
leveraging the model architecture to be trained in an end-to-end manner.

## References

  * [1] Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang.  Bottom-up and top-down attention for image captioning and visual question answering.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6077–6086, 2018.
  * [2] Omer Arshad, Ignazio Gallo, Shah Nawaz, and Alessandro Calefati.  Aiding intra-text representations with visual context for multimodal named entity recognition.  In 2019 International Conference on Document Analysis and Recognition (ICDAR), pages 337–342. IEEE, 2019.
  * [3] Shuang Bai and Shan An.  A survey on automatic image caption generation.  Neurocomputing, 311:291–304, 2018.
  * [4] David M Blei, Andrew Y Ng, and Michael I Jordan.  Latent dirichlet allocation.  the Journal of machine Learning research, 3:993–1022, 2003.
  * [5] Marcella Cornia, Matteo Stefanini, Lorenzo Baraldi, and Rita Cucchiara.  Meshed-memory transformer for image captioning, 2020.
  * [6] Ignazio Gallo, Alessandro Calefati, and Shah Nawaz.  Multimodal classification fusion in real-world scenarios.  In 2017 14th IAPR international conference on document analysis and recognition (ICDAR), volume 5, pages 36–41. IEEE, 2017.
  * [7] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [8] Simao Herdade, Armin Kappeler, Kofi Boakye, and Joao Soares.  Image captioning: Transforming objects into words, 2020.
  * [9] Chiori Hori, Takaaki Hori, Teng-Yok Lee, Ziming Zhang, Bret Harsham, John R Hershey, Tim K Marks, and Kazuhiko Sumi.  Attention-based multimodal fusion for video description.  In Proceedings of the IEEE international conference on computer vision, pages 4193–4202, 2017.
  * [10] MD Zakir Hossain, Ferdous Sohel, Mohd Fairuz Shiratuddin, and Hamid Laga.  A comprehensive survey of deep learning for image captioning.  ACM Computing Surveys (CsUR), 51(6):1–36, 2019.
  * [11] Lun Huang, Wenmin Wang, Jie Chen, and Xiaoyong Wei.  Attention on attention for image captioning.  CoRR, abs/1908.06954, 2019.
  * [12] Wenhao Jiang, Lin Ma, Yu-Gang Jiang, Wei Liu, and Tong Zhang.  Recurrent fusion network for image captioning, 2018.
  * [13] Andrej Karpathy and Li Fei-Fei.  Deep visual-semantic alignments for generating image descriptions.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3128–3137, 2015.
  * [14] Lei Ke, Wenjie Pei, Ruiyu Li, Xiaoyong Shen, and Yu-Wing Tai.  Reflective decoding network for image captioning, 2019.
  * [15] Ryan Kiros, Ruslan Salakhutdinov, and Richard S Zemel.  Unifying visual-semantic embeddings with multimodal neural language models.  arXiv preprint arXiv:1411.2539, 2014.
  * [16] Jiasen Lu, Caiming Xiong, Devi Parikh, and Richard Socher.  Knowing when to look: Adaptive attention via a visual sentinel for image captioning.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 375–383, 2017.
  * [17] Shah Nawaz, Muhammad Kamran Janjua, Ignazio Gallo, Arif Mahmood, Alessandro Calefati, and Faisal Shafait.  Do cross modal systems leverage semantic relationships?  In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pages 0–0, 2019.
  * [18] Steven J. Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel.  Self-critical sequence training for image captioning.  CoRR, abs/1612.00563, 2016.
  * [19] Muhammad Saad Saeed, Muhammad Haris Khan, Shah Nawaz, Muhammad Haroon Yousaf, and Alessio Del Bue.  Fusion and orthogonal projection for improved face-voice association.  arXiv preprint arXiv:2112.10483, 2021.
  * [20] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [21] Moses Soh.  Learning cnn-lstm architectures for image caption generation.  Dept. Comput. Sci., Stanford Univ., Stanford, CA, USA, Tech. Rep, 2016.
  * [22] Mike W Spratling and Mark H Johnson.  A feedback model of visual attention.  Journal of cognitive neuroscience, 16(2):219–237, 2004.
  * [23] Gargi Srivastava and Rajeev Srivastava.  A survey on automatic image captioning.  In International Conference on Mathematics and Computing, pages 74–83. Springer, 2018.
  * [24] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  arXiv preprint arXiv:1706.03762, 2017.
  * [25] Ivan Vendrov, Ryan Kiros, Sanja Fidler, and Raquel Urtasun.  Order-embeddings of images and language, 2016.
  * [26] Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan.  Show and tell: A neural image caption generator.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3156–3164, 2015.
  * [27] Weixuan Wang, Zhihong Chen, and Haifeng Hu.  Hierarchical attention network for image captioning.  Proceedings of the AAAI Conference on Artificial Intelligence, 33(01):8957–8964, Jul. 2019.
  * [28] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio.  Show, attend and tell: Neural image caption generation with visual attention.  In International conference on machine learning, pages 2048–2057. PMLR, 2015.
  * [29] Zhongliang Yang, Yu-Jin Zhang, Sadaqat ur Rehman, and Yongfeng Huang.  Image captioning with object detection and localization, 2017.
  * [30] Ting Yao, Yingwei Pan, Yehao Li, and Tao Mei.  Exploring visual relationship for image captioning.  In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
  * [31] Ting Yao, Yingwei Pan, Yehao Li, Zhaofan Qiu, and Tao Mei.  Boosting image captioning with attributes, 2016.
  * [32] Niange Yu, Xiaolin Hu, Binheng Song, Jian Yang, and Jianwei Zhang.  Topic-oriented image captioning based on order-embedding.  IEEE Transactions on Image Processing, 28(6):2743–2754, 2018.
  * [33] Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason Corso, and Jianfeng Gao.  Unified vision-language pre-training for image captioning and vqa.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 13041–13049, 2020.
  * [34] Zhihao Zhu, Zhan Xue, and Zejian Yuan.  Topic-guided attention for image captioning.  In 2018 25th IEEE International Conference on Image Processing (ICIP), pages 2615–2619, 2018.