# Creating and Reenacting Controllable 3D Humans with Differentiable Rendering

Thiago L. Gomes1 Thiago M. Coutinho1 Rafael Azevedo1
Renato Martins2 Erickson R. Nascimento1
1Universidade Federal de Minas Gerais (UFMG), Brazil 2 Université Bourgogne
Franche-Comté, France
<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

This paper proposes a new end-to-end neural rendering architecture to transfer
appearance and reenact human actors. Our method leverages a carefully designed
graph convolutional network (GCN) to model the human body manifold structure,
jointly with differentiable rendering, to synthesize new videos of people in
different contexts from where they were initially recorded. Unlike recent
appearance transferring methods, our approach can reconstruct a fully
controllable 3D texture-mapped model of a person, while taking into account
the manifold structure from body shape and texture appearance in the view
synthesis. Specifically, our approach models mesh deformations with a three-
stage GCN trained in a self-supervised manner on rendered silhouettes of the
human body. It also infers texture appearance with a convolutional network in
the texture domain, which is trained in an adversarial regime to reconstruct
human texture from rendered images of actors in different poses. Experiments
on different videos show that our method successfully infers specific body
deformations and avoid creating texture artifacts while achieving the best
values for appearance in terms of Structural Similarity (SSIM), Learned
Perceptual Image Patch Similarity (LPIPS), Mean Squared Error (MSE), and
Fréchet Video Distance (FVD). By taking advantages of both differentiable
rendering and the 3D parametric model, our method is fully controllable, which
allows controlling the human synthesis from both pose and rendering
parameters. The source code is available at
https://www.verlab.dcc.ufmg.br/retargeting-motion/wacv2022.

##  1 Introduction

Figure 1: 3D texture-mapped human synthesis. Our method receives a frame of a
person, extracts her/his mesh (left side) and outputs a refined 3D shape and
appearance representations of people (right side).

The research in computer vision and graphics communities has made great
strides forward in the last two decades, with milestones being passed from
semantic segmentation [30], tridimensional reconstruction [31, 38, 39] to
synthesis of human motion [5, 20], realistic images [54] and videos [44]. As
more vision and graphics methods are integrated into new approaches such as
differentiable rendering, more systems will be able to achieve high accuracy
and quality in different tasks, in particular, generative methods for
synthesizing videos with plausible human reenactment.

Over the past few years, a remarkable performance of the Generative
Adversarial Networks (GANs) [10] has shed a new light for the problem of
synthesizing faithful real-world images of humans. Although GAN-based
approaches have been achieving high quality results in generating videos [2,
47, 22] and images [14, 12, 54, 44, 4] of people, in general, they suffer with
the high variability in the poses, unseen images from viewpoints not present
in the training data, and are often limited to reason directly on the
projected 2D image domain. Image-based rendering techniques [8, 9], for their
turn, are effective solutions to create 3D texture-mapped models of people,
being capable to synthesize images from any arbitrary viewpoint without using
large number of images. On the other hand, image-based rendering methods
require a painstaking design of the algorithm and are not capable of improving
the visual quality of the synthesized images by using more data when
available.

In this paper, we take a step towards combining learning and image-based
rendering approaches in a new end-to-end architecture that synthesizes human
avatars capturing both body geometry and texture details. The proposed
architecture comprises a graph convolution network (GCN) operating over the
mesh with differentiable rendering to achieve high-quality results in image
generation of humans. Our approach receives as input a frame of a person,
estimates a generic mesh in the desired pose and outputs a representation of
their shape and appearance, as shown in Figure 1. The method provides a final
3D representation that is compatible with traditional graphic pipelines.
Specifically, our architecture estimates a refined mesh and a detailed texture
map to properly represent the person’s shape and appearance for a given input
frame.

While 2D image-to-image translation methods are capable of generating
plausible images, many applications such as Virtual Reality (VR) and Augmented
Reality (AR) [6, 3, 30] require a fully 3D representation of the person.
Additionally, the view-dependence hinders the creation of new configurations
of scenes where the avatar can be included. Although view interpolation can be
applied to estimate a transition between a pair of camera poses, it may create
artifacts and unrealistic results when adding the virtual avatar into the new
scene using unseen camera poses. Video-based rendering systems are greatly
benefited through the use of realistic 3D texture-mapped models, which make
possible the inclusion of virtual avatars using unrestricted camera poses and
the automatic modification and re-arrangement of video footage. In addition to
being able to render human avatars from different viewpoints, the 3D shape
also allows synthesizing new images under different illumination conditions.
We experimentally showed the capabilities of our new architecture to estimate
realistic 3D texture-mapped models of humans. Experiments on a variety of
videos show that our method excels several state-of-the-art methods achieving
the best values for appearance in terms of Structural Similarity (SSIM),
Learned Perceptual Image Patch Similarity (LPIPS), Mean Squared Error (MSE),
and Fréchet Video Distance (FVD).

#### Contributions.

The main contributions of this paper are threefold: i) a novel formulation for
transferring appearance and reenacting human actors that produces a fully 3D
representation of the person; ii) a graph convolutional architecture for mesh
generation that leverages the human body structure information and keeps
vertex consistency, which results in a refined human mesh model; iii) a new
architecture that takes advantages of both differentiable rendering and the 3D
parametric model and provides a fully controllable human model, _i.e_., the
user can control the human pose and rendering parameters.

##  2 Related Work

#### Human animation by neural rendering.

Recently, we witnessed the explosion of neural rendering approaches to animate
and synthesize images of people on unseen poses [44, 4, 22]. According to
Tewari  _et al_. [41], neural rendering is a deep image or video generation
approach that enables explicit or implicit control of scene properties. The
main difference among these approaches is how the control signal is provided
to the network. Lassner  _et al_. [18] proposed a GAN called ClothNet.
ClothNet produces random people with similar pose and shape in different
clothing styles using as a control signal a silhouette image. In a similar
context, Esser  _et al_. [4] used a conditional U-Net to synthesize new 2D
human views based on estimated edges and body joint locations. Chan  _et al_.
[2] applied an adversarial training to map a 2D source pose to the appearance
of a target subject. Wang  _et al_. [44] presented a general video-to-video
synthesis framework based on conditional GANs, where a 2D dense UV mapping to
body surface (DensePose [37]) is used as a control signal. Similarly, Neverova
_et al_. [32] and Sarkar  _et al_. [40] rely on DensePose as input to
synthesize new views.

These image-to-image translation approaches only allow for implicit control by
way of representative samples, _i.e_., they can copy the scene parameters from
a reference image/video, but not manipulate these parameters explicitly. To
enable the control of the position, rotation, and body pose of a person in a
target image/video, Liu  _et al_. [22] proposed to use a medium-quality
controllable 3D template model of people. In the same line, Liu  _et al_. [24]
proposed a 3D body mesh recovery module based on a parametric statistical
human body model SMPL [25], which disentangles human body into joint rotations
and shape. Wu  _et al_. [46] produced photorealistic free-view-video from
multi-view dynamic human captures. Although these methods are capable of
generating plausible 2D images, they cannot generate a controllable 3D models
of people, unlike our approach, which is a desired feature in many tasks such
as in rendering engines and games or virtual and augmented reality contexts
[6, 3, 30].

#### 3D human pose and mesh reconstruction.

Substantial advances have been made in recent years for human pose and 3D
model estimation from still images. Bogo  _et al_. [1] proposed the SMPLify
method, a fully automated approach for estimating 3D body shape and pose from
2D joints in images. SMPLify uses a CNN to estimate 2D joint locations and
then optimize the re-projection errors of an SMPL body model [25] to these 2D
joints. Similarly, Kanazawa  _et al_. [13] used unpaired 2D keypoint
annotations and 3D scans to train an end-to-end network to regress the 3D mesh
parameters and the camera pose. Kolotouros  _et al_. [17] proposed SPIN, an
hybrid approach combining the ideas of optimization-based method from [1] and
regression deep network [13] to design an efficient method less sensitive to
the optimization initialization, while keeping accuracy from the optimization-
based method.

Despite remarkable results in pose estimation, these methods are limited to
estimate coarse quality generic meshes and textureless characters. Gomes  _et
al_. [8, 9] proposed an image-based rendering technique to create 3D textured
models of people synthesized from arbitrary viewpoints. However, their method
is not end-to-end and it is not capable of improving the visual quality of the
synthesized images by using information of all available data in the training
step. Lazova  _et al_. [19] automatically predict a full 3D textured avatar,
including geometry and 3D segmentation layout for further generation control;
however, their method cannot predict fine details and complex texture
patterns. Methods like PiFu [38, 39], for their turn, are limited to static 3D
models per frame. Their mesh represents a rigid body and the users cannot
drive the human body to new poses, _i.e_., it is not possible to make
animations where the model raise his/her hands since the arms and hands are
not distinguished from the whole mesh.

#### Graph networks (GCN) and adversarial learning.

GCNs recently emerged as a powerful representation for learning from data
lying on manifolds beyond n-dimensional Euclidean vector spaces. They have
been widely adopted to represent 3D point clouds such as PointNet [35] or Mesh
R-CNN [7], and notably, to model the human body structure with state-of-the-
art results in tasks such as human action recognition [50], pose estimation
[53, 43] and human motion synthesis [49, 5, 48]. Often these GCNs have been
combined and trained in adversarial learning schemes, as in human motion [5],
and pose estimation [13]. Our work leverages these features from GCNs and
adversarial training to estimate 3D texture-mapped human models.

#### Differentiable rendering.

Differentiable renderers (DR) are operators allowing the gradients of 3D
objects to be calculated and propagated through images while training neural
networks. As stated in [15], DR connects 2D and 3D processing methods and
allows neural networks to optimize 3D entities while operating on 2D
projections. Loper and Black [26] introduced an approximate differentiable
render which generates derivatives from projected pixels to the 3D parameters.
Kato  _et al_. [11] approximated the backward gradient of rasterization with a
hand-crafted function. Liu  _et al_. [23] proposed a formulation of the
rendering process as an aggregation function fusing the probabilistic
contributions of all mesh triangles with respect to the rendered pixels.
Niemeyer  _et al_. [33] represented surfaces as 3D occupancy fields and used a
numerical method to find the surface intersection for each ray, then they
calculate the gradients using implicit differentiation. Mildenhall  _et al_.
[28] encoded a 3D point and associated view direction on a ray using periodic
activation functions, then they applied classic volume rendering techniques to
project the output colors and densities into an image, which is naturally
differentiable. More recently, techniques [21, 34] based on neural radiance
field (NeRF) learning [29] are being proposed to synthesize novel views of
human geometry and appearance. While these methods achieved high-quality
results, they generally require multi-view data collected with calibrated
cameras and have high computational cost, notably during the inference/test
time.

In this paper, we propose a carefully designed architecture for human neural
rendering, leveraging the new possibilities offered by differentiable
rendering techniques [26, 23, 36, 52]. We present an end-to-end learning
method that i) does not require data capture with calibrated systems, ii) is
computationally efficient in test time, and iii) leverages DR and adversarial
training to improve the estimation capabilities of fully controllable
realistic 3D texture-mapped models of humans.

##  3 Methodology

Figure 2: Human synthesis architecture. Our architecture has three main
networks: a) the Human Mesh Estimation that comprises a three-stage GCN and
learns to fit and deform the mesh based on rendered silhouettes and shape
regularizers; b) the Texture Network, a CNN that is trained conditioned on the
visibility map generated from the deformed mesh to create a coarse texture by
rendering and optimizing on the $l_{1}$ norm; c) the Texture Refinement
Network, a second CNN that is conditioned on the visibility map and the coarse
texture to generate the detailed texture map of the person.

In order to generate deformable 3D texture-mapped human models, our end-to-end
architecture has three main components to be trained during the rendering. The
first component models local deformations on the human 3D body shape extracted
from the images using a three-stage GCN. In the second component, a CNN is
trained to estimate the human appearance map. Similar to the GCN, the CNN is
trained in a self-supervised regime using the gradient signals from the
differentiable renderer. Finally, the third component comprises an adversarial
regularization in the human appearance texture domain to ensure the
reconstruction of photo-realistic images of people.

All frames of a person are used to train the mesh and the texture networks.
The frames provide different viewpoints of the person to build his/her texture
map and to estimate the mesh deformation according to the motion. We apply a
stage-wise training strategy. First, we train the mesh model using the
silhouettes, which produce a texture-less human model. In the sequence, we
freeze the mesh network and train a global texture model. Then, we freeze both
the mesh model and the global texture model and learn the texture refinement
generator and the discriminators parameters. In the inference time, we feed
our architecture with generic meshes parameterized by the SMPL model, and then
to create a refined mesh and a detailed texture map to properly represent the
person’s shape and appearance. Figure 2 outlines these components and their
relations during the training phase and Figure 1 shows these components during
the inference phase.

###  3.1 Human Mesh Estimation

#### Shape and pose representation.

We adopt a shape representation that explores the global and local information
of the human body. We encode the global information using the SMPL model
parametrization [25], which is composed of a learned human shape distribution
$\mathcal{M}$, $24$ 3D joint angles $\boldsymbol{\theta}\in\mathbb{R}^{72}$
(defining 3D rotations of the skeleton joint tree), and shape coefficients
$\boldsymbol{\beta}\in\mathbb{R}^{10}$ that model the proportions and
dimensions of the human body. We used SPIN [17] to regress the SMPL parameters
due to its trade-off between accuracy and efficiency. While the global
information provided by SMPL parametrization enables global control of human
poses, they do not encode the fine details of the human body geometry (local
information). We then characterize the local information by adding a set of
offsets to the mesh produced by the SMPL parametrization from the GCN mesh
network.

#### Mesh Refinement Network.

After computing the global human shape and pose information
$P=\text{SMPL}(\boldsymbol{\theta},\boldsymbol{\beta})$, we model local
deformations on the mesh with the GCN Mesh Network $G_{m}$. Since the SMPL
parametrization only provides a coarse 3D shape, and it cannot accurately
model fine structures like clothes, we feed the mesh network with the initial
SMPL mesh to refine its vertex positions with a sequence of refinement stages.
The network is designed to learn the set of offsets to the SMPL mesh to
increase the realism of the generated views. Drawing inspiration from the
network architecture of [7], our refinement network is composed of three
blocks with six graph convolutional layers with intermediate features of size
$128$. Each block in our mesh network performs four operations:

  * •

Vertex normal computation. This operation computes the normal surface vector
for each vertex as being the weighted sum of the normals of faces containing
the vertex, where the face areas are used as the weights. The resulting normal
is assigned as the node feature ${f_{i}}$ for the vertex $v_{i}$ in the GCN.

  * •

Graph convolution. This convolutional layer propagates information along mesh
edges using a aggregation strategy. Similar to [7], given the input vertex
feature ${f_{i}}$, the layer updates the feature as
${f^{\prime}_{i}=\text{ReLU}(W_{0}f_{i}+\sum_{j\in\mathcal{N}}W_{1}f_{j})}$,
where $W_{0}$ and $W_{1}$ are learned weighting matrices, and $\mathcal{N}(i)$
gives the i-th vertex’s neighbors in the mesh.

  * •

Vertex refinement. To improve the quality of the vertex position estimation,
this operation computes vertex offsets as
$u_{i}=\tanh(W[f^{\prime}_{i};f_{i}])$. $W$ is a learned weighting matrix.

  * •

Vertex refinement clamping. To avoid strong deformations (large
$||u_{i}||_{2}$), we constraint and bound the position update of each vertex
$v_{i}$ as ${v^{\prime}_{i}=v_{i}+\min(\max(u_{i},-K(v_{i})),K(v_{i}))}$,
where $K(v)$ is the 3D update bounds allowed to the vertex $v$, depending on
the body part it belongs to. Each body part, _e.g_., face, footprints, hands,
head, torso, arms, feet, _etc_., have predefined bound thresholds. This
operation ensures that the offsets do not exceed the threshold defined to that
body part, and that the refinement of the mesh geometry do not affect the
body’s topology.

#### Loss function.

For learning the mesh refinement, our model exploits two differentiable
renderers that emulate the image formation process. Techniques such as [23,
36] enable us to invert such renderers and take the advantage of the learning
paradigm to infer shape and texture information from the 2D images.

During the training, the designed loss minimizes the differences between the
image silhouette extracted from the input real image $I_{s}$ and the image
silhouette $\hat{I}_{s}$ of the human body obtained by rendering the deformed
3D human model $M$ into the image by SoftRenderer, a differentiable renderer
$\Pi_{s}(M)$. SoftRenderer is a differentiable that synthesises the silhouette
of the actor. We can define the loss of the Mesh Network $G_{m}$ as:

$\mathcal{L}_{m}=\lambda_{gl}\mathcal{L}_{gl}+\lambda_{gn}\mathcal{L}_{gn}+\mathcal{L}_{s},$
(1)

where $\mathcal{L}_{gl}$ and $\mathcal{L}_{gn}$ regularize the Laplacian and
the normals consistency of the mesh respectively [36], $\lambda_{gl}$ and
$\lambda_{gn}$ are the weights for the geometrical regularizers, and

$\mathcal{L}_{s}=1-\frac{\left\|\hat{I}_{s}\otimes
I_{s}\right\|_{1}}{\left\|(\hat{I}_{s}+I_{s})-\hat{I}_{s}\otimes
I_{s}\right\|_{1}},$ (2)

is the silhouette loss proposed by Liu  _et al_. [23], where
${\hat{I}_{s}=\Pi_{s}(M)}$, $M=G_{m}(P)$ is the refined body mesh model, and
$\otimes$ is the element-wise product.

###  3.2 Human Texture Generation

We represent the appearance of the human model as a texture map in a fixed UV
space that is applied to the refined mesh, in our case, the SMPL mesh with
offsets. Our full pipeline for texture generation is depicted in Figure 2-b-c
and consists of two stages: a coarse texture generation and then texture
refinement. In an initial stage, given the refined 3D meshes of the actor $M$,
the Texture Network $G_{TN}$ learns to render the appearance of the actor in
the image. This texture is also further used to condition and regularize the
Texture Refinement Network $G_{RN}$ in the second stage.

#### Texture Network.

We start estimating a coarse texture map with a U-Net architecture [12]. The
input of the network is a visibility map $x_{v}$ and it outputs the texture
map $x_{p}$. The visibility map indicates which parts in the texture map must
be generated to produce a realistic appearance for the refined mesh. The
visibility map is extracted by a parametric function $x_{v}=U(M)$ that maps
points of refined mesh $M$ with positive dot product between the normal vector
and the camera direction vector to a point $x_{v}$ in the texture space. We
implement $U$ as a render of the 2D UV-map considering only faces with
positive dot product between the normal vector and the camera direction
vector. Thus, the network can learn to synthesize texture maps on demand
focusing on the important parts for each viewpoint.

Figure 2-b shows a schematic representation of the Texture Network training.
The HardRenderer $\Pi_{c}(M,x_{p})$, represents the colored differentiable
renderer that computes the output coarse textured image $\hat{I}$, of model
$M$ and texture map $x_{p}$. In our case, $\hat{I}=\Pi_{c}(M,G_{TN}(U(M))$.
Conversely to the SoftRenderer, the differentiable HardRenderer is used to
propagate the detailed human texture appearance information (color).
Specifically, we train the Texture Network to learn to generate a coarse
texture by imposing the $l_{1}$ norm in the person’s body region of the color
rendered image as:

$\mathcal{L}_{pt}={\left\|(\hat{I}-I)\otimes
B\right\|_{1}}/{\left\|B\right\|_{1}},$ (3)

where $I$ is the real image in the training set and $B$ is the union of the
visibility masks and real body regions.

#### Texture refinement.

We further improve the coarse texture to represent finer details. For that, we
design the Texture Refinement Network $G_{RN}$ to condition the generation of
a new texture map from the coarse texture, on a coherent output of the Texture
Network $G_{TN}$ and the visibility map.

In our adversarial training, the Texture Refinement Network acts as the
generator network $G_{RN}$ and engages in a minimax game against two task-
specific discriminators: the Face Discriminator $D_{1}$ and Image
Discriminator $D_{2}$. The generator is trained to synthesize texture maps in
order to fool the discriminators which must discern between “real” images and
“fake” images, where “fake” images are produced by the neural renderer using
the 3D texture-mapped model estimated by the Mesh and Texture Networks. While
the discriminator $D_{1}$ sees only the face region, the Image Discriminator
$D_{2}$ sees the whole image.

These three networks are trained simultaneously and drive each other to
improve their inference capabilities, as illustrated in Figure 2-c. The
Texture Refinement Network learns to synthesize a more detailed texture map to
deceive the discriminators which in turn learn differences between generated
outputs and ground truth data. The total loss function for the generator and
discriminators for the rendering is then composed of three terms:

$\displaystyle\min\limits_{G_{RN}}(\max\limits_{D_{1}}\mathcal{L}_{GAN}(G_{RN},D_{1})+$
(4)
$\displaystyle\max\limits_{D_{2}}\mathcal{L}_{GAN}(G_{RN},D_{2})+\mathcal{L}_{r}(G_{RN})),$

where both $\mathcal{L}_{GAN}$ terms address the discriminators and
$\mathcal{L}_{r}$ is a regularization loss to reduce the effects from outlier
poses. Each adversarial loss is designed as follows:

$\displaystyle\mathcal{L}_{GAN}(G,D)=\mathbb{E}_{y}[\log D(y)]+$ (5)
$\displaystyle\mathbb{E}_{x_{v},x_{p}}[\log(1-D(\Pi_{c}(M,G(x_{v},x_{p}))))],$

where $x_{v}$ is the visibility map, $x_{p}$ is the output of the Texture
Network, $M$ is the refined mesh, and $y$ is the corresponding segmented real
image $I\otimes B$.

Finally, to reduce the effects of wrong poses, which causes mismatches between
the rendered actor silhouette and silhouette of the real actor, we also add a
regularization loss to prevent the GAN to apply the color of the background
into the human texture. The first term of the regularization loss acts as a
reconstruction of the pixels by imposing the $l_{1}$ norm in the person’s body
region and the second term enforces eventual misaligned regions to stay close
to the coarse texture:

$\displaystyle\mathcal{L}_{r}=\alpha_{1}{\left\|(I-\hat{I}^{RN})\otimes
B\right\|_{1}}/{\left\|B\right\|_{1}}+$ (6)
$\displaystyle\alpha_{2}{\left\|(\hat{I}^{TN}-\hat{I}^{RN})\otimes
C\right\|_{1}}/{\left\|C\right\|_{1}},$

where $\alpha_{1}$ and $\alpha_{2}$ are the weights, $\hat{I}^{TN}$ is the
rendered image using the coarse texture, $\hat{I}^{RN}$ is the rendered image
using the refined texture, and $C$ is the misaligned regions without the face
region, _i.e_., the image region where the predicted silhouette and estimated
silhouette are different.

##  4 Experiments and Results

#### Datasets and baselines.

For the training of both texture models and the mesh human deformation network
we considered four-minute videos provided by [9], where the actors perform
random movements, allowing the model to get different views from the person in
the scene. We use the SMPL model parameters calculated by SPIN [17] and the
silhouette image segmented by MODNet [16] for each frame of the video. In the
evaluation/test time of our 3D human rendering approach, and to provide
comparisons to related methods, we conducted experiments with publicly
available videos used by Chan  _et al_. [2], Liu  _et al_. [24], and Gomes
_et al_. [9] as the evaluation sequences.

We compare our method against five recent methods including V-Unet [4],
Vid2Vid [44], EBDN [2], Retarget [9] and the Impersonator [24]. V-Unet is a
famous method of image-to-image translation that uses conditional variational
autoencoders to generate images based on a 2D skeleton and an image from the
target actor. The approach Retarget is an image-based rendering method based
on image rendering designed to perform human retargeting. Impersonator,
Vid2Vid, and EBDN are generative adversarial models trained to perform human
neural rendering.

#### Metrics and evaluation protocol.

We adopted complementary metrics to evaluate the quality of the approaches to
asset different aspects of the generated images such as structure coherence,
luminance, contrast, perceptual similarity [51], temporal, and spatial
coherence. The metrics used to perform quantitative analysis are SSIM [45],
LPIPS [51], Mean Square Error (MSE), and Fréchet Video Distance (FVD) [42].
Following the protocol proposed by [9], we executed all the methods in the
motion sequences and transfered them to the same background. This protocol
allows us to generate comparisons with the ground truth and compute the
metrics for all the generated images with their respective real peers. Then,
we group the values generated by the metrics in two ways: Motion Types and
Actors. In the first one, for each metric, we calculate the average values of
all actors making a motion sequence (_e.g_., “spinning”), while in the second
one, we calculate the average values of all movements performed by a given
actor. This grouping allows us to analyze the capability of the methods to
render the same actor performing various movements with different levels of
difficulty and also to compare their capacity to render actors with different
morphology performing the same movement.

#### Implementation details.

We trained our body mesh refinement network for $20$ epochs with batch size of
$4$. We used AdamW [27] with parameters $\beta_{1}=0.5$, $\beta_{2}=0.999$,
weight decay = $1\times 10^{-2}$, and learning rate of $1\times 10^{-4}$ with
a linear decay routine to zero starting from the middle of the training. We
empirically set $\lambda_{1}$ and $\lambda_{2}$ to $1.0$ and $0.5$,
respectively. In the Vertex refinement clamping component, we defined the set
of thresholds as follows: $K\in$ {face = $0.0$; footprints = $0.0$; hands =
$0.0$; head = $0.04$; torso = $0.06$; arms = $0.02$; forearms = $0.04$; thighs
= $0.04$; calves = $0.03$; feet = $0.02$} meters. All the training and
inference were performed in a single Titan XP ($12$ GB), where the GCN mesh
model and the human texture networks took around $6$ and $20$ hours per actor,
respectively. The inference time takes $92$ ms per frame ($90$ ms in the GCN
model deformation and $2$ ms in the texture networks).

Due to remarkable performance of pix2pix [12] in synthesizing photo-realistic
images, we build our Texture Network upon its architecture. The optimizers for
the texture models were configured as the same as the Mesh Network, except for
the learning rates. The learning rate for the whole body and face
discriminators, the global texture and refinement texture generators were set
as $2\times 10^{-5}$, $2\times 10^{-5}$, $2\times 10^{-3}$, and $2\times
10^{-4}$, respectively. The parameters of the texture reconstruction was set
to $\alpha_{1}=100$ and the regularization as $\alpha_{2}=100$. We observed
that smaller values led to inconsistency in the final texture. For the
training regime, we used $40$ epochs with batch size $8$. The global texture
model was trained separately from the other models for $2{,}000$ steps, then
we freeze the model, the texture refinement generator and the discriminators
were trained.

Figure 3: Ablation study. a) Results of the texture training without the
refinement stage; b) Model without the Vertex Refinement Clamp layer. We
observe an excessive growth of the mesh without update thresholds. The texture
produced lacks details and even could not preserve the actor’s face; c) shows
the results for our complete model. Table 1: Ablation study. SSIM, LPIPS, MSE,
and FVD comparison by motion types. Best in bold.

Method |  | Metrics
---|---|---
| SSIM1 | LPIPS2 | MSE2 | FVD2
Texture Refinement Removal | 0.869 | $0.136$ | $262.39$ | $795.15$
| Vertex Refinement Clamping | $0.866$ | $0.142$ | $288.04$ | $829.60$
| Complete Model | $0.868$ | 0.134 | 259.79 | 769.54
|  |  1Higher is better |  2Lower is better

#### Ablation Study.

We evaluate the contributions of different parts of the method to the overall
view synthesis performance. We investigated the benefits from the Vertex
refinement clamping component in the Mesh Refinement Network (MRN) and the use
of adversarial training in the texture generation. For the first experiment,
we removed the vertex refinement thresholds, letting the mesh grow loosely.
All other steps of texture training were maintained. Table 1 shows that the
performance dropped drastically when compared to our original model. A
qualitative analysis of the results in Figure 3-b demonstrates that removing
the Vertex refinement clamping component led to strong wrong deformations in
the hands and feet, _i.e_., the regions with higher pose estimation errors.

In the adversarial training analysis, we maintained the original Mesh
Refinement Network and removed the Texture Refinement Network and its
discriminators, training only the Global Texture Network using Equation 3.
Figure 3-a shows the texture quality of the models trained with and without
the adversarial regime. After removing the GAN the model could not generate
textures with fine details, producing blurry results. This result is also
reported in the metrics of Table 1, where we show the average values
calculated from all motion sequences in the test data in which the model
without GAN is outperformed in all results besides SSIM. This result is
coherent, since SSIM is based on low-level image features, and blurred
textures can lead to higher SSIM scores.

Table 2: Comparison with state-of-the-art human neural rendering. SSIM, LPIPS,
MSE, and FVD comparison by motion and actors types. Best in bold.

Metric | Method | Motion type |  | Actor type
---|---|---|---|---
jump | walk | spinning | shake hands | cone | fusion dance | pull down | box |  | S1 | S2 | S3 | S4
SSIM1 | EBDN [2] | $0.878$ | $0.880$ | $0.855$ | $0.859$ | $0.878$ | $0.820$ | $0.857$ | $0.858$ |  | $0.867$ | $0.898$ | $0.844$ | $0.834$
Imper [24] | $0.877$ | $0.880$ | $0.852$ | $0.859$ | $0.877$ | $0.816$ | $0.855$ | $0.856$ |  | $0.867$ | $0.896$ | $0.842$ | $0.831$
Retarget [9] | $0.881$ | $0.885$ | $0.855$ | $0.860$ | $0.879$ | $0.820$ | $0.861$ | $0.869$ |  | $0.872$ | $0.902$ | $0.846$ | $0.834$
Vid2Vid [44] | $0.880$ | $0.884$ | $0.856$ | $0.858$ | $0.878$ | $0.821$ | $0.859$ | $0.866$ |  | $0.868$ | $0.901$ | $0.848$ | $0.835$
V-Unet [4] | $0.870$ | $0.871$ | $0.843$ | $0.847$ | $0.862$ | $0.797$ | $0.847$ | $0.857$ |  | $0.855$ | $0.886$ | $0.830$ | $0.826$
Ours | 0.884 | 0.890 | 0.860 | 0.865 | 0.885 | 0.824 | 0.866 | 0.873 |  | 0.876 | 0.908 | 0.852 | 0.838
LPIPS2 | EBDN [2] | $0.141$ | $0.122$ | $0.139$ | $0.138$ | $0.143$ | $0.215$ | $0.151$ | $0.170$ |  | $0.159$ | $0.145$ | $0.147$ | $0.159$
Imper [24] | $0.151$ | $0.134$ | $0.151$ | $0.151$ | $0.155$ | $0.239$ | $0.168$ | $0.184$ |  | $0.161$ | $0.165$ | $0.170$ | $0.171$
Retarget [9] | 0.125 | $0.099$ | 0.130 | $0.131$ | $0.128$ | 0.206 | 0.131 | 0.127 |  | 0.133 | $0.129$ | 0.133 | 0.143
Vid2Vid [44] | $0.131$ | $0.105$ | $0.126$ | $0.136$ | $0.133$ | $0.203$ | $0.142$ | $0.133$ |  | $0.148$ | $0.131$ | $0.129$ | $0.147$
V-Unet [4] | $0.147$ | $0.132$ | $0.157$ | $0.161$ | $0.174$ | $0.243$ | $0.166$ | $0.158$ |  | $0.184$ | $0.160$ | $0.166$ | $0.158$
Ours | $0.127$ | 0.097 | 0.130 | 0.130 | 0.124 | 0.206 | $0.132$ | 0.127 |  | $0.136$ | 0.124 | $0.134$ | 0.143
MSE2 | EBDN [2] | $306.92$ | $269.58$ | $312.69$ | $312.12$ | $266.17$ | $463.79$ | $384.23$ | $361.57$ |  | $324.40$ | $314.10$ | $331.83$ | $368.19$
Imper [24] | $313.43$ | $275.22$ | $344.94$ | $314.92$ | $267.39$ | $504.00$ | $404.79$ | $377.16$ |  | $277.98$ | $328.07$ | $358.30$ | $436.57$
Retarget [9] | $237.16$ | $178.57$ | $286.86$ | $270.25$ | $237.86$ | $434.64$ | $301.66$ | $245.86$ |  | 243.95 | $260.14$ | $294.88$ | $297.46$
Vid2Vid [44] | $257.33$ | $206.42$ | $286.18$ | $332.09$ | $253.04$ | $452.77$ | $349.28$ | $288.54$ |  | $312.40$ | $274.80$ | $269.81$ | $356.26$
V-Unet [4] | $295.04$ | $269.59$ | $354.33$ | $377.02$ | $328.68$ | $559.75$ | $417.00$ | $346.13$ |  | $381.77$ | $344.71$ | $362.63$ | $384.66$
Ours | 231.20 | 153.23 | 278.75 | 254.38 | 218.76 | 418.58 | 286.02 | 237.42 |  | $247.10$ | 236.49 | 276.17 | 279.42
FVD2 | EBDN [2] | $887.56$ | $273.00$ | $918.94$ | 423.08 | $725.49$ | $952.22$ | $1,113.46$ | $853.26$ |  | $791.98$ | $751.45$ | 560.27 | $826.71$
Imper [24] | $1,770.31$ | $656.07$ | $1,531.64$ | $1,266.14$ | $1,051.42$ | $1,322.72$ | $1,440.94$ | $1,719.55$ |  | $1,270.41$ | $1,092.82$ | $1,395.81$ | $1,214.64$
Retarget [9] | $1,119.50$ | $330.91$ | 674.99 | $478.93$ | $767.68$ | $791.01$ | 988.35 | 760.33 |  | 715.00 | 653.30 | $720.49$ | 515.61
Vid2Vid [44] | 879.94 | $266.6$ | $1,085.49$ | $396.31$ | $790.79$ | $997.42$ | $997.96$ | $1,069.85$ |  | $778.53$ | $719.80$ | $762.46$ | $574.08$
V-Unet [4] | $1,491.63$ | $845.44$ | $1,721.81$ | $1,257.20$ | $1,415.24$ | $1,712.93$ | $2,437.98$ | $1,816.94$ |  | $2,239.14$ | $1,352.10$ | $1,856.78$ | $1,108.34$
Ours | $1,114.43$ | 233.81 | $1019.83$ | $542.24$ | 614.88 | 746.22 | $1010.24$ | $874.69$ |  | $881.49$ | $697.68$ | $718.21$ | $551.06$
|  |  1Higher is better |  2Lower is better |

###  4.1 Results

#### Quantitative comparison with state of the art.

We performed the neural rendering for actors with different body shapes,
gender, clothing styles, and sizes for all considered video sequences. The
video sequences used in the actors’ animations contained motions with
different levels of difficulty, which aims to test the generalization
capabilities of the methods in unseen data. Table 2 shows the performance for
each method considering all motion and actors types in the dataset. We can see
that our method achieves superior peformance as compared to the methods in
most of the motion sequences and actor types considering the SSIM, LPIPS, MSE,
and FVD metrics. These results indicate that our method is capable of
deforming the mesh according to the shape of the given actors and then,
rendering a texture optimized to fit the morphology of the person in the
scene. Furthermore, our training methodology, that considers multiple views
from the actor and the shape parameters, allows the generation of consistent
rendering with less visual artifacts when the character is performing
challenging movements, such as bending or rotating.

#### Qualitative visual analysis.

The visual inspection of synthesized actors also concur with the quantitative
analysis. Figure 4 shows the best frames for each movement using four actors
in the dataset. Our model and Retarget [9] are the only approaches capable of
keeping the body scale of the authors along all scenes, while the other
methods failed, in particular in the movements shake hands and walk. Besides
generating coherent poses, our technique also generated more realistic
textures in comparison to the other methods. Comparing the results of the
movements jump and spinning, one can visualize some details as the shadow of
the shirt sleeve of the actor and the shirt collar, respectively. Figure 5
illustrates a task of transferring the appearance in videos with different
scenes and camera-to-actor translations and camera intrinsics. These results
also demonstrate our capability of generating detailed face and body texture,
producing a good fit to synthesize views of actors observed from different
camera setups.

Figure 4: Qualitative comparison. Movements with four different actors are
represented in the rows. The results for each competitor are represented in
the columns. Figure 5: Human appearance transfer and animation with different
scenes and camera setups. The first line of each scene illustrates the
original frames of the source video. On the second line is the transferred
appearance of the animated virtual actor using our proposed method. The red
squares highlight the face generation quality.

##  5 Conclusions

In this paper we proposed a method that produces a fully 3D controllable
representation of people from images. Our key idea is leveraging
differentiable rendering, GCNs, and adversarial training to improve the
capability of estimating realistic 3D texture-mapped models of humans. Taking
advantages of both differentiable rendering and the 3D parametric model, our
method is fully controllable, which allows controlling the human pose and
rendering parameters. Furthermore, we have introduced a graph convolutional
architecture for mesh generation that refines the human body structure
information, which results in a more accurate human mesh model. The
experiments show that the proposed method has superior quality compared to
recent neural rendering techniques in different scenarios, besides having
several advantages in terms of control and ability to generalize to furthest
views.

#### Acknowledgments.

The authors would like to thank CAPES, CNPq, FAPEMIG, and Petrobras for
funding different parts of this work. R. Martins was also supported by the
French National Research Agency through grants ANR MOBIDEEP
(ANR-17-CE33-0011), ANR CLARA (ANR-18-CE33-0004) and by the French Conseil
Régional de Bourgogne-Franche-Comté. We also thank NVIDIA Corporation for the
donation of a Titan XP GPU used in this research.

## References

  * [1] Federica Bogo, Angjoo Kanazawa, Christoph Lassner, Peter Gehler, Javier Romero, and Michael J Black.  Keep it smpl: Automatic estimation of 3d human pose and shape from a single image.  In European Conference on Computer Vision, pages 561–578. Springer, 2016.
  * [2] Caroline Chan, Shiry Ginosar, Tinghui Zhou, and Alexei A Efros.  Everybody dance now.  In IEEE/CVF International Conference on Computer Vision, pages 5933–5942, 2019.
  * [3] Long Chen, Thomas W. Day, Wen Tang, and Nigel W. John.  Recent developments and future challenges in medical mixed reality.  In Wolfgang Broll, Holger Regenbrecht, and J. Edward Swan II, editors, 2017 IEEE International Symposium on Mixed and Augmented Reality, ISMAR 2017, Nantes, France, October 9-13, 2017, pages 123–135. IEEE Computer Society, 2017.
  * [4] Patrick Esser, Ekaterina Sutter, and Björn Ommer.  A variational u-net for conditional appearance and shape generation.  In IEEE Conference on Computer Vision and Pattern Recognition, 2018.
  * [5] João Pedro Moreira Ferreira, Thiago M. Coutinho, Thiago L. Gomes, José F. Neto, Rafael Azevedo, Renato Martins, and Erickson R. Nascimento.  Learning to dance: A graph convolutional adversarial network to generate realistic dance motions from audio.  Computer & Graphics, 94:11–21, 2021.
  * [6] Abir Gallala, Bassem Hichri, and Peter Plapper.  Survey: The evolution of the usage of augmented reality in industry 4.0.  IOP Conference Series: Materials Science and Engineering, 521:012017, may 2019.
  * [7] Georgia Gkioxari, Jitendra Malik, and Justin Johnson.  Mesh r-cnn.  In IEEE Conference on Computer Vision and Pattern Recognition, October 2019.
  * [8] Thiago L. Gomes, Renato Martins, João P. K. Ferreira, and Erickson R. Nascimento.  Do as I do: Transferring human motion and appearance between monocular videos with spatial and temporal constraints.  In IEEE Winter Conference on Applications of Computer Vision, pages 3355–3364. IEEE, 2020.
  * [9] Thiago L. Gomes, Renato Martins, João Pedro Moreira Ferreira, Rafael Azevedo, Guilherme Torres, and Erickson R. Nascimento.  A shape-aware retargeting approach to transfer human motion and appearance in monocular videos.  International Journal of Computer Vision, 129(7):2057–2075, 2021\.
  * [10] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In Advances in Neural Information Processing Systems (NeurIPS), pages 2672–2680, 2014.
  * [11] Yoshitaka Ushiku Hiroharu Kato and Tatsuya Harada.  Neural 3d mesh renderer.  In IEEE Conference on Computer Vision and Pattern Recognition, 2018.
  * [12] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros.  Image-to-image translation with conditional adversarial networks.  IEEE Conference on Computer Vision and Pattern Recognition, 2017\.
  * [13] Angjoo Kanazawa, Michael J. Black, David W. Jacobs, and Jitendra Malik.  End-to-end recovery of human shape and pose.  In IEEE Conference on Computer Vision and Pattern Recognition, 2018.
  * [14] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen.  Progressive growing of GANs for improved quality, stability, and variation.  In International Conference on Learning Representations, 2018.
  * [15] Hiroharu Kato, Deniz Beker, Mihai Morariu, Takahiro Ando, Toru Matsuoka, Wadim Kehl, and Adrien Gaidon.  Differentiable rendering: A survey.  arXiv preprint, 2020.
  * [16] Zhanghan Ke, Kaican Li, Yurou Zhou, Qiuhua Wu, Xiangyu Mao, Qiong Yan, and Rynson W.H. Lau.  Is a green screen really necessary for real-time portrait matting?  ArXiv, abs/2011.11961, 2020.
  * [17] Nikos Kolotouros, Georgios Pavlakos, Michael J Black, and Kostas Daniilidis.  Learning to reconstruct 3d human pose and shape via model-fitting in the loop.  In IEEE International Conference on Computer Vision, 2019.
  * [18] Christoph Lassner, Gerard Pons-Moll, and Peter V. Gehler.  A generative model for people in clothing.  In IEEE International Conference on Computer Vision, 2017.
  * [19] Verica Lazova, Eldar Insafutdinov, and Gerard Pons-Moll.  360-degree textures of people in clothing from a single image.  In 2019 International Conference on 3D Vision, 3DV 2019, Québec City, QC, Canada, September 16-19, 2019, pages 643–653. IEEE, 2019.
  * [20] Hsin-Ying Lee, Xiaodong Yang, Ming-Yu Liu, Ting-Chun Wang, Yu-Ding Lu, Ming-Hsuan Yang, and Jan Kautz.  Dancing to music.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
  * [21] Lingjie Liu, Marc Habermann, Viktor Rudnev, Kripasindhu Sarkar, Jiatao Gu, and Christian Theobalt.  Neural actor: Neural free-view synthesis of human actors with pose control.  CoRR, abs/2106.02019, 2021.
  * [22] Lingjie Liu, Weipeng Xu, Michael Zollhoefer, Hyeongwoo Kim, Florian Bernard, Marc Habermann, Wenping Wang, and Christian Theobalt.  Neural rendering and reenactment of human actor videos.  ACM Trans. Gr., 38(5):1–14, 2019.
  * [23] Shichen Liu, Tianye Li, Weikai Chen, and Hao Li.  Soft rasterizer: A differentiable renderer for image-based 3d reasoning.  IEEE International Conference on Computer Vision, Oct 2019.
  * [24] Wen Liu, Zhixin Piao, Min Jie, Wenhan Luo, Lin Ma, and Shenghua Gao.  Liquid warping GAN: A unified framework for human motion imitation, appearance transfer and novel view synthesis.  In IEEE International Conference on Computer Vision, 2019.
  * [25] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black.  Smpl: A skinned multi-person linear model.  ACM Transactions on Graphics, 2015.
  * [26] Matthew M Loper and Michael J Black.  Opendr: An approximate differentiable renderer.  In European Conference on Computer Vision, 2014.
  * [27] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In International Conference on Learning Representations, 2019.
  * [28] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In European Conference on Computer Vision, 2020.
  * [29] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In European Conference on Computer Vision, pages 405–421, 2020\.
  * [30] Shervin Minaee, Yuri Boykov, Fatih Porikli, Antonio Plaza, Nasser Kehtarnavaz, and Demetri Terzopoulos.  Image segmentation using deep learning: A survey.  2020\.
  * [31] Ryota Natsume, Shunsuke Saito, Zeng Huang, Weikai Chen, Chongyang Ma, Hao Li, and Shigeo Morishima.  Siclope: Silhouette-based clothed people.  In IEEE Conference on Computer Vision and Pattern Recognition, 2019.
  * [32] Natalia Neverova, Riza Alp Güler, and Iasonas Kokkinos.  Dense pose transfer.  In European Conference on Computer Vision, 2018.
  * [33] Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger.  Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 2020.
  * [34] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer.  D-NeRF: Neural Radiance Fields for Dynamic Scenes.  In IEEE Conference on Computer Vision and Pattern Recognition, 2021.
  * [35] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas.  Pointnet: Deep learning on point sets for 3d classification and segmentation.  In IEEE Conference on Computer Vision and Pattern Recognition, 2017.
  * [36] Nikhila Ravi, Jeremy Reizenstein, David Novotny, Taylor Gordon, Wan-Yen Lo, Justin Johnson, and Georgia Gkioxari.  Accelerating 3d deep learning with pytorch3d.  arXiv:2007.08501, 2020.
  * [37] Iasonas Kokkinos Riza Alp Güler, Natalia Neverova.  Densepose: Dense human pose estimation in the wild.  arXiv, 2018.
  * [38] Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li.  PIFu: Pixel-aligned implicit function for high-resolution clothed human digitization.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
  * [39] Shunsuke Saito, Tomas Simon, Jason Saragih, and Hanbyul Joo.  PIFuHD: Multi-level pixel-aligned implicit function for high-resolution 3d human digitization.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
  * [40] Kripasindhu Sarkar, Dushyant Mehta, Weipeng Xu, Vladislav Golyanik, and Christian Theobalt.  Neural re-rendering of humans from a single image.  In European Conference on Computer Vision, 2020.
  * [41] A. Tewari, O. Fried, J. Thies, V. Sitzmann, S. Lombardi, K. Sunkavalli, R. Martin-Brualla, T. Simon, J. Saragih, M. Niebner, R. Pandey, S. Fanello, G. Wetzstein, J.-Y. Zhu, C. Theobalt, M. Agrawala, E. Shechtman, D. B Goldman, and M. Zollhofer.  State of the art on neural rendering.  Computer Graphics Forum, 39(2):701–727, 2020.
  * [42] Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly.  Towards accurate generative models of video: A new metric & challenges, 2019.
  * [43] Jingbo Wang, Sijie Yan, Yuanjun Xiong, and Dahua Lin.  Motion guided 3d pose estimation from videos.  In European Conference on Computer Vision, 2020.
  * [44] Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, and Bryan Catanzaro.  Video-to-video synthesis.  In NIPS, 2018.
  * [45] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli.  Image quality assessment: from error visibility to structural similarity.  IEEE Transactions on Image Processing, 13(4):600–612, 2004.
  * [46] Minye Wu, Yuehao Wang, Qiang Hu, and Jingyi Yu.  Multi-view neural human rendering.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
  * [47] Wayne Wu, Yunxuan Zhang, Cheng Li, Chen Qian, and Chen Change Loy.  Reenactgan: Learning to reenact faces via boundary transfer.  In European Conference on Computer Vision, 2018.
  * [48] Zijian Huang Qifeng Chen Xuanchi Ren, Haoran Li.  Self-supervised dance video synthesis conditioned on music.  In ACM MM, 2020.
  * [49] Sijie Yan, Zhizhong Li, Yuanjun Xiong, Huahan Yan, and Dahua Lin.  Convolutional sequence generation for skeleton-based action synthesis.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
  * [50] Sijie Yan, Yuanjun Xiong, and Dahua Lin.  Spatial temporal graph convolutional networks for skeleton-based action recognition.  In AAAI Conference on Artificial Intelligence, 2018.
  * [51] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In IEEE Conference on Computer Vision and Pattern Recognition, 2018.
  * [52] Yuxuan Zhang, Wenzheng Chen, Huan Ling, Jun Gao, Yinan Zhang, Antonio Torralba, and Sanja Fidler.  Image GANs meet differentiable rendering for inverse graphics and interpretable 3D neural rendering.  arXiv preprint arXiv:2010.09125, 2020.
  * [53] Long Zhao, Xi Peng, Yu Tian, Mubbasir Kapadia, and Dimitris N Metaxas.  Semantic graph convolutional networks for 3d human pose regression.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
  * [54] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In IEEE International Conference on Computer Vision, 2017.


#  On graviton propagation in curved space-time background

E.V. Arbuzovaa,b, A.D. Dolgovb,c, L. A. Panasenkob

aDepartment of Higher Mathematics, Dubna State University,

Universitetskaya ul. 19, Dubna 141983, Russia

bDepartment of Physics, Novosibirsk State University,

Pirogova 2, Novosibirsk 630090, Russia

cBogolyubov Laboratory of Theoretical Physics, Joint Institute for Nuclear
Research, Joliot-Curie st. 6, Dubna, Moscow region, 141980 Russia

Equation describing propagation of gravitational waves (GW) over arbitrary
curved space-time background is analyzed. New terms, which are absent in the
conventional homogeneous and isotropic Friedmann cosmology, are found. Some
examples of realistic metric, where these new terms manifest themselves, are
presented. Possible implications to very low frequency GW are briefly
discussed.

##  1 Introduction

Gravitational wave (GW) propagation over Minkowski and curved backgrounds was
considered in vast details in the literature, see e.g. textbooks [1, 2, 3, 4,
5, 6]. However, in cosmological situation this consideration is confined to
the conformally flat Friedmann-Le’Maitre-Robertson-Walker (FLRW) spacetime. In
the present work we lift this limitation and derive equation of motion of
gravitational waves in an arbitrary spacetime metric. We show that in the
equation for GW propagation over arbitrary space-time there appear additional
terms which are absent in the FLRW case.

Formally in the classical book [1] the expansion of the proper curvature
tensors up to the first order over small tensor perturbations $h_{\mu\nu}$ is
presented in an arbitrary background metric, see equation 108.4. However, the
equatoin is immediately reduced to the empty space with vanishing Ricci
tensor, $R_{\mu\nu}=0$, in order to obtain the canonical wave equation
$D^{2}h_{\mu\nu}=0$, equation 108.7 of this reference. Nevertheless in ref.
[1] it is explicitly stated that the non-zero Ricci tensor would modify eq.
108.7 and may be of interest.

The corresponding section 1.5 of ref. [4] is the exact replica of the book
[1], see equation 1.172. This equation is also applied to the empty space with
vanishing energy-momentum tensor $T_{\mu\nu}$. Quoting ref. [4]: “Outside the
matter sources $T_{\mu\nu}=0$ … tells us the $R_{\mu\nu}=0$”. This immediately
excludes the equation 1.172 from application to cosmology, except for the
trivial case of asymptotically high frequency. All interesting effects related
to the FLRW metric and moreover to deviations from the FLRW space-time in this
case are washed out.

It is claimed in ref. [4] that it is impossible to fix the background metric
without introduction of the so called “Low” and “High” terms corresponding to
slow and fast varying quantities. This condition contradicts the main stream
in the literature on GW propagation over the FLRW space-time. The background
metric in the world accepted papers is simply taken in the FLRW form and it
does not lead to any problem.

Our paper is a minor (though technically more complicated) modification of the
description of the GW propagation in space-times which differ from the FLRW
one with examples of realistic background metric defined analogously to the
definition of the FLRW background. We have found that there exist some new
terms in the equations describing the GW propagation and mixing of the tensor
modes with the scalar ones which is absent in the FLRW case.

It is important to note that for the spacetime, which differs from the FLRW
one, it is impossible to impose the standard gauge conditions valid for
Minkowski and FLRW metrics. Because of this complication one can not separate
propagation of purely tensor mode from scalar and/or vector ones. The fact
that there can exist mixing between scalar and tensor modes in the presence of
matter and strong anisotropy (like in the Bianchi type I space-time) or
inhomogeneity is well known. We thank the referee for this comment. However,
in this work we made more general derivation not specifying any concrete form
of metric.

As shown in textbook [2], graviton propagation in empty but curved space-time
is governed by the equation

$\displaystyle
D_{\alpha}D^{\alpha}h_{\mu\nu}-2R_{\alpha\mu\nu\beta}h^{\alpha\beta}=0,$ (1)

where $h_{\mu\nu}$ is the tensor perturbation of the total metric:

$\displaystyle\bar{g}_{\mu\nu}=g_{\mu\nu}+h_{\mu\nu},$ (2)

$g_{\mu\nu}$ is the background metric, covariant derivative $D_{\alpha}$ is
defined with respect to the background metric, and $R_{\alpha\mu\nu\beta}$ is
the Riemann tensor in the background space-time, which is supposed to be
nonzero, while the Ricci tensor vanishes, $R_{\mu\nu}=0$.

It was argued in paper [7] that the l.h.s. of Eq. (1) was not changed in the
case of the graviton propagation in the homogeneous and isotropic universe
described by the canonical FLRW-metric and with space-time filled by ideal
liquid. Both the background energy-momentum, $T_{\mu\nu}$, and the first order
correction to it, $T_{\mu\nu}^{(1)}$, as well as the Ricci tensor,
$R_{\mu\nu}$, are supposed to be non-zero. We expand the total energy-momentum
tensor $\overline{T}_{\mu\nu}$ as:

$\displaystyle\overline{T}_{\mu\nu}=T_{\mu\nu}+T_{\mu\nu}^{(1)},$ (3)

where $T_{\mu\nu}$ is the energy-momentum tensor of the background matter, and
$T_{\mu\nu}^{(1)}$ is that induced by perturbations.

The energy density of gravitational waves is of the second order in
$h_{\mu\nu}$ and is neglected, as are all other second order contributions. It
is usually assumed that $T_{\mu}^{\nu(1)}$ for the mixed components is zero.
This condition depends upon the type of matter perturbations and, as is known,
it can be fulfilled for the ideal fluid. Under this condition Eq. (1)
rewritten in terms of mixed components preserves the same form both in empty
and filled space.

There is a subtle point however. If one naively lifts or lowers indices in
$T_{\mu}^{\nu(1)}$ with the background metric, one has to conclude that
$T_{\mu\nu}^{(1)}=0$. If so, the equations for GW propagation in terms of
mixed and lower components would not be equivalent, for arbitrary and not only
for FLRW metric. Indeed, we show in what follows that if $T_{\mu}^{\nu(1)}=0$,
then $T_{\mu\nu}^{(1)}\neq 0$ and an account of this fact leads to equivalent
equations in terms of mixed and lower components in arbitrary metric, as shown
in this work.

On the other hand, if one assumes that $T_{\mu\nu}^{(1)}=0$, the resulting
equation in the filled space would significantly differ, even in FLRW space-
time, from the accepted in the literature canonical one written in terms of
the mixed components $h^{\mu}_{\nu}$, to say nothing about an arbitrary space-
time considered in this paper.

In the conventional approach to the description of the gravitational wave
propagation over FLRW background the following transversality and traceless
conditions on $h^{\mu}_{\nu}$ are imposed:

$\displaystyle D_{\mu}h^{\mu}_{\nu}=0,\,\,\,{\rm and}\,\,\,\,h_{\mu}^{\mu}=0.$
(4)

It is verified in numerous works that Eq. (1) allows to impose these
conditions in the curved Einstein (empty) space with $R_{\mu\nu}=0$. It is
also well known that the same is true also for the filled FLRW space-time.
However, as it is found in what follows, these conditions are violated in the
space-time with an arbitrary metric different from the FLRW one. Hence the
longitudinal or scalar modes of GW may propagate and in the general case
tensor and scalar modes are mixed. This situation is similar to the
propagation of the longitudinal mode of the electromagnetic wave in plasma and
the appearance of non-zero effective photon mass equal to the plasma
frequency.

This statement is criticised by one of the referees of this work, who quoted
Ref. [8] as that where it is proven that no extra degrees of freedom may
appear, namely:

> ”the authors completely misrepresent Ref. [8] as a main point; that paper is
> so simple as to be obviously correct. Nowhere is any new DoF suggested for
> any case there–quite the opposite.”

In Ref. [8] a very special case of Ricci-flat metric (i.e. spaces with
$R_{\mu\nu}=0$) is considered. So there is not much sense to conclude that our
paper is wrong because it disagrees with Ref. [8]. On the opposite, in the
case of the Ricci flat metric with $R_{\mu\nu}=0$ our results completely agree
with Ref. [8]. Moreover we also fully agree with the numerous existing
literature treating GW propagation in FLRW space-time with $R_{\mu\nu}\neq 0$,
to which the results of Ref. [8] are not applicable.

Moreover, the basic Eq. (2) of Ref. [8] contradicts the accepted canonical
equation for GW propagation in FLRW metric if we formally take non-zero Ricci
tensor. It does not mean of course that all world literature contains ”some
obvious errors” quoting this referee. We repeat that we derived equation for
GW propagation in arbitrary space-time. In the limit of FLRW metric (note that
in this metric $R_{\mu\nu}\neq 0$ ) it coincides with the numerous published
works and in the Ricci-flat case it agrees with eq. (2) of Ref. [8]. So the
conclusion of the referee that our paper is incorrect is erroneous.

The noticed above an apparent inconsistency between equations describing GW
propagation in terms of $h_{\mu\nu}$ and $h_{\nu}^{\mu}$ is related to the
fact that $T^{(1)}_{\mu\nu}\neq g_{\mu\alpha}T^{(1)\alpha}_{\nu}$. Indeed

$\displaystyle\bar{T}_{\mu\nu}=\bar{g}_{\alpha\nu}\bar{T}_{\mu}^{\alpha}=\left(g_{\alpha\nu}+h_{\alpha\nu}\right)\left(T^{\alpha}_{\mu}+T^{(1)\alpha}_{\nu}\right),$
(5)

so
$T_{\mu\nu}^{(1)}=h_{\alpha\nu}T^{\alpha}_{\mu}+g_{\mu\alpha}T^{(1)\alpha}_{\nu}$.
The addition of this term into the r.h.s of Eq. (1) allows to impose the
transversality condition on this equation, so the extra propagating modes are
not excited over FLRW background in contrast to the concern expressed in Ref.
[8]. For more detail see the discussion below Eq. (28).

In textbook [6] the detailed derivation of the equation describing GW
propagation over FLRW background, which is fully equivalent to Eq. (1), is
presented for the background interval rewritten in terms of conformal time
$\tau$:

$\displaystyle
ds^{2}=a^{2}(\tau)\left(d\tau^{2}-\delta_{ij}dx^{i}dx^{j}\right)$ (6)

under assumption that $a(\tau)$ is a scalar with respect to coordinate
transformation. Here $\delta_{ij}$ is the Kronecker symbol. The transition to
conformal time is convenient to make for conformally flat FLRW metric, but in
the case of an arbitrary space-time this transition is not particularly useful
because a metric, which is not conformally flat, cannot be transformed to that
proportional to the Minkowski metric.

In this paper we generalise Eq. (1) for graviton propagation over arbitrary
space-time background. We show that in the l.h.s. of this equation there
appear additional terms, which vanish in FLRW metric and which can dominate in
the low frequency limit.

Remind that the study of metric perturbations was pioneered in ref. [9], see
also [10], where it is shown that $h_{\mu\nu}$ could be separated into three
types according to their properties with respect to 3-dimensional space
transformation (rotation): scalar ($h_{t}^{t}$ and $h^{i}_{i})$, vector
($h_{t}^{i}$, $\partial_{i}h^{i}_{t}=0$), and tensor ($h_{i}^{j}$). The
coordinate freedom allows to impose the following conditions on $h_{i}^{j}$:

$\displaystyle\partial_{i}h^{i}_{j}=0,\,\,\,h_{i}^{i}=0,$ (7)

in locally Minkowsky frame. These conditions can be imposed also globally in
conformally flat FLRW metric. So the tensor mode $h_{i}^{j}$ has only two
independent components describing massless quanta propagation with two, as it
should be, transverse helicity states, i.e. gravitational waves. However, in
general space-time this might be incorrect and additional propagating degrees
of freedom could be excited. The problem of gauge fixing in arbitrary space-
time is considered in Sec. 2.

Usually these perturbations are considered either over vacuum solutions of the
Einstein equations or over background FLRW space-time with the metric:

$\displaystyle ds^{2}_{F}=dt^{2}-a^{2}(t)\delta_{ij}dx^{i}dx^{j},$ (8)

In what follows we assume for simplicity that the FLRW metric is spatially
flat. Transition from this metric to conformal one (6) is given by the linear
transformation between time components of the coordinates, $dt=a(\tau)d\tau$.

Gravitational wave production and propagation in FLRW-cosmology was studied in
Refs. [7, 11, 12, 13]. According to the Parker theorem, see Ref. [14, 15],
massless particles production by conformally flat space-time, metric, such as,
in particular, FLRW metric is forbidden if the corresponding field equations
are conformally invariant. This is true for massless fermions, conformally
coupled scalars, and massless vector fields, up to possible breaking of
conformal invariance by the trace anomaly [16]. It was discovered by L.
Grishchuk that gravitons can be produced in conformally flat space-time since
their equation of motion is not conformally invariant [7]. Gravitational waves
could be efficiently produced during cosmological inflation [12, 13]. They may
be observed by the cosmic interferometer LISA which will presumably reveal
information about the mechanism of the primordial inflation.

The paper is organised as follows. Sec. 2 is devoted to the choice of gauge
for four-dimensional tensor modes in arbitrary space-time and comparison that
with the popular gauges in FLRW metric. In Sec. 3 we derive equations
describing propagation of gravitational waves in an arbitrary spacetime making
expansion of the exact Einstein equations up to the first order metric
perturbations $h_{\mu\nu}$. In Sec. 4 the problems with the first order
corrections to the energy-momentum tensor are discussed. Further, in Sec. 5
the equation for the propagation of the mixed components, $h^{\mu}_{\nu}$, is
derived. Section 6 is devoted to the proof that the transversality condition
$D_{\mu}(h^{\mu}_{\nu}-\delta^{\mu}_{\nu}h^{\alpha}_{\alpha}/2)=0$ is
compatible with the equation of GW propagation and so it can be imposed in
arbitrary metric. In Sec. 7 a few examples of realistic metrics, different
from FLRW one, are presented. Lastly, in Sec. 8 possible implications of our
results to low frequency GWs are briefly discussed.

##  2 Choice of gauge

To avoid confusion expressed by one of the referees we stress again that below
we consider an arbitrary space-time metric but not only the FLRW one. In the
latter case the choice of gauge is heavily based on particular properties of
FLRW metric.

We discuss the choice of gauge conditions which can be imposed on $h_{\mu\nu}$
following the lines presented in the textbook [1]. Let us make the coordinate
transformation $\tilde{x}^{\mu}=x^{\mu}+\xi^{(1)\mu}$, where $\xi^{(1)\mu}$ is
a small vector. Under such transformation the first order perturbations,
$h_{\mu\nu}$, of the metric (2) changes as:

$\displaystyle\tilde{h}_{\mu\nu}=h_{\mu\nu}-D_{\mu}\xi^{(1)}_{\nu}-D_{\nu}\xi^{(1)}_{\mu}.$
(9)

Using freedom in the choice of four functions $\xi^{(1)\mu}$ we can impose the
following four conditions:

$\displaystyle D_{\mu}\psi^{\mu}_{\nu}=0,\ \ \ $ (10)

where $\psi^{\mu}_{\nu}=h^{\mu}_{\nu}-\delta^{\mu}_{\nu}\,h/2$ and
$h=h^{\alpha}_{\alpha}$. In the flat spacetime case condition (10) leads to
the wave equation in the classical form, $D^{2}h^{\mu}_{\nu}=0$. The same
equation is true the high frequency (eikonal) limit.

There remains freedom in the coordinate transformation
$\tilde{h}^{\mu}_{\nu}=h^{\mu}_{\nu}-D^{\mu}\xi^{(2)}_{\nu}-D_{\nu}\xi^{(2)\mu}$
with the new parameters $\xi^{(2)\mu}$ which do not violate condition (10).
Accordingly parameters $\xi_{\mu}^{(2)}$ should satisfy the equation:

$\displaystyle D^{2}\xi^{(2)}_{\mu}+R_{\mu}^{\nu}\xi^{(2)}_{\nu}=0.$ (11)

Thus we are allowed to use four more functions $\xi_{\mu}^{(2)}$ to fix the
gauge.

This freedom was used in the textbook [1] to impose the restrictions
$h_{ti}=0$ and $h=0$, where $i$ is the space index. We, however, apply this
freedom to demand $h_{t\alpha}=0$ for any $\alpha$. In this case the condition
$h=0$ may be invalid.

There is still some freedom to make the coordinate transformation with
parameter $\xi^{(3)\mu}$, which, in addition to (11), satisfies the condition:

$\displaystyle D_{\mu}\xi^{(3)\mu}=0.$ (12)

Evidently, the transformation with the functions $\xi^{(3)\mu}$ does not
change the value of $h$.

Detailed discussion of different types of perturbations (scalar, vector, and
tensor) can be found e.g. in the book by Mukhanov [3]. However, all that was
done there only for the FLRW spacetime.

##  3 Basic Equations

We start from the exact Einstein equations:

$\displaystyle\bar{R}_{\mu\nu}-\frac{1}{2}\bar{g}_{\mu\nu}\bar{R}={\overline{T}_{\mu\nu}}.$
(13)

Here and in the rest of the paper $\bar{T}_{\mu\nu}$ and $T_{\mu\nu}$ are
related to the physical energy-momentum tensor through the constant factor as:

$\displaystyle T_{\mu\nu}=\frac{8\pi}{m_{Pl}^{2}}T^{(phys)}_{\mu\nu}.$ (14)

Overline means that the corresponding exact (total) quantities are calculated
in terms of the total metric $\bar{g}_{\mu\nu}$, see Eqs. (2) and (3). We will
consider first-order tensor perturbations $h_{\mu\nu}$ over background metric
$g_{\mu\nu}$ and expand the total Ricci tensor and energy-momentum tensor as

$\displaystyle\bar{R}_{\mu\nu}=R_{\mu\nu}+R_{\mu\nu}^{(1)},\ \ \ \
{\overline{T}_{\mu\nu}}={T}_{\mu\nu}+{T}_{\mu\nu}^{(1)},$ (15)

assuming that all background quantities are taken in the background metric
$g_{\mu\nu}$.

Our goal is to derive the first-order perturbation equation governing
evolution of $h_{\mu\nu}$ without any assumptions about the form of the
background metric.

From the condition

$\displaystyle\bar{g}_{\mu\alpha}\bar{g}^{\nu\alpha}=\delta_{\mu}^{\nu}$ (16)

it follows that:

$\displaystyle\bar{g}^{\mu\nu}=g^{\mu\nu}-h^{\mu\nu}.$ (17)

For the Ricci scalar we have:

$\displaystyle\bar{R}=\bar{g}^{\alpha\beta}\bar{R}_{\alpha\beta}=\left(g^{\alpha\beta}-h^{\alpha\beta}\right)\left(R_{\alpha\beta}+R_{\alpha\beta}^{(1)}\right)=g^{\alpha\beta}R_{\alpha\beta}-h^{\alpha\beta}R_{\alpha\beta}+g^{\alpha\beta}R_{\alpha\beta}^{(1)}\,.$
(18)

Let us note, that the first order correction to the curvature scalar is not
simply obtained from the first order correction to the Ricci tensor through
the contraction of indices by the background metric, but contains an extra
term:

$\displaystyle
R^{(1)}=g^{\alpha\beta}R_{\alpha\beta}^{(1)}-h^{\alpha\beta}R_{\alpha\beta}.$
(19)

Following the book by Landau and Lifshitz [1] we express the perturbation of
the Ricci tensor, $R_{\mu\nu}^{(1)}$, via metric perturbations, $h_{\mu\nu}$,
as

$\displaystyle
R_{\mu\nu}^{(1)}=\frac{1}{2}\left(D_{\alpha}D_{\nu}h^{\alpha}_{\mu}+D_{\alpha}D_{\mu}h^{\alpha}_{\nu}-D_{\alpha}D^{\alpha}h_{\mu\nu}-D_{\mu}D_{\nu}h\right),$
(20)

where covariant derivatives $D_{\beta}$ are taken with respect to the
background metric $g_{\mu\nu}$ and $D^{\alpha}=g^{\alpha\beta}D_{\beta}$. Here
and below we use the background metric, $g^{\mu\nu}$ and $g_{\mu\nu}$, to move
indices up and down.

According to Eq. (10) $h^{\mu}_{\nu}$ satisfies the condition:

$\displaystyle D_{\mu}h^{\mu}_{\nu}=\frac{1}{2}\partial_{\nu}h.$ (21)

Using the commutation rules of covariant derivatives we arrive to the result:

$\displaystyle
R_{\mu\nu}^{(1)}=-\frac{1}{2}D_{\alpha}D^{\alpha}h_{\mu\nu}+h^{\alpha\beta}R_{\alpha\mu\nu\beta}+\frac{1}{2}\left(h_{\alpha\mu}R^{\alpha}_{\nu}+h_{\alpha\nu}R^{\alpha}_{\mu}\right).$
(22)

Substituting Eqs. (22) and (18) into Eq. (13) and keeping only the first-order
quantities we obtain the following equation for tensor perturbations of the
metric:

$\displaystyle
D_{\alpha}D^{\alpha}h_{\mu\nu}-2h^{\alpha\beta}R_{\alpha\mu\nu\beta}-\left(h_{\alpha\mu}R^{\alpha}_{\nu}+h_{\alpha\nu}R^{\alpha}_{\mu}\right)+h_{\mu\nu}R-g_{\mu\nu}h^{\alpha\beta}R_{\alpha\beta}{-\frac{1}{2}g_{\mu\nu}D^{2}h}$
$\displaystyle=-2\,{T}_{\mu\nu}^{(1)}.$ (23)

Taking trace we arrive to the equation:

$\displaystyle
D^{2}h+4h^{\alpha\beta}R_{\alpha\beta}-hR=2{g}^{\mu\nu}{T}_{\mu\nu}^{(1)}.$
(24)

In general case $h^{\alpha\beta}R_{\alpha\beta}\neq 0$, so one can conclude
that $h\neq 0$. On the other hand in FLRW spacetime the condition
$h^{\alpha\beta}R_{\alpha\beta}=0$ is fulfilled and thus equations of motion
(23) and (24) do not contradict the condition $h=0$, if the source
${T}_{\mu\nu}^{(1)}$ is traceless.

Equation (23) coincides with the conventional equation (1) in empty space,
where $R_{\mu\nu}=0$, but essentially differs from Eq. (1) in filled space
even in the Friedmann space-time. Indeed, in FLRW space-time equation (23)
takes the form:

$\displaystyle\left(\partial_{t}^{2}-\frac{1}{a^{2}}\,\partial^{2}_{k}\right)\,h_{ij}-H\partial_{t}h_{ij}-2\left(H^{2}+3\,\frac{\ddot{a}}{a}\right)h_{ij}=-2\,{T}_{ij}^{(1)},$
(25)

while the conventional Eq. (1) with an account of the source term, which we
denote as ${T}_{\mu\nu}^{(1c)}$, becomes

$\displaystyle\left(\partial_{t}^{2}-\frac{1}{a^{2}}\,\partial^{2}_{k}\right)\,h_{ij}-H\partial_{t}h_{ij}-2\,\frac{\ddot{a}}{a}h_{ij}=-2\,{T}_{ij}^{(1c)}.$
(26)

Surprisingly both equations are true and the resolution of the evident
inconsistency is hidden in a difference between ${T}_{\mu\nu}^{(1)}$ and
${T}_{\mu\nu}^{(1c)}$, which according to Eq. (5) is

$\displaystyle
T_{\mu\nu}^{(1c)}=h_{\alpha\nu}T^{\alpha}_{\mu}+g_{\mu\alpha}T^{(1)\alpha}_{\nu}=h_{\alpha\nu}R^{\alpha}_{\mu}+g_{\mu\alpha}T^{(1)\alpha}_{\nu}.$
(27)

Keeping in mind that propagation of GWs in FLRW background is described under
assumption that $T^{(1)\mu}_{\nu}=0$, we can check that the condition
$h_{\mu}^{\mu}=0$ is fulfilled in this background metric, simply because
$h_{\mu\nu}R^{\mu\nu}=0$.

##  4 First order corrections to energy-momentum tensor

The only possible resolution of the discrepancy between Eqs. (25) and (26)
lays in the difference between ${T}_{\mu\nu}^{(1)}$ and ${T}_{\mu\nu}^{(1c)}$.
The standard derivation of the conventional Eqs. (1) or (26) is heavily based
on the condition ${T}_{\mu}^{(1c)\nu}=0$, which is incompatible with seemingly
equivalent condition ${T}_{\mu\nu}^{(1)}=0$. Indeed let us assume, that the
first order corrections to the energy-momentum tensor with mixed components
are zero, $T_{\mu}^{(1)\nu}=0$, as it is usually taken in the case of FLRW
space-time filled by the ideal fluid. Then, if we consider equation for
$h_{\mu\nu}$, we should put down indices using the full metric tensor
$\bar{g}_{\mu\nu}$ and so:

$\displaystyle\bar{T}_{\mu\nu}=\bar{g}_{\mu\alpha}\bar{T}^{\alpha}_{\nu}=T_{\mu\nu}+h_{\mu\alpha}T^{\alpha}_{\nu}{+g_{\mu\alpha}T^{(1)\alpha}_{\nu}}=T_{\mu\nu}+h_{\mu\alpha}T^{\alpha}_{\nu}.$
(28)

The last equality is true, if $T_{\mu}^{(1)\nu}=0$.

Hence

$\displaystyle
T^{(1)}_{\mu\nu}=h_{\mu\alpha}T^{\alpha}_{\nu}=(1/2)\left(h_{\mu\alpha}R^{\alpha}_{\nu}+h_{\nu\alpha}R^{\alpha}_{\mu}-h_{\mu\nu}R\right),$
(29)

where the Einstein equation for the background curvature is used:

$\displaystyle R_{\mu\nu}-g_{\mu\nu}R/2=T_{\mu\nu}.$ (30)

Substituting expression (29) for $T^{(1)}_{\mu\nu}$ we obtain from Eq. (23)

$\displaystyle
D_{\alpha}D^{\alpha}h_{\mu\nu}-2h^{\alpha\beta}R_{\alpha\mu\nu\beta}-g_{\mu\nu}h^{\alpha\beta}R_{\alpha\beta}{-\frac{1}{2}g_{\mu\nu}D^{2}h}=0.$
(31)

Since the last two terms in this equation vanish in the FLRW background we
arrive to the canonical equation (1).

On the other hand, canonical Eq. (1) or Eq. (26), which follows from Eq. (1)
for FLRW metric, is obtained originally from the mixed component equation for
$h_{\mu}^{\nu}$. In this case the indices are put down with the background
metric $g_{\mu\nu}$ and thus starting from $T^{\nu(1)}_{\mu}=0$ we arrive to
$T^{(1c)}_{\mu\nu}=0$, so both ways ultimately lead to the same result.

Let us note that the condition $T^{\nu(1)}_{\mu}=0$ is not necessarily true
and there are some realistic cases when it is not fulfilled. For example,
$T^{\nu(1)}_{\mu}\neq 0$ in the equation describing graviton-to-photon
transition in external magnetic field even over the Minkowsky background, see
e.g. [19]. Another known example is presented by anisotropic stresses which
could be induced e.g. by neutrinos and photons [20, 21]. They all are treated
perturbatively, so the background remains the FLRW one.

Let us stress that if the background metric deviates from the Friedmann one,
the last two terms in Eq. (31) may essentially change the character of
solutions, especially the first of them since it does not vanish in zero
frequency limit.

##  5 Equation for the mixed components

To make the paper self-contained let us now derive equation for the mixed
components, $h^{\mu}_{\nu}$. To this end we start from the equation:

$\displaystyle\bar{R}^{\mu}_{\nu}-\delta^{\mu}_{\nu}\bar{R}/2={\overline{T}^{\mu}_{\nu}}$
(32)

and decompose it up to the first order:

$\displaystyle\bar{R}^{\mu}_{\nu}=R^{\mu}_{\nu}+R^{\mu(1)}_{\nu}$ (33)

The first correction $R^{(1)}_{\mu\nu}$ is calculated in the book of Landau
and Lifshitz [1] and presented here in Eqs. (20) and (22). The indices are
raised according to:

$\displaystyle\bar{R}_{\nu}^{\mu}=\bar{g}^{\beta\mu}\bar{R}_{\nu\beta}=\left(g^{\beta\mu}-h^{\beta\mu}\right)\left(R_{\nu\beta}+R^{(1)}_{\nu\beta}\right)=R_{\nu}^{\mu}-h^{\mu\beta}R_{\nu\beta}+g^{\mu\beta}R^{(1)}_{\nu\beta}.$
(34)

Analogously;

$\displaystyle\bar{R}=\delta^{\alpha}_{\beta}\bar{R}^{\beta}_{\alpha}=R-h^{\mu\beta}R_{\mu\beta}+g^{\mu\beta}R^{(1)}_{\mu\beta}.$
(35)

Finally in the first order we obtain:

$\displaystyle
g^{\mu\beta}R^{(1)}_{\nu\beta}-h^{\mu\beta}R_{\nu\beta}+\frac{1}{2}\delta^{\mu}_{\nu}h^{\alpha\beta}R_{\alpha\beta}-\frac{1}{2}\delta^{\mu}_{\nu}g^{\alpha\beta}R^{(1)}_{\alpha\beta}=0,$
(36)

where according to the discussion in the previous section we took
$\widetilde{T}^{\mu(1)}_{\nu}=0$.

In FLRW background the product $g^{\alpha\beta}R^{(1)}_{\alpha\beta}$ vanishes
due to the conditions $D_{\alpha}h^{\alpha}_{\beta}=0$ and $h^{\mu}_{\mu}=0$,
and $h^{\alpha\beta}R_{\alpha\beta}=0$, since $R_{tj}=0$,
$R_{ij}\sim\delta_{ij}$, $h^{t\mu}=0$ and $h_{i}^{i}=0$. On the other hand,
both these products are generally nonzero if the background deviates from the
FLRW one. According to Eq. (22)
$g^{\alpha\beta}R^{(1)}_{\alpha\beta}=-D^{2}h/2$. As for
$h^{\alpha\beta}R_{\alpha\beta}$, it is generally non vanishing for an
arbitrary taken Ricci tensor.

Taking expression (22) for $R^{(1)}_{\nu\beta}$ and using the condition that
in FLRW background $R^{i}_{j}\sim\delta^{i}_{j}$, we conclude that the last
term in Eq. (22) is equal to $h^{\mu\beta}R_{\nu\beta}$ and cancels out with
the second term in the l.h.s. of Eq. (36). Finally, we obtain the conventional
equation for the mixed components:

$\displaystyle
D_{\alpha}D^{\alpha}h^{\mu}_{\nu}-2h_{\alpha}^{\beta}R^{\mu\alpha}_{\,\,\,\,\,\,\,\beta\nu}=0,$
(37)

where the indices are lifted with the background metric tensor $g^{\mu\nu}$.

This equation in FLRW background turns into:

$\displaystyle\left(\partial_{t}^{2}-\frac{\Delta}{a^{2}}+3H\partial_{t}\right)\,h_{\nu}^{\mu}=0.$
(38)

Making transformation to $h_{\mu\nu}$ according to

$\displaystyle
h^{\mu}_{\nu}=g^{\mu\alpha}h_{\alpha\nu}=-h_{\alpha\nu}\delta^{\mu\alpha}/a^{2}$
(39)

we arrive at

$\displaystyle\left(\partial_{t}^{2}-\frac{\Delta}{a^{2}}-H\partial_{t}-\frac{2\ddot{a}}{a}\right)\,h_{\nu\mu}=0.$
(40)

It coincides with the conventional Eq. (26) but not with Eq. (25) if one
assumes that the right hand sides of both equations vanish. However, we have
shown that taking $T^{\mu(1)}_{\nu}=0$ in FLRW metric for mixed components
leads to $T^{(1)}_{\mu\nu}\neq 0$ with lower indices.

It may be instructive to present equations in terms of conformal time, since
they are frequently analyzed this way. However, if the metric is not
conformally flat, transition to conformal time generally does not make much
sense.

##  6 Verification of the transversality conditions for non-zero
$h_{\mu}^{\mu}$

It is shown here that the action of the covariant divergence $D_{\mu}$ on both
sides of the equation for $h^{\mu}_{\nu}$ gives self-consistent results. For
easier reading we repeat here some equations already presented above.

We start from the exact equations

$\displaystyle\overline{R}^{\mu}_{\nu}-\frac{1}{2}\delta^{\mu}_{\nu}\overline{R}=\overline{{T}}_{\nu}^{\mu}$
(41)

and expand $\overline{{T}}_{\nu}^{\mu}=T^{\mu}_{\nu}+T^{(1)\mu}_{\nu}$ and
$\overline{{R}}_{\nu}^{\mu}=R^{\mu}_{\nu}+R^{(1)\mu}_{\nu}$. So we obtain: for
the background metric, as expected:

$\displaystyle{R}^{\mu}_{\nu}-\frac{1}{2}\delta^{\mu}_{\nu}{R}={{T}}_{\nu}^{\mu}.$
(42)

However, we use the first order correction to $R_{\mu\nu}$ as is presented in
Ref. [1]:

$\displaystyle
R_{\alpha\nu}^{(1)}=(1/2)\left(D_{\beta}D_{\nu}h^{\beta}_{\alpha}+D_{\beta}D_{\alpha}h^{\beta}_{\nu}-D^{2}h_{\alpha\nu}-D_{\alpha}D_{\nu}h\right),$
(43)

so we rise one lower index using ${g}^{\mu\alpha}$ and find:

$\displaystyle
D^{2}h^{\mu}_{\nu}+2R^{\mu\alpha}_{.\,.\,\,\nu\beta}h^{\beta}_{\alpha}+R_{\nu}^{\alpha}h^{\mu}_{\alpha}-R^{\mu}_{\alpha}h_{\nu}^{\alpha}-\delta^{\mu}_{\nu}\left(R^{\alpha}_{\beta}h_{\alpha}^{\beta}+\frac{1}{2}D^{2}h\right)=-2T^{(1)\mu}_{\nu},$
(44)

where $D^{2}=g^{\alpha\beta}D_{\alpha}D_{\beta}$ and it is taken into account
that $h=h^{\alpha}_{\alpha}\neq 0$ and
$D_{\mu}h^{\mu}_{\nu}=\partial_{\nu}h/2$.

Acting by $D_{\mu}$ on the first term and applying commutation rules for the
covariant derivatives we come to:

$\displaystyle
D_{\mu}D^{2}h^{\mu}_{\nu}=D_{\mu}\left(R_{\sigma}^{\mu}h^{\sigma}_{\nu}\right)-D_{\mu}\left(R^{\sigma\,.\,.\,\mu}_{\,\,\nu\alpha}h^{\alpha}_{\sigma}\right)-R^{\sigma.\,.\,\alpha}_{\,\,\nu\mu}D_{\alpha}h^{\mu}_{\sigma}{+\frac{1}{2}D^{2}(D_{\nu}h)}.$
(45)

Substituting Eq. (45) into the divergence of the l.h.s. of eq. (44) we come to
the following result

$\displaystyle{D_{\mu}\left(R_{\sigma}^{\mu}h^{\sigma}_{\nu}\right)-D_{\mu}\left(R^{\sigma\,.\,.\,\mu}_{\,\,\nu\alpha}h^{\alpha}_{\sigma}\right)-R^{\sigma.\,.\,\alpha}_{\,\,\nu\mu}D_{\alpha}h^{\mu}_{\sigma}{+\frac{1}{2}D^{2}(D_{\nu}h)}+2D_{\mu}\left(R^{\mu\sigma}_{.\,\,.\,\,\nu\alpha}h^{\alpha}_{\sigma}\right)+h^{\mu}_{\alpha}D_{\mu}R^{\alpha}_{\nu}}$
$\displaystyle{+\frac{1}{2}R^{\alpha}_{\nu}\,D_{\alpha}h-D_{\mu}\left(h^{\alpha}_{\nu}R^{\mu}_{\alpha}\right)-D_{\nu}\left(R^{\alpha}_{\beta}h_{\alpha}^{\beta}\right)-\frac{1}{2}D_{\nu}D^{2}h=-2D_{\mu}T^{(1)\mu}_{\nu}.}$
(46)

The first and the eighth terms in the l.h.s. of this equation are canceled,
the second and the fifth terms are reduced to
$D_{\mu}\left(R^{\mu\sigma}_{.\,\,.\,\,\nu\alpha}h^{\alpha}_{\sigma}\right)$
and we get:

$\displaystyle{D_{\mu}\left(R^{\mu\sigma}_{.\,\,.\,\,\nu\alpha}h^{\alpha}_{\sigma}\right)-R^{\sigma.\,.\,\mu}_{\,\,\nu\alpha}D_{\mu}h^{\alpha}_{\sigma}{+\frac{1}{2}D^{2}(D_{\nu}h)}+h^{\mu}_{\alpha}D_{\mu}R^{\alpha}_{\nu}}$
$\displaystyle{+\frac{1}{2}R^{\alpha}_{\nu}\,D_{\alpha}h-D_{\nu}\left(R^{\alpha}_{\beta}h_{\alpha}^{\beta}\right)-\frac{1}{2}D_{\nu}D^{2}h=-2D_{\mu}T^{(1)\mu}_{\nu}.}$
(47)

Using commutation relations of covariant derivatives we find:

$\displaystyle D^{2}D_{\nu}h-D_{\nu}D^{2}h=R^{\alpha}_{\nu}D_{\alpha}h.$ (48)

Eventually we arrive to

$\displaystyle{h^{\alpha}_{\sigma}D_{\mu}R^{\mu\sigma}_{.\,\,.\,\,\nu\alpha}+h^{\mu}_{\alpha}D_{\mu}R^{\alpha}_{\nu}-D_{\nu}\left(R^{\alpha}_{\beta}h_{\alpha}^{\beta}\right)+R^{\mu}_{\nu}D_{\mu}h=-2D_{\mu}T^{(1)\mu}_{\nu}.}$
(49)

The first term here can be rewritten via the Bianchi identity as:

$\displaystyle
h^{\alpha}_{\sigma}D_{\mu}R^{\mu\sigma}_{.\,\,.\,\,\nu\alpha}=h^{\alpha}_{\sigma}\left(D_{\nu}R^{\sigma}_{\alpha}-D_{\alpha}R^{\sigma}_{\nu}\right)$
(50)

leading to:

$\displaystyle
h^{\alpha}_{\sigma}\left(D_{\nu}R^{\sigma}_{\alpha}-D_{\alpha}R^{\sigma}_{\nu}\right)+h^{\mu}_{\alpha}D_{\mu}R^{\alpha}_{\nu}-h_{\alpha}^{\beta}D_{\nu}R^{\alpha}_{\beta}-R^{\alpha}_{\beta}D_{\nu}h_{\alpha}^{\beta}+R^{\mu}_{\nu}D_{\mu}h=-2D_{\mu}T^{(1)\mu}_{\nu}.$
(51)

All the terms containing the derivatives of the Ricci tensor in the equation
above neatly cancel out and finally we come to the following equation to be
verified:

$\displaystyle
R^{\alpha}_{\beta}D_{\nu}h_{\alpha}^{\beta}-R^{\mu}_{\nu}D_{\mu}h=2D_{\mu}T^{(1)\mu}_{\nu}.$
(52)

Now we must check if the l.h.s. and r.h.s. of the Eq. (52) are equal. To this
end we will apply the following conservation conditions:

$\displaystyle\overline{D}_{\mu}\overline{T}^{\mu}_{\nu}=0\,\,\,{\rm
and}\,\,\,D_{\mu}T^{\mu}_{\nu}=0.$ (53)

So in the first perturbation order we obtain

$\displaystyle\overline{D}_{\mu}\overline{T}^{\mu}_{\nu}=D_{\mu}T^{\mu}_{\nu}+D_{\mu}T^{(1)\mu}_{\nu}+\Gamma^{(1)\mu}_{\,\,\,\,\,\alpha\mu}T^{\alpha}_{\nu}-\Gamma^{(1)\alpha}_{\,\,\,\,\,\nu\mu}T^{\mu}_{\alpha}.$
(54)

The third term in the r.h.s. of this equation is zero because the first-order
perturbation corrections to the Christoffel symbols has the following form

$\displaystyle\Gamma^{(1)\mu}_{\,\,\,\,\,\nu\alpha}=\frac{1}{2}\left(D_{\nu}h^{\mu}_{\alpha}+D_{\alpha}h^{\mu}_{\nu}-D^{\mu}h_{\nu\alpha}\right)$
(55)

and hence $\Gamma^{(1)\mu}_{\,\,\,\,\,\alpha\mu}=D_{\alpha}h/2$.

From Eqs. (54) and (55) we obtain

$\displaystyle
D_{\mu}{T}^{(1)\mu}_{\nu}=\Gamma^{(1)\alpha}_{\,\,\,\,\,\nu\mu}T^{\mu}_{\alpha}-\Gamma^{(1)\mu}_{\,\,\,\,\,\alpha\mu}T^{\alpha}_{\nu}=$
$\displaystyle\frac{1}{2}\left(D_{\nu}h^{\alpha}_{\mu}+D_{\mu}h^{\alpha}_{\nu}-D^{\alpha}h_{\nu\mu}\right)\left(R^{\mu}_{\alpha}-\frac{1}{2}\delta^{\mu}_{\alpha}R\right)-\frac{1}{2}\left(R^{\alpha}_{\nu}-\frac{1}{2}\delta^{\alpha}_{\nu}R\right)D_{\alpha}h.$
(56)

Here we have taken into account the Einstein equations for the background
metric (42).

After simple algebra we get:

$\displaystyle
D_{\mu}{T}^{(1)\mu}_{\nu}=\left(R^{\mu}_{\alpha}D_{\nu}h^{\alpha}_{\mu}-R^{\mu}_{\nu}D_{\mu}h\right)/2.$
(57)

Thus we see that both sides in equation (52) are equal. It means that
condition (10), $D_{\mu}\psi^{\mu}_{\nu}=0$, is compatible with Eq. (44).

Let us calculate the trace of Eq. (44). We will find:

$\displaystyle D^{2}h+2R^{\mu}_{\nu}h^{\nu}_{\mu}=2T^{(1)\mu}_{\mu}.$ (58)

Seemingly, this equation is inconsistent with Eq. (24). However, if we use the
relation between $T^{(1)\mu}_{\mu}$ and $g^{\mu\nu}T^{(1)}_{\mu\nu}$, which we
can obtain from Eq. (28), we come to:

$\displaystyle
T^{(1)\mu}_{\mu}=g^{\mu\nu}T^{(1)}_{\mu\nu}-h_{\mu}^{\nu}\left(R^{\mu}_{\nu}-\frac{1}{2}\delta^{\mu}_{\nu}R\right)$
(59)

and the consistency is restored.

In the FLRW background
$R^{\mu}_{\alpha}D_{\nu}h^{\alpha}_{\mu}=R^{\mu}_{\nu}h^{\nu}_{\mu}=0$ and the
usual condition $T^{(1)\mu}_{\nu}=0$ allows for $h=0$. However, in arbitrary
background it seems generally impossible to impose both conditions of
covariant conservation of the source and its zero trace.

##  7 Realistic metrics different from FLRW one

Interesting deviations of the cosmological metric from the Friedmann one is
induced by density perturbations over the classical FLRW background. They are
considered in detail in books [6, 3, 5, 17], see also Ref. [18]. Assume that
there exists a cloud of matter with energy density and pressure which are
different from the average cosmological ones. Generally speaking the cloud may
be anisotropic but the impact of the additional term
$h^{\alpha}_{\beta}R_{\alpha}^{\beta}$ (36) would manifest itself even for
isotropic distribution of matter in the cloud. So, for simplicity, we confine
ourselves to this case.

We choose Schwarzschild-like isotropic coordinates in which the metric takes
the form:

$\displaystyle ds^{2}=Adt^{2}-B\,\delta_{ij}\,dx^{i}dx^{j}\,,$ (60)

where the functions $A$ and $B$ depend upon $r$ and $t$. The corresponding
Christoffel symbols are:

$\displaystyle\Gamma^{t}_{tt}=\frac{\dot{A}}{2A}\,,\ \ \
\Gamma^{t}_{jt}=\frac{\partial_{j}A}{2A}\,,\ \ \
\Gamma^{j}_{tt}=\frac{\delta^{jk}\partial_{k}A}{2B}\,,\ \ \
\Gamma^{t}_{jk}=\frac{\delta_{jk}\dot{B}}{2A}\,,$
$\displaystyle\Gamma^{k}_{jt}=\frac{\delta^{k}_{j}\dot{B}}{2B}\,,\ \
\Gamma^{k}_{lj}=\frac{1}{2B}(\delta^{k}_{l}\partial_{j}B+\delta^{k}_{j}\partial_{l}B-\delta_{lj}\delta^{kn}\partial_{n}B)\,.$
(61)

The corresponding Ricci tensor is given by:

$\displaystyle R_{tt}$ $\displaystyle=$ $\displaystyle\frac{\Delta
A}{2B}-\frac{3\ddot{B}}{2B}+\frac{3\dot{B}^{2}}{4B^{2}}+\frac{3\dot{A}\dot{B}}{4AB}+\frac{\partial^{j}A\partial_{j}B}{4B^{2}}-\frac{\partial^{j}A\partial_{j}A}{4AB}\,,$
(62) $\displaystyle R_{tj}$ $\displaystyle=$
$\displaystyle-\frac{\partial_{j}\dot{B}}{B}+\frac{\dot{B}\partial_{j}B}{B^{2}}+\frac{\dot{B}\partial_{j}A}{2AB}\,,$
(63) $\displaystyle R_{ij}$ $\displaystyle=$
$\displaystyle\delta_{ij}\left(\frac{\ddot{B}}{2A}-\frac{\Delta
B}{2B}+\frac{\dot{B}^{2}}{4AB}-\frac{\dot{A}\dot{B}}{4A^{2}}-\frac{\partial^{k}A\partial_{k}B}{4AB}+\frac{\partial^{k}B\partial_{k}B}{4B^{2}}\right)$
(64) $\displaystyle-$
$\displaystyle\frac{\partial_{i}\partial_{j}A}{2A}-\frac{\partial_{i}\partial_{j}B}{2B}+\frac{\partial_{i}A\partial_{j}A}{4A^{2}}+\frac{3\partial_{i}B\partial_{j}B}{4B^{2}}+\frac{\partial_{i}A\partial_{j}B+\partial_{j}A\partial_{i}B}{4AB}\,.$

Here and in what follows the upper space indices are raised with the Kronecker
delta, $\partial^{j}A=\delta^{jk}\partial_{k}A$.

The space derivatives of an arbitrary function of $r$ are equal to:

$\displaystyle\partial_{i}f=\frac{x_{i}}{r}f^{\prime},\,\,\,\,\partial_{i}\partial_{i}f=\left(\frac{\delta_{ij}}{r}-\frac{x_{i}x_{j}}{r^{3}}\right)f^{\prime}+\frac{x_{i}x_{j}}{r^{2}}f^{\prime\prime},$
(65)

where prime means differentiation over $r$. Thus $R_{ij}$ contains some other
terms, except those proportional to $\delta_{ij}$, and consequently the
product $h^{\alpha}_{\beta}R_{\alpha}^{\beta}$ is generally nonvanishing.

Usually perturbations are small, so both $A$ and $B$ weakly deviate from unity
and linear in $A$ and $B$ terms dominate in $R_{ij}$. For a particular shape
of the perturbations $(A-1)\sim r^{2}$ and $(B-1)\sim r^{2}$, as is considered
in Ref. [18], and so $R_{ij}\sim\delta_{ij}$ and
$h^{\alpha}_{\beta}R_{\alpha}^{\beta}=0$. However, it is not the general case.
Moreover, perturbations are known to rise as the cosmological scale factor at
matter dominated regime, and so the Ricci tensor may become close by magnitude
to its background value or even exceed it.

There are some more physically interesting metrics for which the product
$h^{\alpha}_{\beta}R_{\alpha}^{\beta}$ is nonzero. One simple example is
presented by a collapse of dust-like matter, described e.g. in book [1]. This
is the so-called pressureless Tolman solution. More general Tolman solution
with non-zero pressure also possesses the same property.

##  8 Conclusion

We have found that in an arbitrary curved background it is generally
impossible to satisfy both standard conditions $D_{\mu}h^{\mu}_{\nu}=0$ and
$h^{\mu}_{\mu}=0$ on tensor perturbations. As we have shown the condition
$D_{\mu}\psi^{\mu}_{n}u=0$ (10) can be imposed in any space-time. Of course
tensor modes may propagate in any background but they would not be pure tensor
modes or pure transverse modes. According to the equations derived in this
work tensor and scalars modes are mixed and propagate together.

In some sense these phenomena resemble the propagation of electromagnetic
waves in plasma, when longitudinal modes are excited. Moreover, in
inhomogeneous and anisotropic plasma propagating modes may be even more
cumbersome. However, in high frequency limit (eikonal approximation) we return
to ”normality”.

The discovered in this paper additional term in the equation for gravitational
waves (GW) propagation in a background, which differs from the FLRW one, may
have an essential impact on the low frequency tail of GW spectrum in
particular, of GWs which had been produced during inflation, see Ref.[12, 13].
Their intensity at low frequencies might be noticeably suppressed. Because of
that, the strict limit on very long gravitational waves obtained from the CMB
polarization data [22] does not necessarily implies that the traditional
inflation induced by a scalar field, inflaton, [23] is excluded. A study of
this problem is in progress.

## Acknowledgement

We thank V.A. Rubakov for a very important comment and discussions.
The work was supported by RSF Grant 20-42-09010.

## References

  * [1] L.D. Landau, E.M. Lifshitz, The Classical Theory of Fields: Volume 2 (Pergamon press, New York, 1971).
  * [2] C.W. Misner, K.S. Thorne and J.A. Wheeler, Gravitation (W. H. Freeman and company, San Francisco, 1973).
  * [3] V. Mukhanov, Physical Foundations of Cosmology (Cambridge University Press, New York, 2005).
  * [4] M. Maggiore, Gravitational Waves (Oxford University Press, 2008).
  * [5] S. Weinberg, Cosmology (Oxford University Press, 2008).
  * [6] D.S. Gorbunov, V.A. Rubakov, Introduction to the theory of the early universe: Cosmological perturbations and inflationary theory (Hackensack, USA: World Scientific, 2011).
  * [7] L.P. Grishchuk, Amplification of gravitational waves in an isotropic universe, Zh. Eksp. Teor. Fiz. 67, 825 (1974); Sov.Phys.JETP 40, 409 (1975).
  * [8] S. Deser, M. Henneaux, A note on spin two fields in curved backgrounds Class.Quant.Grav. 24, 1683-1686 (2007) [e-Print: arXiv:gr-qc/0611157].
  * [9] E.M. Lifshitz, On the gravitational stability of the expanding universe, Zh. Eksp. Teor. Phys. 16, 587 (1946).
  * [10] E.M. Lifshitz, I.M. Khalatnikov, Problems of relativistic cosmology, Uspekhi Fizicheskikh Nauk 80, 391 (1963).
  * [11] A.V. Zakharov, Influence of collisionless particles on the growth of gravitational perturbations in an isotropic universe, Zh. Eksp. Teor. Fiz. 77, 434-450 (1979).
  * [12] A.A. Starobinsky, Spectrum of relict gravitational radiation and the early state of the universe, JETP Lett. 30, 682-685 (1979); Pisma Zh.Eksp.Teor.Fiz. 30, 719-723 (1979).
  * [13] V.A. Rubakov, M.V. Sazhin, A.V. Veryaskin, Graviton Creation in the Inflationary Universe and the Grand Unification Scale, Phys.Lett.B 115, 189-192 (1982).
  * [14] L. Parker, Particle Creation in Expanding Universes, Phys. Rev. Lett. 21, 562 (1968).
  * [15] L. Parker, Quantized Fields and Particle Creation in Expanding Universes, Phys. Rev. 183, 1057 (1969).
  * [16] A.D. Dolgov, Conformal Anomaly and the Production of Massless Particles by a Conformally Flat Metric, Zh. Eksp. Teor. Fiz. 81, 417 (1981); Sov. Phys. JETP 54, 223 (1981).
  * [17] C. Bambi, A.D. Dolgov, Introduction to Particle Cosmology (Springer, 2015).
  * [18] E.V. Arbuzova, A.D. Dolgov, L. Reverberi, Jeans Instability in Classical and Modified Gravity, Phys. Lett. B 739, 279-284 (2014)[e-Print: 1406.7104 [gr-qc]].
  * [19] A.D. Dolgov, D. Ejlli, Conversion of relic gravitational waves into photons in cosmological magnetic fields, JCAP 12 (2012) 003 • e-Print: 1211.0500 [gr-qc]
  * [20] S. Weinberg, Damping of Tensor Modes in Cosmology, Phys.Rev. D 69, 023503 (2003), astro-ph/0306304.
  * [21] S.Saga, K. Ichiki, N. Sugiyama, Impact of anisotropic stress of free-streaming particles on gravitational waves induced by cosmological density perturbations, Phys. Rev. D 91, 024030 (2015); arXiv:1412.1081 [astro-ph.CO]
  * [22] Planck Collaboration, “Planck 2018 results. VI. Cosmological parameters”, Astron. Astrophys. 641, A6 (2020)(2018) [arXiv:1807.06209].
  * [23] A.D. Linde, Inflationary cosmology, Phys. Repts. 333, 17 (2000).