# Flow Guided Transformable Bottleneck Networks for Motion Retargeting

Jian Ren Menglei Chai Oliver J. Woodford Kyle Olszewski Sergey Tulyakov
Snap Inc.
{jren, mchai, kolszewski<EMAIL_ADDRESS>Work done while at Snap Inc.

###### Abstract

Human motion retargeting aims to transfer the motion of one person in a
“driving” video or set of images to another person. Existing efforts leverage
a long training video from each target person to train a subject-specific
motion transfer model. However, the scalability of such methods is limited, as
each model can only generate videos for the given target subject, and such
training videos are labor-intensive to acquire and process. Few-shot motion
transfer techniques, which only require one or a few images from a target,
have recently drawn considerable attention. Methods addressing this task
generally use either 2D or explicit 3D representations to transfer motion, and
in doing so, sacrifice either accurate geometric modeling or the flexibility
of an end-to-end learned representation. Inspired by the Transformable
Bottleneck Network, which renders novel views and manipulations of _rigid_
objects, we propose an approach based on an implicit volumetric representation
of the image content, which can then be spatially manipulated using volumetric
flow fields. We address the challenging question of how to aggregate
information across different body poses, learning flow fields that allow for
combining content from the appropriate regions of input images of highly _non-
rigid_ human subjects performing complex motions into a single implicit
volumetric representation. This allows us to learn our 3D representation
solely from videos of moving people. Armed with both 3D object understanding
and end-to-end learned rendering, this categorically novel representation
delivers state-of-the-art image generation quality, as shown by our
quantitative and qualitative evaluations.

##  1 Introduction

Retargeting human body motion — transferring motion from a “driving” image or
video of one subject (the _source_) to another subject (the _target_), using
one or more reference images of the target subject in an arbitrary pose — has
received a great deal of attention in recent years, due to numerous practical
and entertaining applications in content generation [62, 8]. Such applications
include transferring sophisticated athletic techniques or dancing performances
to untrained celebrities for special effects for cinema and television;
creating amusing performances for one’s friends or acquaintances for sheer
entertainment; and creating plausible motion sequences from photos or videos
depicting famous and important political figures (including historical figures
who may no longer be alive to perform such actions) for the creation of
plausible full-body “deepfake” videos. However, retaining the target subject’s
identity while rendering them in novel, unseen poses is highly challenging,
and the state-of-the-art is still far from plausible.

Many approaches to this task learn to render a specific person [1, 7, 8, 13,
20, 26, 49, 64, 63, 66, 67, 73] conditioned on the desired pose. This requires
a large number of training frames of that person, and incurs substantial
training time that must be repeated per each new subject. By contrast, in the
few-shot setting, addressed in this work, only a few reference images of the
target are available, and video generation from those images should be fast (,
requiring no subject-specific training). To overcome the lack of data for a
given subject, many other techniques [4, 29, 32, 34, 40, 43, 44, 46] leverage
existing human body models [2, 22, 36] to construct an approximate
representation of the subject that can then be manipulated and rendered. While
the 3D nature of these representations often leads to improved performance
over their purely 2D counterparts [3, 38, 39, 51, 55], their explicit nature,
which faces the limitations of capturing salient details with standard human
body models, also leads to reduced modeling power and therefore fidelity.
Large variations in the clothing (, dresses or jackets that do not conform to
the body shape), body type, or hair of the source and target subjects, for
example, cannot easily be represented with standard models that only represent
the body itself.

In this work we attain more flexible and expressive modeling power by
exploiting a representation that allows for 3D modeling and manipulation, and
yet is fully implicit, it can be fully learned, even though we use no explicit
ground-truth 3D information such as meshes or voxel grids as supervision.
Recently, just such a representation, the Transformable Bottleneck Network
(TBN) [45], has been shown to produce excellent results on novel view
synthesis of rigid objects. In that work, image content is encoded into an
implicit volumetric representation (the “bottleneck”), in which each of the
encoded features in this volume correspond to the local structure and
appearance of the corresponding spatial region in the volume depicted in the
image. However, while it requires no 3D supervision, it is trained using
multi-view datasets of _rigid_ objects depicted from multiple viewpoints to
produce implicit volumes that can then be rigidly transformed to produce novel
views of the depicted content corresponding to changes in viewpoint.

We build upon this approach to address the challenge of performing motion
retargeting for _non-rigid_ humans (for which multiple images of a given
subject may be available, but in dramatically different poses). In doing so,
we address several challenges: how to aggregate volumetric features from
images with changes in camera and body pose, and how to learn this aggregation
from videos without explicit 3D or camera pose supervision. With such an
implicit representation, to synthesize a novel pose, we achieve _non-rigid_
implicit volume aggregation and manipulation by learning a 3D flow to resample
the 3D body model from input images captured with the subject performing
various poses or under different viewpoints. To allow for expressing large-
scale motion while retaining fine-grained details in the synthesized images,
we propose a multi-resolution scheme in which image content is encoded,
transformed and aggregated into bottlenecks of different resolution scales.

As we focus on transferring motion between human subjects, our network
pipeline is designed and trained specifically to extract and manipulate the
foreground of the encoded images, with a separate network for extracting and
compositing the background with the synthesized result. Our training scheme
employs techniques and loss functions precisely designed for the challenging
task of producing plausible motion retargeting without 3D supervision or the
use of explicit 3D models, making use of specialized training techniques to
teach the network to synthesize plausible results when no ground-truth images
corresponding to the applied spatial manipulation is available. We thus avoid
the limitations of explicit body representations [32, 34], which may lead to
unrealistic results due to the limited reconstruction accuracy and mesh
precision. Furthermore, it allows for learning directly from real 2D images
and videos without requiring the tedious and cumbersome collection of copious
high-fidelity 3D data [29, 44, 14].

In our experiments, we demonstrate that our approach qualitatively and
quantitatively outperforms state-of-the-art approaches to human motion
transfer, despite the few images used for inference, and even allows for
plausible motion transfer when using only a single image of the target.

In summary, our key contributions are:

  * •

A novel set of neural network architectures for performing implicit volumetric
human motion retargeting, which exploits the power of 3D human motion modeling
while avoiding the limitations of standard 3D human body modeling techniques.

  * •

A framework to train these networks to attain high-fidelity human motion
transfer using only a few example images of the target subject performing
various poses, _without_ requiring target-specific training.

  * •

Evaluations demonstrating our few-shot approach outperforms state-of-the-art
alternatives both quantitatively and qualitatively, even those requiring
training models for each new subject with substantial training data.

##  2 Related Work

Video-to-Video Generation. Existing works on video-to-video generation can
synthesize high-quality human motion videos using conditional image generation
[7, 20, 64]. Chan et al. [8] apply pre-computed human poses from driving
videos as input for novel view and pose generation of a target person. Along
this line, several works improve the synthesis quality through additional
input signals [13], pose augmentation and pre-processing [49, 66, 67, 73], and
temporal coherence [1, 63]. However, a long recorded video and person-specific
training are required for each target person, limiting the scalability of such
methods. Our work targets a few-shot scenario, with just a handful of source
images available, as discussed below.

Figure 1: Overall pipeline. Given $N$ reference images (left), we extract
implicit volumetric representations at multiple scales using our 2D-3D
encoder. Our flow network (bottom left) computes the 3D warping field used to
warp these representations from the canonical pose (the first source image, to
which the other source image feature volumes are aligned) to the target pose.
Our 3D-2D decoder then synthesizes the subject in the target pose (top right),
while transformations such as rotations can be applied to synthesize novel
views of the subject in this pose (middle right). Our occupancy decoder
(bottom right) improves the bottleneck’s spatial disentanglement and thus
allows for better motion retargeting. It computes an occupancy volume used to
indicate which regions of the volume are occupied by the subject, which is
then decoded into a 2D foreground segmentation mask.

Few-Shot Motion Retargeting. To address scalability, others train generic
models that can use just a few images of the target subject in arbitrary poses
at test time to synthesize novel images with given poses. While high-quality
results have been achieved for face animation [15, 16, 48, 69, 70], animating
bodies remains challenging. Some methods adapt the video-to-video approach to
a few-shot setting, using an extra network to generate identity-specific
weights for image generation network [62], or adapting a pre-trained network
to new subjects [30]. Another class of methods train networks conditioned on
an image of the source identity, and an explicit representation of target pose
[38, 39, 12]. More recent works exploit multi-stage networks to improve
quality [55, 19, 10], in particular using 2D spatial transformers [3, 50, 51]
or deformation [58] to warp the source image into the target pose, or
synthesizing images using attention [61, 74]. However, such purely 2D methods
struggle to capture the complex motions generated by 3D shapes and
transformations. Several works use explicit 3D representations, exploiting
off-the-shelf human pose [2] and shape [22] inference networks and body meshes
[36]. DensePose [2] is used to unwrap appearance from the source image(s) into
a canonical texture map, which is then inpainted and re-rendered in the target
pose [29, 44, 14]. Other works use 3D meshes to compute 2D flow fields to warp
features from source to target images [32, 34], with shallow encoders and
decoders at each end to reduce warping artifacts. Finally, several works use
body models and standard rendering techniques for clothing transfer [4, 40,
43, 46]. However, such explicit representations generally produce synthetic-
looking results. We use an _implicit_ 3D representation on this task for the
first time, unlocking several benefits: the motion representation is more
flexible, no 3D supervision or prior is needed, the image decoder can refine
the output easily, and multiple reference images can be used to improve
synthesis quality.

Unsupervised Motion Retargeting. Unsupervised methods learn retargeting purely
from videos [57, 28, 24, 60, 47], forgoing motion supervision and therefore
also class-specific motion representations. Siarohin et al.learn unsupervised
keypoints in order to warp object parts into novel poses [56, 57]. Lorenz et
al. [37] show that body parts can also be represented by unsupervised
learning, which also helps to disentangle body pose and shape [11].
Nevertheless, without the benefit of explicit pose information from a human
detection model, these methods struggle to generate good results for
challenging driving poses. We therefore use a high quality, off-the-shelf,
pose detection model [5] to facilitate the generation process.

Implicit and Volumetric 3D Representations. The flexibility of implicit 3D
content representations [54, 6, 27], which obviate the use of explicit
surfaces, discrete triangle meshes, make them amenable to recent learning-
based approaches that extract the information needed to render or reconstruct
scenes directly from images. In [59], an object-specific network is trained
from many calibrated images to extract deep features embedded in a 3D grid
that are used to synthesize novel views of the object. [42] perform novel view
synthesis on complex scenes by aggregating samples from a trained network
which, given a point in space and a view direction, provides an opacity and
color value indicating the radiance towards the viewer from this point. This
approach was later extended to handle unconstrained, large-scale scenes [41],
or to use multiple radiance field functions, stored in sparse voxel fields to
better capture detailed scenes with large empty regions [33]. However, these
approaches share the limitation that the networks are trained on many images
with known camera poses for a specific scene, and thus cannot be used to
perform 3D reconstruction or novel view synthesis from new images without re-
training. Furthermore, many images from different viewpoints are required to
train these networks to learn to sufficiently render points between these
images. Other recent efforts use 3D feature grids [23] or pixel-wise implicit
functions [52, 53, 31] to infer dynamic shapes from one or more images, but
these require synthetic ground-truth 3D geometry for supervision, which limits
their applicability to unconstrained real-world conditions. In [45], an
encoder-decoder framework is used to extract a volumetric implicit
representation of image content that is spatially disentangled in a manner
that allows for novel view synthesis, 3D reconstruction and spatial
manipulation of the encoded image content. However, while it allows for
performing non-rigid manipulations of the image content after training, it
must be trained using a multi-view dataset of _rigid_ objects, making it
unsuitable for human motion retargeting. In our work we develop an enhanced
implicit volumetric representation and multi-view fusion techniques to address
these concerns.

##  3 Method

At the heart of our approach to few-shot human body motion retargeting is an
implicit volumetric representation of the shape and appearance of the subject
depicted in the input images. In the following sections, we first describe
this representation, then outline the network architectures employing it to
perform foreground motion transfer and composition into the background
environment. Finally, we discuss the training techniques and loss functions
employed to train these networks to perform flexible motion retargeting while
still achieving high-fidelity image synthesis results. The overall
architecture is illustrated in Figure 1.

###  3.1 Implicit Volumetric Representation

Given an input image, our encoder extracts a feature volume or “bottleneck”
$f\in\mathbb{R}^{N\times D\times H\times W}$, where $D$, $H$, and $W$ are the
depth, height, and width of the encoded volumetric grid used to represent the
image content. Each cell in the grid contains an $N$-dimensional feature
vector describing the structure and appearance of the local image content
corresponding to that region of the volume. The depth dimension of this volume
is aligned with the view direction of the camera, while the height and width
correspond to those of the input image. The feature volume may be passed
directly to the image decoder to synthesize an image corresponding to the
input (in which case it acts as an auto-encoder), or it may be spatially
manipulated in a manner corresponding to the desired transformation of the
image content. Such manipulations include rigid transformations corresponding
to camera viewpoint changes, or non-rigid manipulations corresponding to the
subject’s body motion.

Given the encoded bottleneck $f$ and a dense _flow field_
$\mathrm{T}_{s\rightarrow t}\in\mathbb{R}^{3\times D\times H\times W}$
encoding a 3D coordinate per cell in the transformed bottleneck $f^{\prime}$
pointing to the original volume $f$ that corresponds to the mapping from pose
$s$ to $t$, we employ a sampling layer ${\mathrm{S}:f,\mathrm{T}_{s\rightarrow
t}\rightarrow f^{\prime}}$ to perform trilinear sampling to produce
$f^{\prime}$. This flexible sampling mechanism enables a wide variety of
spatial manipulations, from rigid transformations for novel view synthesis
(simulating camera pose change by rotating the bottleneck) to more complex
non-rigid changes (raising the arms of the subject while keeping the rest of
the body stationary).

While the flow field $\mathrm{T}_{s\rightarrow t}$ for rigid camera viewpoint
changes can be easily computed given the relative camera transformation
between $s$ and $t$, non-rigid body pose transformations can be much more
complicated. It requires the flow field to have the appropriate sampling
location in $f$ for each cell in $f^{\prime}$ that semantically corresponds to
the desired 3D motion of the content extracted from the input image.

Our training process, as described below, in which both rigid viewpoint and
non-rigid pose transformations are used, achieves the _spatial
disentanglement_ required to infer the appropriate flow field and employ it to
guide the desired spatial transformation of the image content.

###  3.2 Flow-Guided Volumetric Resampling

Network Architecture Overview. Our human body synthesis branch
$\mathrm{G}_{\mathrm{fg}}$ has three major components: the encoder, which
consists of a 2D encoding network $\mathrm{Enc}_{\mathrm{2D}}$ and a 3D
encoding network $\mathrm{Enc}_{\mathrm{3D}}$; the decoder, which consists of
a 3D decoding network $\mathrm{Dec}_{\mathrm{3D}}$ and a 2D decoding network
$\mathrm{Dec}_{\mathrm{2D}}$; and the flow-guided transformable bottleneck
network $\mathrm{F}$. A source reference image $\mathbf{x}_{\mathrm{fg}}$,
containing only the foreground region from the original image $\mathbf{x}$
(obtained using a pre-trained segmentation model [9]), is passed through the
encoder to obtain the 3D feature representation $f$ as $f$ =
$\mathrm{Enc}_{\mathrm{3D}}(\mathrm{Enc}_{\mathrm{2D}}(\mathbf{x}_{\mathrm{fg}}))$.
To warp the features, we estimate the flow using the flow network $\mathrm{F}$
with inputs of $\mathbf{x}_{\mathrm{fg}}$ and the source/target poses
$\mathbf{p}_{s},\mathbf{p}_{t}$, which are obtained with the off-the-shelf
pose detection network [5, 2]. For simplicity, we define this operation as
$\mathrm{F}_{s\rightarrow
t}(f)\coloneqq\mathrm{S}(f,\mathrm{F}(\mathbf{x}_{\mathrm{fg}},\mathbf{p}_{s},\mathbf{p}_{t}))$,
and generate the synthesized foreground frame using the flow field from
$\mathrm{F}$ as:

$\hat{\mathbf{y}}_{\mathrm{fg}}=\mathrm{Dec}_{\mathrm{2D}}(\mathrm{Dec}_{\mathrm{3D}}(\mathrm{F}_{s\rightarrow
t}(f))).$ (1)

Multi-Resolution Bottleneck. To increase the fine-scale fidelity of the
synthesized images while retaining the global structure of the target subject,
we adopt a multi-resolution representation, with implicit volumetric
representations at multiple resolutions, as seen in Figure 1. We employ skip
connections between each 3D encoder and 3D decoder. Therefore, the encoder
produces 3D features such that $f=\left\\{f_{1},f_{2},\ldots,f_{m}\right\\}$,
where $m$ is the number of resolutions. The skip connections use 3D warping,
given the estimated flow, to map content to from the correct region in the
encoded bottleneck to the one to be decoded, rather than the direct
connections used in prior art [17]. Thus, for each feature $f_{i}$ obtained
from the 3D encoder, the 3D decoder receives as input
$\mathrm{F}_{s\rightarrow t}(f_{i})$.

Multi-View Aggregation. Given that a single input image only contains partial
information for the depicted human body, we allow for the aggregation of
information, represented with our 3D features, from multiple views to improve
the synthesized image quality. To accelerate the inference, we take the first
source image as the canonical body pose (though this may actually be the
subject in any natural pose), and aggregate features from all other images to
this pose. In this way, we only perform the aggregation once during inference.
More formally, for a total of $N$ source images, the aggregated feature
$f_{a}$ is represented as:

$f_{a}=\frac{1}{N}(f_{1}+\sum_{i=2}^{N}\mathrm{F}_{i\rightarrow 1}(f_{i})),$
(2)

where $f_{i}$ is the 3D representation from the image $i$ and
$\mathrm{F}_{i\rightarrow 1}$ is the warping flow from the image $i$ to the
first image.

###  3.3 Background Modeling

The background environment is modeled through a separate network
$\mathrm{G}_{\mathrm{bg}}$, to which the source images are fed. Specifically,
$\mathrm{G}_{\mathrm{bg}}$ estimates a background image
$\hat{\mathbf{y}}_{\mathrm{bg}}$ and a confidence map $\hat{\mathbf{w}}$, in
which high confidence indicates the foreground, the depicted person, while low
confidence indicates the background. The synthesized image $\hat{\mathbf{y}}$
is:

$\hat{\mathbf{y}}=\hat{\mathbf{w}}\cdot\hat{\mathbf{y}}_{\mathrm{fg}}+(1-\hat{\mathbf{w}})\cdot\hat{\mathbf{y}}_{\mathrm{bg}},$
(3)

where $\cdot$ denotes component-wise multiplication of the confidence map with
the color channels of the synthesized image, and
$\hat{\mathbf{y}}_{\mathrm{bg}}$ indicates the synthesized background image.

###  3.4 Training

Retargeting Supervision. We use a conditional discriminator
$\mathrm{D}_{\mathrm{fg}}$ to determine whether the synthesized foreground
image is real or fake. The concatenation of the input image, 2D source/target
poses, and target image is sent to the discriminator and we apply the
following adversarial loss:

$\begin{split}\mathcal{L}_{\mathrm{D}_{\mathrm{fg}}}=\mathbb{E}_{\mathbf{p}_{t},\mathbf{y}_{\mathrm{fg}}}[\log\mathrm{D}_{\mathrm{fg}}(\mathbf{p}_{t},\mathbf{y}_{\mathrm{fg}})]+\\\
\mathbb{E}_{\mathbf{x}_{\mathrm{fg}},\mathbf{p}_{s},\mathbf{p}_{t}}[\log(1-\mathrm{D}_{\mathrm{fg}}(\mathbf{p}_{t},\mathrm{G}_{\mathrm{fg}}(\mathbf{x}_{\mathrm{fg}},\mathbf{p}_{s},\mathbf{p}_{t})))],\end{split}$
(4)

where $\mathbf{y}_{\mathrm{fg}}$ is the foreground region from the real image.
The generator is trained to minimize this objective, while the discriminator
is trained to maximize it.

Similarly, we have a discriminator $\mathrm{D}_{\mathrm{bg}}$ that works on
the full synthesized with foreground and background:

$\begin{split}\mathcal{L}_{\mathrm{D}_{\mathrm{bg}}}=\mathbb{E}_{\mathbf{p}_{t},\mathbf{y}}[\log\mathrm{D}_{\mathrm{bg}}(\mathbf{p}_{t},\mathbf{y})]+\\\
\mathbb{E}_{\mathbf{x},\mathbf{p}_{s},\mathbf{p}_{t}}[\log(1-\mathrm{D}_{\mathrm{bg}}(\mathbf{p}_{t},\mathrm{G}_{\mathrm{bg}}(\mathbf{x},\mathbf{p}_{s},\mathbf{p}_{t})))],\end{split}$
(5)

where $\mathbf{y}$ denotes the real image.

We also use a perceptual loss [21] $\mathcal{L}_{\mathrm{vgg}}$ between the
real and generated images to improve fidelity of the generated images for both
the foreground and background:

$\mathcal{L}_{\mathrm{per}}=\mathcal{L}_{\mathrm{vgg}}(\mathbf{y}_{\mathrm{fg}},\hat{\mathbf{y}}_{\mathrm{fg}})+\mathcal{L}_{\mathrm{vgg}}(\mathbf{y},\hat{\mathbf{y}}).$
(6)

Additionally, we measure the reconstruction quality of each of the $N$ source
images using the aggregated bottleneck. The reconstructed image is generated
as
$\hat{\mathbf{x}}=\mathrm{Dec}_{\mathrm{2D}}(\mathrm{Dec}_{\mathrm{3D}}((f)))$,
with no flow required for this auto-encoding. The reconstruction loss
$\mathcal{L}_{\mathrm{recon}}$ is:

$\mathcal{L}_{\mathrm{recon}}=\frac{1}{N}(\sum_{i=1}^{N}\left\|\hat{\mathbf{x}}-\mathbf{x}\right\|_{1}).$
(7)

Mask Supervision. We also leverage mask supervision, using the foreground
masks, to better supervise the implicit 3D representation modeling. Similar to
previous work [45], we introduce an occupancy decoder to obtain the estimated
mask directly from the 3D features. Considering that we have multi-resolution
bottlenecks in our architecture, we apply multiple occupancy decoders to get
the mask from each of the bottlenecks. Specifically, given the occupancy
decoder as $\mathrm{Dec}_{\mathrm{occ}}$, the estimated mask
$\hat{\mathbf{w}}_{i}$ for the $i$-th 3D representation is given as
$\hat{\mathbf{w}}_{i}=\mathrm{Dec}_{\mathrm{occ}}(f_{i})$. The mask loss
$\mathcal{L}_{\mathrm{mask}}$ is thus defined as follows:

$\mathcal{L}_{\mathrm{mask}}=\frac{1}{N}(\sum_{j=1}^{N}\frac{1}{m}(\sum_{i=1}^{m}\left\|\hat{\mathbf{w}}_{i}^{j}-\mathbf{w}_{i}^{j}\right\|_{1})),$
(8)

where $\mathbf{w}^{j}$ is the mask obtained from $j$-th source reference image
using the pre-trained segmentation network.

Unsupervised Random Rotation Supervision. We further introduce an unsupervised
training technique to help the network learn implicit volumetric
representations for the encoded subject with appropriate spatial structure.
Specifically, we apply a random rotation around the vertical axis of the
volume to the encoded bottleneck and enforce the corresponding synthesized
image to be indistinguishable from the ground-truth views. The magnitude of
this rotation is sampled from a uniform distribution $r\sim
U(-180^{\circ},180^{\circ})$. The synthesized image is thus
$\hat{\mathbf{y}}_{r}=\mathrm{Dec}_{\mathrm{2D}}(\mathrm{Dec}_{\mathrm{3D}}(\mathrm{F}_{s\rightarrow
r}(f)))$, which should contain a novel view of the subject performing the same
pose as in the source image, where $\mathrm{F}_{s\rightarrow r}$ is the flow
field defined by the rigid transformation between the source pose $s$ and the
random pose $r$. However, since there is no ground-truth image corresponding
to each random rigid transformation, we introduce a discriminator
$\mathrm{D}_{\mathrm{rot}}$ to match the distribution between real images and
$\hat{\mathbf{y}}_{r}$ in an unsupervised manner. We provide the discriminator
with the concatenation of the generated image and the original source to
better maintain the source identity. We employ an adversarial loss to enforce
the rotation constraint on the foreground region as follows:

$\begin{split}\mathcal{L}_{\mathrm{D}_{\mathrm{rot}}}=\mathbb{E}_{\mathbf{x}_{\mathrm{fg}},\mathbf{y}_{\mathrm{fg}}}[\log\mathrm{D}_{\mathrm{rot}}(\mathbf{x}_{\mathrm{fg}},\mathbf{y}_{\mathrm{fg}})]+\\\
\mathbb{E}_{\mathbf{x}_{\mathrm{fg}},\mathbf{p}_{s},r}[\log(1-\mathrm{D}_{\mathrm{rot}}(\mathbf{x}_{\mathrm{fg}},\mathrm{G}_{\mathrm{fg}}(\mathbf{x}_{\mathrm{fg}},\mathbf{p}_{s},r)))].\end{split}$
(9)

Full Objective. The full training objective for the entire motion retargeting
network is thus given as follows:

$\begin{split}\mathcal{L}_{\mathrm{T}}=\lambda_{\mathrm{fg}}\mathcal{L}_{\mathrm{D}_{\mathrm{fg}}}+\lambda_{\mathrm{bg}}\mathcal{L}_{\mathrm{D}_{\mathrm{bg}}}+\lambda_{\mathrm{per}}\mathcal{L}_{\mathrm{per}}+\\\
\lambda_{\mathrm{recon}}\mathcal{L}_{\mathrm{recon}}+\lambda_{\mathrm{mask}}\mathcal{L}_{\mathrm{mask}}+\lambda_{\mathrm{rot}}\mathcal{L}_{\mathrm{D}_{\mathrm{rot}}},\end{split}$
(10)

where $\lambda_{\mathrm{fg}}$, $\lambda_{\mathrm{bg}}$,
$\lambda_{\mathrm{per}}$, $\lambda_{\mathrm{recon}}$,
$\lambda_{\mathrm{mask}}$, and $\lambda_{\mathrm{rot}}$ are hyper-parameters
to control the weight of each loss function.

##  4 Experiments

Table 1: Quantitative results on the iPER [34] dataset. We compare our method with existing works using the SSIM and LPIPS.  Method | SSIM$\uparrow$ | LPIPS$\downarrow$
---|---|---
PG2 [38] | 0.854 | 0.135
SHUP [3] | 0.832 | 0.099
DSC [58] | 0.829 | 0.129
LiquidGAN [34] | 0.840 | 0.087
Ours | 0.868 | 0.086
Figure 2: Qualitative results on the iPER dataset. The leftmost columns show
the reference images. For each example, we show both ground-truth (GT) driving
images (odd rows), and the images generated by our method using the pose from
the GT images (even rows). Our method can generate realistic images from
diverse motion patterns.

In this section, we evaluate our method on two different datasets, and show
qualitative and quantitative comparisons with recent state-of-the-art efforts
on human motion generation. For additional results, including video sequences,
please consult the supplemental material.

###  4.1 Experimental Setting

Datasets. We adopt two datasets for experiments:

  * •

iPER. The first dataset, known as the Impersonator (iPER) dataset, was
recently collected and released by LiquidGAN [34], and serves as a benchmark
dataset for human motion animation tasks. There are $206$ videos with a total
of $241,564$ frames in the dataset. Each video depicts one person performing
different actions. We follow the protocol in previous work, splitting the
training and testing set in the ratio of 8:2. Compared with other datasets
[35, 72, 68], iPER includes human subjects with widely varying shape, height
and appearance, and performing diverse motions.

  * •

Youtube-Dancing. We create this dataset by collecting dancing videos from
Youtube. It includes $675$ dancing videos with $4,063,453$ frames in total for
training, and $200$ videos with $12,965$ frames for testing. The videos are
recorded in unconstrained, in-the-wild environments, and consist of more
challenging and diverse motion patterns compared with the iPER dataset.

Implementation Details. We normalize all images to the range
$\left[-1,1\right]$ for training, and train the networks to synthesize images
with a resolution of $256\times 256$. We apply the Adam optimizer [25] with a
mini-batch size of $32$ for training. Following previous work [64], we apply
multi-scale discriminators, where each discriminator accepts images with the
original resolution, $256\times 256$ and images with half this resolution,
$128\times 128$. The hyper-parameters in Eqn. 10 are set to $1$, except for
$\lambda_{\mathrm{per}}=10$.

Evaluation Metrics. We use three widely-adopted metrics to evaluate the image
synthesis quality. The Structural Similarity (SSIM) [65] index measures the
structural similarity between synthesized and real images. The Learned
Perceptual Similarity (LPIPS) [71] measures the perceptual similarity between
these images. The Fr$\mathrm{\acute{e}}$chet Inception Distance (FID) [18]
calculates the distance between two distributions of real and synthesized
images, and is commonly used to quantify the fidelity of the synthesized
images.

Figure 3: Qualitative results on the Youtube-Dancing dataset. The first and fifth column show one reference image from a target person. For each target person, we show short sequences, including the ground-truth driving images, and the images generated by our and FewShot-V2V [62] using the pose information from the driving images. As seen here, our approach clearly generates more realistic results. Table 2: Quantitative results on the Youtube-Dancing dataset. We report the SSIM, LPIPS, and FID for both methods. Method |  SSIM$\uparrow$ |  LPIPS$\downarrow$ |  FID$\downarrow$
---|---|---|---
FewShot-V2V [62] | 0.770 | 0.144 | 48.39
Ours | 0.787 | 0.131 | 18.82

###  4.2 Comparisons and Results

We first show the experimental results on the iPER dataset. Following previous
work [34], we use three reference images with different degrees of occlusion
from each video to synthesize other images. We report both the SSIM and LPIPS
for our work and existing studies, including PG2 [38], SHUP [3], DSC [58], and
LiquidGAN [34] in Table 1. As shown in the table, we achieve better results
using both metrics than state-of-the-art works, indicating higher synthesized
image quality with our method. Additionally, we provide qualitative results in
Figure 2. We show an example source reference image from a target subject, and
the generated sequence using the driving pose information from the ground-
truth images. As depicted, our method can generate realistic images with
diverse motion patterns.

Table 3: Ablation analysis of our proposed architecture on the iPER dataset. Our Full method achieves results that are superior to all other variants. Method | SSIM$\uparrow$ | LPIPS$\downarrow$
---|---|---
w/o Multi-Resolution TBN | 0.870 | 0.051
w/o Skip-Connections | 0.856 | 0.059
w/o Random Rotations | 0.876 | 0.047
Full | 0.878 | 0.045
Table 4: Analysis of the use of different numbers of reference images from the target subject for image generation on the iPER dataset. The image quality improves with our method as more reference images are used. Method | SSIM$\uparrow$ | LPIPS$\downarrow$
---|---|---
One reference image | 0.878 | 0.045
Two reference images | 0.879 | 0.045
Three reference images | 0.881 | 0.044
Four reference images | 0.882 | 0.043

We then perform these experiments on our collected Youtube-Dancing dataset. We
compare our method with the most recent work, FewShot-V2V [62], which can
generate motion retargeting videos using one or a few images. For evaluation,
we use the first frame from each testing video as a source reference frame and
using poses from the other frames to generate images. Besides the SSIM and
LPIPS, we also follow FewShot-V2V [62] and report the FID scores. The results
are summarized in Table 2. As can be seen, our method achieves better results
than FewShot-V2V [62] on each of the three metrics. We also provide sample
qualitative results in Figure 3. We show the source reference image from a
target subject, and the generated short sequence from our method and
FewShot-V2V using the pose from the ground-truth images. Compared with
FewShot-V2V, our method generates more realistic images with fewer artifacts,
and the identity and the texture from the target subject are much better
preserved.

###  4.3 Ablation Study

Architecture and Training Technique Analysis. We conduct an ablation analysis
of our network architectures and training strategies. For this experiment, we
analyze the effects of each component used to generate the human body, the
foreground region. We thus remove the background using an image segmentation
model [9], and only compute evaluation metrics on the generated foreground
region. We use one source reference image from a target subject and preform
experiments with the following setting on iPER:

  * •

_w/o Multi-Resolution TBN._ Instead of using multi-resolution bottlenecks, we
use a single resolution TBN to encode the implicit 3D representation.

  * •

_w/o Skip-Connections._ The skip-connections (implemented via flow warping)
between the encoded and decoded 3D bottlenecks are removed.

  * •

_w/o Random Rotations._ We remove the unsupervised random rotation supervision
applied to the encoded TBN and its associated loss function, defined in Eqn.
9.

The quantitative results, including the SSIM and LPIPS, are shown in Table 3.
We can see that each component is beneficial and that our full method (_Full_)
achieves the best results.

Multi-View Aggregation Analysis. As our method can leverage multiple reference
images from the target to perform multi-view aggregation, we conduct
experiments to analyze the effect of the number of reference images on the
synthesis result using the iPER dataset. The experimental results, presented
in Table 4, demonstrate that using more reference images improves the quality
of the synthesized human bodies.

##  5 Conclusion

Our approach to few-shot human motion retargeting exploits advantages of 3D
representations of human body while avoiding limitations of the more
straightforward prior methods. Our implicit 3D representation, learned via
spatial disentanglement during training, avoids pitfalls of standard geometric
representations such as dense pose estimations or template meshes, which are
limited in their expressive capacity and for which it is impossible to obtain
accurate ground-truth in unconstrained conditions. However, it allows for
3D-aware motion inference and image content manipulation, and attains state-
of-the-art results on challenging motion retargeting benchmarks. Though we
require 2D human poses, our approach could be extended to allow for more
general motion retargeting for images of articulated animals given their 2D
poses.

## References

  * [1] Kfir Aberman, Mingyi Shi, Jing Liao, Dani Lischinski, Baoquan Chen, and Daniel Cohen-Or.  Deep video-based performance cloning.  In Computer Graphics Forum, volume 38, pages 219–233. Wiley Online Library, 2019.
  * [2] Rıza Alp Güler, Natalia Neverova, and Iasonas Kokkinos.  Densepose: Dense human pose estimation in the wild.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7297–7306, 2018.
  * [3] Guha Balakrishnan, Amy Zhao, Adrian V Dalca, Fredo Durand, and John Guttag.  Synthesizing images of humans in unseen poses.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 8340–8348, 2018\.
  * [4] Bharat Lal Bhatnagar, Garvita Tiwari, Christian Theobalt, and Gerard Pons-Moll.  Multi-garment net: Learning to dress 3d people from images.  In Proceedings of the IEEE International Conference on Computer Vision, pages 5420–5430, 2019.
  * [5] Zhe Cao, Gines Hidalgo, Tomas Simon, Shih-En Wei, and Yaser Sheikh.  Openpose: realtime multi-person 2d pose estimation using part affinity fields.  arXiv preprint arXiv:1812.08008, 2018.
  * [6] J. C. Carr, R. K. Beatson, J. B. Cherrie, T. J. Mitchell, W. R. Fright, B. C. McCallum, and T. R. Evans.  Reconstruction and representation of 3d objects with radial basis functions.  In Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH ’01, page 67–76, New York, NY, USA, 2001. Association for Computing Machinery.
  * [7] Menglei Chai, Jian Ren, and Sergey Tulyakov.  Neural hair rendering.  In Eur. Conf. Comput. Vis., 2020.
  * [8] Caroline Chan, Shiry Ginosar, Tinghui Zhou, and Alexei A. Efros.  Everybody dance now.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
  * [9] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam.  Encoder-decoder with atrous separable convolution for semantic image segmentation.  In Proceedings of the European conference on computer vision (ECCV), pages 801–818, 2018.
  * [10] Haoye Dong, Xiaodan Liang, Ke Gong, Hanjiang Lai, Jia Zhu, and Jian Yin.  Soft-gated warping-gan for pose-guided person image synthesis.  In Advances in neural information processing systems, pages 474–484, 2018.
  * [11] Patrick Esser, Johannes Haux, and Bjorn Ommer.  Unsupervised robust disentangling of latent characteristics for image synthesis.  In Proceedings of the IEEE International Conference on Computer Vision, pages 2699–2709, 2019.
  * [12] Patrick Esser, Ekaterina Sutter, and Björn Ommer.  A variational u-net for conditional appearance and shape generation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8857–8866, 2018.
  * [13] Oran Gafni, Lior Wolf, and Yaniv Taigman.  Vid2game: Controllable characters extracted from real-world videos.  arXiv preprint arXiv:1904.08379, 2019.
  * [14] Artur Grigorev, Artem Sevastopolsky, Alexander Vakhitov, and Victor Lempitsky.  Coordinate-based texture inpainting for pose-guided human image generation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 12135–12144, 2019.
  * [15] Kuangxiao Gu, Yuqian Zhou, and Thomas Huang.  Flnet: Landmark driven fetching and learning network for faithful talking facial animation synthesis.  arXiv preprint arXiv:1911.09224, 2019.
  * [16] Sungjoo Ha, Martin Kersner, Beomsu Kim, Seokjun Seo, and Dongyoung Kim.  Marionette: Few-shot face reenactment preserving identity of unseen targets.  arXiv preprint arXiv:1911.08139, 2019.
  * [17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [18] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  In NeurIPs, 2017.
  * [19] Mohamed Ilyes Lakhal, Oswald Lanz, and Andrea Cavallaro.  Pose guided human image synthesis by view disentanglement and enhanced weighting loss.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 0–0, 2018.
  * [20] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros.  Image-to-image translation with conditional adversarial networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1125–1134, 2017.
  * [21] Justin Johnson, Alexandre Alahi, and Li Fei-Fei.  Perceptual losses for real-time style transfer and super-resolution.  In European conference on computer vision, pages 694–711. Springer, 2016.
  * [22] Angjoo Kanazawa, Michael J Black, David W Jacobs, and Jitendra Malik.  End-to-end recovery of human shape and pose.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7122–7131, 2018.
  * [23] Abhishek Kar, Christian Häne, and Jitendra Malik.  Learning a multi-view stereo machine.  In Adv. Neural Inform. Process. Syst., pages 365–376, 2017.
  * [24] Yunji Kim, Seonghyeon Nam, In Cho, and Seon Joo Kim.  Unsupervised keypoint learning for guiding class-conditional video prediction.  In Advances in Neural Information Processing Systems, pages 3814–3824, 2019.
  * [25] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [26] Markus Knoche, István Sárándi, and Bastian Leibe.  Reposing humans by warping 3d features.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 1044–1045, 2020.
  * [27] Aaron Knoll.  A survey of implicit surface rendering methods, and a proposal for a common sampling framework.  In Lecture Notes in Informatics (LNI), Proceedings - Series of the Gesellschaft fur Informatik (GI), pages 164–177, 01 2007.
  * [28] Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, Nicu Sebe, et al.  Motion-supervised co-part segmentation.  arXiv preprint arXiv:2004.03234, 2020.
  * [29] Verica Lazova, Eldar Insafutdinov, and Gerard Pons-Moll.  360-degree textures of people in clothing from a single image.  In 2019 International Conference on 3D Vision (3DV), pages 643–653. IEEE, 2019.
  * [30] Jessica Lee, Deva Ramanan, and Rohit Girdhar.  Metapix: Few-shot video retargeting.  arXiv preprint arXiv:1910.04742, 2019.
  * [31] Ruilong Li, Yuliang Xiu, Shunsuke Saito, Zeng Huang, Kyle Olszewski, and Hao Li.  Monocular real-time volumetric performance capture.  arXiv preprint arXiv:2007.13988, 2020.
  * [32] Yining Li, Chen Huang, and Chen Change Loy.  Dense intrinsic appearance flow for human pose transfer.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3693–3702, 2019.
  * [33] Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt.  Neural sparse voxel fields.  NeurIPS, 2020.
  * [34] Wen Liu, Zhixin Piao, Jie Min, Wenhan Luo, Lin Ma, and Shenghua Gao.  Liquid warping gan: A unified framework for human motion imitation, appearance transfer and novel view synthesis.  In Int. Conf. Comput. Vis., pages 5904–5913, 2019.
  * [35] Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang, and Xiaoou Tang.  Deepfashion: Powering robust clothes recognition and retrieval with rich annotations.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1096–1104, 2016.
  * [36] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black.  Smpl: A skinned multi-person linear model.  ACM transactions on graphics (TOG), 34(6):1–16, 2015.
  * [37] Dominik Lorenz, Leonard Bereska, Timo Milbich, and Bjorn Ommer.  Unsupervised part-based disentangling of object shape and appearance.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 10955–10964, 2019.
  * [38] Liqian Ma, Xu Jia, Qianru Sun, Bernt Schiele, Tinne Tuytelaars, and Luc Van Gool.  Pose guided person image generation.  In Adv. Neural Inform. Process. Syst., pages 406–416, 2017.
  * [39] Liqian Ma, Qianru Sun, Stamatios Georgoulis, Luc Van Gool, Bernt Schiele, and Mario Fritz.  Disentangled person image generation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 99–108, 2018.
  * [40] Qianli Ma, Jinlong Yang, Anurag Ranjan, Sergi Pujades, Gerard Pons-Moll, Siyu Tang, and Michael J Black.  Learning to dress 3d people in generative clothing.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6469–6478, 2020.
  * [41] Ricardo Martin-Brualla, Noha Radwan, Mehdi S. M. Sajjadi, Jonathan T. Barron, Alexey Dosovitskiy, and Daniel Duckworth.  Nerf in the wild: Neural radiance fields for unconstrained photo collections, 2020.
  * [42] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In ECCV, 2020.
  * [43] Aymen Mir, Thiemo Alldieck, and Gerard Pons-Moll.  Learning to transfer texture from clothing images to 3d humans.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7023–7034, 2020.
  * [44] Natalia Neverova, Riza Alp Guler, and Iasonas Kokkinos.  Dense pose transfer.  In Proceedings of the European conference on computer vision (ECCV), pages 123–138, 2018.
  * [45] Kyle Olszewski, Sergey Tulyakov, Oliver Woodford, Hao Li, and Linjie Luo.  Transformable bottleneck networks.  In Proceedings of the IEEE International Conference on Computer Vision, pages 7648–7657, 2019.
  * [46] Chaitanya Patel, Zhouyingcheng Liao, and Gerard Pons-Moll.  Tailornet: Predicting clothing in 3d as a function of human pose, shape and garment style.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7365–7375, 2020.
  * [47] Albert Pumarola, Antonio Agudo, Alberto Sanfeliu, and Francesc Moreno-Noguer.  Unsupervised person image synthesis in arbitrary poses.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8620–8628, 2018.
  * [48] Shengju Qian, Kwan-Yee Lin, Wayne Wu, Yangxiaokang Liu, Quan Wang, Fumin Shen, Chen Qian, and Ran He.  Make a face: Towards arbitrary high fidelity face manipulation.  In Proceedings of the IEEE International Conference on Computer Vision, pages 10033–10042, 2019.
  * [49] Jian Ren, Menglei Chai, Sergey Tulyakov, Chen Fang, Xiaohui Shen, and Jianchao Yang.  Human motion transfer from poses in the wild.  arXiv preprint arXiv:2004.03142, 2020.
  * [50] Yurui Ren, Ge Li, Shan Liu, and Thomas H Li.  Deep spatial transformation for pose-guided person image generation and animation.  IEEE Trans. Image Process., 29:8622–8635, 2020.
  * [51] Yurui Ren, Xiaoming Yu, Junming Chen, Thomas H Li, and Ge Li.  Deep image spatial transformation for person image generation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7690–7699, 2020.
  * [52] Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li.  Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization.  In ICCV, 2019.
  * [53] Shunsuke Saito, Tomas Simon, Jason Saragih, and Hanbyul Joo.  Pifuhd: Multi-level pixel-aligned implicit function for high-resolution 3d human digitization.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, June 2020.
  * [54] Chen Shen, James F. O’Brien, and Jonathan R. Shewchuk.  Interpolating and approximating implicit surfaces from polygon soup.  ACM Trans. Graph., 23(3):896–904, Aug. 2004.
  * [55] Chenyang Si, Wei Wang, Liang Wang, and Tieniu Tan.  Multistage adversarial losses for pose-based human image synthesis.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 118–126, 2018.
  * [56] Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe.  Animating arbitrary objects via deep motion transfer.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2377–2386, 2019.
  * [57] Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe.  First order motion model for image animation.  In Adv. Neural Inform. Process. Syst., December 2019.
  * [58] Aliaksandr Siarohin, Enver Sangineto, Stéphane Lathuiliere, and Nicu Sebe.  Deformable gans for pose-based human image generation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3408–3416, 2018.
  * [59] Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nießner, Gordon Wetzstein, and Michael Zollhofer.  Deepvoxels: Learning persistent 3d feature embeddings.  In Proc. CVPR, 2019.
  * [60] Sijie Song, Wei Zhang, Jiaying Liu, and Tao Mei.  Unsupervised person image generation with semantic parsing transformation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2357–2366, 2019.
  * [61] Hao Tang, Song Bai, Li Zhang, Philip HS Torr, and Nicu Sebe.  Xinggan for person image generation.  arXiv preprint arXiv:2007.09278, 2020.
  * [62] Ting-Chun Wang, Ming-Yu Liu, Andrew Tao, Guilin Liu, Jan Kautz, and Bryan Catanzaro.  Few-shot video-to-video synthesis.  In Adv. Neural Inform. Process. Syst., 2019.
  * [63] Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, and Bryan Catanzaro.  Video-to-video synthesis.  arXiv preprint arXiv:1808.06601, 2018.
  * [64] Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro.  High-resolution image synthesis and semantic manipulation with conditional gans.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8798–8807, 2018.
  * [65] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli.  Image quality assessment: from error visibility to structural similarity.  TIP, 2004.
  * [66] Dongxu Wei, Xiaowei Xu, Haibin Shen, and Kejie Huang.  Gac-gan: A general method for appearance-controllable human video motion transfer.  IEEE Transactions on Multimedia, 2020.
  * [67] Zhuoqian Yang, Wentao Zhu, Wayne Wu, Chen Qian, Qiang Zhou, Bolei Zhou, and Chen Change Loy.  Transmomo: Invariance-driven unsupervised video motion retargeting.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5306–5315, 2020.
  * [68] Polina Zablotskaia, Aliaksandr Siarohin, Bo Zhao, and Leonid Sigal.  Dwnet: Dense warp-based network for pose-guided human video generation.  arXiv preprint arXiv:1910.09139, 2019.
  * [69] Egor Zakharov, Aleksei Ivakhnenko, Aliaksandra Shysheya, and Victor Lempitsky.  Fast bi-layer neural synthesis of one-shot realistic head avatars.  In European Conference on Computer Vision, pages 524–540. Springer, 2020.
  * [70] Egor Zakharov, Aliaksandra Shysheya, Egor Burkov, and Victor Lempitsky.  Few-shot adversarial learning of realistic neural talking head models.  In Proceedings of the IEEE International Conference on Computer Vision, pages 9459–9468, 2019.
  * [71] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586–595, 2018.
  * [72] Liang Zheng, Liyue Shen, Lu Tian, Shengjin Wang, Jingdong Wang, and Qi Tian.  Scalable person re-identification: A benchmark.  In Proceedings of the IEEE international conference on computer vision, pages 1116–1124, 2015.
  * [73] Yipin Zhou, Zhaowen Wang, Chen Fang, Trung Bui, and Tamara Berg.  Dance dance generation: Motion transfer for internet videos.  In Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 0–0, 2019.
  * [74] Zhen Zhu, Tengteng Huang, Baoguang Shi, Miao Yu, Bofei Wang, and Xiang Bai.  Progressive pose attention transfer for person image generation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2347–2356, 2019.


# Can neutron star discriminate between Dirac and Majorana neutrinos?

Ashutosh Kumar Alok<EMAIL_ADDRESS>Indian Institute of Technology Jodhpur,
Jodhpur 342037, India  Neetu Raj Singh Chundawat<EMAIL_ADDRESS>Indian
Institute of Technology Jodhpur, Jodhpur 342037, India  Arindam Mandal
<EMAIL_ADDRESS>Indian Institute of Technology Jodhpur, Jodhpur 342037,
India  Trisha Sarkar<EMAIL_ADDRESS>Indian Institute of Technology
Jodhpur, Jodhpur 342037, India

###### Abstract

Any observable repercussion of electromagnetic properties of neutrinos will
provide a perspicuous signature of new physics. This includes the phenomenon
of neutrino spin flip in the propinquity of an external magnetic field. In
this work, we study the inklings of spin flip in a neutron star with a
radially varying magnetic field and matter density, known as a magnetar, which
is also a source of profuse production of neutrinos during its initial stage
of thermal evolution. We find that a precise measurement of neutrino flux
emerging from a neutron star would reeducate discrimination between Dirac and
Majorana neutrinos as the flux reduction due to spin flip oscillations are
different for both types of neutrinos. Further, the reduction of flux is more
preeminent near the surface as compared to the core of a neutron star.

##  I Introduction

Ever since Wolfgang Pauli postulated the existence of neutrinos in 1930 as a
saviour to the law of conservation of energy and momentum in $\beta$-decay,
the inscrutable and ubiquitous neutrinos have played a cardinal role in the
understanding of particle physics as well as has an abstruse implications in
the domain of astrophysics and cosmology. The phenomenon of neutrino
oscillations has established the fact that neutrinos have non-zero and non-
degenerate masses [1]. This can be considered as the first substantiation in
benevolence of physics beyond the Standard Model (SM) of particle physics.
However, despite a number of breakthroughs, there are still multifarious
unanswered questions in the neutrino physics sector. These include whether
neutrinos are Dirac or Majorana particles , what is the absolute scale of mass
of neutrinos and $CP$ violation in the leptonic sector. A number of planned
experiments[2] in the neutrino sector will chaperone judicious exploration of
many of the open questions in neutrino physics as well as scrutinizing new
scenarios that are opened up by the recent advances in the domain.

One such facet is neutrino electromagnetic properties. In various theories
involving new physics, neutrinos may develop electromagnetic properties via
quantum loop effects by radiative corrections and hence can acquire a non zero
magnetic moment [3]. Thus, the interaction of neutrinos with any external
electromagnetic field can provide an excellent rostrum to elucidate the
theories beyond SM. Apart from disquisition of new physics, neutrino
electromagnetic properties can also be an exceptional tool to differentiate
between the Dirac and Majorana nature of neutrinos. While coupled to a photon
corresponding to an electromagnetic field, the form factor of interaction at
zero momentum transfer $f_{q}(q^{2}=0)$ which can be measured in any
experimental set ups, are different for Dirac and Majorana neutrinos which
sets a possible distinction between the two kinds. Also, the magnetic moment
of Dirac neutrino can be both diagonal and off-diagonal in nature, while
Majorana neutrino can only possess transitional magnetic moment [3].

In the electroweak (EW) theory of SM, neutrinos are massless left handed (LH)
Weyl fermions and have zero magnetic moment. In the theories of minimally
extended SM (MESM), the RH neutrinos are also taken into account along with
the LH correspondent which is capable of providing a Dirac mass to the
particle and hence a non zero magnetic moment which is extremely small due to
the small mixing between LH and RH current ($\sim
10^{-19}\mu_{B},\leavevmode\nobreak\ \mu_{B}$ is the Bohr magneton) [4]. Such
models are usually left right symmetric (LRSM) and operate under the gauge
group of $SU(3)_{c}\times SU(2)_{L}\times SU(2)_{R}\times U(1)_{B-L}$.
However, in some of the new physics models involving additional charged scalar
particle, the estimated value of neutrino magnetic moment can be quite large
($10^{-12}-10^{-10}\mu_{B}$) as compared to the value predicted from the MESM
theories [5]. The value of the neutrino magnetic moment is constrained from
several terrestrial experiments such as GEMMA [6], Borexino [7], XENON1T [8],
XMASS-I [9] which relies upon the observation of neutrino-electron scattering.
Several astrophysical and cosmological phenomena are also able to put
constraints on the magnetic moment [10, 11, 12].

One of the most engrossing reverberations of electromagnetic properties of
neutrinos is the phenomenon of spin oscillation occurring in neutrino states
in the presence of an external magnetic field. This can have significant
consequences in the context of terrestrial detectors corresponding to several
astrophysical backgrounds. In fact, the phenomena of neutrino spin flavour
precession (SFP) is treated as an alternate theory for the observed abatement
in the solar neutrino flux, beside the framework of neutrino flavour
oscillation. However, the SFP in magnetic field by itself cannot account for
the deficiency in solar neutrino flux. This prospect is ruled out by the
KamLAND experiment data.

In this work, we consider a dense astrophysical compact object having a large
magnetic field, such as a neutron star in which we anatomize the phenomenon of
SFP and contemplate whether this can mitigate to distinguish between Dirac and
Majorana nature of neutrinos. Neutron stars are unique astrophysical objects
which are created as a product of stellar evolution of a main sequence star
having higher mass and in which matter resides at the extreme environmental
conditions. It has a very high intrinsic magnetic field, which can reach even
at an extremely high value of $\sim 10^{18}$ G at its center. The structure of
a neutron star is held by the degenerate pressure of a large number of
neutrons present at its core which is also the source of production of
neutrinos in copious amounts. Neutrino is the significant heat carrier of the
neutron star which cools down the star most efficiently. During the cooling
stage of a neutron star, neutrino emission era is initiated after a brief
period of thermal relaxation which creates a thermal equilibrium between the
core and crust of the star. In the presence of magnetic field, the Dirac
neutrinos are converted into its sterile counterpart, whereas the Majorana
neutrinos are converted into anti-neutrinos.

The plan of the work is as follows. In the next section, we delineate the
formalism of this work. This includes a brief description of the SFP
phenomenon followed by a depiction of magnetic field profile in a neutron
star. The results are furnished in Sec. III. The conclusions are acquainted in
Sec. IV.

##  II Spin flavor oscillations in neutron star

In this section, we silhouette the formalism of our work. In Sec. II.1, we
proffer magnetic moment of neutrino and the phenomena of SFP as its
consequence and in Sec. II.2, we limn neutron star and its magnetic field
profile.

###  II.1 Neutrino magnetic moment and spin flip

The interaction of neutrino with an external electromagnetic field catalyses
four phenomenon: (a) magnetic moment, (b) electric moment, (c) neutrino charge
and (d) anapole moment. In this work, we focus on the magnetic moment of
neutrino ($\mu_{\nu}$). In MESM theories, the magnetic moment of Dirac
neutrino is diagonal and is given by [13]

$\mu_{ii}^{D}=\frac{3eG_{F}m_{i}}{8\sqrt{2}\pi^{2}}\approx\frac{3.2\times
10^{-19}m_{i}}{1eV}\mu_{B}.$ (1)

The magnetic moment of Majorana neutrino is off-diagonal, as the diagonal
counterpart is suppressed due to antisymmetric magnetic moment matrix as
required by the CPT invariance. Magnetic moment for Majorana neutrino can be
expressed as [14]

$\mu_{ij}^{M}=-\frac{3eG_{F}}{32\sqrt{2}\pi^{2}}(m_{i}\pm
m_{j})\sum_{l=e,\mu,\tau}U_{li}^{*}U_{lj}\frac{m_{l}^{2}}{m_{W}^{2}}.$ (2)

In eqn. (1) and (2), $G_{F}$, $m_{i}$ and $e$ represent Fermi constant,
neutrino mass and the charge of an electron, respectively. Further, $m_{l}$
and $m_{W}$ denote mass of the $l$-th lepton and the $W$-boson mass,
respectively. Moreover, $U_{ij}$ is the element of the $3\times 3$ PMNS mixing
matrix. From (1) and (2), it can be seen that the value of magnetic moment
depends on the neutrino mass $i.e.$ larger neutrino magnetic moment, which is
necessary to resolve the solar neutrino problem (SNP), requires higher value
of neutrino mass which poses a serious issue as the current limit of neutrino
mass is extremely small. However, this problem is resolved in several new
physics models including ref. [15] in which a charged scalar is added to the
SM particle spectrum to provide a large neutrino magnetic moment and at the
same time satisfying the current upper bound on the neutrino mass [5]. Apart
from this, an additional approximate horizontal symmetry $SU(2)_{H}$ with the
SM gauge group ($SU(3)_{c}\times SU(2)_{L}\times U(1)_{Y}$) is also capable of
enhancing the neutrino magnetic moment without altering its current mass limit
[16]. Non standard interaction (NSI) [3, 17, 18, 19] and several $R$-parity
breaking supersymmetric (SUSY) [20, 21] models are also capable of providing
large magnetic moment of neutrino.

The value of diagonal and transitional magnetic dipole moments are of the
order of $10^{-19}\mu_{B}$ and $10^{-23}\mu_{B}$ respectively [16] as obtained
from MESM theories which are far below the reach of experimental facilities.
The off-diagonal magnetic moment is much suppressed in comparison to the
diagonal magnetic moment because of the GIM mechanism [22, 23]. Several
reactor experiments such as GEMMA [24], TEXONO [25], ROVNO [26] and
accelerator experimental set ups such as LAPMF [27], LSND [28] put constraints
on the upper bound of neutrino magnetic moment which, on average, lies in the
range $10^{-11}-10^{-10}\leavevmode\nobreak\ \mu_{B}$ through the observation
of $\nu-e$ scattering cross section. Recently an estimation provided by
XENON1T experiment shows the transition magnetic moment $\mu_{e\mu}$ to be
existing in the range $1.65-3.42\times 10^{-11}\leavevmode\nobreak\ \mu_{B}$
[16]. In astrophysics and cosmology from the evolution of stars [29], plasmon
decay in the stellar environment [30], supernova events [31] and
${}^{4}_{2}He$ nucleosynthesis [32], it is also possible to impose indirect
constraints on the magnetic moment value. The constraint obtained from the red
giant branch (RGB) of globular cluster is $\mu_{\nu}<4.5\times
10^{-12}\leavevmode\nobreak\ \mu_{B}$ [33] which is smaller by at least an
order than the limits obtained from different terrestrial experiments .

In SM neutrinos are massless electrically neutral Weyl fermions which cannot
interact with electromagnetic fields at tree level which leads to zero
magnetic moment. However, from higher order perturbative corrections,
neutrinos gain non zero magnetic moment via quantum loop effect . The
Hamiltonian for the neutrino-photon interaction is given by [3]

$\mathcal{H}_{EM}=\bar{\nu}(x)\Lambda^{\mu}\nu(x)A_{\mu},$ (3)

where $A_{\mu}$ is the electromagnetic field and $\Lambda_{\mu}$ denotes the
vertex fuction containing the electromagnetic properties of neutrino which is
a $4\times 4$ matrix in the spinor space. For Dirac neutrinos, it is expressed
as [3]

$\displaystyle\Lambda_{\mu}^{D}(q)$ $\displaystyle=$ $\displaystyle
f_{Q}(q^{2})^{D}q_{\mu}\gamma_{5}-f_{M}^{D}(q^{2})i\sigma_{\mu\nu}q^{\nu}+f_{E}^{D}(q^{2})\sigma_{\mu\nu}q^{\nu}\gamma_{5}$
(4)
$\displaystyle+f_{A}^{D}(q^{2})(q^{2}\gamma_{\mu}-q_{\mu}\not{q})\gamma_{5}.$

and for Majorana neutrinos, it is given by [3]

$\displaystyle\Lambda_{\mu}^{M}(q)$ $\displaystyle=$
$\displaystyle(\gamma_{\mu}-q_{\mu}\not{q}/q^{2})[f_{Q}^{M}(q^{2})+f_{A}^{M}(q^{2})q^{2}\gamma_{5}]$
(5)
$\displaystyle-i\sigma_{\mu\nu}q^{\nu}[f_{M}^{M}(q^{2})+if_{E}^{M}(q^{2})\gamma_{5}]\,.$

Here $f_{i}^{D,M}(q^{2})(i=Q,M,E,A)$ are the form factors corresponding to the
four electromagnetic properties, as mentioned earlier, corresponding to Dirac
(D) and Majorana (M) neutrinos, while $q$ denotes the momentum transfer.
$\gamma^{\mu}$ denotes Dirac matrices, $\not{a}=\gamma^{\mu}a_{\mu}$,
$\gamma_{5}=i\gamma_{0}\gamma_{1}\gamma_{2}\gamma_{3}$,
$\sigma_{\mu\nu}=[\gamma_{\mu},\gamma_{\nu}]$. From eqn. (4) and (5) it is
possible to distinguish between Dirac and Majorana neutrinos due to their
distinct form factors and consequently electromagnetic vertex factor. It is to
be noted further that neutrino electromagnetic properties are invariant under
$CP$ transformation for both Dirac and Majorana neutrinos [34], although it is
violated in the standard $V-A$ theory of weak interaction. The form factors at
zero momentum transfer $q^{2}=0$ are gauge independent and finite which can be
measured by the interaction with the external field, $A_{\mu}$ directly at any
experimental set up.

One of the most interesting consequences of electromagnetic properties is the
precession of neutrino spin in presence of an external magnetic moment which
leads to the mixing between LH and RH neutrino current. In the two flavour
framework, the Hamiltonian for neutrino state evolution is expressed as
$4\times 4$ Hermitian matrix. For Dirac neutrino basis $(\nu_{eL},\nu_{\mu
L},\nu_{eR},\nu_{\mu R})$, the Hamiltonian is expressed as [3]

$H_{D}=\begin{pmatrix}\frac{-\Delta
m^{2}}{4E_{\nu}}cos2\theta+V_{e}&\frac{\Delta
m^{2}}{4E_{\nu}}sin2\theta&\mu_{ee}B_{\perp}&\mu_{e\mu}B_{\perp}\\\
\frac{\Delta m^{2}}{4E_{\nu}}sin2\theta&\frac{\Delta
m^{2}}{4E_{\nu}}cos2\theta+V_{\mu}&\mu_{e\mu}B_{\perp}&\mu_{\mu\mu}B_{\perp}\\\
\mu^{*}_{ee}B_{\perp}&\mu^{*}_{\mu e}B_{\perp}&\frac{-\Delta
m^{2}}{4E_{\nu}}cos2\theta&\frac{\Delta m^{2}}{4E_{\nu}}sin2\theta\\\
\mu^{*}_{e\mu}B_{\perp}&\mu^{*}_{\mu\mu}B_{\perp}&\frac{\Delta
m^{2}}{4E_{\nu}}sin2\theta&\frac{\Delta m^{2}}{4E_{\nu}}cos2\theta\\\
\end{pmatrix}.$ (6)

For Majorana neutrinos in the basis of $(\nu_{eL},\nu_{\mu
L},\bar{\nu}_{e},\bar{\nu}_{\mu})$, the evolution matrix is given by [3]

$H_{M}=\begin{pmatrix}\frac{-\Delta
m^{2}}{4E_{\nu}}cos2\theta+V_{e}&\frac{\Delta
m^{2}}{4E_{\nu}}sin2\theta&0&\mu_{e\mu}B_{\perp}\\\ \frac{\Delta
m^{2}}{4E_{\nu}}sin2\theta&\frac{\Delta
m^{2}}{4E_{\nu}}cos2\theta+V_{\mu}&-\mu_{e\mu}B_{\perp}&0\\\ 0&-\mu^{*}_{\mu
e}B_{\perp}&\frac{-\Delta m^{2}}{4E_{\nu}}cos2\theta-V_{e}&\frac{\Delta
m^{2}}{4E_{\nu}}sin2\theta\\\ \mu^{*}_{e\mu}B_{\perp}&0&\frac{\Delta
m^{2}}{4E_{\nu}}sin2\theta&\frac{\Delta m^{2}}{4E_{\nu}}cos2\theta-V_{\mu}\\\
\end{pmatrix}.$ (7)

In equation (6) and (7), $B_{\perp}$ denote transverse magnetic field, while
$\mu$ is the neutrino magnetic moment. Here, $\Delta
m^{2}=m_{2}^{2}-m_{1}^{2}$. $V_{e}$ and $V_{\mu}$ are the potentials
experienced by $\nu_{e}$ and $\nu_{\mu}$, respectively. Further,
$V_{e}=\sqrt{2}G_{F}(n_{e}-n_{n}/2)$ and $V_{\mu}=-\sqrt{2}G_{F}n_{n}/2$,
where $n_{e}$ and $n_{n}$ are electron and neutron number density,
respectively. In our work, we consider the simplification of zero vacuum
mixing $(\theta_{ij}=0)$ which yields the following Hamiltonian reduced to a
$2\times 2$ matrix, given by [35]

$H=\begin{pmatrix}-\Delta m_{12}^{2}/4E+\Delta V/2&\mu_{e\mu}B\\\
\mu_{e\mu}B&\Delta m_{12}^{2}/4E-\Delta V/2\end{pmatrix}.$ (8)

Here $\Delta V=\sqrt{2}G_{F}\rho Y_{e}^{eff}/m_{N}$ with $\rho$ being the
matter density. $Y_{e}$ and $m_{n}$ denote the effective electron fraction and
nucleonic mass, respectively. $Y_{e}^{eff}=(3Y_{e}-1)/2$ for Dirac basis,
while for Majorana basis, $Y_{e}^{eff}=(2Y_{e}-1)/2$. Following eqn. (8) the
neutrino spin flavour evolution equation in presence of an external magnetic
field is expressed as [35]

$\frac{d^{2}\nu_{eL}}{dr^{2}}-\left(\frac{\mu\dot{B}}{\mu
B}+i\zeta\right)\frac{d\nu_{eL}}{dr}+\left[\phi^{2}+i\frac{d\phi}{dr}+(\mu
B)^{2}-i\phi\frac{\mu\dot{B}}{\mu B}+\phi\zeta\right]\nu_{eL}=0,$ (9)

where

$\displaystyle\phi$ $\displaystyle=$ $\displaystyle\frac{-\Delta
m^{2}}{4E}+\frac{1}{\sqrt{2}}G_{F}n_{e},$ $\displaystyle\zeta$
$\displaystyle=$
$\displaystyle\frac{1}{\sqrt{2}}G_{F}n_{n},\leavevmode\nobreak\
\nu_{eL}\rightarrow\nu_{\mu R},$ $\displaystyle=$
$\displaystyle\sqrt{2}G_{F}n_{n},\leavevmode\nobreak\
\nu_{eL}\rightarrow\bar{\nu}_{\mu}\,.$ (10)

The transition probability of LH electron neutrino can be obtained after
solving (9) alongwith (II.1). It is expected that the solution of these
equations will generate different neutrino survival probabilities in the case
of Dirac and Majorana neutrinos.

Now, to analyse this phenomena, a very suitable medium with high magnetic
field is required. For this, we have taken into account an astrophysical
background such as a neutron star which is able to satisfy both of these
criteria. In the next section, we briefly describe the interior structure and
the magnetic field of a neutron star.

###  II.2 Neutron star and magnetic field

The SFP phenomena has been previously analyzed in several other astrophysical
sources [35, 36, 37, 38, 39, 40]. We have carried out our analysis of SFP in
the matter environment of neutron star due to

  * •

presence of high magnetic field [41],

  * •

extensive production of neutrinos [42],

  * •

there is nearly one billion neutron stars present in our Milky Way galaxy
[43],

  * •

magnetars are the neutron stars having magnetic field greater than that
possessed by any other stellar object [44].

All these points make neutron star a very suitable environment for the SFP
analysis.

Neutron stars are produced as the end stage product of stellar evolution of
medium to higher mass stars having mass within the range $1M_{\odot}\leq M\leq
8M_{\odot}$. After complete exhaustion of nuclear fuel at the stellar core,
gravitational collapsing of the stellar structure is initiated and the entire
core region is packed with iron nuclei because of earlier continuos
nucleosynthesis. This leads to photo disintegration of iron nuclei. After
this, via the process of electron capture, a large nuber of neutrons and
neutrinos are produced. With further compression of the matter, neutrons start
to drip out of the nuclei and with time they build up a degeneracy pressure.
At a certain stage the infalling matter from the outer region to the
degenerate neutron core is bounced back from it and generates an ultrasonic
shock wave which propagates from the core towards the outer region and the
entire structure undergoes a violent explosion of supernova with a sudden
burst of trapped neutrinos. The stellar structure left out after the supernova
forms a proto neutron star in which the hydrostatic equilibrium is set up by
the neutron degeneracy pressure. Although the core of the neutron star mainly
consists of neutrons, there can still be a small admixture of protons and
electrons, possibly with other charged leptons and hyperons. Neutron stars are
extremely dense stellar objects with its mass of the order of
$2\leavevmode\nobreak\ M_{\odot}$ packed inside a sphere of $10$ km. The
interior matter density reaches at a few times of nuclear saturation density.
Till now the maximum observed mass of a neutron star is
$2.3\leavevmode\nobreak\ M_{\odot}$ [45, 46, 47].

After the formation of a neutron star, its interior temperature remains $\sim
10^{10}-10^{11}$ K and there is a temperature gradient existing between the
core and the crust. Within the next $10-100$ years, depending on the initial
mass of the progenitor star, a thermal relaxation period is started in which a
cooling wave passes from the center to the surface of the star to set up a
thermal equilibrium inside the star. After this, the neutrino cooling era is
initiated and the star starts to cool down very fast with the emission of
neutrinos. The neutrino cooling occurs by the set of simultaneous interaction
of $\beta$-decay and electron capture by proton, $n\rightarrow
pe\bar{\nu}_{e},\leavevmode\nobreak\ pe\rightarrow n\nu_{e}$ when the proton
fraction in the star is above a certain critical level, following the
inequality, $k_{Fp}\leq k_{Fn}+k_{Fe}$ where $k_{F}$ denotes the wave number
of the fermions. However, below this critical point, the same set of reactions
occur in presence of a bystander nucleon ($N$) which is less efficient
compared to the previous one. The period of neutrino cooling lasts till $\sim
10^{5}$ years, after which the star loses heat via the emission of photons
[48, 49, 50].

As mentioned earlier, a magnetar can possess an extremely high interior
magnetic field, even of the order of $10^{18}$ G. However, the origin of
magnetic field is still an open question in the domain of
astrophysics,although there are several explanation regarding the subject [51,
52]. There exist several magnetic field profiles of neutron stars. One of the
oldest field profile, which has been useful to many earlier works, is given by
[53]

$B(n_{b}/n_{0})=B_{s}+B_{c}[1-exp(-\beta(n_{b}/n_{0})^{\gamma})],$ (11)

where $B_{c}$ and $B_{s}$ represent the magnetic fields at the core and the
surface, respectively, $n_{0}$ is the nuclear saturation density which is
fixed for a particular neutron star equation of state (EoS) and $n_{b}$ is the
total baryon number density. $\beta$ and $\gamma$ are the two free parameters
of the profile which are set to be random values to obtain the desired
magnetic fields at the core and the surface. Recently, a magnetic field
profile has been developed which does not contain any free parameter and is
expressed as [54, 55]

$B(x)=B_{c}(1-1.6x^{2}-x^{4}+4.2x^{6}-2.4x^{8}).$ (12)

Here, $x=r/R$ with $r$ and $R$ being the radial distance from the center of
the star and its radius, respectively. Neutrinos emitted from the neutron star
have energy in the MeV energy range [42] which are possible to be
distinguished by the neutrino detectors placed on Earth.

##  III Result and discussion

In this section we discuss the analysis of neutrino SFP phenomena occurring in
the interior of a magnetar with the consideration of both Dirac and Majorana
neutrinos and inspect when the magnetar system can possibly show a distinction
between these two kinds of scenarios. We consider an isolated magnetar with
mass $2.3\leavevmode\nobreak\ M_{\odot}$ and radius of $11.8$ km. The magnetar
consists of $npe$ matter and possesses a magnetic field of $10^{18}$ G at the
center and $10^{15}$ G at its surface. We consider the universal magnetic
field profile given by eqn. (12) inside the star. The matter configuration
follows nonlinear Walecka model based on relativistic mean field (RMF) theory
containing $\sigma$, $\omega$ and $\rho$-meson [56, 57] and it obeys the well
known GM1 parametrization [58]. The expressions for electron ($n_{e}(r)$) and
neutron number density ($n_{n}(r)$) are obtained from the nonlinear fit, given
by

$\displaystyle n_{e}(r)$ $\displaystyle=$ $\displaystyle 0.2056-4.502\times
10^{-4}r-2.307\times 10^{-3}r^{2}-3.026\times 10^{-4}r^{3}+8.873\times
10^{-5}r^{4}$ $\displaystyle-7.668\times 10^{-6}r^{5}+2.415\times
10^{-7}r^{6}\,,$ $\displaystyle n_{n}(r)$ $\displaystyle=$ $\displaystyle
0.877+0.01r-0.022\,r^{2}+5.447\times 10^{-3}r^{3}-9.613\times
10^{-4}r^{4}+8.536\times 10^{-5}r^{5}$ (13) $\displaystyle-2.868\times
10^{-6}r^{6}\,.$

Both the number densities are maximum at the centre of the star and decrease
towards the outer regions. In our analysis, we have assumed zero vacuum mixing
($\theta_{12}=0$) and $\Delta m_{12}^{2}=7.5\times 10^{-5}$ eV2. The value of
the neutrino magnetic moment is assented to be $\mu_{\nu}\sim
10^{-11}\mu_{B}$.

Figure 1: Variation of survival spin flip probability ($P_{\nu_{eL}}$) for
Dirac (solid line) and Majorana (dashed line) neutrinos with distance ($R$)
from the center of the star.

Fig. 1 brandishes the clear distinction between Dirac and Majorana neutrinos
near the surface region of the neutron star, around $R\sim 11.5$ km. We
consider an initial flux of $\nu_{eL}$ generated at the inner core of the
magnetar and estimate its survival probability at different radial distances
($R$). It can be espied from the plot that the difference between the two
kinds of neutrinos is not conspicuous near the inner core region where the
matter is ultra dense and resides at the most extreme condition, while the
distinction is maximized at the outer layers of the star. On the surface of
the star, the survival probability for Dirac and Majorana neutrino are
observed to be $\sim 0.64$ and $\sim 0.61$, respectively. Our result implies
that the predicted reduction in the flux of Majorana neutrino is larger by
$5\%$ as compared to the Dirac neutrinos. In case of Majorana neutrinos, the
flux reduction takes place due to its conversion to anti-neutrinos. It is to
be noted here that to distinguish between these two pictures, a powerful
neutrino detector with high efficiency is imperative.

Figure 2: Variation of neutrino spin flip survival probability
($P_{\nu_{eL}}$) with energy ($E$) on the surface of the magnetar ($R=11.76$
km). The left (right) panel depicts the Dirac (Majorana) scenario.

In fig. 2, we display the variation of survival probability with the neutrino
energy, as estimated on the surface of the magnetar. It is deluding from the
two panels of the figure that the survival probability becomes nearly
identical for the neutrinos with higher values of energies ($E\gtrsim 5$ MeV).
It is also apparent that at $E\sim 1$ MeV, the survival probability is $\sim
0.65$ for Dirac neutrinos, while it is $\sim 0.6$ for Majorana neutrinos.
Further, the difference between the Dirac and Majorana neutrinos are well
apprehensible for neutrinos with lower energies. At higher energies, $E\geq 5$
MeV, the survival probability is saturated to a constant value of $\sim 0.9$
for both scenarios.

Figure 3: Variation of survival probability of $\nu_{eL}$ for Dirac neutrinos
with the radial distance ($R$) from the center of the star for different
magnetic moment ($\mu_{\nu}$) values.

In fig. 3 we perform a sensitivity test of neutrino survival probability with
the value of the neutrino magnetic moment in the case of Dirac neutrinos. It
can be observed from the plots that the spin flip survival probability is
extremely sensitive to the value of magnetic moment. It is also avouched that
the oscillatory nature of the probability is increasing gradually with higher
magnetic moments. In the case of Majorana neutrinos, we arrive at similar
conclusions.

##  IV Conclusion

In this work, we study the impingement of the magnetic field of a neutron star
on the propagation of neutrinos produced internally under the assumption that
the neutrino possesses a finite magnetic moment owing to quantum loop
contributions. We consider an isolated magnetar with mass $2.3M_{\odot}$ and
radius $11.76$ km with the central magnetic field of order $10^{18}$ G and the
surface field $\sim 10^{15}$ G. The interior matter is governed by a nonlinear
Walecka model with GM1 parametrization. The value of magnetic moment used in
our analysis is obtained from the measurements of various terrestrial
detectors and different astrophysical and cosmological estimations. The
interaction of the neutrino magnetic moment with the magnetic field of the
neutron star induces the phenomenon of spin flip oscillations. This results in
the reduction of neutrino flux which is $\sim 5\%$ larger for Majorana
neutrinos as compared to Dirac at the surface of the neutron star which is
favorable from the experimental perspective. We find that the distinction
between the Dirac and Majorana nature of neutrinos is visible to a larger
extent near the surface region of the star for low energy neutrinos coming
from the inner core of the magnetar towards its surface. At higher energies,
the distinction between the two kinds of neutrinos is nearly negligible.
Further, the survival probability is extremely sensitive to the assumed value
of the neutrino magnetic moment for both Dirac and Majorana neutrinos. Also,
the oscillatory nature of the survival probability seems to be more
perceptible for larger values of the magnetic moment. Overall, it can be
concluded from our analysis that a magnetar is a very suitable system with its
extreme interior matter conditions and high magnetic field for the analysis of
the phenomenon of spin flavour precession and to discriminate between the
Dirac and Majorana nature of neutrinos.

## References

  * [1] C. Giunti and C. W. Kim, “Fundamentals of Neutrino Physics and Astrophysics,”
  * [2] B. Jamieson [Hyper-Kamiokande], “Future Neutrino Experiments,” [arXiv:2207.05044 [hep-ex]].
  * [3] C. Giunti and A. Studenikin, “Neutrino electromagnetic interactions: a window to new physics,” Rev. Mod. Phys. 87, 531 (2015) [arXiv:1403.6344 [hep-ph]].
  * [4] R. E. Shrock, “Electromagnetic Properties and Decays of Dirac and Majorana Neutrinos in a General Class of Gauge Theories,” Nucl. Phys. B 206, 359-379 (1982)
  * [5] M. Fukugita and T. Yanagida, “A Particle Physics Model for Voloshin-Vysotskii-Okun Solution to the Solar Neutrino Problem,” Phys. Rev. Lett. 58, 1807 (1987)
  * [6] A. G. Beda, V. B. Brudanin, V. G. Egorov, D. V. Medvedev, V. S. Pogosov, M. V. Shirchenko and A. S. Starostin, “The results of search for the neutrino magnetic moment in GEMMA experiment,” Adv. High Energy Phys. 2012, 350150 (2012)
  * [7] M. Agostini et al. [Borexino], “Limiting neutrino magnetic moments with Borexino Phase-II solar neutrino data,” Phys. Rev. D 96, no.9, 091103 (2017) [arXiv:1707.09355 [hep-ex]].
  * [8] O. G. Miranda, D. K. Papoulias, M. Tórtola and J. W. F. Valle, “XENON1T signal from transition neutrino magnetic moments,” Phys. Lett. B 808, 135685 (2020) [arXiv:2007.01765 [hep-ph]].
  * [9] K. Abe et al. [XMASS], “Search for exotic neutrino-electron interactions using solar neutrinos in XMASS-I,” Phys. Lett. B 809, 135741 (2020) [arXiv:2005.11891 [hep-ex]].
  * [10] G. G. Raffelt, “New bound on neutrino dipole moments from globular cluster stars,” Phys. Rev. Lett. 64, 2856-2858 (1990)
  * [11] G. G. Raffelt, “Core Mass at the Helium Flash From Observations and a New Bound on Neutrino Electromagnetic Properties,” Astrophys. J. 365, 559 (1990)
  * [12] G. Raffelt and A. Weiss, “Nonstandard neutrino interactions and the evolution of red giants,” Astron. Astrophys. 264, 536-546 (1992)
  * [13] K. Fujikawa and R. Shrock, “The Magnetic Moment of a Massive Neutrino and Neutrino Spin Rotation,” Phys. Rev. Lett. 45, 963 (1980)
  * [14] P. B. Pal and L. Wolfenstein, “Radiative Decays of Massive Neutrinos,” Phys. Rev. D 25, 766 (1982)
  * [15] M. B. Voloshin and M. I. Vysotsky, “Neutrino Magnetic Moment and Time Variation of Solar Neutrino Flux,” Sov. J. Nucl. Phys. 44, 544 (1986) ITEP-1-1986.
  * [16] K. S. Babu, S. Jana and M. Lindner, “Large Neutrino Magnetic Moments in the Light of Recent Experiments,” JHEP 10, 040 (2020) [arXiv:2007.04291 [hep-ph]].
  * [17] K. J. Healey, A. A. Petrov and D. Zhuridov, “Nonstandard neutrino interactions and transition magnetic moments,” Phys. Rev. D 87, no.11, 117301 (2013) [erratum: Phys. Rev. D 89, no.5, 059904 (2014)] [arXiv:1305.0584 [hep-ph]].
  * [18] D. K. Papoulias and T. S. Kosmas, “Neutrino transition magnetic moments within the non-standard neutrino–nucleus interactions,” Phys. Lett. B 747, 454-459 (2015) [arXiv:1506.05406 [hep-ph]].
  * [19] O. G. Kharlanov and P. I. Shustov, “Effects of nonstandard neutrino self-interactions and magnetic moment on collective Majorana neutrino oscillations,” Phys. Rev. D 103, no.9, 095004 (2021) [arXiv:2010.05329 [hep-ph]].
  * [20] A. Aboubrahim, T. Ibrahim, A. Itani and P. Nath, “Large Neutrino Magnetic Dipole Moments in MSSM Extensions,” Phys. Rev. D 89, no.5, 055009 (2014) [arXiv:1312.2505 [hep-ph]].
  * [21] T. Fukuyama, T. Kikuchi and N. Okada, “Neutrino magnetic moments and minimal supersymmetric SO(10) model,” Int. J. Mod. Phys. A 19, 4825-4834 (2004) [arXiv:hep-ph/0306025 [hep-ph]].
  * [22] C. Giunti and A. Studenikin, “Neutrino electromagnetic properties,” Phys. Atom. Nucl. 72, 2089-2125 (2009) [arXiv:0812.3646 [hep-ph]].
  * [23] Z. z. Xing and Y. L. Zhou, “Enhanced Electromagnetic Transition Dipole Moments and Radiative Decays of Massive Neutrinos due to the Seesaw-induced Non-unitary Effects,” Phys. Lett. B 715, 178-182 (2012) [arXiv:1201.2543 [hep-ph]].
  * [24] A. G. Beda, E. V. Demidova, A. S. Starostin, V. B. Brudanin, V. G. Egorov, D. V. Medvedev, M. V. Shirchenko and T. Vylov, “GEMMA experiment: Three years of the search for the neutrino magnetic moment,” Phys. Part. Nucl. Lett. 7, 406-409 (2010) [arXiv:0906.1926 [hep-ex]].
  * [25] H. T. Wong et al. [TEXONO], “A Search of Neutrino Magnetic Moments with a High-Purity Germanium Detector at the Kuo-Sheng Nuclear Power Station,” Phys. Rev. D 75, 012001 (2007) [arXiv:hep-ex/0605006 [hep-ex]].
  * [26] A. I. Derbin, A. V. Chernyi, L. A. Popeko, V. N. Muratova, G. A. Shishkina and S. I. Bakhlanov, “Experiment on anti-neutrino scattering by electrons at a reactor of the Rovno nuclear power plant,” JETP Lett. 57, 768-772 (1993)
  * [27] R. C. Allen, H. H. Chen, P. J. Doe, R. Hausammann, W. P. Lee, X. Q. Lu, H. J. Mahler, M. E. Potter, K. C. Wang and T. J. Bowles, et al. “Study of electron-neutrino electron elastic scattering at LAMPF,” Phys. Rev. D 47, 11-28 (1993)
  * [28] L. B. Auerbach et al. [LSND], “Measurement of electron - neutrino - electron elastic scattering,” Phys. Rev. D 63, 112001 (2001) [arXiv:hep-ex/0101039 [hep-ex]].
  * [29] A. Heger, A. Friedland, M. Giannotti and V. Cirigliano, “The Impact of Neutrino Magnetic Moments on the Evolution of Massive Stars,” Astrophys. J. 696, 608-619 (2009) [arXiv:0809.4703 [astro-ph]].
  * [30] A. V. Borisov and P. E. Sizin, “Plasmon decay to a neutrino pair via neutrino electromagnetic moments in a strongly magnetized medium,” [arXiv:1406.3301 [hep-ph]].
  * [31] A. de Gouvea and S. Shalgar, “Effect of Transition Magnetic Moments on Collective Supernova Neutrino Oscillations,” JCAP 10, 027 (2012) [arXiv:1207.0516 [astro-ph.HE]].
  * [32] N. Vassh, E. Grohs, A. B. Balantekin and G. M. Fuller, “Majorana Neutrino Magnetic Moment and Neutrino Decoupling in Big Bang Nucleosynthesis,” Phys. Rev. D 92, no.12, 125020 (2015) [arXiv:1510.00428 [astro-ph.CO]].
  * [33] N. Viaux, M. Catelan, P. B. Stetson, G. Raffelt, J. Redondo, A. A. R. Valcarce and A. Weiss, “Particle-physics constraints from the globular cluster M5: Neutrino Dipole Moments,” Astron. Astrophys. 558, A12 (2013) [arXiv:1308.4627 [astro-ph.SR]].
  * [34] C. Broggini, C. Giunti and A. Studenikin, “Electromagnetic Properties of Neutrinos,” Adv. High Energy Phys. 2012, 459526 (2012) [arXiv:1207.3980 [hep-ph]].
  * [35] S. Joshi and S. R. Jain, “Neutrino spin-flavor oscillations in solar environment,” [arXiv:1906.09475 [hep-ph]].
  * [36] D. Yilmaz, “Combined effect of NSI and SFP on solar electron neutrino oscillation,” Adv. High Energy Phys. 2016, 1435191 (2016) [arXiv:1601.03161 [hep-ph]].
  * [37] A. Lichkunov, A. Popov and A. Studenikin, “Neutrino eigenstates and flavour, spin and spin-flavour oscillations in a constant magnetic field,” [arXiv:2012.06880 [hep-ph]].
  * [38] H. Sasaki and T. Takiwaki Phys. Rev. D 104 023018 (2021) Adv. High Energy Phys. 2012, 459526 (2012)
  * [39] J. Adhikary, A. K. Alok, A. Mandal, T. Sarkar and S. Sharma, “Neutrino spin-flavour precession in magnetized white dwarf,” [arXiv:2207.09485 [hep-ph]].
  * [40] A. K. Alok, N. R. S. Chundawat and A. Mandal, “Cosmic neutrino flux and spin flavor oscillations in intergalactic medium,” [arXiv:2207.13034 [hep-ph]].
  * [41] Q. H. Peng and H. Tong, “The physics of strong magnetic fields in neutron stars,” Mon. Not. Roy. Astron. Soc. 378, 159 (2007) [arXiv:0706.0060 [astro-ph]].
  * [42] D. G. Yakovlev, A. D. Kaminker, O. Y. Gnedin and P. Haensel, “Neutrino emission from neutron stars,” Phys. Rept. 354, 1 (2001) [arXiv:astro-ph/0012122 [astro-ph]].
  * [43] A. A. Chrimes, A. J. Levan, P. J. Groot, J. D. Lyman and G. Nelemans, “The Galactic neutron star population – I. An extragalactic view of the Milky Way and the implications for fast radio bursts,” Mon. Not. Roy. Astron. Soc. 508, no.2, 1929-1946 (2021) [arXiv:2105.04549 [astro-ph.GA]].
  * [44] V. M. Kaspi and A. Beloborodov, “Magnetars,” Ann. Rev. Astron. Astrophys. 55, 261-301 (2017) [arXiv:1703.00068 [astro-ph.HE]].
  * [45] D. P. Menezes, “A Neutron Star Is Born,” Universe 7, no.8, 267 (2021) [arXiv:2106.09515 [astro-ph.HE]].
  * [46] I. Vidaña, “Short introduction to the physics of neutron stars,” EPJ Web Conf. 227, 01018 (2020)
  * [47] J. M. Lattimer, “Introduction to neutron stars,” AIP Conf. Proc. 1645, no.1, 61-78 (2015)
  * [48] D. G. Yakovlev, O. Y. Gnedin, M. E. Gusakov, A. D. Kaminker, K. P. Levenfish and A. Y. Potekhin, “Neutron star cooling,” Nucl. Phys. A 752, 590-599 (2005) [arXiv:astro-ph/0409751 [astro-ph]].
  * [49] S. Tsuruta, “NEUTRON STAR COOLING,” Conf. Proc. C 861214, 499-503 (1986)
  * [50] C. J. Pethick, “Cooling of neutron stars,” Rev. Mod. Phys. 64, 1133-1140 (1992)
  * [51] H. C. Spruit, “Origin of neutron star magnetic fields,” AIP Conf. Proc. 983, no.1, 391-398 (2008) [arXiv:0711.3650 [astro-ph]].
  * [52] A. Reisenegger, “Origin and evolution of neutron star magnetic fields,” [arXiv:astro-ph/0307133 [astro-ph]].
  * [53] D. Bandyopadhyay, S. Chakraborty and S. Pal Phys. Rev. Lett. 79 2176 (1997)
  * [54] D. Chatterjee, J. Novak and M. Oertel, “Magnetic field distribution in magnetars,” Phys. Rev. C 99, no.5, 055811 (2019) [arXiv:1808.01778 [nucl-th]].
  * [55] D. Chatterjee, J. Novak and M. Oertel, “Structure of ultra-magnetised neutron stars,” Eur. Phys. J. A 57, no.8, 249 (2021) [arXiv:2108.13733 [nucl-th]].
  * [56] A. M. S. Santos and D. P. Menezes, “Dense and hot matter within the nonlinear Walecka model,” Phys. Rev. C 69, 045803 (2004)
  * [57] H. Mueller and B. D. Serot, “Relativistic mean field theory and the high density nuclear equation of state,” Nucl. Phys. A 606, 508-537 (1996) [arXiv:nucl-th/9603037 [nucl-th]].
  * [58] N. K. Glendenning and S. A. Moszkowski, “Reconciliation of neutron star masses and binding of the lambda in hypernuclei,” Phys. Rev. Lett. 67, 2414-2417 (1991)