# 3D Shape Variational Autoencoder Latent Disentanglement
via Mini-Batch Feature Swapping for Bodies and Faces

Simone Foti Bongjin Koo Danail Stoyanov Matthew J. Clarkson
University College London
<EMAIL_ADDRESS>
###### Abstract

Learning a disentangled, interpretable, and structured latent representation
in 3D generative models of faces and bodies is still an open problem. The
problem is particularly acute when control over identity features is required.
In this paper, we propose an intuitive yet effective self-supervised approach
to train a 3D shape variational autoencoder (VAE) which encourages a
disentangled latent representation of identity features. Curating the mini-
batch generation by swapping arbitrary features across different shapes allows
to define a loss function leveraging known differences and similarities in the
latent representations. Experimental results conducted on 3D meshes show that
state-of-the-art methods for latent disentanglement are not able to
disentangle identity features of faces and bodies. Our proposed method
properly decouples the generation of such features while maintaining good
representation and reconstruction capabilities. Our code and pre-trained
models are available at github.com/simofoti/3DVAE-SwapDisentangled.

Figure 1: Schematic Description of the Proposed Method. Left: an arbitrary
identity feature is selected and a mini-batch of vertices ($\mathbf{X}_{ij}$)
is created by swapping features across different 3D shapes. Colours represent
identities. Notice that features from the same identity have the same colour.
Centre: a 3D-VAE ($\\{E,G\\}$) encodes $\mathbf{X}_{ij}$ in its latent
representations $\mathbf{z}_{ij}=(\mathbf{z}^{f}_{ij}|\mathbf{z}^{c}_{ij})$,
which are subsequently decoded into $\mathbf{X}^{\prime}_{ij}$. In this case
$f$ corresponds to the nose. Therefore, while $\mathbf{z}^{f}_{ij}$ controls
the shape of the nose, $\mathbf{z}^{c}_{ij}$ controls the shape of the rest of
the face. Right: visual representation of all the different mesh features for
which we seek to obtain a disentangled latent representation.

##  1 Introduction

The generation of 3D human faces and bodies is a complex task with multiple
potential applications ranging from movie and game productions, to augmented
and virtual reality, as well as healthcare applications. Currently, the
generation procedure is either manually performed by highly skilled artists or
it involves semi-automated avatar design tools. Even though these tools
greatly simplify the design process, they are usually limited in flexibility
because of the intrinsic constraints of the underlying generative models [17].
Blendshapes [28, 31, 38], 3D morphable models [5, 33, 25], autoencoders [35,
16, 3, 8], and generative adversarial networks [7, 15, 24, 1] are currently
the most used generative models, but they all share one particular issue: the
creation of local features is difficult or even impossible. In fact, not only
do generative coefficients (or latent variables) lack any semantic meaning,
but they also create global changes in the output shape. For this reason, we
focus on the problem of 3D shape creation by enforcing disentanglement among
sets of generative coefficients controlling the identity of a character.

Following [4, 18, 19] we define a disentangled latent representation as one
where changes in one latent unit affects only one factor of variation while
being invariant to changes in other factors. More interpretable and structured
latent representations of data that expose their semantic meaning have been
widely researched in the artificial intelligence community [18, 19, 22, 13,
10], but this is still an open problem especially for generative models of 3D
shapes [3]. Given the superior representation capabilities, the reduced number
of parameters, and the stable training procedures, we decide to focus our
study on deep-learning-based generative models and in particular on
variational autoencoders (VAEs). In this field, recent work has tried to
address the latent disentanglement problem for 3D shapes and managed to
decouple the control over identity and expression (or pose) [3, 8, 1], but
they are still unable to properly disentangle identity features. Some success
has been achieved in the generation of 3D shapes of furniture [29, 43], but
the structural variability of the data requires complex architectures with
multiple encoders and decoders for different furniture parts. In contrast, our
method relies on a single VAE which is trained by curating the mini-batch
generation procedure and with an additional loss. The intuition behind our
method is that if we swap features (e.g. nose, ears, legs, arms, etc.) across
the input data in a controlled manner (Fig. 3D Shape Variational Autoencoder
Latent Disentanglement  via Mini-Batch Feature Swapping for Bodies and Faces,
Left), we not only know a priori which shapes within a mini-batch have (do not
have) the same feature, but we also know which are (are not) created from the
same face (body). These differences and similarities across shapes should be
captured in the latent representation. Therefore, assuming that different
subsets of latent variables correspond to different features, we can partition
the latent space and leverage the structure of the input batch to encourage a
more disentangled, interpretable, and structured representation.

With the objective of building a model capable of generating 3D meshes, we
define our VAE architecture extending [16]. This state-of-the-art model proved
to be fast and capable of better capturing non-linear representations of 3D
meshes, while leveraging very intuitive convolutional operators characterised
by a reduced number of parameters. Nonetheless, the network choice is
arbitrary and we expect our method to be working also with other network
configurations and operators. Even though we consider meshes as our primary
data structures, it is also worth noting that, by providing semantic
segmentations of the different features, our method is applicable to voxel- or
point-cloud-based generative models. We believe that the generality of the
proposed method is particularly important in the current geometric deep
learning field, where definitions of 3D convolutions and pooling operators are
still an open problem.

To summarise, the key contributions of our approach are: (_i_) the definition
of a new mini-batching procedure based on feature swapping, (_ii_) the
introduction of a novel loss function capable of leveraging shape differences
and similarities within each mini-batch, and (_iii_) the consequent creation
of a 3D-VAE capable of generating 3D meshes from a more interpretable and
structured latent representation.

##  2 Related Work

In this section, we first discuss existing work on 3D generative models of
faces and bodies, followed by state-of-the art approaches for latent
disentanglement of autoencoder-based generative models.

#### Generative Models

Blendshape models manually created by artists linearly interpolate local
features between two or more manually selected shapes. These models are common
as consumer-level avatar design tools adopted by several videogame engines.
Even though they guarantee control over the generation of local features, they
are very large models usually built with only a few subjects and are capable
of offering only very limited flexibility and expressivity [17]. A widespread
approach to overcome these limitations is to rely on linear statistical 3D
morphable models (3DMM). These models are based upon the identity space of a
population of 3D shapes, and are usually built by applying a principal
component analysis (PCA) over the entire dataset. They are always built with
the assumption that shapes are registered between each other and in dense
point correspondence. This allows the generation of meaningful and
morphologically realistic shapes as linear combinations of training data. This
technique was pioneered by [5] and further developed and adopted by many
researchers [12]. Interestingly, [17] divided the face in different local
patches and trained a PCA model for each region in order to control the
generation of different facial features. The generation of new faces and
interactive face sculpting are then achieved through a constrained
optimisation. Recently, [33, 32] combined multiple 3DMMs to create the first
combined, large-scale, full-head morphable model. In particular, the universal
head model (Uhm) [33] combines the Large-Scale Face Model (LSFM) [6], which
was built with face scans from $10,000$ subjects, with the LYHM head model
[9]. In [32] it was extended by combining also a detailed ear model, eye and
eye region models, as well as basic models for mouth, teeth, tongue and inner
mouth cavity. As further detailed in Sec. 4, given the high diversity of Uhm
we decided to train our face model on heads from [33].

PCA-based models and blendshapes are often combined. For instance, Smpl[28]
learns linear PCA models of male and female body shapes from approximately
$2,000$ scans per gender, and subsequently uses the resulting principal
components as body shape blendshapes capable of efficiently controlling the
identity of a subject. The same approach is used also by Star [31], which not
only creates more realistic pose deformations than [28], but it also leverages
$10,000$ additional scans to improve the generalisation capabilities of the
model. Given its better generalisation with respect to other state-of-the-art
methods, we trained our body model on shapes generated from Star.

Recently, advances in the geometric deep learning community allowed to
efficiently define convolutional operators on 3D data such as meshes and
point-clouds. [35] is the first AE for 3D meshes of faces based on a graph
convolutional neural network. This model was built using significantly less
parameters than state-of-the-art PCA-based models and showed lower
reconstruction errors as well as better generalisation to unseen faces. Other
AE-based architectures leveraging different convolutional operators over
different datasets were subsequently introduced [3, 8, 26, 44, 46]. Despite
the remarkable performance of these models, we decided to adopt the base
architecture of [16], which further improved upon previous methods by defining
a more intuitive convolutional operator based on dilated spiral convolutions
(i.e. spiral++ convolution).

An alternative line of work considers generative adversarial networks (GANs)
instead of autoencoders. The first GAN operating on 3D meshes was proposed in
[7] and it allowed to disentangle identity from expression generative factors.
Other methods usually map 3D shapes to the image domain and then train
adversarial networks with traditional 2D convolutions [1, 15, 24]. GAN models
are generally able to generate more detailed and realistic 3D shapes than
autoencoders at the cost of being more unstable and difficult to train.

As aforementioned in Sec. 1, with the exception of artistically-created
blendshape models and [17], none of the other methods here described allow to
control local changes during the generation process because their generative
coefficients lack any semantic meaning, are not easily interpretable and are
not properly disentangled.

#### Autoencoder Latent Disentanglement

Latent disentanglement for the generation of 3D shapes has been explored
mostly in relation to the disentanglement of identity and pose generative
factors. [3] created a two level architecture combining a point-cloud AE with
a VAE where the latent space is successfully partitioned by relying on
multiple geometric losses and disentanglement penalties. [8] achieves similar
results by training a point-cloud VAE while controlling the amount of
distortion incurring in the construction of the latent space. As mentioned in
Sec. 1, these methods are not capable of disentangling generative factors
controlling the identity of different subjects. Methods such as [43, 29], on
the other hand, are able to control different parts of furniture meshes, but
they require complex architectures with multiple encoders and decoders
controlling the different parts. Even though part hierarchies have to be
considered in the model formulation, differently from faces and bodies,
discontinuities between different parts are not a problem when generating
furniture.

Research on latent disentanglement of AEs often focuses on the scenario in
which only raw observations are available without any supervision about the
generative factors, and it is usually performed on images. [18] proposed a
simple modification to a VAE [21]. By increasing the weight of the
Kullback–Leibler (KL) divergence, $\beta$-VAE showed better latent
disentanglement properties at the expense of a reduced quality of the
generated samples. Subsequent work, such as [23, 19], tried to overcome this
limitation. The DIP-VAE [23] leverages an additional regularisation term on
the expectation of the approximate posterior over observed data. The Factor
VAE [19] encourages the latent distribution to be factorial, and therefore
independent across dimensions, by using a latent discriminator and by adding a
total correlation term in the VAE loss function. An interesting approach to
encourage latent variables to represent predefined transformations was
proposed in [22], where mini-batches are created combining active and inactive
transformations and gradients influencing the latent are modified during
backpropagation. However, this method requires synthetic datasets created with
known properties that can be used during training to achieve the
disentanglement. Recently, [13] proposed a VAE in which the objective function
is hierarchically decomposed to control the relative levels of statistical
independence between groups of variables and for individual variables in the
same group. The recursive formulation of the loss introduces additional terms
for any variable that has to be disentangled and works only where the factors
of variation are uncorrelated scalar variables, a requirement that hampers the
applicability of the model in real-world scenarios. Finally, the Guided-VAE
[10] in its unsupervised setting leverages a secondary decoder that learns a
set of PCA bases that are used to guide the training over simple geometrical
shapes. Nevertheless, being the secondary decoder based on a PCA, latent
variables suffer the same problems of PCA models.

Among the aforementioned methods for latent disentanglement the DIP-VAE [23]
and Factor VAE [19] showed good disentanglement performance also on in-the-
wild image datasets while requiring only minor modifications to the VAE
formulation. For this reason, we implemented a DIP-VAE and a Factor VAE
operating on meshes and compared them against our method.

##  3 Method

The proposed method (Fig. 3D Shape Variational Autoencoder Latent
Disentanglement  via Mini-Batch Feature Swapping for Bodies and Faces) allows
us to obtain more interpretable and structured latent representations for
self-supervised 3D generative models. This is achieved by training a mesh-
convolutional variational autoencoder (Sec. 3.1) with a mini-batch controlled
feature swapping procedure and a latent consistency loss (Sec. 3.2).

Figure 2: Examples of feature swapping for different features and different
subjects.

###  3.1 Mesh Variational Autoencoder

A manifold triangle mesh is defined as
$\mathcal{M}=\\{\mathbf{X},\boldsymbol{\mathcal{E}},\boldsymbol{\mathcal{F}}\\}$,
where $\mathbf{X}\in\mathbb{R}^{N\times 3}$ is its vertex embedding,
$\boldsymbol{\mathcal{E}}\in\mathbb{N}^{\varepsilon\times 2}$ is the edge
connectivity that defines its topology, and
$\boldsymbol{\mathcal{F}}\in\mathbb{N}^{\Gamma\times 3}$ are its triangular
faces. Assuming that meshes share the same topology across the entire dataset,
$\boldsymbol{\mathcal{E}}$ and $\boldsymbol{\mathcal{F}}$ are constant and
meshes differ from one another only for the position of their vertices, which
are assumed to be consistently aligned, scaled, and with point-wise
correspondences. Since traditional convolutional operators are not compatible
with the non-Euclidean nature of meshes, we build our generative model with
the simple yet efficient approach defined in [16]. Convolution operators are
thus defined as learnable functions over pre-computed dilated spiral sequences
[16]. Pooling and un-pooling operators are defined as sparse matrix
multiplications with pre-computed transformations that are obtained with a
quadric sampling procedure [16, 35] (see Supplementary Materials).

Our 3D-VAE is built as an encoder-decoder pair (Fig. 3D Shape Variational
Autoencoder Latent Disentanglement  via Mini-Batch Feature Swapping for Bodies
and Faces, Centre), where the decoder is used as a generative model and is
referred to as generator. Following this convention, we define our
architecture as a pair of non-linear functions $\\{E,G\\}$. Let $\mathcal{X}$
be the vertex embedding domain and $\mathcal{Z}$ the latent distribution
domain, we have $E:\mathcal{X}\rightarrow\mathcal{Z}$ defined as a variational
distribution $q(\mathbf{z}|\mathbf{X})$ that approximates the intractable
model posterior distribution, and $G:\mathcal{Z}\rightarrow\mathcal{X}$
described by the likelihood $p(\mathbf{X}|\mathbf{z})$. Throughout the entire
network, each spiral++ convolutional layer is followed by an ELU activation
function. However, in $E$ convolutions are interleaved with pooling layers and
in $G$ by un-pooling layers. There are also three fully connected layers: two
of them are the last layers of $E$ predicting the mean and the diagonal
covariance of the variational distribution, the other is the first layer of
$G$ and transforms $\mathbf{z}\sim\mathcal{Z}$ back into a low-dimensional
mesh that can be processed by mesh convolutions.

During training, the following loss is minimised:

$\mathcal{L}_{VAE}=\mathcal{L}_{R}+\alpha\mathcal{L}_{L}+\beta\mathcal{L}_{KL}$
(1)

where $\alpha$ and $\beta$ are weighting constants.
$\mathcal{L}_{R}=\frac{1}{N}\sum_{n=1}^{N}\|\mathbf{x}^{\prime}_{n}-\mathbf{x}_{n}\|_{2}^{2}$
is the mean squared error between the input ($\mathbf{x}_{n}\in\mathbf{X}$)
and the corresponding output
($\mathbf{x}^{\prime}_{n}\in\mathbf{X}^{\prime}=G(E(\mathbf{X}))=G(\mathbf{z})$)
vertices. This reconstruction loss encourages the output of the VAE to be as
close as possible to its input.
$\mathcal{L}_{KL}=KL[q(\mathbf{z}|\mathbf{X})||p(\mathbf{z})]$ is the
Kullback–Leibler (KL) divergence pushing the variational distribution towards
the prior distribution $p(\mathbf{z})$, which is defined as a standard
spherical Gaussian distribution. Finally, $\mathcal{L}_{L}$ is a smoothing
term based on the uniform Laplacian [30] that is computed on the output
vertices as:

$\mathcal{L}_{L}=\frac{1}{N}\sum_{n=1}^{N}\|\mathbf{\delta}_{n}\|_{2}\quad\text{with}\;\mathbf{\delta}_{n}=\frac{1}{|\mathcal{N}_{n}|}\sum_{e\in\mathcal{N}_{n}}\mathbf{x}^{\prime}_{e}-\mathbf{x}^{\prime}_{n}$

where $\mathbf{\delta}_{n}$ is the Laplacian of the n-th output vertex, and
$\mathcal{N}_{n}$ the set of its neighbouring vertices with cardinality
$|\mathcal{N}_{n}|$. $\mathcal{L}_{L}$ is efficiently computed by relying on
matrix operators. Concretely, we have
$\mathbf{\Delta}=[\mathbf{\delta}_{1},...,\mathbf{\delta}_{N}]^{T}=\mathbf{L}\mathbf{X}^{\prime}$,
where $\mathbf{L}=\mathbf{I}-\mathbf{D}^{-1}\mathbf{A}$ is the Laplacian
operator with random walk normalisation, $\mathbf{A}\in\mathbb{N}^{N\times N}$
is the adjacency matrix and $\mathbf{D}\in\mathbb{R}^{N\times N}$ is the
diagonal degree matrix with $D_{aa}=\sum_{a}A_{ab}$. Note that vertices are
normalised by subtracting the per-vertex mean of the training set and dividing
the result by the per-vertex standard deviation of the training set, thus the
losses in Eq. 1 are computed on normalised vertices. Also, all loss terms are
reduced across mini-batches with a mean reduction.

Figure 3: Random samples and vertex-wise distances showing the effects of
traversing three randomly selected latent variables (see Supplementary
Material to observe the effects for all the latent variables.)

###  3.2 Mini-Batch Feature Swapping and Latent Consistency Loss

We aim to obtain a generative model where vertices corresponding to specific
mesh features are controlled by a predefined set of latent variables.
Therefore, we start by defining $F$ arbitrary mesh features on a mesh template
(Fig. 3D Shape Variational Autoencoder Latent Disentanglement  via Mini-Batch
Feature Swapping for Bodies and Faces, Right). Features are manually defined
by colouring mesh vertices. Since vertices have point-wise correspondences
(Sec. 3.1), features can be easily identified for every other mesh in the
dataset without manually segmenting them. This allows us to swap features from
one mesh to another by replacing the vertices corresponding to the selected
feature (Fig. 2).

Feature swapping is at the core of our method and it allows us to curate the
mini-batch generation in order to properly shape and constrain the latent
representation of each mesh. Each mini-batch of size $B$ can be thought of as
a squared matrix of size $\sqrt{B}\times\sqrt{B}$, where each element
$\mathbf{X}_{ij}$ is the vertex embedding of a different mesh. As it can be
seen from Fig. 3D Shape Variational Autoencoder Latent Disentanglement  via
Mini-Batch Feature Swapping for Bodies and Faces (Left), while elements on the
diagonal of this matrix are loaded from the dataset, the remaining elements
are created online by swapping features. Every time a mini-batch is created, a
feature is randomly selected and swapped. Therefore, each row of the matrix
contains the same mesh with different features, while each column contains
different meshes with the same feature. Interestingly, the naive
implementation of the feature swapping causes visible surface discontinuities
in most input meshes (Fig. 2), but discontinuities are not present in
reconstructed meshes thanks to the Laplacian regulariser in Eq. 1.

Obviously, when a mini-batch is encoded we obtain a batched latent. As we can
see in Fig. 3D Shape Variational Autoencoder Latent Disentanglement  via Mini-
Batch Feature Swapping for Bodies and Faces (Centre), for each
$\mathbf{X}_{ij}$ we have a corresponding $\mathbf{z}_{ij}\sim
E(\mathbf{X}_{ij})$ which is evenly split in $F$ subsets of latent variables,
one for each mesh feature
($\mathbf{z}_{ij}=\\{\mathbf{z}^{\omega}_{ij}\\}_{\omega=1}^{F}$). Note that
even though every latent subset $\mathbf{z}^{\omega}_{ij}$ has the same number
of variables, uneven splits are also admissible.

Every time a mini-batch is created by swapping a feature $f$, we can define
$\mathbf{z}_{ij}=(\mathbf{z}^{f}_{ij}|\mathbf{z}^{c}_{ij})$.
$\mathbf{z}^{f}_{ij}$ is the subset of latent variables controlling the
feature swapped across the current mini-batch. $\mathbf{z}^{c}_{ij}$ is the
part that controls everything else and is defined as
$\mathbf{z}^{c}_{ij}=\\{\mathbf{z}^{\omega}_{ij}\\}_{\omega=1}^{F}\setminus\\{\mathbf{z}^{f}_{ij}\\}$.
Inspired by both triplet losses and [37], and thanks to our curated mini-
batching, we can enforce differences and similarities in the latent
representation of the different $\mathbf{X}_{ij}$ by requiring matched
$\mathbf{z}^{\omega}_{ij}$ pairs to have a distance in latent space that is
smaller by a margin, $\eta$, than the distance for unmatched pairs. We
traverse the diagonal of the mini-batch latent matrix and compare all the
elements on the row containing the diagonal element $\mathbf{z}_{ss}$ with
those in the column containing $\mathbf{z}_{ss}$ ($\forall
s\in\\{1,...,\sqrt{B}\\}$). When considering $\mathbf{z}^{f}_{ij}$ we enforce
latent similarities across columns and latent differences across rows by
evaluating:
$\|\mathbf{z}^{f}_{ps}-\mathbf{z}^{f}_{qs}\|_{2}^{2}+\eta_{1}\leq\|\mathbf{z}^{f}_{sp}-\mathbf{z}^{f}_{sq}\|_{2}^{2}$,
$\forall s,p,q\in\\{1,...,\sqrt{B}\\}$ with $p\neq q$. This is justified by
the fact that elements in $\mathbf{X}_{ij}$ have the same mesh feature across
columns and different mesh features across rows. Vice versa, when considering
$\mathbf{z}^{c}_{ij}$, which controls all the other mesh features for the
current mini-batch, we enforce similarities row-wise and differences column-
wise by evaluating:
$\|\mathbf{z}^{c}_{sp}-\mathbf{z}^{c}_{sq}\|_{2}^{2}+\eta_{2}\leq\|\mathbf{z}^{c}_{ps}-\mathbf{z}^{c}_{qs}\|_{2}^{2},$
$\forall s,p,q\in\\{1,...,\sqrt{B}\\}$ with $p\neq q$. We thus define our
latent consistency loss as:

$\begin{split}\mathcal{L}_{c}&=\gamma\sum^{\sqrt{B}}_{\begin{subarray}{c}s,p,q=1\\\
p\neq
q\end{subarray}}\max\Big{[}0,\|\mathbf{z}^{f}_{ps}-\mathbf{z}^{f}_{qs}\|_{2}^{2}-\|\mathbf{z}^{f}_{sp}-\mathbf{z}^{f}_{sq}\|_{2}^{2}+\eta_{1}\Big{]}+\\\
&\qquad+\max\Big{[}0,\|\mathbf{z}^{c}_{sp}-\mathbf{z}^{c}_{sq}\|_{2}^{2}-\|\mathbf{z}^{c}_{ps}-\mathbf{z}^{c}_{qs}\|_{2}^{2}+\eta_{2}\Big{]}\end{split}$
(2)

where $\gamma=\frac{1}{B\sqrt{B}-B}$ is a batch normalisation term that
considers all the latent distances comparisons performed while computing
$\mathcal{L}_{c}$. Combining Eq. 1 with Eq. 2 and said $\kappa\in\mathbb{R}$ a
weighting coefficient, we can formulate the total loss as:

$\mathcal{L}=\mathcal{L}_{VAE}+\kappa\mathcal{L}_{c}=\mathcal{L}_{R}+\alpha\mathcal{L}_{L}+\beta\mathcal{L}_{KL}+\kappa\mathcal{L}_{c}$
(3)

##  4 Experiments

Figure 4: Effects of traversing each latent variable across different mesh
features. For each latent variable (abscissas) we represent the per-feature
mean distances computed after traversing the latent variable from its minimum
to its maximum value. For each latent variable, we expect a high mean distance
in one single feature and low values for all the others.

#### Datasets

Our main objective is to train a generative model capable of generating
different identities from a set of feature-disentangled latent variables. For
our experiments we require datasets containing as many subjects as possible in
a neutral expression. However, most open source datasets for 3D shapes of
faces, bodies, or animals contain only a limited number of subjects captured
in different expressions or poses (e.g. Mpi-Dyna [34], Smpl [28], Surreal
[39], Coma [35], Smal [47], etc.). For this reason, we rely on two linear
models that were built using a conspicuous number of subjects and that are
released for non-commercial scientific research purposes: Uhm [33] and Star
[31] (Sec. 2). From these models we randomly generate $10,000$ meshes and
create one dataset for faces and one for bodies. We use $90\%$ of the data for
training, $5\%$ for validation, and $5\%$ for testing.

#### Implementation Details

All networks were implemented in PyTorch and trained for $40$ epochs using the
Adam optimiser [20] with a fixed learning rate of $lr=1e^{-4}$ and mini-batch
size $B=16$ (note that the feature swapping is applied to our method only).
Spiral convolutions 111The SpiralNet++ implementation was made available with
an MIT license. had spiral length of $9$ and spiral dilation of $1$. The last
convolutional layer of $E$ and the first of $G$ had $64$ features while all
the others $32$. The sampling factors used during the quadric sampling for the
creation of the up- and down-sampling transformation matrices were set to $4$.
Since the two datasets have a significantly different number of vertices
($N_{faces}=71,928$ and $N_{bodies}=6,890$), networks operating on faces have
$4$ convolutional layers interleaved with sampling operators in both $E$ and
$G$, while networks operating on bodies have only $3$. For the same reason
latent sizes are different: $60$ variables for faces and $33$ for bodies.
Considering that the face template was segmented in $12$ regions and the body
template in $11$, each $\mathbf{z}^{\omega}_{ij}$ has $5$ variables for faces
and $3$ for bodies. The weight of the Laplacian regulariser was set to
$\alpha=1$, while the latent consistency weight was $\kappa=0.5$ for faces and
$\kappa=1$ for bodies. $\eta_{1}$ and $\eta_{2}$ were set to
$\eta_{1}=\eta_{2}=0.5$ for both datasets. Training was performed on a single
Nvidia Quadro P5000 for faces and on an Nvidia GeForce GTX 1050Ti for bodies.
We run approximately $120$ experiments in $25$ GPU days.

#### Comparison with Other Methods

Table 1: Quantitative comparison between our model and other state-of-the-art methods for self-supervised latent disentanglement. All methods were trained on the same face dataset. Mean and Max Rec. refer to the the mean and maximum average per-vertex reconstruction errors across the test set. Values are computed in millimetres. The diversity is computed as detailed in Sec. 4. All the other metrics for the evaluation of the generation capabilities were introduced in [42]. Method |  Mean Rec. ($\downarrow$) |  Max Rec. ($\downarrow$) |  Diversity ($\uparrow$) | JSD ($\downarrow$) | MMD ($\downarrow$) | COV(%, $\uparrow$) | 1-NNA ($\Delta$%, $\downarrow$)
---|---|---|---|---|---|---|---
CD | EMD | CD | EMD | CD | EMD
VAE ($\beta=1e^{-2})$ | $1.47$ | $1.99$ | $5.43$ | $\mathbf{1.55}$ | $1.66$ | $0.43$ | $62.99$ | $63.67$ | $7.25$ | $7.50$
VAE ($\beta=1e^{-4})$ | $\mathbf{0.61}$ | $\mathbf{0.74}$ | $4.23$ | $4.89$ | $1.53$ | $0.38$ | $65.49$ | $\mathbf{66.33}$ | $1.17$ | $\mathbf{0.17}$
DIP-VAE-I | $4.65$ | $11.86$ | $4.74$ | $5.32$ | $\mathbf{1.24}$ | $\mathbf{0.29}$ | $55.57$ | $56.42$ | $4.31$ | $4.56$
DIP-VAE-II | $4.76$ | $11.92$ | $4.30$ | $6.44$ | $1.70$ | $0.43$ | $48.48$ | $47.30$ | $17.72$ | $17.15$
Factor VAE | $0.74$ | $1.01$ | $\mathbf{10.51}$ | $12.47$ | $3.60$ | $0.97$ | $41.05$ | $41.05$ | $2.28$ | $2.62$
Proposed | $0.73$ | $0.93$ | $4.23$ | $4.30$ | $1.56$ | $0.38$ | $\mathbf{65.67}$ | $63.67$ | $\mathbf{0.50}$ | $1.67$

We compare our method with other self-supervised methods based on encoder-
decoder pairs. For a fair comparison, all methods share the same underlying
architecture, which we refer to as VAE and which is already detailed in Sec.
3.1. Consistently with the current literature [35, 26, 14, 44], we found that
the weight coefficient ($\beta$) on the KL divergence in VAEs for meshes is
smaller than the one used for images. In fact, with $\beta\geq 1$ the VAE is
not able to reconstruct the data. Thus, we report results on VAEs with
$\beta\in\\{1e^{-2},1e^{-4}\\}$. It is worth noting that the discrepancy
between meshes and images does not allow to define a $\beta$-VAE with the same
criteria used in the literature ($\beta>1$) [18]. We also compare our method
with the DIP-VAE-I, DIP-VAE-II, and Factor VAE. To the best of our knowledge
this is the first attempt to use them in the mesh domain. Therefore, for the
two DIP-VAEs, we set $\beta=1e^{-4}$ and, following the hyperparameter tuning
strategy adopted in the original implementation [23], we tune $\lambda_{d}$
and $\lambda_{od}$. Here we report results for DIP-VAE-I with
$\lambda_{d}=100$ and $\lambda_{od}=10$ as well as for DIP-VAE-II with
$\lambda_{d}=10$ and $\lambda_{od}=10$, which qualitatively showed better
disentanglement performances. Factor VAE is trained with a discriminator
learning rate of $1e^{-6}$, and a total correlation weight $\gamma=0.25$.

We first evaluate the quality of the different models trained on the face
dataset in terms of reconstruction errors, diversity of the generated samples,
Jensen-Shannon Divergence (JSD) [2], Coverage (COV) [2], Minimum matching
distance (MMD) [2], and 1-nearest neighbour accuracy (1-NNA) [42] (Tab. 1).
Mean and maximum reconstruction errors are computed with respect to mean per-
vertex errors across the test set. The diversity is computed as the mean of
mean per-vertex distances among pairs of meshes randomly generated with the
model. The other metrics are computed by leveraging the Chamfer (CD) and Earth
Mover (EMD) distances on $2048$ randomly selected pairs of vertices. Note that
since the original formulation of 1-NNA expects scores converging to $50\%$,
in Tab. 1 we report absolute differences between the original score and the
$50\%$ target value. From Tab. 1 we observe that while most methods for latent
disentanglement have significantly increased reconstruction errors, our method
closely match the VAE. We also notice that while most models have similar
diversity, Factor VAE is able to generate more diverse data. While this
property seems to be desirable, observing some randomly generated sample (Fig.
3), we argue that sampled faces are less realistic. The other metrics used to
evaluate the generation capabilities of the different models show that our
method is comparable with the others, thus proving that our mini-batching
procedure and latent consistency loss do not negatively affect the generation
capabilities.

Figure 5: Results of our method on bodies. A: samples randomly generated with
the proposed method trained on body meshes. B: visual representation of all
the different body features for which we seek to obtain a disentangled latent
representation. C: effects of latent variable traversals for each latent
variable across different body features. D: vertex-wise distances showing the
effects of traversing five latent variables (see Supplementary Material for
all the latent variables).

#### Evaluation of Latent Disentanglement

Previous work evaluated the latent disentanglement on either datasets where
labelled data were available or on custom-made datasets of images whose
generative factors could be used as labels. Examples of such datasets [18, 23,
19] are binary images of geometric shapes (e.g. circles, rectangles, etc.)
where shape deformation parameters are known, or images rendered with
controlled camera and lighting positions. Even though both our datasets are
generated from existing models, these models lack control over the generative
factors, thus traditional metrics such as Z-Diff [18], SAP [23], and Factor
[19] scores cannot be computed. In addition, the few unsupervised
disentanglement metrics currently existing [45], are not suitable for our
evaluation because [27] is tailored for the evaluation of the disentanglement
of style and content information, while [11] is used for model and
hyperparameter selection thus requiring multiple computationally expensive
hyperparameter sweeps. Therefore, we decide to evaluate the effects caused on
the generated meshes while traversing each latent variable. We generate two
meshes corresponding to each latent variable: one is created setting one
latent variable to its minimum ($-3$) and all the remaining to their mean
value ($0$), the other replacing the minimum with the maximum value ($+3$).
The per-vertex Euclidean distances between the two shapes represent the
effects of perturbing a single latent variable. These effects can be
qualitatively assessed by observing meshes rendered with vertex colours
proportional to the distances (Fig. 3, and Fig. 5 D). Alternatively, distances
corresponding to each feature (Fig. 3D Shape Variational Autoencoder Latent
Disentanglement  via Mini-Batch Feature Swapping for Bodies and Faces, Right
and Fig. 5 A) can be averaged and subsequently plotted as in Fig. 4 and Fig. 5
C. This representation clearly highlights how perturbing each latent variable
affects the different features. While most methods appear to be difficult to
interpret and mostly entangled, our method shows a significantly more
structured, interpretable, and disentangled latent representation than other
methods. Interestingly, in the VAE with $\beta=1e^{-2}$ we observe a polarised
regime in which only a subset of latent variables control the generated
shapes. However, these variables are also controlling the same features, thus
disentanglement is not achieved. Since the polarised regime occurs in
$\beta$-VAEs [36], we can consider this VAE to be a $\beta$-VAE operating on
meshes.

#### Direct Manipulation

Similarly to [17], our method supports direct manipulation of the generated 3D
meshes. A user is thus able to select one or multiple vertices, specify their
new desired location, and our method automatically generates a new mesh
locally deformed to satisfy the user edit. This is achieved through a small
optimisation procedure over the latent representation. We use the Adam
optimiser for $50$ iterations and with a fixed learning rate of $lr=0.1$.
Given $S\circ\mathbf{X}^{\prime}=S\circ
G(\mathbf{z})\in\mathbb{R}^{\Upsilon\times 3}$ the subset of vertices manually
selected from the currently generated mesh, and their desired positions
$\mathbf{Y}\in\mathbb{R}^{\Upsilon\times 3}$, with $\Upsilon$ representing the
number of selected vertices, we optimise: $\min_{\mathbf{z}^{f}}\|S\circ
G(\mathbf{z})-\mathbf{Y}\|_{2}^{2}$. Note that the optimisation over
$\mathbf{z}^{f}$ guarantees the locality of the manipulation (Fig. 6, IIa) and
it is achieved by setting to zero the gradients computed over
$\mathbf{z}^{c}$. This is made possible by our method and its improved latent
disentanglement. An optimisation over the entire latent representation would
cause visible global changes (Fig. 6, IIb), thus making impossible the direct
manipulation.

Figure 6: Direct manipulation of the generated mesh. (I) The user selects an
arbitrary number of vertices (blue) and their new desired position (red), then
our method generates a locally edited mesh fitting the desired locations.
Results are reported optimising only $\mathbf{z}^{f}$ (IIa) and optimising the
entire $\mathbf{z}$ (IIb).

##  5 Conclusion

We proposed a novel approach to learn a more disentangled, interpretable, and
structured latent representation for 3D VAEs. This is achieved by curating the
mini-batching procedure with feature swapping and introducing an additional
latent consistency loss. Even though our method is able to disentangle
predefined subsets of latent variables, we do not guarantee orthogonality and
disentanglement among the variables within each subsets. Nonetheless, we can
increase the number of subsets to achieve finer control over the generated
model. The main limitations of our work are the assumptions made on the
training data. A consistent scaling and alignment, as well as dense-point
correspondences, and a fixed mesh topology are common for generative models of
3D faces (and bodies) and useful for an efficient feature swapping. However,
this assumption could be relaxed to make our method suitable for more general
3D problems if a different architecture was implemented and semantic
segmentations of each 3D shape were available. Good semantic segmentations are
not trivial to obtain for raw data, but methods such as [41, 40] could be
used. As future work, we aim at introducing and properly disentangling
expressions (or poses) while retaining the superior latent disentanglement
over identity features made possible by our method.

#### Acknowledgement

This work was supported by the Wellcome Trust/EPSRC [203145Z/16/Z]. The views
expressed in this publication are those of the author(s) and not necessarily
those of the Wellcome Trust.

## References

  * [1] Victoria Fernández Abrevaya, Adnane Boukhayma, Stefanie Wuhrer, and Edmond Boyer.  A decoupled 3d facial shape model by adversarial training.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9419–9428, 2019.
  * [2] Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas Guibas.  Learning representations and generative models for 3d point clouds.  In International conference on machine learning, pages 40–49. PMLR, 2018.
  * [3] Tristan Aumentado-Armstrong, Stavros Tsogkas, Allan Jepson, and Sven Dickinson.  Geometric disentanglement for generative latent shape models.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8181–8190, 2019.
  * [4] Yoshua Bengio, Aaron Courville, and Pascal Vincent.  Representation learning: A review and new perspectives.  IEEE transactions on pattern analysis and machine intelligence, 35(8):1798–1828, 2013.
  * [5] Volker Blanz and Thomas Vetter.  A morphable model for the synthesis of 3d faces.  In Proceedings of the 26th annual conference on Computer graphics and interactive techniques, pages 187–194, 1999.
  * [6] James Booth, Anastasios Roussos, Stefanos Zafeiriou, Allan Ponniah, and David Dunaway.  A 3d morphable model learnt from 10,000 faces.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5543–5552, 2016.
  * [7] Shiyang Cheng, Michael Bronstein, Yuxiang Zhou, Irene Kotsia, Maja Pantic, and Stefanos Zafeiriou.  Meshgan: Non-linear 3d morphable models of faces.  arXiv preprint arXiv:1903.10384, 2019.
  * [8] Luca Cosmo, Antonio Norelli, Oshri Halimi, Ron Kimmel, and Emanuele Rodola.  Limp: Learning latent shape representations with metric preservation priors.  arXiv preprint arXiv:2003.12283, 2, 2020.
  * [9] Hang Dai, Nick Pears, William Smith, and Christian Duncan.  Statistical modeling of craniofacial shape and texture.  International Journal of Computer Vision, 128(2):547–571, 2020\.
  * [10] Zheng Ding, Yifan Xu, Weijian Xu, Gaurav Parmar, Yang Yang, Max Welling, and Zhuowen Tu.  Guided variational autoencoder for disentanglement learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7920–7929, 2020.
  * [11] Sunny Duan, Loic Matthey, Andre Saraiva, Nicholas Watters, Christopher P Burgess, Alexander Lerchner, and Irina Higgins.  Unsupervised model selection for variational disentangled representation learning.  arXiv preprint arXiv:1905.12614, 2019.
  * [12] Bernhard Egger, William AP Smith, Ayush Tewari, Stefanie Wuhrer, Michael Zollhoefer, Thabo Beeler, Florian Bernard, Timo Bolkart, Adam Kortylewski, Sami Romdhani, et al.  3d morphable face models—past, present, and future.  ACM Transactions on Graphics (TOG), 39(5):1–38, 2020.
  * [13] Babak Esmaeili, Hao Wu, Sarthak Jain, Alican Bozkurt, Narayanaswamy Siddharth, Brooks Paige, Dana H Brooks, Jennifer Dy, and Jan-Willem Meent.  Structured disentangled representations.  In The 22nd International Conference on Artificial Intelligence and Statistics, pages 2525–2534. PMLR, 2019.
  * [14] Simone Foti, Bongjin Koo, Thomas Dowrick, Joao Ramalhinho, Moustafa Allam, Brian Davidson, Danail Stoyanov, and Matthew J Clarkson.  Intraoperative liver surface completion with graph convolutional vae.  In Uncertainty for Safe Utilization of Machine Learning in Medical Imaging, and Graphs in Biomedical Image Analysis, pages 198–207. Springer, 2020.
  * [15] Baris Gecer, Alexandros Lattas, Stylianos Ploumpis, Jiankang Deng, Athanasios Papaioannou, Stylianos Moschoglou, and Stefanos Zafeiriou.  Synthesizing coupled 3d face modalities by trunk-branch generative adversarial networks.  In European Conference on Computer Vision, pages 415–433. Springer, 2020.
  * [16] Shunwang Gong, Lei Chen, Michael Bronstein, and Stefanos Zafeiriou.  Spiralnet++: A fast and highly efficient mesh convolution operator.  In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pages 0–0, 2019.
  * [17] Aurel Gruber, Marco Fratarcangeli, Gaspard Zoss, Roman Cattaneo, Thabo Beeler, Markus Gross, and Derek Bradley.  Interactive sculpting of digital faces using an anatomical modeling paradigm.  In Computer Graphics Forum, volume 39, pages 93–102. Wiley Online Library, 2020.
  * [18] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner.  beta-vae: Learning basic visual concepts with a constrained variational framework.  International Conference on Learning Representations, 2016.
  * [19] Hyunjik Kim and Andriy Mnih.  Disentangling by factorising.  In International Conference on Machine Learning, pages 2649–2658. PMLR, 2018.
  * [20] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [21] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  arXiv preprint arXiv:1312.6114, 2013.
  * [22] Tejas D Kulkarni, Will Whitney, Pushmeet Kohli, and Joshua B Tenenbaum.  Deep convolutional inverse graphics network.  arXiv preprint arXiv:1503.03167, 2015.
  * [23] Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan.  Variational inference of disentangled latent concepts from unlabeled observations.  arXiv preprint arXiv:1711.00848, 2017.
  * [24] Ruilong Li, Karl Bladin, Yajie Zhao, Chinmay Chinara, Owen Ingraham, Pengda Xiang, Xinglei Ren, Pratusha Prasad, Bipin Kishore, Jun Xing, et al.  Learning formation of physically-based face attributes.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3410–3419, 2020.
  * [25] Tianye Li, Timo Bolkart, Michael J Black, Hao Li, and Javier Romero.  Learning a model of facial shape and expression from 4d scans.  ACM Trans. Graph., 36(6):194–1, 2017.
  * [26] Or Litany, Alex Bronstein, Michael Bronstein, and Ameesh Makadia.  Deformable shape completion with graph convolutional autoencoders.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1886–1895, 2018.
  * [27] Xiao Liu, Spyridon Thermos, Gabriele Valvano, Agisilaos Chartsias, Alison O’Neil, and Sotirios A Tsaftaris.  Metrics for exposing the biases of content-style disentanglement.  arXiv preprint arXiv:2008.12378, 2020.
  * [28] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black.  Smpl: A skinned multi-person linear model.  ACM transactions on graphics (TOG), 34(6):1–16, 2015.
  * [29] Charlie Nash and Christopher KI Williams.  The shape variational autoencoder: A deep generative model of part-segmented 3d objects.  In Computer Graphics Forum, volume 36, pages 1–12. Wiley Online Library, 2017.
  * [30] Andrew Nealen, Takeo Igarashi, Olga Sorkine, and Marc Alexa.  Laplacian mesh optimization.  In Proceedings of the 4th international conference on Computer graphics and interactive techniques in Australasia and Southeast Asia, pages 381–389, 2006.
  * [31] Ahmed AA Osman, Timo Bolkart, and Michael J Black.  Star: Sparse trained articulated human body regressor.  arXiv preprint arXiv:2008.08535, 2020.
  * [32] Stylianos Ploumpis, Evangelos Ververas, Eimear O’Sullivan, Stylianos Moschoglou, Haoyang Wang, Nick Pears, William Smith, Baris Gecer, and Stefanos P Zafeiriou.  Towards a complete 3d morphable model of the human head.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020\.
  * [33] Stylianos Ploumpis, Haoyang Wang, Nick Pears, William AP Smith, and Stefanos Zafeiriou.  Combining 3d morphable models: A large scale face-and-head model.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 10934–10943, 2019.
  * [34] Gerard Pons-Moll, Javier Romero, Naureen Mahmood, and Michael J Black.  Dyna: A model of dynamic human shape in motion.  ACM Transactions on Graphics (TOG), 34(4):1–14, 2015.
  * [35] Anurag Ranjan, Timo Bolkart, Soubhik Sanyal, and Michael J Black.  Generating 3d faces using convolutional mesh autoencoders.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 704–720, 2018.
  * [36] Michal Rolinek, Dominik Zietlow, and Georg Martius.  Variational autoencoders pursue pca directions (by accident).  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12406–12415, 2019.
  * [37] Soubhik Sanyal, Timo Bolkart, Haiwen Feng, and Michael J Black.  Learning to regress 3d face shape and expression from an image without 3d supervision.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7763–7772, 2019.
  * [38] J Rafael Tena, Fernando De la Torre, and Iain Matthews.  Interactive region-based linear 3d face models.  In ACM SIGGRAPH 2011 papers, pages 1–10. Association for Computing Machinery, 2011.
  * [39] Gul Varol, Javier Romero, Xavier Martin, Naureen Mahmood, Michael J Black, Ivan Laptev, and Cordelia Schmid.  Learning from synthetic humans.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 109–117, 2017.
  * [40] Nitika Verma, Edmond Boyer, and Jakob Verbeek.  Feastnet: Feature-steered graph convolutions for 3d shape analysis.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2598–2606, 2018.
  * [41] Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon.  Dynamic graph cnn for learning on point clouds.  Acm Transactions On Graphics (tog), 38(5):1–12, 2019.
  * [42] Guandao Yang, Xun Huang, Zekun Hao, Ming-Yu Liu, Serge Belongie, and Bharath Hariharan.  Pointflow: 3d point cloud generation with continuous normalizing flows.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4541–4550, 2019.
  * [43] Jie Yang, Kaichun Mo, Yu-Kun Lai, Leonidas J Guibas, and Lin Gao.  Dsm-net: Disentangled structured mesh net for controllable generation of fine geometry.  arXiv preprint arXiv:2008.05440, 2020.
  * [44] Yu-Jie Yuan, Yu-Kun Lai, Jie Yang, Qi Duan, Hongbo Fu, and Lin Gao.  Mesh variational autoencoders with edge contraction pooling.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 274–275, 2020.
  * [45] Julian Zaidi, Jonathan Boilard, Ghyslain Gagnon, and Marc-André Carbonneau.  Measuring disentanglement: A review of metrics.  arXiv preprint arXiv:2012.09276, 2020.
  * [46] Yi Zhou, Chenglei Wu, Zimo Li, Chen Cao, Yuting Ye, Jason Saragih, Hao Li, and Yaser Sheikh.  Fully convolutional mesh autoencoder using efficient spatially varying kernels.  arXiv preprint arXiv:2006.04325, 2020.
  * [47] Silvia Zuffi, Angjoo Kanazawa, David W Jacobs, and Michael J Black.  3d menagerie: Modeling the 3d shape and pose of animals.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6365–6373, 2017.


Faculty of Physics and Applied Computer Science, AGH University of Krakow, al.
A. Mickiewicza 30, 30-059 Kraków, Poland Institute of Nuclear Physics, Polish
Academy of Sciences, PL-31-342 Kraków, Poland Departament de Física Quàntica i
Astrofísica and Institut de Ciències del Cosmos, Universitat de Barcelona,
E-08028 Barcelona, Spain Departamento de Física Teórica, Universidad
Complutense de Madrid and IPARCOS, E-28040 Madrid, Spain Theory Center, Thomas
Jefferson National Accelerator Facility, Newport News, VA 23606, USA Center
for Exploration of Energy and Matter, Indiana University, Bloomington, IN
47403, USA Department of Physics, Indiana University, Bloomington, IN 47405,
USA

# Studying the production mechanisms of light meson resonances in two-pion
photoproduction: a Regge approach

Ł. Bibrzyckiins:krakow1  N. Hammoudins:krakow2  V.
Mathieuins:barcelonains:madrid  R. J. Perryins:barcelona
<EMAIL_ADDRESS>
A. P. Szczepaniakins:jlabins:indiana1ins:indiana2
ins:krakow1ins:krakow1ins:krakow2ins:krakow2ins:barcelonains:barcelonains:madridins:madridins:jlabins:jlabins:indiana1ins:indiana1ins:indiana2ins:indiana2

###### Abstract

A calculation of the angular moments of two-pion photoproduction is presented.
The underlying theoretical model encodes the prominent $\rho(770)$ resonance
and the expected leading background contribution coming from the Deck
mechanism. The model contains a number of free parameters which are fit to
experimental data. A good description of the angular moments is obtained.

##  1 Introduction

This proceedings focuses on a theoretical determination of the angular moments
of two (charged) pion photoproduction at low invariant mass of the
$\pi^{+}\pi^{-}$ system. The angular moments are well-defined experimental
observables which are bilinear in the partial waves of the two-pion subsystem.
They are thus a key spectroscopic observable of interest. After presenting the
theoretical model for two-pion photoproduction, the angular moments are
computed, and compared with experimental results.

##  2 Kinematics and Preliminaries

In this paper a theoretical model for the process
${\gamma}(q,\lambda_{\gamma})+p(p_{1},\lambda_{1})\to\pi^{+}(k_{1})+\pi^{-}(k_{2})+p(p_{2},\lambda_{2})$
is described. The calculation is performed in the helicity frame. In this
frame, the $\pi^{+}\pi^{-}$ system is at rest
($\mathbf{k}_{1}=-\mathbf{k}_{2}$) and the recoiling proton ($\mathbf{p}_{2}$)
defines the negative $z$-axis. The $\pi^{+}$ recoils at an angle
$\Omega^{\text{H}}=(\theta^{\text{H}},\phi^{\text{H}})$ to the $z$ axis. The
scalar amplitudes in such a $2\to 3$ process are maximally described by five
independent kinematical variables. Thus in addition to the two angles for the
$\pi^{+}$, the kinematic invariants $s=(p_{1}+q)^{2}$, $t=(p_{1}-p_{2})^{2}$,
and $s_{12}=(k_{1}+k_{2})^{2}=m_{12}^{2}$ are used. This set of kinematic
variables, $(s,t,s_{12},\Omega^{\text{H}})$ is complete, in the sense that all
other kinematic invariants may be computed by knowing these five. The notation
for the phase-space and intensity is taken from Ref. [1]. The differential
cross section is defined as

$\frac{d\sigma}{dtdm_{12}d\Omega^{\text{H}}}=\kappa\sum_{\lambda_{1}\lambda_{2}\lambda_{\gamma}}|\mathcal{M}_{\lambda_{1}\lambda_{2}\lambda_{\gamma}}(s,t,s_{12},\Omega^{\text{H}})|^{2}\,,$
(1)

where $\kappa$ contains the kinematical factors and
$\mathcal{M}_{\lambda_{1}\lambda_{2}\lambda\gamma}(s,t,s_{12},\Omega^{\text{H}})$
is the invariant matrix element. This paper will focus on a description of the
unpolarized angular moments. These may be obtained from Eq. (1) via

$\expectationvalue{Y_{LM}}(s,t,s_{12})=\sqrt{4\pi}\int
d\Omega^{\text{H}}\,\frac{d\sigma}{dtdm_{12}d\Omega^{\text{H}}}\text{Re}\,Y_{LM}(\Omega^{\text{H}})\,.$
(2)

##  3 Description of Model

The global features of the angular moments, $\expectationvalue{Y_{LM}}$,
measured in two-pion photoproduction can be understood as the result two
competing mechanisms which may be separated based on their effect on the two-
pion invariant mass distribution. In particular, the model incorporates both
explicit low-energy two-pion resonances, and the leading continuum
contribution from the “Deck” or “Drell-Söding” Mechanism [2, 3, 4]. These
contributions are represented diagramatically in Fig. 1. In the following
subsections, these two mechanisms are described in more detail.

feynman_diagrams (40,60) l1,l2 r1,r2,r3 fermionl1,v1,r1 photonl2,v2
dashesv2,r2 dbl_zigzagv1,v2 dashesv2,r3 0.2wv2 $p(p_{1},\lambda_{1})$l1
$\gamma(q,\lambda_{q})$l2 $p(p_{2},\lambda_{2})$r1 $\pi^{-}(k_{2})$r3
$\pi^{+}(k_{1})$r2 (40,60) l1,l2 r1,r2,r3 fermionl1,v1,r1 photonl2,v2
dashesv1,r2 dashesv1,v2,r3 0.2wv1 $p(p_{1},\lambda_{1})$l1
$\gamma(q,\lambda_{q})$l2 $p(p_{2},\lambda_{2})$r1 $\pi^{-}(k_{2})$r2
$\pi^{+}(k_{1})$r3 (40,60) l1,l2 r1,r2,r3 fermionl1,v1,r1 photonl2,v2
dashesv1,r2 dashesv1,v2,r3 0.2wv1 $p(p_{1},\lambda_{1})$l1
$\gamma(q,\lambda_{q})$l2 $p(p_{2},\lambda_{2})$r1 $\pi^{-}(k_{2})$r3
$\pi^{+}(k_{1})$r2

Figure 1: Dominant contributions to two-pion photoproduction at small momentum
transfer. With this approximation, it is possible to relate this $2\to 3$
process to $2\to 2$ processes.

###  3.1 Resonant Model

The resonant model is constructed by assuming that two-pion resonances are
produced via Reggeon-photon fusion, which subsequently decay into two pions.
Symbolically, one may write

$\mathcal{M}_{\lambda_{1}\lambda_{2}\lambda_{\gamma}}(s,t,s_{12},\Omega^{\text{H}})=\sum_{E}\Gamma_{\lambda_{1}\lambda_{2}}^{N\to
EN}(t)R_{E}(s,t)i\mathcal{M}_{\lambda_{\gamma}}^{\gamma
E\to\pi^{+}\pi^{-}}(s_{12},\Omega^{\text{H}})$ (3)

where $\Gamma_{\lambda_{1}\lambda_{2}}^{N\to EN}$ describes the nucleon-
reggeon vertex, $R_{E}(s,t)$ is a Regge propagator and
$i\mathcal{M}_{\lambda_{\gamma}}^{\gamma E\to\pi^{+}\pi^{-}}$ is the $\gamma
E\to\pi^{+}\pi^{-}$ scattering amplitude. The Regge propagator is

$R_{E}(s,t)=\frac{\alpha^{E}(t)}{\alpha^{E}(0)}\frac{1+e^{-i\pi\alpha^{E}(t)}}{\sin\pi\alpha^{E}(t)}\bigg{(}\frac{s}{s_{0}}\bigg{)}^{\alpha^{E}(t)}$
(4)

where $\alpha_{E}(t)=\alpha_{0}^{E}+\alpha_{1}^{E}t$ is the Regge trajectory,
and $s_{0}=1~{}$\mathrm{G}\mathrm{e}\mathrm{V}^{2}$$ is a mass scale
introduced to fix the dimensions. In this study only $\mathbb{P}$, $f_{2}$ and
$\rho/\omega$ exchanges are considered. The $t$-dependence of the vertices is
not predicted from Regge theory and here is obtained from an effective
Lagrangian approach. Residual $t$-dependence can be absorbed into the fitted
parameters. Resonances are described by energy-dependent Breit-Wigner
distributions following the formalism in Ref. [5].

The CLAS database [6] contains experimental measurements of the two-pion
angular moments from near threshold, $m_{12}\sim
0.4~{}$\mathrm{G}\mathrm{e}\mathrm{V}$$, to $m_{12}\sim
1.4~{}$\mathrm{G}\mathrm{e}\mathrm{V}$$. Clear evidence for the presence of
meson resonances may be observed by studying the structures of the angular
moments. In this proceedings, only resonances with a mass of less than
$1~{}$\mathrm{G}\mathrm{e}\mathrm{V}$$ will be considered. In this mass
region, the PDG lists two $S$-wave resonances, the $\sigma(500)$ and
$f_{0}(980)$ and one $P$-wave resonance, the $\rho(770)$.

###  3.2 Continuum Model

The Deck mechanism forms the continuum contribution to the model. For the
reaction considered, it is given by

$\begin{split}\mathcal{M}_{\lambda_{1}\lambda_{2}\lambda_{\gamma}}^{\text{Deck,GI}}&=\sqrt{4\pi\alpha}\bigg{[}\bigg{(}\frac{\epsilon(q,\lambda_{\gamma})\cdot
k_{1}}{q\cdot
k_{1}}-\frac{\epsilon(q,\lambda_{\gamma})\cdot(p_{1}+p_{2})}{q\cdot(p_{1}+p_{2})}\bigg{)}\beta(u_{1}){M}_{\lambda_{1}\lambda_{2}}^{-}(s_{2},t;u_{1})\\\
&-\bigg{(}\frac{\epsilon(q,\lambda_{\gamma})\cdot k_{2}}{q\cdot
k_{2}}-\frac{\epsilon(q,\lambda_{\gamma})\cdot(p_{1}+p_{2})}{q\cdot(p_{1}+p_{2})}\bigg{)}\beta(u_{2}){M}_{\lambda_{1}\lambda_{2}}^{+}(s_{1},t;u_{2})\bigg{]}\end{split}$
(5)

where $\beta(u_{i})=\exp((u_{i}-u_{i}^{\text{min}})/\Lambda_{\pi}^{2})$ are
hadronic form factors introduced to suppress the Born term pion propagator at
large $u_{i}=(q-k_{i})^{2}$,
$\Lambda_{\pi}=0.9~{}$\mathrm{G}\mathrm{e}\mathrm{V}$$ and
${M}_{\lambda_{1}\lambda_{2}}^{\pm}$ is the scattering amplitude for the
process $p+\pi^{*\pm}\to p+\pi^{\pm}$. Note that although this is a binary
amplitude, the virtuality of the initial pion, $u_{i}$ implies that this
amplitude is dependent on three kinematic invariants. In the limit that
$u_{i}\to m_{\pi}^{2}$, these amplitudes may be related to elastic
$\pi^{\pm}p$ scattering, for which there is a wealth of experimental and
theoretical information. The form of the $\pi N$ amplitudes are taken from
Ref. [7]. At low energy, this amplitude is described in terms of the SAID
parameterization [8, 9, 10] of $\pi N$ partial wave amplitudes, which are
extracted from elastic $\pi N$ scattering data, while at high energies, the
model interpolates to a Regge description of the data.

##  4 Results and Discussion

Angular moments of two-pion photoproduction were computed using the model
defined above and compared to the CLAS experimental data [11]. In order to
improve the agreement with experimental data, the relative weights of the
partial waves were refit to the experimental data. The resulting fit to
experimental data is shown in Fig. 2. A good fit to the angular moments is
obtained, except possibly for $m_{12}>1.2~{}$\mathrm{G}\mathrm{e}\mathrm{V}$$.
Note however, that in this region, there are several resonances which are
expected to contribute, but which are not currently incorporated in the model.

Figure 2: Comparing the preliminary predictions for angular moments of two-
pion photoproduction with experimental data from Ref. [11]. After fitting a
number of free parameters, a good description of the experimental data is
obtained.

Figure 3: Preliminary prediction of the absolute value of $S$\- and $P$-wave
amplitudes. Note that at small $-t$, the hierachy of $P$-wave amplitudes is in
accordance with SCHC, but at larger $-t$, the spin-flip amplitude $|P_{0}|$
appears to dominate. This behaviour can be understood from Regge theory.

The resulting absolute values of the partial-waves are shown in Fig. 3. The
notation for the partial waves is taken from Ref. [11]. In particular, $P_{+}$
denotes the $P$-wave partial wave amplitude with the same helicity as the
incident photon, $P_{0}$ denotes the amplitude with one unit of helicity flip
and $P_{-}$ denotes the amplitude with two units of helicity flip. It was
observed in Ref. [12] that at small $-t$, the process $\gamma p\to\rho p$
proceeded primarily via the helicity conserving amplitude. This observation
was called $s$-channel helicity conservation (SCHC). In the notation given
here, this implies that $P_{+}$ dominates over $P_{0}$ and $P_{-}$. Regge
theory implies that for small $-t/s$, the top and bottom vertices factorize,
and the resulting $-t$-dependence of the top vertex is
$\sim(-t)^{|\lambda_{\gamma}-\lambda_{\rho}|}$. Thus in the forward limit,
only the helicity amplitude corresponding to $\lambda_{\gamma}=\lambda_{\rho}$
is expected to contribute. However, compared with Ref. [12], the $-t$ measured
at CLAS is larger. Due to the $-t$ dependence of the $P_{0}$ and $P_{-}$
amplitudes, one expects that at larger $-t$ SCHC should be increasingly
violated, as these amplitudes are $-t$-enhanced. This behaviour is observed in
the $t$-dependence in Fig. 3. Note that at
$-t=0.45~{}$\mathrm{G}\mathrm{e}\mathrm{V}$$, the hierachy of partial waves is
as one would predict from SCHC, with $|P_{+}|>|P_{0}|>|P_{-}|$. However, at
larger $-t$, this hierachy is no-longer obeyed.

##  5 Conclusions

This proceedings presented a theoretical model of two-pion photoproduction
which incorporated the prominent $\rho(770)$ resonance and leading background
contribution from the Deck mechanism. A good agreement between model and data
was obtained after the relative weights of partial waves were fit to
experimental data, suggesting that the model correctly identifies the
production mechanisms and partial waves which are instrumental for the
description of the angular moments. This model can be applied to make
predictions at photon energies relevant for the CLAS12 and GlueX experiments.

###### Acknowledgements.

This work contributes to the aims of the USDOE ExoHad Topical Collaboration,
contract DE-SC0023598. In addition, the support of project
PID2020-118758GB-I00, financed by the Spanish MCIN/ AEI/10.13039/501100011033/
is acknowledged. The support of project No. 2018/29/B/ST2/ 02576 (National
Science Center), partly financed by a Polish research project is also
acknowledged. The authors thank their JPAC colleagues for useful
conversations.

## References

  * [1] Mathieu V., Albaladejo M., Fernández-Ramírez C., Jackura A. W., Mikhasenko M., Pilloni A. Szczepaniak A. P., Phys. Rev. D1002019054017.
  * [2] Drell S. D., Rev. Mod. Phys.331961458.
  * [3] Deck R. T., Phys. Rev. Lett.131964169.
  * [4] Soding P., Phys. Lett.191966702.
  * [5] Workman R. L. et al., PTEP20222022083C01.
  * [6] Jlab experiment clas database.
URL https://clasweb.jlab.org/physicsdb/intro.html

  * [7] Mathieu V., Danilkin I. V., Fernández-Ramírez C., Pennington M. R., Schott D., Szczepaniak A. P. Fox G., Phys. Rev. D922015074004.
  * [8] Arndt R. A., Workman R. L. Pavan M. M., Phys. Rev. C4919942729.
  * [9] Arndt R. A., Strakovsky I. I., Workman R. L. Pavan M. M., Phys. Rev. C5219952120.
  * [10] Arndt R. A., Briscoe W. J., Strakovsky I. I., Workman R. L. Pavan M. M., Phys. Rev. C692004035213.
  * [11] Battaglieri M. et al., Phys. Rev. D802009072005.
  * [12] Ballam J. et al., Phys. Rev. Lett.241970960.