# GaussianTalker: Speaker-specific Talking Head Synthesis
via 3D Gaussian Splatting

Hongyun Yu<EMAIL_ADDRESS>Alibaba GroupHangzhouChina ,  Zhan
Qu<EMAIL_ADDRESS>Zhejiang UniversityHangzhouChina ,  Qihang Yu
<EMAIL_ADDRESS>Zhejiang UniversityHangzhouChina ,  Jianchuan Chen
<EMAIL_ADDRESS>Alibaba GroupHangzhouChina ,  Zhonghua Jiang
Zhejiang UniversityHangzhouChina ,  Zhiwen Chen<EMAIL_ADDRESS>Alibaba GroupHangzhouChina ,  Shengyu Zhang  Zhejiang UniversityHangzhouChina
,  Jimin Xu  Zhejiang UniversityHangzhouChina ,  Fei Wu  Zhejiang
UniversityHangzhouChina ,  Chengfei Lv<EMAIL_ADDRESS>Alibaba
GroupHangzhouChina and  Gang Yu<EMAIL_ADDRESS>Alibaba
GroupHangzhouChina

(2024)

###### Abstract.

Recent works on audio-driven talking head synthesis using Neural Radiance
Fields (NeRF) have achieved impressive results. However, due to inadequate
pose and expression control caused by NeRF implicit representation, these
methods still have some limitations, such as unsynchronized or unnatural lip
movements, and visual jitter and artifacts. In this paper, we propose
GaussianTalker, a novel method for audio-driven talking head synthesis based
on 3D Gaussian Splatting. With the explicit representation property of 3D
Gaussians, intuitive control of the facial motion is achieved by binding
Gaussians to 3D facial models. GaussianTalker consists of two modules,
Speaker-specific Motion Translator and Dynamic Gaussian Renderer. Speaker-
specific Motion Translator achieves accurate lip movements specific to the
target speaker through universalized audio feature extraction and customized
lip motion generation. Dynamic Gaussian Renderer introduces Speaker-specific
BlendShapes to enhance facial detail representation via a latent pose,
delivering stable and realistic rendered videos. Extensive experimental
results suggest that GaussianTalker outperforms existing state-of-the-art
methods in talking head synthesis, delivering precise lip synchronization and
exceptional visual quality. Our method achieves rendering speeds of 130 FPS on
NVIDIA RTX4090 GPU, significantly exceeding the threshold for real-time
rendering performance, and can potentially be deployed on other hardware
platforms. We recommend watching the supplementary
video111https://yuhongyun777.github.io/GaussianTalker/.

talking head synthesis, 3D Gaussian splatting, speaker-specific

††copyright: acmlicensed††journalyear: 2024††doi: XXXXXXX.XXXXXXX††conference:
Proceedings of the 32st ACM International Conference on Multimedia; October
28–November 01, 2024; Melbourne, Australia††booktitle: Proceedings of the 32st
ACM International Conference on Multimedia(MM ’24), October 28–November 01,
2024, Melbourne, Australia††isbn: 978-1-4503-XXXX-X/24/10††ccs: Computing
methodologies††ccs: Computing methodologies Computer vision††ccs: Computing
methodologies Reconstruction

Figure 1. Given the driving audio and reference frames, GaussianTalker can
animate the FLAME mesh of the speaker, which in turn drives the Gaussians
associated with it, ultimately synthesizing a high-quality video with precise
lip movements.

Head Figure.

##  1\. Introduction

Audio-driven talking head synthesis has attracted significant attention in
various fields, including digital avatars, virtual reality, interactive
entertainment, and online meetings(Thies et al., 2020b; Morishima, 1998;
Tanaka and Nakamura, 2022). This task aims to synthesize a video in which the
target speaker’s lip movements are in sync with the given audio input.

In existing research, talking head synthesis approaches are primarily
categorized into 2D-based and 3D-based approaches. Initially, most 2D-based
approaches depended on Generative Adversarial Networks(Goodfellow et al.,
2014) or image-to-image translation(Isola et al., 2017) to synthesize talking
heads in sync with audio. However, the absence of a unified facial model
resulted in shortcomings in the identity preservation and pose control of the
synthesized videos. More recently, research has been conducted to apply the
Neural Radiance Fields (NeRF)(Mildenhall et al., 2020) to this task. NeRF is a
method that models continuous 3D scenes using implicit functions. Compared to
2D-based approaches, NeRF-based approaches can synthesize more lifelike
talking head videos by effectively leveraging 3D facial modeling.
Nevertheless, these approaches often encounter issues such as unsynchronized
or unnatural lip movements, visual jitters, and sporadic artifacts, primarily
because NeRF’s implicit definition entangles static facial geometry with
dynamic motion, complicating the control over facial poses and expressions.

Recently, 3D Gaussian Splatting (3D GS)(Kerbl et al., 2023) has shown
impressive achievements in 3D scene reconstruction. 3D GS employs 3D Gaussians
as discrete geometric primitives, offering an explicit 3D scene representation
and optimizing for real-time rendering performance. In comparison to NeRF, 3D
GS not only significantly enhances rendering efficiency and visual quality,
but also its paradigm based on 3D Gaussians is easier to control, making it
potentially possible to flexibly and intuitively control facial movements. An
intuitive strategy is to drive a Gaussian point cloud for facial motion by
using parametric 3D facial models(Paysan et al., 2009; Cao et al., 2013; Li et
al., 2017). These models typically provide a comprehensive parameter space
that allows for the control of facial attributes, such as shape, pose, and
expression. By binding the Gaussians to the geometric topology of the model,
dynamic talking heads can be generated by synchronizing the displacement of
the bound Gaussians with changes in the facial attribute parameters.

Natural lip movements and realistic visual effects are crucial for talking
head synthesis. However, the deviation of 3D facial models from a specific
speaker’s face makes it challenging to synthesize lifelike videos using such a
basic binding strategy. Specifically, there are two main challenges: 1)
Distribution inconsistencies between the parameters driving the 3D Gaussians
and the actual parameters can lead to deviations in point positions. This
distribution inconsistency specifically refers to differences in speaking
styles. For example, some individuals may speak with a small open mouth, while
the driving signal is a wide open mouth. Lip movements that do not align with
the speaker’s talking style present unnatural effects and even cause lip
artifacts. 2) 3D facial models are limited in modeling complex faces. They can
only capture macroscopic muscle movements but not fine details such as
wrinkles and teeth. This inherent limitation hinders the 3D Gaussians’
accurate representation of a specific face, and the loss of facial details
will also lead to visual flicker and artifacts.

For natural lip movements and realistic visual effects, bridging the subtle
discrepancies between the 3D facial model and the specific speaker’s face is
critical. Firstly, audio typically carries speaker identity information. it is
necessary to decouple the driving audio from the original speaker’s identity
information and ensure that the synthesized lip movements closely match the
target speaker’s style. Secondly, intuitive control of Gaussian attributes
helps break through the representation limitations of facial models. With a
comprehensive set of blendshapes, these Gaussian attributes can be fine-tuned
to capture facial details, refining the facial representation and further
fitting a specific speaker.

In this work, we propose GaussianTalker, a framework designed to generate
highly natural and realistic talking head videos, adaptable to multiple
languages and various timbres of audio input. For dynamic reconstruction, 3D
Gaussians are bound to the geometric topology of the parameterized FLAME(Li et
al., 2017), driving the Gaussians through the deformation of the FLAME mesh,
ensuring accurate facial animation. To address the challenge of unnatural lip
movements caused by inconsistent distributions, we propose a Speaker-specific
Motion Translator. Employing timbre transformation-based contrastive learning
for feature decoupling, a speaker-agnostic audio feature is extracted and
merged with the target identity embedding to generate facial poses and
expressions that closely align with the target speaker. To confront the
challenge posed by unrealistic visual effects caused by the inherent
limitations of 3D facial models, the Dynamic Gaussian Renderer introduces
Speaker-specific BlendShapes designed to predict the target speaker’s latent
pose representation and refine facial details through the computation of
detail-enhancing Gaussian attributes. It is important to note that our
framework is not only capable of achieving speeds well beyond real-time
rendering thresholds on contemporary GPUs, but it is also expected to be
deployed on other hardware platforms, paving the way for its extensive
application across multiple platforms.

The main contributions of this paper are summarized as follows:

  * •

A novel framework for audio-driven talking head synthesis that utilizes 3D
Gaussian Splatting bound to FLAME, which generates lifelike rendered videos by
associating data from different modalities with specific speakers.

  * •

A Speaker-specific Motion Translator decouples identity and employs
personalized embedding for natural lip movements, while a Dynamic Gaussian
Renderer refines Gaussian attributes through latent pose to ensure realistic
visual effects.

  * •

Extensive quantitative and qualitative experiments show GaussianTalker
surpasses state-of-the-art methods in lip-sync and image quality, with its
high rendering speed underscoring multi-hardware platform application
potential.

##  2\. Related Work

Figure 2. Overview of the proposed GaussianTalker. Subfigure (a) depicts
speaker-specific FLAME generated from audio, driving Gaussians for rendering.
Subfigure (b) illustrates the fusion of speaker-agnostic feature with speaker
ID embeddings to decode FLAME. Subfigure (c) shows Gaussians driven by FLAME,
subsequently rendering frames.

Overview of the proposed GaussianTalker.

###  2.1. Audio-Driven Facial Animation

Most early methods of audio-driven facial animation required establishing a
mapping relationship between phonemes and visemes. These methods(Brand, 1999;
Bregler et al., 2023) often overlook individual differences and are overly
complex. The emergence of large-scale high-definition video datasets(Chen et
al., 2023a; Zhang et al., 2021; Afouras et al., 2018; Chung et al., 2018),
coupled with advancements in deep learning, has propelled the domain of
learning-based audio-driven facial animation to the forefront of research.
Initial efforts utilized 2D approaches(Ezzat et al., 2002; Jamaludin et al.,
2019; Wiles et al., 2018; Prajwal et al., 2020; Zhou et al., 2019, 2020;
Vougioukas et al., 2020; Chen et al., 2019; Yu et al., 2020), employing
Generative Adversarial Networks(Goodfellow et al., 2014) or image-to-image
translation(Isola et al., 2017), to generate facial animations. However, these
approaches fell short of accurately reproducing a speaker’s face due to lack
of 3D facial modeling, leading to shortcomings in identity preservation and
pose controllability. The adoption of 3D Morphable Models (3DMM)(Blanz and
Vetter, 2023; Shang et al., 2020; Zollhöfer et al., 2018) has spurred research
into audio-driven facial animation using 3DMM. These approaches(Karras et al.,
2017; Thies et al., 2020a; Wang et al., 2020; Yi et al., 2020; Lu et al.,
2021), leveraging 3D modeling, are capable of rendering a more lifelike
talking style compared to 2D approaches. Nevertheless, intermediate
transformations through 3DMM may lead to information loss(Blanz and Vetter,
2023). Moreover, prior universal 3D audio-driven animation methods(Richard et
al., 2021; Peng et al., 2023) generated vertex predictions with biases toward
specific speakers, resulting in animations that fail to express personal
styles.

Our method utilizes a parametric 3D facial model to represent the human face,
generating facial motion representation from the driving audio. By identity
decoupling and personalized embedding, we accommodate the unique styles of
different speakers.

###  2.2. Human Face Rendering

Recent studies have adopted Neural Radiance Fields (NeRF)(Mildenhall et al.,
2020) for 3D human head modeling and rendering. NeRF encodes the scene into a
continuous volumetric field using a neural network, which enables high-quality
3D rendering. Following its successful application in dynamic scenes(Gafni et
al., 2021; Pumarola et al., 2021; Park et al., 2021), NeRF has been naturally
extended to talking head synthesis tasks(Guo et al., 2021; Yao et al., 2022;
Liu et al., 2022; Tang et al., 2022; Li et al., 2023; Shen et al., 2022; Ye et
al., 2023b, a; Chatziagapi et al., 2023). Some studies have explored the
implementation of talking face video generation in an end-to-end way(Guo et
al., 2021; Yao et al., 2022; Liu et al., 2022). Subsequent studies have
realized several improvements in rendering efficiency(Tang et al., 2022; Li et
al., 2023), few-shot synthesis(Shen et al., 2022; Li et al., 2024), and lip
shape generalization(Ye et al., 2023b, a; Chatziagapi et al., 2023).

With the advent of 3D Gaussian Splatting (3D GS)(Kerbl et al., 2023),
considerable potential has been shown in enhancing both rendering efficiency
and quality. Recent work(Shao et al., 2024; Zhou et al., 2024; Rivero et al.,
2024) has implemented dynamic head reconstruction based on 3D GS.
GaussianHead(Wang et al., 2023) employs learnable Gaussian diffusion for
detailed head generation to accurately reproduce dynamic facial details.
MonoGaussian-Avatar(Chen et al., 2023b) utilizes 3D Gaussian representation
and deformation fields for learning explicit avatars from monocular portrait
videos.

Despite these approaches’ effectiveness, audio-driven talking head synthesis
faces a significant challenge: cross-modal facial parameter generation from
audio fails to precisely capture a specific person’s facial nuances. To our
knowledge, we first apply 3D GS to this task and overcome the above problem.
By Speaker-specific BlendShapes, our method facilitates the synthesis of more
lifelike videos, charting a new course in the field of audio-driven high-
fidelity talking head synthesis.

##  3\. Method

###  3.1. Overview

In this section, we provide a detailed description of the architectural
components of our proposed framework, GaussianTalker. GaussianTalker employs
the parametric FLAME model(Li et al., 2017) to serve as a bridge between
facial animation and rendering. As shown in Figure 2, the overall framework
consists of two main modules: 1) Speaker-Specific Motion Translator, which
converts audio signals into speaker-specific FLAME parameters sequence for
facial animation control; 2) Dynamic Gaussian Renderer, which utilizes FLAME
to drive 3D Gaussians and renders dynamic talking head in real-time. The
following subsections detail the design and training mechanisms of each
module.

###  3.2. Speaker-specific Motion Translator

Given the diversity of audio inputs, we recognize that model generalization is
challenging when solely dependent on video data from specific individuals.
Consequently, we train the module using a multilingual, multi-individual
dataset to improve its adaptability to diverse audio inputs. However, due to
substantial variations in individual speaking styles, the distribution of
FLAME parameters generated by the module might diverge from those of the
target speaker, potentially compromising the rendered videos’ realism. To
address this issue, we develop a Universal Audio Encoder that decouples
identity information from content information, and a Customized Motion Decoder
that integrates personalization features. Additionally, we refer to
SelfTalk(Peng et al., 2023) and introduce a lip-reading constraint mechanism
based on self-supervision to further refine the synchronization of lip
movements. This module’s detailed architecture and workflow are illustrated in
Figure 2(b).

Figure 3. Negative audios $A_{neg,i}$ are obtained by dividing the audio and
positive audio $A_{pos}$ is obtained by timbre conversion. Audios are encoded
to get the corresponding features for adversarial learning to fine-tune the
encoder to become a Universal Audio Encoder.

Pretraining universal encoder.

####  3.2.1. Universal Audio Encoder

Audio signals often contain both speaker identity information and content
information, which are usually tightly coupled. This indicates that in
addition to conveying the content of speech, the audio also carries attributes
unique to the speaker, such as pitch and timbre. In the task of audio feature
extraction, our goal is to capture speech content while excluding information
related to speaker identity. Therefore, we start at the audio encoding stage
to achieve effective decoupling of identity and content information. We
extract audio features using the audio encoder of a pre-trained Wav2Vec
2.0-based multilingual ASR model(Xu et al., 2021). As shown in Figure 3, we
segment a speaker’s audio into $k+1$ segments, and then randomly select a
segment as the anchor audio $A_{anchor}=A_{c1,t1}$. Next, we perform timbre
conversion(Qin et al., 2023) on the anchor audio to obtain positive samples
$A_{pos}=A_{c1,t2}$, while the remaining audio segments are used as negative
samples. The subscripts $c_{i},t_{i}$ denote the content and timbre of the
audio respectively. By encoding the audio, we get the respective feature,
whose formulas are as follows:

(1) $\bm{f}_{t}=E_{uni}({A}_{t};\theta_{uni}),$

where $\bm{f}_{t}$ denotes the audio feature, $E_{uni}$ and $\theta_{uni}$
denote the audio encoder and its learnable parameters, and ${A}_{t}$ denotes
the audio.

Positive samples are consistent in content with the anchor audio, and we
anticipate them to generate similar audio features. For negative sample
features, we aim for maximal dissimilarity from the anchor audio feature to
achieve de-identification of audio encoding. To achieve this goal, we use the
InfoNCE loss(van den Oord et al., 2018) for optimization. To maintain the
original capability of the encoder, we use $E_{asr}$ and $D_{asr}$, encoder
and decoder of the pre-trained Wav2Vec 2.0 model(Xu et al., 2021) to guide the
optimization:

(2) $\displaystyle\mathcal{L}_{adv}$
$\displaystyle=-\log\frac{\exp(\bm{f}_{anchor}\cdot\bm{f}_{0}/\tau)}{\sum_{i=1}^{k}\exp(\bm{f}_{anchor}\cdot\bm{f}_{i}/\tau)},$
(3) $\displaystyle\mathcal{L}_{text}$
$\displaystyle=\mathcal{L}_{ctc}(D_{asr}(\bm{f}_{anchor}),D_{asr}(E_{asr}(A_{anchor}))),$

where $\tau$ denotes the temperature.

####  3.2.2. Customized Motion Decoder

We introduce an identity embedding to generate motions that capture the
speaker’s talking style, such as the difference in the degree of mouth opening
when different speakers talk. The Identity embedding takes personal identity
information as input and generates the personalized feature consistent with
the audio feature dimensions. Next, the personalized feature is fused with the
audio feature to serve as the input of the Customized Motion Decoder. The
decoder based on a transformer-based structure, outputs the FLAME parameters
sequence directly:

(4) $\hat{\bm{Y}}_{1:T}=D_{m}(E_{id}(p;\theta_{id})+E_{uni}({A}),\theta_{m}),$

where $\hat{\bm{Y}}_{1:T}=(\hat{\bm{y}}_{1},...,\hat{\bm{y}}_{T})$ denotes the
generated FLAME parameters sequence, $D_{m}$ denotes the motion decoder,
$E_{id}$ denotes the identity embedding, $p$ denotes the speaker’s identity,
and $\theta_{id},\theta_{m}$ denote the learnable parameters of the
corresponding component, respectively.

####  3.2.3. Training Object

The training objectives of Speaker-specific Motion Translator include three
components: reconstruction, lip motion smoothness, and potential consistency.

Reconstruction. To ensure the generated motion sequence precisely aligns with
the ground truth data, we calculate the mean square error of the ground truth
$\bm{Y}_{1:T}=(\bm{y}_{1},...,\bm{y}_{T})$ and the generated FLAME parameters
sequence $\hat{\bm{Y}}_{1:T}$, as well as the mean square error of the
corresponding vertices of the FLAME mesh
$\bm{V}_{1:T}=(\bm{v}_{1},...,\bm{v}_{T})$ and
$\hat{\bm{V}}_{1:T}=(\hat{\bm{v}}_{1},...,\hat{\bm{v}}_{T})$:

(5) $\mathcal{L}_{rec}=\lambda_{y}\frac{1}{N\times
T}\sum_{t=1}^{T}\left\|\bm{y}_{t}-\hat{\bm{y}}_{t}\right\|^{2}+\lambda_{v}\frac{1}{K\times
T}\sum_{t=1}^{T}\left\|\bm{v}_{t}-\hat{\bm{v}}_{t}\right\|^{2},$

where $N$ denotes the dimension of FLAME parameters, $K$ denotes the number of
FLAME mesh vertices, $T$ denotes the total number of video frames.

Lip motion smoothness. To avoid the potential motion jitter problem that
results from relying solely on reconstruction loss, we enhance lip motion
smoothness by calculating parameter changes across frames:

(6) $\mathcal{L}_{sth}=\lambda_{sth}\frac{1}{N\times
T}\sum_{t=1}^{T}\left\|(\bm{y}_{t}-\bm{y}_{t-1})-(\hat{\bm{y}}_{t}-\hat{\bm{y}}_{t-1})\right\|^{2}.$

Potential consistency. During training, the generated motion sequence is
passed to the lip-reading encoder to extract lip-reading feature. To ensure
coherence between the audio-text feature and lip-reading feature, we compute
their mean square error:

(7) $\mathcal{L}_{lat}=\lambda_{lat}\frac{1}{D\times
T}\left\|E_{asr}({A})-E_{lip}(\hat{\bm{V}}_{1:T},M_{L})\right\|^{2},$

where $M_{L}$ denotes the index of lip region, $E_{lip}$ denotes the lip
encoder, and $D$ denotes the feature dimension.

The final loss function is formulated as follows:

(8) $\mathcal{L}=\mathcal{L}_{rec}+\mathcal{L}_{sth}+\mathcal{L}_{lat}.$

###  3.3. Dynamic Gaussian Renderer

3D GS(Kerbl et al., 2023) is a rasterization technique designed for the real-
time rendering of photorealistic scenes. It utilizes a collection of 3D
Gaussians for modeling, where each Gaussian in the model is characterized by
the following learnable parameters: position $\bm{u}\in\mathbb{R}^{3}$,
indicating the center of the Gaussian; rotation $\bm{r}\in\mathbb{R}^{3\times
3}$, defining its orientation; scaling factor $\bm{s}\in\mathbb{R}^{3}$,
controlling its size; opacity $\bm{\alpha}\in\mathbb{R}$, determining its
visibility; and spherical harmonics coefficients
$\bm{\kappa}\in\mathbb{R}^{3\times(k+1)^{2}}$, which determine the RGB color
$\bm{c}$ through a k-order spherical harmonic function. This set of parameters
for each Gaussian is denoted as
$\mathcal{G}=\\{\bm{u},\bm{r},\bm{s},\bm{\alpha},\bm{\kappa}\\}$. During the
rendering process, the color $C$ of each pixel is determined by aggregating
the contributions of all overlapping Gaussians, as described by the following
equation:

(9) $C=\sum_{i\in
N}\bm{c}_{i}\bm{\alpha}_{i}\prod_{j=1}^{i-1}(1-\bm{\alpha}_{j}).$

We aim to employ explicit 3D Gaussians to represent facial expressions, but
static 3D Gaussians fall short of capturing dynamic changes in expressions.
Consequently, we devised a Dynamic Gaussian Renderer that anchors the
Gaussians to the FLAME triangles. This approach leverages FLAME’s BlendShapes
and Skin Weights to control the deformation of the Gaussians. We also
incorporate some Speaker-specific Blendshapes to enhance geometric and
textural detail in facial rendering and introduce an Inpainting Generator to
seamlessly integrate the rendered face with the original image. The complete
pipeline is presented in Figure 2(c).

####  3.3.1. Dynamic Deformation

Given a set of FLAME parameters, $\bm{\beta}$ for shape, $\bm{\varepsilon}$
for expression, and $\bm{\psi}$ for pose changes, the vertices of the FLAME
model in the global space, denoted as $\bm{v}$, are determined by the
following motion rules:

(10)
$\bm{v}=LBS(\bm{v}_{base}+BS(\\{\bm{\beta};\bm{\varepsilon};\bm{\psi}\\};W_{bs}),J(\bm{\psi}),W),$

where $\bm{v}_{base}$ represents the vertices of the FLAME mesh in canonical
space. $LBS$ signifies the standard Linear Blend Skinning function with
weights $W$ and the function $J$ stands for the joint regressor based on the
pose $\bm{\psi}$, both as defined in the FLAME model. The $BS$ operation
denotes a linear blend shaping process that creates facial animations by
combining blendshapes in accordance with FLAME parameters, weighted by the
blendshape weights $W_{bs}$. For a triangle $\mathcal{T}$ of the FLAME mesh,
which is defined by its vertices $\\{\bm{v}_{0},\bm{v}_{1},\bm{v}_{2}\\}$ and
its edges
$\\{\bm{e}_{01}=\bm{v}_{1}-\bm{v}_{0},\bm{e}_{02}=\bm{v}_{2}-\bm{v}_{0},\bm{e}_{12}=\bm{v}_{2}-\bm{v}_{1}\\}$,
we establish a local coordinate system based on $\mathcal{T}$. The system’s
origin, $P$, representing $\mathcal{T}$’s position in global space, is
determined by a barycentric combination of the triangle’s vertices. The
system’s orientation in global space is captured by the rotation matrix $R$,
and the scale factor $S$ reflects the size of $\mathcal{T}$. These components
are delineated as follows:

(11) $\displaystyle P$
$\displaystyle=\eta_{0}\bm{v}_{0}+\eta_{1}\bm{v}_{1}+\eta_{2}\bm{v}_{2},$ (12)
$\displaystyle R$ $\displaystyle=[\bm{n}_{0};\bm{n}_{1};\bm{n}_{2}],$ (13)
$\displaystyle S$
$\displaystyle=(\|\bm{e}_{01}\|+\|\bm{e}_{02}\|+\|\bm{e}_{12}\|)/3,$

where $\bm{\eta}=(\eta_{0},\eta_{1},\eta_{2})$ is the learnable barycentric
coefficients, and the orthogonal unit vectors
$[\bm{n}_{0};\bm{n}_{1};\bm{n}_{2}]$ are calculated as follows:

(14)
$\displaystyle\bm{n_{0}}=\frac{\bm{e}_{01}}{\|\bm{e}_{01}\|},\bm{n_{1}}=\frac{\bm{e}_{01}\times\bm{e}_{02}}{\|\bm{e}_{01}\times\bm{e}_{02}\|},\bm{n_{2}}=\bm{n_{0}}\times\bm{n_{1}}.$

Then a Gaussian is associated within this local space, denoted as
$\mathcal{G}=\\{\bm{\bar{u}},\bm{\bar{r}},\bm{\bar{s}},\bm{\alpha},\bm{\kappa},\mathcal{T},\bm{\eta}\\}$,
enabling $\mathcal{G}$ to track the movements of $\mathcal{T}$. Specifically,
the attributes $\bm{\bar{u}},\bm{\bar{r}},\bm{\bar{s}}$ of $\mathcal{G}$ are
defined relative to the local coordinate system, rather than the global space.
We initialize its position $\bm{\bar{u}}$ at the origin, rotation
$\bm{\bar{r}}$ as the identity matrix, and scale $\bm{\bar{s}}$ as a unit
vector. When rendering, we transform the $\mathcal{G}$ into global space by:

(15)
$\displaystyle\bm{u}=R\cdot\bm{\bar{u}}+P,\bm{r}=R\cdot\bm{\bar{r}},\bm{s}=S\bm{\bar{s}},$

where $\bm{u},\bm{r},\bm{s}$ denotes the global position, orientation, scale
of $\mathcal{G}$.

During optimization, we employ an adaptive density control strategy similar to
the one described by 3D GS(Kerbl et al., 2023), to dynamically add or remove
Gaussians based on the view space positional gradient and the opacity of each
Gaussian. Specifically, when a Gaussian $\mathcal{G}_{i}$ is anchored to a
triangle $\mathcal{T}_{i}$ of the FLAME mesh, all its derivatives generated by
cloning or splitting from $\mathcal{G}_{i}$, denoted as
($\mathcal{G}_{i,1},\mathcal{G}_{i,2},\mathcal{G}_{i,3},...)$, inherit the
same local coordinate system that we established based on $\mathcal{T}_{i}$.

####  3.3.2. Speaker-specific BlendShapes

We have addressed the challenge of capturing expressive facial motion at a
coarse level by binding Gaussians to the triangles of the FLAME mesh. However,
since FLAME primarily encodes only coarse facial deformations, it remains a
challenge to reproduce speaker-specific facial geometry and texture details,
such as teeth and wrinkles. To overcome this limitation, we introduce a set of
learnable BlendShape (BS) weights that are designed to capture and delineate
these nuanced facial features. Initially, we define a Multilayer Perceptron
(MLP), designated as $W_{\gamma}$, which accepts the pose parameters
$\bm{\psi}$ from the FLAME and generates a latent pose representation
$\bm{\gamma}$ as follows:

(16) $\bm{\gamma}=W_{\gamma}(\bm{\psi}).$

To refine geometrical details, specialized BS weights, $W_{pos}$ for position
and $W_{rot}$ for rotation, are employed to update the corresponding
attributes of Gaussian $\mathcal{G}$ within the local space:

(17)
$\bm{\bar{u}}^{\prime}=\bm{\bar{u}}+BS(\bm{\gamma};W_{pos}),\bm{\bar{r}}^{\prime}=\bm{\bar{r}}\cdot
BS(\bm{\gamma};W_{rot}).$

These revised local attributes, $\bm{\bar{u}}^{\prime}$ and
$\bm{\bar{r}}^{\prime}$, are then converted to global space when rendering, as
explained in Equation 15. Similarly, for texture refinement, we apply
compensation to $\bm{\kappa}_{0}$, the zeroth-order coefficient of
$\bm{\kappa}$ in $\mathcal{G}$, which governs the base color calculated by
zeroth-order spherical harmonics function. This adjustment is achieved through
BS weights designated as $W_{color}$:

(18) $\bm{\kappa}_{0}^{\prime}=\bm{\kappa}_{0}+BS(\bm{\gamma};W_{color}).$

With the incorporation of Speaker-specific BlendShapes for facial detailing,
$\mathcal{G}$ is represented as
$\mathcal{G}=\\{\bm{\bar{u}}^{\prime},\bm{\bar{r}}^{\prime},\bm{\bar{s}},\bm{\alpha},\bm{\kappa}_{0}^{\prime},\bm{\kappa}_{rest},\mathcal{T},\bm{\eta}\\}$,
where $\bm{\kappa}_{rest}$ denotes the higher-order(1 to k-th) coefficients of
$\bm{\kappa}$.

Table 1. Quantitative results under the self-driven setting. The best and second-best results are in bold and underlined. Methods |  PSNR↑ |  SSIM↑ |  LPIPS↓ |  FID↓ |  LMD↓ |  LSE-C↑ |  LSE-D↓ |  FPS↑
---|---|---|---|---|---|---|---|---
Ground Truth |  — |  — |  — |  — |  — |  7.465 |  7.131 |  —
Wav2Lip(Prajwal et al., 2020) |  33.6435 |  0.9506 |  0.0602 |  17.44 |  6.048 |  9.129* |  5.860* |  23
AD-NeRF(Guo et al., 2021) |  30.7756 |  0.9200 |  0.1135 |  26.98 |  3.975 |  6.128 |  8.134 |  0.15
RAD-NeRF(Tang et al., 2022) |  29.9283 |  0.9115 |  0.1075 |  18.73 |  3.519 |  6.096 |  8.147 |  37
ER-NeRF(Li et al., 2023) |  29.5774 |  0.9071 |  0.0694 |  13.22 |  3.555 |  6.340 |  7.955 |  38
GeneFace++(Ye et al., 2023a) |  27.4192 |  0.8870 |  0.0920 |  12.69 |  3.942 |  5.920 |  8.378 |  44
Ours |  37.0775 |  0.9676 |  0.0239 |  4.57 |  3.278 |  7.015 |  7.562 |  130

####  3.3.3. Inpainting Generator

By incorporating Dynamic Deformation and Speaker-specific BlendShapes, we
achieve a consistent alignment of the head pose with the original face image,
which enables a stable reintegration of the rendered face into the original
video frame. Nonetheless, using an alternative audio source may lead to
discrepancies in lip shapes between the rendered and original video frames.
These discrepancies are particularly pronounced around the facial contour and
chin, resulting in visual inconsistencies. To address these issues and enhance
the integrity of the composite image, we introduce an Inpainting Generator,
denoted as $F$. It is designed to seamlessly fill in these mismatches and
improve the visual continuity of the final frames. The blending process is
described by the following equation:

(19) $I=(1-M)\cdot I_{ori}+M\cdot F(I_{gau}+(1-M)I_{ori}),$

where $M$ denotes the facial mask extracted from the original frame by face
parsing(Zheng et al., 2022), and $I_{ori}$ and $I_{gau}$ denote the original
video frame and the output image of the Gaussian renderer, respectively.

####  3.3.4. Training Object

While training the Dynamic Gaussian Renderer, we focus on three types of
optimization objectives: image similarity, Gaussian attributes, and Gaussian
semantic category.

Image Similarity. The image similarity loss comprises a combination of L1 loss
$\mathcal{L}_{1}$, VGG loss $\mathcal{L}_{vgg}$, and SSIM loss
$\mathcal{L}_{ssim}$:

(20)
$\mathcal{L}_{rgb}=\lambda_{1}\mathcal{L}_{1}+\lambda_{2}\mathcal{L}_{vgg}+\lambda_{3}\mathcal{L}_{ssim}.$

Gaussian Attributes. We aim to ensure that during optimization, the positions
of the Gaussians faithfully align with the associated FLAME triangles and
their sizes remain within reasonable bounds to avoid visual jitter and
artifacts. To achieve this, we apply constraints on the Gaussians’ position
and scale:

(21)
$\mathcal{L}_{attr}=\lambda_{p}\left\|max(0,\bm{\bar{u}}^{\prime}-\epsilon_{p})\right\|^{2}+\lambda_{s}\|{max(0,\bar{\bm{s}}-\epsilon_{s})}\|^{2},\\\
$

where $\epsilon_{p}$ and $\epsilon_{s}$ serve as predefined thresholds that
set the maximum allowable positions for $\bm{\bar{u}}^{\prime}$ and scales for
$\bm{\bar{s}}$, respectively.

Gaussian Semantic Category. The dynamic deformation aims for the deformed
Gaussians to approximate the associated FLAME mesh. Ideally, Gaussians bound
to specific triangles, like the lips, should exactly match the corresponding
region’s color and position in the rendered image. However, it’s a challenge
to prevent Gaussians from shifting to implausible positions during
optimization, resulting in jittering artifacts. To address this issue, we
introduce a Gaussian semantic loss $\mathcal{L}_{seg}$. We render an auxiliary
semantic segmentation map $M_{aux}$ using the Gaussian Renderer, where instead
of utilizing the learned color $\bm{c}_{i}$ in Equation 9, we assign a fixed
color $\bm{c}_{fix}$ to each Gaussian point based on its parent triangle’s
category; for example, assigning red to facial points and yellow to lip
points:

(22) $\displaystyle
C_{seg}=\sum_{i=1}\bm{c}_{fix}\bm{\alpha}_{i}\prod_{j=1}^{i-1}(1-\bm{\alpha}_{j}).$

Then, we compute the mean square error between $M_{gau}$ and $M_{gt}$ obtained
by face parsing(Zheng et al., 2022):

(23) $\displaystyle\mathcal{L}_{seg}=\lambda_{seg}\|M_{aux}-M_{gt}\|^{2}.$

Consequently, the final loss function is formulated as follows:

(24) $\mathcal{L}=\mathcal{L}_{rgb}+\mathcal{L}_{attr}+\mathcal{L}_{seg}.$

Table 2. Quantitative results under the cross-driven setting. The best and second-best results are in bold and underlined. Methods | Testset A | Testset B
---|---|---
FID↓ |  LSE-C↑ |  LSE-D↓ | FID↓ |  LSE-C↑ |  LSE-D↓
Wav2Lip(Prajwal et al., 2020) | 18.34 |  8.380 |  6.985 | 17.18 |  9.106 |  6.479
AD-NeRF(Guo et al., 2021) | 29.95 |  3.929 |  11.02 | 29.92 |  3.408 |  11.39
RAD-NeRF(Tang et al., 2022) | 19.55 |  4.028 |  10.78 | 19.18 |  4.035 |  10.93
ER-NeRF(Li et al., 2023) | 13.68 |  4.780 |  10.12 | 13.40 |  4.680 |  10.26
GeneFace++(Ye et al., 2023a) | 12.77 |  5.734 |  9.030 | 12.92 |  6.040 |  8.759
Ours | 5.136 |  5.758 |  8.665 | 4.394 |  7.148 |  7.617

##  4\. Experiment

###  4.1. Experimental Settings

Figure 4. The comparison of generated key frame results. We show the ground
truth frames for comparing and mark the un-sync and bad rendering quality
results with red arrows. Please zoom in for better visualization.

Comparison of methods.

####  4.1.1. Datasets

To facilitate comparison with other state-of-the-art methods, we utilize the
same dataset as AD-NeRF and GeneFace++, comprised of five videos averaging
9085 frames each, at a frame rate of 25 FPS, across English, French, and
Korean languages. To improve the Speaker-specific Motion Translator’s
generalization capabilities, we incorporate data from target speakers,
supplemented with 100 samples of Chinese CN-CVS videos(Chen et al., 2023a) and
100 samples of English HDTF videos(Zhang et al., 2021) for joint training.
During the data preprocessing phase, we begin by extracting audio and video
frames from the source video. Subsequently, facial keypoints are extracted and
smoothed(Bulat and Tzimiropoulos, 2017; Kroeger et al., 2016), then employ
EMOCA(Daněček et al., 2022) to obtain FLAME parameters sequence. Finally, we
apply the face parsing technique(Zheng et al., 2022), which segments each
video frame into distinct facial regions.

####  4.1.2. Compared baselines

We compare our method with several representative methods: 1) Wav2Lip(Prajwal
et al., 2020): Sync-expert supervised learning enhances lip sync accuracy; 2)
AD-NeRF(Guo et al., 2021): NeRF synthesizes audio-driven videos end-to-end; 3)
RAD-NeRF(Tang et al., 2022): Discrete learnable mesh boosts rendering
efficiency; 4) ER-NeRF(Li et al., 2023): Three-plane hash representation
enables high-quality rendering; 5) GeneFace++(Ye et al., 2023a): Pitch-aware
audio-to-motion module and landmark LLE method ensure stable rendering.

####  4.1.3. Implementation Details

The official FLAME model lacks vertices and triangles for teeth, so we
manually attach it with 262 vertices and 546 faces to depict the teeth and
inner mouth regions. These newly added elements are not assigned independent
BS and LBS weights but rather are designed to move in sync with the lip area
of the original FLAME model. For GaussianTalker’s training, we employ the Adam
Optimizer(Kingma and Ba, 2014) across all modules. The Speaker-specific Motion
Translator is trained for 100,000 iterations, with the batch size set to 1.
This training takes about 6 hours, using a learning rate of $1\times 10^{-4}$.
Furthermore, we train the Dynamic Gaussian Renderer for 50,000 iterations
using a batch size of 8, and this training lasts about 4 hours. While
optimizing the Gaussians’ attributes, we fine-tune the FLAME parameters within
the dataset to ensure they more accurately reflect the face of each frame. All
experiments are performed on a single NVIDIA RTX4090 GPU.

††footnotetext: * Wav2Lip is jointly trained with SyncNet, and LSE-C and LSE-D
are its optimization objectives. As a result, it obtains better scores than
the ground truth.

###  4.2. Quantitative Evaluation

####  4.2.1. Comparison settings

In quantitative evaluation, we focus on both synthesized quality of the head
and accuracy of lip movements. Our comparisons are divided into two settings:
1) Self-driven setting, where each speaker’s own audio is used to drive the
corresponding reference frame. 2) Cross-driven setting, where we extract two
extra audio clips from HDTF(Zhang et al., 2021), named Testset A and Testset
B, to drive each speaker. For each generated result, we rescaled frames into
the same size for a fair comparison.

####  4.2.2. Evaluation Metrics

We employ Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index
Measure (SSIM) for assessing overall image quality. To assess image details,
we utilize Learned Perceptual Image Patch Similarity (LPIPS)(Zhang et al.,
2018). To quantify the similarity between the generated video and the ground
truth, we employ Fréchet Inception Distance (FID)(Heusel et al., 2017). For
assessing audio-lip synchronization, we utilize Landmark Distance (LMD)(Chen
et al., 2018) to gauge the extent of synchronization with the ground truth,
and complementarily evaluate lip movement accuracy through Lip Sync Error
Confidence (LSE-C) and Lip Sync Error Distance (LSE-D)(Prajwal et al., 2020).

####  4.2.3. Evaluation Results

The results are presented in Table 1. Our results indicate that, in comparison
with leading audio-driven talking head video synthesis methods, our method
outperforms others, achieving the highest scores across all image quality
assessment metrics. Owing to measures implemented to reduce jitter and
artifacts, the videos produced by our method more accurately resemble the
actual scenes. Regarding lip synchronization, although Wav2Lip excels in the
LSE-C and LSE-D metrics owing to its joint training with SyncNet, our method
still surpasses others in these metrics. Additionally, our method demonstrates
superior performance in the LMD metric, suggesting a closer and more accurate
alignment with actual lip movements.

We also compared the generalization ability of each method using FID score and
LSE-C and LSE-D metrics with out-of-distribution (OOD) audio inputs, detailed
in Table 2. The results show that our method exhibits outstanding image
quality performance, with FID reflecting a considerable enhancement over
competing methods. Furthermore, we have significantly improved the model’s
generalization capabilities by training the Speaker-specific Motion Translator
on a multi-individual dataset. This enhancement enables our method to attain
superior performance in lip synchronization, as evidenced by the LSE-C and
LSE-D.

We also tested the rendering speed. With an NVIDIA RTX4090 GPU and data
preloaded into memory, a sequence of video frames can be output at 130 FPS.
The output video has the same resolution as the original video. This far
exceeds the performance of other methods. In addition, we developed a
rendering pipeline for Gaussians with MNN(Jiang et al., 2020) and OpenGL(Woo
et al., 1999) and subsequently deployed GaussianTalker. On devices with the
Apple M1 chip, it achieved rendering speeds of 36 FPS, demonstrating its
adaptability across various platforms.

###  4.3. Qualitative Evaluation

To more effectively assess image quality and lip synchronization, we present a
comparative analysis of our method with others in Figure 4. Compared to our
method, Wav2Lip primarily concentrates on lip synchronization yet suffers low
image quality, particularly around the mouth area. AD-NeRF and RAD-NeRF cannot
naturally perform the blinking process, as evidenced by artifacts appearing in
eye closure frames. Moreover, these two methods suffer from head-torso
separation when driving cross-identity audio. ER-NeRF demonstrates noticeable
jittering in both the head and torso during speech, which severely impacts the
realism of the video. GeneFace++ shows limited expressiveness and lip movement
variety, failing to perform actions like staring or frowning convincingly.

Most NeRF-based approaches train the head and torso independently, which can
compromise the torso’s rendering quality in scenes with extensive torso
movement. Furthermore, rendering can introduce artifacts stemming from the
incomplete separation of the speaker from the background during preprocessing.
In contrast, our approach not only delivers superior image clarity but also
achieves enhanced lip synchronization performance, particularly with cross-
identity and cross-gender audio inputs, showcasing robust generalization
capabilities. Please see our supplementary for better visualization.

###  4.4. Ablation study

In this section, we conducted ablation studies in a self-driven setting to
validate the effectiveness of each component. Quantitative assessment metrics
included PSNR, LPIPS, and LMD. The results are shown in Table 3 and Figure 5.

Table 3. Ablation study on subject Obama. The best overall results are in bold. Ablation |  PSNR↑ | LPIPS↓ | LMD↓
---|---|---|---
full |  38.37 | 0.0089 | 3.72
replace Universal Audio Encoder
with Wav2Vec 2.0(Xu et al., 2021) |  36.91 | 0.0104 | 4.55
w/o Speaker-specific BlendShapes |  37.46 | 0.0098 | 3.78
w/o Fine-tune FLAME Parameters |  36.87 | 0.0106 | 4.28
w/o Gaussian Semantic Loss |  37.09 | 0.0104 | 3.97

####  4.4.1. Universal Audio Encoder

We investigate the impact of audio decoupling in the audio encoder, which
excludes identity information from the original audio and extracts audio
feature that contains only content information. Removing audio decoupling
decreases all metrics, with the LMD metric suffering the most, indicating
poorer lip synchronization. Figure 5 (a) also illustrates the artifact issues
in some of the lip shapes.

Figure 5. Ablation study on audio decoupling and Speaker-Specific BlendShape.
Removing them will lead to (a) and (b).

Abalation.

####  4.4.2. Speaker-specific BlendShapes

We explore the impact of eliminating the Speaker-specific BlendShapes, which
play a crucial role in depicting facial details. The absence may cause the
Gaussian renderer to generate artifacts in regions that are not explicitly
governed by FLAME parameters, like teeth and wrinkles, resulting in
distortions, particularly in fine features, as shown in Figure 5 (b).

####  4.4.3. Fine-tune FLAME Parameters

This optimization aligns the FLAME parameters with the original video and
reduces inter-frame jitter. Without it, the Gaussian point cloud learns biased
facial representations. As a result, the visual jitter issue becomes more
pronounced and the image quality metrics are significantly declined.

####  4.4.4. Gaussian Semantic Loss

We investigate the effect of ablating Gaussian semantic loss, which clarifies
the binding relationship between 3D Gaussians and FLAME and further normalizes
the motion of Gaussians. Without the Gaussian semantic loss, the speaker will
jitter slightly, leading to a reduction in image quality metrics.

##  5\. Conclusion

In this work, we propose GaussianTalker, a novel framework for audio-driven
talking head synthesis via 3D Gaussian Splatting integrated with the FLAME
model. GaussianTalker associates multimodal data with specific speakers,
reducing potential identity bias between audio, 3D mesh, and video. The
Speaker-specific FLAME Translator employs identity decoupling and personalized
embedding to achieve synchronized and natural lip movements, while the Dynamic
Gaussian Renderer refines Gaussian attributes through a latent pose for stable
and realistic rendering. Extensive experiments showed that GaussianTalker
outperforms state-of-the-art performance in talking head synthesis, while
achieving ultra-high rendering speed that significantly surpasses other
methods. We believe that this innovative approach will encourage future
studies to develop more fluid and lifelike character expressions and
movements. By leveraging advanced Gaussian models and generative techniques,
the animation of characters will go far beyond mere lip-syncing, capturing a
broader range of character dynamics.

## References

  * (1)
  * Afouras et al. (2018) Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman. 2018.  LRS3-TED: a large-scale dataset for visual speech recognition.  _arXiv preprint arXiv:1809.00496_ (2018).
  * Blanz and Vetter (2023) Volker Blanz and Thomas Vetter. 2023.  A morphable model for the synthesis of 3D faces.  In _Seminal Graphics Papers: Pushing the Boundaries, Volume 2_. 157–164.
  * Brand (1999) Matthew Brand. 1999.  Voice puppetry. In _Proceedings of the 26th annual conference on Computer graphics and interactive techniques_. 21–28.
  * Bregler et al. (2023) Christoph Bregler, Michele Covell, and Malcolm Slaney. 2023.  Video rewrite: Driving visual speech with audio.  In _Seminal Graphics Papers: Pushing the Boundaries, Volume 2_. 715–722.
  * Bulat and Tzimiropoulos (2017) Adrian Bulat and Georgios Tzimiropoulos. 2017.  How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks). In _International Conference on Computer Vision_.
  * Cao et al. (2013) Chen Cao, Yanlin Weng, Shun Zhou, Yiying Tong, and Kun Zhou. 2013.  Facewarehouse: A 3d facial expression database for visual computing.  _IEEE Transactions on Visualization and Computer Graphics_ 20, 3 (2013), 413–425.
  * Chatziagapi et al. (2023) Aggelina Chatziagapi, ShahRukh Athar, Abhinav Jain, MV Rohith, Vimal Bhat, and Dimitris Samaras. 2023.  LipNeRF: What is the right feature space to lip-sync a NeRF?. In _2023 IEEE 17th International Conference on Automatic Face and Gesture Recognition (FG)_. IEEE, 1–8.
  * Chen et al. (2023a) Chen Chen, Dong Wang, and Thomas Fang Zheng. 2023a.  CN-CVS: A mandarin audio-visual dataset for large vocabulary continuous visual to speech synthesis. In _ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 1–5.
  * Chen et al. (2018) Lele Chen, Zhiheng Li, Ross K Maddox, Zhiyao Duan, and Chenliang Xu. 2018.  Lip Movements Generation at a Glance. In _Proceedings of the European Conference on Computer Vision (ECCV)_.
  * Chen et al. (2019) Lele Chen, Ross K Maddox, Zhiyao Duan, and Chenliang Xu. 2019.  Hierarchical cross-modal talking face generation with dynamic pixel-wise loss. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 7832–7841.
  * Chen et al. (2023b) Yufan Chen, Lizhen Wang, Qijing Li, Hongjiang Xiao, Shengping Zhang, Hongxun Yao, and Yebin Liu. 2023b.  Monogaussianavatar: Monocular gaussian point-based head avatar.  _arXiv preprint arXiv:2312.04558_ (2023).
  * Chung et al. (2018) Joon Son Chung, Arsha Nagrani, and Andrew Zisserman. 2018.  Voxceleb2: Deep speaker recognition.  _arXiv preprint arXiv:1806.05622_ (2018).
  * Daněček et al. (2022) Radek Daněček, Michael J Black, and Timo Bolkart. 2022.  Emoca: Emotion driven monocular face capture and animation. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 20311–20322.
  * Ezzat et al. (2002) Tony Ezzat, Gadi Geiger, and Tomaso Poggio. 2002.  Trainable videorealistic speech animation.  _ACM Transactions on Graphics (TOG)_ 21, 3 (2002), 388–398.
  * Gafni et al. (2021) Guy Gafni, Justus Thies, Michael Zollhofer, and Matthias Nießner. 2021.  Dynamic neural radiance fields for monocular 4d facial avatar reconstruction. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 8649–8658.
  * Goodfellow et al. (2014) Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014.  Generative adversarial nets.  _Advances in neural information processing systems_ 27 (2014).
  * Guo et al. (2021) Yudong Guo, Keyu Chen, Sen Liang, Yong-Jin Liu, Hujun Bao, and Juyong Zhang. 2021.  Ad-nerf: Audio driven neural radiance fields for talking head synthesis. In _Proceedings of the IEEE/CVF international conference on computer vision_. 5784–5794.
  * Heusel et al. (2017) Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2017.  GANs trained by a two time-scale update rule converge to a local nash equilibrium. In _Proceedings of the 31st International Conference on Neural Information Processing Systems_ (Long Beach, California, USA) _(NIPS’17)_. Curran Associates Inc., Red Hook, NY, USA, 6629–6640.
  * Isola et al. (2017) Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. 2017.  Image-to-image translation with conditional adversarial networks. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 1125–1134.
  * Jamaludin et al. (2019) Amir Jamaludin, Joon Son Chung, and Andrew Zisserman. 2019.  You said that?: Synthesising talking faces from audio.  _International Journal of Computer Vision_ 127 (2019), 1767–1779.
  * Jiang et al. (2020) Xiaotang Jiang, Huan Wang, Yiliu Chen, Ziqi Wu, Lichuan Wang, Bin Zou, Yafeng Yang, Zongyang Cui, Yu Cai, Tianhang Yu, Chengfei Lv, and Zhihua Wu. 2020.  MNN: A Universal and Efficient Inference Engine. In _MLSys_.
  * Karras et al. (2017) Tero Karras, Timo Aila, Samuli Laine, Antti Herva, and Jaakko Lehtinen. 2017.  Audio-driven facial animation by joint end-to-end learning of pose and emotion.  _ACM Transactions on Graphics (TOG)_ 36, 4 (2017), 1–12.
  * Kerbl et al. (2023) Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 2023.  3D Gaussian Splatting for Real-Time Radiance Field Rendering.  (2023).
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ (2014).
  * Kroeger et al. (2016) Till Kroeger, Radu Timofte, Dengxin Dai, and Luc Van Gool. 2016.  Fast Optical Flow using Dense Inverse Search.  arXiv:arXiv:1603.03590
  * Li et al. (2023) Jiahe Li, Jiawei Zhang, Xiao Bai, Jun Zhou, and Lin Gu. 2023.  Efficient region-aware neural radiance fields for high-fidelity talking portrait synthesis. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 7568–7578.
  * Li et al. (2017) Tianye Li, Timo Bolkart, Michael J Black, Hao Li, and Javier Romero. 2017.  Learning a model of facial shape and expression from 4D scans.  _ACM Trans. Graph._ 36, 6 (2017), 194–1.
  * Li et al. (2024) Xueting Li, Shalini De Mello, Sifei Liu, Koki Nagano, Umar Iqbal, and Jan Kautz. 2024.  Generalizable One-shot 3D Neural Head Avatar.  _Advances in Neural Information Processing Systems_ 36 (2024).
  * Liu et al. (2022) Xian Liu, Yinghao Xu, Qianyi Wu, Hang Zhou, Wayne Wu, and Bolei Zhou. 2022.  Semantic-aware implicit neural audio-driven video portrait generation. In _European conference on computer vision_. Springer, 106–125.
  * Lu et al. (2021) Yuanxun Lu, Jinxiang Chai, and Xun Cao. 2021.  Live speech portraits: real-time photorealistic talking-head animation.  _ACM Transactions on Graphics (TOG)_ 40, 6 (2021), 1–17.
  * Mildenhall et al. (2020) B Mildenhall, PP Srinivasan, M Tancik, JT Barron, R Ramamoorthi, and R Ng. 2020.  Nerf: Representing scenes as neural radiance fields for view synthesis. In _European conference on computer vision_.
  * Morishima (1998) Shigeo Morishima. 1998.  Real-time Talking Head Driven by Voice and its Application to Communication and Entertainment. 195–200.
  * Park et al. (2021) Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Martin-Brualla. 2021.  Nerfies: Deformable neural radiance fields. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 5865–5874.
  * Paysan et al. (2009) Pascal Paysan, Reinhard Knothe, Brian Amberg, Sami Romdhani, and Thomas Vetter. 2009.  A 3D face model for pose and illumination invariant face recognition. In _2009 sixth IEEE international conference on advanced video and signal based surveillance_. Ieee, 296–301.
  * Peng et al. (2023) Ziqiao Peng, Yihao Luo, Yue Shi, Hao Xu, Xiangyu Zhu, Hongyan Liu, Jun He, and Zhaoxin Fan. 2023.  Selftalk: A self-supervised commutative training diagram to comprehend 3d talking faces. In _Proceedings of the 31st ACM International Conference on Multimedia_. 5292–5301.
  * Prajwal et al. (2020) KR Prajwal, Rudrabha Mukhopadhyay, Vinay P Namboodiri, and CV Jawahar. 2020.  A lip sync expert is all you need for speech to lip generation in the wild. In _Proceedings of the 28th ACM international conference on multimedia_. 484–492.
  * Pumarola et al. (2021) Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. 2021.  D-nerf: Neural radiance fields for dynamic scenes. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 10318–10327.
  * Qin et al. (2023) Zengyi Qin, Wenliang Zhao, Xumin Yu, and Xin Sun. 2023.  OpenVoice: Versatile Instant Voice Cloning.  _arXiv preprint arXiv:2312.01479_ (2023).
  * Richard et al. (2021) Alexander Richard, Michael Zollhöfer, Yandong Wen, Fernando De la Torre, and Yaser Sheikh. 2021.  Meshtalk: 3d face animation from speech using cross-modality disentanglement. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 1173–1182.
  * Rivero et al. (2024) Alfredo Rivero, ShahRukh Athar, Zhixin Shu, and Dimitris Samaras. 2024.  Rig3DGS: Creating Controllable Portraits from Casual Monocular Videos.  _arXiv preprint arXiv:2402.03723_ (2024).
  * Shang et al. (2020) Jiaxiang Shang, Tianwei Shen, Shiwei Li, Lei Zhou, Mingmin Zhen, Tian Fang, and Long Quan. 2020.  Self-supervised monocular 3d face reconstruction by occlusion-aware multi-view geometry consistency. In _European Conference on Computer Vision_. Springer, 53–70.
  * Shao et al. (2024) Zhijing Shao, Zhaolong Wang, Zhuang Li, Duotun Wang, Xiangru Lin, Yu Zhang, Mingming Fan, and Zeyu Wang. 2024.  SplattingAvatar: Realistic Real-Time Human Avatars with Mesh-Embedded Gaussian Splatting.  _arXiv preprint arXiv:2403.05087_ (2024).
  * Shen et al. (2022) Shuai Shen, Wanhua Li, Zheng Zhu, Yueqi Duan, Jie Zhou, and Jiwen Lu. 2022.  Learning dynamic facial radiance fields for few-shot talking head synthesis. In _European conference on computer vision_. Springer, 666–682.
  * Tanaka and Nakamura (2022) Hiroki Tanaka and Satoshi Nakamura. 2022.  The Acceptability of Virtual Characters as Social Skills Trainers: Usability Study.  _JMIR Hum Factors_ 9, 1 (29 Mar 2022), e35358.  https://doi.org/10.2196/35358
  * Tang et al. (2022) Jiaxiang Tang, Kaisiyuan Wang, Hang Zhou, Xiaokang Chen, Dongliang He, Tianshu Hu, Jingtuo Liu, Gang Zeng, and Jingdong Wang. 2022.  Real-time neural radiance talking portrait synthesis via audio-spatial decomposition.  _arXiv preprint arXiv:2211.12368_ (2022).
  * Thies et al. (2020a) Justus Thies, Mohamed Elgharib, Ayush Tewari, Christian Theobalt, and Matthias Nießner. 2020a.  Neural voice puppetry: Audio-driven facial reenactment. In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XVI 16_. Springer, 716–731.
  * Thies et al. (2020b) Justus Thies, Mohamed Elgharib, Ayush Tewari, Christian Theobalt, and Matthias Nießner. 2020b.  Neural Voice Puppetry: Audio-Driven Facial Reenactment. In _Computer Vision – ECCV 2020_ , Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (Eds.). Springer International Publishing, Cham, 716–731.
  * van den Oord et al. (2018) Aäron van den Oord, Yazhe Li, and Oriol Vinyals. 2018.  Representation Learning with Contrastive Predictive Coding.  _CoRR_ abs/1807.03748 (2018).  arXiv:1807.03748 http://arxiv.org/abs/1807.03748
  * Vougioukas et al. (2020) Konstantinos Vougioukas, Stavros Petridis, and Maja Pantic. 2020.  Realistic speech-driven facial animation with gans.  _International Journal of Computer Vision_ 128, 5 (2020), 1398–1413.
  * Wang et al. (2023) Jie Wang, Jiu-Cheng Xie, Xianyan Li, Feng Xu, Chi-Man Pun, and Hao Gao. 2023.  GaussianHead: Impressive Head Avatars with Learnable Gaussian Diffusion.  _arXiv preprint arXiv:2312.01632_ (2023).
  * Wang et al. (2020) Kaisiyuan Wang, Qianyi Wu, Linsen Song, Zhuoqian Yang, Wayne Wu, Chen Qian, Ran He, Yu Qiao, and Chen Change Loy. 2020.  Mead: A large-scale audio-visual dataset for emotional talking-face generation. In _European Conference on Computer Vision_. Springer, 700–717.
  * Wiles et al. (2018) Olivia Wiles, A Koepke, and Andrew Zisserman. 2018.  X2face: A network for controlling face generation using images, audio, and pose codes. In _Proceedings of the European conference on computer vision (ECCV)_. 670–686.
  * Woo et al. (1999) Mason Woo, Jackie Neider, Tom Davis, and Dave Shreiner. 1999.  _OpenGL programming guide: the official guide to learning OpenGL, version 1.2_.  Addison-Wesley Longman Publishing Co., Inc.
  * Xu et al. (2021) Qiantong Xu, Alexei Baevski, and Michael Auli. 2021.  Simple and Effective Zero-shot Cross-lingual Phoneme Recognition.  arXiv:2109.11680 [cs.CL]
  * Yao et al. (2022) Shunyu Yao, RuiZhe Zhong, Yichao Yan, Guangtao Zhai, and Xiaokang Yang. 2022.  Dfa-nerf: Personalized talking head generation via disentangled face attributes neural rendering.  _arXiv preprint arXiv:2201.00791_ (2022).
  * Ye et al. (2023a) Zhenhui Ye, Jinzheng He, Ziyue Jiang, Rongjie Huang, Jiawei Huang, Jinglin Liu, Yi Ren, Xiang Yin, Zejun Ma, and Zhou Zhao. 2023a.  GeneFace++: Generalized and Stable Real-Time Audio-Driven 3D Talking Face Generation.  _arXiv preprint arXiv:2305.00787_ (2023).
  * Ye et al. (2023b) Zhenhui Ye, Ziyue Jiang, Yi Ren, Jinglin Liu, Jinzheng He, and Zhou Zhao. 2023b.  Geneface: Generalized and high-fidelity audio-driven 3d talking face synthesis.  _arXiv preprint arXiv:2301.13430_ (2023).
  * Yi et al. (2020) Ran Yi, Zipeng Ye, Juyong Zhang, Hujun Bao, and Yong-Jin Liu. 2020.  Audio-driven talking face video generation with natural head pose.  _arXiv preprint arXiv:2002.10137_ 2, 6 (2020), 7.
  * Yu et al. (2020) Lingyun Yu, Jun Yu, Mengyan Li, and Qiang Ling. 2020.  Multimodal inputs driven talking face generation with spatial–temporal dependency.  _IEEE Transactions on Circuits and Systems for Video Technology_ 31, 1 (2020), 203–216.
  * Zhang et al. (2018) Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. 2018.  The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Zhang et al. (2021) Zhimeng Zhang, Lincheng Li, Yu Ding, and Changjie Fan. 2021.  Flow-Guided One-Shot Talking Face Generation With a High-Resolution Audio-Visual Dataset. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 3661–3670.
  * Zheng et al. (2022) Yinglin Zheng, Hao Yang, Ting Zhang, Jianmin Bao, Dongdong Chen, Yangyu Huang, Lu Yuan, Dong Chen, Ming Zeng, and Fang Wen. 2022.  General facial representation learning in a visual-linguistic manner. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 18697–18709.
  * Zhou et al. (2019) Hang Zhou, Yu Liu, Ziwei Liu, Ping Luo, and Xiaogang Wang. 2019.  Talking face generation by adversarially disentangled audio-visual representation. In _Proceedings of the AAAI conference on artificial intelligence_ , Vol. 33. 9299–9306.
  * Zhou et al. (2020) Yang Zhou, Xintong Han, Eli Shechtman, Jose Echevarria, Evangelos Kalogerakis, and Dingzeyu Li. 2020.  Makelttalk: speaker-aware talking-head animation.  _ACM Transactions On Graphics (TOG)_ 39, 6 (2020), 1–15.
  * Zhou et al. (2024) Zhenglin Zhou, Fan Ma, Hehe Fan, and Yi Yang. 2024.  HeadStudio: Text to Animatable Head Avatars with 3D Gaussian Splatting.  _arXiv preprint arXiv:2402.06149_ (2024).
  * Zollhöfer et al. (2018) Michael Zollhöfer, Justus Thies, Pablo Garrido, Derek Bradley, Thabo Beeler, Patrick Pérez, Marc Stamminger, Matthias Nießner, and Christian Theobalt. 2018.  State of the art on monocular 3D face reconstruction, tracking, and applications. In _Computer graphics forum_ , Vol. 37. Wiley Online Library, 523–550.


# Towards efficient Capsule Networks

###### Abstract

From the moment Neural Networks dominated the scene for image processing, the
computational complexity needed to solve the targeted tasks skyrocketed:
against such an unsustainable trend, many strategies have been developed,
ambitiously targeting performance’s preservation. Promoting sparse topologies,
for example, allows the deployment of deep neural networks models on embedded,
resource-constrained devices. Recently, Capsule Networks were introduced to
enhance explainability of a model, where each capsule is an explicit
representation of an object or its parts. These models show promising results
on toy datasets, but their low scalability prevents deployment on more complex
tasks. In this work, we explore sparsity besides capsule representations to
improve their computational efficiency by reducing the number of capsules. We
show how pruning with Capsule Network achieves high generalization with less
memory requirements, computational effort, and inference and training time.

Index Terms—  capsule networks, routing, pruning

##  1 Introduction

00footnotetext: Accepted for publication at the IEEE International Conference
on Image Processing (IEEE ICIP 2022).

©2022 IEEE. Personal use of this material is permitted. Permission from IEEE
must be obtained for all other uses, in any current or future media, including
reprinting/republishing this material for advertising or promotional purposes,
creating new collective works, for resale or redistribution to servers or
lists, or reuse of any copyrighted component of this work in other works.

Capsule Networks (CapsNets) [1, 2] were proposed to overcome the shortcomings
of Convolutional Neural Networks (CNNs). For instance, CNNs need many training
images in different orientation, pose, size etc. to be robust to affine
transformations [1]. Moreover, CNNs employ max-pooling to both reduce the
dimensions of the input space and to route low-level features deeper in the
network. However, the spatial relationships between parts (e.g. the mouth or
the nose) of an object (e.g. a human face) within an image is lost because
max-pooling progressively drops the spatial information [1]. CapsNets build a
hierarchy of parts by means of an iterative routing mechanism [1]. This can be
seen as a parallel attention mechanism which models the connections between
capsules, retaining correct spatial relationships between objects.

Despite CapsNets are successfully applied to various tasks [3, 4], the
research is still in its early stages and CapsNets are far from replacing CNNs
as state-of-the-art neural neural architectures. In fact, their applicability
is limited because stacking multiple capsule layers increase the computational
effort [5, 6]. Applying CapsNets on high resolution data with complex
backgrounds is also critical: stacking many capsule layers or adding more
capsules can lead to instability during training because of the delicate
routing algorithm [7]. Finding the connections between many capsules is not
trivial and tuning the number of routing iterations during training can be
problematic [8].

Fig. 1: A standard CapsNet with many capsules (top). We exploit structured
pruned representations (bottom) to reduce the number of capsules.

However, the problem of over-parametrization and the difficulty of deploying
networks in scenarios with limited memory and resources is not peculiar only
to CapsNets, but also to CNNs. To this end, pruning methods (which reduce the
size and complexity of a neural network model) have gained more and more
attention lately [9]. Learning sparse topologies can effectively reduce the
network footprint and speedup execution with negligible performance loss [10].
So, why not exploiting these techniques also for CapsNets? The reduction their
complexity, especially in early layers, will enable their deployment on
complex datasets. The aim of this work is to exploit structured pruning
approaches to reduce the complexity of CapsNets without performance loss and
test them on data with non-straightforward background such as CIFAR10 and high
dimensional data such as Tiny ImageNet. Fig. 1 provides an overall idea of the
functioning of our approach. We show how employing structurally sparse
backbones, extracting features in early layers for CapsNets, affects the
number of capsules in the overall network. Our experiments show that reducing
the complexity of the backbone is an effective way to achieve high
generalization with less memory requirements, energy consumption, training and
inference time.

##  2 Capsule Networks

###  2.1 Background

CapsNets organize neurons into activity vectors called capsules. Each element
of these vectors accounts for the instantiation parameters of the object that
the capsule represents such as its pose and other properties like thickness,
color, deformation, etc. [1]. The probability that a specific object exists in
the image is simply the magnitude of its capsule vector [1]. CapsNets are
organized in layers and are typically composed by convolutional layers and two
or more capsule layers. The CapsNet architecture introduced in [1] is a
shallow network with two capsule layers. In this work, we replace both
traditional global average pooling layer and final fully connected layer with
these two capsule layers. The poses of the first capsule layer (PrimaryCaps)
$\boldsymbol{u}_{i}$, called primary capsules, are built upon convolutional
layers reshaped into the capsule space. An iterative routing-by-agreement
mechanism is performed to compute the poses of the capsules of the next layer
(OutputCaps). Each capsule $\boldsymbol{u_{i}}$ makes a prediction
$\hat{\boldsymbol{u}}_{{j|i}}$, thanks to a transformation matrix
$\boldsymbol{W}_{ij}$, for the pose of an upper layer output capsule $j$

$\hat{\boldsymbol{u}}_{{j|i}}=\boldsymbol{W}_{{ij}}\boldsymbol{u}_{i}.$ (1)

Then, the total input ${\boldsymbol{s}}_{j}$ of an output capsule $j$ is
computed as the weighted average of votes $\hat{{\boldsymbol{u}}}_{j|i}$

${\boldsymbol{s}}_{j}=\sum\limits_{i}c_{ij}\hat{{\boldsymbol{u}}}_{j|i},$ (2)

where $c_{ij}$ is the coupling coefficient between a primary capsule $i$ and
an output capsule $j$. The un-normalized pose $s_{j}$ of an output capsule $j$
is then “squashed” to compute the normalized pose ${\boldsymbol{v}}_{j}$

${\boldsymbol{v}}_{j}=\textrm{squash}({\boldsymbol{s}}_{j})=\frac{\|{\boldsymbol{s}}_{j}\|^{2}}{1+\|{\boldsymbol{s}}_{j}\|^{2}}\frac{{\boldsymbol{s}}_{j}}{\|{\boldsymbol{s}}_{j}\|}.$
(3)

The coupling coefficients are determined by a “routing softmax” activation
function, whose initial logits $b_{ij}$ are the log prior probabilities the
$i$-th capsule should be coupled to the $j$-th one. At each iteration, the
update rule for the logits is

$b_{ij}\leftarrow
b_{ij}+{\boldsymbol{v}}_{j}\cdot\hat{{\boldsymbol{u}}}_{j|i}.$ (4)

The steps defined in (2), (3), (4) are repeated for $r$ routing iterations.
Sabour  _et al._ introduced the margin loss to train CapsNets [1].

###  2.2 Related work

Sabour  _et al._ [1] proposed a shallow CapsNet to overcome some limitations
of CNNs. They achieved state-of-the-art results on toy datasets like MNIST and
smallNORB. Recent works focused on building deeper and more efficient
architectures of CapsNets in order to apply them on more complex data such as
CIFAR10. We can divide these attempts into three groups: i) pure CapsNets,
namely deep models where only capsule layers are stacked on top of each other
[2, 11, 12]; ii) CapsNets where the primary capsules are extracted with more
than one convolutional layer [7, 13]; iii) CapsNets where the primary capsules
are extracted with a pretrained state-of-the-art backbone [14, 15].

As regards pure capsule layers, Hinton  _et al._ [2] replace the dynamic
routing with Expectation-Maximization, adopting matrix capsules instead of
vector capsules in order to reduce the number of trainable parameters. They
also introduce the idea of convolutional capsules, where the routing is
performed locally between a subset of capsules defined by a receptive field.
Gugglberger  _et al._ [11] introduce residual connections between capsule
layers to train deeper capsule networks.

A simple but effective way to build deeper capsule models consists of
employing several pure convolutional layers before the last capsule layer.
Rajasegaran  _et al._ [7] propose a deep CapsNet composed by four layers. All
the layers have only one routing iteration (so the coupling coefficients are
uniform) except the last one. They approximate the first 3 layers with 2D
convolutional layers with skip connections so only one capsule layer exists in
their architecture. Mazzia  _et al._ [13] extract primary capsules by means of
a set of convolutional and Batch Normalization layers, replacing dynamic
routing with attention routing. They show that even with a very limited number
of parameters their network achieves state-of-the-art results on MNIST,
MultiMNIST and smallNORB.

Finally, instead of stacking pure convolutional layers, one can employ state-
of-the-arts backbones like ResNets as feature extraction networks.For example,
Hahn  _et al._ [14] incorporates a self-routing method such that capsule
models do not require agreements anymore. In all of their experiments, models
share the same base CNN architecture, namely a ResNet-20, requiring less
computation.

Our work relates to those models where primary capsules are extracted using
efficient state-of-the-art backbones like ResNets and MobileNets. We employ
structurally pruned backbones to scale CapsNets to bigger datasets and improve
their efficiency. We show how these pruned Capsule Networks achieve high
generalization with less memory requirements and computational effort.

##  3 Proposed method

Fig. 2: The architecture used in this work is composed by a backbone part and
a capsule part. Primary capsules space has 8 real dimensions while output
capsules are 16-dimensional vectors ($D_{1}=8$ and $D_{2}=16$).

###  3.1 Primary capsules extraction

Following the work of [14, 15], we extract $I$ primary capsule activity
vectors by means of a backbone network. With this approach, we can build deep
capsule networks in an efficient way since the input space of the capsule
network is significantly reduced. Fig. 2 shows the architecture used in this
work. We can build a CapsNet on top of the backbone by replacing the last two
layers by a primary capsule (PrimaryCaps) and fully-connected capsule
(OutputCaps) layers, respectively. Note that PrimaryCaps is a convolutional
capsule layer with $I$ channels of convolutional $D_{1}$ capsules, namely each
capsule contains $D_{1}$ convolutional units with a $K\times K\times S$
kernel. The kernel size used is wide as the spatial dimensions of the output
of the backbone network, which means the number of capsules is dynamically
reduced. In order to compute output capsules, we use the same routing-by-
agreement algorithm described in [1]. In our experiments we use backbones both
as fixed feature extractor, namely we freeze their weights except for the
capsule layer, and as trainable feature extractor, where we finetune all
layers.

###  3.2 Effect of pruned backbones to capsule layers

In order to reduce the computational complexity of CapsNets as in Fig. 2, one
possible strategy is to deploy pruned backbones. Let us define the backbone
network
$\mathcal{B}(\boldsymbol{x}_{\mathcal{M}},\boldsymbol{w}_{\mathcal{B}})$ where
$\boldsymbol{w}_{\mathcal{B}}$ are the backbone parameters and
$\boldsymbol{x}_{\mathcal{M}}$ is the input; it produces as output the tensor
$\boldsymbol{x}_{\mathcal{B}}\in\mathbb{R}^{K\times K\times S}$. Then, tensor
$\boldsymbol{x}_{\mathcal{B}}$ is fed into the CapsNet
$\mathcal{C}(\boldsymbol{x}_{\mathcal{B}},\boldsymbol{w}_{\mathcal{C}})$,
where $\boldsymbol{w}_{\mathcal{C}}$ are the capsule parameters. We can
concatenate the two parts expressing the end-to-end model $\mathcal{N}$ as

$\mathcal{N}(\boldsymbol{x}_{\mathcal{M}},\boldsymbol{w}_{\mathcal{B}},\boldsymbol{w}_{\mathcal{C}})=\mathcal{C}\left[\mathcal{B}(\boldsymbol{x}_{\mathcal{M}},\boldsymbol{w}_{\mathcal{B}}),\boldsymbol{w}_{\mathcal{C}})\right].$
(5)

As we saw in Sec. 3.1, $D_{1}$-dimensional primary capsules poses
$\boldsymbol{u}_{i}$ are built upon convolutional layers. We define the number
of primary capsules as

$I=\left\lfloor\frac{S}{D_{1}}\right\rfloor.$ (6)

From this, we observe that reducing the complexity of the backbone would
result in the overall reduction of the complexity for the entire model
$\mathcal{N}$, and towards this end pruning has already proved to be an
effective approach [9, 10].
Pruning approaches can be divided into two groups. Unstructured pruning
methods aim at minimizing the cardinality $\|\boldsymbol{w}\|_{0}$ of the
parameters in the model, regardless the output topology [16, 17, 18]. On the
other hand, structured approaches drop groups of weight connections entirely,
such as entire channels or filters, imposing a regular pruned topology [9,
19]. As an effect, they minimize the cardinality of some $i$-th intermediate
layer’s output $\|\boldsymbol{x}_{i}\|_{0}$. Bragagnolo  _et al._ [20] showed
that structured sparsity, despite removing significantly less parameters from
the model, yields lower model’s memory footprint and inference time. When
pruning a network in a structured way, a simplification step which practically
reduces the rank of the matrices is possible; on the other side, encoding
unstructured sparse matrices lead to representation overheads [10].
Considering the structure of the primary capsules layer, we are interested in
the reduction of the cardinality for the input of the primary capsules
$\boldsymbol{x}_{\mathcal{B}}$. Indeed, as we can see in (6), the minimization
of $\|\boldsymbol{x}_{\mathcal{B}}\|_{0}$ results in a proportional reduction
of $S$, which in turn reduces $I$, resulting in a reduction of the votes and
the complexity of the routing algorithm [8].

##  4 Experiments

Backbone pruned FLOPS (%) |  Bottleneck size |  Primary caps |  Total FLOPS (B) |  GPU memory consumption (GB) |  Training time (epoch, s) | Accuracy
---|---|---|---|---|---|---
0 | 2048 | 256 | 12.9 | 47.02 | 462 | 0.78
25 | 1970 | 246 | 9.8 | 43.50 | 450 | 0.80
50 | 1947 | 243 | 6.7 | 39.42 | 410 | 0.78
75 | 944 | 118 | 3.2 | 20.11 | 214 | 0.74
Table 1: Dynamic caps Performance per ResNet50 (32 caps, finetuned, Tiny
Imagenet).

We performed several experiments with different backbones on CIFAR10 and Tiny
ImageNet. We choose ResNets and MobileNets networks since they are used as
common baselines in traditional computer vision tasks. In fact, they offer a
good compromise between both efficiency and performance. Furthermore, they are
mainly composed of convolutional layers so it is easy to replace the first
convolutional layer in the original CapsNet introduced in [1] with these
models.

###  4.1 Experimental Setup

We resized CIFAR10 images to $64\times 64$ resolution so as to add a deeper
backbone network as in [7]. We used the 5% of the training set as validation
set for hyper-parameters tuning. We also run experiments on Tiny Imagenet
($224\times 224$ images, 200 classes). Here we used 10% of the training set as
validation set and the original validation set as test set. Our experiments
consider several flavors of the architecture in Fig. 2 with different number
of primary capsules $I$, backbones (ResNet-50 e MobileNetV1) and training
methods (whether the backbone is finetuned or not). We train all the networks
minimizing a margin loss [1] with the Adam optimizer and a batch size of 128.
The experiments were run on a Nvidia Ampere A40 equipped with 48GB RAM, and
the code uses PyTorch 1.10.111The source code is available at
https://github.com/EIDOSLAB/towards-efficient-capsule-networks.
We also used PyNVML library to compute the GPU memory consumption. As
pretrained pruned backbones, we have used both ResNet-50 and MobileNetV1
pruned with EagleEye [19]. EagleEye is a state-of-the-art, open-source,
structured pruning strategy which identifies relevant features at the level of
intermediate outputs $\boldsymbol{x}_{i}$.

###  4.2 Results and discussion

Fig. 3: Performances of ResNet-50 and MobileNetV1 for CIFAR10 (left) and Tiny
ImageNet (right) with several pruned FLOPS. configurations.

The main results are shown in Fig. 3. Each configuration differs from the
dataset used, percentage of pruned parameters in the backbone network (which
affects the number of primary capsules), and training method (backbone
finetuned or not). In Fig. 3 we report the accuracy as a function of the total
number of FLOPS (of both backbone and capsule part) for training the network.
We can see that when the backbone is freezed (in red) the networks achieve
poor performances, but as the number of pruned backbone FLOPS decreases,
namely the number of primary capsules is higher, the accuracy increases. The
poor performance can be explained by the fact that the output features
$\boldsymbol{x}_{\mathcal{B}}$ of the backbone are not directly optimized to
represent objects poses. Therefore, they can not be used directly as input to
capsule layers. To overcome such a drawback, we can either add more primary
capsules or train the whole network. In fact, when the backbone is finetuned
(in blue) the performance is higher since the backbone is able to map more
suitable features to the capsules space. We can see in this case that adding
more capsules does not always lead to an improvement in the classification
accuracy. In Tab. 1 the performance for Tiny ImageNet with a pretrained
ResNet-50 backbone is reported. Employing a backbone with 50% of pruned FLOPS
yields similar accuracy as with a full backbone, with less GPU memory
consumption and training time. With 25% of pruned FLOPS we can even improve
the accuracy compared to the unpruned one. The partial improving in the
performance in a low pruning regime, despite being at a first glance
surprising, is not new to the literature and it is twofold. First, Rajasegaran
_et al._ [7] shows that building too many capsules inhibites learning because
the coupling coefficients are too small, preventing the gradient flow.
Furthermore, Han  _et al._ observed such a phenomenon for unstructured pruning
[16]. Recently, it has been showed that the average entropy in the bottleneck
layer for pruned backbones (in our case, $\boldsymbol{x}_{\mathcal{B}}$) is
higher than in non-pruned ones [21]: this results in the propagation of less
specific and more general information, which prevents features overfit, on top
of which capsules layers can extract much more accurate information.

##  5 Conclusion

In this work we improved the scalability and reduced the computational effort
of CapsNets on complex datasets deploying backbones with structured sparsity.
CapsNets with many capsules are difficult to train: in such a scenario the
routing algorithm, a key mechanism for CapsNets, struggles to find the
necessary agreements between capsules. Therefore, the employment of pruned
backbones (so fewer capsules) for high resolution datasets leads to
competitive results with no performance loss. We also showed how extracting
features in early layers with sparse networks improves efficiency, memory
consumption, inference and training time of the overall CapsNet. Therefore,
with this work, we open the way to apply CapsNets on resource constrained
devices. Future works involve specific methods concerning pruning capsule
layers in a structured way.

## References

  * [1] Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton,  “Dynamic routing between capsules,”  in Advances in Neural Information Processing Systems. 2017, vol. 30, pp. 3856–3866, Curran Associates, Inc.
  * [2] Geoffrey E Hinton, Sara Sabour, and Nicholas Frosst,  “Matrix capsules with EM routing,”  in International Conference on Learning Representations, 2018.
  * [3] Yongheng Zhao, Tolga Birdal, Haowen Deng, and Federico Tombari,  “3d point capsule networks,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 1009–1018.
  * [4] Parnian Afshar, Arash Mohammadi, and Konstantinos N Plataniotis,  “Brain tumor type classification via capsule networks,”  in 2018 25th IEEE International Conference on Image Processing (ICIP). IEEE, 2018, pp. 3129–3133.
  * [5] Chandan Kumar Singh, Vivek Kumar Gangwar, Anima Majumder, Swagat Kumar, Prakash Chanderlal Ambwani, and Rajesh Sinha,  “A light-weight deep feature based capsule network,”  in 2020 International Joint Conference on Neural Networks (IJCNN), 2020, pp. 1–8.
  * [6] Alberto Marchisio, Beatrice Bussolino, Alessio Colucci, Maurizio Martina, Guido Masera, and Muhammad Shafique,  “Q-capsnets: A specialized framework for quantizing capsule networks,”  in Proceedings of the 57th ACM/EDAC/IEEE Design Automation Conference. 2020, DAC ’20, IEEE Press.
  * [7] J. Rajasegaran, V. Jayasundara, S. Jayasekara, H. Jayasekara, S. Seneviratne, and R. Rodrigo,  “Deepcaps: Going deeper with capsule networks,”  in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 10717–10725.
  * [8] Riccardo Renzulli, Enzo Tartaglione, Attilio Fiandrotti, and Marco Grangetto,  “Capsule networks with routing annealing,”  in Artificial Neural Networks and Machine Learning – ICANN 2021, Cham, 2021, pp. 529–540, Springer International Publishing.
  * [9] Enzo Tartaglione, Andrea Bragagnolo, Francesco Odierna, Attilio Fiandrotti, and Marco Grangetto,  “Serene: Sensitivity-based regularization of neurons for structured sparsity in neural networks,”  IEEE Transactions on Neural Networks and Learning Systems, pp. 1–14, 2021.
  * [10] Andrea Bragagnolo and Carlo Alberto Barbano,  “Simplify: A python library for optimizing pruned neural networks,”  SoftwareX, vol. 17, pp. 100907, 2022.
  * [11] Josef Gugglberger, David Peer, and Antonio Rodríguez-Sánchez,  “Training deep capsule networks with residual connections,”  in Artificial Neural Networks and Machine Learning – ICANN 2021, Cham, 2021, pp. 541–552, Springer International Publishing.
  * [12] Yi Xiong, Guiping Su, Shiwe Ye, Yuan Sun, and Yi Sun,  “Deeper capsule network for complex data,”  in 2019 International Joint Conference on Neural Networks (IJCNN), 2019, pp. 1–8.
  * [13] Vittorio Mazzia, Francesco Salvetti, and Marcello Chiaberge,  “Efficient-capsnet: capsule network with self-attention routing,”  Scientific reports, vol. 11, 2021.
  * [14] Taeyoung Hahn, Myeongjang Pyeon, and Gunhee Kim,  “Self-routing capsule networks,”  in Advances in Neural Information Processing Systems. 2019, vol. 32, pp. 7658–7667, Curran Associates, Inc.
  * [15] Inyoung Paik, Taeyeong Kwak, and Injung Kim,  “Capsule networks need an improved routing algorithm,”  in Proceedings of The Eleventh Asian Conference on Machine Learning, Nagoya, Japan, 17–19 Nov 2019, vol. 101 of Proceedings of Machine Learning Research, pp. 489–502, PMLR.
  * [16] Song Han, Jeff Pool, John Tran, and William Dally,  “Learning both weights and connections for efficient neural network,”  in Advances in neural information processing systems, 2015, pp. 1135–1143.
  * [17] Enzo Tartaglione, Skjalg Lepsøy, Attilio Fiandrotti, and Gianluca Francini,  “Learning sparse neural networks via sensitivity-driven regularization,”  in Advances in neural information processing systems, 2018, pp. 3878–3888.
  * [18] Enzo Tartaglione, Andrea Bragagnolo, Attilio Fiandrotti, and Marco Grangetto,  “Loss-based sensitivity regularization: Towards deep sparse neural networks,”  Neural Networks, vol. 146, pp. 230–237, 2022.
  * [19] Bailin Li, Bowen Wu, Jiang Su, Guangrun Wang, and Liang Lin,  “Eagleeye: Fast sub-net evaluation for efficient neural network pruning,” 2020.
  * [20] Andrea Bragagnolo, Enzo Tartaglione, Attilio Fiandrotti, and Marco Grangetto,  “On the role of structured pruning for neural network compression,”  in 2021 IEEE International Conference on Image Processing (ICIP), 2021, pp. 3527–3531.
  * [21] Enzo Tartaglione, Andrea Bragagnolo, and Marco Grangetto,  “Pruning artificial neural networks: A way to find well-generalizing, high-entropy sharp minima,”  in Artificial Neural Networks and Machine Learning – ICANN 2020, Cham, 2020, pp. 67–78, Springer International Publishing.


Figure 39: Hubble WFC3 images of two Class II objects appearing incidentally
in WFC3 observations of HOPS sources.

##  Appendix F The Effect of the Adopted Dust Law on Cavity Morphology

In our modeling, we used the dust opacity models adopted by (Furlan et al.,
2016). To investigate the role of the assumed dust law on the observed
morphology, we compared images generated with two dust opacity models from
Ormel et al. (2011). The opacity model used in this paper, "icsgra3," is
described in Section 3 and adopts a grain coagulation time of 0.3 Myr. We
compare this to the opacity model "icsgra2" which in contrasts adopts a time
of 0.1 Myr with consequently more grains of smaller sizes. When comparing
model protostars from our grid generated with otherwise identical parameters,
we observe that those using "icsgra2" show strongly limb brightened cavity
edge profiles and bright point sources (Figure 40). In contrast, the larger
grains present in "icsgra3" result in more forward scattering where the
intensity peaks toward the center of the cavity instead of the edges. The
"icsgra3" are more consistent with HST observations which typically show the
cavities filled with emission (Figure 5), although there are some examples
that show enhanced edges (Figure 13). This suggests that the larger grains in
"icsgra3" are more representative of our sample (Appendix A). Although beyond
the scope of this investigation, future studies of the observed cavity
morphologies may provide new constraints on grain properties and their
variations.

Figure 40:  Two examples of a radiative transfer models created from different
dust opacity models and otherwise identical parameters. On the left, the
"icgras3" dust law used in this paper displays more forward scattering and
less limb brightening along the cavity edge.

##  Appendix G Comparison Between SED Modeling and Near-IR Morphologies

Figure 41: These four panels contain histograms of inclinations from the SED
fitting of Furlan et al. (2016), for a selection of four scattered-light
morphologies. In each histogram, the color indicates the classification scheme
used by Furlan et al. (2016)

Most of the protostars in this paper have been characterized in detail by
Furlan et al. (2016) using modeling in concert with the SEDs of these objects.
The models used by these authors differ from those discussed in Section 3, and
are fit to the SED from $1.6870\text{\,}\mathrm{\SIUnitSymbolMicro m}$. The
primarily differences that affect the $1.60\text{\,}\mathrm{\SIUnitSymbolMicro
m}$ emission are that our grid uses a finer sampling of high inclination
protostars, a sparser grid of envelope densities, and a different cavity
opening angle exponent (2 vs 1.5).

Figure 41 shows histograms of the number of protostars vs inclinations
determined by SED fitting. Four different histograms are displayed, one each
for bipolar, unipolar, point sources, and non-detections. (Irregular
protostars are not shown). Since the bins are chosen to have equal intervals
in the cosine of the inclination, a random distribution should result in an
equal number of sources in each bin; however, the SED-determined inclination
for the overall sample of HOPS protostars peak at
$6070\text{\,}\mathrm{\SIUnitSymbolDegree}$, suggesting that the SED derived
inclinations may have systematic biases (Furlan et al., 2016). The
distribution of inclinations for point sources is similar in this respect to
the overall sample (see Furlan et al., 2016, Figure 29), except at the highest
inclinations. Obscuration by the disk likely accounts for this deficiency,
both by decreasing the ability to detect a point source and by decreasing the
contrast with the surrounding nebulosity. The unipolar and non-detections also
peak around $6070\text{\,}\mathrm{\SIUnitSymbolDegree}$, although they show a
deficiency of low inclination objects, this is expected since the outflow
cavities cannot be detected at low inclinations the lower obscuration at these
angles makes non-detections unlikely.

Finally, the bipolar protostars show a broad range of inclinations, even
though their observed morphologies require a nearly edge-on perspective.
Furlan et al. (2016) showed that there are large systematic uncertainties in
the SED derived inclinations. This figure further demonstrates the limitations
of using SED derived inclinations, particularly for edge-on protostars. In a
detailed study of the HST morphology of the HOPS 136 protostar, Fischer et al.
(2014) could only find agreement between the SED models and the edge-on
morphology by adding a low density component of dust in the outflow cavity to
increase the scattering at shorter wavelengths. This suggests that our models
are incomplete and therefore under-predict the brightness of protostars in the
near-IR; consequently, model fits erroneously favor inclined models where the
near-IR emission is less absorbed by the disk.

Our measurements of the cavity half-opening angles can also be used to test
the angles derived from SED models. Furlan et al. (2016) fitted models with
discrete cavity half-opening angles of
$515253545\text{\,}\mathrm{\SIUnitSymbolDegree}$ for cavities with a $r^{1.5}$
power-law shape.131313Note that they use the terminology “cavity opening
angle” where we use “cavity half-opening angle”. The consistency of the best
fit cavity angle compared to the mode for various criteria of close models are
shown in Figure 46 of that work; This is further demonstrated in Figure 42,
where we compare half-opening angles inferred from the best SED fits from
Furlan et al. (2016) to the cavity half-opening angles derived in this work.
In spite of the different cavity shapes assumed, a monotonically increasing
correlation should be seen if both are accurate measurements of the cavity
size. Since we do not see the expected correlation, and since most protostars
have decent quality model fits for a range of cavity angles in the SED model
grid, we find the cavity opening angles are not well constrained by SED
modeling. A similar result was found by Seale & Looney (2008).

Figure 42: Half-opening angles compared between this work and those in Furlan
et al. (2016). The blue dotted line indicates equality in the two methods.