(eccv) Package eccv Warning: Package ‘hyperref’ is loaded with option
‘pagebackref’, which is *not* recommended for camera-ready version

$\star$$\star$footnotetext: The initial part of this work was done at
AITRICS.11institutetext: DeepBrain AI Inc., South Korea
22institutetext: Graduate School of AI, KAIST, South Korea
22email<EMAIL_ADDRESS>22email<EMAIL_ADDRESS>22email:
<EMAIL_ADDRESS>
https://export3d.github.io

# Learning to Generate Conditional Tri-plane for 3D-aware Expression
Controllable Portrait Animation

Taekyung Ki⋆ 11 Dongchan Min 22 Gyeongsu Chae 11

###### Abstract

In this paper, we present Export3D, a one-shot 3D-aware portrait animation
method that is able to control the facial expression and camera view of a
given portrait image. To achieve this, we introduce a tri-plane generator that
directly generates a tri-plane of 3D prior by transferring the expression
parameter of 3DMM into the source image. The tri-plane is then decoded into
the image of different view through a differentiable volume rendering.
Existing portrait animation methods heavily rely on image warping to transfer
the expression in the motion space, challenging on disentanglement of
appearance and expression. In contrast, we propose a contrastive pre-training
framework for appearance-free expression parameter, eliminating undesirable
appearance swap when transferring a cross-identity expression. Extensive
experiments show that our pre-training framework can learn the appearance-free
expression representation hidden in 3DMM, and our model can generate 3D-aware
expression controllable portrait image without appearance swap in the cross-
identity manner.

###### Keywords:

Portrait Image Animation Facial Expression Control 3D-aware Synthesis

##  1 Introduction

Portrait image animation aims to generate a video of a given source identity
with the driving motion. It has received a lot of attention due to the
potential of virtual human services, such as cross-lingual film dubbing [31,
13], virtual avatar chatting [41, 69], and video conferencing [57, 56]. In
these scenarios, it is essential to transfer the facial expression (e.g., eye-
blinking, lip motion, etc.) from different person, i.e., cross-identity
transfer, while preserving the source identity. However, it is challenging due
to the ambiguity between appearance and expression [18] and the lack of paired
data (e.g., different faces with the same expression) for disentanglement
representation learning [42].

Most 2D-based methods rely on image warping [52, 70, 64, 58, 24], which warps
the source image to the driving image by estimating the motion between them.
To impose a bottleneck for the motion representation, they encode the motion
into the difference between sparse key-points [52, 70, 24] or latent codes
[58], which are trained in an unsupervised manner. However, in this scenario,
the facial expressions are encoded into the motion space as well, in terms of
local motion, which tends to be neglected due to the relatively large head
motions. Furthermore, since the facial expression and the appearance are
highly entangled in the image space, cross-identity expression transfer often
involves the source appearance change. DPE [42] tackles this entanglement
issue by proposing a self-supervised disentanglement learning framework based
on cycle-consistency learning [71]. However, it shows temporal inconsistency
in the generated video due to its instability of cycle-consistency learning.

Another line of works [34, 38, 66, 35] explores facial expression control in
3D space using the neural radiance fields (NeRFs) [40]. They leverage pre-
trained latent representation of 3D GAN [10] for 3D facial prior where they
design the expression in terms of latent code [38, 66] or predict deformation
field [43] to deform the well-constructed 3D representation, such as tri-plane
[34, 38, 66, 35]. However, the latent code cannot faithfully reconstruct the
source identity [38], and the point-wise deformation fields to those 3D
representations yield video-level artifacts, such as flickers [34].

In this paper, we address the appearance-expression entanglement issue by
proposing a contrastive pre-training framework over video datasets that
produces appearance-free facial expressions with orthogonal structure. Armed
with this representation, we build a one-shot 3D-aware portrait image
animation method, namely Export3D, which controls the facial expression and 3D
camera view of a given source image without appearance swap. To achieve this,
we design a generator architecture consisting of vision transformer (ViT) and
convolution layers [56, 17, 44] that directly generates the tri-planes from
the source image and driving expression parameters instead of predicting the
deformation fields. The main contributions of this work are summarized as
follows:

  * •

We present Export3D, a one-shot 3D-aware portrait image animation method that
can explicitly control the facial expression and camera view of the source
image only using the expression and camera parameters.

  * •

We propose a contrastive pre-training framework for the appearance-free facial
expression latents distilled from the 3DMM parameters where they form an
orthogonal structure for different facial expressions.

  * •

Extensive experiments demonstrate that our pre-training framework can learn
the appearance-free expression, which enables our method to transfer the
cross-identity expression without undesirable appearance swap.

##  2 Related Works

###  2.1 3D-aware Image Synthesis

3D-aware image synthesis aims to generate images with explicit camera pose
control [50, 10, 21, 15, 11, 60]. This is achieved by conditioning the camera
pose parameter into generative features, which are then rendered into an RGB
image through differentiable volume rendering [36, 40, 43, 6]. This rendering
technique has integrated with adversarial learning [20, 50, 10, 21, 15, 11,
60] to learn 3D view consistency from the unposed dataset. GRAM [15] generates
a multi-view consistent image by learning the radiance field on a set of 2D
surface manifolds. AniFaceGAN [59] further learns the deformation fields [43]
for the facial expression on these manifolds [15] for explicit facial
expression control. EG3D [10] introduces a tri-plane representation that
provides a strong 3D position encoding with neural volume rendering and become
the one of the most prominent representation in this field. However, these
methods generate portrait images from noise, requiring further process for
real image manipulation.

Relying on the generateive power of EG3D, several works [62, 65, 54, 38, 33,
7, 67, 56, 66] extend 2D GAN-inversion [1, 2, 55, 48] methods, which is
challenging due to the multi-view consistency for a single-view image.
Specifically, based on facial symmetry, SPI [65] utilizes horizontally flipped
images for pseudo supervision to the occluded facial region. However, it
requires multi-stage latent code optimizations. GOAE [67] proposes an encoder-
based inversion for EG3D which enhances multi-view consistency via an
occlusion-aware tri-plane mixing module. Live3DPortrait [56] can reconstruct
multi-view consistent portrait images by leveraging the synthetic data of pre-
trained EG3D to provide multi-view supervision. However, these methods cannot
explicitly manipulate the expression of the source image.

We propose a tri-plane generator architecture that can generate the tri-plane
of a given source image with explicit expression control. Inspired by [56,
44], we design this generator with ViT and convolution layers [17], and
directly inject expression parameters into the tri-plane generating process
through the expression adaptive layer normalization (EAdaLN). By leveraging
the strong power of NeRF [40, 10, 56, 66, 54], we decode the generated tri-
plane into multi-view images with explicit expression manipulation.

###  2.2 Portrait Image Animation

Portrait image animation, or face reenactment, is a task that animates a given
source image according to the input driving condition, either audio [45, 31,
41, 69, 37, 22] or image [52, 70, 57, 58, 64]. Specifically, image-driven
methods transfer the motion of the driving image into the source image by
learning the motion between them. Most works [52, 70, 57] use facial key-
points as a pivot representation to be aware of motion via the key-point
displacement. FOMM [52] estimates facial key-points in an unsupervised manner,
approximating the motion through the first-order Taylor expansion. LIA [58]
encodes a motion in terms of latent codes by introducing an orthonormal basis
as a motion dictionary. However, the local motion (e.g., facial expression)
and the global motion (e.g. head pose) are still entangled in those
representations. DPE [42] proposes a bidirectional cyclic training strategy to
decouple the pose and expression within the latent codes, while it produces
video-level artifacts due to the instability of the cycle-consistency
learning.

To explicitly control the facial expression, several works leverage the
expression parameters of 3D morphable models (3DMM) [8] in 2D [19, 64] or 3D
spaces [38, 35, 34]. StyleHEAT [64] uses 3DMM to warp 2D spatial features of
pre-trained StyleGAN2 [28] while yielding texture sticking. OTAvatar [38]
proposes a one-shot test-time optimization method that optimizes identity
codes of a single source image and learns expression-aware motion latent codes
in the latent space of pre-trained EG3D. HiDe-NeRF [34] and NOFA [66] take a
different way by predicting an expression-aware deformation field [43] that
deforms the tri-plane [10] reconstructed from the source image.

Our method belongs to image-driven approaches, distinguishing itself by not
depending on 2D image warping or 3D deformation fields. Toward this, we
propose the generator architecture that uses a source image and driving
expression parameters to produce an expression-transferred tri-plane, wherein
the expression parameters directly modulate the source visual features through
the expression adaptive layer normalization (EAdaLN). Furthermore, we mitigate
the appearance swap issue inherent in transferring other person’s expression
by introducing a contrastive pre-training method to obtain appearance-free
expression representations.

Figure 1: Training overview of Export3D. We convert a source image
$S\in\mathbb{R}^{3\times H\times W}$ into a tri-plane $T_{\beta_{D}}(S)$ for
rich 3D priors, conditioned on an expression parameter
$\beta_{D}\in\mathbb{R}^{64}$ from a driving image $D$. A differentiable
volume rendering renders the tri-plane into a raw rendered image
$\hat{D}_{raw}$ using the camera parameter $p_{D}\in\mathbb{R}^{25}$ of $D$,
which is then super-resolved into a final image $\hat{D}$

##  3 Methods

First of all, we formulate our portrait animation method, Export3D. Given a
source image $S\in\mathbb{R}^{3\times H\times W}$, our method transfers the
facial expression and camera view of a driving image $D\in\mathbb{R}^{3\times
H\times W}$ with the expression and camera parameters, respectively. Our
method uses a tri-plane [10] as the intermediate feature representation,
providing a 3D position information for differentiable volume rendering [40,
36]. We directly generate an expression-transferred tri-plane from the source
image and the driving expression parameter [8, 3] through expression adaptive
layer normalization (EAdaLN) (Sec. 3.2). Based on the observation that the
expression parameter still contains the appearance information, we propose a
pre-training framework using contrastive learning to obtain the appearance-
free expression latents, which forms an orthogonal structure for different
expressions (Sec. 3.1). The expression-transferred tri-plane is rendered into
a 3D-aware image through the differentiable volume rendering, and then super-
resolved into the final output (Sec. 3.3).

###  3.1 Contrastive Learned Basis Scaling (CLeBS)

Figure 2: Contrastive pre-training framework for LeBS. We sample the positive
and the negative samples from the same video source so that those samples
share the same appearance. Using contrastive learning, the encoder
$f_{e}(\cdot)$ learns an appearance-free representations.

Natural speaking style comes from the the non-verbal component, such as eye
blinking. To explicitly control the expression of the generated face, we
utilize the expression parameter $\beta\in\mathbb{R}^{64}$ from the widely
used 3D morphable models (3DMM) [8] in 3D face reconstruction. However, simply
using those parameters for transferring the other person’s expression fails to
preserve the facial identity of the source face.

Disentangling Expression and Appearance. In 3DMM-based face reconstruction,
the identity-appearance has been rarely explored. However, [18] shows that a
3D face shape can be reconstructed only using the expression parameters not
using the shape parameters, or vice versa. We also observe that the expression
parameter of 3DMM is highly entangled with the appearance (Fig. 7(a)),
resulting in an undesirable appearance swap when transferring the cross-
identity expressions. We assume that the expression parameter needs to be
refined to represent pure facial expressions. To address this issue, we
propose a contrastive learning based pre-training framework [12, 23, 46, 41]
on video dataset to discard the appearance information hidden in the
expression parameter. Specifically, given a video sequence
$\\{X_{i}\\}_{i=1}^{N}$ and its corresponding expression sequence
$\\{\beta_{i}\\}_{i=1}^{N}$, we sample an aligned image-expression pair
$(X_{k},\beta_{k})$ for the positive and the non-aligned pairs for the
negatives as illustrated in Fig. 2. The images and the expressions are mapped
into $d$-dimensional representations, and the distance between the positive
(or negative) representation pairs is minimized (or maximized) via the
following contrastive objective $\mathcal{L}_{cl}$:

$\mathcal{L}_{cl}=-\log\left(\frac{\exp(\cos(f_{I}(X_{k}),f_{e}(\beta_{k}))/\tau)}{\sum_{j\neq
k}\exp(\cos(f_{I}(X_{j}),f_{e}(\beta_{k}))/\tau)}\right),$ (1)

where $f_{I}(\cdot)$ is an image encoder, $f_{e}(\cdot)$ is an expression
encoder, $\tau$ is the temperature, and $\cos(\cdot,\cdot)$ is the cosine
similarity, respectively. Since all samples are from the same video, they
share the same appearance, thereby the objective (1) enforces the encoders to
learn appearance-free expression.

Moreover, for designing the expression encoder $f_{e}(\cdot)$, we focus on the
orthogonal structure of 3DMM [8] that controls different expressions along
different orthogonal directions. To provide the appearance-free expression
with the orthogonal structure [58], we introduce a learned orthonormal basis
$V$:

$V=\\{\textbf{v}_{1},\textbf{v}_{2},\cdots,\textbf{v}_{n}\\}\subseteq{\mathbb{R}^{d}}~{}~{}~{}\text{and}~{}~{}~{}\langle\textbf{v}_{i},\textbf{v}_{j}\rangle=\delta_{ij},~{}\forall
i,j,$ (2)

spanning our new expression sub-space ($\delta_{ij}$ is the Kroneker delta
function). More precisely, we convert the expression $\beta\in\mathbb{R}^{64}$
into the low-dimensional coefficient
$\lambda=(\lambda_{1},\lambda_{2},\cdots,\lambda_{n})\in\mathbb{R}^{n}~{}(n\ll
64)$ and then scales the learned orthonormal basis $V\in\mathbb{R}^{d}$ to
produce the appearance-free expression representation
$\beta^{\prime}\in\mathbb{R}^{d}$:

$\beta^{\prime}=f_{e}(\beta)=\lambda_{1}\textbf{v}_{1}+\lambda_{2}\textbf{v}_{2}+\cdots+\lambda_{n}\textbf{v}_{n}\in\mathbb{R}^{d}.$
(3)

In this space, an expression is a linear combination of the basis
$V=\\{\textbf{v}_{i}\\}_{i=1}^{n}$ where the coefficient
$\lambda=(\lambda_{1},\lambda_{2},\cdots,\lambda_{n})$ is responsible for the
intensity of each expression direction. We call our encoder $f_{e}(\cdot)$ a
learned basis scaling (LeBS) module with contrastive pre-training (CLeBS).
Note that once CLeBS is pre-trained with the objective (1), no further
training is required as illustrated in Fig. 1 and Fig. 3.

###  3.2 Hybrid Tri-plane Generator

We employ the tri-plane as the intermediate feature representation for 3D
prior to volume rendering. Tri-plane T consists of features assigned on the 3
axis-aligned planes (i.e., $xy,yz,zx$ planes):

$\text{T}=(\text{T}_{xy},\text{T}_{yz},\text{T}_{zx})\in\mathbb{R}^{3\times
32\times\frac{H}{2}\times\frac{W}{2}},$ (4)

where $\text{T}_{ij}\in\mathbb{R}^{32\times\frac{H}{2}\times\frac{W}{2}}$ is
the 32-dimensional feature of $\frac{H}{2}\times\frac{W}{2}$ resolution on the
$ij$-plane. EG3D [10] utilizes StyleGAN2 [28] to generate the tri-plane from a
noise, forming the style latent space $\mathcal{W}\subseteq\mathbb{R}^{512}$.
Several works [62, 65, 54, 38, 33, 7, 66] extend the 2D GAN-inversion methods
[1, 2, 48, 55, 49] to 3D GAN-inversion in terms of reconstructing the tri-
plane from style latent code. However, these methods often face challenges in
preserving facial identity since the style latent code lacks the capacity for
encoding spatial information and person-specific visual details.

We directly generate the expression-transferred tri-plane
$\text{T}_{\beta_{D}}(S)$ from the source $S$ and the driving expression
$\beta_{D}\in\mathbb{R}^{64}$ to reconstruct the driving $D$. Inspired by
Live3DPortrait [56], we construct the tri-plane generator with ViT and
convolution [17, 61]. Specifically, we convert $S\in\mathbb{R}^{3\times
H\times W}$ into a visual feature in
$\mathbb{R}^{\frac{h}{4}\times\frac{H}{2^{3}}\times\frac{W}{2^{3}}}$ through a
stack of convolutional blocks, and then merge it into the $h$-dimensional
$\frac{H\cdot W}{2^{8}}$ visual tokens through a overlap patch merge operator
[61]. These tokens and driving expression are processed through a conditional
ViT [17, 44, 61] blocks, namely EAdaLN-ViT, where the expression modulates
[44] the visual tokens through expression adaptive layer normalization
(EAdaLN) as illustrated in Fig. 3. EAdaLN is applied right before the multi-
head self-attention (MSA) and the mix feed-forward network (Mix-FFN)[61] of
each ViT block to inject the semantic expression into the visual tokens:

$\text{EAdaLN}(x,\beta^{\prime}_{D})=\sigma(\beta^{\prime}_{D})\times\text{LN}(x)+\mu(\beta^{\prime}_{D})\in\mathbb{R}^{h\times\left(\frac{H\cdot
W}{2^{8}}\right)},$ (5)

where $x$ is the input visual token, $\text{LN}(\cdot)$ is the layer
normalization [5], $\sigma(\beta^{\prime}_{D})$ and $\mu(\beta^{\prime}_{D})$
are the $h$-dimensional scale and shift factors computed from
$\beta^{\prime}_{D}=f_{e}(\beta_{D})\in\mathbb{R}^{d}$, respectively.

Figure 3: Hybrid tri-plane generator $\mathbf{G}$ and Expression Adaptive
Layer Normalization (EAdaLN).

To efficiently propagate the visual tokens to the higher resolution, we up-
sample the visual tokens with pixel shuffle [56] followed by the Gaussian low-
pass filter [27]. We experimentally find that the tokens and the pixel shuffle
produce grid artifacts, challenging to eliminate in the image space. Employing
low-pass filters effectively mitigates these artifacts by smoothing the
borderline artifacts over the coordinate. Lastly, we use ViT and convolutional
blocks to output the tri-plane $\text{T}_{\beta_{D}}(S)$:

$\text{T}_{\beta_{D}}(S)={\mathbf{G}}(S,\beta_{D})\in\mathbb{R}^{3\times
32\times\frac{H}{2}\times\frac{W}{2}}.$ (6)

Note that our method does not query the expression parameter to estimate the
motion [42, 64, 19], rather it is used as the multi-dimensional label. To
stabilize the tri-plane generation, we incorporate the online exponential
moving average (EMA) over tri-plane $\text{T}_{EMA}$ which is added to the
generated tri-plane. Please refer to supplementary materials for detailed
architectures.

###  3.3 Volume Rendering and Super-resolution

The tri-plane can be rendered into a 2D RGB image through the differentiable
volume rendering [40, 36, 10]. The expression-transferred tri-plane
$\text{T}_{\beta}(S)$ is projected onto 3 orthogonal planes
($xy,yz,zx$-planes) and then aggregated through average [10]:

$F_{\beta}(S)=\frac{1}{3}(F_{\beta,xy}(S)+F_{\beta,yz}(S)+F_{\beta,zx}(S)),$
(7)

where $F_{\beta,ij}(S)$ are the projected features of $T_{\beta}(S)$ onto the
$ij$ planes. A Light-weight MLP assigns a color $c$ and density
$\mathbf{\sigma}$ to each point $(x,y,z)$ using the aggregated feature
$F_{\beta}(S)$:

$\text{MLP}:F_{\beta}(S)\longrightarrow(c,\mathbf{\sigma}).$ (8)

The differentiable volume rendering [40, 10, 22] composites each color $c$ and
density $\mathbf{\sigma}$ into a RGB value $\mathcal{C}$ along the camera ray
$\mathbf{r}$:

$\mathcal{C}=\int_{t_{n}}^{t_{f}}\mathbf{\sigma}(\mathbf{r}(t))\cdot
c(\mathbf{r}(t))\cdot T(t)dt,$ (9)

where $\mathbf{r}(t)=\mathbf{o}+t\mathbf{d}$, $t\in[t_{n},t_{f}]$, with camera
center $\mathbf{o}\in\mathbb{R}^{3}$, viewing direction
$\mathbf{d}\in\mathbb{R}^{3}$, and $T(t)$ is the accumulation measure along
the ray $\mathbf{r}$ from $t_{n}$ to $t$:

$T(t)=\exp{\left(-\int_{t_{n}}^{t}\mathbf{\sigma}(\mathbf{r}(s))ds\right)}.$
(10)

Note that the ray $\mathbf{r}$ is determined by the driving camera parameter
$p_{D}\in\mathbb{R}^{25}$ to render the generated tri-plane
$\text{T}_{\beta_{D}}(S)$ into a image of the same view with $D$. The
appearance and expression are determined within the tri-plane generation
process, thereby the volume rendering applies the pose the to produce 2D
images of different views.

Directly rendering a target high-resolution image requires high computational
cost. One promising approach to address this issue is to incorporate super-
resolution blocks [10, 60, 56] that up-sample the rendered image of low
resolution. Following this approach, we first render a
$\hat{D}_{raw}\in\mathbb{R}^{3\times\frac{H}{4}\times\frac{W}{4}}$ and then
apply the super-resolution to obtain the target resolution
$\hat{D}\in\mathbb{R}^{3\times H\times W}$, as illustrated in Fig. 1. Instead
of using style-modulated convolution [10, 56], we use plane convolutional
blocks for super-resolution, as we do not leverage the style latent code.
Detailed architecture is provided in supplementary materials.

###  3.4 Training Objectives

Our model is trained with reconstruction manner that reconstruct a driving
frame $D$ from a source frame $S$ with the driving expression parameters
$\beta_{D}$ and camera parameters $p_{D}$ where these frames are randomly
sampled from the same video clip. The training consists of two stages. In the
first phase, we employ MSE loss $\mathcal{L}_{2}$ and VGG16 [53] multi-scale
perceptual loss $\mathcal{L}_{lpips}$ [68] to minimize the perceptual distance
between the generated frame $\hat{D}$ and the driving frame $D$. We also
minimize the distance between the raw rendered image $\hat{D}_{raw}$ and raw
driving image $D_{raw}$ using the same loss functions, denoted by
$\mathcal{L}^{raw}_{2}$ and $\mathcal{L}^{raw}_{lpips}$, respectively:

$\mathcal{L}_{rec}=\mathcal{L}^{raw}_{2}+\mathcal{L}_{2}+\mathcal{L}^{raw}_{lpips}+\mathcal{L}_{lpips}.$
(11)

In the second phase, we integrate the conditional discriminator used in [26],
using the camera parameter as additional condition and employing binary cross-
entropy loss to compute adversarial loss $\mathcal{L}_{adv}$. The total loss
function $\mathcal{L}_{total}$ is

$\mathcal{L}_{total}=\lambda_{rec}\mathcal{L}_{rec}+\lambda_{adv}\mathcal{L}_{adv},$
(12)

where $\lambda_{rec}$ and $\lambda_{adv}$ are balancing coefficients.

##  4 Experiments

###  4.1 Dataset and Pre-processing

We train our model on real video dataset VFHQ [63]. Following the video pre-
processing strategies in [52, 31], we convert the original video into 25 fps
and crop the facial regions of resolution $256\times 256$, ensuring that the
nose is located at the center of the frame. We use a 3DMM extractor [16] to
obtain the expression parameter $\beta\in\mathbb{R}^{64}$. We adopt the pre-
preprocesing strategy of EG3D [10] for the camera parameter
$p\in\mathbb{R}^{25}$ (the concatenation of the camera intrinsic parameters in
$\mathbb{R}^{9}$ and the inverse extrinsic parameter in $\mathbb{R}^{16}$).
After the video pre-processing, 6196 video clips are used for training, and 50
videos are used for test. We also evaluate our model on the test dataset of
TalkingHead-1KH [57]. After the same pre-processing, remaining 20 videos of
different identities are used. Note that we do not use TalkingHead-1KH for
training or fine-tuning.

###  4.2 Implementation Details

We set $n=10$ basis vectors for LeBS in the subspace in $\mathbb{R}^{256}$
($d=256$). For contrastive pre-training of LeBS, we draw 32 negative samples
per 1 positive sample and the temperature $\tau$ in (1) is $0.07$. We
empirically set the balancing coefficients in (12) by $\lambda_{rec}=1$, and
$\lambda_{adv}=0.01$. For all training, we use Adam [32] optimizer with the
learning rate $10^{-4}$ for Export3D, $10^{-4}$ for CLeBS, and $10^{-5}$ for
the discriminator, respectively. We train our model for 300,000 steps with the
reconstruction loss (11) and then incorporate the adversarial loss (12) for
10,000 steps to slightly improve the visual quality. Overall training conducts
on a single A100 GPU about 5 days with batch size 8. In the inference phase,
we use randomly sampled frontal frame as the source frame.

###  4.3 Evaluation

Table 1: Quantitative comparison of on VFHQ. The best score for each metric is
in bold. Note that we only measure CSIM [14], AED and APD [16, 47] for the
cross-identity experiment as no ground-truth is available.
†: Evaluated only on the foreground facial region.

Model | Same-identity | Cross-identity
---|---|---
PSNR $\uparrow$ |  SSIM $\uparrow$ |  AKD $\downarrow$ |  CSIM $\uparrow$ |  AED $\downarrow$ |  APD $\downarrow$ |  CSIM $\uparrow$ |  AED $\downarrow$ |  APD $\downarrow$
StyleHEAT [64] | 14.233 | 0.428 | 30.406 | 0.464 | 0.161 | 0.139 | 0.505 | 0.242 | 0.136
DPE [42] | 23.241 | 0.750 | 3.661 | 0.831 | 0.083 | 0.032 | 0.586 | 0.253 | 0.085
ROME† [30] | 14.185 | 0.642 | 7.281 | 0.737 | 0.111 | 0.051 | 0.641 | 0.224 | 0.074
OTAvatar† [38] | 17.441 | 0.651 | 11.502 | 0.662 | 0.176 | 0.067 | 0.610 | 0.290 | 0.198
HiDe-NeRF† [34] | 21.228 | 0.728 | 8.245 | 0.867 | 0.106 | 0.049 | 0.707 | 0.255 | 0.065
Ours | 23.555 | 0.704 | 3.453 | 0.811 | 0.082 | 0.030 | 0.694 | 0.208 | 0.080

Table 2: Quantitative comparison of on TalkingHead-1KH. The best score for
each metric is in bold. Note that we only measure CSIM [14], AED and APD [16,
47] for the cross-identity experiment as no ground-truth is available.
†: Evaluated only on the foreground facial region.

Method | Same-identity | Cross-identity
---|---|---
PSNR $\uparrow$ |  SSIM $\uparrow$ |  AKD $\downarrow$ |  CSIM $\uparrow$ |  AED $\downarrow$ |  APD $\downarrow$ |  CSIM $\uparrow$ |  AED $\downarrow$ |  APD $\downarrow$
StyleHEAT [64] | 15.613 | 0.517 | 21.198 | 0.575 | 0.148 | 0.095 | 0.571 | 0.218 | 0.102
DPE [42] | 23.201 | 0.786 | 4.281 | 0.807 | 0.093 | 0.029 | 0.714 | 0.216 | 0.081
ROME† [30] | 15.921 | 0.695 | 13.444 | 0.726 | 0.123 | 0.062 | 0.667 | 0.201 | 0.084
OTAvatar† [38] | 16.952 | 0.660 | 11.615 | 0.668 | 0.181 | 0.063 | 0.682 | 0.247 | 0.150
HiDe-NeRF† [34] | 19.759 | 0.729 | 5.746 | 0.843 | 0.112 | 0.043 | 0.757 | 0.232 | 0.085
Ours | 23.239 | 0.797 | 3.581 | 0.764 | 0.092 | 0.033 | 0.772 | 0.204 | 0.076

Figure 4: Comparison on same-identity experiments. For a fair comparison, we
follow the pre-processing strategy of each method.

We compare our model against 2D-based [64, 42] and 3D-based [30, 38, 34]
image-driven portrait animation methods whose official implementations are
available. StyleHEAT [64] warps the 2D spatial features of pre-trained
StyleGAN2 using 3DMM parameters, DPE [42] disentangles the pose and the
expression in the motion latent space without using 3DMM parameters. ROME [30]
is a mesh-based method transferring the expression and pose using 3DMM.
OTAvatar [38] leverages pre-trained EG3D [10] by modeling head motion in terms
of latent codes. HiDe-NeRF [34] deforms the source tri-plane by predicting
expression-aware deformation fields. For evaluation, we employ peak signal-to-
noise ratio (PSNR) and structural similarity index measure (SSIM) for image
quality, average key-point distance (AKD) [47] for facial structure based on
the 68 facial key-points, cosine similarity of identity embedding (CSIM) [14]
for identity preservation, average expression distance (AED), and average pose
distance (APD) [16, 47] for expression transferring and pose matching. For the
cross-identity experiments, we only measure CSIM, AED and APD as no ground-
truth image is available.

Figure 5: Comparison on cross-identity experiments. For a fair comparison, we
follow the pre-processing strategy of each method.

Same-identity experiments. We report the same-identity transfer experiment
results in Tab. 1 and Tab. 2, and illustrate the qualitative results in Fig.
4. For a fair comparison, ROME [30], OTAvatar [38], and HiDe-NeRF [38] are
evaluated on the foreground facial region with different field of view. DPE
[42] shows the stable performance in the same-identity experiments with the
fine-grained expression controls. Among the 3D-based methods, HiDe-NeRF [34]
scores the highest in the identity preservation (CSIM). Our method scores the
best result in the majority of evaluation metrics. Especially, it has an
advantage in expression controls (AKD and AED).

Cross-identity experiments. In Tab. 1 and Tab. 2, we also conduct the cross-
identity transfer experiments that transfers the expression and pose of
different identity into the source identity. As illustrated in Fig. 5, DPE
[42] shows visual artifacts and appearance swap, such as face contours and eye
shape, due to the insufficient disentanglement of expression and pose in the
motion space. HiDe-NeRF [34] scores the highest identity preservation (CSIM)
while un-predictable light changes are involved due to the point-wise
deformation field on the tri-plane. Our method can imitate the driving
expression without appearance swap and generates a video without video-level
artifacts such as light changes and flickers. Please refer to our
supplementary videos.

Table 3: Ablation study on the expression encoding. We employ the same
evaluation setting with Tab. 1. The best score for each metric is in bold.

Method | Same-identity | Cross-identity
---|---|---
PSNR$\uparrow$ | SSIM$\uparrow$ | AKD$\downarrow$ | CSIM$\uparrow$ | AED$\downarrow$ | APD$\downarrow$ | CSIM$\uparrow$ | AED$\downarrow$ | APD$\downarrow$
Direct 3DMM | 23.077 | 0.688 | 3.874 | 0.789 | 0.105 | 0.044 | 0.648 | 0.209 | 0.073
E2E LeBS ($n=25$) | 23.105 | 0.672 | 3.775 | 0.745 | 0.109 | 0.040 | 0.670 | 0.218 | 0.071
E2E LeBS ($n=10$) | 23.235 | 0.676 | 3.755 | 0.751 | 0.110 | 0.038 | 0.672 | 0.238 | 0.079
E2E LeBS ($n=5$) | 22.631 | 0.646 | 4.114 | 0.658 | 0.140 | 0.046 | 0.632 | 0.246 | 0.076
Full (CLeBS) | 23.555 | 0.704 | 3.453 | 0.811 | 0.082 | 0.030 | 0.694 | 0.208 | 0.080

Figure 6: Ablation studies on expression encoding. The first row is the
results of same-identity transfer, and the second row is the results of cross-
identity transfer.

###  4.4 Ablation Studies

Ablation on the expression encoding. In Tab. 3, we conduct ablation studies on
different expression encoding strategies. In Direct 3DMM, we replace our CLeBS
with fully-connected layers to directly inject the expression parameters of
3DMM through EAdaLN. As illustrated in Fig. 6, the direct injection does not
change appearance in the same-identity transfer however, it changes appearance
(e.g., eyebrows, eyelids, and face contour) as well . Furthermore, since the
raw expression parameters inherently contain noise, the generated image
exhibits visual artifacts in the cross-identity transfer. In E2E LeBS, we vary
the the number of basis vectors $n$ (25, 10, 5) in LeBS to demonstrate the
effectiveness of the contrastive pre-training to dicard the appearance
information. In these scenarios, LeBS is trained together with entire model in
end-to-end manner. Due to the entanglement of appearance and expression, both
appearance and expression are changed as a whole as the the number of basis
vector $n$ decreases. LeBS alone is insufficient for extracting appearance-
free expression from the expression parameters.

Visualization of facial expression parameters. In Fig. 8, we sample 10 random
frames from 10 different videos of distinct people in VFHQ [63] and visualize
the low-dimensional t-SNE [39] results of the two expression parameters:
$\beta\in\mathbb{R}^{64}$ and $\beta^{\prime}\in\mathbb{R}^{d}$. As depicted
in Fig. 7(a), the 3DMM expression parameters exhibit entanglement with their
respective source identity, indicating that they still contain appearance
information. On the other hand, contrastive pre-training mitigates the
entanglement between appearance and expression (Fig. 7(b)), thereby resolving
the appearance swap in the cross-identity expression transfer in Fig. 6.

(a) 3DMM expression parameter $\beta$

(b) $\beta^{\prime}$ from the CLeBS.

Figure 7: Visualization of the expression parameters. We plot t-SNE [39] of
raw 3DMM expression and our appearance-free expression parameter.

(c) Basis vector $\mathbf{v}_{4}$

(d) Basis vector $\mathbf{v}_{8}$

Figure 8: Linear scaling along the different basis in CLeBS. We visualize the
different expression directions along the basis vector in
$\mathbf{v}_{4},\mathbf{v}_{8}\in\textbf{V}$.

Linear scaling along the orthogonal directions. In Fig. 8, we verify that
$\beta^{\prime}\in\mathbb{R}^{d}$ has the orthogonal structure where the
learned basis $\mathbf{V}$ determines the different expressions even if
trained in unsupervised manner and the coefficients
$\\{\lambda_{i}\\}_{i=1}^{n}$ scale their intensities. Specifically, we
visualize two orthogonal directions $\mathbf{v}_{4}$ and $\mathbf{v}_{8}$ and
linearly scale their coefficients $\lambda_{4}$ and $\lambda_{8}$ from $1$ to
$10$. As shown in Fig. 7(c), $\mathbf{v}_{4}$ controls eye closing and mouth
closing, while Fig. 7(d) illustrates that $\mathbf{v}_{8}$ controls mouth
opening. Notably, the orthogonal basis does not influence head movements.

Novel-view synthesis with expression transfer. In Fig. 9, we compare the
results of novel-view synthesis with expression transfer to those of HiDe-NeRF
[34]. Both methods utilize the tri-plane and differentiable volume rendering
to generate novel-view images. However, while HiDe-NeRF transfers the driving
expression by deforming the generated tri-plane into a canonical tri-plane
based on driving conditions [43], our method relies on the hybrid generator
$\mathbf{G}$ with EAdaLN. In both the same-identity and cross-identity
transfer scenarios, our approach exhibits more view-consistent results,
highlighting the effectiveness of our method in expression transfer without
relying on deformation. Please refer to supplementary videos.

Figure 9: Novel-view synthesis results with expression transfer.

##  5 Conclusion

We presented Export3D, a 3D-aware portrait image animation model that controls
the facial expression and the camera view of a source image by leveraging the
driving 3DMM expression and camera parameters. Since the expression parameters
are still entangled with appearance information, we proposed a contrastive
pre-training framework to extract appearance-free expression latent from the
parameters. This latent is injected into our generator through expression
adaptive layer normalization (EAdaLN) that produces a tri-plane of source
identity and driving expression. Finally, differentiable volume rendering
renders the tri-plane into 2D images of different views. Extensive experiments
show that our contrastive pre-training framework removes the appearance
information from the expression parameters, enabling our model to transfer the
cross-identity expressions without undesirable appearance swap.

Limitations and future work. While our method can generate realistic face
videos with driving expressions and views, it still has several limitations.
First, our method cannot separately generate the foreground and background
regions as the tri-plane representation construct them as a whole. Several
works address this limitation by extending the tri-plane representation [4],
restricting rendering points in the ray marching process [51], or leveraging
the off-the-shelf segmentation model [29] to manually separate them [34, 35,
38]. Second, our method cannot control non-facial body parts (e.g., neck and
shoulders) and eye gazing as they are beyond the capability of the 3DMM
parameters. We plan to address these limitations for future work.

Ethical consideration. Since our method can generate a realistic video using a
single portrait image, it has the potential for misuse, such as fake news
creations. We have planned to attach visible and invisible watermarks to the
generated videos and restrict the source identities for inference in research
demonstration.

## References

  * [1] Abdal, R., Qin, Y., Wonka, P.: Image2stylegan: How to embed images into the stylegan latent space? In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 4432–4441 (2019)
  * [2] Abdal, R., Qin, Y., Wonka, P.: Image2stylegan++: How to edit the embedded images? In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 8296–8305 (2020)
  * [3] Amberg, B., Knothe, R., Vetter, T.: Expression invariant 3d face recognition with a morphable model. In: 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition. pp. 1–6 (2008)
  * [4] An, S., Xu, H., Shi, Y., Song, G., Ogras, U.Y., Luo, L.: Panohead: Geometry-aware 3d full-head synthesis in 360deg. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 20950–20959 (2023)
  * [5] Ba, J.L., Kiros, J.R., Hinton, G.E.: Layer normalization. arXiv preprint arXiv:1607.06450 (2016)
  * [6] Barron, J.T., Mildenhall, B., Tancik, M., Hedman, P., Martin-Brualla, R., Srinivasan, P.P.: Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 5855–5864 (2021)
  * [7] Bhattarai, A.R., Nießner, M., Sevastopolsky, A.: Triplanenet: An encoder for eg3d inversion. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). pp. 3055–3065 (2024)
  * [8] Blanz, V., Vetter, T.: A morphable model for the synthesis of 3d faces. In: Proceedings of the 26th annual conference on Computer graphics and interactive techniques. pp. 187–194 (1999)
  * [9] Bulat, A., Tzimiropoulos, G.: How far are we from solving the 2d & 3d face alignment problem?(and a dataset of 230,000 3d facial landmarks). In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 1021–1030 (2017)
  * [10] Chan, E.R., Lin, C.Z., Chan, M.A., Nagano, K., Pan, B., De Mello, S., Gallo, O., Guibas, L.J., Tremblay, J., Khamis, S., et al.: Efficient geometry-aware 3d generative adversarial networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 16123–16133 (2022)
  * [11] Chan, E.R., Monteiro, M., Kellnhofer, P., Wu, J., Wetzstein, G.: pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 5799–5809 (2021)
  * [12] Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: International Conference on Machine Learning (ICML). pp. 1597–1607 (2020)
  * [13] Cheng, K., Cun, X., Zhang, Y., Xia, M., Yin, F., Zhu, M., Wang, X., Wang, J., Wang, N.: Videoretalking: Audio-based lip synchronization for talking head video editing in the wild. In: SIGGRAPH Asia 2022 Conference Papers. pp. 1–9 (2022)
  * [14] Deng, J., Guo, J., Xue, N., Zafeiriou, S.: Arcface: Additive angular margin loss for deep face recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 4690–4699 (2019)
  * [15] Deng, Y., Yang, J., Xiang, J., Tong, X.: Gram: Generative radiance manifolds for 3d-aware image generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 10673–10683 (2022)
  * [16] Deng, Y., Yang, J., Xu, S., Chen, D., Jia, Y., Tong, X.: Accurate 3d face reconstruction with weakly-supervised learning: From single image to image set. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). pp. 0–0 (2019)
  * [17] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)
  * [18] Egger, B., Sutherland, S., Medin, S.C., Tenenbaum, J.: Identity-expression ambiguity in 3d morphable face models. In: 2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021). pp. 1–7 (2021)
  * [19] Gao, Y., Zhou, Y., Wang, J., Li, X., Ming, X., Lu, Y.: High-fidelity and freely controllable talking head video generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 5609–5619 (2023)
  * [20] Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial networks. Communications of the ACM 63(11), 139–144 (2020)
  * [21] Gu, J., Liu, L., Wang, P., Theobalt, C.: Stylenerf: A style-based 3d aware generator for high-resolution image synthesis. In: International Conference on Learning Representations (ICLR) (2022), https://openreview.net/forum?id=iUuzzTMUw9K
  * [22] Guo, Y., Chen, K., Liang, S., Liu, Y.J., Bao, H., Zhang, J.: Ad-nerf: Audio driven neural radiance fields for talking head synthesis. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 5784–5794 (2021)
  * [23] He, K., Fan, H., Wu, Y., Xie, S., Girshick, R.: Momentum contrast for unsupervised visual representation learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 9729–9738 (2020)
  * [24] Hong, F.T., Xu, D.: Implicit identity representation conditioned memory compensation network for talking head video generation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 23062–23072 (2023)
  * [25] Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 7132–7141 (2018)
  * [26] Karras, T., Aittala, M., Hellsten, J., Laine, S., Lehtinen, J., Aila, T.: Training generative adversarial networks with limited data. Advances in Neural Information Processing Systems (NeurIPS) 33, 12104–12114 (2020)
  * [27] Karras, T., Aittala, M., Laine, S., Härkönen, E., Hellsten, J., Lehtinen, J., Aila, T.: Alias-free generative adversarial networks. Advances in Neural Information Processing Systems (NeurIPS) 34, 852–863 (2021)
  * [28] Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., Aila, T.: Analyzing and improving the image quality of stylegan. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 8110–8119 (2020)
  * [29] Ke, Z., Sun, J., Li, K., Yan, Q., Lau, R.W.: Modnet: Real-time trimap-free portrait matting via objective decomposition. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 36, pp. 1140–1147 (2022)
  * [30] Khakhulin, T., Sklyarova, V., Lempitsky, V., Zakharov, E.: Realistic one-shot mesh-based head avatars. In: European Conference on Computer Vision (ECCV). pp. 345–362 (2022)
  * [31] Ki, T., Min, D.: Stylelipsync: Style-based personalized lip-sync video generation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 22841–22850 (2023)
  * [32] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  * [33] Ko, J., Cho, K., Choi, D., Ryoo, K., Kim, S.: 3d gan inversion with pose optimization. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). pp. 2967–2976 (2023)
  * [34] Li, W., Zhang, L., Wang, D., Zhao, B., Wang, Z., Chen, M., Zhang, B., Wang, Z., Bo, L., Li, X.: One-shot high-fidelity talking-head synthesis with deformable neural radiance field. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 17969–17978 (2023)
  * [35] Li, X., De Mello, S., Liu, S., Nagano, K., Iqbal, U., Kautz, J.: Generalizable one-shot 3d neural head avatar. Advances in Neural Information Processing Systems (NeurIPS) 36 (2024)
  * [36] Lombardi, S., Simon, T., Saragih, J., Schwartz, G., Lehrmann, A., Sheikh, Y.: Neural volumes: Learning dynamic renderable volumes from images. arXiv preprint arXiv:1906.07751 (2019)
  * [37] Ma, Y., Wang, S., Hu, Z., Fan, C., Lv, T., Ding, Y., Deng, Z., Yu, X.: Styletalk: One-shot talking head generation with controllable speaking styles. arXiv preprint arXiv:2301.01081 (2023)
  * [38] Ma, Z., Zhu, X., Qi, G.J., Lei, Z., Zhang, L.: Otavatar: One-shot talking face avatar with controllable tri-plane rendering. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 16901–16910 (2023)
  * [39] Van der Maaten, L., Hinton, G.: Visualizing data using t-sne. Journal of Machine Learning Research 9(11) (2008)
  * [40] Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM 65(1), 99–106 (2021)
  * [41] Min, D., Song, M., Hwang, S.J.: Styletalker: One-shot style-based audio-driven talking head video generation. arXiv preprint arXiv:2208.10922 (2022)
  * [42] Pang, Y., Zhang, Y., Quan, W., Fan, Y., Cun, X., Shan, Y., Yan, D.m.: Dpe: Disentanglement of pose and expression for general video portrait editing. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 427–436 (2023)
  * [43] Park, K., Sinha, U., Barron, J.T., Bouaziz, S., Goldman, D.B., Seitz, S.M., Martin-Brualla, R.: Nerfies: Deformable neural radiance fields. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 5865–5874 (2021)
  * [44] Peebles, W., Xie, S.: Scalable diffusion models with transformers. arXiv preprint arXiv:2212.09748 (2022)
  * [45] Prajwal, K., Mukhopadhyay, R., Namboodiri, V.P., Jawahar, C.: A lip sync expert is all you need for speech to lip generation in the wild. In: Proceedings of the 28th ACM International Conference on Multimedia. pp. 484–492 (2020)
  * [46] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International Conference on Machine Learning (ICML). pp. 8748–8763 (2021)
  * [47] Ren, Y., Li, G., Chen, Y., Li, T.H., Liu, S.: Pirenderer: Controllable portrait image generation via semantic neural rendering. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 13759–13768 (2021)
  * [48] Richardson, E., Alaluf, Y., Patashnik, O., Nitzan, Y., Azar, Y., Shapiro, S., Cohen-Or, D.: Encoding in style: a stylegan encoder for image-to-image translation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 2287–2296 (2021)
  * [49] Roich, D., Mokady, R., Bermano, A.H., Cohen-Or, D.: Pivotal tuning for latent-based editing of real images. ACM Transactions on Graphics (TOG) 42(1), 1–13 (2022)
  * [50] Schwarz, K., Liao, Y., Niemeyer, M., Geiger, A.: Graf: Generative radiance fields for 3d-aware image synthesis. In: Advances in Neural Information Processing Systems (NeurIPS) (2020)
  * [51] Shin, M., Seo, Y., Bae, J., Choi, Y.S., Kim, H., Byun, H., Uh, Y.: Ballgan: 3d-aware image synthesis with a spherical background. arXiv preprint arXiv:2301.09091 (2023)
  * [52] Siarohin, A., Lathuilière, S., Tulyakov, S., Ricci, E., Sebe, N.: First order motion model for image animation. Advances in Neural Information Processing Systems (NeurIPS) 32 (2019)
  * [53] Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
  * [54] Sun, J., Wang, X., Wang, L., Li, X., Zhang, Y., Zhang, H., Liu, Y.: Next3d: Generative neural texture rasterization for 3d-aware head avatars. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 20991–21002 (2023)
  * [55] Tov, O., Alaluf, Y., Nitzan, Y., Patashnik, O., Cohen-Or, D.: Designing an encoder for stylegan image manipulation. ACM Transactions on Graphics (TOG) 40(4), 1–14 (2021)
  * [56] Trevithick, A., Chan, M., Stengel, M., Chan, E.R., Liu, C., Yu, Z., Khamis, S., Chandraker, M., Ramamoorthi, R., Nagano, K.: Real-time radiance fields for single-image portrait view synthesis. arXiv preprint arXiv:2305.02310 (2023)
  * [57] Wang, T.C., Mallya, A., Liu, M.Y.: One-shot free-view neural talking-head synthesis for video conferencing. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 10039–10049 (2021)
  * [58] Wang, Y., Yang, D., Bremond, F., Dantcheva, A.: Latent image animator: Learning to animate images via latent space navigation. arXiv preprint arXiv:2203.09043 (2022)
  * [59] Wu, Y., Deng, Y., Yang, J., Wei, F., Chen, Q., Tong, X.: Anifacegan: Animatable 3d-aware face image generation for video avatars. Advances in Neural Information Processing Systems (NeurIPS) 35, 36188–36201 (2022)
  * [60] Xiang, J., Yang, J., Deng, Y., Tong, X.: Gram-hd: 3d-consistent image generation at high resolution with generative radiance manifolds. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 2195–2205 (2023)
  * [61] Xie, E., Wang, W., Yu, Z., Anandkumar, A., Alvarez, J.M., Luo, P.: Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information Processing Systems (NeurIPS) 34, 12077–12090 (2021)
  * [62] Xie, J., Ouyang, H., Piao, J., Lei, C., Chen, Q.: High-fidelity 3d gan inversion by pseudo-multi-view optimization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 321–331 (2023)
  * [63] Xie, L., Wang, X., Zhang, H., Dong, C., Shan, Y.: Vfhq: A high-quality dataset and benchmark for video face super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 657–666 (2022)
  * [64] Yin, F., Zhang, Y., Cun, X., Cao, M., Fan, Y., Wang, X., Bai, Q., Wu, B., Wang, J., Yang, Y.: Styleheat: One-shot high-resolution editable talking face generation via pre-trained stylegan. In: European Conference on Computer Vision (ECCV). pp. 85–101 (2022)
  * [65] Yin, F., Zhang, Y., Wang, X., Wang, T., Li, X., Gong, Y., Fan, Y., Cun, X., Shan, Y., Oztireli, C., Yang, Y.: 3d gan inversion with facial symmetry prior. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 342–351 (2023)
  * [66] Yu, W., Fan, Y., Zhang, Y., Wang, X., Yin, F., Bai, Y., Cao, Y.P., Shan, Y., Wu, Y., Sun, Z., et al.: Nofa: Nerf-based one-shot facial avatar reconstruction. In: ACM SIGGRAPH 2023 Conference Proceedings. pp. 1–12 (2023)
  * [67] Yuan, Z., Zhu, Y., Li, Y., Liu, H., Yuan, C.: Make encoder great again in 3d gan inversion through geometry and occlusion-aware encoding. arXiv preprint arXiv:2303.12326 (2023)
  * [68] Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 586–595 (2018)
  * [69] Zhang, W., Cun, X., Wang, X., Zhang, Y., Shen, X., Guo, Y., Shan, Y., Wang, F.: Sadtalker: Learning realistic 3d motion coefficients for stylized audio-driven single image talking face animation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 8652–8661 (2023)
  * [70] Zhao, J., Zhang, H.: Thin-plate spline motion model for image animation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 3657–3666 (2022)
  * [71] Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV). pp. 2223–2232 (2017)

##  6 Supplementary Material

###  6.1 3D Morphable Models (3DMM).

Figure 10: 3DMM [8] vs. Leaned Basis Scaling (LeBS). 3DMM based method
reconstructs 3D facial geometry by scaling the the pre-defined basis
$\mathbf{U}_{exp}$ with expression parameters $\beta\in\mathbb{R}^{64}$. LeBS,
on the other hand, uses the learned basis
$V=\\{v_{i}\\}_{i=1}^{n}\subseteq\mathbb{R}^{d}$ which is scaled by the low-
dimensional coefficients $\lambda=(\lambda_{i})_{i=1}^{n}\in\mathbb{R}^{n}$
($n\ll 64$).

3D Morphable Models (3DMM) [8] are statistical models of 3D shape and their
corresponding texture. In this paper, we only consider the shape
representation of 3DMM. To be specific, a face shape $\mathbf{S}$ is
initialized with the average shape $\bar{\mathbf{S}}$ and further shaped by a
linear combination of expression and identity as follows:

$\mathbf{S}=\bar{\mathbf{S}}+\alpha\mathbf{U}_{id}+\beta\mathbf{U}_{exp},$
(13)

where $\mathbf{U}_{id}\in\mathbb{R}^{80\times d_{3dmm}}$,
$\mathbf{U}_{exp}\in\mathbb{R}^{68\times d_{3dmm}}$ are the pre-defined bases
of identity and expression subspaces of 3D face space, respectively.
$d_{3dmm}$ is the dimension of the 3D face space. The coefficients
$\alpha\in\mathbb{R}^{80}$ and $\beta\in\mathbb{R}^{64}$ determine the facial
identity and expression for the face geometry reconstruction by scaling each
basis [3].

In this paper, we term appearance as the set of geometric features that
determine the facial identity of a given face, such as head size, face
contour, face proportion, eyebrows, eye shape, mouth shape, jaw shape, etc.,
and expression as the motion of these appearance features, such as mouth
opening (closing), eye blinking, etc.

###  6.2 Detailed Model Architectures.

Our model consists of four parts: Learned Basis Scaling (LeBS), Hybrid Tri-
plane Generator $\mathbf{G}$, Light-weight MLP decoder (MLP) for color and
density prediction used in the differentiable volume rendering [40], and
Super-resolution (SR) module. The detailed model architectures are shown in
Fig. 10 and Fig. 11.

LeBS consists of two fully-connected layers with LeakyReLU activation, along
with the learned orthonormal basis $V\subseteq\mathbb{R}^{d}$. For the
contrastive pre-training of LeBS, we employ ResNetSE18 feature extractor [25]
followed by a single fully-connected layer to output the $d$-dimensional
vector, serving as the image encoder $f_{I}(\cdot)$. Notably, we do not
introduce an orthonormal basis to $f_{I}(\cdot)$.

Inspired by [56], we incorporate ViT blocks [17] into our generator
$\mathbf{G}$, specifically utilizing those from SegFormer [61] and DiT [44].
In both EAdaLN-ViT and ViT, we employ four heads with 1024 hidden dimensions
for the multi-head self-attention. It is worth mentioning that the
architectures of EAdaLN-ViT and ViT illustrated in Fig. 11 are the same, with
the exception of EAdaLN integration for expression transfer. We employ the
exponential moving average (EMA) on the tri-planes for stabilizing the
training. More precisely, in the $j$-th gradient step, we calculate and update
the EMA $\text{T}^{j}_{EMA}$ and the current tri-plane $\text{T}^{j}$ as
follows:

$\text{T}^{j}_{EMA}\leftarrow\delta\cdot\text{T}^{j-1}_{EMA}+(1-\delta)\cdot\bar{\text{T}}^{j}\quad\text{and}\quad\text{T}^{j}\leftarrow\text{T}^{j}+\text{T}^{j-1}_{EMA}$
(14)

where $\bar{\text{T}}^{j}$ is the average tri-plane calculated within the
$j$-th batch and $\text{T}^{0}_{EMA}$ is initialized by
$\mathbf{0}\in\mathbb{R}^{3\times 32\times 128\times 128}$. We set
$\delta=0.998$ as the weight for the moving average.

MLP for color and density prediction consists of a stack of fully-connected
layers with soft-plus activation. In contrast to [10], we use two fully-
connected layers to separately predict them.

For SR, we follow the super-resolution module used in [28, 10] except for the
style modulated convolutions.

Figure 11: The detailed model architectures. $k\times k$-Conv-$C$-$s$-$p$ is
the convolution operator with the kernel size $k\times k$, output channel size
$C$, stride step $s$, and padding size $p$. Linear-$C_{0}$-$C_{1}$ is the
fully-connected layer of the input channel size $C_{0}$ and the output channel
size $C_{1}$.

###  6.3 More Implementation Details.

####  6.3.1 Training.

Since our model does not rely on pre-trained EG3D [10, 56], it is trained end-
to-end, except for LeBS. We pre-train LeBS for 60,000 steps with a batch size
of 8, which takes approximately 1 day. Longer pre-training does not result in
significant performance improvements.

###  6.4 Evaluation.

####  6.4.1 Evaluation Metrics.

We provide additional explanations of the evaluation metrics. Average key-
point distance (AKD) is the L1 distance of 68 facial key-points between the
generated image and the driving image, which measures the facial structure
similarity based on the key-points. We use the face-alignment [9] to extract
the key-points. Cosine similarity of identity embedding (CSIM) is the cosine
similarity between the identity embeddings of the source image and the
generated image where the embeddings are extracted from ArcFace [14]. Average
expression distance (AED) and average pose distance (APD) are the L1 distance
between the expression parameters (64 dimensions) and the pose parameters (6
dimensions), respectively extracted from the generated image and the driving
image. We use the 3DMM extractor [16] to extract those parameters.

(a) Basis vector $\mathbf{v}_{1}$

(b) Basis vector $\mathbf{v}_{3}$

(c) Basis vector $\mathbf{v}_{6}$

(d) Basis vector $\mathbf{v}_{9}$

Figure 12: Linear scaling along the different basis in CLeBS. Figure 13:
Ablation studies on expression encoding. We add red markings for clearer
comparison. The dotted lines in the last row are the same length.

###  6.5 Additional Results.

####  6.5.1 Linear Scaling along the Orthonormal Basis.

In Fig. 12, we show additional results of linear scaling along the different
basis vectors [58]. For $\mathbf{v}_{1}$, we scale $\lambda_{1}$ from 1 to -7,
showing mouth opening and eye closing. For $\mathbf{v}_{3}$, we scale
$\lambda_{3}$ from 1 to 20, showing eye closing and lip pursing. For
$\mathbf{v}_{6}$, we scale $\lambda_{6}$ from 1 to -7, showing eyebrow moving.
For $\mathbf{v}_{9}$, we scale $\lambda_{9}$ 1 from to -10, showing eye
closing and smiling. Since our method does not constrain the range of the
coefficients $\lambda=(\lambda_{i})_{i=1}^{10}$, the manipulation can be
realized along the negative scaling. Please refer to video results.

####  6.5.2 Additional Ablation Results on Expression Encoding.

We provide additional ablation results in Fig. 13 with red markings for
clearer comparison. Specifically, in cross-identity transfer, there is
appearance leakage when our full model is not utilized, such as head size,
eyebrows, and eye shape, etc. Please refer to the video results for further
details.

Figure 14: Novel-view synthesis results with expression transfer. Figure 15:
Limitation cases of Export3D. The red arrows indicate the directions of eye
gaze.

####  6.5.3 Additional Comparison with HiDe-NeRF.

In Fig. 14, we exhibit additional comparison results with HiDe-NeRF [34] for
novel-view synthesis with expression transfer. Please refer to the video
results for further details.

###  6.6 Limitations and Future Work.

We exhibit the limitation cases of Export3D in Fig. 15. Since the tri-plane
represents [10] the foreground and the background as a whole, our model
jointly renders them, resulting in head pose-aligned distortion. Several prior
works [35, 34, 30, 38] address this issue by removing the complex background
and providing the volume rendering with a uniform background. However, they
heavily rely on the performance of the background segmentation model [29],
exhibiting the temporal jitters in the generated videos. Additionally, our
model cannot control eye gazing since the 3DMM parameters do not model eye
movement. We leave these limitations for future research.


# Enhancing DMI Interactions by Integrating Haptic Feedback for Intricate
Vibrato Technique

Ziyue Piao

Christian Frisson

Bavo Van Kerrebroeck

IDMIL, CIRMMT, McGill University  Montreal, Quebec, Canada
<EMAIL_ADDRESS>IDMIL, McGill University  Montreal, Quebec, Canada
<EMAIL_ADDRESS>IDMIL, CIRMMT, McGill University  Montreal,
Quebec, Canada<EMAIL_ADDRESS>Marcelo M.Wanderley

IDMIL, CIRMMT, McGill University  Montreal, Quebec, Canada
<EMAIL_ADDRESS>
###### Abstract

This paper investigates the integration of force feedback in Digital Musical
Instruments (DMI), specifically evaluating the reproduction of intricate
vibrato techniques using haptic feedback controllers. We introduce our system
for vibrato modulation using force feedback, composed of Bend-aid (a web-based
sequencer platform using pre-designed haptic feedback models) and TorqueTuner
(an open-source 1 Degree-of-Freedom (DoF) rotary haptic device for generating
programmable haptic effects). We designed a formal user study to assess the
impact of each haptic mode on user experience in a vibrato mimicry task.
Twenty musically trained participants rated their user experience for the
three haptic modes (Smooth, Detent, and Spring) using four Likert-scale scores
(comfort, flexibility, ease of control, and helpfulness for the task).
Finally, we asked participants to share their reflections. Our research
indicates that while the Spring mode can help with Light vibrato, preferences
for haptic modes vary based on musical training background. This emphasizes
the need for adaptable task interfaces and flexible haptic feedback in DMI
design.

###### keywords:

Rotary force feedback, Haptics, Vibrato, TorqueTuner

[500]Applied computing Sound and music computing [500]Hardware Haptic devices

##  1 Introduction

Figure 1: A picture of a person interacting with a Guzheng, embedded in a
larger sketch of a person interacting with a haptic DMI inspired by the
Guzheng with three haptic modes

Adding force feedback to DMIs could potentially enable more complex and
expressive sounds to knobs in traditional musical devices, addressing the
challenges and opportunities identified by [1] for force-feedback digital
musical instruments. In this paper, we explore the application of three force
feedback modes to enhance the expression of Guzheng vibrato technique through
a programmable knob, because of the ubiquity of rotary controls in musical
hardware. We utilized the embedded device TorqueTuner to provide dynamic and
modifiable rotary force feedback to enable precise control over complex
vibrato techniques [2]. We also developed Bend-aid, a web-based MIDI editing
tool that integrated with the TorqueTuner, enabling musicians to apply
specific haptic feedback to pitch-related techniques. To assess the
effectiveness of our haptic feedback system, we conducted a user study
focusing on the replication of the Guzheng’s Light vibrato technique.
Participants were exposed to the three force feedback settings. We evaluated
their experiences through a subjective Likert-scale rating and gathered
reflections on their perceived effectiveness of the force feedback.

##  2 Related work

Musical instrument players often rely on physical feedback to refine their
techniques and achieve desired sound and expressiveness [3]. Touch-free
instruments like the Theremin, when equipped with haptic feedback, have been
shown to enhance playability [4]. Researchers are actively exploring the
integration of haptic feedback in digital interfaces to improve performance
control, expressivity, and user experience [5].

The majority of efforts in adding haptic feedback to virtual instruments aim
to replicate the tactile sensations of acoustic musical instruments. Examples
include virtual drums with vibrotactile actuators [6] and virtual pianos with
mid-air haptic displays [7]. For string instruments, haptic feedback is
achieved through robotic arms and vibrotactile actuators to simulate the
actions of plucking, bowing, and rubbing [8, 9].

In music production, research focuses on enabling control over musical
features through force feedback. Tools like the Haptic Wave, designed for
visually impaired producers and engineers [10], demonstrate the effectiveness
of haptic feedback in audio mixing [11] and audio effects editing [12].
Projects such as SoundFORMS [13] and HaptEQ [14] allow users to interact with
and alter synthesized waveforms using gestures, highlighting force feedback’s
potential to refine musical control in production.

Rotary force-feedback controllers are commonly used in vehicular instrument
control, robot control, and media editors [12]. Rotary control, widely used in
traditional musical instruments like pitch bend wheels and knobs, often lacks
force feedback or provides only constant torque. Tools like THE PLANK address
this gap by providing one-axis force feedback for precise and rapid controls
in live performances [15]. Similarly, the D’groove intelligent DJ system
utilizes a haptic turntable for controlling digital audio playback with added
haptic feedback [16]. TorqueTuner [2, 17], a project using a DC motor for
programmable force feedback in digital musical instruments, shows promise but
lacks experimental validation of its effectiveness.

##  3 Interface Design

Figure 2: TorqueTuner with our easy-to-hold grip

We created Bend-aid, a visual interface, to explore our key question: Does
force feedback improve the mimicry of intricate Guzheng vibrato? Bend-aid
enables bi-directional communication with TorqueTuner, facilitating the
editing of pitch contours using varied force feedbacks. We designed and
3D-printed a grip for the TorqueTuner, facilitating rapid shifting gestures
essential for executing Light vibrato, as illustrated in Figure 2. In this
section, we describe various aspects of the Bend-aid’s interface design,
including visual interface design and haptic modes design. We designed two
haptic modes with force feedback (Spring mode and Detent mode) and one without
force feedback as the baseline (Smooth mode).

###  3.1 Visual Interface Design

Figure 3: UI design of Bend-aid system.

Bend-aid is a web-based interface, as shown in Figure 3, designed for editing
MIDI notes and incorporating various rotary force feedbacks to facilitate
pitch bending. Gesture data can be inputted through TorqueTuner, allowing
interaction with MIDI notes using a mouse and keyboard. The real-time angle of
TorqueTuner is mapped to pitch value in this study.

####  3.1.1 MIDI sequencer

We integrated a MIDI sequencer based on an embeddable piano roll interface
called powerPianoRoll [18] to replicate the typical MIDI editing interaction
in Digital Audio Workstations (DAW). MIDI sequencer facilitates the design of
our user study and potentially makes the editing process intuitive. The audio
synthesizer is implemented using the Tone.js framework and the Guzheng sampler
[19]. The sequencer supports several MIDI editing functions similar to
commonly used DAWs like Logic Pro and Ableton Live, including:

  1. 1.

Double click on an empty cell to add a note.

  2. 2.

Click to highlight a selection.

  3. 3.

Drag a note to move it.

  4. 4.

Drag the beginnings and ends of a note to resize it.

The MIDI sequencer supports editing long pieces by dragging the slider beneath
it and can handle notes from low C1 up to high G8. Notes can be entered
manually, the upload” button allows for importing a MIDI file from a local
folder, and the clear” button can be used to remove all notes on the sequencer
canvas.

####  3.1.2 Data communication

Figure 4: Block diagram of TorqueTuner and Bend-aid system

The overall communication diagram of TorqueTuner and Bend-aid is shown in
Figure 4. Once TorqueTuner is connected to the computer via a USB serial port,
we can press the “connect” button and select the serial port of the
TorqueTuner. The TorqueTuner can then communicate with Bend-aid. In each loop,
the TorqueTuner prints real-time torque, angle, velocity, and mode name to the
serial port, which Bend-aid reads to update its variables. The visual
interface displays the haptic mode on the interface, the LCD of the
TorqueTuner, and the real-time torque and angle of the knob. When changing
haptic modes on the Bend-aid, a stream of the new mode is sent to the
TorqueTuner to change its mode and display the new haptic mode on its LCD
screen.

####  3.1.3 TorqueTuner visualizer

A knob visualizer was designed in Bend-aid to enable observation of the knob’s
status on TorqueTuner. The visualizer smoothly displays the angle with a small
blue circle rotating around a larger circle, which serves as a pointer and
shows a bar-like meter indicating torque value. When changing haptic modes,
the current knob status serves as the zero point, and the angle and torque are
reset until there is another interaction with TorqueTuner. Furthermore, we
have plotted the angle contour below the MIDI Sequencer, which can be saved as
a file for data analysis.

###  3.2 Haptic Modes Design

The haptic modes in Bend-aid have been designed based on the commonly used
rotary input devices in music hardware. Next to the three haptic modes
(Smooth, Detent, and Spring), we designed and tested some haptic modes that
were excluded after an initial pilot study. The Free mode maps a constant
value to the Torque scale. The Vibrato mode, when the knob is turned or
stopped at one angle, it continuously rotates left and right with a small
motion, and the torque changes in a sinusoidal period. We also explored the
Magnet, Friction, Inertia, and Spin haptic modes by adjusting their parameters
from the previous version of TorqueTuner [2]. Finally, we gave up using these
haptic modes during iterations of the initial pilot study because of limited
user experience compared with the other three modes.

Because the Light vibrato require subtle and fast changes between pitch
values, we found that the participant could not finish the mimicry task when
the target pitch range was mapped into a large knob angle degree. We decided
to focus instead on the force feedback design between 0-90 degrees with three
haptic modes (Smooth, Detent, and Spring).

####  3.2.1 Smooth

The Smooth mode has negligible resistance when twisting and serves as the
control condition in the following experiment to compare outcomes with and
without haptic force feedback.

####  3.2.2 Detent

The Detent mode provides a mechanical click when the knob is turned to a
certain position. This mode is commonly found in musical knobs with discrete
settings, like volume control. In each 45 degree segment, the torque
exponentially increases, opposing the turning direction. A transient change in
torque direction at the position of the Detent effectively results in a haptic
click. To facilitate the mimicry task, we increased resistance at the opposing
torque point and make the point directly linked to the target highest pitch in
Light vibrato. We aim at creating a boundary hint, enabling quicker angle
adjustments for rapid pitch changes.

####  3.2.3 Spring

The Spring mode provides resistance that increases as the knob is turned to
simulate the haptic feeling of pressing a string. This can be applied to knobs
adjusting continuous parameters, such as audio effect settings and the pitch
bend wheel. We tested exponential and linear relationships between torque and
angular velocity and finally chose the linear relationship for a more
intuitive interaction similar to pressing a string based on Hooke’s Law.

##  4 User Study

This study evaluated the effects of three force feedback modes (Smooth,
Detent, and Spring) on mimicking the Light vibrato technique in Guzheng.
First, participants completed a pre-experiment questionnaire on their musical
background. Using a within-subjects design, participants then attempted to
memorize and replicate Light vibrato with Bend-aid under three force feedback
conditions. The sequence of feedback conditions was counterbalanced to avoid
order effects. After attempts in each haptic mode, participants rated their
experience, focusing on comfort, ease of control, flexibility, and
helpfulness. Additionally, qualitative feedback was collected through a post-
study interview to delve deeper into their reflections of the task.

###  4.1 Participants

Preliminary tests with both musically trained and untrained individuals
indicated that musical background significantly influenced task performance.
Hence, for the main study, we recruited 20 musically trained participants (7
females, 13 males), compensating them $10 for their 30-minute participation.

###  4.2 Procedure

The experimenter first introduced the study procedure to the participants and
collected the consent form. Before the mimicry task, the participant was
required to fill out a questionnaire on their musical background (Table 1).
Then, the experimenter connected the TorqueTuner to a testing laptop (MacBook
Pro 13), opened Bend-aid interface, and helped the participant put on
headphones (AKG K240 Studio). After introducing the Bend-aid interface, the
experimenter gave the participant the opportunity to become familiar with the
pitch control using TorqueTuner. The experiment session followed. In each
activity, the participant listened to a short piece of audio with three
repetitive C3 notes with Light vibrato as Figure 3 shows (each lasting six
quarter notes in the 4/4 time signature at 120 bpm). The experimenter then
asked the participant to mimic the sound in Bend-aid. Participants were
allowed to try multiple times until they were satisfied with their
performance. Once the participant finished the vibrato mimicry in one haptic
mode, they completed a Likert-scale questionnaire asking for their experienced
comfort, and the device’s flexibility, ease of control, and helpfulness to the
task (Table 2). Subsequently, the participants finished another two sessions
with new haptic modes and rated their experience using the Likert Scale.
Finally, the experimenter conducted a semi-structured post-experiment
interview (Table 1).

Pre-experiment Questionnaire |  Questions
---|---
Q1 |  Do you have experience in learning musical instruments (including singing)?
Q2 |  How many instruments can you play? Please list two of your most proficient instruments and how many years you played them.
Post-experiment Interview |  Questions
Q1 |  Which haptic modes do you like the most and the least? Why?
Q2 |  Do you have any problem using the interface to control vibrato?
Table 1: Pre- and post-experiment questions Likert-scale ratings |  Explanations
---|---
Comfort |  Evaluates how physically pleasant and stress-free the haptic mode feels to the user during the task.
Ease of control |  Measures the user’s flexibility to adjust the angle in the haptic mode to suit the vibrato technique.
Flexibility |  Assesses how easily users can use TorqueTuner to accomplish tasks in the haptic mode.
Helpfulness |  Measures how much the haptic mode helps the user improve their performance and experience in the mimicry task.
Table 2: Explanation for the four Likert-scale ratings

##  5 Results

This section presents the findings from our user study, including subjective
ratings and qualitative responses.

###  5.1 Subjective Ratings

Figure 5: Likert-scale ratings from 20 participants, including means and
standard deviations for comfort, flexibility, ease of control, and task
helpfulness

We hypothesize that the three haptic modes have varying impacts on
participants’ subjective ratings. To test this, we collected data from 20
participants across four categories (comfort, ease of control, flexibility,
and helpfulness) on a 10-point Likert scale. We also calculated the average
score for each haptic mode across the four categories, identifying the mode
with the highest average as the user’s preferred mode. In Figure 5, we
visualized the Likert-scale ratings, including means and standard deviations
for the four categories. We utilized the violin plot to better visualize
statistical distributions than with only box plots [20].

We ran one-way repeated measures ANOVA on the four categories and the
following analysis is based on the new statistics [21].

For comfort, the ANOVA revealed no significant differences among three haptic
modes ($F(2,57)=0.347673,p=0.707817,\eta^{2}=0.012$), with a negligible effect
size suggesting minimal variation attributable to three haptic modes. The
overlapping confidence intervals (CI) among Detent (CI: 6.48 to 7.92), Smooth
(CI: 6.49 to 8.01), and Spring (CI: 6.76 to 8.44) reinforce the conclusion of
no substantive difference in perceived comfort. Flexibility ratings similarly
showed no significant effect of haptic mode condition
($F(2,57)=0.725,p=0.489,\eta^{2}=0.025$). Confidence intervals for Detent (CI:
5.83 to 7.67), Smooth (CI: 6.15 to 8.25), and Spring (CI: 6.70 to 8.30)
further confirm the lack of significant differences across haptic conditions
for Flexibility. These findings suggest that incorporating force feedback into
knobs does not negatively affect comfort or flexibility, even though turning
the knob with haptic feedback requires more effort.

Ease of control ($F(2,57)=2.556,p=0.086,\eta^{2}=0.082$) and helpfulness
($F(2,57)=2.845,p=0.066,\eta^{2}=0.091$) nearly reached significance,
suggesting effects of haptic conditions. For ease of control, the CIs reveal a
noticeable gap between Smooth (CI: 4.62 to 6.18), particularly between Smooth
and Spring (CI: 5.79 to 7.51). For helpfulness, the CIs for Smooth (CI: 4.29
to 6.21) shows a similar gap with that of Spring (CI: 5.63 to 7.77). The
results suggest that for participants, the Spring mode is better than Smooth
mode in making the knob ease of control and helping with finishing the task.

We used the chi-square statistic complemented by Cramér’s V [22] to examine
whether musical background influences participants’ preferences for haptic
feedback. Participants were broadly categorized based on their musical
experience into wind, and string instrument players, omitting other categories
such as piano, voice, and percussion due to either their prevalence or
scarcity. Our findings reveal a statistically significant association between
wind and string instrument players’ preferences ($\chi^{2}(2)=12.898,p<0.05$)
with Cramér’s V indicating a strong relationship ($V=0.568>0.5$). This
suggests that the type of musical instrument experience significantly
influences haptic feedback preferences, particularly distinguishing wind from
string players. Specifically, 77.8% of wind instrument players preferred the
Spring mode, while 62.5% of string players preferred the Detent mode.

###  5.2 Qualitative Responses

After the quantitative experiment, we gathered reflections from all
participants via a post-experiment interview.

Two participants, one with experience in electric guitar and another in
percussion, emphasized their reliance on tactile feedback when interacting
with Bend-aid and TorqueTuner. The guitar player noted that “the Spring mode
is the most similar mode to the tactile sensation of strings, so I can easily
apply the mode in the vibrato task”. Two piano performers shared a consistent
experience, noting: “Initially, the haptic sensation of the Spring mode was
unfamiliar for me. However, once accustomed to it, it became the most
intuitive mode”. Additionally, two participants favored the Spring mode
because of its assistance with the variation towards the end of the vibration.
Two participants expressed a preference for interacting with haptic feedback
at lower force levels, leading them to choose the Smooth mode as their
favorite. However, one electric guitar player found the Smooth mode
challenging for executing rapid vibrato movements due to the lack of haptic
feedback. Given that many instruments use linear motor gestures for pitch
control, a participant with trombone experience proposed that “applying the
results into linear force feedback may make the interaction easier and more
similar with playing the Guzheng".

##  6 Discussion

We explore the outcomes of incorporating diverse force feedback mechanisms
into Guzheng vibratos, as well as the implications for integrating tactile
sensations into future DMIs.

###  6.1 Haptic Feedback and User Experience

The subjective ratings underscore incorporating force feedback into knobs does
not adversely affect comfort or flexibility in the three haptic modes. As for
ease of control and helpfulness, the Smooth mode and Spring mode has a
significant difference. The result shows that Spring mode can improve the
perceived performance in the mimicry task, suggesting that involving physical
feedback may enhance intricate effectiveness of DMIs without compromising user
experience in flexibility and comfort.

Wind instrument players preferred the Spring mode, while string players were
less enthusiastic. This difference may be due to string players’ increased
sensitivity to the slight mismatches between the force feedback from the
TorqueTuner and the feel of real strings. Musicians’ varied experiences with
acoustic instruments shape their expectations and preferences for DMIs. This
variation suggests that a one-size-fits-all approach to haptic design in DMIs
may be less effective than a more tailored strategy that considers the
specific tactile nuances associated with different instrumental backgrounds.

###  6.2 Limitations and Future DMI Interaction

The study can be further explored in these aspects: 1) the need for
quantitative analysis by comparing pitch contours and angle time-series data
with Bend-aid’s “save” function; 2) comparing the effectiveness of linear
force feedback and rotary force feedback in the mimicry task; 3) enhancing
user experience by adjusting force feedback based on participants’ grip
strength; 4) investigating more haptic modes customized to musical backgrounds
to identify similarities and differences compared to our findings.

The preferences based on musical background further emphasize the need for
customizable haptic feedback within digital musical interfaces. The
demonstrated preference for particular haptic modes and their impact on
performance underscores the value of adjustable, background-specific force
feedback in improving both user experience and musical expressiveness.
Furthermore, incorporating force feedback insights from traditional
instruments offers a promising approach to immersive and natural interactions
with DMIs.

##  7 Conclusion

This study delved into enhancing DMIs with force feedback for replicating
Guzheng’s vibrato techniques, utilizing our developed Bend-aid and
TorqueTuner. By examining three haptic modes (Smooth, Detent, and Spring), we
gained insights into their impact on user experience, particularly in terms of
comfort, ease of control, flexibility, and helpfulness to the vibrato mimicry
task. We discovered a marked helpfulness and ease of control for Spring modes
over the Smooth mode. Additionally, musicians’ backgrounds influenced haptic
preferences, revealing differences between wind and string instrument players.
We emphasize the necessity for DMIs to feature customizable haptic feedback
that aligns with the specific requirements of the music and the musician. Our
findings suggest promising avenues for DMI innovation, such as incorporating
force feedback insights from traditional instruments and tailoring feedback
intensity to individual users’ physical capabilities. These developments
potentially further enhance the intuitiveness of force-feedback digital
musical instruments.

##  8 Ethical Standards

All data including questionnaire ratings and post-experiment reflections
during the experiment was approved by the McGill University Research Ethics
Board Office (file number 23-06-074) and consent by participants.

##  9 Acknowledgement

The authors extend their gratitude to João Tragtenberg, Albert-Ngabo
Niyonsenga, Fausto Borem, Maxwell Gentili-Morin, and Lejun Min for their
enthusiastic and invaluable assistance and suggestions. Additionally, we thank
all the participants for their valuable contributions and feedback. This work
was partially supported by the NSERC Discovery Grant #206719.

## References

  * [1] C. Frisson and M. M. Wanderley, “Challenges and opportunities of force feedback in music,” Arts, vol. 12, no. 4, 2023.
  * [2] M. Kirkegaard, M. Bredholt, C. Frisson, and M. Wanderley, “Torquetuner: A self contained module for designing rotary haptic force feedback for digital musical instruments,” in Proceedings of the international conference on new interfaces for musical expression, pp. 273–278, 2020.
  * [3] G. W. Young, D. Murphy, and J. Weeter, “A functional analysis of haptic feedback in digital musical instrument interactions,” Musical Haptics, pp. 95–122, 2018.
  * [4] M. S. O’Modhrain, Playing by feel: incorporating haptic feedback into computer-based musical instruments.  Stanford University, 2001.
  * [5] S. Papetti and C. Saitis, Musical haptics.  Springer Nature, 2018.
  * [6] C. Nichols, “The vbow: development of a virtual violin bow haptic human-computer interface,” in Proceedings of the 2002 conference on new interfaces for musical expression, pp. 1–4, 2002.
  * [7] I. Hwang, H. Son, and J. R. Kim, “Airpiano: Enhancing music playing experience in virtual reality with mid-air haptic feedback,” in 2017 IEEE world haptics conference (WHC), pp. 213–218, IEEE, 2017.
  * [8] M. G. Onofrei, F. Fontana, and S. Serafin, “Perceptual relevance of haptic feedback during virtual plucking, bowing and rubbing of physically-based musical resonators,” in Arts, vol. 12, p. 144, MDPI, 2023.
  * [9] P. Andrea, P. Razvan, N. C. Nilsson, N. S. Andersson, F. Fontana, N. Rolf, S. Stefania, et al., “No strings attached: Force and vibrotactile feedback in a guitar simulation,” in Proceedings of the 16th Sound & Music Computing Conference, pp. 210–216, 2019.
  * [10] A. Tanaka and A. Parkinson, “Haptic wave: A cross-modal interface for visually impaired audio producers,” in Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, pp. 2150–2161, 2016.
  * [11] S. Merchel, E. Altinsoy, and M. Stamm, “Tactile music instrument recognition for audio mixers,” in Audio Engineering Society Convention 128, Audio Engineering Society, 2010.
  * [12] Y. De Pra, F. Fontana, and S. Papetti, “Interacting with digital audio effects through a haptic knob with programmable resistance,” in 2021 24th International Conference on Digital Audio Effects (DAFx), pp. 113–120, IEEE, 2021.
  * [13] A. Colter, P. Davivongsa, D. D. Haddad, H. Moore, B. Tice, and H. Ishii, “Soundforms: Manipulating sound through touch,” in Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems, pp. 2425–2430, 2016.
  * [14] A. Karp and B. Pardo, “Hapteq: A collaborative tool for visually impaired audio producers,” in Proceedings of the 12th International Audio Mostly Conference on Augmented and Participatory Sound and Music Experiences, pp. 1–4, 2017.
  * [15] B. Verplank, M. Gurevich, and M. V. Mathews, “The plank: Designing a simple haptic controller.,” in Proceedings of the 2002 Conference on New Interfaces for Musical Expression, pp. 33–36, 2002.
  * [16] T. Beamish, K. van de Doel, K. MacLean, and S. Fels, “D’groove: A haptic turntable for digital audio control,” in Proceedings of the 2003 International Conference on Auditory Display, 2003.
  * [17] A.-N. Niyonsenga, C. Frisson, and M. M. Wanderley, “TorqueTuner: A Case Study for Sustainable Haptic Development,” in 11th Intl. Workshop on Haptic and Audio Interaction Design, HAID’22, 2022.
  * [18] A. Sarwate and J. Armitage, “powerPianoRoll.” https://github.com/AvneeshSarwate/powerPianoRoll.
  * [19] Cashhammn, “Guzheng-sample.” https://github.com/Cashhammn/Guzheng-sample.
  * [20] J. Matejka and G. Fitzmaurice, “Same stats, different graphs: Generating datasets with varied appearance and identical statistics through simulated annealing,” in Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, CHI ’17, (New York, NY, USA), p. 1290–1294, Association for Computing Machinery, 2017.
  * [21] G. Cumming, Understanding the new statistics: Effect sizes, confidence intervals, and meta-analysis.  Routledge, 2013.
  * [22] H. Cramér, Mathematical Methods of Statistics, ch. Chapter 21. The two-dimensional case, p. 282.  Princeton University Press, 1946.