# Photorealistic and Identity-Preserving Image-Based Emotion Manipulation with
Latent Diffusion Models

Ioannis Pikoulis, Panagiotis P. Filntisis, Petros Maragos
School of ECE, National Technical University of Athens, Greece
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

In this paper, we investigate the emotion manipulation capabilities of
diffusion models with “in-the-wild” images, a rather unexplored application
area relative to the vast and rapidly growing literature for image-to-image
translation tasks. Our proposed method encapsulates several pieces of prior
work, with the most important being Latent Diffusion models and text-driven
manipulation with CLIP latents. We conduct extensive qualitative and
quantitative evaluations on AffectNet, demonstrating the superiority of our
approach in terms of image quality and realism, while achieving competitive
results relative to emotion translation compared to a variety of GAN-based
counterparts. Code is released as a publicly available repo.

Figure 1: Curated examples of LDM-based emotion manipulation on the AffectNet
[30] validation set, using $T_{\textrm{DDIM}}=40$ steps, $\eta=0$, c.f.g scale
$\gamma=3.0$ and variable editing strength $t_{0}\in\\{400,500,600\\}$.
Through the proposed method we are able to manipulate facial expressions of
“in-the-wild” images while preserving realism, but also control emotion
intensity and expression variations.

##  1 Introduction

As generative adversarial networks (GANs) [15] have proven capable of
generating high-quality samples, they managed to draw a lot of attention in
the last 10 years. But recently, even more potent generative methods, like
diffusion models (DMs) [42, 18] have emerged, posing a threat to the dominance
of GANs in the production of synthetic data. DMs are quickly finding use in
both low-level and high-level vision tasks because of their incredible
generative capabilities, including but not limited to image denoising [18,
31], inpainting [11], image super-resolution [28, 25], semantic segmentation
[3, 16, 49], semantic image synthesis [36] and image-to-image translation [5,
21, 51, 38].

In this paper we explicitly focus on image-to-image translation and more
specifically on facial expression manipulation (also known as face
reenactment) on the basis of “in-the-wild” images. Photo-realistic face
reenactment can be used for entertainment purposes, human-computer
interactions, and facial animations, among other things. This area has been
attracting considerable attention both from academic and industrial research
communities and has produced stunning outcomes that broaden the scope of
inventive image editing, and content creation.

##  2 Related Work

###  2.1 GANs

The development of GANs spawned an expanding field of study in the area of
image and video synthesis. A conditional generator is used in the vast
majority of works, meaning that, in the context of emotion manipulation, the
synthesized image is dependent on either another image, a class label or any
kind of expression-specific information (_e.g_. AU activations). This makes it
possible to translate images between various domains (_i.e_., image-to-image
translation) while maintaining the original image’s content, even when
training on unpaired training data. The multi-domain StarGAN [6, 7] framework
illustrated how facial expressions may be altered in images by translating
them in accordance with the provided semantic label (_e.g_. happy, angry,
_etc_.). ExprGAN [10] employed intensity while in [26] the valence-arousal
space was utilized for continuous emotion conditioning. By employing a dataset
annotated with only categorical labels, the proposed approach of GANmut [8]
provided a way to obtain a 2D interpretable emotion-conditional label system.
Notable implementations also include GANimation [34], FACEGAN [45] and ICface
[44] which rather opt for emotion conditioning based on facial landmarks, AU
intensities.

###  2.2 Diffusion Models

A forward and reverse diffusion process make up the type of latent variable
models known as denoising diffusion probabilistic models [42, 18] (DDPM).
While progressively sampling the latent variables $\mathbf{x}_{t}$, noise is
gradually added to the data in the forward process for $t\in[T]$, forming a
Markov chain. The forward process’s individual steps are all Gaussian, _i.e_.
$q(\mathbf{x}_{t}|\mathbf{x}_{t-1}):=\mathcal{N}(\mathbf{x}_{t};\sqrt{1-\beta_{t}}\mathbf{x}_{t-1},\beta_{t}\mathbf{I})$,
where $\\{\beta_{t}\\}^{T}_{t=0}$ denotes a fixed or learned variance
schedule. The forward noising process is summarized as follows:

$\mathbf{x}_{t}=\sqrt{\bar{a}_{t}}\mathbf{x}_{0}+\sqrt{1-\bar{a}_{t}}\bm{\epsilon},\quad\bm{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$
(1)

where $\bar{a}_{s}=\prod_{s=1}^{t}a_{s}$ and $a_{s}:=1-\beta_{s}$. The
reverse, denoising process can also be parameterized in the form of Gaussian
transitions
$p_{\bm{\theta}}(\mathbf{x}_{t-1}|\mathbf{x}_{t}):=\mathcal{N}(\mathbf{x}_{t-1};\mathbf{\mu}_{\bm{\theta}}(\mathbf{x}_{t},t),\sigma_{t}\mathbf{I})$,
where $\mathbf{\mu}_{\bm{\theta}}(\mathbf{x}_{t},t)$ can be decomposed into a
linear combination of the noisy input $\mathbf{x}_{t}$ and a noise
approximation network $\bm{\epsilon}_{\bm{\theta}}(\mathbf{x}_{t},t)$.
Diffusion model training simplifies to the following optimization objective:

$\min_{\bm{\theta}}\mathbb{E}_{\mathbf{x}_{0},\bm{\epsilon},t\sim{U(1,T)}}\lVert\bm{\epsilon}-\bm{\epsilon}_{\bm{\theta}}(\sqrt{\bar{a}_{t}}\mathbf{x}_{0}+\sqrt{1-\bar{a}_{t}}\bm{\epsilon},t)\rVert_{2}^{2}$
(2)

Inferring
$\mathbf{x}_{t-1}\sim{p_{\bm{\theta}}(\mathbf{x}_{t-1}|\mathbf{x}_{t})}$ can
be performed through ancestral sampling:

$\mathbf{x}_{t-1}=\frac{1}{\sqrt{a_{t}}}\bigg{(}\mathbf{x}_{t}-\frac{1-a_{t}}{\sqrt{1-\bar{a}_{t}}}\bm{\epsilon}_{\bm{\theta}}(\mathbf{x}_{t},t)\bigg{)}+\sigma_{t}\bm{\epsilon}$
(3)

where $\bm{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$.

Song _et al_. [43] proposed denoising diffusion implicit models (DDIM),
equipped with a non-Markovian diffusion process and jump-step sampling. DDIM
shares the same forward marginals with DDPM, but differs in terms of the
sampling process:

$\mathbf{x}_{t-1}=\sqrt{\bar{a}_{t-1}}\mathbf{f}(\mathbf{x}_{t},t)+\sqrt{1-\bar{a}_{t-1}-\sigma_{t}^{2}}\bm{\epsilon}_{\bm{\theta}}(\mathbf{x}_{t},t)+\sigma_{t}\bm{\epsilon}$

(4)

where $\mathbf{f}_{\bm{\theta}}$ denotes the prediction of the denoised
observation $\mathbf{x}_{0}$ at $t$, given $\mathbf{x}_{t}$ and
$\bm{\epsilon}_{\bm{\theta}}(\mathbf{x}_{t},t)$:

$\mathbf{f}_{\bm{\theta}}(\mathbf{x}_{t},t):=\frac{\mathbf{x}_{t}-\sqrt{1-\bar{a}_{t}}\bm{\epsilon}_{\bm{\theta}}(\mathbf{x}_{t},t)}{\sqrt{\bar{a}_{t}}}$
(5)

Considering accelerated generation with $\mathbf{\tau}$ being an increasing
subsequence of $[1,\dots,T]$, the variance hyperparameter is often
parameterized as
$\sigma_{\tau_{i}}(\eta)=\eta\sqrt{(1-\bar{a}_{\tau_{i-1}})/(1-\bar{a}_{\tau_{i}})}\sqrt{1-\bar{a}_{\tau_{i}}/\bar{a}_{\tau_{i-1}}}$.
Especially when $\eta=0$, both the forward and reverse processes become
deterministic, enabling fast inversion of the noised input variables. In this
case, the resulting model becomes an implicit probabilistic model [29].

In an attempt to increase speed and computational efficiency, latent diffusion
models (LDM), introduced by Rombach _et al_. [36], proposed transferring both
diffusion and sampling processes from image space to the latent space of an
autoencoder. Given an RGB source image $\mathbf{x}_{0}\in\mathbb{R}^{H\times
W\times 3}$, an encoder $\mathcal{E}$ encodes it into a latent representation
$\mathbf{z}_{0}=\mathcal{E}(\mathbf{x})\in\mathbb{R}^{h\times{w}\times{c}}$. A
decoder $\mathcal{D}$ is jointly-trained to recover the initial images from
their respective latents. The encoder downsamples input images by a factor
$f=H/h=W/w=2^{m},m\in\mathbb{N}^{+}$. Subsequently, diffusion can be applied
on the produced latents via either DDPM of DDIM, resulting in noisy latents
$\mathbf{z}_{t}$, while sampling can still be described with either Eq. 3 or
Eq. 4, after substituting images $\mathbf{x}_{(\cdot)}$ with their respective
latents $\mathbf{z}_{(\cdot)}$. Moreover, the noise approximation network can
be conditioned on additional user inputs $y$, _i.e_.
$\bm{\epsilon}_{\bm{\theta}}(\mathbf{z}_{t},t,\bm{\tau}_{\bm{\theta}}(y))$,
where $\bm{\tau}_{\bm{\theta}}$ denotes a domain-specific encoder that
projects $y$ to an intermediate representation
$\bm{\tau}_{\bm{\theta}}(y)\in\mathbb{R}^{M\times{d_{\tau}}}$.

Emotion manipulation with diffusion models currently constitutes an unexplored
research area, with few available pieces of related literature. Kim _et al_.
[22] introduced DiffFace, a face swapping framework capable of transferring
target attributes such as gaze, structure and pose, while preserving source
identities. Wang _et al_. [47] tackled the head-swapping task with HS-
Diffusion. More recently, Li _et al_. [24] proposed StyO for one-shot face
stylization. Text-guided image manipulation and editing [21, 5, 32] has also
demonstrated competitive results in zero-shot settings.

##  3 Method

We trained a VQGAN [12] as the first stage of our LDM. The LDM is trained
using classifier-free guidance [19], dropout probability
$p_{\textrm{uncond}}=0.2$ and $T_{\textrm{DDPM}}=1000$ steps. For more
details, we refer to the Suppl. Mat.

###  3.1 Baseline

The baseline methodology towards image-based emotion manipulation follows the
paradigm of LDEdit [5], adapted specifically for the task of emotion
translation. In our case, the transformer tokenizer is replaced with a class-
embedder that maps each emotion label to a learnable embedding vector
$\bm{\tau}_{\bm{\theta}}(y)\in\mathbb{R}^{1\times 512}$. More specifically,
the backbone of our approach resides in the near cycle-consistency induced by
the deterministic forward and backward DDIM processes $(\eta=0)$. This is
further demonstrated by the following analysis. The deterministic DDIM
sampling process described in section Eq. 4, in the context of LDMs, can be
rewritten in the following form:

$\sqrt{\frac{1}{\bar{a}_{t-1}}}\mathbf{z}_{t-1}-\sqrt{\frac{1}{\bar{a}_{t}}}\mathbf{z}_{t}=\Bigg{(}\sqrt{\frac{1}{\bar{a}_{t-1}}-1}-\sqrt{\frac{1}{\bar{a}_{t}}-1}\Bigg{)}\bm{\epsilon}_{\bm{\theta}}(\mathbf{z}_{t},t)$

(6)

By setting $\mathbf{y}_{t}:=\sqrt{1/\bar{a}_{t}}\mathbf{z}_{t}$ and
$p_{t}:=\sqrt{1/\bar{a}_{t}-1}$, Eq. 6 can be written as:

$\mathbf{y}_{t-1}-\mathbf{y}_{t}=(p_{t-1}-p_{t})\bm{\epsilon}_{\bm{\theta}}(\mathbf{z}_{t},t)\Rightarrow\mathrm{d}\mathbf{y}_{t}=\bm{\epsilon}_{\bm{\theta}}(\mathbf{z}_{t},t)\mathrm{d}p_{t}$

(7)

The last equation describes an ODE that moves backward in terms of the
diffusion time index $t$. Reversal of this ODE leads to the respective forward
deterministic DDIM process.

Now, given a source image-label pair
$(\mathbf{x}_{\textrm{src}},y_{\textrm{src}})$, its translation w.r.t. a
target emotion $y_{\textrm{trg}}$ can be achieved through the following pair
of equations:

$\mathbf{z}_{t+1}$
$=\sqrt{\bar{a}_{t+1}}\mathbf{f}_{\bm{\theta}}(\mathbf{z}_{t},t,\bm{\tau}_{\bm{\theta}}(y_{\textrm{src}}))+\sqrt{1-\bar{a}_{t+1}}\mathbf{\epsilon}_{\bm{\theta}}(\mathbf{z}_{t},t,\bm{\tau}_{\bm{\theta}}(y_{\textrm{src}}))$
(8) $\mathbf{\hat{z}}_{t-1}$
$=\sqrt{\bar{a}_{t-1}}\mathbf{f}_{\bm{\theta}}(\mathbf{\hat{z}}_{t},t,\bm{\tau}_{\bm{\theta}}(y_{\textrm{trg}}))+\sqrt{1-\bar{a}_{t-1}}\mathbf{\epsilon}_{\bm{\theta}}(\mathbf{\hat{z}}_{t},t,\bm{\tau}_{\bm{\theta}}(y_{\textrm{trg}}))$
(9)

Given the input source image $\mathbf{x}_{\textrm{src}}$, the first stage
encoder $\mathcal{E}$ transforms it into its corresponding latent
representation $\mathbf{z}_{0}$. Then deterministic forward diffusion is
performed up to timestep $t_{0}<T_{\textrm{DDPM}}$, conditioned on the source
emotion label $y_{\textrm{src}}$, denoted as $\mathbf{z}_{t_{0}}$, according
to Eq. 8. The reverse process conditioned on the target emotion label
$y_{\textrm{trg}}$ initiates from the same noised latent code
$\mathbf{z}_{t_{0}}$ with the aim of reconstructing $\hat{\mathbf{z}}_{0}$.
Passing the manipulated latent code $\hat{\mathbf{z}}_{0}$ through the decoder
$\mathcal{D}$ results in the reconstructed target image
$\mathbf{x}_{\textrm{gen}}$. DDIM processes can be significantly sped up by
using fewer discretization steps $\\{\tau_{s}\\}_{s=1}^{T_{\textrm{DDIM}}}$
evenly selected in the range $[1,t_{0}]$ such that $\tau_{1}=1$ and
$\tau_{T_{\textrm{DDIM}}}=t_{0}$. The effects of varying $t_{0}$ are
illustrated in Fig. 1. Lowering the number of sampling steps is essential for
speeding up experiments and in practice $T_{\textrm{DDIM}}\geq 20$ provides
satisfactory results in the context of image-to-image translation.

###  3.2 CLIP-guided finetuning

In order to effectively extract knowledge from CLIP [35] in the context of
image-translation, two different losses have been proposed: a global target
loss [33], and local directional loss [14]. The global CLIP loss tries to
minimize the cosine distance in CLIP space between the generated image and a
given target text. On the other hand, the local directional loss is designed
to alleviate issues of the latter such as low diversity and susceptibility to
adversarial attacks. The local directional CLIP loss induces the direction
between the embeddings of the reference and generated images to be aligned
with the direction between the embeddings of a pair of reference and target
texts in the CLIP space as follows:

$\displaystyle\mathcal{L}_{\textrm{dir}}=1-\textrm{cos}\bigl{\langle}$
$\displaystyle\textrm{CLIP}_{\textrm{img}}(\mathbf{x}_{\textrm{gen}})-\textrm{CLIP}_{\textrm{img}}(\mathbf{x}_{\textrm{src}}),$
(10)
$\displaystyle\textrm{CLIP}_{\textrm{text}}(y_{\textrm{trg}})-\textrm{CLIP}_{\textrm{text}}(y_{\textrm{src}})\bigr{\rangle}$

We adapt DiffusionCLIP [21] to the context of image-based emotion translation.
At the reverse path, our pre-trained LDM is being finetuned to generate
samples driven by target texts $y_{\textrm{trg}}$. Apart from the CLIP
component, the overall finetuning objective is comprised of an identity
preservation loss, implemented as the cosine distance in the embedding space
of a pre-trained IR-SE50 [17, 20] model coupled with ArcFace [9], along with a
$\ell_{2}$ loss in pixel space:

$\mathcal{L}=\lambda_{\textrm{dir}}\mathcal{L}_{\textrm{dir}}+\lambda_{\textrm{id}}\mathcal{L}_{\textrm{id}}(\mathbf{x}_{\textrm{gen}},\mathbf{x}_{\textrm{src}})+\lambda_{\ell_{2}}\lVert\mathbf{x}_{\textrm{gen}}-\mathbf{x}_{\textrm{src}}\rVert_{2}^{2}$
(11)

Our implementation differs from the original one in the following two ways: i)
Instead of DDPM, we use LDM as our backbone. ii) The latter did not take
advantage of source class-label information when applying the directional CLIP
loss, as the majority of experiments were performed in unconditional settings.
Moreover, taking advantage of the available class labels, we allowed images to
be manipulated in effectively 36 different directions, that is
‘neutral’$\rightarrow\\{$‘happy’, ‘sad’, ‘surprised’, ‘scared’, ‘disgusted’,
‘angry’$\\}$ and $y_{\textrm{src}}\rightarrow{y_{\textrm{trg}}}$, with
$y_{\textrm{trg}}\in\\{$‘happy’, ‘sad’, ‘surprised’, ‘scared’, ‘disgusted’,
‘angry’$\\}$ and $y_{\textrm{src}}\neq y_{\textrm{trg}}$ (contempt was mapped
to neutral). As our LDM was trained using c.f.g., we are able to couple CLIP
guidance with different values of unconditional guidance scale $\gamma$.

##  4 Experiments

###  4.1 Dataset & Evaluation Metrics

We train and evaluate our models on AffectNet [30]. AffectNet is by far the
largest database of facial expressions that provides both categorical and VA
annotations, in-the-wild. In the context of image-to-image translation, PSNR,
SSIM [48] and LPIPS [50] constitute the most commonly used evaluation metrics
relative to image quality. We also employ HSEmotion [39, 41, 40], a SOTA
lightweight emotion recognition framework for evaluating the emotion transfer
capabilities of our models in terms of classification accuracy. Lastly, we
evaluate models based on subject identity preservation using feature cosine
similarity (CSIM) in the embedding space of a CosFace [46] model, pre-trained
on the Glint360K [1] dataset.

Method $y_{\textrm{target}}$ | Happy | Sad | Surprised
---|---|---|---
Accuracy$\uparrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | CSIM$\uparrow$ | Accuracy$\uparrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | CSIM$\uparrow$ | Accuracy$\uparrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | CSIM$\uparrow$
Groundtruth [30] | 0.758 | – | – | – | – | 0.638 | – | – | – | – | 0.606 | – | – | – | –
GANimation [34] | 0.645 | 24.07 | 0.816 | 0.099 | 0.547 | 0.212 | 24.52 | 0.830 | 0.097 | 0.582 | 0.360 | 23.82 | 0.817 | 0.101 | 0.559
StarGAN v2 [7] | 0.958 | 17.46 | 0.659 | 0.165 | 0.441 | 0.569 | 18.30 | 0.712 | 0.149 | 0.593 | 0.761 | 17.99 | 0.678 | 0.160 | 0.503
GANmut [8] | 0.879 | 21.42 | 0.809 | 0.106 | 0.663 | 0.888 | 23.85 | 0.857 | 0.094 | 0.755 | 0.829 | 22.43 | 0.810 | 0.112 | 0.675
GGANmut [8] | 0.934 | 21.91 | 0.819 | 0.103 | 0.717 | 0.986 | 22.13 | 0.802 | 0.115 | 0.653 | 0.970 | 22.37 | 0.777 | 0.121 | 0.610
Ours | 0.872 | 25.80 | 0.841 | 0.090 | 0.743 | 0.774 | 25.71 | 0.837 | 0.095 | 0.778 | 0.658 | 25.54 | 0.838 | 0.094 | 0.716
Ours w/ CGF | 0.883 | 24.30 | 0.813 | 0.098 | 0.744 | 0.875 | 24.72 | 0.822 | 0.093 | 0.794 | 0.752 | 23.42 | 0.797 | 0.113 | 0.721
| Scared | Disgusted | Angry
Groundtruth [30] | 0.666 | – | – | – | – | 0.646 | – | – | – | – | 0.514 | – | – | – | –
GANimation [34] | 0.314 | 23.68 | 0.814 | 0.105 | 0.556 | 0.271 | 24.13 | 0.819 | 0.103 | 0.549 | 0.287 | 24.80 | 0.833 | 0.096 | 0.580
StarGAN v2 [7] | 0.860 | 17.76 | 0.676 | 0.162 | 0.507 | 0.879 | 18.10 | 0.691 | 0.154 | 0.528 | 0.666 | 18.14 | 0.689 | 0.154 | 0.509
GANmut [8] | 0.932 | 23.39 | 0.841 | 0.102 | 0.721 | 0.877 | 22.64 | 0.815 | 0.113 | 0.663 | 0.856 | 21.57 | 0.813 | 0.106 | 0.678
GGANmut [8] | 0.967 | 20.13 | 0.763 | 0.135 | 0.556 | 0.987 | 21.68 | 0.772 | 0.128 | 0.562 | 0.969 | 21.77 | 0.767 | 0.133 | 0.590
Ours | 0.764 | 24.89 | 0.824 | 0.103 | 0.770 | 0.676 | 25.50 | 0.835 | 0.100 | 0.707 | 0.710 | 25.66 | 0.840 | 0.094 | 0.714
Ours w/ CGF | 0.764 | 24.83 | 0.826 | 0.096 | 0.764 | 0.450 | 24.87 | 0.822 | 0.089 | 0.714 | 0.886 | 24.18 | 0.796 | 0.106 | 0.735

Table 1: Quantitative comparison among our method and GAN-based
implementations for image-based emotional manipulation on the AffectNet
validation set. Aggregated results can be found in the Suppl. Mat.

Figure 2: Qualitative comparison among our method and GAN-based
implementations for image-based emotional manipulation on curated examples
from the AffectNet validation set.

###  4.2 Results

As at the time of writing, no applicable diffusion-based model with a publicly
available codebase exists, we shift our attention towards well-grounded GAN-
based implementations, with which we will compare our findings, in both a
quantitative and qualitative level. These models are: GANimation [34], StarGAN
v2 [7] and GANmut [8] (linear and Gaussian models, denoted as GANmut and
GGANmut, respectively). For details about used code and training
specifications, please refer to our Suppl. Mat.

#### Objective Results

In Tab. 1 we provide quantitative comparisons among the three aforementioned
GAN-based implementations and our diffusion-based ones, with and without CLIP-
guided finetuning. Across all emotions, our models consistently (with few
exceptions) surpass all GAN-based counterparts in terms of image quality and
subject identity preservation. Our method also achieves higher emotion
classification than GANimation and comparable with StarGAN v2 but is
outperformed by GGANmut. This can be partly explained by the fact the the
latter fits multiple Gaussians with the aim of acquiring a more accurate and
rich representation of the conditional emotional latent space, compared to
discrete emotion labels.

#### User Studies

We conducted two user studies to evaluate the realism and emotion accuracy of
our approach, as assessed by human users, the results of which can be found in
Tab. 2. In the first study, we involved 24 participants, who were presented
with 28 pairwise comparisons between all methods (including original images)
and were asked to choose the most realistic one. As demonstrated by the
results, our method’s manipulated images were perceived significantly
($p<0.01$ with binomial test) more realistic compared to the other methods.
However, as anticipated, they were not as realistic as the actual images.

In the second study, 27 participants were shown 30 images from each method and
asked to identify the displayed emotion from a list of six emotions. The
bottom half of Tab. 2 presents the corresponding accuracy results, which are
closely aligned with the previous objective results. Our approach achieves
performance on-par with StarGAN v2 and surpasses GANimation, although GANmut
achieves the highest accuracy. Notably, the accuracy in the original images of
AffectNet is similar to that of StarGAN v2 and our method. This observation
can potentially be attributed to GANmut generating more ”exaggerated” emotions
that deviate from the emotion distribution in AffectNet, sacrificing realism
in the process. Finally, a qualitative comparison is presented in Fig. 2,
where the superiority of our models in terms of image quality and realism
becomes more evident. For further experimental results and details, we kindly
direct the reader to our Suppl. Mat.

| Realism
---|---
Ours vs. | GANimation | GANmut | StarGAN v2 | Groundtruth
| 0.69 (116) | 0.31 (52) | 0.68 (114) | 0.32 (54) | 0.61 (103) | 0.39 (65) | 0.28 (46) | 0.72 (122)
| Emotion Recognition
0.57 (92/162) | 0.38 (61/162) | 0.73 (118/162) | 0.57 (92/162) | 0.59 (96/162)

Table 2: Head-to-head subjective study regarding realism (top half) and
emotion translation accuracy (bottom half). Numbers in parentheses denote
absolute numbers.

##  5 Conclusion

In this paper, we presented our method for photorealistic and identity
preserving emotion manipulation of images with the use of diffusion models.
Our extensive objective and subjective evaluations on AffectNet showcase the
superiority of method in terms of quality, while achieving competitive results
regarding emotion translation accuracy against both conventional and dedicated
GAN-based methodologies. Hopefully, our work can serve as a breeding ground
for further research in the field of diffusion-based face reenactment.

## References

  * [1] Xiang An, Xuhan Zhu, Yuan Gao, Yang Xiao, Yongle Zhao, Ziyong Feng, Lan Wu, Bin Qin, Ming Zhang, Debing Zhang, et al.  Partial FC: Training 10 million identities on a single machine.  In ICCV, 2021.
  * [2] Tadas Baltrusaitis, Amir Zadeh, Yao Chong Lim, and Louis-Philippe Morency.  OpenFace 2.0: Facial behavior analysis toolkit.  In IEEE FG, 2018.
  * [3] Dmitry Baranchuk, Andrey Voynov, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko.  Label-efficient semantic segmentation with diffusion models.  In ICLR, 2022.
  * [4] Adrian Bulat and Georgios Tzimiropoulos.  How far are we from solving the 2D & 3D face alignment problem? (And a dataset of 230,000 3D facial landmarks).  In ICCV, 2017.
  * [5] Paramanand Chandramouli and Kanchana Vaishnavi Gandikota.  LDEdit: Towards generalized text guided image manipulation via latent diffusion models.  In BMVC, 2022.
  * [6] Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Woo Ha, Sunghun Kim, and Jaegul Choo.  StarGAN: Unified generative adversarial networks for multi-domain image-to-image translation.  In CVPR, 2018.
  * [7] Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Ha.  StarGAN v2: Diverse image synthesis for multiple domains.  In CVPR, 2020.
  * [8] Stefano d’Apolito, Danda Pani Paudel, Zhiwu Huang, Andrés Romero, and Luc Van Gool.  GANmut: Learning interpretable conditional space for gamut of emotions.  In CVPR, 2021.
  * [9] Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou.  ArcFace: Additive angular margin loss for deep face recognition.  In CVPR, 2019.
  * [10] Hui Ding, Kumar Sricharan, and Rama Chellappa.  ExprGAN: Facial expression editing with controllable expression intensity.  In AAAI, 2018.
  * [11] Patrick Esser, Robin Rombach, Andreas Blattmann, and Bjorn Ommer.  ImageBART: Bidirectional context with multinomial diffusion for autoregressive image synthesis.  NeurIPS, 2021.
  * [12] Patrick Esser, Robin Rombach, and Bjorn Ommer.  Taming transformers for high-resolution image synthesis.  In CVPR, 2021.
  * [13] C Fabian Benitez-Quiroz, Ramprakash Srinivasan, and Aleix M Martinez.  EmotioNet: An accurate, real-time algorithm for the automatic annotation of a million facial expressions in the wild.  In CVPR, 2016.
  * [14] Rinon Gal, Or Patashnik, Haggai Maron, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or.  StyleGAN-NADA: CLIP-guided domain adaptation of image generators.  ACM TOG, 41:1–13, 2022.
  * [15] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In NeurIPS, 2014.
  * [16] Alexandros Graikos, Nikolay Malkin, Nebojsa Jojic, and Dimitris Samaras.  Diffusion models as plug-and-play priors.  In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, NeurIPS, 2022.
  * [17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, 2016.
  * [18] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  NeurIPS, 2020.
  * [19] Jonathan Ho and Tim Salimans.  Classifier-free diffusion guidance.  In NeurIPS Workshop on Deep Generative Models and Downstream Applications, 2021.
  * [20] Jie Hu, Li Shen, and Gang Sun.  Squeeze-and-excitation networks.  In CVPR, 2018.
  * [21] Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye.  DiffusionCLIP: Text-guided diffusion models for robust image manipulation.  In CVPR, 2022.
  * [22] Kihong Kim, Yunho Kim, Seokju Cho, Junyoung Seo, Jisu Nam, Kychul Lee, Seungryong Kim, and KwangHee Lee.  DiffFace: Diffusion-based face swapping with facial guidance.  arXiv preprint arXiv:2212.13344, 2022.
  * [23] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In ICLR, 2015.
  * [24] Bonan Li, Zicheng Zhang, Xuecheng Nie, Congying Han, Yinhan Hu, and Tiande Guo.  StyO: Stylize your face in only one-shot.  arXiv preprint arXiv:2303.03231, 2023.
  * [25] Haoying Li, Yifan Yang, Meng Chang, Shiqi Chen, Huajun Feng, Zhihai Xu, Qi Li, and Yueting Chen.  SRDiff: Single image super-resolution with diffusion probabilistic models.  Neurocomputing, 479:47–59, 2022.
  * [26] Alexandra Lindt, Pablo Barros, Henrique Siqueira, and Stefan Wermter.  Facial expression editing with continuous emotion labels.  In IEEE FG, 2019.
  * [27] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In ICLR, 2019.
  * [28] Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool.  RePaint: Inpainting using denoising diffusion probabilistic models.  In CVPR, 2022.
  * [29] Shakir Mohamed and Balaji Lakshminarayanan.  Learning in implicit generative models.  arXiv preprint arXiv:1610.03483, 2016.
  * [30] Ali Mollahosseini, Behzad Hasani, and Mohammad H Mahoor.  AffectNet: A database for facial expression, valence, and arousal computing in the wild.  IEEE Trans. Affect. Comput., 10:18–31, 2017.
  * [31] Alexander Quinn Nichol and Prafulla Dhariwal.  Improved denoising diffusion probabilistic models.  In ICML, 2021.
  * [32] Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yijun Li, Jingwan Lu, and Jun-Yan Zhu.  Zero-shot image-to-image translation.  arXiv preprint arXiv:2302.03027, 2023.
  * [33] Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski.  StyleCLIP: Text-driven manipulation of StyleGAN imagery.  In ICCV, 2021.
  * [34] Albert Pumarola, Antonio Agudo, Aleix M Martinez, Alberto Sanfeliu, and Francesc Moreno-Noguer.  GANimation: Anatomically-aware facial animation from a single image.  In ECCV, 2018.
  * [35] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In ICML, 2021.
  * [36] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In CVPR, 2022.
  * [37] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In MICCAI, 2015.
  * [38] Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi.  Palette: Image-to-image diffusion models.  In ACM SIGGRAPH, 2022.
  * [39] Andrey V Savchenko.  Facial expression and attributes recognition based on multi-task learning of lightweight neural networks.  In IEEE SISY, 2021.
  * [40] Andrey V Savchenko.  Video-based frame-level facial analysis of affective behavior on mobile devices using EfficientNets.  In CVPRW, 2022.
  * [41] Andrey V Savchenko, Lyudmila V Savchenko, and Ilya Makarov.  Classifying emotions and engagement in online learning based on a single facial expression recognition neural network.  IEEE Trans. Affect. Comput., 13:2132–2143, 2022.
  * [42] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  In ICML, 2015.
  * [43] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  In ICLR, 2021.
  * [44] Soumya Tripathy, Juho Kannala, and Esa Rahtu.  ICface: Interpretable and controllable face reenactment using GANs.  In WACV, 2020.
  * [45] Soumya Tripathy, Juho Kannala, and Esa Rahtu.  FACEGAN: Facial attribute controllable reenactment GAN.  In WACV, 2021.
  * [46] Hao Wang, Yitong Wang, Zheng Zhou, Xing Ji, Dihong Gong, Jingchao Zhou, Zhifeng Li, and Wei Liu.  CosFace: Large margin cosine loss for deep face recognition.  In CVPR, 2018.
  * [47] Qinghe Wang, Lijie Liu, Miao Hua, Qian He, Pengfei Zhu, Bing Cao, and Qinghua Hu.  HS-Diffusion: Learning a semantic-guided diffusion model for head swapping.  arXiv preprint arXiv:2212.06458, 2022.
  * [48] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli.  Image quality assessment: From error visibility to structural similarity.  IEEE Trans. Image Process., 13:600–612, 2004.
  * [49] Julia Wolleb, Robin Sandkühler, Florentin Bieder, Philippe Valmaggia, and Philippe C Cattin.  Diffusion models for implicit image segmentation ensembles.  In MIDL, 2022.
  * [50] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In CVPR, 2018.
  * [51] Min Zhao, Fan Bao, Chongxuan Li, and Jun Zhu.  EGSDE: Unpaired image-to-image translation via energy-guided stochastic differential equations.  In NeurIPS, 2022.

Photorealistic and Identity-Preserving Image-Based Emotion Manipulation with
Latent Diffusion Models – Supplementary Material

##  A Model Specifications

###  A.1 Face Cropping & Alignment

The creators of the AffectNet dataset have provided precomputed face bounding
boxes as well as facial landmark keypoint coordinates per image. However, we
rather opted for manually locating both the bounding boxes as well as the
facial landmarks from scratch, using FAN [4]. Before feeding the original
images to FAN, we first resize them to $256^{2}$ pixels in order to avoid
running out of GPU memory. FAN locates the 2D coordinates for 68 facial
landmarks. Based on those landmarks, bounding boxes of size $224^{2}$ are
center-cropped and then aligned by estimating the affine transform required to
warp the set of detected facial keypoints to their canonical coordinates. Any
empty image regions resulting from rotating the original image are dealt with
by reflecting the image near the edges.

###  A.2 First Stage

The used VQGAN [12] follows the U-Net [37] architecture consisting of a series
of downsampling and upsampling residual blocks. The number of residual blocks
per spatial resolution level was set equal to two. The number of basic
channels was set equal to 128. The given input images were first resized to a
resolution of $128^{2}$ pixels. Thus, given input images
$\mathbf{x}\in\mathbb{R}^{128\times 128\times 3}$, the encoder compresses the
latter by a factor $f=4$, resulting in latents
$\hat{\mathbf{z}}\in\mathbb{R}^{32\times 32\times 3}$. Each latent undergoes
element-wise quantization resulting in a collection of $32^{2}$ codebook
entries $\bm{z}_{ij}\in\mathbb{R}^{3},i,j\in[32]$. The cardinality of the
entire codebook is set equal to $|\mathcal{Z}|=16384$. Moreover, the channel
multipliers for the 2 intermediate spatial resolutions is set equal to 2 and 4
respectively. The last spatial resolution level also includes a spatial
attention block. The VQGAN was trained for 6 epochs with the Adam [23]
optimizer, an initial learning rate of $3.6\times 10^{-5}$ and a batch size of
8, reaching a reconstruction loss equal to 0.138 (sum of $\ell_{1}$ and LPIPS
distance) on the AffectNet [30] validation set. The decoder’s structure is
exactly the same as the encoder’s, but with the order of the blocks being
reversed, resulting in approximately 61M trainable parameters.

###  A.3 Denoising Network

The underlying LDM U-Net architecture features two residual blocks per spatial
resolution level, with the number of basic channels being equal to 160. Input
latents are further downsampled by a factor of $f=4$. Attention blocks are
placed at every spatial resolution level, i.e. $32^{2},16^{2},8^{2}$ and the
channel multipliers are set equal to 1, 2 and 4 respectively. Each attention
block is implemented according to original cross-attention mechanism described
in [36]. The class label encoder is implemented as a single learnable
embedding layer with a dimensionality of 512, mapping class labels $y$ to
$\bm{\tau}_{\bm{\theta}}(y)\in\mathbb{R}^{1\times 512}$. All the above led to
a total number of 156M trainable parameters.

In order to better control the intensity of the depicted emotions during the
manipulation phase, we were obliged to train the LDM using classifier-free
guidance [19]. During training and with a probability
$p_{\textrm{uncond}}=0.2$, the input class labels are randomly dropped and
replaced with a $\varnothing$ (null) label, effectively raising the total
number of class labels to nine. Then sampling is performed using the following
linear combination of the conditional and unconditional score estimates:

$\tilde{\bm{\epsilon}}_{\bm{\theta}}(\mathbf{z}_{t},\bm{\tau}_{\bm{\theta}}(y))=(1-\gamma)\bm{\epsilon}_{\bm{\theta}}(\mathbf{z}_{t},\bm{\tau}_{\bm{\theta}}(\varnothing))+\gamma\bm{\epsilon}_{\bm{\theta}}(\mathbf{z}_{t},\bm{\tau}_{\bm{\theta}}(y))$

(12)

The LDM was trained for around 30 epochs with the AdamW [27] optimizer, an
initial learning rate of $2.4\times 10^{-5}$ and a batch size of 24. We used a
linear noise schedule and $T_{\textrm{DDPM}}=1000$ diffusion steps.

##  B Manipulation Hyperparameters

Important hyperparameters of both the baseline and tuned manipulation
processes are the stochasticity parameter $\eta$, the editing strength
$t_{0}$, the number of steps $T_{\textrm{DDIM}}$ and the unconditional
guidance scale $\gamma$. We begin by setting $\eta=0$ for all of the upcoming
experiments. In general, values $\eta>0$ can produce diverse outputs, reducing
the consistency with the original image. In the context of facial expression
manipulation, identity and facial attribute preservation of the depicted
subjects is of primary importance and as the current baseline methodology does
not include any other form of supervision upon the latter, we find ourselves
constrained to use $\eta=0$. Moreover, we chose to name $t_{0}$ as the editing
strength hyperparameter due to the observation that the higher its value, the
more prominent the manipulation effect becomes, compared to the original
image. As we chose to train the LDM using c.f.g., we have one extra degree of
freedom regarding the intensity of the produced manipulation in the form of
the unconditional guidance scale $\gamma$.

Figure 3: Curated examples of latent emotion manipulation on the AffectNet
validation set, using $T_{\textrm{DDIM}}=40$ DDIM steps, $\eta=0$, editing
strength $t_{0}=500$ and c.f.g scale $\gamma=3.0$.

Figure 4: Curated examples of LDM-based emotion manipulation on the AffectNet
validation set, using $T_{\textrm{DDIM}}=40$ steps, $\eta=0$, c.f.g scale
$\gamma=3.0$ and variable editing strength $t_{0}\in\\{400,500,600\\}$.

Figure 5: Curated examples of LDM-based emotion manipulation on the AffectNet
validation set, using $T_{\textrm{DDIM}}=40$ steps, $\eta=0$, editing strength
$t_{0}=500$ and variable c.f.g. scale $\gamma\in\\{1.0,2.0,3.0,4.0,5.0\\}$.

Fig. 3 illustrates curated examples of emotion manipulation on images from the
AffectNet validation set, using $T_{\textrm{DDIM}}=20$, $\gamma=3.0$ and
$t_{0}=500$. The leftmost image in each column is an original sample drawn
from the dataset, while the other six columns include the manipulation results
for each one of the 6 basic expressions of happiness, sadness, surprise, fear,
disgust and anger. We can immediately notice that the results in the ‘happy’,
‘sad’ and ‘angry’ columns are overall satisfactory, the emotions of surprise,
fear and disgust are more difficult to convey. This can be partly justified by
the fact that the aforementioned emotions are underrepresented in the
available AffectNet training set. Fig. 4 illustrates the effect of varying the
editing strength $t_{0}\in\\{400,500,600\\}$ in the case of all six different
target emotions, while $\gamma$ and $T_{\textrm{DDIM}}$ are fixed. One can
easily notice that $t_{0}=400$ is not sufficient for producing emotionally
distinguishable results. On the other end, a value of $t_{0}=600$ produces
better manipulation results but at the cost of subject identity distortion.
Furthermore, Fig. 5 illustrates the effect of varying the unconditional
guidance scale $\gamma\in\\{1.0,2.0,3.0,4.0,5.0\\}$ in the case of all six
different target emotions, while $t_{0}$ and $T_{\textrm{DDIM}}$ are fixed. It
is evident that in the specific setting of $t_{0}=500$, lower values of
$\gamma$ have little to no effect in changing the emotion of a given subject.
Better results can be obtained with $\gamma\in\\{4.0,5.0\\}$ with minimal
distortion of peripheral facial features and subject identity.

###  B.1 Ablation Study

Detailed quantitative results are presented in Tab. 3 and Tab. 4. The quality
of the produced images and consistency with the original ones is measured, as
usual, in terms of PSNR, SSIM and LPIPS111PSNR, SSIM and LPIPS are calculated
using piq.. We also employ HSEmotion222Ranked #1 in 8-emotion classification
accuracy on AffectNet, according to https://paperswithcode.com/sota/facial-
expression-recognition-on-affectnet. [39, 40, 41], a state-of-the-art
lightweight emotion recognition framework for evaluating the emotion transfer
capabilities of our baseline methodology. In Tab. 3, we notice that the most
difficult expressions, as far as emotion transfer is concerned, are (according
to HSEmotion) surprise and disgust, while the easiest one is happiness. More
specifically, a maximum emotion accuracy of 0.961 is achieved in terms of
happiness, while 0.731 and 0.734 for surprise and disgust, respectively. As
expected, the highest recognition accuracy for all six emotions is achieved
using $\gamma=5.0$ and $t_{0}=600$. This setting also results in the lowest
image quality, that regardless of the target emotion, revolves around 22.00 dB
PSNR, 0.730 SSIM and 0.150 LPIPS. Tab. 4, quantifies the effect of varying the
number of steps $T_{\textrm{DDIM}}$ in terms of the three aforementioned image
quality metrics and recognition accuracy. For $t_{0}=400$ and regardless of
the target emotion, increasing $T_{\textrm{DDIM}}$ leads to either no change
or a small drop in recognition performance. This is not the case for higher
$t_{0}\in\\{500,600\\}$. PSNR always increases in accordance with the increase
in sampling steps. SSIM behaves differently, as it increases along with
$T_{\textrm{DDIM}}$ when $t_{0}\in\\{400,500\\}$ and decreases when
$t_{0}=600$. A similar counter-intuitive behavior is noticed in the case of
LPIPS which gradually increases, as the number of DDIM steps increases, when
$t_{0}\in\\{500,600\\}$.

$y_{\textrm{target}}$ | c.f.g. Scale ($\gamma$) | Str. ($t_{0}$) | Image Quality | Emotion Recognition Accuracy w/ HSEmotion [39, 40, 41]
---|---|---|---|---
PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | Neutral | Happy | Sad | Surprised | Disgusted | Disgusted | Angry
Happiness | 1.0 | 400 | $\numprint{35.249}$ | 0.966 | 0.013 | 0.24 | 0.248 | 0.076 | 0.146 | 0.093 | 0.075 | 0.122
2.0 | $\numprint{33.150}$ | 0.95 | 0.021 | 0.254 | 0.344 | 0.047 | 0.154 | 0.058 | 0.052 | 0.092
3.0 | $\numprint{31.706}$ | 0.937 | 0.029 | 0.244 | 0.447 | 0.028 | 0.148 | 0.035 | 0.032 | 0.066
4.0 | $\numprint{30.585}$ | 0.924 | 0.037 | 0.216 | 0.537 | 0.018 | 0.138 | 0.023 | 0.021 | 0.047
5.0 | $\numprint{29.659}$ | 0.911 | 0.045 | 0.179 | 0.626 | 0.01 | 0.121 | 0.016 | 0.018 | 0.031
1.0 | 500 | $\numprint{32.088}$ | 0.939 | 0.03 | 0.223 | 0.364 | 0.057 | 0.138 | 0.068 | 0.06 | 0.09
2.0 | $\numprint{29.525}$ | 0.908 | 0.049 | 0.191 | 0.58 | 0.02 | 0.115 | 0.025 | 0.025 | 0.043
3.0 | $\numprint{27.923}$ | 0.883 | 0.065 | 0.13 | 0.728 | 0.01 | 0.082 | 0.012 | 0.018 | 0.019
4.0 | $\numprint{26.749}$ | 0.861 | 0.078 | 0.084 | 0.819 | 0.006 | 0.066 | 0.007 | 0.009 | 0.01
5.0 | $\numprint{25.8}$ | 0.841 | 0.090 | 0.056 | 0.872 | 0.003 | 0.05 | 0.004 | 0.007 | 0.007
1.0 | 600 | $\numprint{28.385}$ | 0.889 | 0.062 | 0.163 | 0.566 | 0.035 | 0.103 | 0.036 | 0.044 | 0.052
2.0 | $\numprint{25.677}$ | 0.839 | 0.093 | 0.079 | 0.821 | 0.007 | 0.052 | 0.011 | 0.018 | 0.012
3.0 | $\numprint{24.125}$ | 0.803 | 0.113 | 0.038 | 0.916 | 0.003 | 0.027 | 0.004 | 0.008 | 0.004
4.0 | $\numprint{22.995}$ | 0.773 | 0.129 | 0.022 | 0.945 | 0.002 | 0.02 | 0.002 | 0.005 | 0.003
5.0 | $\numprint{22.044}$ | 0.747 | 0.143 | 0.014 | 0.961 | 0.002 | 0.015 | 0.002 | 0.003 | 0.003
Sadness | 1.0 | 400 | $\numprint{35.391}$ | 0.967 | 0.012 | 0.202 | 0.144 | 0.174 | 0.107 | 0.13 | 0.099 | 0.146
2.0 | $\numprint{33.241}$ | 0.952 | 0.020 | 0.188 | 0.092 | 0.27 | 0.079 | 0.144 | 0.094 | 0.135
3.0 | $\numprint{31.676}$ | 0.937 | 0.029 | 0.159 | 0.052 | 0.393 | 0.052 | 0.138 | 0.083 | 0.123
4.0 | $\numprint{30.409}$ | 0.921 | 0.039 | 0.129 | 0.03 | 0.507 | 0.036 | 0.13 | 0.064 | 0.104
5.0 | $\numprint{29.329}$ | 0.906 | 0.049 | 0.098 | 0.014 | 0.609 | 0.022 | 0.121 | 0.05 | 0.086
1.0 | 500 | $\numprint{32.718}$ | 0.946 | 0.025 | 0.212 | 0.104 | 0.23 | 0.091 | 0.131 | 0.092 | 0.14
2.0 | $\numprint{30.083}$ | 0.917 | 0.044 | 0.179 | 0.038 | 0.408 | 0.048 | 0.126 | 0.077 | 0.124
3.0 | $\numprint{28.282}$ | 0.890 | 0.061 | 0.128 | 0.014 | 0.573 | 0.027 | 0.106 | 0.052 | 0.1
4.0 | $\numprint{26.887}$ | 0.864 | 0.078 | 0.084 | 0.009 | 0.696 | 0.014 | 0.086 | 0.034 | 0.077
5.0 | $\numprint{25.713}$ | 0.837 | 0.095 | 0.06 | 0.006 | 0.774 | 0.008 | 0.067 | 0.025 | 0.06
1.0 | 600 | $\numprint{29.853}$ | 0.913 | 0.047 | 0.213 | 0.064 | 0.296 | 0.075 | 0.126 | 0.082 | 0.144
2.0 | $\numprint{26.946}$ | 0.867 | 0.078 | 0.142 | 0.014 | 0.549 | 0.032 | 0.099 | 0.055 | 0.109
3.0 | $\numprint{25.062}$ | 0.827 | 0.103 | 0.09 | 0.008 | 0.712 | 0.017 | 0.069 | 0.029 | 0.076
4.0 | $\numprint{23.587}$ | 0.789 | 0.126 | 0.052 | 0.005 | 0.806 | 0.008 | 0.051 | 0.02 | 0.058
5.0 | $\numprint{22.350}$ | 0.752 | 0.146 | 0.036 | 0.005 | 0.849 | 0.005 | 0.04 | 0.016 | 0.049
Surprise | 1.0 | 400 | $\numprint{35.452}$ | 0.967 | 0.012 | 0.205 | 0.183 | 0.089 | 0.184 | 0.126 | 0.079 | 0.134
2.0 | $\numprint{33.379}$ | 0.953 | 0.020 | 0.204 | 0.19 | 0.063 | 0.252 | 0.126 | 0.053 | 0.112
3.0 | $\numprint{31.873}$ | 0.940 | 0.028 | 0.191 | 0.18 | 0.048 | 0.337 | 0.11 | 0.04 | 0.094
4.0 | $\numprint{30.639}$ | 0.927 | 0.037 | 0.168 | 0.173 | 0.035 | 0.418 | 0.105 | 0.028 | 0.074
5.0 | $\numprint{29.582}$ | 0.913 | 0.046 | 0.135 | 0.167 | 0.027 | 0.482 | 0.106 | 0.024 | 0.059
1.0 | 500 | $\numprint{32.580}$ | 0.945 | 0.026 | 0.209 | 0.165 | 0.072 | 0.242 | 0.137 | 0.063 | 0.112
2.0 | $\numprint{29.901}$ | 0.916 | 0.045 | 0.174 | 0.148 | 0.044 | 0.398 | 0.12 | 0.034 | 0.082
3.0 | $\numprint{28.096}$ | 0.889 | 0.063 | 0.127 | 0.136 | 0.027 | 0.522 | 0.115 | 0.024 | 0.048
4.0 | $\numprint{26.700}$ | 0.863 | 0.079 | 0.091 | 0.125 | 0.017 | 0.603 | 0.12 | 0.014 | 0.03
5.0 | $\numprint{25.536}$ | 0.838 | 0.094 | 0.066 | 0.114 | 0.014 | 0.658 | 0.12 | 0.008 | 0.022
1.0 | 600 | $\numprint{29.426}$ | 0.907 | 0.051 | 0.191 | 0.131 | 0.064 | 0.335 | 0.143 | 0.049 | 0.088
2.0 | $\numprint{26.435}$ | 0.856 | 0.084 | 0.191 | 0.131 | 0.064 | 0.335 | 0.143 | 0.049 | 0.088
3.0 | $\numprint{24.448}$ | 0.813 | 0.110 | 0.078 | 0.085 | 0.016 | 0.662 | 0.133 | 0.008 | 0.018
4.0 | $\numprint{22.957}$ | 0.774 | 0.133 | 0.048 | 0.092 | 0.008 | 0.714 | 0.122 | 0.006 | 0.012
5.0 | $\numprint{21.726}$ | 0.738 | 0.152 | 0.036 | 0.1 | 0.006 | 0.731 | 0.115 | 0.004 | 0.009
Fear | 1.0 | 400 | $\numprint{35.325}$ | 0.967 | 0.012 | 0.164 | 0.15 | 0.122 | 0.148 | 0.174 | 0.093 | 0.148
2.0 | $\numprint{33.121}$ | 0.952 | 0.020 | 0.124 | 0.108 | 0.13 | 0.149 | 0.272 | 0.088 | 0.13
3.0 | $\numprint{31.471}$ | 0.937 | 0.030 | 0.087 | 0.071 | 0.124 | 0.14 | 0.388 | 0.081 | 0.109
4.0 | $\numprint{30.114}$ | 0.921 | 0.040 | 0.053 | 0.048 | 0.115 | 0.13 | 0.498 | 0.066 | 0.09
5.0 | $\numprint{28.935}$ | 0.904 | 0.051 | 0.033 | 0.032 | 0.098 | 0.113 | 0.598 | 0.057 | 0.068
1.0 | 500 | $\numprint{32.566}$ | 0.946 | 0.025 | 0.151 | 0.113 | 0.118 | 0.166 | 0.222 | 0.093 | 0.136
2.0 | $\numprint{29.750}$ | 0.915 | 0.045 | 0.085 | 0.058 | 0.117 | 0.17 | 0.396 | 0.066 | 0.109
3.0 | $\numprint{27.750}$ | 0.884 | 0.065 | 0.043 | 0.033 | 0.092 | 0.148 | 0.562 | 0.05 | 0.072
4.0 | $\numprint{26.185}$ | 0.854 | 0.084 | 0.02 | 0.022 | 0.072 | 0.112 | 0.692 | 0.035 | 0.048
5.0 | $\numprint{24.885}$ | 0.824 | 0.103 | 0.012 | 0.017 | 0.054 | 0.093 | 0.764 | 0.028 | 0.033
1.0 | 600 | $\numprint{29.539}$ | 0.910 | 0.049 | 0.133 | 0.073 | 0.114 | 0.195 | 0.291 | 0.073 | 0.122
2.0 | $\numprint{26.260}$ | 0.855 | 0.085 | 0.052 | 0.024 | 0.089 | 0.18 | 0.541 | 0.044 | 0.071
3.0 | $\numprint{24.115}$ | 0.806 | 0.115 | 0.016 | 0.016 | 0.062 | 0.136 | 0.708 | 0.023 | 0.039
4.0 | $\numprint{22.492}$ | 0.760 | 0.141 | 0.009 | 0.014 | 0.045 | 0.096 | 0.794 | 0.017 | 0.024
5.0 | $\numprint{21.159}$ | 0.717 | 0.164 | 0.008 | 0.012 | 0.038 | 0.081 | 0.832 | 0.013 | 0.016
Disgust | 1.0 | 400 | $\numprint{35.378}$ | 0.967 | 0.012 | 0.177 | 0.151 | 0.111 | 0.122 | 0.122 | 0.144 | 0.173
2.0 | $\numprint{33.184}$ | 0.952 | 0.020 | 0.149 | 0.106 | 0.108 | 0.1 | 0.116 | 0.224 | 0.197
3.0 | $\numprint{31.545}$ | 0.937 | 0.030 | 0.115 | 0.073 | 0.098 | 0.074 | 0.111 | 0.317 | 0.212
4.0 | $\numprint{30.211}$ | 0.921 | 0.040 | 0.094 | 0.05 | 0.082 | 0.056 | 0.094 | 0.413 | 0.211
5.0 | $\numprint{29.068}$ | 0.904 | 0.051 | 0.065 | 0.032 | 0.076 | 0.041 | 0.08 | 0.509 | 0.196
1.0 | 500 | $\numprint{32.759}$ | 0.947 | 0.024 | 0.17 | 0.122 | 0.112 | 0.104 | 0.11 | 0.19 | 0.192
2.0 | $\numprint{30.099}$ | 0.919 | 0.043 | 0.119 | 0.064 | 0.088 | 0.069 | 0.099 | 0.335 | 0.226
3.0 | $\numprint{28.214}$ | 0.891 | 0.062 | 0.073 | 0.04 | 0.072 | 0.04 | 0.073 | 0.478 | 0.224
4.0 | $\numprint{26.745}$ | 0.863 | 0.081 | 0.044 | 0.029 | 0.057 | 0.027 | 0.055 | 0.594 | 0.194
5.0 | $\numprint{25.497}$ | 0.835 | 0.100 | 0.029 | 0.025 | 0.042 | 0.017 | 0.043 | 0.676 | 0.168
1.0 | 600 | $\numprint{29.914}$ | 0.915 | 0.047 | 0.15 | 0.095 | 0.097 | 0.083 | 0.099 | 0.247 | 0.229
2.0 | $\numprint{26.948}$ | 0.869 | 0.078 | 0.082 | 0.042 | 0.069 | 0.043 | 0.069 | 0.452 | 0.243
3.0 | $\numprint{24.981}$ | 0.826 | 0.106 | 0.038 | 0.033 | 0.047 | 0.02 | 0.052 | 0.593 | 0.216
4.0 | $\numprint{23.489}$ | 0.787 | 0.131 | 0.021 | 0.034 | 0.036 | 0.012 | 0.039 | 0.682 | 0.176
5.0 | $\numprint{22.237}$ | 0.750 | 0.154 | 0.015 | 0.036 | 0.028 | 0.009 | 0.031 | 0.734 | 0.147
Anger | 1.0 | 400 | $\numprint{35.566}$ | 0.968 | 0.011 | 0.203 | 0.146 | 0.102 | 0.119 | 0.115 | 0.105 | 0.21
2.0 | $\numprint{33.477}$ | 0.954 | 0.019 | 0.188 | 0.102 | 0.091 | 0.104 | 0.1 | 0.124 | 0.291
3.0 | $\numprint{31.943}$ | 0.940 | 0.027 | 0.168 | 0.071 | 0.085 | 0.084 | 0.088 | 0.132 | 0.372
4.0 | $\numprint{30.698}$ | 0.926 | 0.036 | 0.146 | 0.044 | 0.076 | 0.072 | 0.073 | 0.141 | 0.448
5.0 | $\numprint{29.627}$ | 0.912 | 0.045 | 0.127 | 0.03 | 0.068 | 0.057 | 0.064 | 0.142 | 0.512
1.0 | 500 | $\numprint{32.761}$ | 0.946 | 0.025 | 0.201 | 0.102 | 0.098 | 0.104 | 0.101 | 0.125 | 0.269
2.0 | $\numprint{30.115}$ | 0.918 | 0.043 | 0.168 | 0.041 | 0.077 | 0.076 | 0.071 | 0.13 | 0.436
3.0 | $\numprint{28.304}$ | 0.891 | 0.061 | 0.127 | 0.021 | 0.063 | 0.05 | 0.054 | 0.128 | 0.558
4.0 | $\numprint{26.864}$ | 0.865 | 0.077 | 0.091 | 0.012 | 0.048 | 0.04 | 0.042 | 0.117 | 0.651
5.0 | $\numprint{25.659}$ | 0.840 | 0.094 | 0.072 | 0.008 | 0.039 | 0.025 | 0.037 | 0.108 | 0.71
1.0 | 600 | $\numprint{29.626}$ | 0.910 | 0.049 | 0.191 | 0.058 | 0.088 | 0.085 | 0.084 | 0.129 | 0.364
2.0 | $\numprint{26.599}$ | 0.861 | 0.083 | 0.123 | 0.01 | 0.059 | 0.042 | 0.052 | 0.119 | 0.595
3.0 | $\numprint{24.665}$ | 0.819 | 0.109 | 0.078 | 0.006 | 0.043 | 0.025 | 0.032 | 0.097 | 0.72
4.0 | $\numprint{23.170}$ | 0.781 | 0.132 | 0.046 | 0.007 | 0.028 | 0.017 | 0.023 | 0.086 | 0.794
5.0 | $\numprint{21.882}$ | 0.744 | 0.155 | 0.033 | 0.008 | 0.019 | 0.013 | 0.017 | 0.072 | 0.838

Table 3: LDM-based emotion manipulation on the AffectNet validation set, using
forward/backward DDIM processes with $T_{\textrm{DDIM}}=40$ steps, variable
c.f.g. scale $\gamma\in\\{1.0,2.0,3.0,4.0,5.0\\}$ and editing strength
$t_{0}\in\\{400,500,600\\}$.

$y_{\textrm{target}}$ | DDIM Steps ($T_{\textrm{DDIM}}$) | Str. $(t_{0})$ | Image Quality | Emotion Recognition Accuracy w/ HSEmotion [39, 40, 41]
---|---|---|---|---
PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | Neutral | Happy | Sad | Surprised | Scared | Disgusted | Angry
Happiness | 20 | 400 | $\numprint{31.246}$ | 0.934 | 0.031 | 0.238 | 0.451 | 0.028 | 0.15 | 0.035 | 0.033 | 0.065
40 | $\numprint{31.706}$ | 0.937 | 0.029 | 0.244 | 0.447 | 0.028 | 0.148 | 0.035 | 0.032 | 0.066
80 | $\numprint{32.041}$ | 0.938 | 0.029 | 0.245 | 0.446 | 0.03 | 0.15 | 0.034 | 0.031 | 0.065
20 | 500 | $\numprint{27.667}$ | 0.883 | 0.064 | 0.133 | 0.712 | 0.011 | 0.094 | 0.013 | 0.015 | 0.022
40 | $\numprint{27.923}$ | 0.883 | 0.065 | 0.13 | 0.728 | 0.01 | 0.082 | 0.012 | 0.018 | 0.019
80 | $\numprint{28.177}$ | 0.883 | 0.065 | 0.127 | 0.734 | 0.011 | 0.081 | 0.012 | 0.016 | 0.019
20 | 600 | $\numprint{24.113}$ | 0.811 | 0.107 | 0.045 | 0.905 | 0.002 | 0.031 | 0.004 | 0.007 | 0.005
40 | $\numprint{24.126}$ | 0.803 | 0.113 | 0.038 | 0.916 | 0.003 | 0.027 | 0.004 | 0.008 | 0.004
80 | $\numprint{24.281}$ | 0.800 | 0.115 | 0.037 | 0.919 | 0.003 | 0.026 | 0.004 | 0.007 | 0.005
Sadness | 20 | 400 | $\numprint{31.184}$ | 0.933 | 0.031 | 0.157 | 0.052 | 0.398 | 0.051 | 0.141 | 0.081 | 0.119
40 | $\numprint{31.676}$ | 0.937 | 0.029 | 0.159 | 0.052 | 0.393 | 0.052 | 0.138 | 0.083 | 0.123
80 | $\numprint{32.024}$ | 0.938 | 0.028 | 0.159 | 0.054 | 0.393 | 0.053 | 0.139 | 0.08 | 0.122
20 | 500 | $\numprint{27.974}$ | 0.889 | 0.061 | 0.122 | 0.016 | 0.562 | 0.029 | 0.111 | 0.055 | 0.104
40 | $\numprint{28.282}$ | 0.890 | 0.061 | 0.128 | 0.014 | 0.573 | 0.027 | 0.106 | 0.052 | 0.1
80 | $\numprint{28.560}$ | 0.891 | 0.061 | 0.128 | 0.015 | 0.572 | 0.027 | 0.105 | 0.052 | 0.102
20 | 600 | $\numprint{24.995}$ | 0.833 | 0.097 | 0.089 | 0.006 | 0.694 | 0.015 | 0.082 | 0.032 | 0.082
40 | $\numprint{25.062}$ | 0.827 | 0.103 | 0.09 | 0.008 | 0.712 | 0.017 | 0.069 | 0.029 | 0.076
80 | $\numprint{25.231}$ | 0.824 | 0.106 | 0.089 | 0.008 | 0.716 | 0.016 | 0.066 | 0.029 | 0.076
Surprise | 20 | 400 | $\numprint{31.371}$ | 0.937 | 0.030 | 0.186 | 0.18 | 0.046 | 0.34 | 0.112 | 0.042 | 0.096
40 | $\numprint{31.873}$ | 0.940 | 0.028 | 0.191 | 0.18 | 0.048 | 0.337 | 0.11 | 0.04 | 0.094
80 | $\numprint{32.221}$ | 0.942 | 0.027 | 0.188 | 0.182 | 0.048 | 0.339 | 0.11 | 0.041 | 0.092
20 | 500 | $\numprint{27.821}$ | 0.888 | 0.062 | 0.129 | 0.137 | 0.029 | 0.512 | 0.117 | 0.021 | 0.054
40 | $\numprint{28.096}$ | 0.889 | 0.063 | 0.127 | 0.136 | 0.027 | 0.522 | 0.115 | 0.024 | 0.048
80 | $\numprint{28.361}$ | 0.890 | 0.063 | 0.127 | 0.138 | 0.026 | 0.525 | 0.114 | 0.021 | 0.048
20 | 600 | $\numprint{24.469}$ | 0.821 | 0.104 | 0.078 | 0.086 | 0.017 | 0.661 | 0.128 | 0.01 | 0.021
40 | $\numprint{24.448}$ | 0.813 | 0.110 | 0.078 | 0.085 | 0.016 | 0.662 | 0.133 | 0.008 | 0.018
80 | $\numprint{24.604}$ | 0.810 | 0.113 | 0.075 | 0.083 | 0.014 | 0.67 | 0.131 | 0.008 | 0.018
Fear | 20 | 400 | $\numprint{30.986}$ | 0.933 | 0.031 | 0.084 | 0.068 | 0.126 | 0.144 | 0.389 | 0.08 | 0.108
40 | $\numprint{31.471}$ | 0.937 | 0.030 | 0.087 | 0.071 | 0.124 | 0.14 | 0.388 | 0.081 | 0.109
80 | $\numprint{31.817}$ | 0.938 | 0.029 | 0.087 | 0.072 | 0.121 | 0.142 | 0.388 | 0.08 | 0.11
20 | 500 | $\numprint{27.457}$ | 0.883 | 0.065 | 0.043 | 0.034 | 0.098 | 0.14 | 0.562 | 0.053 | 0.072
40 | $\numprint{27.750}$ | 0.884 | 0.065 | 0.043 | 0.033 | 0.092 | 0.148 | 0.562 | 0.05 | 0.072
80 | $\numprint{28.034}$ | 0.885 | 0.065 | 0.042 | 0.032 | 0.094 | 0.146 | 0.564 | 0.05 | 0.072
20 | 600 | $\numprint{24.092}$ | 0.813 | 0.109 | 0.018 | 0.016 | 0.068 | 0.133 | 0.696 | 0.027 | 0.041
40 | $\numprint{24.115}$ | 0.806 | 0.115 | 0.016 | 0.016 | 0.062 | 0.136 | 0.708 | 0.023 | 0.039
80 | $\numprint{24.286}$ | 0.804 | 0.117 | 0.016 | 0.016 | 0.061 | 0.134 | 0.715 | 0.021 | 0.038
Disgust | 20 | 400 | $\numprint{31.091}$ | 0.934 | 0.031 | 0.116 | 0.072 | 0.099 | 0.074 | 0.11 | 0.316 | 0.214
40 | $\numprint{31.545}$ | 0.937 | 0.030 | 0.115 | 0.073 | 0.098 | 0.074 | 0.111 | 0.317 | 0.212
80 | $\numprint{31.881}$ | 0.938 | 0.029 | 0.115 | 0.073 | 0.097 | 0.077 | 0.108 | 0.317 | 0.213
20 | 500 | $\numprint{27.916}$ | 0.890 | 0.061 | 0.079 | 0.041 | 0.071 | 0.04 | 0.079 | 0.466 | 0.224
40 | $\numprint{28.214}$ | 0.891 | 0.062 | 0.073 | 0.04 | 0.072 | 0.04 | 0.073 | 0.478 | 0.224
80 | $\numprint{28.480}$ | 0.891 | 0.062 | 0.074 | 0.041 | 0.072 | 0.04 | 0.072 | 0.481 | 0.219
20 | 600 | $\numprint{24.938}$ | 0.833 | 0.100 | 0.041 | 0.032 | 0.05 | 0.025 | 0.051 | 0.579 | 0.222
40 | $\numprint{24.981}$ | 0.826 | 0.106 | 0.038 | 0.033 | 0.047 | 0.02 | 0.052 | 0.593 | 0.216
80 | $\numprint{25.154}$ | 0.824 | 0.108 | 0.038 | 0.036 | 0.05 | 0.019 | 0.048 | 0.596 | 0.212
Anger | 20 | 400 | $\numprint{31.424}$ | 0.937 | 0.029 | 0.169 | 0.069 | 0.083 | 0.082 | 0.088 | 0.131 | 0.377
40 | $\numprint{31.943}$ | 0.940 | 0.027 | 0.168 | 0.071 | 0.085 | 0.084 | 0.088 | 0.132 | 0.372
80 | $\numprint{32.298}$ | 0.942 | 0.026 | 0.17 | 0.07 | 0.083 | 0.083 | 0.088 | 0.133 | 0.374
20 | 500 | $\numprint{27.978}$ | 0.890 | 0.060 | 0.125 | 0.02 | 0.064 | 0.053 | 0.055 | 0.128 | 0.554
40 | $\numprint{28.304}$ | 0.891 | 0.061 | 0.127 | 0.021 | 0.063 | 0.05 | 0.054 | 0.128 | 0.558
80 | $\numprint{28.587}$ | 0.892 | 0.061 | 0.13 | 0.02 | 0.061 | 0.051 | 0.052 | 0.125 | 0.562
20 | 600 | $\numprint{24.599}$ | 0.825 | 0.103 | 0.078 | 0.006 | 0.043 | 0.026 | 0.033 | 0.105 | 0.709
40 | $\numprint{24.665}$ | 0.819 | 0.109 | 0.078 | 0.006 | 0.043 | 0.025 | 0.032 | 0.097 | 0.72
80 | $\numprint{24.832}$ | 0.817 | 0.112 | 0.075 | 0.005 | 0.041 | 0.024 | 0.029 | 0.1 | 0.725

Table 4: LDM-based emotion manipulation on the AffectNet validation set, using
forward/backward DDIM processes with c.f.g. scale $\gamma=3.0$, variable
number of steps $T_{\textrm{DDIM}}\in\\{20,40,80\\}$ and editing strength
$t_{0}\in\\{400,500,600\\}$.

##  C CLIP-Guided Finetuning

The first step in finetuning an LDM, involves the precomputation of noisy
latents for a predefined number of training instances and the entirety of the
testing instances. We precomputed noisy latents using deterministic forward
DDIM ($\eta=0$) with $T_{\textrm{DDIM}}=40$ steps and $t_{0}=500$ editing
strength for 4,000 training instances (500 per emotion class). Subsequently,
we randomly sampled 1,000 latents from the latter pool in an attempt to trade
off model performance against reduced tuning time. This procedure was
replicated for all guidance scales $\gamma\in\\{1.0,2.0,3.0,4.0,5.0\\}$. We
ran finetuning for 20 epochs with a base learning rate of $2\times 10^{-6}$,
batch size of 4, using the AdamW [27] optimizer. Each batch of latents
undergoes DDIM sampling for $T_{\textrm{tune}}=6$ steps and $t_{0}=500$.
Obviously this low number of steps in the DDIM sampling process gives
imperfect reconstruction but is empirically chosen [21] as an acceptable
compromise between sample quality and GPU memory requirements. The tuning
procedure is repeated for each one of the six target emotions, leading to six
different models per value of $\gamma$ and combination of
$(\lambda_{\textrm{dir}},\lambda_{\textrm{id}},\lambda_{\ell_{2}})$
coefficients. For simplicity, during all experiments we set
$\lambda_{\textrm{id}}=\lambda_{\ell_{2}}=1$ and only varied
$\lambda_{\textrm{dir}}$.

A qualitative comparison between the baseline and tuned models is presented in
Fig. 6, Fig. 7, where the values of c.f.g scale $\gamma$ and weight
$\lambda_{\textrm{dir}}$ are varied, respectively. Increasing $\gamma$ in the
finetuned models, does not particularly affect the cases of happiness and
disgust while a slight shift in colors can be noticed in some cases. This
could either be a result of the small number of sampling steps used during the
tuning process ($T_{\textrm{tune}}=6$) or higher required coefficients for
$\lambda_{\ell_{2}}$.

Fig. 8 and Fig. 9 present additional quantitative comparisons between baseline
and finetuned models, as well as insights regarding the trade-off between
emotion translation and quality preservation during CGF. More specifically, in
Fig. 8 the finetuned models demonstrate higher classification accuracy scores
for the same values of c.f.g scale $\gamma$, with the exception of the emotion
of ‘disgust’. Images that undergo CLIP guidance with the latter being the
target emotion, end up looking very similar to their ‘angry’ counterparts,
hence the drop in accuracy. The boost in classification accuracy comes at the
cost of a slight dicrease in image quality across all quality metrics, but as
it will be demonstrated later, the end result still remains visually
compelling. Furthermore, in Fig. 9, we can see that even with no unconditional
guidance ($\gamma=1$), a setting in which the manipulation effect of the
baseline method is often unnoticeable to the human eye, the introduction of
directional CLIP loss resolves emotional ambiguity to a large extend, with the
exception of ‘disgust’, as discussed above. An additional qualitative
comparison between baseline and finetuned models on curated examples can be
seen in Fig. 10.

|
---|---

(a)

(b)

Figure 6: Emotion translation comparison between baseline and finetuned models
on curated examples from the AffectNet validation set, using
$T_{\textrm{DDIM}}=40$ steps, $\eta=0$, editing strength $t_{0}=500$, variable
c.f.g. scale $\gamma\in\\{1.0,2.0,3.0,4.0,5.0\\}$,
$\lambda_{\textrm{dir}}=2.0,\lambda_{\textrm{id}}=\lambda_{\ell_{2}}=1.0$.

|
---|---

(a)

(b)

Figure 7: Qualitative comparison between baseline and finetuned models on
curated examples from the AffectNet validation set, using
$T_{\textrm{DDIM}}=40$ steps, $\eta=0$, editing strength $t_{0}=500$,
$\gamma=1$ $\lambda_{\textrm{id}}=\lambda_{\ell_{2}}=1.0$ and variable
$\lambda_{\textrm{dir}}\in\\{1.0,2.0,3.0\\}$.

Figure 8: Quantitative comparison between baseline and CLIP-guided finetuned
(CGF) models, in terms of mean classification accuracy (in the range [0-1]),
LPIPS, SSIM and PSNR of manipulated samples, using $T_{\textrm{DDIM}}=40$
steps, editing strength $t_{0}=500$,
$\lambda_{\textrm{dir}}=2.0,\lambda_{\textrm{id}}=\lambda_{\ell_{2}}=1.0$,
across all c.f.g. scales $\gamma\in\\{1.0,2.0,3.0,4.0,5.0\\}$.

Figure 9: Quantitative comparison between baseline
$(\lambda_{\textrm{dir}}=0)$ and CLIP-guided finetuned (CGF) models
$(\lambda_{\textrm{dir}}>0)$, in terms of mean classification accuracy (in the
range [0-1]), LPIPS, SSIM and PSNR of manipulated samples, using
$T_{\textrm{DDIM}}=40$ steps, editing strength $t_{0}=500$,
$\gamma=1.0,\lambda_{\textrm{id}}=\lambda_{\ell_{2}}=1.0$, across different
values of $\lambda_{\textrm{dir}}\in\\{0.0,1.0,2.0,3.0\\}$.

Figure 10: Qualitative comparison between baseline and finetuned models on
curated examples from the AffectNet validation set, using
$T_{\textrm{DDIM}}=40$ steps, $\eta=0$, editing strength $t_{0}=500$, c.f.g
scale $\gamma=3.0$,
$\lambda_{\textrm{dir}}=2.0,\lambda_{\textrm{id}}=\lambda_{\ell_{2}}=1.0$.

##  D Training & Evaluation Specifications

Code regarding the implementation of our models and reproduction of all
experiments is included as supplementary material. The backbone of our code is
based on the original Stable Diffusion
repository333https://github.com/CompVis/stable-diffusion.

For testing GANmut [8] against our own models, we used the official
codebase444https://github.com/stefanodapolito/GANmut along with publicly
available models that have been pre-trained on AffectNet. We evaluated both
linear (GANmut) and Gaussian (GGANmut) models. In the case of the linear
model, we manipulated images using maximum intensity $(\rho=1)$. In order to
achieve that, we use the learned unit vectors $\bm{v}_{c}=(x_{c},y_{c})$
corresponding to each emotion $c\in\\{1,\dots,C\\}$, and calculate the
principal direction $\theta_{c}=\textrm{arctan2}(y_{c},x_{c})$.

Subsequently, for the application of GANimation [34] in the emotion
manipulation scenario with discrete emotions, we used the following scheme: we
randomly sampled 500 images corresponding to each emotion label from the
training set and computed the required target AU vectors for each one of them,
using the publicly available
OpenFace555https://github.com/TadasBaltrusaitis/OpenFace [2] toolkit. Each
vector contains intensities in the range $[0,5]$ for each on of 17 detected
AUs which are later normalized to $[0,1]$. As GAN training is notoriously
unstable, we opted for using a publicly available model
checkpoint666https://github.com/donydchen/ganimation_replicate, pre-trained on
more than 400K images with annotated AUs belonging to the EmotionNet database
[13].

We trained a StarGAN v2 [7] model, using the publicly available official
codebase777https://github.com/clovaai/stargan-v2, on AffectNet for 100K
iterations using a batch size of 8. The learning rate for $D,E$ and $G$ is set
equal to $10^{-4}$, while for $F$ is set equal to $10^{-6}$, using the Adam
optimizer. The loss weighting coefficients were set equal to
$\lambda_{\textrm{sty}}=\lambda_{\textrm{div}}=\lambda_{\textrm{cyc}}=\lambda_{\textrm{adv}}=1$,
for simplicity. Input image size is $128^{2}$ pixels, with $|\mathcal{Y}|=8$
domains (in accordance with the basic 7 emotions of the AffectNet dataset,
plus neutral). We used latent code dimension $d_{\bm{z}}=16$, style code
dimension $d_{\bm{s}}=64$ and 512 units for the hidden layers of the mapping
network. For image-manipulation, we used latent-based translation, generating
10K latent codes per image, per target emotion. Subsequently, these latent
codes were mapped to their corresponding style codes using the trained mapping
network, while the generator was fed with the average of the computed style
codes, along with the images to be translated.

##  E Comparisons w/ GAN-based Models

Fig. 11 provides a qualitative comparison among the three aforementioned
implementations, considering both linear and Gaussian variants of GANmut, on
curated examples from the AffectNet validation set. Making comparisons between
GAN and diffusion-based models is far from straightforward and we need to take
into consideration the underlying trade-off between emotion classification
accuracy and quality/identity preservation with regard to source images. We
immediately notice that samples manipulated with GANmut, feature accurate
depictions of the target emotions but often suffer from noise and subject
identity deformation. Moreover, as it was expected, it is far more difficult
to capture the desired emotions with GANimation and randomly selected driving
AUs, rather that curated ones, but such a comparison would have been unfair
with regard to other implementations. On the other hand, StarGAN v2 seems to
strike a better balance between image quality and emotion transfer, but mostly
fails to preserve the subjects’ identity, skin tone and illumination. Our
diffusion model successfully compromises between style preservation and
emotion transfer, while also maintaining high image quality.

Method | Mean
---|---
Accuracy$\uparrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | CSIM$\uparrow$
Groundtruth [30] | 0.638 | – | – | – | –
GANimation [34] | 0.348 | 24.17 | 0.822 | 0.100 | 0.562
StarGAN v2 [7] | 0.782 | 17.95 | 0.684 | 0.157 | 0.514
GANmut [8] | 0.877 | 22.55 | 0.824 | 0.106 | 0.693
GGANmut [8] | 0.969 | 21.67 | 0.783 | 0.123 | 0.614
Ours | 0.742 | 25.51 | 0.836 | 0.096 | 0.738
Ours w/ CGF | 0.768 | 24.38 | 0.813 | 0.099 | 0.745

Table 5: Quantitative comparison with mean aggregated evaluation metrics
across all six target emotions.

|  |
---|---|---

(a)

(b)

(c)

Figure 11: Qualitative comparison among our LDM-based model and GAN-based
implementations for image-based emotional manipulation on curated examples
from the AffectNet validation set.

Tab. 5 showcases mean aggregated evaluation measures in terms both image
quality and emotion translation accuracy across all six of the considered
target emotions, between our models (baseline and w/ CGF) and the
aforementioned GAN-based methodologies.