# CustomListener: Text-guided Responsive Interaction for
User-friendly Listening Head Generation

Xi Liu , Ying Guo*, Cheng Zhen, Tong Li, Yingying Ao, Pengfei Yan
Vision AI Department, Meituan
https://customlistener.github.io/ Equal contributionCorresponding Author

###### Abstract

Listening head generation aims to synthesize a non-verbal responsive listener
head by modeling the correlation between the speaker and the listener in
dynamic conversion. The applications of listener agent generation in virtual
interaction have promoted many works achieving diverse and fine-grained motion
generation. However, they can only manipulate motions through simple emotional
labels, but cannot freely control the listener’s motions. Since listener
agents should have human-like attributes (e.g. identity, personality) which
can be freely customized by users, this limits their realism. In this paper,
we propose a user-friendly framework called CustomListener to realize the
free-form text prior guided listener generation. To achieve speaker-listener
coordination, we design a Static to Dynamic Portrait module (SDP), which
interacts with speaker information to transform static text into dynamic
portrait token with completion rhythm and amplitude information. To achieve
coherence between segments, we design a Past Guided Generation module (PGG) to
maintain the consistency of customized listener attributes through the motion
prior, and utilize a diffusion-based structure conditioned on the portrait
token and the motion prior to realize the controllable generation. To train
and evaluate our model, we have constructed two text-annotated listening head
datasets based on ViCo and RealTalk, which provide text-video paired labels.
Extensive experiments have verified the effectiveness of our model.

##  1 Introduction

Listening Head Generation (LHG) is to generate listener motions (face
expressions, head movements, etc.) that respond synchronously to the speaker
during face-to-face communication. Good communication is a two-way process,
and the roles of speakers and listeners are equally important [4]. Whether the
listener responds promptly will affect whether the speaker can actively
express the follow-up content [17, 19]. Unlike speaker head generation (SHG)
[7, 16, 31, 39], listeners can only convey information through non-verbal
feedback (nodding, frowning, etc.), and their responses are affected by both
sides of the interaction (speaker’s motion, tone, and listener’s own
personality and emotion state). Currently, many applications, such as digital
avatar generation [8, 20] and human-computer interaction [45], use the
listener agent to simulate more realistic conversations, which has promoted
many studies in LHG.

Figure 1: The process of text-guided listener generation in our
CustomListener. The text prior provides the basic portrait style of the
listener, which is input into CustomListener and combined with the speaker’s
information to obtain the listener’s motions.

Some early works manually integrate motions based on rules [36, 1, 5], or rely
on 2D facial keypoints [11, 26] to manipulate the head, but they are limited
in the variety of motions and the detailed control ability. Recent works [46,
15, 21, 35, 25] have introduced 3D face coefficients [9], combined with the
post-rendering module to achieve more precise control. Specifically, RLHG [46]
uses a sequence-sequence structure as the baseline method for decoding 3DMM
coefficients. PCH [15] further improves the render module to inpaint image
artifacts. L2L [25] quantizes listener motions into the discrete one-
dimensional codebook by VQ-VAE [37], and ELP [35] expands the codebook to a
composition of several discrete codewords to achieve more fine-grained
control. MFR-Net [21] designs a feature aggregation module to achieve better
identity preservation and motion diversity.

Although the above methods can combine speaker-listener information to
generate diverse and fine-grained listener motions, they have restrictions in
the explicit control of the listener agent’s response. Specifically, L2L [25]
focuses on the motion diversity but cannot control the motion under specific
conditions. RLHG [46], PCH [15], MFR-Net [21], and ELP [35] achieves
controllable listener motion conditioned on the input attitude. However, the
attitude is limited to a few fixed labels (eg. positive, neutral, negative),
posing limitations on the generation of realistic listeners: First, simple
labels are not enough to give accurate responses. For example, for negative
attitudes, the two emotions of doubt and anger correspond to different facial
expressions. More importantly, a realistic listener agent should have its own
identity (parent, friend, teacher, etc.), personality (lively, calm, etc.) and
behavioral habits (e.g. frowning when thinking), which users can customize and
set in advance. However, simple labels cannot achieve the control under these
conditions.

Considering the above limitations of simple labels, in this paper, we propose
a user-friendly framework called CustomListener to enable the text-guided
generation, as shown in Figure 1. Users can pre-customize detailed attributes
of the listener agent. Then using a large language model, we combine speech
content and user-customized attributes to obtain the text prior of the
listener’s basic portrait. In CustomListener, we seamlessly incorporate
speaker information while being guided by the text prior, so as to generate
realistic listener responses that are controllable and interactive.

For the guidance of text prior, due to the interactive nature of
communication, listener motions cannot simply be regarded as text-conditioned
motion generation, but faces the following challenges: 1) Speaker-listener
coordination: the text provides the basic portrait style of the listener’s
response. While to precisely express the listener’s emotional empathy towards
the speaker, the response not only needs to complete the text-specified
motion, but also needs to adjust its completion timing and rhythm, and
fluctuate with the speaker’s semantics, intonation and movement amplitude. 2)
Listener motion coherence: When motions conditioned on different text priors
are combined into a long video, different segments should be coherent. Due to
the customization, apart from ensuring smooth motion switching, we also need
to guarantee that the user-customized listener’s behavioral habits reflected
in past segments are maintained in the current segment. Maintaining the style
of the past while still showing the motion of the current text can be
challenging.

To solve the above challenges, in our CustomListener, we design the Static to
Dynamic Portrait module (SDP) and the Past Guided Generation module (PGG) to
achieve the _coordination_ and the _coherence_ respectively. In SDP, we
transform the static portrait provided by the text prior to be dynamic one,
which interacts with audio to obtain the time-dependent information about the
completion time and rhythm of the gradual motion changes, and refines the
listener’s motions to make the speaker-listener fluctuations relevant. In PGG,
we generate a motion prior based on the similarity of dynamic portrait tokens
between adjacent segments, which provides reference information for features
that need to be maintained in the current compared to the past. Then, the
motion prior and the dynamic portrait token are input as conditions into the
diffusion-based structure [14] with fixed noise at the segment connection,
finally generating listener coefficients that enable coordination and
coherence in interactions.

In summary, this paper has the following contributions:

  * •

We argue that simple labels are not sufficient to achieve freely controllable
listener generation, and thus propose the CustomListener framework that can
freely control listener head motions guided by the text prior.

  * •

We design a SDP module to achieve the speaker-listener coordination, so that
the completion timing and rhythm of text-specified motions correspond to the
speaker, and fluctuate with the speaker’s semantics, tone, and movements.

  * •

For long-term generation, we design a PGG module to meet listener motion
coherence between different text-prior-conditioned segments, so that the
consistency of the listener’s customized behavioral habits between segments
can be maintained and the motion switching is smooth.

  * •

Extensive experiments on ViCo [46] and Realtalk [12] datasets confirm that our
method can realize the state-of-the-art performance, and achieve the
controllable and interactive realistic listener motion generation.

##  2 Related work

Figure 2: Overall framework of CustomListener. Given the text prior providing
the listener’s static portrait style, SDP-Module transforms the static
portrait into a dynamic one. Then in PGG-Module, the dynamic portrait token
are combined with motion prior generated from Past-guided Module and are
utilized as conditions of the diffusion-based structure to realize the
controllable generation. The ’C’ in the figure denotes concatenation and the
small pink squares denotes diffusion time-step token.

#### Listening head generation (LHG):

Some early works [36, 1, 5] use rule-based methods to manually incorporate
interactive motions. Later, data-driven methods [11, 26] generate 2D motions
based on facial keypoints or movement frequencies, but 2D information cannot
fully represent the details of facial movements.

Recently, motion generation based on 3D face coefficients, coupled with the
post-processing renderer, has gradually become mainstream due to its stronger
representation ability of facial movements. RLHG [46] proposes a
representative ViCo dataset and constructs a baseline coefficient decoding
method using a sequence-sequence structure. After that, PCH [15] assembles an
enhanced renderer to make the generated results visually better and more
stable by inpainting the image boundary and background. Learning2Listen (L2L)
[25] proposes to use a sequence-encoding VQ-VAE [37] to learn the discrete
codebook of listener motion. ELP [35] further utilizes a composition of
multiple discrete motion-codewords to represent facial motion in a more fine-
grained manner. MFR-Net [21] achieves the preservation of listener’s identity
and the diversity of generation.

Regarding the listener’s motion status, L2L [25] focuses on the diversity but
cannot explicitly control specific actions. RLHG [46], PCH [15], MFR-Net [21]
achieve controllable generation conditioned on the input emotion, but the
emotion is limited to a few labels and cannot specifically control the
expression and posture of each emotion. Although ELP [35] represents the
latent space under different emotions in a fine-grained manner through the
codebook, what the user can control is still the input emotion rather than the
fine-grained motions. In this way, the listener’s response to each emotion may
depend on the data distribution of the training set.

#### Conditional motion generation:

The motion generation task derives applications under a variety of conditions.
For example, body motion conditioned on text descriptions [18, 27, 43, 28,
13], motion trajectory conditioned on scene image [3], and dance motion
conditioned on music [41]. Recently, large language models promote the
development of free-form text conditioned generation based on auto-encoder
[28], hierarchical model [13], and diffusion models [18]. We also generate
motions conditioned on free-form text. However, for our responsive
interaction, in addition to focusing on the matching of motion and text like
the above methods, we also need to make the motion and the speaker’s action,
speech, tone and rhythm harmonious to express the listener’s empathy. In our
paper, we adopt the SDP module to achieve this harmony. For long-term
generation, to ensure the coherent fluency between clips under different text
commands, some methods have adopted transition sampling [42], specifying noise
[44], etc. However, the coherence for the custom listener, which has some
motion habits, is more complex. It is not just a simple motion switch, but is
related to past habits and styles, and our PGG module solves this challenge.

##  3 Method

###  3.1 Overview

In our CustomListener framework, under the guidance of the fine-grained text
prior, we can synthesize natural and controllable listener motions, which can
seamlessly integrate the speaker’s information (motion, speech content, audio)
to give interactive non-verbal responses. Let $L_{m}\in\mathbb{R}^{T\times
70}$ denote the listener motions to be generated, and $T$ is the number of
frames, 70 denotes 64-dim expression and 6-dim pose. Given speaker’s motions
$S_{m}\in\mathbb{R}^{T\times 70}$, audio $S_{a}$, speech content $S_{c}$ and
text prior $R$, our framework can be formulated as:

$L_{m}=Custom(S_{m},S_{a},S_{c};R)\vspace{-0.2cm}$ (1)

The overview of our proposed framework is shown in Figure 2. Before
generation, for each video segment, we use GPT to generate a text prior based
on user-customized listener attributes and the speaker’s speech content, which
provides the listener’s static portrait reference. Then, we extract the
speaker’s audio features from the Audio Encoder and speaker’s motion
coefficients from a pre-trained face reconstruction model [9]. Following this,
we generate realistic listener motions through a two-stage process: (1) Static
to Dynamic Portrait generation (SDP-Module): it can transform static portrait
tokens into the dynamic through audio-text responsive interaction and speaker-
motion based refinement. The dynamic portrait token includes the time-
dependent information about the motion completion and rhythm, and makes the
speaker-listener motion fluctuations correlated; (2) Past-guided Listener
Motion Generation (PGG-Module): to produce coherent motions and maintain the
listener’s previous behavioral habits in long-term generation, we use a past
guided module to produce the motion prior, combined with the diffusion
denoising process with a fixed initial noise to generate the listener
coefficients. Finally, listener videos are obtained through the post-rendering
process.

###  3.2 Static to Dynamic Portrait generation

In this section, we first introduce the generation of the static portrait-
token, and then present SDP-Module, which can convert static tokens into
dynamic one by two stages: (1) Audio-text Responsive Interaction: static
portrait tokens are converted into time-dependent portrait tokens, which
reflects the completion time and rhythm of motions according to the semantics
and intonation in the audio, ensuring gradual motion changes; (2) Speaker
Motion-based Refinement: it refines the listener’s motions according to the
fluctuations of speaker’s motions, producing the final dynamic portrait token
combined with the time-dependent portrait token.

#### Static Portrait-token Generation

To establish a better connection between the user-customized listener
attributes (identity, personality, relationship, attitude, etc.) and the
listener’s motions, we leverage the powerful GPT for semantic understanding of
both user-customized contents and speaker’s speech contents, and generate the
text prior that accurately describes the listener’s expressions and poses.
Following this, we use RoBERTa [22], a robust and efficient model with high
natural language understanding capability, to encode the text prior into text
embeddings. To obtain portrait-related tokens, we further employ a mapping net
to convert text embeddings into portrait tokens. We consider these portrait
tokens to be static portrait tokens since the text prior describes the
listener’s static portrait (eg. expressions and poses) in the entire video
clip.

#### Audio-text Responsive Interaction

Since static portrait tokens from text prior provide a comprehensive
representation of listener head in a video segment, it lacks the time-
dependent information. Thus, the model would be limited to generating constant
motion and cannot effectively convey gradual variation in movement without a
specific design. To address this critical issue, we make use of the semantic
knowledge in speaker’s utterances, which serves as a guidance for motion
shifts. Specifically, we choose the speaker audio to provide semantics instead
of speech contents. The underlying intuition is the former contains not only
semantic information but also the temporal information. This entire process
can be split into two parts, part A and part B.

Figure 3: Illustration of Audio-text Responsive Interaction. We first generate
a weight matrix by responsive interaction between audio features and static
portrait-token (SP Token), and then generate time-dependent portrait token (TP
Token) guided by weight matrix.

Part A In this part, we generate a weight matrix through responsive
interaction between speaker audio features and the static portrait tokens
obtained from text prior. It provides the correlation degree between the
speaker’s semantics at each timestamp and the motion description in the text
prior. For example, as shown in Figure 3, if the phrase “love you” in the
speaker’s audio at time stamp $k$ exhibits the strongest correlation with the
phrase “raised lip corner” in the description texts, the corresponding weight
value will reach its peak (indicated by a pink circle dot in the weight map).
This weight matrix $M(A,H)$ can be formulated as Equation 2.

Part B After obtaining a weight matrix $M(A,H)$, we can generate time-
dependent portrait tokens based on static portrait tokens with the guidance of
$M(A,H)$. Specifically, as illustrated in Figure 3, the time-dependent
portrait tokens (TP Token) at the time stamp $k$ is generated with the highest
weighting assigned to the static portrait tokens (SP Token) at $l=j$, which
corresponds to the phrase “raised lip corner”. This part can be formulated as
Equation 3.

$M(A,H)=Sm(\frac{1}{\sqrt{\alpha}}(Linear(A)\times Linear(H)^{T})),$ (2)
$E_{t}=\sum_{i=0}^{L-1}M(A,H)_{i,t}\cdot
H_{i,\operatorname{j}=0:C-1},t\in\left\\{0,...,T-1\right\\}$ (3)

where $Sm$ denotes Softmax, $A\in\mathbb{R}^{T\times C}$ is the audio feature,
$H\in\mathbb{R}^{L\times C}$ is the static portrait token,
$M\in\mathbb{R}^{T\times L}$ is the weight map, $\alpha$ is a scaling factor,
$E_{t}\in\mathbb{R}^{1\times C}$ is the time-dependent portrait token at time
stamp $t$. $L$ is the length of text embeddings, and $C$ is the feature
dimension.

With the help of two parts above, we can obtain portrait tokens that contain
time-dependent information about the rhythm of motion completion. This enables
the model to learn the progressive changes in facial/head motions within a
single video segment, as opposed to the constant motions.

#### Speaker Motion-based Refinement

In human-to-human conversations, we believe that the amplitude of the
speaker’s motion fluctuation also has an impact on the changes in the
listener’s motion response[32, 21]. Therefore, we utilize the past speaker’s
motion changes to refine the listener’s motion amplitude. This process can be
well-designed formulated as:

$\overline{E_{t}}=\sum_{i=t-1-w}^{t-1}\exp\left|\frac{\partial S_{i}}{\partial
i}\right|,$ (4) $\widetilde{E_{t}}=E_{t}\oplus\overline{E_{t}},$ (5)

where $\oplus$ means concatenation, $E_{t},\overline{E_{t}},\widetilde{E_{t}}$
is time-dependent portrait token, speaker motion-weighted portrait token and
dynamic portrait token at frame $t$, respectively. $S_{i}$ denotes speaker
motion at frame $i$, $w$ is the time-window length. For efficiency, we set $w$
to 5, which means we only take the past five-frame speaker motions into
account. For each time stamp $t$, the listener’s portrait token $E_{t}$ is
affected by the cumulative gradients of several previous speaker motions. This
process can be illustrated in the lower part of the SDP-Module in Figure 2.
Notably, when generating portrait token at frame $t$, we only focus on the
speaker motions up to frame $t-1$. Due to the listener and the speaker
communicating concurrently, the listener’s state at frame $t$ should not
influenced by the speaker’s motion at the same frame $t$.

###  3.3 Past-guided Listenr Motion Generation

The PGG-Module is to ensure that motions between segments are coherent, and
the customized listener attributes in the previous segment can be maintained
in the current segment. It contains Past-guided Module and diffusion-based
motion generation module, which can generate motion prior based on past
motions and then synthesize listener motions. Following this, a renderer is
used to obtain listener videos.

#### Past-guided Module

To produce long video with arbitrary length, we opt for a segment-by-segment
generation approach during inference. However, we have observed that there are
unsmooth pose transitions between segments. This issue can be attributed to
the neglect of correlations between adjacent segments. To address this, we
take past motions into consideration.

Since our final goal is to synthesize a customized listener, in addition to
ensuring smooth transition, we also need to take the listener’s past
behavioral habits into account (e.g. frowning when thinking). Hence, we sought
assistance from dynamic portrait tokens of two adjacent segments to guide the
generation of motion prior. Specifically, the weight we assign to past motions
at different times is determined by the similarity between portrait tokens in
the current segment and those in the last segment. Formally, we calculate the
motion weight and obtain the motion prior $G_{t:t+L}$ by:

$Sim_{t:t+L}=E_{t:t+L}\times E_{t-L:t}^{T},$ (6) $G_{t:t+L}=Sim_{t:t+L}\times
P_{t-L:t},$ (7)

where $\times$ denotes matrix multiplication, $L$ is the segment length, $t$
is the time stamp, $Sim_{t:t+L}$ denotes the similarity between portrait
tokens in the current segment $E_{t:t+L}$ and those in the last segment
$E_{t-L:t}$, which is the weight we assign to past motions $P_{t-L:t}$. The
higher the similarity, the greater the weight we assign. This approach can not
only encourage continuous transitions between two adjacent segments, but also
ensure the maintenance of customized listener’s behavioral habits between two
adjacent sequences.

#### Diffusion Model

To achieve diverse and natural head motion generation, our generation module
is built on denoising diffusion probabilistic models (DDPM) [14].

During forward diffusion process $q$, we progressively inject noise into the
listener motion $x_{0}$ until it becomes a Gaussian noise $x_{T}$, which can
be defined as a Markov chain:

$q(x_{1:T}\mid x_{0})=\prod_{t=1}^{T}q(x_{t}\mid x_{t-1}),$ (8) $q(x_{t}\mid
x_{t-1})=\mathcal{N}(x_{t};\sqrt{1-\beta_{t}}x_{t-1},\beta_{t}I),$ (9)

where $x_{1},...,x_{T}$ are noised listener motions, $\beta_{1},...,\beta_{T}$
are the variance schedule. In the reverse process $p_{\theta}$, we produce
listener motion $x_{0}$ by removing noise from $x_{T}$ step-by-step:

$p_{\theta}(x_{t-1}\mid
x_{t},c)=\mathcal{N}(x_{t-1};\mu_{\theta}(x_{t},c,t),{\textstyle\sum_{\theta}}(x_{t},c,t)),$
(10)

where $c$ is a conditioning variable. We perform denoising process with a
transformer decoder [38] conditioned on the motion prior and dynamic portrait
tokens.

#### Render

We utilize PIRenderer [30] for rendering, which can produce natural listener
videos by incorporating a single listener reference image as well as the 3DMM
coefficients.

##  4 Experiments

Methods | Testset |  FD $\downarrow$ |  RTLCC $\downarrow$ |  RWTLCC$\downarrow$ |  FID△fm$\downarrow$ |  SND $\downarrow$ |  V-D $\uparrow$
---|---|---|---|---|---|---|---
exp | angle | trans | exp | pose | exp | pose | exp | pose | exp | pose | exp | pose
RLHG∗ [46] | $\mathcal{D}_{test}$ | 15.03 | 7.90 | 6.55 | 0.113 | 0.160 | 0.108 | 0.160 | 11.65 | 0.90 | 2.90 | 0.09 | 0.69 | 0.10
$\mathcal{D}_{ood}$ | 22.81 | 8.58 | 8.90 | 0.124 | 0.101 | 0.104 | 0.100 | 9.32 | 0.82 | 7.20 | 0.11 | 1.01 | 0.12
PCH∗ [15] | $\mathcal{D}_{test}$ | 19.02 | 13.26 | 7.74 | 0.124 | 0.163 | 0.109 | 0.198 | 11.59 | 0.96 | 3.10 | 0.13 | 0.31 | 0.12
$\mathcal{D}_{ood}$ | 18.63 | 17.96 | 8.80 | 0.121 | 0.115 | 0.107 | 0.140 | 8.67 | 0.87 | 7.01 | 0.12 | 0.51 | 0.16
L2L∗ [25] | $\mathcal{D}_{test}$ | 14.89 | 7.35 | 6.34 | 0.102 | 0.153 | 0.101 | 0.113 | 8.64 | 0.87 | 2.62 | 0.09 | 0.57 | 0.20
$\mathcal{D}_{ood}$ | 15.64 | 8.23 | 7.89 | 0.095 | 0.098 | 0.100 | 0.101 | 6.78 | 0.74 | 6.89 | 0.10 | 1.23 | 0.28
MFR-Net† [21] | $\mathcal{D}_{test}$ | 13.37 | 6.82 | 6.02 | - | - | - | - | - | - | - | - | - | -
$\mathcal{D}_{ood}$ | 14.70 | 8.12 | 6.37 | - | - | - | - | - | - | - | - | - | -
Ours | $\mathcal{D}_{test}$ | 11.54 | 6.12 | 5.90 | 0.072 | 0.103 | 0.081 | 0.097 | 3.12 | 0.06 | 2.40 | 0.07 | 1.55 | 0.32
$\mathcal{D}_{ood}$ | 12.67 | 7.49 | 6.01 | 0.081 | 0.073 | 0.084 | 0.075 | 3.59 | 0.07 | 5.94 | 0.05 | 1.25 | 0.30
Table 1: Comparisons of our model with other methods on $\mathcal{D}_{test}$
and $\mathcal{D}_{ood}$ of ViCo. Bold represents the best. The † means we
directly refer to data in their paper and ∗ denotes we retrain the model. The
$\downarrow$ indicates lower is better and the $\uparrow$ indicates higher is
better. The values of FD and FID△fm are multiplied by 100. The quantitative
results on RealTalk [12] are in Appendix B.

###  4.1 Datasets

#### Data Annotation

Since the existing listener head datasets do not contain the full data
required for training our model, we conduct additional data annotations on two
datasets:(1) ViCo [46], a popular dataset in listener head generation; (2)
RealTalk, proposed by [12], which serves as a database for retrieving listener
videos in [12]. Firstly, to learn the relationships between the text prior and
listener expressions, we follow the text annotation pipeline in TalkCLIP [24],
and conduct text annotation on ViCo[46] and RealTalk[12]. Specifically, we
employ [6] to acquire emotion labels, activated AUs as well as their
intensities. Additionally, to learn realistic head movements such as nodding
to convey agreement and shaking the head to signify disagreement, we use
Hopenet [34] to detect head motions. Finally, we generate diverse text prior
description for each video segment according to the format: [A person
$<$EMOTION$>$ and listens with $<$AU$>$ (and $<$HEAD MOTION$>$)], where
$<$EMOTION$>$ denotes the emotional labels, $<$AU$>$ denotes fine-grained
expression-related texts based on AU activation intensities and Facial Action
Coding System [10], and $<$HEAD MOTION$>$ indicates head nodding/shaking
labels. We use “and” to link when there is more than one activated AU.
Detailed examples can be seen in Appendix A.

#### Data Construction

For ViCo, which contains three parts: train set $\mathcal{D}_{train}$, test
set $\mathcal{D}_{test}$, out-of-domain set $\mathcal{D}_{ood}$ (all
identities in $\mathcal{D}_{ood}$ have not appeared in $\mathcal{D}_{train}$),
we obtain the transcripts of the speaker’s audios by a speech recognition
model [29], and get an extended version of ViCo. For RealTalk, to align the
dimension of 3DMM coefficients with ViCo, we re-extract facial 3DMM
coefficients using [9] and then obtain two subsets: $\mathcal{D}_{train}$ and
$\mathcal{D}_{test}$. These two extended datasets are important for future
research on text-guided listener head generation task.

Figure 4: Visual results produced by CustomListener. All listener videos are
generated conditioned on different pre-set text priors, the same speaker video
(the 1st row) and the same reference listener image.

###  4.2 Experimental Settings

#### Loss Fuction

During training, followed by [14], we perform denoising on noised listener
motions $x_{t}$ conditioned on motion prior and dynamic portrait token step-
by-step, and our optimization goal is to predict noise term $\epsilon$ at each
step. The objective function can be defined as a conditional diffusion loss as
follows:

$\vspace{-0.1cm}L_{d}=E_{x_{0},t,\epsilon_{t}\sim\mathcal{N}(0,I)}\left[\left\|\epsilon_{t}-\epsilon_{\theta}(x_{t},c,t)\right\|^{2}\right],\vspace{-0.1cm}$
(11)

where $\epsilon_{t}\sim\mathcal{N}(0,\textbf{I})$ is the noise at step $t$,
$c$ denotes conditions, $x_{0}$ is the original listener motions.

#### Implementation Details

The resolution of videos is 256$\times$256, and the FPS is 30. For training,
we clip the original videos to several 60-frame video segments. We extract
3DMM coefficients from speaker videos and listener videos by [9], which serve
as speaker motions and listener motions, respectively. Following MFR-Net [21],
we extract 45-dim acoustic features from audio. We utilize AdamW optimizer
[23] to train our model with a learning rate of $1e^{-4}$.

Method |  SSIM $\uparrow$ |  CPBD $\uparrow$ |  PSNR $\uparrow$ |  FID $\downarrow$
---|---|---|---|---
RLHG∗ [46] | 0.56 | 0.12 | 17.39 | 27.70
PCH∗ [15] | 0.58 | 0.15 | 18.48 | 21.29
L2L∗ [25] | 0.58 | 0.16 | 17.67 | 22.13
MFR-Net† [21] | 0.59 | 0.18 | 17.82 | 20.08
Ours | 0.60 | 0.19 | 17.98 | 20.06
Table 2: Quantitative results with state-of-the-art methods on image quality.
The best results are highlighted in bold.

###  4.3 Quantitative Results

#### Metrics

It’s non-trivial to measure the realism of generated listener motions. We
choose the following metrics based on that a good custom listener should have
four characteristics: (1) natural-looking and diverse facial/head motions; (2)
highly synchronous with the speaker motions; (3) highly coherent motions in a
long video; (4) high image quality:

$\bullet$ FD: $L1$ distance between the predicted 70-dim listener motions and
the ground-truth listener motions.

$\bullet$ Variation for Diversity (V-D) : Variance across the time series
sequence of 3DMM coefficients.

$\bullet$ Residual Time Lagged Cross Correlation (RTLCC): $L1$ distance
between generated TLCC [2] and TLCC [2] of ground truth. RTLCC indicates the
correlation between listener motions and lagged speaker motions.

$\bullet$ Residual Windowed Time Lagged Cross Correlation (RWTLCC): RTLCC in a
fixed window. We set the window length to 4 seconds following ELP [35].

$\bullet$ FID△fm: FID score of differences in 3DMM coefficients between
adjacent frames, which indicates the temporal naturalness of facial motions,
followed by [44].

$\bullet$ SND: Sequence Naturalness Distance proposed by [44], showing the
distribution difference between generated and ground-truth motions.

$\bullet$ SSIM, CPBD, PSNR and FID: Common metrics for image quality
evaluation.

#### Comparisons with Existing Methods

We retrain PCH [15], RLHG [46] and L2L [25] on ViCo[46]. It should be noted
that, source codes of ELP [35] and MFR-Net [21] is unavailable. For MFR-Net
[21], we utilize the data from the original paper. For ELP [35], the 3DMM
coefficient extraction model used is different from ours, resulting in
distinct dimensions of coefficients (e.g., $\beta\in\mathbb{R}^{100T}$ for ELP
[35], $\beta\in\mathbb{R}^{64T}$ for ours), thus it is not reasonable to
directly compare with evaluation data in ELP [35]. Therefore, we only provide
its visual comparisons in Appendix C. Our quantitative comparisons consist of
four parts: 1) precision and diversity of listener motions: as presented in
Table 1, for the realism of listener motion generation, our proposed method
achieves the lowest FD on both $\mathcal{D}_{test}$ and $\mathcal{D}_{ood}$ in
ViCo. Meanwhile, the V-D of our model is comparable to other methods, which
indicates that our model can maintain certain diversity when generating
controllable listener motions. This may benefit from the probabilistic nature
of the diffusion model. 2) speaker-listener synchronization: compared to all
existing methods, CustomListener can decrease the RTLCC and the RWTLCC, which
justifies the effectiveness of our dynamic portrait tokens. 3) temporal
smoothness: in Table 1, the FID△fm and SND of our model outperform other
existing methods by a large margin, which validates that our model can
mitigate un-smooth transitions. 4) image-level quality: although image quality
is not our main focus, we still have a certain performance gain in related
metrics, as shown in Table 2. These results indicate that our generated motion
is closer to GT motion, thus achieving relatively high image-level quality
without special design in rendering.

Figure 5: Qualitative comparisons with PCH [15], RLHG [46] conditioned on the
same speaker and the same listener reference image.

###  4.4 Qualitative Results

#### Visual Results

Since previous methods cannot generate listener head based on fine-grained
texts, we first present visual results generated by CustomListener. As shown
in Figure 4, given different texts, covering diverse listener’s personalities,
identities and attitudes toward the speaker’s utterances, our framework can
produce fine-grained non-verbal listener motions closely aligned with pre-
customized texts. For example, if the listener is an impatient person who is
not interested in the speaker’s words(the 3rd row in Figure 4), he may appear
indifferent and show impatience. Additionally, the produced listener’s
responses can fluctuate with the speaker’s semantics and movement, showing
great effectiveness of our proposed dynamic portrait tokens.

Figure 6: Ablation study of PGM to verify that PGM can maintain motion coherency. Each group includes adjacent four frames, and the frames enclosed by the red box are the two frames at the transition point between two video clips. Method |  FD $\downarrow$ |  RTLCC $\downarrow$ |  RWTLCC $\downarrow$ |  FID △fm$\downarrow$ |  SND $\downarrow$
---|---|---|---|---|---
(x100) |  |  | (x100) |
w/o ATRI | 20.67 | 0.206 | 0.203 | 5.01 | 4.53
w/o SMW | 19.05 | 0.201 | 0.196 | 5.07 | 4.58
w/o SDP | 21.97 | 0.213 | 0.210 | 5.90 | 4.60
w/o PGM | 20.53 | 0.197 | 0.201 | 6.47 | 4.65
Ours | 18.48 | 0.165 | 0.169 | 3.42 | 4.23
Table 3: Ablation study. Each cell represents the average results of 70-dim
motions (64-dim exp + 6-dim pose) on $\mathcal{D}_{test}$ and
$\mathcal{D}_{ood}$.

#### Comparisons with Existing Methods

In Figure 5, we compare our results with RLHG [46], PCH [15]. It can be seen
that our method is capable of generating more realistic listener heads than
other baselines. In the left four columns, both RLHG [46] and PCH [15] suffer
from unnatural motions (e.g., disorderly opened mouth) and visible facial
artifacts. In the right four columns, RLHG [46] still exhibits incorrect
motions and PCH [15] can not display negative attitude accurately. While our
method can produce natural listener head that highly aligns with the ground
truth conditioned on the given texts. More detailed comparisons can be seen in
Appendix C.

#### User Study

We conduct user studies to evaluate from human perceptual perspective. We
randomly choose 25 speaker videos from ViCo[46] and RealTalk[12] to generate
listeners for each method. 28 people are asked to rate different methods(1-4,
4 is the best) for each video from 4 metrics: similarity to GT, overall
naturalness, speaker-listener coordination and video smooth. The average
scores of each method are shown in Table 4. Our method outperformed other
methods in all aspects.

Method | Similarity | Overall | Speaker-listener | Video
---|---|---|---|---
to GT | Naturalness | Coordination | Smooth
RLHG [46] | 1.28 | 1.12 | 1.64 | 2.04
PCH [15] | 2.08 | 2.40 | 2.48 | 2.28
L2L [25] | 2.68 | 2.56 | 1.92 | 1.80
Ours | 3.96 | 3.92 | 3.96 | 3.88
Table 4: User study results. The best results are highlighted in bold. Figure
7: Ablation study of PGM. For different segments, the habit of frowning is
maintained using PGM.

###  4.5 Ablation Study

#### SDP module

In Section 3.2, we design a SDP module to generate dynamic portrait tokens for
realizing progressive motion shifts that correspond well to the speaker. To
validate its effectiveness, we conduct ablation study on Audio-text Responsive
Interaction (ATRI) and Speaker Motion-based Weighting (SMW). The results are
shown in Table 3. In addition to the improvement in FD, the improvement in the
RTLCC and RWTLCC is more pronounced, which demonstrates the SDP module is
crucial for enhancing speaker-listener synchronization, and each of its
components matters.

#### Past-guided module

To validate the effectiveness of our Past-guided module (PGM), we train
networks without PGM. As presented in Table 3, without PGM, the metrics FID△fm
and SND, which are related to the temporal naturalness of motions, increase
significantly, indicating a decrease in smoothness. This observation shows the
necessity of PGM for mitigating unnatural transitions between video clips. The
visual results are shown in Figure 6. Apart from this, our proposed PGM can
maintain the consistency of listener’s customized behavioral habits between
adjacent clips, as shown in Figure 7. The listener is customized a habit of
frowning when thinking, and when segment 2 is guided by “lips mildly parted
and lip corner raised slightly”, the frowning habit can be still maintained.

##  5 Conclusion

In this paper, we present a user-friendly framework called CustomListener,
which can realize freely-controllable listener head generation conditioned on
text priors. To achieve the speaker-listener coordination, we propose a SDP-
module for dynamic portrait-token generation, which is capable of representing
progressive motion changes. To ensure coherent listener responses in a long
video, we design a PGG-module for smooth transition between adjacent video
segments as well as maintaining customized behavioral habits. Comprehensive
experiments demonstrate the superiority of our whole framework. In future
work, it may be interesting to generalize our framework to listening body
generation, yielding a complete and realistic listener agent.

## References

  * Bohus and Horvitz [2010] Dan Bohus and Eric Horvitz.  Facilitating multiparty dialog with gaze, gesture, and speech.  In _International Conference on Multimodal Interfaces and the Workshop on Machine Learning for Multimodal Interaction_ , pages 1–8, 2010.
  * Boker et al. [2002] Steven M Boker, Jennifer L Rotondo, Minquan Xu, and Kadijah King.  Windowed cross-correlation and peak picking for the analysis of variability in the association between behavioral time series.  _Psychological methods_ , 7(3):338, 2002.
  * Cao et al. [2020] Zhe Cao, Hang Gao, Karttikeya Mangalam, Qi-Zhi Cai, Minh Vo, and Jitendra Malik.  Long-term human motion prediction with scene context.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part I 16_ , pages 387–404. Springer, 2020.
  * Cassell and Thorisson [1999] Justine Cassell and Kristinn R Thorisson.  The power of a nod and a glance: Envelope vs. emotional feedback in animated conversational agents.  _Applied Artificial Intelligence_ , 13(4-5):519–538, 1999.
  * Cassell et al. [1994] Justine Cassell, Catherine Pelachaud, Norman Badler, Mark Steedman, Brett Achorn, Tripp Becket, Brett Douville, Scott Prevost, and Matthew Stone.  Animated conversation: rule-based generation of facial expression, gesture & spoken intonation for multiple conversational agents.  In _Proceedings of the 21st annual conference on Computer graphics and interactive techniques_ , pages 413–420, 1994.
  * Chang et al. [2023] Di Chang, Yufeng Yin, Zongjian Li, Minh Tran, and Mohammad Soleymani.  Libreface: An open-source toolkit for deep facial expression analysis.  _arXiv preprint arXiv:2308.10713_ , 2023.
  * Chen et al. [2019] Lele Chen, Ross K Maddox, Zhiyao Duan, and Chenliang Xu.  Hierarchical cross-modal talking face generation with dynamic pixel-wise loss.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 7832–7841, 2019.
  * Chen et al. [2021] Lele Chen, Chen Cao, Fernando De la Torre, Jason Saragih, Chenliang Xu, and Yaser Sheikh.  High-fidelity face tracking for ar/vr via deep lighting adaptation.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 13059–13069, 2021.
  * Deng et al. [2019] Yu Deng, Jiaolong Yang, Sicheng Xu, Dong Chen, Yunde Jia, and Xin Tong.  Accurate 3d face reconstruction with weakly-supervised learning: From single image to image set.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops_ , pages 0–0, 2019.
  * Ekman [2002] Paul Ekman.  Facial action coding system (facs).  _A human face_ , 2002.
  * Feng et al. [2017] Will Feng, Anitha Kannan, Georgia Gkioxari, and C Lawrence Zitnick.  Learn2smile: Learning non-verbal interaction through observation.  In _2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pages 4131–4138. IEEE, 2017.
  * Geng et al. [2023] Scott Geng, Revant Teotia, Purva Tendulkar, Sachit Menon, and Carl Vondrick.  Affective faces for goal-driven dyadic communication.  _arXiv preprint arXiv:2301.10939_ , 2023.
  * Ghosh et al. [2021] Anindita Ghosh, Noshaba Cheema, Cennet Oguz, Christian Theobalt, and Philipp Slusallek.  Synthesis of compositional animations from textual descriptions.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 1396–1406, 2021.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  _Advances in neural information processing systems_ , 33:6840–6851, 2020.
  * Huang et al. [2022] Ailin Huang, Zhewei Huang, and Shuchang Zhou.  Perceptual conversational head generation with regularized driver and enhanced renderer.  In _Proceedings of the 30th ACM International Conference on Multimedia (MM’22)_ , 2022.
  * Ji et al. [2021] Xinya Ji, Hang Zhou, Kaisiyuan Wang, Wayne Wu, Chen Change Loy, Xun Cao, and Feng Xu.  Audio-driven emotional video portraits.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 14080–14089, 2021.
  * Kendon [1970] Adam Kendon.  Movement coordination in social interaction: Some examples described.  _Acta psychologica_ , 32:101–125, 1970.
  * Kim et al. [2023] Jihoon Kim, Jiseob Kim, and Sungjoon Choi.  Flame: Free-form language-based motion synthesis & editing.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 8255–8263, 2023.
  * LaFrance [1979] Marianne LaFrance.  Nonverbal synchrony and rapport: Analysis by the cross-lag panel technique.  _Social Psychology Quarterly_ , pages 66–70, 1979.
  * Li et al. [2021] Ruilong Li, Shan Yang, David A Ross, and Angjoo Kanazawa.  Ai choreographer: Music conditioned 3d dance generation with aist++.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 13401–13412, 2021.
  * Liu et al. [2023] Jin Liu, Xi Wang, Xiaomeng Fu, Yesheng Chai, Cai Yu, Jiao Dai, and Jizhong Han.  Mfr-net: Multi-faceted responsive listening head generation via denoising diffusion model.  In _Proceedings of the 31th ACM International Conference on Multimedia (MM’23)_ , 2023.
  * Liu et al. [2019] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_ , 2019.
  * Loshchilov and Hutter [2017] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_ , 2017.
  * Ma et al. [2023] Yifeng Ma, Suzhen Wang, Yu Ding, Bowen Ma, Tangjie Lv, Changjie Fan, Zhipeng Hu, Zhidong Deng, and Xin Yu.  Talkclip: Talking head generation with text-guided expressive speaking styles.  _arXiv preprint arXiv:2304.00334_ , 2023.
  * Ng et al. [2022] Evonne Ng, Hanbyul Joo, Liwen Hu, Hao Li, Trevor Darrell, Angjoo Kanazawa, and Shiry Ginosar.  Learning to listen: Modeling non-deterministic dyadic facial motion.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 20395–20405, 2022.
  * Nojavanasghari et al. [2018] Behnaz Nojavanasghari, Yuchi Huang, and Saad Khan.  Interactive generative adversarial networks for facial expression generation in dyadic interactions.  _arXiv preprint arXiv:1801.09092_ , 2018.
  * Petrovich et al. [2021] Mathis Petrovich, Michael J Black, and Gül Varol.  Action-conditioned 3d human motion synthesis with transformer vae.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 10985–10995, 2021.
  * Petrovich et al. [2022] Mathis Petrovich, Michael J Black, and Gül Varol.  Temos: Generating diverse human motions from textual descriptions.  In _European Conference on Computer Vision_ , pages 480–497. Springer, 2022.
  * Radford et al. [2023] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever.  Robust speech recognition via large-scale weak supervision.  In _International Conference on Machine Learning_ , pages 28492–28518. PMLR, 2023.
  * Ren et al. [2021] Yurui Ren, Ge Li, Yuanqi Chen, Thomas H Li, and Shan Liu.  Pirenderer: Controllable portrait image generation via semantic neural rendering.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 13759–13768, 2021.
  * Richard et al. [2021] Alexander Richard, Michael Zollhöfer, Yandong Wen, Fernando De la Torre, and Yaser Sheikh.  Meshtalk: 3d face animation from speech using cross-modality disentanglement.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 1173–1182, 2021.
  * Richardson et al. [2008] Daniel Richardson, Rick Dale, and Kevin Shockley.  Synchrony and swing in conversation: Coordination, temporal dynamics, and communication.  _Embodied communication in humans and machines_ , pages 75–94, 2008.
  * Ronneberger et al. [2015] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In _Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18_ , pages 234–241. Springer, 2015.
  * Ruiz et al. [2018] Nataniel Ruiz, Eunji Chong, and James M Rehg.  Fine-grained head pose estimation without keypoints.  In _Proceedings of the IEEE conference on computer vision and pattern recognition workshops_ , pages 2074–2083, 2018.
  * Song et al. [2023] Luchuan Song, Guojun Yin, Zhenchao Jin, Xiaoyi Dong, and Chenliang Xu.  Emotional listener portrait: Neural listener head generation with emotion.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 20839–20849, 2023.
  * Sonlu et al. [2021] Sinan Sonlu, Uğur Güdükbay, and Funda Durupinar.  A conversational agent framework with multi-modal personality expression.  _ACM Transactions on Graphics (TOG)_ , 40(1):1–16, 2021.
  * Van Den Oord et al. [2017] Aaron Van Den Oord, Oriol Vinyals, et al.  Neural discrete representation learning.  _Advances in neural information processing systems_ , 30, 2017.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Vougioukas et al. [2020] Konstantinos Vougioukas, Stavros Petridis, and Maja Pantic.  Realistic speech-driven facial animation with gans.  _International Journal of Computer Vision_ , 128:1398–1413, 2020.
  * Wei et al. [2022] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al.  Chain-of-thought prompting elicits reasoning in large language models.  _Advances in Neural Information Processing Systems_ , 35:24824–24837, 2022.
  * Yang et al. [2023a] Siqi Yang, Zejun Yang, and Zhisheng Wang.  Longdancediff: Long-term dance generation with conditional diffusion model.  _arXiv preprint arXiv:2308.11945_ , 2023a.
  * Yang et al. [2023b] Zhao Yang, Bing Su, and Ji-Rong Wen.  Synthesizing long-term human motions with diffusion models via coherent sampling.  In _Proceedings of the 31st ACM International Conference on Multimedia_ , pages 3954–3964, 2023b.
  * Yang et al. [2023c] Zhao Yang, Bing Su, and Ji-Rong Wen.  Synthesizing long-term human motions with diffusion models via coherent sampling.  In _Proceedings of the 31st ACM International Conference on Multimedia_ , pages 3954–3964, 2023c.
  * Yu et al. [2023] Zhentao Yu, Zixin Yin, Deyu Zhou, Duomin Wang, Finn Wong, and Baoyuan Wang.  Talking head generation with probabilistic audio-to-visual diffusion priors.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 7645–7655, 2023.
  * Zhang et al. [2021] Zhimeng Zhang, Lincheng Li, Yu Ding, and Changjie Fan.  Flow-guided one-shot talking face generation with a high-resolution audio-visual dataset.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3661–3670, 2021.
  * Zhou et al. [2022] Mohan Zhou, Yalong Bai, Wei Zhang, Ting Yao, Tiejun Zhao, and Tao Mei.  Responsive listening head generation: A benchmark dataset and baseline.  In _Proceedings of the European conference on computer vision (ECCV)_ , 2022.

Supplementary Material

##  Appendix A Implementation Details

###  A.1 Network Details

We introduced Audio Encoder, Mapping Net, Motion Encoder and Transformer
Decoder-based Diffusion Model in Section 3 of our main paper. Due to the page
limitation, we show implementation details of these modules in this Appendix.

#### Audio Encoder

For audio preprocessing, we first extract 45-dim acoustic features from
speaker audio, and then encode these acoustic features into audio features.
The structure of Audio Encoder is shown in Figure 8, which consists of two
convolution layers, a GELU activation function, six 768-dim transformer
layers, and a linear layer.

#### Mapping Net

As discussed in Section 3.2 of the main paper, in order to obtain portrait-
related tokens, we further employ a Mapping Net to convert text embeddings
from RoBERTa [22] into static portrait tokens, which describes the listener’s
static portrait (e.g. expressions, head movements) in the entire video
segment. The structure of Mapping Net is shown in Figure 8.

#### Motion Encoder

We only use a single linear layer to encode the 70-dim noised listener motions
to 768-dim motion tokens (denoted with small blue squares in Figure 2 of our
main paper).

#### Transformer Decoder-based Diffusion Model

We employ transformer decoder as our basic denoising module for two reasons:
First, compared to U-Net [33] used in traditional diffusion model, transformer
[38] can better capture temporal information in listener motion sequences.
Furthermore, compared to transformer encoder which only contains self-
attention, transformer decoder is capable of facilitating information
interaction better between noised listener motion features as well as
concatenated dynamic portrait token and motion prior. Specifically, we use an
8-layer transformer decoder, with 8 heads and 2048-dim fully forward layers.
Additionally, we concatenate the input motion tokens with diffusion time-step
token (denoted with small pink squares in Figure 2 of our main paper) to
inject time-step information.

Figure 8: Stucture of Audio Encoder and Mapping Net.

###  A.2 Inference Details

#### Long-video Inference

During inference, we generate long-term video in a segment-by-segment way.
Specifically, we first sequentially clip the original long video to several
60-frame video segments, containing video frames, audios and speech content.
We then input user-customized listener attributes. Subsequently, by
incorporating speech content of each segment as well as pre-customized
attribute description, text priors of each segment will be produced with the
help of GPT. Finally, our proposed CustomListener will generate long-term
listener motions clip-by-clip.

Figure 9: Details of Text Prior Generation. Figure 10: Illustration of Chain-
of-thought Prompting.

#### Text Prior Generation

To generate text prior in specific format like [A person $<$EMOTION$>$ and
listens with $<$AU$>$ (and $<$HEAD MOTION$>$)]), inspired by [12], we use
chain-of-thought prompting [40] when querying GPT. Specifically, before asking
question, we first provide GPT with a few examples showing the desired answer
format as well as reasoning process. This strategy can also enable GPT to
provide more accurate text prior about facial expressions and head motions
while reducing the misunderstanding probability towards the speaker’s content.
We present some examples of queries in Figure 9 and Figure 10.

After getting the text prior, we randomly replace the adjectives and adverbs
to enhance the diversity of description. For example, for $<$EMOTION$>$, if it
is 'happy', we replace it by randomly selecting one from a group $<$looks
happy, seems joyful, …$>$. For $<$AU$>$, following Facial Action Coding
System, if AU12 (lip corner puller) is activated and level=3, we randomly
choose an adj. from $<$raised, pulled, …$>$, an adv. from $<$fully, extremely,
…$>$, then combine the above and get [ A person seems joyful and listens with
fully raised lip corners.].

Methods |  FD $\downarrow$ |  RTLCC $\downarrow$ |  RWTLCC$\downarrow$ |  FID△fm$\downarrow$ |  SND $\downarrow$ |  V-D $\uparrow$
---|---|---|---|---|---|---
exp | angle | trans | exp | pose | exp | pose | exp | pose | exp | pose | exp | pose
RLHG∗ [46] | 20.11 | 10.32 | 7.16 | 0.162 | 0.365 | 0.160 | 0.363 | 12.42 | 0.76 | 4.75 | 0.09 | 0.54 | 0.14
PCH∗ [15] | 23.07 | 13.46 | 8.04 | 0.170 | 0.368 | 0.163 | 0.391 | 12.38 | 0.81 | 5.02 | 0.11 | 0.29 | 0.10
L2L∗ [25] | 19.01 | 10.16 | 7.03 | 0.158 | 0.319 | 0.156 | 0.315 | 10.80 | 0.71 | 4.67 | 0.10 | 0.56 | 0.40
Ours | 17.63 | 9.30 | 6.49 | 0.138 | 0.218 | 0.134 | 0.207 | 7.70 | 0.14 | 4.37 | 0.07 | 2.90 | 1.01
Table 5: Comparisons of our model with other methods on $\mathcal{D}_{test}$ of RealTalk. Bold represents the best. The ∗ denotes we retrain the model. The $\downarrow$ indicates lower is better and the $\uparrow$ indicates higher is better. The values of FD and FID△fm are multiplied by 100. Method |  SSIM $\uparrow$ |  CPBD $\uparrow$ |  PSNR $\uparrow$ |  FID $\downarrow$
---|---|---|---|---
RLHG∗ [46] | 0.51 | 0.29 | 15.34 | 30.22
PCH∗ [15] | 0.56 | 0.32 | 16.13 | 24.68
L2L∗ [25] | 0.56 | 0.31 | 15.98 | 25.68
Ours | 0.58 | 0.34 | 16.01 | 23.86
Table 6: Quantitative results with other methods on image quality. The ∗
denotes we retrain the model. The best results are highlighted in bold.

##  Appendix B Quantitative Results on RealTalk

We did not present quantitative results on RealTalk in main paper due to page
limitation, thus we give these results in the Appendix B. For a fair
comparison, we retrain PCH [15], RLHG [46] and L2L [25] on RealTalk dataset.
Due to the unavailability of source codes, we cannot compare with MFR-Net [21]
and ELP [35] on RealTalk. Notably, since PCH [15] and RLHG [46] need
listener’s attitude label as input, we use [6] to obtain emotion labels of
each video clip. Then, we group these emotion labels into three basic
attitudes: positive, negative and neutral. As shown in Table 5, we present the
FD, RTLCC, RWTLCC, FID△fm, SND and V-D of our model as well as other methods.
Our proposed CustomListener achieves best performance on RealTalk across all
metrics, which demonstrates the superiority of CustomListener in generating
realistic and coherent listener motions that highly synchronous with the
speaker’s motions. In Table 6, we also present evaluation results related to
image-level quality on RealTalk. Our model achieves optimal results, which
indicates CustomListener can generate relatively high-quality listener images
without special design in the render module, thus justifies our generated
motions are highly precise.

##  Appendix C Supplementary Visual Results

###  C.1 Visual Comparisons with Other Methods

In Figure 11, we compared our results with PCH [15], RLHG [46] and ELP [35].
As there is no source code provided by ELP [35], for fairly comparison, we
first find the speaker video showed in ELP [35], and then generate listener
head based on the speaker video and corresponding description texts instead of
an attitude label. Then, we displayed four generated listener frames, which
are aligned with the four speaker frames in time. As shown in the visual
comparisons, our method has two advantages. Firstly, without any special
design in render model, our generated listener images showed less facial
artifacts and more closed to the ground truth, which indicates the listener
motions produced by CustomListener are highly aligned with the ground-truth
motions. Secondly, without a blink modeling like ELP [35], our generated
listener can perform blinking action, which demonstrates that the listener
motions are natural-looking and photorealistic. More visual comparisons with
PCH [15], RLHG [46] and L2L [25] on ViCo and RealTalk are presented in Figure
12 and Figure 13, respectively.

Figure 11: Qualitative comparisons with PCH [15], RLHG [46] and ELP [35].
Figure 12: Qualitative comparisons with other methods on ViCo dataset. Figure
13: Qualitative comparisons with other methods on RealTalk dataset. Figure 14:
Ablation study of SDP module.

###  C.2 Visual Results of Ablation

In Section 4.5 of our main paper, we presented visual ablation of PGM module.
Due to the page limitation, we show the visual ablation of SDP module in this
subsection. As shown in Figure 14, with the help of SDP module, our generated
listener motions can not only realize progressive motion changes, but also
fluctuate with the speaker’s semantics, intonation and movement amplitude, and
thus achieve the speaker-listener coordination.

###  C.3 Supplementary Video

We provide a supplementary video to present more visual results, which include
visual videos on ViCo and RealTalk, more visual comparisons with other methods
as well as ablation study of each component. The video can be found in the
“Supplementary_Videos.mp4” in the Supplementary Materials. These videos are
also displayed on the project page.


# Active crystallization from power functional theory

Sophie Hermann<EMAIL_ADDRESS>Theoretische Physik II,
Physikalisches Institut, Universität Bayreuth, D-95447 Bayreuth, Germany
Matthias Schmidt<EMAIL_ADDRESS>Theoretische Physik II,
Physikalisches Institut, Universität Bayreuth, D-95447 Bayreuth, Germany
<EMAIL_ADDRESS>
(21 August 2023)

###### Abstract

We address the gas, liquid, and crystal phase behaviour of active Brownian
particles in three dimensions. The nonequilibrium force balance at coexistence
leads to equality of state functions for which we use power functional
approximations. Motility-induced phase separation starts at a critical point
and quickly becomes metastable against active freezing for Péclet numbers
above a nonequilibrium triple point. The mean swim speed acts as a state
variable, similar to the density of depletion agents in colloidal demixing. We
obtain quantitative agreement with recent simulation results and correctly
predict the strength of particle number fluctuations in active fluids.

The occurrence of freezing in a many-body system is often due to the presence
of strong, short-ranged repulsion between the constituent particles hansen2013
; tarazona2000prl . Conditions of high enough density are required for
crystallization as a global ordering phenomenon to occur and these can be
induced by external constraints, such as confinement by walls, or via
interparticle attraction leunissen2005 ; hueckel2020 . In colloidal systems,
attraction between the particles can be generated by adding depletion agents,
such as polymers, colloidal rods, or smaller-sized colloidal spheres. The
depletants create an effective attraction between the primary particles and
the resulting effective interaction potential is accessible via formally
integrating out (averaging over) the depletant degrees of freedom likos2001
and machine learning campos2021ml ; campos2022ml . In general the resulting
interaction potential has a strong many-body character, although notable
exceptions exist, such as the Asakura-Oosawa model binder2014 ; miyazaki2022 ;
dijkstra1999 ; schmidt2000cip , where for sufficiently small polymer-to-
colloid size ratio a description based on an effective pair potential is exact
dijkstra1999 .

In a striking analogy, Turci and Wilding turci2021  have recently related the
phase behaviour of three-dimensional active Brownian particles (ABPs)
turci2021 ; omar2021  to such depletion-driven binary mixtures. ABPs form a
central model system for active matter and their phase behaviour has received
much prior attention bialke2012 ; wysocki2014 ; paliwal2018 ;
solon2018generalized ; moore2021 ; speck2021pre ; speck2022critical ,
including the two-dimensional version of the model bialke2012 ; speck2021pre ;
speck2022critical . The particles undergo overdamped Brownian motion and they
self-propel (swim) along a built-in direction, which diffuses freely. The
system displays motility-induced phase separation (MIPS) into dense and dilute
coexisting nonequilibrium steady states, despite of the absence of explicit
interparticle attraction. The phenomenon was addressed on the basis of a wide
variety of theoretical techniques paliwal2018 ; solon2018generalized ;
speck2021pre ; speck2022critical ; bickmann2020 , including very recent work
by Omar et al. omar2022mips  based on forces. However, none of these
approaches has yet been applied to active freezing.

Despite the significant number of theoretical efforts paliwal2018 ;
solon2018generalized ; speck2021pre ; speck2022critical ; bickmann2020 ;
omar2022mips ; szamel2016 , no consensus has been reached on a common
framework which would act as an uncontested platform for the description of
active systems, such as the theory of simple liquids for spatially
inhomogeneous and phase-separated systems does in equilibrium hansen2013 ;
evans1979 ; evans2016specialIssue ; roth2010review . It is a rather common
point of view that “the link between experiment and theory in active matter is
often rather qualitative” mauleonamieva2020 . Having a predictive theory is
highly valuable though, given that much relevant experimental work is being
carried out, e.g. based on light-controlled systems palacci2013 , as also used
in studies of active polarization soeker2021prl ; auschra2021 , cluster
formation kuemmel2015 , the self-propulsion mechanism of Quincke rollers
mauleonamieva2020 , the experimental study of active sedimentation ginot2018 ,
capillary rise carreira2023verticalWall , and polycrystallinity klongvessa2019
. Equally so, simulations studies of wetting turci2021wetting , vortex crystal
formation james2021vortexCrystal , inertial effects in nematic turbulence
koch2022activeTurbulence , interfacial properties chacon2022  and of dynamical
features martinezroca2021  of active particles could benefit from having a
predictive theory.

In this Letter we use power functional theory schmidt2022rmp , which is a
general framework for the description of the dynamics of many-body systems,
including ABPs schmidt2022rmp ; krinninger2016 ; krinninger2019 ;
hermann2019pre ; hermann2021molPhys ; hermann2019prl . We base our treatment
of freezing on the active force balance, as used in studies of active drag
forces krinninger2016 ; krinninger2019 , motility-induced phase separation
hermann2019pre ; hermann2021molPhys  and the interfacial tension between
phase-separated states hermann2019prl  in two-dimensional ABPs. The theory
satisfies exact sum rules which result from Noether’s theorem for correlation
functions hermann2021noether ; sammueller2023whatIsLiquid  as well as from the
continuity equation for the global polarization hermann2020polarization . We
demonstrate that the framework gives a physically sound and semi-quantitative
account of the full phase behaviour of ABPs in three dimensions. In their
analogy, Turci and Wilding turci2021  suggest that the Péclet number, which
measures the strength of the self-propulsion in the active system relative to
diffusive motion, is akin to the depletants’ fugacity (or polymer reservoir
density) in an equilibrium mixture. We confirm and extend this point of view,
as in our theoretical approach the mean swim speed plays a role akin to the
actual polymer density in the system.

We work on the level of one-body correlation functions, which depend on
position ${\bf r}$ and on particle orientation, as represented by a unit
vector ${\bm{\omega}}$. The continuity equation relates the divergence of the
translational current ${\bf J}({\bf r},{\bm{\omega}},t)$ and of the rotational
current ${\bf J}^{\omega}({\bf r},{\bm{\omega}},t)$ to temporal changes of the
one-body density distribution according to:

$\displaystyle\frac{\partial\rho({\bf r},{\bm{\omega}},t)}{\partial t}$
$\displaystyle=-\nabla\cdot{\bf J}({\bf
r},{\bm{\omega}},t)-\nabla^{\omega}\cdot{\bf J}^{\omega}({\bf
r},{\bm{\omega}},t).$ (1)

Here $\nabla$ and $\nabla^{\omega}$ indicate the derivatives with respect to
${\bf r}$ and ${\bm{\omega}}$, respectively, and the density profile
$\rho({\bf r},{\bm{\omega}},t)$ is position- and orientation-resolved. We
consider steady states such that the left hand side of Eq. (1) vanishes and we
drop the time argument $t$ from here on. As no explicit torques act in the
system, the orientational current stems solely from the free rotational
diffusion of the active spheres: ${\bf J}^{\omega}({\bf
r},{\bm{\omega}})=-D_{\rm rot}\nabla^{\omega}\rho({\bf r},{\bm{\omega}})$,
where $D_{\rm rot}$ indicates the rotational diffusion constant. For the
present case of overdamped active motion, the exact force balance is given by:

$\displaystyle\gamma{\bf v}({\bf r},{\bm{\omega}})$ $\displaystyle={\bf
f}_{\rm id}({\bf r},{\bm{\omega}})+{\bf f}_{\rm int}({\bf
r},{\bm{\omega}})+\gamma s{\bm{\omega}}.$ (2)

The left hand side represents the negative friction force with friction
constant $\gamma$ and the velocity field is the ratio of current and density,
${\bf v}({\bf r},{\bm{\omega}})={\bf J}({\bf r},{\bm{\omega}})/\rho({\bf
r},{\bm{\omega}})$. The three driving contributions on the right hand side of
Eq. (2) are the ideal diffusive force field ${\bf f}_{\rm id}({\bf
r},{\bm{\omega}})=-k_{B}T\nabla\ln\rho({\bf r},{\bm{\omega}})$, the internal
force field ${\bf f}_{\rm int}({\bf r},{\bm{\omega}})$, which arises from the
Weeks-Chandler-Anderson (WCA) interparticle interactions, and the swim force
$\gamma s{\bm{\omega}}$ with $s$ indicating the speed of free swimming. The
one-body interparticle interaction force field ${\bf f}_{\rm int}({\bf
r},{\bm{\omega}})$ is accessible via sampling in simulations krinninger2016 ;
krinninger2019 ; hermann2019pre ; hermann2021molPhys  and via machine-
learning, as recently demonstrated in passive flow delasheras2023perspective
and in equilibrium sammueller2023neural .

We split the interparticle forces according to hermann2019prl ;
delasheras2020fourForces :

$\displaystyle{\bf f}_{\rm int}({\bf r},{\bm{\omega}})$ $\displaystyle={\bf
f}_{\rm ad}({\bf r})+{\bf f}_{\rm flow}({\bf r},{\bm{\omega}})+{\bf f}_{\rm
struc}({\bf r},{\bm{\omega}}),$ (3)

where the right hand side consists of the adiabatic force field ${\bf f}_{\rm
ad}({\bf r})$, the superadiabatic flow force field ${\bf f}_{\rm flow}({\bf
r},{\bm{\omega}})$ and the superadiabatic structural force field ${\bf f}_{\rm
struc}({\bf r},{\bm{\omega}})$. Here the adiabatic force field ${\bf f}_{\rm
ad}({\bf r})$ is defined as acting in an equilibrium system of passive WCA
particles that do not swim. The WCA particles are spheres and hence there is
no nontrivial dependence on ${\bm{\omega}}$ in the adiabatic system. Its
density distribution $\bar{\rho}({\bf r})$ is identical to the orientation-
integrated density distribution in the active system. In the adiabatic system
$\bar{\rho}({\bf r})$ is stabilized by an external potential.

If one wishes to think in terms of functional dependencies, then ${\bf f}_{\rm
ad}({\bf r})$ is an instantaneous density functional, in the sense of
functional dependencies as they form the core of classical density functional
theory of inhomogeneous liquids and solids evans1979 ; hansen2013 ;
evans2016specialIssue ; roth2010review ; schmidt2022rmp . Both the flow and
the structural force contributions in Eq. (2) are of superadiabatic nature,
i.e. they are genuine nonequilibrium force fields which arise from the
interparticle interactions schmidt2022rmp . In equilibrium, as well as in
passive uniaxial flow, the three force contributions were shown to be amenable
to supervised machine learning delasheras2023perspective ;
sammueller2023neural , which we take as confirmation of the general force
splitting concept (3), as is here applied to the active system.

Figure 1: Phase diagram for three-dimensional ABPs. (a) Theoretical result as
a function of the scaled bulk density $\rho_{b}\sigma^{3}$ and the scaled
average swim speed $v_{b}\gamma\sigma/\epsilon$. Shown are stable (solid
lines) and metastable (dashed lines) binodals; slanted tielines connect
coexisting nonequilibrium states. The orange dotted line indicates the line of
maximal compressibility $\chi_{\rm max}$. Note the similarity to a phase
diagram of a colloid-polymer mixture (inset, adapted from Ref. schmidt2000cip
) as a function of the colloid (polymer) packing fraction
$\eta^{C}(\eta^{P})$. (b) Same as (a) but shown as a function of
$\rho_{b}\sigma^{3}$ and the Péclet number Pe. The tie lines are horizontal in
this representation. (c) Same as (b) but obtained from computer simulations in
Ref. turci2021 . Shown are active gas-active fluid (circles), active gas-
crystal (triangles) coexistence densities as well as $\chi_{\rm max}$
(diamonds). The inset is a schematic phase diagram for a polymer-colloid
mixture with size ratio $q=0.6$, taken from Ref. dijkstra1999 .

The flow force ${\bf f}_{\rm flow}({\bf r},{\bm{\omega}})$, as it is part of
Eq. (3), is defined to compensate the friction and the active force in the
force balance relationship (2) such that equality is achieved:

$\displaystyle\gamma{\bf v}({\bf r},{\bm{\omega}})$ $\displaystyle={\bf
f}_{\rm flow}({\bf r},{\bm{\omega}})+\gamma s{\bm{\omega}}.$ (4)

The flow equation (4) is invariant under motion reversal schmidt2022rmp ;
delasheras2020fourForces ; delasheras2023perspective  and it affects the
spatial structure formation as represented by the density profile, only
indirectly, as we detail below. As an approximation we resort to the
superadiabatic drag force of Ref. krinninger2016  with the simple form ${\bf
f}_{\rm flow}({\bm{\omega}})=-\gamma
v_{b}{\bm{\omega}}\rho_{b}/(\rho_{j}-\rho_{b})$, where $v_{b}={\bf
v}\cdot{\bm{\omega}}$ is the mean forward swim speed. This assumption yields
the common linear relationship of the mean swim velocity and the average
density, $v_{b}/s=1-\rho_{b}/\rho_{j}$. The parameter $\rho_{j}=\rm const$
determines the slope of the decay of $v_{b}$ with bulk density and we adjust
its value to $\rho_{j}=1.436\sigma^{-3}$, where $\sigma$ is the lengthscale of
the WCA pair potential.

The superadiabatic structural force field ${\bf f}_{\rm sup}({\bf
r},{\bm{\omega}})$ balances the remaining adiabatic and ideal terms in Eq.
(2), which implies the following force cancellation:

$\displaystyle 0$ $\displaystyle={\bf f}_{\rm id}({\bf r})+{\bf f}_{\rm
ad}({\bf r})+{\bf f}_{\rm struc}({\bf r}).$ (5)

As a consistency check, the sum of Eqs. (4) and (5) recovers the full force
balance relationship (2). The ideal term is generally numerically small, and
we hence approximate the exact ideal force $-k_{B}T\nabla\ln\rho({\bf
r},{\bm{\omega}})\approx-k_{B}T\nabla\ln\bar{\rho}({\bf r})\equiv{\bf f}_{\rm
id}({\bf r})$, where as before $\bar{\rho}({\bf r})$ is the position-dependent
and orientation-averaged one-body density profile. Equation (5) balances the
repulsion that acts in the adiabatic system with the nonequilibrium force
contributions. We recall that the adiabatic system consists of steeply
repulsive spheres without orientations. Hence the structural nonequilibrium
forces necessarily need also be independent of orientation, ${\bf f}_{\rm
struc}({\bf r})$, in order to satisfy Eq. (5).

As all force fields in Eq. (5) are of gradient nature [the non-gradient forces
are contained in Eq. (4)], we can integrate in position and obtain the
following chemical potential balance:

$\displaystyle\mu_{\rm id}({\bf r})+\mu_{\rm ad}({\bf r})+\mu_{\rm struc}({\bf
r})$ $\displaystyle=\mu.$ (6)

Here $\mu=\rm const$ arises from the spatial integration. All terms on the
left hand side of Eq. (6) are solely defined by generating via spatial
differentiation the (negative) force contributions that occur in the
structural force balance (5). Explicitly, we have ${\bf f}_{\rm id}({\bf
r})=-\nabla\mu_{\rm id}({\bf r})$, with the standard ideal gas chemical
potential expression: $\mu_{\rm id}({\bf r})=-k_{B}T\ln\bar{\rho}({\bf r})$;
the adiabatic force field: ${\bf f}_{\rm ad}({\bf r})=-\nabla\mu_{\rm ad}({\bf
r})$; and the superadiabatic structural force field: ${\bf f}_{\rm struc}({\bf
r})=-\nabla\mu_{\rm struc}({\bf r})$.

Up to having neglected the orientation-dependence of the ideal gas
contribution and the assumption of the specific simple form of ${\bf f}_{\rm
flow}({\bm{\omega}})$, the framework thus far developed is exact and we have
to resort to approximations to make further progress. We first turn to the
adiabatic contribution. The adiabatic state is simply the equilibrium WCA
model, which per se has no gas-liquid coexistence due to its lack of
interparticle attraction. Treating fluid states of repulsive spheres is
straightforward. We approximate the system by hard spheres and use a modified
Carnahan-Starling equation of state paricaud2015saft , which correctly
accounts for the behaviour at very high densities, as is relevant for ABPs in
the parameter regime considered here. The corresponding bulk excess free
energy $A_{\rm ad}$ is given by

$\displaystyle\frac{A_{\rm ad}}{Nk_{B}T}$ $\displaystyle=\frac{3\eta}{1-\eta}$
$\displaystyle+\eta\left\\{(1-\eta)\left[(1-\eta\left(1+\frac{1-\eta_{j}}{\eta_{j}}{\rm
e}^{b(\eta-\eta_{j})}\right)\right]\right\\}^{-1}$ (7)

where $\eta=\pi\sigma^{3}\rho_{b}/6$ is the packing fraction, $\eta_{j}=0.655$
is the densest possible packing fraction in this approximation and setting
$b=50$ is an empirical choice paricaud2015saft . An analytical expression for
the bulk chemical potential in the adiabatic system then follows from the
standard identity $\mu_{\rm ad}^{b}(\rho_{b})=[A_{\rm ad}+\eta\partial A_{\rm
ad}/\partial\eta]/N$. We use a local density approximation where we evaluate
the bulk expression at the value of the local density profile, i.e. $\mu_{\rm
ad}({\bf r})=\mu_{\rm ad}^{b}(\bar{\rho}({\bf r}))$.

In order to approximate the equation of state of the adiabatic crystal we
resort to the cell theory hoover1967 ; hoover1968 ; frenkel1984 ;
royall2023review . This yields the chemical potential of the fcc crystal as
$\beta\mu_{\rm
cell}=\ln(\sqrt{2})+3\ln(\Lambda/\sigma)-3\ln(\xi-1)+\xi/(\xi-1)$, where
$\xi=[\pi\sqrt{2}/(6\eta)]^{1/3}$ and $\Lambda$ is the thermal de Broglie
wavelength which we set to $\Lambda=\sigma$. We only take account of the mean
crystal density, and set $\mu_{\rm ad}+\mu_{\rm id}=\mu_{\rm cell}$ for the
treatment of the crystalline phase.

The remaining task is to approximate the superadiabatic structural chemical
potential contribution, $\mu_{\rm struc}({\bf r})$. Here we resort to the
“quiet life” approximation, which was successfully used to describe active
gas-liquid phase separation in two dimensions, along with the force balance
across the free interface between the active bulk states hermann2019pre ;
hermann2021molPhys . This approximation takes into account, in arguably the
simplest correct way, the dependence on both the local density and on the
local velocity. As the force is structural, it is necessarily even in powers
of the velocity. A simple choice which is linear in density and quadratic in
velocity hermann2019pre ; hermann2021molPhys  reads as:

$\displaystyle\mu_{\text{struc}}({\bf r})$
$\displaystyle=\frac{e_{1}\gamma}{6D_{\text{rot}}}v^{2}\frac{\bar{\rho}({\bf
r})}{\rho_{j}},$ (8)

where $e_{1}=0.285$ is a constant that determines the overall strength.
Crucially, we use the same approximation (8) for $\mu_{\rm struc}$ in all
three phases.

Nonequilibrium phase coexistence is obtained via the mechanical balance of the
total force, which in our framework implies equality of the values of the
chemical potential, see Eq. (6), in the coexisting phases, as well as the
equality of the pressure. The pressure is obtained from integrating the
standard relation $\rho_{b}\partial\mu(\rho_{b})/\partial\rho_{b}=\partial
P(\rho_{b})/\partial\rho_{b}$. The resulting phase diagram is shown in Fig. 1
as a function of the bulk density $\rho_{b}$ and either the mean swim speed
$v_{b}$ (a) or the free swim speed $s$ (b), as expressed in scaled form by the
Péclet number ${\rm Pe}=s\sigma\gamma/(k_{B}T)$. The topology of the phase
diagram matches that obtained in simulation work turci2021 ; omar2021  and the
quantitative agreement with the simulation results by Turci and Wilding
turci2021  is very satisfactory; their results for the phase diagram are
displayed in Fig 1(c). Our theory reproduces the marginal stability omar2021
of active gas-liquid coexistence with respect to freezing into a dense fcc
crystal. The coexisting gas has relatively high density, in stark contrast to
the strong dilution of the coexisting gas that occurs quickly in equilibrium
phase separation when moving away from the triple point.

Figure 2: Compressibility $\chi(\rho_{b})$, scaled by its value $\chi(0)$ in
the active ideal gas, as a function of the scaled bulk density
$\rho_{b}/\rho_{c}$, where $\rho_{c}$ is the density at the MIPS critical
point. The theoretical results in panel (a) are obtained from differentiating
$\chi(\rho_{b})=\partial\rho(\mu)/\partial\mu|_{T,\rm Pe}$, where $\mu$ is the
(total) nonequilibrium chemical potential. Results are shown for a sequence of
Péclet numbers (as indicated), scaled by the value at the critical point.
Panel (b) shows the corresponding simulation results of Ref. turci2021 ; for
the purpose of this comparison, we take ${\rm Pe}_{c}=36$, $\rho_{c}=0.94$ in
simulation turci2021  and ${\rm Pe}_{c}=37.6$, $\rho_{c}=0.71$ for the theory.
The lines in (b) connect the data points to guide the eye.

On the basis of the similarity of their simulation results for the active
system to depletion-induced phase behaviour in equilibrium (comparing the main
plot and inset of Fig. 1(c)), Turci and Wilding turci2021  draw conclusions
about the presence and relevance of effective many-body interactions that
govern the active system. While it is well-established that in active systems
the Péclet number plays a role similar to that of temperature in equilibrium,
the proposal by Turci and Wilding leaves open whether one should think of the
activity as only generating many-body effects that are akin to those of
depletants or whether the active system contains actual degrees of freedom
that have not been properly appreciated.

Based on the success of our theory, we argue that the latter is the case and
that besides the density, the velocity field is an intrinsic degree of freedom
that the nonequilibrium system can regulate freely and self-consistently. To
demonstrate the validity of this concept, we use the actual mean velocity
$v_{b}$ instead of Pe as a state variable in Fig. 1(a). The swim speed $v_{b}$
is high in the coexisting gas, low in the coexisting liquid, and even lower in
the coexisting crystal. The latter property is consistent with Caprini et al.
reporting very low swim speeds in solid clusters of the two-dimensional ABP
system, see the Supplemental Material of Ref. caprini2020 . This behaviour is
analogous to what is found in depletion-driven phase separation, when going
from the reservoir density of the depletant to the actual depletant density in
the system dijkstra1999 ; schmidt2000cip . The observed similarity in the form
of the phase diagram is striking, compare the main plot and the inset of Fig.
1(a).

We next investigate whether our proposed theory is predictive beyond the phase
diagram. In their simulation work turci2021 , Turci and Wilding have
investigated the statistics of particle number fluctuations, as they occur in
small virtual subboxes of the global system. The strength of fluctuations
$\chi(\rho_{b})$ is taken to be a proxy for the compressibility, as can in
equilibrium be obtained from the thermodynamical derivative
$\partial\rho_{b}(\mu)/\partial\mu$, carried out in the grand ensemble where
global particle number fluctuations occur. These fluctuations are absent in
the present system, as the particle number is conserved in time [we recall the
validity of even the locally resolved continuity equation (1)].

Within our nonequilibrium framework the partial derivative
$\chi(\mu_{b})=\partial\rho_{b}(\mu)/\partial\mu$ is well-defined. Here $\mu$
is the total chemical potential and we recall its splitting (6) into adiabatic
and superadiabatic contributions. We invert via
$\chi(\mu_{b})=[\partial\mu(\rho_{b})/\partial\rho_{b}]^{-1}$ with the
derivative taken at $T,{\rm Pe}={\rm const}$. We normalize with respect to the
low density behaviour, $\chi(\rho_{b})/\chi(0)$, as has also been done in the
simulations turci2021 . In order to create further common ground we scale the
density axis by the respective value of the critical density $\rho_{c}$. From
the setup of the theory, we expect $\chi(\rho_{b})/\chi(0)$ to be a measure of
particle fluctuations and we show numerical results in Fig. 2(a) as a function
of $\rho_{b}/\rho_{c}$ for a range of different values of ${\rm Pe}/{\rm
Pe}_{c}<1$, where ${\rm Pe}_{c}$ indicates the critical value of the Peclet
number. We find that the theory produces the same bell-shaped variation upon
increasing density at fixed Pe, as is apparent in the simulation results
reproduced in Fig. 2(b). The maximum becomes much more pronounced upon
increasing ${\rm Pe}/{\rm Pe}_{c}$ and the theoretical prediction consistently
diverges at the nonequilibrium critical point. The position of the maximum of
$\chi(\rho_{b})/\chi(0)$ traces a line in the phase diagram. The result is
shown in Fig. 1, which again agrees very well with the simulation data
(compare the orange line in in Fig. 1(b) with the orange symbols (c)).

In conclusion we have investigated the nonequilibrium phase behaviour of ABPs
in three dimensions, based on power functional concepts. The central
assumption is that the formally exact nonequilibrium force balance
relationship contains a nonequilibrium structural force contribution, as
obtained by the negative spatial gradient of a corresponding superadiabatic
chemical potential, Eq. (8). We have shown that the theory predicts the phase
diagram correctly and that nonequilibrium particle number fluctuations are
described in agreement with the observations in simulations. We envisage that
going beyond the simple cell theory for the description of the crystal is
possible with classical density functional theory haering2015  based on
fundamental measure theory tarazona2000prl ; roth2010review  as used to study
the direct correlation function in crystals lin2021dcf . Given the recent
progress in measurement of intercolloidal forces in gel states dong2022 , it
seems not inconceivable that experiments can shed further light on active
forces.

###### Acknowledgements.

We thank Francesco Turci for sending us the simulation data of Ref. turci2021
and him, Nigel Wilding, and Daniel de las Heras for useful and inspiring
discussions.

## References

  * (1) J. P. Hansen and I. R. McDonald, Theory of Simple Liquids, 4th ed. (Academic Press, London, 2013).
  * (2) P. Tarazona, Density functional for hard sphere crystals: a fundamental measure approach, Phys. Rev. Lett. 84, 694 (2000).
  * (3) M. E. Leunissen, C. G. Christova, A.-P. Hynninen, C. P. Royall, A. I. Campbell, A. Imhof, M. Dijkstra, R. van Roij, and A. van Blaaderen, Ionic colloidal crystals of oppositely charged particles, Nature 437, 235 (2005).
  * (4) T. Hueckel, G. M. Hocky, J. Palacci, S. Sacanna, Ionic solids from common colloids, Nature 580, 487 (2020).
  * (5) C. N. Likos, Effective interactions in soft condensed matter physics, Phys. Repo. 348, 267 (2001).
  * (6) G. Campos-Villalobos, E. Boattini, L. Filion, and M. Dijkstra, Machine learning many-body potentials for colloidal systems, J. Chem. Phys. 155, 174902 (2021).
  * (7) G. Campos-Villalobos, G. Giunta, S. Marín-Aguilar, and M. Dijkstra, Machine-learning effective many-body potentials for anisotropic particles using orientation-dependent symmetry functions, J. Chem. Phys. 157, 024902 (2022).
  * (8) K. Miyazaki, K. S. Schweizer, D. Thirumalai, R. Tuinier, and E. Zaccarelli, The Asakura–Oosawa theory: entropic forces in physics, biology, and soft matter, J. Chem. Phys. 156, 080401 (2022).
  * (9) K. Binder, P. Virnau, and A. Statt, Perspective: the Asakura Oosawa model: a colloid prototype for bulk and interfacial phase behavior, J. Chem. Phys. 141, 559 (2014).
  * (10) M. Dijkstra, J. M. Brader, and R. Evans, Phase behaviour and structure of model colloid-polymer mixtures, J. Phys. Condens. Matter 11, 10079 (1999).
  * (11) M. Schmidt, H. Löwen, J. M. Brader, and R. Evans, Density functional for a model colloid-polymer mixture, Phys. Rev. Lett 85, 1934 (2000).
  * (12) F. Turci and N. B. Wilding, Phase separation and multibody effects in three-dimensional active Brownian particles, Phys. Rev. Lett. 126, 038002 (2021).
  * (13) A. K. Omar, K. Klymko, T. GrandPre, and P. L. Geissler, Phase diagram of active Brownian spheres: crystallization and the metastability of motility-induced phase separation, Phys. Rev. Lett. 126, 188002 (2021).
  * (14) J. Bialké, T. Speck, and H. Löwen, Crystallization in a dense suspension of self-propelled particles, Phys. Rev. Lett. 108, 168301 (2012).
  * (15) A. Wysocki, R. G. Winkler, and G. Gompper, Cooperative motion of active Brownian spheres in three-dimensional dense suspensions, Europhysics Letters (EPL) 105, 48004 (2014).
  * (16) F. J. Moore, C. P. Royall, T. B. Liverpool, and J. Russo, Crystallisation and polymorph selection in active Brownian particles, Eur. Phys. J. E 44, 121 (2021).
  * (17) S. Paliwal, J. Rodenburg, R. van Roij, and M. Dijkstra, Chemical potential in active systems: predicting phase equilibrium from bulk equations of state?, New J. Phys. 20, 015003 (2018).
  * (18) A. P. Solon, J. Stenhammar, M. E. Cates, Y. Kafri, and J. Tailleur, Generalized thermodynamics of phase equilibria in scalar active matter, Phys. Rev. E 97, 020602(R) (2018).
  * (19) T. Speck, Coexistence of active Brownian disks: van der Waals theory and analytical results, Phys. Rev. E 103, 012607 (2021).
  * (20) T. Speck, Critical behavior of active Brownian particles: Connection to field theories, Phys. Rev. E 105, 064601 (2022).
  * (21) J. Bickmann and R. Wittkowski, Predictive local field theory for interacting active Brownian spheres in two spatial dimensions, J. Phys.: Condens. Matter 32, 214001 (2020).
  * (22) A. K. Omar, H. Row, S. A. Mallory, and J. F. Brady, Mechanical theory of nonequilibrium coexistence and motility-induced phase separation, Proc. Nat. Acad. Sci. 120, e2219900120 (2023).
  * (23) G. Szamel, Theory for the dynamics of dense systems of athermal self-propelled particles, Phys. Rev. E 93, 012603 (2016).
  * (24) R. Evans, The nature of the liquid-vapour interface and other topics in the statistical mechanics of non-uniform, classical fluids, Adv. Phys. 28, 143 (1979).
  * (25) R. Evans, M. Oettel, R. Roth, and G. Kahl, New developments in classical density functional theory, J. Phys.: Condens. Matter 28, 240401 (2016).
  * (26) R. Roth, Fundamental measure theory for hard-sphere mixtures: a review. J. Phys.: Condens. Matt. 22, 063102 (2010).
  * (27) A. Mauleon-Amieva, M. Mosayebi, J. E. Hallett, F. Turci, T. B. Liverpool, J. S. van Duijneveldt, and C. P. Royall, Competing active and passive interactions drive amoebalike crystallites and ordered bands in active colloids, Phys. Rev. E 102, 032609 (2020).
  * (28) J. Palacci, S. Sacanna, A. P. Steinberg, D. J. Pine, and P. M. Chaikin, Living crystals of light-activated colloidal surfers, Science 339, 936 (2013).
  * (29) N. A. Söker, S. Auschra, V. Holubec, K. Kroy and F. Cichos, How activity landscapes polarize microswimmers without alignment forces, Phys. Rev. Lett. 126, 228001 (2021).
  * (30) S. Auschra, V. Holubec, N. A. Söker, F. Cichos, and K. Kroy, Polarization-density patterns of active particles in motility gradients, Phys. Rev. E 103, 062601 (2021)
  * (31) F. Kümmel, P. Shabestari, C. Lozano, G. Volpe, and C. Bechinger, Formation, compression and surface melting of colloidal clusters by active particles, Soft Matter 11, 6187 (2015).
  * (32) F. Ginot, A. Solon, Y. Kafri, C. Ybert, J. Tailleur and C. Cottin-Bizonne, Sedimentation of self-propelled Janus colloids: polarization and pressure, New J. Phys. 20, 115001 (2018).
  * (33) A. F. Carreira, A. Wysocki, C. Ybert, M. Leocmach, H. Rieger, C. Cottin-Bizonne, How to steer active colloids up a vertical wall, arXiv:2307.02810.
  * (34) N. Klongvessa, F. Ginot, C. Ybert, C. Cottin-Bizonne, and M. Leocmach, Nonmonotonic behavior in dense assemblies of active colloids, Phys. Rev. E 100, 062603 (2019).
  * (35) F. Turci and N. B. Wilding, Wetting transition of active Brownian particles on a thin membrane, Phys. Rev. Lett. 127, 238002 (2021).
  * (36) M. James, D. A. Suchla, J. Dunkel, and M. Wilczek, Emergence and melting of active vortex crystals, Nat. Commun. 12, 5630 (2021).
  * (37) C. M. Koch and M. Wilczek, Role of advective inertia in active nematic turbulence, Phys. Rev. Lett. 127, 268005 (2021).
  * (38) E. Chacón, F. Alarcón, J. Ramirez, P. Tarazona, and C. Valeriani, Intrinsic structure perspective for MIPS interfaces in two-dimensional systems of active Brownian particles, Soft Matter 18, 2646 (2022).
  * (39) J. Martin-Roca, R. Martinez, L. C. Alexander, A. L. Diez, D. G. A. L. Aarts, F. Alarcon, J. Ramírez, and C. Valeriani, Characterization of MIPS in a suspension of repulsive active Brownian particles through dynamical features, J. Chem. Phys. 154, 164901 (2021).
  * (40) M. Schmidt, Power functional theory for many-body dynamics, Rev. Mod. Phys. 94, 015007 (2022).
  * (41) P. Krinninger, M. Schmidt, and J. M. Brader, Nonequilibrium phase behaviour from minimization of free power dissipation, Phys. Rev. Lett. 117, 208003 (2016).
  * (42) P. Krinninger and M. Schmidt, Power functional theory for active Brownian particles: general formulation and power sum rules, J. Chem. Phys. 150, 074112 (2019).
  * (43) S. Hermann, D. de las Heras, and M. Schmidt, Phase separation of active Brownian particles in two dimensions: Anything for a quiet life, Mol. Phys. e1902585 (2021).
  * (44) S. Hermann, P. Krinninger, D. de las Heras, and M. Schmidt, Phase coexistence of active Brownian particles, Phys. Rev. E 100, 052604 (2019).
  * (45) S. Hermann, D. de las Heras, and M. Schmidt, Non-negative interfacial tension in phase-separated active Brownian particles, Phys. Rev. Lett. 123, 268002 (2019).
  * (46) S. Hermann and M. Schmidt, Noether’s theorem in statistical mechanics, Commun. Phys. 4, 176 (2021).
  * (47) F. Sammüller, S. Hermann, D. de las Heras, and M. Schmidt, Noether-constrained correlations in equilibrium liquids, Phys. Rev. Lett. 130, 268203 (2023).
  * (48) S. Hermann and M. Schmidt, Active interface polarization as a state function, Phys. Rev. Research 2, 022003(R) (2020).
  * (49) D. de las Heras, T. Zimmermann, F. Sammüller, S. Hermann and M. Schmidt, Perspective: How to overcome dynamical density functional theory, J. Phys.: Condens. Matter 35, 271501 (2023). (Invited Perspective).
  * (50) F. Sammüller, S. Hermann, D. de las Heras, and M. Schmidt, Neural functional theory for inhomogeneous fluids: Fundamentals and applications, arxiv:2307.04539.
  * (51) D. de las Heras and M. Schmidt, Flow and structure in nonequilibrium Brownian many-body systems, Phys. Rev. Lett. 125, 018001 (2020).
  * (52) L. Caprini, U. Marini Bettolo Marconi, and A. Puglisi, Spontaneous velocity alignment in motility-induced phase separation, Phys. Rev. Lett. 124, 078001 (2020).
  * (53) P. Paricaud, Extension of the BMCSL equation of state for hard spheres to the metastable disordered region: Application to the SAFT approach, J. Chem. Phys. 143, 044507 (2015).
  * (54) W. G. Hoover and F. H. Ree, Use of computer experiments to locate the melting transition and calculate the entropy in the solid phase, J. Chem. Phys. 47, 4837 (1967).
  * (55) W. G. Hoover and F. H. Ree, Melting transition and communal entropy for hard spheres, J. Chem. Phys. 49, 3609 (1968).
  * (56) D. Frenkel and A. J. C. Ladd, New Monte Carlo method to compute the free energy of arbitrary solids. Application to the fcc and hcp phases of hard spheres, J. Chem. Phys. 81, 3188 (1984).
  * (57) C. P. Royall, P. Charbonneau, M. Dijkstra, J. Russo, F. Smallenburg, T. Speck, C. Valeriani, Colloidal Hard Spheres: Triumphs, Challenges and Mysteries arXiv:2305.02452
  * (58) J. M. Häring, C. Walz, G. Szamel, and M. Fuchs, Coarse-grained density and compressibility of nonideal crystals: General theory and an application to cluster crystals. Phys. Rev. B 92, 184103 (2015).
  * (59) S.-C. Lin, M. Oettel, J. M. Häring, R. Haussmann, M. Fuchs, and G. Kahl, Direct correlation function of a crystalline solid, Phys. Rev. Lett. 127, 085501 (2021).
  * (60) J. Dong, F. Turci, R. L. Jack, M. A. Faers, and C. P. Royall, Direct imaging of contacts and forces in colloidal gels, J. Chem. Phys. 156, 214907 (2022).