# Emotion Rendering for Conversational Speech Synthesis with
Heterogeneous Graph-Based Context Modeling

Rui Liu1, Yifan Hu1, Yi Ren2, Xiang Yin2, Haizhou Li3,4 Corrposending Author.

###### Abstract

Conversational Speech Synthesis (CSS) aims to accurately express an utterance
with the appropriate prosody and emotional inflection within a conversational
setting. While recognising the significance of CSS task, the prior studies
have not thoroughly investigated the emotional expressiveness problems due to
the scarcity of emotional conversational datasets and the difficulty of
stateful emotion modeling. In this paper, we propose a novel emotional CSS
model, termed ECSS, that includes two main components: 1) to enhance emotion
understanding, we introduce a heterogeneous graph-based emotional context
modeling mechanism, which takes the multi-source dialogue history as input to
model the dialogue context and learn the emotion cues from the context; 2) to
achieve emotion rendering, we employ a contrastive learning-based emotion
renderer module to infer the accurate emotion style for the target utterance.
To address the issue of data scarcity, we meticulously create emotional labels
in terms of category and intensity, and annotate additional emotional
information on the existing conversational dataset (DailyTalk). Both objective
and subjective evaluations suggest that our model outperforms the baseline
models in understanding and rendering emotions. These evaluations also
underscore the importance of comprehensive emotional annotations. Code and
audio samples can be found at: https://github.com/walker-hyf/ECSS.

## Introduction

Figure 1: Graphical depiction of spoken conversation, where different emotion
cues in the dialogue history perform a direct impact on the emotion expression
for the current utterance.

Conversational speech synthesis (CSS) aims to express a target utterance with
the proper linguistic and affective prosody in a conversational context (Guo
et al. 2020). With the development of human-machine conversations, CSS has
become an integral part of intelligent interactive systems (Tulshan and Dhage
2019; Zhou et al. 2020; Seaborn et al. 2021; McTear 2022) and plays an
important role in areas such as virtual assistants and voice agents, etc.

Unlike the speech synthesis technology for single utterance that just predict
the speaking style according to its linguistic content (Wang et al. 2017; Li
et al. 2019; Ren et al. 2021; Kim, Kong, and Son 2021; Liu et al. 2021a, b, c,
2022a) or attempt to transfer the style information from an additional
reference speech (Wang et al. 2018; Li et al. 2022c; Huang et al. 2022), CSS
methods usually infer the speaking style of the target utterance according to
the dialogue interaction history between two interlocutors. Traditional CSS
works attempt to acquire the speaking style information from various aspects,
such as inter and intra-speaker (Li et al. 2022a), and multi-modal context (Li
et al. 2022a, b; Xue et al. 2023) dependencies modeling, etc. For example, Guo
et al. (2020) construct a coarse-grained context encoder at the sentence level
and use the recurrent neural network (RNN) for dialogue history encoding. Li
et al. (2022a) model inter-speaker and intra-speaker dependencies in a
conversation by a dialog graph convolutional neural (GCN) network and
summarize the output of the graph neural network using the attention
mechanism. In more recent studies, Li et al. (2022b) propose a multi-scale
relational graph convolutional network (MRGCN) to learn the dependencies in
conversations at both global and local scales among the multi-modal
information. The above researches contribute to understanding the
conversational context and determining appropriate speaking styles in
synthesized speeches.

However, emotion understanding and emotion rendering are largely missing in
prior CSS research due to the scarcity of emotional conversational datasets
and the difficulty of stateful emotion modeling. In human-computer
conversations, modeling the emotional expression in the conversational context
is crucial for the speech synthesis system to generate speech with the
appropriate emotional states and improve the user experience in speech-based
interactions. As illustrated by the example in Fig.1, the current utterance in
Fig. 1(a) performs the “Surprise” emotion when the emotion flow of dialogue
history is “Happy $\rightarrow$ Happy $\rightarrow$ Surprise $\rightarrow$
Happy”, while in Fig. 1(b) performs “Disgust” emotion when emotion flow is
“Angry $\rightarrow$ Happy $\rightarrow$ Angry $\rightarrow$ Angry”. We can
conclude that the emotional expression in context can directly affect the
speaking style for the target utterance. In addition, humans tend to have
multiple emotions with varying intensities, such as weak, medium and strong,
while expressing their thoughts and feelings (Firdaus et al. 2020; Im et al.
2022; Liu et al. 2022b; Zhou et al. 2023). Therefore, emotion intensity also
has an essential impact on speech expressiveness. In a nutshell, how to fully
understand the emotional cues of contextual information, and based on this,
adequate emotional rendering in synthesizing conversational speech will be the
focus of this paper. Last but not least, the existing emotion-aware multimodal
data (Busso et al. 2008; Poria et al. 2018; Saganowski et al. 2022; Dias et
al. 2022; Zuo et al. 2023) are mostly targeted at emotion recognition
scenarios, in which the speech fidelity of the audio modality is not high
enough to meet the data requirements of conversational speech synthesis,
resulting in the problem of data scarcity.

To address the above challenges, we propose a novel emotional CSS model,
termed ECSS, that includes two novel mechanisms: 1) to enhance emotion
understanding for the context utterances, heterogeneous graph-based emotional
context modeling module is proposed to learn the emotion cues of emotional
conversational context. Specifically, given the multi-modal context in a
conversation, the text, audio, speaker, emotion, and emotion intensity
information are treated as the multi-source knowledge and used to build the
nodes of the heterogeneous Emotional Conversational Graph called ECG. Then,
the graph encoding module adopts the Heterogeneous Graph Transformer (HGT) (Hu
et al. 2020) as the backbone to learn complex emotional dependencies in the
context, and to learn the impact of such complex dependencies on the emotional
expression of the current utterance; 2) to achieve the emotion rendering for
the current utterance, we employ a contrastive learning-based emotion renderer
module to infer the accurate emotion style for the target utterance.
Specifically, the emotion renderer takes the emotion-aware graph-enhanced node
features from ECG as input and predicts the appropriate emotion, emotion
intensity, and prosody features for the current utterance. This information is
later aggregated with the content and speaker representations of the current
utterance into the acoustic decoder to synthesize the final emotional
conversational speech. Note that the new contrastive learning losses are used
to enhance the differentiation of emotion and emotional intensity expressions
by drawing the same categories of emotion (or intensity) closer together and
pushing different categories farther apart. It’s worth noting that, to
guarantee the successful development of the ECSS model, we designed seven
emotion labels (happy, sad, angry, disgust, fear, surprise, neutral), and
three emotion intensity labels (weak, medium, strong) for a recent expressive
conversational speech synthesis dataset, DailyTalk (Lee, Park, and Kim 2023),
and invited professional practitioners to annotate the labels. All annotated
data will be open-sourced. The main contributions of this paper include:

  * •

We propose a novel emotional conversational speech synthesis model, termed
ECSS. To our best knowledge, this is the first in-depth conversational speech
synthesis study that models emotional expressiveness.

  * •

The proposed heterogeneous graph-based emotional context modeling and emotion
rendering mechanisms ensure the accurate generation of emotional
conversational speech in terms of emotion understanding and expression,
respectively.

  * •

Objective and subjective experiments show that the proposed model outperforms
all state-of-the-art baselines in terms of emotional expressiveness.

## Related works

The emotion modeling for conversation has been studied in both natural
language processing (NLP) and speech processing fields. In NLP field, Zhong,
Wang, and Miao (2019) use context-aware emotion graph attention mechanisms to
utilize external commonsense knowledge for emotion recognition dynamically.
Goel et al. (2021) proposed a novel transformer encoder that adds and
normalizes the word embedding with emotion embedding, thereby integrating the
semantic and affective aspects of the input utterance. However, multi-modal
information other than textual information is rarely considered in NLP.

In speech processing field, the advanced conversational speech synthesis
methods use Graph Neural Networks (GNNs) models (Li et al. 2022a, b) to
understand the multi-modal conversational context and infer the appropriate
speaking style for the target utterance. For example, multi-modal features of
all past utterances in the conversation, including textual information and
speaking style information, are modeled by Dialogue Graph Convolutional
Network (DialogueGCN) (Ghosal et al. 2019) to produce new representations
holding richer context knowledge. However, the GNNs for CSS are designed for
homogeneous graphs, in which all nodes and edges belong to the same types
(such as utterance nodes from different speakers), making them infeasible to
represent the natural heterogeneous structures in conversation. Especially for
emotional expressions in conversation, information such as text, audio,
speaker, emotion, and emotion intensity can be seen as nodes of the
heterogeneous graph.

We note that there are some multi-modal conversational emotion recognition
(MMCER) works that adopt heterogeneous graph networks to model the complex
dependencies of contexts adequately (Li et al. 2022d; Song et al. 2023).
Unlike previous studies, our heterogeneous graph module has some clear
differences from these works: 1) we add emotion and emotion intensity nodes
into the graph structure to model the dynamic emotion cues in conversation
context; 2) we adopt Heterogeneous Graph Transformer as the backbone to encode
the relations between heterogeneous nodes to learn the high-level feature
representation for the constructed graph. Note that our work is the first
attempt to model the emotion understanding and rendering in conversations for
CSS with heterogeneous graph networks.

Figure 2: The overall architecture of ECSS. 1) Multi-source knowledge includes
text, speaker, audio, emotion, and emotion intensity information of the multi-
turn conversation; 2) Heterogeneous graph-based emotional context encoder aims
to model the complex dependencies among the multi-source knowledge with the
Emotional Conversational Graph (ECG), thus understanding the emotion cues in
context; 3) Emotion rendering for CSS seeks to render the accurate emotion
state for synthesized speech, in which Emotion Renderer aims to infer the
emotion feature according to the heterogeneous ECG encoding.

## Task Definition

A conversation can be defined as a sequence of utterances
($utt_{1},utt_{2},...,utt_{\mathcal{J}},utt_{\mathcal{C}}$), where
{$utt_{1},utt_{2},...,utt_{\mathcal{J}}$} is the dialogue history till round
$\mathcal{J}$ while $utt_{\mathcal{C}}$ means the current utterance to be
synthesized. The task of emotional conversational speech synthesis aims to
synthesize the audio $a_{\mathcal{C}}$ given the $utt_{\mathcal{C}}$ and the
dialogue history {$utt_{1},utt_{2},...,utt_{\mathcal{J}}$}. For the multi-
modal context, each utterance $utt_{j}$ ($j\in[1,\mathcal{J}]$) in the
dialogue history can be represented by five-tuples like $<$textj, speakerj,
audioj, emotionj, emotion intensityj$>$, in short for $<u_{j}$, $s_{j}$,
$a_{j}$, $e_{j}$, $i_{j}$$>$. Note that the $utt_{\mathcal{C}}$ can be
represented by only two-tuples like $<$textC, speaker${}_{\mathcal{C}}>$, in
short for $<u_{\mathcal{C}}$, $s_{\mathcal{C}}>$, since emotion and intensity
information need to be generated by ECSS. Particularly, the emotional
expression of the synthesized speech $a_{\mathcal{C}}$ should confirm to the
emotional conversational context characterized by the multi-modal dialogue
history. To this end, the emotional CSS methods need to consider: 1) How to
mine the multi-source emotional information in conversation history that is
important for emotional expression; 2) How to model dynamic emotional cues in
dialogue context while modeling intra-speaker, multi-modal context
dependencies, etc.; 3) How to infer the appropriate emotional expression
information of the current discourse based on understanding the emotional cues
of the conversation.

## Methodology

As shown in the pipeline of Fig. 2, the proposed ECSS consists of three
components, that are 1) Multi-source knowledge; 2) Heterogeneous Graph-based
Emotional Context Encoder and 3) Emotional Conversational Speech Synthesizer.
As mentioned before, the multi-modal context, including text, speaker, audio,
emotion and intensity, contains natural multi-source information and can
therefore be viewed as multi-source knowledge. To enhance emotion
understating, the Heterogeneous Graph-based Emotional Context Encoder
constructs a heterogeneous Emotional Conversational Graph (ECG) by considering
each kind of information in the multi-source knowledge as a node, and obtains
a graph-enhanced emotional contextual representation for each node after
modeling the dependencies among all heterogeneous nodes. To achieve emotion
rendering, the Emotional Conversational Speech Synthesizer utilizes the graph-
enhanced contextual representation in the ECG to make a reasonable prediction
of the emotion expression information of the current sentence and further
generates the emotional conversational speech. We elaborate our ECSS from the
following three aspects: Heterogeneous Graph-based Emotional Context Encoder,
Emotional Conversational Speech Synthesizer and Contrastive Learning Training
Criterion.

### Heterogeneous Graph-based Emotional Context Encoder

As shown in the middle panel of Fig. 2, the heterogeneous graph-based
emotional context encoder consists of three parts: 1) ECG Construction,
constructing the heterogeneous graph with multi-source context; 2) ECG
Initialization, initializing different heterogeneous nodes via their own
feature representations; 3) ECG Encoding, perceiving emotion cues and
generating the emotion-aware feature representation for the heterogeneous
nodes.

#### ECG Construction

Unlike previous GNNs based CSS methods, we aim to introduce the multi-source
knowledge, which are 5 kinds of nodes including text $f_{u}$, audio $f_{a}$,
speaker $f_{s}$, emotion $f_{e}$, and intensity $f_{i}$ and build an emotional
conversational graph or ECG $\mathcal{G=(N,E)}$, where $\mathcal{N}$ denotes
the set of nodes, and $\mathcal{E}$ denotes the set of edges representing the
relations between two nodes. Note that the speaker, audio and text nodes seek
to introduce the basic dialogue attributes, while emotion and intensity nodes
can introduce the dynamic emotion traits and bridge the emotion interaction
between remote utterances. As shown in the middle part of Fig. 2, different
shapes of diagrams mean different kinds of nodes.

Considering the multi-source knowledge, We created 14 different types of
edges, as shown by the different colored connecting lines shown in the middle
part of Fig. 2. However, due to space limits, not all the edges of the nodes
are depicted. In a nutshell, these 14 edges connect 1) the text and each of
the other nodes, 2) the audio and speaker nodes, 3) the emotion and speaker,
emotion intensity, and audio nodes, and 4) emotion intensity and speaker,
audio nodes. Note that all edges include past-to-future and future-to-past
connections to model the bidirectional relation.

#### ECG Initialization

To achieve meaningful heterogeneous graph encoding, we need to initialize all
nodes with their feature representations. As shown in the middle and bottom
panels of Fig. 2, to take the multi-turn dialogue as an example, we employ
various encoders to obtain $f_{u_{j}}$, $f_{s_{j}}$, $f_{a_{j}}$, $f_{e_{j}}$,
$f_{i_{j}}$ ($j\in[1,4]]$) for text, speaker, audio, emotion, and intensity
nodes.

  * •

Text Nodes. We adopt a pre-trained BERT111https://huggingface.co/sentence-
transformers/distiluse-base-multilingual-cased-v1 model to extract the
linguistic feature: $f_{u_{j}}={\rm BERT}(u_{j})$.

  * •

Audio Nodes. We employ the global style token (GST) (Wang et al. 2018) module,
which includes a reference encoder and style token layer, as the audio encoder
to extract the acoustic features contained in each audio $a_{j}$:
$f_{a_{j}}={\rm GST}(a_{j})$.

  * •

Speaker, Emotion, and Emotion Intensity Nodes. The speaker, emotion and
intensity encoders are used to define three randomly initialized trainable
parameter matrices $f_{s_{j}}$, $f_{e_{j}}$, and $f_{i_{j}}$ to learn two
speaker identity features, seven emotion label (happiness, sadness, anger,
disgust, fear, surprise, neutral) features, and three emotion intensity label
(weak, medium, strong) features respectively.

Note that the text and speaker nodes $f_{u_{\mathcal{C}}}$,
$f_{i_{\mathcal{C}}}$ of current utterance are initialized in the same way as
the nodes $f_{u_{j}}$ and $f_{s_{j}}$ in the dialogue history.

#### ECG Encoding

After initializing the constructed Graph, the heterogeneous graph encoding
module is used to encode the emotion cues in dialogue context to obtain the
graph-enhanced representations for each node. Inspired by Heterogeneous Graph
Transformer (HGT) (Hu et al. 2020), we also adopt a three-stage emotional HGT
network, that includes Heterogeneous Mutual Attention (HMA), Heterogeneous
Message Passing (HMP), and Emotional knowledge Aggregation (EKA) operations,
to model the dependencies in emotional conversations. Assuming that one node
in the heterogeneous ECG is the target node, then any node of any type can be
viewed as the source node. The goal of HGT is to aggregate information from
source nodes to get a contextualized representation for the target node.

Firstly, given a target node $\mathcal{N}_{\hat{tgt}}$ and all its neighbor
source nodes $\mathcal{N}_{\hat{src}}$, the HMA mechanism maps
$\mathcal{N}_{\hat{tgt}}$ into a Query vector, and $\mathcal{N}_{\hat{src}}$
into a Key vector, and calculate their dot product as attention, that
indicates the importance of each source node for the target node. Then we
concatenate $h$ attention heads together to get the attention vector for each
node pair
$\mathcal{E}_{\hat{src}\rightarrow\hat{tgt}}=\\{\mathcal{N}_{\hat{src}},\mathcal{N}_{\hat{tgt}}\\}$
($\mathcal{E}_{\hat{src}\rightarrow\hat{tgt}}$ means the edge between
$\mathcal{N}_{\hat{src}}$ and $\mathcal{N}_{\hat{tgt}}$). For each target node
$\mathcal{N}_{\hat{tgt}}$, we gather all attention vectors from its neighbors
$\mathcal{N}_{\hat{src}}$ and conduct softmax to get the final attention
score.

Secondly, HMP is computed parallel to pass the dependency information from
$\mathcal{N}_{\hat{src}}$ to $\mathcal{N}_{\hat{tgt}}$. Specifically, for a
pair of nodes
$\mathcal{E}_{\hat{src}\rightarrow\hat{tgt}}=\\{\mathcal{N}_{\hat{src}},\mathcal{N}_{\hat{tgt}}\\}$,
HMP uses a linear projection to project the feature vector of
$\mathcal{N}_{\hat{src}}$ into a message vector, that then followed by a
matrix $W_{\hat{src}\rightarrow\hat{tgt}}$ for incorporating the edge
dependency. The final step is to concatenate all $h$ message heads to get the
final message vector for each node pair.

At last, the EKA module aims to aggregate the computed attention score and
message vector. We use the attention score as the weight to average the
corresponding messages from all neighbor source nodes
$\mathcal{N}_{\hat{src}}$ and get the emotion-augmented vector
$f_{\mathcal{N}_{\hat{tgt}}}$ for the target node $\mathcal{N}_{\hat{tgt}}$.
It’s worth noting that all ECG nodes, including emotion and intensity nodes,
can be treated as the source or target nodes to learn its high-level
contextual information. Therefore, the EKA operation ultimately incorporates
the emotional information from the dialog history into all ECG nodes. The
graph-enhanced feature representation of each node incorporates the emotional
cues in context.

In this way, the ECG encoding module can obtain the final emotion-aware graph-
enhanced feature representation $f_{u}^{{}^{\prime}}$, $f_{s}^{{}^{\prime}}$,
$f_{a}^{{}^{\prime}}$, $f_{e}^{{}^{\prime}}$ and $f_{i}^{{}^{\prime}}$ for all
text, speaker, audio, emotion and intensity nodes respectively, which can be
fed into subsequent models to provide emotional information.

### Emotional Conversational Speech Synthesizer

As shown in the right panel of Fig. 2, the emotional conversational speech
synthesizer consists of the following four components: Text encoder, Speaker
Encoder, Emotion Renderer and Acoustic Decoder. The text and speaker encoders
seek to encode the content and speaker identity features
$\mathcal{H}^{c}_{\mathcal{C}}$ and $\mathcal{H}^{s}_{\mathcal{C}}$ for the
current utterance and speaker. Note that the emotion renderer attempts to
predict the current utterance’s emotion, intensity, and prosody features
$\mathcal{H}^{e}_{\mathcal{C}}$, $\mathcal{H}^{i}_{\mathcal{C}}$, and
$\mathcal{H}^{p}_{\mathcal{C}}$ using the graph-enhanced node features. To
obtain a robust feature representation of the current utterance, the feature
aggregator module set a set of trainable weight parameters for the above five
features and then output the final mixup feature $\mathcal{H}_{\mathcal{C}}$.
For the acoustic decoder, we use FastSpeech2 (Ren et al. 2021) as the
backbone, which includes the variance adaptor, mel decoder and vocoder. The
variance adapter takes the $\mathcal{H}_{\mathcal{C}}$ as inputs to predict
the duration, energy, and pitch. The Mel Decoder aims to predict the mel-
spectrum features. Finally, a well-trained HiFi-GAN (Kong, Kim, and Bae 2020)
is used as the vocoder to generate speech waveform $a_{\mathcal{C}}$ with
desired emotion style.

Note that to achieve emotion rendering, our emotion renderer extracts the
encoded node features from ECG and predicts the emotion and emotion intensity
feature representations of the current utterance while performing the prosody
prediction. More importantly, to achieve accurate emotion category and emotion
intensity feature representation prediction, we propose contrastive learning-
based emotion and emotion intensity loss functions $\mathcal{L}^{cl}$.

#### Emotion Renderer

As shown in the blue part of Fig. 2, the emotion renderer consists of emotion,
intensity and prosody predictors. The prosody predictor adopts a multi-head
attention layer to infer the speaking prosody information of the current
utterance from the feature representations of text nodes in the dialogue
history. The features of audio nodes are not used because we believe that
audio and text modalities have already interacted during the ECG encoding,
thus the text node already contains audio information. We use the MSE loss for
the prosody predictor to constrain its training, where the target is obtained
from a GST-based prosody extractor. Next, we will introduce emotion and
intensity predictors.

  * •

Emotion Predictor uses the encoded features of the emotion nodes in dialog
history to infer the emotion representation $\mathcal{H}^{e}_{\mathcal{C}}$ of
the current sentence. It includes two convolutional layers, a bidirectional
LSTM layer, and two fully connected layers.

$\mathcal{H}^{e}_{\mathcal{C}}={\rm FC(BiLSTM(CNN(}f_{e}^{{}^{\prime}})))$ (1)

where $f_{e}^{{}^{\prime}}$ represents all emotion-type nodes in the dialogue
history after ECG encoding.

  * •

Intensity Predictor uses the encoded features of the intensity nodes in dialog
history to infer the emotion intensity representation
$\mathcal{H}^{i}_{\mathcal{C}}$ of the current sentence. It consists of two
convolutional layers, a bidirectional LSTM layer, two fully connected layers,
and a mean pooling layer.

$\begin{split}\mathcal{H}^{i}_{\mathcal{C}}={\rm
AvgPooling(FC_{2}(BiLSTM(CNN_{2}(}f_{i}^{{}^{\prime}}))))\end{split}$ (2)

where $f_{i}^{{}^{\prime}}$ is a universal representation of all emotion
intensity nodes in the dialog history after ECG encoding.

#### Contrastive Learning Training Criterion

For emotion and intensity predictors of the emotion renderer, inspired by
(Khosla et al. 2020), we design the emotion-supervised contrastive learning
losses $\mathcal{L}^{cl}$ to motivate the emotion renderer to better
distinguish different emotions categories and intensity degrees. Specifically,
contrastive learning loss $\mathcal{L}^{cl}_{emo}$ for emotion category and
$\mathcal{L}^{cl}_{int}$ for emotion intensity share the same spirits, that is
treating all examples with the same emotion category or intensity label in the
batch as positive examples while different labels as negative.

For the emotion feature $\mathcal{H}^{e}_{\mathcal{C}}$, a batch of $K$
emotion representations is denoted as
$\mathcal{H}^{K}=[\mathcal{H}^{e}_{\mathcal{C}1},\mathcal{H}^{e}_{\mathcal{C}2},...,\mathcal{H}^{e}_{\mathcal{C}K}]$,
$\mathcal{L}^{cl}_{emo}$ for $\mathcal{H}^{e}_{\mathcal{C}k}$ as follows,

$\mathcal{L}^{cl}_{emo}=log\frac{-1}{|\mathcal{P}(k)|}\frac{\underset{\mathcal{H}^{e}_{\mathcal{C}q}\in\mathcal{P}(k)}{\sum}exp(sim(\mathcal{H}^{e}_{\mathcal{C}k},\mathcal{H}^{e}_{\mathcal{C}q})/\tau)}{\underset{\mathcal{H}^{e}_{\mathcal{C}d}\in
B(k)}{\sum}exp(sim(\mathcal{H}^{e}_{\mathcal{C}k},\mathcal{H}^{e}_{\mathcal{C}d})/\tau)}$
(3)

where $sim(\cdot,\cdot)$ is a cosine similarity function. $\tau$ is a scalar
temperature parameter.
$B(k)\equiv\mathcal{H}^{K}\backslash\\{\mathcal{H}^{e}_{\mathcal{C}k}\\}$
contains all representations in $\mathcal{H}^{K}$ except
$\mathcal{H}^{e}_{\mathcal{C}k}$. $\mathcal{P}(k)$ is the set of positive
samples that have the same emotion label with $\mathcal{H}^{e}_{\mathcal{C}k}$
in a batch. Similarity, $\mathcal{L}^{cl}_{int}$ for
$\mathcal{H}^{i}_{\mathcal{C}k}$ as follows,

$\mathcal{L}^{cl}_{int}=log\frac{-1}{|\mathcal{P}(k)|}\frac{\underset{\mathcal{H}^{i}_{\mathcal{C}q}\in\mathcal{P}(k)}{\sum}exp(sim(\mathcal{H}^{i}_{\mathcal{C}k},\mathcal{H}^{i}_{\mathcal{C}q})/\tau)}{\underset{\mathcal{H}^{i}_{\mathcal{C}d}\in
B(k)}{\sum}exp(sim(\mathcal{H}^{i}_{\mathcal{C}k},\mathcal{H}^{i}_{\mathcal{C}d})/\tau)}$
(4)

At last, the total loss function $\mathcal{L}$ is:
$\mathcal{L}=\mathcal{L}^{cl}_{emo}+\mathcal{L}^{cl}_{int}+\mathcal{L}^{mse}_{pro}+\mathcal{L}_{fs2}$,
where $\mathcal{L}_{fs2}$ refers to the acoustic feature loss of traditional
FastSpeech2, including pitch, energy, and duration, as well as the mel
spectrum, $\mathcal{L}^{mse}_{pro}$ indicates the MSE loss for prosody
predictor.

Table 1: Subjective (with 95% confidence interval) and objective results with
different CSS systems. (* means the metric value achieved the suboptimal
result among all systems.)

Systems |  N-DMOS ($\uparrow$) |  E-DMOS ($\uparrow$) |  MAE-M ($\downarrow$) | MAE-P ($\downarrow$) | MAE-E ($\downarrow$) | MAE-D ($\downarrow$)
---|---|---|---|---|---|---
No emotional context modeling | 3.232 $\pm$ 0.030 | 3.100 $\pm$ 0.026 | 0.681 | 0.506 | 0.346 | 0.300
GRU-based | 3.314 $\pm$ 0.022 | 3.288 $\pm$ 0.011 | 0.675 | 0.506 | 0.352 | 0.296
Homogeneous Graph-based | 3.384 $\pm$ 0.027 | 3.493 $\pm$ 0.031 | 0.662 | 0.456 | 0.204 | 0.150
ECSS (Proposed) | 3.506 $\pm$ 0.022 | 3.619 $\pm$ 0.028 | 0.654 | 0.455 | 0.215* | 0.152*
w/o emotion | 3.424 $\pm$ 0.018 | 3.496 $\pm$ 0.026 | 0.658 | 0.467 | 0.224 | 0.150
w/o intensity | 3.487 $\pm$ 0.035 | 3.523 $\pm$ 0.024 | 0.660 | 0.453 | 0.210 | 0.157
w/o speaker | 3.401 $\pm$ 0.019 | 3.511 $\pm$ 0.027 | 0.666 | 0.456 | 0.231 | 0.152
w/o audio | 3.391 $\pm$ 0.023 | 3.505 $\pm$ 0.022 | 0.656 | 0.457 | 0.198 | 0.154
w/o $\mathcal{L}^{cl}$ | 3.388 $\pm$ 0.025 | 3.312 $\pm$ 0.031 | 0.665 | 0.459 | 0.222 | 0.156

## Experiments and Results

#### Dataset

We validate the ECSS on a recently public dataset for conversational speech
synthesis called DailyTalk (Lee, Park, and Kim 2023), that consists of 23,773
audio clips representing 20 hours in total, in which 2,541 conversations were
sampled, modified, and recorded. All dialogues are long enough to represent
the context of each conversation. The dataset was recorded by a male and a
female simultaneously. All speech samples are sampled at 44.10 kHz and coded
in 16 bits. We partition the data into training, validation, and test sets at
a ratio of 8:1:1.

To obtain the multi-source knowledge, we invited a professional practitioner
to perform fine-grained labeling of DailyTalk data for emotion category and
intensity labels while listening to speech and understanding the semantics of
utterance. The final distribution of the data is as follows, the number of
each of the 7 emotion category labels (happy, sad, angry, disgust, fear,
surprise, neutral) are 3871, 722, 226, 186, 74, 497 and 18,197, and the number
of each of the 3 emotion intensity labels (weak, medium, strong) are 19,973,
3,646 and 154.

#### Experimental Setup

In the heterogeneous graph-based emotion context encoder, the dimension of the
text node representation $f_{u_{j}}$ is set to 512, and the dimensions of the
remaining type node representations $f_{e_{j}}$,$f_{i_{j}}$,$f_{s_{j}}$, and
$f_{a_{j}}$ are all set to 256. For multi-head attention-based methods, we set
the head number as 8. For the emotion predictor of emotion renderer, the
convolutional layer has a convolutional kernel of 3, the LSTM input dimension
is 384, the hidden state size is 256, and the forward and backward outputs
from the last time steps of the LSTM are spliced using concat before going
into the linear layer. For the intensity predictor of emotion renderer, the
mean pooling layer convolution kernel size is 2. For the prosody predictor of
emotion renderer, the dimensions of the $Query$, $Key$ and $Value$ and output
feature $\mathcal{H}^{p}_{\mathcal{C}}$ are 512, 384, 384 and 256. The
acoustic decoder is configured with reference to FastSpeech2 (Ren et al.
2021). we use Adam optimizer with $\beta_{1}$ = 0.9, $\beta_{2}$ = 0.98. For
text input, we adopt Grapheme-to-Phoneme (G2P)
toolkit222https://www.github.com/kyubyong/g2p to convert all text into its
phoneme sequence. All speech samples are re-sampled to 22.05 kHz. The mel-
spectrum features are extracted with a window length of 25ms and a shift of
10ms. The model is trained on a Tesla V100 GPU with a batch size of 16 and
600k steps. During ECSS training, the context or dialogue history length is
set to 10. More detailed experimental settings are accessed in the Appendix
section.

#### Evaluation Metrics

For the subjective evaluation metrics, we organized a dialogue-level Mean
Opinion Score (DMOS) (Streijl, Winkler, and Hands 2016) listening test with 30
master students whose second language is English and provided specialized
training on the rules to all listeners. Given the dialogue history, they were
asked to rate the naturalness DMOS (N-DMOS) and emotional DMOS (E-DMOS) of the
synthesized speech of the current utterance, ranging from 1 to 5. Each
listener was access to 50 audio samples. Note that the N-DMOS focuses on the
speaking prosody, while the E-DMOS focuses on the richness of the emotional
expression of the current utterance and whether it matches the emotional
expression of the context.

For the objective evaluation metrics, we calculate the Mean Absolute Error
(MAE) between the predicted and real acoustic features to assess the emotional
expressiveness of the synthesized audio. Specifically, we assess the acoustic
feature in terms of mel-spectrum, pitch, energy, and duration with MAE-M,
MAE-P, MAE-E and MAE-D. In addition, we conduct a visualization study with the
help of a third-party Speech Emotion Recognition (SER) model, that was used to
identify the emotion categories of the synthesized emotional conversational
speech. We plot the confusion matrix to validate the ECSS.

#### Comparative Study

To demonstrate the effectiveness of our ECSS, we employ three advanced
approaches which also employ FastSpeech2 as the TTS backbone as the baseline
systems.

  * •

No emotional context modeling. The first baseline approach is a vanilla
FastSpeech2 (Ren et al. 2021) with no context modeling, which is also
representative of state-of-the-art non-conversational TTS systems.

  * •

GRU-based context modeling. This method involves only text modality and uses
an RNN-based uni-directional GRU network to model contextual dependencies in a
dialogue sequentially (Guo et al. 2020).

  * •

Homogeneous Graph-based emotion context modeling. In the homogeneous graph-
based approach (Li et al. 2022b), each past utterance in the conversation is
represented as a node in the graph. Each node is initialized with the
corresponding multi-modal features. To achieve emotion rendering using this
method, we can extract the graph-enhanced feature of all nodes in dialogue
history to predict the emotion and intensity information.

#### Main Results

As shown in the first five rows of Table 1, the ECSS achieves state-of-the-art
performance on average, which obtains the optimum results in MAE-M (0.654) and
MAE-P (0.455), and suboptimal results in MAE-E (0.215) and MAE-D (0.152).
However, objective experiments may not fully reflect human feelings. By
observing the subjective results, the proposed ECSS model outperforms all
baselines with an N-DMOS score of 3.506 and an E-DMOS score of 3.619, which
reflects the superiority of our ECSS. ECSS contributes to adequate emotion
understanding based on heterogeneous graph context modeling, in which the
emotion renderer fully mines the emotion cues to infer the emotion expression
state of the current sentence, thus achieving satisfactory emotion rendering
effects for conversational speech synthesis.

#### Ablation Results

To evaluate the individual effects of various heterogeneous nodes, including
emotion, intensity, speaker and audio, in ECG and the contrastive learning
loss $\mathcal{L}^{cl}$ for emotion renderer, we remove these components to
build various systems and conduct a series of ablation experiments, and the
subjective and objective results are shown in the rows 6 through 10 of Table
1.

We can find that removing different types of nodes in the heterogeneous ECG
brought about a decrease in objective metrics performance in the vast majority
of metrics, and the subjective DMOS scores also showed a drop. For example,
after removing the emotion node, the MAE-M, MAE-P and MAE-E decreased by
0.004, 0.012 and 0.009 respectively, while N-DMOS and E-DMOS decreased by
0.082 and 0.123. This suggests that our heterogeneous graph nodes can learn
the complex emotional dependencies in the dialog history and achieve full
emotional understanding. In addition, to validate the $\mathcal{L}^{cl}$, we
replace it with the cross-entropy loss. As shown in the last row of Table 1,
all subjective and objective values are reduced after removing the
$\mathcal{L}^{cl}$. This demonstrates that the contrastive learning strategy
allows the emotion renderer to distinguish between different emotion
categories and intensities better.

Table 2: Objective results of various context lengths.

Length | MAE-M ($\downarrow$) | MAE-P ($\downarrow$) | MAE-E ($\downarrow$) | MAE-D ($\downarrow$)
---|---|---|---|---
2 | 0.667 | 0.468 | 0.233 | 0.160
3 | 0.664 | 0.461 | 0.229 | 0.157
6 | 0.662 | 0.456 | 0.223 | 0.155
9 | 0.658 | 0.453 | 0.219 | 0.153
10 | 0.654 | 0.455* | 0.215 | 0.152
13 | 0.660 | 0.456 | 0.220 | 0.154
14 | 0.661 | 0.457 | 0.221 | 0.153

Figure 3: The confusion matrix results about the emotion category rendering
with the help of speech emotion recognition. The X-axis and Y-axis of
subfigures represent perceived and true emotion categories.

#### Context Length Analysis

We also explore the effectiveness of emotional context modeling with different
context lengths. Specifically, considering the average number 9.3 of dialogue
turns in the DailyTalk, we set the utterance length of dialogue history
ranging from 2 to 14 to compare the objective performance. As shown in Table
2, from a general view, all values decrease when enlarging the context length
from 2 to 10, and increase from 10 to 14. This shows that either insufficient
or redundant context information will interfere the understanding of emotion
cues in context.

#### Visualization Study

To demonstrate the emotional expressiveness of synthesized speech more
visually, we employ a pre-trained SER
model333https://huggingface.co/ehcalabres/wav2vec2-lg-xlsr-en-speech-emotion-
recognition to identify the emotion category of 400 audio samples synthesized
from the ECSS and all baselines, respectively. As shown in Fig. 3, we plot the
confusion matrices to show the gap between the different systems. It can be
seen that our ECSS outperforms all baselines by presenting a clear diagonal
line in the confusion matrix. It proves that the emotional conversational
speech synthesized by ECSS performs a clear emotional expression. In addition,
we further conduct a visualization study to validate the emotion intensity
rendering. Specifically, five listeners were invited and asked to rate the
emotion intensity labels of 400 audios. As shown in Fig. 4, the confusion
matrix suggests that the emotional conversational speech synthesized by ECSS
performs a clear emotional intensity expression. The above results also
provide further evidence that our ECSS draws on the contextual modeling
capabilities of heterogeneous graph and the effective constraints of
contrastive learning, thus leading to remarkable performance in both emotion
and emotion intensity rendering.

Figure 4: The confusion matrix results about the emotion intensity rendering.
The X-axis and Y-axis of subfigures represent perceived and true intensity
categories.

## Conclusion

To improve the emotion understanding and rendering in CSS systems, we present
a novel ECSS model whereby a heterogeneous Emotional Conversational Graph
(ECG) armed with multi-source knowledge in context is used for emotional
context modeling, and the emotion renderer with contrastive learning
constraint to achieve accurate emotional style inference. Experimental results
demonstrate the superiority of ECSS over state-of-the-art CSS systems. The
contribution of heterogeneous nodes in ECG and emotion renderer are further
demonstrated in ablation studies. To the best of our knowledge, ECSS is the
first in-depth conversational speech synthesis study that models emotional
expressiveness. We hope that our work will serve as a basis for future
emotional CSS studies.

## Acknowledgments

The research by Rui Liu was funded by the Young Scientists Fund of the
National Natural Science Foundation of China (No. 62206136), and Guangdong
Provincial Key Laboratory of Human Digital Twin (No. 2022B1212010004). The
research by Yifan Hu was funded by the Research and Innovation Projects for
Graduate Students in Inner Mongolia Autonomous Region (No. 11200-121028). The
work by Haizhou Li was supported by the National Natural Science Foundation of
China (Grant No. 62271432), and Shenzhen Science and Technology Research Fund
(Fundamental Research Key Project Grant No. JCYJ20220818103001002).

## References

  * Busso et al. (2008) Busso, C.; Bulut, M.; Lee, C.-C.; Kazemzadeh, A.; Mower, E.; Kim, S.; Chang, J. N.; Lee, S.; and Narayanan, S. S. 2008.  IEMOCAP: Interactive emotional dyadic motion capture database.  _Language resources and evaluation_ , 42: 335–359.
  * Dias et al. (2022) Dias, W.; Andalo, F.; Padilha, R.; Bertocco, G.; Almeida, W.; Costa, P.; and Rocha, A. 2022.  Cross-dataset emotion recognition from facial expressions through convolutional neural networks.  _Journal of Visual Communication and Image Representation_ , 82: 103395\.
  * Firdaus et al. (2020) Firdaus, M.; Chauhan, H.; Ekbal, A.; and Bhattacharyya, P. 2020.  MEISD: A Multimodal Multi-Label Emotion, Intensity and Sentiment Dialogue Dataset for Emotion Recognition and Sentiment Analysis in Conversations.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , 4441–4453. Barcelona, Spain (Online): International Committee on Computational Linguistics.
  * Ghosal et al. (2019) Ghosal, D.; Majumder, N.; Poria, S.; Chhaya, N.; and Gelbukh, A. 2019.  DialogueGCN: A Graph Convolutional Neural Network for Emotion Recognition in Conversation.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , 154–164. Hong Kong, China: Association for Computational Linguistics.
  * Goel et al. (2021) Goel, R.; Susan, S.; Vashisht, S.; and Dhanda, A. 2021.  Emotion-aware transformer encoder for empathetic dialogue generation.  In _2021 9th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW)_ , 1–6. IEEE.
  * Guo et al. (2020) Guo, H.; Zhang, S.; Soong, F. K.; He, L.; and Xie, L. 2020.  Conversational End-to-End TTS for Voice Agent.  _CoRR_ , abs/2005.10438.
  * Hu et al. (2020) Hu, Z.; Dong, Y.; Wang, K.; and Sun, Y. 2020.  Heterogeneous graph transformer.  In _Proceedings of the web conference 2020_ , 2704–2710.
  * Huang et al. (2022) Huang, R.; Ren, Y.; Liu, J.; Cui, C.; and Zhao, Z. 2022.  Generspeech: Towards style transfer for generalizable out-of-domain text-to-speech.  _Advances in Neural Information Processing Systems_ , 35: 10970–10983.
  * Im et al. (2022) Im, C.-B.; Lee, S.-H.; Kim, S.-B.; and Lee, S.-W. 2022.  EMOQ-TTS: Emotion Intensity Quantization for Fine-Grained Controllable Emotional Text-to-Speech.  In _ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 6317–6321.
  * Khosla et al. (2020) Khosla, P.; Teterwak, P.; Wang, C.; Sarna, A.; Tian, Y.; Isola, P.; Maschinot, A.; Liu, C.; and Krishnan, D. 2020.  Supervised Contrastive Learning.  _CoRR_ , abs/2004.11362.
  * Kim, Kong, and Son (2021) Kim, J.; Kong, J.; and Son, J. 2021.  Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech.  In _International Conference on Machine Learning_ , 5530–5540. PMLR.
  * Kong, Kim, and Bae (2020) Kong, J.; Kim, J.; and Bae, J. 2020.  Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis.  _Advances in Neural Information Processing Systems_ , 33: 17022–17033.
  * Lee, Park, and Kim (2023) Lee, K.; Park, K.; and Kim, D. 2023.  Dailytalk: Spoken dialogue dataset for conversational text-to-speech.  In _ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 1–5. IEEE.
  * Li et al. (2022a) Li, J.; Meng, Y.; Li, C.; Wu, Z.; Meng, H.; Weng, C.; and Su, D. 2022a.  Enhancing speaking styles in conversational text-to-speech synthesis with graph-based multi-modal context modeling.  In _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 7917–7921. IEEE.
  * Li et al. (2022b) Li, J.; Meng, Y.; Wu, X.; Wu, Z.; Jia, J.; Meng, H.; Tian, Q.; Wang, Y.; and Wang, Y. 2022b.  Inferring speaking styles from multi-modal conversational context by multi-scale relational graph convolutional networks.  In _Proceedings of the 30th ACM International Conference on Multimedia_ , 5811–5820.
  * Li et al. (2019) Li, N.; Liu, S.; Liu, Y.; Zhao, S.; and Liu, M. 2019.  Neural speech synthesis with transformer network.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 33, 6706–6713.
  * Li et al. (2022c) Li, T.; Wang, X.; Xie, Q.; Wang, Z.; and Xie, L. 2022c.  Cross-Speaker Emotion Disentangling and Transfer for End-to-End Speech Synthesis.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 30: 1448–1460.
  * Li et al. (2022d) Li, Z.; Tu, G.; Liang, X.; and Xu, R. 2022d.  Developing Relationships: A Heterogeneous Graph Network with Learnable Edge Representation for Emotion Identification in Conversations.  In _CAAI International Conference on Artificial Intelligence_ , 310–322. Springer.
  * Liu et al. (2021a) Liu, R.; Sisman, B.; Bao, F.; Yang, J.; Gao, G.; and Li, H. 2021a.  Exploiting Morphological and Phonological Features to Improve Prosodic Phrasing for Mongolian Speech Synthesis.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 29: 274–285.
  * Liu et al. (2021b) Liu, R.; Sisman, B.; Gao, G.; and Li, H. 2021b.  Expressive TTS Training With Frame and Style Reconstruction Loss.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 29: 1806–1818.
  * Liu et al. (2022a) Liu, R.; Sisman, B.; Gao, G.; and Li, H. 2022a.  Decoding Knowledge Transfer for Neural Text-to-Speech Training.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 30: 1789–1802.
  * Liu et al. (2021c) Liu, R.; Sisman, B.; Lin, Y.; and Li, H. 2021c.  Fasttalker: A neural text-to-speech architecture with shallow and group autoregression.  _Neural Networks_ , 141: 306–314.
  * Liu et al. (2022b) Liu, R.; Sisman, B.; Schuller, B.; Gao, G.; and Li, H. 2022b.  Accurate emotion strength assessment for seen and unseen speech based on data-driven deep learning.  _arXiv preprint arXiv:2206.07229_.
  * McTear (2022) McTear, M. 2022.  _Conversational ai: Dialogue systems, conversational agents, and chatbots_.  Springer Nature.
  * Poria et al. (2018) Poria, S.; Hazarika, D.; Majumder, N.; Naik, G.; Cambria, E.; and Mihalcea, R. 2018\.  MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations.  _CoRR_ , abs/1810.02508.
  * Ren et al. (2021) Ren, Y.; Hu, C.; Tan, X.; Qin, T.; Zhao, S.; Zhao, Z.; and Liu, T.-Y. 2021.  FastSpeech 2: Fast and High-Quality End-to-End Text to Speech.  In _International Conference on Learning Representations_.
  * Saganowski et al. (2022) Saganowski, S.; Komoszyńska, J.; Behnke, M.; Perz, B.; Kunc, D.; Klich, B.; Kaczmarek, Ł. D.; and Kazienko, P. 2022.  Emognition dataset: emotion recognition with self-reports, facial expressions, and physiology using wearables.  _Scientific data_ , 9(1): 158.
  * Seaborn et al. (2021) Seaborn, K.; Miyake, N. P.; Pennefather, P.; and Otake-Matsuura, M. 2021.  Voice in human–agent interaction: A survey.  _ACM Computing Surveys (CSUR)_ , 54(4): 1–43.
  * Song et al. (2023) Song, R.; Giunchiglia, F.; Shi, L.; Shen, Q.; and Xu, H. 2023.  SUNET: Speaker-utterance interaction Graph Neural Network for Emotion Recognition in Conversations.  _Engineering Applications of Artificial Intelligence_ , 123: 106315\.
  * Streijl, Winkler, and Hands (2016) Streijl, R. C.; Winkler, S.; and Hands, D. S. 2016.  Mean opinion score (MOS) revisited: methods and applications, limitations and alternatives.  _Multimedia Systems_ , 22(2): 213–227.
  * Tulshan and Dhage (2019) Tulshan, A. S.; and Dhage, S. N. 2019.  Survey on virtual assistant: Google assistant, siri, cortana, alexa.  In _Advances in Signal Processing and Intelligent Recognition Systems: 4th International Symposium SIRS 2018, Bangalore, India, September 19–22, 2018, Revised Selected Papers 4_ , 190–201. Springer.
  * Wang et al. (2017) Wang, Y.; Skerry-Ryan, R.; Stanton, D.; Wu, Y.; Weiss, R. J.; Jaitly, N.; Yang, Z.; Xiao, Y.; Chen, Z.; Bengio, S.; et al. 2017.  Tacotron: Towards End-to-End Speech Synthesis.  _Proc. Interspeech 2017_ , 4006–4010.
  * Wang et al. (2018) Wang, Y.; Stanton, D.; Zhang, Y.; Ryan, R.-S.; Battenberg, E.; Shor, J.; Xiao, Y.; Jia, Y.; Ren, F.; and Saurous, R. A. 2018.  Style tokens: Unsupervised style modeling, control and transfer in end-to-end speech synthesis.  In _International conference on machine learning_ , 5180–5189. PMLR.
  * Xue et al. (2023) Xue, J.; Deng, Y.; Wang, F.; Li, Y.; Gao, Y.; Tao, J.; Sun, J.; and Liang, J. 2023\.  M 2-CTTS: End-to-End Multi-Scale Multi-Modal Conversational Text-to-Speech Synthesis.  In _ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 1–5. IEEE.
  * Zhong, Wang, and Miao (2019) Zhong, P.; Wang, D.; and Miao, C. 2019.  Knowledge-Enriched Transformer for Emotion Detection in Textual Conversations.  _CoRR_ , abs/1909.10681.
  * Zhou et al. (2023) Zhou, K.; Sisman, B.; Rana, R.; Schuller, B. W.; and Li, H. 2023.  Emotion Intensity and its Control for Emotional Voice Conversion.  _IEEE Transactions on Affective Computing_ , 14(1): 31–48.
  * Zhou et al. (2020) Zhou, L.; Gao, J.; Li, D.; and Shum, H.-Y. 2020.  The design and implementation of xiaoice, an empathetic social chatbot.  _Computational Linguistics_ , 46(1): 53–93.
  * Zuo et al. (2023) Zuo, H.; Liu, R.; Zhao, J.; Gao, G.; and Li, H. 2023.  Exploiting Modality-Invariant Feature for Robust Multimodal Emotion Recognition with Missing Modalities.  In _ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 1–5.

##  Appendix A Appendix. Detailed Experimental Settings

We list the detailed model setup of ECSS. The detailed experimental settings
of the Heterogeneous Graph-based Emotional Context Encoder and Emotional
Conversational Speech Synthesizer are shown in Table 3.

Table 3: Detailed Model Setup of ECSS.

ECSS Model Setup
---
|  | Text Node Representation | 512
|  |  Emotion,Intensity, Speaker and Audio Node Representation | 256
|  | HGTConv Hidden Channels | 384
|  | HGTConv Head Number | 2
Heterogeneous Graph-based Emotional Context Encoder | Heterogeneous ECG | HGTConv Layer | 1
|  | ConvBlock2D Kernel | 3
|  | ConvBlock2D Layer | 2
|  | LSTM Input Dimension | 384
|  | LSTM Hidden State Size | 256
|  Emotion Predictor of Emotion Renderer |  | Emotion Predictor
---
Output Dimension
7
|  | ConvBlock2D Kernel | 3
|  | ConvBlock2D Layer | 2
|  | LSTM Input Dimension | 384
|  | LSTM Hidden State Size | 256
|  | AvgPool2d Kernel | 2
|  Intensity Predictor of Emotion Renderer | Output Dimension | 3
|  | Query Dimension | 512
|  | Key and Value Dimension | 384
|  |  | Prosody Predictor
---
Head Number
2
|  Prosody Predictor of Emotion Renderer | Output Dimension | 256
|  |  | Reference Encoder
---
Mel Channels
80
|  |  | Reference Encoder
---
GRU Size
128
|  |  | Style Token
---
Embedding Size
256
|  Prosody Extractor | Style Token Layer GSTs | 7
|  | Word Vector Dimension | 256
|  | Position Encoding Hidden | 256
|  | Transformer FFTBlock Layer | 4
|  | Transformer FFTBlock Head | 2
| Text Encoder | Transformer FFTBlock Dropout | 0.2
| Speaker Encoder | Speaker Embedding Size | 256
| Feature Aggregator |  | Feature Aggregator
---
Output Dimension
256
|  | Word Vector Dimension | 256
|  | Position Encoding Hidden | 256
|  | Transformer FFTBlock Layer | 6
|  | Transformer FFTBlock Head | 2
|  | Transformer FFTBlock Dropout | 0.2
|  | Postnet Embedding Dimension | 512
Emotional Conversational Speech Synthesizer | Acoustic Decoder | Postnet Kernel | 5


# Learning Class-level Prototypes for Few-shot Learning

Anonymous IJCAI 2021 submission Paper ID: 1316  Minglei Yuan1 Wenhai Wang1 Tao
Wang1 Chunhao Cai1 Qian Xu1&Tong Lu∗1 1Nanjing University
<EMAIL_ADDRESS>
###### Abstract

Few-shot learning aims to recognize new categories using very few labeled
samples. Although few-shot learning has witnessed promising development in
recent years, most existing methods adopt an average operation to calculate
prototypes, thus limited by the outlier samples. In this work, we propose a
simple yet effective framework for few-shot classification, which can learn to
generate preferable prototypes from few support data, with the help of an
episodic prototype generator module. The generated prototype is meant to be
close to a certain class-level prototype and is less influenced by outlier
samples. Extensive experiments demonstrate the effectiveness of this module,
and our approach gets a significant raise over baseline models, and get a
competitive result compared to previous methods on miniImageNet,
tieredImageNet, and cross-domain (miniImageNet $\rightarrow$ CUB-200-2011)
datasets.

##  1 Introduction

Although few-shot learning has witnessed promising development in recent
years, most existing methods adopt an average operation to calculate
prototypes, thus limited by the outlier samples. In this work, we propose a
simple yet effective framework for few-shot classification, which can learn to
generate preferable prototypes from few supporting data, with the help of an
episodic prototype generator module. The generated prototype is meant to be
close to a certain class-level prototype and is less influenced by outlier
samples. Extensive experiments demonstrate the effectiveness of this module,
and our approach gets a significant raise over baseline models, and get a
competitive result compared to previous methods on miniImageNet,
tieredImageNet, and cross-domain (miniImageNet $\rightarrow$ CUB-200-2011)
datasets.

The goal of few-shot learning is to learn new categories from limited labeled
data. In past years, many methods Vinyals et al. (2016); Snell et al. (2017);
Finn et al. (2017); Sung et al. (2018); Lifchitz et al. (2019); Rusu et al.
(2019); Nguyen et al. (2020) have been proposed to solve the few-shot
classification problem. Among them, some effective solutions Sung et al.
(2018); Snell et al. (2017); Simon et al. (2020a); Vinyals et al. (2016) are
to learn the embedding of support sets and query sets and then use the
embedding distance to classify the query samples. For example, ProtoNet
(Prototypical Networks) Snell et al. (2017) makes a few-shot classification by
measuring the Euclidean distance between the query embedding and the mean
embedding of each class in support set. RelationNet Sung et al. (2018)
develops a learnable module that learns to compare the query image against
few-shot labeled sample images.

Figure 1: The class-level prototype jitter phenomenon. Class-level prototypes
in a 3-way 5-shot task are subject to jitter due to outliers. The
classification of query samples $x_{q}$ is achieved by measuring the distance
between different classes of prototypes. (a) $x_{q}$ is misclassified due to
the influence of outlier points, as the averaging strategy is easily affected
by the existence of outliers. (b) Our method generates episodic prototypes
that are closer to the class-level prototypes, fixing classification errors.

Although many few-shot classification algorithms have achieved promising
performance, they rarely concentrate on designing a good strategy to generate
class-level prototypes to represent each class in the support set. Some recent
methods Sung et al. (2018); Snell et al. (2017); Oreshkin et al. (2018);
Lifchitz et al. (2019); Wang et al. (2018) adopt the averaging strategy
(_i.e.,_ using the arithmetic mean of embeddings of support samples in each
class) to approximate class-level prototypes. Although this strategy works
well in ideal situations where sufficient samples are given and each of them
is independently sampled from the same distribution Khintchine (1936).
However, in a few-shot classification, the influence of one single outlier on
the averaging strategy may be significant because there are few supporting
samples. This phenomenon is named as class-level prototype jitter in Cai et
al. (2020), which is shown in Figure 1.

To remedy this problem, several methods Simon et al. (2020a, 2019) choose a
preferable class-level prototype from the linear subspace constructed by the
support set, whose main spirit is to calculate a subspace of the feature space
for each category, and then project the feature vector of the query sample
into the subspace, and take the distance measurement in the subspace as the
criterion for predicting the category. An iterative process is proposed in
IFSM Cai et al. (2020) to optimize prototypes, which leads to high time
complexity due to complex iterations. In addition, PSN Simon et al. (2019) and
IFSM Cai et al. (2020) do not work on 1-shot tasks. And all previous methods
omit the guidance of class-level prototypes, which is the average feature of a
large number of samples of the same class in the training set.

To solve this problem, we propose an episodic prototype generator to extract
the prototype supervised with the class-level prototype. In this way, the
proposed episodic prototype generator needs to generate a prototype that can
converge to a more robust prototype. This reduces the impact of outliers in
the prototype calculation.

Specifically, we propose an effective framework for the few-shot
classification task. The proposed method consists of three main modules: the
class-level prototype generator module, the episodic prototype generator
module, and the metrics module. Among them, the class-level prototype
generator module aims to learn class-level prototypes (_i.e.,_ the global mean
of sample features for each class in the training set) from the training data
so that our algorithm can directly benefit from the class-level prototype when
dealing with few-shot classification problem. The episodic prototype generator
is a learnable module constructed with a Transformer Vaswani et al. (2017). It
can help the network capture the contextual information from samples with the
same category. Besides, the output of the episodic prototype generator is
supervised with class-level prototypes produced by class-level prototype
generator module. In this way, the proposed episodic prototype generator model
can tender to get the prototype closer to the class-level prototypes and
reduce the impact of outlier points. Finally, the metric module determines the
category to which the query sample belongs based on the distance.

Extensive experiments demonstrate that the proposed method is competitive with
state-of-the-art few-shot learning methods. Firstly, we directly use class-
level prototype for classification and get a surprising result with an
accuracy of 89.15%. The experiment proves it is wise to choose it as the
classification basis. Secondly, we conduct consistent comparative experiments
on two datasets to compare several representative methods of few-shot
classification. We achieve the state-of-the-art performance on tieredImageNet
5-way 1-shot and 5-shot, and on miniImageNet 5-way 1-shot, with accuracy of
71.78%, 85.91% and 66.68%, respectively. Finally, we further evaluate the
performance of this method on cross-domain tasks. The fact shows that our
model has a strong ability to migrate and is highly competitive with other
methods.

The contributions of our work are as follow:

  1. 1.

We assume that the class-level prototypes can be good candidates for class-
level prototypes in few-shot classification, and justify it through ablation
experiments.

  2. 2.

We propose a mechanism which takes support samples as input and generates
class-level prototypes relatively close to corresponding class-level
prototypes.

  3. 3.

We show that the proposed mechanism brings a significant rise to the
performance of various few-shot learning methods on miniImageNet,
tieredImageNet and a cross-domain task (miniImageNet $\rightarrow$
CUB-200-2011).

##  2 Related Works

Few-shot learning has been of great interest for a long time, and a large
quantify of few-shot learning algorithms have been proposed in the literature.
These methods can be approximately divided into two categories. The first kind
of approach aims at strategies of constructing a general classifier. They show
feasible performance on few-shot tasks. Concretely, this can be done by
inventing a certain data augmentation algorithm Li et al. (2019) or
optimization strategy based on the content of the support and query sets
Vinyals et al. (2016). This can also be achieved by directly predicting the
weights of the classifier Rusu et al. (2019).

The other school of few-shot learning methods Snell et al. (2017); Vinyals et
al. (2016); Sung et al. (2018); Li et al. (2019); Cai et al. (2020); Simon et
al. (2020a) focuses on learning a feature extractor to embed all samples in
the support set and the query set, followed by a classifier making its
decisions based on the embedding of query samples and embeddings of support
samples. These methods are usually called metric-based few-shot learning
methods. Early methods of this category directly compare the embedding of
query samples to embeddings of all support samples, which are similar to
weighted nearest-neighbor classifiers like Matching Network Vinyals et al.
(2016). Prototypical Network Snell et al. (2017) decides that it is more
efficient to use prototypes, _i.e.,_ embeddings of classes to compare with
embeddings of query samples.

Besides, merely using the arithmetic mean as prototypes are far from feasible.
Thus, several methods have been proposed to alleviate this problem through
different means. For instance, CTM Li et al. (2019) introduces a category
traversal module for generating prototypes, which can explicitly extract
intra-class commonality and inter-class uniqueness. IFSM Cai et al. (2020)
assumes a relatively feasible prototype can be represented by a linear
combination of embeddings of support samples from that class and adopts an
iterative process to gradually optimize that linear combination. Subspace
Network Simon et al. (2020a) is based on a similar assumption. However, it
generates an independent set of prototypes for each query sample and is
limited to certain types of distance metrics.

Our work is somehow based on a similar idea compared to these methods, which
is to generate prototypes through a learnable neural network. However, our
work is different in that our model is supervised by the class-level
prototype, which is a good candidate for the prototype. This facilitates us to
obtain a prototype that is closer to class-level prototype.

Figure 2: The overview of our proposed framework for few-shot learning, which
is mainly composed of a class-level prototype generator module, an episodic
prototype generator module, and a metric module.

##  3 Preliminary

In this section, we introduce some notation and formalization of few-shot
learning in an image classification task. Let $(x,y)$ denotes a labeled
example where $x$ is a sample and $y$ is the corresponding ground-truth.
Datasets are often divided into a base class dataset $\mathcal{D}_{train}$ and
a novel class dataset $\mathcal{D}_{novel}$, and $\mathcal{D}_{train}$ and
$\mathcal{D}_{novel}$ do not intersect. The purpose of few-shot learning is to
build a good meta-learner $\Phi_{\theta}$ for the novel classes in
$\mathcal{D}_{novel}$ with a few labeled samples (or called supported
samples).

Meta-learning Vinyals et al. (2016) is an effective way for few-shot learning,
which can utilize the training set in an episodic manner by mimicking the few-
shot learning setting. Meta-learning consists of two phases: meta-training and
meta-test.

In the meta-training phase, data is randomly sampled from
$\mathcal{D}_{train}$ to simulate a scenario $\mathcal{T}$, and the few-shot
learning meta-learner $\Phi_{\theta}$ is trained with thousands of randomly
sampled scenarios. In each scenario, the sampled data are divided into a
support set $\mathcal{S}$ and a query set $\mathcal{Q}$, which satisfies that
the elements of $\mathcal{S}$ and $\mathcal{Q}$ do not intersect, but are
drawn from the same class. The support set $\mathcal{S}$ and the query set
$\mathcal{Q}$ are formalized as follow:

$\begin{split}\mathcal{S}=\\{x_{i}|y_{i}=k,(x_{i},y_{i})\in\mathcal{D}_{s}\\},\\\
\mathcal{Q}=\\{x_{i}|y_{i}=k,(x_{i},y_{i})\in\mathcal{D}_{q}\\}.\end{split}$
(1)

In the meta-test phase, the learned few-shot learning meta-learner
$\Phi_{\theta}$ is tested with a large number of episodes drawn from
$\mathcal{D}_{novel}$, and the average prediction accuracy is used to measure
the learning ability of $\Phi_{\theta}$.

In this paper, we consider only the $N$-way $K$-shot few-shot learning
paradigm, where each support set $\mathcal{S}$ contains $N$ classes, each
class includes $K$ samples, and the query set $\mathcal{Q}$ consists of
unlabeled samples belonging to the classes of support set $\mathcal{S}$.

Meta-learner $\Phi_{\theta}$ is trained to fit few-shot classification tasks
by minimizing the prediction loss on the query set $\mathcal{Q}$ as in Eq.
(2).

$\underset{\theta}{\operatorname{arg~{}min~{}}}\frac{1}{N_{q}}\sum_{(x_{i},y_{i})\in\mathcal{Q}}\mathcal{L}(y_{i},\Phi_{\theta}(x_{i},\mathcal{S})),$
(2)

where $N_{q}$ is the number of samples in $\mathcal{Q}$, and $\mathcal{L}$ is
the loss function. The meta-learner $\Phi_{\theta}$ is required to train with
thousands of randomly sampled tasks under the constraint of the loss function
$\mathcal{L}$.

However, the proposed framework is trained using distance loss. The details
are given in Section 4.

##  4 Method

###  4.1 Framework

The key idea of our model is to elaborate how to get a better prototype
supervised with a class-level prototype in case of insufficient support
samples. To overcome the class-level prototype jitter problem, we propose a
pipeline to fit class-level prototype using few support samples. Figure 2
shows the overall architecture of our proposed model. The proposed algorithm
consists of three main components: a class-level prototype generator module,
an episodic prototype generator module and a metric module.

As illustrated in Figure 2, class-level prototype works in three steps. The
training data are first fed into the class-level prototype generator module. A
generic classifier is first trained with the training set and the parameters
of the classifier are frozen. Then, the features of all samples in each
category are obtained using the trained classifier. Finally, the arithmetic
means of all sample features in each category are used as class-level
prototype. Class-level prototype generator module provides a backbone with
fixed parameters for the following episodic prototype generator module. The
episodic prototype generator is designed to produce prototype that is closer
to class-level prototype, which is detailed in Section 4.3. The metric module
accomplishes the recognition of images by measuring the distance between the
episodic prototype features and the feature vectors of images in query set. We
will introduce the details of these modules in the following.

###  4.2 Class-level Prototype Generator Module

The class-level prototype generator module aims at generating the average of
the features of all samples in the same class. Existing few-shot learning
methods are usually trained and tested on miniImageNet Vinyals et al. (2016)
and tieredImageNet Ren et al. (2018). Each class of the training set
$\mathcal{D}_{train}$ consists of a considerable number of samples. In this
paper, we consider the global class prototype as the best candidate for the
class-level prototype. And then propose the class-level prototype to optimize
the episodic prototype. Specific optimization method will be described in
detail in Section 4.3. We focus on how to calculate class-level prototype in
this part.

To obtain the class-level prototype, a classifier is first trained using
$\mathcal{D}_{train}$. The classifier consists of a backbone network
$\mathcal{F}_{\theta}$ and a fully connected network with a softmax layer and
is optimized using cross-entropy loss. Specifically, the backbone network
$\mathcal{F}_{\theta}$ is used to extract the features of images with a type
of residual network He et al. (2016). The detailed structure of the backbone
network $\mathcal{F}_{\theta}$ is presented in Section 5.2.

Then, the trained backbone network $\mathcal{F}_{\theta}$ is used to extract
the features of samples in each class in the training set, and the arithmetic
mean of all features in the same class is adopted as class-level prototype.

$g_{i}=\frac{1}{M}\sum_{j=1}^{M}\mathcal{F}_{\theta}(x_{ij}),$ (3)

where $M$ is the total number of samples in the _i_ -th class, $x_{ij}$
indicates the input sample.

###  4.3 Episodic Prototype Generator Module

When calculating the prototype of an episode using the arithmetic mean in a
few-shot learning scenario, the prototypes are often affected by outlier
points. In this section, we propose an episodic prototype generator module,
which is designed to narrow the difference between the prototype and the
class-level prototype. The proposed episodic prototype generator generates a
prototype under the supervision of a class-level prototype, and the final
optimization goal is to obtain a prototype that is closer to the class-level
prototype.

To achieve this, in each episode, the samples $\\{x_{i1},\cdots,x_{iK}\\}$ in
the same class are first fed into the pre-trained feature extractor
$\mathcal{F}_{\theta}$, which can obtain their feature vectors, denoted as
$f_{ij}=\mathcal{F}_{\theta}(x_{ij})$. Then, the extracted feature vectors
$\\{f_{i1},\cdots,f_{iK}\\}$ are fed into the episodic prototype generator
$\mathcal{G}_{\gamma}$ and produce episodic prototype $f^{\prime}_{i}$,
denoted as Eq 4.

$f^{\prime}_{i}=\mathcal{G}_{\gamma}(f_{i1},\cdots,f_{iK}),$ (4)

where $\gamma$ is the parameters of the episodic prototype generator
$\mathcal{G}$, $i$ indicates the _i_ -th category.

Finally, we compute the distance between the episodic prototype
$f^{\prime}_{i}$ and class-level prototype $\hat{f}_{i}$ and use the distance
loss to optimize the parameters $\gamma$ of the episodic prototype generator
$\mathcal{G}_{\gamma}$. Euclidean distance is adopted to measure the distance
loss in this paper.

$\mathcal{L}_{distance}=\frac{1}{N}\sum_{i=1}^{N}\mathcal{M}(f^{\prime}_{i},\hat{f}_{i}),$
(5)

where $\mathcal{M}$ is a distance metric module, $N$ indicates the number of
categories in an episode, $\hat{f}_{i}$ is the class-level prototype of the
_i_ -th category in an episode.

In this paper, we employ a multi-head self-attention mechanism Vaswani et al.
(2017) as the feature generator $\mathcal{G}$, which considers the contextual
information of samples in the same episode. The multi-head self-attention
mechanism outputs the refined feature vectors of the input samples. We use the
mean of updated feature vectors as prototypes. These prototypes can converge
to the class-level prototype with the help of the loss function
$\mathcal{L}_{distance}$, thus eliminating the effect caused by outliers in
input samples.

###  4.4 Metric Module

The metric module $\mathcal{M^{{}^{\prime}}}$ classifies the images in the
query set. First, the trained backbone network $\mathcal{F}_{\theta}$ is used
to extract the features of samples in the query set. Then, the metric module
$\mathcal{M^{{}^{\prime}}}$ classifies the query samples according to the
distance between the features of images in the query set and the episodic
prototype features $\\{f^{\prime}_{1},\cdots,f^{\prime}_{N}\\}$. In this
paper, Euclidean distance is chosen as the metric to obtain the classification
accuracy.

Table 1: 5-way few-shot classification accuracy (%) on miniImageNet and
tieredImageNet. We report the average accuracy and 95% confidence interval of
600 randomly generated test sets for each task. The superscript † indicates
that the experimental results are from Chen et al. (2019a). Red: the best;
Blue: the second best.

Methods | Backbone | Source | miniImageNet | tieredImageNet
---|---|---|---|---
5-way 1-shot | 5-way 5-shot | 5-way 1-shot | 5-way 5-shot
MatchNet Vinyals et al. (2016) | 4CONV | NeurIPS16 | 46.6 | 60.0 | - | -
ProtoNet Snell et al. (2017) | 4CONV | NeurIPS17 | 49.42 $\pm$ 0.78 | 68.20 $\pm$ 0.66 | 53.31 $\pm$ 0.89 | 72.69 $\pm$ 0.74
MAML Finn et al. (2017) | 4CONV | ICML17 | 48.70 $\pm$ 1.84 | 63.11 $\pm$ 0.92 | - | -
RelationNet Sung et al. (2018) | 4CONV | CVPR18 | 50.44 $\pm$ 0.82 | 65.32 $\pm$ 0.70 | 54.48 $\pm$ 0.93 | 71.32 $\pm$ 0.78
PSN Simon et al. (2019) | 4CONV | ICLR19 | - | 66.62 $\pm$ 0.69 | - | -
PAC-Bayesian Nguyen et al. (2020) | 4CONV | TPAMI20 | 52.11 $\pm$ 0.43 | 63.87 $\pm$ 0.35 | - | -
PN+IFSM Cai et al. (2020) | 4CONV | ICPR20 | - | 66.98$\pm$ 0.68 | - | 78.81 $\pm$ 0.66
DC Lifchitz et al. (2019) | ResNet-12 | CVPR19 | 61.26 $\pm$ 0.20 | 79.01 $\pm$ 0.13 | - | -
ProtoNet+TRAML Li et al. (2020a) | ResNet-12 | CVPR20 | 60.31 $\pm$ 0.48 | 77.94 $\pm$ 0.57 | - | -
MetaOptNet Lee et al. (2019) | ResNet-12 | CVPR19 | 64.09 $\pm$ 0.62 | 80.00 $\pm$ 0.45 | 65.81 $\pm$ 0.74 | 81.75 $\pm$ 0.53
Ravichandran et al. Ravichandran et al. (2019) | ResNet-12 | ICCV19 | 60.71 | 77.26 | 66.87 | 82.43
Y. Tian et al. Tian et al. (2020) | ResNet-12 | ECCV20 | 62.02 $\pm$ 0.63 | 79.64 $\pm$ 0.44 | 69.74 $\pm$ 0.72 | 84.41 $\pm$ 0.55
DSN-MR Simon et al. (2020a) | ResNet-12 | CVPR20 | 64.60 $\pm$ 0.72 | 79.51 $\pm$ 0.50 | 67.39 $\pm$ 0.82 | 82.85 $\pm$ 0.56
Baseline Chen et al. (2019a) | ResNet-18 | ICLR19 | 51.75 $\pm$ 0.80 | 74.27 $\pm$ 0.63 | - | -
Baseline++ Chen et al. (2019a) | ResNet-18 | ICLR19 | 51.87 $\pm$ 0.77 | 75.68 $\pm$ 0.63 | - | -
MatchNet†Vinyals et al. (2016) | ResNet-18 | ICLR19 | 52.91 $\pm$ 0.88 | 68.88 $\pm$ 0.69 | - | -
ProtoNet†Snell et al. (2017) | ResNet-18 | ICLR19 | 54.16 $\pm$ 0.82 | 73.68 $\pm$ 0.65 | - | -
MAML†Finn et al. (2017) | ResNet-18 | ICLR19 | 49.61 $\pm$ 0.92 | 65.72 $\pm$ 0.77 | - | -
RelationNet†Sung et al. (2018) | ResNet-18 | ICLR19 | 52.48 $\pm$ 0.86 | 69.83 $\pm$ 0.68 | - | -
IDeMe-Net Chen et al. (2019b) | ResNet-18 | CVPR19 | 59.14 $\pm$ 0.86 | 74.63 $\pm$ 0.74 | - | -
Robust 20-dist++ Dvornik et al. (2019) | ResNet-18 | ICCV19 | 59.48 $\pm$ 0.62 | 75.62 $\pm$ 0.48 | - | -
Su et al. Su et al. (2020) | ResNet-18 | ECCV20 | - | 76.6 | - | -
AFHN Li et al. (2020b) | ResNet-18 | CVPR20 | 62.38 $\pm$ 0.72 | 78.16 $\pm$ 0.56 | - | -
MTL+E3BM Liu et al. (2020) | ResNet-25 | ECCV20 | 64.3 | 81.0 | 70.0 | 85.0
CC+rot Gidaris et al. (2019) | WRN-28-10 | ICCV19 | 62.93 $\pm$ 0.45 | 79.87 $\pm$ 0.33 | 62.93 $\pm$ 0.45 | 79.87 $\pm$ 0.33
LEO Rusu et al. (2019) | WRN-28-10 | ICLR19 | 61.76 $\pm$ 0.08 | 77.59 $\pm$ 0.12 | 66.33 $\pm$ 0.05 | 81.44 $\pm$ 0.09
Centroid alignment Afrasiyabi et al. (2020) | WRN-28-10 | ECCV20 | 65.92 $\pm$ 0.60 | 82.85 $\pm$ 0.55 | - | -
PAC-Bayesian Nguyen et al. (2020) | no-standard | TPAMI20 | 63.73 $\pm$ 0.57 | 78.04 $\pm$ 0.45 | 70.82 $\pm$ 0.30 | 81.84 $\pm$ 0.21
Ours | ResNet-12 | - | 66.68$\pm$0.81 | 82.22 $\pm$ 0.56 | 71.78 $\pm$ 0.91 | 85.91$\pm$0.60

##  5 Experiments

###  5.1 Datasets

####  miniImageNet.

It contains 100 classes with 600 images in each class. Following the previous
work Sung et al. (2018); Snell et al. (2017), we divide this dataset into
training, validation, and testing sets, with 64, 16, and 20 classes,
respectively.

####  tieredImageNet.

It consists of a total of 779,165 images with 608 classes. Each image in
tieredImageNet has a size of $84\times 84$. It is a much more challenging
dataset because of the realistic regime of test classes that are less similar
to training ones. In our experiments, we follow the setting of Li et al.
(2019) where $351$ classes are used to make up the training set, 97 for
validation and 160 for testing.

####  miniImageNet $\rightarrow$ CUB-200-2011.

To evaluate the effects of the proposed algorithm in a cross-domain scenario,
we use the miniImageNet as our training set and the validation set and testing
set are from CUB-200-2011 Wah et al. (2011). The miniImageNet involves generic
objects, while CUB-200-2011 contains 200 classes and 11,788 fine-grained bird
images. This dataset mainly validates the cross-domain learning capability of
the model.

###  5.2 Implementation Details

#### Backbone.

We adopt a 12-layer residual network (ResNet-12) He et al. (2016) as the
backbone network of our algorithm Lifchitz et al. (2019); Lee et al. (2019);
Simon et al. (2020b). ResNet-12 is composed of four residual blocks He et al.
(2016). Each residual block contains three $3\times 3$ convolutional layers
and a shortcut connection. At the same time, every convolutional layer is
followed by a BathNorm layer and a LeakyReLU activation function (negative
slope 0.1). In our experiments, the backbone network is pre-trained on the
training set $\mathcal{D}_{base}$ like Rusu et al. (2019).

#### Training Schema.

As many few-shot learning methods Snell et al. (2017); Sung et al. (2018), we
utilize a meta-training strategy to train our model. Specifically, in each
N-way K-shot episode, we randomly sample K images per class to make up a
support set, and $15$ images per class for the query set. The total training
process lasts $200$ epochs, and each epoch contains 200 episodes. We adopt
stochastic gradient descent (SGD) as the optimizer. The learning rate is
initially set to $0.01$ and is multiplied by $0.618$ whenever the validation
score does not improve in $7$ consecutive epochs.

#### Evaluation protocols.

We evaluate our model on the validation at the end of each epoch and verify
the performance of our model in the testing set. Specifically, we test the
models with 600 randomly sampled tasks and calculate the mean and 95%
confidence interval of the prediction accuracy. In each sampling task, we
randomly sampled 15 images for each category.

###  5.3 Comparison with State-of-the-arts

In Table 1, we compare the performance of our algorithm with several state-of-
the-art few-shot learning methods on the miniImageNet and the tieredImageNet.
On the tieredImageNet and the miniImageNet, our algorithm achieves a more
favorable score than other methods on 1-shot classification. On the
tieredImageNet, our solution outperforms other methods on 1-shot and 5-shot
classification by a larger margin. Centroid alignment Afrasiyabi et al. (2020)
is slightly better than ours when dealing with 5-way 5-shot tasks on
miniImageNet. However, Centroid alignment Afrasiyabi et al. (2020) adopts
WRN-28-10 as the backbone, which introduces a lot more parameters than
ResNet-12. As is pointed out in Chen et al. (2019a), the parameter size of the
backbone has a great impact on the performance of a few-shot learning method.
Note that PSN Simon et al. (2019), DSN-MR Simon et al. (2020a) and PN+IFSM Cai
et al. (2020) are all proposed to optimize the class-level prototypes by
learning to form an affine subspace or by an iterative optimization mechanism,
respectively. But PSN Simon et al. (2019) and PN+IFSM Cai et al. (2020) cannot
be applied to the 1-shot classification task. DSN-MR Simon et al. (2020a) has
more flexibility than PSN Simon et al. (2019) and PN+IFSM Cai et al. (2020).
DSN-MR Simon et al. (2020a) uses the same backbone as we do, but our method
has higher accuracy.

###  5.4 Performance in Cross-domain Scenario

In Table 2, we experiment our method on cross-domain tasks, which is trained
on miniImageNet and tested on CUB-200-2011. In our experiment, ResNet-12 is
used as the backbone, which is the same as the comparable method. Table 2
shows that our method is the best performer on cross-domain tasks, which means
that the episodic prototype generator can benefit in the cross-domain
scenario.

Table 2: Few-shot classification accuracy (%) on a cross-domain task (miniImageNet $\rightarrow$ CUB-200-2011). † indicates that the experimental results are from Chen et al. (2019a). Bold: the best. Methods | 5-way 5-shot
---|---
Baseline++† Chen et al. (2019a) | 62.04 $\pm$ 0.76
MatchingNet† Vinyals et al. (2016) | 53.07 $\pm$ 0.74
MAML† Finn et al. (2017) | 51.34 $\pm$ 0.72
ProtoNet† Snell et al. (2017) | 62.02 $\pm$ 0.70
RelationNet† Sung et al. (2018) | 57.71 $\pm$ 0.73
Ours | 62.60 $\pm$ 0.75

###  5.5 The Effect of Episodic Prototype Generator

We conduct ablation experiments to verify the effectiveness of our mechanism.
All the experiments are performed on 5-way 1-shot and 5-way 5-shot tasks, with
miniImageNet being the dataset.

Table 3: Ablation studies on effect of prediction accuracy (%). Methods | 5-way
---|---
1-shot | 5-shot
ProtoNet+Lock | 59.13 $\pm$ 0.86 | 80.88 $\pm$ 0.58
ProtoNet | 61.17 $\pm$ 0.82 | 79.86 $\pm$ 0.55
Ours | 66.68 $\pm$ 0.81 | 82.22 $\pm$ 0.56
Class-level prototype | 89.15 $\pm$ 0.37 | 89.15 $\pm$ 0.37

In Table 3. We first test ProtoNet+Lock and ProtoNet. The former has frozen
backbone parameters, while the latter is learnable during the training phase.
They both use the arithmetic mean as the episodic prototype. Our method is
based on ProtoNet+Lock with the feature generator module applied. As can be
seen from Table 3, the proposed module has a larger improvement of 7.55% in
1-shot setting and 1.34% in 5-shot settings compared with ProtoNet+Lock.
Besides, the proposed module also shows a larger improvement compared to
ProtoNet, which has learnable parameters. This shows the effectiveness of our
episodic prototype generator module.

Figure 3: Visualization of episodic prototype features and mean prototype
features of different models. Different colors represent categories, star-
shaped point represent class-level prototype, triangles represent mean
prototype features of ProtoNet Snell et al. (2017), pentagons are episodic
prototypes generated by our episodic prototype generator module, large dots
are features of support samples, and small dots are features of randomly
selected samples of the same class.

We also report the prediction results using class-level prototype, which are
the ideal results. This experiment uses the class-level prototypes of each
support sample to classify the query samples, and the purpose of reporting
this result is to explore the upper bound of the capability of our proposed
method. At the same time, we can see that the prediction accuracy can reach
89.15% when using the class-level prototype as the episodic prototype. There
leaves a big gap for improvement, and we believe that the gap can be further
reduced by improving the ability of the episodic prototype generator module.

To verify the validity of the proposed method, we visualize the features of
the support samples, class-level prototypes, episodic prototypes, and mean
prototypes using tsne Maaten and Hinton (2008) in Figure 3. The mean prototype
feature is generated by ProtoNet Snell et al. (2017), which uses the same
backbone as our model, and the episodic prototypes are generated by our model.
It shows that there are outliers in the purple and blue classes, resulting in
a large gap between the mean prototype of these two classes and their class-
level prototypes, while the episodic prototype can be closer to class-level
prototype. It shows that our method can obtain a more robust prototype in the
presence of outliers.

The above experiments show that in the presence of outliers, the proposed
episodic prototype generator can obtain more robust prototypes and improve the
prediction accuracy in the presence of few-shot learning.

##  6 Discussion

In a few-shot learning scenario, the arithmetic mean prototype tends to be
influenced by outlier points. Through experimental observations, we find
class-level prototype can greatly improve few-shot learning model. In this
paper, we propose a simple method to learn class-level prototypes from few
support samples. This method is based on the idea that we can generate more
reliable prototypes supervised with class-level prototypes. The proposed
method can produce competitive results than most metric-based few-shot
learning methods, only by improving the prototype.

## References

  * Afrasiyabi et al. [2020] Arman Afrasiyabi, Jean-François Lalonde, and Christian Gagné.  Associative alignment for few-shot image classification.  In ECCV, 2020.
  * Cai et al. [2020] Chunhao Cai, Minglei Yuan, and Tong Lu.  Ifsm: An iterative feature selection mechanism for few-shot image classification.  In ICPR, 2020.
  * Chen et al. [2019a] WeiYu Chen, YenCheng Liu, Zsolt Kira, YuChiang Frank Wang, and JiaBin Huang.  A closer look at few-shot classification.  In ICLR, 2019.
  * Chen et al. [2019b] Zitian Chen, Yanwei Fu, Yu-Xiong Wang, Lin Ma, Wei Liu, and Martial Hebert.  Image deformation meta-networks for one-shot learning.  In CVPR, 2019.
  * Dvornik et al. [2019] Nikita Dvornik, Julien Mairal, and Cordelia Schmid.  Diversity with cooperation: Ensemble methods for few-shot classification.  In ICCV, 2019.
  * Finn et al. [2017] Chelsea Finn, Pieter Abbeel, and Sergey Levine.  Model-agnostic meta-learning for fast adaptation of deep networks.  In ICML, 2017.
  * Gidaris et al. [2019] Spyros Gidaris, Andrei Bursuc, Nikos Komodakis, Patrick Pérez, and Matthieu Cord.  Boosting few-shot visual learning with self-supervision.  In ICCV, 2019.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, 2016.
  * Khintchine [1936] A Ya Khintchine.  Su una legge dei grandi numeri generalizzata.  Giorn. Ist. Ital. Attuari, 1936.
  * Lee et al. [2019] Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto.  Meta-learning with differentiable convex optimization.  In CVPR, 2019.
  * Li et al. [2019] Hongyang Li, David Eigen, Samuel Dodge, Matthew Zeiler, and Xiaogang Wang.  Finding task-relevant features for few-shot learning by category traversal.  In CVPR, 2019.
  * Li et al. [2020a] Aoxue Li, Weiran Huang, Xu Lan, Jiashi Feng, Zhenguo Li, and Liwei Wang.  Boosting few-shot learning with adaptive margin loss.  In CVPR, 2020.
  * Li et al. [2020b] Kai Li, Yulun Zhang, Kunpeng Li, and Yun Fu.  Adversarial feature hallucination networks for few-shot learning.  In CVPR, 2020.
  * Lifchitz et al. [2019] Yann Lifchitz, Yannis Avrithis, Sylvaine Picard, and Andrei Bursuc.  Dense classification and implanting for few-shot learning.  In CVPR, 2019.
  * Liu et al. [2020] Yaoyao Liu, Bernt Schiele, and Qianru Sun.  An ensemble of epoch-wise empirical bayes for few-shot learning.  In ECCV, 2020.
  * Maaten and Hinton [2008] Laurens van der Maaten and Geoffrey Hinton.  Visualizing data using t-sne.  JMLR, 9(Nov):2579–2605, 2008.
  * Nguyen et al. [2020] Cuong Nguyen, Thanh-Toan Do, and Gustavo Carneiro.  Pac-bayesian meta-learning with implicit prior and posterior.  TPAMI, 2020.
  * Oreshkin et al. [2018] Boris Oreshkin, Pau Rodríguez López, and Alexandre Lacoste.  Tadam: Task dependent adaptive metric for improved few-shot learning.  In NIPS, 2018.
  * Ravichandran et al. [2019] Avinash Ravichandran, Rahul Bhotika, and Stefano Soatto.  Few-shot learning with embedded class models and shot-free meta training.  In ICCV, 2019.
  * Ren et al. [2018] Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B Tenenbaum, Hugo Larochelle, and Richard S Zemel.  Meta-learning for semi-supervised few-shot classification.  In ICLR, 2018.
  * Rusu et al. [2019] Andrei A. Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell.  Meta-learning with latent embedding optimization.  In ICLR, 2019.
  * Simon et al. [2019] Christian Simon, Piotr Koniusz, and Mehrtash Harandi.  Projective subspace networks for few-shot learning.  In ICLR, 2019.
  * Simon et al. [2020a] Christian Simon, Piotr Koniusz, Richard Nock, and Mehrtash Harandi.  Adaptive subspaces for few-shot learning.  In CVPR, 2020.
  * Simon et al. [2020b] Christian Simon, Piotr Koniusz, Richard Nock, and Mehrtash Harandi.  Adaptive subspaces for few-shot learning.  In CVPR, 2020.
  * Snell et al. [2017] Jake Snell, Kevin Swersky, and Richard Zemel.  Prototypical networks for few-shot learning.  In NIPS, 2017.
  * Su et al. [2020] Jong-Chyi Su, Subhransu Maji, and Bharath Hariharan.  When does self-supervision improve few-shot learning?  In ECCV, 2020.
  * Sung et al. [2018] Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales.  Learning to compare: Relation network for few-shot learning.  In CVPR, 2018.
  * Tian et al. [2020] Yonglong Tian, Yue Wang, Dilip Krishnan, Joshua B. Tenenbaum, and Phillip Isola.  Rethinking few-shot image classification: A good embedding is all you need?  In ECCV, 2020.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In NIPS, 2017.
  * Vinyals et al. [2016] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al.  Matching networks for one shot learning.  In NIPS, 2016.
  * Wah et al. [2011] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie.  The caltech-ucsd birds-200-2011 dataset.  2011\.
  * Wang et al. [2018] Yu-Xiong Wang, Ross Girshick, Martial Hebert, and Bharath Hariharan.  Low-shot learning from imaginary data.  In CVPR, 2018.