# Interactive Generative Adversarial Networks for Facial Expression Generation
in Dyadic Interactions

Behnaz Nojavanasghari
University of Central Florida
<EMAIL_ADDRESS>Yuchi Huang
Educational Testing Service
<EMAIL_ADDRESS>Saad Khan
Educational Testing Service
<EMAIL_ADDRESS>
###### Abstract

A social interaction is a social exchange between two or more individuals,
where individuals modify and adjust their behaviors in response to their
interaction partners. Our social interactions are one of most fundamental
aspects of our lives and can profoundly affect our mood, both positively and
negatively. With growing interest in virtual reality and avatar-mediated
interactions, it is desirable to make these interactions natural and human
like to promote positive effect in the interactions and applications such as
intelligent tutoring systems, automated interview systems and e-learning. In
this paper, we propose a method to generate facial behaviors for an agent.
These behaviors include facial expressions and head pose and they are
generated considering the users affective state. Our models learn semantically
meaningful representations of the face and generate appropriate and temporally
smooth facial behaviors in dyadic interactions.

##  1 Introduction

Designing interactive virtual agents and robots have gained a lot of attention
in recent years [10, 21, 43]. The popularity and growing interest in these
agents is partially because of their wide applications in real world
scenarios. They can be used for variety of applications from education [30,
31], training [36] and therapy [7, 34] to elderly care [5] and companionship
[6]. One of the most important aspects in human communication is social
intelligence [3]. A part of social intelligence depends on understanding other
people’s affective states and being able to respond appropriately. To have a
natural human- machine interaction, it is critical to enable machines with
social intelligence as well. The first step towards this capability is

Figure 1: The problem we are solving in this paper is, given affective state
of one person in a dyadic interaction, we generate the facial behaviors of the
other person. By facial behaviors, we refer to facial expressions, movements
of facial landmarks and head pose.

observing human communication dynamics and learning from the occurring
behavioral patterns [33, 32]. In this paper, we focus on modeling dyadic
interactions between two partners. The problem we are solving is given the
affective state of one partner, we are interested in generating non-verbal
facial behaviors of the other partner. Figure 1 shows the overview of our
problem. The affective states that we consider in generating facial
expressions include joy, anger, surprise, fear, contempt, disgust, sadness and
neutral. The non-verbal cues that we are generating in this work, are facial
expressions and head pose. These cues help us generate head nods, head shakes,
tilted head and various facial expressions such as smile which are behaviors
that happen in our daily face to face communications. A successful model
should learn meaningful factors of the face for generating of most appropriate
responses for the agent. We have designed our methodology based on the
following factors: 1) humans react to each others’ affective states from an
early age and adjust their behavior accordingly [42]. 2) Each affective state
is characterized by particular facial expressions and head pose[9]. 3) Each
affective state can have different intensities, such as happy vs very happy.
These factors (e.g. affective state and their intensities) effect how fast or
slow or facial expressions and head pose changes over time[9].

There has been great deal of research on facial expression analysis where the
effort is towards describing the facial expressions [40]. Facial expressions
are a result of movements of facial landmarks such as raising eyebrows,
smiling and wrinkling nose. A successful model for our problem, should learn
the movements of these points in response to different affective states.

We are using FACET SDK for extracting 8-dimensional affective state of one
partner in the interaction. We then use it as a condition in generation of
non-verbal facial behaviors of the other person. Our generation process relies
on a conditional generative adversarial networks [29], where the conditioning
vector is affective state of one party in the interaction. Our proposed
methodology is able to extract meaningful information about the face and head
pose and integrate the encoded information with our network to generate
expressive and temporally smooth face images for the agent.

##  2 Related work

Humans communicate using not only words but also non-verbal behaviors such as
facial expressions, body posture and head gestures. In our daily interactions,
we are constantly adjusting our non-verbal behavior based on other peoples
behavior. In collaborative activities such as interviews and negotiations,
people tend to unconsciously mimic other people’s behavior [4, 26].

Figure 2: Overview of our affect-sketch network. In first stage of our two-
stage facial expression generation, we generate face sketches conditioned on
the affective state of the interviewee and using a z vector that carries
contains semantically meaningful information about facial behaviors. This
vector is sampled from generated distributions of rigid and non-rigid face
shape parameters using our proposed methods. Figure 3: Overview of our sketch-
image network. In the second stage of our two-stage facial expression
generation, we generate face images conditioned on the generator sketch of the
interviewer which is generated in the first stage of our network.

With increasing interest in social robotics and virtual reality and their wide
applications in social situations such as automated interview systems [17],
companionship for elder care [5] and therapy for autism [7], there is a
growing need to enable computer systems to understand, interpret and respond
to people’s affective states appropriately.

As a person’s face discloses important information about their affective
state, it is very important to generate appropriate facial expressions in
interactive systems as well.

There has been a number of studies focusing on automatic generation of facial
expressions. These efforts can be categorized into three main groups: 1) Rule
based approaches where affective states are mapped into a pre-defined 2D or 3D
face model [15]. 2) Statistical approaches where face shape is modeled as
linear combination of prototypical expression basis [27]. 3) Deep belief
networks where the models learn the variation of facial expressions in
presence of various affective states and produce convincing samples [39].

While generating facial expressions using previous approaches can result in
promising results, it is important to consider other people’s affective states
while generating facial expressions in dyadic interactions. There has been
previous work in this area, [18] where researchers use conditional generative
adversarial networks for generating facial expressions in dyadic interactions.
While the proposed approach results in appropriate facial expression for one
frame, it does not consider the temporal consistency in a sequence, which
results in generating non-smooth facial expressions over time. Also, some of
the attributes such as head position and orientation that are important cues
for recognizing tilted head, head nods, head shakes are not captured by the
model.

In this paper, we address the problem of generating facial expressions in
dyadic interactions while considering temporal constraints and additional
facial behaviors such as head pose. Our proposed model extracts semantically
meaningful encodings of the face regarding movements of facial landmarks and
head pose and integrate it with the network which results in generation of
appropriate and smooth facial expressions.

##  3 Methodology

Generative adversarial networks (GANs) [13] have been widely used in the field
of computer vision and machine learning for various applications such as image
to image translation [20], face generation [12], semantic segmentation [28]
and etc. GANs are a type of generative model which learn to generate based on
generative and discriminative networks that are trained and updated at the
same time. The generative network tries to find the true distribution of the
data and generate realistic samples and the discriminator network tries to
discriminate between the samples that are generated by generated and samples
from real data. The general formulation of conditional GANs are as follows:

$\min_{G}\max_{D}\mathcal{L}(D,G)=\mathbb{E}_{x,y\sim p_{data}(x,y)}[\log
D(x,y)]+\\\ \mathbb{E}_{x\sim p_{data}(x),z\sim
p_{z}(z)}[\log(1-D(x,G(x,z)))].$ (1)

Conditional GANs are generative models that learn a mapping from random noise
vector z to output image y conditioned on auxiliary information
$x:G:\\{x,z\\}\Rightarrow y$. A conditional GAN consists of a generator G(x,
z) and a discriminator D(x, y) that compete in a two-player minimax game: the
discriminator tries to distinguish real training data from generated images,
and the generator tries to fail the discriminator. That is, D and G play the
following game on V (D, G). As you can see the goal of these networks is to
increase the probability of samples generated by generative networks to
resemble the real data so that the discriminator network fail to differentiate
between the generated samples and the real data.

Conditional generative adversarial networks (CGANs) are a type of GAN that
generate samples considering a conditioning factor. As our problem is
generation of facial expressions conditioned on affective states, we found
CGANs as perfect candidate for addressing our problem. We chose to use CGANs
to generate facial expressions of one party, conditioned on the affective
state of the other partner in a dyadic interaction.The GAN formulation uses a
continuous input noise vector z, that is an input to the generator. However,
this vector is a source of randomness to the model and does not capture
semantically meaningful information. However, using a meaningful distribution
which corresponds to our objective and desired facial behaviors, can help the
model integrate meaningful information with the network and improve the
results of generation. For example, in case of generating facial behaviors,
the important factors are the variation in locations of facial landmarks such
as smiling and head pose such as tilted head.

Figure 4: This figure shows 68 landmarks of the face that we intend to
generate using our model. In the bottom row you see a visualization of non-
rigid shape parameters which correspond to movements of facial landmarks such
as opening the mouth. Our model aims to learn the distribution of these
parameters and use them in generating temporally smooth sequences.

In this paper, we present a modification to the sampling process of the latent
vector z which is an input to the CGAN network. We sample z from the
meaningful distributions of facial behaviors that are generated by two
different approaches 1) Affect-shape dictionary and 2) Conditional LSTM. This
step helps the network to learn interpretable and meaningful representation of
facial behaviors. In the next sections we will first explain the two stage
architecture of our network. Then we will explain preprocessing of the frames
by going over the intuition behind our choice of face representation. Finally,
we provide detailed description of our methods for generating distributions
for vector z which is the input to our CGAN networks.

###  3.1 Two-Stage Conditional Generative Adversarial Network

To generate facial behaviors for agent, we use a two stage architecture, both
of which are based on conditional generative adversarial networks. Figure 2
shows an overview of our first network. In the first stage we use a
conditional generative adversarial network which takes face shape parameter as
input for vector z as well as the conditioning vector which is the 8 affective
states of one partner in the interaction, and it generates sketches of faces
for the other partner in the interaction. Note that the z vector will be
sampled using one of our proposed strategies, which are explained in the
following sections. In the second stage, we input the generated sketches in
the first stage as input to GAN and generate the final facial expressions.
Figure 3 shows an overview of our second network.

Figure 5: Overview of our affect-shape approach for generating clusters of
face shape parameters. We first extract shape descriptors for all faces in the
dataset. Then we cluster them into 8 affect classes corresponding to Joy,
Anger, Surprise, Fear, Contempt, Disgust, Sadness and Neutral. Then, we do a
further inter class clustering for each affect class. The purpose of the
second clustering is to find the inter affect clusters, such as very joyful,
joyful or little joyful.These clusters form the distributions from which the z
vector is sampled in our generative models.  Figure 6: Overview of our
conditional LSTM approach for generating facial shape parameters. In this
approach we consider the affective state of the interviewee and past frames of
the interviewer’s facial expressions as conditioning vector and previous
history. We then learn to generate the future frames of interviewer’s facial
expressions using a conditional LSTM. The output result of this generation is
the input for z vector in our generative models.

###  3.2 Data Preprocessing and Face Representation

Facial landmarks are the salient points on face located at the corners, tips
or midpoints of facial components such as eyes, nose, mouth, etc [41].
Movements of these points form various facial expressions that can convey
different affective states. For example, widening the eyes can communicate
surprise and a smile can be an indication of happiness. Since the locations of
facial landmarks alone are not particularly meaningful in communicating affect
but rather their movements and change over time which plays a rule in affect
recognition [38], we decided to measure their movements and change over time.
To do so, we are projecting a 3D point distribution model on each image. This
model has been trained on in the wild data [25]. The following equation is
used to place a single feature point of the 3D PDM in a given input image:

$\bar{x}_{i}=s.R_{2D}.(\bar{X}_{i}+\phi_{i}q)+t$

In this equation, s shows scale, R shows the head rotation and t is the head
translation which are known as rigid- shape parameters.
$\bar{X}_{i}=[\bar{x}_{i},\bar{y}_{i},\bar{z}_{i}]$ is the mean value of $i$
$th$ feature , $\phi_{i}$ is principle component matrix and q is a matrix
controlling the non-rigid shape parameters, which correspond to deviations of
landmarks locations from an average (neutral) face.
As described, shape parameters of the face carry rich information about head
pose and facial expressions. Hence, we use them as our face representations.
Figure 4 shows 68 facial landmarks that we have considered in this work. We
have also included a visualization of non-rigid shape parameters at the bottom
row for better understanding of these parameters.

###  3.3 Affect-Shape Dictionary

Different facial expressions are indicative of different affective states
[23]. Depending on the affective state and its intensity, the temporal
dynamics and changes in landmarks can happen in various facial expressions
such as a smile vs a frown and they can happen slow or fast depending on how
intense is the affect [8]. Based on this knowledge, we propose the following
strategies to learn meaningful distributions of face shape parameters. Figure
5 shows an overview of our affect-shape dictionary generation.

####  3.3.1 Affect Clustering

While in state of fear landmarks can change very sudden, in state of sadness
or neutral the changes in locations of landmarks will be smaller and slower.
In order to account for this point, we first cluster the calculated shape
parameters for all input frames into 8 clusters, using k-means clustering [14]
where clusters denote the following affective states: Joy, Anger, Surprise,
Fear, Contempt, Disgust, Sadness and Neutral. This stage gives us
distributions of possible facial expressions for each affective state.

####  3.3.2 Inter Affect Clustering

The output of the first clustering step will give us groupings of shape
parameters in 8 clusters corresponding to each category of affective states.
Each affective state can have various arousal levels (a.k.a intensity)[24]. We
did a second level of clustering of shape parameters inside each affect
cluster. To do so, we used a hierarchical agglomerative clustering approach
[11], where we put the constraint that each cluster should contain at least
100 videos. The result of this clustering gives us 3 to 9 sub clusters
corresponding to different arousal levels of each affective state. Having
information about the possible facial expressions that can happen in a
particular affect category and what has been expressed in previous frames by
the agent, we can sample z from an appropriate distribution which results in
an expressive and temporally smooth sequence of facial expressions. To account
for the temporal constraints, for generating $nth$ frame, we pick z from the
appropriate distribution based on the affective state which is the closest to
the previously chosen z in terms of euclidean distance.

###  3.4 Conditional LSTM

Long Short Term Memory networks (LSTM) have been known for their ability in
learning long-term dependencies [16]. We are interested in learning the
dynamics of facial expression, hence LSTMs are an appropriate candidate for
our problem. As the problem we are solving is a conditional problem, we will
use a conditional LSTM (C-LSTM) [2]. The input to CLSTM will be the
concatenated vector of affective state of the interviewer and the shape
parameters of the interviewee for $n$ previous frames and the generated output
will be the future shape parameters of the interviewee. We fix $n$ to be 100
frames in our experiments. We use conditional LSTMS in two different settings
which are described in the following subsections. Figure 6 shows an overview
of our C-LSTM approach.

Overlapping C-LSTM: In overlapping conditional LSTMS, we consider information
across 100 frames as input of the model and the output of the model is
generation of shape parameters for one future frame. In each step, the newly
generated frame is added to the history and the information from the first
input frame is removed instead ( in first in, first out manner) , to predict
futures frames.

Non-overlapping C-LSTM: In non-overlapping C-LSTMs the input to the models is
information across 100 frames and the output is generation of shape parameters
for future 100 frames, with no overlap between the frames.

Figure 7: Examples of generated sketches using both of our approaches. The
first sequence corresponds to the result of affect-shape dictionary approach
and the second row is generated using the conditional LSTM approach. Overall,
our first approach generates smoother sequences.

##  4 Experiments

To assess the effectiveness of our proposed approaches in generating
appropriate and temporally smooth facial expressions, we performed a number of
experiments using the proposed frameworks approaches. Our experiments show the
comparison between our models in terms of result and their pros and cons.

###  4.1 Dataset

The dataset used in this paper is 31 pair of dyadic interactions (videos)
which are interviews for undergraduate admissions process. The purpose of
interviews is to assess English speaking ability of the prospective college.
There are 16 male and 15 female candidates and each candidate is interviewed
by the same interviewer (Caucasian female) who followed a predetermined set of
academic and nonacademic questions designed to encourage open conversation.
The interviews were conducted using Skype video conferencing so the
participants could see and hear each other and the video data from each dyadic
interaction was captured. The duration of interviews varies from 8 to 37
minutes and a total of 24 hours of video data. We have used 70,000 short video
clips for training data and 7000 videos for evaluation.

###  4.2 Affect Recognition Framework

For estimating the affective state of the interviewee, we have used Emotient’s
Facet SDK [1] to process the frames and estimate 8- dimensional affect
descriptor vectors, representing the likelihood of the following classes: joy,
anger, surprise, fear, contempt, disgust, sadness and neutral.

###  4.3 Implementation Details

To create our training set we randomly sampled 70,000 video clips of 100
frames each (∼3.3 seconds) from the interviewee videos and extracted their
affect descriptor using Facet SDK. These affect descriptors are then combined
into a single vector using the approach proposed by Huang et al. [18]. For
each interviewee video clip a single frame from the corresponding interviewer
video is also sampled for training. Our approach is to train the model to
generate a single frame of the interviewer conditioned on facial expression
descriptors from the preceding 100 frames of the interviewee. All face images
are aligned and landmarks are estimated by the method proposed by kazem et al.
[22] to generate ground truth face sketches. For creating our test data, we
randomly sampled 7000 interviewee video clips of 400 frames each (no overlap
with the training set) and used these as input to our two model to generate
7000 facial expression video clips.

For training the first stage of our network (affect to sketch), we first
estimate the landmarks of the interviewer using the proposed method by Kazemi
et al. [22] which gives us 68 facial landmarks (See Figure 4). We then connect
these points by linear lines of one pixel width to generate the sketch images.
The generated sketches are used in a pair with the corresponding image of the
interviewer for training the second network. We masked all training image
pairs by an oval mask after roughly aligned all faces, to reduce the effect of
appearance and lightening variation of the the face in generating our videos.
In the generator, a sketch image is passed through an encoder-decoder network
[20] each containing 8 layers of down sampling and up sampling, in order to
generate the final image. We have adopted the idea proposed by Ronneberger et
al. [37] and have connected feature maps at layer i and layer n-i where n is
total number of layers in the network. In this network receptive fields after
convolution are concatenated with the receptive fields in up-convolving
process, which allows network to use original features in addition to features
after up-convolution and results in overall better performance than a network
that has access to only features after up-sampling.

Figure 8: Examples of generated face images using the second stage of our
architecture. The first sequence corresponds to a single step generation
without considering the sketches and generating face images directly from the
affect vector and noise. The second row shows the results related to
generation of faces using our proposed two stage architecture that we have
used in our experiments. Note that the quality of images in the second row is
much better than the first row.

We adopted the architecture proposed by Radford et. al [35] for our generation
framework and deep convolutional structure for generator and discriminator. We
used modules of the form convolution-BatchNorm-ReLu [19] to stabilize
optimization. In the training phase, we used mini-batch SGD and applied the
Adam solver. To avoid the fast convergence of discriminators, generators were
updated twice for each discriminator update, which differs from original
setting [35] in that the discriminator and generator update alternately.

In order to generate temporally smooth sequences, for each new frame, z is
sampled from a meaningful distribution and based on the previous frames. Note
that the appropriateness of z for a frame is decided based on the possible
variation between two adjacent frames in distribution of face shape parameters
training data which correspond to the movements of facial landmarks and head
pose.

###  4.4 Results and Discussions

We generated sequences of non-verbal facial behaviors for an agent, based on
affect-shape strategy and conditional LSTM. Figure 7 shows the output sketches
generated by these two models. The first row corresponds to the affect-shape
dictionary approach and the second row shows the results based on Conditional
LSTM. We observed that the results from dictionary based is more smooth. In
results of conditional LSTM there seems to be grouping of smooth sequences
followed by sudden jumps between the sequences (In Figure 7 see the transition
between last and one before last frame in the second row). We think that this
can happen because of number of frames we have considered as history, as in
LSTM frameworks all of the history gets summarized and in unrolling step the
time steps are not captured accurately and the generation is more based on an
average of all frames. We compare the two proposed strategies in Table 1. The
advantage of the dictionary based approach is that it does not require
previous history of the face and it can run in real time. However, since z is
randomly sampled from the possible distribution space, it generates a
different sequence each time and it is hard to evaluate this approach.

In table 2 we show an evaluation of conditional LSTM approach for generating
the facial expressions. We have generated the facial expressions using C-LSTM
by considering 100 frames as history and 1) generating the 101 st frame
(complete overlap). 2) generating the next 100 frames (No overlap). As this
method, tries to generate the closest sequence to ground truth, we have
calculated mean squared error as an evaluation metric.

As you can see, C-LSTM with 100 frame overlap has a smaller error value
compared to the C-LSTM with no overlap. Intuitively, it is harder to predict
future with no overlap than having overlap.

Properties Affect-Shape Dictionary Conditional LSTM History No need for
history. Needs history. Diversity Generates diverse sequence. Generates one
particular sequence. Evaluation Hard to evaluate. Easier to evaluate. Speed
Real time Not real time

Table 1: Comparison between Affect-shape dictionary and conditional LSTM
approaches in generating face shape parameters.

Method C-LSTM w overlap C- LSTM w/o overlap MSE 0.101 0.183

Table 2: Comparison between performance of conditional LSTMs with and without
overlap in history.

We have also generated the final face images using our CGAN network. Figure 8
shows the generated images of the interviewer which is the final result of the
our second network, sketch-image GAN. The top row shows the results of such
images if we only had one network and directly went from affect vector and z
into face images. The second row shows the results of a two step network where
we first generate face sketches and then face images. Note that quality of the
images in second row is significantly better than the first row. Also, as our
final goal is to transfer these expressions to an avatar’s face, our mid-level
results can be used to transfer the expressions to various virtual characters.

##  5 Conclusions and Future Work

Human beings react to each other’s affective state and adjust their behaviors
accordingly. In this paper we have proposed a method for generating facial
behaviors in a dyadic interaction. Our model learns semantically meaningful
facial behaviors such as head pose and movements of landmarks and generates
appropriate and temporally smooth sequences of facial expressions.

In our future work, we are interested in transferring the generated facial
expressions to an avatar’s face and generating sequences of expressive avatars
reacting to the interviewee’s affective state using the proposed approaches in
this work. We then will run a user study using pairs of interviewee and
generated avatar’s videos to study the most appropriate responses for each
video and will use the result in building our final models of emotionally
intelligent agents.

## References

  * [1] imotions inc. manual of emotient’s Facet SDK.  2013\.
  * [2] I. Augenstein, T. Rocktäschel, A. Vlachos, and K. Bontcheva.  Stance detection with bidirectional conditional encoding.  arXiv preprint arXiv:1606.05464, 2016.
  * [3] R. Bar-On, D. Tranel, N. L. Denburg, and A. Bechara.  Emotional and social intelligence.  Social neuroscience: key readings, 223, 2004.
  * [4] S. Bilakhia, S. Petridis, and M. Pantic.  Audiovisual detection of behavioural mimicry.  In Affective Computing and Intelligent Interaction (ACII), 2013 Humaine Association Conference on, pages 123–128. IEEE, 2013.
  * [5] J. Broekens, M. Heerink, H. Rosendal, et al.  Assistive social robots in elderly care: a review.  Gerontechnology, 8(2):94–103, 2009.
  * [6] K. Dautenhahn, M. Walters, S. Woods, K. L. Koay, C. L. Nehaniv, A. Sisbot, R. Alami, and T. Siméon.  How may i serve you?: a robot companion approaching a seated person in a helping context.  In Proceedings of the 1st ACM SIGCHI/SIGART conference on Human-robot interaction, pages 172–179. ACM, 2006.
  * [7] K. Dautenhahn and I. Werry.  Towards interactive robots in autism therapy: Background, motivation and challenges.  Pragmatics & Cognition, 12(1):1–35, 2004.
  * [8] J. Decety, L. Skelly, K. J. Yoder, and K. A. Kiehl.  Neural processing of dynamic emotional facial expressions in psychopaths.  Social neuroscience, 9(1):36–49, 2014.
  * [9] P. Ekman.  An argument for basic emotions.  Cognition & emotion, 6(3-4):169–200, 1992.
  * [10] T. Fong, I. Nourbakhsh, and K. Dautenhahn.  A survey of socially interactive robots.  Robotics and autonomous systems, 42(3):143–166, 2003.
  * [11] P. Franti, O. Virmajoki, and V. Hautamaki.  Fast agglomerative clustering using a k-nearest neighbor graph.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(11):1875–1881, 2006.
  * [12] J. Gauthier.  Conditional generative adversarial nets for convolutional face generation.  Class Project for Stanford CS231N: Convolutional Neural Networks for Visual Recognition, Winter semester, 2014(5):2, 2014.
  * [13] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio.  Generative adversarial nets.  In Advances in neural information processing systems, pages 2672–2680, 2014.
  * [14] J. A. Hartigan and M. A. Wong.  Algorithm as 136: A k-means clustering algorithm.  Journal of the Royal Statistical Society. Series C (Applied Statistics), 28(1):100–108, 1979.
  * [15] D. Heylen, M. Poel, A. Nijholt, et al.  Generation of facial expressions from emotion using a fuzzy rule based system.  In Australian Joint Conference on Artificial Intelligence, pages 83–94. Springer, 2001.
  * [16] S. Hochreiter and J. Schmidhuber.  Lstm can solve hard long time lag problems.  In Advances in neural information processing systems, pages 473–479, 1997.
  * [17] M. E. Hoque, M. Courgeon, J.-C. Martin, B. Mutlu, and R. W. Picard.  Mach: My automated conversation coach.  In Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing, pages 697–706. ACM, 2013.
  * [18] Y. Huang and S. M. Khan.  Dyadgan: Generating facial expressions in dyadic interactions.  In Computer Vision and Pattern Recognition Workshops (CVPRW), 2017 IEEE Conference on, pages 2259–2266. IEEE, 2017.
  * [19] S. Ioffe and C. Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In International Conference on Machine Learning, pages 448–456, 2015.
  * [20] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros.  Image-to-image translation with conditional adversarial networks.  arXiv preprint arXiv:1611.07004, 2016.
  * [21] T. Kanda, T. Hirano, D. Eaton, and H. Ishiguro.  Interactive robots as social partners and peer tutors for children: A field trial.  Human-computer interaction, 19(1):61–84, 2004.
  * [22] V. Kazemi and J. Sullivan.  One millisecond face alignment with an ensemble of regression trees.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1867–1874, 2014.
  * [23] D. Keltner, P. Ekman, G. C. Gonzaga, and J. Beer.  Facial expression of emotion.  2003\.
  * [24] E. A. Kensinger and S. Corkin.  Two routes to emotional memory: Distinct neural processes for valence and arousal.  Proceedings of the National Academy of Sciences of the United States of America, 101(9):3310–3315, 2004.
  * [25] M. Koestinger, P. Wohlhart, P. M. Roth, and H. Bischof.  Annotated facial landmarks in the wild: A large-scale, real-world database for facial landmark localization.  In Computer Vision Workshops (ICCV Workshops), 2011 IEEE International Conference on, pages 2144–2151. IEEE, 2011.
  * [26] J. L. Lakin, V. E. Jefferis, C. M. Cheng, and T. L. Chartrand.  The chameleon effect as social glue: Evidence for the evolutionary significance of nonconscious mimicry.  Journal of nonverbal behavior, 27(3):145–162, 2003.
  * [27] C. L. Lisetti and D. J. Schiano.  Automatic facial expression interpretation: Where human-computer interaction, artificial intelligence and cognitive science intersect.  Pragmatics & cognition, 8(1):185–235, 2000.
  * [28] P. Luc, C. Couprie, S. Chintala, and J. Verbeek.  Semantic segmentation using adversarial networks.  arXiv preprint arXiv:1611.08408, 2016.
  * [29] M. Mirza and S. Osindero.  Conditional generative adversarial nets.  arXiv preprint arXiv:1411.1784, 2014.
  * [30] B. Nojavanasghari, T. Baltrušaitis, C. E. Hughes, and L.-P. Morency.  Emoreact: a multimodal approach and dataset for recognizing emotional responses in children.  In Proceedings of the 18th ACM International Conference on Multimodal Interaction, pages 137–144. ACM, 2016.
  * [31] B. Nojavanasghari, T. Baltrušaitis, C. E. Hughes, and L.-P. Morency.  The future belongs to the curious: Towards automatic understanding and recognition of curiosity in children.  In Workshop on Child Computer Interaction, pages 16–22, 2016.
  * [32] B. Nojavanasghari, D. Gopinath, J. Koushik, T. Baltrušaitis, and L.-P. Morency.  Deep multimodal fusion for persuasiveness prediction.  In Proceedings of the 18th ACM International Conference on Multimodal Interaction, pages 284–288. ACM, 2016.
  * [33] B. Nojavanasghari, C. Hughes, T. Baltrusaitis, L.-p. Morency, et al.  Hand2face: Automatic synthesis and recognition of hand over face occlusions.  arXiv preprint arXiv:1708.00370, 2017.
  * [34] B. Nojavanasghari, C. E. Hughes, and L.-P. Morency.  Exceptionally social: Design of an avatar-mediated interactive system for promoting social skills in children with autism.  In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems, pages 1932–1939. ACM, 2017.
  * [35] A. Radford, L. Metz, and S. Chintala.  Unsupervised representation learning with deep convolutional generative adversarial networks.  arXiv preprint arXiv:1511.06434, 2015.
  * [36] J. Rickel.  Intelligent virtual agents for education and training: Opportunities and challenges.  In Intelligent virtual agents, pages 15–22. Springer, 2001.
  * [37] O. Ronneberger, P. Fischer, and T. Brox.  U-net: Convolutional networks for biomedical image segmentation.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 234–241. Springer, 2015.
  * [38] T. Senechal, V. Rapp, and L. Prevost.  Facial feature tracking for emotional dynamic analysis.  Advances Concepts for Intelligent Vision Systems, pages 495–506, 2011.
  * [39] J. M. Susskind, G. E. Hinton, J. R. Movellan, and A. K. Anderson.  Generating facial expressions with deep belief nets.  In Affective Computing. InTech, 2008.
  * [40] Y. Tian, T. Kanade, and J. F. Cohn.  Facial expression recognition.  In Handbook of face recognition, pages 487–519. Springer, 2011\.
  * [41] Y. Tie and L. Guan.  Automatic landmark point detection and tracking for human facial expressions.  EURASIP Journal on Image and Video Processing, 2013(1):8, 2013.
  * [42] E. Tronick.  Still face experiment.  Retrieved October, 22:2016, 2009.
  * [43] B. Ulicny and D. Thalmann.  Crowd simulation for interactive virtual environments and vr training systems.  Computer Animation and Simulation 2001, pages 163–170, 2001.