# Benchmarking Cross-Domain Audio-Visual Deception Detection

Xiaobao Guo, Zitong Yu, , Nithish Muthuchamy Selvaraj, Bingquan Shen,
Adams Wai-Kin Kong, and Alex C. Kot Manuscript received July, 2023.
Corresponding author: Zitong Yu.X. Guo is with ROSE Lab, Interdisciplinary
Graduate Programme, and also with SCSE, Nanyang Technological University,
Singapore. E-mail<EMAIL_ADDRESS>Z. Yu, N. M. Selvaraj, and A. Kot are
with ROSE Lab, Nanyang Technological University, Singapore. E-mail:
<EMAIL_ADDRESS>{ms.nithish<EMAIL_ADDRESS>B. Shen is with DSO
National Laboratories, Singapore. E-mail<EMAIL_ADDRESS>A. W. -K. Kong
is with SCSE, Nanyang Technological University, Singapore. E-mail:
<EMAIL_ADDRESS>
###### Abstract

Automated deception detection is crucial for assisting humans in accurately
assessing truthfulness and identifying deceptive behavior. Conventional
contact-based techniques, like polygraph devices, rely on physiological
signals to determine the authenticity of an individual’s statements.
Nevertheless, recent developments in automated deception detection have
demonstrated that multimodal features derived from both audio and video
modalities may outperform human observers on publicly available datasets.
Despite these positive findings, the generalizability of existing audio-visual
deception detection approaches across different scenarios remains largely
unexplored. To close this gap, we present the first cross-domain audio-visual
deception detection benchmark, that enables us to assess how well these
methods generalize for use in real-world scenarios. We used widely adopted
audio and visual features and different architectures for benchmarking,
comparing single-to-single and multi-to-single domain generalization
performance. To further exploit the impacts using data from multiple source
domains for training, we investigate three types of domain sampling
strategies, including domain-simultaneous, domain-alternating, and domain-by-
domain for multi-to-single domain generalization evaluation. Furthermore, we
proposed the Attention-Mixer fusion method to improve performance, and we
believe that this new cross-domain benchmark will facilitate future research
in audio-visual deception detection. Protocols and source code are available
at https://github.com/Redaimao/cross_domain_DD.

###### Index Terms:

audio-visual, multimodal deception detection, cross-domain, generalization.

##  1 Introduction

Figure 1:  Typical samples from different publicly deception detection
datasets: Real Life Trials [1], Bag of Lies [2], MU3D [3], and Box of Lies
[4]. The samples in each row are from different datasets while those in each
column are with different modalities (visual vs. audio) and ground truth
labels (i.e., truthful vs. deceptive). It can be seen that serious domain
shifts (e.g., resolution/illumination/pose in visual faces and
pitch/loudness/noise in audio) occur among these datasets.

Audio-visual deception detection involves utilizing AI techniques and
algorithms to automatically detect deceptive behavior in speech and facial
movements [5, 6, 7, 8]. Deception detection has a significant impact on
various real-world applications such as law enforcement [9], healthcare [10],
and business [11]. It has the potential to prevent fraud, improve security
measures, and enhance trust and confidence. A reliable deception detection
tool can support more accurate decision-makings.

Traditional deception detection is often a contact-based method. It assesses
whether someone is telling the truth or not by monitoring physiological
responses like skin conductance and heart rate [12]. Experts’ Behavioral
observation and analysis are another technique that evaluates changes in a
person’s body language, speech patterns, and eye movements [13, 14]. However,
such an assessment can be time-consuming and require significant expertise to
perform accurately.

Recently, the development of automated deception detection systems using AI
and machine learning techniques has gained significant attention as the
existing methods have limitations in terms of reliability, accuracy, and
scalability. Various multimodal datasets have been introduced, including real-
life trials from court scenes [1], lab-based setups [2, 3], and game show
scenarios [4]. These datasets provide a wide variety of deceptive samples from
different domains, enabling researchers to examine the effectiveness of AI
models on deception detection. Based on these datasets, progress has been made
in deception detection techniques within specific domains [7, 6, 15]. Recent
studies have utilized rich visual and audio features [16, 17, 18], such as Mel
Spectrogram, emotional states, and facial action units, to enhance the
performance of deception detection tasks.

However, there remains a substantial research gap that needs to be addressed.
Specifically, fewer studies have explored the cross-domain issue, despite the
presence of significant domain shifts in public deception detection datasets.
As shown in Figure 1, domain shifts are observed in both audio and visual
modalities from publicly available datasets. The generalizability of the
models is critical for practical applications. Therefore, such domain shifts
need to be investigated in order to develop deception detection models that
can be generalized across different contexts. Additionally, effective methods
must be proposed to alleviate the domain shift issue by fusing both audio and
visual features in a meaningful way. Addressing these issues can benefit
automated deception detection systems in improving generalizability in real-
world applications.

To address the issue of cross-domain deception detection, we introduce a new
benchmark that evaluates the generalization capacity of AI models using audio
and visual features over publicly available datasets. Our benchmarking
approach utilizes widely adopted audio and visual features, and we compare the
single-to-single domain performance and multi-to-single domain generalization
using different architectures. Specifically, for the multi-to-single setting,
three domain sampling strategies, _i.e.,_ domain simultaneous, domain
alternating, and domain-by-domain, are implemented to conduct cross-domain
testing. To further enhance performance, we propose an Attention-Mixer fusion
method based on MLP-Mixer [19]. This benchmarking framework serves as an
important tool for evaluating the effectiveness of audio-visual deception
detection models in diverse contexts, which will help improve the capabilities
of automated deception detection systems in real-world settings. Additionally,
we hope our work will inspire further research on multimodal models that
address domain shift issues. In summary, our main contributions include:

  * •

Introducing a new benchmark for evaluating the generalization capacity of AI
models using audio and visual features across different domains.

  * •

Comparing the single-to-single domain and multi-to-single domain
generalization using different architectures.

  * •

Providing three domain sampling strategies _i.e.,_ domain simultaneous, domain
alternating, and domain-by-domain, to conduct multi-to-single cross-domain
testing.

  * •

Proposing the Attention-Mixer fusion method to enhance performance.

In the rest of the paper, Sec. 2 provides a review of related psychological
studies on cues to deception and multimodal deception detection works. Sec. 3
introduces our benchmarking approach and fusion method. Sec. 4 provides the
cross-domain benchmark results and fusion results. Finally, conclusions and
future works are given in Sec. 5.

##  2 Related work

###  2.1 Cues to Deception

The research on using behavioral cues for deception has gradually become
active over the past few decades. Psychological researchers have published a
large number of works on the analysis of cues to deception [20, 21, 22]. Among
the studied behavioral cues, verbal and nonverbal cues were preferred as
humans may behave differently between lying and telling the truth. DePaulo _et
al._ [23] studied and reported experimental results on 158 cues to deception.
They revealed that, in general, people who tell lies are less forthcoming and
less convincing than those who tell the truth. Liars usually talk about fewer
details and make fewer spontaneous corrections. They also sound less involved
but more vocally tense. Through the study, the researchers statistically found
that liars often press their lips, repeat words, raise their chins, and show
less genuine smiles. The results show that some behavioral cues do potentially
appear in deception and are even more pronounced when liars are more motivated
to cheat.

Levine _et al._ [24] reviewed the status quo and provided a new perspective on
the theories of deception. They pointed out that lying usually happens when
problematic information is involved. It is critical to understand the verbal
content in the context. Vrij _et al._ [25] realized that interviewers play a
vital role in eliciting and enhancing cues to deceit. The authors proposed the
“interviewing to detect deception” technique to open a new path in the
deception detection research field. They argued that different psychological
states can be exploited by adopting appropriate interview techniques of liars
and truth-tellers. Hirschberg _et al._ [26] proposed a method to distinguish
deceptive from non-deceptive speeches by a large corpus. They also conducted
experiments using acoustic, lexical, and speaker-dependent features, which
showed improved performance by combining multiple feature sets. Warren _et
al._ [27] conducted experiments to investigate the relationship between
affective facial expressions to deception. The results indicated that leaked
emotions with the incongruous intended message can provide useful cues to
deception, which supported the nonverbal leakage theory [28, 29].

Figure 2: Main network and method. (a) Model architecture. Visual modality
includes face and behavior inputs. Audio modality includes Mel Spectrogram
input. The features obtained by the respective encoders. The fusion methods
include score fusion and feature fusion. (b) Domain sampling strategies.
Domain-simultaneous: each batch consists of samples from multiple sources.
Domain-alternating: each batch is alternatively sampled from multiple sources.
Domain-by-domain: the batches are sampled from one source and then from
another.

###  2.2 Multimodal Deception Detection

Recent works for deception detection usually use verbal and non-verbal
features and propose effective fusion methods [30, 15, 16, 17]. For example,
some works utilized facial features from RGB images to perform deception
detection [15, 17, 5]. To capture facial movements, facial action units (AUs)
were utilized. Other features, such as facial expression, were also adopted
[6, 30]. Besides visual features, many works incorporated audio features to
boost performance [7, 6, 15]. For example, Wu _et al._ [6] used MFCC (Mel-
frequency Cepstral Coefficients) features and Karimi _et al._ [15] used raw
audio. Most of the recent works mentioned above have considered multimodal
fusion approaches that extract visual, audio, and text information to boost
performance. In addtion to visual, audio and text, Karnati _et al._ [16]
exploited physiological signals, _i.e.,_ EEG representations for deception
detection.

To better fuse the multimodal features, different fusion methods were
proposed, which can be broadly categorized into feature-level fusion and
decision-level fusion. Specifically, feature-level fusion focused on producing
better multimodal embeddings and used the linear layers to extract crossmodal
dynamics [15, 17, 5, 6, 7]. Rather than that, decision-level fusion aimed to
fuse multimodal dynamics at a late stage, to reduce computational complexity
and learn good marginal representations [16, 7].

However, previous works on multimodal deception detection did not consider
cross-domain issues that occur from one domain to another, which is the focus
of this work.

##  3 Methodology

The mainstream architecture for audio-visual deception detection usually
includes encoders for unimodal feature extraction and/or a fusion module. We
follow the widely adopted architecture to build the benchmark on cross-domain
audio-visual deception detection in this work. As shown in Fig. 2, audio and
visual features are extracted from audio and visual encoders. The fusion
module is performed based on audio and visual features. The fused feature is
input to the classifier for classification. We build the benchmark for cross-
domain generalization performance based on such network architecture with
different encoders. We conducted single-to-single and multi-to-single
evaluations where three domain sampling strategies included, _i.e.,_ domain-
simultaneous, domain-alternating, and domain-by-domain.

###  3.1 Audio and Visual Feature Learning

To establish a benchmark for cross-domain audio-visual deception detection, we
utilize widely adopted audio and visual features along with their respective
encoders. Our approach treats audio and visual features as equally important,
extracting different types of features simultaneously. As depicted in Fig. 1,
this network structure offers several advantages: (1) flexibility in network
selection: different audio or visual encoders can be effortlessly incorporated
and compared in a fair manner, (2) adaptability: the addition or removal of
specific modules and/or losses is straightforward. For instance, a fusion
module can be inserted before classifiers, and (3) easy performance
benchmarking: the system facilitates evaluating performance in various
settings, such as score-level fusion and feature-level fusion. In this work,
we focus on audio and visual modalities for deception detection. In
particular, two kinds of visual features are extracted, _i.e.,_ face features
from RGB face images and behavior features consist of AUs, affects, etc.

As shown in Fig. 1, given a detected RGB face image as input $X_{f}$, the deep
features $F_{f}$ could be extracted via face encoder networks
$\mathcal{E}_{f}$ (e.g., ResNet18 [31]). Similarly, behavior inputs such as
the AU and/or affect features $X_{b}$ are encoded by OpenFace [32] or affect
model (e.g., EmotionNet [33]) $\mathcal{E}_{b}$ to output behavior features
$F_{b}$. Note that we regard both face frames and behavior features as the
visual modality but differentiate them in this work as they have different
types of information and representations. Given audio input $X_{a}$ (either
Mel Spectrogram [34] or waveforms), audio features $F_{a}$ are extracted
through audio encoder $\mathcal{E}_{a}$. The corresponding classification
heads for face frames ($\mathcal{H}_{f}$), behavior features
($\mathcal{H}_{b}$), and audio features ($\mathcal{H}_{a}$) output the
prediction logits $\hat{Y}_{f}$, $\hat{Y}_{b}$, and $\hat{Y}_{a}$,
respectively. The fusion head $\mathcal{G}$ takes $F_{f}$, $F_{b}$, and
$F_{a}$ as input. $\mathcal{G}$ is determined by the actual fusion method,
e.g., liner layer, transformer layers, MLP, etc. The output logit of
$\mathcal{G}$ is denoted by $\hat{Y}_{g}$. Therefore, the audio and visual
learning process can be denoted as follows:

$\small\begin{split}F_{f}&=\mathcal{E}_{f}(X_{f}),\hat{Y}_{f}=\mathcal{H}_{f}(F_{f}),\\\
F_{b}&=\mathcal{E}_{b}(X_{b}),\hat{Y}_{b}=\mathcal{H}_{b}(F_{b}),\\\
F_{a}&=\mathcal{E}_{a}(X_{a}),\hat{Y}_{a}=\mathcal{H}_{a}(F_{a}),\\\
\hat{Y}_{g}&=\mathcal{G}(F_{f},F_{b},F_{a}).\end{split}$ (1)

Loss Function. For deception detection ground truth $Y$, where $Y=0$ for
truthful and $Y=1$ for deception, the binary cross-entropy loss (BCE) is
adopted. The loss for each sample with a certain modality or fused prediction
can be denoted as

$\mathcal{L}_{m}=-(Ylog(\hat{Y}_{m})+(1-Y)log(1-\hat{Y}_{m})),$ (2)

where $m\in\\{f,b,a,g\\}$, $\hat{Y}_{m}$ is the corresponding prediction
logits. In other words, the BCE loss is calculated separately for each type of
modality and/or its fused feature depending on whether a sample has any face
frames, visual inputs, or audio inputs. The overall loss function can be
described as follows:

$\mathcal{L}=\frac{1}{N}\sum_{i=i}^{N}\left(\sum_{m={f,b,a}}\mathcal{L}_{m,i}+\lambda\mathcal{L}_{g,i}\right),$
(3)

where $N$ is the number of data samples and $\lambda$ is a trade-off parameter
between modality loss and fusion loss. $\lambda$ is set to 0.5 in our
experiments.

###  3.2 Cross-domain Generalization

We benchmark the cross-domain generalization on the deception detection task.
First, we introduce the notations and definitions in this section. A domain is
composed of data that are sampled from a distribution (dataset), which can be
denoted as $\mathcal{S}=\\{(X;Y)_{i}\\}_{i=1}^{N}\sim P_{S}$, where
$X=(X_{f},X_{b},X_{a})$, $X_{f},X_{b},X_{a}$ represent samples of face frames,
behavior, and audio modalities, respectively. $Y$ denotes the label, and
$P_{S}$ denotes the joint distribution of the input samples and the output
label. In this paper, for simplicity, we follow similar definitions in [35,
36] to treat each dataset as an individual domain due to their obvious
distribution gaps, but more fine-grained intra-domain factors would be
explored in future work. For domain generalization, $M$ source domains
(training datasets) are given, _i.e.,_
$\mathcal{S}_{train}=\\{S_{j}|j=1,\cdots,M\\}$, where
$\mathcal{S}_{j}=\\{(X;Y)_{i}\\}_{i=1}^{N_{j}}\sim P_{S_{j}}$ denotes the
$j$-th domain, and $P_{S_{i}}\neq P_{S_{j}}$ for $1\leq i,j\leq M$. $N_{j}$ is
the number of total samples in $S_{j}$. The goal of domain generalization is
to learn the predictive function $h$ in $M$ source domains to achieve minimum
error on an unseen test domain $\mathcal{S}_{test}\sim P_{S_{test}}$, and
$P_{S_{test}}\neq P_{S_{j}}$ for $1\leq j\leq M$:

$min~{}\mathbb{E}_{(X;Y)\in\mathcal{S}_{test}}\left[\mathcal{L}(h(X),Y)\right],$
(4)

where $X=(X_{f},X_{b},X_{a})$, $Y$ is the label, $\mathcal{L}$ is the loss
function, and $\mathbb{E}$ is the expectation.

When $M=1$, it is a Single-to-single Cross-domain Generalization task, where
the modal is trained on one training dataset and tested on another dataset.
When $M\geqslant 2$, we propose three strategies to learn from multiple
domains for the Multi-to-single Cross-domain Generalization. Let $B$ denote
one batch of training data with a size of $N_{B}$. Given multiple training
domains $\mathcal{S}_{train}=\\{S_{j}|j=1,\cdots,M\\}$, $B$ is a set of
training data sampled from $\mathcal{S}_{train}$.

Domain-Simultaneous means to train multiple domains in parallel within each
batch of data. In domain simultaneous training, the $k-$th batch of training
data is a group of samples from different domains, _i.e.,_
$B^{k}=(b_{S_{1}}^{k},\cdots,b_{S_{M}}^{k})$, $k\in[1,\cdots K]$, where
$b_{S_{j}}^{k}$ is the batch samples from domain $S_{j}$ for $j=1,\cdots,M$,
$K$ is the number of batches during training. The total number of
$b_{S_{j}}^{k}$ is $N_{B}$. As shown in Fig. 2 (b), each training batch
contains smaller batch samples from all the source domains during training.
Models are trained to learn from different domains simultaneously by feeding
the mixed batch data.

Domain-Alternating is different from domain simultaneous strategy in terms of
batch samples. In domain-alternating, $B^{k}=b_{S_{j}}^{k}$ for
$j={k-\lfloor{(k-1)\over{M}}\rfloor\cdot M}$, where $\lfloor{\cdot}\rfloor$ is
the flooring operator. The number of $b_{S_{j}}^{k}$ is $N_{B}$. Fig. 2 (b)
shows that the consecutive batch samples come from different domains.

Domain-by-Domain aims to train the model by feeding data from source domain
data one by one. $B^{k}=b_{S_{j}}^{k}$ for
$\lceil{\sum_{i=0}^{i=j-1}N_{i}\over{N_{B}}}\rceil\leqslant
k\leqslant\lceil{\sum_{i=0}^{i=j}N_{i}\over{N_{B}}}\rceil$, $N_{0}=0$, where
$\lceil{\cdot}\rceil$ is the ceiling operator. The number of $b_{S_{j}}^{k}$
is $N_{B}$. As shown in Fig. 2, the batch data samples from one domain after
finishing sampling from its previous domains.

###  3.3 Attention-Mixer Fusion

Besides investigating cross-domain sampling strategies, inspired by [19], we
propose Attention-Mixer Fusion to enhance the performance by fusing audio-
visual modalities, where the attention mixer layer takes multimodal features
as input to produce fused features. In particular, an attention mixer layer is
composed of unimodal MLP layers, self-attention layers [37], and crossmodal
MLP layers. First, for batch size $N_{B}$, the input features from different
modalities are concatenated and projected to be a tensor
$U\in\mathbb{R}^{N_{B}\times N_{m}\times D}$ by a Liner layer, followed by
several attention mixer layers, where $N_{m}$ is the number of input
modalities. Specifically, the unimodal MLP layer, the self-attention layer,
and the crossmodal MLP layer can be respectively described as

$\small U^{*,*,i}=F_{g}^{*,*,i}+\mathbf{W_{2}}\ \sigma(\mathbf{W_{1}}\
LN(F_{g}^{*,*,i})),\ i=\left[1,D\right],$ (5) $\footnotesize
U=\left[\left(softmax\left(U{\mathbf{W_{3}}(U\mathbf{W_{4}})^{T}}\over{\sqrt{D}}\right)U\mathbf{W_{5}}\right)_{h}\right]\mathbf{W_{6}},\
h=[1,H],$ (6) $\small U^{*,j,*}=U^{*,j,*}+\mathbf{W_{8}}\
\sigma(\mathbf{W_{7}}\ LN(U^{*,j,*})),\ j=\left[1,N_{m}\right],$ (7)

where $LN(\cdot)$ denotes the Layer Normalization, $\mathbf{W}_{1-8}$ are
trainable weights, $H$ is the number of heads in multihead self-attention, and
$*$ denotes all the entries in that dimension. Several attention mixer layers
are stacked as a deep block, which is set as a hyperparameter in practice. We
set it to 6 in our experiment. Finally, $U\in\mathbb{R}^{N_{B}\times
N_{m}\times D}$ is reduced to $U\in\mathbb{R}^{N_{B}\times N_{m}\times 1}$ by
obtaining the mean value on the feature dimension. In Eq. 5, the unimodal MLP
layer is conducted along the feature dimension to learn the dynamics in each
unimodal feature. Eq. 6 shows the multi-head self-attention operation on the
tensor $U$, which further explores the attention between the unimodal
features. In Eq. 7, the crossmodal MLP layer learns the dynamics across the
modality dimension from the corresponding feature tokens.

TABLE I: The results of single-to-single cross-domain generalization accuracy
(%) on benchmark datasets, Real-life Trial (R), Bag of Lies (B1), Box of Lies
(B2), and MU3D (M).

Modality & Inputs | Method | R to B1 | R to B2 | R to M | B1 to R | B1 to B2 | B1 to M | M to R | M to B1 | M to B2 | Avg
---|---|---|---|---|---|---|---|---|---|---|---
Visual (AU) | LSTM [38] | 48.11 | - | - | 61.21 | - | - | - | - | - | -
Visual (Face frames) | ResNet18 | 52.00 | 61.39 | 51.25 | 50.93 | 57.43 | 50.62 | 57.94 | 51.69 | 57.43 | 54.52
Visual (Face frames) | ResNet18+GRU | 53.54 | 63.37 | 52.81 | 57.41 | 59.41 | 51.56 | 46.73 | 52.92 | 55.45 | 54.80
Visual (AU+Gaze) | MLP | 50.77 | 65.35 | 56.87 | 58.88 | 58.42 | 50.94 | 46.73 | 51.69 | 53.47 | 54.79
Visual (Affect) | MLP | 50.46 | 58.42 | 50.31 | 50.47 | 51.49 | 52.19 | 66.36 | 51.08 | 60.40 | 54.58
Visual (AU+Gaze+Affect) | MLP | 54.46 | 59.41 | 54.37 | 50.47 | 57.43 | 54.69 | 60.75 | 51.69 | 55.45 | 55.41
Audio (Mel spectrogram) | ResNet18 | 46.77 | 53.47 | 52.19 | 50.47 | 66.34 | 50.62 | 54.21 | 51.38 | 55.45 | 53.43
Audio (Waveform) | Wave2Vec | 51.08 | 48.51 | 50.94 | 46.73 | 58.42 | 50.00 | 63.55 | 56.31 | 56.44 | 53.55

TABLE II: The results of multi-to-single cross-domain generalization accuracy
(%) on benchmark datasets, Real-life Trial (R), Bag of Lies (B1), Box of Lies
(B2), and MU3D (M), for different generalization strategies.

Modality & Inputs | Method | R&M to B1 | R&M to B2 | R&B1 to B2 | R&B1 to M | B1&M to R | B1&M to B2 | R&B1&M to B2 | Avg
---|---|---|---|---|---|---|---|---|---
Domain-Simultaneous
Visual (Face frames) | ResNet18 | 53.85 | 49.50 | 49.50 | 50.94 | 44.86 | 60.40 | 44.55 | 50.51
Visual (Face frames) | ResNet18+GRU | 52.62 | 54.46 | 51.49 | 51.88 | 53.27 | 59.41 | 44.55 | 52.53
Visual (AU+Gaze) | MLP | 53.54 | 47.52 | 48.51 | 50.94 | 52.34 | 53.47 | 56.44 | 51.82
Visual (Affect) | MLP | 50.15 | 54.46 | 55.45 | 52.19 | 53.27 | 57.43 | 62.38 | 55.04
Visual (AU+Gaze+Affect) | MLP | 50.46 | 52.48 | 61.39 | 51.25 | 51.4 | 60.4 | 63.37 | 55.82
Audio (Mel spectrogram) | ResNet18 | 48.92 | 45.54 | 53.47 | 53.12 | 43.93 | 62.38 | 50.5 | 51.12
Audio (Waveform) | Wave2Vec | 52.92 | 55.45 | 44.55 | 51.25 | 69.16 | 42.57 | 46.53 | 51.78
Domain-Alternating
Visual (Face frames) | ResNet18 | 50.15 | 45.54 | 56.44 | 51.56 | 50.47 | 54.46 | 65.85 | 53.50
Visual (Face frames) | ResNet18+GRU | 55.38 | 52.48 | 60.40 | 50.00 | 50.47 | 60.40 | 64.62 | 56.25
Visual (AU+Gaze) | MLP | 55.45 | 47.52 | 53.47 | 51.25 | 54.21 | 57.43 | 60.40 | 54.25
Visual (Affect) | MLP | 51.08 | 56.44 | 61.39 | 52.19 | 52.34 | 58.42 | 53.47 | 55.05
Visual (AU+Gaze+Affect) | MLP | 51.38 | 58.42 | 63.37 | 50.31 | 53.27 | 52.48 | 60.40 | 55.66
Audio (Mel spectrogram) | ResNet18 | 50.15 | 60.40 | 53.47 | 50.31 | 58.88 | 51.49 | 47.52 | 53.17
Audio (Waveform) | Wave2Vec | 52.92 | 55.45 | 44.55 | 50.62 | 64.49 | 58.42 | 48.51 | 53.57
Domain-by-Domain
Visual (Face frames) | ResNet18 | 52.00 | 53.47 | 56.44 | 50.00 | 59.81 | 41.58 | 55.45 | 52.68
Visual (Face frames) | ResNet18+GRU | 54.46 | 41.58 | 66.34 | 50.62 | 51.40 | 56.44 | 60.40 | 54.46
Visual (AU+Gaze) | MLP | 51.08 | 43.56 | 55.45 | 53.75 | 57.01 | 53.47 | 54.46 | 52.68
Visual (Affect) | MLP | 55.69 | 57.43 | 57.43 | 51.56 | 52.34 | 49.50 | 61.39 | 55.05
Visual (AU+Gaze+Affect) | MLP | 50.15 | 56.44 | 58.42 | 50.00 | 57.94 | 60.40 | 63.37 | 56.67
Audio (Mel spectrogram) | ResNet18 | 52.31 | 50.50 | 58.42 | 49.38 | 53.27 | 56.44 | 59.41 | 54.24
Audio (Waveform) | Wave2Vec | 56.00 | 47.52 | 44.55 | 53.12 | 67.29 | 57.43 | 58.42 | 54.90

##  4 Experiments

In this part, extensive experiments are conducted to benchmark the cross-
domain performances on public deception detection datasets. In the following,
we sequentially describe the benchmark datasets & metrics (Sec. 4.1),
implementation details (Sec. 4.2), benchmarking results (Sec. 4.3 \- 4.4) and
fusion performances (Sec. 4.5).

###  4.1 Databases and Metrics

Datasets. We benchmarked the cross-domain generalization performance based on
four publicly available datasets. Real Life Trials [1] dataset is a popular
real-world dataset collected from public court trials, which consists of 121
videos including 61 deceptive and 60 truthful video clips. As it is a real-
world dataset, the Real Life trial dataset has more noise on both the video
and audio. We filtered out some corrupted videos and obtained 108 videos (54
truthful and 54 deceptive) with 58 subjects for our experiments. Bag of Lies
[2] is a multimodal dataset collected from well-controlled lab-based
scenarios, where video, audio, EEG, and gaze data are collected. It has 35
subjects, 163 truthful and 162 deceptive video clips. The backgrounds for the
videos are relatively clean and it is less noisy. MU3D [3] has 320 video clips
and 80 subjects that cover different races and genders. It is also a lab-based
dataset that uses the personal description paradigm to stimuli real-world
cases. Each participant tells a positive truth, a positive lie, a negative
truth, and a negative lie. Box of Lies [4] is a deception dataset collected
from an online gameshow, which has 25 videos and 26 participants (6 male and
20 female). The full video set contains 29 truthful and 36 deceptive rounds of
games. However, the quality of the original Box of Lies dataset is not
satisfactory. The visual (the face of the participant) and audio from many
clips are not matching due to the frequent changes of viewpoints. To perform a
fair comparison, we preprocessed and cleaned the Box of Lies dataset. After
preprocessing, 101 video clips were extracted for testing. Some of the typical
samples from these datasets are shown in Fig. 1.

Evaluation Metrics. In this work, we followed the widely adopted metric,
binary classification accuracy (%), for experimental evaluation. The deceptive
clips were labeled as 1 and truthful clips were labeled as 0.

###  4.2 Implementation Details

Feature Extraction. Several widely-adopted audio and visual features were
extracted by different tools. For visual features, OpenFace [32] was used to
extract 35-dimensional AUs and 8-dimensional gaze features. Face frames were
extracted and aligned by MTCNN [39], where we uniformly sampled 64 face frames
for each video clip. Affect features were extracted by Emonet [33], where the
feature included 5-class emotions, arousal, and valence. For audio features,
Mel Spectrograms were extracted by OpenSmile toolkit [34]. Raw audio waveforms
were also used in our experiments.

Protocols. Inspired by [35, 36], we treated each dataset as a domain. To
evaluate the models’ cross-domain generalization capacity and alleviate domain
information leakage, all the preprocessed data including original training and
test data from each dataset was used for either training or testing. Note that
the Box of Lies dataset was only used for testing as many samples were
filtered out due to their unsatisfactory quality. The experiments were
conducted on the single-to-single domain (e.g., R to B1 stands for training on
Real-life Trial (R) and testing on Bag of Lies (B1)) and multi-to-single
domain (e.g., R&M to B2 stands for training on Real-life Trial (R) and MU3D
(M) and testing on Box of Lies (B2)).

Model Selection. Models for audio and visual modalities were selected to fit
the data volume. For face frames, we adopted ResNet18 [31] and Gate Recurrent
Unit (GRU) [40] models for facial feature extraction and temporal modeling,
respectively. Two-layer multilayer perception (MLP) [41] models were used for
AUs, gaze, and affect feature representation. For the audio-based Mel
spectrogram, we used the ResNet18 [31] model for time-frequency feature
representation. For audio waveforms, the Wave2Vec [42] model was applied for
audio feature extraction.

Experimental Setting. Our proposed method was implemented with Pytorch. The
ImageNet pretrained models (e.g., ResNet18) for classification were trained on
the benchmark datasets using SGD optimizer with the initial learning rate
(lr), momentum, and weight decay (wd) were 1e-3, 0.9, and 5e-5, respectively.
We trained models with a maximum of 30 epochs and batchsize 32 on a single
Nvidia V100 GPU. As for the fusion models (e.g., Atten-Mixer on face frames
and Mel Spectrogram), Adam optimizer with initial lr=1e-3 and wd=5e-5 was
used. The models were trained with batchsize 16 for a maximum of 30 epochs.

TABLE III: The fusion results of single-to-single cross-domain generalization
accuracy (%).

Modality & Inputs | Fusion Postion | Fusion Method | R to B1 | R to B2 | R to M | B1 to R | B1 to B2 | B1 to M | M to R |  M to B1 | M to B2 | Avg
---|---|---|---|---|---|---|---|---|---|---|---|---
| Score-level | Average | 53.23 | 41.58 | 51.88 | 64.49 | 62.38 | 50.62 | 57.94 |  53.23 | 62.38 | 55.30
|  | Concat | 51.08 | 55.45 | 51.88 | 54.21 | 58.42 | 51.25 | 57.01 |  50.15 | 61.39 | 54.54
|  | SE-Concat | 53.85 | 60.40 | 51.25 | 55.14 | 58.42 | 50.62 | 56.07 |  51.38 | 65.35 | 55.83
|  | Cross-Atten | 55.38 | 61.39 | 52.19 | 51.40 | 60.40 | 51.25 | 55.14 |  56.31 | 60.40 | 55.98
|  | MLP-Mixer | 55.08 | 48.51 | 53.44 | 56.07 | 58.42 | 53.75 | 55.14 |  59.08 | 59.41 | 55.43
Visual (Face frames) + Visual (AU+Gaze+Affect) | Feature-level | Atten-Mixer(Ours) | 56.92 | 59.41 | 57.94 | 63.37 | 53.75 | 53.75 | 60.75 |  56.00 | 61.39 | 58.14
| Score-level | Average | 53.23 | 49.50 | 51.88 | 50.47 | 59.41 | 53.75 | 65.42 |  56.31 | 49.50 | 54.39
|  | Concat | 50.77 | 53.47 | 50.47 | 62.38 | 51.56 | 51.88 | 54.21 |  52.62 | 58.42 | 53.98
|  | SE-Concat | 50.15 | 44.55 | 51.40 | 61.39 | 53.12 | 52.50 | 65.42 |  56.31 | 66.34 | 55.69
|  | Cross-Atten | 54.46 | 51.49 | 55.14 | 58.42 | 51.25 | 52.19 | 63.55 |  56.62 | 66.34 | 55.95
|  | MLP-Mixer | 52.31 | 55.45 | 57.94 | 63.37 | 53.12 | 51.25 | 64.49 |  57.85 | 62.38 | 57.57
Visual (Face frames) + Audio (Mel spectrogram) | Feature-level | Atten-Mixer(Ours) | 57.54 | 55.45 | 56.07 | 61.39 | 50.94 | 53.12 | 67.29 |  57.85 | 64.36 | 58.22
| Score-level | Average | 49.85 | 58.42 | 54.06 | 45.79 | 60.40 | 50.62 | 49.53 |  56.00 | 63.37 | 54.23
|  | Concat | 49.54 | 53.47 | 47.66 | 61.39 | 53.44 | 51.88 | 57.01 |  50.15 | 58.42 | 53.66
|  | SE-Concat | 50.15 | 48.51 | 55.14 | 60.40 | 53.12 | 50.00 | 57.01 |  49.85 | 63.37 | 54.17
|  | Cross-Atten | 53.23 | 44.55 | 57.94 | 55.45 | 54.69 | 54.06 | 63.55 |  51.69 | 64.36 | 55.50
|  | MLP-Mixer | 49.54 | 57.43 | 50.47 | 63.37 | 54.06 | 53.12 | 59.81 |  55.08 | 69.31 | 56.91
Visual (AU+Gaze+Affect) + Audio (Mel spectrogram) | Feature-level | Atten-Mixer(Ours) | 53.52 | 54.46 | 57.94 | 61.39 | 53.12 | 51.56 | 69.16 |  58.15 | 64.36 | 58.18
| Score-level | Average | 52.00 | 58.42 | 51.88 | 53.27 | 58.42 | 50.94 | 57.01 |  53.54 | 61.39 | 55.20
|  | Concat | 53.23 | 58.42 | 55.14 | 61.39 | 52.50 | 52.19 | 56.07 |  54.15 | 60.40 | 55.94
|  | SE-Concat | 51.38 | 58.42 | 51.40 | 62.38 | 52.81 | 50.94 | 59.81 |  54.77 | 52.48 | 54.93
|  | Cross-Atten | 51.08 | 48.51 | 55.14 | 60.40 | 53.12 | 53.44 | 60.75 |  56.31 | 60.40 | 55.46
|  | MLP-Mixer | 55.69 | 46.53 | 44.86 | 63.37 | 51.56 | 50.94 | 64.49 |  56.00 | 60.40 | 54.87
Visual (Face frames) + Visual (AU+Gaze+Affect) + Audio (Mel spectrogram) | Feature-level | Atten-Mixer(Ours) | 55.08 | 60.40 | 57.01 | 64.36 | 53.44 | 51.25 | 67.29 |  56.00 | 62.38 | 58.58

###  4.3 Cross-domain Testing with Unimodal Features

In this subsection, we present the benchmark results of cross-domain testing
by investigating unimodal features to evaluate their generalization
capacities. For clarity, we use “visual (face frames)” to indicate face inputs
and “visual (AU/gaze/affect)” to indicate behavior inputs.

Single-to-Single Domain. Specifically, the models were trained on one dataset
from one domain and tested on the other dataset from another domain. The
experiments were conducted on the four public datasets, Real-life Trial (R),
Bag of Lies (B1), Box of Lies (B2), and MU3D (M). As shown in Tabel I, for
visual modalities, we extracted the most adopted visual features including
face frames, and behavior features such as AUs, gaze and affect. For audio
modality, Mel spectrogram and waveform were extracted. We also applied several
different backbone networks as audio and visual encoders. We can observe that
R and B1 datasets generalized the best on B2, and M generalized the best on R.
Note that the B2 dataset was not adopted as a source domain dataset as the
original dataset had too much noise and we cleaned it only for testing. On
average, we can observe that the best result was achieved by using visual
(AU+gaze+affect) features.

Multi-to-Single Domain. Here we fully evaluate the performance of multi-to-
single cross-domain generalization using unimodal features. We conducted
experiments for different domain sampling strategies. As shown in Table II,
for domain-simultaneous training, the best generalization performance is
achieved by training on the Bag of Lies and MU3D datasets and testing on the
Real-life Trial dataset (69.16%), which was also the case for domain-by-domain
training strategy (67.29%). For the domain-alternating strategy, the best
result was observed when transferring to the Box of Lies dataset using the
rest three datasets for training (65.85%). The results showed that the best
generalization performances were obtained when transferring to the real-world
dataset and the gameshow dataset by training on the lab-based datasets. This
was because the lab-based datasets (Bag of Lies and MU3D) are relatively clean
compared to the real-world dataset (Real-life Trial) and the gameshow dataset
(Box of Lies). However, in the opposite case, the generalization performance
degraded, for example, R&M to B1 and R&B1 to M. Different domain sampling
strategies reached their best average performance on different input features
and backbone networks. To be specific, for both domain-simultaneous and
domain-by-domain strategies, models trained on visual (AU+gaze+affect)
features reached their highest accuracies, which were 55.82% and 56.67%,
respectively. Using the domain-alternating strategy, the best accuracy of
56.25% was achieved by training on visual (face frames) features. We can
observe that models trained on visual modalities outperformed those trained on
audio modalities across all the generalization strategies. This may be due to
the rich deceptive cues captured by visual modalities in the publicly
available datasets.

###  4.4 Domain-Simultaneous with Gradient Reversal Layer (GRL)

Following the implementation by Ganin _et al._ [43], we compared the multi-to-
single domain generalization accuracies w/w.o GRL. GRL was proposed to
mitigate the domain shift issue by manipulating the training gradients. It
worked by acting as an identity transform in forward propagation and
multiplying the gradient by a certain negative constant during the
backpropagation without having trainable parameters. GRL was inserted between
encoders and domain classifiers, which was easy to implement. As GRL is a
widely adopted method for domain generalization, it is investigated to show
its effectiveness for the deception detection task. We selected domain-
simultaneous as the baseline and added GRL to the original network with the
same training setups. The average accuracies were reported in Fig. 3, where
different types of visual and audio features and methods were compared.
Training with GRL, the performance of ResNet18 and ResNet18+GRU models using
visual (face frames) features and Wave2Vec model using waveform were enhanced.
However, we observed that MLP models using visual (AU/gaze/affect) features
and the ResNet18 model using Mel spectrograms degraded in performance.
Generally, ResNet18 trained with GRL performed better than MLP for visual
modality, and Wave2Vec trained with GRL boosted the performance and surpassed
the model trained on the Mel spectrogram.

Figure 3: Performance comparisons of Domain-simultaneous training w/ and w/o
Gradient Reversal Layer (GRL).  TABLE IV: The fusion results of multi-to-
single cross-domain generalization accuracy (%) for different generalization
strategies.

Modality & Inputs | Fusion Postion | Fusion Method | R&M to B1 | R&M to B2 | R&B1 to B2 | R&B1 to M | B1&M to R | B1&M to B2 | R&B1&M to B2 | Avg
---|---|---|---|---|---|---|---|---|---|---
Domain-Simultaneous
| Score-level | Average | 53.23 | 43.56 | 57.43 | 50.94 | 51.40 | 58.42 | 48.51 | 51.93
|  | Concat | 52.00 | 56.44 | 60.40 | 51.88 | 51.40 | 62.38 | 57.43 | 55.99
|  | SE-Concat | 55.69 | 58.42 | 44.55 | 52.50 | 56.07 | 58.42 | 58.42 | 54.87
|  | Cross-Atten | 51.08 | 52.48 | 55.45 | 51.56 | 57.01 | 60.40 | 57.43 | 55.06
|  | MLP-Mixer | 53.85 | 43.56 | 62.38 | 52.19 | 56.07 | 61.39 | 59.41 | 55.55
Visual (Face frames) + Visual (AU+Gaze+Affect) | Feature-level | Atten-Mixer(Ours) | 53.23 | 57.43 | 63.37 | 52.19 | 57.01 | 63.37 | 62.38 | 58.43
| Score-level | Average | 49.54 | 55.45 | 52.48 | 51.25 | 54.21 | 59.41 | 55.45 | 53.97
|  | Concat | 49.54 | 54.46 | 49.50 | 53.75 | 44.86 | 63.37 | 54.46 | 52.85
|  | SE-Concat | 52.00 | 58.42 | 42.57 | 51.56 | 54.21 | 58.42 | 63.37 | 54.36
|  | Cross-Atten | 50.46 | 52.48 | 60.40 | 53.12 | 52.34 | 58.42 | 59.41 | 55.23
|  | MLP-Mixer | 53.54 | 57.43 | 52.48 | 50.31 | 52.34 | 60.40 | 47.52 | 53.43
Visual (Face frames) + Audio (Mel spectrogram) | Feature-level | Atten-Mixer(Ours) | 55.69 | 64.36 | 56.44 | 53.75 | 58.88 | 59.41 | 58.42 | 58.14
| Score-level | Average | 48.62 | 58.42 | 56.44 | 50.94 | 51.40 | 56.44 | 53.47 | 53.68
|  | Concat | 48.31 | 53.47 | 56.44 | 50.31 | 56.07 | 63.37 | 57.43 | 55.06
|  | SE-Concat | 49.85 | 50.50 | 47.52 | 52.19 | 49.53 | 57.43 | 61.39 | 52.63
|  | Cross-Atten | 51.38 | 63.37 | 61.39 | 51.25 | 48.60 | 59.41 | 58.42 | 56.26
|  | MLP-Mixer | 55.69 | 59.41 | 57.43 | 53.75 | 50.47 | 61.39 | 50.50 | 55.52
Visual (AU+Gaze+Affect) + Audio (Mel spectrogram) | Feature-level | Atten-Mixer(Ours) | 55.08 | 55.45 | 58.42 | 54.37 | 52.34 | 61.39 | 60.40 | 56.78
| Score-level | Average | 54.77 | 57.43 | 57.43 | 52.81 | 60.75 | 58.42 | 56.44 | 56.86
|  | Concat | 54.46 | 54.46 | 47.52 | 53.12 | 53.27 | 61.39 | 56.44 | 54.38
|  | SE-Concat | 53.23 | 55.45 | 54.46 | 52.50 | 59.81 | 56.44 | 58.42 | 55.76
|  | Cross-Atten | 48.62 | 57.43 | 50.50 | 53.12 | 48.60 | 58.42 | 56.44 | 53.30
|  | MLP-Mixer | 48.92 | 48.51 | 59.41 | 52.81 | 54.21 | 57.43 | 51.49 | 53.25
Visual (Face frames) + Visual (AU+Gaze+Affect) + Audio (Mel spectrogram) | Feature-level | Atten-Mixer(Ours) | 52.31 | 59.41 | 58.42 | 52.50 | 58.88 | 61.39 | 60.40 | 57.62
Domain-Alternating
| Score-level | Average | 55.38 | 57.43 | 58.42 | 50.94 | 57.94 | 50.50 | 61.39 | 56.00
|  | Concat | 55.08 | 58.42 | 63.37 | 51.56 | 62.62 | 62.38 | 53.47 | 58.13
|  | SE-Concat | 50.46 | 49.50 | 63.37 | 52.19 | 57.01 | 65.35 | 56.44 | 56.33
|  | Cross-Atten | 55.08 | 59.41 | 60.40 | 52.50 | 48.60 | 58.42 | 61.39 | 56.54
|  | MLP-Mixer | 57.54 | 63.37 | 63.37 | 50.94 | 65.42 | 52.48 | 56.44 | 58.51
Visual (Face frames) + Visual (AU+Gaze+Affect) | Feature-level | Atten-Mixer(Ours) | 56.62 | 61.39 | 63.37 | 51.56 | 58.88 | 62.38 | 58.42 | 58.95
| Score-level | Average | 50.15 | 58.42 | 60.40 | 50.62 | 51.40 | 60.40 | 58.42 | 55.69
|  | Concat | 53.23 | 62.38 | 54.46 | 51.25 | 59.81 | 54.46 | 54.46 | 55.72
|  | SE-Concat | 50.15 | 52.48 | 64.36 | 50.31 | 59.81 | 55.45 | 50.50 | 54.72
|  | Cross-Atten | 52.92 | 57.43 | 49.50 | 52.81 | 65.42 | 53.47 | 66.34 | 56.84
|  | MLP-Mixer | 51.69 | 58.42 | 57.43 | 51.25 | 62.62 | 58.42 | 56.44 | 56.61
Visual (Face frames) + Audio (Mel spectrogram) | Feature-level | Atten-Mixer(Ours) | 53.23 | 63.37 | 56.44 | 51.25 | 63.55 | 59.41 | 56.44 | 57.67
| Score-level | Average | 51.38 | 56.44 | 54.46 | 50.00 | 64.49 | 61.39 | 49.50 | 55.38
|  | Concat | 49.85 | 62.38 | 58.42 | 51.88 | 61.68 | 55.45 | 51.49 | 55.88
|  | SE-Concat | 50.46 | 62.38 | 58.42 | 50.00 | 57.94 | 59.41 | 65.35 | 57.71
|  | Cross-Atten | 49.85 | 62.38 | 50.50 | 53.44 | 58.88 | 56.44 | 56.44 | 55.42
|  | MLP-Mixer | 50.15 | 55.45 | 58.42 | 50.31 | 58.88 | 61.39 | 57.43 | 56.00
Visual (AU+Gaze+Affect) + Audio (Mel spectrogram) | Feature-level | Atten-Mixer(Ours) | 50.15 | 59.41 | 50.50 | 52.19 | 58.88 | 60.40 | 61.39 | 56.13
| Score-level | Average | 56.00 | 52.48 | 59.41 | 51.88 | 51.40 | 60.40 | 56.44 | 55.43
|  | Concat | 51.69 | 58.42 | 58.42 | 52.81 | 57.01 | 61.39 | 56.44 | 56.60
|  | SE-Concat | 56.31 | 59.41 | 58.42 | 50.31 | 53.27 | 61.39 | 55.45 | 56.37
|  | Cross-Atten | 49.54 | 67.33 | 54.46 | 51.56 | 66.36 | 52.48 | 60.40 | 57.45
|  | MLP-Mixer | 50.15 | 63.37 | 60.40 | 51.56 | 62.62 | 61.39 | 61.39 | 58.70
Visual (Face frames) + Visual (AU+Gaze+Affect) + Audio (Mel spectrogram) | Feature-level | Atten-Mixer(Ours) | 51.69 | 66.34 | 60.40 | 51.88 | 59.81 | 61.39 | 62.38 | 59.13
Domain-by-Domain
| Score-level | Average | 54.77 | 46.53 | 59.41 | 50.62 | 57.94 | 59.41 | 58.42 | 55.30
|  | Concat | 53.85 | 53.47 | 63.37 | 51.88 | 57.94 | 65.35 | 58.42 | 57.75
|  | SE-Concat | 56.62 | 44.55 | 63.37 | 52.19 | 54.21 | 58.42 | 59.41 | 55.54
|  | Cross-Atten | 54.15 | 49.50 | 62.38 | 54.69 | 57.01 | 60.40 | 63.37 | 57.36
|  | MLP-Mixer | 57.54 | 62.38 | 71.29 | 51.25 | 54.21 | 62.38 | 62.38 | 60.20
Visual (Face frames) + Visual (AU+Gaze+Affect) | Feature-level | Atten-Mixer(Ours) | 57.54 | 58.42 | 63.37 | 52.19 | 69.16 | 60.40 | 64.36 | 60.78
| Score-level | Average | 52.00 | 50.50 | 58.42 | 51.56 | 57.94 | 61.39 | 60.40 | 56.03
|  | Concat | 50.46 | 59.41 | 62.38 | 52.19 | 60.75 | 56.44 | 57.43 | 57.01
|  | SE-Concat | 55.38 | 48.51 | 58.42 | 52.81 | 62.62 | 57.43 | 59.41 | 56.37
|  | Cross-Atten | 52.31 | 53.47 | 58.42 | 51.25 | 64.49 | 58.42 | 57.43 | 56.54
|  | MLP-Mixer | 53.23 | 60.40 | 62.38 | 52.19 | 61.68 | 58.42 | 58.42 | 58.10
Visual (Face frames) + Audio (Mel spectrogram) | Feature-level | Atten-Mixer(Ours) | 53.85 | 66.34 | 63.37 | 53.75 | 58.88 | 64.36 | 63.37 | 60.56
| Score-level | Average | 48.62 | 59.41 | 51.49 | 51.25 | 60.75 | 62.38 | 63.37 | 56.75
|  | Concat | 57.23 | 60.40 | 56.44 | 51.25 | 56.07 | 57.43 | 56.44 | 56.47
|  | SE-Concat | 55.08 | 60.40 | 59.41 | 54.06 | 60.75 | 61.39 | 57.43 | 58.36
|  | Cross-Atten | 50.15 | 56.44 | 61.49 | 57.50 | 59.81 | 64.36 | 58.42 | 58.31
|  | MLP-Mixer | 52.62 | 56.44 | 58.42 | 52.50 | 60.75 | 61.39 | 58.42 | 57.22
Visual (AU+Gaze+Affect) + Audio (Mel spectrogram) | Feature-level | Atten-Mixer(Ours) | 51.69 | 56.44 | 57.43 | 54.06 | 66.36 | 64.36 | 60.40 | 58.68
| Score-level | Average | 53.85 | 51.49 | 65.35 | 53.75 | 58.88 | 57.43 | 58.42 | 57.02
|  | Concat | 52.92 | 52.48 | 55.45 | 53.75 | 62.62 | 62.38 | 56.44 | 56.58
|  | SE-Concat | 51.38 | 60.40 | 58.42 | 51.56 | 57.01 | 62.38 | 59.41 | 57.22
|  | Cross-Atten | 51.80 | 52.48 | 62.38 | 53.12 | 59.81 | 64.36 | 60.40 | 57.76
|  | MLP-Mixer | 53.85 | 55.45 | 60.40 | 50.94 | 59.81 | 64.36 | 60.40 | 57.89
Visual (Face frames) + Visual (AU+Gaze+Affect) + Audio (Mel spectrogram) | Feature-level | Atten-Mixer(Ours) | 55.69 | 61.39 | 59.41 | 55.00 | 58.88 | 59.41 | 60.40 | 58.60

###  4.5 Cross-domain Testing with Multimodal Fusion

Here we present multimodal fusion results of cross-domain testing to evaluate
models’ generalization capacities.

Single-to-Single Domain with Fusion. In this section, we conducted experiments
of single-to-single cross-domain testing. Two types of fusion positions were
involved including score-level and feature-level fusion. For feature-level
fusion, multiple fusion methods were adopted such as simple concatenation, SE-
Concat [44], Cross-Atten [37], MLP-Mixer [19], and Attention-Mixer fusion
(Ours). To be specific, simple concatenation refers to the concatenation of
the extracted features before the input to the classifier. SE-concat stands
for the SE attention applied to the concatenated features. Cross-Atten means
the crossmodal attention among input features by using the attention mechanism
from the Transformer. MLP-Mixer uses the method in [19] on the extracted
features. The modalities and input include three types, visual (face frames),
visual (AU+gaze+affect), and audio (Mel spectrogram). It results in four
combinations of either two or three types of inputs. As shown in Table IV,
among each type of input combination, the proposed Attention-Mixer fusion
(Atten-Mixer) achieved the best results. The rest fusion methods showed
comparable results, which were less performed than Atten-Mixer.

Multi-to-Single Domain with Fusion. We benchmarked multi-to-single cross-
domain generalization with different fusion methods by three types of cross-
testing strategies. In total, seven sub-experiments were conducted for
different domain combinations. The results are shown in Tabel IV. On average,
the best accuracies were 58.43%, 59.13%, and 60.78% on domain-simultaneous,
domain-alternating, and domain-by-domain strategies, respectively. Among
these, for both domain-simultaneous and domain-by-domain strategies, the best
results were achieved by taking visual (face frames) and visual
(AU+gaze+affect) features as input, while the best result for domain-
alternating was achieved by using visual (face frames), visual
(AU+gaze+affect) and audio (Mel spectrogram) features. Taking a close look at
the average fusion results, for each type of modality input, Atten-Mixer
achieved the best among the six fusion methods. showing the effectiveness of
the proposed method. By using Atten-Mixer, in general, the average result
showed slightly better when using the domain-by-domain strategy and taking
visual (face frames) and visual (AU+gaze+affect) features as input. To sum up,
the results showed that on the current publicly available datasets, visual
features were better when it came to multi-to-single cross-domain
generalizability. However, the performance differences were not a large gap,
and there were also no significant differences by comparing two-to-one domain
and three-to-one domain cross-testing performances.

Figure 4: Ablation study for attention-mixer layers. The number of layers 4,
5, 6, and 7 are compared. The modality and inputs for A, B, C, and D are in
line with those in Table III from the top to the bottom.

Ablation Study for Attention-Mixer Fusion Module. We conducted an ablation
study for the proposed attention-mixer fusion module with the changes in the
number of attention-mixer layers. The experiments were conducted on single-to-
single domain testing, where the average accuracies were compared. As shown in
Fig. 4, the number of attention-mixer layers was set to 4, 5, 6, and 7. The
modalities and inputs in Table III are compared, where “A” had the inputs of
Visual (Face frame) + Visual(AU+Gaze+Affect), “B” had Visual (Face frames) +
Audio(Mel spectrogram), “C” had Visual (AU+Gaze+Affect) + Audio (Mel
spectrogram), and “D” had Visual (Face frame) + Visual (AU+Gaze+Affect) +
Audio (Mel spectrogram). The results showed that models with 6 attention-mixer
layers achieved the best average accuracies, followed by 7 attention-mixer
layers.

###  4.6 Discussion

We can observe that the general performance of cross-domain deception
detection is unsatisfactory because it is challenging to reduce the domain gap
between each dataset. The domain generalization ability of widely-adopted
methods was relatively weak using either audio or visual features. Different
domain sampling strategies worked well for different audiovisual features.
Fusing multiple modalities is able to mitigate the problem. However, the
performance still needs to be improved.

Ethical Consideration. Developing deception detection using AI should
emphasize respecting privacy, minimizing psychological harm, preventing
discrimination, promoting transparency, etc. Researchers should follow
appropriate regulations to develop and deploy AI systems for deception
detection. Potential misuses and negative impacts include invasion of privacy,
discrimination, erosion of trust, etc. Mitigating these risks requires
responsible practices from researchers and developers.

##  5 Conclusion

In this paper, we benchmark the cross-domain generalization performance for
deception detection on publicly available datasets. We compare the single-to-
single domain and multi-to-single domain generalization performances, where
three strategies are used including domain-simultaneous, domain-alternating,
and domain-by-domain. We also investigate the effectiveness of the gradient
reversal layer for domain-simultaneous strategy. Moreover, we propose the
Attention-Mixer fusion method to alleviate the domain shift issue and boost
the performance. Future works for deception detection are encouraged to
propose better methods to improve the domain generalizability on audio-visual
deception detection task.

Acknowledgments. This work was carried out at the Rapid-Rich Object Search
(ROSE) Lab, Nanyang Technological University, Singapore. The research is
supported by the DSO National Laboratories, under project agreement No.
DSOCL21238.

## References

  * [1] V. Pérez-Rosas, M. Abouelenien, R. Mihalcea, and M. Burzo, “Deception detection using real-life trial data,” in _Proceedings of the 2015 ACM on International Conference on Multimodal Interaction_ , 2015, pp. 59–66.
  * [2] V. Gupta, M. Agarwal, M. Arora, T. Chakraborty, R. Singh, and M. Vatsa, “Bag-of-lies: A multimodal dataset for deception detection,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_ , 2019, pp. 0–7.
  * [3] E. P. Lloyd, J. C. Deska, K. Hugenberg, A. R. McConnell, B. T. Humphrey, and J. W. Kunstman, “Miami university deception detection database,” _Behavior research methods_ , vol. 51, no. 1, pp. 429–439, 2019.
  * [4] F. Soldner, V. Pérez-Rosas, and R. Mihalcea, “Box of lies: Multimodal deception detection in dialogues,” in _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , 2019, pp. 1768–1777.
  * [5] M. Ding, A. Zhao, Z. Lu, T. Xiang, and J.-R. Wen, “Face-focused cross-stream network for deception detection in videos,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 7802–7811.
  * [6] Z. Wu, B. Singh, L. Davis, and V. Subrahmanian, “Deception detection in videos,” in _Proceedings of the AAAI conference on artificial intelligence_ , vol. 32, no. 1, 2018.
  * [7] M. Gogate, A. Adeel, and A. Hussain, “Deep learning driven multimodal fusion for automated deception detection,” in _2017 IEEE symposium series on computational intelligence (SSCI)_. IEEE, 2017, pp. 1–6.
  * [8] L. Mathur and M. J. Matarić, “Unsupervised audio-visual subspace alignment for high-stakes deception detection,” in _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2021, pp. 2255–2259.
  * [9] G. Wang, H. Chen, and H. Atabakhsh, “Criminal identity deception and deception detection in law enforcement,” _Group Decision and Negotiation_ , vol. 13, pp. 111–127, 2004.
  * [10] H. Joudaki, A. Rashidian, B. Minaei-Bidgoli, M. Mahmoodi, B. Geraili, M. Nasiri, and M. Arab, “Using data mining to detect health care fraud and abuse: a review of literature.” _Global Journal of Health Science_ , vol. 7, no. 1, pp. 194–202, 2014.
  * [11] F. H. Glancy and S. B. Yadav, “A computational model for financial reporting fraud detection,” _Decision Support Systems_ , vol. 50, no. 3, pp. 595–601, 2011.
  * [12] J. Synnott, D. Dietzel, and M. Ioannou, “A review of the polygraph: history, methodology and current status,” _Crime Psychology Review_ , vol. 1, no. 1, pp. 59–83, 2015.
  * [13] S. Porter and M. Campbell, “A. vrij, detecting lies and deceit: The psychology of lying and implications for professional practice,” _Expert Evidence_ , vol. 7, pp. 227–232, 09 1999.
  * [14] A. Nortje and C. Tredoux, “How good are we at detecting deception? a review of current techniques and theories,” _South African Journal of Psychology_ , vol. 49, no. 4, pp. 491–504, 2019.
  * [15] H. Karimi, J. Tang, and Y. Li, “Toward end-to-end deception detection in videos,” in _2018 IEEE International Conference on Big Data (Big Data)_. IEEE, 2018, pp. 1278–1283.
  * [16] M. Karnati, A. Seal, A. Yazidi, and O. Krejcar, “Lienet: a deep convolution neural networks framework for detecting deception,” _IEEE Transactions on Cognitive and Developmental Systems_ , 2021.
  * [17] D. Avola, L. Cinque, G. L. Foresti, and D. Pannone, “Automatic deception detection in rgb videos using facial action units,” in _Proceedings of the 13th International Conference on Distributed Smart Cameras_ , 2019, pp. 1–6.
  * [18] J.-T. Yang, G.-M. Liu, and S. C.-H. Huang, “Multimodal deception detection in videos via analyzing emotional state-based feature,” _arXiv preprint arXiv:2104.08373_ , 2021.
  * [19] I. O. Tolstikhin, N. Houlsby, A. Kolesnikov, L. Beyer, X. Zhai, T. Unterthiner, J. Yung, A. Steiner, D. Keysers, J. Uszkoreit _et al._ , “Mlp-mixer: An all-mlp architecture for vision,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 24 261–24 272, 2021.
  * [20] D. B. Buller and J. K. Burgoon, “Interpersonal deception theory,” _Communication theory_ , vol. 6, no. 3, pp. 203–242, 1996.
  * [21] M. Hartwig and C. F. Bond Jr, “Why do lie-catchers fail? a lens model meta-analysis of human lie judgments.” _Psychological bulletin_ , vol. 137, no. 4, p. 643, 2011.
  * [22] A. Vrij, _Detecting lies and deceit: The psychology of lying and implications for professional practice_. Wiley, 2000.
  * [23] B. M. DePaulo, J. J. Lindsay, B. E. Malone, L. Muhlenbruck, K. Charlton, and H. Cooper, “Cues to deception.” _Psychological bulletin_ , vol. 129, no. 1, p. 74, 2003.
  * [24] T. R. Levine and S. A. McCornack, “Theorizing about deception,” _Journal of Language and Social Psychology_ , vol. 33, no. 4, pp. 431–440, 2014.
  * [25] A. Vrij and P. A. Granhag, “Eliciting cues to deception and truth: What matters are the questions asked,” _Journal of Applied Research in Memory and Cognition_ , vol. 1, no. 2, pp. 110–117, 2012.
  * [26] J. B. Hirschberg, S. Benus, J. M. Brenier, F. Enos, S. Friedman, S. Gilman, C. Girand, M. Graciarena, A. Kathol, L. Michaelis _et al._ , “Distinguishing deceptive from non-deceptive speech,” 2005.
  * [27] G. Warren, E. Schertler, and P. Bull, “Detecting deception from emotional and unemotional cues,” _Journal of Nonverbal Behavior_ , vol. 33, no. 1, pp. 59–69, 2009.
  * [28] P. Ekman and W. V. Friesen, “Nonverbal leakage and clues to deception,” _Psychiatry_ , vol. 32, no. 1, pp. 88–106, 1969.
  * [29] ——, “Detecting deception from the body or face.” _Journal of personality and Social Psychology_ , vol. 29, no. 3, p. 288, 1974.
  * [30] L. Mathur and M. J. Matarić, “Introducing representations of facial affect in automated multimodal deception detection,” in _Proceedings of the 2020 International Conference on Multimodal Interaction_ , 2020, pp. 305–314.
  * [31] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [32] B. Amos, B. Ludwiczuk, M. Satyanarayanan _et al._ , “Openface: A general-purpose face recognition library with mobile applications,” _CMU School of Computer Science_ , vol. 6, no. 2, p. 20, 2016.
  * [33] A. Toisoul, J. Kossaifi, A. Bulat, G. Tzimiropoulos, and M. Pantic, “Estimation of continuous valence and arousal levels from faces in naturalistic conditions,” _Nature Machine Intelligence_ , vol. 3, no. 1, pp. 42–50, 2021.
  * [34] F. Eyben, M. Wöllmer, and B. Schuller, “Opensmile: the munich versatile and fast open-source audio feature extractor,” in _Proceedings of the 18th ACM international conference on Multimedia_ , 2010, pp. 1459–1462.
  * [35] Z. Wang, Z. Wang, Z. Yu, W. Deng, J. Li, T. Gao, and Z. Wang, “Domain generalization via shuffled style assembly for face anti-spoofing,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 4123–4133.
  * [36] T. Varanka, Y. Li, W. Peng, and G. Zhao, “Data leakage and evaluation issues in micro-expression analysis,” _IEEE Transactions on Affective Computing_ , 2023.
  * [37] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [38] H. U. D. Ahmed, U. I. Bajwa, F. Zhang, and M. W. Anwar, “Deception detection in videos using the facial action coding system,” _arXiv preprint arXiv:2105.13659_ , 2021.
  * [39] K. Zhang, Z. Zhang, Z. Li, and Y. Qiao, “Joint face detection and alignment using multitask cascaded convolutional networks,” _IEEE Signal Processing Letters_ , vol. 23, no. 10, pp. 1499–1503, 2016.
  * [40] J. Chung, C. Gulcehre, K. Cho, and Y. Bengio, “Empirical evaluation of gated recurrent neural networks on sequence modeling,” _arXiv preprint arXiv:1412.3555_ , 2014.
  * [41] L. Noriega, “Multilayer perceptron tutorial,” _School of Computing. Staffordshire University_ , vol. 4, no. 5, p. 444, 2005.
  * [42] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 12 449–12 460, 2020.
  * [43] Y. Ganin and V. Lempitsky, “Unsupervised domain adaptation by backpropagation,” in _International conference on machine learning_. PMLR, 2015, pp. 1180–1189.
  * [44] J. Hu, L. Shen, and G. Sun, “Squeeze-and-excitation networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 7132–7141.