# Unmasking Illusions: Understanding Human Perception of Audiovisual Deepfakes

Ammarah Hashmi, Sahibzada Adil Shahzad, Chia-Wen Lin, , and Yu Tsao, , Hsin-
Min Wang Ammarah Hashmi is with the Social Networks and Human-Centered
Computing Program, Taiwan International Graduate Program, Institute of
Information Science, Academia Sinica, Taipei 11529, Taiwan, and also with the
Institute of Information Systems and Applications, National Tsing Hua
University, Hsinchu 300044, Taiwan. (e-mail: hashmiammarah0@gmail.com).
Sahibzada Adil Shahzad is with the Social Networks and Human-Centered
Computing Program, Taiwan International Graduate Program, Institute of
Information Science, Academia Sinica, and also with the Department of Computer
Science, National Chengchi University, Taipei 11529, Taiwan. (e-mail:
adilshah275@iis.sinica.edu.tw). Chia-Wen Lin is with the Department of
Electrical Engineering and the Institute of Communications Engineering,
National Tsing Hua University, Hsinchu 300044, Taiwan. (e-mail:
cwlin@ee.nthu.edu.tw). Yu Tsao is with the Research Center for Information
Technology Innovation, Academia Sinica, Taipei 11529, Taiwan. (e-mail:
yu.tsao@citi.sinica.edu.tw). Hsin-Min Wang is with the Institute of
Information Science, Academia Sinica, Taipei 11529, Taiwan. (e-mail:
whm@iis.sinica.edu.tw).

###### Abstract

The emergence of contemporary deepfakes has attracted significant attention in
machine learning research, as artificial intelligence (AI) generated synthetic
media increases the incidence of misinterpretation and is difficult to
distinguish from genuine content. Currently, machine learning techniques have
been extensively studied for automatically detecting deepfakes. However, human
perception has been less explored. Malicious deepfakes could ultimately cause
public and social problems. Can we humans correctly perceive the authenticity
of the content of the videos we watch? The answer is obviously uncertain;
therefore, this paper aims to evaluate the human ability to discern deepfake
videos through a subjective study. We present our findings by comparing human
observers to five state-of-the-art audiovisual deepfake detection models. To
this end, we used gamification concepts to provide 110 participants (55 native
English speakers and 55 non-native English speakers) with a web-based platform
where they could access a series of 40 videos (20 real and 20 fake) to
determine their authenticity. Each participant performed the experiment twice
with the same 40 videos in different random orders. The videos are manually
selected from the FakeAVCeleb dataset. We found that all AI models performed
better than humans when evaluated on the same 40 videos. The study also
reveals that while deception is not impossible, humans tend to overestimate
their detection capabilities. Our experimental results may help benchmark
human versus machine performance, advance forensics analysis, and enable
adaptive countermeasures.

###### Index Terms:

Deepfake detection, Video forgery detection, Audiovisual deepfake detection,
Human perception, Subjective evaluation, gamification, crowdsourcing

##  I Introduction

Deepfakes are computer-generated or manipulated multimedia content that
convincingly simulates or conceals reality. The emergence of Artificial
Intelligence (AI) and Deep Learning (DL) technologies has advanced
manipulation tools that are capable of altering a person’s identity by
swapping voice or facial features. Such fabricated multimedia content,
generated or manipulated through cutting-edge AI techniques, can depict a
person in situations they’ve never been in. Deepfake generation technology is
a gold mine with wide applications; however, its dark side poses a significant
threat to society and can have shocking consequences for unsuspecting
citizens. For instance, it can cause chaos when highly realistic videos of
public figures or politicians are generated to spread misinformation. Fake
videos can damage someone’s reputation and emotional well-being by featuring
them in explicit content without their consent. The widespread malicious and
unethical use of synthetic fake content on social media platforms has caused
public unrest and raised serious trust issues in the media.

Figure 1: An audiovisual deepfake example showcasing the simultaneous
manipulation of audio and visual tracks to produce a highly convincing and
deceptive deepfake video.

Exponential advances in hyper-realistic visual effects and the manipulation
across various modalities, including audio, image, and video, have led to the
creation of increasingly plausible content, thereby undermining the trust in
the proverb “Seeing is believing” in digital media. This raises concerns that
audiovisually manipulated videos become visually and aurally
indistinguishable. Audiovisual deepfakes are a form of synthetic media
generated via sophisticated deep learning techniques that manipulate the
acoustic and visual tracks of a video, often created with the intention of
deceiving people. Fig. 1 provides an illustrative example of audiovisual
deepfake creation. In addition, the development of deepfake generation
techniques, such as Generative Adversarial Network (GAN) [1], Autoencoder
(AE), and Variational Autoencoder (VAE) [2], and their growing public
accessibility have had a strong impact on visual and auditory content. This
highlights the importance of caution within the research community as it can
be difficult to distinguish between truth and fabrication, creating new social
challenges. In this paper, we study human perception of audiovisual deepfakes.
Our contribution provides a deeper understanding of human performance in
detecting audiovisual deepfakes, and our analysis may be valuable for
enhancing cybersecurity measures, advancing forensic analysis, and developing
countermeasures.

The generation [1, 3, 4, 5] and detection [5, 6, 7, 8, 9] of deepfakes have
been extensively explored by the research community. While human perception of
deepfake speech, images, or videos respectively has been a major focus of
researchers, to date, there has been less research in the field of human
perception of audiovisual deepfakes. This study is an empirical investigation
designed to evaluate the ability of human participants to detect audiovisual
deepfake videos. The objective of the experiment is to analyze the perceptual
boundaries of audiovisual deepfakes and explore how the average human
observers perceive and interpret fabricated content. Therefore, we conducted a
comprehensive crowdsourced subjective test via a web-based platform, allowing
human participants to watch a series of videos and identify authenticity.
Evaluating human’s ability to detect deepfakes is crucial, but it is also
important to understand how detection algorithms identify them. Therefore, we
also tested five state-of-the-art (SOTA) AI deepfake detectors on the same set
of videos. Finally, we compared the subjective test results with the test
results of the AI detection models. In particular, we explored the following
questions:

  * •

Are participants able to differentiate between real and audiovisual deepfake
videos above chance levels?

  * •

Are participants better than SOTA AI algorithms at identifying audiovisual
deepfake videos?

  * •

After perceiving a video as a deepfake, are participants able to correctly
identify specific forms of manipulation (i.e., audio and video)?

  * •

Does familiarity with and forewarning of deepfakes improve participants’
detection accuracy?

  * •

Does participants’ self-reported level of confidence in their answers align
with their accuracy in detecting audiovisual deepfake videos?

  * •

Do participants’ age, native language, and affinity for technology influence
the perception of deepfake videos?

  * •

Do participants who undergo several rounds of training show a better ability
to identify deepfakes?

The remainder of the paper is organized as follows. Section II briefly reviews
related work. Section III outlines the details of the experiment, including
the setup, design, dataset, and AI models. Section IV presents the human test
results and compares them with the results of the AI models. Section V
discusses the limitations of this subjective study. Finally, Section VI
concludes the paper and proposes future work.

##  II Related Work

Although many models have been proposed for detecting audio, image, video, and
audiovisual deepfakes, in this section we only review related research on
human perception of deepfakes.

In recent years, speech synthesis technology has made significant progress in
producing speech that closely mimics human speech patterns and intonation [3,
4]. Human perception studies conducted from different perspectives have shown
that listeners often have difficulty distinguishing synthesized speech from
natural speech, highlighting the effectiveness of this technology in creating
highly realistic speech. In [10], experimental results in English and Mandarin
highlight the difficulty for humans to identify audio deepfakes, with only 73%
of audio deepfakes correctly identified regardless of language. As speech
synthesis algorithms continue to improve, detecting audio deepfakes may become
more challenging. The study in [11] compared human performance with AI models,
showing that humans and deepfake detection algorithms exhibit the same
strengths and weaknesses, but both have difficulty identifying certain types
of attacks. Recent work in [12] assessed human perception abilities by
engaging participants in scenarios using synthetic voice messages under the
guise of testing the user-friendliness of voice messages. This study showed
that people have difficulty perceiving deception and that no one reacts to
deceptive deepfake messages. [13] presented a comparative analysis between
machine and human perception of adversarial speech, specifically synthetic
audio commands that are recognized as valid messages. Meanwhile, the study in
[14] conducted an in-depth assessment of college students’ perception of audio
deepfakes, including how their academic majors and backgrounds influence their
perceptions.

For images, research on deepfake technology has shown that subtle changes in
faces can generate compelling images that are difficult to detect by human
observers. Therefore, numerous studies have attempted to examine human
perception capabilities to identify fabricated images, and have shown that
human performance is not significantly better than the random chance level.
For example, the subjective study in [15] analyzed the human ability to detect
forgery in digital images, and found that humans can be easily fooled and that
participants not only performed poorly at identifying image forgeries, but
also often doubted the authenticity of real images. [16] evaluated the human
ability to identify image deepfakes of human faces, and found that people are
not naturally good at detecting deepfakes and that the simple interventions
tested do not help. [17] focused on a crowdsourcing-based approach to evaluate
synthetic images specifically generated with GANs. The study in [18] shows
that synthetically generated faces are not just highly photorealistic, they
are nearly indistinguishable from real faces and are judged more trustworthy.
The study in [19] explored the impact of synthetic image deepfakes on college
students’ perception and understanding of the technology.

There are also subjective studies analyzing human ability to distinguish
between authentic and synthetic videos, and comparing human performance with
AI models. The study in [20] shows that both human and AI models exhibit
comparable performance, but with different errors, and that subjects with
access to model predictions perform better than subjects without access to
model predictions. The study in [21] shows that while human perception is very
different from machine perception, both are successfully fooled by deepfakes
but in different ways. Specifically, algorithms have difficulty detecting
deepfake videos that are easily spotted by humans. According to [22], people
cannot reliably detect deepfakes. Raising awareness and financial incentives
do not improve people’s detection accuracy. People tend to mistake deepfakes
as authentic videos (rather than vice versa) and overestimate their own
deepfake detection abilities. Therefore, several studies have attempted to
analyze the factors that influence viewers’ ability to perceive deepfake
videos. The authors of [23] conducted an investigative user study and analyzed
existing AI detection algorithms to uncover the unknown factors behind the
detection of deepfakes. A similar study in [24] shows that two-thirds of
participants were unable to accurately detect a sequence of four videos as
either genuine or deepfake, and that familiarity with the subjects in the
videos had a statistically significant impact on the individuals’ perception
ability. Another study in [25] delved into how humans and machines perceive
deepfake videos, suggesting that incorporating emotional factors and
leveraging specialized visual processing may be promising strategies to
enhance deepfake detection. The study in [26] explored the difference in
detection performance between human observers and automated systems by
comparing their abilities to identify deepfake videos in the presence of noisy
channels. The interesting work in [27] examined tourists’ visit intention by
watching deepfake destination videos, and found that the factors that affect
the tourists’ visit intention after watching deepfake videos include
information manipulation tactics, trust and media richness, while perceived
deception does not influence tourists’ visit intention. The study in [28]
shows that people are more likely to feel uncertain than to be misled by
political deepfakes, but this resulting uncertainty, in turn, reduces trust in
news on social media. The authors of [29] explored the impact of deepfake
videos’ informative cues on individuals’ perceived accuracy of claims and
their intentions to share non-political deepfakes, and found that high
cognitive individuals were less likely to trust news and share content, and
that people were more likely to believe that deepfake claims were true when
informative cues were absent. The results in [30] indicate various challenges
in deepfake user perception that technology developers need to address before
the potential of deepfake applications can be realized for human-computer
interaction.

In addition to the above-mentioned human perception and cognition of single-
modal synthetic (or manipulated) speech, images, and videos, there have also
been a few studies exploring cross-modal human perception. For example, the
authors of [31] conducted a comprehensive survey of 3,002 participants across
three countries to understand humans’ ability to detect synthetic media across
audio, image, and text. The results show that SOTA forgeries are nearly
indistinguishable from “authentic” media, highlighting the difficulty of
identifying deepfakes. The study in [32] shows that individuals are more
vulnerable to, and more likely to share, video deepfakes than other forms of
deepfakes (e.g., audio deepfakes and cheapfakes), and that individuals with
high cognitive ability are less likely to perceive deepfakes as accurate or
share them. Another experimental study in [33] compared the performance of
human perception in detecting political deepfake speeches across text, audio,
and video, finding that audio and visual information enables more accurate
discernment than text alone.

Research on human perception of audiovisual deepfakes is emerging but remains
relatively underexplored. For example, the study in [20] shows that when
synthesized speech is subtle and well synchronized with corresponding lip
movements, audiovisual deepfakes can achieve a high level of realism that
challenges human perception. The authors of [34] conducted a pilot study on
the impact of physical attractiveness on students’ evaluations of teachers and
found that students’ ratings of the quality of teachers’ deepfake videos and
real videos were comparable, and that students did not detect deepfake. The
authors of [35] conducted a subjective evaluation of media consumers’
vulnerability to fake audiovisual content, and found that their detection
performance improved when participants knew the displayed individual or when a
biometric reference video (introducing the individual and its behavior) was
available to them during testing. As technology advances and manipulated
content becomes more common, so does the potential for audiovisual deepfakes
to deceive individuals and influence public opinion. To understand how humans
perceive and interpret incredibly realistic audiovisual deceptive content, our
study adopts a subjective approach that allows for a nuanced understanding of
how individuals perceive and evaluate audiovisual deepfakes, shedding light on
the cognitive processes involved.

Figure 2: The web interface where participants watch videos and identify
whether the videos are ’Real’ or ’Fake’. After classification, participants
are shown the labels of the videos. The web interface can be accessed via
https://jack24.mooo.com/web/experiment/frontend/web/ Figure 3: Distribution of
participants in the experiment.

##  III Experiment

We conducted an online experiment among English-speaking participants with the
primary aim of determining the extent to which synthetically generated
audiovisual content misleads viewers or creates uncertainty about the
authenticity of the video. In other words, we developed an online platform to
analyze human performance and perception of fake videos. Our study delves into
human behavior, exploring the psychological and perceptual factors that lead
people to be misled or experience uncertainty when exposed to content that
include acoustic and visual manipulations. This section outlines the structure
of the experiment, the dataset we used, the sample selection process, and how
we curated the samples presented to participants. Additionally, we discuss
five AI detection models for comparison with human performance. The focus of
our experiment is to evaluate the potential of humans in detecting audiovisual
deepfakes, not to evaluate AI detectors.

###  III-A Experiment Description

We organized an online experiment and recruited subjects through the
crowdsourcing platform Amazon Mechanical Turk. Each participant was paid $10
to complete the study, which typically lasted an average of 30 minutes. We
provided the participants with the URL to conduct the experiment and informed
them of the eligibility criteria for participating in the study. Each
participant was assigned a unique ID. Participants must retain this unique ID
for use in the payment process. When submitting their task on Amazon
Mechanical Turk, participants were required to enter this unique user ID as
proof of completion. Failure to provide a valid user ID will result in
ineligibility for payment processing. This protocol ensures the integrity of
the payment process and helps identify and exclude incomplete or fraudulent
submissions.

Figure 4: Age distribution of participants in the experiment. Figure 5: Sample
videos from the FakeAVCeleb dataset, including individuals of varying ages,
ethnic backgrounds, and genders (faces cropped from the videos).

Our experiment presented a binary classification challenge, in which
participants were shown a series of video clips, one at a time, and had to
discern whether the video was real or manipulated, as shown in Fig. 2.
Participants were informed that there was a 50% chance that each video was
fake and encouraged to rely strictly on their own instincts and observations.

To assess their ability to identify the source of manipulation in situations
where participants thought the video was fake, they were prompted to carefully
specify which modality of the video was altered – video, audio, or both. In
the case of video manipulation, participants were further asked to indicate
which aspects of the visual manipulation were noticed, such as whether the
lips, face, or eyes were manipulated or the audio and video were out of sync.
Likewise, in the case of audio manipulation, participants were asked to choose
between the audio being manipulated or being out of sync with the video. Once
participants classified a video, they were provided with the correct label; in
this way, participants could track their performance. It is worth noting that
the correct label is either real or fake, excluding specific manipulation
types in each modality. Each participant had to complete 80 rounds, i.e.,
judging 80 videos. Our experiment consists of two phases. Participants judged
40 videos presented in a random order in the first phase, and then again
judged the same 40 videos presented in another random order in the second
phase. This was to see if individuals performed consistently the second time
they judged the videos. Upon completion of the 80-rounds experiment,
participants were asked to indicate their level of confidence. Each
participant was asked to indicate how confident they were in the accuracy of
their answers. Finally, participants received feedback on the number of
correctly classified samples out of the total number.

###  III-B Participants

We excluded participants who participated in less than 80 rounds. A total of
110 participants completed the experiment, evenly distributed by gender, with
half being native English speakers and the other half being non-native English
speakers. Half of the participants received a forewarning note about the
threat of deepfakes, while the other half did not. This allowed us to explore
whether prior awareness and understanding of the risks associated with
deepfakes impacted participants’ performance. The overall distribution of
participants is shown in Fig. 3. The age distribution of participants is shown
in Fig. 4. Additionally, we asked participants about their information
technology (IT) skill level by providing four options: beginner, intermediate,
good, or expert. This distribution is shown in Table I.

TABLE I: Distribution of participants at different IT skill levels. IT Skill Level | Number of Participants
---|---
Beginner | 28
Intermediate | 28
Good | 27
Expert | 27

###  III-C Dataset

We conducted the study using the FakeAVCeleb dataset [36], a dataset
specifically designed for the task of audiovisual forgery detection. Its fake
videos cover audio manipulation, video manipulation, or both, using advanced
tools including including SV2TTS [37] (a transfer learning-based real-time
voice cloning tool) for synthetic clone voice generation and Wav2Lip [38],
Faceswap [39], and Fsgan [40] for visual manipulation. The videos in the
dataset are divided into 4 categories, namely Fake-Video-Fake-Audio (FVFA),
Fake-Video-Real-Audio (FVRA), Real-Video-Fake-Audio (RVFA), and Real-Video-
Real-Audio (RVRA). The dataset contains videos of diversity, including four
ethnic backgrounds, various geographic locations, and both genders. Fig. 5
shows some video examples.

By leveraging this dataset, we aim to delve into human behavioral patterns in
identifying audiovisual deepfake videos. Incorporating audio and visual
manipulations into the dataset allows us to gain valuable insights into how
individuals interpret and identify fakes across modalities.

####  III-C1 Data Selection

In our experiment, we manually selected 40 videos from the test set of the
FakeAVCeleb dataset, 20 videos from the real class, and 20 videos from the
fake class. In each class, 10 videos feature males, and 10 videos feature
females.

For the fake class, we selected 4 videos for each gender from the FVFA and
FVRA categories, respectively, and 2 videos for each gender from the RVFA
category. Since the FakeAVCeleb dataset contains various manipulation types
across acoustic and visual modalities, to keep the test set balanced, we
selected samples from each manipulation type.

Figure 6: The average detection accuracy of individual participants across all
videos in Phases I and II.

###  III-D Web Platform

The experiment was conducted using our gamification-based web platform. We
leveraged the popular open-source web server nginx to engage participants in
the task of identifying the authenticity of videos they watched. For backend
development, we used the high-performance framework PHP Yii2 and used PHP 7.4
as the developing language. MongoDB, known for its flexibility and
performance, was used as a NoSQL database, with MongoDB Compass providing an
intuitive graphical user interface (GUI). For efficient code editing, we used
Sublime Text.

###  III-E Metrics

We used accuracy as a metric to evaluate human perception of audiovisual
deepfakes, which measures the overall correctness of participants’ judgments
in distinguishing real videos from deepfake videos. Accuracy refers to the
proportion of correctly identified videos to the total number of videos
presented to participants, and is calculated as follows:

$\text{Accuracy}=\frac{\text{Number of Correctly Identified
Videos}}{\text{Total Number of Videos}}\times 100\%.$ (1)

In conjunction with accuracy, we evaluated human perception of audiovisual
deepfakes using a combination of qualitative analysis and contextual factors
including gender, age, native language, and IT skill level. Qualitative
analysis refers to the examination and interpretation of non-quantitative
data, focusing on the qualitative aspects of participants’ responses,
behaviors, and experiences.

###  III-F Participant Motivation

Participant motivation plays a crucial role in maintaining engagement and
completion rates in subjective studies, especially when the tasks are
extensive or repetitive. The risk of the task becoming boring after a few
rounds poses a significant challenge that can strongly influence the design
process of the study. To alleviate this problem, three approaches were
employed.

First, we offered a monetary reward of $10 upon completion of the study. This
approach not only serves as a powerful incentive for participants to
participate, but also contributes to a more effective and rewarding research
experience overall. Second, we divided the task into two phases. This division
allows participants to experience a sense of accomplishment after completing
the first phase. By providing this tangible sense of accomplishment at the
midpoint, participants are empowered to complete the second phase with renewed
energy and focus. Third, we exposed participants to the correct label after
each round. This immediate feedback keeps their interest and helps them track
their performance. These strategies maintained engagement and ultimately
increased their motivation to complete the study.

###  III-G AI Models

We evaluated five SOTA audiovisual deepfake detection models. These models
were trained on the training set of the FakeAVCeleb dataset and tested on the
same test set used in the subjective evaluation. This comparison provides
scientific insights into human and machine behavior when detecting audiovisual
deepfakes.

  * •

LipForensics [41] focuses on identifying significant semantic inconsistencies
in mouth movements that are often present in AI-generated video content. Note
that it is a visual-only model and cannot detect fake videos with only audio
manipulation.

  * •

AV-Lip-Sync [8] utilizes speech-lip synchronization to detect forgery in
videos. Specifically, it identifies inconsistencies between the lip movements
observed in the video and the synthetic lip movements generated from the audio
using the Wav2lip model.

  * •

AV-Lip-Sync+ [6] is also based on speech-lip synchronization. It extracts
acoustic and visual features via a transformer-based self-supervised learning
(SSL) pre-trained AVHuBERT [42] model.

  * •

CNN-Ensemble is a CNN-based ensemble network [7], consisting of an audio-only
network, a video-only network, and an audiovisual network. The outputs of the
three networks are fused by majority voting.

  * •

AVTENet is a transformer-based ensemble multimodal deepfake detector [9]. The
model consists of an audio-only network, a video-only network, and an
audiovisual network, all of which are based on pre-trained transformer-based
models for feature extraction. It has various fusion strategies. In the
experiment, we used feature fusion.

Figure 7: The average detection accuracy for individual videos across all
participants in Phases I and II.

##  IV Results

In this section, we discuss the results of our subjective analysis, focusing
on the performance comparison between human observers and AI algorithms based
on various factors. We delve deeper into the efficacy of human perception in
detecting audiovisual deepfakes to gain insights. The core findings of our
work are summarized as follows:

  * •

Participants demonstrated an above-chance ability to differentiate between
real videos and deepfake videos.

  * •

Human observers performed worse than SOTA AI algorithms at identifying
deepfake videos.

  * •

Participants had difficulty identifying the specific form of manipulation
after perceiving a video as fake, and tended to attribute it to visual
manipulation even when acoustic manipulation was present.

  * •

Familiarizing them with and being forewarned about deepfakes did not
significantly improve participants’ detection accuracy.

  * •

Participant’s self-reported level of confidence did not align with their
actual accuracy in detecting audiovisual deepfake videos.

  * •

Participants’ age, gender, and native language affected their perception of
deepfake videos, but their affinity for technology did not.

  * •

Participants showed an enhanced ability to discern deepfakes after several
rounds of training.

###  IV-A Overall Detection Accuracy across Participants

Fig. 6 shows the Phase I (40 rounds) and Phase II (40 rounds) deepfake
detection accuracy of individual participants. From Phase I to Phase II, the
average accuracy across all participants increased from 63.30% to 67.98%,
resulting in an overall accuracy of 65.64%.

Human performance at detecting audiovisual deepfakes is marginally better than
random chance. This paper demonstrates that deepfakes have the ability to
deceive the majority of the public. This result is consistent with previous
research [11], [16] and [21].

Most participants performed better in Phase II than in Phase I. The paired
samples t-test analysis showed that the average accuracy improvement from
Phase I to Phase II was significant with a p-value less than 0.0001. The
results show that the training is effective and that increased exposure and
familiarity help them learn and adapt to the stimuli. Over time, their anxiety
may reduce and attention and concentration may increase. Furthermore, after
each round, participants were exposed to the correct label of the video. This
immediate feedback on their performance reflected their correctness and
errors, the former increasing their confidence in the detection, and the
latter enhancing their ability to accurately distinguish real content from
synthetic content.

Figure 8: Proportional distribution of participants’ responses. Figure 9:
Human detection accuracy grouped by (a) age, (b) gender, (c) native or non-
native English speakers, and (d) IT skill level. The box in each box-and-
whisker plot illustrates the interquartile range (IQR), covering the middle
50% of the data, while the upper and lower whiskers extending from the box
show the range of the overall data. Data points outsides the whisker
boundaries are outliers, representing unusually high or low values. The line
within the box represents the median, and the dashed line within the box
represents the mean of the data.

Fig. 7 shows the average detection accuracy for individual videos across all
participants in Phases I and II. Most videos had better detection accuracy the
second time they were shown to participants, especially fake videos. Notably,
participants continued to correctly identify fabricated videos as deepfake;
however, they might experience confusion and struggle to detect genuine videos
as real. Nevertheless, despite the confusion, they still managed to perform
better and show overall improvement in Phase II. This upward trend highlights
the effectiveness of continuous learning to improve detection capabilities
over time.

Interestingly, we also observed a consistent pattern of authenticity bias,
with people being more biased towards detecting videos as real, even though
they were well-informed that half of the videos were deepfake. This cognitive
bias is evident in Fig. 8, where 59% of videos were perceived as real and 41%
were identified as fake. Our findings suggest that people follow the principle
of “seeing is believing” and tend to believe visual information, leading to
interpreting a video as authentic even if it is not. This bias can have a
significant impact on decision-making, as people may rely more on intuition
than objective evidence, which can lead to them being deceived by fabricated
content.

Furthermore, when participants were prompted to indicate the manipulated
modality after identifying a video as fake, they tended to believe that both
modalities (audio and video) were manipulated, or that there was only visual
manipulation. This tendency may stem from visual primacy or cognitive load.
Humans are highly visual creatures, and our brains prioritize visual stimuli
over auditory signals. One potential factor may be sensory discrepancy; in
some cases, deepfake videos exhibit more pronounced visual discrepancies than
auditory ones. Furthermore, processing audio and visual information
simultaneously may cause cognitive challenges; therefore, participants found
it easier to focus on visual cues, encouraging them to prioritize visual
information when detecting manipulations in videos. As shown in Fig. 8, of the
41% of “fake” responses, 16% believed that there were two manipulation modes,
15% believed that there was only visual manipulation, and only 10% believed
that there was only auditory manipulation. However, each manipulation accounts
for about 17% of all videos, and the numbers are balanced.

###  IV-B Detection Accuracy under Different Participant Groupings

In the following analytical study, we examined how the human ability to
perceive audiovisual deepfakes correlates with four factors: age, gender,
native or non-native English speakers, and IT skill level. For each factor, we
aimed to determine important trends that might influence participants’
effectiveness in identifying fakery in the video.

Figure 10: Audiovisual deepfake detection accuracy of individual female and
male participants in Phases I and II. Circle size indicates the overall
accuracy of participants’ judgments, with larger circles indicating higher
accuracy. Figure 11: Participants’ actual detection accuracy and self-reported
confidence in audiovisual deepfake detection. Participants’ average self-
reported confidence is 77.60%, which is higher than the average accuracy of
65.64%.

As shown in Fig. 9(a), older participants (41-50 years old group) had lower
accuracy compared to younger participants in the 20-30 years old and 31-40
years old groups. Differences in digital literacy, exposure to digital
technology, and familiarity with digital media manipulation techniques may be
the reason. Young people who have grown up in the digital age may be more
familiar with media deception. As a result, they have a sharper eye for
identifying deepfakes and experience in critically evaluating online content.
In contrast, older participants may have difficulty identifying subtle signs
of manipulation in deepfake videos due to their limited exposure to digital
media manipulation. Another possible explanation for the lower detection
accuracy in the older participants is that their cognitive abilities decline
with age (e.g., cognitive abilities such as memory and attention may be
affected), while younger individuals may have greater cognitive flexibility,
which helps them to adapt to new information more quickly and distinguish
truth from falsehood.

The analysis results in Fig. 9(b) show that female participants are generally
inferior to male participants in identifying the authenticity of videos. There
may be several potential factors that contribute to this observed difference.
For example, men, on average, are more interested in and exposed to technology
and therefore more familiar with fabricated content. Stereotypes and social
norms are also possible factors, with men more likely to engage in activities
and interests involving technology and digital media manipulation. Cognitive
differences between men and women may also affect their ability to detect
deepfakes, such as differences in attention to detail or spatial reasoning
skills. However, we found that female participants had better learning
capabilities, with the average accuracy increasing from 61.59% in Phase I to
67.09% in Phase II, while male participants’ performance improved less, from
65.00% in Phase I to 68.86% in Phase II. Fig. 10 reflect the results. Thirty-
six of the 55 female participants performed better in Phase II than in Phase
I, whereas only 33 of the 55 male participants performed better in Phase II
than in Phase I. A higher proportion of female participants performed better
in Phase II than in Phase I compared to male participants. Furthermore, Fig.
10 also shows that participants who were more accurate in Phase I also tended
to be more accurate in Phase II, and vice versa.

From Fig. 9(c), we can see that native English speaking participants performed
better than non-native English speaking participants. This may be attributed
to language proficiency, as our experiment was conducted on English-speaking
videos. Native English-speaking participants tended to have a stronger command
of the language, which helped them detect inconsistencies and errors in
speech.

As shown in Fig. 9(d), there is no significant correlation between human
audiovisual deepfake detection and their IT skill level. Participants may have
overestimated their IT skill level, leading to the result that the human
ability to identify authenticity did not improve with increasing IT skill
level as expected.

Notably, participants showed improved performance in phase II regardless of
grouping, as shown in Fig. 9. The reasons have been discussed in the previous
section.

###  IV-C Accuracy vs. Confidence

Next, we analyze the correlation between accuracy and confidence, as this
insight is valuable for assessing participants’ degree of metacognitive
awareness. Therefore, to estimate whether participants correctly perceived
their ability to detect audiovisual deepfakes, we compared their actual
detection accuracy with self-reported confidence across all videos.
Surprisingly, we found that participant’s self-reported confidence in their
answers did not align with their accuracy in detecting audiovisual deepfake
videos. Despite expressing high confidence, their actual accuracy was low. As
shown in Fig. 11, most participants reported average confidence that was
higher than their average accuracy. The average confidence level across all
participants was 77.60%, while the average accuracy was 65.64%. This
difference means that humans’ overconfidence in deepfake detection will cause
deepfakes to bring greater harm to human society.

###  IV-D Detection Intervention

At first, audiovisual deepfakes may appear convincing; however, AI-generated
content can exhibit artifacts, such as irregular shadows, hair anomalies,
speech artifacts, glitches, or inconsistencies. An effective approach to
enhance participants’ performance in detecting audiovisual deepfakes is
raising awareness about these deceptive techniques and their adverse effects.
Surprisingly, our investigation shows that forewarning of these artifacts and
the potential adverse effects of audiovisual deepfake technology did not have
any significant impact on participants’ performance. As shown in Fig. 12,
participants who received a warning showed similar performance to those who
did not receive a warning. Not only did the two groups of participants have
similar median accuracy/confidence, but the distributions were also similar.
The similar performance in both scenarios may be due to human visual and
auditory perception of content. Humans may have cognitive biases and be
overconfident in their ability to detect audiovisual deepfakes. Therefore,
even after receiving forewarning, there is little incentive to improve their
detection capabilities. Moreover, the participants who are forewarned seems to
be more confident in their detection abilities. Their increased confidence in
audiovisual deepfake detection may stem from their increased awareness,
leading them to actively scrutinize the videos for signs of manipulation.

Figure 12: Distribution of confidence and accuracy in audiovisual deepfake
detection for two participant groups (forewarned vs. not forewarned). The
shape and width of the violin body provide insight into the density of data
points within each group, while in the middle of each density curve is a small
boxplot showing the median (dash) and the first and third quartiles at either
end.

###  IV-E Performance of AI Models

We evaluated 5 SOTA AI models on the same 40 videos used in the subjective
evaluation. All models were trained on the FakeAVCeleb dataset [36]. The
results are shown in Table II. The LipForensics model achieved an accuracy of
92.50%, correctly identifying 37 out of 40 samples. In comparison, the AV-Lip-
Sync model that considers audiovisual consistency achieved an accuracy of
87.50% by correctly identifying 35 out of 40 samples. The remaining three
models, AV-Lip-Sync++, CNN-Ensemble, and AVTENet, achieved an accuracy of
97.50%, correctly identifying 39 out of 40 samples.

TABLE II: Detection accuracy of AI models. Model |  | Accuracy (%)
---
LipForensics [41] | 92.50
AV-Lip-Sync [8] | 87.50
AV-Lip-Sync+ [6] | 97.50
CNN-Ensemble [7] | 97.50
AVTENet [9] | 97.50

The accuracy of LipForensics and AV-Lip-Sync is relatively low compared to the
other three AI models. This may be because the former is a unimodal method and
relies only on lip movements, which makes it insufficient for identifying
sophisticated audiovisual deepfakes since manipulation may occur in other
facial areas besides lips. In contrast, the latter focuses on audiovisual
speech-lip synchronization, but its performance may be affected by variations
in audio quality or other factors that impact the synchronization. AV-Lip-
Sync+, CNN-Ensemble, and AVTENet achieved the highest accuracy because these
models adopt a multimodal approach or utilize advanced sophisticated
techniques such as audiovisual synchronization, ensemble network, or
transformer architecture, which may enhance their ability to identify various
deepfakes.

Figure 13: Comparison of the performance of humans and 5 AI models in
audiovisual deepfake detection.

###  IV-F Human vs. AI Models

Fig. 13 compares the performance of humans and 5 AI models in detecting
audiovisual deepfakes. Human performance was well below expectations and
marginally above the chance level. However, all AI models performed
significantly better than human participants. Arguably, AI models have
demonstrated an impressive ability to detect subtle manipulations in
audiovisual content that might escape human perception. AI models excel humans
at integrating information from multiple modalities, and combining audio and
visual modalities in deepfake detection contributes significantly to the
effectiveness of AI models.

##  V Limitations

We are the first to offer insights into the study of human perception of
audiovisual deepfakes. In addition to its contribution, this study has several
limitations that can be mitigated in future studies to thoroughly explore this
field. First, our choice to use a sample size of 40 may not be sufficient to
draw general conclusions about human perception in audiovisual deepfake
detection tasks. Second, other interventions, such as control situations or
providing incentives, can be used to further explore human behavior. Third, to
expand the language range, experiments can be conducted in other languages.
Fourth, participants can be given multiple samples with different manipulation
types to evaluate their performance.

##  VI Conclusions and Future Work

In this paper, we present the results of a subjective study aimed at analyzing
human behavior in detecting audiovisual deepfake videos and comparing human
performance with that of five AI models. We tested 110 human subjects and five
AI models using the same set of 40 manually selected videos (20 real and 20
fake). The findings showed that subjects generally found detecting deepfakes
to be challenging. They performed marginally better than random chance, even
when explicitly informed of the presence of deepfakes. The concern is that
deepfake technology has reached a level of reality that could confuse much of
the public, especially social media users, while people are overconfident in
their ability to determine the authenticity of a video. However, over time,
participants seemed to refine their perception and develop a more accurate
understanding of their detection capabilities. Additionally, we found that
familiarizing participants with the threats posed by deepfakes had no
significant impact on their ability to detect deepfakes. Participants’ age,
gender, and whether they were native speakers of English had an impact on
their detection performance, but not their IT skill level. On the other hand,
AI models outperformed human participants at detecting forgeries in videos.
Furthermore, deepfake detection methods that integrate multiple modalities are
more effective than unimodal methods. In order to reduce the harm of deepfakes
to human society, it is crucial to develop deepfake detection technology.

To further expand and refine our understanding of human perception of
audiovisual deepfakes, we can implement a test design that measures subjective
performance in a more discreet and unbiased way. For example, the
applicability of our findings can be generalized by including a broader and
more diverse population. Detailed analysis of the auditory and visual
components of audiovisual signals will reveal how individuals perceive,
interpret and process audio and visual cues in multimodal deepfake videos.
Furthermore, exploring the impact of the popularity of people in videos (e.g.,
celebrities or politicians) may provide valuable insights into enhancing human
detection capabilities and understanding how prior knowledge affects the
perception of corresponding deepfake content.

## References

  * [1] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio, Generative adversarial nets, in: Proceedings of the Advances in Neural Information Processing Systems, 2014.
  * [2] D. P. Kingma, M. Welling, Auto-encoding variational bayes, in: Proceedings of the International Conference on Learning Representations, 2014.
  * [3] K. Prajwal, R. Mukhopadhyay, V. P. Namboodiri, C. Jawahar, A lip sync expert is all you need for speech to lip generation in the wild, in: Proceedings of the ACM International Conference on Multimedia, 2020, pp. 484–492.
  * [4] S. Choi, S. Han, D. Kim, S. Ha, Attentron: Few-shot text-to-speech utilizing attention-based variable-length embedding, arXiv preprint arXiv:2005.08484 (2020).
  * [5] W. Ahmad, I. Ali, A. Shahzad, A. Hashmi, F. Ghaffar, ResViT: A framework for deepfake videos detection, International Journal of Electrical and Computer Engineering Systems 13 (9) (2022) 807–813.
  * [6] S. A. Shahzad, A. Hashmi, Y.-T. Peng, Y. Tsao, H.-M. Wang, AV-Lip-Sync+: Leveraging AV-HuBERT to exploit multimodal inconsistency for video deepfake detection, arXiv preprint arXiv:2311.02733 (2023).
  * [7] A. Hashmi, S. A. Shahzad, W. Ahmad, C. W. Lin, Y. Tsao, H.-M. Wang, Multimodal forgery detection using ensemble learning, in: Proceedings of the Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, 2022, pp. 1524–1532.
  * [8] S. A. Shahzad, A. Hashmi, S. Khan, Y.-T. Peng, Y. Tsao, H.-M. Wang, Lip Sync Matters: A novel multimodal forgery detector, in: Proceedings of the Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, 2022, pp. 1885–1892.
  * [9] A. Hashmi, S. A. Shahzad, C.-W. Lin, Y. Tsao, H.-M. Wang, AVTENet: Audio-visual transformer-based ensemble network exploiting multiple experts for video deepfake detection, arXiv preprint arXiv:2310.13103 (2023).
  * [10] K. T. Mai, S. Bray, T. Davies, L. D. Griffin, Warning: humans cannot reliably detect speech deepfakes, Plos one 18 (8) (2023) e0285333.
  * [11] N. M. Müller, K. Pizzi, J. Williams, Human perception of audio deepfakes, in: Proceedings of the International Workshop on Deepfake Detection for Audio Multimedia, 2022, pp. 85–91.
  * [12] D. Prudkỳ, A. Firc, K. Malinka, Assessing the human ability to recognize synthetic speech in ordinary conversation, in: Proceedings of the International Conference of the Biometrics Special Interest Group, 2023, pp. 1–5.
  * [13] M. A. Lepori, C. Firestone, Can you hear me now? sensitive comparisons of human and machine perception, Cognitive Science 46 (10) (2022) e13191.
  * [14] G. Watson, Z. Khanjani, V. P. Janeja, Audio deepfake perceptions in college going populations, arXiv preprint arXiv:2112.03351 (2021).
  * [15] V. Schetinger, M. M. Oliveira, R. da Silva, T. J. Carvalho, Humans are easily fooled by digital images, Computers & Graphics 68 (2017) 142–151.
  * [16] S. D. Bray, S. D. Johnson, B. Kleinberg, Testing human ability to detect ‘deepfake’images of human faces, Journal of Cybersecurity 9 (1) (2023) tyad011.
  * [17] S. Zhou, M. Gordon, R. Krishna, A. Narcomey, L. F. Fei-Fei, M. Bernstein, HYPE: A benchmark for human eye perceptual evaluation of generative models, in: Proceedings of the Advances in Neural Information Processing Systems, Vol. 32, 2019.
  * [18] S. J. Nightingale, H. Farid, AI-synthesized faces are indistinguishable from real faces and more trustworthy, Proceedings of the National Academy of Sciences 119 (8) (2022) e2120481119.
  * [19] E. Preu, M. Jackson, N. Choudhury, Perception vs. Reality: understanding and evaluating the impact of synthetic image deepfakes over college students, in: Proceedings of the IEEE Ubiquitous Computing, Electronics & Mobile Communication Conference, IEEE, 2022, pp. 0547–0553.
  * [20] M. Groh, Z. Epstein, C. Firestone, R. Picard, Deepfake detection by human crowds, machines, and machine-informed crowds, Proceedings of the National Academy of Sciences 119 (1) (2022) e2110013119.
  * [21] P. Korshunov, S. Marcel, Subjective and objective evaluation of deepfake videos, in: Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, 2021, pp. 2510–2514.
  * [22] N. C. Köbis, B. Doležalová, I. Soraperra, Fooled twice: People cannot detect deepfakes but think they can, iScience 24 (11) (2021).
  * [23] R. Tahir, B. Batool, H. Jamshed, M. Jameel, M. Anwar, F. Ahmed, M. A. Zaffar, M. F. Zaffar, Seeing is believing: Exploring perceptual differences in deepfake videos, in: Proceedings of the Conference on Human Factors in Computing Systems, 2021, pp. 1–16.
  * [24] C. Allen, B. R. Payne, T. Abegaz, C. Robertson, What You See Is Not What You Know: Studying deception in deepfake video manipulation, Journal of Cybersecurity Education, Research and Practice 2024 (1) (2023) 1.
  * [25] M. Groh, Z. Epstein, C. Firestone, R. Picard, Comparing human and machine deepfake detection with affective and holistic processing, arXiv preprint arXiv:2105.06496 (2021).
  * [26] S. S. Prasad, O. Hadar, T. Vu, I. Polian, Human vs. automatic detection of deepfake videos over noisy channels, in: Proceedings of the IEEE International Conference on Multimedia and Expo, IEEE, 2022, pp. 1–6.
  * [27] B. Sivathanu, R. Pillai, M. Mahtta, A. Gunasekaran, All that glitters is not gold: a study of tourists’ visit intention by watching deepfake destination videos, Journal of Tourism Futures (ahead-of-print) (2023).
  * [28] C. Vaccari, A. Chadwick, Deepfakes and disinformation: Exploring the impact of synthetic political video on deception, uncertainty, and trust in news, Social Media+ Society 6 (1) (2020) 2056305120903408.
  * [29] S. Ahmed, Fooled by the fakes: Cognitive differences in perceived claim accuracy and sharing intention of non-political deepfakes, Personality and Individual Differences 182 (2021) 111074.
  * [30] I. Kaate, J. Salminen, S.-G. Jung, H. Almerekhi, B. J. Jansen, How do users perceive deepfake personas? investigating the deepfake user perception and its implications for human-computer interaction, in: Proceedings of the Conference of the Italian SIGCHI Chapter, 2023, pp. 1–12.
  * [31] J. Frank, F. Herbert, J. Ricker, L. Schönherr, T. Eisenhofer, A. Fischer, M. Dürmuth, T. Holz, A representative study on human detection of artificially generated media across countries, arXiv preprint arXiv:2312.05976 (2023).
  * [32] S. Ahmed, H. W. Chua, Perception and deception: Exploring individual responses to deepfakes across different modalities, Heliyon 9 (10) (2023).
  * [33] M. Groh, A. Sankaranarayanan, A. Lippman, R. Picard, Human detection of political deepfakes across transcripts, audio, and video, arXiv preprint arXiv:2202.12883 (2022).
  * [34] A. Eberl, J. Kühn, T. Wolbring, Using deepfakes for experiments in the social sciences-a pilot study, Frontiers in Sociology 7 (2022) 907199.
  * [35] A. Khodabakhsh, R. Ramachandra, C. Busch, Subjective evaluation of media consumer vulnerability to fake audiovisual content, in: Proceedings of the International Conference on Quality of Multimedia Experience, 2019, pp. 1–6.
  * [36] H. Khalid, S. Tariq, M. Kim, S. S. Woo, FakeAVCeleb: A novel audio-video multimodal deepfake dataset, in: Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, 2021.
  * [37] Y. Jia, Y. Zhang, R. Weiss, Q. Wang, J. Shen, F. Ren, P. Nguyen, R. Pang, I. Lopez Moreno, Y. Wu, et al., Transfer learning from speaker verification to multispeaker text-to-speech synthesis, in: Proceedings of the Advances in Neural Information Processing Systems, Vol. 31, 2018.
  * [38] K. Prajwal, R. Mukhopadhyay, V. P. Namboodiri, C. Jawahar, A lip sync expert is all you need for speech to lip generation in the wild, in: Proceedings of the ACM International Conference on Multimedia, 2020, pp. 484–492.
  * [39] I. Korshunova, W. Shi, J. Dambre, L. Theis, Fast face-swap using convolutional neural networks, in: Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 3677–3685.
  * [40] Y. Nirkin, Y. Keller, T. Hassner, FSGAN: Subject agnostic face swapping and reenactment, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 7184–7193.
  * [41] A. Haliassos, K. Vougioukas, S. Petridis, M. Pantic, Lips don’t lie: A generalisable and robust approach to face forgery detection, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 5039–5049.
  * [42] B. Shi, W.-N. Hsu, K. Lakhotia, A. Mohamed, Learning audio-visual speech representation by masked multimodal cluster prediction, in: Proceedings of the International Conference on Learning Representations, 2022.


# Differentiable Wavetable Synthesis

###### Abstract

Differentiable Wavetable Synthesis (DWTS) is a technique for neural audio
synthesis which learns a dictionary of one-period waveforms i.e. wavetables,
through end-to-end training. We achieve high-fidelity audio synthesis with as
little as 10 to 20 wavetables and demonstrate how a data-driven dictionary of
waveforms opens up unprecedented one-shot learning paradigms on short audio
clips. Notably, we show audio manipulations, such as high quality pitch-
shifting, using only a few seconds of input audio. Lastly, we investigate
performance gains from using learned wavetables for realtime and interactive
audio synthesis.

Index Terms—  Differentiable Digital Signal Processing, Wavetable Synthesis,
Differentiable Dictionaries

Fig. 1: Learned wavetables ordered with highest average attention weights
appearing first (normal English reading order). Wavetables of key harmonics
are highlighted: $f_{0}$ (red), $f_{1}$ (yellow), $f_{2}$ (purple) and $f_{3}$
(green). The remaining wavetables are data-driven combinations of higher
harmonics. The first two wavetables appear to be silence.

##  1 Introduction

Although machine learning (ML) has revolutionized modern audio synthesis with
unprecedented progress [1, 2, 3]; fast, robust and realtime neural audio
synthesis remains a challenge. Purely deep audio models require significant
parallelization [4] or custom kernels [3] for fast performance. Recent
techniques fusing differentiable signal processing with deep neural networks
(DNN) [5] have enabled efficient, interpretable and real-time use cases [6, 7]
with minimal additional engineering.

Wavetable synthesis (WTS) is well-suited to realtime synthesis of periodic and
quasi-periodic signals. Real-world objects that generate sound often exhibit
physics that are well described by harmonic oscillations. These include
vibrating strings, membranes, hollow pipes and human vocal chords [8]. By
using lookup tables composed of single-period waveforms, WTS can be as general
as additive synthesis whilst requiring less realtime computation [9]. We
demonstrate three key contributions to neural audio synthesis using this
approach: high fidelity synthesis, wavetables transferrable to other tasks and
an order of magnitude reduction in computational complexity.

##  2 Related Work

Wavetable Synthesis (WTS): Wavetable synthesis generates audio from a
collection of static, single-period waveforms called “wavetables”, that each
capture a unique harmonic spectrum. Wavetables are typically 256 - 4096
samples in length; a collection can contain a few to several hundred
wavetables depending on use case. Periodic waveforms are synthesized by
indexing into the wavetables as a lookup table and interpolating between
neighbouring samples. WTS was historically used on early hardware synthesizers
where memory and computing power were limited. Today, WTS continues to
underpin commercial sound design and synthesis tools due to its ability to
generate a wide variety of timbres coupled with efficient performance.
Wavetables are normally hand-crafted or extracted programatically from audio
spectra [9, 10]. In this work, we learn data-driven wavetables.

Differentiable dictionaries: ML models can be equipped with differentiable
memory or dictionaries that are learned end-to-end with model parameters.
Models can write to external memory to record and lookup changes in state
[11]. Discriminative models can learn a differentiable dictionary for improved
classification during inference [12]. Here, we formulate wavetables as a
differentiable dictionary optimized jointly with model parameters.

Differentiable Digital Signal Processing: DDSP [5] describes a family of
techniques utilizing strong inductive biases from DSP combined with modern ML.
Successful applications include audio synthesis [5], pitch detection [13],
artificial reverberations [14], IIR filters [7] and audio effect manipualtion
[15]. Building on these works, we add WTS as a new technique for generative
audio tasks.

##  3 Methods

###  3.1 Differentiable Wavetable Synthesizer (DWTS)

Wavetable as learnable dictionary: We define a learnable dictionary
$D=\\{{\bm{w}}_{i}\\}_{i}^{N}$ where $N$ is the number of wavetables and
${\bm{w}}_{i}\in\mathbb{R}^{L}$ denotes a one-cycle wavetable of length $L$.
When a wavetable begins and ends on different values, this discontinuity
causes synthesis artefacts. We append ${\bm{w}}_{i}[L+1]$ to ${\bm{w}}_{i}$
and set ${\bm{w}}_{i}[L+1]={\bm{w}}_{i}[0]$. A wavetable ${\bm{w}}_{i}$ now
contains $L+1$ elements with $L$ learnable parameters. During training, we
learn $D$ using gradient descent jointly with other parameters. During
inference, $D$ is frozen and treated as a traditional, static collection of
wavetables.

Time-varying attention: By sequentially morphing between wavetables, timbre
can be changed over time [10]. Inspired by [16] we generalize morphing as a
time-varying linear attention over all wavetables i.e. $\
c_{1}^{N},c_{2}^{N}...c_{T}^{N}$ where $N$ and $T$ are number of wavetables
and timesteps respectively with constraints $\sum_{i=1}^{N}c_{i}(n)=1$ and
$c_{i}(n)\geq 0$.

Phase: Our model “draws” wavetables directly in the time domain. Phase
relationships within and across wavetables can be controlled without needing
to coherently manage independent magnitudes and phases in the complex
frequency domain. This contrasts with [5], where the initial phase of all
harmonic components are fixed at 0.

Synthesis: At the heart of WTS is a phase accumulator [9]. Given an input
sequence of time-varying fundamental frequency $f_{0}(n)$ over discrete time
steps $n$, we can compute the instantaneous modulo phase $\tilde{\phi}$ by
integrating $f_{0}(n)$:

$\tilde{\phi}(n)=2\pi\sum_{m=0}^{n}f_{0}(m)\mod 2\pi.$ (1)

$\tilde{\phi}(n)$ is normalized into a fractional index
$\tilde{j}(n)=\frac{L}{2\pi}\tilde{\phi}(n)$. We synthesize the signal $x(n)$
by linearly combining wavetables ${\bm{w}}_{i}$ in $D$ via:

$x(n)=A(n)\sum_{i=1}^{N}c_{i}(n)\cdot\Phi({\bm{w}}_{i},\tilde{j}(n),\kappa),$
(2)

where $A(n)$ is a time-varying amplitude controlling the signal’s overall
amplitude and $c_{i}$ denotes the time-varying attention on ${\bm{w}}_{i}$.
$A(n)$ and $c_{i}(n)$ are constrained positive via a sigmoid. The function
$\Phi({\bm{w}}_{i},\tilde{j},\kappa)$ is a fractional indexing operator that
returns the $(\tilde{j})$-th element of the vector ${\bm{w}}_{i}$ by using an
interpolation kernel $\kappa$ to approximate ${\bm{w}}_{i}[\tilde{j}]$ when
$\tilde{j}$ is non integer. Whilst more sophisticated interpolation kernels
exist (cubic, spline etc.), we use linear interpolation for optimal realtime
performance. For readers unfamiliar with fractional indexing, please refer to
the online supplement.

Initialization: ${\bm{w}}_{i}$ is randomly initialized with a zero-centered
Gaussian distribution $\mathcal{N}(0,\sigma^{2})$. We empirically find using a
small $\sigma=0.01$ improves training.

Antialiasing: At high $f_{0}$, the frequencies of upper harmonics in a
wavetable can be above Nyquist and must be removed before lookup to prevent
aliasing. This filter also prevents high frequency components present in the
Gaussian noise initialization from causing aliasing at the start of training.
Without this frequency-dependent anti-aliasing filter, we found aliasing
artefacts alone prevented meaningful learning.

Table 1: Reconstruction error comparison on the Nsynth dataset. Compared to the SOTA additive synthesis approach [5], our wavetable synthesis approach achieves comparable or lower errors. DDSP Additive Synthesis | Wavetable Synthesis (Ours) with $N$=
---|---
5 | 10 | 20 | 100
0.5834 $\pm$ 0.0035 | 0.6448 $\pm$ 0.0041 | 0.5989 $\pm$ 0.0042 | 0.5712 $\pm$ 0.0037 | 0.5756 $\pm$ 0.0034

(a) Target

(b) Reconstructed

(c) Target

(d) Reconstructed

Fig. 2: Spectrograms of two target samples and their corresponding
reconstruction.

##  4 Experiments

We benchmark against the original DDSP autoencoder [5], where a DNN controls
an additive synth and filtered noise synth to produce harmonic and non-
harmonic components of audio respectively. We replace the additive synth with
our wavetable synth and use an identical filtered noise synth. Noise
generation is a stochastic process that must be modelled separately. Like [5],
we omit the optional reverb module when training on the NSynth dataset [17].

Model: We adopt an identical input tuple ($f_{0}(n),l(n),z(n)$) [5].
Fundamental frequency $f_{0}(n)$ in equation 1 is extracted by a pretrained
CREPE model [18] with fixed weights. Loudness $l(n)$ is an A-weighted log-
magnitude extracted deterministically from audio. The residual embedding
$z(n)$ is extracted from MFCC’s via an encoder. For direct comparison, we use
identical encoder and decoder architectures with approximately 7M parameters
in total.

Unlike the autoencoder in [5], our setup contains an additional $N\times L$
learnable parameters in the wavetable dictionary $D$ during training. During
inference however, the wavetables are frozen and the parameter counts near-
identical.

Loss: We use a multi-scale spectral loss similar to [5]:

$L_{\text{reconstruction}}=\sum_{i}||S_{i}-\hat{S}_{i}||_{1},$ (3)

where $S_{i}$ and $\hat{S}_{i}$ respectively denote magnitude spectrums of
target and synthesized audio, and $i$ denotes different FFT sizes. We found
the log term $||\log S_{i}-\log\hat{S}_{i}||_{1}$ caused training
instabilities and excluded it. This modification did not influence the quality
of synthesized audio.

Dataset We use the same subset of the NSynth dataset [17] in [5, 19, 20],
containing 70,000 mono 16kHz samples each 4 seconds long. The examples
comprise mostly of strings, brass, woodwinds and mallets. At 16kHz, a
wavetable length $L=512$ is enough to represent all harmonics at the lowest
fundamental frequency of interest (20Hz).

##  5 Results

###  5.1 Reconstruction Quality of DWTS

Table 1 reports the reconstruction error of the SOTA additive-based
autoencoder from [5] and the proposed DWTS-based autoencoder. We vary the
number of wavetables $N=5,10,20,100$. Our approach achieves the lowest
reconstruction error of 0.5712 using only 20 wavetables. At the expense of a
small reduction in quality compared to the baseline, $N$ can be low as 10. Fig
2 shows spectrograms of two samples and their matching reconstructions using
wavetables. We encourage readers to listen to the online
supplement111https://lamtharnhantrakul.github.io/diffwts.github.io/.

Crucially, the wavetables in $D$ form an alternative, compact set of basis
vectors spanning an $L$-dimensional space extracted directly from the data.
When $N$ is very small at 5, reconstruction suffers due to an insufficient
number of bases. 10-20 wavetables strike an optimal balance for the NSynth
dataset. Wavetables reduce the number of control dimensions by an order of
magnitude compared to the 100 sinusoids in an additive synth [5]. More
importantly, the extracted wavetables are an explicit dictionary that are
portable to other tasks. We show this property in later sections.

###  5.2 Visualizing Wavetables

Fig 1 shows learned wavetables from the NSynth dataset when $N=20$. Despite
being initialized with noise, the learned wavetables are smooth and diverse in
shape. They also match the physics of NSynth sounds. In acoustic instruments,
energy is focused on lower frequencies, particularly the first few harmonics,
compared to higher harmonics [21]. Wavetables in Fig 1 are ordered with
highest average attention weights appearing first. The wavetable highlighted
in red is a phase-shifted sinusoid of one period i.e. the fundamental
frequency $f_{0}$. Other key partials $f_{1}$, $f_{2}$ and $f_{3}$ are
highlighted in yellow, purple and green. The remaining wavetables are data-
driven combinations of higher harmonics, compactly summarizing in a single
wavetable entry what would have taken several sinusoidal components in an
additive synth [5] to represent. Fig 3 shows the attention weights over time
for five audio samples. The attention visibly shifts across wavetables to
output the desired spectrum.

Fig. 3: Visualization of the time-varying attention weights of 5 samples using
20 wavetables.

(a) Original

(b) Add Scratch

(c) Add Pretrain

(d) DWTS Scratch

(e) DWTS Pretrain

Fig. 4: Spectrograms of original audio (a) and synthesized samples from an
input $f_{0}(n)$ pitch shifted down by an octave (b-e)

The asymmetric wavetables are the result of complex behavior in magnitude and
phase. We found phase-locking wavetables to start and end at 0 deteriorated
performance. It suggests the model takes advantage of phase relationships
within and between wavetables. This precise control of wavetable phase will be
particularly valuable in future work exploring synthesis of stereo and
binaural audio [22].

###  5.3 One shot learning and audio manipulations

In domains such as Neural Machine Translation (NMT), word embeddings [23]
extracted from end-to-end language learning are useful for other language
tasks. Analogously, we reason our data-driven wavetables should be useful in
other scenarios like one-shot learning and data-efficient extrapolation.
Unlike an implicit multi-dimensional vector, wavetables are an explicit and
interpretable representation.

One-shot setup: Given only a single 4 second passage of saxophone from the
URMP dataset [24], we train a new autoencoder model initialized with
pretrained wavetables from Fig 1 (DWTS Pretrain). This model only outputs
time-varying attention weights, since the wavetables are now a fixed
dictionary lookup. We compare against three baselines: (1) additive-synth
autoencoder trained from scratch (Add Scratch), (2) finetuning an additive-
synth autoencoder pretrained on Nsynth (Add Pretrain) and (3) Wavetable-synth
autoencoder trained from scratch (DWTS Scratch).

Pitch extrapolation: While all models achieve identical high quality one-shot
reconstructions of the saxophone segment, only DWTS Pretrain is robust to
overfitting during extrapolation. Fig 4 shows how all baselines exhibit high
frequency artefacts when input $f_{0}(n)$ is shifted. DWTS+Pretrain remains
artefact free even at extreme shifts, such as 3 octaves below the original
sample.

We repeat with a 4 second piano passage. A piano is challenging to model due
to the presence of both many harmonics and percussive hammer hits [21]. We
also compare against the Librosa library pitch shift function based on
traditional DSP [25]. When resynthesizing the segment 3 octaves down, DWTS
Pretrain is the only method that preserves the hammer’s percussive impact and
independently shifts harmonic components. Librosa pitch shift loses the
transient impact completely.

Connection to PSOLA: In this scenario, we hypothesize DWTS Pretrain approaches
an optimal Pitch Synchronous Overlap and Add (PSOLA) algorithm [26]. PSOLA
windows a single cycle of the original waveform in the time domain, re-
patching and overlapping these windows at the new pitch. Imperfections in this
windowing and overlapping process can cause artefacts. DWTS Pretrain
guarantees single-cycle waveforms in $D$, where re-pitching is trivially
achieved using a slower phase accumulator $\tilde{\phi}(n)$ reading through a
wavetable ${\bm{w}}_{i}$. This opens up applications like data-efficient
neural sampling, pitch correction and efficient polyphony using multiple phase
accumulators. We leave wavetables extracted from speech or singing voice for
future work.

###  5.4 Computational Complexity

Realtime performance is determined by many factors including algorithmic
complexity, low level optimizations, target hardware and the C++ ML framework
used for matrix operations. Here, we consider only the key difference between
the additive synth-based pipeline in [5], which already runs realtime [6], and
our new approach.

A.) Additive: a bank of 100 sinusoids where harmonic coefficients are updated
at 250 frames per second (FPS) [5]

B.) DWTS: 10 pre-learned wavetables where wavetable weights are also updated
at 250 FPS.

The remaining elements of each approach are assumed identical in specification
and performance. There are two clear areas of improvement with our new method.
Firstly, DWTS requires only ten interpolated wavetable read operations per
sample, compared to the 100 required for Additive. Secondly, both pipelines
require frame-wise linear smoothing of control data to avoid unwanted
artefacts in the synthesized signals. Additive requires all 100 sinusoidal
amplitude coefficients to be smoothed per sample, whereas DWTS requires
smoothing on only 10. We confirm this on a 2.6GHz 2019 Macbook Pro. Over 10k
trials, Additive takes on average 251.32ms to generate all 250 frames in 1
second of audio, whereas DWTS takes only 20.19ms.

##  6 Conclusion

In this paper, we presented a differentiable wavetable synthesizer capable of
high fidelity neural audio synthesis. We demonstrate how explicit and
learnable wavetables offer many advantages including robust one-shot learning
and an order of magnitude reduction in computational complexity. The approach
opens up applications such as data-efficient neural audio sampling and pitch-
shifting.

## References

  * [1] Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew W. Senior, and Koray Kavukcuoglu,  “Wavenet: A generative model for raw audio,”  CoRR, vol. abs/1609.03499, 2016.
  * [2] Yuxuan Wang, R. J. Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J. Weiss, N. Jaitly, Z. Yang, Y. Xiao, Z. Chen, S. Bengio, Q. V. Le, et al.,  “Tacotron: A fully end-to-end text-to-speech synthesis model,”  CoRR, vol. abs/1703.10135, 2017.
  * [3] Nal Kalchbrenner, Erich Elsen, Karen Simonyan, S. Noury, N. Casagrande, E. Lockhart, F. Stimberg, Aäron van den Oord, Sander Dieleman, and K. Kavukcuoglu,  “Efficient neural audio synthesis,”  ICML, 2018.
  * [4] Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu,  “Fastspeech 2: Fast and high-quality end-to-end text to speech,” 2021\.
  * [5] Jesse Engel, Lamtharn Hantrakul, Chenjie Gu, and Adam Roberts,  “Ddsp: Differentiable digital signal processing,”  ICLR, 2020.
  * [6] Francesco Ganis, Erik Frej Knudesn, Søren VK Lyster, Robin Otterbein, David Südholt, and Cumhur Erkut,  “Real-time timbre transfer and sound synthesis using ddsp,”  arXiv preprint arXiv:2103.07220, 2021.
  * [7] Julian D. Parker Boris Kuznetsov and Fabián Esqueda,  “Differentiable iir filters for machine learning applications,”  DaFX, 2020.
  * [8] Julius Smith,  Physical Audio Signal Processing: for Virtual Musical Instruments and Digital Audio Effects,  01 2006.
  * [9] Robert Bristow-Johnson,  “Wavetable synthesis 101, a fundamental perspective,”  in Audio Engineering Society Convention 101. Audio Engineering Society, 1996.
  * [10] Andrew B. Horner and James W. Beauchamp,  Instrument Modeling and Synthesis, pp. 375–397,  Springer New York, New York, NY, 2008.
  * [11] Alex Graves, Greg Wayne, and Ivo Danihelka,  “Neural turing machines,”  arXiv:1410.5401, 2014.
  * [12] Siyuan Shan, Yang Li, and Junier B Oliva,  “Meta-neighborhoods,”  NeurIPS, 2020.
  * [13] Jesse Engel, Lamtharn Hantrakul, Rigel Swavely, Adam Roberts, and Curtis Glenn-Macway Hawthorne,  “Self-supervised pitch detection by inverse audio synthesis,”  in ICML SAS Workshop, 2020.
  * [14] Sungho Lee, Hyeong-Seok Choi, and Kyogu Lee,  “Differentiable artificial reverberation,”  CoRR, vol. abs/2105.13940, 2021.
  * [15] Marco A Martínez Ramírez, Oliver Wang, Paris Smaragdis, and Nicholas J Bryan,  “Differentiable signal processing with black-box audio effects,”  in ICASSP, 2021, pp. 66–70.
  * [16] Andrew Horner,  “Wavetable matching synthesis of dynamic instruments with genetic algorithms,”  vol. 43, pp. 916–931, 11 1995.
  * [17] Jesse Engel, Cinjon Resnick, Adam Roberts, Sander Dieleman, Mohammad Norouzi, Douglas Eck, and Karen Simonyan,  “Neural audio synthesis of musical notes with wavenet autoencoders,”  in ICML, 2017.
  * [18] Jong Wook Kim, Justin Salamon, Peter Li, and Juan Pablo Bello,  “Crepe: A convolutional representation for pitch estimation,”  in ICASSP, 2018.
  * [19] Lamtharn Hantrakul, Jesse H Engel, Adam Roberts, and Chenjie Gu,  “Fast and flexible neural audio synthesis.,”  in ISMIR, 2019.
  * [20] Jesse H. Engel, Kumar Krishna Agrawal, Shuo Chen, I. Gulrajani, C. Donahue, and Adam Roberts,  “Gansynth: Adversarial neural audio synthesis,”  ICLR, 2019.
  * [21] Thomas D. Rossing and Neville H. Fletcher,  Physics of Musical Instruments 2nd Edition,  Springer New York, New York, NY, 1998.
  * [22] Alexander Richard, Dejan Markovic, Israel D Gebru, Steven Krenn, Gladstone Alexander Butler, Fernando Torre, and Yaser Sheikh,  “Neural synthesis of binaural speech from mono audio,”  in ICLR, 2020.
  * [23] Tomas Mikolov, I. Sutskever, K. Chen, Greg S Corrado, and J Dean,  “Distributed representations of words and phrases and their compositionality,”  in NIPS, 2013.
  * [24] Bochen Li, Xinzhao Liu, K. Dinesh, Z. Duan, and Gaurav Sharma,  “Creating a multitrack classical music performance dataset for multimodal music analysis: Challenges, insights, and applications,”  IEEE Transactions on Multimedia, vol. 21, no. 2, pp. 522–535, 2018\.
  * [25] Brian McFee, Colin Raffel, Dawen Liang, Daniel P.W. Ellis, M. McVicar, E. Battenberg, and O. Nieto,  “librosa: Audio and Music Signal Analysis in Python,”  in Proceedings of the 14th Python in Science Conference, 2015, pp. 18 – 24.
  * [26] F. Charpentier and M. Stella,  “Diphone synthesis using an overlap-add technique for speech waveforms concatenation,”  in ICASSP, 1986, vol. 11, pp. 2015–2018.


# PanGEA: The Panoramic Graph Environment Annotation Toolkit

Alexander Ku Peter Anderson11footnotemark: 1 Jordi Pont-Tuset Jason Baldridge
Google Research
{alexku, pjand, jponttuset<EMAIL_ADDRESS>First two authors contributed
equally.

###### Abstract

PanGEA, the Panoramic Graph Environment Annotation toolkit, is a lightweight
toolkit for collecting speech and text annotations in photo-realistic 3D
environments. PanGEA immerses annotators in a web-based simulation and allows
them to move around easily as they speak and/or listen. It includes database
and cloud storage integration, plus utilities for automatically aligning
recorded speech with manual transcriptions and the virtual pose of the
annotators. Out of the box, PanGEA supports two tasks – collecting navigation
instructions and navigation instruction following – and it could be easily
adapted for annotating walking tours, finding and labeling landmarks or
objects, and similar tasks. We share best practices learned from using PanGEA
in a 20,000 hour annotation effort to collect the Room-Across-Room dataset. We
hope that our open-source annotation toolkit and insights will both expedite
future data collection efforts and spur innovation on the kinds of grounded
language tasks such environments can support.

##  1 Introduction

The release of high-quality 3D building and street captures Chang et al.
(2017); Mirowski et al. (2019); Mehta et al. (2020); Xia et al. (2018); Straub
et al. (2019) has galvanized interest in developing embodied navigation agents
that can operate in complex human environments. Based on these environments,
annotations have been collected for a variety of tasks including navigating to
a particular class of object (ObjectNav) Batra et al. (2020), navigating from
language instructions aka vision-and-language navigation (VLN) Anderson et al.
(2018b); Chen et al. (2019); Qi et al. (2020); Ku et al. (2020), and vision-
and-dialog navigation Thomason et al. (2020); Hahn et al. (2020). To date,
most of these data collection efforts have required the development of custom
annotation tools.

To expedite future data collection efforts, in this paper we introduce PanGEA,
an open-sourced annotation toolkit designed for these
settings.111github.com/google-research/pangea Specifically, PanGEA assumes an
environment represented by discrete navigation graphs connecting high-
resolution 360° panoramas, where each node represents a unique viewpoint in
the environment and actions involve moving between these viewpoints. Examples
of suitable environments include the indoor buildings from Matterport3D Chang
et al. (2017) (using the navigation graphs from Anderson et al. (2018b)) and
the street-level environments from StreetLearn Mirowski et al. (2019).

Figure 1: Screenshots of the PanGEA Guide and Follower interfaces. In the
Guide task (left), Guides explore a given path while attempting to create a
navigation instruction for others to follow. Guides can pause and restart the
audio recording at any time. After recording is completed, Guides transcribe
their own audio. In the Follower task (right), annotators listen to a Guide’s
instructions and attempt to follow the intended path. Followers can skip
around the Guide’s audio using the audio waveform at bottom right. In both
tasks, PanGEA tracks the annotators virtual camera pose and automatically
aligns it with the Guide’s audio transcript.

Out of the box, PanGEA supports two annotation modes: the Guide task and the
Follower task. In the Guide task, Guides look around and move through an
environment to follow a pre-defined path and attempt to create a navigation
instruction for others to follow. In the Follower task, annotators listen to a
Guide’s instructions and attempt to follow the path. These annotation modes
are based on the Vision-and-Language Navigation (VLN) setting proposed by
Anderson et al. (2018b). However, compared to similar annotation tools, PanGEA
includes substantial additional capabilities, notably:

  * •

annotation via voice recording (in addition to text entry)

  * •

virtual pose tracking to record what annotators look at

  * •

utilities for aligning a transcript of the words heard or uttered by each
annotator with their visual perceptions and actions

  * •

integration with cloud database and storage platforms

  * •

a modular API facilitating easy extension to new tasks and new environments

PanGEA has already been used in two papers. It was used to collect Room-
Across-Room (RxR) Ku et al. (2020), a dataset of human-annotated navigation
instructions in English, Hindi and Telugu which is the largest VLN dataset by
an order of magnitude. PanGEA was also used to perform human evaluations of
model-generated navigation instructions in Zhao et al. (2021). It could be
trivially adapted to other tasks that combine annotation with movement, such
as annotating walking tours, or finding and labeling particular landmarks or
objects.

We next describe PanGEA’s capabitilities in more detail. In the final section
we share some best practices learned from using PanGEA to collect RxR, which
required more than 20,000 annotation hours.

##  2 PanGEA Toolkit

Figure 2: PanGEA time-aligns each annotators manual audio transcription
(middle) to a pose trace recording their virtual camera movements (bottom).
This is achieved by first generating a noisy-but-timestamped automatic
transcription (top), which is aligned with the manual transcription using
dynamic time warping in order to propagate timestamps to the manual
transcription. Figure adapted from Pont-Tuset et al. (2020)

### Guide Task

In the Guide task (Figure 1, left), Guides look around and move to explore an
environment while recording an audio narration. For the RxR data collection,
the Guide’s movement was restricted to a particular path through the
environment, and annotators were instructed to record navigation instructions
that would be sufficiently descriptive for others to follow the same path.
However, this restriction can be relaxed to allow free movement and narration
for other purposes. Once the Guide is satisfied with their recording, they are
asked to manually transcribe their own voice recording into text. This ensures
high quality transcription results.

During the Guide task, in parallel to the annotator’s voice recording, PanGEA
captures a timestamped record of the annotator’s virtual camera movements,
which we call a pose trace. By default, PanGEA is configured to use
Firebase222https://firebase.google.com, saving the Guide’s audio recording to
a cloud storage bucket, and the transcript, pose trace and other metadata to a
cloud database for post processing. Inspired by Localized Narratives Pont-
Tuset et al. (2020), PanGEA includes a utility to automatically align each
Guide’s pose trace with the manual transcript of their audio recording. This
is achieved by using a Speech to Text
service333https://cloud.google.com/speech-to-text to first generate a noisy-
but-timestamped automatic transcription. PanGEA then using dynamic time
warping to align tokens in the automatic transcript to the manual transcript
before propagating timestamps from the automatic to the manual transcription
(Figure 2). The result is fine-grained synchronization between the transcribed
text, the pixels seen, and the actions taken by the Guide.

### Follower Task

In the Follower task (Figure 1, right), Followers begin at a specified
starting point in an environment and are asked to follow a Guide’s
instructions. They observe the environment and navigate as the Guide’s audio
plays. Followers can skip forward or backward in the audio recording by
clicking on an audio waveform representation of the Guide’s recording. This
allows them to skip over periods of silence or to listen to part of the audio
again. Once the Follower believes they have reached the the end of the path,
or they give up, they indicate they are done and the task ends. Note that
although the Follower task supports audio instructions, it can be easily
adapted to replace the audio instruction with a textual instruction. This was
the approach taken by Zhao et al. (2021).

As with the Guide task, the Follower’s pose trace is recorded and saved to a
cloud database, along with the timestamp of the Guide’s audio that the
Follower listened to at each moment. This allows the Follower’s visual
percepts and actions to be accurately aligned with text tokens in the Guide’s
instructions. Similarity between the annotated (Guide) path and the Follower
path is also a natural measure of the joint quality of both the Guide and the
Follower annotations. In the experiments for RxR, the path extracted from the
Follower’s pose trace was also used as additional supervision when training
Follower agents, since it represents a step-by-step account of how a human
solved the task and the visual inputs they focused on in order to do so Ku et
al. (2020).

### Deployment

PanGEA comes with several demos using a very simplistic environment. To deploy
PanGEA for a new large-scale collection effort requires completing 3 main
steps:

  * •

Creating a new app in Firebase to initialize the cloud storage and cloud
database,

  * •

Setting up an appropriate crowdsourcing platform to serve the PanGEA front-end
to a pool of annotators, and

  * •

Setting up the environment to be used, e.g., hosting the images and navigation
graphs in a storage bucket in an appropriate format.

Further details are provided in the PanGEA readme.

##  3 Observations and Best Practices

PanGEA was developed for the collection of the RxR dataset, a 20,000+ hour
annotation effort based on Matterport3D indoor scenes. Many of the lessons
learned during this collection effort are codified in the PanGEA toolkit. For
example, we found that uploading recorded audio at the end of the Guide task
was time consuming, and so in the final version of PanGEA the wav file is
uploaded in the background while the annotator is busy transcribing their
audio. We also found that audio annotations could include long periods of
silence, so we provided Follower annotators with an audio waveform
visualization and an interface to skip over silence. Some other observations
and best practices for reducing annotation times and improving annotation
quality are shared in this section.

### Annotators Complete Tasks in Creative Ways

PanGEA is designed to capture the alignment between annotators’ visual
percepts, actions and utterances to provide fine-grained spatio-temporal
grounding. In initial trials with PanGEA, we found that some annotators – with
the best of intentions – completely undermined this paradigm. We had
envisioned them speaking while moving and looking at the environment; however,
in an effort to generate more fluent instructions, some annotators first
explored the environment while drafting a navigation instruction separately in
a text editor. Then, having finalized the textual instruction, the annotator
read it all at the end of the audio recording. While this strategy indeed
produced high-quality navigation instructions, the instructions were no longer
time-aligned to the pose trace. Interestingly, the language used in the
instructions also differed. Instructions drafted as text tended to use more
connective phrases — for example, “turn right and then you will see a dining
table” instead of “turn right… now you see a dining table”. We found it
challenging to add guardrails in PanGEA that could prevent this behaviour
without unduly restricting the freedom of the annotators and the flexibility
of the toolkit. Instead, we addressed this issue–successfully–via explicit
training.

### Annotator Training

To overcome the aforementioned issue and to improve annotation quality in
general, for RxR, we conducted an interactive virtual training session with
annotators, providing examples of ideal annotations and various failure modes.
Annotators were also able to ask questions regarding how to best complete the
tasks assigned to them. Although interactive training sessions are not always
possible, at minimum we recommend providing annotators with a training video
that shows a walk-through of the task and notes common pitfalls to avoid. We
provide links to the demo videos for the RxR Guide
task444https://youtu.be/aJkJfB8oI2M and Follower task555https://youtu.be/vcP-
oX1t0CU (initially called the Tourist task).

### Pilot Collections and Learning Periods

Annotating and following navigation instructions in a virtual world is a
complex task. We recommend allowing for several small-scale pilot data
collections to identify issues with the collection process. This includes
having the team creating the dataset perform the tasks using the tool.
Secondly, we recommend allowing for a learning period whenever a new annotator
is introduced to the task, i.e., planning to discard the first 5–10
annotations produced by a new annotator. We found that rotating annotators
between both Guide and Follower tasks early in their experience improved
annotation quality because doing so provides much greater awareness of the
needs of Followers when completing the Guide tasks.

### Data Monitoring Dashboard

We recommend using VLN evaluation metrics such as success rate, navigation
error and SPL Anderson et al. (2018a) (or similar metrics for alternative
tasks) to continually monitor the quality of the collected Guide navigation
instructions and Follower paths. By storing the collected annotations in
Firebase, it is relatively easy to construct web-based interfaces to monitor
these metrics. In the case of RxR, we created a monitoring dashboard that
displayed success rates for each annotator pool and also each annotator, with
the capability to replay the pose traces from individual Guide and Follower
annotations. Annotators were able to see an anonymized view on their progress
as it related to others, which helped them assess whether they were performing
the task correctly or needed additional changes and perhaps explicit guidance.

### Speech versus Writing

In tasks that require a person to perform actions while producing or
comprehending language, it is much easier if people are allowed to use speech
rather than writing because it allows them to use their hands and eyes fully
for performing actions. This has very real consequences for thinking about
future data collection efforts. Speech interactions will be essential for any
tasks that include time pressure, such as collaborative games where players
use language to coordinate. There is also a simple but significant cost
advantage: on average, the transcription portion for an RxR Guide annotation
took three to four times longer than collecting the speech instruction itself,
so either a great deal more instructions could have been collected, or the
cost could have been significantly reduced. Speech also encodes intonation and
is more likely to elicit interesting dialectal differences. For these and
other reasons, we may thus want to encourage more research that works on
language grounding tasks that work with speech directly, and provide current
best automatic speech recognition output for those who insist on working with
text only.

##  4 Future Applications

There are many potential future applications of PanGEA and tools that could be
built based on the design decisions discussed above. We are particularly
excited about multi-agent problems that collect pose traces from multiple
participants as they coordinate via language, such as hide and seek games or
tasks where items must be moved from one location to another to satisfy goals
or solve puzzles, similar to CerealBar Suhr et al. (2019).

## References

  * Anderson et al. (2018a) Peter Anderson, Angel Chang, Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi, Manolis Savva, and Amir R. Zamir. 2018a.  On evaluation of embodied navigation agents.  _arXiv preprint arXiv:1807.06757_.
  * Anderson et al. (2018b) Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. 2018b.  Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments.  In _CVPR_ , pages 3674–3683.
  * Batra et al. (2020) Dhruv Batra, Aaron Gokaslan, Aniruddha Kembhavi, Oleksandr Maksymets, Roozbeh Mottaghi, Manolis Savva, Alexander Toshev, and Erik Wijmans. 2020.  Objectnav revisited: On evaluation of embodied agents navigating to objects.  _arXiv preprint arXiv:2006.13171_.
  * Chang et al. (2017) Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. 2017.  Matterport3d: Learning from RGB-D data in indoor environments.  _International Conference on 3D Vision (3DV)_.
  * Chen et al. (2019) Howard Chen, Alane Suhr, Dipendra Misra, Noah Snavely, and Yoav Artzi. 2019.  Touchdown: Natural language navigation and spatial reasoning in visual street environments.  In _CVPR_.
  * Hahn et al. (2020) Meera Hahn, Jacob Krantz, Dhruv Batra, Devi Parikh, James M. Rehg, Stefan Lee, and Peter Anderson. 2020.  Where are you? localization from embodied dialog.  In _EMNLP_.
  * Ku et al. (2020) Alexander Ku, Peter Anderson, Roma Patel, Eugene Ie, and Jason Baldridge. 2020.  Room-across-room: Multilingual vision-and-language navigation with dense spatiotemporal grounding.  _EMNLP_.
  * Mehta et al. (2020) Harsh Mehta, Yoav Artzi, Jason Baldridge, Eugene Ie, and Piotr Mirowski. 2020.  Retouchdown: Adding touchdown to streetlearn as a shareable resource for language grounding tasks in street view.  _EMNLP Workshop on Spatial Language Understanding (SpLU)_.
  * Mirowski et al. (2019) Piotr Mirowski, Andras Banki-Horvath, Keith Anderson, Denis Teplyashin, Karl Moritz Hermann, Mateusz Malinowski, Matthew Koichi Grimes, Karen Simonyan, Koray Kavukcuoglu, Andrew Zisserman, et al. 2019.  The streetlearn environment and dataset.  _arXiv preprint arXiv:1903.01292_.
  * Pont-Tuset et al. (2020) Jordi Pont-Tuset, Jasper Uijlings, Soravit Changpinyo, Radu Soricut, and Vittorio Ferrari. 2020.  Connecting vision and language with localized narratives.  In _ECCV_.
  * Qi et al. (2020) Yuankai Qi, Qi Wu, Peter Anderson, Xin Wang, William Yang Wang, Chunhua Shen, and Anton van den Hengel. 2020.  Reverie: Remote embodied visual referring expression in real indoor environments.  In _CVPR_.
  * Straub et al. (2019) Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob J. Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, Anton Clarkson, Mingfei Yan, Brian Budge, Yajie Yan, Xiaqing Pan, June Yon, Yuyang Zou, Kimberly Leon, Nigel Carter, Jesus Briales, Tyler Gillingham, Elias Mueggler, Luis Pesqueira, Manolis Savva, Dhruv Batra, Hauke M. Strasdat, Renzo De Nardi, Michael Goesele, Steven Lovegrove, and Richard Newcombe. 2019.  The Replica dataset: A digital replica of indoor spaces.  _arXiv preprint arXiv:1906.05797_.
  * Suhr et al. (2019) Alane Suhr, Claudia Yan, Jack Schluger, Stanley Yu, Hadi Khader, Marwa Mouallem, Iris Zhang, and Yoav Artzi. 2019.  Executing instructions in situated collaborative interactions.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2119–2130, Hong Kong, China. Association for Computational Linguistics.
  * Thomason et al. (2020) Jesse Thomason, Michael Murray, Maya Cakmak, and Luke Zettlemoyer. 2020.  Vision-and-dialog navigation.  In _Conference on Robot Learning (CoRL)_ , pages 394–406. PMLR.
  * Xia et al. (2018) Fei Xia, Amir R. Zamir, Zhi-Yang He, Alexander Sax, Jitendra Malik, and Silvio Savarese. 2018.  Gibson env: real-world perception for embodied agents.  In _CVPR_. IEEE.
  * Zhao et al. (2021) Ming Zhao, Peter Anderson, Vihan Jain, Su Wang, Alex Ku, Jason Baldridge, and Eugene Ie. 2021.  On the evaluation of vision-and-language navigation instructions.  In _Conference of the European Chapter of the Association for Computational Linguistics (EACL)_.