#  _Do You Really Mean That?_ Content Driven Audio-Visual Deepfake Dataset and
Multimodal Method for Temporal Forgery Localization

Zhixi Cai1, Kalin Stefanov1, Abhinav Dhall2,1 and Munawar Hayat1 zhixi.cai,
kalin.stefanov<EMAIL_ADDRESS><EMAIL_ADDRESS>1Monash
University and 2 Indian Institute of Technology Ropar1Australia and 2 India

###### Abstract.

Due to its high societal impact, deepfake detection is getting active
attention in the computer vision community. Most deepfake detection methods
rely on identity, facial attribute and adversarial perturbation based spatio-
temporal modifications at the whole video or random locations, while keeping
the meaning of the content intact. However, a sophisticated deepfake may
contain only a small segment of video/audio manipulation, through which the
meaning of the content can be, for example, completely inverted from sentiment
perspective. To address this gap, we introduce a content driven audio-visual
deepfake dataset, termed as Localized Audio Visual DeepFake (LAV-DF),
explicitly designed for the task of learning temporal forgery localization.
Specifically, the content driven audio-visual manipulations are performed at
strategic locations in order to change the sentiment polarity of the whole
video. Our baseline method for benchmarking the proposed dataset is a 3DCNN
model, termed as Boundary Aware Temporal Forgery Detection (BA-TFD), which is
guided via contrastive, boundary matching and frame classification loss
functions. Our extensive quantitative analysis demonstrates the strong
performance of the proposed method for both tasks of temporal forgery
localization and deepfake detection.

Datasets, Deepfake. Localization, Detection

##  1\. Introduction

Advances in computer vision and deep learning methods (e.g. Autoencoders
(Rumelhart et al., 1985) and Generative Adversarial Networks (Goodfellow et
al., 2020)) have enabled the creation of very realistic fake videos, known as
deepfakes111In the text, deepfake and forgery are used interchangeably.. There
are various ways of creating deepfakes, including voice cloning (Wang et al.,
2017; Jia et al., 2018), face reenactment (Tulyakov et al., 2018; Prajwal et
al., 2020), and face swapping (Korshunova et al., 2017; Nirkin et al., 2019).
Highly realistic deepfakes are a potential tool for spreading harmful
misinformation given our increasing online presence. This success in
generating high quality deepfakes has raised serious concerns about their role
in shaping people’s beliefs with some scholars suggesting that deepfakes are a
“threat to democracy” (Schwartz, 2018; Brandon, 2019; Sample, 2020; Thomas,
2020). As an example of the potentially harmful effect of deepfakes, consider
the recent work (Thies et al., 2020) that uses video of the former United
States President Barack Obama to showcase a novel face reenactment method. In
this work, the lip movements of Barack Obama are synchronised with another
person’s speech, resulting in high quality and realistic video in which the
former president appears to say something he never did. Given the recent surge
in synthesized fake video content on the Internet, it has become increasingly
important to identify deepfakes with more accurate and reliable methods. This
has led to the release of several benchmark datasets (Korshunov and Marcel,
2018; Rossler et al., 2019; Dolhansky et al., 2020) and methods (Mirsky and
Lee, 2021) for fake content detection. These fake video detection methods aim
to correctly classify any given input video as either real or fake. This
suggests that the major assumption behind those datasets and methods is that
fake content is present in the entirety of the video/audio signal, that is,
there is some form of manipulation throughout the content. And current state-
of-the-art deepfake detection methods (Coccomini et al., 2021; Heo et al.,
2021; Wodajo and Atnafu, 2021) achieve impressive results on this problem
using the largest benchmark datasets.

Figure 1. Content driven audio-visual manipulation. On the left is a real
video with the subject saying “Vaccinations are safe”. On the right is audio-
visual deepfake created from the real video based on the change in perceived
sentiment where “safe” is changed to “dangerous”. Green-edge and red-edge
images are real and fake frames, respectively. Through a subtle audio-visual
manipulation, the complete meaning of the video content has changed.

However, fake content might constitute only small part of an otherwise long
real video, as it was initially suggested in (Chugh et al., 2020). Such short
modified segments have the power to completely alter the meaning and sentiment
of the original real content. As an example, consider the manipulation
illustrated in Figure 1. The real video might represent a person saying
“Vaccinations are safe”, while the fake includes only a short modified
segment, for example “safe” is replaced with “dangerous”. Hence, the meaning
and sentiment of the fake video are significantly different from the real.
This type of coordinated manipulation, if done precisely, has the potential to
sway public opinion (e.g. when employed for media of a popular person as the
example with Barack Obama) in a particular direction, for example, based on
target sentiment polarity. Given the discussed major assumption behind current
datasets and methods, the state-of-the-art deepfake detectors might not
perform well on this type of manipulations.

Inspired by this idea and observation, in this paper we tackle the identified
important task of detecting content altering fake segments in videos. The
literature review on benchmark datasets for deepfake detection indicates that
there is no dataset suitable for this task, that is, dataset that consists of
content driven manipulations. Therefore, in this paper, we describe the
process of creating such large-scale dataset that will enable further research
in this important direction. In addition, we propose a novel multimodal method
for precise prediction of the boundaries of fake segments based on visual and
audio information. The main contributions of our work are as follows:

  1. (1)

We introduce a new large-scale public audio-visual dataset, called Localized
Audio Visual DeepFake (LAV-DF), explicitly designed for the task of temporal
forgery localization of content driven manipulations.

  2. (2)

We propose a new multimodal method, called Boundary Aware Temporal Forgery
Detection (BA-TFD) for audio-visual temporal forgery localization of content
driven manipulations.

Table 1. Quantitative comparison of LAV-DF with previous public deepfake
datasets. Cla: Classification, SL: Spatial Localization, TFL: Temporal Forgery
Localization, FS: Face Swapping, and RE: ReEnactment.

Dataset |  Year |  Tasks |  Manipulated |  Method |  #Subjects |  #Real |  #Fake |  #Total
---|---|---|---|---|---|---|---|---
|  |  |  Modality |  Manipulation |  |  |  |
DF-TIMIT (Korshunov and Marcel, 2018) | 2018 | Cla | V | FS | 43 | 320 | 640 | 960
UADFV (Yang et al., 2019) | 2019 | Cla | V | FS | 49 | 49 | 49 | 98
FaceForensics++ (Rossler et al., 2019) | 2019 | Cla | V | FS/RE | - | 1,000 | 4,000 | 5,000
Google DFD (Nick and Andrew, 2019) | 2019 | Cla | V | FS | - | 363 | 3,068 | 3,431
DFDC (Dolhansky et al., 2020) | 2020 | Cla | AV | FS | 960 | 23,654 | 104,500 | 128,154
DeeperForensics (Jiang et al., 2020) | 2020 | Cla | V | FS | 100 | 50,000 | 10,000 | 60,000
Celeb-DF (Li et al., 2020) | 2020 | Cla | V | FS | 59 | 590 | 5,639 | 6,229
WildDeepfake (Zi et al., 2020) | 2021 | Cla | - | - | - | 3,805 | 3,509 | 7,314
FakeAVCeleb (Khalid et al., 2021b) | 2021 | Cla | AV | RE | 600$+$ | 570 | 25,000$+$ | 25,500$+$
ForgeryNet (He et al., 2021) | 2021 | SL/TFL/Cla | V | Random FS/RE | 5400$+$ | 99,630 | 121,617 | 221,247
LAV-DF (Ours) |  2022 |  TFL/Cla |  AV | Content driven RE |  153 |  36,431 |  99,873 |  136,304

##  2\. Related Works

Deepfake Datasets. The body of research in deepfake detection is driven by
seminal datasets curated with different manipulation methods. A summary of the
relevant datasets is presented in Table 1. Korshunov and Marcel (Korshunov and
Marcel, 2018) curated one of the first deepfake datasets DF-TIMIT, where face-
swapping was performed on VidTimit. Down the lane, other important datasets
such as UADFV (Yang et al., 2018), FaceForensics++ (Rossler et al., 2019), and
Google DFD (Nick and Andrew, 2019) were introduced. Due to the complexity of
face manipulation and limited availability of open-source face manipulation
techniques, these datasets are fairly small in size (Li et al., 2020).
Facebook released a large sized dataset DFDC (Dolhansky et al., 2020) in 2020
for the task of deepfake classification. Multiple face manipulation methods
were used to generate a total of 128,154 videos including real videos of 3000
actors. DFDC (Dolhansky et al., 2020) has become a mainstream benchmark
dataset for the task of deepfake detection. With the progress in both audio
and visual deepfake manipulation, post DFDC, several new datasets as Celeb-DF
(Li et al., 2020), DeeperForensics (Jiang et al., 2020), and WildDeepFake (Zi
et al., 2020) were introduced. It is worth noting that all these datasets are
designed for the binary task of deepfake classification and focus primarily on
visual manipulation detection (Chugh et al., 2020). In 2021, OpenForensics (Le
et al., 2021) dataset was introduced for spatial detection and segmentation.
Bounding boxes and masks are provided for each video along with real or fake
labels. The number of faces in each frame is more than one. Recently,
FakeAVCeleb (Khalid et al., 2021b) was released focusing on both face-swap and
face-reenactment methods with manipulated audio and video. ForgeryNet(He et
al., 2021) is the latest contribution to the growing list of deepfake
detection datasets. This large-scale dataset is also centered around video-
only identity manipulation and is suitable for the tasks of video/image
classification and spatial/temporal forgery localization.

To the best of our knowledge, all these previous datasets assume that face
manipulation occurs in most of the frames of the video (Chugh et al., 2020).
Only the latest one, ForgeryNet (He et al., 2021), provides examples of the
important problem of temporal forgery localization since it includes random
face swapping applied to parts of some videos. However, the manipulations
present in that dataset are only identity modifications that do not
necessarily alter the meaning of the content. Our content driven manipulation
dataset LAV-DF addresses this important gap.

Deepfake Detection. Deepfake detection methods draw inspiration from
observations of artifacts such as different eye colors, unnatural blink and
lip-sync issues in deepfake videos. These binary classification methods are
based on both traditional machine learning methods (e.g. EM (Guarnera et al.,
2020) and SVM (Yang et al., 2019)) and deep learning methods (e.g. 3DCNN(de
Lima et al., 2020), GRU(Montserrat et al., 2020) and ViT (Wodajo and Atnafu,
2021; Heo et al., 2021; Coccomini et al., 2021)). Previous methods (Lewis et
al., 2020; Gu et al., 2021) also aim to detect temporal inconsistencies in
deepfake content and recently, several audio-visual deepfake detection methods
such as MDS (Chugh et al., 2020) and M2TR (Wang et al., 2021) were proposed.
The methods above are classification centric and do not focus on temporal
localization. The only exception is the MDS (Chugh et al., 2020), which is
shown to work for localization tasks, however, the method is designed
primarily for classification. The proposed LAV-DF dataset and method are
specifically designed for temporal localization of manipulations.

Temporal Localization. Given that the task of temporal forgery localization is
similar to the task of temporal action localization, previous work in this
area is important. Benchmark datasets in this domain include THUMOS (Idrees et
al., 2017) and ActivityNet (Caba Heilbron et al., 2015) and the proposed
methods can be grouped into two categories: 2-step structures which first
generate segment proposals and then perform multi-class classification to
evaluate the proposals (Zeng et al., 2019; Xu et al., 2020; Liu et al., 2021)
and 1-step methods which directly generate the final segment predictions (Lin
et al., 2017; Buch et al., 2019; Nawhal and Mori, 2021). For the task of
temporal forgery localization there are no classification requirements for the
foreground segments, that is, the background is always real and the foreground
segments are always fake. Therefore, boundary prediction and 1-step methods
are more important and related to our task.

Bagchi et al. (Bagchi et al., 2021) divided the methods for segment proposal
estimation in temporal action localization into two main categories: methods
based on anchors and methods based on predicting the boundary probabilities.
As for the anchor-based, these methods mainly use sliding windows in the
video, such as S-CNN (Shou et al., 2016), CDC (Shou et al., 2017), TURN-TAP
(Gao et al., 2017) and CTAP (Gao et al., 2018). As for the methods predicting
the boundary probabilities, Lin et al. (Lin et al., 2018) introduced BSN in
2018. The method can utilize the global information to overcome the problem
that anchor-based methods cannot generate precise and flexible segment
proposals. Based on BSN, BMN (Lin et al., 2019) and BSN++ (Su et al., 2021)
were introduced for improved performance. It is worth noting that all these
methods are unimodal, which is not optimal for the task of temporal forgery
detection. The importance of multimodality was demonstrated recently by
AVFusion (Bagchi et al., 2021).

Proposed Approach. For the task of temporal forgery detection, both the audio
and visual information are important, in addition to the required precise
boundary proposals. In this paper, we introduce a multimodal method based on
boundary probabilities and compare the performance with BMN (Lin et al.,
2019), AGT (Nawhal and Mori, 2021), MDS (Chugh et al., 2020) and AVFusion
(Bagchi et al., 2021).

##  3\. Localized Audio Visual DeepFake

Figure 2. The pipeline for LAV-DF dataset generation. The green-edge audio and
video frames are the real data and the red-edge audio and video frames are the
generated fake data. The real audio based transcript is used to decide the
location and content to be replaced based on the largest change in sentiment.
The chosen antonyms are used as an input for generating fake audio with voice
cloning. The post-processing and normalization are applied to the audio to
maintain the consistency of the loudness between the generated audio and real
audio in the neighbourhood. The generated audio is used as input for facial
reenactment. Three categories of data are generated: <Fake Audio and Fake
Video>, <Fake Audio and Real Video> and <Real Audio and Fake Video>. The
details are discussed in Section 3.1.

The proposed dataset is a large audio-visual localised deepfake dataset. The
main steps in creating the dataset are: a) Sourcing real videos, b) Processing
real videos to manipulate the transcripts and c) Audio and video synthesis.
The deepfake generation step is based on the hypothesis that changing relevant
word(s) in a statement can lead to change in its perception. The same is also
reflected by changing the sentiment value. The manipulation strategy is to
replace word/words with its/their antonym(s), which leads to a significant
change in the sentiment score of a statement. The overall pipeline of data
generation is shown in Figure 2.

Data Source and Parsing. We sourced the real videos from the VoxCeleb2 (Chung
et al., 2018) dataset. VoxCeleb2 is a facial video dataset with over 1 million
utterance videos of over 6000 speakers. The videos in VoxCeleb2 are collected
from YouTube and the face parts of the video are tracked and cropped by the
CNN facial detector in (King, 2009) with 224$\times$ 224 resolution. The
original dataset contains videos of different duration, spoken language, and
voice loudness. English speaking videos are chosen using the confidence score
from Google Speech-to-Text service. The same service generates the
transcripts, which are used for manipulation.

###  3.1. Data Generation

Transcript Manipulation. After collecting and wrangling the real data from
VoxCeleb2, the next step is to analyse a video’s transcript denoted by
$D=\\{d_{0},d_{1},\cdots,d_{m},\cdots,d_{n}\\}$, where $d_{i}$ denotes word
tokens and $n$ is the number of tokens in the transcript. The aim is to find
the tokens to be replaced in $D$ such that the sentiment score of the
transcript changes the most. This is essentially to create transcript
$D^{\prime}=\\{d_{0},d_{1},\cdots,d_{m}^{\prime},\cdots,d_{n}\\}$, composed of
most of the tokens of $D$ with the exception of a few tokens being replaced.
These replaced tokens $d^{\prime}$ are selected from a set $\hat{d}$ of
antonyms of $d$. We used the sentiment analyzer in NLTK (Bird et al., 2009)
which predicts the sentiment value of a video transcript. Positive sentiment
value means the sentiment in the transcript $S(D)$ is positive, negative
sentiment value means the sentiment is negative, and zero sentiment value
means the sentiment is neutral. For each token $d$ in a transcript $D$, we
find the replacement as follows,

$\tau=\operatorname*{argmax}_{d\in
D,d^{\prime}\in\hat{d}}|S(D)-S(D^{\prime})|$

We find all the replacements in a transcript $D$ as,

$\theta=\operatorname*{argmax}_{\\{\tau_{m}\\}_{m=1}^{M}}|\sum_{i=1}^{M}\Delta
S(\tau_{i})|$

where $\Delta S(\tau_{i})$ is the sentiment difference with the replacement
$\tau_{i}$ and $M$ is the maximum number of replacements in the transcript.
For videos shorter than 10 seconds, there is up to 1 replacement; otherwise,
there are up to 2 replacements. Figure 3 (a) illustrates the significant
change in the sentiment distribution after the manipulations and Figure 3 (b)
presents the histogram of $|\Delta S|$, suggesting that the sentiment of most
transcripts was successfully changed.

Audio Generation. Once the replacement tokens $d^{\prime}$ are available, the
next step is to generate their corresponding audio in the style of the speaker
in the video. We evaluated several recent adaptive TTS methods (Jia et al.,
2018; Casanova et al., 2021; Neekhara et al., 2021), which can generate the
speech style of a person who is not in the training dataset. Based on better
performance, we chose speaker verification based multi-speaker TTS (SV2TTS)
(Jia et al., 2018) as the final method for audio generation. The SV2TTS method
comprises of three modules: a) An encoder for extracting style embedding of
the reference speaker, b) Tacotron 2 (Shen et al., 2018) based spectrogram
generated using the replacement tokens and the speaker style embedding as
input, and c) WaveNet (Oord et al., 2016) based vocoder for generating
realistic audio using the spectrogram generated from the synthesizer. We used
a pre-trained SV2TTS for generating the fake audio segments and later
normalized the loudness of the generated audio using the real audio neighbors.

Video Generation. Once the fake audio is generated, we used it as an input for
generating the corresponding fake video frames. We used Wav2Lip facial
reenactment (Prajwal et al., 2020) for this task as it has been shown to have
better output quality than previous methods (Jamaludin et al., 2019; K R et
al., 2019), and has better generalization and robustness to unseen scenarios.
It is worth noting that newer methods that achieve better video synthesis
quality are not suitable for our task. For example, AD-NeRF (Guo et al., 2021)
is not designed for zero-shot generation of unseen identities and ATVGNet
(Chen et al., 2019) reenacts the face based on a reference static image, which
causes pose inconsistencies on the boundary between fake and real segments.
Wav2Lip takes a reference video and target audio as input, and generates an
output video in which the person in the reference video is speaking the target
audio content with synced lips. We used pre-trained Wav2Lip model and up-
scaled the generated fake video segment to a resolution of $224\times 224$.
Finally, the generated fake audio and video segments were synchronised and
used to replace the original audio and video segments corresponding to the
original tokens.

Based on the generation of fake audio and video, we have three variations in
the final dataset (similar to (Khalid et al., 2021a)):
1\. Fake Audio and Fake Video. Both the audio and corresponding video are
generated for replacement tokens.

2\. Fake Audio and Real Video. Only the audio is generated for replacement
tokens and the corresponding real video is length-normalized.

3\. Real Audio and Fake Video. Only the video is generated for replacement
tokens and the length of the fake video is normalized to match the real audio.

###  3.2. Dataset Statistics

The dataset contains a total of 136,304 videos of which 36,431 are completely
real and 99,873 contain fake segments, with 153 unique identities. We split
the dataset into 3 identity independent subsets for training (78703 videos of
91 identities), validation (31501 videos of 31 identities), and testing (26100
videos of 31 identities).

Summary of the dataset is shown in Figure 3. The total number of fake segments
is 114,253 with duration in the range [0-1.6] seconds and average length of
0.65 seconds, where 89.26% of the segments are shorter than 1 second. The
maximum video length is 20 seconds and 69.61% of the videos are shorter than
10 seconds. As for the modality modification types, the amount of the 4 types
(i.e. video-modified, audio-modified, both-modified, real) are approximately
equal. In most videos (62.72%) there is 1 fake segment, and in some videos
(10.55%) there are 2.

Figure 3. Summary of the LAV-DF dataset: (a) Distribution of sentiment scores,
(b) Distribution of sentiment change, (c) Distribution of fake segment
lengths, (d) Distribution of video lengths, (e) Proportion of fake segments,
and (f) Proportion of modifications.

##  4\. Proposed Method

The proposed method Boundary Aware Temporal Forgery Detection (BA-TFD) is
illustrated in Figure 4. The first step of the method is to extract features
from the input data $X=\\{V,A\\}$, where $V$ is the video and $A$ is the
audio.

###  4.1. Feature Encoders

Figure 4. Structure of the proposed method Boundary Aware Temporal Forgery
Detection (BA-TFD). The video encoder uses raw video as input. The audio
encoder uses spectrograms extracted from raw audio. $\oplus$ denotes
concatenation. During inference, post-processing is applied to generate
segments from the output of the fusion module. The details are discussed in
Section 4.

Video Encoder. The goal of the video encoder is to learn frame-level spatio-
temporal features from the input video $V$ using a 3DCNN. For that purpose we
designed the video encoder $E_{v}$ to take the whole video
$V\in\mathbb{R}^{C\times T\times H\times W}$ as input, where $T$ is the number
of frames, $C$ is the number of channels, and $H$ and $W$ are the height and
width of the frame. The output of the $E_{v}$ are the frame-level features
$F_{v}\in\mathbb{R}^{C_{f}\times T}$, where $C_{f}$ is the features dimension.
$E_{v}$ is composed of 4 blocks, each containing multiple 3D convolutional
layers with kernel size $3\times 3\times 3$ and a final max-pooling layer.

Audio Encoder. The goal of the audio encoder is to learn features from the
input audio $A$ using a 2DCNN. In addition, the learned audio features are
temporarily aligned with the learned frame-level video features. The first
step is to generate the spectrogram $A^{\prime}\in\mathbb{R}^{T_{a}\times
F_{m}}$ of the audio signal in log-space, where $T_{a}$ is the temporal
dimension and $F_{m}$ is the length of mel-frequency cepstrum features. In the
second step, we designed the audio encoder $E_{a}$ to take the spectrogram
$A^{\prime}$ as input. The output of the $E_{a}$ are the audio frame features
$F_{a}\in\mathbb{R}^{C_{f}\times T}$, where $C_{f}$ is the features dimension.
$E_{a}$ is composed of multiple 2D convolutional layers with kernel size
$3\times 3$ and a final max-pooling layer to reduce the temporal dimension
$T_{a}$ to $T$.

###  4.2. Loss Functions

Contrastive Loss. Our hypothesis is that content modification in one or more
modalities will result in miss-synchronization between the modalities (i.e.
video and audio) and contrastive loss has been shown (Chung and Zisserman,
2017; Chugh et al., 2020) to be a powerful objective for similar tasks. In our
method, the audio and video features learned from real videos are regarded as
positive pairs $y=1$, and the audio and video features learned from videos
with at least one modified modality are regarded as negative pairs $y=0$. For
the positive pairs, the contrastive loss minimizes the difference between the
modalities while for negative pairs, the contrastive loss keeps that margin
larger than $\delta$,

$L_{c}=\frac{1}{NC_{f}T}\sum^{N}_{i=1}y_{i}d_{i}^{2}+(1-y_{i})\max(\delta-
d_{i},0)^{2}$ $d_{i}=||F_{vi}-F_{ai}||_{2}$

Auxiliary Frame Classification Loss. Since we have access to the frame-level
features $F_{v}$ and $F_{a}$, we can utilize the labels and train the encoders
to extract powerful and robust features that capture different deepfake
artifacts. For that purpose we designed two frame-level logistic regression
classifiers $FC_{v}$ and $FC_{a}$ using $F_{v}$ and $F_{a}$ as input. The
classifiers consist of 1D convolutional layers and predict the label $\hat{Y}$
as 1 (fake) or 0 (real) for each frame and each modality. The classifiers are
trained with cross-entropy loss,

$L_{f}=-\frac{1}{2NT}\sum_{m\in\\{a,v\\}}\sum^{N}_{i=1}\sum^{T}_{j=t}H(\hat{Y}_{mij},Y_{mij})$
$H(\hat{Y},Y)=Y\log{\hat{Y}}+(1-Y)\log{(1-\hat{Y})}$
$Y_{m}=\eta_{m}Y+(1-\eta_{m})Y_{0}$

where $N$ is the number of samples in the dataset, $T$ is the number of
frames, $m$ is the modality (i.e. audio $a$ or video $v$), $\eta_{m}$
specifies whether modality $m$ is modified, and $Y_{0}$ is the label for real
videos.

Boundary Matching Loss. The ground truth boundary maps are generated following
the procedure in (Lin et al., 2019). Given the fusion boundary map $\hat{M}$,
video boundary map $\hat{M_{v}}$ and audio boundary map $\hat{M_{a}}$
predicted by the model we use mean squared error as boundary matching loss for
$\hat{M}$, $\hat{M_{v}}$ and ${\hat{M_{a}}}$. The fusion boundary matching
loss is,

$L_{b}=\frac{1}{NDT}\sum^{N}_{i=1}\sum^{D}_{j=1}\sum^{T}_{k=1}(\hat{M}_{ijk}-M_{ijk})^{2}$

where, $N$ is the number of samples in the dataset, $D$ is the number of all
possible proposal durations and $T$ is the number of frames. The modality
boundary matching loss is similar to the frame classification loss,

$L_{bm}=\frac{1}{2NDT}\sum_{m\in\\{v,a\\}}\sum^{N}_{i=1}\sum^{D}_{j=1}\sum^{T}_{k=1}(\hat{M}_{mijk}-M_{mijk})^{2}$
$M_{m}=\eta_{m}M+(1-\eta_{m})M_{0}$

where, $m$ is the modality (i.e. video $v$ or audio $a$), $\eta_{m}$ specifies
whether modality $m$ is modified, and $M_{0}$ is the ground truth boundary map
for real videos.

Overall Loss. The overall loss is defined as follows,

$L=\lambda_{c}L_{c}+\lambda_{f}L_{f}+\lambda_{b}L_{b}+\lambda_{bm}L_{bm}$

where, $\lambda_{c}$, $\lambda_{f}$, $\lambda_{b}$ and $\lambda_{bm}$ are
weights for different losses.

Figure 5. The structure of the fusion module. The gray block normalizes the
video and audio weights predicted from the 1D convolutional layers and applies
element-wise weighted average. $\oplus$ denotes element-wise addition and
$\otimes$ denotes element-wise multiplication. BM: boundary map. For details
refer to Section 4.3.

###  4.3. Multimodal Fusion

The predictions of $FC_{v}$ and $FC_{a}$ are concatenated with the features
$F_{v}$ and $F_{a}$, and used by two boundary matching layers $B_{v}$ and
$B_{a}$ (Lin et al., 2019). The goal is to predict the boundary maps
$\hat{M}_{v}\in\mathbb{R}^{D\times T}$ and $\hat{M}_{a}\in\mathbb{R}^{D\times
T}$ for the video and audio, where $T$ is the number of frames and $D$ is the
maximum duration of the fake segments. The fusion module, illustrated in
Figure 5, uses the $\hat{M}_{v}$, $\hat{M}_{a}$, $F_{v}$ and $F_{a}$ as input.
For the video modality, the $\hat{M}_{v}$, $F_{v}$ and $F_{a}$ are used to
calculate the video weights $W_{v}\in\mathbb{R}^{D\times T}$ and for the audio
modality, the $\hat{M}_{a}$, $F_{a}$ and $F_{v}$ are used to calculate the
audio weights $W_{a}\in\mathbb{R}^{D\times T}$. In the final step, we perform
element-wise weighted average and calculate the fusion boundary map prediction
$\hat{M}\in\mathbb{R}^{D\times T}$,

$\hat{M}=\frac{W_{v}\hat{M}_{v}+W_{a}\hat{M}_{a}}{W_{v}+W_{a}}$

where all operations are element-wise.

###  4.4. Inference

During inference, the model uses the video and audio as input and generates a
fusion boundary map $\hat{M}$. The boundary map represents the confidence for
all proposals in the video and is very dense (i.e. there are many duplicated
proposals). Similar to BSN (Lin et al., 2018), we utilize post-processing with
Soft Non-Maximum Suppression (S-NMS) (Bodla et al., 2017) to eliminate the
duplicated proposals.

##  5\. Experiments

We have performed extensive benchmarking of LAV-DF via several state-of-the-
art methods: BMN (Lin et al., 2019), AGT (Nawhal and Mori, 2021), AVFusion
(Bagchi et al., 2021) and MDS (Chugh et al., 2020). Apart from our proposed
dataset LAV-DF, we also validate our method BA-TFD for classification on DFDC
dataset.

Dataset Preparation and Evaluation Protocol. To compare with visual-only
methods, we prepare a subset of the test set where the audio-only modified
data is removed which is denoted as “subset”. The original test set is denoted
as “full set” in the experiments.

Unlike temporal action localization methods (Liu et al., 2021; Nawhal and
Mori, 2021) that are using only average precision, we follow the protocol
proposed in ForgeryNet (He et al., 2021) and use both average precision (AP)
and average recall (AR) as the evaluation metrics for the quantitative
comparison. For AP, we follow the protocol of ActivityNet (Caba Heilbron et
al., 2015) to set the IoU thresholds to 0.5, 0.75 and 0.95. For AR, as the
number of fake segments is small, we set the number of proposals to smaller
numbers like 100, 50, 20 and 10 with the IoU thresholds [0.5:0.05:0.95]. Our
method can also be used for deepfake detection (i.e. classification) task. We
use area under the curve (AUC) for evaluation of the deepfake classification
task.

BA-TFD is a multimodal temporal detection method and if only the visual
component is used, the method is very similar to BMN. In this work we focus on
the problem of temporal localization instead of classification and only DFDC
dataset is used to evaluate the classification performance of the proposed
method.

Implementation Details. BA-TFD is implemented in PyTorch (Paszke et al.,
2019). For hyperparameters, we set $\lambda_{c}$ = $0.1$, $\lambda_{f}$ = $2$,
$\lambda_{b}$ = $1$, $\lambda_{bm}$ = $1$ and $\delta$ = $0.99$. For
comparison, we trained BMN (Lin et al., 2019), AGT (Nawhal and Mori, 2021),
AVFusion (Bagchi et al., 2021) and MDS (Chugh et al., 2020) on temporal
forgery localization task.

In addition, to evaluate the usefulness of the proposed method, we compare
with MDS, EfficientViT (Coccomini et al., 2021) and other methods on
classification task. We followed the original settings for BMN, AGT, MDS and
EfficientViT, and used encoding concatenation fusion for AVFusion. For the
methods that require pre-trained features, we trained them end-to-end with
trainable encoder. For comparison, we also trained BMN with I3D features
(Carreira and Zisserman, 2017) (i.e. fixed encoder). For the models which
require S-NMS post-processing, we used the validation set to search for
optimal parameters for post-processing. Final evaluation and results are based
on the test set.

For DFDC, we consider the whole fake video as one fake segment. For
evaluation, we used 2 methods to generate the classification output for BA-
TFD: (1) Using the highest confidence of the predicted segments as the
confidence of the video being fake and (2) Training a MLP classifier using the
confidences of predicted segments. We chose evaluation method (1) for LAV-DF
and method (2) for DFDC based on better performance on the datasets.

Table 2. Temporal forgery localization results on the full set (please see
Section 5) of LAV-DF. The BA-TFD (visual-only) is the output from video
boundary matching layer in Figure 4, showing the performance when using only
the video modality.

Model |<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|  AR@100 |  AR@50 |  AR@20 |  AR@10
---|---|---|---|---|---|---|---
MDS (Chugh et al., 2020) | 12.78 | 01.62 | 00.00 | 37.88 | 36.71 | 34.39 | 32.15
AGT (Nawhal and Mori, 2021) | 17.85 | 09.42 | 00.11 | 43.15 | 34.23 | 24.59 | 16.71
BMN (Lin et al., 2019) | 24.01 | 07.61 | 00.07 | 53.26 | 41.24 | 31.60 | 26.93
BMN (I3D) | 10.56 | 01.66 | 00.00 | 48.49 | 44.39 | 37.13 | 31.55
AVFusion (Bagchi et al., 2021) | 65.38 | 23.89 | 00.11 | 62.98 | 59.26 | 54.80 | 52.11
BA-TFD (visual-only) | 58.55 | 28.60 | 00.16 | 62.49 | 58.77 | 53.86 | 50.29
BA-TFD |  76.90 |  38.50 |  00.25 |  66.90 |  64.08 |  60.77 |  58.42

Table 3. Temporal forgery localization results on the subset (please see
Section 5) of LAV-DF. The BA-TFD (visual-only) is the output from video
boundary matching layer in Figure 4, showing the performance when using only
the video modality.

Model |<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|  AR@100 |  AR@50 |  AR@20 |  AR@10
---|---|---|---|---|---|---|---
MDS (Chugh et al., 2020) | 23.43 | 03.48 | 00.00 | 58.53 | 56.68 | 53.16 | 49.67
AGT (Nawhal and Mori, 2021) | 15.69 | 10.69 | 00.15 | 49.11 | 40.31 | 31.70 | 23.13
BMN (Lin et al., 2019) | 32.32 | 11.38 | 00.14 | 59.69 | 48.17 | 39.01 | 34.17
BMN (I3D) | 28.10 | 05.47 | 00.01 | 55.49 | 54.44 | 52.14 | 47.72
AVFusion (Bagchi et al., 2021) | 62.01 | 22.77 | 00.11 | 61.98 | 58.08 | 53.31 | 50.52
BA-TFD (visual-only) | 83.55 | 41.88 | 00.24 | 65.79 | 62.30 | 57.95 | 55.34
BA-TFD |  85.20 |  47.06 |  00.29 |  67.34 |  64.52 |  61.19 |  59.32

Table 4. Temporal forgery localization results of BA-TFD with different
losses.

Loss Function |<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|  AR@100 |  AR@50 |  AR@20 |  AR@10
---|---|---|---|---|---|---|---
$L_{b}$ | 53.16 | 11.91 | 00.02 | 53.99 | 50.94 | 47.74 | 45.55
$L_{bm},L_{b}$ | 54.70 | 15.50 | 00.04 | 56.64 | 53.57 | 49.46 | 45.85
$L_{f},L_{bm},L_{b}$ | 76.50 |  39.92 | 00.18 | 66.69 | 63.71 | 60.07 | 57.76
$L_{c},L_{f},L_{bm},L_{b}$ |  76.90 | 38.50 |  00.25 |  66.90 |  64.08 |  60.77 |  58.42

##  6\. Results

Quantitative Analysis.

Temporal Localization: We compare our model on the LAV-DF full set with the
latest methods for temporal action localization and deepfake detection. From
Table 2, our model achieves the best performance, which is 76.9 for<EMAIL_ADDRESS>and
66.9 for AR@100. Unlike temporal action localization datasets such as
ActivityNet, there is a single label for the fake segments, so it is
reasonable that the AP score is relatively high. The multimodal MDS is not
designed for temporal forgery localization tasks and predicts only fixed
length segments (i.e. cannot predict the precise boundaries), hence the scores
for that model are low. As for AGT and BMN, the scores are low because they
are visual-only unimodal methods and cannot detect the fake segments in videos
where only the audio is modified. We also evaluate the visual-only unimodal
output of our model, which shows worse results than the multimodal BA-TFD and
AVFusion. In addition, the results show that when the video encoder is trained
with LAV-DF data, BMN performs significantly better than using I3D features.

We also evaluate those methods on the subset mentioned in Section 5. From
Table 3, the performance of the visual-only models is improved, for BA-TFD,
the visual-only score improves from 58.55 to 83.55<EMAIL_ADDRESS>and the margin
between unimodal and multimodal is decreased from 18.35 to 1.65 (AP@0.5).
Overall, our method still ranks first, which shows that our method has
superior performance for temporal forgery detection.

DeepFake Classification: We also compare our method with previous deepfake
detection methods on LAV-DF and a subset of DFDC. From Table 5, our model
outperforms MDS and EfficientViT on LAV-DF. As for the subset of DFDC, the
performance of our method (see Table 6) is better than previous methods such
as Meso4 (Afchar et al., 2018) and FWA(Li and Lyu, 2019) and is close to MDS.
The reasons of BA-TFD performing better in LAV-DF might be several. Our method
BA-TFD is not designed and trained for classification task with classification
loss. It is trained for temporal forgery localization and then the segment
outputs are summarized as a whole video label prediction. Therefore, the
performance of our method drops as compared to the state-of-the-art
classification method MDS. On the other hand, previous deepfake detection
methods assume that fake videos are entirely fake, so their performance is
reduced on LAV-DF. In summary, our method still performs well on
classification task and has potential to reach the state-of-the-art
performance.

Impact of Loss Functions: To examine the contributions of each loss of BA-TFD,
we train four models with different combinations of losses. From Table 4, all
four losses have positive influences on the performance. By observing the
difference between the scores, the frame classification loss $L_{c}$
contributes the most in the model. With the frame-level label supervising the
model, the encoders are trained to have a better capacity to extract the
features relevant to deepfake artifacts. We visualize the distributions of
video features $F_{v}$ of 10 videos in the dataset in Figure 7. From the
plots, with frame classification loss and contrastive loss added, the features
are more separable, which contributes to the temporal localization and
improves the performance.

Figure 6. Boundary map output visualizations. The first column illustrates the
modality-wise boundary map outputs for a real video. The other columns
illustrate the modality-wise boundary map ouputs of the corresponding fake
videos (i.e. video based modification, audio based modification and audio-
visual based modification). mod: modified and GT: ground truth. Figure 7.
Feature distribution in PCA subspace. Each point is the features of a video
frame marked with real or fake.

Qualitative Analysis. We selected a real video sample and three corresponding
fake videos each with manipulation in a single or both modalities. For these
four videos, we visualize the boundary map outputs $\hat{M}_{v}$,
$\hat{M}_{a}$ and $\hat{M}$ in Figure 6. It is observed that the video output
captures the fake segments when the video modality is manipulated. The same
holds for audio. Regardless of whether the audio or video modality is
modified, the fusion output captures the correct information from audio and
visual outputs and shows the effectiveness of the fusion module.

Failure Analysis. We observe that BA-TFD predictions can be noisy in a few
cases, which contain only a short duration ($\leq 0.5\ sec$) video
manipulations and the corresponding real audio. We argue that for such short
video-only manipulations, if the visual transition from real to fake and then
to real is smooth, it may lead to a noisy result.

##  7\. Conclusion, Limitation and Future Work

In this work we investigated the problem of deepfake detection from the
perspective of partial video modification. To this end, a new dataset LAV-DF
is curated in which the audio and video are modified at specific locations
based on the potential change in sentiment of the video content. For this
problem, we also proposed a new method BA-TFD. The conducted experiments show
that our method achieves better performance than previous relevant state-of-
the-art methods.

Potential Risks and Ethical Concerns. Our work especially LAV-DF might include
potential negative social impacts. As the subjects in the dataset are
celebrities, the content in the dataset may be used for unethical purposes
such as making fake rumours. Also, the dataset generation pipeline can be
applied to create fake videos. To encounter the negative impacts, we prepared
a license for public usage of the dataset and proposed the BA-TFD.

Table 5. Classification results on the full set of LAV-DF.

.  Dataset LAV-DF Model MDS (Chugh et al., 2020) EfficientViT (Coccomini et
al., 2021) BA-TFD AUC 0.828 0.965 0.990

Table 6. Classification results on a subset set of DFDC.

Dataset | DFDC
---|---
Model | Meso4(Afchar et al., 2018) | FWA(Li and Lyu, 2019) | Siamese(Mittal et al., 2020) | MDS (Chugh et al., 2020) | BA-TFD
AUC | 0.753 | 0.727 | 0.844 | 0.916 | 0.846

Limitations. Major limitations are: a) The audio reenactment method used in
the dataset does not always generate the reference style and b) The resolution
of the dataset is constrained on the basis of source videos.

Future work. Major improvement in the future will be increasing LAV-DF with
new token insertion, substitution and deletion of existing tokens and
converting statements into questions.

## References

  * (1)
  * Afchar et al. (2018) Darius Afchar, Vincent Nozick, Junichi Yamagishi, and Isao Echizen. 2018.  MesoNet: a Compact Facial Video Forgery Detection Network. In _2018 IEEE International Workshop on Information Forensics and Security (WIFS)_. 1–7.  https://doi.org/10.1109/WIFS.2018.8630761 ISSN: 2157-4774.
  * Bagchi et al. (2021) Anurag Bagchi, Jazib Mahmood, Dolton Fernandes, and Ravi Kiran Sarvadevabhatla. 2021\.  Hear Me Out: Fusional Approaches for Audio Augmented Temporal Action Localization.  _arXiv:2106.14118 [cs]_ (Aug. 2021).  arXiv: 2106.14118 version: 3.
  * Bird et al. (2009) Steven Bird, Ewan Klein, and Edward Loper. 2009.  _Natural Language Processing with Python: Analyzing Text with the Natural Language Toolkit_.  O’Reilly Media, Inc.  Google-Books-ID: KGIbfiiP1i4C.
  * Bodla et al. (2017) Navaneeth Bodla, Bharat Singh, Rama Chellappa, and Larry S. Davis. 2017.  Soft-NMS – Improving Object Detection With One Line of Code. In _Proceedings of the IEEE International Conference on Computer Vision_. 5561–5569.
  * Brandon (2019) John Brandon. 2019\.  There Are Now 15,000 Deepfake Videos on Social Media. Yes, You Should Worry.  _Forbes_ (Oct. 2019).
  * Buch et al. (2019) Shyamal Buch, Victor Escorcia, Bernard Ghanem, Li Fei-Fei, and Juan Carlos Niebles. 2019\.  End-to-end, single-stream temporal action detection in untrimmed videos.  _Procedings of the British Machine Vision Conference 2017_ (May 2019).  https://doi.org/10.5244/c.31.93 Publisher: British Machine Vision Association.
  * Caba Heilbron et al. (2015) Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. 2015.  ActivityNet: A Large-Scale Video Benchmark for Human Activity Understanding. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 961–970.
  * Carreira and Zisserman (2017) Joao Carreira and Andrew Zisserman. 2017.  Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 6299–6308.
  * Casanova et al. (2021) Edresson Casanova, Christopher Shulby, Eren Gölge, Nicolas Michael Müller, Frederico Santos de Oliveira, Arnaldo Candido Junior, Anderson da Silva Soares, Sandra Maria Aluisio, and Moacir Antonelli Ponti. 2021\.  SC-GlowTTS: an Efficient Zero-Shot Multi-Speaker Text-To-Speech Model.  _arXiv:2104.05557 [cs, eess]_ (June 2021).  arXiv: 2104.05557.
  * Chen et al. (2019) Lele Chen, Ross K. Maddox, Zhiyao Duan, and Chenliang Xu. 2019.  Hierarchical Cross-Modal Talking Face Generation With Dynamic Pixel-Wise Loss. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 7832–7841.
  * Chugh et al. (2020) Komal Chugh, Parul Gupta, Abhinav Dhall, and Ramanathan Subramanian. 2020.  Not made for each other- Audio-Visual Dissonance-based Deepfake Detection and Localization. In _Proceedings of the 28th ACM International Conference on Multimedia_ _(MM ’20)_. Association for Computing Machinery, New York, NY, USA, 439–447.  https://doi.org/10.1145/3394171.3413700
  * Chung et al. (2018) J. S. Chung, A. Nagrani, and A. Zisserman. 2018.  VoxCeleb2: Deep Speaker Recognition. In _INTERSPEECH_.
  * Chung and Zisserman (2017) Joon Son Chung and Andrew Zisserman. 2017.  Out of Time: Automated Lip Sync in the Wild. In _Computer Vision – ACCV 2016 Workshops_ _(Lecture Notes in Computer Science)_ , Chu-Song Chen, Jiwen Lu, and Kai-Kuang Ma (Eds.). Springer International Publishing, Cham, 251–263.  https://doi.org/10.1007/978-3-319-54427-4_19
  * Coccomini et al. (2021) Davide Coccomini, Nicola Messina, Claudio Gennaro, and Fabrizio Falchi. 2021.  Combining EfficientNet and Vision Transformers for Video Deepfake Detection.  _arXiv:2107.02612 [cs]_ (July 2021).  arXiv: 2107.02612 version: 1.
  * de Lima et al. (2020) Oscar de Lima, Sean Franklin, Shreshtha Basu, Blake Karwoski, and Annet George. 2020\.  Deepfake Detection using Spatiotemporal Convolutional Networks.  _arXiv:2006.14749 [cs, eess]_ (June 2020).  arXiv: 2006.14749.
  * Dolhansky et al. (2020) Brian Dolhansky, Joanna Bitton, Ben Pflaum, Jikuo Lu, Russ Howes, Menglin Wang, and Cristian Canton Ferrer. 2020.  The DeepFake Detection Challenge (DFDC) Dataset.  _arXiv:2006.07397 [cs]_ (Oct. 2020).  arXiv: 2006.07397.
  * Gao et al. (2018) Jiyang Gao, Kan Chen, and Ram Nevatia. 2018.  CTAP: Complementary Temporal Action Proposal Generation. In _Proceedings of the European Conference on Computer Vision (ECCV)_. 68–83.
  * Gao et al. (2017) Jiyang Gao, Zhenheng Yang, Kan Chen, Chen Sun, and Ram Nevatia. 2017.  TURN TAP: Temporal Unit Regression Network for Temporal Action Proposals. In _Proceedings of the IEEE International Conference on Computer Vision_. 3628–3636.
  * Goodfellow et al. (2020) Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2020\.  Generative adversarial networks.  _Commun. ACM_ 63, 11 (Oct. 2020), 139–144.  https://doi.org/10.1145/3422622
  * Gu et al. (2021) Zhihao Gu, Yang Chen, Taiping Yao, Shouhong Ding, Jilin Li, Feiyue Huang, and Lizhuang Ma. 2021.  _Spatiotemporal Inconsistency Learning for DeepFake Video Detection_.  Association for Computing Machinery, New York, NY, USA, 3473–3481.  https://doi-org.ezproxy.lib.monash.edu.au/10.1145/3474085.3475508
  * Guarnera et al. (2020) Luca Guarnera, Oliver Giudice, and Sebastiano Battiato. 2020\.  DeepFake Detection by Analyzing Convolutional Traces. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_. 666–667.
  * Guo et al. (2021) Yudong Guo, Keyu Chen, Sen Liang, Yong-Jin Liu, Hujun Bao, and Juyong Zhang. 2021\.  AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 5784–5794.
  * He et al. (2021) Yinan He, Bei Gan, Siyu Chen, Yichun Zhou, Guojun Yin, Luchuan Song, Lu Sheng, Jing Shao, and Ziwei Liu. 2021.  ForgeryNet: A Versatile Benchmark for Comprehensive Forgery Analysis. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 4360–4369.
  * Heo et al. (2021) Young-Jin Heo, Young-Ju Choi, Young-Woon Lee, and Byung-Gyu Kim. 2021.  Deepfake Detection Scheme Based on Vision Transformer and Distillation.  _arXiv:2104.01353 [cs]_ (April 2021).  arXiv: 2104.01353.
  * Idrees et al. (2017) Haroon Idrees, Amir R. Zamir, Yu-Gang Jiang, Alex Gorban, Ivan Laptev, Rahul Sukthankar, and Mubarak Shah. 2017.  The THUMOS Challenge on Action Recognition for Videos ”in the Wild”.  _Computer Vision and Image Understanding_ 155 (Feb. 2017), 1–23.  https://doi.org/10.1016/j.cviu.2016.10.018 arXiv: 1604.06182.
  * Jamaludin et al. (2019) Amir Jamaludin, Joon Son Chung, and Andrew Zisserman. 2019\.  You Said That?: Synthesising Talking Faces from Audio.  _International Journal of Computer Vision_ 127, 11 (Dec. 2019), 1767–1779.  https://doi.org/10.1007/s11263-019-01150-y
  * Jia et al. (2018) Ye Jia, Yu Zhang, Ron J. Weiss, Quan Wang, Jonathan Shen, Fei Ren, Zhifeng Chen, Patrick Nguyen, Ruoming Pang, Ignacio Lopez Moreno, and Yonghui Wu. 2018.  Transfer learning from speaker verification to multispeaker text-to-speech synthesis. In _Proceedings of the 32nd International Conference on Neural Information Processing Systems_ _(NIPS’18)_. Curran Associates Inc., Red Hook, NY, USA, 4485–4495.
  * Jiang et al. (2020) Liming Jiang, Ren Li, Wayne Wu, Chen Qian, and Chen Change Loy. 2020.  DeeperForensics-1.0: A Large-Scale Dataset for Real-World Face Forgery Detection. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 2889–2898.
  * K R et al. (2019) Prajwal K R, Rudrabha Mukhopadhyay, Jerin Philip, Abhishek Jha, Vinay Namboodiri, and C V Jawahar. 2019.  Towards Automatic Face-to-Face Translation. In _Proceedings of the 27th ACM International Conference on Multimedia_ _(MM ’19)_. Association for Computing Machinery, New York, NY, USA, 1428–1436.  https://doi.org/10.1145/3343031.3351066
  * Khalid et al. (2021a) Hasam Khalid, Minha Kim, Shahroz Tariq, and Simon S. Woo. 2021a.  Evaluation of an Audio-Video Multimodal Deepfake Dataset using Unimodal and Multimodal Detectors.  _Proceedings of the 1st Workshop on Synthetic Multimedia - Audiovisual Deepfake Generation and Detection_ (Oct. 2021), 7–15.  https://doi.org/10.1145/3476099.3484315 arXiv: 2109.02993.
  * Khalid et al. (2021b) Hasam Khalid, Shahroz Tariq, and Simon S. Woo. 2021b.  FakeAVCeleb: A Novel Audio-Video Multimodal Deepfake Dataset.  _arXiv:2108.05080 [cs]_ (Aug. 2021).  arXiv: 2108.05080.
  * King (2009) Davis E. King. 2009\.  Dlib-ml: A Machine Learning Toolkit.  _The Journal of Machine Learning Research_ 10 (Dec. 2009), 1755–1758.
  * Korshunov and Marcel (2018) Pavel Korshunov and Sebastien Marcel. 2018.  DeepFakes: a New Threat to Face Recognition? Assessment and Detection.  _arXiv:1812.08685 [cs]_ (Dec. 2018).  arXiv: 1812.08685.
  * Korshunova et al. (2017) Iryna Korshunova, Wenzhe Shi, Joni Dambre, and Lucas Theis. 2017\.  Fast Face-Swap Using Convolutional Neural Networks. In _Proceedings of the IEEE International Conference on Computer Vision_. 3677–3685.
  * Le et al. (2021) Trung-Nghia Le, Huy H. Nguyen, Junichi Yamagishi, and Isao Echizen. 2021.  OpenForensics: Large-Scale Challenging Dataset for Multi-Face Forgery Detection and Segmentation In-the-Wild. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 10117–10127.
  * Lewis et al. (2020) John K. Lewis, Imad Eddine Toubal, Helen Chen, Vishal Sandesera, Michael Lomnitz, Zigfried Hampel-Arias, Calyam Prasad, and Kannappan Palaniappan. 2020.  Deepfake Video Detection Based on Spatial, Spectral, and Temporal Inconsistencies Using Multimodal Deep Learning. In _2020 IEEE Applied Imagery Pattern Recognition Workshop (AIPR)_. 1–9.  https://doi.org/10.1109/AIPR50011.2020.9425167 ISSN: 2332-5615.
  * Li and Lyu (2019) Yuezun Li and Siwei Lyu. 2019.  Exposing DeepFake Videos By Detecting Face Warping Artifacts. In _IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_. 7.
  * Li et al. (2020) Yuezun Li, Xin Yang, Pu Sun, Honggang Qi, and Siwei Lyu. 2020.  Celeb-DF: A Large-Scale Challenging Dataset for DeepFake Forensics. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 3207–3216.
  * Lin et al. (2019) Tianwei Lin, Xiao Liu, Xin Li, Errui Ding, and Shilei Wen. 2019.  BMN: Boundary-Matching Network for Temporal Action Proposal Generation. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 3889–3898.
  * Lin et al. (2017) Tianwei Lin, Xu Zhao, and Zheng Shou. 2017.  Single Shot Temporal Action Detection. In _Proceedings of the 25th ACM international conference on Multimedia_ _(MM ’17)_. Association for Computing Machinery, New York, NY, USA, 988–996.  https://doi.org/10.1145/3123266.3123343
  * Lin et al. (2018) Tianwei Lin, Xu Zhao, Haisheng Su, Chongjing Wang, and Ming Yang. 2018.  BSN: Boundary Sensitive Network for Temporal Action Proposal Generation. In _Proceedings of the European Conference on Computer Vision (ECCV)_. 3–19.
  * Liu et al. (2021) Xiaolong Liu, Yao Hu, Song Bai, Fei Ding, Xiang Bai, and Philip H. S. Torr. 2021\.  Multi-Shot Temporal Event Localization: A Benchmark. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 12596–12606.
  * Mirsky and Lee (2021) Yisroel Mirsky and Wenke Lee. 2021.  The Creation and Detection of Deepfakes: A Survey.  _Comput. Surveys_ 54, 1 (Jan. 2021), 7:1–7:41.  https://doi.org/10.1145/3425780
  * Mittal et al. (2020) Trisha Mittal, Uttaran Bhattacharya, Rohan Chandra, Aniket Bera, and Dinesh Manocha. 2020\.  Emotions Don’t Lie: An Audio-Visual Deepfake Detection Method using Affective Cues. In _Proceedings of the 28th ACM International Conference on Multimedia_ _(MM ’20)_. Association for Computing Machinery, New York, NY, USA, 2823–2832.  https://doi.org/10.1145/3394171.3413570
  * Montserrat et al. (2020) Daniel Mas Montserrat, Hanxiang Hao, Sri K. Yarlagadda, Sriram Baireddy, Ruiting Shao, Janos Horvath, Emily Bartusiak, Justin Yang, David Guera, Fengqing Zhu, and Edward J. Delp. 2020\.  Deepfakes Detection With Automatic Face Weighting. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_. 668–669.
  * Nawhal and Mori (2021) Megha Nawhal and Greg Mori. 2021.  Activity Graph Transformer for Temporal Action Localization.  _arXiv:2101.08540 [cs]_ (Jan. 2021).  arXiv: 2101.08540.
  * Neekhara et al. (2021) Paarth Neekhara, Shehzeen Hussain, Shlomo Dubnov, Farinaz Koushanfar, and Julian McAuley. 2021\.  Expressive Neural Voice Cloning.  _arXiv:2102.00151 [cs, eess]_ (Jan. 2021).  arXiv: 2102.00151.
  * Nick and Andrew (2019) Dufou Nick and Jigsaw Andrew. 2019.  Contributing Data to Deepfake Detection Research.
  * Nirkin et al. (2019) Yuval Nirkin, Yosi Keller, and Tal Hassner. 2019\.  FSGAN: Subject Agnostic Face Swapping and Reenactment. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 7184–7193.
  * Oord et al. (2016) Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. 2016.  WaveNet: A Generative Model for Raw Audio.  _arXiv:1609.03499 [cs]_ (Sept. 2016).  arXiv: 1609.03499.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019.  PyTorch: An Imperative Style, High-Performance Deep Learning Library. In _Advances in Neural Information Processing Systems_ , Vol. 32. Curran Associates, Inc.
  * Prajwal et al. (2020) K R Prajwal, Rudrabha Mukhopadhyay, Vinay P. Namboodiri, and C.V. Jawahar. 2020.  A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild. In _Proceedings of the 28th ACM International Conference on Multimedia_ _(MM ’20)_. Association for Computing Machinery, New York, NY, USA, 484–492.  https://doi.org/10.1145/3394171.3413532
  * Rossler et al. (2019) Andreas Rossler, Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thies, and Matthias Niessner. 2019.  FaceForensics++: Learning to Detect Manipulated Facial Images. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 1–11.
  * Rumelhart et al. (1985) David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. 1985.  _Learning Internal Representations by Error Propagation_.  Technical Report. CALIFORNIA UNIV SAN DIEGO LA JOLLA INST FOR COGNITIVE SCIENCE.  Section: Technical Reports.
  * Sample (2020) Ian Sample. 2020\.  What are deepfakes – and how can you spot them?  _The Guardian_ (Jan. 2020).
  * Schwartz (2018) Oscar Schwartz. 2018\.  You thought fake news was bad? Deep fakes are where truth goes to die.  _The Guardian_ (Nov. 2018).
  * Shen et al. (2018) Jonathan Shen, Ruoming Pang, Ron J. Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan, Rif A. Saurous, Yannis Agiomvrgiannakis, and Yonghui Wu. 2018.  Natural TTS Synthesis by Conditioning Wavenet on MEL Spectrogram Predictions. In _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. 4779–4783.  https://doi.org/10.1109/ICASSP.2018.8461368 ISSN: 2379-190X.
  * Shou et al. (2017) Zheng Shou, Jonathan Chan, Alireza Zareian, Kazuyuki Miyazawa, and Shih-Fu Chang. 2017\.  CDC: Convolutional-De-Convolutional Networks for Precise Temporal Action Localization in Untrimmed Videos. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 5734–5743.
  * Shou et al. (2016) Zheng Shou, Dongang Wang, and Shih-Fu Chang. 2016.  Temporal Action Localization in Untrimmed Videos via Multi-Stage CNNs. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 1049–1058.
  * Su et al. (2021) Haisheng Su, Weihao Gan, Wei Wu, Yu Qiao, and Junjie Yan. 2021.  BSN++: Complementary Boundary Regressor with Scale-Balanced Relation Modeling for Temporal Action Proposal Generation.  _arXiv:2009.07641 [cs]_ (March 2021).  arXiv: 2009.07641.
  * Thies et al. (2020) Justus Thies, Mohamed Elgharib, Ayush Tewari, Christian Theobalt, and Matthias Nießner. 2020\.  Neural Voice Puppetry: Audio-Driven Facial Reenactment. In _ECCV 2020_ _(Lecture Notes in Computer Science)_ , Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (Eds.). Springer International Publishing, Cham, 716–731.  https://doi.org/10.1007/978-3-030-58517-4_42
  * Thomas (2020) Daniel Thomas. 2020\.  Deepfakes: A threat to democracy or just a bit of fun?  _BBC News_ (Jan. 2020).
  * Tulyakov et al. (2018) Sergey Tulyakov, Ming-Yu Liu, Xiaodong Yang, and Jan Kautz. 2018\.  MoCoGAN: Decomposing Motion and Content for Video Generation. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 1526–1535.
  * Wang et al. (2021) Junke Wang, Zuxuan Wu, Jingjing Chen, and Yu-Gang Jiang. 2021\.  M2TR: Multi-modal Multi-scale Transformers for Deepfake Detection.  _arXiv:2104.09770 [cs]_ (April 2021).  arXiv: 2104.09770.
  * Wang et al. (2017) Yuxuan Wang, R. J. Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J. Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, Quoc Le, Yannis Agiomyrgiannakis, Rob Clark, and Rif A. Saurous. 2017\.  Tacotron: Towards End-to-End Speech Synthesis.  _arXiv:1703.10135 [cs]_ (April 2017).  arXiv: 1703.10135.
  * Wodajo and Atnafu (2021) Deressa Wodajo and Solomon Atnafu. 2021.  Deepfake Video Detection Using Convolutional Vision Transformer.  _arXiv:2102.11126 [cs]_ (March 2021).  arXiv: 2102.11126.
  * Xu et al. (2020) Mengmeng Xu, Chen Zhao, David S. Rojas, Ali Thabet, and Bernard Ghanem. 2020.  G-TAD: Sub-Graph Localization for Temporal Action Detection. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 10156–10165.
  * Yang et al. (2018) Ke Yang, Peng Qiao, Dongsheng Li, Shaohe Lv, and Yong Dou. 2018.  Exploring Temporal Preservation Networks for Precise Temporal Action Localization.  _Proceedings of the AAAI Conference on Artificial Intelligence_ 32, 1 (April 2018).  Number: 1.
  * Yang et al. (2019) Xin Yang, Yuezun Li, and Siwei Lyu. 2019.  Exposing Deep Fakes Using Inconsistent Head Poses. In _ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. 8261–8265.  https://doi.org/10.1109/ICASSP.2019.8683164 ISSN: 2379-190X.
  * Zeng et al. (2019) Runhao Zeng, Wenbing Huang, Mingkui Tan, Yu Rong, Peilin Zhao, Junzhou Huang, and Chuang Gan. 2019.  Graph Convolutional Networks for Temporal Action Localization. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 7094–7103.
  * Zi et al. (2020) Bojia Zi, Minghao Chang, Jingjing Chen, Xingjun Ma, and Yu-Gang Jiang. 2020.  WildDeepfake: A Challenging Real-World Dataset for Deepfake Detection. In _Proceedings of the 28th ACM International Conference on Multimedia_ _(MM ’20)_. Association for Computing Machinery, New York, NY, USA, 2382–2390.  https://doi.org/10.1145/3394171.3413769


# Zn2GeO4/SnO2 nanowire architecture: A correlative spatially resolved X-ray
study

Jaime Dolado  European Synchrotron Radiation Facility, 38043 Grenoble, France
<EMAIL_ADDRESS>Fernanda Malato  Instituto de Ciencia de
Materiales de Madrid, Consejo Superior de Investigaciones Científicas, 28049
Cantoblanco, Spain  Jaime Segura-Ruiz  European Synchrotron Radiation
Facility, 38043 Grenoble, France  María Taeño  Departamento de Física de
Materiales, Facultad de Ciencias Físicas, Universidad Complutense de Madrid,
28040-Madrid, Spain  Irina Snigireva  European Synchrotron Radiation Facility,
38043 Grenoble, France  Pedro Hidalgo  Departamento de Física de Materiales,
Facultad de Ciencias Físicas, Universidad Complutense de Madrid, 28040-Madrid,
Spain  Bianchi Méndez  Departamento de Física de Materiales, Facultad de
Ciencias Físicas, Universidad Complutense de Madrid, 28040-Madrid, Spain  Gema
Martínez-Criado  European Synchrotron Radiation Facility, 38043 Grenoble,
France

###### Abstract

Wide bandgap semiconductor heterostructures based on nanowires have attracted
great interest as one of the nanoscale building blocks for the development of
novel optoelectronic devices. Owing to the critical role of heterogeneities
and structural imperfections in the optoelectronic properties, it has become a
challenge to investigate in detail the local structure and elemental
composition of these systems. Typically, transmission electron microscopy and
energy dispersive X-ray spectroscopy are applied to address such issues. Here,
using a synchrotron nanobeam probe, we show a correlative spatially-resolved
study of the composition, luminescence and local structure of a Zn2GeO4/SnO2
nanowire-based system. Unlike other works, the heterostructures were
synthesized by an effective catalyst-free thermal evaporation method and
present a skewer-shaped architecture formed by Plateau-Rayleigh instability.
Our results highlight the role of diffused impurities on the optical
properties of the heterostructure and open new avenues for further
heterogeneity studies using the X-ray nanoprobe.

###### keywords:

X-ray nanoprobe, nanowire-heterostructures, Plateau− Rayleigh instability,
wide bandgap oxides, XRF, XEOL, XAS

Departamento de Física de Materiales, Facultad de Ciencias Físicas,
Universidad Complutense de Madrid, 28040-Madrid, Spain Instituto de Ciencia de
Materiales de Madrid, Consejo Superior de Investigaciones Científicas, 28049
Cantoblanco, Spain

Zn2GeO4/SnO2 nanowire architecture: A correlative spatially resolved X-ray
study

Jaime Dolado, Fernanda Malato, Jaime Segura-Ruiz, María Taeño, Irina
Snigireva, Pedro Hidalgo, Bianchi Méndez, and Gema Martínez-Criado

Synopsis
Herein we present results on a correlative hard X-ray study of Zn2GeO4/SnO2
nanowire heterostructures formed by Plateau-Rayleigh instability. The multi-
technique synchrotron approach allows unraveling the elemental composition,
optical processes and short-range order, making it possible to determine them
independently and spatially resolved at the nanometer length scale. This study
highlight the role of diffused impurities on the optical properties of nano-
heterostructures and open new avenues for further heterogeneity studies using
the X-ray nanoprobe.

Nanowires represent key building blocks for high density device-based
nanotechnology. They are critical elements to implement the “bottom-up”
approach to nanoscale fabrication, whereby well-defined nanosized
architectures with unique physical properties are assembled into electronic
and photonic devices. Moreover, by manipulating growth kinetics it is possible
today to create more complex nanoscale heterostructures. The introduction of
sequential branching, for example, with variations in the composition and/or
doping will make possible novel electronic and photonic functionality into
these building blocks1, 2, 3. Likewise, owing to the Plateau-Rayleigh (PR)
instability, for example, Zn2GeO4/SnO2 heterostructures with skewer-like (SK)
morphologies have been recently synthesized4, emerging as direct competitors
to traditional wide bandgap materials for ultraviolet optoelectronic
nanodevices. Thus, elongated Zn2GeO4 nanorods (E${}_{g}\sim$ 4.5 eV) decorated
with a linear array of SnO2 crystals (E${}_{g}\sim$ 3.6 eV) in a necklace
fashion were made by an effective thermal evaporation method. Yet, the concept
underlying these building blocks rests on controlling the structural and
optical properties at the nanometer scale. Often, the oxide nano-
heterostructures suffer from native defects, heterogeneities, and/or local
structural imperfections that can drastically affect their optoelectronic
response5, 6, 7. So far, the structural and compositional quality of these
nanowire architectures has been mostly gauged by correlative transmission
electron microscopy (TEM) and energy dispersive X-ray spectroscopy (EDS)
measurements4, 8, 9, 10, 11. Additionally, indirect methods such as
cathodoluminescence measurements have been also applied to determine local
heterogeneities in these nanowire-based systems12, 13. However, the formation
of the individual components of these decorated nanostructures also needs a
systematic structural and chemical study at the nanometer scale. Although the
creation of crystalline phases driven by PR mechanism has been demonstrated,
so far there are many open questions. Just few examples include the role of
impurities on coupling formation and local atomic site configuration14, how
these necklace structures are joined to form decorated architectures15, 16,
17, whether there is a full control over composition, diffusion paths, and/or
structural modifications18, 19, 20, coexisting structural polytypes21, 22,
diffusion-driven effects23, phase separation problems24, radial or axial
chemical modulations22, as well as atomic scale variability due to
fluctuations in the growth rate4.

Consequently, apart from a few cases in which single crystal X-ray diffraction
measurements of nanowires could be recorded25, 26, 27, almost no information
about the crystal symmetry of individual nanostructures and their integration
into single architectures is available. Some works have reported the first
applications of nano-X-ray absorption spectroscopy to single nanowires28, 29,
30. Thus, if X-ray excited optical luminescence is added to an X-ray
absorption experiment, the challenge of a correlative study of single
nanostructures can be overcome using hard X-ray nanobeams. Therefore, here we
address such issues by characterizing a representative Zn2GeO4/SnO2 nano-
heterostructure formed by PR mechanism obtained in a one-step thermal
evaporation method4.

In this work we illustrate the use of a hard X-ray nanoprobe to study
decorated nanowire based architectures by multiple contrast mechanisms using
the nano-analysis beamline ID16B of the European Synchrotron Radiation
Facility (ESRF). Using a pair of Kirkpatrick-Baez Si mirrors, our study
involves the collection of X-ray fluorescence (XRF) and X-ray excited optical
luminescence (XEOL) emissions induced by a highly focused and intense hard
X-ray nanobeam. Both XRF and XEOL signals were recorded simultaneously.
Whereas the characteristics XRF is recorded with an energy dispersive Si drift
detector (deadtimes below 20%) at an angle of $\theta$ = (15 ± 5)º with
respect to the sample surface, the luminescence is detected normal to this
surface by a far-field optical collection system31. We apply this experimental
scheme to SK-structures composed by a Zn2GeO4 nanowires and tin oxide
crystallites produced by an effective thermal evaporation method 4. Compared
to most of the previous reports3, here we use a catalyst-free mechanism to
form the wide bandgap heterostructures. The droplets created by the PR
instability act as nucleation centres, giving rise to an unprecedented control
over the surface morphology18, 15.

Figure 1a shows the scanning electron microscopy (SEM) image of a single
Zn2GeO4/SnO2 nanoheterostructure taken with a LEO 1530 SEM equipment. It
consists of a straight central wire surrounded by well-faceted crystallites
(SK-like shape). The wire dimensions are 5 $\mu$m length and 100 nm diameter,
whereas the crystallites present sizes ranging between 500 - 800 nm. Figures
1b-c shows the X-ray nanoimaging results obtained by raster-scanning the SK-
structure in the X-ray nanobeam. The monochromatic X-ray beam was focused to
80 x 60 nm2 spot size ($V$ × $H$) with $\sim$1011 ph/s at room temperature in
air. The XRF maps were taken at 11.25 keV (above the absorption edge of Ge
located at 11.1031 keV) with a pixel size of 50 x 50 nm2 over a 3 × 5 $\mu$m2
sample area with an integration time of 1000 ms per point. In addition to the
major elements (Zn, Ge and Sn) of the Zn2GeO4/SnO2 heterostructure, the
average XRF spectrum (Figure 1b) also reveals unintentional dopants (e.g., Cr,
Fe, Ni and Cu) homogeneously distributed and likely originated from
contaminants in precursors or in the reactor. Figure 1c shows Zn (blue), Ge
(green) and Sn (red) superimposed distributions (individual elemental
distribution maps can be seen in the Supporting Information). The composition
of the nanowire axis is basically Zn2GeO4, as expected. However, the XRF maps
of the SnO2 crystallites exhibit an interesting sandwich like pattern. A
significant incorporation of Ge takes place within the SnO2 crystallites,
which could modify the short-range order and give rise to secondary phases or
defects, affecting accordingly the carrier recombination mechanisms. Moreover,
an axial variation of the Sn localization is observed within the crystallites,
being especially high in the fully faceted areas, whereas Ge is particularly
distributed in the regions not completely faceted (see Supporting Information
for further details). Finally, the Zn signal (Figure SI-2c in Supporting
Information) comes mainly from the nanowire with a much lesser degree
incorporated into the crystallites.

In good agreement with these results, we recently reported the observation of
a high Ge fraction within the SnO2 crystallites (from 8% to 17%) by TEM-EDS.
Our results point to the formation of a solid solution, such as Sn1-xGexO2,
which keeps the rutile crystal structure4. Likely, the faceted areas present a
lower amount of Ge (and a higher amount of Sn) because the Ge out-diffusion to
the Zn2GeO4 nanowire surface (which is arranged in periodic droplets due to PR
instability) acts as nucleation sites for the Sn crystals. The more faceted
the crystal is, the more Sn replaced Ge to form the rutile crystal phase.
Recent studies have shown that the nanowire coating in the PR-type processes
is determined by parameters such as the evolution of the internal crystalline
nanostructure over time, or the exchange of free atoms between the nanowire
surface and the enveloping near-surface layer22, 21. Therefore, these factors
could be involved in the different distribution of Ge and Sn in the SK-
structure.

Figure 1: (a) SEM image of SK-structure. (b) Average XRF spectrum recorded
over the SK- structure shown in (a) at 11.25 keV. (c) XRF map in RGB
visualization that depicts the XRF intensities of Sn (red), Ge (green), and Zn
(blue). Their colour brightness (light represents high counts, dark low
counts) indicates the intensity ranges (more details in the Supporting
Information).

We have also exploited the optical contrast mechanism to record spectrally and
spatially resolved XEOL signals simultaneously to XRF data for each point of
the sample (see Figure 2). After the primary X-ray absorption process, most of
the photogenerated electrons and holes recombine radiatively via secondary XRF
and XEOL pathways. Figure 2a presents the average XEOL spectrum, where a main
emission dominates in the visible range, and consists of two components
located at 1.98 and 2.20 eV (indicated in orange and green colours,
respectively). In addition, there is a weak band in the UV range located at
3.28 eV. The main observed band in the average XEOL spectrum corresponds to
the emission related to the transition of the trapped electrons from oxygen
vacancies to intrinsic surface states from the Sn1-xGexO2 crystallites. The
luminescence of SnO2 is quite complex, exhibiting a broad emission band in the
visible range, consisting of three components (orange, green and blue at 1.98,
2.25, and 2.58 eV, respectively) associated to oxygen vacancies32, 33. These
components were also reported from SnO2 nanoribbons by XEOL measurements
excited with both soft and hard X-rays34, 35. Regarding the weak UV band, this
emission could come from the Zn2GeO4 nanowire, which exhibits a wide emission
associated to oxygen vacancies in the UV range36. However, it has been
observed that SnO2 nanorods and nanoparticles exhibited an UV emission related
to surface states37, so it could also come from Sn1-xGexO2 particles.

Figures 2b and 2c display the spatial distribution of the respective XEOL
bands. The visible emission associated to oxygen vacancies is radially
distributed within the Sn1-xGexO2 crystallites, becoming particularly stronger
from the well-faceted areas, where XRF signal of Sn is higher. Previously we
observed that the crystallites of the SK-structures exhibit {110} facets4, in
which the formation of oxygen vacancies would be easy since they are the
planes of lower surface energy in SnO238. Therefore, this result suggests that
the higher amount of Ge in some areas of the Sn1-xGexO2 particles would affect
the amount of oxygen vacancies, with the consequent vanishing of the visible
emission. On the other hand, the less intense UV emission of unclear origin is
only present from one side of the Sn1-xGexO2 crystallites. The fact that the
UV emission recorded in this work is only observed from one side of the
Sn1-xGexO2 crystallite could suggest an orientation effect linked to the SnO2
crystal structure. Previous work have reported faceted dependent
cathodoluminescence in SnO2 microtubes, which would imply different
distribution of impurities and/or defects.32, 39 Alternatively, the UV
emission could come from the preferential incorporation of defects arising
from the out-diffusion of Ge or Zn from the Zn2GeO4 nanowire, which would
result in new electronic states within the bandgap leading to the UV emission.

In order to spatially resolve at the nanoscale these luminescence related
features with more details, Figure 2d shows the XEOL spectra acquired in two
different points on the Sn1-xGexO2 crystallites (A and B). In both cases, the
visible band related to SnO2 is the most pronounced contribution with some
intensity modulation, resulting from the behaviour of the Sn1-xGexO2
crystallites as optical cavities4. On the other hand, the UV band is detected
only from B, highlighting the inhomogeneous distribution of the radiative
centres associated likely with the different distribution of impurities. As it
is well known, the luminescence bands in semiconductors are related to the
density of surface or defect electronic states37, 40.

Figure 2: (a) Average XEOL spectrum recorded over the SK- structure shown in
Figure 1a. (b) and (c) XEOL integrated intensity maps over the energy ranges
defined by the colour bands in (a). (d) Normalized XEOL spectra taken from two
different points (A and B).

In order to gain further information, given the heterogeneous nature of both
composition and optical emissions, Figure 3 represents the normalized radial
line profiles taken along the SK-structure (see Figure 3a). Figures 3 b, c and
d show the XRF and XEOL profiles data obtained from each of the three
Sn1-xGexO2 crystallites along the perpendicular direction to the central
nanowire. For all cases, the Sn signal (in red colour) follows a similar trend
according to the crystal volume with a maximum intensity at the centre of the
Sn1-xGexO2 crystallites. The Ge signal (in green colour), on the other hand,
presents a different profile for the largest crystal compared to the other two
ones. The largest crystallite appears to be fully formed, producing a higher
intensity from the edges of the structure. The smaller crystallites are
conversely characterized by a more uniform Ge incorporation, with a Gaussian
like profile according to their morphological shape. Finally, the Zn profiles
(in blue colour) display a multipeak shape, indicating a pretty different and
particular behaviour within the crystallite volume. The results not only
reveal the incorporation of Zn within the Sn1-xGexO2 crystallites, but also
its preferential spatial location towards the centre and edges of the
crystallites.

Despite probing different sample depths (XEOL is much more surface sensitive
than XRF), the radial line profiles of the XEOL signals provide complementary
information about the three Sn1-xGexO2 crystallites. Compared to the XRF
signal, the spatial distribution of XEOL is not only governed mainly by the
beam spot size and penetration of the X-ray nanobeam within the sample (energy
dependent, leading to a finite generation volume), but also greatly by the
diffusion of the photogenerated carriers. Many factors can impact the
resulting profile related to the sample itself in terms of sample morphology,
composition, internal absorption and/or reflection mechanisms, as well as
secondary photon emissions. The visible band profile (in orange) follows the
crystal volume with a Gaussian like shape well correlated with Sn [with a
smaller full width at half maximum (FWHM) due to diffusion effects]. The
profile of the UV emission (in violet) presents a quite narrower line width
peaking at the left edge of the Sn1-xGexO2 crystallite, matching one of the
maxima in the Zn profile. The direct matching and interrelation between the UV
band and Zn localization is also supported by axial line profiles recorded
along the SK-structure (see Figure SI-3 in Supporting Information). It is
important to emphasize that the UV emission is not observed in the bare
Zn2GeO4 nanowire, ruling completely out their connection.

In short, the correlative study of the XRF and XEOL line profiles indicates
that for the tin oxide crystallite fully formed (Figure 3b), likely there is
an additional colocalization of Ge and Zn elements. On the other hand, for all
the cases (Figure 3 b, c, d) the UV emission is in close relationship with the
Zn distribution mainly concentrated at one of the edges of the Sn1-xGexO2
crystallites. This finding suggests that the UV emission could come from
recombination centres linked to complexes associated to the presence of Zn.

Figure 3: (a) SK-structure on which the line profiles were collected. (b-d)
Normalized XRF line profiles (top) for Sn (red symbols), Ge (green symbols)
and Zn (blue symbols), and XEOL line profiles (bottom) for visible (orange
symbols) and UV (purple symbols) emissions. All profiles were collected along
the white dashed lines drawn on the SK-structure shown on the left-hand side,
perpendicular to the central nanowire.

Given the key role of Zn within the Sn1-xGexO2 crystallites, in order to get a
deeper insight into its local structure in the SK-structure, XANES data were
taken around the Zn K-edge with nanometer spatial resolution. The measurements
were performed with a beam size of 80 x 60 nm2 using a Si(111) double crystal
monochromator with an energy resolution about 1 eV. Figure 4b shows the
spatially resolved XANES data acquired in XRF detection mode over three
different zones of the SK-structure. The XANES signal is analogous to the
partial density of the empty states of the absorbing atoms; namely, to the
energy band structure for crystalline material. In particular, XANES collected
at the K edge reveals the $p$-partial density of states in the conduction
band. Here, two spectral XANES regions can be distinguished: a prominent white
line assigned mainly to the 1$s$ → 4$p$ dipolar transitions, and the post-edge
resonances, which are rather due to multiple scattering effects. The three
XANES spectra collected on the Sn1-xGexO2 crystallite show an intense and
narrow peak at 9670.2 eV, and the post-edge are similar in all three cases,
indicating a similar short-range order around Zn atoms in the different areas.

An empirical and common practice is to assume that the XANES spectrum from an
unknown local structural order can be understood as a linear superposition of
the spectra of two or more known samples. Therefore, for comparison XANES
spectra around the Zn or Sn K-edges of three model compounds are plotted in
Figure 4c, in addition to one of those acquired from the Sn1-xGexO2
crystallites. In ZnO41, Zn atoms are in tetrahedral sites. In the
Sn0.96Zn0.04O2 alloy, on the other hand, Zn is introduced into the SnO2
structure, inducing numerous lattice defects with no big change of the rutile
crystal structure (octahedral environment)42. Finally, the XANES spectrum
around Sn K-edge acquired on SnO2 with rutile structure is shown43, where each
tin atom is expected to be coordinated by six oxygen atoms. Our results
strongly suggest a substitutional Zn incorporation into the Sn sites. A direct
comparison of XANES data from the post-edge region around the Zn and Sn
K-edges for the Sn1-xGexO2 and SnO2, respectively, indicates the predominant
octahedral coordination of Zn atoms in the Sn1-xGexO2 structure. In addition,
theoretical XANES simulations of ZnO in a rutile-like environment show similar
spectral shapes, corroborating our results44. Thus, the fact that the XANES
spectrum of the Sn1-xGexO2 crystallites is significantly different from the
XANES data of the Sn0.96Zn0.04O2 alloy suggests that Zn is incorporated as a
dopant into the rutile structure, without causing major lattice distortions.
In addition, the white line of the SK-structure is shifted to higher energies
with respect to those of ZnO and the Sn0.96Zn0.04O2 alloy, indicating an
increase in Zn coordination.

In short, XANES results point to Zn atoms in octahedral coordination within
the Sn1-xGexO2 crystallites, indicating that it is being incorporated into the
crystal lattice possibly as a dopant replacing Sn in the rutile structure.
This substitutional Zn defect in the SnO2 lattice may lead to an electronic
state within the gap responsible for the UV emission above observed by XEOL.

Figure 4: (a) Crystallite of the SK-structure in which the XANES spectra shown
in (b) were acquired. Each spectrum was collected on the point of the same
colour marked on the SK-structure. (c) XANES data recorded around the Zn or Sn
K-edges of reference materials with tetrahedral (ZnO41), distorted octahedral
(Sn0.96Zn0.04O242) and octahedral (SnO243) coordination, in addition to the
one acquired in the SK-structure (at the bottom of the graph). The spectra
were shifted vertically for clarity.

In summary, we showed the use of a hard X-ray nanoprobe with optical,
elemental and local structure specificity for the study of Zn2GeO4/SnO2
heterostructures grown by an effective thermal evaporation method. Both the
compositional and optical distributions were probed with nanometre resolution.
Elemental maps showed that Ge and Zn atoms diffused from the central Zn2GeO4
nanowire to the Sn1-xGexO2 crystallite with an inhomogeneous character. On the
other hand, XEOL maps revealed a homogeneous repartition of the dominant SnO2
related visible emission whereas the UV emission is only present on one side
of the crystallite. A correlative study between XRF and XEOL line profiles
allowed us to associate the presence of the UV band with a higher Zn
concentration in certain areas of the Sn1-xGexO2 crystallite. XANES data
collected in different areas of the crystallite around the Zn K-edge confirmed
possible Zn octahedral coordination in the crystallites of the SK-structure.
This finding could likely indicate the substitutional incorporation of Zn into
the Sn sites in the rutile structure, leading to an electronic state within
the bandgap responsible for the UV emission. Thus, this work shows how to
complete the picture through correlative X-ray nano-analysis and how to
develop wide bandgap nanowire-based architectures with bespoke properties at
the nanoscale.

##  1 Associated content

Both distribution maps and normalized line profiles collected along the SK-
structure from XRF and XEOL signals were processed with PyMca software.

This work has been supported by MICINN project RTI2018-097195-B-I00 and by
European Union’s Horizon 2020 research and innovation programme under the
Marie Skłodowska-Curie grant agreement n° 956548, project Quantimony. We thank
the ESRF for the beam time allocated.

Vertical and horizontal X-ray beam profiles; Sn, Ge and Zn XRF maps of the SK-
structure; and normalized XRF and XEOL line profiles collected along the SK-
structure parallel to the central nanowire. This material is available free of
charge.

## References

  * Lauhon et al. 2002 Lauhon, L. J.; Gudiksen, M. S.; Wang, D.; Lieber, C. M. Epitaxial core–shell and core–multishell nanowire heterostructures. _nature_ 2002, _420_ , 57–61
  * Lieber 2003 Lieber, C. M. Nanoscale science and technology: building a big future from small things. _MRS bulletin_ 2003, _28_ , 486–491
  * Jia et al. 2019 Jia, C.; Lin, Z.; Huang, Y.; Duan, X. Nanowire electronics: from nanoscale to macroscale. _Chemical reviews_ 2019, _119_ , 9074–9135
  * Dolado et al. 2019 Dolado, J.; Renforth, K. L.; Nunn, J. E.; Hindsmarsh, S. A.; Hidalgo, P.; Sánchez, A. M.; Méndez, B. Zn2GeO4/SnO2 Nanowire Heterostructures Driven by Plateau-Rayleigh Instability. _Crystal Growth & Design_ 2019, _20_ , 506–513
  * Dayeh et al. 2011 Dayeh, S. A.; Wang, J.; Li, N.; Huang, J. Y.; Gin, A. V.; Picraux, S. T. Growth, defect formation, and morphology control of germanium–silicon semiconductor nanowire heterostructures. _Nano letters_ 2011, _11_ , 4200–4206
  * Huijben et al. 2013 Huijben, M.; Koster, G.; Kruize, M. K.; Wenderich, S.; Verbeeck, J.; Bals, S.; Slooten, E.; Shi, B.; Molegraaf, H. J.; Kleibeuker, J. E., et al. Defect engineering in oxide heterostructures by enhanced oxygen surface exchange. _Advanced functional materials_ 2013, _23_ , 5240–5248
  * Li et al. 2016 Li, Y.; Huang, L.; Li, B.; Wang, X.; Zhou, Z.; Li, J.; Wei, Z. Co-nucleus 1D/2D heterostructures with Bi2S3 nanowire and MoS2 monolayer: One-step growth and defect-induced formation mechanism. _ACS nano_ 2016, _10_ , 8938–8946
  * Jung et al. 2007 Jung, Y.; Ko, D.-K.; Agarwal, R. Synthesis and structural characterization of single-crystalline branched nanowire heterostructures. _Nano letters_ 2007, _7_ , 264–268
  * Liu et al. 2008 Liu, Z.; Elbert, D.; Chien, C.-L.; Searson, P. C. FIB/TEM characterization of the composition and structure of core/shell Cu- Ni nanowires. _Nano letters_ 2008, _8_ , 2166–2170
  * Lim et al. 2009 Lim, S. K.; Brewster, M.; Qian, F.; Li, Y.; Lieber, C. M.; Gradecak, S. Direct correlation between structural and optical properties of III-V nitride nanowire heterostructures with nanoscale resolution. _Nano letters_ 2009, _9_ , 3940–3944
  * Gao et al. 2019 Gao, H.; Sun, W.; Sun, Q.; Tan, H. H.; Jagadish, C.; Zou, J. Compositional varied core–shell InGaP nanowires grown by metal–organic chemical vapor deposition. _Nano letters_ 2019, _19_ , 3782–3788
  * Hetzl et al. 2016 Hetzl, M.; Kraut, M.; Winnerl, J.; Francaviglia, L.; Döblinger, M.; Matich, S.; Fontcuberta i Morral, A.; Stutzmann, M. Strain-induced band gap engineering in selectively grown GaN–(Al, Ga)N core–shell nanowire heterostructures. _Nano letters_ 2016, _16_ , 7098–7106
  * Songmuang et al. 2016 Songmuang, R.; Giang, L. T. T.; Bleuse, J.; Den Hertog, M.; Niquet, Y.-M.; Dang, L. S.; Mariette, H. Determination of the optimal shell thickness for self-catalyzed GaAs/AlGaAs core–shell nanowires on silicon. _Nano letters_ 2016, _16_ , 3426–3433
  * Xu et al. 2018 Xu, S.; Li, P.; Lu, Y. In situ atomic-scale analysis of Rayleigh instability in ultrathin gold nanowires. _Nano Research_ 2018, _11_ , 625–632
  * Day et al. 2016 Day, R. W.; Mankin, M. N.; Lieber, C. M. Plateau–Rayleigh crystal growth of nanowire heterostructures: strain-modified surface chemistry and morphological control in one, two, and three dimensions. _Nano letters_ 2016, _16_ , 2830–2836
  * Li et al. 2017 Li, W.; Jia, Q.; Liu, X.; Zhang, J. Large scale synthesis and photoluminescence properties of necklace-like SiC/SiOx heterojunctions via a molten salt mediated vapor reaction technique. _Ceramics International_ 2017, _43_ , 2950–2955
  * Chu et al. 2018 Chu, Y.; Jing, S.; Yu, X.; Zhao, Y. High-temperature plateau–Rayleigh growth of beaded SiC/SiO2 nanochain heterostructures. _Crystal Growth & Design_ 2018, _18_ , 2941–2947
  * Day et al. 2015 Day, R. W.; Mankin, M. N.; Gao, R.; No, Y.-S.; Kim, S.-K.; Bell, D. C.; Park, H.-G.; Lieber, C. M. Plateau–Rayleigh crystal growth of periodic shells on one-dimensional substrates. _Nature nanotechnology_ 2015, _10_ , 345–352
  * Xue et al. 2016 Xue, Z.; Xu, M.; Zhao, Y.; Wang, J.; Jiang, X.; Yu, L.; Wang, J.; Xu, J.; Shi, Y.; Chen, K., et al. Engineering island-chain silicon nanowires via a droplet mediated Plateau-Rayleigh transformation. _Nature communications_ 2016, _7_ , 1–9
  * Liu et al. 2020 Liu, G.-S.; He, M.; Wang, T.; Wang, L.; He, Z.; Zhan, R.; Chen, L.; Chen, Y.; Yang, B.-R.; Luo, Y., et al. Optically Programmable Plateau–Rayleigh Instability for High-Resolution and Scalable Morphology Manipulation of Silver Nanowires for Flexible Optoelectronics. _ACS Applied Materials & Interfaces_ 2020, _12_ , 53984–53993
  * Gorshkov et al. 2020 Gorshkov, V. N.; Tereshchuk, V. V.; Sareh, P. Diversity of anisotropy effects in the breakup of metallic FCC nanowires into ordered nanodroplet chains. _CrystEngComm_ 2020, _22_ , 2601–2611
  * Walbert et al. 2020 Walbert, T.; Muench, F.; Yang, Y.; Kunz, U.; Xu, B.-X.; Ensinger, W.; Molina-Luna, L. In Situ Transmission Electron Microscopy Analysis of Thermally Decaying Polycrystalline Platinum Nanowires. _ACS nano_ 2020, _14_ , 11309–11318
  * Garín et al. 2017 Garín, M.; Jin, C.; Cardador, D.; Trifonov, T.; Alcubilla, R. Controlling Plateau-Rayleigh instabilities during the reorganization of silicon macropores in the Silicon Millefeuille process. _Scientific reports_ 2017, _7_ , 1–11
  * Allaire et al. 2021 Allaire, R. H.; Kondic, L.; Cummings, L. J.; Rack, P. D.; Fuentes-Cabrera, M. The role of phase separation on Rayleigh-Plateau type instabilities in alloys. _The Journal of Physical Chemistry C_ 2021, _125_ , 5723–5731
  * Bussone et al. 2015 Bussone, G.; Schäfer-Eberwein, H.; Dimakis, E.; Biermanns, A.; Carbone, D.; Tahraoui, A.; Geelhaar, L.; Haring Bolívar, P.; Schülli, T. U.; Pietsch, U. Correlation of electrical and structural properties of single as-grown GaAs nanowires on Si (111) substrates. _Nano letters_ 2015, _15_ , 981–989
  * Li et al. 2020 Li, N.; Labat, S.; Leake, S. J.; Dupraz, M.; Carnis, J.; Cornelius, T. W.; Beutier, G.; Verdier, M.; Favre-Nicolin, V.; Schülli, T. U., et al. Mapping Inversion Domain Boundaries along Single GaN Wires with Bragg Coherent X-Ray Imaging. _ACS nano_ 2020, _14_ , 10305–10312
  * Mostafavi Kashani et al. 2021 Mostafavi Kashani, S. M.; Dubrovskii, V. G.; Baumbach, T.; Pietsch, U. In situ monitoring of MBE growth of a single self-catalyzed GaAs nanowire by X-ray diffraction. _The Journal of Physical Chemistry C_ 2021, _125_ , 22724–22732
  * Segura-Ruiz et al. 2011 Segura-Ruiz, J.; Martinez-Criado, G.; Chu, M.; Geburt, S.; Ronning, C. Nano-X-ray absorption spectroscopy of single Co-implanted ZnO nanowires. _Nano letters_ 2011, _11_ , 5322–5326
  * Segura-Ruiz et al. 2014 Segura-Ruiz, J.; Martínez-Criado, G.; Denker, C.; Malindretos, J.; Rizzi, A. Phase Separation in Single InxGa1-xN Nanowires Revealed through a Hard X-ray Synchrotron Nanoprobe. _Nano letters_ 2014, _14_ , 1300–1305
  * Martínez-Criado et al. 2014 Martínez-Criado, G.; Segura-Ruiz, J.; Alén, B.; Eymery, J.; Rogalev, A.; Tucoulou, R.; Homs, A. Exploring Single Semiconductor Nanowires with a Multimodal Hard X-ray Nanoprobe. _Advanced Materials_ 2014, _26_ , 7873–7879
  * Martínez-Criado et al. 2016 Martínez-Criado, G.; Villanova, J.; Tucoulou, R.; Salomon, D.; Suuronen, J.-P.; Labouré, S.; Guilloud, C.; Valls, V.; Barrett, R.; Gagliardini, E., et al. ID16B: a hard X-ray nanoprobe beamline at the ESRF for nano-analysis. _Journal of synchrotron radiation_ 2016, _23_ , 344–352
  * Maestre et al. 2004 Maestre, D.; Cremades, A.; Piqueras, J. Cathodoluminescence of defects in sintered tin oxide. _Journal of applied physics_ 2004, _95_ , 3027–3030
  * Miller et al. 2017 Miller, D. R.; Williams, R. E.; Akbar, S. A.; Morris, P. A.; McComb, D. W. STEM-cathodoluminescence of SnO2 nanowires and powders. _Sensors and Actuators B: Chemical_ 2017, _240_ , 193–203
  * Zhou et al. 2006 Zhou, X.; Heigl, F.; Murphy, M.; Sham, T.; Regier, T.; Coulthard, I.; Blyth, R. Time-resolved X-ray excited optical luminescence from SnO2 nanoribbons: Direct evidence for the origin of the blue luminescence and the role of surface states. _Applied physics letters_ 2006, _89_ , 213109
  * Zhou et al. 2008 Zhou, X.; Zhou, J.; Murphy, M.; Ko, J.; Heigl, F.; Regier, T.; Blyth, R.; Sham, T. The effect of the surface of SnO2 nanoribbons on their luminescence using X-ray absorption and luminescence spectroscopy. _The Journal of chemical physics_ 2008, _128_ , 144703
  * Dolado et al. 2020 Dolado, J.; Hidalgo, P.; Méndez, B. The role of surface properties in the cathodoluminescence of Zn2GeO4/SnO2 nanowire heterostructures. _Materials letters_ 2020, _275_ , 128152
  * Kar et al. 2011 Kar, A.; Kundu, S.; Patra, A. Surface defect-related luminescence properties of SnO2 nanorods and nanoparticles. _The Journal of Physical Chemistry C_ 2011, _115_ , 118–124
  * Oviedo and Gillan 2000 Oviedo, J.; Gillan, M. Energetics and structure of stoichiometric SnO2 surfaces studied by first-principles calculations. _Surface Science_ 2000, _463_ , 93–101
  * Yuan et al. 2006 Yuan, X.; Lazzarini, L.; Salviati, G.; Zha, M.; Sekiguchi, T. Cathodoluminescence characterization of SnO2 nanoribbons grown by vapor transport technique. _Materials science in semiconductor processing_ 2006, _9_ , 331–336
  * Reshchikov and Morkoç 2005 Reshchikov, M. A.; Morkoç, H. Luminescence properties of defects in GaN. _Journal of applied physics_ 2005, _97_ , 5–19
  * Haug et al. 2011 Haug, J.; Chassé, A.; Dubiel, M.; Eisenschmidt, C.; Khalid, M.; Esquinazi, P. Characterization of lattice defects by X-ray absorption spectroscopy at the Zn K-edge in ferromagnetic, pure ZnO films. _Journal of Applied Physics_ 2011, _110_ , 063507
  * Liu et al. 2010 Liu, X.; Iqbal, J.; Wu, Z.; He, B.; Yu, R. Structure and room-temperature ferromagnetism of Zn-doped SnO2 nanorods prepared by solvothermal method. _The Journal of Physical Chemistry C_ 2010, _114_ , 4790–4796
  * Grandjean et al. 2004 Grandjean, D.; Benfield, R. E.; Nayral, C.; Maisonnat, A.; Chaudret, B. EXAFS and XANES study of a pure and Pd doped novel Sn/SnOx nanomaterial. _The Journal of Physical Chemistry B_ 2004, _108_ , 8876–8887
  * Zhu et al. 2014 Zhu, Q.; Ma, Q.; Buchholz, D.; Chang, R.; Bedzyk, M. J.; Mason, T. O. Structural and physical properties of transparent conducting, amorphous Zn-doped SnO2 films. _Journal of Applied Physics_ 2014, _115_ , 033512