# Statistics-aware Audio-visual DeepFake Detector

###### Abstract

In this paper, we propose an enhanced audio-visual deep detection method.
Recent methods in audio-visual deepfake detection mostly assess the
synchronization between audio and visual features. Although they have shown
promising results, they are based on the maximization/minimization of isolated
feature distances without considering feature statistics. Moreover, they rely
on cumbersome deep learning architectures and are heavily dependent on
empirically fixed hyperparameters. Herein, to overcome these limitations, we
propose: (1) a statistical feature loss to enhance the discrimination
capability of the model, instead of relying solely on feature distances; (2)
using the waveform for describing the audio as a replacement of frequency-
based representations; (3) a post-processing normalization of the fakeness
score; (4) the use of shallower network for reducing the computational
complexity. Experiments on the DFDC and FakeAVCeleb datasets demonstrate the
relevance of the proposed method.

Index Terms—  deepfake detector, multi-modal, audio-visual, distribution,
similarity

##  1 Introduction

The growth of the audio-visual deepfake technology has recently raised major
concerns. Deepfakes, which are person-centric manipulated media, are becoming
increasingly realistic. As a result, there is a pressing need to address their
potential misuse, such as spreading misinformation and identity theft [1], by
introducing effective audio-visual deepfake detection solutions.

Fig. 1: Histograms of features extracted from three real and three fake
samples using an enhanced version of MDS111The baseline is the enhanced MDS
without the statistics-aware loss discussed in Section 3.1. MDS [2] considers
feature distance of whole face region frames and audio to detect deepfake..
Each column represents the feature values in different range.

The most successful audio-visual deepfake detection methods involve the
identification of inconsistencies between audio and visual features for
isolating deepfakes [2, 3, 4]. They usually enforce a high distance between
audio and visual features in the presence of fake data, while they aim to
reduce this distance when dealing with real data.

Despite being promising, these approaches might be improved from several
perspectives: (1) existing methods mostly rely on isolated distances between
audio and visual features extracted from one sample, ignoring feature
statistics [2, 3, 5]. As shown in Fig. 1, histograms of visual and audio
features extracted using our baseline111The baseline is the enhanced MDS
without the statistics-aware loss discussed in Section 3.1. MDS [2] considers
feature distance of whole face region frames and audio to detect deepfake.
from both real and fake test samples exhibit a similar pattern. In both cases,
the audio and visual feature distributions are very similar, making the
discrimination between real and fake data challenging; (2) existing methods
are heavily dependent on empirically fixed hyperparameters. For instance, to
model the audio modality, most deepfake detection methods employ a frequency
representation [2, 6, 7], potentially ignoring discriminative cues and
necessitating a phase of hyper-parameter tuning, as highlighted in [8]. In
addition, deepfakes are usually determined when the distance between the audio
and visual features referred to as fakeness score exceeds a predefined
threshold, similar to [2]; and (3) Lastly, several audio-visual models
incorporate very deep architectures, which can be computationally expensive
[2, 6], making them unsuitable for numerous real-life applications.

To address these challenges, we propose a novel shallow network called
Statistics-aware Audio-visual Deepfake Detector (SADD) for audio-visual
deepfake detection. In fact, recent works suggested that deepfake artifacts
can be effectively modeled in low-level features extracted with shallower
networks [9, 10]. To enhance the model’s discrimination capability, instead of
relying solely on a feature distance loss, we integrate a second loss that
estimates the distance between first-order statistics of audio and visual
feature distributions. This loss is then maximized for fake data, and
minimized otherwise. Moreover, the raw waveform is employed as input to
mitigate potential limitations associated with frequency-based
representations. Finally, to avoid experimentally fixing a classification
threshold, we suggest a fakeness score normalization. The experiments carried
out on two well-known datasets, namely DFDC and FakeAVCeleb, show that our
method necessitates a lower computational cost as compared to existing methods
while maintaining competitive deepfake detection capabilities.

In summary, our contributions can be outlined as follows: (1) Taking the model
from [2], we enhance it by changing the audio input to waveform, reducing the
network depth, and modifying the post-processing steps; (2) We identify an
issue in such models, specifically the lack of distinguishable differences
between real and fake distributions; (3) We introduce a novel loss function to
enforce a desirable separation of audio-visual distributions for fake data
and, conversely, promote their similarity for real data; (4) We conduct
extensive evaluations of our model on the widely-used DFDC [11] benchmark
dataset, demonstrating superior performance compared to state-of-the-art (SoA)
methods. Additionally, we observe an increase in generalization capabilities
when applying our proposed component in a cross-dataset setting, utilizing the
FakeAVCeleb dataset [12].

Organization of the paper: Section 2 describes the related works. Section 3
introduces the proposed approach. Section 4 reports the experiments. Section 5
concludes the paper.

##  2 Related works

Several researchers have developed deepfake detectors for various modalities,
such as images [13, 10] and audios [14, 15]. Nevertheless, these methods are
mostly based on singular modalities, ignoring the complementary information
incorporated in multi-modal data. In fact, leveraging multiple modalities is a
promising way to obtain more robust deepfake detectors. Hence, numerous audio-
visual deepfake detectors have been recently introduced to simultaneously
consider audio and visual information extracted from a given video.

Existing audio-visual methods can be categorized into three classes, namely,
identity-based, fusion-based, and inconsistency-based. Identity-based methods
aim to detect manipulated videos of specific individuals. Agarwal et al.[16]
explore the correlation between observed behavior and speech, enabling not
only the detection of deepfakes but also the identification of doppelgangers
(real persons with highly similar appearances). Cheng et al.[17] attempt to
ascertain if a voice matches a facial image to identify deepfakes. However,
relying solely on images as visual cues may restrict detection to individuals
within the dataset. Cai et al.[18] define antonyms of real words as fake data.
Nonetheless, the meaning of a sentence is heavily dependant on a person’s
personality. While detecting deepfakes of specific individuals has its
specific applications, our interest lies in a more generic approach capable of
robustly detecting deepfakes of unknown persons.

Alternatively, several fusion-based methods simply concatenate information
from both audio and visual sources. Yang et al.[6] combines audio and visual
features using cross-attention after separately extracting audio and visual
features using a transformer-based model. Zhou and Lim [7] aggregate audio and
visual multi-level features after applying cross-attention. Lewis et al.[19]
combine multiple types of visual information (e.g., lip reading, color) and
audio (e.g., spectrogram, phoneme) features using Long Short-Term Memory
networks and a multi-layer perceptron. Ilyas et al.[20] combines the decisions
of the audio and visual models using a voting mechanism, which can be more
sensitive to false positives, as a false positive from either the visual or
audio model may result in an overall false positive prediction.

Inconsistency-based methods explicitly focus on the synchronization between
audio and visual elements to detect deepfakes. The assumption is that, in fake
data, the audio fails to synchronize correctly with visual cues. Gu et al.[3]
focus solely on the lips region as visual input, using a contrastive loss to
train the (a)synchronization of visual and audio features. However, relying
exclusively on the lips may lead to the omission of other crucial facial
regions. Mittal et al.[5] specifically investigate the mismatch between
emotions that are visually perceived and those conveyed through audio/speech.
Nevertheless, this approach relies on emotion recognition models. Feng et
al.[4] employ a synchronization model to predict whether a pair of visual and
audio inputs is synchronized. This model utilizes pseudo-fake data generated
by translating the real audio/visual information to the time domain, assuming
that only translation mismatches exist in fake data. Chugh et al.[2] minimizes
the distance between audio and visual features extracted from real data while
maximizing it in other cases. To enforce a minimum distance on real data while
maximizing it in fake data, MDS utilizes contrastive learning. Contrastive
learning has been applied in various domains, including face verification [21]
and lip reading [22]. Compared to the aforementioned methods, MDS is
straightforward, independent of other models, utilizes the entire face, and
does not solely rely on translation synchronization assumptions. Nevertheless,
as mentioned in Section 1, MDS only align pair of features independently,
without considering their statistics which can lead to limited separation
between audio and visual features extracted from fake data. As a solution, we
introduce a statistics-aware loss to further enhance the separability between
audio and visual features when encountering fake data.

##  3 Methodology

Fig. 2: Our method consists of visual and audio feature extractors that
extract respectively features $\mathbf{f}^{v}$ and $\mathbf{f}^{a}$ from image
sequences $\mathbf{I}^{v}$ and audio waveforms $\mathbf{I}^{a}$. Two separate
classification layers are integrated on top of each extractor. The model is
trained using a cross-entropy loss for each network, along with a feature
distance loss. To enhance feature discrimination between real and fake data,
we introduce an additional feature statistics-aware loss.

We propose a multi-modal deepfake detector which exploits the inconsistencies
between audio and visual features for detecting deepfakes. Fig. 2 gives an
overview of the proposed approach. In particular, the architecture proposed by
Chugh et al.[2] (Section 3.1) is enhanced. Moreover, we employ an additional
loss to quantify the first-order statistics of real and fake data, hence
improving the discriminative power of audio and visual features (Section 3.2).

###  3.1 Enhanced Modality Dissonance Score

To better leverage the inconsistencies between audio and visual cues for
deepfake detection, we seek a model that is simple, free from assumptions,
independent of other models, and capable of utilizing entire facial regions
without discarding potentially important artifacts. Such a model offers
flexibility for improvements. We find the Modality Dissonance Score (MDS) [2]
approach to fit our requirements and propose some modifications for enhancing
it. To reduce the computational cost, we first suggest the use of a shallow
version of MDS by reducing its depth. As mentioned in Section 1, several works
have shown that low-level representations are sufficient for effectively
modeling deepfake artifacts [2], implying the relevance of shallower networks.
Additionally, to reduce the need for hyperparameter tuning related to audio
input conversion, we use the waveform representation as audio input and
introduce a refinement strategy of the fakeness score during inference. More
details about these steps are provided in next subsections.

####  3.1.1 Architecture

We extract features separately for audio input $\mathbf{I}^{a}$ and visual
input $\mathbf{I}^{v}$ using the audio and the visual feature extractor
denoted as $\mathfrak{A}$ and $\mathfrak{V}$, respectively, as follows,

$\mathbf{f}^{a}=\mathfrak{A}(\mathbf{I}^{a})\quad\text{and}\quad\mathbf{f}^{v}=\mathfrak{V}(\mathbf{I}^{v})\text{,}$
(1)

where $f^{a}$ and $f^{v}$ are vectors of the same size.

In [2], $\mathbf{I}^{a}$ is represented by a Mel spectogram. However,
converting the waveform to a Mel spectogram may introduce limitations from the
conversion process, e.g., the necessity of manually fixing hyperparameters
such as the window size, the shift, length, and the feature dimension which
can be sub-optimal [8]. As an alternative, we represent the features
$\mathbf{I}^{a}$ using the raw waveform. Consequently, $\mathfrak{A}$ is
adapted to comply with this kind of input. Specifically, we use 1D
convolutions instead of the 2D convolution layers employed in [2].
Furthermore, we posit that deepfake artifacts are present in shallow features,
hence opting for a reduced number of convolution layers. Specifically, the
size of the visual extractor is reduced from five to three convolution blocks.
Meanwhile, the audio extractors reduced from four to two convolution blocks.
However, extracting features from shallower network can lead to larger feature
maps, which can increase the computational cost in the fully connected layers.
Therefore, we retain only the max pooling from the later blocks of the deep
network to downsample the extracted feature maps.

For real/fake classification, two separated fully connected layers
$\mathfrak{C}^{a}$ and $\mathfrak{C}^{v}$ are considered on top of the audio
and visual extractors as detailed below, respectively,

$\mathbf{y}^{a}=\mathfrak{C}^{a}(\mathbf{f}^{a})\quad\text{and}\quad\mathbf{y}^{v}=\mathfrak{C}^{v}(\mathbf{f}^{v})\text{,}$
(2)

where $\mathbf{y}^{a}$ and $\mathbf{y}^{v}$ represent the class predictions
(i.e., real/fake) for the audio and visual branches, respectively.

####  3.1.2 Training

To train the proposed network, the three losses introduced in [2], namely, the
audio cross-entropy loss $\mathcal{L}^{a}$, the visual cross-entropy loss
$\mathcal{L}^{v}$, and the contrastive-based distance loss $\mathcal{L}^{c}$
are used. Moreover, we propose an additional based loss taking into account
the feature statistics denoted as $\mathcal{L}^{s}$. More details about the
calculation of $\mathcal{L}^{s}$ is given in Section 3.2. The total loss
$\mathcal{L}$ is then computed as follows,

$\mathcal{L}=\mathcal{L}^{v}+\mathcal{L}^{a}+\mathcal{L}^{c}+\alpha\mathcal{L}^{s}\text{,}$
(3)

where $\mathcal{L}^{a}$ and $\mathcal{L}^{v}$ represents the cross-entropy
loss between $\mathbf{y}^{a}$ and $\hat{\mathbf{y}}$, and $\mathbf{y}^{v}$ and
$\hat{\mathbf{y}}$, respectively. The variable $\alpha$ is a weighting
hyperparameter. Despite the presence of noisy labels (i.e., $\hat{\mathbf{y}}$
is “fake” in audio-visual may mean only-audio, only-visual, or both modalities
being fake), Chugh et al.[2] experimentally demonstrates the importance of
these cross-entropy losses.

The contrastive loss enforces consistent audio-visual features for real data
and inconsistency for fake data as follows,

$\mathcal{L}^{c}=\begin{cases}(d^{\mathbf{f}})^{2}&\text{if }\mathbf{y}\text{
is real}\text{,}\\\ (\max(m-d^{\mathbf{f}},0))^{2}&\text{if }\mathbf{y}\text{
is fake}\text{,}\\\ \end{cases}$ (4)

where $m=0.99$ is the margin, as defined in [2], and $d^{\mathbf{f}}$ is the
squared $L_{2}$ distance between $\mathbf{f}^{v}$ and $\mathbf{f}^{a}$.

####  3.1.3 Inference

The fakeness score denoted as $\mu^{d}$ of a test video is determined based on
the mean distance calculated for all subsequences contained in the test video,
as expressed by the following formula,

$\mu^{d}=\frac{1}{n_{s}}\sum_{i=1}^{n_{s}}d^{\mathbf{f}}_{i}\text{,}$ (5)

where $n_{s}$ represents the number of subsequences in a video and
$d^{\mathbf{f}}_{i}$ is $d^{\mathbf{f}}$ of the $i$-th sequence.

In order to constrain the fakeness score to vary between 0 to 1 for a more
intuitive prediction, Chugh et al.[2] implement a post-processing to derive
the final $s$ score as follows,

$s=\begin{cases}1\text{ (real) }&\text{if }\mu^{d}<\tau\text{,}\\\ 0\text{
(fake) }&\text{if }\mu^{d}\geq\tau\text{,}\\\ \end{cases}$ (6)

where $\tau$ is selected by finding the threshold with the highest performance
in the training set.

However, exhaustively searching for the best threshold value may hide the full
potential of the model. Instead, we min-max normalize the score using the
minimum and maximum values of $\mu^{d}$ from the training data. Thus, for a
given test video, the score is calculated as,

$s=\frac{\mu^{d}-\mu^{d}_{min\\_train}}{\mu^{d}_{max\\_train}-\mu^{d}_{min\\_train}}\text{,}$
(7)

where $\mu^{d}_{min\\_train}$ represents the minimum of $\mu^{d}$ across the
training set, and $\mu^{d}_{max\\_train}$ is the maximum. The score is then
clipped to ensure it falls within the range of $0$ to $1$ as follows,

$s=\min(1,\max(s,0))\text{.}$ (8)

###  3.2 Statistics-aware Loss

Given the unclear distinction between the distributions of audio and visual
features of fake data as illustrated in Fig. 1, we propose an additional loss
$\mathcal{L}^{s}$ to enhance the discriminative power of audio and visual
features.

In this work, we suggest to maximize the distance between first-order
statistics (mean) of $\mathbf{f}^{a}$ and $\mathbf{f}^{v}$ denoted as
$\mu^{a}$ and $\mu^{v}$, respectively, in the presence of fake data, while
simultaneously minimizing it when dealing with real data. This is achieved
using a contrastive loss defined as,

$\mathcal{L}^{s}=\begin{cases}(d^{\mu})^{2}&\text{if }y\text{ is
real}\text{,}\\\ \max(m^{\mu}-(d^{\mu})^{2},0)&\text{if }y\text{ is
fake}\text{,}\\\ \end{cases}$ (9)

where $d^{\mu}$ is the $L_{2}$ distance between $\mu^{a}$ and $\mu^{v}$. In
this way, $d^{\mu}$ should be low when encountering a real video and high
otherwise.

Regarding the margin $m^{\mu}$, while it could be set as a constant
hyperparameter, we propose an alternative adaptive approach. Specifically, we
suggest using the standard deviation of $\mathbf{f}^{a}$ and $\mathbf{f}^{v}$
as the margin,

$m^{\mu}=\sigma^{v}+\sigma^{a}\text{,}$ (10)

where $\sigma^{a}$ and $\sigma^{v}$ represent the standard deviations of
$\mathbf{f}^{a}$ and $\mathbf{f}^{v}$, respectively. Leveraging the standard
deviation information instead of using a predefined hyperparameter serves two
purposes. Firstly, it helps reduce the number of predefined hyperparameters.
Secondly, it allows for a dynamic determination of how far apart two mean
values need to be separated based on the standard deviation. In other words,
when separating two distributions, those with a smaller standard deviation may
not require their means to be separated as far as distributions with a larger
standard deviation.

##  4 Experiments

###  4.1 Dataset

DFDC [11]. We employ the DeepFake Detection Challenge (DFDC) dataset for both
training and testing purposes, following the protocol described in [2, 5],
which employs a subset of $18,000$ videos. This subset is divided into train
and test sets with an $85:15$ ratio. Additionally, for accelerated initial
experiments, we also create a smaller set of training data consisting of
$4,930$ videos. Unless specified otherwise, we utilize the larger set.

FakeAVCeleb [12]. We exclusively use FakeAVCeleb for testing our DFDC-trained
models to assess their generalization capabilities to a new dataset. We adhere
to the test setup outlined in [23], where a test set comprises $70$ fake
videos and $70$ real videos.

###  4.2 Experiment Setup

We generally follow the setup proposed in the official code of
[2]222https://github.com/abhinavdhall/deepfake/tree/main/ACM_MM_2020, except
converting audio to a Mel spectogram. Approximately $1$ seconds of audio input
($\mathbf{I}^{a}$) is represented as a waveform with a size of $T^{a}\times
C^{a}=48000\times 1$, where $T^{a}$ and $C^{a}$ denote the time and channel
size, respectively. The video input sequence ($\mathbf{I}^{v}$) is of size
$T^{v}\times C^{v}\times H^{v}\times W^{v}=30\times 3\times 224\times 224$,
where $T^{v}$, $C^{v}$, $H^{v}$, and $W^{v}$ are the time, channel, height,
and width sizes. This video input is extracted concurrently with
$\mathbf{I}^{a}$ but at a different sampling rate ($30$ instead of $48000$).
The model is trained for $50$ epochs with the Adam optimizer [24], a learning
rate of $10^{-3}$, a batch size of $8$, and a weight decay of $10^{-5}$. The
model with the lowest total loss throughout the training epoch is saved for
evaluation. Unless specified otherwise, we set the weighting hyperparameter
$\alpha$ in Eq. (3) to $1$. We report the widely-used Area Under the Curve
(AUC) metric to measure the performance of the proposed model [2, 5].

###  4.3 Ablation Study

To enhance the model proposed in [2], we introduce three modifications: 1)
post-processing by normalizing the fakeness score based on the score
statistics in the training set (Section 3.1.3); 2) changing audio input type
from frequency-based to waveform; and (3) reducing the network size. The
significance of normalization with the training set in post-processing is
evident when comparing Table 1(a) and Table 1(b). Transitioning from Table
1(b) to Table 1(c), we change the audio input type from Mel spectrogram to
waveform, resulting in an improved performance. Reducing the network size in
Table 1(d) decreases the number of parameters by more than half while
maintaining comparable performance, showcasing that a shallower network is
sufficient for deepfake detection. The model in Table 1(d) is defined as our
baseline from here forth. Incorporating the statistics-aware loss in Table
1(e) further enhances performance without introducing additional parameters.
These results underscore the importance of each proposed component.

Table 1: Ablation study: (a) original model of [2], (b) with normalization in
post-processing, (c) with audio format change to waveform, (d) with shallower
network (baseline), and (e) with added distribution loss. “Size”, “Audio”,
“Norm.”, “$\mathcal{L}^{s}$”, “AUC”, “#Param” represent network size, audio
input type (Mel: Mel-spectogram; Wave.: Waveform), normalization in post-
processing, statistics-aware loss, AUC on DFDC, and number of model parameters
in millions, respectively.

| Size | Audio | Norm. | $\mathcal{L}^{s}$ | AUC | #Param
---|---|---|---|---|---|---
(a) | Deep | Mel |  |  | 89.87% | 122.78
(b) | Deep | Mel | ✓ |  | 96.34% | 122.78
(c) | Deep | Wave. | ✓ |  | 97.13% | 263.67
(d) | Shallow | Wave. | ✓ |  | 96.55% | 107.10
(e) | Shallow | Wave. | ✓ | ✓ | 96.69% | 107.10

###  4.4 Comparisons with SoA

Table 2 compares our approach with SoA methods in terms of AUC on DFDC. When
compared to uni-modal methods, either only-visual (V) or only-audio (A),
multi-modal methods generally exhibit better performance. In the comparison
among audio-visual models, our method demonstrates superiority over other
models, including the MDS [2] baseline. We also outperform methods using
emotion-based inconsistencies [5]. Furthermore, our performance surpasses
identity-based models, such as BA-TFD [18] and VFD [17]. Additionally, we
outperform fusion-based models, such as AVoiD-DF [6] and AVFakeNet [20].

Table 2: Comparison of the proposed model with SoA methods in terms of AUC on
the DFDC dataset using only-visual (V), only-audio (A), and audio-visual (AV)
modalities.

Method | Modality | AUC | Method | Modality | AUC
---|---|---|---|---|---
Multi-Attention [25] | V | 84.8% | MDS [2] | AV | 90.66%
SLADD [13] | V | 75.2% | Emotion [5] | AV | 84.4%
Meso4 [10] | V | 75.3% | BA-TFD [18] | AV | 84.6%
AASIST [14] | A | 68.4% | AVFakeNet [20] | AV | 86.2%
ECAPA-TDNN [15] | A | 69.8% | VFD [17] | AV | 85.13%
RawNet [8] | A | 56.2% | AvoiD-DF [6] | AV | 94.8%
|  |  | SADD (Ours) | AV | 96.69%

###  4.5 Additional Discussions

####  4.5.1 How does $\alpha$ affect the model?

We introduce a weighting hyperparameter $\alpha$ in Eq. (3) to adjust the
contribution of the statistics-aware loss within the overall loss. We report
the obtained results on a smaller training set, when varying the value of
$\alpha$ . As shown in Table 3, for different values of $\alpha$ , the model
achieves better performance compared to the baseline. However, setting
$\alpha$ to relatively large value results in a significant drop in AUC.

To further analyze the behavior of the model with different $\alpha$ values,
we visualize in Fig. 3 the distribution of audio and visual features from a
real and a fake test sample. Interestingly, for most $\alpha$ values, instead
of having separate mean values between audio and visual features for fake
samples, the distribution exhibits different standard deviations. This can be
attributed to the margin $m^{\mu}$ in Eq. (9), which is defined using the
standard deviation (Eq. (10)). Consequently, the loss in Eq. (9) can decrease
if the total standard deviation becomes smaller. Hence, a smaller standard
deviation of the audio distribution can lead to a smaller $L^{s}$.
Nevertheless, the observed distinct characteristics between real and fake data
may contribute to the overall improvement of the model. Instead of observing a
difference in terms of standard deviation, a sseperation strats to appear
between the mean values of audio and visual features, with larger $\alpha$
values ($\alpha\geq 10$). However, a high value of $\alpha$ (i.e.,
$\alpha=100$) can also impact the distribution of real data, making the real
and fake data distributions indistinguishable again.

Table 3: Evaluation on DFDC across a wide range of $\alpha$ values. Each model is trained on the smaller training set. In general, our model is robust to a huge range of $\alpha$ values, achieving better performance compared to the baseline. $\alpha$ | AUC | $\alpha$ | AUC
---|---|---|---
0 (baseline) | 86.52% | 1 | 88.55%
0.01 | 87.33% | 5 | 86.64%
0.05 | 88.14% | 10 | 86.92%
0.1 | 86.57% | 50 | 86.59%
0.5 | 87.03% | 100 | 83.51%
Fig. 3:  Feature distribution histograms of real and fake data with different
values of $\alpha$. Adding the statistics-aware loss ($\alpha>0$) results in
distinguishable distribution characteristics between real and fake data.

####  4.5.2 Impact of the statistics-aware loss in limited training data

We train the models detailed in Table 3 using the smaller training dataset.
The model incorporating the statistics-aware loss achieves an improvement of
$2\%$ as compared to the baseline, highlighting the relevance of this loss in
a limited training data environment.

####  4.5.3 Cross-dataset generalization

To assess the generalisation capabilities of the proposed method, we test our
model (trained on the dataset DFDC) on the FakeAVCeleb dataset. The models
reported in Table 1(a)-(e) achieve AUC scores of $48.57\%$, $56.92\%$,
$59.87\%$, $58.82\%$, and $61.39\%$, respectively. The noticable enhancement
in performance suggests improved generalization capabilities resulting from
the utilization of waveform and the incorporation of statistics-aware loss.
However, the proposed model still lags behind SoA, such as AvoiD-DF [6] with
$82.8\%$ AUC, prompting the need for further investigations and refinements.
One potential avenue for improvement could be exploring the benefits of a
transformer-based model, as utilized in AvoiD-DF.

####  4.5.4 Is the behavior induced by the statistics-aware loss consistent?

To further emphasize the importance of the statistics-aware loss, we visualize
in addition to Fig. 1, the feature distributions extracted from models with
different settings. Fig. 4(a) and (b) correspond to the models described in
Table 1(b) and (c), respectively. Fig. 4(c) represents our baseline trained on
a smaller set. As observed, a similar trend occurs in different models,
indicating that the proposed statistics-aware loss has the potential to
enhance their performance.

Fig. 4: Phenomena observed in Fig. 1 are also evident in other models without
statistics-aware loss: (a) Deep network with mel-spectrogram audio input
(Table 1(b)); (b) Deep network with waveform audio input (Table 1(c)); (c)
Shallow network with waveform audio input (Table 1(d)) trained with the
smaller set.

####  4.5.5 Consideration of KL divergence

One well-known metric for measuring the distance between two distributions is
Kullback–Leibler (KL) divergence. However, utilizing KL divergence directly in
our method requires backpropagation through feature histogram generation,
which is not straightforward. For experimental purposes involving KL
divergence, we treat each feature vector as a probability distribution by
applying log_softmax $l(\cdot)$ to $\mathbf{f}^{a}$ and $\mathbf{f}^{v}$.
Consequently, the KL distance loss is defined as,

$\mathcal{L}^{s}=\exp
l(f^{a})\log\frac{l(\mathbf{f}^{a})}{l(\mathbf{f}^{v})}\text{,}$ (11)

For a fair comparison, similar to the proposed statistics-aware loss, we
determine the optimal $\alpha$ hyperparameter value using a small training set
and find $\alpha=0.05$. We then retrain the entire model with the full
training set. However, the model trained using the KL divergence alone
achieved an AUC of $95.91\%$ failing to surpass the baseline with an $96.55\%$
AUC. Alternative approaches, such as reinforcement learning to approximate the
gradients, could be considered in future work.

##  5 Conclusion

In this paper, an enhanced version of MDS [2] has been introduced. MDS which
aims at assessing whether the audio is synchronized with the visual content
has registered promising results. Nevertheless, such an approach is subject to
several drawbacks. Namely, it depends on numerous empirically fixed
hyperparameters, does not consider the overall feature statistics and is based
on a cumbersome architecture. To address that, a statistics-aware loss that
enforces the separation of feature distributions for fake data while bringing
them closer together for real data has been proposed. In addition, a shallower
architecture relying on waveform as input has been introduced. Finally, an
inference post-processing strategy has been followed. Evaluations on the DFDC
and the FakeAVCeleb datasets, confirms the significance of the statistics-
aware. In future work, more suitable and complete distribution losses will be
considered.

## References

  * [1] R. K. Kaliyar, A. Goswami, P. Narang, and V. Chamola, “Understanding the use and abuse of social media: Generalized fake news detection with a multichannel deep neural network,” IEEE Transactions on Computational Social Systems, 2022.
  * [2] K. Chugh, P. Gupta, A. Dhall, and R. Subramanian, “Not made for each other-audio-visual dissonance-based deepfake detection and localization,” in ACM Multimedia, 2020.
  * [3] Y. Gu, X. Zhao, C. Gong, and X. Yi, “Deepfake video detection using audio-visual consistency,” in Digital Forensics and Watermarking: 19th International Workshop, IWDW 2020, Springer, 2021.
  * [4] C. Feng, Z. Chen, and A. Owens, “Self-supervised video forensics by audio-visual anomaly detection,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
  * [5] T. Mittal, U. Bhattacharya, R. Chandra, A. Bera, and D. Manocha, “Emotions don’t lie: An audio-visual deepfake detection method using affective cues,” in ACM Multimedia, 2020.
  * [6] W. Yang, X. Zhou, Z. Chen, B. Guo, Z. Ba, Z. Xia, X. Cao, and K. Ren, “Avoid-df: Audio-visual joint learning for detecting deepfake,” IEEE Transactions on Information Forensics and Security, vol. 18, 2023.
  * [7] Y. Zhou and S.-N. Lim, “Joint audio-visual deepfake detection,” in CVPR, 2021.
  * [8] J.-w. Jung, H.-S. Heo, J.-h. Kim, H.-j. Shim, and H.-J. Yu, “Rawnet: Advanced end-to-end deep neural network using raw waveforms for text-independent speaker verification,” arXiv preprint arXiv:1904.08104, 2019.
  * [9] N. Mejri, K. Papadopoulos, and D. Aouada, “Leveraging high-frequency components for deepfake detection,” in 2021 IEEE 23rd International Workshop on Multimedia Signal Processing (MMSP), IEEE, 2021.
  * [10] D. Afchar, V. Nozick, J. Yamagishi, and I. Echizen, “Mesonet: a compact facial video forgery detection network,” in WIFS, IEEE, 2018.
  * [11] B. Dolhansky, J. Bitton, B. Pflaum, J. Lu, R. Howes, M. Wang, and C. C. Ferrer, “The deepfake detection challenge (dfdc) dataset,” arXiv preprint arXiv:2006.07397, 2020.
  * [12] H. Khalid, S. Tariq, M. Kim, and S. S. Woo, “Fakeavceleb: A novel audio-video multimodal deepfake dataset,” in Proc. Conf. Neural Inf. Process. Syst. Datasets Benchmarks Track, 2021.
  * [13] L. Chen, Y. Zhang, Y. Song, L. Liu, and J. Wang, “Self-supervised learning of adversarial example: Towards good generalizations for deepfake detection,” in CVPR, 2022.
  * [14] J.-w. Jung, H.-S. Heo, H. Tak, H.-j. Shim, J. S. Chung, B.-J. Lee, H.-J. Yu, and N. Evans, “Aasist: Audio anti-spoofing using integrated spectro-temporal graph attention networks,” in ICASSP, IEEE, 2022.
  * [15] B. Desplanques, J. Thienpondt, and K. Demuynck, “Ecapa-tdnn: Emphasized channel attention, propagation and aggregation in tdnn based speaker verification,” arXiv preprint arXiv:2005.07143, 2020.
  * [16] S. Agarwal, L. Hu, E. Ng, T. Darrell, H. Li, and A. Rohrbach, “Watch those words: Video falsification detection using word-conditioned facial motion,” in WACV, 2023.
  * [17] H. Cheng, Y. Guo, T. Wang, Q. Li, X. Chang, and L. Nie, “Voice-face homogeneity tells deepfake,” ACM Transactions on Multimedia Computing, Communications and Applications, 2023.
  * [18] Z. Cai, K. Stefanov, A. Dhall, and M. Hayat, “Do you really mean that? content driven audio-visual deepfake dataset and multimodal method for temporal forgery localization,” in DICTA, IEEE, 2022.
  * [19] J. K. Lewis, I. E. Toubal, H. Chen, V. Sandesera, M. Lomnitz, Z. Hampel-Arias, C. Prasad, and K. Palaniappan, “Deepfake video detection based on spatial, spectral, and temporal inconsistencies using multimodal deep learning,” in AIPR, IEEE, 2020.
  * [20] H. Ilyas, A. Javed, and K. M. Malik, “Avfakenet: A unified end-to-end dense swin transformer deep learning model for audio–visual deepfakes detection,” Applied Soft Computing, 2023.
  * [21] S. Chopra, R. Hadsell, and Y. LeCun, “Learning a similarity metric discriminatively, with application to face verification,” in CVPR, vol. 1, IEEE, 2005.
  * [22] J. S. Chung and A. Zisserman, “Out of time: automated lip sync in the wild,” in ACCV 2016 Workshops, Springer, 2017.
  * [23] H. Khalid, M. Kim, S. Tariq, and S. S. Woo, “Evaluation of an audio-video multimodal deepfake dataset using unimodal and multimodal detectors,” in Proceedings of the 1st workshop on synthetic multimedia-audiovisual deepfake generation and detection, 2021.
  * [24] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014.
  * [25] H. Zhao, W. Zhou, D. Chen, T. Wei, W. Zhang, and N. Yu, “Multi-attentional deepfake detection,” in CVPR, 2021.


# Magnetism in Kitaev Quantum Spin Liquid Candidate RuBr3

T. Weinhold  Institute of Solid State and Materials Physics, TU Dresden,
D-01062 Dresden, Germany  C. Wang  Laboratory for Muon Spin Spectroscopy, PSI,
Villigen, Switzerland  F. Seewald  Institute of Solid State and Materials
Physics, TU Dresden, D-01062 Dresden, Germany  V. Grinenko  Tsung-Dao Lee
Institute, Shanghai Jiao Tong University, Shanghai, China  Y. Imai  Department
of Physics, Graduate School of Science, Tohoku University, 6-3, Aramaki Aza-
Aoba, Aoba-ku, Sendai, Miyagi 980-8578, Japan  F. Sato  K. Ohgushi  Department
of Physics, Graduate School of Science, Tohoku University, 6-3, Aramaki Aza-
Aoba, Aoba-ku, Sendai, Miyagi 980-8578, Japan  H.-H. Klauss  R. Sarkar
<EMAIL_ADDRESS>Institute of Solid State and Materials Physics, TU
Dresden, D-01062 Dresden, Germany

###### Abstract

The present studies show that long-range magnetic order takes place in RuBr3
at $\approx$ 34 K. The observations of clear oscillations in the muon time
spectra demonstrate the presence of well-defined internal fields at the muon
sites. The magnetic ordering appears to be very robust and static suggesting a
more conventional nature of magnetic ordering in the RuBr3 system at zero
field. Present investigations prove that in RuBr3 the Kitaev interactions are
likely to be weakened at zero field in comparison to the $\alpha$-RuCl3
system. This proves that it is possible to tune the Kitaev interactions by
replacing Cl with heavier halogen elements such as Br.

Introduction. A quantum spin liquid (QSL) is an exotic state of matter, in
which electron spins are strongly entangled, but do not exhibit any long-range
magnetic ordering down to $T$ = 0 K. Despite considerable effort in the past,
so far, experimental realizations of a clean QSL remain scarce. [1, 2, 3]
Alexei Kitaev proposed a theoretical model, known as the Kitaev model, where
spins (S = 1/2) are placed on a honeycomb lattice. They are coupled with their
three nearest neighbor spins with bond-dependent ferromagnetic Ising
interactions. [4] Kitaev model is exactly solvable, and the ground state is a
quantum spin liquid. Recently the ruthenium halides are considered promising
candidates for the realization of a Kitaev spin liquid. The bond-dependent
Kitaev magnetic exchange interactions drive this quantum spin liquid phase. To
realize this experimentally, $\alpha$-RuCl3 has been extensively studied.
However, the theoretical descriptions of $\alpha$-RuCl3 are still debated, and
a correct microscopic description of the low-temperature phase is not
available. [5, 6, 7, 8, 9, 10, 11, 12, 13] It appears that a lack of analog
materials for $\alpha$-RuCl3 puts forward certain limitations to find out the
electronic properties as functions of interaction parameters. In
$\alpha$-RuCl3 the covalency between Ru-4d and Cl-p-orbitals is important to
induce Kitaev interaction. Thus, replacing Cl with heavier halogen elements,
such as Br or I, is a sensible approach to strengthen the Kitaev interaction
further. Indeed, a recent comparative study of the electronic structures of
$\alpha$-RuCl3, RuBr3, and RuI3 and the estimation of magnetic exchange
interactions of all three compounds suggests that RuBr3 is a promising
candidate to realize Kitaev spin liquid phases. [14, 15, 16, 17]
The material RuBr3 has a BiI3-type structure (space group R$\bar{3}$) where
Ru3+ forms an ideal honeycomb lattice even at room temperature and it does not
show a structural transition down to low temperatures. RuBr3 orders
antiferromagnetically at $\approx$ 34 K.

To further investigate this system, in particular, its nature of the static
and/or dynamic ground state, we have performed detailed Muon Spin Resonance
($\mu$SR) experiments, both in zero-field (ZF) and in the longitudinal field
(LF) along the initial muon polarisation, in the temperature range 1.5-200 K.
Present $\mu$SR studies confirm the presence of long-range magnetic ordering
at $\approx$ 34 K. Muon relaxation rates ($\lambda$) probe the dynamic/static
spin susceptibility at $\mu$eV energy scales. This gives important information
which is complementary to nuclear magnetic resonance and inelastic neutron
scattering. Present data are typical representations of static magnetism. Thus
in zero-field, the magnetic ordering is robust and static suggesting a more
conventional type of quasi three-dimensional magnetic ordering in RuBr3 at
least in the $\mu$SR time scale. However, the field dependence of the
interactions is not yet known.

Figure 1:  Crystal structure of the RuBr3 system.

Figure 2:  Left panel: At representative temperatures true ZF-$\mu$SR time
spectra measured at PSI . Right panel: Fourier transform $\mu$SR time spectra
at zero fields at a selected temperature. Fourier-transformed spectra are
vertically displaced for clear demonstrations. Lines indicate the theoretical
description as detailed in the text.

Experimental. Polycrystalline sample of RuBr3 was prepared by a high-pressure
synthesis according to Ref [14]. $\mu$SR experiments were performed at the
PSI, SWISS using the GPS instrument. For muon measurements, 300 mg of a powder
sample was used. The $\mu$SR data were analyzed with the muSR fit program.
[sutermuon] The crystal structure of the RuBr3 system is drawn using VESTA.
[18]

$\mu$SR results: the presence of long-range magnetic ordering. Representative
ZF-$\mu$SR asymmetry spectra, measured in wide temperature ranges, are shown
in the left panel of Fig. 2. The right panel shows the Fourier transformed
$\mu$SR time spectra at zero fields at the selected temperature. For visual
clarity, both time spectra and the Fourier-transformed spectra are shifted
vertically. In general, implanted muons (here positive muons) are highly
sensitive to the local magnetic fields with a resolution about
($B=\frac{2\pi}{\gamma_{\mu}}\nu_{\mu}$) $\approx$ 0.1 mT produced by the
adjacent Ru3+ spins. [19] This makes $\mu$SR an ideal probe to detect the
presence of any tiny static magnetism. It is clear that the present ZF-$\mu$SR
spectra display the characteristic signals originating from static magnetism:
1.) Spontaneous coherent oscillations in the studied temperature range down to
1.5 K., 2.) Strong damping of the muon depolarization due to the random
distribution of the static field. These points demonstrate the presence of a
well-defined static magnetic field at the muon stopping site originating due
to the possible long-range magnetic ordered state of Ru3+ moments.
The ZF-time spectra can be best physically described by two magnetic and one
nonmagnetic component:

$\displaystyle
A(t)=A_{1}(\frac{2}{3}\cos(\omega_{H1}t+\phi)e^{-\lambda_{\mathrm{T1}}t}+\frac{1}{3}e^{-\lambda_{\mathrm{L1}}t})$
$\displaystyle+A_{2}(\frac{2}{3}\cos(\omega_{H2}t+\phi)e^{-\lambda_{\mathrm{T2}}t}+\frac{1}{3}e^{-\lambda_{\mathrm{L2}}t})$
$\displaystyle+A_{3}e^{-\lambda_{\mathrm{L2}}t}$ (1)

where $A_{i}$(i=1,2,3) represent the initial asymmetry. $\lambda_{Ti}(i=1,2)$,
$\lambda_{Li}(i=1,2)$ are the width of the static field distribution and muon
relaxation rate, respectively. $\omega_{H2}$ can be well described by the
following equation: $\omega_{H2}(T)$ = a $\omega_{H1}(T)$, where a is a
temperature independent factor. Thus both muon sites observe the same physical
behavior, unlike the observations in $\alpha$-RuCl3. [20]

Figure 3:  Temperature dependence of the internal fields by fitting the ZF
spectra. Dotted lines are ordered parameter fit as detailed in the main text.
Inset shows the amplitude ratios of the ordering fractions estimated from the
fitting of the time spectra.

The right panel of Fig. 2 shows the temperature dependence of Fourier-
transformed $\mu$SR time spectra in zero fields. In the ordered state field
spectra just below $\approx$ 34 K shows a single broad line. With lowering the
temperature lines are shifted toward the high field side and clearly show the
split field spectra suggesting two well-defined internal fields at the muon
sites indicated by the dashed lines.
The estimated internal fields from the fit of the time spectra as a function
of temperature are plotted in Fig. 3. This order parameter plot for site 1 and
site 2 shows that at $\approx$ 34 K both fields merge indicating one magnetic
ordering temperature of the system. However, lowering the temperature they
split, and further lowering the temperature they are saturated to different
fixed values. The overall behavior of the order parameter development appears
to be similar suggesting similar magnetism of the two sites. The obtained
parameters can be described by the following phenomenological equation:

$\displaystyle B(T)=B_{0}{[1-(\frac{T}{T_{N}})^{\alpha}]}^{\beta}$ (2)

The fits give the following parameter values of
$B_{0}$($\mu_{0}H_{0}$)=28.99(2), $\alpha$=2.81(4), $\beta$=0.35(2) and
$T_{N}=34.19(7)$ K. These fits identified the magnetic ordering temperature to
be $T_{N}=34.19(7)$ K which is consistent with the bulk characterization,
neutron scattering, and NMR experiments. Let’s briefly discuss the meaning of
all the parameters used in the phenomenological order parameter equation and
correspondingly relevant physics that one can extract from there. $B_{0}$ is a
constant that determines the saturated magnetic field. $\beta$ is called the
critical exponent and gives information about the type of magnetic ordering.
For three-dimensional Heisenberg ordering the expected value of $\beta$ is
0.367, whereas for three-dimensional Ising magnets the expected $\beta$ is
0.326. [21, 22] The current experimental finding, in particular $\beta$,
indicates that the magnetic ordering in RuBr3 is more like a quasi-three-
dimensional type.

For analog material $\alpha$-RuCl3 the four potential muon stopping sites have
been estimated. [20] Given that RuBr3 has a similar structure to
$\alpha$-RuCl3, it is likely the similar muon-stopping sites for RuBr3. Both
in $\alpha$-RuCl3 and RuBr3 two distinct well-resolved peaks are observed in
the field spectrum in the ordered state. While deep in the ordered state, the
development of the ordered parameter qualitatively behaves similarly for both
compounds, the significant differences are noticeable at the onset of the
ordering temperature. Further $\mu$SR studies on $\alpha$-RuCl3 single
crystals show multiple magnetic phases due to the presence of stacking faults.
[PhysRevB97134410] In the case of RuBr3 magnetic ordering sets in at the same
temperature, whereas in $\alpha$-RuCl3 the ordering sets in at two different
temperature points. One likely scenario is that in the former case, the
presence of less amount of stacking faults makes a unique ordering temperature
for the potential muon-stopping sites. The anisotropy in the interaction may
play a significant role there. However, to what extent this is related to the
anisotropy of the material is not clear so far. This would be clarified by
investigating single crystals which are currently not available to our
knowledge.

In the inset of Fig. 3, the amplitude ratios of the ordering fractions are
displayed as estimated from the fitting of the time spectra. The two signal
fractions are not equal, but rather significantly different. In the solid-
state Nuclear Magnetic Resonance (NMR) technique to evaluate field/frequency
sweep spectra, the area under the curve is taken as the measure of the number
of nuclei effectively contributed to the signal. Thus, the signal intensity
and/or area under the curve is proportional to the number of nuclei. This
helps to quantify the relevant signal fraction from different surroundings of
the nuclei. Conceptually, the muon field spectra are not that different.
Significantly intensity differences suggest among the four potential muon-
stopping sites one-third of the muons (site 2) has its different environments
than the rest (site 1). Although most of the muons encounter qualitatively
similar magnetic field distribution because of the more three-dimensional
magnetic ordering of the Ru3+ spins, their different locations in the lattice
with respect to the Ru3+ spins simply encounter different magnetic field
values.

Figure 4:  Top panel shows the representative LF-$\mu$SR time spectra
collected at 40 K. The lower panel shows the LF spectra at the selected field
at 1.55 K.

Having established that the system has the three-dimensional magnetic ordering
of the Ru3+ spins, we now want to understand the character of the fluctuations
of the Ru3+ spins, namely whether the fluctuations of the spins are static or
dynamic in nature. In general, for a QSL ground state system, the spins are
fluctuating, even if there is static ordering, and this is reflected in the
$\mu$SR relaxation processes by means of persistent spin dynamics or constant
values of relaxation in low temperature. LF-$\mu$SR experiments are an
excellent test case to prove or disprove the character of the spin fluctuation
(static/dynamic). Figure 4 shows the LF-$\mu$SR time spectra at two different
temperatures, namely 40 K and 1.55 K representing the normal state and
magnetically ordered state, respectively. In the normal state at 40 K by
applying a small field of only 50 Gauss, it is possible to completely decouple
the time spectra. The relaxation process at zero field appears to be the
effect of static nuclear relaxation which is easy to suppress/decouple by a
small amount of longitudinal field. So no electronic contribution either
static or dynamic has been encountered by the muons. However, deep in the
ordered state (lower panel) time spectra can be decoupled by the application
of the longitudinal fields reflecting the absence of dynamics.

Figure 5:  Temperature dependence of the ZF transverse relaxation rates in the
ordered state. The inset shows the ZF longitudinal relaxation rates.

Figure 5 shows the ZF transverse ($\lambda_{T}$) and longitudinal relaxation
($\lambda_{L}$) (inset) rates in the magnetically ordered state. Both
relaxation processes clearly represent a static ground state. In the ordered
state for the magnetic component, $\lambda_{L}$ shows the nominal temperature
dependency with values close to zero. Both relaxation processes clearly
represent a static ground state. [23]

Conclusions. In conclusion, detailed $\mu$SR studies on the powder of RuBr3
are carried out. There is clear evidence of long-range magnetic ordering below
$\approx$ 34 K. $\mu$SR relaxation rate $\lambda$ values below the similar
temperature range are typical for a static magnetically ordered system. This
is in line with the decoupling experiments. Taken all together, RuBr3 shows
three-dimensional long-range magnetically ordering at least in zero fields in
$\mu$SR time window. However, unlike the case of $\alpha$-RuCl3 in the $\mu$SR
experimental window, where full three-dimensional order sets in below a
certain temperature after the first ordering takes place, in RuBr3 three-
dimensional order sets in just at $\approx$ 34 K. The problem remains open to
what would be the anisotropic magnetic properties in this system in external
magnetic fields, for which one needs single crystals.

Acknowledgments. This work was financially supported by the Deutsche
Forschungsgemeinschaft (DFG) within the SFB 1143 “Correlated Magnetism – From
Frustration to Topology”, project-id 247310070 (Projects C02), GR 4667/1. This
work was supported by JST, CREST (JPMJCR1901), and JSPS KAKENHI Nos.
JP22H00102, JP19H05823, and JP19H05822. We thank Stephen Blundell for the
helpful discussions.

## References

  * Balents [2010] L. Balents, Spin liquids in frustrated magnets, Nature 464, 199 EP (2010).
  * Savary and Balents [2017] L. Savary and L. Balents, Quantum spin liquids: a review, Reports on Progress in Physics 80, 016502 (2017).
  * Wen  _et al._ [2019] J. Wen, S.-L. Yu, S. Li, W. Yu, and J.-X. Li, Experimental identification of quantum spin liquids, npj Quantum Materials 4, 12 (2019).
  * Kitaev [2006] A. Kitaev, Anyons in an exactly solved model and beyond, Annals of Physics 321, 2 (2006), january Special Issue.
  * Jackeli and Khaliullin [2009] G. Jackeli and G. Khaliullin, Mott insulators in the strong spin-orbit coupling limit: From heisenberg to a quantum compass and kitaev models, Phys. Rev. Lett. 102, 017205 (2009).
  * Winter  _et al._ [2017] S. M. Winter, A. A. Tsirlin, M. Daghofer, J. van den Brink, Y. Singh, P. Gegenwart, and R. Valentí, Models and materials for generalized kitaev magnetism, Journal of Physics: Condensed Matter 29, 493002 (2017).
  * Motome and Nasu [2020] Y. Motome and J. Nasu, Hunting majorana fermions in kitaev magnets, Journal of the Physical Society of Japan 89, 012002 (2020), https://doi.org/10.7566/JPSJ.89.012002 .
  * Johnson  _et al._ [2015] R. D. Johnson, S. C. Williams, A. A. Haghighirad, J. Singleton, V. Zapf, P. Manuel, I. I. Mazin, Y. Li, H. O. Jeschke, R. Valentí, and R. Coldea, Monoclinic crystal structure of $\alpha-{\mathrm{rucl}}_{3}$ and the zigzag antiferromagnetic ground state, Phys. Rev. B 92, 235119 (2015).
  * Cao  _et al._ [2016] H. B. Cao, A. Banerjee, J.-Q. Yan, C. A. Bridges, M. D. Lumsden, D. G. Mandrus, D. A. Tennant, B. C. Chakoumakos, and S. E. Nagler, Low-temperature crystal and magnetic structure of $\alpha-{\mathrm{rucl}}_{3}$, Phys. Rev. B 93, 134423 (2016).
  * Banerjee  _et al._ [2016] A. Banerjee, C. A. Bridges, J.-Q. Yan, A. A. Aczel, L. Li, M. B. Stone, G. E. Granroth, M. D. Lumsden, Y. Yiu, J. Knolle, S. Bhattacharjee, D. L. Kovrizhin, R. Moessner, D. A. Tennant, D. G. Mandrus, and S. E. Nagler, Proximate kitaev quantum spin liquid behaviour in a honeycomb magnet, Nature Materials 15, 733 (2016).
  * Maksimov and Chernyshev [2020] P. A. Maksimov and A. L. Chernyshev, Rethinking $\alpha\text{$-$}{\mathrm{rucl}}_{3}$, Phys. Rev. Res. 2, 033011 (2020).
  * Baek  _et al._ [2017] S.-H. Baek, S.-H. Do, K.-Y. Choi, Y. S. Kwon, A. U. B. Wolter, S. Nishimoto, J. van den Brink, and B. Büchner, Evidence for a field-induced quantum spin liquid in $\alpha$-${\mathrm{rucl}}_{3}$, Phys. Rev. Lett. 119, 037201 (2017).
  * Kasahara  _et al._ [2018] Y. Kasahara, T. Ohnishi, Y. Mizukami, O. Tanaka, S. Ma, K. Sugii, N. Kurita, H. Tanaka, J. Nasu, Y. Motome, T. Shibauchi, and Y. Matsuda, Majorana quantization and half-integer thermal quantum hall effect in a kitaev spin liquid, Nature 559, 227 (2018).
  * Imai  _et al._ [2022] Y. Imai, K. Nawa, Y. Shimizu, W. Yamada, H. Fujihara, T. Aoyama, R. Takahashi, D. Okuyama, T. Ohashi, M. Hagihala, S. Torii, D. Morikawa, M. Terauchi, T. Kawamata, M. Kato, H. Gotou, M. Itoh, T. J. Sato, and K. Ohgushi, Zigzag magnetic order in the kitaev spin-liquid candidate material ${\mathrm{rubr}}_{3}$ with a honeycomb lattice, Phys. Rev. B 105, L041112 (2022).
  * Kim&lowast; [2021] H.-S. Kim&lowast;, Spin-orbit-entangled nature of magnetic moments and kitaev magnetism in layered halides, Applied Science and Convergence Technology 30, 191 (2021).
  * Nawa  _et al._ [2021] K. Nawa, Y. Imai, Y. Yamaji, H. Fujihara, W. Yamada, R. Takahashi, T. Hiraoka, M. Hagihala, S. Torii, T. Aoyama, T. Ohashi, Y. Shimizu, H. Gotou, M. Itoh, K. Ohgushi, and T. J. Sato, Strongly electron-correlated semimetal rui3 with a layered honeycomb structure, Journal of the Physical Society of Japan 90, 123703 (2021), https://doi.org/10.7566/JPSJ.90.123703 .
  * Kaib  _et al._ [2022] D. A. S. Kaib, K. Riedl, A. Razpopov, Y. Li, S. Backes, I. I. Mazin, and R. Valentí, Electronic and magnetic properties of the RuX3 (x = cl, br, i) family: two siblings—and a cousin?, npj Quantum Materials 7, 10.1038/s41535-022-00481-3 (2022).
  * Momma and Izumi [2011] K. Momma and F. Izumi, VESTA3 for three-dimensional visualization of crystal, volumetric and morphology data, Journal of Applied Crystallography 44, 1272 (2011).
  * Sarkar  _et al._ [2017] R. Sarkar, J. Spehling, P. Materne, H. Luetkens, C. Baines, M. Brando, C. Krellner, and H.-H. Klauss, Magnetic order and spin dynamics across a ferromagnetic quantum critical point: $\mu\mathrm{SR}$ investigations of ${\mathrm{ybni}}_{4}{({\mathrm{P}}_{1-x}{\mathrm{As}}_{x})}_{2}$, Phys. Rev. B 95, 121111 (2017).
  * Lang  _et al._ [2016] F. Lang, P. J. Baker, A. A. Haghighirad, Y. Li, D. Prabhakaran, R. Valentí, and S. J. Blundell, Unconventional magnetism on a honeycomb lattice in $\alpha-{\mathrm{rucl}}_{3}$ studied by muon spin rotation, Phys. Rev. B 94, 020407 (2016).
  * [21] T. K. Feng, Ph.D. thesis Critical Dynamics in Classical Antiferromagnets .
  * Pelissetto and Vicari [2002] A. Pelissetto and E. Vicari, Physics Reports 368 (2002).
  * de Réotier and Yaouanc [1997] P. D. de Réotier and A. Yaouanc, Muon spin rotation and relaxation in magnetic materials, Journal of Physics: Condensed Matter 9, 9113 (1997).