# Unsupervised active speaker detection in media content using cross-modal
information

Rahul Sharma, and Shrikanth Narayanan Rahul Sharma<EMAIL_ADDRESS>and
Shrikanth Narayanan<EMAIL_ADDRESS>is with Electrical and Computer
Engineering, University of Southern California, USA

###### Abstract

We present a cross-modal unsupervised framework for active speaker detection
in media content such as TV shows and movies. Machine learning advances have
enabled impressive performance in identifying individuals from speech and
facial images. We leverage speaker identity information from speech and faces,
and formulate active speaker detection as a speech-face assignment task such
that the active speaker’s face and the underlying speech identify the same
person (character). We express the speech segments in terms of their
associated speaker identity distances, from all other speech segments, to
capture a relative identity structure for the video. Then we assign an active
speaker’s face to each speech segment from the concurrently appearing faces
such that the obtained set of active speaker faces displays a similar relative
identity structure. Furthermore, we propose a simple and effective approach to
address speech segments where speakers are present off-screen. We evaluate the
proposed system on three benchmark datasets– Visual Person Clustering dataset,
AVA-active speaker dataset, and Columbia dataset– consisting of videos from
entertainment and broadcast media, and show competitive performance to state-
of-the-art fully supervised methods.

###### Index Terms:

unsupervised, cross-modal, speaker identity, face recognition, active speaker
detection, entertainment media

##  I Introduction

Speech-related activity in a video manifests spatio-temporally both in audio
and visual modalities. Mapping speech activity in the temporal domain related
to _when_ , in time, someone is speaking is often approached from the audio
modality. On the other hand, in the spatial domain, determining _where_ in a
frame the speaker is present relies on the visual signal. In this work, we use
the speech activity information obtained from the audio modality to find the
corresponding speech activity in the visual modality. We address the problem
of finding the speaking face (if any) from the set of candidate faces
appearing in the video. We refer to this task as _active speaker detection_
(ASD).

ASD plays a crucial role in computational media intelligence (CMI) [1],
enabling the study of representation and portrayals of characters portrayed in
media and their societal impact. Specifically, ASD is a crucial component of
tools to discern _who is speaking when?_ in a video, such as for primary
character detection, background character detection [2], speaker diarization
[3, 4, 5], and other human-computer-interaction applications. This work
focuses on ASD in entertainment media content, notably Hollywood movies and TV
shows.

ASD in entertainment media content is a challenging task due to the unknown
and varying number of speakers and faces in the visual frames. The speaking
faces appear in diverse poses because of the rich dynamics in character
interactions. The face detector fails when the speakers appear in extreme
poses including looking away from the camera or moving out of the frame. Non-
ideal conditions in the audio modality, such as background (including off-
screen) speakers and noisy speech, also contribute to problem’s complexity.

The inherent multimodal nature of speech activity has motivated numerous
approaches for active speaker detection to use the connections between the
visual and audio modality. The idea of localizing the visual actions in the
video frames responsible for the audio activity, notably speech, dominates
these approaches. Audio-visual speech modeling typically involves capturing
the activity in the lip region of the detected faces in the visual frames,
explicitly [6, 3] or implicitly [7, 8], and studying its correlation with the
speech waveform to discern the active speaker. Most of these approaches share
the same drawbacks: i) These methods employ computationally expensive 3D
Convolutional neural networks and Recurrent neural networks to model the
visual activity. ii) They train the models in a full-supervised manner
requiring manual annotations, which are tedious and expensive to acquire. iii)
A severe limitation is that the visual activity in the facial gestures
concerning other activities, such as laughing, chewing, etc., can be easily
confused with the speech activity, thus adding false positives.

To complement the noisy visual activity information, we propose to use the
speaker identity information for active speaker face-voice assignment. Using
the knowledge that face and voice both possess information about a speaker’s
identity, we present an unsupervised framework that exploits the fact that
both the active speaker’s face in the visual modality and the concurrent voice
in the audio modality refer to the same speaker. We use well researched and
readily available face and speech recognition systems to represent the faces
and speech, respectively, so that both capture the speaker’s identity.

Using the obtained representations, we construct a speech-identity distance
matrix, capturing the relative identity structure of the speech segments
through the entire video. Figure 1a shows an example of such a distance
matrix. We assign an active speaker face to each speech segment such that the
obtained set of corresponding active speaker faces displays a similar relative
identity structure in form of a face-identity distance matrix, an example is
shown in Figure 1b. Figure 2 shows an overview of the proposed strategy.
Furthermore, studying the similarity between the two distance matrices (speech
and face), we present a simple strategy to address the speech segments with
speakers present off-screen.

We evaluate the system’s performance for active speaker detection on three
benchmark datasets: i) AVA active speaker [9] and ii) Visual Person Clustering
[10] datasets, consisting of videos of TV shows and movies, and iii) Columbia
[11] dataset, consisting of a panel discussion video. The implementation code
and the required dependencies to reproduce all the results in this work will
be publicly available. The contributions of this work are as follows:

  1. 1.

We present a novel framework to harness the speaker identity information,
present within the speech in the audio modality and the speaker’s face in the
visual modality, for the speech-face assignment task. This framework can
complement state-of-the-art visual-activity-based active speaker detection
systems as the acquired information does not depend solely on the visual
activity present in a frame.

  2. 2.

We formulate the active speaker detection task as a cross-modal unsupervised
optimization task deriving information from pretrained speaker recognition and
face recognition models. This eliminates the need for manual annotations and
does not require additional training of any expensive models.

  3. 3.

We present an extensive performance evaluation of the system, validating the
strategy’s applicability for various challenging domains, including videos
from TV shows, Hollywood movies, international movies, and panel discussions.
Furthermore, we present ablation studies to study the performance of the
proposed system and to highlight potential limitations.

Figure 1: a) Speech-identity distance matrix ($SD$) and b) Face-identity
Distance matrix ($FD$) for the movie Hidden Figure. The active speaker faces
are acquired form the ground truth. The speech segments are ordered to gather
the speech segments of each speaker together.

Figure 2: The overview of the proposed framework: We gather temporally
overlapping faces for each speaker-homogeneous speech segment. Using the
speaker recognition embeddings, we construct a speech distance matrix. From
the set of possible sequences of corresponding active speaker faces, we select
the sequence of faces such that the face-identity distance matrix, obtained
using the face-recognition features, displays a high resemblance with the
speech-identity distance matrix.

##  II Related Work

An intuitive solution for detecting an active speaker is to study the activity
in the lip region of faces in a video; this idea has led to a wide range of
methods, from using just the visual signal [6] to posing it as multi-modal [7,
6, 12, 13, 14] and cross-modal [15, 11, 16, 17] formulations. One of the
earlier works proposed explicitly detecting the lip region and quantifying the
observed activity to classify faces as talking or non-talking in TV shows [6].
Along similar lines, sync-net [13] has proposed jointly modeling the activity
in the explicitly detected lip region and the concurrent audio stream using a
ConvNet architecture and training it to detect a speech-face synchronization
leading to a self-supervised active speaker detection system. Following the
success of modeling audio-visual synchronization, various self-supervised
methods have emerged targeting the task of sound source localization [7, 12,
8, 18]. These methods were qualitatively shown to detect an active speaker’s
face as the location for the speech sound event.

The nature of speech activity, interleaving the audio and visual modalities,
has led to various cross-modal approaches, predicting primarily audio-centric
tasks such as voice activity detection [11, 16, 15, 17] using the visual
input. Earlier works [15, 11] have modeled the facial activity using HoG
features and trained SVM-based classifiers using weak supervision from audio
VAD labels. More recently, the authors in [16, 17] introduced a more
sophisticated 3D ConvNets architecture to model the visual frames and trained
a weakly-supervised system for active speaker detection for the more
challenging Hollywood movie video domain. Lately, the introduction of large-
scale audio-visual datasets, such as AVA-active speaker [9] and Active Speaker
in Wild [19] consisting of manual active speaker annotations, has led to the
emergence of a series of fully-supervised methods. These methods rely on the
large-scale nature of labeled data to model the audio-visual activity by
training train a wide variety of massive ConvNet architectures, ranging from
3D convolutions [14, 20] to graph convolutional networks [21, 22].

Unlike the extensively explored idea of modeling the visual activity, efforts
in the direction of utilizing the speaker identity information for active
speaker detection is relatively limited. The work of Hoover et al. [23]
proposed to cluster the speech and faces separately to contain instances of a
single speaker and then derive the information from the temporal co-occurrence
of speech and face clusters to match speech and face for the same speaker.
However, for the audio-visual diarization problem [24], the speaker’s identity
information from speech and faces is widely explored [25, 3, 4]. For instance,
using the temporal co-occurrence of the speech and the face identities
(obtained using face clusters) to derive the speech identities [25]. Inspired
by the same, in this work, we propose to use the well-established speaker
recognition [26, 27] and face recognition [28, 29] systems to capture the
speaker’s identity information from speech and faces and rely on speech and
corresponding active speaker’s face identifying the same character to derive
speech-face associations.

##  III Methodology

###  III-A Problem Formulation

The aim is to generate a speech-face association such that the associated
face, if there is one, for every speech activity in the audio modality, is the
source of the underlying speech. We formulate the problem definition under the
following assumptions: i) that the video content under consideration is shot
to capture the speaker’s face, which stands valid, especially in the case of
entertainment media videos. ii) that at any point in time, there can be at
most one person speaking; thus, no overlapping speech. To investigate the
validity of assuming no overlapping speech, we tabulate the fraction of
overlapping-speech occurrences for several videos from VPCD and AVA active
speaker datasets, in Table I. We observe that the fraction is insignificant
for all the videos, thus validating the assumption.

TABLE I: Fraction of overlapping speech (%) for various videos

Videos |  | Fraction of
---
overlapping speech (%)
Friends | 1.39
TBBT | 4.42
Sherlock | 0.79
Hidden Figures | 0.44
About Last Night | 0.21
AVA active speaker | 2.31

We use the contiguous segments of speech activity in the audio modality of one
speaker as the fundamental unit for our analysis. By definition, such a voiced
segment consists of voices from only one speaker, called a speaker-homogeneous
voiced segment. We denote the set of all the speaker-homogeneous segments
through a video as $S_{all}\equiv\\{s_{1}\dots s_{n},\dots s_{N}\\}$.
Similarly, we use a face-track (locations of one face appearing in consecutive
frames) as the smallest unit for the visual modality. For each $s_{n}$, we
gather the temporally overlapping face-tracks available in visual modality and
denote them as $F_{n}\equiv\\{f_{1},\dots f_{k},\dots f_{K}\\}$.

We call $f_{k}$ as the active speaker face for $s_{n}$, denoted as
$(s_{n}\longleftrightarrow f_{k})$, if face-track $f_{k}$ in video modality is
the source of speech segment $s_{n}$ in audio modality. However, a speech
segment, $s_{n}$, may have no face tracks available in the visual modality in
its course, $F_{n}=\phi$; therein, the speaker is off-screen. Such segments
are not relevant to this work, and hence purged from $S_{all}$. We only
consider the speech segments which have one or more face-tracks overlapping in
time, be the active-speaker’s face is on-screen ($S_{\text{on}}$) or off-
screen ($S_{\text{off}}$), denoted in equation (1) and (2)

$\displaystyle S_{\text{on}}$ $\displaystyle\equiv\\{s_{n}\mid\exists f_{k}\in
F_{n}:(s_{n}\longleftrightarrow f_{k})\\}$ (1) $\displaystyle S_{\text{off}}$
$\displaystyle\equiv\\{s_{n}\mid F_{n}\neq\phi\text{ and }\nexists f_{k}\in
F_{n}:(s_{n}\longleftrightarrow f_{k})\\}$ (2)

We formulate the task of active speaker detection (ASD) as the problem of
finding an active speaker face-track for all the speaker-homogeneous speech
segments, if there exists one. We denote the set of active speaker faces, $A$,
as:

$\displaystyle A\equiv\\{a_{n}\mid a_{n}=f_{k}\text{ and }f_{k}\in F_{n}\text{
and }(s_{n}\longleftrightarrow f_{k})\\}$ (3)

Such a formulation using the speaker-homogeneous speech segments helps reduce
the ASD task to choosing one face track out of temporally overlapping tracks
of (possibly multiple) faces for each speech segment. It also satisfies the
constraint that active speaker faces can only be present when there is speech
activity in the audio modality. At the same time, using a face track as one
entity, rather than each face, helps facilitate that there can be at most one
active speaker face in a frame which aligns with our assumption of non-
overlapping speakers. This formulation intends to find an active speaker face
for each speech segment, $s_{n}$, but as mentioned earlier, the speech
segments may have off-screen speakers ($s_{n}\in S_{\text{off}}$), which may
introduce false positives.

Since the information separating the speech segments into the sets
$S_{\text{on}}$ and $S_{\text{off}}$ is not readily available we propose a
two-stage process. We first devise a system that provides an associated active
speaker face-track for all speech segments in $S_{a}\equiv S_{\text{on}}\cup
S_{\text{off}}$ and call it _stage1_. This is then followed by a system to
remove the speech segments with off-screen speakers from the set
$S_{\text{all}}$, to correct the introduced errors and call it _stage2_.

###  III-B Stage1: Speech-face Assignment

Speech in the audio and face in the visual modality is widely known to possess
an individual character’s identity information. Utilizing it, we propose to
select an active speaker face track for each speech segment $s_{n}$ from the
set of temporally overlapping face tracks, $a_{n}\in F_{n}$, imposing that the
selected face and the underlying speech depict the same character identity.
Since the identity captured in the audio modality (using speech) can not be
directly compared with the captured identity in the visual modality (using
faces), we compute an identity distance matrix for the two modalities
independently, bringing them into a shared space of relative identities.

We start with initializing the set of active speaker face tracks $A$ by
randomly selecting a face track for each speech segment from the set of face
tracks temporally overlapping with the speech segment, $a_{n}=f_{k}\mid
f_{k}\in F_{n}\forall s_{n}\in S_{all}$. Then we construct representations for
the speech segments and the active speaker faces to capture the character
identity information. For speech segments, we extract the speaker recognition
embeddings using a ResNet34 [30] pretrained on VoxCeleb2 [27] with angular
learning objective [26]. We represent the faces $f_{k}$ using SENet-50 [29],
pretrained on MS-Celeb-1M [28], widely used in face recognition systems.

Using the obtained representations, we construct a distance matrix for each
modality. In the audio modality, we represent the speech segments, $s_{n}$ in
terms of its distances from all other speech segments. We compute the distance
between two speech segments $s_{i}$ and $s_{j}$ as the cosine distance between
their speaker recognition embeddings. We call this matrix as the speech-
identity distance matrix, $SD$ and show it in equation 4. Similarly, in the
visual modality, we construct a face-identity distance matrix, $FD$
representing each active speaker face, $a_{n}$ in terms of its distance from
all other active speaker faces. We compute the distances between two active
speaker faces $a_{i}$ and $a_{j}$ as the cosine distance between their
obtained face recognition embeddings, as shown in equation 4. While computing
these distance matrices, we preserve the order of the speech segments and
corresponding active speaker faces, implying that the $i^{th}$ row of $SD$ and
$FD$ represents the speech segment, $s_{i}$, and the corresponding active
speaker face, $a_{i}$, respectively.

$\displaystyle SD[i,j]$ $\displaystyle=\frac{s_{i}\cdot
s_{j}}{\|s_{i}\|\|s_{j}\|}$ $\displaystyle FD[i,j]$
$\displaystyle=\frac{a_{i}\cdot a_{j}}{\|a_{i}\|\|a_{j}\|}$ (4)

The use of identity-capturing speaker recognition representations for speech
segment, $s_{n}$ enables the distance matrix $SD$ to capture the relative
identity structure with context from the entire video. For instance, speech
segments $s_{i}$ and $s_{j}$ from the same speaker will show a smaller value
in $SD[i,j]$ than those from different speakers. Similarly, the face-identity
distance matrix $FD$ constructed using the face recognition representations
captures the relative identity of the active speaker faces in the visual
modality. Figure 1 shows the $SD$ and $FD$ for the Hollywood movie _Hidden
Figures_ , where we have constructed the matrix $FD$ using the ground truth
active speaker faces. For the visualization purposes, we gather the speech
segments of each speaker together. Since the active speaker’s face and the
concurrent speech must identify the same person, we hypothesize that the
relative identity structure captured by $FD$ should have a high resemblance
with $SD$. We observe this similarity in Figure 1, in the form of similar low-
distance square formations along the diagonals of both the matrices.

The $i^{th}$ row in both the matrices depicted as $SD[i]$ and $FD[i]$
represents the identity of the underlying person (character) in terms of its
distance from all other characters in the video. Since the components of the
two rows come from distances computed in two independent dimensions (the audio
and visual modality), they may differ in scale and thus may not be directly
comparable. However, as the components of each row vector show the distance
relative to other identities and all the identities are supposed to be the
same among the two matrices, we expect the row vectors from the two matrices
to show a linear relationship. So we propose to use Pearson’s correlation as
the measure to quantify the similarity between the two matrices. We formalize
the ASD task as an optimization problem where we select the active speaker
faces for each speech segment from the set of temporally overlapping face
tracks, such that the Pearson’s correlation between the speech-identity matrix
and face-identity matrix, computed as an average of the row-wise correlations,
is maximized. The formulation is denoted below in equation 5, where $SD[i]$
and $FD[i]$ denotes the vector corresponding to the $i^{th}$ rows of $SD$ and
$FD$ respectively and $N$ is the number of rows.

$\begin{split}Corr(FD)=\\\
\frac{1}{N}\sum_{i}&\frac{\sum(SD[i]-\overline{SD[i]})(FD[i]-\overline{FD[i]})}{\sqrt{\sum(SD[i]-\overline{SD[i]})^{2}\sum(FD[i]-\overline{FD[i]})^{2}}}\end{split}$
(5) $A\equiv\operatorname*{arg\,max}_{\\{a_{n}\mid a_{n}=f_{k}\text{ and
}f_{k}\in F_{n}\\}}Corr(FD)$ (6)

The notable point is that the face-identity distance matrix, $FD$ is
symmetric; the rows are not independent. The change in the active speaker’s
face for $i^{th}$ speech segment implies the change in the $i^{th}$ row of
$FD$, which reflects a change in all the other rows of $FD$ in the form of the
altered $i^{th}$ column. Changed $FD$ further reflects a change in the
objective function $Corr(FD)$. Thus, in the ideal case scenario, the active
speaker faces for all the speech segments should be jointly selected to
maximize the $Corr(FD)$. Without loss of generality, let us consider that
there are $N$ speech segments in a video and, on average, $k$ faces that
temporally overlap each speech segment. The complexity for computing the
objective function $Corr(FD)$ for a given set of active speaker faces, $A$
would be $O(N^{2})$, and there would exist $k^{N}$ such possible sets, $A$
(all possible combinations of potential active speaker faces). Thus maximizing
the $Corr(FD)$ jointly for all active speaker faces will cost $O(k^{N}N^{2})$.
For a typical TV Hollywood movie, $k\approx 3$ and $N\approx 900$ bring the
cost of jointly optimizing high and out of the scope of current computing
standards.

We propose to approximate the optimization process by iteratively finding the
active speaker’s face corresponding to one speech segment at one time to
maximize the objective function $Corr(FD)$, keeping the active speaker faces
for all other speech segments fixed. Once iterating through all the speech
segments (an epoch), we repeat the process until the $Corr(FD)$ converges.
This approximation reduces the complexity of the optimization from an
exponential to a cubic order of $N$, $O(EkN^{3})$, where E is the number of
epochs, $kN$ is the number of times the $Corr(FD)$ is computed in each epoch
and $O(N^{2})$ is complexity for computing $Corr(FD)$. Furthermore, assuming
that even a portion of a movie has enough information to disambiguate between
the speech and speaker’s faces, we propose to split the number of speech
segments into $p$ partitions and optimize them separately. This helps further
reduce the complexity of the process to $O(Ek\frac{N^{3}}{p^{2}})$ where
$p>1$. The process is detailed in Algorithm 1.

$\text{Random initialization:}A\equiv\\{a_{n}\mid a_{n}\in F_{n}\\}\forall
s_{n}\in S_{\text{all}}$;

Compute $SD$ and $FD$ ;

// using eq: (4)

$objective\leftarrow Corr(FD)$ ;

// using eq: (5)

while  _$objective$ increases_ do

for  _each $a_{i}\in A$_ do

$a_{i}=\operatorname*{arg\,max}\limits_{f_{k}\in F_{n}}Corr(FD)$ ;

// $i^{th}$ row

$\text{Update }FD$;

$\text{Update }A$;

end for

$\text{$objective$}\leftarrow Corr(FD)$;

end while

Algorithm 1 Stage1: Speech-face assignation

###  III-C Stage2: Off-screen speaker correction

Maximizing the objective function, $Corr(FD)$, we assign a corresponding
active speaker face for all the speech segment under consideration, i.e.,
$s_{n}\in S_{\text{all}}$, which includes an active speaker face even for the
speech segments having off-screen speakers, $s_{n}\in S_{\text{Off}}$. This
incorrect assignment of speakers’ faces introduces false positives.

The underlying hypothesis to maximize the correlation between the speech-
identity and the face-identity distance matrices is that since the active
speaker’s faces and the speech segments must identify the same person, the
relative identity patterns from the speech-distance matrix ($SD$) and the
face-distance matrix ($FD$) should show a high resemblance. In contrast, in
the case of speech segments with speaker present off-screen, none of the face
tracks temporally overlapping with the speech segment corresponds to the
active speaker’s face. Thus the speech segment represents an identity
different from any of the concurrent face tracks. We hypothesize that this
discrepancy in the identity leads to a relatively lower similarity between the
identity representations obtained using the underlying speech segment and any
of the temporally overlapping faces for the speech segments with off-screen
speakers. Formally we classify the speech segment, $s_{i}$, as the speech
segment with an off-screen speaker if it displays a low enough row-wise
correlation between the identity representations obtained from the speech-
distance matrix, $SD[i]$ and any of the temporally overlapping face tracks,
$FD[i]$, denoted in Eq 7. Furthermore, we remove the face tracks, earlier
incorrectly classified as active speaker faces corresponding to such speech
segments with off-screen speakers.

$S_{\text{off}}\equiv\left\\{s_{i}\big{|}\frac{\sum(SD[i]-\overline{SD[i]})(FD[i]-\overline{FD[i]})}{\sqrt{\sum(SD[i]-\overline{SD[i]})^{2}\sum(FD[i]-\overline{FD[i]})^{2}}}<\tau\right\\}$
(7)

##  IV Performance Evaluation

###  IV-A Implementation details

We first obtain the active voice regions in a video using an off-the-shelf
speech segmentation tool called pyannote [31]. As the proposed system requires
speaker homogeneous speech segments, we use naive heuristics to segment
further the obtained voice active regions instead of employing a sophisticated
speaker change detection system. We partition the voiced regions at the shot
boundaries, which we gather using
PyScenedetect111http://scenedetect.com/en/latest/. The partitioning at the
shot boundary is motivated by speaker change being a prominent movie cut
attribute [32], thus decreasing the likelihood of observing a speaker change
in the obtained segments. We further partition the obtained segments to have a
maximum duration of 1sec, effectively decreasing the fraction of speech
segments consisting of a speaker change. We represent the speech segments
using a ResNet-34 [26] model pretrained on Voxceleb2 [27] dataset for the
speaker recognition task.

Within the gathered shot boundaries, we use the RetinaFace [33] to obtain face
detections and track them using the SORT tracker [34] to construct face
tracks. We extract identity representations for each face detection box using
a SENet-50 [29], pretrained on MS-Celeb-1M [28] for the face recognition task,
and average them over all the constituting faces to construct a representation
for a face track. Once the representations for all the speaker homogeneous
speech segments and all the faces are gathered, we initialize the active
speaker face for each speech segment with a face randomly selected from a set
of temporally overlapping faces. We partition the set of speech segments to
contain at most 500 speech segments (L=500), grouped in temporal order, which
helps reduce the optimization process’s time complexity. We then employ the
Algorithm 1 to assign an appropriate active speaker face to each of the speech
segments for each partition (stage1) and further improve the predictions by
discovering the speech segments with off-screen speakers (stage2).

To evaluate the performance of the proposed system, we use three benchmark
datasets: i) Visual person clustering dataset (VPCD) [10], ii) AVA-active
speaker dataset [9] and iii) Columbia dataset [11]. The AVA-active speaker
dataset consists of face-wise active speaker annotations for 161 international
movies, freely available on YouTube. We use the publicly available validation
split, consisting of 33 movies, to report and compare the performance of the
proposed strategy against the state-of-the-art systems. VPCD is more
exhaustive, consisting of character identity information in addition to the
active speaker annotations for videos from several widely watched episodes of
TV shows (Friends, Sherlock and TBBT) and 2 Hollywood feature films (Hidden
Figures and About Last Night). It accompanies bounding boxes for the appearing
faces and their corresponding character identities, along with the information
about which character is speaking at any time. One of the challenges of
working with VPCD is that the video files are not freely available. For
purposes of this research, we acquired the video files using the DVDs and
aligned the annotations in VPCD with the acquired videos. The Columbia dataset
consists of manually obtained active speaker annotations for an 85 minutes
long video of a panel discussion with six speakers. The panel discussion
scenario is relatively more controlled than the movies in terms of frontal
face poses and noise-free speech.

TABLE II: F1-scores at various stages for the videos from VPCD. Reported F1
scores are averaged over all the episodes for the TV shows (Friends, TBBT, and
Sherlock).

Videos |  | Random
---
initialization
| Stage 1
---
speech-face
| Stage 2
---
Off-screen speakers
| Stage 2
---
(mAP)
Friends (25 episodes) | 0.54 | 0.78 | 0.80 | 0.82
TBBT (6 episodes) | 0.60 | 0.83 | 0.84 | 0.86
Sherlock (3 episodes) | 0.55 | 0.64 | 0.66 | 0.70
Hidden Figures | 0.45 | 0.62 | 0.67 | 0.66
About Last Night | 0.51 | 0.59 | 0.62 | 0.57

###  IV-B Stage1: Speech-face assignation

Figure 3: Distance matrices for the movie Hidden Figures at various stages of
the system along with the value of objective function, $Corr(FD)$. a) Speech-
identity distance matrix ($SD$) b)Random: Face-identity distance matrix ($FD$)
for randomly initialized ASD. c) Stage1: $FD$ with speech-face assignation,
maximizing $Corr(FD)$. d) Stage2: $FD$ post removing the speech segments with
off-screen speaker. e) Ground truth: $FD$ obtained using ground truth active
speaker faces.

The hypothesis underlying the Algorithm 1 is that the relative identity
patterns captured by the face-identity distance matrix $FD$, constructed using
the active speaker faces, must show a high resemblance with the speech-
identity distance matrix $SD$. In this section, we present the qualitative and
quantitative validation of the hypothesis using videos from the VPCD. We
compare the predictions for each face-bounding box against the ground truth
and report the F1-score in Table II. In Figure 3 we show the evolution of the
$FD$ at various stages for the movie Hidden Figures.

Using ground truth speaker identities, we order the speech segments in $SD$,
and the corresponding predicted active speaker faces in $FD$ to bring the
speech segments from each speaker together. This makes the matrices
interpretable and brings out the visible square patterns along the diagonal in
$SD$ in Figure 3a. Figure 3b shows the $FD$ for the random initialization case
with a low value of the $Corr(FD)$, displayed in the form of minutely visible
similarity with $SD$. However small, the visible similarity signifies the
correct predictions for the trivial case of speech segments with just the
speaker’s face visible in the frames. The performance of the random chance
system is quantified in terms of F1-scores in Table II.

Figure 3c shows the $FD$ post stage1 speech-face assignments, maximizing the
$Corr(FD)$ employing Algorithm 1 and displays significantly higher resemblance
with $SD$, quantified by the higher value of $Corr(FD)$. Figure 4 compares the
objective function value ($Corr(FD)$) at various stages for the videos in
VPCD, and shows the higher value fro stage1 than the random initialization
case, consistently for all the videos. The higher resemblance translates to
the enhanced active speaker detection performance, reported in terms of a
higher F1-score in Table II, compared to the random chance system. We
consistently observe a substantially higher F1-score than the random chance
baseline for all the videos in the VPCD. Although post optimization, the $FD$
is visibly similar to $SD$ (comparing Figure 3a&c), the amount of noise is
still notable when compared against the $FD$ obtained using the ground truth
active speaker faces, shown in Figure 3e. The same is quantitatively visible
in the difference between respective values of $Corr(FD)$ and is due to the
false positives introduced by the system for the speech segments with off-
screen speakers.

Figure 4: Evolution of the objective function $Corr(FD)$ at different stages
for the videos in VPCD.

###  IV-C Stage2: Off-screen speaker correction

The proposed strategy to segregate the speech segments with off-screen
speakers relies on the hypothesis that for such speech segments, since the
speaker’s identity from the speech is different from any of the faces
appearing in the frames, the row-wise correlations between $SD$ and $FD$ are
relatively lower. To validate this hypothesis, we first divide the set of
speech segments, using the ground truth, into $S_{\text{On}}$ (on-screen) and
$S_{\text{Off}}$ (off-screen) and study the distribution of row-wise
correlations between $SD$ and $FD$ for the two groups. We assign ground truth
active speaker face for $s_{i}\in S_{\text{On}}$ and a face randomly selected
from the set of temporally overlapping face tracks for $s_{i}\in
S_{\text{Off}}$ and then construct the $SD$ and $FD$ using all speech segments
$s_{i}\in S_{\text{On}}\cup S_{\text{Off}}$.

In Figure 5, we show the distributions of row-wise correlations between $SD$
and $FD$ for the two sets of speech segments: i) Off-screen, $S_{\text{Off}}$
and ii) On-screen, $S_{\text{On}}$, for the movie Hidden Figures. We observe a
notable difference in the two distributions, which we quantify using the non-
parametric Mann-Whitney U [35] test and obtain a p-value of $10^{-70}$
verifying that the difference between the two distributions is statistically
significant. We performed the same test for all other videos in VPCD and found
that the hypothesis held for all of them.

Figure 5: Plot showing the individual contribution ($corr_{i}$) to $O(D_{v})$
vs the change in contribution ($\Delta_{i}$) when substituted with other
character faces, for speech segments a) having background speakers ($S_{b}$),
b) having on-screen speakers ($S_{o}$).

In Table III we report the fractional duration of speech segments having off-
screen speakers ($S_{\text{Off}}:S_{\text{all}}$) for each video in the VPCD
and observe that the fraction can be significant for some videos and thus
validating the need to address the induced errors in active speaker detection
by the stage1 speech-face assignations. Using various values of $\tau$ in
equation 7, we compute an ROC curve to evaluate the efficacy of the proposed
strategy to classify each speech segment into $S_{\text{Off}}$ and
$S_{\text{On}}$. We report the area under ROC (auROC) curve in Table III for
all the videos and notice the good performance of the simple strategy to
discover speech segments with off-screen speakers.

TABLE III: Performance for the off-screen speech segment classification for
the videos in VPCD, in terms of area under the ROC curve (auROC).

Videos |  | off-screen
---
speech segments (%)
| Classification
---
performance (auROC)
Friends | 0.1 | 0.8074
TBBT | 0.1 | 0.8057
Sherlock | 0.18 | 0.6927
Hidden Figures | 0.21 | 0.8259
About Last Night | 0.17 | 0.6897

For all the videos in VPCD, we use $\tau=0.1$ to gather the speech segments
with off-screen speakers and classify the faces, earlier incorrectly marked as
active speaker faces in stage1, as non-speaking faces. In Figure 3d, we show
the obtained $FD$ for the movie, Hidden Figures and observe a visibly higher
similarity with the speech-identity distance matrix ($SD$), further quantified
in attained a higher value of $Corr(FD)$, compared to $FD$ post stage1. We
further observe that the $FD$ post stage2 looks even more similar to ground
truth $FD$. In Figure 4, we show that the objective function $Corr(FD)$
attains a higher value, after correcting the speech-face assignments for
speech segments with off-screen speakers, compared to stage1 assignments for
all the videos in VPCD.

We report the F1-score for the active speaker detection using the corrected
predictions in Table II (stage2) and compare them with the predictions from
the stage1. We observe that off-screen speaker correction increases the
F1-score consistently for all the videos. In Figure 6, we compare the
performance of the two stages on a precision-recall curve and observe that the
system’s precision improves significantly with a slight drop in the recall.
The enhanced precision is associated with removing the false positives
introduced by assigning an active speaker face to speech segments with off-
screen speakers (stage1), while the decline in the recall is due to the
incorrect classification of a speech segment to have an off-screen speaker in
stage2.

Figure 6: Increase in precision with a slight drop in recall introduced by
correcting the speech-face assignments for speech segments with off-screen
speakers (stage2).

###  IV-D Comparisons with state-of-the-art

We evaluate the performance of the proposed strategy involving finding an
active speaker face track for each speech segment and then removing the speech
segments with off-screen speakers on the videos from the publicly provided
validation split of the AVA dataset. We score the face boxes of the predicted
active speaker face tracks with the row-wise correlation
($correlation(SD[i],FD[i])$) values where a higher correlation value signifies
a better match in identities represented by the face and the speech. All other
face boxes are scored -1, the minimum possible correlation value. We then use
the AVA official implementation to evaluate the performance and report the
mean average precision (mAP) values aggregated over all the videos in Table
IV. We also report the performance of various recent state-of-the-art
techniques in Table IV.

TABLE IV: Comparison with the state-of-the-art methods on AVA active speaker
validation set in terms of mean average precision.

Methods | Stragetgy | mAP (%)
---|---|---
Roth et al. [9] | Supervised | 79.2
Zhang et al. [36] | Supervised | 84.0
Alxazar et al. [21] | Supervised | 87.1
Chung et al [14] | Supervised | 87.8
MAAS-TAN [22] | Supervised | 88.8
Proposed | Unsupervised | 71.40

We observe that the proposed unsupervised strategy of using the identity
information performs well, although not as well as other fully supervised
methods using videos from the AVA train set to model the visual activity in
the frames. The primary reason for the observed lower performance lies with
the nature of videos in the AVA dataset which consists of adverse audio
conditions. Nearly 65% of the speech segments with a visible speaker overlap
with either noise or music [9]. The presence of such noise conditions hinders
the ability of the employed speaker recognition models to capture the identity
information from speech segments, which naturally translates into poor active
speaker detection performance. We further elaborate on the effect of speaker
recognition performance on active speaker detection in section IV-E3.

Using the same method as AVA, we report mAP values for the videos in the VPCD
dataset (averaged over episodes for TV shows) in Table II. We observe that the
performance for videos of TV shows, specifically Friends and TBBT, is notably
higher than the movies (Hidden Figures and About Last Night). The scenes in
Friends and TBBT predominantly consist of just primary characters’ faces
(characters who actively speak in the video), unlike the movies, which have a
significant portion of scenes consisting of faces of background characters
(i.e., characters who do not speak anytime in the video). The presence of
primary characters’ faces provides useful disambiguating information since we
can gather their audio-visual identity information from the parts of videos
where they speak. While the background characters’ faces contribute to
increased confusion since there is no way we can acquire their identity
information. This leads to the displayed difference in the performances for
the TV shows and the movies. Additionally, a higher fraction of off-screen
speakers for the movies than the TV shows further contributes to the lower
performance for movies (Table III). Selected video clips, presenting the
system’s active speaker predictions for several videos from VPCD are present
in supplementary material.

###  IV-E Ablation studies

####  IV-E1 Effects of VAD performance

The availability of the speaker-homogeneous speech segments is fundamental to
the proposed approach. To obtain the speech segments, we use an off-the-shelf
voice activity detector [31]. To make the obtained speech segments speaker-
homogeneous, we use a simple approximation involving partitioning the speech
segments by the scene boundaries and ensuring a maximum duration of 1sec.
Since the number of speaker changes is constant in a video, splitting the
speech segments into a larger number of partitions (obtained by making the
partition length smaller) effectively decreases the fraction of speech
segments consisting of a speaker change. Furthermore, the errors introduced by
the speech segments with a speaker change can be the most for 1sec duration.

To evaluate the effect of the proposed simple proxy for speaker-homogeneous
speech segments, we compare the system’s performance, on videos from VPCD,
with the ideal case scenario. The VPCD annotations consist of character-wise
speech segments, enabling to obtain ideal case speaker-homogeneous speech
segments. We further segment the speech segments using the scene boundaries
and ensuring a maximum duration of 1 sec. As the speech recognition methods’
performance is susceptible to the duration of segments, splitting to maximum
duration of 1 sec eliminates the impact of the difference in durations of
speech segments in the two cases. In Figure 7 we compare the performance in
terms of mAP for all the videos in VPCD for the two cases: i) System VAD using
the proposed speaker-homogeneous proxy and ii) Oracle VAD using the ground
truth speaker-homogeneous speech segments.

We observe that the proposed setup using the VAD system and the speaker-
homogeneous proxy approximates the ideal case scenario quite well, reflected
in nearly equivalent mAP values for the two cases for most videos. On average,
the system’s performance attains nearly 91% of the performance of the ideal
case.

Figure 7: Comparison of the system’s performance against the scenario with
ideal case speaker-homogeneous speech segments, reported in terms of mAP for
video from VPCD. The system’s performance relative to the ideal case scenario
is shown beneath each video.

####  IV-E2 Time complexity vs performance

In section III-B we proposed to split the video into $p$ partitions and
maximize the objective function, $Corr(FD)$ for each partition independently
to reduce the time complexity of the system to $O(Ek\frac{N^{3}}{p^{2}})$. It
assumes that the obtained partitions have enough information to disambiguate
the active speakers. For implementation purposes, we partition a video to have
at maximum $L$ speech segments in each partition. Using $p=\frac{N}{L}$, we
represent the time complexity of the system as $O(EkNL^{2})$, making it a
linear function of the video’s length, $N$ and dependent on the parameter $L$.
The length of a partition $L$ signifies the context available to the video
split to solve the speech-face assignments, implying that the larger values of
$L$ likely provide more disambiguating information and are ideal for the
setup. In contrast, selecting a larger $L$ increases the time complexity of
the system, thus trading off the time for performance.

In Figure 8 we compare the performance of the system (reported in mAP) and the
computational time for various values of $L=[200,500,800]$ and for using total
video length at once $(L=N)$. For visualization purposes, we plot the
computational time on a logarithmic scale. We observe a significant
performance increase for $L=500$ compared with $L=200$ consistently for all
the videos in VPCD. Further increasing $L$, we see marginal improvement in the
performance along with a significant increase in computational time. Therefore
we use $L=500$ for all the videos providing enough context to the video
partitions with reasonable computational time. Interestingly we also observe a
marginal decrease in performance for the movies, Hidden Figures and About Last
Night, and the TV show Sherlock when using the total video length $(L=N)$.
This drop in performance is due to a more significant amount of noise in the
form of off-screen speakers (Table III) compared to other videos, which
accumulate while using a larger context.

Figure 8: Computational time (in logarithmic scale) for the system against the
performance (reported in mAP) for various values of the partition length $L$
for the videos in VPCD.

####  IV-E3 Effects of speech and face recognition quality

The central idea of this work relies on the fact that speech and face both
possess identity information and harness the certitude that the speaker’s face
and the underlying speech identify the same character. It makes capturing the
identity information from speech and face fundamental to the system. Here we
study the effect of the quality of captured speech and face identity
information on the system’s performance. We use two state-of-the-art speech
recognition systems to construct different speech identity representations: i)
ResNet-34 (default to the system) pretrained on Voxceleb2 dataset displaying
1.2% equal error rate (EER) for speaker recognition task on Voxceleb1 test
set. ii) X-vector embeddings extracted from a TDNN model trained on
Voxceleb1+Voxceleb2 training data and displays an EER of 3.2% on the same
Voxceleb1 test set. Note that the lower the EER better is the speaker
recognition performance; thus, ResNet-34 performs better for capturing speaker
identity information from speech than x-vectors.

In Figure 9, we show the performance of the active speaker detection system
for videos from VPCD, utilizing the two speaker recognition systems. We
observe that the performance displayed by the sub-optimal x-vectors embeddings
is consistently lower for all the videos than the ResNet-derived embeddings.
Using a non-parametric ManWhitneyU test, we noticed that the difference
between the performance of the two systems is statistically significant. This
implies that the sub-optimal speaker recognition features lead to degraded
active speaker recognition performance. It further explains that the low
performance for the AVA validation set (reported in Table IV) is due to the
sub-optimal speaker identity embeddings attributed to the noisy speech
conditions.

Similarly, to capture the impact of the face recognition quality, we use two
state-of-the-art face recognition systems to construct the face identity
representations: i) ResNet-50 trained on VGGFace2 (default to the system) and
ii) Facenet trained on VGGFace2, compared on IJB-c displaying 0.95 and 0.66
True acceptance rate (TAR) respectively. Higher TAR implies better face
recognition performance, pointing out that the Resnet-50 is a superior face
recognition system to the Facenet. In Figure 9 we present the active speaker
detection performance comparison of systems employing the two different face
recognition systems and the same speaker recognition system (ResNet-34). We
observe that the difference in performance is marginal and shows no
statistical significance. It suggests that the system is more tolerable to
degradation in face recognition performance.

Figure 9: Comparison of active speaker detection performance for systems using
different speech/face recognition architectures, on videos from the VPCD
dataset. For speech embeddings, Resnet is superior to x-vectors and for face
Resnet is superior to the Facenet.

###  IV-F Discussion

The proposed framework relies on the assumption that the video under
consideration has enough information to disambiguate the characters in the
video. The presence of scenes with just the speaking character in the frames
is one of the simple examples of such character disambiguating information, as
it provides an instant speech-face association. The videos intending to
capture the speaking characters will likely have relevant information to
derive the speech-face association based on the co-occurrence of the speech
and faces through the video. On the contrary, one of the adverse limiting
cases for the system is a video with a fixed set of faces appearing in the
frames at all times. A simple example can be a panel discussion shot with a
static camera, capturing all the panelists at all times. In such a video, the
system will have no way to correctly associate the identity from the speech
with one of the faces on the screen. Columbia dataset [11]: a video of a panel
discussion is very similar to such a scenario. Even though the video is 85 min
long, the annotations are available for 35 min duration, comprising six
speakers.

We evaluate the performance of the proposed system on the Columbia dataset and
report the speaker-wise weighted F1-score, a widely used metric for evaluating
this dataset, in Table V, along with performance reported by various other
methods. We observe that the F1 score is pretty low for three speakers (Lieb,
Long, and Sick), while for the other three, it is at par with other systems.
We further investigated and observed that the face distance matrix $FD$
obtained using the predicted active speaker faces attains a high value of the
objective function $Corr(FD)$. The speech ($SD$) and the face distance ($FD$)
matrices are shown in Figure 10a & c. The obtained $Corr(FD)$ is close to the
one displayed by the ground truth active speaker faces, shown in Figure 10b,
suggesting the intended working for the proposed system.

TABLE V: Comparison of the speaker-wise weighted F1 scores for all the
speakers in Columbia dataset.

Methods | Abbas | Bell | Boll | Lieb | Long | Sick | Avg
---|---|---|---|---|---|---|---
Chakravarty et al. [11] | - | 82.9 | 65.8 | 73.6 | 86.9 | 81.8 | 78.2
Shahid et al [37] | - | 89.2 | 88.8 | 85.8 | 81.4 | 86 | 86.2
Chung et al. [13] | - | 93.7 | 83.4 | 86.8 | 97.7 | 86.1 | 89.5
Afouras et al. [18] | - | 92.6 | 82.4 | 88.7 | 94.4 | 95.9 | 90.8
S-VVAD [38] | - | 92.4 | 97.2 | 92.3 | 95.5 | 92.5 | 94
RealVAD [39] | - | 92 | 98.9 | 94.1 | 89.1 | 92.8 | 93.4
TalkNet [40] | - | 97.1 | 90.0 | 99.1 | 96.6 | 98.1 | 96.2
Proposed | 97.0 | 95.0 | 91.0 | 65.3 | 0.1 | 34.6 | 65.0
Proposed (assisted) | 96.7 | 95.3 | 90.5 | 98.2 | 93.2 | 96.1 | 95.0

Figure 10: Distance matrices for Columbia dataset with $Corr(FD)$. a) Speech
distance matrix $SD$. b) $FD$ with ground truth active speaker faces. c) $FD$
post stage2 for proposed system. d) $FD$ post stage 2 for the system assisted
with 15% ground truth active speaker faces.

On visual inspection of the predicted active speaker faces, we observed that
the system incorrectly associates Long’s speech with Sick’s face and Lieb’s
with Long’s face. The system selects Sick’s face as an active speaker whenever
Long speaks and similarly for the other two speakers. Relevant frames from the
video with the system’s predictions (green box: active speaker) and manually
marked speakers’ identities are shown in Figure 11a. We further observed that
Sick always appears with Long throughout the video, providing the system with
insufficient information to resolve the speech-face association. Similarly,
Lieb always appears with Long, limiting the system to solve the discrepancy.

To close the loop on the system’s performance on the Columbia dataset, we
simulate a setup by manually forcing disambiguating information into the
video, which assists the system in solving the speech-face association. We
randomly select 15% of the speech segments of each speaker and initialize the
corresponding active speaker faces using the ground truth active speaker
faces. While maximizing the objective function $Corr(FD)$ (using Algorithm 1
in stage1), we forbid updating the active speaker face assignment for the
selected speech segments. It replicates the trivial case of a scene with just
the speaker’s face visible in frames, thus synthetically injecting information
to resolve speech-face associations. We show the face distance matrix ($FD$)
for the obtained active speaker faces in Figure 10d and note that the
objective function $Corr(FD)$ is nearly equivalent to the one shown by the
ground truth active speaker faces (Figure 10b). We report speaker-wise
F1-score in Table V and observe that the assisted system performs similar to
state-of-the-art methods for all the speakers. In Figure 11b we show the
frames with corrected speech-face associations. Video clips highlighting the
system’s initial incorrect active speaker predictions and the later obtained
corrected predictions using the assistance from the ground truth are present
in supplementary material. We point out that this easy way of incorporating
external information can be extended to integrate complementary information
from visual-activity-based systems to generate a comprehensive solution to
active speaker detection.

Figure 11: Sample frames from Columbia dataset with manually marked identities
of the faces and the name of the current speaker. a) The frames shows the
incorrect speech-face associations by the proposed system. b) The frames shows
the corrected speech-face associations when assisted with 15% ground truth
faces

##  V Summary and Conclusion

We presented a cross-modal unsupervised framework for active speaker detection
in videos, harnessing the character identity information in the speech and the
faces of the speakers. We evaluated the proposed system on three benchmark
datasets, consisting of various videos from entertainment media (TV shows and
movies) and broadcast media (panel discussion), and showed competitive
performance compared to state-of-the-art fully supervised methods. The
framework utilizes off-the-shelf speech and face identity representation
systems and shares their limitations when speech accompanies noise and music.

The framework provides an easy way to inject external information, enabling it
to collaborate with other methods. It can incorporate high-confidence
predictions of other state-of-the-art methods and even inputs from expert
humans in the loop, enabling a provision to provide supervision at various
degrees. One of the future directions can be to include the complementary set
of information from visual-activity-based methods providing more
disambiguating information to the system and thus further enhance the
performance.

## References

  * [1] K. Somandepalli, T. Guha, V. R. Martinez, N. Kumar, H. Adam, and S. Narayanan, “Computational media intelligence: Human-centered machine analysis of media,” _Proceedings of the IEEE_ , vol. 109, no. 5, 2021.
  * [2] R. Sharma and S. Narayanan, “Audio visual character profiles for detecting background characters in entertainment media,” _arXiv preprint arXiv:2203.11368_ , 2022.
  * [3] H. Bredin and G. Gelly, “Improving speaker diarization of tv series using talking-face detection and clustering,” in _ACM international conference on Multimedia_ , 2016.
  * [4] J. S. Chung, J. Huh, A. Nagrani, T. Afouras, and A. Zisserman, “Spot the conversation: speaker diarisation in the wild,” _arXiv preprint arXiv:2007.01216_ , 2020.
  * [5] R. Sharma and S. Narayanan, “Using active speaker faces for diarization in tv shows,” 2022.
  * [6] M. Bendris, D. Charlet, and G. Chollet, “Lip activity detection for talking faces classification in tv-content,” in _International conference on machine vision_ , 2010, pp. 187–190.
  * [7] A. Owens and A. A. Efros, “Audio-visual scene analysis with self-supervised multisensory features,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 631–648.
  * [8] H. Zhao, C. Gan, A. Rouditchenko, C. Vondrick, J. McDermott, and A. Torralba, “The sound of pixels,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 570–586.
  * [9] J. Roth, S. Chaudhuri, O. Klejch, R. Marvin, A. Gallagher, L. Kaver, S. Ramaswamy, A. Stopczynski, C. Schmid, Z. Xi _et al._ , “Ava active speaker: An audio-visual dataset for active speaker detection,” in _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2020, pp. 4492–4496.
  * [10] A. Brown, V. Kalogeiton, and A. Zisserman, “Face, body, voice: Video person-clustering with multiple modalities,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 3184–3194.
  * [11] P. Chakravarty and T. Tuytelaars, “Cross-modal supervision for learning active speaker detection in video,” in _European Conference on Computer Vision_. Springer, 2016, pp. 285–301.
  * [12] R. Arandjelovic and A. Zisserman, “Objects that sound,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 435–451.
  * [13] J. S. Chung and A. Zisserman, “Out of time: automated lip sync in the wild,” in _Asian conference on computer vision_. Springer, 2016, pp. 251–263.
  * [14] J. S. Chung, “Naver at activitynet challenge 2019–task b active speaker detection (ava),” _arXiv preprint arXiv:1906.10555_ , 2019.
  * [15] P. Chakravarty, S. Mirzaei, T. Tuytelaars, and H. Van hamme, “Who’s speaking? audio-supervised classification of active speakers in video,” in _Proceedings of the 2015 ACM on International Conference on Multimodal Interaction_ , 2015, pp. 87–90.
  * [16] R. Sharma, K. Somandepalli, and S. Narayanan, “Toward visual voice activity detection for unconstrained videos,” in _2019 IEEE International Conference on Image Processing (ICIP)_.
  * [17] ——, “Cross modal video representations for weakly supervised active speaker localization,” _arXiv e-prints_ , pp. arXiv–2003, 2020.
  * [18] T. Afouras, A. Owens, J. S. Chung, and A. Zisserman, “Self-supervised learning of audio-visual objects from video,” in _European Conference on Computer Vision_. Springer, 2020, pp. 208–224.
  * [19] Y. J. Kim, H.-S. Heo, S. Choe, S.-W. Chung, Y. Kwon, B.-J. Lee, Y. Kwon, and J. S. Chung, “Look who’s talking: Active speaker detection in the wild,” _arXiv preprint arXiv:2108.07640_ , 2021.
  * [20] C. Huang and K. Koishida, “Improved active speaker detection based on optical flow,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_ , 2020, pp. 950–951.
  * [21] J. L. Alcázar, F. Caba, L. Mai, F. Perazzi, J.-Y. Lee, P. Arbeláez, and B. Ghanem, “Active speakers in context,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 12 465–12 474.
  * [22] J. León-Alcázar, F. C. Heilbron, A. Thabet, and B. Ghanem, “Maas: Multi-modal assignation for active speaker detection,” _arXiv preprint arXiv:2101.03682_ , 2021.
  * [23] K. Hoover, S. Chaudhuri, C. Pantofaru, I. Sturdy, and M. Slaney, “Using audio-visual information to understand speaker activity: Tracking active speakers on and off screen,” in _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2018, pp. 6558–6562.
  * [24] T. J. Park, N. Kanda, D. Dimitriadis, K. J. Han, S. Watanabe, and S. Narayanan, “A review of speaker diarization: Recent advances with deep learning,” _Computer Speech & Language_, vol. 72, p. 101317, 2022.
  * [25] I. Kapsouras, A. Tefas, N. Nikolaidis, and I. Pitas, “Multimodal speaker diarization utilizing face clustering information,” in _International Conference on Image and Graphics_. Springer, 2015, pp. 547–554.
  * [26] J. S. Chung, J. Huh, S. Mun, M. Lee, H.-S. Heo, S. Choe, C. Ham, S. Jung, B.-J. Lee, and I. Han, “In defence of metric learning for speaker recognition,” _Interspeech 2020_ , Oct 2020. [Online]. Available: http://dx.doi.org/10.21437/interspeech.2020-1064
  * [27] J. S. Chung, A. Nagrani, and A. Zisserman, “Voxceleb2: Deep speaker recognition,” _Interspeech 2018_ , Sep 2018. [Online]. Available: http://dx.doi.org/10.21437/Interspeech.2018-1929
  * [28] Y. Guo, L. Zhang, Y. Hu, X. He, and J. Gao, “Ms-celeb-1m: A dataset and benchmark for large-scale face recognition,” in _European conference on computer vision_. Springer, 2016, pp. 87–102.
  * [29] J. Hu, L. Shen, and G. Sun, “Squeeze-and-excitation networks,” _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , Jun 2018. [Online]. Available: http://dx.doi.org/10.1109/CVPR.2018.00745
  * [30] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” 2015. [Online]. Available: https://arxiv.org/abs/1512.03385
  * [31] H. Bredin, R. Yin, J. M. Coria, G. Gelly, P. Korshunov, M. Lavechin, D. Fustes, H. Titeux, W. Bouaziz, and M.-P. Gill, “pyannote.audio: neural building blocks for speaker diarization,” in _ICASSP 2020, IEEE International Conference on Acoustics, Speech, and Signal Processing_ , 2020.
  * [32] A. Pardo, F. C. Heilbron, J. L. Alcázar, A. Thabet, and B. Ghanem, “Moviecuts: A new dataset and benchmark for cut type recognition,” _arXiv preprint arXiv:2109.05569_ , 2021.
  * [33] J. Deng, J. Guo, E. Ververas, I. Kotsia, and S. Zafeiriou, “Retinaface: Single-shot multi-level face localisation in the wild,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 5203–5212.
  * [34] A. Bewley, Z. Ge, L. Ott, F. Ramos, and B. Upcroft, “Simple online and realtime tracking,” in _2016 IEEE International Conference on Image Processing (ICIP)_ , 2016, pp. 3464–3468.
  * [35] H. B. Mann and D. R. Whitney, “On a test of whether one of two random variables is stochastically larger than the other,” _The annals of mathematical statistics_ , pp. 50–60, 1947.
  * [36] Y.-H. Zhang, J. Xiao, S. Yang, and S. Shan, “Multi-task learning for audio-visual active speaker detection,” _The ActivityNet Large-Scale Activity Recognition Challenge_ , pp. 1–4, 2019.
  * [37] M. Shahid, C. Beyan, and V. Murino, _Comparisons of Visual Activity Primitives for Voice Activity Detection_ , 09 2019, pp. 48–59.
  * [38] ——, “S-vvad: Visual voice activity detection by motion segmentation,” in _2021 IEEE Winter Conference on Applications of Computer Vision (WACV)_ , 2021, pp. 2331–2340.
  * [39] C. Beyan, M. Shahid, and V. Murino, “Realvad: A real-world dataset and a method for voice activity detection by body motion analysis,” _IEEE Transactions on Multimedia_ , vol. 23, pp. 2071–2085, 2021.
  * [40] R. Tao, Z. Pan, R. K. Das, X. Qian, M. Z. Shou, and H. Li, “Is someone speaking?” _Proceedings of the 29th ACM International Conference on Multimedia_ , Oct 2021. [Online]. Available: http://dx.doi.org/10.1145/3474085.3475587