#  Emergent non-invertible symmetries in $\mathcal{N}=4$ Super-Yang-Mills
theory

Orr Sela  Mani L. Bhaumik Institute for Theoretical Physics, Department of
Physics and Astronomy, University of California, Los Angeles, CA 90095, USA

###### Abstract

One of the simplest examples of non-invertible symmetries in higher dimensions
appears in 4d Maxwell theory, where its $SL(2,\mathbb{Z})$ duality group can
be combined with gauging subgroups of its electric and magnetic 1-form
symmetries to yield such defects at many different values of the coupling.
Even though $\mathcal{N}=4$ Super-Yang-Mills (SYM) theory also has an
$SL(2,\mathbb{Z})$ duality group, it only seems to share two types of such
non-invertible defects with Maxwell theory (known as duality and triality
defects). Motivated by this apparent difference, we begin our investigation of
the fate of these symmetries by studying the case of 4d $\mathcal{N}=4$ $U(1)$
gauge theory which contains Maxwell theory in its content. Surprisingly, we
find that the non-invertible defects of Maxwell theory give rise, when
combined with the standard $U(1)$ symmetry acting on the free fermions, to
defects which act on local operators as elements of the $U(1)$ outer-
automorphism of the $\mathcal{N}=4$ superconformal algebra, an operation that
was referred to in the past as the ”bonus symmetry”. Turning to the nonabelian
case of $\mathcal{N}=4$ SYM, the bonus symmetry is not an exact symmetry of
the theory but is known to emerge at the supergravity limit. Based on this
observation we study this limit and show that if it is taken in a certain way,
non-invertible defects that realize different elements of the bonus symmetry
emerge as approximate symmetries, in analogy to the abelian case.

##  I Introduction

A recent exciting development in quantum field theory is the understanding
that symmetries are represented by topological defects with properties that
might go beyond the traditional notion of a symmetry. Non-invertible
symmetries, in particular, correspond to defects with exotic non-group-like
fusion rules and by now have been identified and investigated in diverse
setups in different areas of physics (see [1, 2] for reviews).

One of the simplest instances of such non-invertible symmetries in higher
dimensions appears in 4d Maxwell theory. As shown in [3, 4], when its
$SL(2,\mathbb{Z})$ duality is combined with gauging a $\mathbb{Z}_{N}^{(1)}$
subgroup of its electric 1-form symmetry (with a Dirichlet boundary condition
for the $\mathbb{Z}_{N}^{(1)}$ 2-form gauge field, and possibly with stacking
an SPT phase for it), one is able to construct non-invertible defects, known
as duality and triality defects, at special values of the complexified
coupling $\tau$. Later [5] (see also [6]), employing the observation that
gauging a subgroup of the electric or magnetic 1-form symmetry results again
in Maxwell theory but with a different coupling, the $SL(2,\mathbb{Z})$
duality was combined with gauging such subgroups (in a non-anomalous way) to
yield an $SL(2,\mathbb{Q})$ operation on the theory which acts in an analogous
way to $SL(2,\mathbb{Z})$ (this construction will be reviewed in the next
section). This, in turn, enabled to find a new non-invertible defect at any
value of $\tau$ that is fixed by an element of $SL(2,\mathbb{Q})$ (which acts
on $\tau$ in the standard way by a fractional transformation), thereby
generalizing the results of [3, 4].

It is natural at this point to ask what part of this construction has a
counterpart in non-abelian theories. Due to the central role played by
$SL(2,\mathbb{Z})$ duality, the natural candidate to examine, which will also
be the main focus of this note, is $\mathcal{N}=4$ Super-Yang-Mills (SYM)
theory. Considering for concreteness the theory with $SU(N)$ gauge group, even
though it shares with Maxwell theory an $SL(2,\mathbb{Z})$ duality group, it
only has an electric 1-form symmetry which is $\mathbb{Z}_{N}^{(1)}$ and whose
gauging changes the global structure of the theory and cannot be associated
with an action on the coupling $\tau$. As a result, we do not seem to have an
$SL(2,\mathbb{Q})$ operation as in Maxwell theory, and a non-invertible defect
of the type we discuss only has the chance of being found at values of $\tau$
that are fixed by $SL(2,\mathbb{Z})$.111Notice that we only consider here
defects involving $SL(2,\mathbb{Z})$ duality operations. Other non-invertible
defects might exist at different values of $\tau$, such as condensation
defects [7, 8] which are found at any $\tau$. Indeed, previous works [4, 9]
only found the duality and triality defects in $\mathcal{N}=4$ SYM, with the
additional defects discussed in [5] being absent.

Here, in order to investigate this apparent difference between Maxwell and
$\mathcal{N}=4$ SYM theories in a more systematic way, and to try to extract a
clue that will guide us towards the fate of these additional symmetries in
$\mathcal{N}=4$ SYM, we begin by examining more closely a theory which is in a
sense the intersection of these two: 4d $\mathcal{N}=4$ abelian $U(1)$ gauge
theory. This theory both has an $\mathcal{N}=4$ superconformal algebra and
contains free Maxwell theory in its content. Surprisingly, we will find that
the additional symmetries of [5], when combined with the standard $U(1)$
symmetry acting on the free fermions in the theory, realize at different
values of the coupling different elements of a non-invertible $U(1)$
$R$-symmetry that acts on the local operators as the $U(1)$ outer-automorphism
of the $\mathcal{N}=4$ superconformal algebra. Moreover, since correlators of
local operators do not depend on the coupling in a nontrivial way in this
theory (i.e. one can normalize the operators such that the correlators are
coupling independent), they will satisfy the selection rules of the entire
$U(1)$ at any value of the coupling.

Apriori, the $U(1)$ outer-automorphism of the $\mathcal{N}=4$ superconformal
algebra may or may not be a symmetry of the theory. This question was
discussed in detail in the past in [10, 11] (see also [12, 13, 14, 15]), where
this potential symmetry was referred to as the bonus symmetry. It was
demonstrated that in the abelian theory we consider, the various local
operators have definite charges under this $U(1)$ and that it is respected by
the equations of motion and supersymmetry transformations. However, the nature
of this symmetry seemed to be mysterious as it was clearly not a symmetry of
the Lagrangian and the field strength appeared to be charged under it. One of
the observations made in this note is therefore the identification in modern
terms of the bonus symmetry of the $U(1)$ gauge theory as a non-invertible
symmetry, with different elements realized at different values of the
coupling. These elements, in turn, mainly correspond to the defects discovered
in [5], and while their action on local operators is the one identified in the
past in [10, 11], their action on line operators is highly nontrivial [5].

Once we have identified the non-invertible defects of Maxwell theory discussed
above as the key ingredient giving rise to the bonus symmetry of the
$\mathcal{N}=4$ theory with $U(1)$ gauge group, it is time to turn to the
nonabelian theory and use the bonus symmetry of its algebra as our guide in
searching for new non-invertible symmetries analogous to those of Maxwell
theory. Unlike the abelian case, in $\mathcal{N}=4$ SYM the bonus symmetry is
not an exact symmetry of the theory [10, 11]. This is indeed consistent with
the fact that only the duality and triality defects have been identified as
exactly-topological defects in the past. However, as discussed in detail in
[10], based on holographic duality the bonus symmetry is expected to emerge as
an approximate symmetry in the limit where the gravity dual of $\mathcal{N}=4$
SYM is approximated by type IIB supergravity (i.e. when both $N$ and
$g_{YM}^{2}N$ are very large). This emergence, in turn, follows from the
enhancement of the $SL(2,\mathbb{Z})$ duality symmetry of type IIB string
theory to $SL(2,\mathbb{R})$ in this supergravity limit (at least when the
fields are not treated as quantized), of which the $U(1)$ bonus symmetry is
the maximal compact subgroup. This observation then suggests that the place in
$\mathcal{N}=4$ SYM in which we should look for new defects analogous to those
of Maxwell theory is exactly at this limit.

In order to do it, we will first investigate the global structure of
$\mathcal{N}=4$ SYM at the large-$N$ limit (without specifying the value of
the ’t Hooft coupling $\lambda$). Focusing on a gauge group of the form
$SU(N^{2})$ we will find, following an approach closely related to the one
recently discussed in [16], that the large-$N$ limit can be taken in such a
way that the $\mathbb{Z}_{N^{2}}^{(1)}$ 1-form symmetry turns into
$\mathbb{\widehat{Q}}^{(1)}$ at the limit, where $\mathbb{\widehat{Q}}$ is
called the ring of profinite rational numbers (which will be defined below)
and contains the rationals $\mathbb{Q}$ as a subring. As we will show, this
results in the $SL(2,\mathbb{Z}_{N^{2}})$ group, associated with the
operations of gauging the $\mathbb{Z}_{N^{2}}^{(1)}$ 1-form symmetry and
stacking an SPT phase at finite $N$, turning into $SL(2,\mathbb{\widehat{Q}})$
at this limit. This will then allow us to identity at the supergravity regime
(i.e. when $\lambda$ is taken to be very large) an $SL(2,\mathbb{Q})\subset
SL(2,\mathbb{R})$ duality group, consistent with this structure, which when
combined with the $SL(2,\mathbb{Q})\subset SL(2,\mathbb{\widehat{Q}})$
operations yields new non-invertible defects. We obtain a new such defect at
any value of $\lambda$ (or $\tau$) that is both in the supergravity regime and
is fixed by an element of $SL(2,\mathbb{Q})$, analogously to the situation in
Maxwell theory.222Note that unlike $SL(2,\mathbb{Z})$, $SL(2,\mathbb{Q})$ can
fix a $\tau$ that corresponds to an arbitrarily small gauge coupling (or
$1/\textrm{Im}(\tau)$). In addition, defects at different such couplings
realize different elements of the bonus symmetry, as in the case of the
$\mathcal{N}=4$ $U(1)$ gauge theory.

##  II $\mathcal{N}=4$ abelian $U(1)$ gauge theory

In this section we would like to investigate the way in which the non-
invertible defects of Maxwell theory that include $SL(2,\mathbb{Z})$ duality
transformations appear in the 4d $\mathcal{N}=4$ theory with $U(1)$ gauge
group. As discussed in the previous section, we are mainly interested in the
relation between these defects and the bonus (or outerautomorphism) symmetry
of the $\mathcal{N}=4$ theory.

Before beginning with reviewing the non-invertible defects of Maxwell theory,
let us take a small detour and discuss some hints for them in the classical
theory. Defining the self dual and anti self dual field strengths
$F_{mn}^{\pm}=\frac{1}{2}(F_{mn}\pm\frac{1}{2}\epsilon_{mnpq}F^{pq})$, the
Lagrangian in Euclidean signature takes the form

$\mathcal{L}=\frac{i}{8\pi}\left(\overline{\tau}F_{mn}^{+}F^{+mn}-\tau
F_{mn}^{-}F^{-mn}\right).$ (1)

Clearly, there does not seem to be any (0-form) symmetry beyond charge
conjugation that will act on $F_{mn}^{\pm}$ (and not on $\tau$) while leaving
the Lagrangian invariant. However, since our modern understanding of
symmetries in quantum field theory is in terms of topological defects, we
would actually like to look for symmetries of the stress tensor instead of the
Lagrangian. This is because a topological defect necessarily acts trivially on
the stress tensor (alternatively, the displacement operator on the defect
vanishes), but does not necessarily correspond to a symmetry of the
Lagrangian. Let us therefore look for classical operations on the fields that
will leave the stress tensor invariant.333Note that this is not a sufficient
condition for a symmetry in the quantum theory. For example, the $U(1)$ axial
symmetry of QCD leaves the stress tensor invariant but does not correspond to
a topological defect in the quantum theory (only a discrete subgroup of it
does). In Maxwell theory, we have

$T_{mn}=\frac{\textrm{Im}\left(\tau\right)}{4\pi}\left(F_{mk}^{+}F_{\quad
n}^{-k}+F_{mk}^{-}F_{\quad n}^{+k}\right)$ (2)

and we can readily observe that $F^{\pm}\rightarrow e^{\pm i\varphi}F^{\pm}$
is a symmetry of $T_{mn}$. This operation is also a symmetry of the equations
of motion, but a bit strange at first sight since the field strength is
charged under it.

To understand if and how this $U(1)$ symmetry can appear in the quantum
theory, we notice that its action on $F^{\pm}$ implies that it rotates between
$F$ and $i\star F$, or alternatively between the electric and magnetic fields.
This suggests that if this symmetry is realized at the quantum level, it
involves the $SL(2,\mathbb{Z})$ duality of Maxwell theory. This was indeed
shown to be the case in [5], as we now briefly review.

We begin with the observation that gauging e.g. a $\mathbb{Z}_{p}^{(1)}$
subgroup of the $U(1)^{(1)}_{e}$ electric 1-form symmetry results in a theory
which has the same coupling $\tau$ but with the gauge field $A$ having fluxes
$\mathbb{Z}/p$ (and with the charges of Wilson lines being $p\mathbb{Z}$).
Writing the gauge field as $A=A^{\prime}/p$ with $A^{\prime}$ having fluxes
$\mathbb{Z}$, we can then rewrite the theory in terms of $A^{\prime}$ and end
up with the same Lagrangian and gauge-field fluxes we had at the beginning but
with coupling $\tau/p^{2}$ instead of $\tau$. During this total transformation
consisting of 1-form gauging plus rescaling, a Wilson line with charge $p$ in
the original theory maps to a Wilson line with charge $1$ at the end, while an
’t Hooft line with charge $1$ maps to one with charge $p$. We see that we can
represent this transformation by the matrix $\textrm{diag}(1/p,p)$ acting in
the usual way on the doublet of electric and magnetic 1-form charges $(e,m)$
and by a fractional transformation on the coupling $\tau$, just like the way
$SL(2,\mathbb{Z})$ acts. Overall, combining such (non-anomalous) electric and
magnetic 1-form gauging with the usual $SL(2,\mathbb{Z})$ duality yields the
following $SL(2,\mathbb{Q})$ operation,

$\left(\begin{array}[]{c}e\\\
m\end{array}\right)\rightarrow\left(\begin{array}[]{cc}q_{1}&q_{2}\\\
q_{3}&q_{4}\end{array}\right)\left(\begin{array}[]{c}e\\\
m\end{array}\right)\quad,\quad\tau\rightarrow\frac{q_{1}\tau+q_{2}}{q_{3}\tau+q_{4}}$
(3)

with the corresponding matrix being an element of $SL(2,\mathbb{Q})$. Such a
matrix can be obtained by the following sequence of operations (with
$q_{4}=(1+q_{2}q_{3})/q_{1}$)

$\left(\begin{array}[]{cc}q_{1}&q_{2}\\\
q_{3}&q_{4}\end{array}\right)=\left(\begin{array}[]{cc}1&0\\\
\frac{q_{3}}{q_{1}}&1\end{array}\right)\left(\begin{array}[]{cc}1&q_{1}q_{2}\\\
0&1\end{array}\right)\left(\begin{array}[]{cc}q_{1}&0\\\
0&q_{1}^{-1}\end{array}\right),$ (4)

where the rightmost matrix corresponds to gauging subgroups of the electric
and magnetic 1-form symmetries, the middle matrix to shifting the
$\theta$-term by a rational number which can be obtained by a $CTST$-type
magnetic gauging444Alternatively, this matrix can be obtained by combining
$T$-duality with gauging a subgroup of the electric and magnetic 1-form
symmetries [5]. (where $C$ denotes charge conjugation, $S$ gauging a subgroup
of the magnetic 1-form symmetry and $T$ stacking an SPT phase) [17], and the
left matrix can be obtained from the middle one by combining it with an
$S$-duality operation and its inverse, see [5] for more details. It is
important to comment that the operations in (4) are performed sequentially
such that in each step we return to Maxwell theory with gauge fluxes
$\mathbb{Z}$ and some coupling, such that there is no tension between these
operations and the anomaly between the electric and magnetic 1-form
symmetries.

At this point one can easily check that every coupling of the form

$\tau=\frac{q_{1}-q_{4}+i\sqrt{4-\left(q_{1}+q_{4}\right)^{2}}}{2q_{3}}$ (5)

with $q_{1}$, $q_{2}$ and $q_{4}$ any rationals satisfying $2>q_{1}+q_{4}$ and
$q_{3}\neq 0$ (such that the coupling $g$ is kept real) is invariant under the
corresponding $SL(2,\mathbb{Q})$ element, with $q_{2}=(q_{1}q_{4}-1)/q_{3}$.
We therefore see that topological defects corresponding to different elements
of $SL(2,\mathbb{Q})$ are realized at different values of the coupling.555Let
us comment that each such element of $SL(2,\mathbb{Q})$ can be realized by
more than one defect, differing by condensation defects. Moreover, such
defects rotate between the electric and magnetic fields with different
rotation angles, and since correlators of local operators do not depend on
$\tau$ in a nontrivial way in this theory, such correlators will respect the
selection rules of the entire $U(1)$ operation discussed below Eq. (2).
Notice, however, that this $U(1)$ is not really a symmetry of the theory and
is not associated with a topological operator. For more details on this
construction, as well as explicit Lagrangian descriptions of such
$SL(2,\mathbb{Q})$ defects and a demonstration of their non-invertibility, see
[5].

Turning now to the $\mathcal{N}=4$ $U(1)$ gauge theory, the status of its
$U(1)$ bonus symmetry [10] is very similar to that of the $U(1)$ ”symmetry” of
Maxwell theory we discussed above. To see it, let us begin with the basic
description of the theory. We consider a free theory consisting of a $U(1)$
gauge field with field strength $F_{(\alpha\beta)}$, fermions
$\psi_{I\alpha}$, $\overline{\psi}_{\dot{\alpha}}^{I}$ and real scalars
$\phi_{[IJ]}$, where $I$ is the index of the fundamental representation of
$SU(4)_{R}$ and $\alpha$, $\dot{\alpha}$ the usual indices of $SU(2)_{L,R}$.
The Lagrangian is simply given by the sum of the kinetic terms of the fields,
with the exactly marginal coupling $\tau$ (and $\overline{\tau}$) an overall
factor. Since this theory contains Maxwell theory, it also contains the
defects we discussed above, realizing when acting on local operators different
elements of a $U(1)$ symmetry which we will denote by $U(1)_{F}$. As its name
suggests, among the basic fields of the theory only the field strength is
charged under $U(1)_{F}$ (with charge $1$ for $F_{(\alpha\beta)}$ and $-1$ for
$\overline{F}_{(\dot{\alpha}\dot{\beta})}$). In addition, there is a
$U(1)_{\psi}$ symmetry under which the free fermions $\psi_{I\alpha}$ are
charged with charge 1, and a particularly natural combination of these two
symmetries is $U(1)_{Y}=-U(1)_{\psi}-2U(1)_{F}$ under which the supercurrent
has a well-defined charge of $-1$ (indeed, this will be the charge of terms
like $F_{(\alpha\beta)}\overline{\psi}_{\dot{\alpha}}^{I}$ and
$\partial_{\alpha\dot{\alpha}}\phi^{[IJ]}\psi_{J\beta}$ in it). Since the
supercharges are charged under $U(1)_{Y}$, it is an $R$-symmetry, and in fact
this is exactly the bonus symmetry as defined in [10]. We therefore see that
in this $\mathcal{N}=4$ $U(1)$ gauge theory, the bonus symmetry appears as an
ordinary 0-form $U(1)$ symmetry at the level of local correlators, but is in
fact given by (in general) non-invertible defects realizing at different
values of the coupling different elements of it.

Motivated by this link between the non-invertible defects in Maxwell theory
associated with its $SL(2,\mathbb{Q})$ operation and the bonus symmetry of the
$\mathcal{N}=4$ $U(1)$ gauge theory, we continue in the next section to non-
abelian $\mathcal{N}=4$ theories with the aim of using their bonus symmetry as
a guide for finding new analogous defects.

##  III $\mathcal{N}=4$ Super-Yang-Mills Theory

As discussed in the introduction, the bonus symmetry is not an exact symmetry
of $\mathcal{N}=4$ SYM, in the sense that the corresponding selection rules
are clearly violated in general by local correlators. However, in the
supergravity limit of the theory (when both $N$ and $g_{YM}^{2}N$ are very
large) we obtain a description in terms of type IIB supergravity on
$AdS_{5}\times S^{5}$, and when the fields are not treated as quantized there
is a well-known $SL(2,\mathbb{R})$ symmetry acting on them (enhancing the
standard $SL(2,\mathbb{Z})$ duality of IIB string theory). The bonus symmetry
then emerges as the $U(1)$ subgroup of this $SL(2,\mathbb{R})$ that fixes a
given value of the coupling $\tau$, and the corresponding selection rules are
expected to be satisfied by correlators of local operators that can be
computed using the supergravity approximation [10].

This suggests that defects analogous to the ones discussed in the previous
section might emerge as approximate symmetries in the supergravity limit. For
this to be the case, the possible global structures of the theory at this
limit should allow for transformations analogous to the $SL(2,\mathbb{Q})$ of
Maxwell theory. As we will show, such a description can indeed be obtained if
the large-$N$ limit is taken in a certain way.

Let us begin with recalling the case of finite $N$, considering for
concreteness the gauge group $SU(N)$. Here there is an electric
$\mathbb{Z}_{N}^{(1)}$ 1-form symmetry associated with the $\mathbb{Z}_{N}$
center of $SU(N)$, acting on Wilson lines according to the $N$-ality of their
representation. Gauging this 1-form symmetry or a subgroup of it, possibly
with stacking an SPT phase, changes the spectrum of line operators in the
theory and correspondingly also the 1-form symmetry. The way these different
global forms are encoded holographically is through the topological theory
[18] (also known as the SymTFT [19, 20, 21] of the theory)

$S=\frac{2\pi}{N}\int_{AdS_{5}}\mathsf{B}_{N}\cup\delta\mathsf{C}_{N}$ (6)

obtained near the boundary of $AdS_{5}$, where $\cup$ is the cup product and
$\mathsf{B}_{N}$ and $\mathsf{C}_{N}$ are both $\mathbb{Z}_{N}$ 2-cochains.
The different global forms then correspond to different topological boundary
conditions for the theory (6), which in turn are classified by the Lagrangian
subgroups of its surface operators666We here ignore the issue of topological
refinement [22].

$S_{\left(e,m\right)}\left(\sigma\right)=e^{\frac{2\pi
ie}{N}\int_{\sigma}\mathsf{B}_{N}}e^{\frac{2\pi
im}{N}\int_{\sigma}\mathsf{C}_{N}}\,.$ (7)

Surfaces that can end on the boundary (that is, the ones in the chosen
Lagrangian subgroup) then correspond to the line operators of the boundary
theory which are charged under the 1-form symmetry, while the rest of the
surfaces give rise to the symmetry generators when pushed to the boundary.

Let us now examine the large-$N$ limit of this story. Instead of considering a
specific value of $N$ that is finite but large (in which case the discussion
remains the same as above), we will take the limit in a way that formalizes
more precisely the intuition that combinations such as
$(e\int_{\sigma}\mathsf{B}_{N}\;\textrm{mod}\>N)/N$ (which appears in the
expression for the surface $S_{(e,m)}$, see (7)) are approximately valued in
all of $\mathbb{Q}/\mathbb{Z}$ as $N$ is taken to be very large. In order to
do it, let us focus on the $\mathbb{Z}_{N}$ center of $SU(N)$ and begin by
discussing two different natural ways of taking such a large-$N$ limit of it
(limits of these types have been recently discussed in a physical context in
[16]).

The first, usually referred to as the direct limit, is defined as follows.
Consider the family of groups777We consider in this limit the groups
$\frac{1}{N}\mathbb{Z}_{N}$ instead of $\mathbb{Z}_{N}$ in order to make the
construction more transparent. Working with $\mathbb{Z}_{N}$ will yield an
isomorphic result. $\\{\frac{1}{N}\mathbb{Z}_{N}\\}_{N\in\mathbb{N}}$ together
with the inclusion homomorphisms
$f_{MN}:\frac{1}{M}\mathbb{Z}_{M}\rightarrow\frac{1}{N}\mathbb{Z}_{N}$ between
them when $M|N$. The direct limit, denoted by
$\underset{\longrightarrow}{\lim}$, is then given in this case by the disjoint
union of all the groups but with an equivalence relation that identifies any
element $q\in\frac{1}{M}\mathbb{Z}_{M}$ for some $M$ with all its images under
the maps $f_{MN}$. Explicitly, we have

$\underset{\longrightarrow}{\lim}\,\frac{1}{N}\mathbb{Z}_{N}=\underset{N\in\mathbb{N}}{\bigsqcup}\,\frac{1}{N}\mathbb{Z}_{N}\not\;\;\;\sim$
(8)

where $\sim$ identifies equal rational numbers that appear in different
groups. It is easy to see that the result of this limit is simply
$\mathbb{Q}/\mathbb{Z}$, corresponding to the fact that it can be written as
the union of all cyclic groups. Let us comment that the direct limit is more
general than the specific case considered here, and is applicable to
categories that include objects which are more general than the groups we
examined and with morphisms that are different from the above homomorphisms.
We here followed the general construction and applied it to the large-$N$
limit under consideration.

The second way in which we can take the large-$N$ limit of $\mathbb{Z}_{N}$ is
called the inverse limit, and is the Pontryagin dual of the direct limit we
have previously discussed. We consider the family of groups
$\\{\mathbb{Z}_{N}\\}_{N\in\mathbb{N}}$ with homomorphisms
$g_{MN}:\mathbb{Z}_{N}\rightarrow\mathbb{Z}_{M}$ for $M|N$ given by the
$\textrm{mod}\>M$ map (note that now the map is from $\mathbb{Z}_{N}$ to
$\mathbb{Z}_{M}$ and not vice versa). The inverse limit, denoted by
$\underset{\longleftarrow}{\lim}$, is then given by

$\underset{\longleftarrow}{\lim}\,\mathbb{Z}_{N}=\left\\{\vec{a}\in\underset{N\in\mathbb{N}}{\prod}\,\mathbb{Z}_{N}\mid
a_{M}=a_{N}\;\textrm{mod}\>M\;\;\forall\;\;M|N\right\\}$ (9)

and simply means that any element in the resulting group is specified by the
set of its residues modulo $N$ for all $N$, in a way that is consistent with
the mod map relating different residues. This group is called the ”profinite
integers” and is denoted by $\mathbb{\widehat{Z}}$. It is a certain completion
of the integers $\mathbb{Z}$ and contains them as a subgroup. In particular,
there are profinite integers which are not ordinary integers. As for the
direct limit, the inverse limit can be defined for more general categories and
here we have only considered it in the context of the large-$N$ limit we are
taking.

The natural question at this point is what type of limit should be used for
the large-$N$ limit of the $\mathbb{Z}_{N}$ center of the $SU(N)$ gauge group.
Using the direct limit would mean that the center turns into
$\mathbb{Q}/\mathbb{Z}$ at large $N$, implying that the field $\mathsf{B}_{N}$
in Eq. (6) (which can be regarded as a background field for the electric
1-form symmetry) would turn into a $\mathbb{Q}/\mathbb{Z}$ 2-cochain
$\mathsf{B}_{\mathbb{Q}/\mathbb{Z}}$. The field $\mathsf{C}_{N}$, on the other
hand, which corresponds to the dual symmetry would turn into a
$\mathbb{\widehat{Z}}$ 2-cochain $\mathsf{C}_{\mathbb{\widehat{Z}}}$ (since
$\mathbb{Q}/\mathbb{Z}$ and $\mathbb{\widehat{Z}}$ are Pontryagin duals).
This, however, would mean that the $SL(2,\mathbb{Z}_{N})$ symmetry of the
action in (6) that we have at finite $N$ will not be present at the large-$N$
limit. In addition, there is no natural reason for not treating the fields
$\mathsf{B}_{N}$ and $\mathsf{C}_{N}$ on equal footing, taking a direct limit
for one and the inverse for the other (alternatively, there is no natural way
of choosing which of the global forms, $SU(N)$ or $PSU(N)$, have a
$\mathbb{Q}/\mathbb{Z}$ symmetry and which a $\mathbb{\widehat{Z}}$ one). In
fact, based on tracing the source of different contributions to the path
integral at the large-$N$ limit from those of finite but large $N$, one can
show that in a sense both types of limits should be used for both fields.
Leaving an analysis of the general case for future work, we will here focus on
gauge groups of the form $SU(N^{2})$ and take the large-$N$ limit in a way
that keeps $\mathsf{B}_{N^{2}}$ and $\mathsf{C}_{N^{2}}$ on equal footing and
such that the $SL(2,\mathbb{Z}_{N^{2}})$ symmetry is not broken.

In order to take such a limit of the $\mathbb{Z}_{N^{2}}$ center, we view it
as the following extension888I would like to thank P. Putrov for suggesting
this way of taking the limit to me.

$0\rightarrow\mathbb{Z}_{N}\rightarrow\mathbb{Z}_{N^{2}}\rightarrow\mathbb{Z}_{N}\rightarrow
0$ (10)

where the second arrow denotes multiplication by $N$. Defining the bilinear
pairing $(a,b)_{N^{2}}=ab/N^{2}\;\textrm{mod}\>1$ in $\mathbb{Z}_{N^{2}}$
(which is a map
$(,)_{N^{2}}:\mathbb{Z}_{N^{2}}\times\mathbb{Z}_{N^{2}}\rightarrow\mathbb{Q}/\mathbb{Z}$),
we observe that the first $\mathbb{Z}_{N}$ group in the sequence (10) is a
Lagrangian subgroup with respect to it, and that the two $\mathbb{Z}_{N}$
groups can be regarded as Pontryagin dual to each other using it since an
element $a\in\mathbb{Z}_{N}$ can be understood as
$a\in\textrm{Hom}(\mathbb{Z}_{N^{2}}/\mathbb{Z}_{N},\mathbb{Q}/\mathbb{Z})$
with $a(b)=(a,b)_{N^{2}}$ for $b\in\mathbb{Z}_{N^{2}}/\mathbb{Z}_{N}$ (which
is well-defined since $\mathbb{Z}_{N}$ is Lagrangian). We can then take the
large-$N$ limit of the sequence (10) in such a way that the inverse limit is
taken for the first $\mathbb{Z}_{N}$ group while the direct one is taken for
the other $\mathbb{Z}_{N}$ (i.e. for $\mathbb{Z}_{N^{2}}/\mathbb{Z}_{N}$).
This, in turn, is possible due to the fact that the two $\mathbb{Z}_{N}$
groups are Pontryagin dual to each other and by using the property that
Pontryagin duality exchanges direct and inverse limits,
$\textrm{Hom}(\underset{\longrightarrow}{\lim}\,A_{n},B)=\underset{\longleftarrow}{\lim}\,\textrm{Hom}(A_{n},B)$.
We then end up at this large-$N$ limit with the sequence

$0\rightarrow\mathbb{\widehat{Z}}\rightarrow\mathbb{\widehat{Q}}\rightarrow\mathbb{Q}/\mathbb{Z}\rightarrow
0\,,$ (11)

where the extension $\mathbb{Z}_{N^{2}}$ turns at the limit to the extension
$\mathbb{\widehat{Q}}$ of $\mathbb{Q}/\mathbb{Z}$ by $\mathbb{\widehat{Z}}$.
The group $\mathbb{\widehat{Q}}$ is called the group of profinite rationals,
and is defined in an analogous way to the profinite integers in (9),

$\mathbb{\widehat{Q}}=\left\\{\vec{q}\in\underset{N\in\mathbb{N}}{\prod}\,\left(\mathbb{Q}/N\mathbb{Z}\right)\mid
q_{M}=q_{N}\;\textrm{mod}\>M\;\;\forall\;\;M|N\right\\}.$ (12)

It is self-dual under Pontryagin duality and has a natural ring structure
extending that of $\mathbb{\widehat{Z}}$. Moreover, it includes $\mathbb{Q}$
and $\mathbb{\widehat{Z}}$ as subrings and can be written as
$(\mathbb{\widehat{Z}}\oplus\mathbb{Q})/\mathbb{Z}$.

We therefore obtain that both $\mathsf{B}_{N^{2}}$ and $\mathsf{C}_{N^{2}}$
turn at this limit into $\mathbb{\widehat{Q}}$ 2-cochains, and that the
symmetry between them is preserved. In order to identify this symmetry in full
and find what $SL(2,\mathbb{Z}_{N^{2}})$ turns into at this limit, we should
first find the limit of the action (6). We can do it by rewriting this
finite-$N$ action using the pairing $(,)_{N^{2}}$ of $\mathbb{Z}_{N^{2}}$ we
defined above as

$S=2\pi\int_{AdS_{5}}\left(\mathsf{B}_{N^{2}},\delta\mathsf{C}_{N^{2}}\right)_{N^{2}}$
(13)

and observing that this pairing turns at the limit we are taking into the
pairing
$(q_{1},q_{2})_{\infty}=q_{1}q_{2}\;\textrm{mod}\>\mathbb{\widehat{Z}}\in\mathbb{Q}/\mathbb{Z}$
in $\mathbb{\widehat{Q}}$, where $q_{1},q_{2}\in\mathbb{\widehat{Q}}$ and
$q_{1}q_{2}$ is their product using the standard ring structure of
$\mathbb{\widehat{Q}}$. We therefore find the large-$N$ action

$S_{\infty}=2\pi\int_{AdS_{5}}(\mathsf{B}_{\mathbb{\widehat{Q}}},\delta\mathsf{C}_{\mathbb{\widehat{Q}}})_{\infty}$
(14)

and can identify an $SL(2,\mathbb{\widehat{Q}})$ symmetry acting on
$\mathsf{B}_{\mathbb{\widehat{Q}}}$ and $\mathsf{C}_{\mathbb{\widehat{Q}}}$ as
a doublet using the standard ring structure of $\mathbb{\widehat{Q}}$. Let us
also comment that the surfaces of the theory, which for finite $N$ are
specified by the pair of charges
$(e,m)\in\mathbb{Z}_{N^{2}}\times\mathbb{Z}_{N^{2}}$ (see (7) for the case of
$SU(N)$ gauge group), are now specified by a pair
$(\mathfrak{e},\mathfrak{m})\in\mathbb{\widehat{Q}}\times\mathbb{\widehat{Q}}$
and take the form

$S_{\left(\mathfrak{e},\mathfrak{m}\right)}\left(\sigma\right)=e^{2\pi
i\int_{\sigma}(\mathfrak{e},\mathsf{B}_{\mathbb{\widehat{Q}}})_{\infty}}e^{2\pi
i\int_{\sigma}(\mathfrak{m},\mathsf{C}_{\mathbb{\widehat{Q}}})_{\infty}}\,.$
(15)

We have found that the group $SL(2,\mathbb{Z}_{N^{2}})$, corresponding to
gauging the 1-form symmetry and stacking an SPT phase at finite $N$, turns
into $SL(2,\mathbb{\widehat{Q}})$ at the large-$N$ limit we are considering.
Notice, however, that the duality group of $\mathcal{N}=4$ SYM, which acts
also on the rest of the theory (e.g. on the coupling $\tau$), is still
$SL(2,\mathbb{Z})$ for general values of the ’t Hooft coupling $\lambda$. At
this point we are making use of the supergravity approximation of the string-
theory dual of $\mathcal{N}=4$ SYM which is valid when $\lambda$ is taken to
be large, and of its $SL(2,\mathbb{R})$ enhanced symmetry when the fields are
treated as classical, to identify an $SL(2,\mathbb{Q})\subset
SL(2,\mathbb{R})$ duality group which is consistent at the quantum level with
the global structures (or charge quantization) we have found at the large-$N$
limit we described. Therefore, taking the large-$N$ limit as detailed above
and the ’t Hooft coupling to be large, we expect to have an $SL(2,\mathbb{Q})$
duality group999Note that we only consider those elements of
$SL(2,\mathbb{Q})$ which keep $\lambda$ in the supergravity regime. with
elements that leave values of $\lambda$ corresponding to a $\tau$ of the form
(5) invariant. Performing such a duality operation in half-space and
accompanying it with the corresponding element of $SL(2,\mathbb{Q})\subset
SL(2,\mathbb{\widehat{Q}})$ that brings the global structure to its original
form, we obtain a non-invertible topological defect in the original
theory.101010This is similar to the construction of the duality and triality
defects. For example, the duality defect at $\tau=i$ is obtained by performing
$S$-duality in half-space which is then accompanied by gauging the 1-form
symmetry (or performing an $S$ operation in the modular group of gauging and
stacking SPT phases) in the same half-space. Different such defects, which are
realized at different values of $\lambda$, implement different elements
(corresponding to certain rotation angles) of the $U(1)$ bonus symmetry of
$\mathcal{N}=4$ SYM, in analogy to the case of the $\mathcal{N}=4$ $U(1)$
gauge theory discussed in the previous section. Of course, since such defects
correspond to approximate symmetries at the supergravity regime which become
closer to be topological as $\lambda$ is taken to be larger, we can also
consider at each $\lambda$ $SL(2,\mathbb{Q})$ duality elements that only
approximately fix it (to the same accuracy).111111Such elements correspond to
topological interfaces between two theories with couplings that are almost but
not exactly equal. In order to get a defect in the original theory, such an
interface has to be accompanied with another interface which changes the
coupling back to its original value. These interfaces are only slightly-
nontopological and have a very small displacement operator, and therefore
result (when combined with the other topological interface) in defects that
are similarly almost topological and correspond to approximate symmetries.

### Acknowledgements

I would like to thank O. Bergman, T. Dumitrescu, M. Gutperle, P.-S. Hsin, K.
Intriligator, T. Jacobson, P. Kraus, P. Niro, K. Roumpedakis, G. Zafrir and Y.
Zheng for useful discussions, and especially to P. Putrov for illuminating
discussions about the large-$N$ limit discussed in this paper. I would also
like to thank P.-S. Hsin, P. Niro, P. Putrov and K. Roumpedakis for comments
on the manuscript. This work was supported by the Mani L. Bhaumik Institute
for Theoretical Physics at UCLA.

## References

  * [1] S. Schafer-Nameki, _ICTP Lectures on (Non-)Invertible Generalized Symmetries_ , 2305.18296.
  * [2] S.-H. Shao, _What’s Done Cannot Be Undone: TASI Lectures on Non-Invertible Symmetry_ , 2308.00747.
  * [3] Y. Choi, C. Cordova, P.-S. Hsin, H. T. Lam and S.-H. Shao, _Noninvertible duality defects in 3+1 dimensions_ , _Phys. Rev. D_ 105 (2022) 125016 [2111.01139].
  * [4] Y. Choi, C. Cordova, P.-S. Hsin, H. T. Lam and S.-H. Shao, _Non-invertible Condensation, Duality, and Triality Defects in 3+1 Dimensions_ , _Commun. Math. Phys._ 402 (2023) 489 [2204.09025].
  * [5] P. Niro, K. Roumpedakis and O. Sela, _Exploring non-invertible symmetries in free theories_ , _JHEP_ 03 (2023) 005 [2209.11166].
  * [6] C. Cordova and K. Ohmori, _Quantum Duality in Electromagnetism and the Fine-Structure Constant_ , 2307.12927.
  * [7] K. Roumpedakis, S. Seifnashri and S.-H. Shao, _Higher Gauging and Non-invertible Condensation Defects_ , _Commun. Math. Phys._ 401 (2023) 3043 [2204.02407].
  * [8] D. Gaiotto and T. Johnson-Freyd, _Condensations in higher categories_ , 1905.09566.
  * [9] J. Kaidi, G. Zafrir and Y. Zheng, _Non-invertible symmetries of $\mathcal{N}$ = 4 SYM and twisted compactification_, _JHEP_ 08 (2022) 053 [2205.01104].
  * [10] K. A. Intriligator, _Bonus symmetries of N=4 superYang-Mills correlation functions via AdS duality_ , _Nucl. Phys. B_ 551 (1999) 575 [hep-th/9811047].
  * [11] K. A. Intriligator and W. Skiba, _Bonus symmetry and the operator product expansion of N=4 SuperYang-Mills_ , _Nucl. Phys. B_ 559 (1999) 165 [hep-th/9905020].
  * [12] J. H. Schwarz and P. West, _Symmetries and transformations of chiral n = 2, d = 10 supergravity_ , _Physics Letters B_ 126 (1983) 301.
  * [13] P. Howe and P. West, _The complete n =2, d = 10 supergravity_ , _Nuclear Physics B_ 238 (1984) 181.
  * [14] J. H. Schwarz, _Covariant field equations of chiral n = 2 d = 10 supergravity_ , _Nuclear Physics B_ 226 (1983) 269.
  * [15] M. B. Green and M. Gutperle, _Effects of D instantons_ , _Nucl. Phys. B_ 498 (1997) 195 [hep-th/9701093].
  * [16] P. Putrov, _$\mathbb{Q}/\mathbb{Z}$ symmetry_, 2208.12071.
  * [17] Y. Choi, H. T. Lam and S.-H. Shao, _Noninvertible Global Symmetries in the Standard Model_ , _Phys. Rev. Lett._ 129 (2022) 161601 [2205.05086].
  * [18] E. Witten, _AdS / CFT correspondence and topological field theory_ , _JHEP_ 12 (1998) 012 [hep-th/9812012].
  * [19] D. Gaiotto and J. Kulp, _Orbifold groupoids_ , _JHEP_ 02 (2021) 132 [2008.05960].
  * [20] F. Apruzzi, F. Bonetti, I. n. García Etxebarria, S. S. Hosseini and S. Schafer-Nameki, _Symmetry TFTs from String Theory_ , _Commun. Math. Phys._ 402 (2023) 895 [2112.02092].
  * [21] D. S. Freed, G. W. Moore and C. Teleman, _Topological symmetry in quantum field theory_ , 2209.07471.
  * [22] X. Chen, A. Dua, P.-S. Hsin, C.-M. Jian, W. Shirley and C. Xu, _Loops in 4+1d topological phases_ , _SciPost Phys._ 15 (2023) 001 [2112.02137].


# Self-Supervised Learning of Face Representations for
Video Face Clustering

Vivek Sharma1, Makarand Tapaswi2, M. Saquib Sarfraz1 and Rainer Stiefelhagen1
1 CV:HCI, Karlsruhe Institute of Technology, Karlsruhe, Germany 2 University
of Toronto, Toronto, Canada

###### Abstract

Analyzing the story behind TV series and movies often requires understanding
who the characters are and what they are doing. With improving deep face
models, this may seem like a solved problem. However, as face detectors get
better, clustering/identification needs to be revisited to address increasing
diversity in facial appearance. In this paper, we address video face
clustering using unsupervised methods. Our emphasis is on distilling the
essential information, _identity_ , from the representations obtained using
deep pre-trained face networks. We propose a self-supervised Siamese network
that can be trained without the need for video/track based supervision, and
thus can also be applied to image collections. We evaluate our proposed method
on three video face clustering datasets. The experiments show that our methods
outperform current state-of-the-art methods on all datasets. Video face
clustering is lacking a common benchmark as current works are often evaluated
with different metrics and/or different sets of face tracks. The datasets and
code are available at https://github.com/vivoutlaw/SSIAM.

##  I Introduction

Large videos such as TV series episodes or movies undergo several
preprocessing steps to make the video more accessible, _e.g_. shot detection.
Character identification/clustering has become one such important step with
several emerging research areas [25, 35, 37] requiring it. For example, in
video question-answering [35], most questions center around the characters
asking who they are, what they do, and even why they act in certain ways. The
related task of video captioning [25] often uses a character agnostic way
(replacing names by _someone_) making the captions very artificial and
uninformative (_e.g_.  _someone_ opens the door). However, recent work [24]
suggests that more meaningful captions can be achieved from an improved
understanding of characters. In general, the ability to predict which
characters appear when and where facilitates a deeper video understanding that
is grounded in the storyline.

Motivated by this goal, person clustering [4, 10, 15, 48] and identification
[7, 31, 2, 23, 19] in videos has seen over a decade of research. In
particular, fully automatic person identification is achieved either by
aligning subtitles and transcripts [7, 31, 2], or using web images for actors
and characters [1, 19] as a form of weak supervision. On the other hand,
clustering [48, 4, 41, 42, 47, 5] has mainly relied on _must-link_ and
_cannot-link_ information obtained by tracking faces in a shot and analyzing
their co-occurrence.

As face detectors improve (_e.g_. [13]), clustering and identification need to
be revisited as more faces that exhibit extreme viewpoints, illumination,
resolution, become available and need to be grouped or identified. Along with
improvements to face detection, deep Convolutional Neural Networks (CNNs) have
also yielded large performance gains for face representations [32, 28, 21, 3].
These networks are typically trained using hundreds-of-thousands to millions
of face images gathered from the web, and show super-human performance on face
verification tasks on images (LFW [14]) and videos (YouTubeFaces [40]).
Nevertheless, it is important to note that faces in videos such as TV
series/movies exhibit more variety in comparison to _e.g_. LFW, where the
images are obtained from Yahoo News by cropping mostly frontal faces. While
these deep models generalize well, they are difficult to train from scratch
(require lots of training data), and are typically transferred to other
datasets via _net surgery_ : fine-tuning [29, 47, 48], or use of additional
embeddings on the features from the last layer [27, 6], or both.

Recent work shows that CNN representations can be improved via positive and
negative pairs that are discovered through an MRF [48]; or a revised triplet-
loss [47]. In contrast, we propose simple methods that do not require complex
optimization functions or supervision to improve the feature representation.
We emphasize that while video-level constraints aren’t new, they need to be
used properly to extract the most out of them. This is especially in light of
CNN face representations that are very similar even across different
identities. Fig. 5 proves this point as we see a large overlap between the
cosine similarity distributions of positive (same id) and negative (across id)
track pairs on the base features.

In this paper, we focus on the video face clustering problem. Given a set of
face tracks from several characters, our goal is to group them so that tracks
in a cluster belong to the same character. We summarize the main contributions
of our paper, and also highlight key differences: (1) We propose two variants
of discriminative methods (Sec. III) that build upon deep network
representations, to learn discriminative facial representations. They are
Track-supervised Siamese network (TSiam) and Self-supervised Siamese network
(SSiam). In contrast to previous methods [5], in TSiam, we incorporate
negative pairs for the singleton (no co-occurring) tracks. In SSiam, we obtain
positives and negatives by sorting distances (_i.e_. ranking) on a subset of
frames. Note that methods proposed in this paper are either fully
unsupervised, or use supervision that is obtained automatically, hence can be
thought as unsupervised. Additionally, our fully unsupervised method can mine
positive and negative pairs without the need for tracking. This enables
application of our method to image collections. (2) We perform extensive
empirical studies and demonstrate the effectiveness and generalisation of the
methods. Our methods are powerful and obtain performance comparable or higher
than state-of-the-art when evaluated on three challenging video face
clustering datasets.

##  II Related Work

Over the last decade, video face clustering is typically modeled using
discriminative methods to improve face representations. In the following, we
review some related work in this area.

Video face clustering. Clustering faces in videos commonly uses pairwise
constraints obtained by analyzing tracks and some form of
representation/metric learning. Face image pairs belonging to the same track
are labeled positive (same character), while face images from co-occurring
tracks help create negatives (different characters). This strategy has been
exploited by learning a metric to obtain cast-specific distances [4] (ULDML);
iteratively clustering and associating short sequences based on hidden Markov
Random Field (HMRF) [41, 42]; or performing clustering in a sub-space obtained
by a weighted block-sparse low-rank representation (WBSLRR) [43]. In addition
to pairwise constraints, video editing cues are used in an unsupervised way to
merge tracks [34]. Here, track and cluster representations are learned on-the-
fly with dense-SIFT Fisher vectors [20]. Recently, Jin  _et al_. [15] consider
detection and clustering jointly, and propose a link-based clustering (Erdös-
Rényi) based on rank-1 counts verification. The linking is done by comparing a
given frame with a reference frame and a threshold is learned to merge/not-
merge frames.

Face track clustering/identification methods have also used additional cues
such as clothing appearance [33], speech [22], voice models [19], context
[46], gender [49], name mentions (first, second, and third person references)
in subtitles [18], weak labels using transcripts/subtitles [7, 2], and joint
action and actor labeling [16] using transcripts.

With the popularity of CNNs, there is a growing focus on improving face
representations using video-level constraints. An improved form of triplet
loss is used to fine-tune the network and push the positive and negative
samples apart in addition to requiring anchor and positive to be close, and
anchor and negative far [47]. Zhang  _et al_. [48] learn better
representations by dynamic clustering constraints that are discovered
iteratively during clustering that is performed via a Markov Random Field
(MRF). In contrast to related work, we propose a simple, yet effective
approach (SSiam) to learn good representations by sorting distances on a
subset of frames and not requiring video/track level constraints to generate
positive/negative training pairs.

Further, Zhang  _et al_. [47]-[48] and Datta  _et al_. [5] utilize video-level
constraints only to generate a set of similar and dissimilar face pairs. Thus,
the model does not see negative pairs for singleton (no co-occurring) tracks.
In contrast, our method TSiam incorporates negative pairs for the singleton
tracks by exploiting track-level distance.

Finally, it is worth noting the work on “harvesting” training data from
unlabeled sources which is in the similar spirit of SSiam and TSiam. Fernando
_et al_. [8] and Mishra  _et al_. [17] shuffle the video frames and treat them
as positive/negative training data for reordering video frames; Wang  _et al_.
[38] collect positive/negative training data by tracking bounding boxes
(_i.e_. motion information) in order to learn effective visual
representations.

##  III Refining Face Representations for Clustering

Our goal is to improve face representations using simple methods that build
upon the success of deep CNNs. More precisely, we propose models to refine the
face descriptors automatically, without the need for manual labels. In
contrast, fine-tuning the original CNN typically requires supervised class
labels. Thus, our approach has three key benefits: (i) it is easily applicable
to new videos; (ii) it does not need large amounts of training data (few
hundred tracks are enough); and (iii) specialized networks can be trained on
each episode, film, or series.

Figure 1: Track-supervised Siamese network (TSiam). Illustration of the
Siamese architecture used in our track-supervised Siamese networks. Note that
the MLP is shared across both feature maps. $B$ corresponds to batch size.
Figure 2: Self-supervised Siamese network (SSiam). Illustration of the Siamese
architecture used in our self-supervised Siamese networks. SSiam selects hard
pairs: farthest positives and closest negatives using a ranked list based on
Euclidean distance for learning similarity and dissimilarity respectively.
Note that the MLP is the same across both feature maps. $B$ corresponds to
batch.

Preliminaries. Consider a video with $N$ face tracks
$\\{T^{1},\ldots,T^{N}\\}$ belonging to $C$ characters. Each track corresponds
to one of the characters, and consists of $T^{i}=\\{f_{1},\ldots,f_{M^{i}}\\}$
face images. Our goal is to group tracks into $\\{G_{1},\ldots,G_{|C|}\\}$
such that each track is assigned to only one group, and ideally, each group
contains all tracks from the same character. We use a deep CNN (VGG2 [3]) and
extract a descriptor for each face image
$\mathbf{x}^{i}_{k}\in\mathbb{R}^{D},\,k=1,\ldots,M^{i}$ from the penultimate
layer (before classification) of the network. We refer to these as base
features, and demonstrate that they already achieve a high performance. As a
form of data augmentation, we use 10 crops obtained from an expanded bounding
box surrounding the face image during training. Evaluation is based on one
center crop.

Track-level representations are obtained by aggregating the face image
descriptors

$\mathbf{t}^{i}=\frac{1}{M^{i}}\sum_{k}\mathbf{x}^{i}_{k}\,.$ (1)

We additionally normalize track representations to be unit-norm,
$\hat{\mathbf{t}}^{i}=\mathbf{t}^{i}/\|\mathbf{t}^{i}\|_{2}$ before using them
for clustering.

In [34, 47, 48], the authors use Hierarchical Agglomerative Clustering (HAC)
as the clustering technique. For a fair comparison to [34, 47, 48], we perform
HAC to obtain a fixed number of clusters, equal to the number of characters.
We use the minimum variance ward linkage [39] for all methods. See Fig. 3 for
illustration.

Discriminative models. Discriminative clustering models typically associate a
label $y$ with a pair of features. We choose $y=0$ when a pair of features
$(\mathbf{x}_{1},\mathbf{x}_{2})$ should belong to the same cluster, and $y=1$
otherwise [12].

We use a shallow MLP to reduce the dimensionality and improve generalization
of the features (see Fig. 1, 2). Here, each face image is encoded as
$Q_{\phi}(\mathbf{x}^{i}_{k})$, where $\phi$ corresponds to the trainable
parameters. We find $Q_{\phi}(\cdot)$ to perform best when using a linear
layer (for details see Sec. IV-B). To perform clustering, we compute track-
level aggregated features by average pooling across the embedded frame-level
representations [30]

$\mathbf{t}^{i}=\frac{1}{M^{i}}\sum_{k}Q_{\phi}(\mathbf{x}^{i}_{k})\,,$ (2)

followed by $\ell_{2}$-normalization.

We train our model parameters at the frame-level by minimizing the contrastive
loss [12]:

$\begin{split}\mathcal{L}\left(W,y,Q_{\phi}(\mathbf{x}_{1}),Q_{\phi}(\mathbf{x}_{2})\right)=\qquad\qquad\qquad\qquad\qquad\\\
\frac{1}{2}\left((1-y)\cdot(d_{W})^{2}+y\cdot(\max(0,m-d_{W}))^{2}\right)\,,\end{split}$
(3)

where $\mathbf{x}_{1}$ and $\mathbf{x}_{2}$ are a pair of face representations
with $y=0$ when coming from the same character, and $y=1$ otherwise.
$W:\mathbb{R}^{D\times d}$ is a linear layer that embeds
$Q_{\phi}(\mathbf{x})$ such that $d\ll D$ (in our case, $d=2$). $d_{W}$ is the
Euclidean distance $d_{W}=\|W\cdot Q_{\phi}(\mathbf{x}_{1})-W\cdot
Q_{\phi}(\mathbf{x}_{2})\|^{2}$, and $m$ is the margin, empirically chosen to
be 1.

In the following, we present two strategies to automatically obtain
supervision for pairs of frames: Fig. 1 illustrates the Track-level
supervision, and Fig. 2 shows the Self-supervision for Siamese network
training.

Track-supervised Siamese network (TSiam). Video face clustering often employs
face tracking to group face detections made in each frame. The tracking acts
as a form of high precision clustering (grouping detections within a shot) and
is popularly used to automatically generate positive and negative pairs of
face images [4, 5, 34, 42]. In each frame, we assume that characters appear on
screen only once. Thus, all face images within a track can be used as positive
pairs, while face images from co-occurring tracks are used as negative pairs.
For each frame in the track, we sample two frames within the same track to
form positive pairs, and sample four frames from a co-occurring track (if it
exists) to form negative pairs.

Depending on the filming style of the series/movie, characters may appear
alone or together on screen. As we perform experiments on diverse datasets,
for some videos 35% tracks have co-occurring tracks, while this can be as
large as 70% for other videos. For isolated tracks, we sort all other tracks
based on track-level distances (computed on base features) and randomly sample
frames from the farthest $F=25$ tracks. Note that all previous works ignore
negative pairs for singleton (not co-occurring) tracks. We will highlight
their impact in our experiments.

Self-supervised Siamese network (SSiam). Clustering is an unsupervised task
and supervision from tracks may not always be available. An example is face
clustering within image collections (_e.g_. on social media platforms). To
enable the use of metric learning without any supervision we propose an
effective approach that can generate the required pairs automatically during
training. SSiam is inspired by pseudo-relevance feedback (pseudo-RF) [45, 44]
that is commonly used in information retrieval.

We hypothesize that the first and last samples of a ranked list based on
Euclidean distance are strong candidates for learning similarity and
dissimilarity respectively. We exploit this in a meaningful way and generate
promising similar and dissimilar pairs from a representative subset of the
data.

Formally, consider a subset
$\mathcal{S}=\\{\mathbf{x}_{1},\ldots,\mathbf{x}_{B}\\}$ of frames from the
dataset. We treat each frame $\mathbf{x}_{b},b=1,\ldots,B$ as a query and
compute Euclidean distance against every other frame in the set. We sort rows
of the resulting matrix in an ascending order (smallest to largest distance)
to obtain an ordered index matrix
$\mathcal{O}(\mathcal{S})=[s_{1}^{o};\ldots;s_{B}^{o}]$. Each row $s_{b}^{o}$
contains an ordered index of the closest to farthest faces corresponding to
$\mathbf{x}_{b}$. Note that the first column of such a matrix is the index
itself at distance 0. The second column corresponds to nearest neighbors for
each frame and can be used to form the set of positive pairs
$\mathcal{S}_{+}$. Similarly, the last column corresponds to farthest
neighbors and forms the set of negative pairs $\mathcal{S}_{-}$. Each element
of the above sets stores: query index $b$, nearest/farthest neighbor $r$, and
the Euc. distance $d$. However, we still need to consider the choice of: (i)
the set of frames, and (ii) training pairs from the candidate sets of positive
and negative pairs.

We address this during training and form the pairs dynamically by picking a
random subset of $B$ frames at each iteration. We compute the distances, sort
them, and obtain positive and negative pairs sets
$\mathcal{S}_{+},\mathcal{S}_{-}$, each with $B$ elements. Among them, we
choose $K$ pairs from the positive set that have the largest distances and $K$
pairs from the negative set with the smallest distances. This allows us to
select pairs semi-hard positives and semi-hard negatives from the
representative set of $B$ frames. Finally, these $2K$ pairs form the training
batch for the network.

To encourage variety in the sample set $\mathcal{S}$ and reduce the chance of
false positives/negatives in the chosen $2K$ pairs, $B$ is chosen to be much
larger than $K$ ($B=1000,K=64$). Experiments on difficult datasets and
generalization studies show the benefit and effectiveness of this approach in
collecting positive and negative pairs to train the network.

Note that, SSiam can be thought of as an improved version of pseudo-RF with
batch processing. Rather than selecting farthest negatives and closest
positives for each independent query, we emphasize that SSiam selects 2$K$
hard pairs: farthest positives and closest negatives by looking at the batch
of queries $B$ jointly. This selection of sorted pairs from the positive
$\mathcal{S}_{+}$ and negative $\mathcal{S}_{-}$ sets is quite important. See
experiments for a more detailed comparison of SSiam with pseudo-RF.

##  IV Evaluation

Figure 3: Illustration of the testing evaluation. Given our pre-trained MLP
models: TSiam and SSiam. We extract the frame-level features for the track,
followed by mean pool for a track-level representation, which is then fed to
HAC clustering with fixed number of clusters. Finally, we evaluate the quality
of clustering via Accuracy or BCubed F-measure.

We present our evaluation on three challenging datasets. We first describe the
clustering metric, followed by a thorough analysis of the proposed methods,
ending with a comparison to state-of-the-art.

###  IV-A Experimental Setup

Datasets. We conduct experiments on three challenging video face
identification/clustering datasets: (i) Buffy the Vampire Slayer (BF) [2, 48]
(season 5, episodes 1 to 6): a drama series with several shots in the dark at
night; (ii) Big Bang Theory (BBT) [33, 2, 41, 47] (season 1, episodes 1 to 6):
a sitcom with small cast list shot mainly indoors, and (iii) ACCIO [9]:
Accio-1 first installment of “Harry Potter” movie series with a large number
of dark scenes and several tracks with non-frontal faces.

TABLE I: Dataset statistics for BBT [41, 42, 47], BF [48] and ACCIO [48]. |  | This work |  | Previous work
---|---|---|---|---
Datasets | #Cast | #TR (#FR) | LC/SC (%) | #TR (#FR)
BBT0101 | 5 | 644 (41220) | 37.2 / 4.1 | 182 (11525)
BF0502 | 6 | 568 (39263) | 36.2 / 5.0 | 229 (17337)
ACCIO | 36 | 3243 (166885) | 30.93/0.05 | 3243 (166885)
Figure 4: Example images for a few characters from our dataset. We show one
easy view and one difficult view. The extreme variation in illumination, pose,
resolution, and attributes (spectacles) make the datasets challenging.

Most current works [4, 42, 47, 48] that focus on improving feature
representations for video-face clustering assume the number of main
characters/clusters is known. In this work, for a fair comparison we follow
the same protocols that are widely employed in the previous works [30, 47, 48]
and train on BF episode 2, BBT episode 1, and ACCIO. We also use the same
number of characters as previous methods [48, 47], however, it is important to
note that we do not discard tracks/faces that are small or have large pose
variation. When not mentioned otherwise, we use an updated version of face
tracks released by [2] that incorporate several detectors to encompass all pan
angles and in-plane rotations up to 45 degrees. Tracks are created via an
online tracking-by-detection scheme with a particle filter.

We present a summary of our dataset in Table I, and also indicate the number
of tracks (#TR) and frames (#FR) used in other works, showing that our data is
indeed more challenging. Additionally, it is important to note that different
characters have wide variations in number of tracks, indicated by the class
balance largest class (LC) to smallest class (SC). Fig. 4 shows a few examples
of difficult faces our methods have to deal with.

Evaluation metric. We use Clustering Accuracy (ACC) [48] also called Weighted
Clustering Purity (WCP) [34] as the metric to evaluate the quality of
clustering. As we compare methods providing equal numbers of clusters (number
of main cast), ACC is a fair metric for comparison. ACC is computed as

$\mbox{ACC}=\frac{1}{N}\sum_{c=1}^{|C|}n_{c}\cdot p_{c}\,,$ (4)

where $N$ is the total number of tracks in the video, $n_{c}$ is the number of
samples in the cluster $c$, and cluster purity $p_{c}$ is measured as the
fraction of the largest number of samples from the same label to $n_{c}$.
$|C|$ corresponds to the number of main cast members, and in our case also the
number of clusters. For ACCIO, we report BCubed Precision (P), Recall (R) and
F-measure (F) for a fair comparison with the state-of-the-art. Figure 3
illustrates the steps for the testing evaluation.

###  IV-B Implementation Details

CNN. We adopt the VGG-2 face CNN [3], a ResNet50 model, pre-trained on MS-
Celeb-1M [11] and fine-tuned on 3.31M face images of 9131 subjects (VGG2
data). Input RGB face images are resized to $224\times 224$, and pushed
through the CNN. We extract pool5_7x7_s1 features, resulting in
$\mathbf{x}^{i}_{k}\in\mathbb{R}^{2048}$.

Siamese network MLP. The network comprises of fully-connected layers
($\mathbb{R}^{2048}\rightarrow\mathbb{R}^{256}\rightarrow\mathbb{R}^{2}$).
Note that the second linear layer is part of the contrastive loss (corresponds
to $W$ in Eq. 3), and we use the feature representations at $\mathbb{R}^{256}$
for clustering.

To train our Siamese network with track-level supervision (TSiam), we obtain
about 102k positive and 204k negative frame pairs (for BBT-0101) by mining 2
positive and 4 negative pairs for each frame. For Self-supervised Siamese, we
generate pairs as described in Sec. III, using $B=1000,K=64$. Higher values
values of $B=2000,3000$, did not provide significant improvements.

The MLP is trained using the contrastive loss, and parameters are updated
using Stochastic Gradient Descent (SGD) with a fixed learning rate of
$10^{-3}$. Since the labels are obtained automatically, overfitting is not a
concern. We train our model until convergence (loss does not reduce
significantly).

###  IV-C Clustering Performance and Generalization

TABLE II: Clustering accuracy on the base face representations. Dataset | Track Level | Frame Level
---|---|---
VGG1 | VGG2 | VGG1 | VGG2
BBT-0101 | 0.916 | 0.932 | 0.938 | 0.940
BF-0502 | 0.831 | 0.836 | 0.901 | 0.912

Base features. We compare track- and frame-level performance of two popular
deep face representations: VGG1 [21] and VGG2 [3]. Track-level results use
mean-pool of frames. Number of clusters matches number of characters. Results
are reported in Table II. Note that the differences are typically within 1% of
each other indicating that the results in the following tables are not just
due to having better CNNs trained with more data. We refer to VGG2 features as
Base for the remainder of this paper.

Figure 5: Histograms of pairwise cosine similarity between tracks of same
identity (pos) and different identity (neg) for BBT-0101.

Role of effective mining of +/- pairs. We emphasize that especially in light
of CNN face representations, the features are very similar even across
different identities, and thus positive and negative pairs need to be created
properly to make the most out of them. Fig. 5 proves this point as we see a
large overlap between the cosine similarity distributions of positive (same
id) and negative (across id) track pairs on the base features.

TSiam, impact of singleton tracks. Previous work with video-level constraints
[47, 48] and [5], ignore singleton (not co-occurring) tracks. In TSiam, we
include negative pairs for singletons based on track distances. Table III
shows that 30-70% tracks are singleton and ignoring them lowers accuracy by
4%. This confirms our hypothesis that incorporating negative pairs of
singletons helps improve performance.

TABLE III: Ignoring singleton tracks (and possibly characters) leads to significant performance drop. Accuracy on track-level clustering. | TSiam | # Tracks
---|---|---
Dataset |  w/o Single [5] | Ours | Total | Single | Co-oc
BBT-0101 | 0.936 | 0.964 | 644 | 219 | 425
BF-0502 | 0.849 | 0.893 | 568 | 345 | 223

SSiam, comparison to pseudo-RF [45, 44]. In Pseudo-RF, all samples are treated
independent of each other, there is no batch of data from which $2K$ pairs are
chosen. A pair of samples closest in distance are chosen as positive, and
farthest as negative. However, this usually corresponds to samples that
already satisfy the loss margin, thus leading to small (possibly even 0)
gradient updates. Table IV shows that SSiam that involves sorting a batch of
queries is much more efficient over pseudo-RF as it has the potential to
select harder positives and negatives. We see a consistent gain in
performance, 3% for BBT-0101 and over 9% for BF-0502.

TABLE IV: Comparison of _SSiam_ with _pseudo-RF_. Method | BBT-0101 | BF-0502
---|---|---
Pseudo-RF | 0.930 | 0.814
SSiam | 0.962 | 0.909
TABLE V: Clustering accuracy computed at track-level on the training episodes, with a comparison to all evaluated models. Train/Test | Base | TSiam | SSiam
---|---|---|---
BBT-0101 | 0.932 | 0.964 | 0.962
BF-0502 | 0.836 | 0.893 | 0.909

Performance on training videos. We report the clustering performance on
training videos in Table V. Note that both TSiam and SSiam are trained in an
unsupervised manner, or with automatically generated labels. Both of the
proposed models SSiam and TSiam provide a large performance boost over the
base VGG2 features on BBT and BF.

TABLE VI: Clustering accuracy computed at track-level across episodes within the same TV series. Numbers are averaged across 5 test episodes. Train | Test | Base | TSiam | SSiam
---|---|---|---|---
BBT-0101 | BBT-01[02-06] | 0.935 | 0.930 | 0.914
BF-0502 | BF-05[01,03-06] | 0.892 | 0.889 | 0.904

Generalization within series. In this experiment, we evaluate the
generalization capability of our models. We train on one episode each of
BBT-0101 and BF-0502 and evaluate on all other episodes of the same TV series.
Table VI reports clustering accuracy. Both SSiam or TSiam perform similar
(slightly lower/higher) to the base features, possibly due to overfitting.

TABLE VII: Clustering accuracy when evaluating across video series. Each row
indicates that the model was trained on one episode of BBT / BF, but evaluated
on all 6 episodes of the two series.

| Trained | BBT-01[01-06] | BF-05[01-06]
---|---|---|---
TSiam | BBT-0101 | 0.936 | 0.875
BF-0502 | 0.915 | 0.890
SSiam | BBT-0101 | 0.922 | 0.862
BF-0502 | 0.883 | 0.905

Generalization across series. We further analyze our models by evaluating
generalization across series. From Table VII, we wish to point out an
interesting observation: TSiam and SSiam retain their discriminative power and
can transfer to other series gently. As they learn to score similarity between
pairs of faces, the actual identity and characters do not seem to matter much.
For example, the drop when training TSiam on BBT-0101 and evaluating on BF is
0.890 (train on BF-0502) to 0.875.

Please note that the generalization experiments are presented here to explore
the underlying properties of our models. If achieving high performance is the
only goal, we assert that our models can be trained and evaluated on each
video rapidly and fully automatically.

TABLE VIII: Clustering accuracy when extending to all named characters within the episode. BBT-0101 has 5 main and 6 named characters. BF-0502 has 6 main and 12 named characters. | BBT-0101 | BF-0502
---|---|---
| TSiam | SSiam | TSiam | SSiam
Main cast | 0.964 | 0.922 | 0.893 | 0.905
All named cast | 0.958 | 0.922 | 0.829 | 0.870

Generalization to unseen characters. In the ideal setting, we would like to
cluster all characters appearing in an episode including (main, other named,
unknown, and background). However, this is a very difficult setting, and in
fact, disambiguating background characters is even hard for humans and there
are no datasets that include such labels. For BBT and BF, we do however have
all named characters labeled. Firstly, expanding the clustering experiment to
include them drastically changes the class balance. For example, BF-0502 has 6
main and 12 named characters with class balance 36.2/5.0 to 40.8/0.1.

We present clustering accuracy for this setting in Table VIII. Both proposed
methods show a drop in performance when extending to unseen characters. Note
that the models have been trained on only the main characters data and tested
on all (including unseen) characters. However, the drop is small when adding
just 1 new character (BBT-0101) vs. introduction of 6 in BF-0502.

SSiam’s performance scales well, probably since it is trained with a diverse
set of pairs (dynamically generated during training) and can generalize to
unseen characters.

###  IV-D Comparison with the state-of-the-art

BBT and BF. We compare our proposed methods (TSiam, and SSiam) with the state-
of-the-art approaches in Table IX. We report clustering accuracy (%) on two
videos: BBT-0101 and BF-0502. Historically, previous works have reported
performance at a frame-level. We follow this for TSiam and SSiam.

Note that our evaluation uses 2-4 times larger number of frames than previous
works [48, 47] making direct comparison hard. Specifically in BBT-0101 we have
41,220 frames while [47] uses 11,525 frames. Similarly, we use 39,263 frames
for BF-0502 (vs. 17,337 [48]). Even though we cluster more frames and tracks
(with more visual diversity), our approaches are comparable to or even better
than the current results.

TSiam and SSiam are better than the improved triplet method [47] on BBT-0101.
SSiam obtains 99.04% accuracy which is 3.04% higher performance in absolute
gains. On BF-0502, TSiam performs the best with 92.46% which is 0.33% better
than the JFAC [48].

TABLE IX: Comparison to state-of-the-art. Metric is clustering accuracy (%)
evaluated at frame level. Please note that many previous works use fewer
tracks (# of frames) (also indicated in Table I) making the task relatively
easier. We use an updated version of face tracks provided by [2].

Method | BBT-0101 | BF-0502 | Data Source
---|---|---|---
BBT | BF
ULDML (ICCV ’11) [4] | 57.00 | 41.62 | $-$ | [4]
HMRF (CVPR ’13) [42] | 59.61 | 50.30 | [26] | [7]
HMRF2 (ICCV ’13) [41] | 66.77 | $-$ | [26] | $-$
WBSLRR (ECCV ’14) [43] | 72.00 | 62.76 | $-$ | [7]
VDF (CVPR ’17) [30] | 89.62 | 87.46 | [2] | [2]
Imp-Triplet (PacRim ’16) [47] | 96.00 | $-$ | [26] | $-$
JFAC (ECCV ’16) [48] | $-$ | 92.13 | $-$ | [7]
Ours (with HAC)
TSiam | 98.58 | 92.46 |  |
SSiam | 99.04 | 90.87 |  [2]* |  [2]*

ACCIO. We evaluate our methods on ACCIO dataset with 36 named characters, 3243
tracks, and 166885 faces. The largest to smallest cluster ratios are very
skewed: 30.65% and 0.06%. In fact, half the characters correspond to less than
10% of all tracks. Table X presents the results when performing clustering to
yield 36 clusters (equivalent to the number of characters). In addition, as in
[48], Table XI (num. clusters = 40) shows that our discriminative methods are
not affected much by this skew, and in fact improve performance by a
significant margin over the state-of-the-art.

TABLE X: Performance comparison of TSiam and SSiam with JFAC [48] on ACCIO.

|  | #cluster=36 |
---|---|---|---
Methods | P | R | F
JFAC (ECCV ’16) [48] | 0.690 | 0.350 | 0.460
Ours (with HAC)
TSiam | 0.749 | 0.382 | 0.506
SSiam | 0.766 | 0.386 | 0.514

TABLE XI: Performance comparison of different methods on the ACCIO dataset.

|  | # clusters=40 |
---|---|---|---
Methods | P | R | F
DIFFRAC-DeepID2+ (ICCV ’11) [48] | 0.557 | 0.213 | 0.301
WBSLRR-DeepID2+ (ECCV ’14) [48] | 0.502 | 0.206 | 0.292
HMRF-DeepID2+ (CVPR ’13) [48] | 0.599 | 0.23.0 | 0.332
JFAC (ECCV ’16) [48] | 0.711 | 0.352 | 0.471
Ours (with HAC)
TSiam | 0.763 | 0.362 | 0.491
SSiam | 0.777 | 0.371 | 0.502

Computational complexity. Our models essentially consist of a few linear
layers and are very fast to compute at inference time. In fact, training the
SSiam for about 15 epochs on BBT-0101 requires less than 25 minutes (on a GTX
1080 GPU using the matconvnet framework [36]).

##  V Conclusion

We proposed simple, unsupervised approaches for face clustering in videos, by
distilling the identity factor from deep face representations. We showed that
discriminative models can leverage dynamic generation of positive/negative
constraints based on ordered face distances and do not have to only rely on
track-level information that is typically used. Our proposed models are
unsupervised (or use automatically generated labels) and can be trained and
evaluated efficiently as they involve only a few matrix multiplications.

We conducted experiments on three challenging video datasets, comparing their
differences in usage in past works. Overall, our models are very fast to train
and evaluate and outperform the state-of-the-art while operating on datasets
that contain more tracks with large changes in appearance.

Acknowledgements: This work is supported by the DFG, a German Research
Foundation - funded PLUMCOT project.

## References

  * [1] R. Aljundi, P. Chakravarty, and T. Tuytelaars.  Who’s that Actor? Automatic Labelling of Actors in TV series starting from IMDB Images.  In ACCV, 2016.
  * [2] M. Bäuml, M. Tapaswi, and R. Stiefelhagen.  Semi-supervised Learning with Constraints for Person Identification in Multimedia Data.  In CVPR, 2013.
  * [3] Q. Cao, L. Shen, W. Xie, O. M. Parkhi, and A. Zisserman.  VGGFace2: A Dataset for Recognising Faces across Pose and Age.  In FG, 2018.
  * [4] R. G. Cinbis, J. Verbeek, and C. Schmid.  Unsupervised Metric Learning for Face Identification in TV Video.  In ICCV, 2011.
  * [5] S. Datta, G. Sharma, and C. Jawahar.  Unsupervised learning of face representations.  In FG. IEEE, 2018.
  * [6] A. Diba, V. Sharma, and L. Van Gool.  Deep temporal linear encoding networks.  In CVPR, 2017.
  * [7] M. Everingham, J. Sivic, and A. Zisserman.  “Hello! My name is … Buffy” – Automatic Naming of Characters in TV Video.  In BMVC, 2006.
  * [8] B. Fernando, H. Bilen, E. Gavves, and S. Gould.  Self-supervised video representation learning with odd-one-out networks.  In Computer Vision and Pattern Recognition (CVPR), 2017 IEEE Conference on, pages 5729–5738. IEEE, 2017.
  * [9] E. Ghaleb, M. Tapaswi, Z. Al-Halah, H. K. Ekenel, and R. Stiefelhagen.  Accio: A Dataset for Face Track Retrieval in Movies Across Age.  In ICMR, 2015.
  * [10] M. Guillaumin, J. Verbeek, and C. Schmid.  Is that you? Metric Learning Approaches for Face Identification.  In ICCV, 2009.
  * [11] Y. Guo, L. Zhang, Y. Hu, X. He, and J. Gao.  MS-Celeb-1M: A Dataset and Benchmark for Large-Scale Face Recognition.  In ECCV, 2016.
  * [12] R. Hadsell, S. Chopra, and Y. LeCun.  Dimensionality Reduction by Learning an Invariant Mapping.  In CVPR, 2006.
  * [13] P. Hu and D. Ramanan.  Finding Tiny Faces.  In CVPR, 2017.
  * [14] G. B. Huang, M. Ramesh, T. Berg, and E. Learned-Miller.  Labeled Faces in the Wild: A Database for Studying Face Recognition in Unconstrained Environments.  In ECCV Workshop on Faces in Real-life Images, 2008.
  * [15] S. Jin, H. Su, C. Stauffer, and E. Learned-Miller.  End-to-end Face Detection and Cast Grouping in Movies using Erdős–Rényi Clustering.  In ICCV, 2017.
  * [16] A. Miech, J.-B. Alayrac, P. Bojanowski, I. Laptev, and J. Sivic.  Learning from video and text via large-scale discriminative clustering.  In ICCV, 2017.
  * [17] I. Misra, C. L. Zitnick, and M. Hebert.  Shuffle and learn: unsupervised learning using temporal order verification.  In European Conference on Computer Vision, pages 527–544. Springer, 2016.
  * [18] Z. A.-H. Monica-Laura Haurilet, Makarand Tapaswi and R. Stiefelhagen.  Naming TV Characters by Watching and Analyzing Dialogs.  In WACV, 2016.
  * [19] A. Nagrani and A. Zisserman.  From Benedict Cumberbatch to Sherlock Holmes: Character Identification in TV series without a Script.  In BMVC, 2017.
  * [20] O. M. Parkhi, K. Simonyan, A. Vedaldi, and A. Zisserman.  A Compact and Discriminative Face Track Descriptor.  In CVPR, 2014.
  * [21] O. M. Parkhi, A. Vedaldi, and A. Zisserman.  Deep Face Recognition.  In BMVC, 2015.
  * [22] G. Paul, K. Elie, M. Sylvain, O. Jean-Marc, and D. Paul.  A conditional random field approach for audio-visual people diarization.  In ICASSP, 2014.
  * [23] V. Ramanathan, A. Joulin, P. Liang, and L. Fei-Fei.  Linking people in videos with “their” names using coreference resolution.  In ECCV, 2014.
  * [24] A. Rohrbach, M. Rohrbach, S. Tang, S. J. Oh, and B. Schiele.  Generating Descriptions with Grounded and Co-Referenced People.  In CVPR, 2017.
  * [25] A. Rohrbach, A. Torabi, M. Rohrbach, N. Tandon, C. Pal, H. Larochelle, A. Courville, and B. Schiele.  Movie Description.  IJCV, 123(1):94–120, 2017.
  * [26] M. Roth, M. Bäuml, R. Nevatia, and R. Stiefelhagen.  Robust Multi-pose Face Tracking by Multi-stage Tracklet Association.  In ICPR, 2012.
  * [27] M. S. Sarfraz, A. Schumann, A. Eberle, and R. Stiefelhagen.  A pose-sensitive embedding for person re-identification with expanded cross neighborhood re-ranking.  In CVPR, 2018.
  * [28] F. Schroff, D. Kalenichenko, and J. Philbin.  FaceNet: A Unified Embedding for Face Recognition and Clustering.  In CVPR, 2015.
  * [29] V. Sharma, A. Diba, D. Neven, M. S. Brown, L. Van Gool, and R. Stiefelhagen.  Classification driven dynamic image enhancement.  In CVPR, 2018.
  * [30] V. Sharma, M. S. Sarfraz, and R. Stiefelhagen.  A simple and effective technique for face clustering in tv series.  In CVPR: Brave New Motion Representations Workshop. IEEE, 2017.
  * [31] J. Sivic, M. Everingham, and A. Zisserman.  “Who are you?” – Learning person specific classifiers from video.  In CVPR, 2009.
  * [32] Y. Taigman, M. Yang, M. Ranzato, and L. Wolf.  DeepFace: Closing the Gap to Human-Level Performance in Face Verification.  In CVPR, 2014.
  * [33] M. Tapaswi, M. Bäuml, and R. Stiefelhagen.  “Knock! Knock! Who is it?” Probabilistic Person Identification in TV-Series.  In CVPR, 2012.
  * [34] M. Tapaswi, O. M. Parkhi, E. Rahtu, E. Sommerlade, R. Stiefelhagen, and A. Zisserman.  Total Cluster: A Person Agnostic Clustering Method for Broadcast Videos.  In ICVGIP, 2014.
  * [35] M. Tapaswi, Y. Zhu, R. Stiefelhagen, A. Torralba, R. Urtasun, and S. Fidler.  MovieQA: Understanding Stories in Movies through Question-Answering.  In CVPR, 2016.
  * [36] A. Vedaldi and K. Lenc.  Matconvnet: Convolutional neural networks for matlab.  In ACMMM, 2015.
  * [37] P. Vicol, M. Tapaswi, L. Castrejon, and S. Fidler.  MovieGraphs: Towards Understanding Human-Centric Situations from Videos.  arXiv:1712.06761, 2017.
  * [38] X. Wang and A. Gupta.  Unsupervised learning of visual representations using videos.  In Proceedings of the IEEE International Conference on Computer Vision, pages 2794–2802, 2015.
  * [39] J. H. Ward Jr.  Hierarchical Grouping to Optimize an Objective Function.  Journal of the American Statistical Association, 58(301):236–244, 1963.
  * [40] L. Wolf, T. Hassner, and I. Maoz.  Face Recognition in Unconstrained Videos with Matched Background Similarity.  In CVPR, 2011.
  * [41] B. Wu, S. Lyu, B.-G. Hu, and Q. Ji.  Simultaneous Clustering and Tracklet Linking for Multi-face Tracking in Videos.  In ICCV, 2013.
  * [42] B. Wu, Y. Zhang, B.-G. Hu, and Q. Ji.  Constrained Clustering and its Application to Face Clustering in Videos.  In CVPR, 2013.
  * [43] S. Xiao, M. Tan, and D. Xu.  Weighted Block-sparse Low Rank Representation for Face Clustering in Videos.  In ECCV, 2014.
  * [44] R. Yan, A. Hauptmann, and R. Jin.  Multimedia search with pseudo-relevance feedback.  In International Conference on Image and Video Retrieval, pages 238–247. Springer, 2003.
  * [45] R. Yan, A. G. Hauptmann, and R. Jin.  Negative pseudo-relevance feedback in content-based video retrieval.  In Proceedings of the eleventh ACM international conference on Multimedia, pages 343–346. ACM, 2003.
  * [46] L. Zhang, D. V. Kalashnikov, and S. Mehrotra.  A unified framework for context assisted face clustering.  In ACMMM. ACM.
  * [47] S. Zhang, Y. Gong, and J. Wang.  Deep Metric Learning with Improved Triplet Loss for Face Clustering in Videos.  In Pacific Rim Conference on Multimedia, 2016.
  * [48] Z. Zhang, P. Luo, C. C. Loy, and X. Tang.  Joint Face Representation Adaptation and Clustering in Videos.  In ECCV, 2016.
  * [49] C. Zhou, C. Zhang, H. Fu, R. Wang, and X. Cao.  Multi-cue augmented face clustering.  In ACM’MM, 2015.