# Some remarks on Brauer Classes of $K3$-type

Federica Galluzzi  Dipartimento di Matematica, Università di Torino, Via Carlo
Alberto 10, Torino, Italy<EMAIL_ADDRESS>and  Bert van Geemen
Dipartimento di Matematica, Università di Milano, Via Cesare Saldini 50,
Milano, Italy<EMAIL_ADDRESS>
###### Abstract.

An element in the Brauer group of a general complex projective $K3$ surface
$S$ defines a sublattice of the transcendental lattice of $S$. We consider
those elements of prime order for which this sublattice is Hodge-isometric to
the transcendental lattice of another K3 surface $X$. We recall that this
defines a finite map between moduli spaces of polarized K3 surfaces and we
compute its degree. We show how the Picard lattice of $X$ determines the
Picard lattice of $S$ in the case that the Picard number of $X$ is two.

## Introduction

Let $S$ be a complex projective $K3$ surface and let $T(S)$ be its
transcendental lattice. The Brauer group $Br(S)$ of $S$ can be identified with
([Hu16, §18.1])

$Br(S)\,=\,{\operatorname{Hom}}(T(S),\mathbb{Q}/\mathbb{Z})~{}.$

An element $\alpha$ of order $p$, for a prime number $p$, in the Brauer group
$Br(S)$ then defines a Hodge substructure $T_{\alpha}(S)$ of index $p$ in the
transcendental lattice $T(S)$, it is the kernel of $\alpha$. The isometry
classes of these sublattices were determined in [vG05], [vGK23] for $p=2$ and
in [MSTV17] for $p>2$. In particular, if the Picard rank of $S$ is one and
$Pic(S)=\mathbb{Z}h$ with $h^{2}=2d$, then for any prime number $p$ there is
one class whose lattices are isometric to the transcendental lattice of a K3
surface of degree $2p^{2}d$. We will call these Brauer classes of K3-type.

There is a finite map

$\kappa:\;\mathcal{M}_{2p^{2}d}\,\longrightarrow\,\mathcal{M}_{2d}$

between the moduli spaces of polarized K3 surfaces of degree $2p^{2}d$ and
degree $2d$ whose fiber over $S$, with Picard rank one, consists of all $X$
with $T(X)\cong T_{\alpha}(S)$ for some $\alpha\in Br(S)_{p}$ of K3-type. We
determine the degree of this map in Proposition 3.3.1. Since the Hodge
structure $T(X)$ does not determine $X$ uniquely in general due to the
presence of Fourier-Mukai partners, this degree is not simply the number of
cyclic subgroups in $Br(S)_{p}$ of K3-type.

In case the Picard rank of $X$ is two, its Picard lattice is determined by two
integers $b,c$ and we write $X_{b,c}$ for such a surface. We use the
description of $\kappa$ to determine the Picard lattice of
$S_{b,c}:=\kappa(X_{b,c})$. In the Picard rank two case it can (and does)
happen that $X_{b,c}\cong S_{b,c}$, if so, the determinants of the Picard
lattices are the same. This is an easy necessary, but not sufficient,
criterion for the existence of an isomorphism.

The equality of the determinants occurs exactly when the transcendental
lattice of $X_{b,c}$, which is $T_{\alpha}(S_{b,c})$, is equal to
$T(S_{b,c})$. More precisely, consider a family of K3 surfaces over a disc
with special fiber $S_{b,c}$ and general fiber a polarized K3 surface $S$ of
degree $2d$ of Picard rank one. Then we can identify
$H^{2}(S_{b,c},\mathbb{Z})=H^{2}(S,\mathbb{Z})$ and we have $T(S_{b,c})\subset
T(S)$. Therefore there is a restriction map on the Brauer groups and on their
$p$-torsion subgroups. The kernel of this map is the subgroup of vanishing
Brauer classes, any non-zero element in it is called a vanishing Brauer class
([GvG23]):

$\langle\alpha_{van}\rangle\,=\,\ker(Br(S)_{p}\,\longrightarrow\,Br(S_{b,c})_{p})~{}.$

Let $X$ be a K3 surface of degree $2p^{2}d$ with Picard rank one and let
$S=\kappa(X)$ so that $T(X)$ has index $p$ in $T(S)$. We denote by
$\alpha_{X}\in Br(S)_{p}$ an element such that $\ker(\alpha_{X})=T(X)$. A
specialization of $S$ to $S_{b,c}$ then induces a specialization of $X$ to
$X_{b,c}$ and we have two cyclic subgroups in $Br(S)_{p}$, one is
$\langle\alpha_{X}\rangle$ and the other is $\langle\alpha_{van}\rangle$.
These two subgroups coincide exactly when $\alpha_{X}$ is trivial on
$T(S_{b,c})$, so when $T_{\alpha_{X}}(S_{b,c})=T(S_{b,c})$. In Theorem 3.1.1
we make these subgroups explicit.

We intend to use these results to study degree eight K3 surfaces $S_{\beta}$
associated to certain cubic fourfolds in [GvG23, Proposition 5.1.4]. In
[KS18], Kuznetsov and Shinder study the classes generated by K3 surfaces in
the Grothendieck ring of $K_{0}(\mbox{Var}/{\mathbb{K}})[{\mathbb{L}}^{-1}]$.
They use the geometry of the conic bundles associated to a Brauer class of
K3-type, in particular in the case of a specialization in which this class
vanishes. The very basic results in this paper might give some more insight
into these cases.

##  1\. Brauer groups, vanishing classes and invariants

###  1.2. Brauer classes and B-fields

We recall, following [GvG23], the main definitions but now for the case of an
arbitrary prime number $p$ rather than only $p=2$.

Let $S$ be a $K3$ surface, its transcendental lattice is
$T(S):=Pic(S)^{\perp}$ in $H^{2}(S,\mathbb{Z})$. The Brauer group of $S$ can
be identified with (cf. [Hu16, 18.1])

$Br(S)={\operatorname{Hom}}(T(S),\mathbb{Q}/\mathbb{Z})~{}.$

Since $H^{2}(S,\mathbb{Z})$ is a selfdual lattice, any such homomorphism
$\alpha$ can be defined by an element $B=B_{\alpha}\in H^{2}(X,\mathbb{Q})$,
called a B-field representative of $\alpha$:

$\alpha:\,T(S)\,\longrightarrow\,\mathbb{Q}/\mathbb{Z},\qquad t\,\longmapsto
B\cdot t\mod\mathbb{Z}~{}.$

Let $Br(S)_{p}$ be the $p$-torsion subgroup of $Br(S)$ and let $\alpha\in
Br(S)_{p}$. Then the homomorphism $\alpha$ takes values in
$\mbox{$\frac{1}{p}$}\mathbb{Z}/\mathbb{Z}$. A B-field
$B_{\alpha}\in\mbox{$\frac{1}{p}$}H^{2}(S,\mathbb{Z})$ is unique up to
$\mbox{$\frac{1}{p}$}Pic(S)+H^{2}(S,\mathbb{Z})$ :

$B_{\alpha}^{\prime}=B_{\alpha}+\mbox{$\frac{1}{p}$}D+c,\qquad D\in
Pic(S),\quad c\in H^{2}(S,\mathbb{Z})~{},$

###  1.3. Brauer classes, sublattices and invariants: the case $p=2$

In case the Picard rank of $S$ is one, the lattices $T_{\alpha}(S)$ for $S\in
Br(S)_{p}$ are classified up to isometry by their discriminant groups. This
leads to the following classification, for $p=2$ in Lemma 1.3.1 and for $p>2$
in Lemma 1.4.1. The case $p=2$ is presented in a format similar to the one for
$p>2$.

######  Lemma 1.3.1.

([vGK23, Theorem 2.3]) Let $S$ be a K3 surface such that $Pic(S)=\mathbb{Z}h$,
$h^{2}=2d>0$. Let $\alpha\in Br(S)_{2}$, $\alpha\neq 0$, and
$B\in\mbox{$\frac{1}{2}$}H^{2}(S,\mathbb{Z})\subset H^{2}(S,\mathbb{Q})$ a
B-field representing $\alpha$.

  1. a)

In case $2\centernot{|}d$ there are three isomorphism classes of lattices
$T_{\alpha}(S)$.

    1. i)

$Bh\equiv 0\mod\mathbb{Z}$, in this case there is a unique isomorphism class,
of order $2^{20}-1$, with discriminant group
$\mathbb{Z}/2d\mathbb{Z}\oplus\mathbb{Z}/2\mathbb{Z}\oplus\mathbb{Z}/2\mathbb{Z}$,

    2. ii)

$Bh\equiv 1/2\mod\mathbb{Z}$, $B^{2}\equiv 0\mod\mathbb{Z}$, in this case
there is a unique isomorphism class, of order $2^{9}(2^{10}+1)$, with
discriminant group $\mathbb{Z}/8d\mathbb{Z}$,

    3. iii)

$Bh\equiv 1/2\mod\mathbb{Z}$, $B^{2}\equiv\mbox{$\frac{1}{2}$}\mod\mathbb{Z}$,
in this case there is a unique isomorphism class, of order $2^{9}(2^{10}-1)$,
with discriminant group $\mathbb{Z}/8d\mathbb{Z}$.

  2. b)

In case $2{|}d$ there are three isomorphism classes of lattices
$T_{\alpha}(S)$.

    1. i)

$Bh\equiv 0\mod\mathbb{Z}$, $B^{2}\equiv 0\mod\mathbb{Z}$, in this case there
is a unique isomorphism class of lattices, of order $2^{9}(2^{10}+1)-1$, with
discriminant group
$\mathbb{Z}/2d\mathbb{Z}\oplus\mathbb{Z}/2\mathbb{Z}\oplus\mathbb{Z}/2\mathbb{Z}$,

    2. ii)

$Bh\equiv 0\mod\mathbb{Z}$, $B^{2}\equiv\mbox{$\frac{1}{2}$}\mod\mathbb{Z}$,
in this case there is a unique isomorphism class of lattices, of order
$2^{9}(2^{10}-1)$, with discriminant group
$\mathbb{Z}/2d\mathbb{Z}\oplus\mathbb{Z}/2\mathbb{Z}\oplus\mathbb{Z}/2\mathbb{Z}$,

    3. iii)

$Bh\equiv\mbox{$\frac{1}{2}$}\mod\mathbb{Z}$, in this case there is a unique
isomorphism class of lattices, of order $2^{20}$, with discriminant group
$\mathbb{Z}/8d\mathbb{Z}$.

A non-trivial Brauer class $\alpha\in Br(S)_{2}$ is of K3-type if
$B_{\alpha}h\equiv 1/2\mod\mathbb{Z}$ and $B^{2}_{\alpha}\equiv
0\mod\mathbb{Z}$ ([vG05, Corollary 9.4], the latter is significant only if
$2\centernot{|}d$).

In case $d=1$, let $C_{6}\subset\mathbb{P}^{2}$ be the (smooth) degree six
branch curve of the double cover $S\rightarrow\mathbb{P}^{2}$. Then the Brauer
classes in $Br(S)_{2}$ correspond to ([vG05], [IOOV17]):

  1. i)

If $B_{\alpha}h\equiv 0$, $\alpha$ corresponds to a point of order two $p\in
Jac(C_{6})$.

  2. ii)

If $B_{\alpha}h\equiv\mbox{$\frac{1}{2}$}\,$ and $\,B_{\alpha}^{2}\equiv 0$,
$\alpha$ corresponds to an even theta characteristic on $C_{6}$.

  3. iii)

If $B_{\alpha}h\equiv\mbox{$\frac{1}{2}$}\,$ and
$\,B_{\alpha}^{2}\equiv\mbox{$\frac{1}{2}$}$, $\alpha$ corresponds to an odd
theta characteristic on $C_{6}$.

###  1.4. Brauer classes, sublattices and invariants: the case $p>2$

In case the Picard rank of the degree $2d$ K3 surface $S$ is one, the lattices
$T_{\alpha}(S)$ for $S\in Br(S)_{p}$ and $p>2$ are classified up to isometry
by their discriminant groups, of order $2p^{2}d$, $d(T_{\alpha}(S),q)$, where
$q=q_{\alpha}$ is a quadratic form with values in $\mathbb{Q}/2\mathbb{Z}$. In
case the discriminant group is cyclic, we denote by $v$ a generator of the
discriminant group, $d(T_{\alpha}(S))=\langle v\rangle$.

To give the classification, we fix an isomorphism
$H^{2}(S,\mathbb{Z})=U^{3}\oplus E_{8}(-1)^{2}$ such that $h=(1,d)\in U$, the
first copy of $U$ in the lattice. Then

$T(S)=h^{\perp}=\mathbb{Z}v\oplus\Lambda^{\prime},\quad\mbox{with}\;v=(-1,d)\,\in
U~{}.$

Let $w:=(0,-1)\in U$. Then $wv=1$ and any
$\alpha\in{\operatorname{Hom}}(T(S),\mbox{$\frac{1}{p}$}\mathbb{Z}/\mathbb{Z})$
is determined by a B-field
$B_{\alpha}=\mbox{$\frac{1}{p}$}(i_{\alpha}w+\lambda_{\alpha})\in\mbox{$\frac{1}{p}$}H^{2}(S,\mathbb{Z})$
where $i_{\alpha}\in\mathbb{Z}$ and $\lambda_{\alpha}\in\Lambda^{\prime}$:

$\alpha:\,T(S)\,\longrightarrow\,\mbox{$\frac{1}{p}$}\mathbb{Z}/\mathbb{Z},\qquad\alpha(zv+\lambda^{\prime})\,=\,B_{\alpha}\cdot(zv+\lambda^{\prime})\,=\,\mbox{$\frac{1}{p}$}(i_{\alpha}z+\lambda_{\alpha}\cdot\lambda^{\prime})~{}.$

We define $c_{\alpha}:=-\lambda_{\alpha}^{2}/2\in\mathbb{Z}$ and we observe
that

$B_{\alpha}\cdot
h\,=\,-\mbox{$\frac{1}{p}$}i_{\alpha},\qquad\lambda_{\alpha}^{2}\,=\,-2c_{\alpha}~{}.$

######  Lemma 1.4.1.

([MSTV17, Theorem 9]) Let $S$ be a K3 surface such that $Pic(S)=\mathbb{Z}h$,
$h^{2}=2d>0$. Let $p>2$ be a prime number, $\alpha\in Br(S)_{p}$ and
$B_{\alpha}\in\frac{1}{p}H^{2}(S,\mathbb{Z})\subset H^{2}(S,\mathbb{Q})$ a
B-field representing $\alpha$.

  1. a)

In case $p\centernot{|}d$, there are three isomorphism classes of lattices
$T_{\alpha}(S)$,

    1. i)

the discriminant group is cyclic, so isomorphic to $\mathbb{Z}/2p^{2}d$, and
$-2dp^{2}q(v)\mod p$ is a quadratic residue, there are
$\mbox{$\frac{1}{2}$}p^{10}(p^{10}+1)$ such lattices;

    2. ii)

the discriminant group is cyclic, so isomorphic to $\mathbb{Z}/2p^{2}d$, and
$-2dp^{2}q(v)\mod p$ is not a quadratic residue, there are
$\mbox{$\frac{1}{2}$}p^{10}(p^{10}-1)$ such lattices;

    3. iii)

there is a unique isomorphism class of lattices with discriminant group
$\mathbb{Z}/2d\mathbb{Z}\oplus\mathbb{Z}/p\mathbb{Z}\oplus\mathbb{Z}/p\mathbb{Z}$
and there are ${(p^{20}-1)}/{(p-1)}$ such sublattices.

A Brauer class is of K3-type if the discriminant group is cyclic and
$-2dp^{2}q(v)\mod p$ is a square in $\mathbb{Z}/p\mathbb{Z}$.

  2. b)

In case $p|d$, there are four isomorphism classes of lattices $T_{\alpha}(S)$,

    1. i)

$B_{\alpha}h\equiv 0\mod\mathbb{Z}$, the discriminant group is
$\mathbb{Z}/2d\mathbb{Z}\oplus\mathbb{Z}/p^{2}\mathbb{Z}$ and $c_{\alpha}\mod
p$ is a quadratic residue, there are $\mbox{$\frac{1}{2}$}p^{9}(p^{10}-1)$
such sublattices;

    2. ii)

$B_{\alpha}h\equiv 0\mod\mathbb{Z}$, the discriminant group is
$\mathbb{Z}/2d\mathbb{Z}\oplus\mathbb{Z}/p^{2}\mathbb{Z}$ and $c_{\alpha}\mod
p$ is not a quadratic residue, there are $\mbox{$\frac{1}{2}$}p^{9}(p^{10}-1)$
such sublattices;

    3. iii)

$B_{\alpha}h\equiv 0\mod\mathbb{Z}$, there is a unique isomorphism class of
lattices with discriminant group is
$\mathbb{Z}/2d\mathbb{Z}\oplus\mathbb{Z}/p\mathbb{Z}\oplus\mathbb{Z}/p\mathbb{Z}$,
there are $(p^{9}+1)(p^{10}-1)/(p-1)$ such lattices;

    4. iv)

$B_{\alpha}h\not\equiv 0\mod\mathbb{Z}$, in this case there is a unique
isomorphism class of lattices with discriminant group
$\mathbb{Z}/2p^{2}d\mathbb{Z}$, there are $p^{20}$ such sublattices.

A Brauer class is of K3-type if the discriminant group is cyclic.

###  1.5. Remarks

The intersection number
$B_{\alpha}h\in\mbox{$\frac{1}{p}$}\mathbb{Z}/\mathbb{Z}$ is an invariant of
the Brauer class $\alpha$ only in case $p|d$. In fact, $B_{\alpha}$ and
$B_{\alpha}+\mbox{$\frac{1}{p}$}h$ define the same $\alpha$ but
$(B_{\alpha}+\mbox{$\frac{1}{p}$}h)h=B_{\alpha}h+\mbox{$\frac{1}{p}$}2d$ which
is congruent to $B_{\alpha}h$ only if $p|d$.

Moreover, if $p|d$ and $B_{\alpha}h\equiv 0\mod\mathbb{Z}$, one obtains the
invariant
$B_{\alpha}^{2}\in(\mbox{$\frac{1}{p}$})^{2}\mathbb{Z}/\mbox{$\frac{1}{p}$}\mathbb{Z}$
since any other representative is given by
$B_{\alpha}+\mbox{$\frac{1}{p}$}D+c$ with $D=ah\in Pic(S)$ and $c\in
H^{2}(S,\mathbb{Z})$.

If $p\centernot{|}d$, we see that there is a choice of the B-field
representative such that $B_{\alpha}h=0$, that is,
$B_{\alpha}\in\mbox{$\frac{1}{p}$}T(S)$, any other such representative is then
given by $B_{\alpha}+c$ with $c\in T(S)$.

###  1.6. Vanishing Brauer classes

Let $S$ be a K3 surface with $Pic(S)=\mathbb{Z}h$ and $h^{2}=2d$. We consider
a specialization of $S$ to a K3 surface $S^{\prime}$ with Picard lattice

$Pic(S^{\prime})\,=\,\left(\mathbb{Z}h\,\oplus\,\mathbb{Z}k,\;\begin{pmatrix}h^{2}&hk\\\
hk&k^{2}\end{pmatrix}=\begin{pmatrix}2d&b\\\
b&2c\end{pmatrix}\right)\quad\mbox{for some \;}b,c\in\mathbb{Z}~{}.$

We can identify $H^{2}(S^{\prime},\mathbb{Z})=H^{2}(S,\mathbb{Z})$ and we have
$T(S^{\prime})\subset T(S)$. Thus, there is a restriction map on the
$p$-torsion subgroups of the Brauer groups:

$Br(S)_{p}\,\longrightarrow\,Br(S^{\prime})_{p}~{}.$

A non-zero element in the kernel of this map is a vanishing Brauer class
([GvG23]).

In [GvG23, Proposition 2.1.2 and Corollary 2.1.3] we exhibited a B-field
representative of a vanishing Brauer class $\alpha_{van}\in Br(S)_{2}$ which
can be easily generalized. To do so we identify $H^{2}(S,\mathbb{Z})$ with
$H^{2}(S^{\prime},\mathbb{Z})$ such that $h\in Pic(S)$ specializes to the
element with the same name in $H^{2}(S^{\prime},\mathbb{Z})$.

######  Proposition 1.6.1.

Let $p$ be a prime number. We denote by $\alpha_{van}\in Br(S)_{p}$ a
vanishing Brauer class for the specialization of $(S,h)$ to $S^{\prime}$ as
above.

  * i)

There is an $\alpha_{van}$ with B-field representative given by

$B_{van}\,:=\,\mbox{$\frac{1}{p}$}k\;(\in\mbox{$\frac{1}{p}$}H^{2}(S,\mathbb{Z}))~{}.$

  * ii)

For this $\alpha_{van}\in Br(S)_{p}$ we have

$B_{van}h\,\equiv\,\mbox{$\frac{1}{p}$}b\mod\mathbb{Z},\quad
B_{van}^{2}\,\equiv\,2c(\mbox{$\frac{1}{p}$})^{2}\mod\mathbb{Z}~{}.$

Proof. Notice that
$\mbox{$\frac{1}{p}$}k\not\in\mbox{$\frac{1}{p}$}Pic(S)+H^{2}(S,\mathbb{Z})$,
hence it defines a non-trivial class in $Br(S)_{p}$. But
$\mbox{$\frac{1}{p}$}k\in\mbox{$\frac{1}{p}$}Pic(S^{\prime})$ hence
$\mbox{$\frac{1}{p}$}k$ defines the trivial class in $Br(S^{\prime})_{p}$.
Therefore the Brauer class with B-field representative $\mbox{$\frac{1}{p}$}k$
is the vanishing Brauer class $\alpha_{van}$. ∎

##  2\. K3 surfaces of degree $2d$ and $2p^{2}d$

###  2.1. The Mukai lattice and moduli spaces of sheaves

Let $(X,H)$ be a polarized K3 surface, with $H\in Pic(X)$ primitive, of degree
$H^{2}=2p^{2}d$ where $d>0$ and $p$ is a prime number. As in [MSTV17, §2.6,
§3] we consider the Mukai vector

$v\,:=\,(p,H,pd)\;\in\,\tilde{H}(X)\,:=\,H^{0}(X,\mathbb{Z})\oplus
H^{2}(X,\mathbb{Z})\oplus H^{4}(X,\mathbb{Z})~{}.$

The Mukai lattice $\tilde{H}(X)$ has the bilinear form

$(r,c,s)(r^{\prime},c^{\prime},s^{\prime})\,:=\,-(rs^{\prime}+sr^{\prime})+c\cdot
c^{\prime},\qquad\mbox{so}\quad v^{2}\,=\,0~{},$

where $c\cdot c^{\prime}$ is the intersection product of $c,c^{\prime}\in
H^{2}(X,\mathbb{Z})$ and $H^{0},H^{4}$ are naturally identified with
$\mathbb{Z}$. The Mukai lattice has the weight two Hodge structure defined by
the one on $H^{2}(S,\mathbb{Z})$. The sublattice of integral $(1,1)$-classes
is thus generated by the summands $H^{0},H^{4}$ and $Pic(S)\,(\subset H^{2}$).
In particular, $v$ is of type $(1,1)$.

From the work of Mukai [Mu84] it follows that the moduli space $M_{X}(v)$ of
sheaves $\mathcal{E}$ with

$v\,=\,v(\mathcal{E})\,:=\,\Big{(}\mbox{rank}(\mathcal{E}),c_{1}(\mathcal{E}),\mbox{rank}(\mathcal{E})+(1/2)c_{1}(\mathcal{E})^{2}-c_{2}(\mathcal{E})\Big{)}$

is a K3 surface $S$. It is the unique K3 surface for which there is a Hodge
isometry $H^{2}(S,\mathbb{Z})\cong v^{\perp}/v$. This implies that the image
of the transcendental lattice $T(X)$ of $X$ under the map $T(X)\hookrightarrow
v^{\perp}\rightarrow v^{\perp}/v$ has finite index in $T(S)$. The Picard ranks
of $X$ and $S$ are thus the same. The sublattice generated by
$H^{0}(X,\mathbb{Z}),H^{4}(X,\mathbb{Z})$ and $\mathbb{Z}H$ intersects
$v^{\perp}$ in a rank two sublattice whose image in $v^{\perp}/v$ has rank
one. Then $(S,h)$, where $h$ is a generator of this rank one lattice, is a
polarized K3 surface of degree $2d$ (cf. the proof of Theorem 2.2.1).

###  2.2. A map between moduli spaces

This defines a finite map

$\kappa\,=\,\kappa_{v}:\;\mathcal{M}_{2dp^{2}}\,\longrightarrow\,\mathcal{M}_{2d},\qquad(X,H)\,\longmapsto(M_{X}(v),h)$

where $\mathcal{M}_{e}$ is the coarse moduli space of K3 surfaces of degree
$e$. The case $d=1$ was used by Kondo in [Ko93].

The geometry behind this map is well understood in the case that $d=1$, $p=2$:
a general K3 surface $(X,H)$ of degree eight determines a K3 surface $(S,h)$
of degree two as follows. The line bundle $H$ gives an embedding of $X$ as a
complete intersection of three quadrics in $\mathbb{P}^{5}$. The surface $S$
is the double cover of the $\mathbb{P}^{2}$ that parametrizes the quadrics
which is branched over discriminant curve $C_{6}\subset\mathbb{P}^{2}$ which
parametrizes the singular quadrics ([Kh05], [IKh13], [IKh15], [KS18], [MSTV17,
3.2]).

######  Theorem 2.2.1.

Let, for a given $d>0$ and prime number $p$, $(X_{b,c},H)$ be a K3 surface of
degree $2p^{2}d$ with Picard lattice

$Pic(X_{b,c})\,=\,\left(\mathbb{Z}H\oplus\mathbb{Z}K,\;\begin{pmatrix}2p^{2}d&b\\\
b&2c\end{pmatrix}\right).$

Let $S_{b,c}:=\kappa(X_{b,c})$. The K3 surface $(S_{b,c},h)$ of degree $2d$
has Picard lattice

$Pic(S_{b,c})\,=\,\left\\{\begin{array}[]{rclc}\left(\mathbb{Z}h\oplus\mathbb{Z}k,\;\begin{pmatrix}2d&b\\\
b&2cp^{2}\end{pmatrix}\right)&\mbox{if $p\centernot{|}b$}~{},\\\ &&&\\\
\left(\mathbb{Z}h\oplus\mathbb{Z}k,\;\begin{pmatrix}2d&b/p\\\
b/p&2c\end{pmatrix}\right)&\mbox{if $p|b$}~{}.\\\ \end{array}\right.$

Proof. First of all we show that the general $S_{b,c}$ has a polarization of
degree $2d$. The sublattice of $(1,1)$ classes in $\tilde{H}(X_{b,c})$
contains the primitive sublattice $N$ generated by $(1,0,0),(0,H,0),(0,0,1)$.
One easily finds that

$N\cap v^{\perp}\,=\,\langle\alpha:=(-1,0,d),\;\beta:=(2p,H,0)\rangle,\qquad
v\,=\,p\alpha+\beta~{}.$

Therefore $(N\cap v^{\perp})/v\cong\mathbb{Z}h$ where $h$ is represented by
$\alpha$ and $h^{2}=\alpha^{2}=2d$, and $h$ is primitive in
$v^{\perp}/v=H^{2}(S_{b,c},\mathbb{Z})$, of type $(1,1)$ and $(S_{b,c},h)$
defines a point in $\mathcal{M}_{2d}$.

The Picard lattice of $S_{b,c}$ is the image of $N+\mathbb{Z}(0,K,0)$, the
sublattice of all $(1,1)$ classes in $\tilde{H}(X_{b,c})$, in
$H^{2}(S_{b,c},\mathbb{Z})$.

$(N+\mathbb{Z}(0,K,0))\cap
v^{\perp}\,=\,\langle\alpha,\;\beta,\;\gamma\rangle\quad\mbox{with}\quad\gamma:=\,\left\\{\begin{array}[]{cr}(0,pK,b),&p\centernot{|}\,b,\\\
(0,K,b^{\prime}),&b=pb^{\prime}.\end{array}\right.$

As $v=p\alpha+\beta$, the image of this sublattice is generated by the images
$h,k$ of $\alpha$ and $\gamma$ respectively and one finds the Gram matrices as
in the theorem. ∎

######  Corollary 2.2.2.

If the K3 surfaces $X_{b,c}$ and $S_{b,c}$ are isomorphic, then the prime
number $p$ does not divide $b$.

Proof. If the surfaces are isomorphic, the determinants of the Gram matrices
of the Picard groups must be the same. This is the case only if $p$ does not
divide $b$. (In general it is not the case however that if $p$ does not divide
$b$ then $X$ and $S$ are isomorphic, nor that their Picard lattices are
isomorphic.) ∎

##  3\. The map $\kappa$ and Brauer groups

###  3.1. The cyclic subgroup $C\subset Br(S)$ determined by $X$

For a general $(X,H)$, the transcendental lattice $T(X)=H^{\perp}$ maps to a
sublattice of index $p$ in $h^{\perp}$, in fact $H^{2}=p^{2}h^{2}$. In
particular there is an isomorphism $T(S)/T(X)\cong\mathbb{Z}/p\mathbb{Z}$ and
hence there is a surjective map
$T(S)\rightarrow\mbox{$\frac{1}{p}$}\mathbb{Z}/\mathbb{Z}$ whose kernel is
$T(X)$. Thus $S=\kappa(X)$ comes with a subgroup $C=C_{X}\subset Br(S)_{p}$ of
order $p$.

The following theorem identifies the subgroup $C\subset Br(S)_{p}$. It also
determines the vanishing Brauer class for a specialization of an $(S,h)$ with
Picard rank one to $S_{b,c}$. We recall that one can choose the isomorphism
$H^{2}(S,\mathbb{Z})\cong U^{3}\oplus E_{8}(-1)^{2}$ in such a way that $h$
maps to the vector $(1,d)$ in the first copy of $U$. Then $T(S)=h^{\perp}$ is
the sublattice

$T(S)\,=\,\mathbb{Z}t_{S}\oplus U^{2}\oplus
E_{8}^{2}\qquad(\subset\Lambda_{K3}\,:=\,U^{3}\oplus E_{8}(-1)^{2}).\qquad
t_{S}\,:=\,\left(\begin{smallmatrix}-1\\\ d\end{smallmatrix}\right)~{}.$

In case $X$ has higher Picard rank, let $H\cdot Pic(X)=\gamma\mathbb{Z}$. A
result of Mukai implies that the index of $T(X)$ in $T(S)$ is $GCD(p,\gamma)$,
as we verify below for the Picard rank two case.

######  Theorem 3.1.1.

Let $(X,H)$ be a K3 surface of degree $2p^{2}d$ with $Pic(X)=\mathbb{Z}H$ and
let $(S,h)=\kappa(X,H)$. Then there is an isomorphism
$H^{2}(S,\mathbb{Z})\cong U^{3}\oplus E_{8}(-1)^{2}=U\oplus\Lambda^{\prime}$
such that

$h\,\longmapsto\,\left(\left(\begin{smallmatrix}1\\\
d\end{smallmatrix}\right),0\right),\quad
T(S)\,\stackrel{{\scriptstyle\cong}}{{\longrightarrow}}\,\left(\begin{smallmatrix}-1\\\
d\end{smallmatrix}\right)\mathbb{Z}\oplus\Lambda^{\prime}~{},$

and

$T(X)\,=\,\ker(\alpha_{X}:\,T(S)\,\longrightarrow\,\mbox{$\frac{1}{p}$}\mathbb{Z}/\mathbb{Z},\quad
t\,\longmapsto B_{X}\cdot h\mod\mathbb{Z}),$

where the B-field is

$\quad B_{X}\,:=\,\mbox{$\frac{1}{p}$}\left(\left(\begin{smallmatrix}0\\\
1\end{smallmatrix}\right),0\right)\,\in\,(U\oplus\Lambda^{\prime})\otimes\mathbb{Q}~{}.$

In the specialization of $S$ to $S_{b,c}$, the subgroups of $Br(S)_{p}$
generated by $\alpha_{X}$ and $\alpha_{van}$ coincide if and only if $b$ is
odd.

In case $p=2$, the invariants of $B_{X}$ are $B_{X}h=1/2$ and $B_{X}^{2}=0$.
If also $d=1$, $S$ is a double plane, the Brauer class corresponds to an even
theta characteristic and $X$ is a K3 surface of degree eight, moreover, for
the specialization of $S$ to $S_{b,c}$ the vanishing Brauer class
$\alpha_{van}$:

  1. i)

is $\alpha_{X}$ and corresponds to an even theta for $b$ odd,

  2. ii)

corresponds to a theta characteristic for $b\equiv 2\mod 4$ which is even if
$c$ is even, but $\alpha_{van}\not=\alpha_{X}$, and is odd otherwise,

  3. iii)

corresponds to a point $p$ of order two in the Jacobian $Jac(C_{6})$ for
$b\equiv 0\mod 4$.

In case $b\equiv 0\mod 4$ the theta characteristic $\alpha_{van}+\alpha_{X}$
is even/odd exactly when $c$ is even/odd.

Proof. Up to isometry, there is a unique embedding of $Pic(X_{b,c})$ in the
K3-lattice $\Lambda_{K3}=U\oplus\Lambda^{\prime}$ with
$\Lambda^{\prime}=U^{2}\oplus E_{8}(-1)^{2}$ ([Ni80, Thm. 1.14.4]). We choose
the isometry such that, for some $K^{\prime}\in\Lambda^{\prime}$,

$H\,=\,((1,p^{2}d),0)\in U\oplus\Lambda^{\prime},\qquad
K=((0,b),K^{\prime})\in U\oplus\Lambda^{\prime},\quad
K^{2}=(K^{\prime})^{2}=2c~{}.$

As $Pic(X)=\mathbb{Z}H$ we get

$T(X)=H^{\perp}=\mathbb{Z}t_{X}\oplus\Lambda^{\prime},\quad
t_{X}\,:=\,\left(\begin{smallmatrix}-1\\\ p^{2}d\end{smallmatrix}\right)~{}.$

Notice that $\tilde{H}=(H^{0}(X)\oplus U\oplus
H^{4}(X))\oplus\Lambda^{\prime}$, let

$\tilde{U}\,:=\,H^{0}\oplus U\oplus
H^{4}\,\subset\,\tilde{H}(X,\mathbb{Z})~{},$

where $U$ is the first summand of $\Lambda_{K3}$. Then
$v=(p,H,pd)\in\tilde{U}$. As $\Lambda^{\prime}\subset v^{\perp}$ and $\langle
v\rangle\cap\Lambda^{\prime}=\\{0\\}$, this unimodular lattice maps
isomorphically to the sublattice $\Lambda^{\prime}\subset
v^{\perp}/v=H^{2}(S,\mathbb{Z})$. To find the image of $T(X)$, it remains to
find the image in $v^{\perp}/v$ of $(0,(-1,p^{2}d),0)\in\tilde{U}\cap
v^{\perp}$.

With the notation in the proof of Theorem 2.2.1,

$\tilde{U}\cap
v^{\perp}\,=\,\langle\alpha=(1,0,-d),\;\beta_{1}=(0,\left(\begin{smallmatrix}1\\\
0\end{smallmatrix}\right),pd),\;\beta_{2}=(0,\left(\begin{smallmatrix}0\\\
p\end{smallmatrix}\right),1)\rangle~{}.$

Notice that

$v\,=\,p\alpha\,+\,\beta_{1}\,+\,pd\beta_{2},\qquad(0,(-1,p^{2}d),0)\,=\,-\beta_{1}+pd\beta_{2}~{}.$

Hence $(\tilde{U}\cap v^{\perp})/v$ is generated by the images $h,k^{\prime}$
of $\alpha$ and $\beta_{2}$ whereas $\beta_{1}$ maps to $-ph-
pdk^{\prime}=-p(h+dk^{\prime})$. The intersection products are $h^{2}=2d$,
$hk^{\prime}=-1$, $(k^{\prime})^{2}=0$. The sublattice $\langle
h,k^{\prime}\rangle\subset H^{2}(S,\mathbb{Z})$ is isomorphic to $U$:

$\langle
h,k^{\prime}\rangle\,\stackrel{{\scriptstyle\simeq}}{{\longrightarrow}}\,U,\qquad
h\,\longmapsto(1,d),\quad k^{\prime}\,\longmapsto(0,-1)~{}.$

Then $(0,(-1,p^{2}d),0)=-\beta_{1}+pd\beta_{2}$ maps to
$p(h+dk^{\prime})+pdk^{\prime}=p(h+2dk^{\prime})$ which maps to $p(1,-d)\in
U$. So we can choose the isomorphism between $H^{2}(S,\mathbb{Z})$ and
$\Lambda_{K3}$ in such a way that $h\mapsto(1,d)$ and the image of
$(0,(-1,p^{2}d),0)$ maps to $p(1,-d)$. The image of $T(X)$ in
$H^{2}(S,\mathbb{Z})$ is then the sublattice

$T(X)\,\stackrel{{\scriptstyle\simeq}}{{\longrightarrow}}\,p\left(\begin{smallmatrix}-1\\\
d\end{smallmatrix}\right)\mathbb{Z}\,\oplus\,\Lambda^{\prime}\quad(\subset
T(S))~{}.$

Notice that for $B_{X}=(0,1)\in U$, we have $B_{X}(1,-d)=-1$ which implies
that $\ker(B_{X})=T(X)$.

A simple computation shows that
$B_{X}h=\mbox{$\frac{1}{p}$}(0,1)\cdot(1,d)=\mbox{$\frac{1}{p}$}$ and
$B_{X}^{2}=0$.

The subgroups generated by $\alpha_{X}$ and $\alpha_{van}$ are the same if and
only if the inclusion $T({X_{b,c}})\subset T({S_{b,c}})$ is an equality, which
is equivalent to these lattices having the same discriminants. This is again
equivalent to the determinants of the Picard lattices of $X_{b,c}$ and
$S_{b,c}$ being the same and by Theorem 2.2.1 we see that this happens if and
only if $p\not|\,b$.

The invariants of $\alpha_{van}$ are determined by the second column of a Gram
matrix of $Pic(S_{b,c})$ by Proposition 1.6.1. A Gram matrix is given in
Theorem 2.2.1 and (i)-(iii) follow.

In case $B_{van}$ corresponds to a point of order two, the sum
$B_{s}:=B_{X}+B_{van}$ corresponds to a theta characteristic. The parity of
this characteristic is determined by $B_{s}^{2}\mod\mathbb{Z}$. We find
$B_{X}^{2}=0$ and, by Proposition 1.6.1, $B_{van}=(1/2)k$ with $k$ as in
Theorem 2.2.1, hence $B_{van}^{2}=(1/4)(2c)=c/2$. It remains to compute
$2B_{X}\cdot B_{van}$ which we claim is $0$, so that $B_{s}^{2}=c/2$ and the
last statement of the theorem is proven.

To verify the claim, we recall that $k$ has representative $\gamma\in
v^{\perp}$ and since $b\equiv 0\mod 4$ we have $\gamma=(0,K,b^{\prime})$ where
$b=2b^{\prime}$. With our choice of embedding, $K=((0,b),K^{\prime})$ and then
$\gamma=(0,(0,b),b^{\prime})+(0,K^{\prime},0)=b^{\prime}\beta_{2}+(0,K^{\prime},0)$
which maps to $b^{\prime}(0,-1)+K^{\prime}\in
U\oplus\Lambda^{\prime}=H^{2}(S,\mathbb{Z})$. Hence $B_{X}\cdot
B_{van}\,=\,(0,1)\cdot(0,-b^{\prime}/2)=0$ since $B_{X}\cdot K^{\prime}=0$. ∎

###  3.2. An intrinsic description of the map $\kappa$

To define $\kappa:\mathcal{M}_{2p^{2}d}\rightarrow\mathcal{M}_{2d}$, we used a
Mukai vector $v$. Here we give another way to define the map, where we use
some of the notation from the (proofs of the) previous results.

Let $(X,H)$ be a polarized K3 surface of degree $2p^{2}d$. Recall that in
$H^{2}(X,\mathbb{Z})$ we have the sublattices $\mathbb{Z}H$ and
$H^{\perp}=\mathbb{Z}t_{X}\oplus\Lambda^{\prime}$, their direct sum has index
$2p^{2}d$ in $H^{2}(X,\mathbb{Z})$. To get all of the second cohomology group
one has to add the ‘glue vector’ $(H+t_{X})/2p^{2}d$. Since the discriminant
group $(H^{\perp})^{*}/H^{\perp}$ of $H^{\perp}$ is cyclic, there is a unique
overlattice, denoted by $h^{\perp}$, such that $H^{\perp}\subset h^{\perp}$
has index $p$. This overlattice is generated by $H^{\perp}$ and a $t_{S}\in
h^{\perp}$ with $pt_{S}=t_{X}$. Let $\mathbb{Z}h$ be the rank one lattice with
$h^{2}=2d$. Then the overlattice of $\mathbb{Z}h\oplus h^{\perp}$ defined by
the glue vector $(h+t_{S})/2d$ is an even unimodular lattice and hence is
isometric to $\Lambda_{K3}$. This lattice has the Hodge structure induced by
the one on $T(X)\subset h^{\perp}$ and hence defines a unique polarized K3
surface $(S,h):=\kappa((X,H))$ by surjectivity of the period map and the
Torelli theorem.

###  3.3. FM partners

We consider the fibers of the map $\kappa$ in the case $d=1,p=2$. Given a
general K3 surface $(S,h)\in\mathcal{M}_{2}$, by Theorem 3.1.1 an $(X,H)$ in
the fiber over it determines an order $2$ subgroup $C=\ker(\alpha_{X})\subset
Br(S)_{2}$. Moreover, the unique non-trivial Brauer class $\alpha\in C$
corresponds to an even theta characteristic on $C_{6}$. Given an even theta
characteristic on a general $C_{6}$, this invertible sheaf has no non-trivial
global sections and using [Be00] one obtains a K3 surface $X$ of degree $8$
from a resolution of this sheaf. Since there are $2^{9}(2^{10}+1)$ even theta
characteristics on the genus $10$ curve $C_{6}$, this number is also the
degree of $\kappa:\mathcal{M}_{8}\rightarrow\mathcal{M}_{2}$.

For a general $d\geq 1$ and a prime number $p$ however, the order $p$ subgroup
$C=\ker(\alpha_{X})\subset Br(S)_{p}$ only determines the sublattice

$T_{C}\,=\,T_{\alpha_{X}}(S)\,:=\,\ker(\alpha_{X}:\,T(S)\,\longrightarrow\,\mbox{$\frac{1}{p}$}\mathbb{Z}/\mathbb{Z})~{},$

with the induced Hodge structure from $T(S)$. To obtain a K3 surface $X$, one
must embed $T_{C}$ primitively into a K3 lattice, which can be done only if
$\alpha_{X}$, and thus $C$, is of K3-type, and even then the embedding need
not be unique up to isometry.

######  Proposition 3.3.1.

Let $(S,h)\in\mathcal{M}_{2d}$ be a polarized K3 surface with
$Pic(S)=\mathbb{Z}h$ and $h^{2}=2d$. For a prime number $p$, the cardinality
of the fiber of $\kappa:\mathcal{M}_{2p^{2}d}\rightarrow\mathcal{M}_{2d}$ over
$(S,h)$ is

$\sharp\kappa^{-1}(S,h)\,=\,\left\\{\begin{array}[]{cl}\mbox{$\frac{1}{2}$}p^{10}(p^{10}+1)&\mbox{if}\quad
d=1,\\\ p^{10}(p^{10}+1)&\mbox{if}\quad p\centernot{|}d,d>1,\\\
p^{20}&\mbox{if}\quad p|d~{}.\end{array}\right.$

A polarized K3 surface $(X,H)$ is in the fiber $\kappa^{-1}(S,h)$ if and only
if $T(X)$ is Hodge isometric to a sublattice of index $p$ of $T(S)$.

Proof. If $(X,H)\in\kappa^{-1}(S,h)$ then $T(X)$ is an index $p$ sublattice of
$T(S)$, hence it is defined by an order $p$ subgroup of K3-type of
$Br(S)_{p}$. Conversely such a subgroup $C$ defines a sublattice $T_{C}$ of
index $p$ that can be primitively embedded into the K3 lattice, that is, this
sublattice is isometric to $H^{\perp}$ for some $(X,H)$ if $C$ is of K3-type.

The ‘forgetful’ map that associates to a polarized K3 surface $(X,H)$, with
$H^{2}=2p^{2}d$, the Hodge structure $H^{\perp}$ defines a map
$\mathcal{M}_{2p^{2}d}\rightarrow\mathcal{M}^{T}_{2p^{2}d}$ which is finite
and has degree equal to the number of FM partners of $X$ for a K3 surface $X$
with $Pic(X)=\mathbb{Z}H$. This number is ([Og02, Proposition 1.10])

$|FM(X)|\,=\,2^{\tau(p^{2}d)-1},$

where $\tau(p^{2}d)$ is the number of prime factors of $p^{2}d$. Thus, given
the Hodge structure $T_{C}$, there are $2^{\tau(p^{2}d)-1}$ K3 surfaces $X$
with $T(X)\cong T_{C}$. However, there are also $2^{\tau(d)-1}$ K3 surfaces
$S^{\prime}$ with $T(S^{\prime})\cong T(S)$.

In particular, if $p|d$ then $\tau(d)=\tau(p^{2}d)$ and thus, given $T_{C}$,
each $S^{\prime}$ with $T(S^{\prime})\cong T(S)$ determines a unique
$X^{\prime}$ with $T_{C}\cong T(X^{\prime})$. The same is true if $d=1$:
$\tau(1)=\tau(p^{2})=1$.

If however $p\centernot{|}d$ and $d>1$ then $\tau(p^{2}d)=\tau(d)+1$ and
$S^{\prime}$ determines two K3 surfaces $X^{\prime}$.

The degree of the map, for $p=2$, $p>2$ then follows from [vGK23, Theorem 2.3]
and [MSTV17, §2.6] respectively, which gives the number of $C$ of K3-type. (In
[MSTV17, §2.7, Remark], it is stated that, based on a result of Kondo, for
$d=1$ the degree of $\kappa$ is $p^{10}(p^{10}+1)$, however Kondo [Ko24]
confirmed that the degree is $\mbox{$\frac{1}{2}$}p^{10}(p^{10}+1)$.) ∎

##  4\. Examples

###  4.1. K3 surfaces with a line

We consider some special cases of $\kappa:X_{2p^{2}d}\mapsto S_{2d}$ with
Picard rank two. In the literature we found the cases

$(d,p)\,=\,(1,2),\quad(1,3),\quad(2,2),\quad(2,3),\quad(3,2)~{}.$

We consider these cases where moreover $X$ has a line. Then $X=X_{1,-1}$ in
the notation of Theorem 2.2.1, so $b=1$. The Picard lattices of $X_{1,-1}$ and
$S_{1,-1}$ have the same determinant by Corollary 2.2.2. We then consider
whether the K3 surfaces $X_{1,-1}$ and $S_{1,-1}$ are isomorphic, which is not
always the case as we will see.

Since the determinants are the same, we have the equality of the Brauer
classes $\alpha_{X}=\alpha_{van}\in Br(S)$, that is, the Bauer class
$\alpha_{X}\in Br(S_{1,-1})$ defining $X_{1,-1}$ (up to possible FM partners)
is trivial. It would be interesting to see explicitly a rational section of
the specialization of the conic bundles with Brauer class $\alpha_{X}$ on a
general $S$ to $S_{1,-1}$ in this case. Such conic bundles are quite
explicitly known in some of the examples.

###  4.2. $X_{8}$ and $S_{2}$ : square determinants

We consider first the K3 surfaces $X_{b,c}$ of degree $8=2p^{2}d$ with
$d=1,p=2$, such that the Picard lattice has determinant $D$ which is a square:
$D=-\det(Pic(X_{8}))=b^{2}-16c$. In that case $X_{b,c}$ has a genus one
fibration given by a divisor class $E$ with $E^{2}=0$.

In [KS18, Lemma 3.10] it is shown that there are infinitely many
$D=-\det(Pic(X_{8}))=b^{2}-16c$ for which $X_{b,c}$ and $S_{b,c}$, with $b$
odd, are not isomorphic, even if the Picard lattices have the same
determinant. For this they consider the case $D=m^{2}$ for an odd integer $m$,
for example take $b=m$ and $c=0$. There is an isomorphism $X_{b,c}\cong
S_{b,c}$ if and only if $r^{2}-Ds^{2}=\pm 8$ has an integer solution according
to [MN03]. This can now be written as $r^{2}-(ms)^{2}=\pm 8$ and if one takes
$ms>3$ then $|r^{2}-(ms)^{2}|>8$ for any $r\neq\pm ms$, hence the result.

A simple geometric example where these surfaces are not isomorphic is thus the
case that $X$ is a smooth complete intersection of three quadrics that
contains a rational normal cubic curve. Then $X=X_{3,-1}$ and $K$ is the class
of the cubic curve:

$Pic(X_{3,-1})\,=\,\left(\mathbb{Z}H\oplus\mathbb{Z}K,\;\begin{pmatrix}8&3\\\
3&-2\end{pmatrix}\right),\qquad D=-\det(Pic(X_{8}))=25,$

and $X_{3,-1}\ncong S_{3,-1}$, in fact, the Picard lattices are not isometric.
From Theorem 2.2.1 we have

$Pic(S_{3,-1})\,=\,\left(\mathbb{Z}h\oplus\mathbb{Z}k,\;\begin{pmatrix}2&3\\\
3&-8\end{pmatrix}\right)\,\cong\,\left(\mathbb{Z}h\oplus\mathbb{Z}e,\;\begin{pmatrix}2&5\\\
5&0\end{pmatrix}\right),\quad e:=h+k~{}.$

If the Picard lattices were isometric, there should also be a $(-2)$-class,
like $K$, in $Pic(S_{3,-1})$. However, $(xh+ye)^{2}=2x^{2}+10xy=2x(x+5y)$ and
this is $-2$ only if either $x=1$, $x+5y=-1$ or $x=-1$, $x+5y=1$, however both
are impossible for $(x,y)\in\mathbb{Z}^{2}$.

On the other hand, in the cases $D=1,9$, one can take $X_{1,0}$ and $X_{3,0}$
respectively and these are isomorphic to $S_{1,0}$, $S_{3,0}$ respectively
since $(r,s)=(\pm 3,\pm 1)$, $(r,s)=(\pm 1,\pm 3)$ give solutions to
$r^{2}-Ds^{2}=\pm 8$. These two cases are well-known.

For $D=1$ the two Picard lattices are both isomorphic to the hyperbolic plane
$U$ and $X_{1,0}\cong S_{1,0}$ since the glueing of $U$ to $T(X)=T(S)$ is just
a direct sum. These surfaces have a unique elliptic fibration. (see [MN03,
Proposition 3.2.1]).

In the case $D=9$ the surface $X_{3,0}\cong S_{3,0}$ is a $K3$ surface of
bidegree $(2,3)$ in $\mathbb{P}^{1}\times\mathbb{P}^{2}$. See [MN03,
Proposition 3.2.1], [vG05, §5.8], [Be22]),[IKh13, Proposition 3.7]).

The classical association $X_{8}\mapsto S_{2}$, already mentioned in §3.3, is
studied in (the list is surely not complete) [IKh13], [IKh15], [Kh05], [KS18],
[MN03], [MSTV17].

###  4.3. $X_{16}$ with a line and $S_{4}$

For $d=2$, $p=2$ one has $X_{2p^{2}d}=X_{16}$. We assume that this surface
contains a line $L$. Then $X_{16}=X_{1,-1}$ and with $K$ the class of the
line:

$Pic(X_{1,-1})\,=\,\left(\mathbb{Z}H\oplus\mathbb{Z}K,\;M_{16}\,=\,\begin{pmatrix}16&1\\\
1&-2\end{pmatrix}\right).$

From Theorem 2.2.1 one finds

$Pic(S_{1,-1})\,=\,\left(\mathbb{Z}h\,\oplus\,\mathbb{Z}k,\;\begin{pmatrix}4&1\\\
1&-8\end{pmatrix}\right).$

The two Picard lattices are isomorphic:

$\left\\{\begin{array}[]{rcl}h&=&H-2K,\\\
k&=&-H+3K,\end{array}\right.\qquad\left\\{\begin{array}[]{rcl}H&=&3h+2k,\\\
K&=&h+k,\end{array}\right.$

and $\det(Pic(X_{16}))=\det(Pic(S_{4}))=-33$. To show that $X_{1,-1}\cong
S_{1,-1}$ it suffices to show that the glueing of the Picard lattice to the
transcendental lattice is unique up to isomorphisms. That again follows from
the surjectivity of the map $O(Pic(X_{1,-1}))\rightarrow O(D_{P})$ where
$D_{P}$ is the discriminant group of $Pic(X_{1,-1})$. In fact, let

$S\,:=\,\begin{pmatrix}19&64\\\ 8&27\end{pmatrix},\qquad\mbox{then}\quad
SM_{16}S^{t}\,=\,M_{16}~{},$

so that $S\in O(Pic(X_{16}))$. The discriminant group of the Picard group is
generated by $\delta:=(2,1)/33$ and one finds that $\delta S\equiv 23\delta$.
Since
$\mathbb{Z}/33\mathbb{Z}\cong\mathbb{Z}/3\mathbb{Z}\times\mathbb{Z}/11\mathbb{Z}$,
with $23\mapsto(-1,1)$, we see that $-I,S\in O(Pic(X_{16}))$ generate
$O(D_{P})$.

Geometrically, the isomorphism $X_{1,-1}\rightarrow S_{1,-1}$ is given by the
‘double projection’ from the line $L\subset X_{1,-1}\subset\mathbb{P}^{9}$.
First one projects from the line: $\phi_{H-K}:X_{16}\rightarrow
X^{\prime}_{12}\subset\mathbb{P}^{7}$, notice that $(H-K)^{2}=12$. The image
of $L$ is a rational normal curve of degree $(H-K)K=3$ which spans a
$\mathbb{P}^{3}\subset\mathbb{P}^{7}$. Projection from the span of the normal
curve induces the map $\phi_{H-2K}:X_{16}\rightarrow
S_{4}\subset\mathbb{P}^{3}$, the image of $L$ is a quintic rational curve in
the quartic surface $S_{4}$ since $(H-2K)K=5$.

See [IR05], [IR07], [MSTV17, §3.4], [vGK23, §5.3] for geometrical aspects of
the map $X_{16}\mapsto S_{4}$.

###  4.4. $X_{18}$ with a line and $S_{2}$

For $d=1$, $p=3$ one has $X_{2p^{2}d}=X_{18}$ and $S_{2d}=S_{2}$. Assume that
$X$ contains a line, then $X=X_{1,-1}$ and

$Pic(X_{1,-1})\,=\,\left(\mathbb{Z}H\oplus\mathbb{Z}K,\;\begin{pmatrix}18&1\\\
1&-2\end{pmatrix}\right)~{},$

therefore

$Pic(S_{1,-1})\,=\,\left(\mathbb{Z}h\,\oplus\,\mathbb{Z}k,\;\begin{pmatrix}2&1\\\
1\,&-18\end{pmatrix}\right).$

The two Picard lattices are isomorphic and have determinant $-37$, for example
an isomorphism is:

$\left\\{\begin{array}[]{rcl}h&=&2H-5K,\\\
k&=&-5H+13K,\end{array}\right.\qquad\left\\{\begin{array}[]{rcl}H&=&13h+5k,\\\
K&=&5h+2k~{}.\end{array}\right.$

The two $K3$’s are also isomorphic since the (sufficient) conditions in [MN04,
Theorem 3.1.5] are satisfied. More precisely, there exists $h_{1}\in
Pic(X_{18})$ such that $h_{1}^{2}=2p$ and $h_{1}H\equiv 0\mod p$ (here $p=3$),
for example $h_{1}=H+3K$.

Another way to see this is to notice that, since the order of the discriminant
groups is $37,$ a prime number, the orthogonal group of the discriminant
lattice of the Picard groups is $\\{\pm 1\\}$. Thus the glueing of the Picard
lattice to the transcendental lattice is unique and the surfaces are
isomorphic.

See also [MSTV17, §3.3] for the map $X_{18}\mapsto S_{2}$.

###  4.5. $X_{24}$ with a line and $S_{6}$

For $d=3$, $p=2$ one has $X_{2p^{2}d}=X_{24}$ and $S_{2d}=S_{6}$. Assume that
$X$ contains a line, then $X=X_{1,-1}$ and the Picard lattices of $X$ and
$S_{1,-1}$ have the Gram matrices

$P_{24}\,=\,\begin{pmatrix}24&1\\\ 1&-2\end{pmatrix},\qquad
P_{6}\,=\,\begin{pmatrix}6&1\\\ 1\,&-8\end{pmatrix}.$

The determinants are $-49$, so we are in the case of square determinants as in
§4.2. Let $e=h+k$ in $Pic(S_{1,-1})$, then $e^{2}=(h+k)^{2}=6+2-8=0$. Then
$e=h+k,h$ is a basis of the Picard lattice of $S_{1,-1}$ and

$(xe+yh)^{2}\,=\,x^{2}e^{2}+2xyeh+y^{2}h^{2}\,=\,14xy\,+\,6y^{2}\qquad(e=h+k)~{}.$

Notice that there is no $(-2)$-vector in this lattice since $y(7x+3y)=-1$ has
no integer solutions. Therefore the Picard lattices are not isometric and
hence $X_{1,-1}\not\cong S_{1,1}$.

The case $X_{24}\mapsto S_{6}$ was studied in detail in the recent paper
[KM23].

###  4.6. $X_{36}$ with a line and $S_{4}$

For $d=2$ and $p=3$ one has $X_{2p^{2}d}=X_{36}$, which has genus $19$, and
$S_{2d}=S_{4}$. This case was considered in [KM23, §1.2], see also [BBFM23,
Remark 4.16]. Assume that $X$ contains a line, then $X=X_{1,-1}$ and the
Picard lattices of $X$ and $S_{1,-1}$ have the Gram matrices

$P_{36}\,=\,\begin{pmatrix}36&1\\\ 1&-2\end{pmatrix},\qquad
P_{4}\,=\,\begin{pmatrix}4&1\\\ 1\,&-18\end{pmatrix}.$

The determinants are $-73$. The Gram matrices are equivalent:

$SP_{36}S^{t}\,=\,P_{4},\qquad S\,=\,\begin{pmatrix}57&272\\\
136&649\end{pmatrix}~{},$

hence the Picard lattices are isomorphic. Since the orthogonal group of the
discriminant group, which is $\mathbb{Z}/73\mathbb{Z}$, is $\\{\pm 1\\}$, the
glueing of the Picard lattice to the transcendental lattice is unique. Thus
$X_{1,-1}\cong S_{1,-1}$.

## References

  * [Be00] A. Beauville, Determinantal hypersurfaces, Michigan Math. J. 48 (2000) 39–64.
  * [Be22] A. Beauville A remark on the generalized Franchetta conjecture for $K3$ surfaces Math. Z. 300 (4) (2022) pp.3337–3340.
  * [BBFM23] V. Benedetti, M. Bolognesi, D. Faenzi, L. Manivel, Hecke cycles on moduli of vector bundles and orbital degeneracy loci, arXiv:2310.06482.
  * [GvG23] F. Galluzzi, B. van Geemen Invariants of Vanishing Brauer Classes, arXiv:2310.18200 .
  * [vG05] B. van Geemen, Some remarks on Brauer groups of K3 surfaces, Adv. Math. 197 (2005) 222–247.
  * [vGK23] B. van Geemen, G. Kapustka, Contractions of hyper-Kähler fourfolds and the Brauer group., Adv. Math. 412 (2023) Paper No. 108814.
  * [Hu16] D. Huybrechts, Lectures on K3 surfaces, Cambridge Studies in Advanced Mathematics 158 2016\.
  * [IR05] A. Iliev, K. Ranestad, Geometry of the Lagrangian Grassmannian LG(3, 6) with applications to Brill-Noether loci, Michigan Math. J. 53 (2005) 383–417.
  * [IR07] A. Iliev, K. Ranestad, The abelian fibration on the Hilbert cube of a K3 surface of genus 9, Internat. J. Math. 18 (2007) 1–26.
  * [IKh13] A. Ingalls, M. Khalid, Rank Two Sheaves on K3 Surfaces: A Special Construction, Q. J. Math., 64, 2, (2013), 443–-470.
  * [IKh15] C. Ingalls, M. Khalid, An explicit derived equivalence of Azumaya algebras on K3 surfaces via Koszul duality, J. Algebra 432 (2015) 300–327.
  * [IOOV17] A. Ingalls, A. Obus, E. Ozman, B. Viray, Unramified Brauer Classes on Cyclic Covers of the Projective Plane, in “Brauer Groups and Obstruction Problems”, Progress in Mathematics 320 2017 115–153.
  * [KM23] A. Kanemitsu, S. Mukai, Polarized K3 surfaces of genus thirteen and curves of genus three, arXiv:2310.02078.
  * [Kh05] M. Khalid, On K3 correspondences, J. reine angew. Math., 589, 2, (2005), 57–78.
  * [Ko93] S. Kondo, On the Kodaira dimension of the moduli space of K3 surfaces, Compositio Math. 89 (1993) 251–299.
  * [Ko24] S. Kondo, private communication, May 2014.
  * [KS18] A. Kuznetsov, E. Shinder, Grothendieck ring of varieties, $D$\- and $L$-equivalence, and families of quadrics, Sel. Math. New Ser. 24 (2018) 3475–3500.
  * [MN03] C. Madonna, V.V. Nikulin, On a classical correspondence between $K3$ surfaces (Russian) Proc.Steklov Inst.Math. 241 (2003) 132–168.
  * [MN04] C. Madonna, V.V. Nikulin, On a classical correspondence between $K3$ surfaces II in : M. Douglas, J. Gauntlett, M. Gross (eds.) Clay Mathematics Proceedings, 3  (Strings and Geometry) (2004) 285–300 .
  * [MSTV17] K. McKinnie, J. Sawon, S. Tanimoto, A. Várilly-Alvarado, Brauer groups on K3 surfaces and arithmetic applications, in: Brauer Groups and Obstruction Problems, in: Progr. Math., vol. 320, Birkhäuser, 2017, pp. 177–218.
  * [Mu84] S. Mukai, Symplectic structure of the moduli space of simple sheaves on an abelian or K3 surface, Invent. Math. 77 (1984) 101–116.
  * [Ni80] V. Nikulin, Integral symmetric bilinear forms and some of their applications, Math. USSR Izv. 14 (1980) 103–167.
  * [Og02] K. Oguiso, K3 surfaces via almost-primes, Math. Res. Lett. 9 (2002) 47–63.


# Multimodal Fusion Refiner Networks

Sethuraman Sankaran, David Yang, Ser-Nam Lim
Facebook AI, New York, NY, USA
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Tasks that rely on multi-modal information typically include a fusion module
that combines information from different modalities. In this work, we develop
a Refiner Fusion Network (ReFNet) that enables fusion modules to combine
strong unimodal representation with strong multimodal representations. ReFNet
combines the fusion network with a decoding/defusing module, which imposes a
modality-centric responsibility condition. This approach addresses a big gap
in existing multimodal fusion frameworks by ensuring that both unimodal and
fused representations are strongly encoded in the latent fusion space. We
demonstrate that the Refiner Fusion Network can improve upon performance of
powerful baseline fusion modules such as multimodal transformers. The refiner
network enables inducing graphical representations of the fused embeddings in
the latent space, which we prove under certain conditions and is supported by
strong empirical results in the numerical experiments. These graph structures
are further strengthened by combining the ReFNet with a Multi-Similarity
contrastive loss function. The modular nature of Refiner Fusion Network lends
itself to be combined with different fusion architectures easily, and in
addition, the refiner step can be applied for pre-training on unlabeled
datasets, thus leveraging unsupervised data towards improving performance. We
demonstrate the power of Refiner Fusion Networks on three datasets, and
further show that they can maintain performance with only a small fraction of
labeled data.

##  1 Introduction

Figure 1: Illustration of the idea behind refiner network that infuses a
responsibility condition on the fusion architecture.

In several real-world applications, decision making involves integrating
multiple modalities such as vision, text, auditory, and possibly even the
content creator and how people engage with the input [35, 3, 37]. The
application of multi-modal inference or decision making systems span several
fields such as hate speech detection [15], misinformation detection [22],
reasoning tasks [7], etc. Multimodal modeling includes two broad steps:
extraction of features from different modalities such as images and text, and
fusion of the different modalities. Several choices are available for fusion,
including late fusion, mid-fusion or early fusion [14, 33]. Early fusion
integrates features extracted from multiple modalities, and uses the
integrated feature representation for learning downstream tasks. On the other
hand, late fusion integrates classification scores of different features [41]
to obtain the final classification score. Whether it is early or late fusion,
different fusion strategies exist. These include Concat fusion [50], where
different features are concatenated first and input into a MLP to get an
integrated feature, as well as Set-based fusion [42] that is based on
permutation invariant functions or Graph-based fusion modules [1].

The approach in this paper is partly motivated by a limitation of current
fusion strategies that predominantly encode multimodal information, ignoring
potentially the importance of retaining unimodal signals. Even powerful
multimodal architectures such as those with co-attention transformer layers
[29] across modalities, which attend from the visual stream over language
stream, leverage such approaches for the final fusion.

We propose a complementary approach wherein we refine these fusion
representations to optimally balance independent information that each
modality carries with multimodal representations to inform the downstream
task. To this end, we leverage notions of responsibility [58] and contrastive
learning [23] to build what we refer to as a Refiner Fusion Network (ReFNet).
The main idea behind ReFNet is to balance the fusion module with a
‘responsibility’ driven refiner module [19, 57] as illustrated in Figure 1.
The refiner module drives creation of sets of artificial neurons
preferentially tuned to specific modal inputs, while the fusion module drives
creation of artificial neurons with mixed representations. The responsibility
condition is akin to a regularizer that helps guard against over-fitting of
the fusion module to the downstream task, and reduces vanishing gradients with
respect to specific input fusion modes. By simultaneously balancing the number
of unimodal and multimodal neurons during training, a better representation of
the input modalities is achieved to perform multi-modal fusion informed by the
downstream task.

One interesting finding of this work is that by imposing a responsibility
condition in addition to the downstream task, the refiner module induces
strong graphical structures between modalities, which we can prove under
certain conditions. Such a latent inductive graph, when coupled with metric
learning, encourages a latent graphical structure across modalities and
samples, which we believe leads to a stronger fusion representation and
performance on downstream tasks. More importantly, as opposed to transductive
graph modeling [28], an inductive graph avoids the burden of needing to
“carry” an adjacency matrix during inference time, which can get very large in
real-world use cases.

Our contributions can be summarized as follow:

  1. 1.

We propose ReFNet, a refiner module that can be added to any given fusion
module that helps to induce neurons that are each responsible for a specific
modality. We show that ReFNet can boost performance even over powerful
transformers and can help induce latent graphical structures that we can show
under certain conditions.

  2. 2.

When coupled with metric learning, which we call ReFNetMS, we observe a
further boost in performance, and surmise from the T-SNE plots that we can
generate stronger clustering and representation of the different classes.

  3. 3.

Lastly, we demonstrate that ReFNet has increased level of tolerance to lesser
amount of labeled data, which also helps reduce annotation needs.

##  2 Related Work

Responsibility problem The notion of responsibility was first introduced by
Zhang and colleagues [58]. The goal of this paper was that when permutation
invariant sets are mapped to a latent space, the neurons of the encoding
function must be faithful to the discontinuities introduced from the input
space to the set entries. This was expanded into Set and Graph Refiner
networks in [19] wherein an inner loop optimization was performed to divide
inputs into set elements that satisfy the responsibility problem, in contrast
to say splitting just the featurizer of the CNN of an image. This showed
better performance in relational reasoning tasks, and this approach can be
used with other permutation invariant architectures such as Deep Multimodal
Sets [42],

Multitask Learning The idea behind multitask learning is to learn tasks in
parallel but using a shared representation [8]. A CentralNet architecture
expanded this idea for multimodal fusion networks [47, 39, 39]. Their approach
was to create a central network that links modality specific networks. Each
modality is allowed to make decisions independent of other modalities, while a
central network aims to leverage the mixed modalities. Taskonomy [55] builds
the shared representation space by first learning several low level tasks that
can be of generic value to several downstream tasks. ReFNet is designed to
complement these approaches where the refiner operates on the shared
representation space to decode back the unimodal signals.

Graph based Fusion In Graph based fusion modules, each input modality can be
considered nodes of a graph with a known adjacency matrix [56, 1] or
explicitly modeling interaction across modalities [31]. The GINFusion model
[53] creates a representation of the graph in an embedding space using a dense
graph connection. Adding ReFNet to the downstream loss pushes the fusion
architecture to have strong unimodal and strong multimodal components, and to
induce edges between modalities, when they exist (as we show later in the
manuscript).

Autoencoder Autoencoders play a big role in unsupervised learning, transfer
learning and dimensionality reduction [4, 6, 32]. Set autoencoders can be used
for dimensionality reduction of a set of features to a lower dimensional space
[9]. Multi-modal autodecoders have been developed for filling in missing data
[20]. A special case of the Refiner Network will be the cyclic loss function
introduced in [59]. ReFNet enables encoding of feature sets that feed into a
fusion module.

Metric Learning Supervised deep metric learning has been the focus of several
research efforts [27, 21, 13]. Contrastive loss [16, 18] and triplet losses
[43, 11] are being widely used in several applications. In contrastive
learning [23, 51], samples with similar labels are pulled together in the
fusion embedding space while those with dissimilar labels are pulled apart.
Contrastive loss has two ingredients for a given anchor: pool samples with
similar labels (positive) and those with dissimilar labels (negative), and
minimize distances in the embedding space for the former and maximize
distances for the latter. Triplet loss uses anchors, where one positive and
negative sample are chosen per anchor, which are typically the hardest
examples for a given anchor. Other approaches include lifted structures [36],
n-pair losses [46], quadruplets [10], angular loss [48], adapted triplet loss
[54], and multi-similarity loss functions [49] that utilize pair-wise
relations across samples in a batch. A recent work [34] demonstrated that
gains with metric learning are more modest compared to what is commonly
reported. In this paper, we use the Multi-Similarity contrastive loss function
in combination with the refiner network. We call this method ReFNetMS, wherein
the responsible weights are trained in combination with maximizing separation
of dissimilar embeddings in the fusion space, that can simultaneously elicit
the underlying graphical structure across modalities and samples.

Figure 2: Schematic of the proposed algorithm with a refiner and contrastive
loss module. The image based features $F_{1},F_{2},F_{3},F_{4}$ and text based
features $F_{5},F_{6}$ are fused together, and a refiner is applied on the
fused embedding to generate refiner outputs $R_{1},R_{2},\cdots,R_{6}$ which
are used to define a self-supervised loss function and a supervised Multi-
similarity contrastive loss is also used across samples in a batch.

##  3 Multimodal Refiner Fusion Network Design

Let $F_{1}$, $F_{2}$, $\cdots$ $F_{M}$ refer to featurized inputs of $M$
modalities to a fusion module. A fusion module then aggregates these inputs,
and creates a fused embedding of the multi-modal features as

${\text{F}}_{\text{emb}}=\mathcal{A}([F_{1},F_{2},\cdots F_{M}])$ (1)

where $\mathcal{A}$ maps the input features to an embedding space. In the
context of this paper, $\mathcal{A}$ can encompass many of the fusion methods
in literature such as Concat/MLP or other fusion modules used with Concat-
Bert, ViLBERT, MMBT, etc. The fused embedding is subsequently used to train a
downstream task such as classification.

The refiner module that we introduce in this paper decomposes the fused
embedding into set elements, and imposes a refiner loss that the decomposed
set elements can capture the featurized inputs to the fusion module (Fig. 2).

$R_{i}=\mathcal{D}_{i}({\text{F}}_{\text{emb}})\forall i=1,2,\cdots M$ (2)

where $R_{i}$ are the set elements generated by the refiner module,
$\mathcal{D}$. We then introduce a self-supervised loss function,
$\mathcal{C}_{\text{i,ss}}$, for each fusion input

$\mathcal{C}_{\text{i,ss}}=1-\text{Cosine
Similarity}(R_{i},H_{i}(F_{i}))\forall i=1,2,\cdots M$ (3)

where $H_{i}$ is a mapping of the features to the refiner space. In general
$H_{i}$ can be the identity transformation unless the problem necessitates a
lower dimensional refiner space than the feature space (e.g. when not
sufficient training samples are present). The total refiner cost function is
$\sum_{i=1}^{M}\gamma_{i}\mathcal{C}_{\text{i,ss}}$ where $\gamma_{i}$ are the
weights of the refiner cost function associated with the different features.

###  3.1 Refiner as a self-supervision module

One advantage of the refiner module is that it is self-supervised and can
leverage unlabeled data (refer Fig. 2). Hence, unsupervised data, when
available, can be pre-trained initially by the refiner module before training
on the downstream task. This helps in reducing the amount of training labels
required, especially when considering that labeling on multi-modal tasks is
typically more expensive than in unimodal tasks such as classification.

###  3.2 Refiner Fusion Module on Multi-modal transformers

Refiner module can be applied on top any fusion architecture that takes as
input feature streams from the multiple modalities that are finally fused
together. These streams can be in the form of a set or a graph. Multi-modal
transformers such as MMBT [24], Visual Bert [26] and ViLBERT [29] have emerged
as strong multi-modal models in the recent past. They combine the power of
BERT model [12] for processing text, caption or OCR, with powerful ResNet
models [17] to capture image features. In ViLBERT, a multi-modal co-attention
model is used that has demonstrated powerful state-of-the-art performance on
many public multi-modal benchmark tasks.

In the rest of the paper, we apply ReFNet on top of ViLBERT architecture. The
output post the co-attention layers of the text and visual streams are fused,
and a decoder is applied to the fused embedding to decode back the text and
visual embeddings using an MLP with hidden layers.

##  4 Inducing Latent Graph Structures

While we do not explicitly model graph in this paper, we show in this section
that our proposed method can exploit hidden graphical connections in the data,
both across modes within a training sample and across samples in a batch as
illustrated in Figure 3. In Theorem 4.1, we show that when an (unknown)
adjacency matrix, ${\bf A}$, exists that contains the connections across
modalities and when the fusion network and refiner network are linear, then
the inverse of the weights of the refiner network contains the weighted
adjacency matrix.

######  Theorem 4.1.

Let ${\bf A}$ be an unknown adjacency matrix and ${\bf W}$ be the weights of
an affine transformation to generate the fusion embedding, ${\bf E}$. The
weights $\Gamma$ of a linear refiner satisfy the property, $\Gamma{\bf W}{\bf
A}={\bf I}_{m}$.

###### Proof.

Let ${\bf F}^{m\times d}$ represent features where $m$ is the number of
modalities and $d$ is the size of each feature vector. We assume that each
modality has a feature vector of dimension $d$ without any loss of generality
(otherwise they can be padded with zeros).

The unknown adjacency matrix, ${\bf A}^{m\times m}$ contains ones whenever
modality $i$ and $j$ have an edge between them. Let ${\bf W}^{k\times m}$ be
the unknown coefficients that create the fusion embeddings, ${\bf E}^{k\times
d}$ from the features as shown in Equation 4.

${\bf E}={\bf W}{\bf A}{\bf F}$ (4)

where ${\bf E}$ is the embedding in a $k\times d$ space. The fusion module
generates weights, ${\bf W}^{*}={\bf W}{\bf A}$. The refiner calculates
weights $\Gamma^{m\times k}$ such that $\tilde{\bf F}=\Gamma{\bf E}=\Gamma{\bf
W}{\bf A}{\bf F}$. Since the refiner network finds weights such that
$\tilde{\bf F}={\bf F}$ (refer Eq. 3 where ${\mathcal{D}}({\bf x})=\Gamma{\bf
x}$ and $H_{i}$ is identity), we have

${\bf F}=\Gamma{\bf W}{\bf A}{\bf F}.$

Since the above holds $\forall{\bf F}$, $\Gamma{\bf W}{\bf A}={\bf I}_{m}$
where ${\bf I}_{m}$ is an identify matrix of size $m\times m$.

If ${\bf W}{\bf A}$ is invertible, then $\Gamma=({\bf W}{\bf A})^{-1}$.
Without using refiner, the weights, ${\bf W}^{*}$ will be tuned by a
downstream task of much lower dimension than the refiner module, and therefore
the weights will be tuned to generate a good representation of the graph in a
much lower dimensional space, thereby failing to induce the latent graphical
structure. ∎

######  Corollary 4.1.

When $k=m$ and ${\bf W}$ has a rank equal to $m$, $\Gamma^{-1}$ is the
weighted adjacency matrix, ${\bf W}{\bf A}$.

The above follows directly from Equation 4 because ${\bf W}{\bf A}$ has a rank
of $m$.

Figure 3: Illustration of the connections that the refiner induces (inter-
modality) and those that the metric learning induces (across samples).

###  4.1 Contrastive Loss

When the self-supervised refiner module is combined with a contrastive loss,
weights are driven to induce graphical structures both within modalities and
across samples. Without refiner, metric learning maximizes separation of fused
embeddings based on the downstream classification task. But the addition of
refiner to metric learning enables separation also across modalities (because
the refiner is responsible) as we demonstrate with a T-SNE figure later in the
manuscript. We use the Multi-similarity loss in this paper, though other
contrastive loss functions can be used. The Multi-similarity loss for $T$
training samples in a batch is calculated as [49]

$\displaystyle{\mathcal{L}}_{MS}=\frac{1}{T}\sum_{i=1}^{T}{\left[\frac{1}{\alpha}log[1+\sum
e^{-\alpha(S_{ik}-\lambda)}]\right]}+$
$\displaystyle{\left[\frac{1}{\beta}log[1+\sum
e^{\beta(S_{ik}-\lambda)}]\right]}$ (5)

where $S_{ij}$ is the similarity (dot product) between samples and $\alpha$,
$\beta$ and $\lambda$ are hyperparameters. We provide the overall algorithm in
Alg. 1 where $\gamma_{i}$ are the self-supervised loss coefficients, $\zeta$
is the loss coefficient for contrastive loss, $w_{k}$ are the weights of the
fusion and refiner networks and $\eta$ is the learning rate.

Algorithm 1 Algorithm for training Multi-modal Fusion Networks

Pretraining (optional)

Compute features, $F_{1},F_{2},\cdots,F_{M}$.

while ${\text{e}poch}\leq\text{max epochs}$ do

${\mathcal{L}}_{\text{p}retrain}=\sum\gamma_{i}{\mathcal{C}}_{i,ss}$

$w_{k+1}=w_{k}-\eta\frac{\partial{\mathcal{L}}_{\text{p}retrain}}{\partial w}$

end while

Training

while ${\text{e}poch}\leq\text{max epochs}$ and stop criterion is not met do

${\text{F}}_{\text{emb}}=\mathcal{A}([F_{1},F_{2},\cdots F_{M}])$

$R_{i}=\mathcal{D}_{i}({\text{F}}_{\text{emb}})\forall i=1,2,\cdots M$

$\mathcal{C}_{\text{i,ss}}=1-\text{Cosine
Similarity}(R_{i},H_{i}({\mathcal{F}}_{i})$

${\mathcal{L}}_{\text{t}rain}={\mathcal{L}}_{downstream}+\sum\gamma_{i}{\mathcal{C}}_{i,ss}+\zeta*{\mathcal{L}}_{MS}$

$w_{k+1}\leftarrow
w_{k}-\eta\frac{\partial{\mathcal{L}}_{\text{t}rain}}{\partial w}$

end while

##  5 Experiments

For this study, we use the pre-trained ViLBERT model [29] as a baseline for
training and testing ReFNet and ReFNetMS. Similar to [25], the ViLBert model
was only unimodally pretrained. Instead of using the element-wise product [29]
to fuse the unimodal representations of images and texts, we concatenate them
and feed them into a new linear layer to formulate the overall representation.
We are able to reproduce the baselines reported for each of examples in this
section. We feed the fused embedding to two new linear layers. The Multi-
similarity loss function is integrated with the fused embedding, and included
during the final training runs. We used the MMF framework [44] built on
PyTorch [38] for setting up the training and evaluation pipelines.

###  5.1 MM-IMDB

The multi-modal IMDB dataset [2] contains 25,959 movies and their poster,
genre, plot and other metadata fields such as year, language, writer,
director, and aspect ratio. The goal is to classify the movie into 24
categories. Each movie can contain more than one class. The micro-f1 and
macro-f1 scores were used to evaluate performance. The original dataset
contains a baseline using Gated Multimodal units [2]. The ViLBERT baseline
[45] is used here which improved upon the Gated Multimodal methods, and is
compared with ReFNet and ReFNetMS. Since each movie can simultaneously have
multiple classes present, the precision and recall scores are calculated based
on the f-score as follows [30].

The macro f1 score is calculated from the precision, $p_{j}$ and recall
$r_{j}$ of each class as

$f_{1}^{\text{macro}}=\frac{1}{N}\sum_{i=1}^{N}\frac{2\times p_{j}\times
r_{j}}{p_{j}+r_{j}}.$

The micro f1 score is calculated using all the class labels together as

$f_{1}^{\text{micro}}=\frac{2\times p^{\text{micro}}\times
r^{\text{micro}}}{p^{micro}+r^{micro}}$

where $p^{micro}$ and $r^{micro}$ are the precision and recall across all
classes calculated based on the total number of true positives, false
positives and false negatives.

We used the AdamW optimizer, with a Cosine warmup and Cosine decay learning
rate scheduler. The value of $\epsilon$ for AdamW optimizer was set to
$1e^{-8}$ with corresponding $\beta_{1}$ and $\beta_{2}$ as 0.9 and 0.999. The
batch size was set to 32, learning rate was set to $5e^{-}5$ and the fused
embedding dimension was set to $512$. An MLP with a hidden layer was used for
the decoding refiner module. For the metric loss function, values of
$\alpha=50$ and $\beta=2$ were chosen. We used values of
$\eta_{1}=0.1,\eta_{2}=0.1,\zeta=0.1$.

###  5.2 Hateful Memes

Hateful Memes dataset [25] contains over 10,000 multi-modal examples (image
and test) with the goal of detecting if an input is hateful or not. The
dataset is constructed such that unimodal models struggle and only multi-modal
models can succeed (see Fig. 4). Difficult examples (“benign confounders”) are
added to the dataset to make it hard to rely on unimodal signals. The dataset
comprises of five different types of memes: multimodal hate, where benign
confounders were found for both modalities, unimodal hate where one or both
modalities were already hateful on their own, benign image and benign text
confounders and finally random not-hateful examples. There were 1,000 samples
in the validation dataset and 2,000 examples in the test dataset. We used
values of $\eta_{i}=0.1,\zeta=0.25$.

Figure 4: A few examples of memes that may be considered benign (top) or mean
(bottom).

We use a cross entropy loss for the two-label classification. We train on 8
Navidia Volt100 GPUs with a total batch size of 32 for a total of 22000
updates. We evaluate every 1000 updates and save the model with the best AUROC
metric on the validation set. We use the AdamW optimizer with an initial
learning rate of 1.0e-05. We use a linear decay learning rate schedule with
warm up to train the model.

###  5.3 SNLI Visual Entailment

The SNLI Visual Entailment (SNLI-VE) dataset [52] consists of image-sentence
pairs whereby a real world image premise and a natural language hypothesis are
given. The goal is to determine if the natural language hypothesis can be
concluded given the information provided by the image premise. Three labels,
entailment (hypothesis is true), neutral or contradiction (hypothesis is
false), are assigned to image-sentence pairs. The dataset has 550k image-
sentence pairs generated based on the Stanford Natural Language Inference
(SNLI) [5] corpus and Flickr30k [40] dataset. The training setup is the same
as Hateful Memes dataset except we use a batch size of 480, an initial
learning rate of 5.0e-05, and a total of 10000 updates. We used values of
$\eta_{i}=0.1,\zeta=0.05$.

##  6 Results

###  6.1 MM-IMDB

Table 1 compares the performance of ReFNet and ReFNetMS on the test set of MM-
IMDB dataset to a ViLBERT baseline model in combination with different
pretrained models. In general, pretraining ViLBERT reduced the performance on
MM-IMDB dataset, but ReFNet was able to improve upon the performance. A
relative gain of 0.8% in the micro f1 score and 0.82% in the macro f1 score
were observed. Compared to other pretraining modules, ReFNet had gains between
0.48% and 2.12 %. The improvement on micro and macro f1 score using ReFNetMS
were statistically significant based on a t-test (p = 0.02 and 0.03
respectively).

model | macro f1 test. | micro f1 test.
---|---|---
ViLBERT | 58.48 $\pm$ 0.25 | 66.77 $\pm$ 0.14
ViLBERT-VQA2 | 57.70 | 66.42
ViLBERT-COCO | 57.72 | 65.63
ViLBERT-cc small | 58.20 | 66.70
ReFNet | 58.75 $\pm$ 0.07 | 67.02 $\pm$ 0.15
ReFNetMS | 58.96 $\pm$ 0.09 | 67.31 $\pm$ 0.19
Table 1: Comparison of the macro and micro f1 score on the test set across
ViLBERT in combination with different pretraining datasets [45].

###  6.2 Hateful Memes

Table 2 compares the performance of ReFNet and ReFNetMS with respect to the
ViLBERT baseline. ReFNet improves the accuracy by 3.30% with a relative gain
in the AUC Of 1.84%. The use of ReFNetMS improves the Accuracy further by
0.80%. The overall relative gain in AUC was 2.17%. Based on a t-test, both
ReFNet and ReFNetMS had a statistically significant improvement on the AUC
(p-value = 1e-4 and 0.006 respectively) and the accuracy (p-value $<$ 1e-6 for
both) on the test set.

model | Acc. val. | AUC val. | Acc. test. | AUC test.
---|---|---|---|---
base | 60.71 $\pm 0.29$ | 70.62 $\pm 0.42$ | 59.70 $\pm 0.20$ | 70.53 $\pm 0.07$
ReFNet | 62.45 $\pm$ 1.09 | 70.87 $\pm$ 0.41 | 63.00 $\pm$ 0.31 | 71.83 $\pm$ 0.13
ReFNetMS | 63.29 $\pm$ 1.31 | 70.99 $\pm$ 0.37 | 63.80 $\pm$ 0.36 | 72.06 $\pm$ 0.49
Table 2: Comparison of the Accuracy and AUC on both the validation and test
sets for the Hateful Memes dataset. ”base” refers to the baseline ViLBERT
model.

###  6.3 SNLI Visual Entailment

ReFNet showed a small relative improvement on the test set (improvement in
accuracy $0.1\%\pm 0.07$ on the test set which was not significant). However,
ReFNetMS improves the Accuracy relatively by $0.71\%$ which was significant
(p-value = 0.001). Since the dataset contains more than a hundred thousand
examples, even a 1% improvement results in thousands of images being correctly
classified.

Figure 5: Visualization of fusion features in reduced dimensions using T-SNE
with perplexity set to 100. Top: fusion features of ViLBERT baseline showing
the 3 clusters with entanglements. Middle: fusion features of ReFNet showing
the 3 clusters are better separated with less entanglements. Bottom: fusion
features of RefNetMS showing the Refiner and Contrastive loss inducing six
clusters across modalities and classes (three clusters for each of the vision
and text modalities). The colors red, blue, and green represent three classes
contradiction, entailment, and neutral.

##  7 Ablation Studies

For the ablation study for MM-IMDB, we successively chose a fraction of the
labeled dataset (5, 10 and 20%) for downstream classification while the self-
supervised refiner module leverages unlabeled data. The baselines were rerun
using the ViLBERT model. Metrics were reported on the test dataset based on
the model corresponding to the best validation performance on 20000
iterations. ReFNet was able to achieve a 4.03% higher micro f1 and 11.83%
higher macro f1 score compared to the ViLBERT baseline. ReFNetMS boosted the
performance over baseline to 6.51% micro f1 score and 13.77% macro f1 score.
These were 1.66 and 5.00 when 10% of the labeled data was used and 0.88 and
1.04 when 20% of the labeled data was used. A full summary of the ablation
study is provided in Table 3 and illustrated in Fig. 6.

Figure 6: Comparison of performance of the baseline ViLBERT model, ReFNet and ReFNetMS across datasets with 5%, 10% and 20% of labeled training data available. Figure on the left shows micro and macro f1 scores on the validation dataset and that on the right shows the scores on the test dataset. model | frac labeled | macro f1 test | micro f1 test
---|---|---|---
baseline | 5% | 34.06 | 56.62
ReFNet | 5% | 45.89 | 60.65
ReFNetMS | 5% | 47.83 | 63.13
baseline | 10% | 46.73 | 62.65
ReFNet | 10% | 48.70 | 63.41
ReFNetMS | 10% | 51.73 | 64.31
baseline | 20% | 56.08 | 65.75
ReFNet | 20% | 56.43 | 66.24
ReFNetMS | 20% | 57.12 | 66.63
Table 3: Ablation study on the MM-IMDB dataset. Fraction labeled is the
fraction of labeled samples used during training for the logit binary cross
entropy function.

On the Hateful Memes dataset, the results based on successive reduction in the
fraction of labeled data are summarized in table 4. Using just 5% of the
labeled data, the area under the ROC curve on the test set improves by 2.52%
and 3.84% using ReFNet and ReFNetMS respectively. The corresponding
improvements in accuracy are 4.14% and 4.60%. When using 10% of the labeled
dataset, the accuracy improved by 1.50% and the AUC improved by 0.50%, and
when using 20% of the labeled dataset, these were 3.50% and 2.59%
respectively.

| 5% V | 5% V | 5% T | 5% T | 10% V | 10% V | 10% T | 10% T | 20% V | 20% V | 20% T | 20% T
---|---|---|---|---|---|---|---|---|---|---|---|---
model | Acc | AUC | Acc | AUC | Acc | AUC | Acc | AUC | Acc | AUC | Acc | AUC
baseline | 55.56 | 59.4 | 52.8 | 57.12 | 56.05 | 60.32 | 54.45 | 60.28 | 61.08 | 50.20 | 50.2 | 61.31
ReFNet | 56.20 | 58.65 | 56.94 | 59.64 | 56.15 | 61.84 | 54.65 | 60.87 | 54.96 | 61.95 | 53.70 | 63.90
ReFNetMS | 57.34 | 62.02 | 57.40 | 60.96 | 55.65 | 61.82 | 54.25 | 60.78 | 58.13 | 62.78 | 58.30 | 62.86
Table 4: Ablation study on the Hateful Meme dataset. V refers to validation
set, T refers to the test set, and 5%, 10% and 20% refer to the corresponding
fraction of labeled training data used. AUC is the area under the ROC curve
and ACC is the accuracy of the predicted model. All the reported values
correspond to the model with the best performance on the validation dataset.

##  8 Discussion

We started with the hypothesis that imposing a responsibility condition on
Multimodal fusion architectures can improve performance on downstream tasks.
We imposed the responsibility condition using a refiner module to decode each
feature feeding into the fusion. This self supervised loss function for each
input feature used the cosine similarity between the decoded and original
fusion features. We demonstrated this by integrating refiner on a strong
transformer based multimodal baseline - the ViLBERT model. We showed that
imposing responsibility can result in relative gains of over 5.53% on the
accuracy and 1.84% on the Hateful memes dataset.

Reducing the amount of labeled data available showed that the self-supervised
ReFNet can effectively leverage unlabeled data and maintain performance much
better than the baseline ViLBERT model. For instance, with 5% of labeled data
available for MM-IMDB, ReFNet had an improved micro f1 performance of 4.05,
and adding metric loss (ReFNetMS) improved it further by 2.48, yielding a net
gain score of 6.53. The macro-f1 score had a net gain of 13.77. These
demonstrate how well the responsibility module combined with the metric loss
function could perform compared to baseline transformer models for multi-modal
classification. This is also beneficial for the purposes of multi-modal
labeling of datasets. In general, labeling for multi-modal problems such as
misinformation can be significantly more expensive compared to classification
tasks. Leveraging ReFNet with a baseline model such as ViLBERT can help reduce
the amount of labeling required to attain the same performance.

Choosing the refiner space to be the actual input (e.g. image pixels) rather
than the input to the fusion module resulted in a reduction in performance and
was not considered to be used with the ViLBERT architecture. This is likely
because the co-attention mechanism that ViLBERT models is nullified by
imposing responsibility prior to applying the co-attention. However, this
might provide a richer option for other architectures, especially when
significantly more supervised data is available (since the input space is
typically of a larger dimension than the input to the fusion module).

Metric Learning methods have shown strong improvements for multi-class
classification problems. When integrated with the self-supervised Refiner
network, the metric Refiner network is able to elicit representations of the
fused embeddings in a responsible setting, (i.e.) derive distance metrics in
the embedding space characterized by strong unimodal and mixed representations
of the input features. In order to demonstrate how the refiner network enables
inducing graphical representations of the fused embedding in the latent space,
we generated T-SNE plots using the fusion features with ViLBERT, ReFNet and
ReFNetMS algorithms. Fig. 5 shows that ViLBERT model generates 3 vaguely
separated fusion feature clusters due to their entanglements while the
clusters generated by ReFNet has a better separation between clusters and
clearly delineates them. The metric learning algorithm is able to further
separate the clusters across modalities, therefore we observe six clusters,
three different classes for each modality (vision and language). This
responsibility property can mitigate the problem of vanishing gradient by
driving the weights to have some unimodal adherence. In addition, we can
obtain performance boost when a few modalities dominate performance of the
downstream task, and the fusion module is unable to encode the other
modalities without the existence of a high dimensional regularizer such as the
Refiner module.

Limitations and Future Work: Based on some initial experiments, we chose the
Multi-Similarity loss function but did not really explore the full space of
Metric Loss functions. Further, we did not showcase examples containing multi-
modal problems with strong transductive graph baselines.

## References

  * [1] Michalis Angelou, Vassilis Solachidis, Nicholas Vretos, and Petros Daras.  Graph-based multimodal fusion with metric learning for multimodal classification.  Pattern Recognition, 95:296–307, 2019.
  * [2] John Arevalo, Thamar Solorio, Manuel Montes-y Gómez, and Fabio A González.  Gated multimodal units for information fusion.  arXiv preprint arXiv:1702.01992, 2017.
  * [3] Pradeep K Atrey, M Anwar Hossain, Abdulmotaleb El Saddik, and Mohan S Kankanhalli.  Multimodal fusion for multimedia analysis: a survey.  Multimedia systems, 16(6):345–379, 2010.
  * [4] Pierre Baldi.  Autoencoders, unsupervised learning, and deep architectures.  In Proceedings of ICML workshop on unsupervised and transfer learning, pages 37–49. JMLR Workshop and Conference Proceedings, 2012.
  * [5] Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning.  A large annotated corpus for learning natural language inference, 2015\.
  * [6] Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov.  Importance weighted autoencoders.  arXiv preprint arXiv:1509.00519, 2015.
  * [7] Remi Cadene, Hedi Ben-Younes, Matthieu Cord, and Nicolas Thome.  Murel: Multimodal relational reasoning for visual question answering.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1989–1998, 2019.
  * [8] Rich Caruana.  Multitask learning.  Machine learning, 28(1):41–75, 1997.
  * [9] Hung-I Harry Chen, Yu-Chiao Chiu, Tinghe Zhang, Songyao Zhang, Yufei Huang, and Yidong Chen.  Gsae: an autoencoder with embedded gene-set nodes for genomics functional characterization.  BMC systems biology, 12(8):45–57, 2018.
  * [10] Weihua Chen, Xiaotang Chen, Jianguo Zhang, and Kaiqi Huang.  Beyond triplet loss: a deep quadruplet network for person re-identification.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 403–412, 2017.
  * [11] De Cheng, Yihong Gong, Sanping Zhou, Jinjun Wang, and Nanning Zheng.  Person re-identification by multi-channel parts-based cnn with improved triplet loss function.  In Proceedings of the iEEE conference on computer vision and pattern recognition, pages 1335–1344, 2016.
  * [12] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [13] Yueqi Duan, Wenzhao Zheng, Xudong Lin, Jiwen Lu, and Jie Zhou.  Deep adversarial metric learning.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2780–2789, 2018.
  * [14] Konrad Gadzicki, Razieh Khamsehashari, and Christoph Zetzsche.  Early vs late fusion in multimodal convolutional neural networks.  In 2020 IEEE 23rd International Conference on Information Fusion (FUSION), pages 1–6. IEEE, 2020.
  * [15] Raul Gomez, Jaume Gibert, Lluis Gomez, and Dimosthenis Karatzas.  Exploring hate speech detection in multimodal publications.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1470–1478, 2020.
  * [16] Raia Hadsell, Sumit Chopra, and Yann LeCun.  Dimensionality reduction by learning an invariant mapping.  In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06), volume 2, pages 1735–1742. IEEE, 2006.
  * [17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [18] Junlin Hu, Jiwen Lu, and Yap-Peng Tan.  Discriminative deep metric learning for face verification in the wild.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1875–1882, 2014.
  * [19] Qian Huang, Horace He, Abhay Singh, Yan Zhang, Ser Nam Lim, and Austin R Benson.  Better set representations for relational reasoning.  Advances in Neural Information Processing Systems, 33, 2020.
  * [20] Natasha Jaques, Sara Taylor, Akane Sano, and Rosalind Picard.  Multimodal autoencoder: A deep learning approach to filling in missing sensor data and enabling better mood prediction.  In 2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII), pages 202–208. IEEE, 2017.
  * [21] Mahmut Kaya and Hasan Şakir Bilge.  Deep metric learning: A survey.  Symmetry, 11(9):1066, 2019.
  * [22] Dhruv Khattar, Jaipal Singh Goud, Manish Gupta, and Vasudeva Varma.  Mvae: Multimodal variational autoencoder for fake news detection.  In The World Wide Web Conference, pages 2915–2921, 2019.
  * [23] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan.  Supervised contrastive learning.  arXiv preprint arXiv:2004.11362, 2020.
  * [24] Douwe Kiela, Suvrat Bhooshan, Hamed Firooz, and Davide Testuggine.  Supervised multimodal bitransformers for classifying images and text.  arXiv preprint arXiv:1909.02950, 2019.
  * [25] Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, and Davide Testuggine.  The hateful memes challenge: Detecting hate speech in multimodal memes, 2020.
  * [26] Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang.  Visualbert: A simple and performant baseline for vision and language.  arXiv preprint arXiv:1908.03557, 2019.
  * [27] Zechao Li and Jinhui Tang.  Weakly supervised deep metric learning for community-contributed image retrieval.  IEEE Transactions on Multimedia, 17(11):1989–1999, 2015.
  * [28] Wei Liu and Shih-Fu Chang.  Robust multi-class transductive learning with graphs.  In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 381–388. IEEE, 2009.
  * [29] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  arXiv preprint arXiv:1908.02265, 2019.
  * [30] Gjorgji Madjarov, Dragi Kocev, Dejan Gjorgjevikj, and Sašo Džeroski.  An extensive experimental comparison of methods for multi-label learning.  Pattern recognition, 45(9):3084–3104, 2012.
  * [31] Sijie Mai, Haifeng Hu, and Songlong Xing.  Modality to modality translation: An adversarial representation learning and graph fusion network for multimodal fusion.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 164–172, 2020.
  * [32] Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey.  Adversarial autoencoders.  arXiv preprint arXiv:1511.05644, 2015.
  * [33] Vicente P Minotto, Claudio R Jung, and Bowon Lee.  Simultaneous-speaker voice activity detection and localization using mid-fusion of svm and hmms.  IEEE Transactions on Multimedia, 16(4):1032–1044, 2014.
  * [34] Kevin Musgrave, Serge Belongie, and Ser-Nam Lim.  A metric learning reality check.  In European Conference on Computer Vision, pages 681–699. Springer, 2020.
  * [35] Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Y Ng.  Multimodal deep learning.  In ICML, 2011.
  * [36] Hyun Oh Song, Yu Xiang, Stefanie Jegelka, and Silvio Savarese.  Deep metric learning via lifted structured feature embedding.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4004–4012, 2016.
  * [37] Sharon Oviatt et al.  Multimodal interfaces.  The human-computer interaction handbook: Fundamentals, evolving technologies and emerging applications, 14:286–304, 2003.
  * [38] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.  Pytorch: An imperative style, high-performance deep learning library.  arXiv preprint arXiv:1912.01703, 2019.
  * [39] Juan-Manuel Pérez-Rúa, Valentin Vielzeuf, Stéphane Pateux, Moez Baccouche, and Frédéric Jurie.  Mfas: Multimodal fusion architecture search.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6966–6975, 2019.
  * [40] Bryan A. Plummer, Liwei Wang, Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazebnik.  Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models, 2016.
  * [41] Soujanya Poria, Erik Cambria, Rajiv Bajpai, and Amir Hussain.  A review of affective computing: From unimodal analysis to multimodal fusion.  Information Fusion, 37:98–125, 2017.
  * [42] Austin Reiter, Menglin Jia, Pu Yang, and Ser-Nam Lim.  Deep multi-modal sets.  arXiv preprint arXiv:2003.01607, 2020.
  * [43] Florian Schroff, Dmitry Kalenichenko, and James Philbin.  Facenet: A unified embedding for face recognition and clustering.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 815–823, 2015.
  * [44] Amanpreet Singh, Vedanuj Goswami, Vivek Natarajan, Yu Jiang, Xinlei Chen, Meet Shah, Marcus Rohrbach, Dhruv Batra, and Devi Parikh.  Mmf: A multimodal framework for vision and language research, 2020.
  * [45] Amanpreet Singh, Vedanuj Goswami, and Devi Parikh.  Are we pretraining it right? digging deeper into visio-linguistic pretraining.  arXiv preprint arXiv:2004.08744, 2020.
  * [46] Kihyuk Sohn.  Improved deep metric learning with multi-class n-pair loss objective.  In Proceedings of the 30th International Conference on Neural Information Processing Systems, pages 1857–1865, 2016.
  * [47] Valentin Vielzeuf, Alexis Lechervy, Stéphane Pateux, and Frédéric Jurie.  Centralnet: a multilayer approach for multimodal fusion.  In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, pages 0–0, 2018.
  * [48] Jian Wang, Feng Zhou, Shilei Wen, Xiao Liu, and Yuanqing Lin.  Deep metric learning with angular loss.  In Proceedings of the IEEE International Conference on Computer Vision, pages 2593–2601, 2017.
  * [49] Xun Wang, Xintong Han, Weilin Huang, Dengke Dong, and Matthew R Scott.  Multi-similarity loss with general pair weighting for deep metric learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5022–5030, 2019.
  * [50] Yikai Wang, Wenbing Huang, Fuchun Sun, Tingyang Xu, Yu Rong, and Junzhou Huang.  Deep multimodal fusion by channel exchanging.  arXiv preprint arXiv:2011.05005, 2020.
  * [51] Kilian Q Weinberger and Lawrence K Saul.  Distance metric learning for large margin nearest neighbor classification.  Journal of machine learning research, 10(2), 2009.
  * [52] Ning Xie, Farley Lai, Derek Doran, and Asim Kadav.  Visual entailment: A novel task for fine-grained image understanding, 2019\.
  * [53] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka.  How powerful are graph neural networks?  arXiv preprint arXiv:1810.00826, 2018.
  * [54] Baosheng Yu, Tongliang Liu, Mingming Gong, Changxing Ding, and Dacheng Tao.  Correcting the triplet selection bias for triplet loss.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 71–87, 2018.
  * [55] Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese.  Taskonomy: Disentangling task transfer learning.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3712–3722, 2018.
  * [56] Haigang Zhang, Shuyi Li, Yihua Shi, and Jinfeng Yang.  Graph fusion for finger multimodal biometrics.  IEEE Access, 7:28607–28615, 2019.
  * [57] Yan Zhang, Jonathon Hare, and Adam Prügel-Bennett.  Deep set prediction networks.  arXiv preprint arXiv:1906.06565, 2019.
  * [58] Yan Zhang, Jonathon Hare, and Adam Prügel-Bennett.  Fspool: Learning set representations with featurewise sort pooling.  arXiv preprint arXiv:1906.02795, 2019.
  * [59] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In Proceedings of the IEEE international conference on computer vision, pages 2223–2232, 2017.