Bath]Department of Physics, University of Bath, Bath BA2 7AY, United Kingdom
Bath]Department of Physics, University of Bath, Bath BA2 7AY, United Kingdom
Bath]Department of Physics, University of Bath, Bath BA2 7AY, United Kingdom
Warwick]Department of Physics, University of Warwick, Coventry CV4 7AL, United
Kingdom Bath]Department of Physics, University of Bath, Bath BA2 7AY, United
Kingdom Bath]Department of Physics, University of Bath, Bath BA2 7AY, United
Kingdom

# Superconducting Quantum Interference in Twisted van der Waals
Heterostructures

Liam S. Farrar<EMAIL_ADDRESS>[  Aimee Nevill  [  Zhen Jieh Lim  [
Geetha Balakrishnan  [  Sara Dale  [  Simon J. Bending  [

###### Abstract

Modern Superconducting QUantum Interference Devices (SQUIDs) are commonly
fabricated from either Al or Nb electrodes, with an in-situ oxidation process
to create a weak link between them. However, common problems of such planar
nano- and micro-SQUIDs are hysteretic current-voltage curves, and a shallow
flux modulation depth. Here, we demonstrate the formation of both Josephson
junctions and SQUIDs using a dry transfer technique to stack and
deterministically misalign flakes of NbSe2; allowing one to overcome these
issues. The Josephson dynamics of the resulting twisted NbSe2-NbSe2 junctions
are found to be sensitive to the misalignment angle of the crystallographic
axes. A single lithographic process was then implemented to shape the
Josephson junction into a SQUID geometry with typical loop areas of $\simeq$
25 $\mu m^{2}$ and weak links $\simeq$ 600 nm wide. These devices display
large stable current and voltage modulation depths of up to $\Delta
I_{c}\simeq$ 75$\%$ and $\Delta V\simeq$ 1.4 mV respectively.

Superconducting QUantum Interference Devices (SQUIDs) are key components in
the development of ultra-sensitive electric and magnetic measurement systems.1
The basic SQUID design consists of a superconducting ring intersected by one
(rf SQUID) or two (dc SQUID) Josephson junctions; the latter consisting of two
superconducting electrodes coupled by weak links that allow the flow of
supercurrent. The weak link can take a variety of forms including point
contacts, physical constrictions, or heterostructures consisting of a thin
normal metal or insulating barrier separating the two superconductors.2 The
latter type is typically fabricated by deposition of metallic superconductors
such as Al and Nb, with the tunnelling barrier formed by in-situ oxidation.3
This oxide barrier varies in thickness on the atomic scale and often contains
defect traps, which can lead to highly non-uniform supercurrent
distributions.4 In addition, over time oxygen atoms in the oxide barrier
diffuse out, altering the normal state resistance of the junction resulting in
a variation of the critical current, a process which can be detrimental to the
longevity of the device.5 Since Josephson junctions are the basis of many
superconducting technologies such as qubits,6 quantum metrology,7 and
superconducting quantum interference devices (SQUIDs), development and
incorporation of new materials with improved properties and functionality is
vital.

Figure 1:  a) Schematic of the device fabrication method. i) Au contacts are
deposited onto a Si/SiO2 substrate. ii) A single exfoliation is made of a bulk
NbSe2 crystals and the resulting flakes are transferred onto a PDMS stamp.
iii) The PDMS is brought into contact with the substrate which is tilted at a
small angle. The PDMS is slowly pressed into the substrate until the boundary
of the PDMS-substrate contact region lies beyond one of the flakes. iv) The
PDMS is then retracted, leaving the first flake deposited onto the Au
contacts. The substrate is then rotated such that the crystallographic axes of
the two NbSe2 flakes are now misaligned by an angle $\theta$. v) The second
flake is positioned above the first and the PDMS is brought into contact with
the substrate. vi) The PDMS is now retracted leaving the second flake
contacting both the first flake and the Au contacts. b) Optical micrograph of
two NbSe2 flakes on a PDMS stamp. c) Optical micrograph of the two overlapping
NbSe2 flakes on SiO2/Si. Here one of the flakes (labelled $\\#$2) has been
rotated by 60∘. d) Temperature-dependence of the normalised resistance
R(T)/R(294 K) for the two NbSe2 flakes and the overlapping junction region
between them.

An alternative route to the fabrication of oxide-free Josephson junctions is
the stacking of two-dimensional materials (2D) into vertical van der Waals
(vdW) heterostructures,8 a technique which holds promise for creating
atomically clean and defect free interfaces. Additionally, vdW-based devices
provide access to a new selection of superconducting materials such as
Bi2Sr2CaCu2O8+δ,9 FeSe,10 and 2H-NbSe2.11. These materials all share a common
characteristic in that they have a layered structure and easily cleave
perpendicular to the $c$-axis. These properties, along with the availability
of dry transfer techniques,12 has allowed the fabrication of vdW
heterostructure devices comprised of two or more mechanically exfoliated
flakes. Such devices include junctions with dielectric tunnelling barriers,13,
14 superconducting-normal-superconducting Josephson junctions,15 and van der
Waals interface Josephson junctions,16 which may prove useful in the formation
of superconducting qubits.17 Furthermore, a vdW-based device offers an
additional variable in the form of controlling the relative twist angle
between the crystallographic axes of the two materials; a degree of freedom
which is not available in conventional heterostructures. This twisting creates
a misalignment between the two crystals, changing the atomic registry at the
interface and leading to angular-dependent interlayer interactions. This
enables one to tune the electronic coupling via the twist angle, leading to
effects such as band hybridisation,18, 19 minigaps and band replicas due to
scattering on moiré potentials,20 charge transfer and changes in effective
masses 21, and has led to the discovery of a moiré superlattice in
graphene/h-BN,22 as well as electrically tunable superconductivity in twisted
bilayer graphene.23 This new field of study named twistronics, may prove
valuable in the incorporation of two-dimensional materials into more complex
vdW superconducting devices such as qubits and SQUIDs.

Figure 2:  (a) Current-voltage ($I$-$V$) characteristics of a NbSe2-NbSe2
junction (sample S1, $\theta$ $\approx$ 6∘) for temperatures between $T$ = 3 K
and 7.5 K. b) Temperature dependence of the critical current $I_{\rm c}$ (red)
and retrapping current (white). The dashed line is calculated from AB theory
(see text). c) $I$-$V$ characteristics for sample S2 ($\theta$ $\approx$ 20∘)
at $T$ = 3.75 K. d) Temperature dependence of d$I$/d$V$ versus V between $T$ =
3.75 and 6.75 K. e) Temperature dependence of the estimated superconducting
gap $\Delta$. The dashed line is calculated from an interpolation
approximation to BCS theory.

In this study, twisted NbSe2-NbSe2 heterostructures are formed by stacking two
NbSe2 flake with a well-determined misalignment of the crystallographic axes
using the procedure described in Methods and highlighted in Fig 1a. Briefly,
two NbSe2 are sequentially stacked with a small overlap onto a Si/SiO2
substrate with pre-patterned Au contacts and encapsulated with a thin layer of
hexagonal boron nitride. Typical junction areas are on the order of 20-60
$\mu$m2 (Fig. 1c).

Before describing the characterisation of devices patterned with SQUID
geometries, we investigate the transport properties of twisted NbSe2-NbSe2
junctions. The temperature-dependence of the normalised four-point resistance
R(T)/R(294K) from 4 to 294 K is shown in Fig. 1d for two NbSe2 flakes with a
relative twist angle of 60∘, as well as the resulting junction formed by the
overlap region. We note that there is an uncertainty of $\pm$1∘ in the
misalignment angle of devices arising from both the resolution of the rotation
stage and unwanted movement of the PDMS during the stamping progress. Both
flakes and junction show metallic transport behaviour which is phonon-limited
at high temperature (R $\propto$ T) and disorder limited at low temperature (R
approaches a constant value) before reaching the superconducting state at Tc
$\sim$ 7 K.24 As the $T_{c}$ of NbSe2 is known to reduce in flakes $<$ 10
layers,13, 14 all flakes used were at least 10 nm, ensuring negligible
suppression of $T_{\rm c}$. The residual resistance ratio (RRR), defined as
R(T=294K)/R(T=8K), is found to be $\sim$ 30 for the two NbSe2 flakes, with a
small variation due to the slightly different flake thicknesses. This is in
contrast to the RRR of the junction region which is found to vary between
$\sim$10-25 depending on the twist angle. A close up of the superconducting
transition is shown in Fig 1(d), where it can be seen that the junction
displays a slightly broader transition width when compared to the individual
NbSe2 flakes, indicative of additional disorder at the interface.

###  0.0.1 Josephson junction performance

Next, we examine the current-voltage (I-V) characteristics of a NbSe2-NbSe2
junction for temperatures $T$ = 3 to 7.5 K (S1, Fig. 2a). Here the
misorientation angle of the junction is estimated to be $\theta\sim$ 6∘ from
comparison of optical micrographs of the two flakes. From the presence of
distinct retrapping currents in the I-V curves, it is clear that the junction
behaviour is typical of an underdamped Josephson junction. This indicates that
the vdW interface decouples the superconducting order parameters and creates a
weak link that allows the flow of a Josephson supercurrent. Each NbSe2 flake
has multiple electrical contacts, allowing the use of different voltage lead
configurations to confirm that the bulk critical current of each flake is
substantially larger than that of the junction. Next, we quantify the junction
dynamics by examining the McCumber parameter $\beta_{c}$, found by fitting the
ratio between the retrapping current $I_{r}$ and the critical current $I_{c}$
and comparing it to a microscopic model.25 $I_{r}/I_{c}$ is found to be
$\sim$0.89 at $T$ = 3 K, leading to a McCumber parameter $\beta_{c}=$ 0.6,
placing this device in the ’weakly underdamped’ category. Using the
relationship $\beta_{c}=2eI_{c}R^{2}_{N}C\hbar^{-1}$ the junction capacitance,
$C$, was found to be 1.6x$10^{-13}$F, yielding a specific capacitance of $C/A$
= 0.8 $\mu Fcm^{-2}$. Assuming the permittivity is that of vacuum, this leads
to a gap between the two flakes of $<$1 nm, comparable to that of the $c$-axis
period in 2H-NbSe2 of 0.63 nm.

The temperature dependence of the critical current, $I_{\rm c}$, can be
calculated using Ambegaokar-Baratoff (AB) theory26, in which $I_{\rm c}(T)$ is
expressed as

$\frac{I_{\rm c}(T)}{I_{\rm
c}(0)}=\frac{\Delta(T)}{\Delta(0)}\tanh{\left[\frac{\Delta(T)}{2k_{B}T}\right]},$
(1)

where $I_{\rm c}(0)$ and $\Delta(0)$ are the zero temperature critical current
and superconducting gap respectively. $\Delta(T)$ can be described by an
interpolation approximation to the Bardeen-Cooper-Schrieffer (BCS) theory
temperature-dependent superconducting energy gap,

$\Delta(T)=\Delta(0)\tanh{2.2\sqrt{(T_{c}-T)/T)}}.$ (2)

The fit in Fig 2b shows that the critical current of sample S1 is reasonably
well described by AB theory, in agreement with previous reports on NbSe2
Josephson junctions,16 despite the assumption of a symmetric single gap
superconductor. This allows us to estimate $\Delta(0)$ = 0.90 mV and
$I_{c}(0)$ = 1.06 mA.

However, not all devices fabricated behave as underdamped junctions, and upon
examination of a second device presented in Fig 2c (S2, $\theta\sim$ 20∘), we
observe $I$-$V$ characteristics which are non hysteretic, indicative of a
junction which is in the strongly overdamped regime ($\beta_{c}\ll 1$). Here,
non-linear features arise due to quasiparticle currents that reveal
information about the superconducting density of states. We examine this in
Fig 2d by taking the numerical differential of the measured $I$-$V$ curves.
From this we observe an intricate non-BCS like gap structure, with peaks
located at $\pm$1.79 mV at $T$ = 3.75 K. Furthermore, we observe a shoulder-
like hump at lower energy ($\pm$1 mV), similar to that seen in tunnelling
spectroscopy of NbSe2 using van der Waals tunnel barriers.13, 14 Using the BCS
approximation in Eqn 2, we can extract the zero temperature superconducting
gap $\Delta(0)$. Despite sub-gap features indicative of non BCS-like
behaviour, the measured $\Delta(T)$ fits reasonably well with $\Delta(0)$ =
0.93 mV as shown in Fig 2e.

###  0.0.2 Angular dependence

Figure 3: a) Current-voltage (I-V) characteristics for twisted NbSe2-NbSe2
junctions with twist angles $\theta$ $\approx$ 0∘, b) 10∘, c) 30∘, d) 60∘. The
red and black arrows indicate the direction of the current sweep. Figure 4:
(a) Optical image of an etched SQUID structure. The insert shows a close up of
the structure, with the overlap region between the two NbSe2 flakes
highlighted in red. (b) Schematic diagram of the SQUID device shown in (a).
(c) SQUID critical current as a function of temperature in zero applied field.
(d) Current-voltage ($I$-$V$) characteristics of a NbSe2-NbSe2 SQUID at 4 K.
The red (blue) curve corresponds to the maximum (minimum) value of the
positive critical current $I_{\rm c}$ within one period. (e) Modulation of
$I_{c}$ as a function of the applied magnetic flux under a voltage bias of $V$
= 900 $\mu$V (f) and $V$ = 10 $\mu$V. (g) Voltage modulation as a function of
the magnetic flux under a current bias of $I$ = 50 $\mu$A.

Having observed strikingly different characteristics in NbSe2-NbSe2 junctions
prepared using the same fabrication method, we turn our attention to the role
of twist angle in these devices. Using the method presented in Fig 1a, we are
able to fabricate devices with a determined misalignment angle respective to
the crystallographic axes of the two flakes. The results are shown in Fig 3a-d
in which we present the current-voltage ($I$-$V$) characteristics of four
NbSe2-NbSe2 junctions with twist angles in the range $\theta$ $\approx$ 0 -
60∘. First we examine the $\theta\approx$ 0∘ device, in which hysteresis is
observed in the I-V characteristics, indicating the junction is an underdamped
Josephson junction. In samples with larger twist angles ($>$10∘, Fig 3b), this
hysteresis disappears, giving rise to reversible junctions which show features
of quasiparticle gap structure at angles $>$20∘ (Fig 3c). As the twist angle
is increased beyond about 40∘, the gap features disappear and the hysteresis
eventually returns near $\theta$ $\approx$ 60∘ (Fig 3d).

Angle resolved photo-emission spectroscopy (ARPES) data27 and density
functional theory calculations28 show that there are five electronic bands
crossing the Fermi energy in NbSe2. Of these, one is a small Se-$4p$
$p_{z}$-derived "pancake"-shaped hole pocket, while the other four are
Nb-$4d$-derived bands with roughly cylindrical Fermi surfaces centered at the
$\Gamma$ and $K$ points in the Fermi surface. Based on ARPES measurements, the
Se pancake has been shown to exhibit no superconducting gap, while the Nb-
derived sheets display superconductivity which is strongly anisotropic in
$k$.29, 30 This anisotropy of the superconducting order parameter is
characterised by maxima in the gap at 60∘ intervals around the Fermi sheets.
From this, we hypothesise that at twist angles close to 30∘, tunnelling is
dominated by processes that couple regions of the Fermi surface with maximum
gap in one layer with regions of minimum gap in the other layer. This angular-
dependent selectivity in the tunnelling process suppresses the critical
current $I_{c}$, leading to a lower value of $\beta_{c}$, and non-hysteretic
$I$-$V$ characteristics.

The observed twist dependence could also be related to the specific atomic
arrangement at the interface which depends on the flake terminations as well
as any effects arising from relative lateral displacements of the two flakes.
However, the latter are expected to be negligible in our large overlap
regions, which are many hundreds of unit cells wide. We note that we have
minimised any variation in the interface quality and homogeneity by performing
all fabrication in the inert environment provided by a nitrogen glovebox.

###  0.0.3 SQUID performance

Having established the ability to fabricate high quality non-hysteretic
Josephson junctions, we examine the ability to use them in more complex
superconducting devices such as superconducting quantum interference devices
(SQUIDs). Junction devices were patterned into a SQUID geometry using reactive
ion etching as described in Methods. An image of a typical device is shown in
Fig 4a, and consists of a superconducting loop between two $\sim$600 nm wide
Josephson junctions formed in the overlap region of the two NbSe2 flakes. The
SQUID loop is of width, $W$ $\approx$ 3.5 $\mu$m, and length, $L$ $\approx$
7.0 $\mu$m, creating an internal hole of area $\approx$ 24.5 $\mu$m2. When an
external magnetic field $B\parallel_{c}$ is applied, phase shifts are induced
between the two junctions leading to interference and a net critical current
that oscillates with a period $\Phi_{0}/A$. In practice all real SQUIDs have a
finite self-inductance that gives rise to screening currents that can often
result in undesirably low values of current modulation $\Delta
I_{c}$($B\parallel_{c}$).

To investigate this in our NbSe2-NbSe2 SQUIDs, we used electrical transport
measurements to characterise them at temperatures down to $T$ = 3.75 K. Figure
4c shows the temperature dependence of the critical current. We observe no
reduction of the critical temperature, indicating that etching caused no
deterioration in the junction quality. Fig 4d shows the current-voltage
characteristics for a NbSe2 SQUID, with the two curves corresponding to the
maximum, $I_{c}^{max}$, and minimum, $I_{c}^{min}$, measured values of the
critical current within a single oscillation period. The modulation of the
critical current, $I_{c}$($\Phi$) for two different voltage set points, is
shown in Fig 4e,f where $\Phi$ is the applied magnetic flux. Under a bias
voltage of $V$ = 900 $\mu$V the current modulation depth is 33$\%$, which
increases to 75$\%$ for a bias of $V$ = 10 $\mu$V. The noise level of the
signal increased slightly at $V$ = 10 $\mu$V due to small temperature
oscillations arising from the refrigeration cycle of the closed-cycle
cryostat. The inductance parameter characterises the amplitude of the current
modulation and is given by $\beta_{L}=2\pi LI_{c}/\Phi_{0}$, where $\Phi_{0}$
is the superconducting flux quantum and $L$ is the inductance of the SQUID
loop. Fitting our measured $\Delta I_{c}$ data to numerical models we estimate
that $\beta_{L}\approx$ 2, and $L\approx$ 10 pH, which is consistent with
other estimates of the self-inductance based on the SQUID geometry. The size
of the SQUID loop $A$ = 24.5 $\mu$m2 leads to a theoretical value of
$\Phi$/$\Phi_{0}$ $\simeq$ 84 $\mu T$, which is close to the measured value of
$\Phi/\Phi_{0}$ = 78 $\mu T$. Additionally, no decrease in the overall current
modulation, $\Delta I_{c}$, was observed up to 20 mT, the maximum output of
the electromagnetic coil.

An important property of modern SQUIDs is the realisation of non-hysteretic
$I$-$V$ characteristics ($\beta_{c}\ll$ 1), allowing the use of flux-locked
loop feedback schemes to dramatically increase the flux resolution.31, 32 We
are able to tune this desirable property in our NbSe2 SQUIDS by
deterministically controlling the twist angle during device fabrication, with
a wide range of angles over which $\beta_{c}$ is found to be sufficiently
small. Figure 4g shows the voltage under a bias current of 50 $\mu$A, as a
function of magnetic field applied to the SQUID, revealing large voltage
modulations with a depth of $\approx$ 1.4 mV. The voltage modulation depth is
characterised by $\Delta V\simeq\delta R\Delta I_{c}$, where $\Delta I_{c}$ is
the critical current modulation depth, and $\delta$R is the differential
resistance of the SQUID calculated here to be $\delta$R $\simeq$ 28 $\Omega$.
A large value of $\Delta V$ is required to achieve low flux noise SQUID
devices as it minimises the noise contribution from the input amplifier
$S^{1/2}_{V,a}$ to the white flux noise, given by
$S^{1/2}_{\Phi,a}\simeq\Phi_{0}S^{1/2}_{V,a}/\pi\Delta V$, in flux-locked
operation.33 No degradation in $T_{c}$ or $I_{c}$ of these SQUIDs was observed
after multiple cooling cycles or when remeasured after storage in a nitrogen
glovebox for 4 weeks, indicating that they are both stable and suitable for
repeated long term use.

##  0.1 Discussion

In conclusion, we have developed a method to fabricate Josephson junctions and
SQUIDs based on a twisted van der Waal heterostructure architecture. The
formation of Josephson junctions is achieved using a dry transfer method and
requires no ’wet’ lithographic steps. The result is a junction device whose
$I$-$V$ characteristics can be deterministically tuned via the twist angle. A
single lithographic process is then implemented to shape the Josephson
junction into a SQUID geometry with typical loop areas of $\simeq$ 25 $\mu
m^{2}$ and weak links $\simeq$ 600 nm wide. We obtain voltage modulation
depths of $\Delta$V $\simeq$ 1.4 mV and current modulation depths of $\Delta$I
$\simeq$ 75$\%$ from devices which display long term stability.

Our method demonstrates the ability to fully integrate 2D materials into the
design of a high-performance SQUID, paving the way for designer
superconducting circuits through incorporation of other 2D materials, each
with distinct electronic properties. The utilisation of van der Waals bonded
circuits may also have other advantages: here the devices are only anchored to
the substrate by the weak vdW force, allowing the possible pick-up and
transfer of characterised SQUIDs onto other substrates or materials, while the
intrinsically thin nature of 2D materials may allow their incorporation into
flexible electronic circuits.34 Other applications for the technology is in
the design of superconducting qubits such as those based on flux qubit
architectures.35, 36 It is well known that the performance of these is often
limited by dissipation, decoherence and noise from two-level defect systems.
The single crystalline structure of 2D flakes, along with low defect
densities, may provide circuit components with superior performance.

##  0.2 Method.

Thin flakes of NbSe2 were mechanically exfoliated from high quality single
crystals onto silicone elastomer polydimethylsiloxane (PDMS) stamps. Flakes of
suitable geometry and thickness were then sequentially transferred onto
Si/SiO2 (300 nm oxide) substrates with pre-patterned Au contacts. To ensure
crystallographic alignment of the twisted vdW junctions, one single
exfoliation of the bulk single crystal is performed which results in multiple
thin flakes with aligned crystallographic axes. From this, two neighbouring
flakes were chosen with a rotation of the substrate performed after stamping
the first flake. The resulting device is thus a twisted vdW junction with the
two flakes misaligned by a chosen angle with $\pm$1∘ accuracy. The entire dry
transfer set-up is housed in a nitrogen glovebox with an oxygen and moisture
content <1 ppm, ensuring an oxide free interface. Devices selected for
fabrication into SQUIDs were subsequently capped with a thin layer ($\sim$20
nm) of hexagonal boron nitride (h-BN) using the same dry transfer technique.
Following this, a polymethyl methacrylate e-beam resist was spin coated onto
the substrate, which was then kept under vacuum (10-6 mbar) overnight to
remove any solvent. Next, standard e-beam lithography (EBL) was used to define
the area of the junction to be etched to form the SQUID loop, before transfer
into an inductively coupled plasma etcher. The sample was then reactively ion
etched using an O2 \+ SF6 mixture, after which the sample was immersed in
acetone to remove leftover resist before storage in a nitrogen glovebox.

##  0.3 Associated Content

###  0.3.1 Supporting Information

The Supporting Information is available at xxx.

##  0.4 Author Information

###  0.4.1 Corresponding Author

*E-mail<EMAIL_ADDRESS>
###  0.4.2 Present Address

†Department of Physics, University of Bath, Bath BA2 7AY, United Kingdom

###  0.4.3 Acknowledgements

The authors thank Peter Heard at the IAC, University of Bristol, for their
useful discussions and supporting work at the early phases of this study. The
research was funded by the Bath/Bristol Centre for Doctoral Training in
Condensed Matter Physics, under the EPSRC (UK) Grant No. EP/L015544. The work
at the University of Warwick is supported through EPSRC Grant EP/T005963/1.
A.N, Z.J.L, and S.D acknowledge financial support from the Royal Society.

###  0.4.4 Author contributions

L.F fabricated the devices under the supervision of S.B, with support from
A.N. and Z.J.L under the supervision of S.D. G.B grew the single crystals. L.F
performed the measurements. L.F and S.B wrote the paper with comments and
discussion from all authors.

## References

  * 1 Fagaly, R.L.  Superconducting quantum interference device instruments and applications.  _Rev. Sci. Instrum._ 2006, 77, 101101\.
  * 2 Likharev, K.K.  Superconducting weak links.  _Rev. Mod. Phys._ 1979, 51, 101-159.
  * 3 Gurvitch, M.; Washington, M.A.; Huggins, H.A.  High quality refractory Josephson tunnel junctions utilizing thin aluminum layers.  _Appl. Phys. Lett._ 1983, 42, 472-474.
  * 4 Zeng, L.J.; Nik, S.; Greibe, T.; Krantz, P.; Wilson, C.M.; Delsing, P.; Olsson, E.  Direct observation of the thickness distribution of ultra thin AlOx barriers in Al/AlOx/Al Josephson junctions.  _J. Phys. D Appl. Phys._ 2015, 48, 395308\.
  * 5 Pop, I.M.; Fournier, T.; Crozes, T.; Lecocq, F.; Matei, I.; Pannetier, B.; Buisson, O.; Guichard, W.  Fabrication of stable and reproducible submicron tunnel junctions.  _J. Vac. Sci. Technol._ 2012, 30, 010607\.
  * 6 Kjaergaard, M.; Schwartz, M.E.; Braumüller, J.; Krantz, P.; Wang, J.I.J.; Gustavsson, S.; Oliver, W.D.  Superconducting qubits: Current state of play.  _Annu. Rev. Condens. Matter Phys._ 2020, 11, 369-395.
  * 7 Scherer, H.; Camarota, B.  Quantum metrology triangle experiments: a status review.  _Meas. Sci. Technol._ 2012, 23, 124010\.
  * 8 Wang, L.; Meric, I.; Huang, P.Y.; Gao, Q.; Gao, Y.; Tran, H.; Taniguchi, T.; Watanabe, K.  One-dimensional electrical contact to a two-dimensional material.  _Science_ 2013, 342, 614-617.
  * 9 Yu, Y.; Ma, L.; Cai, P.; Zhong, R.; Ye C.; Shen J.; Gu, G.D.; Chen, X.H.; Zhang, Y.  High-temperature superconductivity in monolayer Bi2 Sr2 CaCu2 O8+δ.  _Nature_ 2019, 575, 156-163.
  * 10 Farrar, L.S.; Bristow, M.; Haghighirad, A.A.; McCollam, A.; Bending, S.J.; Coldea, A.I.  Suppression of superconductivity and enhanced critical field anisotropy in thin flakes of FeSe.  _npj Quantum Mater._ 2020, 5, 1-7.
  * 11 Xi, X.; Wang, Z.; Zhao, W.; Park, J.H.; Law, K.T.; Berger, H.; Forró, L.; Shan, J.; Mak, K.F.  Ising pairing in superconducting NbSe2 atomic layers.  _Nat. Phys._ 2016, 12, 139-143.
  * 12 Castellanos-Gomez, A.; Buscema, M.; Molenaar, R.; Singh, V.; Janssen, L.; Van Der Zant, H.S.; Steele, G.A.  Deterministic transfer of two-dimensional materials by all-dry viscoelastic stamping.  _2D Mater._ 2014, 1, 011002\.
  * 13 Khestanova, E.; Birkbeck, J.; Zhu, M.; Cao, Y.; Yu, G.L.; Ghazaryan, D.; Yin, J.; Berger, H., Forro, L.; Taniguchi, T.; Watanabe, K.  Unusual Suppression of the Superconducting Energy Gap and Critical Temperature in Atomically Thin NbSe2.  _Nano Lett._ 2018, 18, 2623-2629.
  * 14 Dvir, T.; Massee, F.; Attias, L.; Khodas, M.; Aprili, M.; Quay, C.H.; Steinberg, H.  Spectroscopy of bulk and few-layer superconducting NbSe2 with van der Waals tunnel junctions.  _Nat. Commun._ 2018, 9, 1-6.
  * 15 Kim, M.; Park, G.H.; Lee, J.; Lee, J.H.; Park, J.; Lee, H.; Lee, G.H.; Lee, H.J.  Strong proximity Josephson coupling in vertically stacked NbSe2–graphene–NbSe2 van der Waals junctions.  _Nano Lett._ 2017, 10, 6125-6130.
  * 16 Yabuki, N.; Moriya, R.; Arai, M.; Sata, Y.; Morikawa, S.; Masubuchi, S.; Machida, T.  Supercurrent in van der Waals Josephson junction.  _Nat. Commun._ 2016, 7, 1-5.
  * 17 Liu, X; Hersam, M.C.  2D materials for quantum information science.  _Nat. Rev. Mater._ 2019, 10, 669-684.
  * 18 Ohta, T.; Robinson, J.T.; Feibelman, P.J.; Bostwick, A.; Rotenberg, E.; Beechem, T.E.  Evidence for interlayer coupling and moiré periodic potentials in twisted bilayer graphene.  _Phys. Rev. Lett._ 2012, 109, 186807\.
  * 19 Thompson, J.J.P.; Pei, D.; Peng, H.; Wang, H.; Channa, N.; Peng, H.L.; Barinov, A., Schröter, N.B.M.; Chen, Y.; Mucha-Kruczyński, M.  Determination of interatomic coupling between two-dimensional crystals using angle-resolved photoemission spectroscopy.  _Nat. Commun._ 2020, 11, 1-7.
  * 20 Pierucci, D.; Henck, H.; Avila, J.; Balan, A.; Naylor, C.H.; Patriarche, G.; Dappe, Y.J.; Silly, M.G.; Sirotti, F.; Johnson, A.C.; Asensio, M.C.  Band alignment and minigaps in monolayer MoS2-graphene van der Waals heterostructures. Nano letters.  _Nano Lett._ 2016, 16, 4054-4061.
  * 21 Yeh, P.C.; Jin, W.; Zaki, N.; Kunstmann, J.; Chenet, D.; Arefe, G.; Sadowski, J.T.; Dadap, J.I.; Sutter, P.; Hone, J.; Osgood Jr, R.M.  Direct measurement of the tunable electronic structure of bilayer MoS2 by interlayer twist..  _Nano Lett._ 2016, 16, 953-959.
  * 22 Dean, C.R.; Wang, L.; Maher, P.; Forsythe, C.; Ghahari, F.; Gao, Y.; Katoch, J.  Hofstadter’s butterfly and the fractal quantum Hall effect in moiré superlattices.  _Nature_ 2013, 497, 598-602.
  * 23 Cao, Y.; Fatemi, V.; Fang, S.; Watanabe, K.; Taniguchi, T.; Kaxiras, E.; Jarillo-Herrero, P.  Unconventional superconductivity in magic-angle graphene superlattices.  _Nature_ 2018, 556, 43-50.
  * 24 Naito, M.; Tanaka, S.  Electrical Transport Properties in 2H-NbS2,-NbSe2,-TaS2 and-TaSe2.  _J. Phys. Soc. Japan_ 1982, 51, 219-227.
  * 25 Likharev, K. K.  Dyanmics of Josephson Junctions and Circuits.  _Gordon and Breach_ 1986.
  * 26 Barone, A.; Paterno, G.  Physics and applications of the Josephson effect.  _Wiley_ 1982.
  * 27 Yokoya, T.; Kiss, T.; Chainani, A.; Shin, S.; Nohara, M.; Takagi, H.  Fermi surface sheet-dependent superconductivity in 2H-NbSe2.  _Science_ 2001, 294, 2518-2520.
  * 28 Johannes, M.D.; Mazin, I.I.; Howells, C.A.  Fermi-surface nesting and the origin of the charge-density wave in NbSe2..  _Phys. Rev. B_ 2006, 73, 205102\.
  * 29 Borisenko, S.V.; Kordyuk, A.A.; Zabolotnyy, V.B.; Inosov, D.S.; Evtushinsky, D.; Büchner, B.; Yaresko, A.N.; Varykhalov, A.; Follath, R.; Eberhardt, W.; Patthey, L.  Two energy gaps and Fermi-surface “arcs” in NbSe2.  _Phys. Rev. Lett._ 2009, 102, 166402\.
  * 30 Rahn, D.J.; Hellmann, S.; Kalläne, M.; Sohrt, C.; Kim, T.K.; Kipp, L.; Rossnagel, K.  Gaps and kinks in the electronic structure of the superconductor 2H-NbSe2 from angle-resolved photoemission at 1 K.  _Phys. Rev. B_ 2012, 85, 224532\.
  * 31 Kleiner, R.; Koelle, D.; Ludwig, F.; Clarke, J.  Superconducting quantum interference devices: State of the art and applications..  _Proc. IEEE_ 2004, 92, 1534-1548.
  * 32 Martínez-Pérez, M.J.; Koelle  NanoSQUIDs: Basics $\&$ recent advances.  _Phys. Sci. Rev._ 2017, 2(8), 1-27.
  * 33 Trabaldo, E.; Pfeiffer, C.; Andersson, E.; Arpaia, R.; Kalaboukhov, A.; Winkler, D.; Lombardi, F.; Bauch, T.  Grooved Dayem nanobridges as building blocks of high-performance YBa2Cu3O7-δ SQUID magnetometers.  _Nano Lett._ 2009, 19, 1902-1907.
  * 34 Kim, S.J.; Choi, K.; Lee, B.; Kim, Y.; Hong, B.H.  Materials for flexible, stretchable electronics: graphene and 2D materials.  _Annu. Rev. Mater. Res._ 2015, 45, 63-84.
  * 35 Chiorescu, I.; Bertet, P.; Semba, K.; Nakamura, Y.; Harmans, C.J.P.M.; Mooij, J.E.  Coherent dynamics of a flux qubit coupled to a harmonic oscillator..  _Nature_ 2004, 431, 159-162.
  * 36 Yan, F.; Gustavsson, S.; Kamal, A.; Birenbaum, J.; Sears, A.P.; Hover, D.; Gudmundsen, T.J.; Rosenberg, D.; Samach, G.; Weber, S.; Yoder, J.L.  The flux qubit revisited to enhance coherence and reproducibility.  _Nat. Commun._ 2016, 7, 1-9.


# Rank4Class: A Ranking Formulation for Multiclass Classification

Aeiau Zzzz  Bauiu C. Yyyy  Cieua Vvvvv  Iaesut Saoeu  Fiuea Rrrr  Tateu H.
Yasehe  Aaoeu Iasoh  Buiui Eueu  Aeuia Zzzz  Bieea C. Yyyy  Teoau Xxxx  Eee
Pppp

###### Abstract

Multiclass classification (MCC) is a fundamental machine learning problem of
classifying each instance into one of a predefined set of classes. Given an
instance, an MCC model computes a score for each class, all of which are used
to sort the classes. The performance of a model is usually measured by Top-$K$
Accuracy/Error (e.g. $K=1$ or $5$). In this paper, we do not aim to propose
new neural network architectures as most recent works do, but to show that it
is promising to boost MCC performance with a novel formulation through the
lens of ranking. In particular, by viewing MCC as _an instance class ranking
problem_ , we first argue that ranking metrics, such as Normalized Discounted
Cumulative Gain, can be more informative than the existing Top-$K$ metrics. We
further demonstrate that the dominant neural MCC recipe can be transformed to
a _neural ranking pipeline_. Based on such generalization, we show that it is
intuitive to leverage techniques from the rich information retrieval
literature to improve the MCC performance out of the box. Extensive empirical
results on both text and image classification tasks with diverse datasets and
backbone neural models show the value of our proposed framework.

Machine Learning, ICML

##  1 Introduction

Multiclass classification (MCC) is the problem of classifying each instance
into one of _a predefined set of classes_ (hastie01statisticallearning;
bishop2016pattern). It is one of the most fundamental machine learning
problems that has broad applications in many fields such as natural language
processing (sun2020finetune) and computer vision (he2016resnet). For example,
deciding the category of a news article (minaee2021deep) or the subject of an
image (krizhevsky2012imagenet) is formulated as an MCC problem.

Numerous MCC models have been proposed in the past, ranging from the linear
models to nonlinear decision trees and neural models (aly05survey). In the
modern deep learning era, while there are significant advances in neural
network architectures, dominant MCC methods share the same recipe: an input
instance, being it a feature vector, a sentence, or an image, is fed into a
neural model and scored against a predefined set of classes. The model is
trained by using a loss function, typically the _softmax cross entropy loss_ ,
between the labels and scores over all candidate classes (goodfellow2016deep).
During inference, the classes are sorted after an instance is scored against
them. Metrics such as Top-$K$ Accuracy (the percentage of test instances whose
correct class label is in the top $K$ predicted classes, also known as simply
“classification accuracy” when $K=1$) are usually used for evaluating MCC
performance. To better display the models’ headroom, Top-$K$ Error ($1-$
Top-$K$ Accuracy) is widely adopted to compare different models
(krizhevsky2012imagenet). Following this recipe, most efforts in the
literature focus on designing more powerful neural network architectures for
representation learning to improve MCC performance (he2016resnet; densenet;
dosovitskiy2020vit; minaee2021deep). Few works have studied a formulation
different from the recipe above.

Different from existing work, we propose a novel formulation for MCC by
examining it _through the lens of ranking_ , or more specifically, learning to
rank (LTR), a rich research field stemming from information retrieval (IR)
(liu2009learning). As shown in the rest of the paper, firstly, such a
formulation allows us to better evaluate MCC performance by applying more
informative ranking metrics. Secondly, it improves MCC performance out of the
box, agnostic to the representation learning architecture used. This is
achieved by utilizing advanced ranking losses, and more flexible matching
schemes between the input instance and candidate classes.

In this paper, we first formalize MCC as a ranking problem. As hinted above,
dominant neural MCC models score a given instance on a predefined set of
classes and sort them. This is equivalent to a LTR setting where a set of
items (i.e., classes) are scored and ranked given a query (i.e., the input
instance). In fact, a Top-$K$ Accuracy measure itself can be viewed as a
ranking metric, similar to Precision@$K$ in the IR literature. However, other
ranking metrics such as Normalized Discounted Cumulative Gain are more
commonly used in IR because they are more informative than Precision@$K$ in
real-world user-facing applications. They can be borrowed to enrich the MCC
evaluation metrics, but have not been adopted widely. We provide an in-depth
discussion of ranking metrics for better MCC evaluation, which also motivates
our modeling efforts in the new formulation.

For modeling, we show a general _“equivalent view”_ for MCC from a ranking
perspective, where the dominant MCC model architecture is equivalent to a
neural ranking architecture with _a specific set of design choices_. This view
gives us insights into existing MCC models’ groundings as well as limitations,
which in turn inspires more design options. We further propose several
intuitive approaches under the LTR formulation to improve MCC performance from
two aspects: loss functions and model architectures. For loss functions, we
leverage the rich LTR study of advanced ranking losses specifically designed
for certain metrics. For model architecture, we realize that the vast MCC
literature focuses on only one component in the equivalent ranking
architecture, i.e., the input instance embedding. With the LTR view, we can
enhance the modeling capacity of other components as well. In this paper, we
especially study the effect of enhancing the interactions between instances
and classes, a popular setup in LTR to match queries and items
(li2014semantic).

We report experimental results on a variety of MCC tasks, including different
datasets and backbone models for different modalities. Examples include text
classification on GoEmotions (demszky2020goemotions) with BERT (bert) and
image classification on ImageNet (krizhevsky2012imagenet) with ResNet
(he2016resnet). Results show that the proposed methods outperform or perform
competitively with existing MCC models in all settings. We expect that the
promising results can encourage the community to further examine MCC from LTR
perspectives.

By exploring the connection between LTR and MCC, our contributions can be
summarized as follows.

  * •

We prove that ranking metrics are more informative than the Top-$K$
Accuracy/Error for MCC evaluation and propose to use them additionally
(Section 2).

  * •

We show mathematically that dominant neural MCC architecture is equivalent to
a specific ranking architecture with several limitations (Section 3).

  * •

Under the LTR formulation, we show that it is natural to adopt advanced
ranking losses and richer instance-class interactions for MCC (Section 4).

  * •

We conduct experiments on a wide range of MCC tasks to validate the
effectiveness and generality of our LTR formulation for MCC (Section 5).

##  2 Ranking Metrics for MCC

###  2.1 Metrics for Classification

The basic binary classification problem classifies instances into positive and
negative classes. MCC extends binary classification to more than two classes.
Going from binary to multiclass is not trivial for evaluation. Binary
classification metrics are usually _class-oriented_. For example, metrics such
as AUC and Accuracy are based on measures like true positives (TP) and false
negatives (FN), which are computed with respect to positive and negative
classes (sokolova2009metric). These metrics are not directly used in MCC for
more than two classes. In contrast, MCC metrics are usually _instance-
oriented_. The commonly used metrics are the Top-$K$ Accuracy/Error metrics,
popularized by the ImageNet competition (ILSVRC15). Some earlier works like
(crammer2002svm) defined the “empirical error” when working on multiclass SVM
algorithms, which is equivalent to Top-1 Error.

###  2.2 Relation to Ranking Metrics

Figure 1: We show an simulated example of evaluating image classification
performance of two models with Top-1/5 Accuracy and NDCG@5. The Top-1/5
Accuracy of two models are exactly the same, and can not differentiate the two
models. While NDCG@5 can successfully tell that model A has a better
performance than model B.

For easier illustration, we use Top-$K$ Accuracy in the rest of this section,
which is simply $1-$ Top-$K$ Error. As discussed above, MCC metrics are
instance-oriented, and the metric value is simply averaged over all instances
in a given evaluation set. Thus we study the metric calculated on a single
instance to be more concise. In particular, given an instance and $n$
candidate classes, let $\mathbf{y}$ be the true labels of the classes and
$\mathbf{y}[c]\in\\{0,1\\}$ be the label for class $c$. We consider the
standard MCC setting where there is exactly one class with label $1$ per
instance (aly05survey; bishop2016pattern). Let $\pi_{A}$ be a ranking of
classes produced by model A, with the i-th most likely class being
$\pi_{A}(i)$. Then the Top-$K$ Accuracy ($1\leq K\leq n$, abbreviated as
acc@$K$ in Eq 1) is

$\textrm{acc@$K$}(\pi_{A},\mathbf{y})=\sum_{i=1}^{K}\mathbf{y}[\pi_{A}(i)].$
(1)

The Top-$K$ Accuracy can be thought of a type of a ranking metric. It is very
close to the Precision@$K$: Top-$K$ Accuracy is the same as
$\min(1,K\cdot\textrm{Precision@$K$})$ and Top-1 Accuracy is the same as
Precision@1. Besides Precision@$K$, there are other commonly used ranking
metrics such as Normalized Discounted Cumulative Gains (NDCG) and Mean
Reciprocal Rank (MRR) (ndcg2002paper). To the best of our knowledge, they are
not commonly used for MCC evaluation, but can be more informative.

We use NDCG@$K$ (simply called NDCG when $K=n$) as an example. Consider the
instance as a query, and all the classes as candidate items with relevance
labels $\mathbf{y}$, we have

$\displaystyle\textrm{NDCG@$K$}(\pi_{A},\mathbf{y})=\frac{\textrm{DCG@$K$}(\pi_{A},\mathbf{y})}{\textrm{DCG@$K$}(\pi_{*},\mathbf{y})},$
(2)
$\displaystyle\textrm{and}\;\textrm{DCG@$K$}(\pi,\mathbf{y})=\sum_{i=1}^{K}\frac{2^{\mathbf{y}[\pi(i)]}-1}{\log_{2}(1+i)}.$

where $\pi_{*}$ is the ideal ranking sorted by $\mathbf{y}$. It’s easy to
prove that NDCG@1 is equivalent to Top-1 Accuracy.

######  Theorem 1.

Given an instance with class labels $\mathbf{y}\in\\{0,1\\}^{n}$ and
$\sum_{c=1}^{n}\mathbf{y}[c]=1$, for any $1<K\leq n$, NDCG@$K$ preserves more
information than Top-$K$ Accuracy in evaluating the class rankings.

The proof of Theorem 1 is in Appendix A. The basic idea is to measure the
information of a metric by the entropy of its value evaluated on any class
ranking as a random variable. This information measure indicates that, given
the metric’s value, how much can we say about its input class ranking.

Theorem 1 shows that for $K>1$, NDCG@$K$ is strictly more informative than
Top-$K$ Accuracy. To better illustrate this, let us consider two models A and
B, which rank the correct class at position $p_{A}$ and $p_{B}$, respectively,
and $p_{A}<p_{B}\leq K,2\leq K\leq n$. Thus, both models can rank the correct
class in the top $K$ positions while model A ranks it higher than model B.
With Top-$K$ Accuracy, we have
$\textrm{acc@$K$}(\pi_{A},\mathbf{y})=\textrm{acc@$K$}(\pi_{B},\mathbf{y})=1$.
But with NDCG@$K$, we have
$\textrm{NDCG@$K$}(\pi_{A},\mathbf{y})=1/\log_{2}(1+p_{A})$, and
$\textrm{NDCG@$K$}(\pi_{B},\mathbf{y})=1/\log_{2}(1+p_{B})$, which gives us
$\textrm{NDCG@$K$}(\pi_{A},\mathbf{y})>\textrm{NDCG@$K$}(\pi_{B},\mathbf{y})$.
Therefore, when model A consistently ranks the correct class higher than model
B in top $K$ positions, Top-$K$ Accuracy may not be able to reflect the better
performance of A, while NDCG@$K$ will always detect it. More importantly,
although both metrics ignore the correct classes ranked below the $K$’th
position, we can simply set $K=n$ to evaluate the whole ranked list of
classes. In this case, NDCG is still a meaningful metric that reflects the
position of the correct class in the full list, while Top-$K$ Accuracy becomes
meaningless (always $1$).

In Figure 1, we simulate an image MCC application in a _user-facing interface_
and compare the results from two models with different metrics. NDCG@5 is
clearly more capable of distinguishing the two models, while both Top-1
Accuracy and Top-5 fails to. The reason is that there is a position
discounting function in NDCG@$K$, while Top-$K$ metrics impose a simple hard
cut at position $K$, which loses information about the exact ranked position
of the correct class. Other ranking metrics such as MRR share similar
characteristics with NDCG. In the rest of this paper, we choose NDCG@$K$ as
our representatives for ranking metrics and use it in addition to Top-$K$
metrics in experiments.

##  3 MCC from a Ranking Perspective

###  3.1 Classical MCC Model Architectures

The general model architecture for MCC based on deep neural networks (DNN) is
composed of three parts: an input instance to be classified, an encoder to
extract latent representation (embedding) of the input, and a classification
layer for generating scores on candidate classes, as shown in Figure 2 Left.
We use $\mathbf{x}$ to represent the input instance such as a textual sentence
or an image. The encoder can have different structure designs based on the
modality of the input, such as transformers (vaswani2017attention) to encode
textual sentences; and convolutional neural networks (krizhevsky2012imagenet)
to encode images. It can be represented by a function $\mathcal{H}(\cdot)$
that maps $\mathbf{x}$ to a $d$-dimensional embedding vector
$\mathbf{h}=\mathcal{H}(\mathbf{x})$. The classification layer is in most
cases a dense layer with weight matrix $W\in\mathbb{R}^{n\times(d+1)}$. The
classification scores are calculated by $\mathbf{s}=W\mathbf{h}^{\prime}$,
where $\mathbf{h}^{\prime}\coloneqq[\mathbf{h},1]$ with an added bias
dimension of value $1$ for the bias. $\mathbf{s}$ is an $n$-dimensional score
vector for $n$ classes with
$s_{i}={\mathbf{e}}_{i}^{\top}W\mathbf{h}^{\prime}$ for the $i$-th class,
where ${\mathbf{e}}_{i}$ is a $n$-dimensional one-hot vector with the $i$-th
dimension being $1$.

For training neural MCC models, the softmax cross entropy loss (SoftmaxCE) is
used by default in almost all prior work (goodfellow2016deep; zhang2021dive).
However, whether it is the most suitable loss for optimizing the evaluation
metrics of interest is not carefully studied. Very recently, empirical results
show that the mean squared error (MSE) can sometimes outperform SoftmaxCE in
MCC tasks (hui2021evaluation), but it is sensitive to an extra rescaling
parameter for tasks with many classes.

###  3.2 An Equivalent View from Neural Ranking Models

Figure 2:  Left: the classical DNN architecture for MCC; Middle: the
equivalent neural ranking architecture; Right: The ranking generalization for
MCC which allows the exploration of different class encoders, interaction
layers, and loss functions.

LTR learns a ranking model to rank a set of items (e.g., documents, news,
etc.) based on their relevance to a query. Neural ranking models (guo2019deep)
adopt DNNs to match the query and items with their latent representations.

######  Claim 1.

The classical DNN model for MCC as described in Section 3.1 is equivalent to a
specific neural ranking model trained with softmax cross entropy loss.

To prove Claim 1, we show the equivalent view of a classical MCC model as a
neural ranking model in Figure 2 Middle. In particular, we treat the input
instance $\mathbf{x}$ as the query, and the $n$ candidate classes as input
items to be ranked by the neural ranking model. The same as in the classical
MCC model, we apply the encoder $\mathcal{H}(\cdot)$ to $\mathbf{x}$ to get a
$d$-dimensional embedding $\mathbf{h}=\mathcal{H}(\mathbf{x})$, and
$\mathbf{h}^{\prime}=[\mathbf{h},1]$. For representing the classes, we use
one-hot vector $\mathbf{e}_{i}$ for the $i$-th class and obtain its embedding
$\mathbf{c}_{i}=W^{\top}\mathbf{e}_{i}$, where $W$ is the weight matrix from
the classification layer of the classical MCC model. Finally, the ranking
score of the $i$-th class can be calculated by simple _dot product_
$s^{\prime}_{i}\equiv\mathbf{c}_{i}^{\top}\mathbf{h}^{\prime}={\mathbf{e}}_{i}^{\top}W\mathbf{h}^{\prime}=s_{i}$.
In this way, we show the classical MCC model can be transformed to an
equivalent neural ranking model when both are trained by the SoftmaxCE.

##  4 Ranking Architectures for MCC

In this section, we discuss the possible new designs of different components
in the equivalent ranking view of MCC models. We call the resulting new
framework ranking for multiclass classification, or _Rank4Class_.

As discussed in Section 1, existing work mostly focus on representation
learning in the instance encoder. From the ranking perspective, however, we
can see that there are several other promising directions to improve MCC
performance. First, existing work mainly use SoftmaxCE, while there exist many
advanced ranking losses that can be leveraged. Second, the interaction between
the instance and class embeddings is simply a dot product, while richer
interaction patterns can be explored. Third, while existing work focus on
instance encoder, the class encoder is a naive linear projection that uses
one-hot encoding of classes as input, where more powerful class encoders with
richer inputs (e.g., class metadata) can be deployed. We illustrate the
potential improvements in Figure 2 Right. In this paper, we focus on ranking
losses and interaction patterns for improving MCC performance, and leave the
rest for future work.

###  4.1 Ranking Losses for MCC

Although SoftmaxCE is widely adopted for training MCC models, whether it is
the best option to optimize existing evaluation metrics is not clear
(hui2021evaluation). On the other hand, we have shown that ranking metrics are
better for evaluating MCC tasks. Thus, ranking losses are natural
considerations since many of them are theoretically grounded to directly
optimize certain ranking metrics. Next, we first discuss the soundness of
using SoftmaxCE in MCC tasks with respect to ranking metrics. Then we present
two example ranking losses, i.e., the pairwise logistic loss (PairLogLoss) and
the approximate NDCG loss (ApproxNDCG).

We re-denote the label of the $i$-th class for an instance by
$y_{i}\in\\{0,1\\},i\in\\{1,\dots,n\\}$, where only the correct class has
label $1$. To compute the SoftmaxCE, the classification scores
$s_{i},i\in\\{1,\dots,n\\}$ produced by the model are first projected to the
probability simplex $\mathbf{p}$ by softmax activation as
$p_{i}=\frac{e^{s_{i}}}{\sum_{j=1}^{n}e^{s_{j}}}$. Then SoftmaxCE is defined
as

$\ell_{ce}(\mathbf{y},\mathbf{p})=-\sum_{i=1}^{n}y_{i}\log p_{i}.$

Intuitively, the SoftmaxCE is promoting the correct class against all other
classes since only the term with $y_{i}=1$ is counted in it. It can be viewed
as a listwise ranking loss, which aims to rank the correct class above all
other classes.

######  Theorem 2 (bruch2019analysis).

Softmax cross entropy loss is a bound on mean reciprocal rank and mean
normalized discounted cumulative gain in log-scale.

The proof of Theorem 2 can be found in bruch2019analysis (Theorems 1-3). It
explains the promising performance of SoftmaxCE in MCC tasks. To the best of
our knowledge, there is no such bounds for MSE, so the findings in
(hui2021evaluation) may need further theoretical investigation.

On the other hand, there is rich literature of developing ranking losses for
optimizing ranking metrics (burges2005learning; burges2006learning;
burges2010ranknet). Such ranking losses are usually directly derived from the
target ranking metric, and bound the metric by approximation techniques
(qin2010general). One of the most historical and popularly used ranking losses
is the PairLogLoss.

$\ell_{pl}(\mathbf{y},\mathbf{s})=\sum_{i=1}^{n}\sum_{j=1}^{n}\mathbb{I}_{y_{i}>y_{j}}\log(1+e^{-\sigma(s_{i}-s_{j})}),$
(3)

where $\mathbb{I}$ is the indicator and $\sigma$ is a hyper-parameter. The
PairLogLoss is proved to be able to minimize the rank of the relevant item
(wang2018lambdaloss).

Another popular ranking loss is the ApproxNDCG (qin2010general), which
directly optimizes the NDCG metric in Eq 2. The rank of an item $i$ can be
computed as $\pi_{s}(i)=1+\sum_{j\neq i}\mathbb{I}_{s_{i}<s_{j}}$, where the
indicator $\mathbb{I}_{s<t}$ is discrete but can be approximated by a sigmoid
function to be smooth:

$\mathbb{I}_{s<t}=\mathbb{I}_{t-s>0}\approx\frac{1}{1+e^{-\alpha(t-s)}},$ (4)

where $\alpha>0$ is a parameter to control how tightly the indicator is
approximated.

To summarize, SoftmaxCE can be viewed as a ranking loss that bounds specific
ranking metrics. Other ranking losses can also be valuable. We further
investigate the empirical use of different ranking losses for MCC in Section
5.

###  4.2 Enhancing Instance-Class Interactions

After obtaining the embeddings $\mathbf{h}$ for the input instance and
$\mathbf{c}_{i}$ for class $i$, it is also important to design the matching
mechnism for producing the score $s_{i}$ between the embeddings of the
instance and the class. For this purpose, we can add different interactions
between $\mathbf{h}$ and $\mathbf{c}_{i}$ rather than simple dot-product for
producing the score, $s_{i}=\mathcal{I}(\mathbf{h},\mathbf{c}_{i})$.
$\mathcal{I}(\cdot,\cdot)$ is a function to represent the interaction between
instance and class embeddings which produces a ranking score. In this paper,
we consider the following two patterns as examples to enhance the interaction,
but much richer interactions can be deployed based on the LTR formulation:

  * •

LC+MLP: We first apply element-wise multiplication, also known as the latent
cross (LC) operation (beutel2018latentcross) on $\mathbf{h},\mathbf{c}_{i}$,
and then follow with a multilayer perceptron (MLP) to get the score $s_{i}$.
LC has shown to be a simple and efficient way to generate higher-order
matching interactions in DNNs.

  * •

Concat+MLP: We first concatenate the two embeddings
$\mathbf{h},\mathbf{c}_{i}$ and follow with an MLP to compute $s_{i}$.

Table 1: Statistics of datasets used in experiments. Dataset | #classes | Train | Validation | Test
---|---|---|---|---
GoEmotions | 28 | 36.3K | 4.5K | 4.6K
MIND | 18 | 78.8K | 25.7K | 25.9K
ImageNet | 1000 | 1.28M | 50K | -
CIFAR-10 | 10 | 50K | - | 10K
Table 2: Results from classical MCC models and Rank4Class evaluated by Top-1/5 Error and NDCG@5. Bold font indicates the best value in each row. Relative Improvement means relative reduction in Top-1/5 Error and relative increase in NDCG@5. Dataset | Encoder | Metrics | Classical MCC | Rank4Class | Relative Improvement
---|---|---|---|---|---
GoEmotions | BERT | Top-1 Error | 41.00 | 40.65 | 0.85%
Top-5 Error | 13.35 | 11.85 | 11.24%
NDCG@5 | 73.90 | 74.62 | 0.97%
ELECTRA | Top-1 Error | 38.45 | 37.36 | 2.83%
Top-5 Error | 10.45 | 8.65 | 17.22%
NDCG@5 | 76.96 | 78.30 | 1.74%
MIND | BERT | Top-1 Error | 30.90 | 30.20 | 2.27%
Top-5 Error | 6.15 | 5.05 | 17.89%
NDCG@5 | 82.78 | 83.56 | 0.94%
ELECTRA | Top-1 Error | 27.09 | 26.40 | 2.55%
Top-5 Error | 4.25 | 3.70 | 12.94%
NDCG@5 | 85.73 | 86.13 | 0.47%
ImageNet | ResNet50 | Top-1 Error | 23.74 | 23.58 | 0.67%
Top-5 Error | 6.90 | 6.75 | 2.17%
NDCG@5 | 85.74 | 85.85 | 0.13%
CIFAR-10 | VGG16 | Top-1 Error | 6.56 | 6.40 | 2.44%
Top-5 Error | 0.20 | 0.15 | 25.00%
NDCG@5 | 97.19 | 97.23 | 0.04%

##  5 Experiments

We study Rank4Class on both text classification and image classification tasks
in comparison to classical MCC models. We consider several datasets and
instance encoders for evaluation. The datasets are summarized in Table 1. For
text classification, we include two recent large-scale datasets, GoEmotions
(demszky2020goemotions) and MIND (wu2020mind). The GoEmotions dataset contains
instances with multiple labels, which are filtered out, since we focus on
single-label MCC tasks. We adopt ELECTRA (clark2020electra) and BERT (bert) as
text encoders in the experiments. For image classification, we use the popular
ImageNet (ILSVRC15) and CIFAR-10 (krizhevsky2009cifar10). We adopt ResNet50
(he2016resnet) as image encoder for ImageNet and VGG16 (simonyan2014very) for
CIFAR-10. More details of the datasets and instance encoders are given in
Appendix B.

For text datasets with both validation and test sets provided, we tune hyper-
parameters on the validation set and report results on the test set. For
ImageNet/CIFAR-10, we tune hyper-parameters and report the performance on the
validation/test set respectively, which is the norm in the literature. More
details on experimental settings such as data processing and hyper-parameter
tuning are included in Appendix C. We use Top-1 Error (equivalent to $1-$
NDCG@1, lower the better), Top-5 Error, and NDCG@5 (higher the better) for
evaluation. All result points are multiplied by $100$ for better illustration
as commonly done in the literature.

In Section 5.1, we summarize the overall MCC performance of Rank4Class with
respect to different configurations of loss functions and interaction
patterns. In Section 5.2, we study different ranking losses in optimizing MCC
tasks with respect to different evaluation metrics. In Section 5.3, we examine
the effect of different interaction patterns between the instance and class
embeddings. We discuss the effectiveness of ranking metrics in MCC evaluation
in both sections.

Table 3: Results on text classification tasks trained with different losses. Dataset | Encoder | Metrics | SoftmaxCE | PairLogLoss | ApproxNDCG
---|---|---|---|---|---
GoEmotions | BERT | Top-1 Error | 41.00 | 41.79 | 41.42
Top-5 Error | 13.35 | 12.00 | 13.00
NDCG@5 | 73.90 | 74.27 | 74.02
ELECTRA | Top-1 Error | 38.45 | 39.78 | 37.67
Top-5 Error | 10.45 | 8.80 | 9.25
NDCG@5 | 76.96 | 77.17 | 78.03
MIND | BERT | Top-1 Error | 30.90 | 31.21 | 30.97
Top-5 Error | 6.15 | 5.25 | 5.25
NDCG@5 | 82.78 | 83.25 | 83.36
ELECTRA | Top-1 Error | 27.09 | 27.72 | 26.65
Top-5 Error | 4.25 | 3.75 | 4.30
NDCG@5 | 85.73 | 85.74 | 85.89
Table 4: Results on image classification tasks trained with different losses. Dataset | Encoder | Metrics | SoftmaxCE | PairLogLoss | ApproxNDCG
---|---|---|---|---|---
ImageNet | ResNet50 | Top-1 Error | 23.74 | 23.64 | 23.62
Top-5 Error | 6.90 | 6.80 | 6.85
NDCG@5 | 85.74 | 85.82 | 85.80
CIFAR-10 | VGG16 | Top-1 Error | 6.56 | 6.43 | 6.41
Top-5 Error | 0.20 | 0.15 | 0.15
NDCG@5 | 97.19 | 97.21 | 97.23

###  5.1 Overall Performance of Rank4Class

Besides SoftmaxCE, PairLogLoss, and ApproxNDCG discussed in Section 4.1, we
also include Gumbel-ApproxNDCG (bruch2020stochastic) loss and MSE
(hui2021evaluation) in experiments. We study different combinations between
these five losses and the two interaction patterns introduced in Section 4.2
in the Rank4Class framework. In Table 2, we report the best performance of
different configurations for Rank4Class under each metric in comparison to the
baseline MCC models. The complete results of all combinations are included in
Appendix D, where we use $*$ to mark the combinations that achieve the best
performance in each task under each metric. As shown in the table, Rank4Class
can improve the MCC performance in virtually all tasks through specific
combinations of losses and interaction patterns. This shows the increased
capacity from Rank4Class in MCC tasks evaluated on different metrics, which is
achieved by adding more flexible design options in different components from
LTR perspectives.

###  5.2 Effect of Ranking Losses

In this section, we study the use of ranking losses for optimizing MCC
performance. In particular, we use PairLogLoss and ApproxNDCG as two most
representative ranking losses in comparison to the SoftmaxCE. The results on
other losses can be find in Appendix D. We only vary the loss function in the
base Rank4Class structure in Figure 2 (Middle), so the “SoftmaxCE” method is
the baseline that is equivalent to classical MCC models.

The results are shown in Table 3 and 4 for text and image classification tasks
respectively. Overall, PairLogLoss or ApproxNDCG can outperform SoftmaxCE in
nearly all tasks and metrics except for the Top-1 Error on GoEmotions and MIND
datasets with BERT model. Besides, PairLogLoss is generally good at reducing
Top-5 Error than SoftmaxCE and ApproxNDCG, achieving the lowest Top-5 Error in
all tasks. ApproxNDCG performs well on both Top-1 Error and NDCG@5 (achieves
the top in four out of six tasks), which shows its effectiveness in directly
optimizing NDCG metrics. Moreover, we see that the improvements from
PairLogLoss and ApproxNDCG are more significant on text classification tasks
than that on image classification tasks. Our observation is similar to that in
(hui2021evaluation), which hypothesized the reason being that structures and
parameters of popular image encoders are all heavily tuned with the SoftmaxCE.

Finally, we observe that different metrics are not always consistent in
evaluating MCC tasks. For example, on GoEmotions with ELECTRA, PairLogLoss
performs better than ApproxNDCG on Top-5 Error, while ApproxNDCG outperforms
PairLogLoss on NDCG@5. This means that the PairLogLoss tends to rank the
correct class in top 5 positions in more instances than ApproxNDCG, but
ApproxNDCG can put the correct class relatively higher in the rankings.
Furthermore, on MIND with BERT, Top-5 Error can not tell if PairLogLoss or
ApproxNDCG is better. But NDCG@5 can successfully differentiate them since it
also takes the absolute rank of the correct class in top 5 positions into
account and thus is more informative.

Table 5: Results on text classification tasks of different interactions trained with ApproxNDCG. Dataset | Encoder | Metrics |  | dot product
---
LC+MLP | Concat+MLP
GoEmotions | BERT | Top-1 Error | 41.42 | 40.65 | 40.92
Top-5 Error | 13.00 | 12.40 | 12.20
NDCG@5 | 74.02 | 74.62 | 74.61
ELECTRA | Top-1 Error | 37.67 | 37.67 | 37.80
Top-5 Error | 9.25 | 8.65 | 9.15
NDCG@5 | 78.03 | 78.30 | 77.97
MIND | BERT | Top-1 Error | 30.97 | 30.81 | 30.77
Top-5 Error | 5.25 | 5.25 | 5.05
NDCG@5 | 83.36 | 83.43 | 83.35
ELECTRA | Top-1 Error | 26.65 | 26.74 | 26.40
Top-5 Error | 4.30 | 3.70 | 4.30
NDCG@5 | 85.89 | 86.13 | 86.05
Table 6: Results on image classification tasks of different interactions trained with SoftmaxCE. Dataset | Encoder | Metrics | dot product | LC+MLP | Concat+MLP
---|---|---|---|---|---
ImageNet | ResNet50 | Top-1 Error | 23.74 | 23.62 | 23.66
Top-5 Error | 6.90 | 6.80 | 6.75
NDCG@5 | 85.74 | 85.84 | 85.83
CIFAR-10 | VGG16 | Top-1 Error | 6.56 | 6.40 | 6.43
Top-5 Error | 0.20 | 0.20 | 0.20
NDCG@5 | 97.19 | 97.22 | 97.23

###  5.3 Effect of Instance and Class Interactions

In this section, we study the effect of different interaction patterns between
instance and class embeddings for producing ranking scores. Specifically, we
use the two interaction patterns in section 4.2 with simple 2-layer MLPs.

We compare these two types of interactions with the dot-product baseline
between the instance and class embeddings by fixing the loss function. In
particular, we use ApproxNDCG in text classification tasks and SoftmaxCE in
image classification tasks. The study of different interactions on other
losses are included in Appendix D. The results are shown in Table 5 and 6 for
text and image classification tasks. The results demonstrate that the two
added interactions can outperform or achieve competitive performance than
simple dot-product in all tasks and metrics. This shows the effectiveness of
adding enhanced interactions based on the Rank4Class framework. In particular,
latent-cross embedding tends to perform better than the concatenation of
instance and class embeddings, achieving top performance in four out of six on
Top-1 Error and five out of six tasks on NDCG@5. Again, we see that different
evaluation metrics are not always consistent with each other, and NDCG@5 can
be more informative than Top-5 Error, as observed and discussed in Section
5.2.

##  6 Related Work

Modern deep neural networks for MCC converge to the same recipe: given an
input instance, a neural model is learned to output scores for a set of
classes. The vast research literature focus on developing more effective
representation learning encoders in diverse domains, such as computer vision
(he2016resnet; krizhevsky2012imagenet; densenet; dosovitskiy2020vit; 50333),
natural language processing (sun2020finetune; tay2021pre; minaee2021deep), and
automatic speech recognition (moritz2019triggered), among others. The softmax
cross entropy loss is virtually the dominant loss function discussed in these
papers. Only very recently, hui2021evaluation studied the mean squared error
as an alternative loss for training MCC models. Our work is orthogonal to the
extensive research on neural encoders in that we provide a new formulation
from the LTR perspective. Such a perspective inspires more diverse loss
functions and flexible model architectures that can model interactions between
inputs and classes more effectively.

Learning to rank (LTR) is a long-established research area at the intersection
of machine learning and information retrieval (liu2009learning). Neural
ranking models are dominant in ranking virtually all modalities recently,
including text ranking (jimmylinbook), image retrieval (gordo2016deep), and
tabular data ranking (dasalc). Many LTR papers focus on more effective loss
functions (qin2010general; bruch2020stochastic) to rank items with respect to
a query. One of the purposes of this paper is to introduce new techniques
stemming from LTR to solve MCC problems.

Multi-label classification (MLC) differs from MCC in that there are more than
one labels for each instance. MLC is generally treated as a different problem
from classical MCC: the number of labels assigned to an instance could be
arbitrary and one research focus is to decide the threshold to cutoff the
prediction list. Its recently popular sub-field, extreme multi-label
classification (zhang2018deep), also has a different and specific setting
(e.g., the number of labels is huge) and research focus (e.g., efficiency to
rank in the huge label space). LTR techniques have been proposed for MLC in
the past (yang2012multilabel). While the perspective is similar to ours, the
work is not in the deep learning setting and thus does not have the
equivalence view of MCC in LTR. Our work makes the connection and also
proposes extensions for MCC in the deep learning setting.

##  7 Conclusion

In this paper we examine the classical MCC problem through the lens of
ranking. Such a perspective brings benefits to MCC from three aspects: ranking
metrics, ranking losses, and ranking architectures. We first show that ranking
metrics can be more informative for MCC evaluations. Then, in the deep
learning setting, we show an equivalent view of MCC models in LTR setting.
Such a generalization provides new perspectives for MCC with respect to loss
functions and model architectures. We study these new formulations of MCC on
various datasets and observe consistent performance improvements.

Our work opens up several research directions. First, the new ranking
architectures allow to take more class information such as class metadata into
account and it is interesting to study how this additional information can
improve MCC. Second, it is also possible to apply the proposed Rank4Class
framework to binary classification. Third, classes are usually not
independent, and our framework makes it possible to incorporate the
relationship between classes into MCC through attention or graph-based
mechanisms.

##  Appendix A Proof of Theorem 1

See 1

###### Proof.

Consider NDCG@$K$ and Top-$K$ Accuracy as functions of any input class ranking
$\pi\in\Pi$, where $\Pi$ is the space of all possible rankings (of size n!).
Let $P$ be a distribution over $\Pi$ from which the evaluation set is sampled.
Let the probability of the correct class being at position $i$ in the
evaluation set be $p_{i},p_{i}\geq 0,i\in[n]$, and $\sum_{i=1}^{n}p_{i}=1$.
Thus $\mathbb{E}[p_{i}]=\sum_{\mathbf{y}[\pi(i)]=1}P(\pi)$, where $P(\pi)$ is
the probability that a class ranking $\pi$ being sampled from $\Pi$.

Note that NDCG@$K$ has $K$ distinct values corresponding to the correct class
being ranked at the top $K$ positions plus $0$ for not in top $K$ positions.
Therefore, the entropy of the values of NDCG@$K$ can be obtained as:

$H(\textrm{NDCG@$K$})=-\sum_{i=1}^{K}p_{i}\log p_{i}-p_{-}\log p_{-},$ (5)

where $p_{-}=\sum_{i=K+1}^{n}$ is the probability that the correct class is
not ranked in top $K$ positions. $p_{-}\log p_{-}=0$ when $p_{-}=0$ as a
convention in information theory.

However, Top-$K$ Accuracy only has two possible values: $1$ when the correct
class is in top $K$ positions, and $0$ otherwise. Hence the entropy of Top-$K$
Accuracy values is

$H(\textrm{acc@$K$})=-p_{+}\log p_{+}-p_{-}\log p_{-},$ (6)

where $p_{+}=\sum_{i=1}^{K}$ is the probability that the correct class is
ranked in top $K$ positions.

Finally, we have

$\displaystyle H(\textrm{NDCG@$K$})-H(\textrm{acc@$K$})$ (7) $\displaystyle=$
$\displaystyle-\sum_{i=1}^{K}p_{i}\log p_{i}+p_{+}\log p_{+}$ $\displaystyle=$
$\displaystyle\sum_{i=1}^{K}p_{i}\big{(}\log p_{+}-\log p_{i}\big{)}\geq 0.$

Therefore, NDCG@$K$ has more information than Top-$K$ Accuracy in terms of the
entropy of the evaluation results. ∎

##  Appendix B Datasets and Encoders

###  B.1 Datasets

  * •

GoEmotions (demszky2020goemotions) is the largest manually annotated dataset
for fine-grained emotion classification. It contains 58k English Reddit
comments, labeled with 27 emotion categories and a default “Neutral” category.
On top of the raw data, the authors also provided a version that only includes
comments with two or more raters agreeing on at least one label, split into
train/test/validation sets. In our experiments, we filter out comments with
more than 1 emotion labels in all three sets.

  * •

MIND (wu2020mind) is a large-scale dataset for news recommendation. MIND
contains about 130k English news articles. Every news article contains rich
textual content including title, abstract, body, category and entities. We use
the concatenation of title and abstract as the content of the news and use the
category of the news as the classification label. We split all news instances
into train/test/validation sets by roughly 60/20/20 for experiments.

  * •

ImageNet (krizhevsky2012imagenet) is an image dataset with around 1.28 million
images in the training set and 50k images in the validation set. Each image is
labeled by one of 1,000 classes. The images are cropped and resized to
$224\times 224\times 3$ pixels as the input following the preprocessing in
https://github.com/tensorflow/models/tree/master/official/vision/image_classification/resnet.
We follow the common practices of using ImageNet and report the results on the
validation set.

  * •

CIFAR-10 (krizhevsky2009cifar10) contains 50k training images and 10k test
images. There are 10 classes and each class has 6k images. All images have the
same size of $32\times 32\times 3$.

###  B.2 Instance Encoders

  * •

BERT (vaswani2017attention) is a model based on transformers pretrained on a
large corpus of English data. We use the BERT-Base-uncased model from
https://github.com/google-research/bert for finetuning in our experiments. The
maximum sequence length is set to 32 for GoEmotions and 128 for MIND.

  * •

ELECTRA (clark2020electra) is another pretrained transformer model. We use the
ELECTRA-Base-uncased model from https://github.com/google-research/electra in
our experiments. The maximum sequence length is set in the same way as BERT.

  * •

ResNet50 (he2016resnet) is a 50 layers deep convolutional neural network with
residual connections. We use the implementation of ResNet50 from the
tensorflow official models at
https://github.com/tensorflow/models/tree/master/official/vision/image_classification/resnet.
We adopt the pretrained network on ImageNet and finetune it in our Rank4Class
framework.

  * •

VGG16 (simonyan2014very) is a convolutional neural network with 16 layers. We
use the implementation of VGG16 from https://github.com/geifmany/cifar-vgg
with input size $32\times 32$ in our experiments on CIFAR-10.

##  Appendix C Experimental Settings

We provide the implementation and hyper-parameter tuning details for
reproducibility. For text classification tasks, we tokenize the raw sentences
into word ids based on BERT vocabulary, and create the input masks and segment
ids following standard BERT input formats. For MIND, we concatenate the title
and abstract of each news by adding a “[SEP]” token between them. ELECTRA uses
exactly the same input formats as BERT. The data processing of image datasets
follow the standard methods as given in the original papers. We implement the
Rank4Class pipeline based on TF-ranking (rama2019tfrank). We adopt TF-
ranking’s implementation of SoftmaxCE, MSE and all the ranking losses.

For instance encoders, we use the default hyper-parameters suggested in the
original implementations listed in Appendix B without further tuning. We use
the same number of hidden units in the two layers of MLP for interactions, and
tune the number of hidden units in {64, 128, 256, 512}. For all experiments,
we use Adam (kingma2017adam) and Adagrad (john2011adaptive) as optimizers for
training models. We tune the initial learning rate for all experiments in the
range of $1e^{-7}$ to $0.1$ with a multiplicative step size of 3. Adam has a
slight edge over Adagrad in most experiments and the best learning rate for
Adam are $3e^{-6}$ and $1e^{-5}$ for most experiments. We use a batch size of
32 for text classification tasks, and a batch size of 64 for image
classification tasks. For all configurations of the models and datasets, we
train the model for 100,000 steps in text classification tasks, and 50 epochs
in image classification tasks. We pick the best checkpoint on the validation
set (if provided) for evaluation.

##  Appendix D Results on different combinations of ranking losses and
interaction patterns

Here we provide the experiment results on different combinations of loss
functions and interaction patterns. Besides PairLogLoss and ApproxNDCG
included in Section 5, we also include results from Gumbel-ApproxNDCG and MSE
here. Note that we use the rescaled MSE as suggested in (hui2021evaluation) on
ImageNet dataset, since there is only 1 correct class in 1,000 classes, and
regular MSE performs poorly on such imbalanced data. In particular, as the
number of combinations is large, we display the results with respect to each
dataset and instance encoder in each table for better visualization and
comparison. Then in each table, we group the results from different
combinations of loss functions and interaction patterns according to the three
evaluation metrics. We underline the top 3 combinations (ties are included)
under each metric, and use $*$ to mark the best. The results of all datasets
and instance encoders are shown in Tables 7 \- 12.

Firstly, the conclusions on loss functions, interactions and evaluation
metrics from the tables are the same as those in Sections 5.2 and 5.3. The
Gumbel-ApproxNDCG performs well when more complicated interactions such as
latent cross and concatenating embeddings are applied. MSE achieves the best
top-1 Error in GoEmotions with BERT and CIFAR-10 with VGG16. It indicates that
MSE can be effective in optimizing top-1 metrics in certain tasks, which
aligns with the observation in hui2021evaluation. However, MSE is not suitable
for direct application in problems with highly imbalanced correct and
incorrect classes as on the ImageNet classification task. A few rescaling
techniques with hyper-parameters need to be applied for MSE to perform
properly, which also increase the burden of hyper-parameters tuning. Besides,
adding different combinations of losses and interaction patterns can always
improve the performance, as indicated by $*$ under each metric. Note that we
did not further tune the hyper-parameters of instance encoders for different
architectures of Rank4Class. It is possible that the hyper-parameters of
encoders are more suitable for the classical MCC models, and fine-tuning may
further boost the performance of Rank4Class. Last but not least, we see that
NDCG is more informative in evaluating MCC performance than Top-5 Error which
creates many ties. For example, In Table 9 for MIND with BERT, Top-5 Error
fails to find the best performing model, while NDCG@5 can successfully
differentiate them.

Table 7: Results on GoEmotions with BERT as text encoder. GoEmotions + BERT
---
Metric | Interaction | SoftmaxCE | PairLogLoss | ApproxNDCG |  | Gumbel
---
ApproxNDCG
MSE
Top-1 Error | dot product | 41.00 | 41.79 | 41.42 | 41.11 | 41.20
LC+MLP | 41.35 | 41.98 |  40.65* |  40.70 | 41.31
Concat+MLP | 41.02 | 41.55 |  40.92 | 41.59 | 41.63
Top-5 Error | dot product | 13.35 |  12.00 | 13.00 | 13.80 | 15.00
LC+MLP | 12.85 | 12.25 | 12.40 | 12.85 | 14.30
Concat+MLP | 12.40 |  11.85* |  12.20 | 12.90 | 13.85
NDCG@5 | dot product | 73.90 | 74.27 | 74.02 | 73.71 | 73.00
LC+MLP | 74.11 | 74.22 |  74.62* |  74.53 | 73.50
Concat+MLP | 74.36 | 74.19 |  74.61 | 74.05 | 73.55
Table 8: Results on GoEmotions with ELECTRA as text encoder. GoEmotions +
ELECTRA
---
Metric | Interaction | SoftmaxCE | PairLogLoss | ApproxNDCG |  | Gumbel
---
ApproxNDCG
MSE
Top-1 Error | dot product | 38.45 | 39.78 |  37.67 | 38.63 | 37.97
LC+MLP | 38.69 | 39.13 |  37.67 | 38.45 |  37.36*
Concat+MLP | 39.00 | 39.48 | 37.80 | 38.06 | 38.15
Top-5 Error | dot product | 10.45 |  8.80 | 9.25 | 10.65 | 10.30
LC+MLP | 9.30 | 8.90 |  8.65* | 10.00 | 10.15
Concat+MLP | 9.75 |  8.70 | 9.15 | 9.90 | 10.40
NDCG@5 | dot product | 76.96 | 77.17 |  78.03 | 76.72 | 77.37
LC+MLP | 77.45 | 77.49 |  78.30* | 77.41 | 77.69
Concat+MLP | 77.06 | 77.47 |  77.97 | 77.35 | 77.24
Table 9: Results on MIND with BERT as text encoder. MIND + BERT
---
Metric | Interaction | SoftmaxCE | PairLogLoss | ApproxNDCG |  | Gumbel
---
ApproxNDCG
MSE
Top-1 Error | dot product | 30.90 | 31.21 | 30.97 | 30.88 | 30.71
LC+MLP |  30.20* |  30.51 | 30.81 | 30.79 | 30.56
Concat+MLP |  30.51 | 31.29 | 30.77 | 30.92 | 30.71
Top-5 Error | dot product | 6.15 |  5.25 |  5.25 | 5.75 | 6.95
LC+MLP | 5.35 | 5.30 |  5.25 | 5.45 | 7.15
Concat+MLP | 5.55 |  5.25 |  5.05* | 5.65 | 7.30
NDCG@5 | dot product | 82.78 | 83.25 | 83.36 | 83.11 | 82.59
LC+MLP |  83.56* |  83.54 |  83.43 | 83.28 | 82.48
Concat+MLP | 83.31 | 83.27 | 83.35 | 83.16 | 82.37
Table 10: Results on MIND with ELECTRA as text encoder. MIND + ELECTRA
---
Metric | Interaction | SoftmaxCE | PairLogLoss | ApproxNDCG |  | Gumbel
---
ApproxNDCG
MSE
Top-1 Error | dot product | 27.09 | 27.72 | 26.65 | 26.77 | 26.99
LC+MLP | 26.89 | 27.79 | 26.74 |  26.55 | 26.92
Concat+MLP | 26.78 | 27.53 |  26.40* |  26.55 | 26.65
Top-5 Error | dot product | 4.25 |  3.75 | 4.30 | 4.25 | 5.35
LC+MLP | 4.04 |  3.75 |  3.70* | 3.90 | 5.20
Concat+MLP | 4.25 |  3.70* | 4.30 | 3.95 | 5.55
NDCG@5 | dot product | 85.73 | 85.74 | 85.89 | 85.89 | 85.26
LC+MLP | 85.67 | 85.71 |  86.13* |  86.09 | 85.31
Concat+MLP | 85.81 | 85.85 |  86.05 | 85.99 | 85.25
Table 11: Results on ImageNet with ResNet50 as image encoder. ImageNet +
ResNet50
---
Metric | Interaction | SoftmaxCE | PairLogLoss | ApproxNDCG |  | Gumbel
---
ApproxNDCG
| MSE
---
(rescaled)
Top-1 Error | dot product | 23.74 | 23.64 | 23.62 | 23.65 | 26.46
LC+MLP | 23.62 | 23.70 | 23.65 |  23.60 | 24.18
Concat+MLP | 23.66 | 23.79 |  23.61 |  23.58* | 23.83
Top-5 Error | dot product | 6.90 | 6.80 | 6.85 | 6.80 | 8.95
LC+MLP | 6.80 |  6.75* | 6.80 | 6.80 | 6.90
Concat+MLP |  6.75* |  6.75* | 6.80 | 6.80 | 6.80
NDCG@5 | dot product | 85.74 | 85.82 | 85.80 | 85.82 | 83.24
LC+MLP |  85.84 | 85.80 | 85.82 |  85.84 | 85.44
Concat+MLP | 85.83 | 85.77 | 85.82 |  85.85* | 85.72
Table 12: Results on CIFAR-10 with VGG16 as image encoder. CIFAR-10 + VGG16
---
Metric | Interaction | SoftmaxCE | PairLogLoss | ApproxNDCG |  | Gumbel
---
ApproxNDCG
| MSE
---
Top-1 Error | dot product | 6.56 | 6.43 | 6.41 | 6.45 | 6.48
LC+MLP |  6.40* | 6.42 | 6.46 |  6.40* | 6.42
Concat+MLP | 6.43 | 6.44 | 6.44 | 6.47 |  6.40*
Top-5 Error | dot product | 0.20 |  0.15* |  0.15* | 0.20 |  0.15*
LC+MLP | 0.20 |  0.15* | 0.25 | 0.20 | 0.25
Concat+MLP | 0.20 | 0.20 | 0.25 |  0.15* | 0.35
NDCG@5 | dot product | 97.19 | 97.21 |  97.23* | 97.21 | 97.16
LC+MLP |  97.22 | 97.21 | 97.17 | 97.17 | 97.14
Concat+MLP |  97.23* | 97.21 | 97.19 |  97.22 | 97.16


Copyright for this paper by its authors. Use permitted under Creative Commons
License Attribution 4.0 International (CC BY 4.0).

MediaEval’22: Multimedia Evaluation Workshop, January 13–15, 2023, Bergen,
Norway and Online

[1]

[1]

[1] [1]<EMAIL_ADDRESS>]

[1]Corresponding author. [1]These authors contributed equally.

# Floods Relevancy and Identification of Location from Twitter Posts using NLP
Techniques

Muhammad Suleman  Muhammad Asif  Tayyab Zamir  Ayaz Mehmood  Jebran Khan
Nasir Ahmad  Kashif Ahmad  DCSE, University of Engineering and Technology,
Peshawar, Pakistan  Abasyn University Islamabad Campus, Pakistan  Department
of AI, AJOU University, South Korea  Department of Computer Science, Munster
Technological University, Cork, Ireland

(2022)

###### Abstract

This paper presents our solutions for the MediaEval 2022 task on DisasterMM.
The task is composed of two subtasks, namely (i) Relevance Classification of
Twitter Posts (RCTP), and (ii) Location Extraction from Twitter Texts (LETT).
The RCTP subtask aims at differentiating flood-related and non-relevant social
posts while LETT is a Named Entity Recognition (NER) task and aims at the
extraction of location information from the text. For RCTP, we proposed four
different solutions based on BERT, RoBERTa, Distil BERT, and ALBERT obtaining
an F1-score of 0.7934, 0.7970, 0.7613, and 0.7924, respectively. For LETT, we
used three models namely BERT, RoBERTa, and Distil BERTA obtaining an F1-score
of 0.6256, 0.6744, and 0.6723, respectively.

##  1 Introduction

Natural disasters represent hazardous events that are generally caused by
geophysical, hydrological, climatological, and meteorological elements. These
hazardous events may have an adverse impact on human lives and infrastructure.
Floods are one such event and it frequently occurs in different parts of the
world. Similar to other natural disasters, floods may have a significant
impact on public health and infrastructure. For instance, it has been noticed
on numerous occasions that roads and communication infrastructure are badly
damaged during floods [1].

A rapid and effective response to disasters may help in mitigating their
adverse impact. Access to relevant and timely information is critical for an
effective response. The literature demonstrates several situations where
access to relevant information may be possible due to several reasons, such as
the unavailability of reporters in the area and damage to communication [2].
Recently social media and crowdsourcing have been explored as a source of
communication, information collection, and dissemination in emergency
situations. To this aim, several interesting solutions have been proposed to
collect, analyze, and extract meaningful insights from social media content
[2]. However, social media content also comes with several limitations. For
instance, social media content is generally noisy, thus, making access to
relevant information very challenging. Similarly, geolocation information,
which is critical for the relevance of the content, is not necessarily
available for all the relevant posts.

Considering the importance and applications of social media content in
disaster analytics floods detection in social media content has been also
included in the MediaEval benchmark competition as a shared task for several
years. This paper presents a solution for the MMDisaster task presented in
MediaEval 2022 [3]. The challenge aims to solve two key challenges to disaster
analytics in social media. The first subtask aims at reducing social media
noise by automatically filtering social media content to obtain relevant
content. The second subtask aims at extracting location information from
social media text, allowing automatic positioning of a potential incident due
to floods. For both subtasks, we proposed several interesting solutions as
described in Section 3.

##  2 Related Work

In recent years, the potential of social media has been widely explored in
different application domains [4, 5]. Some of the key applications where
social media content has been already proven very effective include public
health [6], education [7], and public resource management [8]. Social media
outlets have also been widely explored for a diversified set of applications
in disasters and emergency situations [2]. For instance, Hao et al. [9]
proposed a multi-modal framework utilizing multi-social media imagery and
textual information for damage assessment in disaster-hit areas. The key
factors analyzed in the work include hazard/disaster type, severity, and
damage type. Wu et al. [10] also utilized social media data and the associated
geo-location information for generating early warnings and damage assessment
analysis after disasters. Ahmad et al. [11], on the other hand, used social
media imagery for the analysis of road conditions after the floods. More
specifically, the authors proposed an early and late-fusion framework to
identify passable roads in flooded regions. Alam et al. [12] explored the
potential of social media content in another relevant task of assessing flood
severity. To this aim, the authors collected a large-scale benchmark dataset
namely CrisisMD. The dataset provides a large collection of Twitter posts
including textual and visual content. Hassan et al. [13] explored a slightly
different aspect of natural disasters by extracting sentiments and emotions
from visual content shared in social media outlets. The authors detailed how
visual sentiment analysis of disaster-related social media visual content can
be utilized by different stakeholders, such as news agencies, public
authorities, and humanitarian organizations.

Despite being proven very effective in different tasks of disaster analytics,
social media content has several limitations, such as noisy data and the
unavailability of geolocation information. In this paper, we propose a
solution to overcome such challenges.

##  3 Approach

###  3.1 Relevance Classification of Twitter Posts (RCTP)

As a first step, we analyzed the available multimedia content. During the
analysis, we observed that most of the posts missing visual content. Moreover,
most of the images were irrelevant. Thus, we decided to use textual
information only in our solution. Our framework for the RCTP subtask is
composed of two steps. In the first step, we performed some pre-processing
techniques to clean the data by removing unnecessary information, such as
usernames, URLs, emojis, and stop words.

After pre-processing, several state-of-the-art NLP algorithms including BERT
[14], Roberta [15], Distil BERT [16], and ALBERT [17] are used for the
classification of the text. Since its a binary classification task, in all
methods, our cost function is based on binary crossentropy. Moreover, we used
Adam optimizer with a mini batch size of 32 for 20 epochs.

###  3.2 Location Extraction from Twitter Texts (LETT)

LETT subtask is treated as Named Entity Recognition (NER) task. NER involves
locating and classifying named entities in text into pre-defined categories
[18]. In this task, we are interested in the identification of words
representing the starting and subsequent words of a text sequence referring to
a location. In LETT, annotations are provided at the word level. Similar to
the RCTP task, in this task, we rely on multiple state-of-the-art algorithms
including BERT, Roberta, Distil BERT, and ALBERT. We note that in this task,
since annotations are provided at the word level, we did not use any pre-
processing technique before training our models.

###  3.3 Dataset

For both subtasks, separate datasets are released. The dataset for RCTP
subtask contains data from a total of 8,000 tweets. The tweets are collected
between May 25, 2020, and June 12, 2020, using flood-related keywords in the
Italian Language, such as ”alluvione”, ”allagamento”, and ”esondazione”. The
dataset is provided in two different sets namely the development set and the
test set. The development set is composed of 5,337 tweets while the test set
contains a total of 1,315 tweets.

The dataset for the LETT subtask is composed of around 6,000 tweets collected
between March 25, 2017, and August 1, 2018, using flood-related Italian
keywords. The annotations for this subtask are available per word in the
tweets.

##  4 Results and Analysis

###  4.1 Runs Description of RCTP Subtask

Table 1 shows the experimental results of the proposed solutions on the
development set. We note that during the experiments on the development set,
we used 70% samples of the development set for training, 20% for testing, and
10% samples for validation. As can be seen in the table, no significant
differences can be observed in the performance of the models on the clean and
un-clean datasets. As far as the performance of the individual models is
concerned, slightly better results are obtained with BERT compared to the
other models.

Table 1: Experimental results of RCTP task on the development set. Model | F1-Score on the Clean Dataset | F1-Score on the Un-clean Dataset
---|---|---
BERT | 0.95 | 0.94
RoBERTa | 0.94 | 0.93
Distil BERT | 0.93 | 0.93
ALBERT | 0.92 | 0.92

Table 2 provides the official results of the proposed solutions on the test
set. We note that for the experiments on the test set the models are trained
on the complete development set. In total, 4 different runs are submitted for
the task. Our first, second, and fourth runs are based on BERT, RoBERTa, and
Distil Bert models trained on the un-cleaned dataset, respectively. Our third
run is based on the BERT model trained on the cleaned dataset. The performance
of the models trained on the un-cleaned dataset is higher than the models
trained on the cleaned dataset. This indicates that the pre-processing
information resulted in the removal of some relevant features and thus has a
negative impact on the results.

Table 2: Experimental results of the RCTP task on the test set.

Run | Precision | Recall | F1-Score
---|---|---|---
1 (BERT on Un-clean Dataset) | 0.6949 | 0.9251 | 0.7934
2 (RoBERTa on Un-lean Dataset) | 0.6947 | 0.9347 | 0.7970
3 (BERT on Clean Dataset) | 0.6486 | 0.9213 | 0.7613
4 (Distil BERT on Un-clean Dataset) | 0.6940 | 0.9232 | 0.7924

###  4.2 Runs Description of LETT Subtask

Table 3 provides the experimental results of the proposed solutions on the
development set for the LETT subtask. Similar to RCTP, we used 70% samples of
the development set for training, 20% for testing, and 10% samples for
validation. A significant variation can be observed in the results of the
models on the development set. Overall, better results are obtained for
RoBerta with a significant improvement of 3% over the second-highest results
obtained with the BERT model.

Table 3: Experimental results of the LETT task on the development set.

Model | F1-Score
---|---
BERT | 0.7752
RoBERTa | 0.8014
Distil BERT | 0.7658
ALBERT | 0.6827

Table 4 provides the official results for the LETT subtask. In the experiments
on the test set, the models are trained on the complete development set. We
note that in the partial results the omitted samples are counted as false
while in the partial results the omitted samples are completely ignored
without any penalty. As can be seen in the table, overall, better results are
obtained with Roberta and Distil BERT compared to the original implementation
of the BERT model.

Table 4: Experimental results of LETT task on the test set.

Run | Exact Results | Partial Results
---|---|---
Precision | Recall | F1-Score | Precision | Recall | F1-Score
1 (BERT) | 0.596 | 0.522 | 0.556 | 0.628 | 0.622 | 0.625
2 (RoBERTa) | 0.540 | 0.676 | 0.600 | 0.577 | 0.810 | 0.674
3 (Distil BERT) | 0.563 | 0.604 | 0.583 | 0.610 | 0.760 | 0.677

##  5 Conclusions

In this paper, we presented our solutions for the DisasterMM challenge posted
in MediaEval 2022. For both subtasks, multiple state-of-the-art NLP algorithms
are employed. In the current implementation, all the models are used
individually, however, we believe these models can complement each other if
jointly utilized in a merit-based fusion method. In the future, we aim to
employ different merit-based fusion methods to jointly utilize the
capabilities of the individual models in both subtasks.

## References

  * Nguyen et al. [2017] D. T. Nguyen, F. Ofli, M. Imran, P. Mitra,  Damage assessment from social media imagery data during disasters,  in: Proceedings of international conference on advances in social networks analysis and mining 2017, 2017, pp. 569–576.
  * Said et al. [2019] N. Said, K. Ahmad, M. Riegler, K. Pogorelov, L. Hassan, N. Ahmad, N. Conci,  Natural disasters detection in social media and satellite imagery: a survey,  Multimedia Tools and Applications 78 (2019) 31267–31302.
  * Andreadis et al. [2023] S. Andreadis, A. Bozas, I. Gialampoukidis, A. Moumtzidou, R. Fiorin, F. Lombardo, T. Mavropoulos, D. Norbiato, S. Vrochidis, M. Ferri, I. Kompatsiaris,  DisasterMM: Multimedia Analysis of Disaster-Related Social Media Data Task at MediaEval 2022,  in: Proceedings of the MediaEval 2022 Workshop, Bergen, Norway and Online, 2023.
  * Ahmad et al. [2019] K. Ahmad, K. Pogorelov, M. Riegler, N. Conci, P. Halvorsen,  Social media and satellites,  Multimedia Tools and Applications 78 (2019) 2837–2875.
  * Alsmadi et al. [2022] I. Alsmadi, K. Ahmad, M. Nazzal, F. Alam, A. Al-Fuqaha, A. Khreishah, A. Algosaibi,  Adversarial nlp for social network applications: Attacks, defenses, and research directions,  IEEE Transactions on Computational Social Systems (2022).
  * Andreadis et al. [2021] S. Andreadis, G. Antzoulatos, T. Mavropoulos, P. Giannakeris, G. Tzionis, N. Pantelidis, K. Ioannidis, A. Karakostas, I. Gialampoukidis, S. Vrochidis, et al.,  A social media analytics platform visualising the spread of covid-19 in italy via exploitation of automatically geotagged tweets,  Online Social Networks and Media 23 (2021) 100134.
  * et al. [2021] A.-R. et al.,  The influence of information system success and technology acceptance model on social media factors in education,  Sustainability 13 (2021) 7770.
  * Ahmad et al. [2022] K. Ahmad, M. Ayub, J. Khan, N. Ahmad, A. Al-Fuqaha,  Social media as an instant source of feedback on water quality,  IEEE Transactions on Technology and Society (2022).
  * Hao and Wang [2020] H. Hao, Y. Wang,  Leveraging multimodal social media data for rapid disaster damage assessment,  International Journal of Disaster Risk Reduction 51 (2020) 101760.
  * Wu and Cui [2018] D. Wu, Y. Cui,  Disaster early warning and damage assessment analysis using social media data and geo-location information,  Decision support systems 111 (2018) 48–59.
  * Ahmad et al. [2019] K. Ahmad, K. Pogorelov, M. Riegler, O. Ostroukhova, P. Halvorsen, N. Conci, R. Dahyot,  Automatic detection of passable roads after floods in remote sensed and social media data,  Signal Processing: Image Communication 74 (2019) 110–118.
  * Alam et al. [2018] F. Alam, F. Ofli, M. Imran,  Crisismmd: Multimodal twitter datasets from natural disasters,  in: Twelfth international AAAI conference on web and social media, 2018.
  * Hassan et al. [2022] S. Z. Hassan, K. Ahmad, S. Hicks, P. Halvorsen, A. Al-Fuqaha, N. Conci, M. Riegler,  Visual sentiment analysis from disaster images in social media,  Sensors 22 (2022) 3628.
  * Devlin et al. [2018] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova,  Bert: Pre-training of deep bidirectional transformers for language understanding,  arXiv preprint arXiv:1810.04805 (2018).
  * Liu et al. [2019] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, V. Stoyanov,  Roberta: A robustly optimized bert pretraining approach,  arXiv preprint arXiv:1907.11692 (2019).
  * Sanh et al. [2019] V. Sanh, L. Debut, J. Chaumond, T. Wolf,  Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter,  arXiv preprint arXiv:1910.01108 (2019).
  * Lan et al. [2019] Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, R. Soricut,  Albert: A lite bert for self-supervised learning of language representations,  arXiv preprint arXiv:1909.11942 (2019).
  * Li et al. [2020] J. Li, A. Sun, J. Han, C. Li,  A survey on deep learning for named entity recognition,  IEEE Transactions on Knowledge and Data Engineering 34 (2020) 50–70.