# The rise and fall of light stops in the LHC top quark sample

Emanuele Bagnaschi§,†, Gennaro Corcella†, Roberto Franceschini‡, Dibyashree
Sengupta†<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>$\S$ CERN, Theoretical Physics Department,
1211 Geneva 23, Switzerland
† INFN, Laboratori Nazionali di Frascati, Via E. Fermi 40, 00044 Frascati
(RM), Italy
‡ Università degli Studi Roma and INFN, Via della Vasca Navale 84, 00146, Roma

###### Abstract

We discuss the possibility that light new physics in the top quark sample at
the LHC can be found by investigating with greater care well known kinematic
distributions, such as the invariant mass $m_{b\ell}$ of the $b$-jet and the
charged lepton in fully leptonic $t\bar{t}$ events. We demonstrate that new
physics can be probed in the rising part of the already measured $m_{b\ell}$
distribution. To this end we analyze a concrete supersymmetric scenario with
light right-handed stop quark, chargino and neutralino. The corresponding
spectra are characterized by small mass differences, which make them not yet
excluded by current LHC searches and give rise to a specific end-point in the
shape of the $m_{b\ell}$ distribution. We argue that this sharp feature is
general for models of light new physics that have so far escaped the LHC
searches and can offer a precious handle for the implementation of robust
searches that exploit, rather than suffer from, soft bottom quarks and
leptons. Recasting public data on searches for new physics, we identify
candidate models that are not yet excluded. For these models we study the
$m_{b\ell}$ distribution and derive the expected signal yields, finding that
there is untapped potential for discovery of new physics using the $m_{b\ell}$
distribution.

††preprint:  CERN-TH-2023-241

### Introduction

Supersymmetry (SUSY) at the TeV scale is a pillar of modern Beyond Standard
Model (BSM) phenomenology. It has shaped the thinking and the expectations for
new physics for the past few decades. After the lack of new physics signals in
the flavor experiments in the late 90s and early 2000s, great hopes were put
in the possibility to observe super-partners in high energy collisions at the
Large Hadron Collider (LHC). Indeed, it is entirely possible to build
supersymmetric models in which there is no signal of new physics in flavor
observables or in CP violation. These matters are entirely ruled by the
breaking of SUSY and there is no deep conceptual motivation to involve SUSY in
flavor dynamics.

On the contrary, SUSY features a microscopic dynamics deeply connected to the
spontaneous breaking of the electroweak symmetry. It implies the existence of
two Higgs doublets with fermionic partners relatively close in mass, thus
fueling the expectation for new physics close to the mass scale of the Higgs
boson. In addition, in weakly coupled SUSY models, such as the Minimal
Supersymmetric Standard Model (MSSM) Dimopoulos and Georgi (1981), the
particle properties at the weak scale are a calculable output of the
microscopic properties of the supersymmetric model Martin (1998). A prominent
role in this computation is played by the spectrum and couplings of the top
quark and its scalar partner (see, e.g., Slavich  _et al._ (2021) on the role
played by top and stop quarks on the MSSM Higgs masses). Thus the
investigation of new physics related to top quark phenomenology Franceschini
(2023) emerged as a prime target for searches for new physics already at the
onset of the LHC program.

Despite the great efforts put in these searches, no signs of new physics have
been spotted at the LHC yet. In light of these results, the widespread
attitude today is to favor physics scenarios with new physics characterized by
a mass scale beyond the reach of the LHC, but possibly accessible to future
larger machines Mangano, M. et al. (2019); J. de Blas et al. (2018); ALEGRO
collaboration (2019); Accettura, et al. (2023). For this reason it has become
customary to parameterize new physics using contact operators than encode
micro-physics in a similar fashion to how the Fermi four-fermion contact
interaction preluded to the $SU(2)$ weak interactions Grzadkowski  _et al._
(2010).

In spite of the general trend, in this letter we take a complementary attitude
and investigate the possibility of light new physics in the top quark sector
still not excluded at the LHC. We find that it is still possible for new
physics to appear in signals that are are quite similar to the simplest
manifestations that one can imagine in the MSSM. Furthermore, we will provide
new methods to probe this enticing possibility and show that there is a
significant potential to make a discovery in the current LHC dataset.

The majority of searches have so far concentrated on signals characterized by
_large energy releases_ in the new physics events, in the TeV range, giving
rise to BSM signal that appear in regions of the phase-space where the SM has
a scarce rate. Much less activity has been devoted to searches in SM-rich
signal regions. To fill this gap it is necessary to pursue a search method
that confronts the SM backgrounds where they are largest, that is to say in
events where the energy release is in the range of _tens or few hundreds of
GeV_. In this region of phase-space, thanks to the enormous progress in SM
high-precision calculations, it is possible to carry out measurements with
exquisite precision, e.g. ATLAS Collaboration (2023),ATL (2018); Andari
(2017); ATLAS Collaboration (2017a), CMS (2022); CMS Collaboration (2015),
ATLAS Collaboration (2019); ATLAS Collaboration (2017b); CMS (2016).
Therefore, in this letter we propose to carry out new searches for BSM in
regions of phase-space and in physical observables that were previously used
_only_ for SM measurements. Following our novel use of the data acquired for
these measurements, we demonstrate that it is possible to obtain sensitivity
to new physics scenarios that have not been probed yet by current searches or
suffer from large uncertainty in the reach of these searches.

To ascertain what new physics scenarios are currently probed by the present
results of the LHC, we will recast publicly available data using a simplified-
model approach Kraml  _et al._ (2014); Mahdi Altakach  _et al._ (2023). This
method offers a reproducible and relatively reliable procedure to determine
what new physics models, beyond those explicitly tested by the experiments,
can be considered as excluded. This recast allows us to focus our attention on
the models that are likely still experimentally allowed. Our search for new
physics models not yet excluded by recast of public information has also the
value to be a stress-test of the present strategy for the publication of
experimental results and their reinterpretation. We believe this is a valuable
contribution to the assessment of the quality of the reinterpretation effort
carried out by the LHC community.

### Elusive New Physics in top-quark samples

New physics in the top quark sector has been searched for in a large number of
final states (see e.g. Ref. Franceschini (2023) for a recent review). A large
fraction of the signatures involves top quarks accompanied by some extra
handle characteristic of new physics productions, e.g.
$t\bar{t}+\cancel{E}_{T}$ or $t\bar{t}+\phi$, with $\cancel{E}_{T}$ being
missing transverse momentum and $\phi$ a new boson. Other possible signatures
are pursued with bottom quarks, that are expected from the weak $SU(2)$
symmetry, and final states from off- and on-shell top quarks. Generally these
searches are sensitive to new physics that results in large energy release,
because of large mass differences between the new states themselves and
between new states and SM ones. Unfortunately this search strategy hits a
blindstpot when the spectrum has small mass differences.

The issue of possible blind spots in LHC searches, due to small mass
differences in the new physics spectrum and possible closeness of the new
physics masses with the SM ones, has emerged early on in the exploitation of
the LHC data Fan  _et al._ (2011, 2015). Solutions that have been suggested
involved the precision measurement of the total cross-section for $t\bar{t}$
production Eifert and Nachman (2015); Cohen  _et al._ (2019), possible
disagreement in the extraction of $m_{t}$ Cohen  _et al._ (2019) and angular
distributions Cohen  _et al._ (2018); Han  _et al._ (2012) that may be
sensitive to new physics, as well as features in the kinematic distributions
of very high-$p_{T}$ top quarks Macaluso  _et al._ (2015).

Despite these proposals, the status of weak scale supersymmetry, including
that of superpartners charged under QCD, remains unset. One reason is that not
all of the above proposals have yet been incorporated in the suite of searches
that the experiments carry out, which can be interpreted in a twofold manner.
On the one hand this is signaling there is still room for improvement, on the
other hand this can be read as a signal of the objective difficulty to use new
observables for searches, e.g. for the difficulty to obtain the desired data
with sufficiently small uncertainty as required by the proposed new searches.
This motivates our effort to seek a method that requires mere reinterpretation
of measurements already carried out and no new observables to be measured with
high accuracy.

To fill in this gap we propose a new method to identify new physics hidden in
the top quark sample. Our method leverages the notable feature of the
candidate new physics models to involve energy releases in the decay of the
new physics particles that are typically smaller than those of the SM
$t\bar{t}$ production and other SM background processes.

Figure 1: The $m_{b\ell}$ distribution of the SM (grey line, rescaled by 1/8)
and MSSM signals (colored lines) for the benchmark points ON1, OFF1, OFF2, T1,
T2.The signal distribution has a characteristic rise-and-fall shape, which
makes it easier to observe. In the bottom panel we display the ratio of the
signals over the SM contribution.

As a consequence of this feature the range of the Lorentz invariants that can
be used to characterize the kinematics of the events involving new physics is
different from that of the SM processes. In particular the maximum of the
invariant mass of the bottom and the lepton that arise from the top quark
decay, denoted by $m_{b\ell}$, turns out to be significantly smaller than in
SM $t\bar{t}$ production for models that have not been excluded from present
searches. This gives rise to a notable rise-and-fall shape of the new physics
signal in the low energy part of the $m_{b\ell}$ spectrum. This rise and fall
shape changes for each new physics model spectrum, but it is generic of the
whole class of new physics models not presently excluded. Some examples of
$m_{b\ell}$ spectra that arise for MSSM parameter choice not presently
excluded by recast of public data are presented in Fig. 1 for illustration.

Remarkably, the quantity $m_{b\ell}$ is routinely measured by the LHC
experiments, thus our observation can be readily applied to measurements
already carried out, with no required new measurements to be performed to
search for new physics. 111We stress that in principle other quantities can be
used to test new physics following the same logic that lead us to $m_{b\ell}$.
E.g. the spectrum of the energy of the $b$ jets is sensitive to new physics
Agashe  _et al._ (2013); Franceschini (2017), but there is no published data
on this quantity except a preliminary CMS measurement CMS collaboration
(2015).

Figure 2: MSSM signal significance eq. (2) versus ${\chi}_{1}^{\pm}$ and
${\chi}_{1}^{0}$ at fixed $m_{\tilde{t}_{1}}=200$ GeV evaluated from the
$m_{b\ell}$ spectrum and its uncertainty in ATLAS ATLAS Collaboration (2019)
(before the fit of SM backgrounds). The black dots correspond to the points in
the plane we have explicitly generated. The colored shades are step-wise
interpolations of the generated points. The color scale is saturated at $z=5$.
The black line is the approximate contour for the exclusion from present
searches. The full result of our recast is presented in the Appendix in Fig.
3. The region of the plane at the left of the black line is deemed as excluded
from our recast.

Based on the $m_{b\ell}$ spectrum of each new physics model we derive an
estimate for the sensitivity to new physics of precision $m_{b\ell}$
measurements carried out at the LHC. We take ATLAS ATLAS Collaboration (2019)
and CMS Sirunyan  _et al._ (2019a) recent results on the $m_{b\ell}$ spectrum
(and its uncertainty in each bin) to compute the expected statistical
significance that a new physics signal from the MSSM would have. The obtained
expected significance is given in Fig. 2 from which we can see that our method
has the potential to probe large areas of the MSSM parameter space. In
particular it is sensitive to mass spectra that are not currently probed by
reinterpretation of LHC searches. In addition the sensitivity of our method
depends on a different combination of the new physics masses with respect to
standard searches. Thus our method can probe new physics in previously
unexplored corners of the models space and also provide independent
information on models that can be studied with already proposed strategies.
This is particularly useful in view of the complexity to set bounds with
traditional methods due to systematics and modelling difficulties, e.g.
related to the transition from on- to off-shell intermediate resonances.

### Methodology

The top quark is pair-produced at the LHC through strong interactions with a
cross-section about 1 nb Czakon  _et al._ (2013); Catani  _et al._ (2019),
yielding a sample of ${\cal O}(10^{8})$ $t\bar{t}$ pairs produced so far.
Given such large numbers, it is possible to study the top quark in great
detail, including rare decay modes. We consider the decay to a double lepton
final state, which leads to a final state containing 2 $b$-jets, 2 oppositely
charged leptons and their corresponding neutral partners which manifest as
missing transverse momentum at the LHC. In this sample we carefully study a
well-known observable, $m_{b\ell}$, the invariant mass of the lepton and the
$b$-jet. This quantity has been used to extract the mass of the top quark CMS
collaboration (2014, 2009) and its width Aaboud  _et al._ (2018); Khachatryan
_et al._ (2014), demonstrating the great control that the experiments can
achieve on this observable.

As argued above, the observable $m_{b\ell}$ can show deviations from the SM
prediction if it gets contaminated by a new particle which has mass close to
$m_{t}$ and can mimic the final state of fully leptonic $t\bar{t}$ events.
Referring to the MSSM as a case study, such final states are given by events
with pair-production of the lightest stop quark $\tilde{t_{1}}$, decaying into
the lightest chargino ${\chi_{1}^{\pm}}$ and a $b$-quark, followed by
${\chi_{1}^{\pm}}$ decay into the lightest neutralino ${\chi_{1}^{0}}$ and
lepton pair via a real or a virtual $W$:

$\tilde{t_{1}}\to\chi^{\pm}_{1}b\to b\ell\nu\chi_{1}^{0}\;.$

The lightest neutralino and the neutrino manifest themselves as missing
transverse momentum, so that this signals leads to the same final state as
fully leptonic $t\bar{t}$ pairs. An analogous final state could be achieved in
the MSSM if the lightest stop quark $\tilde{t_{1}}$ decays into a
(leptonically decaying) $t$ quark and ${\chi_{1}^{0}}$. In this case the
dominant decay is ($\tilde{t_{1}}\to t{\chi_{1}^{0}}$), which suffers much
stronger bounds from specific searches Sirunyan  _et al._ (2019b) and
therefore it is not in the focus of our work.

To root our study in a concrete and reproducible setup, we consider a large
set of points in the MSSM parameter space and study the signal in the
$m_{b\ell}$ spectrum for each MSSM point. For concreteness we consider three
possible values for the lightest stop mass: $m_{\tilde{t_{1}}}$ = $180$, $200$
and $220$ GeV. For each value of $m_{\tilde{t_{1}}}$ we scan the parameter
space to get different values of $m_{{\chi_{1}^{\pm}}}$ and
$m_{{\chi_{1}^{0}}}$. In order to obtain our reference points, we make use of
SPheno 4.0.3 Porod and Staub (2012), interfaced with SARAH 4.15.1 Staub
(2014). The SPheno input parameters are set at a high scale $Q$ and then run
down to the weak scale by means of Renormalization Group Equations. The
description of the inputs used is provided in the Appendix.

Each of these points has been checked against searches available for recast
using SModelS 2.2.1 Alguero  _et al._ (2022). In particular we check the value
of the metric $r$ computed by SModelS 2.2.1. Points for which $r>1$ are deemed
to be excluded by the recast, while for $r<1$ we consider the present public
data to be insufficient to exclude that model. Clearly, it is possible that
the full dataset held by the experimental collaborations, as well as
combinations of signal regions not taken into account by SModelS 2.2.1, can
still exclude the points were we find $r<1$.

Next we simulate the contribution to $m_{b\ell}$ for each parameter space
point using Pythia 8.3 Bierlich  _et al._ (2022) in the region of phase space
identified by the following selection:

$\displaystyle p_{T}(\ell)\geq 25~{}\text{GeV},$
$\displaystyle|\eta(\ell)|<2.5,$ (1) $\displaystyle p_{T}(j)\geq
25~{}\text{GeV},$ $\displaystyle|\eta(j)|<2.5,$

for jets made with anti-kT Cacciari  _et al._ (2008) algorithm with $R=0.4$
and separations between jets and leptons $\Delta R(\ell,j)>0.2$, $\Delta
R(j,j)>0.4$ and $\Delta R(\ell,\ell)>0.1$. This is a selection closely
following that of the experimental collaborations, e.g. ATLAS Collaboration
(2019); Aaboud  _et al._ (2018); CMS (2016), except for minor differences in
the selection for $\ell=e$ and $\ell=\mu$ that we do not pursue. We have
considered variations of the cuts and found that, if attainable, softer
selections on the transverse momenta would magnify the signal in the
$m_{b\ell}$ distribution even further, but we limit ourselves to the
conservative choice of cuts as in eq. (1). The $m_{b\ell}$ spectra that we
obtain are compared with the spectrum measured by ATLAS ATLAS Collaboration
(2019) and CMS Sirunyan  _et al._ (2019a) for 139 fb-1 integrated luminosity.
As the experimental results are endowed with an uncertainty on each bin of the
measured differential cross-section $d\sigma/dm_{b\ell}$, we can use the
expected rate of MSSM signal to compute a significance

$z=\sqrt{\sum_{i}\left(\frac{S_{i}}{\delta B_{i}}\right)^{2}}\,,$ (2)

where $S_{i}$ is the MSSM signal yield expected in the $i$-th bin of the
published histogram and $\delta B_{i}$ is the uncertainty on each bin as
published by the experiments. In absence of more precise information from the
experiments, the uncertainty in each bin is assumed to be uncorrelated with
the others.

We note that both experimental collaborations provide two set of
uncertainties: one is obtained with nominal Monte Carlo predictions and
uncertainties, while a second one is provided after the measured $m_{b\ell}$
spectrum is used as a constrain on the sum of the Monte Carlo predictions for
several SM processes contributing to the relevant region of phase-space. These
two results are indicated by the experiments as “pre-fit” and “post-fit”
measurements of the $m_{b\ell}$ distribution. The post-fit one has smaller
uncertainties and leads to stronger bounds on new physics. For reference we
note that the smallest uncertainty in a single bin for the “pre-fit” ATLAS
result we use is about 5%. Using the “post-fit” result would give even
stronger exclusions, as the smallest uncertainty in a single bin would be
reduced to 0.8% in that scenario. However, we argue that it should be used
with care, because it is obtained assuming that the $m_{b\ell}$ spectrum is
due solely to the SM and no new physics.

In Fig. 2 we show the more conservative “pre-fit” result of the significance
eq. (2) from the ATLAS result ATLAS Collaboration (2019). Points for which
$z>2$ can be excluded at 95% confidence level with the new proposed analysis
of $m_{b\ell}$. Strikingly, the region excluded by our proposal covers a large
area of the chargino-neutralino mass plane not excluded by the recast of the
present searches.

We observe that the contours of $z$ in the chargino-neutralino mass plane
closely follow the contours of the maximal $m_{b\ell}$ value that can be
obtained for a cascade decay Franceschini  _et al._ (2023); Barr and Lester
(2010), thus they depend on a different combination of the masses compared to
the present searches. This is apparent comparing the contours of $r$ in Fig. 3
in the Appendix and the contours of $z$ in Fig. 2.

For greater detail, in Tab. 1 we present the result for several points that
are not excluded by recast of present searches, i.e. SModelS 2.2.1 gives $r<1$
222The most recent version of SModelS at the time of writing is 2.3.2. We
checked that the new searches included in the newest release of SModelS do not
change the values of $r$ for the points in this table.. We note that in
several cases one expects deviations from the SM in the $m_{b\ell}$
distribution much larger than the uncertainties published by the experiments.
These include cases for chargino-neutralino mass differences close to $m_{W}$,
where the present searches have a marked blindspot. We note that CMS results
tend to give a weaker sensitivity: this is due to the coarser binning of the
data published by CMS with respect to ATLAS. The table presents results for
three masses $m_{\tilde{t}}$ considered. The complementarity of the proposed
search using $m_{b\ell}$ is evident for all the masses $m_{\tilde{t}}$
considered, as to testify the general validity of the point that we make in
this letter.

BM | $\mu$ | $M_{1}$ | $A_{t}$ | $m_{\chi^{+}}$ | $m_{\chi^{0}}$ |  $z$ Sirunyan  _et al._ (2019a) |  $z$ ATLAS Collaboration (2019) | $r$
---|---|---|---|---|---|---|---|---
| $m_{\tilde{t}}=200$ GeV
ON1 | 185 | 95 | 2820.5 | 186.6 | 85.6 |  [0.8,1.7] |  [2.7,14.3] | 0.9
OFF1 | 155 | 160 | 2857.5 | 156.4 | 123.3 |  [0.9,1.8] |  [2.6,14.8] | 0.7
OFF2 | 175 | 145 | 2839.5 | 176.6 | 123.5 |  [1.5,3.] |  [5.1,25.5] | 0.8
T1 | 135 | 65 | 2895.5 | 136.2 | 54. |  [4.,7.7] |  [10.7,61.3] | 0.8
T2 | 135 | 60 | 2895.5 | 136.2 | 49.9 |  [4.1,7.9] |  [10.8,60.6] | 0.8
| $m_{\tilde{t}}=220$ GeV
OFF3 | 155 | 150 | 3140.5 | 156.4 | 118.6 |  [0.7,1.4] |  [1.9,10.9] | 0.8
OFF4 | 170 | 160 | 3122 | 171.5 | 130.8 |  [0.9,1.8] |  [2.5,13.7] | 0.6
ON2 | 190 | 95 | 3104 | 191.7 | 86.1 |  [2.1,4.3] |  [6.1,32.8] | 0.7
OFF5 | 190 | 145 | 3104 | 191.7 | 127.7 |  [1.4,2.8] |  [4.2,22.5] | 0.6
ON3 | 190 | 65 | 3104 | 191.7 | 58.9 |  [1.9,3.7] |  [5.3,28.7] | 0.8
| $m_{\tilde{t}}=180$ GeV
OFF6 | 165 | 115 | 2570.5 | 166.5 | 99.2 |  [1.2,2.5] |  [4.8,22.9] | 0.8
OFF7 | 160 | 105 | 2580 | 161.5 | 90.4 |  [2.2,4.5] |  [7.2,36.3] | 0.8
OFF8 | 160 | 170 | 2570 | 161.5 | 130.3 |  [0.6,1.2] |  [2.4,11.2] | 0.6
OFF9 | 155 | 150 | 2579.5 | 156.4 | 118.5 |  [1.6,3.2] |  [5.3,27.2] | 0.8
OFF10 | 145 | 175 | 2598.5 | 146.3 | 122.2 |  [0.8,1.6] |  [2.4,12.7] | 0.8
Table 1: Chargino and neutralino masses, input parameters $\mu$, $M_{1}$ and
$A_{t}$, all given in GeV for few benchmarks (BM). Resulting value of $r$
computed from SModelS 2.2.1 and the range of the significance eq. (2) expected
from the $m_{b\ell}$ spectrum analysis using ATLAS ATLAS Collaboration (2019)
or CMS Sirunyan  _et al._ (2019a) measurements. The low (high) end the
significance range corresponds to uncertainties on the $m_{b\ell}$ spectrum
before(after) a fit using SM predictions for the known backgrounds.

### Summary and outlook

The presently available public information from LHC searches for new physics
can be re-interpreted to test models for which the LHC experiments have not
provided explicit results. The MSSM is the model for which the majority of the
efforts to provide reinterpretation data has been carried out so far. In our
work we have used the publicly available information and assembled it to the
extent that it is possible with a standard tool such as SModelS 2.2.1. We have
investigated the bounds on weak scale supersymmetry spectra featuring light
$SU(2)$ singlet stops squarks and light bino/Higgsino states with masses close
to the top quark mass. We find that the LHC search results, reinterpreted by
using the most recent data, still cannot exclude large part of the models we
tested.

At face value this result implies that the LHC experiments, even after years
of efforts, have not yet reached a level at which weak-scale supersymmetry can
be said to be ruled completely, not even in the case of colored super
partners.

Exclusion of supersymmetric models with light stops, bino and Higgsino may be
possible using the full power of the data that is held by the experimental
collaborations, but this cannot be deduced neither from publicly available
information on searches results nor from a recast of the public material.
Therefore, we urge the experimental collaborations to present data for these
models in a similar fashion to our results in Fig.3, as to provide results for
these specific types of models. We also advise that they should do all that is
in their power to release and maintain public information suitable to obtain
exclusions on new physics models in a systematic and reproducible way. We have
explained that the generation of spectra of this type in the MSSM may require
a very focused setting of MSSM spectrum generators, thus a specific effort,
beyond the “wide-net” studies in the context of the pMSSM, may be needed to
tackle this issue.

In addition to raising a flag about the actual reach of the LHC searches for
new physics and their reinterpretability, we have provided an example of
search strategy that would cover the gaps that we have highlighted. The novel
strategy leverages the precision in top quark measurements and in particular
the spectrum of the b-jet+lepton invariant mass $m_{b\ell}$, that is used to
measure the top quark mass and its width with high precision. Relying on
publicly available data on the measured $m_{b\ell}$ spectrum we have
identified the sensitivity contours in the chargino-neutralino mass plane at
various representative values of the stop mass. Remarkably the sensitivity of
our search strategy depends on a different combination of the masses that that
relevant for the other searches. In particularly it seems to cope well with
the transition between on- and off-shell intermediate resonances, such as
$\chi^{\pm}\to\chi^{0}W^{\pm}\longmapsto\chi^{\pm}\to\chi^{0}\ell^{\pm}\nu$.
Therefore, the novel search strategy that we propose offers a very valuable
complementary constraint that can fill the gaps and extend the reach of the
searches for weak scale supersymmetry that have been devised so far.
Furthermore, our proposal holds the promise to become an exemplary incarnation
of the paradigm of new physics searches through high precision Standard Model
measurements. Thus we urge the experimental collaborations to start
interpreting the measurement of $m_{b\ell}$ and possibly other top quark
precision properties as searches for new physics and to provide explicit
results for bounds on models from these observables.

## Acknowledgements

The work of RF is supported in part by the MUR PRIN2022 Grant n.202289JEW4. It
is a pleasure for RF to thanks Federico Meloni, Javier Montejo Berlingen,
Krzysztof Rolbiecki for useful discussions. We thank the Galileo Galilei
Institute for Theoretical Physics for the hospitality and the INFN for partial
support during the completion of this work.

## References

  * Dimopoulos and Georgi (1981) S. Dimopoulos and H. Georgi, Nucl.Phys. B193, 150 (1981).
  * Martin (1998) S. P. Martin, in  _Perspectives on Supersymmetry_ , edited by G. L. Kane (1998) p. 1, arXiv:hep-ph/9709356 .
  * Slavich  _et al._ (2021) P. Slavich _et al._ , Eur. Phys. J. C 81, 450 (2021), arXiv:2012.15629 [hep-ph] .
  * Franceschini (2023) R. Franceschini, Annual Review of Nuclear and Particle Science 73, arXiv:2301.04407 (2023), arXiv:2301.04407 [hep-ph] .
  * Mangano, M. et al. (2019) Mangano, M. et al., Eur. Phys. J. C 79 (2019), 10.1140/epjc/s10052-019-6904-3.
  * J. de Blas et al. (2018) J. de Blas et al. , CERN Yellow Rep. Monogr. Vol. 3 (2018) (2018), CYRM-2018-003, arXiv:1812.02093 [hep-ph] .
  * ALEGRO collaboration (2019) ALEGRO collaboration, (2019), arXiv:1901.10370v2 [physics.acc-ph] .
  * Accettura, et al. (2023) Accettura, et al., arXiv e-prints , arXiv:2303.08533 (2023), arXiv:2303.08533 [physics.acc-ph] .
  * Grzadkowski  _et al._ (2010) B. Grzadkowski, M. Iskrzynski, M. Misiak, and J. Rosiek, JHEP 10, 085 (2010), arXiv:1008.4884 [hep-ph] .
  * ATLAS Collaboration (2023) ATLAS Collaboration,  _Improved W boson Mass Measurement using 7 TeV Proton-Proton Collisions with the ATLAS Detector_, Tech. Rep. ATLAS-CONF-2023-004 (CERN, Geneva, 2023).
  * ATL (2018) _Prospects for the measurement of the W-boson mass at the HL- and HE-LHC_, Tech. Rep. ATL-PHYS-PUB-2018-026 (CERN, Geneva, 2018).
  * Andari (2017) N. Andari (ATLAS),  _W mass measurement_, Tech. Rep. ATL-PHYS-PROC-2017-051 (CERN, Geneva, 2017).
  * ATLAS Collaboration (2017a) ATLAS Collaboration, ArXiv e-prints (2017a), arXiv:1701.07240 [hep-ex] .
  * CMS (2022) _Projection of the top quark spin correlation measurement and search for top squark pair production at the HL-LHC_, Tech. Rep. CMS-PAS-FTR-18-034 (CERN, Geneva, 2022).
  * CMS Collaboration (2015) CMS Collaboration, ArXiv e-prints (2015), arXiv:1511.06170 [hep-ex] .
  * ATLAS Collaboration (2019) ATLAS Collaboration (ATLAS),  _Measurement of the top-quark decay width in top-quark pair events in the dilepton channel at $\sqrt{s}=13$ TeV with the ATLAS detector_, Tech. Rep. ATLAS-CONF-2019-038 (CERN, Geneva, 2019).
  * ATLAS Collaboration (2017b) ATLAS Collaboration, arXiv e-prints , arXiv:1709.04207 (2017b), arXiv:1709.04207 [hep-ex] .
  * CMS (2016) _Bounding the top quark width using final states with two charged leptons and two jets at $\sqrt{s}=13~{}\mathrm{TeV}$_, Tech. Rep. (CERN, Geneva, 2016).
  * Kraml  _et al._ (2014) S. Kraml, S. Kulkarni, U. Laa, A. Lessa, W. Magerl, D. Proschofsky-Spindler, and W. Waltenberger, Eur. Phys. J. C 74, 2868 (2014), arXiv:1312.4175 [hep-ph] .
  * Mahdi Altakach  _et al._ (2023) M. Mahdi Altakach, S. Kraml, A. Lessa, S. Narasimha, T. Pascal, and W. Waltenberger, SciPost Phys. 15, 185 (2023), arXiv:2306.17676 [hep-ph] .
  * Fan  _et al._ (2011) J. Fan, M. Reece, and J. T. Ruderman, Journal of High Energy Physics 11, 12 (2011), arXiv:1105.5135 [hep-ph] .
  * Fan  _et al._ (2015) J. Fan, R. Krall, D. Pinner, M. Reece, and J. T. Ruderman, (2015), arXiv:1512.05781v1 [hep-ph] .
  * Eifert and Nachman (2015) T. Eifert and B. Nachman, Phys. Lett. B 743, 218 (2015), arXiv:1410.7025 [hep-ph] .
  * Cohen  _et al._ (2019) T. Cohen, S. Majewski, B. Ostdiek, and P. Zheng, arXiv e-prints (2019), arXiv:1909.09670 [hep-ph] .
  * Cohen  _et al._ (2018) T. Cohen, W. Hopkins, S. Majewski, and B. Ostdiek, ArXiv e-prints (2018), arXiv:1804.00111 [hep-ph] .
  * Han  _et al._ (2012) Z. Han, A. Katz, D. Krohn, and M. Reece, JHEP 8, 083 (2012), arXiv:1205.5808 [hep-ph] .
  * Macaluso  _et al._ (2015) S. Macaluso, M. Park, D. Shih, and B. Tweedie, ArXiv e-prints (2015), arXiv:1506.07885 [hep-ph] .
  * Agashe  _et al._ (2013) K. Agashe, R. Franceschini, and D. Kim, Phys. Rev. D 88, 057701 (2013), arXiv:1209.0772 [hep-ph] .
  * Franceschini (2017) R. Franceschini, Nuovo Cim. C 39, 340 (2017).
  * CMS collaboration (2015) CMS collaboration, CMS-PAS-TOP-15-002 (2015).
  * Sirunyan  _et al._ (2019a) A. M. Sirunyan _et al._ (CMS), Eur. Phys. J. C 79, 368 (2019a), arXiv:1812.10505 [hep-ex] .
  * Czakon  _et al._ (2013) M. Czakon, P. Fiedler, and A. Mitov, Phys. Rev. Lett. 110, 252004 (2013), arXiv:1303.6254 [hep-ph] .
  * Catani  _et al._ (2019) S. Catani, S. Devoto, M. Grazzini, S. Kallweit, and J. Mazzitelli, JHEP 07, 100 (2019), arXiv:1906.06535 [hep-ph] .
  * CMS collaboration (2014) CMS collaboration, CMS-PAS-TOP-14-014 (2014).
  * CMS collaboration (2009) CMS collaboration, CMS-PAS-TOP-09-002 (2009).
  * Aaboud  _et al._ (2018) M. Aaboud _et al._ (ATLAS), Eur. Phys. J. C 78, 129 (2018), arXiv:1709.04207 [hep-ex] .
  * Khachatryan  _et al._ (2014) V. Khachatryan _et al._ (CMS), Phys. Lett. B 736, 33 (2014), arXiv:1404.2292 [hep-ex] .
  * Sirunyan  _et al._ (2019b) A. M. Sirunyan _et al._ (CMS), JHEP 03, 101 (2019b), arXiv:1901.01288 [hep-ex] .
  * Porod and Staub (2012) W. Porod and F. Staub, Comput. Phys. Commun. 183, 2458 (2012), arXiv:1104.1573 [hep-ph] .
  * Staub (2014) F. Staub, Comput. Phys. Commun. 185, 1773 (2014), arXiv:1309.7223 [hep-ph] .
  * Alguero  _et al._ (2022) G. Alguero, J. Heisig, C. K. Khosa, S. Kraml, S. Kulkarni, A. Lessa, H. Reyes-González, W. Waltenberger, and A. Wongel, JHEP 08, 068 (2022), arXiv:2112.00769 [hep-ph] .
  * Bierlich  _et al._ (2022) C. Bierlich _et al._ ,  (2022), 10.21468/SciPostPhysCodeb.8, arXiv:2203.11601 [hep-ph] .
  * Cacciari  _et al._ (2008) M. Cacciari, G. P. Salam, and G. Soyez, JHEP 04, 063 (2008), arXiv:0802.1189 [hep-ph] .
  * Franceschini  _et al._ (2023) R. Franceschini, D. Kim, K. Kong, K. T. Matchev, M. Park, and P. Shyamsundar, Rev. Mod. Phys. 95, 045004 (2023), arXiv:2206.13431 [hep-ph] .
  * Barr and Lester (2010) A. J. Barr and C. G. Lester, J. Phys. G 37, 123001 (2010), arXiv:1004.2732 [hep-ph] .
  * LEP (2003) _A Combination of Preliminary Results on Gauge Boson Couplings Measured by the LEP experiments_ , Tech. Rep. LEPEWWG-TGC-2003-01. DELPHI-2003-068-PHYS-936. L3-Note-2826. LEPEWWG-2006-01. OPAL-TN-739. ALEPH-2006-016-CONF-2003-012 (CERN, Geneva, 2003) 2003 Summer Conferences.

##  Appendix A Sensitivity of present searches

The full result on the exclusion metric $r$ computed by SModelS 2.2.1 is given
in Fig. 3. A notable drop of the strength of the bounds from the searches
appears along the line diving the on-shell and off-shell intermediate $W$ in
the $\tilde{\chi}^{+}$ decay. Similar plots can be obtained for other values
of $m_{\tilde{t}_{1}}$. The overall look is quite similar, thus we do not
display these figures here for brevity. The general trend is for lighter
$m_{\tilde{t}_{1}}$ larger parts of the chargino-neutralino mass plane are
constrained. As clear from Tab. 1 there are regions unconstrained by the
recast of the present searches for all the three masses $m_{\tilde{t}_{1}}$ we
have tested. It should be recalled that the approach of SModelS 2.2.1does not
allow to combine exclusions from different signal regions, as this requires to
account for correlations, that are in general not available. Hence it is
possible that a more complete analysis, that can only be carried out by the
experiments, may find that larger regions of the chargino-neutralino mass
plane are excluded.

Figure 3: Values of $r$ for different masses of $\tilde{\chi}_{1}^{\pm}$ and
$\tilde{\chi}_{1}^{0}$ at fixed $m_{\tilde{t}_{1}}=$ 200 GeV calculated using
SModelS 2.2.1. A value of $r>1$ implies the model is excluded. The color scale
is saturated at $r=2$.

##  Appendix B MSSM points generation

In general we focused our generation on spectra in which the decay via
chargino is prominent, as to pursue this elusive decay mode. To obtain viable
spectra from a full MSSM generator we had to set inputs at a particularly low
scale. This is an indication of the fact that the models we are studying are
far from being simplest options in an MSSM with a single mass scale. Clearly,
if the MSSM realizes this type of spectrum, a non-trivial pattern of SUSY
breaking giving rise to multi-scale super-partners spectrum must be arranged.
We do not interpret this fact and we limit ourselves to use the MSSM to
generate mass spectra, whose existence can be read as a mere example and proof
of consistency of the new physics we are studying. For the reproducibility of
our work we list all the relevant inputs of the spectrum generator. At $Q=1.6$
TeV, we set:

  * •

$m_{\tilde{u}}(1,1)^{2}$ = $m_{\tilde{u}}(2,2)^{2}$ = $m_{\tilde{q}}(i,i)^{2}$
= $m_{\tilde{l}}(i,i)^{2}$ = $m_{\tilde{e}}(i,i)^{2}$ =
$m_{\tilde{d}}(i,i)^{2}$ = 1.2 $\cdot$ $10^{7}$ GeV2 for $i=1,2,3$, where
$\tilde{q},\tilde{l}$ are charged under SU(2), while
$\tilde{d},\tilde{u},\tilde{e}$ are charged only under hyper-charge. All the
off-diagonal squark and slepton mass terms are set to zero;

  * •

$m_{\tilde{u}}(3,3)^{2}$ = 1.7 $\cdot$ $10^{5}$ GeV2 which governs the
lightest stop eigenstate and results in a $\tilde{t}_{1}$ almost pure SU(2)
singlet state;

  * •

$M_{1}\in$ [5,1000] GeV, $M_{2}=1$ TeV, $M_{3}=3.5$ TeV;

  * •

$M_{A}^{2}$ = 2 $\cdot$ $10^{6}$ GeV2, $\tan\beta$ = 10;

  * •

$\mu\in[100~{}\text{GeV},m_{\tilde{t}_{1}}]$;

  * •

$A(3,3)$: the trilinear scalar soft SUSY breaking interaction for the stop in
the range $X_{t}+\mu\cdot\cot\beta+[-100,100]\,\text{GeV}$ where the exact
value is obtained by trial-and-error to get the desired $m_{\tilde{t}_{1}}$.
All other trilinear couplings are set to zero.

In order to optimize our effort we did not consider $M_{1}$ below 5 GeV as it
tends to make the decay channel $\tilde{t_{1}}\to t{\chi_{1}^{0}}$ very
copious and the spectrum is typically excluded by dedicated searches, e.g.
Ref. Sirunyan  _et al._ (2019b). The lower limit on $\mu$ takes into account
limits from LEP-II experiments LEP (2003).


# Meta Cross-Modal Hashing on Long-Tailed Data

Runmin Wang College of Computer and Information Sciences, Southwest
University, Chongqing, China. Guoxian Yu School of Software, Shandong
University. Corresponding Author<EMAIL_ADDRESS>Carlotta Domeniconi
Department of Computer Science, George Mason University, VA, USA. Xiangliang
Zhang CEMSE, King Abdullah University of Science and Technology, Thuwal, SA.

###### Abstract

Due to the advantage of reducing storage while speeding up query time on big
heterogeneous data, cross-modal hashing has been extensively studied for
approximate nearest neighbor search of multi-modal data. Most hashing methods
assume that training data is class-balanced. However, in practice, real world
data often have a long-tailed distribution. In this paper, we introduce a
meta-learning based cross-modal hashing method (MetaCMH) to handle long-tailed
data. Due to the lack of training samples in the tail classes, MetaCMH first
learns direct features from data in different modalities, and then introduces
an associative memory module to learn the memory features of samples of the
tail classes. It then combines the direct and memory features to obtain meta
features for each sample. For samples of the head classes of the long tail
distribution, the weight of the direct features is larger, because there are
enough training data to learn them well; while for rare classes, the weight of
the memory features is larger. Finally, MetaCMH uses a likelihood loss
function to preserve the similarity in different modalities and learns hash
functions in an end-to-end fashion. Experiments on long-tailed datasets show
that MetaCMH performs significantly better than state-of-the-art methods,
especially on the tail classes.

##  1 Introduction

With the rapid expansion of the Internet, a massive amount of multimedia data
is being generated in our daily lives. To retrieve data in this scenario,
approximate nearest neighbor (ANN) search plays an essential role in all kinds
of applications. As a widely studied solution for ANN, hashing has sparked
increasing research interest, due to its advantage in reducing storage
consumption and speeding up query time [1, 2]. Hashing aims at mapping data
vectors into fixed length binary codes, while preserving the proximity in the
original space, and thus facilitates approximate search in constant time.

Learning to hashing (LTH) is a popular category of hashing methods. LTH aims
at learning an optimal compound hash function $\mathbf{b}=h(\mathbf{x})$ from
the training data, and maps each input data sample onto a compact code vector
$\mathbf{b}$. Therefore, LTH methods are data-dependent. Most hashing methods
simply assume that the training data is balanced across different classes [3,
4]. However, in the real world, the frequency distribution of visual classes
is long-tailed [5, 6], which includes a few frequent (or head) classes and a
large number of rare (or tail) ones. The training samples of the tail classes
are too scarce to learn the corresponding hashing function well. Thus, few-
shot hashing methods have been proposed to deal with insufficient training
samples [7, 8].

In many applications, the data has different modalities (e.g., image, text,
video and audio) and we need to search for data items across the modalities.
For example, we may want to find images or videos semantically related to the
text of a query or to keywords. Thus, single-modal hashing methods are
extended to multi-modal scenarios, and diverse cross-modal hashing methods
have been proposed [9, 10, 11, 12]. These methods neglect the problem of
imbalanced classes and insufficient training samples in the training set.
Cross-modal zero-shot learning methods have been proposed to model new
classes, unseen in the training set but available in the test set [12, 11];
however, they still assume that the data distribution is balanced, or
approximately balanced. In addition, these methods have been mainly applied to
balanced datasets with only 10 to 30 different categories (e.g., _Flickr_ ,
one of the most widely used datasets, contains samples from 10 different
categories). But in practice, the frequency distribution of visual classes is
long-tailed [5, 6]. Most of the existing cross-modal hashing methods only
consider data samples in the head classes and ignore samples in the tail ones,
which add up to a large portion of the whole distribution. Therefore, these
methods show poor robustness on real world long-tailed data, as shown in our
experiments.

Recently, meta-learning has been used extensively and has witnessed great
success in a variety of fields [13]. Meta-learning provides an alternative
paradigm, where a machine learning method gains experience (meta knowledge)
over learning episodes, to eventually boost its performance on few-shot data.
In addition, it has been recognized that neural networks with memory
capacities can empower meta-learning [14]. Therefore, we use an associative
memory module to solve the problem of few-shot learning and imbalanced classes
simultaneously. Particularly, we embed the meta-learning paradigm into the
cross-modal hashing framework, and propose a meta learning based cross-modal
hashing method (MetaCMH). MetaCMH first introduces a memory-augmented feature
learning network to transfer meta-knowledge from head classes with sufficient
samples to tail ones with insufficient samples, and thus to learn meta
features for samples from both head and tail classes. At the same time,
MetaCMH minimizes a likelihood loss and a quantization loss to preserve the
similarity in meta features and to learn hash codes of samples. In addition, a
balance loss is used to make each bit of the hash code being balanced on all
the training samples. Our main contributions are as follows:

  * •

We consider the most common long-tailed data distribution and propose a novel
CMH method (MetaCMH) to attack the problem of few-shot learning and imbalanced
classes simultaneously.

  * •

We embed the meta-learning paradigm into the cross-modal hashing framework. We
design an associative memory module to learn the meta-knowledge from samples
of head classes and use this memory-based meta-knowledge to boost the
performance on samples of tail classes.

  * •

The results on long-tailed datasets show that MetaCMH outperforms state-of-
the-art cross-modal hashing methods, especially for the samples from tail
classes.

##  2 Related Works

Our work is closely related to cross-modal hashing and meta-learning. We
briefly review relevant recent work in these areas.

###  2.1 Cross-modal hashing

CMH methods can be divided into two main groups, supervised and unsupervised,
based on whether they use the semantic (class) labels of the samples. Since in
this paper we focus on the distribution of the number of samples from
different classes, we also consider zero-shot cross-modal hashing solutions,
which can handle samples of new classes unseen during training.

Unsupervised CMH methods typically project the features of different
modalities into a shared binary space to learn the hash functions.
Unsupervised CMH was first inspired by canonical correlation analysis (CCA).
For example, cross view hashing (CVH) [15] aims to find a common Hamming space
to maximize the correlation among different modalities. Collective matrix
factorization hashing (CMFH) [9] finds different projection functions from
different modalities to a unified subspace, and takes these projection
functions as hash functions. Deep neural networks have achieved great success
for their feature extraction capability. Thus, CMH methods with a deep
architecture have attracted increasing interest in recent years. Unsupervised
Deep Cross-Modal Hashing (UDCMH) [16] integrates deep learning and matrix
factorization with binary latent factor models, and explicitly constrains the
hash codes to preserve the neighborhood structure of the original data. It
obtains hash codes via solving a discrete-constrained objective function
directly without relaxation. The methods mentioned above are all unsupervised
approaches which do not make use of valuable class labels of training data.

Supervised methods take advantage of semantic labels of training data and
often achieve a performance superior to unsupervised ones. To name a few,
cross-modal similarity sensitive hashing (CMSSH) [17] considers every bit of a
hash code as a classification task, and learns the whole bits one by one.
Semantic correlation maximization (SCM) [18] optimizes the hashing functions
by maximizing the correlation between two modalities with respect to the
semantic similarity obtained from labels. Semantics preserving hashing (SePH)
[19] transforms the semantic affinities obtained from class labels into a
probability distribution and approximates this distribution with to-be-learned
hash codes in the Hamming space, and finally learns the hash codes via kernel
logistic regression. Recently, supervised CMH methods combined with deep
architectures have been extensively studied. For example, deep cross-modal
hashing (DCMH) [20] integrates feature learning and hash code learning into a
same deep framework, and minimizes a joint loss function to preserve the
similarity and generate hash codes simultaneously. Self-supervised adversarial
hashing (SSAH) [21] introduces two adversarial networks and a uniform label-
net to maximize the semantic correlation and consistency of the
representations between different modalities. Most of these CMH solutions
assume that the labels are fully available during training, and the number of
samples from different classes is balanced. In real world scenarios, though,
the labels of the training data are only partially known, or not fully
reliable. For this reason, weakly-supervised CMH methods have been proposed.
Weakly-supervised cross-modal Hashing (WCHash) [22] enriches the labels of
training data and defines a latent central modality to make efficient hashing
on more than three modalities. Flexible cross-modal hashing (FlexCMH) [23]
extends weakly-supervised cross-modal hashing on partially paired modality
data by jointly optimizing the match between samples of different modalities
and collaborative matrix factorization based hashing.

In real world applications, the number of samples from different classes is
often imbalanced, and to expand cross-modal hashing to emerging new labels,
zero-shot CMH methods have been recently proposed. Attribute-guided network
(AgNet) [12] aligns different modality data into a semantically rich attribute
space via different attribute extraction networks, and then maps attribute
vectors into hash codes. The shared attribute space enables the knowledge
transfer from seen classes to unseen ones. Cross-modal zero-shot hashing
(CZHash) [11] first learns a category space under the guidance of attribute
data matrices, then learns hash functions via jointly optimizing the composite
(inter-modal and intra-modal) similarity and category attribute space.

To the best of our knowledge, almost all the existing cross-modal hashing
methods explicitly assume that the training set is class-balanced. As a
result, they are vulnerable to long-tailed training sets, especially for
samples from rare classes, as our experiments show. To overcome these
limitations, we propose a novel CMH method (MetaCMH) based on meta learning
for long-tail datasets.

###  2.2 Meta-learning

To date, no single or general definition of meta-learning exists [13].
Intuitively, meta-learning is similar to transfer learning [24], where the
model acquires experience (or meta-knowledge) over multiple learning episodes
from source domains (data distributions) to improve the performance on new
tasks with few training samples per class. Here we roughly group meta-learning
methods into three categories according to the representation of meta-
knowledge $\omega$.
Parameter Initialization here $\omega$ corresponds to the initial parameters
of a neural network. For example, model-agnostic meta-learning (MAML) [25] and
probabilistic model-agnostic meta-learning (PMAML) [26] take meta-knowledge
$\omega$ as initial conditions of the inner optimization. These methods aim to
find a good initialization for the models, so that they can produce a good
generalization performance on a new task with a small amount of training data.
Latent embedding optimization (LEO) [27] first learns a low-dimensional
representation for the parameters and then does the meta-learning process in
the low-dimensional subspace.
Embedding Functions first learn an embedding network $\omega$ in the meta-
learning process and transform raw input data into a representation which can
preserve the original data structure. Prototypical networks [28] learn a
metric space in which few-shot classification is achieved by computing the
Euclidean distances between prototype representations of each class. Parameter
networks [29] assume that similar categories have similar parameters, and
predict parameters from activations.
Black-Box Models directly train learner $\omega$ from the support set (i.e.
$\theta=f_{\omega}(\mathcal{D}^{meta-train})$) rather than relying on the
gradient. For example, memory-augmented neural networks can learn parameters
from old data and assimilate new data quickly [30]. [31] changes the memory
retrieval mechanism of Neural Turing Machines and introduces an external
memory module to learn meta-knowledge in the training procedure. Then the
meta-knowledge (memory parameter) is used to enhance the neural networks and
improve the rapid generalization ability. Meta networks [32] improve on this
neural model by combining slow weights and fast weights. Slow weights are
learned by SGD like ordinary neural networks and the updates are slow. Fast
weights are predicted by an extra neural network and improve the rapid
generalization ability of the whole network.

Our MetaCMH builds on a memory-augmented network to learn meta features for
each sample, so as to achieve the unification of samples from head classes and
those from tail ones. In MetaCMH, the memory feature is not simply added as a
direct feature with a learned weight, as in [6]. We determine the weights
based on the extent to which the sample belongs to the tail class. This makes
it more interpretable and effective, as shown in the experiments.

##  3 Proposed Method

###  3.1 Notation and Problem Definition

Without loss of generality, in this paper we present the solution for two
different modalities (i.e., image and text). Suppose that
$\mathcal{O}=\\{o_{i}\\}^{n}_{i=1}$ is the dataset with $n$ samples, where
each sample $o_{i}$ contains data from two different modalities
$\\{\mathbf{x}_{i},\mathbf{y}_{i}\\}$. Let $\mathbf{X}\in\mathbb{R}^{n\times
d_{x}}$ denote the image modality and $\mathbf{x}_{i}\in\mathbb{R}^{d_{x}}$
corresponds to an image of $o_{i}$. Similarly,
$\mathbf{Y}\in\mathbb{R}^{n\times d_{y}}$ is the text modality. For each
sample $o_{i}$, we have a class label vector $\mathbf{l}_{i}\in\\{0,1\\}^{L}$
to encode which classes it belongs to, and $L$ is the number of distinct
classes. If the $t$-th element of $\mathbf{l}_{i}$ is equal to 1, then $o_{i}$
belongs to class $t$; if it’s equal to 0, $o_{i}$ does not belong to class
$t$. Here, the number of samples in different classes subject to the long-
tailed distribution (or power law distribution). In other words, a small
number of classes has a moderate number of samples, and a large number of
classes has a very small number of samples. We denote the affinity
(similarity) matrix as $\mathbf{A}$ to identify whether the samples are
similar to one another. If sample $i$ and $j$ have at least one label in
common, we assume they are similar to each other and set $a_{ij}=1$. On the
other hand, if sample $i$ and sample $j$ do not have any label in common, we
set $a_{ij}=0$.

The goal of cross-modal hashing is to find a series of hash functions for each
modality to map samples into a fixed length binary code while preserving their
original proximity. In this paper, we aim to find two hash functions
$f_{x}(\mathbf{x})$ and $f_{y}(\mathbf{y})$ for the image and text modalities,
where $f_{x}(\mathbf{x}_{i})\in\\{-1,+1\\}^{c}$ is the hash code of sample
$o_{i}$ obtained from the image modality, and
$f_{y}(\mathbf{y}_{i})\in\\{-1,+1\\}^{c}$ is the hash code obtained from the
text modality. Here $c$ is the hash code length. With the cross-modal
similarity preserved, if samples $o_{i}$ and $o_{j}$ are similar (i.e.,
$a_{ij}=1$), the Hamming distance between the hash codes
$f_{x}(\mathbf{x}_{i})$ and $f_{y}(\mathbf{y}_{j})$ will be smaller;
otherwise, the Hamming distance will be larger.

The whole framework of our method is illustrated in Figure 1. There are two
basic modules in our framework, the meta embedding and the hash code learning
modules. The meta module first learns meta features for each sample from
different modalities. The weights of memory features for samples from head
classes will be small and that for samples from tail classes will be large,
therefore we can enhance the feature extraction ability of our meta embedding
module especially for tail classes, for which there are not sufficient
training samples. We then obtain the hash code from our hash learning module,
which preserves the similarity by minimizing a likelihood loss function and
making hash codes of samples from the same class close to each other. The
details of our meta network and hash code learning module are presented in the
following subsections.

Figure 1: The MetaCMH framework. The meta embedding network for each modality
consists of two parts, a basic network and a meta network. For each sample, we
first obtain its direct features of the image (text) modality using a basic
image (text) network, and use a memory-augmented meta network to enhance the
direct features with meta features. We then quantify the meta features and
generate hash codes for each sample.

###  3.2 Asymmetric Weighted Similarity

There are two main challenges in face of long-tailed data. As we can see in
the Figure 2, the first challenge is the category imbalance problem.
Specifically, there are more instances in the head categories than tail
categories, and this will lead to an imbalance weight of different categories
in the model learning process. The second challenge is the few-shot learning
problem in the tail categories which we will deal with in the later section.
To overcome the category imbalance problem, we propose an asymmetric weighted
similarity.

Figure 2: The distribution of instance numbers of different categories.

###  3.3 Meta Embedding

In long-tailed data, samples can be roughly divided into two categories,
samples from head classes and those from tail ones. Head classes have
sufficient training samples, thus the neural network can learn their features
well. For tail classes, the training samples are insufficient and ordinary
neural networks perform poorly. Therefore, we use a meta embedding network to
transfer meta-knowledge from head classes to tail ones, and enable our feature
learning module to rapidly generalize to samples from tail ones. Firstly, we
learn _direct features_ $\mathbf{v}_{direct}$ from each modality using a
neural network. For the image modality, given the strong feature engineering
ability of Convolutional Neural Networks, we choose a pre-trained CNN named
CNN-F [33] to learn the features. CNN-F is a CNN with 8 layers. The detailed
structure of the network is given in Table 1. The first five layers are
convolutional layers denoted by “conv1-5”. In Table 1, “filters:$num\times
size\times size$” means there are $num$ convolution filters in the layer and
each filter size is $size\times size$. “st.$t$” means the convolution stride
is $t$. “pad $n$” means adding $n$ pixels to each edge of the input image.
“LRN” means Local Response Normalisation is applied to this layer. “pool”
denotes the down-sampling factor. For text modality, we use a neural network
with two full-connected layers. The detailed structure is also given in Table
1. “nodes:$n$” means there are $n$ nodes in the layer. We want to remark that
we mainly focus on the meta-embedding module, thus other alternative networks
can be also adopted to extract features from image and text modalities.

CNN performs well under the prerequisite of sufficient training data [34]. But
for long-tailed datasets, especially for samples from tail classes, we lack
enough data for training. Given this, we introduce the _memory features_
$\mathbf{v}_{memory}$ to enhance the direct feature vector
$\mathbf{v}_{direct}$. We know that the feature learning procedure projects
the raw samples data into a semantic subspace. Samples of different classes
may also have partially similar semantic attributes. For example, pandas and
tigers belong to different classes but they are both furry. Thus, if we take
the linear combination of prototypes of different classes as the memory
features $\mathbf{v}_{memory}$, then the latter can be a good supplement for
the direct features $\mathbf{v}_{direct}$. Inspired by this observation, we
denote the meta feature vector as $\mathbf{v}_{meta}$:

(3.1) $\small\mathbf{v}_{meta}=\mathbf{v}_{direct}+\eta\mathbf{v}_{memory}$

Similarly to LSTM [35], in our meta-embedding network, both
$\mathbf{v}_{memory}$ and $\mathbf{v}_{meta}$ are dependent on the input
$\mathbf{v}_{direct}$.

Specifically, for the memory features $\mathbf{v}_{memory}$, we follow the
idea in [6] and set $\mathbf{v}_{memory}$ to a linear combination of
discriminative centroids (prototypes) of different classes. We take the
centroid of each class as an attribute and use their linear combination to
enhance the direct features. We denote the centroid of the $i$-th class as
$\mathbf{C}_{i}$, and
$\mathbf{C}_{i}=\frac{1}{K}\sum_{j=1}^{K}\mathbf{v}_{j}$. Here $K$ is the
number of samples of class $i$ and $\mathbf{v}_{j}$ is the direct feature
vector for samples from class $i$. Then, we can define our memory feature
vector as follows:

(3.2) $\small\mathbf{v}_{memory}=\sum_{i}w_{i}\mathbf{C}_{i}$

where $w_{i}$ is the weight of the prototype and it depends on
$\mathbf{v}_{direct}$. Concretely, $w_{i}$ is learned by a shallow neural
network whose input is $\mathbf{v}_{direct}$ and output is $w_{i}$. The
parameters of this weight learning network $\theta_{\omega}$ are updated by
SGD and back-propagation via minimizing the loss function, which is embedded
in the learning process of the entire image-net or text-net.

In Eq. (3.1), $\eta$ is the weight for memory feature vector
$\mathbf{v}_{memory}$. Intuitively, if the sample is from head classes, there
are sufficient supervised training data in the same class, so the direct
features $\mathbf{v}_{direct}$ can work well alone. Therefore, the weight of
memory features $\eta$ should be smaller. On the other hand, if the sample is
from tail classes, $\eta$ should be larger. Thus, we define the trade-off
parameter as:

(3.3) $\small\eta=\frac{d_{min}^{tail}}{d_{min}^{head}}$

where
$d_{min}^{head}=\underset{j\in\mathcal{C}_{head}}{\min}|\mathbf{v}_{i}-\mathbf{C}_{j}|_{2}^{2}$
is the minimum distance to the prototypes of the head classes and
$d_{min}^{tail}=\underset{j\in\mathcal{C}_{tail}}{\min}|\mathbf{v}_{i}-\mathbf{C}_{j}|_{2}^{2}$.
In this way, we can ensure that the trade-off parameter $\eta_{i}$ for the
samples of the head classes is small, while $\eta_{i}$ for the samples of the
tail ones is large. In other meta-learning methods like [6], a parameter
similar to $\eta$ is learned by an extra shallow neural network. Our strategy
is more interpretable and more effective, as shown in our experiments.

Table 1: Configuration of Image and Text networks

Network |  Layers |  Configuration
---|---|---
Image-net |  conv1 |  filters:64 $\times$ 11 $\times$ 11, st. 4, pad 0, LRN, $\times$ 2 pool
|  conv2 |  filters:256 $\times$ 5 $\times$ 5, st. 1, pad 2, LRN, $\times$ 2 pool
|  conv3 |  filters:256 $\times$ 3 $\times$ 3, st. 1, pad 1
|  conv4 |  filters:256 $\times$ 3 $\times$ 3, st. 1, pad 1
|  conv5 |  filters:256 $\times$ 3 $\times$ 3, st. 1, pad 1, $\times$ 2 pool
|  fc6 |  nodes:4096
|  fc7 |  nodes:4096
|  fc8 |  nodes:code length $c$
Text-net |  fc1 |  nodes:8192
|  fc2 |  nodes:code length $c$

Figure 3: The meta feature learning network. We consider a linear combination
of prototypes as memory features. The memory and direct features are combined
to obtain the meta features.

###  3.4 Hash code learning

After extracting meta features for each sample, we then focus on how to obtain
the hash code for each sample. Suppose $\mathbf{A}\in\\{0,1\\}^{n\times n}$ is
the affinity matrix of samples and each $a_{ij}=1$ in $\mathbf{A}$ means
sample $i$ is similar to sample $j$, and $a_{ij}=0$ means sample $i$ is not
similar to $j$. Here, we need to learn hash codes and hash functions while
preserving the similarity information in $\mathbf{A}$. Suppose that
$\mathbf{H}^{x}=[\mathbf{h}^{x}_{1},\cdots,\mathbf{h}^{x}_{n}]\in\mathbb{R}^{c\times
n}$ is the hash code matrix obtained from the image modality and
$\mathbf{h}^{x}_{i}$ is the hash code obtained from the image modality for
image $i$;
$\mathbf{H}^{y}=[\mathbf{h}^{y}_{1},\cdots,\mathbf{h}^{y}_{n}]\in\mathbb{R}^{c\times
n}$ is the hash code matrix obtained from the text modality and
$\mathbf{h}^{y}_{i}$ is the hash code obtained from the text modality for text
$i$. Since $\mathbf{A}$ is known, we want to obtain the $\mathbf{H}^{x}$ and
$\mathbf{H}^{x}$ corresponding to the highest probability. Therefore, we apply
a maximum a posteriori approach:

(3.4) $\displaystyle\log p(\mathbf{H}^{x},\mathbf{H}^{y}|\mathbf{A})$
$\displaystyle\propto\log
p(\mathbf{A}|\mathbf{H}^{x},\mathbf{H}^{y})p(\mathbf{H}^{x})p(\mathbf{H}^{y})$
$\displaystyle=\sum_{i,j}\log
p(a_{ij}|\mathbf{h}^{x}_{i},\mathbf{h}^{y}_{j})p(\mathbf{h}^{x}_{i})p(\mathbf{h}^{y}_{j})$

where $p(a_{ij}|\mathbf{h}^{x}_{i},\mathbf{h}^{y}_{j})$ is the likelihood
function, $p(\mathbf{h}^{x})$ and $p(\mathbf{h}^{y})$ are prior probabilities
of $\mathbf{h}^{x}$ and $\mathbf{h}^{y}$, which are unknown fixed values.
Thus, we only consider the definition of the likelihood function
$p(a_{ij}|\mathbf{h}^{x}_{i},\mathbf{h}^{y}_{j})$. As mentioned earlier, the
principle of hashing is to preserve the proximity between samples, i.e., the
smaller the Hamming distance between $\mathbf{h}^{x}_{i}$ and
$\mathbf{h}^{y}_{j}$, the higher the probability that $a_{ij}$ will be 1.
Hamming distance is non-differentiable, thus we should find a surrogate.
$\mathbf{h}^{x}_{i}$ and $\mathbf{h}^{y}_{j}$ here are both binary code
vectors, so the Hamming distance between $\mathbf{h}^{x}_{i}$ and
$\mathbf{h}^{y}_{j}$ can be converted into an inner product:

(3.5) $\small
dist_{H}(\mathbf{h}^{x}_{i},\mathbf{h}^{y}_{j})=\frac{1}{2}(c-\left\langle\mathbf{h}^{x}_{i},\mathbf{h}^{y}_{j}\right\rangle)$

where $c$ is the length of hash codes, and
$\left\langle\mathbf{h}^{x}_{i},\mathbf{h}^{y}_{j}\right\rangle$ denotes the
inner product of $\mathbf{h}^{x}_{i}$ and $\mathbf{h}^{y}_{j}$. From Eq.
(3.5), we know that $dist_{H}(\mathbf{h}^{x}_{i},\mathbf{h}^{y}_{j})$ is
negatively correlated with
$\left\langle\mathbf{h}^{x}_{i},\mathbf{h}^{y}_{j}\right\rangle$. Thus, we
define the likelihood function of $a_{ij}$ as follows:

(3.6) $\displaystyle p(a_{ij}|\mathbf{h}^{x}_{i},\mathbf{h}^{y}_{j})$
$\displaystyle=\begin{cases}\sigma(\phi_{ij})&a_{ij}=1\\\
1-\sigma(\phi_{ij})&a_{ij}=0\end{cases}$
$\displaystyle=\sigma(\phi_{ij})^{a_{ij}}(1-\sigma(\phi_{ij}))^{1-a_{ij}}$

where
$\phi_{ij}=\frac{1}{2}\left\langle\mathbf{h}^{x}_{i},\mathbf{h}^{y}_{j}\right\rangle$
and $\sigma(\phi)=\frac{1}{1+e^{-\phi}}$ denotes the Sigmoid function.
$\sigma(\phi)$ is positively correlated with argument $\phi$, so the smaller
the Hamming distance between $\mathbf{h}^{x}_{i}$ and $\mathbf{h}^{y}_{j}$,
the larger the inner product
$\left\langle\mathbf{h}^{x}_{i},\mathbf{h}^{y}_{j}\right\rangle$ is. This
results in a higher probability that $a_{ij}$ will be 1 and vice versa.
Therefore, Eq. (3.6) is a good approximation of the likelihood function.

Let $\mathbf{v}^{x}_{i}=f^{x}(\mathbf{x}_{i};\theta_{x})\in\mathbb{R}^{c}$ and
$\mathbf{v}^{y}_{j}=f^{y}(\mathbf{y}_{j};\theta_{y})\in\mathbb{R}^{c}$ denote
the learned meta features of the image modality of sample $i$ and the text
modality of sample $j$. $\theta_{x}$ and $\theta_{y}$ are the parameters of
image-net and text-net, respectively.
$\mathbf{V}^{x}=[\mathbf{v}^{x}_{1},\cdots,\mathbf{v}^{x}_{n}]$ and
$\mathbf{V}^{y}=[\mathbf{v}^{y}_{1},\cdots,\mathbf{v}^{y}_{n}]$ are the
feature matrices of the image modality and the text modality. We can define
our loss function as follows:

(3.7)
$\small\begin{split}\underset{\mathbf{B},\theta_{x},\theta_{y}}{min}\mathcal{J}=&-\sum_{i,j=1}^{n}(a_{ij}\Phi_{ij}-log(1+\mathrm{e}^{\Phi_{ij}}))\\\
&+\alpha(||\mathbf{B}-\mathbf{V}^{x}||_{F}^{2}+||\mathbf{B}-\mathbf{V}^{y}||_{F}^{2})\\\
&+\beta(||\mathbf{V}^{x}\mathbf{1}||_{F}^{2}+||\mathbf{V}^{y}\mathbf{1}||_{F}^{2})\\\
s.t.\quad\mathbf{B}\in&\\{+1,-1\\}^{c\times n}\end{split}$

where
$\Phi_{ij}=\frac{1}{2}\langle\mathbf{v}^{x}_{i},\mathbf{v}^{y}_{j}\rangle$ is
the inner product between $\mathbf{v}^{x}_{i}$ and $\mathbf{v}^{y}_{j}$,
$\mathbf{B}$ is the unified hash code. $\mathbf{1}$ is a vector with all
elements being 1. The first term
$-\sum_{i,j=1}^{n}(a_{ij}\Phi_{ij}-log(1+\mathrm{e}^{\Phi_{ij}}))$ is the
negative log likelihood of the affinity matrix $\mathbf{A}$. It is easy to see
that minimizing the first term can make the inner product between
$\mathbf{v}_{i}^{x}$ and $\mathbf{v}_{j}^{y}$ small when $a_{ij}=0$, and large
when $a_{ij}=1$. In other words, samples similar to each other will be
assigned similar meta features. The second term
$||\mathbf{B}-\mathbf{V}^{x}||_{F}^{2}+||\mathbf{B}-\mathbf{V}^{y}||_{F}^{2}$
is the quantization loss. Minimizing this term forces hash codes learnt from
paired samples across modalities to be as similar as possible. The third term
$||\mathbf{V}^{x}\mathbf{1}||_{F}^{2}+||\mathbf{V}^{y}\mathbf{1}||_{F}^{2}$ is
the balance loss; minimizing this term can make the number of zeros and ones
in the hash codes as close as possible, and thus enlarge the coding space.

###  3.5 Optimization

There are three parameters in our loss function Eq. (3.7), $\mathbf{B}$,
$\theta_{x}$, and $\theta_{y}$. It is hard to optimize these parameters
simultaneously. Thus, we optimize these parameters by an alternative strategy.
We iteratively learn one of the three parameters at a time while fixing the
others and give the main idea here.
Optimize $\mathbf{\theta_{x}}$($\mathbf{\theta_{y}}$). We learn the parameters
of image-net $\theta_{x}$ with $\theta_{y}$ and keep $\mathbf{B}$ fixed. We
first calculate the derivative of the loss function $\mathcal{J}$ with respect
to the meta feature vector $\mathbf{v}_{i}^{x}$ for each sample
$\mathbf{x}_{i}$, and then use the back-propagation (BP) algorithm and
stochastic gradient descent (SGD) to update $\theta_{x}$ iteratively.
$\frac{\partial\mathcal{J}}{\partial\mathbf{V}^{x}_{*i}}$ is calculated as
follows:

(3.8)
$\small\begin{split}\frac{\partial\mathcal{J}}{\partial\mathbf{V}^{x}_{*i}}=&\frac{1}{2}\sum_{j=1}^{n}(\sigma(\Phi_{ij})\mathbf{V}^{y}_{*j}-a_{ij}\mathbf{V}^{y}_{*j})\\\
&+2\alpha(\mathbf{V}^{x}_{*i}-\mathbf{B}_{*i})+2\beta\mathbf{V}^{x}\mathbf{1}\end{split}$

where $\mathbf{V}^{x}_{*i}$ denotes the $i$-th column of $\mathbf{V}^{x}$.
Then we can compute $\frac{\partial\mathcal{J}}{\partial\theta_{x}}$ using the
chain rule, and then update $\theta_{x}$ based on BP. The optimization of
$\theta_{y}$ is similar.
Optimize $\mathbf{B}$. When $\theta_{x}$ and $\theta_{y}$ are fixed, the
optimization problem can be reformulated as:

(3.9) $\displaystyle\max_{\mathbf{B}}\quad$ $\displaystyle
tr(\mathbf{B}(\mathbf{V}^{x}+\mathbf{V}^{y})^{T})$ s.t.
$\displaystyle\mathbf{B}\in\\{+1,-1\\}^{c\times n}$

We can easily solve this problem by letting $\mathbf{B}_{ij}$ have the same
sign as $(\mathbf{V}^{x}+\mathbf{V}^{y})_{ij}$:

(3.10) $\small\mathbf{B}=sign(\mathbf{V}^{x}+\mathbf{V}^{y})$

##  4 Experiments

###  4.1 Experimental setup

We conduct our experiments on two different long-tailed datasets. As we stated
earlier, most existing cross-modal hashing methods only focus on balanced
data. There is no off-the-shelf benchmark multi-modal long-tailed dataset.
Given this, we manually filter and modify the two datasets _Flickr_ [36] and
_NUS-wide_ [37]. The original _Flickr_ dataset contains of 25,015 images
collected from the Flickr website. Each image is annotated with its
corresponding textual tags and at least one label out of 24 different semantic
labels. The textual tags of each image are represented with a 1386-dimensional
BOW (bag of word) vector. Each image-tag pair is treated as an sample. For
methods based on hand-crafted features rather than raw picture data, each
image is converted to a 512-dimensional GIST vector. The original _NUS-wide_
dataset contains 269,648 images and corresponding textual tags collected from
Flickr website. Each image-tag pair is treated as an sample and each sample is
labeled with at least one label of 81 concepts. The textual tags are converted
into a series of 1000-dimensional BOW vectors. For hand-crafted features based
methods, each image is converted into a 500-dimensional bag-of-visual-words
(BOVW) vector.

We convert these two datasets into long-tailed datasets in a similar way.
Because most samples in the two datasets are labeled with not only one label,
we first get rid of the excess labels on the samples. For samples with more
than 3 labels, we randomly keep 2-3 high discriminative labels. Here we assume
that class labels with fewer samples are more discriminative. For example, a
tiger has two labels (“animal” and “feline”). Obviously, more samples have the
label “animal”, and fewer the label “feline”. To keep the tail labels, we
select samples and construct our long-tailed training sets. We select a few
classes with a large number of samples and a large number of classes with a
few samples (the specific quantities are shown in Table 2). In this way, the
number of samples in different classes follow a long-tailed distribution. As
for the test (query) set, we select 50 samples from each class. We use the
whole original dataset (excluding samples in the training set) as the
retrieval set.

Table 2: Distribution of head and tail classes of the training set. Dataset | Item | head | tail1 | tail2
---|---|---|---|---
Flickr | Number of classes | 4 | 10 | 10
Samples in each class | 2000 | 200 | 50
NUS-wide | Number of classes | 9 | 35 | 35
Samples in each class | 1000 | 100 | 25

We use five representative methods for empirical comparison, including Deep
Cross-Modal Hashing (DCMH) [20], Semantic Correlation Maximization (SCM-seq
and SCM-orth) [18], Semantics Preserving Hashing (SePH) [38], and Collective
Matrix Factorization Hashing (CMFH) [9]. SCM-seq, SCM-orth, and SePH are
supervised methods with shallow architectures. DCMH and SDCH are supervised
methods with deep architectures. CMFH is an unsupervised hash learning method.
In addition, MetaCMH-v is introduced as a variant of MetaCMH and it uses a
shallow neural network to learn the parameter $\eta$ rather than adopting Eq.
(3.3). For each comparison method, we use the hyper-parameters recommended in
the corresponding article. For MetaCMH and MetaCMH-v, we set $\alpha=\beta=1$.
We use the canonically used mean average precision (MAP) [17, 18] to quantify
the performance of hashing retrieval methods.

###  4.2 Results and Analysis

We report the experimental results in Table 3, where “I to T” means the query
is the image and the retrieval data is the text, and “I to T” means the query
is the text and the retrieval data is the image. The standard deviations of
MAP results of compared methods are quite small (generally less than 3%), thus
we do not report the standard deviations in the table. From Table 3, we have
the following observations and conclusions:

Table 3: Mean Average Precision of all methods on Flickr and NUS-wide.

Dataset | Flickr | NUS-wide
---|---|---
Class | All classes | Head Classes | Tail Classes | All classes | Head Classes | Tail Classes
bits | 32bit | 64bit | 32bit | 64bit | 32bit | 64bit | 32bit | 64bit | 32bit | 64bit | 32bit | 64bit
I to T | DCMH | 0.492 | 0.506 | 0.597 | 0.601 | 0.351 | 0.367 | 0.387 | 0.386 | 0.496 | 0.501 | 0.258 | 0.255
CMFH | 0.377 | 0.382 | 0.444 | 0.446 | 0.265 | 0.274 | 0.256 | 0.261 | 0.289 | 0.302 | 0.204 | 0.215
SCM-seq | 0.387 | 0.393 | 0.451 | 0.467 | 0.255 | 0.259 | 0.264 | 0.269 | 0.304 | 0.307 | 0.213 | 0.219
SCM-orth | 0.386 | 0.392 | 0.462 | 0.477 | 0.251 | 0.255 | 0.265 | 0.271 | 0.316 | 0.319 | 0.217 | 0.225
SePH | 0.467 | 0.475 | 0.510 | 0.521 | 0.339 | 0.346 | 0.314 | 0.326 | 0.417 | 0.421 | 0.249 | 0.254
MetaCMH | 0.521 | 0.527 | 0.581 | 0.587 | 0.485 | 0.491 | 0.431 | 0.440 | 0.505 | 0.509 | 0.307 | 0.319
MetaCMH-v | 0.501 | 0.503 | 0.590 | 0.594 | 0.401 | 0.433 | 0.420 | 0.427 | 0.507 | 0.514 | 0.287 | 0.289
T to I | DCMH | 0.510 | 0.514 | 0.583 | 0.580 | 0.376 | 0.381 | 0.413 | 0.416 | 0.541 | 0.543 | 0.303 | 0.307
CMFH | 0.387 | 0.396 | 0.435 | 0.437 | 0.277 | 0.284 | 0.279 | 0.287 | 0.312 | 0.322 | 0.235 | 0.239
SCM-seq | 0.399 | 0.406 | 0.491 | 0.495 | 0.261 | 0.269 | 0.295 | 0.297 | 0.327 | 0.336 | 0.247 | 0.253
SCM-orth | 0.404 | 0.413 | 0.499 | 0.501 | 0.265 | 0.276 | 0.299 | 0.312 | 0.335 | 0.336 | 0.259 | 0.264
SePH | 0.463 | 0.475 | 0.524 | 0.526 | 0.367 | 0.374 | 0.337 | 0.346 | 0.448 | 0.457 | 0.297 | 0.305
MetaCMH | 0.541 | 0.543 | 0.588 | 0.590 | 0.495 | 0.503 | 0.458 | 0.462 | 0.538 | 0.547 | 0.356 | 0.359
MetaCMH-v | 0.522 | 0.523 | 0.591 | 0.593 | 0.404 | 0.415 | 0.435 | 0.439 | 0.541 | 0.550 | 0.290 | 0.302

(i) The methods with a deep architecture (DCMH, MetaCMH and MetaCMH-v)
outperform solutions with a shallow architecture (CMFH, SCM-seq, SCM-orth,
SePH), and the former methods perform better than the latter ones on both head
and tail classes. Therefore, methods with deep architectures achieve higher
overall MAP values. This is mainly because the deep neural networks based
methods have a stronger feature extraction capability than shallow ones.

(ii) Supervised methods perform better than unsupervised methods (CMFH). This
is very intuitive, since supervised methods take advantage of class labels
which can effectively improve the performance of hashing. The gap between CMFH
and SCM-seq (or SCM-orth) is small because SCM simply uses a linear
transformation to learn the hash function while trying to reconstruct the
cosine similarity matrix with obtained hash codes, and it ignores structure
information. In contrast, CMFH learns a latent semantic space via feature
matrix decomposition, which can mine structure information.

(iii) MetaCMH uses a similar hash code learning method as DCMH, but DCMH only
uses a basic deep neural network to learn the features of instances. We
observe that, DCMH performs well on head classes, but it clearly loses to
MetaCMH on tail ones. This is due to the fact that the memory-augmented
network can transfer meta knowledge from head classes to tail ones, and can
enhance the effectiveness on tail classes. We also observe that in some cases,
the MAP values of DCMH on head classes are higher than those of MetaCMH. This
is because for head classes with abundant training samples, the basic deep
neural network can learn features of instances very well, and the extra memory
features may in part damage the direct features.

(iv) MetaCMH-v achieves results comparable to MetaCMH on head classes, but it
loses to MetaCMH on tail classes. This is because MetaCMH-v uses a shallow
neural network to learn the weight of memory features, which does not
guarantee that the weights of memory features of instances from the head
classes are significantly smaller than those from tail ones. As a result,
direct features of instances from head classes are well learned, but those for
tail classes are not, and the lower weights of instances from tail classes
cannot well enhance the direct features.

Overall, these results and the ablation study confirm the necessity and
effectiveness of MetaCMH on performing meta feature learning for long-tailed
data.

###  4.3 Parameter analysis

In the MetaCMH model, there are two hyper-parameters: $\alpha$ and $\beta$. We
conduct a series of experiments on the long-tailed _Flicrk_ dataset to study
the influence of these two parameters. We set $\beta=1$ when testing the
sensitivity of $\alpha$ and set $\alpha=1$ when testing the sensitivity of
$\beta$. Results are shown in Figure 4 where the hash code length is fixed to
32 bits. We observe that when $\alpha=1$, the MAP values reach their maximum
values, and the model works best. As $\alpha$ increases or decreases, the
effectiveness of the model decreases. Compared to $\alpha$, the model is less
sensitive to the input value of $\beta$, but the overall pattern is similar as
$\alpha$. When $\beta=1$, the MAP values reach their maximum values, and the
model works best. Overall, a too small $\alpha$ cannot ensure consistent
hashing codes across modalities, while a too small $\beta$ cannot generate
balanced binary codes. On the other hand, over-weighing them brings down the
quality of cross-modal hashing functions. From these observations, we adopt
$\alpha=1$ and $\beta=1$ in the experiments.

(a) $\alpha$ vs. MAP

(b) $\beta$ vs. MAP

Figure 4: Parameter sensitivity analysis of $\alpha$ and $\beta$ on the long-
tailed _Flickr_ dataset.

###  4.4 Ablation study

In this section, we conduct a series of ablation study to validation the
effectiveness of each components of the whole model. Thus, we propose several
variations of MetaCMH.

##  5 Conclusion

In this paper, we proposed a novel cross-modal hashing method called Meta
Cross-Modal Hashing (MetaCMH) for long-tailed data. MetaCMH introduces a meta-
learning framework into cross-modal hashing and uses a memory-augmented
network to transfer meta knowledge from head classes to tail ones. Our
experiments on two long-tailed datasets show that meta-learning and CMH can
work well together, and MetaCMH outperforms competitive methods on long-tailed
datasets. The codes of MetaCMH are available at
http://mlda.swu.edu.cn/codes.php?name=MetaCMH.

## References

  * [1] Jun Wang, Wei Liu, Sanjiv Kumar, and Shih Fu Chang.  Learning to hash for indexing big data - a survey.  Proceedings of the IEEE, 104(1):34–57, 2016.
  * [2] Jingdong Wang, Ting Zhang, Jingkuan Song, Nicu Sebe, and Heng Tao Shen.  A survey on learning to hash.  TPAMI, 40(4):769–790, 2018.
  * [3] Yair Weiss, Antonio Torralba, and Rob Fergus.  Spectral hashing.  In NeurIPS, pages 1753–1760, 2009.
  * [4] Yunchao Gong, Svetlana Lazebnik, Albert Gordo, and Florent Perronnin.  Iterative quantization: A procrustean approach to learning binary codes for large-scale image retrieval.  TPAMI, 35(12):2916–2929, 2013.
  * [5] William J Reed.  The pareto, zipf and other power laws.  Economics Letters, 74(1):15–19, 2001.
  * [6] Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu.  Large-scale long-tailed recognition in an open world.  In CVPR, pages 2537–2546, 2019.
  * [7] Yu-Xiong Wang, Liangke Gui, and Martial Hebert.  Few-shot hash learning for image retrieval.  In ICCV, pages 1228–1237, 2017.
  * [8] Chengcheng Liu, Huikai Shao, Dexing Zhong, and Jun Du.  Siamese-hashing network for few-shot palmprint recognition.  In SSCI, pages 3251–3258, 2019.
  * [9] Guiguang Ding, Yuchen Guo, and Jile Zhou.  Collective matrix factorization hashing for multimodal data.  In CVPR, pages 2075–2082, 2014.
  * [10] Yahui Xu, Yang Yang, Fumin Shen, Xing Xu, Yuxuan Zhou, and Heng Tao Shen.  Attribute hashing for zero-shot image retrieval.  In ICME, pages 133–138, 2017.
  * [11] Xuanwu Liu, Zhao Li, Jun Wang, Guoxian Yu, Carlotta Domeniconi, and Xiangliang Zhang.  Cross-modal zero-shot hashing.  In ICDM, pages 449–458, 2019.
  * [12] Z. Ji, Y. Sun, Y. Yu, Y. Pang, and J. Han.  Attribute-guided network for cross-modal zero-shot hashing.  IEEE Transactions on Neural Networks, 31(1):321–330, 2020.
  * [13] Joaquin Vanschoren.  Meta-learning: A survey.  arXiv preprint, 2018.
  * [14] Sepp Hochreiter, A Steven Younger, and Peter R Conwell.  Learning to learn using gradient descent.  In ICANN, pages 87–94, 2001.
  * [15] Shaishav Kumar and Raghavendra Udupa.  Learning hash functions for cross-view similarity search.  In IJCAI, pages 1360–1365, 2011.
  * [16] G. Wu, Z. Lin, J. Han, L. Liu, G. Ding, B. Zhang, and J. Shen.  Unsupervised deep hashing via binary latent factor models for large-scale cross-modal retrieval.  In IJCAI, pages 2854–2860, 2018.
  * [17] Michael M Bronstein, Alexander M Bronstein, Fabrice Michel, and Nikos Paragios.  Data fusion through cross-modality metric learning using similarity-sensitive hashing.  In CVPR, pages 3594–3601, 2010.
  * [18] Dongqing Zhang and Wujun Li.  Large-scale supervised multimodal hashing with semantic correlation maximization.  In AAAI, pages 2177–2183, 2014.
  * [19] Zijia Lin, Guiguang Ding, Mingqing Hu, and Jianmin Wang.  Semantics-preserving hashing for cross-view retrieval.  In CVPR, pages 3864–3872, 2015.
  * [20] Qing-Yuan Jiang and Wu-Jun Li.  Deep cross-modal hashing.  In CVPR, pages 3232–3240, 2017.
  * [21] C. Li, C. Deng, N. Li, W. Liu, X. Gao, and D. Tao.  Self-supervised adversarial hashing networks for cross-modal retrieval.  In CVPR, pages 4242–4251, 2018.
  * [22] X. Liu, G. Yu, C. Domeniconi, J. Wang, G. Xiao, and M. Guo.  Weakly-supervised cross-modal hashing.  TBD, 99(1):1–14, 2020.
  * [23] Xuanwu Liu, Jun Wang, Guoxian Yu, Carlotta Domeniconi, and Xiangliang Zhang.  Weakly-paired cross-modal hashing.  TNNLS, 99(1):1–11, 2020.
  * [24] Sinno Jialin Pan and Qiang Yang.  A survey on transfer learning.  TKDE, 22(10):1345–1359, 2010.
  * [25] Chelsea Finn, Pieter Abbeel, and Sergey Levine.  Model-agnostic meta-learning for fast adaptation of deep networks.  In ICML, page 1126¨C1135, 2017.
  * [26] Chelsea Finn, Kelvin Xu, and Sergey Levine.  Probabilistic model-agnostic meta-learning.  In NeurIPS, pages 9516–9527, 2018.
  * [27] Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell.  Meta-learning with latent embedding optimization.  In ICRL, 2019.
  * [28] Jake Snell, Kevin Swersky, and Richard S Zemel.  Prototypical networks for few-shot learning.  In NIPS, pages 4077–4087, 2017.
  * [29] Siyuan Qiao, Chenxi Liu, Wei Shen, and Alan L Yuille.  Few-shot image recognition by predicting parameters from activations.  In CVPR, pages 7229–7238, 2018.
  * [30] Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap.  Meta-learning with memory-augmented neural networks.  In ICML, pages 1842–1850, 2016.
  * [31] Alex Graves, Greg Wayne, and Ivo Danihelka.  Neural turing machines.  arXiv: Neural and Evolutionary Computing, 2014.
  * [32] Tsendsuren Munkhdalai and Hong Yu.  Meta networks.  In ICML, pages 2554–2563, 2017.
  * [33] Ken Chatfield, Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman.  Return of the devil in the details: Delving deep into convolutional nets.  In British Machine Vision Conference, pages 1–12, 2014.
  * [34] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  In NeurIPS, pages 1097–1105, 2012.
  * [35] Sepp Hochreiter and Jurgen Schmidhuber.  Long short-term memory.  Neural Computation, 9(8):1735–1780, 1997.
  * [36] Mark J Huiskes and Michael S Lew.  The mir flickr retrieval evaluation.  In ACM MIRM, pages 39–43, 2008.
  * [37] Tatseng Chua, Jinhui Tang, Richang Hong, Haojie Li, Zhiping Luo, and Yantao Zheng.  Nus-wide: a real-world web image database from national university of singapore.  In CIVR, pages 48–52, 2009.
  * [38] Zijia Lin, Guiguang Ding, Mingqing Hu, and Jianmin Wang.  Semantics-preserving hashing for cross-view retrieval.  In CVPR, pages 3864–3872, 2015.