# Spontaneous Human Combustion
rules out all standard candidates for Dark Matter

Frederic V. Hessman  Institut für Astrophysik und Geophysik, University of
Göttingen  J. Craig Wheeler  Dept. of Astronomy, University of Texas at Austin

(April 1, 2023)

###### Abstract

We argue that the reported cases of Spontaneous Human Combustion (SHC) are
most likely due to the impact of the human body with an extremely high energy
particle like cosmic rays or Dark Matter. Normal and antimatter cosmic rays
and classical weakly-interacting massive particles (WIMPs) with energies of
GeV to ZeV can be easily ruled out due to their inability to dump enough
energy into a small region of human tissue, leaving as the single remaining
candidate massive Dark Matter particles. While primordial Black Holes would
appear to be very good candidates for inducing the SHC phenomenon, we show
that the estimated local Dark Matter density requires that the particles have
masses of $\sim 10$ kg, clearly ruling out this candidate. All of the other
classic DM candidates – from scalar and pseudo-scalar spin 1/2 and spin 2
gauge singlets to nuclearitic strange quark “bowling balls” – can be ruled
out. Axions tailored to solve the CP-problem also cannot be invoked, no matter
what mass is considered. The only particles left are massive mega-axions
(MaMAs), for which there are an infinite number of possible string models.

cosmic rays – dark matter – black holes – particles: axions – medical history

††preprint: arXiv: April 1, 2023

##  I Introduction

Given all of the indirect signs of the existence of Dark Matter (DM) – the
rotation curves of spiral galaxies, the velocity dispersions within and
macroscopic gravitational lensing from galaxy clusters, the angular structure
of the cosmic microwave background, observations of the baryonic acoustic
oscillations in the large-scale structure of the universe – it is particularly
frustrating that the search for the underlying particle has been so
unsuccessful. The limits on classical weakly-interacting massive particles
(WIMPs) in the range of $\sim$1-1000 GeV are nearly at the point of rejecting
this class of candidate. A spate of new detector experiments is in the process
of doing the same for very light particles ($\leq 1\,eV$) like axions. On the
other end of the conceivable mass-range, the limits on the density of
primordial black holes (BH) are now also very restrictive, including
experiments searching for the signs of BH passages though kitchen cabinet tops
[1]. Similarly, the gravitational microlensing surveys like MACHO and EROS in
our Galaxy and Quasar microlensing observations have restricted the mass-
density of stellar- and planetary-mass candidates [2, 3]. The only DM particle
mass range that has not yet been so exhaustively probed is that between the
WIMPs and primordial BH, i.e. between about 1 TeV and $10^{11}$ kg. On the low
end of this mass range – less than about 1 gram – is so-called “nuclearitic”
matter with nuclear densities [4] or generic “macro” DM with cross-sections of
$\sim\\!cm^{2}$ [5].

We consider what constraints on the DM particle mass can be derived based on a
natural experiment that may have been carried out since the beginning of human
life on Earth – Spontaneous Human Combustion (SHC) [6].

##  II Spontaneous Human Combustion

SHC is the reported occurrence of massive fatal burns in humans with no
obvious explanation for the source of the ignition. These events purportedly
originate from within the body of the victims and generally result in no
damage to the surroundings other than that due to contact with the body
itself.

The idea of SHC was first reported in the scientific literature in 1746 [7].
Thurston [8] maintained in the British Medical Journal that the phenomenon “…
attracted the attention … of the medical profession” throughout the 19th
century. Arnold [9] relates the detailed history of the idea and reviews the
most prominent cases. All in all, there have been about 200 well-documented
reports during the last 300 years, mostly within Europe and North America. The
assumption is that occurrences before the 18th century or in less developed
societies would have been explained as magical events due to sinister
supernatural forces and so gone largely undocumented. Of course, humans are
not the only living beings that would be suffering from this effect – there
should be just as many spontaneously combusted sheep, cows, raccoon dogs, and
African swallows – but the occurrence of such an event in a non-human victim
is much less likely to be noticed as something extremely odd and much less
likely to produce such an emotional impact that it gets reported as such.

While there have been several rational/medical attempts at explaining SHC –
sleeping while smoking, acetone produced by kertosis [10], and the mast-cell
activation syndrome [11] – these mundane medical explanations have never been
convincing. Given our current knowledge of the types of highly energetic
particles in the universe, we propose that SHC is much more likely to have
been caused by a collision with a cosmic particle. Such an interaction with
human tissue could principally result in the sudden release of considerable
energy within a fairly small volume [12]. Once started, such an event could
then induce the combustion of fatty tissue, resulting in a fatal burning event
with a natural origin within the body itself. Likely physical candidates are
the most extreme cosmic ray particles or Dark Matter particles, being
omnipresent and potentially having large specific kinetic energies.

##  III Cosmic rays and WIMPs?

Most cosmic rays have energies of MeV to GeV, but some have been observed up
to energies of ZeV [13]. As such, they would seem to be good candidates for
the occurrence of SHC. While they have enough energy to cause considerable
damage, this energy must be deposited in a small volume in order to cause SHC.
Clinical studies have shown that the higher energy particles do not, in fact,
have this effect [14] : considerable damage can be done by single collisions
with, e.g. DNA molecules, but the entire energy of the particle is not
deposited. Lower energy particles interact more often and hence can do more
thermal damage, but their total energies are not enough to cause combustion.
Thus, normal atomic and nuclear particles are inherently poor SHC candidates.
The same argument can be made for weakly-interacting massive particles (WIMPs)
with similar energies that are the classic Dark Matter candidates [12].

Cosmic rays out of antimatter could, in principle, result in the release of
lots of energy in a very localized region. The problem with this hypothesis is
that all of the energy needed to trigger an SHC event must come from a single
annihilation that produces both the right amount and the right type of
annihilation radiation. The energy involved in triggering a single SHC event
that initially burns a fraction $f_{SHC}$ of it’s victim is roughly

$\displaystyle E_{SHC}$ $\displaystyle\approx$
$\displaystyle~{}m_{h}~{}f_{SHC}~{}\epsilon_{fat}$ $\displaystyle\approx$
$\displaystyle 2\times
10^{6}\,\textrm{J}~{}\left(\frac{m_{h}}{60\,\textrm{kg}}\right)\left(\frac{f_{SHC}}{10^{-3}}\right)\left(\frac{\epsilon_{fat}}{17000\,\textrm{BTU/lb}}\right)$

where $m_{h}$ is the mass of a typical victim and $\epsilon_{fat}$ is the
energy produced by burning animal fat [15]. The equivalent annihilation rest
mass is $3\times 10^{-11}$ kg or an atomic number equivalent of $2\times
10^{16}$! Even if one could trigger the event with a factor of 1000 less fatty
tissue, the cosmic ray nucleus involved would have an atomic number far beyond
that of any known element. If the annihilation event produced high-energy
radiation (X-rays, gamma rays), the radiation would simply exit the victim’s
body and the burning of the tissue wouldn’t be triggered.

All of these effects prove that normal matter in the form of energetic
particles or even massive antimatter cosmic rays are not capable of inducing
SHC. We must turn to forms of DM other than WIMPs in order to find an
appropriate candidate.

##  IV Primordial Black Holes?

One way to increase the interaction of an energetic particle with human tissue
is to change the interaction mechanism from simple collisions to, for example,
accretion. Primordial Dark Matter in the form of BH is thus a plausible
candidate: the particles would be universally present, have high kinetic
energies, and the energy released into the tissue would be due to the intense
radiation produced by mass-accretion near the singularity.

In order to exist now, primordial BH must have masses larger than

$M_{BH,primordial}>\left(\frac{13.4\,\textrm{Gyr}~{}c^{4}}{5120\pi\,G^{2}}\right)^{1/3}=1.7\times
10^{11}\,\textrm{kg}$ (2)

(about the mass of a major mountain range; [16]). Carr et al. [3] suggest that
primordial BH should come in a distinct range of masses – about $10^{-6}$,
$1$, $30$, and $10^{6}$ M⊙ – due to instabilities at expected QCD state
transitions, and that such objects would fulfill all of the known
astrophysical constraints.

SHC can add an additional constraint on the possible masses: could the
collision of a primordial BH with a human result only in damage to the victim
and not to the surroundings? In particularly, wouldn’t there be a noticeable
gravitational ”jolt” due to the passage of a large mass that would cause
collateral damage that has not been reported in SHC cases (see [16] for a
simulation of such an encounter)?

The asymptotic perpendicular velocity given to a human by the hyperbolic
passage of a significantly more massive BH in the laboratory frame is

$\displaystyle V_{\bot}$ $\displaystyle\approx$
$\displaystyle\frac{GM_{BH}}{bV_{DM}}$ $\displaystyle\approx$ $\displaystyle
0.05\,\textrm{cm/s}~{}\left(\frac{M_{BH}}{2\,10^{11}\,\textrm{kg}}\right)\\!\left(\frac{b}{10\,\textrm{cm}}\right)^{-1}\\!\left(\frac{V_{DM}}{220\,\textrm{km/s}}\right)^{-1}$

for an impact parameter $b$ small enough so that SHC occurs, a typical local
Galactic DM kinematic velocity, and a minimal-mass primordial BH. Such a
gravitational “jolt” is unlikely to have been noticed, much less resulted in
any collateral damage, knocking victims off of chairs or otherwise noticeable
effects. Thus, the only effects would be due to the release of energy due to
the effects of the collision itself. However, when a planet-sized primordial
QCD BH is substituted, the jolt is relativistic, a case clearly ruled out by
SHC: if there are primordial BH, they must have “small” masses near the
cosmological lifetime limit.

The Schwarzschild radius of a minimum-mass primordial BH with this mass is
minuscule even by subatomic scales, but all matter within the Hoyle-Littleton
radius [17] of the path

$r_{HL}=\frac{2GM_{BH}}{V_{DM}^{2}}\approx 0.47\,nm$

should be accreted during passage. This cross-section is small, but the high
density of tissue results in an accretion rate of

$\dot{M}_{HL}=\frac{4\pi G^{2}M_{BH}^{2}\rho_{human}}{V_{DM}^{3}}\approx
1.5\times 10^{-10}\,\textrm{kg/s}$ (4)

or an instantaneous accretion power of about

$L_{acc}\approx 0.1\dot{M}_{HL}c^{2}\approx\textrm{1400\,kW}$ (5)

i.e. more than enough energy input to cause fatty tissue to inflame. The
accretion energy will be emitted with a typical equivalent black-body
temperature of

$T_{acc}>\left(\frac{3GM_{BH}\dot{M}_{HB}}{8\pi
r_{HL}^{2}\sigma}\right)^{1/4}\approx 2\times 10^{6}\,\textrm{K}$ (6)

The soft X-ray and EUV part of this energy will be deposited locally due to
the rapidly rising opacity of human tissue towards the extreme ultraviolet
[18], producing highly localized heating and, hence, nearly spontaneous
combustion.

Though the available power is large enough, the interaction time is very
short, of order $L_{h}/V_{DM}\approx 2\,\mu$s, resulting in a very small total
energy release. In order to increase the total effect, one can attempt to use
a primordial BH with the largest mass consistent with SHC: assuming that the
asymptotic perpendicular velocity of the victim can be no larger than $\sim
10$ cm/s (otherwise, the victim would be ejected from her chair), the BH
cannot be more massive than about

$\displaystyle M_{BH}$ $\displaystyle\approx$
$\displaystyle\frac{bV_{DM}V_{\bot}}{G}$ $\displaystyle\approx$ $\displaystyle
3\times
10^{13}\,\textrm{kg}~{}\left(\frac{b}{10\,\textrm{cm}}\right)\\!\left(\frac{V_{DM}}{220\,\textrm{km/s}}\right)\\!\left(\frac{V_{\bot}}{10\,\textrm{cm/s}}\right)$

and would result in an instantaneous accretion power of $\sim 51$ GW and a
total energy deposition of $5\times 10^{4}$ J – perhaps enough to cause SHC
(see Eqn. III). Thus, if primordial BH exist and collide with humans, they
will result in the observed SHC phenomena if they have masses less than about
$10^{13}$ kg. The final question is whether the occurrence rate of SHC is
compatible with the existence of primordial BH.

##  V Generic constraints from the local DM density

The local DM density has been estimated using a variety of methods, e.g. based
upon the rotation curve of the Milky Way and the kinematics of local stars
[19, 20, 21]. Despite many questions about the data and the reliability of the
models [22, 23], the canonical value is about 0.01 M⊙/pc3 (0.4 GeV/cm3). Given
this estimate for the local DM density, it is possible to constrain the mass
of the purported particle using the occurrence rate of SHC alone. Using a
typical human height of 160 cm during the 18th and 19th centuries [24], the
corresponding cross-sectional surface area per human $A_{human}$ was about
$0.4$ m2. The DM flux is then

$\displaystyle\phi_{SHC}$ $\displaystyle\approx$
$\displaystyle\frac{N_{events}}{\Delta t_{events}N_{pop}A_{human}}$ (8)
$\displaystyle\approx$ $\displaystyle 2\times
10^{-17}\,\textrm{events}/\textrm{m}^{2}/\textrm{s}$ (9)

where $N_{pop}\approx 2\times 10^{9}$ is the mean population of the Earth
during the last 300 years. For a typical local DM particle velocity of
$220\,km/s$, the mass of the DM particle must then be

$M_{DM}=\frac{\rho_{DM}V_{DM}}{\phi_{SHC}}~{}\approx~{}8\,kg~{}\approx~{}4\times
10^{27}\,\textrm{GeV}$ (10)

This value is an upper limit to the actual mass, since it is proportional to
$N_{pop}$ and it is unclear what fraction of the total population at what
epoch would have reported cases of SHC. For instance, if one only considers
the populations of Europe and North America, where most of the reports of SHC
have been made, the relevant mean population size would drop to about 450
million and the corresponding DM particle mass to 2 kg. Thus, DM particles can
best be described as astrophysical “bowling balls” and certainly cannot be due
to primordial BH produced by QCD instabilities.

##  VI SHC-consistent DM candidates?

Given a DM particle mass of $\sim$10 kg, most of the classic DM candiates can
be ruled out.

  * •

WIMPs are not massive enough and cannot provide the needed SHC interaction
cross-sections.

  * •

Neutrinos are obviously bad candidates for SHC interactions even if it was
possible to scale them to such high masses.

  * •

Primordial BH must have drastically larger masses in order to have survived
until now and would have SHC effects that would be much stronger than
observed.

  * •

There is no known mechanism to turn baryonic MACHO candidate objects like
planets and brown dwarfs into microscopic bowling balls.

  * •

Scalar, pseudo- scalar, spin 1/2 and spin 2 dark matter particles which are
gauge singlets are ruled out [25].

  * •

Massive primordial strange quark matter “nuggets” [4] with baryon numbers of
only $A\simeq 10^{28}$ would not have survived baryon evaporation [26].

We are only left with axions [27] : not only do they have a cool name but one
can assume whatever mass one wants.

Axions fulfilling the SHC constraints must have masses that are drastically
larger than those previously considered. The Pecci-Quinn mechanism for
explaining CP-violation [28] – the original reason for postulating the
existence of axions – has an energy scale $f_{A}$ that determines the mass of
the axion and the strength of all couplings. A high axion mass would force
this energy scale to be extremely low: classically

$f_{A}\approx
10^{12}\,\textrm{GeV}~{}\left(\frac{5.70\,\mu\textrm{eV}}{m_{A}}\right)$ (11)

but for a mass of 10 kg ($6\times 10^{27}$ GeV), this scale shrinks down to
about $10^{-21}$ eV – way below any QCD scale. This has profound implications
for the cosmological use of axions [29] : whereas “normal” axions with small
masses are massless in the early universe, SHC axion cannonballs are not and
never will be.

An extremely small axion energy scale has dramatic consequences for couplings
with photons and matter. For example, in a standard Lagrangian, the axion-
photon coupling constant $g_{A}$ is

$g_{A}\equiv\frac{\alpha}{2\pi}\frac{C_{A\gamma}}{f_{A}}$ (12)

where the adimensional constant

$C_{A\gamma}=\frac{E}{N}-\frac{2}{3}\frac{4m_{d}+m_{u}}{m_{u}+m_{d}},$ (13)

depends upon the mass of up and down quarks and the ratio of two anomaly
coefficients $E$ and $N$ (the axion “model”). For an extremely low $f_{A}$ and
$C_{A\gamma}\sim O(1)$, the coupling would be so strong that every photon
would immediately create a shower of axions. The only reasonable alternative
is that the two $C_{A\gamma}$ terms nearly cancel each other, amazingly
producing a “normal” coupling constant despite the tiny axion energy scale.
Given that we need an SHC-interaction with the axion, the canceling cannot be
perfect ($C_{A\gamma}\not\equiv 0$), so a PQ-axion explanation comes at the
price of yet another symmetry breaking.

Since there are good reasons for having “normal” axions (CP-violation) and
there is no reason there cannot be many different axion varieties – in string
theory, there can be an infinite number of varieties [30]. We need to invoke a
fully new massive mega-axion (“MaMA”) model beyond the PQ-mechanism, one that
is tailored to maximize the transfer of energy to human tissue once a
collision occurs. Since high-energy particles are not good candidates for this
transfer – otherwise they themselves would have been good SHC candidates – we
require the post-collision creation of massive numbers of soft X-ray and EUV
photons. Indeed, this scenario is not too much different from that used to
detect neutrinos: the giant containers used in instruments like
SuperKamiokande contains water with similar densities as human tissue and the
containers are monitored by thousands of photomultiplier tubes that detect the
Cherenkov light produced by electrons created by neutrino collisions. If
SuperKamiokande has not detected SHC-like events, then practically all of the
light must be of higher frequency than that seen by photomultipliers – i.e.
EUV and beyond – and of low enough frequency to be absorbed locally by the
tissue, i.e. soft X-rays. Alternatively, the MaMAs react not with single
protons but with the higher mass (carbon, oxygen,…) nuclei, so that
experiments like SuperKamiokande would not show any SHC-like events. Given the
largely unconstrained freedoms of string theory, we are confident that an
appropriate model for an axion-like particle with bowling-ball masses can be
found.

##  VII Conclusions

We have argued that the reports of spontaneous human combustion can only be
due to interactions with astrophysical particles and that such particles must
necessarily be those responsible for Dark Matter. The SHC phenomena and
observed rate are powerful constraints on the properties of such particles:
the mass must be of order 10 kg. This is a mass that is wholly unexplored in
both theory and experiment and is totally inconsistent with all previous
hypotheses. We are left with a single plausible candidate: massive mega-axions
– “MaMAs” – for which there undoubtedly exists a string model consistent with
the SHC phenomenon.

###### Acknowledgements.

We want to thank David “Doddy” Marsh for brief hallway conversations about
axions that led us to think we might know what we are doing. This research was
supported by a grant from the European Association of Fire-Extinguisher
Manufacturers. JCW would like to acknowledge additional support from the Jim
Bob and Betty Sue Johnson Foundation for Non-Woke Cosmological Research.

## References

  * Sidhu  _et al._ [2019] J. S. Sidhu, G. Starkman, and R. Harvey, Counter-top search for macroscopic dark matter, Phys. Rev. D 100, 103015 (2019).
  * Hawkins [2015] M. R. S. Hawkins, A new look at microlensing limits on dark matter in the Galactic halo, Astron. Astrophys. 575, A107 (2015), arXiv:1503.01935 [astro-ph.GA] .
  * Carr  _et al._ [2021] B. Carr, S. Clesse, J. García-Bellido, and F. Kühnel, Cosmic conundra explained by thermal history and primordial black holes, Physics of the Dark Universe 31, 100755 (2021).
  * de Rujula and Glashow [1984] A. de Rujula and S. L. Glashow, Nuclearites-a novel form of cosmic radiation, Nature 312, 734 (1984).
  * Jacobs  _et al._ [2015] D. M. Jacobs, G. D. Starkman, and B. W. Lynn, Macro dark matter, Mon. Not. Roy. Ast. Soc. 450, 3418 (2015), arXiv:1410.2236 [astro-ph.CO] .
  * wik [2022] Spontaneous Human Combustion, https://en.wikipedia.org/wiki/Spontaneous_human_combustion (2022), accessed: 2021-04-01.
  * Rolli [1746] P. Rolli, An Extract, by Mr. Paul Rolli, F.R.S. of an Italian Treatise, written by the Rev. Joseph Bianchini, a Prebend in the City of Verona; upon the Death of the Countess Cornelia Zangari and Bandi, of Cesena, Philosophical transactions of the Royal Society of London 21, 447 (1746).
  * Thurston [1938] G. Thurston, Spontaneous human combustion, The British Medical Journal 1, 1340 (1938).
  * Arnold [1995] L. Arnold,  _Ablaze!: The Mysterious Fires of Spontaneous Human Combustion_ (M. Evans & Co., 1995).
  * Ford [2012] B. J. Ford, Solving the Mystery of Spontaneous Human Combustion, The Microscope , 63–72 (2012).
  * Afrin [2016] L. B. Afrin,  _Never Bet Against Occam: Mast Cell Activation Disease and the Modern Epidemics of Chronic Illness and Medical Complexity_ (Sisters Media, LLC., 2016).
  * Freese and Savage [2012] K. Freese and C. Savage, Dark matter collisions with the human body, Physics Letters B 717, 25 (2012), arXiv:1204.1339 [astro-ph.CO] .
  * Kotera  _et al._ [2010] K. Kotera, D. Allard, and A. V. Olinto, Cosmogenic neutrinos: parameter space and detectabilty from PeV to ZeV, JCAP 2010 (10), 013, arXiv:1009.1382 [astro-ph.HE] .
  * Bennett  _et al._ [2007] P. Bennett, N. Cutter, and B. Sutherland, Split-dose exposures versus dual ion exposure in human cell neoplastic transformation, Radiation and Environmental Biophysics 46, 119–123 (2007).
  * bal [2022] BTU Values of Various Solid and Liquid Wastes, http://balboa-pacific.com/WasteToEnergy/BTU_Values.pdf (2022), accessed: 2022-03-27.
  * Wheeler and Wheeler [1999] J. R. Wheeler and J. C. Wheeler, The Krone Experiment, independent film, http://thekroneexperiment.com (1999).
  * Hoyle and Lyttleton [1941] F. Hoyle and R. A. Lyttleton, On the accretion theory of stellar evolution, Mon. Not. Roy. Ast. Soc. 101, 227 (1941).
  * Jacques [2013] S. L. Jacques, Optical properties of biological tissues: a review, Phys. Med. Biol. 58, 37 (2013).
  * Bovy and Rix [2013] J. Bovy and H.-W. Rix, A Direct Dynamical Measurement of the Milky Way’s Disk Surface Density Profile, Disk Scale Length, and Dark Matter Profile at 4 kpc $<$ R $<$ 9 kpc, Astrophys. J. 779, 115 (2013), arXiv:1309.0809 [astro-ph.GA] .
  * Wang  _et al._ [2022] J. Wang, F. Hammer, and Y. Yang, Milky Way total mass derived by rotation curve and globular cluster kinematics from Gaia EDR3, Mon. Not. Roy. Ast. Soc. 510, 2242 (2022), arXiv:2111.09324 [astro-ph.GA] .
  * Binney and Vasiliev [2023] J. Binney and E. Vasiliev, Self-consistent models of our Galaxy, Mon. Not. Roy. Ast. Soc. 520, 1832 (2023), arXiv:2206.03523 [astro-ph.GA] .
  * Hessman [2015] F. V. Hessman, The difficulty of measuring the local dark matter density, Astron. Astrophys. 579, A123 (2015), arXiv:1506.00384 [astro-ph.GA] .
  * Sivertsson  _et al._ [2022] S. Sivertsson, J. I. Read, H. Silverwood, P. F. de Salas, K. Malhan, A. Widmark, C. F. P. Laporte, S. Garbari, and K. Freese, Estimating the local dark matter density in a non-axisymmetric wobbling disc, Mon. Not. Roy. Ast. Soc. 511, 1977 (2022), arXiv:2201.01822 [astro-ph.GA] .
  * [24] N. R. F. C. (NCD-RisC), A century of trends in adult human height, eLife 5, e13410 (2016).
  * Calmet and Kuipers [2021] X. Calmet and F. Kuipers, Theoretical bounds on dark matter masses, Physics Letters B 814, 136068 (2021), arXiv:2009.11575 [hep-ph] .
  * Bhattacharjee  _et al._ [1993] P. Bhattacharjee, J.-E. Alam, B. Sinha, and S. Raha, Survivability of cosmological quark nuggets in the chromoelectric flux-tube fission model of baryon evaporation, Phys. Rev. D 48, 4630 (1993).
  * Irastorza [2021] I. G. Irastorza, An introduction to axions and their detection, arXiv e-prints , arXiv:2109.07376 (2021), arXiv:2109.07376 [hep-ph] .
  * Peccei and Quinn [1977] R. D. Peccei and H. R. Quinn, $\mathrm{CP}$ conservation in the presence of pseudoparticles, Phys. Rev. Lett. 38, 1440 (1977).
  * Marsh [2016] D. J. Marsh, Axion cosmology, Physics Reports 643, 1 (2016), axion cosmology.
  * Arvanitaki  _et al._ [2010] A. Arvanitaki, S. Dimopoulos, S. Dubovsky, N. Kaloper, and J. March-Russell, String axiverse, Phys. Rev. D 81, 123530 (2010).


# Bi-directional Feature Reconstruction Network for Fine-Grained
Few-Shot Image Classification

Jijie Wu1, Dongliang Chang2, Aneeshan Sain3,
Xiaoxu Li1, Zhanyu Ma2, Jie Cao1, Jun Guo2, and Yi-Zhe Song3 indicates
corresponding author.

###### Abstract

The main challenge for fine-grained few-shot image classification is to learn
feature representations with higher inter-class and lower intra-class
variations, with a mere few labelled samples. Conventional few-shot learning
methods however cannot be naively adopted for this fine-grained setting – a
quick pilot study reveals that they in fact push for the opposite (i.e., lower
inter-class variations and higher intra-class variations). To alleviate this
problem, prior works predominately use a support set to reconstruct the query
image and then utilize metric learning to determine its category. Upon careful
inspection, we further reveal that such unidirectional reconstruction methods
only help to increase inter-class variations and are not effective in tackling
intra-class variations. In this paper, we for the first time introduce a bi-
reconstruction mechanism that can simultaneously accommodate for inter-class
and intra-class variations. In addition to using the support set to
reconstruct the query set for increasing inter-class variations, we further
use the query set to reconstruct the support set for reducing intra-class
variations. This design effectively helps the model to explore more subtle and
discriminative features which is key for the fine-grained problem in hand.
Furthermore, we also construct a self-reconstruction module to work alongside
the bi-directional module to make the features even more discriminative.
Experimental results on three widely used fine-grained image classification
datasets consistently show considerable improvements compared with other
methods. Codes are available at: https://github.com/PRIS-CV/Bi-FRN.

## Introduction

Figure 1: Visualization of the localized regions returned from Eigen-CAM
(Muhammad and Yeasin 2020) based on a ProtoNet (Snell, Swersky, and Zemel
2017) model (trained on CUB-200-211 dataset) optimized by the traditional
method, the FRN (Wertheimer, Tang, and Hariharan 2021), and the proposed
method. The higher energy region denotes the more discriminative part in the
image.

Few-shot fine-grained image classification (Huang et al. 2021; Li et al.
2021a) has emerged very recently as an important means to tackle the data
scarcity problem widely facing fine-grained analysis (Wei et al. 2021). As per
the conventional few-shot setting, it asks for effective model transfer given
just a few support samples. The differences however lies in the unique
characteristics brought by the fine-grained nature of the problem – the model
needs to focus on learning subtle and discriminative features to discriminate
not only on the category level (as per conventional few-shot), but importantly
also on intra-class level differentiating fine-grained visual differences
amongst class instances.

It is therefore not surprising that traditional few-shot methods, when applied
to the fine-grained problem, can no longer hold their promises. This is
clearly seen in Figure 1, where we apply ProtoNet (Snell, Swersky, and Zemel
2017) on the “birds” dataset (Wah et al. 2011) – the model tends to encourage
lower inter-class variations and higher intra-class variations, which is the
very opposite of our goal of fine-grained image classification.

Attempts have been made on adapting traditional few-shot learning methods to
the fine-grained scenario. Early attempts (Huang et al. 2021; Sun et al. 2020;
Zhu, Liu, and Jiang 2020) have however focused on devising complex
architectural designs which resulted in marginal gains over their vanilla
counterparts. It was not until very recently that reconstruction-based methods
have gained popularity (Wertheimer, Tang, and Hariharan 2021; Doersch, Gupta,
and Zisserman 2020) and consequently achieved state-of-the art performance.
Through specifically engineering for support-query feature alignment, they
naturally encourage fine-grained transfer. Effects of this can be observed in
Figure 1, where feature regions learned by FRN (Wertheimer, Tang, and
Hariharan 2021) tend to be more fine-grained when compared with ProtoNet
(Snell, Swersky, and Zemel 2017).

Figure 2: (a) is the traditional metric based method. (b) is the method
proposed in (Wertheimer, Tang, and Hariharan 2021). (b) + (c) is the method
proposed in this paper. (b) can help the model increase the inter-class
variations, and (c) can help the model decrease the intra-class variations.

However upon careful inspection, we importantly observe that although the
model can focus on more subtle and discriminative regions, the semantic
information they represented is nonetheless disparate between any two samples
belonging to the same class (e.g., “Class 1” FRN focused on different parts of
a bird). This seems to suggest that these reconstruction-based methods still
suffer from having large intra-class variations, which is counter-productive
for overall fine-grained learning (large inter-class variations and small
intra-class variations). In other words, such methods only help to increase
inter-class variations and do not reduce intra-class variations very well. The
reason, we conjecture, is that the existing reconstruction-based methods
(Wertheimer, Tang, and Hariharan 2021; Doersch, Gupta, and Zisserman 2020) (as
shown in Figure 2(b)) works in a unidirectional fashion, i.e., reconstruction
only happens one way from support features to query features. Consequently, it
can only constrain the relationship between reconstructed query features and
original query features to increase inter-class variations. This however fails
to constrain the relationship between samples within each class in the support
set, which is pivotal in decreasing intra-class variations.

As such, in this paper, we for the first time introduce a bi-directional
reconstruction mechanism for few-shot fine-grained classification. Instead of
using the support set to reconstruct the query set to increase inter-class
variations only (as shown in Figure 2(b)), we additionally use the query set
to reconstruct the support set to simultaneously reduce intra-class variations
(as shown in Figure 2(c)). This modification might sound overly simple at
first sight, it however importantly fulfills both desired learning outcomes
for the fine-grained setting – support to query to encourage large inter-class
variations, and query to support to encourage small intra-class variations
(see Section Ablation Study for a detailed explanation).

Our bi-directional feature reconstruction framework mainly contains four
modules: (i) a feature extraction module, (ii) a self-reconstruction module,
(iii) a mutual bi-directional reconstruction module, and (iv) a distance
metric module. (i) and (iv) are common to a reconstruction-based approach
(Wertheimer, Tang, and Hariharan 2021). In addition to the proposed bi-
directional module (iii), we also find use of a self-reconstruction module
(ii) to benefit fine-grained feature learning, and works well with the former
in terms of increasing inter-class variations and reducing intra-class
variations. Ablative studies in Section Ablation Study show both to be
effective and indispensable for few-shot fine-grained learning.

In summary, our contributions are three-fold: 1) We reveal the key problem in
current reconstruction-based few-shot fine-grained classification lies with
its inability in minimising intra-class variations. 2) We for the first time
propose a bi-directional reconstruction network that simultaneously increase
inter-class variations while reducing intra-class variations by way of a
mutual support-query and query-support reconstruction. 3) Experimental results
and ablative analyses on three fine-grained few-shot image datasets
consistently demonstrate the superiority of the proposed method and reveal
insights on why the bi-directional approach is effective.

## Related works

Metric-Based Few-shot Learning: In few-shot learning methods, metric-based
methods have received extensive attention owing to their simplicity and
efficiency (Li et al. 2021b). A plethora of earlier works have adopted fixed
metric or learnable module to learn a good metric embedding (Snell, Swersky,
and Zemel 2017; Vinyals et al. 2016; Sung et al. 2018). These methods classify
samples according to distance or similarity. Recently, GNN-based few-shot
methods (Satorras and Estrach 2018; Kim et al. 2019; Yang et al. 2020) adopted
graph neural networks (GNN) to model the similarity measurement – their
advantage being that samples have a rich relational structure. In addition to
these classic ones, some metric-based methods for fine-grained image
classification have also emerged. While DN4 (Li et al. 2019) proposed a local
descriptor-based image-to-class measure, using local features of samples to
learn feature metric, BSNet (Li et al. 2021a) used a bi-similarity network to
learn fewer but more discriminative regions using a combination of two
different metrics. NDPNet (Zhang et al. 2021) designed a feature re-
abstraction embedding network that projects the local features into the
proposed similarity metric learning network to learn discriminative projection
factors. Our method adopts a fixed euclidean distance to measure the error
between construction features and origin features.

Alignment-based Few-shot Learning: Alignment-based fine-grained few-shot
methods pay more attention to align the spatial positions of objects in images
and then classify them based on a similarity measure. Unlike PARN (Wu et al.
2019) which is a position-aware relational network that aligns similar objects
in spatial position and learns more flexible and robust metric capabilities,
Semantic Alignment Metric Learning (SAML) (Hao et al. 2019) aligns the
semantically relevant dominant objects in fine-grained images using a collect-
and-select strategy. DeepEMD (Zhang et al. 2020) uses the Earth Mover’s
Distance to generate the optimal matching flows between two local feature sets
and computes the distance between two images based on the optimal matching
flows and matching costs. In addition to these alignment-based methods above,
reconstruction-based methods are also good for aligning spatial positions of
objects from different fine-grained images. CTX (Doersch, Gupta, and Zisserman
2020) constructs a novel transformer-based neural network, which can find a
coarse spatial relationship between the query and the labelled images, and
then compute the distances between spatially-corresponding features to predict
the label of samples. Alleviating the need for any new modules or large-scale
learnable parameters FRN (Wertheimer, Tang, and Hariharan 2021) obtains the
optimal reconstruction weights in the closed form solution to reconstruct
query features from support features. Unlike these existing reconstruction
based methods, we introduce bi-directional reconstruction method, that not
only reconstructs query samples based on support samples for increasing the
inter-class variations, but also reconstructs support samples based on a query
sample for reducing the intra-class variations.

Attention Mechanism: Vaswani et al. (Vaswani et al. 2017) first proposed the
self-attention mechanism and then used it to built a new simple network
architecture, namely Transformer. Besides achieving success in natural
language processing (Vaswani et al. 2017; Kenton and Toutanova 2019; Brown et
al. 2020), this architecture also works well in computer vision tasks
(Dosovitskiy et al. 2021; Hassani et al. 2021; Chen, Fan, and Panda 2021).
Recently, some few-shot learning works have gradually begun to adopt the self-
attention mechanism. Han-Jia Ye et al. (Ye et al. 2020) proposed a Few-shot
Embedding Adaptation with Transformer (FEAT) for few-shot learning. Here
authors constructed set-to-set functions using a variety of approximators and
found Transformer to be the most efficient option that can model interactions
between images in a set and hence enable co-adaptation of each image. Unlike
FEAT where the transformer structure is only used for support samples, CTX
(Doersch, Gupta, and Zisserman 2020) uses self-attention to calculate the
spatial attention weights of query sample and support samples and learn a
query-aligned class prototype. It then calculates the distance between the
query sample and the aligned class prototype to classify the query sample. The
proposed few-shot classification method also introduced the self-attention
mechanism adopted in the transformer to learn optimal reconstruction weights
in our self-reconstruction module and mutual reconstruction module.

## Methodology

In this section, we describe our proposed method, starting with a formulation
of the proposed method followed by an overview and an in-depth description of
each component.

### The Problem Formulation

Given a dataset $D=\\{(x_{i},y_{i}),y_{i}\in Y\\}$, we divide it into three
parts, that is, $D_{base}=\\{(x_{i},y_{i}),y_{i}\in Y_{base}\\}$,
$D_{val}=\\{(x_{i},y_{i}),y_{i}\in Y_{val}\\}$ and
$D_{novel}=\\{(x_{i},y_{i}),y_{i}\in Y_{novel}\\}$, where $x_{i}$ and $y_{i}$
are the original feature vector and class label of the $i^{\text{th}}$ image,
respectively. The categories of these three parts are disjoint, i.e.,
$Y_{base}\cap Y_{val}\cap Y_{novel}=\varnothing$, and $Y_{base}\cup
Y_{val}\cup Y_{novel}=Y$. Few-shot classification aims to improve a $C$-way
$K$-shot classification performance on $D_{novel}$ by learning knowledge from
$D_{base}$ and $D_{val}$. For such a task, there are $C$ classes sampled
randomly from $D_{novel}$, and each class only has $K$ randomly sampled
labelled (support) samples $S$, and $M$ randomly sampled unlabelled (query)
samples $Q$.

Many few-shot methods adopt the meta-learning paradigm (Finn, Abbeel, and
Levine 2017; Ye and Chao 2022). Specifically, the training process of these
methods on $D_{base}$ is the same as the prediction process on $D_{novel}$.
Their purpose is to learn the meta-knowledge of learning a category concept
based on a few labelled samples. In the meta-training phase, the few-shot
classification model learns the categories of query samples based on few
support samples in each task on $D_{base}$. The optimal model is selected by
evaluating performance of the few-shot classification model on multiple tasks
on $D_{val}$. In meta-test phase, the final performance of the optimal model
are commonly evaluated by the average accuracy on all sampled tasks of
$D_{novel}$.

### Overview

Figure 3: The proposed Bi-Directional feature reconstruction network. FSRM
refers to Feature Self-reconstruction Module and FMRM refers to Feature Mutual
Reconstruction Module.

Learning subtle and discriminative features is crucial for fine-grained few-
shot image classification. Considering that the existing reconstruction-based
methods, which mostly reconstruct query features based on support features,
fail to reduce intra-class variations adequately, we propose a bi-directional
reconstruction network.

As shown in Figure 3, our model consists of four modules: the first is
embedding module $f_{\theta}$ for extracting deep convolutional image
features. This can be a traditional convolutional neural network or a residual
network. The second is a feature self-reconstruction module $g_{\phi}$, in
which the convolutional features of each image are reconstructed by themselves
based on a self-attention mechanism. This module can make the similar local
features become more similar, while dissimilar ones even more dissimilar, and
additionally benefits the following feature mutual reconstruction. The third
is a feature mutual reconstruction module, $h_{\gamma}$, which reconstructs
sample features in a bidirectional form. This module not only uses the support
sample to reconstruct the query sample but also reconstructs the support
sample from the query sample. Compared with the existing unidirectional
reconstruction which only focuses on increasing inter-class variations of
features, the bi-directional reconstruction adds another function – reducing
the intra-class variations of features. Finally, the fourth is a Euclidean
metric module, which is in charge of calculating the distance between query
sample and reconstructed query sample, as well as support samples and
reconstructed support samples. The weighted sum of the two distances are used
for classifying query samples.

### Feature Self-Reconstruction Module (FSRM)

Figure 4: Feature self-reconstruction module.

We constructed the feature self-reconstruction module (FSRM), as shown in
Figure 4. For a $C$-way $K$-shot classification task, we input $C\times(K+M)$
samples $x_{i}$ into the embedding module $f_{\theta}$ to extract features
$\hat{x_{i}}=f_{\theta}(x_{i})\in\mathbb{R}^{d\times h\times w}$, where $d$ is
the channel number, $h$ and $w$ represents the height and the width of
features, respectively. Each image feature $\hat{x_{i}}$ is the input of the
FSRM, $g_{\phi}$, and the output of the FSRM is recorded as
${\hat{z_{i}}}\in\mathbb{R}^{r\times d}$.

First, we reshape the feature $\hat{x_{i}}$ as $r$ local features in spatial
positions $[\hat{x}_{i}^{1},\hat{x}_{i}^{2},\cdots,\hat{x}_{i}^{r}]$, where
$r=h\times w$. The traditional vision transformers (Dosovitskiy et al. 2021)
build on standard transformers (Vaswani et al. 2017), taking the sequence of
image patches as input. We however compute the summation of the sequence of
local features $\hat{x}_{i}^{j}$ and the corresponding spatial position
embedding $E_{pos}\in R^{r\times d}$ as the input of the transformer, i.e.,
${z_{i}}=[\hat{x}_{i}^{1},\hat{x}_{i}^{2},\cdots,\hat{x}_{i}^{r}]+E_{pos}$,
where $E_{pos}$ adopts sinusoidal position encoding (Vaswani et al. 2017).

The output of FSRM module is computed based on the standard self-attention
operation in Transformer Encoder, and the computing operation is shown as
follows:

$Attention(Q,K,V)=Softmax(\frac{QK^{T}}{\sqrt{d_{k}}})V.$ (1)

Therefore, we can obtain $\hat{z_{i}}$ as,

$\hat{z_{i}}=Attention(z_{i}W_{\phi}^{Q},z_{i}W_{\phi}^{K},z_{i}W_{\phi}^{V}),\hat{z_{i}}\in\mathbb{R}^{r\times
d},$ (2)

where $W_{\phi}^{Q}$, $W_{\phi}^{K}$, and $W_{\phi}^{V}$ are a set of
learnable weight parameters with $d\times d$ size. Next, the $\hat{z_{i}}$ are
calculated continually by a Layer Normalization ($LN$) and a Multi-Layer
Perceptron ($MLP$).

$\hat{z_{i}}=MLP(LN(z_{i}+\hat{z_{i}})).$ (3)

### Feature Mutual Reconstruction Module (FMRM)

Figure 5: Feature mutual reconstruction module.

We propose Feature Mutual Reconstruction Module (FMRM), as shown in Figure 5,
which contains two operations: reconstructing support features in one class
given a query feature and reconstructing query feature given support features
in one class.

For $C$-way $K$-shot classification task, after going through FSRM, we can
obtain reconstructed support features of the $c^{\text{th}}$ class, i.e.
$S_{c}=[\hat{z}_{k}^{c}]\in\mathbb{R}^{kr\times d}$, where $k\in[1,\cdots,K]$
and $c\in[1,\cdots,C]$, and reconstructed query feature
$Q_{i}=\hat{z_{i}}\in\mathbb{R}^{r\times d}$, where $i\in[1,\cdots,C\times
M]$. $S_{c}$ multiplies by weights $W_{\gamma}^{Q}$, $W_{\gamma}^{K}$ and
$W_{\gamma}^{V}$,respectively, obtaining $S_{c}^{Q}$, $S_{c}^{K}$ and
$S_{c}^{V}$, where
$W_{\gamma}^{Q},W_{\gamma}^{K},W_{\gamma}^{V}\in\mathbb{R}^{d\times d}$.
Similarly, $Q_{i}$ multiplies by weights $W_{\gamma}^{Q}$, $W_{\gamma}^{K}$
and $W_{\gamma}^{V}$, respectively, obtaining $Q_{i}^{Q}$, $Q_{i}^{K}$ and
$Q_{i}^{V}$.

We calculate the reconstructed $i^{\text{th}}$ query feature $\hat{Q}_{(c,i)}$
from support features $S_{c}^{V}$ in the $c^{\text{th}}$ class and the
reconstructed support features $\hat{S}_{(i,c)}$ in the $c^{\text{th}}$ class
from $i^{\text{th}}$ query using the two equations below.

$\hat{Q}_{(c,i)}=Attention(Q_{i}^{Q},S_{c}^{K},S_{c}^{V}),\hat{Q}_{(c,i)}\in\mathbb{R}^{r\times
d},$ (4)

$\hat{S}_{(i,c)}=Attention(S_{c}^{Q},Q_{i}^{K},Q_{i}^{V}),\hat{S}_{(i,c)}\in\mathbb{R}^{kr\times
d}.$ (5)

where $Attention(\cdot,\cdot,\cdot)$ is shown in Equation 1.

To our best knowledge, the existing reconstruction methods only use
unidirectional reconstruction – using support features to reconstruct the
query feature. Building on existing methods, we add the reverse reconstruction
shown in Equation 5 – using query feature to reconstruct support features.

### Learning Objectives

After FMRM, we use the Euclidean metric to compute the distance from this
query sample $Q_{i}$ to the support samples in the $c^{\text{th}}$ class
(Figure 2(b)) as:

$d_{Q_{i}\to S_{c}}=||Q_{i}^{V}-\hat{Q}_{(c,i)}||^{2},$ (6)

and compute the distance from the support samples in the $c^{\text{th}}$ class
to the query sample $Q_{i}$ (Figure 2(c)) as:

$d_{S_{c}\to Q_{i}}=||S_{c}^{V}-\hat{S}_{(i,c)}||^{2}.$ (7)

The total distance can be obtained by weighted summation of two distances
$d_{Q_{i}\to S_{c}}$ and $d_{S_{c}\to Q_{i}}$, as,

$d_{i}^{c}=\tau(\lambda_{1}d_{Q_{i}\to S_{c}}+\lambda_{2}d_{S_{c}\to Q_{i}}),$
(8)

where $\lambda_{1}$ and $\lambda_{2}$ are learnable weight parameters, and
both of them are initialized as $0.5$. $\tau$ is a learnable temperature
factor, following (Wertheimer, Tang, and Hariharan 2021; Ye et al. 2020; Chen
et al. 2020; Gidaris and Komodakis 2018). We normalize $d_{i}^{c}$ to obtain
$\hat{d_{i}^{c}}$ as:

$\hat{d_{i}^{c}}=\frac{e^{-d_{i}^{c}}}{\sum_{c=1}^{C}e^{-d_{i}^{c}}}.$ (9)

Based on $\hat{d_{i}^{c}}$, the total loss in one $C$-way $K$-shot task can be
calculated as:

$\mathcal{L}=-\frac{1}{M\times C}\sum_{i=1}^{M\times
C}\sum_{c=1}^{C}\textbf{1}(y_{i}==c)log(\hat{d_{i}^{c}}),$ (10)

where $\textbf{1}(y_{i}==c)$ equals $1$ when $y_{i}$ and $c$ are equal,
otherwise $0$.

During the training process, for a $C$-way $K$-shot task on $D_{base}$, we
minimize $\mathcal{L}$ to update the proposed network, and repeat this process
on all randomly generated tasks.

## Experimental results and Analysis

### Datasets and Implementation Details

Datasets: To evaluate the performance of the proposed method, we selected
three benchmark fine-grained datasets, CUB-200-2011 (CUB) (Wah et al. 2011) is
a classic fine-grained image classification dataset. It contains $11,788$
images from $200$ bird species. Following (Zhang et al. 2020; Ye et al. 2020),
we crop each image to a human annotated bounding box. Stanford-Dogs (Dogs)
(Khosla et al. 2011) is a challenging fine-grained image categorization
dataset. The dataset includes $20,580$ annotated images of $120$ breeds of
dogs from around the world. Stanford-Cars (Cars) (Krause et al. 2013) is also
a commonly used benchmark dataset for fine-grained image classification. The
dataset contains $16,185$ images of $196$ car-types. For each dataset, we
divided it into $D_{train}$, $D_{val}$ and $D_{test}$. The ratio of
$D_{train}$, $D_{val}$ and $D_{test}$ is same as the literature (Zhu, Liu, and
Jiang 2020), and all images are resized to $84\times 84$.

Implementation Details: We conducted experiments on two widely used backbone
architectures: Conv-4 and ResNet-12. The architectures of Conv-4 and ResNet-12
are the same as that of (Wertheimer, Tang, and Hariharan 2021; Ye et al.
2020). We implemented all our experiments on NVIDIA 3090Ti GPUs via Pytorch
(Paszke et al. 2019). In our experiments, we train all Conv-4 and ResNet-12
models for $1,200$ epochs using SGD with Nesterov momentum of $0.9$. The
initial learning rate is set to $0.1$ and weight decay to 5e-4. Learning rate
is decreased by a scaling factor of 10 after every 400 epochs. For Conv-4
models, we train the proposed models using $30$-way $5$-shot episodes, and
test for $1$-shot and $5$-shot episodes. We use $15$ query images per class in
both settings. Furthermore, for ResNet-12 models we train our proposed model
using $15$-way $5$-shot episodes, in order to save memory. We employ standard
data augmentation, including center crop, random horizontal flip and colour
jitter for better training stability. Thereafter, we select the best-
performing model based on the validation set, and validate every $20$ epochs.
For all experiments, we report the mean accuracy of $10,000$ randomly
generated tasks on $D_{test}$ with $95\%$ confidence intervals on the standard
$5$-way, $1$-shot and $5$-shot settings.

### Comparison with State-of-the-Arts

Table 1: 5-way few-shot classification performance on the CUB, Dogs and Cars datasets. The top block uses Conv-4 backbone and the bottom block uses ResNet-12 backbone. Methods labeled by ${\dagger}$ denote our implementations. Method | CUB | Dogs | Cars
---|---|---|---
$1$-shot |  $5$-shot |  $1$-shot |  $5$-shot |  $1$-shot |  $5$-shot
ProtoNet${\dagger}$ (NeurIPS 2017) | 64.82$\pm$0.23 | 85.74$\pm$0.14 | 46.66$\pm$0.21 | 70.77$\pm$0.16 | 50.88$\pm$0.23 | 74.89$\pm$0.18
Relation (CVPR 2018) | 63.94$\pm$0.92 | 77.87$\pm$0.64 | 47.35$\pm$0.88 | 66.20$\pm$0.74 | 46.04$\pm$0.91 | 68.52$\pm$0.78
DN4 (CVPR 2019) | 57.45$\pm$0.89 | 84.41$\pm$0.58 | 39.08$\pm$0.76 | 69.81$\pm$0.69 | 34.12$\pm$0.68 | 87.47$\pm$0.47
PARN${\dagger}$ (ICCV 2019) | 74.43$\pm$0.95 | 83.11$\pm$0.67 | 55.86$\pm$0.97 | 68.06$\pm$0.72 | 66.01$\pm$0.94 | 73.74$\pm$0.70
SAML (ICCV 2019) | 65.35$\pm$0.65 | 78.47$\pm$0.41 | 45.46$\pm$0.36 | 59.65$\pm$0.51 | 61.07$\pm$0.47 | 88.73$\pm$0.49
DeepEMD (CVPR 2020) | 64.08$\pm$0.50 | 80.55$\pm$0.71 | 46.73$\pm$0.49 | 65.74$\pm$0.63 | 61.63$\pm$0.27 | 72.95$\pm$0.38
LRPABN (TMM 2021) | 63.63$\pm$0.77 | 76.06$\pm$0.58 | 45.72$\pm$0.75 | 60.94$\pm$0.66 | 60.28$\pm$0.76 | 73.29$\pm$0.58
BSNet(D&C) (TIP 2021) | 62.84$\pm$0.95 | 85.39$\pm$0.56 | 43.42$\pm$0.86 | 71.90$\pm$0.68 | 40.89$\pm$0.77 | 86.88$\pm$0.50
CTX${\dagger}$ (NeurrIPS 2020) | 72.61$\pm$0.21 | 86.23$\pm$0.14 | 57.86$\pm$0.21 | 73.59$\pm$0.16 | 66.35$\pm$0.21 | 82.25$\pm$0.14
FRN${\dagger}$ (CVPR 2021) | 74.90$\pm$0.21 | 89.39$\pm$0.12 | 60.41$\pm$0.21 | 79.26$\pm$0.15 | 67.48$\pm$0.22 | 87.97$\pm$0.11
FRN+TDM${\dagger}$ (CVPR 2022) | 72.01$\pm$0.22 | 89.05$\pm$0.12 | 51.57$\pm$0.23 | 75.25$\pm$0.16 | 65.67$\pm$0.22 | 86.44$\pm$0.12
Ours | 79.08$\pm$0.20 | 92.22$\pm$0.10 | 64.74$\pm$0.22 | 81.29$\pm$0.14 | 75.74$\pm$0.20 | 91.58$\pm$0.09
ProtoNet${\dagger}$ (NeurIPS 2017) | 81.02$\pm$0.20 | 91.93$\pm$0.11 | 73.81$\pm$0.21 | 87.39$\pm$0.12 | 85.46$\pm$0.19 | 95.08$\pm$0.08
CTX${\dagger}$ (NeurIPS 2020) | 80.39$\pm$0.20 | 91.01$\pm$0.11 | 73.22$\pm$0.22 | 85.90$\pm$0.13 | 85.03$\pm$0.19 | 92.63$\pm$0.11
DeepEMD${\dagger}$ (CVPR 2020) | 75.59$\pm$0.30 | 88.23$\pm$0.18 | 70.38$\pm$0.30 | 85.24$\pm$0.18 | 80.62$\pm$0.26 | 92.63$\pm$0.13
FRN${\dagger}$ (CVPR 2021) | 84.30$\pm$0.18 | 93.34$\pm$0.10 | 76.76$\pm$0.21 | 88.74$\pm$0.12 | 88.01$\pm$0.17 | 95.75$\pm$0.07
FRN+TDM${\dagger}$ (CVPR 2022) | 85.15$\pm$0.18 | 93.99$\pm$0.09 | 78.02$\pm$0.20 | 89.85$\pm$0.11 | 88.92$\pm$0.16 | 96.88$\pm$0.06
Ours | 85.44$\pm$0.18 | 94.73$\pm$0.09 | 76.89$\pm$0.21 | 88.27$\pm$0.12 | 90.44$\pm$0.15 | 97.49$\pm$0.05
Table 2: Ablation studies using only FSRM module or FMRM module. Backbone | Method | CUB | Dogs | Cars
---|---|---|---|---
$1$-shot |  $5$-shot |  $1$-shot |  $5$-shot |  $1$-shot |  $5$-shot
Conv-4 | Baseline (ProtoNet) | 64.82$\pm$0.23 | 85.74$\pm$0.14 | 46.66$\pm$0.21 | 70.77$\pm$0.16 | 50.88$\pm$0.23 | 74.89$\pm$0.18
(FSRM) | 75.37$\pm$0.21 | 88.61$\pm$0.12 | 65.10$\pm$0.22 | 79.94$\pm$0.15 | 71.61$\pm$0.22 | 84.70$\pm$0.14
(FMRM) | 74.92$\pm$0.21 | 89.97$\pm$0.11 | 61.28$\pm$0.21 | 80.07$\pm$0.14 | 70.22$\pm$0.21 | 88.45$\pm$0.11
(FSRM+FMRM) | 79.08$\pm$0.20 | 92.22$\pm$0.10 | 64.74$\pm$0.22 | 81.29$\pm$0.14 | 75.74$\pm$0.20 | 91.58$\pm$0.09
ResNet-12 | Baseline (ProtoNet) | 81.02$\pm$0.20 | 91.93$\pm$0.11 | 73.81$\pm$0.21 | 87.39$\pm$0.12 | 85.46$\pm$0.19 | 95.08$\pm$0.08
(FSRM) | 82.53$\pm$0.19 | 92.43$\pm$0.10 | 75.64$\pm$0.21 | 87.44$\pm$0.12 | 85.95$\pm$0.18 | 94.44$\pm$0.08
(FMRM) | 84.33$\pm$0.18 | 94.25$\pm$0.09 | 76.29$\pm$0.21 | 89.06$\pm$0.11 | 89.62$\pm$0.15 | 97.45$\pm$0.05
(FSRM+FMRM) | 85.44$\pm$0.18 | 94.73$\pm$0.09 | 76.89$\pm$0.21 | 88.27$\pm$0.12 | 90.44$\pm$0.15 | 97.49$\pm$0.05
Table 3: Ablation on reconstruction designs of FMRM. Backbone | Method | CUB | Dogs | Cars
---|---|---|---|---
$1$-shot |  $5$-shot |  $1$-shot |  $5$-shot |  $1$-shot |  $5$-shot
Conv-4 | Baseline (ProtoNet) | 64.82$\pm$0.23 | 85.74$\pm$0.14 | 46.66$\pm$0.21 | 70.77$\pm$0.16 | 50.88$\pm$0.23 | 74.89$\pm$0.18
Ours (Q$\to$S) | 79.88$\pm$0.20 | 91.76$\pm$0.11 | 65.26$\pm$0.22 | 80.81$\pm$0.14 | 75.61$\pm$0.20 | 90.49$\pm$0.10
Ours (S$\to$Q) | 76.54$\pm$0.21 | 88.03$\pm$0.14 | 64.39$\pm$0.22 | 78.36$\pm$0.15 | 72.71$\pm$0.22 | 85.11$\pm$0.14
Ours (Mutual) | 79.08$\pm$0.20 | 92.22$\pm$0.10 | 64.74$\pm$0.22 | 81.29$\pm$0.14 | 75.74$\pm$0.20 | 91.58$\pm$0.09
ResNet-12 | Baseline (ProtoNet) | 81.02$\pm$0.20 | 91.93$\pm$0.11 | 73.81$\pm$0.21 | 87.39$\pm$0.12 | 85.46$\pm$0.19 | 95.08$\pm$0.08
Ours (Q$\to$S) | 83.72$\pm$0.19 | 93.31$\pm$0.09 | 76.50$\pm$0.21 | 87.95$\pm$0.12 | 87.37$\pm$0.17 | 95.10$\pm$0.08
Ours (S$\to$Q) | 81.72$\pm$0.19 | 90.83$\pm$0.11 | 75.62$\pm$0.22 | 86.47$\pm$0.13 | 85.90$\pm$0.18 | 93.17$\pm$0.10
Ours (Mutual) | 85.44$\pm$0.18 | 94.73$\pm$0.09 | 76.89$\pm$0.21 | 88.27$\pm$0.12 | 90.44$\pm$0.15 | 97.49$\pm$0.05

To validate the efficiency of our method for fine-grained few-shot image
classification, we conducted experiments on the three fine-grained image
classification datasets discussed earlier. The results of Relation (Sung et
al. 2018), DN4 (Li et al. 2019) and BSNet (Li et al. 2021a) are from
literature BSNet (Li et al. 2021a), and the results of SAML (Hao et al. 2019)
and DeepEMD (Zhang et al. 2020) are from DLG (Cao et al. 2022), and the
results of LRPABN (Huang et al. 2021) are from MattML (Zhu, Liu, and Jiang
2020). The results of methods marked with ${\dagger}$, such as ProtoNet
(Snell, Swersky, and Zemel 2017), PARN (Wu et al. 2019), CTX (Doersch, Gupta,
and Zisserman 2020), FRN (Wertheimer, Tang, and Hariharan 2021), FRN+TDM (Lee,
Moon, and Heo 2022) and DeepEMD (Zhang et al. 2020) are obtained via the code
officially provided by the author, which is replaced by the dataset used in
this paper.

Figure 6: Recovered images of different features by our method for the CUB
dataset.

We use Conv-4 and ResNet-12 as the backbone of all compared methods and test
$5$-way $1$-shot and $5$-way $5$-shot classification performance. As seen from
Table 1, the proposed method achieves highest accuracy on all three datasets
when the Conv-4 is adopted. Apart from the result on the Dogs (Khosla et al.
2011) dataset where performance falls slightly behind the FRN+TDM method when
the ResNet-12 is adopted, our method achieves highest accuracy.

In a nutshell, compared with other newly proposed methods, our method achieves
stable and excellent performance on the three fine-grained image datasets on
both $5$-way $1$-shot and $5$-way $5$-shot classification tasks, majorly owing
to our network structure. The two modules FSRM and FMRM, conditions the model
to learn the subtle and discriminative features for fine-grained
classification.

### Ablation Study

To further justify the design choices of our method and model components
towards efficiency and accuracy, we perform a few ablation studies on three
datasets using both Conv-4 and ResNet-12 as backbone networks.

The Effectiveness of FSRM and FMRM: We compare our method’s efficacy by
removing components in a strip-down fashion. In other words, we conduct
experiments and report in Table 2, where we remove FSRM module (FMRM), FMRM
module (FSRM), and then both which becomes equivalent to ProtoNet (Snell,
Swersky, and Zemel 2017) (Baseline (ProtoNet)). Evidently, performance
improves further in our method where both FSRM and FMRM are used together
(FSRM+FMRM). Therefore, FSRM and FMRM modules are indispensable and
complementary.

The Effectiveness of Mutual Reconstruction in FMRM: For our method, we remove
reconstruction of support samples based on query samples in FMRM, i.g.,
$\lambda_{2}$ becoming $0$ in Equation 8, which is marked as Ours ($Q\to S$).
And we remove reconstruction of query samples based on support samples, i.g.,
$\lambda_{1}$ becoming $0$ in Equation 8, which is marked as Ours ($S\to Q$).
As per Table 3, both unidirectional reconstruction methods score lower than
the proposed feature mutual reconstruction method (FMRM) in most cases.
Therefore, it can be concluded that the design of the Feature Mutual
Reconstruction Module (FMRM) is reasonable and effective.

### Visualization Analysis

To demonstrate the efficiency of our proposed network better, we recovered the
original and reconstructed features. We trained an inverse ResNet as a
decoder, the input of which is the feature value in the mutual reconstruction
module, and the output of which is a $3\times 84\times 84$ recovered image. In
the training process, we use an Adam optimizer with an initial learning rate
of $0.01$, set batch size as $200$ and train for $1,000$ epochs, with an L1
loss to measure the prediction error.

As per Figure 6, the left-most block shows the support images of $5$ classes,
each of which occupies one row and contains $5$ images. The right-most column
is a query image, which we copied $5$ times simply for convenient comparison
and aesthetics. The second block from left is the recovered images of support
features $S_{c}^{V}$, whereas second column from the right is the recovered
images of the $i^{\text{th}}$ query features $Q_{i}^{V}$. The third block from
left is the recovered images of reconstructed support feature
$\hat{S}_{(i,c)}$ based on $i^{\text{th}}$ query feature, while that from the
right is the recovered images of reconstructed query features
$\hat{Q}_{(c,i)}$ based on each $c$ class support feature.

From the first and second blocks (first being left-most), it can be seen that
the decoder we trained is capable of recovering images from features. For the
third and fourth ones, it can be seen that if a query sample is used to
reconstruct the support samples which has same class as the query sample, the
reconstructed features will be similar to the original support features.
However if we use a query sample to reconstruct the support samples which has
a class different from the query sample, the reconstructed features will be
quite different from the original support features. Likewise, when we use
support samples to reconstruct the query samples, similar results are
obtained. This indicates that the proposed network can effectively alleviate
inaccurate similarity measure of the unaligned fine-grained images.

## Conclusion

In this paper, we proposed a bi-directional feature reconstruction network for
few-shot fine-grained image classification. Our major contribution is a mutual
reconstruction module that works in both directions, i.e., support to test and
test to support. Compared to the existing reconstruction-based methods, the
proposed method can achieve larger inter-class variations and lower the intra-
class variations which is crucial to fine-grained learning. Extensive
experiments show that the proposed network can perform well on three fine-
grained image datasets consistently, competing strongly and at times
surpassing contemporary state-of-the-arts.

## References

  * Brown et al. (2020) Brown, T. B.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al. 2020.  Language Models are Few-Shot Learners.  _ArXiv_.
  * Cao et al. (2022) Cao, S.; Wang, W.; Zhang, J.; Zheng, M.; and Li, Q. 2022.  A Few-shot Fine-grained Image Classification Method Leveraging Global and Local Structures.  _International Journal of Machine Learning and Cybernetics_.
  * Chen, Fan, and Panda (2021) Chen, C.-F.; Fan, Q.; and Panda, R. 2021.  CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification.  In _ICCV_.
  * Chen et al. (2020) Chen, Y.; Wang, X.; Liu, Z.; Xu, H.; and Darrell, T. 2020.  A New Meta-Baseline for Few-Shot Learning.  _ArXiv_.
  * Doersch, Gupta, and Zisserman (2020) Doersch, C.; Gupta, A.; and Zisserman, A. 2020.  CrossTransformers: Spatially-aware Few-shot Transfer.  In _NerulPS_.
  * Dosovitskiy et al. (2021) Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; et al. 2021\.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.  In _ICLR_.
  * Finn, Abbeel, and Levine (2017) Finn, C.; Abbeel, P.; and Levine, S. 2017.  Model-agnostic Meta-learning for Fast Adaptation of Deep Networks.  In _ICML_.
  * Gidaris and Komodakis (2018) Gidaris, S.; and Komodakis, N. 2018.  Dynamic Few-Shot Visual Learning Without Forgetting.  In _CVPR_.
  * Hao et al. (2019) Hao, F.; He, F.; Cheng, J.; Wang, L.; Cao, J.; and Tao, D. 2019.  Collect and Select: Semantic Alignment Metric Learning for Few-shot Learning.  In _ICCV_.
  * Hassani et al. (2021) Hassani, A.; Walton, S.; Shah, N.; Abuduweili, A.; Li, J.; and Shi, H. 2021.  Escaping the Big Data Paradigm with Compact Transformers.  _ArXiv_.
  * Huang et al. (2021) Huang, H.; Zhang, J.; Zhang, J.; Xu, J.; and Wu, Q. 2021.  Low-rank Pairwise Alignment Bilinear Network for Few-shot Fine-grained Image Classification.  _IEEE Transactions on Multimedia_.
  * Kenton and Toutanova (2019) Kenton, J. D. M.-W. C.; and Toutanova, L. K. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In _NAACL-HLT_.
  * Khosla et al. (2011) Khosla, A.; Jayadevaprakash, N.; Yao, B.; and Li, F.-F. 2011.  Novel Dataset for Fine-grained Image Categorization: Stanford Dogs.  In _CVPR workshops_.
  * Kim et al. (2019) Kim, J.; Kim, T.; Kim, S.; and Yoo, C. D. 2019.  Edge-Labeling Graph Neural Network for Few-shot Learning.  In _CVPR_.
  * Krause et al. (2013) Krause, J.; Stark, M.; Deng, J.; and Fei-Fei, L. 2013.  3D Object Representations for Fine-Grained Categorization.  In _ICCV Workshops_.
  * Lee, Moon, and Heo (2022) Lee, S.; Moon, W.; and Heo, J.-P. 2022.  Task Discrepancy Maximization for Fine-Grained Few-Shot Classification.  In _CVPR_.
  * Li et al. (2019) Li, W.; Wang, L.; Xu, J.; Huo, J.; Gao, Y.; and Luo, J. 2019.  Revisiting Local Descriptor Based Image-To-Class Measure for Few-Shot Learning.  In _CVPR_.
  * Li et al. (2021a) Li, X.; Wu, J.; Sun, Z.; Ma, Z.; Cao, J.; and Xue, J.-H. 2021a.  BSNet: Bi-Similarity Network for Few-shot Fine-grained Image Classification.  _IEEE Transactions on Image Processing_.
  * Li et al. (2021b) Li, X.; Yang, X.; Ma, Z.; and Xue, J.-H. 2021b.  Deep Metric Learning for Few-shot Image Classification: A Selective Review.  _ArXiv_.
  * Muhammad and Yeasin (2020) Muhammad, M. B.; and Yeasin, M. 2020.  Eigen-cam: Class Activation Map Using Principal Components.  In _IJCNN_.
  * Paszke et al. (2019) Paszke, A.; Gross, S.; Massa, F.; Lerer, A.; Bradbury, J.; Chanan, G.; Killeen, T.; Lin, Z.; Gimelshein, N.; Antiga, L.; et al. 2019.  PyTorch: An Imperative Style, High-Performance Deep Learning Library.  In _NeurIPS_.
  * Satorras and Estrach (2018) Satorras, V. G.; and Estrach, J. B. 2018.  Few-shot Learning with Graph Neural Networks.  In _ICLR_.
  * Snell, Swersky, and Zemel (2017) Snell, J.; Swersky, K.; and Zemel, R. 2017.  Prototypical Networks for Few-shot Learning.  In _NerulPS_.
  * Sun et al. (2020) Sun, X.; Xv, H.; Dong, J.; Zhou, H.; Chen, C.; and Li, Q. 2020.  Few-shot Learning for Domain-specific Fine-grained Image Classification.  _IEEE Transactions on Industrial Electronics_.
  * Sung et al. (2018) Sung, F.; Yang, Y.; Zhang, L.; Xiang, T.; Torr, P. H.; and Hospedales, T. M. 2018\.  Learning to Compare: Relation Network for Few-Shot Learning.  In _CVPR_.
  * Vaswani et al. (2017) Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, L. u.; and Polosukhin, I. 2017.  Attention is All you Need.  In _NerulPS_.
  * Vinyals et al. (2016) Vinyals, O.; Blundell, C.; Lillicrap, T.; Kavukcuoglu, K.; and Wierstra, D. 2016\.  Matching Networks for One Shot Learning.  In _NerulPS_.
  * Wah et al. (2011) Wah, C.; Branson, S.; Welinder, P.; Perona, P.; and Belongie, S. J. 2011.  The Caltech-UCSD Birds-200-2011 Dataset.  Technical report, California Institute of Technology.
  * Wei et al. (2021) Wei, X.-S.; Song, Y.-Z.; Mac Aodha, O.; Wu, J.; Peng, Y.; Tang, J.; Yang, J.; and Belongie, S. 2021.  Fine-Grained Image Analysis with Deep Learning: A Survey.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_.
  * Wertheimer, Tang, and Hariharan (2021) Wertheimer, D.; Tang, L.; and Hariharan, B. 2021.  Few-shot Classification With Feature Map Reconstruction Networks.  In _CVPR_.
  * Wu et al. (2019) Wu, Z.; Li, Y.; Guo, L.; and Jia, K. 2019.  PARN: Position-aware Relation Networks for Few-shot Learning.  In _ICCV_.
  * Yang et al. (2020) Yang, L.; Li, L.; Zhang, Z.; Zhou, X.; Zhou, E.; and Liu, Y. 2020.  DPGN: Distribution Propagation Graph Network for Few-shot Learning.  In _CVPR_.
  * Ye and Chao (2022) Ye, H.-J.; and Chao, W.-L. 2022.  How to Train Your MAML to Excel in Few-shot Classification.  In _ICLR_.
  * Ye et al. (2020) Ye, H.-J.; Hu, H.; Zhan, D.-C.; and Sha, F. 2020.  Few-shot Learning via Embedding Adaptation With Set-to-Set Functions.  In _CVPR_.
  * Zhang et al. (2020) Zhang, C.; Cai, Y.; Lin, G.; and Shen, C. 2020.  DeepEMD: Few-shot Image Classification With Differentiable Earth Mover’s Distance and Structured Classifiers.  In _CVPR_.
  * Zhang et al. (2021) Zhang, W.; Liu, X.; Xue, Z.; Gao, Y.; and Sun, C. 2021.  NDPNet: A Novel Non-linear Data Projection Network for Few-shot Fine-grained Image Classification.  _ArXiv_.
  * Zhu, Liu, and Jiang (2020) Zhu, Y.; Liu, C.; and Jiang, S. 2020.  Multi-attention Meta Learning for Few-shot Fine-grained Image Recognition.  In _IJCAI_.