# How likely are Snowball episodes near the inner edge of the habitable zone?

R. Wordsworth  School of Engineering and Applied Sciences, Harvard, Cambridge,
MA 02138, USA  Department of Earth and Planetary Sciences, Harvard, Cambridge,
MA 02138, USA<EMAIL_ADDRESS>
###### Abstract

Understanding when global glaciations occur on Earth-like planets is a major
challenge in climate evolution research. Most models of how greenhouse gases
like CO2 evolve with time on terrestrial planets are deterministic, but the
complex, nonlinear nature of Earth’s climate history motivates study of non-
deterministic climate models. Here a maximally simple stochastic model of CO2
evolution and climate on an Earth-like planet with an imperfect CO2 thermostat
is investigated. It is shown that as stellar luminosity is increased in this
model, the decrease in the average atmospheric CO2 concentration renders the
climate increasingly unstable, with excursions to a low-temperature state
common once the received stellar flux approaches that of present-day Earth.
Unless climate feedbacks always force the variance in CO2 concentration to
decline rapidly with received stellar flux, this means that terrestrial
planets near the inner edge of the habitable zone may enter Snowball states
quite frequently. Observations of the albedos and color variation of
terrestrial-type exoplanets should allow this prediction to be tested directly
in the future.

###### Subject headings:

planets and satellites: atmospheres—planets and satellites: terrestrial
planets

Investigating the processes that determine planetary habitability and
predicting their observable consequences is a key objective of exoplanet
climate modeling (Seager, 2013). Today, Earth is still our only confirmed
example of a habitable planet, so its climate and chemistry continues to drive
our understanding of habitability in general. One of the most influential
models of long-term CO2 evolution on Earth is the carbonate-silicate
weathering feedback (Walker et al., 1981), which is the basis for the
‘canonical’ definition of the habitable zone (Kasting et al., 1993; Kopparapu
et al., 2013). Despite the popularity of this model, the nature of Earth’s CO2
cycle through geologic time remains a highly active area of research, and a
number of processes likely cause Earth’s carbon cycle to deviate significantly
from the standard weathering feedback (Maher and Chamberlain, 2014; Macdonald
et al., 2019; Graham and Pierrehumbert, 2020).

Accurate estimates of temperature and atmospheric CO2 in Earth’s deep-time
history are difficult to obtain, but there is no evidence for secular warming
of the climate over the last 4 Gy (Feulner, 2012). Because the Sun’s
luminosity has increased with time (by about 30 to 40% over the last 4 Gy) and
CO2 has likely been a key greenhouse gas throughout Earth history, a secular
decline in atmospheric CO2 with time seems almost certain. However, this
decline has been far from monotonic: current anthropogenic emissions aside,
the variations in Earth’s surface temperature and atmospheric CO2 levels just
in the last 400 My have been substantial, for reasons that are still the
subject of intensive study (Franks et al., 2014; Montañez et al., 2016;
Lenardic et al., 2016; Macdonald et al., 2019).

Motivated by these observations and previous modeling efforts, the purpose of
this note is to construct a simple stochastic model of CO2 evolution, and to
apply it to terrestrial-type planets. The model is intentionally semi-
empirical, rather than mechanistic, because many of the processes that affect
Earth’s CO2 levels remain so uncertain. As will be shown, the transition to a
stochastic view of CO2 evolution leads to qualitatively different conclusions
compared to the deterministic picture.

Surface temperature evolution in the model is represented as

$C\frac{dT}{dt}=\frac{1}{4}F[1-A(T)]-OLR$ (1)

where $T$ is surface temperature, $C$ is the heat capacity of the ocean-
atmosphere system (here in J/m2/K), $F$ is incident stellar flux, and $OLR$ is
the outgoing longwave radiation at the top of the atmosphere, which we will
take to be a function of $T$ and the molar concentration of CO2 in the
atmosphere. Internal climate variability, which would add a stochastic term to
(1), is neglected here to keep the focus on the impact of variability in the
CO2 cycle.

The aim here is to point out general model features rather than to make
precise predictions, so we linearize the OLR around Earth’s preindustrial
surface temperature $T_{0}=288$ K and CO2 molar concentration
$f_{\ce{CO2},0}=280$ ppmv:

$OLR\approx OLR_{0}+a(T-T_{0})-b\log(f_{\ce{CO2}}/f_{\ce{CO2},0}).$ (2)

Here $a=2$ W/m2/K following Abbot (2016), and $b=5.35$ W/m2 is the radiative
forcing coefficient for CO2 (Myhre et al., 1998). Logarithmic dependence of
OLR on $f_{\ce{CO2}}$ is a reasonable approximation in the 10 to $10^{5}$ ppmv
CO2 and 280 to 290 K temperature range, although the value of $b$ begins to
increase at high CO2 concentrations.

Setting $F=F_{0}+\Delta F$ and noting that
$\frac{1}{4}F_{0}(1-A_{0})=S_{0}=OLR_{0}$, where $F_{0}=1366$ W/m2 and
$A_{0}=0.3$ are Earth’s present-day received solar flux and albedo,
respectively, we can write the time evolution of the temperature deviation
from the baseline state $x=T-T_{0}$ as

$C\frac{dx}{dt}=\Delta S-\frac{1}{4}F\Delta A-ax+b\log y$ (3)

where $y=f_{\ce{CO2}}/f_{\ce{CO2},0}$, $\Delta A=A(T)-A_{0}$ and $\Delta
S=\Delta F(1-A_{0})/4$ (Abbot, 2016). $\Delta A$ is quite hard to assess in
general due to cloud effects, but its dependence on surface ice coverage acts
to accelerate Snowball transitions as the transition temperature is
approached. As the main aim here is to assess the likelihood of a Snowball
transition as a function of atmospheric CO2 concentration, rather than to
study the Snowball state itself, it can be safely set to zero. In addition,
the climate achieves thermal balance far more rapidly than atmospheric CO2
levels change, so we set ${dx}/{dt}=0$. This allows the temperature deviation
from the present-day Earth value to be written as

$x=\frac{\Delta S+b\log y}{a}.$ (4)

Next, we incorporate CO2 evolution. The evolution of the CO2 molar
concentration $y$ with time is modeled as an Ornstein-Uhlenbeck process with
an offset term $\chi$ (Jacobs, 2010). For a given timestep $dt$, this means
that the increment in $y$ is

$dy=-\tau^{-1}(y-\chi)dt+gdW$ (5)

Here $g$ is a constant and $dW$ represents a Wiener process such that for
every timestep $dt$, $dW$ is equal to a value taken from a gaussian
distribution with variance $dt$. $\tau$ is a timescale that determines how
rapidly $y$ is drawn back to the mean value (either by carbonate-silicate
weathering feedbacks, or some other process). At each timestep, $y$ is set to
$-y$ if $y<0$, ensuring that $y$ always remains positive-valued.

Equation (5) provides an inherently non-deterministic representation of CO2
evolution, with a linear restoration term that prevents unbounded growth in
the probability distribution for $y$ with time. From any starting condition,
the system evolves towards a statistically steady state on a timescale $\tau$.
Once a steady state is reached, the probability density function for $y$ has
the form

$q(y)=Q\mathrm{e}^{-(y-\chi)^{2}/2\sigma_{y}^{2}}$ (6)

where the standard deviation $\sigma_{y}=\sqrt{\tau/2}g$ and the normalization
factor

$Q=\frac{2}{\sigma_{y}\sqrt{2\pi}}\frac{1}{1+\mathrm{erf}[\chi/\sqrt{2}\sigma_{y}]}$
(7)

ensure

$\int_{0}^{\infty}q(y)dy=1.$ (8)

Because $y$ can only take positive values, the mean of this distribution is

$\overline{y}=\int_{0}^{\infty}yq(y)dy=\chi+\sigma_{y}^{2}q_{0}.$ (9)

where $q_{0}=q(0)$. The distribution variance is

$V=\int_{0}^{\infty}(y-\overline{y})^{2}q(y)dy=\sigma_{y}^{2}(1-\overline{y}q_{0}).$
(10)

Note that when $\chi>>\sigma_{y}$, $\overline{y}=\chi$ and $V=\sigma_{y}^{2}$.

Converting (5) into a statement about the probability of a Snowball transition
under given conditions requires the parameter $\chi$ to be determined. Given
the lack of evidence for secular warming of Earth over the last 4 Gy as the
Sun’s luminosity has increased, the simplest assumption we can make is that
CO2 feedbacks set $\chi$ at a value that yields $\overline{T}=T_{0}$ (and
hence $\overline{x}=0$) on long timescales. Taking the time mean of (3) and
assuming separation of timescales between slow ($>100$ My) evolution of
$\Delta S$ and more rapid stochastic CO2 fluctuations yields

$\overline{\log y}=-\Delta S/b=S_{0}\left(1-\alpha\right)/b$ (11)

where $\alpha=F/F_{0}$ is the stellar flux received relative to Earth’s
present-day received flux. $\chi$ is then calculated for use in (5) by finding
the root of the function

$\Phi(\chi)=\overline{\log y}-\int_{0}^{\infty}\log yq(y,\chi)dy$ (12)

numerically. Situations where $\Phi(\chi)$ has no root for $\chi>0$ occur at
low $\chi/\sigma_{y}$ values, but this is of little practical significance
here, because the planet enters a Snowball state before they are reached.

Because temperature depends directly on the CO2 concentration, a second
probability density function $p(x)$ for the temperature deviation $x$ can be
written as

$p(x)=q[y(x)]\left|\frac{dy}{dx}\right|.$ (13)

We can rearrange (4) in terms of $y$, take the derivative, and substitute in
the result along with (6) to get

$p(x)=cQ\mathrm{e}^{cx-\Delta S/b-\frac{1}{2}(\mathrm{e}^{cx-\Delta
S/b}-\chi)^{2}/\sigma_{y}^{2}}.$ (14)

where $c=a/b$. Given (13), we can also write
$q(f_{\ce{CO2}})=q(y)/f_{\ce{CO2},0}$ and $p(T)=p(x)$. The function $p(x)$ is
asymmetric, with rapid decline at high $x$ values but a long tail stretching
to low $x$ values (Fig. 1). The implication is that for a gaussian CO2
concentration distribution with a given variance, very low temperatures are
reached more frequently than very high temperatures. Hence even when the mean
CO2 concentration is well above the threshold for a Snowball event, there
remains a finite probability of a transition occurring.

Figure 1 shows the results of solving (5) numerically via the Euler method
over 1 Gy of constant stellar luminosity. CO2 and temperature time series are
shown for a single run (panels a and c), and probability density functions
$q(f_{\ce{CO2}})$ and $p(T)$ are shown for an ensemble of 1024 runs (panels b
and d). The asymmetry of the temperature evolution indicated by (14) is clear
from Fig. 1d. In the time series shown, CO2 levels temporarily dip low enough
to make temperature fall below the Snowball threshold (set here at 280 K,
following Pierrehumbert et al., 2011) just after 300 My. This was verified to
cause a Snowball transition when ice-albedo effects were included in the model
(results not shown).

Figure 2 shows the output of the same model when secular evolution of stellar
flux is included. Here, 3.5 Gy of evolution is simulated and stellar flux
evolution is represented as

$\alpha(t)=\frac{1}{1+\frac{2}{5}(1-t/{4.5~{}\mbox{Gy}})}$ (15)

which is appropriate for a Sun-like star (Gough, 1981). Generalization of the
results to other star types is straightforward in principle, but it is not
pursued here, in part because Snowball transitions on low-mass stars may be
strongly affected by the stellar spectrum and the planet’s spin-orbit
configuration (Joshi and Haberle, 2012; Shields et al., 2013; Checlair et al.,
2017).

As can be seen from Fig. 2, for a fixed value of $\sigma_{y}$, temperature
fluctuations steadily increase with time until a Snowball transition occurs,
with larger $\sigma_{y}$ values yielding earlier Snowball transitions. A
Snowball event at some point in time is therefore inevitable unless
$\sigma_{y}$ declines at least as fast as $\chi$ does. This result shows that
if a planet possess an effective CO2 thermostat on long timescales ($>100$
My), as Earth appears to, but the shorter term variance in CO2 does not
decline rapidly as stellar luminosity increases, the chance of undergoing a
Snowball glaciation should _increase_ as the planet gets closer to the inner
edge of the habitable zone. This is a very different prediction from that of
deterministic models of CO2 cycles on Earth-like planets, which either predict
permanently clement conditions, or glaciations that only begin to occur
towards the outer edge of the habitable zone (e.g., Tajika, 2007; Haqq-Misra
et al., 2016).

Figure 3 shows the time of Snowball transition for 32 simulations with
different values of $\sigma_{y}$. There is some scatter because of the
stochastic nature of the simulations, but the strong dependence of transition
time on the CO2 variance is clear. As a general rule, once $\chi$ drops to
below 2 to 4 times the value of $\sigma_{y}$, a Snowball transition becomes
likely. The effect of $\tau$ on the results was also tested, and it was found
that a larger $\tau$ caused transitions to occur at a given time at slightly
higher $\sigma_{y}/\chi$ ratios in general, although the effect was not large
in the $\tau=0.1$ to $10$ My range.

Short-term climate variability due to effects like stellar fluctuations,
ocean-atmosphere feedbacks and volcanic aerosol emissions (e.g., Foster and
Rahmstorf, 2011; Macdonald and Wordsworth, 2017; Arnscheidt and Rothman, 2020)
has been neglected here. Recent work has elegantly demonstrated the impact of
short-term climate variability on Snowball transitions in a probabilistic
framework (Lucarini and Bódai, 2019). Including this variability would simply
increase the probability of a Snowball transition for a given set of
parameters in equation (5). Of course, on a planet where stellar, albedo and
dynamical variability always dominates variations in greenhouse gas
concentrations, the trend described above would be masked by these effects.
However, for Earth at least it is clear that variations in CO2 concentration
have been a fundamental driver of climate change over geologic time.

Reliable constraints on $\sigma_{y}$ on Earth on long timescales are hard to
come by, although it does appear to have declined over the last Gy or so since
the Neoproterozoic Snowball events. Three-dimensional climate modeling
suggests the Marinoan Snowball glaciation that terminated 635 My ago occurred
at a CO2 concentration of between 280 and 560 ppmv (Voigt et al., 2011; Voigt
and Abbot, 2012). This can be compared with the 3000 ppmv or more CO2 that
would likely have been present under temperate or warm climate conditions
(Pierrehumbert et al., 2011). Over the last 400 My, the characteristics of
fossil leaf stomata and other proxies constrain CO2 to around 250–2000 ppmv,
with the greatest uncertainty at the highest concentrations (Franks et al.,
2014). Finally, over just the last 800 ky until the industrial era, CO2 has
varied between about 280 and 180 ppmv (Lüthi et al., 2008), with a standard
deviation $\sigma_{y}$ of $0.11$ of the mean value (mean $f_{\ce{CO2}}=224$
ppmv, $\sigma_{f}=25$ ppmv). A few hundred My in the future, the stochastic
model predicts that CO2 fluctuations of this order ($\sigma_{f}$ of a few 10s
of ppmv) would be sufficient to start a Snowball transition (Fig. 3).

The model presented here is extremely simple and empirical. However, it is
arguably at least as justified for exoplanet habitability modeling as the many
other more sophisticated deterministic models of CO2 evolution that currently
exist. It is of course possible that variance in $f_{\ce{CO2}}$ does always
decrease rapidly enough as stellar luminosity increases to prevent Snowball
transitions. However, even if CO2 variance has decreased on Earth since the
Neoproterozoic, it is not at all obvious based on our current understanding of
the carbon cycle that this trend will continue to hold in the future, or apply
in general to Earth-like exoplanets.

The long-term CO2 source in the carbon cycle (volcanism) behaves largely
independently of climate until Venus-like atmospheric pressures are reached,
with possible modest _positive_ feedbacks due to couplings between sea level
and the rate of mid-ocean ridge volcanism (Huybers and Langmuir, 2017). The
CO2 weathering sink has a temperature dependence, but this can readily become
saturated because of local physical weathering rate limits, even when the
global climate is temperate. Earth’s history in the Phanerozoic (Maher and
Chamberlain, 2014; Macdonald et al., 2019) and Neoproterozoic (Hoffman et al.,
1998) indicates that the effects of tectonic processes and continental drift
on CO2 variability are both extremely important. Large Igneous Province (LIP)
eruptions, which have appeared intermittently throughout Earth history, are
capable of supplying huge quantities of weatherable basalt to the surface and
hence drawing down large quantities of CO2, even at the cooler equatorial
temperatures expected near a Snowball transition. Indeed, weathering
associated with LIPs likely played a major role in the first Neoproterozoic
Snowball Earth transition (Cox et al., 2016).

The final major source of complexity in Earth’s CO2 cycle is the biosphere. It
is plausible, although certainly not guaranteed, that life itself can rapidly
reduce CO2 variance as stellar luminosity increases, which would be a clear
example of a ‘Gaian’ feedback (Lovelock and Margulis, 1974). Surface
weathering by land plants is an important part of the modern carbon cycle, and
it has been suggested that weathering feedbacks involving C3-photosynthetic
plants may have buffered minimum $f_{\ce{CO2}}$ values to 100-200 ppmv on
Earth over the last 24 My (Pagani et al., 2009). However, positive
biogeochemical ocean feedbacks are a plausible explanation for the ice-age CO2
oscillations observed over the last 800,000 ky (Sigman et al., 2010), and coal
formation in the Carboniferous may have brought Earth closer to a Snowball
than at any time since the Neoproterozoic (Feulner, 2017). Even during the
last glacial maximum ca. 20,000 years ago, estimates of global mean
temperature (Tierney et al., 2020) suggest only a few Kelvins of additional
cooling could have been sufficient to push Earth into a Snowball state. As our
current era of anthropogenic global warming makes clear, when the starting
atmospheric CO2 inventory is small, even relatively small changes in exchange
rates with other reservoirs in the system have the capacity to cause sudden
and dramatic shifts in climate.

Tests of the canonical carbonate-silicate weathering hypothesis via
observations of atmospheric CO2 on exoplanets has been proposed (Bean et al.,
2017), although they require a large sample size of planetary targets and
highly capable observing systems to be successful (Lehmer et al., 2020).
Broadband or spectrally resolved albedo measurements to identify planets in a
Snowball state provide an alternative approach, at least as long as
degeneracies associated with planetary radius and the presence of thick cloud
decks can be addressed (Cowan et al., 2011; Guimond and Cowan, 2018). Such
observations would allow a powerful probe into the level of control that
climate feedback mechanisms on terrestrial-type planets provide, and the
extent to which Earth’s climate history has been unusual.

## Acknowledgments

This article has benefitted from discussions with E. Tziperman and A. Knoll
and helpful comments from an anonymous reviewer. Code to reproduce the plots
in the paper is available open-source at
_https://github.com/wordsworthgroup/stochastic_snowball_2021_. Finally, I
thank R. Pierrehumbert for bringing a manuscript preprint by R. J. Graham
(Graham, 2021) to my attention during the review process that independently
puts forward ideas related to those presented here, although with a different
focus and modeling approach.

## References

  * Abbot (2016) D. S. Abbot.  Analytical investigation of the decrease in the size of the habitable zone due to a limited CO2 outgassing rate.  _The Astrophysical Journal_ , 827(2):117, 2016\.
  * Arnscheidt and Rothman (2020) C. W. Arnscheidt and D. H. Rothman.  Routes to global glaciation.  _Proceedings of the Royal Society A_ , 476(2239):20200303, 2020.
  * Bean et al. (2017) J. L. Bean, D. S. Abbot, and E. M.-R. Kempton.  A statistical comparative planetology approach to the hunt for habitable exoplanets and life beyond the solar system.  _The Astrophysical Journal Letters_ , 841(2):L24, 2017.
  * Checlair et al. (2017) J. Checlair, K. Menou, and D. S. Abbot.  No snowball on habitable tidally locked planets.  _The Astrophysical Journal_ , 845(2):132, 2017\.
  * Cowan et al. (2011) N. B. Cowan, T. Robinson, T. A. Livengood, D. Deming, E. Agol, M. F. A’Hearn, D. Charbonneau, C. M. Lisse, V. S. Meadows, S. Seager, et al.  Rotational variability of Earth’s polar regions: implications for detecting snowball planets.  2011\.
  * Cox et al. (2016) G. M. Cox, G. P. Halverson, R. K. Stevenson, M. Vokaty, A. Poirier, M. Kunzmann, Z.-X. Li, S. W. Denyszyn, J. V. Strauss, and F. A. Macdonald.  Continental flood basalt weathering as a trigger for Neoproterozoic Snowball Earth.  _Earth and Planetary Science Letters_ , 446:89–99, 2016\.
  * Feulner (2012) G. Feulner.  The faint young Sun problem.  _Reviews of Geophysics_ , 50(2), 2012.
  * Feulner (2017) G. Feulner.  Formation of most of our coal brought Earth close to global glaciation.  _Proceedings of the National Academy of Sciences_ , 114(43):11333–11337, 2017.
  * Foster and Rahmstorf (2011) G. Foster and S. Rahmstorf.  Global temperature evolution 1979–2010.  _Environmental Research Letters_ , 6(4):044022, 2011.
  * Franks et al. (2014) P. J. Franks, D. L. Royer, D. J. Beerling, P. K. Van de Water, D. J. Cantrill, M. M. Barbour, and J. A. Berry.  New constraints on atmospheric CO2 concentration for the Phanerozoic.  _Geophysical Research Letters_ , 41(13):4685–4694, 2014.
  * Gough (1981) D. O. Gough.  Solar interior structure and luminosity variations.  In _Physics of Solar Variations_ , pages 21–34. Springer, 1981.
  * Graham (2021) R. J. Graham.  High pCO2 reduces sensitivity to CO2 perturbations on temperate, Earth-like planets throughout most of habitable zone, 2021.  _arXiv preprint arXiv:2104.01224_.
  * Graham and Pierrehumbert (2020) R. J. Graham and R. Pierrehumbert.  Thermodynamic and energetic limits on continental silicate weathering strongly impact the climate and habitability of wet, rocky worlds.  _The Astrophysical Journal_ , 896(2):115, 2020\.
  * Guimond and Cowan (2018) C. M. Guimond and N. B. Cowan.  The direct imaging search for Earth 2.0: Quantifying biases and planetary false positives.  _The Astronomical Journal_ , 155(6):230, 2018\.
  * Haqq-Misra et al. (2016) J. Haqq-Misra, R. K. Kopparapu, N. E. Batalha, C. E. Harman, and J. F. Kasting.  Limit cycles can reduce the width of the habitable zone.  _The Astrophysical Journal_ , 827(2):120, 2016\.
  * Hoffman et al. (1998) P. F. Hoffman, A. J. Kaufman, G. P. Halverson, and D. P. Schrag.  A Neoproterozoic Snowball Earth.  _Science_ , 281(5381):1342–1346, 1998.
  * Huybers and Langmuir (2017) P. Huybers and C. H. Langmuir.  Delayed CO2 emissions from mid-ocean ridge volcanism as a possible cause of late-Pleistocene glacial cycles.  _Earth and Planetary Science Letters_ , 457:238–249, 2017\.
  * Jacobs (2010) K. Jacobs.  _Stochastic processes for physicists: understanding noisy systems_.  Cambridge University Press, 2010.
  * Joshi and Haberle (2012) M. M. Joshi and R. M. Haberle.  Suppression of the water ice and snow albedo feedback on planets orbiting red dwarf stars and the subsequent widening of the habitable zone.  _Astrobiology_ , 12(1):3–8, 2012.
  * Kasting et al. (1993) J. F. Kasting, D. P. Whitmire, and R. T. Reynolds.  Habitable Zones around Main Sequence Stars.  _Icarus_ , 101:108–128, 1993.
  * Kopparapu et al. (2013) R. K. Kopparapu, R. Ramirez, J. F. Kasting, V. Eymet, T. D. Robinson, S. Mahadevan, R. C. Terrien, S. Domagal-Goldman, V. Meadows, and R. Deshpande.  Habitable Zones around Main-sequence Stars: New Estimates.  _The Astrophysical Journal_ , 765:131, 2013.
  * Lehmer et al. (2020) O. R. Lehmer, D. C. Catling, and J. Krissansen-Totton.  Carbonate-silicate cycle predictions of Earth-like planetary climates and testing the habitable zone concept.  _Nature Communications_ , 11(1):1–10, 2020.
  * Lenardic et al. (2016) A. Lenardic, A. M. Jellinek, B. Foley, C. O’Neill, and W. B. Moore.  Climate-tectonic coupling: Variations in the mean, variations about the mean, and variations in mode.  _Journal of Geophysical Research: Planets_ , 121(10):1831–1864, 2016.
  * Lovelock and Margulis (1974) J. E. Lovelock and L. Margulis.  Atmospheric homeostasis by and for the biosphere: the Gaia hypothesis.  _Tellus_ , 26(1-2):2–10, 1974.
  * Lucarini and Bódai (2019) V. Lucarini and T. Bódai.  Transitions across melancholia states in a climate model: Reconciling the deterministic and stochastic points of view.  _Physical Review Letters_ , 122(15):158701, 2019\.
  * Lüthi et al. (2008) D. Lüthi, M. Le Floch, B. Bereiter, T. Blunier, J.-M. Barnola, U. Siegenthaler, D. Raynaud, J. Jouzel, H. Fischer, K. Kawamura, and T. F. Stocker.  High-resolution carbon dioxide concentration record 650,000–800,000 years before present.  _Nature_ , 453(7193):379–382, 2008.
  * Macdonald and Wordsworth (2017) F. A. Macdonald and R. Wordsworth.  Initiation of Snowball Earth with volcanic sulfur aerosol emissions.  _Geophysical Research Letters_ , 44(4):1938–1946, 2017.
  * Macdonald et al. (2019) F. A. Macdonald, N. L. Swanson-Hysell, Y. Park, L. Lisiecki, and O. Jagoutz.  Arc-continent collisions in the tropics set Earth’s climate state.  _Science_ , 364(6436):181–184, 2019.
  * Maher and Chamberlain (2014) K. Maher and C. P. Chamberlain.  Hydrologic regulation of chemical weathering and the geologic carbon cycle.  _Science_ , 343(6178):1502–1504, 2014.
  * Montañez et al. (2016) I. P. Montañez, J. C. McElwain, C. J. Poulsen, J. D. White, W. A. DiMichele, J. P. Wilson, G. Griggs, and M. T. Hren.  Climate, pCO2 and terrestrial carbon cycle linkages during late Palaeozoic glacial–interglacial cycles.  _Nature Geoscience_ , 9(11):824–828, 2016.
  * Myhre et al. (1998) G. Myhre, E. J. Highwood, K. P. Shine, and F. Stordal.  New estimates of radiative forcing due to well mixed greenhouse gases.  _Geophysical research letters_ , 25(14):2715–2718, 1998.
  * Pagani et al. (2009) M. Pagani, K. Caldeira, R. Berner, and D. J. Beerling.  The role of terrestrial plants in limiting atmospheric CO2 decline over the past 24 million years.  _Nature_ , 460(7251):85–88, 2009.
  * Pierrehumbert et al. (2011) R. T. Pierrehumbert, D. S. Abbot, A. Voigt, and D. Koll.  Climate of the Neoproterozoic.  _Annual Review of Earth and Planetary Sciences_ , 39:417–460, 2011.
  * Seager (2013) S. Seager.  Exoplanet habitability.  _Science_ , 340(6132):577–581, 2013.
  * Shields et al. (2013) A. L. Shields, V. S. Meadows, C. M. Bitz, R. T. Pierrehumbert, M. M. Joshi, and T. D. Robinson.  The effect of host star spectral energy distribution and ice-albedo feedback on the climate of extrasolar planets.  _Astrobiology_ , 13(8):715–739, 2013.
  * Sigman et al. (2010) D. M. Sigman, M. P. Hain, and G. H. Haug.  The polar ocean and glacial cycles in atmospheric CO2 concentration.  _Nature_ , 466(7302):47–55, 2010.
  * Tajika (2007) E. Tajika.  Long-term stability of climate and global glaciations throughout the evolution of the Earth.  _Earth, Planets and Space_ , 59(4):293–299, 2007\.
  * Tierney et al. (2020) J. E. Tierney, J. Zhu, J. King, S. B. Malevich, G. J. Hakim, and C. J. Poulsen.  Glacial cooling and climate sensitivity revisited.  _Nature_ , 584(7822):569–573, 2020.
  * Voigt and Abbot (2012) A. Voigt and D. S. Abbot.  Sea-ice dynamics strongly promote Snowball Earth initiation and destabilize tropical sea-ice margins.  _Climate of the Past_ , 8(6):2079–2092, 2012\.
  * Voigt et al. (2011) A. Voigt, D. S. Abbot, R. T. Pierrehumbert, and J. Marotzke.  Initiation of a Marinoan Snowball Earth in a state-of-the-art atmosphere-ocean general circulation model.  _Climate of the Past_ , 7(1):249–263, 2011.
  * Walker et al. (1981) J. C. G. Walker, P. B. Hays, and J. F. Kasting.  A negative feedback mechanism for the long-term stabilization of the earth’s surface temperature.  _Journal of Geophysical Research_ , 86:9776–9782, 1981\.

Figure 1.— Output of the numerical stochastic model over 500 My for an
ensemble of 1024 runs, given fixed stellar luminosity ($\alpha=0.98$),
relaxation timescale $\tau=2.5$ My, and $\sigma_{y}=0.7$. a) CO2 molar
concentration vs. time, b) normalized histogram of CO2 molar concentration
values, c) temperature vs. time and d) normalized histogram of temperature
values. For a) and c), a single run where temperature dropped below the
Snowball limit is shown in black. In c) the light blue shading indicates the
Snowball transition region, while in both a) and c), the green line indicates
the mean value. Finally, in b) and d) the solid black and red dashed lines
indicate the numerical results and analytic results according to (6) and (14),
respectively. The ice-albedo feedback is not included in these simulations.
Figure 2.— Output of the numerical stochastic model for an Earth-like planet
around a G-star with evolving stellar luminosity over 3.5 Gy, starting from 3
Gy before present. a) Atmospheric CO2 molar concentration and b) global mean
temperature as a function of time. Received stellar flux is also shown on the
$x$-axis. In b), the light blue shading indicates the Snowball transition
region. Figure 3.— Time of Snowball transition as a function of CO2
concentration standard deviation $\sigma_{f}$ in ppmv
($\sigma_{f}=\sigma_{y}\times 280$ ppmv). Black dots show numerical model
results, while the red lines show fractions of the steady-state parameter
$\chi$, which is calculated from (12).


# SSR-HEF: Crowd Counting with Multi-Scale Semantic Refining and Hard Example
Focusing

Jiwei Chen∗ Kewei Wang∗ Wen Su  Zengfu Wang ∗ These authors have contributed
equally to this work.This work was supported by the Special Project of
Strategic Leading Science and Technology of Chinese Academy of Sciences (No.
XDC08020000 and No. XDC08020400) and the National Natural Science Foundation
of China (No.61472393).Jiwei Chen is with the Institute of Intelligent
Machines, Hefei Institutes of Physical Science, Chinese Academy of Sciences,
Hefei 230031, China, and also with University of Science and Technology of
China, Hefei 230026, China, and Hefei University of Technology, Hefei 230009,
China.Kewei Wang is with the School of Computer Science, Faculty of
Engineering, University of Sydney.Wen Su is with the virtual reality
laboratory in Zhejiang Sci-Tech University.Zengfu Wang is with the Institute
of Intelligent Machines, Hefei Institutes of Physical Science, Chinese Academy
of Sciences, Hefei 230031, China, and also with University of Science and
Technology of China, Hefei 230026, China. (e-mail: zfwang@ustc.edu.cn).

###### Abstract

Crowd counting based on density maps is generally regarded as a regression
task.Deep learning is used to learn the mapping between image content and
crowd density distribution. Although great success has been achieved, some
pedestrians far away from the camera are difficult to be detected. And the
number of hard examples is often larger. Existing methods with simple
Euclidean distance algorithm indiscriminately optimize the hard and easy
examples so that the densities of hard examples are usually incorrectly
predicted to be lower or even zero, which results in large counting errors. To
address this problem, we are the first to propose the Hard Example Focusing
(HEF) algorithm for the regression task of crowd counting. The HEF algorithm
makes our model rapidly focus on hard examples by attenuating the contribution
of easy examples. Then higher importance will be given to the hard examples
with wrong estimations. Moreover, the scale variations in crowd scenes are
large, and the scale annotations are labor-intensive and expensive. By
proposing a multi-Scale Semantic Refining (SSR) strategy, lower layers of our
model can break through the limitation of deep learning to capture semantic
features of different scales to sufficiently deal with the scale variation. We
perform extensive experiments on six benchmark datasets to verify the proposed
method. Results indicate the superiority of our proposed method over the
state-of-the-art methods. Moreover, our designed model is smaller and faster.

Crowd counting, density map, multi-scale semantic refining strategy, hard
example focusing.

##  1 Introduction

Crowd counting based on computer vision can estimate the number of people in
video frames or still images. It has attracted remarkable attention since its
wide practical applications including public safety, urban planning, and
traffic management [1, 2, 3]. To be specific, crowd density estimation can be
utilized to monitor crowds in real time to prevent injuries caused by
overcrowding. It can also provide advice on public resource allocation and
real-time traffic dispatch. Moreover, the methods of crowd counting can be
also extended in the cell counting, vehicle counting, and so on. However, due
to the scale variation, high density, and non-uniform distribution in crowd
scenes, this task is still challenging.

Great efforts have been devoted and various algorithms have been proposed in
crowd counting. The algorithms are generally divided into detection and
regression based methods. In detection based methods [4, 5, 6], pedestrians
are detected and the number of positive detections denotes the crowd count.
However, some pedestrians are too small to be detected in the dense crowd
scene. In regression based methods [7, 8, 2], the extracted features are
mapped to the crowd counting result. However, the location of each human is
omitted. In recent years, the convolutional neural network (CNN) based methods
occupy an undisputed dominance. In CNN based methods [9, 3, 10], the proposed
models can automatically grasp the visual features and further encode them to
a density map that records the quantity and location of pedestrians in images.
They acquire the crowd counting result by summing up all values in the density
map.

Figure 1: Examples of two issues in the crowd. 1) Hard example: pedestrians in
red rectangles are difficult to be detected. 2) Scale variation: pedestrian
head size in yellow rectangles of (a) varies with the distance from camera.
(a) represents the input image. (b) represents the ground-truth density map.
(c) represents the density map estimated by MCNN. (d) represents the density
map estimated by MCNN with our HEF algorithm. GT represents the ground-truth
count. ET represents the estimated count. Figure 2: The architecture of the
proposed SSR-HEF. The segmentation map of each scale is generated and mapped
back to network to adaptively refine and extract multi-scale features in the
SSRM. The hard example focusing task is completed in the HEFM. Its parameters
of the convolutional layers are represented as “Con(kernel size)-(number of
filters)-(dilation rate)”. Its parameters of the Max-Pooling layers are
represented as “MP(kernel size)-(strides).

In recent years, tremendous CNN models have been proposed to regress the crowd
density distribution. Then the counting result can be obtained by integrating
the final density map. Although they have achieved significant improvement in
crowd counting, we find that the hard sample problem in crowd scenes has not
been solved. As shown in Fig. 1 (a), due to long distances from the camera,
some pedestrians in red rectangles are difficult to be detected. And their
densities are imprecisely predicted by MCNN [9] to be lower or even zero in
Fig. 1 (c). We think that existing methods with simple L2 loss give the same
weight coefficient to each example and indiscriminately optimize them so that
the hard examples aren’t effectively focused on and optimized, which results
in large counting errors.

Although the focal loss [11] used in the classification task is proposed to
successfully deal with the class imbalance, this algorithm can not be utilized
in crowd counting because it is a regression task. There are essential
differences between classification and regression tasks. The former is
qualitative and the latter is quantitative. To address this issue, we are the
first to propose a novel hard example focusing algorithm for the regression
task of crowd counting. It can be used to tackle the hard example problem in
crowd scenes. By applying this algorithm, higher importance will be given to
the hard examples with wrong estimations. In principle, easy samples will be
severely penalized and their weights will be attenuated exponentially, which
in turn increases the importance of hard examples. From Fig. 1 (d), we can
observe that the MCNN with our HEF algorithm can effectively address the
problem of hard examples and improve the accuracy of crowd counting.

Additionally, as illustrated in Fig. 1 (a), pedestrian size in yellow
rectangles varies with the distance from camera. Many CNN based methods [12,
13, 14] concatenate features from different depth layers of deep network to
cope with the scale variation. However, as reported in [15], higher layers of
deep CNN encode the semantic concepts, whereas lower layers capture primitive
features. The semantic concepts can not be encoded in the lower layers of
aforementioned methods. So negative concatenation of different depth features
is not the best way to solve the problem of large scale variations.

To sufficiently overcome the scale variation problem, we propose a multi-scale
semantic refining strategy (as seen in Fig. 2). With the supervision of
ground-truth segmentation of each scale, the multi-scale segmentation maps,
where the highlighted places represent the crowd area, are generated and
mapped back to network to adaptively refine the multi-scale features. Then the
features with semantic concepts of different scales are further encoded to
automatically contribute to generating the final robust density map.
Unfortunately, the present crowd counting datasets do not provide the ground-
truth segmentation of each scale. To obtain them without the substantial
amount of extra manual labeling cost, we propose an automatic labeling method
where the ground-truth segmentation is encoded by pasting the matrix of ones
on the position of pedestrian head on a binary map, then the labels of multi-
scale segmentation maps are obtained by a simple downsampling transformation.

Our contributions are summarized as follows:

1\. We find the hard sample problem in the regression task of crowd counting.
To address this problem, the weighted hard example focusing algorithm is
proposed to make model focus on the hard samples with wrong estimations in the
quantitative regression task.

2\. To sufficiently overcome the scale variation problem, we first propose a
novel multi-scale semantic refining strategy. Multi-scale segmentation maps
with semantic prior are proposed and mapped backed to network to adaptively
refine and extract multi-scale features. This strategy also breaks through the
limitation of deep learning to make lower layers of deep CNN capture semantic
concepts.

3\. Extensive experiments are conducted on six benchmark datasets i.e.
Shanghaitech Part_A, Shanghaitech Part_B, UCF_CC_50, UCF-QNRF, WorldExpo’10,
and JHU-CROWD. Results demonstrate the superiority of our lightweight SSR-HEF
over many state-of-the-art methods. Moreover, our designed model is smaller
and faster.

In the remainder of this paper, we will discuss the related work in Section
II. The detail of proposed method will be given in Section III. Experiment
results with analysis will be shown in Section IV, followed by the conclusion
in Section V.

##  2 Related Work

Recently, various algorithms have been proposed in crowd counting, especially
for deep learning. We will introduce some of them.

###  2.1 Detection based methods

In preliminary researches, the detection framework is popular for crowd
counting. Li et al. [4] detected the head of humans by Haar wavelets. Lin et
al. [5] leveraged the part-template tree to detect and segment the crowd. Zeng
et al. [6] utilized the HOG and LBP to detect key parts of human body. They
get the crowd counting result by summing the number of positive detections.
However, they may not generalize to dense crowds because of some too small
humans.

###  2.2 Regression based methods

The regression based methods are more robust than the detection based methods
in dense crowd scenes. They extract some pivotal features, such as edge,
texture, and segmentation features, from the input image and further map them
to the crowd counting result without detection. In [7], the crowd count was
regressed by the Gaussian model. In [8], the Poisson model was reported to map
extracted features to the crowd counting result. In [2], the features of human
heads were employed to count the crowd. However, they lost the location of the
crowd.

###  2.3 CNN based methods

Deep learning has obtained commendable performance in many fields. And it
improves the accuracy of crowd counting dramatically. The CNN based methods
[9, 16, 17] regress to the density map where the location and quantity of
crowd are recorded. Due to the serious scale variation in the crowd, many
methods [9, 3, 13, 18, 19] are proposed to overcome it. In [9], the MCNN with
three-branch CNNs was designed. In [3], the IG-CNN could adapt the scale
variation by increasing its branches. Multi-scale feature maps of different
depth layers were concatenated in [13]. PACNN [18] introduced the perspective
map into density regression. MBTTB-SCFB [12] fused the bottom-top and top-
bottom features. Yang et al. [19] proposed a reverse perspective network to
evaluate and correct the distortions of input images; In [20], Wang et al.
used game characters to synthesize plenty of data to train models to mitigate
the overfitting caused by insufficient data. In [21], the semantic prior was
exploited to contribute to crowd counting. In [14], the S-DCNet trained on
closed sets could generalize well to open sets. In [22], a local counting map
was proposed to reduce the inconsistency between training targets and
evaluation metrics. In [17], a topological method was proposed to reduce the
error of crowd localization.

Figure 3: Some visualization results of ground truth.

In [23], the optical and thermal information was mined to recognize
pedestrians. In [24], a cross-view cross-scene multi-view counting method was
proposed to improve the performance and generalization of the model. In [25],
a generalized loss function based on an unbalanced optimal transport was
introduced for crowd counting and localization. In [26], a purely point-based
framework with a new metric was designed for joint crowd counting and
individual localization. In [27], a spatial uncertainty-aware semi-supervised
method with regularized surrogate task was proposed to reduce the annotation
cost. In [28], the domain-specific knowledge propagating model was constructed
to unbiasedly learn the knowledge from multi-domain data at the same time. In
[29], the temporal consistency was explored to estimate crowd flows and infer
crowd densities. In [30], a locality-aware data partition method was proposed
to tackle the severe under-estimation and over-estimation problems.

Although they have achieved great success in crowd counting, we notice that
all of them do not find the problem of hard examples in crowd counting. To the
best of our knowledge, we are the first to propose the novel hard example
focusing model to address this problem. In addition, lower layers of all the
aforementioned methods only capture primitive features and can not encode
semantic concepts in deep CNN. We propose a multi-scale semantic refining
strategy to break the limitation and sufficiently overcome the scale
variation. The multi-layer convolution feature fusion [31] and feature pyramid
fusion [32] are effective in dealing with the large scale variation. Moreover,
the features with high discrimination [33] and low correlation [34] can be
selected and provided with high weights in fusion, which inspires us to
further explore the multi-scale feature fusion with adaptive weighting to cope
with the scale variation in future works.

##  3 Proposed Method

In this paper, we propose a multi-Scale Segmentation Refining and Hard Example
Focusing model named SSR-HEF. It is a novel lightweight CNN model. An overview
of our SSR-HEF is shown in Fig. 2. It consists of two modules: Multi-Scale
Segmentation Refining Module (SSRM) and Hard Example Focusing Module (HEFM).
In the SSRM, multi-scale segmentation maps are encoded and mapped backed to
network to adaptively refine and extract multi-scale features. In the HEFM,
the hard examples are focused on automatically and the final highly refined
density map is generated. They are elaborated as follows.

###  3.1 Ground Truth Generation

The head position $p$ of each human is annotated in crowd counting dataset.
Following the method reported in [35], we utilize the Gaussian kernel centered
on each head position $p$ to encode the ground-truth density map. To ensure
that the actual crowd counting result can be obtained by integrating each
value over the entire ground-truth density map, the Gaussian kernel has been
normalized to 1. The density at any pixel position $a$ in the ground-truth
density map $D$ is formulated by Eq.(1):

$D(a)=\sum_{p\in P}N(a;\mu,\beta),$ (1)

where $P$ represents the complete set of annotated points in each image.
$N(a;\mu,\beta)$ represents the Gaussian kernel with mean $\mu$ and variance
$\beta$.

Figure 4: The configurations of designed backbone in the SSRM. Its parameters
of convolutional layers are represented as “Con(kernel size)-(number of
filters)-(dilation rate)”. Its parameters of the Max-Pooling layers are
represented as “MP(kernel size)-(strides).

Due to the large and dense crowd in datasets, it is expensive to obtain multi-
scale segmentation labels by manual labeling. For example, ShanghaiTeach
Part_A needs 241,677 annotations. Hence, an automatic labeling method is
proposed to encode multi-scale segmentation labels. Specifically, as with the
input to generate the ground-truth density map, the position $p$ of each head
is annotated and given. The ground-truth segmentation map $S^{0}$ with the
size of original image can be encoded by pasting the ones template on a binary
map, and each ones template is centered on each head position $p$. The size of
ones template is 15$\times$15 and values in the ones template are all 1. The
output is the ground-truth segmentation map $S^{0}$ and it is a binary map.
Then multi-scale segmentation labels $S^{1/2},S^{1/4},S^{1/8}$ can be obtained
through the following two steps. Firstly, $S^{0}$ is the input and the multi-
scale feature maps (MSFM) with the sizes of 1/2, 1/4, 1/8 of original image
are resized from $S^{0}$. Secondly, these multi-scale segmentation labels
$S^{1/2},S^{1/4},S^{1/8}$ are encoded from MSFM by the binarization where if
the pixel value is greater than 0, it is set to 1, otherwise, it is set to 0.
The outputs are multi-scale segmentation labels $S^{1/2},S^{1/4},S^{1/8}$ and
they are also the binary map. Some visualization results are given in Fig. 3.

###  3.2 Multi-Scale Semantic Refining Module

As shown in Fig. 2, this module is first proposed to sufficiently cope with
the scale variation problem by mapping predicted multi-scale segmentation maps
back to network to adaptively refine and extract multi-scale features, which
enhances the scale encoding ability of the network. The configurations of
designed backbone in this module are shown in Fig. 4. To improve the
robustness of our model to the scale variation, four different dilation
convolution scales are utilized to encode the input image at the beginning of
this backbone. Every convolution is followed by ReLU. The dilation
convolutions are widely used to broaden receptive fields with fewer
parameters, which can also avoid overfitting. Additionally, three max-pooling
layers (size 2$\times$2, strides 2) are exploited to extract multi-scale
features.

Table 1: The configurations of the classification task. Input (MSRF)
---
Spatial Pyramid Pooling (16$\times$16)
FC-64
FC-15

The detailed structure of multi-scale semantic refinement is illustrated in
Fig. 2. With the supervision of multi-scale segmentation labels, the multi-
scale segmentation maps where the highlighted places represent the crowd area
are generated and lower layers of the network can capture semantic concepts.
To adaptively refine multi-scale features, after the sigmoid layer $\sigma$,
the segmentation maps with semantic prior are mapped back to the output
features (denoted by $F_{m}^{h\times w\times c}$) of different depth layers by
performing an element-wise multiplication i.e., $\hat{F}_{m}^{h\times w\times
c}=F_{m}^{h\times w\times c}\bigodot\sigma(\hat{S}_{m})$. Then they are
further fused to obtain multi-scale semantic refining features (MSRF). The
Dice loss [36] is utilized to optimize these segmentation tasks.
Mathematically,

$L_{segs}=\sum_{m=1}^{M}[1-\frac{2\hat{S}_{m}S_{m}}{\hat{S}_{m}^{2}+S_{m}^{2}}],$
(2)

where $M$ represents the number of scales and it is set to 3 in this paper.
$S_{m}$ represents the ground-truth segmentation map of each scale which
respectively corresponds to $S^{1/2},S^{1/4},S^{1/8}$ encoded in the Ground
Truth Generation. $\hat{S}_{m}$ represents the predicted segmentation map of
each scale. Although [37, 38] also use segmentation tasks, there are essential
differences. [37] treats it as an auxiliary multi-task learning, and the
segmentation task is used to distinguish the foreground and background in
[38]. While we propose the multi-scale semantic refining strategy to
explicitly solve multi-scale problems and also break through the limitation of
deep learning to make lower layers of deep CNN capture some semantic concepts.

To incorporate the global-level cue into the MSRF to contribute in encoding
the final highly refined density map, the auxiliary classification task [35]
divides these crowd counts into different classes to give an overall rough
estimation of the total population in the image. In contrast to [35] where the
classification task with an additional CNN branch adds a lot of redundant
parameters, our classification task directly shares the backbone with other
tasks, which reduces the redundancy of our model and makes our model easier to
train. Additionally, our SSR-HEF solves the scale variation and hard example
problem for crowd counting. The configurations of the classification task are
given in Table 1. The cross-entropy loss is utilized to optimize this task.

$L_{cla}=-\frac{1}{N}\sum^{N}_{n=1}\sum^{K}_{k=1}\left[\left(y^{k}_{n}\right)log(\hat{y}_{n})\right],$
(3)

where $N$ represents the number of test samples. $\hat{y}_{n}$ represents the
output of classification network. $y^{k}_{n}$ represents the actual class of
crowd counts. $K$ represents the classes of crowd counts and it is set to 15
in this paper. Since the populations in different images of different datasets
vary greatly, we propose the adaptive classification criteria to encode the
labels of classification tasks for different datasets. Specifically, we obtain
the maximum count of people $C$ in the dataset, and we use $Thr=C/K$ as the
adaptive threshold to divide these crowd counts of the dataset into different
classes. A simple example is given to illustrate the encoding process of the
actual class of crowd counts. For instance, if the number of people in a
dataset ranges from 0 to 30, and we quantify these crowd counts of this
dataset into three classes. $Thr=10$. And the images of populations between 0
and 10 fall into the first class. The images of populations between 10 and 20
fall into the second class. And the images of populations between 20 and 30
fall into the third class. $Thrs$ of different datasets are given in Table 2.

Figure 5: The architecture of the feature enhancement layer. Its parameters of convolutional layers are represented as “Con(kernel size)-(number of filters)- (dilation rate)”. Table 2: Statistics of these datasets: Ave Resolution denotes the average resolution of images. Color denotes the color of images. Num Images denote the number of images in the dataset; Max Count denotes the maximum crowd counts in the dataset; Classification Thr denotes the threshold for classifying crowd counts of the dataset; Ave count denotes the average crowd counts in the dataset; Annotations denote the total number of labeled people in the dataset. Dataset | Ave Resolution | Color | Num Images | Max Count | Classification Thr | Ave Count | Annotations
---|---|---|---|---|---|---|---
ShanghaiTech Part_A [9] | 589 $\times$ 868 | RGB,Grey | 482 | 3139 | 209.3 | 501.4 | 241,677
ShanghaiTech Part_B [9] | 768 $\times$ 1024 | RGB | 716 | 578 | 38.5 | 123.6 | 88,488
UCF_CC_50 [2] | 2101 $\times$ 2888 | Grey | 50 | 4543 | 302.9 | 1279.5 | 63,974
UCF-QNRF [39] | 2013 $\times$ 2902 | RGB,Grey | 1535 | 12865 | 857.7 | 815.4 | 1,251,642
WorldExpo’10 [1] | 576 $\times$ 720 | RGB | 3980 | 253 | 16.9 | 50.2 | 199,923
JHU-CROWD [40] | 1450 $\times$ 900 | RGB,Grey | 4250 | 7286 | 485.7 | 262.3 | 1,114,785

###  3.3 Hard Example Focusing Module

Strong crowd characteristics are crucial for counting. The Feature Enhancement
Layer [41] is used to strengthen the crowd feature representations for the
Hard Example Focusing task where the counting result is obtained.

Feature Enhancement Layer: This layer is included in the HEFM. Details of this
layer are illustrated in Fig. 5. The outputs of SSRM are fed into this layer.
The convolution layer followed by ReLU is employed to encode them. Then the
max pool and average pool along the channel axis are utilized to respectively
extract the highlighted representations from the feature maps. After
concatenation, these highlighted representations followed by convolution and
sigmoid layer are further encoded and mapped back to network to enhance the
feature responses of the crowd.

Figure 6: Some examples of these datasets.

Hard Example Focusing: We first find the hard example problem in the
regression task of crowd counting. Existing methods with L2 loss give the same
weight coefficient to each example and indiscriminately optimize them. The
hard examples have not received effective focus and optimization. So the
density of them is usually predicted incorrectly, which often shows that the
estimated density is lower than the density of ground truth. Inspired by the
focal loss [11] used in the classification task, we propose a novel HEF
algorithm which can be directly used in the regression task of crowd counting
to address this problem. By using this HEF algorithm, the hard examples with
wrong estimations can be rapidly focused on. In principle, the weight
modulating factor reduces the contribution of L2 loss of easy examples, which
in turn increases the importance of hard examples. More formally, the proposed
algorithm with Euclidean distance is utilized as a loss function to optimize
the final density map.

$\begin{aligned}
L_{hef}=\frac{1}{W*H}\sum^{W}_{w=1}\sum^{H}_{h=1}\quad\quad\quad\quad\quad\quad\\\
(1-\sigma(D\left(X_{(w,h)};\theta\right)))^{\gamma}\left\|D\left(X_{(w,h)};\theta\right)-D_{(w,h)}\right\|^{2}_{2},\end{aligned}$
(4)

where $W$ and $H$ represent the width and height of estimated density map,
respectively. $\sigma$ represents the sigmoid function.
$D\left(X_{(w,h)};\theta\right)$ represents the density of position $(w,h)$ in
the final estimated density map. $\theta$ represents the parameters of
network. And all values in the final estimated density map are summed to
acquire the crowd counting result. $D_{(w,h)}$ represents the density in the
ground-truth density map. As recommended by [11], exponent $\gamma$ is set to
2. The weight modulating factor
$(1-\sigma(D\left(X_{(w,h)};\theta\right)))^{\gamma}$ is introduced to the
Euclidean distance to attenuate the contribution from easy examples. For
instance, when the hard sample isn’t detected correctly and its density was
incorrectly predicted to be lower or even 0, the weight modulating factor
tends to 1, and the loss is almost unaffected. For the easy sample, its
density is easy to estimate, and the weight modulating factor tends to 0, then
its loss is down-weighted. In this way, its weights will be greatly reduced
during backpropagation, which in turn increases the importance of hard
samples.

In [40], the L2-loss reweighted with uncertainty is proposed for crowd
counting. This work is different from ours. The L2-loss reweighted with
uncertainty is designed to increase the effectiveness of the residual
learning, where only highly confident residuals are allowed to get propagated
to the output. Our HEF algorithm with the focal factor is proposed to focus on
hard examples of the crowd, where hard examples can be focused on by
attenuating the contribution of easy examples. In [42], the cross-modal
learning to rank with the self-paced learning theory is proposed to learn the
samples from simple to complex, which points us to a new direction for hard
examples mining in crowd counting.

##  4 Experiments

With the development of crowd counting, more than 20 datasets have been
proposed. And extensive experiments are separately conducted on six offline
benchmark datasets including Shanghaitech Part_A [9], Shanghaitech Part_B [9],
UCF_CC_50 [2], UCF-QNRF [39], WorldExpo’10 [1], and JHU-CROWD [40]. Statistics
of them are elaborated in Table 2. Some representative examples of these
datasets are shown in Fig. 6. We also conducted experiments on the online
dataset (NWPU-Crowd [43]). Real-time results can be observed at
https://www.crowdbenchmark.com/nwpucrowd.html. To alleviate overfitting caused
by limited training data, patches with the sizes of 1/16, 1/4, 1 of original
image are randomly cropped from every image. The numbers of patches in each
size are set to 9, 4, and 1, respectively. Meanwhile, random noise and
flipping are also employed on them with a probability of 0.5 during the
training process. The learning rate is set as 3e-5. And the Adam optimization
with a batch size of 1 is used to optimize the overall loss function in an
end-to-end manner. The overall loss is given as follows:

$L_{overall}=L_{hef}+\lambda_{1}L_{segs}+\lambda_{2}L_{cla},$ (5)

during the tuning process, we find that $L_{segs}$, $L_{cla}$ are large and
overwhelm the losses $L_{hef}$ in the hard example focusing task. To balance
the magnitude of different tasks, $\lambda_{1}$ is set to 1e-2 and
$\lambda_{2}$ is set to 1e-3.

###  4.1 Evaluation Metric

Following existing standard metrics in [9, 20, 19], the Mean Absolute Error
(MAE) and Mean Squared Error (MSE) are leveraged to evaluate the performance
of different methods, which are defined as follows:

$MAE=\frac{1}{E}\sum_{e=1}^{E}\left|z_{e}-\widetilde{z_{e}}\right|,MSE=\sqrt{\frac{1}{E}\sum^{E}_{e=1}\left(z_{e}-\widetilde{z_{e}}\right)^{2}}.$
(6)

Where $E$ represents the total number of test images, and $z_{e}$ represents
the actual crowd count by summing each value in the ground-truth density map.
$\widetilde{z_{e}}$ represents the estimated crowd count by summing each value
in the final density map. MAE represents the accuracy of crowd counting and
MSE represents the robustness of proposed methods.

Table 3: Estimation errors on the WorldExpo’10 dataset. Method | Scene 1 | Scene 2 | Scene 3 | Scene 4 | Scene 5 | Average
---|---|---|---|---|---|---
LBP+RR | 13.6 | 58.9 | 37.1 | 21.8 | 23.4 | 31.9
Wang et al. [20] | 4.3 | 59.1 | 43.7 | 17.0 | 7.6 | 26.3
IG-CNN [3] | 2.6 | 16.1 | 10.2 | 20.2 | 7.6 | 11.3
Wan et al. [21] | 2.9 | 15.0 | 7.2 | 14.7 | 2.6 | 8.5
Yang et al. [19] | 2.4 | 10.2 | 9.7 | 11.5 | 3.8 | 8.2
TEDnet [13] | 2.3 | 10.1 | 11.3 | 13.8 | 2.6 | 8.0
PACNN [18] | 2.3 | 12.5 | 9.1 | 11.2 | 3.8 | 7.8
SSR-HEF(OURS) | 1.9 | 9.8 | 8.4 | 10.0 | 2.4 | 6.5
Table 4: Estimation errors on the JHU-CROWD dataset. Category | Distractors | Low | Medium | High | Weather | Overall
---|---|---|---|---|---|---
Methods | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE
MCNN [9] | 103.8 | 238.5 | 37.7 | 92.5 | 84.1 | 185.2 | 499.6 | 795.5 | 128.2 | 288.3 | 109.3 | 291.0
CMTL [35] | 138.5 | 263.8 | 47.0 | 106.0 | 82.4 | 198.3 | 407.8 | 660.2 | 117.8 | 260.1 | 102.5 | 262.6
Switching CNN [44] | 100.5 | 235.5 | 32.1 | 80.5 | 76.1 | 173.1 | 395.1 | 640.1 | 105.1 | 245.2 | 99.1 | 255.1
SA-Net [45] | 71.9 | 167.7 | 30.0 | 76.6 | 65.4 | 121.5 | 516.3 | 762.7 | 99.4 | 234.9 | 98.0 | 260.3
CSR-Net [46] | 44.3 | 102.4 | 15.8 | 39.9 | 48.4 | 77.7 | 463.5 | 746.1 | 96.5 | 284.6 | 78.4 | 242.7
CG-DRCN [40] | 43.4 | 97.8 | 15.7 | 38.9 | 44.0 | 73.2 | 346.2 | 569.5 | 80.9 | 227.3 | 66.1 | 195.5
SSR-HEF | 32.2 | 71.8 | 11.6 | 20.7 | 34.8 | 51.9 | 268.3 | 378.6 | 69.5 | 120.1 | 51.3 | 101.6
Table 5: Estimation errors on the ShanghaiTeach. Methods | Part_A | Part_B
---|---|---
MAE | MSE | MAE | MSE
LBP + RR | 303.2 | 371.0 | 59.1 | 81.7
Wang et al. [20] | 123.4 | 193.4 | 19.9 | 28.3
IG-CNN [3] | 72.5 | 118.2 | 13.6 | 21.1
TEDnet [13] | 64.2 | 109.1 | 8.2 | 12.8
PACNN [18] | 62.4 | 102.0 | 8.9 | 13.5
AMRNet [22] | 61.6 | 98.4 | 7.0 | 11.0
Wan et al. [21] | 61.3 | 101.4 | 8.7 | 13.6
Yang et al. [19] | 61.2 | 96.9 | 8.1 | 11.6
TopoCount [17] | 61.2 | 104.6 | 7.8 | 13.7
MBTTB-SCFB [12] | 60.2 | 94.1 | 8.0 | 15.5
S-DCNet [14] | 58.3 | 95.0 | 6.7 | 10.7
SSR-HEF(OURS) | 55.0 | 88.3 | 6.1 | 9.5
Table 6: Estimation errors on the UCF_CC_50 and UCF-QNRF datasets. Methods | UCF_CC_50 | UCF-QNRF
---|---|---
MAE | MSE | MAE | MSE
Idrees et al. [2] | 419.5 | 541.6 | 315.0 | 508.0
Wang et al. [20] | 373.4 | 528.8 | - | -
Wan et al. [21] | 355.0 | 560.2 | - | -
IG-CNN [3] | 291.4 | 349.4 | - | -
TEDnet [13] | 249.4 | 354.5 | 113.0 | 188.0
PACNN [18] | 241.7 | 320.7 | - | -
MBTTB-SCFB [12] | 233.1 | 300.9 | 97.5 | 165.2
S-DCNet [14] | 204.2 | 301.3 | 104.4 | 176.1
TopoCount [17] | 184.1 | 258.3 | 89.0 | 159.0
AMRNet [22] | 184.0 | 265.8 | 86.6 | 152.2
SSR-HEF(OURS) | 173.3 | 260.4 | 70.2 | 128.6
Table 7: Estimation errors of different configurations. Methods | Part_A | Part_B | UCF_CC_50 | UCF-QNRF | WorldExpo’10
---|---|---|---|---|---
MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE
Without SSR strategy | 70.5 | 108.6 | 12.6 | 20.5 | 205.3 | 271.3 | 87.6 | 157.2 | 8.9 | 12.3
Without Feature Enhancement Layer | 66.8 | 108.0 | 9.4 | 15.2 | 191.5 | 288.3 | 79.8 | 140.5 | 7.6 | 9.5
Without HEF algorithm | 72.4 | 118.4 | 15.1 | 25.1 | 226.2 | 301.5 | 95.8 | 165.9 | 10.1 | 15.6
SSR-HEF | 55.0 | 88.3 | 6.1 | 9.5 | 173.3 | 260.4 | 70.2 | 128.6 | 6.5 | 8.1
Figure 7: Some visualization results of the segmentation maps in different
depth layers. Image column represents the test images. SM1, SM2, and SM3
Columns represent the segmentation maps of different depths.

###  4.2 Evaluations and Comparisons

ShanghaiTeach [9]. This dataset consists of two parts: Part_A and Part_B.
Part_A contains a training set with 300 images and a test set with 182 images.
All of them are collected from the Internet in a random way. Part_B contains a
training set with 400 images and a test set with 316 images. They are taken
from the Shanghai streets. Since the density of Part_A is higher than that of
Part_B, it is difficult to predict the crowd count accurately in this part.

Comparisons with other state-of-the-art methods are tabulated in Table 5. LBP
+ RR are traditional feature extractors. Others are the CNN based methods. We
can observe that the CNN based models outperform the traditional feature
extractors (LBP+RR) significantly. In CNN based models, we can also find that
the scale variation is extremely challenging and it limits the performance of
proposed models. So many methods [3, 13, 18, 12, 19] are proposed to overcome
it. Moreover, although the model (113M) [14] is 56 times larger than ours
(2M), it is straightforward to see that the proposed SSR-HEF obtains the best
result on both Part_A and Part_B.

WorldExpo’10 [1]. The WorldExpo’10 contains 3,980 frames sampled from 1,132
video sequences that are captured from 108 different scenes. The training set
with 3,380 frames is produced from 103 different scenes, while the test set
with 600 frames is generated from other 5 different scenes (Scene 1-Scene 5).
The region of interest (ROI) has been given for each scene in this dataset.
Following the experimental setting in [1], the provided perspective maps are
also used to encode the ground-truth density map.

Following the recommendation of the author of this database [1], MAE is
leveraged to evaluate the results of other state-of-the-art methods in Table
3. It represents the accuracy of the method. In the same way, a clear review
can be found that the CNN based methods outperform the traditional methods
(LBP+RR). We can also observe that the proposed SSR-HEF leads the best
performance in terms of Scene 1, 2, 4, 5, and average of five scenes, with
$17\%$, $22\%$, $11\%$, $37\%$, and $17\%$ improvement over the second best
method [18]. Our proposed SSR-HEF is more robust, unlike others that will fail
in a certain scene(e.g., Scene 4 in [3]). So our Average of 5 scenes is the
best. The competitive result in Scene 3 is obtained. By reviewing the test
set, we find that the ROI mask removes many hard examples in Scene 3. Hence,
our strength has not been fully demonstrated.

JHU-CROWD [40]. The JHU-CROWD is an unconstrained dataset with 4,250 images
that are collected under various weather-based degradations such as rain,
snow, haze, etc. This dataset contains 1.11 million annotations at both image-
level and head-level. Comparisons with other state-of-the-art methods are
tabulated in Table 4. We can observe that the proposed SSR-HEF outperforms
other methods in all scenes, which shows the robustness of our proposed
method.

UCF_CC_50 [2]. This dataset consists of 50 images randomly crawled from the
Internet. Because of the challenging issues such as limited image, high
density, and low resolution, the 5-fold cross validation is leveraged to
evaluate the proposed methods.

We compare our method with other state-of-the-art methods in Table 6.
Traditional classical algorithms, such as SIFT, MRF, are used to extract crowd
features in [2]. Other methods on the ShanghaiTeach are still compared on this
dataset. It can be also found that the deep learning outperforms the
traditional algorithm [2]. In addition, although [22] uses the larger model
(VGG-16) and is pretrained on the ImageNet dataset, we can observe that the
proposed SSR-HEF achieves the best results.

UCF-QNRF [39]. The UCF-QNRF is a large dataset with 1201 training images and
334 testing images. These high-resolution images contain a high density of
crowd. There are 1,251,642 annotated heads in this dataset, with an average of
815 per image. The comparison results with recent state-of-the-art methods are
given in Table 6. In the same way, the performance of hand-crafted features
[2] is lost to deep learning methods. In deep learning methods, our SSR-HEF
leads the best performance on both MAE and MSE, with the MAE/MSE 16.4/23.6
lower than that of second best performance [22].

###  4.3 Ablation Study

To further verify the effectiveness of our proposed method, we perform
extensive ablation studies with analysis on these datasets, especially on the
most popular ShanghaiTeach Part_A [9].

Benefits of Multi-Scale Semantic Refining Module: To sufficiently overcome the
scale variation in crowd scene, we propose a multi-scale semantic refining
strategy. Multi-scale segmentation maps with semantic prior are first encoded,
then they are mapped back to network to adaptively refine and extract multi-
scale features. To demonstrate the effectiveness of this strategy, a set of
comparative experiments is conducted. In comparative experiments, this
strategy is removed from our model. $L_{segs}$ and corresponding parameters
are also removed. Comparison results of our model with and without this
strategy are elaborated in Table 7. It can be observed that there is an
evident utility in improving the performance by using this multi-scale
semantic refining strategy. Since the scale variations of ShanghaiTeach
Part_A, UCF_CC_50, and UCF-QNRF are large, their improvements are more
obvious.

Table 8: Estimation errors about the classification task. categories | 0 | 5 | 10 | 15 | 20
---|---|---|---|---|---
MAE | 68.2 | 63.4 | 61.8 | 55.0 | 65.0
MSE | 116.3 | 105.1 | 104.7 | 88.3 | 112.7
Figure 8: The crowd count comparisons with and without the HEF. X-axis: the ID
of images, Y-axis: the crowd counts of images.

To further demonstrate the mechanism of this strategy, some predicted multi-
scale segmentation maps in different depth layers on the ShanghaiTeach Part_A
are resized and visualized in Fig. 7. It can be observed that lower layers
have learned some semantic concepts to roughly locate the crowd area. And
layers of different depths play their respective advantages in feature
extraction. As shown in red rectangles, small-scale crowds are captured in the
SM1. Medium-scale crowds are highlighted in the SM2. Large-scale crowds are
deeply depicted in the SM3, which is vital to refine the multi-scale features.
Finding from this study and previous studies suggests that scale variations
restrict the performance of crowd counting and effective extraction of multi-
scale features is critical to overcoming the scale variation problem.

The auxiliary classification task digs into the level cue of density for
performance improvement. To verify this effectiveness, comparative experiments
with different classes are conducted on the ShanghaiTeach Part_A. The results
are given in Table 8. ’0’ column represents the SSR-HEF without classification
task. We can see that all results with the classification task are better than
that without this task, which suggests that the classification task is helpful
to improve the counting accuracy of the model. And the SSR-HEF with 15 classes
leads the best result. We think that 15 classes are sufficient because of the
limitations of model capacity.

Benefits of Feature Enhancement Layer: To recognize the merits of this layer,
comparative experiments with and without this layer are conducted. As shown in
Table 7, a clear review can be found that the SSR-HEF with this layer achieves
higher accuracy on these datasets. We argue that crowd areas are enhanced in
this layer, which is powerful to improve the accuracy of crowd counting.

Figure 9: Some visualization results of the HEF algorithm. First Row: test
image. Second Row: ground-truth density map. Third Row: the final estimated
density map without hard example focusing algorithm. Fourth Row: the final
estimated density map with hard example focusing algorithm. GT denotes the
ground-truth crowd count. ET denotes the estimated crowd count.

Benefits of Hard Example Focusing Algorithm: To set up a comprehensive
evaluation of this algorithm, we perform a set of comparative experiments. In
the comparative experiment, the HEF algorithm is removed and only the
Euclidean distance is utilized to optimize the final density map. Comparison
results are respectively elaborated in Table 7. A clear review can be found
that the performances are further enhanced by the proposed HEF algorithm on
these datasets.

To further visualize the effectiveness of this HEF algorithm on the
ShanghaiTeach Part_A, the final density maps respectively optimized by the
Euclidean distance (third row) and our proposed HEF algorithm (fourth row) are
illustrated in Fig. 9. As expected, the density (heat) of hard examples in red
rectangles of the fourth row with the HEF is larger than that of the third row
without the HEF. The density of hard examples in the red rectangle of the
third row is mispredicted to be lower or even zero. The density maps of the
fourth row and its ground-truth density map (second row) are more similar. And
the crowd counting results of the fourth row are almost the same with the
ground-truth counts, which further confirms that our proposed SSR-HEF focuses
well on the hard example.

The predicted counting results of each image on the ShanghaiTeach Part_A are
shown in Fig. 8. We can observe that blue circles (estimated crowd counts with
the HEF) are closer to red triangles (ground-truth crowd counts) than green
pentagrams (estimated crowd counts without the HEF) in most images, which
represents that the HEF is effective in reducing crowd counting errors.
However, the performance of our algorithm is still limited by the number of
labeled training samples. Inspired by [47] where abundant unlabeled samples
can be used for self-supervised learning, we will explore self-supervised
methods to break the limitations to further improve the performance of our
algorithm in future works.

Table 9: Comparisons of the real time. Size represents the model size. Time represents the average time. Method | Size | Time (s)
---|---|---
AMRNet [22] | 226M | 0.445
Wang et al. [20] | 155M | 0.417
Wan et al. [21] | 66M | 0.054
S-DCNet [14] | 113M | 0.049
TopoCount [17] | 103M | 0.009
SSR-HEF | 2M | 0.003

###  4.4 Evaluations of The Real Time:

To demonstrate the real-time effectiveness of our designed model, the
comparative experiments of recent open source methods and ours are conducted
with NVIDIA GTX 1080Ti and Intel Core i7. The test images of the ShanghaiTech
Part_A are resized to 512$\times$1024\. Comparison results are tabulated in
Table 9. It can be observed that although three segmentation tasks introduce a
few additional convolution operations, the model size of our designed model is
still small (2M). And our model is about 3 times faster than [17] which is the
second fast.

##  5 Conclusion

In this paper, we find the hard example problem in the regression task of
crowd counting. The hard example focusing algorithm is proposed to solve it.
The lightweight SSR-HEF with this algorithm can rapidly focus on hard examples
by automatically attenuating the weights of easy examples. To sufficiently
address the scale variation problem, we first propose a novel multi-scale
semantic refining strategy. Multi-scale segmentation maps with semantic prior
are generated and mapped back to network to adaptively refine and extract
multi-scale features. This strategy also makes lower layers of deep CNN
capture the semantic concepts. Extensive experiments are conducted on six
benchmark datasets to evaluate our method. Results imply that our method
outperforms many state-of-the-art methods. Moreover, our designed model is
smaller and faster. In the future, we will explore the self-supervised method
to break the limitations of labeling cost and improve the performance of crowd
counting.

## References

  * [1] C. Zhang, H. Li, X. Wang, and X. Yang, “Cross-scene crowd counting via deep convolutional neural networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 833–841.
  * [2] H. Idrees, I. Saleemi, C. Seibert, and M. Shah, “Multi-source multi-scale counting in extremely dense crowd images,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2013, pp. 2547–2554.
  * [3] D. Babu Sam, N. N. Sajjan, R. Venkatesh Babu, and M. Srinivasan, “Divide and grow: Capturing huge diversity in crowd images with incrementally growing cnn,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 3618–3626.
  * [4] M. Li, Z. Zhang, K. Huang, and T. Tan, “Estimating the number of people in crowded scenes by mid based foreground segmentation and head-shoulder detection,” in _2008 19th International Conference on Pattern Recognition_. IEEE, 2008, pp. 1–4.
  * [5] Z. Lin and L. S. Davis, “Shape-based human detection and segmentation via hierarchical part-template matching,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 32, no. 4, pp. 604–618, 2010.
  * [6] C. Zeng and H. Ma, “Robust head-shoulder detection by pca-based multilevel hog-lbp detector for people counting,” in _2010 20th International Conference on Pattern Recognition_. IEEE, 2010, pp. 2069–2072.
  * [7] A. B. Chan, Z.-S. J. Liang, and N. Vasconcelos, “Privacy preserving crowd monitoring: Counting people without people models or tracking,” in _2008 IEEE Conference on Computer Vision and Pattern Recognition_. IEEE, 2008, pp. 1–7.
  * [8] A. B. Chan and N. Vasconcelos, “Bayesian poisson regression for crowd counting,” in _Computer Vision, 2009 IEEE 12th International Conference on_. IEEE, 2009, pp. 545–551.
  * [9] Y. Zhang, D. Zhou, S. Chen, S. Gao, and Y. Ma, “Single-image crowd counting via multi-column convolutional neural network,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 589–597.
  * [10] J. Chen, S. Wen, and Z. Wang, “Crowd counting with crowd attention convolutional neural network,” _Neurocomputing_ , vol. 382, pp. 210–220, 2020.
  * [11] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, “Focal loss for dense object detection,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 2980–2988.
  * [12] V. A. Sindagi and V. M. Patel, “Multi-level bottom-top and top-bottom feature fusion for crowd counting,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , October 2019.
  * [13] X. Jiang, Z. Xiao, B. Zhang, X. Zhen, X. Cao, D. Doermann, and L. Shao, “Crowd counting and density estimation by trellis encoder-decoder networks,” in _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2019.
  * [14] H. Xiong, H. Lu, C. Liu, L. Liu, Z. Cao, and C. Shen, “From open set to closed set: Counting objects by spatial divide-and-conquer,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , October 2019\.
  * [15] M. D. Zeiler and R. Fergus, “Visualizing and understanding convolutional networks,” in _Computer Vision – ECCV 2014_ , D. Fleet, T. Pajdla, B. Schiele, and T. Tuytelaars, Eds. Cham: Springer International Publishing, 2014, pp. 818–833.
  * [16] J. Chen and Z. Wang, “Crowd counting with segmentation attention convolutional neural network,” _IET Image Processing_ , vol. 15, no. 6, pp. 1221–1231, 2021.
  * [17] S. Abousamra, M. Hoai, D. Samaras, and C. Chen, “Localization in the crowd with topological constraints,” in _Thirty-Fifth AAAI Conference on Artificial Intelligence_ , 2021.
  * [18] M. Shi, Z. Yang, C. Xu, and Q. Chen, “Revisiting perspective information for efficient crowd counting,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2019.
  * [19] Y. Yang, G. Li, Z. Wu, L. Su, Q. Huang, and N. Sebe, “Reverse perspective network for perspective-aware object counting,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2020.
  * [20] Q. Wang, J. Gao, W. Lin, and Y. Yuan, “Learning from synthetic data for crowd counting in the wild,” in _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2019.
  * [21] J. Wan, W. Luo, B. Wu, A. B. Chan, and W. Liu, “Residual regression with semantic prior for crowd counting,” in _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2019.
  * [22] X. Liu, J. Yang, W. Ding, T. Wang, Z. Wang, and J. Xiong, “Adaptive mixture regression network with local counting map for crowd counting,” in _Computer Vision – ECCV 2020_ , A. Vedaldi, H. Bischof, T. Brox, and J.-M. Frahm, Eds., 2020.
  * [23] L. Liu, J. Chen, H. Wu, G. Li, C. Li, and L. Lin, “Cross-modal collaborative representation learning and a large-scale rgbt benchmark for crowd counting,” in _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021.
  * [24] Q. Zhang, W. Lin, and A. B. Chan, “Cross-view cross-scene multi-view crowd counting,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2021, pp. 557–567.
  * [25] J. Wan, Z. Liu, and A. B. Chan, “A generalized loss function for crowd counting and localization,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2021, pp. 1974–1983.
  * [26] Q. Song, C. Wang, Z. Jiang, Y. Wang, Y. Tai, C. Wang, J. Li, F. Huang, and Y. Wu, “Rethinking counting and localization in crowds: A purely point-based framework,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , October 2021, pp. 3365–3374.
  * [27] Y. Meng, H. Zhang, Y. Zhao, X. Yang, X. Qian, X. Huang, and Y. Zheng, “Spatial uncertainty-aware semi-supervised crowd counting,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , October 2021, pp. 15 549–15 559.
  * [28] B. Chen, Z. Yan, K. Li, P. Li, B. Wang, W. Zuo, and L. Zhang, “Variational attention: Propagating domain-specific knowledge for multi-domain learning in crowd counting,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , October 2021, pp. 16 065–16 075.
  * [29] W. Liu, M. Salzmann, and P. Fua, “Counting people by estimating people flows,” _CoRR_ , vol. abs/2012.00452, 2020.
  * [30] J. T. Zhou, L. Zhang, D. Jiawei, X. Peng, Z. Fang, Z. Xiao, and H. Zhu, “Locality-aware crowd counting,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , pp. 1–1, 2021.
  * [31] J. Chu, Z. Guo, and L. Leng, “Object detection based on multi-layer convolution feature fusion and online hard example mining,” _IEEE Access_ , pp. 19 959–19 967, 2018.
  * [32] Y. Zhang, J. Chu, L. Leng, and J. Miao, “Mask-refined r-cnn: A network for refining object details in instance segmentation,” _Sensors_ , vol. 20, no. 4, p. 1010, 2020.
  * [33] L. Leng, M. Li, C. Kim, and X. Bi, “Dual-source discrimination power analysis for multi-instance contactless palmprint recognition,” _Multimedia Tools & Applications_, vol. 76, no. 1, pp. 333–354, 2017.
  * [34] L. Leng and J. Zhang, “Palmhash code vs. palmphasor code,” _Neurocomputing_ , vol. 108, no. may 2, pp. 1–12, 2013.
  * [35] V. A. Sindagi and V. M. Patel, “Cnn-based cascaded multi-task learning of high-level prior and density estimation for crowd counting,” in _2017 14th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS)_. IEEE, 2017, pp. 1–6.
  * [36] F. Milletari, N. Navab, and S.-A. Ahmadi, “V-net: Fully convolutional neural networks for volumetric medical image segmentation,” in _2016 Fourth International Conference on 3D Vision_. IEEE, 2016, pp. 565–571.
  * [37] M. Zhao, J. Zhang, C. Zhang, and W. Zhang, “Leveraging heterogeneous auxiliary tasks to assist crowd counting,” in _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * [38] J. Gao, Q. Wang, and X. Li, “Pcc net: Perspective crowd counting via spatial convolutional network,” _IEEE Transactions on Circuits and Systems for Video Technology_ , 2019.
  * [39] H. Idrees, M. Tayyab, K. Athrey, D. Zhang, S. Al-Maadeed, N. Rajpoot, and M. Shah, “Composition loss for counting, density map estimation and localization in dense crowds,” in _The European Conference on Computer Vision (ECCV)_ , September 2018.
  * [40] V. A. Sindagi, R. Yasarla, and V. M. Patel, “Pushing the frontiers of unconstrained crowd counting: New dataset and benchmark method,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , October 2019.
  * [41] S. Woo, J. Park, J.-Y. Lee, and I. So Kweon, “Cbam: Convolutional block attention module,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 3–19.
  * [42] M. Luo, X. Chang, Z. Li, L. Nie, A. G. Hauptmann, and Q. Zheng, “Simple to complex cross-modal learning to rank,” _Computer Vision & Image Understanding_, vol. 163, no. oct., pp. 67–77, 2017.
  * [43] Q. Wang, J. Gao, W. Lin, and X. Li, “Nwpu-crowd: A large-scale benchmark for crowd counting and localization,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2020.
  * [44] D. B. Sam, S. Surya, and R. V. Babu, “Switching convolutional neural network for crowd counting,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , vol. 1, no. 3, 2017, p. 6.
  * [45] X. Cao, Z. Wang, Y. Zhao, and F. Su, “Scale aggregation network for accurate and efficient crowd counting,” in _European Conference on Computer Vision_. Springer, 2018, pp. 757–773.
  * [46] Y. Li, X. Zhang, and D. Chen, “Csrnet: Dilated convolutional neural networks for understanding the highly congested scenes,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 1091–1100.
  * [47] D. Yuan, X. Chang, P.-Y. Huang, Q. Liu, and Z. He, “Self-supervised deep correlation tracking,” _IEEE Transactions on Image Processing_ , vol. 30, pp. 976–985, 2021.

[]Jiwei Chen is currently pursuing the Ph.D. degree in the Hefei Institutes of
Physical Science, University of Science and Technology of China. His research
interests include computer vision, few-shot learning, semi-supervised learning
and crowd counting.

[]Kewei Wang is currently pursuing her M.Phil. degree at the School of
Computer Science, Faculty of Engineering, the University of Sydney. Her
research interests include computer vision, machine learning and crowd
counting.

[]Wen Su received Ph.D. degree in control science and engineering from
University of Science and Technology of China in 2018 and B.E. degree in
engineering from Automation Department, University of Science and Technology
of China in 2013, respectively. Now, she works in virtual reality laboratory
in Zhejiang Sci-Tech University. At present her research interests are image
segmentation and depth scene understanding based on deep learning.

[]Zengfu Wang received the B.S. degree in electronic engineering from the
University of Science and Technology of China in 1982 and the Ph.D. degree in
control engineering from Osaka University, Japan, in 1992. He is currently a
Professor with the Institute of Intelligent Machines, Chinese Academy of
Sciences, and the Department of Automation, University of Science and
Technology of China. He has published more than 300 journal articles and
conference papers. His research interests include computer vision,
human–computer interaction and intelligent robots. He received the Best Paper
Award at the ACM International Conference on Multimedia 2009 and the IET Image
Processing Premium Award 2017.