[a]Jérémy Auffinger

# BlackHawk: A tool for computing Black Hole evaporation

Alexandre Arbey

###### Abstract

We present the public code BlackHawk, a powerful tool to compute the
evaporation of any distribution of Schwarzschild and Kerr Black Holes and the
emission spectra of Hawking radiation.

##  1 Introduction

The nature of dark matter (DM) is a long standing problem in modern cosmology.
For now, particle physics candidates explaining the missing mass have not been
observed, neither in direct (see e.g. [1] for an overview) nor indirect
searches (see e.g. [2] for a recent review). On the other hand, the concept of
primordial Black Holes (PBHs) which originates from the 60s [3] has raised
revived interest in the last decade. The recent detection of gravitational
waves from the mergers of binary BHs by the LIGO/Virgo Collaborations [4] and
the observation of the shadow of the supermassive BH at the center of M87 [5]
have shown that BHs are ubiquitous in our universe. The observed BHs are
either of stellar mass or supermassive. There is a never-ending discussion
about their origin: most, or at least some of them may be primordial. Many
astrophysical constraints limit the fraction of DM that PBHs could represent,
see for example [6] and references therein for a long review, and [7] for a
more concise one. Recent work focuses on the nature of LIGO/Virgo BHs and the
discrimination between the primordial and stellar origin using mass
distribution or spin measurements [8, 9, 10, 11, 12, 13, 14]. Most of those
studies consider PBHs with very low spin formed during the radiation
domination era, but it has been shown that PBHs could have a very high spin if
formed during a transient matter dominated era, evading Thorne’s limit [15].

The most striking characteristics of BHs is that they emit radiation close to
the blackbody one and thus slowly evaporate away by losing mass, angular
momentum and – putative – electric charge. This phenomenon first described by
Hawking [16] led to the idea that BHs could be detected via the radiation they
emit when evaporating [17]. Hawking radiation is sizeable only for small BHs
of less than $\sim 10^{16}-10^{19}\,$g, leading to radiation in the
MeV$\,-\,$keV band. This points directly towards PBHs since stellar mass BHs
are heavier than $\sim 2\,M_{\odot}$ due to the Tolman-Oppenheimer-Volkoff
limit.

PBHs may have formed in the early universe through the collapse of strong
density inhomogeneities $\Delta\rho/\rho\gtrsim 1$ resulting from quantum
fluctuations during inflation, from the collapses of topological defects
(cosmic strings, domain walls), or from bubble collisions during a first-order
phase transition (see [6] for an extended discussion). The mass of PBHs formed
in the early universe is linked to the content of the Hubble radius at the
time of formation

$M^{\rm init}_{\rm PBH}(t)\sim 10^{15}\left(\dfrac{t}{10^{-23}\,{\rm
s}}\right)\,{\rm g}\,,$ (1)

which corresponds approximately to the Hubble mass at time $t$. Evaporation
equations show that (non-rotating) BHs have a lifetime linked to their mass by

$\tau(M^{\rm init}_{\rm PBH})\sim 10^{64}\left(\dfrac{M^{\rm init}_{\rm
PBH}}{M_{\odot}}\right)^{3}\,{\rm yr}\,.$ (2)

Thus PBHs with a mass $M_{\rm PBH}^{\rm init}\lesssim 10^{15}\,$g have
completely evaporated by now. These PBHs may have however left imprints of
their evaporation in cosmological observations. Anyway, they cannot represent
a sizeable fraction of DM. Heavier PBHs, on the other hand, are almost eternal
compared to the age of the universe due to the power $3$ dependence in Eq.
(2). If they are of asteroid mass $M_{\rm PBH}^{\rm init}\lesssim 10^{19}\,$g,
then their Hawking radiation may be detected by astrophysical observation and
they could represent a fraction, if not all, of DM.

The above discussion shows that there is a need for constraints on PBH DM
linked to observations in order to infirm or confirm this scenario.
Alternatively, mixed models assuming the existence of more than one DM
component (e.g. WIMPs and PBHs [18]) can also be constrained using Hawking
radiation. For now, there still exists an open window $10^{17}\,{\rm
g}\lesssim M_{\rm PBH}\lesssim 10^{21}\,{\rm g}$ in the asteroid and sublunar
mass range for PBHs to represent all of DM [6]. PBHs are also quite
unconstrained in the lowest possible mass range $10^{-1}\,{\rm g}\lesssim
M_{\rm PBH}\lesssim 10^{9}\,{\rm g}$ as there is no known cosmological
observable from those times, although gravitational astronomy may soon provide
new possibilities [19, 20]. This is the general context in which the public
tool BlackHawk [21] has been created.

In section 2, we present rapidly the fundamental equations of Hawking
radiation, in section 3 we discuss in some detail the capabilities of
BlackHawk, in section 4 we present some examples of the work that can be done
with BlackHawk and we conclude in section 5.

##  2 Hawking radiation

Hawking radiation [16] is a semi-classical phenomenon by which pairs of
particles and anti-particles are created in the vicinity of the horizon of a
BH and can get causally separated by this horizon. There results a net flux of
energy at spatial infinity. The creation of pairs at the horizon is a purely
gravitational phenomenon whose thermodynamics is directed by the effective
temperature of the BH111In the following, we use natural units for which
$G=\hbar=c=k_{\rm B}=1$.

$T\equiv\dfrac{\kappa}{2\pi}\,,$ (3)

where $\kappa$ is the surface gravity. Thus, the rate of Hawking radiation is
simply given by the blackbody law

$\dfrac{{\rm d}^{2}N_{i}}{{\rm d}t{\rm d}E}=\sum_{\rm
dof}\dfrac{\Gamma_{i}(E,M,x_{j})/2\pi}{e^{E^{\prime}(E,x_{j})/T}-(-1)^{2s_{i}}}\,,$
(4)

where

  * •

${\rm d}^{2}N_{i}/{\rm d}t{\rm d}E$ is the number of particles of type $i$
emitted per unit time $t$ and energy $E$;

  * •

$\Gamma_{i}$ is the greybody factor describing the probability that a particle
generated at the horizon escapes at infinity;

  * •

$E^{\prime}$ is the total energy of the particles corrected for e.g. horizon
rotation or electric charge;

  * •

$s_{i}$ is the particle spin;

  * •

$x_{j}$ are all the internal degrees of freedom of the BH, namely angular
momentum $a^{*}$ and charge $Q$ following the no-hair theorem;

  * •

the sum runs over the internal degrees of freedom of the particles (helicity,
color, _etc_).

In this formula, the most important component is the greybody factors which
encode the discrepancy between BH radiation and pure blackbody radiation. It
is evaluated by evolving a plane wave radially from the BH horizon to the
spatial infinity and comparing the amplitudes of the wave at these two extreme
positions

$\Gamma_{i}(E,M,x_{j})\equiv\left|\dfrac{Z_{\infty}}{Z_{\rm
hor}}\right|^{2}\,.$ (5)

Hawking radiation yield thus depends on the spin of the emitted particle, and
on the other internal degrees of freedom such as its charge and angular
momentum. Couplings between the particle and the BH electric charges, or
between the particle and the BH angular momentum [22, 23, 24] give rise to
interesting phenomena such as superradiance [25]. Although numerical and
analytical estimates existed for the greybody factors for various BH solutions
([26] is often cited), precise numerical computations of these factors lacked
in the literature.

We can deduce the time evolution of BHs by integrating over the energy and
momentum loss

$\displaystyle f(M,a^{*})\equiv-M^{2}\dfrac{{\rm d}M}{{\rm
d}t}=M^{2}\int_{0}^{+\infty}\sum_{i}\sum_{\rm
dof.}\dfrac{E}{2\pi}\dfrac{\Gamma_{i}(E,M,a^{*})}{e^{E^{\prime}/T}\pm 1}{\rm
d}E\,,$ (6) $\displaystyle g(M,a^{*})\equiv-\dfrac{M}{a^{*}}\dfrac{{\rm
d}J}{{\rm d}t}=\dfrac{M}{a^{*}}\int_{0}^{+\infty}\sum_{i}\sum_{\rm
dof.}\dfrac{m}{2\pi}\dfrac{\Gamma_{i}(E,M,a^{*})}{e^{E^{\prime}/T}\pm 1}{\rm
d}E\,,$ (7)

where $J\equiv a^{*}M^{2}$ is the angular momentum of the BH and $m$ that of
the particle. These Page factors allow to write differential equations for the
mass and spin of the BH

$\displaystyle\dfrac{{\rm d}M}{{\rm d}t}=-\dfrac{f(M,a^{*})}{M^{2}}\,,$ (8)
$\displaystyle\dfrac{{\rm d}a^{*}}{{\rm
d}t}=\dfrac{a^{*}(2f(M,a^{*})-g(M,a^{*}))}{M^{3}}\,,$ (9)

where the mass evolution equation indeed shows the behaviour of Eq. (2) if
$f(M,a^{*})$ is taken as a constant.

##  3 BlackHawk content

BlackHawk has been designed as a public tool to compute the Hawking radiation
emitted by a variety of BH solutions, with any initial mass and spin
distribution, and to provide spectra of final particles that can be compared
to astrophysical observations222Many more details can be found in the
BlackHawk manual [21]..

Various PBHs distributions taken from the literature have been implemented in
BlackHawk, like the monochromatic one with a single BH mass and spin, and the
log-normal one which is the result of PBH production by a narrow peak in the
power spectrum of primordial density fluctuations. A lot of efforts have been
put in the computation of the greybody factors for Schwarzschild (non-
rotating, uncharged) and Kerr (rotating, uncharged) BHs, which are _per se_
interesting results. These greybody factors are used to compute the Page
factors Eq. (6) and Eq. (7), which are then used to compute the mass and spin
evolution of the distribution of BHs.

Hawking radiation is described in terms of Standard Model (SM) particle
emission. Of course, any particle beyond the SM would be emitted by the BH
horizon, such as any particle DM candidate, or the putative graviton (included
in BlackHawk). On the other hand, astrophysical observations are based on the
detection of stable particles, or at least on the imprints left in some
cosmological era by stable particles at that timescale. Thus, particle physics
codes (PYTHIA [27] and HERWIG [28]) are used to hadronize and decay the SM
particles to stable ones333Stable in the sense of cosmologically stable, or
stable at the time of BBN for example.. In BlackHawk, they come in the form of
tabulated transfer functions from primary SM particles to secondary stable
particles.

##  4 Primordial Black Hole detection and constraints

BlackHawk is a versatile tool. The tabulated values of greybody factors, Page
factors, the tabulated transfer functions between primary and secondary
particles can be used outside the main code for e.g. BH evolution study [15]
(see Fig. 1) or DM indirect searches (on the same model as for the tables of
M. Cirelli _et al._ [29]). Of course, the main aim of BlackHawk is to provide
spectra for constraining the abundance of PBHs in the universe, or for
predicting signals emitted by known or putative astrophysical objects. Here we
give a short overview of the work that has been done so far.

Figure 1: Evolution of a BH mass and spin following Eqs. (6) and (7). Taken
from [15].

First, BlackHawk can predict the instantaneous spectrum of a population of
PBHs in our Galaxy. If PBHs represent a substantial fraction or all of DM,
they are expected to be distributed as DM in the galactic halo, following for
example the Navarro-Frenk-White (NFW) profile given by [30]

$\rho_{\rm PBH}(r)=\rho_{\rm DM}(r)\sim\dfrac{1}{(r/r_{s})(1+r/rs)^{2}}\,,$
(10)

where $r_{s}$ is a characteristic radius. Then, if PBHs are monochromatic,
i.e. with the same unique mass, it is easy to deduce the flux of particles of
type $i$ on an Earth or space based detector resulting from the integration
over the galactic bulk

$\dfrac{{\rm d}F_{i}}{{\rm d}E}=\int\dfrac{{\rm d}\Omega}{4\pi}\int{\rm
d}l\dfrac{\rho_{\rm PBH}(l)}{M_{\rm PBH}}\dfrac{{\rm d}^{2}N_{i}}{{\rm d}t{\rm
d}E}\,,$ (11)

where $l$ is the line of sight coordinate. Of course, this flux depends on the
precise distribution of DM, and deviations from NFW induce modified
constraints/signals. For example, BlackHawk can predict the density of high-
energy particles emitted by PBHs at the galactic center. These particles
evolve in a strong magnetic field and thus produce synchrotron radiation in
the radio band, that can be used to constrain the amount of PBHs [31]. PBHs of
$M_{\rm PBH}\lesssim 10^{17}\,$g emit electrons and positrons. The
annihilation of the $e^{\pm}$ in the galactic center may contribute to the
$511\,$keV line measured in the direction of the galactic center [32]. On the
same level, the INTEGRAL measure of the soft gamma-ray spectrum of the Galaxy
sets constraints on the abundance of light PBHs [33, 34] and neutrino
scintillators such as JUNO are sensitive to (anti)neutrinos [35, 36].

Second, BlackHawk can be used to constrain the cosmological amount of PBHs. If
PBHs represent a substantial fraction of DM, then they can explain both the
cosmological DM and galactic DM. For that, BlackHawk can compute the emission
of particles throughout the whole BH lifetime, and integrating this emission,
taking the cosmological redshift into account, gives access to the isotropic
background flux of e.g. (soft) gamma-rays [37, 38]

$I\equiv E\dfrac{{\rm d}F_{\gamma}}{{\rm d}E}=\dfrac{1}{4\pi}n_{\rm
PBH}(t_{0})E\int_{t_{\rm min}}^{t_{\rm max}}(1+z)\dfrac{{\rm
d}^{2}N_{\gamma}}{{\rm d}t{\rm d}E}((1+z)E){\rm d}t\,,$ (12)

where $n_{\rm PBH}(t_{0})$ is the number density of PBHs today, $z(t)$ is the
redshift and ${\rm d}^{2}N_{\gamma}/{\rm d}t{\rm d}E$ is the secondary
spectrum of photons. The results of this computation, taking the PBH spin into
account, are presented in Fig. 2. This prediction has been adapted to PBHs in
scenarios with large extra spatial dimensions [39], opening an interesting
window on the ability of BlackHawk to probe alternative gravitational
theories.

Figure 2: Constraints on the DM fraction composed of PBHs from the isotropic
extragalactic gamma-ray background. Taken from [38].

Third, BlackHawk can be used together with Big-Bang nucleosynthesis (BBN)
codes to provide constraints on very light PBHs that evaporate when the light
elements are forged at the beginning of the universe. Indeed, energetic
particles emitted by $M_{\rm PBH}\sim 10^{9}-10^{13}\,$g PBHs can change the
neutron to proton ratio at the onset of BBN, induce nuclear reactions during
BBN or photo-dissociate the final light elements after BBN. Hence, the
modified Deuterium and/or Helium abundances can provide strong constraints on
the abundance of light PBHs at the BBN epoch [40].

Fourth, BlackHawk can predict the signals emitted by astrophysical structures
in terms of Hawking radiation in various energy bands. These constraints are
independent from and complementary to the cosmological and galactic ones. For
example, PBHs evaporating in nearby dwarf galaxies such as Leo T would deposit
energy – heat and ionization – in such structures. Constraints can be set on
the abundance of PBHs in these strongly DM-dominated objects [41, 34, 42].
More exotic is the prediction of the radio ($\gamma$) and gravitational waves
($g$) power emission of a putative Planet 9, would it be a PBH of $M_{\rm
P9}\sim 10\,M_{\oplus}$ [43]

$\frac{{\rm d}^{2}\mathcal{P}_{\gamma/g}}{{\rm d}\nu{\rm d}S}=\frac{1}{4\pi
r_{\rm S}^{2}}E\frac{{\rm d}^{2}N_{\gamma/g}}{{\rm d}t{\rm d}\nu}\,,$ (13)

where $r_{\rm S}$ is the Planet 9 Schwarzschild radius. Prediction of the
signal in terms of radio photons is given in Fig. 3.

Figure 3: Comparison between the Planet 9 radio signal and the CMB background.
Adapted from [43].

Interestingly, most of these studies used the fact that BlackHawk allows the
user to compute the spectra for realistic spinning BHs and extended mass
functions. This is the main improvement of this public code over previous
constraints, beside the more precise computation of the Hawking radiation
rates.

##  5 Conclusion

We have summarized here the context of development of the public code
BlackHawk, which provides a powerful and versatile tool to compute the Hawking
radiation spectra of any distribution of BHs. We have described the basics of
Hawking radiation and the main capabilities of BlackHawk and given an overview
of the studies this code has already helped to pursue. BlackHawk is not a
fixed entity and receives regularly updates to correct bugs and improve its
smoothness. Moreover, work is invested towards a major update that will
enhance its capabilities: new BH solutions as well as a better low-energy
treatment and additional particles will be added in the near future.

## Acknowledgments

We would like to thank the organizers of the TOOLS 2020 conference for giving
us the opportunity to present our work.

## References

  * [1] C. Tao, _Dark Matter searches: an overview_ , _Journal of Instrumentation_ 15 (2020) C06054.
  * [2] C. Pérez de los Heros, _Status, Challenges and Directions in Indirect Dark Matter Searches_ , _arXiv e-prints_ (2020) arXiv:2008.11561 [2008.11561].
  * [3] Y.B. Zel’dovich and I.D. Novikov, _The Hypothesis of Cores Retarded during Expansion and the Hot Cosmological Model_ , _Soviet Astronomy_ 10 (1967) 602.
  * [4] LIGO Scientific Collaboration and Virgo Collaboration, _Binary Black Hole Population Properties Inferred from the First and Second Observing Runs of Advanced LIGO and Advanced Virgo_ , _Astrophys. J. Lett._ 882 (2019) L24 [1811.12940].
  * [5] Event Horizon Telescope Collaboration, K. Akiyama et al., _First M87 Event Horizon Telescope Results. I. The Shadow of the Supermassive Black Hole_ , _Astrophys. J. Lett._ 875 (2019) L1 [1906.11238].
  * [6] B. Carr, K. Kohri, Y. Sendouda and J. Yokoyama, _Constraints on Primordial Black Holes_ , _arXiv e-prints_ (2020) arXiv:2002.12778 [2002.12778].
  * [7] A.M. Green and B.J. Kavanagh, _Primordial Black Holes as a dark matter candidate_ , _arXiv e-prints_ (2020) arXiv:2007.10722 [2007.10722].
  * [8] E. Khalouei, H. Ghodsi, S. Rahvar and J. Abedi, _Possibility of Primordial Black holes as the source of gravitational wave events in the advanced LIGO detector_ , _arXiv e-prints_ (2020) arXiv:2011.02772 [2011.02772].
  * [9] C. Boehm, A. Kobakhidze, C.A.J. O’Hare, Z.S.C. Picker and M. Sakellariadou, _Eliminating the LIGO bounds on primordial black hole dark matter_ , _arXiv e-prints_ (2020) arXiv:2008.10743 [2008.10743].
  * [10] J.L.G. Sobrinho and P. Augusto, _Stellar mass primordial black holes as cold dark matter_ , _MNRAS_ 496 (2020) 60 [2005.10037].
  * [11] K.W.K. Wong, G. Franciolini, V. De Luca, V. Baibhav, E. Berti, P. Pani et al., _Constraining the primordial black hole scenario with Bayesian inference and machine learning: the GWTC-2 gravitational wave catalog_ , _arXiv e-prints_ (2020) arXiv:2011.01865 [2011.01865].
  * [12] J. Garcia-Bellido, J.F. Nuño Siles and E. Ruiz Morales, _Bayesian analysis of the spin distribution of LIGO/Virgo black holes_ , _arXiv e-prints_ (2020) arXiv:2010.13811 [2010.13811].
  * [13] O. Burke, J.R. Gair, J. Simón and M.C. Edwards, _Extreme precision for extreme spin: probing near-extremal black holes using LISA_ , _arXiv e-prints_ (2020) arXiv:2010.05932 [2010.05932].
  * [14] V. De Luca, G. Franciolini, P. Pani and A. Riotto, _The evolution of primordial black holes and their final observable spins_ , _JCAP_ 2020 (2020) 052 [2003.02778].
  * [15] A. Arbey, J. Auffinger and J. Silk, _Evolution of primordial black hole spin due to Hawking radiation_ , _MNRAS_ 494 (2020) 1257 [1906.04196].
  * [16] S.W. Hawking, _Particle creation by black holes_ , _Communications in Mathematical Physics_ 43 (1975) 199.
  * [17] S.W. Hawking, _Black hole explosions?_ , _Nature_ 248 (1974) 30.
  * [18] B. Carr, F. Kuhnel and L. Visinelli, _Black Holes and WIMPs: All or Nothing or Something Else_ , _arXiv e-prints_ (2020) arXiv:2011.01930 [2011.01930].
  * [19] K. Inomata, M. Kawasaki, K. Mukaida, T. Terada and T.T. Yanagida, _Gravitational wave production right after a primordial black hole evaporation_ , _Phys. Rev. D_ 101 (2020) 123533 [2003.10455].
  * [20] D. Hooper, G. Krnjaic, J. March-Russell, S.D. McDermott and R. Petrossian-Byrne, _Hot Gravitons and Gravitational Waves From Kerr Black Holes in the Early Universe_ , _arXiv e-prints_ (2020) arXiv:2004.00618 [2004.00618].
  * [21] A. Arbey and J. Auffinger, _BlackHawk: a public code for calculating the Hawking evaporation spectra of any black hole distribution_ , _European Physical Journal C_ 79 (2019) 693 [1905.04268].
  * [22] D.N. Page, _Particle emission rates from a black hole: Massless particles from an uncharged, nonrotating hole_ , _Phys. Rev. D_ 13 (1976) 198.
  * [23] D.N. Page, _Particle emission rates from a black hole. II. Massless particles from a rotating hole_ , _Phys. Rev. D_ 14 (1976) 3260.
  * [24] D.N. Page, _Particle emission rates from a black hole. III. Charged leptons from a nonrotating hole_ , _Phys. Rev. D_ 16 (1977) 2402.
  * [25] R. Brito, V. Cardoso and P. Pani, _Superradiance: New Frontiers in Black Hole Physics_ , vol. 906, Springer (2015), [1501.06570].
  * [26] J.H. MacGibbon and B.R. Webber, _Quark- and gluon-jet emission from primordial black holes: The instantaneous spectra_ , _Phys. Rev. D_ 41 (1990) 3052.
  * [27] T. Sjöstrand, S. Ask, J.R. Christiansen, R. Corke, N. Desai, P. Ilten et al., _An introduction to PYTHIA 8.2_ , _Computer Physics Communications_ 191 (2015) 159 [1410.3012].
  * [28] J. Bellm, G. Bewick, S. Ferrario Ravasio, S. Gieseke, D. Grellscheid, P. Kirchgaeßer et al., _Herwig 7.2 release note_ , _European Physical Journal C_ 80 (2020) 452 [1912.06509].
  * [29] M. Cirelli, G. Corcella, A. Hektor, G. Hütsi, M. Kadastik, P. Panci et al., _PPPC 4 DM ID: a poor particle physicist cookbook for dark matter indirect detection_ , _JCAP_ 2011 (2011) 051 [1012.4515].
  * [30] J.F. Navarro, C.S. Frenk and S.D.M. White, _The Structure of Cold Dark Matter Halos_ , _Astrophys. J._ 462 (1996) 563 [astro-ph/9508025].
  * [31] M.H. Chan and C.M. Lee, _Constraining primordial black hole fraction at the galactic centre using radio observational data_ , _MNRAS_ 497 (2020) 1212 [2007.05677].
  * [32] R. Laha, _Primordial Black Holes as a Dark Matter Candidate Are Severely Constrained by the Galactic Center 511 keV $\gamma$ -Ray Line_, _Phys. Rev. Lett._ 123 (2019) 251101 [1906.09994].
  * [33] R. Laha, J.B. Muñoz and T.R. Slatyer, _I N T E G R A L constraints on primordial black holes and particle dark matter_ , _Phys. Rev. D_ 101 (2020) 123514 [2004.00627].
  * [34] R. Laha, P. Lu and V. Takhistov, _Gas Heating from Spinning and Non-Spinning Evaporating Primordial Black Holes_ , _arXiv e-prints_ (2020) arXiv:2009.11837 [2009.11837].
  * [35] B. Dasgupta, R. Laha and A. Ray, _Neutrino and Positron Constraints on Spinning Primordial Black Hole Dark Matter_ , _Phys. Rev. Lett._ 125 (2020) 101101 [1912.01014].
  * [36] S. Wang, D.-M. Xia, X. Zhang, S. Zhou and Z. Chang, _Constraining the Primordial Black Holes as Dark Matter at JUNO_ , _arXiv e-prints_ (2020) arXiv:2010.16053 [2010.16053].
  * [37] G. Ballesteros, J. Coronado-Blázquez and D. Gaggero, _X-ray and gamma-ray limits on the primordial black hole abundance from Hawking radiation_ , _Physics Letters B_ 808 (2020) 135624 [1906.10113].
  * [38] A. Arbey, J. Auffinger and J. Silk, _Constraining primordial black hole masses with the isotropic gamma ray background_ , _Phys. Rev. D_ 101 (2020) 023010 [1906.04750].
  * [39] G. Johnson, _Primordial black hole constraints with large extra dimensions_ , _JCAP_ 2020 (2020) 046 [2005.07467].
  * [40] Y. Luo, C. Chen, M. Kusakabe and T. Kajino, _Impacts of Hawking Radiation from Primordial Black Holes in Critical Collapse Model on the Light Element Abundances_ , _arXiv e-prints_ (2020) arXiv:2011.10937 [2011.10937].
  * [41] H. Kim, _A constraint on light primordial black holes from the interstellar medium temperature_ , _arXiv e-prints_ (2020) arXiv:2007.07739 [2007.07739].
  * [42] A. Coogan, L. Morrison and S. Profumo, _Direct Detection of Hawking Radiation from Asteroid-Mass Primordial Black Holes_ , _arXiv e-prints_ (2020) arXiv:2010.04797 [2010.04797].
  * [43] A. Arbey and J. Auffinger, _Detecting Planet 9 via Hawking radiation_ , _arXiv e-prints_ (2020) arXiv:2006.02944 [2006.02944].


Animal Avatars: Reconstructing Animatable 3D Animals from Casual Videos

Remy Sabathier1,2 Niloy J. Mitra2 David Novotny1

R. Sabathier, N. Mitra, D. Novotny

Meta University College London

We present a method to build animatable dog avatars from monocular videos.
This is challenging as animals display a range of (unpredictable) non-rigid movements and have a variety of appearance details (e.g., fur, spots, tails).
We develop an approach that links the video frames via a 4D solution that jointly solves for animal's pose variation, and its appearance (in a canonical pose).
To this end, we significantly improve the quality of template-based shape fitting by endowing the SMAL parametric model with Continuous Surface Embeddings (CSE), which brings image-to-mesh reprojection constaints that are denser, and thus stronger, than the previously used sparse semantic keypoint correspondences.
To model appearance, we propose an implicit duplex-mesh texture that is defined in the canonical pose, but can be deformed using SMAL pose coefficients and later rendered to enforce a photometric compatibility with the input video frames.
On the challenging CoP3D and APTv2 datasets, we demonstrate superior results (both in terms of pose estimates and predicted appearance) to existing template-free (RAC) and template-based approaches (BARC, BITE).
Video results and additional information accessible on the .

§ INTRODUCTION

Building poseable reconstructions of animals captured by consumer imaging devices is a valuable technology with numerous applications in augmented and virtual reality.
Among many possible animal species, the reconstruction of canines is of particular interest primarily due to their important role in the lives of their two-legged friends.

While it is nowadays possible to reliably reconstruct rigid scenes captured from a moving vantage point <cit.>, the reconstruction of non-rigid shapes is significantly less constrained and, as such, a more challenging problem.
Here, many recent works focused on generic animal reconstruction from multi-view (videos) <cit.> or single-view 2D image supervision  <cit.> without prior knowledge of the animal shape.
While the latter demonstrated impressive progress, the challenging nature of the problem limits the applicability to scenarios with simple deformations and good multi-view test-time coverage.

Animal Avatars. Given a monocular video of a dog, we propose a template-based method to reconstruct the shape $\beta$, motion $\theta_{t}$ and texture $\psi$. We address the challenge of insufficient supervision for unconventional views by integrating Continuous Surface Embeddings with an articulated mesh. We introduce a novel implicit duplex-mesh texture model, jointly optimized alongside motion parameters.

To better constrain the non-rigid 3D reconstruction, inspired by state-of-the-art methods for Human 3D reconstruction <cit.>, we leverage a known category template.
Specifically, we use the SMAL template introduced by
Zuffi  <cit.> - a quadruped-equivalent of the seminal SMPL parametric human model <cit.>.
The template has been used to enable single-view 3D dog reconstruction trained on an extensive collection of dog images semi-manually annotated with SMAL poses <cit.>.
While this approach provides a clear state-of-the-art in monocular 3D dog-shape reconstruction, the inherent ambiguity of single-view reconstruction provides many challenges.

To further increase our chances of successful reconstruction, besides leveraging the SMAL model, we turn our attention towards reconstructing casual video captures of dogs, which was first explored in <cit.>.
Indeed, videos provide a stronger multi-view supervision which significantly simplifies the 3D shape fitting problem.
However, regardless of monocular <cit.> or multi-view conditioning <cit.>, we observed a common failure mode in existing methods when animals are viewed from non-frontal views.
This issue arises because fitting relies on sparse joint-reprojection constraints that mainly cover front-facing parts, offering limited supervision for rear and side views.

Hence, our first contribution is to replace the sparse keypoint supervision with a denser alternative.
Specifically, we exploit Continuous Surface Embeddings <cit.> (CSE), which annotate each vertex of the CSE dog mesh with a unique descriptor.
Furthermore, CSE provides a pre-trained deep image-to-CSE predictor, that labels image pixels with their corresponding CSE descriptor and transitively with the matching mesh vertex.
In this work, in an one-time process, we first transform CSE descriptors to the SMAL mesh by means of a semi-manual non-rigid aligment.
This then enables a stronger keypoint loss providing reprojection constraints for all points on the animal's body, even in rear views.

Secondly, we enhance fits by exploiting the inherent smoothness of animal movements over time.
Previous attempts incorporated this knowledge by enforcing temporal smoothness on the deformation coefficients of the SMAL template.
However, we found this approach flawed because the coefficients have to represent both the smooth animal motion and the camera motion, which is often non-smooth due to the unstable camera operator.
Instead, we propose to represent the SMAL deformation as a combination of accurate Structure-from-Motion camera and optimized animal motion, allowing for proper temporal regularization.
Importantly, SfM also provides intrinsic parameters of each camera (focal length), which facilitates more accurate shape fitting through loss terms that require rendering.

Finally, we are the first to enable texturing of the SMAL mesh by leveraging it as a scaffold for an implicit duplex-mesh neural radiance field, which can be rendered while accounting for body deformations.
Our approach involves defining implicit shape and color functions on a subset of the 3D domain bounded by enlarged and downsized versions of the mesh template.
This allows for articulation of the corresponding implicit surface by posing the boundary meshes similarly to the original mesh.

We evaluate the color and 3D shape-fitting accuracy on the CoP3D dataset <cit.>, containing crowd-sourced “turntable” videos of dogs captured by smartphone cameras, achieving performance superior to template-based <cit.> and template-free <cit.> baselines. We also compare our pose estimation quality on the recent APTv2 dataset <cit.>
and report results superior compared to video-based methods dog-specific <cit.>.

§ RELATED WORK

Video reconstruction on humans.

Recent works in 3D human pose reconstruction show impressive results on videos, representing detailed motions and robustness to occlusions <cit.>. Several factors support the progress in this area.

The majority of methods rely on the parametric SMPL model introduced in <cit.> and refined in <cit.>. Unlike the existing animal models, it is learned from a large collection of real 3D scans of humans, which entails stronger expressiveness. The model provides parametric handles to both shape and pose variations.
Additionally, human-centric models benefit from large real datasets with keypoint annotations <cit.> as well as synthetic 3D datasets <cit.>.

We also note the availability of off-the-shelf models for related tasks such as human key-point identification <cit.>.
Such models can guide training of 3D reconstruction or provide soft annotation on large unlabelled datasets <cit.>.
These factors explain why 3D animal reconstruction cannot directly benefit from breakthroughs in the human domain.

Additionally, there are several works <cit.> targeting human reconstruction with texture from monocular and multi-view sources achieving high rendering quality by leveraging off-the-shelf human pose estimation models.

Template-based animal reconstruction.
Based on the success of template-based human reconstruction using SMPL, <cit.> introduced SMAL, a parametric model for quadruped animals.
Unlike the SMPL model, which is supervised by scans of real humans, and due to the many challenges of scanning live quadruped animals, the SMAL model is only trained with scans of toy animal models.

To add to the challenge, no large-scale dataset with 3D annotations for dog reconstructions exists.
The only 3D-annotated dataset <cit.> does not adequately represent the diversity of dog breeds and poses.
The most relevant image dataset is StanfordExtra <cit.>, a collection of dog images with silhouette and joint annotations.
Despite being diverse, spanning different dog breeds and environments, the dataset is biased towards front-facing views.
This motivates our choice of the CoP3D dataset <cit.>, an extensive collection of pet videos with high view-point variability in each video.

Similar to human reconstruction research, multiple works estimate shape attributes $\beta$ and pose attributes $\theta$ for the SMAL model from a single image, relying on 2D reprojection constraints.
<cit.> predicts skeleton joints to find an initial solution, which is then refined to match keypoints estimated from the animal silhouette. <cit.> propose a coarse-to-fine strategy, where an initial solution is refined through a graph-convolutional network. BARC <cit.> enforces similar shape attributes for dogs of the same breed, which is predicted by a deep network. BITE <cit.>, an extension of BARC, improve prediction plausibility with ground-contact and ground-plane losses, and improve accuracy with an iterative refinement loop. We compare ours against BARC and BITE to show how multi-view supervision and CSE embeddings significantly improve performance, especially on the challenging videos from the COP3D dataset. We note some additional related works on other species <cit.>.

Template-free animal reconstruction.
The template-free setup enabling reconstruction of a wider range of animals has also been considered.
These approaches build a 3D representation by analyzing a collection of images, or frames of a single video, or videos of the same species.

Progress made in differentiable rendering <cit.> supported the analysis-by-synthesis method for animal reconstruction from a single video. Recent works
<cit.> propose to minimize silhouette and photometric losses in order to jointly learn camera parameters and a textured frame-independent linear deformation 3D model.
We are inspired by Viser <cit.> that recovered articulated humans from monocular videos.
They learn long-range 2D point tracks using object masks and optical flow, and a video-specific embedding linking pixel appearance to surface points on a canonical deformable mesh.
However, such approaches are vulnerable to (self-)occlusions, especially in significantly dynamic scenes.

To overcome these issues, several works train on a set of videos. BANMo <cit.> leverages multiple videos of the same subject to build an implicit canonical representation, posed via a differentiable neural blend skinning method. Similar to our work, they use pixel CSE predictions on the different images to link it with a 3D embedding defined in the canonical space.
As such, satisfactory new views are generated only when positioned relatively close to the training views.
Recently, RAC <cit.> extended BANMo to learn a general and instance-specific model from a set of videos from the same category of deformable objects, including the dog category analyzed here.
TrackeRF <cit.> extends PixelNeRF <cit.> to time-deforming shapes but it only predicts novel views without a full-scale 3D animal-body model.
Note that the method additionally needs to be initialized with a basic hierarchical joint skeleton.
We compare with RAC in <ref> and omit comparison to TrackeRF because its source code is not available.

Method overview. Two stage process: First, we initialize root pose $g_{t}^{0}$ via PnP-RANSAC, utilizing CSE mesh-pixel correspondences. Then, we jointly optimize shape $\beta$, time-varying pose $\theta_{t}$ and implicit texture $\psi$ through an analysis-by-synthesis approach, leveraging mask $\mathcal{L}^{mask}$, dense correspondence $\mathcal{L}^{cse}$ (optimized models in orange), and
photometric $\mathcal{L}^{photo}$ signals.

§ METHOD

Our goal is to reconstruct the 3D shape and appearance of a dog captured in a monocular video, i.e., given a tuple $(I_t)_{t=1}^{T}$ of $T \in \mathbb{N}$ video frames
we output a tuple $(S_t)_{t=1}^{T}$ of colored 3D shapes $S_t$ of the animal in each frame $I_t \in \mathbb{R}^{3 \times H \times W}$.

In <ref>, we detail the shape representation $S$,
while <ref> describe the optimization process recovering the shape $S$ given input images $I$.

§.§ Shape and Appearance Representation

Our method defines the colored shape representation $S_t$ at time $t$ as a 3-tuple

\begin{equation} \label{eq:main_shape_rep}
S_t := (\beta, \theta_t, \psi),
\end{equation}

where $\beta$ and $\psi$ define the intrinsic (i.e., time-invariant) deformation and the texture of the dog body respectively,
and $\theta_t$ is the time-dependent pose of its skeleton.
In what follows, we detail these three sets of parameters.

3D shape representation $\beta,\theta_t$.
There is a plethora of articulated 3D shape representations ranging from universal less-constrained 3D-flow functions <cit.> to category-specific Linear-Blend-Skinning (LBS) models attached to a fixed surface-mesh template <cit.>.
Since we focus on a certain animal category (i.e., dog) with a well-defined space of plausible articulations and body deformations, we opt for the latter, i.e., a template-based shape model.

Specifically, we represent the 3D geometry (i.e., the parameters $\theta_t, \beta$ in <ref>) of each dog with the SMAL model <cit.>.
The latter comprises a deformable template mesh $(\hat{\mathcal{V}}, \mathcal{F})$ with vertices $\hat{\mathcal{V}} \in \mathbb{R}^{\nsmallpts \times 3}$ and a list of triangular faces $\mathcal{F} \in \mathbb{N}^{\nsmallfaces \times 3}$.
The deformation of $\hat{\mathcal{V}}$ is defined as a function

\begin{equation} \label{eq:smal_deformation}
F(\hat{\mathcal{V}}, \beta, \theta_t) := \mathcal{V} \in \mathbb{R}^{\nsmallpts \times 3},
\end{equation}

conditioned on shape parameters (PCA coefficients and bone lengths) $\beta \in \mathbb{R}^{d_\beta}$, controlling the non-rigid deformation of the animal shape in its canonical pose, and pose parameters $\theta_t := (g^0, \theta^J)$.
The latter has two components: (i) a root transformation $g^0 \in \mathbb{SE}(3)$ that represents the overall rigid transformation of the dog body; and (ii) angles $\theta^J \in \mathbb{R}^{d_J}$ of the animal joints that control the deformation of its limbs.

Time-deforming SMAL.
To represent the time-varying deformation of a dog, we estimate a tuple $(\theta_t)_{t=1}^{T}$ comprising SMAL pose coefficients $\theta_t$ for each of the $T$ video frames, and a single vector
$\beta$ because, typically, the intrinsic deformation is time-invariant.
Since the animals often move smoothly in time, the pose $\theta_t$ is defined as a function
of a smooth temporal basis $\gamma(\tau(t))$ as follows:

\begin{equation}
\theta_t := \theta^\text{MLP}(\tau(t)),
\end{equation}

implemented by a shallow multi-layer perceptron $\theta^\text{MLP}$ accepting
positional encoding $\gamma$ <cit.> applied to the timestamp $\tau(t) \in \mathbb{R}^+$ of frame $I_t$.

Continuous SMAL Embeddings.
Besides the deformation parameters ($\theta, \beta$), SMAL also defines joint locations $\hat{\mathcal{J}} \in \mathbb{R}^{N_J \times 3}$ as a sparse subset of 3D surface points,
which are linearly regressed from the vertex locations.
Since the joints correspond to semantic parts of the animal body (paws, ear tips), they can improve shape-fitting accuracy via establishing correspondences between the SMAL mesh and the detections of the body parts in the images.
However, while these keypoints improve performance in <cit.>, where most animals are photographed from their side or frontal views, they are inadequate for our video fly-arounds containing rear views with little-to-no visible keypoints (see <ref> and <ref>).

Hence, instead of sparse landmarks, we exploit Continuous Surface Embeddings (CSE) <cit.>, which attach a unique embedding vector $e_k \in \mathbb{R}^{d_e}$ to each vertex $X_k \in \hat{\mathcal{V}}$ of the (canonical) SMAL template such that the dimensions of $e$ vary smoothly over the mesh surface.
CSE also provides a deep predictor that annotates each image pixel with its corresponding mesh coordinate $e_k$.
Thus, unlike sparse keypoints, the latter densely annotates images of animals from arbitrary viewpoints, including rear views.

Because the original template mesh of CSE <cit.> is different from the SMAL template, following <cit.>, we transform the CSE coordinate map to SMAL using a customized variant of the Zoom-Out method <cit.>.
The latter results in the final set of SMAL-CSE vertex coordinates $e_k$ (alignment visualized in <ref>).

SMAL CSE. We align the CSE mesh template (top left) with the SMAL template (top right) in order to setup the CSE coordinate frame over the surface of the SMAL mesh.
In combination with a pretrained image-to-CSE predictor, this allows establishing dense correspondences between surface points of the SMAL template and the corresponding image pixels.
Note that the image CSE detections (rows labelled “CSE”) provide dense correspondence covering all parts of the dog body, which is not the case of sparse keypoints (“Keypoints” rows).

Double-mesh implicit texture $\psi$.
Besides reconstructing 3D animal shapes, we also aim to recover the texture of the animal body.
We require an exact supporting 3D shape to learn an accurate texture model.
However, due to the low expressivity of the SMAL deformation space, the posed mesh can only represent the surface of a dog up to a certain error.
Thus, following recent advances in new-view synthesis of humans <cit.>, we leverage the template mesh as a scaffold supporting a more accurate implicit radiance field <cit.>, which we describe next.

Our method is inspired by duplex radiance fields <cit.>, originally proposed for speeding up rendering of neural radiance fields <cit.>.
Specifically, we first extrude the 2-manifold canonical surface to a 3D volume by defining an $\epsilon$-neighborhood $\hat{\mathcal{N}}^\epsilon \subset \mathbb{R}^3$ as a 3D subspace bounded by an outer mesh with vertices $\hat{\mathcal{V}}^\uparrow = (1 + \epsilon) \hat{\mathcal{V}}$, and an inner mesh $\hat{\mathcal{V}}^\downarrow = (1 - \epsilon) \hat{\mathcal{V}}$, both sharing the same faces $\mathcal{F}$.
In practice, we obtain the offset meshes by moving along directions of vertex normals.
Similar to the template mesh itself, both $\hat{\mathcal{V}}^\uparrow$ and $\hat{\mathcal{V}}^\downarrow$ can be deformed with $\theta, \beta$ resulting in the posed neighborhood $\mathcal{N}^\epsilon$ bounded by $F(\hat{\mathcal{V}}^\uparrow, \beta, \theta)$ and $F(\hat{\mathcal{V}}^\downarrow, \beta, \theta)$.

Within this neighborhood, we then define an opacity function $\psi_\sigma: \mathcal{N}^\epsilon \mapsto \mathbb{R}^+$, annotating 3D locations $\hat{\bx}$ in the mid-space with presence ($\psi_\sigma(\hat{\bx}) > 0$) or absence ($\psi_\sigma(\hat{\bx}) \approx 0$) of the surface, and the radiance function
$\psi_c: \mathcal{N}^\epsilon \times \mathbb{S}^2 \mapsto[0, 1]^3$,
which colors the space depending on the direction $\br \in \mathbb{S}^2$ from which the input point $\hat{\bx}$ is observed.
The coloring and opacity functions are implemented as in <cit.>, i.e., using a shallow MLP decoding a learnable triplane feature grid.
Please refer to the supplementary material for details.

Implicit duplex-mesh model.
We propose a novel deformable implicit shape model.
Using the radiance and opacity functions $\psi_c$ and $\psi_\sigma$ defined over an $\mathbb{R}^3$ subspace bounded by a canonical duplex mesh with vertices $\mathcal{\hat{V}}^\uparrow$ and $\mathcal{\hat{V}}^\downarrow$, we render a color of the posed duplex mesh via EA raymarching over a canononical ray $\hat{\br}_\bu$.
The latter is obtained by transforming into the canonical space the intersections of the view-space ray $\br_\bu$ with the posed duplex mesh $F(\mathcal{\hat{V}}^\uparrow, \beta, \theta), F(\mathcal{\hat{V}}^\downarrow, \beta, \theta)$.

Double-mesh rendering.
Having defined the animal shape $(\mathcal{V}, \mathcal{F})$ and the implicit texture $\psi$, we image the latter from an arbitrary camera viewpoint $P$ using a differentiable rendering function $\mathcal{R}$:

\begin{equation}
\mathcal{R}(P, \mathcal{V}, \mathcal{F}, \psi) := \bar{I},
\end{equation}

which outputs the render $\bar{I} \in [0, 1]^{3 \times H \times W}$ as observed from the camera with projection matrix $P \subset \mathbb{R}^{3 \times 4}$.

To obtain $\bar{I}$, we iterate over all its pixels $\bu \in [1..H] \times [1..W]$ and march with Emission-Absorption (EA) over the canonical ray $\hat{\br}_\bu$ defined as the camera-space ray $\br_\bu = P^{-1} \bu$ in the rest-pose coordinates.
Specifically, $\br_\bu$ is first intersected with the posed outer boundary mesh $F(\hat{\mathcal{V}}^\uparrow, \beta, \theta)$, and then the intersection's barycentric coordinates are applied to the corresponding canonical mesh $\hat{\mathcal{V}}^\uparrow$ yielding a 3D point $\hat{\bx}_\bu^\uparrow \in \hat{\mathcal{N}}^\epsilon$ in the canonical neighborhood $\hat{\mathcal{N}}^\epsilon$.
Repeating the same for the for the inner mesh $F(\hat{\mathcal{V}}^\uparrow, \beta, \theta)$ yields a second 3D point $\hat{\bx}^\downarrow_\bu \in \hat{\mathcal{N}}^\epsilon$.
The two points $\hat{\bx}_\bu^\uparrow$ and $\hat{\bx}^\downarrow_\bu$ then define the canonical ray $\hat{\br}_\bu$, over which we march with EA, accumulating the outputs of the radiance functions $\psi_c$ and $\psi_\sigma$ in the process, to render the final color of pixel $\bu$ (details in the supplementary).

Note that our EA rendering differs from the duplex radiance fields <cit.>, which instead leverage an MLP to directly map positional encodings of the two intersection points  $\hat{\bx}^\downarrow_\bu$, $\hat{\bx}_\bu^\uparrow$ to a surface color.

Appearance representation $\psi$
Besides reconstructing 3D animal shape, our work also aims to recover the texture of the animal body.
To do so, we describe the mesh color via a time-invariant coloring function
$c: \mathbb{R}^{d_e} \mapsto[0, 1]^3$
assigning to each coordinate $e \in \mathbb{R}^{d_e}$ on the surface of the mesh $(\mathcal{V}, \mathcal{F})$ a three-dimensional RGB color $c(e) \in [0, 1]^3$.
Here, $c$ is implemented as a two-layer perceptron

\begin{equation}
c^\text{MLP}_\psi(e) = \text{MLP}_\psi(\gamma(e)),
\end{equation}

with parameters $\psi$ accepting the positional encoding $\gamma(e)$ <cit.> of CSE coordinates $e$.
Note that this is convenient because we are effectively representing the surface-smooth texture as a function of the similarly-smooth coordinates $e$.

Shape rendering
Having defined the animal shape $(\mathcal{V}, \mathcal{F})$ and texture $c$, we image the latter from an arbitrary camera viewpoint $P$ using a differentiable rendering function $R$:

\begin{equation}
R(P, \mathcal{V}, \mathcal{F}, \psi) = \bar{I},
\end{equation}

which outputs the render $\bar{I} \in [0, 1]^{3 \times H \times W}$ as observed from the camera with projection matrix $P \subset \mathbb{R}^{3 \times 4}$.
Along with the latter, $R$ takes as input the SMAL faces $\mathcal{F}$ and the deformed vertices $\mathcal{V} = (\hat{\mathcal{V}}, \beta, \theta)$ of the SMAL template.

$R$ also accepts parameters $\psi$ of the appearance MLP $c^\text{MLP}_\psi$ which define the rendered colors.
To render the color image $\bar{I}$, we first output a CSE render via barycentric interpolation of the SMAL-CSE vertex texture $\{e_k\}_{k=1}^{\nsmallpts}$ using the rendered fragment buffer, followed by evaluating the coloring MLP $c^\text{MLP}_\psi$ at each pixel of the CSE render.
We implement $R$ using the differentiable mesh rasterizer from PyTorch3D <cit.>.

§.§ Pose Initialization

Fitting a non-rigid shape to a monocular video is a challenging task and, as such, a trivial random initialization of the shape parameters (the weights of MLPs $\psi, \chi$) inevitably leads to a failure.
We thus employ two fitting stages, where the first initializes parameters to ensure convergence of the second stage.

Root pose estimation.
We observed that a suitable initialization of the root pose $g^0$, while initializing the rest of the parameters (limb angles $\theta^J$, intrinsic deformation $\beta$, implicit MLP $\psi$) randomly, is sufficient to avoid the most common local minima, such as flipping of the dog body along its symmetry axes.

The goal of the initial fitting stage is thus to recover the root rigid transformations $g^0_t$ for $t\in[1,T]$ so that the perspective projection of the unposed canonical template $\hat{\mathcal{V}}$ into each camera $P_t$ matches the depicted dog in frame $t$.

PnP with CSE.
To this end, we leverage the CSE coordinate map of the SMAL mesh (<ref>).
First, for each image $I_t$, a pre-trained CSE convolutional network annotates pixels $\mathbf{u}_t \in [1..H]\times[1..W]$ with their CSE embedding $e(\mathbf{u}_t)$.
Then, we establish correspondences between each pixel $\mathbf{u}_t$ and the vertices $\hat{X}_{\text{NN}^e(\mathbf{u}_t)} \in \hat{\mathcal{V}}$ on the template mesh $\hat{\mathcal{V}}$ by recovering the nearest neighbors
$\text{NN}^e(\bu_t) := \argmin_{1 \leq k \leq |\mathcal{V}|} \| e(\mathbf{u}_t) - e_k \|$
in the CSE embedding space.
Given the set
$\{(\bu_t, \hat{X}_{\text{NN}^e(\bu_t)})\}$
of pixel-to-vertex correspondences, PnP-RANSAC <cit.> estimates the best camera $P_t^\text{PnP}$ aligning the projections of the vertex points with their corresponding pixels.

In order to increase robustness to occasional failures of PnP caused by inaccurate CSE predictions, we employ a collective pose refinement that finds a single global rigid transformation $g^\text{PnP} \in \mathbb{SE}(3)$ aligning the sequence of PnP-estimated cameras $(P_t^\text{PnP})_{t=1}^T$ with the scene SfM cameras $(P_t^\text{SfM})_{t=1}^T$ (the SfM cameras are detailed in the next section).
The latter then initializes the root-rigid transformation of each frame, i.e., $\forall t: g^0_t = g^\text{PnP}$.

§.§ Shape Fitting

We now detail the second fine-fitting stage, which optimizes all shape parameters ($\beta, \theta_t, \psi$) given the initial poses $g^0$.

Factoring the rigid pose.
Even with near-perfect initialization of the root rigid pose, it is challenging to converge to a good solution when, as done in previous works <cit.>, the rigid component $g_t = g^0_t \in \mathbb{SE}(3)$ of the rendering camera $P_t$ is solely represented with the root transformation $g^0_t$.
Such optimization is challenging because $g^0_t$ has to represent the animal pose and also compensate for complex camera motions such as the jitter caused by unstable handling.

Instead, we factor the extrinsics $g_t = g^\text{cam}_t \cdot g^0_t$ of each rendering camera $P_t$ as a composition of the camera motion $g^\text{cam}_t \in \mathbb{SE}(3)$ w.r.t. the rigid scene background and the motion $g^0_t \in \mathbb{SE}(3)$ of the dog w.r.t. the background.
Here, $g^\text{cam}_t := g^\text{SfM}_t$ is fixed to the SfM camera $g^\text{SfM}_t$ estimated by COLMAP <cit.> which we empirically found to be very accurate.
Offloading the camera estimate to SfM, we are then left with the simpler task of regressing the temporally-smooth rigid animal motion $g^0_t$.

CSE-guided fine shape fitting.
The second fine fitting stage outputs all parameters ($\beta, \theta_t, \psi$) by optimizing the shape predictor $\theta^\text{MLP}$ and the implicit shape $\psi$ using the following loss function:

\begin{equation} \label{eq:main_objective}
\sum_{t=1}^T (
\mathcal{L}^\text{cse}_t
+ \mathcal{L}^\text{kp}_t
+ \mathcal{L}^\text{photo}_t
+ \mathcal{L}^\text{mask}_t
+ \mathcal{L}^\text{reg}_t).
\end{equation}

The latter calculates frame-specific losses $\mathcal{L}_t^\cdot$ and sums them over all $T$ training images.
The next paragraphs detail each loss term.

(i) CSE-keypoint loss.
Besides leveraging CSE for the pose initialization, we also guide the fitting process with the following CSE keypoint loss:

\begin{equation} \label{eq:cse_keypoint_loss}
\mathcal{L}^\text{cse}_t = \sum_{\mathbf{u}_t \in M_t} \| \mathbf{u}_t - P_t F(\mathcal{V}, \beta, \theta_t)_{\text{NN}^e(\mathbf{u}_t)} \|,
\end{equation}

integrating the reprojection distances between each foreground pixel $\mathbf{u}_t \in M_t$ and the corresponding 2D projections $P_t F(\mathcal{V}, \beta, \theta_t)_{\text{NN}^e(\mathbf{u}_t)}$ of $\mathbf{u}_t$'s CSE correspondence $\text{NN}^e(\mathbf{u}_t)$ on the mesh $F(\mathcal{V}, \theta_t, \beta)_{\text{NN}^e(\mathbf{u}_t)}$.
The mesh is deformed with parameters $\theta_t = \theta^\text{MLP}(\tau(t))$ given by the pose predictor $\theta^\text{MLP}$ for time-stamp $\tau(t)$ of the frame $t$.
Here, $M_t \in \{0, 1\}^{H \times W}$ corresponds to the foreground segmentation extracted with a pre-trained segmenter <cit.>.

(ii) Sparse-keypoint loss.
We complement the main CSE-keypoint training signal with the standard sparse-keypoint loss
\mathcal{L}^\text{kp}_t = \sum_{k=1}^{N_J} \| h(I_t)_k - P_t J_k \|,
minimizing the distance between the projection $P_t J_k$ and the detection $h(I_t)_k$ of the $j$-th SMAL joint $J_k = F(\mathcal{\hat{J}}, \beta, \theta_t)_k$ in the articulation of the image $I_t$.
Here $h(I_t) \in \mathbb{R}^{N_J \times 2}$ denotes the 2D keypoint detections from BARC's sparse dog keypoint detector <cit.>.
We discovered that sparse keypoints improve fitting on thin body structures, such as paws, where the CSE detections are less accurate.

(iii) Photometric loss.
To train the appearance predictor, and to provide an additional supervision for fine 3D fitting, we minimize the photometric loss:

\begin{equation} \label{eq:photometric_loss}
\mathcal{L}^\text{photo}_t =
\text{LPIPS}(
M_t \odot I_t
\mathcal{R}(
F(\mathcal{V}, \theta_t, \beta),
\mathcal{F},
\psi^\text{MLP}
\end{equation}

comprising the Learned Perceptual Image Patch Similarity <cit.> (LPIPS) between the masked ground-truth image $M_t \odot I_t$ and the RGB render of the posed mesh $F(\mathcal{V}, \theta_t, \beta)$ colored by the implicit duplex-mesh MLP $\psi^\text{MLP}$.

(iv) Chamfer mask loss.
Similar to <cit.>, we minimize a silhouette loss $\mathcal{L}^\text{mask}_t$:

\begin{equation} \label{eq:mask_loss}
\mathcal{L}^\text{mask}_t =
\{ \bu_t | \bu_t \in M_t \},
\{ P_t X_k | X_k \in F(\mathcal{V}, \theta_t, \beta)\}
\end{equation}

evaluating Chamfer distance between the set of occupied 2D pixels $\bu_t \in M_t$ of the g. t. mask $M_t$,
and the 2D projections of the posed-mesh vertices $F(\mathcal{V}, \theta_t, \beta)$.

(v) Shape regularizers.
To avoid implausible mesh articulations, we also employ shape regularization loss
\mathcal{L}^\text{reg}_t =
\mathcal{L}^\text{arap}_t
+ \mathcal{L}^\text{edge}_t
comprising the As-Rigid-As-Possible (ARAP) regularizer $\mathcal{L}^\text{arap}_t$ <cit.>,
and the edge-length penalty $\mathcal{L}^\text{edge}_t$ <cit.> enforcing local rigidity of the posed template. See supplementary for details.

Qualitative comparison.
We note that, unlike template-based approaches, the reconstructed meshes from RAC are very far from the actual shape of a dog. To evaluate temporal consistency, please refer to the result videos.

§ EXPERIMENTS

Average results on 50 sequences from COP3D and 39 sequences from APTv2. Quality of pose estimates is measured by IoU, IoUw5, $err_\text{track}$; appearance quality is measured by PSNR, PSNRw5, LPIPS.
Note that BARC and BITE only estimate pose and hence we cannot evaluate appearance quality, indicated by `$\times$'.

Dataset     5c| <cit.>      <cit.>
IoU $\uparrow$      IoUw5 $\uparrow$     PSNR $\uparrow$     PSNRw5 $\uparrow$     LPIPS $\downarrow$     $err_\text{track}$ $\downarrow$
BARC <cit.>            0.75      0.47     $\times$      $\times$         $\times$      0.047

BITE <cit.>             0.81      0.59      $\times$      $\times$         $\times$     0.047

RAC <cit.>     0.76      0.52     21.86      17.51         0.164     0.093

Ours                                    0.84      0.79     22.12      19.40         0.041     0.035

We evaluate all models on COP3D <cit.>, an open-source dataset containing
fly-around videos of pets annotated with cameras and segmentation masks.
We select a subset of 50 dog videos with a variety of poses, movements and textures.
Each test video contains 200 frames, which we split to the training and test sets by considering contiguous blocks of 15 frames as train, interleaved by blocks of 5 frames as test (we further evaluate the impact of this protocol in <ref>).
We evaluate all models at 256$\times$256 resolution.

We also propose a tracking evaluation using APTv2 <cit.>, a comprehensive dataset that includes
videos featuring 30 distinct animal species in motion. Each video frame in this dataset is annotated with 17 keypoints, which we use as ground-truth for quantitative evaluation. All videos in the dataset have a consistent length of 15 frames. We restrict to a subset of 39 sequences, where the video contains a single-instance of a dog.

On COP3D <cit.>, we report three evaluation metrics assessing the quality of the predicted shape and texture:
(i) IoU reports the average over the intersection-over-union between the silhouette render of the posed shape $S_t$ and the ground truth mask $M_t$ computed for each frame $t$;
(ii) PSNR computes the average peak-signal-to-noise ratio between the renders $\bar{I}_t$ of the fitted shape at time $t$, and the ground truth image $I_t$; and
(iii) LPIPS <cit.>  is a perceptual metric that measures the average visual similarity between the rendered images $\bar{I}_t$ of the fitted shape at time $t$, and the ground truth images $I_t$. This metric is particularly useful as it takes into account human visual perception and the structural information of the images, providing a more accurate measure of visual similarity compared to PSNR. Additionally, we report worst 5% variant PSNRw5, IoUw5.
On APTv2 <cit.>, we evaluate $\mathbf{err_\text{track}}$ measuring the tracking error of the annotated keypoints via the following protocol:
(i) On the first frame, we pair each ground-truth keypoint $kp_{i}^{0}$ with a vertex $v_{i} \in \mathbb{R}^{3}$ on the mesh via the predicted pose.
(ii) On the remaining frames, we compute L2 distance between ground-truth keypoints and the projection of paired vertices.

First, we compare to template-based reconstructors BARC <cit.> and BITE <cit.> which, similar to us, predict parameters of the SMAL model, but take as input only a single image (frame).
Here, BITE extends BARC with a test-time iterative refinement.
Secondly we compare to RAC <cit.>, which is a template-free reconstructor with deformable shape and texture priors learned by observing various animals in videos.
For fair comparison, we execute their shape/texture training pipeline on the original training set extended with our CoP3D videos.

The quality of our textures can only be directly compared to RAC's model, which also includes texture, but we cannot compare them to BITE or BARC because they only output 3D shapes.

<Ref> contains the results of our experiments on COP3D and APTv2. On COP3D, our method outperforms BARC,BITE and RAC in the quality of the predicted texture (LPIPS). In terms of IoU, we outperform BARC and RAC, and achieve similar performance to BITE. However, since our method is temporally-consistent, we significantly outperform BITE on worst 5% IoU.
On APTv2 <cit.>, our method outperforms BARC, BITE and RAC on the tracking evaluation. We note that RAC achieves significantly worst than all the other methods. We argue that this is caused by the poor quality of the rendered shape.
We provide qualitative comparison in <ref> and videos for visual evaluation.

Texture transfer.
Models optimized on a scene can be re-animated via the template articulations. We demonstrate shape and motion projection from one optimized scene to other textures.

Ablation on CoP3D reporting performance with various loss terms removed and without camera motion factorization ($g_t^\text{cam}=g^0_t$).
w/o     $\mathcal{L}^\text{chamfer}$     $\mathcal{L}^\text{cse}$     $\mathcal{L}^\text{keypoint}$     $\mathcal{L}^\text{color}$     $\mathcal{L}^\text{arap}$     $\mathcal{L}^\text{edge}$
IoU $\uparrow$     0.70     0.81     0.80     0.81     0.81     0.83     0.72     0.84

PSNR $\uparrow$     20.65     20.89     21.54     21.62     21.61     21.88     19.12     22.40

LPIPS $\downarrow$     0.060     0.051     0.048     0.047     0.047     0.045     0.067    0.038

Ablation study.
To validate our design choices, we ablate individual components of our model and record the incurred changes in performance.
(i) Loss ablation.
In <ref>, we remove each loss term of Equation <ref> and report the resulting PSNR/LPIPS/IoU.
The results indicate a performance drop when any loss is removed, which confirms the merit of each loss.
(ii) Motion factorization.
We also demonstrate the benefits of our factorization of measured rigid motion to the motion of the camera and the motion of the shape (<ref>).
Specifically, we design an experiment where the rigid motion $g_t$ of each rendering camera $P_t$ is represented with the root rigid component $g^0_t$ of the SMAL deformation coefficients $\theta_t$, i.e., $\forall t \in [1..T] g_t^\text{cam}=g^0_t$, ignoring the SfM estimate $g_t^\text{SfM}$.
Results in <ref> indicate that this leads to a significant decrease in performance across all metrics justifying our rigid motion factorization.

Varying frame-split difficulty
For COP3D evaluations <cit.>, we follow the original train/test split protocol (contiguous blocks of 15 frames as train, interleaved by blocks of 5 frames as test).
Removing excessive visible frames would render the problem unsolvable (e.g. it is impossible to guess the exact motion of a dog's legs given two boundary frames that are too far apart). Regardless, in <ref> we evaluate the impact of reducing the supervision with two additional splits.
Our method outperforms RAC on all splits, thus showing stronger robustness to weaker supervision.

Evaluation with a varying train/test frame split on COP3D. Note that our method consistently beats RAC across a range of train/test splits.
Split Train/Test         IoU $\uparrow$     IoUw5 $\uparrow$     PSNR $\uparrow$     PSNRw5 $\uparrow$     LPIPS $\downarrow$
15/5     RAC <cit.>      0.76      0.52     21.86      17.51         0.164

Ours                                    0.84      0.79     22.12      19.40         0.041
15/10     RAC <cit.>     0.71     0.4      21.54     16.46     0.175

Ours                                       0.82     0.69     22.07     18.54     0.048
15/15     RAC <cit.>     0.66     0.37     20.67     15.38     0.196

Ours                                       0.81     0.63     21.62     17.81     0.056

§ CONCLUSIONS

In this paper we proposed a novel method for generating textured animatable 3D mesh models given a casually captured monocular video of a dog.
Our method augments the animatable SMAL mesh template with Continuous Surface Embeddings to setup a surface coordinate system which, in combination with a pretrained image-to-CSE predictor, allows to estimate accurate image-to-mesh correspondences that eventually lead to significantly more accurate fits.
The better shape fitting then enables us to optimize an implicit opacity-color texture supported by a scaffold defined by the mesh in its rest pose. Our experiments reveal performance superior to existing template-free and template-based approaches on the challenging CoP3D dataset.

– Supplementary material –

§ ADDITIONAL RESULTS

We include additional qualitative results in Figures <ref> and <ref>. We recommend looking at the video results to assess the quality of our reconstruction compared to other methods.
Notably, we observe that:
(1) Our reconstruction is consistent in time, which is not the case of BITE and BARC which reconstruct frames independently.
(2) The meshes reconstructed with RAC achieve relatively good score across all metrics, except LPIPS, yet they are qualitatively very poor, and they fall short in accurately representing the subject's true shape.

§ IMPLICIT DUPLEX-MESH RENDERING

We provide additional details for our implicit duplex-mesh model. During the rendering of $\bar{I} \in [0, 1]^{3 \times H \times W}$ from an arbitrary camera viewpoint $P$, we explained earlier that for each pixel $\bu \in [1..H] \times [1..W]$:

* We compute ${\bx}_\bu^\uparrow$, the first intersection between the ray in posed-space $\br_\bu$ and the posed outer boundary mesh $F(\hat{\mathcal{V}}^\uparrow, \beta, \theta)$. We repeat the same for the inner mesh $F(\hat{\mathcal{V}}^\uparrow, \beta, \theta)$ to get intersection point ${\bx}^\downarrow_\bu$.

* Since ${\bx}_\bu^\uparrow$ and ${\bx}^\downarrow_\bu$ are defined on the posed outer and inner mesh, we can compute their corresponding position $\hat{\bx}_\bu^\uparrow$ and $\hat{\bx}^\downarrow_\bu$ in the canonical space.

* We march in the canonical space with Emission-Absorption (EA) over the canonical ray $\hat{\br}_\bu$, defined between the two canonical points: $\hat{\bx}_\bu^\uparrow$ and $\hat{\bx}^\downarrow_\bu$

Trajectory sampling Given a ray in posed-space $\br_\bu$, the computation of (${\bx}_\bu^\uparrow$, ${\bx}^\downarrow_\bu$) can lead to 3 scenarios:

* $\br_\bu$ first intersects the outer and then the inner mesh. This is the standard scenario.

* $\br_\bu$ only intersect the outer, and not the inner mesh. In that case, we compute ${\bx}_\bu^\uparrow$ as usual. However, since ${\bx}^\downarrow_\bu$ cannot be defined on the inner mesh, we replace it with the second intersection point between $\br_\bu$ and  $F(\hat{\mathcal{V}}^\uparrow, \beta, \theta)$.

* $\br_\bu$ does not intersect with the inner or the outer mesh. Here, we do not render the pixel $\bu$.

Additionally, even if the inner and outer mesh do not overlap in the canonical space, they may intersect in some cases when they are posed. This can lead to unexpected scenarios where $\br_\bu$ first intersect the inner mesh and then the outer mesh, or only the inner mesh. We simply decide to discard these points from the rendering.

In the method section, we described how our approach simultaneously learns the shape $\beta$, motion $\theta_{t}$, and texture $\psi$ of a sequence. During the optimization process, the implicit duplex-mesh model is randomly initialized and progressively refined via the photometric loss $\mathcal{L}^\text{photo}_t$. This loss is also backpropagated to the pose predictor. We note that the quality of texture reconstruction is highly dependent on the quality of the pose reconstruction.

§ SHAPE REGULARIZERS

We include details about the different regularizers applied on the shape during optimization.

As-Rigid-As-Possible (ARAP) In the context of shape manipulation and deformation in computer graphics and geometric modeling, ARAP regularization has emerged as a pivotal technique. The main objective is to preserve the local rigidity of structures while allowing for global transformations, making it ideal for applications where maintaining the original characteristics of the shape is crucial. ARAP regularization achieves this by minimizing the deviation from a rigid transformation at a local level, effectively balancing between rigidity and flexibility.

In the specific context of our project, we observe that balancing the ARAP loss is very important to achieve a harmonious blend of realism and flexibility, and ensure that the dog's movements are both natural and within the bounds of plausible deformation. When the loss is set too high, it results in an overly rigid dog model, exhibiting minimal motion and lacking fluidity. This rigidity, while preserving local structure, inadvertently leads to an unnatural appearance in motion sequences. Conversely, setting the ARAP loss too low often lead to invalid deformations, where the mesh of the dog may contort or distort in unrealistic ways.

We follow implementation defined in <cit.>. Please refer to the original paper for the implementation details.

Edge regularization We include a regularization on the edge length between connected vertices of the mesh $\mathcal{V}$ after the mesh is posed.

The principle behind the edge length regularizer is to minimize the variation in the length of each edge during the deformation process (when the mesh is posed). For a given edge connecting vertices \( v_i \) and \( v_j \), the length of this edge can be denoted as \( l_{ij} \). The goal is to keep \( l_{ij} \) as close as possible to its original length in canonical pose \( l_{ij}^\text{canonical} \). Mathematically, this can be formulated as a loss:

\[
\mathcal{L}^\text{edge}_t = \sum_{(i, j) \in \text{Edges}(\mathcal{F})} \left( l_{ij} - l_{ij}^\text{canonical} \right)^2
\]

By tuning the weight of the edge-length regularizer, we observe the same behaviour as the ARAP loss.

§ INDEPENDENT DUPLEX-MESH RENDERER

We emphasize that our texture rendering module is independent from our pose prediction framework, and can be applied on fixed pose predictions $\{\theta_{t}\}$ by optimizing a photometric loss.
We propose to extend BARC and BITE method (which only predict pose) with our texture module, and show qualitative results in <Ref>. Our texture model achieves strong reconstruction even on BARC and BITE predictions. In practice, since BARC and BITE generate non-temporally consistent pose, we observe a drop in texture quality for scene with substantial motions from the camera or the subject.

Texture reconstruction from BARC BITE pose predictions. We optimize our duplex-mesh renderer model with fixed set of pose $\{\theta_{t}\}$ estimated by BARC and BITE on sequences from COP3D.

Additional qualitative comparison.

Additional qualitative comparison.