# Exotic Compact Objects and the Fate of the Light-Ring Instability

Pedro V. P. Cunha  Departamento de Matemática da Universidade de Aveiro and
Centre for Research and Development in Mathematics and Applications (CIDMA),
Campus de Santiago, 3810-183 Aveiro, Portugal  Carlos Herdeiro  Departamento
de Matemática da Universidade de Aveiro and Centre for Research and
Development in Mathematics and Applications (CIDMA), Campus de Santiago,
3810-183 Aveiro, Portugal  Eugen Radu  Departamento de Matemática da
Universidade de Aveiro and Centre for Research and Development in Mathematics
and Applications (CIDMA), Campus de Santiago, 3810-183 Aveiro, Portugal
Nicolas Sanchis-Gual  Departamento de Astronomía y Astrofísica, Universitat de
València, Dr. Moliner 50, 46100, Burjassot (València), Spain  Departamento de
Matemática da Universidade de Aveiro and Centre for Research and Development
in Mathematics and Applications (CIDMA), Campus de Santiago, 3810-183 Aveiro,
Portugal

(July 2022)

###### Abstract

Ultracompact objects with light-rings (LRs) but without an event horizon could
mimic black holes (BHs) in their strong gravity phenomenology. But are such
objects dynamically viable? Stationary and axisymmetric ultracompact objects
that can form from smooth, quasi-Minkowski initial data must have at least one
stable LR, which has been argued to trigger a spacetime instability; but its
development and fate have been unknown. Using fully non-linear numerical
evolutions of ultracompact bosonic stars free of any other known instabilities
and introducing a novel adiabatic effective potential technique, we confirm
the LRs triggered instability, identifying two possible fates: migration to
non-ultracompact configurations or collapse to BHs. In concrete examples we
show that typical migration/collapse time scales are not larger than $\sim
10^{3}$ light-crossing times, unless the stable LR potential well is very
shallow. Our results show that the LR instability is effective in destroying
horizonless ultracompact objects that could be plausible BH imitators.

Introduction. A few years after the first gravitational wave detection from a
collision of two black holes (BHs) Abbott  _et al._ (2016) and the first image
of a BH resolving its horizon scale structure Akiyama  _et al._ (2019), there
is a scientific consensus about the physical reality of BHs. Yet, both the
inability to observationally proof the “BH hypothesis” Berti  _et al._ (2015);
Barack  _et al._ (2019); Arun  _et al._ (2022); Herdeiro (2022) and its
challenging and far-reaching theoretical consequences Penrose (1965); Hawking
(1976); Almheiri  _et al._ (2013), demand a thorough scrutiny of its
alternatives.

In this spirit, a variety of horizonless exotic compact objects (ECOs) have
been proposed Cardoso and Pani (2019): the “ECO hypothesis”. Any putative ECO
model must overcome theoretical and observational tests to become a contender.
Of special interest are ultracompact ECOs (UCOs, for short), $i.e.$ possessing
light rings (LRs): planar bound photon orbits that asymptotically flat BHs
must possess Cunha and Herdeiro (2020). UCOs can imitate the (initial)
ringdown Cardoso  _et al._ (2016); Cardoso and Pani (2017) and (to some
extent) the shadow Cunha and Herdeiro (2018) of BHs making them plausible BH
foils if they are dynamically viable.

Conditions for dynamical viability include $(i)$ a plausible formation
mechanism, $(ii)$ sufficient stability against the ubiquitous astrophysical
perturbations and $(iii)$ embedding in a physically sound effective field
theory. It was shown in Cunha  _et al._ (2017a) that, under generic
assumptions, an equilibrium UCO that forms from smooth, quasi-Minkowski
initial data, must have at least a pair of LRs, one of which is stable. It has
been argued that the existence of stable LRs can trigger a spacetime
instability, by trapping massless perturbations that eventually pile up and
backreact on the spacetime Keir (2016); Cardoso  _et al._ (2014); Benomio
(2021). The development and fate of this hypothetical generic obstruction to
UCOs has, however, been so far unknown.

In this letter we present the development and fate of the LR instability in
concrete models, providing evidence that UCOs with a plausible formation
mechanism are destroyed in astrophysical timescales, therefore questioning
their viability as BH alternatives.

UCOs and stable LRs. Consider equilibrium, finite ADM mass $M$, asymptotically
flat, UCOs described by a stationary, axially-symmetric, circular metric,
$g_{\mu\nu}$. In $(t,r,\theta,\varphi)$ coordinates, such that $\partial_{t}$
and $\partial_{\varphi}$ are the commuting Killing vectors adapted to
stationarity and axisymmetry, respectively, (see Cunha  _et al._ (2017a);
Cunha and Herdeiro (2020) for details), the effective dimensionless potentials

$V_{\pm}=\frac{g_{t\varphi}\mp\sqrt{g_{t\varphi}^{2}-g_{tt}g_{\varphi\varphi}}}{g_{\varphi\varphi}}\,M\
,$ (1)

determine LRs (if they exist) as critical points: $\nabla V_{\pm}=0$. The
$\pm$ sign is connected to the LRs rotation sense.

One can associate a topological charge, $\mathcal{Q}_{i}$ to individual LRs
Cunha  _et al._ (2017a); Cunha and Herdeiro (2020). The total spacetime
topological charge $\mathcal{Q}=\sum_{LRs}\mathcal{Q}_{i}$ is then invariant
under smooth spacetime deformations preserving the asymptotic structure and
internal regularity. This means that any horizonless, asymptotically flat,
everywhere regular UCO has the same topological LR charge as Minkowski
spacetime, $\mathcal{Q}=0$. Since unstable (stable) LRs have
$\mathcal{Q}_{i}=-1$ ($\mathcal{Q}_{i}=+1$), this implies that in the
dynamical formation of UCOs LRs emerge as a stable-unstable pair. This
conclusion does not depend on the specific (metric) theory of gravity or on
the details of the (incomplete) gravitational collapse 111In particular, the
theorem in Cunha  _et al._ (2017a) does not require stationarity and axi-
symmetry during this incomplete collapse.. It implies that for UCOs with a
plausible formation mechanism, the existence of a (Schwarzschild-like)
unstable LR - as to imitate BH phenomenology - is accompanied by a potentially
dangerous stable LR.

Adiabatic effective potential. We wish to follow the dynamics of UCOs. It has
been argued that the instability triggered by stable LRs is non-linear Keir
(2016). Unveiling its final state, moreover, requires a non-linear analysis.
Thus, we shall resort to numerical evolutions of appropriate UCO models, using
a 3+1 spacetime split:

$\displaystyle ds^{2}$
$\displaystyle=-N^{2}\,dt^{2}+\gamma_{ij}\left(dx^{i}+\beta^{i}\,dt\right)\left(dx^{j}+\beta^{j}\,dt\right)\
,$ (2)

where $N$ is the lapse function, $\bm{\beta}$ is the shift, and $\bm{\gamma}$
is the projected 3-metric on the $t=$constant slice $\Sigma_{t}$, which uses
Cartesian-like coordinates $\\{x,y,z\\}$.

To monitor the evolution of LRs we introduce, from the numerical evolution 3+1
data in (2), an adiabatic effective potential (AEP) analog to (1) at each time
step, taking $M$ in (2) to be $M_{t}$, the UCO mass at each time slice $t$, to
allow comparisons at different $t$. The AEP carries physical information if
the evolution is sufficiently slow and departures for axial-symmetry are mild.
The lack of stationarity and axi-symmetry can, moreover, be washed away by
appropriate averaging procedures.

A detailed discussion of the assumptions in constructing the AEP is given in
Appendix A; here we state the key ones; $(i)$ At each $t$, approximate Killing
vectors $\partial_{t}$ and $\partial_{\varphi}$ exist for the evolving,
asymptotically flat and approximately circular UCO. This implies that at each
point a $\lambda\in\mathbb{R}$ exists such that
$\partial_{\varphi}=\lambda{\bm{\beta}}$. $(ii)$ $\partial_{\varphi}$ is
assumed to be tangent to surfaces with $x^{2}+y^{2}\equiv r^{2}=$ constant
(but $r$ needs not be a geometric distance). At each $t$ the coordinate center
$\mathcal{O}$: $(x,y)=(0,0)$ is linearly shifted to the UCO center, to account
for a possible (slow) drift of the star, which indeed occurs in the examples
below. $(iii)$ The evolving UCO is assumed to possess a $\mathbb{Z}_{2}$
reflection symmetry around the equatorial plane surface $z=0$.

Under these assumptions, the $(g_{tt},g_{t\varphi},g_{\varphi\varphi})$ data
necessary for the AEP (1) is obtained from the 3+1 data in (2),
$(N,\bm{\beta},\bm{\gamma})$ as follows (see Appendix A for details). $1)$
$g_{tt}=-N^{2}+\gamma_{ij}\beta^{i}\beta^{j}$; $2)$ $g_{\varphi\varphi}$
determines the perimeter $\mathcal{P}$ of a circumference $r=$constant,
$\mathcal{P}=2\pi\sqrt{g_{\varphi\varphi}(r)}=\int_{0}^{2\pi}\sqrt{\gamma_{xx}y^{2}-2\gamma_{xy}xy+\gamma_{yy}x^{2}}\,d\phi$,
where the auxiliary coordinate $\phi$ is defined via
$x=r\cos\phi,y=r\sin\phi$, and may differ from $\varphi$. Computing
$\mathcal{P}$ numerically via this integral, yields $g_{\varphi\varphi}$; 3)
since $g_{t\varphi}=g_{\varphi\varphi}\beta^{\varphi}$, then
$g_{t\varphi}=\pm\sqrt{g_{\varphi\varphi}}\,\sqrt{\gamma_{ij}\beta^{i}\beta^{j}}$.
The sign is positive (negative) if $\beta^{y}>0$ ($\beta^{y}<0$) when $y=0$.

Finally, two averaging procedures are used: $(1)$ to wash away (mild) axial-
symmetry deviations, we work with averaged functions over surfaces with
$r=$constant, $e.g.$
$\left<{N^{2}}\right>\equiv\frac{1}{2\pi}\int_{0}^{2\pi}N^{2}\,d\phi.$ Then,
$\left<g_{t\varphi}\right>\equiv\pm\sqrt{g_{\varphi\varphi}}\,\sqrt{\left<{\bm{\beta}}\cdot{\bm{\beta}}\right>}$,
and
$\left<g_{tt}\right>\equiv-\left<{N^{2}}\right>+\left<{\bm{\beta}}\cdot{\bm{\beta}}\right>$.
$(2)$ to wash away oscillations in the UCO’s time evolution around a trend, we
introduce time-averaged potentials $V^{*}_{\pm}$:
$V^{*}_{\pm}(r,t)=[2\,T(t)]^{-1}\int_{t-2\,T(t)}^{t}V_{\pm}(r,\tau)\,d\tau$,
where $T(t)$ is a measure of the oscillation period of the lapse $N(t)$, taken
to represent physical oscillations of the evolving UCO.

Testing UCO dynamics with bosonic stars (BSs). Testing the dynamical viability
of UCOs, requires models obeying conditions $(i)$ and $(iii)$ in the
Introduction, to then assess the impact of LRs on condition $(ii)$. The former
are obeyed in families of bosonic stars (BSs) Schunck and Mielke (2003); Brito
_et al._ (2016), which can form via gravitational cooling Seidel and Suen
(1994); Di Giovanni  _et al._ (2018) and emerge in simple and robust
scalar/vector effective field theories minimally coupled to Einstein’s gravity
222The non-self interacting vector model used here is free of known
pathologies Clough  _et al._ (2022); Coates and Ramazanoğlu (2022); Mou and
Zhang (2022)., where numerical evolutions are under control Liebling and
Palenzuela (2012). Additionally, our analysis requires UCO models free of
other instabilities ($e.g.$ perturbative or from ergo-regions), which would
mask the impact of LRs. This rules out the simplest, spherical scalar/vector
BSs, which only become UCOs in a perturbatively unstable regime Cunha  _et
al._ (2017b).

We have identified two appropriate models of BSs for studying the LR
instability, both described by the Lagrangian density, $\mathcal{L}=R/(16\pi
G)+\mathcal{L}_{m}$, where $R$, $G$ are the Ricci scalar and Newton’s constant
333We use geometrized units, $G=1=c$ here after.. Model 1 has
$\mathcal{L}_{m}=-\mathcal{F}_{\alpha\beta}\bar{\mathcal{F}}^{\alpha\beta}/4-\mu^{2}\mathcal{A}_{\alpha}\bar{\mathcal{A}}^{\alpha}/2$,
describes a complex Proca model with mass $\mu$ 444Overbar denotes complex
conjugation. and we focus on its fundamental, spinning, mini-Proca star
solutions Herdeiro and Radu (2017); Herdeiro  _et al._ (2019). They are
labelled by their ADM mass $M$ or their oscillation frequency $\omega$ and are
dynamically robust Sanchis-Gual  _et al._ (2019a). These Proca stars become
UCOs for $\omega/\mu\lesssim 0.711$; an ergoregion emerges for
$\omega/\mu\lesssim 0.602$ and perturbative instabilities are expected beyond
the maximal $M$, for $\omega/\mu\lesssim 0.562$ \- Fig. 1 (main panel). Model
2 has
$\mathcal{L}_{m}=-\partial_{\alpha}\Phi\partial^{\alpha}\bar{\Phi}-\mu^{2}|\Phi|^{2}\left[1-2|\Phi|^{2}/\sigma_{0}^{2}\right]^{2}$,
describes a self-interacting complex scalar model with mass $\mu$ and coupling
$\sigma_{0}$. We focus on its fundamental, spinning, solitonic boson star
solutions Kleihaus  _et al._ (2005); Siemonsen and East (2021), again labelled
by $M$ or $\omega$, choosing the illustrative value $\sigma_{0}=0.05$ Bezares
_et al._ (2017). There is a dynamically robust relativistic branch of
solutions for $\omega/\mu\lesssim 0.493$ Siemonsen and East (2021), that
become UCOs for $\omega/\mu\lesssim 0.188$; perturbative instabilities are
expected beyond the maximal M, for $\omega/\mu\lesssim 0.132$ and the ergo-
region appears in this unstable branch for $\omega/\mu\lesssim 0.134$ \- Fig.
1 (inset) 555In both models solutions with ergo-regions are UCOs, in agreement
with Ghosh and Sarkar (2021)..

Figure 1: Space of solutions for Model 1 (main panel) and Model 2 (inset).
Starting at $\omega/\mu=1$ and moving leftwards along the solutions curves,
one encounters the first solution with a LR (ergo-region) at the pink diamond
(blue star). The maximal mass BS, at which an unstable mode is expected to
appear (see $e.g.$ Siemonsen and East (2021)) is marked with a black pentagon.

The just described Proca (solitonic boson) stars in the range
$0.602<\omega/\mu<0.711$ ($0.132<\omega/\mu<0.188$) provide tests for the LR
instability (dashed line segments in Fig. 1). Using them as initial data, we
have performed fully non-linear numerical evolutions of the Einstein-bosonic
systems with spacetime variables in the BSSN formulation and the Einstein
Toolkit EinsteinToolkit ; Löffler (2012) evolved using McLachlan Brown  _et
al._ (2009); McLachlan  and Lean Witek and Zilhão  \- codes available in Witek
_et al._ (2021) and described in Zilhão  _et al._ (2015); Cunha  _et al._
(2017b); Sanchis-Gual  _et al._ (2019b).

Results Model 1: migration. We have evolved the aforementioned Proca stars up
to $t\mu=10^{4}$ and confirmed the existence of an instability for the UCOs.
Fig. 2 (main panel) shows the time at which the instability starts $vs.$
$\omega$. The timescale tends to diverge when approaching the first star with
a LR ($\omega/\mu\simeq 0.711$), thus associating the instability with UCOs.
For some of the models with LR we could not see the development of
instability, since the simulations last only up to $t\mu=10^{4}$ but our
results suggest that they will become unstable if evolved for longer. The
instability is not seen for stars with $\omega/\mu>0.711$.

Figure 2: Instability timescale for Model 1 (main panel) and Model 2 (inset).
The dashed vertical line marks the first UCO.

To grasp the nature of the instability, Fig. 3 shows snapshots of the time
evolution of the energy density on the equatorial plane for stars with
$\omega/\mu=0.68$, 0.69, 0.70, and angular momentum density for
$\omega/\mu=0.68$. The energy density first acquires an octogonal shape (see
second row of Fig. 3) which evolves into a “starfish-like” pattern (third
row). The axisymmetry of the star is mildly broken and the star suffers a
small kick, slowly moving away from the grid center. The instability triggers
a balanced mass and angular momentum loss, mostly carried by the Proca field,
with almost no gravitational wave emission. This balanced loss allows a
migration to less massive and compact but still spinning Proca stars without
LRs - see arrow in Fig. 1 and Appendix B for details.

$\ \ \ \ \ \omega/\mu=0.68$ |  $\ \ \ \ \omega/\mu=0.69$ |  $\omega/\mu=0.70$
---|---|---

Figure 3: Snapshots of the energy density (and angular momentum density in
middle left column) of three spinning Proca Star UCOs with $\omega/\mu=0.68$
(left and midlle left column), 0.69 (middle right column) and 0.70 (right
column). Time runs from top to bottom, given in code units.

Further clear evidence that the fate of this instability is a non-UCO spinning
Proca star is obtained by examining the averaged AEP $V^{*}_{-}$ 666For these
Proca stars the LRs emerge as critical points of $V_{-}$, rather than
$V_{+}$.. At each time slice, this potential has the typical radial profile
for an UCO displaying a stable and an unstable LR - Fig. 4 (top - inset) -
from which one can extract a potential depth $h$, defined as the (positive)
potential difference of $V^{*}_{-}$ computed at the LRs. $h$ can be
represented as a function of time in the evolutions. Fig. 4 (top) shows it for
$\omega/\mu=0.68$. It is clear that the depth $h(t)$ reaches zero at
$t_{c}\,\mu\simeq 7480$, $i.e.$ the two LRs have merged in a finite time. This
merger is further seen on the bottom panel of Fig. 4, where the perimetral
radius of both LRs converges to a single value. After this point the LRs
essentially disappear in terms of $V^{*}_{-}$. However, for a limited couple
of instances right after $t>t_{c}$ the LRs can be recreated and destroyed
again due to sporadic fluctuations of the potential (not shown in Fig. 4),
before disappearing altogether. This confirms the migration robustly evolves
in the direction of destroying the LR pair. In Appendix A, the time evolution
of the AEP is further detailed, legitimating the validity of the adiabatic
approximation underlying the AEP.

Figure 4: (Top) Potential-well depth $h$ (illustrated in inset) as a function
of time for $\omega/\mu=0.68$. A guiding fit function (dotted line) is defined
in terms of the inverse time $\tau=1/t$, with adjusted constants
$\\{a,A,\tau_{c}\\}$. (Bottom) The LRs’ circumferential radii
$\sqrt{g_{\varphi\varphi}}/M_{t}$ for different values of time $t$. The blue
line is an auxiliary spline interpolation to help convey the underlying
pattern. The time-averaged potential $V_{-}^{*}$ can suppress several, but not
all, oscillation modes of the raw potential $V_{-}$. This is the likely cause
of the observed zig-zagging of the red curve, a manifestation of residual star
radial oscillations around their unexcited state (at that time).

Results Model 2: collapse. We have also evolved the spinning solitonic scalar
boson stars described above. In the UCO region, $\omega/\mu\lesssim 0.188$, we
have again observed an instability, absent for $\omega>0.188$ (but within the
relativistic stable branch). Its timescale again tends to diverge, as we
approach the critical frequency - Fig. 2 (inset), correlating the instability
with the existence of LRs. The initial development of the instability
qualitatively resembles the previous case in the loss of axi-symmetry - see
Fig. 5 for evolution snapshots of the star with $\omega/\mu=0.16$. However,
the outcome is different; the star collapses and forms a spinning BH,
diagnosed by the emergence of an apparent horizon (detailed in Appendix B).
The culprit of this different outcome may be the self-interaction potential,
that confines the star and suppresses the dissipation through gravitational
cooling, the essential channel by which migration occurs in model 1. The AEP
shows violent oscillations and appears to deepen near the location of the
stable LR before the collapse, implying the technique looses validity.

Figure 5: Time evolution of a spinning solitonic BSs with $\omega/\mu=0.16$
which has LRs. (Left and middle left column) Energy density. Time runs from
top to bottom and then from left to right. (Middle right and right column)
Angular momentum density. Both quantities are extracted on the equatorial
plane.

Remarks. Proving the BH hypothesis is as challenging as disproving the ECO
hypothesis. A smaller, but informative step, is to rule out classes of
inadequate models as BH alternatives. The evidence shown in this letter
supports the inadequacy, as BH foils, of a large class of UCOs, for which a
plausible formation mechanism exists via an incomplete gravitational collapse
of quasi-Minkowski initial data. Even if such UCOs could form as a transient
state, their unavoidable stable LR triggers an instability that, generically,
develops in a moderate time scale, either promoting collapse to BHs or
migration to a non-UCO. It remains to be seen whether UCOs near the critical
configuration, for which the timescale can grow (likely) arbitrarily large,
still retain any effectiveness as BH foils, and if other fates are possible
for the LR instability, $e.g.$ considering different UCO models. It also
remains an open problem to find a practical estimate of the LR instability
timescale in a generic spacetime. One of our main results is a concrete
timescale within two specific models. The difficulty arises from the putative
non-linear character of the instability. Perhaps some lesson can be taken from
other gravitational non-linear instabilities, namely the Anti-de Sitter (AdS)
turbulent instability Bizon and Rostworowski (2011). In fact, the trapping of
modes in the stable LR potential well has some conceptual similarities with
the trapping of perturbations in the AdS “box”.

There are mechanisms to avoid the conclusion in Cunha  _et al._ (2017a) that
UCOs have stable LRs, $e.g.$: $(i)$ topological non-triviality; $e.g.$
ultracompact wormholes, since such UCOs are not diffeomorphic to Minkowski
spacetime. But any mechanism forming such macroscopic UCOs is beyond General
Relativity and remains to be established; $(ii)$ non-axisymmetry; but the
known astrophysical compact objects exhibit to a high degree axisymmetry;
$(iii)$ non-circularity; but very few (analytic or numerical) solutions
describing compact objects in Einstein’s or modified gravity without
circularity are known.

The ability of ECOs to imitate BH phenomenology does not demand LRs. Examples
are known, both for gravitational wave ($e.g.$ Calderón Bustillo  _et al._
(2021); Calderon Bustillo  _et al._ (2022)) and electromagnetic ($e.g.$
Herdeiro  _et al._ (2021)) observations, where ECOs without LRs could mimic,
in limited circumstances, some strong gravity BH features. It seems
implausible, however, that ECOs without LRs may mimic all BH phenomenology and
replace them entirely as physical players in the Universe. Under this
rationale, our results challenge the ECO hypothesis as a complete replacement
of the BH hypothesis, but not as another ingredient of the physical Universe.

Acknowledgements. This work was supported by the Spanish Agencia Estatal de
Investigación (grant PGC2018-095984-B-I00), by the Generalitat Valenciana
(PROMETEO/2019/071), by the Center for Research and Development in Mathematics
and Applications (CIDMA) through the Portuguese Foundation for Science and
Technology (FCT - Fundação para a Ciência e a Tecnologia), references
UIDB/04106/2020 and UIDP/04106/2020, by national funds (OE), through FCT,
I.P., in the scope of the framework contract foreseen in the numbers 4, 5 and
6 of the article 23, of the Decree-Law 57/2016, of August 29, changed by Law
57/2017, of July 19 and by the projects PTDC/FIS-OUT/28407/2017, CERN/FIS-
PAR/0027/2019, PTDC/FIS-AST/3041/2020 and CERN/FIS- PAR/0024/2021. This work
has further been supported by the European Union’s Horizon 2020 research and
innovation (RISE) programme H2020-MSCA-RISE-2017 Grant No. FunFiCO-777740 and
by FCT through Project No. UIDB/00099/2020. PC is supported by the Individual
CEEC program 2020 funded by the FCT. N.S.G. acknowledges financial support by
the Spanish Ministerio de Universidades, reference UP2021-044, within the
European Union-Next Generation EU. Computations have been performed at the
Servei d’Informàtica de la Universitat de València, and the Argus and Blafis
cluster at the U. Aveiro.

Appendix A. Details on the Adiabatic Effective Potential (AEP). We shall
consider a 3+1 spacetime metric satisfying the following assumptions:

[Adiabatic approximation] The spacetime at each time slice $t$ is
approximately stationary, and thus exists an approximate Killing vector
$\partial_{t}$ connected to the time coordinate $t$. Under this assumption,
the evolution is regarded as adiabatic.

[Axial-symmetry] The spacetime is approximately axially-symmetric, and thus
exists an approximate Killing vector $\partial_{\varphi}$ connected to this
assumption.

[Circularity & Asymptotic flatness] The spacetime satisfies approximate
circularity and asymptotic flatness. Both these assumptions imply the
spacetime admits an (approximate) 2-space orthogonal to
$\\{\partial_{t},\partial_{\varphi}\\}$ and possesses a discrete symmetry
$(t,\varphi)\to(-t,-\varphi)$. This also implies that each point exists
$\lambda\in\mathbb{R}$ such that $\partial_{\varphi}=\lambda{\bm{\beta}}$.

[Adaptation to axial-symmetry] The approximate Killing vector
$\partial_{\varphi}$ is assumed to be tangent to surfaces with
$x^{2}+y^{2}\equiv r^{2}=constant$. The coordinate center $\mathcal{O}$:
$(x,y)=(0,0)$ is linearly shifted at each time $t$ to the center of the
gravitational field distribution, using the lapse $N$ as a reference.

[Equatorial symmetry] The spacetime is assumed to possess a $\mathbb{Z}_{2}$
reflection symmetry on the equatorial plane surface $z=0$. Together with the
assumed axial-symmetry, metric data on this geometrically invariant surface
can be directly compared across different times. For this reason, and
considering that LRs are also typically found on the equatorial plane, the AEP
analysis in this work will be restricted to the surface $z=0$ (Equatorial
plane).

Given these assumptions, one can readily note that:

$\displaystyle
g_{tt}=-N^{2}+\gamma_{ij}\beta^{i}\beta^{j}\equiv-N^{2}+{\bm{\beta}}\cdot{\bm{\beta}}$
(3) $\displaystyle
g_{t\varphi}=\gamma_{\varphi\varphi}\beta^{\varphi}=g_{\varphi\varphi}\beta^{\varphi}=\beta_{\varphi}$
(4) $\displaystyle g_{r\varphi}=\partial_{r}\cdot\partial_{\varphi}=0$ (5)

The metric component $g_{tt}$ can be directly obtained from the 3+1 numerical
relativity data. In contrast, obtaining both $g_{t\varphi}$ and
$g_{\varphi\varphi}$ from the numerical relativity data, necessary to define
the AEP, is less straightforward. Critically, one can realise that the
perimeter $\mathcal{P}$ of a circumference of constant coordinate $r$ is
determined by $g_{\varphi\varphi}$. Writing the 2-metric on the equatorial
plane as

$\displaystyle ds_{(2)}^{2}$
$\displaystyle=g_{\varphi\varphi}d\varphi^{2}+g_{rr}dr^{2}$
$\displaystyle=\gamma_{xx}dx^{2}+2\gamma_{xy}dxdy+\gamma_{yy}dy^{2}\,,$

we can obtain the integral of the proper distance over a circle of constant
coordinate $r$:

$\displaystyle\mathcal{P}$
$\displaystyle=\oint_{r=\textrm{const.}}ds=2\pi\sqrt{g_{\varphi\varphi}(r)}$
$\displaystyle=\int_{0}^{2\pi}\sqrt{\gamma_{xx}y^{2}-2\gamma_{xy}xy+\gamma_{yy}x^{2}}\,d\phi\,.$

The last integral can be performed numerically using the 3+1 data, with
Cartesian-like coordinates $(x,y)$ parameterized as
$\left(x=r\cos\phi,\,y=r\sin\phi\right)$. The introduced auxiliary coordinate
$\phi$ might be different from $\varphi$. This procedure leads us to
$g_{\varphi\varphi}(r)={\mathcal{P}^{2}}/\left(4\pi^{2}\right)$. In addition
to $g_{\varphi\varphi}$, the computation of
$g_{t\varphi}=g_{\varphi\varphi}\beta^{\varphi}$ also requires knowledge of
$\beta^{\varphi}$. Taking the square of
$\bm{\beta}=\beta^{\varphi}\,\partial_{\varphi}$ yields:

$\bm{\beta}\cdot{\bm{\beta}}=\left[\beta^{\varphi}\right]^{2}\,g_{\varphi\varphi}\implies\beta^{\varphi}=\pm\sqrt{\frac{\bm{\beta}\cdot{\bm{\beta}}}{g_{\varphi\varphi}}}\,.$

This leads to the relation presented in the main text,
$g_{t\varphi}=\pm\sqrt{g_{\varphi\varphi}}\,\sqrt{{\bm{\beta}}\cdot{\bm{\beta}}}$.
The sign of $g_{t\varphi}$ is determined by the circulation sense of
$\bm{\beta}$ around the coordinate center, being positive (negative) if
$\beta^{y}>0$ ($\beta^{y}<0$) when $y=0$.

It is helpful to assess how much the spacetime symmetry assumptions are
violated. Indeed, one can notice that

$\displaystyle{\bm{\beta}}=\beta^{x}\,\partial_{x}+\beta^{y}\,\partial_{y}=\underbrace{\beta^{r}}_{=\,0}\partial_{r}+\beta^{\varphi}\,\partial_{\varphi}$
(6) $\displaystyle\bm{dr}=\frac{x\,\bm{dx}+y\,\bm{dy}}{r}$ (7)
$\displaystyle\beta^{r}=\bm{\beta}\bm{dr}=\frac{\beta^{x}\,x+y\,\beta^{y}}{r}=0\,.$
(8)

Figure 6: Show in red are the potentials $V_{-}(r,t)$ and its time-average
$V_{-}^{*}(r,t)$ for the dynamical evolution of the Proca Star with
$\omega=0.68\mu$, shown at different times $t$. The dashed-grey lines
represent the potential of the stationary Proca Star solutions with different
$\omega$. The dimensionless circumferential radius
$\sqrt{g_{\varphi\varphi}(r)}/M_{t}$ is used as a geometrical measure of the
radial distance to the star center.

For real numerical relativity data, the quantity $\beta^{r}$, is not
identically zero, and it can be used as a simple measure of the violation of
axi-symmetry. For instance, in the Proca Star case departures from symmetry
lead to maximum values of $\beta^{r}\sim 10^{-3}-10^{-2}$. However, these
departures from axial-symmetry can be diluted by working with averaged squared
functions over surfaces with $r=const.$:

$\left<\bm{\beta}\cdot\bm{\beta}\right>\equiv\frac{1}{2\pi}\int_{0}^{2\pi}\left(\bm{\beta}\cdot\bm{\beta}\right)\,d\phi,\qquad\left<{N^{2}}\right>\equiv\frac{1}{2\pi}\int_{0}^{2\pi}N^{2}\,d\phi\,.$

Using the latter we define an averaged version of the metric functions:

$\displaystyle\left<g_{t\varphi}\right>$
$\displaystyle\equiv\pm\sqrt{g_{\varphi\varphi}}\,\sqrt{\left<{\bm{\beta}}\cdot{\bm{\beta}}\right>}$
$\displaystyle\left<g_{tt}\right>$
$\displaystyle\equiv-\left<{N^{2}}\right>+\left<{\bm{\beta}}\cdot{\bm{\beta}}\right>$
$\displaystyle\left<g_{\varphi\varphi}\right>$ $\displaystyle\equiv
g_{\varphi\varphi}\,,$

with the AEP taking the explicit form:

$V_{\pm}(r,t)=M_{t}\,\frac{\left<g_{t\varphi}\right>\mp\sqrt{\left<g_{t\varphi}\right>^{2}-\left<g_{tt}\right>\left<g_{\varphi\varphi}\right>}}{\left<g_{\varphi\varphi}\right>}\,.$
(9)

This dimensionless potential can be defined for each time slice $t$, assuming
an adiabatic evolution of the spacetime. Profiles of $V_{\pm}$ taken at
different times can be compared directly, with possible mass loss during the
evolution already accounted for.

As a concrete spacetime example, we focus on Proca Star evolutions described
in the main text. Close to the end of their evolution, Proca Stars can be
considered in a perturbed excited state, with metric functions ($i.e.$ the
lapse) oscillating periodically around an average behaviour. Some long-term
patterns, such as the slow merger of LRs during the evolution, become much
clearer in terms of the time-averaged potential $V_{\pm}^{*}$, defined in the
main text, rather than the original potential $V_{\pm}(r,t)$. The fact that a
typical Proca Star oscillation period is around $\sim 100$ times larger than
the light-crossing time between the LRs further validates the adiabatic
approximation. For the Proca Stars considered here, the LRs appear only when
selecting the negative sign of the potential, $i.e.$ for $V^{*}_{-}$.

The evolution of the potential $V_{-}(r,t)$ and its time-average
$V_{-}^{*}(r,t)$ are represented in Fig. 6 for the Proca Star with
$\omega=0.68\mu$. We can notice that even for large $t$, the profile of the
dynamical potentials closely resemble those of stationary Proca Star solutions
with different $\omega$. This might suggest that dynamical Proca Stars closely
match the structure of different stationary solutions during their time
evolution, although this interpretation is likely a naive approximation.
Nevertheless, the consistency of the AEP profile with that of several
stationary Proca Stars, independently of the numerical relativity time
evolution, is a further indication on the validity of the AEP framework.

The bottom row of Fig. 6 displays three snapshots of the potentials during a
single oscillation of the star. The potential $V_{-}(r,t)$ possesses two LRs
at $t\mu=8293$, which are then destroyed as we cross $t\mu=8323$ and
$t\mu=8352$. In contrast, the time-averaged potential $V^{*}(r,t)$ remains
fairly unchanged in the course of a single oscillation of the star.
Ultimately, the purpose of $V_{-}^{*}$ is to represent the unexcited reference
state of the star at that time.

Appendix B. Loss of energy and angular momentum in the migration scenario. The
loss of Proca field during the instability is shown in Fig. 7 where we plot
the energy density in logscale on the equatorial plane and the $xz$ ($y=0$)
plane of the Proca star with $\omega/\mu=0.70$. The energy (and angular
momentum) densities are computed from Komar integrals as in Cunha  _et al._
(2017b); Sanchis-Gual  _et al._ (2019c). The Proca field is emitted mainly
along the equatorial plane, with some of it falling back onto the star,
yielding a breathing behaviour. Such loss is not present in the scalar
solitonic bosonic stars studied, and may play an important role in the
different outcomes. The real part of the scalar potential $\mathcal{X}_{\phi}$
of the Proca field (see Sanchis-Gual  _et al._ (2019c) for the definition of
this potential) is shown in the second column of Fig. 7; it illustrates how
the instability affects the star at the level of the Proca field (see the
panels in the third, fourth, and fifth rows). The “fleur-de-lys” pattern in
the Proca field is what configures the polynomial features in the energy
density. The bottom middle panel proves that the end-point of the instability
is still an $m=1$ spinning Proca star.

Figure 7: Time evolution of an ultracompact spinning Proca Star with
$\omega/\mu=0.70$: energy density in logscale (first column) and the real part
of the scalar potential (second column) both on the equatorial plane, and the
energy density in logscale in the $xz$ plane (third column). For comparison,
the energy density in logscale of a stable model with $\omega/\mu=0.72$
(fourth column).

Fig. 8 exhibits the time evolution of the mass and angular momentum of four
spinning Proca stars with $\omega/\mu=0.68$, 0.69, 0.70 (with LR), and 0.72
(without LR). The models with LR suffer an important loss of angular momentum
when the instability kicks in, but part of the angular momentum is accreted
back by the Proca star. After an initial more violent process of the
instability, the stars continue to lose angular momentum and mass at a slower
rate, through gravitational cooling, tending towards values corresponding to
configurations without LR. This is the part wherein we expect the AEP to be a
good physical description of the LRs’ evolution. On the other hand, as
expected, the model with $\omega/\mu=0.72$ does not show any instability. In
the latter a drift is seen in the conservation of the total energy and angular
momentum due to numerical error, but this drift goes away with resolution
Sanchis-Gual  _et al._ (2019a). The deviation at the end of the simulation
with respect to the initial values is less than 1%. In the bottom panels of
Fig. 7 we can see that the gravitational cooling is still at work and there is
a physical ejection of material, which is not present in the stable case.

Figure 8: Time evolution of the mass and angular momentum of four Proca star
models. For comparison, the critical Proca star, for which LRs first appear
has $\omega/\mu\simeq 0.711$, $M\mu\simeq 1.053$ (horizontal black solid line)
and $J\mu^{2}\simeq 1.149$ (horizontal black dashed line). It is clear that
the evolution of the ultracompact Proca stars tends to migrate them to non-
ultracompact stars.

Finally, in Fig. 9 we plot the BH formation after the collapse of the spinning
solitonic scalar boson star with $\omega/\mu=0.16$. The inset shows the
minimum value of the lapse function $N$ that approximately vanishes when the
apparent horizon forms.

Figure 9: Time evolution of the scalar field energy $E_{\varphi}$ and BH mass
$M_{\rm BH}$ for the ultracompact solitonic spinning scalar boson star with
$\omega/\mu=0.16$, which collapses into a BH.

## References

  * Abbott  _et al._ (2016) B. P. Abbott, R. Abbott, T. D. Abbott, M. R. Abernathy, F. Acernese, K. Ackley, C. Adams, T. Adams, P. Addesso, R. X. Adhikari, and et al., Physical Review Letters 116, 061102 (2016), arXiv:1602.03837 [gr-qc] .
  * Akiyama  _et al._ (2019) K. Akiyama _et al._ (Event Horizon Telescope), Astrophys. J. 875, L1 (2019).
  * Berti  _et al._ (2015) E. Berti _et al._ , Class. Quant. Grav. 32, 243001 (2015), arXiv:1501.07274 [gr-qc] .
  * Barack  _et al._ (2019) L. Barack _et al._ , Class. Quant. Grav. 36, 143001 (2019), arXiv:1806.05195 [gr-qc] .
  * Arun  _et al._ (2022) K. G. Arun _et al._ (LISA), Living Rev. Rel. 25, 4 (2022), arXiv:2205.01597 [gr-qc] .
  * Herdeiro (2022) C. A. R. Herdeiro, (2022), arXiv:2204.05640 [gr-qc] .
  * Penrose (1965) R. Penrose, Phys. Rev. Lett. 14, 57 (1965).
  * Hawking (1976) S. W. Hawking, Phys. Rev. D 14, 2460 (1976).
  * Almheiri  _et al._ (2013) A. Almheiri, D. Marolf, J. Polchinski, and J. Sully, JHEP 02, 062 (2013), arXiv:1207.3123 [hep-th] .
  * Cardoso and Pani (2019) V. Cardoso and P. Pani, Living Rev. Rel. 22, 4 (2019), arXiv:1904.05363 [gr-qc] .
  * Cunha and Herdeiro (2020) P. V. P. Cunha and C. A. R. Herdeiro, Phys. Rev. Lett. 124, 181101 (2020), arXiv:2003.06445 [gr-qc] .
  * Cardoso  _et al._ (2016) V. Cardoso, E. Franzin, and P. Pani, Phys. Rev. Lett. 116, 171101 (2016), [Erratum: Phys.Rev.Lett. 117, 089902 (2016)], arXiv:1602.07309 [gr-qc] .
  * Cardoso and Pani (2017) V. Cardoso and P. Pani, Nature Astron. 1, 586 (2017), arXiv:1709.01525 [gr-qc] .
  * Cunha and Herdeiro (2018) P. V. P. Cunha and C. A. R. Herdeiro, Gen. Rel. Grav. 50, 42 (2018), arXiv:1801.00860 [gr-qc] .
  * Cunha  _et al._ (2017a) P. V. P. Cunha, E. Berti, and C. A. R. Herdeiro, Phys. Rev. Lett. 119, 251102 (2017a), arXiv:1708.04211 [gr-qc] .
  * Keir (2016) J. Keir, Class. Quant. Grav. 33, 135009 (2016), arXiv:1404.7036 [gr-qc] .
  * Cardoso  _et al._ (2014) V. Cardoso, L. C. B. Crispino, C. F. B. Macedo, H. Okawa, and P. Pani, Phys. Rev. D 90, 044069 (2014), arXiv:1406.5510 [gr-qc] .
  * Benomio (2021) G. Benomio, Anal. Part. Diff. Eq. 14, 2427 (2021), arXiv:1809.07795 [gr-qc] .
  * Note (1) In particular, the theorem in Cunha  _et al._ (2017a) does not require stationarity and axi-symmetry during this incomplete collapse.
  * Schunck and Mielke (2003) F. E. Schunck and E. W. Mielke, Class. Quant. Grav. 20, R301 (2003), arXiv:0801.0307 [astro-ph] .
  * Brito  _et al._ (2016) R. Brito, V. Cardoso, C. A. R. Herdeiro, and E. Radu, Phys. Lett. B752, 291 (2016), arXiv:1508.05395 [gr-qc] .
  * Seidel and Suen (1994) E. Seidel and W.-M. Suen, Physical review letters 72, 2516 (1994).
  * Di Giovanni  _et al._ (2018) F. Di Giovanni, N. Sanchis-Gual, C. A. R. Herdeiro, and J. A. Font, Phys. Rev. D98, 064044 (2018), arXiv:1803.04802 [gr-qc] .
  * Note (2) The non-self interacting vector model used here is free of known pathologies Clough  _et al._ (2022); Coates and Ramazanoğlu (2022); Mou and Zhang (2022).
  * Liebling and Palenzuela (2012) S. L. Liebling and C. Palenzuela, Living Rev. Rel. 15, 6 (2012), arXiv:1202.5809 [gr-qc] .
  * Cunha  _et al._ (2017b) P. V. P. Cunha, J. A. Font, C. Herdeiro, E. Radu, N. Sanchis-Gual, and M. Zilhão, Phys. Rev. D96, 104040 (2017b), arXiv:1709.06118 [gr-qc] .
  * Note (3) We use geometrized units, $G=1=c$ here after.
  * Note (4) Overbar denotes complex conjugation.
  * Herdeiro and Radu (2017) C. A. R. Herdeiro and E. Radu, Phys. Rev. Lett. 119, 261101 (2017), arXiv:1706.06597 [gr-qc] .
  * Herdeiro  _et al._ (2019) C. Herdeiro, I. Perapechka, E. Radu, and Y. Shnir, Phys. Lett. B 797, 134845 (2019), arXiv:1906.05386 [gr-qc] .
  * Sanchis-Gual  _et al._ (2019a) N. Sanchis-Gual, F. Di Giovanni, M. Zilhão, C. Herdeiro, P. Cerdá-Durán, J. Font, and E. Radu, Physical Review Letters 123, 221101 (2019a).
  * Kleihaus  _et al._ (2005) B. Kleihaus, J. Kunz, and M. List, Phys. Rev. D 72, 064002 (2005), arXiv:gr-qc/0505143 .
  * Siemonsen and East (2021) N. Siemonsen and W. E. East, Phys. Rev. D 103, 044022 (2021), arXiv:2011.08247 [gr-qc] .
  * Bezares  _et al._ (2017) M. Bezares, C. Palenzuela, and C. Bona, Phys. Rev. D95, 124005 (2017), arXiv:1705.01071 [gr-qc] .
  * Note (5) In both models solutions with ergo-regions are UCOs, in agreement with Ghosh and Sarkar (2021).
  * (36) EinsteinToolkit, “Einstein Toolkit: Open software for relativistic astrophysics,” http://einsteintoolkit.org/.
  * Löffler (2012) F. Löffler, Classical Quantum Gravity 29, 115001 (2012).
  * Brown  _et al._ (2009) J. D. Brown, P. Diener, O. Sarbach, E. Schnetter, and M. Tiglio, Phys. Rev. D 79, 044023 (2009), arXiv:0809.3533 [gr-qc] .
  * (39) McLachlan, “McLachlan, a public BSSN code,” .
  * (40) H. Witek and M. Zilhão, “Canuda,” https://bitbucket.org/canuda/.
  * Witek  _et al._ (2021) H. Witek, M. Zilhao, G. Bozzola, M. Elley, G. Ficarra, T. Ikeda, N. Sanchis-Gual, and H. Silva,  (2021), 10.5281/zenodo.3565474.
  * Zilhão  _et al._ (2015) M. Zilhão, H. Witek, and V. Cardoso, Class. Quant. Grav. 32, 234003 (2015), arXiv:1505.00797 [gr-qc] .
  * Sanchis-Gual  _et al._ (2019b) N. Sanchis-Gual, C. Herdeiro, J. A. Font, E. Radu, and F. Di Giovanni, Phys. Rev. D 99, 024017 (2019b).
  * Note (6) For these Proca stars the LRs emerge as critical points of $V_{-}$, rather than $V_{+}$.
  * Bizon and Rostworowski (2011) P. Bizon and A. Rostworowski, Phys. Rev. Lett. 107, 031102 (2011), arXiv:1104.3702 [gr-qc] .
  * Calderón Bustillo  _et al._ (2021) J. Calderón Bustillo, N. Sanchis-Gual, A. Torres-Forné, J. A. Font, A. Vajpeyi, R. Smith, C. Herdeiro, E. Radu, and S. H. W. Leong, Phys. Rev. Lett. 126, 081101 (2021), arXiv:2009.05376 [gr-qc] .
  * Calderon Bustillo  _et al._ (2022) J. Calderon Bustillo, N. Sanchis-Gual, S. H. W. Leong, K. Chandra, A. Torres-Forne, J. A. Font, C. Herdeiro, E. Radu, I. C. F. Wong, and T. G. F. Li, (2022), arXiv:2206.02551 [gr-qc] .
  * Herdeiro  _et al._ (2021) C. A. R. Herdeiro, A. M. Pombo, E. Radu, P. V. P. Cunha, and N. Sanchis-Gual, (2021), arXiv:2102.01703 [gr-qc] .
  * Sanchis-Gual  _et al._ (2019c) N. Sanchis-Gual, C. Herdeiro, J. A. Font, E. Radu, and F. Di Giovanni, Phys. Rev. D 99, 024017 (2019c), arXiv:1806.07779 [gr-qc] .
  * Clough  _et al._ (2022) K. Clough, T. Helfer, H. Witek, and E. Berti, (2022), arXiv:2204.10868 [gr-qc] .
  * Coates and Ramazanoğlu (2022) A. Coates and F. M. Ramazanoğlu, (2022), arXiv:2205.07784 [gr-qc] .
  * Mou and Zhang (2022) Z.-G. Mou and H.-Y. Zhang, (2022), arXiv:2204.11324 [hep-th] .
  * Ghosh and Sarkar (2021) R. Ghosh and S. Sarkar, Phys. Rev. D 104, 044019 (2021), arXiv:2107.07370 [gr-qc] .


# FastFCN: Rethinking Dilated Convolution in the Backbone for Semantic
Segmentation

Huikai Wu, Junge Zhang, Kaiqi Huang
Institute of Automation, Chinese Academy of Sciences
{huikai.wu, jgzhang<EMAIL_ADDRESS>Kongming Liang, Yizhou Yu
Deepwise AI Lab
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Modern approaches for semantic segmentation usually employ dilated
convolutions in the backbone to extract high-resolution feature maps, which
brings heavy computation complexity and memory footprint. To replace the time
and memory consuming dilated convolutions, we propose a novel joint upsampling
module named Joint Pyramid Upsampling (JPU) by formulating the task of
extracting high-resolution feature maps into a joint upsampling problem. With
the proposed JPU, our method reduces the computation complexity by more than
three times without performance loss. Experiments show that JPU is superior to
other upsampling modules, which can be plugged into many existing approaches
to reduce computation complexity and improve performance. By replacing dilated
convolutions with the proposed JPU module, our method achieves the state-of-
the-art performance in Pascal Context dataset (mIoU of 53.13%) and ADE20K
dataset (final score of 0.5584) while running 3 times faster. Code is
available in https://github.com/wuhuikai/FastFCN.

##  1 Introduction

(a) FCN [22]

(b) EncoderDecoder

(c) DilatedFCN

Figure 1: Different types of networks for semantic segmentation. (a) is the
original FCN, (b) follows the encoder-decoder style, and (c) employs dilated
convolutions to obtain high-resolution final feature maps. Best viewed in
color.

Semantic segmentation [23, 40, 4] is one of the fundamental tasks in computer
vision, with the goal of assigning a semantic label to each pixel of an image.
Modern approaches usually employ a Fully Convolution Network (FCN) [22] to
address this task, achieving tremendous success among several segmentation
benchmarks.

The original FCN is proposed by Long _et al_. [22], which is transformed from
a Convolutional Neural Network (CNN) [16, 15] designed for image
classification. Inheriting from the design for image classification, the
original FCN downsamples the input image progressively by stride convolutions
and/or spatial pooling layers, resulting in a final feature map in low
resolution. Although the final feature map encodes rich semantic information,
the fine image structure information is lost, leading to inaccurate
predictions around the object boundaries. As shown in Figure 1(a), the
original FCN typically downsamples the input image 5 times, reducing the
spatial resolution of the final feature map by a factor of 32.

To obtain a high-resolution final feature map, [3, 28, 18, 30, 27] employ the
original FCN as the encoder to capture high-level semantic information, and a
decoder is designed to gradually recover the spatial information by combining
multi-level feature maps from the encoder. As shown in Figure 1(b), we term
such methods EncoderDecoder, of which the final prediction generated by the
decoder is in high resolution. Alternatively, DeepLab [5] removes the last two
downsampling operations from the original FCN and introduces dilated (atrous)
convolutions to maintain the receptive field of view unchanged.111In most
cases, dilated convolutions in this paper refer to (1) removing downsampling
operations and (2) replacing regular convolutions with dilated convolutions.
Following DeepLab, [38, 6, 36] employ a multi-scale context module on top of
the final feature map, outperforming most EncoderDecoder methods significantly
on several segmentation benchmarks. As shown in Figure 1(c), the spatial
resolution of the last feature map in DilatedFCN is 4 times larger than that
in the original FCN, thus maintaining more structure and location information.

The dilated convolutions play an important role in maintaining the spatial
resolution of the final feature map, leading to superior performance compared
to most methods in EncoderDecoder. However, the introduced dilated
convolutions bring heavy computation complexity and memory footprint, which
limit the usage in many real-time applications. Taking ResNet-101 [13] as an
example, compared to the original FCN, 23 residual blocks (69 convolution
layers) in DilatedFCN require to take 4 times more computation resources and
memory usages, and 3 residual blocks (9 convolution layers) need to take 16
times more resources.

We aim at tackling the aforementioned issue caused by dilated convolutions in
this paper. To achieve this, we propose a novel joint upsampling module to
replace the time and memory consuming dilated convolutions, namely Joint
Pyramid Upsampling (JPU). As a result, our method employs the original FCN as
the backbone while applying JPU to upsample the low-resolution final feature
map with output stride (OS) 32, resulting in a high-resolution feature map
(OS=8). Accordingly, the computation time and memory footprint of the whole
segmentation framework is dramatically reduced. Meanwhile, there’s no
performance loss when replacing the dilated convolutions with the proposed
JPU. We attribute this to the ability of JPU to exploit multi-scale context
across multi-level feature maps.

To validate the effectiveness of our method, we first conduct a systematical
experiment, showing that the proposed JPU can replace dilated convolutions in
several popular approaches without performance loss. We then test the proposed
method on several segmentation benchmarks. Results show that our method
achieves the state-of-the-art performance while running more than 3 times
faster. Concretely, we outperform all the baselines on Pascal Context dataset
[23] by a large margin, which achieves the state-of-the-art performance with
mIoU of 53.13%. On ADE20K dataset [40], we obtain the mIoU of 42.75% with
ResNet-50 as the backbone, which sets a new record on the val set. Moreover,
our method with ResNet-101 achieves the state-of-the-art performance in the
test set of ADE20K dataset.

In summary, our contributions are three folds, which are: (1) We propose a
computationally efficient joint upsampling module named JPU to replace the
time and memory consuming dilated convolutions in the backbone. (2) Based on
the proposed JPU, the computation time and memory footprint of the whole
segmentation framework can be reduced by a factor of more than 3 and meanwhile
achieves better performance. (3) Our method achieves the new state-of-the-art
performance in both Pascal Context dataset (mIoU of 53.13%) and ADE20K dataset
(mIoU of 42.75% with ResNet-50 as the backbone on the val set and final score
of 0.5584 with ResNet-101 on the test set).

Figure 2: Framework Overview of Our Method. Our method employs the same
backbone as the original FCN. After the backbone, a novel upsampling module
named Joint Pyramid Upsampling (JPU) is proposed, which takes the last three
feature maps as the inputs and generates a high-resolution feature map. A
multi-scale/global context module is then employed to produce the final label
map. Best viewed in color.

##  2 Related Work

In this section, we first give an overview on methods for semantic
segmentation, which can be categorized into two directions. We then introduce
some related works on upsampling.

###  2.1 Semantic Segmentation

FCNs [22] have achieved huge success in semantic segmentation. Following FCN,
there’re two prominent directions, namely DilatedFCN and EncoderDecoder.
DilatedFCNs [11, 34, 7, 6, 38, 36, 5] utilize dilated convolutions to keep the
receptive field of view and employ a multi-scale context module to process
high-level feature maps. Alternatively, EncoderDecoders [24, 28, 18, 1, 26,
12, 33, 37] propose to utilize an encoder to extract multi-level feature maps,
which are then combined into the final prediction by a decoder.

##### DilatedFCN

In order to capture multi-scale context information on the high-resolution
final feature map, PSPNet [38] performs pooling operations at multiple grid
scales while DeepLabV3 [6] employs parallel atrous convolutions with different
rates named ASPP. Alternatively, EncNet [36] utilizes the Context Encoding
Module to capture global contextual information. Differently, our method
proposes a joint upsampling module named JPU to replace the dilated
convolutions in the backbone of DilatedFCNs, which can reduce computation
complexity dramatically without performance loss.

##### EncoderDecoder

To gradually recover the spatial information, [28] introduces skip connections
to construct U-Net, which combines the encoder features and the corresponding
decoder activations. [18] proposes a multi-path refinement network, which
explicitly exploits all the information available along the down-sampling
process. DeepLabV3+ [8] combines the advantages of DilatedFCN and
EncoderDecoder, which employs DeepLabV3 as the encoder. Our method is
complementary to DeepLabV3+, which can reduce the computation overload of
DeepLabV3 without performance loss.

###  2.2 Upsampling

In our method, we propose a module to upsample a low-resolution feature map
given high-resolution feature maps as guidance, which is closely related to
joint upsampling as well as data-dependent upsampling.

##### Joint Upsampling

In the literature of image processing, joint upsampling aims at leveraging the
guidance image as a prior and transferring the structural details from the
guidance image to the target image. [17] constructs a joint filter based on
CNNs, which learns to recover the structure details in the guidance image.
[31] proposes an end-to-end trainable guided filtering module, which upsamples
a low-resolution image conditionally. Our method is related to the
aforementioned approaches. However, the proposed JPU is designed for
processing feature maps with a large number of channels while [17, 31] are
specially designed for processing 3-channel images, which fail to capture the
complex relations in high dimensional feature maps. Besides, the motivation
and target of our method is completely different.

##### Data-Dependent Upsampling

DUpsampling [29] is also related to our method, which takes advantages of the
redundancy in the segmentation label space and is able to recover the pixel-
wise prediction from low-resolution outputs of CNNs. Compared to our method,
DUpsampling has a strong dependency on the label space, which generalizes
poorly to a larger or more complex label space.

##  3 Method

In this section, we first introduce the most popular methods for semantic
segmentation, named DilatedFCNs. We then reform the architecture of
DilatedFCNs with a novel joint upsampling module, Joint Pyramid Upsampling
(JPU). Finally, we discuss the proposed JPU in details, before which joint
upsampling, dilated convolution and stride convolution are briefly introduced.

###  3.1 DilatedFCN

To exploit Deep CNNs in semantic segmentation, Long  _et al_. [22] transform
the CNN designed for image classification into FCN. Taking ResNet-101 as an
example, the original CNN contains 5 convolution stages, a global average
pooling layer and a linear layer. To construct an FCN, the global average
pooling layer and the linear layer are replaced by a convolution layer, which
is used to generate the final label map, as shown in Figure 1(a). Between each
two consecutive convolution stages, stride convolutions and/or spatial pooling
layers are employed, resulting in 5 feature maps with gradually reduced
spatial resolutions.

The spatial resolution of the last feature map in FCN is reduced by a factor
of 32, leading to inaccurate predictions about the locations and details. To
obtain a final feature map with high resolution, DeepLab [5] removes the
downsampling operations before the last two feature maps, as shown in Figure
1(c). Besides, the convolution layers inside the last two convolution stages
are replaced by dilated convolutions to maintain the receptive field of view,
thus named DilatedFCN. As a result, the resolution of the last feature map is
reduced by a factor of 8, which reserves more location and detail information.
Following DeepLab, [38, 6] propose a multi-scale context module to capture
context information from the last feature map, achieving tremendous success in
several segmentation benchmarks.

(a) Dilated Convolution in 1D (d=2)

(b) Stride Convolution in 1D (s=2)

Figure 3: Dilated Convolution (dilation rate=2) and Stride Convolution
(stride=2) in 1D. Best viewed in color.

###  3.2 The Framework of Our Method

To obtain a high-resolution final feature map, methods in DilatedFCN remove
the last two downsampling operations from the original FCN, which bring in
heavy computation complexity and memory footprint due to the enlarged feature
maps. In this paper, we aim at seeking an alternative way to approximate the
final feature map of DilatedFCN without computation and memory overload.
Meanwhile, we expect the performance of our method to be as good as that of
the original DilatedFCNs.

To achieve this, we first put back all the stride convolutions removed by
DilatedFCN, while replacing all the dilated convolutions with regular
convolution layers. As shown in Figure 2, the backbone of our method is the
same as that of the original FCN, where the spatial resolutions of the five
feature maps (Conv1$-$Conv5) are gradually reduced by a factor of 2. To obtain
a feature map similar to the final feature map of DilatedFCN, we propose a
novel module named Joint Pyramid Upsampling (JPU), which takes the last three
feature maps (Conv3$-$Conv5) as inputs. Then a multi-scale context module (PSP
[38]/ASPP [6]) or a global context module (Encoding [36]) is employed to
produce the final predictions.

Compared to DilatedFCN, our method takes 4 times fewer computation and memory
resources in 23 residual blocks (69 layers) and 16 times fewer in 3 blocks (9
layers) when the backbone is ResNet-101. Thus, our method runs much faster
than DilatedFCN while consuming less memory.

###  3.3 Joint Pyramid Upsampling

The proposed JPU is designed for generating a feature map that approximates
the activations of the final feature map from the backbone of DilatedFCN. Such
a problem can be reformulated into joint upsampling, which is then resolved by
a CNN designed for this task.

####  3.3.1 Background

##### Joint Upsampling

Given a low-resolution target image and a high-resolution guidance image,
joint upsampling aims at generating a high-resolution target image by
transferring details and structures from the guidance image. Generally, the
low-resolution target image $y_{l}$ is generated by employing a transformation
$f(\cdot)$ on the low-resolution guidance image $x_{l}$, _i.e_.
$y_{l}=f(x_{l})$. Given $x_{l}$ and $y_{l}$, we are required to obtain a
transformation $\hat{f}(\cdot)$ to approximate $f(\cdot)$, where the
computation complexity of $\hat{f}(\cdot)$ is much lower than $f(\cdot)$. For
example, if $f(\cdot)$ is a multi-layer perceptron (MLP), then
$\hat{f}(\cdot)$ can be simplified as a linear transformation. The high-
resolution target image $y_{h}$ is then obtained by applying $\hat{f}(\cdot)$
on the high-resolution guidance image $x_{h}$, _i.e_. $y_{h}=\hat{f}(x_{h})$.
Formally, given $x_{l}$, $y_{l}$ and $x_{h}$, joint upsampling is defined as
follows:

$y_{h}=\hat{f}(x_{h}),~{}\text{where}~{}\hat{f}(\cdot)=\operatorname*{argmin}_{h(\cdot)\in\mathcal{H}}||y_{l}-h(x_{l})||,$
(1)

where $\mathcal{H}$ is a set of all possible transformation functions, and
$||\cdot||$ is a pre-defined distance metric.

##### Dilated Convolution

Dilated convolution is introduced in DeepLab [5] for obtaining high-resolution
feature maps while maintaining the receptive field of view. Figure 3(a) gives
an illustration of the dilated convolution in 1D (dilation rate = 2), which
can be divided into the following three steps: (1) split the input feature
$f_{in}$ into two groups $f_{in}^{0}$ and $f_{in}^{1}$ according to the parity
of the index, (2) process each feature with the same convolution layer,
resulting in $f_{out}^{0}$ and $f_{out}^{1}$, and (3) merge the two generated
features interlaced to obtain the output feature $f_{out}$.

##### Stride Convolution

Stride convolution is proposed to transform the input feature into an output
feature with reduced spatial resolution, which is equivalent to the following
two steps as shown in Figure 3(b): (1) process the input feature $f_{in}$ with
a regular convolution to obtain the intermediate feature $f_{m}$, and (2)
remove the elements with an odd index, resulting in $f_{out}$.

####  3.3.2 Reformulating into Joint Upsampling

The differences between the backbone of our method and DilatedFCN lie on the
last two convolution stages. Taking the 4th convolution stage (Conv4) as an
example, in DilatedFCN, the input feature map is first processed by a regular
convolution layer, followed by a series of dilated convolutions (d=2).
Differently, our method first processes the input feature map with a stride
convolution (s=2), and then employs several regular convolutions to generate
the output.

Formally, given the input feature map $x$, the output feature map $y_{d}$ in
DilatedFCN is obtained as follows:

$\displaystyle y_{d}$ $\displaystyle=x\to
C_{r}\to\underbrace{C_{d}\to......\to C_{d}}_{n}$ (2) $\displaystyle=x\to
C_{r}\to\underbrace{SC_{r}M\to......\to
SC_{r}M}_{n}~{}\text{(Fig~{}\ref{fig:dilation:dilated_conv})}$
$\displaystyle=x\to C_{r}\to S\to\underbrace{C_{r}\to......\to C_{r}}_{n}\to
M$ $\displaystyle=y_{m}\to S\to C_{r}^{n}\to M$
$\displaystyle=\\{y_{m}^{0},y_{m}^{1}\\}\to C_{r}^{n}\to
M~{}\text{(Fig~{}\ref{fig:dilation:dilated_conv})},$

while in our method, the output feature map $y_{s}$ is generated as follows:

$\displaystyle y_{s}$ $\displaystyle=x\to
C_{s}\to\underbrace{C_{r}\to......\to C_{r}}_{n}$ (3) $\displaystyle=x\to
C_{r}\to R\to\underbrace{C_{r}\to......\to
C_{r}}_{n}~{}\text{(Fig~{}\ref{fig:dilation:stride_conv})}$
$\displaystyle=y_{m}\to R\to C_{r}^{n}=y_{m}^{0}\to
C_{r}^{n}~{}\text{(Fig~{}\ref{fig:dilation:stride_conv})}.$

$C_{r}$, $C_{d}$, and $C_{s}$ represent a regular/dilated/stride convolution
respectively, and $C_{r}^{n}$ is $n$ layers of regular convolutions. $S$, $M$
and $R$ are split, merge, and reduce operations in Figure 3, where adjacent
$S$ and $M$ operations can be canceled out. Notably, the convolutions in
Equations 2 and 3 are in 1D, which is for simplicity. Similar results can be
obtained for 2D convolutions.

The aforementioned equations show that $y_{s}$ and $y_{d}$ can be obtained
with the same function $C_{r}^{n}$ with different inputs: $y_{m}^{0}$ and
$y_{m}$, where the former is downsampled from the latter. Thus, given $x$ and
$y_{s}$, the feature map $y$ that approximates $y_{d}$ can be obtained as
follows:

$\displaystyle y=$ $\displaystyle\\{y_{m}^{0},y_{m}^{1}\\}\to\hat{h}\to M$ (4)
$\displaystyle\text{where}~{}\hat{h}=\operatorname*{argmin}_{h\in\mathcal{H}}||y_{s}-h(y_{m}^{0})||,$
$\displaystyle\quad\quad~{}~{}~{}y_{m}=x\to C_{r},$

which is the same as the joint upsampling problem defined in Equation 1.
Similar conclusions can be easily obtained for the 5th convolution stage
(Conv5).

####  3.3.3 Solving with CNNs

Equation 4 is an optimization problem, which takes lots of time to converge
through the iterative gradient descent. Alternatively, we propose to
approximate the optimization process with a CNN module. To achieve this, we
first require to generate $y_{m}$ given $x$, as shown in Equation 4. Then,
features from $y_{m}^{0}$ and $y_{s}$ need to be gathered for learning the
mapping $\hat{h}$. Finally, a convolution block is required to transform the
gathered features into the final prediction $y$.

Following the aforementioned analysis, we design the JPU module as in Figure
4. Concretely, each input feature map is firstly processed by a regular
convolution block (Fig. 4a), which is designed for (1) generating $y_{m}$
given $x$, and (2) transforming $f_{m}$ into an embedding space with reduced
dimensions. As a result, all the input features are mapped into the same
space, which enables a better fusion and reduces the computation complexity.

Then, the generated feature maps are upsampled and concatenated, resulting in
$y_{c}$ (Fig. 4b). Four separable convolutions [14, 9] with different dilation
rates (1, 2, 4, and 8) are employed in parallel to extract features from
$y_{c}$, where different dilation rates take different functions. Concretely,
the convolution with dilation rate 1 is employed to capture the relation
between $y_{m}^{0}$ and the rest part of $y_{m}$, as shown by the blue box in
Figure 5. Alternatively, the convolutions with dilation rate 2, 4 and 8 are
designed for learning the mapping $\hat{h}$ to transform $y_{m}^{0}$ into
$y_{s}$, as shown by the green boxes in Figure 5. Thus, JPU can extract multi-
scale context information from multi-level feature maps, which leads to a
better performance. This is significantly different from ASPP [6], which only
exploit the information in the last feature map.

The extracted features encode the mapping between $y_{m}^{0}$ and $y_{s}$ as
well as the relation between $y_{m}^{0}$ and the rest part of $y_{m}$. Thus,
another regular convolution block is employed, which transforms the features
into the final predictions (Fig. 4c).

Notably, the proposed JPU module solves two closely related joint upsampling
problems jointly, which are (1) upsampling Conv4 based on Conv3 (the 4th
convolution stage), and (2) upscaling Conv5 with the guidance of the enlarged
Conv4 (the 5th convolution stage).

Figure 4: The Proposed Joint Pyramid Upsampling (JPU). Best viewed in color.
Figure 5: The convolution with dilation rate 1 focuses on $y_{m}^{0}$ and the
rest part of $y_{m}$, and the convolution with dilation rate 2 aims at
$y_{m}^{0}$ and $y_{s}$. Best viewed in color.

##  4 Experiment

In this section, we first introduce the datasets used in our experiments as
well as the implementation details. We then conduct a systematic ablation
study to show the effectiveness of the proposed JPU from the view of both
performance and efficiency. Finally, to compare with the state-of-the-art
methods, we report the performance on two segmentation datasets, Pascal
Context [23] and ADE20K [40], which are widely used as the segmentation
benchmarks. Moreover, we also show some visual results to demonstrate the
superiority of our method.

###  4.1 Experimental Settings

##### Dataset

Pascal Context dataset [23] is based on the PASCAL VOC 2010 detection
challenge, which provides additional pixel-wise semantic annotations. There’re
4,998 images for training (train) and 5,105 images for testing (val).
Following the prior works [18, 5, 36], we use the most frequent 59 object
categories plus background (60 classes in total) as the semantic labels.

##### Implementation Details

Our method is implemented in PyTorch [25]. For training on Pascal Context, we
follow the protocol presented in [36]. Concretely, we set the learning rate to
0.001 initially, which gradually decreases to 0 by following the ”poly”
strategy (power = 0.9). For data augmentation, we randomly scale (from 0.5 to
2.0) and left-right flip the input images. The images are then cropped to
$480\times 480$ and grouped with batch size 16. The network is trained for 80
epochs with SGD, of which the momentum is set to 0.9 and weight decay is set
to 1e-4. All the experiments are conducted in a workstation with 4 Titan-Xp
GPUs (12G per GPU). We employ pixel-wise cross-entropy as the loss function.
ResNet-50 and ResNet-101 are used as the backbone, which are widely used in
most existing segmentation methods as the standard backbones.

###  4.2 Ablation Study

To show the effectiveness of the proposed method, we conduct a systematical
ablation study on Pascal Context dataset with ResNet-50 as the backbone, as
shown in Table 1. We report the standard evaluation metrics of pixel accuracy
(pixAcc) and mean Intersection of Union (mIoU). Notably, no multi-scale
testing and left-right flipping are applied to the val images.

##### Dilated Convolutions

For methods in DilatedFCN, the downsampling operations in the last two
convolution stages are removed, resulting in the output stride (OS) to be 8.
Encoding-8-None in Table 1 represents the original EncNet [36]. To show the
effect of dilated convolutions, we replace the backbone of EncNet with that of
the original FCN (the same as our method), resulting in the OS to be 32. We
then upsample the last feature map by 4 times with bilinear interpolation
before feeding it into the Encoding Head, noted as Encoding-32-Bilinear. As
shown in Table 1, Encoding-32-Bilinear performs significantly worse than
Encoding-8-None, which shows that it’s not trivial to replace the dilated
convolutions in the backbone of DilatedFCNs.

##### Upsampling Module

To show the effectiveness of the proposed JPU, we compare it with other
classic upsampling methods, bilinear upsampling and feature pyramid network
(FPN) [20]. As shown in Table 1, FPN outperforms bilinear interpolation by a
large margin. Even compared with EncNet, FPN achieves comparable performance
in both pixAcc and mIoU. By replacing FPN with our JPU, our method outperforms
both FPN and EncNet by more than 1% in mIoU, which achieves the state-of-the-
art performance.

The visual results are shown in Figure 6. Encoding-32-Bilinear (Fig. 6(c))
captures the global semantic information successfully, which gives a rough
segmentation of the bird and sky. However, the boundary of the bird is
inaccurate, and most parts of the branch are failed to be labeled out. When
replacing bilinear interpolation with FPN (Fig. 6(d)), the bird and branch are
labeled out successfully with accurate boundaries, which shows the effect of
combing low-level and high-level feature maps. A slightly better result can be
obtained with dilated convolutions (Fig. 6(e)). As for our method (Fig. 6(f)),
it labels out both the main branch and the side shoot accurately, which shows
the effectiveness of the proposed joint upsampling module. Particularly, the
side shoot demonstrates the ability of JPU to extract multi-scale context from
multi-level feature maps. Thus, our method can achieve a better performance.

##### Generalization to Other Methods

To show the generalization ability of the proposed JPU, we replace EncNet with
two popular methods in DilatedFCN, namely DeepLabV3 (ASPP Head) [6] and PSPNet
[38]. As shown in Table 1, our methods transformed from DeepLabV3 and PSP
outperforms the corresponding original methods consistently.

Head | OS | Upsampling | pixAcc% | mIoU%
---|---|---|---|---
Encoding [36] | 8 | None | 78.39 | 49.91
32 | Bilinear | 76.10 | 46.47
FPN [20] | 78.16 | 49.59
JPU (ours) | 78.98 | 51.05
ASPP [6] | 8 | None | 78.27 | 49.19
32 | JPU (ours) | 78.79 | 50.07
PSP [38] | 8 | None | 78.60 | 50.58
32 | JPU (ours) | 78.91 | 50.89
Table 1: Performance on the val set of Pascal Context dataset with the
ResNet-50 as the backbone.

(a) Input

(b) GT

(c) Bilinear

(d) FPN [20]

(e) EncNet [36]

(f) Ours

Figure 6: Visual comparison of different upsampling modules with Encoding Head and ResNet-50 as the backbone. Backbone | Head | Upsampling | FPS
---|---|---|---
ResNet-50 | Encoding [36] | None | 18.77
Bilinear | 45.67
FPN [20] | 37.87
JPU (ours) | 37.56
ASPP [6] | None | 15.99
JPU (ours) | 20.67
PSP [38] | None | 18.08
JPU (ours) | 28.48
ResNet-101 | Encoding [36] | None | 10.51
Bilinear | 35.20
FPN [20] | 32.40
JPU (ours) | 32.02
ASPP [6] | None | 10.46
JPU (ours) | 18.08
PSP [38] | None | 11.36
JPU (ours) | 23.87
Table 2: Comparison of Computation Complexity. The FPS is measured on a Titan-
Xp GPU with a $512\times 512$ image as input, which is averaged among 100
runs.

##### FPS

To compare the computation complexity, we employ frame per second (FPS) as the
evaluation metric, which is measured on a Titan-Xp GPU with a $512\times 512$
image as input. As shown in Table 2, the reported FPS is averaged among 100
runs. For ResNet-50, our method (Encoding-JPU) runs about two times faster
than EncNet (Encoding-None). When changing the backbone to ResNet-101, our
method runs more than three times faster than EncNet. The speed of our method
is also comparable to FPN, but our method achieves much better performance. As
for DeepLabV3 (ASPP) and PSP, our method can accelerate them to a certain
degree while having a better performance.

###  4.3 Comparison with Other Methods

Method | Backbone | mIoU%
---|---|---
FCN-8s [22] |  | 37.8
CRF-RNN [39] |  | 39.3
ParseNet [21] |  | 40.4
BoxSup [10] |  | 40.5
HO_CRF [2] |  | 41.3
Piecewise [19] |  | 43.3
VeryDeep [32] |  | 44.5
DeepLabV2 [5] | ResNet-101 + COCO | 45.7
RefineNet [18] | ResNet-152 | 47.3
EncNet [36] | ResNet-101 | 51.7
DUpsampling [29] | Xception-71 | 52.5
EncNet+JPU (ours) | ResNet-50 | 51.2222Following [36], the mIoU reported in Table 1 is on 59 classes w/o background. In this table, the mIoU is measured on 60 classes w/ background for a fair comparison with other methods. Besides, we average the network prediction in multiple scales for evaluation in this table.
EncNet+JPU (ours) | ResNet-101 | 53.1
Table 3: The state-of-the-art methods on the val set of the Pascal Context dataset. Method | Backbone | pixAcc% | mIoU%
---|---|---|---
FCN [22] |  | 71.32 | 29.39
SegNet [3] |  | 71.00 | 21.64
DilatedNet [35] |  | 73.55 | 32.31
CascadeNet [40] |  | 74.52 | 34.90
RefineNet [18] | ResNet-152 | - | 40.7
PSPNet [38] | ResNet-101 | 81.39 | 43.29
ResNet-269 | 81.69 | 44.94
EncNet [36] | ResNet-50 | 79.73 | 41.11
ResNet-101 | 81.69 | 44.65
Ours | ResNet-50 | 80.39 | 42.75
ResNet-101 | 80.99 | 44.34
Table 4: Results on the val set of ADE20K dataset. Rank | Team | Single Model | Final Score
---|---|---|---
1 | CASIA_IVA_JD | ✗ | 0.5547
2 | WinterIsComing | ✗ | 0.5544
- | PSPNet [38] | ResNet-269 | 0.5538
- | EncNet [36] | ResNet-101 | 0.5567
- | Ours | ResNet-101 | 0.5584
Table 5: Results on ADE20K test set. The first two entries ranked 1st and 2nd
place in COCO-Place challenge 2017.

##### Pascal Context

In Table 1, our method employs ResNet-50 as the backbone without multi-scale
evaluation, and the metrics are calculated on 59 classes excluding background
by following [36]. To compare fairly with the state-of-the-art methods, we
average the prediction in multiple scales and calculate the metrics among 60
classes including background, which are then reported in Table 3. With
ResNet-50 as the backbone, our method outperforms DeepLabV2 (with COCO
pretraining) and RefineNet by a large margin, which employ ResNet-101 and
ResNet-152 as the backbone, respectively. Moreover, our method (ResNet-50)
achieves competitive performance compared to EncNet with ResNet-101 as the
backbone. By replacing ResNet-50 with a deeper network ResNet-101, our method
gets an additional 1.9% improvement in mIoU, which outperforms EncNet
(ResNet-101) and DUpsampling (Xception-71) significantly and achieves the
state-of-the-art performance. Notably, Xception-71 is a much stronger backbone
than ResNet-101. For completeness, we also report the mIoU on 59 classes (w/o
background), which is 52.10%(ResNet-50) and 54.03% (ResNet-101).

##### ADE20K

ADE20K dataset [40] is a scene parsing benchmark, which contains 150
stuff/object categories. The dataset includes 20K/2K/3K images for training
(train), validation (val), and testing (test).

We train our network on the train set for 120 epochs with learning rate 0.01.
We then evaluate the model on the val set and report pixAcc and mIoU in Table
4. When employing ResNet-50 as the backbone, our method outperforms EncNet
(ResNet-50) by 1.64% in mIoU, while achieving a much better performance
compared to RefineNet (ResNet-152). By replacing ResNet-50 with ResNet-101,
our method obtains competitive performance compared to EncNet (ResNet-101) and
PSPNet (ResNet-269). Our method (ResNet-101) performs a little worse than
EncNet, and we attribute this to the spatial resolution of the training
images. Concretely, in our method, the training images are cropped to
$480\times 480$ for processing 4 images in a GPU with 12G memory. However,
EncNet is trained with $576\times 576$ images on GPUs with memory larger than
12G.

We then fine-tune our network on the train set and val set for another 20
epochs with learning rate 0.001. The predictions on the test set are submitted
to the evaluation server. As shown in Table 5, our method outperforms two
winning entries from the COCO-Place challenge 2017. Moreover, our method also
achieves better performance compared to PSPNet and EncNet, although it
performs worse on the val set. Notably, Final Score is the metric used in the
evaluation server, which is the average of pixAcc and mIoU.

The visual results from both the Pascal Context dataset and the ADE20K dataset
are shown in Figure 7. More results are shown in the supplementary material.

(a) Input

(b) GT

(c) EncNet [36]

(d) Ours

Figure 7: Visual results of our method (ResNet-101). The first row is from
Pascal Context val set, while the second row is from ADE20K val set. Best
viewed in color.

##  5 Conclusion

In this paper, we have analyzed the differences and connections between
dilated convolution and stride convolution. Based on the analysis, we
formulated the task of extracting high-resolution feature maps into a joint
upsampling problem and proposed a novel CNN module JPU to solve the problem.
By replacing the time and memory consuming dilated convolutions with our JPU,
the computation complexity is reduced by more than three times without
performance loss. The ablation study shows that the proposed JPU is superior
to other upsampling modules. By plugging JPU, several modern approaches for
semantic segmentation achieve a better performance while runs much faster than
before. Results on two segmentation datasets show that our method achieves the
state-of-the-art performance while reducing the computation complexity
dramatically.

## References

  * [1] M. Amirul Islam, M. Rochan, N. D. Bruce, and Y. Wang.  Gated feedback refinement network for dense image labeling.  In CVPR, 2017.
  * [2] A. Arnab, S. Jayasumana, S. Zheng, and P. H. Torr.  Higher order conditional random fields in deep neural networks.  In ECCV, 2016.
  * [3] V. Badrinarayanan, A. Kendall, and R. Cipolla.  Segnet: A deep convolutional encoder-decoder architecture for image segmentation.  arXiv, 2015.
  * [4] H. Caesar, J. Uijlings, and V. Ferrari.  Coco-stuff: Thing and stuff classes in context.  In CVPR, 2018.
  * [5] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille.  Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs.  TPAMI, 2018.
  * [6] L.-C. Chen, G. Papandreou, F. Schroff, and H. Adam.  Rethinking atrous convolution for semantic image segmentation.  arXiv, 2017.
  * [7] L.-C. Chen, Y. Yang, J. Wang, W. Xu, and A. L. Yuille.  Attention to scale: Scale-aware semantic image segmentation.  In CVPR, 2016.
  * [8] L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam.  Encoder-decoder with atrous separable convolution for semantic image segmentation.  In ECCV, 2018.
  * [9] F. Chollet.  Xception: Deep learning with depthwise separable convolutions.  In CVPR, 2017.
  * [10] J. Dai, K. He, and J. Sun.  Boxsup: Exploiting bounding boxes to supervise convolutional networks for semantic segmentation.  In ICCV, 2015.
  * [11] C. Farabet, C. Couprie, L. Najman, and Y. LeCun.  Learning hierarchical features for scene labeling.  TPAMI, 2013.
  * [12] J. Fu, J. Liu, Y. Wang, and H. Lu.  Stacked deconvolutional network for semantic segmentation.  arXiv, 2017.
  * [13] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  In CVPR, 2016.
  * [14] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam.  Mobilenets: Efficient convolutional neural networks for mobile vision applications.  arXiv, 2017.
  * [15] A. Krizhevsky, I. Sutskever, and G. E. Hinton.  Imagenet classification with deep convolutional neural networks.  In NIPS, 2012.
  * [16] Y. LeCun, L. Bottou, Y. Bengio, P. Haffner, et al.  Gradient-based learning applied to document recognition.  IEEE, 1998.
  * [17] Y. Li, J.-B. Huang, N. Ahuja, and M.-H. Yang.  Deep joint image filtering.  In ECCV, 2016.
  * [18] G. Lin, A. Milan, C. Shen, and I. D. Reid.  Refinenet: Multi-path refinement networks for high-resolution semantic segmentation.  In CVPR, 2017.
  * [19] G. Lin, C. Shen, A. Van Den Hengel, and I. Reid.  Efficient piecewise training of deep structured models for semantic segmentation.  In CVPR, 2016.
  * [20] T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie.  Feature pyramid networks for object detection.  In CVPR, 2017.
  * [21] W. Liu, A. Rabinovich, and A. C. Berg.  Parsenet: Looking wider to see better.  arXiv, 2015.
  * [22] J. Long, E. Shelhamer, and T. Darrell.  Fully convolutional networks for semantic segmentation.  In CVPR, 2015.
  * [23] R. Mottaghi, X. Chen, X. Liu, N.-G. Cho, S.-W. Lee, S. Fidler, R. Urtasun, and A. Yuille.  The role of context for object detection and semantic segmentation in the wild.  In CVPR, 2014.
  * [24] H. Noh, S. Hong, and B. Han.  Learning deconvolution network for semantic segmentation.  In ICCV, 2015.
  * [25] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer.  Automatic differentiation in pytorch.  In NIPS Workshop, 2017.
  * [26] C. Peng, X. Zhang, G. Yu, G. Luo, and J. Sun.  Large kernel matters—improve semantic segmentation by global convolutional network.  In CVPR, 2017.
  * [27] T. Pohlen, A. Hermans, M. Mathias, and B. Leibe.  Full-resolution residual networks for semantic segmentation in street scenes.  In CVPR, 2017.
  * [28] O. Ronneberger, P. Fischer, and T. Brox.  U-net: Convolutional networks for biomedical image segmentation.  In MICCAI, 2015.
  * [29] Z. Tian, C. Shen, T. He, and Y. Yan.  Decoders matter for semantic segmentation: Data-dependent decoding enables flexible feature aggregation.  arXiv, 2019.
  * [30] Z. Wojna, V. Ferrari, S. Guadarrama, N. Silberman, L.-C. Chen, A. Fathi, and J. Uijlings.  The devil is in the decoder.  arXiv, 2017.
  * [31] H. Wu, S. Zheng, J. Zhang, and K. Huang.  Fast end-to-end trainable guided filter.  In CVPR, 2018.
  * [32] Z. Wu, C. Shen, and A. v. d. Hengel.  Bridging category-level and instance-level semantic image segmentation.  arXiv, 2016.
  * [33] C. Yu, J. Wang, C. Peng, C. Gao, G. Yu, and N. Sang.  Learning a discriminative feature network for semantic segmentation.  In CVPR, 2018.
  * [34] F. Yu and V. Koltun.  Multi-scale context aggregation by dilated convolutions.  In ICLR, 2016.
  * [35] F. Yu and V. Koltun.  Multi-scale context aggregation by dilated convolutions.  arXiv, 2016.
  * [36] H. Zhang, K. Dana, J. Shi, Z. Zhang, X. Wang, A. Tyagi, and A. Agrawal.  Context encoding for semantic segmentation.  In CVPR, 2018.
  * [37] Z. Zhang, X. Zhang, C. Peng, D. Cheng, and J. Sun.  Exfuse: Enhancing feature fusion for semantic segmentation.  In ECCV, 2018.
  * [38] H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia.  Pyramid scene parsing network.  In CVPR, 2017.
  * [39] S. Zheng, S. Jayasumana, B. Romera-Paredes, V. Vineet, Z. Su, D. Du, C. Huang, and P. H. Torr.  Conditional random fields as recurrent neural networks.  In ICCV, 2015.
  * [40] B. Zhou, H. Zhao, X. Puig, S. Fidler, A. Barriuso, and A. Torralba.  Scene parsing through ade20k dataset.  In CVPR, 2017.

## More Visual Results

See Figure 8$-$12 in the following pages for more visual results.

(a) Input

(b) GT

(c) EncNet

(d) Ours

Figure 8: Visual results of our method (ResNet-101) on the Pascal Context val
set. Best viewed in color.

(a) Input

(b) GT

(c) EncNet

(d) Ours

Figure 9: Visual results of our method (ResNet-101) on the Pascal Context val
set. Best viewed in color.

(a) Input

(b) GT

(c) EncNet

(d) Ours

Figure 10: Visual results of our method (ResNet-101) on the Pascal Context val
set. Best viewed in color.

(a) Input

(b) GT

(c) EncNet

(d) Ours

Figure 11: Visual results of our method (ResNet-101) on the ADE20K val set.
Best viewed in color.

(a) Input

(b) GT

(c) EncNet

(d) Ours

Figure 12: Visual results of our method (ResNet-101) on the ADE20K val set.
Best viewed in color.