# Anisotropic Photon and Electron Scattering without Ultrarelativistic
Approximation

Anderson C.M. Lai  0000-0003-2741-4556<EMAIL_ADDRESS>Department of
Physics, The Chinese University of Hong Kong, Shatin, New Territories, Hong
Kong, China  Kenny C.Y. Ng  0000-0001-8016-2170<EMAIL_ADDRESS>Department
of Physics, The Chinese University of Hong Kong, Shatin, New Territories, Hong
Kong, China

(November 28, 2022)

###### Abstract

Interactions between photons and electrons are ubiquitous in astrophysics.
Photons can be down scattered (Compton scattering) or up scattered (inverse
Compton scattering) by moving electrons. Inverse Compton scattering, in
particular, is an essential process for the production of astrophysical gamma
rays. Computations of inverse Compton emission typically adopts an isotropic
or an ultrarelativistic assumption to simplify the calculation, which makes
them unable to broadcast the formula to the whole phase space of source
particles. In view of this, we develop a numerical scheme to compute the
interactions between anisotropic photons and electrons without taking
ultrarelativistic approximations. Compared to the ultrarelativistic limit, our
exact results show major deviations when target photons are down scattered or
when they possess energy comparable to source electrons. We also consider two
test cases of high-energy inverse Compton emission to validate our results in
the ultrarelativistic limit. In general, our formalism can be applied to cases
of anisotropic electron-photon scattering in various energy regimes, and for
computing the polarizations of the scattered photons.

##  I Introduction

Interactions between electrons and photons are responsible for a variety of
astrophysical phenomena. In particular, inverse Compton (IC) scattering—the up
scattering of low-energy photons by high-energy electrons—is one of the main
mechanisms for the production of astrophysical X-rays and gamma rays. For
example, cosmic-ray electrons can IC scatter with the interstellar radiation
field Hunter  _et al._ (1997); Moskalenko and Strong (1999); Strong  _et al._
(2000) and produce part of the Galactic diffuse gamma-ray emission Thompson
_et al._ (1997); Strong  _et al._ (2004a, b); Ackermann  _et al._ (2011).
Cosmic-ray electrons can interact with solar photons and produce a gamma-ray
halo around the Sun Orlando and Strong (2007); Moskalenko  _et al._ (2006);
Zhou  _et al._ (2017); Orlando and Strong (2008); Orlando  _et al._ (2008);
Abdo  _et al._ (2011); Ng  _et al._ (2016); Orlando and Strong (2021); Tang
_et al._ (2018); Linden  _et al._ (2022, 2018). Gamma rays can be produced in
astrophysical sources, such as pulsars, Blazars, and gamma-ray bursts, via
external IC emission or synchrotron self-Compton (SSC) emission Chiang and
Dermer (1999); Meszaros  _et al._ (1994); Yuksel  _et al._ (2009); Linden  _et
al._ (2017); Sudoh  _et al._ (2019). The up scattering of Cosmic Microwave
Background (CMB) radiation is also responsible for the Sunyaev–Zeldovich (SZ)
effect Sunyaev and Zeldovich (1970).

The IC emission formulation was studied in detail by Jones Jones (1968).
Analytic expressions were obtained by considering isotropic distributions of
electrons and photons for arbitrary electron energies. Jones also derived the
photon spectrum from IC scattering in the ultrarelativistic limit, which was
further developed by Blumenthal & Gould (BG70 Hereafter)BLUMENTHAL and GOULD
(1970) and Rybicki & Lightman Rybicki and Lightman (1986) by considering
electrons with a power-law energy spectrum. These results have found numerous
applications in high-energy astrophysics, such as the calculation of SSC
emission associated with relativistic jets Inoue  _et al._ (2019); Banik and
Bhadra (2019) and remnants from binary neutron star mergers Takami  _et al._
(2014), etc. While the ultrarelativistic results work well, the general
formalism by Jones suffers from numerical instability over a broad range of
photon and electron energies due to large number subtraction Belmont (2009).

Substantial efforts have been devoted to mitigating the numerical instability
in Jones’ expression, including reformulations of or corrections to Jones’
formula Nagirner and Poutanen (1994); Pe’er and Waxman (2005), as well as
interpolation from ultrarelativistic limit Coppi and Blandford (1990). These
improvements have been applied to numerical calculations of radiative
processes Pe’er  _et al._ (2006); Vurm and Poutanen (2009) and CMB spectral
distortions Sarkar  _et al._ (2019). Nevertheless, these works only considered
isotropic scattering, and usually focused on specific kinematic regimes and
target energy distributions. (E.g., ultrarelativistic electrons or thermal
photons/electrons.)

For the more general cases of anisotropic photon-electron scatterings,
Aharonian & Atoyan Aharonian and Atoyan (1981) derived the differential cross
section for the scattering between anisotropic photons and isotropic
electrons, and was applied in Chen and Bastian (2012); Murase  _et al._
(2010). These were also later rederived by Nagirner and Poutanen (1993) and
Brunetti (2000). In addition, Poutanen & Vurm Poutanen and Vurm (2010)
considered electrons in the weak anisotropic approximation scattering with
isotropic photons. Kelner et. al. Kelner  _et al._ (2014) studied anisotropic
electrons scattering with isotropic photons but took the ultrarelativistic
limit. Alternatively, a Monte Carlo approach was proposed to tackle the
anisotropic scattering Molnar and Birkinshaw (1999). The scattering between
anisotropic photon and isotropic electrons, in the ultrarelativistic limit,
was also considered by Moskalenko & Strong (MS hereafter) Moskalenko and
Strong (2000), which is often applied in computing the IC emission from
cosmic-ray interactions in the galaxy Strong  _et al._ (2000) and in the solar
system Moskalenko  _et al._ (2006); Orlando and Strong (2007). However, these
analytic expressions were sometimes found to be numerically unstable (Vurm and
Poutanen (2009); Belmont (2009)). A general formalism for anisotropic
electron-photon scatterings that is applicable in all kinematic regimes
remains unavailable.

In this paper, we present a numerical integration approach that solves the
general anisotropic IC scattering problem and calculates the resulting photon
spectrum. In section II, we describe the formalism on how we handle the
kinematic constraints arising from the differential cross section. In section
III, we validate our calculations using solar IC and SSC emission as test
cases. We show that our results are numerically stable and correctly converge
in the ultrarelativistic limit; we also discuss the cases where the exact
calculation deviate from the ultrarelativistic limit. We conclude and discuss
the outlook of this work in section V.

##  II Formulation

###  II.1 Master Equation for IC intensity

The master equation for line-of-sight (LOS) IC spectral intensity is given by:

$\displaystyle\frac{dI}{dE_{2}}(\Omega_{o},E_{2})=\frac{c}{4\pi}\int ds\int
n_{e}dE_{e}\int n_{ph}dE_{1}\left<\frac{d^{2}\sigma}{d\Omega dE_{2}}\right>,$
(1)

where $\Omega_{o}(\theta_{o},\phi_{o})$ is the LOS direction w.r.t the
observer, $s$ is the LOS distance, $E_{e}$, $E_{1}$, $E_{2}$ are the electron,
the target photon, and the scattered photon energies, respectively.
$n_{e}(E_{e})$ is the differential number density of source electron flux,
$n_{ph}(E_{1})$ is the differential number density of target photon field. The
angular averaged differential cross section is given by

$\displaystyle\left<\frac{d^{2}\sigma}{d\Omega dE_{2}}\right>=\int
d\Omega_{ph}f_{ph}\int d\Omega_{e}f_{e}\frac{d^{2}\sigma}{d\Omega
dE_{2}}(1-\beta\cos\theta_{e}),$ (2)

where $\Omega(\theta,\phi)$, $\Omega_{ph}(\theta_{ph},\phi_{ph})$, and
$\Omega_{e}(\theta_{e},\phi_{e})$ are the directions of the scattered photon,
target photons, and source electrons, respectively.
$f_{ph}(E_{ph},\Omega_{ph})$ and $f_{e}(E_{e},\Omega_{e})$ represent
normalized angular distributions of target photons and source electrons with
$\int fd\Omega=1$. $\beta$ is the speed of incident electrons and the factor
$(1-\beta\cos\theta_{e})$ comes in as the correction factor on electron
flux/photon density for non-parallel target photon and source electron. We
will use the simplified expression:

$\displaystyle\frac{d^{2}\sigma^{\prime}}{d\Omega
dE_{2}}=\frac{d^{2}\sigma}{d\Omega dE_{2}}(1-\beta\cos\theta_{e})$ (3)

hereafter. The definitions of the variables are schematically shown in Fig. 1.

Figure 1: A schematic diagram that depicts the IC scattering for the case of a
point source of target photons. $\vec{k}_{1}$, $\vec{k}_{2}$, and
$\vec{\beta}$ are the directions of target photon, scattered photon and source
electron. See the text for a detailed description of the variables. Note that
in general $\vec{k}_{1}$, $\vec{k}_{2}$, and $\vec{\beta}$ do not lie in the
same plane, see Fig. 2.

###  II.2 Assumptions

In general, the differential cross section Eq. 2 depends on many variables. To
simplify, we assume a unidirectional target photon along direction
$\Omega^{\prime}_{ph}$ from a point source, we also assume an isotropic
distribution of source electron flux. That is:

$\begin{split}f_{ph}&=\delta({\Omega_{ph}-\Omega^{\prime}_{ph}})\,,\\\
f_{e}&=\frac{1}{4\pi}\,.\end{split}$ (4)

Eq. 2 then becomes:

$\displaystyle\left<\frac{d^{2}\sigma}{d\Omega
dE_{2}}\right>=\frac{1}{4\pi}\int
d\Omega_{e}\frac{d^{2}\sigma^{\prime}}{d\Omega
dE_{2}}(E_{1},E_{2},E_{e},\Omega,\Omega_{e}),$ (5)

where $\Omega_{ph}$ is dropped by the delta function and $\Omega_{e}$ is now
defined w.r.t to the direction of target photon. The explicit form of the
differential cross section in Eq. 3 is given by Jauch and Rohrlich (2012):

$\displaystyle\frac{d^{2}\sigma^{\prime}}{d\Omega
dE_{2}}=\frac{r_{e}^{2}\bar{X}}{2\gamma^{2}(1-\beta\cos\theta_{e})}\frac{E_{2}^{2}}{E_{1}^{2}}\delta(E_{2}-\bar{E}_{2}),$
(6)

where $r_{e}$ is the classical electron radius, $\gamma$ is the electron
Lorentz factor and

$\displaystyle\bar{X}=\left[\frac{\kappa_{1}}{\kappa_{2}}+\frac{\kappa_{2}}{\kappa_{1}}+2m_{e}^{2}\left(\frac{1}{\kappa_{1}}-\frac{1}{\kappa_{2}}\right)+m_{e}^{4}\left(\frac{1}{\kappa_{1}}-\frac{1}{\kappa_{2}}\right)^{2}\right],$
(7)

where $\kappa_{1}=p\cdot k_{1}=p_{0}k_{0}-\vec{p}\cdot\vec{k}_{1}$ and
$\kappa_{2}=p\cdot k_{2}$ are the dot products of four momentum of incident
electron and incident/scattered photon, the subscript 0 denotes the time
component. $\bar{E}_{2}$ can be found from conservation of 4-momentum:

$\displaystyle\bar{E}_{2}=\frac{E_{1}(1-\beta\cos\theta_{e})}{\frac{E_{1}}{E_{e}}(1-\cos\theta)+1-\beta\cos\theta_{1}},$
(8)

where $\theta_{1}$ is the angle between source electron and scattered photon,

$\displaystyle\cos\theta_{1}=\cos\theta\cos\theta_{e}+\sin\theta\sin\theta_{e}\cos(\phi_{e}-\phi).$
(9)

We note that although Eq. 9 depends on 4 angular variables
$\phi_{e},\theta_{e},\phi,\theta$, the two polar angles only come in as their
difference $\phi_{e}-\phi$. This implies a polar symmetry on $\phi$ as we
integrate $\phi_{e}$. In other words, Eq. 6 does not depend on the $\phi$, and
we chose $\phi=0$ hereafter.

In the limit $\beta\ll 1$, Eq. 8 reduces to the Compton scattering formula and
$\kappa_{1},\kappa_{2}$ become $m_{e}E_{1}$ and $m_{e}E_{2}$. Putting these
expressions back in Eq. 6 yields the Klein-Nishina (KN) differential cross
section in electron-rest frame (ERF):

$\frac{d^{2}\sigma^{\prime}_{\text{KN}}}{d\Omega
dE_{2}}=\frac{r_{e}^{2}}{2}\frac{E_{2}^{2}}{E_{1}^{2}}\left(\frac{E_{2}}{E_{1}}+\frac{E_{1}}{E_{2}}-\sin^{2}\theta\right)\delta(E_{2}-\bar{E}_{2}).$
(10)

Figure 2: A diagram for the IC scattering between a photon and an electron.
With the assumptions made in Eq. 4, we can choose to align the target photon
direction with $+z-$axis and set $x-z$ plane as the scattering plane.

###  II.3 Reduction to MS result

As mentioned in the introduction, MS Moskalenko and Strong (2000) adopted an
ultrarelativistic assumption, in particular, $\gamma\gg 1$ and $\theta_{1}=0$.
The latter implies that scattered photons are unidirectional along the
direction of electron, or $\Omega=\Omega_{e}$. To illustrate the effect of
such approximation, we first rewrite the delta function in Eq. 6 as:

$\displaystyle\delta(E_{2}-\bar{E}_{2})=\frac{\delta(\Omega_{e}-\Omega_{e,sol})}{\left|\frac{d\bar{E}_{2}}{d\Omega_{e}}\right|},$
(11)

where $\Omega_{e,sol}$ is the solution to the condition of
$E_{2}=\bar{E}_{2}(\Omega_{e,sol})$. The unidirectional approximation is then
equivalent to:

$\displaystyle\delta(\Omega_{e}-\Omega_{e,sol})$
$\displaystyle=\delta(\phi_{e})~{}\delta(\cos\theta_{e}-\cos\theta)$ (12)

With this simplification, integration in Eq. 5 implies removing the
$\Omega_{e}$ dependence and evaluate the denominator in Eq. 11 at
$\Omega_{e}=\Omega$. By restoring a general source photon distribution
$f_{ph}$ and integrating over $\Omega_{ph}$, it can be shown that the whole
expression reduces to MS’s Eq. (8)Moskalenko and Strong (2000):

$\displaystyle\left<\frac{d\sigma}{dE_{2}}\right>=\frac{\pi
r_{e}^{2}}{E_{1}(\gamma-E_{2})^{2}}\int
d\Omega_{ph}f_{ph}\left[2-2\frac{E_{2}}{\gamma}\left(\frac{1}{E^{\prime}_{1}}+2\right)\right.$
$\displaystyle\left.+\frac{E_{2}^{2}}{\gamma^{2}}\left(\frac{1}{{E^{\prime}}_{1}^{2}}+2\frac{1}{E^{\prime}_{1}}+3\right)-\frac{E_{2}^{3}}{\gamma^{3}}\right].$
(13)

###  II.4 General treatment without unidirectional approximation

Instead of doing the unidirectional approximation in the last section, we look
for the exact solution of $\cos\theta_{e,sol}$ in terms of
$E_{1},E_{2},E_{e},\theta$ and $\phi_{e}$. To begin with, we write the
complete form of Eq. 5 using Eq. 6, 7 and 11:

$\displaystyle\left<\frac{d^{2}\sigma}{d\Omega dE_{2}}\right>$
$\displaystyle=$ $\displaystyle\frac{r^{2}_{e}}{8\pi\gamma^{2}}\int
d\Omega_{e}\frac{\bar{X}E^{2}_{2}}{(1-\beta\cos\theta_{e})E^{2}_{1}}\frac{\delta(\cos\theta_{e}-\cos\theta_{e,sol})}{\left|\frac{d\bar{E}_{2}}{d\cos\theta_{e}}\right|_{\cos\theta_{e,sol}}}$
$\displaystyle=$
$\displaystyle\frac{r^{2}_{e}}{8\pi\gamma^{2}}\int^{2\pi}_{0}d\phi_{e}\sum_{\cos\bar{\theta}_{e,sol}}\left[\frac{\bar{X}E^{2}_{2}}{(1-\beta\cos\theta_{e})E^{2}_{1}\left|\frac{d\bar{E}_{2}}{d\cos\theta_{e}}\right|}\right]\,,$

where

$\displaystyle\frac{d\bar{E}_{2}}{d\cos\theta_{e}}$ $\displaystyle=$
$\displaystyle\frac{\beta\bar{E}_{2}}{1-\beta\cos\theta_{e}}\left[\frac{\bar{E}_{2}}{E_{1}}(\cos\theta-\sin\theta\cos\phi_{e}\cot\theta_{e})-1\right].$

Figure 3: The differential cross sections $E_{2}^{2}\>d\sigma/dE_{2}$ against
scattered photon energy $E_{2}$ in the Thomson regime $E_{1}\ll E_{e}$ (Sec.
III.1.1). Left: target photon energy $E_{1}=10^{-6}$ MeV, source electron
Lorentz factor $\gamma=1.5$. BG70 refers to the Eq. (2.48) in BLUMENTHAL and
GOULD (1970), the blue line marks the value of $E_{1}$. See paragraph for more
details; Right: target photon energy $E_{1}=10^{-6}$ MeV, source electron
Lorentz factor $\gamma=100$.

The problem of finding the differential cross section then reduces to summing
and finding the possible solutions $\cos\theta_{e,sol}$, and then numerically
integrate over $\phi_{e}$. It can be achieved by putting Eq. 8 into a
quadratic equation of $\tan(\theta_{e}/2)$:

$\displaystyle(C-A)\tan^{2}\left(\frac{\theta_{e}}{2}\right)+2B\tan\left(\frac{\theta_{e}}{2}\right)+(C+A)=0,$
(16)

where $A,B$ and $C$ are given by:

$\displaystyle\begin{split}A&=\beta(E_{1}-E_{2}\cos\theta)\,,\\\
B&=-E_{2}\beta\sin\theta\cos\phi_{e}\,,\\\
C&=\frac{E_{2}E_{1}(1-\cos\theta)}{E_{e}}+E_{2}-E_{1}.\end{split}$ (17)

Hence, the two solutions to the equation

$\tan(\theta_{e}/2)=\frac{-B\pm\sqrt{A^{2}+B^{2}-C^{2}}}{C-A}$ (18)

correspond to two possible IC scattering geometry for a given set of
$(E_{2},E_{1},E_{e},\theta,\phi_{e})$. Although both solutions lead to
physical scattering geometries, only positive solutions are retained. That is
because a negative $\tan(\theta_{e}/2)$ can be mapped to a positive one
$\theta_{e}\rightarrow\theta_{e}+\pi$ together with
$\phi_{e}\rightarrow\phi_{e}+\pi$. Therefore, it represents a duplicated a
solution at another $\phi_{e}$. The relation between a positive determinant
$A^{2}+B^{2}-C^{2}$ and the physical limit of $E_{2}$ is explained in Appendix
A.

##  III Results

###  III.1 Scattering between isotropic photon and isotropic electrons

In this section, we compare the results obtained with our general formalism to
that from BG70 BLUMENTHAL and GOULD (1970), which was obtained using the
high-$\gamma$ approximation and is used frequently in the IC literature.

BG70 also assumed isotropic photon and electron distributions. To match that,
we evaluate the differential cross section by averaging over the scattering
angle of Eq. II.4. Although Eq. II.4 is derived from isotropic source electron
and unidirectional target photon (Eq. 4), averaging over the scattering angle
is equivalent to averaging over the incident photon directions, and thus
corresponds to an isotropic source photon distribution.

We note that in the context of thermal SZ effect, Sarkar et. al. Sarkar  _et
al._ (2019) also considered general isotropic scatterings. They defined the
kinematic regimes by comparing the source photon energy $E_{1}$ and the
electron energy $E_{e}$. This is different from us, as we consider mainly the
energies in the electron-rest frame (ERF). Below we refer to primed variables
as ERF quantities (e.g., $E_{1}^{\prime}$) and unprimed variables as the
observer frame quantities (e.g., $E_{1}$).

####  III.1.1 Thomson regime ($E_{1}^{\prime}\ll m_{e}$)

When the ERF target photon energy $E_{1}^{\prime}=\gamma
E_{1}(1-\beta\cos\theta_{e})$ is much smaller than the electron rest mass, the
scattering between target photons and electrons falls into the Thomson regime
regardless of the value of $\gamma$. In the Thomson limit, photon energies are
the same before and after scattering in the ERF, $E_{1}^{\prime}\approx
E_{2}^{\prime}$. In the ultrarelativistic limit ($\gamma\gg 1$), the maximum
cutoff of the IC scattered photon energy in the observer frame, $E_{2,\rm
max}$, is well approximated by $4\gamma^{2}E_{1}$, which is the limit used by
BG70.

Figure 4: Similar to Fig. 3, but for the KN regime, $E^{\prime}_{1}\gtrsim
m_{e}$, $E_{1}\lesssim E_{e}$ (Sec. III.1.2). Left: $E_{1}=0.5$ MeV and
$\gamma=1.5$, which corresponds to mildly-relativistic scattering; Right:
$E_{1}=100$ MeV and $\gamma=500$, which corresponds to ultrarelativistic
scattering.

Our exact formalism is expected to agree well with BG70 in the
ultrarelativistic limit. However, in the mildly-relativistic limit, when the
$\gamma$ value is smaller, the approximation $E_{2,\rm max}\sim
4\gamma^{2}E_{1}$ from head-on scattering between source electron and photon
no longer holds. In this case, the upscattered photon energy

$E_{2}\simeq\frac{1-\beta\cos\theta_{e}}{1-\beta\cos\theta_{1}}E_{1}$ (19)

can be obtained by taking the limit $E_{1}\ll E_{e}$ in Eq. 8 or by
considering the relation $E_{1}^{\prime}=E_{2}^{\prime}$ and express
$E_{1}^{\prime},E_{2}^{\prime}$ in observer frame quantities. The exact
$E_{2,\rm max}$ is then yielded by maximizing Eq. 19 with respect to
$\theta_{e}$ and $\theta_{1}$.

The left panel in Fig. 3 plots the differential cross section against $E_{2}$
for the mildly-relativistic case of $\gamma=1.5$ electrons scattering with
source photons at $10^{-6}$ MeV. The scattering falls into the Thomson regime
as $E_{1}^{\prime}\approx 3\times 10^{-6}$ MeV. The blue line marks the value
of $E_{1}$ and separates the spectrum into upscattering ($E_{1}<E_{2}$) and
downscattering ($E_{1}>E_{2}$) regions. In our exact calculation, the cutoff
energy is lower than BG70, and the differential cross section shifts to the
left. Our differential cross section also differs from BG70 in the down-
scattering region, which we discuss in detail later in Sec. III.1.4.

The right panel of Fig. 3 shows the same plot but with $\gamma=100$ for source
electrons to depict the ultrarelativistic IC scattering. The scattering is
still in the Thomson regime as $E_{1}^{\prime}\sim 2\times 10^{-3}$ MeV $\ll
m_{e}$. As expected, in the upscattering domain $E_{2}>E_{1}$, our result
agrees well with BG70, and have produced the same $E_{2,\rm
max}=4\gamma^{2}E_{1}\approx 4\times 10^{-2}$ MeV. In addition, the total
Thomson cross section $\sigma_{T}=8\pi/3~{}r_{e}^{2}$ can be recovered by
integrating the area under $d\sigma/dE_{2}$ in Fig. 3 over $E_{2}$. Therefore,
it validates our formalism on the differential cross section in the
ultrarelativistic limit.

####  III.1.2 KN regime ($E_{1}\lesssim E_{e}$, $E^{\prime}_{1}\gtrsim m_{e}$)

For larger values of $E_{1}$ that approaches $E_{e}$, the ERF target photon
energy $E_{1}^{\prime}$ can easily surpass the rest mass of the electron
$m_{e}$; this corresponds to the KN regime. In this regime, BG70 approximated
the maximum scattered photon energy $E_{2,\rm max}$ to be:

$E_{2,\rm max}\approx\frac{4\gamma^{2}E_{1}}{1+4\gamma\frac{E_{1}}{m_{e}}}\,,$
(20)

which follows from applying the large $\gamma$, head-on scattering with the
photon scattered backward approximation ($\theta_{e}=\pi,\theta_{1}=0$) to the
conservation of energy Eq. 8. When $E_{1}^{\prime}\gg m_{e}$, the term
$4\gamma E_{1}/m_{e}$ in the denominator of Eq. 20 dominates and $E_{2,\rm
max}\simeq E_{e}$. Using the general differential cross section, we instead
find that $E_{2,\rm max}=(\gamma-1)E_{e}+E_{1}$, which is simply the case when
the electron transfer all its kinetic energy to the photon and is valid for
any values of $\gamma$.

The left panel in Fig. 4 shows the differential cross section of $\gamma=1.5$
electrons scattering with 0.5 MeV target photons. As $E_{1}^{\prime}\approx
2\gamma E_{1}=1.5$ MeV, which implies that photons undergo Compton scattering
in ERF and a large portion of the energy is transferred to the electron. This
corresponds to the KN regime.

The right panel in Fig. 4 shows the ultrarelativistic scattering $\gamma=500$
electron and 100 MeV target photons, which is in the regime of
$E_{1}^{\prime}\approx 2\gamma E_{1}\gg m_{e}$. As expected, our results agree
with BG70 in the $E_{1}<E_{2}<E_{e}$ energy range. BG70’s formula, however,
does not work above $E_{e}$, while our results extend correctly to the true
maximum at $E_{2}=(\gamma-1)E_{e}+E_{1}$. We note that for large $\gamma$
cases, the differential cross section strongly peaks at the electron energy
$E_{e}$. This can be clearly seen in this plot (as well as in the right panel
of Fig. 5.)

Figure 5: Similar to Fig. 3, but for the trans-Compton regime, $E_{1}>E_{e}$
(Sec. III.1.3). Left: $E_{1}=1$ MeV, $\gamma=1.5$, corresponds to mildly-
relativistic scattering; Right: $E_{1}=100$ MeV, $\gamma=100$, corresponds to
ultrarelativistic scattering.

####  III.1.3 Trans-Compton regime ($E_{1}>E_{e}$)

When $E_{1}>E_{e}$, that is, when the target photon energy is greater than the
electron energy, the scattering in ERF entirely falls in the KN regime. But in
this case, the target photon mostly experiences energy loss, similar to the
case of Compton scattering. Although the case of $E_{1}>E_{e}$ was not
considered in BG70, we compare to their formula for completeness. Our
formalism works for all energy range up to $E_{2,\rm
max}=E_{1}+(\gamma-1)E_{e}$.

The left panel in Fig. 5 shows the differential cross section of $\gamma=1.5$
electrons scattering with 1 MeV target photons, which corresponds to a mildly-
relativistic scattering. The ultrarelativistic case ($\gamma=100$, $E_{1}=100$
MeV) is illustrated in the right panel of Fig. 5. The large deviation of
BG70’s differential cross section from ours implies the breakdown of
ultrarelativistic approximation in this kinematic regime.

####  III.1.4 Down-scattering cases ($E_{2}<E_{1}$)

In all the cases discussed above, our general formalism includes the effect of
down scattering (when $E_{2}<E_{1}$), which is not included in BG70. From Fig.
3 to Fig. 5, we note that the correct differential cross sections always
decline more rapidly than BG70 in the down scattering regime. In addition, we
also correctly calculate the minimum scattered photon energy $E_{2,\rm min}$
using Eq. 8, which corresponds to a photon and an electron initially
travelling in the same direction with the photon scattered backward
($\theta_{e}=0,~{}\theta_{1}=\pi$). In contrast, there is no $E_{2,\rm min}$
from BG70.

##  IV Case studies

We consider two simple cases of high-energy IC emission to show that our
formalism can correctly reproduce the results in ultrarelativistic limits, and
find the regime where the ultrarelativistic assumption would break down.

###  IV.1 Solar Inverse Compton Emission

Solar IC emission are produced when cosmic-ray electrons up scatter solar
photons Moskalenko  _et al._ (2006); Orlando  _et al._ (2008); Orlando and
Strong (2007, 2008). The anisotropic IC scattering cross section in MS
Moskalenko and Strong (2000) (with ultrarelativistic approximations) was
adopted to produce the IC emission package StellarICs Orlando and Strong
(2013). In this section, we compare our exactly formalism with the latest
StellarICs calculation on solar gamma ray in Ref. Orlando and Strong (2021).

Fig. 6 shows the LOS solar IC spectral intensity from the general electron-
photon scattering formalism and StellarICs. The computation of our spectrum
follows the master equation Eq. 1 with an observation angle
$\theta_{0}=0.3^{\circ}$. We employ the same $n_{ph},f_{ph},n_{e},f_{e}$ as in
StellarICs. Specifically, the differential number density of target photons
follows a black-body spectrum at 5770 K, with a spherical and uniform
distribution. The differential number density of source electrons is inferred
from the curve fitted with AMS02 data in the Fig. 3 of Orlando and Strong
(2021) and the distribution is isotropic. From Fig. 6, our spectrum agrees
well with StellarICs in the range of $E_{2}=10^{-2}-10^{3}$ MeV. The
correction from releasing the ultrarelativistic approximation in the
differential cross section cannot be seen in the figure, since electrons with
$\gamma=10^{2}-10^{4}$ are responsible for scattering solar photons to the
range of hard X-rays and gamma rays. The general formula thus reduces to
ultrarelativistic approximation in this regime and converge to StellarICs’s.

Figure 6: The LOS solar IC intensity at $0.3^{\circ}$ away from the centre of
the sun. The red spectrum labelled with ’StellarICs’ refers to the solid blue
curve in Orlando and Strong (2021) Fig. 4. The intensity spectrum in this work
was computed using Eq. 1 with the same solar photon and cosmic electron
spectrum in Aguilar  _et al._ (2014). In particular, a thermal spectrum from
$10^{-7}$ to $10^{-5}$ MeV was used for the photon field. An electron spectrum
from $10^{1}$ to 10 5 MeV that follows the curve fitted with AMS02 data in
Fig. 3 of Orlando and Strong (2021), was used. Figure 7: A decomposition of
LOS solar IC spectrum. The decomposed spectra with $\gamma=1-1.5$,
$\gamma=1.5-5$, $\gamma=5-10$, $\gamma=10-25$ and $\gamma=25-100$ are given by
the magenta, blue, green, cyan, and yellow curves, respectively. Solid lines
represent the StellarICs results and our results (general formalism) are shown
in dashed lines. Linear vertical scale is used to highlight the mildly-
relativistic correction.

For smaller values of $E_{2}$, we expect to see some discrepancies between our
exact calculation and the ultrarelativistic approximations. For illustration,
we extend the electron spectrum to lower energies, adopting a $E_{e}^{-3.2}$
power law between $\gamma=1$ to 100.

Fig. 7 shows the full energy range for the anisotropic solar IC scattering. It
is clear that the solar emission above $E_{2}\approx 10^{-3}$ MeV is indeed
dominated by larger $\gamma>10$. The ultrarelativistic approximation holds in
this regime and the two emission spectra coincide. When $E_{2}<10^{-3}$ MeV,
our calculation deviates from StellarICs. From the spectrum decomposition, we
see the deviation is due to mildly-relativistic correction from general
formalism. The deviation is the largest in the interval $\gamma=1.5-5$. The
overall correction to the total emission is a reduced intensity for
$E_{2}<5\times 10^{-3}$ MeV.

###  IV.2 Synchrotron Self-Compton

Synchrotron photons are emitted when energetic electrons gyrate along strong
magnetic fields. These synchrotron photons can also undergo IC scattering with
the gyrating electrons, resulting high-energy gamma-ray emission. This
synchrotron self-Compton (SSC) mechanism have been used to model gamma-ray
emissions from various sources, such as blazars and relativistic jets Potter
and Cotter (2012); Inoue  _et al._ (2019); Banik and Bhadra (2019).

In the SSC mechanism, the target photon energy is higher than the solar IC
case. We therefore consider a simplified SSC model to see the effect of
relaxing the ultrarelativistic assumption.

We consider a relativistic jet, where the Lorentz factor of the jet is 500
with a $0.5^{\circ}$ observation angle from the jet direction. The resulted
boost from the jet frame (JF) to the observer frame is about 50. Both the
electron and photon spectra are taken to be isotropic in JF so that we can
compare the result with BG70. In the JF (quantities denoted with the “JF”
subscript), we consider the electron spectrum to be a power law
$E_{e,\text{JF}}^{-3}$ between $\gamma_{\text{JF}}=1$ to $10^{4}$. The photon
spectrum is taken to be a $E_{1,\text{JF}}^{-2}$ power law between $10^{-3}$
MeV to $10^{3}$ MeV.

Fig. 8 shows the SSC emission spectrum $E_{2}^{2}d\epsilon/dE_{2}$ in the
observer frame with the above configuration. The red dashed line represents
the total emission from our calculation. For comparison, results obtained from
BG70 is shown in solid lines. Our results agrees well to that obtained with
BG70 at high photon energies, but large deviations start to appear at low
energy.

Figure 8: The SSC emission spectrum at observation angle $0.5^{\circ}$ away
from the jet with $E_{1,\text{JF}}$ from $10^{-3}$ to $10^{3}$ MeV and
$\gamma_{\text{JF}}$ from 1 to $10^{4}$. Both of target photons and electrons
are assumed to be isotropic in JF. The emission is plotted in scattered photon
energy in observer frame $E_{2}$. Red line gives the total emission. The
individual contributions from
$\gamma_{\text{JF}}=1-10^{1},10^{1}-10^{2},10^{2}-10^{3}$, and $10^{3}-10^{4}$
are presented in the black, blue, cyan, and yellow lines, respectively. The
transition points $P_{1}$ and $P_{2}$ for the blue line are highlighted. See
the text for details about the decomposition spectrum and transition points.

For $E_{2,\text{JF}}$ smaller than the minimum photon energy, $10^{-3}$ MeV
($E_{2}$ $\approx 0.05$ MeV, in Fig. 8), these are dominated by down-scattered
photons produced in the Thomson regime. In this case, BG70’s formula
corresponds to a flat differential cross section, and thus results in a
$E_{2}^{2}$ behaviour in the $E_{2}^{2}d\epsilon/dE_{2}$ plot. As discussed in
Sec. III.1.4, our results deviates considerably from the ultrarelativistic
approximation.

To further illustrate the physics of the IC scattering, consider the blue line
in Fig. 8, which is the spectrum produced by electrons with
$\gamma_{\text{JF}}$ between $10^{1}$ and $10^{2}$. For $E_{2}>0.05$ MeV
($E_{2,\text{JF}}>10^{-3}$), the spectrum enters a region where all scattering
geometries are accessible by Thomson scattering, thus resulting in the
$E_{2}^{2}$ behaviour. The $E_{2}^{2}$ trend continues until reaching the
point $P_{1}$, which corresponds to
$E_{2,\text{JF}}=4\gamma_{\text{JF,m}}^{2}E_{1,\text{JF,m}}$, where both
$\gamma_{\text{JF,m}}$ and $E_{1,\text{JF,m}}$ are the lower cutoff of the
source photon and electron spectrum. Above $P_{1}$, the spectrum then softens
mainly due to the spectral shape of source photons. Finally, above the point
$P_{2}$, the scattering transits into the KN regime, and the spectrum steepens
further mainly due to the spectral shape of source electrons. We note that the
transition into the KN regime can be rapid in the case of large $\gamma$
values. E.g., for the case of the the yellow line in Fig. 8.

Finally, as mentioned in Sec. III.1.2, our formalism works correctly for cases
with $E_{2}>E_{e,\rm{max}}$ when there are appreciable energy in the target
photon. These contribution appears as the ’bump’ features in Fig. 8. In these
cases, the extra contributions are overwhelmed by those from higher energy
electrons.

##  V Conclusions and Outlook

In this work, we consider the scattering between energetic electrons and
photons, and study the differential cross section for the outgoing photon
spectrum. This process is frequently used in high-energy astrophysics for the
production of IC photon emission.

We demonstrate that for the general case (without ultrarelativistic
approximation) of anisotropic photons scattering with isotropic electrons, the
differential cross section can be written as Eq. II.4, which can be easily
integrated numerically by finding the analytic solutions to the scattering
geometry, given by Eq. 18.

Focusing on isotropic photon and isotropic electron scatterings, we compare
our formalism with that from BG70, which considered the ultrarelativistic
approximation. We find that the scattering can be divided into three regimes,
the Thomson regime, the KN regime, and the Trans-Compton regime. In general,
we find that there would be deviations to the BG70 formula in the
downscattering limit ($E_{2}<E_{1}$), as well as when the target photon energy
becomes comparable to the electron energy.

To validate our formalism in the ultrarelativistic limit and show that it is
numerically stable, we consider two cases of high-energy IC emission. The
first is solar IC emission produced by anisotropic solar photon scattering
with isotropic cosmic-ray electrons. We find that our results agree well with
that from StellarICs Orlando and Strong (2021). Due to solar photons being
relatively low in energy, corrections from the exact formalism only appear
below keV photon energy. The second case we consider is SSC emission, where
isotropic synchrotron photons can scatter with isotropic electrons in
astrophysical jets. Due to the synchrotron photons being comparatively
energetic, the corrections from downscattering can be significant in the low
energy part of the SSC. At high energies, our results agree well with that
produced by the BG70 formula.

In these case studies, although we only consider scattering between isotropic
source electrons with isotropic photons (SSC) or anisotropic photons (solar
IC), it is straightforward to generalise the calculations to include different
angular distributions for photons and electrons in Eq. 2. For example, for the
case of external Compton emission in astrophysical jets Dermer (1995); Kelner
_et al._ (2014), photons emission are produced by jet electrons scattering
with photons from CMB, accretion disk, or dusty torus Finke (2016). In these
cases, both photon and electron distribution can be anisotropic, and thus a
more general formalism like ours is required.

In this work, we have focused much of our discussion in the gamma-ray regime,
where IC emission is typically considered. This is in part to show that our
formalism can be reduced to the well established works in the
ultrarelativistic regime. In general, we expect that a general formalism like
this is required whenever mildly relativistic electrons are involved, or when
the target photon energy is not small compared to electron energy.

Finally, through Eq. 18, we have obtained an exact solution to the scattering
angle geometry. This allows us to obtain the photon polarization caused by
anisotropic scattering, which we will discuss in detail in a followup work Lai
and Ng . In the literature, results of photon polarization caused by IC
scattering are somewhat inconsistent Krawczynski (2012). This work forms the
basis for a numerical framework to obtain the polarization spectrum. This is
especially timely, given that there are recent and future X-ray and gamma-ray
telescopes that are capable of detecting photon polarizations Weisskopf  _et
al._ (2016); Zhang  _et al._ (2016); Alessandro  _et al._ (2021).

##  VI Acknowledgement

We thank Ming-Chung Chu for helpful discussions. This work makes use of
StellarICs, an open source code that is available from Ref. Orlando and Strong
(2021). This project is supported by a grant from the Research Grant Council
of the Hong Kong Special Administrative Region, China (Project No. 24302721).

## References

  * Hunter  _et al._ (1997) S. D. Hunter _et al._ , “EGRET observations of the diffuse gamma-ray emission from the galactic plane,” Astrophys. J. 481, 205–240 (1997).
  * Moskalenko and Strong (1999) I. V. Moskalenko and A. W. Strong, “Puzzles of Galactic continuum gamma-rays,” Astrophysical Letters and Communications 38, 445–448 (1999), arXiv:astro-ph/9811221 [astro-ph] .
  * Strong  _et al._ (2000) Andrew W. Strong, Igor V. Moskalenko, and Olaf Reimer, “Diffuse Continuum Gamma Rays from the Galaxy,” ApJ 537, 763–784 (2000), arXiv:astro-ph/9811296 [astro-ph] .
  * Thompson  _et al._ (1997) D. J. Thompson, D. L. Bertsch, D. J. Morris, and R. Mukherjee, “Energetic gamma ray experiment telescope high-energy gamma ray observations of the Moon and quiet Sun,” J. Geophys. Res. 102, 14735–14740 (1997).
  * Strong  _et al._ (2004a) Andrew W. Strong, Igor V. Moskalenko, and Olaf Reimer, “Diffuse galactic continuum gamma rays: A model compatible with EGRET data and cosmic-ray measurements,” The Astrophysical Journal 613, 962–976 (2004a).
  * Strong  _et al._ (2004b) Andrew W. Strong, Igor V. Moskalenko, and Olaf Reimer, “A new determination of the extragalactic diffuse gamma-ray background from EGRET data,” The Astrophysical Journal 613, 956–961 (2004b).
  * Ackermann  _et al._ (2011) M. Ackermann _et al._ , “A Cocoon of Freshly Accelerated Cosmic Rays Detected by Fermi in the Cygnus Superbubble,” Science 334, 1103–1107 (2011).
  * Orlando and Strong (2007) E. Orlando and A. W. Strong, “Gamma rays from halos around stars and the sun,” Astrophysics and Space Science 309, 359–363 (2007).
  * Moskalenko  _et al._ (2006) Igor V. Moskalenko, Troy A. Porter, and Seth W. Digel, “Inverse Compton Scattering on Solar Photons, Heliospheric Modulation, and Neutrino Astrophysics,” ApJ 652, L65–L68 (2006), arXiv:astro-ph/0607521 [astro-ph] .
  * Zhou  _et al._ (2017) Bei Zhou, Kenny C. Y. Ng, John F. Beacom, and Annika H. G. Peter, “TeV solar gamma rays from cosmic-ray interactions,” Physical Review D 96 (2017), 10.1103/physrevd.96.023015.
  * Orlando and Strong (2008) E. Orlando and A. W. Strong, “Gamma-ray emission from the solar halo and disk: a study with EGRET data,” A&A 480, 847–857 (2008), arXiv:0801.2178 [astro-ph] .
  * Orlando  _et al._ (2008) E. Orlando, D. Petry, and A. W. Strong, “Extended inverse-Compton gamma-ray emission from the Sun seen by EGRET,” in  _International Cosmic Ray Conference_ , International Cosmic Ray Conference, Vol. 1 (2008) pp. 11–14.
  * Abdo  _et al._ (2011) A. A. Abdo _et al._ (Fermi-LAT), “Fermi-LAT Observations of Two Gamma-Ray Emission Components from the Quiescent Sun,” Astrophys. J. 734, 116 (2011), arXiv:1104.2093 [astro-ph.HE] .
  * Ng  _et al._ (2016) Kenny C. Y. Ng, John F. Beacom, Annika H. G. Peter, and Carsten Rott, “First observation of time variation in the solar-disk gamma-ray flux with fermi,” Physical Review D 94 (2016), 10.1103/physrevd.94.023004.
  * Orlando and Strong (2021) Elena Orlando and Andrew Strong, “StellarICS: inverse Compton emission from the quiet Sun and stars from keV to TeV,” J. Cosmology Astropart. Phys 2021, 004 (2021), arXiv:2012.13126 [astro-ph.HE] .
  * Tang  _et al._ (2018) Qing-Wen Tang, Kenny C. Y. Ng, Tim Linden, Bei Zhou, John F. Beacom, and Annika H. G. Peter, “Unexpected dip in the solar gamma-ray spectrum,” Phys. Rev. D 98, 063019 (2018), arXiv:1804.06846 [astro-ph.HE] .
  * Linden  _et al._ (2022) Tim Linden, John F. Beacom, Annika H. G. Peter, Benjamin J. Buckman, Bei Zhou, and Guanying Zhu, “First observations of solar disk gamma rays over a full solar cycle,” Phys. Rev. D 105, 063013 (2022), arXiv:2012.04654 [astro-ph.HE] .
  * Linden  _et al._ (2018) Tim Linden, Bei Zhou, John F. Beacom, Annika H. G. Peter, Kenny C. Y. Ng, and Qing-Wen Tang, “Evidence for a New Component of High-Energy Solar Gamma-Ray Production,” Phys. Rev. Lett. 121, 131103 (2018), arXiv:1803.05436 [astro-ph.HE] .
  * Chiang and Dermer (1999) J. Chiang and C. D. Dermer, “Synchrotron and ssc emission and the blast-wave model of gamma-ray bursts,” Astrophys. J. 512, 699 (1999), arXiv:astro-ph/9803339 .
  * Meszaros  _et al._ (1994) P. Meszaros, M. J. Rees, and H. Papathanassiou, “Spectral properties of blast wave models of gamma-ray burst sources,” Astrophys. J. 432, 181–193 (1994), arXiv:astro-ph/9311071 .
  * Yuksel  _et al._ (2009) Hasan Yuksel, Matthew D. Kistler, and Todor Stanev, “TeV Gamma Rays from Geminga and the Origin of the GeV Positron Excess,” Phys. Rev. Lett. 103, 051101 (2009), arXiv:0810.2784 [astro-ph] .
  * Linden  _et al._ (2017) Tim Linden, Katie Auchettl, Joseph Bramante, Ilias Cholis, Ke Fang, Dan Hooper, Tanvi Karwal, and Shirley Weishi Li, “Using HAWC to discover invisible pulsars,” Phys. Rev. D 96, 103016 (2017), arXiv:1703.09704 [astro-ph.HE] .
  * Sudoh  _et al._ (2019) Takahiro Sudoh, Tim Linden, and John F. Beacom, “TeV Halos are Everywhere: Prospects for New Discoveries,” Phys. Rev. D 100, 043016 (2019), arXiv:1902.08203 [astro-ph.HE] .
  * Sunyaev and Zeldovich (1970) R. A. Sunyaev and Ya. B. Zeldovich, “Small scale fluctuations of relic radiation,” Astrophys. Space Sci. 7, 3–19 (1970).
  * Jones (1968) F C Jones, “Calculated spectrum in inverse-compton-scattered photons.” Phys. Rev., 167: 1159-69(Mar. 25, 1968). (1968), 10.1103/PhysRev.167.1159.
  * BLUMENTHAL and GOULD (1970) GEORGE R. BLUMENTHAL and ROBERT J. GOULD, “Bremsstrahlung, synchrotron radiation, and compton scattering of high-energy electrons traversing dilute gases,” Rev. Mod. Phys. 42, 237–270 (1970).
  * Rybicki and Lightman (1986) George B. Rybicki and Alan P. Lightman,  _Radiative Processes in Astrophysics_ (1986).
  * Inoue  _et al._ (2019) Yoshiyuki Inoue, Dmitry Khangulyan, Susumu Inoue, and Akihiro Doi, “On high-energy particles in accretion disk coronae of supermassive black holes: implications for MeV gamma rays and high-energy neutrinos from AGN cores,”  (2019), 10.3847/1538-4357/ab2715, arXiv:1904.00554 [astro-ph.HE] .
  * Banik and Bhadra (2019) Prabir Banik and Arunava Bhadra, “Describing correlated observations of neutrinos and gamma-ray flares from the blazar TXS 0506+056 with a proton blazar model,” Phys. Rev. D 99, 103006 (2019), arXiv:1908.11849 [astro-ph.HE] .
  * Takami  _et al._ (2014) Hajime Takami, Koutarou Kyutoku, and Kunihito Ioka, “High-Energy Radiation from Remnants of Neutron Star Binary Mergers,” Phys. Rev. D 89, 063006 (2014), arXiv:1307.6805 [astro-ph.HE] .
  * Belmont (2009) R. Belmont, “Numerical computation of isotropic Compton scattering,” Astron. Astrophys. 506, 589 (2009), arXiv:0908.2705 [astro-ph.HE] .
  * Nagirner and Poutanen (1994) D. I. Nagirner and J. Poutanen,  _Single Compton scattering_ , Vol. 9 (1994).
  * Pe’er and Waxman (2005) Asaf Pe’er and Eli Waxman, “Time dependent numerical model for the emission of radiation from relativistic plasma,” Astrophys. J. 628, 857–866 (2005), arXiv:astro-ph/0409539 .
  * Coppi and Blandford (1990) P S Coppi and R D Blandford, “Reaction rates and energy distributions for elementary processes in relativistic pair plasmas,” Monthly Notices of the Royal Astronomical Society 245, 453–453 (1990), https://academic.oup.com/mnras/article-pdf/245/3/453/42466511/mnras0453.pdf .
  * Pe’er  _et al._ (2006) Asaf Pe’er, Peter Meszaros, and Martin J. Rees, “The observable effects of a photospheric component on grb’s and xrf’s prompt emission spectrum,” Astrophys. J. 642, 995–1003 (2006), arXiv:astro-ph/0510114 .
  * Vurm and Poutanen (2009) Indrek Vurm and Juri Poutanen, “Time-dependent modelling of radiative processes in hot magnetized plasmas,” Astrophys. J. 698, 293–316 (2009), arXiv:0807.2540 [astro-ph] .
  * Sarkar  _et al._ (2019) Abir Sarkar, Jens Chluba, and Elizabeth Lee, “Dissecting the Compton scattering kernel I: Isotropic media,” Mon. Not. Roy. Astron. Soc. 490, 3705–3726 (2019), arXiv:1905.00868 [astro-ph.CO] .
  * Aharonian and Atoyan (1981) F. A. Aharonian and A. M. Atoyan, “Compton Scattering of Relativistic Electrons in Compact X-Ray Sources,” Ap&SS 79, 321–336 (1981).
  * Chen and Bastian (2012) Bin Chen and Timothy S. Bastian, “The Role of Inverse Compton Scattering in Solar Coronal Hard X-ray and Gamma-ray Sources,” Astrophys. J. 750, 35 (2012), arXiv:1108.0131 [astro-ph.SR] .
  * Murase  _et al._ (2010) Kohta Murase, Kenji Toma, Ryo Yamazaki, Shigehiro Nagataki, and Kunihito Ioka, “High-energy emission as a test of the prior emission model for gamma-ray burst afterglows,” MNRAS 402, L54–L58 (2010), arXiv:0910.0232 [astro-ph.HE] .
  * Nagirner and Poutanen (1993) D. I. Nagirner and Yu. J. Poutanen, “Compton scattering by Maxwellian electrons - Redistribution of radiation according to frequencies and directions,” Astronomy Letters 19, 262–267 (1993).
  * Brunetti (2000) G. Brunetti, “Anisotropic inverse Compton scattering from the trans-relativistic to the ultrarelativistic regime and application to the radio galaxies,” Astropart. Phys. 13, 107–125 (2000), arXiv:astro-ph/9908236 .
  * Poutanen and Vurm (2010) Juri Poutanen and Indrek Vurm, “Theory of Compton scattering by anisotropic electrons,” Astrophys. J. Suppl. 189, 286–308 (2010), arXiv:1006.2397 [astro-ph.HE] .
  * Kelner  _et al._ (2014) S. R. Kelner, E. Lefa, F. M. Rieger, and F. A. Aharonian, “The Beaming Pattern of External Compton Emission from Relativistic Outflows: The Case of Anisotropic Distribution of Electrons,” ApJ 785, 141 (2014), arXiv:1308.5157 [astro-ph.HE] .
  * Molnar and Birkinshaw (1999) S. M. Molnar and M. Birkinshaw, “Inverse Compton scattering in mildly relativistic plasma,” Astrophys. J. 523, 78 (1999), arXiv:astro-ph/9903444 .
  * Moskalenko and Strong (2000) Igor V. Moskalenko and Andrew W. Strong, “Anisotropic inverse compton scattering in the galaxy,” The Astrophysical Journal 528, 357–367 (2000).
  * Jauch and Rohrlich (2012) J.M. Jauch and F. Rohrlich,  _The Theory of Photons and Electrons: The Relativistic Quantum Field Theory of Charged Particles with Spin One-half_, Theoretical and Mathematical Physics (Springer Berlin Heidelberg, 2012).
  * Orlando and Strong (2013) Elena Orlando and Andrew Strong, “Stellarics: Stellar and solar inverse compton emission package,”  (2013).
  * Aguilar  _et al._ (2014) M. Aguilar _et al._ (AMS), “Electron and Positron Fluxes in Primary Cosmic Rays Measured with the Alpha Magnetic Spectrometer on the International Space Station,” Phys. Rev. Lett. 113, 121102 (2014).
  * Potter and Cotter (2012) William J. Potter and Garret Cotter, “Synchrotron and inverse-Compton emission from blazar jets I: a uniform conical jet model,” Mon. Not. Roy. Astron. Soc. 423, 756 (2012), arXiv:1203.3881 [astro-ph.HE] .
  * Dermer (1995) Charles D. Dermer, “On the Beaming Statistics of Gamma-Ray Sources,” ApJ 446, L63 (1995).
  * Finke (2016) Justin D. Finke, “External Compton Scattering in Blazar Jets and the Location of the Gamma-Ray Emitting Region,” Astrophys. J. 830, 94 (2016), [Erratum: Astrophys.J. 860, 178 (2018)], arXiv:1607.03907 [astro-ph.HE] .
  * (53) Anderson C.M. Lai and Kenny C. Y. Ng, “In prep.” .
  * Krawczynski (2012) H. Krawczynski, “The Polarization Properties of Inverse Compton Emission and Implications for Blazar Observations with the GEMS X-Ray Polarimeter,” ApJ 744, 30 (2012), arXiv:1109.2186 [astro-ph.HE] .
  * Weisskopf  _et al._ (2016) Martin C. Weisskopf, Brian Ramsey, Stephen L. O’Dell, Allyn Tennant, Ronald Elsner, Paolo Soffita, Ronaldo Bellazzini, Enrico Costa, Jeffery Kolodziejczak, Victoria Kaspi, Fabio Mulieri, Herman Marshall, Giorgio Matt, and Roger Romani, “The imaging x-ray polarimetry explorer (ixpe),” Results in Physics 6, 1179–1180 (2016).
  * Zhang  _et al._ (2016) S. N. Zhang _et al._ (eXTP), “eXTP – enhanced X-ray Timing and Polarimetry Mission,” Proc. SPIE Int. Soc. Opt. Eng. 9905, 99051Q (2016), arXiv:1607.08823 [astro-ph.IM] .
  * Alessandro  _et al._ (2021) D. A. Alessandro _et al._ , “Gamma-ray Astrophysics in the MeV Range: the ASTROGAM Concept and Beyond,” arXiv e-prints , arXiv:2102.02460 (2021), arXiv:2102.02460 [astro-ph.IM] .

##  Appendix A Relating positive determinant to the kinematic constraint of
$E_{2}$

A non-negative determinant in the quadratic equation Eq. 16
$\Delta=A^{2}+B^{2}-C^{2}\geq 0$ secures real solutions and sets the kinematic
constraint on the possible range of $E_{2}$. To illustrate this, rewrite the
determinant into a quadratic equation of $E_{2}$:

$\displaystyle K_{1}E^{2}_{2}+K_{2}E_{2}+K_{3}\geq 0$ (21) $\displaystyle
K_{1}$ $\displaystyle=$
$\displaystyle\beta^{2}(\cos^{2}\theta+\sin^{2}\theta\cos^{2}\phi_{e})-\left[\frac{E_{1}(1-\cos\theta)}{E_{e}}+1\right]^{2}$
$\displaystyle K_{2}$ $\displaystyle=$ $\displaystyle
2E_{1}\left[\frac{E_{1}(1-\cos\theta)}{E_{e}}+1-\beta\cos\theta\right]$
$\displaystyle K_{3}$ $\displaystyle=$ $\displaystyle(\beta^{2}-1)E^{2}_{1}.$

Eq. 21 has another determinant $\Delta^{\prime}=K_{2}^{2}-4K_{1}K_{3}$ which
is positive definite for a physical solution. In the non-relativistic KN
regime, $\beta\rightarrow 0$, $E_{1}\sim m_{e}\approx E_{e}$, We have
$K_{1}<0$ such that $E_{2}$ is bounded the roots of Eq. 21. We also have
$\Delta^{\prime}\approx 0$, so:

$\displaystyle\frac{E_{1}}{\frac{E_{1}}{m_{e}}(1-\cos\theta)+1}$
$\displaystyle\leq$ $\displaystyle
E_{2}\leq\frac{E_{1}}{\frac{E_{1}}{m_{e}}(1-\cos\theta)+1}$ (22)
$\displaystyle\implies E_{2}$ $\displaystyle=$
$\displaystyle\frac{E_{1}}{\frac{E_{1}}{m_{e}}(1-\cos\theta)+1}=E_{\text{Compton}}\,,$

as expected.

In the Thomson regime, $E^{\prime}_{1}\ll m_{e}$ and $E_{2}$ is again bounded
by the roots as $K_{1}<0$. We further consider ultrarelativistic limit such
that $\gamma\ll 1$, both maximum and minimum scattered photon energy $E_{2}$
occurs at $\cos\theta=-1$ scattering in this limit. Manipulation on
$\Delta^{\prime}$ yields:

$\displaystyle\begin{split}K^{2}_{2}-4K_{1}K_{3}&=(1+\beta)^{2}-(1-\beta^{2})^{2}\\\
&=(1+\beta)^{2}[1-(1-\beta)^{2}]\\\
&\approx(1+\beta)^{2}\left(1-\frac{1}{4\gamma^{4}}\right),\end{split}$ (23)

where the relativistic approximation $\beta\approx 1-1/(2\gamma^{2})$ is used
in last line. With Eq. 23 and additional calculations, the conditional
$\Delta\geq 0$ becomes:

$\displaystyle\frac{(1-(1-1/(8\gamma^{4}))}{1-\beta}E_{1}\leq$ $\displaystyle
E_{2}$ $\displaystyle\leq\frac{1+(1-1/(8\gamma^{4}))}{1-\beta}E_{1}$
$\displaystyle\frac{1}{4\gamma^{2}}E_{1}\leq$ $\displaystyle E_{2}$
$\displaystyle\leq 4\gamma^{2}E_{1}.$ (24)

Again, it agrees with the limits mentioned in Rybicki and Lightman (1986).
Therefore, a real solution of Eq. 18 implies a physical solution for the
scattering.


11institutetext: Faculty of Technical Sciences, Novi Sad 21125, Serbia
11email<EMAIL_ADDRESS>
22institutetext: University of Cambridge, Cambridge CB3 0FD, United Kingdom
22email<EMAIL_ADDRESS>
# An Effective Loss Function for Generating 3D Models from Single 2D Image
without Rendering

Nikola Zubić Work performed while the author was Research Intern Apprentice
under the supervision of professor Pietro Liò.11 0000-0001-9816-2718 Pietro
Liò 22 0000-0002-0540-5053

###### Abstract

Differentiable rendering is a very successful technique that applies to a
Single-View 3D Reconstruction. Current renderers use losses based on pixels
between a rendered image of some 3D reconstructed object and ground-truth
images from given matched viewpoints to optimise parameters of the 3D shape.

These models require a rendering step, along with visibility handling and
evaluation of the shading model. The main goal of this paper is to demonstrate
that we can avoid these steps and still get reconstruction results as other
state-of-the-art models that are equal or even better than existing category-
specific reconstruction methods. First, we use the same CNN architecture for
the prediction of a point cloud shape and pose prediction like the one used by
Insafutdinov & Dosovitskiy. Secondly, we propose the novel effective loss
function that evaluates how well the projections of reconstructed 3D point
clouds cover the ground-truth object’s silhouette. Then we use Poisson Surface
Reconstruction to transform the reconstructed point cloud into a 3D mesh.
Finally, we perform a GAN-based texture mapping on a particular 3D mesh and
produce a textured 3D mesh from a single 2D image. We evaluate our method on
different datasets (including ShapeNet, CUB-200-2011, and Pascal3D+) and
achieve state-of-the-art results, outperforming all the other supervised and
unsupervised methods and 3D representations, all in terms of performance,
accuracy, and training time.

###### Keywords:

3D Reconstruction Single-View 3D Reconstruction

##  1 Introduction

One of the main problems in 3D Computer Graphics and Vision is the ability of
a model to learn 3D structure representation and reconstruction [9].
Supervised 3D Deep Learning is highly efficient in direct learning from 3D
data representations [1], such as meshes, voxels, and point clouds. They
require a large amount of 3D data for the training process, and also, their
representation is sometimes complex for the task of direct learning. These
factors lead to the abandonment of this approach because of its inefficient
performance and time consumption. Unsupervised 3D structural learning learns
3D structure without 3D supervision and represents a promising approach.

Differentiable rendering is a novel field that allows the gradients of 3D
objects to be calculated and propagated through images [11]. It also reduces
the requirement of 3D data collection and annotation, while enabling a higher
success rate in various applications. Their ability to create a bond between
3D and 2D representations, by computing gradients of 2D loss functions with
the respect to 3D structure, makes them a key component in unsupervised 3D
structure learning. These loss functions are based on differences between RGB
pixel values [13]. By rendering the predicted 3D structure from a specific
viewpoint and then evaluating the loss function based on pixel-wise loss
between rendered and ground-truth image, model parameters are optimised to
reconstruct the desired 3D structure.

However, these evaluation techniques are very time-consuming. They don’t
contribute at all to an accurate 3D structure reconstruction. Here, we propose
a novel idea for fast 3D structure reconstruction (in the form of a point
cloud silhouette) and then we convert it to a 3D mesh and transfer the
object’s texture from a 2D image onto the reconstructed 3D object. Hence,
unlike in loss functions that are based on pixels, our approach has an
effective loss function that arises exclusively from the 2D projections of 3D
points, without interpolation based on pixels, shading and visibility
handling.

##  2 Related work

###  2.1 3D representations

Previous works [23, 29] have concentrated on mesh reconstruction by using the
full 3D supervision approach. The main problem with these approaches, besides
inefficiency, is the usage of ground-truth 3D meshes, and they are mostly
available in a limited number of datasets. Some approaches [24, 25] solved
this problem by using 2D supervision from multiple-scene images based on
voxels.

###  2.2 Differentiable rendering

Prediction of 3D models from single images while achieving high-quality visual
results is possible by using the differentiable renderer. A differentiable
rendering framework allows gradients to be analytically (or approximately)
computed for all pixels in an image. Famous frameworks include: RenderNet [19]
and OpenDR [17].

###  2.3 Unsupervised learning of shape and pose with differentiable point
clouds

The work that inspired us addresses the learning of an accurate 3D shape and
camera pose from a collection of unlabeled category-specific images [8]. It
uses a specific convolutional neural network architecture to predict both
model’s shape and the pose from a single image. However, it is still time-
consuming since it uses differentiable point cloud projection.

##  3 Proposed method

###  3.1 Intuitive overview

In order to overcome the problems of structural 3D learning, unsupervised
methods introduced different differentiable renderers [8, 9, 11, 3, 16, 17,
19] to first render the reconstructed 3D shape into 2D images from different
view-angles and then portray them as what got obtained through complete
supervision. After this, we can calculate the pixel-wise losses between those
2D images from different view-angles and real (ground-truth) images from the
dataset. Since the renderer is differentiable, the loss between these images
back-propagates through the network to train it.

Figure 1: Our method removes the rendering process and requires only 2D
projections of 3D point clouds. During the generation of 3D shapes using
multiple silhouette images (from different viewing angles), 2D projections of
all points on the shape should uniformly cover the silhouette from each
viewing angle. We implement this using two key ideas (that together form
effective loss function). (1) For 3D shapes formed by 3D points, their
projections for each view should locate within the silhouette. (2) All
projections for each silhouette should distribute uniformly. We achieve this
by maximising the loss between each of the pairs of these 2D projections. P \-
Point cloud representation, c \- Pose of camera.

To evaluate the pixel-wise loss, previous differentiable renderers rendered
the images by taking into account some form of interpolation [3] of the
reconstructed 3D structure over each pixel, such as rasterisation and
visibility handling.

We train a network that learns to generate a 3D point cloud based on a single
image using the images from a dataset (from different view-angles) as
supervision which is opposed to those that use ground-truth point clouds as
supervision.

Current methods render based on differentiable renderers that render images of
the reconstructed 3D shape and actual images and then minimise the pixel-wise
loss to optimise the reconstructed 3D shape.

Total effective loss informs us how well the projected points cover the
objective silhouette. The process includes two terms, one that forces all the
projections into the silhouette where the projections initialise randomly, and
the other term moves the projections such that the distance between every two
of them is the maximum possible, which allows the projections to cover the
silhouette uniformly. Starting from some point cloud (randomly initialised),
we can force all the projections in the silhouette using the first term, and
then using the second term, we can uniformly distribute projections to cover
the whole silhouette.

Figure 2: Generated 3D point cloud is transformed into 3D mesh and then
textured.

After completing the process shown in Figure 1, we generate 3D point cloud for
a desired image. After this, we apply Poisson Surface Reconstruction [12] to
generate 3D mesh from given 3D point cloud and then we use GAN for texture
mapping on a particular 3D mesh and produce a textured 3D mesh based on the
input image texture, which is shown in the Figure 2. Main paper deals with
implementation details of an Effective Loss Function $E_{L}$ which is our
novelty, and compares the results with other approaches. More details and case
study is available in Appendix 0.A.

###  3.2 Implementation details

Our goal is to learn the structure of 3D point clouds ($P$) formed by $N$
points $n_{j}$ only from $G_{t}$ ground-truth images of the silhouette
$S_{i}$, where $j$ $\in$ $[1,N]$ and $i$ $\in$ $[1,G_{t}]$. Current
differentiable renderers rely on point clouds ($P$) rendering into raster
images $S^{\prime}_{i}$ from $i$-th viewing angle, which are used to produce a
loss by comparing $S^{\prime}_{i}$ and $S_{i}$ pixel by pixel. These steps are
not necessary to get a precise solution.

Let the projection of the point $n_{j}$ in view $i$ be ${p}_{j}^{i}$. The
error evaluates how well the sets of projected points
$\left\\{\boldsymbol{p}_{j}^{i}\mid j\in[1,N]\right\\}$ cover the silhouette
of the object. So, the loss is composed of two parts.

Projection$p_{j}$$n_{j}$Projection$p_{j}$$n_{j}$$p_{j^{\prime}}$$n_{j^{\prime}}$

Figure 3: The left and right grids represent two ground-truth silhouette
images $S_{i}$. $n_{j}$ point projects onto an image, and its projection is
$p_{j}$. Left: For 3D shapes formed by 3D points, their projections for each
view should locate within the silhouette, where the whole white grid is a
silhouette, and its pixel values are $1$ (red square). So, we are minimising
differences between pixel values of projections and $1$ for every projection.
Right: Besides that, we must not only minimise the first term loss but also
the second term loss which maximises the distance between two different point
projections from a 3D point cloud: $p_{j}$ and $p_{j^{\prime}}$.

If we have a predicted 3D point cloud and a binary image of the silhouette,
the loss calculates as follows: First, we project the points $n_{j}$ and get
projections $p_{j}$ (we write abbreviated without $i$, this is
$\boldsymbol{p}_{j}^{i}$) on the images of the silhouette $S_{i}$, where the
pixel value of the projection $p_{j}$ is denoted by $\boldsymbol{\pi_{i}}$.
The first term penalises points outside of the foreground by calculating the
difference $\boldsymbol{1-\pi_{i}}$, assuming that the foreground in the
binary silhouette image has a value of $1$. Minimising this loss will force
all projections into the foreground. Additionally, the second term adjusts the
spatial distribution of the projected points. It forces the pairs of
projections in the foreground to be as far apart from each other as possible
(right grid shown in the Figure 3). Thus, such a system arranges the 3D
locations of the points $n_{j}$ through their projections $p_{j}$ by
simultaneously optimising these two losses.

The first term is calculated as the difference between $1$ and the pixel value
$\pi_{i}$ of each projection $p_{j}^{i}$ on the silhouette image $S_{i}$. We
make use of bilinear interpolation to calculate the value $\pi_{i}$ using the
binary pixel values of the nearest pixels around $p_{j}^{i}$. All projections
are forced to the foreground for all silhouette images by minimising the
following $L_{1}$ norm:

$L_{1}\left(\boldsymbol{\pi}_{i}\right)=\left\|1-\boldsymbol{\pi}_{i}\right\|_{1}$
(1)

However, it is impossible to force all projections into the foreground by
minimising this $\boldsymbol{L1}$ norm. If we optimise point cloud according
to some silhouette image (a) and start from some randomly initialised points
(b), then we will get inadequate point projections if we use $L1$ norm, as
shown in the Figure 4.

There are two reasons why this problem occurs. One reason is the fact that
$L1$ norm is non-differentiable. Even if we only look at the difference, the
second reason is that we only examine the pixel intensity based on the
difference between $1$ and the interpolated pixel value $\pi_{i}$ based on the
four closest binary pixel values. This prevents training if the projections
$p_{j}^{i}$ are too far from the foreground.

Figure 4: We are given a silhouette image (a) and randomly initialized
projections (b). Then we cannot force projections into the foreground (c)
because standard first term loss has a local minimum problem, which results in
a non-uniform disposition of projections (blue dots). This problem is solved
by smoothing the original silhouette to obtain pixel values of projections and
calculate the difference.

Our goal is to produce non-zero gradients anywhere in the background part,
while the pixel values in the foreground part do not require a modification.
We will denote these processed silhouette images as $S_{i}^{G}$, to
distinguish between the original silhouette image $S_{i}$ and the processed
image. For each pixel $x$ on the background of the silhouette image $S_{i}$,
we write:

$\boldsymbol{S}_{i}^{G}(\boldsymbol{x})=\left\\{\begin{aligned}
1,&\boldsymbol{x}\in\mathcal{F}\\\
1-d(\boldsymbol{x},\partial\mathcal{F}),&\boldsymbol{x}\in\bar{\mathcal{F}}\end{aligned}\right.$
(2)

where
$\mathcal{F}=\left\\{\boldsymbol{x}\mid\boldsymbol{\pi}_{i}(\boldsymbol{x})=1\right\\}$
is the foreground, while
$\bar{\mathcal{F}}=\left\\{\boldsymbol{x}\mid\boldsymbol{\pi}_{i}(\boldsymbol{x})=0\right\\}$
is a background, and $\partial\mathcal{F}$ is the foreground’s boundary.
$d(\boldsymbol{x},\partial\mathcal{F})$ represents the $L_{2}$ distance
between $x$ and his closest $\partial\mathcal{F}$, which is normalised by the
resolution of the $S_{i}$.

Normalisation is also performed on the processed pixel values in the
background for them to lie in the interval $(0,1)$. Min-max normalisation is
used for this sub-task:
$\boldsymbol{S}_{i}^{G}(\bar{\mathcal{F}})=\operatorname{minmax}\left(\boldsymbol{S}_{i}^{G}(\bar{\mathcal{F}})\right)$.
Finally, the modified first term loss function is:

$\mathcal{M}_{1}\left(\boldsymbol{\pi}_{i}\right)=\left\|1-\boldsymbol{\pi}_{i}^{G}\right\|_{1}$
(3)

Figure 5: With modified first term loss function, it is possible to force all
projections of randomly initialised points (b) from a smoothed silhouette
image (a) into the foreground (c) by minimising the (3). Blue dots represent
the projections.

According to Figure 5, using only the first term loss leads to non-uniform
point projections in the foreground. To accurately represent the 3D shape and
cover the silhouette, we will use a second term loss. Through this loss, we
will model the spatial relationship between every two pairs of projections.
That loss should force projections inside the foreground. They should be as
far away from each other as possible.

To solve this problem, we propose a second-term loss function that increases
the distance between projection pairs that are deeper within the foreground
and reduces it for projection pairs around the foreground’s boundary. It skips
projections within the background.

For every projection pair $p_{j}^{i}$ and $p_{j^{\prime}}^{i}$, the $L2$
distance is calculated by the formula:

$d\left(p_{j}^{i},p_{j^{\prime}}^{i}\right)=\left\|p_{j}^{i}-p_{j^{\prime}}^{i}\right\|_{2},$
(4)

which we then normalise according to the resolution of the silhouette image.

This approach tends to maximise the distance
$d\left(p_{j}^{i},p_{j^{\prime}}^{i}\right)$. We use the Gaussian function
[14] to obtain a loss based on the invariance of the structure which decreases
with increasing the distance. So, we can essentially minimise the loss of
invariance along with the modified first term loss $\mathcal{M}_{1}$.

For each projection $p_{j}^{i}$, loss based on the invariance of the structure
models its spatial relationship with all other projections
$p_{j^{\prime}}^{i}$:

$\mathcal{L}_{2}\left(p_{j}^{i},\left\\{p_{j^{\prime}}^{i}\right\\}\right)=w_{j}^{i}\sum_{j^{\prime}=1}^{N}\left[w_{j^{\prime}}^{i}\cdot\exp\left(\frac{-d\left(p_{j^{\prime}}^{i}p_{j^{\prime}}^{i}\right)}{\theta}+\mu_{j}^{i}\right)\right],$
(5)

where $w_{j}^{i}$ and $w_{j^{\prime}}^{i}$ are weights corresponding to the
projections $p_{j}^{i}$ and $p_{j^{\prime}}^{i}$ respectively, $\theta>0$ is
the decay parameter, $\mu_{j}^{i}>0$ is the boundary bias for the projection
$p_{j}^{i}$.

$w_{j}^{i}$ expresses to what level the projection $p_{j}^{i}$ merges with the
background. If that weight is set to zero, the projection $p_{j}^{i}$ is such
that the invariance of the structure is completely removed so that the
modified first term loss $\mathcal{M}_{1}$ immediately forces $p_{j}^{i}$
within the foreground. The decay (merge) parameter controls the merge interval
(invariance of the structure intensity) of a given background 3D model. The
projection boundary bias $\mu_{j}^{i}$ for projection $p_{j}^{i}$ controls the
distance to the foreground’s boundary where the invariance over that
projection reduces.

Weight $w_{j}^{i}$ is calculated using bilinear interpolation based on the
closest binary pixel values in the silhouette image $S_{i}$, as shown in the
Figure 3. We use multi-scale gradients [21] to compute $\mu_{j}^{i}$. Binary
pixel values are extracted from adjacent points located at the vertices of the
squares in the grid (around the projection $p_{j}^{i}$) - Figure 3. We perform
interpolations over them, and we take the mean value of all of these
interpolations to calculate $\mu_{j}^{i}$. This approach progressively reduces
invariance of the structure as $p_{j}^{i}$ approaches the foreground’s
boundary.

Finally, the effective loss function $E_{L}$ is calculated through
simultaneous minimisation of the modified first term loss function
$\mathcal{M}_{1}$ and the second term loss function $\mathcal{L}_{2}$ based on
the invariance of the structure. The total error $E_{L}$ is obtained by the
following formula ($\alpha$ and $\beta$ are used for balancing the losses, we
average over points $N$ and views $G_{t}$):

$E_{L}=\frac{\sum_{i=1}^{G_{t}}\sum_{j=1}^{N}\left(\alpha\mathcal{M}_{1}\left(\pi_{i}\right)+\beta\mathcal{L}_{2}\left(p_{j}^{i},\left\\{p_{j^{\prime}}^{i}\right\\}\right)\right)}{G_{t}\cdot
N}$ (6)

After this process, we have a 3D point cloud which is then transformed to a 3D
mesh using Poisson Surface Reconstruction [12]. We use GAN [30] for texture
mapping on a particular 3D mesh and produce a textured 3D mesh based on the
input image texture, which is shown in Figure 2. The generator generates
displacement maps and textures, and the discriminator discriminates between
real/fake displacement maps and textures.

##  4 Results

In this section, we succinctly report the results, primarily through
comparison with other approaches. More details and case study is available in
Appendix 0.A.

The quantitative results using Chamfer’s distance [22] are shown in Table 1.
Our point cloud output (Ours) outperforms its voxel equivalent (Ours-V) in all
cases. Chamfer’s distance improves with the increase of resolution. We also
outperform the previous best method that used rendering [8] and DRC method
[24].

Table 1:  Quantitative results on shape prediction with known camera pose (on ShapeNet dataset). We report the Chamfer’s distance between normalised point clouds, multiplied by $100$ and use three categories: Airplanes, Cars and Chairs. Our point cloud output outperforms all other methods in terms of Chamfer’s distance. Lower value is better; bold = best. |  |  Resolution $32$ |  |  Resolution $64$ |  |  Resolution $128$
---|---|---|---|---|---|---
|  |  DRC [25] |  DPC [8] | Ours-V | Ours |  |  DPC [8] | Ours |  |  DPC [8] | Ours
Airplane |  | $8.35$ | $4.52$ | $4.49$ | $\mathbf{3.99}$ |  | $3.50$ | $\mathbf{3.15}$ |  | 2.84 | $\mathbf{2.63}$
Car |  | $4.35$ | $4.22$ | $3.75$ | $\mathbf{3.79}$ |  | $2.98$ | $\mathbf{2.86}$ |  | 2.42 | $\mathbf{2.37}$
Chair |  | $8.01$ | $5.10$ | $5.34$ | $\mathbf{4.64}$ |  | $4.15$ | $\mathbf{3.99}$ |  | 3.62 | $\mathbf{3.46}$
Mean |  | $6.90$ | $4.61$ | $4.53$ | $\mathbf{4.14}$ |  | $3.55$ | $\mathbf{3.33}$ |  | 2.96 | $\mathbf{2.82}$
Table 2:  Quantitative Volumetric IoU [20] comparison with differentiable
renderers for different 3D representations and supervised methods (on ShapeNet
dataset). We use three categories: Airplanes, Cars and Chairs. Bigger value is
better; bold = best.

|  | Unsupervised learning |  |  | Supervised learning
---|---|---|---|---|---
|  |  SoftRas [16] |  DIB-R [3] | Ours |  |  |  P2M [27] IN [15] |  |  RN [4] AN [6] |  |  DSN [32] 3DN [28] |  |  ON [18] Ours
Airplane |  | $58.4$ | $57.0$ | $\mathbf{62.4}$ |  |  | $51.5$ | $55.4$ |  | $42.6$ | $39.2$ |  | $57.5$ | $54.3$ |  | $57.1$ | $\mathbf{75.3}$
Car |  | $77.1$ | $\mathbf{78.8}$ | $75.6$ |  |  | $50.1$ | $74.5$ |  | $66.1$ | $22.0$ |  | $74.3$ | $59.4$ |  | $73.7$ | $\mathbf{75.1}$
Chair |  | $49.7$ | $52.7$ | $\mathbf{58.3}$ |  |  | $40.2$ | $52.2$ |  | $43.9$ | $25.7$ |  | $54.3$ | $34.4$ |  | $50.1$ | $\mathbf{57.8}$
Mean |  | $61.7$ | $62.8$ | $\mathbf{65.43}$ |  |  | $47.3$ | $60.7$ |  | $50.9$ | $29.0$ |  | $62.0$ | $49.4$ |  | $60.3$ | $\mathbf{64.97}$

Our results outperform state-of-the-art differentiable renderers in the
Volumetric IoU metric [20] while simultaneously being less time-consuming
during the training phase, as shown in Table 2. For cars, our outcome is
better than renderers based on voxels but very similar to renderers based on
meshes because meshes represent a superior initial 3D representation for large
areas of flat surfaces [10] (such as cars).

Table 3: Training time efficiency in hours. | $3D$ representations | Rendering |  $32^{2}$ image |  $64^{2}$ image |  $128^{2}$ image
---|---|---|---|---|---
| 2000 points/ | 8000 points/ | 16000 points/
|  $32^{3}$ voxels |  $64^{3}$ voxels |  $128^{3}$ voxels
DRC [25] | Voxels | Yes |  $\approx$14h |  $\approx$60h |  $\approx$216h
DPC [8] | Point clouds | Yes |  $\approx$14h |  $\approx$24h |  $\approx$72h
Ours | Point clouds | No |  $\approx$6.5h |  $\approx$11h |  $\approx$34.5h

Also, FID scores on Mesh (produced from 3D point cloud), Texture (extracted by
a GAN) and Both (final output - textured 3D mesh) produced state-of-the art
results, which can be seen in Figure 6.

##  5 Datasets, metrics & code

#### Datasets

We used the following datasets: ShapeNet [2] (train/test split from [8]),
CUB-200-2011 [26] (train/test split from [10]), and Pascal3D+ dataset [31]
(train/test split from [10]).

#### Metrics

Numerical evaluation for point clouds is performed by using Chamfer’s distance
[22] between predicted and real (ground-truth) point clouds.

Volumetric IoU [20] comparison is used by comparing the 3D grid voxelised from
the predicted point cloud with the one voxelised from the ground-truth point
cloud.

Fréchet Inception Distance (FID) is widely used as an evaluation metric [7]
(not only for 2D GANs but also for our task). FID scores will evaluate 2D
projections of generated point clouds to meshes. 3D mesh and textures are
evaluated separately in this process.

#### Code

Implementation, data and trained models are available at:
https://github.com/NikolaZubic/2dimageto3dmodel

##  6 Possible extensions and limitations

Our work can be used as part of more complex software that deals with video
games, animation, or any aspect where it is necessary to have base 3D models
which can be additionally polished with more sculpting. The work can be
extended by taking even more account of the smooth characteristics of the
functions. Our work is the first one to tackle the challenging problem of
Single-View 3D Reconstruction without Rendering. Results are impressive, but
this task is far from being fully solved. Our model struggles to predict
camera poses that are rare in the training dataset. Also, it captures the
major shape characteristics of each instance but ignores some details. For
example, legs of zebras, cows and horses are not separated (Figure 12 and
Figure 13).

Figure 6: We use real-world 2D bird images as input for generating a 3D model.
In the first column is the input where we have images of the real birds, in
the second column, there is a generated 3D mesh (obtained from 3D point cloud
after Poisson Surface Reconstruction [12]), and in the next four columns,
there is a predicted 3D model visible in 4 poses.

##  7 Conclusion

In this paper, we proposed a 3D reconstruction based on a single image, a
method for learning the pose and shape of 3D objects given only their 2D
projections, using the initial point cloud representation and then converting
that representation to a 3D mesh. Mesh is textured using GANs to produce the
final output. Extensive experiments have shown that point clouds compare well
with the voxel-based representation, such as performance and accuracy. The
proposed framework learns to predict shape, texture, and pose from single
images, without rendering step, based solely on 2D projections of 3D point
clouds and their coverage of the ground-truth silhouette. While rendering
requires exhaustive computation, our key finding is that it does not endow
accuracy in 3D structure learning.

##  8 Acknowledgements

We would like to thank the anonymous reviewers for reviewing the paper before
final submission and providing helpful and detailed comments.

## References

  * [1] Ahmed, E., Saint, A., Shabayek, A.E.R., Cherenkova, K., Das, R., Gusev, G., Aouada, D., Ottersten, B.: A survey on deep learning advances on different 3d data representations (2019)
  * [2] Chang, A.X., Funkhouser, T., Guibas, L., Hanrahan, P., Huang, Q., Li, Z., Savarese, S., Savva, M., Song, S., Su, H., Xiao, J., Yi, L., Yu, F.: Shapenet: An information-rich 3d model repository (2015)
  * [3] Chen, W., Ling, H., Gao, J., Smith, E., Lehtinen, J., Jacobson, A., Fidler, S.: Learning to predict 3d objects with an interpolation-based differentiable renderer. In: Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F., Fox, E., Garnett, R. (eds.) Advances in Neural Information Processing Systems. vol. 32, pp. 9609–9619. Curran Associates, Inc. (2019), https://proceedings.neurips.cc/paper/2019/file/f5ac21cd0ef1b88e9848571aeb53551a-Paper.pdf
  * [4] Choy, C.B., Xu, D., Gwak, J., Chen, K., Savarese, S.: 3d-r2n2: A unified approach for single and multi-view 3d object reconstruction (2016)
  * [5] Community, B.O.: Blender - a 3D modelling and rendering package. Blender Foundation, Stichting Blender Foundation, Amsterdam (2018), http://www.blender.org
  * [6] Groueix, T., Fisher, M., Kim, V.G., Russell, B.C., Aubry, M.: Atlasnet: A papier-mâché approach to learning 3d surface generation (2018)
  * [7] Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: Gans trained by a two time-scale update rule converge to a local nash equilibrium (2018)
  * [8] Insafutdinov, E., Dosovitskiy, A.: Unsupervised learning of shape and pose with differentiable point clouds. CoRR abs/1810.09381 (2018), http://arxiv.org/abs/1810.09381
  * [9] Jimenez Rezende, D., Eslami, S.M.A., Mohamed, S., Battaglia, P., Jaderberg, M., Heess, N.: Unsupervised learning of 3d structure from images. In: Lee, D., Sugiyama, M., Luxburg, U., Guyon, I., Garnett, R. (eds.) Advances in Neural Information Processing Systems. vol. 29, pp. 4996–5004. Curran Associates, Inc. (2016), https://proceedings.neurips.cc/paper/2016/file/1d94108e907bb8311d8802b48fd54b4a-Paper.pdf
  * [10] Kanazawa, A., Tulsiani, S., Efros, A.A., Malik, J.: Learning category-specific mesh reconstruction from image collections (2018)
  * [11] Kato, H., Beker, D., Morariu, M., Ando, T., Matsuoka, T., Kehl, W., Gaidon, A.: Differentiable rendering: A survey (2020)
  * [12] Kazhdan, M., Bolitho, M., Hoppe, H.: Poisson surface reconstruction. In: Proceedings of the Fourth Eurographics Symposium on Geometry Processing. p. 61–70. SGP ’06, Eurographics Association, Goslar, DEU (2006)
  * [13] Kumar, T., Verma, K.: A theory based on conversion of rgb image to gray image. International Journal of Computer Applications 7(2), 7–10 (2010)
  * [14] Li, Z., Shafiei, M., Ramamoorthi, R., Sunkavalli, K., Chandraker, M.: Inverse rendering for complex indoor scenes: Shape, spatially-varying lighting and svbrdf from a single image (2019)
  * [15] Liu, J., Lu, H.: Imnet: A learning based detector for index modulation aided mimo-ofdm systems (2019)
  * [16] Liu, S., Chen, W., Li, T., Li, H.: Soft rasterizer: Differentiable rendering for unsupervised single-view mesh reconstruction (2019)
  * [17] Loper, M.M., Black, M.J.: Opendr: An approximate differentiable renderer. In: European Conference on Computer Vision. pp. 154–169. Springer (2014)
  * [18] Mescheder, L., Oechsle, M., Niemeyer, M., Nowozin, S., Geiger, A.: Occupancy networks: Learning 3d reconstruction in function space (2019)
  * [19] Nguyen-Phuoc, T., Li, C., Balaban, S., Yang, Y.L.: Rendernet: A deep convolutional network for differentiable rendering from 3d shapes (2019)
  * [20] Niemeyer, M., Mescheder, L., Oechsle, M., Geiger, A.: Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision (2020)
  * [21] Sreegadha, G.: Image interpolation based on multi scale gradients. Procedia Computer Science 85, 713–724 (2016). https://doi.org/https://doi.org/10.1016/j.procs.2016.05.258, https://www.sciencedirect.com/science/article/pii/S1877050916306081, international Conference on Computational Modelling and Security (CMS 2016)
  * [22] Sun, X., Wu, J., Zhang, X., Zhang, Z., Zhang, C., Xue, T., Tenenbaum, J.B., Freeman, W.T.: Pix3d: Dataset and methods for single-image 3d shape modeling (2018)
  * [23] Tatarchenko, M., Dosovitskiy, A., Brox, T.: Octree generating networks: Efficient convolutional architectures for high-resolution 3d outputs (2017)
  * [24] Tulsiani, S., Efros, A.A., Malik, J.: Multi-view consistency as supervisory signal for learning shape and pose prediction (2018)
  * [25] Tulsiani, S., Zhou, T., Efros, A.A., Malik, J.: Multi-view supervision for single-view reconstruction via differentiable ray consistency (2017)
  * [26] Wah, C., Branson, S., Welinder, P., Perona, P., Belongie, S.J.: The caltech-ucsd birds-200-2011 dataset (2011)
  * [27] Wang, N., Zhang, Y., Li, Z., Fu, Y., Liu, W., Jiang, Y.G.: Pixel2mesh: Generating 3d mesh models from single rgb images (2018)
  * [28] Wang, W., Ceylan, D., Mech, R., Neumann, U.: 3dn: 3d deformation network (2019)
  * [29] Wu, J., Wang, Y., Xue, T., Sun, X., Freeman, W.T., Tenenbaum, J.B.: Marrnet: 3d shape reconstruction via 2.5d sketches (2017)
  * [30] Xian, W., Sangkloy, P., Agrawal, V., Raj, A., Lu, J., Fang, C., Yu, F., Hays, J.: Texturegan: Controlling deep image synthesis with texture patches (2018)
  * [31] Xiang, Y., Mottaghi, R., Savarese, S.: Beyond pascal: A benchmark for 3d object detection in the wild. In: IEEE Winter Conference on Applications of Computer Vision (WACV) (2014)
  * [32] Xu, Q., Wang, W., Ceylan, D., Mech, R., Neumann, U.: Disn: Deep implicit surface network for high-quality single-view 3d reconstruction (2019)

##  Appendix 0.A Appendix - Network architecture details

The network we used for the $3D$ reconstruction (in a point cloud
representation) based on a single image is composed of a $2D$ encoder and a
$3D$ point cloud decoder. The $2D$ encoder represents a $7$-layer CNN. The
first layer consists of a $5\times 5$ kernel with $16$ channels and a stride
of $2$. Each of the remaining layers has three kernels and comes in pairs,
where the given layer in pairs has a stride of $2$, while the second one has a
stride of $1$. The number of channels increases by a factor of $2$ after each
layer with a stride. These convolutional layers are followed by two fully
connected layers whose dimensions are $1024$. The $3D$ point cloud decoder has
one fully connected layer whose dimensions are $1024$, and it then predicts
the point cloud representation. The point cloud that is consisted of $N$
points gets predicted as a vector whose dimensions are $3N$ (point
coordinates).

We have chosen this architecture because it achieved state-of-the-art results
for the problem of Single-View 3D Reconstruction, but the process of
differentiable rendering was unnecessary, and we obtained a more precise
solution without it. This architecture represents an optimal solution because
it can firmly reconstruct real-world data, despite the absence of accurate
ground-truth camera poses. Also, it can be used as a basis to learn colors and
textures, but that would require explicit reasoning about lighting and
shading.

##  Appendix 0.B Appendix - Metrics details

The following section will explain all the details about metrics used for
comparison with other approaches.

###  0.B.1 Chamfer’s distance

Numerical evaluation is performed by using Chamfer’s distance [22] between
predicted and real (ground-truth) point clouds:

$\displaystyle d_{Chamfer}\left(P_{1},P_{2}\right)=$
$\displaystyle\frac{1}{\left|P_{1}\right|}\sum_{r\in P_{1}}\min_{s\in
P_{2}}\|r-s\|_{2}$ (7) $\displaystyle+\frac{1}{\left|P_{2}\right|}\sum_{s\in
P_{2}}\min_{r\in P_{1}}\|s-r\|_{2},$

where $P_{1}$ is the predicted point cloud and $P_{2}$ is the ground-truth
point cloud, $r$ is a point on $P_{1}$ and $s$ is a point on $P_{2}$.
$\left|P_{1}\right|$ and $\left|P_{2}\right|$ represent the number of points
for point clouds $P_{1}$ and $P_{2}$. The first sum evaluates the precision of
the predicted point cloud by computing how far on average is the closest
ground-truth point from a predicted point. The second sum measures the
coverage of the ground-truth by the predicted point cloud: how far is on
average the closest predicted point from a ground-truth point [8].

###  0.B.2 Volumetric IoU

When comparing with voxel-based results, we discretise the $3D$ space, where
the predicted or the ground-truth point clouds are located, into a $3D$ voxel
grid, where a voxel is set to $1$ if it contains a point. Volumetric IoU [20]
comparison is used by comparing the 3D grid voxelised from the predicted point
cloud with the one voxelised from the ground-truth point cloud.

###  0.B.3 Texture evaluation

Fréchet Inception Distance (FID) is widely used as an evaluation metric [7]
(not only for 2D GANs but also for our task). FID scores will evaluate 2D
projections of generated point clouds to meshes. 3D mesh and textures are
evaluated separately in this process.

In addition to the full FID, we report the Texture FID, where we used meshes
estimated using our method, and the Mesh FID, where we replaced generated
textures with pseudo-ground-truth ones. While we mostly rely on the Full FID
to discuss the results, the individual ones represent a convenient tool to
analyze how the model responds to different hyperparameters. Generated samples
render at $299\times 299$, and ground-truth images scale to this resolution.
We provide visualizations that give more insights into the conceptual
differences between these types of FID metrics, which can be shown in Figure
7.

Figure 7: Images on which we computed the FID scores. We generated them from
the viewpoint corresponding to the randomly selected image from the training
set. In the Mesh FID case, we output the mesh using the pseudo-ground-truth
texture from the real-world image. In the Texture FID, the actual mesh gets
textured using the generated texture from 2D GAN. In the Full FID and Mesh FID
case of the top-left vehicle, we can observe that the silhouette of the mesh
looks fine but straight lines and stripes present unsteady structural effect
caused by the underlying mesh, while in the Texture FID (which does not use
generated meshes) the lines appear more straight.

##  Appendix 0.C Appendix - Experiments and discussion

###  0.C.1 Datasets details

We carried out experiments that involve 3D shapes in three categories from the
ShapeNet [2] dataset, including chairs, airplanes, and cars. They are commonly
used for evaluation by others, and if the method is working well on them, then
we can say that it will also generalize on the other shapes. We followed the
same train/test splitting process as in [8], and employed the five rendered
views from each 3D shape and the ground-truth point clouds by [8]. More
precisely, we had three different resolutions for rendered views ($32^{2}$,
$64^{2}$, and $128^{2}$), all corresponding to the same set of ground-truth
point clouds with different numbers of points, as shown in Table 1.

For the CUB-200-2011 birds [26] dataset, we used the train/test split of [10],
which consists of $\approx$6k training images and $\approx$5.7k test images.
Each image has an annotated class label (out of 200 classes). We evaluated FID
on test images using poses (where applicable) from the training set, although
we concluded that the FID is almost identical between the two sets.

As for the Pascal3D+ [31] dataset, we used the same split as [10] to train our
model. The 2D GAN is trained only on the ImageNet subset ($4.7$k usable
images) since we noticed that the images in the Pascal3D+ dataset were too
small for practical purposes. The test split of [10] does not contain any
ImageNet images, so we evaluated FID scores on training images, motivated by
the previous observation of the CUB-200-2011 dataset.

###  0.C.2 Experiments details

We used the same architecture as described in Appendix section 0.A, which was
introduced by [8], but we replaced the differentiable rendering module with
our Effective Loss Function $E_{L}$. Their approach uses structural learning
of the 3D point clouds by employing pairs of RGB images. For each pair, the
network first outputs a point cloud representation from the first RGB image
and then renders the predicted point cloud from the view angle of the second
image. Their differentiable rendering module generates a rendered silhouette
image, and the neural network was trained by minimizing the pixel-wise error
between the rendered silhouette image and the silhouette of the second input
image. Our approach removes differentiable rendering and exploits the
projected positions of generated point clouds to create the loss $E_{L}$
required during the training. At the test time, the trained network with loss
$E_{L}$ outputs a 3D point cloud from a single RGB image.

Firstly, we compared our results with rendering-based approaches in terms of
Chamfer’s distance. All compared renderers produce silhouettes of the
predicted shapes to compute their loss concerning the ground-truth
silhouettes. We carried out the comparison by training the networks using
silhouette images at three different resolutions, as already mentioned. We
reported quantitative comparison in Table 1. Our results outperformed all
compared methods under all classes at all three resolutions. Our approach
showed an obvious advantage over voxel-based differentiable renderers, where
we recovered more geometry details in a more memory-efficient manner. In
addition, by omitting the rendering process, our approach achieves higher
accuracies of the reconstructed point clouds. These results further
demonstrate that our method is robust to changes in image resolutions and the
number of points.

Secondly, we compared our results with rendering-based methods for other 3D
representations (such as meshes and voxel grids) in terms of IoU. The
comparison includes Perspective Transform Nets [11], Neural Mesh Renderer
[11], Soft Rasterizer [16], and Interpolation-based Differentiable Renderer
[3]. The first two methods are voxel-based, while the other two are mesh-
based. To produce our IoU, we voxelized the point clouds predicted from images
with a resolution of $128^{2}$ in Table 1 into voxel grids with a resolution
$32^{3}$ to compare to the same ground-truth as other ones. Quantitative
comparison in Table 2 shows that we significantly outperform the state-of-the-
art differentiable renderers in terms of mean IoU, where we achieved the best
results under airplane and chair classes. Mesh-based approaches are limited to
a fixed (usually spherical) mesh topology. This leads to inaccuracies when
representing more complex surfaces, such as chairs, which often display non-
spherical topology.

Finally, we compared the results with the latest 3D supervised methods in
Table 2. In the first experiment, we conducted comparison with Multiview
aggregation for learning category-specific shape reconstruction [11]. Results
were reported by using their evaluation code. Following the same setting, we
used point clouds reconstructed from input images with a resolution of
$64^{2}$ in Table 1, scaled each predicted point cloud such that the diagonal
of its bounding box is $1$, and resampled a ground-truth point cloud to $8000$
points if there are more than $8000$ points. Table 4 shows that our results
significantly outperform Multiview aggregation under all three classes.

Table 4: Chamfer’s distance comparison with the latest supervised method
Multiview aggregation.

| Cars | Airplanes | Chairs
---|---|---|---
MV aggregation | 0.3331 | 0.2795 | 0.4637
Ours | 0.0342 | 0.0414 | 0.0437

Figure 8 shows a few generated, textured meshes rendered from the multiple
views in Blender [5], and also their corresponding textures. Results on the
CUB-200-2011 dataset have high resolution, but the back of the cars in the
Pascal3D+ dataset has some irregularities. After further analysis, we found
that the dataset is very imbalanced, with only $20\%$ of the images showing
the back of the car and the majority of them showed the frontal part. So, this
issue could be solved by using more training data.

Figure 8: Qualitative results on Pascal3D+ (left) and CUB-200-2011 (right) dataset. Each object has been rendered from three views (in Blender), and the top row represents the texture learned by GAN. Figure 9: All generated 3D models can be visualised in the Blender tool [5] and viewed in real-time from an user-free angle. Left: 3D mesh without texture. Right: Textured 3D mesh which represents the final output of our model based on a single (in this case: car) image. Table 5: FID scores on Mesh (produced from 3D point cloud), Texture (extracted by a GAN) and Both (final output - textured 3D mesh) grouped by dataset, texture resolution, and condition, in truncated case. Lower is better; bold = best in that dataset. |  |  | FID (truncated $\sigma$) |
---|---|---|---|---
Dataset | Texture resolution | Condition | $\sigma$ | Both | Tex. | Mesh |
CUB-200-2011 | 512x512 | None | 1 | 40.33 | 43.73 | 17.89 |
Class | 0.25 | 33.61 | 26.67 | 18.53 |
256x256 | Class | 0.25 | 32.28 | 29.62 | 18.88 |
Pascal3D+ | 512x512 | None | 1 | 40.99 | 30.76 | 27.17 |
Class | 0.75 | 26.23 | 21.89 | 22.96 |
Class + Colour | 0.5 | 30.50 | 21.10 | 26.86 |
256x256 | Class + Colour | 0.5 | 38.19 | 25.43 | 35.71 |

##  Appendix 0.D Appendix - Ablation study and efficiency

We carried out ablation studies to justify our claims in terms of the
effectiveness of each element of our method under airplanes at a resolution of
$2000$ points in Table 1. In Table 6, we report results with only some losses,
fewer views (like $G_{t}=3$ and $G_{t}=2$), and without weights and biases.

Table 6: Ablation studies in terms of Chamfer’s distance (CD).

| $\mathcal{M}_{1}$ | $\mathcal{L}_{2}$ | Pixel+$\mathcal{L}_{2}$ |  $\mathcal{M}_{1}$ \+ no $w_{j}^{i}$ |  $\mathcal{M}_{1}$ \+ no $\mu_{j}^{i}$ | $G_{t}=2$ | $G_{t}=3$ | $G_{t}=4$
---|---|---|---|---|---|---|---|---
CD | 19.50 | 139.10 | 24.59 | 4.58 | 4.41 | 4.79 | 4.54 | 4.01

This study shows that our loss $E_{L}$ cannot learn the structure of shapes
using only $\mathcal{M}_{1}$ or $\mathcal{L}_{2}$ loss, also not with standard
$L_{1}$ loss because of the local minimum issue and non-differentiability. The
second term loss and its hyperparameters (indicator weights and boundary bias)
contribute to the reconstruction accuracy and efficiency of optimization.
Parameters $\alpha$ and $\beta$ contribute to the conflict and trade-off
between modified first term loss and second term loss based on structure
invariance. Using fewer views than our $G_{t}=4$ views in training degenerates
the structure learning performance.

As for efficiency, we compared our model’s training time with state-of-the-art
differentiable renderers for 3D shapes, as shown in Table 3. The voxel-based
method (DRC) has a weakness in terms of a vast computational burden due to the
cubic complexity of voxel grids, which limits it to work only in low
resolutions such as $32^{3}$ and $64^{3}$ with a slow convergence rate.
Although the point cloud-based method by Insafutdinov & Dosovitskiy [8] does
not require 3D convolutional layers as DRC, the rendering procedure still
requires intensive computation with discrete 3D grids. So, this method
requires more time ($6\times 10^{5}$ mini-batch iterations) during training
than our method ($1\times 10^{5}$ mini-batch iterations).

We used parameters learned in different steps during training to reconstruct a
shape from a corresponding image in a test set. Additionally, by using an
image from test rather than the training set we demonstrated the
generalization ability learned in optimization, which strongly justifies our
effectiveness. Also, it is shown that our model adapts well to real-world
images. More examples can be seen in the last section of Appendix 0.F.

##  Appendix 0.E Appendix - Training details

We evaluated our loss using this network with ground-truth camera poses during
projections. Additionally, we used RGB images with three different resolutions
to train and to evaluate the generated point clouds in three resolutions that
include $2000$, $8000$, and $16000$ points. The network was trained using the
Adam optimizer with a batch size of 16 rendered images (4 views of 4 shapes),
where we iterated over $1\times 10^{5}$ batches in each experiment.

Figure 10: Total unsupervised loss decreases through time (more steps of
training) which is an indicator that our model is learning the desired
objective if we assume that objective is correctly set up. Figure 11: Ensemble
of pose regressors for each example where an additional pose is student’s
prediction. For every view we can produce point clouds and get projections.

##  Appendix 0.F Appendix - More visual results

Figure 12: More qualitative results on classes: horses & cows. Figure 13: More
qualitative results on classes: penguins & zebras.