††thanks: equally contributed††thanks: equally contributed††thanks: equally
contributed

# Quantized conductance with non-zero shot noise as a signature of Andreev
edge state

Manas Ranjan Sahu  Department of Physics, Indian Institute of Science,
Bangalore 560012, India  Arup Kumar Paul  Department of Physics, Indian
Institute of Science, Bangalore 560012, India  Jagannath Sutradhar  Department
of Physics, Indian Institute of Science, Bangalore 560012, India  K. Watanabe
National Institute for Materials Science, 1-1 Namiki, Tsukuba 305-0044, Japan
T. Taniguchi  National Institute for Materials Science, 1-1 Namiki, Tsukuba
305-0044, Japan  Vibhor Singh  Department of Physics, Indian Institute of
Science, Bangalore 560012, India  Subroto Mukerjee  Department of Physics,
Indian Institute of Science, Bangalore 560012, India  Sumilan Banerjee
<EMAIL_ADDRESS>Department of Physics, Indian Institute of Science,
Bangalore 560012, India  Anindya Das<EMAIL_ADDRESS>Department of
Physics, Indian Institute of Science, Bangalore 560012, India

###### Abstract

Electrical conductance measurements have limited scope in identifying Andreev
edge states (AESs), which form the basis for realizing various topological
excitations in quantum Hall (QH) - superconductor (SC) junctions. To
unambiguously detect AESs, we measure shot noise along with electrical
conductance in a graphene-based QH-SC junction at integer filling $\nu=2$.
Remarkably, we find that the Fano factor of shot noise approaches $\it{half}$
when the bias energy is less than the superconducting gap ($2\Delta$), whereas
it is close to zero above $2\Delta$. This is striking, given that, at the same
time, the electrical conductance remains quantized at $2e^{2}/h$ within and
above $2\Delta$. A quantized conductance is expected to produce zero-shot
noise due to its dissipationless flow. However, at a QH-SC interface, AESs
carry the current in the zero-bias limit and an equal mixing of electron and
hole-like states produces $\it{half}$ of the Poissonian shot noise with
quantized conductance. The observed results are in accord with our detailed
theoretical calculations of electrical conductance and shot noise based on
non-equilibrium Green’s function method in the presence of disorder. Our
results pave the way in using shot noise as a detection tool in the search of
exotic topological excitations in QH-SC hybrids.

Chiral quantum Hall (QH) edge states in proximity with a superconductor (SC)
can give rise to exotic excitations Beenakker (2015); Bishara and Nayak
(2007); Nayak  _et al._ (2008); Mong  _et al._ (2014); Qi  _et al._ (2010);
Fisher (1994) like Majorana fermion. There are several promising theoretical
proposals of realizing chiral Majorana fermion at QH-SC interfaces Qi  _et
al._ (2010); Mong  _et al._ (2014); San-Jose  _et al._ (2015), however, its
evidence is still inconclusive Rahmani and Franz (2019); Manousakis  _et al._
(2020); Motome and Nasu (2020); Kayyalha  _et al._ (2020). Realization of the
electron-hole hybrid states called Andreev edge states (AESs) at the QH-SC
interface is an important step in this quest and graphene hosting clean QH
edge states at moderate magnetic field is an ideal platform. The recent
developments of several superconductors with large critical magnetic field and
transparent interfaces with high quality graphene have paved the way for a
number of interesting experimental observations Rickhaus  _et al._ (2012);
Shalom  _et al._ (2016); Amet  _et al._ (2016); Park  _et al._ (2017); Lee
_et al._ (2017); Sahu  _et al._ (2018); Seredinski  _et al._ (2019); Zhao  _et
al._ (2020); Sahu  _et al._ (2019); Lee and Lee (2018); Finkelstein and Amet
(2017); Dvir  _et al._ (2021), such as crossed Andreev conversion Lee  _et
al._ (2017), supercurrent in QH regime Amet  _et al._ (2016), inter-Landau-
level Andreev reflection Sahu  _et al._ (2018) and interference of chiral AESs
Zhao  _et al._ (2020). Despite these progresses, the identification of AESs
remains scarce, and its dynamics have remained unexplored in the presence of
disorder and dissipation.

Figure 1: (a) Schematic of the AESs (wavy lines). (b) Experimentally measured
low-bias Fano factor, $F_{S}$ and high-bias Fano factor, $F_{N}$ are plotted
as a function of magnetic field. The different transport regimes are
highlighted by the gradient colours. The dashed horizontal line correspond to
$\it{half}$ Fano.

AESs result from the repeated Andreev reflections at a QH-SC interface, where
an incident electron successively turns into a hole and back into an electron.
Quantum mechanically, AESs are fermionic modes that arise from linear
combinations of electron and hole, and propagate along the interface as shown
schematically in Fig. 1(a). The nature of the resultant fermion coming out
from the interface relies on the interference of the AESs. An electron or hole
or mixture of electron and hole can exit at the end of the interface depending
on the phase difference accumulated by the AESs. Thus, in a conventional
conductance measurement, the conductance is expected to oscillate between zero
and $4e^{2}/h$ at filling factor $\nu=2$ as a function of parameters like
chemical potential and magnetic field, giving a robust signature of AESs.
However, the inevitable presence of disorder and interface roughness
randomizes the phase. As a result, low visibility of conductance oscillations
around $2e^{2}/h$ is observed experimentally for shorter QH-SC interfaceZhao
_et al._ (2020). For a wider QH-SC interface, the oscillations are expected to
vanish due to complete phase averaging and will exhibit quantized conductance
of $2e^{2}/h$, exactly like a QH-normal metal (NM) junction. Hence, the
conductance measurement is a limited tool to explore the AESs at QH-SC
interfaces. On the contrary, as we demonstrate here, the shot noise, which
originates from the discrete nature of the current carriers Blanter and
Büttiker (2000), does not vanish due to the phase averaging and can provide
new insights into the AESs interference. The shot noise of a system is
quantified by the Fano factor ($F$), namely the ratio of measured current
noise with the Poissonian value of shot noise, $2eI$, for a current $I$
Tworzydło  _et al._ (2006); DiCarlo  _et al._ (2008); Danneau  _et al._
(2008); Kumada  _et al._ (2015); Matsuo  _et al._ (2015); Sahu  _et al._
(2019). Remarkably, we experimentally observe a large non-zero Fano factor,
close to $\it{half}$, simultaneously with a quantized conductance in a
graphene QH-SC junction, thus providing a robust and unambiguous signature of
AESs.

To this end, we perform conductance and shot noise measurement in a graphene
QH-SC junction at filling factor $\nu=2$ as a function of excitation or bias
energy ($eV_{SD}$) at several magnetic fields ($B$). As shown in Fig. 1(b),
our shot noise results can be divided into three regions; bulk dominated
transport at lower $B$, AESs dominated transport at intermediate $B$, and
normal quasiparticles dominated transport at higher $B$. In the intermediate
$B$, the Fano factor approaches a value close to half when the $eV_{SD}\ll
2\Delta$, whereas the Fano factor is close to zero for normal quasiparticle
transport for $eV_{SD}\gg 2\Delta$ [Fig. 1(b)]. In contrast the conductance
always remains $2e^{2}/h$ across the excitation energy $2\Delta$. Our
experimental observations are supported by an effective model [Fig. 1(a)],
discussed later, which predicts a Fano factor of half for a QH-SC junction for
complete phase averaging of interference of AESs. This is in stark contrast to
the zero Fano factor expected for a QH-NM junction. Moreover, we also
calculate conductance and Fano factor for a microscopic model of graphene QH-
SC junction in the presence of disorder using non-equilibrium Green’s function
(NEGF) method. We find disorder averaged conductance ($G$) with quantized
plateaus ($\nu e^{2}/h$), as in QH-NM junction, and a large non-zero quantized
Fano factor for $\nu=2$ plateau as a consequence of equal mixing of electron
and hole like states.

Figure 2: (a) The schematic of MoRe-graphene-MoRe device along with the shot
noise measurement setup. (b) The I-V response of the MoRe-graphene-MoRe device
showing supercurrent at zero magnetic field. (c) The I-V response of MoRe lead
showing supercurrent at $B=8$T. (d) The conductance of the MoRe-graphene-MoRe
device at $\nu=2$ QH plateau at $B=2$T plotted as a function of back gate
voltage. The inset shows conductance at $\nu=2$ with bias energy. The vertical
lines correspond to the superconducting gap.

To realize coupling of QH and superconductivity, we fabricated hexagonal Boron
Nitride (hBN) encapsulated graphene device edge contacted with type-II
Molybdenum Rhenium (MoRe) superconductor on a Si/SiO2 substrate. The width of
the graphene-SC interface was $\sim 2\leavevmode\nobreak\ \mathrm{\mu m}$ and
the channel length was $\sim 0.8\leavevmode\nobreak\ \mathrm{\mu m}$. Fig.
2(a) shows the schematic of the device with measurement setup. The device
fabrication and measurement setup are discussed in details in the Supplemental
Material (SM)sup . The MoRe leads show supercondcting transition at $T_{c}\sim
8.7\leavevmode\nobreak\ K$ (Fig. S5). Fig. 2(b) shows the supercurrent of the
MoRe-graphene-MoRe junction at zero magnetic field demonstrating the high
quality of graphene-MoRe interfaces (details in SM Sec. S8). The supercurrent
of the junction is killed by applying a tiny field of 100mT. Whereas, the MoRe
leads remain superconducting at large magnetic fields as shown by the I-V
characteristic in Fig. 2(c) depicting supercurrent at B=8T.

Clean QH plateaus of the MoRe-graphene-MoRe junction are observed at magnetic
field as low as 1T (Fig. S6). Fig. 2(d) shows two-probe conductance ($G$)
around $\nu=2$ filling at $B=2$T as a function of back gate voltage ($V_{BG}$)
and the conductance plateau remains very close to $2e^{2}/h$ similar to QH-NM
interface. Further, the conductance remains almost unchanged by application of
bias energy as shown in Fig. 2(d)-inset. The vertical dashed lines mark the
the proximity induced superconducting gap, $2\Delta\sim 1meV$ as can be seen
from our shot noise data discussed later. The conductance values for $\nu=2$
and $\nu=6$ filling factors remain very close to $2e^{2}/h$ and $6e^{2}/h$,
respectively, for the full range of magnetic field from 1T to 10T (Fig. S6).
These observations are in accordance with the recent experiments Lee  _et al._
(2017); Zhao  _et al._ (2020). However, there were no noticeable oscillations
at $\nu=2$ [Fig. 2(d)] as compared to observed oscillations by Zhao et al.
Zhao  _et al._ (2020).

Figure 3: (a) The measured $S_{I}$ is plotted as a function of $I_{SD}$ at the
center of $\nu$=2 plateau at several magnetic fields at 40 mK. (b)
Differential Fano factor for the $\nu=2$ ($B=1.6$T) plateau plotted as a
function of $I_{SD}$. Top axis shows the corresponding excitation voltage
($V_{SD}$). The two vertical dashed lines marks the approximate proximity
induced superconducting gap $2\Delta\sim 1\leavevmode\nobreak\ \mathrm{meV}$.
Inset shows the low bias fitting (using eq.(1)) of the same $B=1.6$T and
$\nu=2$ shot noise data giving $F_{S}\sim 0.44$. (c) Low ($eV_{SD}\ll
2\Delta$) and high bias ($eV_{SD}\gg 2\Delta$) fittings are shown for the shot
noise data at $\nu=2$ QH plateau at $B=1.4$T. (d) $F_{S}$ and $F_{N}$ for
$\nu=2$ QH plateau at B=4T as a function of temperature.

Now we present shot noise data ($S_{I}$) of the device. Fig. 3(a) shows the
shot noise data taken at the center of the $\nu=2$ QH plateau plotted as a
function of applied DC current ($I_{SD}$) at several magnetic fields.
Interestingly, the $S_{I}$ does not increase linearly for the full range of
current. Rather, a local linearity can be seen either at lower current
($<20\leavevmode\nobreak\ \mathrm{nA}$) or at larger current
($>100\leavevmode\nobreak\ \mathrm{nA}$) with the slope being significantly
larger for the former case. For better understanding this, we plot the
differential Fano factor, $F=(1/2e)(dS_{I}/dI_{SD})$, for $B=1.6$T as a
function of $I_{SD}$ in Fig. 3(b). The differential Fano factor peaks around
zero $I_{SD}$ and saturates to very small magnitude at larger $I_{SD}$. Such
transition is expected around the superconducting gap ($2\Delta\sim
1\leavevmode\nobreak\ $meV) marked by vertical dashed lines. The observed gap
is smaller than the expected BCS gap $2\Delta_{BCS}\sim 2\leavevmode\nobreak\
\mathrm{meV}$ as calculated from the critical temperature ($\sim
7-8\leavevmode\nobreak\ $K) of the bulk MoRe leads. Such reduction is observed
in past and is attributed to superconducting proximity effectTikhonov  _et
al._ (2016); Sau  _et al._ (2010). We define a low-bias Fano factor $F_{S}$
for $eV_{SD}<2\Delta$, anticipating transport via Andreev reflections.
Whereas, the high-bias Fano factor for $eV_{SD}>2\Delta$ is denoted as
$F_{N}$, which is expected to be very small since the transport happens via
normal quasi-particles. From the differential Fano factor we indeed find
$F_{S}$ to be $\sim$ 0.4 near the zero bias and $F_{N}$ to saturate around
$\sim$0.05.

More accurate estimation of $F_{S}$ and $F_{N}$ requires fitting of $S_{I}$
data with the finite-temperature expression of shot noise Blanter and Büttiker
(2000); Sahu  _et al._ (2019); Paul  _et al._ (2020),

$S_{I}=\left\\{\begin{array}[]{ll}2eI_{SD}F_{S}\,[coth(\frac{e^{*}V_{SD}}{2k_{B}T_{e}})-\frac{2k_{B}T_{e}}{e^{*}V_{SD}}],&\text{for
}|eV_{SD}|<2\Delta\\\ K+2eI_{SD}F_{N},&\text{for
}|eV_{SD}|>2\Delta\end{array}\right.\ $ (1)

Where, $V_{SD}=\frac{I_{SD}}{G}$ is the excitation voltage, $k_{B}$ is the
Boltzmann constant and $T_{e}$ ($\sim 40mK$) is the electron temperature. Fig.
3(b)-inset shows the low bias fitting of $B=1.6$T shot noise data, which gives
$F_{S}$ to be $\sim 0.44$. We extracted Fano factors for different magnetic
fields by fitting the corresponding shot noise data with Eq.1 as shown in Fig.
3(c) for $B=1.4T$. The extracted $F_{S}$ and $F_{N}$ are plotted as a function
of $B$ in Fig. 1(b), where it can be seen that $F_{N}$ remains almost constant
($\sim 0.05$) at magnetic field as low as 1.4T to high magnetic field.
However, it increases rapidly below 1.4T due to the bulk contribution. The
presence of bulk contribution below $1.4$T is further justified from the rapid
degradation of quality of $\nu=2$ QH plateau in this regime, which is shown in
Fig. S14. More interestingly, $F_{S}$ has larger value and approaches close to
$\it{half}$ in the range 1.4-1.6T and then decreases with increasing magnetic
field. In SM Sec. S5, we discuss how the presence of vortices at the QH-SC
interface can enable the normal quasi-particle transport and hence reduce the
value of Fano factor with increasing magnetic field. In Fig. 3(d), we show the
$F_{S}$ and $F_{N}$ for $\nu=2$ QH plateau with temperature at B=4T. It can be
seen that $F_{S}$ decreases with increasing temperature and becomes almost
equal to $F_{N}$ at $\sim 0.5K$ due to the normal quasiparticle transport. The
$T\sim 0.5K$ is much smaller than the $T_{C}\sim 7-8K$ of bulk MoRe lead. This
reduction of temperature is possibly due to superconducting proximity effect
and is discussed in details in SM Sec. S9. Note that, for normal transport via
QH edge states no shot noise is expected as the transmission probability
equals to one. For a junction with reduced transparency ($T_{N}<1$) shot noise
can be non-zero with $F_{N}=1-T_{N}$. From our conductance value for $\nu=2$
QH plateau, the $T_{N}\sim 0.98$. The observed $F_{N}\sim 0.05$ is close to
the theoretically expected Fano factor $\sim 0.02$. The slight deviation at
smaller magnetic fields ($1.4-1.8T$) could be due to small bulk contribution
at large applied bias ($\sim 2$meV), which does not affect $F_{S}$ measured at
very small bias energy ($<0.2$meV).

Next, we discuss an effective model for AESs as well as microscopic NEGF
calculations to understand the observation of quantized conductance with large
non-zero Fano factor in graphene QH-SC junction.

An effective model for AESs: An effective model of QH-SC interface is
schematically shown in Fig. 1(a)Khaymovich  _et al._ (2010); Zhao  _et al._
(2020). The zero-energy eigenstates of the AESs can be written in the
electron-hole basis $\\{\ket{e},\ket{h}\\}$ as
$\ket{\psi_{1}}=\alpha\ket{e}+\beta\ket{h}$ and
$\ket{\psi_{2}}=\beta^{*}\ket{e}-\alpha^{*}\ket{h}$ with wave vectors $k_{1}$
and $k_{2}$ along the edge and $|\alpha|^{2}+|\beta|^{2}=1$. By writing an
incoming electron-like state as a linear combination of AES eigenstates, the
Andreev reflection probability ($P_{\mathrm{AR}}$) is obtained as
$P_{\mathrm{AR}}=4|\alpha|^{2}|\beta|^{2}\sin^{2}(\phi/2)$, where
$\phi=(k_{1}-k_{2})L$ is the phase difference acquired between the two AES
eigenstates over the length $L$ (SM Sec. S1). When the AESs are neutral
electron-hole hybrids, i.e. $|\alpha|^{2}=|\beta|^{2}=1/2$,
$P_{\mathrm{AR}}(\phi)=\sin^{2}(\phi/2)$. For $\phi=0,2\pi$, no Cooper pair is
transmitted due to the complete reflection of all the incident electrons. In
the opposite limit, $\phi=\pi$, all the incident electrons transmit as Cooper
pair due to perfect Andreev reflection. For $\nu=2$ filling factor, the
conductance is zero in the former limit, and twice the normal state
conductance, $4e^{2}/h$, in the later. In general, the junction conductance
oscillates between these two limits if the phase difference is tuned either by
carrier density or by magnetic field. For a realistic junction, the presence
of disorder and inelastic processes at the current carrying edge Marguerite
_et al._ (2019), will introduce dephasing and decoherence, making the phase
differences random. For a complete phase averaging of the transmitted current
with uniform distribution of phase, the conductance of QH-SC junction becomes
exactly same as that of QH-NM junction i.e $2e^{2}/h$. Moreover, for a given
phase difference, $\phi$, the power spectral density of shot noise ($S_{I}$)
Blanter and Büttiker (2000) can be written as
$S_{I}(\phi)=2e^{*}I_{t}[1-P_{\mathrm{AR}}(\phi)]=8eI\sin^{2}(\phi/2)[1-\sin^{2}(\phi/2)]$,
with the Cooper pair charge, $e^{*}=2e$. For a complete phase averaging,
$S_{I}=eI$. Thus, the effective model gives a Fano factor of $\it{half}$ for
QH-SC junction with perfectly quantized conductance plateaus, in agreement of
our experimental observations.

NEGF calculations: For the NEGF calculations, as in an earlier work Sun and
Xie (2009), we consider an infinite strip of finite width having a hexagonal
graphene lattice connected to two leads one either side, as shown
schematically in Fig.4(a). Details of the model, parameters and numerical
calculations are given in SM Sec. S2. Within the NEGF formalism, the zero-
temperature two-probe conductance (in units of $e^{2}/h$) in the linear
response regime ($I=GV$) for small voltage bias $V$ is given by
$G=2\langle\left[\Tr(T_{N})+2\Tr(T_{A})\right]\rangle$, where $T_{N}$ and
$T_{A}$ are normal and Adreev transmission matrices, respectively, for each
spin species. The transmission matrices can be expressed in terms of Green’s
function and self energy of the leads (SM Sec. S2). Here $\langle..\rangle$
represents disorder averaging over many disorder realizations of
$\\{\epsilon_{i}\\}$. We define the disorder averaged transmissions matrices
as $\overline{T}_{N(A)}=\langle\Tr(T_{N(A)})\rangle$.

Figure 4: (a) System for NEGF calculations. The graphene region has a nearest
neighbour hopping $t$ and magnetic field perpendicular to its plane and the
varying marker size of the lattice represents a particular realization of
onsite disorder. (b) Conductance of the system is plotted as a function of
chemical potential for a disorder strength $W=0.5t$, flux $\Phi=0.07$ and SC
gap $\Delta=1/2750t$ in the right lead. Conductance plateaus (blue) are
realized by averaging over disorder and the plateaus appear at exactly the
same conductance (green) expected for QH-NM case. Inset: the normal and
Andreev transmission coefficient for QH-SC are plotted as a function of
excitation energy at the chemical potentials highlighted by the arrow. (c)
Calculated Fano factor is plotted as a function of $\mu$ showing the 2/3 Fano
factor at the $\nu=2$ plateau. The Fano factor vanishes for QH-NM case.

Fig. 4(b) shows the disorder averaged conductance as a function of chemical
potential ($\mu$) with SC ($\Delta=1/2750t$ with nearest neighbour hopping $t$
in graphene) and normal ($\Delta=0$) right lead. The conductance for both the
cases show the same QH plateaus, as expected from the effective edge model.
But, from Fig. 4(b)-inset we can see that only Andreev transmission takes part
in the transport, i.e, $\overline{T}_{N}=0$ and $\overline{T}_{A}\neq 0$,
within the superconducting gap for the SC lead. Following Blanter et
al.Blanter and Büttiker (2000), the shot noise for normal-superconductor
junction is given by $S=(16e^{3}V/h)\Tr[T_{A}(1-T_{A})]$, such that the Fano
factor is $F=2\langle\Tr[T_{A}(1-T_{A})]\rangle/\langle\Tr[T_{A}]\rangle$. The
numerically computed Fano factor for the graphene QH-SC junction is shown in
Fig. 4(c) and compared with that of QH-NM system. As evident, Fano factor for
the conductance plateaus for QH-SC is non-zero, in contrast to the QH-NM
system. In particular, $F=2/3$ for $\nu=2$ plateau, unlike $F=1/2$ obtained
from the edge model. This difference in values of $F$ from the NEGF
calculations and the effective model can be traced back to the difference of
the distribution $\Tr T_{A}$ over disorder realizations for the two models. As
discussed in detail in SM Sec. S3, in case of NEGF calculations, we find that
$\Tr T_{A}$ is uniformly distributed for $\nu=2$, thus producing $F=2/3$. On
the contrary, in the effective model of AES in the preceding section, we
assumed uniform distribution of $\phi$ as opposed to the uniform distribution
of $\sin^{2}(\phi/2)\propto T_{A}$. This difference between NEGF calculation
and effective model does not affect the conductance.

To conclude, our result of close to half Fano factor along with quantized
conductance for Andreev reflection dominated transport is an unambiguous
signature of the existence of AESs at the QH-SC interface. The closer
agreement of experimental results with the scenario of uniform phase averaging
gives an indication of the crucial involvements both static disorder and
inelastic processes along the current-carrying edge even at very low
temperature Marguerite  _et al._ (2019). Getting insights into such intriguing
dynamical processes for AESs would be of great importance for the eventual
realization of novel excitations, e.g. Majorana and para-fermions, in various
QH insulator-superconductor hybrid systems Qi  _et al._ (2010); Mong  _et al._
(2014); Clarke  _et al._ (2014); Alicea and Fendley (2016); San-Jose  _et al._
(2015).

A.D. acknowledges supports from the MHRD, Govt. of India under STARS research
funding (STARS/APR2019/PS/156/FS), and also thanks Swarnajayanti Fellowship of
the DST/SJF/PSA-03/2018-19.

## References

  * Beenakker (2015) C. Beenakker, Reviews of Modern Physics 87, 1037 (2015).
  * Bishara and Nayak (2007) W. Bishara and C. Nayak, Physical review letters 99, 066401 (2007).
  * Nayak  _et al._ (2008) C. Nayak, S. H. Simon, A. Stern, M. Freedman, and S. D. Sarma, Reviews of Modern Physics 80, 1083 (2008).
  * Mong  _et al._ (2014) R. S. Mong, D. J. Clarke, J. Alicea, N. H. Lindner, P. Fendley, C. Nayak, Y. Oreg, A. Stern, E. Berg, K. Shtengel, _et al._ , Physical Review X 4, 011036 (2014).
  * Qi  _et al._ (2010) X.-L. Qi, T. L. Hughes, and S.-C. Zhang, Physical Review B 82, 184516 (2010).
  * Fisher (1994) M. P. Fisher, Physical Review B 49, 14550 (1994).
  * San-Jose  _et al._ (2015) P. San-Jose, J. L. Lado, R. Aguado, F. Guinea, and J. Fernández-Rossier, Phys. Rev. X 5, 041042 (2015).
  * Rahmani and Franz (2019) A. Rahmani and M. Franz, Reports on Progress in Physics 82, 084501 (2019).
  * Manousakis  _et al._ (2020) J. Manousakis, C. Wille, A. Altland, R. Egger, K. Flensberg, and F. Hassler, Physical Review Letters 124, 096801 (2020).
  * Motome and Nasu (2020) Y. Motome and J. Nasu, Journal of the Physical Society of Japan 89, 012002 (2020).
  * Kayyalha  _et al._ (2020) M. Kayyalha, D. Xiao, R. Zhang, J. Shin, J. Jiang, F. Wang, Y.-F. Zhao, R. Xiao, L. Zhang, K. M. Fijalkowski, _et al._ , Science 367, 64 (2020).
  * Rickhaus  _et al._ (2012) P. Rickhaus, M. Weiss, L. Marot, and C. Schonenberger, Nano letters 12, 1942 (2012).
  * Shalom  _et al._ (2016) M. B. Shalom, M. Zhu, V. FalKo, A. Mishchenko, A. Kretinin, K. Novoselov, C. Woods, K. Watanabe, T. Taniguchi, A. Geim, _et al._ , Nature Physics 12, 318 (2016).
  * Amet  _et al._ (2016) F. Amet, C. T. Ke, I. V. Borzenets, J. Wang, K. Watanabe, T. Taniguchi, R. S. Deacon, M. Yamamoto, Y. Bomze, S. Tarucha, _et al._ , Science 352, 966 (2016).
  * Park  _et al._ (2017) G.-H. Park, M. Kim, K. Watanabe, T. Taniguchi, and H.-J. Lee, Scientific reports 7, 1 (2017).
  * Lee  _et al._ (2017) G.-H. Lee, K.-F. Huang, D. K. Efetov, D. S. Wei, S. Hart, T. Taniguchi, K. Watanabe, A. Yacoby, and P. Kim, Nature Physics 13, 693 (2017).
  * Sahu  _et al._ (2018) M. R. Sahu, X. Liu, A. K. Paul, S. Das, P. Raychaudhuri, J. Jain, and A. Das, Physical review letters 121, 086809 (2018).
  * Seredinski  _et al._ (2019) A. Seredinski, A. W. Draelos, E. G. Arnault, M.-T. Wei, H. Li, T. Fleming, K. Watanabe, T. Taniguchi, F. Amet, and G. Finkelstein, Science advances 5, eaaw8693 (2019).
  * Zhao  _et al._ (2020) L. Zhao, E. G. Arnault, A. Bondarev, A. Seredinski, T. F. Larson, A. W. Draelos, H. Li, K. Watanabe, T. Taniguchi, F. Amet, _et al._ , Nature Physics , 1 (2020).
  * Sahu  _et al._ (2019) M. R. Sahu, A. K. Paul, A. Soori, K. Watanabe, T. Taniguchi, S. Mukerjee, and A. Das, Physical Review B 100, 235414 (2019).
  * Lee and Lee (2018) G.-H. Lee and H.-J. Lee, Reports on Progress in Physics 81, 056502 (2018).
  * Finkelstein and Amet (2017) G. Finkelstein and F. Amet, Nature Physics 13, 625 (2017).
  * Dvir  _et al._ (2021) T. Dvir, A. Zalic, E. H. Fyhn, M. Amundsen, T. Taniguchi, K. Watanabe, J. Linder, and H. Steinberg, Physical Review B 103, 115401 (2021).
  * Blanter and Büttiker (2000) Y. M. Blanter and M. Büttiker, Physics reports 336, 1 (2000).
  * Tworzydło  _et al._ (2006) J. Tworzydło, B. Trauzettel, M. Titov, A. Rycerz, and C. W. Beenakker, Physical review letters 96, 246802 (2006).
  * DiCarlo  _et al._ (2008) L. DiCarlo, J. Williams, Y. Zhang, D. McClure, and C. Marcus, Physical review letters 100, 156801 (2008).
  * Danneau  _et al._ (2008) R. Danneau, F. Wu, M. Craciun, S. Russo, M. Tomi, J. Salmilehto, A. Morpurgo, and P. J. Hakonen, Physical review letters 100, 196802 (2008).
  * Kumada  _et al._ (2015) N. Kumada, F. Parmentier, H. Hibino, D. Glattli, and P. Roulleau, Nature communications 6, 1 (2015).
  * Matsuo  _et al._ (2015) S. Matsuo, S. Takeshita, T. Tanaka, S. Nakaharai, K. Tsukagoshi, T. Moriyama, T. Ono, and K. Kobayashi, Nature communications 6, 1 (2015).
  * (30) See supplemental material for additional data on device fabrication and characterization, measurement techniques, and details of theoretical models used. The supplemental material includes additional references MacKinnon and Kramer (1983); Lee and Fisher (1981); Verges (1999); Jehl  _et al._ (1999); Komatsu  _et al._ (2012); Miyoshi  _et al._ (2005); Zomer  _et al._ (2014); Pizzocchero  _et al._ (2016); Purdie  _et al._ (2018); Ronen  _et al._ (2016); Srivastav  _et al._ (2019); Heersche  _et al._ (2007); Calado  _et al._ (2015); Schmidt  _et al._ (2018); Zhu  _et al._ (2018); Borzenets  _et al._ (2013); Rycerz  _et al._ (2009) .
  * Tikhonov  _et al._ (2016) E. Tikhonov, D. Shovkun, M. Snelder, M. Stehno, Y. Huang, M. Golden, A. Golubov, A. Brinkman, and V. Khrapai, Physical review letters 117, 147001 (2016).
  * Sau  _et al._ (2010) J. D. Sau, R. M. Lutchyn, S. Tewari, and S. D. Sarma, Physical Review B 82, 094522 (2010).
  * Paul  _et al._ (2020) A. K. Paul, M. R. Sahu, C. Kumar, K. Watanabe, T. Taniguchi, and A. Das, Communications Physics 3, 1 (2020).
  * Khaymovich  _et al._ (2010) I. Khaymovich, N. Chtchelkatchev, I. Shereshevskii, and A. Mel’nikov, EPL (Europhysics Letters) 91, 17005 (2010).
  * Marguerite  _et al._ (2019) A. Marguerite, J. Birkbeck, A. Aharon-Steinberg, D. Halbertal, K. Bagani, I. Marcus, Y. Myasoedov, A. Geim, D. Perello, and E. Zeldov, Nature 576, E6 (2019).
  * Sun and Xie (2009) Q.-f. Sun and X. Xie, Journal of Physics: Condensed Matter 21, 344204 (2009).
  * Clarke  _et al._ (2014) D. J. Clarke, J. Alicea, and K. Shtengel, Nature Physics 10, 877 (2014), arXiv:1312.6123 [cond-mat.str-el] .
  * Alicea and Fendley (2016) J. Alicea and P. Fendley, Annual Review of Condensed Matter Physics 7, 119 (2016).
  * MacKinnon and Kramer (1983) A. MacKinnon and B. Kramer, Zeitschrift fur Physik B Condensed Matter 53, 1 (1983).
  * Lee and Fisher (1981) P. A. Lee and D. S. Fisher, Phys. Rev. Lett. 47, 882 (1981).
  * Verges (1999) J. A. Verges, Computer Physics Communications 118, 71 (1999).
  * Jehl  _et al._ (1999) X. Jehl, P. Payet-Burin, C. Baraduc, R. Calemczuk, and M. Sanquer, Physical review letters 83, 1660 (1999).
  * Komatsu  _et al._ (2012) K. Komatsu, C. Li, S. Autier-Laurent, H. Bouchiat, and S. Guéron, Physical Review B 86, 115412 (2012).
  * Miyoshi  _et al._ (2005) Y. Miyoshi, Y. Bugoslavsky, and L. Cohen, Physical Review B 72, 012502 (2005).
  * Zomer  _et al._ (2014) P. Zomer, M. Guimarães, J. Brant, N. Tombros, and B. Van Wees, Applied Physics Letters 105, 013101 (2014).
  * Pizzocchero  _et al._ (2016) F. Pizzocchero, L. Gammelgaard, B. S. Jessen, J. M. Caridad, L. Wang, J. Hone, P. Bøggild, and T. J. Booth, Nature communications 7, 1 (2016).
  * Purdie  _et al._ (2018) D. Purdie, N. Pugno, T. Taniguchi, K. Watanabe, A. Ferrari, and A. Lombardo, Nature communications 9, 1 (2018).
  * Ronen  _et al._ (2016) Y. Ronen, Y. Cohen, J.-H. Kang, A. Haim, M.-T. Rieder, M. Heiblum, D. Mahalu, and H. Shtrikman, Proceedings of the National Academy of Sciences 113, 1743 (2016).
  * Srivastav  _et al._ (2019) S. K. Srivastav, M. R. Sahu, K. Watanabe, T. Taniguchi, S. Banerjee, and A. Das, Science advances 5, eaaw5798 (2019).
  * Heersche  _et al._ (2007) H. B. Heersche, P. Jarillo-Herrero, J. B. Oostinga, L. M. Vandersypen, and A. F. Morpurgo, Nature 446, 56 (2007).
  * Calado  _et al._ (2015) V. E. Calado, S. Goswami, G. Nanda, M. Diez, A. R. Akhmerov, K. Watanabe, T. Taniguchi, T. M. Klapwijk, and L. M. Vandersypen, Nature nanotechnology 10, 761 (2015).
  * Schmidt  _et al._ (2018) F. E. Schmidt, M. D. Jenkins, K. Watanabe, T. Taniguchi, and G. A. Steele, Nature communications 9, 1 (2018).
  * Zhu  _et al._ (2018) M. Zhu, M. B. Shalom, A. Mishchsenko, V. Falko, K. Novoselov, and A. Geim, Nanoscale 10, 3020 (2018).
  * Borzenets  _et al._ (2013) I. Borzenets, U. Coskun, H. Mebrahtu, Y. V. Bomze, A. Smirnov, and G. Finkelstein, Physical review letters 111, 027001 (2013).
  * Rycerz  _et al._ (2009) A. Rycerz, P. Recher, and M. Wimmer, Physical Review B 80, 125417 (2009).

See pages - of SM.pdf


# ABO: Dataset and Benchmarks for Real-World 3D Object Understanding

Jasmine Collins1, Shubham Goel1, Kenan Deng2, Achleshwar Luthra3, Leon Xu1,2,
Erhan Gundogdu2, Xi Zhang2, Tomas F. Yago Vicente2, Thomas Dideriksen2,
Himanshu Arora2, Matthieu Guillaumin2, and Jitendra Malik1

1 UC Berkeley, 2 Amazon, 3 BITS Pilani

###### Abstract

We introduce Amazon Berkeley Objects (ABO), a new large-scale dataset designed
to help bridge the gap between real and virtual 3D worlds. ABO contains
product catalog images, metadata, and artist-created 3D models with complex
geometries and physically-based materials that correspond to real, household
objects. We derive challenging benchmarks that exploit the unique properties
of ABO and measure the current limits of the state-of-the-art on three open
problems for real-world 3D object understanding: single-view 3D
reconstruction, material estimation, and cross-domain multi-view object
retrieval.

Figure 1: ABO is a dataset of product images and realistic, high-resolution,
physically-based 3D models of household objects. We use ABO to benchmark the
performance of state-of-the-art methods on a variety of realistic object
understanding tasks.

##  1 Introduction

Progress in 2D image recognition has been driven by large-scale datasets [37,
15, 44, 58, 26]. The ease of collecting 2D annotations (such as class labels
or segmentation masks) has led to the large scale of these diverse, in-the-
wild datasets, which in turn has enabled the development of 2D computer vision
systems that work in the real world. Theoretically, progress in 3D computer
vision should follow from equally large-scale datasets of 3D objects. However,
collecting large amounts of high-quality 3D annotations (such as voxels or
meshes) for individual real-world objects poses a challenge. One way around
the challenging problem of getting 3D annotations for real images is to focus
only on synthetic, computer-aided design (CAD) models [10, 72, 35]. This has
the advantage that the data is large in scale (as there are many 3D CAD models
available for download online) but many of the models are low quality or
untextured and do not exist in the real world. This has led to a variety of 3D
reconstruction methods that work well on clear-background renderings of
synthetic objects [13, 24, 67, 48] but do not necessarily generalize to real
images, new categories, or more complex object geometries [60, 5, 6].

To enable better real-world transfer, another class of 3D datasets aims to
link existing 3D models with real-world images [66, 65]. These datasets find
the closest matching CAD models for the objects in an image and have human
annotators align the pose of each model to best match the image. While this
has enabled the evaluation of 3D reconstruction methods in-the-wild, the shape
(and thus pose) matches are approximate. Further, because this approach relies
on matching CAD models to images, it inherits the limitations of the existing
CAD model datasets (_i.e_. poor coverage of real-world objects, basic
geometries and textures).

The IKEA [42] and Pix3D [59] datasets sought to improve upon this by
annotating real images with exact, pixel-aligned 3D models. The exact nature
of such datasets has allowed them to be used as training data for single-view
reconstruction [21] and has bridged some of the synthetic-to-real domain gap.
However, the size of the datasets are relatively small (90 and 395 unique 3D
models, respectively), likely due to the difficulty of finding images that
exactly match 3D models. Further, the larger of the two datasets [59] only
contains 9 categories of objects. The provided 3D models are also untextured,
thus the annotations in these datasets are typically used for shape or pose-
based tasks, rather than tasks such as material prediction.

Rather than trying to match images to synthetic 3D models, another approach to
collecting 3D datasets is to start with real images (or video) and reconstruct
the scene by classical reconstruction techniques such as structure from
motion, multi-view stereo and texture mapping [12, 57, 56]. The benefit of
these methods is that the reconstructed geometry faithfully represents an
object of the real world. However, the collection process requires a great
deal of manual effort and thus datasets of this nature tend to also be quite
small (398, 125, and 1032 unique 3D models, respectively). The objects are
also typically imaged in a controlled lab setting and do not have
corresponding real images of the object “in context”. Further, included
textured surfaces are assumed to be Lambertian and thus do not display
realistic reflectance properties.

Motivated by the lack of large-scale datasets with realistic 3D objects from a
diverse set of categories and corresponding real-world multi-view images, we
introduce Amazon Berkeley Objects (ABO). This dataset is derived from
Amazon.com product listings, and as a result, contains imagery and 3D models
that correspond to modern, real-world, household items. Overall, ABO contains
147,702 product listings associated with 398,212 unique catalog images, and up
to 18 unique metadata attributes (category, color, material, weight,
dimensions, etc.) per product. ABO also includes “360º View” turntable-style
images for $8,222$ products and 7,953 products with corresponding artist-
designed 3D meshes. In contrast to existing 3D computer vision datasets, the
3D models in ABO have complex geometries and high-resolution, physically-based
materials that allow for photorealistic rendering. A sample of the kinds of
real-world images associated with a 3D model from ABO can be found in Figure
ABO: Dataset and Benchmarks for Real-World 3D Object Understanding, and sample
metadata attributes are shown in Figure 4. The dataset is released under CC
BY-NC 4.0 license and can be downloaded at https://amazon-berkeley-
objects.s3.amazonaws.com/index.html.

To facilitate future research, we benchmark the performance of various methods
on three computer vision tasks that can benefit from more realistic 3D
datasets: (i) single-view shape reconstruction, where we measure the domain
gap for networks trained on synthetic objects, (ii) material estimation, where
we introduce a baseline for spatially-varying BRDF from single- and multi-view
images of complex real world objects, and (iii) image-based multi-view object
retrieval, where we leverage the 3D nature of ABO to evaluate the robustness
of deep metric learning algorithms to object viewpoint and scenes.

Dataset | # Models | # Classes | Real images | Full 3D | PBR
---|---|---|---|---|---
ShapeNet [10] | 51.3K | 55 | ✗ | ✓ | ✗
3D-Future [19] | 16.6K | 8 | ✗ | ✓ | ✗
Google Scans [56] | 1K | - | ✗ | ✓ | ✗
CO3D [55] | 18.6K | 50 | ✓ | ✗ | ✗
IKEA [43] | 219 | 11 | ✓ | ✓ | ✗
Pix3D [59] | 395 | 9 | ✓ | ✓ | ✗
PhotoShape [53] | 5.8K | 1 | ✗ | ✓ | ✓
ABO (Ours) | 8K | 63 | ✓ | ✓ | ✓
Table 1: A comparison of the 3D models in ABO and other commonly used object-
centric 3D datasets. ABO contains nearly 8K 3D models with physically-based
rendering (PBR) materials and corresponding real-world catalog images.  Figure
2: Posed 3D models in catalog images. We use instance masks to automatically
generate 6-DOF pose annotations.

Figure 3: Sample catalog images and attributes that accompany ABO objects.
Each object has up to 18 attribute annotations.

Figure 4: 3D model categories. Each category is also mapped to a synset in the WordNet hierarchy. Note the y-axis is in log scale. Benchmark | Domain | Classes | Instances |  | Images | Structure | Recall@1 |
---|---|---|---|---|---|---|---|---
train | val | test |  | train | val | test-target | test-query |
CUB-200-2011 | Birds | 200 | - | - | - |  | 5994 | 0 | - | 5794 | 15 parts | 79.2% | [30]
Cars-196 | Cars | 196 | - | - | - |  | 8144 | 0 | - | 8041 | - | 94.8% | [30]
In-Shop | Clothes | 25 | 3997 | 0 | 3985 |  | 25882 | 0 | 12612 | 14218 | Landmarks, poses, masks | 92.6% | [33]
SOP | Ebay | 12 | 11318 | 0 | 11316 |  | 59551 | 0 | - | 60502 | - | 84.2% | [30]
ABO (MVR) | Amazon | 562 | 49066 | 854 | 836 |  | 298840 | 26235 | 4313 | 23328 | Subset with 3D models | 30.0% |
Table 2: Common image retrieval benchmarks for deep metric learning and their
statistics. Our proposed multi-view retrieval (MVR) benchmark based on ABO is
significantly larger, more diverse and challenging than existing benchmarks,
and exploits 3D models.

##  2 Related Work

3D Object Datasets ShapeNet [10] is a large-scale database of synthetic 3D CAD
models commonly used for training single- and multi-view reconstruction
models. IKEA Objects [43] and Pix3D [59] are image collections with 2D-3D
alignment between CAD models and real images, however these images are limited
to objects for which there is an exact CAD model match. Similarly, Pascal3D+
[66] and ObjectNet3D [65] provide 2D-3D alignment for images and provide more
instances and categories, however the 3D annotations are only approximate
matches. The Object Scans dataset [12] and Objectron [3] are both video
datasets that have the camera operator walk around various objects, but are
limited in the number of categories represented. CO3D [55] also offers videos
of common objects from 50 different categories, however they do not provide
full 3D mesh reconstructions.

Existing 3D datasets typically assume very simplistic texture models that are
not physically realistic. To improve on this, PhotoShapes [53] augmented
ShapeNet CAD models by automatically mapping spatially varying (SV-)
bidirectional reflectance distribution functions (BRDFs) to meshes, yet the
dataset consists only of chairs. The works in [17, 20] provide high-quality
SV-BRDF maps, but only for planar surfaces. The dataset used in [32] contains
only homogenous BRDFs for various objects. [41] and [7] introduce datasets
containing full SV-BRDFs, however their models are procedurally generated
shapes that do not correspond to real objects. In contrast, ABO provides
shapes and SV-BRDFs created by professional artists for real-life objects that
can be directly used for photorealistic rendering.

Table 1 compares the 3D subset of ABO with other commonly used 3D datasets in
terms of size (number of objects and classes) and properties such as the
presence of real images, full 3D meshes and physically-based rendering (PBR)
materials. ABO is the only dataset that contains all of these properties and
is much more diverse in number of categories than existing 3D datasets.

3D Shape Reconstruction Recent methods for single-view 3D reconstruction
differ mainly in the type of supervision and 3D representation used, whether
it be voxels, point clouds, meshes, or implicit functions. Methods that
require full shape supervision in the single-view [18, 71, 59, 48, 22] and
multi-view [31, 13, 67] case are often trained using ShapeNet. There are other
approaches that use more natural forms of multi-view supervision such as
images, depth maps, and silhouettes [68, 64, 31, 61], with known cameras. Of
course, multi-view 3D reconstruction has long been studied with classical
computer vision techniques [27] like multi-view stereo and visual hull
reconstruction. Learning-based methods are typically trained in a category-
specific way and evaluated on new instances from the same category. Out of the
works mentioned, only [71] claims to be category-agnostic. In this work we are
interested in how well these ShapeNet-trained networks [13, 71, 48, 22]
generalize to more realistic objects.

Figure 5: Qualitative 3D reconstruction results for R2N2, Occupancy Networks,
GenRe, and Mesh-RCNN on ABO. All methods are pre-trained on ShapeNet and show
a decrease in performance on objects from ABO.

Material Estimation Several works have focused on modeling object appearance
from a single image, however realistic datasets available for this task are
relatively scarce and small in size. [39] use two networks to estimate a
homogeneous BRDF and an SV-BRDF of a flat surface from a single image, using a
self-augmentation scheme to alleviate the need for a large training set.
However, their work is limited to a specific family of materials, and each
separate material requires another trained network. [69] extend the idea of
self-augmentation to train with unlabeled data, but their work is limited by
the same constraints. [16] use a modified U-Net and rendering loss to predict
the SV-BRDFs of flash-lit photographs consisting of only a flat surface. To
enable prediction for arbitrary shapes, [41] propose a cascaded CNN
architecture with a single encoder and separate decoder for each SV-BRDF
parameter. While the method achieves good results on semi-uncontrolled
lighting environments, it requires using the intermediate bounces of global
illumination rendering as supervision. More recent works have turned towards
using multiple images to improve SV-BRDF estimation, but still only with
simplistic object geometries. For instance, [17] and [20] use multiple input
images with a flash lit light source, but only for a single planar surface.
[7] and [8] both use procedurally generated shapes to estimate SV-BRDFs from
multi-view images. ABO addresses the lack of sufficient realistic data for
material estimation, and in this work we propose a simple baseline method that
can estimate materials from single or multi-view images of complex, real-world
shapes.

2D/3D Image Retrieval Learning to represent 3D shapes and natural images of
products in a single embedding space has been tackled by [40]. They consider
various relevant tasks, including cross-view image retrieval, shape-based
image retrieval and image-based shape retrieval, but all are inherently
constrained by the limitations of ShapeNet [10] (cross-view image retrieval is
only considered for chairs and cars). [36] introduced 3D object
representations for fine-grained recognition and a dataset of cars with real-
world 2D imagery (CARS-196), which is now widely used for deep metric learning
(DML) evaluation. Likewise, other datasets for DML focus on instances/fine
categories of few object types, such as birds [62], clothes [45], or a few
object categories [52].

Due to the limited diversity and the similar nature of query and target images
in existing retrieval benchmarks, the performance of state-of-the-art DML
algorithms are near saturation. Moreover, since these datasets come with
little structure, the opportunities to analyze failure cases and improve
algorithms are limited. Motivated by this, we derive a challenging large-scale
benchmark dataset from ABO with hundreds of diverse categories and a proper
validation set. We also leverage the 3D nature of ABO to measure and improve
the robustness of representations with respect to changes in viewpoint and
scene. A comparison of ABO and existing benchmarks for DML can be found in
Table 2.

##  3 The ABO Dataset

Dataset Properties The ABO dataset originates from worldwide product listings,
metadata, images and 3D models provided by Amazon.com. This data consists of
147,702 listings of products from 576 product types sold by various Amazon-
owned stores and websites (e.g. Amazon, PrimeNow, Whole Foods). Each listing
is identified by an item ID and is provided with structured metadata
corresponding to information that is publicly available on the listing’s main
webpage (such as product type, material, color, and dimensions) as well as the
media available for that product. This includes $398,212$ high-resolution
catalog images, and, when available, the turntable images that are used for
the “360º View” feature that shows the product imaged at 5º or 15º azimuth
intervals ($8,222$ products).

|  Chamfer Distance ($\downarrow$) |  |  Absolute Normal Consistency ($\uparrow$)
---|---|---|---
| bench | chair | couch | cabinet | lamp | table |  | bench | chair | couch | cabinet | lamp | table
3D R2N2 [13] | 2.46/0.85 | 1.46/0.77 | 1.15/0.59 | 1.88/0.25 | 3.79/2.02 | 2.83/0.66 |  | 0.51/0.55 | 0.59/0.61 | 0.57/0.62 | 0.53/0.67 | 0.51/0.54 | 0.51/0.65
Occ Nets [48] | 1.72/0.51 | 0.72/0.39 | 0.86/0.30 | 0.80/0.23 | 2.53/1.66 | 1.79/0.41 |  | 0.66/0.68 | 0.67/0.76 | 0.70/0.77 | 0.71/0.77 | 0.65/0.69 | 0.67/0.78
GenRe [71] | 1.54/2.86 | 0.89/0.79 | 1.08/2.18 | 1.40/2.03 | 3.72/2.47 | 2.26/2.37 |  | 0.63/0.56 | 0.69/0.67 | 0.66/0.60 | 0.62/0.59 | 0.59/0.57 | 0.61/0.59
Mesh R-CNN [22] | 1.05/0.09 | 0.78/0.13 | 0.45/0.10 | 0.80/0.11 | 1.97/0.24 | 1.15/0.12 |  | 0.62/0.65 | 0.62/0.70 | 0.62/0.72 | 0.65/0.74 | 0.57/0.66 | 0.62/0.74
Table 3: Single-view 3D reconstruction generalization from ShapeNet to ABO.
Chamfer distance and absolute normal consistency of predictions made on ABO
objects from common ShapeNet classes. We report the same metrics for ShapeNet
objects (denoted in  gray), following the same evaluation protocol. All
methods, with the exception of GenRe, are trained on all of the ShapeNet
categories listed.

3D Models ABO also includes $7,953$ artist-created high-quality 3D models in
glTF 2.0 format. The 3D models are oriented in a canonical coordinate system
where the “front” (when well defined) of all objects are aligned and each have
a scale corresponding to real world units. To enable these meshes to easily be
used for comparison with existing methods trained on 3D datasets such as
ShapeNet, we have collected category annotations for each 3D model and mapped
them to noun synsets under the WordNet [49] taxonomy. Figure 4 shows a
histogram of the 3D model categories.

Catalog Image Pose Annotations We additionally provide 6-DOF pose annotations
for $6,334$ of the catalog images. To achieve this, we develop an automated
pipeline for pose estimation based on the knowledge of the 3D model in the
image, off-the-shelf instance masks [28, 34], and differentiable rendering.
For each mask M, we estimate $\textbf{R}\in SO(3)$ and
$\textbf{T}\in\mathbb{R}^{3}$ such that the following silhouette loss is
minimized

$\textbf{R}^{*},\textbf{T}^{*}=\operatorname*{argmin}_{\textbf{R},\textbf{T}}\left\lVert
DR(\textbf{R},\textbf{T})-\textbf{M}\right\rVert$

where $DR(\cdot)$ is a differentiable renderer implemented in PyTorch3D [54].
Examples of results from this approach can be found in Figure 2. Unlike
previous approaches to CAD-to-image alignment [65, 59] that use human
annotators in-the-loop to provide pose or correspondences, our approach is
fully automatic except for a final human verification step.

Material Estimation Dataset To perform material estimation from images, we use
the Disney [9] base color, metallic, roughness parameterization given in glTF
2.0 specification [25]. We render 512x512 images from 91 camera positions
along an upper icosphere of the object with a $60^{\circ}$ field-of-view using
Blender’s [14] Cycles path-tracer. To ensure diverse realistic lighting
conditions and backgrounds, we illuminate the scene using 3 random environment
maps out of 108 indoor HDRIs [23]. For these rendered images, we generate the
corresponding ground truth base color, metallicness, roughness, and normal
maps along with the object depth map and segmentation mask. The resulting
dataset consists of 2.1 million rendered images and corresponding camera
intrinsics and extrinsics.

##  4 Experiments

###  4.1 Evaluating Single-View 3D Reconstruction

As existing methods are largely trained in a fully supervised manner using
ShapeNet [10], we are interested in how well they will transfer to more real-
world objects. To measure how well these models transfer to real object
instances, we evaluate the performance of a variety of these methods on
objects from ABO. Specifically we evaluate 3D-R2N2 [13], GenRe [71], Occupancy
Networks [48], and Mesh R-CNN [22] pre-trained on ShapeNet. We selected these
methods because they capture some of the top-performing single-view 3D
reconstruction methods from the past few years and are varied in the type of
3D representation that they use (voxels in [13], spherical maps in [71],
implicit functions in [48], and meshes in [22]) and the coordinate system used
(canonical vs. view-space). While all the models we consider are pre-trained
on ShapeNet, GenRe trains on a different set of classes and takes as input a
silhouette mask at train and test time.

To study this question (irrespective of the question of cross-category
generalization), we consider only the subset of ABO models objects that fall
into ShapeNet training categories. Out of the 63 categories in ABO with 3D
models, we consider 6 classes that intersect with commonly used ShapeNet
classes, capturing 4,170 of the 7,953 3D models. Some common ShapeNet classes,
such as “airplane”, have no matching ABO category; similarly, some categories
in ABO like “air conditioner” and “weights” do not map well to ShapeNet
classes.

For this experiment, we render a dataset (distinct from the ABO Material
Estimation Dataset) of objects on a blank background from a similar
distribution of viewpoints as in the rendered ShapeNet training set. We render
30 viewpoints of each mesh using Blender [14], each with a $40^{\circ}$ field-
of-view and such that the entire object is visible. Camera azimuth and
elevation are sampled uniformly on the surface of a unit sphere with a
$-10^{\circ}$ lower limit on elevations to avoid uncommon bottom views.

GenRe and Mesh-RCNN make their predictions in “view-space” (i.e. pose aligned
to the image view), whereas R2N2 and Occupancy Networks perform predictions in
canonical space (predictions are made in the same category-specific, canonical
pose despite the pose of the object in an image). For each method we evaluate
Chamfer Distance and Absolute Normal Consistency and largely follow the
evaluation protocol of [22].

Results A quantitative comparison of the four methods we considered on ABO
objects can be found in Table 3. We also re-evaluated each method’s
predictions on the ShapeNet test set from R2N2 [13] with our evaluation
protocol and report those metrics. We observe that Mesh R-CNN [22] outperforms
all other methods across the board on both ABO and ShapeNet in terms of
Chamfer Distance, whereas Occupancy Networks performs the best in terms of
Absolute Normal Consistency. As can be seen, there is a large performance gap
between all ShapeNet and ABO predictions. This suggests that shapes and
textures from ABO, while derived from the same categories but from the real
world, are out of distribution and more challenging for the models trained on
ShapeNet. Further, we notice that the lamp category has a particularly large
performance drop from ShapeNet to ABO. Qualitative results suggest that this
is likely due to the difficulty in reconstructing thin structures. We
highlight some qualitative results in Figure 5, including one particularly
challenging lamp instance.

Figure 6: Qualitative material estimation results for single-view (SV-net) and
multi-view (MV-net) networks. We show estimated SV-BRDF properties (base
color, roughness, metallicness, surface normals) for each input view of an
object compared to the ground truth.

###  4.2 Material Prediction

To date, there are not many available datasets tailored to the material
prediction task. Most publicly available datasets with large collections of 3D
objects [10, 12, 19] do not contain physically-accurate reflectance parameters
that can be used for physically-based rendering to generate photorealistic
images. Datasets like PhotoShape [53] do contain such parameters but are
limited to a single category. In contrast, the realistic 3D models in ABO are
artist-created and have highly varied shapes and SV-BRDFs. We leverage this
unique property to derive a benchmark for material prediction with large
amounts of photorealistic synthetic data. We also present a simple baseline
approach for both single- and multi-view material estimation of complex
geometries.

Method To evaluate single-view and multi-view material prediction and
establish a baseline approach, we use a U-Net-based model with a ResNet-34
backbone to estimate SV-BRDFs from a single viewpoint. The U-Net has a common
encoder that takes an RGB image as input and has a multi-head decoder to
output each component of the SV-BRDF separately. Inspired by recent networks
in [7, 17], we align images from multiple viewpoints by projection using depth
maps, and bundle the original image and projected image pairs as input data to
enable an analogous approach for the multi-view network. We reuse the single-
view architecture for the multi-view network and use global max pooling to
handle an arbitrary number of input images. Similar to [16], we utilize a
differentiable rendering layer to render the flash illuminated ground truth
and compare it to similarly rendered images from our predictions to better
regularize the network and guide the training process. Ground truth material
maps are used for direct supervision.

Our model takes as input 256x256 rendered images. For training, we randomly
subsample 40 views on the icosphere for each object. In the case of the multi-
view network, for each reference view we select its immediate 4 adjacent views
as neighboring views. We use mean squared error as the loss function for base
color, roughness, metallicness, surface normal and render losses. Each network
is trained for 17 epochs using the AdamW optimizer [47] with a learning rate
of 1e-3 and weight decay of 1e-4.

Figure 7: Qualitative multi-view material estimation results on real catalog
images. Each of the multiple views is aligned to the reference view using the
catalog image pose annotations.

Results Results for the single-view network (SV-net) and multi-view network
(MV-net) can be found in Table 4. The multi-view network has better
performance compared to single-view network in terms of the base color,
roughness, metallicness, and surface normal prediction tasks. The multi-view
network is especially better at predicting properties that affect view-
dependent specular components like roughness and metallicness.

We also run an ablation study on our multi-view network without using 3D
structure to align neighboring views to reference view (denoted as _MV-net: no
projection_). First, we observe that even without 3D structure-based
alignment, the network still outperforms the single-view network on roughness
and metallic predictions. Comparing to the multi-view network, which uses 3D
structure-based alignment, we can see structure information leads to better
performance for all parameters. We show some qualitative results from the test
set in Figure 6.

As a focus of ABO is enabling real-world transfer, we also test our multi-view
network on catalog images of objects from the test set using the pose
annotations gathered by the methodology in Section 3, and use the inferred
material parameters to relight the object (Figure 7). Despite the domain gap
in lighting and background, and shift from synthetic to real, our network
trained on rendered images makes reasonable predictions on the real catalog
images. In one case (last row), the network fails to accurately infer the true
base color, likely due to the presence of self-shadow.

| SV-net | MV-net (no proj.) | MV-net
---|---|---|---
Base Color ($\downarrow$) | 0.129 | 0.132 | 0.127
Roughness ($\downarrow$) | 0.163 | 0.155 | 0.129
Metallicness ($\downarrow$) | 0.170 | 0.167 | 0.162
Normals ($\uparrow$) | 0.970 | 0.949 | 0.976
Render ($\downarrow$) | 0.096 | 0.090 | 0.086
Table 4: ABO material estimation results for the single-view, multi-view, and
multi-view network without projection (MV-net no proj.) ablation. Base color,
roughness, metallicness and rendering loss are measured using RMSE (lower is
better) - normal similarity is measured using cosine similarity (higher is
better).

###  4.3 Multi-View Cross-Domain Object Retrieval

Merging the available catalog images and 3D models in ABO, we derive a novel
benchmark for object retrieval with the unique ability to measure the
robustness of algorithms with respect to viewpoint changes. Specifically, we
leverage the renderings described in Section 3, with known azimuth and
elevation, to provide more diverse views and scenes for training deep metric
learning (DML) algorithms. We also use these renderings to evaluate the
retrieval performance with respect to a large gallery of catalog images from
ABO. This new benchmark is very challenging because the rendered images have
complex and cluttered indoor backgrounds (compared to the cleaner catalog
images) and display products with viewpoints that are not typically present in
the catalog images. These two sources of images are indeed two separate image
domains, making the test scenario a multi-view cross-domain retrieval task.

Method To compare the performance of state-of-the-art DML methods on our
multi-view cross-domain retrieval benchmark, we use PyTorch Metric Learning
[2] implementations that cover the main approaches to DML: NormSoftmax [70]
(classification-based), ProxyNCA [50] (proxy-based) and Contrastive,
TripletMargin, NTXent [11] and Multi-similarity [63] (tuple-based). We
leveraged the Powerful Benchmarker framework [1] to run fair and controlled
comparisons as in [51], including Bayesian hyperparameter optimization.

We opted for a ResNet-50 [29] backbone, projected it to 128D after a LayerNorm
[4] layer, did not freeze the BatchNorm parameters and added an image padding
transformation to obtain undistorted square images before resizing to 256x256.
We used batches of 256 samples with 4 samples per class, except for
NormSoftmax and ProxyNCA where we obtained better results with a batch size of
32 and 1 sample per class. After hyperparameter optimization, we trained all
losses for 1000 epochs and chose the best epoch based on the validation
Recall@1 metric, computing it only every other epoch.

Importantly, whereas catalog and rendered images in the training set are
balanced (188K vs 111K), classes with and without renderings are not (4K vs.
45K). Balancing them in each batch proved necessary to obtain good
performance: not only do we want to exploit the novel viewpoints and scenes
provided by the renderings to improve the retrieval performance, but there are
otherwise simply not sufficiently many negative pairs of rendered images being
sampled.

Results As shown in Table 5, the ResNet-50 baseline trained on ImageNet
largely fails at the task (Recall@1 of 5%). This confirms the challenging
nature of our novel benchmark. DML is thus key to obtain significant
improvements. In our experiments, NormSoftmax, ProxyNCA and Contrastive
performed better ($\approx\\!29\%$) than the Multi-similarity, NTXent or
TripletMargin losses ($\approx\\!23\%$), a gap which was not apparent in other
datasets, and is not as large when using cleaner catalog images as queries.
Moreover, it is worth noting that the overall performance on ABO is
significantly lower than for existing common benchmarks (see Table 2). This
confirms their likely saturation [51], the value in new and more challenging
retrieval tasks, and the need for novel metric learning approaches to handle
the large scale and unique properties of our new benchmark.

Further, the azimuth ($\theta$) and elevation ($\varphi$) angles available for
rendered test queries allow us to measure how performance degrades as these
parameters diverge from typical product viewpoints in ABO’s catalog images.
Figure 8 highlights two main regimes for both azimuth and elevation: azimuths
beyond $|\theta|\\!=\\!75^{\circ}$ and elevations above
$\varphi\\!=\\!50^{\circ}$ are significantly more challenging to match,
consistently for all approaches. Closing this gap is an interesting direction
of future research on DML for multi-view object retrieval. For one, the
current losses do not explicitly model the geometric information in training
data.

| Rendered images | Catalog
---|---|---
Recall@k (%) | $k\\!=\\!1$ | $k\\!=\\!2$ | $k\\!=\\!4$ | $k\\!=\\!8$ | $k\\!=\\!1$
Pre-trained | 5.0 | 8.1 | 11.4 | 15.3 | 18.0
Constrastive | 28.6 | 38.3 | 48.9 | 59.1 | 39.7
Multi-similarity | 23.1 | 32.2 | 41.9 | 52.1 | 38.0
NormSoftmax | 30.0 | 40.3 | 50.2 | 60.0 | 35.5
NTXent | 23.9 | 33.0 | 42.6 | 52.0 | 37.5
ProxyNCA | 29.4 | 39.5 | 50.0 | 60.1 | 35.6
TripletMargin | 22.1 | 31.1 | 41.3 | 51.9 | 36.9
Table 5: Test performance of state-of-the-art deep metric learning methods on
the ABO retrieval benchmark. Retrieving products from rendered images
highlights performance gaps that are not as apparent when using catalog
images.

Figure 8: Recall@1 as a function of the azimuth and elevation of the product
view. For all methods, retrieval performance degrades rapidly beyond azimuth
$|\theta|>75^{\circ}$ and elevation $\varphi>50^{\circ}$.

##  5 Conclusion

In this work we introduced ABO, a new dataset to help bridge the gap between
real and synthetic 3D worlds. We demonstrated that the set of real-world
derived 3D models in ABO are a challenging test set for ShapeNet-trained 3D
reconstruction approaches, and that both view- and canonical-space methods do
not generalize well to ABO meshes despite sampling them from the same
distribution of training classes. We also trained both single-view and multi-
view networks for SV-BRDF material estimation of complex, real-world
geometries - a task that is uniquely enabled by the nature of our 3D dataset.
We found that incorporating multiple views leads to more accurate
disentanglement of SV-BRDF properties. Finally, joining the larger set of
products images with synthetic renders from ABO 3D models, we proposed a
challenging multi-view retrieval task that alleviates some of the limitations
in diversity and structure of existing datasets, which are close to
performance saturation. The 3D models in ABO allowed us to exploit novel
viewpoints and scenes during training and benchmark the performance of deep
metric learning algorithms with respect to the azimuth and elevation of query
images.

While not considered in this work, the large amounts of text annotations
(product descriptions and keywords) and non-rigid products (apparel, home
linens) enable a wide array of possible language and vision tasks, such as
predicting styles, patterns, captions or keywords from product images.
Furthermore, the 3D objects in ABO correspond to items that naturally occur in
a home, and have associated object weight and dimensions. This can benefit
robotics research and support simulations of manipulation and navigation.

Acknowledgements We thank Pietro Perona and Frederic Devernay. This work was
funded in part by an NSF GRFP (#1752814) and the Amazon-BAIR Commons Program.

## References

  * [1] Powerful benchmarker.  https://kevinmusgrave.github.io/powerful-benchmarker.  Accessed: 2022-03-19.
  * [2] Pytorch metric learning.  https://kevinmusgrave.github.io/pytorch-metric-learning.  Accessed: 2022-03-19.
  * [3] Adel Ahmadyan, Liangkai Zhang, Jianing Wei, Artsiom Ablavatski, and Matthias Grundmann.  Objectron: A large scale dataset of object-centric videos in the wild with pose annotations.  arXiv preprint arXiv:2012.09988, 2020.
  * [4] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton.  Layer normalization.  arXiv preprint arXiv:1607.06450, 2016.
  * [5] Miguel Angel Bautista, Walter Talbott, Shuangfei Zhai, Nitish Srivastava, and Joshua M Susskind.  On the generalization of learning-based 3d reconstruction.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2180–2189, 2021.
  * [6] Jan Bechtold, Maxim Tatarchenko, Volker Fischer, and Thomas Brox.  Fostering generalization in single-view 3d reconstruction by learning a hierarchy of local and global shape priors.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15880–15889, 2021.
  * [7] Sai Bi, Zexiang Xu, Kalyan Sunkavalli, David Kriegman, and Ravi Ramamoorthi.  Deep 3d capture: Geometry and reflectance from sparse multi-view images.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5960–5969, 2020.
  * [8] Mark Boss, Varun Jampani, Kihwan Kim, Hendrik Lensch, and Jan Kautz.  Two-shot spatially-varying brdf and shape estimation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3982–3991, 2020.
  * [9] Brent Burley and Walt Disney Animation Studios.  Physically-based shading at disney.  In ACM SIGGRAPH, volume 2012, pages 1–7. vol. 2012, 2012.
  * [10] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al.  Shapenet: An information-rich 3d model repository.  arXiv preprint arXiv:1512.03012, 2015.
  * [11] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In International conference on machine learning, pages 1597–1607. PMLR, 2020.
  * [12] Sungjoon Choi, Qian-Yi Zhou, Stephen Miller, and Vladlen Koltun.  A large dataset of object scans.  arXiv preprint arXiv:1602.02481, 2016.
  * [13] Christopher B Choy, Danfei Xu, JunYoung Gwak, Kevin Chen, and Silvio Savarese.  3d-r2n2: A unified approach for single and multi-view 3d object reconstruction.  In European conference on computer vision, pages 628–644. Springer, 2016.
  * [14] Blender Online Community.  Blender - a 3D modelling and rendering package.  Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018.
  * [15] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * [16] Valentin Deschaintre, Miika Aittala, Fredo Durand, George Drettakis, and Adrien Bousseau.  Single-image svbrdf capture with a rendering-aware deep network.  ACM Transactions on Graphics (TOG), 37(4):128, 2018.
  * [17] Valentin Deschaintre, Miika Aittala, Frédo Durand, George Drettakis, and Adrien Bousseau.  Flexible svbrdf capture with a multi-image deep network.  In Computer Graphics Forum, volume 38, pages 1–13. Wiley Online Library, 2019.
  * [18] Haoqiang Fan, Hao Su, and Leonidas J Guibas.  A point set generation network for 3d object reconstruction from a single image.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 605–613, 2017.
  * [19] Huan Fu, Rongfei Jia, Lin Gao, Mingming Gong, Binqiang Zhao, Steve Maybank, and Dacheng Tao.  3d-future: 3d furniture shape with texture.  arXiv preprint arXiv:2009.09633, 2020.
  * [20] Duan Gao, Xiao Li, Yue Dong, Pieter Peers, Kun Xu, and Xin Tong.  Deep inverse rendering for high-resolution svbrdf estimation from an arbitrary number of images.  ACM Transactions on Graphics (TOG), 38(4):134, 2019.
  * [21] Georgia Gkioxari, Jitendra Malik, and Justin Johnson.  Mesh r-cnn.  In ICCV, 2019.
  * [22] Georgia Gkioxari, Jitendra Malik, and Justin Johnson.  Mesh r-cnn.  In Proceedings of the IEEE International Conference on Computer Vision, pages 9785–9795, 2019.
  * [23] Andreas Mischok Greg Zaal, Sergej Majboroda.  Hdrihaven.  https://hdrihaven.com/.  Accessed: 2020-11-16.
  * [24] Thibault Groueix, Matthew Fisher, Vladimir G Kim, Bryan C Russell, and Mathieu Aubry.  A papier-mâché approach to learning 3d surface generation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 216–224, 2018.
  * [25] Khronos Group.  gltf 2.0 specification.  https://github.com/KhronosGroup/glTF.  Accessed: 2020-11-16.
  * [26] Agrim Gupta, Piotr Dollar, and Ross Girshick.  Lvis: A dataset for large vocabulary instance segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5356–5364, 2019.
  * [27] Richard Hartley and Andrew Zisserman.  Multiple view geometry in computer vision.  Cambridge university press, 2003.
  * [28] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In ICCV, 2017.
  * [29] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
  * [30] HeeJae Jun, ByungSoo Ko, Youngjoon Kim, Insik Kim, and Jongtack Kim.  Combination of multiple global descriptors for image retrieval.  arXiv preprint arXiv:1903.10663, 2019.
  * [31] Abhishek Kar, Christian Häne, and Jitendra Malik.  Learning a multi-view stereo machine.  In Advances in neural information processing systems, pages 365–376, 2017.
  * [32] Kihwan Kim, Jinwei Gu, Stephen Tyree, Pavlo Molchanov, Matthias Nießner, and Jan Kautz.  A lightweight approach for on-the-fly reflectance estimation.  In Proceedings of the IEEE International Conference on Computer Vision, pages 20–28, 2017.
  * [33] Sungyeon Kim, Dongwon Kim, Minsu Cho, and Suha Kwak.  Proxy anchor loss for deep metric learning.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [34] Alexander Kirillov, Yuxin Wu, Kaiming He, and Ross Girshick.  Pointrend: Image segmentation as rendering.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9799–9808, 2020.
  * [35] Sebastian Koch, Albert Matveev, Zhongshi Jiang, Francis Williams, Alexey Artemov, Evgeny Burnaev, Marc Alexa, Denis Zorin, and Daniele Panozzo.  Abc: A big cad model dataset for geometric deep learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9601–9611, 2019.
  * [36] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei.  3d object representations for fine-grained categorization.  In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13), Sydney, Australia, 2013.
  * [37] Alex Krizhevsky et al.  Learning multiple layers of features from tiny images.  2009\.
  * [38] Jake Levinson, Carlos Esteves, Kefan Chen, Noah Snavely, Angjoo Kanazawa, Afshin Rostamizadeh, and Ameesh Makadia.  An analysis of svd for deep rotation estimation.  arXiv preprint arXiv:2006.14616, 2020.
  * [39] Xiao Li, Yue Dong, Pieter Peers, and Xin Tong.  Modeling surface appearance from a single photograph using self-augmented convolutional neural networks.  ACM Transactions on Graphics (TOG), 36(4):45, 2017.
  * [40] Yangyan Li, Hao Su, Charles Ruizhongtai Qi, Noa Fish, Daniel Cohen-Or, and Leonidas J. Guibas.  Joint embeddings of shapes and images via cnn image purification.  ACM Trans. Graph., 2015.
  * [41] Zhengqin Li, Zexiang Xu, Ravi Ramamoorthi, Kalyan Sunkavalli, and Manmohan Chandraker.  Learning to reconstruct shape and spatially-varying reflectance from a single image.  In SIGGRAPH Asia 2018 Technical Papers, page 269. ACM, 2018.
  * [42] Joseph J Lim, Hamed Pirsiavash, and Antonio Torralba.  Parsing ikea objects: Fine pose estimation.  In Proceedings of the IEEE International Conference on Computer Vision, pages 2992–2999, 2013.
  * [43] Joseph J Lim, Hamed Pirsiavash, and Antonio Torralba.  Parsing ikea objects: Fine pose estimation.  In Proceedings of the IEEE International Conference on Computer Vision, pages 2992–2999, 2013.
  * [44] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In European conference on computer vision, pages 740–755. Springer, 2014.
  * [45] Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang, and Xiaoou Tang.  Deepfashion: Powering robust clothes recognition and retrieval with rich annotations.  In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
  * [46] William E Lorensen and Harvey E Cline.  Marching cubes: A high resolution 3d surface construction algorithm.  ACM siggraph computer graphics, 21(4):163–169, 1987.
  * [47] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  arXiv preprint arXiv:1711.05101, 2017.
  * [48] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger.  Occupancy networks: Learning 3d reconstruction in function space.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4460–4470, 2019.
  * [49] George A Miller.  Wordnet: a lexical database for english.  Communications of the ACM, 38(11):39–41, 1995.
  * [50] Yair Movshovitz-Attias, Alexander Toshev, Thomas K. Leung, Sergey Ioffe, and Saurabh Singh.  No fuss distance metric learning using proxies.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017.
  * [51] Kevin Musgrave, Serge Belongie, and Ser-Nam Lim.  A metric learning reality check.  In European Conference on Computer Vision, pages 681–699. Springer, 2020.
  * [52] Hyun Oh Song, Yu Xiang, Stefanie Jegelka, and Silvio Savarese.  Deep metric learning via lifted structured feature embedding.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
  * [53] Keunhong Park, Konstantinos Rematas, Ali Farhadi, and Steven M Seitz.  Photoshape: Photorealistic materials for large-scale shape collections.  arXiv preprint arXiv:1809.09761, 2018.
  * [54] Nikhila Ravi, Jeremy Reizenstein, David Novotny, Taylor Gordon, Wan-Yen Lo, Justin Johnson, and Georgia Gkioxari.  Accelerating 3d deep learning with pytorch3d.  arXiv preprint arXiv:2007.08501, 2020.
  * [55] Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotny.  Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction.  arXiv preprint arXiv:2109.00512, 2021.
  * [56] Google Research.  Google scanned objects, August.
  * [57] Arjun Singh, James Sha, Karthik S Narayan, Tudor Achim, and Pieter Abbeel.  Bigbird: A large-scale 3d database of object instances.  In 2014 IEEE international conference on robotics and automation (ICRA), pages 509–516. IEEE, 2014.
  * [58] Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta.  Revisiting unreasonable effectiveness of data in deep learning era.  In Proceedings of the IEEE international conference on computer vision, pages 843–852, 2017.
  * [59] Xingyuan Sun, Jiajun Wu, Xiuming Zhang, Zhoutong Zhang, Chengkai Zhang, Tianfan Xue, Joshua B Tenenbaum, and William T Freeman.  Pix3d: Dataset and methods for single-image 3d shape modeling.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2974–2983, 2018.
  * [60] Maxim Tatarchenko, Stephan R Richter, René Ranftl, Zhuwen Li, Vladlen Koltun, and Thomas Brox.  What do single-view 3d reconstruction networks learn?  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3405–3414, 2019.
  * [61] Shubham Tulsiani, Tinghui Zhou, Alexei A Efros, and Jitendra Malik.  Multi-view supervision for single-view reconstruction via differentiable ray consistency.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2626–2634, 2017.
  * [62] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie.  The Caltech-UCSD Birds-200-2011 Dataset.  Technical Report CNS-TR-2011-001, California Institute of Technology, 2011\.
  * [63] Xun Wang, Xintong Han, Weilin Huang, Dengke Dong, and Matthew R Scott.  Multi-similarity loss with general pair weighting for deep metric learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5022–5030, 2019.
  * [64] Olivia Wiles and Andrew Zisserman.  Silnet: Single-and multi-view reconstruction by learning from silhouettes.  arXiv preprint arXiv:1711.07888, 2017.
  * [65] Yu Xiang, Wonhui Kim, Wei Chen, Jingwei Ji, Christopher Choy, Hao Su, Roozbeh Mottaghi, Leonidas Guibas, and Silvio Savarese.  Objectnet3d: A large scale database for 3d object recognition.  In European Conference on Computer Vision, pages 160–176. Springer, 2016.
  * [66] Yu Xiang, Roozbeh Mottaghi, and Silvio Savarese.  Beyond pascal: A benchmark for 3d object detection in the wild.  In IEEE winter conference on applications of computer vision, pages 75–82. IEEE, 2014.
  * [67] Haozhe Xie, Hongxun Yao, Xiaoshuai Sun, Shangchen Zhou, and Shengping Zhang.  Pix2vox: Context-aware 3d reconstruction from single and multi-view images.  In Proceedings of the IEEE International Conference on Computer Vision, pages 2690–2698, 2019.
  * [68] Xinchen Yan, Jimei Yang, Ersin Yumer, Yijie Guo, and Honglak Lee.  Perspective transformer nets: Learning single-view 3d object reconstruction without 3d supervision.  In Advances in neural information processing systems, pages 1696–1704, 2016.
  * [69] Wenjie Ye, Xiao Li, Yue Dong, Pieter Peers, and Xin Tong.  Single image surface appearance modeling with self-augmented cnns and inexact supervision.  In Computer Graphics Forum, volume 37, pages 201–211. Wiley Online Library, 2018.
  * [70] Andrew Zhai and Hao-Yu Wu.  Classification is a strong baseline for deep metric learning.  In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (BMVC), 2019.
  * [71] Xiuming Zhang, Zhoutong Zhang, Chengkai Zhang, Josh Tenenbaum, Bill Freeman, and Jiajun Wu.  Learning to reconstruct shapes from unseen classes.  In Advances in Neural Information Processing Systems, pages 2257–2268, 2018.
  * [72] Qingnan Zhou and Alec Jacobson.  Thingi10k: A dataset of 10,000 3d-printing models.  arXiv preprint arXiv:1605.04797, 2016.

##  Appendix A Dataset Properties

Metadata Visualization We visualize additional products according to their
metadata attributes in Figure 9. For each attribute (unit count and weight),
each row depicts products that fall into that label (for categorical
attributes) or bin (for continuous-valued attributes). For continuous-valued
attributes, products within the same row are ordered from lowest to highest.

##  Appendix B Dataset Organization

In this work we used ABO to derive benchmarks for different tasks such as 3D
reconstruction, material estimation, and multi-view retrieval. Table 6
outlines how each data subset in ABO is used at both train and test time. “No-
BG Renders” refers to the white-background rendered dataset of ABO objects
described in Section 4.1 of the main text, while “BG Renders” refers to the
Material Estimation Dataset described in Section 3 of the main text.

##  Appendix C 6DOF Pose Optimization

Instance Masks We use instance masks generated both from MaskRCNN [28] trained
on LVIS [26] as well as PointRend [34] trained on COCO [44]. Both model
checkpoints were obtained from the Detectron2
repository111https://github.com/facebookresearch/detectron2. We kept predicted
masks from all categories with a confidence score greater than $0.1$.

Pose Optimization For each instance mask, we initialize $24$ different runs
with random rotations and optimize R and T for $1,000$ steps using the Adam
optimizer with a learning rate of 1e-2. We parameterize the rotation matrix
using the symmetric orthogonalization procedure of [38]. At the end of the
$24$ runs, we pick the pose that has the lowest loss and validate its
correctness via a human check.

##  Appendix D Single-View 3D Reconstruction Evaluation

View-Space Evaluations Since view-space predictions can be scaled and
$z$-translated simultaneously while preserving the projection onto the camera,
we must solve the depth-ambiguity to align the predicted and ground truth (GT)
meshes for benchmarking. We use known camera extrinsics to transform the GT
mesh into view-space, and align it with the predicted mesh by solving for a
Chamfer-distance minimizing depth. In practice, we normalize average vertex
depths for each mesh independently and then search through $51$ candidate
depths. We compare the predicted and GT meshes after alignment following the
evaluation protocol in [18, 22] and report Chamfer distance and Absolute
Normal Consistency. Since Chamfer varies with the scale of the mesh, we follow
[18, 22] and scale the meshes such that the longest edge of the GT mesh
bounding box is of length 10.

Figure 9: Metadata visualization. Catalog image samples for “unit count” and “weight” metadata attributes. For each product we show its main image. |  | 3D Recon. | Material Est. | Retrieval
---|---|---|---|---
Train | No-BG Renders |  |  |
BG Renders |  | ✓ | ✓
Catalog Images |  |  | ✓
Test | No-BG Renders | ✓ |  |
BG Renders |  | ✓ | ✓
Catalog Images |  | ✓ | ✓
Table 6: ABO data subsets used in each benchmarking experiment at both train
and test time. “BG” and “No-BG” Renders refer to renderings from 3D models in
ABO with and without backgrounds.

Canonical-Space Evaluations To evaluate shapes predicted in canonical space,
we must first align them with the GT shapes. Relying on cross-category
semantic alignment of models in both ShapeNet and ABO, we use a single
(manually-set) rotation-alignment for the entire data. We then solve for
relative translation and scale, which remain inherently ambiguous, to minimize
the Chamfer distance between the two meshes. In practice, we search over a
$3^{4}$ grid of candidate scale/translation after mean-centering (to vertex
centroid) and re-scaling (via standard deviation of vertex distances to
centroid) the two meshes independently. Note that R2N2 [13] predicts a voxel
grid so we convert it to a mesh for the purposes of benchmarking. Marching
Cubes [46] is one such way to achieve this, however, we follow the more
efficient and easily batched protocol of [22] that replaces every occupied
voxel with a cube, merges vertices, and removes internal faces.

Additional Qualitative Examples We show additional reconstructions of ABO
objects from the best performing single-view 3D reconstruction method, Mesh
R-CNN, and the method that claims to be category-agnostic, GenRe in Figure 12.
We find that both methods generally fail to reconstruct objects with thin
structures, but that the failure occurs in different ways for each method.
GenRe simply does not reconstruct them, whereas Mesh R-CNN produces a
reconstruction that qualitatively appears more like the 3D convex hull of the
object.

Quantitative Evaluation on Test Split As the focus of this work was to measure
the domain gap of ShapeNet-trained 3D reconstruction methods to ABO, we
evaluated reconstruction performance for all 3D models in ABO that came from
ShapeNet categories. To facilitate future research that may want to train 3D
reconstruction methods using ABO and compare to ShapeNet trained methods, we
generate a train/val/test split (80%/10%/10%) that we release along with the
dataset. We also report the performance (averaged across all categories) of
each method on this test split in Table 7.

|  Chamfer ($\downarrow$) |  Abs. Normal Consistency ($\uparrow$)
---|---|---
3D R2N2 [13] | 1.97 | 0.55
OccNets [48] | 1.19 | 0.70
GenRe [71] | 1.61 | 0.66
Mesh R-CNN [21] | 0.82 | 0.62
Table 7: 3D reconstruction on ABO test split. Chamfer distance and absolute
normal consistency averaged across all categories

##  Appendix E Material Estimation

Dataset Curation We use the Material Estimation Dataset outlined in the main
text, omitting object with transparencies, resulting in 7,679 models. We split
the 3D models into a non-overlapping train/test set of 6,897 and 782 models,
respectively. To test generalization to new lighting conditions, we reserve 10
out of 108 HDRI environment maps for the test set only.

Network Details A visualization of the single-view material estimation network
(SV-net) and multi-view network (MV-net) can be found in Figure 10 (middle and
bottom, respectively). The MV-net uses camera poses to establish pixel-level
correspondences. Given a pixel $p$ in one viewpoint with image coordinate $x$
and depth $z$, the image coordinate $x^{\prime}$ of the its corresponding
pixel in another viewpoint can be computed as $x^{\prime}=KRK^{-1}x+Kt/z$,
where $K$ is the camera intrinsic matrix, $R$ and $t$ are the rotation and
translation between the two viewpoints. Some pixels in one view can be
occluded and hence not visible in other views. These can be determined by
using a depth-based occlusion test. We fill these pixels using values from the
reference view.

Full Texture Map Reconstruction Using per-view predicted material maps, we can
also generate a full textured PBR model. This is achieved by back-projecting
the predicted material maps to the UV domain and using a learned network to
aggregate and smooth the predictions. Figure 11 shows the full UV map
reconstruction pipeline.

Figure 10: Top: Encoder-decoder architecture. Encoder uses a ResNet-34
(conv1-conv5) backbone. KxK-N-M-X denotes a double convolution block of KxK
filter, N input channels, M intermediate channels, and X output channels. We
use BatchNorm and leaky ReLU. Middle: Single-view baseline. Bottom: Multi-view
baseline. Given a reference view, neighboring views are selected and projected
to the reference view and passed as input to the network.  Figure 11: Pipeline
for full UV map prediction. Per-view predictions are back projected onto the
UV and fed to an encoder-decoder network for smoothing and aggregation.

##  Appendix F Multi-View Cross-Domain Object Retrieval

Dataset Curation For this task, we used product type annotations to focus only
on rigid objects, removing items such as garments, home linens, and some
accessories (cellphone accessories, animal harnesses, cables, etc.). As the
set of products beyond just those with 3D models are likely to contain near-
duplicates (i.e. different sizes of the same shoe), we then applied a
hierarchical Union-Find algorithm for near-duplicate detection and product
grouping, based on shared imagery as a heuristic. We considered near-
duplicates as correct matches of a single instance and thus assigned a unique
_instance id_ to all near-duplicate listings. Product groups are sets of such
instances that are from product lines that may share design details,
materials, patterns and thus may have common images (close-up detail of
fabric, fact sheet image, …). Consequently, we ensured that all instances in a
group are assigned to the same data split (train, val or test). The val and
test sets contain only instances with 3D models and, while their catalog
images compose their respective target set (val-target and test-target), we
use rendered images as queries (val-query and test-query).

Dataset Curation Statistics The hierarchical Union-Find algorithm yielded
$29,988$ groups of $50,756$ instances, of which $1,334$ have 3D models
($5,683$ instances). We then sampled groups accounting for $836$ instances
with 3D models for the _test_ set, using their combined $4,313$ catalog images
as _test-target_ images and sampled 8 rendered views for each of the
environment maps as _test-query_ images. Again, we sampled groups with $854$
of the remaining instances with 3D models for the _validation_ set, using
catalog images ($4,707$) as _val-target_ and 8 rendered images per envmap as
_val-query_. We used the rest of the instances as the _train_ set: $49,066$
instances ($3,993$ with 3D model), $187,912$ catalog images and $110,928$
rendered images ($298,840$ total).

Implementation Details Train images are pre-processed by the following: square
padding, resize to 256x256, random resized crop of 227x227 (scale between
$0.16$ and $1$ and ratio between $0.75$ and $1.33$) and random horizontal flip
($p=0.5$). Test images are padding to square, resized to 256x256 and center
cropped to 227x227.

The trunk used is ResNet-50 pre-trained on ImageNet, leading to a 2048D
vector. We did not freeze these weights, including the BatchNorm parameters.
The embedding module is a LayerNorm normalization followed by linear
projection to 128D embedding. The batch sampler is class-balanced and domain-
balanced. Domain-balancing is implemented as a hierarchical sampling: we first
sample $N$ classes with rendered images and $N$ classes without, then sample
$K$ images for each class among all available images for the class, leading to
batches of $2NK$ samples. For NormSoftmax and ProxyNCA we used batches of 32
samples, 1 sample per class, 16 classes with rendered images and 16 without.
For all other methods we used batches of 256 samples, 4 samples per class, 64
classes with rendered images and 64 classes without.

One epoch consists of 200 batches sampled from the above procedure. We trained
the models for 1000 epochs, using early stopping when no improvements on the
validation accuracy are found for over 250 consecutive epochs. The epoch for
testing is selected based on the maximum validation Recall@1 for validation.
We used 8 workers for data loading on a AWS p3.8xlarge instance (32 cores, 4
Nvidia V100 GPUs). As [51], we did not use tuple mining within a batch. The
embeddings are normalized before indexing and querying. We used Recall@1 using
_val-query_ rendered images as queries against _val-target_ catalog images as
the validation metric, computing it at every even epoch. RMSProp with a
learning rate $1e\\!-\\!6$, weight decay $1e\\!-\\!4$, and momentum 0.9 is
used to optimize both the trunk and mebedding layers. For metric losses, where
applicable, the learning rate is optimized via hyperparameter optimization.

Loss | Recall@1 (%) | Recall@2 (%) | Recall@4 (%) | Recall@8 (%) | MAP (%) | MAP@R (%) | R-Precision (%)
---|---|---|---|---|---|---|---
Pre-trained | 4.97 | 8.10 | 11.41 | 15.30 | 7.69 | 2.27 | 3.44
Constrastive | 28.56 | 38.34 | 48.85 | 59.10 | 31.19 | 14.16 | 19.19
Multi-similarity | 23.12 | 32.24 | 41.86 | 52.13 | 26.77 | 11.72 | 16.29
NormSoftmax | 30.02 | 40.32 |  . 50.19 | 59.96 | 32.61 | 14.03 | 18.76
NTXent | 23.86 | 33.04 | 42.59 | 51.98 | 27.00 | 12.05 | 16.51
ProxyNCA | 29.36 | 39.47 | 50.05 | 60.11 | 32.38 | 14.05 | 19.00
TripletMargin | 22.15 | 31.10 | 41.32 | 51.90 | 25.80 | 10.87 | 15.41
Table 8: Test metrics for the ABO-MVR benchmark, using rendered images as queries. The gallery images are derived from catalog images and contain classes from the train and test classes. Loss | Recall@1 (%) | Recall@2 (%) | Recall@4 (%) | Recall@8 (%) | MAP (%) | MAP@R (%) | R-Precision (%)
---|---|---|---|---|---|---|---
Pre-trained | 17.99 | 23.93 | 31.72 | 38.65 | 22.57 | 6.99 | 9.55
Constrastive | 39.67 |  . 52.21 | 64.41 | 71.64 | 42.96 | 22.52 | 28.07
Multi-similarity | 38.05 | 50.06 | 61.79 | 68.17 | 40.87 | 21.06 | 26.32
NormSoftmax | 35.50 | 46.70 | 57.38 | 64.78 | 38.07 | 18.63 | 23.42
NTXent | 37.51 | 49.34 | 61.37 | 69.23 | 40.12 | 20.03 | 25.32
ProxyNCA | 35.64 | 46.53 | 57.36 | 65.06 | 38.50 | 18.81 | 23.65
TripletMargin | 36.87 | 48.34 | 60.98 | 69.44 | 40.03 | 19.94 | 25.46
Table 9: Test metrics for the ABO-MVR benchmark, using catalog images as
queries. The gallery images are derived from catalog images and contain
classes from the train and test classes.

Hyperparameters are estimated by 30 runs of Bayesian hyperparameter
optimization for 100 epochs, using the implementation of Powerful-Benchmarker,
the epoch is chosen on the full run after optimization:

  * •

Contrastive. Pos margin: $0.5287$; Neg margin: $1.0523$; Epoch: $614$

  * •

Multi-similarity. Alpha: $0.0240$; Beta: $49.1918$; Base: $0.5567$; Epoch:
$76$

  * •

NormSoftmax. Temperature: $0.0776$; Metric loss learning rate: $0.0013$;
Epoch: $448$

  * •

NTXent. Temperature: $0.0665$; Epoch: $114$

  * •

ProxyNCA. Softmax scale: $5.5915$; Metric loss learning rate: $0.0010$; Epoch:
$566$

  * •

Triplet. Margin: $0.0743$; Epoch: $706$

Additional Metrics Tables 8 and 9 provide a more complete version of Table 5
from the main manuscript. In addition to Recall@k, we also report mean average
precision (MAP), mean average precision at R (MAP@R) and R-Precision as
desribed and implemented by [51]. These metrics provide additional insights
compared to recall, as they give higher value to retrieval results with
correct rankings and having as many correct results as possible in the first
ranks.

In Table 8, we compare the results when using rendered images of test classes
as queries, against the union of catalog images of test classes and catalog
images of train classes. In Table 9, we instead compare the results when using
catalog images of test classes as queries, against the same union of catalog
images of test classes and catalog images of train classes. Naturally, the
image used as query is discarded from its own search results before computing
the metrics. Still, as we can see, these is a significant gap between
retrieving from rendered queries compared to catalog images. This highlights
the difficulty of this new benchmark.

Qualitative Results In Figure 13 we show qualitative results for a few queries
(low elevation, mid elevation, high elevation), showing some success and
failure cases of NormSoftmax, ProxyNCA and Contrastive.

Figure 12: Additional qualitative 3D reconstruction results for GenRe and Mesh
R-CNN. The first four rows display ABO objects that overlap with common
ShapeNet categories, such as cabinet, chair and table. The bottom four rows
display reconstructions for ABO objects from categories not represented in
ShapeNet.

Figure 13: Qualitative retrieval results for low, medium, and high elevation
products. The leftmost column shows the query image, the other 8 columns show
the top-8 results, highlighted in green if correct and red if incorrect. The
top 3 rows are results of Contrastive, the middle 3 are of NormSoftmax and the
bottom 3 are ProxyNCA. Each group of 3 rows have the same queries: one of low
elevation (side table), one of mid-elevation (air conditioner), one of high
elevation (cart).