# Anomalous interfacial electron transfer kinetics in twisted trilayer
graphene caused by layer-specific localization

Kaidi Zhang  Department of Chemistry, University of California, Berkeley, CA
94720, USA Yun Yu  Department of Chemistry, University of California,
Berkeley, CA 94720, USA Stephen Carr  Brown Theoretical Physics Center, Brown
University, Providence, RI, USA Mohammad Babar  Department of Mechanical
Engineering, Carnegie Mellon University, Pittsburgh, PA, USA Ziyan Zhu  SLAC
National Accelerator Laboratory, Stanford, CA, USA Bryan Kim  Department of
Chemistry, University of California, Berkeley, CA 94720, USA Catherine
Groschner  Department of Chemistry, University of California, Berkeley, CA
94720, USA Nikta Khaloo  Department of Chemistry, University of California,
Berkeley, CA 94720, USA Takashi Taniguchi  International Center for Materials
Nanoarchitectonics, National Institute for Materials Science, Tsukuba, Japan
Kenji Watanabe  Research Center for Functional Materials, National Institute
for Materials Science, Tsukuba, Japan Venkatasubramanian Viswanathan
Department of Mechanical Engineering, Carnegie Mellon University, Pittsburgh,
PA, USA D. Kwabena Bediako  Department of Chemistry, University of California,
Berkeley, CA 94720, USA Chemical Sciences Division, Lawrence Berkeley National
Laboratory, Berkeley, CA 94720, USA

## Abstract

Interfacial electron-transfer (ET) reactions underpin the interconversion of
electrical and chemical energy. Pioneering experiments showed that the ET rate
depends on the Fermi–Dirac distribution of the electronic density of states
(DOS) of the electrode, formalized in the Marcus–Hush–Chidsey (MHC) model.
Here, by controlling interlayer twists in well-defined trilayer graphene
moirés, we show that ET rates are strikingly dependent on electronic
localization in each atomic layer, and not the overall DOS. The large degree
of tunability inherent to moiré electrodes leads to local ET kinetics that
range over three orders of magnitude across different constructions of only
three atomic layers, even exceeding rates at bulk metals. Our results
demonstrate that beyond the ensemble DOS, electronic localization is critical
in facilitating interfacial ET, with implications for understanding the origin
of high interfacial reactivity typically exhibited by defects at
electrode–electrolyte interfaces.

## Introduction

Interfacial electron-transfer reactions underpin the interconversion of
electrical and chemical energy[1, 2, 3]. Within the microscopic theory of
electron transfer, the MHC model[4, 5, 6, 7, 8, 4, 5] highlights the
dependence of electrochemical rates on the DOS of an electrode, motivating the
discovery of new approaches to manipulating the band structure of electrodes
as a means of controlling the performance limits of energy conversion and
storage devices. However, it is conventionally assumed that the electrode DOS
are invariant with energy/overpotential and delocalized[8]. Atomic defects at
electrode surfaces provide a striking, albeit challenging to control, example
of the pronounced effect of local structural/electronic modifications on
interfacial reactivity. Atomic vacancies[9], kinks, and step edges[10, 11, 12]
are typically associated with massively enhanced interfacial reactivity
compared to atomically pristine surfaces. The effect of these defects is
typically explained in the context of providing increased DOS at energies that
are desirable for charge transfer or formation of a surface-bound catalytic
intermediate (such as mid-gap states in a semiconducting material[10, 9]).
However, the dangling bonds at such sites would invariably introduce a strong
spatial localization of these large electronic DOS. For this reason, beyond
the augmented DOS magnitude, we might consider that localization may play a
key role in facilitating interfacial ET to the necessarily localized
electronic states on the solution-phase molecule/complex/ion. Yet, a
systematic experimental examination of the effects of electronic localization
on heterogeneous interfacial charge transfer has been intractable owing to the
considerable synthetic challenge of constructing pristine electrode materials
that would allow a deterministic modulation of this property separate from the
overall DOS.

Azimuthal misalignment of atomically thin layers produces moiré superlattices
and alters the electronic band structure, in a manner that is systematically
dependent on the interlayer twist angle[13, 14]. The formation of flat
electronic bands, particularly at a series of ‘magic’ moiré angles leads to a
diversity of correlated electron physics[15, 16, 17, 18]. Notably, these flat
bands imply a large DOS that is highly localized in real space[19]. Small-
angle twisted bilayer graphene (TBG) exhibits a recently discovered angle-
dependent electrochemical behavior[20], where outer-sphere ET kinetics can be
tuned nearly tenfold simply by varying moiré twist angle, $\theta_{m}$,
between 0° and 2°.

The stacking order of graphene in multilayers strongly alters the resulting
electronic properties of the system[21, 23, 24, 25, 26, 27, 28, 29, 16, 22].
Where Bernal (ABA-stacked) trilayer graphene displays dispersive bands,
rhombohedral (ABC) graphene possesses a non-dispersive, or ‘flat’, electronic
band close to the Fermi level, which is responsible for the emergence of
correlated electron phenomena at low temperatures[30, 31].

Fig. 1: Polytypes of twisted trilayer graphene. (A,B) Illustrations of two
twisted trilayer graphene polytypes, with moiré wavelength $\lambda$. The
black parallelogram outlines the moiré unit cell in each case. (C,D) Computed
local DOS (see Methods) for 1.2° M-t-B (C) and 1.5° A-t-A (D).

More pronounced flat bands are produced in twisted trilayer graphene (TTG)
structures. Rotationally misaligned (by a moiré ‘twist’ angle $\theta_{m}$)
monolayer and Bernal stacked bilayer forms a ‘monolayer-twist-bilayer’ (M-t-B)
heterostructure (Fig. 1A)[32, 33]. Systematically alternating the angle
between adjacent graphene layers such that the top layer is perfectly aligned
with the bottom layer results in an ‘A-t-A’ heterostructure (Fig. 1B)[29, 16,
22] that possesses extremely flat bands at a magic angle around $1.5\degree$.
These flattened electronic bands, which manifest as large DOS that are
localized on AAB and AAA sites in M-t-B and A-t-A TTG, respectively (Figs.
1C,D), now introduce distinctive possibilities for systematically probing the
dependence of interfacial ET on electronic structure generally and in
particular, the effects of electronic localization. For example, even within
the TTG family, larger DOS are found in A-t-A as compared to M-t-B near their
respective magic angles (Figs. 1C,D), properties that naively might be
expected to correlate with interfacial ET rates, based on MHC theory.

## Results and Discussion

Scanning electrochemical cell microscopy (SECCM)[12] measurements were carried
out on non-twisted (ABA, ABC) and twisted trilayer graphene samples that were
fabricated into devices (see Methods)[20]. As shown in Fig. 2A, naturally
occurring ABA and ABC trilayers were mechanically exfoliated from bulk
graphite and identified using optical microscopy together with confocal Raman
spectroscopy (see Methods) [34, 35]. M-t-B and A-t-A TTG samples were prepared
by the ‘cut-and-stack’ approach (see Methods), resulting in samples possessing
uniform $\theta_{m}$ around the ‘magic angles’ of about 1.34° for an M-t-B
device and 1.53° for an A-t-A device. Piezoelectric force microscopy (PFM) and
scanning tunneling microscopy (STM) were used to evaluate the twist angle
distribution and uniformity across the moiré samples (Fig. 2B)[36]. Using
SECCM, cyclic voltammograms (CVs) were measured with 2.0 mM Ru(NH3)6^3+—an
ideal and well-established redox couple for interrogating outer-sphere ET
kinetics[11, 20]—and 0.10 M KCl as the supporting electrolyte. In Fig. 2C, a
representative set of CVs collected from these different trilayer samples is
shown. We find that the ABA domain of the flake shown in Fig. 2A exhibited the
most sluggish rates of Ru(NH3)6^3+ electro-reduction, evinced by a half-wave
potential ($E_{1/2}$) of –0.32 V, which is cathodically shifted substantially
from the equilibrium potential, $E^{0}$, of –0.25 V for Ru(NH_3)_6^3+/2+ (all
potentials are reported relative to the Ag/AgCl quasi-counter/reference
electrode). However, the $E_{1/2}$ measured from the CV acquired at region II
(ABC domain) of the same flake was –0.27 V, pointing to considerably more
facile electroreduction kinetics on the rhombohedral trilayer as compared to
the Bernal trilayer. For both TTG samples, reversible CVs with $E_{1/2}$
$\approx$ –0.25 V were obtained, indicative of highly facile electrokinetics
and heterogeneous electrochemical rate constants that exceed those of both ABA
and ABC graphene considerably. These observations motivated the measurement of
the variation of interfacial ET rates with $\theta_{m}$.

Fig. 2: Fabrication and electrochemistry of twisted trilayer graphene. (A)
Left: Optical micrograph of a device fabricated from an exfoliated trilayer
graphene flake on hBN. Right: Confocal Raman spectra acquired in the sites in
A marked with red (ABC domain) and blue (ABA domain) dots, along with the
Raman map of the region indicated with a yellow box in A. Scale bars: 10
$\mu$m. (B) Left: Optical micrograph of an M-t-B device on hBN (Scale bar: 10
$\mu$m). Right: A lateral PFM phase image over the yellow boxed region in B
reveals the moiré superlattice pattern. Scale bar: 50 nm. (C) Representative
steady-state voltammograms of 2 mM Ru(NH3)6^3+ in 0.1 M KCl solution obtained
at ABA and ABC trilayer graphene, along with 1.3° M-t-B and 1.5° A-t-A,
compared to that obtained at a $\sim 40$ nm thick platinum film. Scan rate,
100 mVs-1. Inset illustrates the SECCM technique.

To quantitatively assess differences in interfacial kinetics associated with
disparate electronic structures, we compared experimental CVs to those
simulated with different standard rate constants, $k^{0}$, calculated with the
Butler–Volmer model (see Methods). Here, it is critical to account for the
relatively small and potential-dependent quantum capacitance, $C_{q}$ (see
Methods)[20, 11] in these low-dimensional electrodes, which for a given
applied potential, $V_{app}$, produces a dynamic electron or hole doping of
the few-layer graphene by an energy of $eV_{q}$ (where $e$ is the elementary
charge and $V_{q}$ is the chemical potential relative to the charge neutrality
potential). The remainder, $V_{dl}$, persists as a drop across the electric
double layer (so that $V_{app}=V_{q}+V_{dl}$). $C_{q}(V_{q})$ was calculated
for all trilayer systems (ABA, ABC, as well as M-t-B and A-t-A at various
$\theta_{m}$) (Fig. 3A) using the respective computed band structures and DOS
profiles (see Methods). The corresponding plots of $V_{dl}/V_{app}$ as a
function of $V_{app}$ are shown in Fig. 3B. Taken together, these data reveal
that flat electronic bands result in a more significant fraction of $V_{app}$
partitioning into $V_{dl}$ near the charge neutrality potential. Notably, as
shown in Fig. 3A, changes in $\theta_{m}$ tune $C_{q}(V_{q})$ and ‘magic-
angle’ ($\approx 1.5\degree$) A-t-A displays higher $C_{q}$ than ‘magic-angle’
($1.2-1.3\degree$) M-t-B consistent with its overall greater DOS (Fig. 1D).

Fig. 3: Angle dependent quantum capacitance and interfacial ET. (A) Calculated
$C_{q}$ as a function of the chemical potential ($V_{q}$) for ABA, ABC, and
TTG using the respective computed band structures and DOS profiles (see
Methods). (B) Calculated fraction of applied potential on the double layer
($V_{dl}/V_{app}$) as a function of the applied potential ($V_{app}$) for ABA,
ABC, and TTG. $V_{q}$ and $V_{app}$ are relative to the charge neutrality
potential. Taken together, these data reveal that flat electronic bands result
in a more significant fraction of $V_{app}$ partitioning into $V_{dl}$ near
the charge neutrality potential. (C) Dependence of ET rate constant, $k^{0}$,
on trilayer graphene stacking type (ABA, ABC) and $\theta_{m}$ for M-t-B TTG.
Each marker denotes the mean of measurements made on samples within a standard
deviation of the mean twist angle. The horizontal and vertical error bars
represent the standard deviations of $\theta_{m}$ and the standard error of
$k^{0}$. The inset shows comparison of $k^{0}$ values for M-t-B, B-t-M, and
A-t-A TTG at $\theta_{m}=0.82\pm 0.05\degree$.

After determining $V_{dl}$ in this manner, we extracted $k^{0}$ values by
identifying the simulated CV that was in closest agreement with the
experiment[20] (see Methods). The $\theta_{m}$ dependence of $k^{0}$ was
measured by preparing M-t-B TTG devices with varying $\theta_{m}$ between
0.08° and 8.0° (see Methods) and acquiring CVs of Ru(NH3)6^3+ electroreduction
by SECCM for each sample. Fig. 3C shows the strong, non-monotonic variation in
$k^{0}$ over two orders of magnitude from ABA and ABC graphene to
$\theta_{m}=8\degree$ M-t-B. For samples with $1\degree\leq\theta_{m}\leq$ 2°
ET appears reversible within our accessible scan rates and so we cannot
extract any kinetic information beyond noting that within this range of
$\theta_{m}$, $k^{0}\geq 0.35$ cm/s. The quenched dependence of $\theta_{m}$
on $k^{0}$ (blue markers in Fig. 3C) in analogous electrochemical measurements
of the trisphenanthroline cobalt(III/II) redox couple, Co(phen)_3^3+/2+ (see
Methods) provides compelling evidence that it is the moiré flat bands that
drive the observed angle-dependent electrokinetic modulation in TTG, as in
TBG[20].

An unexpected observation of the factors controlling interfacial ET is made by
comparing the electrochemical responses of TTG polytypes. A-t-A TTG, on the
basis of its massive DOS (Fig. 1D) and giant $C_{q}$—which exceeds that of
M-t-B (Fig. 3A)—should be expected to yield the highest ET rates. However,
while an effect of $\theta_{m}$ on $k^{0}$ is also observed in A-t-A samples,
this variant of TTG displays consistently lower $k^{0}$ than M-t-B at similar
$\theta_{m}$ (Fig. 3C, inset). Further, B-t-M heterostructures, which consist
of a Bernal bilayer placed with a twist atop a monolayer (i.e., flipped
versions of M-t-B), display markedly lower $k^{0}$ values than the
corresponding M-t-B electrodes, notwithstanding an ostensibly identical
overall electronic structure. These striking observations point clearly to
effects governing the interfacial ET kinetics beyond simply the ensemble DOS.

Fig. 4: Lattice relaxation and stacking area fractions in TTG. (A,B,D,E)
Constant current STM images representative M-t-B (A,B) and A-t-A (D,E)
samples. Scale bars: 50 nm (C,D) Qualitative illustrations of different
stacking domains in rigid and relaxed M-t-B (C) and A-t-A (F) moiré unit
cells. (G) Extracted area fraction of different stacking domains in M-t-B TTG.
Error bars represent standard errors.

To fully understand these $\theta_{m}$ dependencies as well as the disparities
between interfacial electron transfer kinetics of M-t-B, B-t-M, and A-t-A, we
used STM (room temperature, constant current) to evaluate the role of lattice
relaxation in controlling the area fraction of stacking domains in M-t-B and
A-t-A TTG. In Fig. 4A, a representative STM map of small-angle
($\theta_{m}=0.14\degree$) M-t-B shows a clear contrast between the various
stacking domains. Regions with higher local DOS appear brighter than those
with lower DOS since a larger tip–sample distance is required to maintain a
constant current[33]. ABC domains, therefore, appear brighter than ABA domains
owing to the native flatband of the ABC stacking type. These ABA and ABC
domains (black and red regions, respectively) form alternating triangular
patterns while the AAB region forms small circles of diameter $\approx$ 11 nm,
which appear with the brightest contrast owing to the localization of the
moiré flatband and associated large DOS on these AAB sites as shown in Fig. 1C
(this is analogous to the localization of moiré flatbands on AA sites in
TBG[19]). For $\theta_{m}=0.78\degree$ (Fig. 4B), while the triangular ABA/ABC
patterns have shrunk in size compared to those in Fig. 4A, the diameters of
AAB regions remained largely unchanged. For A-t-A, AAA domains are visible as
bright spots (Figs. 4D,E), consistent with the localization of the large DOS
on these regions (Fig. 1D)[37], with degenerate ABA and BAB regions requiring
smaller tip–sample distances (dark regions) to sustain a constant STM current
because of a lower local DOS.

The measured area distribution of stacking domains in TTG, therefore, differ
significantly from those of rigid moiré structures and are instead relaxed as
depicted schematically in Fig 4C,F minimizing (maximizing) high (low) energy
domains in a manner that is conceptually analogous to that reported for
TBG[38, 19, 39]. To support these experiments, we also performed finite
element method (FEM) simulations to model relaxation in TTG, finding results
that lie in good agreement with our STM and dark-field transmission electron
microscopy data. Importantly, these structural measurements and calculations
permit a quantitative determination of the area fractions in TTG after
reconstruction as a function of $\theta_{m}$ as plotted in Fig. 4G.

These area fraction distributions after structural relaxation explain the
origin of the kinetic modulation observed in Fig. 3C at $\theta_{m}<2\degree$
as being driven by $\theta_{m}$-dependent area fractions of the ‘topological
defect’[40, 41] AAB and AAA sites. Our relaxation simulations also show that
at $\theta_{m}\leq 0.3\degree$, relaxation of these moiré superlattices
reestablishes nearly commensurate ABA, BAB, and/or ABC domains with local DOS
that should not deviate substantially from those of freestanding ABA and ABC
trilayers. This observation is in line with previous experimental[39, 38, 33,
41] and theoretical studies[42, 41] of lattice relaxation in bilayer
analogues. Therefore, by considering $k^{0}$ variations at
$\theta_{m}<1\degree$ in Fig. 3C (which are also within the range of
kinetically resolvable $k^{0}$), we can extract the local rate constant
associated with the AAB and AAA stacking domains through Eq.1 and Eq.2 where
$\beta_{i}$ and $\kappa_{i}^{0}$ represent the area fraction and local
standard heterogeneous electron transfer rate constant, respectively, for
stacking domain $i$.

$k_{MtB}^{0}=\beta_{AAB}\kappa_{AAB}^{0}+\beta_{ABC}\kappa_{ABC}^{0}+\beta_{ABA}\kappa_{ABA}^{0}+\beta_{SP}\kappa_{SP}^{0}$
(1)

$k_{AtA}^{0}=\beta_{AAA}\kappa_{AAA}^{0}+\beta_{ABA}\kappa_{ABA}^{0}+\beta_{BAB}\kappa_{BAB}^{0}+\beta_{AAB}\kappa_{AAB}^{0}$
(2)

As a result of the lattice relaxation effect discussed above, we can determine
$\kappa_{ABA}$ and $\kappa_{ABC}$ from independent measurements of
freestanding Bernal and rhombohedral trilayers (Figs. 2C and 3C). In addition,
we can assume that $\kappa_{SP}^{0}\approx\kappa_{ABA}^{0}$, which is
justified on the basis of the STM images and calculated local DOS. This
analysis allows us to extract standard electron transfer rate constants for
the AAB (M-t-B), ABB (B-t-M), and AAA (A-t-A) topological defects.

Combined with previous electrochemical measurements at TBG surfaces[20] we
compare ET kinetics of Ru(NH3)6^3+/2+ among a wide array of stacking
configurations from monolayer to trilayer graphene in Fig. 5A. For atomic
stacking orders naturally found in bulk graphite, we observed a gradual
enhancement as the number of layers increases from monolayer to Bernal
trilayer. This can be explained by a modest increase in DOS close to the Fermi
level as the number of layers increases.[11] ABC graphene displays a
pronounced augmentation in $k^{0}$ from that of ABA graphene associated with
the intrinsic flat band of the rhombohedral system. Most notably, ‘artificial’
high energy stacking (AA, AAA, AAB, ABB) topological defects created by moiré
superlattices exhibit extraordinarily high $k^{0}$ values, with that of AAB
exceeding 3 cm/s, which is greater than that measured on bulk platinum
electrodes ($0.85–1.2$ cm/s)[43], notwithstanding only consisting of three
atomic layers.

Fig. 5: ET rates of few-layer graphene and layer-dependent DOS localization
(A) Local standard Ru(NH3)6^3+/2+ ET rate constants at few-layer graphene in
different stacking configurations. ‘Artificial’ moiré-derived stacking domains
are labeled with an asterisk. Each bar is the mean local rate either measured
(for natural stacking) or calculated (for aritifical stacking) for small twist
angle samples. The error bars represent the standard errors for the rates. (B)
Schematic of M-t-B/B-t-M graphene layers. (C) Layer-dependent DOS profile (see
Methods) for AAB stacking domains in M-t-B and B-t-M graphene at
$\theta_{m}=1.2\degree$. Insets show real space DOS maps of each layer at
$\epsilon$ = -3 meV. (D) Schematic of the A-t-A layers. (E) Layer-dependent
DOS profile for AAA stacking domains in A-t-A graphene at
$\theta_{m}=1.2\degree$. The insets show real space DOS maps of each layer at
$\epsilon$ = -1 meV for $\theta_{m}=1.2\degree$.

Fig. 5A also shows the unexpected result that AAA sites display lower ET rates
than AAB notwistanding the higher DOS and $C_{q}$ of AAA than AAB (Fig. 3A).
Strikingly, we also find that ABB sites yield slower ET kinetics than both AAB
(despite identical overall DOS) and AA (despite higher overall DOS). Thus,
while in-plane electronic localization and structural relaxation effects
explain the dependence of $k^{0}$ on $\theta_{m}$ in TTG, the relative
interfacial ET rates of AAB (M-t-B), ABB (B-t-M), and AAA (A-t-A) (Fig. 3C
inset and Fig. 5A) appear not to correlate with DOS.

To explain these trends, Figs. 5C–E show layer-isolated local DOS($\epsilon$)
profiles (Figs. 5C,E) at the topological defects (AAB/ABB, AAA) along with
calculated real-space DOS maps (insets in Figs. 5C,E). These calculations show
how the DOS enhancements at AAB sites are distinctly localized on the top two
layers of M-t-B structures (i.e. the ‘AA’ portions of AAB)[44]. In contrast,
the DOS at AAA sites are most strongly localized on the middle layer of A-t-A.
This three-dimensional electronic localization (within a thickness of only
three atomic layers) arising from different symmetries of these topological
defects unveils the fundamental basis for the unexpected trends in ET rate
constants at AAB, ABB, and AAA (Figs. 3C and 5A): though the electrodes are
only three atomic layers thick, ET rate constants are only correlated with the
electronic properties precisely at the electrode–electrolyte interface.

These observations strongly hint at the role of interfacial electronic
coupling (between the localized states on the electrode and the electron
donor/acceptor in solution), electric double-layer effects, and/or interfacial
reorganization energy as even more crucial than the overall DOS alone. Indeed,
theoretical calculations based on the MHC model that accounts only for the
$\theta_{m}$-dependent DOS, but with a coupling strength, $\nu$, and
reorganization energy, $\lambda$, that are invariant with $\theta_{m}$, vastly
underestimate the dependence of $k^{0}$ on $\theta_{m}$. These MHC
calculations also likewise predict identical interfacial ET rates for M-t-B
and B-t-M, which is clearly at odds with the experiment. Our experimental
results, therefore, now motivate future theoretical work to adapt these MHC
models to consider how electronic localization, which is deterministically
tuned here by varying $\theta_{m}$ or TTG structure, modifies $\nu$[45] —
and/or $\lambda$[46], to bridge the gap between theory and experiment and
extend our microscopic understanding of interfacial ET.

## Conclusions

Controlling stacking geometries and twist angles in few-layer graphene
therefore enables the manipulation of standard ET rate constants over three
orders of magnitude. In particular, energetically unfavorable topological
defects (AAA, AAB stacking domains), which are attainable only through the
construction of a moiré superlattice, exhibit extraordinarily high standard
rate constants. This electrochemical behavior arises from the moiré-derived
flat bands that are localized in these topological defects. In addition to the
effects of in-plane structural relaxation and electronic localization, the
out-of-plane localization of the electron wavefunction on specific layers of
twisted trilayer graphene results in measurable differences in ET rates at
topological defects possessing different symmetries.

These results provide a powerful demonstration of the sensitivity of
interfacial ET kinetics to the three-dimensional localization of electronic
states at electrochemical surfaces, and raise the question of whether
traditional measurements of ET rates at macroscopic electrodes might severely
underestimate the true local rate constant, which may be mediated by atomic
defects that strongly localize electronic DOS, at these interfaces. In turn,
SECCM measurements are shown to be powerful tools for probing layer-dependent
electronic localization in atomic heterostructure electrodes.

Future experimental and theoretical work is needed to shed more light on the
microscopic origin of these electron-transfer modulations in the context of
reorganization energy, electronic coupling, and even electric double-layer
structure. This work also heralds the use of moiré materials as a versatile
and systematically tunable experimental platform for theoretical adaptations
of the Marcus–Hush–Chidsey framework applied to interfaces with localized
electronic states, which are representative of defective surfaces that are
ubiquitous to nearly all real electrochemical systems. In an applied context,
twistronics is shown to be a powerful pathway for engineering pristine 2D
material surfaces to execute charge transfer processes with facile kinetics,
holding implications for electrocatalysis[47, 48] and other energy conversion
device schemes that could benefit from ultrathin, flexible, and/or transparent
electrodes that retain high electron-transfer kinetics.

## Materials and Methods

Chemicals
Natural Kish graphite crystals were purchased from Graphene Supermarket.
Si/SiO2 wafers (0.5 mm thick with 285 nm SiO2 or 90 nm SiO2) were purchased
from NOVA Electronic Materials. Polydimethylsiloxane (PDMS) stamps were
purchased from MTI Corporation. Sn/In alloy was purchased from Custom
Thermoelectric. Poly(bisphenol-A carbonate) (Mw 45000), dichlorodimethylsilane
($>$99.5%), hexaammineruthenium(III) chloride (98%), cobalt(II) chloride
hexahydrate (98%), 1,10-phenanthroline ($>$99%), calcium chloride ($>$93%) and
potassium chloride ($>$99%) were purchased from Sigma-Aldrich and used as
received. All aqueous electrolyte solutions were prepared with type I water
(EMD Millipore, 18.2 M$\Omega$ cm resistivity). The 2 mM solutions of
tris(1,10-phenanthroline)cobalt(II) were prepared by dissolving 1:3 molar
ratios of solid cobalt(II) chloride and 1,10-phenanthroline in water. In both
Ru(NH3)6^3+ and Co(phen)3^3+ solutions, 0.1 M of KCl was added as a supporting
electrolyte.

Sample Fabrication
Graphite and hexagonal boron nitride (hBN) were exfoliated from the bulk
crystals with Scotch tape. Exfoliated films were surveyed by an optical
microscope (Laxco LMC-5000). Monolayer, bilayer and trilayer graphene were
identified with their characteristic optical contrast of 7%, 12% and 18%
respectively in the green channel[49]. Trilayer graphene films were further
confirmed by Raman spectroscopy (HORIBA LabRAM Evo) of the 2D peak (around
2600 – 2700 cm-1)[34]. The 2D peak was used to distinguish different stacking
domains (ABC/ABA) as ABC trilayer graphene exhibits an enhanced shoulder at
around 2640 cm-1. Trilayer graphene and twisted trilayer graphene samples were
fabricated by the well-established ‘cut and stack’ dry transfer method[20].
All transfers were done on a temperature-controlled heating stage (Instec), an
optical microscope (Mitutoyo FS70) and a micromanipulator (MP-285, Sutter
Instrument). For monolayer twist bilayer or bilayer twist monolayer samples,
graphene flakes with both bilayer and monolayer parts were carefully selected.
The monolayer section was severed from the bilayer with a scanning tunneling
microscopy (STM) tip. For a-twist-a samples, a large piece of graphene ($>$ 50
µm by 20 µm) was cut evenly into three pieces. A thin piece of
poly(bisphenol-A carbonate) (PC) film ($\sim$ 3 by 3 mm) attached to a PDMS
chunk ($\sim$ 7 by 7 mm) was used to pick up an hBN ($\sim$ 10 – 20 nm) from
the SiO2/Si substrate at 120 °C. This hBN was carefully aligned with the
bottom layer of the graphene stack and lowered to pick up that piece. The
stage was rotated (usually to a slightly larger angle than the desired twist)
and the second piece of graphene was overlapped by the already picked-up
graphene and thus delaminated from the substrate. For a-twist-a samples, a
third piece of graphene was picked up after the stage was rotated back to the
original orientation. A piece of graphite ($\sim$ 20 nm, $>$ 50 µm by 50 µm)
was then picked up such that it was connected to the graphene. The PC film was
carefully removed from the PDMS and placed onto a clean SiO2/Si. In/Sn was
painted onto the graphite via micro-soldering[50] to a metallic plate which is
attached beneath the SiO2/Si.

Finite Element Simulation and Cyclic Voltammograms Fitting
All finite element simulations of electron transport were performed on COMSOL
Multiphysics v5.6 (COMSOL) to capture the effects of quantum capacitance. The
fitting of the CVs was achieved by statistical analysis of experimental and
simulated CVs.

Raman Mapping
Confocal Raman spectra were collected by recording from 2550 – 2800 cm-1 with
a 532 nm laser at 3.2 mW. Raman maps were generated by collecting spectrum
across the trilayer films with a step size of 2 µm. The spectrum was fitted
with single Lorentzian functions. The full-width half maxima of the fitted
functions were used to differentiate ABA and ABC trilayers.

PFM Measurements
PFMs were performed on AIST-NT OmegaScope Reflection. Ti/Ir coated silicon
probes from Nanosensor with a force constant of 2.8 Nm-1 and resonance
frequency of 75 kHz were used. 2 V of AC bias with resonance frequencies at
820 kHz was used and the force was set at 25 nN.

STM Measurements
STM measurements were conducted using a Park NX10 STM module (Park Systems) at
room temperature and atmospheric pressure. Pt-Ir tips were prepared by
electrochemical etching of 0.25 mm Pt-Ir wires (Nanosurf) in 1.5 M CaCl2
solutions[51]. The scan images were taken with 0.2 V tip-sample bias and 100
pA current set point. Twist angles of various samples were determined using
Delaunay triangulation on the Gaussian centers[20, 19].

Electron Microscopy Measurements
The transmission electron microscopy images of the nanopipettes were obtained
with a JEOl 1200EX transmission electron microscope operated at 100 keV. The
top $\sim$ 1 mm portion of the piette was attached to the grid (PELCO Hole
Grids) such that the piette tip was positioned in the cetner hole, and the
rest of the pipette was broken off. Selected area electron diffraction
patterns were collected on a FEI Tecnai T20 S-TWIN transmission electron
microscope with a LaB6 filament operated at 200 kV. Selected area electron
diffraction was used to resolve the twist anlges for samples with twist angles
larger than $3\degree$. To obtain the diffraction patterns, the fabricated
TLG/hBN samples were measured at the National Center for Electron Microscopy
facility in the Molecular Foundry at Lawrence Berkeley National Laboratory.
Low-magnification DF-TEM images were acquired using a Gatan UltraScan camera
on a Thermo Fisher Scientific Titan-class microscope operated at 60 kV.

Calculation of band structure and DOS
The DOS for trilayer graphene structures was calculated as a function of
$\theta_{m}$ using the ab initio perturbation continuum model developed
previously [52]. The low-energy electronic structure is based on a momentum
expansion about the valley K point of the super-cell Brillouin zone, allowing
a smooth dependence of bands on the twist angle. It has been shown that the
perturbation continuum model exactly reproduces the results of the more
expensive ab initio tight-binding model, and both are in good agreement with
full density functional theory (DFT) calculations [52, 53, 54, 55]. The energy
range of integration for the DOS was fixed at $\pm$0.5 eV around the charge
neutrality point (CNP). For evaluation of the LDOS, the normalized moiré
supercell was divided into a 90 $\times$ 90 grid in real space and sampled
over 36 k points in the Brillouin zone. We kept the sublattice symmetry intact
and assumed no extra screening of the interlayer coupling constants.

Quantum Capacitance Calculation
Quantum capacitance ($C_{q}$) describes the variation of electrical charges
with respect to the chemical potential ($V_{q}$). Theoretical $C_{q}$ values
with respect to $V_{q}$ was calculated based on the following equation[56]:

$C_{\mathrm{q}}=e^{2}\mathop{\int}\limits_{-\infty}^{+\infty}D\left({{\epsilon}}\right)F_{\mathrm{T}}\left({{{\epsilon}}-eV_{\mathrm{q}}}\right){\mathrm{d}}{{\epsilon}}$
(3)

$F_{\mathrm{T}}\left({\it{\epsilon}}\right)=\left({4k_{\mathrm{B}}T}\right)^{-1}{\mathrm{sech}}^{2}\left({{\it{\epsilon}}/2k_{\mathrm{B}}T}\right)$
(4)

where $D(\epsilon)$ is the density of states, which we center at the CNP,
$F_{T}(\epsilon)$ is the thermal broadening function and $kB$ is Boltzmann’s
constant. We assumed $T$ = 300 K for our experimental conditions. The total
electric double-layer capacitance is governed by the compact layer
capacitance. Hence, we used a constant $C_{dl}$ = 10 µF cm-2 to simplify the
calculation [57]. We solved the self-consistent equations relating $V_{app}$,
$V_{q}$, $V_{dl}$, $C_{q}$ and $C_{dl}$ using Simpson integration and
nonlinear least squares

$V_{{\mathrm{app}}}=V_{{\mathrm{dl}}}+V_{\mathrm{q}}$ (5)

$\frac{{V_{{\mathrm{dl}}}}}{{V_{\mathrm{q}}}}=\frac{{C_{\mathrm{q}}}}{{C_{{\mathrm{dl}}}}}$
(6)

to obtain $C_{q}$ vs. $V_{q}$ and $V_{dl}/V_{app}$ vs. $V_{app}$ shown in Fig.
3.

SECCM Measurements
The SECCM nanopipettes were fabricated from single-channel quartz capillaries
(inner and out diameters of 0.7 mm and 1.0 mm from Sutter Instrument) in a
laser nano pipette puller (Sutter Instrument Model 2000). The program was set
to heat 700, filament 4, velocity 20, delay 127 and pull 140 to generate
pipettes of diameters around 200 nm, later confirmed with bright field
TEM[20]. The outer surfaces of the pipettes were silanized by dipping them
into dichlorodimethylsilane for 3 - 5 seconds when nitrogen flowed through the
inside of the pipettes. They were then filled with either Ru(NH3)6^3+ or
Co(phen)3^3+ solutions through a micro syringe. The pipettes were gently
tapped and a gentle string of nitrogen was used to eliminate the bubbles. The
pipettes were then inserted with a silver AgCl wire as a quasi-counter
reference electrode (QCRE). The pipettes were carefully approached (0.2 µm/s)
to the locations of interest while a -0.5 V (0.5 V for Co(phen)3^3+) bias was
applied. The meniscus achieved contact when a current of larger than 2 pA (or
smaller than -2 pA) was observed. The pipette was allowed to stabilize for 30
s. Cyclic voltammograms (CVs) were then conducted by sweeping the potential at
100 mVs-1 between -0.6 to 0 V (0 to 0.8 V for Co(phen)3^3+/2+) for 5 cycles.
Multiple CVs were collected for each sample and for small twist samples
($\theta\leq$ 0.15°) with moiré wavelengths of more than 80 nm, only CVs
recorded with nanopipettes more than 200 nm in diameter were included to
ensure they surveyed multiple stacking domains. To survey electrochemical
activities across a large sample, the pipette was retracted by 1 µm after CVs
were measured and horizontally moved to a new location for a new approach.

## Acknowledgements

This material is based upon work supported by the US Department of Energy,
Office of Science, Office of Basic Energy Sciences, under award no. DE-
SC0021049 (experimental studies by K.Z., Y.Y., B.K., N.K., and D.K.B.) and the
Office of Naval Research under award no. N00014-18-S-F009 (computational work
by M.B. and V.V.). S.C. acknowledges support from the National Science
Foundation under grant no. OIA-1921199. C.G. was supported by a grant from the
W.M. Keck Foundation (Award no. 993922). Experimental work at the Molecular
Foundry, LBNL was supported by the Office of Science, Office of Basic Energy
Sciences, the U.S. Department of Energy under Contract no. DE-AC02-05CH11231.
Confocal Raman spectroscopy was supported by a Defense University Research
Instrumentation Program grant through the Office of Naval Research under award
no. N00014-20-1-2599 (D.K.B.). Other instrumentation used in this work was
supported by grants from the Canadian Institute for Advanced Research
(CIFAR–Azrieli Global Scholar, Award no. GS21-011), the Gordon and Betty Moore
Foundation EPiQS Initiative (Award no. 10637), and the 3M Foundation through
the 3M Non-Tenured Faculty Award (no. 67507585). K.W. and T.T. acknowledge
support from JSPS KAKENHI (Grant Numbers 19H05790, 20H00354 and 21H05233). We
thank Isaac M. Craig for the helpful discussion regarding STM analysis.

## Author Contributions

K.Z., Y.Y., and D.K.B. conceived the study. K.Z., B.K., C.G., and N.K.
performed the experiments. K.Z. performed the COMSOL simulations. M.B., S.C.
and V.V. carried out the theoretical calculations. K.Z. performed the quantum
capacitance calculations. K.Z. and B.K. performed STM image analysis. T.T. and
K.W. provided the hBN crystals. K.Z., Y.Y., and D.K.B. analysed the data. K.Z.
and D.K.B. wrote the manuscript with input from all co-authors.

## References

  * [1] Zhi Wei Seh et al. “Combining theory and experiment in electrocatalysis: Insights into materials design” In _Science_ 355.6321 American Association for the Advancement of Science, 2017, pp. eaad4998
  * [2] Jens Kehlet Nørskov, Thomas Bligaard, Jan Rossmeisl and Claus Hviid Christensen “Towards the computational design of solid catalysts” In _Nature Chemistry_ 1.1 Nature Publishing Group, 2009, pp. 37–46
  * [3] Jonathan Hwang et al. “Perovskites in catalysis and electrocatalysis” In _Science_ 358.6364 American Association for the Advancement of Science, 2017, pp. 751–756
  * [4] Wolfgang Schmickler and Elizabeth Santos “Interfacial electrochemistry” Springer Science & Business Media, 2010
  * [5] Allen J Bard, Larry R Faulkner and Henry S White “Electrochemical methods: fundamentals and applications” John Wiley & Sons, 2022
  * [6] Christopher E.. Chidsey “Free Energy and Temperature Dependence of Electron Transfer at the Metal-Electrolyte Interface” In _Science_ 251.4996, 1991, pp. 919–922
  * [7] Martin C Henstridge, Eduardo Laborda, Neil V Rees and Richard G Compton “Marcus–Hush–Chidsey theory of electron transfer applied to voltammetry: A review” In _Electrochimica Acta_ 84 Elsevier, 2012, pp. 12–20
  * [8] Rachel Kurchin and Venkatasubramanian Viswanathan “Marcus–Hush–Chidsey kinetics at electrode–electrolyte interfaces” In _The Journal of Chemical Physics_ 153.13 AIP Publishing LLC, 2020, pp. 134706
  * [9] Hong Li et al. “Activating and optimizing MoS2 basal planes for hydrogen evolution through the formation of strained sulphur vacancies” In _Nature Materials_ 15.1 Nature Publishing Group, 2016, pp. 48–53
  * [10] Thomas F. Jaramillo et al. “Identification of Active Edge Sites for Electrochemical H2 Evolution from MoS2 Nanocatalysts” In _Science_ 317.5834, 2007, pp. 100–102 DOI: 10.1126/science.1141483
  * [11] Aleix G Güell et al. “Redox-dependent spatially resolved electrochemistry at graphene and graphite step edges” In _ACS Nano_ 9.4 ACS Publications, 2015, pp. 3558–3571
  * [12] Patrick R Unwin, Aleix G Guell and Guohui Zhang “Nanoscale electrochemistry of sp2 carbon materials: from graphite and graphene to carbon nanotubes” In _Accounts of Chemical Research_ 49.9 ACS Publications, 2016, pp. 2041–2048
  * [13] Kyounghwan Kim et al. “van der Waals heterostructures with high accuracy rotational alignment” In _Nano Letters_ 16.3 ACS Publications, 2016, pp. 1989–1995
  * [14] Chun Ning Lau, Marc W Bockrath, Kin Fai Mak and Fan Zhang “Reproducibility in the fabrication and physics of moiré materials” In _Nature_ 602.7895 Nature Publishing Group, 2022, pp. 41–50
  * [15] Yuan Cao et al. “Correlated insulator behaviour at half-filling in magic-angle graphene superlattices” In _Nature_ 556.7699 Nature Publishing Group, 2018, pp. 80–84
  * [16] Zeyu Hao et al. “Electric field–tunable superconductivity in alternating-twist magic-angle trilayer graphene” In _Science_ 371.6534 American Association for the Advancement of Science, 2021, pp. 1133–1138
  * [17] Guorui Chen et al. “Evidence of a gate-tunable Mott insulator in a trilayer graphene moiré superlattice” In _Nature Physics_ 15.3 Nature Publishing Group, 2019, pp. 237–241
  * [18] Leon Balents, Cory R Dean, Dmitri K Efetov and Andrea F Young “Superconductivity and strong correlations in moiré flat bands” In _Nat Phys_ 16.7 Nature Publishing Group, 2020, pp. 725–733
  * [19] Alexander Kerelsky et al. “Maximized electron interactions at the magic angle in twisted bilayer graphene” In _Nature_ 572.7767 Nature Publishing Group, 2019, pp. 95–100
  * [20] Yun Yu et al. “Tunable angle-dependent electrochemistry at twisted bilayer graphene with moiré flat bands” In _Nature Chemistry_ 14.3 Nature Publishing Group, 2022, pp. 267–273
  * [21] Tataiana Latychevskaia et al. “Stacking transition in rhombohedral graphite” In _Frontiers of Physics_ 14.1 Springer, 2019, pp. 1–7
  * [22] Ammon Fischer et al. “Unconventional superconductivity in magic-angle twisted trilayer graphene” In _npj Quantum Materials_ 7.1 Nature Publishing Group, 2022, pp. 1–10
  * [23] Sylvain Latil and Luc Henrard “Charge carriers in few-layer graphene films” In _Physical Review Letters_ 97.3 APS, 2006, pp. 036803
  * [24] Rui Xu et al. “Direct probing of the stacking order and electronic spectrum of rhombohedral trilayer graphene with scanning tunneling microscopy” In _Physical Review B_ 91.3 APS, 2015, pp. 035410
  * [25] Ziyan Zhu et al. “Twisted trilayer graphene: A precisely tunable platform for correlated electrons” In _Physical Review Letters_ 125.11 APS, 2020, pp. 116404
  * [26] Hongyuan Li et al. “Global control of stacking-order phase transition by doping and electric field in few-layer graphene” In _Nano Letters_ 20.5 ACS Publications, 2020, pp. 3106–3112
  * [27] Shuai Zhang et al. “Domino-like stacking order switching in twisted monolayer–multilayer graphene” In _Nature Materials_ 21.6 Nature Publishing Group, 2022, pp. 621–626
  * [28] Youngju Park, Bheema Lingam Chittari and Jeil Jung “Gate-tunable topological flat bands in twisted monolayer-bilayer graphene” In _Physical Review B_ 102.3 APS, 2020, pp. 035411
  * [29] Jeong Min Park et al. “Tunable strongly coupled superconductivity in magic-angle twisted trilayer graphene” In _Nature_ 590.7845 Nature Publishing Group, 2021, pp. 249–255
  * [30] Haoxin Zhou et al. “Superconductivity in rhombohedral trilayer graphene” In _Nature_ 598.7881 Nature Publishing Group, 2021, pp. 434–438
  * [31] Guorui Chen et al. “Tunable correlated Chern insulator and ferromagnetism in a moiré superlattice” In _Nature_ 579.7797 Nature Publishing Group, 2020, pp. 56–61
  * [32] Shuigang Xu et al. “Tunable van Hove singularities and correlated states in twisted monolayer–bilayer graphene” In _Nature Physics_ 17.5 Nature Publishing Group, 2021, pp. 619–626
  * [33] Si-yu Li et al. “Imaging topological and correlated insulating states in twisted monolayer-bilayer graphene” In _Nature Communications_ 13.1 Nature Publishing Group, 2022, pp. 1–7
  * [34] Chun Hung Lui et al. “Imaging stacking order in few-layer graphene” In _Nano Letters_ 11.1 ACS Publications, 2011, pp. 164–169
  * [35] Chunxiao Cong et al. “Raman characterization of ABA-and ABC-stacked trilayer graphene” In _ACS Nano_ 5.11 ACS Publications, 2011, pp. 8760–8768
  * [36] Leo J McGilly et al. “Visualization of moiré superlattices” In _Nature Nanotechnology_ 15.7 Nature Publishing Group, 2020, pp. 580–584
  * [37] Simon Turkel et al. “Orderly disorder in magic-angle twisted trilayer graphene” In _Science_ 376.6589 American Association for the Advancement of Science, 2022, pp. 193–199
  * [38] Hyobin Yoo et al. “Atomic and electronic reconstruction at the van der Waals interface in twisted bilayer graphene” In _Nature Materials_ 18.5 Nature Publishing Group, 2019, pp. 448–453
  * [39] Nathanael P Kazmierczak et al. “Strain fields in twisted bilayer graphene” In _Nature Materials_ 20.7 Nature Publishing Group, 2021, pp. 956–963
  * [40] “Strain solitons and topological defects in bilayer graphene” In _Proceedings of the National Academy of Sciences_ 110.28, 2013, pp. 11256–11260
  * [41] Rebecca Engelke et al. “Non-Abelian topological defects and strain mapping in 2D moiré materials” arXiv, 2022 DOI: 10.48550/ARXIV.2207.05276
  * [42] Stephen Carr et al. “Relaxation and domain formation in incommensurate two-dimensional heterostructures” In _Physical Review B_ 98.22 APS, 2018, pp. 224102
  * [43] E Santos, T Iwasita and W Vielstich “On the use of the coulostatic method for the investigation of fast redox systems” In _Electrochimica acta_ 31.4 Elsevier, 1986, pp. 431–437
  * [44] Ling-Hui Tong et al. “Spectroscopic visualization of flat bands in magic-angle twisted monolayer-bilayer graphene: coexistence of localization and delocalization” In _Physical Review Letters_ 128.12 APS, 2022, pp. 126401
  * [45] Sergey V Pavlov, Renat R Nazmutdinov, Maxim V Fedorov and Sergey A Kislenko “Role of graphene edges in the electron transfer kinetics: insight from theory and molecular modeling” In _The Journal of Physical Chemistry C_ 123.11 ACS Publications, 2019, pp. 6627–6634
  * [46] Aditya M. Limaye, Wendu Ding and Adam P. Willard “Understanding attenuated solvent reorganization energies near electrode interfaces” In _The Journal of Chemical Physics_ 152.11, 2020, pp. 114706
  * [47] Yun Yu, Madeline Van Winkle and D. Bediako “Tuning interfacial chemistry with twistronics” In _Trends in Chemistry_ 4, 2022, pp. 857–859
  * [48] Yang Zhang, Claudia Felser and Liang Fu “Moiré metal for catalysis” arXiv, 2021 DOI: 10.48550/ARXIV.2111.03058
  * [49] Hai Li et al. “Rapid and reliable thickness identification of two-dimensional nanosheets using optical microscopy” In _ACS Nano_ 7.11 ACS Publications, 2013, pp. 10344–10353
  * [50] Çağlar Ö Girit and Alex Zettl “Soldering to a single atomic layer” In _Applied Physics Letters_ 91.19 American Institute of Physics, 2007, pp. 193512
  * [51] L Libioulle, Y Houbion and J-M Gilles “Very sharp platinum tips for scanning tunneling microscopy” In _Review of Scientific Instruments_ 66.1 American Institute of Physics, 1995, pp. 97–100
  * [52] Stephen Carr, Shiang Fang, Ziyan Zhu and Efthimios Kaxiras “Exact continuum model for low-energy electronic states of twisted bilayer graphene” In _Physical Review Research_ 1.1 APS, 2019, pp. 013001
  * [53] Stephen Carr, Shiang Fang and Efthimios Kaxiras “Electronic-structure methods for twisted moiré layers” In _Nature Reviews Materials_ 5.10 Nature Publishing Group, 2020, pp. 748–763
  * [54] Shiang Fang and Efthimios Kaxiras “Electronic structure theory of weakly interacting bilayers” In _Physical Review B_ 93.23 APS, 2016, pp. 235153
  * [55] Procolo Lucignano et al. “Crucial role of atomic corrugation on the flat bands and energy gaps of twisted bilayer graphene at the magic angle $\theta$ 1.08 $\degree$” In _Physical Review B_ 99.19 APS, 2019, pp. 195419
  * [56] GM Yang, HZ Zhang, XF Fan and WT Zheng “Density functional theory calculations for the quantum capacitance performance of graphene-based electrode material” In _The Journal of Physical Chemistry C_ 119.12 ACS Publications, 2015, pp. 6464–6470
  * [57] Jilin Xia, Fang Chen, Jinghong Li and Nongjian Tao “Measurement of the quantum capacitance of graphene” In _Nature Nanotechnology_ 4.8 Nature Publishing Group, 2009, pp. 505–509


# DeOccNet: Learning to See Through Foreground Occlusions in Light Fields

Yingqian Wang1, Tianhao Wu1, Jungang Yang1,
Longguang Wang1, Wei An1, Yulan Guo1,2

1College of Electronic Science and Technology, National University of Defense
Technology, China
2School of Electronics and Communication Engineering, Sun Yat-sen University,
China
{wangyingqian16, yangjungang<EMAIL_ADDRESS>Corresponding author

###### Abstract

Background objects occluded in some views of a light field (LF) camera can be
seen by other views. Consequently, occluded surfaces are possible to be
reconstructed from LF images. In this paper, we handle the LF de-occlusion
(LF-DeOcc) problem using a deep encoder-decoder network (namely, DeOccNet). In
our method, sub-aperture images (SAIs) are first given to the encoder to
incorporate both spatial and angular information. The encoded representations
are then used by the decoder to render an occlusion-free center-view SAI. To
the best of our knowledge, DeOccNet is the first deep learning-based LF-DeOcc
method. To handle the insufficiency of training data, we propose an LF
synthesis approach to embed selected occlusion masks into existing LF images.
Besides, several synthetic and real-world LFs are developed for performance
evaluation. Experimental results show that, after training on the generated
data, our DeOccNet can effectively remove foreground occlusions and achieves
superior performance as compared to other state-of-the-art methods. Source
codes are available at: https://github.com/YingqianWang/DeOccNet.

##  1 Introduction

Seeing through foreground occlusions is beneficial to many computer vision
applications such as detection and tracking in surveillance [5, 13, 32, 33].
However, due to foreground occlusions, some rays cannot hit the sensors of
traditional single-view cameras (_e.g_., Digital Single Lens Reflex).
Therefore, objects behind occlusions cannot be fully observed and reliably
reconstructed. In recent years, camera arrays [26, 25, 18, 19, 8] have
undergone a rapid development since they can record light fields (LFs) and
provide a large number of viewpoints with rich angular information. The
complementary information among different viewpoints is beneficial for the
reconstruction of occluded surfaces since background objects occluded in some
views can be seen by other views.

|
---|---
(a) | (b)
|
(c) | (d)
Figure 1: An illustration of LF-DeOcc using our rendered scenes Syn01. (a)
Configuration of the scene. Yellow boxes with $5\times 5$ blocks represent
camera arrays. (b) Occluded center-view SAI. (c) Results of our DeOccNet. (d)
Occlusion-free groundtruth.

As illustrated in Fig. 1, light field de-occlusion (LF-DeOcc) aims at removing
foreground occlusions using sub-aperture images (SAIs) captured by a camera
array111In the area of LF-DeOcc, images captured by camera arrays are widely
used due to their wide baselines. Therefore, we follow the existing work [18,
12, 34, 30, 11] and use camera arrays for LF-DeOcc.. The pioneering work on
LF-DeOcc is proposed by Vaish _et al_. [18] using a refocusing method.
However, this method cannot recover a clean surface of occluded objects since
rays from occlusions and background are mixed. In fact, it is important but
challenging to correctly select pixels only belonging to occluded objects. To
this end, existing methods [12, 34, 30, 11] generally built different models
to handle LF-DeOcc problem. Due to the highly complex structures of scenes in
real world, these methods with handcrafted feature extraction and stereo
matching techniques cannot achieve satisfactory performance. In recent years,
deep learning has been successfully used in different LF tasks such as depth
estimation [15, 14], image super-resolution [38, 22, 35], view synthesis [28,
21, 27], and LF intrinsics [2, 1]. These networks have achieved state-of-the-
art performance in numerous areas. However, to the best of our knowledge, deep
learning has not been used for LF-DeOcc due to several issues. In this paper,
we design a novel and effective paradigm, and propose the first deep learning
network (_i.e_., DeOccNet) to handle LF-DeOcc problem. Specifically, we
summarize three major challenges in deep learning-based LF-DeOcc, and provide
solutions to these challenges using our proposed paradigm.

The first challenge is that, as compared to LF depth estimation networks [15,
14] and LF super-resolution networks [38, 22, 35], LF-DeOcc networks should
use as much information from occluded surfaces as possible, while maintaining
a larger receptive field to cover occlusions of different types and scales. We
address this challenge by employing an encoder-decoder network to encode LF
structures. We concatenate all SAIs along the channel dimension to fully use
the information of occluded surfaces. Besides, we use a residual atrous
spatial pyramid pooling (ASPP) module to extract multi-scale features and
enlarge receptive fields.

The second challenge is that, as compared to single image inpainting networks
[10, 31, 37, 9], LF-DeOcc networks have to learn the scene structure to
automatically recognize, label and remove foreground occlusions. We address
this challenge by setting the occlusion-free center-view SAI as groundtruth,
and train our DeOccNet in an end-to-end manner. In this way, our network can
recognize occlusions from background through disparity discrepancy, and
automatically remove foreground occlusions.

The third challenge is that, LF-DeOcc networks face an insufficiency of
training data since large-scale LF datasets with removable foreground
occlusions are unavailable. Moreover, test scenes are also insufficient for
performance evaluation. We address this challenge by proposing a data
synthesis approach to embed different occlusion masks into existing LF images.
Using this approach, more than 1000 LFs are generated to train our network.
Moreover, we develop several synthetic and real-world LFs for performance
evaluation.

Experimental results have demonstrated the effectiveness of our paradigm. Our
DeOccNet achieves superior performance on both synthetic and real-world scenes
as compared to other state-of-the-art methods.

##  2 Related Works

###  2.1 Single image inpainting

Single image inpainting methods aim at filling holes in an image using both
neighborhood information and global priors. The major challenge of single
image inpainting lies in synthesizing visually realistic and semantically
plausible pixels for missing regions. Recent deep learning based methods [10,
31, 37, 9] have achieved promising results for inpainting large missing
regions in an image. Specifically, Yu _et al_. [37] proposed a deep generative
model-based inpainting method to synthesize novel image structures and
textures. Liu _et al_. [9] used partial convolutions for inpainting with
irregular holes and achieved the state-of-the-art performance.

Compared to single image inpainting, LF-DeOcc can use complementary
information provided by SAIs to generate improved results. The difference
between single image inpainting and LF-DeOcc is significant. In single image
inpainting, holes or masks are always pre-defined. However, LF-DeOcc requires
automatical extraction of foreground occlusions by analyzing scene structures.
That is, occlusions are closer to cameras than background objects, and thus
have larger disparities. Due to the complex structures of real-world scenes,
it is highly challenging for algorithms to correctly select rays originating
from occluded objects.

Figure 2: An overview of our DeOccNet. (a) The overall architecture. (b) The
structure of the residual ASPP module.

###  2.2 Light field de-occlusion

LF-DeOcc is an active research topic and has been investigated for decades
[18, 17, 12, 34, 30, 11]. Vaish _et al_. [18] proposed a refocusing method by
warping each SAI by a specific value, and then averaging the warped SAIs along
angular dimension. Due to the large equivalent aperture of camera arrays, when
background is refocused on, occlusions are extremely blurred and the see
through effect can be achieved. However, the resulting images are always
blurred due to the indiscriminate use of rays from both occlusions and
background. Vaish _et al_. further proposed an improved version using both
median cost and entropy cost [17]. Since these methods [18, 17] do not exactly
exploit scene structures, their performance is limited for scenes with heavy
occlusions.

To solve this problem, Pei _et al_. proposed a pixel-labeling method [12] to
remove occlusions. Specifically, pixels corresponding to occlusions are
labeled by stereo matching and masked out during refocusing process, resulting
in a clean image. However, this method can only generate images refocused on a
specific depth, leaving objects in other depth ranges suffering from various
degrees of blurs. Subsequently, they used an image-matting approach to perform
all-in-focus synthetic aperture imaging [11]. Besides, Yang _et al_. used
visible layers [34] to address the all-in-focus imaging issue. Xiao _et al_.
[30] used k-means clustering to classify pixels of occlusions and background.
All these methods [12, 34, 30, 11] use handcrafted feature extraction and
stereo matching techniques, and cannot achieve a satisfactory performance in
scenes with complex structures and heavy occlusions.

###  2.3 Deep learning in light field

Deep neural networks have been widely used in various LF tasks such as image
super-resolution [36, 22, 35, 38], view synthesis [28, 21, 27], and depth
estimation [15, 14]. Compared to these tasks, networks for LF-DeOcc should
have a larger receptive field and use more information of occluded surfaces.
Currently, no existing work on deep learning based LF-DeOcc is available in
literature. It is worth noting that works in [2, 1] are similar to ours.
Specifically, a fully convolutional auto-encoder is proposed in [2] to
separate diffuse and specular components of an LF. Both [2] and our work
require high-level features and global priors of scene structures.
Consequently, we built our network upon encoder-decoder architecture to encode
LF structures. Note that, there are two significant differences between [2]
and our network. First, only horizontal and vertical SAIs (_e.g_., 9 cross-
views in a 5$\times$5 LF) are used in [2]. In contrast, all SAIs are used in
our network to fully exploit the information of occluded objects. Second, our
DeOccNet uses residual ASPP module to enlarge the receptive field, and uses
multiple skip layers to have a holistic understanding of the scene while
preserving fine details.

##  3 The Proposed Method

###  3.1 Network architecture

The task of our DeOccNet is to replace pixels of occlusions with pixels from
the background. To achieve this task, our network is required to find
correspondence and incorporate complementary information from SAIs. Note that,
foreground occlusions generally have shallow depths and large disparities.
That is, pixels of occlusions always have very large position variations among
SAIs. Therefore, multi-scale features with large receptive fields are required
for our network. In this paper, we first use a residual ASPP module for
hierarchical feature extraction, and then use an auto-encoder to incorporate
both spatial and angular information. The architecture of our DeOccNet is
shown in Fig. 2. Different from existing LF networks [2, 15, 38] where only
part of SAIs are stacked as inputs, we stacked all SAIs along the channel
dimension (_e.g_., $75$ channels for $5\times 5$ RGB SAIs) to use as much
information as possible since LF-DeOcc highly depends on the information of
occluded objects. Consequently, our DeOccNet takes the stacked SAIs as its
input, and finally generates an occlusion-free center-view SAI.

Residual ASPP module. In our network, the input volume is first processed by a
$1\times 1$ convolution layer to generate features with a fixed depth (_i.e_.,
$64$ in this paper). Then, a residual ASPP module is used to generate
hierarchical features. As shown in Fig. 2(b), similar to the module used in
[20], we first combine six dilated convolution layers (with dilation rates of
$1$, $2$, $4$, $8$, $16$, and $32$) to form an ASPP group, then cascade three
ASPP groups in residual manner to achieve high learning efficiency. The
residual ASPP module can enlarge the receptive field and extract multi-scale
information around occlusions. It is demonstrated in the ablation study (see
Table 1) that our residual ASPP module is beneficial to the overall LF-DeOcc
performance.

Figure 3: The structure of the encoder and decoder blocks. Note that, the encoder and decoder blocks share mirrored structures. That is, strided convolution is used in the third unit in each encoder block, while de-convolution is used in the first unit in each decoder block.  |
---|---
(a) | (b)
Figure 4: An illustration of our Mask Embedding approach. (a) Masks used in
our approach. Note that, cropping and scaling are performed for better
visualization. (b) The pipeline of our Mask Embedding approach. Here, a
$3\times 3$ LF is used as an example.

Encoder pathway. Features generated by the residual ASPP module are then
transferred to the encoder pathway, where $4$ encoder blocks are cascaded to
incorporate both spatial and angular information. Specifically, as shown in
Fig. 3, each encoder block contains three cascaded units. In each unit, the
batch-normalized features are given to two separated paths to achieve local
residual learning. The first path includes a $3\times 3$ convolution and a
Leaky ReLU (with a leaky factor of $0.1$), while the second path either keeps
the input unchanged or passes it through a strided convolution or de-
convolution. Therefore, features produced by the two paths have the same
resolution. Features from both paths are added to produce the output of this
unit. For each encoder block, the first two units keep depth and resolution
unchanged, while the third unit applies a strided convolution (with a stride
of $2$) to halve the resolution and double the feature depth. Consequently,
the final feature generated by the encoder (bottleneck of our DeOccNet) is
$1/16$ time of the input feature in resolution and has $1024$ channels in
depths.

Decoder pathway. After passing through the bottleneck, features are decoded
through the decoder pathway. Note that, decoder blocks have mirrored
structures as encoder blocks. That is, the decoder block is also chained by
three residual units, and the first unit uses a de-convolution to exactly
revert the encoder block on the corresponding level. Moreover, to preserve
fine details in the final output image, features of different resolutions on
the encoder pathway are concatenated with their counterparts on the decoder
pathway by skip connections. Consequently, the decoder can be guided to
gradually add details onto restored images. Since feature depth is doubled by
concatenation, an additional $1\times 1$ convolution layer is employed before
the last three decoder blocks to halve the feature depth. Similarly, a
$1\times 1$ convolution layer is also applied to the output feature to reduce
its channel to $3$.

###  3.2 Mask embedding for training data synthesis

It is important to provide sufficient data to train our DeOccNet. Although LFs
with removable occlusions can be acquired by capturing real-world scenes
with/without foreground occlusions, or by rendering synthetic scenes using
softwares such as 3dsMax222https://www.autodesk.eu/products/3ds-max/overview
and Blender333https://www.blender.org/, these approaches are significantly
labour-intensive and even infeasible. Consequently, it is important to design
an efficient approach to generate a large amount of data for network training.
In this paper, we propose Mask Embedding, a training data synthesis approach
to synthesize LFs with removable foreground occlusions. An illustration of our
Mask Embedding approach is shown in Fig. 4.

As shown in Fig. 4(a), we manually collected $80$ mask images from Internet
using tags such as salix leaves, grids, fences, and paper cuts. All these
masks are common foreground occlusions in daily life. Meanwhile, we collected
$60$ LFs from the Stanford LF dataset [16], the Old HCI 4D LF dataset [24],
the New HCI 4D LF benchmark [4], and the MIT Synthetic LF Archive [7]. Note
that, to improve the generalization capability of our network, the RGB
channels of both LFs and mask images are randomly shuffled, and the masks are
randomly selected to be embedded into shuffled LFs.

The pipeline of our Mask Embedding approach is illustrated in Fig. 4(b). We
randomly pick a mask from the mask set and then embed it into each SAI
according to LF configurations. Specifically, a disparity with a shallow depth
is randomly set and allocated to the selected mask. Then, the mask is warped
according to the angular coordinate of the target view and the allocated
disparity. Note that, bilinear interpolation is used when masks do not fall
into integer coordinates. Next, warped masks are added to each SAI, resulting
in LFs with foreground occlusions. Finally, refocusing is performed on each
generated LF to check the LF configuration.

Although the Mask Embedding approach can easily generate a large number of LFs
for training, the generated LFs only have occlusions in a single depth, which
is significantly different from real-word scenarios. To address this issue, we
use generated LFs as original LFs and repeat this process twice to synthesize
LFs with occlusions at two and three depth layers. Using the proposed
approach, we totally synthesize $1500$ LFs to train our models. Although LFs
synthesized by our approach only have fence-like and front-parallel
occlusions, experimental results show that, our network trained on the
synthetic data can generalize well to real-world cases (_e.g_., CD scene in
Fig. 6). That is, our network can successfully learn the scene structure
through disparity discrepancy using LFs synthesized by our Mask Embedding
approach.

###  3.3 Training details

We trained two models with $5\times 15$ and $5\times 5$ input SAIs. The
$5\times 15$ model was used for the Stanford CD scene [16], and the $5\times
5$ model was used for our self-developed scenes. During the training phase, we
used LFs synthesized by the Mask Embedding approach as the training data. All
the $60$ LFs were used for the $5\times 5$ model. In contrast, due to the
angular resolution limitation of existing LF datasets, only LFs from the
Stanford LF dataset [16] were used to generate training data for the $5\times
15$ model.

It is worth noting that foreground and background in a scene are relative
concepts. That is, some occlusions can also be considered as background
objects in multi-occlusion situations as shown in Fig. 5. Consequently, both
training and test scenes should be rectified to specific depths for LF-DeOcc.
In this paper, we perform rectification by cropping each SAI accordingly to
make occlusions have positive disparity values while backgrounds have negative
disparity values. In this way, our DeOccNet can effectively achieve LF-DeOcc
by simply removing objects with positive disparity values. Finally, we cropped
occluded SAIs into $224\times 224$ pixel patches by a stride of $112$, and
performed $2\times$ upsampling for data augmentation. Meanwhile, occlusion-
free center-view SAI was cropped and upsampled accordingly to generate
groundtruths.

Our DeOccNet was implemented in Pytorch on a PC with an Nvidia RTX 2080Ti GPU.
All models were trained using an MSE loss and optimized using the Adam method
[6] with $\beta_{1}=0.9$, $\beta_{2}=0.999$, and a batch size of $8$. The
initial learning rate was set to $1\times 10^{-3}$ and reduced to $1\times
10^{-4}$ after $100$ epochs. The training took about $2$ days and was stopped
at $200$ epochs.

Figure 5: Multi-occlusion situation in LF-DeOcc. (a) Occluded center-view SAI. (b) Result of our DeOccNet with inputs being rectified at a shallow depth (the blue dotted line in (a)). The front-most tree is considered as a foreground occlusion. (c) Result of our DeOccNet with inputs being rectified at a deep depth (the red dotted line in (a)). Three front trees are considered as foreground occlusions. Consequently, different results can be generated by our DeOccNet with the same inputs being rectified at different depth values. |  |  |
---|---|---|---
(a) Occluded |  (b) Refocus [18] |  (c) Median [17] |  (d) Pei _et al_. [11]
|  |  |
(e) Liu _et al_. [9] |  (f) DeOccNet (75 center views) |  (g) DeOccNet | (h) Groundtruth
Figure 6: Qualitative results achieved on the CD scene [16] (with an occlusion
rate of $40.2\%$). (a) Occluded center-view SAI. (b)-(e) Comparative results
achieved by different methods. (f) Result achieved by our DeOccNet using $75$
identical center-view SAIs as its inputs (discussed in Section 4.4). (g) Our
result. (h) Occlusion-free center-view SAI.

##  4 Experiments

In this section, we first introduce the test scenes used in our experiments,
and then compare our method to several state-of-the-art methods. Finally, we
present ablation study and analyses.

###  4.1 Test scenes

Real-world scenes. We followed [11, 17, 34] and tested our method on the
publicly available CD scene [16]. The original CD scene consists of 105 views
distributed on a $5\times 21$ grid. We selected the central $5\times 15$ views
for performance evaluation. Groundtruth image is provided by a second capture
with occlusions being removed. Besides, we captured several real-world scenes
using a moving Leica Q camera (with an $F=10$, $f=28$ mm lens) mounted on a
gantry. It is argued in [29, 23] that the scanning scheme is equivalent to a
single shot by a camera array in static occasions. We shifted the camera to
$25$ positions on a $5\times 5$ grid with $3$ cm baselines. The captured
images were calibrated using the method in [39].

Synthetic scenes. Since the number of real-world test scenes is very small, we
rendered $4$ synthetic scenes with removable foreground occlusions for further
evaluation. All elements in our synthetic scenes were collected from Internet,
and parameters (_e.g_., lighting, depth range) were tuned to better reflect
real scenes. The angular resolution of each scene was set to $5\times 5$,
while baselines and occlusion ranges were varied in different scenes.
Occlusion-free center-view SAIs were also rendered for quantitative
evaluation.

|  |  |  |
---|---|---|---|---
(a) Occluded (Bike01) |  (b) Refocus [18] |  (c) Liu _et al_. [9] |  (d) Pei _et al_. [11] | (e) Ours
|  |  |  |
(f) Occluded (Bike02) |  (g) Refocus [18] |  (h) Liu _et al_. [9] |  (i) Pei _et al_. [11] | (j) Ours
|  |  |  |
(k) Occluded (Handrail) |  (l) Refocus [18] |  (m) Liu _et al_. [9] |  (n) Pei _et al_. [11] | (o) Ours
Figure 7: Qualitative results achieved on our self-developed real-world scenes. Note that, the occlusion rates of the three scenes are $61.8\%$, $57.7\%$, and $39.1\%$, respectively. |  |  |  |
---|---|---|---|---
(a) Overview (Syn02) |  (b) Liu _et al_. [9] |  (c) Pei _et al_. [11] | (d) Ours | (e) Groundtruth
|  |  |  |
(f) Overview (Syn03) |  (g) Liu _et al_. [9] |  (h) Pei _et al_. [11] | (i) Ours | (j) Groundtruth
|  |  |  |
(k) Overview (Syn04) |  (l) Liu _et al_. [9] |  (m) Pei _et al_. [11] | (n) Ours | (o) Groundtruth
Figure 8: Qualitative results achieved on our synthetic scenes. Sub-figures on
the leftmost column represent the configurations of different scenes, yellow
boxes with $5\times 5$ blocks represent camera arrays. Note that, the
occlusion rates of these scenes are $30.2\%$, $19.6\%$ and $14.8\%$,
respectively.

###  4.2 Comparison to the state-of-the-arts

We compared our DeOccNet to the state-of-the-art LF-DeOcc method [11]. We also
used traditional refocusing method [18] and its improved version [17] as
baselines. Moreover, to investigate the benefits of complementary information
introduced by additional perspectives, we compared our method to the state-of-
the-art image inpainting method [9]. Note that, the inpainting method [9]
cannot automatically recognize occlusions in an image. Therefore, we manually
labeled occlusions in each center-view image. Since the codes of [11, 18, 17]
are unavailable, we used our own implementations with their default parameter
settings. Following the state-of-the-art image inpainting methods [37, 9],
mean $l_{1}$ error, peak signal-to-noise ratio (PSNR) and structure similarity
(SSIM) are used as quantitative evaluation metrics in this paper. Readers are
referred to [37, 9] for more details about these metrics. Qualitative results
on real-world and synthetic datasets are shown in Figs. 6, 7, and 8, and
quantitative results are listed in Table 1.

It can be seen from Fig. 6(a) that occlusions in the CD scene occupy a large
portion. Consequently, even though $75$ views are provided, it is still highly
challenging to reconstruct occluded objects. Method [18] removes occlusions by
warping and averaging SAIs. As shown in Fig. 6(b), although occlusions are
extremely blurred at the focused depth, occluded object is also unclear and
its contrast is low. Besides, since results produced by [18] only focus on
limited depth ranges, objects in other depth ranges are highly blurred and
cannot be recognized. In contrast, Vaish _et al_. [17] use median cost to
achieve all-in-focus synthetic aperture imaging. However, the performance
achieved by [17] is very limited.

Pei _et al_. [11] achieve a better performance than method [17]. That is
because, only rays belonging to occluded objects are used in [11]. However,
due to various textures and shapes of occlusions and background, it is
difficult to exactly select rays only from background, and the resulting
images can be deteriorated by incorrect classification of occlusion and
background. More specifically, if pixels from occlusions are misused, the
resulting images will be blurred (_e.g_., Fig. 6(d)). In contrast, if
background are incorrectly considered as occlusions, the resulting images will
have blank areas (_e.g_., Figs. 7(d) and 7(i)) since all information in these
areas is removed.

The single image inpainting method [9] tries to hallucinate missing part and
render reasonable results using knowledge learned from a large number of daily
scenes (_e.g_., the ImageNet database [3]). It can generate promising results
on scenes with simple structure and small occlusions (_e.g_., Fig. 8(g)).
However, since no additional information is used in the inpainting process,
this method cannot handle scenes with rich textures/details or heavy
occlusions (_e.g_., Figs. 7(c)and 8(l)).

As compared to existing methods, our method achieves superior performance,
especially on scenes with heavy occlusions (_e.g_., scenes CD and Syn04). That
is, DeOccNet successfully learns to discriminate occlusions from backgrounds,
and incorporates information of occluded objects from different viewpoints. It
can be also seen from Table 1 that our method achieves the best quantitative
results on scenes CD, Syn01 and Syn02. Note that, our method is slightly
inferior to [9] on scene Syn03. That is because, method [9] uses manually
labeled groundtruth occlusions while our method do not rely on any groundtruth
information. Since scene Syn03 has relatively small occlusions and simple
textures, method [9] works well using learned spatial priors and labeled
groundtruth occlusions.

Table 1: Quantitative results achieved by different methods and different design choices of DeOccNet. Note that, for $l_{1}$-error, smaller scores indicate better performance, and for PSNR and SSIM, higher scores indicate better performance. |  CD |  Syn01 |  Syn02 |  Syn03 |  Syn04 |  Average
---|---|---|---|---|---|---
$l_{1}$-error (Pei _et al_. [11]) |  0.233 |  0.188 |  0.240 |  0.204 |  0.156 |  0.204
$l_{1}$-error (Liu _et al_. [9]) |  0.196 |  0.264 |  0.180 |  0.078 |  0.187 |  0.181
$l_{1}$-error (ours_noASPP) |  0.296 |  0.200 |  0.271 |  0.220 |  0.339 |  0.265
$l_{1}$-error (ours_3skips) |  0.222 |  0.202 |  0.346 |  0.319 |  0.207 |  0.259
$l_{1}$-error (ours) |  0.185 |  0.138 |  0.165 |  0.163 |  0.242 |  0.178
PSNR (Pei _et al_. [11]) |  16.75 |  19.78 |  18.10 |  19.60 |  21.57 |  19.16
PSNR(Liu _et al_. [9]) |  15.95 |  18.19 |  18.99 |  25.58 |  20.44 |  19.83
PSNR (ours_noASPP) |  18.80 |  20.37 |  17.69 |  20.78 |  17.79 |  19.09
PSNR (ours_3skips) |  19.95 |  20.41 |  16.25 |  17.74 |  21.20 |  19.11
PSNR (ours) |  21.27 |  24.68 |  21.74 |  23.98 |  20.56 |  22.45
SSIM (Pei _et al_. [11]) |  0.508 |  0.636 |  0.568 |  0.656 |  0.569 |  0.587
SSIM (Liu _et al_. [9]) |  0.647 |  0.595 |  0.682 |  0.848 |  0.485 |  0.651
SSIM (ours_noASPP) |  0.625 |  0.586 |  0.617 |  0.809 |  0.523 |  0.632
SSIM (ours_3skips) |  0.621 |  0.613 |  0.560 |  0.742 |  0.530 |  0.613
SSIM (ours) |  0.694 |  0.699 |  0.734 |  0.858 |  0.650 |  0.727

###  4.3 Ablation study

We conducted ablation study to investigate the improvement introduced by the
residual ASPP module and skip connections. Quantitative results are presented
in Table 1.

First, we removed the residual ASPP module and retrained DeOccNet from scratch
using the same training data. We can observe from Table 1 that the residual
ASPP module significantly contributes to the overall LF-DeOcc performance.
Specifically, it introduces more than $3$ dB improvement in average PSNR, and
nearly $0.1$ improvement in average SSIM. That is because, the residual ASPP
module can help the network to have a large receptive field to cover
foreground occlusions of various shapes and scales.

Then, we investigated the benefits introduced by skip connections. Note that,
during the training process, we discovered that the network cannot achieve a
reasonable convergence without any skip connection. Therefore, we only removed
the outermost skip connection in the ablation study, and infer the
contribution of skip layers from experimental results. As shown in Table 1,
the outermost skip layer introduces $3.34$ dB improvement in average PSNR and
$1.14$ improvement in average SSIM. That is because, the high-frequency
details are tend to be lost by strided convolutions, and it is difficult to
recover these details from low-resolution features. Therefore, skip
connections are necessary to provide a short path for low-level and high-
frequency information.

###  4.4 Further discussion

It is worth noting that LFs generated by our Mask Embedding approach were used
as the only training data of our DeOccNet, and there is no intersection
between the training and test scenes. In fact, the styles of occlusion and
background between the training and test scenes differ significantly. That is,
all occlusions in our training data are fence-like and front-parallel masks.
Although we ran the Mask Embedding approach three times to simulate occlusions
at multiple depths, we cannot generate occlusions within a continuous range of
depths as in the CD and Bike scenes. However, as shown in Figs. 6(g) and 7(e),
our network can generalize well to these cases to handle slant occlusions.
That is because, both training and test scenes share the same LF structures.
Specifically, occlusions have positive disparity values while backgrounds have
negative disparity values.

To investigate the intrinsic mechanism of our network in dealing with
foreground occlusions, we stacked $75$ identical center-view SAIs of the CD
scene, and fed them into our DeOccNet. It can be seen from Fig. 6(f) that our
DeOccNet network cannot work with replicated inputs with identical
information. That is, the mechanism of our DeOccNet is significantly different
from that of the single image inpainting network [9]. Specifically, rather
than using spatial information within one perspective in [9], our network uses
complementary information from different viewpoints. It introduces significant
performance improvements in scenes with heavy but pierced occlusions (_e.g_.,
basket in scene Bike01) because useful information can be introduced by
different perspectives. However, only focusing on complementary information
from different views makes our network perform unsatisfactorily on scenes with
solid block regions (_e.g_., the right bottom corner in scene Bike02) since
background is occluded in all perspectives and spatial information is need for
LF-DeOcc. In the future, we will incorporate perspective information with
neighborhood prior for LF-DeOcc, which is likely to introduce further
performance improvement.

##  5 Conclusion

In this paper, we propose DeOccNet, the first deep learning based method for
LF-DeOcc. We embed masks into existing LFs to generate a large training
dataset. Experiments on both synthetic and real-world scenes show that, our
DeOccNet can automatically remove foreground occlusions through disparity
discrepancy, and achieve superior performance as compared to existing methods.

##  6 Acknowlegement

This work was partially supported by the National Natural Science Foundation
of China (Nos. 61401474, 61602499, 61972435), and Fundamental Research Funds
for the Central Universities (No. 18lgzd06).

## References

  * [1] A. Alperovich, O. Johannsen, and B. Goldluecke.  Intrinsic light field decomposition and disparity estimation with deep encoder-decoder network.  In 2018 26th European Signal Processing Conference (EUSIPCO), pages 2165–2169. IEEE, 2018.
  * [2] A. Alperovich, O. Johannsen, M. Strecke, and B. Goldluecke.  Light field intrinsics with a deep encoder-decoder network.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 9145–9154, 2018.
  * [3] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 248–255. Ieee, 2009.
  * [4] K. Honauer, O. Johannsen, D. Kondermann, and B. Goldluecke.  A dataset and evaluation methodology for depth estimation on 4d light fields.  In Asian Conference on Computer Vision (ACCV), pages 19–34. Springer, 2016.
  * [5] N. Joshi, S. Avidan, W. Matusik, and D. J. Kriegman.  Synthetic aperture tracking: tracking through occlusions.  In International Conference on Computer Vision (ICCV), pages 1–8. IEEE, 2007.
  * [6] D. P. Kingma and J. Ba.  Adam: A method for stochastic optimization.  International Conference on Learning Representations (ICLR), 2014\.
  * [7] D. Lanman, G. Wetzstein, M. Hirsch, W. Heidrich, and R. Raskar.  Polarization fields: dynamic light field display using multi-layer lcds.  In ACM Transactions on Graphics, volume 30, page 186. ACM, 2011\.
  * [8] X. Lin, J. Wu, G. Zheng, and Q. Dai.  Camera array based light field microscopy.  Biomedical optics express, 6(9):3179–3189, 2015.
  * [9] G. Liu, F. A. Reda, K. J. Shih, T.-C. Wang, A. Tao, and B. Catanzaro.  Image inpainting for irregular holes using partial convolutions.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 85–100, 2018.
  * [10] D. Pathak, P. Krahenbuhl, J. Donahue, T. Darrell, and A. A. Efros.  Context encoders: Feature learning by inpainting.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2536–2544, 2016.
  * [11] Z. Pei, X. Chen, and Y.-H. Yang.  All-in-focus synthetic aperture imaging using image matting.  IEEE Transactions on Circuits and Systems for Video Technology, 28(2):288–301, 2018.
  * [12] Z. Pei, Y. Zhang, X. Chen, and Y.-H. Yang.  Synthetic aperture imaging using pixel labeling via energy minimization.  Pattern Recognition, 46(1):174–187, 2013.
  * [13] Z. Pei, Y. Zhang, T. Yang, X. Zhang, and Y.-H. Yang.  A novel multi-object detection method in complex scene using synthetic aperture imaging.  Pattern Recognition, 45(4):1637–1658, 2012.
  * [14] J. Peng, Z. Xiong, D. Liu, and X. Chen.  Unsupervised depth estimation from light field using a convolutional neural network.  In 2018 International Conference on 3D Vision (3DV), pages 295–303. IEEE, 2018.
  * [15] C. Shin, H.-G. Jeon, Y. Yoon, I. So Kweon, and S. Joo Kim.  Epinet: A fully-convolutional neural network using epipolar geometry for depth from light field images.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4748–4757, 2018.
  * [16] V. Vaish and A. Adams.  The (new) stanford light field archive.  Computer Graphics Laboratory, Stanford University, 6(7), 2008.
  * [17] V. Vaish, M. Levoy, R. Szeliski, C. L. Zitnick, and S. B. Kang.  Reconstructing occluded surfaces using synthetic apertures: Stereo, focus and robust measures.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), volume 2, pages 2331–2338. IEEE, 2006.
  * [18] V. Vaish, B. Wilburn, N. Joshi, and M. Levoy.  Using plane+ parallax for calibrating dense camera arrays.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), volume 1, pages I–I. IEEE, 2004.
  * [19] K. Venkataraman, D. Lelescu, J. Duparré, A. McMahon, G. Molina, P. Chatterjee, R. Mullis, and S. Nayar.  Picam: An ultra-thin high performance monolithic camera array.  ACM Transactions on Graphics, 32(6):166, 2013.
  * [20] L. Wang, Y. Wang, Z. Liang, Z. Lin, J. Yang, W. An, and Y. Guo.  Learning parallax attention for stereo image super-resolution.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [21] Y. Wang, F. Liu, Z. Wang, G. Hou, Z. Sun, and T. Tan.  End-to-end view synthesis for light field imaging with pseudo 4dcnn.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 333–348, 2018.
  * [22] Y. Wang, F. Liu, K. Zhang, G. Hou, Z. Sun, and T. Tan.  Lfnet: A novel bidirectional recurrent convolutional neural network for light-field image super-resolution.  IEEE Transactions on Image Processing, 27(9):4274–4286, 2018.
  * [23] Y. Wang, J. Yang, Y. Guo, C. Xiao, and W. An.  Selective light field refocusing for camera arrays using bokeh rendering and superresolution.  IEEE Signal Processing Letters, 26(1):204–208, 2018.
  * [24] S. Wanner, S. Meister, and B. Goldluecke.  Datasets and benchmarks for densely sampled 4d light fields.  In Vision, Modelling and Visualization (VMV), volume 13, pages 225–226. Citeseer, 2013.
  * [25] B. Wilburn, N. Joshi, V. Vaish, M. Levoy, and M. Horowitz.  High-speed videography using a dense camera array.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), volume 2, pages II–II. IEEE, 2004.
  * [26] B. Wilburn, N. Joshi, V. Vaish, E.-V. Talvala, E. Antunez, A. Barth, A. Adams, M. Horowitz, and M. Levoy.  High performance imaging using large camera arrays.  In ACM Transactions on Graphics, volume 24, pages 765–776. ACM, 2005.
  * [27] G. Wu, Y. Liu, Q. Dai, and T. Chai.  Learning sheared epi structure for light field reconstruction.  IEEE Transactions on Image Processing, 2019.
  * [28] G. Wu, Y. Liu, L. Fang, Q. Dai, and T. Chai.  Light field reconstruction using convolutional network on epi and extended applications.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018\.
  * [29] G. Wu, B. Masia, A. Jarabo, Y. Zhang, L. Wang, Q. Dai, T. Chai, and Y. Liu.  Light field image processing: An overview.  IEEE Journal of Selected Topics in Signal Processing, 11(7):926–954, 2017.
  * [30] Z. Xiao, L. Si, and G. Zhou.  Seeing beyond foreground occlusion: a joint framework for sap-based scene depth and appearance reconstruction.  IEEE Journal of Selected Topics in Signal Processing, 11(7):979–991, 2017.
  * [31] C. Yang, X. Lu, Z. Lin, E. Shechtman, O. Wang, and H. Li.  High-resolution image inpainting using multi-scale neural patch synthesis.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6721–6729, 2017.
  * [32] T. Yang, Y. Zhang, X. Tong, X. Zhang, and R. Yu.  Continuously tracking and see-through occlusion based on a new hybrid synthetic aperture imaging model.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3409–3416. IEEE, 2011.
  * [33] T. Yang, Y. Zhang, X. Tong, X. Zhang, and R. Yu.  A new hybrid synthetic aperture imaging model for tracking and seeing people through occlusion.  IEEE Transactions on Circuits and Systems for Video Technology, 23(9):1461–1475, 2013.
  * [34] T. Yang, Y. Zhang, J. Yu, J. Li, W. Ma, X. Tong, R. Yu, and L. Ran.  All-in-focus synthetic aperture imaging.  In European Conference on Computer Vision (ECCV), pages 1–15. Springer, 2014.
  * [35] H. W. F. Yeung, J. Hou, X. Chen, J. Chen, Z. Chen, and Y. Y. Chung.  Light field spatial super-resolution using deep efficient spatial-angular separable convolution.  IEEE Transactions on Image Processing, 28(5):2319–2330, 2019.
  * [36] Y. Yoon, H.-G. Jeon, D. Yoo, J.-Y. Lee, and I. So Kweon.  Learning a deep convolutional network for light-field image super-resolution.  In Proceedings of the IEEE International Conference on Computer Vision Workshops (ICCVW), pages 24–32, 2015.
  * [37] J. Yu, Z. Lin, J. Yang, X. Shen, X. Lu, and T. S. Huang.  Generative image inpainting with contextual attention.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5505–5514, 2018.
  * [38] S. Zhang, Y. Lin, and H. Sheng.  Residual networks for light field image super-resolution.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 11046–11055, 2019.
  * [39] Z. Zhang.  A flexible new technique for camera calibration.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 22, 2000.