# Joint 3D Shape and Motion Estimation from Rolling Shutter Light-Field Images

Hermes McGriff${}^{~{}1,3}$ Renato Martins1,2 Nicolas Andreff${}^{~{}3}$
Cédric Demonceaux1,2
1Université de Bourgogne, CNRS UMR 6303 ICB 2Université de Lorraine, CNRS,
Inria, LORIA
3Université de Franche-Comté, CNRS UMR 6174 FEMTO-ST
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

In this paper, we propose an approach to address the problem of 3D
reconstruction of scenes from a single image captured by a light-field camera
equipped with a rolling shutter sensor. Our method leverages the 3D
information cues present in the light-field and the motion information
provided by the rolling shutter effect. We present a generic model for the
imaging process of this sensor and a two-stage algorithm that minimizes the
re-projection error while considering the position and motion of the camera in
a motion-shape bundle adjustment estimation strategy. Thereby, we provide an
instantaneous 3D shape-and-pose-and-velocity sensing paradigm. To the best of
our knowledge, this is the first study to leverage this type of sensor for
this purpose. We also present a new benchmark dataset composed of different
light-fields showing rolling shutter effects, which can be used as a common
base to improve the evaluation and tracking the progress in the field. We
demonstrate the effectiveness and advantages of our approach through several
experiments conducted for different scenes and types of motions. The source
code and dataset are publicly available at: https://github.com/ICB-Vision-
AI/RSLF.

##  1 Introduction

Light-field (LF) cameras (also known as plenoptic), introduced by Adelson and
Wang [1] and prototyped by Ng [25], consist of a conventional camera with a
micro-lens array in front of the photosensitive sensor. This type of imaging
sensor has the particularity of being able to capture a light field of a scene
in a single capture. LF cameras are now an established solution used in
computer vision, photogrammetry and robotics [7, 15, 37].

Figure 1: Some central views of the proposed RSLF dataset. From left to right:
A global shutter view, the ground truth depth, and four different rolling
shutter views with increasing camera motions.

The miniaturization of these cameras, _e.g_. in the context of applications
such as intra-corporeal micro-robotics, requires the choice of a rolling
shutter (RS) photosensitive sensor. Conversely to global shutter (GS) cameras,
where all pixels of the image are acquired at the same time, the image
acquisition by rolling shutter is sequential [24]. Notably, a RS sensor
creates image deformations in the case of dynamic scenes or when the camera is
moving, as depicted in Fig. 1. The rolling shutter then often degrades the
performance and challenge existing reconstruction and pose estimation
approaches. Ait-Aider _et al_. [3] have shown that, in the case of a
conventional perspective monocular camera, these deformations can be leveraged
in order to compute the motion of the scene with respect to the camera.
However, their proposed model and subsequent improved strategies [4, 18] have
the strong limitation of requiring the shape of the object/scenes to be known.
Conversely, this paper proposes to jointly estimate the motion and the
structure of a scene from a single view shot and in less constrained
conditions. Although the possibilities given by RS, when properly modelled,
has been shown for several computer vision and graphics problems, the
combination with LF has not been yet exploited in a unified approach. One
important motivation of this paper is to show the possibilities that this
sensor modality presents, such as of being able to allow the estimation of the
camera motion (or from the scene/object) from a single view without priors on
the scene shape. We are particularly motivated by showing the interest of a
unified approach (and its properties) that is capable of leveraging RS with
existing light-field consumer devices. Indeed, such sensors are available,
like the entry level cameras of Raytrix (R8, R42, R10$\mu$, R20) or any camera
array with RS sensors (like Pelican Imaging), but unfortunately no public
dataset is available to be the best of the authors’ knowledge. In this
context, another core motivation of this paper is to present a suitable and
challenging LF dataset collected with a RS camera with different motion levels
and scene geometries. For that, we have generated new models and leveraged
existing scene models (from Matterport3D) into an adapted rendering engine
(based on Blender) to create LF data affected by RS distortions in different
conditions (e.g., from mild to strong motions). The main contributions of this
paper are as follows:

  * •

We propose a generic projection model of a rolling shutter light-field (RSLF)
camera. This model is capable to represent a light-field with both global
shutter and rolling shutter settings.

  * •

A non-linear bundle adjustment strategy is designed to estimate jointly the 3D
shape and motion for this sensor modality. We also design a linear
initialization strategy in order to recover a first coarse estimate of the 3D
shape. This initialization is essential for the convergence of our approach as
shown in the ablation studies.

  * •

We also present a new dataset composed of Rolling Shutter Light Fields (RSLF)
paired with ground truth depth maps, on several synthetic scenes and with
different types and levels of motion. We aim this dataset to be used as a
common base to improve the evaluation and help tracking the progress in the
field.

##  2 Related Work

#### Depth estimation from light-fields.

Light-field contains rich information cues about the geometry of the scene.
The seminal work of Adelson and Wang [1] for the plenoptic camera exploit this
ability for “single lens stereo”. They used sub-aperture images (SAI) to
perform a standard two frame displacement analysis with multiple pairs
horizontally and vertically. In the same direction, multi-view stereo
matching-based methods try to reproduce the results of classical stereo with
plenoptic images [12, 27, 16, 38]. In this context, Georgiev and Lumsdaine
[12] introduced the focused plenoptic camera and proposed a complete setup in
order to recover depth with this new design. The method simultaneously render
the image and estimate a per micro-lens depth map by computing the cross
correlation between patches in micro images. Similarly, Perwass and Wietzke
[27] introduced a multi-focused plenoptic camera model alongside a depth
estimation algorithm based on point correspondences between micro images. Jeon
and Park [16] explored the phase-shift theorem of the Fourier transform to
estimate an accurate sub-pixel disparity map by computing a matching cost
volume between SAI. Zeller _et al_. [38] proposed a filtering method for the
estimation of semi-dense probabilistic depth maps for focused plenoptic
cameras, with a Kalman filter like approach preserving discontinuities in the
depth map. Ferreira and Goncalves [11] proposed a similar but faster depth map
estimation method, with SIFT correspondences and through epipolar lines on the
micro images. Bok _et al_. [5] proposed a calibration of the light-field
camera based on a bundle adjustment method and Zhang _et al_. [39] proposed a
generic multi-projection model (along with its calibration algorithm) for LF
cameras. Most of these techniques rely on generating SAI and then applying
classic stereo matching algorithms to estimate the depth of the scene.
However, they assume GS cameras (or with slow moving objects and camera
motions). Our approach, on the other hand, can handle scenes with a camera in
movement and is far less affected by RS distortions due to camera motions.

#### Epipolar plane images and learning-based LF analysis.

The scene structure can also be extracted from Epipolar Plane Images (EPI) [6,
8, 36, 32]. These approaches estimate depth information from the slopes of the
lines observed in the Epipolar planes. Tao et al. [34] improved the accuracy
of the depth estimation with a weighted sum between the defocus and
correspondence cues present in EPIs. Zhang _et al_. [40] proposed a spinning
parallelogram operator to determine the line slopes. Lin _et al_. [21]
leveraged the refocus capability of light-fields and the possibility to use
Shape-From-Focus. Closely related to our work, Srinivasan _et al_. [33]
proposed a motion estimation from a single view with a light-field camera
based on motion blur. Heber and Pock [13] first used a Convolutional Neural
Network to compute depth from light-field images. Shin _et al_. [31] proposed
a fast and accurate light field depth estimation method based on a fully-
convolutional neural network and a light-field image-specific data
augmentation. These techniques suffer by the lack of generalization to
new/unseen scenes and often dependence on significant amount of data.

#### Rolling shutter structure-from-motion estimation.

The potential of RS images received increased attention for scene analysis.
Meingast _et al_. [24] developed a general projection equation for a rolling
shutter camera and also proposed a calibration to estimate the rate of the
rolling shutter. Ait-Aider _et al_. [3] first showed that the rolling shutter
effect could be leveraged to estimate the motion of an object, but of known
shape, when the majority of previous studies on the rolling shutter were about
compensating it [20, 17]. This is notably done for blur compensation with both
model and learning-based approaches [23, 10]. Saurer _et al_. [30] and Ait-
Aider _et al_. [4] investigated RS effects for stereo vision. Recently Lao _et
al_. [19] proposed a analogy with non-rigidity to solve shape estimation with
a monocular rolling shutter image. Different than these previous methods, we
address the ambiguity between shape and motion inherent to RS images
exploiting the properties of the LF. We show that a micro-lens array in front
of the RS sensor allows to model the RS effect in the case of 3D scenes and to
estimate the movement of the scene with respect to the camera without prior
knowledge of the scene geometry.

##  3 Method

Our joint 3D scene reconstruction and camera motion estimation approach has
two main stages. Firstly, a coarse linear solution is computed to provide an
initialization for a non-linear bundle adjustment method. This method is
designed to handle the geometric and temporal constraints that are present in
the Rolling Shutter Light-Field setting.

#### Light-field modeling and RS projection.

Figure 2: left \- A raw plenoptic image from a near viewpoint in the scene
shown in Fig. 1 and a detail of the micro-images. right \- The adopted LF
coordinate frames: The 3D point is projected in a 3D virtual scene by thin
lens projection, then on the 2D image plane by pinhole projection which
coordinate frame depends on the considered viewpoint.

To provide a comprehensive theoretical framework for our proposed approach, we
begin by presenting an overview of the light field projection modeling.
Subsequently, we use this framework to introduce a projection model
formulation that is specifically designed for the RSLF setting. A more
detailed description of the projection model formulation and theoretical
analysis are given in the Supplementary material. An overview of the adopted
light-field modeling and geometry is shown in Fig. 6. The pose of the camera
with respect to the scene expressed in the world coordinates frame
$(\mathbf{O}_{w},X_{w},Y_{w},Z_{w})$ is
$[\mathbf{R}\mid\mathbf{T}]\in\mathbb{SE}(3)$. The camera position defines a
new coordinate frame $(\mathbf{O}_{c},X_{c},Y_{c},Z_{c})$ with origin placed
in the optical center of the main lens. The view plane (MLA plane) has
coordinate frame $(\mathbf{O},X,Y,Z)$ expressed in relation to the camera
frame by a pure translation $(O_{x},O_{y},d)$ expressed by the transformation
matrix $\mathbf{D}\in SE(3)$, with $\mathbf{O}=(O_{x},O_{y},0)^{\top}$ the
intersection of the optical axis and the view plane, and $d$ the distance
between the optical center of the main lens and the view plane. The micro-
image local frames $(x,y)$ are attached to the image plane and are dependent
of the considered viewpoint, as shown in Fig. 6. Given a point in the world
homogeneous coordinates frame
${}^{w}\widetilde{\mathbf{p}}=(x_{w},y_{w},z_{w},1)^{\top}$ and the matrices
${}^{c}\mathbf{M}_{w}$ (the transformation between the camera to world
coordinates) and $\mathbf{K}_{c}$ (thin lens projection matrix), we can obtain
the virtual projection of the 3D point inside the camera as:

$\lambda_{c}\widetilde{\mathbf{p}}=\mathbf{D}\mathbf{K}_{c}{}^{c}\mathbf{M}_{w}{}^{w}\widetilde{\mathbf{p}},$
(1)

with $\lambda_{c}$ a scaling factor. For a given viewpoint
$\mathbf{c}=(s,t,0)^{\top}$, _i.e_. a projection center, the projection of the
point $\widetilde{\mathbf{p}}$ onto the image plane is given by:

$\lambda_{s}^{s,t}\widetilde{\mathbf{m}}^{s,t}=\mathbf{K}_{s}^{s,t}\widetilde{\mathbf{p}}=\begin{bmatrix}f&0&0&-fs\\\
0&f&0&-ft\\\ 0&0&1&0\\\ \end{bmatrix}\widetilde{\mathbf{p}},$ (2)

with $\widetilde{\mathbf{m}}^{s,t}=(x^{s,t},y^{s,t},1)^{\top}$ the final LF
image points, $f$ the focal length of the micro-lenses and $\lambda_{s}$ a
scaling factor.

#### Rolling shutter modeling.

We follow a similar formalism to Ait-Aider _et al_. [3] to represent an RS
imaging process. The main insight is to define a projection model dependent of
the camera pose and as a function of the micro-image line $t$ being observed.
We adopt the hypothesis that the speeds $(\mathbf{v},\Omega)$ are constant
during the LF acquisition. Adapting the initial projection defined in Eq. 1
for the RS we have:

$\lambda_{c}\widetilde{\mathbf{p}}=\mathbf{D}\mathbf{K}_{c}\left[\begin{array}[]{c
c}\delta\mathbf{R}^{t}{}^{c}\mathbf{R}_{w}&{}^{c}\mathbf{T}_{w}+\delta\mathbf{T}^{t}\\\
\boldsymbol{0}^{\top}&1\end{array}\right]{}^{w}\widetilde{\mathbf{p}},$ (3)

with $\delta\mathbf{R}^{t}=\mathbf{a}\mathbf{a}^{\top}(1-\cos(\Omega\tau
t))+\mathbf{I}\cos(\Omega\tau t)+[\mathbf{a}]_{\wedge}\sin(\Omega\tau
t),\mbox{ and }\delta\mathbf{T}^{t}=\mathbf{v}\tau t,$ where $\mathbf{a}$
(axis of rotation), $\Omega$ (angular velocity) and $\mathbf{v}$ (linear
velocity) describe the uniform movement of the camera coordinate frame with
respect to the world coordinates frame and $\tau$ the time between the
acquisition of two lines of the micro-images. The full Rolling Shutter LF
projection from Eq. 15 that projects the 3D point
${}^{w}\widetilde{\mathbf{p}}_{i}$ to an image point
$\mathbf{m}_{i}^{s,t}\in\mathbb{P}^{2}$, given a center of projection
$\mathbf{c}=(s,t,0)^{\top}$ is then

$\lambda\mathbf{m}_{i}^{s,t}=\mathbf{K}_{s}^{s,t}\mathbf{D}\mathbf{K}_{c}[\delta\mathbf{R}^{t}{}^{c}\mathbf{R}_{w}\mid{}^{c}\mathbf{T}_{w}+\delta\mathbf{T}^{t}]{}^{w}\widetilde{\mathbf{p}}_{i},$
(4)

where $\mathbf{K}_{s}^{s,t}\mathbf{D}\mathbf{K}_{c}$ can be represented as a
single compact intrinsic Rolling Shutter LF tensor:

$\mathbf{K}_{s}^{s,t}\mathbf{D}\mathbf{K}_{c}=\begin{bmatrix}f&0&-\frac{f}{F}(O_{x}-s)&f(O_{x}-s)\\\
0&f&-\frac{f}{F}(O_{y}-t)&f(O_{y}-t)\\\ 0&0&1-\frac{d}{F}&d\end{bmatrix},$ (5)

with $F$ the focal length of the main lens. This formulation has the strong
advantage of being generic and represent both GS and RS configurations.
Another advantage is that all parameters of this unified model can be
calibrated with existing techniques such as Bok _et al_. [5] for the intrinsic
parameters and Meingast _et al_. [24] for the rolling shutter rate.

#### Generalization and particular cases.

When $\tau=0$ (_i.e_., no temporal delay between two consecutive lines), this
model can be simplified to a GS light-field camera as the position of the
sensor with respect to the scene will be identical for any $t$. The situation
where the camera has no velocity with respect to the scene can also be seen as
GS for similar reasons. The proposed projection model in Eq. 17 generalizes to
a conventional pinhole camera projection in the case where the MLA is composed
of a unique lens. More details are given in the Supplementary material.

###  3.1 Scene Structure and Motion Estimation

For a given set of matching points inside a calibrated LF and assuming that
all points belongs to the same rigid scene in a uniform movement with respect
to the camera, we can recover the position of the points in the 3D world as
well as their motion at a given time. We will use a re-projection error
minimization in order to find jointly these 3D coordinates
${}^{w}\widetilde{\mathbf{p}}_{i}$ and the dynamic parameters describing the
movement of the camera.

#### Linear initialization.

A classical multi-view stereo strategy is applied to provide a first estimate
of the 3D points in the scene. In order to reduce the influence of the RS
effect, we apply the multi-view stereo only horizontally, thereby ensuring
that each measured point ${}^{w}\widetilde{\mathbf{p}}_{i}\in\mathbb{R}^{3}$
is captured at the same instant. From the experiments, this first estimate is
essential to allow convergence of the following non-linear optimization.

#### Non-linear bundle adjustment.

Using this 3D initialization of the observed points in the light field and our
projection model, we design a re-projection error in order to recover
simultaneously a refined structure of the scene and the camera motion. From
our projection in Eq. 17 we compute the point:

$\left(u_{i}^{s,t},v_{i}^{s,t},w_{i}^{s,t}\right)^{T}=\mathbf{K}_{s}^{s,t}\mathbf{D}\mathbf{K}_{c}[\delta\mathbf{R}^{t}\mid\delta\mathbf{T}^{t}]{}^{w}\widetilde{\mathbf{p}}_{i}$
(6)

and deduce the Euclidean pixel coordinates, the scalars $x_{i}^{s,t}$ and
$y_{i}^{s,t}$, computed as:

$\displaystyle x_{i}^{s,t}$
$\displaystyle=\frac{u_{i}^{s,t}}{w_{i}^{s,t}}:=\xi_{(x)}^{s,t}({}^{w}\widetilde{\mathbf{p}}_{i},\Omega,\mathbf{a},\mathbf{v}),\text{\hskip
14.22636pt and}$ (7) $\displaystyle y_{i}^{s,t}$
$\displaystyle=\frac{v_{i}^{s,t}}{w_{i}^{s,t}}:=\xi_{(y)}^{s,t}({}^{w}\widetilde{\mathbf{p}}_{i},\Omega,\mathbf{a},\mathbf{v}),$

with $\xi^{s,t}$ the projection function that, given a center of projection
$\mathbf{c}=(s,t,0)^{\top}$, return the coordinates of the image point with
respect to its static position and its movement. The re-projection error
function is obtained by computing the distance between the measured points
$\mathbf{\tilde{m}}_{i}^{s,t_{0}}(\tilde{x_{i}}^{s,t},\tilde{y_{i}}^{s,t})$
and the coordinates estimated with $\xi_{(x)}^{s,t}$ and $\xi_{(y)}^{s,t}$
from Eq. 7 as follows:

$\displaystyle\epsilon=$
$\displaystyle\sum_{s}\sum_{t}\sum_{i}\left(\tilde{x_{i}}^{s,t}-\xi_{(x)}^{s,t}({}^{w}\widetilde{\mathbf{p}}_{i},\Omega,\mathbf{a},\mathbf{v})\right)^{2}$
(8)
$\displaystyle+\left(\tilde{y_{i}}^{s,t}-\xi_{(y)}^{s,t}({}^{w}\widetilde{\mathbf{p}}_{i},\Omega,\mathbf{a},\mathbf{v})\right)^{2}.$

This problem has three unknowns for $\Omega\mathbf{a}$, three unknowns for
$\mathbf{v}$, and three unknowns for every ${}^{w}\widetilde{\mathbf{p}}_{i}$.
It can be solved if at least four non-coplanar 3D points can be observed,
meaning that they need to be located at least an LF image in two different
lines and at two different columns of micro-images.

#### Regularization.

For the moment, the rotation axis $\mathbf{a}$ in Eq. 8 is defined to pass
through the world origin, which corresponds to the optical center of the main
lens. However, this is generally not the instantaneous center of rotation of
the movement between the camera and the scene. To ease the description of the
movement, we regularize the optimization by providing a “center of rotation”
$\mathbf{g}$ to the point cloud. This allows to express all points
${}^{w}\mathbf{p}_{i}$ in a new coordinate frame centered on this center of
rotation. It also allows to compute normalized points ${}^{n}\mathbf{p}_{i}$
from which the coordinates are lying in the range $[-1,1]$. The final non-
linear adapted re-projection error from Eq. 8 using the normalized points and
the center of rotation regularization is then:

$\displaystyle\epsilon=$
$\displaystyle\sum_{s}\sum_{t}\sum_{i}\left(\tilde{x_{i}}^{s,t}-{}^{n}\xi_{(x)}^{s,t}({}^{n}\mathbf{p}_{i},\mathbf{g},\Omega,\mathbf{a},\mathbf{v})\right)^{2}$
(9)
$\displaystyle+\left(\tilde{y_{i}}^{s,t}-{}^{n}\xi_{(y)}^{s,t}({}^{n}\mathbf{p}_{i},\mathbf{g},\Omega,\mathbf{a},\mathbf{v})\right)^{2},$

where ${}^{n}\xi_{(x)}^{s,t}$ and ${}^{n}\xi_{(y)}^{s,t}$ are designed to
handle the normalization, and $\mathbf{g}$ is also optimized in the loop so
that the model is able to find the optimal center of rotation on-the-fly.
Further details on the optimization are provided in the supplementary
materials.

##  4 Rolling Shutter Light-Field Dataset

| abs rel $\downarrow$ | abs diff $\downarrow$ | RMS $\downarrow$
---|---|---|---
Method | GS | slow | fast | GS | slow | fast | GS | slow | fast
Jeon-CVPR [16] | 0.040 | 0.053 | 0.110 | 0.027 | 0.036 | 0.071 | 0.035 | 0.048 | 0.092
OACC-Net [35] | 0.143 | 0.171 | 0.196 | 0.091 | 0.109 | 0.125 | 0.109 | 0.128 | 0.144
Ours | 0.040 | 0.041 | 0.059 | 0.031 | 0.032 | 0.044 | 0.046 | 0.051 | 0.064
| $\delta<1.25\uparrow$ | $\delta<1.25^{2}\uparrow$ | $\delta<1.25^{3}\uparrow$
Method | GS | slow | fast | GS | slow | fast | GS | slow | fast
Jeon-CVPR [16] | 0.993 | 0.976 | 0.894 | 1.000 | 0.999 | 0.973 | 1.000 | 1.000 | 0.998
OACC-Net [35] | 0.767 | 0.720 | 0.676 | 0.959 | 0.945 | 0.933 | 1.000 | 0.997 | 0.997
Ours | 0.958 | 0.961 | 0.949 | 0.989 | 0.988 | 0.982 | 0.999 | 0.999 | 0.999

Table 1: Average reconstruction error metrics in different scenarios for all
dataset sequences: GS (global shutter, equivalent to a static camera
scenario), slow (RS with small camera linear and angular velocities), and fast
(RS with camera motion three times higher velocities than in the _slow_ case).
The upward arrow means that a higher score is better. Our approach is
significantly better than the considered methods, and with competitive results
even for the GS case. Please see the text for details.

| abs rel $\downarrow$ | $\delta<1.25\uparrow$
---|---|---
rabbit | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10
Jeon-CVPR [16] | 0.06 | 0.08 | 0.07 | 0.07 | 0.07 | 0.1 | 0.19 | 0.12 | 0.13 | 0.34 | 0.39 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 0.82 | 1.0 | 0.91 | 0.59 | 0.35
OACC-Net [35] | 0.4 | 0.48 | 0.5 | 0.44 | 0.38 | 0.49 | 0.47 | 0.48 | 0.44 | 0.5 | 0.5 | 0.26 | 0.08 | 0.06 | 0.14 | 0.29 | 0.09 | 0.1 | 0.1 | 0.1 | 0.13 | 0.1
Ours | 0.03 | 0.03 | 0.03 | 0.02 | 0.03 | 0.03 | 0.03 | 0.03 | 0.02 | 0.03 | 0.03 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0
table | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10
Jeon-CVPR [16] | 0.03 | 0.03 | 0.04 | 0.03 | 0.05 | 0.03 | 0.05 | 0.09 | 0.05 | 0.17 | 0.07 | 1.0 | 1.0 | 0.99 | 1.0 | 1.0 | 1.0 | 0.97 | 0.96 | 0.99 | 0.76 | 0.94
OACC-Net [35] | 0.17 | 0.21 | 0.2 | 0.19 | 0.19 | 0.2 | 0.19 | 0.24 | 0.15 | 0.25 | 0.2 | 0.69 | 0.6 | 0.64 | 0.64 | 0.63 | 0.59 | 0.67 | 0.55 | 0.79 | 0.5 | 0.65
Ours | 0.02 | 0.02 | 0.02 | 0.03 | 0.02 | 0.03 | 0.04 | 0.02 | 0.04 | 0.03 | 0.04 | 0.995 | 0.995 | 0.99 | 1.0 | 0.99 | 1.0 | 1.0 | 0.99 | 0.99 | 0.98 | 1.0
bedroom | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10
Jeon-CVPR [16] | 0.02 | 0.03 | 0.02 | 0.04 | 0.06 | 0.03 | 0.07 | 0.02 | 0.11 | 0.03 | 0.07 | 1.0 | 1.0 | 1.0 | 1.0 | 0.97 | 1.0 | 0.99 | 1.0 | 0.89 | 1.0 | 0.94
OACC-Net [35] | 0.03 | 0.05 | 0.03 | 0.06 | 0.1 | 0.05 | 0.13 | 0.03 | 0.13 | 0.05 | 0.13 | 1.0 | 0.98 | 1.0 | 0.99 | 0.93 | 0.98 | 0.8 | 1.0 | 0.77 | 1.0 | 0.79
Ours | 0.03 | 0.03 | 0.03 | 0.03 | 0.03 | 0.03 | 0.04 | 0.03 | 0.04 | 0.04 | 0.05 | 1.0 | 1.0 | 1.0 | 0.999 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 0.99

Table 2: Detailed reconstruction error metrics for three representative scenes
“rabbit”, “table” and “bedroom” considering the eleven different motion
scenarios (from 0 to 10) of the dataset. The upward arrow means that a higher
score is better.

Despite the potential of rolling shutter plenoptic cameras, to the best of the
authors’ knowledge, all existing LF datasets are done assuming a global
shutter hypothesis [9, 26, 29, 2]. Unfortunately, there is no public data
available showing the rolling shutter effect on light-field images. Therefore,
we have carefully designed and collected a new dataset with seven different
synthetic scenes build on Blender, containing notably pseudo-real scenes
created from Habitat-Matterport benchmark [28]. This new dataset (inspired by
the HCI 4D LF benchmark [14]) is composed of four photo-realistic scenes from
Matterport and three synthetic ones (as the examples depicted in Fig. 1 and
Fig. 4). We provide, per scene, the following data:

  1. (i)

Config files with camera settings and disparity ranges.

  2. (ii)

Different motion scenarios:

     * •

GS: This is the static configuration. It allows to have a good measure of the
performance difference with or without RS distortion by having the same scene
in both scenarios. It is equivalent to a GS light field.

     * •

slow: The motions affect the image enough to affect largely the perception of
the scene geometry.

     * •

fast: The linear and angular camera velocities are about three times more
important than for the slow motions.

We collect 11 light field sequences per scene (1 GS, 5 slow, 5 fast). Please
see the table in the supplementary with the velocity intervals for each motion
scenario.

  3. (iii)

Each light field is of dimension $9\times 9\times 512\times 512\times 3$,
which is equivalent to a light field captured from a plenoptic camera with a
$512\times 512$ micro-lense array and $9\times 9$ micro-images.

  4. (iv)

A depth map corresponding to the geometry of the scene at middle time of
exposition (_i.e_., the pose of the camera during the acquisition of the
center line).

We believe this dataset has the potential to help the evaluation and to
promote futher investigation of RS applications for scene analysis with light
fields. Visualizations and additional details of the dataset are given in the
Supplementary material.

##  5 Experiments

#### Metrics and competitors.

We have selected two representative algorithms for comparison: the model-based
approach of Jeon _et al_. [16], and a recent learning-based 3D estimation from
LF of Wang _et al_. [35]. The comparison is done in both GS and RS scenarios
for all methods with the aim of fair conditions for the competitors. Six
commonly used metrics are selected for the evaluation abs rel, abs diff, RMS,
$\delta<1.25$, $\delta<1.25^{2}$ and $\delta<1.25^{3}$. abs rel is the
absolute difference between the estimation and the ground truth (gt),
normalized by the gt. abs diff is the absolute difference between the
estimation and the gt. RMS is the Root Mean Square Error between the
estimation and the gt. $\delta<1.25$, $\delta<1.25^{2}$ and $\delta<1.25^{3}$
are respectively the proportion of the points in a range of $1.25$ times the
gt, $1.25^{2}$ times the gt and $1.25^{3}$ times the gt.

###  5.1 Results

The evaluation and averaged metrics for all scenes (and different motion
conditions) are shown in Tab. 1. We can observe the proposed method achieves
the best scores overall in several of the considered metrics (e.g., “abs rel”
and “abs diff”), and notably for all metrics of the “fast” sequences’ split.
We can also notice that it has even a competitive performance to the recent
competitors in the GS scenario. This aspect will be further investigated in
the ablation and sensitivity analysis. As we can observe, the two competitors
perform far worse when motion is present. The detailed metrics for three
representative scenes considering the eleven light fields sequences per scene
(1 GS, 5 slow, 5 fast) are shown in Tab. 7, where we can see that our method
performs better in most cases. Please check some qualitative examples of the
obtained shape reconstructions for these three scenes shown in Fig. 4. We
alternate, for these three scenes, the GS case and a RS case with high
velocity (motion scenario number 9). We can clearly see the capacity of our
algorithm to model the RS deformations. In the scene “bedroom”, motion
scenario 9, (the last line of Fig. 4), one can clearly notice from visual
inspection the compensation done on the painting (the rectangle is less
stretched). Unfortunately, this qualitative observation is not highlighted in
the detailed quantitative metrics Tab. 7. Indeed, the painting is stretched in
the estimation given by the competitors, but is still close to the wall plane,
resulting in similar scores. However, the proposed formulation is at least
twice as accurate than the competitors for the other two scenes detailed in
Tab. 7 for all motion profiles, accordingly to the average scores for all
sequences shown in Tab. 1. The detailed results for all sequences and scenes
are included for completeness in the Supplementary materials due to page space
limitations.

Finally, we analyse the performance of the approaches in the easy to
understand ‘‘chart” scene as shown in the quantitative results from Tab. 3 and
visualizations in Fig. 3. Similarly to all other scenes, it is composed of
eleven light fields (1 GS, 5 slow, 5 fast), where a double checkerboard
pattern is joint in a $90^{\circ}$ angle configuration. Our method achieves
the best scores for every metric in both the slow and fast scenarios. However,
we can also obtain competitive results to both strong competitors in the case
of GS. We can also notice that sometimes our obtained estimation is more
accurate when the camera is moving slowly than when the camera is static. This
will be discussed in the ablation study Sec. 5.2. Tab. 3 also indicates that
our method slightly degrades with the augmentation of the camera speed, but it
still considerably outperforms all the competitors in the fast scenarios for
the four first metrics. Fig. 3 shows some qualitative examples of the three
methods in the different scenarios and the associated point clouds. We can
observe how our method is still capable of fitting the object shape even with
the presence of RS and fast camera motions. Looking at the object 3D
reconstruction results obtained by the other methods, we can clearly observe
deformation effects caused by the misinterpretation of the RS checkerboard
images. These degradation of performance can be explained if we observe that
the computed disparity maps of the competitors map the distortions of the
scene due to RS from the center view. They also interpret the movement of the
camera between vertically distant views only as spatial disparity. Thus, if a
point moves vertically downwards during acquisition, it will have a bigger
disparity than it should (between two viewpoints, where the point is moving
because of changes in point of view but also because of its own movement).
Inversely, if a point moves vertically upwards during acquisition, it will
have a smaller disparity than it should. These two effects contribute to
degrade the performance of GS-designed algorithms in the estimation of the 3D
geometry of the scene.

| abs rel $\downarrow$ | abs diff $\downarrow$ | RMS $\downarrow$
---|---|---|---
Method | GS | slow | fast | GS | slow | fast | GS | slow | fast
Jeon-CVPR [16] | 0.003 | 0.013 | 0.049 | 8.464 | 30.293 | 76.824 | 17.489 | 47.647 | 129.720
OACC-Net [35] | 0.003 | 0.013 | 0.051 | 12.214 | 30.882 | 79.938 | 26.197 | 54.799 | 140.215
Ours | 0.004 | 0.003 | 0.003 | 13.692 | 15.395 | 23.754 | 22.146 | 25.327 | 44.791
| $\delta<1.25\uparrow$ | $\delta<1.25^{2}\uparrow$ | $\delta<1.25^{3}\uparrow$
Method | GS | slow | fast | GS | slow | fast | GS | slow | fast
Jeon-CVPR [16] | 1.000 | 0.923 | 0.745 | 1.000 | 0.992 | 0.898 | 1.000 | 0.995 | 0.991
OACC-Net [35] | 1.000 | 0.922 | 0.730 | 1.000 | 0.993 | 0.895 | 1.000 | 0.996 | 0.939
Ours | 0.988 | 0.982 | 0.973 | 0.996 | 0.999 | 0.995 | 1.000 | 1.000 | 0.998

Table 3: Detailed reconstruction error metrics in different scenarios for the
“chart” sequence: GS (global shutter, equivalent to a static camera scenario),
slow (RS with small camera linear and angular velocities), and fast (RS with
camera motion three times higher velocities than in the _slow_ case). The
upward arrow means that a higher score is better. Our approach performed
significantly better than the two recent considered methods. Figure 3:
Qualitative examples of reconstruction for different motion scenarios for the
“chart” sequence. The ”GS” scenario on the left. A ”slow” scenario in the
middle. A ”fast” scenario on the right. - first column: The central view of
the LF, the disparity map of Jeon-CVPR [16], the disparity map of OACC-Net
[35]. - Second column: The 3D point clouds (red dots) obtained for our method,
Jeon-CVPR [16], OACC-Net [35]. Despite the fact that the images look
different, due to the rolling shutter effect, the reconstruction is supposed
to give the same result (in green crosses in the point clouds). Figure 4: Some
central views and associated point cloud reconstructions for the scenes and
results shown in Tab. 7. From right to left, OACC-Net [35], Jeon-CVPR [16] and
Ours. Ground truth points in gray and estimated in green.

###  5.2 Ablation study

We performed different ablation studies in order to evaluate the relevance of
the different parts of the method. In the first ablation, we retained two
major components for evaluation, the contribution of i) linear initialization
strategy (No Init.), and ii) the regularization (No Reg.) as shown in Tab. 5.
For the ablation of the initialization, we initialized the optimization Eq. 9
with all the points clustered in a position near the center of mass of the
point cloud we should have found with the linear initialization. We show in
Tab. 5 that, even after convergence, the solution is still far from correct.
For the ablation of the regularization, we see that the method without the
regularization gives worst results in the RS scenarios. These evaluations
confirm the importance of these components in the designed method.

A second ablation study was designed to evaluate the performance of our method
without the RS modelling (Ours No RS) depicted in Tab. 5. By modeling the RS
effect we also have additional degrees of freedom that lead to a slight
degradation of the results when compared to a GS scheme for the GS scenes. We
performed an evaluation to verify the effect of constraining the dynamic
degrees of freedom ($\Omega=0$ and $\mathbf{v}=\mathbf{0}$) in case of GS
would result in the estimation. The results in Tab. 5 show an improvement on
all the metrics of up to about 6%. This concurs with the aforementioned
hypothesis. The obtained performance is on par with the competitors which are
specifically designed for GS settings.

###  5.3 Discussion

From the experiments, we can observe that our method is capable of handling
different camera motions and provides improved scene structure estimates. The
proposed model is designed to handle rigid scenes, yet it can estimate the
structure and motion parameters for 3D scene points independently if at least
four image points are available, _i.e_., to compute a “3D scene flow” from a
single LF image. We assumed rigidity in order to compute a common set of
dynamic parameters to each point (corresponding to a camera motion in a rigid
scene). We believe our strategy could be also extended to handle scenes with
dynamic objects independently (or non-rigid) with multiple camera motion
hypotheses. The adopted RS projection also assumes that both linear and
angular velocities to be uniform during the LF image acquisition (i.e., zero
acceleration). However, RS devices, while having a sequential acquisition,
usually have a small time of total exposure per frame (about 0.1 s for a 4K
image). Therefore the assumption of constant camera speeds during the frame
acquisition holds in typical motion-scene scale scenarios. Nevertheless, the
proposed approach could still be applied for accelerated motions with a piece-
wise decomposition of the plenoptic image in horizontal bands. Such a strategy
of piece-wise decomposition in horizontal bands for classic images has been
investigated in [22] for a classic monocular RS sensor. The motion and shape
estimation could then be done at different time instants and allow to recover
more complex scenes (_e.g_., non-rigid) and motion scenarios.

| RMS $\downarrow$
---|---
Abl. | GS | slow | fast
No Init. | 0.243 | 0.242 | 0.240
No Reg. | 0.045 | 0.060 | 0.086
Full | 0.046 | 0.051 | 0.064
| $\delta_{1}<1.25\uparrow$
Abl. | GS | slow | fast
No Init. | 0.650 | 0.646 | 0.630
No Reg. | 0.969 | 0.950 | 0.895
Full | 0.958 | 0.961 | 0.949

Table 4: Reconstruction errors for the ablation study of our method for the
initialization and regularization steps.

Abl. | abs rel $\downarrow$ | abs diff $\downarrow$ | RMS $\downarrow$ | $\delta<1.25\uparrow$
---|---|---|---|---
Jeon-CVPR [16] | 0.040 | 0.027 | 0.035 | 0.993
Ours Full | 0.040 | 0.031 | 0.046 | 0.958
Ours No RS | 0.040 | 0.029 | 0.041 | 0.976

Table 5: Ablation study of the dynamic motion parameters with a static GS
scene.

##  6 Conclusion

In this paper, we proposed a projection model for a light-field camera
equipped with a rolling shutter sensor. This model allows us to jointly
estimate the shape and motion on unknown scenes from a single light field
image. The approach has been evaluated on different motions and 3D scenes.
Furthermore, it does not suffer from shape/motion ambiguity thanks to the
relatively reasonable assumption of a row-wise GS. To fill the lack of
publicly available rolling-shutter LF data, we created a dataset that includes
simulated photo-realistic light fields with different motion scenarios, and we
will make it publicly available. We plan to build upon this model to generate
denser depth maps and extend the motion estimations to non-rigid scenes. Our
proposed model shows improved 3D scene geometry estimates, and we believe that
it will inspire further research in this area, notably for applications in the
context of robot vision, manipulation and micro-robotics.

#### Acknowledgements.

The authors would like to thank the funding from the French “Investissements
d’Avenir” program, project ISITE-BFC, contract ANR-15-IDEX-03, by the Conseil
Régional BFC from the project ANER-MOVIS and by ”Grand Prix Scientifique 2018,
Fond. Ch. Defforey-Institut de France”.

## References

  * [1] Edward H Adelson and John YA Wang.  Single lens stereo with a plenoptic camera.  IEEE Trans. Pattern Anal. Mach. Intell., 14(2):99–106, 1992.
  * [2] Vamsi Kiran Adhikarla, Marek Vinkler, Denis Sumin, Rafał Mantiuk, Karol Myszkowski, Hans-Peter Seidel, and Piotr Didyk.  Towards a quality metric for dense light fields.  In IEEE Conf. Comput. Vis. Pattern Recog., 2017.
  * [3] Omar Ait-Aider, Nicolas Andreff, Jean Marc Lavest, and Philippe Martinet.  Simultaneous object pose and velocity computation using a single view from a rolling shutter camera.  In Eur. Conf. Comput. Vis., 2006.
  * [4] Omar Ait-Aider and François Berry.  Structure and kinematics triangulation with a rolling shutter stereo rig.  In Int. Conf. Comput. Vis., 2009.
  * [5] Yunsu Bok, Hae-Gon Jeon, and In So Kweon.  Geometric calibration of micro-lens-based light field cameras using line features.  IEEE Trans. Pattern Anal. Mach. Intell., 39(2):287–300, 2016.
  * [6] Robert C Bolles, H Harlyn Baker, and David H Marimont.  Epipolar-plane image analysis: An approach to determining structure from motion.  Int. J. Comput. Vis., 1(1):7–55, 1987.
  * [7] Caroline Conti, Luís Ducla Soares, and Paulo Nunes.  Dense light field coding: A survey.  Access, 8:49244–49284, 2020.
  * [8] Don Dansereau and Len Bruton.  Gradient-based depth estimation from 4d light fields.  In Int. Symposium on Circuits and Systems. IEEE, 2004.
  * [9] Donald G. Dansereau, Bernd Girod, and Gordon Wetzstein.  LiFF: Light field features in scale and depth.  In IEEE Conf. Comput. Vis. Pattern Recog., 2019.
  * [10] Bin Fan, Yuchao Dai, and Mingyi He.  Sunet: symmetric undistortion network for rolling shutter correction.  In Int. Conf. Comput. Vis., 2021.
  * [11] Rodrigo Ferreira and Nuno Goncalves.  Fast and accurate micro lenses depth maps for multi-focus light field cameras.  In German Conf. on Pattern Recog. Springer, 2016.
  * [12] Todor G Georgiev and Andrew Lumsdaine.  Focused plenoptic camera and rendering.  J. of Electronic Imaging, 19(2):021106, 2010.
  * [13] Stefan Heber and Thomas Pock.  Convolutional networks for shape from light field.  In IEEE Conf. Comput. Vis. Pattern Recog., 2016.
  * [14] Katrin Honauer, Ole Johannsen, Daniel Kondermann, and Bastian Goldluecke.  A dataset and evaluation methodology for depth estimation on 4d light fields.  In Asian Conf. on Comput. Vis., pages 19–34. Springer, 2017.
  * [15] Ivo Ihrke, John Restrepo, and Lois Mignard-Debise.  Principles of light field imaging: Briefly revisiting 25 years of research.  Sign. Proc. Magazine, 33(5):59–69, 2016.
  * [16] Hae-Gon Jeon, Jaesik Park, Gyeongmin Choe, Jinsun Park, Yunsu Bok, Yu-Wing Tai, and In So Kweon.  Accurate depth map estimation from a lenslet light field camera.  In IEEE Conf. Comput. Vis. Pattern Recog., 2015.
  * [17] Alexandre Karpenko, David Jacobs, Jongmin Baek, and Marc Levoy.  Digital video stabilization and rolling shutter correction using gyroscopes.  CSTR, 1(2):13, 2011.
  * [18] Yizhen Lao, Omar Ait-Aider, and Helder Araujo.  Robustified structure from motion with rolling-shutter camera using straightness constraint.  Pattern Recognition Letters, 111:1–8, 2018.
  * [19] Yizhen Lao, Omar Ait-Aider, and Adrien Bartoli.  Solving rolling shutter 3d vision problems using analogies with non-rigidity.  Int. J. Comput. Vis., 129(1):100–122, 2021.
  * [20] Chia-Kai Liang, Li-Wen Chang, and Homer H Chen.  Analysis and compensation of rolling shutter effect.  IEEE Trans. Image Process., 17(8):1323–1330, 2008.
  * [21] Haiting Lin, Can Chen, Sing Bing Kang, and Jingyi Yu.  Depth recovery from light field using focal stack symmetry.  In Int. Conf. Comput. Vis., 2015.
  * [22] Ludovic Magerand and Adrien Bartoli.  A generic rolling shutter camera model and its application to dynamic pose estimation.  In Int. symposium on 3D Data Proc., Visualiz. and Transmis., 2010\.
  * [23] Maxime Meilland, Tom Drummond, and Andrew I Comport.  A unified rolling shutter and motion blur model for 3d visual registration.  In Int. Conf. Comput. Vis., 2013.
  * [24] Marci Meingast, Christopher Geyer, and Shankar Sastry.  Geometric models of rolling-shutter cameras.  arXiv preprint cs/0503076, 2005.
  * [25] Ren Ng, Marc Levoy, Mathieu Brédif, Gene Duval, Mark Horowitz, and Pat Hanrahan.  Light field photography with a hand-held plenoptic camera.  PhD thesis, Stanford University, 2005.
  * [26] Said Pertuz, Edith Pulido-Herrera, and Joni-Kristian Kamarainen.  Focus model for metric depth estimation in standard plenoptic cameras.  J. of Photogrammetry and Remote Sensing, 144:38–47, 2018.
  * [27] Christian Perwass and Lennart Wietzke.  Single lens 3d-camera with extended depth-of-field.  In Human Vis. and Elect. imaging. SPIE, 2012.
  * [28] Santhosh Kumar Ramakrishnan, Aaron Gokaslan, Erik Wijmans, Oleksandr Maksymets, Alexander Clegg, John M Turner, Eric Undersander, Wojciech Galuba, Andrew Westbury, Angel X Chang, Manolis Savva, Yili Zhao, and Dhruv Batra.  Habitat-matterport 3d dataset (HM3d): 1000 large-scale 3d environments for embodied ai.  In Adv. Neural Inform. Process. Syst., 2021.
  * [29] Martin Rerabek and Touradj Ebrahimi.  New light field image dataset.  In Int. Confe. on Qual. of Multimed. Exp., 2016.
  * [30] Olivier Saurer, Kevin Koser, Jean-Yves Bouguet, and Marc Pollefeys.  Rolling shutter stereo.  In Int. Conf. Comput. Vis., 2013.
  * [31] Changha Shin, Hae-Gon Jeon, Youngjin Yoon, In So Kweon, and Seon Joo Kim.  Epinet: A fully-convolutional neural network using epipolar geometry for depth from light field images.  In IEEE Conf. Comput. Vis. Pattern Recog., 2018.
  * [32] Vincent Sitzmann, Semon Rezchikov, Bill Freeman, Josh Tenenbaum, and Fredo Durand.  Light field networks: Neural scene representations with single-evaluation rendering.  In Adv. Neural Inform. Process. Syst., 2021.
  * [33] Pratul P Srinivasan, Ren Ng, and Ravi Ramamoorthi.  Light field blind motion deblurring.  In IEEE Conf. Comput. Vis. Pattern Recog., 2017.
  * [34] Michael W Tao, Pratul P Srinivasan, Jitendra Malik, Szymon Rusinkiewicz, and Ravi Ramamoorthi.  Depth from shading, defocus, and correspondence using light-field angular coherence.  In IEEE Conf. Comput. Vis. Pattern Recog., 2015.
  * [35] Yingqian Wang, Longguang Wang, Zhengyu Liang, Jungang Yang, Wei An, and Yulan Guo.  Occlusion-aware cost constructor for light field depth estimation.  In IEEE Conf. Comput. Vis. Pattern Recog., 2022.
  * [36] Sven Wanner and Bastian Goldluecke.  Globally consistent depth labeling of 4d light fields.  In IEEE Conf. Comput. Vis. Pattern Recog., 2012.
  * [37] Gaochang Wu, Belen Masia, Adrian Jarabo, Yuchen Zhang, Liangyong Wang, Qionghai Dai, Tianyou Chai, and Yebin Liu.  Light field image processing: An overview.  J. of Selected Topics in Sign. Proc., 11(7):926–954, 2017.
  * [38] Niclas Zeller, Franz Quint, and Uwe Stilla.  Depth estimation and camera calibration of a focused plenoptic camera for visual odometry.  J. of Photogrammetry and Remote Sensing, 118:83–100, 2016.
  * [39] Qi Zhang, Chunping Zhang, Jinbo Ling, Qing Wang, and Jingyi Yu.  A generic multi-projection-center model and calibration method for light field cameras.  IEEE Trans. Pattern Anal. Mach. Intell., 41(11):2539–2552, 2018\.
  * [40] Shuo Zhang, Hao Sheng, Chao Li, Jun Zhang, and Zhang Xiong.  Robust depth estimation for light field via spinning parallelogram operator.  Comput. Vis. and Image Underst., 145:148–159, 2016.

## [Supplementary material WACV 2024]:
Joint 3D Shape and Motion Estimation from Rolling Shutter Light-Field Images

In this supplementary material to our paper, we provide additional
visualizations of the sequences from our dataset with rolling shutter light-
field images, as well as more details about the projection model presented in
the paper.

##  1 Rolling Shutter Light-Field Dataset

Figure 5: This figure is already presented in the main paper and is duplicated
here with bigger dimensions - Visualizations of the center views from each of
the seven different scenes. The four bottom rows are generated from samples of
the Habitat-Matterport benchmark [28]. The first column shows the GS scenario;
the second column shows the associated depth maps, and the subsequent columns
are from different motion scenarios (numbers 1, 2, 9 and 10 in Tab. 6).

Visualizations of some sequences scenes from our dataset (discussed in Section
4) are shown in Fig. 5. For each scene, we provide eleven velocity scenarios
from which every middle-exposition time position is similar, _i.e_. the center
line of pixel is the same for every image of the same scene. All the scenes
share the same eleven velocity profiles, but still possess different
deformations due to the RS effect since the center of rotation of the camera
is always different. Please notice the far right column in Fig. 5 where the
same movement creates some “squishing” in some sequences (_e.g_. second and
fourth rows) and some “stretching” in others (_e.g_. third and fifth rows).
The velocities for each scenarios are presented in Tab. 6.

Scenario number | 1 | 2 | 3 | 4 | 5
---|---|---|---|---|---
Rotations (euler angles) | [0,0,$\pi$/12] | [0,0,0] | [-$\pi$/18,0,0] | [$\pi$/18,$\pi$/18,0] | [0,0,$\pi$/12]
Translations | [0,0,0] | [0,-0.2,0] | [0,-0.05,0.05] | [0,0,0.2] | [0,-0.2,0]
Scenario number | 6 | 7 | 8 | 9 | 10
Rotations (euler angles) | [0,0,$\pi$/3] | [0,0,0] | [-$\pi$/3,0,0] | [2$\pi$/9,0,0] | [0,0,$\pi$/2]
Translations | [0,0,0] | [0,-0.8,0] | [0,0.4,0.2] | [0.4,-1.6,-0.8] | [0,-0.8,0]

Table 6: Different velocity scenarios, given in radians per frames and meters
per frames. The scenarios from 1 to 5 are the slow scenarios and the ones from
6 to 10 are the fast scenarios.

All sequences were created using Blender and the render engine Cycles. The
cameras have a $50mm$ focal length, and they are placed in a plane normal to
the $z$ axis organized in a $9\times 9$ grid with $6mm$ between each view
point. They are all oriented in a way that they have their optical axes
passing through the point $[0,0,7m]$. We also provide $1024x1024$ depth maps
generated from the camera placed at the same position that of the center view,
but with a $25mm$ focal length. Since the scene is moving with respect to the
camera, it is necessary to have a wider field of view as more of the scene can
be seen in the RS scenarios. The depth maps are normalized for a range from
$0m$ to $7m$. Some vizualisations of the depth maps are shown in the second
column of Fig. 5. The detailed results mentioned in the main paper for all
sequences and scenes are presented Tab. 7.

| abs rel $\downarrow$ | $\delta<1.25\uparrow$
---|---|---
rabbit | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10
Jeon-CVPR [16] | 0.06 | 0.08 | 0.07 | 0.07 | 0.07 | 0.1 | 0.19 | 0.12 | 0.13 | 0.34 | 0.39 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 0.82 | 1.0 | 0.91 | 0.59 | 0.35
OACC-Net [35] | 0.4 | 0.48 | 0.5 | 0.44 | 0.38 | 0.49 | 0.47 | 0.48 | 0.44 | 0.5 | 0.5 | 0.26 | 0.08 | 0.06 | 0.14 | 0.29 | 0.09 | 0.1 | 0.1 | 0.1 | 0.13 | 0.1
Ours | 0.03 | 0.03 | 0.03 | 0.02 | 0.03 | 0.03 | 0.03 | 0.03 | 0.02 | 0.03 | 0.03 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0
table | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10
Jeon-CVPR [16] | 0.03 | 0.03 | 0.04 | 0.03 | 0.05 | 0.03 | 0.05 | 0.09 | 0.05 | 0.17 | 0.07 | 1.0 | 1.0 | 0.99 | 1.0 | 1.0 | 1.0 | 0.97 | 0.96 | 0.99 | 0.76 | 0.94
OACC-Net [35] | 0.17 | 0.21 | 0.2 | 0.19 | 0.19 | 0.2 | 0.19 | 0.24 | 0.15 | 0.25 | 0.2 | 0.69 | 0.6 | 0.64 | 0.64 | 0.63 | 0.59 | 0.67 | 0.55 | 0.79 | 0.5 | 0.65
Ours | 0.02 | 0.02 | 0.02 | 0.03 | 0.02 | 0.03 | 0.04 | 0.02 | 0.04 | 0.03 | 0.04 | 0.995 | 0.995 | 0.99 | 1.0 | 0.99 | 1.0 | 1.0 | 0.99 | 0.99 | 0.98 | 1.0
bedroom | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10
Jeon-CVPR [16] | 0.02 | 0.03 | 0.02 | 0.04 | 0.06 | 0.03 | 0.07 | 0.02 | 0.11 | 0.03 | 0.07 | 1.0 | 1.0 | 1.0 | 1.0 | 0.97 | 1.0 | 0.99 | 1.0 | 0.89 | 1.0 | 0.94
OACC-Net [35] | 0.03 | 0.05 | 0.03 | 0.06 | 0.1 | 0.05 | 0.13 | 0.03 | 0.13 | 0.05 | 0.13 | 1.0 | 0.98 | 1.0 | 0.99 | 0.93 | 0.98 | 0.8 | 1.0 | 0.77 | 1.0 | 0.79
Ours | 0.03 | 0.03 | 0.03 | 0.03 | 0.03 | 0.03 | 0.04 | 0.03 | 0.04 | 0.04 | 0.05 | 1.0 | 1.0 | 1.0 | 0.999 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 0.99
couch | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10
Jeon-CVPR [16] | 0.02 | 0.03 | 0.03 | 0.03 | 0.03 | 0.02 | 0.06 | 0.05 | 0.06 | 0.05 | 0.06 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 0.98 | 1.0
OACC-Net [35] | 0.04 | 0.06 | 0.06 | 0.05 | 0.06 | 0.05 | 0.07 | 0.09 | 0.06 | 0.12 | 0.06 | 1.0 | 0.99 | 0.99 | 1.0 | 0.98 | 0.99 | 0.99 | 0.92 | 1.0 | 0.84 | 0.98
Ours | 0.03 | 0.04 | 0.03 | 0.03 | 0.03 | 0.04 | 0.06 | 0.03 | 0.11 | 0.04 | 0.06 | 1.0 | 0.999 | 1.0 | 0.999 | 0.989 | 0.999 | 0.989 | 0.992 | 0.964 | 0.958 | 0.971
fireplace | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10
Jeon-CVPR [16] | 0.05 | 0.08 | 0.08 | 0.09 | 0.1 | 0.09 | 0.09 | 0.1 | 0.32 | 0.09 | 0.14 | 0.96 | 0.9 | 0.87 | 0.9 | 0.86 | 0.84 | 0.85 | 0.89 | 0.57 | 0.86 | 0.71
OACC-Net [35] | 0.08 | 0.13 | 0.14 | 0.11 | 0.13 | 0.15 | 0.14 | 0.16 | 0.35 | 0.14 | 0.15 | 0.95 | 0.86 | 0.81 | 0.9 | 0.84 | 0.84 | 0.85 | 0.86 | 0.58 | 0.84 | 0.8
Ours | 0.12 | 0.1 | 0.12 | 0.13 | 0.1 | 0.09 | 0.09 | 0.09 | 0.32 | 0.07 | 0.13 | 0.741 | 0.778 | 0.751 | 0.716 | 0.8 | 0.837 | 0.828 | 0.875 | 0.558 | 0.92 | 0.753
living room | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10
Jeon-CVPR [16] | 0.04 | 0.04 | 0.05 | 0.04 | 0.05 | 0.05 | 0.06 | 0.07 | 0.07 | 0.1 | 0.1 | 0.99 | 1.0 | 0.99 | 0.99 | 0.96 | 0.99 | 0.99 | 0.95 | 0.99 | 0.92 | 0.94
OACC-Net [35] | 0.04 | 0.04 | 0.05 | 0.04 | 0.08 | 0.05 | 0.07 | 0.08 | 0.06 | 0.12 | 0.09 | 1.0 | 1.0 | 0.99 | 0.99 | 0.96 | 0.99 | 0.99 | 0.95 | 1.0 | 0.87 | 0.96
Ours | 0.05 | 0.05 | 0.04 | 0.04 | 0.07 | 0.05 | 0.08 | 0.06 | 0.12 | 0.08 | 0.07 | 0.974 | 0.98 | 0.983 | 0.974 | 0.93 | 0.975 | 0.92 | 0.946 | 0.928 | 0.915 | 0.878

Table 7: Detailed reconstruction error metrics for all the scenes considering
the eleven different motion scenarios (from 0 to 10) of the dataset. The
upward arrow means that a higher score is better.

##  2 Projection Model Formulation

We provide additional details of the construction of the formulation for the
RSLF projection model presented in Section 3, until Eq. (5), of the main
paper. In order to describe the projection of a point in the world coordinate
frame, we first apply a thin lens projection through the main lens and then a
pinhole projection through every micro-lens of the micro-lens array (MLA)
independently (as shown in Fig. 6).

Figure 6: This figure is already presented in the main paper and is duplicated
here for the convenience of the reader - The adopted LF coordinate frames: The
3D point is projected in a 3D virtual scene by thin lens projection, then on
the 2D image plane by pinhole projection which coordinate frame depends on the
considered viewpoint.

Given a point in the world homogeneous coordinate frame
${}^{w}\widetilde{\mathbf{p}}=(x_{w},y_{w},z_{w},1)^{\top}$, its coordinates
in the camera coordinate frame will be given by the matrix
${}^{c}\mathbf{M}_{w}{}^{w}$ as

${}^{c}\widetilde{\mathbf{p}}={}^{c}\mathbf{M}_{w}{}^{w}\widetilde{\mathbf{p}},$
(10)

where $\widetilde{\mathbf{p}}$ represents the homogeneous coordinates of point
$\mathbf{p}$ in the left-upperscripted reference frame, while
${}^{c}\mathbf{M}_{w}$ is the homogeneous matrix displacing the camera frame
onto the world frame:

${}^{c}\mathbf{M}_{w}=\left[\begin{array}[]{c
c}{}^{c}\mathbf{R}_{w}&{}^{c}\mathbf{T}_{w}\\\
\boldsymbol{0}^{\top}&1\end{array}\right].$ (11)

We will then project it through the main lens as
${}^{v}\widetilde{\mathbf{p}}=(x_{v},y_{v},z_{v},1)^{\top}$, using the
projection matrix for a thin lens projection as

$\lambda_{c}{}^{v}\widetilde{\mathbf{p}}=\underbrace{\begin{bmatrix}1&0&0&0\\\
0&1&0&0\\\ 0&0&1&0\\\ 0&0&-\frac{1}{F}&1\\\
\end{bmatrix}}_{=:\mathbf{K}_{c}}{}^{c}\widetilde{\mathbf{p}},$ (12)

with $F$ the focal distance of the main lens and $\lambda_{c}$ a scaling
factor. To express this point $\widetilde{\mathbf{p}}=(x,y,z,1)^{\top}$ in the
sensor coordinate frame, we use the homogeneous matrix $\mathbf{D}$ linked to
the geometry of the sensor as

$\widetilde{\mathbf{p}}=\underbrace{\begin{bmatrix}1&0&0&O_{x}\\\
0&1&0&O_{y}\\\ 0&0&1&d\\\ 0&0&0&1\\\
\end{bmatrix}}_{=:\mathbf{D}}{}^{v}\widetilde{\mathbf{p}},$ (13)

with $\mathbf{O}=(O_{x},O_{y},0)^{\top}$ the intersection of the optical axis
and the view plane and $d$ the distance between the optical center of the main
lens and the view plane.
Given a point $\mathbf{c}=(s,t,0)^{\top}$ from the view plane, _i.e_. a
projection center, the pinhole projection to an image point
$\widetilde{\mathbf{m}}^{s,t}=(x^{s,t},y^{s,t},1)^{\top}$ is given by the
matrix

$\lambda_{s}^{s,t}\widetilde{\mathbf{m}}^{s,t}=\underbrace{\begin{bmatrix}f&0&0&-fs\\\
0&f&0&-ft\\\ 0&0&1&0\\\
\end{bmatrix}}_{=:\mathbf{K}_{s}^{s,t}}\widetilde{\mathbf{p}},$ (14)

with $f$ the focal distance of the micro-lenses, _i.e_. the distance between
the view plane and the image plane and $\lambda_{s}$ a scaling factor. This is
a classical pinhole projection that take into account the position of the
micro-lens $\mathbf{c}$. From these equations we can find the LF point
$(x,y,s,t)$ for any 3D point in the world, in the case of a LF camera modeled
with the GS hypothesis. In order to add the RS effect in our model, and thus
the movement of the camera in our equations, we adopt a similar RS formalism
from Ait-Aider _et al_. [3]. They considered that the camera moves by the same
little uniform movement between any two lines of pixels and define camera pose
in function of the pixel line. We also make the hypothesis that the
acquisition time inside a micro-image is instantaneous (_i.e_. micro-images
are considered GS). Assuming a uniform movement $[\delta\mathbf{R}^{\delta
t}\mid\delta\mathbf{T}^{\delta t}]$ between any two lines of micro-images, we
can express the camera pose in function of the viewpoint line and rewrite Eq.
10 as

${}^{c}\widetilde{\mathbf{p}}=\left[\begin{array}[]{c
c}\delta\mathbf{R}^{t}{}^{c}\mathbf{R}_{w}&{}^{c}\mathbf{T}_{w}+\delta\mathbf{T}^{t}\\\
\boldsymbol{0}^{\top}&1\end{array}\right]{}^{w}\widetilde{\mathbf{p}},$ (15)

with

$\delta\mathbf{R}^{t}=\mathbf{a}\mathbf{a}^{\top}(1-\cos(\Omega\tau
t))+\boldsymbol{I}\cos(\Omega\tau t)+[\mathbf{a}]_{\wedge}\sin(\Omega\tau
t),\mbox{ and }\delta\mathbf{T}^{t}=\mathbf{v}\tau t,$ (16)

with $\mathbf{a}$ (axis of rotation) $\Omega$ (angular velocity) and
$\mathbf{v}$ (linear velocity) describes the uniform movement of the camera
coordinate frame with respect to the world coordinate frame. $\tau$ is the
time between the acquisition of two lines of point of view and $t$ is the line
coordinate of the point of view $\mathbf{c}=(s,t,0)^{\top}$ from Eq. 14. The
complete RSLF projection of the 3D point ${}^{w}\widetilde{\mathbf{p}}_{i}$ to
a image point $\mathbf{m}_{i}^{s,t}$, given a center of projection
$\mathbf{c}=(s,t,0)^{\top}$, is then

$\lambda\mathbf{m}_{i}^{s,t}=\mathbf{K}_{s}^{s,t}\mathbf{D}\mathbf{K}_{c}[\delta\mathbf{R}^{t}{}^{c}\mathbf{R}_{w}\mid{}^{c}\mathbf{T}_{w}+\delta\mathbf{T}^{t}]{}^{w}\widetilde{\mathbf{p}}_{i}.$
(17)

That can be simplified as

$\lambda\mathbf{m}_{i}^{s,t}=\mathbf{K}^{s,t}[\delta\mathbf{R}^{t}{}^{c}\mathbf{R}_{w}\mid{}^{c}\mathbf{T}_{w}+\delta\mathbf{T}^{t}]{}^{w}\widetilde{\mathbf{p}}_{i},$
(18)

with

$\mathbf{K}^{s,t}=\begin{bmatrix}f&0&-\frac{f}{F}(O_{x}-s)&f(O_{x}-s)\\\
0&f&-\frac{f}{F}(O_{y}-t)&f(O_{y}-t)\\\ 0&0&1-\frac{d}{F}&d\end{bmatrix},$
(19)

which is presented in Eq. (5) of the main paper. As discussed in the main
paper, this projection model combines at the same time the property of a
light-field sensor and a rolling shutter sensor. Specifically, the model can
be extended to a GS light-field camera, when the temporal delay between two
consecutive lines is zero, $\tau=0$, and thus the position of the sensor with
respect to the scene will be identical for any $t$. In fact, Eq. 15 will be
simplified into Eq. 10, that is the global shutter case of the light-field
projection described earlier. For similar reasons, the model will act like a
global shutter light-field camera when the camera has no velocity with respect
to the object. The RSLF projection model generalizes to a conventional camera
projection in the case where the MLA is composed of a unique lens. Indeed if
we set $s=0$ and $t=0$, Eq. 19 becomes

$\mathbf{K}^{0,0}=\begin{bmatrix}f&0&-\frac{f}{F}O_{x}&fO_{x}\\\
0&f&-\frac{f}{F}O_{y}&fO_{y}\\\ 0&0&1-\frac{d}{F}&d\end{bmatrix}$ (20)

which correspond to a pinhole camera with projection matrix
$\mathbf{K}^{\prime}$ at position $\mathbf{D}^{\prime}$, with

$\mathbf{K}^{\prime}=\begin{bmatrix}f&0&c_{x}\\\ 0&f&c_{y}\\\
0&0&1\end{bmatrix},~{}~{}\mathbf{D}^{\prime}=\begin{bmatrix}1&0&0&O_{x}-c_{x}\frac{d}{F}\\\
0&1&0&O_{y}-c_{y}\frac{d}{F}\\\ 0&0&\frac{fO_{x}}{Fc_{x}}&d\end{bmatrix},$
(21)

with $c_{x}=(\frac{d}{F}-1)\frac{f}{F}O_{x}$ and
$c_{y}=(\frac{d}{F}-1)\frac{f}{F}O_{y}$. Since we consider that the micro-
images are locally global shutter, this pinhole model is global shutter.

#### Implementation details.

The non-linar bundle adjustment discussed in Section 3.1 of the main paper was
implemented using PyTorch with the Adam optimizer, with learning rate 0.01 for
5000 iterations to ensure convergence. For the regularization of our
optimization method discussed in the end of Section 3 of the main paper, we
use a new coordinate frame in order to provide a center of rotation to be
optimized and a normalization for the point cloud. We define the new points
${}^{n}\mathbf{p}_{i}$ as:

${}^{n}\mathbf{p}_{i}=\frac{\mathbf{p}_{i}-\mathbf{g}}{\lambda_{n}},$ (22)

with $\mathbf{g}$ the center of rotation and $\lambda_{n}$ the normalization
factor. $\mathbf{g}$ is initialized as the mean position of the initial points
$\mathbf{p}_{i}$ and $\lambda_{n}$ is defined before the optimization and is
calculated as

$\lambda_{n}=\text{max}(\mathbf{p}_{i}-\mathbf{g}).$ (23)


# Berry phases in Coulomb drag of double-layer graphene system

Jianghui Pan  International Center for Quantum Materials, School of Physics,
Peking University, Beijing 100871, China  Lijun Zhu  CAS Key Laboratory of
Strongly-Coupled Quantum Matter Physics, and Department of Physics, University
of Science and Technology of China, Hefei 230026, China  International Center
for Quantum Design of Functional Materials (ICQD), Hefei National Research
Center for Physical Sciences at the Microscale, University of Science and
Technology of China, Hefei 230026, China  Xiaoqiang Liu  International Center
for Quantum Design of Functional Materials (ICQD), Hefei National Research
Center for Physical Sciences at the Microscale, University of Science and
Technology of China, Hefei 230026, China  Lin Li  CAS Key Laboratory of
Strongly-Coupled Quantum Matter Physics, and Department of Physics, University
of Science and Technology of China, Hefei 230026, China  International Center
for Quantum Design of Functional Materials (ICQD), Hefei National Research
Center for Physical Sciences at the Microscale, University of Science and
Technology of China, Hefei 230026, China  Hefei National Laboratory, Hefei
230088, China  Changgan Zeng  CAS Key Laboratory of Strongly-Coupled Quantum
Matter Physics, and Department of Physics, University of Science and
Technology of China, Hefei 230026, China  International Center for Quantum
Design of Functional Materials (ICQD), Hefei National Research Center for
Physical Sciences at the Microscale, University of Science and Technology of
China, Hefei 230026, China  Hefei National Laboratory, Hefei 230088, China  Ji
Feng<EMAIL_ADDRESS>International Center for Quantum Materials, School of
Physics, Peking University, Beijing 100871, China  Hefei National Laboratory,
Hefei 230088, China

###### Abstract

Recent experiments suggest quantum interference effects in the Coulomb drag of
double-layer graphene systems. By accounting for correlated interlayer
impurity scattering under a weak magnetic field, our theoretical results
reveal drag resistivities resembling those in weak (anti-)localization. It is
established that the quantum interference effect is most significant when the
chemical potentials match. The theory clarifies the roles of intra- and
interlayer Berry phases in Coulomb drag in double-layer graphene systems and
helps delineate the intra- and intervalley contributions. These insights are
valuable for designing graphene-based electronic devices exploiting quantum
effects.

††preprint: APS/123-QED

Introduction. Coulomb drag, effected by momentum transfer between the charge
carriers of two separated conductors[1], offers a non-contact method for
characterizing basic properties of a wide range of materials. Coulomb drag
measurement is particularly suitable for low-dimensional systems, typically in
a double-layer device configuration. Specifically, Coulomb drag has been
observed in two graphene layers separated by a dielectric,[2, 3, 4, 5, 6, 7,
8, 9, 10, 11, 12, 13, 14, 15, 16] as well as in related systems.[17, 18] In
monolayer graphene, quantum interference is also affected by the unusual Berry
phase of a Dirac Fermion.[19, 20, 21] The situation is more complicated in
double-layer devices, where the intralayer localization, described by the
Cooperon propagator, has been argued not to contribute to the drag
current.[22] Recent experimental observations in a graphene double-layer
system indicate that the magneto-transresistance deviates from the usual $\sim
B^{2}$ law at low fields, suggesting weak localization (WL)-like or weak anti-
localization (WAL)-like behaviors in different doping regimes.[23, 16]

Indeed, unlike the intralayer case where momentum conservation ensures that
the Berry phase enters into backscattering, such kinematic constraints are
apparently absent in interlayer processes. Consequently, the interlayer Berry
phase manifested in the interlayer Cooperon [24, 25] is expected weigh equally
into forward and backward scattering, even when correlated scattering of
carriers residing on different layers becomes important with a long phase
diffusion time $\tau_{g}$. Meanwhile, it is unclear how the intralayer Berry
phase creeps into correlated interlayer scattering processes.

Figure 1:  (a) A schematic diagram of the Coulomb drag effect and the
interlayer impurity scattering. (b) Also shown are the Feynman diagrams for
the second-order drag conductivity (up) without and (down) with correlated
interlayer impurity scattering. Solid lines represent electron free
propagation, while dashed and wavy lines represent the interlayer impurity
potentials and the screened Coulomb interaction, respectively. (c) Bethe-
Salpeter equation for the interlayer Cooperon. (d) Feynman diagrams for the
second-order Coulomb drag with interlayer Cooperons (shaded block).

In this report, we examine the role of correlated interlayer scattering (CIS)
caused by impurities in the Coulomb drag of double-layer graphene systems,
considering both intra- and intervalley scattering. Based on finite
temperature perturbation theory, we systematically calculate the drag
resistivity correction from quantum interference with interlayer Cooperon
under a weak magnetic field, and provide the dependence of the correction on
chemical potentials, temperature, and interlayer thickness. The microscopic
calculation uncovers that there is a remarkable interplay between the
intralayer and interlayer Berry phases in the Cooperon-mediated drag
transport, leading to a dominance of forward scattering. Additionally, the
correlated scattering time $\tau_{g}$ for CIS indicates that quantum
interference effects are most significant when the chemical potentials match.

Theory. We consider two separated graphene layers coupled solely by Coulomb
interaction. The low-energy excitations in a monolayer graphene are described
by the usual Dirac Hamiltonian:

$H_{l}=v_{\text{F}}(\xi p_{l}^{x}\sigma_{x}+p_{l}^{y}\sigma_{y}),$ (1)

where $\sigma_{i}\,(i=x,y,z)$ are the Pauli matrices for lattice pseudospin,
$l=1,2$ is the layer index, $\xi=\pm 1$ is the valley index, and $p_{l}^{x}$
and $p_{l}^{y}$ are the momenta in the absence of interlayer coupling. The
eigenvalues of $H_{l}$ are given by
$\varepsilon_{l\alpha}=b_{l}v_{\text{F}}k_{l}$ ($\hbar=1$), with $\alpha$
encapsulating valley $\xi=\pm 1$, band $b=\pm 1$, and wavevector $\bm{k}$. The
retarded/advanced single-particle Green’s functions read
$G^{\text{R/A}}_{l\alpha}(\varepsilon)=1/(\varepsilon-\varepsilon_{l\alpha}\pm\text{i}/2\tau_{l}),$
where $1/2\tau_{l}=-\operatorname{Im}\Sigma_{l}$ is assumed to arise primarily
from intralayer scattering.

In a double-layer device depicted in Fig. 1(a), the linear current response is
generally described by
$J_{l}^{a}=\sigma_{ll^{\prime}}^{ab}E_{l^{\prime}}^{b}$, where
$l,l^{\prime}=1,2$ and $a,b=x,y$. The drag conductivities $\sigma_{12}$ and
$\sigma_{21}$ arise from the momentum exchange of charge carriers in the two
separate layers via Coulomb scattering. The longitudinal retarded current-
current correlation function can be obtained by standard perturbation theory
[22, 26], from which the leading-order drag conductivity corresponds to the
Feynman diagram in Fig. 1(a). The diagram is second order in the screened
Coulomb potential $U(\bm{q})$, and involves two triangular correlation
functions
$\Delta^{a}_{l}(\bm{q},\bm{q}^{\prime};\omega,\omega^{\prime};\bm{B})$[22],
depicted as triangles in Fig. 1(a). Correlated interlayer scattering can be
modeled by the interlayer Cooperon inserted between $\Delta_{1}$ and
$\Delta_{2}$, represented as blue-shaded four-legged vertices in Fig. 1(c),
leading to a total 18 diagrams $\\{\Pi_{i}\\}$ as shown in Fig. 1(d). The
corresponding correction to drag resistivity involving the interlayer Cooperon
is then:

$\delta\rho_{\text{D}}=\frac{e^{2}}{2\sigma_{11}\sigma_{22}}\sum_{i=1}^{18}\int\frac{\text{d}\bm{q}\text{d}\omega}{(2\pi)^{3}}\frac{\partial
n_{B}(\omega)}{\partial\omega}\Pi_{i}(\bm{q},\omega;\bm{B}),$ (2)

where $n_{B}(\omega)$ is the Bose function and $\bm{B}$ is the magnetic field
perpendicular to the plane.

The interlayer Cooperon differs from the usual intralayer Cooperon. The
intralayer Cooperon represents the interference between paths related by time
reversal taken by the same electron. Time reversal corresponds to electron
backscattering, resulting in constructive interference and localization. As
depicted in Fig. 1(a), the interlayer Cooperon describes electrons from
different layers moving along superimposed planar paths under the effect of
interlayer impurity scattering.[16] This correlated electron motion can result
in a phenomenon similar to weak localization in a single layer. The interlayer
Cooperon satisfies the Bethe-Salpeter equation (BSE) depicted in Fig. 1(d).
The bare vertex is
$C_{\beta\beta^{\prime}\alpha\alpha^{\prime}}^{0}=\overline{U_{\beta\alpha}U_{\beta^{\prime}\alpha^{\prime}}}$
(overline indicates disorder averaging), evaluated to be
$C^{0}_{\beta\alpha}=\frac{sv^{2}}{2\tau_{i}\mu}\left(1+2e^{\text{i}\phi}+e^{2\text{i}\phi}\right)$
for intravalley scattering, and
$C^{0}_{\beta\alpha}=\frac{s\xi_{\alpha}\xi_{\beta}v^{2}}{2\tau_{i}\mu}e^{\text{i}\phi}$
for intervalley scattering. Here, $\tau_{i}$ is the interlayer impurity
relaxation time, $\phi$ is the angle between $\bm{k}_{\beta}$ and
$\bm{k}_{\alpha}$, and $s=b_{\alpha}b_{\beta}=\pm 1$. Due to momentum
conservation, we have
$\bm{Q}=\bm{k}_{\alpha}+\bm{k}_{\alpha^{\prime}}=\bm{k}_{\beta}+\bm{k}_{\beta^{\prime}}$
and $\Xi=\xi_{\alpha}+\xi_{\alpha^{\prime}}=\xi_{\beta}+\xi_{\beta^{\prime}}$.
Since for large $\mu_{i}$, interband processes are unimportant, the BSE can be
written as

$\displaystyle
C_{\beta\alpha}(\bm{Q},\Xi;\bm{B})=C^{0}_{\beta\alpha}(\bm{Q},\Xi)+\sum_{\bm{k}_{\lambda},\xi_{\lambda}}C^{0}_{\beta\gamma}(\bm{Q},\Xi)$
(3) $\displaystyle\times G^{\text{
R/A}}_{1,\bm{k}_{\gamma}}(\mu;\bm{B})G^{\text{R/A}}_{2,\bm{Q}-\bm{k}_{\gamma}}(\mu+\Delta\mu;\bm{B})C_{\gamma\alpha}(\bm{Q},\Xi;\bm{B}).$

In contrast to the single-layer system, the Cooperon depends on the chemical
potentials, $\mu_{l}$, of the two separate layers, which can be tuned
independently. The combination of Green’s functions that appear on the right-
hand side of Eq. (3) depends on $s=\operatorname{sign}(\mu_{1}\mu_{2})$:
$G_{1}^{\text{R}}G_{2}^{\text{A}}$ and $G_{1}^{\text{A}}G_{2}^{\text{R}}$ for
$s=1$, and $G_{1}^{\text{R}}G_{2}^{\text{R}}$ and
$G_{1}^{\text{A}}G_{2}^{\text{A}}$ for $s=-1$. Under a weak magnetic field,
the momentum $\bm{Q}$ satisfies the semiclassical Einstein-Brillouin-Keller
quantization condition $Q_{n}^{2}=4eB(n+\gamma/4-\gamma_{C}/2\pi)$, where
$\gamma=2$ is the Maslov index in this system and $\gamma_{C}=2\pi(1+s)$ is
the total intralayer Berry phase. Keeping only the divergent part in the weak-
scattering limit, the Cooperon has the form:

$C_{\beta\alpha}(\bm{Q})=\frac{sv^{2}}{\tau\tau_{i}\mu}\frac{\xi_{\alpha}\xi_{\beta}e^{\text{i}\phi}}{\tau_{g}^{-1}\pm\mathrm{i}\Delta\mu+DQ_{n}^{2}}$
(4)

where $\Delta\mu=|\mu_{2}|-|\mu_{1}|$ plays the role of a frequency in static
limit, $2\mu=|\mu_{1}|+|\mu_{2}|$, $2/\tau=1/\tau_{1}+1/\tau_{2}$, and the
correlated scattering time is

$\frac{1}{\tau_{g}}=\frac{\tau_{i}}{\tau^{2}}-\left(1-\frac{\Delta\mu}{\mu}\right)\frac{1}{\tau},$
(5)

and the interlayer phase diffusion coefficient is $D=v^{2}\tau_{\text{tr}}/2$.
For a divergent Cooperon, it is required that $\tau_{i}=\tau$ and
$\Delta\mu=0$ simultaneously. The angle subtended by $\bm{k}_{\beta}$ and
$\bm{k}_{\alpha}$, $\phi$, is the interlayer Berry phase, as it is the
geometric phase from CIS, whose role in the drag resistivity will be clarified
in the next section.

Role of interlayer Berry phase. Despite the apparent divergence of the
Cooperon in the CIS regime, it is not obvious whether or how the Berry phase
enters into the drag conductivity. In the monolayer case[19], the conservation
of momentum at the current vertices immediately makes the Berry phase of the
Cooperon directly relevant to backscattering. In the double-layer drag setup,
however, this kinematic constraint is absent. To clarify this issue, we
examine the dominant terms of all $\Pi_{i}$’s, where the Cooperon is inserted
between the Coulomb vertices (bottom pair in Fig. 1(d)). As $Q\rightarrow 0$,
the product of the Coulomb vertices

$U(\bm{q})U\left(\bm{q}+\bm{p}\right)\equiv e^{f(\bm{q},\bm{p})}$ (6)

is peaked at $\bm{q}_{\star}=-\bm{p}/2$, with
$\bm{p}=\bm{k}_{\beta}-\bm{k}_{\alpha}$. Applying the Laplace method [27] to
$e^{f}$ in the integral over $\bm{q}$ yields

$\displaystyle\delta\rho_{\text{D}}$
$\displaystyle\approx\frac{-e^{2}T^{2}}{6\sigma_{11}\sigma_{22}}C(\mu)\int\limits_{0}^{2\pi}\frac{\text{d}\phi}{2\pi}\frac{2\pi-\phi-\sin\phi}{\sqrt{\det(-f^{(2)}_{\star}})}G_{\bm{k}+\bm{k}^{\prime}}^{\text{R}}G^{\text{A}}_{\bm{k}+\bm{k}^{\prime}}$
(7)
$\displaystyle\times\cos\phi\left(1+\cos\phi+2|\cos\tfrac{\phi}{2}|\right)^{2}\times
e^{f(\bm{q_{\star}})}$

where

$C(\mu)=\frac{v^{2}}{2D\tau\tau_{i}\mu}\left[\psi(\frac{\ell^{2}_{B}}{2\ell^{2}_{\tau}}+\frac{1}{2})-\psi(\frac{\ell^{2}_{B}}{2\ell^{2}_{\phi}}+\frac{1}{2})\right]$
(8)

is for intravalley scattering in which $\ell_{\phi}$ and $\ell_{\tau}$ are
phase coherence and mean free lengths, respectively. $\ell_{B}=(2eB)^{-1/2}$
is the magnetic length and $\psi$ is the digamma function. $f^{(2)}_{\star}$
is the Hessian of $f(\bm{q})$ at $\bm{q}_{\star}$.

Eq. (7) leads immediately to a simple relation between intra- and intervalley
$\delta\rho_{D}$ when impurity scattering times coincide for ultra-short
ranged impurities

$\eta=\delta\rho_{\mathrm{D},\mathrm{intra}}/\delta\rho_{\mathrm{D},\mathrm{inter}}=-2.$
(9)

This relation originates from the difference in numbers of intra- and
intervalley Cooperons, as can be deduced from the summation in Eq. (4). In
general, it is expected that the inter- and intravalley $\delta\rho_{D}$ may
deviate from this exact relation, but may generally have opposite signs. In
what follows, we will discuss exclusively the intravalley $\delta\rho_{D}$.

The drag resistivity correction exhibits a broadening of approximately
$0.5~{}\mathrm{T}$ under the magnetic field, displaying a trend similar to
that of the intralayer weak localization. The presence of a weak magnetic
field primarily influences the overall value of $\delta\rho_{D}$; therefore,
the dependence of $\delta\rho_{D}$ on various parameters will be investigated
without a magnetic field in the following sections.

We have collected terms from the Berry phase on the second line in Eq. (7),
which reflect the angular dependence of the integrand: while the first term
$\cos\phi$ represents the interlayer Berry phase from the Cooperon in Eq. (4),
the second term within the parentheses arises from the intralayer Berry phase.
Evidently, it is the interplay of both intra- and interlayer Berry phase terms
that determines how the CIS impacts the drag resistivity. For calibration, we
note that in the monolayer case,[19] only $\cos\phi$ enters, but momentum
conservation dictates that $\phi\sim\pi$, which leads to WAL/WL in the
inter/intravalley processes. In the CIS process in a double-layer system,
however, such a kinematic constraint is absent. It is noted that the
intralayer term vanishes at $\phi=\pi$, but increases to a maximum at
$\phi=0$.

It is inferred that the intralayer Berry phase term diminishes backscattering
and favors forward scattering. The interlayer Berry phase then determines the
overall sign of the drag resistivity, in conjunction with the intralayer Berry
phase along with $s$ and valley index. In the weak-scattering regime
$\tau\mu\gg 1$ and $B=0$, Eq. (7) reduces to

$\frac{\delta\rho_{\text{D}}}{\rho_{\text{D}}}\approx-s\frac{48\pi^{3}}{\zeta(3)}\sqrt{2|\mu|\tau}\log\frac{\ell_{\phi}}{\ell_{\tau}}.$
(10)

We see that there is a significant CIS correction to the drag resistivity, and
$\delta\rho_{\text{D}}/\rho_{\text{D}}$ depends on temperature only through
the temperature dependencies of $\tau$ and $\tau_{\phi}$.

Figure 2:  The angular $\phi$-dependence of $\delta\rho_{D}$ is shown for
various $d$ at $T=200\mathrm{K}$ and $A=100$. The integrands of
$\delta\rho_{D}$ are displayed in a polar plot. Dots and lines correspond to
the integrands in Eqs. (2) and (7), respectively. The integral values of
$\delta\rho_{D}$ at various $d$ are normalized to 1.

Numerical results. While the foregoing analysis reveals the interplay of
intra- and interlayer Berry phase terms in the drag resistivity, it is
purported to pertain to the weak scattering regime. To further explore the
effects of temperature $T$, interlayer thickness $d$, and impurity strength
$A^{-1}$[28] on drag resistivity, we numerically integrate Eq. (2) using the
stratified Monte Carlo technique [29]. The interlayer impurity relaxation time
is expressed as $1/\tau_{i}=2\pi\mu/A$, and the intralayer impurity relaxation
time $\tau$ is determined through the self-consistent Born approximation[30].
The screened Coulomb potential takes the form

$U(\bm{q})=\frac{e^{2}q}{\epsilon q_{\text{s}1}q_{\text{s}2}\sinh(qd)}$ (11)

in which $q_{\text{s}l}$ are the Thomas-Fermi wavevectors of layer $l$[31].
The summation over $\bm{Q}$ in the Cooperon will be carried out with the upper
and lower momentum cutoffs being $\ell^{-1}_{\tau}$ and $\ell^{-1}_{\phi}$,
respectively. For simplicity, we estimate that $\ell_{\phi}$ is four orders of
magnitude larger than $\ell_{\tau}$[3].

Fig. 2 shows numerical results of the $\phi$-dependence for $\delta\rho_{D}$,
illustrating the impact of the Berry phase and Coulomb interaction. The
integrand as a function of $\phi$ in Eq. (2) is numerically computed (dots in
Fig. 2), and the integrand of Eq. (7) is plotted as lines for comparison. It
can be seen that the contribution of forward scattering completely dominates,
and strict backscattering is completely suppressed, as expected from our
theory. It should also be noted that the Coulomb interaction contributes to
the anisotropy by virtue of the Coulomb term in Eq.(6), which peaks at
$\bm{q}_{\star}=-\bm{p}/2$, further favoring forward scattering. This argument
is fairly generic, only assuming $U(\bm{q})$ is a simple peaked function.

Fig. 3(a) shows $\delta\rho_{\mathrm{D}}/\rho_{\mathrm{D}}$ as a function of
carrier densities in the two layers with only intravalley scattering. Since
our model pertains only to intraband impurity scattering. our results are only
meaningful at relatively high carrier densities ($|n_{l}|>5\times
10^{11}~{}\mathrm{cm^{-2}}$). We can divide the diagram into four parts based
on carrier types (e for $n_{l}>0$, h for $n_{l}<0$) in the two layers: e-e,
e-h, h-e and h-h sectors. Due to particle-hole symmetry,
$\delta\rho_{D}/\rho_{D}$ in the four sectors is interrelated by $s=\pm 1$, in
accordance with (10). The most important feature we find is that the quantum
interference mainly occurs where $|n_{1}|\approx|n_{2}|$, showing WAL-like
behavior in the e-e (h-h) sectors, and WL-like behavior in the e-h (h-e)
sectors.

Figure 3: The relationship of $\delta\rho_{\mathrm{D}}/\rho_{\mathrm{D}}$ (a)
as a function of $n_{1}$ and $n_{2}$ at
$T=200~{}\mathrm{K},A=100,d=11.3~{}\mathrm{nm}$, (b) as a function of $n_{1}$
at various $n_{2}$. (c) The temperature and (d) the impurity strength
dependence of $\delta\rho_{\mathrm{D}}/\rho_{\mathrm{D}}$ for $n=n_{1}=n_{2}$
at $A=100$ or $T=200~{}\mathrm{K}$.

In Fig. 3(b), the ratio $\delta\rho_{\text{D}}/\rho_{\text{D}}$ is plotted as
a function of $n_{2}$, for two chosen values of $n_{1}$ (vertical lines in
Fig. 3(a)). The maximum occurs with identical densities. Note that $|n_{l}|$
need not be exactly eqaul for significant Cooperon contribution to $\rho_{D}$,
at finite temperatures and small $\tau_{g}^{-1}$ values. We observe a
crossover from WL-like to WAL-like to WL-like again in the e-e (h-h) region,
and vice versa in the e-h (h-e) region, although as has been emphasized, the
low-density results are not physically relevant. Fig. 3(c,d) shows the
temperature and impurity strength dependence of $\delta\rho_{D}/\rho_{D}$. The
key feature is that $\delta\rho_{D}/\rho_{D}$ decreases as carrier densities
$n$, temperatures $T$ or impurity strength $A^{-1}$ increases. This can be
simply understood, as thermal excitation and/or varied impurity strength
causes a mismatch in the inter- and intra-layer scattering times, destroying
the CIS.

Figure 4: (a) $\tau_{g}$ as a function of $n_{1}$ in layer-1 with $A=100$.
Dashed lines: $\tau_{g}$ when $n_{2}=2\times 10^{12}\mathrm{cm^{-2}}$. Solid
lines: $\tau_{g}$ when chemical potentials match. Dash-dot line: $\tau_{g}$ in
SLG-BLG with ${n_{2}=2.6\times 10^{11}\mathrm{cm^{-2}}}$ which maximizes
$\tau_{g}/\tau$ when Fermi surfaces match. (b) Experimentally measured (dots)
$\delta\rho_{\text{D}}/\rho_{\text{D}}$ in the BLG-BLG system. The solid lines
are Lorentzian fits. (c) Experimental $\delta\rho_{D}/\rho_{D}$ with
$n_{2}=0.35\times 10^{12}~{}\mathrm{{cm}^{-2}}$ at various $n_{1}$. The total
$\delta\rho_{D}$ (blue) is partitioned into intra- (orange dash-dot) and
intervalley contributions (green dash-dot) (see SM).

It is actually possible to generalize the foregoing discussions on single
layer-single layer (SS) systems to other graphene bilayer devices, such as
bilayer-bilayer (BB) and single layer-bilayer (SB) systems. The appearance of
interlayer quantum interference effects arises from CIS characterized by
$\tau_{g}/\tau$, which is shown for different systems in Fig (4)(a).
Interlayer coherence occurs when the Fermi surfaces of the two layers
coincide. In the case of SS and BB systems, the two layers will have identical
carrier densities. In the SB case, the two layers have different carrier
densities when the coincidence occurs. As the intralayer Berry phase $2\pi$
also enters into the sign of $\delta\rho_{D}$ (see Eq. (7)), the system shows
WAL-like behavior in the e-e (h-h) sectors and WL-like behavior in the e-h
(h-e) sectors. However, for an SB system, the total intralayer Berry phase is
$\pi$, leading to the opposite behavior.

Finally, we briefly relate the theory to recent experiments [23, 16]. The
experimental $\delta\rho_{\text{D}}/\rho_{\text{D}}$ versus electron density
for a BLG-BLG system is shown in Fig. (4)(b). It is noted that
$\delta\rho_{\text{D}}/\rho_{\text{D}}$ increases as the difference in carrier
densities diminishes, aligning with theoretical expectations. However, the
experimental $\delta\rho_{\text{D}}/\rho_{\text{D}}$ does not vanish at high
carrier densities, which cannot be explained by the current theory.
Intriguingly, the experimental $\delta\rho_{\text{D}}/\rho_{\text{D}}$ can be
perfectly fitted with the sum of two concentric Gaussians with opposite signs,
as shown in Fig. (4)(c). It is found that the ratio $\eta$ of the heights of
the two Gaussians falls in the range [-2.5, -1.9]. Relating to Eq. (9), we
ascribe the two opposing contributions to inter- and intravalley contributions
to $\delta\rho$ from correlated interlayer scattering (CIS). It is then
inferred from the observation that $\eta\sim-2$ that the experimental devices
are closer to the weak scattering limit with strongly localized interlayer
impurities.

###### Acknowledgements.

We acknowledge the financial support from the National Key R&D Program of
China (Grant No. 2021YFA1400100 and 2023YFA1406300), the National Natural
Science Foundation of China (Grants No. 12274003 and 11934001), the Innovation
Program for Quantum Science and Technology (Grant No. 2021ZD0302600), and the
CAS Project for Young Scientists in Basic Research (Grant No. YSBR-046).

## References

  * Pogrebinskii [1977] M. Pogrebinskii, Mutual drag of carriers in a semiconductor-insulator-semiconductor system, Sov. Phys. Semicond. 11, 372 (1977).
  * Castro Neto  _et al._ [2009] A. H. Castro Neto, F. Guinea, N. M. R. Peres, K. S. Novoselov, and A. K. Geim, The electronic properties of graphene, Rev. Mod. Phys. 81, 109 (2009).
  * Das Sarma  _et al._ [2011] S. Das Sarma, S. Adam, E. H. Hwang, and E. Rossi, Electronic transport in two-dimensional graphene, Rev. Mod. Phys. 83, 407 (2011).
  * Min  _et al._ [2008] H. Min, G. Borghi, M. Polini, and A. H. MacDonald, Pseudospin magnetism in graphene, Phys. Rev. B 77, 041407 (2008).
  * Hwang and Das Sarma [2009] E. H. Hwang and S. Das Sarma, Plasmon modes of spatially separated double-layer graphene, Phys. Rev. B 80, 205405 (2009).
  * Feldman  _et al._ [2009] B. Feldman, J. Martin, and A. Yacoby, A. broken-symmetry states and divergent resistance in suspended bilayer graphene, Nature Phys. 5, 889 (2009).
  * Hirata  _et al._ [2021] M. Hirata, A. Kobayashi, C. Berthier, and K. Kanoda, Interacting chiral electrons at the 2D Dirac points: a review, Rep. Prog. Phys. 84, 036502 (2021).
  * Kim  _et al._ [2011] S. Kim, I. Jo, J. Nah, Z. Yao, S. K. Banerjee, and E. Tutuc, Coulomb drag of massless fermions in graphene, Phys. Rev. B 83, 161401 (2011).
  * Hwang  _et al._ [2011] E. H. Hwang, R. Sensarma, and S. Das Sarma, Coulomb drag in monolayer and bilayer graphene, Phys. Rev. B 84, 245441 (2011).
  * Titov  _et al._ [2013] M. Titov, R. V. Gorbachev, B. N. Narozhny, T. Tudorovskiy, M. Schütt, P. M. Ostrovsky, I. V. Gornyi, A. D. Mirlin, M. I. Katsnelson, K. S. Novoselov, A. K. Geim, and L. A. Ponomarenko, Giant magnetodrag in graphene at charge neutrality, Phys. Rev. Lett. 111, 166601 (2013).
  * Song and Levitov [2013] J. C. W. Song and L. S. Levitov, Hall drag and magnetodrag in graphene, Phys. Rev. Lett. 111, 126601 (2013).
  * Lee  _et al._ [2016] K. Lee, J. Xue, D. C. Dillen, K. Watanabe, T. Taniguchi, and E. Tutuc, Giant frictional drag in double bilayer graphene heterostructures, Phys. Rev. Lett. 117, 046803 (2016).
  * Liu  _et al._ [2017] X. Liu, L. Wang, K. C. Fong, Y. Gao, P. Maher, K. Watanabe, T. Taniguchi, J. Hone, C. Dean, and P. Kim, Frictional magneto-coulomb drag in graphene double-layer heterostructures, Phys. Rev. Lett. 119, 056802 (2017).
  * Tse  _et al._ [2019] W.-K. Tse, B. Y.-K. Hu, J. N. Hong, and A. H. MacDonald, Magneto-coulomb drag and hall drag in double-layer dirac systems, Phys. Rev. Lett. 122, 186602 (2019).
  * Zhu  _et al._ [2020a] L. Zhu, L. Li, R. Tao, X. Fan, X. Wan, and C. Zeng, Frictional drag effect between massless and massive fermions in single-layer/bilayer graphene heterostructures, Nano Lett. 20, 1396 (2020a).
  * Zhu  _et al._ [2023] L. Zhu, X. Liu, L. Li, X. Wan, R. Tao, Z. Xie, J. Feng, and C. Zeng, Signature of quantum interference effect in inter-layer coulomb drag in graphene-based electronic double-layer systems, Nature Communications 14, 1465 (2023).
  * Mitra  _et al._ [2020] R. Mitra, M. R. Sahu, K. Watanabe, T. Taniguchi, H. Shtrikman, A. K. Sood, and A. Das, Anomalous coulomb drag between inas nanowire and graphene heterostructures, Phys. Rev. Lett. 124, 116803 (2020).
  * Anderson  _et al._ [2021] L. Anderson, A. Cheng, T. Taniguchi, K. Watanabe, and P. Kim, Coulomb drag between a carbon nanotube and monolayer graphene, Phys. Rev. Lett. 127, 257701 (2021).
  * Suzuura and Ando [2002] H. Suzuura and T. Ando, Crossover from symplectic to orthogonal class in a two-dimensional honeycomb lattice, Physical review letters 89, 266603 (2002).
  * Wu  _et al._ [2007] X. Wu, X. Li, Z. Song, C. Berger, and W. A. de Heer, Weak antilocalization in epitaxial graphene: Evidence for chiral electrons, Physical review letters 98, 136801 (2007).
  * Lu and Shen [2015] H.-Z. Lu and S.-Q. Shen, Weak antilocalization and localization in disordered and interacting weyl semimetals, Physical Review B 92, 035203 (2015).
  * Flensberg  _et al._ [1995] K. Flensberg, B. Y.-K. Hu, A.-P. Jauho, and J. M. Kinaret, Linear-response theory of coulomb drag in coupled electron systems, Physical Review B 52, 14761 (1995).
  * Zhu  _et al._ [2020b] L. Zhu, L. Li, R. Tao, X. Fan, X. Wan, and C. Zeng, Frictional drag effect between massless and massive fermions in single-layer/bilayer graphene heterostructures, Nano letters 20, 1396 (2020b).
  * Gornyi  _et al._ [1999] I. Gornyi, A. Yashenkin, and D. Khveshchenko, Coulomb drag in double layers with correlated disorder, Physical review letters 83, 152 (1999).
  * Garate and Glazman [2012] I. Garate and L. Glazman, Weak localization and antilocalization in topological insulator thin films with coherent bulk-surface coupling, Physical Review B 86, 035422 (2012).
  * Kamenev and Oreg [1995] A. Kamenev and Y. Oreg, Coulomb drag in normal metals and superconductors: Diagrammatic approach, Phys. Rev. B 52, 7516 (1995).
  * Butler [2007] R. W. Butler,  _Saddlepoint approximations with applications_ , Vol. 22 (Cambridge University Press, 2007).
  * Shon and Ando [1998] N. H. Shon and T. Ando, Quantum transport in two-dimensional graphite system, Journal of the Physical Society of Japan 67, 2421 (1998).
  * Lepage [2021] G. P. Lepage, Adaptive multidimensional integration: Vegas enhanced, Journal of Computational Physics 439, 110386 (2021).
  * Ando  _et al._ [2002] T. Ando, Y. Zheng, and H. Suzuura, Dynamical conductivity and zero-mode anomaly in honeycomb lattices, Journal of the Physical Society of Japan 71, 1318 (2002).
  * Tse  _et al._ [2007] W.-K. Tse, B. Y.-K. Hu, and S. D. Sarma, Theory of coulomb drag in graphene, Physical Review B 76, 081401 (2007).