# Dynamical Casimir Effect and Work in Fermionic Fields

Gianluca Francica<EMAIL_ADDRESS>
###### Abstract

We consider a quantum massless fermionic field in (1+1) dimensions in the case
of moving boundaries. We work in the canonical approach in order to find a
Hamiltonian describing the dynamics of the field. Thus, we study the
statistics of work and particles produced from the vacuum by a linear driving
of a boundary.

##  I Introduction

The dynamical Casimir effect is the phenomenon for which moving boundaries of
a quantum field could induce particle creation from the vacuum. It was
originally foreseen by Moore, which has used a simplified one-dimensional
model moore70 , and afterwards studied by DeWittdewitt75 , Fulling and Davies
fulling76 ; davies77 , and many others. Anyway, differently from the bosonic
fields for which there have been many studies (see, e.g., Ref. dodonov20  for
a recent review), the dynamical Casimir effect in fermionic fields remains
less investigated mazzitelli87 .

An interesting question is how the dynamical Casimir effect is related to the
thermodynamics of the process. Recently, many studies has consolidated the
emerging field of quantum thermodynamics kosloff13 ; Vinjanampathy16 ;
bookthermo18 . In this perspective, the statistics of the work performed and
the change of photon number has been studied in the case of a cavity with an
oscillating boundary fei19 .

In this paper, we aim to study the dynamical Casimir effect in the less
investigated case of a massless fermionic field in $(1+1)$ dimensions, by
adopting a canonical approach previously employed for the bosonic case
schutzhold98 . In particular, we study the statistics of work and particles
produced from the vacuum by a linear driving of a boundary.

##  II  Model

We consider a quantum fermionic field in $(1+1)$ dimensions between moving
boundaries. In the massless case the Dirac Lagrangian density being

$\mathcal{L}=\frac{iv}{2}\left(\bar{\psi}\gamma^{\alpha}\partial_{\alpha}\psi-(\partial_{\alpha}\bar{\psi})\gamma^{\alpha}\psi\right)$
(1)

where $\partial_{\alpha}=\partial/\partial x^{\alpha}$ with $x^{0}=vt$ and
$x^{1}=x$. The adjoint field $\bar{\psi}$ is defined by
$\bar{\psi}=\psi^{\dagger}\gamma^{0}$, and the Dirac matrices satisfy the
anti-commutation relations $\\{\gamma^{\mu},\gamma^{\nu}\\}=2g^{\mu\nu}$ with
$g^{\mu\nu}=\text{diag}(+,-)$. The Dirac matrices can be taken to be two-
dimensional, we consider $\gamma^{0}=\sigma_{x}$ and $\gamma^{1}=-i\sigma_{y}$
where $\sigma_{\alpha}$, with $\alpha=x,y,z$, are the Pauli matrices. We
assume that the boundaries are placed at the points $x=0$ and $x=l(t)$ and we
consider the bag-like boundary conditions mazzitelli87

$e^{i\theta\gamma_{5}}\psi=in_{\mu}\gamma^{\mu}\psi$ (2)

where $\gamma_{5}=\gamma^{0}\gamma^{1}=\sigma_{z}$ and $n^{\mu}$ is the unit,
exterior normal to the boundary. In particular, we will put
$\theta=\theta_{0}$ constant for $x=0$ and $x=l(t)$.

The energy is $E=\int T_{00}dx$ where $T_{00}$ is the density

$T_{00}=-iv\psi^{\dagger}\gamma_{5}\partial_{x}\psi$ (3)

The instantaneous eigenfunctions $\psi_{n}$ of the one-particle energy
$-iv\gamma_{5}\partial_{x}$ are such that
$-iv\gamma_{5}\partial_{x}\psi_{n}=\omega_{n}\psi_{n}$. We find

$\psi_{n}=\frac{1}{\sqrt{2l}}\left(\begin{array}[]{c}e^{i\frac{\omega_{n}}{v}x}\\\
ie^{-i\left(\frac{\omega_{n}}{v}x-\theta_{0}\right)}\\\ \end{array}\right)$
(4)

which are normalized such that
$\int\psi_{n}^{\dagger}\psi_{m}dx=\delta_{m,n}$, with one-particle energies
$\omega_{n}=(n+1/2)\pi v/l$. In order to quantize the Dirac field we expand it
in terms of the eigenfunctions $\psi_{n}$ as

$\psi=\sum_{n\geq
0}a_{n}\psi_{n}+b_{n}^{\dagger}\psi_{-1-n}=\sum_{n}q_{n}\psi_{n}$ (5)

such that automatically satisfies the boundary conditions. The Lagrangian
$L=\int\mathcal{L}dx$ reads

$L=\sum_{n}\left(\frac{i}{2}(q^{\dagger}_{n}\dot{q}_{n}-\dot{q}^{\dagger}_{n}q_{n})-\omega_{n}q^{\dagger}_{n}q_{n}\right)+i\sum_{nm}q^{\dagger}_{m}M_{mn}q_{n}$
(6)

where we have defined the matrix elements

$M_{mn}=\frac{1}{2}\int(\psi_{m}^{\dagger}\dot{\psi}_{n}-\dot{\psi}_{m}^{\dagger}\psi_{n})dx$
(7)

By considering $q_{n}$ as independent variables, their canonical conjugate
momenta will be $p_{n}=\partial L/\partial\dot{q}_{n}=iq_{n}^{\dagger}/2$.
Thus the Hamiltonian $H$ describing the dynamics reads

$H=E-i\sum_{nm}q_{m}^{\dagger}M_{mn}q_{n}$ (8)

where the energy is $E=\sum\omega_{n}q_{n}^{\dagger}q_{n}$ and $M$ needs to be
anti-hermitian in order to have $H$ hermitian. We impose the anti-commutation
relations
$\\{a_{n},a_{m}^{\dagger}\\}=\\{b_{n},b_{m}^{\dagger}\\}=\delta_{m,n}$ and
$\\{a_{n},a_{m}\\}=\\{b_{n},b_{m}\\}=\\{a_{n},b_{m}\\}=\\{a_{n},b_{m}^{\dagger}\\}=0$.
By subtracting the zero-point energy, we have
$E=\sum\left|\omega_{n}\right|(a_{n}^{\dagger}a_{n}+b_{n}^{\dagger}b_{n})$ and
the ground-state is the vacuum state $|0\rangle$ defined by
$a_{n}|0\rangle=b_{n}|0\rangle=0$.

The elements of the matrix $M$ are

$M_{mn}=\frac{(-1)^{m-n}}{2}\frac{m+n+1}{m-n}\frac{\dot{l}}{l}$ (9)

for $m\neq n$, and $M_{nn}=0$, such that $M^{T}=-M$. We observe that, since
$E|0\rangle=0$ at any time, only due to the presence of the matrix $M$ it is
possible to acquire a dynamical Casimir effect.

##  III  Dynamics and Statistics

We focus on a linear changing of the boundary $l(t)=l_{0}+\alpha t$, such that
$[H(t),H(t^{\prime})]=0$ and the time-evolution operator is
$U=e^{-i\tilde{H}\Delta l}$, where $\Delta l=\ln(l(t)/l(0))$ and $\tilde{H}$
is time-independent and such that $H=\tilde{H}\alpha/l(t)$. In order to study
the dynamical Casimir effect, we consider the initial state $|0\rangle$. In
this case the distribution of work performed is equal to $p(w)=\left|\langle
n|U|0\rangle\right|^{2}\delta(w-\epsilon_{n}(t))$ where $|n\rangle$ is the
eigenstate of $E(t)$ with eigenvalue $\epsilon_{n}(t)$. The characteristic
function $\chi(u)=\int e^{iuw}p(w)dw$ can be expressed as

$\chi(u)=\langle 0|e^{iu\tilde{E}}|0\rangle$ (10)

where $\tilde{E}=U^{\dagger}E(t)U$ and the work moments are given by $\langle
w^{n}\rangle=(-i)^{n}\chi^{(n)}(0)$. We note that the work is completely
irreversible since the vacuum energy remains equal to zero. Similarly, the
characteristic function of the particle created is $\chi_{N}(u)=\langle
0|e^{iu\tilde{N}}|0\rangle$ with $\tilde{N}=U^{\dagger}NU$ where $N$ is the
number operator $N=\sum a^{\dagger}_{n}a_{n}+b^{\dagger}_{n}b_{n}$.

For small $\Delta l$, we get

$U|0\rangle\approx|0\rangle+\frac{\Delta l}{2}\sum_{m\geq 0,n\geq
0}(-1)^{m+n}\frac{m-n}{m+n+1}a_{m}^{\dagger}b_{n}^{\dagger}|0\rangle$ (11)

such that couples of fermions $(a,b)$ are created from the vacuum. In
particular, it is more likely to create fermions $a$ with high energy and $b$
with low energy, and viceversa to create fermions $b$ with high energy and $a$
with low energy.

Lets show how to calculate the characteristic functions. We start by noting
that the operator $\tilde{H}$ can be expressed as

$\tilde{H}=\sum_{mn}\left(c^{\dagger}_{m}A_{mn}c_{n}+\frac{1}{2}(c^{\dagger}_{m}B_{mn}c_{n}^{\dagger}+h.c.)\right)$
(12)

where $c_{n}=a_{n}$ and $c_{-n-1}=b_{n}$ for $n\geq 0$, and the matrices $A$
and $B$ are such that $A^{\dagger}=A$ and $B^{T}=-B$. In particular
$A_{nn}=\left|n+1/2\right|\pi v/\alpha$ and for $n\neq m$,
$A_{mn}=-iM_{mn}l/\dot{l}$ if $m,n\geq 0$ or $m,n<0$ and $A_{mn}=0$ otherwise,
$B_{mn}=-B_{nm}=-iM_{mn}l/\dot{l}$ for $m\geq 0$ and $n<0$ and $B_{mn}=0$
otherwise. In order to solve the dynamics, we note that the operator
$\tilde{H}$ can be diagonalized by a Bogoliubov transformation lieb61
$c_{n}=\sum_{k}u_{nk}\eta_{k}+v_{nk}\eta^{\dagger}_{k}$ such that
$\tilde{H}=\sum\Lambda_{k}\eta_{k}^{\dagger}\eta_{k}$. The matrices $u$ and
$v$ are defined by $Au+Bv^{*}=\Lambda u$ and $Av+Bv^{*}=\Lambda v$, where
$\Lambda$ is the diagonal matrix with diagonal elements $\Lambda_{k}\geq 0$.
Thus, since $U^{\dagger}\eta_{k}U=e^{-i\Lambda_{k}\Delta l}\eta_{k}$, the time
evolved $\tilde{c}_{n}=U^{\dagger}c_{n}U$ reads

$\tilde{c}_{n}=\sum_{m}\tilde{u}_{nm}c_{m}+\tilde{v}_{nm}c_{m}^{\dagger}$ (13)

with $\tilde{u}=uDu^{\dagger}+vD^{*}v^{\dagger}$ and
$\tilde{v}=uDv^{T}+vD^{*}u^{T}$ where $D$ is the diagonal matrix with elements
$D_{nn}=e^{-i\Lambda_{n}\Delta l}$. Of course
$\tilde{E}=\sum\left|\omega_{n}\right|\tilde{c}^{\dagger}_{n}\tilde{c}_{n}$ is
diagonal in terms of the fermions $\tilde{c}$. The vacuum state of the time-
evolved fermion operators is $|\tilde{0}\rangle$ defined by
$\tilde{c}_{n}|\tilde{0}\rangle=0$. We have the relation between the two
vacuum states chung01

$|0\rangle=Ke^{\frac{1}{2}\sum
G_{mn}\tilde{c}^{\dagger}_{m}\tilde{c}^{\dagger}_{n}}|\tilde{0}\rangle$ (14)

with $G$ skew-symmetric defined by $\tilde{u}G+\tilde{v}=0$. Thus

$e^{iu\tilde{E}}|0\rangle=Ke^{\frac{1}{2}\sum\tilde{G}_{mn}\tilde{c}^{\dagger}_{m}\tilde{c}^{\dagger}_{n}}|\tilde{0}\rangle$
(15)

with $\tilde{G}=\tilde{D}G\tilde{D}$ where $\tilde{D}$ is the diagonal matrix
with elements $\tilde{D}_{nn}=e^{iu\left|\omega_{n}\right|}$. In order to
calculate the inner product of Eq. (10), we use the fermionic coherent states
defined by $\tilde{c}_{n}|\xi\rangle=\xi_{n}|\xi\rangle$ where $\xi_{n}$ are
Grassmann variables negele98 . We obtain the Gaussian integral

$\langle 0|e^{iu\tilde{E}}|0\rangle=\left|K\right|^{2}\int d\xi
d\xi^{*}e^{-\frac{1}{2}(\xi,\xi^{*})\Gamma(\xi,\xi^{*})^{T}}$ (16)

where

$\Gamma=\left(\begin{array}[]{cc}G^{*}&-1\\\ 1&-\tilde{G}\\\
\end{array}\right)$ (17)

which can be easily calculated, such that

$\chi(u)=\sqrt{\frac{\det\Gamma(u)}{\det\Gamma(0)}}$ (18)

with $\det\Gamma(u)=\det(1+G^{\dagger}\tilde{G})$. On the other hand, the
characteristic function of the number of particles is similarly obtained by
considering $\tilde{D}_{nn}=e^{iu}$.

We calculate numerically the moments $\langle w^{n}\rangle$ and $\langle
N^{n}\rangle$ by employing the respective characteristic functions and we find
that the moments diverge as the number of modes $\psi_{n}$ goes to infinity.
By truncate the space to the $2L$ modes
$(\psi_{-L},\psi_{-L+1},\cdots,\psi_{L-1})$, we find two different behaviors
depending on the value of the speed $\alpha/v$ (see Fig. 1).

Figure 1:  The plots of the moments in function of the cut-off $L$. The energy
scale is in unit of $\pi v/l(t)$. We put $\Delta l=\ln 2$, $\alpha/v=0.1$ (top
panels) and $\alpha/v=2$ (bottom panels).

For small $\alpha/v$ we have $\langle w^{n}\rangle\sim L^{n}$ and $\langle
N^{n}\rangle\sim\ln L$ as $L\to\infty$. On the other hand, for large
$\alpha/v$ we get $\langle w^{n}\rangle\sim L^{2n}$ and $\langle
N^{n}\rangle\sim L^{n}$. We find numerical values of the parameters
$\beta_{\alpha},\gamma_{\alpha}$ that make the functions
$f_{w}(L)=\beta_{0}+\beta_{1}L+\beta_{2}L^{2}$ and
$f_{N}(L)=\gamma_{0}+\gamma_{1}L+\gamma_{l}\ln L$ give a best fit to $\langle
w\rangle$ and $\langle N\rangle$ as a function of $L$ (see Fig. 2).

Figure 2:  The plots of the coefficients $\beta_{2}$ and $\gamma_{1}$ in
function of $\alpha/v$. We put $\Delta l=\ln 2$ and $\beta_{2}$ is in unit of
$\pi v/l(t)$

It results that $\gamma_{1}$ and $\beta_{2}$ are practically zero for small
$\alpha/v$ and suddenly increase as $\alpha/v$ reaches a value
$\alpha/v\approx 1$. Thus, there are two dynamical regimes: an adiabatic
regime, for slow driving, where the work and the number of particles generated
increase linearly and logarithmically with $L$, and a diabatic one, for fast
driving, where the work and the number of particles generated increase
quadratically and linearly with $L$. Of course, if $v$ is the speed of light
we need a superluminal speed $\alpha$ to observe the diabatic regime. We note
that such behavior does not change for an expansion or a compression of the
box. In particular, we find that the number moments $\langle N^{n}\rangle$ are
invariant with respect to the transformation $(\alpha,\Delta
l)\to-(\alpha,\Delta l)$, differently from the work moments which can change
noticeably.

##  IV  Conclusion

In summary, we have investigated the dynamical Casimir effect in a two-
dimensional massless fermionic field by adopting a canonical approach. We
focus on the case of a linear driving of a boundary, showing how to solve the
dynamics and study the statistics of work and number of particles produced
from the vacuum. By performing a numerical analysis by truncating the number
of modes, we find that there are two different dynamical regimes. In both
regimes, work and number of fermions generated are unbounded as the number of
modes goes to infinity. In the end, we note that, for a finite cut-off $L$,
for a slow driving, e.g. $\alpha/v\ll 1$, the number of particles generated
can be very small, $\langle N\rangle\approx 0.03(\alpha/v)^{2}\ln L$ for
$\Delta l=\ln 2$, but the work produced can be relatively big, $\langle
w\rangle\approx 0.02(\alpha/v)^{2}L\pi v/l(t)$, which can be observed as heat
given to the environment in a thermalization process at zero temperature
plastina14 .

## References

  * (1) G.T. Moore, J. Math. Phys. 11, 2679 (1970)
  * (2) B. S. DeWitt, Phys. Rep. 19, 295-357 (1975)
  * (3) S. A. Fulling and P. C. W. Davies, Proc. R. Soc. Lond. A 348, 393-414 (1976)
  * (4) P. C. W. Davies and S. A. Fulling, Proc. R. Soc. Lond. A 356, 237-257 (1977)
  * (5) V. Dodonov, Physics, 2(1), 67-104 (2020)
  * (6) F. D. Mazzitelli, J. P. Paz and M. A. Castagnino, Phys. Lett. B 189, 132-136 (1987)
  * (7) R. Kosloff, Entropy 15, 2100 (2013)
  * (8) S. Vinjanampathy and J. Anders, Contemp. Phys. 57, 545 (2016)
  * (9) Thermodynamics in the Quantum Regime - Fundamental Aspects and New Directions, (eds.) F. Binder, L. A. Correa, C.Gogolin, J. Anders, G. Adesso, Springer (2018)
  * (10) Z. Fei, J.-N. Zhang, R. Pan, T. Qiu and H. T. Quan, Phys. Rev. A 99, 052508 (2019)
  * (11) R. Schützhold, G. Plunien and G. Soff, Phys. Rev. A 57, 2311 (1998)
  * (12) E. Lieb, T. Schultz, and D. Mattis, Annals of Physics 16, 407 (1961)
  * (13) M.-C. Chung and I. Peschel, Phys. Rev. B 64, 064412 (2001)
  * (14) J. W. Negele and H. Orland, Quantum Many-particle Systems, Taylor & Francis Inc (1998)
  * (15) F. Plastina, A. Alecce, T. J. G. Apollaro, G. Falcone, G. Francica, F. Galve, N. Lo Gullo and R. Zambrini, Phys. Rev. Lett. 113, 260601 (2014)


RReferences

# Attentive Multimodal Fusion for
Optical and Scene Flow

Youjie Zhou1, Guofeng Mei2, Yiming Wang2, Fabio Poiesi2, Yi Wan1 This work was
supported by the China government project (2019JZZY010112), (2020JMRH0202),
(YC-KYXM-07-2021) and by the PNRR project FAIR - Future AI Research
(PE00000013), under the NRRP MUR program funded by the
NextGenerationEU.1Youjie Zhou and Yi Wan are with the School of Mechanical
Engineering, Shandong University, China, and the Key Laboratory of High
Efficiency and Clean Mechanical Manufacture of Ministry of Education, Shandong
University, China. 202020511<EMAIL_ADDRESS>Yi Wan is the
corresponding author.2Guofeng Mei, Yiming Wang, and Fabio Poiesi are with
Fondazione Bruno Kessler, Italy<EMAIL_ADDRESS>
###### Abstract

This paper presents an investigation into the estimation of optical and scene
flow using RGBD information in scenarios where the RGB modality is affected by
noise or captured in dark environments. Existing methods typically rely solely
on RGB images or fuse the modalities at later stages, which can result in
lower accuracy when the RGB information is unreliable. To address this issue,
we propose a novel deep neural network approach named FusionRAFT, which
enables early-stage information fusion between sensor modalities (RGB and
depth). Our approach incorporates self- and cross-attention layers at
different network levels to construct informative features that leverage the
strengths of both modalities. Through comparative experiments, we demonstrate
that our approach outperforms recent methods in terms of performance on the
synthetic dataset Flyingthings3D, as well as the generalization on the real-
world dataset KITTI. We illustrate that our approach exhibits improved
robustness in the presence of noise and low-lighting conditions that affect
the RGB images. We release the code, models and dataset at
https://github.com/jiesico/FusionRAFT.

###### Index Terms:

Optical and scene flow, multimodal fusion, self- and cross-attention.

##  I Introduction

Optical flow algorithms are essential for determining the motion of objects or
regions within images between consecutive video frames. They generate a 2D
vector field that describes the apparent movement of pixels over time. In
contrast, scene flow focuses on estimating the pixel-level 3D motion in stereo
or RGBD video frames [1]. These algorithms find wide applications in robotics
[2, 3] and surveillance [4, 5]. Computing optical flow becomes particularly
challenging in environments with non-informative textures or when scenes are
captured under low-lighting conditions. To address these difficulties, deep
learning methods have emerged as effective solutions for optical flow
estimation, formulating the problem as an energy minimization task [6, 7, 8,
9]. Deep learning-based optical flow approaches have demonstrated significant
improvements over traditional methods [10, 11, 12].

Several approaches utilize the computation of a correlation volume in the
visible spectrum (RGB) to estimate the optical flow between two frames [6, 10,
11]. The correlation volume captures inter-frame similarity by taking the dot
product of the corresponding convolutional feature vectors and can be
generated through an end-to-end deep network. This deep network can be
designed to minimize an underlying energy function. However, relying solely on
RGB information can be limited in scenes affected by motion blurs, non-
informative textures, or low illumination conditions. To address this
limitation, some approaches have incorporated multimodal information. For
example, depth or point cloud data can provide an alternative representation
of the underlying scene structure. This multimodal information can be
integrated through late fusion, where feature vectors are combined without
intermediate information exchange [13, 1], or through exchanging information
between branches while sacrificing the independence of the single-modality
representation [12].

In this paper, we present a novel multimodal fusion approach, named
FusionRAFT, for optical and scene flow estimation, specifically designed to
handle data captured in noisy or low-lighting conditions, for example those
that can be encountered in search and rescue applications [14]. Our approach
introduces three key components to address these challenges. Firstly, we
propose a feature-level fusion technique that seamlessly blends RGB and depth
information using a shared loss function. Secondly, we introduce a self-
attention mechanism that enhances the expressiveness of feature vectors by
dynamically balancing the importance of features within each individual
modality. Lastly, we incorporate an optimized cross-attention module that
facilitates information exchange and balance between RGB and depth modalities.
We integrate these new modules within RAFT [10] and RAFT-3D [1], using an
application-oriented data augmentation strategy to learn robust feature
representations that make optical and scene flow estimation effective in
complex environments. We conduct extensive evaluations on standard optical and
scene flow benchmarks, as well as on two new settings that we introduce to
assess robustness against photometric noise and challenging illumination
conditions. Our method achieves state-of-the-art performance on the synthetic
dataset FlyingThings3D [15] and demonstrates superior generalization
capabilities on the real-world dataset KITTI [16] without fine-tuning.

##  II Related work

We provide a comprehensive analysis of the recent progress in optical flow
estimation using deep learning, followed by an in-depth investigation into the
integration of multimodal fusion techniques for improving flow estimation
performance.

Optical flow. FlowNet [6] pioneered the use of deep neural networks to
estimate optical flow as a supervised learning task. FlowNet learns features
across scales and abstraction levels to determine pixel correspondences.
FlowNet inspired FlowNet2.0 [7], PWC-Net [17], MaskFlowNet [18] and
LiteFlowNet3 [19]. FlowNet2.0 presents a warping operation and a method for
stacking multiple networks through this operation [7]. PWC-Net utilizes
pyramidal processing, warping, and a cost volume approach to improve both the
size and accuracy of optical flow models [17]. MaskFlowNet incorporates an
asymmetric occlusion-aware feature matching module, which learns to filter out
occluded regions through feature warping without the need for explicit
supervision [18]. LiteFlowNet3 tackles the challenge of estimating optical
flow in the presence of partially occluded or homogeneous regions by using an
adaptive affine transformation and a confidence map that identifies unreliable
flow [19]. The confidence map is used to guide the generation of
transformation parameters.

RAFT [10] is a per-pixel feature extraction approach that constructs multi-
scale 4D correlation volumes for each pixel pair, and updates the flow field
iteratively through a recurrent unit. Like FlowNet, RAFT has inspired GMA [20]
and CRAFT [11]. GMA addresses occlusions by modeling image self-similarities
by using a global motion aggregation module, a transformer-based approach for
finding long-range dependencies between pixels in the first image, and a
global aggregation of the corresponding motion features. CRAFT aims to
estimate the large motion displacements through a semantic smoothing
transformer layer that integrates the features of one image and a cross-
attention layer that replaces the original dot-product operator for
correlation used in RAFT. Unlike these approaches, we tackle the problem of
estimating optical flow in situations of unreliable RGB information, such as
noises and scarce illuminations, by appropriately fusing multiple modalities
through self and cross attention within feature extraction layers.

Multimodal fusion. Multimodal fusion can be performed at various stages:
early-, mid-, and late-fusion. In early-fusion, multiple channels are created
within the network to process multiple modalities together [21]. Mid-fusion
maintains different branches for each modality and then merges the
corresponding features at the end of the network [22, 23]. In late-fusion, the
network is trained on each modality separately and then fuses the results from
the independent branches [24]. RAFT [10], GMA [20], and CRAFT [11] estimate
the relationships between two consecutive frames using RGB images. Inspired by
multimodal fusion, some of these works have been improved to compute both
scene and optical flow by utilizing additional modalities such as depth, and
point clouds.

####  II-1 RGB + Point Cloud Data.

DeepLiDARFlow [13] exhibits improved performance in challenging conditions,
such as reflective surfaces, poor illumination, and shadows. Images and point
clouds are processed by using multi-scale feature pyramid networks. Late-
fusion based on differentiable confidence volumes produces the fused features.
CamLiFlow [12] improves upon DeepLiDARFlow by fusing dense image features and
sparse point features more effectively. Instead of late-fusion, CamLiFlow
adopts a multi-stage, bidirectional fusion strategy, in which the two
modalities are learned in separate branches using modality-specific
architectures. CamLiRAFT [25] further improves the performance based on the
RAFT [10] framework, leading to superior results compared to CamLiFlow [12].
Our method differs from previous methods in that it ensures the independence
of each modality through the use of two separate branches and balances the
information between the modalities through multi-stage information exchange.

####  II-2 RGB + depth

RAFT-3D [1] extends RAFT to estimate both optical and scene flow from RGBD
data. RGB images serve as inputs to the feature network, where a 4D
correlation volume is constructed and a soft grouping of pixels into rigid
objects is formed with the aid of depth information. Unlike RAFT [10], RAFT-3D
employs late-fusion with the depth information and the RGB features in the
prediction module, improving the stability of flow prediction. However, RAFT’s
feature extraction method may not sufficiently capture the rich 3D structural
information. To address this, our approach employs early-fusion, in which
features are extracted from both RGB and depth information, enabling stable
estimation even in cases where RGB information is unreliable.

##  III Our approach

We present a Multimodal Feature Fusion (MFF) Encoder that performs early
fusion of RGB and depth modalities to improve the estimation of both optical
and scene flow under noisy or poor lighting conditions. Our encoder is
flexible and can be integrated into flow estimation frameworks by replacing
their original feature encoder. To achieve this, we employ self-attention,
cross-attention, and Multimodal Transfer Module (MMTM) [26]. We extract low-
level features from each modality and improve their expressivity using self-
attention. Cross-attention enables the network to attend to the most
informative modality. MMTM is used to further fuse the attended features that
are computed from the two modalities. Fig. 1(a) shows the architecture of our
encoder.

Figure 1: Block diagram of FusionRAFT. (a) Our encoder architecture: RGB and
depth frames are taken as inputs. The encoder network is a two-branch network
with a transformer (self-attention plus cross-attention) and a Multimodal
Transfer Module. (b) Optical flow and (c) scene flow architectures. Two
consecutive RGBD frames are taken as inputs by the MFF for the feature
encoder, and the first RGBD frame is taken as input by the MFF for the context
encoder.

###  III-A Multimodal Feature Fusion Encoder

The Multimodal Feature Fusion Encoder takes a pair of consecutive RGBD frames
$(P^{t},~{}P^{t+1})$ at time $t$ as input. Each frame
$P^{t}=\\{I^{t},Z^{t}\\}$ is composed of a RGB image $I^{t}$ and a depth image
$Z^{t}$.

We first obtain low-level features $\bm{{F}}^{t}_{r}\in\mathbb{R}^{W\times
H\times D}$ and $\bm{{F}}^{t}_{d}\in\mathbb{R}^{W\times H\times D}$ from each
modality with convolutional blocks, where we use the subscript ${r}$ to
represent the RGB branch and ${d}$ for the depth branch (Fig. 1(a)). ${D}$ is
the feature dimension and ${W\times H}$ is the resolution of the features.

Self-attention. The local features $\bm{{F}}^{t}_{r}$ and $\bm{{F}}^{t}_{d}$
are obtained with convolutions that have limited receptive fields, therefore
we model global structures by establishing long-range dependencies through a
self-attention module (${S_{\theta}}(\cdot)$ in Fig. 1(a)). To mitigate the
high computational cost of self-attention, we downsample
$\bm{{F}}^{t}_{r}\in\mathbb{R}^{N\times D}$ and
$\bm{{F}}^{t}_{d}\in\mathbb{R}^{N\times D}$ to obtain $\bm{\bar{F}}^{t}_{r}$
and $\bm{\bar{F}}^{t}_{d}$ via $3\times 3$ and $5\times 5$ max-pooling layers.
With these downsampled features, we can use a multi-attention layer with four
parallel attention heads to process $\bm{F}^{t}_{r}$ and
$\bm{\bar{F}}^{t}_{r}$ (or $\bm{F}^{t}_{d}$ and $\bm{\bar{F}}^{t}_{d}$) in
parallel and get $\bm{\hat{F}}^{t}_{k}$:

$\displaystyle\bm{\hat{F}}^{t}_{k}\leftarrow{S_{\theta}}(\bm{{F}}^{t}_{k},\bm{\bar{F}}^{t}_{k})$
(1)
$\displaystyle=\bm{{F}}^{t}_{k}{+}\mbox{MLP}\left(\sigma\left({\bm{W}^{t}_{Q_{s}}\bm{F}^{t}_{k}\left(\bm{W}^{t}_{K_{s}}\bm{\bar{F}}^{t}_{k}\right)^{\top}}\big{/}{\sqrt{D}}\right)\bm{W}^{t}_{V_{s}}\bm{\bar{F}}^{t}_{k}\right),$

where $k\in\\{r,d\\}$ and $\sigma$ is the softmax function. $D$ is the feature
dimension. $\bm{W}^{t}_{Q_{s}}\in\mathbb{R}^{N\times
D},\bm{W}^{t}_{K_{s}}\in\mathbb{R}^{J\times D}$ and
$\bm{W}^{t}_{V_{s}}\in\mathbb{R}^{J\times D}$ are the query, key and value
matrices, where $N=W\times H$, $J=(W\times H)/(3\times 5)$.
$\mbox{MLP}(\cdot)$ denotes a three-layer fully connected network with
instance normalization[27] and ReLU [28] activation after the first two
layers.

Cross-attention. We promote information exchange between the two modalities
via cross-attention, which we implement through the network
${C_{\theta}}(\cdot)$ (Fig. 1(a)). Attention signals from one modality (e.g.
RGB) emphasize the features of another modality (e.g. depth), and vice versa.
Given the self-attended features $\bm{\hat{F}}^{t}_{r}\in\mathbb{R}^{N\times
D}$ and $\bm{\hat{F}}^{t}_{d}\in\mathbb{R}^{N\times D}$, we also adopt two
downsampled networks max-pooling ($3\times 3$), and max-pooling ($5\times 5$)
to generate the downsampled image feature map $\bm{\bar{\hat{F}}}^{t}_{r}$ (or
$\bm{\bar{\hat{F}}}^{t}_{d}$). We denote the transformed features as
$\bm{\hat{\hat{F}}}^{t}_{r}\in\mathbb{R}^{N\times D}$ and
$\bm{\hat{\hat{F}}}^{t}_{d}\in\mathbb{R}^{N\times D}$ attained by cross-
attention via

$\displaystyle\bm{\hat{\hat{F}}}^{t}_{r}\leftarrow{C_{\theta}}(\bm{\hat{F}}^{t}_{r},\bm{\bar{\hat{F}}}^{t}_{d})$
(2)
$\displaystyle=\bm{\hat{F}}^{t}_{r}{+}\mbox{MLP}\left(\sigma\left(\bm{W}^{t}_{Q_{c}}\bm{\hat{F}}^{t}_{r}\left(\bm{W}^{t}_{K_{c}}\bm{\bar{\hat{F}}}^{t}_{d}\right)^{\top}\big{/}{\sqrt{D}}\right)\bm{W}^{t}_{V_{c}}\bm{\bar{\hat{F}}}^{t}_{d}\right),$

where $W^{t}_{Q_{c}}\in\mathbb{R}^{N\times
D},W^{t}_{K_{c}}\in\mathbb{R}^{J\times D}$ and
$W^{t}_{V_{c}}\in\mathbb{R}^{J\times D}$ are the query, key and value
matrices. This cross-attention block is also applied in the reverse direction
so that information flows in both directions, i.e., RGB$\rightarrow$depth and
depth$\rightarrow$RGB.

Multimodal Transfer Module. Because our architecture operates with multimodal
information, we further promote information exchange between modalities after
attention. Let ${M_{\theta}}(\cdot)$ be the Multimodal Transfer Module [26] we
use to improve the balance between RGB and depth information (Fig. 1(a)). Let
$\bm{\hat{\hat{F}}}^{t}_{r}\in\mathbb{R}^{N\times D_{M}}$ and
$\bm{\hat{\hat{F}}}^{t}_{d}\in\mathbb{R}^{N\times D_{M}}$ be the input
multimodal features to MMTM, and $\bm{\tilde{F}}_{r}^{t}\in\mathbb{R}^{N\times
D_{M}}$ and $\bm{\tilde{F}}_{d}^{t}\in\mathbb{R}^{N\times D_{M}}$ be the
respective outputs. MMTM first squeezes the feature vectors into
${S_{\bm{\tilde{F}}^{t}_{r}}}$ and ${S_{\bm{\tilde{F}}^{t}_{d}}}$ via a global
average pooling. MMTM then maps these tensors to a joint representation _Z_
through concatenation and a fully-connected layer. Based on _Z_ , MMTM finally
balances RGB and depth information by gating the channel-wise features:

$\displaystyle{S_{\bm{\tilde{F}}^{t}_{k}}}$
$\displaystyle=\frac{1}{{\Pi_{t=1}^{K}{N_{k}}}}\sum\limits_{{n_{1,{\cdots},}}{n_{K}}}{\bm{\hat{\hat{F}}}^{t}_{k}({n_{1}},\cdots,{n_{K}})},$
(3) $\displaystyle{{Z}}$
$\displaystyle=\bm{\mbox{W}}[{{S_{\bm{\tilde{F}}^{t}_{r}}}},{{S_{\bm{\tilde{F}}^{t}_{d}}}}]+b$
$\displaystyle\tilde{\bm{{F}}}^{t}_{k}$
$\displaystyle=2\sigma({\bm{\mbox{W}}_{\bm{\tilde{F}}^{t}_{k}}}{Z})\odot\bm{\hat{\hat{F}}}^{t}_{k},$

where $\left[\cdot,\cdot\right]$ is the concatenation operator and
$k\in\\{r,d\\}$. $N_{k}$ represents the spatial dimensions of
$\bm{\hat{\hat{F}}}^{t}_{k}$ and $D_{M}$ represents the number of channels of
the features. $\bm{\mbox{W}}\in\mathbb{R}^{D_{Z}\times 2D_{M}}$,
${\bm{\mbox{W}}_{\bm{\tilde{F}}^{i}_{k}}}\in\mathbb{R}^{D_{M}\times D_{Z}}$
are the weights, and $b\in\mathbb{R}^{D_{Z}}$ are the biases of the fully
connected layers.

###  III-B Optical and scene flow estimation

The inputs of optical and scene flow estimation are the feature vectors
$[\tilde{\bm{{F}}}^{t}_{r},\tilde{\bm{{F}}}^{t}_{d}]$ and
$[\tilde{\bm{{F}}}^{t+1}_{r},\tilde{\bm{{F}}}^{t+1}_{d}]$. By calculating the
dot product of feature vectors between the inputs, a 4D correlation volume C
is generated:

$\displaystyle
fnet({P}^{t})=[\tilde{\bm{{F}}}^{t}_{r},\tilde{\bm{{F}}}^{t}_{d}]=[{M_{\theta}}({C_{\theta}}({S_{\theta}}({I^{t}}),{S_{\theta}}({Z^{t}})))],$
(4) $\displaystyle\textbf{C}({P^{t}},{P^{t+1}})=\langle
fnet({P^{t}}),fnet({P^{t+1}})\rangle.$

where $\left<\cdot,\cdot\right>$ is the dot product operator. A four-layer
pyramid
$\\{{\textbf{C}_{1}},{\textbf{C}_{2}},{\textbf{C}_{3}},{\textbf{C}_{4}}\\}$ is
generated by reducing the last two dimensions of the correlation volume
through pooling with kernels of size 1, 2, 4, and 8.

We compute 4D correlation volumes to estimate optical and scene flow [10, 1].
Through
$\\{{\textbf{C}_{1}},{\textbf{C}_{2}},{\textbf{C}_{3}},{\textbf{C}_{4}}\\}$,
we iteratively estimate the dense displacement field
$\\{\textbf{f}^{1}_{est},\textbf{f}^{2}_{est},...,\textbf{f}^{M}_{est}\\}$
with M iterations to update the optical and scene flow. We train our network
by computing the loss between the estimated flow and the ground-truth flow
${\textbf{f}_{gt}}$ as

$\small\mathcal{L}=\sum\limits_{k=1}^{M}{{\gamma^{M-k}}{{\left\|{\textbf{f}^{k}_{est}-{\textbf{f}_{gt}}}\right\|}_{1}}},\
$ (5)

where as the iteration $k$ increases, the weight per loss term exponentially
increases with a base $\gamma$. Fig. 1(b,c) show how our Multimodal Feature
Fusion Encoder is integrated in RAFT and RAFT-3D to estimate the optical flow
and the scene flow, respectively. Our module can be integrated seamlessly and
does not require any modification to RAFT and RAFT-3D’s modules after the 4D
correlation volume computation.

##  IV Experiments

We compare FusionRAFT against state-of-the-art approaches on the
FlyingThings3D [15] and KITTI [16] datasets. We design two experimental
settings to mimic corrupted RGB images and poor lighting condition scenarios.
We also evaluate on data we acquired with a RGBD sensor in various lighting
conditions. We report both quantitative and qualitative results, and carry out
ablation studies.

###  IV-A Experimental setup

Datasets. FlyingThings3D [15] is split into clean and final sets containing
dynamic synthetic scenes. The former is composed of 27K RGBD images including
changing lighting and shading effects, while the latter is an augmented
version of the former with simulated challenging motions and blurs. Each set
contains train and test splits. Previous methods [10, 1, 12] exclude samples
containing fast-moving objects during the evaluation. However, as such visual
challenges is of interests to our problem, we use the whole training set of
FlyingThings3D and sample 1K RGBD image pairs from the whole test set for the
evaluation. KITTI consists of real-world scenes captured from vehicles in
urban scenarios. Because the original dataset does not provide depth data, we
use the disparity estimated by GA-Net [29] as in [1]. We exploit KITTI to
assess the ability of our model and the compared ones in generalizing from
synthetic to real data, without training or finetuning using any of the
KITTI’s sequences. We use the training set of KITTI as our evaluation set
since KITTI’s test set is not publicly available. To further validate the
performance of FusionRAFT in real-world scenarios, we collect an RGBD dataset
using a Realsense D415 camera in an indoor office with moving people under
three lighting setups, named Bright, Dimmed, and Dark. The Bright setting
features bright lighting, where the moving objects are clearly visible. The
Dimmed setting features dimmed lighting, where the moving objects can be
observed with a lower visual quality. The Dark setting features very low
lighting where the moving objects can be barely seen. We only qualitatively
evaluate this dataset because we could not produce optical flow ground truth.

Evaluation metrics. We quantify the optical and scene flow results using
conventional evaluation metrics [10, 11, 1]: for the optical flow we use $\rm
AEPE_{2D}$(pixel), $\rm{ACC}_{1px}$(%) and $\rm Fl^{all}_{2D}$(%), for the
scene flow we use $\rm AEPE_{3D}$(m), $\rm{ACC}_{0.05m}$(%),
$\rm{ACC}_{0.10m}$(%) and $\rm Fl^{all}_{3D}$(%). $\rm AEPE_{2D}$ measures the
average end-point error (EPE) [10], which is an average value of all the 2D
flow errors. $\rm AEPE^{epe{\textless{100}}}_{2D}$ measures the average end-
point error (EPE) among the 2D flow errors that are less than 100 pixels. $\rm
AEPE_{3D}$ is the average of euclidean distance (EPE for 3D) between the
ground-truth 3D scene flow and the predicted results. $\rm
AEPE^{epe{\textless{1}}}_{3D}$ measures the average end-point error (EPE)
among the 3D flow errors that are less than 1 meter. $\rm{ACC}_{1px}$ [1]
measures the portion of errors that are within a threshold of one pixel.
$\rm{ACC}_{0.05m}$ [1] measures the portion of errors that are within a
threshold of 0.05 meters, while $\rm{ACC}_{0.10m}$ [1] measures the portion of
errors that are within a threshold of 0.10 meters. $\rm MEAN_{AEPE}$ and $\rm
MEAN_{ACC}$ are the average values of $\rm AEPE^{all}_{2D}$ and $\rm
ACC_{1px}$, respectively, calculated over FlyingThings3D-clean and
FlyingThings3D-final. $\rm Fl^{all}_{2D}$ [11] is the percentage of outlier
pixels whose end-point error is $>3$ pixels or $5\%$ of the ground-truth flow
magnitude. $\rm Fl^{all}_{3D}$ [30] is the percentage of outlier pixels whose
3D Euclidean distance between the ground-truth 3D scene flow and the predicted
one is $>0.3$ m or $5\%$ of the ground-truth flow magnitude.

Evaluation settings. Environments with poor light conditions lead to weak
texture information that can compromise the stability of feature
representation. Also additive Gaussian noises can affect optical and scene
flow estimation. To assess the robustness, we design three experimental
settings on the public FlyingThings3D and KITTI datasets: _Standard_ : we use
the original version of the dataset; _AGN_ : we apply Additive Gaussian Noise
on RGB images; _Dark_ : we darken RGB images. In AGN we randomly sample noise
values ($\alpha$) from a normal distribution centered in zero with a standard
deviation equal to 35. In Dark we divide pixel values by a random factor
$\beta\sim\mbox{U}(\\{1,2,\cdots,9\\})$.

Implementation details. We implemented FusionRAFT in PyTorch with all modules
initialized with random weights. We train our network for 100K iterations with
the batch size of 6 on 3 Nvidia 3090 GPUs. During training, we set the initial
learning rate at $1.25\cdot 10^{-4}$ and use linear decay. We apply MMTM
sequentially with N = 3 times as suggested in the original paper [26]. We set
$\gamma{=}0.8$ in Eq. (5) as in RAFT [10].

###  IV-B Comparisons

We compare FusionRAFT against RGB methods for 2D optical flow estimation, i.e.
RAFT [10], GMA [20], CRAFT [11], and Separable flow [31], and against methods
for 3D scene flow estimation, i.e. RAFT-3D [1] and CamLiRAFT [25]. See Sec. II
for the description of these methods.

TABLE I:  Optical flow estimation in the Standard, AGN, and Dark settings on
FlyingThings3D-clean, FlyingThings3D-final, and KITTI. All models are trained
with FlyingThings3D, without fine-tuning on KITTI. Bold font indicates the
best-performing method.

| Method | Input | FlyingThings3D-clean | FlyingThings3D-final | KITTI-Train
---|---|---|---|---|---
| $\rm ACC_{1px}$ | $\rm AEPE^{epe{\textless{100}}}_{2D}$ | $\rm AEPE^{all}_{2D}$ | $\rm ACC_{1px}$ | $\rm AEPE^{epe{\textless{100}}}_{2D}$ | $\rm AEPE^{all}_{2D}$ | $\rm AEPE^{all}_{2D}$ | $\rm Fl^{all}_{2D}$
Standard setting |  RAFT [10] | RGB | 77.06 | 2.65 | 4.69 | 76.91 | 2.67 | 4.39 | 6.76 | 20.99
GMA [20] | RGB | 78.81 | 2.58 | 4.43 | 78.66 | 2.57 | 4.20 | 6.10 | 20.47
Separable flow [31] | RGB | 75.39 | 2.88 | 4.57 | 75.29 | 2.84 | 4.29 | 6.40 | 20.66
CRAFT [11] | RGB | 77.90 | 2.79 | 4.85 | 77.70 | 2.77 | 4.66 | 6.82 | 21.95
FusionRAFT-2D | RGBD | 80.37 | 2.17 | 3.52 | 80.21 | 2.22 | 3.42 | 5.49 | 18.05
RAFT-3D [1] | RGBD | 86.01 | 1.79 | 3.58 | 85.97 | 1.76 | 3.57 | 5.91 | 17.80
CamLiRAFT [25] | RGB+LiDAR | 83.59 | 1.81 | 3.03 | 83.26 | 1.80 | 2.84 | 4.84 | 14.76
FusionRAFT-3D | RGBD | 87.45 | 1.57 | 2.58 | 87.39 | 1.58 | 2.69 | 4.70 | 12.36
AGN setting |  RAFT [10] | RGB | 71.42 | 2.98 | 4.89 | 71.01 | 2.96 | 4.64 | 7.23 | 23.45
GMA [20] | RGB | 72.63 | 2.99 | 5.23 | 72.20 | 2.94 | 5.24 | 7.01 | 23.52
Separable flow [31] | RGB | 68.95 | 3.15 | 5.32 | 68.57 | 3.18 | 5.23 | 8.26 | 25.79
CRAFT [11] | RGB | 73.30 | 2.86 | 4.65 | 72.81 | 2.88 | 4.67 | 7.45 | 23.65
FusionRAFT-2D | RGBD | 77.24 | 2.24 | 3.50 | 76.77 | 2.30 | 3.38 | 5.47 | 19.15
RAFT-3D [1] | RGBD | 84.59 | 1.79 | 3.14 | 84.26 | 1.84 | 3.21 | 5.50 | 17.94
CamLiRAFT [25] | RGB+LiDAR | 76.98 | 2.23 | 3.98 | 76.31 | 2.33 | 3.71 | 5.26 | 16.98
FusionRAFT-3D | RGBD | 86.75 | 1.55 | 2.63 | 86.71 | 1.53 | 2.60 | 4.53 | 11.57
Dark setting |  RAFT [10] | RGB | 60.26 | 4.00 | 8.15 | 60.36 | 4.01 | 7.85 | 11.75 | 31.81
GMA [20] | RGB | 63.36 | 4.50 | 9.96 | 62.10 | 4.62 | 10.34 | 9.65 | 27.87
Separable flow [31] | RGB | 68.20 | 4.84 | 7.96 | 68.03 | 4.86 | 7.79 | 10.09 | 28.41
CRAFT [11] | RGB | 70.07 | 4.84 | 8.46 | 69.77 | 4.87 | 8.44 | 11.10 | 29.47
FusionRAFT-2D | RGBD | 76.70 | 2.39 | 3.65 | 76.57 | 2.38 | 3.66 | 8.29 | 23.87
RAFT-3D [1] | RGBD | 81.03 | 2.20 | 3.78 | 80.96 | 2.20 | 3.56 | 15.14 | 32.08
CamLiRAFT [25] | RGB+LiDAR | 74.80 | 2.54 | 4.54 | 74.73 | 2.64 | 4.11 | 7.44 | 16.97
FusionRAFT-3D | RGBD | 87.11 | 1.55 | 2.91 | 87.03 | 1.58 | 2.84 | 7.26 | 20.07

TABLE II:  Scene flow estimation in the Standard, AGN, and Dark settings on
FlyingThings3D-clean, FlyingThings3D-final, and KITTI. All models are trained
with FlyingThings3D, without fine-tuning on KITTI. Bold font indicates the
best-performing method.

Method | Setting | FlyingThings3D-clean | FlyingThings3D-final | KITTI-Train
---|---|---|---|---
$\rm ACC_{0.05m}$ | $\rm ACC_{0.10m}$ | $\rm AEPE^{epe{\textless{1}}}_{3D}$ | $\rm AEPE^{all}_{3D}$ | $\rm ACC_{0.05m}$ | $\rm ACC_{0.10m}$ | $\rm AEPE^{epe{\textless{1}}}_{3D}$ | $\rm AEPE^{all}_{3D}$ | $\rm AEPE^{all}_{3D}$ | $\rm Fl^{all}_{3D}$
RAFT-3D [1] | Standard | 74.01 | 81.22 | 0.064 | 0.186 | 74.25 | 81.43 | 0.064 | 0.180 | 0.136 | 5.20
CamLiRAFT [25] | Standard | 76.83 | 87.98 | 0.049 | 0.104 | 76.87 | 88.20 | 0.049 | 0.102 | 0.121 | 7.13
FusionRAFT-3D | Standard | 77.04 | 83.74 | 0.056 | 0.100 | 76.80 | 83.58 | 0.057 | 0.101 | 0.134 | 4.90
RAFT-3D[1] | AGN | 75.05 | 81.77 | 0.065 | 0.193 | 74.75 | 81.56 | 0.066 | 0.144 | 0.134 | 5.36
CamLiRAFT [25] | AGN | 73.83 | 86.59 | 0.055 | 0.116 | 73.38 | 86.31 | 0.055 | 0.126 | 0.122 | 7.81
FusionRAFT-3D | AGN | 76.60 | 82.71 | 0.061 | 0.104 | 76.35 | 82.55 | 0.062 | 0.107 | 0.134 | 5.52
RAFT-3D[1] | Dark | 71.21 | 79.33 | 0.071 | 0.203 | 71.00 | 79.24 | 0.072 | 0.145 | 0.145 | 9.39
CamLiRAFT[25] | Dark | 66.70 | 82.24 | 0.068 | 0.141 | 66.65 | 82.01 | 0.069 | 0.127 | 0.171 | 9.39
FusionRAFT-3D | Dark | 76.72 | 83.62 | 0.057 | 0.175 | 76.58 | 83.51 | 0.057 | 0.112 | 0.136 | 6.20

####  IV-B1 Quantitative results

Tab. I (top) reports optical flow results in Standard setting. FusionRAFT-2D
outperforms GMA by $+1.56\%$ and $+1.55\%$ in terms of $\rm ACC_{1px}$, and
$+0.91$ and $+0.78$ in terms of $\rm AEPE^{all}_{2D}$ in FlyingThings3D-clean
and FlyingThings3D-final, respectively. FusionRAFT-3D outperforms RAFT-3D by
$+1.44\%$ and $+1.42\%$ in terms of $\rm ACC_{1px}$, and $+1.00$ and $+0.88$
in terms of $\rm AEPE^{all}_{2D}$. While RAFT-3D extracts features only from
RGB images, our MFF encoder extracts features from both RGB and depth,
producing more informative internal representations. FusionRAFT-3D outperforms
CamLiRAFT by $+3.86\%$ and $+4.13\%$ in terms of $\rm ACC_{1px}$, and $+0.45$
and $+0.15$ in terms of $\rm AEPE^{all}_{2D}$.

\begin{overpic}[width=52.03227pt]{sections/images/flow-error/setting1-rgb.png}
\put(38.0,-12.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize{RGB}}
\end{overpic} \begin{overpic}[width=52.03227pt]{sections/images/flow-
error/depth.jpg}
\put(38.0,-12.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize{depth}}
\end{overpic} \begin{overpic}[width=52.03227pt]{sections/images/flow-
error/RAFT1.png}
\put(38.0,62.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize{RAFT}}
\put(10.0,-12.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize$\rm
AEPE^{all}_{2D}$=10.22} \end{overpic}
\begin{overpic}[width=52.03227pt]{sections/images/flow-error/GMA1.png}
\put(36.0,62.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize{GMA}}
\put(10.0,-12.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize$\rm
AEPE^{all}_{2D}$=12.06} \end{overpic}
\begin{overpic}[width=52.03227pt]{sections/images/flow-error/ours-2d1.png}
\put(10.0,62.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize{{\mbox{FusionRAFT}}-2D}}
\put(10.0,-12.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize$\rm
AEPE^{all}_{2D}$=7.18} \end{overpic}
\begin{overpic}[width=52.03227pt]{sections/images/flow-error/RAFT-3d1.jpg}
\put(29.0,62.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize{RAFT-3D}}
\put(10.0,-12.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize$\rm
AEPE^{all}_{2D}$=8.52} \end{overpic}
\begin{overpic}[width=52.03227pt]{sections/images/flow-
error/camliraft/setting1-3.4394.png}
\put(19.0,62.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize{CamLiRAFT}}
\put(10.0,-12.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize$\rm
AEPE^{all}_{2D}$=3.44} \end{overpic}
\begin{overpic}[width=52.03227pt]{sections/images/flow-error/ours-3d1.jpg}
\put(10.0,62.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize{{\mbox{FusionRAFT}}-3D}}
\put(10.0,-12.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize$\rm
AEPE^{all}_{2D}$=2.46} \end{overpic}
---
\begin{overpic}[width=52.03227pt]{sections/images/flow-error/setting2-rgb.png}
\put(38.0,-12.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize{RGB}}
\end{overpic} \begin{overpic}[width=52.03227pt]{sections/images/flow-
error/depth.jpg}
\put(38.0,-12.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize{depth}}
\end{overpic} \begin{overpic}[width=52.03227pt]{sections/images/flow-
error/RAFT2.png}
\put(10.0,-12.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize$\rm
AEPE^{all}_{2D}$=11.24} \end{overpic}
\begin{overpic}[width=52.03227pt]{sections/images/flow-error/GMA2.png}
\put(10.0,-12.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize$\rm
AEPE^{all}_{2D}$=11.13} \end{overpic}
\begin{overpic}[width=52.03227pt]{sections/images/flow-error/ours-2d2.png}
\put(10.0,-12.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize$\rm
AEPE^{all}_{2D}$=5.56} \end{overpic}
\begin{overpic}[width=52.03227pt]{sections/images/flow-error/RAFT-3d2.jpg}
\put(10.0,-12.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize$\rm
AEPE^{all}_{2D}$=4.03} \end{overpic}
\begin{overpic}[width=52.03227pt]{sections/images/flow-
error/camliraft/setting2-3.7638.png}
\put(10.0,-12.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize$\rm
AEPE^{all}_{2D}$=3.76} \end{overpic}
\begin{overpic}[width=52.03227pt]{sections/images/flow-error/ours-3d2.jpg}
\put(10.0,-12.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize$\rm
AEPE^{all}_{2D}$=2.63} \end{overpic}
\begin{overpic}[width=52.03227pt]{sections/images/flow-error/setting3-rgb.png}
\put(38.0,-12.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize{RGB}}
\end{overpic} \begin{overpic}[width=52.03227pt]{sections/images/flow-
error/depth.jpg}
\put(38.0,-12.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize{depth}}
\end{overpic} \begin{overpic}[width=52.03227pt]{sections/images/flow-
error/RAFT3.png}
\put(10.0,-12.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize$\rm
AEPE^{all}_{2D}$=14.69} \end{overpic}
\begin{overpic}[width=52.03227pt]{sections/images/flow-error/GMA3.png}
\put(10.0,-12.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize$\rm
AEPE^{all}_{2D}$=11.34} \end{overpic}
\begin{overpic}[width=52.03227pt]{sections/images/flow-error/ours-2d3.png}
\put(10.0,-12.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize$\rm
AEPE^{all}_{2D}$=7.69} \end{overpic}
\begin{overpic}[width=52.03227pt]{sections/images/flow-error/RAFT-3d3.jpg}
\put(10.0,-12.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize$\rm
AEPE^{all}_{2D}$=5.18} \end{overpic}
\begin{overpic}[width=52.03227pt]{sections/images/flow-
error/camliraft/setting3-5.9515.png}
\put(10.0,-12.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize$\rm
AEPE^{all}_{2D}$=5.95} \end{overpic}
\begin{overpic}[width=52.03227pt]{sections/images/flow-error/ours-3d3.jpg}
\put(10.0,-12.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize$\rm
AEPE^{all}_{2D}$=2.35} \end{overpic}
Figure 2:  Examples of optical flow estimation error in the
FlyingThings3D-clean dataset. The more vivid the magenta, the higher the
error. FusionRAFT-2D method handles optical flow estimation better than RGB-
based methods, while FusionRAFT-3D method outperforms RAFT-3D with a smaller
AEPE. Best viewed in color.
\begin{overpic}[width=52.03227pt]{sections/images/flow-error-
kitti/imagekitti.png}
\put(38.0,-12.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize{RGB}}
\end{overpic} \begin{overpic}[width=52.03227pt]{sections/images/flow-error-
kitti/depthkitti.jpg}
\put(38.0,-12.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize{depth}}
\end{overpic} \begin{overpic}[width=52.03227pt]{sections/images/flow-error-
kitti/RAFT-1-4.99.png}
\put(38.0,35.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize{RAFT}}
\put(10.0,-12.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize$\rm
AEPE^{all}_{2D}$=4.99} \end{overpic}
\begin{overpic}[width=52.03227pt]{sections/images/flow-error-
kitti/GMA-1-4.30.png}
\put(36.0,35.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize{GMA}}
\put(10.0,-12.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize$\rm
AEPE^{all}_{2D}$=4.30} \end{overpic}
\begin{overpic}[width=52.03227pt]{sections/images/flow-error-
kitti/ours2d-1-4.11.png}
\put(10.0,35.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize{{\mbox{FusionRAFT}}-2D}}
\put(10.0,-12.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize$\rm
AEPE^{all}_{2D}$=4.11} \end{overpic}
\begin{overpic}[width=52.03227pt]{sections/images/flow-error-
kitti/RAFT3d-1-3.80.png}
\put(29.0,35.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize{RAFT-3D}}
\put(10.0,-12.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize$\rm
AEPE^{all}_{2D}$=3.80} \end{overpic}
\begin{overpic}[width=52.03227pt]{sections/images/flow-
error/camliraft/setting1-kitti-3.4892.png}
\put(19.0,35.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize{CamLiRAFT}}
\put(10.0,-12.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize$\rm
AEPE^{all}_{2D}$=3.49} \end{overpic}
\begin{overpic}[width=52.03227pt]{sections/images/flow-error-
kitti/ours3d-1-2.513.png}
\put(10.0,35.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize{{\mbox{FusionRAFT}}-3D}}
\put(10.0,-12.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize$\rm
AEPE^{all}_{2D}$=2.51} \end{overpic}
---
\begin{overpic}[width=52.03227pt]{sections/images/flow-error-kitti/rgb-2.png}
\put(38.0,-12.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize{RGB}}
\end{overpic} \begin{overpic}[width=52.03227pt]{sections/images/flow-error-
kitti/depthkitti.jpg}
\put(38.0,-12.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize{depth}}
\end{overpic} \begin{overpic}[width=52.03227pt]{sections/images/flow-error-
kitti/RAFT-2-5.93.png}
\put(10.0,-12.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize$\rm
AEPE^{all}_{2D}$=5.93} \end{overpic}
\begin{overpic}[width=52.03227pt]{sections/images/flow-error-
kitti/GMA-2-4.75.png}
\put(10.0,-12.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize$\rm
AEPE^{all}_{2D}$=4.75} \end{overpic}
\begin{overpic}[width=52.03227pt]{sections/images/flow-error-
kitti/ours2d-2-4.74.png}
\put(10.0,-12.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize$\rm
AEPE^{all}_{2D}$=4.74} \end{overpic}
\begin{overpic}[width=52.03227pt]{sections/images/flow-error-
kitti/RAFT3d-2-3.09.png}
\put(10.0,-12.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize$\rm
AEPE^{all}_{2D}$=3.09} \end{overpic}
\begin{overpic}[width=52.03227pt]{sections/images/flow-
error/camliraft/setting2-kitti-3.5997.png}
\put(10.0,-12.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize$\rm
AEPE^{all}_{2D}$=3.60} \end{overpic}
\begin{overpic}[width=52.03227pt]{sections/images/flow-error-
kitti/ours3d-2-2.704.png}
\put(10.0,-12.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize$\rm
AEPE^{all}_{2D}$=2.70} \end{overpic}
\begin{overpic}[width=52.03227pt]{sections/images/flow-error-kitti/rgb-3.png}
\put(38.0,-12.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize{RGB}}
\end{overpic} \begin{overpic}[width=52.03227pt]{sections/images/flow-error-
kitti/depthkitti.jpg}
\put(38.0,-12.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize{depth}}
\end{overpic} \begin{overpic}[width=52.03227pt]{sections/images/flow-error-
kitti/RAFT-3-52.98.png}
\put(10.0,-12.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize$\rm
AEPE^{all}_{2D}$=52.98} \end{overpic}
\begin{overpic}[width=52.03227pt]{sections/images/flow-error-
kitti/GMA-3-5.52.png}
\put(10.0,-12.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize$\rm
AEPE^{all}_{2D}$=5.52} \end{overpic}
\begin{overpic}[width=52.03227pt]{sections/images/flow-error-
kitti/ours2d-3-4.82.png}
\put(10.0,-12.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize$\rm
AEPE^{all}_{2D}$=4.82} \end{overpic}
\begin{overpic}[width=52.03227pt]{sections/images/flow-error-
kitti/RAFT3d-3-8.75.png}
\put(10.0,-12.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize$\rm
AEPE^{all}_{2D}$=8.75} \end{overpic}
\begin{overpic}[width=52.03227pt]{sections/images/flow-
error/camliraft/setting3-kitti-3.7383.png}
\put(10.0,-12.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize$\rm
AEPE^{all}_{2D}$=3.74} \end{overpic}
\begin{overpic}[width=52.03227pt]{sections/images/flow-error-
kitti/ours3d-3-3.050.png}
\put(10.0,-12.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize$\rm
AEPE^{all}_{2D}$=3.05} \end{overpic}
Figure 3: Examples of optical flow estimation error in the KITTI dataset. The
more vivid the magenta, the higher the error. FusionRAFT-2D method handles
optical flow estimation better than all RGB-based methods. FusionRAFT-3D
method outperforms RAFT-3D with a smaller AEPE. Best viewed in color.
\begin{overpic}[width=52.03227pt]{sections/images/realtest/0059.png}
\put(38.0,-12.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize{RGB}}
\end{overpic}
\begin{overpic}[width=52.03227pt]{sections/images/realtest/0059depth.png}
\put(38.0,-12.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize{depth}}
\end{overpic}
\begin{overpic}[width=52.03227pt]{sections/images/realtest/RAFT-61.png}
\put(38.0,60.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize{RAFT}}
\end{overpic}
\begin{overpic}[width=52.03227pt]{sections/images/realtest/GMA-61.png}
\put(36.0,60.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize{GMA}}
\end{overpic}
\begin{overpic}[width=52.03227pt]{sections/images/realtest/ours-2D-61.png}
\put(10.0,60.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize{{\mbox{FusionRAFT}}-2D}}
\end{overpic}
\begin{overpic}[width=52.03227pt]{sections/images/realtest/RAFT-3D-61.png}
\put(29.0,60.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize{RAFT-3D}}
\end{overpic}
\begin{overpic}[width=52.03227pt]{sections/images/realtest/CamliRAFT-59.png}
\put(22.0,60.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize{CamLiRAFT}}
\end{overpic}
\begin{overpic}[width=52.03227pt]{sections/images/realtest/ours-3D-61.png}
\put(10.0,60.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize{{\mbox{FusionRAFT}}-3D}}
\end{overpic}
---
\begin{overpic}[width=52.03227pt]{sections/images/realtest1/0457.png}
\put(38.0,-12.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize{RGB}}
\end{overpic}
\begin{overpic}[width=52.03227pt]{sections/images/realtest1/0457depth.png}
\put(38.0,-12.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize{depth}}
\end{overpic}
\begin{overpic}[width=52.03227pt]{sections/images/realtest1/RAFT-459.png}
\end{overpic}
\begin{overpic}[width=52.03227pt]{sections/images/realtest1/GMA-459.png}
\end{overpic}
\begin{overpic}[width=52.03227pt]{sections/images/realtest1/ours-2D-459.png}
\end{overpic}
\begin{overpic}[width=52.03227pt]{sections/images/realtest1/RAFT-3D-459.png}
\end{overpic}
\begin{overpic}[width=52.03227pt]{sections/images/realtest1/457.png}
\end{overpic}
\begin{overpic}[width=52.03227pt]{sections/images/realtest1/ours-3D-459.png}
\end{overpic}
\begin{overpic}[width=52.03227pt]{sections/images/realtest2/0676.png}
\put(38.0,-12.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize{RGB}}
\end{overpic}
\begin{overpic}[width=52.03227pt]{sections/images/realtest2/0676depth.png}
\put(38.0,-12.0){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\scriptsize{depth}}
\end{overpic}
\begin{overpic}[width=52.03227pt]{sections/images/realtest2/RAFT-678.png}
\end{overpic}
\begin{overpic}[width=52.03227pt]{sections/images/realtest2/GMA-678.png}
\end{overpic}
\begin{overpic}[width=52.03227pt]{sections/images/realtest2/ours-2D-678.png}
\end{overpic}
\begin{overpic}[width=52.03227pt]{sections/images/realtest2/RAFT-3D-678.png}
\end{overpic}
\begin{overpic}[width=52.03227pt]{sections/images/realtest2/CamliRAFT-676.png}
\end{overpic}
\begin{overpic}[width=52.03227pt]{sections/images/realtest2/our-3D-678.png}
\end{overpic}
Figure 4: Examples of optical flow estimation error in our real-world dataset.
(top) Bright setting, (middle) Dimmed setting, (bottom) Dark setting.
FusionRAFT method can handle also the Dark setting (see sharper flow
boundaries). Best viewed in color.

Tab. I (middle) reports optical flow results in AGN setting. FusionRAFT-2D
outperforms CRAFT by $+3.94\%$ and $+3.96\%$ in terms of $\rm ACC_{1px}$, and
$+1.15$ and $+1.29$ in terms of $\rm AEPE^{all}_{2D}$ in FlyingThings3D-clean
and FlyingThings3D-final, respectively. FusionRAFT-3D outperforms RAFT-3D by
$+2.16\%$ and $+2.45\%$ in in terms of $\rm ACC_{1px}$ and $+0.51$ and $+0.61$
in terms of $\rm AEPE^{all}_{2D}$. $\rm AEPE^{all}_{2D}$ of RAFT-3D is lower
than that of the Standard setting. This is because $\rm AEPE^{all}_{2D}$
computes the average of all errors, and the average is known to be sensitive
to outliers. In fact by computing the median (less sensitive to outliers), the
performance of RAFT-3D in AGN setting is worse than the Standard setting: e.g.
RAFT-3D in FlyingThings3D-clean achieves a median $\rm EPE^{all}_{2D}$ of
0.127 and 0.130 in the Standard and AGN settings, respectively.

Tab. I (bottom) reports optical flow results in Dark setting. RGB methods
perform worse than the previous settings, whereas our FusionRAFT methods
outperform RGB methods, RAFT-3D, and CamLiRAFT. Tab. I also reports $\rm
AEPE^{all}_{2D}$ and $\rm Fl^{all}_{2D}$ on KITTI without fine-tuning the
models. Although CamLiRAFT demonstrates a better generalization capability
than RAFT-3D, FusionRAFT outperforms almost all the other methods in all three
settings, demonstrating its robustness and adaptability in real-world
scenarios.

Tab. II reports scene flow results. FusionRAFT-3D outperforms RAFT-3D on both
FlyingThings3D-clean and FlyingThings3D-final. CamLiRAFT scores on par with
FusionRAFT-3D in all three settings. FusionRAFT-3D performs better in terms of
$\rm ACC_{0.05m}$, while CamLiRAFT performs better in terms of $\rm
ACC_{0.10m}$. This suggests that FusionRAFT-3D produces more small flow errors
than CamLiRAFT. In general, FusionRAFT-3D performs stably across all three
settings, while the performance of RAFT-3D and CamLiRAFT degrades in the AGN
and Dark settings. On KITTI, FusionRAFT-3D is the best-performing method on
all three settings in terms of $\rm Fl^{all}_{3D}$. In terms of $\rm
AEPE_{3D}^{all}$, CamLiRAFT performs better in the Standard and AGN settings,
while FusionRAFT-3D scores the best in the Dark setting.

####  IV-B2 Qualitative results

We provide examples of qualitative optical flow results indicated with their
corresponding $\rm AEPE^{all}_{2D}$. We visualize the errors with respect to
the ground-truth: the stronger the magenta, the higher the error. Fig. 2 and
Fig. 3 show the results of optical flow errors on FlyingThings3D and KITTI,
respectively. Both FusionRAFT-2D and FusionRAFT-3D consistently produce
smaller $\rm AEPE^{all}_{2D}$ values than the other methods, which can also be
visually verified with less magenta areas produced by our models. Fig. 4 shows
the flow estimation on our acquired indoor dataset with RAFT, GMA, RAFT-3D,
CamLiRAFT, and FusionRAFT. In the Bright setting (top), all compared methods
produce good-quality results. In the Dimmed setting (middle), RAFT, GMA, and
CamLiRAFT show low-quality results, which we can observe from the poor edges
produced by the moving objects. In the Dark setting, FusionRAFT is the only
method that produces results where the moving objects are distinguishable.

###  IV-C Ablation study

Tab. III reports the ablation study on self-attention (SA), cross-attention
(CA), and Multimodal Transfer Module (MMTM) on the FlyingThings3D dataset in
both Standard and Dark settings. Overall, we can observe that all the
components we added provide an incremental contribution to improve the quality
of the output optical flow compared to the RGB baseline. SA and CA
consistently improve performance (see Exp 3 vs 6 vs 8, 4 vs 7 vs 9, and
similarly for the Dark setting). The SA applied to both depth and RGB is
better than applying it to the RGB branch only (see Exp 5 vs 6 for the
Standard setting, and 14 vs 15 for the Dark setting). MMTM fusion consistently
outperforms the simple concatenation of RGB and depth branches in the Dark
setting (see Exp 12 vs 13, 15 vs 16, 17 vs 18). There is one case in the
Standard setting where this last does not occur (see Exp 6 vs 7). In general,
SA focuses on intra-modality relationships while CA focuses on inter-modality
relationships. MMTM further exchanges information across modalities at a
deeper level. The best performance is achieved when all the modules are
activated.

TABLE III: Ablation study in Standard and Dark settings on FlyingThings3D.
$\rm MEAN_{ACC}$ and $\rm MEAN_{AEPE}$ are the mean of $\rm ACC_{1px}$ and
$\rm AEPE^{all}_{2D}$, respectively, on FlyingThings3D-clean and
FlyingThings3D-final. ✓(RGB): self-attention is performed on the RGB.

| Exp | RGB | depth | SA | CA | Fusion | FlyingThings3D-clean | FlyingThings3D-final | $\rm MEAN_{ACC}$ | $\rm MEAN_{AEPE}$
---|---|---|---|---|---|---|---|---|---|---
| $\rm ACC_{1px}$ | $\rm AEPE^{all}_{2D}$ | $\rm ACC_{1px}$ | $\rm AEPE^{all}_{2D}$
Standard setting | 1 | ✓ | - | - | - | - | 77.06 | 4.69 | 76.91 | 4.39 | 76.99 | 4.54
2 | ✓ | - | ✓ | - | - | 77.81 | 4.50 | 77.72 | 4.26 | 77.77 | 4.38
3 | ✓ | ✓ | - | - | concat | 79.21 | 3.74 | 79.06 | 3.71 | 79.14 | 3.73
4 | ✓ | ✓ | - | - | MMTM | 79.43 | 3.79 | 79.27 | 3.69 | 79.35 | 3.74
5 | ✓ | ✓ | ✓(RGB) | - | concat | 79.35 | 3.72 | 79.18 | 3.66 | 79.27 | 3.69
6 | ✓ | ✓ | ✓ | - | concat | 79.71 | 3.81 | 79.55 | 3.63 | 79.63 | 3.72
7 | ✓ | ✓ | ✓ | - | MMTM | 78.97 | 3.73 | 78.77 | 3.65 | 78.87 | 3.69
8 | ✓ | ✓ | ✓ | ✓ | concat | 80.14 | 3.56 | 79.95 | 3.53 | 80.05 | 3.55
9 | ✓ | ✓ | ✓ | ✓ | MMTM | 80.37 | 3.52 | 80.21 | 3.42 | 80.29 | 3.47
Dark setting | 10 | ✓ | - | - | - | - | 60.26 | 8.15 | 60.36 | 7.85 | 60.31 | 8.00
11 | ✓ | - | ✓ | - | - | 67.92 | 8.01 | 67.80 | 7.77 | 67.86 | 7.89
12 | ✓ | ✓ | - | - | concat | 75.33 | 4.06 | 75.17 | 4.05 | 75.25 | 4.06
13 | ✓ | ✓ | - | - | MMTM | 75.56 | 3.97 | 75.40 | 3.99 | 75.48 | 3.98
14 | ✓ | ✓ | ✓(RGB) | - | concat | 75.47 | 3.95 | 75.32 | 3.94 | 75.40 | 3.95
15 | ✓ | ✓ | ✓ | - | concat | 75.69 | 3.96 | 75.57 | 3.90 | 75.63 | 3.93
16 | ✓ | ✓ | ✓ | - | MMTM | 75.75 | 3.81 | 75.60 | 3.76 | 75.68 | 3.79
17 | ✓ | ✓ | ✓ | ✓ | concat | 76.43 | 3.79 | 76.26 | 3.72 | 76.35 | 3.76
18 | ✓ | ✓ | ✓ | ✓ | MMTM | 76.70 | 3.65 | 76.57 | 3.66 | 76.64 | 3.66

###  IV-D Computation analysis

We measure the number of parameters, Floating-Point Operations (FLOPs), and
inference time of all compared methods using FlyingThings3D. We conducted the
experiments with a Nvidia 3090 GPU (24G) and I9-10900 CPUs, and reported the
results in Tab. IV. Despite FusionRAFT-2D has the second-largest number of
parameters, its number of FLOPs and inference time are in-between the other
methods for optical flow estimation. The inference time of FusionRAFT-3D is
slightly higher than that of CamLiRAFT, although our number of parameters is
one order of magnitude larger than CamLiRAFT. From the per-component analysis
of FusionRAFT-2D in Tab. V, we can observe that Self-attention and Cross-
attention have a higher computational cost than MMTM and the two-branch
encoder. The most time-consuming component is Others which includes all the
other modules to compute the optical flow.

TABLE IV: Comparative computational analysis by using 960$\times$540-sized
images on a Nvidia 3090.

Models | Params [M] | FLOPs [T] | Inference time [ms]
---|---|---|---
RAFT [10] | 5.31 | 0.78 | 99
GMA [20] | 5.88 | 0.59 | 87
Separable flow [31] | 8.35 | 0.50 | 639
CRAFT [11] | 6.31 | 0.99 | 302
FusionRAFT-2D | 8.13 | 0.68 | 219
RAFT-3D [1] | 44.50 | 0.51 | 277
CamLiRAFT [25] | 8.41 | 0.67 | 312
FusionRAFT-3D | 86.32 | 0.76 | 398

TABLE V: Ablations of computational performance on FusionRAFT-2D by using
960$\times$540-sized images on a Nvidia 3090.

Models | Params [M] | FLOPs [T] | Inference time [ms]
---|---|---|---
Image encoder | 2.33 | 0.11 | 13
Depth encoder | 2.33 | 0.11 | 13
Self-attention | 0.16 | 0.06 | 63
Cross-attention | 0.09 | 0.04 | 43
MMTM | 0.24 | 2M | 1
Others | 2.98 | 0.36 | 86
Total | 8.13 | 0.68 | 219

##  V Conclusions

We presented FusionRAFT, a novel approach for optical and scene flow
estimation. FusionRAFT improves feature extraction with an early-fusion
Multimodal Feature Fusion (MFF) Encoder. MFF attends to informative features
and enables information exchange within and across modalities by using self-
attention, cross-attention, and the Multimodal Transfer Module. Through
experimental validation, we showed that FusionRAFT generates more stable and
informative feature descriptions by exploiting the different modalities.
FusionRAFT scores state-of-the-art results in Standard setting, but also in
our newly introduced AGN and Dark settings where RGB information is corrupted.
Future research directions may include the integration of FusionRAFT in
robotic systems for autonomous navigation.

## References

  * [1] Z. Teed and J. Deng, “Raft-3d: Scene flow using rigidmotion embeddings,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2021.
  * [2] T. Zhang, H. Zhang, Y. Li, Y. Nakamura, and L. Zhang, “FlowFusion: Dynamic Dense RGB-D SLAM Based on Optical Flow,” in _Proceedings of IEEE International Conference on Robotics and Automation (ICRA)_ , Paris, Fr, June 2020\.
  * [3] G. de Croon, C. De-Wagter, and T. Seidl, “Enhancing optical-flow-based control by learning visual appearance cues for flying robots,” _Nature Machine Intelligence_ , vol. 3, no. 1, pp. 33–41, 2021.
  * [4] L. Sevilla-Lara, Y. Liao, F. Guney, V. Jampani, A. Geiger, and M. J. Black, “On the integration of optical flow and action recognition,” in _Proceedings of German Conference on Pattern Recognition (GCPR)_ , Stuttgart, Ger, October 2018.
  * [5] S. Sun, Z. Kuang, L. Sheng, W. Ouyang, and W. Zhang, “Optical flow guided feature: A fast and robust motion representation for video action recognition,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , Utah, US, June 2018.
  * [6] A. Dosovitskiy, P. Fischer, , E. Ilg, P. Hausser, C. Hazirbas, V. Golkov, P. der Smagt, D. Cremers, and T. Brox, “FlowNet: Learning Optical Flow with Convolutional Networks,” in _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_ , Santiago, Chile, Dec 2015.
  * [7] E. Ilg, N. Mayer, T. Saikia, M. Keuper, A. Dosovitskiy, and T. Brox, “Flownet 2.0: Evolution of optical flow estimation with deep networks,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , Hawaii, US, July 2017.
  * [8] C. Bailer, B. Taetz, and D. Stricker, “Flow fields: Dense correspondence fields for highly accurate large displacement optical flow estimation,” in _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_ , Santiago, Chi, August 2015.
  * [9] R. Schuster, C. Bailer, O. Wasenm, and D. Stricker, “Flowfields++: Accurate optical flow correspondences meet robust interpolation,” in _Proceedings of the IEEE International Conference on Image Processing (ICIP)_ , Athens, GR, October 2018.
  * [10] Z. Teed and J. Deng, “Raft: Recurrent all-pairs field transforms for optical flow,” in _Proceedings of the European conference on computer vision European (ECCV)_ , August 2020.
  * [11] X. Sui, S. Li, X. Geng, Y. Wu, X. Xu, Y. Liu, R. Goh, and H. Zhu, “CRAFT: Cross-Attentional Flow Transformer for Robust Optical Flow,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , Louisiana, US, June 2022.
  * [12] H. Liu, T. Lu, Y. Xu, J. Liu, W. Li, and L. Chen, “CamLiFlow: Bidirectional Camera-LiDAR Fusion for Joint Optical Flow and Scene Flow Estimation,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , New Orleans, US, June 2022.
  * [13] R. Rishav, R. Battrawy, R. Schuster, O. Wasenmuller, and D. Stricker, “DeepLiDARFlow: A Deep Learning Architecture For Scene Flow Estimation Using Monocular Camera and Sparse LiDAR,” in _Proceedings of IEEE International Conference on Intelligent Robots and Systems (IROS)_ , Las Vegas, US, June 2020.
  * [14] R. Murphy, J. Kravitz, S. Stover, and R. Shoureshi, “Mobile robots in mine rescue and recovery,” _IEEE Robotics & Automation Magazine_, 2009.
  * [15] N. Mayer, E. Ilg, P. Hausser, and P. Fischer, “A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , Las Vegas, US, June 2016.
  * [16] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun, “Vision meets robotics: The kitti dataset,” _The International Journal of Robotics Research_ , vol. 32, no. 11, pp. 1231–1237, 2013.
  * [17] D. Sun, X. Yang, M. Liu, and J. Kautz, “PWC-Net: CNNs for optical flow using pyramid, warping, and cost volume,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , Utah, US, June 2018\.
  * [18] S. Zhao, Y. Sheng, Y. Dong, E. Chang, and Y. Xu, “Maskflownet: Asymmetric feature matching with learnable occlusion mask,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2020\.
  * [19] T. Hui and C. Change, “LiteFlowNet3: Resolving Correspondence Ambiguity for More Accurate Optical Flow Estimation,” in _Proceedings of the European conference on computer vision European (ECCV)_ , August 2020.
  * [20] S. Jiang, D. Campbell, Y. Lu, H. Li, and R. Hartley, “Learning to estimate hidden motions with global motion aggregation,” in _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_ , 2021.
  * [21] J. Wagner, V. Fischer, M. Herman, and S. Behnke, “Multispectral pedestrian detection using deep fusion convolutional neural networks,” in _Proceedings of European Symposium on Artificial Neural Networks (ESANN)_ , Bruges, BEL, April 2016.
  * [22] D. Konig, M. Adam, C. Jarvers, G. Layher, H. Neumann, and M. Teutsch, “Fully convolutional region proposal networks for multispectral person detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , Hawaii, US, July 2017.
  * [23] J. Liu, S. Zhang, S. Wang, and D. Metaxas, “Multispectral deep neural networks for pedestrian detection,” in _Proceedings of the British Machine Vision Conference (BMVC)_ , York, UK, September 2016.
  * [24] Y. Chen, J. Shi, Z. Ye, C. Mertz, S. Kong, and D. Ramanan, “Multimodal Object Detection via Probabilistic Ensembling,” in _Proceedings of the European conference on computer vision (ECCV)_ , Tel Aviv, Israel, October 2022\.
  * [25] H. Liu, T. Lu, Y. Xu, J. Liu, and L. Wang, “Learning optical flow and scene flow with bidirectional camera-lidar fusion,” _arXiv preprint arXiv:2303.12017_ , 2023.
  * [26] H. Reza, A. Shaban, M. L. Iuzzolino, and K. Koishida, “MMTM: Multimodal Transfer Module for CNN Fusion,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2020.
  * [27] D. Ulyanov, A. Vedaldi, and V. Lempitsky, “Instance normalization: The missing ingredient for fast stylization,” _arXiv preprint arXiv:1607.08022_ , 2016\.
  * [28] B. Xu, N. Wang, T. Chen, and M. Li, “Empirical evaluation of rectified activations in convolutional network,” _arXiv_ , 2015.
  * [29] F. Zhang, V. Prisacariu, R. Yang, and P. Torr, “GA-Net: Guided Aggregation Net for End-To-End Stereo Matching,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , Long Beach, US, June 2019.
  * [30] Z. Wang, S. Li, H. Howard-Jenkins, V. Prisacariu, and M. Chen, “Flownet3d++: Geometric losses for deep scene flow estimation,” in _Proceedings of IEEE Winter Conference on Applications of Computer Vision (WACV)_ , Waikoloa, US, March 2020.
  * [31] F. Zhang, O. J-Woodford, V. Prisacariu, and P. Torr, “Separable flow: Learning motion cost volumes for optical flow estimation,” in _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_ , March 2021.