# Discovery of a large and faint nebula at the Triangulum galaxy

Aleix Roig  Parc Astronòmic de les Muntanyes de Prades, Muralla 5, 43364
Prades, Tarragona, Spain<EMAIL_ADDRESS>Raúl Infante-Sainz Centro de
Estudios de Física del Cosmos de Aragón (CEFCA), Plaza San Juan, 1, E-44001,
Teruel, Spain  Judith Ardèvol Institut de Ciències del Cosmos (ICCUB),
Universitat de Barcelona (UB), Martí i Franquès 1, E-08028 Barcelona, Spain
Dept. de Física Quàntica i Astrofísica (FQA), Universitat de Barcelona (UB),
Martí i Franquès 1, E-08028 Barcelona, Spain  Institut d’Estudis Espacials de
Catalunya (IEEC), c. Gran Capità, 2-4, 08034 Barcelona, Spain

###### Abstract

We report the discovery of a previously uncatalogued arch-shaped filamentary
nebula at the outer part of the Triangulum galaxy (M33) centred at
$\alpha=1^{h}34^{m}25^{s}$, $\delta=+30^{\circ}20^{\prime}17^{\prime\prime}$
(ICRS). This discovery stems from meticulous observations employing deep
exposures of M33, using both H$\alpha$ and [OIII] narrow-band filters. The
nebula, designated as _Roig1 Prades Sky_ , exhibits an H$\alpha$ surface
brightness of $23.9\,\rm{mag}\,\rm{arcsec}^{-2}$. Its sky projected location
is $21\,$arcmin away from the M33 galactic centre towards the southeast
direction with an extent of $(120\times 440)\pm 30\,$pc. Deep spectroscopic
observations are required to unveil its real nature.

Astrophotography (97) — H alpha photometry (691) — Nebulae (1095) — Optical
astronomy (1776) — Triangulum Galaxy (1712)

##  1 Introduction

Amateur astronomers have more access to telescope observation time in
comparison to their professional counterparts, enabling them to conduct longer
observations covering wider areas of the sky. Aleix Roig used this advantage
to capture a very deep image of M33 covering an area of $2.4^{\circ}\times
1.6^{\circ}$ around this galaxy. The primary objective was to seek for [OIII]
and H$\alpha$ emission regions in the outskirts of M33.

These observations took place at the Astrocat
Observatory111https://astrocat.info, an amateur facility located in the dark
skies of Prades (Spain), characterised by a sky brightness of
$\mu_{\rm{V}}\sim 21\,\rm{mag}\,\rm{arcsec}^{-2}$. The exceptional conditions
allowed Aleix Roig to set up this observatory back in 2014. He held other
professional collaborations, such as the 100 hours image of the M101 galaxy
presented at the 355 IAU Symposium222https://www.iac.es/es/node/188587.

##  2 Data

The images were acquired employing two small refractor telescopes, both
synchronised by a MESU200 mount. Each telescope is equipped with its
independent camera and filters, configured to observe the same region of the
sky. The images were taken during two distinct periods: from September 22 to
26, and from October 5 to 11, 2023. Periods of high Moon illumination were
carefully avoided.

The first telescope, a Takahashi FSQ85 (85 mm aperture, F/5.3), captured a
total of 969 images, each lasting 300 seconds (total exposure time of 80.75
hours) using the [OIII] filter centred at 500.7 nm with a passband of 6.5 nm.
We used a ZWO ASI294MM 4/3” camera with a pixel size of 2.1 arcsec/pixel and a
field of view (FOV) of $2.44^{\circ}\times 1.66^{\circ}$.

The second telescope, a Takahashi FSQ106 (106 mm aperture, F/5.0), captured a
total of 534 images, each lasting 300 seconds (total exposure time of 44.5
hours) using the H$\alpha$ filter centred at 656.3 nm with a passband of 3 nm.
The camera used was a ZWO ASI2600MM APS-C with a pixel size of 1.5
arcsec/pixel and a FOV of $2.54^{\circ}\times 1.74^{\circ}$.

The Takahashi FSQ106 was used to capture images with luminance (L), red (R),
green (G), and blue (B) filters that were combined to obtain an LRGB-coloured
image with a total exposure time of 28.5 hours.

The data processing was mainly done with PixInsight333https://pixinsight.com/
to obtain the colour images. The use of J-PLUS (Cenarro et al., 2019) data,
and more in particular its H$\alpha$ (J0660) filter, allowed us to perform a
crude photometric calibration and the estimation of surface brightness
measurements by using Gnuastro (Akhlaghi & Ichikawa, 2015). The depth of the
combined H$\alpha$ image is $24.9\,\rm{mag}\,\rm{arcsec}^{-2}$ ($3\sigma$ in
100 arcsec2). Due to the lack of good-quality calibrated data, we could not
obtain reliable brightness measurements in the [OIII] filter.

##  3 Results

The resulting LRGB image is presented in the left panel of Figure 1. It shows
the details in the H$\alpha$ and [OIII] bands of the central part of M33. We
found no emission at these wavelengths on the outskirts of the galaxy.
Foreground Galactic cirrus can be distinguished with LRGB filters across those
outer regions. We refer to the Astrocat Observatory
web444https://astrocat.info/a-deep-triangulum-galaxy/ to see different
versions of the same data.

Figure 1:  Left: LRGB image of M33. B and G represent [OIII], R represents
H$\alpha$. The outskirts of M33 are mapped according to a classic LRGB
composition with no [OIII] or H$\alpha$ added. _Roig1 Prades Sky_ is
identified by two perpendicular lines. Top-right: Zoom-in of _Roig1 Prades
Sky_ from the left panel. The HII region _[LHK2017] 318_ is at the top right
corner. Bottom-right: Same as above in grey scale with a box indicating the
size measurement ($0.48\times 1.80$ arcmin).

The top right panel of Figure 1 shows a zoom-in of the discovered nebula. It
has an H$\alpha$ surface brightness of $\sim
23.9\,\rm{mag}\,\rm{arcsec}^{-2}$. The detection of the same feature using two
independent telescopes and several nights rejects the possibility of being an
instrumental artefact.

Furthermore, _Roig1 Prades Sky_ is also visible in the deep image of M33 from
Magrini et al. (2000b)555https://www.ing.iac.es/PR/newsletter/news2/m33.pdf
obtained with the Isaac Newton Telescope. The nebula is outlined as a bright
[OIII] emitter at the top right corner of the second part of their image. This
is a completely independent confirmation of our finding. The goal of these
authors was obtaining a catalogue of planetary nebulae in M33 (Magrini et al.,
2000a); they did not include our nebula among their candidates and it is not
mentioned in the paper. Indeed, _Roig1 Prades Sky_ is much more extended than
the targets they studied. Galera-Rosillo et al. (2018) also searched for
planetary nebulae at the outskirts of M33, but they do not cover our region of
interest.

The centre of the brightest part of this arch-shaped nebula is at
$\alpha=1^{h}34^{m}25^{s}$, $\delta=+30^{\circ}20^{\prime}17^{\prime\prime}$
(ICRS). It has an angular size of $(0.48\times 1.80)\pm 0.12\,$arcmin (bottom
right panel of Figure 1). Assuming a distance to M33 of $840\pm 11\,$kpc
(Breuval et al., 2023), this translates into a physical size of $(120\times
440)\pm 30\,$pc. We detect two fainter filaments that extend even further
towards the southeast direction.

_Roig1 Prades Sky_ might be the brightest part of a spherical shell. In that
case, a lower limit of the diameter of this bubble would be $\sim 470\pm 30$
pc. When including the two filaments extending southeast, this shell almost
doubles its size. The angular distance between the nebula and the M33 centre
is $21\,$arcmin. This translates to $\sim 5\,$kpc at the aforementioned
distance without considering the inclination of the M33 disc.

The southern part of _Roig1 Prades Sky_ coincides with the Wolf-Rayet (WR)
star _[NM2011]-J013425.11+301950.3_ (_Gaia-DR3-303312480229584384_). However,
it does not seem to be at the centre of our nebula. Moreover, _Roig1 Prades
Sky_ is much bigger than previous WR nebulae found in M33 (typical diameters
of a few tens of parsecs according to Drissen et al., 1991). These two
properties are the reasons why we think that this structure might not be the
nebula associated with the aforementioned WR. On the other hand, the high
emission of excited oxygen makes us consider _Roig1 Prades Sky_ as a possible
supernova remnant.

The real nature of _Roig1 Prades Sky_ is unknown. We plan to perform new
observations and obtain deep spectroscopic data to determine its
morphological, chemical and kinematic properties.

##  4 Acknowledgements

We acknowledge Dr. Jorge García-Rojas, Dr. Ángel López-Sánchez, Dr. Ignacio
Trujillo, Dr. Kike Herrero and Josep M. Drudis for their helpful advice and
comments. AR thanks Georgina Serven for her help during this project. RIS
acknowledges governments of Spain and Aragón, FITE, and Science Ministry
(PGC2018-097585-B-C21, PID2021-124918NA-C43). JA acknowledges the funding by
Spanish MCIN/AEI/10.13039/501100011033 and by ESF+ through the PRE2021-100596
grant.

## References

  * Akhlaghi & Ichikawa (2015) Akhlaghi, M., & Ichikawa, T. 2015, ApJS, 220, 1, doi: 10.1088/0067-0049/220/1/1
  * Breuval et al. (2023) Breuval, L., Riess, A. G., Macri, L. M., et al. 2023, ApJ, 951, 118, doi: 10.3847/1538-4357/acd3f4
  * Cenarro et al. (2019) Cenarro, A. J., Moles, M., Cristóbal-Hornillos, D., et al. 2019, A&A, 622, A176, doi: 10.1051/0004-6361/201833036
  * Drissen et al. (1991) Drissen, L., Shara, M. M., & Moffat, A. F. J. 1991, AJ, 101, 1659, doi: 10.1086/115795
  * Galera-Rosillo et al. (2018) Galera-Rosillo, R., Corradi, R. L. M., & Mampaso, A. 2018, A&A, 612, A35, doi: 10.1051/0004-6361/201731383
  * Magrini et al. (2000a) Magrini, L., Corradi, R. L. M., Mampaso, A., & Perinotto, M. 2000a, A&A, 355, 713
  * Magrini et al. (2000b) Magrini, L., Perinotto, M., & Corradi, R. 2000b, The Newsletter of the Isaac Newton Group of Telescopes, 2, 17


# FP8 Formats For Deep Learning

Paulius Micikevicius, Dusan Stosic, Patrick Judd, John Kamalu, Stuart Oberman,
Mohammad Shoeybi,
Michael Siu, Hao Wu
NVIDIA
{pauliusm, dstosic, pjudd, jkamalu, soberman, mshoeybi, msiu<EMAIL_ADDRESS>
Neil Burgess, Sangwon Ha, Richard Grisenthwaite
Arm
{neil.burgess, sangwon.ha<EMAIL_ADDRESS>
Naveen Mellempudi, Marius Cornea, Alexander Heinecke, Pradeep Dubey
Intel
{naveen.k.mellempudi, marius.cornea, alexander.heinecke,
<EMAIL_ADDRESS>

###### Abstract

FP8 is a natural progression for accelerating deep learning training inference
beyond the 16-bit formats common in modern processors. In this paper we
propose an 8-bit floating point (FP8) binary interchange format consisting of
two encodings - E4M3 (4-bit exponent and 3-bit mantissa) and E5M2 (5-bit
exponent and 2-bit mantissa). While E5M2 follows IEEE 754 conventions for
representatio of special values, E4M3’s dynamic range is extended by not
representing infinities and having only one mantissa bit-pattern for NaNs. We
demonstrate the efficacy of the FP8 format on a variety of image and language
tasks, effectively matching the result quality achieved by 16-bit training
sessions. Our study covers the main modern neural network architectures -
CNNs, RNNs, and Transformer-based models, leaving all the hyperparameters
unchanged from the 16-bit baseline training sessions. Our training experiments
include large, up to 175B parameter, language models. We also examine FP8
post-training-quantization of language models trained using 16-bit formats
that resisted fixed point int8 quantization.

##  1 Introduction

Continued improvement in state of the art deep learning (DL) results has
required continued increase in neural network model sizes and compute
resources needed to train them. For example, large natural language models
such as GPT-3 [2], Turing-Megatron [18], PaLM [4], and OPT [25] take weeks to
train on thousands of processors. Reduced precision representation of numbers
has been the corner stoen for deep learning training and inference
acceleration. Common floating point types for training include IEEE single
precision, TF32 mode for single precision [19], IEEE half precision [14], and
bfloat16 [9]. While some research publications have taken bit-reduction to the
extreme, i.e. 1-bit binary networks [5, 7, 26, 24, 17], they have not been
successful in maintaining result quality needed for many practical
applications. For inference fixed-point int8 representation is a popular
option. In some cases even int8 inference can encounter challenges in
maintaining the accuracy required for application deployment [1]. Additional
number representations, such as log-format [15, 11], posits, and log with
posit exponent values [8] have been proposed in literature but have not been
adopted in practice because the demonstrated benefits have not been sufficient
to justify new math pipeline hardware designs.

FP8 is a natural progression from 16-bit floating point types, reducing the
compute requirements of neural network training. Furthermore, due to its non-
linear sampling of the real numbers, FP8 can also have advantages for
inference when compared to int8. Wang et al. [22] proposed using 5-bit
exponent format for training neural networks, confirming their methodology on
the convolutional neural networks (CNNs) for image classification on CIFAR-10
and ILSVRC12 datasets. Mellempudi et al. [12] study the 5-bit exponent format
for training on the larger CNNs as well as language translation networks based
on recurrent and transformer blocks. Both papers investigate 16-bit weight
updates as well as stochastic rounding. Use of two FP8 formats, 4- and 5-bit
exponent fields, for training is introduced in [20], studying a wider range of
CNNs as well as speech and language translation models. [20] also investigates
FP8 inference of networks trained in higher precision and introduces returning
of batch normalization statistics to improve result accuracy. Noune et al [16]
propose a modified FP8 representation that dedicates a single encoding to
special values in order to increase the represented dynamic range and present
an extensive study of exponent bias effect on result quality. 8-bit inference
with various formats, including FP8, with networks trained in higher precision
is the focus of [10].

In this paper we describe an 8-bit binary format for floating point
representation, using two encodings for FP8. Basic principles of using FP8 for
deep learning are summarized in Section 2. In Section 3 we describe the bit
encodings and reasoning behind them. Empirical evaluation of training and
inference with a variety of tasks and models is presented in Section 4. We
show that FP8 training matches FP16 or bfloat16 training results for a variety
of tasks and neural network model architectures and sizes, without changing
any model or optimizer hyperparameters. Our study includes the training of
very large language models, up to 175B parameters. It is important to consider
a wide range of model sizes since it has been shown that models of different
sizes may have different numerical behaviors (for example, the different
behavior of Resnet-18 and Resnet-50 observed in [12].

##  2 Aspects of FP8 Usage in Deep Learning

Some aspects of FP8 usage affect the choices for binary interchange format.
For example, the dynamic ranges required by various networks dictate the need
for the two formats as well as the preference for scaling factor handling in
software rather than via exponent bias. Other aspects, such as type conversion
specifics are orthogonal to the binary format. Both aspects are briefly
reviewed in this section.

It is expected that mathematical operations on FP8 inputs will produce outputs
in higher precision, optionally converting the results to FP8 prior to writing
them to memory. This is common practice today for the 16-bit floating point
formats (FP16 and bfloat16) found on CPUs, GPUs, and TPUs [3, 14]. For
example, matrix-multiplication or dot-product instructions produce single-
precision outputs but less arithmetically-intensive operations are typically
performed after casting the 16-bit inputs to single precision. Thus, FP8
tensors will be generated by converting to FP8 from wider types, such as
single precision floating point.

Higher precision values need to be multiplied with a scaling factor prior to
their casting to FP8 in order to move them into a range that better overlaps
with the representable range of a corresponding FP8 format. This is very
similar to the purpose loss-scaling serves in mixed-precision training with
FP16, where gradients are moved into FP16-representable range [14],[13](slides
13-16). However, some networks require per-tensor scaling factors as FP8
dynamic range is not sufficient to cover the union of all tensors’ important
values (see Section 3.2). Details of the heuristics to select the scaling
factors are beyond the scope of this paper, but the general idea is to choose
a scaling factor such that the maximum magnitude in the tensor becomes close
to the maximum representable magnitude in the corresponding format. Values
that overflow are then saturated to the maximum representable value. Weight
update skipping (and reduction of the scaling factor) on overflows, as used by
FP16 automatic mixed precision training [13], is not a good choice for FP8 as
overflows are much more likely due to the narrower dynamic range, resulting in
too many skipped updates. Values in higher precision get unscaled by
multiplying with the inverse of the scaling factor, either after conversion
from FP8 or after arithmetic instructions for a linear operation have produced
a higher-precision output. In both cases only a minimal amount of additional
arithmetic is required. For matrix multiplications, unscaling is applied once
per dot-product, thus amortized by many multiply-accumulates with FP8 inputs.
Less arithmetic intensive operations (such as nonlinearities, normalizations,
or weight updates by optimizers) are typically memory-bandwidth limited and
not sensitive to one additional arithmetic instruction per value.

While the mechanics of type conversion are orthogonal to the binary format, we
briefly touch on some aspects for completeness of DL uses. Converting a
special value from a wider precision to FP8 results in the corresponding
special value in FP8. For conversions to E4M3 this means that both infinities
and NaNs in the wider type (for example, single precision) turn into NaNs in
FP8. This handling of special values is needed when using mixed precision
training that involves both FP8 and FP16 types, since automatic mixed
precision [13] run-time adjustment of loss-scaling relies on causing and
detecting overflows. In addition, non-saturating mode of conversion can be
provided for usecases that may require a strict handling of overflows.
Rounding mode (round to nearest even, stochastic, etc.) choice is orthogonal
to the interchange format is left up to the implementation, software and
possibly hardware, for maximum flexibility.

##  3 FP8 Binary Interchange Format

FP8 consists of two encodings - E4M3 and E5M2, where the name explicitly
states the number of exponent (E) and mantissa (M) bits. We use the common
term "mantissa" as a synonym for IEEE 754 standard’s trailing significand
field (i.e. bits not including the implied leading 1 bit for normal floating
point numbers). The recommended use of FP8 encodings is E4M3 for weight and
activation tensors, and E5M2 for gradient tensors. While some networks can
train with just the E4M3 or the E5M2 type, there are networks that require
both types (or must maintain many fewer tensors in FP8). This is consistent
with findings in [20, 16], where inference and forward pass of training use a
variant of E4M3, gradients in the backward pass of training use a variant of
E5M2.

FP8 encoding details are specified in Table 1. We use the $S$.E.M notation to
describe binary encodings in the table, where $S$ is the sign bit, E is the
exponent field (either 4 or 5 bits containing biased exponent), M is either a
3- or a 2-bit mantissa. Values with a 2 in the subscript are binary, otherwise
they are decimal.

Table 1: Details of FP8 Binary Formats | E4M3 | E5M2
---|---|---
Exponent bias | 7 | 15
Infinities | N/A | $S.11111.00_{2}$
NaN | $S.1111.111_{2}$ | $S.11111.\\{01,10,11\\}_{2}$
Zeros | $S.0000.000_{2}$ | $S.00000.00_{2}$
Max normal | $S.1111.110_{2}=1.75*2^{8}=448$ | $S.11110.11_{2}=1.75*2^{15}=57,344$
Min normal | $S.0001.000_{2}=2^{-6}$ | $S.00001.00_{2}=2^{-14}$
Max subnorm | $S.0000.111_{2}=0.875*2^{-6}$ | $S.00000.11_{2}=0.75*2^{-14}$
Min subnorm | $S.0000.001_{2}=2^{-9}$ | $S.00000.01_{2}=2^{-16}$

Design of these FP8 format followed the principle of staying consistent with
IEEE-754 conventions, deviating only if a significant benefit is expected for
DL application accuracy. Consequently, the E5M2 format follows the IEEE 754
conventions for exponent and special values and can be viewed as IEEE half
precision with fewer mantissa bits (similar to how bfloat16 and TF32 can be
viewed as IEEE single precision with fewer bits). This allows for
straightforward conversion between E5M2 and IEEE FP16 formats. By contrast,
the dynamic range of E4M3 is extended by reclaiming most of the bit patterns
used for special values because in this case the greater range achieved is
much more useful than supporting multiple encodings for the special values.

###  3.1 Special value representations

We extend the narrow dynamic range of the E4M3 format by representing fewer
special values, adopting their bit patterns for normal values. Infinities are
not represented (see Section 2 for overflow handling details) and we retain
only one mantissa bit-pattern for NaNs. This modification extents the dynamic
range by one extra power of 2, from 17 to 18 binades. We gain the
representation of seven more magnitudes (256, 288, 320, 352, 384, 416, 448),
corresponding to the biased exponent value $1111_{2}$. The maximum
representable magnitude without this modification would be 240. For
consistency with IEEE 754 conventions we retain positive and negative
representations for zero and NaN. While we could gain one additional
representable magnitude, 480, by having just one encoding for zero and one for
NaN, this would require breaking the symmetry of positive and negative
representations inherent in the IEEE 754 formats, complicating or invalidating
algorithm implementations that rely on this property. For example, IEEE
floating point formats allow comparison and sorting of floating point values
using integer operations. The benefit of increasing the maximum value from 448
to 480 for DL is not significant to warrant deviating from IEEE convention and
losing software implementations that rely on it.

As mentioned earlier, E5M2 represents all the special values (infinities,
NaNs, and zeros) consistently with IEEE conventions. Our extensive empirical
studies (Section 4) indicate that 5 bits of exponent provide sufficient per
tensor dynamic range (32 binades, including the subnormal values) for DL.
Furthermore, the benefit of having fewer representations of special values
would be much smaller for E5M2 than it was for E4M3 - only 3 additional
magnitude values would be added due to the smaller mantissa, one additional
binade is much less impactful when E5M2 already provides 32 (compared to
E4M3’s 17 without the adjustment).

###  3.2 Exponent bias

Both E4M3 and E5M2 retain IEEE-like exponent biases: 7 and 15 for E4M3 and
E5M2, respectively. Exponent bias controls the placement of representable
range on the real number line. The same effect is achieved when maintaining a
scale factor per tensor. Our experiments indicate that there are neural
networks that cannot use the same exponent bias for all tensors of a given
type, requiring a per-tensor adjustment. One such example is discussed in
Section 4.3. Consequently, we chose to not deviate from the IEEE convention
for exponent bias. Leaving the per-tensor scaling to software implementation
enables more flexibility than is possible with a programmable exponent bias
approach - the scaling factor can take on any real value (typically
represented in higher precision), while programmable bias is equivalent to
allowing only powers of 2 as scaling factors.

##  4 Empirical Results

Training experiments were carried out with simulated FP8 - tensor values were
clipped to only those that could be represented in FP8 (including the scaling
factor application and saturation). For example, prior to matrix
multiplication for a fully-connected layer, both the incoming activations and
the weight tensors were converted to FP8 and back to the wider representation
(either FP16 or bfloat16). Arithmetic was carried out using the wider
representation for two reasons: the interchange format is the focus of this
paper since different processors may choose different vector- and matrix-
instruction implementations, emulation of arithmetic not supported in hardware
would be prohibitively slow for training of the large models. Obtaining
results for large models is imperative as previous studies have identified
different numerical behavior for models of different sizes (for example, R18
and R50 in [12]).

###  4.1 Training

In the FP8 training experiments we retain the same model architectures, weight
initializations, and optimizer hyper-parameters as are used for higher-
precision baseline training sessions. Baselines were trained in either FP16 or
bfloat16, which have been shown to match the results of single-precision
training sessions [14, 9]. In this study we focused on the input tensors for
math-intensive operations - convolutions and matrix multiplies, to which we’ll
refer as GEMM-operations as they involve dot-product computations. Thus,
unless otherwise specified, we clip to FP8-representable values the
activation, weight, and activation gradient tensors that are inputs to GEMMs.
Output tensors were left in higher precision as they typically are consumed by
non-GEMM operations, such as a non-linearities or normalizations, and in a
number of cases get fused with the preceding GEMM operation. Moving more
tensors to FP8 is the subject of future study.

Table 2: Image Classification Models, ILSVRC12 Validation Top-1 Accuracy Model | Baseline | FP8
---|---|---
VGG-16 | 71.27 | 71.11
VGG-16 BN | 73.95 | 73.69
Inception v3 | 77.23 | 77.06
DenseNet 121 | 75.59 | 75.33
DenseNet 169 | 76.97 | 76.83
Resnet18 | 70.58 | 70.12
Resnet34 | 73.84 | 73.72
Resnet50 v1.5 | 76.71 | 76.76
Resnet101 v1.5 | 77.51 | 77.48
ResNeXt50 | 77.68 | 77.62
Xception | 79.46 | 79.17
MobileNet v2 | 71.65 | 71.04
DeiT small | 80.08 | 80.02

Results for image classification task are listed in Table 2. All networks were
trained on ImageNet ILSVRC12 dataset, top-1 accuracy was computed on the
validation dataset. All GEMM operations’ inputs were clipped to FP8, including
the first convolution and the last fully-connected layer, which were left in
higher precision by previous studies [12, 22]. DeiT [21] is a Trasnformer-
based architecture, the rest of the models are CNNs. With the exception of
MobileNet v2, accuracy achieved by FP8 training is within run-to-run variation
of higher-precision training (run-to-run variation in achieved accuracy is
observed when running training sessions initialized with different random
seeds). We continue work on recovering the remaining accuracy for MobilNet v2.

Language translation task was tested using both Transformers and LSTM-based
recurrent GNMT neural network. Even though Transformer-based translation
models have superseded RNNs in practice, we include GNMT to more completely
cover model architecture types, as well as a proxy for other tasks that still
use recurrent networks. Models were trained on the WMT 2016 English->German
dataset, evaluated using sacreBLEU on newstest2014 data (higher BLEU scores
are better). Evaluation scores for FP8-trained models are within run-to-run
variation variation bounds when compared to the baseline training sessions.

Table 3: Language Translation Models, English->German, BLEU Scores Model | Baseline | FP8
---|---|---
GNMT | $24.83$ | $24.65$
Transformer Base | $26.87$ | $26.83$
Transformer Large | $28.43$ | $28.35$

Training losses (perplexity, lower is better) for a variety of language models
are listed in Table 4. Transformer models were trained on the Wikipedia
dataset. GPT models were trained on a variant of The Pile dataset [6],
augmented with Common Crawl and Common Crawl-derived datasets, as described in
Section 3 of [18]. As was seen with image networks, training results of the
FP8 sessions is within run-to-run noise of 16-bit training sessions. Note that
175B parameter model perplexity is reported at 75% training, as the bfloat16
baseline run has not yet completed. The FP8 training session has completed and
its loss curve is consistent with successful training as shown in Figure 1. As
with the vision and language translation models, we conclude that FP8 training
results match those of 16-bit training sessions.

Table 4: NLP Models, Perplexity Model | Baseline | FP8
---|---|---
Transformer-XL Base | $22.98$ | $22.99$
Transformer-XL Large | $17.80$ | $17.75$
GPT 126M | $19.14$ | $19.24$
GPT 1.3B | $10.62$ | $10.66$
GPT 5B | $8.94$ | $8.98$
GPT 22B | $7.21$ | $7.24$
GPT 175B | $6.65$ | $6.68$
Figure 1: Training loss (perplexity) curves for various GPT-3 models. x-axis
is normalized number of iterations.

###  4.2 Inference

8-bit inference deployment is greatly simplified by FP8 training, as inference
and training use the same datatypes. This is in contrast to int8 inference
with networks trained in 32- or 16-bit floating point, which require post-
training quantization (PTQ) calibration and sometimes quantization-aware
training (QAT) in order to maintain model accuracy. Furthermore, even with
quantization aware training some int8-quantized models may not completely
recover the accuracy achieved with floating point [1].

We evaluate FP8 post-training quantization of models trained in 16-bit
floating point. Table 5 lists inference accuracies for FP16-trained models
quantized to either int8 or E4M3 for inference. Both quantizations use per-
channel scaling factors for weights, per-tensor scaling factors for
activations, as is common for int8 fixed-point. All input tensors to matrix-
multiply operations (including attention batched matrix multiplies) were
quantized. Max-calibration (choosing the scaling factor so that the maximum
magnitude in a tensor is represented) is used for weights, activation tensors
are calibrated using the best calibration chosen from max, percentile, and MSE
methods. BERT language model evaluation on Stanford Question Answering Dataset
shows that FP8 PTQ maintains accuracy while int8 PTQ leads to a significant
loss of model accuracy. We also tried casting the tensors to FP8 without
applying a scaling factor, which resulted in a significant accuracy loss,
increasing the perplexity to 11.0. Evaluation of GPT models on wikitext103
dataset shows that while FP8 PTQ is much better at retaining model accuracy
compared to int8.

Table 5: Post training quantization of models trained in 16-bit floating point. For F1 metrics higher is better, for perplexity lower is better. Best 8-bit result is bolded. Model | Dataset (metric) | 16-bit FP | int8 | E4M3
---|---|---|---|---
BERT Base | SQuAD v1.1 (F1) | $88.19$ | $76.89$ | 88.09
BERT Large | SQuAD v1.1 (F1) | $90.87$ | $89.65$ | 90.94
GPT3 126M | wikitext103 (perplexity) | $19.01$ | $28.37$ | 19.43
GPT3 1.3B | wikitext103 (perplexity) | $10.19$ | $12.74$ | 10.29
GPT3 6.7B | wikitext103 (perplexity) | $8.51$ | $10.29$ | 8.41
Figure 2: 1.3B GPT3 perplexity when bfloat16-trained model weight and
activation input tensors are cast to E4M3 format with various exponent biases,
no per-tensor scaling.

###  4.3 Per-tensor scaling factors

While training and inference for a number of networks can be successfully
carried out in FP8 with the same scaling factor for all the tensors of the
same type (in other words, choosing a single exponent bias could be possible),
there are cases where per-tensor scaling factors are needed to maintain
accuracy. This need becomes more pronounced when we store more of the tensors
in FP8, not just inputs for GEMM operations. Figure 2 shows the FP8 inference
perplexity (measured on wikitext103 dataset), when using post-training-
quantization of a bfloat16-trained network. No calibration was done, weight
and activation tensors were cast from bfloat16 to E4M3 type with the
corresponding exponent bias. As we can see, when casting to FP8 only the
inputs to GEMM operations (both weighted GEMMs as wells as two attention
batched matrix multiplies that involve only activations), several exponent
bias choices in the $[7,10]$ range lead to results matching the bfloat16
baseline. However, if we also quantize to FP8 the residual connections (input
tensors for the Add operations, which further reduces pressure on both storage
and memory bandwidth) then no single exponent bias value leads to sufficient
accuracy - even exponent bias of 7 results in perplexity of 12.59 which is
significantly higher (worse) than 10.19 for the bfloat16 baseline. However, if
instead we calibrate the tensors to have their own scaling factors (following
the convention of int8 quantization to use per-channel and per-tensor scaling
factors for weights and activations, respectively [23]) we achieve 10.29 and
10.44 perplexities for GEMM-only and GEMM+residuals FP8 inference.

##  5 Conclusions

In this paper we propose an FP8 binary interchange format, consisting of E4M3
and E5M2 encodings. By minimally deviating from IEEE-754 conventions for
binary encoding of floating point values, we ensure that that software
implementations can continue to rely on such IEEE FP properties as ability to
compare and sort values using integer operations. The primary motivator for
the format is acceleration of Deep Learning training and inference, by
enabling smaller and more power efficient math pipelines as well as reducing
memory bandwidth pressure. We demonstrate that a wide variety of neural
network models for image and language tasks can be trained in FP8 to match
model accuracy achieved with 16-bit training sessions, using the same model,
optimizer, and training hyperparameters. Using FP8 not only accelerates and
reduces resources required to train, but also simplifies 8-bit inference
deployment by using the same datatypes for training and inference. Prior to
FP8 8-bit inference required calibrating or fine-tuning for int8 models
trained in floating point, which added complexity to the deployment process
and in some cases failed to maintain accuracy.

## References

  * [1] Michael J. Anderson, Benny Chen, Stephen Chen, Summer Deng, Jordan Fix, Michael Gschwind, Aravind Kalaiah, Changkyu Kim, Jaewon Lee, Jason Liang, Haixin Liu, Yinghai Lu, Jack Montgomery, Arun Moorthy, Nadathur Satish, Sam Naghshineh, Avinash Nayak, Jongsoo Park, Chris Petersen, Martin Schatz, Narayanan Sundaram, Bangsheng Tang, Peter Tang, Amy Yang, Jiecao Yu, Hector Yuen, Ying Zhang, Aravind Anbudurai, Vandana Balan, Harsha Bojja, Joe Boyd, Matthew Breitbach, Claudio Caldato, Anna Calvo, Garret Catron, Sneh Chandwani, Panos Christeas, Brad Cottel, Brian Coutinho, Arun Dalli, Abhishek Dhanotia, Oniel Duncan, Roman Dzhabarov, Simon Elmir, Chunli Fu, Wenyin Fu, Michael Fulthorp, Adi Gangidi, Nick Gibson, Sean Gordon, Beatriz Padilla Hernandez, Daniel Ho, Yu-Cheng Huang, Olof Johansson, Shishir Juluri, and et al.  First-generation inference accelerator deployment at facebook.  arxiv, abs/2107.04140, 2021.
  * [2] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners.  In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc., 2020.
  * [3] Neil Burgess, Jelena Milanovic, Nigel Stephens, Konstantinos Monachopoulos, and David Mansell.  Bfloat16 processing for neural networks.  In Martin Langhammer Sylvie Boldo, editor, 26th IEEE Symposium on Computer Arithmetic, ARITH 2019, Kyoto, Japan, June 10-12, 2019, pages 88–91. IEEE, 2017.
  * [4] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel.  Palm: Scaling language modeling with pathways, 2022.
  * [5] Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David.  Binaryconnect: Training deep neural networks with binary weights during propagations.  Advances in neural information processing systems, 28, 2015.
  * [6] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy.  The Pile: An 800gb dataset of diverse text for language modeling.  arXiv preprint arXiv:2101.00027, 2020.
  * [7] Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio.  Binarized neural networks.  Advances in neural information processing systems, 29, 2016.
  * [8] Jeff Johnson.  Rethinking floating point for deep learning.  CoRR, abs/1811.01721, 2018.
  * [9] Dhiraj D. Kalamkar, Dheevatsa Mudigere, Naveen Mellempudi, Dipankar Das, Kunal Banerjee, Sasikanth Avancha, Dharma Teja Vooturi, Nataraj Jammalamadaka, Jianyu Huang, Hector Yuen, Jiyan Yang, Jongsoo Park, Alexander Heinecke, Evangelos Georganas, Sudarshan Srinivasan, Abhisek Kundu, Misha Smelyanskiy, Bharat Kaul, and Pradeep Dubey.  A study of BFLOAT16 for deep learning training.  arxiv, abs/1905.12322, 2019.
  * [10] Andrey Kuzmin, Mart Van Baalen, Yuwei Ren, Markus Nagel, Jorn Peters, and Tijmen Blankevoort.  FP8 quantization: The power of the exponent.  arXiv, 2208.09225, 2022.
  * [11] Edward H. Lee, Daisuke Miyashita, Elaina Chai, Boris Murmann, and S. Simon Wong.  Lognet: Energy-efficient neural networks using logarithmic computation.  In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2017, New Orleans, LA, USA, March 5-9, 2017, pages 5900–5904. IEEE, 2017.
  * [12] Naveen Mellempudi, Sudarshan Srinivasan, Dipankar Das, and Bharat Kaul.  Mixed precision training with 8-bit floating point, 2019.
  * [13] Paulius Micikevicius.  Mixed precision training: theory and practice, 2018.  https://on-demand.gputechconf.com/gtc/2018/presentation/s8923-training-neural-networks-with-mixed-precision-theory-and-practice.pdf, Accessed on 2022-09-11.
  * [14] Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory F. Diamos, Erich Elsen, David García, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu.  Mixed precision training.  arxiv, 1710.03740, 2017.
  * [15] Daisuke Miyashita, Edward H. Lee, and Boris Murmann.  Convolutional neural networks using logarithmic data representation.  CoRR, abs/1603.01025, 2016.
  * [16] Badreddine Noune, Philip Jones, Daniel Justus, Dominic Masters, and Carlo Luschi.  8-bit numerical formats for deep neural networks.  arXiv preprint arXiv:2206.02915, 2022.
  * [17] Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi.  Xnor-net: Imagenet classification using binary convolutional neural networks.  In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision – ECCV 2016, pages 525–542, Cham, 2016. Springer International Publishing.
  * [18] Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, Elton Zheng, Rewon Child, Reza Yazdani Aminabadi, Julie Bernauer, Xia Song, Mohammad Shoeybi, Yuxiong He, Michael Houston, Saurabh Tiwary, and Bryan Catanzaro.  Using deepspeed and megatron to train megatron-turing NLG 530b, A large-scale generative language model.  arXiv, 2201.11990, 2022.
  * [19] Dusan Stosic and Paulius Micikevicius.  Accelerating ai training with nvidia tf32 tensor cores, 2021.  https://developer.nvidia.com/blog/accelerating-ai-training-with-tf32-tensor-cores/, Accessed on 2022-09-4.
  * [20] Xiao Sun, Jungwook Choi, Chia-Yu Chen, Naigang Wang, Swagath Venkataramani, Vijayalakshmi (Viji) Srinivasan, Xiaodong Cui, Wei Zhang, and Kailash Gopalakrishnan.  Hybrid 8-bit floating point (hfp8) training and inference for deep neural networks.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
  * [21] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou.  Training data-efficient image transformers & distillation through attention.  In International Conference on Machine Learning, volume 139, pages 10347–10357, July 2021.
  * [22] Naigang Wang, Jungwook Choi, Daniel Brand, Chia-Yu Chen, and Kailash Gopalakrishnan.  Training deep neural networks with 8-bit floating point numbers.  In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
  * [23] Hao Wu, Patrick Judd, Xiaojie Zhang, Mikhail Isaev, and Paulius Micikevicius.  Integer quantization for deep learning inference: Principles and empirical evaluation.  arxiv, 2004.09602, 2020.
  * [24] Dongqing Zhang, Jiaolong Yang, Dongqiangzi Ye, and Gang Hua.  Lq-nets: Learned quantization for highly accurate and compact deep neural networks.  European conference on computer vision (ECCV), pages 365–382, 2018\.
  * [25] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer.  Opt: Open pre-trained transformer language models, 2022.
  * [26] Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou.  Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients.  arXiv preprint, 1606.06160, 2016.


# DeepInteraction: 3D Object Detection via Modality Interaction

Zeyu Yang1 Jiaqi Chen1 Zhenwei Miao2 Wei Li3 Xiatian Zhu4 Li Zhang1
1Fudan University 2Alibaba DAMO Academy 3NTU 4University of Surrey
https://github.com/fudan-zvg/DeepInteraction Li Zhang<EMAIL_ADDRESS>is the corresponding author with School of Data Science, Fudan University.

###### Abstract

Existing top-performance 3D object detectors typically rely on the multi-modal
fusion strategy. This design is however fundamentally restricted due to
overlooking the modality-specific useful information and finally hampering the
model performance. To address this limitation, in this work we introduce a
novel modality interaction strategy where individual per-modality
representations are learned and maintained throughout for enabling their
unique characteristics to be exploited during object detection. To realize
this proposed strategy, we design a DeepInteraction architecture characterized
by a multi-modal representational interaction encoder and a multi-modal
predictive interaction decoder. Experiments on the large-scale nuScenes
dataset show that our proposed method surpasses all prior arts often by a
large margin. Crucially, our method is ranked at the first position at the
highly competitive nuScenes object detection leaderboard.

##  1 Introduction

3D object detection is critical for autonomous driving by localizing and
recognizing decision-sensitive objects in a 3D world. For reliable object
detection, LiDAR and camera sensors have been simultaneously deployed to
provide point clouds and RGB images for more stronger perception. The two
modalities exhibit naturally strong complementary effects due to their
different perceiving characteristics. Point clouds offer necessary
localization and geometry information at low resolution, whilst images give
rich appearance information at high resolution. Therefore, information fusion
across modalities becomes particularly crucial for strong 3D object detection
performance.

Figure 1:  Schematic strategy comparison. (a) Existing multi-modality fusion
based 3D detection: Fusing individual per-modality representations into a
single hybrid representation and from which the detection results are further
decoded. (b) Our multi-modality interaction based 3D detection: Maintaining
two modality-specific representations throughout the whole pipeline with both
representational interaction in the encoder and predictive interaction in the
decoder.

Existing multi-modal 3D objection detection methods typically adopt a modality
fusion strategy (Figure 1(a)) by combining individual per-modality
representations into a single hybrid representation. For example,
PointPainting vora2020pointpainting  and its variants wang2021pointaugmenting
; Yin2021MVP ; Xu2021FusionPaintingMF  aggregate category scores or semantic
features from the image space into the 3D point cloud space. AutoAlign
chen2022autoalign  and VFF li2022vff  similarly integrate image
representations into the 3D grid space. Latest alternatives li2022deepfusion ;
liu2022bevfusion ; liang2022bevfusion  merge the image and point cloud
features into a joint bird’s-eye view (BEV) representation. This fusion
approach is, however, structurally restricted due to its intrinsic limitation
of potentially dropping off a large fraction of modality-specific
representational strengths due to largely imperfect information fusion into a
unified representation.

To overcome the aforementioned limitations, in this work a novel modality
interaction strategy (Figure 1(b)) for multi-modal 3D object detection is
introduced. Our key idea is that, instead of deriving a fused single
representation, we learn and maintain two modality-specific representations
throughout to enable inter-modality interaction so that both information
exchange and modality-specific strengths can be achieved spontaneously. Our
strategy is implemented by formulating a DeepInteraction architecture. It
starts by mapping 3D point clouds and 2D multi-view images into LiDAR BEV
feature and image perspective feature with two separate feature backbones in
parallel. Subsequently, an encoder interacts the two features for progressive
information exchange and representation learning in a bilateral manner. To
fully exploit per-modality representations, a decoder/head is further designed
to conduct multi-modal predictive interaction in a cascaded manner.

Our contributions are summarized as follows: (i) We propose a novel modality
interaction strategy for multi-modal 3D object detection, with the aim to
resolve a fundamental limitation of previous modality fusion strategy in
dropping the unique perception strengths per modality. (ii) To implement our
proposed strategy, we formulate a DeepInteraction architecture with a multi-
modal representational interaction encoder and a multi-modal predictive
interaction decoder. (iii) Extensive experiments on the nuScenes dataset show
that our DeepInteraction yields new state of the art for multi-modality 3D
object detection and achieves the first position at the highly competitive
nuScenes leaderboard.

##  2 Related work

#### 3D object detection with single modality

Automated driving vehicles are generally equipped with both LiDAR and multiple
surround-view cameras. But many previous methods perform 3D object detection
by exploiting data captured from only a single form of sensor. For camera-
based 3D object detection, since depth information is not directly accessible
from RGB images, some previous works huang2021bevdet ; wang2019pseudo ;
reading2021caddn  lift 2D features into a 3D space by conducting depth
estimation, followed by performing object detection in the 3D space. Another
line of works wang2022detr3d ; liu2022petr ; li2022bevformer ; lu2022ego3rt ;
jiang2022polar  resort to the detection Transformer carion2020detr
architecture. They leverage 3D object queries and 3D-2D correspondence to
incorporate 3D computation into the detection pipelines. Despite the rapid
progress of camera-based approaches, the state-of-the-art of 3D object
detection is still dominated by LiDAR-based methods. Most of LiDAR-based
detectors quantify point clouds into regular grid structures such as voxels
zhou2018voxelnet ; yan2018second , pillars pointpillars  or range images
bewley2020range ; fan2021rangedet ; Chai2021ToTP  before processing them. Due
to the sampling characteristics of LiDAR, these grids are naturally sparse and
hence fit the Transformer design. So a number of approaches mao2021votr ;
fan2021sst  have applied the Transformer for point cloud feature extraction.
Differently, several methods use the Transformer decoder or its variants as
their detection head bai2022transfusion ; wang2021objectdgcnn . 3DETR
misra2021-3detr  adopts a complete Transformer encoder-decoder architecture
with less priori in design. Due to intrinsic limitations with either sensor,
these methods are largely limited in performance.

#### Multi-modality fusion for 3D object detection

Leveraging the perception data from both camera and LiDAR sensors usually
leads to improved performance. This has emerged as a promising direction.
Existing 3D detection methods typically perform multi-modal fusion at one of
the three stages: raw input, intermediate feature, and object proposal. For
example, PointPainting vora2020pointpainting  is the pioneering input fusion
method wang2021pointaugmenting ; huang2020epnet ; Yin2021MVP . The main idea
is to decorate the 3D point clouds with the category scores or semantic
features from the 2D instance segmentation network. Whilst 4D-Net
piergiovanni20214d  placed the fusion module in the point cloud feature
extractor for allowing the point cloud features to dynamically attend to the
image features. ImVoteNet qi2020imvotenet  injects visual information into a
set of 3D seed points abstracted from raw point clouds. The proposal based
fusion methods ku2018joint ; chen2022futr3d  keep the feature extraction of
two modalities independently and aggregate multi-modal features via proposals
or queries at the detection head. The first two categories of methods take a
unilateral fusion strategy with bias to 3D LiDAR modality due to the
superiority of point clouds in distance and spatial perception. Instead, the
last category fully ignores the intrinsic association between the two
modalities in representation. As a result, all above previous methods fail to
fully exploit both modalities, in particular their strong complementary
nature. Besides, a couple of concurrent works have explored fusion of the two
modalities in a shared representation space liu2022bevfusion ;
liang2022bevfusion . They conduct view transformation in the same way
philion2020lift  as in the camera-only approach. This design is however less
effective in exploiting the spatial cues of point clouds during view
transformation, potentially compromising the quality of camera BEV
representation. This gives rise to an extra need of calibrating such
misalignment in network capacity.

In this work, we address the aforementioned limitations in all previous
solutions with a novel multi-modal interaction strategy. The key insight
behind our approach is that we maintain two modality-specific feature
representations and conduct representational and predictive interaction for
maximally exploring their complementary benefits whilst preserving their
respective strengths.

##  3 Method

We present a novel modality interaction framework, dubbed DeepInteraction, for
multi-modal (3D point clouds and 2D multi-camera images) 3D object detection.
In contrast to all prior arts, we learn two representations specific for 3D
LiDAR and 2D image modalities respectively, whilst conducting multi-modal
interaction through both model encoding and decoding. An overview of
DeepInteraction is shown in Figure 1(b). It consists of two main components:
An encoder with multi-modal representational interaction (Section 3.1), and a
decoder with multi-modal predictive interaction (Section 3).

###  3.1 Encoder: Multi-modal representational interaction

Unlike conventional modality fusion strategy that often aggregates multi-modal
inputs into a hybrid feature map, individual per-modality representations are
learned and maintained via multi-modal representational interaction within our
encoder. Specifically, our encoder is formulated as a multi-input-multi-output
(MIMO) structure: Taking as input two modality-specific scene representations
which are independently extracted by LiDAR and image backbones, and producing
two refined representations as output. Overall, it is composed by stacking
multiple encoder layers each with (I) multi-modal representational interaction
(MMRI), (II) intra-modal representational learning (IML), and (III)
representational integration.

Figure 2:  Illustration of the multi-modal representational interactions.
Given two modality-specific representations, the image-to-LiDAR feature
interaction (a) spread the visual signal in the image representation to the
LiDAR BEV representation, and the LiDAR-to-image feature interaction (b) takes
cross-modal relative contexts from LiDAR representation to enhance the image
representations.

#### (I) Multi-modal representational interaction (MMRI)

Each encoder layer takes the representations of two modalities, i.e., the
image perspective representation $\bm{h}_{c}$ and the LiDAR BEV representation
$\bm{h}_{p}$, as input. Our multi-modal representational interaction aims to
exchange the neighboring context in a bilateral cross-modal manner, as shown
in Figure 2. It consists of two steps:

#### (i) Cross-modal correspondence mapping and sampling

To define cross-modality adjacency, we first need to build the pixel-to-
pixel(s) correspondence between the representations ${\bm{h}}_{p}$ and
${\bm{h}}_{c}$. To that end, we construct dense mappings between the image
coordinate frame $c$ and the BEV coordinate frame $p$
($\mathcal{M}_{p\rightarrow c}$ and $\mathcal{M}_{c\rightarrow p}$).

From image to LiDAR BEV coordinate frame $\mathcal{M}_{c\rightarrow
p}:\mathbb{R}^{2}\rightarrow 2^{\mathbb{R}^{2}}$ (Figure 2(a)): We first
project each point $(x,y,z)$ in 3D point cloud to multi-camera images to form
a sparse depth map ${\bm{d}}_{sparse}$, followed by depth completion
ku2018defense  leading to a dense depth map ${\bm{d}}_{dense}$. We further
utilize ${\bm{d}}_{dense}$ to back-project each pixel in the image space into
the 3D point space. This results in the corresponding 3D coordinate $(x,y,z)$,
given an image pixel $(i,j)$ with depth ${\bm{d}}_{dense}^{[i,j]}$. Next,
$(x,y)$ is used to locate the corresponding BEV coordinate $(i_{p},j_{p})$. We
denote the above mapping as $T(i,j)=(i_{p},j_{p})$. We obtain this
correspondence via $(2k+1)\times(2k+1)$ sized neighbor sampling as
$\mathcal{M}_{c\rightarrow p}(i,j)=\left\\{T(i+\Delta i,j+\Delta j)|\Delta
i,~{}\Delta j\in[-k,+k]\right\\}$.

From LiDAR BEV to image coordinate frame $\mathcal{M}_{p\rightarrow
c}:\mathbb{R}^{2}\rightarrow 2^{\mathbb{R}^{2}}$ (Figure 2(b)): Given a
coordinate $(i_{p},j_{p})$ in BEV, we first obtain the LiDAR points
$\left\\{(x,y,z)\right\\}$ within the pillar corresponding to $(i_{p},j_{p})$.
Then we project these 3D points into camera image coordinate frame
$\left\\{(i,j)\right\\}$ according to the camera intrinsics and extrinsics.
This correspondence is obtained as: $\mathcal{M}_{p\rightarrow
c}(i_{p},j_{p})=\left\\{(i,j)\right\\}$.

#### (ii) Attention-based feature interaction

For an image feature point as query $\bm{q}=\bm{h}_{c}^{[i_{c},j_{c}]}$, its
cross-modality neighbors $\mathcal{N}_{q}$, denoted as
$\mathcal{N}_{q}={\bm{h}}_{p}^{\left[\mathcal{M}_{c\rightarrow
p}(i_{c},j_{c})\right]}$, are used as the key $\bm{k}$ and value $\bm{v}$ for
cross-attention learning:

$f_{{\phi}_{c\rightarrow
p}}\left({\bm{h}}_{c},{\bm{h}}_{p}\right)^{[i,j]}=\sum_{\bm{k},\bm{v}\in\mathcal{N}_{q}}\text{softmax}\left(\frac{\bm{qk}}{\sqrt{d}}\right)\bm{v},$
(1)

where ${\bm{h}}^{[i,j]}$ denotes indexing the element at location $(i,j)$ on
the 2D representation ${\bm{h}}$. This is image-to-LiDAR representational
interaction.

The other way around, given a LiDAR BEV feature point as query
$\bm{q}=\bm{h}_{p}^{[i_{p},j_{p}]}$, we similarly obtain its cross-modality
neighbors as $\mathcal{N}_{q}={\bm{h}}_{c}^{\left[\mathcal{M}_{p\rightarrow
c}(i_{p},j_{p})\right]}$. The same process (Eq. (1)) is applied for realizing
LiDAR-to-image representational interaction $f_{{\phi}_{p\rightarrow
c}}\left({\bm{h}}_{c},{\bm{h}}_{p}\right)$.

#### (II) Intra-modal representational learning (IML)

Concurrently, we conduct intra-modal representational learning complementary
to multi-modal interaction. The same local attention as defined in Eq. (1) is
consistently applied. For either modality, we use a $k\times k$ grid
neighborhood as the key and value. Formally, we denote
$f_{{\phi}_{c\rightarrow c}}(\bm{h}_{c})$ for image representation and
$f_{{\phi}_{p\rightarrow p}}(\bm{h}_{p})$ for LiDAR representation.

#### (III) Representational integration

Each layer ends up by integrating the two outputs per modality:

$\displaystyle\bm{h}_{p}^{\prime}$
$\displaystyle=\text{FFN}(\text{Concat}(\text{FFN}(\text{Concat}(\bm{h}_{p}^{p\rightarrow
p},~{}\bm{h}_{p}^{c\rightarrow p})),~{}\bm{h}_{p})),$ (2)
$\displaystyle\bm{h}_{c}^{\prime}$
$\displaystyle=\text{FFN}(\text{Concat}(\text{FFN}(\text{Concat}(\bm{h}_{c}^{c\rightarrow
c},~{}\bm{h}_{c}^{p\rightarrow c})),~{}\bm{h}_{c})),$

where FFN specifies a feed-forward network, and Concat denotes element-wise
concatenation.

###  3.2 Decoder: Multi-modal predictive interaction

Figure 3: Illustration of our multi-modal predictive interaction. Our
predictive interaction decoder (a) generates predictions via (b) progressively
interacting with two modality-specific representations.

Beyond considering the multi-modal interaction at the representation-level, we
further introduce a decoder with multi-modal predictive interaction (MMPI) to
maximize the complementary effects in prediction. As depicted in Figure 3(a),
our core idea is to enhance the 3D object detection of one modality
conditioned on the other modality. In particular, the decoder is built by
stacking multiple multi-modal predictive interaction layers, within which
predictive interactions are formulated in an alternative and progressive
manner. Similar to the decoder of DETR carion2020detr , we cast the 3D object
detection as a set prediction problem. Here, we define a set of $N$ object
queries $\left\\{\bm{Q}_{n}\right\\}^{N}_{n=1}$ and the resulting $N$ object
predictions $\left\\{({\bm{b}}_{n},{\bm{c}}_{n})\right\\}^{N}_{n=1}$, where
${\bm{b}}_{n}$ and ${\bm{c}}_{n}$ denote the predicted bounding box and
category for the $n$-th prediction.

#### Multi-modal predictive interaction layer (MMPI)

For the $l$-th decoding layer, the set prediction is computed by taking the
object queries $\left\\{\bm{Q}_{n}^{(l-1)}\right\\}^{N}_{n=1}$ and the
bounding box predictions $\left\\{\bm{b}_{n}^{(l-1)}\right\\}^{N}_{n=1}$ from
previous layer as inputs and enabling interaction with the intensified image
${\bm{h}}_{p}^{\prime}$ or LiDAR ${\bm{h}}_{c}^{\prime}$ representations
(${\bm{h}}_{c}^{\prime}$ if $l$ is odd, ${\bm{h}}_{p}^{\prime}$ if $l$ is
even). We formulate the multi-modal predictive interaction layer (Figure 3(b))
for specific modality as follows:

#### (I) Multi-modal predictive interaction on image representation (MMPI-
image)

Taking as input 3D object proposals
$\left\\{\bm{b}_{n}^{(l-1)}\right\\}^{N}_{n=1}$ and object queries
$\left\\{\bm{Q}_{n}^{(l-1)}\right\\}^{N}_{n=1}$ generated by the previous
layer, this layer leverages the image representation ${\bm{h}}_{c}^{\prime}$
for further prediction refinement. To integrate the previous predictions
$\left\\{\bm{b}_{n}^{(l-1)}\right\\}^{N}_{n=1}$, we first extract $N$ Region
of Interest (RoI) he2017maskrcnn  features
$\left\\{\bm{R}_{n}\right\\}^{N}_{n=1}$ from the image representation
${\bm{h}}_{c}^{\prime}$, where $\bm{R}_{n}\in\mathbb{R}^{S\times S\times C}$
is the extracted RoI feature for the $n$-th query, $(S\times S)$ is RoI size,
and $C$ is the number of channels of RoI features. Specifically, for each 3D
bounding box, we project it onto image representation ${\bm{h}}_{c}^{\prime}$
to get the 2D convex polygon and take the minimum axis-aligned circumscribed
rectangle. We then design a multi-modal predictive interaction operator that
maps $\left\\{\bm{Q}_{n}^{(l-1)}\right\\}^{N}_{n=1}$ into the parameters of a
series of $1\times 1$ convolutions and then applies them consecutively to
$\left\\{\bm{R}_{n}\right\\}^{N}_{n=1}$; The resulted interactive
representation is further used to obtain the updated object query
$\left\\{\bm{Q}_{n}^{l}\right\\}^{N}_{n=1}$.

#### (II) Multi-modal predictive interaction on LiDAR representation (MMPI-
LiDAR)

This layer shares the same design as the above except that it takes as input
LiDAR representation instead. With regards to the RoI for LiDAR
representation, we project the 3D bounding boxes from previous layer to the
LiDAR BEV representation ${\bm{h}}_{p}^{\prime}$ and take the minimum axis-
aligned rectangle. It is worth mentioning that due to the scale of objects in
autonomous driving scenarios is usually tiny in the BEV coordinate frame, we
enlarge the scale of the 3D bounding box by 2 times. The shape of RoI features
cropped from the LiDAR BEV representation ${\bm{h}}_{p}^{\prime}$ is also set
to be $S\times S\times C$. Here $C$ is the $C$ is the number of channels of
RoI features as well as the height of BEV representation. The multi-modal
predictive interaction layer on LiDAR representation is stacked on the above
image counterpart.

For object detection, a feed-forward network is appended on the
$\left\\{\bm{Q}_{n}^{l}\right\\}^{N}_{n=1}$ for each multi-modal predictive
interaction layer to infer the locations, dimensions, orientations and
velocities. During training, the matching cost and loss function as
bai2022transfusion  are applied.

##  4 Experiments

###  4.1 Experimental setup

#### Dataset

We evaluate our approach on the nuScenes dataset caesar2020nuscenes . It
provides point clouds from 32-beam LiDAR and images with a resolution of
$1600\times 900$ from 6 surrounding cameras. The total of 1000 scenes, where
each sequence is roughly 20 seconds long and annotated every 0.5 second, is
officially split into train/val/test set with 700/150/150 scenes. For the 3D
object detection task, 1.4M objects in scenes are annotated with 3D bounding
boxes and classified into 10 categories: car, truck, bus, trailer,
construction vehicle, pedestrian, motorcycle, bicycle, barrier, and traffic
cone.

#### Metric

Mean average precision (mAP) everingham2010pascal  and nuScenes detection
score (NDS) caesar2020nuscenes  are used as the evaluation metric of 3D
detection performance. The final mAP is computed by averaging over the
distance thresholds of 0.5m, 1m, 2m, 4m across 10 classes. NDS is a weighted
average of mAP and other attribute metrics, including translation, scale,
orientation, velocity, and other box attributes.

Table 1: Comparison with state-of-the-art methods on the nuScenes test set. Metrics: mAP(%), NDS(%). ‘L’ and ‘C’ represent LiDAR and camera, respectively. † denotes test-time augmentation is used. § denotes that test-time augmentation and model ensemble both are applied for testing. Method | Modality | Backbones | validation | test
---|---|---|---|---
Image | LiDAR |  mAP$\uparrow$ |  NDS$\uparrow$ |  mAP$\uparrow$ |  NDS$\uparrow$
BEVDet4D huang2021bevdet  | C | Swin-Base | - | 42.1 | 54.5 | 45.1 | 56.9
BEVFormer li2022bevformer  | C | V99 | - | - | - | 48.1 | 56.9
Ego3RT lu2022ego3rt  | C | V99 | - | 47.8 | 53.4 | 42.5 | 47.9
PolarFormer jiang2022polar  | C | V99 | - | 50.0 | 56.2 | 49.3 | 57.2
CenterPoint yin2021center  | L | - | VoxelNet | 59.6 | 66.8 | 60.3 | 67.3
Focals Conv chen2022focal  | L | - | VoxelNet-FocalsConv | 61.2 | 68.1 | 63.8 | 70.0
Transfusion-L bai2022transfusion  | L | - | VoxelNet | 65.1 | 70.1 | 65.5 | 70.2
LargeKernel3D chen2022scaling  | L | - | VoxelNet-LargeKernel3D | 63.3 | 69.1 | 65.3 | 70.5
FUTR3D chen2022futr3d  | L+C | R101 | VoxelNet | 64.5 | 68.3 | - | -
PointAugmenting wang2021pointaugmenting † | L+C | DLA34 | VoxelNet | - | - | 66.8 | 71.0
MVP Yin2021MVP  | L+C | DLA34 | VoxelNet | 67.1 | 70.8 | 66.4 | 70.5
AutoAlignV2 chen2022autoalign  | L+C | CSPNet | VoxelNet | 67.1 | 71.2 | 68.4 | 72.4
TransFusion bai2022transfusion  | L+C | R50 | VoxelNet | 67.5 | 71.3 | 68.9 | 71.6
BEVFusion liang2022bevfusion  | L+C | Swin-Tiny | VoxelNet | 67.9 | 71.0 | 69.2 | 71.8
BEVFusion liu2022bevfusion  | L+C | Swin-Tiny | VoxelNet | 68.5 | 71.4 | 70.2 | 72.9
DeepInteraction-base | L+C | R50 | VoxelNet | 69.9 | 72.6 | 70.8 | 73.4
Focals Conv-F chen2022focal † | L+C | R50 | VoxelNet-FocalsConv | 67.1 | 71.5 | 70.1 | 73.6
LargeKernel3D-F chen2022scaling † | L+C | R50 | VoxelNet-LargeKernel | - | - | 71.1 | 74.2
DeepInteraction-large† | L+C | Swin-Tiny | VoxelNet | 72.6 | 74.4 | 74.1 | 75.5
BEVFusion-e liu2022bevfusion § | L+C | Swin-Tiny | VoxelNet | 73.7 | 74.9 | 75.0 | 76.1
DeepInteraction-e§ | L+C | Swin-Tiny | VoxelNet | 73.9 | 75.0 | 75.6 | 76.3

###  4.2 Implementation details

Our implementation is based on the public code base mmdetection3d mmdet3d2020
. For the image branch backbone, we use a simple ResNet-50 he2016deep  and
initialize it from the instance segmentation model Cascade Mask R-CNN
cai2019cascade  pretrained on COCO lin2014coco  and then nuImage
caesar2020nuscenes , which is same as Transfusion bai2022transfusion . To save
the computation cost, we rescale the input image to 1/2 of its original size
before feeding into the network, and freeze the weights of image branch during
training. The voxel size is set to $(0.075m,0.075m,0.2m)$, and the detection
range is set to $[-54m,54m]$ for $X$ and $Y$ axis and $[-5m,3m]$ for $Z$ axis.
The representational interaction encoder is composed by stacking two
representational interaction layers. For the multi-modal predictive
interaction decoder, we use 5 cascaded decoder layers. We set the query number
to 200 for training and testing and use the same query initialization method
as Transfusion bai2022transfusion . The above configuration is termed
DeepInteraction-base.

We also adopt another two widely used settings for online submission, i.e.,
test-time augmentation (TTA) and model ensemble. In the following, we refer to
the two settings as DeepInteraction-large and DeepInteraction-e respectively.
In particular, DeepInteraction-large uses Swin-Tiny liu2021Swin  as image
backbone, and doubles the number of channel for each convolution block in
LiDAR backbone. The voxel size of DeepInteraction-large is set to
$[0.5m,0.5m,0.2m]$. Following the common practice, we use double flipping and
rotation with yaw angles $[0^{\circ},\pm 6.25^{\circ},\pm 12.5^{\circ}]$ for
test-time augmentation. DeepInteraction-e ensembles multiple DeepInteraction-
large models with input LiDAR BEV grid size between $[0.5m,0.5m]$ and
$[1.5m,1.5m]$.

For data augmentation, following TransFusion bai2022transfusion  we adopt
random rotation with a range of $r\in\left[-\pi/4,\pi/4\right]$, random
scaling with a factor of $r\in\left[0.9,1.1\right]$, random translation with
standard deviation 0.5 in three axis, and random horizontal flipping. We also
use the class-balanced re-sampling in CBGS zhu2019class  to balance the class
distribution for nuScenes. Following bai2022transfusion , we adopt a two stage
training recipe. We take TransFusion-L bai2022transfusion  as our LiDAR-only
baseline. We use Adam optimizer with one-cycle learning rate policy, with max
learning rate $1\times 10^{-3}$, weight decay 0.01 and momentum 0.85 to 0.95,
following CBGS zhu2019class . Our LiDAR-only baseline is trained for 20 epochs
and LiDAR-image fusion for 6 epochs with batch size of 16 using 8 NVIDIA V100
GPUs.

###  4.3 Comparison to the state of the art

#### Performance

We compare with state-of-the-art alternatives on the nuScenes test set. As
shown in Table 1, DeepInteraction achieves new state-of-the-art performance
under all settings. The base variant without TTA and model ensemble,
DeepInteraction-base, with a simple ResNet-50 image backbone, surpasses all
the prior arts as well as the concurrent work BEVFusion liu2022bevfusion  even
with a Swin-Tiny image backbone. DeepInteraction-large beats the closest rival
LargeKernel3D-F chen2022scaling  with the same TTA and test time augmentation
(single model) by a considerable margin. Our ensemble version
DeepInteraction-e achieves the first rank among all the solutions on the
nuScenes leaderboard. These results verify the performance advantages of our
multi-modal interaction approach. Per-category results are shown in Appendix
A.1. Qualitative results are provided in Figure 4 and Appendix A.4.

Figure 4: Qualitative results on nuScenes val set. In LiDAR BEV (right), green
boxes are the ground-truth and blue boxes are the predictions. Best viewed
when zooming in.

#### Run time

We compare inference speed tested on NVIDIA V100, A6000 GPUs and A100
separately. As shown in Table 2, our method achieves the best performance
while running faster than alternative painting-based wang2021pointaugmenting
and query-based chen2022futr3d  fusion approaches. This validates superior
trade-off between detection performance and inference speed achieved by our
method. As found in wang2021pointaugmenting , feature extraction for multi-
view high resolution camera images contributes the most of overall latency in
a multi-modal 3D detector. Indeed, from Table 3(c) we observe that increasing
the number of decoder layers only brings negligible extra latency, which
concurs with the same conclusion.

###  4.4 Ablations on the decoder

Table 2:  Run time comparison. If not specified, the performance and efficiency are evaluated on the nuScenes val set.  Method |  mAP(%)$\uparrow$ |  NDS(%)$\uparrow$ |  FPS(A100)$\uparrow$ |  FPS(A6000)$\uparrow$ |  FPS(V100)$\uparrow$
---|---|---|---|---|---
PointAugmenting wang2021pointaugmenting  | 66.8 (Test) | 71.0 (Test) | 1.4 | 2.8 | 2.3
FUTR3D chen2022futr3d  | 64.2 | 68.0 | 4.5 | 2.3 | 1.8
Transfusion bai2022transfusion  | 67.5 | 71.3 | 6.2 | 5.5 | 3.8
DeepInteraction | 69.9 | 72.6 | 4.9 | 3.1 | 2.6
Table 3: Ablation studies on the decoder. The mAP and NDS are evaluated on the
nuScenes val set.

LiDAR | Image |  mAP |  NDS
---|---|---|---
DETR carion2020detr  | DETR carion2020detr  | 68.6 | 71.6
DETR carion2020detr  | MMPI | 69.3 | 72.1
MMPI | MMPI | 69.9 | 72.6
(a) The type of decoder layer.

Modality | mAP | NDS
---|---|---
Fully LiDAR | 69.2 | 72.2
LiDAR-image alternating | 69.9 | 72.6
(b) Single vs. multiple representations.

# of decoder layers | mAP | NDS | FPS$\uparrow$
---|---|---|---
1 (LiDAR-only) | 65.1 | 70.1 | 8.7
2 | 69.5 | 72.3 | 2.8
3 | 69.7 | 72.5 | 2.7
4 | 69.8 | 72.5 | 2.7
5 | 69.9 | 72.6 | 2.6
6 | 69.7 | 72.1 | 2.5
(c) The number of decoder layers.

Train |  Inference |  mAP |  NDS
---|---|---|---
| 200 | 69.9 | 72.6
200 | 300 | 70.1 | 72.7
| 400 | 70.0 | 72.6
| 200 | 69.7 | 72.5
300 | 300 | 69.9 | 72.6
| 400 | 70.0 | 72.6
(d) The number of queries.

Multi-modal predictive interaction layer vs. DETR carion2020detr  decoder
layer In Table 3(a) we evaluate the design of decoder layer by comparing our
multi-modal predictive interaction (MMPI) with DETR carion2020detr  decoder
layer. Note the DETR decoder layer means the conventional Transformer deocder
layer is used to aggregate multi-modal information same as in Transfusion
bai2022transfusion . We further test a mixing design: using vanilla DETR
decoder layer for aggregating features in LiDAR representation and our MMPI
for aggregating features in image representation (second row). It is evident
that MMPI is significantly superior over DETR by improving 1.3% mAP and 1.0%
NDS, with combinational flexibility in design.

How many representations/modalities? In Table 3(b), we evaluate the effect of
using different numbers of representations/modalities in decoding. We compare
our MMPI using both representations in an alternating manner with a variant
using LiDAR representation in all decoder layers. It is observed that using
both representations is beneficial, verifying our design consideration.

Number of decoder layers As shown in Table 3(c), increasing the number of
decoder layers up to 5 layers can consistently improve the performance whilst
introducing negligible latency. LiDAR-only denotes the Transfusion-L
bai2022transfusion  baseline.

Number of queries Since our query embeddings are initialized in a non-
parametric and input-dependent manner as in bai2022transfusion , the number of
queries are adjustable during inference. In Table 3(d), we evaluate different
combinations of query numbers for training and test. Overall, the performance
is stable over different choices with 200/300 for training/test as the best
setting.

###  4.5 Ablations on the encoder

Figure 5:  Illustrations of the heatmaps predicted from BEV representations
before and after representational interactions. All samples are from the
nuScenes val split. (a) A case of the occluded tiny object. (b) A case of
small object at long distance. (c) An example of two adjacent barriers which
are connected together in LiDAR point clouds and thus it is difficult to have
their instance-level understanding without the help of visual clues.  Table 4:
Ablation studies on the representational interaction encoder. The mAP and NDS
are evaluated on the nuScenes val set.

IML |  MMRI |  mAP |  NDS
---|---|---|---
✓ |  | 68.1 | 71.9
| ✓ | 69.5 | 72.5
✓ | ✓ | 69.9 | 72.6
(a) Encoder design. IML: Intra-modal learning; MMRI: Multi-modal
representational interaction.

# of encoder layers |  mAP |  NDS
---|---|---
w/o | 66.4 | 70.7
1 | 67.7 | 71.2
2 | 69.9 | 72.6
(b) The number of encoder layers.

Method |  mAP |  NDS
---|---|---
Representational fusion | 67.5 | 71.3
Representational interaction (Ours) | 68.7 | 71.9
(c) Our representational interaction vs. conventional representational fusion
(e.g., Transfusion bai2022transfusion ).

Table 5: Evaluation on different LiDAR backbones. The mAP and NDS are
evaluated on the nuScenes val set.

Methods | Modility | mAP | NDS
---|---|---|---
PointPillars pointpillars  | L | 46.2 | 59.1
+Transfusion-L bai2022transfusion  | L | 54.5 | 62.7
+Transfusion bai2022transfusion  | L+C | 58.3 | 64.5
+DeepInteraction | L+C | 60.0 | 65.6
(a) Comparison between pillar-based methods.

Methods | Modility | mAP | NDS
---|---|---|---
VoxelNet zhou2018voxelnet  | L | 52.6 | 63.0
+Transfusion-L bai2022transfusion  | L | 65.1 | 70.1
+Transfusion bai2022transfusion  | L+C | 67.5 | 71.3
+DeepInteraction | L+C | 69.9 | 72.6
(b) Comparison between voxel-based methods.

Table 6:  Comparison with the LiDAR-only baseline Transfusion-L bai2022transfusion  on nuScenes val split.  Method | mAP | NDS | Car | Truck | C.V. | Bus | T.L. | B.R. | M.T. | Bike | Ped. | T.C.
---|---|---|---|---|---|---|---|---|---|---|---|---
Transfusion-L bai2022transfusion  | 65.1 | 70.2 | 86.5 | 59.6 | 25.4 | 74.4 | 42.2 | 74.1 | 72.1 | 56.0 | 86.6 | 74.1
DeepInteraction | 69.9 | 72.7 | 88.5 | 64.4 | 30.1 | 79.2 | 44.6 | 76.4 | 79.0 | 67.8 | 88.9 | 80.0

Multi-modal representational interaction vs. fusion To precisely demonstrate
the superiority of our multi-modal representational interaction, we compare a
naive version of our DeepInteraction with conventional representational fusion
strategy as presented in Transfusion bai2022transfusion . We limit our
DeepInteraction using the same number of encoder and decoder layers as
bai2022transfusion  for a fair comparison. Table 4(c) shows that our
representational interaction is clearly more effective.

Encoder design We ablate the design of our encoder with focus on multi-modal
representational interaction (MMRI) and intra-modal representational learning
(IML). We have a couple of observations from Table 4(a): (1) Our MMRI can
significantly improve the performance over IML; (2) MMRI and IML can work well
together for further performance gain. As seen from Table 4(b), stacking our
encoder layers for iterative MMRI is beneficial.

Qualitative results of representational interaction To gain more insight about
the effect of our multi-modal representational interaction (MMRI), we
visualize the heatmaps of challenging cases. We observe from Figure 5 that
without the assistance of MMRI, some objects cannot be detected when using
LiDAR only (the middle column). The locations of these objects are highlighted
by red circles in the heatmap and white bounding boxes in the RGB image below.

Concretely, the sample (a) suggests that camera information is helpful to
recover partially obscured tiny objects with sparse observation in the point
cloud. The sample (b) shows a representative case where some distant objects
can be recognized successfully due to the help of visual information. From the
sample (c), we observe that the centers of some barriers yield a more distinct
activation in the heatmap after representational interaction. This is probably
due to that it is too difficult to locate the boundaries of several
consecutive barriers from LiDAR point clouds only.

###  4.6 Ablation on LiDAR backbones

We examine the generality of our framework with two different LiDAR backbones:
PointPillars pointpillars  and VoxelNet zhou2018voxelnet . For PointPillars,
we set the voxel size to (0.2m, 0.2m) while keeping the remaining settings
same as DeepInteraction-base. For fair comparison, we use the same number of
queries as TransFusion bai2022transfusion . As shown in Table 5, due to the
proposed multi-modal interaction strategy, DeepInteraction exhibits consistent
improvements over LiDAR-only baseline using either backbone (by 5.5% mAP for
voxel-based backbone, and 4.4% mAP for pillar-based backbone). This manifests
the generality of our DeepInteraction across varying point cloud encoder.

###  4.7 Performance breakdown

To demonstrate more fine-grained performance analysis, we compare our
DeepInteraction and our LiDAR-only baseline Transfusion bai2022transfusion  at
the category level in terms of mAP on nuScenes val set. We can see from Table
6 that our fusion approach achieves remarkable improvements on all the
categories, especially on tiny or rare object categories (+11.8% mAP for
bicycle, +6.9% mAP for motorcycle, and +5.9% mAP for traffic cone).

##  5 Conclusion

In this work, we have presented a novel 3D object detection method
DeepInteraction for exploring the intrinsic multi-modal complementary nature.
This key idea is to maintain two modality-specific representations and
establish interactions between them for both representation learning and
predictive decoding. This strategy is designed particularly to resolve the
fundamental limitation of existing unilateral fusion approaches that image
representation are insufficiently exploited due to their auxiliary-source role
treatment. Extensive experiments demonstrate our proposed DeepInteraction
yields new state of the art on the nuScenes benchmark dataset and achieves the
first position at the highly competitive nuScenes 3D object detection
leaderboard.

#### Acknowledgement

This work was supported in part by National Natural Science Foundation of
China (Grant No. 6210020439), Lingang Laboratory (Grant No. LG-QS-202202-07),
Natural Science Foundation of Shanghai (Grant No. 22ZR1407500), Science and
Technology Innovation 2030 - Brain Science and Brain-Inspired Intelligence
Project (Grant No. 2021ZD0200204) and Shanghai Municipal Science and
Technology Major Project (Grant No. 2018SHZDZX01).

## References

  * (1) Xuyang Bai, Zeyu Hu, Xinge Zhu, Qingqiu Huang, Yilun Chen, Hongbo Fu, and Chiew-Lan Tai.  Transfusion: Robust lidar-camera fusion for 3d object detection with transformers.  In CVPR, 2022.
  * (2) Alex Bewley, Pei Sun, Thomas Mensink, Dragomir Anguelov, and Cristian Sminchisescu.  Range conditioned dilated convolutions for scale invariant 3d object detection.  arXiv preprint, 2020.
  * (3) Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom.  nuscenes: A multimodal dataset for autonomous driving.  In CVPR, 2020.
  * (4) Zhaowei Cai and Nuno Vasconcelos.  Cascade r-cnn: high quality object detection and instance segmentation.  IEEE TPAMI, 2019.
  * (5) Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In ECCV, 2020.
  * (6) Yuning Chai, Pei Sun, Jiquan Ngiam, Weiyue Wang, Benjamin Caine, Vijay Vasudevan, Xiao Zhang, and Drago Anguelov.  To the point: Efficient 3d object detection in the range image with graph convolution kernels.  In CVPR, 2021.
  * (7) Xuanyao Chen, Tianyuan Zhang, Yue Wang, Yilun Wang, and Hang Zhao.  Futr3d: A unified sensor fusion framework for 3d detection.  arXiv preprint, 2022.
  * (8) Yukang Chen, Yanwei Li, Xiangyu Zhang, Jian Sun, and Jiaya Jia.  Focal sparse convolutional networks for 3d object detection.  arXiv preprint, 2022.
  * (9) Yukang Chen, Jianhui Liu, Xiaojuan Qi, Xiangyu Zhang, Jian Sun, and Jiaya Jia.  Scaling up kernels in 3d cnns.  arXiv preprint, 2022.
  * (10) Zehui Chen, Zhenyu Li, Shiquan Zhang, Liangji Fang, Qinghong Jiang, Feng Zhao, Bolei Zhou, and Hang Zhao.  Autoalign: Pixel-instance feature aggregation for multi-modal 3d object detection.  arXiv preprint, 2022.
  * (11) MMDetection3D Contributors.  MMDetection3D: OpenMMLab next-generation platform for general 3D object detection.  https://github.com/open-mmlab/mmdetection3d, 2020.
  * (12) Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman.  The pascal visual object classes (voc) challenge.  IJCV, 2010.
  * (13) Lue Fan, Ziqi Pang, Tianyuan Zhang, Yu-Xiong Wang, Hang Zhao, Feng Wang, Naiyan Wang, and Zhaoxiang Zhang.  Embracing single stride 3d object detector with sparse transformer.  arXiv preprint, 2021.
  * (14) Lue Fan, Xuan Xiong, Feng Wang, Naiyan Wang, and ZhaoXiang Zhang.  Rangedet: In defense of range view for lidar-based 3d object detection.  In ICCV, 2021.
  * (15) Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In ICCV, 2017.
  * (16) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, 2016.
  * (17) Junjie Huang, Guan Huang, Zheng Zhu, and Dalong Du.  Bevdet: High-performance multi-camera 3d object detection in bird-eye-view.  arXiv preprint, 2021.
  * (18) Tengteng Huang, Zhe Liu, Xiwu Chen, and Xiang Bai.  Epnet: Enhancing point features with image semantics for 3d object detection.  In ECCV, 2020.
  * (19) Yanqin Jiang, Li Zhang, Zhenwei Miao, Xiatian Zhu, Jin Gao, Weiming Hu, and Yu-Gang Jiang.  Polarformer: Multi-camera 3d object detection with polar transformers.  arXiv preprint, 2022.
  * (20) Jason Ku, Ali Harakeh, and Steven L Waslander.  In defense of classical image processing: Fast depth completion on the cpu.  In CRV, 2018.
  * (21) Jason Ku, Melissa Mozifian, Jungwook Lee, Ali Harakeh, and Steven L Waslander.  Joint 3d proposal generation and object detection from view aggregation.  In IROS, 2018.
  * (22) Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom.  Pointpillars: Fast encoders for object detection from point clouds.  In CVPR, 2019.
  * (23) Yanwei Li, Xiaojuan Qi, Yukang Chen, Liwei Wang, Zeming Li, Jian Sun, and Jiaya Jia.  Voxel field fusion for 3d object detection.  In CVPR, 2022.
  * (24) Yingwei Li, Adams Wei Yu, Tianjian Meng, Ben Caine, Jiquan Ngiam, Daiyi Peng, Junyang Shen, Bo Wu, Yifeng Lu, Denny Zhou, et al.  Deepfusion: Lidar-camera deep fusion for multi-modal 3d object detection.  arXiv preprint, 2022.
  * (25) Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Qiao Yu, and Jifeng Dai.  Bevformer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers.  arXiv preprint, 2022.
  * (26) Tingting Liang, Hongwei Xie, Kaicheng Yu, Zhongyu Xia, Zhiwei Lin, Yongtao Wang, Tao Tang, Bing Wang, and Zhi Tang.  BEVFusion: A Simple and Robust LiDAR-Camera Fusion Framework.  arXiv preprint, 2022.
  * (27) T. Y. Lin, M. Maire, S. Belongie, J. Hays, and C. L. Zitnick.  Microsoft coco: Common objects in context.  In ECCV, 2014.
  * (28) Yingfei Liu, Tiancai Wang, Xiangyu Zhang, and Jian Sun.  Petr: Position embedding transformation for multi-view 3d object detection.  arXiv preprint, 2022.
  * (29) Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In ICCV, 2021.
  * (30) Zhijian Liu, Haotian Tang, Alexander Amini, Xingyu Yang, Huizi Mao, Daniela Rus, and Song Han.  Bevfusion: Multi-task multi-sensor fusion with unified bird’s-eye view representation.  arXiv preprint, 2022.
  * (31) Jiachen Lu, Zheyuan Zhou, Xiatian Zhu, Hang Xu, and Li Zhang.  Learning ego 3d representation as ray tracing.  In ECCV, 2022.
  * (32) Jiageng Mao, Yujing Xue, Minzhe Niu, Haoyue Bai, Jiashi Feng, Xiaodan Liang, Hang Xu, and Chunjing Xu.  Voxel transformer for 3d object detection.  In CVPR, 2021.
  * (33) Ishan Misra, Rohit Girdhar, and Armand Joulin.  An End-to-End Transformer Model for 3D Object Detection.  In ICCV, 2021.
  * (34) Jonah Philion and Sanja Fidler.  Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d.  In ECCV, 2020.
  * (35) AJ Piergiovanni, Vincent Casser, Michael S Ryoo, and Anelia Angelova.  4d-net for learned multi-modal alignment.  In CVPR, 2021.
  * (36) Charles R Qi, Xinlei Chen, Or Litany, and Leonidas J Guibas.  Imvotenet: Boosting 3d object detection in point clouds with image votes.  In CVPR, 2020.
  * (37) Cody Reading, Ali Harakeh, Julia Chae, and Steven L. Waslander.  Categorical depth distributionnetwork for monocular 3d object detection.  In CVPR, 2021.
  * (38) Sourabh Vora, Alex H Lang, Bassam Helou, and Oscar Beijbom.  Pointpainting: Sequential fusion for 3d object detection.  In CVPR, 2020.
  * (39) Chunwei Wang, Chao Ma, Ming Zhu, and Xiaokang Yang.  Pointaugmenting: Cross-modal augmentation for 3d object detection.  In CVPR, 2021.
  * (40) Yan Wang, Wei-Lun Chao, Divyansh Garg, Bharath Hariharan, Mark Campbell, and Kilian Weinberger.  Pseudo-lidar from visual depth estimation: Bridging the gap in 3d object detection for autonomous driving.  In CVPR, 2019.
  * (41) Yue Wang, Vitor Campagnolo Guizilini, Tianyuan Zhang, Yilun Wang, Hang Zhao, and Justin Solomon.  Detr3d: 3d object detection from multi-view images via 3d-to-2d queries.  In CoRL, 2022.
  * (42) Yue Wang and Justin M. Solomon.  Object dgcnn: 3d object detection using dynamic graphs.  In NeurIPS, 2021.
  * (43) Shaoqing Xu, Dingfu Zhou, Jin Fang, Junbo Yin, Bin Zhou, and Liangjun Zhang.  FusionPainting: Multimodal fusion with adaptive attention for 3d object detection.  ITSC, 2021.
  * (44) Yan Yan, Yuxing Mao, and Bo Li.  Second: Sparsely embedded convolutional detection.  Sensors, 2018.
  * (45) Tianwei Yin, Xingyi Zhou, and Philipp Krahenbuhl.  Center-based 3d object detection and tracking.  In CVPR, 2021.
  * (46) Tianwei Yin, Xingyi Zhou, and Philipp Krähenbühl.  Multimodal virtual point 3d detection.  NeurIPS, 2021.
  * (47) Yin Zhou and Oncel Tuzel.  Voxelnet: End-to-end learning for point cloud based 3d object detection.  In CVPR, 2018.
  * (48) Benjin Zhu, Zhengkai Jiang, Xiangxin Zhou, Zeming Li, and Gang Yu.  Class-balanced grouping and sampling for point cloud 3d object detection.  arXiv preprint, 2019.

## Checklist

  1. 1.

For all authors…

    1. (a)

Do the main claims made in the abstract and introduction accurately reflect
the paper’s contributions and scope? [Yes] See Section 4.

    2. (b)

Did you describe the limitations of your work? [Yes] See supplementary
material.

    3. (c)

Did you discuss any potential negative societal impacts of your work? [Yes]
See supplementary material.

    4. (d)

Have you read the ethics review guidelines and ensured that your paper
conforms to them? [Yes]

  2. 2.

If you are including theoretical results…

    1. (a)

Did you state the full set of assumptions of all theoretical results? [N/A]

    2. (b)

Did you include complete proofs of all theoretical results? [N/A]

  3. 3.

If you ran experiments…

    1. (a)

Did you include the code, data, and instructions needed to reproduce the main
experimental results (either in the supplemental material or as a URL)? [N/A]

    2. (b)

Did you specify all the training details (e.g., data splits, hyperparameters,
how they were chosen)? [Yes] See Section 4.

    3. (c)

Did you report error bars (e.g., with respect to the random seed after running
experiments multiple times)? [N/A]

    4. (d)

Did you include the total amount of compute and the type of resources used
(e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section
4.

  4. 4.

If you are using existing assets (e.g., code, data, models) or
curating/releasing new assets…

    1. (a)

If your work uses existing assets, did you cite the creators? [Yes]

    2. (b)

Did you mention the license of the assets? [N/A]

    3. (c)

Did you include any new assets either in the supplemental material or as a
URL? [No]

    4. (d)

Did you discuss whether and how consent was obtained from people whose data
you’re using/curating? [N/A]

    5. (e)

Did you discuss whether the data you are using/curating contains personally
identifiable information or offensive content? [N/A]

  5. 5.

If you used crowdsourcing or conducted research with human subjects…

    1. (a)

Did you include the full text of instructions given to participants and
screenshots, if applicable? [N/A]

    2. (b)

Did you describe any potential participant risks, with links to Institutional
Review Board (IRB) approvals, if applicable? [N/A]

    3. (c)

Did you include the estimated hourly wage paid to participants and the total
amount spent on participant compensation? [N/A]

##  Appendix A Appendix

###  A.1 Performance breakdown for categories

In table 7, we report the performance on each category. It is shown that our
DeepInteraction performs the best among all the competitors across the most of
object categories.

Table 7: Comparison with state-of-the-art methods on the nuScenes test set. Metrics: mAP(%)$\uparrow$, NDS(%)$\uparrow$, and AP(%)$\uparrow$ for each category. ‘C.V.’, ‘Ped.’, and ‘T.C.’,‘M.T.’ and ‘T.L.’ are short for construction vehicle, pedestrian, traffic cone, motor and trailer respectively. ‘L’ and ‘C’ represent LiDAR and camera, respectively. † denotes test-time augmentation is used. § denotes that test-time augmentation and model ensemble both are applied for testing. Method | Mod. | mAP | NDS | Car | Truck | C.V. | Bus | T.L. | B.R. | M.T. | Bike | Ped. | T.C.
---|---|---|---|---|---|---|---|---|---|---|---|---|---
CenterPoint yin2021center † | L | 60.3 | 67.3 | 85.2 | 53.5 | 20.0 | 63.6 | 56.0 | 71.1 | 59.5 | 30.7 | 84.6 | 78.4
Focals Conv chen2022focal  | L | 63.8 | 70.0 | 86.7 | 56.3 | 23.8 | 67.7 | 59.5 | 74.1 | 64.5 | 36.3 | 87.5 | 81.4
TransFusion-L bai2022transfusion  | L | 65.5 | 70.2 | 86.2 | 56.7 | 28.2 | 66.3 | 58.8 | 78.2 | 68.3 | 44.2 | 86.1 | 82.0
LargeKernel chen2022scaling  | L | 65.3 | 70.5 | 85.9 | 55.3 | 26.8 | 66.2 | 60.2 | 74.3 | 72.5 | 46.6 | 85.6 | 80.0
PointAug. wang2021pointaugmenting † | L+C | 66.8 | 71.0 | 87.5 | 57.3 | 28.0 | 65.2 | 60.7 | 72.6 | 74.3 | 50.9 | 87.9 | 83.6
MVP Yin2021MVP  | L+C | 66.4 | 70.5 | 86.8 | 58.5 | 26.1 | 67.4 | 57.3 | 74.8 | 70.0 | 49.3 | 89.1 | 85.0
FusionPainting Xu2021FusionPaintingMF  | L+C | 68.1 | 71.6 | 87.1 | 60.8 | 30.0 | 68.5 | 61.7 | 71.8 | 74.7 | 53.5 | 88.3 | 85.0
AutoAlign chen2022autoalign  | L+C | 68.4 | 72.4 | 87.0 | 59.0 | 33.1 | 69.3 | 59.3 | 78.0 | 72.9 | 52.1 | 87.6 | 85.1
FUTR3D chen2022autoalign  | L+C | 68.4 | 72.4 | 87.0 | 59.0 | 33.1 | 69.3 | 59.3 | 78.0 | 72.9 | 52.1 | 87.6 | 85.1
TransFusion bai2022transfusion  | L+C | 68.9 | 71.7 | 87.1 | 60.0 | 33.1 | 68.3 | 60.8 | 78.1 | 73.6 | 52.9 | 88.4 | 86.7
BEVFusion liang2022bevfusion  | L+C | 69.2 | 71.8 | 88.1 | 60.9 | 34.4 | 69.3 | 62.1 | 78.2 | 72.2 | 52.2 | 89.2 | 85.5
BEVFusion liu2022bevfusion  | L+C | 70.2 | 72.9 | 88.6 | 60.1 | 39.3 | 69.8 | 63.8 | 80.0 | 74.1 | 51.0 | 89.2 | 86.5
DeepInteraction-base | L+C | 70.8 | 73.4 | 87.9 | 60.2 | 37.5 | 70.8 | 63.8 | 80.4 | 75.4 | 54.5 | 91.7 | 87.2
Focals Conv-F chen2022focal † | L+C | 70.1 | 73.6 | 87.5 | 60.0 | 32.6 | 69.9 | 64.0 | 71.8 | 81.1 | 59.2 | 89.0 | 85.5
LargeKernel3D-F chen2022scaling † | L+C | 71.1 | 74.2 | 88.1 | 60.3 | 34.3 | 69.1 | 66.5 | 75.5 | 82.0 | 60.3 | 89.6 | 85.7
DeepInteraction-large† | L+C | 74.1 | 75.5 | 88.8 | 64.0 | 40.8 | 70.9 | 62.7 | 82.3 | 85.3 | 64.5 | 92.6 | 89.3
BEVFusion-e liu2022bevfusion § | L+C | 75.0 | 76.1 | 90.5 | 65.8 | 42.6 | 74.2 | 67.4 | 81.1 | 84.4 | 62.9 | 91.8 | 89.4
DeepInteraction-e§ | L+C | 75.7 | 76.3 | 89.0 | 64.5 | 44.7 | 74.2 | 66.0 | 83.5 | 85.4 | 66.4 | 92.8 | 90.9

###  A.2 Discussions of potential societal impacts

Fusing multi-modal information allows to compensate for the shortcomings of
single modality in 3D object detection, leading to more more accurate and
robust performance. In practice, a stronger 3D object detection method as our
DeepInteraction model is expected to reduce the potential accidents of self-
driving cars. This improves the safety and reliability of autonomous driving.
However, multi-modal algorithms often require more powerful computing devices
and run at a higher cost. This raises a need for improving the system
efficiency to be resolved in the future.

###  A.3 Limitations

All the components for multi-modal fusion in our DeepInteraction have no
preference to any per-modal representations. However, the initial queries are
derived from LiDAR BEV, albeit fused with image features. We will explore how
to generate initial queries from both modalities (i.e., LiDAR’s bird-eyes-view
and camera’s front-view).

Our method involves explicit 2D-3D mapping, hence is conditioned on the
calibration quality of the sensors. To relax this condition, a potential
method is to exploit the attention mechanism to allow the network to
automatically establish alignment between multi-modal features.

Finally, our current model design does not take into account model efficiency.
In the future, we will develop a more advanced framework which can adaptively
select more cost-effective combinations of interaction operators in order to
optimize the trade-off between performance, efficiency and robustness.

###  A.4 More visualizations

Figure 6: Qualitative comparison between LiDAR-only baseline and our
DeepInteraction. Blue boxes and green boxes are the predictions and ground-
truth, respectively. Solid eclipse indicates false negative, and dashed
eclipse indicates false positive. Figure 7: Detection results shown in point
clouds by DeepInteraction on nuScenes val set. The bounding boxes of ground-
truth and predictions are in the color blue and green respectively. Best
viewed on screen.