11institutetext: Wikimedia Foundation 11email<EMAIL_ADDRESS>
# Analyzing Wikidata Transclusion on English Wikipedia

Isaac Johnson

###### Abstract

Wikidata is steadily becoming more central to Wikipedia, not just in
maintaining interlanguage links, but in automated population of content within
the articles themselves. It is not well understood, however, how widespread
this transclusion of Wikidata content is within Wikipedia. This work presents
a taxonomy of Wikidata transclusion from the perspective of its potential
impact on readers and an associated in-depth analysis of Wikidata transclusion
within English Wikipedia. It finds that Wikidata transclusion that impacts the
content of Wikipedia articles happens at a much lower rate (5%) than previous
statistics had suggested (61%). Recommendations are made for how to adjust
current tracking mechanisms of Wikidata transclusion to better support metrics
and patrollers in their evaluation of Wikidata transclusion.

###### Keywords:

Wikidata Wikipedia Patrolling

##  1 Introduction

Wikidata is steadily becoming more central to Wikipedia, not just in
maintaining interlanguage links, but in automated population of content within
the articles themselves. This transclusion of Wikidata content within
Wikipedia can help to reduce maintenance of certain facts and links by
shifting the burden to maintain up-to-date, referenced material from each
individual Wikipedia to a single repository, Wikidata.

Current best estimates suggest that, as of August 2020, 62% of Wikipedia
articles across all languages transclude Wikidata content. This statistic
ranges from Arabic Wikipedia (arwiki) and Basque Wikipedia (euwiki), where
nearly 100% of articles transclude Wikidata content in some form, to Japanese
Wikipedia (jawiki) at 38% of articles and many small wikis that lack any
Wikidata transclusion. English Wikipedia, the largest language edition, sits
in the middle at 61% articles having some form of Wikidata
transclusion.111https://wmdeanalytics.wmflabs.org/WD˙percentUsageDashboard/

The challenge in interpreting these numbers, however, is that they are based
on a database table called
wbc_entity_usage222https://www.mediawiki.org/wiki/Wikibase/Schema/wbc˙entity˙usage
that was not built to provide insight into the nature of Wikidata
transclusion. Specifically, the table serves to support the Mediawiki software
in updating its cache of rendered Wikipedia articles when updates are made to
transcluded Wikidata items. In many cases, however, the data is very coarse
and gives little indication as to how the Wikidata content appears in the
Wikipedia article. Anecdotally, Portuguese Wikipedia (ptwiki) has a featured
list
article333https://pt.wikipedia.org/wiki/Lista˙de˙mortos˙e˙desaparecidos˙pol%C3%ADticos˙na˙ditadura˙militar˙brasileira
that is generated purely from Wikidata content using a
template444https://en.wikipedia.org/wiki/Template:Wikidata˙list and many
languages make extensive use of Wikidata-powered infoboxes whereas, by
consensus,555https://en.wikipedia.org/wiki/Wikipedia:Requests˙for˙comment/Wikidata˙Phase˙2
transclusion of Wikidata into infoboxes and article content in English
Wikipedia (enwiki) is quite restricted.

The goal of this research is to contextualize the coarse Wikidata transclusion
numbers so as to better understand the diversity of ways in which Wikidata is
being used within Wikipedia and how these different ways might impact
Wikipedia readers. Informed by design decisions and community consensus, a
taxonomy of Wikidata transclusion is developed and applied to English
Wikipedia via a mixture of qualitative coding and quantitative analyses.
Specifically, this research makes the following contributions:

  * •

Taxonomy of transclusion: a framework is provided to categorize different
types of Wikidata transclusion based upon their perceived potential impact on
readers.

  * •

Analysis of transclusion on English Wikipedia: it is determined that the vast
majority of the 61% of articles with evidence of Wikidata transclusion on
English Wikipedia can be classified as low-importance and only 5% of articles
on English Wikipedia have medium- or high-importance Wikidata transclusion.

##  2 Background

Relatively little research has focused on how Wikidata is transcluded within
Wikipedia, with most of the focus of studies of Wikidata (outside of the large
body of work building tools with Wikidata) either being on Wikidata as an
isolated entity—e.g., detecting vandalism on Wikidata [8], estimating item
completeness [1]—or studying the distribution of knowledge across languages
via labels and Wikipedia sitelinks [3, 4]. While this past research is not
directly related to the question of in what ways Wikidata is transcluded in
Wikipedia articles, the results of this analysis do provide insight into one
measure of a Wikidata item’s importance, a relevant aspect of understanding
completeness and prioritizing work such as vandalism detection. It has been
noted, as well, that Wikidata transclusion introduces challenges to Wikipedia
editors ability to patrol changes to articles [6].

###  2.1 Wikidata Transclusion

Wikidata transclusion is tracked via the wbc_entity_usage database
table666https://www.mediawiki.org/wiki/Wikibase/Schema/wbc˙entity˙usage, the
intention of which is to keep the cache of parsed Wikipedia articles up-to-
date. It thus tracks which Wikidata items, and which aspects of these items,
are transcluded in Wikipedia articles. It is based on what Wikidata content is
requested when rendering a Wikipedia article, however, which is a superset of
what Wikidata content actually is transcluded (and displayed) in an article.

There are a variety of mechanisms through which Wikidata transclusion happens
within Wikipedia articles such as templates that depend on Lua modules or more
specific parser
calls.777https://en.wikipedia.org/wiki/Wikipedia:Wikidata#Inserting˙Wikidata˙values˙into˙Wikipedia˙articles
While the end result should not depend on the method employed, the exact
method used does determine how the transclusion is recorded in the
wbc_entity_usage table. In particular, whereas the specific parser calls often
allow the table to record the specific statements used within an article, the
more general Lua modules often trigger a much broader usage aspect (“some
statements”) that obscures which particular properties are being transcluded.

Calculating which articles have some form of Wikidata transclusion through the
wbc_entity_usage table, then, is a very coarse measure of what Wikidiata
content is actually included in a Wikipedia article. Improving the insights
that can be gleaned from this table would require substantial reengineering of
how the table works. This research, however, does provide some insight into
how to interpret the data in this table as well as offer an alternative,
richer, though less automatic, framework for evaluating Wikidata transclusion
within Wikipedia.

##  3 Methods

###  3.1 Taxonomy of Wikidata Transclusion

Not all forms of Wikidata transclusion have the same potential impact. In this
research, impact was defined as how strongly it is believed a reader could be
affected if the content that was transcluded was incorrect. There are many
alternative definitions—e.g., impact on editors’ backlog to maintain
Wikipedia—that would undoubtedly lead to different classifications. This
definition was chosen as it informs how important it is for patrollers to be
aware of the transclusion and track changes to the content [6].

Instances of Wikidata transclusion was classified as either high, medium, or
low importance—i.e. impact—depending on three different criteria:

  * •

Platform: does the transclusion show up on all platforms—i.e. desktop, mobile
web, and Android/iOS apps—or only in certain form factors? Transclusion that
shows up on more platforms has a higher potential for impact because more
readers can see it. English Wikipedia (like most other language editions) is
largely split between desktop and mobile web readers with only a small
percentage of pageviews coming from the
apps.888https://stats.wikimedia.org/#/en.wikipedia.org/reading/total-page-
views/normal—bar—2-year—access~desktop*mobile-app*mobile-web—monthly

  * •

Salience: when transclusion appears, where on the page does it appear and how
obvious is it? Content that appears at the top of articles is more likely to
be interacted with by readers versus content that appears at the end of
articles [2] and thus is more impactful. Links generated by templates
generally have low click-through rates [5].

  * •

Reader impact if viewed: finally, assuming the reader sees the content, what
is the potential impact? Is it a fact that might affect their perception of
the topic [7, 9], a link that could lead to offensive content or
disinformation such as media repositories that might contain offensive images,
a link that could lead to incorrect content but points to a well-curated
knowledge base such as VIAF (libraries) or GNIS (geography) and would be
likely recognized as incorrect if followed and not be offensive, or is it just
a hidden tracking
category999https://en.wikipedia.org/wiki/Wikipedia:Categorization#Wikipedia˙administrative˙categories
for the article that assists in maintenance of the wiki but has no relevance
to the reader?

Table 1 shows the main types of templates that were encountered on English
Wikipedia and how they were coded according to the above taxonomy.

Use-case | Platform | Salience | Impact | Importance | Usage |  Example Templates
---|---|---|---|---|---|---
Infobox | All | High | High | High | Low |  Ib person Wikidata; Ib company
External Links | All | Medium | Medium | Medium | Medium |  IMDb title; Commons category
Search Descriptions | App | High | High | Low | High |  Default without shortdesc
Metadata | Desktop | Low | Low | Low | High |  Authority control; Taxonbar
Tracking Categories | Desktop | Low | Low | Low | High |  Birth date; Coord (when hard-coded)
References | All | Low | Low | Low | Medium |  USGS gazetteer; Cite Q
Table 1: Overview of the primary forms in which Wikidata transclusion happens
on English Wikipedia along with how each use-case is coded for its importance
to readers. “Ib” stands for “Infobox”. Usage refers to roughly how often each
use-case appears in English Wikipedia based on the results of this study.

###  3.2 Qualitative Coding

One hundred random articles on English Wikipedia were chosen and evaluated to
determine what, if any, templates were being recorded as transcluding Wikidata
content in the article and categorized into the transclusion categories (low-,
medium-, high-importance) described above.101010The data from the
wbc_entity_usage table can be easily viewed from within the article in the
“Page information” tab under “Wikidata entities used in this page”. An article
was categorized with its highest level of transclusion—e.g., an article with a
low-importance transclusion and high-importance transclusion would be coded as
high-importance transclusion. Only a single researcher evaluated the data
given the time-intensive nature of the coding. While there is in theory only a
single correct answer for what templates are triggering Wikidata transclusion,
in practice it can be difficult to trace back the calls made by templates to
determine whether they might be causing the transclusion listed within the
wbc_entity_usage table. The articles were evaluated in their current form on
31 July 2020.

###  3.3 Quantitative Verification

The final step was to verify the representativeness of the sample of articles
coded and prototype an approach to continuing to track Wikidata transclusion
without having to re-code a new sample of articles. Specifically, for the
templates that were identified as major sources of Wikidata transclusion, a
Python script was developed that would detect usage of these templates in
wikitext dumps111111https://github.com/geohci/wikidata-transclusion and
determine whether the template was actually transcluding content or just
generating tracking categories based on a series of
heuristics.121212Alternative strategies such as selectively ablating templates
from a page in a test environment to see how content and tracking changes
would be excessively complex because most templates are written to use the
Wikidata item of the article containing them and so cannot be evaluated
outside of the context of the article they are in. This included two common
metadata templates (Template:Authority control and Template:Taxonbar), all
templates that generate external links from
Wikidata131313https://en.wikipedia.org/wiki/Category:External˙link˙templates˙using˙Wikidata
as they follow a very similar pattern, and the templates for inserting
latitude/longitude coordinates (Template:Coord) and birth date (Template:Birth
date and age) as very common Wikidata templates. Redirects to these templates
were also included. Infobox templates were not included in this script as
there are approximately 100 with varying but low
usage141414https://en.wikipedia.org/wiki/Category:Infobox˙templates˙using˙Wikidata
and each template has its own specific parameters that control Wikidata
transclusion. The 1 August 2020 wikitext dumps were used for this
evaluation.151515https://dumps.wikimedia.org/enwiki/20200801/enwiki-20200801-pages-
articles.xml.bz2

##  4 Results

As a baseline, the wbc_entity_usage table indicates as of August 2020 that 61%
of articles on English Wikipedia include some form of Wikidata
transclusion.161616https://wmdeanalytics.wmflabs.org/WD˙percentUsageDashboard/
In line with this statistic, from the 100 article sample selected to be
evaluated (full data in Table LABEL:tab:articlesample in the Appendix), 59 had
some form of Wikidata transclusion recorded in the wbc_entity_usage table.

For those 59 articles with some evidence of Wikidata transclusion, however,
only 5 had medium- or high-importance Wikidata transclusion on further
inspection. Specifically, 2 articles had Wikidata-powered infobox templates, 3
articles had Wikidata-powered external link templates, 21 articles just had
metadata templates, and 33 articles just had Wikidata tracking categories.
Overlapping with these categories, 54 of the 100 articles did not override the
Wikidata description for use in article previews in Search on the official iOS
and Android
apps.171717https://www.mediawiki.org/wiki/API:Page˙info˙in˙search˙results This
form of transclusion is not tracked by the wbc_entity_usage table though.

The quantitative evaluation supported the validity of the transclusion rates
seen in the qualitative coding. From all of English Wikipedia (6,125,693
articles), 3% of articles had external link templates that transcluded
Wikidata (only 47.5% of the 3,132,833 instances found of external link
templates that could transclude Wikidata did while the rest provided hard-
coded parameters and thus only served to generate tracking categories), 29% of
articles had metadata templates181818This actually tracks closely to the 21%
statistic from the qualitative evaluation as that number is for articles that
only had metadata templates but an additional 3% of articles were found to
have metadata templates and higher-importance templates., and 27% of articles
were found to just have tracking categories generated via Wikidata templates.
For instance, out of the 1,841,505 times in which the coordinates template
(displays latitude-longitude coordinates in an article with a link to a map)
appears in English Wikipedia, it only uses Wikidata to provide coordinates
2,730 times (0.1%) and the rest of the template instances have latitude-
longitude coordinates hard-coded.191919This is likely the result of English
Wikidata determining that existing information should not be overwritten with
Wikidata data:
https://en.wikipedia.org/wiki/Wikipedia:Requests˙for˙comment/Wikidata˙Phase˙2

##  5 Discussion

###  5.1 Tracking Wikidata Transclusion

This analysis indicates that the prior measure of Wikidata transclusion in
English Wikipedia (61%) both greatly underestimates and overestimates Wikidata
transclusion. Transclusion that affects the content of articles (high- and
medium-importance) was determined to be 5% of articles, indicating that 61% is
far too high of a measure for these types. On the other hand, usage of the
Wikidata description in the mobile apps has not been previously tracked and so
this 61% number does exclude some articles that are indeed using Wikidata
descriptions within the context of the Wikipedia apps.

Metrics that track how well-incorporated Wikidata is into Wikipedia articles
will need more nuanced approaches that capture these distinctions. Two fixes
would make a huge impact. First, if standard statements—e.g., date of
birth—were tracked separately from identifiers—e.g., VIAF, GNIS IDs—this would
separate out metadata templates. This analysis found that metadata templates
account for much of Wikidata transclusion. They speak to Wikidata’s value as a
linked data repository but are a very different use-case than transclusion of
content like date of birth or coordinates. Secondly, transclusion that merely
generates tracking categories—e.g., Category:Date of birth not in
Wikidata—could be tracked separately from transclusion that inserts content
into the page.

Together, these two changes would greatly improve the ease with which the
different importance-levels of Wikidata transclusion could be tracked without
in-depth qualitative coding wiki by wiki. This would both support more nuanced
metrics of Wikidata transclusion—e.g., for tracking progress—but also support
much more nuanced filters for patrollers of changes to Wikidata that might
affect articles on their watchlist. Currently, patrollers of vandalism can
choose to see or not see Wikidata
transclusion,202020https://en.wikipedia.org/wiki/Wikipedia:Wikidata#Recent˙changes
but do not have good tools for distinguishing from transclusion that might
have a sizeable impact on an article and therefore warrant checking and
changes to Wikidata items that would have no or only a very minimal impact on
an article. As these analyses demonstrated, much of Wikidata transclusion
could reasonably be considered “noise” from a patrolling perspective,
especially because the metadata templates are tracked via wbc_entity_usage in
such a way that any change to any part of the article’s associated Wikidata
item shows up in recent changes (even if it has no impact on the article
whatsoever).

###  5.2 Generalization

English Wikipedia was analyzed in this work as the most accessible Wikipedia
for qualitative coding for the researcher. While there are both many wikis
with more and with less evidence of transclusion, English Wikipedia does have
some very strong policies with regards to Wikidata transclusion that are not
necessarily found in other language editions. Further analyses of Wikidata
transclusion in other languages would undoubtedly discover different
distributions of usage and new types of templates that have not shown up on
English Wikipedia.

A different approach to the taxonomy would also change the results. For
instance, this taxonomy does not take into account any measure of the
importance of the article that is transcluding the Wikidata content—e.g.,
number of views the article gets [10], whether it’s a biography of a living
person212121https://en.wikipedia.org/wiki/Wikipedia:Biographies˙of˙living˙persons—which
would likely mediate the expected impact of incorrect content being
transcluded. Different taxonomies—e.g., focusing on the reduction in editor
backlog that would result from a particular instance of Wikidata
transclusion—would also likely lead to very different results.

## References

  * [1] Balaraman, V., Razniewski, S., Nutt, W.: Recoin: relative completeness in wikidata. In: Companion Proceedings of the The Web Conference 2018. pp. 1787–1792 (2018)
  * [2] Dimitrov, D., Singer, P., Lemmerich, F., Strohmaier, M.: What makes a link successful on wikipedia? In: Proceedings of the 26th International Conference on World Wide Web. pp. 917–926 (2017)
  * [3] Kaffee, L.A., Piscopo, A., Vougiouklis, P., Simperl, E., Carr, L., Pintscher, L.: A glimpse into babel: an analysis of multilinguality in wikidata. In: Proceedings of the 13th International Symposium on Open Collaboration. pp. 1–5 (2017)
  * [4] Konieczny, P., Klein, M.: Gender gap through time and space: A journey through wikipedia biographies via the wikidata human gender indicator. New Media & Society 20(12), 4608–4633 (2018)
  * [5] Mitrevski, B., Piccardi, T., West, R.: Wikihist. html: English wikipedia’s full revision history in html format. In: Proceedings of the International AAAI Conference on Web and Social Media. vol. 14, pp. 878–884 (2020)
  * [6] Morgan, J.: Patrolling on wikipedia (2019), https://meta.wikimedia.org/wiki/Research:Patrolling˙on˙Wikipedia
  * [7] Rothshild, A., Lurie, E., Mustafaraj, E.: How the interplay of google and wikipedia affects perceptions of online news sources. In: Computation+ Journalism Symposium (2019)
  * [8] Sarabadani, A., Halfaker, A., Taraborelli, D.: Building automated vandalism detection tools for wikidata. In: Proceedings of the 26th International Conference on World Wide Web Companion. pp. 1647–1654 (2017)
  * [9] Singer, P., Lemmerich, F., West, R., Zia, L., Wulczyn, E., Strohmaier, M., Leskovec, J.: Why we read wikipedia. In: Proceedings of the 26th International Conference on World Wide Web. pp. 1591–1600 (2017)
  * [10] Warncke-Wang, M., Ranjan, V., Terveen, L., Hecht, B.: Misalignment between supply and demand of quality content in peer production communities. In: Ninth International AAAI Conference on Web and Social Media (2015)

##  6 Appendix

Table 2: Complete listing of 100 random articles that were sampled from English Wikipedia and their Wikidata transclusion categories. Abbreviations: “S. only” is “Sitelinks only” which means no transclusion; “T. only” is “Tracking only” which means there was transclusion but it only generated tracking categories and no content; “No shortdesc” means that the article uses Wikidata’s description in the mobile apps; “Ib” stands for “Infobox”; “Commons C.” indicates “Commons category” template, which inserts a link to a Commons category in the page. Note that Infobox Anatomy is listed as metadata because it inserts metadata links as opposed to facts, though it is in the context of an infobox. Pages were evaluated in their current form on 31 July 2020 and they (or the templates they transclude) may have changed since then. No. |  Page Title | S. only | T. only | No shortdesc | Metadata |  Content | Import.
---|---|---|---|---|---|---|---
1 |  Don’t Go Breaking My Heart (Sonic Dream Collective song) | Y |  | Y |  |  | None
2 |  Donja Raštelica |  | Y |  |  |  | Low
3 |  Salió el Sol | Y |  | Y |  |  | None
4 |  Archimantis sobrina |  |  | Y | Taxonbar |  | Low
5 |  Fiafia | Y |  | Y |  |  | None
6 |  Daniel Webster |  |  | Y | Auth. Ctrl. |  | Low
7 |  1997 European Athletics U23 Championships – Women’s 5000 metres | Y |  | Y |  |  | None
8 |  Urige Buta |  |  |  | Auth. Ctrl. |  Sports links | Medium
9 |  Wolfgang Martin Stroh |  |  |  | Auth. Ctrl. |  | Low
10 |  Olu Irame |  | Y |  |  |  | Low
11 |  Travis McGee |  | Y | Y |  |  | Low
12 |  Online Bible | Y |  | Y |  |  | None
13 |  Caledonian Brewery |  |  | Y |  |  Ib company | High
14 |  Networking cables |  |  | Y | Auth. Ctrl. |  Commons C. | Medium
15 |  Arachno Creek |  |  |  | Auth. Ctrl. |  | Low
16 |  Malinvestment | Y |  | Y |  |  | None
17 |  Steirastoma poeyi |  |  |  | Taxonbar |  | Low
18 |  1927 William & Mary Indians football team | Y |  |  |  |  | None
19 |  1919 Uruguayan parliamentary election | Y |  | Y |  |  | None
20 |  Abbey of the Holy Ghost |  | Y | Y |  |  | Low
21 |  Lucky Lynx | Y |  | Y |  |  | None
22 |  South West Pacific (film) |  | Y | Y |  |  | Low
23 |  Lawrence Townsend |  |  | Y | Auth. Ctrl. |  | Low
24 |  Anahuarque |  | Y | Y |  |  | Low
25 |  Platycephalus fuscus |  |  | Y | Taxonbar |  | Low
26 |  Cambarus williami |  |  | Y | Taxonbar |  | Low
27 |  Senoussi (cigarette) | Y |  | Y |  |  | None
28 |  Nuhiji | Y |  |  |  |  | None
29 |  Songs of Life (The Gufs album) | Y |  |  |  |  | None
30 |  Darren McKenzie-Potter |  | Y |  |  |  | Low
31 |  List of aircraft (Sb) | Y |  |  |  |  | None
32 |  Res Gestae Divi Augusti |  | Y |  |  |  | Low
33 |  Nightrider (chess) | Y |  |  |  |  | None
34 |  Afonso V of Portugal |  |  |  | Auth. Ctrl. |  | Low
35 |  Panikos Hatziloizou |  | Y |  |  |  | Low
36 |  Bongo Comics | Y |  | Y |  |  | None
37 |  Gustavia longepetiolata |  |  | Y | Taxonbar |  | Low
38 |  1935 SANFL season | Y |  | Y |  |  | None
39 |  Cat Orgy |  | Y |  |  |  | Low
40 |  Ulex |  |  |  | Taxonbar |  | Low
41 |  Roseate | Y |  |  |  |  | None
42 |  Tourism Corporation of Khyber Pakhtunkhwa | Y |  | Y |  |  | None
43 |  Fever 121614 | Y |  |  |  |  | None
44 |  Inside Detroit |  | Y | Y |  |  | Low
45 |  KKSE | Y |  |  |  |  | None
46 |  Philip II, Count of Schaumburg-Lippe |  |  | Y | Auth. Ctrl. |  | Low
47 |  Lyndon, Ohio |  | Y | Y |  |  | Low
48 |  Principality of Lippe |  | Y | Y |  |  | Low
49 |  They Are Billions |  | Y | Y |  |  | Low
50 |  List of fellows of the Royal Society elected in 1973 | Y |  | Y |  |  | None
51 |  Epiphysis |  |  | Y | Ib Anatomy |  | Low
52 |  João Valente Bank |  | Y | Y |  |  | Low
53 |  Clitophon | Y |  |  |  |  | None
54 |  Johnstown Center, Wisconsin |  | Y |  |  |  | Low
55 |  List of mergers in Shimane Prefecture | Y |  | Y |  |  | None
56 |  Highbury Union F.C. | Y |  | Y |  |  | None
57 |  Babaha |  | Y |  |  |  | Low
58 |  Metropolis (Anatolia) |  |  | Y | Auth. Ctrl. |  | Low
59 |  Mat Latos |  | Y | Y |  |  | Low
60 |  Woodstock Railway | Y |  | Y |  |  | None
61 |  Megachile osea |  |  |  | Taxonbar |  | Low
62 |  Public holidays in Japan | Y |  |  |  |  | None
63 |  Danish Contemporary Bible 2020 |  | Y | Y |  |  | Low
64 |  Onion River (Minnesota) |  | Y | Y |  |  | Low
65 |  Zémidjan | Y |  | Y |  |  | None
66 |  Mena (album) | Y |  |  |  |  | None
67 |  Volvarina habanera |  |  | Y | Taxonbar |  | Low
68 |  Flying High (1931 film) |  | Y |  |  |  | Low
69 |  Myponie Point |  | Y | Y |  |  | Low
70 |  Elizabeth Mayer |  |  | Y | Auth. Ctrl. |  | Low
71 |  Notre Dame Catholic School |  | Y |  |  |  | Low
72 |  List of Canadian Hot 100 top 10 singles in 2011 | Y |  | Y |  |  | None
73 |  Arthur Koegel | Y |  | Y |  |  | None
74 |  Lists of Bulgarian films | Y |  |  |  |  | None
75 |  6th Parliament of King William III | Y |  | Y |  |  | None
76 |  DK King of Swing |  | Y | Y |  |  | Low
77 |  Benjamin Dwyer |  |  | Y | Auth. Ctrl. |  | Low
78 |  Patton Glacier |  |  | Y |  |  USGS gazetteer | Medium
79 |  McAfee Peak |  | Y | Y |  |  | Low
80 |  Sommatino |  |  |  | Auth. Ctrl. |  Ib Italian Comune | High
81 |  Pay Takht-e Varzard |  | Y |  |  |  | Low
82 |  89 Albert Embankment |  | Y |  |  |  | Low
83 |  Acalles carinatus |  |  |  | Taxonbar |  | Low
84 |  Saudia Aerospace Engineering Industries |  | Y | Y |  |  | Low
85 |  FromeFM | Y |  | Y |  |  | None
86 |  Clonakilty Cowboys | Y |  |  |  |  | None
87 |  Gorilla (James Taylor album) | Y |  |  |  |  | None
88 |  Union Township, Lawrence County, Ohio |  | Y |  |  |  | Low
89 |  Radio KAOS | Y |  |  |  |  | None
90 |  Ralph Emery |  | Y |  |  |  | Low
91 |  Marine Technology Society |  |  |  | Auth. Ctrl. |  | Low
92 |  ABMA | Y |  |  |  |  | None
93 |  Sten Heckscher |  |  | Y | Auth. Ctrl. |  | Low
94 |  Keiokaku Velodrome |  | Y | Y |  |  | Low
95 |  Mena (given name) | Y |  |  |  |  | None
96 |  Speed skating at the 1924 Winter Olympics – Men’s all-round | Y |  |  |  |  | None
97 |  List of Gintama. Shirogane no Tamashii-hen episodes | Y |  |  |  |  | None
98 |  The Singing Hotel |  | Y |  |  |  | Low
99 |  Kedrick | Y |  |  |  |  | None
100 |  Arbinda Department |  | Y |  |  |  | Low


# Point2Seq: Detecting 3D Objects as Sequences

Yujing Xue1,2 Jiageng Mao311footnotemark: 1 Minzhe Niu2 Hang Xu2 Michael Bi
Mi4
Wei Zhang2 Xiaogang Wang3 Xinchao Wang1
1National University of Singapore 2Huawei Noah’s Ark Lab
3The Chinese University of Hong Kong 4Huawei International Pte Ltd  Joint
first authors with equal contribution. Corresponding author. Email:
<EMAIL_ADDRESS>
###### Abstract

We present a simple and effective framework, named Point2Seq, for 3D object
detection from point clouds. In contrast to previous methods that normally
predict attributes of 3D objects all at once, we expressively model the
interdependencies between attributes of 3D objects, which in turn enables a
better detection accuracy. Specifically, we view each 3D object as a sequence
of words and reformulate the 3D object detection task as decoding words from
3D scenes in an auto-regressive manner. We further propose a lightweight
scene-to-sequence decoder that can auto-regressively generate words
conditioned on features from a 3D scene as well as cues from the preceding
words. The predicted words eventually constitute a set of sequences that
completely describe the 3D objects in the scene, and all the predicted
sequences are then automatically assigned to the respective ground truths
through similarity-based sequence matching. Our approach is conceptually
intuitive and can be readily plugged upon most existing 3D-detection backbones
without adding too much computational overhead; the sequential decoding
paradigm we proposed, on the other hand, can better exploit information from
complex 3D scenes with the aid of preceding predicted words. Without bells and
whistles, our method significantly outperforms previous anchor- and center-
based 3D object detection frameworks, yielding the new state of the art on the
challenging ONCE dataset as well as the Waymo Open Dataset. Code is available
at https://github.com/ocNflag/point2seq.

##  1 Introduction

Figure 1: Point2Seq reformulates the 3D object detection problem as auto-
regressively generating sequences of words that can represent the 3D objects.
The sequential decoding paradigm attains better detection performance compared
to previous works that predict all attributes of the 3D objects in parallel,
thanks to its competence in exploring intrinsic dependencies among object
attributes.

3D object detection is a critical component of intelligent perception systems
for self-driving, aiming to localize and recognize cars, pedestrians, and
other key objects around an autonomous vehicle. With the increasing popularity
of LiDAR sensors, 3D object detection from point clouds has been receiving
much attention owing to the advanced detection accuracy compared to other
input modalities.

In driving scenes, most 3D objects are extremely small relative to the
detection range, and numerous approaches have been employed to detect those
small objects from point clouds accurately. Anchor-based methods [38, 13]
place predefined anchors on each pixel center of a Bird-Eye-View (BEV) feature
map, while center-based approaches [42, 9] treat the pixels near object
centers as positives and predict boxes using those pixel features. These
methods rely on complex hand-crafted procedures of label assignments and post-
processing, and quantization errors introduced by the BEV representations lead
to severe misalignment between the object locations and the pixel features
used to predict those objects. Approaches like [25, 27, 16] rely on a second
refinement stage to mitigate the misalignment issue, yet at the cost of adding
too much computational overhead. Therefore, learning more spatially-aligned
features to detect the 3D objects accurately while maintaining a high
efficiency poses an open challenge to the research community.

To address the challenge, in this paper, we introduce Point2Seq, a flexible
and streamlined framework for 3D object detection from point clouds. Unlike
prior methods that typically predict all attributes of a 3D object (_e.g_.,
location, class, size) simultaneously, we represent each object as a sequence,
in which each word corresponds to an object attribute, and we explicitly
explore the inherent dependencies among words alongside their relations with
the input 3D scene to progressively predict each attribute of 3D objects. Our
motivation is quite intuitive: given the fact that each object is represented
by sequential words, the existing words will provide 3D detectors with cues to
better exploit spatial features and help detectors predict the following words
more accurately. For instance, 3D detectors can leverage more spatially-
aligned features for an object if the object location has been formerly
predicted and better recognize the object class if its size information has
already been known. It is therefore desirable to design a detection framework
that may sequentially predict words of 3D objects conditioned on the preceding
generated words as well as the spatial features until all the words form a set
of sequences describing the 3D objects in the scene.

To achieve this goal, we must address two critical challenges: how to design
the sequential object words prediction module and make it compatible with the
existing 3D detection pipelines and how to optimize the 3D detector with the
ground truth and predicted sequences. To resolve the first problem, we propose
a novel scene-to-sequence decoder, which takes the BEV feature map and a set
of initial region cues as input and auto-regressively decodes sequences for
all objects in parallel. The scene-to-sequence decoder is compatible with most
grid-based 3D backbones [38, 44, 35, 13, 19] and can effectively aggregate
features from those backbones based on the information of the preceding words.
By virtue of the highly-parallel deep learning libraries, the scene-to-
sequence decoder can generate the sequences of all 3D objects in one shot,
with little added time and memory cost.

To handle the second issue, we adopt the set-to-set loss to match the
predicted sequences with the ground truths. Unlike existing approaches [36,
21, 3] that utilize the sum of the classification and regression loss as the
cost function for bipartite matching, in this paper, we propose a novel metric
to measure the similarity between two sequences. Then we perform bipartite
matching by maximizing the global similarity of the prediction and the ground
truth set using the proposed metric. In this manner, each predicted sequence
can be automatically assigned to a respective ground truth without pre-defined
anchors or centers. The assignments are globally optimal and result in better
performance compared to previous methods.

With the lightweight scene-to-sequence decoder, our method can progressively
predict the words of 3D objects, yielding reliable predictions that
significantly outperform state-of-the-art. In addition, our method is free
from the human-designed procedures of label assignments with similarity-based
sequence matching. Our key contributions are summarized as follows:

$\bullet$ We present an effective and flexible framework for 3D object
detection from point clouds. We represent each 3D object as a sequence of
words and model the 3D object detection problem as decoding the words from the
3D scenes in an auto-regressive manner.

$\bullet$ We propose a scene-to-sequence decoder that can auto-regressively
generate sequences representing the detected 3D objects and introduce the
similarity-based sequence matching scheme to enable automatic assignments of
the predicted sequences to the respective ground truths for end-to-end
training.

$\bullet$ Our method significantly outperforms the anchor-based and center-
based 3D detectors with the same backbone, attaining $66.16\%$ mAP on the ONCE
dataset and $77.52\%$ vehicle L1 mAP on the Waymo Open Dataset.

##  2 Related Work

Backbones for 3D object detection. 3D detectors rely on various backbone
networks to extract features from input point clouds. The existing backbones
of 3D detectors can be divided into $3$ streams: point-based, range-based, and
grid-based. The point-based backbones [26, 39, 28, 41, 40, 15, 22] operate
directly on raw point clouds with the point operators [23, 18] to extract the
point-wise features. The range-based backbones [31, 14, 20, 8, 1] take range
images, which are the raw data from LiDAR sensors, as the input
representation. Customized operators, _e.g_., range conditional convolutions
[1], meta-kernels [8], are applied on the range images for feature extraction.
The grid-based backbones [13, 35, 38, 44, 19] firstly rasterize point clouds
into voxels or pillars. Those voxels or pillars are fed into a 3D network and
then projected into a BEV feature map, followed by a 2D convolutional neural
network to detect 3D objects. Among the $3$ kinds of backbones, the grid-based
backbones can obtain superior detection performance while maintaining high
efficiency. Our Point2Seq is a flexible detection framework and can be applied
to most grid-based backbones.

Figure 2: The overall architecture of our framework. Point2Seq contains three
major components: the 3D backbone, the scene-to-sequence decoder, and the
similarity-based sequence matching scheme. The 3D backbone takes the
rasterized point cloud as input and outputs the Bird-Eye-View (BEV) feature
map for the 3D scene. The scene-to-sequence decoder operates on the BEV
feature map and sequentially predicts the words of 3D objects based on the
information from the preceding predicted words. Finally, the predicted
sequences are automatically assigned to the corresponding ground truths by the
proposed similarity-based sequence matching scheme.

3D objects prediction mechanisms. 3D detectors adopt various prediction
mechanisms to generate detected 3D objects from the backbone features. For the
point-based backbones, PointRCNN [26] directly generates object proposals on
the key points’ locations. For the range-based backbones, RangeDet [8]
generates 3D bounding boxes on the pixels of range images. For the grid-based
backbones, SECOND [38] places a set of 3D anchors on every grid center of a
BEV map. The anchors that have a high overlap with the ground truth 3D objects
are set to positives, and the objects are then predicted on the positive
anchors. SA-SSD [11] applies the part-sensitive warping scheme to enhance the
spatial features. CenterPoints [42] treats BEV pixels near the object centers
as positives and generates predicted bounding boxes near the object centers.
Existing methods usually predict all attributes of the 3D objects
simultaneously without considering the intra-object information, while
Point2Seq can model the interdependencies among the object’s attributes with
the scene-to-sequence decoder.

Set-to-set matching for object detection. The set-to-set matching mechanism is
first introduced in DETR [3] in which a set of object queries are assigned to
the respective ground truths through bipartite matching. This method is
improved by a series of works [45, 32, 29, 34] on image-based object
detection. The set-to-set matching scheme has also been adopted in 3D object
detection. 3DETR [21] utilizes sampled points as object queries. Object DGCNN
[36] leverages a sparse set of object queries to iteratively interact with a
BEV feature map to generate 3D objects. Compared to previous methods,
Point2Seq does not require the sparse object queries or the multi-step
refinement modules in [33, 37]. The sequences are generated densely from the
BEV map in parallel and are automatically matched to the corresponding ground
truths by similarity-based sequence matching, without additional modules or
parameters.

##  3 Detecting 3D Objects as Sequences

###  3.1 Architecture

For each 3D scene, Point2Seq takes a point cloud as input and outputs a set of
3D bounding boxes $\mathcal{B}=\\{B_{1},\cdots,B_{M}\\}\in\mathbb{R}^{M\times
8}$ that represent the detected 3D objects, _e.g_., vehicles, pedestrians,
cyclists, _etc_. A 3D point cloud is an $N\times d$ matrix, where $N$ denotes
the number of points in the scene and $d$ denotes the initial features of
points, _i.e_., 3D coordinates, intensity, _etc_. Each 3D object
$B_{i}\in\mathbb{R}^{8}$ is a vector: $[x,y,z,l,w,h,\theta,c]$, where
$[x,y,z]$ is the location of the object’s center, $[l,w,h]$ is the object’s
size, $\theta$ is the object’s orientation, and $c$ is the class of the
object.

As is shown in Figure 2, the architecture of Point2Seq is composed of $3$
parts: the 3D backbone, the scene-to-sequence decoder, and the similarity-
based sequence matching scheme. The 3D backbone first consumes a point cloud
and generates a BEV feature map from the point cloud. Then the scene-to-
sequence decoder takes both the BEV feature map and an initial set of region
cues as input and decodes sequences of words that describe the detected 3D
objects in the scene. Finally, the similarity-based sequence matching is
applied to assign the predicted sentences to the respective ground truths. The
choice of 3D backbones in Point2Seq can be flexible: most grid-based backbones
[38, 13, 35, 19] can be applied in our framework. The grid-based backbones
first transform point clouds into voxels or pillars, and then 3D features are
extracted from those voxels or pillars by sparse convolutions [10] or set
abstraction [23], respectively. The 3D features are then projected to Bird-
Eye-View (BEV). A 2D convolutional neural network is applied on the projected
features to obtain the final BEV feature map $F\in\mathbb{R}^{H\times W\times
C}$, where the detection space is divided into an $H\times W$ grid, and $C$
denotes the number of feature channels.

In the scene-to-sequence decoder, the 3D objects $\mathcal{B}$ are transformed
into a set of sequences $\\{S_{1},\cdots,S_{M}\\}$, where each sequence
$S_{i}$ corresponds to a 3D object $B_{i}$ and contains $K$ words
$\\{W_{i}^{0},\cdots,W_{i}^{K-1}\\}$ that can represent the 3D object. The
scene-to-sequence decoder operates on the BEV feature map $F$ and can auto-
regressively predict a word $W_{i}^{j}$ conditioning on $F$ and the preceding
words $W_{i}^{0:j-1}$. We will introduce the detailed design of the scene-to-
sequence decoder in Sec. 3.2 and then illustrate how to optimize the scene-to-
sequence decoder and the 3D backbone through similarity-based sequence
matching in Sec. 3.3. Finally, we discuss and compare our method with previous
literature in Sec. 3.4.

###  3.2 Scene-to-Sequence Decoder

Problem formulation. Previous single-stage 3D detectors model the 3D object
detection problem as predicting all attributes of the 3D objects $\mathcal{B}$
simultaneously from the features $F$. The learning process can be formulated
as the optimization problem:

$\max\sum_{i=1}^{M}logP(B_{i}|\mathcal{D}(F)),$ (1)

where $B_{i}\in\mathcal{B}$ is the attributes $[x,y,z,l,w,h,\theta,c]$ of the
$i$th ground truth 3D object, $M=|\mathcal{B}|$, and $\mathcal{D}$ is normally
a convolutional prediction head applied on the BEV feature map $F$. The
parallel prediction paradigm is widely adopted for its efficiency. However, it
suffers from the misalignment between the object’s actual location and the BEV
features used for prediction due to the high quantization errors introduced by
rasterization. Multi-stage refinements can be employed to mitigate the
misalignment problem but will consequently introduce too much computational
overhead.

In this paper, we draw inspirations from Language Modeling (LM) [2, 7] in
natural language processing applications and translate each 3D object $B$ into
a sequence $S$ containing $K$ words $\\{W^{0},\cdots,W^{K-1}\\}$:

$B=\mathcal{T}(S)=\mathcal{T}(W^{0},\cdots,W^{K-1}).$ (2)

The translation $\mathcal{T}$ is parameter-free and bidirectional, so the 3D
objects and their corresponding words can be easily transformed into each
other. Then similar to the language models, we can reformulate the detection
problem as maximizing the probability production of all target words
$\tilde{W_{i}}^{j}$ conditioning on the feature $F$ and the preceding
predicted words $W_{i}^{0:j-1}$:

$\max\sum_{i=1}^{M}\sum_{j=1}^{K-1}logP(\tilde{W_{i}}^{j}|\mathcal{D}(F,W_{i}^{0},\cdots,W_{i}^{j-1})).$
(3)

The main insight of our approach is that each 3D object is decomposed into
several words, and predicting these words sequentially, instead of
simultaneously in previous methods, enables more effective exploitation of the
BEV features with the cues from the preceding predicted words.

3D object as a sequence of words. Translating every 3D object into words is a
pivotal step in our method. Different from those methods that adopt discrete
tokens as words in natural language processing tasks, we represent the words
in a continuous format in our method. The use of continuous representations
for object words is preferable in 3D object detection since most attributes of
3D objects, _e.g_., locations, sizes, orientations, are continuous values, and
the predicted words can be directly transformed back into the corresponding
object’s attributes without loss of accuracy.

In this paper, each 3D object $B=[x,y,z,l,w,h,\theta,c]$ is translated into
$5$ words:

$B=\mathcal{T}(S)=\mathcal{T}(W^{R},W^{L},W^{O},W^{S},W^{C}).$ (4)

The region word $W^{R}=[R_{x},R_{y}]\in\mathbb{R}^{2}$ indicates the possible
region in which the 3D object is likely to appear, where $[R_{x},R_{y}]$ is
the BEV center coordinate of the region, and additional parameters
$[R_{l},R_{w}]$ are introduced to describe the spatial range of the region on
the BEV feature map. The location word
$W^{L}=[L_{x},L_{y},z]\in\mathbb{R}^{3}$ denotes the location of the object’s
center, where $L_{x}=(x-R_{x})/R_{l}$ and $L_{y}=(y-R_{y})/R_{w}$ denote the
relative location in the region. The orientation word
$W^{O}=[sin(\theta),cos(\theta)]\in\mathbb{R}^{2}$ encodes the object’s
orientation $\theta$ by trigonometric functions. The size word
$W^{S}=[log(l),log(w),log(h)]\in\mathbb{R}^{3}$ applies logarithmic functions
to the object’s size. The category word $W^{C}\in\mathbb{R}^{n+1}$ indicates
the probabilities of $n$ detected classes and the background class.

Scene-to-sequence prediction. Our proposed scene-to-sequence decoder takes the
BEV features $F$ and a set of region words $W^{R}$ as the initial inputs and
sequentially predicts $W^{L}$, $W^{O}$, $W^{S}$, $W^{C}$, _i.e_., $W^{1}$,
$W^{2}$, $W^{3}$, $W^{4}$ in $4$ steps for each region $W^{R}$. In each step,
the words are predicted on a hidden state map $H\in\mathbb{R}^{H\times W\times
C}$ that encodes the historical information of the preceding steps.
Specifically, the hidden state $H_{1}$ is firstly initialized as the input BEV
feature map $F$:

$H_{1}=F.$ (5)

Then at the $j$th step, the word $W^{j}$ will be directly predicted from the
hidden state $H_{j}$ at the corresponding region center $W^{R}$, _i.e_.,
$H_{j}[W^{R}]\in\mathbb{R}^{C}$, through a single linear projection layer
$f_{linear}$:

$W^{j}=f_{linear}(H_{j}[W^{R}]),$ (6)

where $[\cdot]$ is the indexing operator. The hidden state $H_{j}[W^{R}]$ at
$W^{R}$ will then be updated to the next step $H_{j+1}[W^{R}]$ based on the
already learned knowledge from the former predicted words
$\\{W^{0},\cdots,W^{j}\\}$:

$H_{j+1}[W^{R}]=\Phi(H_{j}[W^{R}];W^{0},\cdots,W^{j}),$ (7)

where $\Phi$ is the update function. To model the hidden state update process
at the $j$th step, near each region $W^{R}$, we first sample a sparse set of
points $\\{p^{j}_{1},\cdots,p^{j}_{n}\\}\in\mathbb{R}^{n\times 2}$ by a
spatial sampler $\mathcal{S}$ parameterized by the predicted words:

$\\{p^{j}_{1},\cdots,p^{j}_{n}\\}=\mathcal{S}(W^{0},\cdots,W^{j}).$ (8)

The sampling patterns of $\mathcal{S}$ are shown in Figure 3, and the detailed
formulations are demonstrated in the appendix. Then for each region $W^{R}$,
we can update to $H_{j+1}[W^{R}]$ by aggregating hidden vectors at those
sampled locations on $H_{j}$, which can be formulated as

$H_{j+1}[W^{R}]=\mathcal{A}(H_{j}[p^{j}_{1}],\cdots,H_{j}[p^{j}_{n}]),$ (9)

where the aggregation function $\mathcal{A}$ concatenates the sampled hidden
vectors and projects them into the $\mathbb{R}^{C}$ space.

The initial set of the region words $W^{R}$ indicates where the 3D objects are
likely to appear in the 3D scene. Since there is no such prior information, we
employ a dense prediction strategy in this paper. Namely, we treat each pixel
in the BEV feature map as a region word and predict a sequence for each BEV
pixel. The category word $W^{C}$ is expected to predict the highest
probability for the background class if there is no 3D object near the
corresponding pixel region. The dense prediction paradigm benefits from the
highly-parallel characteristics of the modern deep learning libraries, and the
sequences for all pixels can be predicted in parallel by the scene-to-sequence
decoder efficiently using shared MLPs and sampling operators.

Figure 3: The sample locations at each step. For each region, we progressively
obtain $1$, $4$, $4$ points from the spatial sampler $\mathcal{S}$, based on
the information from the predicted words.

###  3.3 Similarity-based Sequence Matching

In this section, we will introduce how to optimize our detection framework by
similarity-based sequence matching. Our approach is inspired by the set-to-set
loss employed in the image-based detection frameworks [3]. Our main
contribution lies in the design of a new cost function tailored for 3D object
detection in the set-to-set matching problem. Specifically, we propose a novel
metric $Sim(S,\tilde{S})$ to measure the similarity of two sequences $S$ and
$\tilde{S}$ from the predicted and the ground truth sequence set,
respectively. We match the predicted sequences to the corresponding target
sequences by maximizing the global similarity of the two sequence sets.
Finally, losses can be applied to the matched sequence pairs for back-
propagation. In this manner, we can eliminate the hand-crafted label
assignment process and make our model end-to-end trainable without non-maximum
suppression.

The scene-to-sequence decoder outputs a set of predicted sequences
$\\{S_{1},\cdots,S_{HW}\\}$ containing $H\times W$ sequences for all BEV
pixels in total. For the ground truth 3D objects, we also construct a sequence
set $\\{\tilde{S}_{1},\cdots,\tilde{S}_{M},\varnothing,\cdots,\varnothing\\}$
where the size of the ground truth set is equal to that of the prediction set,
and we pad the remaining sequences with $\varnothing$ if $M<H\times W$. To
measure the similarity between a predicted sequence $S$ and a ground truth
sequence $\tilde{S}$, we define a new similarity metric that can be formulated
as

$Sim(S,\tilde{S})=(W^{C}\tilde{W}^{C})^{\alpha}\cdot
e^{-(1-\alpha)\sum_{j\in\\{R,L,O,S\\}}|W^{j}-\tilde{W}^{j}|},$ (10)

where the first term $(W^{C}\tilde{W}^{C})^{\alpha}$ measures the class
similarity between the predicted and the ground truth objects, and the second
term measures the shape and location similarity. The hyper-parameter $\alpha$
is utilized to balance the two similarities and set to $0.25$ in our
experiments. $Sim(S,\varnothing)=0$ if a predicted sequence is matched to
$\varnothing$.

The proposed similarity metric is a more stringent criterion to match the
predictions with the ground truths, and even slight differences between the
two sequences can make the similarity score tend to $0$. Given the fact that
the 3D objects are innately small and the mismatch should be completely
avoided, the stringent similarity metric we proposed is preferable in the task
of 3D object detection.

With the similarity metric, we can further establish the optimal set-to-set
matching $\Pi^{*}$ by considering the bipartite matching problem:

$\Pi^{*}=\operatorname*{arg\,max}_{\Pi}\sum_{(i\rightarrow
j)\in\Pi}Sim(S_{i},\tilde{S}_{j}),$ (11)

where $\Pi$ is a bijective function that enables a one-to-one mapping from the
predicted sequence set to the ground truth set. The bipartite matching problem
aims to find the optimal $\Pi^{*}$ so that the maximum overall similarity of
the two sets can be achieved. With $\Pi^{*}$, every ground truth sequence can
be automatically assigned to the corresponding predicted sequence that has the
highest similarity. The optimal bipartite matching $\Pi^{*}$ can be calculated
efficiently by the Hungarian algorithm [12].

Once the matched pairs of $S$ and $\tilde{S}$ are established, the proposed
loss function tailored for 3D object detection can be computed as:

$\displaystyle\mathcal{L}_{det}=$ $\displaystyle\sum_{(i\rightarrow
j)\in\Pi^{*}}[\mathcal{L}_{cls}(W^{C}_{i},\tilde{W}^{C}_{j})+$ (12)
$\displaystyle\mathbbm{1}_{\\{\tilde{S}_{i}\neq\varnothing\\}}\lambda_{reg}\mathcal{L}_{reg}(W^{\\{R,L,O,S\\}}_{i},\tilde{W}^{\\{R,L,O,S\\}}_{j})],$

where $\mathcal{L}_{cls}$ is the focal loss applied on the predicted and
target category words, and $\mathcal{L}_{reg}$ takes the words
$W^{\\{R,L,O,S\\}}$ and $\tilde{W}^{\\{R,L,O,S\\}}$ as input and translates
them back to the respective object’s location and shape $[x,y,z,l,w,h,\theta]$
on which the smooth-$L_{1}$ loss is then applied. The indicator function
$\mathbbm{1}_{\\{\tilde{S}_{i}\neq\varnothing\\}}$ implies that we only apply
$\mathcal{L}_{reg}$ on those sequences that are matched to the ground truths,
and $\lambda_{reg}$ is a coefficient that balances the two losses.

Since each ground truth 3D object is matched with only one predicted sequence,
the scene-to-sequence head does not produce duplicated boxes for an individual
object. Hence the time-consuming process of non-maximum suppression can be
eliminated in our framework. During the inference stage, we simply filter out
those low-quality sequences in which the maximum class probability in $W^{C}$
is below a certain threshold, and we translate the remaining sequences into 3D
objects as the final detection results.

###  3.4 Discussion

Our proposed Point2Seq shares a similar intuition with the concurrent work
Pix2Seq [5], which is proposed for image-based object detection, in terms of
leveraging objects as words that can be read out from a feature map. However,
our method is intrinsically different from [5] in $3$ aspects: 1) Unlike [5]
that merges all objects into an individual sequence, we treat each object as a
sequence and predict all objects in parallel, while words in each object are
generated sequentially. In this manner, we can circumvent the object ordering
problem in [5], and our method is much more efficient at the inference stage
compared to [5] in which the inference latency will be heavily influenced by
the total object count in an image. 2) We adopt the continuous word
representations instead of discrete tokens in [5]. The use of continuous
representations relieves the need for quantization and makes our method
compatible with the existing loss functions tailored for 3D object detection.
3) We propose the scene-to-sequence decoder to generate words for each object,
in lieu of the Transformer architecture in [5]. The scene-to-sequence decoder
is lightweight and leverages a sparse set of features to predict each object,
which is more suitable for 3D object detection where the detected targets
usually are small and sparse.

Method | Backbone | Head | Vehicle LEVEL 1 | Vehicle LEVEL 2
---|---|---|---|---
3D mAP(%) | 3D mAPH(%) | 3D mAP(%) | 3D mAPH(%)
LaserNet [20] | Range | Anchor | 52.1 | 50.1 | - | -
RCD [1] | Range | Center | 69.0 | 68.5 | - | -
RangeDet [8] | Range | Center | 72.85 | - | - | -
RSN [31] | Range | Center | 75.1 | 74.6 | 66.0 | 65.6
PointPillars [13] | Pillar | Anchor | 63.3 | 62.7 | 55.2 | 54.7
Pillar-OD [35] | Pillar | Anchor | 69.8 | - | - | -
MVF [43] | Voxel | Anchor | 62.93 | - | - | -
PV-RCNN [25] | Voxel | Anchor | 77.51 | 76.89 | 68.98 | 68.41
VoTr-TSD [19] | Voxel | Anchor | 74.95 | 74.25 | 65.91 | 65.29
Voxel R-CNN [6] | Voxel | Anchor | 75.59 | - | 66.59 | -
Pyramid-RCNN [16] | Voxel | Anchor | 76.3 | 75.68 | 67.23 | 66.68
CT3D [24] | Voxel | Anchor | 76.3 | - | 69.04 | -
CVCNet [4] | Voxel | Center | 65.20 | - | - | -
AFDet [9] | Voxel | Center | 63.69 | - | - | -
CenterPoints [42] | Voxel | Center | 76.7 | 76.2 | 68.8 | 68.3
SECOND† [38] | Voxel | Anchor | 73.62 | 73.14 | 64.86 | 64.40
CenterPoints† [42] | Voxel | Center | 75.58 | 75.01 | 67.00 | 66.52
Point2Seq (Ours) | Voxel | Sequence | 77.52 | 77.03 | 68.80 | 68.36
Table 1: Performance comparison on the Waymo Open Dataset with 202 validation sequences for vehicle detection. ${\dagger}$: re-implemented using the official code. Point2Seq maintains the same backbone, data augmentations, and training epochs with the re-implemented baselines. Method | mAP(%) | Vehicle mAP(%) | Pedestrian mAP(%) | Cyclist mAP(%)
---|---|---|---|---
overall | 0-30m | 30-50m | 50m-inf | overall | 0-30m | 30-50m | 50m-inf | overall | 0-30m | 30-50m | 50m-inf
PointRCNN [25] | 28.74 | 52.09 | 74.45 | 40.89 | 16.81 | 4.28 | 6.17 | 2.4 | 0.91 | 29.84 | 46.03 | 20.94 | 5.46
PointPillars [13] | 44.34 | 68.57 | 80.86 | 62.07 | 47.04 | 17.63 | 19.74 | 15.15 | 10.23 | 46.81 | 58.33 | 40.32 | 25.86
PV-RCNN [25] | 53.55 | 77.77 | 89.39 | 72.55 | 58.64 | 23.50 | 25.61 | 22.84 | 17.27 | 59.37 | 71.66 | 52.58 | 36.17
SECOND [38] | 51.89 | 71.19 | 84.04 | 63.02 | 47.25 | 26.44 | 29.33 | 24.05 | 18.05 | 58.04 | 69.96 | 52.43 | 34.61
CenterPoints [42] | 60.05 | 66.79 | 80.10 | 59.55 | 43.39 | 49.90 | 56.24 | 42.61 | 26.27 | 63.45 | 74.28 | 57.94 | 41.48
Point2Seq (Ours) | 66.16 | 73.43 | 85.16 | 66.21 | 50.76 | 57.53 | 68.21 | 47.15 | 25.18 | 67.53 | 77.95 | 62.14 | 46.06
Table 2: Performance comparison on the ONCE dataset validation split.
Point2Seq maintains the same backbone architecture and training configurations
with the baselines on the ONCE benchmark.

##  4 Experiment

In this section, we evaluate Point2Seq on the commonly-used Waymo Open Dataset
[30] and the ONCE dataset [17]. We first introduce the experimental settings
in Sec. 4.1. Then we compare our approach with previous state-of-the-art
methods on the Waymo Open Dataset (Sec. 4.2) and the ONCE dataset (Sec. 4.3).
Finally, we report the inference speed and the number of parameters, as well
as the efficacy of different components in our model in Sec. 4.4.

###  4.1 Experimental Setup

Waymo Open Dataset. The Waymo Open Dataset is composed of $1000$ sequences of
point clouds, in which $798$ sequences (nearly $158k$ point cloud samples) are
used as the training set, and $202$ sequences (nearly $40k$ point cloud
samples) are utilized as the validation set. The evaluation metrics on the
Waymo Open Dataset are 3D mean Average Precision (mAP) and mAP weighted by
heading accuracy (mAPH). The IoU threshold used for vehicles is $0.7$ and
$0.5$ for other categories. The detection results are reported based on the
difficulty levels: LEVEL 1 for boxes with more than $5$ points and LEVEL 2 for
boxes with at least $1$ point.

ONCE Dataset. The ONCE dataset contains one million point clouds in total, in
which $5$k, $3$k, $8$k point clouds are annotated as the training, validation,
testing split, respectively. The remaining point clouds are kept unannotated
for self-/semi-supervised learning. In this paper, we train our model on the
training split and report the detection results of vehicles, pedestrians, and
cyclists on the validation and testing split, without using the unlabeled
data. The official evaluation metric is mean Average Precision (mAP), and the
detection results are divided according to the objects’ distances to the
sensor: $0$-$30$m, $30$-$50$m, and $50$m-inf.

Implementation Details. On the Waymo Open Dataset, we use the same 3D sparse
convolutional neural network and 2D convolutional neural network as [42]. The
input voxel size and the output resolution of the BEV feature map are also
kept the same as [42] for a fair comparison. On the ONCE dataset, all the
voxel-based detectors use the same type of 3D backbone [38] in their official
benchmark implementations. We also follow the setting and use the 3D backbone
in [38]. For other model configurations, we adopt the same as those on the
ONCE benchmark.

Training and Inference Details. We train our model with the ADAM optimizer and
the cosine annealing learning rate scheduler. On the Waymo Open Dataset, we
uniformly sample $20\%$ of the point cloud samples for training and use the
full validation set for evaluation following [25]. We train our model with the
batch size $32$ and the initial learning rate $0.006$ for $180$ epochs on $8$
V100 GPUs. $\lambda_{reg}$ in the loss function is set to $2$. Data
augmentations are kept the same as [42]. On the ONCE dataset, we follow the
training settings of the respective benchmark and train our model with the
batch size $32$ and the initial learning rate $0.003$ for $80$ epochs on $8$
V100 GPUs. $\lambda_{reg}$ in the loss function is set to $0.5$. Data
augmentations are kept the same as [17]. On both two datasets, we filter out
those objects with the maximum foreground class probability in $W^{C}$ below
$0.2$ and keep the remaining objects as the final detection results during the
inference stage, without any other post-processing.

###  4.2 Comparisons on the Waymo Open Dataset

Since our contribution focuses on the 3D object prediction mechanism, the
fairest way to evaluate our method and compare it with the anchor-based and
center-based methods is to only replace the center or anchor head with our
scene-to-sequence decoder while maintaining other components as the same. We
follow this principle and re-implement two baseline models from their official
implementations: SECOND [38] with the anchor head and CenterPoints [42] with
the center head. Our proposed Point2Seq, re-implemented SECOND, and
CenterPoints have the same voxel-based 3D backbone, data augmentations, and
training epochs to ensure a completely fair comparison.

Table 1 shows the detection results on the Waymo validation set. Simply
switching from the anchor and center head to our Point2Seq gives $3.90\%$ and
$1.94\%$ LEVEL 1 mAP improvements, respectively. Our method attains $77.52\%$
LEVEL 1 mAP and $68.80\%$ LEVEL 2 mAP for vehicle detection, surpassing
existing methods by a significant margin. Our approach even outperforms those
time-consuming two-stage 3D detectors [25, 16, 24], which indicates the
effectiveness of the scene-to-sequence decoder.

###  4.3 Comparisons on the ONCE dataset

The ONCE dataset benchmarks different voxel-based detectors using the same
backbone network, and we also follow this rule for a fair comparison. As is
shown in Table 2, Point2Seq attains the state-of-the-art results on all
classes, with $73.43\%$ mAP for vehicle detection, $57.53\%$ mAP for
pedestrian detection, and $67.53\%$ for cyclist detection. The overall mAP of
our approach is $66.16\%$, $6.11\%$ higher than the center-based 3D object
detector [42] and $14.27\%$ higher than the anchor-based 3D object detector
[38]. The observations on the ONCE dataset are consistent with those on the
Waymo Open Dataset.

###  4.4 Ablation Studies

Inference speed and model parameters. Table 3 demonstrates the inference time
and the number of parameters of our method. Since the 3D objects in a scene
are predicted in parallel, Point2Seq can obtain high efficiency with $70.4$ms
inference latency on average for a single model on a V100 GPU. The scene-to-
sequence head only contains several linear projection layers, and the sampling
operation is parameter-free, so the model only introduces $0.1$M additional
parameters compared to the center-based baseline.

Generalizability on different 3D backbones. To verify whether Point2Seq can
achieve superior performances upon different backbones, we apply the scene-to-
sequence head on both the voxel-based [42] and pillar-based [13] 3D backbones
and compare the results with the center and anchor head, respectively. Table 4
demonstrates that on both two types of backbone networks, our method
consistently outperforms the anchor-based and center-based detectors.

Effects of different components in Point2Seq. Table 5 shows the effectiveness
of the scene-to-sequence decoder and the similarity-based sequence matching
scheme. Similarity-based sequence matching can be independently applied on the
previously used convolutional head and boost the detection performance by
$3.1\%$ mAP compared to the anchor-based baseline. Combing the two proposed
components, we can obtain a performance gain of $3.9\%$ mAP.

Method | Vehicle | #Param | Latency (ms)
---|---|---|---
L1/L2 mAP(%)
PV-RCNN [25] | 77.51/68.98 | 13.05M | 300
CenterPoints [42] | 76.7/68.8 | 8.74M | 77
SECOND† [38] | 73.62/64.86 | 7.28M | 66.5
CenterPoints† [42] | 75.58/67.00 | 7.76M | 69.5
Point2Seq† (Ours) | 77.52/68.80 | 7.86M | 70.4
Table 3: Inference speed and parameters amount. ${\dagger}$: tested under the same environment using a single model on a V100 GPU. Backbone | Head | Veh. LEVEL 1 | Veh. LEVEL 2
---|---|---|---
mAP/mAPH(%) | mAP/mAPH(%)
Pillar | Anchor | 63.31/62.74 | 55.24/54.72
Center | 65.46/64.66 | 57.59/56.88
Point2Seq | 69.01/68.25 | 60.72/60.03
Voxel | Anchor | 73.62/73.14 | 64.86/64.40
Center | 75.58/75.01 | 67.00/66.52
Point2Seq | 77.52/77.03 | 68.80/68.36
Table 4: Performances on different backbone networks.

The order of words in a sequence. We explored the influence of changing the
predicted word orders in our method. The results in Table 6 indicate that the
order of words plays a non-negligible role on the detection performance. For
example, putting $W^{O}$ at the $4$th place greatly reduces the detection
accuracy, which may indicate the importance of predicting the object
orientation at an earlier position. Putting $W^{C}$ at the end will exhibit
better performance compared to putting $W^{C}$ at the beginning.

The choice of different similarity metrics. We evaluated different formulas of
the similarity metric used in similarity-based sequence matching. Table 7
exhibits the results of the $3$ formulas we have examined. The formula (3) is
the currently adopted similarity metric. The formula (2) replaces the term
$e^{-(1-\alpha)\sum_{j\in\\{R,O,S,L\\}}|W^{j}-\tilde{W}^{j}|}$ in Eq. 10 with
$e^{-(1-\alpha)3DIoU(B,\tilde{B})}$ where $3DIoU(B,\tilde{B})$ computes the 3D
IoU score of two bounding boxes. The formula (1) replaces the same term with
$e^{-(1-\alpha)\sum_{j=1}^{8}|C^{j}-\tilde{C}^{j}|}$, where we calculate the
differences of $8$ respective corners $C$ of two bounding boxes. The results
indicate that Eq. 10 is the best among the $3$ formulas as the similarity
metric.

##  5 Conclusion

We present Point2Seq, an effective and general 3D object detection framework
that can be applied to most grid-based backbone networks. Point2Seq contains a
scene-to-sequence decoder, which can auto-regressively generate sequences
describing the detected 3D objects, and similarity-based sequence matching is
proposed to enable end-to-end training without human-designed label
assignments. For future works, we plan to extend our framework to multi-
modality 3D object detection.

Assignment | Module | Veh. LEVEL 1 | Veh. LEVEL 2
---|---|---|---
mAP/mAPH(%) | mAP/mAPH(%)
Anchor | C.H | 73.62/73.14 | 64.86/64.40
Center | C.H | 75.58/75.01 | 67.00/66.52
S.S.M. | C.H | 76.72/76.19 | 68.02/67.54
S.S.M. | S.S.D | 77.52/77.03 | 68.80/68.36
Table 5: Effects of different components in Point2Seq. C.H: Convolutional Head in previous works. S.S.M.: Similarity-based Sequence Matching. S.S.D: Scene-to-Sequence Decoder. Order | Veh. LEVEL 1 | Veh. LEVEL 2
---|---|---
mAP/mAPH(%) | mAP/mAPH(%)
$W^{R},W^{O},W^{S},W^{L},W^{C}$ | 77.40/76.91 | 68.71/68.27
$W^{R},W^{O},W^{L},W^{S},W^{C}$ | 77.43/76.94 | 68.74/68.30
$W^{R},W^{L},W^{O},W^{S},W^{C}$ | 77.52/77.03 | 68.80/68.36
$W^{R},W^{L},W^{S},W^{O},W^{C}$ | 73.82/73.24 | 66.16/65.61
$W^{R},W^{C},W^{L},W^{O},W^{S}$ | 75.96/75.41 | 67.26/66.76
$W^{R},W^{C},W^{S},W^{O},W^{L}$ | 74.52/73.99 | 66.86/66.35
$W^{R},W^{C},W^{O},W^{L},W^{S}$ | 77.05/76.54 | 68.37/67.90
$W^{R},W^{C},W^{O},W^{S},W^{L}$ | 76.82/76.28 | 68.11/67.62
Table 6: Effects of different word orders in the sequences. Metrics | Veh. LEVEL 1 | Veh. LEVEL 2
---|---|---
mAP/mAPH(%) | mAP/mAPH(%)
(1) | 74.31/73.52 | 65.61/65.04
(2) | 75.07/74.43 | 66.36/65.73
(3) | 75.40/74.81 | 66.76/66.23
Table 7: Comparisons of different similarity metrics.

## Acknowledgement

This work is supported by NUS Faculty Research Committee Grant (WBS:
A-0009440-00-00), and NRF Centre for Advanced Robotics Technology Innovation
(CARTIN) Project.

## References

  * [1] Alex Bewley, Pei Sun, Thomas Mensink, Dragomir Anguelov, and Cristian Sminchisescu.  Range conditioned dilated convolutions for scale invariant 3d object detection.  arXiv preprint arXiv:2005.09927, 2020.
  * [2] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  arXiv preprint arXiv:2005.14165, 2020.
  * [3] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In European Conference on Computer Vision, pages 213–229. Springer, 2020.
  * [4] Qi Chen, Lin Sun, Ernest Cheung, and Alan L Yuille.  Every view counts: Cross-view consistency in 3d object detection with hybrid-cylindrical-spherical voxelization.  Advances in Neural Information Processing Systems, 2020.
  * [5] Ting Chen, Saurabh Saxena, Lala Li, David J Fleet, and Geoffrey Hinton.  Pix2seq: A language modeling framework for object detection.  arXiv preprint arXiv:2109.10852, 2021.
  * [6] Jiajun Deng, Shaoshuai Shi, Peiwei Li, Wengang Zhou, Yanyong Zhang, and Houqiang Li.  Voxel r-cnn: Towards high performance voxel-based 3d object detection.  arXiv:2012.15712, 2020.
  * [7] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [8] Lue Fan, Xuan Xiong, Feng Wang, Naiyan Wang, and Zhaoxiang Zhang.  Rangedet: In defense of range view for lidar-based 3d object detection.  arXiv preprint arXiv:2103.10039, 2021.
  * [9] Runzhou Ge, Zhuangzhuang Ding, Yihan Hu, Yu Wang, Sijia Chen, Li Huang, and Yuan Li.  Afdet: Anchor free one stage 3d object detection.  arXiv preprint arXiv:2006.12671, 2020.
  * [10] Benjamin Graham, Martin Engelcke, and Laurens Van Der Maaten.  3d semantic segmentation with submanifold sparse convolutional networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9224–9232, 2018.
  * [11] Chenhang He, Hui Zeng, Jianqiang Huang, Xian-Sheng Hua, and Lei Zhang.  Structure aware single-stage 3d object detection from point cloud.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11873–11882, 2020.
  * [12] Harold W Kuhn.  The hungarian method for the assignment problem.  Naval research logistics quarterly, 2(1-2):83–97, 1955.
  * [13] Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom.  Pointpillars: Fast encoders for object detection from point clouds.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12697–12705, 2019.
  * [14] Zhidong Liang, Ming Zhang, Zehan Zhang, Xian Zhao, and Shiliang Pu.  Rangercnn: Towards fast and accurate 3d object detection with range image representation.  arXiv preprint arXiv:2009.00206, 2020.
  * [15] Huihui Liu, Yiding Yang, and Xinchao Wang.  Overcoming catastrophic forgetting in graph neural networks.  In AAAI Conference on Artificial Intelligence, 2021.
  * [16] Jiageng Mao, Minzhe Niu, Haoyue Bai, Xiaodan Liang, Hang Xu, and Chunjing Xu.  Pyramid r-cnn: Towards better performance and adaptability for 3d object detection.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2723–2732, 2021.
  * [17] Jiageng Mao, Minzhe Niu, Chenhan Jiang, Hanxue Liang, Jingheng Chen, Xiaodan Liang, Yamin Li, Chaoqiang Ye, Wei Zhang, Zhenguo Li, et al.  One million scenes for autonomous driving: Once dataset.  arXiv preprint arXiv:2106.11037, 2021.
  * [18] Jiageng Mao, Xiaogang Wang, and Hongsheng Li.  Interpolated convolutional networks for 3d point cloud understanding.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1578–1587, 2019.
  * [19] Jiageng Mao, Yujing Xue, Minzhe Niu, Haoyue Bai, Jiashi Feng, Xiaodan Liang, Hang Xu, and Chunjing Xu.  Voxel transformer for 3d object detection.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3164–3173, 2021.
  * [20] Gregory P. Meyer, Ankita Gajanan Laddha, Eric Kee, Carlos Vallespi-Gonzalez, and Carl K. Wellington.  Lasernet: An efficient probabilistic 3d object detector for autonomous driving.  2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12669–12678, 2019.
  * [21] Ishan Misra, Rohit Girdhar, and Armand Joulin.  An end-to-end transformer model for 3d object detection.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2906–2917, 2021.
  * [22] Xuran Pan, Zhuofan Xia, Shiji Song, Li Erran Li, and Gao Huang.  3d object detection with pointformer.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7463–7472, 2021.
  * [23] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas.  Pointnet: Deep learning on point sets for 3d classification and segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652–660, 2017.
  * [24] Hualian Sheng, Sijia Cai, Yuan Liu, Bing Deng, Jianqiang Huang, Xian-Sheng Hua, and Min-Jian Zhao.  Improving 3d object detection with channel-wise transformer.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 2743–2752, October 2021.
  * [25] Shaoshuai Shi, Chaoxu Guo, Li Jiang, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li.  Pv-rcnn: Point-voxel feature set abstraction for 3d object detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10529–10538, 2020.
  * [26] Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li.  Pointrcnn: 3d object proposal generation and detection from point cloud.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 770–779, 2019.
  * [27] Shaoshuai Shi, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li.  From points to parts: 3d object detection from point cloud with part-aware and part-aggregation network.  IEEE transactions on pattern analysis and machine intelligence, 2020\.
  * [28] Weijing Shi and Raj Rajkumar.  Point-gnn: Graph neural network for 3d object detection in a point cloud.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1711–1719, 2020.
  * [29] Peize Sun, Yi Jiang, Enze Xie, Zehuan Yuan, Changhu Wang, and Ping Luo.  Onenet: Towards end-to-end one-stage object detection.  arXiv preprint arXiv:2012.05780, 2020.
  * [30] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al.  Scalability in perception for autonomous driving: Waymo open dataset.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2446–2454, 2020.
  * [31] Pei Sun, Weiyue Wang, Yuning Chai, Gamaleldin Elsayed, Alex Bewley, Xiao Zhang, Cristian Sminchisescu, and Dragomir Anguelov.  Rsn: Range sparse net for efficient, accurate lidar 3d object detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5725–5734, 2021.
  * [32] Peize Sun, Rufeng Zhang, Yi Jiang, Tao Kong, Chenfeng Xu, Wei Zhan, Masayoshi Tomizuka, Lei Li, Zehuan Yuan, Changhu Wang, et al.  Sparse r-cnn: End-to-end object detection with learnable proposals.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14454–14463, 2021.
  * [33] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Advances in neural information processing systems, pages 5998–6008, 2017.
  * [34] Jianfeng Wang, Lin Song, Zeming Li, Hongbin Sun, Jian Sun, and Nanning Zheng.  End-to-end object detection with fully convolutional network.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15849–15858, 2021.
  * [35] Yue Wang, Alireza Fathi, Abhijit Kundu, David A Ross, Caroline Pantofaru, Tom Funkhouser, and Justin Solomon.  Pillar-based object detection for autonomous driving.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXII 16, pages 18–34. Springer, 2020.
  * [36] Yue Wang and Justin Solomon.  Object dgcnn: 3d object detection using dynamic graphs.  arXiv preprint arXiv:2110.06923, 2021.
  * [37] Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon.  Dynamic graph cnn for learning on point clouds.  Acm Transactions On Graphics (tog), 38(5):1–12, 2019.
  * [38] Yan Yan, Yuxing Mao, and Bo Li.  Second: Sparsely embedded convolutional detection.  Sensors, 18(10):3337, 2018.
  * [39] Yiding Yang, Zunlei Feng, Mingli Song, and Xinchao Wang.  Factorizable graph convolutional networks.  In Advances in Neural Information Processing Systems, 2020.
  * [40] Yiding Yang, Jiayan Qiu, Mingli Song, Dacheng Tao, and Xinchao Wang.  Distilling knowledge from graph convolutional networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
  * [41] Zetong Yang, Yanan Sun, Shu Liu, and Jiaya Jia.  3dssd: Point-based 3d single stage object detector.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11040–11048, 2020.
  * [42] Tianwei Yin, Xingyi Zhou, and Philipp Krahenbuhl.  Center-based 3d object detection and tracking.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11784–11793, 2021.
  * [43] Yin Zhou, Pei Sun, Yu Zhang, Dragomir Anguelov, Jiyang Gao, Tom Ouyang, James Guo, Jiquan Ngiam, and Vijay Vasudevan.  End-to-end multi-view fusion for 3d object detection in lidar point clouds.  In Conference on Robot Learning, pages 923–932. PMLR, 2020.
  * [44] Yin Zhou and Oncel Tuzel.  Voxelnet: End-to-end learning for point cloud based 3d object detection.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4490–4499, 2018.
  * [45] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai.  Deformable detr: Deformable transformers for end-to-end object detection.  arXiv preprint arXiv:2010.04159, 2020.