# Explaining Autonomous Driving Actions with Visual Question Answering

Shahin Atakishiyev1, Mohammad Salameh2, Housam Babiker1, Randy Goebel1 This
work was supported by the Alberta Machine Intelligence Institute (Amii),
Computing Science Department of the University of Alberta, and the Natural
Sciences and Engineering Research Council of Canada (NSERC).Shahin
Atakishiyev, Housam Babiker, and Randy Goebel are with the Department of
Computing Science, University of Alberta, Edmonton, Alberta, Canada. Mohammad
Salameh is with Huawei Technologies Canada Co., Ltd., Edmonton, Alberta,
Canada.Correspondence<EMAIL_ADDRESS>
###### Abstract

The end-to-end learning ability of self-driving vehicles has achieved
significant milestones over the last decade owing to rapid advances in deep
learning and computer vision algorithms. However, as autonomous driving
technology is a safety-critical application of artificial intelligence (AI),
road accidents and established regulatory principles necessitate the need for
the explainability of intelligent action choices for self-driving vehicles. To
facilitate interpretability of decision-making in autonomous driving, we
present a Visual Question Answering (VQA) framework, which explains driving
actions with question-answering-based causal reasoning. To do so, we first
collect driving videos in a simulation environment using reinforcement
learning (RL) and extract consecutive frames from this log data uniformly for
five selected action categories. Further, we manually annotate the extracted
frames using question-answer pairs as justifications for the actions chosen in
each scenario. Finally, we evaluate the correctness of the VQA-predicted
answers for actions on unseen driving scenes. The empirical results suggest
that the VQA mechanism can provide support to interpret real-time decisions of
autonomous vehicles and help enhance overall driving safety.

##  I INTRODUCTION

Urban autonomous driving is one of the most challenging tasks for self-driving
vehicles, especially considering the potential interaction with other cars,
road-crossing pedestrians, bystanders, traffic lights, and other conditions of
dynamically changing environments. As highly automated vehicles increasingly
rely on mapping sensory data to control the commands of a vehicle, applicable
end-to-end learning techniques should be acceptably safe and computationally
transparent. In particular, the remarkable success of deep learning and
computer vision algorithms has expedited progress in safe autonomous driving
on real roads and urban areas. For example, in February 2023, Waymo reported
that their autonomous vehicle drove more than one million rider-only miles
across several US cities with no reported injuries or events involving
vulnerable road participants [1]. The report also describes that Waymo’s
vehicle was involved in two accidents, where one of the accidents was caused
by the driver of another car being distracted by their phone while approaching
a red traffic light, according to Waymo’s claim. Moreover, other recently
reported traffic accidents

Figure 1: An example of the most probable answers with softmax probability
scores predicted by our VQA framework on the action of an ego vehicle.

with self-driving cars [2] and resulting fatalities call for a scrutinized
regulation of vehicle autonomy within a legal framework. Such road mishaps
trigger safety, transparency, and other legal culpability issues. Inherently,
a self-driving vehicle also needs to justify its temporal decisions with some
form of explanation. As self-driving decisions directly impact passengers on
board and other road users, consumers and transportation jurisdictions
intrinsically expect transparency and rely on the correctness of such
decisions. As a concrete example, the European Union (EU) adopted the General
Data Protection Regulation (GDPR) [3] that proposed a recital of the right of
explanation, which entitles consumers to receive an explanation on decisions
of autonomous systems. Article 22 of GDPR also describes general principles
regarding stakeholders’ rights and responsibilities to use automated decision-
making systems [4]. Thus, the need for the explainability of autonomous
driving decisions has legal, socio-technical, psychological, and philosophical
perspectives, in general [5, 6].
The delivery of explanations is another important topic in autonomous driving.
As both consumers and engaged technical people have different backgrounds and
knowledge about how self-driving cars work, it is necessary that explanations
are provided in accordance with an explanation receiver’s (i.e., an explainee)
relevant identity, as described in the recent surveys of [5, 6, 7]. In this
context, self-driving explanations must be correct, sufficiently informative,
and intelligible for explainees.
In this study, we propose a VQA-based explanation approach111The source code,
data, and related resources are available at https://github.com/Shahin-01/VQA-
AD. to justify RL-based self-driving decisions in a simulation environment. At
its core, VQA is a task in the intersection of natural language processing and
computer

Figure 2: The most common approaches to explaining autonomous driving actions.

vision, which produces an answer to a text-based question about an image [8].
Such an objective makes this reasoning technique intuitively applicable to
autonomous driving. When humans drive or are a passenger on board, they
inherently analyze real-time and upcoming traffic scenes and think about
relevant causal, temporal, and descriptive questions, such as “Why is the car
turning left?”, “What action will the car in the left lane perform at the
T-junction?” and “What is the speed of the vehicle in front?” as examples.
Getting answers to such questions by any means helps us have a reliable and
safe trip. In this regard, we leverage the VQA mechanism to pose a question on
an autonomous car’s chosen action within the driving scene and justify the
question with a causal answer reflecting the car’s decision-making in that
scenario.
Motivated by this point, we build our framework as follows. We train an RL
agent (i.e., an ego car) to operate in an autonomous driving environment and
record its decisions (actions) in correspondence to the video frames (states).
We then utilize a VQA system to justify actions of the ego car: the VQA
framework inputs an image frame with a question reflecting the action of the
car in the scene and tries to predict the relevant answer for such an action
(e.g., Figure 1).
Overall, the main contributions of our paper can be summarized as follows:

  * •

We present the first empirical study on explaining autonomous driving actions
with a VQA approach.

  * •

We release a dataset of image-question-answer triplets justifying an
autonomous car’s actions in the scene.

  * •

We show that connecting vision and natural language could rationalize an RL
agent’s decision-making in an intelligible way.

  * •

We propose further directions to develop more rigorous VQA frameworks for
explanatory self-driving actions.

The rest of the paper is organized as follows. In Section II, we provide
state-of-the-art explainability approaches for autonomous driving. We then
present details of the data generated by our RL agent and visual feature
extraction in Section III. Finally, we report empirical results and the
discussion of these results in Section IV and sum up the article with
concluding remarks and future directions.

##  II Related Work

Since Bojarski et al.’s [9] CNN-based end-to-end learning approach, the
autonomous driving community has shown increasing interest in interpreting
self-driving decisions. In general, primarily explored explanation provision
techniques for autonomous driving are visual, textual explanations, feature
importance scores, and hybrid or multimodal explanations comprising two or
more of these methods (see Figure 2 for the relevant classification).
Visual explanations in the context of autonomous driving identify which parts
of the perceived image (i.e., driving scene) have more influence on the
vehicle’s decision, as justifications for the performed action [7]. For
instance, a visual explanation can show an image of a red traffic light
captured by the vehicle’s video camera as a saliency map (i.e., a heatmap)
pointing out that the perception algorithm classified it as a primary reason
for stopping. In this context, Kim and Canny proposed a causal attention-based
visualization technique to show which groups of pixel values (i.e., blobs)
have a true causal impact on the model’s prediction [10]. After analyzing
attention maps in a post-hoc manner, they remove more than half of the blobs
and analyze the model’s output. The empirical results show that the network
produces more convincing and correct predictions in driving decisions, just
like real drivers do in a realistic environment. Furthermore, as an augmented
version of their initial work [9], Bojarski et al. developed VisualBackProp, a
saliency map-based visual explanation framework highlighting which sets of
input pixels have more influence on a vehicle’s decisions [11]. They show that
the VisualBackProp method correctly identifies the most important traffic
elements, such as lane markings and other cars in the scene, as a basis for
decision-making. In addition, VisualBackProp has been proven to be an
effective interpretable approach to detecting the failure cases [12] in the
original vision-based end-to-end learning method of [9].
Another popular vision-based rationalization technique uses the idea of
counterfactual visual explanations. These explanations aim to identify whether
changing some parts of the original image leads to a different prediction than
the original prediction made on the original input. Bansal et al. [13]
modified hand-crafted inputs by removing some objects in the image to see
whether their introduced ChauffeurNet makes different predictions with the
altered image. A similar strategy is followed by Li et al. [14], where the
goal is to find the risk objects for driving. They show that manipulated

Figure 3: An aerial view of Town 1 and 2 on the CARLA simulator [15].

removal of a pedestrian in an intersection changes the driving command from
“Stop” to “Go”; thus, the pedestrian is considered a “risk object”, which
causes the driving decision to change to the contrastive class. Finally, as a
more recent counterfactual analysis, Jacob et al. [16] investigated a style
modification of image regions on the driving model’s predictions. The
experimental study shows that their presented framework, STEEX, generates
counterfactual explanations in case of manual interventions to the driving
scene. Therefore, visual explanations can enable people to ensure that the
intelligent driving system accurately senses the operational environment.
Textual descriptions are another way of conveying rationales to the end-users
for driving decisions. This approach generates natural language text that
explains driving actions with descriptive, temporal, and causal information.
The first successful textual explanation work is Kim et al.’s study [17],
where the authors leverage an attention-based video-to-text approach to
generate textual explanations for autonomous vehicles. They further extend
this work by incorporating human advice [18] and observation-to-action rules
[19] to the underlying model and provide text-based explanations on performed
actions. In another study, Xu et al. introduce BDD-OIA [20], an extension of
the BDD100K dataset. Based on the action-inducing objects, they provide 21
explanations for a set of 4 actions (move forward, stop/slow down, turn left,
turn right). Lastly, in this context, Ben Younes et al. [21] proposed BEEF, an
architecture that explains the behavior of trajectory prediction with textual
justifications based on features fused from multi-levels, such as late
features comprising the system-wise decisions and spatio-temporal features
consisting of perceptual driving information.
Feature importance scores, as well-known quantitative evaluation metrics, have
also recently been investigated in various autonomous driving tasks. The
applications of these methods to autonomous driving include decision trees
[22], Shapley values [23], and partial dependence plots [24]. The primary goal
of these methods in self-driving is to understand the weights and
contributions of scene features used in predictive modeling across the
explored self-driving tasks.
Finally, except for visual, textual, and quantitative explanations, recent
studies have attempted to use multi-modal explanatory techniques to convey
information on the chosen course of actions of self-driving vehicles. For
example, in

Figure 4: State space representation of the ego car in its environment. An
ideal state is that the car follows the direction of the lane within the lane.

their two studies [25, 26], Schneider et al. propose a combination of visual,
textual, audio, light, and vibration feedback to provide retrospective and
live explanations on action decisions of autonomous driving. These studies
show that while visualization and light-based driving information improve the
user experience (UX), multi-modal explanations can enhance perceived control
and understanding of a vehicle’s decision-making by connecting UX, autonomous
driving, and explainable AI. Moreover, real-time driving information delivered
via vibration, tactile sensation, or haptic feedback with a relevant degree of
an alert may have a crucial role in the smooth and timely transfer of control
between a self-driving vehicle and a backup driver.
With the inherent ability to reason about visual information, such as images,
videos, and related multimedia data, VQA has recently been explored in several
safety-critical and security-concerning domains. These works include
applications to the healthcare and medical field [27, 28] and visual
surveillance [29]. Interestingly, the topic has not been investigated deeply
in autonomous driving. As far as we know, there are only two instances that
utilize the VQA mechanism in the transportation domain. The first is the
CLEVRER dataset, which describes the collision events with video
representation and reasoning [30]. The other contribution is the SUTD-
TrafficQA benchmark, which basically predicts traffic situations with
question-answer pairs ranging from basic understanding (i.e., What is the type
of the road?) to reverse reasoning (i.e., What might have happened moments
ago?) [31]. In our study, we focus on action-based explanations and show that
question-answering-based causal event reasoning has significant benefits for
explaining real-time decisions of self-driving cars. We describe the details
of the framework and the experimental results in the following sections.

##  III Experimental Design and Methodology

Our framework is designed in three primary steps. First, we use a deep RL
agent to control an autonomous car in a simulation environment and collect a
driving video from

TABLE I: Annotated question-answer pairs in our VQA framework Action category |  Question | Answer
---|---|---
Go straight |  Why is the car going straight? | Because the road is clear.
Turn left |  Why is the car turning to the left? | Because the road is bending to the left.
Turn left at T-junction |  Why is the car turning left at T-junction? | Because there is no obstacle on the right side and turning left can be performed safely.
Turn right |  Why is the car turning to the right? | Because the road is bending to the right.
Turn right at T-junction |  Why is the car turning right at T-junction? | Because there is no obstacle on the left side and turning right can be performed safely.
Figure 5: Learning curve of DDPG in Town 1 with the specified parameters. Our
VQA framework is further fine-tuned on driving data collected here.

the simulator. We then convert this recorded video to image sequences
uniformly. Finally, we select five specific action categories in the extracted
driving frames and annotate them using question-answer pairs that justify the
car’s action in the scene (Table I). The high-level description of the
components and overall architecture is provided in Figure 6. Given such a
setup, the objective of our architecture is to predict the correct answer to a
posed question about an autonomous car’s performed action in an unseen driving
scene. The details of the data collection, data annotation, and question-
answering steps are described in the following subsections.

###  III-1 Data Collection

To obtain driving data, we trained an RL agent (i.e., a self-driving car) on
the CARLA simulator [15]. We used the Deep Deterministic Policy Gradient
(DDPG) algorithm [32] for the control of a self-driving car in a simulation
environment. Control commands of automated driving have continuous actions
including braking, acceleration, and steering angle which themselves can have
a broad range of values as a representation. DDPG, as an augmented version of
the Deep Q-learning algorithm, is particularly well-adapted for continuous
action spaces and therefore is appropriate for driving control tasks.
Furthermore, DDPG uses experience replay, a memory storing the agent’s past
experiences ($s_{t},a_{t},r_{t},s_{t+1}$), out of which the algorithm can
sample randomly to train the agent. This ability to reuse samples makes DDPG a
computationally efficient learning approach. Moreover, DDPG has an actor-
critic architecture, in which the actor learns an observation-to-action
mapping, and the critic learns to evaluate the quality of an agent’s chosen
actions. DDPG also uses target networks \- the target actor network
$\mu^{\prime}$, and target critic networks $Q^{\prime}$. These networks are
time-delayed copies of their original networks that help stabilize the
training process. The parameters of target networks are updated as follows:

TABLE II: The training parameters of DDPG on CARLA Actor learning rate |  Critic learning rate |  Target network hyper-parameter |  Replay buffer size |  Batch size |  Discount factor
---|---|---|---|---|---
0.0001 |  0.001 |  0.001 |  100000 |  32 |  0.99

$\theta^{Q^{\prime}}\leftarrow\tau\theta^{Q}+(1-\tau)\theta^{Q^{\prime}}$ (1)

$\theta^{\mu^{\prime}}\leftarrow\tau\theta^{\mu}+(1-\tau)\theta^{\mu^{\prime}}$
(2)

where $\tau$ $\ll$ 1\. For an effective action exploration, the term additive
noise is usually added to the exploration policy and action is selected
accordingly:

$a_{t}=\mu(s_{t}|\theta^{\mu})+\mathcal{N}_{t}$ (3)

Such a learning technique enables the DDPG agent to learn a policy that
maximizes its expected reward while also considering the quality of the chosen
actions.
RL Training Details: We generated driving data by training the agent on Town 1
within CARLA. Town 1 (see Figure 3, a) is a map containing straight lines,
left turns, right turns, T-junctions, traffic lights, speed signs, and various
stationary objects around the curbs. We first use the A* motion planning
algorithm [33] to generate a route with an initial and final point of a motion
trajectory inside the simulated town, which shows consecutive waypoints
linking these points. In our experiment, we set the number of waypoints to 15.
By

Figure 6: A diagram of the proposed VQA architecture for autonomous driving.

default, the waypoints are referenced to the origin point (0,0,0) in the map.
To ensure that they are referenced to the dynamic position of the self-driving
car while in motion, we use Perez et al.s’ methodology [34] and apply a
transformation matrix to represent the state of the agent with these points,
the vehicle’s yaw angle, and its global position on the map as follows:

$\begin{bmatrix}\@add@centering\centering\cos\phi_{c}&-\sin\phi_{c}&0&X_{c}\\\
\sin\phi_{c}&\cos\phi_{c}&0&Y_{c}\\\ 0&0&1&Z_{c}\\\ 0&0&1&1\end{bmatrix}$ (4)

The goal of the task is that the ego car follows this predefined route and
reaches the final destination by performing the relevant actions along its
trip.
As seen from Figure 4, the agent acquires a driving vector $f_{t}$= ($v_{t}$,
$d_{t}$, $\phi_{t}$) from the simulation environment where these parameters
reflect the vehicle’s velocity, lateral distance, and yaw angle, respectively.
Ideally, the goal of driving is to move on in the direction of the lane as
long as possible without lane departure and collisions. In this sense, the
reward shaping can be conditioned for the vehicle’s 1) perfect longitudinal
direction, 2) deviation from the lane direction with yaw angle, and 3) lane
departure and collision. Based on these criteria, we adopt the relevant reward
formulation from Perez et al. [34] for an ego car:

$R=\begin{cases}-200&\text{road departures or collisions},\\\
\sum_{t}\left|v_{t}\cos\phi_{t}\right|-\left|v_{t}\sin\phi_{t}\right|-\left|v_{t}\right|\left|d_{t}\right|&\text{
driving inside the lane},\\\ +100&\text{ arriving at the goal
position}.\end{cases}$

(5)

Finally, action space is continuous and can receive values from the interval
[-1,1]. By defining this setting, we trained our agent in Town 1. The training
parameters of DDPG can be seen in Table II.

Figure 7: Example scenarios from an ego vehicle’s field of view on CARLA. During the decision-making process of the agent, we are given visual signals and we ask action-related questions and try to find an answer given the current state. The green arrow shows the ego car’s chosen action and the white arrows indicate the other route at T-junction scenarios. We show the top 5 answers predicted by our model. The green-colored text shows the correct answer to the question for the performed action of the car. Except for the turn left scenario, justifications for other actions are predicted correctly by the model. TABLE III: Number of correct predictions for each action category Go straight | Turn left | Turn left at T-junction | Turn right | Turn right at T-junction | Total
---|---|---|---|---|---
20/20 | 0/20 | 20/20 | 20/20 | 20/20 | 80/100

###  III-2 Data Annotation

As we obtained the driving video with the DDPG agent, we selected 5 action
categories (go straight, turn left, turn right, turn left at T-junction, and
turn right at T-junction), and extracted consecutive frames uniformly (30
frames per second) for 5 video segments. We then chose 10 frames from each
segment. We ensured that these frames were extracted from driving segments,
where the car followed the predefined route and performed the relevant action
safely without lane departure or collision. We distinguish left and right
turns in the current line from left and right turns at T-junction as in the
latter an ego car also has an alternative route. So, our training data
includes 5 action categories with 50 high-quality frames per category,
denoting a total of 250 driving scenes obtained from the recorded video. We
manually annotated the training data with five causal question-answer (QA)
pairs (see Table I) ensuring the annotations reflected the scene correctly.
Each of 250 frames has its single and scenario-related annotation. As test
data, we selected a collection of 100 frames from both Town 1 and Town 2 on
the CARLA simulator, as the map of Town 2 is similar to Town 1. Similar to the
training data annotation, we selected 20 frames for each action category from
various segments of Town 1 and Town 2 and annotated each of them with a
relevant QA pair. The goal is to assess the generalization ability of the
employed VQA framework on these action categories in unseen traffic scenarios.

###  III-3 Question-Answering Framework

On the question-answering side, we fine-tune the original VQA framework [35]
trained on the MS COCO dataset [36]. At the highest level, our VQA model takes
an encoded driving image and a question embedding as input, to predict the
answer (i.e., explanation) for a performed action in the scene. The model is
composed of two neural networks. The first one is a multilayer-feedforward
network with 2 hidden layers each containing 1000 hidden units and tanh
activation function. We apply the dropout regularization with 0.5 in each
layer. Finally, a long short-term memory (LSTM) [37] followed by a softmax
layer is employed to produce an answer for the asked question about the
driving action. On the image encoding part, we eliminate the output layer and
use the last hidden layer of the pre-trained VGG-19 architecture [38],
producing a 4096-dimensional feature vector. Further, a linear transformation
is applied to make the image features 1024-dimensional. The LSTM model for the
question encoder has 2 hidden layers with 512 hidden units, and thus it is a
1024-dimensional vector, the same as image features. An interesting aspect is
the unification of the question and image vectors from a mathematical
perspective. Previous studies have generally either preferred the
concatenation or multiplication of these vectors, but [35] and [39] have shown
that multiplying the image and question encoder usually leads to a better
joint representation. Consequently, given the image vector, $V_{i}$, and
question embedding $V_{q}$, the resulting vector passed to the fully connected
layer of the VQA pipeline is represented as their element-wise multiplication,
as a fused feature vector:

$V_{r}=V_{i}\times V_{q}$ (6)

We use the question and answer vocabularies of the original VQA framework,
which have sizes of more than 17K unique tokens and 1000 candidate answers
(which are either single tokens such as “yes,” “white,” or expressions
consisting of two or more strings such as “playing video game”), respectively,
obtained by descriptions from the MS COCO [36] images. We customize candidate
answers by adding our answers of 5 action questions to that answer vocabulary.
The expectation is that our VQA model picks the most correct answer with the
highest softmax probability score out of the 1K candidates for the asked “Why”
question about the action within the driving scene.

##  IV Experimental Results and Discussion

On the data collection side, we trained the DDPG agent on the CARLA 0.9.11
version in 500 episodes using a TensorFlow backend to get a driving video. As
described above, we used 250 frames from Town 1 for training our VQA network
and evaluated its performance on 100 frames collected from Town 1 and Town 2
(Figure 3). We used the PyTorch backend for training and evaluating our VQA
architecture. The experiments were performed on an NVIDIA RTX 3090 GPU machine
with a 32 GB memory size. All the frames were set to have a size of 640
$\times$ 480 both in training and test. As we have ground-truth answers (see
Table I) for the asked question about an image, we compare the top prediction
of our model on the test data (i.e., an answer with the highest softmax
probability score) with these ground-truth answers. Thus, we use accuracy as
an evaluation metric, which is defined as follows:

Figure 8: The average softmax probability scores for top predictions in each
action category.

$\text{Accuracy}=\frac{\\#~{}frames~{}with~{}correct~{}predictions}{total~{}number~{}of~{}test~{}frames}$
(7)

Based on this evaluation criterion, our VQA model predicted 80 correct answers
to the asked questions for 100 images. Hence, the accuracy of the prediction
is 0.8 or 80%.

###  IV-1 Discussion

Except for turn left actions, our model predicted explanatory answers
correctly for all remaining action classes. Interestingly, in frames with
turn-left scenarios, the VQA framework primarily recognized these actions as
turn right. In Figure 7, we provide exemplary driving scenes for the five
action categories. As seen, the model was able to predict the highest
probability scores for all actions in the scenes correctly, except for the
misclassified turn left action in the second image. This misclassification
could be due to ambiguity in the tested driving frames, the shape of curves in
the scene, and road conditions in the training data. Hence, it is important to
increase the size of the training data considering the shapes of road lanes
and curves, lighting, and other road objects to potentially improve the
accuracy of the predictions of the VQA network on self-driving actions.
Another implication of our work is that unifying computer vision with a
natural language provides an opportunity to explain temporal actions of an RL
agent. As explored in a recent study [40], explaining RL in sequential
decision-making problems is an important and emerging topic, particularly when
explanation receivers do not have a technical background. As autonomous
driving is a safety-critical application area, justifying reinforcement
learning-based decisions to end users with natural language-based reasoning is
an effective and easily understandable approach. A natural foundation for
explainable reinforcement learning (XRL) would be to provide reward-based
justifications on action decisions. However, as self-driving explanations are
intended for a general community, it is essential to ensure that such
explanations are intelligible and informative. While [40] has attempted to
build an inherently explainable RL architecture, we build our explanations
independent of an agent’s decisions. We also acknowledge the need to be
cautious about providing explanations that are independent of an agent’s
behavior; it is possible that post-hoc explanations may not always reflect an
agent’s real decision-making process. For example, in an actual left turn
scenario, a model’s response to the question “Why is the car turning to the
right?” as ”Because the road is bending to the right.” may be a hallucination
of a VQA architecture. Consequently, it is important to further investigate
the topic of generating linguistic explanations for an agent’s actions and
evaluate such explanations with human-adversarial examples as well.

###  IV-2 Limitations

Real roads are more complex and dynamic with the presence of traffic lights,
bystanders, passengers, other vehicles, and adverse weather conditions. In the
current version of our framework, the ego car only interacts with the
stationary environment and explains actions associated with such interactions.
Moreover, our dataset is small in size. Hence, these features are limitations
of our present framework, and as a next step, we plan to work on explaining
self-driving actions in more dynamic and complex scenarios with enriched data,
where details are provided in the conclusions section.

###  IV-3 Practical use cases

In practice, the VQA mechanism can be leveraged at least in two ways on real
autonomous vehicles. First, it can help passengers on board monitor driving
safety by “judging” the vehicle’s decisions. For instance, a user interface or
dashboard set up on a back seat may provide voice-to-text functionality, and a
passenger can observe driving surrounding, ask a question about the vehicle’s
chosen action, and get an answer. Such a feature can help monitor the
reliability of self-driving and instill trust in vehicle autonomy during the
trip. Another practical application is to retain a history of action-question-
answer triplets (…$a_{t},q_{t},ans_{t},a_{t+1},q_{t+1},ans_{t+1}...$) and use
it for forensic analysis in possible accident investigations with self-driving
vehicles. Such log data can help understand why the self-driving vehicle made
a specific decision at a particular time just before being involved in an
accident.

##  V Conclusions and Future Work

We have presented a preliminary study on explaining autonomous driving actions
with a VQA approach. We used driving data generated by an RL agent on the
CARLA simulator and developed our question-answering system as an explanatory
approach to the agent’s decisions. The experimental results show that a simple
and straightforward VQA mechanism can help interpret the real-time decisions
of an autonomous car and also help understand its correct and incorrect
decisions as safety implications. The results also suggest that unifying VQA
with RL-based decision-making will likely do well for actions in a dynamic
environment, provided that we have more training dataset. In this sense, we
plan to explore three potential directions:
1\. Augmenting data and using other VQA architectures: We will increase the
size of training data (ideally $>$50K driving frames), perform fine-tuning on
our model using more recent ConvNet architectures, such as Vision Transformer
(ViT) [41], and try out other VQA frameworks as well. By making a comparative
analysis of these pre-trained deep neural architectures on driving data, we
can observe their empirical performance in terms of accuracy and potentially
produce a large-scale and curated benchmark dataset.
2\. Training an RL agent on dynamic environments: We will run the RL agent in
other towns on the CARLA simulator, which have more vehicles, pedestrians, and
complex intersections, and annotate the ego vehicle’s interaction with them
accordingly to provide more image-question-answer triplets.
3\. Leveraging large language models (LLMs): Finally, a recent breakthrough in
LLMs gives a reason to use this architecture in autonomous driving problems.
As our current task combines vision and natural language-based reasoning for
explaining self-driving actions, multimodal transformers (e.g., GPT-4 [42] and
its variations) could serve a purpose in this context. As multimodal
transformers can input an image and text, and provide contextual information
about the joint semantics, it seems promising to fine-tune such state-of-the-
art learning architectures for the self-driving domain and generate rigorously
structured explanations.
We believe that the empirical work and further directions proposed in this
paper can help improve safety, transparency, and trustworthiness of autonomous
driving technology.

## References

  * [1] Waymo’s Blog, “First Million Rider-Only Miles: How the Waymo Driver is Improving Road Safety,” Accessed online on March 12, 2023. [Online]. Available: https://blog.waymo.com/2023/02/first-million-rider-only-miles-how.html
  * [2] NTS Board, “Collision between a sport utility vehicle operating with partial driving automation and a crash attenuator Mountain View, California.” Accessed online on March 9, 2023.
  * [3] P. Voigt and A. Von dem Bussche, “The EU General Data Protection Regulation (GDPR),” _A Practical Guide, 1st Ed., Cham: Springer International Publishing_ , vol. 10, no. 3152676, pp. 10–5555, 2017.
  * [4] GDPR, “Art. 22 GDPR Automated individual decision-making, including profiling ,” (Accessed on March 9, 2023). [Online]. Available: https://gdpr-info.eu/art-22-gdpr/
  * [5] D. Omeiza, H. Webb, M. Jirotka, and L. Kunze, “Explanations in Autonomous Driving: A Survey,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 23, no. 8, pp. 10 142–10 162, 2021.
  * [6] S. Atakishiyev, M. Salameh, H. Yao, and R. Goebel, “Explainable Artificial Intelligence for Autonomous Driving: A Comprehensive Overview and Field Guide for Future Research Directions,” _arXiv preprint arXiv:2112.11561_ , 2021\.
  * [7] É. Zablocki, H. Ben-Younes, P. Pérez, and M. Cord, “Explainability of Deep Vision-Based Autonomous Driving Systems: Review and Challenges,” _International Journal of Computer Vision_ , vol. 130, no. 10, pp. 2425–2452, 2022.
  * [8] K. Kafle and C. Kanan, “Visual question answering: Datasets, algorithms, and future challenges,” _Computer Vision and Image Understanding_ , vol. 163, pp. 3–20, 2017.
  * [9] M. Bojarski, D. Del Testa, D. Dworakowski, B. Firner, B. Flepp, P. Goyal, L. D. Jackel, M. Monfort, U. Muller, J. Zhang, _et al._ , “End to end learning for self-driving cars,” _arXiv preprint arXiv:1604.07316_ , 2016.
  * [10] J. Kim and J. Canny, “Interpretable Learning for Self-Driving Cars by Visualizing Causal Attention,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2017, pp. 2942–2950.
  * [11] M. Bojarski, A. Choromanska, K. Choromanski, B. Firner, L. J. Ackel, U. Muller, P. Yeres, and K. Zieba, “VisualBackProp: Efficient Visualization of CNNs for Autonomous Driving,” in _2018 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2018, pp. 4701–4708.
  * [12] S. Mohseni, A. Jagadeesh, and Z. Wang, “Predicting model failure using saliency maps in autonomous driving systems,” _ICML 2019 Workshop on Uncertainty and Robustness in Deep Learning_ , 2019.
  * [13] M. Bansal, A. Krizhevsky, and A. Ogale, “ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst,” _Robotics: Science and Systems_ , 2019.
  * [14] C. Li, S. H. Chan, and Y.-T. Chen, “Who Make Drivers Stop? Towards Driver-centric Risk Assessment: Risk Object Identification via Causal Inference,” in _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2020, pp. 10 711–10 718.
  * [15] A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun, “CARLA: An Open Urban Driving Simulator,” in _Conference on Robot Learning_. PMLR, 2017, pp. 1–16.
  * [16] P. Jacob, É. Zablocki, H. Ben-Younes, M. Chen, P. Pérez, and M. Cord, “STEEX: Steering Counterfactual Explanations with Semantics,” in _Proceedings of the European Conference on Computer Vision (ECCV)_. Springer, 2022, pp. 387–403.
  * [17] J. Kim, A. Rohrbach, T. Darrell, J. Canny, and Z. Akata, “Textual Explanations for Self-Driving Vehicles,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 563–578.
  * [18] J. Kim, T. Misu, Y.-T. Chen, A. Tawari, and J. Canny, “Grounding Human-to-Vehicle Advice for Self-driving Vehicles,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 10 591–10 599.
  * [19] J. Kim, S. Moon, A. Rohrbach, T. Darrell, and J. Canny, “Advisable Learning for Self-Driving Vehicles by Internalizing Observation-to-Action Rules,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 9661–9670.
  * [20] Y. Xu, X. Yang, L. Gong, H.-C. Lin, T.-Y. Wu, Y. Li, and N. Vasconcelos, “Explainable Object-induced Action Decision for Autonomous Vehicles,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 9523–9532.
  * [21] H. Ben-Younes, É. Zablocki, P. Pérez, and M. Cord, “Driving behavior explanation with multi-level fusion,” _Pattern Recognition_ , vol. 123, p. 108421, 2022.
  * [22] C. Brewitt, B. Gyevnar, S. Garcin, and S. V. Albrecht, “GRIT: Fast, Interpretable, and Verifiable Goal Recognition with Learned Decision Trees for Autonomous Driving,” in _2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2021, pp. 1023–1030.
  * [23] Y. Almalioglu, M. Turan, N. Trigoni, and A. Markham, “Deep learning-based robust positioning for all-weather autonomous driving,” _Nature Machine Intelligence_ , vol. 4, no. 9, pp. 749–760, 2022.
  * [24] D. Lee, J. Mulrow, C. J. Haboucha, S. Derrible, and Y. Shiftan, “Attitudes on Autonomous Vehicle Adoption using Interpretable Gradient Boosting Machine,” _Transportation Research Record_ , vol. 2673, no. 11, pp. 865–878, 2019.
  * [25] T. Schneider, S. Ghellal, S. Love, and A. R. Gerlicher, “Increasing the User Experience in Autonomous Driving through different Feedback Modalities,” in _26th International Conference on Intelligent User Interfaces_ , 2021, pp. 7–10.
  * [26] T. Schneider, J. Hois, A. Rosenstein, S. Ghellal, D. Theofanou-Fülbier, and A. R. Gerlicher, “ExplAIn Yourself! Transparency for Positive UX in Autonomous Driving,” in _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_ , 2021, pp. 1–12.
  * [27] D. Gurari, Q. Li, A. J. Stangl, A. Guo, C. Lin, K. Grauman, J. Luo, and J. P. Bigham, “VizWiz Grand Challenge: Answering Visual Questions From Blind People,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 3608–3617.
  * [28] L.-M. Zhan, B. Liu, L. Fan, J. Chen, and X.-M. Wu, “Medical Visual Question Answering via Conditional Reasoning,” in _Proceedings of the 28th ACM International Conference on Multimedia_ , 2020, pp. 2345–2354.
  * [29] D. Li, Z. Zhang, K. Yu, K. Huang, and T. Tan, “ISEE: An Intelligent Scene Exploration and Evaluation Platform for Large-Scale Visual Surveillance,” _IEEE Transactions on Parallel and Distributed Systems_ , vol. 30, no. 12, pp. 2743–2758, 2019.
  * [30] K. Yi, C. Gan, Y. Li, P. Kohli, J. Wu, A. Torralba, and J. B. Tenenbaum, “CLEVRER: Collision Events for Video Representation and Reasoning,” in _International Conference on Learning Representations_ , 2020.
  * [31] L. Xu, H. Huang, and J. Liu, “SUTD-TrafficQA: A Question Answering Benchmark and an Efficient Network for Video Reasoning Over Traffic Events,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 9878–9888.
  * [32] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, “Continuous Control with Deep Reinforcement Learning,” _International Conference on Learning Representations_ , 2016\.
  * [33] P. E. Hart, N. J. Nilsson, and B. Raphael, “A Formal Basis for the Heuristic Determination of Minimum Cost Paths,” _IEEE Transactions on Systems Science and Cybernetics_ , vol. 4, no. 2, pp. 100–107, 1968.
  * [34] Ó. Pérez-Gil, R. Barea, E. López-Guillén, L. M. Bergasa, C. Gomez-Huelamo, R. Gutiérrez, and A. Diaz-Diaz, “Deep reinforcement learning based control for Autonomous Vehicles in CARLA,” _Multimedia Tools and Applications_ , vol. 81, no. 3, pp. 3553–3576, 2022.
  * [35] S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh, “VQA: Visual Question Answering,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2015, pp. 2425–2433.
  * [36] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft COCO: Common Objects in Context,” in _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13_. Springer, 2014, pp. 740–755.
  * [37] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” _Neural computation_ , vol. 9, no. 8, pp. 1735–1780, 1997.
  * [38] K. Simonyan and A. Zisserman, “Very Deep Convolutional Networks for Large-Scale Image Recognition,” _International Conference on Learning Representations_ , 2015.
  * [39] S. Garg and R. Srivastava, “Object sequences: encoding categorical and spatial information for a yes/no visual question answering task,” _IET Computer Vision_ , vol. 12, no. 8, pp. 1141–1150, 2018.
  * [40] X. Peng, M. Riedl, and P. Ammanabrolu, “Inherently Explainable Reinforcement Learning in Natural Language,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 16 178–16 190, 2022.
  * [41] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale,” in _International Conference on Learning Representations_ , 2021.
  * [42] OpenAI, “GPT-4 Technical Report,” https://arxiv.org/abs/2303.08774, 2023\.


# Accurate Instance-Level CAD Model Retrieval
in a Large-Scale Database

Jiaxin Wei1, Lan Hu1,2, Chenyu Wang1, Laurent Kneip1 1Mobile Perception Lab,
School of Information Science and Technology, ShanghaiTech University;
<EMAIL_ADDRESS>Institute of Microsystem
and Information Technology, Chinese Academy of Sciences and University of
Chinese Academy of Sciences<EMAIL_ADDRESS>
###### Abstract

We present a new solution to the fine-grained retrieval of clean CAD models
from a large-scale database in order to recover detailed object shape
geometries for RGBD scans. Unlike previous work simply indexing into a
moderately small database using an object shape descriptor and accepting the
top retrieval result, we argue that in the case of a large-scale database a
more accurate model may be found within a neighborhood of the descriptor. More
importantly, we propose that the distinctiveness deficiency of shape
descriptors at the instance level can be compensated by a geometry-based re-
ranking of its neighborhood. Our approach first leverages the discriminative
power of learned representations to distinguish between different categories
of models and then uses a novel robust point set distance metric to re-rank
the CAD neighborhood, enabling fine-grained retrieval in a large shape
database. Evaluation on a real-world dataset shows that our geometry-based re-
ranking is a conceptually simple but highly effective method that can lead to
a significant improvement in retrieval accuracy compared to the state-of-the-
art.

##  I INTRODUCTION

Nowadays, emerging robotics and augmented reality [3] applications have a huge
demand for real-time 3D reconstructions of object-level scene models from RGBD
scans. Traditional incremental fusion solutions [4][5] often fail to recover
detailed scene geometries due to the noisy nature of the measurements captured
by consumer-grade hardware. Instead of focusing on the exhaustive refinement
of 3D scans [6], an increasing number of approaches [7][1][8] turn to an
offline CAD database for better reconstructions. Given sufficient object
detection, segmentation, and representation abilities, the original incomplete
scans may be greatly improved by substituting corresponding measurement parts
with similar CAD models.

However, the imperfection of object detection and segmentation, together with
the incapability of 3D shape descriptors to encode precise geometry
information, jointly place a major restriction on CAD model retrieval at a
finer level. Current methods, such as [1][8], concentrate on computing the
9DoF alignment between scanned objects and CAD models while only requiring the
retrieval of a same-category model due to the usage of a small ground truth
pool. Zhao et al. [9] and Dahnert et al. [10] claim retrieval of CAD instances
with higher similarity, but they still constrain the model search to a
relatively small database containing an augmented ground truth set for each
scanned object. We instead focus on instance-level retrieval of CAD models in
a large shape database, which is more practical in real-world scenarios.

CAD models from the same category usually possess a universal geometric
structure. Therefore, it is quite easy for learned representations
[11][12][13] to achieve outstanding performance in 3D shape classification.
Yet, individual CAD models within the same class vary drastically in terms of
shape details, making it hard to differentiate them from one another. To this
end, we propose a two-step pipeline to tackle this challenging problem. We
first retrieve multiple CAD candidates from the shape descriptor space using
nearest neighbor search, and then use geometric residuals to re-rank those
candidates, thereby obtaining much better results both quantitatively and
qualitatively. Our method is based on two important insights. First, the
discriminative power of learned features can help narrow down the search to
models with the same semantic class as that of the scanned object. And a
suitable hierarchical data structure on top of our large-scale database makes
the search more efficiently. Second, after figuring out which part of a
database is of interest, we rely on point set distance metrics to further
perform a local search and refine the ranking order of initial candidates. We
also suggest a new variant of Chamfer Distance to deal with pathological cases
in evaluating candidates.

The main contributions of this paper are summarized as follows:

  * •

We present a new pipeline for instance-level CAD model retrieval at large
scale. It relies on a feature-based $k$-nearest neighbor search followed by a
geometry-based re-ranking, which can considerably improve the quality of
retrieved CAD models.

  * •

We introduce a variant of the Single-direction Chamfer Distance [9] that is
more robust to outliers when measuring point set distances between partial
scans and CAD models.

As demonstrated by our experiments, the combination of geometric re-ranking
outperforms the plain application of state-of-the-art feature embedding
networks. From a qualitative perspective, it also produces much more visually
similar results, and thus for the first time enables instance-level retrieval
of CAD models within a large-scale database.

##  II Related Work

CAD Model Retrieval: There are mainly two directions for 3D model retrieval:
view-based methods and model-based methods. View-based methods extract and
merge 2D features from multiple projections of a 3D object and merge those
features to generate a global shape descriptor for later recognition while
model-based methods directly deal with 3D data. View-based methods are out of
scope of this paper. The interested readers are referred to [14][15][16] for
more details.

Most traditional 3D model retrieval algorithms [17][18][19][20] are based on
hand-designed descriptors [21][22] that may easily suffer from the
inconsistency between clean CAD models and noisy real-world data. Meanwhile,
learning-based methods are getting more and more popular. Scan2CAD [1] carries
out implicit retrieval by evaluating the semantic compatibility between CAD
models and a patch from the scan. Avetisyan et al. [8] insert an extra object
detection module such that the relevant parts of a scan can be directly used
to calculate descriptors. Dahnert et al. [10] segment the foreground from the
input scan and complete the segmented objects by stacking hourglass encoder-
decoders, resulting in a joint embedding of 3D scans and CAD models. More
recently, CORSAIR [9] extends FCGF [23] to simultaneously learn a global
feature for retrieval as well as local point features for alignment.

The aforementioned methods have shown promising results yet are not practical.
They either assume a small pool of exact CAD models as input or consider a
pool with augmented ground truth models for each scanned object. In contrast,
our proposed method performs retrieval in a more realistic setting where the
database has thousands of models and a one-to-many relationship with scanned
objects.

3D Shape Descriptors: Given that there are plenty of excellent works for
instance segmentation [24][25][26], we can safely assume that objects have
already been segmented from scans. 3D model retrieval can thus be achieved by
learning a global shape descriptor for object scans, which has attracted
increasing attention over the past few years. For example, VoxNet [27] applies
a supervised 3D CNN on volumetric occupancy grids to predict object class
labels. Alternative approaches directly work on point clouds. PointNet [11]
extracts point features using shared MLPs and aggregates them with a max
pooling operator to obtain a permutation-invariant global feature. Later on,
Qi et al. propose PointNet++ [12] to further capture the local geometric
structures of point clouds from neighborhood points. In the spirit of graph
neural networks, DGCNN [13] constructs a local graph on the point cloud which
can be dynamically updated using aggregated edge features after each layer of
the network. As a generalization of 2D CNN, PointCNN [2] uses
$\mathcal{X}$-Conv to transform the input point clouds and raw features into a
canonical form and then applies typical convolutions. However, those works
usually do not work well with noisy or partial measurements. We resort to
geometric information to reduce the uncertainty.

Set Similarity for Point Clouds: Fan et al. [28] propose two distance metrics
to compare generated point clouds against ground truth: Chamfer Distance (CD)
and Earth Mover’s Distance (EMD). EMD gives the minimum effort of transforming
one point set into the other and has been proven to be more accurate than CD
in point cloud generation [28][29]. The biggest issue impeaching EMD from
widespread use is the high computational cost for exact solutions. CD is
preferred [30][31][32] when efficiency is of major concern.

##  III Method

Figure 1: An illustration of our pipeline. CAD models are processed offline to
form a large database in which we can do $k$-nearest neighbor search for each
query object. Based on the coarsely retrieved CAD candidates, we further
perform geometry-based re-ranking using Modified Single-Direction Chamfer
distance (MSCD) to make sure that higher-ranked candidates are generally more
similar to the query object.

This section is organized as follows. Section III-A reveals our motivation of
establishing instance-level retrieval in a large CAD database and provides an
overview of our method. In Section III-B and III-C, we describe our two-step
pipeline in all details, comprising of feature-based ranking and geometry-
based re-ranking.

###  III-A Overview

The retrieval of a similar CAD model is tightly entangled with the cross-
instance alignment between a CAD model and a scanned object. With the rise of
category-level object alignment [33], it is possible to align different object
instances within a category. For ease of implementation, we simply assume that
all the retrieved CAD models are already aligned with the query object by
leveraging such alignment methods.

Avetisyan et al. [1][8] retrieve CAD models from a small pool tailored for
each scene. The model pool contains only the ground truth CAD models
corresponding to the objects that are effectively present in the scene. On the
contrary, we perform retrieval in a large-scale CAD database, which is more
closer to the real situation. We furthermore focus on instance-level rather
than category-level retrieval. For category-level retrieval, the matching of
categories between the retrieved CAD and the query object is enough to be
regarded as a success. However, real-world applications such as robot
interactions with surrounding environment require a detailed understanding of
object shapes. In other words, the retrieved CAD model should replicate the
object shape as closely as possible. In contrast to the works of Ishimtsev et
al. [34] and Uy et al. [35], our method avoids the involvement of expensive
CAD model deformation, and makes full use of a large-scale database to
identify the model that is most similar to the scanned object.

Our method breaks down into two procedures. First, we calculate a shape
descriptor using a neural network and then perform coarse retrieval by means
of $k$-nearest neighbour search to obtain an initial set of CAD candidates. It
points out regions of interest in the database that are highly likely to
contain the best shape. Second, we apply geometric verification to assess and
re-rank the initial candidates. As it is hard to evaluate the similarity
between retrieved CAD models and incomplete scanned objects, we introduce a
modified version of Chamfer Distance to better handle the outliers. This step
can significantly eliminate the uncertainty incurred by neural networks. In
addition, the refinement is quite efficient given that the number of CAD
candidates is far fewer that of models in the whole database. Our complete
pipeline is shown in Figure 1.

###  III-B Feature-Based Ranking

Although neural networks may not yet be able to capture the exact geometry of
an object, it is still easy for them to categorize shapes into the right
semantic classes. Consequently, we utilize the discriminative power of learned
global features to characterize inter-class dissimilarity. All the query
objects and CAD models are processed through the same backbone encoder. Note
that there are little constraints on the network architecture as long as it
outputs a global feature for each shape. Here we adopt neural networks
originally designed for 3D shape classification.

Given a query object $\mathbf{X}\in\mathbb{R}^{N\times 3}$ and a CAD database
$\mathcal{Y}=\\{\mathbf{Y}_{i}|\mathbf{Y}_{i}\in\mathbb{R}^{M_{i}\times
3},1\leq i\leq S\\}$ consisting of $S$ models, all represented in the form of
point clouds, our goal is to retrieve a CAD model $\mathbf{Y}\in\mathcal{Y}$
that is not only semantically compatible to $\mathbf{X}$ but also
geometrically close to it. Let $f(\cdot)$ denote the backbone encoder. It
takes as input a point cloud $\mathbf{P}$ and outputs a $d$-dimensional shape
descriptor

$f:\mathbf{P}\mapsto f(\mathbf{P})\in\mathbb{R}^{d}.$

Out of consideration for efficiency, features for all CAD models in the
database are pre-computed offline. Let
$\mathbf{F}^{\mathbf{X}}\in\mathbb{R}^{d}$ and
$\mathbf{F}^{\mathcal{Y}}=\\{\mathbf{F}^{\mathcal{Y}}_{i}|\mathbf{F}^{\mathcal{Y}}_{i}\in\mathbb{R}^{d},1\leq
i\leq S\\}$ be the extracted global features for the query object and the
models, respectively. Each time a new query arrives, we perform a $k$-nearest
neighbor search using the object feature to obtain an ordered candidate set:

$\mathcal{C}=\\{\mathbf{C}_{j}|\mathbf{C}_{j}\in\mathcal{Y},1\leq j\leq k\\}$

such that $d(\mathbf{F}^{\mathbf{X}},\mathbf{F}^{\mathcal{C}}_{j})$ is the
$j$-th smallest element in
$d(\mathbf{F}^{\mathbf{X}},\mathbf{F}^{\mathcal{Y}})$. Here $d(\cdot)$
represents the Euclidean distance between feature vectors. To reduce the
search complexity, we explore the database via hierarchical data structures,
such as a kd-tree.

###  III-C Geometry-Based Re-ranking

In most cases, the initial set may contain a very similar model but its
ranking order is unsatisfactory owing to the limited ability of learned
features to encode precise shape information. The distinctiveness can be
further brought down when some parts of the query object are occluded or
unobserved. Meanwhile, the bounded number of CAD models in the database means
that there usually do not exist identical matches. Therefore, we exploit
geometric information to address the intra-class similarity. Specifically, we
re-rank the initial CAD candidates by measuring the geometric difference
between point clouds.

Chamfer Distance (CD) is a popular point set distance metric used in many
learning based point cloud generation frameworks [28][29]. For two point
clouds $P$ and $Q$, a common definition of CD is to find the nearest neighbor
of $P$ in $Q$ and vice versa, and then add up all squared distances. That is,

$d_{CD}(P,Q)=\frac{1}{|P|}\sum_{p\in P}\min_{q\in
Q}\|p-q\|_{2}^{2}+\frac{1}{|Q|}\sum_{q\in Q}\min_{p\in P}\|p-q\|_{2}^{2},$ (1)

where $|P|$ and $|Q|$ are the number of points in $P$ and $Q$, respectively.
The division by $|P|$ and $|Q|$ acts as a normalization which can balance the
influence caused by different point densities in $P$ and $Q$. Note that the
term Chamfer Distance is a slight abuse of notation as it does not actually
satisfy the triangle inequality required by a proper distance metric [28].

An interesting variant of CD has been introduced in [9], called the Single-
direction Chamfer Distance (SCD). It simply drops one side of squared
distances in (1), resulting in

$d_{SCD}(P,Q)=\frac{1}{|P|}\sum_{p\in P}\min_{q\in Q}\|p-q\|_{2}^{2}.$ (2)

Hence, SCD only maintains the distances from scanned object to CAD model. It
is intuitively clear that this simplification helps SCD to concentrate on the
visible parts of a scanned object while neglecting unobserved parts.

For a better understanding of (2), let us define a distance vector

$\mathbf{d}=\left[\begin{matrix}\min_{q\in Q}\|p_{1}-q\|_{2}\\\ \min_{q\in
Q}\|p_{2}-q\|_{2}\\\ ...\\\ \min_{q\in
Q}\|p_{|P|}-q\|_{2}\end{matrix}\right]\in\mathbb{R}^{|P|}.$

Thus, SCD can be reformulated as

$d_{SCD}(P,Q)=\frac{1}{|P|}\|\mathbf{d}\|_{2}^{2},$ (3)

the $L_{2}$-norm of the distance vector $\mathbf{d}$. However, it is well
known from the literature of robust optimization that the $L_{2}$-norm is
sensitive to outliers as it emphasises on residuals by squaring them. In
practical scenarios, object scans often contain outlier measurements owing to
the existence of segmentation errors and other artifacts such as floating
point measurements. In order to improve the robustness of the above distance
against such artifacts, we propose to use the $L_{1}$-norm of $\mathbf{d}$
instead. We denote this distance the Modified Single-direction Chamfer
Distance (MSCD), and it is given by

$d_{MSCD}(P,Q)=\frac{1}{|P|}\sum_{p\in P}\min_{q\in Q}\|p-q\|_{2}.$ (4)

Though the change seems subtle, our experimental results demonstrate that MSCD
achieves better results in practice.

To conclude, we take each candidate CAD model from $\mathcal{C}$ and calculate
the MSCD with respect to our object scan, yielding a new ordered set

$\hat{\mathcal{C}}=\\{\hat{\mathbf{C}}_{j}|\hat{\mathbf{C}}_{j}\in\mathcal{C},1\leq
j\leq k\\}$

such that $d_{MSCD}(\mathbf{X},\hat{\mathbf{C}}_{j})$ is the $j$-th smallest
element in $d_{MSCD}(\mathbf{X},\mathcal{C})$.

##  IV Experiments

Table I: Top1/Top5 retrieval accuracy Method | bed | bookshelf | cabinet | chair | display | sofa | table | other | class avg. | instance avg.
---|---|---|---|---|---|---|---|---|---|---
VFH [36] | 0.00/0.00 | 0.00/0.02 | 0.00/0.00 | 0.01/0.02 | 0.01/0.03 | 0.00/0.01 | 0.01/0.04 | 0.00/0.00 | 0.00/0.01 | 0.01/0.02
DGCNN [13] | 0.04/0.17 | 0.02/0.10 | 0.00/0.06 | 0.01/0.04 | 0.01/0.04 | 0.01/0.07 | 0.01/0.04 | 0.06/0.24 | 0.25/0.47 | 0.02/0.09
VoxNet [27] | 0.12/0.37 | 0.06/0.17 | 0.02/0.06 | 0.02/0.05 | 0.03/0.08 | 0.04/0.12 | 0.03/0.10 | 0.07/0.22 | 0.22/0.50 | 0.04/0.11
PointNet2 [12] | 0.10/0.28 | 0.02/0.08 | 0.01/0.05 | 0.01/0.05 | 0.01/0.07 | 0.02/0.08 | 0.02/0.08 | 0.09/0.28 | 0.29/0.51 | 0.03/0.11
PointCNN [2] | 0.05/0.29 | 0.16/0.31 | 0.04/0.11 | 0.02/0.08 | 0.04/0.11 | 0.02/0.10 | 0.05/0.15 | 0.11/0.32 | 0.29/0.57 | 0.06/0.16
VFH+MSCD | 0.01/0.01 | 0.05/0.14 | 0.02/0.03 | 0.02/0.08 | 0.03/0.07 | 0.03/0.05 | 0.08/0.11 | 0.01/0.01 | 0.02/0.04 | 0.03/0.07
DGCNN+MSCD |  0.42/0.74 | 0.28/0.53 | 0.18/0.31 | 0.08/0.17 | 0.10/0.26 | 0.18/0.33 | 0.13/0.21 | 0.30/0.63 | 0.39/0.70 | 0.17/0.33
VoxNet+MSCD | 0.41/0.75 | 0.26/0.51 | 0.21/0.43 | 0.08/0.18 | 0.11/0.29 |  0.25/0.40 | 0.17/0.31 |  0.30/0.62 |  0.43/0.69 | 0.18/0.36
PointNet2+MSCD | 0.41/0.72 | 0.28/0.53 | 0.23/0.44 | 0.10/0.21 | 0.12/0.29 | 0.22/0.37 | 0.19/0.32 | 0.30/0.63 |  0.43/0.70 | 0.19/0.37
PointCNN+MSCD | 0.41/0.72 | 0.29/0.56 | 0.27/0.54 | 0.11/0.26 | 0.16/0.35 | 0.25/0.42 | 0.27/0.47 | 0.30/0.63 | 0.43/0.71 | 0.22/0.43

This section is organized as follows. Section IV-A introduces the dataset on
which we conduct all our experiments. Section IV-B describes the alternative
state-of-the-art methods and Section IV-C defines three evaluation metrics.
Discussions of the performance and ablation studies are presented in Section
IV-D and Section IV-E, respectively. Finally, in Section IV-F, we show some
qualitative resutls to further demonstrate the superiority of our method.

###  IV-A Dataset

We test our method on the real-world dataset Scan2CAD [1], which contains
14225 (3049 unique) CAD models from ShapeNet [37] and corresponding scanned
objects from ScanNet [38]. For CAD model retrieval, we assume that instance
segmentations are known, and apply the segmentation masks over the 3D scans to
obtain object surfaces. We furthermore sample points on the surface of the CAD
models to generate point clouds for the database. Note that we filter out some
poor quality point clouds with too few points. Finally, we work with 12598
object scans spreading over 1506 scenes, and a database of 2827 CAD models.

###  IV-B Shape Descriptor Alternatives

For comparison purposes, we choose several state-of-the-art 3D shape
descriptors. This includes the handcrafted feature VFH [36] as well as the
learning-based features DGCNN [13], VoxNet [27], PointNet2 (or PointNet++)
[12], and PointCNN [2]. Viewpoint Feature Histogram (VFH) is a global
descriptor based on FPFH [21]. We compute the 308-dimensional VFH feature
using the PCL implementation [39]. As for DGCNN, VoxNet, PointNet2 and
PointCNN, they are representatives of graph-based, volumetric-based, point-
based, and convolution-based methods for 3D shape classification,
respectively. All networks are trained on ModelNet40 [40] and we extract the
global feature from the input of the final classification layer. Both DGCNN
and PointNet2 output 256-dimensional feature vectors while VoxNet and PointCNN
produce 128-dimensional features.

###  IV-C Evaluation Metrics

We use three different metrics to evaluate the retrieval performance on
Scan2CAD:

Topk Retrieval Accuracy. Suppose we have $T$ query objects in $\mathcal{X}$
and there is a ground truth CAD model $\mathbf{Y}^{*}\in\mathcal{Y}$ for each
query object $\mathbf{X}\in\mathcal{X}$. The Topk retrieval accuracy (RA) is
defined as

$\mathrm{RA}=\frac{1}{T}\sum_{\mathbf{X}\in\mathcal{X}}\mathbf{1}_{\mathcal{A}_{k}}(\mathbf{Y}^{*}),$
(5)

where $\mathcal{A}_{k}$ can be replaced either by the top-k subset of the
initial ordered set $\mathcal{C}$, or the top-k subset of the re-ordered set
$\hat{\mathcal{C}}$. Note that
$\mathbf{1}_{\mathcal{A}_{k}}:\mathcal{Y}\rightarrow\\{0,1\\}$ is an indicator
function:

$\mathbf{1}_{\mathcal{A}_{k}}(\mathbf{Y}^{*})=\left\\{\begin{aligned}
1\quad&\mathrm{if\leavevmode\nobreak\ }\mathbf{Y}^{*}\in\mathcal{A}_{k},\\\
0\quad&\mathrm{otherwise}.\end{aligned}\right.$

Top1 Chamfer Distance. With a slight modification of the metric defined in
[9], the Top1 Chamfer Distance is computed by measuring MSCD between the
ground truth CAD model and the retrieved top 1 CAD model:

$d=\frac{1}{T}\sum_{\mathbf{X}\in\mathcal{X}}d_{MSCD}(\mathcal{A}_{1},\mathbf{Y}^{*}).$
(6)

Ground truth Ranking. The ranking order of the ground truth CAD model in the
returned CAD candidate set:

$\mathrm{R}=\frac{1}{T}\sum_{\mathbf{X}\in\mathcal{X}}\mathrm{ranking}_{\mathcal{A}}(\mathbf{Y}^{*})$
(7)

where $\mathrm{ranking}_{\mathcal{A}}(\cdot)$ denotes the ranking position in
$\mathcal{A}$.

###  IV-D Results

We use the methods mentioned in Section IV-B as our front-end, followed by
MSCD-based re-ranking to improve the CAD candidate set returned by the initial
feature-based search. Note that the nearest neighbour search is configured to
find 90 nearest neighbors. Table I indicates Top1 and Top5 retrieval accuracy
of different methods and demonstrates the impact of adding MSCD-based re-
ranking. It is obvious that the proposed geometric re-ranking strategy leads
to a significant performance boost over the original feature-based ranking
results. In addition, our method achieves much better results when combined
with PointCNN.

Table II: Top5 Category Ratio Method | Top5 Category Ratio
---|---
VFH [36] | 0.32
DGCNN [13] | 0.99
VoxNet [27] | 1.00
PointNet2 [12] | 0.99
PointCNN [2] | 0.99
Figure 2: Comparative results of Top1 Chamfer Distance on Scan2CAD. Figure 3:
Comparative results of Ground truth Ranking on Scan2CAD.

###  IV-E Ablation Studies

Figure 4: Top1 and Top5 retrieval accuracy using different metrics and
different search ranges based on PointCNN. Figure 5: Qualitative results of
CAD model retrieval on Scan2CAD [1] based on PointCNN [2]. Rows 1 to 5 show
different kinds of examples for classes chair, table, bed, lamp and trash bin,
respectively. The retrieved CAD model identical to ground truth is highlighted
with corresponding color while the models of clearly inferior quality are
highlighted with red. The last two rows are failure cases.

In Table II, we show that the top 5 CAD models produced by 3D shape
descriptors almost always have the same category as that of the query object,
only with the exception of VFH. This proves the usefulness of neural networks
in handling inter-class dissimilarity and also explains the poor performance
of VFH.

We further test our MSCD-based re-ranking method using Top1 Chamfer Distance.
As shown in Figure 2, the MSCD between the ground truth and the retrieved top
1 CAD model declines for all 3D shape descriptors when combining with a re-
ranking method. We also compare the rank of the ground truth CAD model with
and without re-ranking to validate its effectiveness (see Figure 3). After re-
ranking, the ground truth CAD model is promoted to a higher rank.

Our method has very few hyper-parameters. To figure out the influence of
hyper-parameters on retrieval performance, we run a series of experiments
based on PointCNN using different nearest neighbor search ranges and different
point cloud distance metrics (introduced in Section III-C). Figure 4 shows
that MSCD outperforms both CD and SCD in terms of Top1 and Top5 retrieval
accuracy. The performance gradually improves as the search range becomes
larger.

###  IV-F Qualitative Results

Qualitative results for CAD model retrieval on Scan2CAD are shown in Figure 5.
Our method takes good advantage of visible parts in the query object and
retrieves more reasonable CAD models with similar structure. For example, let
us consider the first row in Figure 5. The wheel and armrest fragments in the
query indicate that this is a swivel chair with armrests. It is obvious that
our proposed re-ranking method returns CAD models with both of these
properties while the feature-based search struggles to get reasonable results.
Furthermore, our method successfully rejects models of clearly inferior
quality (cf. row 3 and row 4 of Fig. 5).

The last two rows in Figure 5 show failure cases of our approach. Most of them
are caused by the inherent ambiguity in query objects and inaccurate ground
truth annotations. Taking the bottom row as an example, the query barely
captures the back and the seat of a chair, which creates possibilities for
arbitrary types of legs. Though our method retrieves some similar CAD models,
it fails to find the exact model of the ground truth.

##  V Conclusion

We have presented a two-step pipeline for instance-level CAD model retrieval
in a large-scale database. Since state-of-the-art learned representations
still have limitations on understanding all aspects of 3D shapes, we have
proposed a geometry-based re-ranking method to facilitate fine-grained
retrieval. We have furthermore introduced a Modified Single-direction Chamfer
Distance to measure the point set distance between partial objects and CAD
models. Thorough experiments on real-world test cases validate the superiority
of our method. Future work will investigate the combination of instance-level
retrieval and category-level alignment in order to simultaneously perform CAD
model retrieval and alignment with high accuracy.

## References

  * [1] A. Avetisyan, M. Dahnert, A. Dai, M. Savva, A. X. Chang, and M. Nießner, “Scan2cad: Learning cad model alignment in rgb-d scans,” _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 2609–2618, 2019.
  * [2] Y. Li, R. Bu, M. Sun, W. Wu, X. Di, and B. Chen, “Pointcnn: Convolution on x-transformed points,” in _NeurIPS_ , 2018.
  * [3] A. J. Davison, “Futuremapping: The computational structure of spatial ai systems,” _arXiv preprint arXiv:1803.11288_ , 2018.
  * [4] R. A. Newcombe, S. Izadi, O. Hilliges, D. Molyneaux, D. Kim, A. Davison, P. Kohli, J. Shotton, S. Hodges, and A. Fitzgibbon, “Kinectfusion: Real-time dense surface mapping and tracking,” _2011 10th IEEE International Symposium on Mixed and Augmented Reality_ , pp. 127–136, 2011.
  * [5] J. McCormac, R. Clark, M. Bloesch, A. Davison, and S. Leutenegger, “Fusion++: Volumetric object-level slam,” _2018 International Conference on 3D Vision (3DV)_ , pp. 32–41, 2018.
  * [6] J. Hou, A. Dai, and M. Nießner, “Revealnet: Seeing behind objects in rgb-d scans,” _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 2095–2104, 2020.
  * [7] S. Gupta, P. Arbeláez, R. Girshick, and J. Malik, “Aligning 3d models to rgb-d images of cluttered scenes,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 4731–4740.
  * [8] A. Avetisyan, A. Dai, and M. Nießner, “End-to-end cad model retrieval and 9dof alignment in 3d scans,” _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pp. 2551–2560, 2019.
  * [9] T. Zhao, Q. Feng, S. Jadhav, and N. A. Atanasov, “Corsair: Convolutional object retrieval and symmetry-aided registration,” _ArXiv_ , vol. abs/2103.06911, 2021.
  * [10] M. Dahnert, A. Dai, L. Guibas, and M. Nießner, “Joint embedding of 3d scan and cad objects,” _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pp. 8748–8757, 2019.
  * [11] C. Qi, H. Su, K. Mo, and L. Guibas, “Pointnet: Deep learning on point sets for 3d classification and segmentation,” _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 77–85, 2017.
  * [12] C. Qi, L. Yi, H. Su, and L. Guibas, “Pointnet++: Deep hierarchical feature learning on point sets in a metric space,” in _NIPS_ , 2017.
  * [13] Y. Wang, Y. Sun, Z. Liu, S. E. Sarma, M. Bronstein, and J. Solomon, “Dynamic graph cnn for learning on point clouds,” _ACM Transactions on Graphics (TOG)_ , vol. 38, pp. 1 – 12, 2019.
  * [14] A. Grabner, P. M. Roth, and V. Lepetit, “Location field descriptors: Single image 3d model retrieval in the wild,” in _2019 International Conference on 3D Vision (3DV)_. IEEE, 2019, pp. 583–593.
  * [15] A. Bansal, B. Russell, and A. Gupta, “Marr revisited: 2d-3d alignment via surface normal prediction,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 5965–5974.
  * [16] X. He, Y. Zhou, Z. Zhou, S. Bai, and X. Bai, “Triplet-center loss for multi-view 3d object retrieval,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 1945–1954.
  * [17] Y. M. Kim, N. J. Mitra, D.-M. Yan, and L. Guibas, “Acquiring 3d indoor environments with variability and repetition,” _ACM Transactions on Graphics (TOG)_ , vol. 31, no. 6, pp. 1–11, 2012.
  * [18] Y. Li, A. Dai, L. Guibas, and M. Nießner, “Database-assisted object retrieval for real-time 3d reconstruction,” in _Computer Graphics Forum_ , vol. 34, no. 2. Wiley Online Library, 2015, pp. 435–446.
  * [19] Y. M. Kim, N. J. Mitra, Q. Huang, and L. Guibas, “Guided real-time scanning of indoor objects,” in _Computer Graphics Forum_ , vol. 32, no. 7. Wiley Online Library, 2013, pp. 177–186.
  * [20] B. Drost and S. Ilic, “3d object detection and localization using multimodal point pair features,” in _2012 Second International Conference on 3D Imaging, Modeling, Processing, Visualization & Transmission_. IEEE, 2012, pp. 9–16.
  * [21] R. Rusu, N. Blodow, and M. Beetz, “Fast point feature histograms (fpfh) for 3d registration,” _2009 IEEE International Conference on Robotics and Automation_ , pp. 3212–3217, 2009.
  * [22] A. E. Johnson, “Spin-images: a representation for 3-d surface matching,” 1997\.
  * [23] C. Choy, J. Park, and V. Koltun, “Fully convolutional geometric features,” _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pp. 8957–8965, 2019.
  * [24] H. Chen, K. Sun, Z. Tian, C. Shen, Y. Huang, and Y. Yan, “Blendmask: Top-down meets bottom-up for instance segmentation,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 8573–8581.
  * [25] D. Bolya, C. Zhou, F. Xiao, and Y. J. Lee, “Yolact: Real-time instance segmentation,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 9157–9166.
  * [26] K. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask r-cnn,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 2961–2969.
  * [27] D. Maturana and S. Scherer, “Voxnet: A 3d convolutional neural network for real-time object recognition,” _2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pp. 922–928, 2015.
  * [28] H. Fan, H. Su, and L. Guibas, “A point set generation network for 3d object reconstruction from a single image,” _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 2463–2471, 2017.
  * [29] P. Achlioptas, O. Diamanti, I. Mitliagkas, and L. Guibas, “Learning representations and generative models for 3d point clouds,” in _ICML_ , 2018\.
  * [30] H. Deng, T. Birdal, and S. Ilic, “Ppf-foldnet: Unsupervised learning of rotation invariant 3d local descriptors,” _ArXiv_ , vol. abs/1808.10322, 2018\.
  * [31] ——, “3d local features for direct pairwise registration,” _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 3239–3248, 2019.
  * [32] C. Duan, S. Chen, and J. Kovacevic, “3d point cloud denoising via deep neural network based local surface estimation,” _ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 8553–8557, 2019.
  * [33] Q. Feng and N. A. Atanasov, “Fully convolutional geometric features for category-level object alignment,” _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pp. 8492–8498, 2020.
  * [34] V. Ishimtsev, A. Bokhovkin, A. Artemov, S. Ignatyev, M. Nießner, D. Zorin, and E. Burnaev, “Cad-deform: Deformable fitting of cad models to 3d scans,” in _ECCV_ , 2020.
  * [35] M. A. Uy, V. G. Kim, M. Sung, N. Aigerman, S. Chaudhuri, and L. Guibas, “Joint learning of 3d shape retrieval and deformation,” _ArXiv_ , vol. abs/2101.07889, 2021.
  * [36] R. Rusu, G. Bradski, R. Thibaux, and J. M. Hsu, “Fast 3d recognition and pose using the viewpoint feature histogram,” _2010 IEEE/RSJ International Conference on Intelligent Robots and Systems_ , pp. 2155–2162, 2010.
  * [37] A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su, J. Xiao, L. Yi, and F. Yu, “Shapenet: An information-rich 3d model repository,” _ArXiv_ , vol. abs/1512.03012, 2015\.
  * [38] A. Dai, A. X. Chang, M. Savva, M. Halber, T. Funkhouser, and M. Nießner, “Scannet: Richly-annotated 3d reconstructions of indoor scenes,” _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 2432–2443, 2017.
  * [39] R. B. Rusu and S. Cousins, “3D is here: Point Cloud Library (PCL),” in _IEEE International Conference on Robotics and Automation (ICRA)_. Shanghai, China: IEEE, May 9-13 2011.
  * [40] Z. Wu, S. Song, A. Khosla, F. Yu, L. Zhang, X. Tang, and J. Xiao, “3d shapenets: A deep representation for volumetric shapes,” _2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 1912–1920, 2015.