# Understanding Time Variations of DNN Inference in Autonomous Driving

Liangkai Liu1, Yanzhi Wang2, and Weisong Shi3  1Department of Computer
Science, Wayne State University
2Department of Electrical & Computer Engineering, Northeastern University
3Department of Computer and Information Sciences, University of Delaware

###### Abstract

Deep neural networks (DNNs) are widely used in autonomous driving due to their
high accuracy for perception, decision, and control. In safety-critical
systems like autonomous driving, executing tasks like sensing and perception
in real-time is vital to the vehicle’s safety, which requires the
application’s execution time to be predictable. However, non-negligible time
variations are observed in DNN inference. Current DNN inference studies either
ignore the time variation issue or rely on the scheduler to handle it. None of
the current work explains the root causes of DNN inference time variations.
Understanding the time variations of the DNN inference becomes a fundamental
challenge in real-time scheduling for autonomous driving. In this work, we
analyze the time variation in DNN inference in fine granularity from six
perspectives: data, I/O, model, runtime, hardware, and end-to-end perception
system. Six insights are derived in understanding the time variations for DNN
inference.

###### Index Terms:

time variations, deep neural networks, autonomous driving

##  I Introduction

Owing to its high safety and efficiency, autonomous driving has become the
fundamental technology for the next generation of transportation. Deep Neural
Networks (DNNs) are widely deployed in the autonomous driving system for
sensing, perception, decision, and control. Typical examples include: YOLOv3
and Faster R-CNN for object detection [43, 45]; Deeplabv3 for image semantic
segmentation [26, 16]; LaneNet and PINet for lane detection [38, 28]. There
are two main reasons for the success of DNNs in autonomous driving systems.
The first is the higher accuracy compared with traditional computer vision-
based approaches [23]. The other is that DNNs can process raw data, making it
suitable for autonomous driving vehicles since it generates terabytes of raw
sensor data daily [31].

As a safety-critical system, autonomous driving sets high requirements in
accuracy, real-time, robustness, etc. The high accuracy of DNN-based
algorithms promotes the development of autonomous vehicles. However,
satisfying the real-time requirements of the sensing, perception, and planning
tasks are still significant challenges. According to [27], when the vehicle
drives at 40 km per hour in urban areas, these autonomous functions should be
effective every 1 m with task execution time less than 100ms. As DNN models
are widely used in object detection/classification, lane tracking, and
decision-making applications, guaranteeing the real-time execution of the DNN
inference becomes the key to satisfying the real-time requirements of
autonomous driving.

Generally, for safety-critical applications like sensing, perception, control,
etc., deadline-based scheduling is used by the real-time scheduler to
guarantee safety. Setting up deadlines is usually based on the worst observed
execution time. However, although the model structure and weights are fixed,
non-negligible time variations still exist in DNN inference [51, 49]. Prior
works [51] observed the time variations for DNN inference in mobile devices
and found that inference time follows an approximately Gaussian distribution.
However, the statistic-based approach performs poorly when time variations are
enormous. Another work [49] on the anytime DNN system also observed the time
variations issue and provided a Kalman Filter-based estimation for latency
distribution. $D^{3}$ is a work that addresses the time variations in AV
systems with dynamic deadlines rather than static deadlines [21]. However,
none of the existing approaches could handle huge time variations since they
do not consider the roots causing DNN inference time variations. Time
variations bring a big challenge for real-time schedulers because the deadline
with the worst observed execution time could waste many processor resources.
Many resources would be saved if the DNN inference time could be fixed or
narrowed down to a specific range. A detailed and in-depth analysis of DNN
inference time variation is missing. Therefore, understanding the DNN
inference time variation is key to optimizing the DNN inference runtime for
autonomous driving.

In this paper, we undertake a comprehensive analysis of DNN inference time
variations in a general autonomous driving system. We analyze the time
variation issues in fine granularity. For typical DNN models, we consider the
variability in DNN inference from six perspectives: data, I/O, model, runtime,
hardware, and end-to-end perception system. Six insights are derived for
reducing DNN inference time variations. In summary, this paper makes the
following contributions:

  * •

The time variation issues of DNN inference in autonomous driving are
thoroughly studied. We found that the majority of DNN models show variations
larger than 100ms, which significantly affects autonomous driving safety.

  * •

Through a comprehensive analysis of the time variation of DNN model inference
from six perspectives, we derive six insights into the relationship between
DNN inference time variations and variability of data, I/O, model, runtime,
hardware, and end-to-end perception system.

The rest of the paper is organized as follows. Section II presents the
background and motivation of this work. Section III discusses the time
variation analysis for typical DNN models. Section IV present the
implementation of the end-to-end perception system. Section V dicussed the
time variation analysis for the end-to-end perception system. Section VI
describes the related work. Section VII concludes the paper.

##  II Background and Motivation

The recent proliferation of computing technologies, e.g., sensors, computer
vision, machine learning, hardware acceleration, and the broad deployment of
communication mechanisms, e.g., DSRC, C-V2X, 5G, have pushed the horizon of
autonomous driving, which automates the decision and control of vehicles by
leveraging the perception results based on multiple sensors. The key to the
success of these autonomous systems is making reliable decisions in a real-
time fashion. However, since the sensing data and runtime of the computing
devices vary, the end-to-end latency for the perception, planning, and vehicle
controls show non-negligible variation. This time variation affects the system
predictability, making the operating system’s scheduling less efficient.

In this section, we present the motivation for this work by answering three
questions:

  * •

How prevalent are DNNs for autonomous driving?

  * •

How vital is timing variation for autonomous driving, and what is the state-
of-the-art?

  * •

What are the potential issues that affected the time variations in DNN
inference?

###  II-A DNN Inference for Autonomous Driving

DNN models have been widely used in the autonomous driving system for sensing,
perception, and localization [33]. Figure 1 shows an overview of the state-of-
the-art autonomous driving systems. The design purely relies on cameras for
sensing the environment. As shown in the figure, eight primary components are
divided into three parts: sensing, perception, and decision [32]. Sensing is
the process that sensors capture information from the environment. Perception
represents understanding the environment with algorithms applied to the
sensing data, including localization, detection, semantic segmentation, and
sensor fusion. The role of sensor fusion is to collaborate among all the
perception components and generate locations for objects, lanes, and open
spaces for the planning module. The decision is composed of global planning,
local planning, and vehicle control. Global planning generates the routes
between origin and destination, while local planning generates the trajectory
and control commands on brake, throttle, and steering. Finally, the control
commands are sent to the drive-by-wire system and applied to the vehicle.

We can find that the core modules on perception and decision are all based on
neural networks. The main reason is that DNN has an excellent performance in
many applications. Take image classification as an example. The ImageNet
challenge winner in 2017 decreased the classification error to 0.023, while
human beings’ error is 0.05 [29]. DNNs can also learn features from raw,
heterogeneous, and noisy data, suitable for autonomous driving scenarios since
terabytes of raw sensor data are generated on the vehicle [18].

Figure 1: A typical autonomous driving system.

###  II-B Time Variations in DNN Inference

The proliferation of deep learning achieves enormous performance improvement
and brings lots of issues in computation complexity, energy consumption, and
time variation for safety-critical systems [51]. For autonomous driving
vehicles, time variation affects predictability, which guarantees the
vehicle’s safety and improves resource utilization [40].

However, current DNN-based computing systems show poor performance in terms of
time variation [30]. To illustrate the time variation issue in a state-of-the-
art DNN-based autonomous driving system, we choose eleven models/algorithms
covering the whole pipeline of the autonomous driving system and measure the
end-to-end latency with the same input. There are seven DNN models for
perception: YOLOv3 [43], Faster R-CNN ResNet101 [45], Mask R-CNN Inceptionv2
[25], and SSD MobileNetv2 [34, 26] are for object detection; PINet [28] and
LaneNet [38] for lane detection; Deeplabv3 with MobileNetv2 for semantic
segmentation [16, 26]. Localization and planning algorithms are tested by
running ROS Navigation offline with recorded sensor data [7]. Adaptive Monte
Carlo localization (AMCL) [1] and ORB-SLAM2 [37] are deployed for
localization. A∗ [3] and Dynamic Window Approach (DWA) [2] are deployed for
global and local path planning, respectively. Table I shows the mean, the
range, and the division of the range and the mean of the end-to-end latency
for eleven models/algorithms. The range is defined as the difference between
the maximum value and the minimum value. From Table I, we can observe that
majority of the time is consumed by the DNN-based perception tasks: object
detection, lane detection, and segmentation. Among all the seven DNN models,
four models have a range larger than 100ms. LaneNet shows the biggest range
with 282ms. If we consider the percentage of the range over the mean, the
variations of the last three models for lane detection, localization, and
planning are larger than 100 percent. AMCL shows the poorest performance with
675.5 percent. However, localization and planning tasks only occupy a limited
portion of the end-to-end autonomous driving pipeline.

TABLE I: The mean, range, and variations of the eleven models used in the
autonomous driving system.

Task | Model | Mean (ms) | Range (ms) | Range / Mean (%)
---|---|---|---|---
Object Detection | YOLOv3 [43] | 173 | 57 | 32.8
Faster R-CNN Resnet101 [45] | 413 | 128 | 31
Mask R-CNN Inceptionv2 [25] | 266 | 104 | 39.1
SSD MobileNetv2 [34, 26] | 144 | 70 | 48.6
Lane Detection | LaneNet [38] | 82 | 282 | 344
PINet [28] | 127 | 263 | 207.1
Segmentation | Deeplabv3 [16, 26] | 149 | 19 | 12.8
Localization | AMCL [1] | 1.3 | 8.7 | 675.5
ORB-SLAM2 [37] | 53 | 56 | 105.6
Planning | A* [3] | 79 | 97 | 122.3
DWA [2] | 23 | 73 | 323.8

Figure 2: The box plot of end-to-end latency for typical models/algorithms in
autonomous driving pipeline.

How about the distributions of the end-to-end latency? Figure 2 shows the box
plot of the end-to-end inference time of the above eleven models/algorithms.
We can observe that perception tasks have a much wider latency distribution
than localization and planning tasks. Non-negligible amount of latencies that
lie above the 75th percentile. Besides, there are many outliers (abnormal
data) among all the perception tasks. Although these outliers have a low
probability, they affect the scheduler’s performance, especially for the
safety-critical system like autonomous driving. If we consider the autonomous
driving system a hard real-time system, the scheduler assigns deadlines for
each task based on the worst-observed execution time. Take LaneNet as an
example. The scheduler would set the deadline larger than the worst observed
execution time, which is 340ms [50]. Although it guarantees the safety of this
task, it also brings enormous inefficiency because the actual execution time
is less than 160ms over 95 percent of the time, which means around 180ms is
wasted for most jobs. If we narrow the execution time range to less than 50ms,
we can save almost 110ms in LaneNet for every job.

Time variation is expected to make a significant impact on safety-critical
systems. Since perception tasks consume most of the time, this work’s focus
would be on the DNN models in perception of the autonomous driving system. We
propose using profiling tools to understand DNN inference time variations in
current autonomous driving systems and model their time variation.

###  II-C Uncertainties in DNN Inference

Figure 3: The timeline of DNN inference.

What are the potential issues that affect the time variations in DNN
inference? To answer this question, we first analyze the timeline of DNN
inference. Figure 3 shows a general DNN inference timeline in TensorFlow. The
timeline starts by calling inference(), which first loads the graphs and
weights into the memory. The process starts reading the input, where sub() is
used to subscribe to an image stream. The Robot Operating System [41] (ROS)
provides data communications between different components. Next, it pre-
processes the image, including resizing, converting color space from one to
another, etc. Next, the processed image is passed to the Session and loaded
into the processor to run network inference. Finally, post-processing
transforms the bounding boxes transform() or fit pixel proposals into lanes
(lane_fit()) on the image.

From the timeline of DNN inference, we derive several uncertainties that
contribute to the time variation issue. The first is data, which means the
value and distribution of image pixels. The sparsity matrix is expected to
have less time on inference than the dense matrix [17]. The second is the data
I/O to the session graph. How the running graph reads data from ROS messages
could affect the inference time variations. The third is the model, which
means the model’s structure and complexity in multiply and accumulate (MAC)
operations. The fourth is runtime, owing to the contention of concurrent jobs
for resources like memory, CPU, and GPU. How many processes allow preemption,
their scheduling policies, priorities, etc., are the factors that affect the
runtime variation. Finally, the hardware also affects the time variation of
DNN inference. GPU is expected to run faster than CPU, but a multi-core system
is supposed to show more time variations than a single-core system. How will
different architectures affect the time variations of DNN inference? In
summary, five aspects of uncertainties in DNN inference need to be studied:
data, I/O, model, runtime, and hardware.

###  II-D Profiling Tools

For current DNN-based autonomous driving systems, a big challenge is how to
explore the time variation in DNN model inference. Our approach is to profile
the system with a variety of granularity. In general, we use three profiling
tools: code level (cProfiler) and system level (nvprof and Linux Perf).

cProfiler [12] is a python library for code-level profiling. It collects
statistics that describe how often and for how long parts of the program are
executed. The number of function calls can identify code bugs and possible
inline-expansion points (high call counts). Internal time statistics can
identify “hot loops” that should be carefully optimized. Cumulative time
statistics are used to identify high-level errors in the selection of
algorithms. We use cProfiler to get the call graph with time breakdowns of the
code.

Although application-level profiling gives us some explanations of DNN
inference time variations, it is not enough to explain the variability of the
DNN inference for different architectures. Therefore, we conduct system call
level profiling with Linux perf to show model inference performance at the
system call level. Linux perf uses the system performance counter to monitor
the whole system [22]. We use Linux perf to collect the system-level metrics,
including CPU cycles, context switches, CPU migrations, page faults,
instructions, branches, branch misses, cache misses, etc.

GPU has been widely used in DNN execution acceleration. nvprof is a profiling
tool provided by NVIDIA that enables the collection of a timeline of CUDA-
related activities on CPU and GPU, including kernel execution, memory
transfers, memory set, CUDA API calls, and events or metrics for CUDA kernels
[15]. We use nvprof to collect timing analysis for GPU activities to find the
roots for DNN inference time variations.

##  III Model Inference Profiling

To understand the uncertainties and quantify their impacts on DNN inference
time variations, we need to profile the execution of model inference in fine
granularity. This section discusses the profiling of typical DNN models based
on the uncertainty analysis in Section II.

###  III-A Experiment Setup

To begin with, we present the experimental setup for DNN inference profiling.
We choose three types of computing devices that cover CPU and GPU-based
processors. Besides, we create an image dataset based on the KITTI dataset
[20].

Hardware and software setup. The devices we use for profiling include NVIDIA
Jetson AGX Xavier, Xavier NX, and Intel Fog Reference. Table II shows these
devices’ memory, CPU, and GPU configurations [5]. Both the Jetson boards are
installed with JetPack 4.4-DP [4] (L4T R32.4.2) and use an l4t-ml docker image
[9] as the base image for system setup. The l4t-ml image includes several
libraries for machine learning-related applications: TensorFlow 1.15, PyTorch
v1.5.0, torchvision v0.6.0, CUDA 10.2, cuDNN 8.0.0, OpenCV 4.1, etc. On top of
the l4t-ml image, we implemented a ROS-based perception pipeline for
autonomous driving. ROS Melodic and ROS Galactic are deployed as the
communication middleware. To reduce the impact of different hardware on the
time variation, the model inference profiling of data, I/O, model, and runtime
variances are conducted on NVIDIA Jetson AGX boards, which have the same chip
as the auto-graded NVIDIA DRIVE AGX Xavier board [8]. Besides, we disable the
Dynamic Voltage and Frequency Scaling (DVFS) on Jetson AGX Xavier with
jetson_benchmark [6] and turn off all other user applications before the
experiment. The Jetson broad is set at MAXN power mode, where both CPU and GPU
run with the highest frequency. Since accuracy is essential for the autonomous
driving scenario, all the DNN models are trained and tested with full
precision (FP32).

TABLE II: Hardware configurations of devices in profiling.

Devices | CPU | GPU | Memory | AI Performance
---|---|---|---|---
AGX Xavier |  | 8-core NVIDIA Carmel Arm®v8.2
---
64-bit CPU 8MB L2 + 4MB L3
| 512-core NVIDIA Volta™
---
with 64 Tensor Cores
| 32 GB 256-bit LPDDR4x
---
136.5 GB/s
32 TOPs
Xavier NX |  | 6-core NVIDIA Carmel ARM®v8.2
---
64-bit CPU 6MB L2 + 4MB L3
| 384-core NVIDIA Volta™
---
with 48 Tensor Cores
| 8 GB 128-bit LPDDR4x
---
59.7 GB/s
21 TOPs
Fog Node |  | 8 Intel(R) Xeon(R)
---
CPU E3-1275 v5 @ 3.60GHz
- |  | 32 GB DDR4
---
34.1 GB/s
-
GPU Workstation |  | 28 Intel® Core™ i9-9940X
---
CPU @ 3.30GHz
4 NVIDIA GeForce RTX 2080 Ti/PCIe/SSE2 |  | 64 GB DDR4
---
85 GB/s
312 TOPS

Dataset Descriptions. We create an image dataset as a uniform input to the
profiling process based on the KITTI dataset [11]. The image dataset covers
three scenarios (city, residential, and road) with 1,800 images. By sampling
the image dataset with different frequencies (1, 2, 5, and 10 FPS), we get
four groups of images for each scenario, with 60, 120, 300, and 600 images for
each scenario. In addition to the image dataset, an auto-grade camera with
1920x1080 resolution and 30 FPS is also used as input to the DNNs.

Metrics. To give detailed profiling results of the DNN-based applications, we
measure several metrics, including latency, processor utilization, memory
utilization, etc. We calculate latency’s statistic metrics, including the
range and the coefficient of variation. Equation (1) defines the range as the
difference between the maximum and minimum values. The coefficient of
variation ($c_{v}$) is used to evaluate the relative variability, and it is
calculated using the standard variation $\sigma$ divided by the mean value
$\mu$. $c_{v}$ is a positive value. The higher the value is, the higher
variations the data has.

Range:

$R=max(t_{i})-min(t_{i})$ (1)

Coefficient of Variation:

$c_{v}=\frac{\sigma}{\mu}$ (2)

###  III-B Data Variability

As the input to the DNN inference, data variability is expected to affect the
whole pipeline significantly. The application scenarios and FPS are expected
to affect the execution time of the DNN models since more lanes and objects
are supposed to be detected downtown than in the countryside. Meanwhile, since
the critical operations of DNN inference are multiply and accumulate (MAC),
the distribution of the matrix/pixel might also affect the execution time. One
example is that the sparse matrix needs fewer operations than the dense
matrix. Moreover, the weather brings another uncertainty for sensor data when
the model is deployed in real-world environment. Therefore, we discuss the
impact of the data variability on the time variation of DNN inference in three
aspects: scenarios, pixel sizes & distribution, and weather’s impact.

Figure 4: CDF for latency results of YOLOv3, Faster R-CNN, and PINet under
different scenarios.

Scenarios. We use the datasets from three scenarios (i.e., city, residential,
and road) to evaluate the impact of data variability. Three DNN inferences are
covered: YOLOv3 and Faster R-CNN for object detection, PINet for lane
detection. The results under different scenarios are shown in Figure 4. The
performance for different scenarios shows differently: PINet and Faster R-CNN
show massive time variations between those three scenarios, while YOLOv3 does
not. The reason is that different scenarios bring variable possibilities to
detect lanes and objects. As a representative of one-stage based object
detection, YOLOv3’s latency is not affected by the scenarios. However, Faster
R-CNN is a two-stage based object detection. Different scenarios have
different numbers of potential objects, contributing to the inference time
variations. This observation also implies that the scheduling of object and
lane detection tasks should take the running scenario into account. We will
discuss it in detail in model variability.

Figure 5: The post-processing time and the number of detected objects in
Faster R-CNN ResNet101 for a sequence of images. Right table contains the
correlation coefficient of post-processing time and number of detected objects
for five object detection models.

Since the different scenarios mainly affect the possibility of detecting
objects, the number of objects becomes a potential connection between the
scenario and the time variation of DNN inference. To prove it, we choose
several DNN models for object detection and get the time sequence of the
latency breakdown and the number of objects. Figure 5 shows an example of the
results for Faster R-CNN ResNet101. The variation in the number of objects is
almost consistent with the variations in the post-processing time. To get more
accurate correlation results, we calculate the correlation coefficient ($p$)
of five object detection models’ post-processing and the number of detected
objects. The results for Faster R-CNN, Mask R-CNN, YOLOv3, SSD MobileNetv2,
and SSD Inceptionv2 are 0.98, 0.98, 0.43, 0.91, and 0.96, respectively. These
results explain the relationship between usage scenarios and inference time
variations. Since lane detection can be seen as pixel-level regression, the
scenarios’ impact on its DNN inference is similar.

Figure 6: The CDF of end-to-end latency for YOLOv3, Faster R-CNN, and PINet
with different pixel distributions; The CDF of end-to-end latency for Faster
R-CNN with difference image sizes.

Pixel sizes & distributions. In addition to the scenarios of the input data,
the pixel distribution and sizes are also expected to affect the variation of
DNN inference time. To show the effect of pixel distributions, we choose three
types of data inputs to the DNNs and measure time variations. These image
inputs include all zero (black), all 255 (white), and random matrix. We get
the results for running these three cases on YOLOv3, Faster R-CNN, and PINet,
as shown in Figure 6. We can observe significant time variation in PINet when
the random matrix is applied because PINet is a pixel-level regression to
lanes. Random values of the pixels add the computations for detecting pixel
proposals for lanes. On the contrary, there is no significant difference
between YOLOv3 and Faster R-CNN because object detection is a box-level
detection. Changing limited pixels does not make a big difference for a box
containing thousands of pixels.

For profiling pixel size’s impact on inference time variations, we choose the
Faster R-CNN model and scale the same input image ([3, 375, 1242]) by
multiplying the width and heights with different ratios $\lambda$: 0.01, 0.05,
0.1, 1, 2, 5, and 10. Each scaled image is sent to execute model inference 100
times. The results for CDF of the end-to-end latency are shown in Figure 6. We
can find that the CDFs are close to each other except for the case when
$\lambda$ equals 10, which shows higher average latency and inference time
variations than others. Through the study on the implementation, we found that
the difference is mainly caused by the pre-processing, where the input image
will be transposed and cropped if the size is larger than the maximum value.

TABLE IV: The mean ($\mu$), variation ($\sigma$), and variation coefficient
($c_{v}$) of the end-to-end inference time for Faster R-CNN and PINet under
different raining cases.

End-to-End Inference Time (ms) | Faster R-CNN | PINet
---|---|---
Raining Case | $\mu$ | $\sigma$ | $c_{v}$ | $\mu$ | $\sigma$ | $c_{v}$
0 mm/hour | 320.7 | 1217.9 | 3.8 | 228.9 | 7862.6 | 34.4
25 mm/hour | 326.5 | 1129.6 | 3.5 | 216.9 | 7142.3 | 32.9
50 mm/hour | 324.3 | 1082.5 | 3.3 | 215 | 7033.4 | 32.7
100 mm/hour | 319.3 | 950.4 | 3 | 205.9 | 6633.3 | 32.2
150 mm/hour | 314.9 | 898.9 | 2.9 | 199.2 | 5410.6 | 27.2
200 mm/hour | 309.1 | 805.5 | 2.6 | 190.4 | 5068.1 | 26.6

Weather’s impact. When the model is deployed in a real-world environment, the
weather is expected to affect the accuracy and the inference time of DNN
models. To show the weather’s impact on the DNN inference time variation, we
render different levels of rain into the KITTI dataset and measure the end-to-
end model inference time [48]. Six raining cases (0/25/50/100/150/200 mm/hour)
are covered, and we choose Faster R-CNN and PINet to show the impact of rains
on the inference time. Table IV shows the results of end-to-end inference time
for Faster R-CNN and PINet under six rainy cases. By comparing the mean
($\mu$), the variation ($\sigma$), and the coefficient of variation ($c_{v}$)
under different raining cases, we find that the end-to-end inference time’s
value and variation decrease as the rain level increases. The reason is that
heavy rain makes thousands of pixel value changes, and the probability for a
group of pixels to be lanes and objects is decreased. To prove it, we record
the number of object box proposals, pixel lane proposals, detected objects,
and lanes from the inference pipeline of Faster R-CNN and PINet, shown in
Figure 7. We can find the number of proposals for objects and lanes is
decreased when the rain level increases. Besides, considering the 25th and
75th percentile of the distribution, the range and the variation of objects
and lanes are also decreased, which is highly correlated with the end-to-end
inference time variations.

Figure 7: The box plot of proposals and objects for Faster R-CNN and PINet
under different raining levels.

Insight 1: The scenario affects the DNN inference time variations by the
potential number of detected lanes/objects. The pixel distributions have a
higher impact on lane detection than object detection. The time variations of
object and lane detection decrease as the rain level increases.

(a) (b)

Figure 8: The publish-subscribe model in (a) ROS1 IPC (b) ROS2 DDS. Each
publisher/subscriber is corresponding to a Node in ROS1 or a participant in
ROS2.

###  III-C I/O Variability

As an essential step for the DNN inference, access to the data could affect
the whole execution pipeline. ROS is an anonymous publish-subscribe middleware
system widely used for data communication. The publisher-subscriber mechanism
is the main communication pattern in the autonomous driving system since
multiple models need sensor data for localization, detection, segmentation,
etc. The socket-based Inter-Process Communication (IPC) mechanism in ROS1
brings high compatibility and extensibility [35]. ROS2 is an evolved version
of ROS1. ROS2 uses Data Distribution Service (DDS) [39] as its communication
foundation, which works well in real-time distributed systems. In this part,
we compare the time variations under two types of communication mechanisms:
ROS IPC and ROS2 DDS.

TABLE V: Three messages used in comparison of ROS1 IPC and ROS2 DDS.

Name | Message type | Source | Dimention( [width, height, channel] ) | Size
---|---|---|---|---
msg1 | Image | Image File | [192, 108, 3] | 62KB
msg2 | Image | Image File | [1920, 1080, 3] | 6.2MB
msg3 | Image | Camera | [1920, 1080, 3] | 6.2MB

Figure 9: The box plot for message transmission latency of ROS1 IPC and ROS2
DDS when the number of subscribers increased from one to eight.

Figure 8 shows the structure of the publish-subscribe model used in ROS1
Inter-Process Communications (IPC) and ROS2 (Data Distribution Service) DDS.
ROS1 has a centralized design for data communication. It has a master node
that provides naming and registration services for all the other ROS nodes,
topics, services, and actions. ROS node is a process to perform a particular
computation, while ROS topics are named buses for ROS nodes to exchange
messages [13]. Figure 8(a) is a general 1 to N communication pattern (one data
publisher, N data subscribers). The underlying transmission is based on TCPROS
by default [10]. When N nodes subscribe to a topic, the message would be
copied N-1 times and sent to the subscriber in sequence order. Unlike ROS1,
ROS2 follows a distributed design for fault tolerance purposes. As is shown in
Figure 8(b), a global data space is implemented in DDS, which all independent
applications can access. The underlying communication in ROS2 DDS is based on
UDP and shared memory to avoid data copies [19].

To compare the performance of ROS1 IPC and ROS2 DDS, we deployed three Image
messages in ROS/ROS2. Table V shows the descriptions of these messages: one is
read directly from a USB camera with a resolution of 1920x1080, another is
randomly generated with the same size as the camera’s frame, and the remaining
one has a smaller size than the former two messages. The Image publisher’s
queue size is 1 for both ROS1 IPC and ROS2 DDS. The deployed ROS DDS is
eProsima Fast DDS, explicitly optimized for ROS2 [19, 52]. We measure the
latency of message transmission from the time a message is published until the
time another node subscribes to it. In addition, we set up a different number
of subscribers between one to eight and recorded the communication latency for
each subscriber.

The results of the box plot for communication latency with ROS1 IPC and ROS2
DDS are shown in Figure 9. By comparing the latency distribution, we find that
for both ROS1 IPC and ROS2 DDS, the range of communication latency increases
when the number of subscribers increases, which indicates the I/O time
variations increase when more subscribers are accessing the same Topic.
Moreover, when the message size is small (msg1), ROS2 DDS shows lower
communication latency and variations than ROS1 IPC. This is owing to the
overhead of message copy in ROS1 IPC. However, when the message size becomes
larger (msg2), ROS1 IPC begins to show better performance than ROS2 DDS. The
reason is that ROS2 DDS invokes UDP calls for communication, while the maximum
UDP datagram size in ROS2 is 64KB. Plenty of time is consumed by the message
splitting and merging in ROS2 DDS, which is much higher than message copy
overhead in ROS1 IPC. Besides, the communication latency variation among
subscribers in ROS1 IPC is also lower than in ROS2 DDS. We can find that when
transmitting msg2 and msg3 with one to eight patterns, four have lower latency
and a smaller range, while the other four subscribers have much higher
results. The main reason is that the message splitting and merging in UDP
consumes so many CPU calls that the communication cannot support eight links
simultaneously.

Insight 2: The variations of I/O latency increase significantly when the
number of subscribers to the same topic increases. ROS2 DDS shows lower
latency and time variations for small messages, while ROS1 IPC performs better
for large messages.

###  III-D Model Variability

In the end-to-end timeline of DNN inference, the model plays an essential part
in its time variations. Models trained under different scenarios or network
structures are expected to perform differently. Within the DNN inference, most
of the time is consumed by the model inference, which raises the question of
whether the inference’s variation also dominates the model’s time variations.
Since the model’s variability is a complex topic, we focus on six detection
models in this part.

Figure 10: The latency breakdown and time variations for four object detection
models and two lane detection models with the images dataset. We can observe
the relationship of the end-to-end inference time variations with the
breakdown’s latency. Among all these six models, the time variations of
YOLOv3, SSD-MobileNetv2 are determined by model inference, while the post-
processing determines the time variations of the remaining four models.

Inference and post-processing dominated. In Figure 10, we apply four DNN
models for object detection and two for lane detection on the image dataset to
show the changing latency breakdowns with different images. Based on each
part’s trend in each model, we can divide the four models into two groups:
inference-dominated and post-processing dominated. The inference-dominated
model means the variation of the end-to-end latency is correlated with the
inference time, which contains YOLOv3 and SSD-MobileNetv2. The post-processing
dominated model means the variation of the end-to-end latency is greatly
affected by the post-processing part. The detailed correlation analysis
results for end-to-end latency with breakdowns (reading, pre-processing,
inference, and post-processing) are shown in Table VI, which gives
quantitative proof of dominating factors of DNN inference time.

Why does it happen for these DNN models? We found the original answer when
looking inside the design for the DNNs. As we have learned from the design of
these six models, YOLOv3 and SSD use a one-stage approach that uniformly
samples on the image to get a certain number of bounding boxes and relies on
the convolution layers for feature extraction to calculate the probability of
object class. This one-stage design leads to a static number of objects from
the inference part, enabling minor time variations of the post-processing
step. In contrast, Faster R-CNN ResNet101 and Mask R-CNN Inceptionv2 are based
on a two-stage approach. The first stage generates a sparse set of candidate
object proposals, and the second one determines the accurate object regions
and the corresponding class/lane labels using convolutional neural networks
[53]. This two-stage design causes slight time variations in the convolution
neural networks. However, it makes the number of objects in post-processing
dynamic, which explains why the time variations are correlated to post-
processing time.

Similarly, LaneNet and PINet also follow a two-stage design, where the first
stage generates pixel proposals and the second stage clusters pixel proposals
into lane groups. To prove our assumption, we collect the number of proposals
and the post-processing time from DNN inference. Then we normalize them and
calculate their correlation coefficients. Figure 11 shows the time sequence
and correlation coefficient results for Faster R-CNN, LaneNet, and PINet. The
correlation coefficient between the number of proposals and post-processing
time is constantly higher than 0.89.

TABLE VI: Correlation coefficients of End-to-End latency with the breakdowns.
The more closer to 1, the more correlated.

Correlation Coefficients | read | pre-processing | inference | post-processing
---|---|---|---|---
YOLOv3 | 0.220 | 0.429 | 0.906 | 0.378
Faster R-CNN ResNet101 | -0.108 | 0.060 | 0.681 | 0.896
Mask R-CNN Inceptionv2 | -0.131 | 0.035 | 0.619 | 0.946
SSD MobileNetv2 | 0.169 | 0.056 | 0.963 | 0.525

Figure 11: The time sequence and correlation coefficients of proposals and
post-processing time

Insight 3: The design of the model’s structure significantly impacts the time
variations of the DNN inference. For object detection, one-stage models show
less variation than two-stage based ones. The time variations for two-stage
based object detection and lane detection models are mainly caused by the
number of objects/lanes proposals from the first stage.

###  III-E Runtime Variability

With fixed input, model, and hardware, runtime becomes an essential factor
contributing to the time variation of DNN inference. However, since a simple
application can invoke thousands of system calls with millions of instructions
to computing architecture, it is tough to accurately predict the execution
time for a given application. Since GPU tasks are non-preemptive, CPU
scheduling becomes the main uncertainty in runtime [14]. In this paper, the
DNN inference runtime profiling focuses on finding the connections between the
CPU scheduling policy and the time variation.

Scheduling policy setup. Due to the hard real-time requirements from safety-
critical applications, real-time operating systems or systems with real-time
kernel patches are widely used in computing systems for autonomous vehicles.
In this paper, we use the NVIDIA Jetson AGX board and configure it with an RT-
kernel patch. In L4T R32.4.2, the RT kernel allows preemption for most system
calls [44], giving more space to guarantee the deadlines for safety-critical
applications.

We choose four scheduling policies: SCHED_OTHER, SCHED_FIFO, SCHED_RR, and
SCHED_DEADLINE. SCHED_OTHER is AGX’s default scheduling policy for user
applications for maximum processor utilization. SCHED_FIFO schedules in a
first-come-first-serve method, while SCHED_RR schedules in a round-robin way.
SCHED_DEADLINE is a CPU scheduler based on the Earliest Deadline First (EDF)
[46]. We choose two DNN models, PINet and YOLOv3, for the runtime profiling.
The priorities for PINet and YOLOv3 are shown in Table VII. It shows that the
priorities of SCHED_FIFO and SCHED_RR are all 99, while others are 0. For the
setting up of deadlines, we choose two deadlines for each of them. Deadline-1
is set up based on the worst observed execution time (225ms for YOLOv3 and
300ms for PINet). Deadline-2 is based on the average end-to-end DNN inference
time (200ms for YOLOv3 and 150ms for PINet). Under each scheduling policy, the
mean value, 50, 80, and 99 percentiles, is used to show end-to-end latency
distribution. Besides, the coefficient of variation is also calculated to show
the quantified time variations.

Since the competition of applications significantly impacts the execution
time, we conduct experiments for each model in two steps. First, we run the
model inference without competition, called the single test. Then we conduct
the experiments with the resource competition from another DNN model, called
the compete test. By default, the competition model will use SCHED_OTHER as
the scheduling policy.

TABLE VII: The scheduling setup for PINet and YOLOv3.

Scheduling Policies |  | Priority
---
min/max
PINet | YOLOv3
SCHED_OTHER | 0/0 | 0 | 0
SCHED_FIFO | 1/99 | 99 | 99
SCHED_RR | 1/99 | 99 | 99
SCHED_DEADLINE | 0/0 |  | 0
---
deadline-1: 300ms
deadline-2: 150ms
| 0
---
deadline-1: 225ms
deadline-2: 200ms

Time variations under RT kernel. The RT kernel results in latency’s mean, 50,
80, and 99 percentiles are shown in Figure 12. Deadline-based scheduling shows
the worst time variations among all the RT scheduling policies (FIFO, RR, and
Deadline). One explanation is that the scheduler does not terminate tasks even
when it has already passed the deadline. The variation of the deadline-based
approach can be decreased with the termination, but it will reduce the FPS of
the detection. Besides, deadline-based scheduling with the average time is
much better than the deadline with the worst observed execution time, which
raises another question in selecting an appropriate deadline. The quantified
results for the coefficient of variation are shown in Table VIII. The results
prove our analysis above that the deadline-based scheduling shows the worst
performance in variation. Besides, the DNN inference time variations are much
more severe in the compete test than in the single test, mainly caused by
contention for non-preemptive system resources. Analysis of the time
variations for multi-tenant DNN inference exceeds the topic of this paper and
will be our future work.

Figure 12: Latency’s mean, 50, 80, and 99 percentiles of PINet and YOLOv3.

Insight 4: Deadline-based scheduling shows more time variations than other
real-time scheduling policies. Setting average time as the deadline has much
fewer time variations than the worst observed execution time in DNN inference.

TABLE VIII: Coefficient of variation under RT kernel.

$c_{v}$ \- RT | PINet-single | PINet-compete | YOLOv3-single | YOLOv3-compete
---|---|---|---|---
Deadline-1 | 0.30 | 0.33 | 0.13 | 0.21
Deadline-2 | 0.44 | 0.44 | 0.27 | 0.23
FIFO | 0.33 | 0.31 | 0.02 | 0.14
OTHER | 0.32 | 0.30 | 0.02 | 0.15
RR | 0.33 | 0.31 | 0.02 | 0.15

###  III-F Hardware Variability

Since most autonomous driving applications are DNN-based and require massive
computations, the variability of the hardware in CPU, GPU, and memory
configurations dramatically affects the DNN inference performance. This part
discusses the hardware’s variability in two aspects: end-to-end latency for
different devices and time variations under CPU and GPU-based architectures.

End-to-end latency for different devices. We use all four devices listed in
Table II for running PINet and YOLOv3 model inference experiments to collect
the end-to-end latency with breakdowns. The results are shown in Figure 13.
The GPU workstation performs the best in average end-to-end inference latency
among all the devices. However, all devices show long tail latency with non-
negligible time variations. For two-stage based PINet, the Fog Node tends to
have a smaller time variation range than Jetson AGX and Xavier NX. The reason
is that the post-processing mainly causes the time variations of PINet on the
CPU side, while the Fog node has more powerful CPUs. For YOLOv3, which is one-
stage based, the GPU desktop shows the lowest inference time variations
because the inference part mainly dominates the time variations of the one-
stage based model on the GPU side. The GPU workstation has much higher AI
performance in TOPs than the other two Jetson boards.

Figure 13: The CDF of end-to-end latency for PINet and YOLOv3 with different
hardware.

Time variations under CPU & GPU architecture. Since the average latency of
PINet and YOLOv3 on Jetson AGX, Fog Node, and GPU workstation are close. At
the same time, the inference time variations have a big difference. We use
these three devices to profile the impact of CPU and GPU architectures on DNN
time variations. To begin with, we use nvprof to profile the GPU activities on
AGX Xavier. We found that for PINet and YOLOv3, only the inference part is
executed on GPU while read, pre-processing, and post-processing are usually
executed on the CPU. To find the lower level roots causing the time variations
in post-processing, we use Linux perf to monitor the system events when PINet
and YOLOv3 models are executed with/without post-processing. The collected
system events include the instructions, cache references, cache misses, CPU
cycles, context switches, CPU migrations, etc. Table IX shows the ratio of
collected events on DNN inference between with and without post-processing.
The higher the value is, the more events happen during the post-processing.
Based on the comparison results, we can find that more events happen in AGX
Xavier than Fog node, including cache misses, instructions, cycles, context
switches, CPU clocks, and CPU migrations. Events like CPU migrations, context
switches, cycles, and CPU clocks directly contributed to the time variations
in post-processing. The Fog node has a server-level CPU, while the AGX
Xavier’s CPU is mainly used on embedding and mobile systems. We conclude that
embedding systems on autonomous vehicles with more powerful CPUs could help
decrease the time variations in DNN inference.

TABLE IX: The events comparison of running PINet and YOLOv3 on Jetson AGX and
Fog Node with/without post-processing.

w. / w. o. post-process | PINet | YOLOv3
---|---|---
metrics | AGX | Fog | GPU | AGX | Fog | GPU
branch-misses | 1.18 | 1.02 | 1.07 | 1 | 1.02 | 1.02
cache-misses | 1.04 | 1.03 | 1.25 | 1.03 | 1.03 | 1.03
cache-references | 1.09 | 1.03 | 1.07 | 1.03 | 1 | 1.01
instructions | 1.06 | 1.04 | 1.11 | 1.04 | 1 | 1
cycles | 1.12 | 1.02 | 1.04 | 1.04 | 1 | 1.02
context-switches | 1.06 | 0.31 | 1.15 | 1.36 | 0.99 | 1.15
cpu-migrations | 0.95 | 0.66 | 4.54 | 1.10 | 1.01 | 1.03
page-faults | 1 | 0.65 | 1 | 1 | 1.04 | 0.98

Insight 5: The time variations in PINet are mainly caused by the post-
processing on the CPU. Although GPU performs better in conducting matrix
operations, having a powerful CPU in the autonomous vehicle embedding system
would help to decrease the time variations in DNN inference.

##  IV End-to-End System for Autonomous Driving

Autonomous driving vehicles are composed of a variety of applications for
sensing, perception, and control. The performance of the system is expected to
rely on the coordination of several modules. To evaluate autonomous driving
system’s variation, we propose to build an end-to-end prototype based on ROS
in the NVIDIA Jetson AGX board with RT kernel.

###  IV-A Overview of the End-to-End System

Based on the overview autonomous driving system in Figure 1, we develop a ROS
framework for the end-to-end system, as shown in Figure 14. Since this paper
focuses on the variations of DNN inference and the planning modules are all
rule-based algorithms with stable execution time, the ROS framework does not
include that part. The whole pipeline starts with the /image node, capturing
images from the cameras or image files, and publishing it. Three perception
nodes subscribe /image_raw messages and execute the DNN inference on the
images. Three perception nodes are responsible for Simultaneous Localization
and Mapping (SLAM), object detection, and semantic segmentation. After DNN
inference, three nodes publish their results, covering the position
information, objects, and semantics. Another node called /fusion subscribes to
these three nodes. It synchronizes them to get the sensor fusion results,
which gives the control module the location of the vehicle and obstacles and
open space for driving.

In our design, the algorithm used for SLAM is ORB-SLAM2, which is a pure
camera-based approach to capture key points in pixels and localize the
vehicles and generate maps simultaneously [37]. YOLOv3 is used for object
detection, while Deeplabv3 is used for semantic segmentation [43, 16].

Figure 14: The ROS framework of the end-to-end system.

###  IV-B ROS Nodes and Topics

A ROS node (i.e., the nodes in the figure) is a process to perform a
particular computation while ROS topics (i.e., the arrows in the figure) are
named buses for ROS nodes to exchange messages [13]. In the end-to-end system,
we implement five ROS nodes to access the sensor data and process the data:
/image, /darknet_ros, /slam, /segmentation, and /fusion. Publish-subscriber-
based message sharing is used to transmit messages between these nodes.

The ROS topics are defined to exchange messages between ROS nodes. Ten ROS
topics are implemented to exchange messages, including images, positions,
objects, and other customized messages. The summarized descriptions of some
ROS topics are reported in Table X. There are two messages based on Image
type: header, height, width, encoding, data, etc. The ROS topic’s header
contains the sequence ID, timestamp, and frame ID to represent a specific
message. Since timestamp and sequence ID are needed for the synchronization,
we implement /pose_timestamp based on /pose, which contains the position and
orientation data. For object detection, bounding boxes are used to present the
detected objects, which is determined by min and max values of the x and
y-axis and the probability and the object class. /bounding_boxes includes all
the bounding boxes for one image and contains a header inside. For semantic
segmentation, the results are shown as different colors inside the image to
represent different segments. An Image-based topic called /semantic is used to
represent the results with the message header.

TABLE X: ROS topics used in the end-to-end system

ROS Topics | Library | Type | Fields
---|---|---|---
/image_raw | sensor_msgs | Image |  | header, height, width, encoding, data, etc
---
/pose_timestamp | geometry_msgs | PoseStamped | header, pose
/pose | geometry_msgs | Pose |  | position (x, y, z float64),
---
orientation (x, y, z, w, float64)
/bounding_boxes | darknet_ros_msgs | BoundingBoxes |  | header, image_header, bounding_boxes
---
/bounding_box | darknet_ros_msgs | BoundingBox |  | probability, xmin, ymin, xmax, ymax, id, class
---
/semantics | sensor_msgs | Image |  | header, height, width, encoding, data, etc
---

###  IV-C Message Synchronization

For ROS nodes that need to subscribe to multiple ROS topics and process them
together, message synchronization becomes one of the implementation issue.
Typically, message synchronization is based on the timestamp and sequence ID,
so we convert the /pose message to /pose_timestamp to add on the header.

The /fusion node’s objective is to combine all the perception results of the
same image frame. The first thing is to make a unique ID for each image frame.
In the beginning, the /image node attaches timestamp information and frame ID
to each message it publishes out. For three perception nodes, after DNN
inference on the coming image frame, the timestamp and sequence ID of the
coming images will be used as the header’s timestamp and sequence ID of the
new message like /pose_timestamp, /semantic, etc. With unique IDs on each
image frame and detection results, the remaining question is how to make them
synchronized. In our design, we use
message_filter111http://wiki.ros.org/message_filters with Approximate Time
Synchronizer to manage the fusion process. The approximate synchronizer sets
queue size as 100 and 100ms as the slop, which means the message with a time
difference less than 100ms is considered synchronized.

##  V System-level Profiling

Compared with the typical model inference, the end-to-end system is expected
to have more uncertainties, contributing to higher variations. In this
section, with the ROS-based end-to-end system, we investigate the time
variation issues for the real autonomous driving system in two aspects:
typical module time analysis and end-to-end system time analysis.

###  V-A Latency Analysis for each Module

There are three typical modules in the ROS-based prototype: ORB-SLAM2 based
localization and mapping, YOLOv3 based object detection, and Deeplabv3 for
semantic segmentation. These modules subscribe to the image topic from the
/image node and execute the DNN inference, which means the I/O changes into
ROS communications. The total delay is defined as the difference between the
model inference finish time and the image’s sending timestamp from /image
node. In contrast, the total inference time is defined as the total time spent
within the perception module. The results for running each module separately
is shown in Figure 15. We can observe that for YOLOv3 and Deeplabv3, the
difference between total delay and total inference is huge, while ORB-SLAM2 is
small. The difference is because ROS communication has overhead in
transmitting images using pub/sub mechanisms. It is supposed to convert RGB
images into ROS images before publishing out and needs to convert it back to
RGB images for DNN inference after receiving it. The results of $C_{v}$
present that the total delay variation is higher than the total inference for
YOLOv3 and Deeplabv3.

Figure 15: The latency analysis for each ROS perception module when running
separately.

###  V-B System Latency Analysis

With multiple perception modules subscribing to the /image_raw topics
simultaneously and the ROS node /fusion combines their results, the system
latency is expected to show more variations than the separate module case. The
results for the system latency of each module is shown in Figure 16. We can
find that the time variations of YOLOv3 and Deeplabv3’s total delay are much
higher than the typical module case - with the highest delay almost attains
4000ms, which is dangerous for the safety-critical autonomous driving systems.
Besides, the 99 percentile for YOLOv3 and Deeplabv3 is more than the
combination of that in a typical module case, which implies huge tail latency
caused by the accumulation of variations with the competing of concurrent
tasks. $C_{v}$ shows models’ variations in which ORB-SLAM2 shows low variation
while two DNN-based module show huge variations.

Figure 16: The latency analysis for each ROS module when the end-to-end system
is running. Figure 17: The delay between fusion messages.

To understand the real impact of this enormous variation to the perception
system, we record the timestamps for the /fusion node to get synchronized
messages from three perception modules and generate the fusion data. Figure 17
shows the delays between the fusion messages in milliseconds. Queue size with
100 and 1000 of the buffer within the synchronizer are tested, respectively.
The delay variations under queue size with 1000 are less than that with 100.
For a queue size of 100, we can found that there are tremendous variations for
the delay. The worst case of the fusion message’s delay goes to over 10000ms,
which means that the control module gets the lane or objects’ perception
results after 10 seconds since the camera captures objects or lane. The
average and minimum value of the delay also goes to 773ms and 242ms,
respectively. This huge variation makes the control of the vehicle very
challenging because it is hard to predict.

Insight 6: Communication middleware like ROS brings huge extra variations to
the end-to-end autonomous driving systems. Long-tail latency exists, and it
accumulates with competing with concurrent tasks. Adding queue size inside the
synchronizer helps to reduce the variations.

##  VI Related Work

Autonomous vehicles are proposed to understand the environment and drive
without human intervention [33]. DNNs play an essential role in the sensing,
perception, decision, and control tasks in autonomous driving. Generally, the
research on DNNs for autonomous driving can be divided into two categories:
training DNN models with higher accuracy and improving the runtime performance
of trained DNN models.

Many DNN-based algorithms are deployed in autonomous vehicles for object
detection, lane detection, semantic segmentation, localization, etc. [31]. The
object detection algorithms can be divided into two types: one-stage based
algorithms like YOLO and SSD [43, 34]; two-stage based algorithms like Fast
R-CNN, Mask R-CNN, etc. [45, 25]. The key difference is whether there is a
proposal bounding box stage. Semantic segmentation is used to detect driving
segments. The fully convolutional neural network has been applied and achieves
good performance [36]. LaneNet is a lane detection algorithm that uses an
instance segmentation problem and applies image semantic segmentation
algorithms [38]. Another approach called PINet adds key points estimation with
the instance segmentation [28].

After the DNN models get trained, optimizing the model inference in latency,
energy consumption, and memory utilization becomes a big challenge. In 2015,
Han et al. [24] proposed pruning redundant connections and retraining the deep
learning models to fine-tune the weights effectively, reducing computing
complexity. Reducing the precision of operations and operands is another
direction for the runtime optimization of DNN inference. Reducing precision is
usually achieved by reducing the number of bits/levels representing the data,
decreasing the computation requirements and storage costs [47]. Besides, the
profiling of the DNN inference also gets more attention [51]. MLPerf targets a
uniformed profiling benchmark for the machine learning algorithms at the edge
[42].

##  VII Conclusion

DNNs are widely used in autonomous driving due to their high accuracy for
perception, decision, etc. Understanding the variation of the DNN inference of
autonomous driving becomes a fundamental challenge in real-time and efficient
scheduling. Non-negligible time variations are observed in DNN inference,
which significantly challenges scheduling safety-critical tasks. Therefore, in
this work, we analyze the time variation in DNN inference in fine granularity
and derive six insights into understanding DNN inference time variations.

## References

  * [1] AMCL.  http://wiki.ros.org/amcl.
  * [2] dwa_local_planner.  http://wiki.ros.org/dwa\\_local\\_planner.
  * [3] global_planner.  http://wiki.ros.org/global_planner.
  * [4] JetPack SDK 4.4 DP Archive.  https://developer.nvidia.com/jetpack-sdk-44-dp-archive.
  * [5] Jetson Modules.  https://developer.nvidia.com/embedded/jetson-modules.
  * [6] jetson_benchmarks.  https://github.com/NVIDIA-AI-IOT/jetson\\_benchmarks.
  * [7] Navigation.  http://wiki.ros.org/navigation.
  * [8] NVIDIA DRIVE AGX Developer Kit.  https://developer.nvidia.com/drive/drive-agx.
  * [9] NVIDIA L4T ML.  https://ngc.nvidia.com/catalog/containers/nvidia:l4t-ml.
  * [10] TCPROS.  http://wiki.ros.org/ROS/TCPROS.
  * [11] The KITTI Vision Benchmark Suite.  http://www.cvlibs.net/datasets/kitti/.
  * [12] The Python Profilers.  https://docs.python.org/3.6/library/profile.html#module-cProfile.
  * [13] Robot Operating System(ROS), Powering the World’s Robots, 2019.
  * [14] Can Basaran and Kyoung-Don Kang.  Supporting preemptive task executions and memory copies in GPGPUs.  In 2012 24th Euromicro Conference on Real-Time Systems, pages 287–296. IEEE, 2012.
  * [15] Thomas Bradley.  GPU performance analysis and optimisation.  NVIDIA Corporation, 2012.
  * [16] Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam.  Rethinking atrous convolution for semantic image segmentation.  arXiv preprint arXiv:1706.05587, 2017.
  * [17] Yu-Hsin Chen, Joel Emer, and Vivienne Sze.  Eyeriss: A spatial architecture for energy-efficient dataflow for convolutional neural networks.  In ACM SIGARCH Computer Architecture News, volume 44, pages 367–379. IEEE Press, 2016.
  * [18] Jeffrey Dean.  The deep learning revolution and its implications for computer architecture and chip design.  In 2020 IEEE International Solid-State Circuits Conference-(ISSCC), pages 8–14. IEEE, 2020.
  * [19] ePROSIMA.  eProsima Fast RTPS.  [Online].  https://www.eprosima.com/index.php/products-all/eprosima-fast-rtps.
  * [20] Andreas Geiger, Philip Lenz, and Raquel Urtasun.  Are we ready for autonomous driving? the KITTI vision benchmark suite.  In Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on, pages 3354–3361. IEEE, 2012.
  * [21] Ionel Gog, Sukrit Kalra, Peter Schafhalter, Joseph E Gonzalez, and Ion Stoica.  D3: a dynamic deadline-driven approach for building autonomous vehicles.  In Proceedings of the Seventeenth European Conference on Computer Systems, pages 453–471, 2022.
  * [22] Brendan Gregg.  Linux performance analysis and tools.  Technical report, Technical report, Joyent, 2013.
  * [23] Sorin Grigorescu, Bogdan Trasnea, Tiberiu Cocias, and Gigel Macesanu.  A survey of deep learning techniques for autonomous driving.  Journal of Field Robotics, 37(3):362–386, 2020.
  * [24] Song Han, Jeff Pool, John Tran, and William J Dally.  Learning both weights and connections for efficient neural networks.  arXiv preprint arXiv:1506.02626, 2015.
  * [25] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask R-CNN.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 2961–2969, 2017.
  * [26] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam.  MobileNets: efficient convolutional neural networks for mobile vision applications.  arXiv preprint arXiv:1704.04861, 2017.
  * [27] Shinpei Kato, Eijiro Takeuchi, Yoshio Ishiguro, Yoshiki Ninomiya, Kazuya Takeda, and Tsuyoshi Hamada.  An open approach to autonomous vehicles.  IEEE Micro, 35(6):60–68, 2015.
  * [28] Yeongmin Ko, Jiwon Jun, Donghwuy Ko, and Moongu Jeon.  Key points estimation and point instance segmentation approach for lane detection.  arXiv preprint arXiv:2002.06604, 2020.
  * [29] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  ImageNet classification with deep convolutional neural networks.  In Advances in neural information processing systems, pages 1097–1105, 2012.
  * [30] Liangkai Liu, Jiamin Chen, Marco Brocanelli, and Weisong Shi.  E2M: an energy-efficient middleware for computer vision applications on autonomous mobile robots.  In Proceedings of the 4th ACM/IEEE Symposium on Edge Computing (SEC), pages 59–73, 2019.
  * [31] Liangkai Liu, Sidi Lu, Ren Zhong, Baofu Wu, Yongtao Yao, Qingyang Zhang, and Weisong Shi.  Computing systems for autonomous driving: State-of-the-art and challenges.  arXiv preprint arXiv:2009.14349, 2020.
  * [32] Shaoshan Liu, Liyun Li, Jie Tang, Shuang Wu, and Jean-Luc Gaudiot.  Creating autonomous vehicle systems.  Synthesis Lectures on Computer Science, 6(1):i–186, 2017.
  * [33] Shaoshan Liu, Liangkai Liu, Jie Tang, Bo Yu, Yifan Wang, and Weisong Shi.  Edge computing for autonomous driving: Opportunities and challenges.  Proceedings of the IEEE, 107(8):1697–1716, 2019.
  * [34] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C Berg.  SSD: Single shot multibox detector.  In European conference on computer vision, pages 21–37. Springer, 2016.
  * [35] Wei Liu, Hao Wu, Ziyue Jiang, Yifan Gong, and Jiangming Jin.  A robotic communication middleware combining high performance and high reliability.  In 2020 IEEE 32nd International Symposium on Computer Architecture and High Performance Computing (SBAC-PAD), pages 217–224. IEEE, 2020.
  * [36] Jonathan Long, Evan Shelhamer, and Trevor Darrell.  Fully convolutional networks for semantic segmentation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3431–3440, 2015.
  * [37] Raul Mur-Artal et al.  ORB-SLAM2: an open-source SLAM system for monocular, stereo and rgb-d cameras.  arXiv preprint arXiv:1610.06475, 2016.
  * [38] Davy Neven, Bert De Brabandere, Stamatios Georgoulis, Marc Proesmans, and Luc Van Gool.  Towards end-to-end lane detection: an instance segmentation approach.  In 2018 IEEE Intelligent Vehicles Symposium (IV), pages 286–291. IEEE, 2018.
  * [39] Gerardo Pardo-Castellote.  Omg data-distribution service: Architectural overview.  In 23rd International Conference on Distributed Computing Systems Workshops, 2003. Proceedings., pages 200–206. IEEE, 2003.
  * [40] Chang Yun Park.  Predicting deterministic execution times of real-time programs.  1992\.
  * [41] Morgan Quigley, Ken Conley, Brian Gerkey, Josh Faust, Tully Foote, Jeremy Leibs, Rob Wheeler, and Andrew Y Ng.  ROS: an open-source robot operating system.  In ICRA workshop on open source software, volume 3, page 5. Kobe, Japan, 2009.
  * [42] Vijay Janapa Reddi, Christine Cheng, David Kanter, Peter Mattson, Guenther Schmuelling, Carole-Jean Wu, Brian Anderson, Maximilien Breughe, Mark Charlebois, William Chou, et al.  MLPerf inference benchmark.  In 2020 ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA), pages 446–459. IEEE, 2020.
  * [43] Joseph Redmon and Ali Farhadi.  YOLOv3: an incremental improvement.  arXiv preprint arXiv:1804.02767, 2018.
  * [44] Federico Reghenzani, Giuseppe Massari, and William Fornaciari.  The real-time linux kernel: A survey on Preempt_RT.  ACM Computing Surveys (CSUR), 52(1):1–36, 2019.
  * [45] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster R-CNN: Towards real-time object detection with region proposal networks.  In Advances in neural information processing systems, pages 91–99, 2015.
  * [46] Marco Spuri and Giorgio C Buttazzo.  Efficient aperiodic service under earliest deadline scheduling.  In RTSS, pages 2–11, 1994.
  * [47] Vivienne Sze, Yu-Hsin Chen, Tien-Ju Yang, and Joel S Emer.  Efficient processing of deep neural networks: A tutorial and survey.  Proceedings of the IEEE, 105(12):2295–2329, 2017.
  * [48] Maxime Tremblay, Shirsendu Sukanta Halder, Raoul de Charette, and Jean-François Lalonde.  Rain rendering for evaluating and improving robustness to bad weather.  International Journal of Computer Vision, 129(2):341–360, 2021\.
  * [49] Chengcheng Wan, Muhammad Santriaji, Eri Rogers, Henry Hoffmann, Michael Maire, and Shan Lu.  ALERT: Accurate learning for energy and timeliness.  In 2020 USENIX Annual Technical Conference (USENIX ATC 20), pages 353–369, 2020.
  * [50] Reinhard Wilhelm, Jakob Engblom, Andreas Ermedahl, Niklas Holsti, Stephan Thesing, David Whalley, Guillem Bernat, Christian Ferdinand, Reinhold Heckmann, Tulika Mitra, et al.  The worst-case execution-time problem—overview of methods and survey of tools.  ACM Transactions on Embedded Computing Systems (TECS), 7(3):1–53, 2008.
  * [51] Carole-Jean Wu, David Brooks, Kevin Chen, Douglas Chen, Sy Choudhury, Marat Dukhan, Kim Hazelwood, Eldad Isaac, Yangqing Jia, Bill Jia, et al.  Machine learning at Facebook: Understanding inference at the edge.  In 2019 IEEE International Symposium on High Performance Computer Architecture (HPCA), pages 331–344. IEEE, 2019.
  * [52] Tianze Wu, Baofu Wu, Sa Wang, Liangkai Liu, Shaoshan Liu, Yungang Bao, and Weisong Shi.  Oops! it’s too late. your autonomous driving system needs a faster middleware.  IEEE Robotics and Automation Letters, 6(4):7301–7308, 2021.
  * [53] Shifeng Zhang, Longyin Wen, Xiao Bian, Zhen Lei, and Stan Z Li.  Single-shot refinement neural network for object detection.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4203–4212, 2018.