# Rethinking BiSeNet For Real-time Semantic Segmentation

Mingyuan Fan Equal contribution. Shenqi Lai11footnotemark: 1 Junshi Huang Co-
corresponding author. Xiaoming Wei22footnotemark: 2 Zhenhua Chai
Junfeng Luo  Xiaolin Wei
Meituan
{fanmingyuan laishenqi  huangjunshi  weixiaoming  chaizhenhua
luojunfeng<EMAIL_ADDRESS>
###### Abstract

BiSeNet [28, 27] has been proved to be a popular two-stream network for real-
time segmentation. However, its principle of adding an extra path to encode
spatial information is time-consuming, and the backbones borrowed from
pretrained tasks, e.g., image classification, may be inefficient for image
segmentation due to the deficiency of task-specific design. To handle these
problems, we propose a novel and efficient structure named Short-Term Dense
Concatenate network (STDC network) by removing structure redundancy.
Specifically, we gradually reduce the dimension of feature maps and use the
aggregation of them for image representation, which forms the basic module of
STDC network. In the decoder, we propose a Detail Aggregation module by
integrating the learning of spatial information into low-level layers in
single-stream manner. Finally, the low-level features and deep features are
fused to predict the final segmentation results. Extensive experiments on
Cityscapes and CamVid dataset demonstrate the effectiveness of our method by
achieving promising trade-off between segmentation accuracy and inference
speed. On Cityscapes, we achieve 71.9% mIoU on the test set with a speed of
250.4 FPS on NVIDIA GTX 1080Ti, which is 45.2% faster than the latest methods,
and achieve 76.8% mIoU with 97.0 FPS while inferring on higher resolution
images. Code is available at https://github.com/MichaelFan01/STDC-Seg.

##  1 Introduction

Figure 1: Speed-Accuracy performance comparison on the Cityscapes test set.
Our methods are presented in red dots while other methods are presented in
blue dots. Our approaches achieve state-of-the-art speed-accuracy trade-off.

Semantic segmentation is a classic and fundamental topic in computer vision,
which aims to assign pixel-level labels in images. The prosperity of deep
learning greatly promotes the performance of semantic segmentation by making
various breakthroughs [18, 27, 22, 4], coming with fast-growing demands in
many applications, e.g., autonomous driving, video surveillance, robot
sensing, and so on. These applications motivate researchers to explore
effective and efficient segmentation networks, particularly for mobile field.

To fulfill those demands, many researchers propose to design low-latency,
high-efficiency CNN models with satisfactory segmentation accuracy. These
real-time semantic segmentation methods have achieved promising performance on
various benchmarks. For real-time inference, some works, e.g., DFANet [18] and
BiSeNetV1 [28] choose the lightweight backbones and investigate ways of
feature fusion or aggregation modules to compensate for the drop of accuracy.
However, these lightweight backbones borrowed from image classification task
may not be perfect for image segmentation problem due to the deficiency of
task-specific design. Besides the choice of lightweight backbones, restricting
the input image size is another commonly used method to promote the inference
speed. Smaller input resolution seems to be effective, but it can easily
neglect the detailed appearance around boundaries and small objects. To tackle
this problem, as shown in Figure 2(a), BiSeNet [28, 27] adopt multi-path
framework to combine the low-level details and high-level semantics. However,
adding an additional path to get low-level features is time-consuming, and the
auxiliary path is always lack of low-level information guidance.

Figure 2: Illustration of architectures of BiSeNet [28] and our proposed
approach. (a) presents Bilateral Segmentation Network (BiSeNet [28]), which
use an extra Spatial Path to encode spatial information. (b) demonstrates our
proposed method, which use a Detail Guidance module to encode spatial
information in the low-level features without an extra time-cosuming path.

To this end, we propose a novel hand-craft network for the purpose of faster
inference speed, explainable structure, and competitive performance to that of
existing methods. First, we design a novel structure, called Short-Term Dense
Concatenate module (STDC module), to get variant scalable receptive fields
with a few parameters. Then, the STDC modules are seamlessly integrated into
U-net architecture to form the STDC network, which greatly promote network
performance in semantic segmentation task.

In details, as shown in Figure 3, we concatenate response maps from multiple
continuous layers, each of which encodes input image/feature in different
scales and respective fields, leading to multi-scale feature representation.
To speed up, the filter size of layers is gradually reduced with negligible
loss in segmentation performance. The details structure of STDC networks can
be found in Table 2.

In the phase of decoding, as shown in Figure 2(b), instead of utilizing an
extra time-consuming path, Detail Guidance are adopted to guide the low-level
layers for the learning of spatial details. We first utilize Detail
Aggregation module to generate detail ground-truth. Then, the binary cross-
entropy loss and dice loss are employed to optimize the learning task of
detail information, which is considered as one type of side-information
learning. It should be noted that this side-information is not required in the
inference time. Finally, the spatial details from low-level layers and
semantic information from deep layers are fused to predict the semantic
segmentation results. The whole architecture of our method is shown in Figure
4.

Our main contributions can be summarized as follows:

  * •

We design a Short-Term Dense Concatenate module (STDC module) to extract deep
features with scalable receptive field and multi-scale information. This
module promotes the performance of our STDC network with affordable
computational cost.

  * •

We propose the Detail Aggregation module to learn the decoder, leading to more
precise preservation of spatial details in low-level layers without extra
computation cost in the inference time.

  * •

We conduct extensive experiments to present the effectiveness of our methods.
The experiment results present that STDC networks achieve new state-of-the-art
results on ImageNet, Cityscapes and CamVid. Specifically, our STDC1-Seg50
achieves 71.9% mIoU on the Cityscapes test set at a speed of 250.4 FPS on one
NVIDIA GTX 1080Ti card. Under the same experiment setting, our STDC2-Seg75
achieves 76.8% mIoU at a speed of 97.0 FPS.

##  2 Related Work

###  2.1 Efficient Network Designs

Model design plays an important role in computer vision tasks. SqueezeNet [16]
used the fire module and certain strategies to reduce the model parameters.
MobileNet V1 [13] utilized depth-wise separable convolution to reduce the
FLOPs in inference phase. ResNet [9] [10] adopted residual building layers to
achieve outstanding performance. MobileNet V2 [25] and ShuffleNet [29] used
group convolution to reduce computation cost while maintaining comparable
accuracy. These works are particularly designed for the image classification
tasks, and their extensions to semantic segmentation application should be
carefully tuned.

###  2.2 Generic Semantic Segmentation

Traditional segmentation algorithms, e.g., threshold selection, super-pixel,
utilized the hand-crafted features to assign pixel-level labels in images.
With the development of convolution neural network, methods [3, 1, 32, 14]
based on FCN [23] achieved impressive performance on various benchmarks. The
Deeplabv3 [3] adopted an atrous spatial pyramid pooling module to capture
multi-scale context. The SegNet [1] utilized the encoder-decoder structure to
recover the high-resolution feature maps. The PSPNet [32] devised a pyramid
pooling to capture both local and global context information on the dilation
backbone. Both dilation backbone and encoder-decoder structure can
simultaneously learn the low-level details and high-level semantics. However,
most approaches require large computation cost due to the high-resolution
feature and the complicate network connections. In this paper, we propose an
efficient and effective architecture which achieves good trade-off between
speed and accuracy.

###  2.3 Real-time Semantic Segmentation

Recently, there are fast-growing practical applications for real-time semantic
segmentation. In this circumstance, there are two mainstreams to devise
efficient segmentation methods. (i) lightweight backbone. DFANet [18] adopted
a lightweight backbone to reduce computation cost and devised a cross-level
feature aggregation module to enhance performance. DFNet [21] utilized
“Partial Order Pruning” algorithm to obtain a lightweight backbone and
efficient decoder. (ii) multi-branch architecture. ICNet [31] devised the
multi-scale image cascade to achieve good speed-accuracy trade-off. BiSeNetV1
[28] and BiSeNetV2 [27] proposed two-stream paths for low-level details and
high-level context information, separately. In this paper, we propose an
efficient lightweight backbone to provide scalable receptive field.
Furthermore, we set a single path decoder which uses detail information
guidance to learn the low-level details.

Figure 3:  (a) General STDC network architecture. ConvX operation refers to
the Conv-BN-ReLU. (b) Short-Term Dense Concatenate module (STDC module) used
in our network. M denotes the dimension of input channels, N denotes the
dimension of output channels. Each block is a ConvX operation with different
kernel size. (c) STDC module with stride=2.

##  3 Proposed Method

BiSeNetV1 [28] utilizes lightweight backbones, e.g., ResNet18 and spatial path
as encoding networks to form two-steam segmentation architecture. However, the
classification backbones and two-stream architecture may be inefficient due to
the structure redundancy. In this section, we first introduce the details of
our proposed STDC network. Then we present the whole arhitecture of our
single-stream method with detail guidance.

###  3.1 Design of Encoding Network

####  3.1.1 Short-Term Dense Concatenate Module

STDC module | Block1 | Block2 | Block3 | Block4 | Fusion
---|---|---|---|---|---
RF(S = 1) | $1\times 1$ | $3\times 3$ | $5\times 5$ | $7\times 7$ |  $1\times 1$, $3\times 3$ $5\times 5$, $7\times 7$
RF(S = 2) | $1\times 1$ | $3\times 3$ | $7\times 7$ | $11\times 11$ |  $3\times 3$ $7\times 7$, $11\times 11$
|  |  |  |  |
Table 1: Receptive Field of blocks in our STDC module. RF denotes Receptive
Field, S means stride, Note that if stride=2, the $1\times 1$ RF of Block1 is
turned into $3\times 3$ RF by Average Pool operation.

The key component of our proposed network is the Short-Term Dense Concatenate
module (STDC module). Figure 3(b) and (c) illustrate the layout of STDC
module. Specifically, each module is separated into several blocks, and we use
$ConvX_{i}$ to denote the operations of $i$-th block. Therefore, the output of
$i$-th block is calculated as follows:

$x_{i}=ConvX_{i}(x_{i-1},k_{i})$ (1)

where $x_{i-1}$ and $x_{i}$ are the input and output of $i$-th block,
separately. ConvX includes one convolutional layer, one batch normalization
layer and ReLU activation layer, and $k_{i}$ is the kernel size of
convolutional layer.

In STDC module, the kernel size of first block is 1, and the rest of them are
simply set as 3. Given the channel number of STDC module’s output $N$, the
filter number of convolutional layer in $i$-th block is $N/2^{i}$, except the
filters of last convolutional layer, whose number is the same to that of
previous convolutional layer. In image classification tasks, its a common
practice to using more channels in higher layers. But in semantic segmentation
tasks, we focus on scalable receptive field and multi-scale informations. Low-
level layers need enough channels to encode more fine-grained informations
with small receptive field, while high-level layers with large receptive field
focus more on high-level information induction, setting the same channel with
low-level layers may cause information redundancy. Down-sample is only
happened in Block2. To enrich the feature information, we concatenate $x_{1}$
to $x_{n}$ feature maps as the output of STDC module by skip-path. Before
concatenation, the response maps of different blocks in STDC module is down-
sampled to the same spatial size by average pooling operation with $3\times 3$
pooling size, as shown in Figure 3(c). In our setting, the final output of
STDC module is:

$x_{output}=F(x_{1},x_{2},...,x_{n})$ (2)

where $x_{output}$ denotes the STDC module output, $F$ is the fusion operation
in our method, while $x_{1},x_{2},...,x_{n}$ are feature maps from all $n$
blocks. In the consideration of efficiency, we adopt concatenation as our
fusion operation. In our method, we use the STDC module in $4$ blocks.

Table 1 presents the receptive field of blocks in STDC module, and
$x_{output}$ thus gathers multi-scale information from all blocks, We claim
that our STDC module has two advantages: (1) we elaborately tune the filter
size of blocks by gradually decreasing in geometric progression manner,
leading to significant reduction in computation complexity. (2) the final
output of STDC module is concatenated from all blocks, which preserves
scalable respective fields and multi-scale information.

Given the input channel dimension $M$ and output channel dimension $N$, the
parameter number of STDC module is:

$\displaystyle S_{param}$ $\displaystyle=M\times 1\times
1\times\frac{N}{2^{1}}+\sum_{i=2}^{n-1}\frac{N}{2^{i-1}}\times 3\times
3\times\frac{N}{2^{i}}+$ $\displaystyle\frac{N}{2^{n-1}}\times 3\times
3\times\frac{N}{2^{n-1}}$
$\displaystyle=\frac{NM}{2}+\frac{9N^{2}}{2^{3}}\times\sum_{i=0}^{n-3}\frac{1}{2^{2i}}+\frac{9N^{2}}{2^{2n-2}}$
$\displaystyle=\frac{NM}{2}+\frac{3N^{2}}{2}\times(1+\frac{1}{2^{2n-3}})$ (3)

As shown in Equation 3, the parameter number of STDC module is dominated by
the predefined input and output channel dimension, while the number of blocks
has slight impact on the parameter size. Particularly, if $n$ reaches the
maximum limit, the parameter number of STDC module almost keeps constant,
which is only defined by $M$ and $N$.

Stages | Output size | KSize | S | STDC1 | STDC2
---|---|---|---|---|---
R | C |  R | C
Image | 224$\times$224 |  |  |  | 3 |  | 3
ConvX1 | 112$\times$112 | 3$\times$3 | 2 | 1 | 32 | 1 | 32
ConvX2 | 56$\times$56 | 3$\times$3 | 2 | 1 | 64 | 1 | 64
Stage3 |  28$\times$28 28$\times$28 |  |  2 1 |  1 1 |  256 |  1 3 |  256
Stage4 |  14$\times$14 14$\times$14 |  |  2 1 |  1 1 |  512 |  1 4 |  512
Stage5 |  7$\times$7 7$\times$7 |  |  2 1 |  1 1 |  1024 |  1 2 |  1024
ConvX6 | 7$\times$7 | 1$\times$1 | 1 | 1 | 1024 | 1 | 1024
GlobalPool | 1$\times$1 | 7$\times$7 |  |  |  |  |
FC1 |  |  |  |  | 1024 |  | 1024
FC2 |  |  |  |  | 1000 |  | 1000
FLOPs |  |  |  | 813M | 1446M
Params |  |  |  | 8.44M | 12.47M
|  |  |  |  |  |  |
Table 2: Detailed architecture of STDC networks. Note that ConvX shown in the
table refers to the Conv-BN-ReLU. The basic module of Stage 3, 4 and 5 is STDC
module. KSize mean kernel size. S, R, C denote stride, repeat times and output
channels respectively.

####  3.1.2 Network Architecture

We demonstrate our network architecture in Figure 3(a). It consists of 6
stages except input layer and prediction layer. Generally, Stage 1$\sim$5
down-sample the spatial resolution of the input with a stride of 2,
respectively, and the Stage 6 outputs the prediction logits by one ConvX, one
global average pooling layer and two fully connected layer.

The Stage 1&2 are usually regarded as low-level layers for appearance feature
extraction. In pursuit of efficiency, we only use one convolutional block in
each of Stage 1&2, which is proved to be sufficient according to our
experiences. The number of STDC module in Stage 3, 4, 5 is carefully tuned in
our network. Within those stages, the first STDC module in each stage down-
samples the spatial resolution with a stride of 2. The following STDC modules
in each stage keep the spatial resolution unchanged.

We denote the output channel number of stage as $N_{l}$, where $l$ is the
index of stage. In practice, we empirically set $N_{6}$ as 1024, and carefully
tune the channel number of rest stages, until reaching a good trade-off
between accuracy and efficiency. Since our network mainly consists of Short-
Term Dense Concatenate modules, we call our network STDC network. Table 2
shows the detailed structure of our STDC networks.

Figure 4: Overview of the STDC Segmentation network. ARM denotes Attention
Refine module, and FFM denotes Feature Fusion Module in [28]. The operation in
the dashed red box is our STDC network. The operation in the dashed blue box
is Detail Aggregation Module.

###  3.2 Design of Decoder

####  3.2.1 Segmentation Architecture

We use the pretrained STDC networks as the backbone of our encoder and adopt
the context path of BiSeNet [28] to encode the context information. As shown
in Figure 4(a), we use the Stage 3, 4, 5 to produce the feature maps with
down-sample ratio 1/8, 1/16, 1/32, respectively. Then we use global average
pooling to provide global context information with large receptive field. The
U-shape structure are deployed to up-sample the features stem from global
feature, and combine each of them with the counterparts from last two stages
(Stage 4&5) in our encoding phase. Following BiSeNet [28], we use Attention
Refine module to refine the combination features of every two stages. For the
final semantic segmentation prediction, we adopt Feature Fusion module in
BiSeNet [28] to fuse the 1/8 down-sampled feature from Stage 3 in the encoder
and the counterpart from the decoder. We claim that the features of these two
stages are in different levels of feature representation. The feature from the
encoding backbone preserves rich detail information, while the feature from
the decoder contains context information due to the input from global pooling
layer. Specifically, the Seg Head includes a $3\times 3$ Conv-BN-ReLU operator
followed with a $1\times 1$ convolution to get the output dimension $N$, which
is set as the number of classes. We adopt cross-entry loss with Online Hard
Example Mining to optimize the semantic segmentation learning task.

Figure 5: Visual explanations for features in the spatial path and Stage 3
without and with Detail Guidance. The column with subscript D denotes results
with Detail Guidance. The visualization shows that spatial path can encode
more spatial detail,e.g., boundary, corners, than backbone’s low-level layers,
while our Detail Guidance module can do the same thing without extra
computation cost.

####  3.2.2 Detail Guidance of Low-level Features

We visualize the features of BiSeNet’s spatial path in Figure 5(b). Compared
with the backbone’s low-level layers(Stage 3) of same downsample ratio,
spatial path can encode more spatial detail, e.g., boundary, corners. Based on
this observation, we propose a Detail Guidance module to guide the low-level
layers to learn the spatial information in single-stream manner. We model the
detail prediction as a binary segmentation task. We first generate the detail
map ground-truth from the segmentation ground-truth by Laplacian operator as
shown in Figure 4 (c). As illustrated in Figure 4(a), we insert the Detail
Head in Stage 3 to generate the detail feature map. Then we use the detail
ground-truth as the guidance of detail feature map to guide the low-level
layers to learn the feature of spatial details. As shown in Figure 5(d), the
feature map with detail guidance can encode more spatial details than
aforementioned result presented in Figure 5(c). Finally, the learned detail
features are fused with the context features from the deep block of the
decoder for segmentation prediction.
Detail Ground-truth Generation: We generate the binary detail ground-truth
from the semantic segmentation ground-truth by our Detail Aggregation module,
as shown in dashed blue box of Figure 4(c). This operation can be carried out
by 2-D convolution kernel named Laplacian kernel and a trainable $1\times 1$
convolution. We use the Laplacian operator shown in Figure 4(e) to produce
soft thin detail feature maps with different strides to obtain mult-scale
detail informations. Then we upsample the detail feature maps to the original
size and fuse it with a trainable $1\times 1$ convolution for dynamic re-
wegihting. Finally, we adopt a threshold $0.1$ to convert the predicted
details to the final binary detail ground-truth with boundary and corner
informations.
Detail Loss: Since the number of detail pixels is much less than the non-
detail pixels, detail prediction is a class-imbalance problem. Because
weighted cross-entropy always leads to coarse results, following [7], we adopt
binary cross-entropy and dice loss to jointly optimize the detail learning.
Dice loss measures the overlap between predict maps and ground-truth. Also, it
is insensitive to the number of foreground/background pixels, which means it
can alleviating the class-imbalance problem. So for the predicted detail map
with the height $H$ and the width $W$, the detail loss $L_{detail}$ is
formulated as follows:

$L_{detail}(p_{d},g_{d})=L_{dice}(p_{d},g_{d})+L_{bce}(p_{d},g_{d})$ (4)

where $p_{d}\in\mathbb{R}^{H\times W}$ denotes the predicted detail and
$g_{d}\in\mathbb{R}^{H\times W}$ denotes the corresponding detail ground-
truth. $L_{bce}$ denotes the binary cross-entropy loss while $L_{dice}$
denotes the dice loss, which is given as follows:

$L_{dice}(p_{d},g_{d})=1-\frac{2\sum_{i}^{H\times
W}p_{d}^{i}g_{d}^{i}+\epsilon}{\sum_{i}^{H\times
W}(p_{d}^{i})^{2}+\sum_{i}^{H\times W}(g_{d}^{i})^{2}+\epsilon}$ (5)

where $i$ denotes the $i$-th pixel and $\epsilon$ is a Laplace smoothing item
to avoid zero division. In this paper we set $\epsilon=1$.

As shown in Figure 4(b), we use a Detail Head to produce the detail map, which
guide the shallow layer to encode spatial information. Detail Head includes a
$3\times 3$ Conv-BN-ReLU operator followed with a $1\times 1$ convolution to
get the output detail map. In the experiment, the Detail Head is proved to be
effective to enhance the feature representation. Note that this branch is
discarded in the inference phase. Therefore, this side-information can easily
boost the accuracy of segmentation task without any cost in inference.

Figure 6: Visual comparison of our Detail Guidance on Cityscapes val set. The
column with subscript D denotes results with Detail Guidance. The first row
(a) shows the input images. (b) and (c) illustrate the heatmap of Stage 3
without and with Detail Guidance. (d) and (e) demonstrate the predictions
without and with Detail Guidance. (f) is the ground-truth of input images.
Figure 7: Comparisons with different block number of STDC2 on ImageNet.

##  4 Experimental Results

We implement our method on three datasets: ImageNet [6], Cityscapes [5] and
CamVid [2] to evaluate the effectiveness of our proposed backbone and
segmentation network, respectively. We first introduce the datasets and
implementation details. Then, we report our accuracy and speed results on
different benchmarks compared with other algorithms. Finally, we discuss the
impact of components in our proposed approach.

###  4.1 Benchmarks and Evaluation Metrics

ImageNet. The ILSVRC [6] 2012 is the most popular image classification
dataset. It contains 1.2 million images for training, and 50,000 for
validation with 1,000 categories. It is also widely used for training a
pretrained model for downstream tasks, like object detection or semantic
segmentation.
Cityscapes. Cityscapes [5] is a semantic scene parsing dataset, which is taken
from a car perspective. It contains 5,000 fine annotated images and split into
training, validation and test sets, with 2,975, 500 and 1,525 images
respectively. The annotation includes 30 classes, 19 of which are used for
semantic segmentation task. The images have a high resolution of $2,048\times
1,024$, thus it is challenging for the real-time semantic segmentation. For
fair comparison, we only use the fine annotated images in our experiments.
CamVid. Cambridge-driving Labeled Video Database (Camvid) [2] is a road scene
dataset, which is taken from a driving automobile perspective. This dataset
contains 701 annotated images extracted from the video sequence, in which 367
for training, 101 for validation and 233 for testing. The images have a
resolution of $960\times 720$ and $32$ semantic categories, in which the
subset of $11$ classes are used for segmentation experiments.
Evaluation Metrics. For classification evaluation, we use evaluate top-1
accuracy as the evaluation metrics following [9]. For segmentation evaluation,
we adopt mean of class-wise intersection over union (mIoU) and Frames Per
Second (FPS) as the evaluation metrics.

###  4.2 Implementation Details

Image Classification. We use mini-batch stochastic gradient descent (SGD) with
batch size 64, momentum $0.9$ and weight decay $1e^{-4}$ to train the model.
Three training methods from [11] are adopted, including learning rate warmup,
cosine learning rate policy and label smoothing. The total epochs is 300 with
warmup strategy at the first 5 epochs, within which the learning rate starts
from 0.001 to 0.1. The dropout before classification block is set to 0.2. We
do not use other special data augmentations, and all of them are the same as
[9].
Semantic Segmentation. We use mini-batch stochastic gradient descent (SGD)
with momentum $0.9$, weight decay $5e^{-4}$. The batch size is set as $48$,
$24$ for the Cityscapes, CamVid dataset respectively. As common configuration,
we utilize ”poly” learning rate policy in which the initial rate is multiplied
by $(1-\frac{iter}{max\\_iter})^{power}$. The power is set to $0.9$ and the
initial learning rate is set as $0.01$. Besides, we train the model for
$60,000$, $10,000$ iterations for the Cityscapes, CamVid dataset respectively,
in which we adopt warmup strategy at the first $1000$, $200$ iterations.

Data augmentation contains color jittering, random horizontal flip, random
crop and random resize. The scale ranges in [0.125, 1.5] and cropped
resolution is $1024\times 512$ for training Cityscapes. For training CamVid,
the scale ranges in [0.5, 2.5] and cropped resolution is $960\times 720$.

In all experiments, we conduct our experiments base on pytorch-1.1 on a
docker. We perform all experiments under CUDA 10.0, CUDNN 7.6.4 and TensorRT
5.0.1.5 on NVIDIA GTX 1080Ti GPU with batch size $1$ for benchmarking the
computing power of our method.

###  4.3 Ablation Study

Backbone | Resolution | mIoU(%) | FPS
---|---|---|---
GhostNet [8] | $512\times 1024$ | 67.8 | 135.0
MobileNetV3 [12] | $512\times 1024$ | 70.1 | 148.3
EfficientNet-B0 [26] | $512\times 1024$ | 72.2 | 99.9
STDC2 | $512\times 1024$ | 74.2 | 188.6
GhostNet [8] | $768\times 1536$ | 71.3 | 60.9
MobileNetV3 [12] | $768\times 1536$ | 73.0 | 70.4
EfficientNet-B0 [26] | $768\times 1536$ | 73.9 | 45.9
STDC2 | $768\times 1536$ | 77.0 | 97.0
|  |  |
Table 3: Lightweight backbone comparison on Cityscapes val set. All experients utilize the same decoder and same experiment settings. Method | SP | DG | mIoU(%) | FPS
---|---|---|---|---
4x | 2x | 1x
BiSeNetV1 [28] | ✓ |  |  |  | 69.0 | 105.8
STDC2-50 | ✓ |  |  |  | 73.7 | 171.6
STDC2-50 |  |  |  |  | 73.0 | 188.6
STDC2-50 |  | ✓ |  |  | 73.4 | 188.6
STDC2-50 |  |  | ✓ |  | 73.6 | 188.6
STDC2-50 |  |  |  | ✓ | 73.8 | 188.6
STDC2-50 |  |  | ✓ | ✓ | 73.9 | 188.6
STDC2-50 |  | ✓ | ✓ | ✓ | 74.2 | 188.6
|  |  |  |  |  |
Table 4: Detail information comparison on Cityscapes val set. SP means method
with Spatial Path and DG indicates Detail Guidance, inwhich 1x, 2x, 4x denotes
detail features with different down-sample strides in Detail Aggregation
module.

This section introduces the ablation experiments to validate the effectiveness
of each component in our method.
Effectiveness of STDC Module. We adjust the block number of STDC module in
STDC2 and present the result in Figure 7. According to our Equation 3, as the
group number increases, the FLOPs decrease obviously. And the best performance
is in 4 blocks. The benefits of more blocks become very small and a deeper
network is bad for the parallel calculation and FPS. Hence, in this paper, we
set the block number in STDC1 and STDC2 to 4.
Effectiveness of Our backbone. To verify the effectiveness of our backbone
designed for real-time segmentation, we adopt the latest lightweight backbones
which has comparable classification performance compared with STDC2 to
formulate a semantic segmentation network with our decoder. As show in Table.
3, our STDC2 yield the best speed-accuracy trade-off comapred with other
lightweight backbones.
Effectiveness of Detail Guidance. We first visualize the heatmap of the
feature map of Stage $3$ as shown in Figure 6. The features of Stage 3 with
detail guidance encode more spatial information comparing to that of Stage $3$
without detail guidance. Hence the final prediction of small objects and
boundaries are more precise. We show some quantitative results in Table 4. To
verify the effectiveness of our Detail Guidance, we show the comparison of
different detail guidance strategies of STDC2-Seg on Cityscapes val dataset.
To further demonstrate the capability of Detail Guidance, we first use the
Spatial Path in BiSeNetV1 [28] to encode the spatial information, then use the
features generated from the Spatial Path to replace the features from Stage
$3_{D}$. The setting of experiment with Spatial Path are exactly the same with
other experiments. As shown in Table 4, Detail Guidance in STDC2-Seg can
improve the mIoU without harming the inference speed. Adding Spatial Path to
encode spatial information can also improve the performance on accuracy, but
it increases the computation cost at the same time. Also we find our Detail
Aggregation module encode the abundant detail information and yield the
highest mIoU with aggregation of 1x, 2x, 4x detail features.

Model | Top1 Acc. | Params | FLOPs | FPS
---|---|---|---|---
ResNet-18 [9] | 69.0% | 11.2M | 1800M | 1058.7
ResNet-50 [9] | 75.3% | 23.5M | 3800M | 378.7
DF1 [21] | 69.8% | 8.0M | 746M | 1281.3
DF2 [21] | 73.9% | 17.5M | 1770M | 713.2
DenseNet121 [15] | 75.0% | 9.9M | 2882M | 363.6
DenseNet161 [15] | 76.2% | 28.6M | 7818M | 255.0
GhostNet(x1.0) [8] | 73.9% | 5.2M | 141M | 699.1
GhostNet(x1.3) [8] | 75.7% | 7.3M | 226M | 566.2
MobileNetV2 [25] | 72.0% | 3.4M | 300M | 998.8
MobileNetV3 [12] | 75.2% | 5.4M | 219M | 661.2
EfficientNet-B0 [26] | 76.3% | 5.3M | 390M | 443.0
STDC1 | 73.9% | 8.4M | 813M | 1289.0
STDC2 |  76.4% | 12.5M | 1446M | 813.6
|  |  |  |
Table 5: Comparisons with other popular networks on ImageNet Classification.

###  4.4 Compare with State-of-the-arts

In this part, we compare our methods with other existing state-of-the-art
methods on three benchmarks, ImageNet, Cityscapes and CamVid.
Results on ImageNet. As shown in Table 5, our STDC networks achieves higher
speed and accuracy compared with other lightweight backbones. Compared with
the lightweight backbones used in real-time segmentation, e.g., DF1Net, the
top-1 classification accuracy of our STDC1 network is 4.1% higher than
baseline on the ImageNet validation set. Compared with populuar lightweight
networks, e.g. EfficientNet-B0, the FPS of STDC2 network is 83.7% higher than
that of baseline with competitive classification result.

Model | Resolution | Backbone | mIoU(%) | FPS
---|---|---|---|---
val | test
ENet [24] | $512\times 1024$ | no | - | 58.3 | 76.9
ICNet [31] | $1024\times 2048$ | PSPNet50 | - | 69.5 | 30.3
DABNet [17] | $1024\times 2048$ | no | - | 70.1 | 27.7
DFANet B [18] | $1024\times 1024$ | Xception B | - | 67.1 | 120
DFANet A’ [18] | $512\times 1024$ | Xception A | - | 70.3 | 160
DFANet A [18] | $1024\times 1024$ | Xception A | - | 71.3 | 100
BiSeNetV1 [28] | $768\times 1536$ | Xception39 | 69.0 | 68.4 | 105.8
BiSeNetV1 [28] | $768\times 1536$ | ResNet18 | 74.8 | 74.7 | 65.5
CAS [30] | $768\times 1536$ | no | - | 70.5 | 108.0
GAS [22] | $769\times 1537$ | no | - | 71.8 | 108.4
DF1-Seg-d8 [21] | $1024\times 2048$ | DF1 | 72.4 | 71.4 | 136.9
DF1-Seg[21] | $1024\times 2048$ | DF1 | 74.1 | 73.0 | 106.4
DF2-Seg1[21] | $1024\times 2048$ | DF2 | 75.9 | 74.8 | 67.2
DF2-Seg2[21] | $1024\times 2048$ | DF2 | 76.9 | 75.3 | 56.3
SFNet [20] | $1024\times 2048$ | DF1 | - | 74.5 | 121
HMSeg [19] | $768\times 1536$ | no | - | 74.3 | 83.2
TinyHMSeg [19] | $768\times 1536$ | no | - | 71.4 | 172.4
BiSeNetV2 [27] | $512\times 1024$ | no | 73.4 | 72.6 | 156
BiSeNetV2-L [27] | $512\times 1024$ | no | 75.8 | 75.3 | 47.3
FasterSeg [4] | $1024\times 2048$ | no | 73.1 | 71.5 | 163.9
STDC1-Seg50 | $512\times 1024$ | STDC1 | 72.2 | 71.9 | 250.4
STDC2-Seg50 | $512\times 1024$ | STDC2 | 74.2 | 73.4 | 188.6
STDC1-Seg75 | $768\times 1536$ | STDC1 | 74.5 | 75.3 | 126.7
STDC2-Seg75 | $768\times 1536$ | STDC2 | 77.0 | 76.8 | 97.0
|  |  |  |  |
Table 6: Comparisons with other state-of-the-art methods on Cityscapes. no
indicates the method do not have a backbone.

Results on Cityscapes. As shown in Table 6, we present the segmentation
accuracy and inference speed of our proposed method on Cityscapes validation
and test set. Following the previous methods [27, 22], we use the training set
and validation set to train our models before submitting to Cityscapes online
server. At test phase, we first resize the image into the fixed size
$512\times 1024$ or $768\times 1536$ to inference, then we up-sample the
results to $1024\times 2048$. Overall, our methods get the best speed-accuracy
trade-off among all methods. We use $50$ and $75$ after the method name to
represent the input size $512\times 1024$ and $768\times 1536$ respectively.
For example, with the STDC1 backbone and $512\times 1024$ input size, we name
the method STDC1-Seg50. As shown in Table 6, our STDC1-Seg50 achieves a
significantly faster speed than baselines, i.e., 250.4 FPS, and still has
71.9% mIoU on test set, which is over 45.2% faster than the runner-up. Our
STDC2-Seg50 using $512\times 1024$ input size achieves 73.4% mIOU with 188.6
FPS, which is the state-of-the-art trade-off between performance and speed.
For $768\times 1536$ input size, our STDC2-Seg75 achieves the best mIOU 77.0%
in validation set and 76.8% on test set at 97.0 FPS.
Results on CamVid. We also evaluate our method on CamVid dataset. Table 7
shows the comparison results with other methods. With the input size
$720\times 960$, STDC1-Seg achieves 73.0% mIoU with 197.6 FPS which is the
state-of-the-art trade-off between performance and speed. This further
demonstrates the superior capability of our method.

Model | Resolution | Backbone | mIoU(%) | FPS
---|---|---|---|---
ENet [24] | $720\times 960$ | no | 51.3 | 61.2
ICNet [31] | $720\times 960$ | PSPNet50 | 67.1 | 34.5
BiSeNetV1 [28] | $720\times 960$ | Xception39 | 65.6 | 175
BiSeNetV1 [28] | $720\times 960$ | ResNet18 | 68.7 | 116.3
CAS [30] | $720\times 960$ | no | 71.2 | 169
GAS [22] | $720\times 960$ | no | 72.8 | 153.1
BiSeNetV2 [27] | $720\times 960$ | no | 72.4 | 124.5
BiSeNetV2-L [27] | $720\times 960$ | no | 73.2 | 32.7
STDC1-Seg | $720\times 960$ | STDC1 | 73.0 | 197.6
STDC2-Seg | $720\times 960$ | STDC2 | 73.9 | 152.2
|  |  |  |
Table 7: Comparisons with other state-of-the-art methods on CamVid. no
indicates the method do not have a backbone.

##  5 Conclusions

In this paper, we revisit the classical segmentation architecture BiSeNet [28,
27] for structure optimization. Generally, the classification backbone and
extra spatial path of BiSeNet greatly hinder the inference efficiency.
Therefore, we propose a novel Short-Term Dense Concatenate Module to extract
deep features with scalable receptive field and multi-scale information. Based
on this module, STDC networks are designed and achieve competitive accuracy
with high FPS in image classification.Using STDC networks as backbone, our
detail-guided STDC-Seg achieves state-of-the-art speed-accuracy trade-off in
real-time semantic segmentation. Extensive experiments and visualization
results indicates the effectiveness of our proposed STDC-Seg networks. In
future, we extend our method by following directions: (i) the backbone will be
validated in more tasks, e.g., object detection. (ii) we will explore deeper
on the utilization of spatial boundary in semantic segmentation tasks.

Acknowledgements This research is supported by Beijing Science and Technology
Project. (No.Z181100008918018).

## References

  * [1] Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla.  Segnet: A deep convolutional encoder-decoder architecture for image segmentation.  IEEE transactions on pattern analysis and machine intelligence, 39(12):2481–2495, 2017.
  * [2] Gabriel J. Brostow, Jamie Shotton, Julien Fauqueur, and Roberto Cipolla.  Segmentation and recognition using structure from motion point clouds.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 44–57, 2008.
  * [3] Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam.  Rethinking atrous convolution for semantic image segmentation.  arXiv preprint arXiv:1706.05587, 2017.
  * [4] Wuyang Chen, Xinyu Gong, Xianming Liu, Qian Zhang, Yuan Li, and Zhangyang Wang.  Fasterseg: Searching for faster real-time semantic segmentation.  In International Conference on Learning Representations, 2020.
  * [5] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele.  The cityscapes dataset for semantic urban scene understanding.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3213–3223, 2016.
  * [6] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * [7] Ruoxi Deng, Chunhua Shen, Shengjun Liu, Huibing Wang, and Xinru Liu.  Learning to predict crisp boundaries.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 562–578, 2018.
  * [8] Kai Han, Yunhe Wang, Qi Tian, Jianyuan Guo, Chunjing Xu, and Chang Xu.  Ghostnet: More features from cheap operations.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1580–1589, 2020.
  * [9] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [10] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Identity mappings in deep residual networks.  In European conference on computer vision, pages 630–645. Springer, 2016.
  * [11] Tong He, Zhi Zhang, Hang Zhang, Zhongyue Zhang, Junyuan Xie, and Mu Li.  Bag of tricks for image classification with convolutional neural networks.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 558–567, 2019.
  * [12] Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al.  Searching for mobilenetv3.  In Proceedings of the IEEE International Conference on Computer Vision, pages 1314–1324, 2019.
  * [13] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam.  Mobilenets: Efficient convolutional neural networks for mobile vision applications.  arXiv preprint arXiv:1704.04861, 2017.
  * [14] Ping Hu, Fabian Caba, Oliver Wang, Zhe Lin, Stan Sclaroff, and Federico Perazzi.  Temporally distributed networks for fast video semantic segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8818–8827, 2020.
  * [15] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger.  Densely connected convolutional networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700–4708, 2017.
  * [16] Forrest N Iandola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William J Dally, and Kurt Keutzer.  Squeezenet: Alexnet-level accuracy with 50x fewer parameters and $\textless$0.5mb model size.  arXiv preprint arXiv:1602.07360, 2016.
  * [17] Gen Li and Joongkyu Kim.  Dabnet: Depth-wise asymmetric bottleneck for real-time semantic segmentation.  In British Machine Vision Conference, 2019.
  * [18] Hanchao Li, Pengfei Xiong, Haoqiang Fan, and Jian Sun.  Dfanet: Deep feature aggregation for real-time semantic segmentation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9522–9531, 2019.
  * [19] Peike Li, Xuanyi Dong, Xin Yu, and Yi Yang.  a. when humans meet machines: Towards efficient segmentation networks.  In Proceedings of the British Machine Vision Conference (BMVC), 2020\.
  * [20] Xiangtai Li, Ansheng You, Zhen Zhu, Houlong Zhao, Maoke Yang, Kuiyuan Yang, and Yunhai Tong.  Semantic flow for fast and accurate scene parsing.  arXiv preprint arXiv:2002.10120, 2020.
  * [21] Xin Li, Yiming Zhou, Zheng Pan, and Jiashi Feng.  Partial order pruning: for best speed/accuracy trade-off in neural architecture search.  In Proceedings of the IEEE Conference on computer vision and pattern recognition, pages 9145–9153, 2019.
  * [22] Peiwen Lin, Peng Sun, Guangliang Cheng, Sirui Xie, Xi Li, and Jianping Shi.  Graph-guided architecture search for real-time semantic segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4203–4212, 2020.
  * [23] Jonathan Long, Evan Shelhamer, and Trevor Darrell.  Fully convolutional networks for semantic segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3431–3440, 2015.
  * [24] Adam Paszke, Abhishek Chaurasia, Sangpil Kim, and Eugenio Culurciello.  Enet: A deep neural network architecture for real-time semantic segmentation.  arXiv preprint arXiv:1606.02147, 2016.
  * [25] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen.  Mobilenetv2: Inverted residuals and linear bottlenecks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510–4520, 2018.
  * [26] Mingxing Tan and Quoc V Le.  Efficientnet: Rethinking model scaling for convolutional neural networks.  arXiv preprint arXiv:1905.11946, 2019.
  * [27] Changqian Yu, Changxin Gao, Jingbo Wang, Gang Yu, and Nong Sang.  Bisenet v2: Bilateral network with guided aggregation for real-time semantic segmentation.  arXiv preprint arXiv:2004.02147, 2020.
  * [28] Changqian Yu, Jingbo Wang, Chao Peng, Changxin Gao, Gang Yu, and Nong Sang.  Bisenet: Bilateral segmentation network for real-time semantic segmentation.  In Proceedings of the European conference on computer vision (ECCV), pages 325–341, 2018.
  * [29] Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun.  Shufflenet: An extremely efficient convolutional neural network for mobile devices.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6848–6856, 2018.
  * [30] Yiheng Zhang, Zhaofan Qiu, Jingen Liu, Ting Yao, Dong Liu, and Tao Mei.  Customizable architecture search for semantic segmentation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 11641–11650, 2019.
  * [31] Hengshuang Zhao, Xiaojuan Qi, Xiaoyong Shen, Jianping Shi, and Jiaya Jia.  Icnet for real-time semantic segmentation on high-resolution images.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 405–420, 2018.
  * [32] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia.  Pyramid scene parsing network.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2881–2890, 2017.


# Radar Spectra-Language Model for Automotive Scene Parsing

Mariia Pushkareva1, Yuri Feldman1†, Csaba Domokos2, Kilian Rambach2, Dotan Di
Castro1
Bosch Center for Artificial Intelligence, Haifa, Israel1 and Renningen,
Germany2
†Corresponding author. Email<EMAIL_ADDRESS>
###### Abstract

Radar sensors are low cost, long-range, and weather-resilient. Therefore, they
are widely used for driver assistance functions, and are expected to be
crucial for the success of autonomous driving in the future. In many
perception tasks only pre-processed radar point clouds are considered. In
contrast, radar spectra are a raw form of radar measurements and contain more
information than radar point clouds. However, radar spectra are rather
difficult to interpret. In this work, we aim to explore the semantic
information contained in spectra in the context of automated driving, thereby
moving towards better interpretability of radar spectra. To this end, we
create a radar spectra-language model, allowing us to query radar spectra
measurements for the presence of scene elements using free text. We overcome
the scarcity of radar spectra data by matching the embedding space of an
existing vision-language model (VLM). Finally, we explore the benefit of the
learned representation for scene parsing, and obtain improvements in free
space segmentation and object detection merely by injecting the spectra
embedding into a baseline model.

###### Index Terms:

radar deep learning, vision language model

##  I Introduction

Radar is a valuable sensing modality in the automotive domain, combining the
benefits of low hardware cost with long-range and weather-resilient sensing.
Radar sensors are already used for driver assistance functions and are
expected to be crucial for autonomous driving. Nevertheless, developing a well
performing perception algorithm, e.g., to detect all relevant objects, is a
challenging task. Numerous radar perception algorithms are based on radar
point cloud data as input [1, 2, 3, 4]. To compute the point cloud data, first
the measured baseband time signal is converted to radar spectra. Then local
intensity maxima, the radar reflections, are filtered out. This results in a
list of radar reflections, the radar point cloud. Therefore, information that
is available in the raw spectral radar data, is inevitably lost in the point
cloud data [5]. Recent work [6, 7, 8, 9, 10, 11] shows that perception
algorithms applied on radar spectra can achieve improved performance.
Nevertheless, working on radar spectra introduces new challenges: First of
all, there are only a small number of labeled datasets available providing
radar spectra. Furthermore, radar spectra data is difficult to interpret by
humans, as evidenced by Fig. 1 depicting the RGB image alongside its
corresponding range-Doppler spectrum. This leads naturally to the question:
what scene information is captured in radar spectra?

Figure 1: Training of a radar spectra-language model utilizes a frozen vision-
language model for supervision. Radar spectra encoder is trained to match
image embeddings of the corresponding RGB images. In this way, text embeddings
get aligned to radar embeddings as well.

To address the above, we propose to train a radar spectra-language model
(RSLM) for automotive scenarios, motivated by the tremendous success of
vision-language models (VLMs) like CLIP [12], or DALL-E [13]. Said model can
subsequently be used to query radar measurements for contents of interest
using free text, a step towards understanding the semantic content of radar
spectra.

To train the radar encoder, we utilize the frozen image encoder of a VLM, i.
e. the weights of the image encoder are not adapted during training of the
radar encoder, cf. Fig. 1. During training, the radar encoder embeddings are
forced to match the feature embeddings of the image encoder. In the VLM, the
output feature embeddings of the text encoder are aligned to the feature
embeddings of the image encoder, i. e. text domain is connected to image
domain. By aligning the feature embeddings of the radar encoder to the ones of
the image encoder, the feature embeddings of the radar encoder are aligned to
the ones of the text encoder as well, i. e. text domain is connected to radar
spectra domain, see Fig. 1. In this way, we construct the radar spectra-
language model. To the best of our knowledge, we are the first ones to train a
radar-language model. Note that for training the radar encoder only paired
image-radar spectra samples are needed, no labeled spectral radar data is
necessary. This tackles the problem of a large labeled radar spectra dataset,
which is usually needed for a supervised training.

We are especially interested in automotive applications. Since performance of
off-the-shelf VLMs is not satisfactory, we fine-tune VLMs for automotive
scenes. To explore the semantic content of radar spectra, we benchmark the
RSLM on scene retrieval tasks: Free text is used to describe a scene, and the
RSLM is used to search for data samples which fit to this scene description.
Moreover, we show that the RSLM can be used to improve the performance on two
downstream tasks, object detection on radar spectra and free space space
estimation.

Our main contributions can be summarized as follows:
1) We propose training and evaluation of the first radar spectra-language
model. 2) We benchmark scene retrieval using the radar spectra-language model,
exploring semantic content of radar spectra. 3) We investigate the benefits of
the learned radar feature embeddings on two downstream tasks: object detection
and free space estimation.

##  II Related Work

#### Vision-language Models

Large VLMs have shown great potential in learning representations that are
transferable across a wide range of downstream tasks. An efficient way to
learn image representations by making use of contrastive training on image-
caption pairs was proposed in [12]. [14] shows the advantage of fine-tuning
text models with frozen (pre-trained) image models. However, the connection
between text and other modalities has received less attention. [15] proposed
to train encoders of several modalities. The closest approach to our work is
LidarCLIP [16], which learns a mapping from Lidar point clouds to CLIP [12]
embedding space, effectively relating text and Lidar data through the image
domain. Our work was inspired by that idea, however, we consider a new input
modality, namely radar spectra. We leverage vision-language models to achieve
a better representation for radar spectra input.

#### VLMs for Automotive Scene Understanding

In recent research, VLMs are used for automotive applications [17, 18]. Scene
understanding with VLMs is investigated in the form of object detection [19,
20] and visual questioning answering (VQA) [21, 22], producing usually
descriptions which capture only a subset of scene elements. Captioning
approaches [20] require expensive ground truth annotation and [16] relies on a
large-scale automotive dataset. Both are not available in our case. Romero et
al. [23] propose an approach, that matches the input scene measurement to an
embedding vector, which lies in the same representation space as the text
embedding. Thus the model can be queried using free text. However, it utilizes
out-of-the-box CLIP, and is limited by its performance.

#### Object Detection on Spectra

Object detection on automotive radar spectra is attracting increasing interest
since recent introduction of public datasets [10, 8, 9]. A radar dataset and a
one-stage detector generating both 3D and 2D bounding boxes was proposed in
[11]. The CRUW dataset and an object detection network on range-azimuth radar
spectra was presented in [8]. FFT-RadNet [10] eliminates the overhead of
computing the range-azimuth-Doppler tensor by learning to recover angles from
a range-Doppler spectrum. DAROD [24] is an adaptation of Faster R-CNN for
automotive radar on range-Doppler spectra. [25] proposed hierarchical Swin
vision transformers for radar object detection.

##  III Proposed Approach

Since the introduction of VLMs [12], the coupling of image and text latent
representation spaces has been leveraged to enable semantic perception “in-
the-wild” accross modalities [15]. We aim to harness this generalization
ability to examine the semantic content of radar spectra of automotive scenes.
To this end, we train a radar spectra-language model, consisting of a spectra
encoder and a text encoder with a shared embedding space, representing the
observed scenes. The radar spectra encoder is trained by using paired radar
spectra-image measurements from automotive driving datasets. It is trained to
match the embedding space of a VLM, following [16]. To obtain an embedding
space that better fits our data we formulate a process for fine-tuning a VLM
using generated captions of automotive scenes, without any human annotations.
We validate our approach in two ways: 1) We evaluate our method on a spectra
retrieval task using text queries, directly attempting to shed light on what
elements of the scene are captured in radar spectra. 2) We inject the spectra
embedding in a baseline object-detection and segmentation network to observe
an improvement in detection and segmentation performance.

The rest of this section is organized as follows: The proposed process of
fine-tuning a VLM with automotive data is described in Section III-A. We
explain the training and architecture of the proposed radar spectra encoder in
Section III-B, and present its application for downstream tasks in Section
III-C.

###  III-A VLM Fine-tuning

Publicly available VLMs are generally not specifically adapted to automotive
scenes, e.g., CLIP accuracy of zero-shot classification for KITTI dataset
varies from 21% to 44% [12]. Therefore, we fine-tune a baseline VLM for road
scenes. We use a segmentation model to generate labels for the presence and
position of different objects within each image. Using these labels, multiple
different captions are generated for each dataset frame, based on the objects
which are present in this frame. This way, diverse captions can be generated
automatically. Those captions along with the corresponding images are used to
fine-tune the VLM.

###  III-B Radar Spectra-language Model

To obtain paired radar-spectra and text encoders we use a similar concept as
presented in [16]. We train a radar encoder to output similar embeddings to a
VLM image encoder. During training of the radar encoder, the VLM model is
frozen, i. e. the weights of the VLM model are fixed, see Fig. 1. Matching
pairs of radar spectra and images are input to the network: an image to the
frozen VLM image encoder and a corresponding radar spectrum to the radar
encoder. We train the radar encoder to minimize the difference between the
image and radar encoder outputs, where both outputs, the radar embeddings and
image embeddings, have the same size. In this work, we compare two networks
for the radar encoder: A network with a CNN backbone and a network with a
Feature Pyramid Network (FPN) backbone, cf. Fig. 2.

CNN radar encoder The CNN network includes three convolutional layers, batch-
normalization, average pooling, a fully-connected layer and a layer
normalization. The parameters of the first convolutional layer depend on the
radar spectrum type and the number of input channels of the dataset at hand.

For the RADIal dataset, which includes range-Doppler spectra, we use the
recommended paramters given by [10]. For the CRUW dataset, which consists of
range-azimuth spectra, the parameters are chosen according to the spectra
dimensions.

FPN radar encoder We choose the Feature Pyramid Network (FPN) of FFT-RadNet
[10] as the radar encoder. Detection, and segmentation heads are not included
in the radar encoder. A convolutional layer and fully-connected layer are
added, to project the output to the same space as the CLIP embeddings. The
parameters of the first convolutional layer in the radar encoder are the same
as for the CNN radar encoder, and depend on the dataset at hand.

###  III-C Downstream Tasks

To investigate the benefit of the learned radar spectra feature embeddings for
different downstream tasks, we consider object detection as well as free space
estimation as two applications. To this end, we combine the trained radar
encoder with a detection and segmentation network. The overall architecture is
depicted in Fig. 3. We propose to inject embeddings from the pre-trained RSLM
radar encoder into the detection network. We hypothesize that this would
introduce a semantic prior benefiting detection and segmentation. Below, we
provide details on this architecture and the loss function for its training.

Detection Backbone We choose FFT-RadNet as used by [10] as our detection
backbone. There exists an optimized version of FFT-RadNet [26], but
hyperparameters haven’t been made public. T-FFTRadNet by [25] uses a Swin [27]
backbone as opposed to FPN in FFT-RadNet. Since code and exact parameters are
not available for T-FFTRadNet, we have choosen FFT-RadNet as baseline. We use
both the detection and the driveable space segmentation heads, as defined in
[10].

Incorporating the RSLM Embeddings The input radar spectra tensor is
concurrently fed into the detection backbone and the radar encoder. The radar
feature embeddings output by the radar encoder are transformed by an adapter
branch to match the size of the output features of the detection backbone, and
are summed with those. As radar encoder we use the FPN radar encoder.

Loss The loss function is defined as $L=L_{\text{det}}+\lambda
L_{\text{seg}}$, where $0<\lambda\in\mathbb{R}$ is a weighting factor and the
detection and segmentation losses are defined as follows:

$\displaystyle
L_{\text{det}}=\text{focal}(y_{\text{class}},\hat{y}_{\text{class}})+\beta~{}\text{smooth-}L_{1}(y_{\text{reg}}-\hat{y}_{\text{reg}}),$
(1) $\displaystyle
L_{\text{seg}}=\sum_{r,a}\text{BCE}(y_{\text{seg}}(r,a),\hat{y}_{\text{seg}}(r,a)),$
(2)

where $0<\beta\in\mathbb{R}$;
$\text{focal}(y_{\text{class}},\hat{y}_{\text{class}})$ is the focal loss for
true $y_{\text{class}}$ and predicted $\hat{y}_{\text{class}}$ class labels;
smooth-$L_{1}$ is the smooth $L_{1}$ loss, where $y_{\text{reg}}$ and
$\hat{y}_{\text{reg}}$ denote the ground-truth polar coordinates of the
centers of the bounding boxes and the predicted ones, respectively. BCE
denotes the binary cross entropy loss for true free-space map $y_{\text{seg}}$
and predicted map $\hat{y}_{\text{seg}}$, where $r$ and $a$ stand for range
and azimuth coordinates, respectively.

Figure 2: Architecture of the radar encoder, with FPN or CNN radar backbone.
The MIMO encoder is chosen according to the dataset (CRUW or RADIal). Figure
3: RSLM-Aided detection and segmentation architecture. Input spectra are
concurrently fed into the detection backbone and the radar encoder from the
pre-trained RSLM.

##  IV Experiments

We present experimental results for fine-tuning the CLIP image encoder for
automotive scenarios. The semantic content of radar spectra is analyzed in
Section IV-C, where the RSLM is evaluated on a retrieval task. In Section IV-D
the benefit of the trained radar encoder is evaluated on two downstream tasks:
radar object detection and free space space estimation.

###  IV-A Datasets

In this paper we use three datasets for autonomous driving: 1) RADIal [10] has
8252 annotated frames, each with a range-Doppler spectrum of size $512\times
256\times 16$, Lidar, radar reflections, images, centers of cars, and free
space annotations in birds-eye view. 2) The CRUW dataset [8] has 40,734
annotated frames with range-azimuth spectra of size $128\times 128\times 8$,
RGB images, centers of cars, pedestrians and cyclists in range-azimuth
coordinates. 3) The nuScenes dataset [28] includes Lidar and radar point
clouds, camera, IMU and GPS data. In this work, we only use images from the
train-validation split with 40,157 samples.

###  IV-B VLM finetuning

We use Open CLIP [29] ViT-L/14, pretrained on datacomp_xl, as the VLM. This
model is fine-tuned using image-caption pairs, as described in Section III-A,
using RADIal, CRUW, and nuScenes datasets. We compare the original and fine-
tuned VLM by evaluating retrieval performance for classes on the CRUW dataset,
which are particularly relevant for autonomous driving. To compute the model
predictions, the cosine similarity of the text and the image embeddings is
computed. We rank the retrieved data samples by the cosine similarity values.
The top-10 and top-100 retrieval metrics are listed in Table I. The results
show, that the fine-tuned VLM outperforms the original VLM on average.
Performance is only worse on some classes like bicycle, likely due to those
classes being underrepresented in the datasets used for fine-tuning.

###  IV-C Radar spectra-language model

TABLE I: Comparison of top-1 and top-100 precision scores for retrieval task for original VLM, fine-tuned VLM and RSLM on CRUW dataset. | Top 10 | Top 100
---|---|---
Label |  Original VLM |  Fine- tuned VLM |  CNN RSLM |  FPN RSLM |  Original VLM |  Fine- tuned VLM |  CNN RSLM |  FPN RSLM
sidewalk | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 0.99
building | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1
wall | 0.9 | 1 | 0.4 | 0.9 | 0.8 | 1 | 0.72 | 0.86
fence | 1 | 1 | 0.9 | 1 | 0.99 | 1 | 0.91 | 0.96
traffic light | 0.3 | 0.2 | 0.1 | 0.1 | 0.57 | 0.17 | 0.05 | 0.06
traffic sign | 1 | 1 | 1 | 0.8 | 1 | 1 | 0.88 | 0.87
person | 0.2 | 1 | 0 | 1 | 0.68 | 1 | 0.08 | 0.95
rider | 0.8 | 1 | 0.3 | 0.6 | 0.94 | 1 | 0.13 | 0.4
car | 0.7 | 1 | 1 | 1 | 0.84 | 0.92 | 1 | 0.93
truck | 1 | 1 | 0.6 | 0.7 | 1 | 0.85 | 0.28 | 0.25
bicycle | 1 | 1 | 0.7 | 0.4 | 1 | 0.26 | 0.26 | 0.2
Mean | 0.809 | 0.927 | 0.636 | 0.773 | 0.893 | 0.904 | 0.574 | 0.679

Setup We train a radar spectra encoder by matching the embedding of the
corresponding image produced by the image encoder of the frozen VLM. Here we
use the Open CLIP model fine-tuned to automotive scenes. Separate encoders are
trained for the range-Doppler spectra from RADIal dataset and range-azimuth
spectra from CRUW dataset. We trained the CNN and FPN radar encoder with mean
squared error (MSE) loss for matching the embeddings.

Evaluation of RSLM The trained RSLMs are evaluated on a retrieval task as
described above. The CLIP text encoder and our trained radar encoder are used
to compute the radar spectra-language model predictions. The retrieved data
samples are ranked by the cosine similarity values.

Results Retrieval performance is shown in Table I. The FPN radar encoder
outperforms the CNN radar encoder for most of the classes and by mean top-10
and top-100 accuracy. For person prompt, FPN achieves better results than the
original VLM, due to fine-tuning. Thus, the RSLM can be successfully applied
to retrieval tasks. In Fig. 4 images are shown, which correspond to the
retrieved spectra with maximal cosine similarity value for the given caption.
It shows, that the RSLM can retrieve objects and scenes like parking lots and
trucks, which were not presented in the ground truth. This shows, that radar
spectra and language can be succesfully connected using the RSLM.

(a) Parking lot with many cars

(b) truck cruising confidently on the open road

Figure 4: Data retrieval using the trained RSLM. The corresponding images are
shown for visualization only, they are not used for data retrieval. The used
query appears in the caption of each image.

###  IV-D Object detection and free-space segmentation with RSLM radar
embeddings

To evaluate the benefit of the learned radar embeddings of the RSLM, we
compare a) the baseline network “FFT-RadNet” [10] with b) the baseline network
including the radar encoder of the RSLM “FFT-RadNet + RSLM encoder”, cf.
Section III-C. We were not able to reproduce the results of “FFT-RadNet”
reported in [10] using the provided hyperparameters. Therefore, we provide the
results of our training to be able to compare it to the model including the
RSLM radar encoder.

Metrics We evaluate the models on the RADIal dataset and use the same metrics
as in [10]: For evaluating the object detection task the mean average
precision (mAP), mean average recall (mAR), and F1-score are computed. For
free space segmentation the intersection over union (IoU) is used.

Results Table II summarizes the results. Using the pre-trained RLSM radar
encoder (“FFT-RadNet + RLSM encoder”) improves object detection (mAP and
F1-score) and segmentation performance (IoU). mAR is very similar for both
models. Thus, simply injecting the pre-trained radar spectra embeddings of the
RSLM encoder improves the performance. Note that no additional labeled data is
necessary to pre-train the RLSM encoder. Visualizations for detection and
segmentation results can be found in Fig. 5 and Fig. 6.

TABLE II: Comparison of the baseline model “FFT-RadNet” with our proposed
model “FFT-RadNet + RSLM encoder”. Simply injecting the pre-trained radar
spectra embeddings of the RSLM encoder improves object detection (mAP, mAR,
F1) and segmentation (IoU) performance. Not that pre-training the RSLM encoder
does not require any additional ground truth data.

Model | mAP (%) | mAR (%) | F1 (%) | IoU (%)
---|---|---|---|---
FFT-RadNet(∗) |  88.8 $\pm$ 1.7 |  81.2 $\pm$ 1.8 | 84.2 |  67.3 $\pm$ 1
FFT-RadNet + RLSM encoder |  90.7 $\pm$ 1.1 |  81.8 $\pm$ 2 | 86.0 |  71.2 $\pm$ 2.3

$\mathbf{(*)}$ FFT-RadNet architecture from [10], trained by us.

TABLE III: Ablation studies for detection (mAP, mAR, and $F_{1}$-score) and
segmentation task (IoU). The different model architectures are described in
Section IV-D. The “frozen enc” and “from scratch” models, achieve the best
results. Note that training the “frozen enc” model doesn’t require any
additional ground truth data.

Model |  Detect backbone |  Radar enc |  RSLM weights |  Fine-tuned enc | mAP (%) | mAR (%) | F1 (%) | IoU (%)
---|---|---|---|---|---|---|---|---
frozen enc | + | + | + | - |  90.7 $\pm$ 1.1 | 81.8 $\pm$ 2 | 86.0 | 71.2 $\pm$ 2.3
fine-tuned enc | + | + | + | + | 90.4 $\pm$ 1.2 | 81.4 $\pm$ 2.1 | 85.6 | 69.9 $\pm$ 2.6
only-frozen enc | - | + | + | - | 0.1 $\pm$ 0 | 2.4 $\pm$ 0.6 | 0.1 | 55 $\pm$ 16.7
only fine-tuned enc | - | + | + | + |  $0.0\pm$ 0 | 2.7 $\pm$ 1.1 | 0 | 59.1 $\pm$ 9.9
from-scratch | + | + | - | + | 88.1 $\pm$ 2.8 |  82.9 $\pm$ 0.7 | 85.4 |  72.6 $\pm$ 1.9

$\mathbf{(*)}$ FFT-RadNet architecture from [10], trained by us.

Ablation Study For better understanding of the model and training methods, we
conduct an ablation study and compare the following models: “baseline” network
is FFT-RadNet [10], “frozen-enc” denotes the network “FFT-RadNet + RSLM
encoder” with the frozen, pre-trained radar encoder. “fine-tuned enc” is the
same network as “frozen-enc”, however, the radar encoder is fine-tuned on the
last 10 epochs. “only frozen enc” includes the pre-trained radar encoder,
radar adapter, detection and segmentation head only. “only fine-tuned enc” is
the same as “only frozen enc”, but the radar encoder is fine-tuned on last 10
epochs. “from-scratch” is a random initialized network with the ”frozen-enc”
architecture.

Table II summarizes object detection and segmentation performance for the
models described above. The columns of the table correspond to model features:
“detect backbone” signifies the use of the detection backbone (MIMO pre-
encoder and FPN Radar backbone) of FFT-RadNet. “radar enc” denotes the
incorporation of the radar encoder from RSLM. “RSLM weights” indicates the
usage of weights from the RSLM model for the radar encoder; otherwise, it is
randomly initialized. “Fine-tuned enc” signifies that the radar encoder was
fine-tuned during detection training; otherwise it is frozen. Results in the
table exhibit performance improvements when adding frozen radar embeddings
into the model architecture. This enhancement is observed in both detection
and free-space segmentation tasks, as compared to the baseline model without
embeddings (“frozen enc” and “fine-tuned enc” vs. “baseline”). Fine-tuning the
radar encoder does not improve object detection or segmentation performance.
Furthermore, the model “from-scratch” with the same architecture as the
“frozen-enc” variant, exhibits slightly higher IoU scores for free-space
segmentation and similar detection performance compared to the “frozen-enc”
model. In contrast, models that exclusively incorporate the radar encoder
component of RSLM (“only-frozen enc”, “only fine-tuned enc”), whether frozen
or trained during the last 10 epochs, do not successfully accomplish the
detection task.

This shows, that using the pre-trained radar encoder from RSLM in addition to
the detection backbone improves performance in downstream tasks (“frozen enc”
vs. “baseline”), i. e. learning the feature embeddings is helpful. Note, that
the pre-training does not use any labeled radar spectra data, and the weights
of the radar encoder lead to similar performance as weights trained in a fully
supervised manner (“frozen enc” vs. “from-scratch”). We emphasize that
improvements are achieved with the same hyperparameters as the baseline model
by just adding the RSLM radar encoder.

Discussion and Future Directions The proposed RSLM relies on a pre-trained
vision-language model, and therefore depends on the quality of the caption-
image pairs the underlying model was trained on. More captions would help to
fine-tune the corresponding VLM, yielding a better RSLM. This performance
dependence is an obvious limitation of the RSLM, as the VLM has a limited
performance on some fine-grained cases, e.g. it often cannot recognize traffic
signs.

Our results demonstrate that the proposed RSLM can learn relevant features for
scene retrieval. While only scene-level descriptions have been considered in
this paper, object-, or region- level descriptions (a 3D variant of e.g. [30,
31, 32]) would also be beneficial in future work.

Our experiments show that the learned features are relevant for downstream-
tasks. We emphasize that the observed performance boost is obtained without
the need for any additional labeled data, only by making use of image-radar
pairs.

Radar measurements are not significantly affected by bad weather conditions or
time of day, which is one main advantage. Therefore, while the proposed RSLM
was trained on images that are taken at daytime, it can be expected to work as
well in “rainy”, and “night” scenarios. However, this is not possible to
verify since images in difficult conditions are not available for the
considered datasets. New publicly released datasets with difficult weather
conditions, such as the recently available [9], will be beneficial. As another
future application, the proposed model might be used for radar data
generation.

Figure 5: Detection results of FFT-RadNet (green) and our proposed network
“FFT-RadNet + RLSM encoder”(blue). The bounding box prediction of FFT-RadNet
is displaced w.r.t the ground truth bounding box (red), whereas the “FFT-
RadNet + RLSM encoder” predictions align well with the ground truth.
Confidence score equals 0.1. Left: Bounding boxes in Cartesian coordinates
displayed on radar point clouds. Right: Bounding boxes projected on image.

(a) RGB

(b) FFT-RadNet

(c) Ours

Figure 6: Example of segmentation result of FFT-RadNet and our network “FFT-
RadNet + RLSM encoder”. Red color denotes ground truth open driving space,
green color represents free-space predicted by the corresponding model, and
yellow color denotes the intersection of ground truth and predicted drive-able
space. The predictions of our proposed method “FFT-RadNet + RLSM encoder” are
better aligned to the ground truth. Note that the models use radar spectra
only as input.

##  V Conclusion

We developed a radar spectra-language model (RSLM), to the best of our
knowledge the first such model, for automotive scenes. Our method makes use of
vision-language models (VLMs), which we first fine-tuned on automotive image
data to improve their performance. We investigated the semantic content of
radar spectra, by querying the RSLM with text descriptions and evaluating
radar scene retrieval. In this way the model can even be used to query for
different object types, for which no corresponding labels exist in the
dataset. Moreover, the proposed methods overcomes the scarcity of labeled
radar spectra data, since no labeled radar data is needed to train the RSLM.
Finally, we showed that the performance in downstream tasks can be improved by
injecting radar feature embeddings from the RSLM into a detection and
segmentation model.

## References

  * [1] M. Ulrich _et al._ , “Improved orientation estimation and detection with hybrid object detection networks for automotive radar,” in _ITSC_ , 2022\.
  * [2] A. Danzer _et al._ , “2D car detection in radar data with PointNets,” in _ITSC_ , 2019.
  * [3] P. Svenningsson _et al._ , “Radar–PointGNN: Graph based object recognition for unstructured radar point-cloud data,” in _RadarConf_ , 2021\.
  * [4] G. Bang _et al._ , “RadarDistill: Boosting Radar-based Object Detection Performance via Knowledge Distillation from LiDAR Features,” 2024.
  * [5] S. Yao _et al._ , “Radar perception in autonomous driving: Exploring different data representations,” _ArXiv_ , 2023.
  * [6] B. Major _et al._ , “Vehicle detection with automotive radar using deep learning on range-azimuth-doppler tensors,” in _ICCVWorkshop_ , 2019.
  * [7] A.-E. Cozma _et al._ , “DeepHybrid: Deep learning on automotive radar spectra and reflections for object classification,” in _ITSC_ , 2021.
  * [8] Y. Wang _et al._ , “RODNet: Radar object detection using cross-modal supervision,” in _WACV_ , 2021.
  * [9] D.-H. Paek _et al._ , “K-Radar: 4D radar object detection for autonomous driving in various weather conditions,” in _NeurIPS_ , 2022.
  * [10] J. Rebut _et al._ , “Raw high-definition radar for multi-task learning,” in _CVPR_ , 2022.
  * [11] A. Zhang _et al._ , “RADDet: Range-azimuth-doppler based radar object detection for dynamic road users,” in _CRV_ , 2021.
  * [12] A. Radford _et al._ , “Learning transferable visual models from natural language supervision,” in _ICML_ , 2021.
  * [13] A. Ramesh _et al._ , “Zero-shot text-to-image generation,” in _ICML_ , 2021.
  * [14] X. Zhai _et al._ , “LiT: Zero-shot transfer with locked-image text tuning,” 2022.
  * [15] R. Girdhar _et al._ , “ImageBind one embedding space to bind them all,” in _CVPR_ , 2023.
  * [16] G. Hess _et al._ , “LidarCLIP or: How I Learned to Talk to Point Clouds,” in _WACV_ , 2024.
  * [17] X. Zhou _et al._ , “Vision language models in autonomous driving and intelligent transportation systems,” _ArXiv_ , 2023.
  * [18] Z. Yang _et al._ , “A survey of large language models for autonomous driving,” _ArXiv_ , 2023.
  * [19] M. Najibi _et al._ , “Unsupervised 3D perception with 2D vision-language distillation for autonomous driving,” _ArXiv_ , 2023.
  * [20] X. Ding _et al._ , “HiLM-D: Towards high-resolution understanding in multimodal large language models for autonomous driving,” _ArXiv_ , 2023\.
  * [21] T. Qian _et al._ , “NuScenes-QA: A multi-modal visual question answering benchmark for autonomous driving scenario,” _ArXiv_ , 2023.
  * [22] V. Dewangan _et al._ , “Talk2BEV: Language-enhanced bird’s-eye view maps for autonomous driving,” _ArXiv_ , 2023.
  * [23] F. Romero _et al._ , “Zelda: Video analytics using vision-language models,” _ArXiv_ , 2023.
  * [24] C. Decourt _et al._ , “DAROD: A deep automotive radar object detector on range-doppler maps,” in _IV_ , 2022.
  * [25] J. Giroux _et al._ , “T-FFTRadNet: Object detection with Swin vision transformers from raw ADC radar signals,” _ArXiv_ , 2023.
  * [26] B. Yang _et al._ , “ADCNet: End-to-end perception with raw radar ADC data,” _ArXiv_ , 2023.
  * [27] Z. Liu _et al._ , “Swin transformer: Hierarchical vision transformer using shifted windows,” _ICCV_ , 2021.
  * [28] H. Caesar _et al._ , “nuScenes: A multimodal dataset for autonomous driving,” in _CVPR_ , 2020.
  * [29] M. Cherti _et al._ , “Reproducible scaling laws for contrastive language-image learning,” in _CVPR_ , 2023.
  * [30] L. Li _et al._ , “Grounded language-image pre-training,” in _CVPR_ , 2022\.
  * [31] Y. Zhong _et al._ , “RegionCLIP: Region-based Language-Image Pretraining,” _CVPR_ , 2021.
  * [32] L. Yao _et al._ , “DetCLIPv2: Scalable Open-Vocabulary Object Detection Pre-training via Word-Region Alignment,” _CVPR_ , 2023.