11institutetext: University of Technology Sydney, Australia 22institutetext:
Tencent AI Lab 33institutetext: Fujian Normal University 44institutetext:
University of Nottingham Ningbo China

# CAR: Class-aware Regularizations for
Semantic Segmentation

Ye Huang 11 Di Kang 22 Liang Chen 33 Xuefei Zhe 22
Wenjing Jia 11 Linchao Bao 22 Xiangjian He Corresponding author44

###### Abstract

Recent segmentation methods, such as OCR and CPNet, utilizing “class level”
information in addition to pixel features, have achieved notable success for
boosting the accuracy of existing network modules. However, the extracted
class-level information was simply concatenated to pixel features, without
explicitly being exploited for better pixel representation learning. Moreover,
these approaches learn soft class centers based on coarse mask prediction,
which is prone to error accumulation. In this paper, aiming to use class level
information more effectively, we propose a universal Class-Aware
Regularization (CAR) approach to optimize the intra-class variance and inter-
class distance during feature learning, motivated by the fact that humans can
recognize an object by itself no matter which other objects it appears with.
Three novel loss functions are proposed. The first loss function encourages
more compact class representations within each class, the second directly
maximizes the distance between different class centers, and the third further
pushes the distance between inter-class centers and pixels. Furthermore, the
class center in our approach is directly generated from ground truth instead
of from the error-prone coarse prediction. Our method can be easily applied to
most existing segmentation models during training, including OCR and CPNet,
and can largely improve their accuracy at no additional inference overhead.
Extensive experiments and ablation studies conducted on multiple benchmark
datasets demonstrate that the proposed CAR can boost the accuracy of all
baseline models by up to 2.23% mIOU with superior generalization ability. The
complete code is available at https://github.com/edwardyehuang/CAR.

###### Keywords:

Class-aware regularizations, semantic segmentation

##  1 Introduction

Semantic segmentation, which assigns a class label for each pixel in an image,
is a fundamental task in computer vision. It has been widely used in many
real-world scenarios that require the results of scene parsing for further
processing, e.g., image editing, autopilot, etc. It also benefits many other
computer vision tasks such as object detection and depth estimation.

After the early work FCN [16] which used fully convolutional networks to make
the dense per-pixel segmentation task more efficient, many works [36, 2] have
been proposed which have greatly advanced the segmentation accuracy on various
benchmark datasets. Among these methods, many of them have focused on better
fusing spatial domain context information to obtain more powerful feature
representations (termed _pixel features_ in this work) for the final per-pixel
classification. For example, VGG [21] utilized large square context
information by successfully training a very deep network, and DeepLab [2] and
PSPNet [36] utilized multi-scale features with the ASPP and PPM modules.

Recently, methods based on dot-product self-attention (SA) have become very
popular since they can easily capture the long-range relationship between
pixels [26, 7, 32, 35, 11, 37, 6, 20, 22]. SA aggregates information
dynamically (by different attention maps for different inputs) and selectively
(using weighted averaging spatial features according to their similarity
scores). Using multi-scale and self-attention techniques during spatial
information aggregation has worked very well (e.g., 80% mIOU on Cityscapes
[17] (single-scale w/o flipping)).

As complements to the above methods, many recent works have proposed various
modules to utilize class-level contextual information. The class-level
information is often represented by the class center/context prior which are
the mean features of each class in the images. OCR [31] and ACFNet [33]
extract “soft” class centers according to the predicted coarse segmentation
mask by using the weighted sum. CPNet [30] proposed a context prior
map/affinity map, which indicates if two spatial locations belong to the same
class, and used this predicted context prior map for feature aggregation.
However, they [31, 33, 30] simply concatenated these class-level features with
the original pixel features for the final classification.

In this paper, we also focus on utilizing class level information. Instead of
focusing on how to better extract class-level features like the existing
methods [31, 33, 30], we use the simple, but accurate, average feature
according to the GT mask, and focus on maximizing the inter-class distance
during feature learning. This is because it mirrors how humans can robustly
recognize an object by itself no matter what other objects it appears with.

Learning more separable features makes the features of a class less dependent
upon other classes, resulting in improved generalization ability, especially
when the training set contains only limited and biased class combinations
(e.g., cows and grass, boats and beach). Fig. 1 illustrates an example of such
a problem, where the classification of dog and sheep depends on the
classification of grass class, and has been mis-classified as cow. In
comparison, networks trained with our proposed CAR successfully generalize to
these unseen class combinations.

Figure 1: The concept of the proposed CAR. Our CAR optimizes existing models
with three regularization targets: 1) reducing pixels’ intra-class distance,
2) reducing inter-class center-to-center dependency, and 3) reducing pixels’
inter-class dependency. As highlighted in this example (indicated with a red
dot in the image), with our CAR, the grass class does not affect the
classification of dog/sheep as much as before, and hence successfully avoids
previous (w/o CAR) mis-classification.

To better achieve this goal, we propose CAR, a class-aware regularizations
module, that optimizes the class center (intra-class) and inter-class
dependencies during feature learning. Three loss functions are devised: the
first encourages more compact class representations within each class, and the
other two directly maximize the distance between different classes.
Specifically, an intra-class center-to-pixel loss (termed as “intra-c2p”, Eq.
(3)) is first devised to produce more compact representation within a class by
minimizing the distance between all pixels and their class center. In our
work, a class center is calculated as the averaged feature of all pixels
belonging to the same class according to the GT mask. More compact intra-class
representations leave a relatively large margin between classes, thus
contributing to more separable representations. Then, an inter-class center-
to-center loss (“inter-c2c”, Eq. (6)) is devised to maximize the distance
between any two different class centers. This inter-class center-to-center
loss alone does not necessarily produce separable representations for every
individual pixels. Therefore, a third inter-class center-to-pixel loss
(“inter-c2p”, Eq. (13)) is proposed to enlarge the distance between every
class center and all pixels that do not belong to the class.

In summary, our contributions in this work are:

  1. 1.

We propose a universal class-aware regularization module that can be
integrated into various segmentation models to largely improve the accuracy.

  2. 2.

We devise three novel regularization terms to achieve more separable and less
class-dependent feature representations by minimizing the intra-class variance
and maximizing the inter-class distance.

  3. 3.

We calculate the class centers directly from ground truth during training,
thus avoiding the error accumulation issue of the existing methods and
introducing no computational overhead during inference.

  4. 4.

We provide image-level feature-similarity heatmaps to visualize the learned
inter-class features with our CAR are indeed less related to each other.

##  2 Related Work

Self-Attention. Dot-product self-attention proposed in [26, 23] has been
widely used in semantic segmentation [7, 32, 35, 37]. Specifically, self-
attention determines the similarity between a pixel with every other pixel in
the feature map by calculating their dot product, followed by softmax
normalization. With this attention map, the feature representation of a given
pixel is enhanced by aggregating features from the whole feature map weighted
by the aforementioned attention value, thus easily taking long-range
relationship into consideration and yielding boosted performance. In self-
attention, in order to achieve correct pixel classification, the
representation of pixels belonging to the same class should be similar to gain
greater weights in the final representation augmentation.

Class Center. In 2019 [33, 31], the concept of _class center_ was introduced
to describe the overall representation of each class from the categorical
context perspective. In these approaches, the center representation of each
class was determined by calculating the dot product of the feature map and the
coarse prediction (i.e., weighted average) from an auxiliary task branch,
supervised by the ground truth [36]. After that, those intra-class centers are
assigned to the corresponding pixels on feature map. Furthermore, in 2020
[30], a learnable kernel and one-hot ground truth were used to separate the
intra-class center from inter-class center, and then concatenated with the
original feature representation.

All of these works [31, 33, 30] have focused on extracting the intra (inter)
class centers, but they then simply concatenated the resultant class centers
with the original pixel representations to perform the final logits. We argue
that the categorical context information can be utilized in a more effective
way so as to reduce the inter-class dependency.

To this end, we propose a CAR approach, where the extracted class center is
used to directly regularize the feature extraction process so as to boost the
differentiability of the learned feature representations (see Fig. 1) and
reduce their dependency on other classes. Fig. 2 contrasts the two different
designs. More details of the proposed CAR are provided in Sect. 3.

(a) Design of OCR, ACFNet and CPNet

(b) Our CAR

Figure 2: The difference between the proposed CAR and previous methods that
use class-level information. Previous models focus on extracting class center
while using simple concatenation of the original pixel feature and the
class/context feature for later classification. In contrast, our CAR uses
direct supervision related to class center as regularization during training,
resulting in small intra-class variance and low inter-class dependency. See
Fig. 1 and Sec. 3 for details.

Inter-Class Reasoning. Recently, [5, 13] studied the class dependency as a
dataset prior and demonstrated that inter-class reasoning could improve the
classification performance. For example, a car usually does not appear in the
sky, and therefore the classification of sky can help reduce the chance of
mis-classifying an object in the sky as a car. However, due to the limited
training data, such class-dependency prior may also contain bias, especially
when the desired class relation rarely appears in the training set.

Fig. 1 shows such an example. In the training set, cow and grass are dependent
on each other. However, as shown in this example, when there is a dog or sheep
standing on the grass, the class dependency learned from the limited training
data may result in errors and predict the target into a class that appears
more often in the training data, i.e., cow in this case. In our CAR, we design
inter-class and intra-class loss functions to reduce such inter-class
dependency and achieve more robust segmentation results.

##  3 Methodology

Figure 3: The proposed CAR approach. CAR can be inserted into various
segmentation models, right before the logit prediction module (A1-A4). CAR
contains three regularization terms, including (C) intra-class center-to-
center loss $\mathcal{L}_{\text{intra-c2p}}$ (Sec. 3.2.2), (D) inter-class
center-to-center loss $\mathcal{L}_{\text{inter-c2c}}$ (Sec. 3.3.2), and (E)
inter-class center-to-pixel loss $\mathcal{L}_{\text{inter-c2p}}$ (Sec.
3.3.3).

###  3.1 Extracting Class Centers from Ground Truth

Denote a feature map and its corresponding resized one-hot encoded ground-
truth mask as $\mathbf{X}\in\mathbb{R}^{H\times W\times C}$111$H$, $W$ and $C$
denote images’ height and width, and number of channels, respectively. and
$\mathbf{Y}\in\mathbb{R}^{H\times W\times N_{\text{class}}}$, respectively. We
first get the spatially flattened class mask
$\mathbf{Y}_{\text{flat}}\in\mathbb{R}^{HW\times N_{\text{class}}}$ and
flattened feature map $\mathbf{X}_{\text{flat}}\in\mathbb{R}^{HW\times C}$.
Then, the class center222It is termed as _class center_ in [33] and _object
region representations_ in [31]., which is the average features of all pixel
features of a class, can be calculated by:

$\bm{\mu}_{image}=\frac{\mathbf{Y}_{\text{flat}}^{T}\cdot\mathbf{X}_{\text{flat}}}{\mathbf{N}_{\text{non-
zero}}}\in\mathbb{R}^{N_{\text{class}}\times C},$ (1)

where $\mathbf{N}_{\text{non-zero}}$ denotes the number of non-zero values in
the corresponding map of the ground-truth mask $\mathbf{Y}$. In our
experiments, to alleviate the negative impact of noisy images, we calculate
the class centers using all the training images in a batch, and denote them as
$\bm{\mu_{\text{batch}}}$333We use $\bm{\mu}$ and omit the subscript $batch$
for clarity..

###  3.2 Reducing Intra-Class Feature Variance

####  3.2.1 Motivation.

More compact intra-class representation can lead to a relatively larger margin
between classes, and therefore result in more separable features. In order to
reduce the intra-class feature variance, existing works [26, 7, 37, 30, 11,
32] usually use self-attention to calculate the dot-product similarity in
spatial space to encourage similar pixels to have a compact distance
implicitly. For example, the self-attention in [26] implicitly pushed the
feature representation of pixels belonging to the same class to be more
similar to each other than those of pixels belonging to other classes. In our
work, we devise a simple _intra-class center-to-pixel loss_ to guide the
training, which can achieve this goal very effectively and produce improved
accuracy.

####  3.2.2 Intra-class Center-to-pixel Loss.

We define a simple but effective intra-class center-to-pixel loss to suppress
the intra-class feature variance by penalizing large distance between a pixel
feature and its class center. The Intra-class Center-to-pixel Loss
$\mathcal{L}_{\text{intra-c2p}}$ is defined by:

$\small\mathcal{L}_{\text{intra-c2p}}=f_{\text{mse}}(\mathcal{D}_{\text{intra-c2p}}),$
(2)

where

$\small\mathcal{D}_{\text{intra-c2p}}=(1-\mathbf{\sigma})\lvert\mathbf{Y}_{\text{flat}}\cdot\bm{\mu}-\mathbf{X}_{\text{flat}}\rvert.$
(3)

In Eq. (3), $\mathbf{\sigma}$ is a spatial mask indicating pixels being
ignored (i.e., ignore label), $\mathbf{Y}_{\text{flat}}\cdot\bm{\mu}$
distributes the class centers $\bm{\mu}$ to the corresponding positions in
each image. Thus, our intra-class loss $\mathcal{L}_{\text{intra-c2p}}$ will
push the pixel representations to their corresponding class center, using mean
squared error (MSE) in Eq. (3).

###  3.3 Maximizing Inter-class Separation

####  3.3.1 Motivation.

Humans can robustly recognize an object by itself regardless which other
objects it appears with. Conversely, if a classifier _heavily_ relies on the
information from other classes to determine the classification result, it will
easily produce wrong classification results when a rather rare class
combination appears during inference. Maximizing inter-class separation, or in
another words, reducing the inter-class dependency, can therefore help the
network generalize better, especially when the training set is small or is
biased. As shown in Fig. 1, the dog and sheep are mis-classified as the cow
because cow and grass appear together more often in the training set. To
improve the robustness of the model, we propose to reduce this inter-class
dependency. To this end, the following two loss functions are defined.

####  3.3.2 Inter-class Center-to-center Loss.

The first loss function is to maximize the distance between any two different
class centers. Inspired by the center loss used in face recognition [27], we
propose to reduce the similarity between class centers $\bm{\mu}$, which are
the averaged features of each class calculated according to the GT mask. The
_inter-class relation_ is defined by the dot-product similarity [23] between
any two classes as:

$\footnotesize\mathbf{A}_{\text{c2c}}=\text{softmax}(\frac{\bm{\mu}^{T}\cdot\bm{\mu}}{\sqrt{C}}),\>\>\
\mathbf{A}_{\text{c2c}}\in\mathbb{R}^{N_{class}\times N_{class}}.$ (4)

Moreover, since we only need to constrain the inter-class distance, only the
non-diagonal elements are retained for the later loss calculation as:

$\mathbf{D}_{\text{inter-c2c}}=\Big{(}1-eye(N_{class})\Big{)}\mathbf{A}_{\text{c2c}}.$
(5)

We only penalize larger similarity values between any two different classes
than a pre-defined threshold $\frac{\epsilon_{0}}{N_{class}-1}$, i.e.,

$\mathcal{D}_{\text{inter-c2c}}=f_{\text{sum}}\Big{(}\text{max}(\mathbf{D}_{\text{inter-c2c}}-\frac{\epsilon_{0}}{N_{class}-1},0)\Big{)}.$
(6)

Thus, the Inter-class Center-to-center Loss $\mathcal{L}_{\text{inter-c2c}}$
is defined by:

$\mathcal{L}_{\text{inter-c2c}}=f_{\text{mse}}(\mathcal{D}_{\text{inter-c2c}}).$
(7)

Here, a small margin is used in consideration of the feature space size and
the mislabeled ground truth.

####  3.3.3 Inter-class Center-to-pixel Loss.

Maximizing only the distances between class centers does not necessarily
result in separable representation for every individual pixels. We further
maximize the distance between a class center and any pixel that does not
belong to this class. More concretely, we first compute the center-to-pixel
dot product as:

$\mathbf{\Lambda}_{\text{c2p}}=\bm{\mu}^{T}\cdot\mathbf{X_{\text{flat}}},\>\>\
\mathbf{\Lambda}_{\text{c2p}}\in\mathbb{R}^{HW\times N_{\text{class}}}.$ (8)

Ideally, with the previous loss $\mathcal{L}_{\text{inter-c2c}}$, the features
of all pixels belonging to the same class should be equal to that of the class
center. Therefore, we replace the intra-class dot product with its ideal
value, namely using the class center $\bm{\mu}$ for calculating the intra-
class dot product as:

$\mathbf{\Lambda}_{c}=diag(\bm{\mu}^{T}\cdot\bm{\mu}),$ (9)

and the replacement effect is achieved by using masks as:

$\mathbf{\Lambda^{\prime}}=\mathbf{\Lambda}_{\text{c2p}}(1-\mathbf{Y}_{\text{flat}})+\mathbf{\Lambda}_{c}.$
(10)

This updated dot product $\mathbf{\Lambda^{\prime}}$ is then used to calculate
similarity across class axis with a softmax as:

$\mathbf{A}_{\text{c2p}}=\text{softmax}(\mathbf{\Lambda^{\prime}}),\>\>\
\mathbf{A}_{\text{c2p}}\in\mathbb{R}^{HW\times N_{\text{class}}}.$ (11)

Similar to the calculation of $\mathcal{L}_{\text{inter-c2c}}$ in the previous
subsection, we have

$\mathbf{D}_{\text{inter-c2p}}=(1-\mathbf{Y}_{\text{flat}})\mathbf{A}_{\text{c2p}},$
(12)
$\mathcal{D}_{\text{inter-c2p}}=f_{\text{sum}}\Big{(}\text{max}(\mathbf{D}_{\text{inter-c2p}}-\frac{\epsilon_{1}}{N_{\text{class}}-1},0)\Big{)}.$
(13)

Thus, the Inter-class Center-to-pixel Loss $\mathcal{L}_{\text{inter-c2p}}$ is
defined by:

$\mathcal{L}_{\text{inter-c2p}}=f_{\text{mse}}(\mathcal{D}_{\text{inter-c2p}}).$
(14)

###  3.4 Differences with OCR, ACFNet, CPNet, and CIPC

Methods that are closely related to ours are OCR [31], ACFNet [33] and CPNet
[30], which all focus on better utilizing class-level features and differ on
how to extract the class centers and context features. However, they all use a
simple concatenation to fuse the original pixel feature and the complementary
context feature. For example, OCR and ACFNet first produce a coarse
segmentation, which is supervised by the GT mask with a categorical cross-
entropy loss, and then use this predicted coarse mask to generate the (soft)
class centers by weighted summing all the pixel features. OCR then aggregates
these class centers according to their similarity to the original pixel
feature termed as “pixel-region relation”, resulting in a “contextual
feature”. Slightly differently from OCR, ACFNet directly uses the probability
(from the predicted coarse mask) to aggregate class center, obtaining a
similar context feature termed as “attentional class feature”. CPNet defines
an affinity map, which is a binary map indicating if two spatial locations
belong to the same class. Then, they use a sub-network to predict their ideal
affinity map and use the soft version affinity map termed as “Context Prior
Map” for feature aggregation, obtaining a class feature (center) and a context
feature. Note that CPNet concatenates class feature, which is the updated
pixel feature, and the context feature.

We also propose to utilize class-level contextual features. Instead of
extracting and fusing pixel features with sub-networks, we propose three loss
functions to directly regularize training and encourage the learned features
to maintain certain desired properties. The approach is simple but more
effective thanks to the direct supervision (validated in Tab. 2). Moreover,
our class center estimate is more accurate because we use the GT mask. This
strategy largely reduces the complexity of the network and introduces no
computational overhead during inference. Furthermore, it is compatible with
all existing methods, including OCR, ACFNet and CPNet, demonstrating great
generalization capability.

We also notice that Cross-Image Pixel Contrast (CIPC) [25] shares a similar
high-level goal as our CAR, i.e., learning more similar representations for
pixels belonging to the same class than to a different class. However, the
approaches of achieving this goal are very different. CIPC is motivated by
contrastive learning while our CAR is motivated by the compositionality of the
scene, for better generalization in the cases of rare class combinations.
Therefore, CIPC adopts the _one-vs-rest_ style InfoNCE loss, including the
typical pixel-to-pixel loss and a special pixel-to-center loss. In contrast,
(1) we propose an additional _center-to-center_ loss to regularize the inter-
class dependency explicitly and effectively (see Table 1); (2) we use _one-vs-
one_ style inter-class losses while CIPC uses _one-vs-rest_ style NCE loss.
Compared to our _one-vs-one_ loss, using _one-vs-rest_ loss for training does
not necessarily result in small inter-class similarity between the current
class and every individual “other” classes and may increase the inter-class
similarity among those “other” classes. (3) we leave margins to prevent CAR
regularizations, which is not the primary task of pixel classification, from
dominating the learning process.

##  4 Experiments

###  4.1 Implementation

##### Training Settings.

For both CAR and baselines, we apply the settings common to most works [34,
35, 11, 10, 37], including SyncBatchNorm, batch size = 16, weight decay
(0.001), 0.01 initial LR, and poly learning decay with SGD during training. In
addition, for the CNN backbones (e.g., ResNet), we set output stride = 8 (see
[3]). Training iteration is set to 30k iterations unless otherwise specified.
For the thresholds in Eq. 6 and Eq. 13, we set $\epsilon_{0}=0.5$ and
$\epsilon_{1}=0.25$.

##### Determinism and Reproducibility

Our implementations are based on the latest NVIDIA deterministic framework
(2022), which means exactly the same results can be always reproduced with the
same hardware and same training settings (including random seed). To
demonstrate the effectiveness of our CAR with equal comparisons, we reproduced
all the baselines that we compare, all conducted with exactly the same
settings unless otherwise specified.

###  4.2 Experiments on Pascal Context

The Pascal Context [19] dataset is split into 4,998/5,105 for training/test
set. We use its 59 semantic classes following the common practice [31, 35].
Unless otherwise specified, both baselines and CAR are trained on the training
set with 30k iterations. The ablation studies are presented in Sect. 4.2.1.

####  4.2.1 Ablation Studies on Pascal Context

Table 1: Ablation studies of adding CAR to different methods on Pascal Context dataset. All results are obtained with single scale test without flip. “A” means replacing the $3\times 3$ conv with $1\times 1$ conv (detailed in Sec. 4.2.1). CAR improves the performance of different types of backbones (CNN & Transformer) and head blocks (SA & Uper), showing the generalizability of the proposed CAR.  | Methods | $\mathcal{L}_{\text{intra-c2p}}$ | $\mathcal{L}_{\text{inter-c2c}}$ | $\mathcal{L}_{\text{inter-c2p}}$ |  A | mIOU (%)
---|---|---|---|---|---|---
R1 |  ResNet-50 + Self-Attention [26] | - | - |  |  | 48.32
R2 |  |  |  |  | ✓ | 48.56
R3 | \+ CAR | ✓ |  |  |  | 49.17
R4 |  | ✓ | ✓ |  |  | 49.79
R5 |  | ✓ | ✓ | ✓ |  | 50.01
R6 |  | ✓ |  |  | ✓ | 49.62
R7 |  | ✓ | ✓ |  | ✓ | 50.00
R8 |  | ✓ | ✓ | ✓ | ✓ | 50.50
S1 |  Swin-Tiny + UperNet [29] | - | - |  |  | 49.62
S2 |  |  |  |  | ✓ | 49.82
S3 | \+ CAR | ✓ |  |  |  | 49.01
S4 |  | ✓ | ✓ |  |  | 50.63
S5 |  | ✓ | ✓ | ✓ |  | 50.26
S6 |  | ✓ |  |  | ✓ | 49.62
S7 |  | ✓ | ✓ |  | ✓ | 50.58
S8 |  | ✓ | ✓ | ✓ | ✓ | 50.78

##### CAR on ResNet-50 + Self-Attention.

We firstly test the CAR with “ResNet-50 + Self-Attention” (w/o image-level
block in [35]) to verify the effectiveness of the proposed loss functions,
i.e., $\mathcal{L}_{\text{intra-c2p}}$, $\mathcal{L}_{\text{inter-c2c}}$, and
$\mathcal{L}_{\text{inter-c2p}}$.

As shown in Tab. 1, using $\mathcal{L}_{\text{intra-c2p}}$ directly improves
1.30 mIOU (48.32 vs 49.62); Introducing $\mathcal{L}_{\text{inter-c2c}}$ and
$\mathcal{L}_{\text{inter-c2p}}$ further improves 0.38 mIOU and 0.50 mIOU;
Finally, with all three loss functions, the proposed CAR improves 2.18 mIOU
from the regular ResNet-50 + Self-attention (48.32 vs 50.50).

##### CAR on Swin-Tiny + Uper.

“Swin-Tiny + Uper” is a totally different architecture from “ResNet-50 + Self-
Attention [26]”. Swin [14] is a recent Transformer-based backbone network.
Uper [29] is based on the pyramid pooling modules (PPM) [36] and FPN [12],
focusing on extracting multi-scale context information. Similarly, as shown in
Tab. 1, after adding CAR, the performance of Swin-Tiny + Uper also increases
by 1.16, which shows our CAR can generalize to different architectures well.

##### The Devil is In the Architecture’s Detail.

We find it important to replace the leading $3\times 3$ conv (in the original
method) with $1\times 1$ conv (Fig. 3B). For example,
$\mathcal{L}_{\text{intra-c2p}}$ and $\mathcal{L}_{\text{inter-c2p}}$ did not
improve the performance in Swin-Tiny + Uper (Row S3 vs S1, and S5 vs S4 in
Tab. 1). A possible reason is that the network is trained to maximize the
separation between different classes. However, if the two pixels lie on
different sides of the segmentation boundary, a $3\times 3$ conv will merge
the pixel representations from different classes, making the proposed losses
harder to optimize.

To keep simplicity and maximize generalization, we use the same network
configurations in our all experiments. However, performance may be further
improved with some minor dedicated modifications for each baseline when
deploying our CAR. For example, decreasing the filter number to 256 for the
last conv layer of ResNet-50 + Self-Attention + CAR results in a further
improvement to 51.00 mIOU (from 50.50). Replacing the conv layer after PPM
(inside Uper block, A3 in Fig. 3) from $3\times 3$ to $1\times 1$ in Swin-Tiny
+ UperNet boosts Swin (tiny & large) + UperNet + CAR by an extra 0.5-1.0 mIOU.
We intentionally did _not_ exhaustively search these variants and _not_ report
these results in any table since they did not generalize.

Table 2: Ablation studies of adding CAR to different baselines on Pascal Context [19] and COCOStuff-10K [1]. We deterministically reproduced all the baselines with the same settings. All results are single-scale without flipping. CAR works very well in most existing methods. § means reducing the class-level threshold to 0.25 from 0.5. We found it is sensitive for some model variants to handle a large number of class. Affinity loss [30] and Auxiliary loss [36] are applied on CPNet and OCR, respectively, since they highly rely on those losses.  Methods | Backbone | Reference | mIOU(%)
---|---|---|---
|  |  | Pascal Context | COCO-Stuff10K
FCN [16] |  ResNet-50 [8] | CVPR2015 | 47.72 | 34.10
FCN + CAR |  ResNet-50 [8] |  |  48.40(+0.68) |  34.91(+0.81)§
FCN [16] |  ResNet-101 [8] | CVPR2015 | 50.93 | 35.93
FCN + CAR |  ResNet-101 [8] |  |  51.39(+0.49) |  36.88(+0.95)§
DeepLabV3 [4] |  ResNet-50 [8] | ECCV2018 | 48.59 | 34.96
DeepLabV3 + CAR |  ResNet-50 [8] |  |  49.53(+0.94) |  35.13(+0.17)
DeepLabV3 [4] |  ResNet-101 [8] | ECCV2018 | 51.69 | 36.92
DeepLabV3 + CAR |  ResNet-101 [8] |  |  52.58(+0.89) |  37.39(+0.47)
Self-Attention [26] |  ResNet-50 [8] | CVPR2018 | 48.32 | 34.35
Self-Attention + CAR |  ResNet-50 [8] |  |  50.50(+2.18) |  36.58(+2.23)§
Self-Attention [26] |  ResNet-101 [8] | CVPR2018 | 51.59 | 36.53
Self-Attention + CAR |  ResNet-101 [8] |  |  52.49(+0.9) |  38.15(+1.62)
CCNet [10] |  ResNet-50 [8] | ICCV2019 | 49.15 | 35.10
CCNet + CAR |  ResNet-50 [8] |  |  49.56(+0.41) |  36.39(+1.29)
CCNet [10] |  ResNet-101 [8] | ICCV2019 | 51.41 | 36.88
CCNet + CAR |  ResNet-101 [8] |  |  51.97(+0.56) |  37.56(+0.68)
DANet [7] |  ResNet-101 [8] | CVPR2019 | 51.45 | 35.80
DANet + CAR |  ResNet-101 [8] |  |  52.57(+1.12) |  37.47(+1.67)
CPNet [30] |  ResNet-101 [8] | CVPR2020 | 51.29 | 36.92
CPNet + CAR |  ResNet-101 [8] |  |  51.98(+0.69) |  37.12(+0.20)§
OCR [31] |  HRNet-W48 [24] | ECCV2020 | 54.37 | 38.22
OCR + CAR |  HRNet-W48 [24] |  |  54.99(+0.62) |  39.53(+1.31)
UperNet [29] |  Swin-Tiny [14] | ICCV2021 | 49.62 | 36.07
UperNet + CAR |  Swin-Tiny [14] |  |  50.78(+1.16) |  36.63(+0.56)§
UperNet [29] |  Swin-Large [14] | ICCV2021 | 57.48 | 44.25
UperNet + CAR |  Swin-Large [14] |  |  58.97(+1.49) |  44.88(+0.63)
CAA [9] |  EfficientNet-B5 [18] | AAAI2022 | 57.79 | 43.40
CAA + CAR |  EfficientNet-B5 [18] |  |  58.96(+1.17) |  43.93(+0.53)

##### CAR on Different Baselines.

After we have verified the effectiveness of each part of the proposed CAR, we
then tested CAR on multiple well-known baselines. All of the baselines were
reproduced under similar conditions (see Sect. 4.1). Experimental results
shown in Tab. 2 demonstrate the generalizability of our CAR on different
backbones and methods.

##### Visualization of Class Dependency Maps.

In Fig. 4, we present the class dependency maps calculated on the complete
Pascal Context _test_ set, where every pixel stores the dot-product
similarities between every two class centers. The maps indicate the inter-
class dependency obtained with the standard ResNet-50 + Self-Attention and
Swin-Tiny + UperNet, and the effect of applying our CAR. A hotter color means
that the class has higher dependency on the corresponding class, and vice
versa. According to Fig. 4 a1-a2, we can easily observe that the inter-class
dependency has been significantly reduced with CAR on ResNet50 + Self-
Attention. Fig. 4 b1-b2 show a similar trend when tested with different
backbones and head blocks. This partially explains the reason why baselines
with CAR generalize better on rarely seen class combinations (Figs. 1 and 5).
Interestingly, we find that the class-dependency issue is more serious in
Swin-Tiny + Uper, but our CAR can still reduce its dependency level
significantly.

Figure 4: Class dependency maps generated on Pascal Context test set. One may
zoom in to see class names. A hotter color means that the class has higher
dependency to the corresponding class, and vice versa. It is obvious that our
CAR reduces the inter-class dependency, thus providing better generalizability
(see Figs. 1 and 5).

##### Visualization of Pixel-relation Maps.

In Fig. 5, we visualize the pixel-to-pixel relation energy map, based on the
dot-product similarity between a red-dot marked pixel and other pixels, as
well as the predicted results for different methods, for comparison. Examples
are from Pascal Context test set. As we can see, with CAR supervision, the
existing models focus better on objects themselves rather than other objects.
Therefore, this reduces the possibility of the classification errors because
of the class-dependency bias.

(a) ResNet50 + Self-Attention

(b) Swin-Tiny + UperNet

Figure 5:  Visualization of the feature similarity between a given pixel
(marked with a red dot in the image) and all pixels, as well as the
segmentation results on Pascal Context test set. A hotter color denotes larger
similarity value. Apparently, our CAR reduces the inter-class dependency and
exhibits better generalization ability, where energies are better restrained
in the intra-class pixels.

###  4.3 Experiments on COCOStuff-10K

COCOStuff-10K dataset [1] is widely used for evaluating the robustness of
semantic segmentation models [11, 31]. The COCOStuff-10k dataset is a very
challenging dataset containing 171 labeled classes and 9000/1000 images for
training/test. As shown in Tab. 2, all of the tested baselines gain
performance boost ranging from 0.17% to 2.23% with our proposed CAR. This
demonstrates the generalization ability of our CAR when handling a large
number of classes.

##  5 Conclusions and Future Work

In this paper, we have aimed to make a better use of class level context
information. We have proposed a universal class-aware regularizations (CAR)
approach to regularize the training process and boost the differentiability of
the learned pixel representations. To this end, we have proposed to minimize
the intra-class feature variance and maximize the inter-class separation
simultaneously. Experiments conducted on benchmark datasets with extensive
ablation studies have validated the effectiveness of the proposed CAR
approach, which has boosted the existing models’ performance by up to 2.18%
mIOU on Pascal Context and 2.23% on COCOStuff-10k with no extra inference
overhead.

Acknowledgement This research depends on the NVIDIA determinism framework. We
appreciate the support from @duncanriach and @reedwm at NVIDIA and TensorFlow
team.

## References

  * [1] Caesar, H., Uijlings, J., Ferrari, V.: COCO-Stuff: Thing and Stuff Classes in Context. In: IEEE Conference on Computer Vision and Pattern Recognition (2018)
  * [2] Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L.: Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Transactions on Pattern Analysis and Machine Intelligence (2017)
  * [3] Chen, L.C., Papandreou, G., Schroff, F., Adam, H.: Rethinking atrous convolution for semantic image segmentation (2017)
  * [4] Chen, L.C., Zhu, Y., Papandreou, G., Schroff, F., Adam, H.: Encoder-decoder with atrous separable convolution for semantic image segmentation. In: European Conference on Computer Vision (2018)
  * [5] Choi, S., Kim, J.T., Choo, J.: Cars can’t fly up in the sky: Improving urban-scene segmentation via height-driven attention networks. In: IEEE Conference on Computer Vision and Pattern Recognition (2020)
  * [6] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16x16 words: Transformers for image recognition at scale. In: International Conference on Learning Representations (2021)
  * [7] Fu, J., Liu, J., Tian, H., Li, Y., Bao, Y., Fang, Z., Lu, H.: Dual attention network for scene segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition (2019)
  * [8] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: IEEE Conference on Computer Vision and Pattern Recognition (2016)
  * [9] Huang, Y., Kang, D., Jia, W., He, X., liu, L.: Channelized axial attention - considering channel relation within spatial attention for semantic segmentation. In: AAAI (2022)
  * [10] Huang, Z., Wang, X., Wei, Y., Huang, L., Shi, H., Liu, W., Huang, T.S.: Ccnet: Criss-cross attention for semantic segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence (2020)
  * [11] Li, X., Zhong, Z., Wu, J., Yang, Y., Lin, Z., Liu, H.: Expectation-maximization attention networks for semantic segmentation. In: International Conference on Computer Vision (2019)
  * [12] Lin, T.Y., Dollá, P., Girshick, R., He, K., Hariharan, B., Belongie, S.: Feature pyramid networks for object detection. In: IEEE Conference on Computer Vision and Pattern Recognition (2017)
  * [13] Liu, M., Schonfeld, D., Tang, W.: Exploit visual dependency relations for semantic segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition (2021)
  * [14] Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B.: Swin transformer: Hierarchical vision transformer using shifted windows. In: ICCV (2021)
  * [15] Liu, Z., Mao, H., Wu, C.Y., Feichtenhofer, C., Darrell, T., Xie, S.: A convnet for the 2020s. In: IEEE Conference on Computer Vision and Pattern Recognition (2022)
  * [16] Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition (2015)
  * [17] Marius, C., Mohamed, O., Sebastian, R., Timo, R., Markus, E., Rodrigo, B., Uwe, F., Roth, S., Bernt, S.: The cityscapes dataset for semantic urban scene understanding. In: IEEE Conference on Computer Vision and Pattern Recognition (2016)
  * [18] Mingxing, T., Quoc, L.: Efficientnet: Rethinking model scaling for convolutional neural networks. In: International Conference on Machine Learning (2019)
  * [19] Mottaghi, R., Chen, X., Liu, X., Cho, N.G., Lee, S.W., Fidler, S., Urtasun, R., Yuille, A.: The role of context for object detection and semantic segmentation in the wild. In: IEEE Conference on Computer Vision and Pattern Recognition (2014)
  * [20] Ranftl, R., Bochkovskiy, A., Koltun, V.: Vision transformers for dense prediction. In: ICCV (2021)
  * [21] Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: International Conference on Learning Representations (2015)
  * [22] Sixiao, Z., Jiachen, L., Hengshuang, Z., Xiatian, Z., Zekun, L., Yabiao, W., Yanwei, F., Jianfeng, F., Tao, X., H.S., T.P., Li, Z.: Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. In: IEEE Conference on Computer Vision and Pattern Recognition (2021)
  * [23] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Łukasz Kaiser, Polosukhin, I.: Attention is all you need. In: Conference on Neural Information Processing Systems (2017)
  * [24] Wang, J., Sun, K., Cheng, T., Jiang, B., Deng, C., Zhao, Y., Liu, D., Mu, Y., Tan, M., Wang, X., Liu, W., Xiao, B.: Deep high-resolution representation learning for visual recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence (2020)
  * [25] Wang, W., Zhou, T., Yu, F., Dai, J., Konukoglu, E., Van Gool, L.: Exploring cross-image pixel contrast for semantic segmentation. In: ICCV. pp. 7303–7313 (2021)
  * [26] Wang, X., Girshick, R., Gupta, A., He, K.: Non-local neural networks. In: IEEE Conference on Computer Vision and Pattern Recognition (2018)
  * [27] Wen1, Y., Zhang, K., Li, Z., Qiao, Y.: Discriminative feature learning approach for deep face recognition. In: European Conference on Computer Vision (2016)
  * [28] Wu, H., Zhang, J., Huang, K., Liang, K., Yizhou, Y.: Fastfcn: Rethinking dilated convolution in the backbone for semantic segmentation (2019)
  * [29] Xiao, T., Liu, Y., Zhou, B., Jiang, Y., Sun, J.: Unified perceptual parsing for scene understanding. In: European Conference on Computer Vision (2018)
  * [30] Yu, C., Wang, J., Gao, C., Yu, G., Shen, C., Sang, N.: Context prior for scene segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition (2020)
  * [31] Yuan, Y., Chen, X., Wang, J.: Object-contextual representations for semantic segmentation. In: European Conference on Computer Vision (2020)
  * [32] Yuan, Y., Huang, L., Guo, J., Zhang, C., Chen, X., Wang, J.: Ocnet: Object context network for scene parsing. International Journal of Computer Vision (2021)
  * [33] Zhang, F., Chen, Y., Li, Z., Hong, Z., Liu, J., Ma, F., Han, J., Ding, E.: Acfnet: Attentional class feature network for semantic segmentation. In: International Conference on Computer Vision (2019)
  * [34] Zhang, H., Dana, K., Shi, J., Zhang, Z., Wang, X., Tyagi, A., Agrawal, A.: Context encoding for semantic segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition (2018)
  * [35] Zhang, H., Zhan, H., Wang, C., Xie, J.: Semantic correlation promoted shape-variant context for segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition (2019)
  * [36] Zhao, H., Shi, J., Qi, X., Wang, X., Jia, J.: Pyramid scene parsing network. In: IEEE Conference on Computer Vision and Pattern Recognition (2017)
  * [37] Zhu, Z., Xu, M., Bai, S., Huang, T., Bai, X.: Asymmetric non-local neural networks for semantic segmentation. In: International Conference on Computer Vision (2019)

##  Appendix 0.A Appendix

###  0.A.1 Extra technical details

####  0.A.1.1 Deterministic

Control variables are very important for all scientific research. In computer
vision, we always use the same backbones and the same datasets when verifying
the difference between two methods.

Without using “deterministic” technology, all operations in neural networks
contain some randomness. Nowadays, with the latest deterministic technology
and fixed seeds, experiments can be conducted in a fully-controlled
environment. This means that the performance difference between different
settings (i.e., w/ and w/o CAR) is not affected by this randomness any more
but faithfully reflects the effectiveness of different methods.

In Tab. 3, we report the performance of our proposed CAR (ResNet-50 + Self-
attention and Swin-Tiny + UperNet) with different seeds for readers who are
interested in how our CAR performs when trained with different random seeds.
As it is shown, our CAR consistently improves the mIOU over its baseline using
different random seeds, demonstrating the effectiveness of the CAR.

Table 3: Ablation studies of our proposed CAR using different random seeds on the Pascal Context dataset. Methods | Seed (mIOU%)
---|---
| 0 (default) | 1 | 2
ResNet-50 + Self-Attention | 48.32 | 47.54 | 47.69
ResNet-50 + Self-Attention + CAR |  50.50(+2.18) |  50.20(+2.66) |  50.59(+2.90)
Swin-Tiny + UperNet | 49.62 | 49.24 | 49.54
Swin-Tiny + UperNet + CAR |  50.78(+1.16) |  50.57(+1.33) |  50.75(+1.21)

###  0.A.2 Extra experiments

####  0.A.2.1 Ablation studies on batch class center

In our experiments, we calculated the class centers using all the training
images in a _batch_ to alleviate the negative impact of noisy images. Here, we
investigate the impact of using the class center of each individual image for
class-aware regularizations.

Table 4: Comparison of mIOUs (%) obtained when using batch class center vs image class center in CAR. Methods |  Baseline | CAR
---|---|---
|  | Image Class Center | Batch Class Center
ResNet-50 + Self-Attention | 48.32 | 49.78 | 50.50
Swin-Tiny + UperNet | 49.62 | 49.45 | 50.78

####  0.A.2.2 CAR using Moving Average.

We also implemented a moving average version of CAR which tracks the class
center $\bm{\mu}$ with moving average similar to BatchNorm. As shown in Tab.
5, we find this moving average version of CAR negatively impacts both
ResNet-50 + Self-Attention and Swin-Tiny + Uper.

Table 5: Ablation studies of adding moving average to CAR on Pascal Context. Decay rate stands for the effect of old class center. Methods |  CAR | CAR (Moving Average)
---|---|---
|  | 0.8 | 0.9 | 0.99
ResNet-50 + Self-Attention | 50.50 |  49.80($-$0.70) |  50.26($-$0.24) |  49.96($-$0.54)
Swin-Tiny + UperNet | 50.78 |  49.56($-$1.22) |  50.03($-$0.75) |  48.93($-$1.85)

####  0.A.2.3 CAR without intra-class loss.

Table 6 below shows that, including only inter-c2c loss and inter-c2p still
improves the result.

Table 6: Extra ablation studies for CAR without intra-class loss. | $\mathcal{L}_{\text{intra-c2p}}$ | $\mathcal{L}_{\text{inter-c2c}}$ | $\mathcal{L}_{\text{inter-c2p}}$ | A | mIOU
---|---|---|---|---|---
ResNet-50 + Self-Attention | - | - |  |  | 48.32
|  |  |  | ✓ | 48.56
\+ CAR |  | ✓ | ✓ |  | 49.31
|  | ✓ | ✓ | ✓ | 50.23

####  0.A.2.4 Exceeding state-of-the-art (SOTA) in Pascal Context

The main motivation of our CAR is to utilize class-level information as
regularizations during training to boost the performance of all existing
methods. However, following the convention and also for readers who are
interested, we compare with state-of-the-art methods in Tab. 7 regardless
their architectures are related to ours or not. Since Swin [14] is not
compatible with dilation, we use JPU [28] as the substitution to obtain
features with output stride = 8. Uper contains an FPN [12] module that can
obtain features with output stride = 4.

Boosted by our CAR, the strong model ConvNext-Large [15] \+ CAA [9] achieved
the performance of 62.70% mIOU under single-scale testing, and 63.91% under
multi-scale testing. Also, we found increasing the training iterations from
the default 30K to 40K when using Adam optimizer can further increase
performance in Pascal Context dataset. Thus, the SOTA single model performance
has now been boosted to 62.97% under single-scale testing, and 64.12% under
multi-scale testing. This has outperformed the previous SOTA single model,
i.e., EfficientNetB7 + CAA, by a large margin.

Table 7: Experiments on boosting the SOTA single-model performance on Pascal Context by our CAR. See Sec. 0.A.2.4 for the details. §: We report previous SOTA score as reference. SS: mIOU on Single scale without flipping. MF: Multi-scale with flipping. JPU is used to get features with output stride = 8. Aux: Apply auxiliary loss during training, see [36]. Iters: training iterations. WP:Linear learning rate warmup  Methods | Backbone | Aux | Optimizer | Iters | WP | SS(%) | MF(%)
---|---|---|---|---|---|---|---
CAA§ | EfficientNet-B7-D8 | ✓ | SGD | 30K |  | - | 60.30
UperNet | Swin-Large |  | SGD | 30K |  | 57.48 | 59.45
UperNet + CAR | Swin-Large |  | SGD | 30K |  | 58.97 | 60.76
CAA | Swin-Large + JPU |  | SGD | 30K |  | 58.31 | 59.75
CAA + CAR | Swin-Large + JPU |  | SGD | 30K |  | 59.84 | 61.46
CAA + CAR | Swin-Large + JPU |  | Adam | 30K |  | 60.68 | 62.21
CAA | ConvNeXt-Large + JPU |  | SGD | 30K |  | 60.48 | 61.80
CAA + CAR | ConvNeXt-Large + JPU |  | SGD | 30K |  | 61.40 | 62.69
CAA + CAR | ConvNeXt-Large + JPU |  | Adam | 30K |  | 62.65 | 63.77
CAA + CAR | ConvNeXt-Large + JPU | ✓ | Adam | 30K |  | 62.70 | 63.91
CAA + CAR | ConvNeXt-Large + JPU | ✓ | Adam | 40K |  | 62.97 |  64.12
CAA + CAR | ConvNeXt-Large + JPU | ✓ | Adam | 40K | ✓ | 63.13 | 64.17

####  0.A.2.5 Exceeding SOTA performance in COCOStuff-10K

Simliar to Sec. 0.A.2.4, in Tab. 8, boosted by our CAR, the strong model
ConvNext-Large [15] \+ CAA achieved the performance of 49.03% mIOU under
single-scale testing, and 50.01% under multi-scale testing. This has also
outperformed the previous SOTA single model, i.e., EfficientNetB7 + CAA, by a
large margin.

Table 8: Experiments on boosting SOTA on COCOStuff10k, levering the previous single model SOTA and boosted by our CAR. See Sec. 0.A.2.5 for details. §: We report the original SOTA scores. SS: Single scale without flipping. MF: Multi-scale with flipping. Aux Apply auxiliary loss during training, see [36].  Methods | Backbone | Aux | Optimizer | SS mIOU(%) | MF mIOU(%)
---|---|---|---|---|---
CAA§ | EfficientNet-B7-D8 | ✓ | SGD | - | 45.40
UperNet | Swin-Large |  | SGD | 44.25 | 46.10
UperNet + CAR | Swin-Large |  | SGD | 44.88 | 46.64
CAA | Swin-Large + JPU |  | SGD | 44.22 | 45.31
CAA + CAR | Swin-Large + JPU |  | SGD | 45.48 | 46.99
CAA | ConvNeXt-Large + JPU |  | SGD | 46.49 | 47.23
CAA + CAR | ConvNeXt-Large + JPU |  | SGD | 46.70 | 47.77
CAA + CAR | ConvNeXt-Large + JPU |  | Adam | 48.20 | 48.83
CAA + CAR | ConvNeXt-Large + JPU | ✓ | Adam | 49.03 | 50.01

###  0.A.3 Extra Visualizations

####  0.A.3.1 Visualization of OCRNet in Pascal Context

Similar to the main paper, in Fig. 6, we visualize the pixel-to-pixel relation
energy maps obtained with HRNetW48 [31] \+ OCR [31]. This figure shows that
our CAR can further improve the robustness of class center based models by
making better use of the class center. Interestingly, as shown in C12 of Fig.
6 and Fig. 1 shown in our main paper what is predicted by ResNet-50 + Self-
Attention, we find cow/sheep/dog misclassification is a common issue in many
semantic segmentation models, especially when i.e. grass and cow co-exist
frequently during training. This issue is better addressed by our CAR due to
its reduced inter-class dependency.

Figure 6: Visualization of the feature similarity between a given pixel
(marked with a red dot in the image) and all other pixels, as well as the
segmentation results of HRNetW48 [24] \+ OCR [31] on Pascal Context test set.
A hotter color denotes a greater similarity value.

####  0.A.3.2 Visualization of DeepLab in Pascal Context

We also visualize the pixel-to-pixel relation energy map of ResNet-50 [8] \+
DeepLabV3 [3] in Fig. 7. These visualizations clearly show that the reduced
inter-class dependency helps to correct the classification.

Figure 7: Visualization of the feature similarity between a given pixel
(marked with a red dot in the image) and all pixels, as well as the
segmentation results of  ResNet-50 [8] \+ DeepLab [3] on Pascal Context test
set. A hotter color denotes a greater similarity value.


# CLONeR: Camera-Lidar Fusion for Occupancy Grid-aided Neural Representations

Alexandra Carlson∗,1, Manikandasriram S. Ramanagopal∗,2,
Nathan Tseng1, Matthew Johnson-Roberson3, Ram Vasudevan2, and Katherine A.
Skinner2 ∗denotes equal contributionManuscript received: January 04, 2023;
Accepted March 08, 2023. This paper was recommended for publication by Editor
Cesar Cadena upon evaluation of the Associate Editor and Reviewers’ comments.
This work was supported by a grant from Ford Motor Company via the Ford-UM
Alliance under award N028603.1A. Carlson and N. Tseng are with Ford Motor
Company, but completed this work during graduate studies at the Department of
Robotics, University of Michigan, Ann Arbor, MI, USA
<EMAIL_ADDRESS>Ramanagopal, R. Vasudevan, and K. Skinner are with
the Department of Robotics, University of Michigan, Ann Arbor, MI, USA
<EMAIL_ADDRESS>Johnson-Roberson is with the Robotics
Institute, Carnegie Mellon University, Pittsburgh, PA, USA
<EMAIL_ADDRESS>https://fcav.engin.umich.edu/clonerDigital Object
Identifier (DOI): 10.1109/LRA.2023.3262139.

###### Abstract

Recent advances in neural radiance fields (NeRFs) achieve state-of-the-art
novel view synthesis and facilitate dense estimation of scene properties.
However, NeRFs often fail for outdoor, unbounded scenes that are captured
under very sparse views with the scene content concentrated far away from the
camera, as is typical for field robotics applications. In particular, NeRF-
style algorithms perform poorly: (1) when there are insufficient views with
little pose diversity, (2) when scenes contain saturation and shadows, and (3)
when finely sampling large unbounded scenes with fine structures becomes
computationally intensive. This paper proposes CLONeR, which significantly
improves upon NeRF by allowing it to model large unbounded outdoor driving
scenes that are observed from sparse input sensor views. This is achieved by
decoupling occupancy and color learning within the NeRF framework into
separate Multi-Layer Perceptrons (MLPs) trained using LiDAR and camera data,
respectively. In addition, this paper proposes a novel method to build
differentiable 3D Occupancy Grid Maps (OGM) alongside the NeRF model, and
leverage this occupancy grid for improved sampling of points along a ray for
volumetric rendering in metric space. Through extensive quantitative and
qualitative experiments on scenes from the KITTI dataset, this paper
demonstrates that the proposed method outperforms state-of-the-art NeRF models
on both novel view synthesis and dense depth prediction tasks when trained on
sparse input data.

###### Index Terms:

Deep Learning for Visual Perception, Sensor Fusion, Computer Vision for
Transportation

##  I Introduction

The estimation of dense scene properties from sparse and low diversity sensor
data is a critical task in robot perception. In particular, for autonomous
vehicle perception, estimating dense and accurate depth is essential for scene
understanding used for downstream planning and navigation tasks. Neural
radiance fields (NeRFs) are an attractive solution for producing joint dense
depth and color maps of a scene [1] . NeRFs use implicit functions to encode
volumetric density and color observations from multiple images with known
poses. Recent advances in NeRFs have demonstrated state-of-the-art performance
for novel view synthesis and 3D modeling.

Still, outdoor scenes captured in autonomous vehicle datasets remain
challenging for NeRFs. These scenes are typically captured from sparse
viewpoints with limited view diversity, i.e., a camera mounted on a vehicle
moving forward. They tend to be large and unbounded, with much of the scene
content concentrated far away from the camera. Lastly, images captured
outdoors feature varying exposure and brightness levels depending upon the
camera settings and weather conditions. These challenges can lead to learning
inaccurate scene geometry, blurry scene renderings and poorly modeled thin
structures, especially at long distances.

Figure 1: An illustration of the sparse inputs and dense outputs from CLONeR.
From a single image and single LiDAR scan as input (shown on the left, under
‘Training Inputs’), CLONeR (middle, represented by the blue arrow) can
generate accurate dense depth, novel views, and an Occupancy grid map (shown
on the right, under ‘Rendered Outputs’).

This paper presents CLONeR, Camera-Lidar fusion for Occupancy grid-aided
Neural Representation, a novel NeRF framework that addresses key challenges to
applying NeRFs to outdoor driving scenes. The proposed method can accurately
clone a 3D scene from a sparse set of image views and LiDAR scans.
Impressively, CLONeR can render novel views and dense depth maps with fine
detail from a single image and LiDAR scan (see Fig. 1). CLONeR accurately
models the operation of both LiDAR and RGB camera sensors within a NeRF
framework to learn a dense 3D representation of a scene. In addition, CLONeR
simultaneously builds a differentiable 3D Occupancy Grid Map (OGM) using LiDAR
data, which it leverages to efficiently sample large outdoor scenes in metric
space. This learned occupancy grid map is capable of maintaining known versus
unknown regions in a probabilistic manner, and replaces the standard coarse
MLP sampling scheme used in NeRFs. Critically, CLONeR decouples occupancy and
color learning within the NeRF by utilizing two separate MLPs. One of these
MLPs learns occupancy from LiDAR ray data, and the other MLP learns the RGB
model of the scene from camera rays. By doing this, sensor artifacts and
illumination effects in the RGB images do not negatively influence the learned
scene depth.

In summary, the key contributions of the proposed work, CLONeR, are:

  * •

A decoupled NeRF model where LiDAR is used to supervise geometry learning
while camera images are used to supervise color learning. This allows one to
perform novel view synthesis using as few as a single image when LiDAR scans
are available.

  * •

A differentiable 3D occupancy grid learned directly on the GPU alongside the
decoupled NeRF model. This 3D occupancy grid explicitly maintains information
about known and unknown regions, and outperforms standard coarse NeRF based
importance sampling methods.

We demonstrate that our method outperforms state-of-the-art NeRF models on
both novel view synthesis and depth prediction tasks through extensive
quantitative and qualitative experiments on several scenes from the KITTI
dataset.

The remainder of this paper is organized as follows: We review related work in
Section II, and give a detailed description of the proposed method, CLONeR, in
Section III. We present extensive quantitative and qualitative results in
Section IV. Section V provides the conclusion and future work.

##  II Related Works

The impressive performance achieved by the original NeRF [1] has inspired many
subsequent works to extend it to various applications including generative
modeling [2] and scene relighting [3]. In this work, we develop a novel NeRF
framework that leverages camera and LiDAR to enable depth-guided learning for
unbounded, outdoor scenes observed from sparse/limited viewpoints.

###  II-A Depth-guided Learning in Neural Radiance Fields

In prior work, NeRF-based models have been used along with depth supervision
from structure-from-motion techniques [4], RGB-D cameras [5, 6, 7] and
continuous-wave Time-of-Flight sensors [8]. However, RGB-D cameras do not
operate well outdoors in the presence of sunlight, and SfM pipelines operate
poorly in low-texture environments, making this depth supervision very
difficult to obtain for outdoor scenes that would be encountered in autonomous
driving applications.

###  II-B Neural radiance fields for outdoor scenes

A significant number of NeRF models have tried to tackle more complex static
3D objects, as well as larger indoor scenes [5, 9, 10, 11, 12]. In particular,
in the surface reconstruction problem space, both NeuS [10] and VolSDF [11]
both separate the color and opacity learning. We extend their intuitions to
volume rendering in this work, and demonstrate that decoupling is key in
removing several failure modes when training NeRFs on outdoor scenes. However,
outdoor scenes still present a unique challenge for NeRFs. Many outdoor scenes
have a relatively large and unbounded. Much of the scene content is
concentrated far away from the camera(s) with a significant amount of empty
space. The scenes are large enough that sampling the 3D scene volume to learn
accurate scene structures becomes computationally challenging [7]. As a
result, NeRFs struggle to learn even coarse scene occupancy. This results in
blurry scene renderings and poorly modeled fine/thin structures, especially at
long distances [13, 14].

There have been many successful methods, including BlockNerF [15] and CityNeRF
[16], that focus on large scale scenes that cover a large spatial extent, such
as a city block or entire city. These methods leverage multiple NeRF models to
learn to model the 3D space [15] and they require a significant number of
images to train. In our proposed method, we instead focus on scenes with a
long depth range from a single camera view, which is specific to the type of
data we would encounter in autonomous vehicle datasets. Additionally, we
consider a single scene from a limited viewpoint and develop our novel
contributions in the OGM and decoupled models for handling LiDAR for these
cases. This differs greatly in the configuration of the scenes in works such
as BlockNeRF, which focuses on combining smaller NeRF models to handle the
large spatial extent of their chosen scenes. There has also been very
promising work in modeling 360∘ unbounded scenes [17, 13, 18, 19], but these
methods require dense multiview datasets which are difficult to capture for
autonomous vehicle datasets.

Very few NeRF models have tried to use LiDAR information to learn scene
occupancy in outdoor scenes. Of particular relevance to our work is Urban
Radiance Fields [19] and Neural Point Light Fields [20]. The method proposed
in Urban Radiance fields utilizes LiDAR data along with images from panoramic
cameras captured in outdoor scenes. Their data collection trajectory
specifically ensures a large number of views are available with diversity in
the camera poses in addition to their LiDAR data. In comparison, the proposed
method is designed to operate on very sparse camera views and LiDAR data [19].

In Neural Point Light Fields, they represent outdoor scenes with a light field
learned on top of a sparse point cloud. They aggregate point features along
each ray to get a final RGB prediction. However, this method learns a feature
vector (via CNN encoding) for each point in the point cloud, and thus requires
long driving trajectories/videos to learn a sufficient scene representation
[20]. In contrast, our proposed method can produce accurate scene models from
as few as a single image paired with a single point cloud. In summary, neither
[19, 20] have demonstrated the ability to learn dense depth using just LiDAR
returns and sparse views.

###  II-C Sampling in Neural Radiance Fields

A key challenge in the original NeRF formulation is the need to finely sample
along a ray to approximate the line integral. Mip-NeRF [21] addressed the
sampling and aliasing challenges in the original NeRF formulation and Mip-NeRF
360 [17] extended it to address unbounded scenes with focus on a central
object. TermiNeRF [9] and DO-NeRF [7] replace the coarse MLP of the original
NeRF model with a separate sampling MLP that estimates where the fine MLP
network should be applied in the 3D world volume. However, in order to train
the aforementioned networks, the input data still needs to cover a large and
diverse set of views.

Recent developments have popularized the use of occupancy grids for sampling
during test-time in NeRFs [22, 23]. Most similar to ours, in [24], the authors
propose to maintain a cascaded set of occupancy grids in order to skip ray
marching on empty space. They present heuristic methods to update the
occupancy grids as training of the NeRF model progresses. In contrast, we
propose to jointly perform occupancy grid mapping while training the NeRF
model. We further distinguish our approach by leveraging LiDAR measurements to
learn and construct these occupancy grids in a probabilistic manner.

##  III Methods

Figure 2: Overview of proposed CLONeR framework. Panel A shows the forward
pass of the framework. In Panel B, LiDAR data is used as the only supervision
for the Sigma MLP during training. In Panel C, camera data is the only
supervision used to train the Color MLP.

Figure 2 provides an overview of the proposed method. Its two key components
are (i) the differentiable occupancy grid map (OGM), which is used to learn
the coarse occupancy of the 3D metric space of the scene, and (ii) the
decoupled NeRF MLPs, which independently learn a fine color and depth model of
the scene. We describe each model component and the training process of the
framework in the following subsections.

###  III-A Overview of Neural Radiance Fields

A Neural Radiance Field (NeRF) [1] models the scene using a 5D vector-valued
function using an MLP network. This MLP takes a 3D location $\vec{x}=(x,y,z)$
and a viewing direction $\vec{d}=(x_{d},y_{d},z_{d})$ as inputs and outputs
the color $\vec{c}=(r,g,b)$ emitted by that scene point and a scalar volume
density $\sigma$.

An image pixel $(u,v)$ is mapped to a camera ray $\vec{r}(t)=\vec{o}+t\vec{d}$
using a simple pinhole camera model. The expected color of the camera ray is
computed using the quadrature approximation of the volumetric rendering
equation between the near and far bounds $t_{n}$ and $t_{f}$ [1]:

$\begin{split}\vec{C}(\vec{r})=\sum_{i=1}^{N}T_{i}(1-\exp(-\sigma_{i}\delta_{i}))\vec{c}_{i},\quad\\\
\mathrm{where}\quad
T_{i}=\exp\big{(}-\sum_{j=1}^{i-1}\sigma_{j}\delta_{j}\big{)},\end{split}$ (1)

and $\delta_{i}=(t_{i+1}-t_{i})$ is the distance between samples along the ray
where the MLP is applied. The weights of the MLP are then optimized by
minimizing the mean squared error between the rendered color
$\hat{\vec{C}}(\vec{r})$ of a pixel and the ground truth color of that pixel.
The termination depth of a ray $\hat{D}(\vec{r})$ is computed as:

$\begin{split}\hat{D}(\vec{r})=\sum_{i=1}^{N}T_{i}(1-\exp(-\sigma_{i}\delta_{i}))t_{i}.\end{split}$
(2)

###  III-B CLONeR World Coordinate System

Given the set of camera and LiDAR poses from an external source such as a GNSS
system, we follow [1] and subtract the average translation and normalize the
rotation such that the local $z-$axis for each camera points into the camera.
The sensor poses and the point clouds from the LiDAR are further scaled by a
common factor such that the region of interest falls completely within
$[-1,1]^{3}$, as required by most types of positional encodings used in NeRFs
[1]. We refer to this scaled, bounded domain as the _world cube_.

For generic outdoor driving scenes, we determine the region of interest in the
following way: for each input camera pose, we use its intrinsic parameters and
user-defined near and far values to compute the 8 corners of its view frustum
in world coordinates. We choose a set of desired camera poses along with their
intrinsics and compute corresponding view frustum points. We then compute a
common scale factor such that all the sensor poses in addition to all the
above points fall within the world cube. We explicitly avoid using the LiDAR
point clouds to define the scale factor as LiDAR data could contain regions
never seen in any of the images, such as points behind the camera.

###  III-C Integrating LiDAR Measurements into NeRF

A 3D scanning LiDAR continuously sweeps the scene using a set of laser beams.
Let $H_{L}$ denote the pose of the LiDAR in world coordinates (as a $3\times
4$ matrix) and let $\vec{x}$ denote a 3D point measured in the local sensor
frame at some time $t$. This corresponds to a ray in the world coordinates
with ray origin $\vec{o}=H_{L}[0,0,0,1]^{T}$ and ray direction
$\vec{d}=H_{L}[\vec{x}^{T}1]^{T}-\vec{o}$. Recall from Sec. III-A that one can
render a camera ray using the NeRF MLP and obtain a termination depth,
expected color, as well as opacity at samples along the ray. Similarly, based
upon how we model the LiDAR ray, we can also render these quantities for each
LiDAR ray cast into the scene.

###  III-D Decoupled NeRF Model

We propose a decoupled NeRF model that contains two separate MLP networks, one
for learning the scene geometry and one for scene color:
$F^{g}_{\Theta}:\mathrm{R}^{3}\rightarrow\mathrm{R}$, which we call the sigma
MLP, and $F^{c}_{\Phi}:\mathrm{R}^{5}\rightarrow\mathrm{R}^{n_{c}}$,which we
call the color MLP. Note that $n_{c}$ is the number of color channels. These
decoupled MLP networks are trained simultaneously such that
$\sigma_{i}=F^{g}_{\Theta}(\vec{r}(t_{i}))$ and
$\vec{c}_{i}=F^{c}_{\Phi}(\vec{r}(t_{i}),\vec{d})$.

These two networks are shown in panels B and C of Fig. 2. Our sigma MLP takes
the positionally encoded 3D positions only as input. It has 1 hidden layer of
64 neurons with the estimated differential opacity $\sigma$ as output. Our
color MLP takes the positionally encoded 3D positions and encoded view
directions as input. It has 2 hidden layers of 64 neurons each and outputs
colors with $n_{c}$ channels. We use multi-resolution hash tables [24] for
positional encoding and use Spherical Harmonics [22] for encoding view
direction. We use separate hash tables for positional encoding of the sigma
and color MLPs. Unlike the original NeRF, there is no interaction between the
parameters of the sigma MLP and the color MLP.

When rendering a LiDAR ray, we only evaluate the sigma MLP while computing
gradients. When rendering a camera ray, we evaluate the sigma MLP, without
computing gradients, to estimate the differential opacity needed in Eq. 1 to
compute the expected color. In contrast to the original NeRF model, which uses
a simple pinhole camera model, we instead use the full pinhole camera model to
compute ray directions. This is needed to maintain consistency with camera-
LiDAR calibration parameters. We use multiple samples per pixel by generating
rays corresponding to fractional pixel coordinates. To obtain ground truth
color information for the fractional pixels, we use bilinear interpolation of
the input images.

###  III-E Loss Functions for NeRF MLPs

We use separate loss functions for LiDAR and camera rays:

#### LiDAR Line-of-sight (LOS) loss

We use a variant of the line-of-sight loss used in [19]. For a given LiDAR
ray, let $z^{*}$ be the ground truth termination depth along the ray. Let
$t_{i}$ be the samples along the ray and $w_{i}$ be the associated accumulated
opacity at those samples. We define
$w_{i}^{*}=\mathcal{K}_{\epsilon}(t_{i}-z^{*})$ where $\mathcal{K}_{\epsilon}$
is a truncated Gaussian distribution with variance equal to $(\epsilon/3)^{2}$
and $\epsilon$ is a non-zero number that is decayed as training progresses. We
can now define the line-of-sight loss as

$\mathcal{L}_{\mathtt{sight}}(\Theta)=\|w_{i}-w_{i}^{*}\|_{1}$ (3)

Ideally, we would like $w_{i}$ to be non-zero only at a single point and zero
everywhere else. To encourage this sparse behavior, we use an L1 penalty.

#### LiDAR Opacity loss

The volumetric rendering equation as implemented in the original NeRF model
only requires that the sum of accumulated opacity does not exceed 1. However,
Eq. 2 implicitly assumes that the accumulated opacities sum to 1. Therefore,
we use the penalty method to force the accumulated opacity for a ray to sum to
1 by using the following opacity loss:

$\mathcal{L}_{\mathtt{opacity_{l}}}(\Theta)=\|1-\sum_{i}w_{i}\|$ (4)

where $\Theta$ is the trainable parameters of the sigma MLP.

#### Camera Color loss

Following [5], we use L1 penalty for supervising camera rays.

$\mathcal{L}_{\mathtt{color}}(\Phi)=\|\hat{\vec{C}}(\vec{r})-\vec{C}(\vec{r})\|_{1}$
(5)

#### Total Loss

The complete loss function is given by

$\mathcal{L}(\Theta,\Phi)=\lambda_{1}\mathcal{L}_{\mathtt{sight}}(\Theta)+\mathcal{L}_{\mathtt{opacity_{l}}}(\Theta)+\\\
\lambda_{2}\mathcal{L}_{\mathtt{color}}(\Phi)$ (6)

where $\lambda_{1}=1000$ and $\lambda_{2}=1$, and only $\lambda_{1}$ is
decayed over training iterations.

Figure 3: Occupancy Grid Mapping as Stochastic Gradient Descent. Panel A
describes the interpolation process to predict occupancy values at each ray
sample from the neighboring $\gamma$ variables in the OGM. Panel B describes
how the $\gamma$ variables of the OGM are updated using stochastic gradient
descent.

###  III-F Occupancy Grid Mapping

Occupancy Grid Mapping (OGM) has long been used in robotics to model 3D space
in a probabilistic manner [25]. In this paper, we cast OGM as an optimization
problem that can be solved using stochastic gradient descent. We then use the
learned OGM to efficiently sample along sensor rays, taking the place of a
coarse MLP in standard NeRF algorithms.

Let $\mathcal{O}\in R^{N\times N\times N}$ denote a regular 3D grid
partitioning a bounded region of interest into voxels. If $p_{k}$ refers to
the probability that a single grid element is occupied, then the corresponding
log-odds is defined as $l_{k}=\log\frac{p_{k}}{1-p_{k}}$. In a typical OGM
algorithm, at each iteration $n$, given a range measurement $z_{n}$, each grid
element is updated as follows:

$l_{n,k}=l_{n-1,k}+\mathcal{S}(k,z_{n})-l_{0}$ (7)

where $l_{0}$ is the prior of occupancy represented in log-odds, $\mathcal{S}$
is the inverse sensor model that returns the measurement update for grid
element $k$ given $z_{n}$ and the known pose of the sensor [25]. We propose to
treat the log-odds variables $l_{k}$ as learnable parameters
$\gamma_{k}\in\mathcal{O}$ initialized to $l_{0}$. We can then re-write the
map update equation Eq. (7) from OGM to match a parameter update equation from
SGD as follows:

$\gamma_{k}:=\gamma_{k}-\alpha\nabla\mathcal{L}(\mathcal{O},z_{n})$ (8)

where $\alpha$ is the learning rate and $\mathcal{L}$ is an objective function
operating on the parameters $\mathcal{O}$ and a measurement $z_{n}$. Matching
the two equations, we get:

$-\alpha\nabla\mathcal{L}(\mathcal{O},z_{n})=\mathcal{S}(k,z_{n})-\gamma_{0}$
(9)

Under this new setting, performing OGM translates to optimizing an unknown
objective function $\mathcal{L}$, whose gradients play the role of the desired
map update provided by the inverse measurement model $\mathcal{S}$.

A trained OGM, $\mathcal{O}$, can be used to efficiently sample points along
sensor rays $\vec{r}(t)$ during volumetric rendering in the following manner.
Given the ray origin $\vec{o}$ and direction $\vec{d}$, we uniformly sample
$N/2$ points $\vec{\zeta}_{j}=\vec{o}+\zeta_{j}\vec{d}$ along this ray where
$N$ is the total number of samples needed for volumetric rendering. We compute
the log-odds values at these continuous coordinates, $l_{\vec{\zeta}_{j}}$,
using trilinear interpolation of the discrete grid $\mathcal{O}$. The
corresponding occupancy is then computed as

$p_{\vec{\zeta}_{j}}=\frac{1}{1+\exp(-l_{\vec{\zeta}_{j}})}$ (10)

Note that a value of $p=0.5$ corresponds to unknown regions. We clamp and re-
scale these probabilities from $[0.5,1.0]$ to $[0.0,1.0]$ and perform
importance sampling to get $N/2$ additional samples around occupied regions.
Finally, both the uniform samples and the additional samples are concatenated
to construct $N$ samples $\vec{\beta}_{j}=\vec{o}+\beta_{j}\vec{d}$ that are
then used in volumetric rendering.

In order to train $\mathcal{O}$, we define a function $g$ that defines the
gradients of $l_{\vec{\beta}_{j}}$ as follows:

$\begin{split}\frac{\partial\mathcal{L}}{\partial
l_{\vec{\beta}_{j}}}=g(\beta_{j},z_{n})&=l_{\mathtt{free}}\mathcal{U}((z_{t}-\delta)-\beta_{j})+\\\
&-l_{\mathtt{occ}}\mathcal{U}(\beta_{j}-(z_{t}-\delta))\mathcal{U}((z_{t}+\delta)-\beta_{j})\end{split}$
(11)

where $\mathcal{U}$ is the Heaviside step function, which takes the value of 1
for positive inputs and zero otherwise, and $\delta$ is a hyperparameter.
These gradients can then be backpropagated through the trilinear interpolation
operator to compute updates for $\gamma_{k}$, as shown pictorially in Panel B
of Fig. 3, which shows an illustration of $g$. Note that we do not explicitly
define $\mathcal{L}(\mathcal{O})$.

###  III-G Training Details

We perform training of the two NeRF MLPs in three stages. In Stage 1, we only
train the sigma MLP using LiDAR data for $2500$ iterations with a batch size
of $1024$ rays per iteration. Note that rendering a LiDAR ray is significantly
faster than rendering a camera ray since only the sigma MLP needs to be
evaluated. In Stage 2, we freeze the weights of the sigma MLP and train the
color MLP using camera data for $2500$ iterations with $1024$ rays per
iteration. Finally, in Stage 3, we jointly fine-tune the full model for
$10000$ iterations with $1024$ LiDAR rays and $1024$ camera rays per
iteration. In stages 1 and 3, we also train the OGM using the same batch of
LiDAR data. We make one update with the OGM optimizer after accumulating
gradients every 10 updates to the NeRF model. With the above settings, the
model takes less than 12 minutes and 45 seconds to train on a single NVIDIA
A100 GPU.

Figure 4: Qualitative comparison of novel view synthesis results for 3
different scenes. Best viewed using a monitor, please zoom in to view details.
Figure 5: Qualitative comparison of rendered depth for the scenes
corresponding to Fig. 4. Ground truth depth is from the KITTI depth prediction
benchmark. The top 1/3rd of the depth maps, which is not covered by LiDAR,
have been cropped out, as done in the KITTI depth estimation benchmark. Best
viewed using a monitor, please zoom in to view details.

##  IV Experimental Results

TABLE I: Quantitative results for novel view synthesis. Metrics reported are averages across the 19 scenes. Algorithm | PSNR $\uparrow$ | MSSIM $\uparrow$ | LPIPS $\downarrow$
---|---|---|---
NeRF | $3.123$ | $0.169$ | $0.788$
MIP-NeRF | $13.274$ | $0.500$ | $0.255$
DSNeRF-gt | $12.849$ | $0.511$ | $0.268$
DSNeRF* | $19.932$ | $0.856$ | 0.120
CLONeR | 20.042 | 0.856 | 0.128
TABLE II: Quantitative results for dense depth prediction. Metrics reported are averages across the 19 scenes. Ground truth depth is from the KITTI depth prediction benchmark. Regions of the scene that do not include LiDAR are not considered, as is done in the KITTI depth estimation benchmark. Algorithm |  $\mu$SILog $\downarrow$ | absErrRel $\downarrow$ | sqErrRel $\downarrow$
---|---|---|---
NeRF | $0.417$ | $0.308$ | $0.568$
MIP-NeRF | $0.568$ | $0.405$ | $0.266$
DSNeRF-gt | $0.442$ | $0.365$ | $0.232$
DSNeRF* | $0.196$ | $0.085$ | $0.027$
CLONeR | 0.101 | 0.073 | 0.017

In this section we present image-only results from for Novel View Synthesis
and Depth Prediction tasks. For video results of Fig. 4 and Fig. 5, please
refer to the project website.

###  IV-A Dataset and Training/Evaluation Protocol

We train and evaluate the proposed model and baselines on 19 scenes from the
KITTI dataset [26]. Each scene consists of three rectified stereo image pairs
and three LiDAR scans corresponding to consecutive time steps, $t-1,t$ and
$t+1$. For training each baseline, we use camera poses and intrinsics
estimated by COLMAP [27, 28]. Since our proposed method operates in metric
space, we use the pose and calibration information provided by the KITTI
dataset to train CLONeR. For each scene, the two LiDAR scans and two left
camera images at time $t-1$ and $t+1$, referred to as Image 000 and Image 002,
are used for training. Note that only CLONeR uses the two LiDAR scans
corresponding to time $t-1$ and $t+1$ for training. Quantitative and
qualitative evaluation is performed on the remaining four images - left image
at time $t$ and all three right images. We refer to these test images as Image
001, Image 003, Image 004, and Image 005, respectively.

###  IV-B Baseline methods

We compare our method against the following state-of-the-art methods: NeRF
[1], Mip-NeRF [21], and DSNeRF [4]. We train each of these models using the
provided hyperparameters within each of the cited code bases. For NeRF, we
used the original Tensorflow implementation. For Mip-NeRF, we use the
available pytorch-lightning implementation [29]. For DSNeRF, we used the
original pytorch implementation [4]. Note that DSNeRF is trained using 3D
keypoints estimated by COLMAP computed on all 6 images. In this setting,
COLMAP would have likely used information from the 4 test images as well to
estimate 3D keypoints visible from the 2 training images. We attempted to
train DSNeRF using only keypoints that were estimated from the 2 training
images but the training did not converge. This gives an advantage to DSNeRF
compared to the other methods. We add an asterisk to all DSNeRF results to
denote this advantage. In addition, we consider a fourth baseline, which we
call DSNeRF-gt, where we replace pose, focal length and 3D keypoint
information from COLMAP with ground truth information provided by the KITTI
dataset. Specifically, we aggregate points from the two LiDAR scans and treat
them as 3D keypoints. We projected the points onto each frame and retained
only points within its field of view. Note that DSNeRF-gt and our proposed
method use the same amount of ground truth input information.

###  IV-C Novel View Synthesis Results

We report three standard metrics for quantitative evaluation in Table I: Peak
Signal to Noise Ratio (PSNR) [30], Multiscale Structural Similarity Index
(MSSIM) [31], and LPIPS [32]. CLONeR meets or exceeds DSNeRF* with respect to
PSNR and MSSIM while scoring marginally lower with respect to LPIPS. Note that
CLONeR expects LiDAR data by design and is therefore at a disadvantage for the
upper regions of the image where there is no LiDAR coverage. Qualitative
results on selected images from three scenes are shown in Fig. 4. While all
methods were able to recover general scene details, Mip-NeRF performed
significantly worse than others. Both CLONeR and DSNeRF* achieve similar level
of detail in the rendered novel views. In our experiments, DSNeRF-gt performed
worse than DSNeRF*. Although LiDAR-based depth is more precise than values
obtained from COLMAP, different points in the LiDAR would be occluded from
each of the camera viewpoint. And when projected, it associates a wrong depth
to a scene point. As a result, we see that the trained model learns the image
at $t-1$ and $t+1$ to be on two different planes.

###  IV-D Depth Prediction Results

For quantitative evaluation, we use the evaluation protocol from the KITTI
depth prediction benchmark. This means that both our reported metrics and
ground truth depth maps are from the KITTI depth prediction benchmark suite,
and regions of the scene that do not include LiDAR are not considered, such as
the sky. While CLONeR and DSNeRF-gt operate in metric space, all the other
baselines can only estimate depth up-to scale (as determined by the COLMAP
poses). Thus, we follow the evaluation protocol in [33] and align the median
depth of the predicted depth maps with the ground truth before computing the
metrics. As seen in Table II, CLONeR outperforms all baselines across the
three metrics when averaged across the 19 scenes.

The pseudo-ground truth depth maps used in the quantitative evaluation along
with qualitative results are shown in Fig. 5. Note that the pseudo-ground
truth depth maps are sparse and contain many empty pixels where there is no
ground truth. A visual inspection of the qualitative results demonstrate the
improved performance of CLONeR compared to all the baselines. CLONeR captures
greater details in all scenes along with fine structures in the regions where
LiDAR data is present. MIP-NeRF suffers from significant planar artifacts,
potentially from the geometry ambiguity failure mode that NeRF models
experience when trained on sparse views [17]. The depth maps from NeRF lack
any structural detail. For DSNeRF-gt, the left training image (Image 000)
seems to be learnt on a plane close to the camera. As a result, the depth map
for Image 003 (corresponding to right test image) has large planar portions
that is behind the camera pose for the second training image (Image 002).
Consequently, depth maps for Image 004 and Image 005 show more structural
detail. While DSNeRF* is able to capture global structure in the depth maps,
the object boundaries are not clearly defined, see the results for scene 2,
column 2 in the figure. It also struggles in regions with fine structures,
such as the bushes in scene 3, final column of Fig. 5.

###  IV-E Ablation Experiments

TABLE III: Ablation experiments. Decoupled indicates whether the proposed decoupled Color/Sigma MLPs are used (instead of a single MLP) for fine sampling of the scene, LiDAR indicates if LiDAR rays are used as depth guidance, LOS/Term indicates if Line of Sight loss or Termination depth loss is used, and CF indicates if a coarse MLP is used (instead of the OGM). Type | Decoupled | LiDAR | LOS / Term | OGM | CF
---|---|---|---|---|---
Ablation A | ✗ | ✗ | N/A | ✗ | ✓
Ablation B | ✗ | ✓ | ✓/ ✗ | ✓ | ✗
Ablation C | ✓ | ✓ | ✗/ ✓ | ✓ | ✗
Ablation D | ✓ | ✓ | ✓/ ✗ | ✗ | ✓
Proposed | ✓ | ✓ | ✓/ ✗ | ✓ | ✗
Figure 6: Qualitative comparison of rendered depth maps from a test image for different ablations. The yellow boxes highlight regions of saturation and red boxes highlight hard cast shadows TABLE IV: Ablation quantitative evaluation for Novel View Synthesis and Depth estimation. Metrics reported are averages across the 19 scenes. Regions of the scene that do not include LiDAR are not considered, as is done in the KITTI depth estimation benchmark. Ablation | Novel View Synthesis
---|---
PSNR $\uparrow$ | MSSIM $\uparrow$ | LPIPS $\downarrow$
Ablation A | $18.491$ | $0.781$ | $0.148$
Ablation B | $15.803$ | $0.742$ | $0.179$
Ablation C | $17.142$ | $0.735$ | $0.189$
Ablation D | $17.657$ | $0.782$ | $0.156$
Proposed |  20.042 |  0.856 |  0.128
Ablation | Depth Prediction
SILog $\downarrow$ | absErrRel $\downarrow$ | sqrErrRel $\downarrow$
Ablation A | $0.554$ | $0.461$ | $0.422$
Ablation B | $0.282$ | $0.141$ | $0.048$
Ablation C | $0.132$ | $0.097$ | $0.027$
Ablation D | $0.353$ | $0.314$ | $0.264$
Proposed | 0.101 | 0.073 | 0.017

To analyze the different components of the proposed work, we consider 5
ablation experiments, described in Table III. We present the quantitative
results in Table IV. The proposed method demonstrates comparable performance
across ablation experiments for novel view synthesis, but yields the highest
performance for depth prediction. In Fig. 6, we show how the different
features of CLONeR eliminate the failure modes that NeRFs have in outdoor
scenes. Ablation A and B, which both use coupled NeRFs, experience the failure
mode where both saturation and hard cast shadows are learned as separate
objects or planes; these regions are highlighted in the yellow callout box and
red callout box (respectively) in each depth map of Fig. 6. Ablation A, B, and
C all experience the failure mode where fine structures cannot be learned. We
observe that all the structures in the yellow callout box for these depth maps
are noticeably blurry, which highlights not only the importance of LiDAR but
also the proposed volumetric carving losses. Ablation D experiences the
failure mode where both global and local smooth scene structure is difficult
to learn without proper sampling in a large outdoor scene, highlighting the
importance of the proposed OGM. Finally, we observe that all ablation models
struggle to learn correct scene geometry in the upper parts of the image.

##  V Conclusion

This paper tackles the problem of novel view synthesis and dense depth
prediction for large unbounded outdoor scenes with sparse input views. We
proposed CLONeR, which extends NeRFs to operate on camera and LiDAR data and
leverages occupancy grid mapping for efficient sampling, thus replacing the
standard coarse NeRF MLP. Through both quantitative and qualitative
experiments on the KITTI dataset, we demonstrate that the proposed method
outperforms state-of-the-art algorithms on both tasks. Future work will
investigate improving the performance of the proposed method in regions where
LiDAR data is not present, but camera data is available, namely the sky and
regions of the scene where LiDAR cannot be collected, such as through glass.
Ultimately, this method opens up significant opportunity to begin using NeRFs
to learn dense maps of outdoor scenes and can allow researchers to augment
existing datasets with novel views and corresponding depth maps.

## References

  * [1] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, “Nerf: Representing scenes as neural radiance fields for view synthesis,” in _European conference on computer vision_. Springer, 2020, pp. 405–421.
  * [2] E. R. Chan, M. Monteiro, P. Kellnhofer, J. Wu, and G. Wetzstein, “Pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2021, pp. 5799–5809.
  * [3] X. Zhang, P. P. Srinivasan, B. Deng, P. Debevec, W. T. Freeman, and J. T. Barron, “Nerfactor: Neural factorization of shape and reflectance under an unknown illumination,” _ACM Transactions on Graphics (TOG)_ , vol. 40, no. 6, pp. 1–18, 2021.
  * [4] K. Deng, A. Liu, J.-Y. Zhu, and D. Ramanan, “Depth-supervised nerf: Fewer views and faster training for free,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 12 882–12 891.
  * [5] E. Sucar, S. Liu, J. Ortiz, and A. J. Davison, “imap: Implicit mapping and positioning in real-time,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 6229–6238.
  * [6] Z. Zhu, S. Peng, V. Larsson, W. Xu, H. Bao, Z. Cui, M. R. Oswald, and M. Pollefeys, “Nice-slam: Neural implicit scalable encoding for slam,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2022.
  * [7] T. Neff, P. Stadlbauer, M. Parger, A. Kurz, J. H. Mueller, C. R. A. Chaitanya, A. Kaplanyan, and M. Steinberger, “Donerf: Towards real-time rendering of compact neural radiance fields using depth oracle networks,” in _Computer Graphics Forum_ , vol. 40, no. 4. Wiley Online Library, 2021, pp. 45–59.
  * [8] B. Attal, E. Laidlaw, A. Gokaslan, C. Kim, C. Richardt, J. Tompkin, and M. O’Toole, “Törf: Time-of-flight radiance fields for dynamic scene view synthesis,” _Advances in neural information processing systems_ , vol. 34, pp. 26 289–26 301, 2021.
  * [9] M. Piala and R. Clark, “Terminerf: Ray termination prediction for efficient neural rendering,” in _2021 International Conference on 3D Vision (3DV)_. IEEE, 2021, pp. 1106–1114.
  * [10] P. Wang, L. Liu, Y. Liu, C. Theobalt, T. Komura, and W. Wang, “Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction,” _arXiv preprint arXiv:2106.10689_ , 2021.
  * [11] L. Yariv, J. Gu, Y. Kasten, and Y. Lipman, “Volume rendering of neural implicit surfaces,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 4805–4815, 2021.
  * [12] M. Niemeyer, J. T. Barron, B. Mildenhall, M. S. Sajjadi, A. Geiger, and N. Radwan, “Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 5480–5490.
  * [13] K. Zhang, G. Riegler, N. Snavely, and V. Koltun, “Nerf++: Analyzing and improving neural radiance fields,” _arXiv preprint arXiv:2010.07492_ , 2020\.
  * [14] J. T. Barron, B. Mildenhall, D. Verbin, P. P. Srinivasan, and P. Hedman, “Mip-nerf 360: Unbounded anti-aliased neural radiance fields,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 5470–5479.
  * [15] M. Tancik, V. Casser, X. Yan, S. Pradhan, B. Mildenhall, P. P. Srinivasan, J. T. Barron, and H. Kretzschmar, “Block-nerf: Scalable large scene neural view synthesis,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 8248–8258.
  * [16] Y. Xiangli, L. Xu, X. Pan, N. Zhao, A. Rao, C. Theobalt, B. Dai, and D. Lin, “Citynerf: Building nerf at city scale,” _arXiv preprint arXiv:2112.05504_ , 2021.
  * [17] J. T. Barron, B. Mildenhall, D. Verbin, P. P. Srinivasan, and P. Hedman, “Mip-nerf 360: Unbounded anti-aliased neural radiance fields,” _arXiv preprint arXiv:2111.12077_ , 2021.
  * [18] R. Martin-Brualla, N. Radwan, M. S. Sajjadi, J. T. Barron, A. Dosovitskiy, and D. Duckworth, “Nerf in the wild: Neural radiance fields for unconstrained photo collections,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 7210–7219.
  * [19] K. Rematas, A. Liu, P. P. Srinivasan, J. T. Barron, A. Tagliasacchi, T. Funkhouser, and V. Ferrari, “Urban radiance fields,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 12 932–12 942.
  * [20] J. Ost, I. Laradji, A. Newell, Y. Bahat, and F. Heide, “Neural point light fields,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 18 419–18 429.
  * [21] J. T. Barron, B. Mildenhall, M. Tancik, P. Hedman, R. Martin-Brualla, and P. P. Srinivasan, “Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 5855–5864.
  * [22] A. Yu, R. Li, M. Tancik, H. Li, R. Ng, and A. Kanazawa, “Plenoctrees for real-time rendering of neural radiance fields,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 5752–5761.
  * [23] C. Sun, M. Sun, and H.-T. Chen, “Improved direct voxel grid optimization for radiance fields reconstruction,” _arXiv preprint arXiv:2206.05085_ , 2022\.
  * [24] T. Müller, A. Evans, C. Schied, and A. Keller, “Instant neural graphics primitives with a multiresolution hash encoding,” _ACM Trans. Graph._ , vol. 41, no. 4, pp. 102:1–102:15, Jul. 2022. [Online]. Available: https://doi.org/10.1145/3528223.3530127
  * [25] S. Thrun, “Probabilistic robotics,” _Communications of the ACM_ , vol. 45, no. 3, pp. 52–57, 2002.
  * [26] A. Geiger, P. Lenz, and R. Urtasun, “Are we ready for autonomous driving? the kitti vision benchmark suite,” in _2012 IEEE conference on computer vision and pattern recognition_. IEEE, 2012, pp. 3354–3361.
  * [27] J. L. Schönberger, E. Zheng, M. Pollefeys, and J.-M. Frahm, “Pixelwise view selection for unstructured multi-view stereo,” in _European Conference on Computer Vision (ECCV)_ , 2016.
  * [28] J. L. Schönberger and J.-M. Frahm, “Structure-from-motion revisited,” in _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016.
  * [29] J. Huang, “Unofficial pytorch-lightening implementation of mipnerf,” 2022. [Online]. Available: $https://github.com/hjxwhy/mipnerf_{p}l$
  * [30] J. Korhonen and J. You, “Peak signal-to-noise ratio revisited: Is simple beautiful?” in _2012 Fourth International Workshop on Quality of Multimedia Experience_. IEEE, 2012, pp. 37–38.
  * [31] Z. Wang, E. P. Simoncelli, and A. C. Bovik, “Multiscale structural similarity for image quality assessment,” in _The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003_, vol. 2. Ieee, 2003, pp. 1398–1402.
  * [32] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, “The unreasonable effectiveness of deep features as a perceptual metric,” in _CVPR_ , 2018\.
  * [33] L. Wang, Y. Wang, L. Wang, Y. Zhan, Y. Wang, and H. Lu, “Can scale-consistent monocular depth be learned in a self-supervised scale-invariant manner?” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 12 727–12 736.