# MOS: Towards Scaling Out-of-distribution Detection for Large Semantic Space

Rui Huang
Department of Computer Sciences
University of Wisconsin-Madison
<EMAIL_ADDRESS>Yixuan Li
Department of Computer Sciences
University of Wisconsin-Madison
<EMAIL_ADDRESS>
###### Abstract

Detecting out-of-distribution (OOD) inputs is a central challenge for safely
deploying machine learning models in the real world. Existing solutions are
mainly driven by small datasets, with low resolution and very few class labels
(, CIFAR). As a result, OOD detection for large-scale image classification
tasks remains largely unexplored. In this paper, we bridge this critical gap
by proposing a group-based OOD detection framework, along with a novel OOD
scoring function termed MOS. Our key idea is to decompose the large semantic
space into smaller groups with similar concepts, which allows simplifying the
decision boundaries between in- vs. out-of-distribution data for effective OOD
detection. Our method scales substantially better for high-dimensional class
space than previous approaches. We evaluate models trained on ImageNet against
four carefully curated OOD datasets, spanning diverse semantics. MOS
establishes state-of-the-art performance, reducing the average FPR95 by 14.33%
while achieving 6x speedup in inference compared to the previous best method.

Figure 1: Top: Examples of in-distribution images sampled from ImageNet-1k (in
green) and OOD images sampled from 4 datasets described in Section 4.1 (in
orange). Bottom: Overview of the proposed group-based OOD detection framework.
The key idea is to decompose the large semantic space into smaller groups,
which allows simplifying the decision boundary between in- and out-of-
distribution data. A category others is added to each group. An OOD image is
mapped to others with high confidence for all groups, whereas an in-
distribution image will have a lower score for others in the group it belongs
to (, artifact group). The minimum score on category others among all groups,
MOS, allows effective differentiation of OOD data.

##  1 Introduction

Out-of-distribution (OOD) detection has become a central challenge in safely
deploying machine learning models in the open world, where the test data may
be distributionally different from the training data. A plethora of literature
has emerged in addressing the problem of OOD detection [3, 14, 16, 24, 27, 29,
36, 4, 20, 32, 30]. However, existing solutions are mainly driven by small,
low-resolution datasets such as CIFAR [23] and MNIST [25]. Deployed systems
like autonomous vehicles often operate on images that have far greater
resolution and perceive environments with far more categories. As a result, a
critical research gap exists in developing and evaluating OOD detection
algorithms for large-scale image classification tasks.

While one may be eager to conclude that solutions for small datasets should
transfer to a large-scale setting, we argue that this is far from the truth.
The main challenges posed in OOD detection stem from the fact that it is
impossible to comprehensively define and anticipate anomalous data in advance,
resulting in a large space of uncertainty. As the number of semantic classes
increases, the plethora of ways that OOD data may occur increases
correspondingly. For example, our analysis reveals that the average false
positive rate (at 95% true positive rate) of a common baseline [16] would rise
from 17.34% to 76.94% as the number of classes increases from 50 to 1,000 on
ImageNet-1k [8]. Very few works have studied OOD detection in the large-scale
setting, with limited evaluations and effectiveness [42, 15]. This begs the
following question: _how can we design an OOD detection algorithm that scales
effectively for classification with large semantic space_?

Motivated by this, we take an important step to bridge this gap and propose a
group-based OOD detection framework that is effective for large-scale image
classification. Our key idea is to decompose the large semantic space into
smaller groups with similar concepts, which allows simplifying the decision
boundary and reducing the uncertainty space between in- vs. out-of-
distribution data. Intuitively, for OOD detection, it is simpler to estimate
whether an image belongs to one of the coarser-level semantic groups than to
estimate whether an image belongs to one of the finer-grained classes. For
example, consider a model tasked with classifying 200 categories of
plantations and another 200 categories of marine animals. A truck image can be
easily classified as OOD data since it does not resemble either the plantation
group or the marine animal group.

Formally, our proposed method leverages group softmax and derives a novel OOD
scoring function. Specifically, the group softmax computes probability
distributions within each semantic group. A key component is to utilize a
category others in each group, which measures the probabilistic score for an
image to be OOD with respect to the group. Our proposed OOD scoring function,
_Minimum Others Score_ (MOS), exploits the information carried by the others
category. As illustrated in Figure 1, MOS is higher for OOD inputs as they
will be mapped to others with high confidence in all groups, and is lower for
in-distribution inputs.

We extensively evaluate our approach on models trained with the ImageNet-1k
dataset, leveraging the state-of-the-art pre-trained BiT-S models [22] as
backbones. We explore label space of size 10-100 times larger than that of
previous works [16, 29, 27, 32, 4, 17]. Compared to the best baseline [15],
our method improves the average performance of OOD detection by 14.33% (FPR95)
over four diverse OOD test datasets. More importantly, our method achieves
improved OOD detection performance while preserving the classification
accuracy on in-distribution datasets. We note that while group-based learning
has been used for improving tasks such as long-tail object detection [28], our
objective and motivation are very different—we are interested in reducing the
uncertainty between in- and out-of-distribution data, rather than reducing the
confusion among in-distribution data themselves. Below we summarize our key
results and contributions:

  * •

We propose a group-based OOD detection framework, along with a novel OOD
scoring function MOS, that scales substantially better for large label space.
Our method establishes the new state-of-the-art performance, reducing the
average FPR95 by 14.33% while achieving 6x speedup in inference time compared
to the best baseline.

  * •

We conduct extensive ablations which improve understandings of our method for
large-scale OOD detection under (1) different grouping strategies, (2)
different sizes of semantic class space, (3) different backbone architectures,
and (4) varying fine-tuning capacities.

  * •

We curate diverse OOD evaluation datasets from four real-world high-resolution
image databases, which enables future research to evaluate OOD detection
methods in a large-scale setting111 Code and data for reproducing our results
are available at: https://github.com/deeplearning-wisc/large_scale_ood.

##  2 Preliminary and Analysis

##### Preliminaries

We consider a training dataset drawn i.i.d. from the in-distribution
$P_{\bm{X}}$, with label space ${Y}=\\{1,2,\cdots,C\\}$. For OOD detection
problem, it is common to train a classifier $f(\mathbf{x})$ on the in-
distribution $P_{\bm{X}}$, and evaluate on samples that are drawn from a
different distribution $Q_{\bm{X}}$. An OOD detector $G(\mathbf{x})$ is a
binary classifier:

$G(\mathbf{x})=\begin{cases}\text{in},&\text{if}\ S(\mathbf{x})\geq\gamma\\\
\text{out},&\text{if}\ S(\mathbf{x})<\gamma,\end{cases}$

where $S(\mathbf{x})$ is the scoring function, and $\gamma$ is the threshold
chosen so that a high fraction (, 95%) of in-distribution data is correctly
classified.

##### Effect of Number of Classes on OOD Detection

We first revisit the common baseline approach [16], which uses the maximum
softmax probability (MSP),
$S(\mathbf{x})=\max_{i}\frac{e^{f_{i}(\mathbf{x})}}{\sum_{j=1}^{C}e^{f_{j}(\mathbf{x})}}$,
for OOD detection. We investigate the effect of label space size on the OOD
detection performance. In particular, we use a ResNetv2-101 architecture [13]
trained on different subsets222To create the training subset, we first
randomly select $C$ ($C\in\\{50,200,300,400,500,600,700,800,900,1000\\}$)
labels from the 1,000 ImageNet classes. For each of the chosen label, we then
sample 700 images for training. of ImageNet with varying numbers of classes
$C$. As shown in Figure 2, the performance (FPR95) degrades rapidly from
17.34% to 76.94% as the number of in-distribution classes increases from 50 to
1,000. This trend signifies that current OOD detection methods are indeed
challenged by the increasingly large label space, which motivates our work.

Figure 2: OOD detection performance of a common baseline MSP [16] decreases
rapidly as the number of ImageNet-1k classes increases (_left_ : AUROC;
_right_ : FPR95). Figure 3: A toy example in 2D space of group-based OOD
detection framework. _Left_ : without grouping, the decision boundary between
in- vs. out-of-distribution data becomes increasingly complex with more
classes. _Right_ : our group-based method simplifies the decision boundary and
reduces the uncertainty space for OOD data.

##  3 Method

Our novel group-based OOD detection framework is illustrated in Figure 1. In
what follows, we first provide an overview and then describe the group softmax
training technique in Section 3.1. We introduce our proposed OOD detection
algorithm MOS in Section 3.2, followed with grouping strategies in Section
3.3.

##### Method Overview: A Conceptual Example

As aforementioned, OOD detection performance can suffer notably from the
increasing number of in-distribution classes. To mitigate this issue, our key
idea is to decompose the large semantic space into smaller groups with similar
concepts, which allows simplifying the decision boundary and reducing the
uncertainty space between in- vs. out-of-distribution data. We illustrate our
idea with a toy example in Figure 3, where the in-distribution data consists
of class-conditional Gaussians. Without grouping (left), the decision boundary
between in- vs. OOD data is determined by _all_ classes and becomes
increasingly complex as the number of classes grows. In contrast, with
grouping (right), the decision boundary for OOD detection can be significantly
simplified, as shown by the dotted curves.

In other words, by way of grouping, the OOD detector only needs to make a
small number of relatively simple estimations about whether an image belongs
to this group, as opposed to making a large number of hard decisions about
whether an image belongs to this class. An image will be classified as OOD if
it belongs to none of the groups. We proceed with describing the training
mechanism that achieves our novel conceptual idea.

###  3.1 Group-based Learning

We divide the total number of $C$ categories into $K$ groups,
$\mathcal{G}_{1},\mathcal{G}_{2},...,\mathcal{G}_{K}$. We calculate the
standard group-wise softmax for each group $\mathcal{G}_{k}$:

$p_{c}^{k}(\mathbf{x})=\frac{e^{f_{c}^{k}(\mathbf{x})}}{\sum_{c^{\prime}\in\mathcal{G}_{k}}e^{f_{c^{\prime}}^{k}(\mathbf{x})}},\
c\in\mathcal{G}_{k},\vspace{-0.1cm}$ (1)

where $f_{c}^{k}(\mathbf{x})$ and $p_{c}^{k}(\mathbf{x})$ denote the output
logit and the softmax probability for class $c$ in group $\mathcal{G}_{k}$,
respectively.

##### Category “Others”

Standard group softmax is insufficient as it can only discriminate classes
within the group, but cannot estimate the OOD uncertainty between inside vs.
outside the group. To this end, a new category others is introduced to every
group, as shown in Figure 1. The model can predict others if the input
$\mathbf{x}$ does not belong to this group. In other words, the others
category allows explicitly learning the decision boundary between inside vs.
outside the group, as illustrated by the dashed curves surrounding classes
C1/C2/C3 in Figure 3. This is desirable for OOD detection, as an OOD input can
be mapped to others for all groups, whereas an in-distribution input will be
mapped to one of the semantic categories in some group with high confidence.

Importantly, our use of the category others creates “virtual” group-level
outlier data without relying on any external data. Each training example
$\mathbf{x}$ not only helps estimate the decision boundary for the
classification problem, but also effectively improves the OOD uncertainty
estimation for groups to which it does not belong. We show the formulation can
in fact achieve the dual objective of in-distribution classification, as well
as OOD detection.

##### Training and Inference

During training, the ground-truth labels are re-mapped in each group. In
groups where $c$ is not included, class others will be defined as the ground-
truth class. The training objective is a sum of cross-entropy losses in each
group:

$\mathcal{L}_{GS}=-\frac{1}{N}\sum_{n=1}^{N}\sum_{k=1}^{K}\sum_{c\in\mathcal{G}_{k}}y_{c}^{k}\log(p_{c}^{k}(\mathbf{x})),\vspace{-0.1cm}$
(2)

where $y_{c}^{k}$ and $p_{c}^{k}$ represent the label and the softmax
probability of category $c$ in $\mathcal{G}_{k}$, and $N$ is the total number
of training samples.

We denote the set of all valid (non-others) classes in each group as
$\mathcal{G}^{\prime}_{k}=\mathcal{G}_{k}\backslash\\{\text{{others}}\\}$.
During inference time, we derive the group-wise class prediction in the valid
set for each group:

$\hat{p}^{k}=\max_{c\in\mathcal{G}^{\prime}_{k}}p_{c}^{k}(\mathbf{x}),\
\hat{c}^{k}=\operatorname*{arg\,max}_{c\in\mathcal{G}^{\prime}_{k}}p_{c}^{k}(\mathbf{x}).\vspace{-0.1cm}$

Then we use the maximum group-wise softmax score and the corresponding class
for final prediction:

$k_{*}=\operatorname*{arg\,max}_{1\leq k\leq K}\hat{p}^{k}.\vspace{-0.1cm}$

The final prediction is category $\hat{c}^{k_{*}}$ from group
$\mathcal{G}_{k_{*}}$.

###  3.2 OOD Detection with MOS

For a classification model trained with the group softmax loss, we propose a
novel OOD scoring function, Minimum Others Score (MOS), that allows effective
differentiation between in- vs. out-of-distribution data. Our key observation
is that category others carries useful information for how likely an image is
OOD with respect to each group.

As discussed in Section 3.1, an OOD input will be mapped to others with high
confidence in all groups, whereas an in-distribution input will have a low
score on category others in the group it belongs to. Therefore, the lowest
others score among all groups is crucial for distinguishing between in- vs.
out-of-distribution data. This leads to the following OOD scoring function,
termed as _Minimum Others Score_ :

$S_{\text{MOS}}(\mathbf{x})=-\min_{1\leq k\leq
K}{p_{\text{others}}^{k}(\mathbf{x})}.\vspace{-0.1cm}$ (3)

Note that we negate the sign to align with the conventional notion that
$S_{\text{MOS}}(\mathbf{x})$ is higher for in-distribution data and lower for
out-of-distribution.

To provide an interpretation and intuition behind MOS, we show in Figure 4 the
average scores for the category others in each group for both in-distribution
and OOD images. For in-distribution, we select all validation images from the
animal group in the ImageNet-1k dataset. The minimum others score among all
groups is significantly lower for in-distribution data than that for OOD data,
allowing for effective differentiation between them.

Figure 4: Average of others scores in each group for both in-distribution data (left) and OOD data (right). Method |  Test Time (min) | iNaturalist | SUN | Places | Textures | Average
---|---|---|---|---|---|---
AUROC | FPR95 | AUROC | FPR95 | AUROC | FPR95 | AUROC | FPR95 | AUROC | FPR95
$\uparrow$ | $\downarrow$ | $\uparrow$ | $\downarrow$ | $\uparrow$ | $\downarrow$ | $\uparrow$ | $\downarrow$ | $\uparrow$ | $\downarrow$
MSP [16] | 3.1 | 87.59 | 63.69 | 78.34 | 79.98 | 76.76 | 81.44 | 74.45 | 82.73 | 79.29 | 76.96
ODIN [29] | 23.6 | 89.36 | 62.69 | 83.92 | 71.67 | 80.67 | 76.27 | 76.30 | 81.31 | 82.56 | 72.99
Mahalanobis [27] | 145.4 | 46.33 | 96.34 | 65.20 | 88.43 | 64.46 | 89.75 | 72.10 | 52.23 | 62.02 | 81.69
Energy [32] | 3.1 | 88.48 | 64.91 | 85.32 | 65.33 | 81.37 | 73.02 | 75.79 | 80.87 | 82.74 | 71.03
KL Matching [15] | 20.6 | 93.00 | 27.36 | 78.72 | 67.52 | 76.49 | 72.61 | 87.07 | 49.70 | 83.82 | 54.30
MOS (ours) | 3.2 | 98.15 | 9.28 | 92.01 | 40.63 | 89.06 | 49.54 | 81.23 | 60.43 | 90.11 | 39.97
Table 1: OOD detection performance comparison between MOS and baselines. All
methods are fine-tuned from the same pre-trained BiT-S-R101x1 backbone with
ImageNet-1k as in-distribution dataset. The description of 4 OOD test datasets
is provided in Section 4.1. $\uparrow$ indicates larger values are better,
while $\downarrow$ indicates smaller values are better. All values are
percentages. Bold numbers are superior results. Test time for all methods are
evaluated with the same in- and out-of-distribution datasets (60k images in
total).

###  3.3 Grouping Strategies

Given the dependency on the group structure, a natural question arises: _how
do different grouping strategies affect the performance of OOD detection_? To
answer this, we systematically consider three grouping strategies: (1)
taxonomy, (2) feature clustering, and (3) random grouping.

##### Taxonomy

The first grouping strategy is applicable when the taxonomy of the label space
is known. For example, in the case of ImageNet, each class is associated with
a synset in WordNet [35], from which we can build the taxonomy as a
hierarchical tree. In particular, we adopt the 8 super-classes defined by
ImageNet333http://image-net.org/explore as our groups and map each category
into one of the 8 groups: animal, artifact, geological formation, fungus,
misc, natural object, person, and plant.

##### Feature Clustering

When taxonomy is not available, we can approximately estimate the structure of
semantic classes through feature clustering. Specifically, we extract feature
representations for each training image from a pre-trained feature extractor.
Then, the feature representation of each class is the average of feature
embeddings in that class. Finally, we perform a K-Means clustering [33] on
categorical feature representations, one for each class.

##### Random Grouping

Lastly, we contrast the taxonomy and the feature clustering strategies with
random grouping, where each class is randomly assigned to a group. This allows
us to estimate the lower bound of OOD detection performance with MOS.

By default, we use taxonomy as the grouping strategy if not specified
otherwise. In Section 4.3.3, we experimentally compare the OOD detection
performance using all three grouping strategies.

##  4 Experiments

We first describe the evaluation datasets (Section 4.1) and experimental
setups (Section 4.2). In Section 4.3, we show that MOS achieves state-of-the-
art OOD detection performance, followed by extensive ablations that improve
the understandings of MOS for large-scale OOD detection.

###  4.1 Datasets

####  4.1.1 In-distribution Dataset

We use ImageNet-1k [8] as the in-distribution dataset, which covers a wide
range of real-world objects. ImageNet-1k has at least 10 times more labels
compared to CIFAR datasets used in prior literature. In addition, the image
resolution is also significantly higher than CIFAR (32$\times$32) and MNIST
(28$\times$28).

####  4.1.2 Out-of-distribution Datasets

To evaluate our approach, we consider a diverse collection of OOD test
datasets, spanning various domains including fine-grained images, scene
images, and textural images. We carefully curate the OOD evaluation benchmarks
to make sure concepts in these datasets do not overlap with ImageNet-1k. Below
we describe the construction of each evaluation dataset in detail. Samples of
each OOD dataset are provided in Figure 1. We provide the list of concepts
chosen for each OOD dataset in Appendix A.

##### iNaturalist

iNaturalist [47] is a fine-grained dataset containing 859,000 images across
more than 5,000 species of plants and animals. All images are resized to have
a max dimension of 800 pixels. We manually select 110 plant classes not
present in ImageNet-1k, and randomly sample 10,000 images for these 110
classes.

##### SUN

SUN [49] is a scene database of 397 categories and 130,519 images with sizes
larger than $200\times 200$. SUN and ImageNet-1k have overlapping categories.
Therefore, we carefully select 50 nature-related concepts that are unique in
SUN, such as forest and iceberg. We randomly sample 10,000 images for these 50
classes.

##### Places

Places365 [51] is another scene dataset with similar concept coverage as SUN.
All images in this dataset have been resized to have a minimum dimension of
512. We manually select 50 categories from this dataset that are not present
in ImageNet-1k and then randomly sample 10,000 images for these 50 categories.

##### Textures

Textures [6] consists of 5,640 images of textural patterns, with sizes ranging
between $300\times 300$ and $640\times 640$. We use the entire dataset for
evaluation.

Figure 5: OOD detection performance of MOS (blue) and the MSP baseline (gray).
MOS exhibits more stabilized performance as the number of in-distribution
classes increases. For each OOD dataset, we show AUROC (top) and FPR95 (
bottom).

###  4.2 Experiment Setup

##### Pre-trained Backbone

We use Google BiT-S models [22] as our feature extractor in all experiments.
The models are trained on ImageNet-1k, with ResNetv2 architectures [13] at
varying capacities. Pre-trained models allow extracting high-quality features
with minimal time and energy consumption. In practice, one can always choose
to train from scratch.

For the main results, we use the BiT-S-R101x1 model with depth 101 and width
factor 1, unless specified otherwise. We provide a comparison of using feature
extractors of varying model sizes in Section 4.3.4. For efficiency, we fix the
backbone and only fine-tune the last fully-connected (FC) layer in the main
experiments. We additionally explore the effect of fine-tuning more layers
beyond the last FC layer in Section 4.3.5.

##### Training Details

We follow the procedure in BiT-HyperRule [22] and fine-tune the pre-trained
BiT-S model for 20k steps with a batch size of 512. We use SGD with an initial
learning rate of 0.003 and a momentum of 0.9. The learning rate is decayed by
a factor of 10 at 30%, 60%, and 90% of the training steps. During training,
all images are resized to 512 $\times$ 512 and randomly cropped to 480
$\times$ 480\. At test time, all images are resized to 480 $\times$ 480\. A
learning rate warm-up is used for the first 500 steps. We perform all
experiments on NVIDIA GeForce RTX 2080Ti GPUs.

##### Evaluation Metrics

We measure the following metrics that are commonly used for OOD detection: (1)
the false positive rate of OOD examples when the true positive rate of in-
distribution examples is at 95% (FPR95); (2) the area under the receiver
operating characteristic curve (AUROC). We additionally report the area under
the precision-recall curve (AUPR) in Appendix D.

###  4.3 Results

####  4.3.1 MOS vs. Existing Methods

The main results are shown in Table 1. We report performance for each dataset
described in Section 4.1, as well as the average performance. For fair
evaluation, we compare with competitive methods in the literature that derive
OOD scoring functions from a model trained on in-distribution data and do not
rely on auxiliary outlier data. We first compare with approaches driven by
small datasets, including MSP [16], ODIN [29], Mahalanobis [27], as well as
Energy [32]. All these methods rely on networks trained with flat softmax.
Under the same network backbone (BiT-S-R101x1), MOS outperforms the best
baseline Energy [32] by 31.06% in FPR95. It is also worth noting that fine-
tuning with group softmax maintains competitive classification accuracy
(75.16%) on in-distribution data compared with its flat softmax counterpart
(75.20%).

We also compare our method with KL matching [15], a competitive baseline
evaluated on large-scale image classification. MOS reduces FPR95 by 14.33%
compared to KL matching. Note that for each input, KL matching needs to
calculate its KL divergence to all class centers. Therefore, the running time
of KL matching increases linearly with the number of in-distribution
categories, which can be computationally expensive for a very large label
space. As shown in Table 1, our method achieves a 6x speedup compared to KL
matching.

####  4.3.2 MOS with Increasing Numbers of Classes

In Figure 5, we show the OOD detection performance as we increase the number
of in-distribution classes $C\in\\{50,200,300,400,500,600,700,800,900,1000\\}$
on ImageNet-1k. For each $C$, we create training data by first randomly
sampling $C$ labels from the entire 1k classes, and then sampling 700 images
for each chosen label. Importantly, we observe that MOS (in blue) exhibits
more stabilized performance as $C$ increases, compared to MSP [16] (in gray).
For example, on the iNaturalist OOD dataset, FPR95 rises from 21.02% to 63.36%
using MSP, whilst MOS degrades by only 4.76%. This trend signifies that MOS is
an effective approach for scaling OOD detection towards a large semantic
space.

We also explore an alternative setting where we fix the total number of
training images, as we vary the number of classes $C$. In this setting, the
model is trained on fewer images per class as the number of classes increases,
making the problem even more challenging. We report those results in Appendix
B.1. Overall, MOS remains less sensitive to the number of classes compared to
the MSP baseline.

Figure 6: OOD detection performance comparison between MOS with different
grouping strategies and the MSP baseline on 4 OOD datasets. (top: AUROC;
bottom: FPR95).

####  4.3.3 MOS with Different Grouping Strategies

In this ablation, we contrast the performance of three different grouping
strategies described in Section 3.3. For a fair comparison, we use the number
of groups $K=8$ for all methods, since ImageNet taxonomy has 8 super-classes.
For feature clustering, we first extract the feature vector from the
penultimate layer of the pre-trained BiT-S model for each training image. The
feature representation for each category is the average feature vector among
all images in that category. We then perform a K-Means clustering on the 1,000
categorical feature vectors (one for each class) with $K=8$. For random
grouping, we randomly split 1,000 classes into 8 groups with equal sizes (125
classes each).

We compare the performance of MOS under different grouping strategies in
Figure 6. We observe that feature clustering works substantially better than
the MSP baseline [16] while maintaining similar in-distribution classification
accuracy (-0.16%) to the taxonomy-based grouping. Interestingly, random
grouping achieves better performance than the MSP baseline [16] on 3 out of 4
OOD datasets. However, we do observe a drop of in-distribution classification
accuracy (-0.98%) using random grouping, compared to the taxonomy-based
grouping. We argue that feature clustering is a viable strategy when taxonomy
is unavailable, as it outperforms MSP by 18.2% (FPR95) on average. We
additionally report how different numbers of groups $K$ affect the OOD
detection performance for all three grouping strategies in Appendix B.2.

Figure 7: Effect of using different pre-trained feature extractors. The x-axis
indicates feature extractors with larger capacities from left to right. Only
the top FC layer is fine-tuned in all experiments. Both OOD detection (bars)
and image classification (dashed lines) benefit from improved feature
extractors.

####  4.3.4 MOS with Different Feature Extractors

We investigate how the performance of OOD detection changes as we employ
different pre-trained feature extractors. In Figure 7, we compare the
performance of using a family of 5 feature extractors (in increasing size):
BiT-S-R50x1, BiT-S-R101x1, BiT-S-R50x3, BiT-S-R152x2, BiT-
S-R101x3444https://github.com/google-research/big_transfer. All models are
ResNetv2 architectures with varying depths and width factors. It is important
to note that since we fix the entire backbone and only fine-tune the last FC
layer, this ablation is about the effect of the quality of feature extractors
rather than model capacities.

As we use feature extractors trained on larger capacities, the classification
accuracy increases, with comparable performance between using the flat vs.
group softmax. Overall the OOD detection performance improves as the capacity
of feature extractors increases. More importantly, MOS consistently
outperforms MSP [16] in all cases. These results suggest that using pre-
trained models with better feature representations will not only improve
classification accuracy but also benefit OOD detection performance.

Figure 8: Effect of fine-tuning different numbers of residual blocks in BiT-
S-R101x1. We show both OOD detection (bars) and image classification (dashed
lines) performance.

####  4.3.5 MOS with Varying Fine-tuning Capacities

In this ablation, we explore the efficacy of fine-tuning more layers.
Concretely, we go beyond the FC layer and fine-tune different numbers of
residual blocks in BiT-S-R101x1. Figure 8 shows the classification accuracy
and OOD detection performance under different fine-tuning capacities.
Noticeably, MOS consistently outperforms MSP [16] in OOD detection under all
fine-tuning capacities. As expected, we observe that fine-tuning more layers
leads to better classification accuracy. However, increasing the number of
fine-tuning layers would adversely affect OOD detection in some cases. We
hypothesize that fine-tuning with more layers will result in more label-
overfitted predictions, and undesirably produce higher confidence scores for
OOD data. This suggests that only fine-tuning the top FC layers is not only
computationally efficient but also in fact desirable for OOD detection
performance.

##  5 Related Work

##### OOD Detection with Pre-trained Models

Hendrycks and Gimpel [16] establish a common baseline for OOD detection by
using maximum softmax probability (MSP). Several works attempt to improve the
OOD uncertainty estimation by using ODIN score [29], deep ensembles [24],
Mahalanobis distance-based confidence score [27], generalized ODIN score [20],
and energy score [32]. Lin et al. [30] propose a dynamic OOD inference
framework that improved the computational efficiency. However, previous
methods driven by small datasets are suboptimal in a large-scale setting. In
contrast, MOS scales better with large label space, outperforming existing
methods by a large margin.

##### OOD Detection with Model Fine-tuning

An orthogonal line of work explores training with auxiliary outlier data for
model regularization [3, 11, 34, 36, 44, 32]. Auxiliary outlier data can
either be realistic images [17, 36, 38, 32, 4] or synthetic images generated
by GANs [26]. Several loss functions have been designed to regularize model
predictions of the auxiliary outlier data towards uniform distributions [26],
a background class for OOD data [4, 36], or higher energies [32]. In this
work, our model is fine-tuned only on in-distribution data, as we do not
assume the availability of auxiliary outlier data. Different from previous
settings, it can be prohibitive to construct an auxiliary outlier dataset in
large-scale image classification, since the in-distribution data has a much
wider coverage of concepts.

##### Generative Modeling Based OOD Detection

Generative models [21, 45, 41, 10, 46] estimate the probability density of
input data and can thus be directly utilized as OOD detectors with high
density indicating in-distribution and low density indicating out-of-
distribution. However, as shown in [37], deep generative models can
undesirably assign a high likelihood to OOD data. Several strategies have been
proposed to mitigate this issue, such as improved metrics [5], likelihood
ratio [40, 43], and modified training techniques [17]. In this work, we mainly
focus on the discriminative-based approaches. It is important to note that
generative models [19] can be prohibitively challenging to train and optimize
on large-scale real-world datasets.

##### OOD Detection for Large-scale Classification

Several works make pioneering efforts in large-scale OOD detection. Roady et
al. [42] sample half of the classes from ImageNet-1k as in-distribution data,
and evaluate the other half as OOD test data. They use a one-vs-rest training
strategy and background class regularization, which requires access to an
auxiliary dataset. KL matching was employed as the OOD scoring function in
[15]. In this work, we propose a novel group-based solution that scales more
effectively and efficiently for large-scale OOD detection. We also perform
evaluations on more diverse real-world OOD datasets and conduct thorough
ablations that improve understandings of the problem and solutions in many
aspects.

##### Learning with Hierarchical Labels

The hierarchical structure of the class categories has been utilized for
efficient inference [9, 31], improved classification accuracy [7], and
stronger object detection performance [39]. Some works aim to learn a label
tree structure when taxonomy is unavailable [2, 9, 31]. As a typical
hierarchy, group-based learning has been widely adopted in image
classification tasks [18, 1, 50, 48, 12]. Recently, a group softmax classifier
is proposed to tackle the problem of long-tail object detection, where
categories are grouped according to the number of training instances [28]. We
contribute to this field by showing the promise of using a group label
structure for effective OOD detection.

##  6 Conclusion

In this paper, we propose a group-based OOD detection framework, along with a
novel OOD scoring function, MOS, that effectively scales the OOD detection to
a real-world setting with a large label space. We curate four diverse OOD
evaluation datasets that allow future research to evaluate OOD detection
methods in a large-scale setting. Extensive experiments show our group-based
framework can significantly improve the performance of OOD detection in this
large-scale setting compared to existing approaches. We hope our research can
raise more attention to expand the view of OOD detection from small benchmarks
to a large-scale real-world setting.

## References

  * [1] Karim Ahmed, Mohammad Haris Baig, and Lorenzo Torresani.  Network of experts for large-scale image categorization.  In European Conference on Computer Vision, pages 516–532. Springer, 2016.
  * [2] Samy Bengio, Jason Weston, and David Grangier.  Label embedding trees for large multi-class tasks.  In Advances in Neural Information Processing Systems, pages 163–171, 2010.
  * [3] Petra Bevandić, Ivan Krešo, Marin Oršić, and Siniša Šegvić.  Discriminative out-of-distribution detection for semantic segmentation.  arXiv preprint arXiv:1808.07703, 2018.
  * [4] Jiefeng Chen, Yixuan Li, Xi Wu, Yingyu Liang, and Somesh Jha.  Robust out-of-distribution detection via informative outlier mining.  arXiv preprint arXiv:2006.15207, 2020.
  * [5] Hyunsun Choi and Eric Jang.  Generative ensembles for robust anomaly detection.  2018\.
  * [6] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi.  Describing textures in the wild.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3606–3613, 2014.
  * [7] Jia Deng, Nan Ding, Yangqing Jia, Andrea Frome, Kevin Murphy, Samy Bengio, Yuan Li, Hartmut Neven, and Hartwig Adam.  Large-scale object classification using label relation graphs.  In European conference on computer vision, pages 48–64. Springer, 2014.
  * [8] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * [9] Jia Deng, Sanjeev Satheesh, Alexander C Berg, and Fei Li.  Fast and balanced: Efficient label tree learning for large scale object recognition.  In Advances in Neural Information Processing Systems, pages 567–575, 2011.
  * [10] Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio.  Density estimation using real NVP.  In 5th International Conference on Learning Representations, ICLR 2017, 2017.
  * [11] Yonatan Geifman and Ran El-Yaniv.  Selectivenet: A deep neural network with an integrated reject option.  In International Conference on Machine Learning, pages 2151–2159. PMLR, 2019.
  * [12] Sam Gross, Marc’Aurelio Ranzato, and Arthur Szlam.  Hard mixtures of experts for large scale weakly supervised vision.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6865–6873, 2017.
  * [13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Identity mappings in deep residual networks.  In European conference on computer vision, pages 630–645. Springer, 2016.
  * [14] Matthias Hein, Maksym Andriushchenko, and Julian Bitterwolf.  Why relu networks yield high-confidence predictions far away from the training data and how to mitigate the problem.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 41–50, 2019.
  * [15] Dan Hendrycks, Steven Basart, Mantas Mazeika, Mohammadreza Mostajabi, Jacob Steinhardt, and Dawn Song.  A benchmark for anomaly segmentation.  arXiv preprint arXiv:1911.11132, 2019.
  * [16] Dan Hendrycks and Kevin Gimpel.  A baseline for detecting misclassified and out-of-distribution examples in neural networks.  In 5th International Conference on Learning Representations, ICLR 2017, 2017.
  * [17] Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich.  Deep anomaly detection with outlier exposure.  In International Conference on Learning Representations, 2018.
  * [18] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.  Distilling the knowledge in a neural network.  arXiv preprint arXiv:1503.02531, 2015.
  * [19] Tobias Hinz, Stefan Heinrich, and Stefan Wermter.  Generating multiple objects at spatially distinct locations.  In International Conference on Learning Representations, 2018.
  * [20] Yen-Chang Hsu, Yilin Shen, Hongxia Jin, and Zsolt Kira.  Generalized odin: Detecting out-of-distribution image without learning from out-of-distribution data.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10951–10960, 2020.
  * [21] Diederik P. Kingma and Max Welling.  Auto-encoding variational bayes.  In Yoshua Bengio and Yann LeCun, editors, 2nd International Conference on Learning Representations, ICLR 2014.
  * [22] Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby.  Big transfer (bit): General visual representation learning.  In ECCV 2020, 2020.
  * [23] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * [24] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell.  Simple and scalable predictive uncertainty estimation using deep ensembles.  In Advances in neural information processing systems, pages 6402–6413, 2017.
  * [25] Yann LeCun, Corinna Cortes, and CJ Burges.  Mnist handwritten digit database.  ATT Labs [Online]. Available: http://yann.lecun.com/exdb/mnist, 2, 2010.
  * [26] Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin.  Training confidence-calibrated classifiers for detecting out-of-distribution samples.  In International Conference on Learning Representations, 2018.
  * [27] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin.  A simple unified framework for detecting out-of-distribution samples and adversarial attacks.  In Advances in Neural Information Processing Systems, pages 7167–7177, 2018.
  * [28] Yu Li, Tao Wang, Bingyi Kang, Sheng Tang, Chunfeng Wang, Jintao Li, and Jiashi Feng.  Overcoming classifier imbalance for long-tail object detection with balanced group softmax.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10991–11000, 2020.
  * [29] Shiyu Liang, Yixuan Li, and Rayadurgam Srikant.  Enhancing the reliability of out-of-distribution image detection in neural networks.  In 6th International Conference on Learning Representations, ICLR 2018, 2018.
  * [30] Ziqian Lin, Sreya Dutta, and Yixuan Li.  Mood: Multi-level out-of-distribution detection.  Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [31] Baoyuan Liu, Fereshteh Sadeghi, Marshall Tappen, Ohad Shamir, and Ce Liu.  Probabilistic label trees for efficient large scale image classification.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 843–850, 2013.
  * [32] Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li.  Energy-based out-of-distribution detection.  Advances in Neural Information Processing Systems, 2020.
  * [33] James MacQueen et al.  Some methods for classification and analysis of multivariate observations.  In Proceedings of the fifth Berkeley symposium on mathematical statistics and probability, volume 1, pages 281–297. Oakland, CA, USA, 1967\.
  * [34] Andrey Malinin and Mark Gales.  Predictive uncertainty estimation via prior networks.  In Advances in Neural Information Processing Systems, pages 7047–7058, 2018.
  * [35] George A Miller.  Wordnet: a lexical database for english.  Communications of the ACM, 38(11):39–41, 1995.
  * [36] Sina Mohseni, Mandar Pitale, JBS Yadawa, and Zhangyang Wang.  Self-supervised learning for generalizable out-of-distribution detection.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 5216–5223, 2020.
  * [37] Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan.  Do deep generative models know what they don’t know?  In International Conference on Learning Representations, 2018.
  * [38] Aristotelis-Angelos Papadopoulos, Mohammad Reza Rajati, Nazim Shaikh, and Jiamian Wang.  Outlier exposure with confidence control for out-of-distribution detection.  arXiv preprint arXiv:1906.03509, 2019.
  * [39] Joseph Redmon and Ali Farhadi.  Yolo9000: better, faster, stronger.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7263–7271, 2017.
  * [40] Jie Ren, Peter J Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark Depristo, Joshua Dillon, and Balaji Lakshminarayanan.  Likelihood ratios for out-of-distribution detection.  In Advances in Neural Information Processing Systems, pages 14680–14691, 2019.
  * [41] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra.  Stochastic backpropagation and approximate inference in deep generative models.  In International conference on machine learning, pages 1278–1286. PMLR, 2014.
  * [42] Ryne Roady, Tyler L. Hayes, Ronald Kemker, Ayesha Gonzales, and Christopher Kanan.  Are out-of-distribution detection methods effective on large-scale datasets?, 2019.
  * [43] Joan Serrà, David Álvarez, Vicenç Gómez, Olga Slizovskaia, José F. Núñez, and Jordi Luque.  Input complexity and out-of-distribution detection with likelihood-based generative models.  In International Conference on Learning Representations, 2020.
  * [44] Akshayvarun Subramanya, Suraj Srinivas, and R Venkatesh Babu.  Confidence estimation in deep neural networks via density modelling.  arXiv preprint arXiv:1707.07013, 2017.
  * [45] Esteban G Tabak and Cristina V Turner.  A family of nonparametric density estimation algorithms.  Communications on Pure and Applied Mathematics, 66(2):145–164, 2013\.
  * [46] Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al.  Conditional image generation with pixelcnn decoders.  In Advances in neural information processing systems, pages 4790–4798, 2016.
  * [47] Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie.  The inaturalist species classification and detection dataset.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8769–8778, 2018.
  * [48] David Warde-Farley, Andrew Rabinovich, and Dragomir Anguelov.  Self-informed neural network structure learning.  arXiv preprint arXiv:1412.6563, 2014.
  * [49] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba.  Sun database: Large-scale scene recognition from abbey to zoo.  In 2010 IEEE computer society conference on computer vision and pattern recognition, pages 3485–3492. IEEE, 2010.
  * [50] Zhicheng Yan, Hao Zhang, Robinson Piramuthu, Vignesh Jagadeesh, Dennis DeCoste, Wei Di, and Yizhou Yu.  Hd-cnn: hierarchical deep convolutional neural networks for large scale visual recognition.  In Proceedings of the IEEE international conference on computer vision, pages 2740–2748, 2015.
  * [51] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba.  Places: A 10 million image database for scene recognition.  IEEE transactions on pattern analysis and machine intelligence, 40(6):1452–1464, 2017.

MOS: Towards Scaling Out-of-distribution Detection for Large Semantic Space

(Supplementary Material)

##  Appendix A Selected Categories in OOD Datasets

To evaluate our approach, we consider a diverse collection of OOD test
datasets, spanning various domains. We carefully curate the OOD evaluation
benchmarks to make sure concepts in these datasets do not overlap with
ImageNet-1k [8]. Below we provide the list of concepts chosen for each OOD
dataset, including iNaturalist [47], SUN [49], Places365 [51], and Textures
[6]. We hope the information would allow future research to reproduce our
results.

##### iNaturalist

Coprosma lucida, Cucurbita foetidissima, Mitella diphylla, Selaginella
bigelovii, Toxicodendron vernix, Rumex obtusifolius, Ceratophyllum demersum,
Streptopus amplexifolius, Portulaca oleracea, Cynodon dactylon, Agave
lechuguilla, Pennantia corymbosa, Sapindus saponaria, Prunus serotina,
Chondracanthus exasperatus, Sambucus racemosa, Polypodium vulgare, Rhus
integrifolia, Woodwardia areolata, Epifagus virginiana, Rubus idaeus, Croton
setiger, Mammillaria dioica, Opuntia littoralis, Cercis canadensis, Psidium
guajava, Asclepias exaltata, Linaria purpurea, Ferocactus wislizeni, Briza
minor, Arbutus menziesii, Corylus americana, Pleopeltis polypodioides,
Myoporum laetum, Persea americana, Avena fatua, Blechnum discolor, Physocarpus
capitatus, Ungnadia speciosa, Cercocarpus betuloides, Arisaema dracontium,
Juniperus californica, Euphorbia prostrata, Leptopteris hymenophylloides, Arum
italicum, Raphanus sativus, Myrsine australis, Lupinus stiversii, Pinus
echinata, Geum macrophyllum, Ripogonum scandens, Echinocereus triglochidiatus,
Cupressus macrocarpa, Ulmus crassifolia, Phormium tenax, Aptenia cordifolia,
Osmunda claytoniana, Datura wrightii, Solanum rostratum, Viola adunca,
Toxicodendron diversilobum, Viola sororia, Uropappus lindleyi, Veronica
chamaedrys, Adenocaulon bicolor, Clintonia uniflora, Cirsium scariosum, Arum
maculatum, Taraxacum officinale officinale, Orthilia secunda, Eryngium
yuccifolium, Diodia virginiana, Cuscuta gronovii, Sisyrinchium montanum, Lotus
corniculatus, Lamium purpureum, Ranunculus repens, Hirschfeldia incana, Phlox
divaricata laphamii, Lilium martagon, Clarkia purpurea, Hibiscus moscheutos,
Polanisia dodecandra, Fallugia paradoxa, Oenothera rosea, Proboscidea
louisianica, Packera glabella, Impatiens parviflora, Glaucium flavum, Cirsium
andersonii, Heliopsis helianthoides, Hesperis matronalis, Callirhoe pedata,
Crocosmia $\times$ crocosmiiflora, Calochortus albus, Nuttallanthus
canadensis, Argemone albiflora, Eriogonum fasciculatum, Pyrrhopappus
pauciflorus, Zantedeschia aethiopica, Melilotus officinalis, Peritoma arborea,
Sisyrinchium bellum, Lobelia siphilitica, Sorghastrum nutans, Typha
domingensis, Rubus laciniatus, Dichelostemma congestum, Chimaphila maculata,
Echinocactus texensis

##### SUN

badlands, bamboo forest, bayou, botanical garden, canal (natural), canal
(urban), catacomb, cavern (indoor), corn field, creek, crevasse, desert
(sand), desert (vegetation), field (cultivated), field (wild), fishpond,
forest (broadleaf), forest (needleleaf), forest path, forest road, hayfield,
ice floe, ice shelf, iceberg, islet, marsh, ocean, orchard, pond, rainforest,
rice paddy, river, rock arch, sky, snowfield, swamp, tree farm, trench,
vineyard, waterfall (block), waterfall (fan), waterfall (plunge), wave, wheat
field, herb garden, putting green, ski slope, topiary garden, vegetable
garden, formal garden

##### Places

badlands, bamboo forest, canal (natural), canal (urban), corn field, creek,
crevasse, desert (sand), desert (vegetation), desert road, field (cultivated),
field (wild), field road, forest (broadleaf), forest path, forest road, formal
garden, glacier, grotto, hayfield, ice floe, ice shelf, iceberg, igloo, islet,
japanese garden, lagoon, lawn, marsh, ocean, orchard, pond, rainforest, rice
paddy, river, rock arch, ski slope, sky, snowfield, swamp, swimming hole,
topiary garden, tree farm, trench, tundra, underwater (ocean deep), vegetable
garden, waterfall, wave, wheat field

##### Textures

all images in this dataset

##  Appendix B More Ablation Studies

###  B.1 MOS with Increasing Numbers of Classes (A More Challenging Setting)

Figure 9: OOD detection performance of MOS (blue) and the MSP baseline (gray).
MOS exhibits more stabilized performance as the number of in-distribution
classes increases. For each OOD dataset, we show AUROC (top) and FPR95 (
bottom). Different from Figure 5, we fix the number of _total training images_
instead of the number of _training images in each category_ in this
experiment.

In Section 4.3.2, we increase the number of in-distribution classes while
fixing the number of _training images in each class_ and observe the
degradation of OOD detection performance. Here we investigate an alternative
setting where we fix the number of _total training images_ to be $35,000$, as
we increase the number of classes
$C\in\\{50,200,300,400,500,600,700,800,900,1000\\}$. For each $C$, we create
training data by first randomly sampling $C$ labels from the entire 1,000
classes in ImageNet-1k, and then sampling 35,000 / $C$ images for each chosen
label. In Figure 9, we show the OOD detection performance with varying numbers
of in-distribution classes $C$.

Figure 10: WordNet hierarchy. Super-classes of ImageNet-1k are based on the
leaf nodes (in ellipses) except for misc. The super-class of misc contains 3
leaf nodes: abstract entity, matter, and location.

This is a more challenging setting because when the number of classes
increases, there will be fewer training images for each class. Unsurprisingly,
the performance degradation of OOD detection is more severe in this setting.
For instance, on the iNaturalist OOD dataset, the FPR95 performance of MSP
[16] degrades by 56.74% when the number of classes increases from 50 to 1,000,
while the corresponding degradation is only 42.34% in the previous setting.
Importantly, MOS remains much less sensitive to the change of the number of
in-distribution classes compared to the MSP baseline (without grouping). In
particular, on the Places OOD dataset, the FPR95 performance drops from 14.45%
to 88.02% using MSP, while MOS degrades by only 38.73%.

###  B.2 MOS with Varying Numbers of Groups

Level |  Number of Groups |  Grouping Strategy | iNaturalist | SUN | Places | Textures | Average
---|---|---|---|---|---|---|---
AUROC | FPR95 | AUROC | FPR95 | AUROC | FPR95 | AUROC | FPR95 | AUROC | FPR95
$\uparrow$ | $\downarrow$ | $\uparrow$ | $\downarrow$ | $\uparrow$ | $\downarrow$ | $\uparrow$ | $\downarrow$ | $\uparrow$ | $\downarrow$
-3 | 2 | taxonomy | 97.66 | 11.20 | 85.27 | 65.86 | 82.21 | 70.31 | 79.06 | 63.88 | 86.05 | 52.81
feature clustering | 94.91 | 33.12 | 87.87 | 57.02 | 84.31 | 66.23 | 84.58 | 56.06 | 87.92 | 53.11
random grouping | 91.12 | 46.82 | 79.28 | 78.73 | 78.64 | 76.50 | 79.02 | 63.53 | 82.02 | 66.40
-2 | 3 | taxonomy | 97.21 | 15.78 | 92.28 | 40.08 | 89.35 | 49.74 | 81.00 | 60.64 | 89.96 | 41.56
feature clustering | 94.57 | 33.58 | 87.23 | 57.18 | 83.60 | 65.34 | 83.06 | 57.23 | 87.12 | 53.33
random grouping | 90.75 | 47.55 | 76.57 | 83.00 | 75.89 | 81.33 | 80.40 | 61.93 | 80.90 | 68.45
-1 | 6 | taxonomy | 97.16 | 16.16 | 92.07 | 40.28 | 89.12 | 49.53 | 81.34 | 60.27 | 89.92 | 41.56
feature clustering | 90.68 | 53.49 | 82.95 | 72.24 | 79.48 | 75.06 | 81.78 | 62.50 | 83.72 | 65.82
random grouping | 91.55 | 44.73 | 79.11 | 78.99 | 78.31 | 76.17 | 80.93 | 62.30 | 82.48 | 65.55
0 | 8 | taxonomy | 98.15 | 9.28 | 92.01 | 40.63 | 89.06 | 49.54 | 81.23 | 60.43 | 90.11 | 39.97
feature clustering | 93.29 | 41.13 | 84.84 | 63.15 | 81.84 | 66.33 | 80.62 | 64.61 | 85.15 | 58.81
random grouping | 90.63 | 47.47 | 76.95 | 81.89 | 76.65 | 78.47 | 79.02 | 65.25 | 80.81 | 68.27
1 | 36 | taxonomy | 95.85 | 23.73 | 90.51 | 47.53 | 87.74 | 52.51 | 88.47 | 45.55 | 90.64 | 42.33
feature clustering | 91.22 | 47.73 | 79.25 | 79.48 | 76.53 | 79.00 | 82.81 | 61.72 | 82.45 | 66.98
random grouping | 91.01 | 46.96 | 79.66 | 77.79 | 79.36 | 74.35 | 78.91 | 68.72 | 82.24 | 66.96
2 | 85 | taxonomy | 92.22 | 46.19 | 88.07 | 59.41 | 86.02 | 60.28 | 85.40 | 57.70 | 87.93 | 55.90
feature clustering | 93.13 | 40.07 | 81.05 | 77.06 | 78.33 | 76.44 | 82.28 | 64.57 | 83.70 | 64.54
random grouping | 90.58 | 50.04 | 78.75 | 81.21 | 78.81 | 77.23 | 76.95 | 76.17 | 81.27 | 71.16
3 | 225 | taxonomy | 90.35 | 57.38 | 85.19 | 71.72 | 83.57 | 69.99 | 81.40 | 72.27 | 85.13 | 67.84
feature clustering | 91.49 | 48.90 | 79.59 | 82.16 | 78.06 | 79.97 | 79.40 | 75.09 | 82.14 | 71.53
random grouping | 89.66 | 56.81 | 77.55 | 84.73 | 78.16 | 79.61 | 75.07 | 82.43 | 80.11 | 75.90
4 | 416 | taxonomy | 89.18 | 63.48 | 82.34 | 80.60 | 81.30 | 76.88 | 78.37 | 81.17 | 82.80 | 75.53
feature clustering | 91.66 | 47.91 | 80.40 | 79.40 | 79.12 | 77.26 | 78.24 | 80.00 | 82.36 | 71.14
random grouping | 88.68 | 61.29 | 76.94 | 86.01 | 77.67 | 81.41 | 73.24 | 86.91 | 79.13 | 78.91
5 | 642 | taxonomy | 88.10 | 67.11 | 80.07 | 84.04 | 79.65 | 79.89 | 75.17 | 87.22 | 80.75 | 79.57
feature clustering | 90.45 | 55.74 | 79.83 | 82.89 | 79.22 | 79.18 | 75.77 | 86.01 | 81.32 | 75.96
random grouping | 88.31 | 63.92 | 77.09 | 85.94 | 77.60 | 81.69 | 72.16 | 89.08 | 78.79 | 80.16
6 | 789 | taxonomy | 87.39 | 69.61 | 78.57 | 85.73 | 78.64 | 81.04 | 73.57 | 89.29 | 79.54 | 81.42
feature clustering | 89.81 | 59.68 | 78.78 | 84.75 | 78.31 | 80.82 | 74.66 | 88.65 | 80.39 | 78.48
random grouping | 88.07 | 65.12 | 76.91 | 86.65 | 77.52 | 82.15 | 71.84 | 89.84 | 78.59 | 80.94
Table 2: Effect of different numbers of groups on OOD detection performance
for 3 grouping strategies (taxonomy, feature clustering, and random grouping).
Level 0 represents the level of super-classes in the taxonomy tree (main
setting). Positive levels indicate splitting the super-classes into more
groups (tracing down the taxonomy tree), while negative levels indicate
merging the super-classes into fewer groups (tracing up the taxonomy tree).

In this ablation we investigate how different numbers of groups $K$ affect the
OOD detection performance of MOS under three grouping strategies: (1)
taxonomy, (2) feature clustering, and (3) random grouping. For taxonomy-based
grouping, in order to increase the number of groups, we split the nodes of
each super-class into their descendants in the label tree and map the 1,000
classes into one of the descendants instead of the super-classes themselves;
in order to decrease the number of groups, we merge some of the super-classes
into one group based on Figure 10. Specifically, we construct 10 taxonomy
levels with increasing numbers of groups based on the label tree in the
following way:

##### Level -3

There are 2 groups in Level -3: {animal, plant, fungus}, {artifact, natural
object, geological formation, person, misc}.

##### Level -2

There are 3 groups in Level -2: {animal, plant, fungus}, {artifact, natural
object}, {geological formation, person, misc}.

##### Level -1

There are 6 groups in Level -1: {animal, plant, fungus}, {artifact}, {natural
object}, {geological formation}, {person}, {misc}.

##### Level 0

This is the level of 8 super-classes (main setting).

##### Level 1$\sim$6

Groups in Level $i$ are direct children of the nodes in Level $(i-1)$.

For feature clustering and random grouping, we set the numbers of groups to be
equal to the corresponding numbers at each of the taxonomy levels for fair
comparisons.

As shown in Table 2, for taxonomy-based grouping, the performance of OOD
detection is almost optimal when the number of groups is 8 (Level 0), and
further increasing or decreasing the number of groups will not lead to
improved performance. Moreover, taxonomy-based grouping outperforms feature
clustering and random grouping when $K$ is small and mildly large. However,
feature clustering surpasses taxonomy-based grouping when the number of groups
is sufficiently large. We hypothesize that as we trace down the label tree,
the numbers of categories in each group become more imbalanced, which could
adversely impact the performance of OOD detection using taxonomy-based
grouping.

##  Appendix C AUROC Curves

Figure 11 shows the AUROC curves of MOS and MSP for OOD detection. All
settings and training details are the same as in Table 1. The gray curve
corresponds to the MSP baseline [16], while the blue curve corresponds to MOS
with taxonomy-based grouping. We observe huge gaps between the gray and the
blue AUROC curves on all OOD datasets. For instance, when TPR = 95%, the FPR
can be reduced from 63.69% to 9.28% on the iNaturalist OOD dataset.

Figure 11: AUROC curves of MOS (blue) and MSP (gray) on four OOD datasets.

##  Appendix D AUPR Results

OOD Dataset | Method | AUROC | FPR95 | AUPR
---|---|---|---|---
$\uparrow$ | $\downarrow$ | $\uparrow$
iNaturalist | MSP | 87.59 | 63.69 | 97.23
ODIN | 89.36 | 62.69 | 97.76
Mahalanobis | 46.33 | 96.34 | 81.14
Energy | 88.48 | 64.91 | 97.58
KL Matching | 93.00 | 27.36 | 97.93
MOS (ours) | 98.15 | 9.28 | 99.62
SUN | MSP | 78.34 | 79.98 | 94.45
ODIN | 83.92 | 71.67 | 96.26
Mahalanobis | 65.20 | 88.43 | 88.81
Energy | 85.32 | 65.33 | 96.57
KL Matching | 78.72 | 67.52 | 94.10
MOS (ours) | 92.01 | 40.63 | 98.17
Places | MSP | 76.76 | 81.44 | 94.15
ODIN | 80.67 | 76.27 | 95.35
Mahalanobis | 64.46 | 89.75 | 88.85
Energy | 81.37 | 73.02 | 95.49
KL Matching | 76.49 | 72.61 | 93.61
MOS (ours) | 89.06 | 49.54 | 97.36
Textures | MSP | 74.45 | 82.73 | 95.65
ODIN | 76.30 | 81.31 | 96.12
Mahalanobis | 72.10 | 52.23 | 91.89
Energy | 75.79 | 80.87 | 96.05
KL Matching | 87.07 | 49.70 | 97.97
MOS (ours) | 81.23 | 60.43 | 96.65
Average | MSP | 79.29 | 76.96 | 95.37
ODIN | 82.56 | 72.99 | 96.37
Mahalanobis | 62.02 | 81.69 | 87.67
Energy | 82.74 | 71.03 | 96.42
KL Matching | 83.82 | 54.30 | 95.90
MOS (ours) | 90.11 | 39.97 | 97.95
Table 3: Main results with AUPR. Experimental setups are the same as in Table
1.

In Table 3 we report the area under the precision-recall curve (AUPR)
complementing the AUROC and FPR95 results in Table 1. AUPR is an informative
metric in the presence of class imbalance, which is common in OOD detection.
Again, MOS demonstrates state-of-the-art performance in terms of AUPR.

##  Appendix E Others Scores for All In-distribution Groups and OOD Datasets

Figure 12 and Figure 13 show average others scores for 8 in-distribution
groups and 4 OOD datasets, respectively. For in-distribution groups, others
scores are averaged among all validation images in each group in ImageNet-1k;
for OOD datasets, others scores are averaged among all sampled images in the
curated datasets.

These histograms provide visual justifications for our method MOS: in-
distribution images will have low others scores in at least one group (shown
in red boxes), while out-of-distribution images will have high others scores
in all 8 groups. Therefore, MOS is effective in distinguishing between in- vs.
out-of-distribution data.

Figure 12: Average others scores for all in-distribution groups. Red boxes
indicate the corresponding groups these images belong to. Figure 13: Average
others scores for all OOD datasets


# DarkGS: Learning Neural Illumination and 3D Gaussians Relighting for Robotic
Exploration in the Dark

Tianyi Zhang1, Kaining Huang1, Weiming Zhi1, Matthew Johnson-Roberson1
1Authors are with the Robotics Institute, School of Computer Science, Carnegie
Mellon University, Pittsburgh, PA 15213, USA
<EMAIL_ADDRESS>
###### Abstract

Humans have the remarkable ability to construct consistent mental models of an
environment, even under limited or varying levels of illumination. We wish to
endow robots with this same capability. In this paper, we tackle the challenge
of constructing a photorealistic scene representation under poorly illuminated
conditions and with a moving light source. We approach the task of modeling
illumination as a learning problem, and utilize the developed illumination
model to aid in scene reconstruction. We introduce an innovative framework
that uses a data-driven approach, _Neural Light Simulators_ (NeLiS), to model
and calibrate the camera-light system. Furthermore, we present DarkGS, a
method that applies NeLiS to create a relightable 3D Gaussian scene model
capable of real-time, photorealistic rendering from novel viewpoints. We show
the applicability and robustness of our proposed simulator and system in a
variety of real-world environments. Code released at
https://github.com/tyz1030/neuralight.git

##  I INTRODUCTION

Robots and autonomous vehicles have been routinely deployed in poorly
illuminated environments for critical missions and tasks such as exploration,
inspection, transportation, search and rescue, etc. (see Fig. 1(a)). Imaging
systems consisting of one or multiple RGB cameras and light sources are often
equipped on the robot to illuminate and sense the surrounding environment. The
streamed image sequence can be further used in downstream tasks, e.g.
navigation, mapping, and visualization, to boost the robot autonomy and human
understanding of the environment.

(a) Robots in the dark with onboard light sources

(b) Our work build 3D Gaussians and relight the scene in dark

Figure 1: Robotic imaging systems working in the dark consist of cameras and
light sources. Examples as shown in (a): Carla Simulator [1], Team CoStar in
SubT Challenge [2] and HoloOcean underwater robot simulator [3]. In this work,
we propose a pipeline that calibrates the camera-light system which helps
photorealistic scene reconstruction and relighting from images collected in
the dark.

Scene reconstruction, or the capability to create accurate internal
representations of the environment, is vital for robots operating in unknown
environments. Previous vision-only approaches largely rely on identifying
common feature points over a set of multiple-view images, and then minimizing
a reprojection error [4]. Such procedures like SfM (SfM) or SLAM (SLAM) also
estimate the camera poses of the images. Based on these camera poses, NeRF
(NeRF) [5] is capable of achieving photorealistic scene reconstruction by
optimizing a photometric loss between the representation and the images.
However, while achieving huge success in the graphics community, the
transition of objective function from reprojection error to photometric loss
has raised a new challenge to the robotics community: Can we still build
consistent scene representations with a moving light source on the robot
platform?

In this work, we identify _illumination-inconsistency_ as the main challenge
in building a photorealistic scene representation from images taken with a
moving light source. That is, as the robot operates in the environment, the
effect of moving light source results in images captured of the same region
appearing visually different. Handcrafted feature descriptors that serve as
backbone in SfM and SLAM [6, 7, 8] are designed to be invariant to changes in
pixel intensities. Such changes also have limited effects on the optimization
pipeline, which uses reprojection errors as objective functions. Hence, the
classic SfM methods are generally robust against _illumination-inconsistency_.
However, on the way towards photorealistic scene reconstruction and switching
to optimization of the photometric loss, we find that state-of-the-art NeRF
methods [5, 9] fail on the images taken in the dark with a moving light
source. Although variants such as RawNeRF [10] and Relightable 3DG [11] claim
to be able to handle dark images in RAW format and model the reflection
properties of the scene, they still fail to handle the varying illumination.

Concretely, the problem tackled in this paper is as such: Given a sequence of
images taken in poorly illuminated environments, with one major light source
moving with one camera as a rigid body, reconstruct the scene by minimizing
photometric loss and achieve photorealistic novel view image synthesis (Fig.
1(b)). Our contributions are as follows:

  1. 1.

A pipeline that consists of light source modeling, camera-light calibration,
building 3D Gaussians and scene relighting from illumination-inconsistent
images.

  2. 2.

NeLiS (NeLiS), a data-driven and physically interpretable illumination model
and software for light source modeling and calibration.

  3. 3.

DarkGS (DarkGS), a variant of the 3DGS (3DGS) model that builds photorealistic
scene representations under poorly illuminated conditions and relights the
scene with global illumination, based on COLMAP [12] and NeLiS results.

##  II RELATED WORK

###  II-A Extroceptive Sensor Calibration on Robots

Figure 2: Our proposed workflow: Images for camera-light calibration are first
collected at a calibration target. With NeLiS, we manually initialize the
parameters and then optimize the light model. The model can then be used to
build DarkGS, present the scene with learned or relighted illumination.
(Learning Gamma tone mapping is supported by NeLiS but not further discussed
in this paper.)

Autonomous robots are usually equipped with perceptual sensors such as cameras
and LiDARs. Taking camera calibration as an example, a calibration target with
AprilTag [13] or checkerboard pattern is often used. With observations of the
target from different perspectives, the focal length, center of projection,
distortion coefficient, and pose of the camera can be estimated [14]. Similar
target-based approaches have been used to calibrate LiDARs [15], radars [16]
and acoustic sensors [17] for downstream sensor fusion tasks [18]. Analogous
to sensor calibrations mentioned above, in this paper, we propose to use a
target consisting of AprilTags [13] and blank space for calibrating the light
in a camera-light system, including estimating the transformation between
camera’s and light’s coordinate system, RID (RID) and light fall-off curve of
the light source.

###  II-B Light Calibration

Existing methods use various kinds of customized calibration targets: [19][20]
propose to use a target of AprilTags and pins to infer the position of point
light source from the shadows of the pins. Alternatively, [21] [22] propose to
use a Lambertian sphere and estimate the light source parameters by learning
to reconstruct the sphere. Furthermore, [23] shows that the location of the
light source can also be recovered from the shadow of a sphere. With a
Lambertian plane and customized markers, [24] shows that it is possible to
calibrate a point light source with a single image. [25] shows the most
relevance to our work, which calibrates the pose of a light source and the
metric scale given the RID curve. However, the above work either assumes an
oversimplified model, e.g. a point light source with inverse square light
falloff, in a pure dark environment, or that the RID is given, which makes the
method less applicable. In this work, we use a minimum calibration target
design that consists of only AprilTags [13] and a Lambertian plane similar to
[25]. With NeLiS we show that modeling and learning RID, light falloff and
ambient light together with camera-light pose can improve the quality of image
reconstruction and make our calibration method more feasible for real-world
robot deployments.

###  II-C NeRF and 3D Gaussian Splatting

Based on the success of SfM [4], NeRF learns to represent scenes using a
continuous function, e.g. a MLP (MLP), and can achieve photorealistic novel-
view image synthesis. Popular variations of NeRF such as Mip-NeRF[26] and
Instant-NGP [27] have improved NeRF by rendering accuracy and speed. Recent
innovation introduced as 3DGS [9] has revolutionized this field by replacing
the neural network backbone with spatial Gaussians, allowing rendering speed
of 100+ fps while maintaining the accuracy and differentiability of the
system. For our robotic setup, the problem of varying illumination is
attempted by [28, 29] which model the physical property of the scene, e.g.
roughness, albedo, and normal, instead of the radiance as a constant. However,
these works oversimplify the light model as a Lambertian point light source,
co-centered with the camera, making it not applicable on many real robotic
platforms. Gaussianshader [30] and Relightable 3DG [11] introduce physical
properties into the 3DGS framework but model illumination as a constant that
does not change frame-by-frame. According to our experiments, none of the
above mentioned methods handles the illumination-inconsistancy issue on a
real-robot imaging system. In our proposed DarkGS, by modeling the physical
property of the scene and taking advantage of NeLiS, we can not only build a
consistent 3DGS from poorly illuminated images, but also relight the scene
with global illumination.

###  II-D Photometric Stereo

Photometric stereo refers to the estimation of the shape of an object with
multiple images taken from a single viewpoint with different lighting
orientations [31]. Early approaches confine this problem to known light source
and Lambertian surfaces with uniform albedo. Further work has allowed
spatially varying albedo [32] and unknown light [33]. However, our robotic
setup differs from this problem by the fact that our camera and light source
always move together as a rigid body. And we choose to tackle this problem
within the framework of 3DGS [9] which allows us to synthesize novel views
with photorealistic quality.

##  III METHODOLOGY

Our proposed workflow for building NeLiS and DarkGS is shown in Fig. 2. We
build NeLiS by calibrating the camera-light system at a planar calibration
target. By accounting for the calibration image sequence and hand measurement
of the light pose, an in-the-loop human operator can initialize the light
parameters using our intuitive GUI. Since the calibration target can be
localized by AprilTags [13], a shading model can render the image with the
given parameters to approximate the calibration image. Then, we run the
optimization to minimizes the photometric error of the calibration image
sequence and rendered image. After obtaining a calibrated NeLiS, the robot can
be deployed in a dark environment, using the collected images to build a
DarkGS, finetune the model, and relight the scene.

In this section, we will discuss the details of the NeLiS model in III-A,
DarkGS in III-B and NeLiS frontend software and workflow in III-C.

###  III-A Shading Model for NeLiS

Figure 3: Our shading model: (Left) In NeLiS, camera poses are localized by
AprilTags on the calibration target. The pose of light $R_{l}^{c}$ and
$t_{l}^{c}$, albedo $\mathbf{c}$ of the calibration target, ambient light $A$,
RID $\Phi_{\theta}$, and light falloff function $\Psi_{\tau}$ will be learned.
(Right) In building DarkGS, each Gaussian $g_{i}$ is modeled with an albedo
$\mathbf{c}_{i}$ and normal $\mathbf{n}_{i}$. The ambient light $A$ and scale
$s$ will also be optimized in this process.

The calibration data is taken by capturing photos at a calibration target from
different views while the light source moves with the camera as a rigid body.
The calibration target is a white plane with four AprilTags positioned as four
corners of a rectangle (as shown in Fig. 3). We attach the origin of the world
coordinate to the top-left corner of the calibration target. We assume that
the camera is precalibrated with distortions removed. One of the key problems
for NeLiS to solve is estimating the relative pose of light to the camera
$R_{l}^{c}\in{SO(3)}$ and $t_{l}^{c}\in\mathbb{R}^{3}$.

Given the true size of the calibration target, we can apply PnP (PnP) [34] to
extract $R_{c}^{w}\in SO(3)$ and $t_{c}^{w}\in\mathbb{R}^{3}$ which transform
points from the camera coordinate to the world coordinate. The position of the
light source is then given by
$\mathbf{o}_{l}=R_{c}^{w}t_{l}^{c}+t_{c}^{w}\in\mathbb{R}^{3}$. Since we align
the $z$ axis of the light coordinate with the centerline of the light, the
direction of the centerline can be denoted by
$\boldsymbol{\omega}_{l}=R_{c}^{w}R_{l}^{c}[0,0,1]^{\top}\in\mathbb{R}^{3}$.
Both $\mathbf{o}_{l}$ and $\boldsymbol{\omega}_{l}$ are in the world
coordinate frame.

We only use the area bounded by 4 AprilTags as the ROI (ROI) to do
calibrations. We assume that this area is a Lambertian plane and has the same
normal $\mathbf{n}\in\mathbb{R}^{3}$ and diffusive albedo
$\mathbf{c}\in\mathbb{R}^{\lambda}$ anywhere on the plane ($\lambda=1$ for
grayscale images and $\lambda=3$ for RGB). For each pixel in the ROI, we find
the intersection of the corresponding camera ray and the target plane in the
world coordinate system, denoted by $\mathbf{x}\in\mathbb{R}^{3}$. To infer
the incident radiance at $\mathbf{x}$, one needs to model the RID, light
falloff function, and ambient light.

####  III-A1 RID

RID is commonly modeled as a function of the angle between the centerline
$\boldsymbol{\omega}_{l}$ of the light and light ray
$\boldsymbol{\omega}_{\mathbf{x}}=\mathbf{x}-\mathbf{o}_{l}$. Previous work
assume that RID is given [25, 24, 22]. However, this assumption may not hold
for in-the-wild robot deployment. Instead, we remove this dependency by
learning a neural RID from calibration data:

$\Phi_{\theta}(\mathbf{x})=\text{MLP}_{\theta}(\cos^{-1}(\frac{\boldsymbol{\omega}_{\mathbf{x}}}{\lVert\boldsymbol{\omega}_{\mathbf{x}}\rVert_{2}}\cdot\boldsymbol{\omega}_{l}))$
(1)

here $\theta$ denotes the parameters of the MLP.

####  III-A2 Light Falloff Curve

Inverse square law is widely used to model light falloff, based on the
assumption of a point light source. However, when objects are closer to the
light source, the inverse square law starts to fail. We choose to model the
light falloff in the form of a Lorentzian function of the distance
$\lVert\boldsymbol{\omega}_{\mathbf{x}}\rVert_{2}$, as suggested by [35]:

$\Psi_{\tau}(\mathbf{x})=\frac{1}{\tau+\lVert\boldsymbol{\omega}_{\mathbf{x}}\rVert_{2}^{2}}$
(2)

Instead of estimating $\tau$ from hand measurement of the light surface [35],
we designate it to be a learnable parameter.

####  III-A3 Ambient Light

Imaging and lighting problems are often studied in a dark room to better
simplify and constrain the model by removing ambient light. However, a
perfectly dark space for calibration might not be accessible for real-world
robot deployment. Instead, we model ambient light as a learnable parameter
$A$. The incident radiance at the point $\mathbf{x}$ can thereby be modeled
as:

$I_{\mathbf{x}}=\Psi_{\tau}(\mathbf{x})\Phi_{\theta}(\mathbf{x})+A$ (3)

####  III-A4 BRDF

Although there are BRDF (BRDF) models capable of handling complex reflection
effects, they exceed the scope of this paper. We opt for the Lambertian
reflection model, eliminating the need to optimize any parameters in our BRDF.
We use
$f_{r}(\boldsymbol{\omega}_{\mathbf{x}},\mathbf{n})=\text{max}(\boldsymbol{\omega}_{\mathbf{x}}\cdot\mathbf{n},0)$
as our BRDF, giving the rendering equation:

$\hat{L}_{\mathbf{x}}=I_{\mathbf{x}}f_{r}(\boldsymbol{\omega}_{\mathbf{x}},\mathbf{n})\mathbf{c}$
(4)

With captured pixel intensity $L_{\mathbf{x}}$, we use L1 loss and formulate
the NeLiS optimization problem as:

$\min_{\theta,A,\tau,R_{l}^{c},t_{l}^{c},{\mathbf{c}}}\sum_{\mathbf{x}\in{\text{ROI}}}\|L_{\mathbf{x}}-\hat{L}_{\mathbf{x}}\|_{1}$
(5)

###  III-B Building DarkGS

Within the framework of 3DGS, we model the scene with a point cloud of
Gaussians $\mathbf{G}$ (as shown in Fig. 3 right). Each Gaussian ${g}_{i}$ in
the point cloud encompasses attributes including position $\mathbf{p}_{i}$,
covariance $\Sigma_{i}$, opacity $\alpha_{i}$, albedo $\mathbf{c}_{i}$ and
normal $\mathbf{n}_{i}$, that
${g}_{i}=\\{\mathbf{p}_{i},\Sigma_{i},\alpha_{i},\mathbf{c}_{i},\mathbf{n}_{i}\\}\in\mathbf{G}$.

Given $\mathbf{p}_{i}$, the incident radiance $I_{i}$ can be calculated by Eq.
3. With $\mathcal{N}$ ordered points for pixel $(u,v)$, the rendering equation
then becomes:

$\hat{L}_{u,v}=\sum_{i\in\mathcal{N}}I_{i}f_{r}(\boldsymbol{\omega}_{i},\mathbf{n}_{i})\mathbf{c}_{i}\alpha_{i}\prod_{j}^{i-1}(1-\alpha_{j})$
(6)

####  III-B1 Scale Recovery

The framework of 3DGS and its variants are often based on monocular SfM
solutions such as COLMAP [4]. While calibration using AprilTags recovers poses
in true metric scale with NeLiS, monocular SfM only gives up-to-scale poses
for building 3D Gaussians. Here, we introduce a scaling factor $s>0$ as a
learnable parameter so that we can obtain the positions with scale
$\mathbf{p}_{i}^{\prime}=s\mathbf{p}_{i}$. With captured pixel intensity
$L_{u,v}$, the 3D Gaussian can be built by solving the follow optimization
problem:

$\min_{A,\mathbf{G},s}\sum\|L_{u,v}-\hat{L}_{u,v}\|_{1}$ (7)

####  III-B2 Training Warm-Up

In some cases, a large discrepancy between the initial scale and the true
scale often leads to divergence and local minimum at the beginning of
optimization. We therefore propose a warm-up strategy to tackle this problem.
The overall idea behind this warm-up strategy is that we overwrite our light
pose to be co-centered and parallel with the camera. In this co-centered
configuration, the baseline between camera and light will be zero, so that the
scale difference between the baseline and the point cloud will have a minor
effect on shading at coarse level. Then, we gradually recover the pose to the
calibration result over the first $k$ iterations. During this process, Eq. 7
is being solved and $s$ will be optimized. We define a warm-up factor that
grows with iterations, so that in the $m^{th}$ iteration, this factor is
$\frac{m}{k}$. Here, we denote Lie exponential map and Lie logarithm map as
follows:

$\text{exp}(\cdot):\mathbb{R}^{3}\rightarrow SO(3),\ \
\text{log}(\cdot):SO(3)\rightarrow\mathbb{R}^{3}$ (8)

so that in warm-up stage we replace $R_{l}^{c}$ and $t_{l}^{c}$ with:

$\hat{R}_{l}^{c}=\text{exp}(\frac{m}{k}\text{log}(R_{l}^{c})),\ \
\hat{t}_{l}^{c}=\frac{m}{k}t_{l}^{c}$ (9)

####  III-B3 Relighting

Once the DarkGS model is built, we can relight the scene by replacing the
components in Eq. 3, i.e. $\Psi_{\tau}$, $\Phi_{\theta}$ and $A$, by carefully
designed values and functions. For example, replacing the MLP function in
$\Phi_{\theta}$ with a constant can create Lambertian illumination without a
strong pattern; replacing the light falloff function $\Psi_{\tau}$ with a
constant can simulate the illumination from a parallel light source.

###  III-C NeLiS frontend: Human-in-the-loop calibration

Although front-end software is not the main contribution of this paper, it
provides an interactive interface for NeLiS visualization, manual
initialization of parameters, and hyperparameter tuning, as partially shown in
Fig. 2 and elaborated on our github. We find that the NeLiS model is prone to
local minimum, especially when learning an MLP and camera-light pose from
scratch at the same time. In addition, optimizing MLP and albedo together
creates an under-constrained problem, often leading to numerical instability.
With our GUI, we first initialize the camera-light pose with a Gaussian
distribution as initial RID, and fine-tune the pose, albedo, and other
parameters. Then we train an MLP-based RID with pose and albedo freezed,
preventing divergence in concurrently learning multiple parameters. Once the
model has converged, we unfreeze all the learnable parameters and optimize
them together to reach the global minimum.

##  IV EXPERIMENTS

###  IV-A Experiments Setup

Figure 4: Our experiment setup: The imaging system is installed on a legged
robot platform (Unitree GO1). We use a FLIR machine vision camera to stream
the images in RAW format. The calibration target (as shown behind the dog) is
a white wall with four AprilTags positioned in a rectangle.

Our imaging system consists of a FLIR machine vision camera (Model Firefly S)
and a light source, together mounted on a rigid structure on top of our legged
robot, as shown in Fig. 4. The baseline between the camera and the light
source is approximately 32 cm, measured by hand. Although the ray-tracing
problem can be simplified by placing the light source and camera close enough
and modeling them as co-centered [28], in many applications a long baseline
between them is a preferred design worth studying. For example, deep-sea
robots usually have a long baseline between light and camera to avoid massive
light backscattering from the water.

We experimented with three different light sources: a flashlight, a diving
light, and a flood light. To build NeLiS, we take $\sim 40$ calibration images
from different ranges and perspectives for each light configuration and use
$25\%$ of the images as a testing set to evaluate the NeLiS performance. For
building DarkGS, we take $50\sim 150$ images for each scene. All images are
streamed in RAW format without any tone mapping or white balancing. Parameters
optimized in both NeLiS and DarkGS use Adam optimizer [36] with LieTorch [37]
integration.

###  IV-B Can existing 3DGS methods do the job?

We first investigated whether existing 3DGS methods can reconstruct the scene
using the images we collected with our robotic setup in the dark. We
experimented with:  1. vanilla 3DGS [9] which models the scene with constant
radiance  2. RawNeRF [10] which is developed to reconstruct scene from RAW HDR
images  3. Relightable 3DGS [11] which models the shading and environmental
light map.  To make a fair comparison, we apply gamma tone mapping to images
for vanilla 3DGS [9] and Relightable 3DGS [11] which brightens the image and
smooth the discrepancy between illuminated and under-illuminated areas. We
also replace RawNeRF’s backbone MLP-based renderer [10] with Gaussian render
[9].

As shown in Fig. 5, we found that none of the existing methods is able to
build valid scene reconstructions. We see an excessive amount of artifacts in
the center area of the synthesized image. The key reason is that the
capability to model varying illumination is missing from existing methods.

Figure 5: None of the existing methods can solve the problem: Results of
Vanilla 3DGS [9], RawNeRF [10] Relightable 3DGS [11] show heavy artifacts and
fail to converge. The key reason for the failures is that the existing method
does not model the illumination change.

(a) RID

(b) Light Fall-off Curve

Figure 6: (a) Light sources on real robots have various RID patterns; (b) Real
world measurements of light falloff show that the inverse square law is
insufficient to model any of our light sources, but Lorentzian functions [35]
with learnable parameter $\tau$ fit them well.

###  IV-C Why do we need to learn RID?

As shown in Fig. 6(a), different light has different RID patterns. Existing
methods often model RID as known [25], or general functions such as the power
of cosine functions [22] or a Gaussian distribution. However, for most light
sources, and all light sources we used in this research, RID is not given.
Modeling them with general functions such as a Gaussian distribution will not
be adaptive and expressive enough to reflect the true RID of different light
patterns for building a photorealistic renderer. The ablation study in Fig. 7
(columns 2 and 5) shows that while a Gaussian distribution fits the RID of
flood light well, it performs much worse on the other two types of light. In
comparison, our MLP-based model fits all 3 kinds of light with the best
performance, showing good adaptivity to different light patterns.

Figure 7: Ablation study: We show that our model with an MLP-based RID,
learnable light fall-off parameter $\tau$ and ambient light $A$ can
effectively improve the rendering performance. MSE of the testing set are
highlighted in yellow. The red dashed box bounds the regions to be rendered.
Figure 8: Visulization of results from multiple scenes: We show that with
DarkGS, we can reconstruct the scene with RAW images from robotic deployments
in dark environments, and relight the scene to reveal more information that is
corrupted in the RAW image due to uneven and partial illumination. Results as
shown are all from the flashlight setup which is the most challenging
according to our numerical results.

###  IV-D Why do we need to learn light falloff?

For simplicity, previous studies [24, 22, 25] use an inverse square law to
model the light falloff. It is not a good fit for real-world light systems. We
measure the light falloff of the center point of 3 different light sources as
shown in Fig. 6(b). The plot show that when the range is close, light fall-off
does not follow the inverse square law as the point light source assumption
starts to fail. However, with a Lorentzian model suggested by [35], the light
falloff can be better fitted with our introduced learnable parameter $\tau$.
We further show numerical results in Fig. 7 (columns 2 and 3) that by learning
$\tau$, the rendering performance on testing set of all 3 kinds of lights gets
improved, which implies that our light fall-off model better approximates the
true light falloff.

###  IV-E Do we need a perfectly dark environment?

No. Ambient light is universal in all kinds of environment and a perfectly
dark room for calibration is rare for robot deployments in the wild. Neither
our calibration with NeLiS nor deployment experiments with DarkGS are
performed in a perfectly dark environment. As shown in Fig. 7 (columns 2 and
4), we show that with our modeling of ambient light $A$, the performance of
the testing set is improved on all three different lights.

###  IV-F Results Visualization

We deployed our system in various real-world environments and the results are
shown in Fig. 8. Compared with ground truth, our model is able to reconstruct
the image with photorealistic quality with learned illumination. Then we
replace the light source in our model with a Lambertian light to create a
global illumination that illuminates the entire FOV (FOV) of the camera with
photorealistic rendering quality.

##  V LIMITATIONS

A limitation of our work is that our model currently does not support modeling
objects with heavy shadows or specular reflection (Fig. 9(a)). However, recent
concurrent works [38, 11] have shown the potential to integrate them into the
3DGS framework. It is also obvious that our relighted image is in green-blue
tone (Fig. 9(b)). This is because  1. We are sticking with the RAW image
format throughout the pipeline, so images appear green as the RAW image nature
2. Our LED light has a high color temperature (cool color) so that the
relighted image appears blue.

(a) Shadow & Specularity

(b) White Balancing

Figure 9: Limitations: (a) Our current method is not able to model shadows and
specularity. (b) The relighted image is still in green-blue tone due to the
nature of RAW images format and that white balancing is missing from our
pipeline.

##  VI CONCLUSIONS AND FUTURE WORK

This work aims to solve the problem of building 3D Gaussians and scene
relighting from images taken by a moving camera-light system. Our proposed
pipeline consists of NeLiS, a camera-light simulation and calibration model,
and DarkGS which build’s photolistic representation for scenes in the dark.
The results show that our proposed pipeline can build relightable Gaussians
from images taken by the robot platform deployed in the dark, while other
current approaches are not able to do the job. Ablations show that components
in our proposed model effectively improve the performance of our shading
model, so we can learn RID with an MLP for arbitrary light pattern, better
approximate the light-falloff curve, and allow us to calibrate and deploy the
system in environments that are not perfectly dark. These efforts make our
model more applicable in real-world robot deployments. Future work includes
modeling shadows and non-Lambertian objects, and bringing white balance and
tone mapping into the loop.

## ACKNOWLEDGEMENTS

This work is supported by NOAA NA22OAR0110624.

## References

  * [1] Alexey Dosovitskiy et al.  “CARLA: An Open Urban Driving Simulator”  In _Proceedings of the 1st Annual Conference on Robot Learning_ , 2017, pp. 1–16
  * [2] Ali Agha et al.  “Nebula: Quest for robotic autonomy in challenging environments; team costar at the darpa subterranean challenge”  In _arXiv preprint arXiv:2103.11470_ , 2021
  * [3] Easton Potokar, Spencer Ashford, Michael Kaess and Joshua G. Mangelson  “HoloOcean: An Underwater Robotics Simulator”  In _2022 International Conference on Robotics and Automation (ICRA)_ , 2022, pp. 3040–3046  DOI: 10.1109/ICRA46639.2022.9812353
  * [4] Sameer Agarwal et al.  “Building Rome in a day”  In _Communications of the ACM_ 54, 2011, pp. 105–112
  * [5] Ben Mildenhall et al.  “NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis”  In _ECCV_ , 2020
  * [6] David G Lowe  “Distinctive image features from scale-invariant keypoints”  In _International journal of computer vision_ 60 Springer, 2004, pp. 91–110
  * [7] Raúl Mur-Artal, J. M. M. Montiel and Juan D. Tardós  “ORB-SLAM: a Versatile and Accurate Monocular SLAM System”  In _IEEE Transactions on Robotics_ 31.5, 2015, pp. 1147–1163  DOI: 10.1109/TRO.2015.2463671
  * [8] Carlos Campos et al.  “ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual-Inertial and Multi-Map SLAM”  In _IEEE Transactions on Robotics_ 37.6, 2021, pp. 1874–1890
  * [9] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler and George Drettakis  “3D Gaussian Splatting for Real-Time Radiance Field Rendering”  In _ACM Transactions on Graphics_ 42.4, 2023  URL: https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/
  * [10] Ben Mildenhall et al.  “NeRF in the Dark: High Dynamic Range View Synthesis from Noisy Raw Images”  In _CVPR_ , 2022
  * [11] Jian Gao et al.  “Relightable 3D Gaussian: Real-time Point Cloud Relighting with BRDF Decomposition and Ray Tracing”  In _arXiv:2311.16043_ , 2023
  * [12] Johannes Lutz Schönberger and Jan-Michael Frahm  “Structure-from-Motion Revisited”  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016
  * [13] Edwin Olson  “AprilTag: A robust and flexible visual fiducial system”  In _2011 IEEE International Conference on Robotics and Automation_ , 2011, pp. 3400–3407  DOI: 10.1109/ICRA.2011.5979561
  * [14] Joern Rehder et al.  “Extending kalibr: Calibrating the extrinsics of multiple IMUs and of individual axes”  In _2016 IEEE International Conference on Robotics and Automation (ICRA)_ , 2016, pp. 4304–4311  IEEE
  * [15] Jorge Beltrán, Carlos Guindel, Arturo Escalera and Fernando García  “Automatic Extrinsic Calibration Method for LiDAR and Camera Sensor Setups”  In _IEEE Transactions on Intelligent Transportation Systems_ , 2022  DOI: 10.1109/TITS.2022.3155228
  * [16] Joris Domhof, Julian F. P. Kooij and Dariu M. Gavrila  “A Joint Extrinsic Calibration Tool for Radar, Camera and Lidar”  In _IEEE Transactions on Intelligent Vehicles_ 6.3, 2021, pp. 571–582  DOI: 10.1109/TIV.2021.3065208
  * [17] Disai Yang, Bingwei He, Mingzhu Zhu and Jiantao Liu  “An Extrinsic Calibration Method With Closed-Form Solution for Underwater Opti-Acoustic Imaging System”  In _IEEE Transactions on Instrumentation and Measurement_ 69.9, 2020, pp. 6828–6842  DOI: 10.1109/TIM.2020.2976082
  * [18] Jingyu Song, Lingjun Zhao and Katherine A Skinner  “LiRaFusion: Deep Adaptive LiDAR-Radar Fusion for 3D Object Detection”  In _arXiv preprint arXiv:2402.11735_ , 2024
  * [19] Hiroaki Santo et al.  “Light Structure from Pin Motion: Simple and Accurate Point Light Calibration for Physics-based Modeling”  In _European Conference on Computer Vision (ECCV)_ , 2018
  * [20] Hiroaki Santo et al.  “Light structure from pin motion: Geometric point light source calibration”  In _International Journal of Computer Vision (IJCV)_ , 2020  DOI: 10.1007/s11263-020-01312-3
  * [21] Long Ma et al.  “A fast LED calibration method under near field lighting based on photometric stereo”  In _Optics and Lasers in Engineering_ 147, 2021, pp. 106749  DOI: https://doi.org/10.1016/j.optlaseng.2021.106749
  * [22] Long Ma et al.  “Robust point light source calibration method for near-field photometric stereo using feature points selection”  In _Appl. Opt._ 62.36 Optica Publishing Group, 2023, pp. 9512–9522  DOI: 10.1364/AO.505234
  * [23] Wei Jin et al.  “Shadow-Based Lightsource Localization With Direct Camera–Lightsource Geometry”  In _IEEE Transactions on Instrumentation and Measurement_ 73, 2024, pp. 1–12  DOI: 10.1109/TIM.2023.3344148
  * [24] Long Ma et al.  “Calibration of position and orientation for point light source synchronously with single image in photometric stereo”  In _Opt. Express_ 27.4 Optica Publishing Group, 2019, pp. 4024–4033  DOI: 10.1364/OE.27.004024
  * [25] Yifan Song et al.  “Light Pose Calibration for Camera-light Vision Systems”, 2020  arXiv:2006.15389 [cs.CV]
  * [26] Jonathan T. Barron et al.  “Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields”  In _2021 IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2021, pp. 5835–5844  URL: https://api.semanticscholar.org/CorpusID:232352655
  * [27] Thomas Müller, Alex Evans, Christoph Schied and Alexander Keller  “Instant Neural Graphics Primitives with a Multiresolution Hash Encoding”  In _ACM Trans. Graph._ 41.4 New York, NY, USA: ACM, 2022, pp. 102:1–102:15  DOI: 10.1145/3528223.3530127
  * [28] Sai Bi et al.  “Neural Reflectance Fields for Appearance Acquisition”  In _arXiv preprint arXiv:2008.03824_ arXiv, 2020  DOI: 10.48550/ARXIV.2008.03824
  * [29] Tianyi Zhang and Matthew Johnson-Roberson  “Beyond NeRF Underwater: Learning Neural Reflectance Fields for True Color Correction of Marine Imagery”  In _IEEE Robotics and Automation Letters_ 8.10, 2023, pp. 6467–6474  DOI: 10.1109/LRA.2023.3307287
  * [30] Yingwenqi Jiang et al.  “GaussianShader: 3D Gaussian Splatting with Shading Functions for Reflective Surfaces”  In _arXiv preprint arXiv:2311.17977_ , 2023
  * [31] Robert Woodham  “Photometric Method for Determining Surface Orientation from Multiple Images”  In _Optical Engineering_ 19, 1992  DOI: 10.1117/12.7972479
  * [32] Ayan Chakrabarti and Kalyan Sunkavalli  “Single-image RGB photometric stereo with spatially-varying albedo”  In _2016 Fourth International Conference on 3D Vision (3DV)_ , 2016, pp. 258–266  IEEE
  * [33] Ronen Basri, David Jacobs and Ira Kemelmacher  “Photometric Stereo with General, Unknown Lighting”  In _International Journal of Computer Vision_ 72, 2007, pp. 239–257  DOI: 10.1109/CVPR.2001.990985
  * [34] Martin A. Fischler and Robert C. Bolles  “Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography”  In _Commun. ACM_ 24.6 New York, NY, USA: Association for Computing Machinery, 1981, pp. 381–395  DOI: 10.1145/358669.358692
  * [35] A. Ryer  “Light Measurement Handbook”  International Light, 1997  URL: https://books.google.com/books?id=e-JIPwAACAAJ
  * [36] Diederik Kingma and Jimmy Ba  “Adam: A Method for Stochastic Optimization”  In _International Conference on Learning Representations_ , 2014
  * [37] Zachary Teed and Jia Deng  “Tangent Space Backpropagation for 3D Transformation Groups”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021
  * [38] Luis Bolanos, Shih-Yang Su and Helge Rhodin  “Gaussian Shadow Casting for Neural Characters”, 2024  arXiv:2401.06116 [cs.CV]

  *[SfM]:
  *[SLAM]:
  *[NeRF]:
  *[NeLiS]:
  *[DarkGS]:
  *[3DGS]:
  *[RID]:
  *[MLP]:
  *[PnP]:
  *[ROI]:
  *[BRDF]:
  *[FOV]: