# Solving ImageNet: a Unified Scheme for Training any Backbone to Top Results

Tal Ridnik  Hussam Lawen  Emanuel Ben-Baruch  Asaf Noy
DAMO Academy  Alibaba Group
<EMAIL_ADDRESS>
###### Abstract

ImageNet serves as the primary dataset for evaluating the quality of computer-
vision models. The common practice today is training each architecture with a
tailor-made scheme, designed and tuned by an expert. In this paper, we present
a unified scheme for training any backbone on ImageNet. The scheme, named USI
(Unified Scheme for ImageNet), is based on knowledge distillation and modern
tricks. It requires no adjustments or hyper-parameters tuning between
different models, and is efficient in terms of training times. We test USI on
a wide variety of architectures, including CNNs, Transformers, Mobile-oriented
and MLP-only. On all models tested, USI outperforms previous state-of-the-art
results. Hence, we are able to transform training on ImageNet from an expert-
oriented task to an automatic seamless routine. Since USI accepts any backbone
and trains it to top results, it also enables to perform methodical
comparisons, and identify the most efficient backbones along the speed-
accuracy Pareto curve.
Implementation is available at: https://github.com/Alibaba-
MIIL/Solving_ImageNet

##  1 Introduction

(a) Various architecture families

(b) USI training scheme

(c) Top-1 accuracy comparison (%)

Figure 1: Our unified training scheme for ImageNet, USI. With USI, we can
train any backbone to top results on ImageNet, without any hyper-parameter
tuning or adjustments per architecture.

ImageNet (1K) dataset, introduced for the ILSVRC2012 visual recognition
challenge [31], has been at the center of modern advances in deep learning
[21, 14, 32]. It serves as a main dataset for pretraining computer-vision
models [34, 22, 17], and measuring the accuracy of models on ImageNet was
found to be a good proxy to actual performance on various downstream tasks
[20, 29, 42].

However, training on ImageNet remains an ongoing challenge. Since the seminal
work of AlexNet [21], new training tricks, regularizations and enhancements to
improve results have been continuously suggested: One-cycle learning rate
scheduling [33]; Stronger augmentations based on AutoAugment [5] and
RandAugment [6]; Scaling learning rate with batch size [43]; Exponential-
moving average (EMA) of model weights [19]; Improved weights initializations
[13, 15]; Image-based regularizations such as Cutout [7], Cutmix [46] and
Mixup [48]; Architecture regularizations like drop-path [4] and drop-block
[9]; Label-smoothing [27]; Different train-test resolutions [39]; More
training epochs [42]; Progressive image resizing during training [35]; True
weight decay [26]; Dedicated optimizer for large batch size [44], and more.

Almost any new architecture proposed is accompanied by a dedicated training
scheme (e.g. [25, 38, 29, 35, 15]). These schemes can significantly differ,
and a tailored scheme for one model often underperforms when used for other
models. For example, when a dedicated scheme for ResNet50 model [42] was used
for training EfficieneNetV2 model, it yielded $3.3\%$ lower accuracy than a
tailor-made scheme [35].

Broadly speaking, deep learning backbones for computer vision can be divided
into four categories: ResNet-like, Mobile-oriented, Transformers, and MLP-
only. ResNet model [14] and its variants (TResNet [30], SEResNet [18],
ResNet-D [15] to name a few) usually work well over a wide variety of training
schemes [29]. A top-performance training scheme for ResNet models was
suggested in [42], and became the standard for this type of models. Mobile-
oriented models are architectures that heavily rely on depthwise convolutions,
and efficient CPU-oriented design [32, 17, 34]. Their dedicated training
schemes usually consist of RMSProp optimizer, waterfall learning rate
scheduling and EMA. Due to lack of inductive bias, Transformer-based [8, 24,
11] and MLP-only [36, 37] models for vision are harder to train, and less
stable [29, 23]. A dedicated training scheme for these models was proposed in
[38], which includes longer training (1000 epochs), strong cutmix-mixup and
drop-path regularizations, large weight-decay and repeated augmentations [1].

Knowledge Distillation (KD), originally proposed in [16], uses a teacher model
to guide the target network (often referred to as the student model) along the
training. The student receives supervision from both the ground-truth label
and the teacher’s prediction for each image. KL-divergence measures the extra
loss between the teacher the and student. [38] proposed a dedicated KD scheme
for training Transformers-based models on ImageNet, that relies on a special
distillation token, hard-label thresholding and long training. [3] used KD
from super-network to train sub-networks during neural architecture search
process. [22] showed marginal improvement for ResNet50 model when using KD.
[40] demonstrated that KD suppresses noise from data augmentations, and
enables using stronger augmentations. [47] suggested a variant of KD by
cropping the spatial prediction layer of the teacher.

However, KD is not a common practice for ImageNet training. Most training
schemes proposed throughout the years have not utilized KD, and it is not an
integral part of the popular repositories for ImageNet training, such as [41].
For obtaining top results, a more frequent option is pretraining a model on
the larger ImageNet-21K dataset [29], and fine-tuning it on ImageNet-1K [24,
35, 25, 37, 30]. This alternative, of course, requires longer training and
larger computational budget. It is a common practice to separate between
results obtained from using only ImageNet-1K images, to results obtained using
extra-data, such as ImageNet-21K pretraining.

In this paper, we introduce a unified training scheme for ImageNet, called USI
(Unified Scheme for ImageNet). USI can train any backbone to state-of-the-art
results, without any hyper-parameter tuning or tailor-made tricks per model.
An illustration of USI training scheme appears in Figure 1. The scheme is
based on the observation that vanilla KD, which imposes an additional KL-
divergence loss between the teacher and student predictions, works very well
on any backbone. The reason for the effectiveness of KD is the introduction of
additional information, that does not exist in the original ground-truth
labels. The teacher’s predictions per image account for correlations and
similarities between classes, they handle better pictures with several
objects, and even compensate for ground-truth mistakes. KD also handle better
augmentations, and removes the need for label smoothing. All these factors
lead to a more robust and effective optimization process, that requires fewer
training tricks and regularizations.

We thoroughly test USI on a wide variety of modern deep learning models,
including ResNet-like, Mobile-oriented, Transformer-based and MLP models. On
all model tested, USI outperforms previously reported state-of-the-art
results, that were obtained with a dedicated scheme per model. USI is also
efficient, and requires only $300$ epochs of training.

Since the proposed scheme consistently leads to top results, it also enables a
fair comparison of speed-accuracy trade-offs. We benchmark various models on
GPU and CPU, and identify leading models that provide the best speed-accuracy
trade-off along the Pareto curve.

The paper’s contributions can be summarized as follow:

  * •

We introduce a unified, efficient training scheme for ImageNet dataset, USI,
that does not require hyper-parameter tuning. Exactly the same recipe is
applied to any backbone. Hence, ImageNet training is transformed from an
expert-oriented task into an automatic seamless procedure.

  * •

We test USI on various deep learning models, including ResNet-like, Mobile-
oriented, Transformer-based and MLP-only models. We show it consistently and
reliably achieves state-of-the-art results, compared to tailor-made schemes
per model.

  * •

We use USI to perform a methodological speed-accuracy comparison of modern
deep learning models, and identify efficient backbones along the Pareto curve.

Figure 2: Examples for teacher predictions. ImageNet ground-truth labels are
highlighted in red. Unlike the ground-truth, the teacher predictions account
for similarities and correlations between classes, objects’ saliency, pictures
with several objects, and more. The teacher predictions would also better
represent the content of images under strong augmentations.

##  2 Method

In this section, we will first review how to apply knowledge distillation (KD)
for classification. Then we will discuss why we need KD in ImageNet training,
and finally present USI, our KD-based training scheme for ImageNet.

###  2.1 KD for Classification

For any input image, a classification network outputs a logit vector
$\mathbf{z}=\\{z_{i}\\}_{i=1}^{K}$, where $K$ is the number of classes. The
softened prediction vector is denoted by
$\mathbf{p}(\tau)=\\{p_{i}\\}_{i=1}^{K}$, where each element $p_{i}(\tau)$ is
given by applying the softmax activation function, with a temperature-scaling
parameter $\tau$:

$p_{i}(\tau)=\frac{\exp({z_{i}/\tau})}{\sum_{j=1}^{K}\exp({z_{j}/\tau})}.$ (1)

Let us define the softened prediction vectors of the student and teacher
models as $\mathbf{p}^{s}(\tau)$ and $\mathbf{p}^{t}(\tau)$, respectively. The
objective function used for training the student model is a combination of the
Cross-Entropy (CE) loss between the student’s prediction and the ground-truth
vector $\mathbf{y}$, and the Kullback-Leibler (KL) divergence between the
student and the teacher predictions,

$\mathcal{L}=\mathcal{L}_{\text{CE}}(\mathbf{p}^{s}(1),\mathbf{y})+\alpha_{\text{kd}}\mathcal{L}_{\text{KL}}(\mathbf{p}^{s}(\tau),\mathbf{p}^{t}(\tau)),$
(2)

where $\alpha_{kd}$ is a hyper-parameter for adjusting the relative importance
of the KD loss. The CE loss is given by,

$\mathcal{L}_{\text{CE}}(\mathbf{p}^{s}(1),\mathbf{y})=-\sum_{j}y_{j}\log
p^{s}_{j}(1),$ (3)

and the KL divergence loss is given by,

$\mathcal{L}_{\text{KL}}(\mathbf{p}^{s}(\tau),\mathbf{p}^{t}(\tau))=\tau^{2}\sum_{j}p^{t}_{j}(\tau)\log\frac{p^{t}_{j}(\tau)}{p^{s}_{j}(\tau)}.$
(4)

###  2.2 Why Do We Need KD in ImageNet Training ?

ImageNet serves as the primary dataset for pretraining and evaluating
computer-vision models. Unlike other classification datasets, on ImageNet we
are training models from scratch, and not doing transfer learning. Training
from scratch is, in general, harder, and requires higher learning rates,
stronger regularizations, and more training epochs. Hence, the optimization
process on ImageNet is more sensitive to different hyper-parameters, and the
architecture used.

To gain more insight and motivation into the impact of KD, we present in
Figure 2 some typical examples for the predictions of the teacher model,
compared to ImageNet ground-truth labels.

  * •

Picture (a) contains big salient nails. That is the ground-truth, and that’s
also the teacher leading prediction ($99.9\%$). Notice that the teacher’s 2nd
and 3rd top predictions are related to nails (screw and hammer), but with
negligible probability.

  * •

Picture (b) contains an airliner. That is the top prediction of the teacher
($83.6\%$). However, the teacher also predicts wing with a non-negligible
probability ($11.3\%$). That is not a mistake - an airliner has wings. The
teacher here mitigates the case where the ground-truth labels are not
mutually-exclusive, and provides more accurate information about the content
of the image.

  * •

Picture (c) contains a hen (female chicken). However, the hen is not very big
and salient. The teacher’s predictions reflects that, by identifying an hen
with lower probability ($55.5\%$). The teacher also gives non-negligible
probability to cock (male chicken) label - $8.9\%$. This is a mistake by the
teacher, but a logical one - hen and cock are quite similar.

  * •

In picture (d) the teacher disagrees with the ground-truth. The ground-truth
is ice-lolly, while the teacher top prediction is English setter (a type of
dog). The teacher is correct - the dog is more salient in the picture.

We see from the above examples that the teacher’s predictions contain more
information than the plain (single-label) ground-truth. The rich predictions
provided by the teacher account for correlations and similarities between
classes. They handle better pictures with several objects, and even compensate
for ground-truth mistakes. KD predictions also handles better strong
augmentations, since they represent the correct content of the augmented
image. They also removes the need for label smoothing, since the teacher
inherently outputs soft predictions.

Due to these factors, training with a teacher provides better supervision,
leading to a more effective and robust optimization process, compared to
training with hard-labels only.

###  2.3 The Proposed Training Scheme

Our proposed training scheme for ImageNet, USI, is based on utilizing KD. When
training on ImageNet with KD, we observe that the training process is far more
robust to hyper-parameter selection, and requires fewer training tricks and
regularizations. In addition, the need for dedicated tricks per backbone is
eliminated - a single unified scheme can train any backbone to top results. An
illustration of USI scheme appears in Figure 1. In Table 1 we present the full
training configuration.

Procedure | Value
---|---
Train resolution | 224
Test resolution | 224
Epochs | 300
Optimizer | AdamW
Weight decay | 2e-2
Learning rate | 2e-3
LR decay | One-cycle policy
Mixup alpha | 0.8
Cutmix alpha | 1.0
Augmentations | Rand-augment (7/0.5)
Test crop ratio | 0.95
Repeated Augs | 3
Base loss | Cross entropy
KD loss | KL-divergence
KD temperature | 1
$\alpha_{kd}$ | 5
Teacher | TResNet-L
Batch size | 512 to 3456
Table 1: USI training configuration. With USI, exactly the same training
recipe is applied to any backbone, and no hyper-parameter tuning is needed.

Some observations and insights into the proposed scheme:

##### Batch size selection

The maximal batch size allowed by Different backbones varies significantly
(see Table 10 in the appendix). Hence, using a fixed batch size for all
backbones is not always possible. It is beneficial to choose a batch size as
large as possible, since it enables fully utilizing the GPU cores, reducing
communication overheads, and increasing training speed. Previous schemes
suggested that larger batches require larger learning rates or a dedicated
optimizer [43, 10].

USI, which is KD-based training scheme with AdamW optimizer, is more robust to
batch size and learning rate tuning. We will show in Section 3.2 that with the
same learning rate, USI consistently provides top results for a wide range of
batch sizes. Hence, in Table 1 we state a range of batch sizes, instead of a
fixed one. Any value along this range can be chosen. We recommend using $0.8$
to $0.9$ of the maximal possible batch size possible, for optimizing training
speeds.

##### Which teacher to choose

Our primary requirement is to choose a teacher who outperforms the student, a
common requirement in KD [16]. Given that constraint, we suggest selecting a
teacher model with good speed-accuracy trade-off (see Figure 5). We will show
in Section 3.3 that our scheme is robust to teacher and student types.
Teachers with similar accuracy train students to similar accuracy, regardless
of their type (CNNs or Transformers).

Model Type |  Model Name |  USI Top1 Acc. [%] | Comparable Training Scheme
---|---|---|---
Top1 Acc. [%] | Epochs | KD | Additional Details
CNN | ResNet50 | 81.0 | 80.4 [42] | 600 | no | ResNet-strike-back, A1 config
80.2 [47] | 300 | yes | Relabel-based KD
TResNet-M | 82.4 | 80.7 [30] | 300 | no |
ConvNext-S | 84.0 | 83.1 [25] | 300 | no |
RegNetY-32 | 82.8 | 81.0 [28] | 100 | no |
Transformer | ViT-S | 81.8 | 79.8 [8] | 300 | no | Original paper scheme
81.2 [38] | 1000 | yes | DeiT scheme
LeViT-384 | 82.7 | 82.6 [11] | 1000 | yes | DeiT scheme
Swin-S | 84.0 | 83.2 [24] | 300 | no |
TNT-S | 82.5 | 81.5 [12] | 300 | no |
Mobile- Oriented CNN | OFA-595m | 80.6 | 80.0 [3] | 255 | yes | KD from super network
EfficientnetB0 | 79.3 | 77.1 [34] | not stated | no |
MobileNetV3 | 77.3 | 75.2 [17] | not stated | no |
MLP-Only | Mixer-B | 82.0 | 76.4 [36] | 255 | no |
ResMLP-S12 | 78.8 | 77.8 [37] | 400 | yes |
|  |  |  |  |  |
Table 2: Comparison of our proposed scheme, USI, to previous state-of-the-art
results. Train and test resolution - $224$.

##### KD impact on training speed

Adding KD supervision incurs additional overhead, and reduces training speed.
However, the additional overhead is usually small. while the student network
needs to do forward pass, store intermediate maps, do backward pass and update
weights, the teacher network only needs forward passes. In addition, since the
teacher model is fixed, we can apply various optimizations to it, like batch-
norm fusion, channels-last and jit [30]. We found that the relative overhead
of KD decreases as we increase the batch size, which is another reason to
prefer large batch sizes. For TResNet-L [30] teacher model ($83.9\%$
accuracy), the additional overhead from KD reduces the training speed by
$10\%$-$20\%$.

##  3 Results

###  3.1 Comparison to Previous Schemes

In Table 2 we present the ImageNet top-1 accuracy obtained for various deep
learning architectures, when trained with our proposed scheme, USI. We compare
USI to previous state-of-the-art results, obtained with a tailor-made scheme
per architecture. We see from Table 2 that on all architecture tested (CNN,
Transformer, Mobile-oriented, MLP-only), USI reaches results better than
previously reported top results:

  * •

For CNN architectures, USI significantly outperforms previous results. This
applies also to ResNet50 model, where we compare USI to a recently proposed
dedicated scheme [42], and a KD-based scheme [47].

  * •

For Transformer architectures, on two prominent models, ViT-S and LeViT-384,
USI reaches better results compared to DeiT scheme [38], that also utilized
KD. Notice that DeiT used a tailored KD training recipe for Transformers,
which includes longer training ($1000$ epochs), an auxiliary KD head from
special distillation token, and modifications to the KD algorithm (hard-
thresholding the teacher predictions). In contrast, USI uses vanilla KD, less
than a third of the epochs, and no architecture-tailored modifications. With
equal number of epochs, the gap between USI and DeiT is even bigger (see
Section 3.4).

  * •

For Mobile-oriented and MLP-based architectures, USI shows significant
improvement compared to previously reported results.

To conclude, Table 2 presents results for a wide variety of modern deep
learning models. For each model, we compare results from our proposed scheme,
USI, to top results from the literature, obtained with tailor-made schemes per
model. On all models tested, without any hyper-parameter tuning, USI achieves
state-of-the-art results. We believe that this is a fair comparison, that
demonstrates that USI enables to transform high-quality training on ImageNet
into an automatic routine, that requires no dedicated expertise and cumbersome
tuning process.

###  3.2 Robustness to Batch-size

As discussed in Section 2.3, larger batch size leads to faster training speed.
In Table 3 we test the robustness of USI, with fixed learning rate, to
different batch sizes. For the tests we used TResNet-M model, which enables
large maximal batch size due to usage of inplace-activated batch-norm instead
of regular batch-norm [30] Table 3 demonstrates that over a large range of
batch sizes, $512$-$3456$, the accuracy remains almost the same. This
indicates that USI operates well with a fixed learning rate.

| Batch
---
Size
| Top1 Acc.
---
[%]
| Training speed
---
[img/sec]
512 | 82.3 | 1100
1024 | 82.5 | 1900
2048 | 82.3 | 3000
2752 | 82.4 | 4100
3456 | 82.4 | 4300
3456 |  |  | 4900
---
(no-KD reference)
Table 3: Accuracy and training speed, for different batch sizes. Model tested
- TResNet-M.

For our runs, we used an 8xV100 Nvidia machine, TResNet-M student, and
TResNet-L teacher. In terms of training speed, with a batch size of $512$
($64$ per GPU) we reached a training speed of $1100$ img/sec, while with a
batch size of $3456$ ($432$ per GPU) we reached $4300$ img/sec. Hence,
increasing the batch size significantly aids the training speed. Without KD,
we reached a training speed of $4900$ img/sec.

###  3.3 Robustness to Teacher Type

The results in Table 2 were obtained with TResNet-L teacher. We chose this
model as a teacher since it provides high accuracy, and a good speed-accuracy
trade-off on GPU (see Figure 5). In Table 4 we test if different student
models benefit from different teacher models. Since TResNet-L is a CNN, we
compared it to a Transformer-based teacher, Volo-d1 [45], which has similar
top-1 accuracy ($83.9\%$ for TResNet-L, $84.1\%$ for Volo-d1).

Student |  | Student
---
Type
Teacher |  | Teacher
---
Type
| Top1
---
Acc. [%]
ResNet50 | CNN | TResNet-L | CNN | 81.0
Volo-d1 | Transformer | 80.9
LeViT384 | Transformer | TResNet-L | CNN | 82.7
Volo-d1 | Transformer | 82.7

Table 4: Testing different students with different teachers.

We see from Table 4 that both CNN and Transformer students work well with CNN
and Transformer teachers. This implies we have flexibility in choosing the
teacher type.

###  3.4 Number of Training Epochs

USI default configuration is KD training, for $300$ epochs. However, for
reaching the maximal possible accuracy, $300$ epochs is not enough, and longer
training would further improve the accuracy. This phenomenon, where in KD the
student model benefits from very long training, is called patient teacher [2].
In Table 5 we present the accuracies obtained for various training lengths.

Training epochs | Top1 Acc. [%]
---|---
100 | 80.0
200 | 81.9
300 | 82.7
600 | 83.0
1000 | 83.2
Table 5: Accuracy for different numbers of epochs. Model tested - LeViT-384.

As can be seen, the accuracy continues to improve as we increase training
epoch from $300$ to $600$ and to $1000$.

Our default training configuration was chosen to be $300$ epochs since this
value provides a good compromise - training times are reasonable ($1$-$3$ days
on 8xV100 GPU, depending on the model), while we consistently achieve good
results, as can be seen in Figure 1 (c). However, if training times are not a
limitation, we recommend increasing the number of training epochs.

###  3.5 Speed-Accuracy Measurements

When using a tailor-made training scheme per backbone, that depends on hyper-
parameter tuning, computing power and other factors, it is challenging to
reliably compare different models. With USI, that consistently provides top
results for any backbone, we can do a methodological, reproducible and
reliable comparison of speed-accuracy trade-off on ImageNet. In Figure 3 and
Figure 4 we compare various modern architectures, on GPU and CPU inference.

$1{,}000$$2{,}000$$3{,}000$$4{,}000$$5{,}000$$6{,}000$$7{,}000$$8{,}000$$9{,}000$$10{,}000$$11{,}000$$80$$82$$84$LeViT-768*LeViT-384ConvNext-
SOFA-595mResMLP-S12Swin-STResNet-MTResNet-LViT-
SLeViT-256ResNet-50RegNetY-32EfficientNet-B0TnT-SMixer-
BEfficientNet-B3Inference Speed [images/sec]Top-1 Accuracy
[%]TransformerCNNMobile-OrientedMLP-only Figure 3: Speed-Accuracy comparison
on an Nvidia V100 GPU, using TensorRT inference engine.
$0$$100$$200$$300$$400$$500$$600$$700$$800$$78$$80$$82$$84$LeViT-768*LeViT-384ConvNext-
SOFA-595mResMLP-S12Swin-STResNet-MTResNet-LViT-
SLeViT-256ResNet-50RegNetY-32MobileNetV3-LEfficientNet-B0EfficientNet-B3EfficientNet-B2TnT-
SMixer-BInference Speed [images/sec]Top-1 Accuracy [%]TransformerCNNMobile-
OrientedMLP-only Figure 4: Speed-Accuracy comparison on an Intel Xeon Cascade
Lake 2.5Ghz CPU, using OpenVINO inference engine.

##### Implementation details

On GPU, The throughput was tested with TensorRT inference engine, FP16, and
batch size of 256. On CPU, The throughput was tested using Intel’s OpenVINO
Inference Engine, FP16, a batch size of 1 and 16 streams (equivalent to the
number of CPU cores). All measurements were done after models were optimized
to inference by batch-norm fusion. This significantly accelerates models that
utilize batch-norm, like ResNet50, TResNet and LeViT. Note that LeViT-768*
model is not a part of the original paper [11], but a model defined by us, to
test LeViT design on higher accuracies regime.

##### GPU inference analysis

On GPU, for low-to-medium accuracies, the most efficient models are LeViT-256
and LeViT-384. For higher accuracies ($>83.5\%$), TResNet-L and LeViT-768*
models provides the best trade-off among the models tested. Note that besides
LeViT, other transformer models, such as Swin, TnT and ViT, provide inferior
speed-accuracy trade-off compared to modern CNNs. Mobile-oriented models also
do not excel on GPU inference, compared to top CNNs. Also note that several
modern architectures, titled ”Small” (ConvNext-S, Swin-S, TnT-S), are in fact
quite resource-intensive - their inference speed is approximately three times
slower compared to a plain ResNet50 model.

##### CPU inference analysis

On CPU, Mobile oriented models (OFA, MobileNet, EfficientNet) provide the best
speed-accuracy trade-off. LeViT models, who excelled on GPU inference, are not
as efficient for CPU inference.

###  3.6 Additional Ablations

In this section we will present additional ablations and tests, to further
validate our USI training configuration.

####  3.6.1 KD teacher relative weight

In Table 6 we test the impact of the relative weight between the KD loss and
and supervision loss ($\alpha_{kd}$ in Eq. 2).

KD relative weight, $\alpha_{kd}$ | Top1 Acc. [%]
---|---
0 (no KD loss) | 76.2
1 | 80.8
5 | 82.7
10 | 82.6
20 | 82.7
$\infty$ (no CE loss) | 82.7
Table 6: Accuracy for different KD relative weights. Model tested - LeViT-384

As can be seen, without KD ($\alpha_{kd}=0$), our training scheme performs
poorly - $6.5\%$ less than accuracy obtained with the default value,
$\alpha_{kd}=5$. If the KD relative weight is too low ($\alpha_{kd}=1$), there
is also a decline in scores. For $\alpha_{kd}\geq 5$ we achieve top results.
Interestingly, results remain the same even when training without the original
hard-label supervision, and relying only on the teacher
($\alpha_{kd}=\infty)$. This further demonstrates the effectiveness of KD in
ImageNet training.

####  3.6.2 KD Temperature

In Table 7 we investigate the impact of KD Temperature ($\tau$ in Eq. 2) on
the accuracy.

KD Temperature, $\tau$ | Top1 Acc. [%]
---|---
0.1 | 79.3
1 | 82.7
2 | 82.7
5 | 81.7
10 | 81.4
Table 7: Accuracy for different KD temperatures. Model tested - LeViT-384

Table 7 shows that there is no benefit from using temperature in our KD loss.
Both $\tau<1$ (sharpening the teacher predictions) and $\tau>1$ (softening the
teacher predictions) reduce the accuracy. Utilizing vanilla softmax
probabilities leads to the best results.

####  3.6.3 Mixup-Cutmix vs. Cutout

Cutout, Mixup and Cutmix are important augmentations, that significantly
improved ImageNet Scores [30]. While Cutout only augments the input image,
Cutmix and Mixup also alter the labels. Cutout is more prevalent when training
CNNs and Mobile-oriented models [17, 15], while a combination of Mixup and
Cutmix, as proposed in [41], is popular for training Transformer-based models
[11, 38]. In Table 8 we compare using Cutout to Mixup-Cutmix in our unified
scheme.

Augmentation Type | Top1 Acc. [%]
---|---
None | 82.0
Cutout | 82.4
Mixup-Cutmix | 82.7
Table 8: Accuracy for different augmentations. Model tested - LeViT-384

As can be seen, applying each augmentation is beneficial, but Mixup-Cutmix
augmentation contributes more to improving the accuracy.

####  3.6.4 Architecture-based regularizations

It is common to apply architecture-based regularizations, such as drop-path
[4] and drop-block [9], mainly when training Transformer-based models [8, 38].
These regularizations are not always applicable to other types of
architectures, and their implementations sometimes differ between different
models. Hence we preferred to avoid using them in our unified scheme.

In Table 9 we test whether adding drop-path to our scheme, when training a
Transformer-based model, would improve results.

Drop-path | Top1 Acc. [%]
---|---
0 | 82.7
0.1 | 82.6
0.2 | 82.5
Table 9: Accuracy for different values of drop-path regularization. Model
tested - LeViT-384

As can be seen, there is no gain from adding drop-path regularizations to USI
scheme.

####  3.6.5 Repository

Due to implementation details and other factors, training schemes on a
specific repository sometimes under-perform when used by a different
repository. We developed USI on an inner private repository. To increase our
confidence in its validity, we re-implemented it on a publicly available
repository - timm [41]. We were able to reproduce on timm all the results
obtained on our private repo. In 111https://github.com/Alibaba-
MIIL/Solving_ImageNet we will share our timm-based implementation.

##  4 Discussion and Conclusions

In this paper, we introduced a unified scheme for ImageNet training, called
USI. USI, which utilizes KD and modern training tricks, requires no hyper-
parameters tuning between different models, and enables to train any backbone
to top results. Hence, it transforms training on ImageNet from an expert-
oriented task to an automatic procedure. With USI, we are also able to perform
a methodical speed-accuracy comparison, and reliably identify efficient
computer-vision backbones.

##### Applicability to other classification datasets

For ImageNet, we are training a model from scratch. This implies using high
learning rates, strong regularization, and more epochs. Hence, our ImageNet-
dedicated USI scheme is not directly applicable to other classification
datasets, which usually use transfer learning. However, KD remains a highly
effective technique also for the transfer learning case. It not only
contributes to improving scores, but also leads to a more robust training
procedure, which is less sensitive to hyper-parameter tuning. We believe that
various AutoML schemes would benefit from KD usage, and plan to demonstrate
this in future work.

## References

  * [1] Maxim Berman, Hervé Jégou, Andrea Vedaldi, Iasonas Kokkinos, and Matthijs Douze.  Multigrain: a unified image embedding for classes and instances.  arXiv preprint arXiv:1902.05509, 2019.
  * [2] Lucas Beyer, Xiaohua Zhai, Amélie Royer, Larisa Markeeva, Rohan Anil, and Alexander Kolesnikov.  Knowledge distillation: A good teacher is patient and consistent.  arXiv preprint arXiv:2106.05237, 2021.
  * [3] Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han.  Once for all: Train one network and specialize it for efficient deployment.  In International Conference on Learning Representations, 2020.
  * [4] Shaofeng Cai, Yao Shu, Gang Chen, Beng Chin Ooi, Wei Wang, and Meihui Zhang.  Effective and efficient dropout for deep convolutional neural networks.  arXiv preprint arXiv:1904.03392, 2019.
  * [5] Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le.  Autoaugment: Learning augmentation strategies from data.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 113–123, 2019.
  * [6] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le.  Randaugment: Practical automated data augmentation with a reduced search space.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 702–703, 2020.
  * [7] Terrance DeVries and Graham W Taylor.  Improved regularization of convolutional neural networks with cutout.  arXiv preprint arXiv:1708.04552, 2017.
  * [8] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [9] Golnaz Ghiasi, Tsung-Yi Lin, and Quoc V Le.  Dropblock: A regularization method for convolutional networks.  Advances in neural information processing systems, 31, 2018.
  * [10] Akhilesh Gotmare, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher.  A closer look at deep learning heuristics: Learning rate restarts, warmup and distillation.  arXiv preprint arXiv:1810.13243, 2018.
  * [11] Benjamin Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Hervé Jégou, and Matthijs Douze.  Levit: a vision transformer in convnet’s clothing for faster inference.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12259–12269, 2021.
  * [12] Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang.  Transformer in transformer.  Advances in Neural Information Processing Systems, 34, 2021.
  * [13] Boris Hanin and David Rolnick.  How to start training: The effect of initialization and architecture.  Advances in Neural Information Processing Systems, 31, 2018.
  * [14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [15] Tong He, Zhi Zhang, Hang Zhang, Zhongyue Zhang, Junyuan Xie, and Mu Li.  Bag of tricks for image classification with convolutional neural networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 558–567, 2019.
  * [16] Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al.  Distilling the knowledge in a neural network.  arXiv preprint arXiv:1503.02531, 2(7), 2015.
  * [17] Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al.  Searching for mobilenetv3.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1314–1324, 2019.
  * [18] Jie Hu, Li Shen, and Gang Sun.  Squeeze-and-excitation networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7132–7141, 2018.
  * [19] Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson.  Averaging weights leads to wider optima and better generalization.  arXiv preprint arXiv:1803.05407, 2018.
  * [20] Simon Kornblith, Jonathon Shlens, and Quoc V Le.  Do better imagenet models transfer better?  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2661–2671, 2019.
  * [21] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  Advances in neural information processing systems, 25:1097–1105, 2012.
  * [22] Jungkyu Lee, Taeryun Won, Tae Kwan Lee, Hyemin Lee, Geonmo Gu, and Kiho Hong.  Compounding the performance improvements of assembled techniques in a convolutional neural network.  arXiv preprint arXiv:2001.06268, 2020.
  * [23] Liyuan Liu, Xiaodong Liu, Jianfeng Gao, Weizhu Chen, and Jiawei Han.  Understanding the difficulty of training transformers.  arXiv preprint arXiv:2004.08249, 2020.
  * [24] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012–10022, 2021.
  * [25] Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie.  A convnet for the 2020s.  arXiv preprint arXiv:2201.03545, 2022.
  * [26] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  arXiv preprint arXiv:1711.05101, 2017.
  * [27] Rafael Müller, Simon Kornblith, and Geoffrey E Hinton.  When does label smoothing help?  Advances in neural information processing systems, 32, 2019.
  * [28] Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár.  Designing network design spaces.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10428–10436, 2020.
  * [29] Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor.  Imagenet-21k pretraining for the masses.  arXiv preprint arXiv:2104.10972, 2021.
  * [30] Tal Ridnik, Hussam Lawen, Asaf Noy, Emanuel Ben Baruch, Gilad Sharir, and Itamar Friedman.  Tresnet: High performance gpu-dedicated architecture.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1400–1409, 2021.
  * [31] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei.  ImageNet Large Scale Visual Recognition Challenge.  International Journal of Computer Vision (IJCV), 115(3):211–252, 2015.
  * [32] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen.  Mobilenetv2: Inverted residuals and linear bottlenecks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510–4520, 2018.
  * [33] Leslie N Smith and Nicholay Topin.  Super-convergence: Very fast training of neural networks using large learning rates.  In Artificial intelligence and machine learning for multi-domain operations applications, volume 11006, page 1100612. International Society for Optics and Photonics, 2019.
  * [34] Mingxing Tan and Quoc Le.  Efficientnet: Rethinking model scaling for convolutional neural networks.  In International conference on machine learning, pages 6105–6114. PMLR, 2019.
  * [35] Mingxing Tan and Quoc Le.  Efficientnetv2: Smaller models and faster training.  In International Conference on Machine Learning, pages 10096–10106. PMLR, 2021.
  * [36] Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al.  Mlp-mixer: An all-mlp architecture for vision.  Advances in Neural Information Processing Systems, 34, 2021.
  * [37] Hugo Touvron, Piotr Bojanowski, Mathilde Caron, Matthieu Cord, Alaaeldin El-Nouby, Edouard Grave, Gautier Izacard, Armand Joulin, Gabriel Synnaeve, Jakob Verbeek, et al.  Resmlp: Feedforward networks for image classification with data-efficient training.  arXiv preprint arXiv:2105.03404, 2021.
  * [38] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou.  Training data-efficient image transformers & distillation through attention.  In International Conference on Machine Learning, pages 10347–10357. PMLR, 2021.
  * [39] Hugo Touvron, Andrea Vedaldi, Matthijs Douze, and Hervé Jégou.  Fixing the train-test resolution discrepancy.  Advances in neural information processing systems, 32, 2019.
  * [40] Longhui Wei, An Xiao, Lingxi Xie, Xiaopeng Zhang, Xin Chen, and Qi Tian.  Circumventing outliers of autoaugment with knowledge distillation.  In European Conference on Computer Vision, pages 608–625. Springer, 2020.
  * [41] Ross Wightman.  Pytorch image models.  https://github.com/rwightman/pytorch-image-models, 2019.
  * [42] Ross Wightman, Hugo Touvron, and Hervé Jégou.  Resnet strikes back: An improved training procedure in timm.  arXiv preprint arXiv:2110.00476, 2021.
  * [43] Yang You, Igor Gitman, and Boris Ginsburg.  Scaling sgd batch size to 32k for imagenet training.  arXiv preprint arXiv:1708.03888, 6(12):6, 2017.
  * [44] Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh.  Large batch optimization for deep learning: Training bert in 76 minutes.  arXiv preprint arXiv:1904.00962, 2019.
  * [45] Li Yuan, Qibin Hou, Zihang Jiang, Jiashi Feng, and Shuicheng Yan.  Volo: Vision outlooker for visual recognition.  arXiv preprint arXiv:2106.13112, 2021.
  * [46] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo.  Cutmix: Regularization strategy to train strong classifiers with localizable features.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 6023–6032, 2019.
  * [47] Sangdoo Yun, Seong Joon Oh, Byeongho Heo, Dongyoon Han, Junsuk Choe, and Sanghyuk Chun.  Re-labeling imagenet: from single to multi-labels, from global to localized labels, 2021.
  * [48] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz.  mixup: Beyond empirical risk minimization.  arXiv preprint arXiv:1710.09412, 2017.

##  Appendix A Model Details

In Table 10 we provide full details for various models. Measurements were done
using PyTorch engine.

Model name |  | Top-1 Accuracy
---
[%]
| Maximal Inference Speed
---
[img/sec]
| Maximal Training Speed
---
[img/sec]
| Maximal
---
Batch Size
MobileNetV3 | 77.3 | 6359 | 1316 | 480
ResMLP-S12 | 78.8 | 4909 | 1474 | 440
EfficienetNet-B0 | 79.3 | 5112 | 1047 | 316
OFA-595m | 80.6 | 3919 | 601 | 292
ResNet50 | 81.0 | 2819 | 785 | 316
ViT-S | 81.8 | 2432 | 724 | 244
Mixer-B | 82.0 | 1388 | 457 | 168
EfficienetNet-B3 | 82.1 | 2604 | 511 | 168
TResNet-M | 82.4 | 2917 | 720 | 504
TnT-S | 82.6 | 782 | 152 | 146
LeViT-384 | 82.7 | 5175 | 1400 | 448
LeViT-768 | 84.2 | 2116 | 594 | 196
RegNetY-32 | 82.8 | 2013 | 546 | 260
TResNet-L | 83.9 | 1563 | 340 | 228
Swin-S | 84.0 | 858 | 223 | 112
ConvNext-S | 84.0 | 1172 | 350 | 128
Table 10: Full model details. All measurements were done on Nvidia V100 GPU,
with 16GB. Train and test resolution - $224$.

##  Appendix B GPU Inference using PyTorch inference engine

10002000300040005000600070008000$78$$80$$82$$84$LeViT-768*LeViT-384ConvNext-
SOFA-595mResMLP-S12Swin-STResNet-MTResNet-LViT-
SLeViT-256ResNet-50RegNetY-32MobileNetV3-LEfficientNet-B0TnT-SMixer-
BEfficientNet-B3Inference Speed [images/sec]Top-1 Accuracy
[%]TransformerCNNMobile-OrientedMLP-only Figure 5: Speed-Accuracy comparison
on an Nvidia V100 GPU, with PyTorch inference engine.

In Figure 5 we compare GPU speed-accuracy trade off of different models with a
PyTorch inference engine.


# Online Learning of Piecewise Polynomial Signed Distance Fields
for Manipulation Tasks

Ante Marić, Yiming Li, and Sylvain Calinon This work was supported by the
State Secretariat for Education, Research and Innovation in Switzerland and
the European Commission’s Horizon Europe Program through the INTELLIMAN
project (https://intelliman-project.eu/, HORIZON-CL4-Digital-Emerging Grant
101070136) and the SESTOSENSO project (http://sestosenso.eu/, HORIZON-
CL4-Digital-Emerging Grant 101070310).The authors are with the Idiap Research
Institute, CH-1920 Martigny, Switzerland and also with the EPFL, 1015
Lausanne, Switzerland<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Reasoning about distance is indispensable for establishing or avoiding contact
in manipulation tasks. To this end, we present an online method for learning
implicit representations of signed distance using piecewise polynomial basis
functions. Starting from an arbitrary prior shape, our approach incrementally
constructs a continuous representation from incoming point cloud data. It
offers fast access to distance and analytical gradients without the need to
store training data. We assess the accuracy of our model on a diverse set of
household objects and compare it to neural network and Gaussian process
counterparts. Distance reconstruction and real-time updates are further
evaluated in a physical experiment by simultaneously collecting sparse point
cloud data and using the evolving model to control a manipulator.

###### Index Terms:

Signed Distance Fields; Representation Learning; Incremental Learning; Machine
Learning for Robot Control

##  I Introduction

Scene representation is a naturally emerging topic in robotics as a basis for
physical interaction. In recent years, implicit modeling methods have been
used as compact representations of environment properties such as distance,
occupancy, and color. Signed distance functions (SDFs) model distances to
closest occupied points by assigning zero values to surfaces, negative values
to surface interiors, and positive values elsewhere. Previously used in
environment mapping and collision avoidance settings [1], they have recently
seen use in robotic manipulation as the field moves towards exploring contact-
rich behaviors [2]. In such scenarios, distance representations can be
exploited to quickly and robustly retrieve gradients for a variety of tasks.
Furthermore, modeling the full range of distances, as opposed to only the zero
level-set, can be beneficial for reasoning about making or breaking contact,
deformation, or penetration, with possible extensions for representing active
agents such as users or robots.

Figure 1: Mesh and distance level-sets of the Stanford Bunny reconstructed
from piecewise polynomial basis functions. The model is learned incrementally
from $1283$ randomly sampled surface points and corresponding normal vectors.

Commonly used implicit SDF models in robotics rely mainly on neural
architectures or Gaussian process models, while alternative formulations
remain less explored. Earlier computer graphics work implicitly encodes SDFs
using basis functions, with recent extensions showing promising results aimed
at simulation environments using piecewise polynomial representations [3]. In
robotics, basis functions have been used to encode movements as superpositions
of primitives [4]. Building on this viewpoint, basis function representations
of implicit distance can be seen as a step toward combining movement with
shape primitives of robot environments or objects of interest.

In many robotics settings, quickly building an environment model from sparse
incoming data takes priority over achieving highly accurate representations.
Doing so in an online manner additionally enables the integration of feedback
for adaptive behavior in previously unseen environments. We formulate an
online method for learning implicit signed distance fields represented as
piecewise polynomial basis functions. Our method uses a simple incremental
least squares approach and regularization scheme in order to approximate
distance fields from incoming surface points and normals. Accuracy and
performance of the piecewise polynomial representation can be balanced through
interpretable hyperparameters like polynomial degree and number of segments.
Model behavior can also easily be influenced by imposing priors. Our
representation does not require storage of training data and can be run in
real-time on both GPU and CPU. We evaluate its accuracy on a set of diverse
household objects, with comparisons to Gaussian process and neural network
counterparts. The smoothness and continuity of our representation are
highlighted by using an evolving model to guide the movement of a physical
manipulator.

##  II Implicit environment representations

In robotics, widely adopted scene representations use mapping approaches that
rely on discretized occupancy grids and account for uncertainty [5].
Subsequent methods explicitly store distance information for real-time usage
in dynamic settings by introducing distance fields [1, 6]. Following
advancements in computer vision, recent work has been exploring implicit
representations as scalable and compact alternatives for describing scenes
without storing data in dense grids. Implicit representations of distance,
volume, and color information have found application in robotics, with initial
uses in navigation [7] and mapping [8]. Recent work uses implicit
representations as visual encodings for grasping [9], whole-body manipulation
[2], human-robot interaction [10], planning [11], and control [12].

###  II-A Implicit signed distance functions

Implicit signed distance functions model geometry through continuous
functions, thus decoupling memory from spatial resolution. Seminal work
utilizes neural networks for shape representation, showing higher performance
than point cloud, mesh, or grid-based counterparts [13]. Additional efforts
have been put into investigating regularization and modeling methods that
allow learning such representations in online scenarios using raw point cloud
data [14, 15]. Recent methods introduce neural architectures that jointly
represent SDF and color to track and reconstruct unknown objects [16].
Similarly, Neural radiance fields (NeRFs) [17] jointly encode volume density
and color. They have garnered much attention as environment representations,
with recent extensions targeting real-time rendering [18] and dynamic scenes
[19]. However, NeRFs do not offer direct access to distance or derivative
information, which can be beneficial for interpreting task execution.
Furthermore, neural representations often require large amounts of data and do
not translate readily to lower data regimes found in modalities such as
tactile or proximity sensing.

###  II-B Gaussian process implicit surfaces

Probabilistic models like Gaussian process implicit surfaces (GPIS) have been
used to represent distances with account for uncertainty [20]. To extend the
approach for mapping purposes, scaling issues of the Gaussian process have
been addressed through the use of clustering and hierarchical models [21].
Subsequent combinations with implicit regularization methods enable accurate
modeling of unsigned distances [22]. These methods were later combined to give
a unified mapping, odometry, and planning framework [23].

###  II-C Basis function representations

Basis functions can describe complex representations as weighted
superpositions of simple signals. In robotics, they are well-known as the
underlying representation used to encode movement primitives [24, 25]. A
detailed review can be found in [4]. Their role in computer graphics extends
to higher-dimensional input space to implicitly represent shapes using SDFs
[26]. Learning such representations from point clouds and normals can be
achieved simply by solving a linear system of equations using a least squares
approach or iterative optimization procedures [27]. The resulting implicit SDF
models are continuous and smooth combinations of shape primitives with
analytical access to distance and gradients. A recent example utilizes
piecewise polynomial basis functions to represent detailed SDFs for high-
definition rendering in simulation settings [3]. The following section
describes an online formulation of piecewise polynomial SDF based on
incremental learning, with the aim of guiding movement in manipulation tasks.

##  III Piecewise polynomial SDF

###  III-A Bernstein polynomial basis functions

Figure 2: Incremental model updates used to model a 2D shape, starting from a
circular prior. Sampled points and normals are shown in dark blue, and the
reconstructed zero-level contour in black. The reconstructed SDF is visualized
as a color map with distance contours. The normal ray and regularization
points of a single sample are displayed in the second image.

The value of a univariate function $f(x)$ at input $x$ can be represented as a
weighted sum of $K$ basis functions with

$f(x)=\sum_{k=1}^{K}\bm{\phi}_{k}(x)\;w_{k}=\bm{\phi}(x)\;\bm{w},$ (1)

where $\bm{\phi}$ can come from any family of basis functions. For our SDF
representation, we use Bernstein polynomial basis functions, which can be
computed as

$\bm{\phi}_{k}(x)=\frac{(K-1)!}{(k-1)!(K-k)!}\;{(1-x)}^{K-k}\;x^{k-1},$ (2)

$\forall k\in\\{1,\ldots,K\\}$. Instead of considering a global encoding which
might require the use of high-order polynomials, we split the problem into a
set of local fitting problems that can consider lower-order polynomials. We
retain $C^{1}$ continuity between the concatenated segments by adding
constraints on the weights of the form

$\displaystyle w_{K}^{a}$ $\displaystyle=w_{0}^{b}$ (3) $\displaystyle
w_{1}^{b}$ $\displaystyle=-w_{K-1}^{a}+2w_{0}^{b},$ (4)

where $a$ and $b$ are concatenated polynomials of order $K$, and $w_{k}^{a}$
is used to denote the $k$-th weight of polynomial $a$. Polynomial bases and
their derivatives can then be expressed in matrix form as

$\displaystyle\bm{\phi}(x)$ $\displaystyle=\bm{T}(x)\bm{B}\bm{C},$ (5)
$\displaystyle\frac{\partial\bm{\phi}(x)}{\partial x}$
$\displaystyle=\frac{\partial\bm{T}(x)}{\partial x}\bm{B}\bm{C},$ (6)

with $\bm{T}(x)=\left[1~{}x~{}x^{2}~{}\cdots~{}x^{K}\right]$ being a
polynomial feature map of input $x$, $\bm{B}$ the corresponding Bernstein
coefficient matrix, and $\bm{C}$ a constraint matrix of the form

$\bm{C}=\begin{bmatrix}1&0&\cdots&0&0&\cdots\\\ 0&1&\cdots&0&0&\cdots\\\
\vdots&\vdots&\ddots&\vdots&\vdots&\ddots\\\ 0&0&\cdots&1&0&\cdots\\\
0&0&\cdots&0&1&\cdots\\\ 0&0&\cdots&0&1&\cdots\\\ 0&0&\cdots&-1&2&\cdots\\\
\vdots&\vdots&\ddots&\vdots&\vdots&\ddots\end{bmatrix},$ (7)

enforcing (3) and (4).

Successive Kronecker products can be used to extend the described
representation to any number of input and output dimensions. For clarity and
visualization purposes, we will continue the method description for a two-
dimensional case. Namely, an extension to two-dimensional input space
(Cartesian coordinates) and one-dimensional output (signed distance) can be
calculated as

$\displaystyle\bm{\Psi}(x,y)$
$\displaystyle=\bm{\phi}(x)\,\otimes\,\bm{\phi}(y),$ (8)

with partial derivatives and gradient computed analytically as

$\displaystyle\frac{\partial\bm{\Psi}(x,y)}{\partial x}$
$\displaystyle=\frac{\partial\bm{\phi}(x)}{\partial
x}\,\otimes\,\bm{\phi}(y),$ (9)
$\displaystyle\frac{\partial\bm{\Psi}(x,y)}{\partial y}$
$\displaystyle=\bm{\phi}(x)\,\otimes\,\frac{\partial\bm{\phi}(y)}{\partial
y},$ (10) $\displaystyle\bm{\nabla}\bm{\Psi}(x,y)$
$\displaystyle=\frac{\partial\bm{\Psi}(x,y)}{\partial
x}\,\otimes\,\frac{\partial\bm{\Psi}(x,y)}{\partial y},$ (11)

This can then be used to compute the distance and gradient values of the SDF
at coordinate $(x,y)$ with

$\displaystyle f(x,y)$ $\displaystyle=\bm{\Psi}(x,y)\,\bm{w},$ (12)
$\displaystyle\bm{\nabla}f(x,y)$
$\displaystyle=\bm{\nabla}\bm{\Psi}(x,y)\,\bm{w}.$ (13)

The same procedure can be applied to calculate the Laplacian
$\bm{\Delta}f(x,y)=\bm{\Delta}\bm{\Psi}(x,y)\,\bm{w}$ and higher-order
derivatives. The above representation extends analogously to accommodate
three-dimensional Cartesian coordinates as input by applying an additional
Kronecker product.

###  III-B Computation of weights

To approximate the SDF using polynomial basis functions, any method capable of
solving a system of linear equations of the form $\bm{A}\bm{w}=\bm{s}$ can be
employed. The simplest case can utilize a batch least squares estimate of the
form

$\displaystyle\bm{w}$
$\displaystyle=(\bm{A}^{\top}\bm{A})^{-1}\bm{A}^{\top}\bm{s},$ (14)

or ridge regression as the regularized variant [28].

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

Figure 3: Mesh and SDF reconstruction results for objects from the YCB test
set: (a) 003_cracker_box, (b) 006_mustard_bottle, (c) 011_banana, (d)
016_pear, (e) 019_pitcher_base, (f) 021_bleach_cleanser, (g) 048_hammer, (h)
035_power_drill, (i) 063-a_marbles, (j) 053_mini_soccer_ball. All models were
learned from $800$ non-uniformly sampled points and normals.

We use quadratic error terms in order to evaluate the fitting of distance and
normal data for $N$ incoming samples

$\displaystyle c_{d}(\bm{x}_{n})$
$\displaystyle=\left|\left|\bm{\Psi}(\bm{x}_{n})\bm{w}\right|\right|^{2},$
(15) $\displaystyle c_{g}(\bm{x}_{n})$
$\displaystyle=\left|\left|\bm{\nabla}\bm{\Psi}(\bm{x}_{n})\bm{w}-\bm{g}_{n}\right|\right|^{2},$
(16)

with $\bm{x}_{n}=\left(x_{n},y_{n}\right)$ denoting the $n$-th input sample,
and $\bm{g}_{n}$ the corresponding ground truth normal. An additional tension
term is used to constrain the curvature of the resulting distance field on $R$
control points

$\displaystyle c_{t}(\bm{x}_{r})$
$\displaystyle=\left|\left|\bm{\Delta}\bm{\Psi}(\bm{x}_{r})\bm{w}\right|\right|^{2},$
(17)

$\forall r\in\\{1,\ldots,R\\}$.

The model weights are learned by minimizing a combined cost

$\displaystyle
c=\sum_{n=1}^{N}\lambda_{d}^{2}c_{d}(\bm{x}_{n})+\lambda_{g}^{2}c_{g}(\bm{x}_{n})+\sum_{r=1}^{R}\lambda_{t}^{2}c_{t}(\bm{x}_{r}),$
(18)

with cost-tuning coefficients $\lambda_{d}$, $\lambda_{g}$, and $\lambda_{t}$.
We construct our input and output vectors as concatenations of distance,
normal, and tension features

$\displaystyle\bm{\Psi}_{d}$
$\displaystyle=\left[\bm{\Psi}(\bm{x}_{1})\cdots\bm{\Psi}(\bm{x}_{N})\right]^{\top},$
(19) $\displaystyle\bm{\nabla\Psi}_{g}$
$\displaystyle=\left[\bm{\nabla\Psi}(\bm{x}_{1})\cdots\bm{\nabla\Psi}(\bm{x}_{N})\right]^{\top},$
(20) $\displaystyle\bm{\Delta\Psi}_{t}$
$\displaystyle=\left[\bm{\Delta\Psi}(\bm{x}_{1})\cdots\bm{\Delta\Psi}(\bm{x}_{R})\right]^{\top}.$
(21)

In the above, multidimensional features are flattened by stacking components
to keep compatible dimensions. We finally minimize (18) by calculating (14)
with

$\displaystyle\bm{A}=\begin{bmatrix}\lambda_{d}\bm{\Psi}_{d}\\\
\lambda_{g}\bm{\nabla}\bm{\Psi}_{g}\\\
\lambda_{t}\bm{\Delta}\bm{\Psi}_{t}\end{bmatrix},\quad\bm{s}=\begin{bmatrix}\bm{0}_{d}\\\
\lambda_{g}\bm{g}\\\ \bm{0}_{t}\end{bmatrix},$ (22)

where $\bm{g}$ is a vector of sampled normal components, and $\bm{0}_{d}$ and
$\bm{0}_{t}$ are zero vectors with lengths compatible to $\bm{\Psi}_{d}$ and
$\bm{\Delta}\bm{\Psi}_{g}$, respectively.

###  III-C Incremental formulation

For online approximation, we employ an incremental variant of the least
squares algorithm which allows us to gradually refine an initial estimate by
providing samples one-by-one or in batches. Similar approaches have previously
been used in the context of control [29]. The algorithm exploits the Sherman-
Morrison-Woodbury relations [30] which connect subsequent inverses of a matrix
after small-rank perturbations. After initializing the weight precision matrix
$\bm{P}=\mathrm{cov}(\bm{w})^{-1}$ to $\bm{P}_{0}$, it can be incrementally
updated as

$\bm{P}_{\\!\scriptscriptstyle\mathrm{new}}=\bm{P}-\underbrace{\bm{P}\bm{A}_{n}^{\scriptscriptstyle\top}\left(\sigma^{2}\bm{I}+\bm{A}_{n}\bm{P}\bm{A}_{n}^{\scriptscriptstyle\top}\right)^{-1}}_{\bm{K}}\bm{A}_{n}\bm{P},$
(23)

where $\sigma^{2}$ is the measurement noise variance, $\bm{A}_{n}$ the input
matrix, and $\bm{I}$ an identity matrix of compatible dimensions. Starting
from a prior $\bm{w}=\bm{w}_{0}$, weight updates can then be calculated by
using the Kalman gain $\bm{K}$

$\bm{w}_{\\!\scriptscriptstyle\mathrm{new}}=\bm{w}+\bm{K}\Big{(}\bm{s}_{n}-\bm{A}_{n}\bm{w}\Big{)}.$
(24)

The above iterative computation has no requirement of storing the training
points and enables us to impose priors on our model through $\bm{P}_{0}$ and
$\bm{w}_{0}$. We initialize the precision matrix as
$\bm{P}_{0}=(\bm{M}^{\top}\bm{M})^{-1}$, with $\bm{M}=\bm{B}\bm{C}$ following
(5). The evolving precision matrix can be used to incrementally track weight
covariance as the model is learned.

Algorithm 1 Incremental computation of weights.

$\bm{P}=\bm{P}_{0}$ // Initialize precision matrix

$\bm{w}=\bm{w}_{0}$ // Initialize weights

for  _$n\leftarrow 1$ to $N$_ do

$\bm{A}_{n}=\bm{A}(\bm{x}_{n})$ // Construct input matrix
$\bm{K}=\bm{P}\bm{A}_{n}^{\scriptscriptstyle\top}\left(\sigma^{2}\bm{I}+\bm{A}_{n}\bm{P}\bm{A}_{n}^{\scriptscriptstyle\top}\right)^{-1}$//
Compute gain  $\bm{P}\leftarrow\bm{P}-\bm{K}\bm{A}_{n}\bm{P}$ // Update
precision matrix
$\bm{w}\leftarrow\bm{w}+\bm{K}\Big{(}\bm{s}_{n}-\bm{A}_{n}\bm{w}\Big{)}$ //
Update weights

end for

In order for our model to accurately approximate a distance function, the
tension term needs to be enforced throughout the input space. We achieve this
in an online setting by uniformly sampling a number of control points on the
normal rays of incoming surface samples, as displayed in Figure 2. The full
computation steps are summarized in Algorithm 1. We apply the same approach
for three-dimensional inputs, with example reconstructions shown in Figures 1
and 3.

##  IV Experiments

###  IV-A Reconstruction accuracy

To evaluate the reconstruction accuracy of our method we use error metrics
similar to [15]. Reconstructed distances are evaluated using the mean absolute
error (MAE)

$\displaystyle\text{MAE}(\bm{x})=\big{|}\hat{s}(\bm{x})-s(\bm{x})\big{|},$
(25)

with $\hat{s}(\bm{x})$ denoting the estimated signed distance at point
$\bm{x}$, and $s(\bm{x})$ the ground truth value. The accuracy of our model is
first tested on different grid sizes by using cubic polynomials with a
variable number of segments.

Figure 4: Mean absolute error (MAE) using cubic polynomials with a varying
number of segments.

Figure 4 displays the MAE results on our test data, which comprises of real
point clouds from the Yale-CMU-Berkeley (YCB) dataset [31]. The test set
consists of 10 household objects of diverse shapes, depicted in Figure 3.
Ground truth SDFs are reconstructed from high-definition meshes with 512k
polygons.

We compare the accuracy of our approach against two methods used in online
settings: a LogGP model using the Matérn 3/2 kernel as in [22], and a 4-layer
neural network with 256 neurons in each layer, using the loss and positional
embedding described in [15]. Our comparison model uses cubic Bernstein
polynomials with 6 segments per input dimension. All methods are evaluated on
the same point cloud and normal data for each object from the test set.
Accuracy is compared across different data volumes by varying the number of
sampled points. Since LogGP models unsigned distance, we evaluate it against
the absolute value of the ground truth. Figure 5(a) shows the resulting MAE
comparisons. Distance accuracy near and far from object surfaces for different
amounts of training samples is displayed in Table I.

(a) Mean absolute error (MAE)

(b) Gradient cosine distance (GCD)

Figure 5: Distance and gradient reconstruction accuracy compared on varying
amounts of data.

$|s|<0.05$
---
Samples | Ours | GP | NN
5000 | $$0.0574$\pm$0.0389$$ | $$0.0577$\pm$0.0379$$ | $$0.0245$\pm$0.0066$$
2000 | $$0.0562$\pm$0.0366$$ | $$0.0585$\pm$0.0425$$ | $$0.0385$\pm$0.0226$$
800 | $$0.0558$\pm$0.0368$$ | $$0.0539$\pm$0.0395$$ | $$0.0386$\pm$0.0249$$

$|s|>0.05$
---
Samples | Ours | GP | NN
5000 | $$0.1043$\pm$0.0525$$ | $$0.0926$\pm$0.0392$$ | $$0.1174$\pm$0.0561$$
2000 | $$0.1044$\pm$0.0551$$ | $$0.0916$\pm$0.0325$$ | $$0.0818$\pm$0.0339$$
800 | $$0.1079$\pm$0.0522$$ | $$0.1252$\pm$0.0461$$ | $$0.0798$\pm$0.0299$$

TABLE I: Comparison of the mean absolute error (MAE) near and far from object
surfaces for varying numbers of training samples.

Additional comparisons are made with respect to reconstructed distance
gradients by calculating the gradient cosine distance (GCD)

$\displaystyle
GCD(\bm{x})=1-\frac{\nabla_{\bm{x}}\hat{s}(\bm{x})\cdot\nabla_{\bm{x}}s(\bm{x})}{\|\nabla_{\bm{x}}\hat{s}(\bm{x})\|\|\nabla_{\bm{x}}s(\bm{x})\|},$
(26)

with $\nabla_{\bm{x}}\hat{s}(\bm{x})$ denoting the estimated distance
gradient, and $\nabla_{\bm{x}}s(\bm{x})$ the corresponding ground truth.
Gradients are recovered analytically for our method and numerically for the NN
and GP models. Ground truth gradients are calculated numerically from
corresponding ground truth SDFs. All numerical calculations are done on a
dense grid of $128^{3}$ points. Figure 5(b) displays the GCD comparisons. For
qualitative evaluation, Figure 3 shows the mesh and distance fields of the
test objects, reconstructed from learned piecewise polynomial representations.

###  IV-B Computation time & memory requirements

The computation time of updates and queries to our model increases
quadratically based on the number of weights, which is determined by the
desired number of segments and basis functions (polynomial degree).
Computation time and memory requirements do not increase with the total number
of training points, and we only store the combination weights of our basis
function representation. For a set input dimension, the time complexity of a
single incremental update is $\mathcal{O}(B^{3}+S^{2}K^{2}B+SKB^{2})$, with
$S$ denoting the number of segments, $K$ the number of basis functions, and
$B$ the sample batch size. Figure 6 shows mean update times for different
numbers of segments and varying batch sizes using a PyTorch implementation and
NVIDIA GeForce MX550 GPU.

Figure 6: Update time for cubic polynomials with varying numbers of segments
and different batch sizes. The desired real-time cutoff of $30~{}ms$ is
denoted by a horizontal line. Figure 7: A visualization of our physical
experiment and results, showing the evolution of a piecewise polynomial SDF
model starting from a spherical prior. As $n$ samples are collected, the
updated model is queried in real-time to control the manipulator by following
a tangential trajectory on the SDF level-sets while keeping normal
orientation.

During accuracy evaluation, the mean time of a single incremental update for
$S=6$, $K=4$, and $B=10$ was $0.1176~{}s$ on the described setup. Update time
can be further reduced by reducing the density of sampled regularization
points. For queries and reconstruction, our method scales linearly with the
number of weights, as we calculate only a weighted sum of basis functions. The
time complexity of querying our model for a batch of $B$ points is
$\mathcal{O}(SKB)$ with mean time $0.17~{}ms$ for a single query point in our
tests. The mean time of full SDF reconstruction on a dense $128^{3}$ grid,
required only for visualization of the results, was $28.79~{}s$ using the
marching cubes algorithm [32].

###  IV-C Physical experiment

In order to test the viability of our method in a real-world setting, we
perform a physical experiment in an online learning scenario using a Franka
Emika 7-DoF manipulator. Starting from a spherical prior, we incrementally
update our model with sparse surface points and estimated normals collected by
an in-hand Intel RealSense D415 sensor. At each timestep, we query normals
from our evolving SDF model to keep the camera oriented toward the object
while following a tangential trajectory on the level sets. Model updates are
done in real-time and used as feedback for our controller, with a mean
timestep of $31.4~{}ms$ on a $3.6~{}GHz$ Intel Core i9-9900K CPU. To enable
real-time operation on CPU, our SDF model uses cubic polynomials and 4
segments per input dimension with point-by-point updates. Figure 7 shows our
setup and the evolution of an SDF approximated from sparse partial point cloud
data.

##  V Discussion

The results in Figure 4 demonstrate a reduction in the reconstruction error
when increasing the number of segments. For cubic polynomials, this decline
becomes less pronounced beyond 5 segments, partly due to the size and
complexity of our test set. Qualitatively, employing a higher number of
segments results in more detailed reconstructions, a characteristic that might
not be fully captured by the MAE metric. For instance, comparing Figure 3(b),
reconstructed from a model using 6 segments, with the final result in Figure
7, reconstructed from a model using 4 segments, shows this effect. The number
of segments and polynomial degree can be adjusted to balance accuracy and
performance for the task at hand. Evaluations in Figure 5 and Table I
demonstrate consistent performance across various data volumes. Employing 6
segments per input dimension yields MAE and GCD values similar to those of the
Gaussian process and neural network comparison models. These results are
maintained both near object surfaces and at greater distances, affirming the
effectiveness of our regularization approach. Qualitative reconstructions in
Figure 3 display visually accurate distance and mesh reconstruction results
across our test set. It can be noted that the current regularization approach,
paired with the tested hyperparameters, tends to over-smooth and compromise
detail. This is especially visible when modeling thin or sharp objects, as
depicted by Figure 3(g).

The computation time and memory requirements of our formulation are
independent of the total number of training points. As illustrated in Figure
6, the update time scales quadratically with the number of segments and
cubically with batch size. Consequently, this constrains the applicability of
our current formulation to smaller environments and data regimes typically
found in tabletop manipulation settings. Our approach achieves fast query
times as inference amounts to calculating a weighted superposition of basis
functions. The physical experiment showcases real-time updates and
intermediary reconstruction results, starting from a spherical prior. It
demonstrates that continuity, smoothness, and fast access to distance and
analytical gradients enable direct usage of our model for feedback-loop
control, while validating it in a scenario with higher noise and sparse
samples.

Future work will investigate the adaptation of the described method to
clustering or hierarchical models, such as octrees. These approaches have been
shown to improve SDF reconstructions and enable the handling of larger and
more complex environments [3]. Varying other hyperparameters like polynomial
degree or family of basis functions might yield additional improvements to the
accuracy and detail of the reconstructed SDFs. Further efforts in scaling and
increasing performance might focus on using local weight updates and optimized
computation pipelines. On the method level, an interesting direction would
involve further analysis of the evolving precision matrix to evaluate the
quality of intermediary results during the learning process, or as an
uncertainty measure in planning scenarios. The use of incrementally updated
priors might also be of particular interest for extensions to dynamic scenes.
Finally, combinations of multiple sensing modalities (e.g., tactile and
proximity) need to be further explored and validated on a physical setup.
Further experiments could also make use of the analytic representation by
exploiting Boolean operators to combine the SDFs of multiple bodies and
evaluate overlap (e.g., for uses in whole-body manipulation).

##  VI Conclusion

We presented an online formulation of piecewise polynomial signed distance
functions. Starting from an arbitrary prior shape, our method incrementally
builds a continuous and smooth distance representation from incoming surface
point clouds and normal data. The results display reconstruction accuracy
similar to Gaussian process and neural network comparison models on a test set
consisting of various household objects. We have shown that memory and
computation-time requirements of our underlying model do not increase with the
total number of training points, and highlight the possibility of real-time
updates. Finally, we validated our approach on a physical manipulator with
noisy observations by using the evolving distance model for feedback-loop
control.

## References

  * [1] H. Oleynikova, Z. Taylor, M. Fehr, R. Siegwart, and J. Nieto, “Voxblox: Incremental 3d euclidean signed distance fields for on-board mav planning,” in Proc. IEEE/RSJ IROS, pp. 1366–1373, 2017.
  * [2] Y. Li, Y. Zhang, A. Razmjoo, and S. Calinon, “Learning robot geometry as distance fields: Applications to whole-body manipulation,” arXiv:2307.00533, 2023.
  * [3] E. Pujol and A. Chica, “Adaptive approximation of signed distance fields through piecewise continuous interpolation,” Computers & Graphics, vol. 114, pp. 337–346, 2023.
  * [4] S. Calinon, “Mixture models for the analysis, edition, and synthesis of continuous time series,” in Mixture Models and Applications (N. Bouguila and W. Fan, eds.), pp. 39–57, Springer, Cham, 2019.
  * [5] A. Hornung, K. M. Wurm, M. Bennewitz, C. Stachniss, and W. Burgard, “OctoMap: An efficient probabilistic 3D mapping framework based on octrees,” Autonomous Robots, vol. 34, no. 3, pp. 189–206, 2013.
  * [6] L. Han, F. Gao, B. Zhou, and S. Shen, “Fiesta: Fast incremental euclidean distance fields for online motion planning of aerial robots,” Proc. IEEE/RSJ IROS, pp. 4423–4430, 2019.
  * [7] M. Adamkiewicz, T. Chen, A. Caccavale, R. Gardner, P. Culbertson, J. Bohg, and M. Schwager, “Vision-only robot navigation in a neural radiance world,” IEEE Robotics and Automation Letters (RA-L), vol. 7, no. 2, pp. 4606–4613, 2022.
  * [8] E. Sucar, S. Liu, J. Ortiz, and A. Davison, “iMAP: Implicit mapping and positioning in real-time,” Proc. IEEE/CVF ICCV, pp. 6209–6218, 2021.
  * [9] M. Breyer, J. J. Chung, L. Ott, S. Roland, and N. Juan, “Volumetric grasping network: Real-time 6 dof grasp detection in clutter,” in Proc. CoRL, pp. 1602–1611, 2020.
  * [10] P. Liu, K. Zhang, D. Tateo, S. Jauhri, J. Peters, and G. Chalvatzaki, “Regularized deep signed distance fields for reactive motion generation,” in Proc. IEEE/RSJ IROS, pp. 6673–6680, 2022.
  * [11] D. Driess, J.-S. Ha, M. Toussaint, and R. Tedrake, “Learning models as functionals of signed-distance fields for manipulation planning,” in Proc. CoRL, 2021.
  * [12] Y. Li, S. Li, V. Sitzmann, P. Agrawal, and A. Torralba, “3d neural scene representations for visuomotor control,” in Proc. CoRL, pp. 112–123, 2021.
  * [13] J. J. Park, P. R. Florence, J. Straub, R. A. Newcombe, and S. Lovegrove, “Deepsdf: Learning continuous signed distance functions for shape representation,” Proc. IEEE CVPR, pp. 165–174, 2019.
  * [14] A. Gropp, L. Yariv, N. Haim, M. Atzmon, and Y. Lipman, “Implicit geometric regularization for learning shapes,” in Proc. ICML, vol. 119, pp. 3789–3799, 2020.
  * [15] J. Ortiz, A. Clegg, J. Dong, E. Sucar, D. Novotny, M. Zollhoefer, and M. Mukadam, “isdf: Real-time neural signed distance fields for robot perception,” in Proc. RSS, 2022.
  * [16] B. Wen, J. Tremblay, V. Blukis, S. Tyree, T. Muller, A. Evans, D. Fox, J. Kautz, and S. Birchfield, “Bundlesdf: Neural 6-dof tracking and 3d reconstruction of unknown objects,” Proc. IEEE CVPR, pp. 606–617, 2023.
  * [17] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, “Nerf: Representing scenes as neural radiance fields for view synthesis,” Comm. ACM, vol. 65, p. 99–106, Dec 2021.
  * [18] B. Kerbl, G. Kopanas, T. Leimkühler, and G. Drettakis, “3d gaussian splatting for real-time radiance field rendering,” ACM Transactions on Graphics, vol. 42, July 2023.
  * [19] A. Pumarola, E. Corona, G. Pons-Moll, and F. Moreno-Noguer, “D-NeRF: Neural Radiance Fields for Dynamic Scenes,” in Proc. IEEE CVPR, 2020.
  * [20] S. Dragiev, M. Toussaint, and M. Gienger, “Gaussian process implicit surfaces for shape estimation and grasping,” in Proc. IEEE ICRA, pp. 2845–2850, 2011.
  * [21] B. Lee, C. Zhang, Z. Huang, and D. D. Lee, “Online continuous mapping using gaussian process implicit surfaces,” Proc. IEEE ICRA, pp. 6884–6890, 2019.
  * [22] L. Wu, K. M. B. Lee, L. Liu, and T. Vidal-Calleja, “Faithful euclidean distance field from log-gaussian process implicit surfaces,” IEEE Robotics and Automation Letters (RA-L), vol. 6, no. 2, pp. 2461–2468, 2021.
  * [23] L. Wu, K. M. B. Lee, C. Le Gentil, and T. Vidal-Calleja, “Log-gpis-mop: A unified representation for mapping, odometry, and planning,” IEEE Transactions on Robotics (T-RO), vol. 39, no. 5, pp. 4078–4094, 2023.
  * [24] A. J. Ijspeert, J. Nakanishi, H. Hoffmann, P. Pastor, and S. Schaal, “Dynamical Movement Primitives: Learning Attractor Models for Motor Behaviors,” Neural Computation, vol. 25, pp. 328–373, 02 2013.
  * [25] A. Paraschos, C. Daniel, J. Peters, and G. Neumann, “Probabilistic movement primitives,” in Proc. NIPS, pp. 2616–2624, 2013.
  * [26] B. Jüttler and A. Felis, “Least-squares fitting of algebraic spline surfaces,” Advances in Computational Mathematics, vol. 17, pp. 135–152, Jul 2002.
  * [27] G. Taubin, “Smooth signed distance surface reconstruction and applications,” in Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications, pp. 38–45, 2012.
  * [28] A. E. Hoerl and R. W. Kennard, “Ridge regression: Biased estimation for nonorthogonal problems,” Technometrics, vol. 12, no. 1, pp. 55–67, 1970.
  * [29] J.-A. Ting, S. Vijayakumar, and S. Schaal, “Locally weighted regression for control,” in Encyclopedia of Machine Learning, pp. 613–624, Springer, 2010.
  * [30] W. W. Hager, “Updating the inverse of a matrix,” SIAM Rev., vol. 31, pp. 221–239, 1989.
  * [31] B. Calli et al., “Yale-cmu-berkeley dataset for robotic manipulation research,” Intl Journal of Robotics Research, vol. 36, no. 3, pp. 261–268, 2017.
  * [32] W. E. Lorensen and H. E. Cline, “Marching cubes: A high resolution 3d surface construction algorithm,” in Seminal graphics: pioneering efforts that shaped the field, pp. 347–353, 1998.