# ExtremeMETA: High-speed Lightweight Image Segmentation Model by Remodeling
Multi-channel Metamaterial Imagers

Quan Liu, Brandon T. Swartz, Ivan Kravchenko, Jason G. Valentine, Yuankai Huo
Yuankai Huo is the corresponding author, Vanderbilt University, Nashville, TN
37212, USA. E-mail: (yuankai.huo@vanderbilt.edu)Quan Liu, Brandon T. Swartz
and Jason G. Valentine are with the Vanderbilt University, Nashville, TN
37212, USA. E-mail<EMAIL_ADDRESS><EMAIL_ADDRESS>jason.g.valentine@vanderbilt.edu).Ivan Kravchenko is with Oak Ridge National
Laboratory, Oak Ridge, TN 37830, USA. E-mail<EMAIL_ADDRESS>
###### Abstract

Deep neural networks (DNNs) have heavily relied on traditional computational
units like CPUs and GPUs. However, this conventional approach brings
significant computational burdens, latency issues, and high power consumption,
limiting their effectiveness. This has sparked the need for lightweight
networks like ExtremeC3Net. On the other hand, there have been notable
advancements in optical computational units, particularly with metamaterials,
offering the exciting prospect of energy-efficient neural networks operating
at the speed of light. Yet, the digital design of metamaterial neural networks
(MNNs) faces challenges such as precision, noise, and bandwidth, limiting
their application to intuitive tasks and low-resolution images. In this paper,
we propose a large kernel lightweight segmentation model, ExtremeMETA. Based
on the ExtremeC3Net, the ExtremeMETA maximizes the ability of the first
convolution layer by exploring a larger convolution kernel and multiple
processing paths. With the proposed large kernel convolution model, we extend
the optic neural network application boundary to the segmentation task. To
further lighten the computation burden of the digital processing part, a set
of model compression methods is applied to improve model efficiency in the
inference stage. The experimental results on three publicly available datasets
demonstrate that the optimized efficient design improved segmentation
performance from 92.45 to 95.97 on mIoU while reducing computational FLOPs
from 461.07 MMacs to 166.03 MMacs. The proposed the large kernel lightweight
model ExtremeMETA showcases the hybrid design’s ability on complex tasks.

###### Index Terms:

large convolution kernel, model compression, segmentation, meta-material.

##  I Introduction

In the realm of modern computer vision, digital neural networks play a pivotal
role. The convolutional neural network (CNN) stands out as arguably the most
extensively employed AI approach, particularly in tasks like image
classification, segmentation, and detection. Despite the advent of vision
transformer-based models, convolution remains integral for extracting local
image features. Presently, CNNs are typically implemented on computational
units like CPUs and GPUs. However, this conventional design approach brings
forth substantial challenges, including a formidable computational load,
notable latency issues, and heightened power consumption. These limitations
become particularly pronounced in applications such as the Internet of Things
(IoT), edge computing, and drone operations, where requires the lightweight
model for efficient analysis. Recognizing the critical need for DNN models
with reduced energy consumption and lower latency, the AI community has
embarked on a quest for more efficient solutions. Despite these efforts,
achieving energy-free and light-speed DNNs within the current research trends
seems to be an elusive goal.

Recent breakthroughs in optical computational units, including metamaterials
(refer to Fig. 1), have brought to light the potential for neural networks
that operate without energy consumption and at unprecedented speeds. The
current cutting-edge metamaterial neural network (MNN) takes on a hybrid form,
leveraging optical processors as a light-speed and energy-free front-end
convolutional operator alongside a digital feature aggregator. This inventive
approach serves to significantly reduce computational latency. By assigning
the convolution operations to optical units, more than 90 percent of the
floating-point operations (FLOPs) inherent in conventional CNN backbones like
VGG and ResNet are effectively off-loaded. This marks a noteworthy departure
from traditional architectures, opening up new avenues for efficient and high-
performance neural network designs. However, the hybrid design is
fundamentally influenced by the physical structure including the limited
kernel size, and channel number. Besides that, the hybrid system is also
limited by what can be fabricated as the first optical layer of the neural
network.

Figure 1: This study provides a hybrid pipeline for designing and optimizing a
large kernel digital neural network. The proposed ExtremeMETA is efficient for
segmentation tasks with less FLOPs in computation.

Based on our proposed LMNN model, the hybrid design achieves promising
performance on the classification task. However, there are limitations of
LMNN, namely: (1) the LMNN model can only perform image classification tasks
instead of model complex tasks like image segmentation and object detection;
(2) input images are in low resolution (28$\times$28), and (3) besides
leverage the computation burden to the optic, the digital part requires
efficiency improvement operation like model compression in the inference
stage.

In this paper, we propose a novel large kernel lightweight segmentation model
ExtremeMETA that maximizes the efficiency advantages of optic signal
computation, while also compressing the digital processing model to further
improve the model segmentation efficiency. To adapt the segmentation task on
large images, the proposed lightweight large kernel model achieves larger
receptive fields, the ability to larger image analysis, and covers general
vision tasks, image classification segmentation, and detection. Furthermore,
the complexity of the model digital processing part is explicitly addressed
via a set of model compression methods. We evaluate our design on image
segmentation tasks using three public datasets: the portrait dataset, the
Stanford dataset, and KITTI dataset. The proposed lightweight large kernel
model achieved superior segmentation accuracy as compared with the SOTA
segmentation model. Overall, the system’s contributions can be summarized in
four-fold:

  * •

We propose a new large convolution kernel CNN network to achieve a large
reception field, less energy consumption, and less latency.

  * •

We introduce the model re-parameterization to improve large convolution kernel
performance and sparse convolution kernel compression mechanism to compress
the multi-branch sparse-convolution design to a single layer for the hybrid
system implementation. The model compression mechanism improves the model
efficiency for digital processing.

  * •

The task limitations of large convolution hybrid models are explicitly
addressed via performing segmentation tasks on multiple datasets from
different categories.

The rest of the paper is organized as follows. In Section II, we introduce
background and related research relevant to large kernel convolution, model
compression, and optical neural networks on image processing tasks. In Section
III, our proposed lightweight lightspeed model is presented. It includes the
large kernel re-parameterization, sparse convolution compression, and multi-
path model compression. Section IV focuses on presenting the dataset and
experiment implementation details. Section V provides the experimental results
and ablation study. Then, in Sections VI and VII, we provide the discussion
and conclude our work.

##  II Related work

###  II-A Large kernel convolution design

In the realm of convolutional neural networks (CNNs), the design and
utilization of large kernel convolutions have garnered significant attention
in recent years. Numerous studies have explored the benefits of using larger
convolutional kernels, such as 7x7 or 11x11, to capture broader spatial
contexts and more intricate patterns within images [1, 2]. Early research
efforts focused on understanding the impact of kernel size on model
performance, with findings suggesting that larger kernels can lead to improved
feature extraction and recognition accuracy, especially for complex visual
tasks [3].

Building upon these findings, subsequent works have proposed various
strategies to incorporate the large kernel convolutions into CNN architectures
effectively. These strategies often involve modifying network architectures,
adjusting kernel sizes, or integrating multi-scale features to enhance the
robustness and versatility of CNN models [4, 5]. Additionally, advancements in
hardware acceleration and parallel processing have facilitated the efficient
implementation of large kernel convolutions, enabling their widespread
adoption across diverse computer vision applications [6, 7].

Overall, the related work on large kernel convolution design underscores its
pivotal role in advancing the capabilities of CNNs for tackling increasingly
complex and demanding visual recognition tasks [8, 9].

###  II-B Optic neural network

Optic neural networks (ONNs) have emerged as a promising paradigm for
accelerating neural network computations by leveraging the unique properties
of optical computing. Inspired by the principles of light-based signal
processing, ONNs exploit the parallelism, high bandwidth, and low energy
consumption inherent in optical systems to achieve significant computational
efficiency gains compared to traditional electronic implementations. A
considerable body of research has focused on exploring various aspects of
ONNs, including optical device design, system architectures, and algorithmic
frameworks tailored to optical computing platforms [10, 11, 12].

Early studies laid the groundwork for ONNs by demonstrating their potential
for accelerating matrix-vector multiplications, a fundamental operation in
neural network inference [13, 14]. Subsequent works have extended ONN
capabilities to encompass more complex neural network layers and
architectures, paving the way for practical applications in tasks such as
image classification, object detection, and natural language processing [15,
16].

Key challenges in ONN research include addressing optical noise, device
nonlinearity, and scalability issues, which require interdisciplinary efforts
spanning optics, photonics, and machine learning [17, 18]. Despite these
challenges, ONNs hold great promise for enabling ultra-fast and energy-
efficient neural network computations, with the potential to revolutionize
various domains of artificial intelligence and computing [19, 20].

###  II-C Convolution neural network model compression

In the field of convolutional neural networks (CNNs), model compression
techniques have garnered significant attention as a means to reduce the
computational complexity and memory footprint of deep learning models without
sacrificing performance. A diverse range of methods has been proposed to
compress CNNs, including pruning, quantization, low-rank approximation,
knowledge distillation, and weight sharing. Pruning techniques aim to remove
redundant or less important parameters from the network, thereby reducing its
size and computational cost [21, 22]. Quantization methods reduce the
precision of network parameters, often by representing weights and activations
with fewer bits, to decrease memory requirements and improve inference speed
[23]. Low-rank approximation techniques exploit the underlying structure of
weight matrices to factorize them into smaller, more computationally efficient
components [24]. Knowledge distillation involves training a compact ”student”
network to mimic the predictions of a larger ”teacher” network, transferring
knowledge from the latter to the former [25]. Additionally, weight sharing
approaches aim to reduce redundancy by sharing parameters across different
parts of the network [26].

Collectively, these model compression techniques offer effective strategies
for deploying CNNs on resource-constrained devices or accelerating inference
in large-scale deployment scenarios. Ongoing research in this area continues
to explore novel compression algorithms, optimization strategies, and
application-specific considerations to further improve the efficiency and
effectiveness of compressed CNN models.

##  III Method

Problem statement We extensively study the trainability of large kernels on
metamaterial neural networks (MNN) and unveil three main observations: (i)
traditional convolution kernel shows limited improvement on large images; (ii)
the MNN is only available on classification task; (iii) metamaterial
implementation limited the computation ratio on segmentation model which is
normally in complex structure.

Figure 2: Lightweight segmentation model with hybrid meta optics design. The
model has two parts: CoarseNet and FineNet. The large kernel block is composed
of depthwise convolution layers.

###  III-A Large convolution design with multiple path design

Limited by the image size and the task for the model, our previous proposed
model LMNN achieved the prediction performance with kernel size $9\times 9$.
Two major limitations exist when applying the large kernel design to the MNN:
(1) the metamaterial implementation limits the image size to a small range;
(2) only the classification task is available to be validated on the MNN model
when the segmentation task and detection task is too difficult to be
implemented under the optic implementation limitation. To address the
challenges, we proposed our model from two perspectives: (1) from kernel
design, we employ the large convolution kernel with parameterization design to
construct the convolution layer (larger than $9\times 9$); (2) from model
design, our proposed lightweight segmentation model based on the multipath
model structure composed by a course segmentation path and a light refinement
path proposed by [27].

Figure 3: Model compression on segmentation model digital processing part. The
left panel shows the multipath structure of the advanced C3 block. The right
panel shows the compression mechanism.

###  III-B Model compression with sparse convolution

Model compression is a crucial technique aimed at enhancing the efficiency of
deep learning models by reducing their size and computational demands while
maintaining their performance standards. Among the various strategies employed
in model compression, pruning, and quantization stand out as widely adopted
methodologies. Pruning, a prominent technique in model compression, involves
the systematic removal of redundant or unnecessary parameters from neural
networks. By identifying and eliminating connections that contribute minimally
to the model’s performance, pruning effectively reduces the model’s size and
computational requirements. This process allows for a more streamlined network
architecture without sacrificing accuracy, making it particularly valuable for
resource-constrained environments or deployment on edge devices.

We applied model compression and parameterization together for the sparse
convolution kernel which is shown in Fig. Sparse convolution refers to a
convolution operation where the kernel (filter) contains mostly zero values,
resulting in a sparse structure. When using a kernel size of 1x3 (1 row and 3
columns), the convolution operation typically involves sliding this kernel
over the input data and performing element-wise multiplication followed by
summation along the spatial dimensions.

$O_{h,w,c^{\prime}}=\sum_{i=0}^{2}\sum_{j=0}^{C-1}I_{h,w+i,j}\times
K_{0,i,j,c^{\prime}}$ (1)

I as the input tensor, K as the kernel tensor, O as the output tensor, and
$\times$ as the convolution operation.

For the ExtremeC3 block, we have three convolution paths with kernel size
$k\times k$, $1\times k$, and $k\times 1$. The compressed convolution kernel
follows Eq. Let’s denote the individual kernels as $k_{1\times k}$,
$k_{k\times k}$, $k_{k\times 1}$.

$K_{combined}(i,j)=w_{1xk}\times K_{1xk}(i,j)+w_{kxk}\times
K_{kxk}(i,j)+w_{kx1}\times K_{kx1}(i,j)$ (2)

The compressed multipath convolution block saves computation complexity in the
inference stage.

##  IV Data and experimental design

###  IV-A Data description

Three public datasets, EG1800 [28], Stanford Car dataset [29], and KITTI
dataset [30], were used to evaluate the lightweight large kernel model on
segmentation tasks. For the EG1800 dataset, we employ 1887 images in
$600\times 800$ resolution with semantic segmentation masks. The EG1800
dataset is collected from Flicker with the manually annotated mask of the
portrait. The Stanford Car dataset is composed of 16,185 RGB images of cars
with the point coordinate where the car is located in images. The KITTI
dataset is popular in mobile robotics and autonomous driving and features
diverse traffic scenarios captured using high-resolution RGB, grayscale stereo
cameras, and a 3D laser scanner. However, it lacks inherent ground truth
annotations for semantic segmentation. To adapt to the segmentation task, both
the Stanford Car dataset and the KITTI dataset need to address the annotation
limitation.

###  IV-B Data generation with foundation model

Regarding the Stanford Car dataset and KITTI dataset lacking of segmentation
annotation, we employ the Segment Anything Model (SAM) [31] to generate the
object mask based on the prompts of object location. The SAM model is a
foundation model that has a zero-shot ability to segment objects on new image
distributions. The RGB image of Standford Car and KITTI datasets and bounding
box coordinate is provided for the SAM model and SAM model will generate the
object masks. With the help of the SAM model, the RGB images with object mask
annotations are available for model training.

###  IV-C Large kernel digital design on segmentation model

The large kernel design is applied to the segmentation network’s first
convolution layer design. Since the first layer is designed to be substituted
by the metaoptic lens in the inference stage, our large kernel design is under
physical limitation. On the other hand, the optic lens provides light-speed
computation which we can take advantage of. Based on the multipath
segmentation network, the first convolution layers of both the Coarse-net part
and Fine-net part are redesigned with the large convolution kernel with
parameterization following the strategy in our previous work LMNN [32]. Since
the image is large compared with FashionMNIST previously used, our kernel size
is larger from $9\times 9$ to $15\times 15$. The channel number is expanded
from 12 to 48. The Larger convolution kernel and channel number provide the
large capability of the first layers and handle the complex situation.

###  IV-D Model design with optic constrain

Under the meta-optic fabrication limitation, the meta-optic layer has
limitations on both channel number and input size. The trade-off in model
performance between input size and channel number is discussed. The size-first
design uses the largest input image size under fabrication constrain. Channel-
first design prefers more channel numbers under the fabrication limitation.

###  IV-E Model compression efficiency

Besides enlarging the capability of the first layer, our proposed lightweight
segmentation network is compressed in the digital part. Since the model
compression affects the model’s complexity and efficiency, we evaluate if the
compressed model loses accuracy. To test the efficiency of the model
compression strategy, the model FLOPs, parameters, and FLOPs ratio of the
first convolution layer.

##  V Result

In this section, we evaluate our proposed lightweight segmentation network
with a simple model structure, using the EG1800 dataset, Stanford Car dataset,
and KITTI dataset. Since the Stanford Car dataset and KITTI dataset are car
images, we train the model and test the two datasets together.

###  V-A Segmentation performance on portrait dataset

We evaluate the lightweight segmentation model on EG1800 dataset together with
model parameters and first convolution FLOPs ratio. As shown in Table. I, the
original ExtremeC3 model cannot take advantage of the large convolution kernel
on the first layer, $15\times 15$ kernel shows even lower performance than
$11\times 11$. The model performance without the first convolution layer shows
a 2$\%$ drop compared with the ExtremeC3 model with $3\times 3$ kernel size.
Our proposed hybrid lightweight segmentation model achieves the best
performance with $15\times 15$ convolution kernel which has the same digital
computation FLOPs.

TABLE I: Segmentation performance on EG1800

Model | Kernel size | 1st conv FLOPs ($\%$) | Model FLOPs | Digital FLOPs | Test (mIoU)
---|---|---|---|---|---
ExtremeC3 | 3$\times$3 | 10.87 | 199.4 | 199.4 | 0.9249
11$\times$11 | 62.11 | 469.14 | 469.14 | 0.9323
15$\times$15 | 75.30 | 719.62 | 719.62 | 0.9301
Digital | N/A | N/A | 174.10 | 174.10 | 0.9086
Ours | 1$\times$1 | 2.80 | 182.06 | 174.10 | 0.9137
3$\times$3 | 10.87 | 199.40 | 174.10 | 0.9234
11$\times$11 | 59.68 | 431.81 | 174.10 | 0.9415
15$\times$15 | 63.36 | 475.16 | 174.10 | 0.9418

Model FLOPs and digital FLOPs unit is MMacs.

Besides improving the model performance with advanced design on the first
convolution layer, we evaluate the model efficiency improvement by model
compression. Following the experiment setting in Table. I, we applied model
compression, including sparse convolution kernel compression and multipath
parameterization, to each model design and shows the efficiency evaluation
matrix in Table. II. The compression method shows efficient computation on
digital FLOPs without affecting model performance (mIoU).

TABLE II: Segmentation performance on EG1800 after model compression

Model | Kernel size | 1st conv FLOPs ($\%$) | Model FLOPs | Digital FLOPs | Test (mIoU)
---|---|---|---|---|---
ExtremeC3 | 3$\times$3 | 11.33 | 191.32 | 191.32 | 0.9233
11$\times$11 | 63.21 | 461.07 | 461.07 | 0.9315
15$\times$15 | 76.16 | 711.55 | 711.55 | 0.9289
Digital | N/A | N/A | 166.03 | 166.03 | 0.9031
Ours | 1$\times$1 | 3.17 | 174.25 | 166.03 | 0.9121
3$\times$3 | 11.33 | 191.32 | 166.03 | 0.9217
11$\times$11 | 60.81 | 423.74 | 166.03 | 0.9404
15$\times$15 | 64.45 | 467.09 | 166.03 | 0.9420

Model FLOPs and digital FLOPs unit is MMacs.

###  V-B Segmentation performance on car dataset

To validate our lightweight segmentation model with more datasets, we conduct
experiments on the car dataset, including the Stanford Car dataset and KITTI
dataset both with semantic segmentation mask as ground truth. Since the
Stanford Car dataset and KITTI dataset are in different resolutions. Both the
Standford Car dataset and the KITTI dataset are used for model training.

TABLE III: Segmentation performance on car dataset

Model | Kernel size | Train (KITTI+Stanford) | Test (mIoU) | KITTI | Stanford
---|---|---|---|---|---
ExtremeC3 | 3*3 | 95.02 | 92.51 | 84.45 | 95.23
11*11 | 95.12 | 92.09 | 84.37 | 95.39
15*15 | 76.09 | 70.25 | 22.69 | 95.22
Digital | N/A | 93.31 | 89.11 | 78.47 | 94.27
Ours | 1*1 | 94.13 | 90.94 | 82.68 | 93.15
3*3 | 94.97 | 92.01 | 85.05 | 94.77
11*11 | 95.79 | 92.91 | 85.33 | 95.97
15*15 | 96.05 | 93.17 | 87.41 | 95.19

Model FLOPs and digital FLOPs unit is MMacs.

TABLE IV: Segmentation performance on car dataset after model compression

Model | Kernel size | 1st conv FLOPs (%) | Model FLOPs | Digital FLOPs | Test (mIoU)
---|---|---|---|---|---
ExtremeC3 | 3*3 | 11.33 | 191.32 | 191.32 | 91.36
11*11 | 63.21 | 461.07 | 461.07 | 92.45
15*15 | 76.16 | 711.55 | 711.55 | 70.01
Digital | N/A | N/A | 166.03 | 166.03 | 88.97
Ours | 1*1 | 3.17 | 174.25 | 166.03 | 90.94
3*3 | 11.33 | 191.32 | 166.03 | 94.25
11*11 | 60.81 | 423.74 | 166.03 | 95.32
| 15*15 | 64.45 | 467.09 | 166.03 | 93.05

Model FLOPs and digital FLOPs unit is MMacs.

###  V-C Ablation studies

Due to the fabrication limitation of the meta-lens array, the priority of
channel number and input image size need to be decided. The results of the
experiment are shown in Figure 4. The left panel illustrates how increasing
the input image size enhances performance compared to expanding the number of
channels in a convolution layer. The gray area highlights the performance
disparity in terms of mean Intersection over Union (mIoU). On the right panel,
the effectiveness of utilizing large convolution kernels is assessed. Circles
of various colors represent different convolution layer architectures, with
the area of each circle indicating the ratio of FLOPs (Floating Point
Operations per Second) for the layer when implemented using meta-optic
materials. The x-axis represents the model’s FLOPs, excluding the layer
intended for fabrication.

Figure 4: Model ablation study. Left panel: trade-off between input image size
and channel number of convolution layer. Right panel: model efficiency
visualization comparing model FLOPs and mIoU.

###  V-D Model compression

Figure 5 demonstrates that the compressed model achieves a reduction of 8
MMacs in FLOPs, decreasing from 174.10 MMacs to 166.03 MMacs. The right panel
indicates that the compressed model maintains equivalent performance to the
original model. This consistency in performance illustrates that our
ExtremeMETA not only enhances the efficiency of the digital components but
also contributes to the overall optimization of the hybrid system.

Figure 5: Model compression performance. Left panel: origin model,
ExtremeMETA, and compressed model parameters comparison; right panel: model
performance after compression.

##  VI Discussion

Given the demonstrated superior performance of large convolution kernels in
tasks such as image classification and segmentation, there exists substantial
potential for their application in a wider array of complex computer vision
tasks. Large convolution kernels have shown remarkable effectiveness in tasks
like image classification and segmentation, primarily due to their ability to
capture more extensive spatial information and intricate patterns within
images. This success suggests that employing large convolution kernels in
other computer vision tasks could yield significant improvements.

One such task is object detection, where accurately identifying and localizing
objects within images is crucial. By utilizing large convolution kernels, the
model can better discern the detailed features of objects, leading to more
precise detection results. This can be particularly beneficial in scenarios
with small or occluded objects, where finer details are essential for accurate
recognition as the results shown in the experiments on the car dataset.

Furthermore, in tasks involving image generation or synthesis, such as style
transfer or super-resolution, large convolution kernels can enhance the
model’s ability to capture intricate textures and details, resulting in more
realistic and high-fidelity output images. These kernels can effectively
extract and preserve fine-grained features, which are instrumental in
faithfully replicating the characteristics of the input images.

The application can be extended to video processing tasks like action
recognition or video segmentation, large convolution kernels can enhance the
model’s capability to analyze temporal and spatial dependencies across frames.
By incorporating information from a broader context, these kernels enable a
more robust understanding of dynamic scenes, leading to improved performance
in tasks requiring temporal coherence and contextual understanding.

The adoption of large convolution kernels holds promise for advancing various
complex computer vision tasks beyond traditional image classification and
segmentation. Their ability to capture intricate details and spatial
relationships makes them a valuable tool for enhancing the performance and
capabilities of computer vision models across diverse applications.

##  VII Conclusion

In conclusion, we have introduced a novel large kernel lightweight
segmentation model that harnesses the efficiency advantages of optical signal
computation while integrating digital processing model compression techniques
to further enhance segmentation efficiency. Our model offers larger receptive
fields tailored for segmentation tasks on large images, extending its
applicability to various vision tasks including image classification,
segmentation, and detection. Through extensive evaluations on diverse
datasets, including the portrait, Stanford, and KITTI datasets, our proposed
approach has demonstrated superior segmentation accuracy compared to state-of-
the-art models. Our contributions encompass the introduction of a novel large
convolution kernel CNN network for larger reception fields, reduced energy
consumption, and lower latency, alongside the introduction of model re-
parameterization and sparse convolution kernel compression mechanisms to
enhance model performance and efficiency in digital processing. By explicitly
addressing task limitations and conducting segmentation tasks on multiple
datasets from different categories, our work represents a significant step
forward in the development of efficient and effective segmentation models for
a wide range of computer vision applications.

## Acknowledgment

YH and QL acknowledge support from NIH under contract R01DK135597. YH is the
corresponding author. BTS and JGV acknowledge support from DARPA under
contract HR001118C0015, NAVAIR under contract N6893622C0030 and ONR under
contract N000142112468. Meta-optic devices were manufactured as part of a user
project at the Center for Nanophase Materials Sciences (CNMS), which is a US
Department of Energy, Office of Science User Facility, Oak Ridge National
Laboratory.

## References

  * [1] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” _arXiv preprint arXiv:1409.1556_ , 2014.
  * [2] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 1–9.
  * [3] M. D. Zeiler and R. Fergus, “Visualizing and understanding convolutional networks,” _arXiv preprint arXiv:1311.2901_ , 2014.
  * [4] C. Szegedy, S. Ioffe, and V. Vanhoucke, “Rethinking the inception architecture for computer vision,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 2818–2826.
  * [5] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016.
  * [6] X. Zhang, X. Zhou, M. Lin, and J. Sun, “Efficient and accurate approximations of nonlinear convolutional networks,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 1984–1992.
  * [7] M. Sun, Z. Liu, X. Wang, W. Qiao, and K. Lin, “Efficientnet: Rethinking model scaling for convolutional neural networks,” in _International Conference on Machine Learning_ , 2018, pp. 6105–6114.
  * [8] M. Lin, Q. Chen, and S. Yan, “Network in network,” _arXiv preprint arXiv:1312.4400_ , 2013.
  * [9] G. Huang, Z. Liu, L. van der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 4700–4708, 2017.
  * [10] Y. Shen, N. C. Harris, S. Skirlo, M. Prabhu, T. Baehr-Jones, M. Hochberg, X. Sun, S. Zhao, H. Larochelle, and D. Englund, “Deep learning with coherent nanophotonic circuits,” _Nature Photonics_ , vol. 11, no. 7, pp. 441–446, 2017.
  * [11] X. Lin, Y. Rivenson, D. Teng, L. Wei, H. Günaydın, Y. Zhang, and A. Ozcan, “All-optical machine learning using diffractive deep neural networks,” _Science_ , vol. 361, no. 6406, pp. 1004–1008, 2018.
  * [12] T. W. Hughes, M. Minkov, I. A. Williamson, Y. Shi, and S. Fan, “Training of photonic neural networks through in situ backpropagation and gradient measurement: supplementary material,” _Optica_ , vol. Part F127-, 2018.
  * [13] A. N. Tait, M. A. Nahmias, B. J. Shastri, P. R. Prucnal, and J. S. Harris, “The physics of optical neural networks,” _Applied Physics Reviews_ , vol. 4, no. 2, p. 021105, 2017.
  * [14] ——, “Optical implementation of deep networks,” _Applied Optics_ , vol. 55, no. 36, pp. A71–A82, 2016.
  * [15] M. Miscuglio, J. Dambre, and P. Bienstman, “All-optical nonlinear activation function for photonic neural networks [invited],” _Optical Materials Express_ , vol. 8, no. 12, pp. 3851–3863, 2018.
  * [16] L. Larger, M. C. Soriano, D. Brunner, L. Appeltant, J. M. Gutiérrez, I. Fischer, and C. R. Mirasso, “Photonic information processing beyond turing: an optoelectronic implementation of reservoir computing,” _Optics Express_ , vol. 20, no. 3, pp. 3241–3249, 2012.
  * [17] S. Jutamulia and F. T. S. Yu, “Overview of hybrid optical neural networks,” _Optics & Laser Technology_, vol. 28, no. 2, pp. 85–97, 1996.
  * [18] K. M. Boehm, P. Khosravi, R. Vanguri, J. Gao, and S. P. Shah, “Harnessing multimodal data integration to advance precision oncology,” _Nature Reviews Cancer_ , vol. 22, no. 2, pp. 71–88, 2022.
  * [19] M. Zhuge _et al._ , “Kaleido-bert: Vision-language pre-training on fashion domain,” in _Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition_ , 2021.
  * [20] Y. B. Ovchinnikov _et al._ , “Diffraction of a released bose-einstein condensate by a pulsed standing light wave,” _Physical Review Letters_ , vol. 83, no. 2, p. 284, 1999.
  * [21] S. Han, J. Pool, J. Tran, and W. Dally, “Learning both weights and connections for efficient neural network,” in _Advances in Neural Information Processing Systems_ , 2015, pp. 1135–1143.
  * [22] P. Molchanov, S. Tyree, T. Karras, T. Aila, and J. Kautz, “Pruning convolutional neural networks for resource efficient inference,” in _International Conference on Learning Representations_ , 2016.
  * [23] I. Hubara, M. Courbariaux, D. Soudry, R. El-Yaniv, and Y. Bengio, “Quantized neural networks: Training neural networks with low precision weights and activations,” in _arXiv preprint arXiv:1609.07061_ , 2017.
  * [24] E. L. Denton, W. Zaremba, J. Bruna, Y. LeCun, and R. Fergus, “Exploiting linear structure within convolutional networks for efficient evaluation,” in _Advances in Neural Information Processing Systems_ , 2014, pp. 1269–1277.
  * [25] G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” in _arXiv preprint arXiv:1503.02531_ , 2015.
  * [26] W. Chen, J. Wilson, S. Tyree, K. Weinberger, and Y. Chen, “Compressing neural networks with the hashing trick,” in _International Conference on Machine Learning_ , 2015, pp. 2285–2294.
  * [27] H. Park, L. L. Sjösund, Y. Yoo, J. Bang, and N. Kwak, “Extremec3net: Extreme lightweight portrait segmentation networks using advanced c3-modules,” _arXiv preprint arXiv:1908.03093_ , 2019.
  * [28] X. Shen, A. Hertzmann, J. Jia, S. Paris, B. Price, E. Shechtman, and I. Sachs, “Automatic portrait segmentation for image stylization,” in _Computer Graphics Forum_ , vol. 35, no. 2. Wiley Online Library, 2016, pp. 93–102.
  * [29] J. Krause, J. Deng, M. Stark, and L. Fei-Fei, “Collecting a large-scale dataset of fine-grained cars,” 2013.
  * [30] A. Geiger, P. Lenz, and R. Urtasun, “Are we ready for autonomous driving? the kitti vision benchmark suite,” in _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2012.
  * [31] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo _et al._ , “Segment anything,” _arXiv preprint arXiv:2304.02643_ , 2023.
  * [32] Q. Liu, H. Zheng, B. T. Swartz, Z. Asad, I. Kravchenko, J. G. Valentine, Y. Huo _et al._ , “Digital modeling on large kernel metamaterial neural network,” _arXiv preprint arXiv:2307.11862_ , 2023.


# TULIP: Transformer for Upsampling of LiDAR Point Clouds

Bin Yang1, Patrick Pfreundschuh1, Roland Siegwart1, Marco Hutter2, Peyman
Moghadam3,4†, Vaishakh Patil2†
1Autonomous Systems Lab, ETH Zurich, Switzerland, 2Robotic Systems Lab, ETH
Zurich, Switzerland
3School of EER, Queensland University of Technology (QUT), Australia, 4Data61,
CSIRO, Brisbane, Australia
{biyang, patripfr, rolandsi, mahutter<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

LiDAR Upsampling is a challenging task for the perception systems of robots
and autonomous vehicles, due to the sparse and irregular structure of large-
scale scene contexts. Recent works propose to solve this problem by converting
LiDAR data from 3D Euclidean space into an image super-resolution problem in
2D image space. Although their methods can generate high-resolution range
images with fine-grained details, the resulting 3D point clouds often blur out
details and predict invalid points. In this paper, we propose TULIP, a new
method to reconstruct high-resolution LiDAR point clouds from low-resolution
LiDAR input. We also follow a range image-based approach but specifically
modify the patch and window geometries of a Swin-Transformer-based network to
better fit the characteristics of range images. We conducted several
experiments on three different public real-world and simulated datasets. TULIP
outperforms state-of-the-art methods in all relevant metrics and generates
robust and more realistic point clouds than prior works. We plan to publish
the code upon acceptance.

$\dagger$$\dagger$footnotetext: Authors share last authorship.

##  1 Introduction

(a) Input

(b) Ground Truth

(c) SwinIR [26]

(d) TULIP (Ours)

Figure 1: TULIP generates more realistic LiDAR point clouds from low-
resolution LiDAR input (a) compared to the existing image super-resolution
approach [26].

Light Detection And Ranging (LiDAR) is one of the most common sensors for
perception in various fields of autonomy, such as autonomous driving, and
unmanned aerial vehicles (UAV). LiDARs are used to generate 3D point clouds of
the scene. These point clouds are essential for mapping, localization, and
object detection tasks. However, the accuracy of these tasks often depends on
the resolution of the point cloud [48]. Furthermore, the resolution of a LiDAR
is inherently associated with increased energy consumption and cost, making
its use impractical for various applications. LiDARs also have different
vertical and horizontal resolutions. The vertical resolution of rotating 3D
LiDARs is typically much lower than the horizontal resolution. These
limitations make an upsampling technique necessary to increase the resolution
of the LiDAR data, especially in the vertical direction. Moreover, upsampling
LiDAR data has the potential to counter domain shift problems in LiDAR-based
learning methods. Notably, the use of lower resolution LiDAR data in a system
that has been trained on higher resolution data leads to a significant drop in
performance [3]. Therefore, such upsampling techniques can not only help to
mitigate the domain gap by creating a virtual high-resolution sensor that
matches the target domain [37, 56] but also reduce the high cost of collecting
new LiDAR data, annotating and retraining the methods.

Deep learning has led to remarkable advancements in LiDAR upsampling
techniques in recent years. Several recent methods [51, 53, 37] have focused
on learning the upsampling process within the 3D Euclidean space. However, the
processing of 3D data in deep learning can be quite resource-intensive. One
approach to this is the representation of the point cloud as a range image
[38, 24, 17, 44]. This approach facilitates the adoption of well-established
image-based super-resolution methods. Image-based super-resolution methods
mostly employ autoencoder-style networks with convolutional and
deconvolutional layers [38, 57, 13]. However, these methods are not directly
transferable to range images. Range images generated by back projection of
sparse 3D point clouds have distinct sharp edges at the object boundaries.
Convolutional neural network based methods often induce edge smoothing due to
the regularization effect, making them unsuitable for LiDAR range image
processing.

As an alternative, Transformer [43] has shown great success in RGB image
super-resolution and producing less blurry outputs [26]. Vision transformers,
however, face challenges such as the need for large amounts of training data
[9, 26, 31] and the global computation of self-attention, limiting the capture
of local information in the range images. To address this, Swin Transformer
[29] introduced tokenizing the input image and applying the self-attention
mechanism locally in independent windows. While the modifications to Swin
Transformer have led to robust training and promising results compared to
Vision Transformer, its default structure and settings are not ideally suited
for processing LiDAR range images. These range images consist of single-
channel data, representing 3D spatial information, as opposed to three-channel
color images that depict visual appearance. Furthermore, range images feature
large, smooth areas within 3D objects, separated by distinct sharp edges
between objects. As a result, minor inaccuracies in the 2D prediction (_e.g_.
blurry output) can lead to significant differences in the projected 3D
occupancy. This non-uniform distribution of image details further complicates
network training. The challenging details in the range image are concentrated
in a few specific pixels, contrasting with RGB images where relevant details
are evenly distributed. This makes range images fundamentally different from
RGB images. Inspired by the limitations of state-of-the-art range image
upsampling methods and the potential of Swin-Transformer [29], we propose a
novel network named TULIP. Our proposed geometry-aware architecture is
tailored to accommodate LiDAR range image data. The proposed method takes a
low-resolution range image as input and produces a high-resolution range
image, which can then be projected into a 3D point cloud. To accommodate LiDAR
range image geometry, we utilize one-line row patches to tokenize the input
range image, as opposed to the square patches used for RGB images. This
approach helps to preserve vertical information for the upsampling process
while capturing boundary discontinuities effectively. This also creates a
trade-off between local detail capture and model complexity in the training
process. Additionally, we incorporate a non-square window for local self-
attention computation. This enhances learning of spatial contexts at different
scales in the range image. Further, we extensively train and evaluate TULIP on
two real-world and one simulated autonomous driving datasets: KITTI [19],
DurLAR [25] and CARLA [24], respectively. Comparisons to existing approaches
demonstrate that TULIP outperforms the state-of-the-art methods on all three
datasets.

##  2 Related Works

Image super-resolution has seen significant progress in recent years. The
technique aims to construct high-resolution (HR) images from low-resolution
(LR) observations, often leveraging advances in Convolutional Neural Networks
(CNNs) to enhance the fidelity and detail of images for better visualization
and information extraction. Subsequently, several enhanced frameworks have
been developed [13, 14, 54, 23, 22].

Driven by their success in the field of natural language processing (NLP) [21,
4], Transformers [43] has been extended to solve a variety of vision-related
tasks such as object detection [6] and semantic segmentation [2, 7]. Such
Vision Transformers (ViT) [16] excels at learning to focus on relevant image
regions by exploring global interactions among different regions. Their
impressive performance has led to their adoption in image restoration tasks
[26, 9, 11] as well. While they brought significant improvements in RGB image
super-resolution, Vision Transformer [16] comes with the drawback of the
quadratic computational complexity of self-attention. Additionally, they
capture mostly global dependencies within the data and require large amounts
of data for training.

To address these challenges, Swin Transformer [29] has been proposed. Unlike
the Vision Transformer, which relies on global self-attention across the
entire image, the Swin Transformer employs a local window-based attention
mechanism and establishes a pyramid-like architecture. This approach processes
images hierarchically by gradually merging smaller image patches into larger
ones. This strategy enables more effective handling of different scales and
facilitates the processing of multi-scale features, making it particularly
suitable for range image processing. Furthermore, due to Swin Transformer’s
hierarchical structure, the increase in the number of model parameters scales
linearly with the input image size, which mitigates the challenges associated
with handling large range images. Additionally, they require less training
data than the classical Vision Transformer, which is beneficial for LiDAR
upsampling as LiDAR datasets are typically magnitudes smaller than RGB
datasets used for Vision Transformer. The Swin and Vision Transformer have
also been used for image super-resolution on omnidirectional panoramic camera
images [40, 50]. However, these approaches are specifically designed for the
unique characteristics of these cameras (_e.g_. strong distortions, spherical
continuity) that are vastly different from the LiDAR sensing model, which
renders them unsuitable for application to range images.

Other related works have focussed on upsampling high-density point clouds from
low-density point clouds [56, 51, 49, 55, 34]. While those approaches also
provide qualitative results for LiDAR point clouds, they focus on increasing
the general 3D point density of point clouds of single objects instead of a
large scene. Another area closely associated with LiDAR upsampling is depth
completion [12, 20, 47, 41, 35, 33, 32, 27]. The primary objective of depth
completion is to improve sparse depth estimates acquired from LiDAR by
integrating data from multiple sensors, predominantly RGB cameras. The result
of depth completion methods is a dense depth map that provides depth values to
every pixel in the input depth map. Nonetheless, this field slightly diverges
from LiDAR upsampling due to its reliance on multimodal sensors and its common
deployment within a restricted field of view, making it not directly
comparable within the scope of this paper. LiDAR point clouds exhibit specific
characteristics, such as a distinct point pattern (due to the stacked lasers
in rotating 3D LiDARs) and a decrease in point density with increasing
distance from the sensor. Differently from arbitrary point cloud upsampling,
LiDAR upsampling tries to mimic a realistic point cloud of a high-resolution
LiDAR given the point cloud of low-resolution LiDAR and thus targets a
different result than the approaches above.

Due to the difficulty and high computational cost of upsampling LiDAR point
clouds in 3D space [37, 1, 8], most works [24, 38, 42] represent the point
cloud as a range image and perform the LiDAR upsampling task in 2D image
space. LiDAR-CNN [42] introduces a CNN-based architecture with semantic and
perceptual guidance. The method specifies two further loss functions besides a
per-point reconstruction loss to acquire a better synthesis of the high-
resolution range image. However, this approach is limited to data with
semantic annotations, which drastically reduces its applicability. LiDAR-SR
[38] deploys a CNN-based network with U-Net [36] architecture. It additionally
uses a Monte-Carlo dropout post-processing to reduce the amount of noisy
points in the prediction. These approaches based on convolutional operations
tend to fail to reconstruct the sharpness in the range image effectively.
While the previous approaches directly predict the range image, Implicit LiDAR
Network (ILN) [24] uses an implicit neural architecture that learns
interpolation weight to fill in new pixels instead of their values directly.
Although the method outperforms CNN-based approaches in training speed and
preserving the geometrical details in the input, it can still suffer from
limited neighboring information in the LiDAR data, especially in areas distant
from the sensor due to the extremely sparse input data. Differently from the
previous works, our approach builds on Swin-Transformer as a backbone and is
specifically customized to effectively process range images.

##  3 Methodology

LiDAR upsampling is achieved by upsampling a range image, effectively
transforming it from a 3D upsampling problem into a 2D image super-resolution
problem [24, 42, 38, 17]. It is, therefore, an evident approach to build on
RGB image super-resolution. However, image super-resolution aims to solve a
different problem than LiDAR upsampling. Image super-resolution enhances the
visual appearance of a low-resolution RGB image by recreating a high-
resolution image. While quantitative metrics for evaluating these works exist,
there is no one specific correct solution. Differently, LiDAR upsampling tries
to recreate a high-resolution, 3D LiDAR sensor output from a low-resolution
LiDAR input point cloud. Range images contain more geometrical and spatial
contexts rather than visual information, and the scene’s geometry strictly
dictates the correct solution. The underlying sensing model is also
drastically different, as LiDARs are active sensors. The respective range
pixel values purely depend on distance, while pixels in an RGB camera depend
on many factors, such as scene appearance, lighting conditions, exposure time,
and white balance. Furthermore, range images have a highly asymmetric field-
of-view (FoV), between 1:8 to 1:64, which differs drastically from regular
camera images. Accordingly, most works [42, 17, 38] only perform vertical
upsampling, in contrast to image super-resolution, which operates in both
dimensions. A naive application of state-of-the-art image super-resolution
networks [26] to range images does not result in adequate performance (Fig.
1). To this end, we developed TULIP, a novel range image-based LiDAR
Upsampling method. Based on the observations above, we specifically adapt a
network based on SwinUnet [5] to incorporate range images better. In the
following sections, we will describe the technical details of our method, with
a special focus on how our approach accommodates range images.

###  3.1 Problem Definition

Figure 2: Using a spherical projection, the point cloud (top) can be
represented as the range image (bottom) and vice versa.

A LiDAR point cloud consists of points captured during one revolution
$\mathcal{P}=\\{\mathbf{p}_{1},,...,\mathbf{p}_{n}\\}$. Each measurement
represent at 3D point $\mathbf{p}_{i}=\\{x_{i},y_{i},z_{i}\\}$. As input, we
assume a low-resolution point cloud $\mathcal{P}_{l}$ with $n_{l}=H_{l}\times
W_{l}$ points, where $H_{l}$ and $W_{l}$ correspond to vertical and horizontal
resolution. We aim to predict a high-resolution point cloud $\mathcal{P}_{h}$,
that has the same FoV as $\mathcal{P}_{l}$ but contains $n_{h}=\beta*n_{l}$
points. In our experiments, we set $\beta=4$. We only increase the point
cloud’s vertical resolution (amount of LiDAR active beams), _i.e_.,
$n_{h}=H_{h}\times W_{h}$, where $H_{h}=\beta*H_{l}$ and $W_{h}=W_{l}$. We
project the point cloud into a 2D range image. In this image, each row and
column coordinate $v,u$ correspond to the respective elevation and azimuth
angles of the LiDAR points, while the pixel value contains the range of the
point $r=\sqrt{x^{2}+y^{2}+z^{2}}$. The image coordinate of a 3D point can be
calculated using a spherical projection model:

$\begin{bmatrix}u\\\
v\end{bmatrix}=\begin{bmatrix}\frac{W}{2}-\frac{W}{2\pi}arctan(\frac{y}{x})\\\
\frac{H}{\Theta_{max}-\Theta_{min}}*(\Theta_{max}-arctan(\frac{z}{\sqrt{x^{2}+y^{2}}})\end{bmatrix}$
(1)

$\Theta_{max}$ and $\Theta_{min}$ correspond to the limits of the vertical
field of view of the LiDAR. We formulate the LiDAR upsampling problem as an
image upsampling problem, _i.e_. we aim to predict a high-resolution range
image $I_{h}\in\mathbf{R}^{1\times H_{h}\times W}$ given a low-resolution
range image $I_{l}\in\mathbf{R}^{1\times H_{l}\times W}$. We can then
calculate the high-resolution point cloud $\mathcal{P}_{h}$ by inverting the
projection in Eq. 1.

###  3.2 Architecture Design and Overview

Our network builds upon Swin-Unet [5] which is originally designed for image
segmentation. As shown in Fig. 3, we deploy a U-shaped network structure
featuring skip connections that link the encoder and decoder modules. The
fundamental building block of TULIP is the Swin Transformer [29]. TULIP first
uses a convolutional layer to patchify and map the input to a high-dimensional
feature space. The tokenized low-level feature maps are passed through a
multi-stage encoder. Each stage contains a pair of Swin Transformer [29]
blocks and a patch merging layer to downsample the resolution by a factor of
two and increase the dimension by a factor of four. This step realizes the
hierarchic computation and extraction of multi-scale features through
encoding. Within each stage, multiple instances of the self-attention
mechanism are performed locally in parallel, which is so-called Window Multi-
Head Self Attention (W-MSA). After passing through a two-layer MLP and
residual connection with the input, the feature vector is further passed to
the second part of the Swin Transformer block, utilizing a Shifted Window
Multi-Head Self Attention (SW-MSA) which extends W-MSA by employing a shifted
window operation. This enhances the model capacity as it compensates for the
lack of interaction between the local windows in W-MSA. The bottleneck layer,
which consists of two subsequent Swin Transformer [29] blocks, is applied to
enhance the high-level feature representation further. The decoder, which
realizes the upsampling has a symmetric design with respect to the encoder and
operates in a reverse way of the encoder. The resolution of the feature maps
is first expanded, and the dimension is reduced by a factor of two
accordingly. We feed existing geometrical information via skip connections.
The feature maps are transformed into a single-channel, high-resolution range
image in the last layer. The projection head comprises a $1\times 1$
convolutional layer for feature expansion, followed by a Leaky ReLU
activation, a pixel shuffle layer [39], and another $1\times 1$ convolutional
layer for final projection. As the loss function, we select the pixel-wise L1
loss.

Figure 3: Network architecture of TULIP: We adapt the network presented in [5]
for range image upsampling. The low-resolution input is encoded using Swin
Transformer blocks and patch merging layers (left). Patch unmerging layers are
used to rearrange the feature patches for upsampling (left). The projection
head generates the high-resolution range image from the latent features.

####  3.2.1 Patch Partition

The patch partition layer creates an initial feature embedding from the input
range image. This initial feature representation significantly influences the
network’s performance. Specifically, the selection of this layer determines
how the inherent characteristics, patterns, and relevant information of the
LiDAR range image data are encoded into a format that the network can
effectively learn from. Most transformer-based RGB image super-resolution
approaches utilize a relatively large patch size to tokenize the input [26, 9,
31]. These larger patches enable the construction of a global spatial context,
facilitating an understanding of the overall structure and high-level features
of the RGB image. These networks employ square-shaped patches and implement
upsampling in both vertical and horizontal directions. In contrast, LiDAR
upsampling is primarily aiming to enhance the vertical resolution of the
input, given that range images possess properties that are vastly different
from RGB images. Furthermore, attention across more pixels in range image is
less useful than in RGB image, since there is almost no geometrical relation
for the spatial contexts that are far from each other in the 2D range image
space. Motivated by this, we propose two adjustments to effectively process
range images within a Swin-Unet:

Row-Based Patch Tokenization: Our model builds row patches with a dimension of
$1\times 4$ for range image tokenization. The new patch geometry is designed
to retain full vertical information while compressing horizontally. This
aligns with our objective of conserving and extending vertical details in the
range image. Besides that, row patches excel at capturing boundary
discontinuities between distinct objects and the background within the scene.
Moreover, the selection of the patch size strikes a balance between enhancing
model performance and limiting model capacity. Smaller patches can capture
more detailed local information, but they also increase the number of model
parameters, which can slow down the model’s training process. Circular Padding
(CD) in Horizontal Dimension: Instead of adding zero pixels padding, our model
uses circular padding in the horizontal dimension. This avoids introducing
artificial features and naturally matches the sensor model of a rotating 3D
LiDAR. This preserves accurate neighbor relations along the edges, which would
otherwise be compromised during the projection to the 2D range image. The
Circular Padding (CD) also allows us to fit flexible input sizes without
changing the patch geometry.

####  3.2.2 Non-Square window for local self-attention

Inside Swin Transformer, input feature maps are window-partitioned along width
and height to compute the self-attention locally in each window. In addition
to square-shaped RGB images, some recent work applies a square window for
panoramic data as well [28, 52]. However, using square windows one range
images is unfavorable. On one hand, small windows can focus more on attention
within near areas but lead to increasing computational complexity and can
struggle with the reconstruction of objects at a further distance in the
scene. On the other hand, a larger square window can blend scene-related
contexts at different scales, which can potentially degrade the network
performance. In a prior study done by Eskandar _et al_. [17], they
demonstrated the significant enhancement in range image super-resolution by
separately upsampling the upper and lower portions of the data. In contrast to
their approach of utilizing CNN-based shallower and deeper network branches
for distinct feature extraction, our proposal involves the direct partitioning
of the input using a rectangular window. This window geometry facilitates
robust attention interaction among distant points through W-MSA and subtle
cross-attention between objects at multiple scales via SW-MSA and hierarchical
processing (refer to Fig. 4). In addition, range images, as typical panorama
data, inherently contain abundant information along the horizontal direction.
Therefore, directing attention more toward the image width rather than the
height can aid the network in capturing a greater amount of geometrical
information. At the same time, the computational complexity remains the same
as using a large square window.

Figure 4: Our network uses a hierarchical structure for patch merging on the
range image. We propose to use rectangle window to fit the range data and to
handle depth discontinuity at different scales.

####  3.2.3 Additional Modifications and Postprocessing

Patch Unmerging (PU) and Pixel Shuffle (PS): Similar to the patch merging
layer that is responsible for downsampling at the encoder stage, a mechanism
is necessary within the decoder to upsample the feature maps. Uformer [45]
achieves upsampling through a de-convolutional layer. As they tend to smooth
out image sharpness [26], we design a reverse operation of patch merging. We
re-arrange the channels of patches into their respective positions in a grid
to upsample the resolution of input feature maps, which we denote as patch
unmerging. Furthermore, we built the projection head based on the pixel
shuffle operation [39]. Monte Carlo Dropout: We additionally use Monte Carlo
Dropout (MC-Dropout) [38] to filter unreliable points. MC-Dropout executes
several feed-forward passes during inference with different active dropouts,
which yields a distribution of outputs. We refine the results by removing
points with high output variance, as they often come down to noisy, invalid
points.

##  4 Experiments

###  4.1 Experimental Settings

Datasets: We conduct experiments on three different datasets, that include the
two large-scale real-world datasets DurLAR [25] and KITTI [19] (Sec. 4.3.2),
as well as on a dataset [24] that was generated using the CARLA simulator [15]
(Sec. 4.3.1). For all datasets, we select test sequences that are recorded in
different locations than the train set to avoid spatial overlap. We vertically
downsample the high-resolution range images with a factor of four by skipping
the respective lines to generate low-resolution input images. For ablation
studies in Sec. 4.2, we select the KITTI dataset. Implementation Details: In
all experiments, we use distributed processing with $4\times$ GeForce RTX 2080
Ti. For training, we use a fixed batch size of eight per GPU for all datasets.
We use the AdamW [30] optimizer with a weight decay factor of 0.05 and a base
learning rate of 5e-4. Evaluation metrics: We evaluate the Mean Absolute Error
(MAE) for all the pixels in the generated 2-D range images. Additionally, we
assess the performance by considering the 3D points reconstructed by our
neural networks. The Chamfer Distance (CD) measures the Euclidean distance
between two point clouds. We also follow the approach in [24] to evaluate the
volumetric occupancy similarity. To do so, the point clouds are voxelized
using a voxel size of $0.1m$. A voxel is classified as occupied for each point
cloud if it contains at least one point. We then calculate the Intersection-
over-Unit (IoU) based on the occupancy.

###  4.2 Ablation Studies

Model | Patch | Window | CP/PU/PS | MC |  MAE $\downarrow$ |  IoU $\uparrow$ |  CD $\downarrow$
---|---|---|---|---|---|---|---
Baseline [5] | $4\times 4$ | Square | $\times$ | $\times$ | 0.7138 | 0.3250 | 0.1940
Model 1 | $2\times 2$ | Square | $\times$ | $\times$ | 0.6251 | 0.3337 | 0.1661
Model 2 | $1\times 4$ | Square | $\times$ | $\times$ | 0.4814 | 0.3667 | 0.1391
Model 3 | $1\times 4$ | Rectangular | $\times$ | $\times$ | 0.4228 | 0.4040 | 0.1218
Model 4 | $1\times 4$ | Rectangular | $\checkmark$ | $\times$ | 0.4186 | 0.4094 | 0.1199
TULIP | $1\times 4$ | Rectangular | $\checkmark$ | $\checkmark$ | 0.4183 | 0.4134 | 0.1198
TULIP-L | $1\times 4$ | Rectangular | $\checkmark$ | $\checkmark$ | 0.3678 | 0.4329 | 0.0971
Table 1: Ablation study results for $4\times$ upsampling on KITTI. Input
Resolution: $16\times 1024$, Output Resolution: $64\times 1024$.

Patch size: In Tab. 1 Baseline and Model 1-2, we compare the upsampling
results with different patch sizes. We observe that the smaller patch
($2\times 2$) outperforms the larger one ($4\times 4$) while the proposed row
patch ($1\times 4$) can further refine the network performance significantly.
In particular, the row patch improves $32.6\%$ in MAE, $12.8\%$ in IoU, and
$28.3\%$ in CD, compared to a $4\times 4$ patch. As visualized in Fig. 5, the
row patch better preserves sharpness around edges and corners compared to the
larger patch sizes.

(a) Scene

(b) GT

(c) $1\times 4$

(d) $2\times 2$

(e) $4\times 4$

Figure 5: Upsampling results with different patch sizes: We observe that
square patches ($4\times 4$, $2\times 2$) blur out edges and create invalid
connections between separate walls, while the proposed $1\times 4$ patch
generates sharp edges and correct discontinuities.

(a) Scene

(b) GT

(c) Rectangular

(d) Square

Figure 6: Upsampling results with different window geometries: Rectangular
windows improve reconstruction of objects at different ranges, indicated by
more distinct discontinuities (top) and a better reconstruction of the car
(bottom).

Window geometry: To verify the effectiveness of modifying window geometry for
range images, we conduct experiments with square and rectangular windows (Tab.
1 Model 2-3. Based on the quantitative results, we can observe the following:
reconstruction quality produced by local attention within large square windows
is inferior to the rectangular window with respect to both 2D and 3D
evaluation metrics, while they result in the same computational complexity.
The underlying reason for this performance difference lies in the perceptive
field of the windows. A wider window is more likely to include discontinuities
between objects at different scales. Accordingly, the network performs better
at separating objects from each other, which can visually be validated by
clearer surface boundaries in Fig. 6.

Additional Modifications and Postprocessing: We additionally apply Circular
Padding (CP), Patch Unmerging (PU), and Pixel Shuffle (PS) layer in Model 4.
Due to space restrictions and a smaller impact on the performance, we evaluate
them together. For TULIP, we also apply Monte Carlo dropout [18] as a post-
processing step, which helps to further reduce ghost points in between
objects, which is reflected by the enhancement in IoU. For TULIP-L, we
increase the model capacity, using four Swin-Transformer layers instead of
three in both encoder and decoder, which demonstrates further performance
boost on all metrics.

Inference Time: We evaluate the inference time on the KITTI dataset [19]. A
single forward pass for TULIP-L takes 1.33s. For the Monte-Carlo Dropout we
run the separate forward passes in a batch and calculate the filtered result,
which increases the effective inference time of TULIP-L to 1.61s. This is
slower than ILN [24] (1.14s) and LIDAR-SR [38] (1.16s) and comparable to Swin-
IR [26] (1.63s).

###  4.3 Benchmark Results

Besides bilinear interpolation, which computes the interpolation weights from
the four nearest neighbors for upsampling, we compete against the state-of-
the-art LiDAR upsampling approaches Implicit LiDAR Network (ILN) [24] and
LiDAR-SR [38]. We also compare against several image super-resolution works:
SRNO [46], and LIIF [10] learn implicit features for interpolation instead of
2D coordinates of new pixels while HAT [9] and Swin-IR [26] are pixel-based
approaches that directly reconstruct high-resolution images from low-
resolution feature embeddings. Similar to our work, they are also both based
on Swin Transformer [29]. We train all methods on the specific datasets using
their default parameters.

CARLA 4x Output Resolution: $128\times 2048$
---
Model | MAE $\downarrow$ | IoU $\uparrow$ | CD $\downarrow$
Bilinear | 1.8128 | 0.1382 | 0.7262
SRNO [46] | 2.4640 | 0.1343 | 2.0230
HAT [9] | 1.6032 | 0.2698 | 0.6337
SWIN-IR [26] | 1.9560 | 0.2718 | 0.4840
LIIF [10] | 0.8064 | 0.3502 | 0.174
LIDAR-SR [38] | 0.8216 | 0.2581 | 0.2044
ILN [24] | 0.8592 | 0.5206 | 0.1855
TULIP (Ours) | 0.7768 | 0.5291 | 0.1168
TULIP-L (Ours) | 0.7748 | 0.5332 | 0.1032
Table 2: Quantitative comparison against state-of-the-art LiDAR and image
super-resolution methods on simulation dataset. All methods are trained and
evaluated on the same splits. KITTI 4x Output Resolution: $64\times 1024$
---
Model | MAE $\downarrow$ | IoU $\uparrow$ | CD $\downarrow$
Bilinear | 2.0892 | 0.1063 | 0.6000
SRNO [46] | 0.8350 | 0.2035 | 0.4417
HAT [9] | 0.6856 | 0.2035 | 0.2516
SWIN-IR [26] | 1.2972 | 0.2774 | 0.7347
LIIF [10] | 0.6143 | 0.3226 | 0.1916
LIDAR-SR [38] | 0.5674 | 0.1005 | 0.2165
ILN [24] | 1.0528 | 0.3342 | 0.2787
TULIP (Ours) | 0.4183 | 0.4134 | 0.1198
TULIP-L (Ours) | 0.3678 | 0.4329 | 0.0971
DurLAR 4x Output Resolution: $128\times 2048$
Bilinear | 2.4384 | 0.1266 | 0.6346
SRNO [46] | 1.5396 | 0.1507 | 0.5108
HAT [9] | 1.7820 | 0.2353 | 0.1973
SWIN-IR [26] | 1.9416 | 0.2157 | 0.2279
LIIF [10] | 1.5672 | 0.2469 | 0.1548
LIDAR-SR [38] | 1.5312 | 0.1370 | 0.1128
ILN [24] | 1.5720 | 0.3430 | 0.0893
TULIP (Ours) | 1.5382 | 0.3552 | 0.06560
TULIP-L (Ours) | 1.5386 | 0.3620 | 0.06371
Table 3: Quantitative comparison against state-of-the-art LiDAR and image super-resolution methods on real-world dataset. All methods are trained and evaluated on the same splits. Input |  |  |
---|---|---|---
Swin-IR [26] |  |  |
LIIF [10] |  |  |
LiDA-SR [38] |  |  |
ILN [24] |  |  |
TULIP (Ours) |  |  |
GT |  |  |
|  A |  B |  C
Figure 7: Qualitative results on KITTI: We provide visualizations for the
top-4 results from Tab. 3 and Swin-IR [26]. Our approach outperforms other
state-of-the-art methods in upsampling realistic point clouds. A: TULIP is the
only approach that does not generate invalid ghost points in between the car
(right bottom) and the wall behind. It can also be observed that all
approaches except for ILN match the characteristic line pattern of the LiDAR
on the ground (middle). B: Swin-IR and LIIF generate a blurry reconstruction.
TULIP achieves the clearest point cloud and even reconstructs small details
such as the car’s side mirrors. C: Swin-IR, LIIF, and LiDAR-SR do not clearly
separate the car from the wall behind. ILN generates a clear point cloud but
does not properly reconstruct the shape of the cars. TULIP achieves the
clearest point cloud, with distinct discontinuities between objects, and
closely resembles the shape of the three cars.

####  4.3.1 Simulation Results

Similar to prior works, we first evaluate the performance on noise-free,
simulated data. We train and test all approaches on the dataset presented in
[24] that was recorded using the CARLA simulator and select the ground-truth
range images of size $128\times 2048$ that capture a vertical FoV of 30∘. We
use a (20699/2618) train/test split. We observe that the performance between
image super-resolution approaches varies greatly, and LIIF performs the best
among them. While LIIF [10] achieves a comparable MAE to the LiDAR upsampling
methods, the 3D metrics are significantly worse, indicating that it is
unsuitable for the underlying geometric input data. Nevertheless, LIIF [10]
still outperforms the LiDAR upsampling method in LIDAR-SR. The interpolation-
based approach in ILN shows decent results for the noise-free data, as
indicated by their high IoU. Nevertheless, our approaches achieve the best
performance in all evaluated metrics, with the strongest improvement in
Chamfer Distance, which validates that our design is well suited for the
geometric reconstruction in range images.

####  4.3.2 Real-World Results

To evaluate our approach on real-world data, we downsample the DurLAR and
KITTI datasets to a comparable number of frames to the CARLA dataset by
temporally skipping frames in the dataset sequences. In particular, we have
the following (train/test) split: DurLAR (24372/2508), and KITTI (20000/2500).

KITTI: The KITTI dataset [19] was collected using a Velodyne HDL-64E LiDAR
with a vertical FoV of 26.8∘ and a resolution of $64\times 1024$. For the
image super-resolution approaches, we observe comparable results and trends on
KITTI as on the simulated data in Sec.4.3.1. Even though LIDAR-SR [38] is
designed for LIDAR upsampling and outperforms image super-resolution
approaches in terms of MAE, it achieves the lowest IoU among all approaches,
which can be explained by a large amount of invalid floating points between
real objects. LIIF [10] achieves better CD and comparable IoU compared to ILN.
We observe that the IoU of ILN [24] is drastically lower than on CARLA. The
interpolation scheme in ILN is thus not suitable to properly handle the lower
resolution data with noise from the real-world sensor. Our approaches
significantly outperform all competing methods, especially in terms of IoU as
it can better handle discontinuities between objects. As indicated in the
visualizations in Fig.7, our approach predicts fewer invalid float points
between objects, as well as fewer blurry edges, which results in much clearer
point clouds.

DurLAR: The DurLAR dataset [25] was recorded using an Ouster OS1-128 LiDAR,
that captures point clouds with a resolution of $128\times 2048$ with a 45∘
vertical FoV. Compared to CARLA and KITTI, we see a large performance drop for
all approaches in MAE and IoU, which results from the larger maximum range and
vertical FoV of the sensor as well as stronger sensor noise. We observe that
all approaches designed for LiDAR upsampling clearly improve in Chamfer
Distance compared to image super-resolution approaches. ILN and TULIP
outperform all methods by a large margin in IoU. TULIP-L achieves the best 3D
metrics, which validates that it can also handle the challenging DurLAR data
better than prior works.

Range Analysis: As the density of LiDAR point clouds decreases with increasing
distance from the sensor, we additionally compare the performance at different
ranges for a more fine-grained evaluation. We observe that our approach
achieves better or similar performance compared to all approaches in all
metrics at all ranges. Up to 30m, TULIP clearly outperforms the baseline
approaches. TULIP handles discontinuities between objects better than other
approaches, which can be verified visually through sharper edges and fewer
”floating” points between objects. This is especially reflected in the high
IoU. At ranges above 30m, our approach performs comparable to other methods.
Above this range, LiDAR data is extremely sparse, meaning that points
neighboring in the low-resolution range image are actually distant in the
Euclidean space. It is, therefore, not feasible to infer meaningful
information about the scene from such neighboring points at a high range,
which is also represented in the poor performance of all approaches.

(a) IoU $\uparrow$

(b) Chamfer distance $\downarrow$

Figure 8: 3D error metrics visualized at different ranges for KITTI.

###  4.4 Failure Cases

Although TULIP outperforms other state-of-the-art methods both qualitatively
and quantitatively, the upsampling quality can still be limited in certain
cases. For instance, as shown in Fig 9, our approach shows inferior upsampling
results for the specific scene. The irregularity of the scene leads to high
uncertainty, which makes the reconstruction noisy. Observing the details in
Fig. 9(a), the network fails to reconstruct the car at the top of the scene.

(a) Full scene

(b) Details

Figure 9: Failure Case: noisy point cloud generated by TULIP and missing
reconstruction of the object. (Ours and GT)

##  5 Conclusion

This work presents TULIP, a novel method for LiDAR Upsampling that achieves
incredible performance in upsampling the range image. Our approach transforms
3D point clouds into 2D range images and performs the upsampling in 2D space.
We build upon a Swin-Transformer-based network and specifically modify the
patch partition and attention windows to better accommodate the
characteristics of range images. Throughout various experiments, testing on
three different benchmarks, it shows that TULIP outperforms state-of-the-art
methods quantitatively up to 35.3% in MAE, 49.3% in Chamfer distance, and
29.5% in IoU. Quantitatively, TULIP demonstrates more promising results in
upsampling the realistic LiDAR data.

## References

  * Akhtar et al. [2022] Anique Akhtar, Zhu Li, Geert Van der Auwera, Li Li, and Jianle Chen.  Pu-dense: Sparse tensor-based point cloud geometry upsampling.  _IEEE Transactions on Image Processing_ , 31:4133–4148, 2022.
  * Amir et al. [2022] Shir Amir, Yossi Gandelsman, Shai Bagon, and Tali Dekel.  Deep vit features as dense visual descriptors.  _ECCVW What is Motion For?_ , 2022.
  * Bešić et al. [2022] Borna Bešić, Nikhil Gosala, Daniele Cattaneo, and Abhinav Valada.  Unsupervised domain adaptation for lidar panoptic segmentation.  _IEEE Robotics and Automation Letters_ , 7(2):3404–3411, 2022.
  * Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Cao et al. [2022] Hu Cao, Yueyue Wang, Joy Chen, Dongsheng Jiang, Xiaopeng Zhang, Qi Tian, and Manning Wang.  Swin-unet: Unet-like pure transformer for medical image segmentation.  In _European conference on computer vision_ , pages 205–218. Springer, 2022.
  * Carion et al. [2020] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In _European conference on computer vision_ , pages 213–229. Springer, 2020.
  * Caron et al. [2021] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 9650–9660, 2021.
  * Chen et al. [2022] Tso Yuan Chen, Ching Chun Hsiao, and Ching-Chun Huang.  Density-imbalance-eased lidar point cloud upsampling via feature consistency learning.  _IEEE Transactions on Intelligent Vehicles_ , 2022.
  * Chen et al. [2023a] Xiangyu Chen, Xintao Wang, Jiantao Zhou, Yu Qiao, and Chao Dong.  Activating more pixels in image super-resolution transformer.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 22367–22377, 2023a.
  * Chen et al. [2021] Yinbo Chen, Sifei Liu, and Xiaolong Wang.  Learning continuous image representation with local implicit image function.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 8628–8638, 2021.
  * Chen et al. [2023b] Zheng Chen, Yulun Zhang, Jinjin Gu, Linghe Kong, Xiaokang Yang, and Fisher Yu.  Dual aggregation transformer for image super-resolution.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 12312–12321, 2023b.
  * Cheng et al. [2018] Xinjing Cheng, Peng Wang, and Ruigang Yang.  Depth estimation via affinity learned with convolutional spatial propagation network.  In _Proceedings of the European conference on computer vision (ECCV)_ , pages 103–119, 2018.
  * Dong et al. [2015] Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang.  Image super-resolution using deep convolutional networks.  _IEEE transactions on pattern analysis and machine intelligence_ , 38(2):295–307, 2015.
  * Dong et al. [2016] Chao Dong, Chen Change Loy, and Xiaoou Tang.  Accelerating the super-resolution convolutional neural network.  In _Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14_ , pages 391–407. Springer, 2016.
  * Dosovitskiy et al. [2017] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun.  Carla: An open urban driving simulator.  In _Conference on robot learning_ , pages 1–16. PMLR, 2017.
  * Dosovitskiy et al. [2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _International Conference on Learning Representations_ , 2021.
  * Eskandar et al. [2022] George Eskandar, Sanjeev Sudarsan, Karim Guirguis, Janaranjani Palaniswamy, Bharath Somashekar, and Bin Yang.  Hals: A height-aware lidar super-resolution framework for autonomous driving.  _arXiv preprint arXiv:2202.03901_ , 2022.
  * Gal and Ghahramani [2016] Yarin Gal and Zoubin Ghahramani.  Dropout as a bayesian approximation: Representing model uncertainty in deep learning.  In _international conference on machine learning_ , pages 1050–1059. PMLR, 2016.
  * Geiger et al. [2012] Andreas Geiger, Philip Lenz, and Raquel Urtasun.  Are we ready for autonomous driving? the kitti vision benchmark suite.  In _2012 IEEE conference on computer vision and pattern recognition_ , pages 3354–3361. IEEE, 2012.
  * Hu et al. [2021] Mu Hu, Shuling Wang, Bin Li, Shiyu Ning, Li Fan, and Xiaojin Gong.  Penet: Towards precise and efficient image guided depth completion.  In _2021 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 13656–13662. IEEE, 2021.
  * Kenton and Toutanova [2019] Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of naacL-HLT_ , page 2, 2019.
  * Kim et al. [2016a] Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee.  Accurate image super-resolution using very deep convolutional networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1646–1654, 2016a.
  * Kim et al. [2016b] Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee.  Deeply-recursive convolutional network for image super-resolution.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1637–1645, 2016b.
  * Kwon et al. [2022] Youngsun Kwon, Minhyuk Sung, and Sung-Eui Yoon.  Implicit lidar network: Lidar super-resolution via interpolation weight prediction.  In _2022 International Conference on Robotics and Automation (ICRA)_ , pages 8424–8430. IEEE, 2022.
  * Li et al. [2021] L. Li, K.N. Ismail, H.P.H. Shum, and T.P. Breckon.  Durlar: A high-fidelity 128-channel lidar dataset with panoramic ambient and reflectivity imagery for multi-modal autonomous driving applications.  In _Proc. Int. Conf. on 3D Vision_. IEEE, 2021.
  * Liang et al. [2021] Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte.  Swinir: Image restoration using swin transformer.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 1833–1844, 2021.
  * Lin et al. [2022] Yuankai Lin, Tao Cheng, Qi Zhong, Wending Zhou, and Hua Yang.  Dynamic spatial propagation network for depth completion.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 1638–1646, 2022.
  * Ling et al. [2023] Zhixin Ling, Zhen Xing, Xiangdong Zhou, Manliang Cao, and Guichun Zhou.  Panoswin: A pano-style swin transformer for panorama understanding.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 17755–17764, 2023.
  * Liu et al. [2021] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 10012–10022, 2021.
  * Loshchilov and Hutter [2017] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In _International Conference on Learning Representations_ , 2017.
  * Lu et al. [2022] Zhisheng Lu, Juncheng Li, Hong Liu, Chaoyan Huang, Linlin Zhang, and Tieyong Zeng.  Transformer for single image super-resolution.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 457–466, 2022.
  * Park et al. [2020] Jinsun Park, Kyungdon Joo, Zhe Hu, Chi-Kuei Liu, and In So Kweon.  Non-local spatial propagation network for depth completion.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIII 16_ , pages 120–136. Springer, 2020.
  * Qiu et al. [2019] Jiaxiong Qiu, Zhaopeng Cui, Yinda Zhang, Xingdi Zhang, Shuaicheng Liu, Bing Zeng, and Marc Pollefeys.  Deeplidar: Deep surface normal guided depth prediction for outdoor scene from sparse lidar data and single color image.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3313–3322, 2019.
  * Qiu et al. [2022] Shi Qiu, Saeed Anwar, and Nick Barnes.  Pu-transformer: Point cloud upsampling transformer.  In _Proceedings of the Asian Conference on Computer Vision_ , pages 2475–2493, 2022.
  * Rho et al. [2022] Kyeongha Rho, Jinsung Ha, and Youngjung Kim.  Guideformer: Transformers for image guided depth completion.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6250–6259, 2022.
  * Ronneberger et al. [2015] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In _Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18_ , pages 234–241. Springer, 2015.
  * Savkin et al. [2022] Artem Savkin, Yida Wang, Sebastian Wirkert, Nassir Navab, and Federico Tombari.  Lidar upsampling with sliced wasserstein distance.  _IEEE Robotics and Automation Letters_ , 8(1):392–399, 2022.
  * Shan et al. [2020] Tixiao Shan, Jinkun Wang, Fanfei Chen, Paul Szenher, and Brendan Englot.  Simulation-based lidar super-resolution for ground vehicles.  _Robotics and Autonomous Systems_ , 134:103647, 2020.
  * Shi et al. [2016] Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang.  Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1874–1883, 2016.
  * Sun et al. [2023] Xiaopeng Sun, Weiqi Li, Zhenyu Zhang, Qiufang Ma, Xuhan Sheng, Ming Cheng, Haoyu Ma, Shijie Zhao, Jian Zhang, Junlin Li, et al.  Opdn: Omnidirectional position-aware deformable network for omnidirectional image super-resolution.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 1293–1301, 2023.
  * Tang et al. [2020] Jie Tang, Fei-Peng Tian, Wei Feng, Jian Li, and Ping Tan.  Learning guided convolutional network for depth completion.  _IEEE Transactions on Image Processing_ , 30:1116–1129, 2020.
  * Triess et al. [2019] Larissa T Triess, David Peter, Christoph B Rist, Markus Enzweiler, and J Marius Zöllner.  Cnn-based synthesis of realistic high-resolution lidar data.  In _2019 IEEE Intelligent Vehicles Symposium (IV)_ , pages 1512–1519. IEEE, 2019.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Wang et al. [2023] Suhong Wang, Hongqing Wang, Shufeng She, Yanping Zhang, Qingju Qiu, and Zhifeng Xiao.  Swin-t-nfc crfs: An encoder–decoder neural model for high-precision uav positioning via point cloud super resolution and image semantic segmentation.  _Computer Communications_ , 197:52–60, 2023.
  * Wang et al. [2022] Zhendong Wang, Xiaodong Cun, Jianmin Bao, Wengang Zhou, Jianzhuang Liu, and Houqiang Li.  Uformer: A general u-shaped transformer for image restoration.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 17683–17693, 2022.
  * Wei and Zhang [2023] Min Wei and Xuesong Zhang.  Super-resolution neural operator.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 18247–18256, 2023.
  * Yan et al. [2021] Zhiqiang Yan, Kun Wang, Xiang Li, Zhenyu Zhang, Baobei Xu, Jun Li, and Jian Yang.  Rignet: Repetitive image guided network for depth completion.  In _European Conference on Computer Vision_ , 2021.
  * Yang et al. [2019] Zetong Yang, Yanan Sun, Shu Liu, Xiaoyong Shen, and Jiaya Jia.  Std: Sparse-to-dense 3d object detector for point cloud.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 1951–1960, 2019.
  * Ye et al. [2021] Shuquan Ye, Dongdong Chen, Songfang Han, Ziyu Wan, and Jing Liao.  Meta-pu: An arbitrary-scale upsampling network for point cloud.  _IEEE transactions on visualization and computer graphics_ , 28(9):3206–3218, 2021.
  * Yu et al. [2023] Fanghua Yu, Xintao Wang, Mingdeng Cao, Gen Li, Ying Shan, and Chao Dong.  Osrt: Omnidirectional image super-resolution with distortion-aware transformer.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 13283–13292, 2023.
  * Yu et al. [2018] Lequan Yu, Xianzhi Li, Chi-Wing Fu, Daniel Cohen-Or, and Pheng-Ann Heng.  Pu-net: Point cloud upsampling network.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2790–2799, 2018.
  * Yun et al. [2022] Heeseung Yun, Sehun Lee, and Gunhee Kim.  Panoramic vision transformer for saliency detection in 360 videos.  In _ECCV_ , 2022.
  * Zhang et al. [2019] Wentai Zhang, Haoliang Jiang, Zhangsihao Yang, Soji Yamakawa, Kenji Shimada, and Levent Burak Kara.  Data-driven upsampling of point clouds.  _Computer-Aided Design_ , 112:1–13, 2019.
  * Zhang et al. [2020] Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu.  Residual dense network for image restoration.  _TPAMI_ , 2020.
  * Zhao et al. [2022] Wenbo Zhao, Xianming Liu, Zhiwei Zhong, Junjun Jiang, Wei Gao, Ge Li, and Xiangyang Ji.  Self-supervised arbitrary-scale point clouds upsampling via implicit neural representation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 1999–2007, 2022.
  * Zhao et al. [2021] Yifan Zhao, Le Hui, and Jin Xie.  Sspu-net: Self-supervised point cloud upsampling via differentiable rendering.  In _Proceedings of the 29th ACM International Conference on Multimedia_ , pages 2214–2223, 2021.
  * Zováthi et al. [2023] Örkény Zováthi, Balázs Pálffy, Zsolt Jankó, and Csaba Benedek.  St-depthnet: A spatio-temporal deep network for depth completion using a single non-repetitive circular scanning lidar.  _IEEE Robotics and Automation Letters_ , 2023.