# Z-Splat: Z-Axis Gaussian Splatting for Camera-Sonar Fusion

Ziyuan Qu1, Omkar Vengurlekar2, Mohamad Qadri3, Kevin Zhang4, Michael Kaess3,
Christopher Metzler4, Suren Jayasuriya2, and Adithya Pediredla1
1Dartmouth College,  2Arizona State University,  3Carnegie Mellon University,
4University of Maryland

###### Abstract

Differentiable 3D-Gaussian splatting (GS) is emerging as a prominent technique
in computer vision and graphics for reconstructing 3D scenes. GS represents a
scene as a set of 3D Gaussians with varying opacities and employs a
computationally efficient splatting operation along with analytical
derivatives to compute the 3D Gaussian parameters given scene images captured
from various viewpoints. Unfortunately, capturing surround view (360°
viewpoint) images is impossible or impractical in many real-world imaging
scenarios, including underwater imaging, rooms inside a building, and
autonomous navigation. In these restricted baseline imaging scenarios, the GS
algorithm suffers from a well-known ‘missing cone’ problem, which results in
poor reconstruction along the depth axis. In this manuscript, we demonstrate
that using transient data (from sonars) allows us to address the missing cone
problem by sampling high-frequency data along the depth axis. We extend the
Gaussian splatting algorithms for two commonly used sonars and propose fusion
algorithms that simultaneously utilize RGB camera data and sonar data. Through
simulations, emulations, and hardware experiments across various imaging
scenarios, we show that the proposed fusion algorithms lead to significantly
better novel view synthesis ($5$ dB improvement in PSNR) and 3D geometry
reconstruction (60% lower Chamfer distance).

###### Index Terms:

Gaussian Splatting, Acoustic-Optic Vision, Sensor fusion.

##  1 Introduction

Differentiable Gaussian spatting (GS) [1], a resurgence of elliptical weighted
average (EWA) splatting [2] from almost two decades back, has been emerging as
the dominant differentiable representation for scenes [3, 4]. GS represents
the scene as a volume density map similar to neural radiance fields (Nerf) [1,
5]. However, unlike Nerfs, they explicitly represent the scene as a sum of
densities of anisotropic 3D Gaussians. The explicit representation facilitates
easier geometric interpretation and manipulation of the scene, leading to an
explosion of GS-based techniques for scene relighting [6], shading [7], novel
view synthesis [8], text-based manipulation [9], and non-rigid manipulation
[10]. Furthermore, the GS rendering algorithm is fast, thanks to the splatting
operator and analytical derivatives which leads to extensions to dynamic
scenes [11, 12], 4D manipulation [13, 14, 15] and time-efficient editing [16,
9].

Unfortunately, when the training dataset has a limited baseline, optimizing
through the GS algorithm causes overfitting leading to poor novel-view
rendering and 3D reconstruction. In section 3.4, we show that limited baseline
results in the well-known ‘missing cone’ problem [17, 18] where the
frequencies along the depth-axis are not captured in the training data. To
mitigate the problem of missing depth information from the training images,
Chung et al. [19] proposed regularizing 3D Gaussian splatting with monocular
depth estimates. However, as this method hallucinates depth, the scene
reconstructions are no longer physically based and suffer from training bias.
In this manuscript, we use sonar images to sample the missing cone region.

Recent progress in sonar technologies has resulted in several low-cost sensors
that augment 2D spatial data measured by RGB cameras. Examples include
echosounder [20], forward-looking sonar [21], synthetic aperture sonar [22]
which find applications in SLAM [23], navigation [24], underwater imaging
[25], and imaging through scattering media [26]. These sonars offer
complementary information compared to the standard RGB cameras.

In this paper, we extend Gaussian splatting for sonar and build fusion
techniques that reconstruct geometry using the complementary information from
both the cameras and sonars. Our extension involves the development of
splatting operations along the $z$-axis tailored for these sensor types.
Through simulations, emulations, and hardware experiments, we show that
combining sonars and cameras results in significantly better (60%) 3D geometry
reconstruction. We also show that the accuracy of novel view synthesis
improves by 5 dB by regularizing camera data with sonars.

The specific contributions of this manuscript include

  1. 1.

A novel forward model to render the transient of Gaussian point clouds for two
types of sonars: Echosounder and Forward-Looking Sonar (FLS).

  2. 2.

Fusion algorithms for cameras and sonars.

  3. 3.

Validation on synthetic, emulated hardware, and real hardware datasets showing
that fusion GS splatting results in better geometric (60%) and photometric (5
dB) reconstruction than standard camera-only Gaussian splatting.

We will make the source code and datasets public after peer-review. We hope
that our work inspires the use of complementary sensor modalities to improve
3D scene representation in the future.

##  2 Related Work

Gaussian splatting: In computer graphics and rendering, splatting algorithms
were introduced over two decades ago [2] for texture filtering [27] and point
cloud rendering [28, 29], where the scene is represented as a sum of
anisotropic Gaussian kernel that can be efficiently rendered and without
aliasing artifacts.

The Gaussian splatting paper [1] uses this scene representation and fast
differentiable rendering pipeline to compute the scene parameters (density,
color, mean, and variance of the 3D Gaussians). Thanks to the decreased
computation on empty spaces, analytical derivatives, rich-quality
reconstructions, and explicit representations, Gaussian splatting has seen an
explosion of extensions, even though it was introduced less than a year ago
(see Fei et al. [30] for a review of the state-of-the-art). Related to this
manuscript, Matsuki et al. [31], Keetha et al. [32], Yan et al. [33], Sun et
al. [34] have independently proposed similar SLAM algorithms using RGB-D
cameras. The key idea is to splat depth similar to color values and minimize
the weighted sum of photometric and geometric loss functions. However, this
idea cannot be extended to sonar data as these techniques require a single
depth value per pixel, necessitating the dataset to be a depth map with high
spatial resolution on the x-y plane. Instead, sonar data typically resembles a
transient histogram of time-of-flight returns.

Camera and sonar fusion techniques: There has been previous research on fusing
complementary information from sonars and cameras in the literature [35, 36,
37]. For example, optical cameras suffer from hazing and scattering problems,
especially in turbid waters [35]. Sonars do not suffer from scattering, though
they have poor spatial resolution. To address this, Raaj et al. [36] combined
FLS and optical cameras for 3D object localization via particle filter in
scattering environments. Williams and Mahon [37] combined an echosounder (also
known as depth sounder) and a vision camera to develop a SLAM algorithm for
underwater robotic navigation on the Great Barrier Reef. This manuscript
focuses on the complementary geometric information that sonars and cameras
provide and analyzes how the missing cone present in camera-based 3D
reconstruction can be resolved via our Gaussian splatting algorithms for
transient histograms to reconstruct better geometry and photometry.

Most related to our work, Babaee and Negahdaripour [38] reconstructed 3D
objects from RGB and imaging sonars. However, their technique requires
matching occluding contours across RGB and imaging sonar, which restricts its
applicability to small baseline scenarios. Qadri et al. [21] recently propose
combining FLS sonar and cameras using implicit neural representations (INR).
Unlike ours, Qadri et al.’s technique cannot easily extend to echosounder and
reconstructs only scene geometry, not photometry.

Other fusion techniques: In addition to sonar, researchers have proposed
fusing radar measurements with radar for better object detection [39], imaging
of cluttered environment [40], and estimating pose and motion of vehicles for
autonomous navigation. Fusing Lidars or other optical time-of-flight cameras
is similar to our problem and has been targeted at imaging through scattering
media [41], densification of lidar point scans [42], and improved depth
estimation [43, 44]. The proposed Gaussian splatting algorithm could be
extended to fusion with Radars and Lidars.

##  3 Technical background

Figure 1: Sonar measurements provide complementary information. (a) Volumetric
scene captured with three pairs of cameras and sonars (echo sounder). We
assume the sensors are in the far field (i.e., the affine approximation to the
projective transform in Gaussian splatting research is valued). For the center
camera-sonar pair, camera measurements are obtained by projecting the
volumetric data along the vertical axis, and sonar measurements are obtained
by projecting the volumetric data along the horizontal axis. (b) If only
camera measurements are considered, then using the Fourier-slice theorem, we
are capturing only a few slices of the Fourier transform of the volume and
missing information on a large cone. (c) Sonar (time-resolved data) captures
orthogonal slices in the Fourier space, and hence, 3D reconstruction of the
scene is better conditioned if we do the camera-sensor fusion instead of using
only camera data.

In this section, we will briefly review the scene representation, rendering
equation, and splatting operation associated with the Gaussian splatting
algorithm. We will show that traditional GS algorithms do not appropriately
capture the $z$-related scene parameters, resulting in the missing cone
problem. This problem motivates our camera and sonar fusion solution to
recover this lost information.

###  3.1 Scene representation

In Gaussian splatting, the scene is represented as a volumetric density map.
This density ($\sigma\in\mathbb{R}^{+}$) at a point
($\bar{x}\in\mathbb{R}^{3}$) is given as the sum of densities contributed by
several Gaussians located as positions ($\bar{\mu}_{n}$ $\in\mathbb{R}^{3}$)
and $3\times 3$ anisotropic variance ($\varSigma_{n}$). Analytically,

$\displaystyle\sigma(\bar{x})=\sum_{n=1}^{N}\sigma_{n}\mathcal{N}(\bar{x};\bar{\mu}_{n},\varSigma_{n}),$
(1)

where $\sigma_{n}$ is a scaling factor representing the density of each
Gaussian. The color at each 3D point is similarly defined as

$\displaystyle\bar{c}(\bar{x})=\sum_{n=1}^{N}\bar{c}_{n}\mathcal{N}(\bar{x};\bar{\mu}_{n},\varSigma_{n}),$
(2)

where $\bar{c}_{n}$ is the color of each Gaussian kernel, which is encoded
using spherical harmonics [1].

The covariance of a 3D Gaussian is represented using the scaling matrix
$S_{n}$ and a rotation matrix $R_{n}$ to maintain the positive definiteness
as:

$\varSigma_{n}=R_{n}S_{n}S_{n}^{T}R_{n}^{T}.$ (3)

The rotation matrix $R_{n}$ is calculated from a normalized quaternion
$q_{n}$. For brevity, we will drop the index $n$ wherever the index is
implied.

###  3.2 Volume rendering equation

Gaussian splatting uses the same volume rendering equation as point-based
$\alpha$-blending [45] or NeRF-style [5] volumetric rendering. Therefore, if a
ray ($\bar{x}(l)=\bar{o}+l\bar{d}$) is cast along a pixel on the camera, the
color $C$ of the pixel will be

$\displaystyle C=\sum_{m=1}^{M}T_{m}\alpha_{m}c_{m},$ (4)

where the index $m$ represents the $m^{\text{th}}$ segment of the ray with
small length $\delta l$, and

$\displaystyle\alpha_{m}$ $\displaystyle=1-e^{-\sigma(\bar{x})\delta
l},\,\,\,T_{m}=\prod_{k=1}^{m-1}(1-\alpha_{k}),\text{and}$ $\displaystyle
c_{m}$ $\displaystyle=c(\bar{x}(m\delta_{l})).$ (5)

###  3.3 Splatting operation

Instead of expensive ray tracing similar to NeRF-style algorithms, the GS
algorithms [2, 1] use splatting, which is similar to rasterization. For a
camera viewing transform $W$, the 3D volume is transformed appropriately to
the camera view, but the GS algorithm approximates the projection operation
with an affine transformation so that any 3D Gaussian remains a 3D Gaussian
with covariance matrix:

$\displaystyle\varSigma^{\prime}=JW\varSigma W^{T}J^{T},\,\text{and
mean}\,\mu^{\prime}=JW\mu$ (6)

where $J$ is the Jacobian for the affine approximation and

$J=\begin{bmatrix}\frac{1}{\mu_{z}}&0&\frac{-\mu_{x}}{\mu_{z}^{2}}\\\
0&\frac{1}{\mu_{z}}&\frac{-\mu_{y}}{\mu_{z}^{2}}\\\
\frac{\mu_{x}}{l}&\frac{\mu_{y}}{l}&\frac{\mu_{z}}{l},\end{bmatrix}$ (7)

where $\mu=[\mu_{x},\mu_{y},\mu_{z}]$ and $l=\|\mu\|_{2}$.

The next step after the approximate projection operation is orthographic
projection. The 3D Gaussians are converted to 2D Gaussians with covariance
matrix $\varSigma_{\text{2D}}^{\prime}$ obtained by dropping the last row and
column, mathematically,

$\displaystyle\varSigma_{\text{2D}}^{\prime}=\begin{bmatrix}1&0&0\\\
0&1&0\end{bmatrix}\varSigma^{\prime}\begin{bmatrix}1&0\\\ 0&1\\\
0&0\end{bmatrix}$ (8)

The 2D Gaussians are $\alpha$-blended using eq. 4 to synthesize the camera
image.

###  3.4 The Missing Cone

In small baseline imaging scenarios, information about the covariances
($\sigma_{xz},\sigma_{yz}$) and variance ($\sigma_{zz}$) associated with the
depth-axis and the mean ($\mu_{z}$) along the depth-axis are not captured by
the camera images.

To illustrate this, consider a simple volumetric imaging scenario in fig. 1.
We consider 2D volume (the arguments are extendable to 3D volume) and assume
no inter-reflections and far-field sensing modalities, i.e., volumetric scene
after the affine approximation (with one of the views) in the GS algorithm.
For these volumetric scenes, the rendered RGB image will be the $x$-axis
projection (splat) of the volume.

Using the Fourier-slice theorem, the Fourier transform of the camera image is
the same as the x-slice of the volume’s Fourier transform. By taking several
images from various angles, we are capturing various slices of the volume’s
Fourier transform. Unfortunately, for the small baseline scenarios [21], we
have the missing cone problem [18, 17, 46], which refers to a cone in the
Fourier space that was not sensed by the camera images. This limits the
fidelity of the 3D reconstructions due to missing frequency information.

Our approach to solving this problem is to utilize complementary information
present in time-resolved measurements, particularly time-resolved measurements
from sonar, which can be obtained by taking the z-axis projection of the
volume as we will discuss next in (section 4). From the Fourier-slice theorem,
this is equivalent to obtaining a z-slice of the volume’s Fourier transform as
shown in fig. 1(c). Hence, sonars can help capture the missing cone and
augment camera data, particularly when camera baselines are limited.

##  4 Z-Axis Gaussian Splatting for Camera-Sonar Fusion

We will first extend the Gaussian splatting model to two commonly used sonars,
i.e. echosounder and FLS, by exploiting the physics of how the sonars operate.
After that, we will combine the camera and sonar Gaussian splatting to build a
fusion algorithm capable of reconstructing the 3D Gaussian parameters
accurately even for small baseline imaging scenarios.

###  4.1 Echosounder

A single-beam echosounder utilizes a single transducer element to emit a sound
pulse toward the target area and captures the echo, a time-varying sound pulse
that bounces back from the scene. This echoed signal is a function of acoustic
reflectivity and depth of various scene points. For simplicity, we assume the
acoustic albedo is constant for all the objects in the scene. Therefore, the
echoed signal can also be interpreted as a 1D histogram of scene depths
computed through acoustic time-of-flight.

In fig. 2, we illustrate the proposed splatting operation for echosounder. We
splat the 3D Gaussians along the $z$-axis, while computing the transmission
and alpha values similar to the camera splatting. The covariance of the 1D
projection of 3D Gaussians is:

$\varSigma_{\text{1D}}^{\prime}=\begin{bmatrix}0&0&1\\\
\end{bmatrix}\varSigma^{\prime}\begin{bmatrix}0\\\ 0\\\
1\end{bmatrix}=\sigma_{zz}.$ (9)

Therefore, the echosounder captures the information about mean $\mu_{z}$ and
variance $\sigma_{zz}$ along the depth axis that was missing in the camera
data. The proposed splatting operation computes the depth histogram of the
volumetric scene while accounting for the visibility term accurately. We show
the steps of the $z$-axis splatting for echosounder in algorithm 1.

Algorithm 1 Echosounder rendering

$Z[d]=0;$

for each point on the $xy$-plane do

for each visible Gaussian Kernel do

for each bin $i$ within $\mu_{z}\pm 3\sigma_{zz}$ do

$z=$ center of the bin

$Z[i]+=T\cdot\alpha\cdot\exp(-\frac{(z-\mu_{z})^{2}}{2\sigma_{zz}})$

end for

end for

end for

###  4.2 Forward Looking Sonar (FLS):

FLS is similar to an echosounder, except it contains multiple linear
transducer elements. Through beamforming, FLS recovers the range and azimuth
information but not the elevation. Therefore, in the orthographic view (fig.
2(b)), FLS measures depth histograms for various $y$ values.

To build a rendering algorithm for FLS, we splat the 3D Gaussians on the $y-z$
plane, resulting in 2D Gaussians with covariance matrix:

$\varSigma^{\prime}_{2D}=\begin{bmatrix}\sigma_{yy}&\sigma_{yz}\\\
\sigma_{yz}&\sigma_{zz}\end{bmatrix}.$ (10)

We compute the transmission and alpha values similar to camera splatting, as
the visibility terms do not change for FLS. We identify the steps in FLS
rendering in algorithm 2

In practice, an RGB camera often has a higher spatial resolution than a sonar,
so the information on the y-axis mainly comes from the RGB camera. Therefore,
we expect the reconstruction accuracy and quality of novel synthetic views
supervised by FLS to be slightly better than those from echo sonar, but not
significantly better.

Algorithm 2 FLS rendering

$Z[h,d]=0;$

for each point on the $xy$-plane do

for each visible Gaussian Kernel do

for each bin $i$ within $\mu\pm 3\cdot\sigma_{zz}$ do

$z=$ center of the bin

$j=$ y-center of the Gaussian

$Z[j,i]+=T\cdot\alpha\cdot\exp(-\frac{(z-\mu_{z})^{2}}{2\sigma_{zz}})$

end for

end for

end for

Figure 2: Ray View Transformation and Z-Axis Splatting (a) This illustration
shows the camera view. The covariance of Gaussians in the camera view is
$\varSigma=W^{T}\varSigma W$, which transforms the Gaussians from the world
view to the camera view. (b) The Gaussians are transformed into the ray view
through an affine approximation of the projection transform using the Jacobian
matrix ($J$). The covariance matrix of the Gaussians will be
$\varSigma^{\prime}=J^{T}\varSigma J$. (c) The transformed 3D Gaussian is then
projected (splat) onto the $xy$-plane for rendering camera and $z$-axis for
rendering echosounder (for collocated camera and echosounder). The gray
Gaussian is occluded by the Gaussian in the front, so the Transmission($T$) of
that Gaussian is smaller than the others independent of whether we are
rendering camera or sonar.

###  4.3 Fusion of cameras and sonars

We fuse the camera and sonar information by jointly minimizing the error
between the rendered data and measured data for both sensing modalities. We
define the camera loss $\mathcal{L}_{c}$ as:

$\mathcal{L}_{c}=\|I(x,y)-I_{\text{gs}}(x,y)\|_{1},$ (11)

where $I(x,y)$ represents the measured camera image and $I_{gs}(x,y)$ is the
rendered image from Gaussian splatting.

The sonar loss ($\mathcal{L}_{s}$) is defined as

$\mathcal{L}_{s}=\begin{cases}\|S(z)-S_{\text{gs}}(z)\|_{2},&\text{for Echo
Sonar}\\\ \|S(y,z)-S_{\text{gs}}(y,z)\|_{2},&\text{for FLS}.\end{cases}$ (12)

Here $S(z)$ and $S(y,z)$ are the measured echo sonar and FLS data;
$S_{\text{gs}}$ and $S_{\text{gs}}(y,z)$ are the Gaussian splatted renderings
discussed in section 4.1 and section 4.2.

We define the loss function as a weighted combination of camera and sonar loss
functions:

$\mathcal{L}=\mathcal{L}_{i}+w\cdot\mathcal{L}_{d}.$ (13)

Empirically, we found that setting $0.1\leq w\leq 3$ is suitable for most
scenarios. We can tune $w$ based on the confidence between camera and sonar
measurements. For instance, in fig. 4, we choose $w=3$ for the living room
scene because the scene does not have a lot of texture. We can also choose
adaptive weighing schemes similar to Qadri et al. [21] where more importance
is given to sonar measurements in the beginning and camera measurements in the
later iterations, though we did not find that weighing scheme useful in this
case.

###  4.4 Implementation

We have implemented the proposed algorithms by extending the publicly
available Gaussian splatting source code shared by Kerbl et al. [1]. We
computed the derivatives analytically and wrote cuda kernels to improve the
rendering and training speed. The average training time for all the simulated
experiments in the manuscript is around $5$ minutes on a NVIDIA 4090 GPU,
while real experiments took $20$ minutes for echosounder and 8 minutes for
FLS.

Figure 3:  Simulation and emulation training for both echosounder and FLS
fusion techniques (a) Raw depth image captured with Time-of-Flight (ToF)
camera. (b) An RGB image captured with a camera. (c) Simulated echo sonar
intensity was generated using the depth histogram and utilized as ground truth
during training. (d) A 3D Gaussian scene. We use $xy$-splatting to render RGB
images and $z$-splatting to render echosounder depth intensity distribution.
(e) Simulated FLS intensity generated by histogramming depth per row. (f) A 3D
Gaussian scene, and we splat along $xy$-direction to render RGB image and
along $yz$-direction to render FLS image. We minimize the sum of RGB loss and
corresponding depth loss to train the camera-sonar fusion algorithms.

##  5 Experimental Results

In this section, we systematically evaluate the proposed fusion techniques
using simulated, emulated, and hardware-capture data. Novel view synthesis
refers to generating images from camera views not in the training data. Using
standard photometric metrics (PSNR, SSIM, and LPIPS), we compare the novel
views generated by traditional Gaussian splatting and the proposed fusion
techniques. We also compare the reconstructed geometry as 3D point clouds
using Chamfer distance, F1 score, precision, and recall.

###  5.1 Simulation results

We use Mitsuba [47] to generate the simulated datasets. For each camera and
sonar view, we render the image from the camera view and the depth map from
the sonar view. We simulate echosounder intensity by histogramming the depth
values of the entire scene, resulting in an intensity distribution across
depth that mimics the transient sonar measurements. This histogram serves as
the ground truth either for training or for testing. The left part of fig. 3
illustrates the process of generating the ground truth echo sonar intensity
and the predicted echo sonar intensity using the proposed method.

To simulate FLS data, we histogram the depth values per row rather than the
whole image. Each row represents one azimuth angle $\theta$ in the FLS. The
resolution of the histogram along the z-axis is determined by the range $r$
that the FLS can resolve. The right part of fig. 3 illustrates the process of
generating ground-truth FLS scans. To initialize Gaussians, we found that
random initialization to be more reliable than COLMAP initialization, as the
latter sometimes suffered due to small baselines in our datasets.

Figure 4:  Novel view synthesis comparison: The incorporation of depth
information notably mitigates the presence of floaters in the reconstructed
scene. Moreover, depth information accurately positions the Gaussian kernels,
particularly in scenes with uniform color or overexposure. The average SSIM,
PSNR, and LPIPS metrics for the entire test set comprising 263 novel views are
presented in table I.

We provide two different types of scenes in the simulation process: (i) Room-
sized scenes from Mitsuba Gallery (fig. 4) and (ii) Single-object scenes from
Mitsuba Gallery, Stanford 3D Scanning Repository and online free resources. In
the room-sized scenes shown in fig. 4, we cannot freely maneuver the camera
around the scene or object and obtain a full 360-degree scan due to the
geometric constraints, and hence, we are in a restricted baseline scenario.
Further, the scenes contain saturated regions due to windows and little
texture. For these imaging scenarios, the camera-based reconstruction is ill-
posed, and the addition of depth information better constrains the
reconstruction problem.

We present the novel view synthesis comparisons (photometric comparisons) in
table II and geometry reconstruction comparisons in table I. For novel view
synthesis, we outperform the RGB-only method on PSNR, SSIM, and LPIPS metrics.
Specifically, we observe an average 5 dB improvement in PSNR compared to
camera-only GS algorithms and a 10 dB increase on challenging scenes that
contain little texture (living room scene).

The reconstructed 3D Gaussians typically have a small variance (of a couple of
pixels) and, hence, can act as reconstructed point clouds. We tessellated the
ground truth mesh and considered the mesh vertices as the ground truth point
cloud. This tessellation allows us to compute various metrics (Chamfer
distance, Recall, precision, F1-score) between the reconstructed point cloud
and the ground truth. In table I, we show the comparisons across all the
methods. The proposed techniques consistently outperform the traditional GS
algorithms. we observe an average 50% improvement in Chamfer distance and a
tenfold increase in the scenes with little texture (living room scene).

The primary reason for these improvements lies in our method’s effective
mitigation of the issue of floaters in the scene, a common challenge
encountered in volumetric rendering methods. By reducing erroneous placement
of Gaussians and enhancing both geometry accuracy and photometric details, we
achieve significant advancements in overall performance. The results also
indicate that FLS outperforms echo sonar, albeit with only a slight
improvement. This confirms our hypothesis from section 4.2 that FLS will
provide little additional information in the $y$-axis, as cameras also provide
information along that dimension and at a higher resolution.

TABLE I: Simulation: Novel view synthesis comparisons (Room sized scenes)

| RGB Only | Echo Sonar (Ours) | FLS (Ours)
---|---|---|---
Scene | PSNR↑ | SSIM↑ | LPIPS↓ | PSNR↑ | SSIM↑ | LPIPS↓ | PSNR↑ | SSIM↑ | LPIPS↓
Bedroom | 31.855 | 0.881 | 0.242 | 35.264 | 0.893 | 0.205 | 35.348 | 0.891 | 0.177
Living room | 27.508 | 0.874 | 0.331 | 37.790 | 0.948 | 0.182 | 38.457 | 0.951 | 0.176
Bathroom | 27.465 | 0.872 | 0.177 | 33.381 | 0.929 | 0.092 | 35.753 | 0.952 | 0.073

  * •

PSNR, SSIM, and LPIPS metrics. We use PSNR, SSIM, and LPIPS to evaluate the
quality of the predicted novel view RGB images. The best results are in green,
while the worst are in red. We can see that our methods outperform the RGB-
only method in all scenes. While the FLS method has the best performance, as
expected, the improvement compared to the echosounder is small.

TABLE II: Simulation: Geometric comparisons (Room sized scenes)

| RGB Only | Echo Sonar (Ours) | FLS (Ours)
---|---|---|---
Scene | Chamfer↓ | Precision↑ | Recall↑ | F1↑ | Chamfer↓ | Precision↑ | Recall↑ | F1↑ | Chamfer↓ | Precision↑ | Recall↑ | F1↑
Bedroom | 0.374 | 0.893 | 0.549 | 0.680 | 0.163 | 0.997 | 0.667 | 0.799 | 0.198 | 0.998 | 0.622 | 0.767
Living Room | 3.382 | 0.825 | 0.084 | 0.152 | 0.291 | 0.977 | 0.512 | 0.672 | 0.359 | 0.998 | 0.540 | 0.701
Bathroom | 4.545 | 0.616 | 0.594 | 0.605 | 4.136 | 0.882 | 0.521 | 0.655 | 3.912 | 0.907 | 0.485 | 0.632

  * •

Chamfer, Precision, Recall, and F1 score metrics. We use Chamfer, Precision,
Recall, and F1 score to evaluate the quality of the predicted 3D Gaussian
splatting.The best results are in green, while the worst are in red. The
metrics are calculated by comparing the point cloud of the predicted Gaussian
splatting with the ground truth mesh vertices. Our methods outperform the RGB
only method in most metrics, and the FLS method has the best performance.

In the second set of scenes, instead of complex scenes with multiple objects,
we created a series of scenes with only one diffuse object without texture.
Similar to Qadri et al. [21], we restricted camera and sonar movement along
the $x$-axis within a small range (2.3 degree on the circle) across these
simplified scenes. We visualize the results in fig. 5, showcasing the ground
truth as a mesh and the predicted Gaussian kernels as red points. In table
III, we quantify the accuracy of reconstructed geometry with various metrics.
We can observe that the proposed fusion consistently performs better than
camera-only techniques.

Figure 5:  Geometry comparison on one-object scenes. We captured the data by
moving the camera only along the $x$-axis. We show ground truth meshes and
superimpose the reconstructed Gaussians as point clouds. In the highlighted
regions, we can observe that camera-only methods reconstruct the geometry
inaccurately along the $z$-axis, whereas the proposed fusion techniques
reconstruct the geometry accurately.  TABLE III: Simulation: Geometric
comparisons (One-object scenes)

| RGB Only | Echo Sonar (Ours) | FLS (Ours)
---|---|---|---
Scene | Chamfer↓ | Precision↑ | Recall↑ | F1↑ | Chamfer↓ | Precision↑ | Recall↑ | F1↑ | Chamfer↓ | Precision↑ | Recall↑ | F1↑
Bauhaus | 0.184 | 0.943 | 0.693 | 0.799 | 0.134 | 0.899 | 0.729 | 0.805 | 0.127 | 0.894 | 0.772 | 0.829
Loong | 0.0184 | 0.8462 | 0.916 | 0.846 | 0.0093 | 0.8627 | 0.944 | 0.902 | 0.0086 | 0.8815 | 0.938 | 0.909
Titanic | 0.164 | 0.949 | 0.808 | 0.873 | 0.103 | 0.997 | 0.887 | 0.939 | 0.069 | 0.984 | 0.932 | 0.957

  * •

Chamfer, Precision, Recall, and F1 score metrics. We use Chamfer, Precision,
Recall, and F1 score to evaluate the quality of the predicted 3D Gaussian
splatting. The best results are in green, while the worst are in red. These
metrics are computed by comparing the point cloud of the predicted Gaussian
splatting with the ground truth mesh vertices. Our methods consistently
outperform the RGB-only approach across most metrics, and FLS exhibits
slightly better performance than echosounder.

###  5.2 Hardware emulation results

Figure 6:  Novel view synthesis comparison on emulated hardware. We set up a Cornell box in the lab, captured both RGB and depth images, and emulated the echosounder and FLS data. In the reconstructed scene, all methods work well on the objects with high-contrast textures. However, the RGB-only technique fails to reconstruct the white object with the same color as the background and also suffers from color bleeding. Our methods, on the other hand, successfully reconstruct the white object and do not suffer from color bleeding. For different random seeds, RGB-only techniques have high variance in the reconstructed results and have poor reconstructions (b), while our methods have consistent performance. We plot the variance across different runs in fig. 7, and we can observe that our techniques are robust to random initial seeds.  TABLE IV: Emulation Performance: Cornell Box | PSNR↑ | SSIM↑ | LPIPS↓
---|---|---|---
RGB Only | 37.499 | 0.977 | 0.093
Echo Sonar (Ours) | 42.089 | 0.987 | 0.037
FLS (Ours) | 42.142 | 0.988 | 0.036

In addition to the simulation, we emulate the sonars using the camera and
Lidar on the Sony Xperia II smartphone. We have built a Cornell box scene with
multiple objects in our lab. We translate the phone inch-by-inch on the
$xy$-plane to obtain training data. We capture RGB images and pixel-wise depth
measurements simultaneously, mirroring the process employed during simulation
in section 5.1. We position the phone in between the training samples to
generate the test dataset. We extract camera poses and Structure-from-Motion
(SFM) points using COLMAP and train the three models of GS, Echosounder, and
FLS.

The test results are shown in fig. 6 and quantified in table IV. From fig. 6,
it is evident that the RGB-only reconstruction results in significantly more
blur on the white crane, box edges, and egg doll. This could be attributed to
COLMAP creating denser points on clear edges but not on the background or
white objects that share the same color as the background. During
optimization, the RGB-only method would not produce more Gaussians because of
the similar color. However, our methods surpass the RGB-only approach by
producing clear and sharp edges with the aid of $z$-axis information. The
average PSNR of our methods is 5 dB higher than that of RGB-only, and our
techniques also have higher SSIM and lower LPIPS, as shown in table IV.

Figure 7: Variance comparison for emulations. We ran the GS algorithms 10
times with the same training and test datasets but with different random
seeds. We can see that the variance of our methods is much lower than the RGB-
only method. This indicates that our methods have consistent performance,
while the RGB-only method can sometimes result in poor results. Note that even
the best results with RGB-only are worse than the proposed techniques.

During the experiments, we observed that the RGB-only method exhibits high
variability. To investigate further, we conducted 10 training runs for each
model using the same training dataset, test dataset, and initialized SFM
points, but with random initial seeds. In fig. 7, we show the box and whisker
plot to visualize the variance between the runs. The RGB-only method can
perform poorly at times and exhibits significant variance across each training
process. In contrast, both of our methods are robust and have consistent
performance. Furthermore, even our worst-performing run outperforms the RGB-
only method.

###  5.3 Hardware results - Echo Sounder

For our set of real hardware experiments, we first utilize our proposed method
for sensor fusion between RGB cameras and a monostatic active sonar to capture
acoustic ToF data. Echo sounding is the technique of using an active sonar to
determine depth/range. Thus we have built a prototype of an echo sounder using
a single speaker and microphone that is attached to a DSLR camera for
collecting real acoustic ToF data along with RGB images. We describe our setup
and data processing procedures below.

Figure 8: Hardware prototype consists of a DSLR camera with speaker and
microphone. It is used to image a scene on the motorized turntable.  Figure 9:
Qualitative comparison of echo sonar real-data results. The comparison
presents two scenes captured using a DSLR camera and echo-sonar with a
turntable setup. Our method demonstrates a noticeable improvement in
performance over the baseline RGB-only method, both quantitatively and
qualitatively.

Experimental setup and data capture: Our setup, visualized in Figure 8 and
similar to other circular in-air systems [48, 49, 50], consists of (1)
acoustic transducer array consisting of loudspeaker tweeter (Peerless by
Tymphany OX20SC02-04) and a microphone (GRAS 46AM); (2) a Nikon D5600 DSLR
camera; and (3) a motorized circular platform (Rotary Table RTLA-90-200M) to
rotate targets. As seen in the figure, white paper and a white board was used
to cover the turntable and background, otherwise parts of the scene would be
static while the objects moved, not reflecting the experimental scenario of a
moving sonar. The tweeter transmits a linear frequency modulated chirp with
peak voltage of 3V for a duration of 1 ms waveform from starting frequency
$10kHz$ to stopping frequency $30kHz$ with a sampling frequency of $100kHz$.

For data capture, we rotate the target scene at 1-degree increments from 0 to
40 degrees while simultaneously capturing RGB video frames and the acoustic
signal from the microphone. We first use COLMAP to perform SfM to obtain an
initial point cloud and camera pose. We perform model alignment for SFM points
using the information from the experimental setup (radius of the turn table
and incremental angle) to roughly align the SFM points in 3D space with the
transient histogram to be computed. For the audio signal, we perform similar
processing for matched filtering by Reed et al. [50, 22]: (1) we implement
group delay correction in the frequency domain; (2) we subtract background
measurement of the scene with no object; (3) we convert the signal into the
analytic domain using the Hilbert Transform; (3) we perform matched filtering
in the frequency domain, and then (4) take the magnitude of the complex signal
to form a transient histogram.

For initialization of the Gaussians, we utilize the inital SfM point cloud
generated from COLMAP as well as include additional 3D Gaussians ($N=10,000$
for our experiments) who are placed at $(x,y,z)$ locations corresponding to
maximum energy in the transient histogram. This greatly improved the
performance of the method in practice compared to the baseline Gaussian
splatting.

Main experimental results: In Fig. 9, we show a qualitative comparison of our
method compared to an RGB only 3D reconstruction. This scene is a difficult
scene as COLMAP SfM points are noisy and inaccurate, which contributes to
errors in the RGB only Gaussian-splatting reconstruction. In contrast, our
method better resolves the objects in the scene as well as the background,
leveraging the transient information from the echo sounder.

TABLE V: Echo-Sonar Performance | RGB Only | Echo Sonar (Ours)
---|---|---
Scene | PSNR↑ | SSIM↑ | LPIPS↓ | PSNR↑ | SSIM↑ | LPIPS↓
Rabbit | 25.882 | 0.874 | 0.282 | 30.785 | 0.940 | 0.237
Teapot | 39.664 | 0.982 | 0.138 | 39.717 | 0.985 | 0.114

In Tab. V, we see an improvement in PSNR/SSIM/LPIPS for test views
(photometric error).

###  5.4 Hardware results - FLS

We additionally use our algorithm to fuse measurements from an optical camera
and an imaging sonar (FLS) to reconstruct a real object submerged underwater
in a water test tank. Acoustic FLS images (example in fig.10a ) resolve both
the range and azimuth of the reflecting object but not the elevation angle
which remains ambiguous. In other words, one can view each column of an
imaging sonar image as capturing the transient at a particular azimuth angle
which makes this data easily integrable with our method.

Figure 10: Underwater FLS data. (a) Example imaging sonar measurement: the
range $r$ and azimuth $\theta$ are resolved but the elevation angle $\phi$
remains ambiguous. (b) Sample camera image of the submerged test structure.

We use an existing dataset containing both imaging sonar measurements (with
$14^{\circ}$ elevation) captured using a Didson imaging sonar mounted on a
Bluefin Autonomous Underwater Vehicle and optical camera images captured using
a FLIR camera. See [51, 21] for more detail regarding the data and hardware
setup.

Since the dataset contains only front-facing camera poses with negligible
rotational changes, baseline Gaussian splatting resulted in noisy
reconstructions. Hence, for both the camera-only and FLS+camera
reconstructions, we first filter out all points outside a bounding box around
the object and then align the resulting point cloud with the GT model of the
target object using ICP.

We observe from table VI that integrating FLS information with camera images
results in higher reconstruction accuracy, as showcased by the Chamfer
distance/precision/recall/F1 metrics. This can also be observed quantitatively
in fig. 11, which shows that when only using optical cameras, the resulting
reconstruction contains high errors along the depth axis. On the other hand,
integrating FLS information allows our algorithm to better resolve depth using
the additional range information. We note that, although our result is
slightly worse compared to AONeuS [21] (although do expect better performance
with more thorough parameter tuning), the runtime of our algorithm is $\sim
9.95\times$ better.

TABLE VI: Hardware FLS: Geometric Metrics- threshold 0.05

| Chamfer ↓ | Precision↑ | Recall↑ | F1↑
---|---|---|---|---
RGB Only | 0.205 | 0.414 | 0.670 | 0.512
FLS (Ours) | 0.124 | 0.457 | 0.775 | 0.575

  * •

Figure 11: Experimental reconstructions using RGB-only and RGB+FLS. When using
RGB-only measurements, we observe high error along depth (red box).
Integrating FLS measurements improves depth resolution.

##  6 Conclusion

In this paper, we introduce a $z$-axis splatting pipeline for Gaussian
splatting, which enables the fusion of RGB camera information with various
types of depth-resolved acoustic measurements, such as echo-sounders and FLS.
Through extensive validation on simulated, emulated, and real-world hardware
experimental data, we demonstrate superior performance compared to RGB-only
methods for both novel view synthesis and geometric reconstruction. By
leveraging the depth information provided by acoustic sensors, our method
offers a promising approach to reconstructing scenes with small baselines.

Our method currently does not account for scattering. Modeling scattering may
produce more accurate reconstructions, particularly in the underwater setting.
In the fusion step, we used a simple linear model to combine the losses of
various imaging modalities. Non-linear models and adaptive models that change
the relative weights over iterations, such as the ones that Qadri et al. [21]
have used, may result in better 3D reconstruction. Exploring various
combinations of loss functions could be another interesting future direction.

While this work focused on $z$-axis splatting for sonar fusion, an interesting
future direction is to extend our approach to handle radar and lidar systems,
which share a similar acquisition model as sonars. Finally, generalizing our
method to dynamic scenes and to sensors that operate in this space, such as
Doppler cameras and frequency-modulated continuous-wave time-of-flight
cameras, could be another interesting extension of our technique.

## Acknowledgments

A.P. and Z.Q. were supported in part by NSF IIS-2326904. O.V. and S.J. were
supported in part by ONR grant N00014-23-1-2406, and NSF IIS-2326905. M.Q. and
M.K. were supported in part by ONR grant N00014-21-1-2482. K.Z. and C.A.M.
were supported in part by AFOSR Young Investigator Program award no.
FA9550-22-1-0208, ONR award no. N00014-23-1-2752, and a seed grant from SAAB,
Inc.

## References

  * [1] B. Kerbl, G. Kopanas, T. Leimkühler, and G. Drettakis, “3d gaussian splatting for real-time radiance field rendering,” _ACM Transactions on Graphics_ , vol. 42, no. 4, pp. 1–14, 2023.
  * [2] M. Zwicker, H. Pfister, J. Van Baar, and M. Gross, “Ewa volume splatting,” in _Proceedings Visualization, 2001. VIS’01._ IEEE, 2001, pp. 29–538.
  * [3] T. Lu, M. Yu, L. Xu, Y. Xiangli, L. Wang, D. Lin, and B. Dai, “Scaffold-gs: Structured 3d gaussians for view-adaptive rendering,” _arXiv preprint arXiv:2312.00109_ , 2023.
  * [4] Z. Yu, A. Chen, B. Huang, T. Sattler, and A. Geiger, “Mip-splatting: Alias-free 3d gaussian splatting,” _arXiv preprint arXiv:2311.16493_ , 2023.
  * [5] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, “Nerf: Representing scenes as neural radiance fields for view synthesis,” _Communications of the ACM_ , vol. 65, no. 1, pp. 99–106, 2021.
  * [6] J. Gao, C. Gu, Y. Lin, H. Zhu, X. Cao, L. Zhang, and Y. Yao, “Relightable 3d gaussian: Real-time point cloud relighting with brdf decomposition and ray tracing,” _arXiv preprint arXiv:2311.16043_ , 2023.
  * [7] Y. Jiang, J. Tu, Y. Liu, X. Gao, X. Long, W. Wang, and Y. Ma, “Gaussianshader: 3d gaussian splatting with shading functions for reflective surfaces,” _arXiv preprint arXiv:2311.17977_ , 2023.
  * [8] Z. Zhu, Z. Fan, Y. Jiang, and Z. Wang, “Fsgs: Real-time few-shot view synthesis using gaussian splatting,” _arXiv preprint arXiv:2312.00451_ , 2023.
  * [9] J. Fang, J. Wang, X. Zhang, L. Xie, and Q. Tian, “Gaussianeditor: Editing 3d gaussians delicately with text instructions,” _arXiv preprint arXiv:2311.16037_ , 2023.
  * [10] D. Das, C. Wewer, R. Yunus, E. Ilg, and J. E. Lenssen, “Neural parametric gaussians for monocular non-rigid object reconstruction,” _arXiv preprint arXiv:2312.01196_ , 2023.
  * [11] Z. Yang, H. Yang, Z. Pan, X. Zhu, and L. Zhang, “Real-time photorealistic dynamic scene representation and rendering with 4d gaussian splatting,” _arXiv preprint arXiv:2310.10642_ , 2023.
  * [12] A. Kratimenos, J. Lei, and K. Daniilidis, “Dynmf: Neural motion factorization for real-time dynamic view synthesis with 3d gaussian splatting,” _arXiv preprint arXiv:2312.00112_ , 2023.
  * [13] R. Shao, J. Sun, C. Peng, Z. Zheng, B. Zhou, H. Zhang, and Y. Liu, “Control4d: Dynamic portrait editing by learning 4d gan from 2d diffusion-based editor,” _arXiv preprint arXiv:2305.20082_ , 2023.
  * [14] Y.-H. Huang, Y.-T. Sun, Z. Yang, X. Lyu, Y.-P. Cao, and X. Qi, “Sc-gs: Sparse-controlled gaussian splatting for editable dynamic scenes,” _arXiv preprint arXiv:2312.14937_ , 2023.
  * [15] H. Yu, J. Julin, Z. Á. Milacski, K. Niinuma, and L. A. Jeni, “Cogs: Controllable gaussian splatting,” _arXiv preprint arXiv:2312.05664_ , 2023.
  * [16] J. Huang and H. Yu, “Point’n move: Interactive scene object manipulation on gaussian splatting radiance fields,” _arXiv preprint arXiv:2311.16737_ , 2023.
  * [17] F. Macias-Garza, K. R. Diller, and A. C. Bovik, “Missing cone of frequencies and low-pass distortion in three-dimensional microscopic images,” _Optical Engineering_ , vol. 27, no. 6, pp. 461–465, 1988.
  * [18] X. Liu, S. Bauer, and A. Velten, “Analysis of feature visibility in non-line-of-sight measurements,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 10 140–10 148.
  * [19] J. Chung, J. Oh, and K. M. Lee, “Depth-regularized optimization for 3d gaussian splatting in few-shot images,” 2024.
  * [20] H. Liu, M. Roznere, and A. Q. Li, “Deep underwater monocular depth estimation with single-beam echosounder,” in _2023 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2023, pp. 1090–1097.
  * [21] M. Qadri, K. Zhang, A. Hinduja, M. Kaess, A. Pediredla, and C. A. Metzler, “Aoneus: A neural rendering framework for acoustic-optical sensor fusion,” 2024.
  * [22] A. Reed, J. Kim, T. Blanford, A. Pediredla, D. Brown, and S. Jayasuriya, “Neural volumetric reconstruction for coherent synthetic aperture sonar,” _ACM Transactions on Graphics (TOG)_ , vol. 42, no. 4, pp. 1–20, 2023.
  * [23] M. F. Fallon, J. Folkesson, H. McClelland, and J. J. Leonard, “Relocating underwater features autonomously using sonar-based slam,” _IEEE Journal of Oceanic Engineering_ , vol. 38, no. 3, pp. 500–513, 2013.
  * [24] P. Yang, H. Liu, M. Roznere, and A. Q. Li, “Monocular camera and single-beam sonar-based underwater collision-free navigation with domain randomization,” in _The International Symposium of Robotics Research_. Springer, 2022, pp. 85–101.
  * [25] M. Roznere and A. Q. Li, “Underwater monocular image depth estimation using single-beam echosounder,” in _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2020, pp. 1785–1790.
  * [26] T. Zhang, S. Liu, X. He, H. Huang, and K. Hao, “Underwater target tracking using forward-looking sonar for autonomous underwater vehicles,” _Sensors_ , vol. 20, no. 1, p. 102, 2019.
  * [27] P. S. Heckbert, “Fundamentals of texture mapping and image warping,” 1989.
  * [28] M. Zwicker, H. Pfister, J. Van Baar, and M. Gross, “Surface splatting,” in _Proceedings of the 28th annual conference on Computer graphics and interactive techniques_ , 2001, pp. 371–378.
  * [29] ——, “Ewa splatting,” _IEEE Transactions on Visualization and Computer Graphics_ , vol. 8, no. 3, pp. 223–238, 2002.
  * [30] B. Fei, J. Xu, R. Zhang, Q. Zhou, W. Yang, and Y. He, “3d gaussian as a new vision era: A survey,” _arXiv preprint arXiv:2402.07181_ , 2024.
  * [31] H. Matsuki, R. Murai, P. H. Kelly, and A. J. Davison, “Gaussian splatting slam,” _arXiv preprint arXiv:2312.06741_ , 2023.
  * [32] N. Keetha, J. Karhade, K. M. Jatavallabhula, G. Yang, S. Scherer, D. Ramanan, and J. Luiten, “Splatam: Splat, track & map 3d gaussians for dense rgb-d slam,” _arXiv preprint arXiv:2312.02126_ , 2023.
  * [33] C. Yan, D. Qu, D. Wang, D. Xu, Z. Wang, B. Zhao, and X. Li, “Gs-slam: Dense visual slam with 3d gaussian splatting,” _arXiv preprint arXiv:2311.11700_ , 2023.
  * [34] S. Sun, M. Mielle, A. J. Lilienthal, and M. Magnusson, “High-fidelity slam using gaussian splatting with rendering-guided densification and regularized optimization,” _arXiv preprint arXiv:2403.12535_ , 2024.
  * [35] F. Ferreira, D. Machado, G. Ferri, S. Dugelay, and J. Potter, “Underwater optical and acoustic imaging: A time for fusion? a brief overview of the state-of-the-art,” _OCEANS 2016 MTS/IEEE Monterey_ , pp. 1–6, 2016.
  * [36] Y. Raaj, A. John, and T. Jin, “3d object localization using forward looking sonar (fls) and optical camera via particle filter based calibration and fusion,” _OCEANS 2016 MTS/IEEE Monterey_ , pp. 1–10, 2016.
  * [37] S. Williams and I. Mahon, “Simultaneous localisation and mapping on the great barrier reef,” _IEEE International Conference on Robotics and Automation, 2004. Proceedings. ICRA’04. 2004_ , vol. 2, pp. 1771–1776, 2004.
  * [38] M. Babaee and S. Negahdaripour, “3-d object modeling from 2-d occluding contour correspondences by opti-acoustic stereo imaging,” _Computer Vision and Image Understanding_ , vol. 132, pp. 56–74, 2015.
  * [39] R. Nabati and H. Qi, “Centerfusion: Center-based radar and camera fusion for 3d object detection,” in _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2021, pp. 1527–1536.
  * [40] R. Grover, G. Brooker, and H. F. Durrant-Whyte, “A low level fusion of millimeter wave radar and night-vision imaging for enhanced characterization of a cluttered environment,” in _Proceedings 2001 Australian Conference on Robotics and Automation_ , 2001.
  * [41] M. Bijelic, T. Gruber, F. Mannan, F. Kraus, W. Ritter, K. Dietmayer, and F. Heide, “Seeing through fog without seeing fog: Deep multimodal sensor fusion in unseen adverse weather,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 11 682–11 692.
  * [42] Y. M. Kim, C. Theobalt, J. Diebel, J. Kosecka, B. Miscusik, and S. Thrun, “Multi-view image and tof sensor fusion for dense 3d reconstruction,” in _2009 IEEE 12th international conference on computer vision workshops, ICCV workshops_. IEEE, 2009, pp. 1542–1549.
  * [43] D. B. Lindell, M. O’Toole, and G. Wetzstein, “Single-photon 3d imaging with deep sensor fusion.” _ACM Trans. Graph._ , vol. 37, no. 4, p. 113, 2018.
  * [44] M. Nishimura, D. B. Lindell, C. Metzler, and G. Wetzstein, “Disambiguating monocular depth estimation with a single transient,” in _European Conference on Computer Vision_. Springer, 2020, pp. 139–155.
  * [45] M. Salvi and K. Vaidyanathan, “Multi-layer alpha blending,” in _Proceedings of the 18th Meeting of the ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games_ , 2014, pp. 151–158.
  * [46] A. H. Delaney and Y. Bresler, “Globally convergent edge-preserving regularized reconstruction: an application to limited-angle tomography,” _IEEE Transactions on image processing_ , vol. 7, no. 2, pp. 204–221, 1998.
  * [47] M. Nimier-David, D. Vicini, T. Zeltner, and W. Jakob, “Mitsuba 2: A retargetable forward and inverse renderer,” _ACM Transactions on Graphics (TOG)_ , vol. 38, no. 6, pp. 1–17, 2019.
  * [48] T. E. Blanford, J. D. McKay, D. C. Brown, J. D. Park, and S. F. Johnson, “Development of an in-air circular synthetic aperture sonar system as an educational tool,” vol. 36. ASA, 8 2019, p. 070002.
  * [49] J. D. Park, T. E. Blanford, D. C. Brown, and D. Plotnick, “Alternative representations and object classification of circular synthetic aperture in-air acoustic data,” _The Journal of the Acoustical Society of America_ , vol. 148, no. 4_Supplement, pp. 2661–2661, 2020.
  * [50] A. Reed, T. Blanford, D. C. Brown, and S. Jayasuriya, “Sinr: Deconvolving circular sas images using implicit neural representations,” _IEEE Journal of Selected Topics in Signal Processing_ , 2022.
  * [51] M. Qadri, M. Kaess, and I. Gkioulekas, “Neural implicit surface reconstruction using imaging sonar,” in _2023 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2023, pp. 1040–1047.


# Unexplained Spectral Phenomena in the Interstellar Medium: an introduction

Sun Kwok11affiliationmark<EMAIL_ADDRESS>
###### Abstract

There exists a number of astronomical spectral phenomena that have remained
unidentified after decades of extensive observations. The diffuse interstellar
bands, the 220 nm feature, unidentified infrared emission bands, extended red
emissions, and 21 and 30 $\mu$m emission features are seen in a wide variety
of astrophysical environments. The strengths of these features suggest that
they originate from chemical compounds made of common elements, possibly
organic in nature. The quest to understand how such organic materials are
synthesized and distributed across the Galaxy represents a major challenge to
our understanding of the chemical content of the Universe.

00footnotetext: Department of Earth, Ocean, and Atmospheric Sciences,
University of British Columbia, Vancouver, Canada; corresponding
<EMAIL_ADDRESS>
Astrophysics and Space Science (in press)

Keywords astrobiology; astrochemistry; ISM: lines and bands; ISM: molecules;
planetary nebulae: general; stars: AGB and post-AGB

##  1 Historical background

In 1814, Joseph Fraunhofer observed and tabulated 473 dark lines in the
spectrum of the Sun. By comparing the Fraunhofer lines with the bright color
lines observed in heated chemical elements, Gustav Kirchhoff and Robert Bunsen
identified the solar lines as originating from elements sodium, calcium,
magnesium, iron, chromium, nickel, barium, copper, and zinc. This led to the
realization that the Sun is made of the same chemical elements as those of the
Earth and marked the demise of the Aristotelian concept of celestial objects
being composed of ether.

In 1868, Norman Lockyer discovered a bright yellow-orange line in the solar
spectrum, which he suggested to be due to a new element and named it “helium”.
The existence of this new element was confirmed by the discovery of its
terrestrial counterpart in 1895. In 1869, a green line was found in the
spectrum of the corona of the Sun during the total solar eclipse, and this
line was also thought to be a new element and was named “coronium”. In 1864,
William Huggins found a bright green emission line in the spectrum of the
planetary nebula NGC 6543. Since this line did not match any lines from known
chemical elements, it was suggested to be due to a new element “nebulium”. It
was not until the early 20th century that “coronium” was identified as
electronic transitions from ionized iron (Fe13+) due to the high temperature
in the solar corona, and “nebulium” was identified as forbidden lines of
ionized oxygen (O++) arising under low-density nebular conditions. These
applications of quantum theory of atoms and laboratory atomic spectroscopy to
explain astronomical observations represented the beginning of the modern
discipline of astrophysics.

In the early 21st century, we are facing similar challenges in the form of a
number of unexplained spectral phenomena in the interstellar medium. The
diffuse interstellar bands (DIBs) were discovered in 1922, when two optical
absorption lines of interstellar origin were seen in the spectra of stars
(Heger, 1922). As of 2021, over 500 DIBs in the ultraviolet, visible, and
infrared wavelength regions have been cataloged along the line of sights of
over 100 stars.

The 220 nm ultraviolet feature was discovered in 1965 (Stecher, 1965). This
feature is seen in the extinction curves of many stars, with
characteristically consistent profiles and peak wavelengths. Its wide presence
suggests that the carrier is a common constituent of the diffuse interstellar
medium.

The extended red emission (ERE) is a broad ($\Delta\lambda\sim$ 80 nm)
emission band peaking between 650 and 800 nm, first discovered in the
reflection nebula HD44179 (Cohen et al., 1975). ERE has been detected in
reflection nebulae, dark nebulae, cirrus clouds, planetary nebulae, H ii
regions, diffuse interstellar medium, and haloes of galaxies. It is commonly
attributed to photoluminescence powered by far UV photons. It is estimated
that $\sim$4% of the energy absorbed by interstellar dust at $\lambda<$0.55
$\mu$m is emitted in the form of the ERE.

A family of unidentified infrared emission (UIE) features at 3.3, 6.2, 7.7,
8.6, and 11.3 $\mu$m was discovered in the spectrum of the planetary nebula
NGC 7027 (Russell, Soifer, and Willner, 1977). The 3.3 $\mu$m feature was
first identified as the C$-$H stretching mode of aromatic compounds by Knacke
(1977). The organic origin of the UIE bands was extensively discussed by Duley
& Williams (1981), who assigned the 3.3 and 11.3 $\mu$m features to graphitic
(aromatic) materials.

Also present in astronomical spectra are emission features around 3.4 $\mu$m,
which arise from symmetric and anti-symmetric C$-$H stretching modes of methyl
and methylene groups (Jourdain de Muizon, D’Hendecourt and Geballe, 1990). The
bending modes of these groups also manifest themselves at 6.9 and 7.3 $\mu$m.
In addition, there are unidentified emission features at 15.8, 16.4, 17.4,
17.8, and 18.9 $\mu$m. The emission bands themselves are often accompanied by
strong, broad emission plateaus features at 6$-$9, 10$-$15, and 15$-$20
$\mu$m. The first two plateau features have been identified as superpositions
of in-plane and out-of-plane bending modes emitted by a mixture of aliphatic
side groups attached to aromatic rings (Kwok, Volk, and Bernath, 2001). This
collection of features in the UIE family has been observed in planetary
nebulae, reflection nebulae, novae, Hii regions, and galaxies.

The unidentified infrared emission feature around 30 $\mu$m was discovered
from Kuiper Airborne Observatory observations (Forrest, Houck, and McCarthy,
1981). It was first seen in carbon-rich asymptotic giant branch stars,
planetary nebulae, and proto-planetary nebulae. Among planetary nebulae in the
Magellanic Clouds, about half of them possess the 30 $\mu$m feature (Bernard-
Salas et al., 2009).

The 21 $\mu$m emission feature was first discovered in Infrared Astronomical
Satellite Low Resolution Spectroscopic survey (Kwok, Volk, and Hrivnak, 1989).
The feature peaks around 20.1 $\mu$m and shows a broad ($\sim$ 2 $\mu$m) and
smooth profile. The 21 $\mu$m feature is almost always accompanied by the 30
$\mu$m feature. The 21 $\mu$m feature is primarily observed in carbon-rich
post-asymptotic-giant-branch stars. From the spectral energy distribution of
21 $\mu$m sources, it is found that the 21 and 30 $\mu$m features can carry
respectively up to 8 and 20% of the total energy output of the objects
(Hrivnak, Volk, and Kwok, 2000).

##  2 Distribution in the Universe

Although these unexplained spectral phenomena were first discovered in the
spectra of stars, they are not isolated phenomena as they are observed in a
wide range of celestial objects throughout the Universe. DIBs have been
detected in external galaxies with redshifts up to 0.5 (Sarre, 2006). The 220
nm feature has been detected in interplanetary dust particles in the Solar
System (Bradley et al., 2005) as well as in distant galaxies with redshift
$>$2 (Elíasdóttir et al., 2009). A survey of 150 galaxies by the AKARI
satellite found that $\sim$0.1% of the total energy of the parent galaxies is
emitted through the 3.3 $\mu$m UIE band (Imanishi et al., 2010). In some
active galaxies, up to 20% of the total luminosity of the galaxy is emitted in
the UIE bands (Smith et al., 2007). The 3.4 $\mu$m aliphatic feature has been
detected in absorption in ultraluminous infrared galaxies (Mason et al., 2004;
Risaliti et al., 2006). The detection of UIE bands in high-redshift galaxies
(Teplitz et al., 2007) and quasars (Lutz et al., 2007) implies that organic
compounds were widely present as early as 10 billion years ago. This suggests
that abiological synthesis of complex organics was active through most of the
history of the Universe.

##  3 Chemical nature of the carriers

Because of the strengths and ubiquitous nature of the features, the carrier
must be made of common, abundant elements, with the element carbon probably
playing a major role. While the DIBs are commonly believed to be due to
electronic transitions of gas-phase carbon-based molecules, the carrier of the
220 nm feature is more likely to be a carbonaceous solid such as amorphous
carbon (Mennella et al., 1998), carbon onions (Iglesias-Groth 2004),
hydrogenated fullerences (Cataldo & Iglesias-Groth 2009), or polycrystalline
graphite (Papoular & Papoular 2009). Among the hundreds of DIBs, only two
(963.2 and 957.7 nm) have been positively identified as originating from
ionized fullerene (C${}_{60}^{+}$, Foing & Ehrenfreund, 1994; Campbell et al.,
2015). Two weaker lines of C${}_{60}^{+}$ at 942.8 and 936.6 nm have also been
suggested to have counterparts in DIBs (Walker et al., 2015).

A variety of chemical structures have been suggested as the carriers of the
UIE bands. These include polycyclic aromatic hydrocarbon (PAH) molecules
(Léger and Puget, 1984; Allamandola, Tielens, and Barker, 1989), small
carbonaceous molecules (Bernstein and Lynch, 2009), hydrogenated amorphous
carbon (HAC), soot and carbon nanoparticles (Hu and Duley, 2008), quenched
carbonaceous composite particles (QCC, Sakata et al., 1987), coal and kerogen
(Papoular et al., 1989; Papoular, 2001), petroleum fractions (Cataldo,
Feheyan, and Heymann, 2002), and mixed aromatic/aliphatic organic
nanoparticles (MAON, Kwok & Zhang, 2013).

Proposed carriers for the 21 $\mu$m feature include hydrogenated fullerenes
(Webster 1995), titanium carbide (von Helden et al., 2000), silicon carbide
(Speck and Hofmeister, 2004), and thiourea groups attached to
aromatic/aliphatic structures (Papoular, 2011). Since ERE is the result of
photoluminescience, the carrier is likely a semiconductor with a nonzero band
gap. Other proposed carbon-based carriers include QCC (Sakata etal, 1992), C60
(Webster, 1993), and nanodiamonds (Chang, Chen, & Kwok, 2006).

##  4 Future outlook

In this topical collection, the recent developments in our understanding of
the ERE and the 21/30 $\mu$m features are reviewed by Witt and Lai (2020), and
Volk, Sloan, and Kraemer (2020) respectively. The possible ERE-DIB connection
is also discussed by Witt and Lai (2020). The scenario of coal and petroleum
related compounds as carriers of UIE is discussed by Cataldo, García-Hernández
and Manchado (2020). The possibilities of hydrogenated fullerenes as carriers
of DIB, ERE, and 220 nm features are reviewed by Zhang, Sadjadi, and Hsia
(2020).

Is it possible that these unexplained spectral phenomena be related to each
other? The identification of two DIBs with C${}_{60}^{+}$ suggests the
possibility that the DIB carrier molecules may be breakdown products of large,
complex organic compounds, such as MAONs or other amorphous hydrocarbons. The
presence of the 8 and 12 $\mu$m UIE plateau features in fullerene sources also
suggests that fullerenes and UIE carriers may share common precursors.

While we do not know when and where the carriers of DIBs and 220 nm features
are synthesized, the observations of UIE bands, ERE, and 21/30 $\mu$m features
in the circumstellar environment dictate that the chemical synthesis time
scale is constrained by the evolutionary and dynamical time scale of the
circumstellar envelopes. For example, UIE bands are observed to emerge over
time scales of $\sim$103 years in the proto-planetary nebulae phase (Kwok,
Volk, & Hrivnak, 1999), and over time scales of weeks in novae (Helton et al.,
2011). This suggests that the synthesis of the UIE carriers is extremely
efficient. How such synthesis can occur so rapidly under low-density
conditions is not understood by our current chemical models.

Could the carriers of these unexplained spectral phenomena be new chemical
compounds unobserved on Earth, as in the case of the discovery of helium in
the Sun, or could they be the result of unusual physical environment as in the
case of “coronium” and “nebulium”? Further development in laboratory
spectroscopy will hold the key to the identification of the carriers.

Snow (2014) has suggested that DIBs alone represent the largest reservoir of
organic material in the Galaxy, so no matter what the exact nature of the
carriers is, they must be an important constituent of the Cosmos. The
resolution of these spectral mysteries may bring about a new view of our
understanding of the chemical content of the Universe.

Acknowledgements

This work is supported by a grant from the Natural and Engineering Research
Council of Canada.

## References

  * Allamandola, Tielens, and Barker (1989) Allamandola, L.J., Tielens, A.G.G.M., Barker, J.R.: Astrophys. J. Suppl. 71. 733 (1989)
  * Bernard-Salas et al. (2009) Bernard-Salas, J., Peeters, E., Sloan, G. C., et al. , Astrophys. J., 699. 1541 (2009)
  * Bernstein and Lynch (2009) Bernstein, L.S., Lynch, D.K.: Astrophys J 704. 226 (2009)
  * Bradley et al. (2005) Bradley, J., Dai, Z.R., Erni, R., Browning, N., Graham, G., Weber, P., Smith, J., Hutcheon, I., Ishii, H., Bajt, S., Floss, C., Stadermann, F., Sandford, S.: Science 307. 244 (2005)
  * Campbell et al. (2015) Campbell, E.K., Holz, M., Gerlich, D., Maier, J.P.: Nature 523. 322 (2015)
  * Cataldo, Feheyan, and Heymann (2002) Cataldo, F., Keheyan, Y., Heymann, D.: International J. of Astrobiology 1. 79 (2002)
  * Cataldo, García-Hernández and Manchado (2020) Cataldo, F., García-Hernández, D. A., & Manchado, A.: Astrophys. Space Sci.365. 81 (2020)
  * Chang, Chen, & Kwok (2006) Chang, H.-C., Chen, K., Kwok, S.: Astrophys. J. Lett. 639. L63 (2006)
  * Cohen et al. (1975) Cohen, M., Anderson, C. M., Cowley, A., et al.: Astrophys. J. 196. 179 (1975)
  * Duley & Williams (1981) Duley, W.W., Williams, D.A.: Mon. Not. R. Astron. Soc. 196. 269 (1981)
  * Elíasdóttir et al. (2009) Elíasdóttir, Á. et al.: Astrophys. J. 697. 1725 (2009)
  * Foing & Ehrenfreund (1994) Foing, B.H., Ehrenfreund, P.: Nature 369. 296 (1994)
  * Forrest, Houck, and McCarthy (1981) Forrest, W. J., Houck, J. R., & McCarthy, J. F. 1981, Astrophys. J. 248. 195 (1981)
  * Heger (1922) Heger, M. L.: Lick Observatory Bulletin 10. 141 (1922)
  * Helton et al. (2011) Helton, L. A., Evans, A., Woodward, C. E., Gehrz, R. D.: in Proceedings of the EAS Publications Series 46. 407
  * Hrivnak, Volk, and Kwok (2000) Hrivnak, B. J., Volk, K., Kwok, S.: Astrophys. J. 535. 275 (2000)
  * Hu and Duley (2008) Hu, A., Duley, W.W.: Astrophys. J. Lett. 677. L153 (2008)
  * Imanishi et al. (2010) Imanishi, M., Nakagawa, T., Shirahata, M., Ohyama, Y., Onaka, T.: Astrophys. J. 721 1233 (2010)
  * Jourdain de Muizon, D’Hendecourt and Geballe (1990) Jourdain de Muizon, M., D’Hendecourt, L. B., Geballe, T. R.: Astron. Astrophys. 235. 367 (1990)
  * Knacke (1977) Knacke, R.F.: Nature 269. 132 (1977)
  * Kwok & Zhang (2013) Kwok, S., Zhang, Y.: Astrophys. J. 771. 5 (2013)
  * Kwok, Volk, and Bernath (2001) Kwok, S., Volk, K., Bernath, P.: Astrophys. J. Lett. 554. L87 (2001)
  * Kwok, Volk, and Hrivnak (1989) Kwok, S., Volk, K. M., & Hrivnak, B. J.: Astrophys. J. Lett. 345. L51 (1989)
  * Kwok, Volk, & Hrivnak (1999) Kwok, S., Volk, K., & Hrivnak, B. J.: Astron. Astrophys. 350. L35 (1999)
  * Léger and Puget (1984) Léger, A., Puget, J.L.: Astron. Astrophys. 137. L5 (1984)
  * Lutz et al. (2007) Lutz, D., et al.: Astrophys. J. Lett. 661. L25 (2007)
  * Mason et al. (2004) Mason, R.E., Wright, G., Pendleton, Y., Adamson, A.: Astrophys. J. 613. 770 (2004)
  * Mennella et al. (1998) Mennella, V., Colangeli, L., Bussoletti, E., Palumbo, P., Rotundi, A.: Astrophys. J. Lett. 507. L177 (1998)
  * Papoular (2001) Papoular, R.: Astron. Astrophys. 378. 597 (2001)
  * Papoular (2011) Papoular, R.: Mon. Not. R. Astron. Soc. 415. 494 (2011)
  * Papoular & Papoular (2009) Papoular, R.J., Papoular, R.: Mon. Not. R. Astron. Soc. 394. 2175 (2009)
  * Papoular et al. (1989) Papoular, R., Conrad, J., Giuliano, M., Kister, J., Mille, G.: Astron. Astrophys. 217. 204 (1989)
  * Risaliti et al. (2006) Risaliti, G., Maiolino, R., Marconi, A., et al. Mon. Not. R. Astron. Soc. 365. 303 (2006)
  * Russell, Soifer, and Willner (1977) Russell, R.W., Soifer, B.T., Willner, S.P.: Astrophys. J. Lett. 217 L149 (1977)
  * Sakata et al. (1987) Sakata, A., Wada, S., Onaka, T., Tokunaga, A.T.: Astrophys. J. Lett. 320. L63 (1987)
  * Sakata etal (1992) Sakata, A. etal: Astrophys. J. 393. L83 (1992)
  * Sarre (2006) Sarre, P.J.: J. Molecular Spectroscopy 238(1). 1 (2006)
  * Smith et al. (2007) Smith, J.D.T., et al.: Astrophys. J. 656. 770 (2007)
  * Snow (2014) Snow, T.P.: in IAU Symposium 297: the Interstellar Diffuse Bands, eds. J. Cami & N.L.J. Cox, CUP, p. 3 (2014)
  * Speck and Hofmeister (2004) Speck, A. K., Hofmeister, A. M.: Astrophys. J. 600. 986 (2004)
  * Stecher (1965) Stecher, T. P.: Astrophys. J. 142. 1683 (1965)
  * Teplitz et al. (2007) Teplitz, H.I., et al.: Astrophys. J. 659. 941 (2007)
  * Walker et al. (2015) Walker, G. A. H., Bohlender, D. A., Maier, J. P., Campbell, E. K.: Astrophys. J. Lett. 812. L8 (2015)
  * Volk, Sloan, and Kraemer (2020) Volk, K., Sloan, G. C., Kraemer, K. E.: Astrophys. Space Sci. 365. 88 (2020)
  * von Helden et al. (2000) von Helden, G. et al.: Science 288. 313 (2000)
  * Webster (1993) Webster, A.: Mon. Not. R. Astron. Soc. 264. L1 (1993)
  * Webster (1995) Webster, A.: Mon. Not. R. Astron. Soc. 277. 1555 (1995)
  * Witt and Lai (2020) Witt, A. N., Lai, T. S.-Y.: Astrophys. Space Sci. 365. 58 (2020)
  * Zhang, Sadjadi, and Hsia (2020) Zhang, Y., Sadjadi, S., Hsia, C.-H.: Astrophys. Space Sci. 365. 67 (2020)