# Concept-aware clustering for decentralized deep learning under temporal
shift

Marcus Toftås  Emilie Klefbom  Edvin Listo Zec  Martin Willbo  Olof Mogren

###### Abstract

Decentralized deep learning requires dealing with non-iid data across clients,
which may also change over time due to temporal shifts. While non-iid data has
been extensively studied in distributed settings, temporal shifts have
received no attention. To the best of our knowledge, we are first with
tackling the novel and challenging problem of decentralized learning with non-
iid and dynamic data. We propose a novel algorithm that can automatically
discover and adapt to the evolving concepts in the network, without any prior
knowledge or estimation of the number of concepts. We evaluate our algorithm
on standard benchmark datasets and demonstrate that it outperforms previous
methods for decentralized learning.

Decentralized learning, ICML

Figure 1: An illustration of the proposed solution. Each client $k$ has a
model consisting of a feature extractor $\mathcal{F}^{k}_{\theta}$ and a
classifier $\mathcal{F}^{k}_{\phi}$. The training consists of three steps: (1)
Uniform sampling: A subset of clients $\mathcal{R}$ is randomly selected and
client $k$ updates both model components using FedAvg with $\mathcal{R}$. (2)
Similarity-based sampling: A subset of clients $\mathcal{S}$ is selected based
on their similarity $s$ to client $k$, which updates only the classifier
component using FedAvg with $\mathcal{S}$. (3) Local training: Client $k$
trains both model components locally and fine-tunes the classifier component.

##  1 Introduction

The proliferation of smartphones and other devices that can continuously
collect and transmit data has substantially increased the amount of data
available for machine learning-based applications. However, sharing data
explicitly may not be possible or desirable in some scenarios. For instance,
users and businesses may have privacy concerns preventing disclosure of
private or sensitive data. Alternatively, legal regulations such as the GDPR
(Voigt & Von dem Bussche, 2017) and other data protection acts may prohibit
data sharing. Moreover, practical limitations such as large distributed
datasets or low network bandwidth may make transmitting data to a centralized
location infeasible. In such scenarios, distributed frameworks like federated
learning (FL) can be a viable solution, as they communicate model parameters
instead of data, and where the models are aggregated using federated averaging
(FedAvg) (McMahan et al., 2017). FL has already demonstrated its scalability
and applicability in various domains, such as hospitals (Dayan et al., 2021),
retail stores (Yang et al., 2019), and at companies such as Google (McMahan &
Thakurta, 2022).

In the FL framework multiple clients collaborate to train a shared model
without exchanging their local data. However, FL often relies on a central
node to coordinate the communication among the clients, which can cause
communication bottlenecks and single points of failure. To overcome these
limitations, decentralized learning proposes a peer-to-peer communication
protocol that eliminates the need for a central node and reduces the
vulnerability and computational load of any single node. However,
decentralized learning also poses new challenges, such as how to optimize the
client models in a decentralized manner.

Traditionally, decentralized machine learning has used consensus optimization,
where clients aim to agree on a common model using a gossip learning approach
(Kempe et al., 2003; Boyd et al., 2006; Blot et al., 2016). This works well
when the data distributions across clients are similar or identical. However,
when the data distributions or tasks differ significantly across clients,
consensus optimization can be detrimental. Therefore, recent work has
suggested viewing decentralized learning as a clustering problem, where
clients try to find suitable collaborators in a network of peers and avoid
merging their models with dissimilar ones (Onoszko et al., 2021; Li et al.,
2022; Listo Zec et al., 2023). All of these works consider non-iid data in
decentralized deep learning, however they still assume that the data
distributions are stationary in time. For real world scenarios, this is often
not the case for edge devices that continuously collect new data.

This work presents the first study of decentralized deep learning with
temporal shifts, which account for the dynamic nature of data distributions
over time. Our main contribution is an algorithm that allows clients to learn
personalized models in non-iid settings where their concepts may evolve over
time. Our problem setting is related to the recent work of (Jothimurugesan et
al., 2023), which investigates non-iid data across clients and time in
federated learning with a central server.

##  2 Background

###  2.1 Problem formulation

Decentralized learning involves clients solving their own optimization
problems, such as supervised learning. Previous work has shown that clients
can benefit from communicating models with other clients who have similar
tasks (Listo Zec et al., 2023). However, the authors assumed that the data
distributions of each client are stationary. We address the challenges that
emerge when the data distributions of clients vary over time. Our main
contribution is to demonstrate that existing decentralized learning methods
are not resilient to temporal shifts, and to propose a simple solution based
on novel hierarchical model aggregation. We empirically show that our solution
leads to improved performance for this problem.

We consider an empirical risk minimization (ERM) setup with $K$ clients that
communicate (synchronously) in a peer-to-peer network, where any pair of
clients can communicate at each communication round. Each client $k$ has a
(private) training set $\mathcal{D}^{t}_{k}(x,y)$ generated by the underlying
distribution $p^{t}_{k}(x,y)$ at time step $t$. We assume that the data is
non-iid both across clients and over time, which is more realistic than the
common assumption of only having non-iid data across clients. We follow
Jothimurugesan et al. (2023) and Gama et al. (2014) and say that there is a
concept shift at client $k$ if $p_{k}^{t}\neq p_{k}^{t-1}$.

This work aims to design an algorithm that can handle shifts in a distributed
setting. Concept shift can be caused by various types of shifts over time. In
this paper, we focus on two main types of shifts: covariate shift (where the
input distribution $p(x)$ changes but the label distribution $p(y|x)$ is
stationary) and label shift (where the label distribution $p(y)$ changes but
the distribution $p(x|y)$ is stationary.) (Kairouz et al., 2021).

###  2.2 Motivation

Decentralized learning is appealing in situations where a central server is
undesirable; this could be due to privacy concerns (lack of trust in the
central server) or scalability issues (central server being a bottleneck). For
example, consider $K$ users with their own data distributions. A possible
scenario is $K$ smartphones that collect images that reflect the users’
preferences and activities. Rather than relying on a central server, each user
can communicate with peers in a network to jointly solve some optimization
problem, such as learning a supervised image recognition task. However, the
data distributions may differ across clients due to their personal
preferences. In realistic scenarios, the images collected may also vary over
time, which has not been studied in decentralized peer-to-peer learning
before. Therefore, the goal of this work is to study distributed temporal
shifts.

##  3 Method

We model each client as having a concept that reflects its true optimization
objective at any point in time. A client benefits from communicating with
other clients that have the same concept as itself (i.e. similar data
distribution), since FedAvg becomes detrimental otherwise (McMahan et al.,
2017). This can thus be framed as a clustering problem, where each client
tries to find clients in the network which have a similar concept. However,
the clustering is challenging because the clients cannot share their local
data due to privacy concerns, and can only exchange gradients (or model
parameters).

###  3.1 Hierarchical aggregation with similarity based tuning (HAST)

We propose a novel clustering algorithm that can adapt to different concepts
that emerge at different times. Our algorithm, called Hierarchical Aggregation
with Similarity based Tuning (HAST), extends the Decentralized Adaptive
Clustering (DAC) method (Listo Zec et al., 2023) by incorporating a
hierarchical structure and a tuning mechanism based on empirical training
loss. We demonstrate the effectiveness of our algorithm on two scenarios: one
with two concepts and one with four concepts. We show that our algorithm
outperforms DAC and other baselines in terms of accuracy and robustness under
concept shift.

HAST works as follows. Each client $k$ has a neural network consisting of a
feature extractor $\mathcal{F}^{k}_{\theta}$ and a classifier
$\mathcal{F}^{k}_{\phi}$ (the whole model being
$\mathcal{F}^{k}_{\phi}\circ\mathcal{F}^{k}_{\theta}$). The training procedure
in HAST consists of three steps, illustrated in figure 1. For each client $k$
and for each communication round:

  1. 1.

A subset of clients $\mathcal{R}$ is sampled uniformly and at random. Client
$k$ updates all layers $\mathcal{F}^{k}_{\phi}\circ\mathcal{F}^{k}_{\theta}$
as in FedAvg.

  2. 2.

A subset of clients $\mathcal{S}$ is randomly sampled based on their
similarity to client $k$. Client $k$ updates only the classifier layers
$\mathcal{F}^{k}_{\phi}$ using average aggregation.

  3. 3.

Client $k$ performs local training on its own data, updating the whole model
$\mathcal{F}^{k}_{\phi}\circ\mathcal{F}^{k}_{\theta}$. Afterwards, it fine-
tunes only the classifier $\mathcal{F}^{k}_{\phi}$.

The similarity function is the same as used by (Listo Zec et al., 2023), based
on the empirical training loss of client model $i$ on the data of client $j$:
$s_{ij}=1/\ell(w_{i};x_{j})$. This similarity score is transformed using a
softmax with a temperature scaling $\tau$ in order to get a probability vector
$\tilde{s}_{ij}$ for each client and each communication round.

###  3.2 Experimental setup

Our goal is to develop an algorithm that is robust to distributional shifts
(not decreasing significantly in accuracy between tasks). To demonstrate the
challenge of decentralized shift, we measure test accuracy over time and study
how robust different methods are to temporal shifts on two computer vision
datasets. Our code is found on Github. 111https://github.com/EmilieKar/HAST

To investigate the effects of covariate shift, we consider the PACS dataset
(Li et al., 2017) which is typically used for domain adaptation. It consists
of four domains with the same seven labels in each domain. We simulate
covariate shift by changing domains for a client $k$ over time $t$, i.e.
$\mathcal{D}^{t}_{k}(x)$ varies over time but keeping
$\mathcal{D}^{t}_{k}(y|x)$ fixed. To investigate the effects of label shift,
we consider the CIFAR-10 dataset (Krizhevsky et al., 2009). We simulate label
shift by varying $\mathcal{D}_{k}^{t}(y)$ over time but keeping
$\mathcal{D}^{t}_{k}(x|y)$ fixed, creating two clusters based on the labels:
one animal cluster (four labels) and one vehicle cluster (four labels).

Baselines. We compare our proposed method to two main baselines. The first we
refer to as Random, where all clients communicate randomly using a gossip
protocol. The second is DAC (Listo Zec et al., 2023), where clients
communicate using the similarity metric based on empirical training loss. For
a fair comparison with the baselines, we allow all methods to sample the same
number of clients per communication round. Since HAST performs two stages of
aggregation (random and similarity-based), it effectively samples $2n$
clients, where $n=3$ in this paper. Therefore, we allow the baselines to also
sample $2n$ clients for each round.

Hyperparameters. We performed a grid search over the hyperparameters of each
baseline and selected the ones that achieved the highest validation accuracy.
We also verified that the optimal learning rate was not at the boundary of the
searched grid.

Models. We use a small CNN model for each client in all experiments. The model
has two convolutional layers ($\mathcal{F}_{\theta}$), two fully-connected
layers and an output layer ($\mathcal{F}_{\phi}$). The model is not designed
to achieve state-of-the-art performance on the supervised tasks, but rather to
have enough capacity to solve the tasks while exploring decentralized temporal
shifts. The number of clients is set to 50 in the CIFAR-10 experiments, and 20
in the PACS experiments.

(a) $C=4$.

(b) $C=2$, animal/vehicles.

(c) $C=2$, random labels.

(d) $C=1$.

Figure 2: Test accuracy as a function of communication rounds for varying
datasets and number of clusters $C$. (a): PACS, (b-d): CIFAR-10.

##  4 Results and discussion

Firstly, we evaluate our method on the PACS dataset, which consists of four
domains: photo, art painting, cartoon, and sketch. We simulate a dynamic
environment where clients may experience domain shifts over time.
Specifically, at a certain communication round (marked by the vertical dashed
line in figure 2(a)), each client randomly switches to a different domain with
probability $\frac{3}{4}$. Figure 2(a) shows the mean test accuracy over
clients as a function of communication rounds. We can see that our method is
robust to domain shifts and outperforms the baselines. Interestingly, the DAC
baseline performs even worse than the Random baseline. We attribute this to
overfitting, as DAC merges all layers of the model with clients selected based
on the similarity score. In contrast, HAST first aggregates the model with
random clients and then only aggregates the classifier layer with similar
clients, resulting in a more generalizable model.

We conduct experiments to investigate the effect of the aggregation scheme in
HAST on the performance of personalized models. We vary the number of layers
that are aggregated in the second step of HAST using the similarity sampling
and compare it with DAC and Random. Figure 2(b) shows the results on the
CIFAR-10 dataset with $C=2$ clusters divided into animals and vehicles. The
depth parameter indicates how many layers are included in the aggregation. For
example, a depth of 4 means that all layers except the first one are
aggregated, while a depth of 1 means that only the output layer is aggregated.
We conclude that the optimal depth for HAST is 3, which is used in all other
experiments.

Figure 2(c) presents the results on CIFAR-10 with two clusters that have
random labels. This setting creates a smaller distribution gap between the two
clusters (compared to the setting with animals vs vehicles), which makes the
Random baseline more competitive. Nevertheless, HAST still surpasses both
baselines in this setting. We also note that unlike DAC, which suffers from
severe overfitting to the current domain and shows a large performance drop
under domain shifts, HAST maintains its robustness across domains.

We perform an ablation study to investigate the effect of the third step of
HAST, where we finetune the classifier layer. We compare HAST with DAC and
Random, both with and without finetuning, on the CIFAR-10 dataset with one
cluster. This means that the data is iid among clients. Figure 2(d) shows the
results of this experiment. We observe that finetuning improves the
performance of all methods and that HAST outperforms both baselines,
regardless of finetuning.

##  5 Conclusions

We have presented a novel aggregation method for decentralized deep learning
that can cope with temporal shifts in a peer-to-peer network. This is the
first work to address this problem and to demonstrate the benefits of
aggregating different layers of a neural network using FedAvg for robustness
and generalization under temporal shifts. Our proposed algorithm uses soft
clustering to group clients with similar concepts and allows them to update
their beliefs of potential collaborators over time. This enables clients to
smoothly transition between different collaboration groups as their concepts
change. Moreover, our algorithm employs a two-stage aggregation scheme that
makes the personalized models robust to concept changes by leveraging the
knowledge from other clients.

##  6 Related work

Gossip learning is a peer-to-peer communication protocol that has been studied
in various machine learning settings (Kempe et al., 2003; Boyd et al., 2006;
Ormándi et al., 2012) and applied in a decentralized deep learning setup to
learn personalized models (Blot et al., 2016). However, gossip learning is not
suitable for situations where client data is distributed non-iid, as it
assumes all clients share the same objective.

Previous work in federated learning has addressed this problem by introducing
multiple central models to which clients are assigned, and clustering clients
with similar objectives (Ghosh et al., 2020). However, this approach relies on
determining the appropriate number of central models on beforehand and assigns
clients to hard clusters, which may not capture the overlapping client
interests.

Listo Zec et al. (2023) proposed an algorithm that assigns clients to soft
clusters. These soft cluster assignments are continuously learned over
communication rounds based on client similarity, which is approximated by
training losses as by Onoszko et al. (2021). This work tackles many challenges
in decentralized learning where data is non-iid distributed among clients,
both in terms of cluster sizes and different types of distributional shifts.
However, they do not consider time-dependent distributional shifts.

Jothimurugesan et al. (2023) addresses federated learning with distributed
concept shift, where clients have different and dynamic data distributions.
They present FedDrift, which learns multiple global models for different
concepts. They are the first to tackle FL with concept shift, challenging the
single-model paradigm. The work treats shift adaptation as a time-varying
clustering problem, and uses hierarchical clustering to handle an unknown
number of concepts.

Similarly to Jothimurugesan et al. (2023), we also view concept shift
adaptation as a time-varying clustering problem. However, our solution differs
in several aspects. First, we consider a decentralized learning framework,
where there is no central server. Second, we build on previous work on round-
based client similarity approximation and soft-cluster assignment, enabling
clients to aggregate their models with similar peers. Third, we introduce a
novel hierarchical model aggregation step, which ensures that every client in
the network can perform well on their own current distribution and also adapt
quickly to new concepts.

## References

  * Blot et al. (2016) Blot, M., Picard, D., Cord, M., and Thome, N.  Gossip training for deep learning, 2016.  URL https://arxiv.org/abs/1611.09726.
  * Boyd et al. (2006) Boyd, S., Ghosh, A., Prabhakar, B., and Shah, D.  Randomized gossip algorithms.  _IEEE Transactions on Information Theory_ , 52(6):2508–2530, 2006.  doi: 10.1109/TIT.2006.874516.
  * Dayan et al. (2021) Dayan, I., Roth, H. R., Zhong, A., Harouni, A., Gentili, A., Abidin, A. Z., Liu, A., Costa, A. B., Wood, B. J., Tsai, C.-S., et al.  Federated learning for predicting clinical outcomes in patients with covid-19.  _Nature medicine_ , 27(10):1735–1743, 2021.
  * Gama et al. (2014) Gama, J., Žliobaitė, I., Bifet, A., Pechenizkiy, M., and Bouchachia, A.  A survey on concept drift adaptation.  _ACM computing surveys (CSUR)_ , 46(4):1–37, 2014\.
  * Ghosh et al. (2020) Ghosh, A., Chung, J., Yin, D., and Ramchandran, K.  An efficient framework for clustered federated learning.  _Advances in Neural Information Processing Systems_ , 33:19586–19597, 2020.
  * Jothimurugesan et al. (2023) Jothimurugesan, E., Hsieh, K., Wang, J., Joshi, G., and Gibbons, P. B.  Federated learning under distributed concept drift.  In _International Conference on Artificial Intelligence and Statistics_ , pp. 5834–5853. PMLR, 2023.
  * Kairouz et al. (2021) Kairouz, P., McMahan, H. B., Avent, B., Bellet, A., Bennis, M., Bhagoji, A. N., Bonawitz, K., Charles, Z., Cormode, G., Cummings, R., et al.  Advances and open problems in federated learning.  _Foundations and Trends® in Machine Learning_ , 14(1–2):1–210, 2021.
  * Kempe et al. (2003) Kempe, D., Dobra, A., and Gehrke, J.  Gossip-based computation of aggregate information.  In _44th Annual IEEE Symposium on Foundations of Computer Science, 2003. Proceedings._ , pp. 482–491. IEEE, 2003.
  * Krizhevsky et al. (2009) Krizhevsky, A., Hinton, G., et al.  Learning multiple layers of features from tiny images.  2009\.
  * Li et al. (2017) Li, D., Yang, Y., Song, Y.-Z., and Hospedales, T. M.  Deeper, broader and artier domain generalization.  In _Proceedings of the IEEE international conference on computer vision_ , pp. 5542–5550, 2017.
  * Li et al. (2022) Li, Z., Lu, J., Luo, S., Zhu, D., Shao, Y., Li, Y., Zhang, Z., and Wu, C.  Mining latent relationships among clients: Peer-to-peer federated learning with adaptive neighbor matching.  _arXiv preprint arXiv:2203.12285_ , 2022.
  * Listo Zec et al. (2023) Listo Zec, E., Ekblom, E., Willbo, M., Mogren, O., and Girdzijauskas, S.  Decentralized adaptive clustering of deep nets is beneficial for client collaboration.  In Goebel, R., Yu, H., Faltings, B., Fan, L., and Xiong, Z. (eds.), _Trustworthy Federated Learning_ , pp. 59–71, Cham, 2023. Springer International Publishing.  ISBN 978-3-031-28996-5.
  * McMahan & Thakurta (2022) McMahan, B. and Thakurta, A.  Federated learning with formal differential privacy guarantees.  _Google AI Blog_ , 2022.
  * McMahan et al. (2017) McMahan, B., Moore, E., Ramage, D., Hampson, S., and y Arcas, B. A.  Communication-efficient learning of deep networks from decentralized data.  In _Artificial intelligence and statistics_ , pp. 1273–1282. PMLR, 2017.
  * Onoszko et al. (2021) Onoszko, N., Karlsson, G., Mogren, O., and Listo Zec, E.  Decentralized federated learning of deep neural networks on non-iid data.  _ICML Workshop on Federated Learning for User Privacy and Data Confidentiality_ , 2021.
  * Ormándi et al. (2012) Ormándi, R., Hegedűs, I., and Jelasity, M.  Gossip learning with linear models on fully distributed data.  _Concurrency and Computation: Practice and Experience_ , 25(4):556–571, may 2012.  doi: 10.1002/cpe.2858.  URL https://doi.org/10.1002%2Fcpe.2858.
  * Voigt & Von dem Bussche (2017) Voigt, P. and Von dem Bussche, A.  The eu general data protection regulation (gdpr).  _A Practical Guide, 1st Ed., Cham: Springer International Publishing_ , 10(3152676):10–5555, 2017.
  * Yang et al. (2019) Yang, Q., Liu, Y., Chen, T., and Tong, Y.  Federated machine learning: Concept and applications.  _ACM Transactions on Intelligent Systems and Technology (TIST)_ , 10(2):1–19, 2019.

##  Appendix A Appendix

###  A.1 Concept shift experiment

(a) $C=2$, FedDrift pattern.

(b) $C=4$, FedDrift pattern.

(c) $C=4$, Random pattern.

Figure 3: Test accuracy as a function of communication rounds for varying
number of clusters and client concept shift patterns. CIFAR-10.

In addition to the experiments shown in the main paper we also evaluated our
algorithm on the concept shift label swap, that is shifts where
$p^{t}_{k}(y|x)\neq p^{t-1}_{k}(y|x)$. Not to be confused with the more
general term we defined in the paper meaning simply that $p^{t}_{k}\neq
p^{t-1}_{k}$.

Experimental setup. We grouped the clients into either two or four clusters.
In Figures 3(a) and 3(b), we used 20 clients in total, while in Figure 3(c),
we used 50 clients. The label distribution across the clusters was consistent
in all experiments, with eight distinct labels. The label distribution across
the clusters remained consistent in all experiments, consisting of eight
distinct labels. However, in the two-cluster experiment, a label swap was
simulated by swapping two labels within one of the clusters. Similarly, in the
four-cluster experiments, the label swaps within clusters were conducted with
care to ensure non-overlapping exchanges of labels between the clusters. Note
that Figures 3(a) and 3(b) followed a fixed client shift pattern as described
in previous work (Jothimurugesan et al., 2023), specifically Figures 2 and 3
in their paper. On the other hand, Figure 3(c) randomly assigned clients to
clusters, similar to our main paper.

Result. Figure 3 shows the results of the three label swap experiments, where
HAST outperforms all the baselines. In Figure 3(a), we see that HAST achieves
the best performance in the two-cluster experiment, regardless of the client
shift phase. HAST performs better than other methods throughout the
experiment. It does well in the initial phase when all clients share the same
cluster, in the middle phase when clients start to move to the second cluster,
and in the final phase when all clients are in the second cluster only.
Similarly, in Figure 3(b), HAST dominates the four-cluster experiment
following the FedDrift pattern. Finally, in Figure 3(c), HAST shows a
significant advantage over the baselines in the random cluster assignment
experiment.


(eccv) Package eccv Warning: Package ‘hyperref’ is loaded with option
‘pagebackref’, which is *not* recommended for camera-ready version

11institutetext: MIT CSAIL

# TetSphere Splatting: Representing High-Quality Geometry with Lagrangian
Volumetric Meshes

Minghao Guo Both authors contributed equally to this research. Bohan Wang⋆
Kaiming He  Wojciech Matusik

###### Abstract

We present TetSphere splatting, an explicit, Lagrangian representation for
reconstructing 3D shapes with high-quality geometry. In contrast to
conventional object reconstruction methods which predominantly use Eulerian
representations, including both neural implicit (e.g., NeRF, NeuS) and
explicit representations (e.g., DMTet), and often struggle with high
computational demands and suboptimal mesh quality, TetSphere splatting
utilizes an underused but highly effective geometric primitive—tetrahedral
meshes. This approach directly yields superior mesh quality without relying on
neural networks or post-processing. It deforms multiple initial tetrahedral
spheres to accurately reconstruct the 3D shape through a combination of
differentiable rendering and geometric energy optimization, resulting in
significant computational efficiency. Serving as a robust and versatile
geometry representation, TetSphere splatting seamlessly integrates into
diverse applications, including single-view 3D reconstruction, image-/text-
to-3D content generation. Experimental results demonstrate that TetSphere
splatting outperforms existing representations, delivering faster optimization
speed, enhanced mesh quality, and reliable preservation of thin structures.

##  1 Introduction

Reconstructing 3D geometry stands as a fundamental task in computer vision and
graphics. The field has seen significant strides with the advent of diffusion
models, showing exceptional capabilities in generating images. This success
has spurred research into employing 2D generative methods for 3D
reconstruction [67, 46, 94, 10, 80, 50, 53, 49, 56, 68, 47].

These advances prominently featured the use of Eulerian representations, where
the geometry is defined on fixed coordinates in the 3D world space. One
prevalent category includes _neural implicit_ representations, such as Neural
Radiance Fields (NeRF) [58], which uses a density function, and Neural
Implicit Surfaces (NeuS) [91], which employs a signed distance function. Both
methods are parameterized by neural networks and require an additional mesher
step to obtain geometric shapes. These representations take spatial
coordinates of the world space as input, inherently classified as Eulerian. In
parallel, _explicit_ representation methods such as Deep Marching Tetrahedra
(DMTet) [77] and its variants [25, 52] use an explicit deformable tetrahedral
grid, with signed distance values defined at the grid vertices – an
alternative take on Eulerian representations which facilitates the integration
of explicit shading materials. Optimizing Eulerian models, including both
implicit and explicit ones, however, is both time- and memory-consuming. This
challenge is primarily due to the necessity of employing high-capacity neural
networks or high-resolution grids, which are essential for capturing intricate
geometric details and inherently contain numerous parameters. Moreover,
Eulerian representations are vulnerable in modeling thin structures, often
leading to floating artifacts. Extracting surfaces from these models is also
required for subsequent applications like rendering and simulation, adding
another layer of complexity.

Figure 1: (a) Eulerian vs. Lagrangian geometry representations: DMTet employs
a deformable tetrahedral grid, assigning signed distance values at vertices,
whereas TetSphere splatting reconstructs 3D shapes by directly deforming tet-
spheres, enhancing computational and memory efficiency. TetSphere splatting
supports a range of applications, including (b) single-view 3D reconstruction
from Google Scanned Objects dataset [19], (c) SDS-based image-to-3D
generation, and (d) text-to-3D generation.

To address these challenges, we propose TetSphere splatting, an _explicit,
Lagrangian_ geometry representation that efficiently constructs high-quality
meshes. Lagrangian representations track the deformation of geometry
primitives through space, offering a more efficient approach than the
typically more resource-intensive Eulerian representations. An illustrative
comparison between Eulerian and Lagrangian geometry representation is shown in
Fig. 1 (a). Our TetSphere splatting leverages tetrahedral meshes – a
volumetric, Lagrangian representation that remains relatively underexplored in
the realm of 3D reconstruction. The 3D object is represented by the union of a
set of deformed tetrahedral spheres, termed _TetSphere_. Compared to Gaussian
point cloud, TetSphere imposes structured constraints between points owing to
tetrahedralization and also offers a clear distinction between the interior
and boundary of the object. Compared to other Lagrangian representations,
particularly surface mesh-based methods [1, 65] which struggle with
regularization issues leading to self-intersections and compromised mesh
quality, TetSphere exhibits greater stability due to its volumetric nature. It
also effectively handles complex topologies without re-meshing, an often-
necessary requirement in surface mesh representations during optimization
[65]. Overall, TetSphere offers the following advantages: 1) its explicit
representation ensures fast optimization (in a matter of minutes) and reduced
memory cost compared to Eulerian representations; 2) its Lagrangian nature
shows robustness to thin structures and reducing floating artifacts; 3) it
enables the integration of volumetric regularization, significantly improving
mesh quality; and 4) it can handle shapes with arbitrary topologies.

Figure 2: Comparisons of mesh quality across NeRF, NeuS, GS, DMTet, and
TetSphere splatting, highlighting a critical yet overlooked criterion of 3D
reconstruction. Existing methods are not inherently mesh-based but are derived
through post-processing like Marching Cubes. On the contrary, our method
demonstrates superior mesh quality without remeshing. Results are obtained
from a single image of a dog in Fig. 3.

We further present a computational framework for TetSphere splatting. Similar
to Gaussian splatting [41], our method “splats” TetSpheres to conform to the
target object. We formulate the deformation of TetSpheres as a geometric
energy optimization, incorporating differentiable rendering loss, bi-harmonic
energy of the deformation gradient field, and non-inversion constraints, to
ensure robust and accurate reconstruction. Our framework is versatile,
enabling seamless integration with applications including single-view object
reconstruction, image-to-3D generation, and text-to-3D generation. Notably,
TetSphere splatting’s fast optimization and reduced computational cost offer
significant benefits for the latter two tasks that typically require intensive
time and memory.

In our evaluation, we underscore a frequently overlooked aspect of 3D
reconstruction: the quality of the reconstructed mesh. The concept of mesh
quality encompasses multiple attributes key to the usability of 3D models,
such as the uniformity of surface triangles, the absence of undesired bumps,
and manifoldness. Despite its paramount importance, particularly in rendering
and simulation applications, the evaluation of mesh quality has not received
adequate focus in the current 3D reconstruction research. Recognizing this
oversight, we introduce three evaluation metrics to assess mesh quality. We
conduct a comprehensive evaluation using the Google Scanned Objects (GSO)
dataset [19]. Compared to state-of-the-art methods, our TetSphere splatting
exhibits superior performance when measured under these newly proposed
criteria. It maintains competitive performance on other commonly employed
metrics as well. Furthermore, we illustrate its utility in 3D content
generation, where it produces qualitatively superior results compared to
existing state-of-the-art methods.

##  2 Related Work

Eulerian and Lagrangian geometry representations. The differentiation between
Eulerian and Lagrangian representations originates from computational fluid
dynamics [14] but extends more broadly into computational geometry and
physics. Eulerian representations characterize the geometry within a fixed
spatial reference, usually utilizing grids or voxels to define the space. In
contrast, Lagrangian methods track the movement of individual particles or
elements, providing a dynamic representation of geometry that adapts to
changes over time. Using fluid simulation as an analogy, an Eulerian view
would analyze fluid presence at fixed points in space, whereas a Lagrangian
perspective follows specific fluid particles. Neural implicit representations,
such as DeepSDF [66], NeRF [59], and InstantNGP [62], are modern adaptations
of Eulerian concepts, processing 3D positions as inputs to neural networks.
These methods theoretically allow for infinite resolution through NN-based
parameterization but can result in slow optimization speeds due to NN
training. Beyond implicit ones, explicit or hybrid Eulerian representations,
such as DMTet [77], DefTet [24], and TetGAN [102], incorporate explicit
irregular grids but can still cause substantial memory usage for high-
resolution shapes. Gaussian splatting [82] exemplifies a Lagrangian approach
by moving Gaussian point clouds in space. Our proposed TetSphere is an
explicit Lagrangian representation, which can be viewed as introducing
explicit constraints among points due to tetrahedral meshing, with enhanced
efficiency and reconstruction quality.

Single image-to-3D object reconstruction. Single-image 3D reconstruction is an
inherently ill-posed problem, and extensive research has been dedicated to
addressing it [23, 20]. Early approaches utilized a combination of 2D image
encoders and 3D decoders trained on 3D data with both explicit
representations, including voxels [13, 88, 12, 97, 98], meshes [90, 26], and
point clouds [55, 22, 21, 29], and implicit representations such as NeRF [103,
38, 61], SDF[66, 60, 100], and occupancy networks[57, 5]. Many of these
methods were trained on categorized 3D datasets such as 3D templates [72, 27,
40] and semantics [45], yet faced challenges in generalizing to unseen
categories. Recently, an active research direction has been leveraging 2D
generative models for 3D reconstruction, including the use of SDS and
supplementary losses [46, 80, 49, 56, 76, 99, 30, 17]. Alternative approaches
train view-conditioned 2D diffusion models to directly produce multi-view
images for 3D reconstruction [49, 48, 50, 9, 86, 85, 53]. Our method adopts a
similar strategy but introduces an explicit and Lagrangian geometry
representation to overcome the limitation of prior representations. The recent
introduction of large-scale 3D dataset propelled feed-forward large
reconstruction models [33, 92, 101, 96, 81, 75, 15, 16, 8, 41], which directly
reconstruct triplane-based NeRF [8] or 3D Gaussians [41]. Their feed-forward
inference accelerates the speed of 3D object reconstruction, but often at a
sacrifice of relatively low resolution and geometry quality, as well as,
coupled geometry and materials.

Our method also relates to text-to-3D content generation as it is one of the
applications of TetSphere splatting. We leave a detailed discussion on these
related works to Appendix 0.F.

#####

Figure 3: Overall pipeline: TetSphere splatting selects 3D feature points to
initialize a set of tetrahedral spheres. The geometry optimization stage
deforms these spheres to reconstruct the object. Texture optimization is then
applied to obtain the texture or physical material of the object.

##  3 Overview

We use a set of TetSpheres to represent 3D shapes. Our TetSphere splatting
takes multi-view images as inputs. We initiate it using a set of tetrahedral
spheres with varying centers and sizes (radii). These spheres are chosen to
ensure that, collectively, the initial spheres approximately cover the target
shape under different input views. Subsequently, these spheres undergo a two-
stage optimization process to precisely reconstruct the 3D object. The first
stage, geometry optimization, deforms the tetrahedral spheres through the
minimization of rendering loss and two geometric regularization energies. The
second stage optimizes the surface texture or Physically Based Rendering (PBR)
materials of the tetrahedral spheres. Both stages leverage differentiable
rasterizers. The rendering loss typically incorporates supervision from ground
truth images or SDS. An illustration of the overall pipeline is shown in Fig.
3.

##  4 TetSphere Splatting

Our TetSphere splatting is an explicit, Lagrangian representation for 3D
shapes. It positions itself alongside Gaussian splatting [41], which is also a
neural-network free Lagrangian representation by representing shapes using
Gaussian point clouds as primitives. Although the use of point clouds is
notably general, it lacks local constraints within individual Gaussian
kernels, resulting in an undefined boundary between the interior and exterior
of the object. Converting these Gaussian kernels into explicit meshes
necessitates surface reconstruction, which often leads to uneven, bumpy
surfaces due to the inherent limitation in defining surface from point clouds.

Figure 4: TetSphere splatting with deforming tetrahedral spheres. Color-coded
regions represent the bi-harmonic energy values (red: high, blue: low) across
tetrahedra, one of the geometric regularizations employed in our deformation
optimization process.

We employ tetrahedral spheres as our primitive of choice. Unlike point clouds,
tetrahedral meshes enforce structured local connectivity between points owing
to tetrahedralization. This preserves the geometric integrity of the 3D shape
and also enhances the surface quality by imposing geometric regularization
across the entire mesh interior. We formulate the reconstruction of shapes
through TetSphere splatting as a deformation of tetrahedron spheres. Starting
from a set of tetrahedral spheres, we adjust the positions of their vertices
to align the rendered images of these meshes with the corresponding target
multi-view images. Vertex movement is constrained by two geometric
regularizations on the tetrahedral meshes, derived from the field of geometry
processing [4]. These constraints, which penalize the nonsmooth deformation
(via bi-harmonic energy) and prevent the inversion of mesh elements (via local
injectivity), have proven effective in ensuring that the resulting tetrahedral
meshes are of superior quality and maintain structural integrity. Fig. 4
illustrates the iterative process of TetSphere splatting with deforming
tetrahedral spheres.

###  4.1 Tetrahedral Sphere Primitive

The primitive of TetSphere splatting is a tetrahedralized sphere, called
TetSphere, with $N$ vertices and $T$ tetrahedra. By applying principles from
the Finite Element Method (FEM) [79], the mesh of each sphere is composed of
tetrahedral elements, with each tetrahedron constituting a 3D discrete
piecewise linear volumetric entity. We denote the position vector of all
vertices of the $i$-th deformed sphere mesh as ${x}_{i}\in\mathbb{R}^{3N}$.
The deformation gradient of the $j$-th tetrahedron in the $i$-th sphere is
denoted as $\mathbf{F}^{(i,j)}_{\mathbf{x}}\in\mathbb{R}^{3\times 3}$, which
quantitatively describes how each tetrahedron’s shape transforms [79].
Essentially, the deformation gradient $\mathbf{F}^{(i,j)}_{\mathbf{x}}$ serves
as a measure of the spatial changes a tetrahedron undergoes from its original
configuration to its deformed state. Refer to the inset figure for a visual
explanation and Appendix 0.D for an in-depth derivation.

Rather than using a single sphere, our representation employs a collection of
spheres to accurately represent arbitrary shapes. Consequently, the complete
shape is the union of all spheres. By adopting multiple spheres, this approach
ensures that each local region of a shape is detailed independently, enabling
a highly accurate representation. Moreover, it allows for the representation
of shapes with arbitrary topologies. Such a claim is theoretically guaranteed
by the paracompactness property of manifold shapes [37].

Using tetrahedral spheres offers several technical benefits compared with
prevalent representations for object reconstruction, as demonstrated in Fig.
2:

  * •

Compared to neural representations (e.g., NeRF, NeuS), our tetrahedral
representation does not rely on neural networks, thus inherently accelerating
the optimization process.

  * •

Compared to Eulerian representations (such as DMTet), our approach entirely
avoids the need for iso-surface extraction – an operation that often degrades
mesh quality owing to the predetermined resolution of the grid space.
Furthermore, Eulerian methods rely on level-set functions, a dependency that
can lead to undesirable floating and noisy artifacts given sparsely populated
or unpopulated grid values during the training phase. The deformation process
of our representation inherently prevents individual spheres from breaking,
thereby eliminating floating noises.

  * •

Compared to other Lagrangian representations, such as triangle meshes and
Gaussian point clouds, our method offers a volumetric representation through
the use of tetrahedral meshes. It is more robust to thin shapes, which often
pose a risk of self-penetration in surface meshes, and to fragile structures,
which are a challenge for point clouds. Furthermore, unlike Gaussian point
clouds, which require additional steps to produce a mesh, our tetrahedral
representation naturally forms a mesh. Each tetrahedron also imposes
constraints among vertices, leading to superior mesh quality.

###  4.2 TetSphere Splatting as Shape Deformation

To reconstruct the geometry of the target object, we deform the initial
TetSpheres by changing their vertex positions. This process is governed by two
primary goals: ensuring the deformed TetSpheres align with the input multi-
view images and maintaining high mesh quality that adheres to necessary
geometry constraints.

To maintain the mesh quality, we leverage bi-harmonic energy – defined in the
literature on geometry processing [6] as an energy quantifying smoothness
throughout a field – to the deformation gradient field. This geometric
regularization ensures the smoothness of the deformation gradient field across
the deformation process, thus preventing irregular mesh or bumpy surfaces. It
is important to highlight that this bi-harmonic regularization does _not_ lead
to over-smoothness of the final result. This is because the energy targets the
deformation gradient field, which measures the _relative_ changes in vertex
positions, rather than the _absolute_ positions themselves. Such an approach
allows for the preservation of sharp local geometric details, akin to
techniques used in physical simulations [95]. Furthermore, we introduce a
geometric constraint to guarantee local injectivity in all deformed elements
[74]. This ensures that the elements maintain their orientation during the
deformation, avoiding inversions or inside-out configurations. This constraint
can be mathematically expressed as
$\mathrm{det}(\mathbf{F}^{(i,j)}_{\mathbf{x}})>0.$ Importantly, these two
terms – bi-harmonic energy for smoothness and local injectivity for element
orientation – are universally applicable to any tetrahedral meshes, stemming
from their fundamental basis in geometry processing [4].

Let $\mathbf{x}=[x_{1},...,x_{M}]\in\mathbb{R}^{3NM}$ denote the positions of
vertices across all $M$ TetSpheres, and
$\mathbf{F}_{\mathbf{x}}\in\mathbb{R}^{9MT}=[\mathrm{vec}(\mathbf{F}^{(1,1)}_{\mathbf{x}}),...,\mathrm{vec}(\mathbf{F}^{(M,T)}_{\mathbf{x}})]$
denote the flattened deformation gradient fields of all TetSpheres. In the bi-
harmonic energy, the Laplacian matrix is defined based on the connectivity of
the tetrahedron faces, denoted as $\mathbf{L}\in\mathbb{R}^{9MT\times 9MT}$.
This matrix is block symmetric, where each block
$\mathbf{L}_{pq}\in\mathbb{R}^{9\times 9},p\neq q$ is set to a negative
identity matrix $-I$ if the $p$-th and $q$-th tetrahedron shares a common
triangle; or $kI$ for $\mathbf{L}_{pp}$, where $k$ is the number of neighbors
of the $p$-th tetrahedron. The deformation of the TetSpheres is formulated as
an optimization problem:

$\displaystyle\min_{\textbf{x}}\ \ $
$\displaystyle\mathbf{\Phi}(R(\mathbf{x}))+||\mathbf{L}\mathbf{F}_{\mathbf{x}}||_{2}^{2}$
$\displaystyle\mathrm{s.t.}\ \ $
$\displaystyle\mathrm{det}(\mathbf{F}^{(i,j)}_{\mathbf{x}})>0,\ \forall
i\in\\{1,...,M\\},\ j\in\\{1,...,T\\},$ (1)

where $R(\cdot)$ is the rendering function, $\mathbf{\Phi}(\cdot)$ is the
rendering loss matching the union of deformed tetrahedral spheres with the
input images. The second term regulates the bi-harmonic energy across the
deformation gradient field. The non-inversion constraint ensures that
tetrahedra maintain their orientation. To manage this constrained
optimization, we reformulate it by incorporating the non-inversion hard
constraint as a soft penalty term into the objective,

$\displaystyle\min_{\textbf{x}}\ \ $
$\displaystyle\mathbf{\Phi}(R(\mathbf{x}))+w_{1}||\mathbf{L}\mathbf{F}_{\mathbf{x}}||_{2}^{2}+w_{2}\sum_{i,j}(\mathrm{min}\\{0,\mathrm{det}(\mathbf{F}^{(i,j)}_{\mathbf{x}})\\})^{2},$
(2)

allowing for optimization via standard gradient descent solvers.

In the proposed optimization framework, three considerations have been
outlined. 1) The adaptive loss function $\mathbf{\Phi}(\cdot)$, designed to be
flexible, supports a variety of metrics, including $l_{1}$ for color images,
MSE for depth images, cosine embedding loss for normal images, or SDS loss. 2)
Given that the tetrahedron is a linear element, the deformation gradient
$\mathbf{F}^{(i,j)}_{\mathbf{x}}$ is a linear function of $\mathbf{x}$, making
the bi-harmonic energy a quadratic term. 3) The weights $w_{1}$ and $w_{2}$
are dynamically adjusted using a cosine scheduler. We provide details of the
scheduler’s hyperparameters in Appendix 0.A.

##  5 Optimization Framework

The overall framework of optimization is illustrated in Fig. 3. The geometry
optimization follows the shape deformation as detailed in Sec. 4. In this
section, we introduce the remaining two components: initialization of
TetSphere splatting and texture optimization.

###  5.1 TetSphere Initialization

Given multi-view images as inputs, we select feature points to initialize the
3D center positions of the TetSpheres. We aim to achieve a uniform
distribution of these TetSpheres within the object, ensuring comprehensive
coverage of the silhouette depicted in the multi-view images.

Figure 5: Feature point selection via silhouette coverage algorithm. Our
method automates the feature point selection from a sparse voxel constructed
from multi-view images. By solving a linear programming problem, we determine
these feature points and initialize tetrahedral spheres for subsequent
geometry optimization.

We introduce an algorithm, silhouette coverage, inspired by Coverage Axis [18]
to automatically select initial centers of TetSpheres for an arbitrary shape.
This process begins with constructing a coarse voxel grid and initially
assigning a zero value to each voxel. By projecting these voxels into the
image spaces using the same camera poses as the input multi-view images,
voxels within the foreground of all images are marked with a value of $1$.
These voxel positions are identified as candidate positions of TetSphere
centers. From these marked positions, the objective is to pick a minimal
subset of candidates that ensure thatall candidates are fully encapsulated by
TetSpheres centered on these points. This involves placing uniform spheres
with varying radius values at all candidate points and choosing a minimal
subset that collectively covers all the candidate points. We formulate a
linear programming problem to efficiently perform the selection. The detailed
formulation is provided in Appendix 0.E. Fig. 5 shows the pipeline of the
silhouette coverage algorithm. In our implementation, with a voxel grid
resolution $300\times 300$ and $n=20$, the whole TetSphere initialization
completes in ${\sim}1$ minute on average.

###  5.2 Texture/PBR Material Optimization

The final stage of our reconstruction process focuses on optimizing the
textures or material properties. Our method, with its explicit representation,
allows textures and materials to be directly applied to the surface vertices
and faces of the TetSpheres. This enables the use of sophisticated material
models, such as Disney’s principled BRDF [7], with physically-based rendering.

Material optimization is facilitated through the use of differentiable
rasterizers [43], which adjust the textures or materials to closely match the
input multi-view color images. A significant advantage of TetSphere splatting
is that the deformation of tetrahedral spheres does not alter the surface
topology. Unlike methods such as DMTet, which require isosurface extraction
and subsequent texture parameterization at each step due to potential changes
in the underlying shape, our method necessitates only a single texture
parameterization at the beginning of optimization. This parameterization
remains consistent throughout the process, significantly enhancing the
efficiency of texture optimization.

For scenarios with dense input views, we have found that using textured images
as optimization variables is straightforward and yields high-quality results.
In cases with sparse input views, we adopt a two-layer multilayer perceptron
(MLP) that takes the surface vertex positions as inputs and outputs the
material parameters, a practice in line with existing methods [80, 69].

Table 1: Single-View reconstruction results on the GSO Dataset: Evaluating
reconstruction accuracy with Chamfer Distance (Cham.) and Volume IoU,
alongside new mesh quality metrics: Area-Length Ratio (ALR), Manifoldness Rate
(MR), and Connected Component Discrepancy (CC Diff.). TetSphere splatting
demonstrates superior performance on these criteria and maintains competitive
reconstruction accuracy.

Method | Mesher |  Cham.$\downarrow$ |  Vol. IoU$\uparrow$ |  ALR$\uparrow$ |  MR(%)$\uparrow$ |  CC Diff.$\downarrow$
---|---|---|---|---|---|---
Realfusion [56] | Marching Cubes | 0.0819 | 0.2741 | 0.0561 | 100% | 47.7
Magic123 [68] | Marching Tets | 0.0516 | 0.4528 | 0.0383 | 100% | 13.7
One-2-3-45 [47] | Marching Cubes | 0.0629 | 0.4086 | 0.0574 | 96% | 0.83
Point-E [64] | Marching Cubes | 0.0426 | 0.2875 | 0.2421 | 100% | 18.38
Shap-E [39] | Marching Tets | 0.0436 | 0.3584 | 0.1236 | 100% | 9.03
Zero123 [49] | Marching Cubes | 0.0339 | 0.5035 | 0.0543 | 100% | 0.18
SyncDreamer [51] | Marching Cubes | 0.0261 | 0.5421 | 0.0201 | 10% | 0.3
Wonder3d [53] | Marching Cubes | 0.0329 | 0.5768 | 0.0281 | 100% | 0.0
Open-LRM [31] | Marching Cubes | 0.0285 | 0.5945 | 0.0252 | 100% | 0.0
DreamGaussian [83] | Marching Cubes | 0.0641 | 0.3476 | 0.0812 | 100% | 237.4
Ours | N/A | 0.0351 | 0.6317 | 0.3665 | 100% | 0.0

Figure 6: Qualitative comparison on image-to-3D generation with surface mesh
visualizations and rendered normal maps (1/2). Our generated meshes exhibit
minimal noise and high quality, featuring more regular triangle meshing.

Figure 7: Qualitative comparison on image-to-3D generation with surface mesh
visualizations and rendered normal maps (2/2). Our generated meshes exhibit
minimal noise and high quality, featuring more regular triangle meshing.

##  6 Experiments and Results

We conduct experiments on three applications to demonstrate TetSphere’s
versatility and effectiveness: single-view reconstruction, image-to-3D shape
generation, and text-to-3D shape generation. We choose single-view
reconstruction because it allows for comprehensive evaluation using the well-
established benchmark on the Google Scanned Objects (GSO) dataset. We choose
the latter two applications because the existing methods are known for their
high computational costs, specifically in terms of GPU quantity and memory
capacity. We conduct experiments on these applications to demonstrate that
TetSphere splatting effectively addresses and mitigates these issues.

For the first two applications, the input is a single image from which a
textured 3D shape is reconstructed. For text-to-3D generation, the input is a
text prompt. All the output targets are textured 3D shapes. For single-view
reconstruction, we conduct a quantitative evaluation of our method on GSO
dataset, comparing its performance with other state-of-the-art methods. The
evaluation focuses on both the reconstruction accuracy and the geometry
quality. In addition, we show a series of qualitative comparisons using web-
collected images or text prompts. Appendix 0.A provides the implementation
details.

###  6.1 Baselines and Evaluation Protocol

####  6.1.1 Baselines.

For single-view reconstruction, we quantitatively compare with several state-
of-the-art methods: RealFusion [56], Magic123 [68], One-2-3-45 [47], Point-E
[64], Shap-E [39], Zero123 [49], SyncDreamer [50], Wonder3d [53], Open-LRM
[33, 31], and DreamGaussian [83]. For image-to-3D generation, we compare
TetSphere splatting with Magic123 and Dreamcraft3D [80]. Both are multi-stage
methods starting with NeRF optimization followed by DMTet for optimizing mesh
and texture. For text-to-3D, our comparison focuses on RichDreamer, a state-
of-the-art method known for incorporating PBR materials and generating shapes,
showcasing notable results in 3D generation from text prompts.

####  6.1.2 Evaluation Datasets.

For single-view reconstruction, following prior research [51, 53], we use the
GSO dataset for our evaluation, which covers a broad range of everyday
objects. The evaluation dataset aligns with those used by SyncDreamer and
Wonder3D, featuring 30 diverse objects ranging from common household items to
animals. For image-to-3D generation, we include a variety of internet-
collected images in different styles in our evaluation. For text-to-3D
generation, we employ text prompts from the official implementation of
RichDreamer.

####  6.1.3 Metrics.

To assess the accuracy of single-view reconstruction, we use two commonly used
metrics: Chamfer Distance (Cham.) and Volume IoU, comparing the ground-truth
shapes with the reconstructed ones. Following established practices, we use
the rigid Iterative Closest Point (ICP) [3] algorithm to align the generated
shapes with their ground-truth counterparts before metric calculation.

While Cham. and Volume IoU effectively gauge volumetric and point-based shape
conformity, they fall short of evaluating mesh quality. Recognizing this gap,
we introduce three additional metrics to comprehensively assess the geometry
quality of the generated shapes: 1) Area-length Ratio (ALR): This metric
computes the average ratio of a triangle’s area to its perimeter (scaled by a
constant coefficient) within the surface mesh. Values range from $0$ to $1$,
where meshes with higher ALR values contain mostly equilateral triangles,
thereby indicating superior triangle quality; 2) Manifoldness Rate (MR):
Manifoldness verifies whether a mesh qualifies as a closed manifold. Non-
manifold meshes can manifest anomalies, such as edges shared by more than two
faces, vertices connected by edges but not by a surface, isolated vertices and
edges, or open boundaries, which can cause significant problems in downstream
applications such as simulation and rendering. We report the percentage of
manifold shapes within the evaluation dataset as MR; and 3) Connected
Component Discrepancy (CC Diff.) from the ground-truth shape: This measure
identifies the presence of floating artifacts or structural discontinuities
within the mesh, highlighting the integrity and cohesion of the reconstructed
shape.

Table 2: Comparison of memory cost and run-time speed on image-to-3D
generation with SDS loss. We report the maximal batch size of $256\times 256$
images that can occupy a 40GB A100 and the run-time speed for training with
batch size $4$.

Method |  Maximal |
---|---|---
|  Speed$\uparrow$ | Batch Size$\uparrow$ |
| (#iter./s) |  |
NeRF | Make-it-3D[84] | 4 | 1.22
Magic123[68] | 4 | 1.03
NeuS | SyncDreamer[50] | 48 | 1.8
DMTet | DreamCraft3D[80] | 8 | 1.23
GeoDream[54] | 8 | 1.30
3D GS | DreamGaussian[83] | 80 | 4.43
Ours | 120 | 6.59

Figure 8: Analysis on energy coefficients for geometry optimization. Dark
regions indicate flipping of the surface triangles.

###  6.2 Results

Single-view Reconstruction. We use the $8$ colored images and normal maps
generated by Wonder3d to reconstruct the shapes. Table 1 shows the comparison
results. Our TetSphere splatting technique excels over baseline methods in
terms of mesh quality while also achieving competitive levels of
reconstruction accuracy (Fig. 1(b)). This improvement is attributed to the
regularizations embedded within the TetSphere splatting optimization. These
results demonstrate the superior capabilities of our representation compared
to existing ones.

Image-to-3D Shape Generation. Fig. 6 and 7 illustrate the comparison results
on image-to-3D shape generation. Our approach outperforms Magic123 and
DreamCraft3D regarding mesh quality, achieving smoother surfaces for broad
regions, such as animal bodies while retaining local sharp details in areas
like eyes and noses. Magic123 tends to produce overly smooth surfaces that
sometimes deviate from the correct geometry, as observed in the mushroom and
the teddy bear in Fig. 7. DreamCraft3D suffers from noisy and bumpy surface
meshes, indicating lower mesh quality. Furthermore, we highlight the
computational efficiency of TetSphere splatting in Table 2. Compared to
various geometry representations, TetSphere splatting stands out for its
minimal memory usage and achieves the fastest run-time speed. This efficiency
underscores the benefits of TetSphere’s explicit and Lagrangian properties.

Figure 9: Results on text-to-3D shape generation. Our TetSphere splatting
excels in creating slender forms, demonstrating its strength in handling thin
structures effectively.

Text-to-3D Shape Generation. Fig. 9 shows comparison results on text-to-3D
generation, with additional results provided in Appendix 0.C. Our TetSphere
splatting is capable of producing slender structures such as the dragon’s head
and the goblet. Furthermore, the results demonstrate that the integration of
TetSphere splatting with SDS enhances the geometric detail of the generated
shapes, resulting in both diverse and high-quality textures.

###  6.3 Analysis

Effects of Energy Coefficients. Fig. 8 demonstrates how different energy
coefficients influence the reconstruction outcome. Larger coefficients lead to
a smoother surface, but excessively small coefficients may cause tetrahedron
inversion. In our experiments, we choose $w_{1}=5\times 10^{-6},w_{2}=2\times
10^{-5}$ and apply a cosine increasing schedule to balance surface smoothness
and structural integrity.

Dense-view Inverse Rendering. In Appendix 0.B, we show an additional
application of TetSphere splatting: inverse rendering with densly sampled
views and compare it with surface-mesh representation [65]. Our method
demonstrates fast and robust optimization results in this context. This
further illustrates TetSphere splatting’s potential for broader reconstruction
scenarios.

##  7 Conclusion

We introduced TetSphere splatting, a geometry representation for
reconstructing textured shapes using a tetrahedral mesh framework. This method
addresses the limitations of existing reconstruction methods, such as the high
computational cost of neural implicit representations and the suboptimal mesh
quality inherent in Eulerian geometry methods. Future work could extend
TetSphere splatting to leverage direct 3D supervision with volumetric data.
The current limitation of TetSphere splatting lies in its inability to
guarantee topology preservation due to the union of all tetrahedral spheres.
This underscores the necessity for future development of shape generation that
can adhere to topology constraints.

## References

  * [1] Alliegro, A., Siddiqui, Y., Tommasi, T., Nießner, M.: Polydiff: Generating 3d polygonal meshes with diffusion models. arXiv preprint arXiv:2312.11417 (2023)
  * [2] Armandpour, M., Sadeghian, A., Zheng, H., Sadeghian, A., Zhou, M.: Re-imagine the negative prompt algorithm: Transform 2D diffusion into 3d, alleviate janus problem and beyond (Apr 2023)
  * [3] Arun, K.S., Huang, T.S., Blostein, S.D.: Least-squares fitting of two 3-d point sets. IEEE Trans. Pattern Anal. Mach. Intell. 9(5), 698–700 (May 1987)
  * [4] Bærentzen, J.A., Gravesen, J., Anton, F., Aanæs, H.: Guide to computational geometry processing: foundations, algorithms, and methods. Springer Science & Business Media (2012)
  * [5] Bian, W., Wang, Z., Li, K., Prisacariu, V.A.: Ray-onet: Efficient 3d reconstruction from a single rgb image. In: BMVC (2021)
  * [6] Botsch, M., Sorkine, O.: On linear variational surface deformation methods. IEEE transactions on visualization and computer graphics 14(1), 213–230 (2007)
  * [7] Burley, B., Studios, W.D.A.: Physically-based shading at disney. In: Acm Siggraph. vol. 2012, pp. 1–7. vol. 2012 (2012)
  * [8] Chan, E.R., Lin, C.Z., Chan, M.A., Nagano, K., Pan, B., De Mello, S., Gallo, O., Guibas, L., Tremblay, J., Khamis, S., Karras, T., Wetzstein, G.: Efficient geometry-aware 3D generative adversarial networks (Dec 2021)
  * [9] Chan, E.R., Nagano, K., Chan, M.A., Bergman, A.W., Park, J.J., Levy, A., Aittala, M., De Mello, S., Karras, T., Wetzstein, G.: Generative novel view synthesis with 3D-Aware diffusion models (Apr 2023)
  * [10] Chen, R., Chen, Y., Jiao, N., Jia, K.: Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. arXiv preprint arXiv:2303.13873 (2023)
  * [11] Chen, R., Chen, Y., Jiao, N., Jia, K.: Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. arXiv preprint arXiv:2303.13873 (2023)
  * [12] Chen, Z., Zhang, H.: Learning implicit fields for generative shape modeling. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5939–5948 (2019)
  * [13] Choy, C.B., Xu, D., Gwak, J., Chen, K., Savarese, S.: 3d-r2n2: A unified approach for single and multi-view 3d object reconstruction. In: Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VIII 14. pp. 628–644. Springer (2016)
  * [14] Chung, T.J.: Computational fluid dynamics. Cambridge university press (2002)
  * [15] Deitke, M., Liu, R., Wallingford, M., Ngo, H., Michel, O., Kusupati, A., Fan, A., Laforte, C., Voleti, V., Gadre, S.Y., VanderBilt, E., Kembhavi, A., Vondrick, C., Gkioxari, G., Ehsani, K., Schmidt, L., Farhadi, A.: Objaverse-XL: A universe of 10m+ 3D objects (Jul 2023)
  * [16] Deitke, M., Schwenk, D., Salvador, J., Weihs, L., Michel, O., VanderBilt, E., Schmidt, L., Ehsani, K., Kembhavi, A., Farhadi, A.: Objaverse: A universe of annotated 3D objects (Dec 2022)
  * [17] Deng, C., Jiang, C.m., Qi, C.R., Yan, X., Zhou, Y., Guibas, L., Anguelov, D.: NeRDi: Single-View NeRF synthesis with Language-Guided diffusion as general image priors (Dec 2022)
  * [18] Dou, Z., Lin, C., Xu, R., Yang, L., Xin, S., Komura, T., Wang, W.: Coverage axis: Inner point selection for 3d shape skeletonization. In: Computer Graphics Forum. vol. 41, pp. 419–432. Wiley Online Library (2022)
  * [19] Downs, L., Francis, A., Koenig, N., Kinman, B., Hickman, R., Reymann, K., McHugh, T.B., Vanhoucke, V.: Google scanned objects: A High-Quality dataset of 3D scanned household items (Apr 2022)
  * [20] Fahim, G., Amin, K., Zarif, S.: Single-view 3d reconstruction: A survey of deep learning methods. Computers & Graphics 94, 164–190 (2021)
  * [21] Fan, H., Su, H., Guibas, L.: A point set generation network for 3D object reconstruction from a single image. arXiv [cs.CV] (Dec 2016)
  * [22] Fan, H., Su, H., Guibas, L.J.: A point set generation network for 3d object reconstruction from a single image. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 605–613 (2017)
  * [23] Fu, K., Peng, J., He, Q., Zhang, H.: Single image 3d object reconstruction based on deep learning: A review. Multimedia Tools and Applications 80, 463–498 (2021)
  * [24] Gao, J., Chen, W., Xiang, T., Jacobson, A., McGuire, M., Fidler, S.: Learning deformable tetrahedral meshes for 3d reconstruction. Advances In Neural Information Processing Systems 33, 9936–9947 (2020)
  * [25] Gao, W., Wang, A., Metzer, G., Yeh, R.A., Hanocka, R.: Tetgan: A convolutional neural network for tetrahedral mesh generation. arXiv preprint arXiv:2210.05735 (2022)
  * [26] Gkioxari, G., Malik, J., Johnson, J.: Mesh r-cnn. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 9785–9795 (2019)
  * [27] Goel, S., Kanazawa, A., Malik, J.: Shape and viewpoint without keypoints. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XV 16. pp. 88–104. Springer (2020)
  * [28] González, Á.: Measurement of areas on a sphere using fibonacci and latitude–longitude lattices. Mathematical Geosciences 42, 49–64 (2010)
  * [29] Groueix, T., Fisher, M., Kim, V.G., Russell, B.C., Aubry, M.: AtlasNet: A Papier-Mâché approach to learning 3D surface generation (Feb 2018)
  * [30] Gu, J., Trevithick, A., Lin, K.E., Susskind, J., Theobalt, C., Liu, L., Ramamoorthi, R.: Nerfdiff: Single-image view synthesis with nerf-guided distillation from 3d-aware diffusion. In: ICML (2023)
  * [31] He, Z., Wang, T.: Openlrm: Open-source large reconstruction models. https://github.com/3DTopia/OpenLRM (2023)
  * [32] Hong, S., Ahn, D., Kim, S.: Debiasing scores and prompts of 2D diffusion for view-consistent Text-to-3D generation (Mar 2023)
  * [33] Hong, Y., Zhang, K., Gu, J., Bi, S., Zhou, Y., Liu, D., Liu, F., Sunkavalli, K., Bui, T., Tan, H.: LRM: Large reconstruction model for single image to 3D (Nov 2023)
  * [34] Huang, Y., Wang, J., Shi, Y., Qi, X., Zha, Z.J., Zhang, L.: DreamTime: An improved optimization strategy for Text-to-3D content creation (Jun 2023)
  * [35] Jain, A., Mildenhall, B., Barron, J.T., Abbeel, P., Poole, B.: Zero-shot text-guided object generation with dream fields. CVPR (2022)
  * [36] Jakob, W., Speierer, S., Roussel, N., Nimier-David, M., Vicini, D., Zeltner, T., Nicolet, B., Crespo, M., Leroy, V., Zhang, Z.: Mitsuba 3 renderer (2022), https://mitsuba-renderer.org
  * [37] James, R.M.: Topology. Prentic Hall of India Private Limited, New delhi (2000)
  * [38] Jang, W., Agapito, L.: CodeNeRF: Disentangled neural radiance fields for object categories (Sep 2021)
  * [39] Jun, H., Nichol, A.: Shap-E: Generating conditional 3D implicit functions (May 2023)
  * [40] Kanazawa, A., Tulsiani, S., Efros, A.A., Malik, J.: Learning category-specific mesh reconstruction from image collections. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 371–386 (2018)
  * [41] Kerbl, B., Kopanas, G., Leimkühler, T., Drettakis, G.: 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics 42(4) (2023)
  * [42] Khalid, N.M., Xie, T., Belilovsky, E., Tiberiu, P.: Clip-mesh: Generating textured meshes from text using pretrained image-text models. SIGGRAPH Asia 2022 Conference Papers (December 2022)
  * [43] Laine, S., Hellsten, J., Karras, T., Seol, Y., Lehtinen, J., Aila, T.: Modular primitives for high-performance differentiable rendering. ACM Transactions on Graphics 39(6) (2020)
  * [44] Li, W., Chen, R., Chen, X., Tan, P.: SweetDreamer: Aligning geometric priors in 2D diffusion for consistent Text-to-3D (Oct 2023)
  * [45] Li, X., Liu, S., Kim, K., De Mello, S., Jampani, V., Yang, M.H., Kautz, J.: Self-supervised single-view 3d reconstruction via semantic consistency. In: ECCV (2020)
  * [46] Lin, C.H., Gao, J., Tang, L., Takikawa, T., Zeng, X., Huang, X., Kreis, K., Fidler, S., Liu, M.Y., Lin, T.Y.: Magic3d: High-resolution text-to-3d content creation. In: CVPR (2023)
  * [47] Liu, M., Xu, C., Jin, H., Chen, L., Mukund, V.T., Xu, Z., Su, H.: One-2-3-45: Any single image to 3D mesh in 45 seconds without Per-Shape optimization (Jun 2023)
  * [48] Liu, M., Xu, C., Jin, H., Chen, L., T, M.V., Xu, Z., Su, H.: One-2-3-45: Any single image to 3d mesh in 45 seconds without per-shape optimization (2023)
  * [49] Liu, R., Wu, R., Hoorick, B.V., Tokmakov, P., Zakharov, S., Vondrick, C.: Zero-1-to-3: Zero-shot one image to 3d object (2023)
  * [50] Liu, Y., Lin, C., Zeng, Z., Long, X., Liu, L., Komura, T., Wang, W.: SyncDreamer: Generating multiview-consistent images from a single-view image (Sep 2023)
  * [51] Liu, Y., Lin, C., Zeng, Z., Long, X., Liu, L., Komura, T., Wang, W.: Syncdreamer: Learning to generate multiview-consistent images from a single-view image. arXiv preprint arXiv:2309.03453 (2023)
  * [52] Liu, Z., Feng, Y., Black, M.J., Nowrouzezahrai, D., Paull, L., Liu, W.: Meshdiffusion: Score-based generative 3d mesh modeling. arXiv preprint arXiv:2303.08133 (2023)
  * [53] Long, X., Guo, Y.C., Lin, C., Liu, Y., Dou, Z., Liu, L., Ma, Y., Zhang, S.H., Habermann, M., Theobalt, C., Wang, W.: Wonder3D: Single image to 3D using Cross-Domain diffusion (Oct 2023)
  * [54] Ma, B., Deng, H., Zhou, J., Liu, Y.S., Huang, T., Wang, X.: Geodream: Disentangling 2d and geometric priors for high-fidelity and consistent 3d generation. arXiv preprint arXiv:2311.17971 (2023)
  * [55] Mandikal, P., Navaneet, K.L., Agarwal, M., Venkatesh Babu, R.: 3D-LMNet: Latent embedding matching for accurate and diverse 3D point cloud reconstruction from a single image (Jul 2018)
  * [56] Melas-Kyriazi, L., Rupprecht, C., Laina, I., Vedaldi, A.: RealFusion: 360° reconstruction of any object from a single image (Feb 2023)
  * [57] Mescheder, L., Oechsle, M., Niemeyer, M., Nowozin, S., Geiger, A.: Occupancy networks: Learning 3d reconstruction in function space. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 4460–4470 (2019)
  * [58] Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: Nerf: Representing scenes as neural radiance fields for view synthesis. ECCV (2020)
  * [59] Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM 65(1), 99–106 (2021)
  * [60] Mittal, P., Cheng, Y.C., Singh, M., Tulsiani, S.: Autosdf: Shape priors for 3d completion, reconstruction and generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 306–315 (2022)
  * [61] Müller, N., Simonelli, A., Porzi, L., Bulo, S.R., Nießner, M., Kontschieder, P.: Autorf: Learning 3d object radiance fields from single view observations. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3971–3980 (2022)
  * [62] Müller, T., Evans, A., Schied, C., Keller, A.: Instant neural graphics primitives with a multiresolution hash encoding. ACM Transactions on Graphics (ToG) 41(4), 1–15 (2022)
  * [63] Munkberg, J., Hasselgren, J., Shen, T., Gao, J., Chen, W., Evans, A., Müller, T., Fidler, S.: Extracting Triangular 3D Models, Materials, and Lighting From Images. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 8280–8290 (June 2022)
  * [64] Nichol, A., Jun, H., Dhariwal, P., Mishkin, P., Chen, M.: Point-E: A system for generating 3D point clouds from complex prompts (Dec 2022)
  * [65] Nicolet, B., Jacobson, A., Jakob, W.: Large steps in inverse rendering of geometry. ACM Transactions on Graphics (TOG) 40(6), 1–13 (2021)
  * [66] Park, J.J., Florence, P., Straub, J., Newcombe, R., Lovegrove, S.: Deepsdf: Learning continuous signed distance functions for shape representation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 165–174 (2019)
  * [67] Poole, B., Jain, A., Barron, J.T., Mildenhall, B.: Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209.14988 (2022)
  * [68] Qian, G., Mai, J., Hamdi, A., Ren, J., Siarohin, A., Li, B., Lee, H.Y., Skorokhodov, I., Wonka, P., Tulyakov, S., Ghanem, B.: Magic123: One image to high-quality 3d object generation using both 2d and 3d diffusion priors. arXiv preprint arXiv:2306.17843 (2023)
  * [69] Qiu, L., Chen, G., Gu, X., Zuo, Q., Xu, M., Wu, Y., Yuan, W., Dong, Z., Bo, L., Han, X.: RichDreamer: A generalizable Normal-Depth diffusion model for detail richness in Text-to-3D (Nov 2023)
  * [70] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: ICML (2021)
  * [71] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: CVPR (2022)
  * [72] Roth, J., Tong, Y., Liu, X.: Adaptive 3d face reconstruction from unconstrained photo collections. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 4197–4206 (2016)
  * [73] Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E.L., Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., et al.: Photorealistic text-to-image diffusion models with deep language understanding. NeurIPS (2022)
  * [74] Schüller, C., Kavan, L., Panozzo, D., Sorkine-Hornung, O.: Locally injective mappings. In: Computer Graphics Forum. vol. 32, pp. 125–135. Wiley Online Library (2013)
  * [75] Shen, B., Yan, X., Qi, C.R., Najibi, M., Deng, B., Guibas, L., Zhou, Y., Anguelov, D.: GINA-3D: Learning to generate implicit neural assets in the wild. arXiv [cs.CV] (Apr 2023)
  * [76] Shen, Q., Yang, X., Wang, X.: Anything-3D: Towards single-view anything reconstruction in the wild (Apr 2023)
  * [77] Shen, T., Gao, J., Yin, K., Liu, M.Y., Fidler, S.: Deep marching tetrahedra: a hybrid representation for high-resolution 3d shape synthesis. In: NeurIPS (2021)
  * [78] Shi, Y., Wang, P., Ye, J., Long, M., Li, K., Yang, X.: MVDream: Multi-view diffusion for 3D generation (Aug 2023)
  * [79] Sifakis, E., Barbic, J.: Fem simulation of 3d deformable solids: a practitioner’s guide to theory, discretization and model reduction. In: Acm siggraph 2012 courses, pp. 1–50 (2012)
  * [80] Sun, J., Zhang, B., Shao, R., Wang, L., Liu, W., Xie, Z., Liu, Y.: DreamCraft3D: Hierarchical 3D generation with bootstrapped diffusion prior (Oct 2023)
  * [81] Tang, J., Chen, Z., Chen, X., Wang, T., Zeng, G., Liu, Z.: LGM: Large Multi-View gaussian model for High-Resolution 3D content creation (Feb 2024)
  * [82] Tang, J., Ren, J., Zhou, H., Liu, Z., Zeng, G.: Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. arXiv preprint arXiv:2309.16653 (2023)
  * [83] Tang, J., Ren, J., Zhou, H., Liu, Z., Zeng, G.: DreamGaussian: Generative gaussian splatting for efficient 3D content creation (Sep 2023)
  * [84] Tang, J., Wang, T., Zhang, B., Zhang, T., Yi, R., Ma, L., Chen, D.: Make-it-3d: High-fidelity 3d creation from a single image with diffusion prior. arXiv preprint arXiv:2303.14184 (2023)
  * [85] Tang, S., Chen, J., Wang, D., Tang, C., Zhang, F., Fan, Y., Chandra, V., Furukawa, Y., Ranjan, R.: MVDiffusion++: A dense high-resolution multi-view diffusion model for single or sparse-view 3D object reconstruction (Feb 2024)
  * [86] Tang, S., Zhang, F., Chen, J., Wang, P., Furukawa, Y.: MVDiffusion: Enabling holistic multi-view image generation with Correspondence-Aware diffusion (Jul 2023)
  * [87] Tsalicoglou, C., Manhardt, F., Tonioni, A., Niemeyer, M., Tombari, F.: TextMesh: Generation of realistic 3D meshes from text prompts (Apr 2023)
  * [88] Tulsiani, S., Zhou, T., Efros, A.A., Malik, J.: Multi-view supervision for single-view reconstruction via differentiable ray consistency. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2626–2634 (2017)
  * [89] Wang, H., Du, X., Li, J., Yeh, R.A., Shakhnarovich, G.: Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation. CVPR (2023)
  * [90] Wang, N., Zhang, Y., Li, Z., Fu, Y., Liu, W., Jiang, Y.G.: Pixel2mesh: Generating 3d mesh models from single rgb images. In: Proceedings of the European conference on computer vision (ECCV). pp. 52–67 (2018)
  * [91] Wang, P., Liu, L., Liu, Y., Theobalt, C., Komura, T., Wang, W.: NeuS: Learning neural implicit surfaces by volume rendering for multi-view reconstruction (Jun 2021)
  * [92] Wang, P., Tan, H., Bi, S., Xu, Y., Luan, F., Sunkavalli, K., Wang, W., Xu, Z., Zhang, K.: PF-LRM: Pose-Free large reconstruction model for joint pose and shape prediction (Nov 2023)
  * [93] Wang, Z., Lu, C., Wang, Y., Bao, F., Li, C., Su, H., Zhu, J.: Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. NeurIPS (2023)
  * [94] Wang, Z., Lu, C., Wang, Y., Bao, F., Li, C., Su, H., Zhu, J.: Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. Advances in Neural Information Processing Systems 36 (2024)
  * [95] Wen, J., Wang, B., Barbic, J.: Large-strain surface modeling using plasticity. IEEE Transactions on Visualization and Computer Graphics (2023)
  * [96] Weng, Z., Liu, J., Tan, H., Xu, Z., Zhou, Y., Yeung-Levy, S., Yang, J.: Single-View 3D human digitalization with large reconstruction models (Jan 2024)
  * [97] Xie, H., Yao, H., Sun, X., Zhou, S., Zhang, S.: Pix2Vox: Context-aware 3D reconstruction from single and multi-view images (Jan 2019)
  * [98] Xie, H., Yao, H., Zhang, S., Zhou, S., Sun, W.: Pix2Vox++: Multi-scale context-aware 3D object reconstruction from single and multiple images. Int. J. Comput. Vis. 128(12), 2919–2935 (Dec 2020)
  * [99] Xu, D., Jiang, Y., Wang, P., Fan, Z., Wang, Y., Wang, Z.: NeuralLift-360: Lifting an in-the-wild 2D photo to a 3D object with 360° views (Nov 2022)
  * [100] Xu, Q., Wang, W., Ceylan, D., Mech, R., Neumann, U.: DISN: Deep implicit surface network for high-quality single-view 3D reconstruction (May 2019)
  * [101] Xu, Y., Tan, H., Luan, F., Bi, S., Wang, P., Li, J., Shi, Z., Sunkavalli, K., Wetzstein, G., Xu, Z., Zhang, K.: DMV3D: Denoising Multi-View diffusion using 3D large reconstruction model (Nov 2023)
  * [102] Yang, S., Liu, J., Wang, W., Guo, Z.: Tet-gan: Text effects transfer via stylization and destylization. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 33, pp. 1238–1245 (2019)
  * [103] Yu, A., Ye, V., Tancik, M., Kanazawa, A.: pixelNeRF: Neural radiance fields from one or few images (Dec 2020)

##  Appendix 0.A Implementation Details

Baselines. For the baseline methods, including RealFusion [56], Magic123 [68],
One-2-3-45 [47], Point-E [64], Shap-E [39], Zero123 [49], SyncDreamer [50],
Wonder3d [53], Open-LRM [33, 31], DreamGaussian [83], Dreamcraft3D [80], and
RichDreamer [69], we follow their original implementations and use their
publicly available codebase for getting the results in this paper.

Our method. Our implementation of the TetSphere initialization algorithm is
developed in C++ and makes use of the Gurobi linear programming solver. The
optimization of geometric energies is implemented using CUDA as a PyTorch
extension to enhance computational efficiency. 1) For single-view
reconstruction, we employ Wonder3d [53] to generate six multi-view images,
including both color and normal images for each view, using predefined camera
poses for reconstruction with TetSphere splatting. We use a 2-layer MLP for
texture representation. The optimization objective $\mathbf{\Phi}(\cdot)$
encompasses both the rendering loss and the normal loss. The rendering loss
consists of an $l_{1}$ norm on tone-mapped color and MSE on the alpha mask,
along with a cosine loss on normals, similar to that described in [63]; 2) For
image-to-3D shape generation, we obtain multi-view images from the initial
stage of DreamCraft3D (coarse NeRF fitting only), generating $360$ views
sampled on a Fibonacci sphere [28]. The texture optimization directly employs
a $2048\times 2048$ 2D texture image, circumventing the need for additional
neural networks. Here, the optimization objective $\mathbf{\Phi}(\cdot)$
focuses solely on the rendering loss; 3) For text-to-3D generation, we use the
initial stage of RichDreamer [69] to obtain multi-view images. In this
scenario, we optimize our TetSphere splatting using both the rendering loss
and the SDS loss. The SDS loss for geometry is calculated using the Normal-
depth diffusion model as described in [69]. Likewise, the SDS loss for texture
leverages the albedo diffusion model from the same work. We implement a
2-layer MLP to parameterize and optimize the PBR material of the shape. To
enhance robustness and efficiency, we apply a cosine scheduler to scale the
coefficients of the geometry loss for all applications, formulated as
$\eta=4^{\sin(\frac{t\pi}{2n})}$, where $t$ denotes the current iteration and
$T$ represents the total number of iterations.

##  Appendix 0.B Dense-view Inverse Rendering

Figure 10: Results on 3D shape reconstruction from dense multi-view images
(1/2). Shapes reconstructed using our method are depicted in gray, whereas
those reconstructed with the state-of-the-art method from [65] are presented
in blue. While both methods effectively minimize the rendering loss, our
technique does not produce artifacts, such as undesired wrinkles and flipping
triangles (highlighted in red), and producing high-quality triangles, as
illustrated by the wireframes.

Fig. 10, 11, and 12 show the results of 3D shape reconstruction from dense
multi-view images. These multi-view images were captured by densely sampling
camera positions around each object (employing 360 views sampled on a sphere
based on the Fibonacci sequence [28]) and subsequently rendered using Mitsuba
[36]. The chosen shapes for this demonstration are from the GSO dataset.

To establish the efficacy of our method, we compare its performance with the
state-of-the-art surface mesh-based approach as described in [65] by applying
both techniques to the aforementioned dense multi-view images. Our method
utilizes a consistent hyperparameter set for all shapes undergoing $3,000$
optimization iterations, mirroring the parameters described in the main paper.
In contrast, for the method in [65], we adhere to the parameters in the
original paper – namely, using $\lambda=19$ and a step size of $10^{-2}$ – and
conduct $10,000$ iterations of optimization. This number of iterations is
necessary due to difficulties in achieving convergence within a few thousand
iterations for this method.

While both methods effectively minimize the rendering loss, our method
consistently delivers results of superior quality. Fig. 10 and 11 illustrate
that our reconstructions are devoid of common artifacts such as wiggles or
kinks, and showcase high-quality triangles, as evidenced by the wireframe
representations. Furthermore, our approach demonstrates faster convergence and
notable scalability advantages when compared to the second-order method in
[65]. Additionally, as Fig. 12 indicates, our method is also capable of
handling shapes with complex topologies, further underscoring its versatility
and effectiveness.

##  Appendix 0.C Additional Results

###  0.C.1 Qualitative Results on Single-view Reconstruction

Fig. 13 shows the results of single-view reconstruction performed on the GSO
dataset. Our TetSphere demonstrates superior mesh quality, effectively
capturing sharp geometric features, including the boundaries of the shoes.

Figure 11: Results on 3D shape reconstruction from dense multi-view images
(2/2).

Figure 12:  Our method is capable of handling shapes with complex topologies,
an advantage that existing methods fail to achieve [65]. In this example, both
shapes are computed by incorporating the rendering loss and normal loss.

Figure 13: Qualitative results on single-view reconstruction.

###  0.C.2 More Results on Text-to-3D Shape Generation

Fig. 14 shows additional results of text-to-3D shape generation. These results
highlight our method’s capability to construct complicated material, such as
reflections, by leveraging the explicit geometry representation.

Figure 14: More results on text-to-3D shape generation.

##  Appendix 0.D Tetrahedron and its Deformation Gradient

Following [79], we treat a tetrahedron as a piecewise linear element. The
initial (undeformed) positions of the four vertices of a tetrahedron are
denoted by
$\mathbf{X}=[\mathbf{X}^{(1)},\mathbf{X}^{(2)},\mathbf{X}^{(3)},\mathbf{X}^{(4)}]$,
with each $\mathbf{X}^{(i)}\in\mathbb{R}^{3}$. Similarly, the positions of the
four vertices of a deformed tetrahedron are represented by
$\mathbf{x}=[\mathbf{x}^{(1)},\mathbf{x}^{(2)},\mathbf{x}^{(3)},\mathbf{x}^{(4)}]$,
where each $\mathbf{x}^{(i)}\in\mathbb{R}^{3}$. The deformation gradient
$\mathbf{F}\in\mathbb{R}^{3\times 3}$, which quantifies the local deformation
of the tetrahedron, is given by:

$\displaystyle\mathbf{F}$ $\displaystyle=\mathbf{D}_{s}\mathbf{D}_{m}^{-1},$
(3) $\displaystyle\mathbf{D}_{s}$
$\displaystyle:=\left[\begin{array}[]{ccc}\mathbf{x}^{(1)}-\mathbf{x}^{(4)}&\
\mathbf{x}^{(2)}-\mathbf{x}^{(4)}&\
\mathbf{x}^{(3)}-\mathbf{x}^{(4)}\end{array}\right],$ (5)
$\displaystyle\mathbf{D}_{m}$
$\displaystyle:=\left[\begin{array}[]{ccc}\mathbf{X}^{(1)}-\mathbf{X}^{(4)}&\
\mathbf{X}^{(2)}-\mathbf{X}^{(4)}&\
\mathbf{X}^{(3)}-\mathbf{X}^{(4)}\end{array}\right].$ (7)

The deformation gradient $\mathbf{F}$ essentially captures how a tetrahedron
transforms from its initial state to its deformed state, encompassing both
rotation and stretching effects.

##  Appendix 0.E Formulation Details of TetSphere Initialization

Assuming there are a total of $m$ candidate positions obtained from the coarse
voxel grid (as shown in Fig. 5), our goal with TetSphere initialization is to
select a subset of these candidate points such that the object’s shape is
adequately covered by tetrahedral spheres centered at these positions. We
first initialize a sphere of fixed radius at each candidate position, where
the radius is calculated as $\alpha r+\beta$, where $r$ is the minimum
distance from each candidate position to the voxel surface. We use
$\alpha=1.2,\beta=0.07$ in all our examples. The objective is to select a
subset of spheres that collectively cover all voxel positions.

We define a coverage matrix $\mathbf{D}\in\\{0,1\\}^{m\times m}$, where each
element $d_{ji}\in\\{0,1\\}$ indicates whether voxel position $j$ is covered
by a sphere centered on candidate position $i$. A binary vector
$\mathbf{v}\in\\{0,1\\}^{m}$ identifies selected candidate positions, with
each element denoting the selection status of corresponding voxel positions.
The selection of feature points is formulated as a mixed-integer linear
programming problem:

$\displaystyle\min_{\mathbf{v}}|\mathbf{v}|\ \ \ \mathrm{s.t.}\ \
\mathbf{D}\mathbf{v}\geq\mathbbm{1},$ (8)

where $|\cdot|$ is the $l_{1}$ norm, $\mathbbm{1}$ is a vector of ones. This
optimization is efficiently solved using standard linear programming solvers.

##  Appendix 0.F Related work – Text-to-3D content generation

The recent success of text-to-2D image generation models has spurred a growing
interest in generating 3D output from text input. In light of the limited
availablity of text-annotated 3D datasets, methods have been developed that
leverage the pre-trained text-to-image models to reason between 2D renderings
of 3D models and text descriptions. Early works [42, 35] adopt the pre-trained
CLIP model [70] to supervise the generation by aligning the clip text and
image embeddings. More recently, 2D diffusion-based generative models [73, 71]
have powered direct supervision in the image/latent space and achieved
superior 3D quality [67, 84, 11, 46, 93]. Notably, DreamFusion [67] introduces
the Score Distillation Sampling (SDS) for supervising the NeRF [58]
optimization using diffusion priors as a score function (i.e., by minimizing
the added noise and the predicted noise under the text condition). Follow-up
works have since been proposed to improve score sampling formula with Perturb-
and-Average and variational method [93, 89], improved noise sampling schedules
[34], various 3D representations [46, 87, 11], text prompts [2], 3D
consistency [44, 32], and prior quality with dedicated diffusion models [69,
78]. Our method leverages the Normal-Depth diffusion model used in [69] for
text-to-3D shape generation, which is trained on the large-scale LAION
dataset, but replaces the geometry representation with our TetSphere.