# LEVERAGING 3D INFORMATION IN UNSUPERVISED BRAIN MRI SEGMENTATION

###### Abstract

Automatic segmentation of brain abnormalities is challenging, as they vary
considerably from one pathology to another. Current methods are supervised and
require numerous annotated images for each pathology, a strenuous task. To
tackle anatomical variability, Unsupervised Anomaly Detection (UAD) methods
are proposed, detecting anomalies as outliers of a healthy model learned using
a Variational Autoencoder (VAE). Previous work on UAD adopted a 2D approach,
meaning that MRIs are processed as a collection of independent slices. Yet, it
does not fully exploit the spatial information contained in MRI. Here, we
propose to perform UAD in a 3D fashion and compare 2D and 3D VAEs. As a side
contribution, we present a new loss function guarantying a robust training.
Learning is performed using a multicentric dataset of healthy brain MRIs, and
segmentation performances are estimated on White-Matter Hyperintensities and
tumors lesions. Experiments demonstrate the interest of 3D methods which
outperform their 2D counterparts.

††*Data used in preparation of this article were partially obtained from the
Alzheimer’s Disease Neuroimaging Initiative (ADNI) database
(adni.loni.usc.edu). As such, the investigators within the ADNI contributed to
the design and implementation of ADNI and/or provided data but did not
participate in analysis or writing of this report. A complete listing of ADNI
investigators can be found at: http://adni.loni.usc.edu/wp-
content/uploads/how_to_apply/ADNI_Acknowledgement_List.pdf

Index Terms—  Deep Learning, Variational Autoencoder, Anomaly Detection,
Medical Imaging

##  1 Introduction

Brain anomalies are widely used as biomarkers indicating the presence or
progress of many neurological disorders. Magnetic Resonance Imaging (MRI) is
today an essential modality to reveal these markers. In recent years, a
multitude of algorithms for the automatic detection of anomalies in brain MRI
has been proposed, with promising results achieved by Deep Learning (DL)
approaches.

Anomalies can manifest in a wide range of shape, location and intensities. To
automatically detect them, most state-of-the-art algorithms use a supervised
approach, trained on an as large as possible manually annotated dataset [1].
Acquisition of such dataset can be both time-consuming and expensive. Besides,
these algorithms are specific to the task they are trained for and will
therefore perform poorly when applied on images presenting unseen type of
anomalies.

To overcome these limitations, Unsupervised Anomaly Detection (UAD) methods
are proposed. They consist in the construction of a healthy model, which
allows to detect anomalies from their deviation from the model. Recent DL
methods propose to learn a manifold of healthy patients by computing lower-
dimensional latent representations of the scans. At inference, anomalous
images are mapped to the learned healthy manifold. An anomaly map is then
computed using the voxel-wise difference from its healthy projection. Although
failing for the time being to meet the state-of-the-art results of supervised
methods, UAD approaches have several advantages: they do not require any
labeled data and can be adapted with very limited adjustments to different
types of pathologies.

Two types of neural networks are particularly investigated in UAD : Generative
Adversarial Networks (GANs) [2] and Variational Autoencoders (VAEs) [3]. GANs
use an adversarial training approach to construct the healthy manifold [4].
VAEs rather encode images into latent distributions and use a regularization
term in their loss function to ensure consistency in the latent space [5].
Yet, training of VAEs is known to be unstable [6]. As an attempt to alleviate
this issue, we propose a robust loss function to ensure a stable training.

To the best of our knowledge, all previous research related to UAD tackle the
problem in a 2D fashion, meaning that the MRI volume is processed as a
collection of independent 2D slices. Despite being computationally efficient,
this approach has several limitations. First, it requires the selection of
homogeneous slices. Top and bottom slices, which contain no or little brain
tissue, are excluded, as they may compromise the training of 2D networks. It
is an impairing step because it requires hand-tuning and prevents the
detection of anomalies in the entire MRI. Second, the processing of 2D data
does not fully exploit the spatial information available in the scans. This
restriction to 2D may arise from the significant difficulty of 3D training,
which suffers from the high dimensionality of data and the reduction of the
number of training samples. Another limitation of the state of the art is the
restriction of most previous work on UAD to monocentric datasets for training
[5, 7, 8], which fails to ensure a clinically realistic setup.

In this work, we present a 3D VAE framework for UAD. We train 2D and 3D models
using our proposed robust loss function and compare them on multiple
pathologies acquired in real clinical conditions. Experiments show that
exploiting the full images in a 3D fashion leads to better performances than
the usual 2D approaches.

##  2 Methods

###  2.1 UAD using Variational Autoencoders

VAEs are composed of an encoder followed by a decoder. The encoder compresses
the input image $X$ into a distribution over the latent space, which is
regularized to be close to a prior distribution conventionally set to be a
Gaussian multivariate distribution $N(z;0,I)$. A point $z$ is then sampled
from this latent distribution and presented to the decoder which produces a
reconstruction $\hat{X}$ of the input.

The bottleneck separating the encoder and the decoder distinguishes spatial
and dense VAEs. In the spatial configuration, VAEs are fully-convolutional,
meaning that the latent vector is a multi-dimensional tensor
$z\in\mathbb{R}^{N\times h\times w}$ in 2D and $z\in\mathbb{R}^{N\times
d\times h\times w}$ in 3D with $N$ the latent space dimension, $h$ the height,
$w$ the width and $d$ the depth. In this configuration, spatial information is
preserved during the encoding-decoding process. In the dense configuration,
the image is encoded into a 1-dimensional tensor $z\in\mathbb{R}^{N}$ with the
use of fully-connected layers, and as a consequence spatial information is
discarded.

After training on healthy subjects, we perform UAD using a reconstruction-
based approach. Reconstruction is the classical approach for DL-based UAD. It
consists in the computation of anomaly maps by subtracting its reconstruction
$\hat{X}$ to the input image $X$. As a result of the training, anomalies are
badly reconstructed once projected to the healthy manifold, and thus yield to
high values in the anomaly map. By binarizing them with an appropriate
threshold, anomalies segmentations are obtained.

###  2.2 Collapsing-robust loss function

Training of a VAE is performed by minimizing a two-terms loss function: the
evidence lower-bound ($ELBO(X,\hat{X})$):

$\pazocal{L}=|X-\hat{X}|_{d}+D_{KL}(q_{\phi}(z|X)|P(z)),d\in\\{1,2\\}$ (1)

The left-hand side term of this equation is the reconstruction term, which
computes the distance between the image $X$ and its reconstruction $\hat{X}$.
The second term is the Kullback-Leibler Divergence (KL) between the posterior
$q_{\phi}(z|X)$ and the defined prior $P(z)$, which acts as a regularization
term in the latent space. Training of VAE using equation (1) is prone to
_posterior collapse_ , meaning that the network may learn to neglect a subset
of latent variables to match the prior, and thus the generative power of the
network degrades [6]. Solutions have been proposed to tackle the problem,
including the addition of a hyperparameter $\beta$ to balance both terms of
the equation [9]. Following the same motivation, KL annealing schedules have
been presented, including the KL cyclical annealing schedule described in
[10]. We build on this work and propose a custom loss function defined as
follows:

$\displaystyle\pazocal{L}_{T}$
$\displaystyle=\dfrac{|X-\hat{X}|_{1}}{\Sigma}+\beta(t)D_{KL}(q_{\phi}(z|X)|P(z))$
(2) $\displaystyle\text{with }\beta(t)$
$\displaystyle=\left\\{\begin{array}[]{ll}\dfrac{2t}{T}\text{ for
t}\in[0,\dfrac{T}{2}[\\\ \hskip 2.0pt1\hskip
8.0pt\text{else}\end{array}\right.\text{and }\beta(t+T)=\beta(t)$

where $\Sigma$ is the moving mean computed on the $L$ last values of the
reconstruction term, $t$ is the current iteration and T the cycle duration.
$T$ and $L$ are two hyperparameters that we set to 50 and 10 respectively in
our experiments. We choose the $\ell_{1}$ norm as reconstruction term to
reduce blurring of the reconstructed images. Normalizing this term by its
moving mean maintains the reconstruction term near one, which encourages the
network to learn throughout the entire training stage even when reconstruction
becomes satisfying. Coupled with the KL cyclical annealing schedule, we obtain
a loss robust to collapsing which ensures a stable training of VAEs. We use
this loss without any additional parameter tuning in all our experiments.

###  2.3 Architecture Details

Our encoders are composed of 6 layers: 4 convolutional layers compressing the
input image, and 2 additional layers to encode the result into a normal
distribution by computing its parameters $\mu$ and $\sigma$. These two last
layers are convolutional in the spatial configuration, and fully-connected for
dense VAEs. A latent vector $z$ is then sampled from this distribution and
passed to the decoder.

Decoders adopt a symmetric architecture with 5 layers decompressing $z$ in
order to reconstruct the input image. Similarly, the first layer of the
decoder is convolutional in the spatial configuration, and fully-connected
otherwise. 3D adaptations of our architectures are obtained by replacing 2D
convolutions by their 3D counterparts.

##  3 Experiments

We focus on brain MRI FLAIR. To train our networks, we gather 79 healthy scans
from several opensource datasets : the ADNI dataset, IBC 111This data was
obtained from the OpenfMRI database. Its accession number is ds000244.[11] and
Kirby [12]. Evaluation of our networks is performed on 2 different datasets
presenting various brain lesions. We use a collection of 196 scans with
annotated White-Matter Hyperintensities (WMH) gathered from several opensource
datasets : the MICCAI MSSEG Challenge [13], the ISBI MS Lesion Challenge [14],
and the WMH Challenge [15]. Additionally, we select 100 brain tumors scans
from the BraTS 2018 dataset [16]. These datasets are multi-centric and
comprise various scanners and clinical protocols, hence assuring a clinically
realistic setup.

Scans are rigidly registered on a $170\times 204\times 170$ template with an
isotropic resolution of $1mm$, corrected for bias using [17] and skull-
stripped using the HD-BET algorithm [18]. Volumes are then cropped to
$160\times 192\times 75$ to concentrate on central slices, for which brain
tissue is abundant. Finally, intensity range is set to $[0,1]$.

Segmentations are produced by binarizing anomaly maps, obtained as the
difference between an image and its reconstruction. Binarization is performed
using a threshold above which voxels are considered as anomalies. In order to
find its optimal value, we split each testing set in half. On the first half,
we compute Dice scores between ground truths and segmentations obtained with
15 different thresholds, ranged between $0\text{ and }0.15$, which we found to
be a relevant range in our experiments. Then, the threshold which provided the
best performance is applied on the second half.

The obtained segmentations are then multiplied by a slightly eroded brain
mask, in order to remove false positives occurring near the brain contour.
Finally, a median filter is applied to remove regions with less than 10
voxels.

Fig. 1: Reconstructions and segmentations of our networks. Left : WMH lesions.
Right : tumors lesions.

##  4 Results

We evaluate our networks segmentation performance using Dice, sensitivity and
specificity scores. Reconstruction performance is estimated using the voxel-
wise mean average error (MAE) between the input scan and its reconstruction.
To fairly compare 2D and 3D networks, they are all evaluated on the same 75
central slices for each test scan. Performances of the Spatial 2D and Dense
2D, as well as their 3D adaptations, are presented in Table 1. Illustrative
reconstructions and segmentations can be found in Figure 1 for both WMH and
tumor lesions. 3D adaptations demonstrate a boost in performance as compared
to 2D. On both the WMH and Tumors datasets, the best network is the Dense 3D
VAE, with dice scores of $0.463\pm 0.259$ and $0.650\pm 0.190$, a specificity
of $0.996$ and $0.981$, and a sensitivity of $0.511$ and $0.711$ respectively.

| WMH | Tumors
---|---|---
Model | Dice | Spe | Sen | MAE | Dice | Spe | Sen | MAE
Spatial 2D | $0.280\pm 0.174$ | $0.986$ | $0.547$ | $0.049$ | $0.260\pm 0.121$ | $0.912$ | $0.376$ | $0.079$
Spatial 3D | $0.336\pm 0.203$ | $0.989$ | $0.605$ | $0.056$ | $0.386\pm 0.160$ | $0.898$ | $0.655$ | $0.065$
Dense 2D | $0.460\pm 0.262$ | $0.994$ | $0.604$ | $0.102$ | $0.618\pm 0.167$ | $0.978$ | $0.676$ | $0.133$
Dense 3D | $\textbf{0.463}\pm\textbf{0.259}$ | $0.996$ | $0.511$ | $0.104$ | $\textbf{0.650}\pm\textbf{0.190}$ | $0.981$ | $0.711$ | $0.151$
Table 1:  Performance metrics obtained on the test datasets. Spe =
Specificity, Sen = Sensitivity, MAE = Mean Average Error.

##  5 Discussion and Conclusion

In this work, the interest of 3D methods for UAD in brain MRIs was evaluated.
Using our proposed collapsing-robust loss function, 2D and 3D models were
compared on two pathological datasets showing an overall increase in
performance in the 3D setting. For spatial VAEs, this gain was more
significant than for dense networks. In the dense configuration, spatial
information was not preserved during the encoding-decoding scheme. Thus, the
introduction of additional spatial context with the use of 3D did not
significantly increase the segmentation performance. Additionally, we observed
that dense networks outperformed spatial networks in both the 2D and 3D cases.
They indeed operated a significantly higher compression of the scan as
compared to spatial networks. Consequently, reconstructions presented very
little details, and anomalies were entirely discarded, which made them more
easily detectable.

Performances of our approaches, although being satisfying, remain lower than
state-of-the-art supervised methods. In [19], a Dice score above $0.9$ is
obtained in tumor segmentation task, and a score of $0.56$ is achieved in [20]
for WMH lesions segmentation with supervised approaches. Although UAD methods
fail at reaching these standards, they are very promising as they allow to
detect anomalies in a generic fashion and without the use of annotated
training samples. One interesting application of UAD approaches is the
generation of raw segmentations, which can be used as a starting point for
manual segmentations, saving time for the rater.

In future work, advanced 3D architectures could be tested with the use of
deeper networks. Extensions to multi-sequential MRI data is also of interest.

##  6 Compliance with ethical standards

This research study was conducted retrospectively using human subject data
made available by the following sources: ADNI, OpenfMRI, NITRC, the MICCAI
MSSEG Challenge, the ISBI MS Lesion Challenge, the WMH Challenge and BraTS
2018. Ethical approval was not required as confirmed by the license attached
with the data.

##  7 Acknowledgments

BL, ML, SD and AT are employees of the Pixyl Company. MD and FF serve on Pixyl
advisory board.

## References

  * [1] Z. Akkus et al.,  “Deep learning for brain MRI segmentation: state of the art and future directions,”  J Dig Im, vol. 30, no. 4, 2017.
  * [2] Ian Goodfellow et al.,  “Generative adversarial nets,”  Adv Neural Inf Proc Sys, 2014.
  * [3] D. P. Kingma et al.,  “Auto-encoding variational Bayes,”  Int Conf Learn Repres, 2013.
  * [4] F. Di Mattia et al.,  “A survey on GANs for anomaly detection,”  CoRR, 2019.
  * [5] C. Baur et al.,  “Autoencoders for unsupervised anomaly segmentation in brain MR images: A comparative study,”  CoRR, 2020.
  * [6] J. Lucas et al.,  “Understanding posterior collapse in generative latent variable models,”  Int Conf Learn Repres, 2019.
  * [7] X. Chen et al.,  “Unsupervised detection of lesions in brain MRI using constrained adversarial auto-encoders,”  CoRR, 2018.
  * [8] D. Zimmerer et al.,  “Context-encoding variational autoencoder for unsupervised anomaly detection,”  CoRR, 2018.
  * [9] I. Higgins et al.,  “beta-VAE: Learning basic visual concepts with a constrained variational framework,”  Int Conf Learn Repres, 2016.
  * [10] H. Fu et al.,  “Cyclical annealing schedule: A simple approach to mitigating KL vanishing,”  NAACL-HLT, vol. 1, 2019.
  * [11] A. L. Pinho et al.,  “Individual brain charting, a high-resolution fmri dataset for cognitive mapping,”  Scientific data, vol. 5, 2018.
  * [12] B. A. Landman et al.,  “Multi-parametric neuroimaging reproducibility: a 3T resource study,”  NeuroIm, 2011.
  * [13] Olivier Commowick et al.,  “Objective evaluation of multiple sclerosis lesion segmentation using a data management and processing infrastructure,”  Scientific reports, vol. 8, no. 1, 2018.
  * [14] Aaron Carass et al.,  “Longitudinal multiple sclerosis lesion segmentation: resource and challenge,”  NeuroImage, vol. 148, 2017.
  * [15] Hugo J. Kuijf et al.,  “Standardized assessment of automatic segmentation of white matter hyperintensities and results of the wmh segmentation challenge,”  IEEE transactions on medical imaging, vol. 38, no. 11, 2019.
  * [16] B. H. Menze et al.,  “The multimodal brain tumor image segmentation benchmark (BRATS),”  TMI, vol. 34, no. 10, 2014.
  * [17] N. J. Tustison et al.,  “N4itk: improved N3 bias correction,”  TMI, vol. 29, no. 6, 2010.
  * [18] F. Isensee et al.,  “Automated brain extraction of multisequence MRI using artificial neural networks,”  HBM, vol. 40, no. 17, 2019.
  * [19] D. Lachinov et al.,  “Glioma segmentation with cascaded UNet,”  in BrainLes. Springer, 2018.
  * [20] S. Valverde et al.,  “Improving automated multiple sclerosis lesion segmentation with a cascaded 3D convolutional neural network approach,”  NeuroIm, vol. 155, 2017.


#  _Real-World Textured Things_ : a Repository of Textured Models Generated
with Modern Photo-Reconstruction Tools

Andrea Maggiordomo<EMAIL_ADDRESS>Federico Ponchio
<EMAIL_ADDRESS>Paolo Cignoni<EMAIL_ADDRESS>Marco
Tarini<EMAIL_ADDRESS>University of Milan, Italy  ISTI - CNR, Italy

###### Abstract

We are witnessing a proliferation of textured 3D models captured from the real
world with automatic photo-reconstruction tools by people and professionals
without a proper technical background in computer graphics. Digital 3D models
of this class come with a unique set of characteristics and defects –
especially concerning their parametrization – setting them starkly apart from
3D models originating from other, more traditional, sources. We study this
class of 3D models by collecting a significant number of representatives and
quantitatively evaluating their quality according to several metrics. These
include a new _invariant_ metric we carefully design to assess the amount of
fragmentation of the UV map, which is one of the main weaknesses potentially
hindering the usability of these models. Our results back the widely shared
notion that models of this new class are still not fit for direct use in
downstream applications (such as videogames), and require challenging
processing steps. Regrettably, existing automatic geometry processing tools
are not always up to the task: for example, we verify that available tools for
UV optimization often fail due mesh inconsistencies, geometric and topological
noise, excessive resolution, or other factors; moreover, even when an output
is produced, it is rarely a significant improvement over the input (according
to the aforementioned measures). Therefore, we argue that further advancements
are required by the Computer Graphics and Geometry Processing research
communities specifically targeted at this class of models. Towards this goal,
we share the models we collected in this study in the form of a new public
repository, _Real-World Textured Things_ (RWTT), intended as a benchmark to
systematic field-test and compare future algorithms. RWTT consists of 568
carefully selected textured 3D models representative of all the main modern
off-the-shelf photo-reconstruction tools. The repository is web-browsable by
the metadata we collected during our experiments, and comes with a tool,
TexMetro, providing the same set of measures for generic UV mapped datasets.

###### keywords:

Benchmarks , Photo-reconstruction , 3D Acquisition , Quality Measures ,
Surface Parametrization , UV maps , Textures , Real-World Models

††journal: Computer Aided and Geometric Design

##  1 Introduction

In the last decade, many research and implementation efforts have culminated
in the development of algorithms, technologies, and suites that facilitate
creating and publishing 3D content by professionals and hobbyists, not
necessarily equipped with a technical background in computer graphics or
geometry processing. Among the most influential and exciting advancements,
photo-reconstruction techniques played a primary role, impacting a variety of
application fields such as artwork and architectural documentation,
environmental monitoring, data acquisition for manufacturing, 3D assets
creation. Commercial and free software solutions are now available to both
professionals and hobbyists, and used to generate accurate 3D digital copies
of real-world objects in an almost fully automatic way.

This democratization of 3D content creation determined the diffusion of
textured 3D digital models sharing certain specific characteristics, differing
both qualitatively and quantitatively from those of 3D representations
typically targeted – and assumed – by the large majority of academic works on
Computer Graphics and Geometry Processing. Consequently, there is now an
increased demand for robust techniques designed to process, manipulate, and
ameliorate this class of 3D model representation, and that further research
efforts are required to fill this gap.

##### Shared Characteristics of typical Real-World, Textured models

Digital 3D models acquired with photo-reconstruction techniques share a
combination of traits that are not found in any other class of 3D models (such
as 3D meshes digitally sculpted or manually designed by professional 3D
modelers, range-scanned models, or procedurally generated models).

The meshing is typically high-resolution but low quality, triangle-based, and
completely irregular. The frequent inaccuracies in the 3D reconstruction
result in geometrical noise and meshing defects. While many solutions have
been proposed to face these problems, (like for example Attene et al. [2013],
Ju [2004], Chu et al. [2019]) the proposed approaches focus on the geometrical
aspect and do not take into account the presence of an existing
parametrization. Even more importantly, geometry is but one aspect of the
description: photo-reconstructed meshes are enriched with one or multiple
high-resolution textures derived from the same set of photographs used to
reconstruct the shape; these are a central part of the description of the
object, as well as the object UV map, i.e. the parametrization defined to map
the texture data onto the object surface.

Figure 1: One model of the RWTT dataset, with its texture and UV map.

The UV maps of the 3D models present the most distinctive traits, directly
reflecting the way they have been constructed (see Sec. 1.1 below).
Specifically, they are atlas-based parametrizations where the atlases are
typically extremely fragmented and with irregularly shaped charts, to a much
larger extent than other categories of models (see for example Fig. 1).

This exacerbates by several times over all the numerous defects and costs
associated to texture seams of the UV maps, impacting any downstream
application. The reader is referred to Yuksel et al. [2019] for a more
comprehensive analysis of these issues. In summary, they include poor GPU
memory usage, due to a combination of packing inefficiency and the need to
replicate texels on a small band around each seam to avoid noticeable texture
“bleeding” artifacts; unavoidable minor rendering artifacts, appearing to are
due to the inconsistent bilinear sampling on the seams (even in the presence
of said texel duplication); inability to use MIP-mapping without incurring in
additional bleeding artifacts (unless the width of the texel duplication bands
is enlarged exponentially to the maximal MIP-map level); vertex duplications
in the data structure encoding the mesh, which is necessary to be able to
encode the seams in the first place (implying a small additional cost both in
memory and transform workload); consequent constraints on processing or
preprocessing performed over the mesh data structure, for example hindering to
some extent compression, streaming coarsening, construction of multi-
resolution structures (such as Cignoni et al. [2005]). The latter factors are
due to the necessity of the texture seams to be represented by (duplicated)
edges in the meshing.

##### Contributions

With this in mind, in this paper we offer the following contributions. We
present an analysis of the aforementioned class of 3D models and their
characterizing features, by collecting a large sample of representative models
and performing systematic measurements according to several criteria we
identify as relevant to geometry processing algorithms and computer graphics
applications in general, focusing our analysis on the UV maps as its presence
is the defining characteristics of these models.

To support our analysis, we design a new quality measure, the atlas
_crumbliness_ , that captures the severity of texture seams in a given UV map.
This new measure is compact, easy to compute, and has desirable properties
such as being invariant to UV scale and mesh subdivision.

Since our analysis shows that the quality of the parametrizations in our
sample data is low, we evaluate commercial, open-source and academic software
on the task of recomputing UV maps for the 3D models, and show how existing
algorithms and implementations typically either fail, are unable to improve
the quality of these maps or are not practical.

We make our dataset, Real-World Textured Things (RWTT), publicly available on
the web. The dataset has been curated and organized to facilitate batch-
processing, and is suited to test, benchmark, and challenge geometry
processing algorithms, both existing and future. Overall, RWTT features 568
models generated by real-world users of the aforementioned photo-
reconstruction technologies and released with permissive licenses, striving to
represent the wide variety of user categories, used photo-reconstruction tools
(including all existing off-the-shelf ones), application fields and styles.
Our aim is to provide a standard, realistic, and challenging testbed for
algorithms, able to indicate their usability in the real world. All models in
RWTT come pre-assessed according to the measures we collected, and are
browsable and searchable by them.

Finally, we provide the measuring tool we used to evaluate the models, as a
publicly available, stand-alone software application: TexMetro (Sec. 7).
TexMetro automatically grades user-provided datasets (in the same way dataset
of RWTT are graded), and is intended as a tool to help researchers and
practitioners assessing and comparing the effectiveness of future algorithms
and techniques designed to target this class of models. In this, we are
inspired by the impact that tools of this kind had in the geometry processing
community (Sec. 2.1).

###  1.1 Background in 3D Photo-Reconstruction

Photo-reconstruction techniques (Hartley and Zisserman [2003], Seitz et al.
[2006], Remondino et al. [2014]) generate a textured 3D mesh starting from a
set of high-resolution 2D pictures shot at the subject.

There are many possible variations, but the general principle can be
summarized as follows. First, Structure-From-Motion techniques are used to
estimate camera parameters: feature points are identified on each input image,
then matching features points are identified across images using similarity
measures. In a global adjustment phase, a set of camera positions is
estimated, together with 3D positions of feature points, to maximize
consistency, e.g. by minimize total back-projection errors or maximize color
consistency across images. Once the camera parameters have computed, Multi-
View-Stereo algorithms are employed to construct a dense point cloud. Finally,
the point cloud is processed to remove noise and outliers, and fused into one
polygonal mesh representation. The mesh representation is used to estimate the
occlusion and visibility of each pixel in the picture. A subset of the visible
pixels are then copied into one or more high-resolution textures, which
provide the majority of the visual detail of the final result. Textures are
sometimes post-processed to alleviate shadow artifacts, color jumps when
switching from one source image to another, or can optionally store additional
material and surface information (e.g. surface normals).

Each step of this chain constitutes a challenging problem, which has been
studied in separation or within complete systems. The research on each sub-
problem has recently matured enough so that a large number of off-the-shelf
commercial tools integrating implementation of all the used techniques, both
commercial and open-source, are now available CapturingReality [2014–2018],
Autodesk [2018], Agisoft LLC [2010–2018], 3DFlow [2014–2018], Pix4D SA [2011--
2018], Moulon et al. , Ranzuglia et al. [2013], Bentley [2014–2018], itSeez3D
[2019].

For our purposes, the most relevant part is the construction of the UV map.
Parametrization construction techniques traditionally employed in other
contexts are often not adequate to the task, in part because these methods
often lack the full automatism required in this scenario. Moreover, many
methods (automatic or user-assisted) are hindered by the characteristics of
the meshing (i.e. its large resolution and various defects). At the same time,
the presence of a set of images that features every part of the object surface
(by construction) unlocks UV map construction strategies that are available
only in this scenario. Namely, each mesh triangle can be assigned to one of
the photos using heuristics (based on estimated visibility or other factors).
Thus defines islands inside the photos which then just need to be packed into
(one or more) final textures. Note also that that mapping distortions,
traditionally of central importance in the effort of constructing a good
quality UV map for a given mesh, are not particularly relevant in this
context.

Differently from most other scenarios, where a UV map is constructed before
the texture is filled with the final signal, here the texture content (color
from images) _precedes_ the UV map construction. Because the source of the
texture colors are limited to the images, there is no gain in diminishing the
mapping distortions which can already present from these images to the 3D
surface. Therefore, instead of diminishing the distortion by optimizing a
suitable map, the optimization strategy can only consist in choosing the least
distorted input image to use as a source for a given triangle, which is (for
example) the image offering the most orthogonal view of that portion of the
surface (barring other considerations, such as occlusion, presence of
highlights, optimization of island shapes, and others).

These factors determine that the UV found on this important class of models
features unique and specific characteristics, motivating the present work.

##  2 Related Work

The practice of creating good, comprehensive, realistic, challenging
benchmarks is well recognized, and there is no need to state their importance
in promoting good practices and improving repeatability, comparability, and
validation of research. In this section we briefly summarize other dataset
contributions made to the geometry processing community and public
repositories of textured models. There are many geometric and computer vision
datasets available that target a large variety of tasks.

Datasets for machine learning generally feature large collections of objects,
and come with metadata to facilitate training and validation of learning-based
algorithms. The ModelNet benchmark [Wu et al., 2015] is a collection of more
than a hundred thousands CAD models, designed to train and benchmark object
recognition algorithms. ShapeNet [Chang et al., 2015] is a large scale dataset
of annotated 3D models. Datasets designed to benchmark recognition and
classification algorithms provide metadata about categories and annotations to
facilitate the training of the geometric networks. On a similar note, the ABC
dataset [Koch et al., 2019] is a massive dataset of one million CAD models
proposed to test the effectiveness of learning-based approaches to the
solution of common geometric problems such as normals computation.

Related to object scanning, there exists a large quantity of RGB-D datasets.
The review of those datasets is beyond our scope, and we refer the reader to
the excellent survey Firman [2016].

Scanning campaigns have produced popular model repositories, such as the
Digital Michelangelo Project [Levoy et al., 2000] and the Stanford Scanning
Repository [Levoy et al., ], whose models are known to the entire computer
graphics community. None of these models, however, have textures, and they
have been generated using 3D scanners; in contrast, the models featured in our
dataset are created with photo-reconstruction algorithms.

The Princeton Shape Benchmark [Shilane et al., 2004] is a large collection of
low-resolution models with very simple geometries. The Turk and Mullins [2003]
dataset focuses instead on high-resolution models. Both datasets are now
rather outdated.

More related datasets are the Thingi10K dataset [Zhou and Jacobson, 2016],
which features real-world objects and targets 3D scanning. [Choi et al., 2016]
is a large dataset of more than ten thousand object scans and the associated
RGB-D data. The collection is extensive and, like ours, features object
obtained from 3D reconstruction but again without textures.

SceneNN [Hua et al., 2016] and ScanNet [Dai et al., 2017] feature annotated
scenes reconstructed from RGB-D video sequences, and among the existing
datasets are, to our knowledge, the most similar to ours. However, they do not
contain models reconstructed with photogrammetric techniques, features mostly
indoor scenes and the 3D reconstructions are not particularly accurate.

As outlined above, geometric and computer vision datasets seldomly include
textures. Textured models can be found in asset stores such as the Unity Asset
store [Unity technologies, 2019], the Unreal Marketplace [Epic Games, 2004 --
2019], and online repositories such as TurboSquid [TurboSquid, 2019]. Models
from these sources however are typically non-free, sculpted, and textured by
hand by 3D artists.

In contrast to all of the above, Real-World Textured Things is a collection of
large, textured 3D models generated with a variety of photo-reconstruction
software tools, and features models published with permissive licenses to
allow use in research and other practical contexts.

###  2.1 Quality Measurement Tools

The idea of providing public tools intended to be used to measure specific
quality measures of user-provided models has several predecessors. Among the
most notable, is _Metro_ , a tool presented in Cignoni et al. [1998], which
computes and reports the Hausdorff distance between pairs of surfaces provided
as triangular meshes. Over the last two full decades, _Metro_ has helped to
assess and compare vast amounts of algorithms, e.g. for surface
simplification, remeshing, surface reconstruction, and other geometry
processing tasks – as reported by thousands of research papers [Google
Scholar, 2019]. As an additional benefit, the adoption of _Metro_ contributed
to uniform the way quality measures are defined and implemented. Overall,
_Metro_ has proven to be a tremendous aid in the research community striving
to invent new algorithms and solutions toward a given set of geometry
processing problems.

##  3 Dataset

As mentioned in the introduction, one of the motivations behind the creation
of the Real-World Textured Things dataset is to provide researchers and
developers a challenging testbed of textured 3D models indicative of the
actual models that can be encountered in practice. Rather than creating the
models in-house, we have collected publicly available models created and
released by professionals practitioners and hobbyists. This strategy has been
successfully employed by other authors (e.g. Zhou and Jacobson [2016], Koch et
al. [2019]) and has the important advantage of allowing us to include models
generated with various software packages, which is crucial to obtain a
representative sample of the 3D data that can be found in the real world. In
particular, it is reasonable to expect the emergence of some common traits
characterizing the geometry and textures of the models in the dataset
according to the implementation details and algorithmic choices of different
reconstruction pipelines.

At the time of writing, the dataset features 568 models created by 129 unique
authors using at least 18 different acquisition pipelines. Figure 2 reports a
taxonomy of the models featured in the RWTT dataset categorized according to
the reconstruction software used for their generation; this information was
obtained from the textual descriptions and tags available when the models were
retrieved.

We have collected models that were published across 6 years from 2013 to 2019
from three sources: SketchFab [Sketchfab, 2011--2018], the Smithsonian 3D
Program [Smithsonian Institution, 2013--2019] and the Zamani Project
[ruther2020]. SketchFab is a popular online repository for publishing and
sharing 3D content used by thousands of users and constantly growing; it
features a large collection of 3D models reconstructed from photographs and
conveniently exposes a web API to automatically retrieve metadata such as the
model author, description and tags among others. The Smithsonian 3D Program is
a digitization project from the Smithsonian Institution published on the web
and made available a selection of 3D reconstructions of artifacts and relics.
The Zamani Project is an ongoing effort to document important heritage and
archaeological sites in Africa by providing, among other material, high
resolution textured 3D models.

We have been careful to almost exclusively include models that have been
released with permissive licenses to allow anyone using the dataset to freely
share their results: all of the models from SketchFab are released under
various declinations of the Creative Commons 4.0 license; the models from the
Smithsonian 3D Project follow the “Smithsonian Terms of Use”, allowing fair
use as long as the original source is cited. We plan to keep the dataset
updated by adding new models in the future.

###  3.1 Dataset Curation

The dataset is organized as a collection of folders using a progressive naming
scheme to facilitate the automatic processing of the 3D models. Each model in
the dataset is assigned a progressive identifier $ID$ and is stored in a
folder named RWT$ID$.

For each model, we provide the following data:

  * 1.

The 3D model in OBJ format and its material file

  * 2.

The textures referenced by the material file

  * 3.

A JSON file with the provenance metadata of the model

  * 4.

A JSON file with geometric and parametrization measures

Although the original models and textures came in a variety of formats, we
have converted all of them to unique formats for convenience; in several
cases, manual fixing of the object files was required to restore correct
material libraries and texture image references. All the models are provided
as OBJ files, and the textures as PNG images. Whenever necessary, the
conversion only involved the file formats while the 3D and image data was left
untouched; the models archived in their original formats are available
separately. The textures we provide generally encode surface colors; however,
several models also come with high-resolution textures storing other signals
such as surface normals, roughness, specularity, and ambient occlusion.

Figure 2: Overview of the different software tools used to create the models
featured in Real-World Textured Things. There are at least 18 different
reconstruction pipelines represented in the dataset; here for compactness we
only report the most frequent occurrences.

Each model comes with its _metadata.json_ file reporting information such as
the name of the 3D model, the author, the license under which the model was
released, a textual description of the model provided by the author, the date
in which the model was published, a collection of tags chosen by the author,
and a resource identifier. For models obtained from SketchFab the resource
identifier is the URI assigned from SketchFab itself, whereas for the ones
obtained from the Smithsonian 3D Program the URI is the link to the web-page
showcasing the model. When available, we also include the software used to
generate the model and a semantic category we assigned to the models for
classification purposes.

Finally, we also include a _texmetro.json_ file generated with the TexMetro
tool. This JSON file contains information about the geometry,
parameterization, and textures of the models; a detailed description of the
TexMetro tool and its output is given in section 7.

##  4 Quality Measures for Textured 3D Meshes

The quality of a textured mesh can be assessed by several quantifiable
criteria, reflecting the resources that will be necessary to deploy the model
(e.g. store it, stream it, or render it) and the quality of the rendered
images, and thus the degree of usability of a given model in a specific
context.

Models in RWTT are scored according to a suitable set of such measures, which
is described in this Section. Different measures can impact downstream
applications in different ways. Collectively, the set of measures reveals the
peculiarity of the class of meshes we are focusing on, setting it apart from
meshes commonly encountered in contexts different from automated photo
reconstruction.

Our set of measures includes: general criteria (such as data size, surface
genus), UV map measures (concerning the parametrization), and defects (number
of occurrences of typical inconsistencies).

###  4.1 General Measures

Geometric and topological measures include the model resolution in terms of
faces, vertices, and edges, the surface area, the number and total length of
the mesh boundaries and the surface genus. Topological information is relevant
since many geometry processing algorithms require strict assumptions on the
topology of the input data. Texture resolution and number of textures, are
also meaningful measures, as these are all quantities that would make this
data impractical to be handled by algorithms that do not scale well with the
input size.

###  4.2 UV map Measures

The quality of the surface parametrizations and texture maps are the main
focus of our analysis and we list here the metrics we have adopted.

####  4.2.1 UV Occupancy

We define UV occupancy as the ratio of the number of texels strictly inside
any UV triangle over the total number of texels. A better UV occupancy results
in a higher texel density over the surface for a fixed texture resolution. or
lower GPU memory footprint for the same rendering quality. Occupancy is
affected by the quality of the atlas packing: tightly packed atlases waste
fewer texels between charts. Atlases that are highly fragmented have also a
worse occupancy.

####  4.2.2 Atlas Crumbliness and Solidity

Figure 3: Three models with their texture seams highlighted. While some
texture maps featured in our dataset have a relatively low chart count (left),
many others exhibit a surprisingly large number of texture seams unrelated to
the geometrical and topological complexity of the models (center and right).
Crumbliness and solidity values conveniently and compactly model the
fragmentation of the texture atlas.

As summarized in the introduction, the presence of texture seams is somewhat
detrimental to the usability of the dataset, for several different reasons.
While texture seams are, to some extent, necessary in any textured mesh, we
focus on the presence of redundant texture seams, due to the fragmentation of
the UV atlas, and independently from the complexity of the 3D reconstructed
shape (see figure 3). Therefore, we need to adopt a quality measure to assess
the level of fragmentation of an existing texture atlas.

Because there is no standard, we devise a new measure for this characteristic,
which is one defining feature of the class of 3D models of our dataset. We
considered several alternatives. We observe that the number of charts is not,
alone, a good descriptor, for two reasons: first, charts can have varying
sizes and boundaries of varying complexity; second, the chart may include any
amount of “non-disconnecting” cuts, i.e. cuts that do not split the chart. The
number of seams edges, or the ratio of seam edges to non-seam edges, would
also not be adequate: these measures would be affected by a regular
subdivision of the mesh, which does not affect the level of fragmentation.
Similarly, total length of seam edges would not be invariant to a rescaling of
UV.

Instead, we define a new measure, which we term _crumbliness_. Crumbliness is
the ratio of the total length of the perimeter of the atlas charts (summed
over all charts) over to the perimeter of a circle having the same area as the
summed area of all charts. Because the circle is the shape maximizing the area
for a given perimeter, this measure has 1 as the lower (and best) bound. In
formulas, if $a_{i}$ is the area of chart $i$ and $p_{i}$ is its perimeter,
the crumbliness of the atlas $C$ if given by

$C=\left(\sum_{i}{p_{i}}\right)\left({4\pi\sum_{i}{a_{i}}}\right)^{-\frac{1}{2}}$

Crumbliness is independent of the mesh resolution, invariant under element
subdivision, and scale-independent. As such, it can be used to compare atlases
of different models and for textures of varying sizes.

We define _atlas solidity_ as the inverse of crumbliness. Compatibly with many
other quality measures, atlas solidity has an upper bound of 1, and higher
values indicate a better quality.

##### Uniformity of texture sampling

A well-understood characteristic of any parametrization (the UV map) is the
distortion it introduces. In the ideal case, the mapping between the surface
and the texture preserves areas, angles, or lengths (implying both areas and
angles); the failure in preserving these features is measured by various
_distortion measures_. Conformal distortion reflects angles variation, area
distortion reflects area variation, and isometric distortion reflects length
variation. A multitude of different measures have been presented in the
literature (for each case), usually to minimize them during automatic UV map
construction (see Floater and Hormann [2005] for a survey, and, for more
recent examples, Nöll and Stricker [2011], Jiang et al. [2017], Poranne et al.
[2017] among many others).

In our context, which is texture mapping, distortion is relevant because it
directly determines how texture samples (texels) will be distributed on the
surface. Area distortion translates into nonuniform sampling density, and
conformal distortion translates into anisotropic sampling.

Sampling density can be hugely varying in models of RWTT, due to the different
relative resolution of the pictures used to reconstruct the models, and the
different distance between the acquiring camera and the captured object.

Instead of adopting any specific measure of area distortion, we choose to
measure directly its effect, by analyzing the sampling density of texels over
the mesh.

Figure 4: Visualization of the scalar field $s_{f}$ on a model affected by
uneven sampling frequency. Regions densely and sparsely sampled are colorized
in blue and red respectively, reflecting the amount of color information
encoded in the texture image. In this case, the model was clearly
reconstructed using close-ups of the statue with the pavement left in the
background.

To this end, we define a per-element scalar quantity $s_{f}$ that represents
the texture sampling resolution as the ratio between the local (per-element)
and global (per-mesh) change of scale from 3D to UV:

$s_{f}=\frac{A_{\text{UV}}(f)/A_{\text{3D}}(f)}{\sum_{f}A_{\text{UV}}(f)/\sum_{f}{A_{\text{3D}}(f)}}.$

Where $A_{\text{UV}}$ and $A_{3D}$ indicate element areas computed in UV and
3D space respectively. The local value of $s_{f}$ is close to 1 for uniform
sampling distributions, and approaches zero as the mapping resolution of $f$
gets smaller relative to the global resolution of the map and becomes greater
than one otherwise. Therefore the scalar field $s_{f}$ indicates, over a
model, where the color signal is undersampled and oversampled w.r.t. to the
average sampling density of one given model. Figure 4 shows an example of how
uneven sampling can impact the quality of a textured model when it is
rendered.

As an aggregate measure for a model, we provide the variance of $s_{f}$ over
the mesh (its average is 1.0 by construction). Small values reflect more
uniform sampling, and the minimal value 0 represents a perfectly uniform
sampling. Note that this is not necessarily desirable for models in our class;
in fact, even a vastly nonuniform sampling density can correctly reflect the
different semantic importance of certain parts of the model relative to others
(such as an object in focus vs. background) or different quantities of
available color data.

####  4.2.3 UV Conformal Distortion

Almost conformal mappings (i.e. UV maps exhibiting low conformal distortions)
are desirable because they imply a more isotropic texel sampling distribution
over the surface, which positively affects the quality of the signal
reconstruction and avoids texture stretching artifacts. Because this factor is
not captured by the sampling density, we measure conformal distortion
separately.

To this end, we adopt a formulation similar to Kharevych et al. [2006]: which
defines the quasi-conformal distortion ($qcd$) of a map as the ratio of the
smallest to largest singular value of the Jacobian of the mapping. This is the
inverse of what is proposed in Kharevych et al. [2006]; we opt for this
formulation so that, the maximal of value 1.0 corresponds to the best case (a
perfectly conformal map), consistently with most other quality measures in our
set.

####  4.2.4 Discrepancy in the Signal Reconstruction at Seam Edges

One specific defect linked to the presence of seams is the potential
discrepancy of the texture signal on the two sides of the texture seam, which
can be due to either mismatching texel values, and to an inconsistent bilinear
interpolation between texture values. This does not depend on the UV map
alone, but on the texture values as well. Therefore, we include a separate
measure to assess the extent of this problem for a given specific model.

To this end, we adopt the measure recently defined (for the purpose of
minimizing it) in Liu et al. [2017], which, for completeness, we describe in
the following.

Given an UV edge $e$, $e(t)$ denotes the linear combination of its endpoints,
with the parameter $t$ restricted to the $[0,1]$ interval. A (manifold) edge
$e$ of the 3D mesh is a seam edge if corresponds to two non coinciding UV
edges $e_{1}$, $e_{2}$. Let $f$ be the function returning the texture value
for a given UV coordinates, using bilinear interpolation of the four closest
texels. The reconstruction discrepancy at a seam edge $e$ is then defined as:

$D(e)=\int_{0}^{1}|f(e_{1}(t))-f(e_{2}(t))|\,dt.$

As a global discrepancy measure, we report the average discrepancy of all the
seam edges in a 3D model

$D(S)=\frac{\sum_{e\in S}\|e\|\cdot D(e)}{\sum_{e\in S}\|e\|}$

where $S$ denotes the set of seam edges, and the each seam edge contribution
is weighted by its length $\|e\|$ to capture the visibility and severity of
the resulting artifacts independently from the mesh subdivision. In our
implementation, edge discrepancies are estimated using discrete samples.

###  4.3 Defects

One defining characteristic of datasets of RWTT is the common occurrence of
defects and inconsistencies of specific nature. These issues are usually not
considered when geometry processing algorithms are developed on paper, which
often assume to be receiving sanitized input. Different issues can be
encountered in the geometric part, or the parametrization part.

Defects of geometry include extremely irregular triangulations, zero-area
triangles, topologically degenerate triangles (referring the same vertex more
than once), and unreferenced or duplicated primitives. Topological defects
include errors such as local loss of two-manifoldness, holes, and incomplete
meshes. Another potential source of issues is the presence of “topological
noise”, i.e., small handles unrelated to the ideal topology of the
reconstructed 3D shape (typically generated during the surface reconstruction
phase from noisy dense point clouds), which is revealed by incongruous genus
number with respect to the overall model shape.

Defects of the UV maps include loss of injectivity, resulting in the same
texture data to be erroneously mapped into multiple parts of the mesh.
Overlaps in the UV map can be either _local_ , when triangles flip in UV space
and change orientation relative to their neighbors, or _global_ , when the
boundary of one or several charts intersect in UV space. While they cause the
same problem, it is useful to distinguish between local and global overlaps,
because the strategies available to counter them are typically distinct.
Additionally, several dataset feature only partial maps, that is UV maps that
do not map a model surface in its entirety; this is likely due to insufficient
photographic data.

##  5 Measurements of RWTT Models: an Analysis

We applied our set of measures overall models of RWTT. In this section, we
offer an analysis of the results, which serves as a picture of the merits and
limitations of the currently available off-the-shelf photo-reconstruction
software.

###  5.1 Measures on Geometry

Figure 5: Geometric complexity of the 3D models found in our dataset. Models
generated from 3D reconstruction technology are typically stored at high
resolution to maximize accuracy and fidelity to their real-world counterpart.

Real-World Textured Things features models of many different sizes, ranging
from small models made of only a few thousand triangles to large models
composed of millions of faces. The histogram of the geometric complexity of
the dataset, expressed as the number of vertices, is reported in figure 5. On
average, our models tend to be fairly large compared to those found in similar
datasets: as an example, 37% of the RWTT models have more than 250K vertices
compared to less than 2% of the models found in Thingi10K [Zhou and Jacobson,
2016]. This is expected, as objects are typically reconstructed at high
resolution and then optimized for efficient use in downstream applications if
needed.

###  5.2 Texture Maps and Parametrizations

Texture Size Modern photo-reconstruction pipelines are able to generate
accurate digital reproductions from hundreds or thousands of high-resolution
images. Consequently, the majority of textures resolutions found in RWTT are
ranging from 4K upwards. Many models feature multiple textures images,
including non-squared ones.

Figure 6: Distribution of the RWTT textures occupancy. Many textures are not
efficiently packed, leading to a waste of GPU memory when the models are
rendered in real-time applications.

Texture Space Occupancy Figure 6 plots the histogram of texture occupancy
values found in RWTT and clearly shows how atlas packing is often sub-par,
leading to unnecessarily large memory requirements for texture signals sampled
at relatively low densities. This is particularly significant as GPU memory is
often a critical bottleneck for graphics applications.

Figure 7: Distribution of atlas Solidity for models featured in the RWTT
dataset. Many atlases exhibit high-to-extreme fragmentation: the model with
the highest chart count has 555252 charts.

Atlas Solidity Figure 7 plots the distribution of the atlas solidity across
the dataset, and clearly shows a bias towards low solidity values, indicating
a very frequent occurrence of texture maps affected by the presence of very
large amounts of seams.

The surprising fact, however, is that the degree of fragmentation witnessed in
the majority of the models is almost never justified by the topological and
geometrical complexity of the models we have collected (see figure 3). We
conjecture that this happens because of two distinct causes. First, this
happens because the textures are synthesized using large numbers of
photographs, with reconstruction tools optimizing for color fidelity and
attempting to store as much information as possible from the available images,
disregarding the creation of many atlas charts. Second, this may also happen
because of intrinsic limitations in existing automatic UV mapping technology,
which we investigate in section 6.

Figure 8: Distribution of atlas Conformality, where a lower value implies
larger angle deviations. Despite many fine-grain chart decompositions, many
parametrizations are still affected by non-negligible conformal distortion.

Conformality and Sampling Frequency Figure 8 reports the distribution of the
average quasi-conformal distortion values for our dataset. Unsurprisingly,
most of the texture maps exhibit good angle preservation; this is to be
expected given the high degree of fragmentation affecting many of the atlases.

Figure 9: Visualization of the UV scaling ranges present in the dataset. We
plot the 1st, 50th and 99th percentiles for the models sorted by UV scaling
range width, highlighting how a significant portion of the dataset exhibits
large variances in the mapping resolution.

A scatter plot of the 1st, 50th and 99th percentile values of the models UV
scaling, sorted by range from $P_{1}$ to $P_{99}$ is reported in figure 9,
showing how a significant amount of models exhibit a non-negligible spread in
the UV scaling values and, consequently, non-uniform texel density across the
surface.

Models with less uniform sampling frequency distributions were likely
generated from photographs that captured surface details at different scales.

Figure 10: Distribution of average seam color discrepancy (measured in RGB
space).

Seam Color Discrepancy The distribution of average color reconstruction
discrepancies at texture seams is reported in figure 10. Overall, our dataset
offers a fairly heterogeneous representation of color discrepancies in texture
seams. There are two explanations for this diversity. First, there can be
differences in the uniformity of the texture colors whereby models with more
varied textures are likely to exhibit more severe artifacts at texture seams
compared to models with surface colors that are more uniform. Second, there
can be differences in the way texture seams are handled by the reconstruction
tools during the texturing process: some tools may attempt to generate
textures that are smoother across seams by optimizing for consistency in the
texture data whereas others may prioritize fidelity to the photographic data.

###  5.3 Defects

Figure 11: Common defects found in Real-World Textured Things models

The ability to deal with arbitrary input meshes without requiring preemptive
cleaning is a key property of any robust implementation, whose importance is
being increasingly recognized as the emergence of more and more “real world”
dataset and algorithms specifically developed to deal with them. RWTT is a
further exemplification of this, and features meshes with all kinds of meshing
and geometric defects, as well as defects related to texture mapping such as
local and global overlap of UV coordinates, and models that are only partially
textured. An overview of the situation is reported in figure 11.

##  6 Recomputing UV maps

Given the number of issues these UV maps exhibit, an obvious question is
whether or not these results are peculiar to the tools that generated the
textures and parametrizations as part of the 3D reconstruction process, or if
there is an intrinsic limitation in the current technology regarding automatic
UV mapping of complex and high-resolution model. Therefore, we have decided to
evaluate the performance of existing state of the art UV mapping tools and see
if they can produce UV maps of better quality than the ones found in our
dataset.

We have performed experiments with commercial, open-source and academic
automatic UV mapping solutions. Maya [Autodesk, Inc, 2019] is a commercial 3D
modeling software that provides a simple automatic UV unwrapping procedure by
projecting 3D faces to a fixed set of orthogonal planes. Blender [Blender
Foundation, 2019] also has automatic projection-based UV unwrapping
functionality, but the set of projection planes is not fixed: instead, charts
are computed by grouping together faces with similar normal vectors, and the
projection plane of each chart is orthogonal to the average normal. Finally,
we also experimented with OptCuts [Li et al., 2018], which is a recent
academic work that automatically computes UV maps by alternating steps in
which the distortion of the mapping is reduced, and steps in which cuts are
added to the atlas charts to further reduce distortion.

For our experiments, we decided to set a time limit on the overall running
time of 20 minutes. Since these tools cannot conveniently produce multiple
textures (rather, UV channels) we have restricted the experiment only to
models with single textures. We also restricted the experiments to models
comprised of a unique connected component as the available implementation of
OptCut fails to initialize multi-component models. To meet the topological
requirements of OptCuts (the input must be two-manifold), we implemented a
simple preliminary cleaning step that removes non-manifold vertices and edges,
unreferenced vertices, and degenerate geometry. Moreover, since OptCuts is an
iterative algorithm we saved the result obtained after 20 minutes of running
time, rather than simply interrupting the task.

Figure 12: Performance of existing UV mapping tools on models generated with
photo-reconstruction technology.

The results of this experiment are reported in figure 12 as a 3D scatter plot
of the texture occupancy, mapping conformality, and atlas solidity of the
input data and the outputs produced by the three implementations used in the
experiment, with failure cases marked in red. Ideally, the scatter plots of
the outputs should move towards the top corner at coordinates (1, 1, 1).
Blender performs worse in terms of atlas solidity compared to the input UV
maps, without managing to improve under any of the remaining two metrics. Maya
also generates heavily fragmented output atlases, but succeeds in
significantly reduce the conformal distortion of the UV maps; note however
that the results are also affected by packing issues as shown by the poor
performance in terms of texture occupancy. OptCuts shows the best results
despite being often interrupted before completing the atlas optimization, but
it is affected by a very high failure rate (54%). OptCuts failures are either
due to numerical issues that prevent the correct initialization of the UV
configuration, or inability to reduce the surface genus before starting.
Finally, it is worth mentioning that, of the three implementations we tested,
OptCuts is the only one that guarantees the bijectivity of the mapping;
however, it is a much less efficient, practical and reliable solution.

Overall, this experimental evaluation shows that automatically computing UV
maps for this category of 3D models is intrinsically challenging regardless of
the method employed, and current technology is unable to provide satisfactory
results in most cases.

##  7 The TexMetro Tool

Alongside the dataset, we also release an open-source command line tool
(TexMetro) that can be used to compute all the measures described in Sec. 4 on
arbitrary user provided photo-reconstruction models. This tool can be used to
assess and compare existing or future alternative photo-reconstruction tools
(e.g. starting from the same set of images), as well as to assess the efficacy
of existing or future methods aimed to improve the quality of 3D models.

The UV-map, geometry, and connectivity of the an input models are analyzed.
The statistics in full are collected in a JSON file while only a relevant
subset of them is displayed on the standard output, consisting of salient
information about the mesh geometry, its parameterization and texture data. A
description of the JSON file structure can be found in the documentation from
the TexMetro repository.

A few of the measures are dependent on the texture resolution (in texels). To
compute these, TexMetro takes as input the actual texture image, or, simply,
its dimensions, provided as command-line arguments.

The TexMetro tool is open-source and is implemented using C++, Qt (for
interfacing), and OpenGL (to accelerate texture overlap detecting, which is
implemented by rasterization over off-screen buffers).

##  8 Conclusions

In this work, we introduce a dataset of textured 3D models generated with
automatic software, carefully selected and gathered from publicly available
sources, and enriched with relevant measures, including new ones tailored to
highlight the defects of this increasingly important (but often neglected)
class of 3D models; we also provide an open-source tool to collect measures
and statistics on the geometry, parametrization and texture maps of user-
provided models.

The analysis of our dataset reveals interesting traits common to these models.
In particular, it is extremely common to find heavily fragmented texture
atlases; this could be attributed to the fact that these parametrizations are
generated automatically and tailored to the available pictures. Moreover,
experimental evaluation of current state of the art automatic UV mapping
tools, both commercial and academic, suggests that the issues that affect the
UV maps of our dataset are not peculiar to the implementations of the software
used to reconstruct the mesh; rather, this seems to be a limitation of current
automatic UV mapping tools and algorithms that either fail to produce better
results or are not practical and robust enough to reliably undertake the task.

Our aspiration is that this dataset and tool will help researchers and
developers in their efforts to improve the quality of photo-reconstruction
algorithms and implementations, as well as new tools targeted at post-
processing the resulting models.

## Acknowledgements

This research was partially funded by the Italian PRIN project DSurf (grant
no. 2015B8TRFM) and by the EU project PARTHENOS: Pooling Activities, Resources
and Tools for Heritage E-Research Networking, Optimization and Synergies
(grant no. 2015-2019 H2020-INFRADEV-1-2014).

## References

  * 3DFlow [2014–2018] 3DFlow, 2014–2018.  Zephyr.  https://www.3dflow.net/3df-zephyr-pro-3d-models-from-photos/.
  * Agisoft LLC [2010–2018] Agisoft LLC, 2010–2018.  Photoscan: photogrammetric processing of digital images and 3d spatial data generation.  http://www.agisoft.com.
  * Attene et al. [2013] Attene, M., Campen, M., Kobbelt, L., 2013.  Polygon mesh repairing: An application perspective.  ACM Comput. Surv. 45.  URL: https://doi.org/10.1145/2431211.2431214, doi:10.1145/2431211.2431214.
  * Autodesk [2018] Autodesk, I., 2018.  ReCap: Create accurate 3D models with reality capture.  https://www.autodesk.com/products/recap/overview.
  * Autodesk, Inc [2019] Autodesk, Inc, 2019.  Autodesk Maya 2019.  https://www.autodesk.com/products/maya/overview.
  * Bentley [2014–2018] Bentley, 2014–2018.  ContextCapture.  https://www.bentley.com/en/products/brands/contextcapture/.
  * Blender Foundation [2019] Blender Foundation, 2019.  Blender - Free and Open 3D Creation Software.  URL: http://www.blender.org.
  * CapturingReality [2014–2018] CapturingReality, 2014–2018.  RealityCapture: state-of-the-art all-in-one photogrammetry software solution.  https://www.capturingreality.com/Product/.
  * Chang et al. [2015] Chang, A.X., Funkhouser, T., Guibas, L., Hanrahan, P., Huang, Q., Li, Z., Savarese, S., Savva, M., Song, S., Su, H., Xiao, J., Yi, L., Yu, F., 2015\.  ShapeNet: An Information-Rich 3D Model Repository.  Technical Report arXiv:1512.03012 [cs.GR]. Stanford University — Princeton University — Toyota Technological Institute at Chicago.
  * Choi et al. [2016] Choi, S., Zhou, Q.Y., Miller, S., Koltun, V., 2016\.  A large dataset of object scans.  arXiv:1602.02481 .
  * Chu et al. [2019] Chu, L., Pan, H., Liu, Y., Wang, W., 2019.  Repairing man-made meshes via visual driven global optimization with minimum intrusion.  ACM Transactions on Graphics (TOG) 38, 158.
  * Cignoni et al. [2005] Cignoni, P., Ganovelli, F., Gobbetti, E., Marton, F., Ponchio, F., Scopigno, R., 2005\.  Batched multi triangulation.  URL: http://vcg.isti.cnr.it/Publications/2005/CGGMPS05.
  * Cignoni et al. [1998] Cignoni, P., Rocchini, C., Scopigno, R., 1998.  Metro: Measuring error on simplified surfaces.  Computer Graphics Forum 17, 167–174.  URL: https://onlinelibrary.wiley.com/doi/abs/10.1111/1467-8659.00236, doi:10.1111/1467-8659.00236, arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/1467-8659.00236.
  * Dai et al. [2017] Dai, A., Chang, A.X., Savva, M., Halber, M., Funkhouser, T., Nießner, M., 2017\.  Scannet: Richly-annotated 3d reconstructions of indoor scenes, in: Proc. Computer Vision and Pattern Recognition (CVPR), IEEE.
  * Epic Games [2004 -- 2019] Epic Games, 2004 -- 2019.  Unreal engine 4 marketplace.  https://www.unrealengine.com/marketplace/.
  * Firman [2016] Firman, M., 2016.  RGBD Datasets: Past, Present and Future, in: CVPR Workshop on Large Scale 3D Data: Acquisition, Modelling and Analysis.
  * Floater and Hormann [2005] Floater, M.S., Hormann, K., 2005\.  Surface parameterization: a tutorial and survey, in: Advances in multiresolution for geometric modelling. Springer, pp. 157--186.
  * Google Scholar [2019] Google Scholar, 2019.  Cignoni et al. [1998] citation page.  https://scholar.google.it/scholar?cites=4029217030792744566.
  * Hartley and Zisserman [2003] Hartley, R., Zisserman, A., 2003\.  Multiple view geometry in computer vision.  Cambridge university press.
  * Hua et al. [2016] Hua, B.S., Pham, Q.H., Nguyen, D.T., Tran, M.K., Yu, L.F., Yeung, S.K., 2016\.  Scenenn: A scene meshes dataset with annotations, in: International Conference on 3D Vision (3DV).
  * itSeez3D [2019] itSeez3D, 2019.  itSeez3D -- #1 Mobile 3D scanner for iPad.  https://itseez3d.com/.
  * Jiang et al. [2017] Jiang, Z., Schaefer, S., Panozzo, D., 2017.  Simplicial complex augmentation framework for bijective maps.  ACM Trans. Graph. 36, 186:1--186:9.  URL: http://doi.acm.org/10.1145/3130800.3130895, doi:10.1145/3130800.3130895.
  * Ju [2004] Ju, T., 2004.  Robust repair of polygonal models.  ACM Transactions on Graphics (TOG) 23, 888--895.
  * Kharevych et al. [2006] Kharevych, L., Springborn, B., Schröder, P., 2006.  Discrete conformal mappings via circle patterns.  ACM Trans. Graph. 25, 412--438.  URL: http://doi.acm.org/10.1145/1138450.1138461, doi:10.1145/1138450.1138461.
  * Koch et al. [2019] Koch, S., Matveev, A., Jiang, Z., Williams, F., Artemov, A., Burnaev, E., Alexa, M., Zorin, D., Panozzo, D., 2019.  Abc: A big cad model dataset for geometric deep learning, in: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
  * [26] Levoy, M., Gerth, J., Curless, B., Pull, K., .
  * Levoy et al. [2000] Levoy, M., Pulli, K., Curless, B., Rusinkiewicz, S., Koller, D., Pereira, L., Ginzton, M., Anderson, S., Davis, J., Ginsberg, J., et al., 2000\.  The digital michelangelo project: 3d scanning of large statues, in: Proceedings of the 27th annual conference on Computer graphics and interactive techniques, ACM Press/Addison-Wesley Publishing Co.. pp. 131--144.
  * Li et al. [2018] Li, M., Kaufman, D.M., Kim, V.G., Solomon, J., Sheffer, A., 2018.  Optcuts: Joint optimization of surface cuts and parameterization.  ACM Transactions on Graphics 37.  doi:http://dx.doi.org/10.1145/3272127.3275042.
  * Liu et al. [2017] Liu, S., Ferguson, Z., Jacobson, A., Gingold, Y., 2017\.  Seamless: Seam erasure and seam-aware decoupling of shape from mesh resolution.  ACM Transactions on Graphics (TOG) 36, 216:1--216:15.  doi:10.1145/3130800.3130897.
  * [30] Moulon, P., Monasse, P., Marlet, R., Others, .  Openmvg. an open multiple view geometry library.  https://github.com/openMVG/openMVG.
  * Nöll and Stricker [2011] Nöll, T., Stricker, D., 2011\.  Efficient packing of arbitrarily shaped charts for automatic texture atlas generation, in: Proceedings of the Twenty-second Eurographics Conference on Rendering, Eurographics Association, Aire-la-Ville, Switzerland, Switzerland. pp. 1309--1317.  URL: http://dx.doi.org/10.1111/j.1467-8659.2011.01990.x, doi:10.1111/j.1467-8659.2011.01990.x.
  * Pix4D SA [2011--2018] Pix4D SA, 2011--2018.  Pix4dmodel.  https://pix4d.com/product/pix4dmodel.
  * Poranne et al. [2017] Poranne, R., Tarini, M., Huber, S., Panozzo, D., Sorkine-Hornung, O., 2017.  Autocuts: simultaneous distortion and cut optimization for uv mapping.  ACM Transactions on Graphics (TOG) 36, 215.
  * Ranzuglia et al. [2013] Ranzuglia, G., Callieri, M., Dellepiane, M., Cignoni, P., Scopigno, R., 2013.  Meshlab as a complete tool for the integration of photos and color with high resolution 3d geometry data, in: CAA 2012 Conference Proceedings, Pallas Publications - Amsterdam University Press (AUP). pp. 406--416.  URL: http://vcg.isti.cnr.it/Publications/2013/RCDCS13.
  * Remondino et al. [2014] Remondino, F., Spera, M.G., Nocerino, E., Menna, F., Nex, F., 2014.  State of the art in high density image matching.  The Photogrammetric Record 29, 144--166.
  * Seitz et al. [2006] Seitz, S.M., Curless, B., Diebel, J., Scharstein, D., Szeliski, R., 2006.  A comparison and evaluation of multi-view stereo reconstruction algorithms, in: Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 1, IEEE Computer Society, Washington, DC, USA. pp. 519--528.  URL: http://dx.doi.org/10.1109/CVPR.2006.19, doi:10.1109/CVPR.2006.19.
  * Shilane et al. [2004] Shilane, P., Min, P., Kazhdan, M., Funkhouser, T., 2004\.  The princeton shape benchmark, in: Proceedings Shape Modeling Applications, 2004., IEEE. pp. 167--178.
  * Sketchfab [2011--2018] Sketchfab, 2011--2018.  Sketchfab - your 3d content on web, mobile, ar, and vr.  https://sketchfab.com/.
  * Smithsonian Institution [2013--2019] Smithsonian Institution, 2013--2019.  Smithsonian 3D program.  https://3d.si.edu/.
  * TurboSquid [2019] TurboSquid, 2019.  Turbosquid: 3d models for professionals.  https://www.turbosquid.com/.
  * Turk and Mullins [2003] Turk, G., Mullins, B., 2003\.  Large geometric models archive.  https://www.cc.gatech.edu/projects/large_models/.
  * Unity technologies [2019] Unity technologies, 2019.  Unity asset store.  https://assetstore.unity.com/.
  * Wu et al. [2015] Wu, Z., Song, S., Khosla, A., Yu, F., Zhang, L., Tang, X., Xiao, J., 2015.  3d shapenets: A deep representation for volumetric shapes, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1912--1920.
  * Yuksel et al. [2019] Yuksel, C., Lefebvre, S., Tarini, M., 2019.  Rethinking texture mapping.  Computer Graphics Forum 38, 535--551.  URL: https://onlinelibrary.wiley.com/doi/abs/10.1111/cgf.13656, doi:10.1111/cgf.13656, arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/cgf.13656.
  * Zhou and Jacobson [2016] Zhou, Q., Jacobson, A., 2016\.  Thingi10k: A dataset of 10,000 3d-printing models.  arXiv preprint arXiv:1605.04797 .