# FaceLift: A transparent deep learning framework to beautify urban scenes

Sagar Joglekar
Nokia Bell Labs
Cambridge, UK
<EMAIL_ADDRESS>
&Daniele Quercia
Nokia Bell Labs
Cambridge, UK
<EMAIL_ADDRESS>
&Miriam Redi
Nokia Bell Labs
Cambridge, UK
&Luca Maria Aiello
Nokia Bell Labs
Cambridge, UK
&Tobias Kauer
Nokia Bell Labs
Cambridge, UK
&Nishanth Sastry
Department of Informatics
King’ College
London, UK

###### Abstract

In the area of computer vision, deep learning techniques have recently been
used to predict whether urban scenes are likely to be considered beautiful: it
turns out that these techniques are able to make accurate predictions. Yet
they fall short when it comes to generating actionable insights for urban
design. To support urban interventions, one needs to go beyond _predicting_
beauty, and tackle the challenge of _recreating_ beauty. Unfortunately, deep
learning techniques have not been designed with that challenge in mind. Given
their “black-box nature”, these models cannot be directly used to explain why
a particular urban scene is deemed to be beautiful. To partly fix that, we
propose a deep learning framework (which we name
FaceLift111http://goodcitylife.org/facelift/) that is able to both _beautify_
existing urban scenes (Google Street views) and _explain_ which urban elements
make those transformed scenes beautiful. To quantitatively evaluate our
framework, we cannot resort to any existing metric (as the research problem at
hand has never been tackled before) and need to formulate new ones. These new
metrics should ideally capture the presence (or absence) of elements that make
urban spaces great. Upon a review of the urban planning literature, we
identify five main metrics: walkability, green spaces, openness, landmarks and
visual complexity. We find that, across all the five metrics, the beautified
scenes meet the expectations set by the literature on what great spaces tend
to be made of. This result is further confirmed by a 20-participant expert
survey in which FaceLift have been found to be effective in promoting citizen
participation. All this suggests that, in the future, as our framework’s
components are further researched and become better and more sophisticated, it
is not hard to imagine technologies that will be able to accurately and
efficiently support architects and planners in the design of the spaces we
intuitively love.

_K_ eywords Deep learning, Urban Design, Generative models, Urban beauty,
Explainable models

##  1 Introduction

Whether a street is considered beautiful is subjective, yet research has shown
that there are specific urban elements that are universally considered
beautiful: from greenery, to small streets, to memorable spaces [1, 2, 3].
These elements are those that contribute to the creation of what the urban
sociologist Jane Jacobs called ‘urban vitality’ [4].

Given that, it comes as no surprise that computer vision techniques can
automatically analyse pictures of urban scenes and accurately determine the
extent to which these scenes are considered, _on average_ , beautiful. Deep
learning has greatly contributed to increase these techniques’ accuracy [5].

However, urban planners and architects are interested in urban interventions
and, as such, they would welcome machine learning technologies that help them
recreate beauty in urban design [6] rather than simply predicting beauty
scores. As we shall see in Section 2, deep learning, by itself, is not fit for
purpose. It is not meant to recreate beautiful scenes, not least because it
cannot provide any explanation on why a scene is deemed beautiful, or which
urban elements are predictors of beauty.

To partly fix that, we propose a deep learning framework (which we name
FaceLift) that is able to both _generate_ a beautiful scene (or, better,
_beautify_ an existing one) and _explain_ which parts make that scene
beautiful. Our work contributes to the field of urban informatics, an
interdisciplinary area of research that studies practices and experiences
across urban contexts and creates new digital tools to improve those
experiences [7, 8]. Specifically, we make two main contributions:

  * •

We propose a deep learning framework that is able to learn whether a
particular set of Google Street Views (urban scenes) are beautiful or not, and
based on that training, the framework is then able to both _beautify_ existing
views and _explain_ which urban elements make them beautiful (Section 3).

  * •

We quantitatively evaluate whether the framework is able to actually produce
beautified scenes (Section 4). We do so by proposing a family of five urban
design metrics that we have formulated based on a thorough review of the
literature in urban planning. For all these five metrics, the framework passes
with flying colours: with minimal interventions, beautified scenes are twice
as walkable as the original ones, for example. Also, after building an
interactive tool with “FaceLifted” scenes in Boston and presenting it to
twenty experts in architecture, we found that the majority of them agreed on
three main areas of our work’s impact: decision making, participatory
urbanism, and the promotion of restorative spaces.

##  2 Related Work

Previous work has focused on: collecting ground truth data about how people
perceive urban spaces; predicting urban qualities from visual data; and
generating synthetic images that enhance a given quality (e.g., beauty).

Perception of physical spaces. From Jane Jacobs’s seminal work on urban
vitality [4] to Christopher Alexander’s cataloging of typical “patterns” of
good urban design [1], there has been a continuous effort to understand what
makes our cities livable and enjoyable. In the fields of psychology,
environmental design and behavioral sciences, research has studied the
relationship between urban aesthetics [9] and a variety of objective measures
(e.g., scene complexity [10], presence of nature [11]) and subjective ones
(e.g., people’s affective responses [12]).

Ground truth of urban perceptions. So far, the most detailed studies of
perceptions of urban environments and their visual appearance have relied on
personal interviews and observation: some researchers relied on annotations of
video recordings by experts [13], while others have used participant ratings
of simulated (rather than existing) street scenes [14]. The Web has recently
been used to survey a large number of individuals. Place Pulse is a website
that asks a series of binary perception questions (such as ‘Which place looks
safer [between the two]?’) across a large number of geo-tagged images [3]. In
a similar way, Quercia _et al._ collected pairwise judgments about the extent
to which urban scenes are considered quiet, beautiful and happy [2] to then
recommend pleasant paths in the city [15]. They were then able to analyze the
scenes together with their ratings using image-processing tools, and found
that the amount of greenery in any given scene was associated with all three
attributes and that cars and fortress-like buildings were associated with
sadness. Taken all together, their results pointed in the same direction:
urban elements that hinder social interactions were undesirable, while
elements that increase interactions were the ones that should be integrated by
urban planners to retrofit cities for happiness. Urban perceptions translate
in concrete outcomes. Based on 3.3k self-reported survey responses, Ball et
al. [16] found that urban scenes that are aesthetically beautiful not only are
visually pleasurable but also promote walkability. Similar findings were
obtained by Giles et al. [17].

Deep learning and the city. Computer vision techniques have increasingly
become more sophisticated. Deep learning techniques, in particular, have been
recently used to accurately predict urban beauty [5, 18], urban change [19],
and even crime [20, 21]. Recent works have also showed the utility of deep
learning techniques in predicting house prices from urban frontages [22], and
from a combination of satellite data and street view images [23].

Generative models. Since the introduction of Generative Adversarial Networks
(GANs) [24], deep learning has been used not only to analyze existing images
but also to generate new ones altogether. This family of deep networks has
evolved into various forms, from super resolution image generators [25] to
fine-grained in-painting technologies [26]. Recent approaches have been used
to generate images conditioned on specific visual attributes [27], and these
images range from faces [28] to people [29]. In a similar vein, Nguyen _et
al._ [30] used generative networks to create a natural-looking image that
maximizes a specific neuron (the beauty neuron). In theory, the resulting
image is the one that “best activates” the neuron under consideration. In
practice, it is still a synthetic template that needs further processing to
look realistic. Finally, with the recent advancement in Augmented Reality, the
application of GANs to generate urban objects in simulated urban scenes have
also been attempted [31].

To sum up, a lot of work has gone into collecting ground truth data about how
people tend to perceive urban spaces, and into building accurate predictions
models of urban qualities. Yet little work has gone into models that generate
realistic urban scenes that maximize a specific property and that offer human-
interpretable explanations of what they generate.

##  3 FaceLift Framework

Symbol |  Meaning
---|---
$I_{i}$ |  Original urban scene
$Y$ |  Set of annotation classes for urban scenes (e.g., beautiful, ugly)
$y_{i}$ |  Annotation class in $Y$ (e.g., beautiful)
$\hat{I_{j}}$ |  Template scene (synthetic image)
$I^{\prime}$ |  Target Image
$C$ |  Beauty Classifier

Table 1: Notations Figure 1: An illustration of the FaceLift framework. The
training data of urban scenes with beautiful and ugly labels is fed into the
FaceLift algorithms (GAN and CNN). These transform the original image into its
beautified version. The two images - original and beautified - are compared in
terms of urban elements that have been added or removed.

The main goal of FaceLift is to beautify an existing urban scene and explain
its beautification. To meet that goal, it performs five steps:

Step 1 Curating urban scenes. Deep learning systems need considerable amounts
of training data. Our initial set of data is limited and to augment it, we
develop a new way of curating and augmenting the number of annotated images.

Step 2 Training a beauty classifier. We design and train a deep learning model
that is able to distinguish beautiful urban scenes from non-beautiful ones.

Step 3 Generating a synthetic beautified scene. Based on our classifier’s
learned representation of beauty, we train a generative model that is able to
beautify an urban scene in input.

Step 4 Retrieving a realistic beautified scene. The generated image has a
“synthetic look” - it does not look realistic. To fix that, we retrieve the
image in the set of curated urban scenes most similar to the generated one. We
use the euclidean distance to compute similarity.

Step 5 Identifying the urban elements characterizing the beautified scene. In
the final step, the framework explains the changes introduced in the
transformation process by comparing the beautified scene to the original one
in terms of addition and removal of specific urban elements.

### Step 1 Curating Urban Scenes

To begin with, we need highly curated training data with labels reflecting
urban beauty. We start with the Place Pulse dataset that contains a set of
110k Google street view images from 56 major cities across 28 countries around
the world [5]. The pictures were labeled by volunteers through an ad-hoc
crowdsourcing website222http://pulse.media.mit.edu. Volunteers were shown
random pairs of images and asked to select which scene looked more beautiful,
safe, lively, boring, wealthy, and depressing. At the time of writing, 1.2
million pairwise comparisons were generated by 82k online volunteers from 162
countries, with a good mix of people residing in both developed and developing
countries. To our knowledge, no independent systematic analysis of the biases
of Place Pulse has been conducted yet. However, it is reasonable to expect
that representation biases are minimized by the substantial size of the
dataset, the wide variety of places represented, and the diversity of gender,
racial, and cultural backgrounds of the raters. We focus only on those scenes
that are labelled in terms of beauty (the focus of this study) and that have
at least three judgments. This leave us with roughly 20,000 scenes. To
transform judgments into beauty scores, we use the TrueSkill algorithm [32],
which gives us a way of partitioning the scenes into two sets (Figure 2): one
containing beautiful scenes, and the other containing ugly scenes. The
resulting set of scenes is too small for training any deep learning model
without avoiding over-fitting though. As such, we need to augment such a set.

We do so in two ways. First, we feed each scene’s location into the Google
Streetview API to obtain the snapshots of the same location at different
camera angles (i.e., at
$\theta\in{-30^{\circ},-15^{\circ},15^{\circ},30^{\circ}}$). Yet the resulting
dataset is still too small for robust training. So we again feed each scene’s
location into the Google Streetview API, but this time we do so to obtain
scenes at increasing distance $d\in\\{10,20,40,60\\}$ meters. A real example
of one such augmentation instance can be seen in Figure 3. This expands our
set of scenes, but does so at the price of introducing scenes whose beauty
scores have little to do with the original one’s. To fix that, we take only
the scenes that are _similar_ to the original one (we call this way of
augmenting “conservative translation”). Two scenes are considered similar if
the similarity of their two feature vectors (derived from the FC7 layer of
PlacesNet [33]) is above a certain threshold. In a conservative fashion, we
choose that threshold to be the median similarity between rotated and original
scenes.

To make sure this additional augmentation has not introduced any unwanted
noise, we consider two sets of scenes: one containing those that have been
taken during this last step, i.e., the one with high similarity to the
original scenes (_taken-set_), and the other containing those that have been
filtered away (_filtered-set_). Each scene is represented with the five most
confident scene labels extracted by PlacesNet [33]. We then aggregate labels
at set level by computing each label’s frequency $fr$ on the _taken-set_ and
that on the _filtered-set_. Finally, we characterize each label’s propensity
to be correctly augmented as:
$\textrm{prone}(label)=fr(label,\textrm{\emph{taken-
set}})-fr(label,\textrm{\emph{filtered-set}}).$ This reflects the extent to
which a scene with a given label is prone to be augmented or not. From Figure
4, we find that, as one would expect, scenes that contain highways, fields and
bridges can be augmented at increasing distances while still showing
resemblances to the original scene; by contrast, scenes that contain gardens,
residential neighborhoods, plazas, and skyscrapers cannot be easily augmented,
as they are often found in high density parts of the city in which diversity
within short distances is likely to be experienced.

Figure 2: Frequency distribution of beauty scores. The red and green lines
represent the thresholds below and above which images are considered ugly and
beautiful. Conservatively, images in between are discarded.

(a)

(b)

Figure 3: Two types of augmentation: (a) _rotation_ of the Street Views camera; and (b) _translation_ of scenes at increasing distances. Figure 4: The types of scene that have greater propensity to be correctly augmented with similar scenes at increasing distances. Augmentation | Accuracy (Percentage)
---|---
None | 63
Rotation | 68
Rotation + Translation | 64
Rotation + Conservative Translation | 73.5
Table 2: Percentage accuracy for our beauty classifier trained on sets of
urban scenes that have been differently augmented.

### Step 2 Training a beauty classifier

Having this highly curated set of labeled urban scenes, we are now ready to
train a classifier. We choose the CaffeNet architecture as our classifier $C$.
This is a modified version of AlexNet [34, 35]. Its Softmax layer classifies
the input image into one of two classes of beautiful(1) and ugly(0).

Having $C$ at hand, we now turn to training it. The training is done on a 70%
split of each of the training sets, and the testing on the remaining 30%. The
training sets are constructed as increasingly augmented sets of data. We start
from our 20k images and progressively augment them with the snapshots obtained
with the 5-angle camera rotations, and then with the exploration of scenes at
increasing distance $d\in\\{10,20,40,60\\}$ meters. The idea behind this data
augmentation is that the model’s accuracy should increase with increasing
levels of augmentation. Indeed it does (Table 2): it goes from 63% on the set
of original scenes to a value as high as 73.5% on the set of fully augmented
scenes, which is a notable increase in accuracy for this type of
classification tasks (the state-of-the-art classifier was reported to have an
accuracy of 70% [5]).

### Step 3 Generating a synthetic beautified scene

Figure 5: Architecture of the synthetic beauty generator. This consists of a generator of synthetic scenes concatenated with a beauty classifier. The green block is the beauty maximized template $\hat{I_{j}}$, which is subject to forward and backward passes (red arrow) when optimizing for beauty. Original | Generated
---|---
|
|
|
Table 3: Examples of our generator’s outputs. The original scenes and the
generated ones are shown side by side.

Having this trained classifier at hand, we can then build a generator of
synthetic beautified scenes. To build such a generator, we retrain the GAN
described by Dosovitskiy and Brox [36] on our curated urban scene dataset.
This network is trained by maximizing the confusion for the discriminator
between the generated images $G(f)$ and the original ones $I_{f}$ [24]. Some
examples of the output of this generator can be seen in Table 3. This table
shows the comparison between the original ($I_{f}$) and the GAN’s generated
image ($G(f)$) side by side. The resulting generator is concatenated with our
beauty classifier (Figure 5). As a result, given the two classes of ugly
$y_{i}$ and beautiful $y_{j}$, the end-to-end model transforms any original
scene $I_{i}$ of class $y_{i}$ (e.g., ugly scene) into template scene
$\hat{I_{j}}$ that maximizes class $y_{j}$ (e.g., beautified template scene).

More specifically, given an input image $I_{i}$ known to be of class $y_{i}$
(e.g., ugly), our technique outputs $\hat{I_{j}}$, which is a more beautiful
version of it (e.g., $I_{i}$ is morphed towards the average representation of
a beautiful scene) while preserving the way $I_{i}$ looks. The technique does
so using the “Deep Generator Network for Activation Maximization” (_DGN-AM_)
[30]. Given an input image $I_{i}$, _DGN-AM_ iteratively re-calculates the
color of $I_{i}$’s pixels in a way the output image $\hat{I_{j}}$ both
maximizes the activation of neuron $y_{j}$ (e.g., the “beauty neuron”) and
looks “photo realistic”. This is equivalent to finding the feature vector $f$
that maximizes the following expression:

$\hat{I_{j}}=G(f):\underset{f}{\arg\max}(C_{j}(G(f))-\lambda||f||)$ (1)

where:

  * •

$G(f)$ is the image synthetically generated from the candidate feature vector
$f$;

  * •

$C_{j}(G(f))$ is the activation value of neuron $y_{j}$ in the scene
classifier $C$ (the value to be maximized);

  * •

$\lambda$ is a $L_{2}$ regularization term.

Here the initialization of $f$ is key. If $f$ were to be initialized with
random noise, the resulting $G(f)$ would be the average representation of
category $y_{j}$ (of, e.g., beauty). Instead, $f$ is initialized with the
feature vector corresponding to $I_{i}$ and, as such, the resulting maximized
$G(f)$ is $I_{i}$’s version “morphed to become more beautiful”.

The input image is also key. It makes little sense to beautify an already
beautiful image, not least because such beautification process would result in
a saturated template $\hat{I_{j}}$. For this reason, to generate an image that
maximizes the beauty neuron in the classifier $C$, we restrict the input set
to ugly scenes. We do the opposite when maximizing the ugly neuron.

### Step 4 Returning a realistic beautified scene

We now have template scene $\hat{I_{j}}$ (which is a synthetic beautified
version of original scene $I_{i}$) and need to retrieve a realistic looking
version of it. We do so by: _i)_ representing each of the scenes in our
augmented set plus the synthetic image $\hat{I_{j}}$ as a 4096 dimensional
feature vector derived from the FC7 layer of the PlacesNet [33]; _ii)_
computing the Euclidean distance (as $L_{2}$ Norm) between $\hat{I_{j}}$’s
feature vector and each original scene’s feature vector; and _iii)_ selecting
the scene in our augmented set most similar (smaller distance) to
$\hat{I_{j}}$. This results into the selection of the beautified scene
$I_{j}$.

### Step 5 Identifying characterizing urban elements

Since original scene $I_{i}$ and beautified scene $I_{j}$ are real scenes with
the same structural characteristics (e.g., point of view, layout), we can
easily compare them in terms of presence or absence of urban elements
extracted by computer vision tools such as SegNet and PlacesNet. That is, we
can determine how the original scene and its beautified version differ in
terms of urban design elements.

Original ($I_{i}$) |  Latent Beauty representation ($\hat{I_{j}}$) |  Beautified ($I_{j}$)
---|---|---
|  |
|  |
|  |
|  |
|  |
Table 4: Examples of the “FaceLifting” process, which tends to add greenery,
narrow roads, and pavements.

##  4 Evaluation

The goal of FaceLift is to transform existing urban scenes into versions that:
_i)_ people perceive more beautiful; _ii)_ contain urban elements typical of
great urban spaces; _iii)_ are easy to interpret; and _iv)_ architects and
urban planners find useful. To ascertain whether FaceLift meets that composite
goal, we answer the following questions next:

Q1 Do individuals perceive “FaceLifted” scenes to be beautiful?

Q2 Does our framework produce scenes that possess urban elements typical of
great spaces?

Q3 Which urban elements are mostly associated with beautiful scenes?

Q4 Do architects and urban planners find FaceLift useful?

### Q1 People’s perceptions of beautified scenes

To ascertain whether “FaceLifted” scenes are perceived by individuals as they
are supposed to, we run a crowd-sourcing experiment on Amazon Mechanical Turk.
We randomly select 200 scenes, 100 beautiful and 100 ugly (taken at the bottom
10 and top 10 percentiles of the Trueskill’s score distribution of Figure 2).
Our framework then transforms each ugly scene into its beautified version, and
each beautiful scene into its corresponding ‘uglified’ version. These scenes
are arranged into pairs, each of which contains the original scene and its
beautified or uglified version. On Mechanical Turk, we only select verified
masters as our crowd-sourcing workers (those with an approval rate above 90%
during the past 30 days), pay them $0.1 per task, and ask each of them to
choose the most beautiful scene for each given pair. We make sure to have at
least 3 votes for each scene pair. Overall, our workers end up selecting the
scenes that are actually beautiful 77.5% of the times, suggesting that
“FaceLifted” scenes are indeed perceived to be more beautiful by people.

### Q2 Are beautified scenes great urban spaces?

To answer that question, we need to understand what makes a space great. After
reviewing the literature in urban planning, we identify four factors
associated with great places [37, 1] (Table 5): they mainly tend to be
walkable, offer greenery, feel cozy, and be visually rich.

Metric |  Description
---|---
Walkability |  Walkable streets support people’s natural tendency to explore spaces [37, 38, 39].
Green Spaces |  The presence of greenery has repeatedly been found to impact people’s well-being [1]. Under certain conditions, it could also promote social interactions [2]. Not all types of greenery have to be considered the same though: dense forests or unkempt greens might well have a negative impact [4].
Landmarks |  Feeling lost is not a pleasant experience, and the presence of landmarks have been shown to contribute to the legibility and navigability of spaces [40, 2, 37, 41].
Privacy-Openness |  The sense of privacy conveyed by a place’s structure (as opposed to a sense of openness) impacts its perception [37].
Visual Complexity |  Visual complexity is a measure of how diverse an urban scene is in terms of design materials, textures, and objects [37]. The relationship between complexity and preferences generally follows an ‘inverted-U’ shape: we prefer places of medium complexity rather than places of low or high complexity [12].

Table 5: Urban Design Metrics.

To automatically extract visual cues related to these four factors, we select
500 ugly scenes and 500 beautiful ones at random, transform them into their
opposite aesthetic qualities (i.e., the ugly ones are beautified, and the
beautiful ones are ‘uglified’), and compare which urban elements related to
the four factors distinguish uglified scenes from beautified ones.

We extract labels from each of our 1,000 scenes using two image classifiers.
First, using PlacesNet [33], we label each of our scenes according to a
classification containing 205 labels (reflecting, for example, landmarks,
natural elements), and retain the five labels with highest confidence scores
for the scene. We used a list of 8 properties of walkable streets defined in
previous work [42] as a guide to manually select only the PlacesNet labels
that are related to walkability. These labels include, for example: _abbey_ ,
_plaza_ , _courtyard_ , _garden_ , _picnic area_ , and _park_ (Table 6
contains the exhaustive list). Second, using Segnet [43], we label each of our
scenes according to a classification containing 12 labels. That is because
Segnet is trained on dash-cam images, and classifies each scene pixel with one
of these twelve labels: road, sky, trees, buildings, poles, signage,
pedestrians, vehicles, bicycles, pavement, fences, and road markings.

Architectural | Walkable | Landmark | Natural
---|---|---|---
Apartment building | Abbey | Airport | Badlands
Building Facade | Alley | Amphitheatre | Bamboo Forest
Construction Site | Boardwalk | Amusement Park | Canyon
Courthouse | Botanical Garden | Arch | Coast
Drive way | Corridor | Amphitheatre | Corn field
Door way | Cottage garden | Baseball Field | Creek
Forest road | Courtyard | Basilica | Desert (Sand)
Garbage dump | Crosswalk | Bridge | Field (cultivated)
Golf course | Fairway | Castle | Field (wild)
Highway | Food court | Wind mill | Mountain
Hotel | Forest path | Cathedral | Snowy Mountain
Inn | Formal Garden | Church | Ocean
Ice skating rink | Herb Garden | Dam | Orchard
Motel | Outdoor Market | Dock | Pond
Office building | Nursery | Cemetery | Rainforest
Parking Lot | Patio | Fire station | Rice paddy
Railroad track | Pavilion | Fountain | River
Residential neighbourhood | Picnic area | Gas Station | Rock arch
Restaurant | Playground | Harbour | Sand bar
Runway | Plaza | Hospital | Sea Cliff
School House | Patio | Lighthouse | Ski slope
Skyscraper | Shopfront | Mansion | Sky
Slum | Topiary garden | Mausoleum | Snow field
Supermarket | Tree farm | Pagoda | Swamp
Outdoor swimming pool | Veranda | Palace | Valley
Tower | Vegetable garden | Racecourse | Wheat field
Water tower | Yard | Ruin | Desert (vegetation)
Wind farm |  | Rope Bridge |
|  | Ski Resort |
|  | Baseball stadium |
|  | Football stadium |
|  | Subway Station |
|  | Train Station |
|  | Temple |
Table 6: Classification of the PlacesNet labels into the four categories.
Figure 6: Number of labels in specific urban design categories (on the
$x$-axis) found in beautified scenes as opposed to those found in uglified
scenes. Figure 7: Count of specific walkability-related labels (on the
$x$-axis) found in beautified scenes minus the count of the same labels found
in uglified scenes.

Having these two ways of labeling scenes, we can now test whether the
expectations set by the literature of what makes urban spaces great (Table 5)
are met in the FaceLifted scenes.

_H1 Beautified scenes tend to be walkable._ We manually select only the
PlacesNet labels that are related to walkability. These labels include, for
example, abbey, plaza, courtyard, garden, picnic area, and park. To test
hypothesis _H1_ , we count the number of walkability-related labels found in
beautified scenes as opposed to those found in uglified scenes (Figure 6): the
former contain twice as many walkability labels than the latter. We then
determine which types of scenes are associated with beauty (Figure 7).
Unsurprisingly, beautified scenes tend to show gardens, yards, and small
paths. By contrast, uglified ones tend to show built environment features such
as shop fronts and broad roads. It is worth noting that walkability often acts
as an enabler for other desirable properties of urban space (e.g., its
restorative capability), and this might be the ultimate reason why our measure
of walkability correlates with beauty.

_H2 Beautified scenes tend to offer green spaces._ We manually select only the
PlacesNet labels that are related to greenery. These labels include, for
example, fields, pasture, forest, ocean, and beach. Then, in our 1,000 scenes,
to test hypothesis _H2_ , we count the number of nature-related labels found
in beautified scenes as opposed to those found in uglified scenes (Figure 6):
the former contain more than twice as many nature-related labels than the
latter. To test this hypothesis further, we compute the fraction of ‘tree’
pixels (using SegNet’s label ‘tree’) in beautified and uglified scenes, and
find that beautification adds 32% of tree pixels, while uglification removes
17% of them.

(a)

(b)

Figure 8: The percentage of beautified scenes ($y$-axis): (a) having an
increasing presence of sky (on the $x$-axis); and (b) having an increasing
level of visual richness (on the $x$-axis). The error bars represent standard
errors obtained by random re-sampling of the data for 500 iterations.

_H3 Beautified scenes tend to feel private and ‘cozy’._ To test hypothesis
_H3_ , we count the fraction of pixels that Segnet labeled as ‘sky’ and show
the results in a bin plot in Figure 8a: the $x$-axis has six bins (each of
which represents a given range of sky fraction), and the $y$-axis shows the
percentage of beautified _vs._ uglified scenes that fall into each bin.
Beautified scenes tend to be cozier (lower sky presence) than the
corresponding original scenes.

_H4 Beautified scenes tend to be visually rich._ To quantify to which extent
scenes are visually rich, we measure their visual complexity [37] as the
amount of disorder in terms of distribution of (Segnet) urban elements in the
scene:

$H_{I}=-\sum p(i)\log p(i)$ (2)

where $i$ is the $i^{th}$ Segnet’s label, and $p(i)$ is the proportion of
urban scene $I$ containing the $i^{th}$ element. The total number of labels is
twelve. The higher $H_{I}$, the higher the scene’s entropy, that is, the
higher the scene’s complexity. It has been suggested that the relationship
between complexity and pleasantness follows an ‘inverted U’ shape [12]: we
prefer places of medium complexity rather than places of low or high
complexity. To test that, we show the percentage of beautified scenes that
fall into each complexity bin (Figure 8b): we do not find a strong evidence of
the ‘inverted U’ shape hypothesis, in that, beautified scenes are of low to
medium complexity, while uglified ones are of high complexity.

### Q3 Urban elements characterizing beautified scenes

Pair of urban elements | $\beta_{1}$ | $\beta_{2}$ | $\beta_{3}$ | Error Rate (Percentage)
---|---|---|---|---
Buildings - Trees | -0.032 | 0.084 | 0.005 | 12.7
Sky - Buildings | -0.08 | -0.11 | 0.064 | 14.4
Roads - Vehicles | -0.015 | -0.05 | 0.023 | 40.6
Sky - Trees | 0.03 | 0.11 | -0.012 | 12.8
Roads - Trees | 0.04 | 0.10 | -0.031 | 13.5
Roads - Buildings | -0.05 | -0.097 | 0.04 | 20.2

Table 7: Coefficients of logistic regressions run on one pair of predictors at
the time.

To determine which urban elements are the best predictors of urban beauty and
the extent to which they are so, we run a logistic regression, and, to ease
interpretation, we do so on one pair of predictors at the time:

$Pr(\textrm{beautiful})=logit^{-1}(\alpha+\beta_{1}*V_{1}+\beta_{2}*V_{2}+\beta_{3}*V_{1}.V_{2})$
(3)

where $V1$ is the fraction of the scene’s pixels marked with one Segnet’s
label, say, “buildings” (over the total number of pixels), and $V2$ is the
fraction of the scene’s pixels marked with another label, say, “trees”. The
result consists of three beta coefficients: $\beta_{1}$ reflects $V1$’s
contribution in predicting beauty, $\beta_{2}$ reflects $V2$’s contribution,
and $\beta_{3}$ is the interaction effect, that is, it reflects the
contribution of the dependency between $V1$ and $V2$ in predicting beauty. We
run logistic regressions on the five factors that have been found to be most
predictive of urban beauty [2, 37, 1], and show the results in Table 7.

Since we are using logistic regressions, the quantitative interpretation of
the beta coefficients is eased by the “divide by 4 rule” [44]: we can take the
$\beta$ coefficients and “divide them by 4 to get an upper bound of the
predictive difference corresponding to a unit difference” in beauty [44]. For
example, take the results in the first row of Table 7. In the model
$Pr(beautiful)=logit^{-1}(\alpha-0.032\cdot buildings+0.084\cdot
trees+0.005\cdot buildings\cdot trees)$, we can divide - 0.032/4 to get
-0.008: a difference of 1% in the fraction of pixels being buildings
corresponds to no more than a 0.8% _negative_ difference in the probability of
the scene being beautiful. In a similar way, a difference of 1% in the
fraction of pixels being trees corresponds to no more than a 0.021% _positive_
difference in the probability of the scene being beautiful. By considering the
remaining results in Table 7, we find that, across all pairwise comparisons,
trees is the most positive element associated with beauty, while roads and
buildings are the most negative ones. These results match previous literature
on what makes urban design of spaces great [40, 1, 4, 38, 2], adding further
external validity to our framework’s beautification.

### Q4 Do architects and urban planners find it useful?

Use case | Definitely Not | Probably Not | Probably | Very Probably | Definitely
---|---|---|---|---|---
Decision Making | 4.8% | 9.5% | 38% | 28.6% | 19%
Participatory Urban Planning | 0% | 4.8% | 52.4% | 23.8% | 19%
Promote Green Cities | 4.8% | 0% | 47.6% | 19% | 28.6%

Table 8: Urban experts polled about the extent to which an interactive map of
“FaceLifted” scenes promotes: (a) decision making; (b) citizen participation
in urban planning; and (c) promotion of green cities Figure 9: Interactive map
of FaceLifted scenes in Boston.

To ascertain whether practitioners find FaceLift potentially useful, we build
an interactive map of the city of Boston in which, for selected points, we
show pairs of urban scenes before/after beautification (Figure 9). We then
send that map along with a survey to 20 experts in architecture, urban
planning, and data visualization around the world. Questions were asked with a
non-neutral response Likert scale (Table 8). That is because previous work
[45, 46] has shown that such a scale: _(i)_ pushes respondents to “take a
stance”, given the absence of a neutral response; and _(ii)_ works best if
respondents are experts in the subject matter of the survey as responses of
the “I don’t know” type tend to be rare (as it is has been indeed the case for
our survey). The experts had to complete tasks in which they rated FaceLift
based on how well it supports decision making, participatory urbanism, and the
promotion of green spaces. According to our experts (Table 8), the tool can
very probably supports decision making, probably support participatory
urbanism, and definitely promote green spaces. These results are also
qualitatively supported by our experts’ comments, which include: “The maps
reveal patterns that might not otherwise be apparent”, “The tool helps
focusing on parameters to identify beauty in the city while exploring it”, and
“The metrics are nice. It made me think more about beautiful places needing a
combination of criteria, rather than a high score on one or two dimensions. It
made me realize that these criteria are probably spatially correlated”.

##  5 Discussion

FaceLift is a framework that automatically beautifies urban scenes by
combining recent approaches of Generative Adversarial Networks and Deep
Convolutional Networks. To make it usable by practitioners, the framework is
also able to explain which urban elements have been added/removed during the
beautification process.

###  5.1 Limitations

Facelift still faces some important challenges. The main limitation is that
generative image models are still hard to control, especially when dealing
with complex scenes containing multiple elements. Some of the beautifications
suggested by our tool modify the scenes too dramatically to use them as
blueprints for urban interventions (e.g., shifting buildings or broadening
roads). This undesired effect is compounded by the restricted size and
potential biases of the data that we use both for training and for selecting
the scene most similar to the machine-generated image—which might result, for
example, in generated scenes that are set in seasons or weather conditions
that differ from the input image. To address these limitations, more work has
to go into offering principled ways of fine-tuning the generative process, as
well as into collecting reliable ground truth data on human perceptions. This
data should ideally be stratified according to the people’s characteristics
that impact their perceptions. Performance assessment frameworks for the built
environment (like the Living Building Challenge333https://living-
future.org/lbc/beauty-petal) could provide a good source of non-traditional
qualitative measures useful for training and validating the Facelift
algorithm. Facelift still faces some important challenges. The main limitation
is that generative image models are still hard to control, especially when
dealing with complex scenes containing multiple elements. Some of the
beautifications suggested by our tool modify the scenes too dramatically to
use them as blueprints for urban interventions (e.g., shifting buildings or
broadening roads). This undesired effect is compounded by the restricted size
and potential biases of the data that we use both for training and for
selecting the scene most similar to the machine-generated image—which might
result, for example, in generated scenes that are set in seasons or weather
conditions that differ from the input image. To address these limitations,
more work has to go into offering principled ways of fine-tuning the
generative process, as well as into collecting reliable ground truth data on
human perceptions. This data should ideally be stratified according to the
people’s characteristics that impact their perceptions. Performance assessment
frameworks for the built environment (like the Living Building
Challenge444https://living-future.org/lbc/beauty-petal) could provide a good
source of non-traditional qualitative measures useful for training and
validating the Facelift algorithm.

Another important limitation has to do with the complexity of the notion of
beauty. There exists a wide spectrum of perceptive measures by which urban
scenes could be considered beautiful. This is because the “essence” of a place
is socio-cultural and time-specific [47]. The collective perception of the
urban environment evolves over time as its appearance and function change [48]
as a result of shifting cultures, new urban policies, and placemaking
initiatives [49]. An undiscerning, mechanistic application of machine learning
tools to urban beautification might be undesirable because current technology
cannot take into account most of these crucial aspects. FaceLift is no
exception, and this is why we envision its use as a way to support new forms
of placemaking rather than as a tool to replace traditional approaches.
Nevertheless, we emphasize the need of a critical reflection on the
implications of deploying such a technology, even when just in support of
placemaking activities. In particular, it would be beneficial to study the
impact of the transformative effect of FaceLift-inspired interventions on the
ecosystem of the city [50, 51] as well as exploring the need to pair its usage
with practices and principles that might reduce any potential undesired side
effect [52]

###  5.2 Conclusion

Despite these limitations, FaceLift has the potential to support urban
interventions in scalable and replicable ways: it can be applied to an entire
city (scalable), across a variety of cities (replicable).

We conceived FaceLift not as a technology to _replace_ the decision making
process of planners and architects, but rather as a tool to _support_ their
work. FaceLift could aid the creative process of beautification of a city by
suggesting imagined versions of what urban spaces could become after applying
certain sets of interventions. We do not expect machine-generated scenes to
equal the quality of designs done by experts. However, unlike the work of an
expert, FaceLift is able to generate beautified scenes very fast (in seconds)
and at scale (for an entire city), while quickly providing a numerical
estimate of how much some urban elements should change to increase beauty. The
user study we conducted suggests that these features make it possible to
inspire the work of decision makers and to nudge them into considering
alternative approaches to urban interventions that might not otherwise be
apparent. We believe this source of inspiration could benefit non-experts too,
for example by helping residents to imagine a possible future for their cities
and by motivating citizen action in the deployment of micro-interventions.

To turn existing spaces into something more beautiful, that will still be the
duty of architecture. Yet, with technologies similar to FaceLift more readily
available, the complex job of recreating restorative spaces in an increasingly
urbanized world will be greatly simplified.

## Data accessibility statement

The aggregated version of the data, the best performing models and all the
supporting code can be accessed freely from http://goodcitylife.org/data.php

## References

  * [1] Christopher Alexander.  A Pattern Language: Towns, Buildings, Constructions.  Oxford University Press, 1977.
  * [2] Daniele Quercia, Neil Keith O’Hare, and Henriette Cramer.  Aesthetic capital: what makes london look beautiful, quiet, and happy?  In Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing, pages 945–955. ACM, 2014.
  * [3] Philip Salesses, Katja Schechtner, and César A Hidalgo.  The collaborative image of the city: mapping the inequality of urban perception.  PloS one, 8(7):e68400, 2013.
  * [4] J. Jacobs.  The Death and Life of Great American Cities.  Random House, 1961.
  * [5] Abhimanyu Dubey, Nikhil Naik, Devi Parikh, Ramesh Raskar, and César A Hidalgo.  Deep learning the city: Quantifying urban perception at a global scale.  arXiv preprint arXiv:1608.01769, 2016.
  * [6] A. De Botton.  The Architecture of Happiness.  Vintage Series. Knopf Doubleday Publishing Group, 2008.
  * [7] Marcus Foth.  Handbook of research on urban informatics: The practice and promise of the real-time city.  Information Science Reference Hershey, PA, 2009.
  * [8] Marcus Foth, Jaz Hee-jeong Choi, and Christine Satchell.  Urban informatics.  In Proceedings of the ACM 2011 conference on Computer supported cooperative work, pages 1–8. ACM, 2011.
  * [9] Eulogio Real, Constantino Arce, and José Manuel Sabucedo.  Classification of landscapes using quantitative and categorical data, and prediction of their scenic beauty in north-western spain.  Journal of environmental psychology, 20(4):355–373, 2000.
  * [10] Stephen Kaplan, Rachel Kaplan, and John S Wendt.  Rated preference and complexity for natural and urban visual material.  Perception & Psychophysics, 12(4):354–356, 1972.
  * [11] Rachel Kaplan and Stephen Kaplan.  The experience of nature: A psychological perspective.  CUP Archive, 1989.
  * [12] Roger S Ulrich.  Aesthetic and affective response to natural environment.  In Behavior and the Natural Environment, pages 85–125. Springer, 1983.
  * [13] Robert J. Sampson and Stephen W. Raudenbush.  Seeing Disorder: Neighborhood Stigma and the Social Construction of Broken Windows.  Social Psychology Quarterly, 67(4), 2004.
  * [14] Pall Jakob Lindal and Terry Hartig.  Architectural variation, building height, and the restorative quality of urban residential streetscapes.  Journal of Environmental Psychology, 2012.
  * [15] Daniele Quercia, Rossano Schifanella, and Luca Maria Aiello.  The shortest path to happiness: Recommending beautiful, quiet, and happy routes in the city.  In Proceedings of the 25th ACM Conference on Hypertext and Social Media, pages 116–125, 2014.
  * [16] Kylie Ball, Adrian Bauman, Eva Leslie, and Neville Owen.  Perceived environmental aesthetics and convenience and company are associated with walking for exercise among australian adults.  Journal of Preventive Medicine, 33(5):434–440, 2001.
  * [17] Billie Giles-Corti, Melissa H Broomhall, Matthew Knuiman, Catherine Collins, Kate Douglas, Kevin Ng, Andrea Lange, and Robert J Donovan.  Increasing walking: how important is distance to, attractiveness, and size of public open space?  American Journal of Preventive Medicine, 28(2):169–176, 2005\.
  * [18] Chanuki Illushka Seresinhe, Tobias Preis, and Helen Susannah Moat.  Using deep learning to quantify the beauty of outdoor places.  Royal Society Open Science, 4(7):170170, 2017.
  * [19] Nikhil Naik, Scott Duke Kominers, Ramesh Raskar, Edward L Glaeser, and César A Hidalgo.  Computer vision uncovers predictors of physical urban change.  Proceedings of the National Academy of Sciences, 114(29):7571–7576, 2017.
  * [20] Marco De Nadai, Radu Laurentiu Vieriu, Gloria Zen, Stefan Dragicevic, Nikhil Naik, Michele Caraviello, Cesar Augusto Hidalgo, Nicu Sebe, and Bruno Lepri.  Are Safer Looking Neighborhoods More Lively?: A Multimodal Investigation into Urban Life.  In Proceedings of the ACM on Multimedia Conference (MM), 2016\.
  * [21] Sean M Arietta, Alexei A Efros, Ravi Ramamoorthi, and Maneesh Agrawala.  City forensics: Using visual elements to predict non-visual city attributes.  IEEE Transactions on Visualization and Computer Graphics, 20(12):2624–2633, 2014.
  * [22] Stephen Law, Chanuki Illushka Seresinhe, Yao Shen, and Mario Gutierrez-Roig.  Street-Frontage-Net: urban image classification using deep convolutional neural networks.  International Journal of Geographical Information Science, pages 1–27, 2018.
  * [23] Stephen Law, Brooks Paige, and Chris Russell.  Take a look around: using street view and satellite images to estimate house prices.  arXiv preprint arXiv:1807.07155, 2018.
  * [24] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In Advances in Neural Information Processing Systems, pages 2672–2680, 2014.
  * [25] Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al.  Photo-realistic single image super-resolution using a generative adversarial network.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4681–4690, 2017.
  * [26] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros.  Context encoders: Feature learning by inpainting.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2536–2544, 2016.
  * [27] Xinchen Yan, Jimei Yang, Kihyuk Sohn, and Honglak Lee.  Attribute2image: Conditional image generation from visual attributes. corr abs/1512.00570, 2015.
  * [28] Yaniv Taigman, Adam Polyak, and Lior Wolf.  Unsupervised cross-domain image generation.  arXiv preprint arXiv:1611.02200, 2016.
  * [29] Liqian Ma, Qianru Sun, Stamatios Georgoulis, Luc Van Gool, Bernt Schiele, and Mario Fritz.  Disentangled person image generation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 99–108, 2018.
  * [30] Anh Nguyen, Alexey Dosovitskiy, Jason Yosinski, Thomas Brox, and Jeff Clune.  Synthesizing the preferred inputs for neurons in neural networks via deep generator networks.  In Advances in Neural Information Processing Systems, pages 3387–3395, 2016.
  * [31] Hassan Abu Alhaija, Siva Karthik Mustikovela, Lars Mescheder, Andreas Geiger, and Carsten Rother.  Augmented reality meets computer vision: Efficient data generation for urban driving scenes.  International Journal of Computer Vision, 126(9):961–972, 2018\.
  * [32] Ralf Herbrich, Tom Minka, and Thore Graepel.  Trueskill: a bayesian skill rating system.  In Advances in Neural Information Processing Systems, pages 569–576, 2007.
  * [33] Bolei Zhou, Agata Lapedriza, Jianxiong Xiao, Antonio Torralba, and Aude Oliva.  Learning deep features for scene recognition using places database.  In Advances in Neural Information Processing Systems, pages 487–495, 2014.
  * [34] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  In Advances in Neural Information Processing Systems, pages 1097–1105, 2012.
  * [35] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich.  Going deeper with convolutions.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1–9, 2015.
  * [36] Alexey Dosovitskiy and Thomas Brox.  Generating images with perceptual similarity metrics based on deep networks.  In Advances in Neural Information Processing Systems, pages 658–666, 2016.
  * [37] Reid Ewing and Otto Clemente.  Measuring Urban Design: Metrics for livable places.  Island Press, 2013.
  * [38] Daniele Quercia, Luca Maria Aiello, Rossano Schifanella, and Adam Davies.  The Digital Life of Walkable Streets.  In Proceedings of the 24th ACM Conference on World Wide Web (WWW), pages 875–884, 2015.
  * [39] J. Speck.  Walkable City: How Downtown Can Save America, One Step at a Time.  In Farrar, Straus and Giroux, 2012.
  * [40] Kevin Lynch.  The image of the city.  1960\.
  * [41] Daniele Quercia, Joao Paulo Pesce, Virgilio Almeida, and Jon Crowcroft.  Psychological Maps 2.0: A web gamification enterprise starting in London.  In Proceedings of ACM International Conference on World Wide Web (WWW), 2013.
  * [42] Daniele Quercia, Luca Maria Aiello, Rossano Schifanella, and Adam Davies.  The digital life of walkable streets.  In Proceedings of the 24th international conference on World Wide Web, pages 875–884. International World Wide Web Conferences Steering Committee, 2015.
  * [43] Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla.  Segnet: A deep convolutional encoder-decoder architecture for image segmentation.  arXiv preprint arXiv:1511.00561, 2015.
  * [44] Brandon K Vaughn.  Data analysis using regression and multilevel/hierarchical models.  Journal of Educational Measurement, 45(1):94–97, 2008.
  * [45] Aphrodite Baka, Lia Figgou, and Vasiliki Triga.  ’neither agree, nor disagree’: a critical analysis of the middle answer category in voting advice applications.  International Journal of Electronic Governance, 5(3-4):244–263, 2012.
  * [46] Guy Moors.  Exploring the effect of a middle response category on response style in attitude measurement.  Quality and Quantity, 42(6):779–794, 2008.
  * [47] Christian Norberg-Schulz.  Genius Loci: Towards a Phenomenology of Architecture.  Rizzoli, 1980.
  * [48] Stewart Brand.  How buildings learn: What happens after they’re built.  Penguin, 1995.
  * [49] Marcus Foth.  Lessons from urban guerrilla placemaking for smart city commons.  In Proceedings of the 8th International Conference on Communities and Technologies, pages 32–35. ACM, 2017.
  * [50] Paul Dourish.  Algorithms and their others: Algorithmic culture in context.  Big Data & Society, 3(2):2053951716665128, 2016.
  * [51] Rob Kitchin.  Thinking critically about and researching algorithms.  Information, Communication & Society, 20(1):14–29, 2017.
  * [52] Rob Kitchin.  The ethics of smart cities and urban science.  Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 374(2083):20160115, 2016.


11institutetext: Univ. Lille, CNRS, Centrale Lille, UMR 9189 CRIStAL, F-59000
Lille, France 22institutetext: Department of Computer Science (DIKU),
University of Copenhagen, Denmark

# A function space perspective on stochastic shape evolution

Elizabeth Baker 22 Thomas Besnier 11 Stefan Sommer 22

###### Abstract

Modelling randomness in shape data, for example, the evolution of shapes of
organisms in biology, requires stochastic models of shapes. This paper
presents a new stochastic shape model based on a description of shapes as
functions in a Sobolev space. Using an explicit orthonormal basis as a
reference frame for the noise, the model is independent of the
parameterisation of the mesh. We define the stochastic model, explore its
properties, and illustrate examples of stochastic shape evolutions using the
resulting numerical framework.

###### Keywords:

shape space diffusions 3D mesh processing

##  1 Introduction

In fields from medical imaging to biology, realistic models of shape must
allow randomness in shape evolutions. For example, in evolutionary biology,
random gene changes through evolution can be hypothesised to cause random
shape variation. Therefore, there is a need for stochastic models of shape.
Moreover, recent years have seen the rise of diffusion models in deep learning
[15, 12] relying on a progressive addition of noise onto sample data which
could motivate the search for new ways to define diffusion processes in
particular data spaces such as shape spaces. In this paper, we define a
stochastic process for shape spaces that immerse a base shape in
$\mathbb{R}^{d}$, $d=2,3$. The noise is related to the shape structure without
reference to the ambient space in which the shape is embedded. Building on
related models in the case of outer shape spaces, we construct a framework
that is independent of the chosen shape representation and that gives
numerically efficient ways of simulating stochastic shape evolutions.

###  1.1 Motivation and contribution

We seek to define random paths (stochastic processes) in shape spaces starting
from a source shape $S_{0}$. The major difficulty here resides in shape spaces
generally being non-Euclidean, infinite-dimensional manifolds. Recent work has
perturbed the momentum of Hamilton’s equations [27, 28] or used stochastic
perturbations defined in the ambient space [24, 2, 3]. Here, we wish to define
the stochastic perturbations directly in the shape space without referring to
the ambient space. This paper explores an idea to do so through an intrinsic
description of compact surfaces as functions. Thus, it defines a framework
independent of the discretisation of the shapes. However, a guiding principle
for our work is that it should be computationally feasible to simulate from
the constructed process. We demonstrate this with numerical simulations for
which the code is accessible at https://github.com/tbesnier/bm-shapes. The
method can efficiently integrate random shape trajectories. We highlight two
distinct methods, one for shapes represented by functions and another for
shapes represented explicitly by point clouds or meshes.

###  1.2 Related Work

This paper fits into a body of work on the analysis of shapes in shape spaces
as follows. Shape spaces generally encompass definitions for shapes, paths
between shapes, and lengths of such paths. There are multiple methods to
define shape spaces [5]; we mention two of them here. The first approach, used
in this paper, identifies shapes as maps from an underlying manifold into
$\bbbr^{d},$ with $d=2,3$. Paths in the shape space are paths in the space of
functions. In the second approach, shapes are considered as a subset of
$\bbbr^{d}$. Variations of shapes arise from the action of the diffeomorphism
group over $\bbbr^{d}$. This approach leads to the Large Deformation
Diffeomorphic Metric Mapping (LDDMM) framework [30, 7]. The metric structures
appearing in the first class are often referred to as inner metrics. The
second approach correspondingly leads to outer metrics.

####  1.2.1 Inner approach

Usually, the function space is taken to be the space of immersions or
embeddings. Immersions are smooth maps for which the differential map is
injective. An embedding is an injective immersion. Because embeddings are
injective, it prevents the shape from self-intersecting. Different classes of
metrics have been introduced in the literature to define distances between
shapes represented as functions. One example is the square-root normal field
(SRNF) [16] in which shapes are elements of $L^{2}(\bbbs^{2},\bbbr^{3})$ and
the $L^{2}$ distance is used to define a (pseudo-)distance on the space of
immersions. Due to their simplicity and ease of calculation, SRNF has led to
several numerical frameworks [20, 6]. Other tools have been developed with
stronger Sobolev metrics (also called elastic metrics) with additional
theoretical properties [14]. One example is from Su et al. [26]: shapes are
decomposed into a spherical harmonics basis, and a framework to find geodesics
between the decomposed shapes is introduced. In this work, we will also use
spherical decompositions of shapes.

####  1.2.2 Outer approach

The LDDMM framework applies in a matching context from a source shape $S_{0}$
to a target shape $S_{1}$. An optimisation problem is solved to find the
“best” diffeomorphism acting on $S_{0}$ and its ambient space to match
$S_{1}$, resulting in geodesics in the shape space. A probabilistic framework
can be built around this by considering the Hamiltonian formulation of the
geodesic. That is, the geodesic equation is written in terms of a momentum and
velocity equation, and these are perturbed. In [27], the momentum map is
perturbed. This is executed in a finite setting where shapes are approximated
by a finite number of points (called landmarks). Adding noise to the momentum
equation can be interpreted as a random force acting on each landmark. In
[28], the approach is extended to the case where the number of landmarks
approaches infinity. This is similar to our approach, in that we too consider
stochastic perturbations of maps in $L^{2}$-space. But, where [27, 28] perturb
the momentum map of the geodesic equation, we perturb the shape directly by
considering the shape as a function. More recent work [2] has perturbed both
the momentum and the velocity maps.

##  2 Background

###  2.1 Shape space

####  2.1.1 Shapes as immersions and embeddings

One way of modelling shapes is as functions from an underlying manifold $M$
into $\bbbr^{d}$. The shape space is usually taken as either the space of
immersions or embeddings [5]. The underlying manifold $M$ can be chosen based
on the dimension and the topological features of the shape to be modelled
[14]. A common choice is $M=\bbbs^{2}$. Then, a shape $s$ is modelled as an
immersion (or an embedding) $s:\bbbs^{2}\rightarrow\mathbb{R}^{3}$. In this
way, the shape $s$ deforms the sphere. In particular, $s$ belongs to
$L^{2}(\bbbs^{2},\mathbb{R}^{3})$, the Hilbert space of square-integrable
functions.

Taking shapes as immersions (resp. embeddings) gives parameterised shapes; two
shapes that have identical images in $\bbbr^{d}$ with different
parameterisations are treated as different shapes. We can also consider
unparameterised shapes by taking the shape space to be the set of immersions
(resp. embeddings) quotiented by the space of diffeomorphisms
$\text{Diff}(M)$. Two shapes $s,s^{\prime}$ are equivalent up to a
reparameterisation if there exists a diffeomorphism $\phi\in\text{Diff}(M)$,
such that $s=s^{\prime}\circ\phi$. This forms an equivalence class
$\text{Imm}(M,\bbbr^{d})/\text{Diff}(M)$ over the space of immersions (resp.
embeddings). Therefore, the space of immersions (resp. embeddings) is called
the preshape space. Paths in the preshape space can be projected to the shape
space via the projection map. Equivalently, the stochastic model defined in
this paper can be mapped to the shape space by applying the projection to the
process.

####  2.1.2 Spherical harmonic decomposition of shapes

Figure 1: Visualisation of the three first orders of elementary functions in
the spherical harmonic basis

Considering a shape as an element of $L^{2}(\bbbs^{2},\mathbb{R}^{3})$ enables
an orthonormal expansion of the shape. We choose spherical harmonics for the
orthonormal basis as it is a natural choice for functions over the sphere [9];
loosely speaking, it is the “spherical counterpart” of the Fourier
decomposition.

Let $L^{2}(\mathbb{S}^{2})$ be the space of square-integrable functions
$f:\mathbb{S}^{2}\to\mathbb{R}$, with $\bbbs^{2}$ parameterised by
$\omega=(\theta,\phi)\in[0,2\pi)\times[0,\pi)$. Then $L^{2}(\bbbs^{2})$ forms
a Hilbert space when equipped with the inner product

$\langle
f,g\rangle_{L^{2}(\bbbs^{2})}:=\int_{\mathbb{S}^{2}}f(\omega)g(\omega)\mathrm{d}\omega=\int_{0}^{\pi}\int_{0}^{2\pi}f(\theta,\phi)g(\theta,\phi)\sin(\phi)\mathrm{d}\theta\mathrm{d}\phi,$

for $f,g\in L^{2}(\bbbs^{2}).$

Let $\Delta_{\mathbb{S}^{2}}$ be the Laplace-Beltrami operator on the sphere.
The spherical harmonics $Y_{l}^{m}:\bbbs^{2}\to\bbbr$ are defined as the
eigenfunctions of $\Delta_{\mathbf{S}^{2}}$ with respective eigenvalues
$-l(l+1)$:

$-\Delta_{\mathbb{S}^{2}}Y_{l}^{m}=l(l+1)Y_{l}^{m}.$

The spherical harmonics form a complete orthonormal basis of
$L^{2}(\bbbs^{2})$: any function $s\in L^{2}(\bbbs^{2})$ can be written as

$s(\theta,\phi)=\sum_{l=0}^{\infty}\sum_{|m|\leq l}\langle
s,Y_{l}^{m}\rangle_{L^{2}(\bbbs^{2})}Y_{l}^{m}(\theta,\phi)=\sum_{l=0}^{\infty}\sum_{|m|\leq
l}\hat{s}_{l,m}Y_{l}^{m}(\theta,\phi).$ (1)

The spherical harmonics have an explicit formula for every
$l\in\mathbb{N},|m|\leq l$ as

$Y_{l}^{m}(\theta,\phi)=\sqrt{\frac{2l+1}{4\pi}\frac{(l-m)!}{(l+m)!}}P_{l}^{m}(\cos(\theta))e^{im\phi}$
(2)

with $P_{l}^{m}$ being the associated Legendre polynomials:

$P_{l}^{m}(x)=(-1)^{m}2^{l}(1-x^{2})^{m/2}\sum_{k=m}^{l}\frac{k!}{(k-m)!}x^{k-m}\binom{l}{k}\binom{{(l+k-1)}/{2}}{l}.$
(3)

The space $L^{2}(\bbbs^{2},\bbbr^{3})$ is defined as triplets of functions in
$L^{2}(\bbbs^{2})$. That is

$L^{2}(\bbbs^{2},\bbbr^{3}):=\\{f=(f_{1},f_{2},f_{3})\mid f_{1},f_{2},f_{3}\in
L^{2}(\bbbs^{2})\\}.$

The spherical harmonic decomposition can therefore be extended from functions
in $L^{2}(\bbbs^{2},\bbbr)$ to functions $f=(f_{1},f_{2},f_{3})$ in
$L^{2}(\bbbs^{2},\bbbr^{3})$ by taking the spherical harmonic decompositions
of $f_{1},f_{2}$ and $f_{3}$.

Here we have only detailed the spherical harmonics basis. However, any
orthonormal basis $\\{e_{i}\\}_{i=0}^{\infty}$ of $L^{2}(\bbbs^{2})$ could be
used. Another potential choice could be spherical wavelets [1, 21]. Then, each
shape $S$ can be represented as the weighted sum of basis elements
$S=\sum_{i=0}^{\infty}\alpha_{i}e_{i}$ for some coefficients
$\alpha_{i}\in\bbbr$, and orthonormal basis $\\{e_{i}\\}_{i=0}^{\infty}$ of
$L^{2}(\bbbs^{2})$.

####  2.1.3 Sobolev spaces

Our aim is to use stochastic processes in the Hilbert space
$L^{2}(\bbbs^{2},\bbbr^{3})$ to define stochastic evolutions of shapes. The
advantage of working in the $L^{2}$ space is that there is an explicit basis.
However, functions in $L^{2}(\bbbs^{2},\bbbr^{3})$ need not even be
continuous, leading to highly irregular surfaces (see fig. 2 for an example).
To circumvent this, we will work in the Sobolev space
$H^{\nu}(\bbbs^{2},\bbbr^{3})$, a subspace of $L^{2}(\bbbs^{2},\bbbr^{3})$.

The Sobolev space $H^{\nu}(\bbbs^{2})$ of order $\nu\in\mathbb{N}_{\geq 0}$ is
defined by

$H^{\nu}(\mathbb{S}^{2}):=\left\\{f\in
L^{2}(\mathbb{S}^{2})\;\middle|\;\sum_{l\in\bbbn}\sum_{|m|\leq
l}(l+1)^{2\nu}|\hat{f}_{l,m}|^{2}<\infty\right\\},$ (4)

where $\hat{f}_{l,m}=\langle f,Y_{l}^{m}\rangle_{L^{2}(\bbbs^{2})}$ [13]. This
is a Hilbert space when endowed with the inner product

$\langle f,g\rangle_{H^{\nu}(\bbbs^{2})}:=\langle
f,g\rangle_{L^{2}(\bbbs^{2})}+\langle(-\Delta_{\mathbb{S}^{2}})^{\nu/2}f,(-\Delta_{\mathbb{S}^{2}})^{\nu/2}g\rangle_{L^{2}(\bbbs^{2})},$

where $\Delta_{\bbbs^{2}}$ is the Laplace-Beltrami operator on the sphere, and
the fractional power of the Laplace-Beltrami operator is defined in terms of
spherical harmonics as

$(-\Delta_{\mathbb{S}^{2}})^{\nu/2}f=\sum_{l\in\bbbn}\sum_{|m|\leq
l}(l(l+1))^{\nu/2}\hat{f}_{l,m}Y_{l}^{m}.$

By the embedding theorem, functions in $H^{\nu}(\bbbs^{2})$ are continuous for
any $\nu\geq 2$. Essentially, functions in Sobolev spaces have fast-decaying
spherical harmonic coefficients. In other words, their spectral information
concentrates around low frequencies. Moreover, this gives us a method for
mapping functions from $L^{2}(\bbbs^{2})$ into $H^{\nu}(\bbbs^{2})$: If

$f=\sum_{l,m}\hat{f}_{l,m}Y_{l}^{m}\in L^{2}(\bbbs^{2})$

then

$g=\sum_{l,m}(l+1)^{-\nu}\hat{f}_{l,m}Y_{l}^{m}\in H^{\nu}(\bbbs^{2}).$

###  2.2 Stochastic processes in Hilbert spaces

Throughout the rest of this paper, we assume our stochastic processes are
defined over some probability space $(\Omega,\mathcal{F},\bbbp).$

####  2.2.1 Wiener processes in Hilbert spaces

To define stochastic processes, we apply the theory of stochastic processes in
infinite dimensional Hilbert spaces [10]. For this, we discuss briefly how to
define a $Q$-Wiener process $(W_{t}^{Q})_{t\in[0,T]}$, with ending time $T>0$,
in a Hilbert space.

Let $\mathcal{H}$ be a Hilbert space and $Q$ a non-negative, trace-class
operator on $\mathcal{H}$. Then, there exists some orthonormal basis
$\\{e_{i}\\}_{i=0}^{\infty}$ of $\mathcal{H}$, and values
$\lambda_{i}\in\bbbr$ such that

$\displaystyle Qe_{i}=\lambda_{i}e_{i},\;\text{for all }i\in\bbbn.$ (5)

Define a $Q$-Wiener process as an $\mathcal{H}$-valued stochastic process

$W^{Q}_{t}:=\sum_{i=0}^{\infty}\sqrt{\lambda_{i}}B^{i}_{t}e_{i},$

where $t\in[0,T]$ and $\\{B^{i}_{t}\\}_{i=0}^{\infty}$ are independent, real-
valued Brownian motions on the probability space $(\Omega,\mathcal{F},\bbbp)$.
The series eq. 5 has the expected properties of a Wiener process: it converges
in $L^{2}(\Omega,\mathcal{F},\bbbp;C([0,T],\mathcal{H}))$ where
$C([0,T],\mathcal{H})$ is equipped with the supremum norm, it has a continuous
modification, and it has independent increments, with Gaussian laws [10].

####  2.2.2 Real-valued Itô processes

We want to compute stochastic processes over the coefficients of the spectral
decompositions of shapes. To this end, we define an Itô process and state its
convergence properties (see [23] for details). A (one dimensional) Itô process
$(X_{t})_{t\in[0,T]}$ is defined as the solution of a stochastic differential
equation (SDE) of the form

$\mathrm{d}X_{t}=b(t,X_{t})\mathrm{d}t+\sigma(t,X_{t})\mathrm{d}B_{t},\;X_{0}\in
L^{2}(\bbbp)$ (6)

where $B_{t}$ is Brownian motion on $\bbbr$ with respect to a filtration
$(\mathcal{F}_{t})_{t\in[0,T]}$ and $b:[0,T]\times\bbbr\to\bbbr$ and
$\sigma:[0,T]\times\bbbr\to\bbbr$ satisfy the Lipschitz continuity condition:

$|b(t,x)-b(t,y)|^{2}+|\sigma(t,x)-\sigma(t,y)|^{2}\leq L_{T}|x-y|^{2},$

where $x,y\in\bbbr,t\in[0,T]$ and $L_{T}<\infty$. We call $b$ and $\sigma$ the
drift and diffusion terms, respectively. The solution to eq. 6 is unique for
$\mathcal{F}_{0}$-measurable initial conditions $x_{0}\in L^{2}(\bbbp)$ and
satisfies

$\displaystyle\mathbb{E}\left[\sup_{t\in[0,T]}\|x_{t}\|^{2}\right]\leq\kappa_{T}\cdot\mathbb{E}\left[(1+|x_{0}|)^{2}\right].$
(7)

The choice of $b$ and $\sigma$ affects the behaviour of the Itô processes.

##  3 Inner approach: Spectral diffusion

Our aim is to introduce an inner shape space approach to stochastics in shape
spaces. In order to develop stochastic evolutions of shapes, we represent
shapes via a spherical harmonic decomposition. We can consider the stochastic
evolution of shapes by adding a $Q$-Wiener process (or any diffusion process)
directly to the decomposition and then constraining the process to
$H^{\nu}(\mathbb{S}^{2},\mathbb{R}^{3}),\;\nu\geq 2$.

Given a shape $u_{0}\in L^{2}(\bbbs^{2},\bbbr^{3})$ and an operator $Q$ on
$L^{2}(\bbbs^{2},\bbbr^{3})$, we define a $Q$-Wiener process in the spectral
domain of a shape, $u_{t}^{Q}$ as

$u^{Q}_{t}=u_{0}+W_{t}^{Q}=\underbrace{\sum_{l=0}^{\infty}\sum_{|m|<l}\langle
u_{0},Y_{l}^{m}\rangle Y_{l}^{m}}_{\text{source
shape}}+\underbrace{\sum_{l=0}^{\infty}\sum_{|m|<l}B_{t}^{l,m}Q^{1/2}(Y_{l}^{m})}_{\text{$Q$-Wiener
process}}$ (8)

where $\\{B^{l,m}_{t}\\}_{l,m}$ are independent real-valued Brownian motions.

In eq. 8, $Q$ controls the diffusion to guarantee the convergence of the
process in $H$. If we choose $Q$ to be a non-negative trace-class operator,
the process converges in $L^{2}(\bbbs^{2},\bbbr^{3})$. For any basis
$\\{e_{i}\\}_{i\in\bbbn}$ of $L^{2}$ and any sequence of positive real numbers
$\\{\lambda_{i}\\}_{i\in\bbbn}$ satisfying
$\sum_{i\in\bbbn}\lambda_{i}<\infty$, the operator

$Q(\cdot)=\sum_{i\in\bbbn}\lambda_{i}\langle\cdot,e_{i}\rangle e_{i}$

is non-negative and of trace class. Therefore, we use a spherical harmonics
basis weighted with a positive decaying sequence
$\\{\lambda_{i}\\}_{i\in\bbbn},$ where $\sum_{i}\lambda_{i}<\infty,$ defining
the $Q$-Wiener process

$W_{t}^{Q}=\sum_{l\in\bbbn}\sum_{|m|\leq
l}\sqrt{\lambda_{l}}B^{l,m}_{t}Y_{l}^{m}.$

The rate of decay of the coefficients $\\{\lambda_{i}\\}_{i}$ controls the
regularity of the space in which the process converges. If

$\sum_{l\in\bbbn}\sum_{|m|\leq l}\lambda_{l}(l+1)^{2\nu}<\infty$

the sum converges in $L^{2}(\Omega;C([0,1],H^{\nu}(\bbbs^{2},\bbbr^{3})),$
where $C([0,1],H^{\nu}(\bbbs^{2},\bbbr^{3}))$ is equipped with the supremum
norm.

The process in eq. 8 can be generalised by exchanging the independent Brownian
motions $\\{B^{l,m}_{t}\\}_{l,m}$ with other, more general independent
stochastic processes $\\{x_{t}^{l,m}\\}_{l,m}$, for example, Itô processes. In
this way, we have stochastic processes on the coefficients of the shape.
Letting $x^{l,m}_{0}=0$ means that eq. 7 is bounded. When this is satisfied,
the process

$\displaystyle X_{t}^{Q}:=\sum_{l\in\bbbn}\sum_{|m|\leq
l}\sqrt{\lambda_{l}}x^{l,m}_{t}Y_{l}^{m}$ (9)

also converges in
$L^{2}(\Omega,\mathcal{F},\bbbp;C([0,1],H^{\nu}(\bbbs^{2},\bbbr^{3}))$ [10].

##  4 Numerical experiments for spectral diffusions

We here aim to illustrate the constructed stochastic process. We address two
situations. The first assumes shapes are provided via a function in its
spherical harmonic decomposition, and the second assumes shapes are
represented by meshes (with vertex coordinates and face connectivity
information). In the first situation, all that remains is to simulate the
real-valued stochastic process $x^{l,m}_{t}$ from eq. 9. For the numerical
integration, we use an Euler–Maruyama scheme [18] such that the stochastic
process $u_{t}$ defined by

$\mathrm{d}u_{t}=b(t,u_{t})\mathrm{d}t+\sigma(t,u_{t})\mathrm{d}W_{t}$ (10)

is approximated by

$u_{t_{k+1}}\approx u_{t_{k}}+b(t_{k},u_{t_{k}})\Delta
t_{k}+\sigma(t_{k},u_{t_{k}})\Delta W_{t_{k}}$ (11)

with $\Delta t_{k}=t_{k+1}-t_{k}$ and $\Delta
W_{t_{k}}=W_{t_{k+1}}-W_{t_{k}}\sim\mathcal{N}(0,\sqrt{t_{k+1}-t_{k}})$.

###  4.1 Effects of truncation and covariance operators

We start by illustrating the effects of different covariance operators within
our framework. To this end, we use the sphere $\bbbs^{2}$, which we view as
the image of

$\displaystyle s:\begin{cases}[0,\pi)\times[0,2\pi)&\to\bbbr^{3}\\\
(\theta,\phi)&\mapsto(\sin{\theta}\cos{\phi},\sin{\theta}\sin{\phi},\cos{\theta}),\end{cases}$

and then decompose $s$ with respect to the spherical harmonics basis

$s=\left(\sum_{\begin{subarray}{c}l\in\bbbn\\\ |m|\leq
l\end{subarray}}s^{1}_{l,m}Y_{l}^{m},\;\sum_{\begin{subarray}{c}l\in\bbbn\\\
|m|\leq
l\end{subarray}}s^{2}_{l,m}Y_{l}^{m},\;\sum_{\begin{subarray}{c}l\in\bbbn\\\
|m|\leq l\end{subarray}}s^{3}_{l,m}Y_{l}^{m}\right).$

The results in fig. 2 show some processes $u(t)=(u^{1}(t),u^{2}(t),u^{3}(t))$
defined by

$u^{i}(t)=s+\sum_{l=0}^{N}\sum_{|m|\leq l}\lambda_{l}B_{t}^{l,m}Y_{l}^{m}\;\in
L^{2}(\bbbs^{2},\bbbr),\quad i\in\\{1,2,3\\}$

where $N\in\bbbn$, $\\{B_{t}^{l,m}\\}_{l,m}$ are independent real-valued
Brownian motions and $\\{\lambda_{l}\\}_{l\leq N}$ are chosen to define
various operators. We also note the role the truncation value $N\in\bbbn$
plays. The spectral information of the series is concentrated around lower
frequencies. Taking lower values for $N$ results in notably smoother shapes
but also gives less detail to the surface of the shapes. We encourage the
reader to also look at the .gif files of the simulations in the GitHub
repository https://github.com/tbesnier/bm-shapes.

Figure 2: Each row presents some frames of a $Q$-Wiener process starting from
the unit (discretised) sphere. 1st row: $\lambda_{l}=1$ up to $N=25$ (no
decay), 2nd row: $\lambda_{l}=\frac{1}{l+1}$, 3rd row:
$\lambda_{l}=\frac{1}{(l+1))^{2}}$ and 4th row: $\lambda_{l}=1$ up to $N=225$
(no decay). We highlight the loss of regularity in the last row when high
spectral orders are not weighted down.

###  4.2 Simulations on radial projections of meshes

3D shape data can be explicitly represented as meshes rather than as maps.
Some work [19, 22] has been done on converting from mesh representations to
spherical harmonic representations. In this section, we use an easier method
using radial projection (see fig. 3 for a visualisation) before computing a
diffusion process from the sphere as described before. Finally, we take the
inverse radial projection mapping of the deformed sphere.

Figure 3: Numerical framework of our method on mesh data. First, we project on
the sphere (with a radial projection, for instance), then, we compute the
diffusion on the sphere and transfer the diffusion back to the mesh.

We used this setting because of its simplicity regarding computations, but
other types of spherical parameterisation of meshes [29, 4] can be used. When
using radial projections, the topology of the shape is no longer a problem
since shapes of any genus can be projected onto the sphere. However, using the
radial projection in this way means points with similar angular coordinates
are highly correlated. For example, on the torus, it means that the outer and
inner rings behave similarly (see fig. 4).

Figure 4: Simulation of a $Q$-Wiener process with $Q=\mathrm{Id}_{49}$. We
show in this example how the spatial correlation prevents the sides of the
torus from intersecting each other.

In fig. 5 we illustrate our framework with radial projections for assorted
meshes.

Figure 5: Q-Wiener processes applied on different meshes. Here,
$Q=(1-\Delta_{\bbbs^{2}})^{-1}$ and we take the first 25 coefficients.

###  4.3 Other types of processes

We can define and simulate any diffusion processes, as in eq. 9, instead of
using Brownian motion. For instance, the Ornstein-Uhlenbeck process is defined
by an SDE with the drift and diffusion terms given by $b(t,x):=x$ and
$\sigma(t,x):=C$ respectively, where $C$ is a real constant. We observe in
fig. 6 how the mesh inflates into a sphere which is expected for this type of
process.

Figure 6: Simulation of an Ornstein-Uhlenbeck diffusion, solution to an SDE
with a drift term $b(t,x)=x$ and diffusion term $\sigma(t,x)=0.1$. We
highlight that, as expected, the process converges towards the sphere.

In case a different time covariance structure is needed, we can also simulate
fractional $Q$-Wiener processes, which are generalisations of the Wiener
process with a covariance function
$C_{h}(s,t)=\frac{1}{2}(|t|^{h}+|s|^{h}-|t-s|^{2h})$ with Hurst index
$h\in(0,1)$. The covariance function of Brownian motion corresponds to
$h=0.5$.

Figure 7: Simulation of a fractional Wiener process from the sphere with
different Hurst indices: 1st row $h=0.3$, 2nd row $h=0.7$, 3rd row $h=0.95$.
The processes are simulated with the method described in [8]. As $H$
increases, the mesh is less subject to strong variations over short time
intervals.

A whole theory is derived from this process, and we invite the interested
reader to read [17] for more details. If $h<0.5$ or $h>0.5$, the process has
negatively or positively correlated increments. We show resulting paths in
fig. 7 with different Hurst indices.

##  5 Limitations and future work

###  5.1 Precision of the numerical reconstruction

Decomposing a mesh with spherical harmonics is one of many choices for a basis
of $L^{2}(\bbbs^{2},\mathbb{R}^{3})$. Wavelets could be better suited if the
signal has sharp, irregular details or discontinuities (as demonstrated in
fig. 8) [25]. But, the relation between Sobolev spaces and spherical wavelets
is not as straightforward as for spherical harmonics.

Figure 8: Reconstruction of a (projected) spherical signal. The original
signal (on the left) is hardly reconstructed with spherical harmonics up to
order 5. On the right, also with a resolution of 5, the wavelet (symlet 8 in
[11]) reconstruction shows near-perfect results.

###  5.2 Limitations regarding the control of the process

If we only have mesh data, the diffusion process is computed on the sphere and
then mapped back to the mesh. It has the advantage of keeping spatial
coherence as described before, but the spectral diffusion process has an
isotropic spatial variance because of the symmetry properties of spherical
harmonics. It becomes challenging to localise larger variance on some regions
of the mesh, which could be desirable as one can assume changes in the
variability of some areas with morphological data.

Finally, we point out that the sequence of weighting coefficients
$\\{\lambda_{l}\\}_{l\in\mathbb{N}}$ is fixed and comes from the Laplace-
Beltrami operator on the sphere $\mathbb{S}^{2}$. Rigorously, to apply the
Bessel potential, we should recompute the eigenvalues of the Laplacian at each
time step to stay in the shape space. We chose to avoid this extra step as it
is a computationally costly operation, but it can be added in further
developments.

##  6 Conclusion

In this paper, we proposed a function-based approach to compute stochastic
processes (Itô diffusions) between discretised surfaces (effectively meshes)
in $\mathbb{R}^{3}$. We can compute stochastics independently of the
underlying parameterisation by modelling the mesh through its spherical
harmonic decomposition. In addition, restraining the process to a Sobolev
space ensures spatial regularity at all times. Our framework enables us to use
various stochastic processes and visualise their behaviour in the mesh space
with publicly available code. Testing the behaviour of our framework on more
complex 3D structures with different stochastic processes and applying it to
geometry processing tasks is future work.

####  6.0.1 Acknowledgements

The work presented in this article was done at the Center for Computational
Evolutionary Morphometry and is partly supported by Novo Nordisk Foundation
grant NNF18OC0052000 as well as VILLUM FONDEN research grant 40582 and UCPH
Data+ Strategy 2023 funds for interdisciplinary research.

## References

  * [1] Jean-Pierre Antoine and Daniela Roşca.  The wavelet transform on the two-sphere and related manifolds: A review.  In Peter Schelkens, Touradj Ebrahimi, Gabriel Cristóbal, and Frédéric Truchetet, editors, Optical and Digital Image Processing, volume 7000 of Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, page 70000B, 2008.
  * [2] Alexis Arnaudon, Darryl D. Holm, and Stefan Sommer.  A geometric framework for stochastic shape analysis.  Foundations of Computational Mathematics, 19(3):653–701, 2019.
  * [3] Alexis Arnaudon, Frank van der Meulen, Moritz Schauer, and Stefan Sommer.  Diffusion bridges for stochastic Hamiltonian systems and shape evolutions.  SIAM Journal on Imaging Sciences, 15(1):293–323, 2022.
  * [4] Arul Asirvatham, Emil Praun, and Hugues Hoppe.  Consistent spherical parameterization.  In Proceedings of the 5th International Conference on Computational Science - Volume Part II, ICCS’05, page 265–272. Springer Berlin Heidelberg, 2005.
  * [5] Martin Bauer, Martins Bruveris, and Peter Michor.  Overview of the geometries of shape spaces and diffeomorphism groups.  Journal of Mathematical Imaging and Vision, 2014.
  * [6] Martin Bauer, Nicolas Charon, and Philipp Harms.  Inexact elastic shape matching in the square root normal field framework.  In International Conference on Geometric Science of Information, pages 13–20. Springer, 2019.
  * [7] Mirza Faisal Beg, Michael Miller, Alain Trouvé, and Laurent Younes.  Computing large deformation metric mappings via geodesic flows of diffeomorphisms.  International Journal of Computer Vision, 61:139–157, 02 2005.
  * [8] Jean-Francois Coeurjolly and Emilio Porcu.  Fast and exact simulation of complex-valued stationary Gaussian processes through embedding circulant matrix.  Journal of Computational and Graphical Statistics, 27(2):278–290, 2018.
  * [9] R. Courant and D. Hilbert.  Methods of Mathematical Physics: Partial Differential Equations.  Wiley, 1st edition, 1989.
  * [10] Giuseppe Da Prato and Jerzy Zabczyk.  Stochastic Equations in Infinite Dimensions.  Encyclopedia of Mathematics and its Applications. Cambridge University Press, 2nd edition, 2014.
  * [11] Ingrid Daubechles.  Orthonormal Bases of Compactly Supported Wavelets, pages 564–652.  Princeton University Press, Princeton, 2006.
  * [12] Prafulla Dhariwal and Alex Nichol.  Diffusion models beat gans on image synthesis.  CoRR, abs/2105.05233, 2021.
  * [13] Q. T. Le Gia, I. H. Sloan, and H. Wendland.  Multiscale analysis in Sobolev spaces on the sphere.  SIAM Journal on Numerical Analysis, 48(6):2065–2090, 2010.
  * [14] Emmanuel Hartman, Yashil Sukurdeep, Eric Klassen, Nicolas Charon, and Martin Bauer.  Elastic shape analysis of surfaces with second-order Sobolev metrics: a comprehensive numerical framework, 2022.
  * [15] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 6840–6851, 2020.
  * [16] Ian H. Jermyn, Sebastian Kurtek, Eric Klassen, and Anuj Srivastava.  Elastic shape matching of parameterized surfaces using square root normal fields.  In Andrew Fitzgibbon, Svetlana Lazebnik, Pietro Perona, Yoichi Sato, and Cordelia Schmid, editors, Computer Vision – ECCV 2012, Lecture Notes in Computer Science, page 804–817, Berlin, Heidelberg, 2012. Springer.
  * [17] Guy Jumarie.  Complex-Valued Fractional Brownian Motion of Order n. Part I, page 129–155.  Springer Netherlands, Dordrecht, 2000.
  * [18] Peter E. Kloeden and Eckhard Platen.  Numerical Solution of Stochastic Differential Equations.  Stochastic Modelling and Applied Probability. Springer Berlin Heidelberg, 2011.
  * [19] Sebastian Kurtek, Anuj Srivastava, Eric Klassen, and Hamid Laga.  Landmark-guided elastic shape analysis of spherically-parameterized surfaces.  Computer Graphics Forum, 32:429–438, 2013.
  * [20] Hamid Laga, Qian Xie, Ian H. Jermyn, and Anuj Srivastava.  Numerical inversion of SRNF maps for elastic shape analysis of genus-zero surfaces.  IEEE Transactions on Pattern Analysis & Machine Intelligence, 39(12):2451–2464, 2017.
  * [21] Christian Lessig and Eugene Fiume.  SOHO: orthogonal and symmetric haar wavelets on the sphere.  ACM Transactions on Graphics, 27(1):4:1–4:11, 2008.
  * [22] Emil Praun and Hugues Hoppe.  Spherical parametrization and remeshing.  ACM Transactions on Graphics, 22(3):340–349, Jul 2003.
  * [23] René L. Schilling and Lothar Partzsch.  Brownian motion: an introduction to stochastic processes.  De Gruyter textbook. de Gruyter, Berlin; Boston, second edition edition, 2014.
  * [24] Stefan Sommer, Alexis Arnaudon, Line Kühnel, and Sarang C. Joshi.  Bridge simulation and metric estimation on landmark manifolds.  CoRR, abs/1705.10943, 2017.
  * [25] Gilbert Strang.  Wavelet transforms versus fourier transforms.  Bulletin of the American Mathematical Society, 28(2):288–305, 1993\.
  * [26] Zhe Su, Martin Bauer, Stephen Preston, Hamid Laga, and Eric Klassen.  Shape analysis of surfaces using general elastic metrics.  Journal of Mathematical Imaging and Vision, 62, 10 2020.
  * [27] Alain Trouvé and François-Xavier Vialard.  Shape splines and stochastic shape evolutions: A second order point of view.  Quarterly of Applied Mathematics, 70(2):219–251, 2012.
  * [28] François-Xavier Vialard.  Extension to infinite dimensions of a stochastic second-order model associated with shape splines.  Stochastic Processes and their Applications, 123(6):2110–2157, 2013\.
  * [29] Chunxue Wang, Zheng Liu, and Ligang Liu.  As-rigid-as-possible spherical parametrization.  Graphical Models, 76(5):457–467, 2014.
  * [30] Laurent Younes.  Shapes and Diffeomorphisms, volume 171 of Applied Mathematical Sciences.  Springer Berlin Heidelberg, 2019.