#### Sampling distribution of $\mu_{W}$ and $\sigma_{W}$.

In order to derive the sampling distributions of the membrane potentials, we
first need to derive the sampling distribution of the synaptic weights. That
is, for independently drawn weight vectors
$\\{\vec{w}_{1},\vec{w}_{2},...\vec{w}_{m}\\}$ of size $n$, we are interested
in the distributions of the sample mean $\hat{\mu}_{W}$ and the sample
variance $\hat{\sigma}_{W}$. For weights drawn from the zero-mean distribution
$W\sim\mathcal{N}(0,\,\sigma_{W}^{2})$, the former is simply

$\hat{\mu}_{W}\sim\mathcal{N}\left(0,\frac{\sigma_{W}^{2}}{n}\right)\leavevmode\nobreak\
.$ (69)

To obtain the sampling distribution of the variance, we first observe that,
according to Cochran’s Theorem [57],

$n\frac{\hat{\sigma}_{W}^{2}}{\sigma_{W}^{2}}\sim\chi^{2}_{n-1}\leavevmode\nobreak\
,$ (70)

which we can alternatively express as a special case of a Gamma distribution
$\Gamma\left(k,\theta\right)$ with shape parameter $k=\frac{n-1}{2}$ and scale
parameter $\theta=2$. Hence the above equation can be written as

$n\frac{\hat{\sigma}_{W}^{2}}{\sigma_{W}^{2}}\sim\Gamma\left(\frac{n-1}{2},2\right)\leavevmode\nobreak\
.$ (71)

Using the scaling property of the gamma function, we can solve Eq. (71) for
$\hat{\sigma}_{W}^{2}$ under the requirement $\frac{\sigma_{W}^{2}}{n}>0$.
This holds as long as we have variance in the weight initialization. Hence we
can express the distribution of the sample variance $\hat{\sigma}_{W}^{2}$ as

$\hat{\sigma}_{W}^{2}\sim\Gamma\left(\frac{n-1}{2},\frac{2\sigma_{W}^{2}}{n}\right)\leavevmode\nobreak\
.$ (72)

#### Sampling distribution of $\hat{\mu}_{U}$.

We start by observing that the membrane potential $U$ is normally distributed
according to the Central Limit Theorem and its mean $\mu_{U}$ and variance
$\sigma_{U}^{2}$ were given in Eqs. (4) and (5) in the main text (see Fig. 1).
To derive the sampling distribution of $\hat{\mu}_{U}$, we observe that the
sample $\hat{\mu}_{U}$ is related to $\hat{\mu}_{W}$ through

$\hat{\mu}_{U}=n\nu\bar{\epsilon}\hat{\mu}_{W}\leavevmode\nobreak\ .$ (73)

From the above equation and our derivation of the sampling distribution of
$\hat{\mu}_{W}$, it becomes apparent that

$\hat{\mu}_{U}\sim\mathcal{N}\left(0,n^{2}\nu^{2}\bar{\epsilon}^{2}\left(\frac{\sigma_{W}^{2}}{n}\right)\right)\leavevmode\nobreak\
,$ (74)

which can be further simplified to

$\hat{\mu}_{U}\sim\mathcal{N}\left(0,\sigma_{U}^{2}\frac{\nu\bar{\epsilon}^{2}}{\hat{\epsilon}}\right)$
(75)

by expressing it in terms of the initialization target $\sigma_{U}$, for which
we inserted Eq. (9) from the main text (Fig. S3a-d).

We can conclude that random sampling of the weights induces systematic
variance in the expected membrane potential means $\hat{\mu}_{U}$ of neurons
receiving inputs from $n$ homogeneous Poisson processes with firing rate
$\nu$. Specifically, the expected variance on the population level is
independent of the number of inputs $n$, but scales with the target
fluctuation magnitude $\sigma_{U}$ and the input firing rate $\nu$.

#### Sampling distribution of $\hat{\sigma}_{U}$.

We follow a similar approach to derive the sampling distribution of
fluctuation magnitudes in the population. Formally, we are looking for the
distribution of $\hat{\sigma}_{U}^{2}$, which can be derived from observing
that for a neuron $i$

$\left(\hat{\sigma}_{U}^{(i)}\right)^{2}=n\left(\left(\hat{\sigma}_{W}^{(i)}\right)^{2}+\left(\hat{\mu}_{W}^{(i)}\right)^{2}\right)\nu\hat{\epsilon}\leavevmode\nobreak\
.$ (76)

To get the distribution of $\hat{\sigma}_{U}$, we first need to determine the
distribution of the right hand side. Starting with the distribution of
$\hat{\mu}_{W}^{2}$, we observe that the standardized form of
$\hat{\mu}_{W}^{2}$ follows a chi-square distribution with one degree of
freedom:

$\frac{\hat{\mu}_{W}^{2}}{\frac{\sigma_{W}^{2}}{n}}\sim\chi^{2}_{1}\leavevmode\nobreak\
.$ (77)

Similarly to the first paragraph, we can rewrite the chi-square distribution
as a Gamma distribution and use its scaling properties to obtain the
distribution of $\hat{\mu}_{W}^{2}$:

$\hat{\mu}_{W}^{2}\sim\Gamma\left(\frac{1}{2},\frac{2\sigma_{W}^{2}}{n}\right)\leavevmode\nobreak\
.$ (78)

Note that both $\hat{\mu}_{W}^{2}$ and $\hat{\sigma}_{W}^{2}$ are Gamma
distributed with a shared scale parameter $\theta=\frac{2\sigma_{W}^{2}}{n}$
and that these random variable are independent. We can therefore use the
summation property of the Gamma distribution to obtain

$\left(\hat{\sigma}_{W}^{2}+\hat{\mu}_{W}^{2}\right)\sim\Gamma\left(\frac{n}{2},\frac{2\sigma_{W}^{2}}{n}\right)$
(79)

and finally plug this result back into Eq. (76) to obtain the distribution

$\hat{\sigma}_{U}^{2}\sim\Gamma\left(\frac{n}{2},2\nu\hat{\epsilon}\sigma_{W}^{2}\right)\leavevmode\nobreak\
.$ (80)

As we did in the previous paragraph, we can insert the solution for
$\sigma_{W}^{2}$ in the case of centered weights, given by Eq. (9), to
simplify the distribution to

$\hat{\sigma}_{U}^{2}\sim\Gamma\left(\frac{n}{2},\frac{2\sigma_{U}^{2}}{n}\right)$
(81)

as displayed in Fig. S3f. We can alternatively express the expected
variability as the distribution of standard deviations, which follows the
Nakagami distribution [58]

$\hat{\sigma}_{U}\sim\mathrm{Nakagami}\left(\frac{n}{2},\sigma_{U}^{2}\right)$
(82)

with shape parameter $m=\frac{n}{2}$ and spread parameter
$\Omega=\sigma_{U}^{2}$ (Fig. S3f). Thus, random sampling of synaptic weights
induces a systematic variance in $\sigma_{U}$ that scales with
$\sigma_{U}^{2}$ and inversely with the number of inputs.

###  S3 Fluctuation-driven initialization of Dalian networks using log-
normally distributed weights

The initialization of the weights in separate excitatory and inhibitory
neuronal populations, as is the case with Dalian networks, requires weights
sampled from one-sided distributions. Inspired by neurobiological evidence
[59], we consider weights sampled from the log-normal distribution
parameterized by $\mu$ and $\sigma>0$ that gives rise to a random variable
$X=e^{\mu+\sigma Z}$ where $Z\sim N(0,1)$ is a standard normal random
variable. The expected value and the variance of $X$ are then defined as

$\displaystyle\mathbb{E}[X]$ $\displaystyle=$
$\displaystyle\exp\left(\mu+\frac{\sigma^{2}}{2}\right)$ (83)
$\displaystyle\mathbb{V}[X]$ $\displaystyle=$
$\displaystyle\left(\exp\left(\sigma^{2}\right)-1\right)\exp\left(2\mu+\sigma^{2}\right)\leavevmode\nobreak\
.$ (84)

To parameterize synaptic weights, we use the log-normal distribution to obtain
excitatory weights $\ln(W^{E})\sim\mathcal{N}(\mu_{E},\sigma_{E})$ and
inhibitory weights $\ln(W^{I})\sim\mathcal{N}(\mu_{I},\sigma_{I})$. As we only
have two equations, which we use to describe the membrane potential of a
neuron in the fluctuation-driven regime (mean and variance of the membrane
potential), we also need to restrict the parameterization to two parameters in
total. Hence, we set $\sigma_{E}=\sigma_{I}=1$. We can therefore simplify the
above to

$\displaystyle\mathbb{E}[W]$ $\displaystyle=$ $\displaystyle e^{\mu+1/2}$ (85)
$\displaystyle\mathbb{V}[W]$ $\displaystyle=$ $\displaystyle
e^{2\mu+2}-e^{2\mu+1}$ (86)

and also note that the second moment of the distribution is

$\mathbb{E}[W^{2}]=e^{2\mu+2}\leavevmode\nobreak\ .$ (87)

Given these definitions, we can write down the values of $\mu_{U}$ and
$\sigma_{U}^{2}$ as

$\displaystyle\mu_{U}$ $\displaystyle=$ $\displaystyle
N_{E}\nu_{E}\bar{\epsilon}_{E}e^{(\mu_{E}+1/2)}-N_{I}\nu_{I}\bar{\epsilon}_{I}e^{(\mu_{I}+1/2)}$
(88) $\displaystyle\sigma^{2}_{U}$ $\displaystyle=$ $\displaystyle
N_{E}\nu_{E}\hat{\epsilon}_{E}e^{(2\mu_{E}+2)}+N_{I}\nu_{I}\hat{\epsilon}_{I}e^{(2\mu_{I}+2)}\leavevmode\nobreak\
.$ (89)

We once more set a target mean membrane potential $\mu_{U}=0$ to achieve a
balanced state at initialization. Using this, we solve Eq. (88) to receive an
expression for $\mu_{I}$

$\mu_{I}=\mu_{E}+\log\left(\frac{N_{E}\nu_{E}\bar{\epsilon}_{E}}{N_{I}\nu_{I}\bar{\epsilon}_{I}}\right)\leavevmode\nobreak\
,$ (90)

which can be further simplified to

$\mu_{I}=\mu_{E}+\log\left(\frac{1}{\Delta_{EI}}\right)$ (91)

by using the definition of $\Delta_{EI}$ from Eq. (45). Substituting this
result into Eq. (89) allows us to solve for $\mu_{E}$ and we receive

$\mu_{E}=\frac{1}{2}\log\left(\frac{\sigma_{U}^{2}}{N_{E}\nu_{E}\hat{\epsilon}_{E}+N_{I}\nu_{I}\hat{\epsilon}_{I}\left(\frac{1}{\Delta_{EI}}\right)^{2}}\right)-1\leavevmode\nobreak\
.$ (92)

Finally, Equations (91) and (92) together with $\sigma_{E}=\sigma_{I}=1$ and
$\mu_{U}=0$ provide us the parameters to initialize the inhibitory and
excitatory weights sampled from a log normal distribution.

#### Dalian networks with excitatory recurrence.

We start from the mean and variance of the membrane potential for a neuron
receiving a feed-forward excitatory, recurrent excitatory and recurrent
inhibitory input, where we assume $\nu_{R}=\nu_{F}=\nu_{I}=\nu$,

$\displaystyle\mu_{U}$ $\displaystyle=$
$\displaystyle\left(N_{F}\nu\bar{\epsilon}_{E}\right)e^{\mu_{F}+1/2}+\left(N_{R}\nu\bar{\epsilon}_{E}\right)e^{\mu_{R}+1/2}-\left(N_{I}\nu\bar{\epsilon}_{I}\right)e^{\mu_{I}+1/2}$
(93) $\displaystyle\sigma^{2}_{U}$ $\displaystyle=$
$\displaystyle\left(N_{F}\nu\hat{\epsilon}_{E}\right)e^{2\mu_{F}+2}+\left(N_{R}\nu\hat{\epsilon}_{E}\right)e^{2\mu_{R}+2}+\left(N_{I}\nu\hat{\epsilon}_{I}\right)e^{2\mu_{I}+2}$
(94)

and the definition of $\alpha$

$\displaystyle\alpha$ $\displaystyle=$ $\displaystyle\frac{\text{Part of
}\sigma_{U}^{2}\text{ caused by excitatory feed-forward
connections}}{\text{Part of }\sigma_{U}^{2}\text{ caused by all excitatory
connections}}$ (95) $\displaystyle=$
$\displaystyle\frac{\left(N_{F}\nu\hat{\epsilon}_{E}\right)e^{2\mu_{F}+2}}{\left(N_{F}\nu\hat{\epsilon}_{E}\right)e^{2\mu_{F}+2}+\left(N_{R}\nu\hat{\epsilon}_{E}\right)e^{2\mu_{R}+2}}\leavevmode\nobreak\
.$ (96)

Here we are again requiring a mean membrane potential $\mu_{U}=0$ and we set
the variances of the log-normal distributions to one,
$\sigma_{R}=\sigma_{F}=\sigma_{I}=1$.
Performing the same sequence of solving and substituting as in the above
paragraph, we find explicit equations for the three weight distribution
parameters:

$\displaystyle\mu_{R}$ $\displaystyle=$
$\displaystyle\mu_{F}+\frac{1}{2}\log\left(N_{F}-\alpha
N_{F}\right)-\log\left(\alpha N_{R}\right)=\mu_{F}+\Delta_{R}$ (97)
$\displaystyle\mu_{I}$ $\displaystyle=$
$\displaystyle\mu_{F}+\frac{1}{2}\log\left(\frac{\bar{\epsilon}_{E}\left(e^{\Delta_{R}}N_{R}+N_{F}\right)}{N_{I}\bar{\epsilon}_{I}}\right)=\mu_{F}+\Delta_{EI}^{R}$
(98) $\displaystyle\mu_{F}$ $\displaystyle=$
$\displaystyle\frac{1}{2}\log\left(\frac{\sigma_{U}^{2}}{\nu\left(e^{2\Delta_{R}}N_{R}\hat{\epsilon}_{E}+e^{2\Delta_{EI}^{R}}\hat{\epsilon}_{I}N_{I}+N_{F}\hat{\epsilon}_{E}\right)}\right)-1\leavevmode\nobreak\
.$ (99)

  *[SNNs]: spiking neural network
  *[ANNs]: artificial neural network
  *[SNN]: spiking neural network
  *[LIF]: leaky integrate-and-fire
  *[SG]: surrogate gradient
  *[SGs]: surrogate gradient
  *[PSP]: postsynaptic potential
  *[SHD]: Spiking Heidelberg Digits
  *[BPTT]: back-propagation through time
  *[CSNN]: convolutional spiking neural network
  *[CSNNs]: convolutional spiking neural network
  *[SGD]: stochastic gradient descent
  *[ReLU]: rectified linear unit
  *[LIFs]: leaky integrate-and-fire
  *[ANN]: artificial neural network


3D point cloud segmentation using GIS
Chao-Jung Liu, Vladimir Krylov & Rozenn Dahyot

ADAPT Centre, School of Computer Science and Statistics,
Trinity College Dublin, Ireland

In this paper we propose an approach to perform semantic segmentation of 3D point cloud data by importing the geographic information from a 2D GIS layer (OpenStreetMap). The proposed automatic procedure identifies meaningful units such as buildings and adjusts their locations to achieve best fit between the GIS polygonal perimeters and the point cloud. Our processing pipeline is presented and illustrated by segmenting point cloud data of Trinity College Dublin (Ireland) campus constructed from optical imagery collected by a drone.

Keywords: Segmentation, semantic labelling, 3D point clouds, 3D environment.

§ INTRODUCTION

Up-to-date accurate 2D and 3D maps are growing increasingly important for localisation and navigation employed by both humans and machines. Various technologies and data collection modalities are available nowadays to capture and encode a digital twin of the world such as Lidar <cit.> and drone imagery <cit.>. Once created, such digital twins can be seamlessly manipulated and visualised with the help of game engines, and used in applications, such as education (e.g., driving simulators), and entertainment (e.g.,  virtual visits and gaming <cit.>).
More recently these virtual environments have also found applications in providing valuable labeled data for training machines  using data driven artificial intelligence. For instance project Airsim [https://github.com/Microsoft/AirSim] uses the Unreal game engine to provide training data for  autonomous drones and cars.

In this paper, our aim is to label automatically unstructured geolocated 3D point cloud data. For this purpose we propose to register heterogeneous sources of information: the semantic  information  provided by  OpenStreetMap (OSM) and  3D point clouds covering the same geographic area (Fig. <ref>). We show that the registration of these two sources of information allows one to segment the point cloud in useful semantic units with arbitrary geometrical shapes. After introducing some related work (Sec. <ref>), our approach  is presented in Section <ref>.

Trinity 3D model dataset <cit.>

Overlay OSM data onto game terrain

3D point cloud of Trinity College Dublin campus, 2015.

§ RELATED WORK

Structure from motion techniques have been widely used for reconstruction of large scale scenes and image geo-location. Practically, it has become a cheaper alternative to costly high-end LiDAR technology. Structure from motion relies on keypoint detection, image matching, bundle adjustment and, finally, generation of dense point clouds. These points can then be condensed into meshes and be presented in game engine as terrain (cf. Fig. <ref>).
For instance, Agarwal et al. <cit.> use a large collection of pictures harvested from the web to create a sparse point cloud of a city.
Bódis-Szomorú et al. <cit.> proposed to reconstruct a mesh of a city area by registering and fusing two point clouds, one airborne and  the other generated from street view and aerial images to reconstruct the city view, and then generate the mesh by using fusion techniques.
Rumpler et al. <cit.>  register  OSM data for retrieving the information of outlines of buildings in the form of 2D vectors, and use them with satellite pictures to generate 3D mesh building.
Bulbul and Dahyot <cit.> reconstruct cubical model of a city using OSM  geolocated building footprints  and heights, and associate  textures  extracted from Google Street view imagery to recover building facade. The resulting 3D city model is then used in a game engine for visualisation of   social media activity in the area, using  pictures posted on social media platforms.

There is a large body of work for image region labeling into generic categories such as cars, trees or buildings.
For instance, Tighe and Lazebnik <cit.> proposed an  image parsing method to segment the region of images by trained descriptors. The semantic labels are classified by predefined class for geometric / semantic context.
The semantic labeling work is conducted through all of the overlapping images  for reconstruction, which is very time consuming.
Having a 3D mesh generated multiple view imagery,
Riemensschneider et al. <cit.> improve computation complexity and accuracy by predicting the  best  image view for clustering and propagate the labels to the mesh. It is shown to be more efficient that than clustering of all images and merging labels for a same object in the scene. Tchapmi et al. <cit.> label raw point clouds using  neural networks  with Conditional Random Field  to predict class of each point in the cloud.
Kaiser et al. <cit.> segment the 2D aerial imagery into object by using the data from OSM, transforming geographic coordinate to local pixel coordinates, and use neural network architecture to refine the boundary to make a finer-grained labeling for objects.
Becker et al. <cit.> proposed to classify photogrammetric  3D point  clouds into specific classes (e.g. road, building, vegetation).

Krylov et al. <cit.> proposed to geolocate street furniture from Google Street View (GSV) imagery  using fully convolutional neural networks for object segmentation and distance estimation followed by a Markov Random Field to coherently geolocate individual objects in images. This was extended in <cit.> to the fusion of street level imagery and LiDAR data to object detection at increased spatial accuracy.

Similarly, Branson et al. <cit.> use  GSV images together with optical satellite imagery for cataloguing street trees.

§ GEOLOCATED DATA REGISTRATION

Our approach is demonstrated using the  data  for Trinity College Dublin Campus recorded using a drone in 2017, which is available open source <cit.>. Our methodology employs  the 3D point cloud generated from drone imagery (Sec. <ref>) but can also be applied to Lidar data (e.g <cit.>).
We employ OSM data as GIS source of information <cit.>.

§.§ 3D point clouds generated from drone imagery

The Trinity campus mesh  has been  generated from optical imagery captured by a drone <cit.> in 2017. Pix4D software[https://pix4d.com/] has been employed to process the images to generate the geolocated point cloud using structure from motion techniques.

§.§ OSM data structure and parsing

To acquire geolocation and semantic label data, we employ the OSM API that allows one to extract information about specific geographic area from their database in XML format. We use this interface to collect semantic information about the Trinity campus. The XML dump file contains all 3 types of OSM attributes: ways, nodes and relations. Ways are used to encode polygon-shaped areas like buildings, roads, etc. A way contains semantic labels (tags) and references to corresponding nodes. Each node contains the geolocation (longitude and latitude) of an individual point on the map (e.g., corner of a building). Relations are used to model local logical or geographic relationships between objects. Using the name of any individual building we can search among all the way attributes to find all the nodes establishing its location inside the Trinity campus.

§.§ Mercator projection

OSM uses WGS84 spatial reference system. This is the reference system also used by GPS. The corresponding coordinates are referred to as longitude and latitude. WGS84 models Earth as a spheroid. The real shape is however ellipsoidal, i.e. flat at both poles. To resolve this inconsistency, the Mercator projection[https://wiki.openstreetmap.org/wiki/Mercator] is employed.

Registration is performed in two steps: affine registration for the whole dataset, followed by a separate position adjustment for each building.

Intersection (orange)  between OSM building positions (yellow) and 3d point cloud ground truth locations (red).

IoU      Distance (meters)

Average    0.567   5.594

Median    0.607   4.871

Distance between centroids defined by OSM and 3d point cloud across 16 buildings.

Fit between point cloud data and OSM GIS data after global registration.

§.§ User defined correspondences between heterogeneous data streams

To align the reference location from Mercator projection to 3D model, we manually selected  18  correspondences (control points) between the two media streams
to estimate an affine transformation matrix  handling translation, rotation and scale with Least Squares.

The center of Trinity campus model (0,0)is on the Museum building, has  OSM coordinate (53.34380,-6.25532) and Mercator projection coordinate(-696339.0371489801,7012543.77625507).

Fig. <ref> presents our processing pipeline.

§.§ Evaluation of Global Affine registration with user defined correspondences

To evaluate how accurate the geo-locations of nodes from OSM fit the considered point cloud dataset, we report assessment on 16 buildings across the campus. We also manually label the point cloud by photo-interpretation to have a ground truth of semantic units corresponding to these 16 buildings.
Each building is defined by its perimeter, i.e. a polygon shape on x-y plane in orthographic view (see red polygons in Fig. <ref>).
The information about building locations coming from OSM is reported as yellow polygons in Fig. <ref>. We compare theses two polygons as 2D shapes (disacrding the third dimension of data) to assess the correspondance via overlap using Intersection of Unions (IoU) <cit.>.
The IoU is a real number between 0 to 1, where higher values mean better fit (equivalently, stronger overlap).
We compute the shape's centroid points and use these to find distance between polygons.

ymin=0, ymax=1,
,width=,height = 4cm
xticklabel style = rotate=45, anchor=east, align=right,text width=1.5cm,font=,yshift=-.5ex,
,xticklabels=Chapel, Regent House, Berkeley Library, O'Reilly, Museum Building, Botany Building, Campanile, Trinity Long Room Hub, The Old Dining Hall, Reading Room, Old Civil Eng. Building, FitzGerald Building, Lloyd Institute, Zoology Building, Old Library, Public Theather
only marks,
mark size=2.5pt]

IoU per building after global affine registration.

As can be seen (Fig. <ref>), there is a substantial misalignment between geolocations reported by OSM and 3d point cloud.
This is mainly because one single affine transformation does not capture well the deformation between the two streams of information. Furthermore, the location information in OSM is crowd-sourced and is not always of the highest spatial accuracy.
Figure <ref> reports the averages and median values for IoU and the distance between OSM and 3d point cloud polygons' centroids.

Fig. <ref> demonstrates the IoU for each of building and it reflects to the misalignment for each of building.

§ ADJUSTMENT OF OSM LOCATIONS

Alongside the considered 16 buildings our 3d point cloud is comprised of other points, describing ground, trees and other buildings. We first remove the ground points by elevation surface thresholding using CloudCompare(available at <https://www.danielgm.net/cc/>). We then discard the third coordinate data in the 3d point cloud (elevation above the reference level) transforming it into 2d point cloud $D_{2d}$.

We employ IoU as the main metric to assess the quality of fit between the data and ground truth locations.
In order to improve IoU for buildings, we propose to estimate the optimal translation from OSM-defined positions towards 3d point cloud clusters. We assume that the spatial orientation (rotation) and scaling are correct in OSM.
In other words, to identify parts of the point cloud that correspond to specific buildings we assume that the polygon defining the perimeter of the building is recovered by affine translation of the OSM position within a certain maximum radius.

Adjust OSM position
[1]
function $f$
Floating gap for every looping time : $\Delta_i$
Looping times : $K_i$
Threshold distance : $T_d$
Threshold IoU : $T_I$
For each building :
Select region of interest($r$);
Retrieve 2d vector $S_i$ and compute centroid of $C{S_i}$;
Initialize number of points clustered: $P$;
not the end of $K_i$
Searching from top-left, floating by $\Delta_i$;
$C{T_{x,y} S_i}$ $\leftarrow$ Compute centroid of $T_{x,y} S_i$ ;
$\mid$ $C{S_i}  - C{T_{x,y} S_i}$ $\mid$ $<$ $T_d$
$P'$ $\leftarrow$ Compute number of points clustered
$P'$ $>$  $P$ AND $IoU$ > $T_I$
$P\leftarrow P'$
goto loop

End procedure

Zoology bld., $r_3=5.5m$

Zoology bld., $r_2=8m$

Zoology bld., $r_1=12m$

FitzGerald bld., $r_3=5.5m$

FitzGerald bld., $r_2=8m$

FitzGerald bld., $r_1=12m$

2D point cloud after ground removal (blue points) and the segmentation (red points) obtained by the proposed OSM adjustment procedure for various search radii $r$ exemplified on Zoology and FitzGerald buildings. Green polygons represent the original OSM building positions.

2c|$r_1=12m$     2c|$r_2=8m$     2c$r_3=5.5m$

Metric        IoU     Distance     IoU     Distance     IoU     Distance

Average ($\Delta_1=0.8m$)     0.791        2.411      0.774       2.269      0.764       2.577

Median ($\Delta_1=0.8m$)      0.779        2.232      0.801       2.128      0.773       2.235

Average ($\Delta_2=1.15m$)     0.791         2.266      0.784       2.197      0.759       2.806

Median ($\Delta_2=1.15m$)      0.787        2.065      0.799       2.056      0.783       2.876

Performance of the proposed position adjustment method for different values of $r$: IoU and distances between centroids of estimated locations and ground truth (higher values of IoU correspond to better fit).

Let $S_i$ be the OSM perimeter of a building (polygon), and $T_{x,y}S_i$ be its translation by $x$ meters horizontally (east-west) and $y$ meters vertically (north-south).
Let $\hat{T}$ be the optimal translated position of the OSM building delivering maximum to our fit criterion IoU, where we consider overlap between the OSM-translated polygon and the 2d point cloud. Then the perimeter of the building in our point cloud can be found as $\hat {T}S_i$, with

$$\hat {T}   = \operatorname{arg\,max}_T \text{\bf \, IoU}(T_{x,y} S_i, D_{2d}), \text{\; with \;} x^2+y^2 < r^2.$$

To establish the overlap we also take into account the density of the 2d point cloud, which is non-uniform, due to substantially higher density of points along the edges of buildings (originating from points on the vertical surfaces).

Practically we simplify the above maximization problem by considering fixed sizes of increments for $x$ and $y$ translations in $T_{x,y}$.
We apply two step sizes $\Delta_1=1.15m$ and $\Delta_2=0.8m$, and conduct the search of the optimal position within the $r_1=12m$, $r_1=8m$ and $r_3=5.5m$. The overall result for $(r_1,\Delta_1)$ is demonstrated in Fig. <ref>. The green polygon represents the initial OSM polygon, and the red data points are the segmented parts of the point cloud identified by our procedure.

In Fig. <ref> we demonstrate the performance of adjustment procedure with the maximal radius increasing from $r_3=5.5m$ to $r_1=12m$ with a spatial translation step set to $\Delta_1=1.15m$. The initial OSM position for both highlighted buildings has a limited overlap with the point cloud data.
For the Zoology building the IoU increases significantly from $0.15$ (OSM after global registration) to $0.798$ (after adjustment in $r_3=12m$). Fig. <ref>-<ref> demonstrates the gradual improvement of the OSM position with the increase of radius. In case of reasonably isolated buildings with little vegetation around this behaviour is typical. FitzGerald buildings in Fig. <ref>-<ref> is an example of an isolated building closely surrounded by trees, which limits the performance of the position refinement procedure. Specifically, after the initial improvement from $r_3=5.5m$ to $r_2=8m$, the estimated position in $r_1=12m$ drops due to an adjacent cluster of dense tall vegetation which biases the estimated position.

Fitting OSM to point cloud: blue patches are non-ground points from the point cloud (buildings and trees), green bounding boxes are original OSM structures, and in red are the estimated adjusted OSM positions.

Table <ref> demonstrates performance of the proposed position adjustment algorithm for various values of search radius $r$ and translation step $\Delta$. We show the average and median for these 16 buildings and we found that the IoU increases significantly. From 0.567 on average, after the global registration of OSM data, to above $0.75$ after position adjustment.

The distance between centroid point of polygons from 5.594 meters down to around 2-2.4 meters after automatically estimated translation adjustment $\hat{T}$. In addition, in Fig. <ref> we demonstrate the improvement in IoU  achieved with the proposed position adjustment approach for the considered individual buildings with the translation step $\Delta_2=1.15m$ and three radii $r$. The proposed greedy search strategy consistently improves the overlap between OSM shapes and point cloud data. We implement this algorithm on CPU(Intel i7 with 8 threads) in MATLAB. The whole segmentation process takes around 6-8 minutes depending on maximum  distance $r_i$ and step $\Delta_i$.

§ CONCLUSION & FUTURE WORK

Our overall results show that aligning GIS (OSM) data with 3D point cloud is promising for segmenting it in meaningful subsets of vertices.
We are currently working on improving the robustness of the pipeline by integrating the colour information into the decision process.

ymin=0, ymax=1,
,width=,height = 9cm
xticklabel style = rotate=45, anchor=east, align=right,text width=1.5cm,font=,yshift=-.5ex,
,xticklabels=Chapel, Regent House, Berkeley Library, O'Reilly, Museum Building, Botany Building, Campanile, Trinity Long Room Hub, The Old Dining Hall, Reading Room, Old Civil Eng. Building, FitzGerald Building, Lloyd Institute, Zoology Building, Old Library, Public Theather,
legend style=at=(0.932,0.39),anchor=north,legend cell align=left
only marks,
mark =*,
color= gray,
mark options=fill=gray,
mark size=3pt]
only marks,
mark =*,
color= red,
mark options=fill=red,
mark size=3pt]

only marks,
mark =*,
color= green,
mark options=fill=green,
mark size=3pt]
only marks,
mark =*,
color= blue,
mark options=fill=blue,
mark size=3pt]

IoU per building: initial OSM (after global affine registration) and adjusted position with three radii $r$.

§ ACKNOWLEDGEMENT

This work was supported by the ADAPT Centre for Digital Content Technology, funded by the
Science Foundation Ireland Research Centres Programme (Grant 13/RC/2106) and the European Regional Development Fund.