Article title
Dark-field light scattering microscope with focus stabilization

Authors
Anna Peters*, Zhu Zhang, Sanli Faez

Affiliations
Nanophotonics, Debye Institute for Nanomaterials Science, Utrecht University,
3584CC Utrecht, The Netherlands

Corresponding author’s email address and Twitter handle
<EMAIL_ADDRESS>
Abstract
We present detailed design and operation instructions for a single-objective
inverted microscope. Our design is suitable for two dark-field modes of
operation: 1- total internal reflection scattering, and 2- cross-polarization
backscattering. The user can switch between the two modes by exchanging one
mode-steering element, which is also adapted to the Thorlabs cage system. To
establish a stable background speckle for differential microscopy the imaging
plane is stabilized with active feedback. We validate the stabilization
efficacy by performing long-term scattering measurement on single
nanoparticles. This setup can be extended for simultaneous scattering,
fluorescence, and confocal imaging modes.

Keywords
Cross-polarization Microscopy, Dark-field Imaging, Focus Stabilization,
Inverted Microscope.

Specifications table
o —X—X[3,l]— Hardware name Single-Objective stabLe Inverted Dark-field
microscope (SOLID) Subject area • Coherent imaging • Engineering and Material
Science • Analytical Chemistry Hardware type • Imaging tools Closest
commercial analog While some commercial microscopes can be customized to
provide this imaging modality, we are not aware of a system that provides
cross-polarization mode in reflection. Open source license CC BY-SA 4.0 Cost
of hardware 38,500 EUR up to 59,541 EUR (with Hamamatsu Orca Flash 4.0 sCMOS
camera) Source file repository https://doi.org/10.17605/osf.io/t3yvd OSHWA
certification UID NL000012

##  1\. Hardware in context

Optical detection of particles smaller than one micrometers has become
relevant in many analytical methods in medicine, nanomaterials fabrication,
and the cosmetic industry. While electron microscopy is widely used for
specialized investigations of particles below one micrometer, optical methods
are preferred for desktop applications and point-of-care investigations. In
dark-field microscopy, unlabeled nanoparticles can be visualized under an
optical microscope[1] even inside a liquid. By monitoring the nanoparticle
dynamics one can estimate and further manipulate their characteristics, e.g.
the charge state, binding with the environment or adhesion on a substrate[2],
and their composition. Small particles are difficult to detect with coherent
scattering as the scattered intensity scales with the sixth power of the
particles’ radius. Regarding biological samples, it becomes more challenging
due to their low refractive index, heterogeneous distribution in the optical
field, and low scattering cross-section. However, interferometric-enhanced
methods such as iSCAT [3, 4, 5, 6] and phase-contrast imaging [7] offer high
sensitivity. In some coherent methods, the properties of the reference beam
can be altered, which leads to a higher signal-to-noise ratio (SNR) [8]. Dark-
field microscopy gains popularity as a background-free detection tool, where
small, bright features appear on a dark background. However, methods based on
scattering are inherently more sensitive to speckle creation by partial
reflection from each optical elements. This background speckle is an
interference pattern that can rapidly vary due to drift in the optical path
and the illumination laser frequency. For this reason, scattering-based
optical detection tools often require a more stable and robust set-up than
methods based on fluorescence. Therefore, mechanical vibrations and other
drifts, such as fluctuations in the laser mode or convection in optical light
path, must be minimized or eliminated. While several groups [9, 10, 11, 12,
13, 14] have reported their open-source designs for fluorescent-based
microscopy methods, to our knowledge, no coherent imaging microscope with open
hardware have been implemented with the specification required for stable
imaging of highly sub-wavelength nanoparticles.

##  2\. Hardware description

Here we present an inverted reflection-based dark-field microscope with two
modes of operation. These two modes are 1- total internal reflection
scattering and 2- cross-polarization imaging, which can be applied inside the
same skeleton by replacing a single part: a compound beam-directing element.
To ensure high sensitivity, we need a robust skeleton with few moving parts
and solid-on-solid contacts in between the individual components. The
fundamental design concept is inspired by the Thorlabs cage system; a cost-
effective and accessible solution for coherent optical setups. Our customised
set-up can be extended for fluorescence imaging, but we will not further
discuss this application in this paper. The setup, depicted in Fig 1, is an
inverted dark-field microscope in reflection mode, where the sample is
illuminated through a thin cover glass. The sample of choice can be a solid,
e.g. small features on microscope coverslips or thin films, or particle
suspensions sandwiched between two glass slides. The sample holder is made
from aluminium (see 3\. Design files summary) and adapted to the exact size of
the sample (usually 24x40 $mm$ coverslips). The sample is additionally fixed
with magnets on top to minimize the drift when positioned on the oil-immersion
objective, while making sample exchange easy and fast. The fiber-coupled
illumination beam is directed onto the dichroic mirror (Component No.12),
where the short wavelength of 488 $nm$ is reflected. The blue laser hits the
sample from below, perpendicular to the object’s surface plane. The scattered
light is collected through the high NA objective and the imaging optics onto a
scientific CMOS camera. While operating in cross-polarization imaging, a
second laser beam at 780 $nm$ (Component No.0) needs to be installed for
optical feedback control. The reflected infrared laser beam is guided onto the
QPD (Component No.56). The difference signal on the QPD is recorded is used as
feedback for correcting the drift of the glass slide by the piezo-actuated
stage (Component No.27). In the following section we walk through the setup
parts and discuss them with more details.

### Total internal reflection-based dark field microscope

In total internal reflection illumination, the incident beam hits the glass-
sample interface under an oblique angle larger than the critical angle (Glass-
Air interface: 42 degrees and Glass-Water interface: 62.45 degrees). The back-
reflected light leaves the interface under the same angle. Most of the
scattered light propagates towards the higher-index side of the interface.
Here, the specularly-reflected beam is used as the optical feedback as its
direction is highly sensitive to displacement of the glass interface. The
central beam-stirring in this setup is a custom-designed element, matching the
cage system, that keeps the micromirrors behind the objective; see Fig 3. Each
of the arms is mounted on a kinematic stage, which provides two degrees of
freedom necessary for precisely aligning the illumination and reflection
paths. To connect this customized part to the Thorlabs cage system, we used
the base mount (Component No.35), providing another (rotational) degree of
freedom for both arms. During alignment, we need to make sure that the maximal
intensity is hitting the first gold mirror to have the greatest laser-particle
interaction, hence the largest scattering signal on the camera, and afterwards
ensure that the maximal intensity is hitting the second gold mirror to read
the highest possible value on the QPD for feedback stabilization, and to
remove the specular reflection from the imaging path, as much as possible.

### Cross polarization-based dark field microscope

For cross-polarization imaging, the directly reflected beam is automatically
cancelled because a single reflection, ideally, does not rotate the
polarization. Here, the incident beam needs to be linearly polarized. Here, we
work with a p-polarized beam. The polarization is parallel to the optical
table. This polarization is set using a polarizer in front of the laser. The
illumination beam follows the beam path shown in Fig 1, however is guided
through a polarizing beamsplitter instead of hitting the gold micro-mirror.
P-polarization is reflected upwards, and s-polarization is transmitted. After
laser-particle interaction, the back-reflected beam and the scattered beam
share the same path and can not be distinguished. Arriving again at the
polarizing beamsplitter, the beams are filtered by polarization;
p-polarization is reflected and s-polarization is transmitted and collected by
the camera. Because of the high-NA objective, the light is tightly focused on
the sample and partially rotates the beam polarization. Direct reflection from
a flat interface does not change the polarization, so it will be filtered out
by the beamsplitter. This type of microscope is more sensitive to curved
features on the interface, giving more contrast to the scattered light from
the particle. If required, you can replace the high-NA objective with a lower
NA objective and improve the field-of-view, at the cost of imaging resolution.
Here, we make use of the total internal reflection for stabilization of the
imaging plane, by installing a second light source. This second beam hits the
glass-sample interface at an angle larger that total internal reflection, to
separate it from the other coherent light source. It is aligned such that it
hits the backside of the objective and is not going on the optical axis. Part
of the out-going beam is directed through the polarizing beamsplitter again to
be collected by the QPD. A dichroic mirror (Component No.44) separated the
beams; the longer wavelength is transmitted and the shorter wavelength is
reflected.

Figure 1: Top view of experimental setup. The illumination beam path is shown
in the blue colored region. The beam hits the sample from below (orange
colored region) and the reflected beam is guided further to the camera (green
colored region). If you prefer to work with the total internal reflection
imaging setting, you do not need to install the red beam path. The reflected
blue beam acts as the optical feedback and hits the quadrant photodiode
(yellow colored region). While working with the cross-polarization imaging
setting, it is required to install the red beam path. The near-infrared beam
acts as the feedback control. Figure 2: Enlarged image of orange colored
region from Fig 1 is shown when operating in cross-polarization. The beam-
guiding region and parts of the feedback control are inside the aluminium
skeleton (depicted as the grey walls on the outside of the image).

### Optical feedback control system

Both setups have an integrated optical feedback control system, which
increases the sensitivity and stability during measurements. The incoming
laser beam hits the QPD sensor and detects the sum. The QPD difference signal
corresponds to the actual position of the glass slide surface, and is fed to
the to the NanoDrive as feedback on a PID signal. The NanoDrive has an in-
built reader which translates voltage to $\mu m$ displacement. If you conduct
your experiments in closed-loop configuration, the NanoDrive adjusts its
position according to the signal it receives from the optical beam.

### Image acquisition

Microscope images are acquired by the Hamamatsu Orca Flash 4.0 sCMOS camera
C11440-22CU. The camera runs at 30 fps at full size image if connected via USB
3.0 to the computer. Switching to Camera Link would increase the frame rate up
to 100 fps at full size image 2048x2048 pixels. The current setup description
guarantees reliable single-particle detection of submicrometer-sized particles
in a 50x50 $\mu m$ field-of-view. Combining the hardware with the open-source
software ’PyNTA’ [faez2019pynta], single-particle tracking for long time
periods is facilitated, and stores information about the particle’s position
(position coordinates), scattered intensity (pixel intensity of specified
area), electronic output signal and camera trigger.

##  Design files

### Micro mirror arm adapter

This customized adapter is fabricated out of aluminium. Each metal arm is
fixed on a compact kinematic mirror mount (Component No.36). On the front
cutout, a triangular mirror is placed and tightened with a screw. Each mirror
is coated with a high-reflective material e.g. gold. For extra secure hold,
the mirrors are glued on their edges to the metal holder. Next, the base
(Component No.35) needs to be prepared by cutting out an hole with a diameter
of 20 $mm$. After assembling the arms, they are mounted at a certain distance
apart on the base. The distance is necessary to bring the mirrors close or
further apart from each other during the laser alignment. A gap in-between the
mirrors allows the scattered beam to go through to the imaging path. This base
has been chosen to fit the C6WR cage system from Thorlabs.

(a) Front view.

(b) Top view.

(c) Side view.

Figure 3: Custom-built micro mirror arm adapter. The arms are made out of
metal and are fixed on kinematic stages. At the front, the high-reflective
mirror (mounted on the right arm) directs the incoming beam upwards. The
reflected beam is guided by the second mirror (on the left arm) to the
quadrant photodiode (QPD). That beam acts as the optical feedback control. The
scattered beam propagates between the two arms. This design has been chosen to
be compatible with the C6WR cage system from Thorlabs.

### Sample holder

The sample holder is made of aluminium to ensure hard contact onto the other
metal parts. Glass slides with dimensions of 24x40 $mm$ fit exactly in the
milled lowest region.

### Aluminium skeleton

The aluminium skeleton protects the sensitive optical components from external
influences such as dust particles, and air flow distortion, and serves as a
base for the installation of the MicroDrive and NanoDrive stages. Precise
cutouts on the walls are necessary that light can enter and leave the system.

##  3\. Design files summary

o —X—X—X[1.5,1]—X[1.5,1]— Design filename | File type | Open source license | Location of the file
---|---|---|---
SampleHolder | CAD file | CC BY-SA 4.0 | https://doi.org/10.17605/osf.io/t3yvd
mirror_arms | CAD file | CC BY-SA 4.0 | https://doi.org/10.17605/osf.io/t3yvd
Aluminium_skeleton | CAD file | CC BY-SA 4.0 | https://doi.org/10.17605/osf.io/t3yvd

##  Bill of materials

A detailed list of all numbered components and associated costs can be found
here https://doi.org/10.17605/osf.io/t3yvd. Please note that suppliers
mentioned here are just suggestions. Material costs may vary over time and
depend on location, and alternative sources may provide better prices than
those listed here. We divided the setup into parts:

  * $\bullet$

Red beam path.

  * $\bullet$

Blue beam path.

  * $\bullet$

Option 1: Total internal reflection.

  * $\bullet$

Option 2: Cross-polarization.

  * $\bullet$

Feedback control.

  * $\bullet$

Image acquisition.

as depicted in Fig 1. In the designated ”Red beam path”, the fiber-coupled
laser beam can be replaced with a solid state laser improving the laser
stability. Whether you choose for Option 1: Total internal reflection or
Option 2: Cross-polarization microscope, most of the components are the same,
but you need to switch the micro mirror arms with the polarising beamsplitter.
The base (Component No.35) remains the same to fit the cage cube C6WR
(Component No.30). In the designated ”Image acquisition”, you can choose your
camera of choice. Here, we have installed a Hamamatsu Orca Flash 4.0.

##  4\. Build instructions

The framework builds up on Thorlabs Optical Cage System 30 mm metric units,
more information can be found online under the link
https://www.thorlabs.com/navigation.cfm?guide_id=2002.

### Laser alignment

### Total internal reflection microscope

Before you start aligning the illumination beam, you assure that every optical
component is mounted and fixed as described in section 2 and according to the
steps explained in section 4. After leaving the fiber coupling output
collimator (Component No.17), the beam hits the dichroic mirror (Component
No.12) close to the center and is guided further to the next optical element
No.14 by using the two knobs on the back of the first right-angle mirror mount
(Component No.13, top right). The general rule of thumb is to adjust the blue
light source to the center of each optical component. The second right-angle
mirror mount has again two knobs on the backside to correct for the beam path.
The light beam hits the center of the first mirror of the micromirror arm
adapter (see Fig 3). Under optimal conditions, the beam should be reflected
exactly under a 90∘ degree angle upwards. The light beam is guided on the
backside of the objective and hits the glass slide under an oblique angle
(more information provided in section 2). If you take a small piece of paper
and hold it in a vertical position close to the hole on the sample holder, you
should see a thin blue line. The reflected beam is leaving the system under
the same incoming, oblique angle. Our next goal is to align the reflected beam
such, that it will hit onto the QPD. You need to position the second mirror
accordingly. On the backside of the micromirror arm adapter are two knobs per
mirror, which you need to modify. Additionally, you need to guarantee that the
space between the mirrors is large enough to pass a large fraction of the
scattered light. Remark: You don’t need the red beam path at all. The
reflected blue laser beam can also act here as your optical feedback system.

### Cross-polarization microscope

Before you start aligning the illumination beam, you assure that every optical
component is mounted and fixed as described in section 2. After leaving the
fiber coupling output collimator (Component No.17), the beam hits the dichroic
mirror (Component No.12) close to the center and is guided further to the next
optical element No.14 by using the two knobs on the back of the first right-
angle mirror mount (Component No.13, top right). The general rule of thumb is
to adjust the blue light source to the center of each optical component. The
second right-angle mirror mount has again two knobs on the backside to correct
for the beam path. You remove the sample holder and unscrew the clean
microscope objective (no residue of oil on the objective), and put them aside.
Now, you have a clear view from top on the beam path. The light beam is
reflected of the center of the polarizing beamsplitter (Component No.36). You
can see the outcoming laser from top through the cutout of the sample holder.
To check for the accuracy, you need to investigate the far-field laser beam-
profile and its position. It is helpful to place a white sheet of paper at a
large distance (roughly 1-2 meters) above your optical table and mark the
center of the cutout. Again you use the two knobs on the second right-angle
mirror mount (Component No.13, lower left) to guide the blue laser beam on the
marked cross. Screwing the objective back at its position, should not alter
the central position of the beam on your paper guideline (but will change its
size). The p-polarization of the reflected beam should follow the same beam
path as the incoming beam, because it is reflected on the polarizing
beamsplitter. Only the s-polarized component of the scattered light is
transmitted through the polarizing beamsplitter. This signal is relatively
weak, preferably you can dim the lights or you already work in a dark room.
After passing through the beamsplitter, it hits the elliptical mirror
(Component No.40). An elliptical mirror is used to maintain symmetry (circular
aperture) and reduce losses. The beam is further guided onto a shortpass
dichroic mirror (Component No.44), which reflect the blue beam. Afterwards,
the beam is going through two achromatic doublet lenses (Component No.59 & 63)
to minimize spherical aberration. Finally, the short wavelength beam reaches
the camera. Wavelengths above 550 $\mu m$ are filtered out before entering the
camera to cancel the near-infrared light used for stabilization. For
installing the optical feedback loop, you need a second coherent light source.
In the schematic Fig 1, we used a fiber-coupled laser diode. The red laser
beam hits the center of dielectric mirror (Component No.4) and through the
quarter-wave plate (Component No.6). We create a circularly polarized light,
otherwise the beam can not pass through the polarizing beamsplitter in
reflection. From now on, we align the red beam laser off-center of every
optical element to spatially separate the illumination path from the optical
feedback path. Another reason is the protection of the laser diode itself. As
a reference, you can install an iris before the dichroic mirror (Component
No.12) to know the exact position of the beam. The beam should hit the iris
and later the dichroic mirror on the right of the center. Further it hits the
dielectric mirror (Component No.14, lower left) again on the right of the
center. Before reaching the polarizing beamsplitter, the beam is guided
through the plano-convex lens (Component No.23) to focus the light further.
Within the beamsplitter, the beam hits the lower end of the cube and is
reflected upwards. The beam is guided on the backside of the objective, and
leaves the sample under the same angle. Some part of the light is reflected
back. Remark: Make sure that the reflected beam does not hit the onset of the
fiber-coupled laser diode. It might damage the diode and deteriorate the laser
power. The other part is transmitted and guided through the elliptical mirror
(Component No.40) and the dichroic mirror (Component No.44). The long-
wavelength laser beam is transmitted and collected by the QPD (Component
No.56).

##  5\. Operation instructions

Turn on the cooling system (air/water) of the camera and let it run for a
while. After a couple of minutes, you can start the camera by sliding the
button. The light bulb next to the button should turn green. This indicates
that everything works smoothly. If any other color code should appear, e.g.
orange or red, read the manual or consult the company. Next, insert your flow
cell or microfluidic chip into the sample holder. Be aware only microscope
coverslips or other systems having the dimensions 24x40mm can fit into the
sample holder. Fix the glass slide with magnets to ensure minimal drift.
Before embedding the sample holder into its proper position, put sufficient
(one to two drops) immersion oil on top of the objective. Keep in mind to use
a proper amount of oil for index-matching. If you are using not enough oil,
you will see the effect on the microscope image, resulting in a smaller field
of view or no sight of any particle. Place the sample holder in its intended
position on top of the objective. Based on the smearing of the oil on the
glass, you can tell if you used the right amount or not. Turn on the blue
laser, by enabling the button in its own software. If you need to adjust the
power or other parameters, you can do so in the software itself. We keep the
power low (1 mW) to lower the background scattering and not dealing with pixel
saturation. To move around in the x- and y-plane, you need to operate the
MicroDrive system (Component No.26). By adjusting the speed, you can regulate
how fast to want to move within your system. After finding an object of
interest, we use the adjustment knob next to the objective or the NanoDrive
itself (Component No.27) to move along the z-direction and bringing the object
into focus. As a reference, Fig 4 shows a spherical polystyrene bead of size:
800 $nm$ in focus while operating the cross-polarization microscope.

Figure 4: The point spread function (PSF) of a spherical polystyrene bead of
800 $nm$ in diameter, measured in cross-polarization mode. The red circle in
the center marks the actual size of the particle and on the background, its
diffraction pattern is shown. The characteristic clover-leaf pattern arises
from this type of microscope.

The NanoDrive allows accurate displacement along the optical axis with sub-
nanometer resolution. We can turn on the red laser by tapping the ON button on
its interface. You can change the power manually on the screen but keep in
mind to stay in the low regime. The sum on the QPD should be still high
enough, so that the received signal is readable. The object is now in focus
and we attempt to bring as much reflected light from the red laser beam into
the QPD by playing around with this two knobs on the backside of Component
No.8. If we can ensure the position of the QPD signal at its center and a high
SUM bar, we can fix the focal plane by switching from an open loop to a closed
loop. It is important to find the correct settings for the PID loop by playing
around first with the proportional gain, then the integral gain and at last
the differential gain. We use an proportional gain of 0.42, an integral gain
of 0.01 and a differential gain of 0.0, but it may vary for your experimental
setup. In section 6, you can find more information on how the stability in an
open and closed loop configuration should look like. Try to minimize
overcompensation by the PID loop, which will emerge as large oscillations on
the display. You are ready to take some measurements. After aligning the whole
setup, we recommend not to touch or modify any components in the illumination
path because it will interfere with the rest of the setup and you might end up
repeating all the steps above again.

##  6\. Validation and characterization

We investigate the stability of the experimental set-up by analyzing how the
point-spread function, the scattering intensity per nanoparticle and the QPD
voltage change over time. All of the presented results below (Fig 5-Fig 9) are
obtained with the total internal reflection microscopy setting. The feedback
system allows two settings: open-loop and closed-loop. In open-loop, the
optical signal is just monitored, but no input is send to the NanoDrive to
correct for displacements along the optical axis of the objective. Hence in
closed-loop, the microscope is continuously communicating with the NanoDrive
via the QPD sensor, where the optical signal impinges. Figures 5 and 6 are
obtained in open-loop. In a standard lab environment, a drift of roughly 0.8
$\mu m$ in 18 $min$ is presented. Upon switching to closed-loop, the drift can
be reduced significantly (see Figures 7 and 8).

b)a)

Figure 5: The point spread function(PSF) of 30 $nm$ and 40 $nm$
nanoparticles(NPs) before and after stage drift in open loop. a) The particles
landed on the glass slide before stage drift at time t=0 $min$. Before
starting monitoring the stage drift over time, the NPs are brought into focus.
b) The PSF of NPS after t=18 $min$ of drifting. The NPs are out of the focus
plane of the objective. Scale bars: 4 $\mu$m

b)a)

Figure 6: The pixel intensity of each particle (out of nine from Fig 5) has
been recorded and normalized. All particles were measured simultaneously. The
output voltage of the NanoDrive in open loop configuration is shown in the
lower panel. After calibration, the voltage can be directly related to $\mu m$
displacement. a) The intensities of the NPs are not constant during the stage
drifting. b) The differential voltage ($V_{u}-V_{b}$) between the top and
lower parts of the QPD during the stage drifting.

b)a)

Figure 7: PSF of NPs before and after 24-hour stage stabilization. The
measurement regards 30 $nm$ and 40 $nm$ nanoparticles at the focal plane. a)
The PSF of the NPs at the beginning of activating stage stabilization at time
t=0 $min$. b) The PSF of NPS after 24 hours of activating stage stabilization.
The NPs are still at the focus plane of the objective. Scale bar: 4 $\mu m$

b)a)

Figure 8: a) NPs intensities tracks during activation of the stage Z-lock
close loop control. The averaged intensity remains constant over time,
indicating no observable stage drift. b) The displacement is compensated by
the Piezo during the Z-lock close loop control. Left panel: first 60 minutes
of monitoring right after activating the Z-lock. Middle panel: 60 minutes of
monitoring 5 hours after activating the Z-lock. Right panel: 60 minutes of
monitoring 24 hours after activating the Z-lock.

b)a)

Figure 9:  a) Intensity tracks of NPs in a vibrations environment which is
introduced artificial physical activities. b) The compensated displacement of
piezo stage during this vibration environment.

Ethics statements

CRediT author statement Peters, Anna: Visualization, Writing - Original Draft.
Zhang Zhu: Formal analysis, Validation. Faez, Sanli: Conceptualization,
Supervision, Writing - Review & Editing, Project administration, Funding
acquisition. Acknowledgements We thank Paul Jurrius, JanBonne Aans, and Dante
Killian for technical support. This project was supported by the Dutch
Organisation for Scientific Research (NWO) with grant project: Photonics
Translational Research – Medical Photonics (MEDPHOT).

## References

  * [1] X. Meng, A. Sonn-Segev, A. Schumacher, D. Cole, G. Young, S. Thorpe, R. W. Style, E. R. Dufresne, and P. Kukura, “Micromirror total internal reflection microscopy for high-performance single particle tracking at interfaces,” ACS photonics, vol. 8, no. 10, pp. 3111–3118, 2021.
  * [2] A. Wang, J. W. Zwanikken, D. M. Kaz, R. McGorty, A. M. Goldfain, W. B. Rogers, and V. N. Manoharan, “Before the breach: Interactions between colloidal particles and liquid interfaces at nanoscale separations,” Physical Review E, vol. 100, no. 4, p. 042605, 2019.
  * [3] J. Ortega Arroyo, D. Cole, and P. Kukura, “Interferometric scattering microscopy and its combination with single-molecule fluorescence imaging,” Nature protocols, vol. 11, no. 4, pp. 617–633, 2016.
  * [4] R. W. Taylor and V. Sandoghdar, “Interferometric scattering microscopy: seeing single nanoparticles and molecules via rayleigh scattering,” Nano letters, vol. 19, no. 8, pp. 4827–4835, 2019.
  * [5] M. Liebel, J. T. Hugall, and N. F. van Hulst, “Ultrasensitive label-free nanosensing and high-speed tracking of single proteins,” Nano letters, vol. 17, no. 2, pp. 1277–1281, 2017.
  * [6] T. Jollans, M. D. Baaske, and M. Orrit, “Nonfluorescent optical probing of single molecules and nanoparticles,” The Journal of Physical Chemistry C, vol. 123, no. 23, pp. 14107–14117, 2019.
  * [7] A. Matlock, O. Avci, S. Ünlü, and L. Tian, “Differential phase contrast and digital refocusing in a computational reflection interferometric microscope for nanoparticle imaging,” in Computational Optical Sensing and Imaging, pp. CTh4B–2, Optical Society of America, 2017.
  * [8] G. Daaboul, A. Yurt, X. Zhang, G. Hwang, B. Goldberg, and M. Unlu, “High-throughput detection and sizing of individual low-index nanoparticles and viruses for pathogen identification,” Nano letters, vol. 10, no. 11, pp. 4727–4731, 2010.
  * [9] K. Kwakwa, A. Savell, T. Davies, I. Munro, S. Parrinello, M. A. Purbhoo, C. Dunsby, M. A. Neil, and P. M. French, “easystorm: a robust, lower-cost approach to localisation and tirf microscopy,” Journal of biophotonics, vol. 9, no. 9, pp. 948–957, 2016.
  * [10] M. N. Alsamsam, A. Kopūstas, M. Jurevičiūtė, and M. Tutkus, “The mieye: Bench-top super-resolution microscope with cost-effective equipment,” HardwareX, vol. 12, p. e00368, 2022.
  * [11] G. O. Merces, C. Kennedy, B. Lenoci, E. G. Reynaud, N. Burke, and M. Pickering, “The incubot: A 3d printer-based microscope for long-term live cell imaging within a tissue culture incubator,” HardwareX, vol. 9, p. e00189, 2021\.
  * [12] A. Maia Chagas, L. L. Prieto-Godino, A. B. Arrenberg, and T. Baden, “The€ 100 lab: A 3d-printable open-source platform for fluorescence microscopy, optogenetics, and accurate temperature control during behaviour of zebrafish, drosophila, and caenorhabditis elegans,” PLoS biology, vol. 15, no. 7, p. e2002702, 2017.
  * [13] G. N. Meloni, “3d printed and microcontrolled: the one hundred dollars scanning electrochemical microscope,” Analytical chemistry, vol. 89, no. 17, pp. 8643–8649, 2017.
  * [14] A. Guver, N. Fifita, P. Milas, M. Straker, M. Guy, K. Green, T. Yildirim, I. Unlu, M. V. Yigit, and B. Ozturk, “A low-cost and high-precision scanning electrochemical microscope built with open source tools,” HardwareX, vol. 6, p. e00082, 2019.


# Window Detection In Facade Imagery:
A Deep Learning Approach Using Mask R-CNN

Nils Nordmark, University of Gothenburg
Mola Ayenew, University of Gothenburg

###### Abstract

The parsing of windows in building facades is a long-desired but challenging
task in computer vision. It is crucial to urban analysis, semantic
reconstruction, life cycle analysis, digital twins and scene parsing amongst
other building-related tasks that require high-quality semantic data. In this
article, we investigate the usage of the Mask R-CNN framework to be used for
window detection of facade imagery input. We utilize transfer learning to
train our proposed method on COCO weights with our own collected dataset of
street view images of facades to produce instance segmentations of our new
window class. Experimental results show that our suggested approach can with a
relatively small dataset train the network only with transfer learning and
augmentation achieve results on par with prior state-of-the-art window
detection approaches, even without post-optimization techniques.

##  1 Introduction

Gathering semantic data regarding building features from images is desired as
manual gathering can often be a challenging, time consuming and resource
intensive task. For example, information regarding location, shapes and sizes,
materials of building structures such as windows, doors and balconies are
often not available, can change over time and the data gathering is usually a
manual process.

Many different approaches have been applied to solve the gathering of semantic
data within many different domains. Within building facade parsing this task
has mostly been treated as a process of semantic segmentation of facade
imagery into architectural structural classes such as windows, doors,
balconies and so on. This semantic segmentation approach provides information
about facade classes as it assigns each pixel with a class label. This means
that it does not differentiate instances of the same object as each pixel is
classified to belong to a particular class. Specifically for parsing of
windows instances in building facades, we suggest an approach that can
decouple segmentation and classification to detect windows in an image with a
bounding box and segmentation mask for each window instance.

With instance segmentation, we can analyze the windows independently with
detailed and comprehensive information for each pixel and specifically to
which window it belongs to. To separate the pixels of the same category into
different instances, we propose an approach based on the Mask R-CNN framework.

##  2 Related Work

Facade parsing has been studied for a long time and there exists a large body
of work on how to deal with such a problem.

###  Earlier approaches

[32] proposed parsing ground-level images into architectural units which could
be used for city modelling. [31] used Scale–Invariant Feature Transform (SIFT)
and [25] used K-means clustering to tackle the same problem.

Many others proposed a procedural shape grammar approach. This approach could
at a high-level be explained as a hand-crafted set of rules of basic shapes
which represents structured geometries, i.e. the structure of the object is
encoded as a set of parametric rules. Models could then be generated by
iteratively applying the rules on a starting shape.

[20] used a procedural modelling pipeline of shape grammars combined with
image analysis to derive hierarchical facade subdivisions. [10] extended the
work with an attribute grammar which used a more advanced bottom-up and top-
down mechanism for recognizing objects. Their method was based on previous
object recognition work, such as [33] and [30].

[29] combined machine learning algorithms such as Reinforcement Learning (RL)
with procedural modelling and shape grammar. Although their method could be
said to improve the works of [20] and [10], the learning process to some
extent still was based on hand-crafted rules.

[19] took another approach based on architectural styles for automatic
classification. They tried to extend [20] work with appropriate style
information as they argued that it could be used to select a more appropriate
procedural grammar for the task of building reconstruction. Their approach was
sort of an ‘inverse’ procedural modelling, i.e. an attempt to not rely on the
assumption that the building style was known and needed to be conducted as a
manual task. Instead, they tried to automate the classification of
architectural styles, by using facade images to train their classifier to
distinguish between three distinct architectural styles. For classifying those
specific styles they were quite accurate but it also required a rather
specific set of circumstances to succeed.

As most of the earlier approaches used hand-crafted features and traditional
classifiers a common limitation was that the required hand-crafted features
often failed to represent the diversity of more complex objects, i.e. could
not handle deviations from these set of user-defined rules well. They also
suffered in accuracy when the input image was more complex, such as street-
view images where the image was not perfectly rectified. Another common
limitation was that they often suffered from heavy computational costs and
slow running times ([5], [14]).

###  Convolutional Neural Networks Approaches

More recently, convolutional neural networks (CNNs) based models have gained
in popularity due to their good performances. With the success of AlexNet in
2012, many semantic segmentation approaches which used to apply traditional
machine learning algorithms have instead been primarily based on CNNs [5];
[22].

Improved models quickly followed AlexNet such as Segnet ([4]) which was one of
the first to tackle the problem of semantic segmentation of a whole image.

Another major milestone came when [17] showed that a fully convolutional
network (FCN) trained end-to-end, pixels-to-pixels could exceed then state-of-
the-art semantic image segmentation methods by retaining spatial information
of an image. In more detail, they used three pre-trained base models (AlexNet,
VGGNet and GoogleNet) and transferred them from classifiers by replacing fully
connected layers with convolutional layers. Since then a semantic segmentation
model trend based on FCNs followed with improving results ([28]).

[6] use dilated convolution instead of plain convolution to set new state-of-
art results with their proposed DeepLab system.

Recently, more automated methods and entirely based on deep learning have
achieved remarkable improvements in image analysis, such as [23] and [21].

### Deep Learning Approaches

Deep learning and specifically deep convolutional neural networks have shown
to outperform other image analysis methods for traditional vision tasks
(classification, object detection, semantic segmentation and instance
segmentation) in a lot of benchmarks and a variety of fields, e.g. medicine,
surveillance, economics, sociology. They have been applied to a diversity of
tasks such as face detection, speech recognition, autonomous vehicles and so
on.

### Deep Learning Approaches Related to Facade Parsing

Naturally, deep learning was also applied to building-related image analysis
problems as well. Recently there are some examples of work where deep learning
has been applied for facade parsing quite specifically and rather
successfully.

[27] method for semantic interpretation of facade images used a convolutional
network and a key insight was that even smaller datasets could train the
network when transfer learning could be employed. For validation, they used
the eTRIMS dataset to measure the overall accuracy, precision and recall. The
F1 score had an accuracy of 0.82± 0.09 for the facade categories, e.g. windows
0.86 ± 0.05 0.67.

[16] pushed the development of facade parsing even further with their method
DeepFacade which combined both the learning capacity of deep convolutional
neural networks with rules and a loss function based on symmetry found in
building facades structures. They trained an FCN-8s network with their novel
loss function to obtain experimental results on both the ECP dataset and the
eTRIMS dataset that significantly outperformed previous state-of-the-art
methods. Pixel accuracies on the eTRIMS dataset reached as high as 0.91 for
windows.

[8] used appearance and layout cues combined with the VGG-model. Their method
only had a single object of focus and overall accuracy was on par with other
methods, although supplementary materials had to be asked for an in-depth
understanding of the results.

[14] developed a multi-scale patch-based pattern extraction that combined with
a CNN (AlexNet), to automatically estimate the condition of buildings. They
showed to some degree their method could serve as a proxy for condition
estimates by appraisers.

[3] used stereoscopic image and tachometer data combined with a deep learning-
based facade segmentation stage based on generative adversarial networks
(GANs). The task was to reconstruct 3D models of buildings facades, in the
urban environment for cultural heritage.

[12] used the bounding boxes detected by YOLO architecture in real-time to
guide facade reconstruction in an interactive environment. Results showed
accuracies above 0.82, which they deemed as sufficient for their 3D
reconstruction task.

[24] developed a hybrid pipeline for his master thesis which consisted of both
traditional and modern machine learning approaches. His system was trained for
windows delineation segmentation, i.e. detecting window frames, mullions,
transoms and so on. For small datasets it showed promising results, however,
this method is not competitive compared to complete end-to-end state-of-the-
art deep learning approaches such as [16].

[18] used Faster R-CNN architecture for window detection, they specifically
wanted to obtain the locations of the windows in their images for 3D
reconstruction. They called their method WD-Net which showed 0.5 per cent
improved accuracy compared with the baseline Faster R-CNN.

##  3 OUR APPROACH

### Mask R-CNN Architecture

In this paper, we suggested the Mask R-CNN model, which is an intuitive
extension of the Faster R-CNN model and outputs the predicting segmentation
masks on each Region of Interest (RoI), in parallel with the existing branch
for classification and bounding box regression. Faster R-CNN is a popular
framework for object detection but it is not designed for pixel-to-pixel
alignment between network inputs and outputs. The segmentation mask in Mask
R-CNN represents the pixel-level segmentation and alignment of each object
that helps to extract each instance of an object separately without
background. Mask R-CNN is a two-stage framework. The first stage scans the
image and generates proposals, the second stage classifies the proposals and
generates bounding boxes and masks [2]; [11].

Figure 1: Mask R-CNN framework

Mask R-CNN architecture consists of four high-level modules those are;
Backbone, Region Proposal Network (RPN), ROI Classifier & Bounding Box
Regressor, and Segmentation Masks as shown in the (Figure
LABEL:tab:internalarchit) below.

Figure 2: Mask R-CNN Internal Architecture

### Backbone

The input image is fed into the CNN architecture, called the backbone. It is a
pre-trained standard convolutional neural network, which is either ResNet50 or
ResNet101. The backbone network serves as a feature extractor over an entire
image, at the early layers the backbone detects low-level features like edges
and corners, and later layers successively detect higher-level features like
the car, person, sky. When the image passes through the backbone network, it
is converted from 1024x1024px x 3 (RGB) to a feature map of shape 32x32x2048.
This feature map becomes the input for the following stages.

Another more effective backbone, called a Feature Pyramid Network (FPN)
applied on the top of the ResNet backbone to improve the feature extraction
method. FPN uses a top-down architecture with lateral connections to build an
in-network feature pyramid from a single-scale input. It improves the standard
feature extraction pyramid by adding a second pyramid that takes the high-
level features from the first pyramid and passes them down to lower layers. By
doing so, it allows features at every level to have access to both, lower and
higher-level features. Using a ResNet-FPN backbone for feature extraction with
Mask R-CNN gives excellent gains in both accuracy and speed [2].

###  Region Proposal Network (RPN)

A Region Proposal Network (RPN) was introduced by [26] that takes an image as
input and outputs a set of rectangular object proposals and finds areas that
contain objects. To generate region proposals, the author scans the images in
a sliding-window fashion over the convolutional feature map received from the
ResNet101 + FPN backbone layers. The network takes as input an n × n spatial
window of the input convolutional feature map. Each sliding window is mapped
to a lower-dimensional feature. The RPN doesn’t scan over the image directly,
instead, the RPN scans over the backbone feature map. This allows the RPN to
reuse the extracted features efficiently and avoid duplicate calculations.Then
this feature is fed into two fully connected layers, a box-regression layer,
and a box-classification layer.

Figure 3: Region Proposal Network (RPN)

RPN has a classifier(cls) and a regressor(reg) as shown in the (Figure
LABEL:tab:rpn). The regions that the RPN scans over are called anchors, which
are boxes distributed over the image area. The anchor is the central point of
the sliding window. For the ZF Model which was an extension of AlexNet, the
dimensions are 256-d and for VGG-16, it was 512-d. The classifier determines
the probability of a proposal having the target object. Regression regresses
the coordinates of the proposals. For any image, scale and aspect-ratio are
two important parameters. Aspect ratio is width of image/height of the image,
the scale is the size of the image. (Figure 3) shows that if 3 scales and 3
aspect-ratio are chosen, the total of 9 proposals are possible for each pixel,
this is how the value of k is decided, K=9 for this case, k being the number
of anchors. For the whole image, the number of anchors will be W*H*K, where W
and H are the width and height of the image respectively [13].

The RPN generates two outputs for each anchor:
Anchor Class:  The anchor class can be either foreground or background
classes. The foreground class implies that there is likely an object in that
box.

Bounding Box Refinement: A foreground anchor might not be centered perfectly
over the object. So the RPN estimates a delta change to refine the anchor box
to fit the object better. Using the RPN predictions, the model picks the top
anchors that are likely to contain objects and refine their location and
size.If several anchors overlap too much, it keeps the one with the highest
foreground score and discards the rest (Non-max Suppression). After getting
the final proposals (regions of interest) then, it passes to the next stage.

Figure 4: Example, a) Anchor boxes per object b) Top three anchor boxes pre
object

(a) (b)

### ROI Classifier & Bounding Box Regressor

This architecture was introduced by ([9]), by removing the SVM classifiers for
training to a regression layer and classification layer. This stage runs on
the regions of interest (ROIs) proposed by the RPN.

First, the network processes all images with several convolutional and max-
pooling layers to produce convolutional feature map. Then a region of interest
(RoI) pooling layer extracts a fixed length feature vector for each object
proposal from the feature map. Then, each feature vectored into the fully
convolutional layers and produces softmax probability over different object
classes with background class and output the probability of confidence for
each object.

This module generates two outputs for each ROI: Class: The class of the object
in the ROI. Since this network is deeper and has the capacity to classify
regions to specific classes like person, car, chair,etc. It can also generate
a background class, which causes the ROI to be discarded.

Bounding Box Refinement: Further refine the location and size of the bounding
box to encapsulate the object.

RoI pooling layer: Due to the bounding box refinement step in the RPN, the ROI
boxes can have different sizes. ROI pooling is used to crop a part of a
feature map and resize it to a fixed size. The RoI pooling layer uses max
pooling to convert the features inside any valid region of interest into a
small feature map with a fixed spatial extent of H×W (e.g., 7 × 7), where H
and W are layer hyper-parameters that are independent of any particular RoI
[9].

### Segmentation Masks

The mask network is the addition that the Mask R-CNN introduces. The mask
branch is a convolutional network that takes the positive regions selected by
the ROI classifier and generates masks for them. The generated masks are low
resolution: 28x28 pixels. But they are soft masks, represented by float
numbers, so they hold more details than binary masks. The small mask size
helps to keep the mask branch light. During training, the ground-truth masks.
scale down to 28x28 to compute the loss, and during inference, the predicted
masks scale up to the size of the ROI bounding box and that gives us the final
masks, one per object.

### Mask R-CNN Losses

The loss function in Mask R-CNN is the weighted sum of different losses at
different stages of the model. Five general losses occur during training the
weight [2].

Rpn class loss: Occur when an improper classification of anchor boxes by the
Region Proposal Network. This loss increases when the multiple objects are not
being detected by the model in the last output. This loss determines how well
the Region Proposal Network separates background with objects.
Rpn bbox loss: This loss occurs during the localization accuracy of the RPN.
The object is being detected but the bounding box is not properly fit with the
position of the object.
Mrcnn class loss: Occurs when an improper classification of objects that are
present in the region proposal. This is to be increased in case the object is
being detected from the image, but misclassified. It determines how well the
Mask RCNN recognizes each class of object.
Mrcnn bbox loss: This loss occurs during the localization of the bounding box
of the identified class. The loss increases when the correct classification of
the object is done, but localization is not precise.
Mrcnn mask loss: This loss corresponds to masks created on the identified
objects, which is related to how well the Mask RCNN segment objects.

### Public datasets

As for current dataset resources, the Ecole Centrale Paris (ECP) Facades
dataset [29] and the eTRIMS [15] database are the two most relevant public
facade datasets we are aware of as both have been used as a benchmark for
facade parsing in previous literature.

The ECP dataset consists of 104 annotated images which are rectified and
cropped facades of Haussmannian style buildings in Paris. Images are annotated
with seven classes (balcony, chimney, door, roof, shop, wall and window).
Ground truth object segmentation assigns each pixel to either one of the
annotated objects or background. The object segmentation is represented as an
image that consists of a colourmap (Fig. 5). The original annotations were
improved by [19] to better fit the ground truth.

Figure 5: Example from ECP dataset, a) the input image b) the annotation

(a) (b)

Note. Image from ([29])
(a) (b)

Figure 6: Example from eTRIMS, (a) the input image (b) the annotation

The eTRIMS consists of 60 annotated street view images but different from the
ECP dataset the images are not rectified and the facades are of various
architectural styles. Images are annotated with eight classes (sky, building,
door, vegetation, car, road, pavement and window). Ground truth object
segmentation assigns each pixel to either one of the annotated objects or
background. The object segmentation is represented as an image that consists
of an array and a color-map matrix (Fig. 6).

After inspecting these dataset we found it unwise to use either datasets for
training our model to detect windows in real street view images because of
multiple reasons. The ECP dataset only contains facades that are manually
rectified and viewed in a front-parallel direction, thus using it as training
data would not be wise since it is not highly representative of real street
view images, which is the imagery we want to detect windows instances in.
However, the most important reasons were that we found both to be imprecise or
even wrongly annotated regarding what exactly defines the window boundaries.
For example, the eTRIMS dataset often annotated the window boundaries to
include not only the window frame but even other details such as the exterior
casing (See Figure 6).

We searched and found a few more public datasets that contained window
annotations but none were of very high quality and many had the same issues as
mentioned above. Therefore we had to take a decision to build a quality
dataset from scratch.

### Our dataset

We decided to use Google Street View images (Map data ©2020 Google) of facades
mainly from central Gothenburg, Sweden. The cropped images were manually
collected with an intention to contain a diversity of typical Swedish urban
building facades that encompassed a large variety of window types.

We annotated them manually with the VGG Image Annotator (VIA) tool [7]. There
exists a lot of other tools to annotate images, e.g. LabelMe, RectLabel,
LabelBox and COCO UI, but we chose VIA because of its efficiency and
simplicity. For example, it runs in most modern web browsers so it does not
require any other installation or setup. Regarding the annotated segmentation
masks there is not a universally standard data format to store them. Some
datasets save them as PNG images, while others store them in XML-format, and
so on. The VIA tool saves its annotations in a JSON-format file where each
mask is a set of polygon points (Figure LABEL:tab:via).

Figure 7: Two examples from images in our dataset annotated with the VIA tool

(a) (b)

Since we tried to be consistent to follow each window’s exact window frame and
not to include other window parts as window boundaries it made annotation a
very time-consuming and complex task, especially as specific decisions had to
be taken for each window type. This precision and attention to detail made
annotating the earlier images very gruelling, but once we got used to both the
user interface and figured out most of the different window types and their
frames, we were annotating one window about every 5-10 seconds. However, there
were a lot of windows in some pictures and we wanted to be as consistent and
precise as possible.

After gradually expanding our dataset we ended up with a collection of 100
images that contained 1540 annotated windows (an average of 15,4 windows per
image). We then divide our dataset into a training set (80 images) and a
validation set (20 images), i.e. we split the generated JSON-file into two
different files (Figure LABEL:tab:json) with their corresponding images.

Figure 8: Examples of the JSON-files generated with the VIA tool

### Optimization Methods

To use the Mask R-CNN framework we used an open-source code implementation
(Matterport) of it written on Python 3, Keras, and TensorFlow by [1]. Then we
hosted the implementation on the Google Colab platform for free GPU usage and
easy of shareability amongst other positive factors.

Assume to start with the structure of the Mask R-CNN as described in the paper
[11] and the source code provided in [1], we needed to investigate the
similarities and differences between the Matterport implementation running on
Keras and TensorFlow and the official paper implementation build on the Caffe
deep learning framework.

From our understanding, the implementation followed the paper with only a few
exceptions and those were done mostly for code simplicity and generalization.

For example, to support training multiple images per batch, the Matterport
implementation resizes all images to the same size and preserve the aspect
ratio by padding with zeros if the image is not a perfect square.

To support the training of different dataset’s annotations they chose to
generate bounding boxes in the code on the fly, thus making image
augmentations easier to apply. The downside is a slight decrease in bounding
box placement accuracy, about only 2% of bounding boxes differed by 1px or
more.

The paper uses a more aggressive learning rate of 0.02, but for the Matterport
implementation that was too high and causes the weights to explode in
TensorFlow as it computes gradients differently from Caffe. To mitigate this
problem, gradient clipping was implemented.

With an understanding of the differences between Matterport’s and the paper’s
features, parameters, hyper-parameters and so on, we could start to optimize
our model. The optimization adjustments and modifications were updated
continuously during the whole process, but we can perhaps give some intuition
on how they were done by describing them as iteratively separate steps. For
example, optimization of the network structure with its corresponding
configuration can be considered ’low level’, while optimizing training was
done with more regards to the properties of our dataset. Finally, in inference
optimization, we look at the outcome and try to adjust our result within the
context of our research problem or goal. Furthermore, each step often needed
distinct configuration adjustment depending on which model weights and dataset
changes were used, i.e. training resumed with COCO-weights, augmentation flip,
augmentation affine and so on, all needed different configurations to train
properly.

Figure 9: Examples of the many configurations options in the Matterport
implementation

##  4 EXPERIMENTS

In this section, we will first present some metrics used in our experiments,
then give insight into the thought process of some of our structure and
configuration optimization, training optimization and inference optimization.
Finally, we will present our experimental settings and some of the best
results we achieved.

### Metrics

To use the networks inference results for detecting windows and analysing the
performance of our model, we are interested in detecting every existing window
as well as reducing the number of wrongly detected windows. In other words,
recall is a crucial value to improve. To evaluate the images overall detection
results, we first use IoU (Intersection over Union) to measure how much our
predicted boundary overlaps with the ground truth (the real object boundary).
A predefined IoU threshold of 0.5 or greater is used for classifying the
prediction as a true positive otherwise a false positive.

Figure 10: Examples of IoU.

Then we use the typical precision and recall rate:

$\begin{split}Precision&=\frac{TP}{TP+FP}\\\ \end{split}$ (1)
$\begin{split}Recall&=\frac{TP}{TP+FN}\end{split}$ (2)

where, TP, FP, FN denotes the true positive, false positive and false
negative, respectively.

Finally, we use the mean Average Precision (mAP):

$mAPI\&=\int_{0}^{1}P(R)\,dR$ (3)

where the P represent the precision rate and and R the recall rate.

### Structure and configuration optimization

Overall, we found it a challenging task to test and tune the many parameters,
mainly due to the adaptive process of such a large search space, the
continuous updates to our training data, but also the time it takes to train
and do inference to calculate the accuracy. However, we will try to summarize
the more important optimization decision we took.

We tested many different ‘lower-level’ configurations of the neural network,
for example, we investigated Mask R-CNN in combination with both the ResNet50
and the ResNet101 Backbone as described in [11]. Those experiments showed that
the latter often yielded better results.

We kept the base configuration of input images of size 1024x1024 px for best
accuracy even if some of our images were a bit smaller since the model resized
them automatically. Further testing could benefit separate tests with batches
of different image sizes.

We found the gradient clipping norm to be appropriately set a 5 with a
learning rate at 0.001 and momentum of 0.9 and so on.

By plotting each test we could observe that training and validation losses
indicated overfitting, i.e. we got an increasingly lower training loss but the
validation loss at certain points started to fluctuate or stagnate (Figure
11).

Figure 11: Examples of plots from our training

To overcome overfitting we experimented with weight decay and dropout amongst
other things. Unfortunately, GridSearch with cross-validation, to
automatically search for the optimal hyper-parameter values, required more
computational power than we had access to, but we believe a valid option given
the appropriate resources.

Another option and the most intuitive, to prevent overfitting is often to
extend the dataset. However, this was neither feasible given the remaining
project time and also did not fit well with our interest in investigating the
performance of a relatively small dataset.

Finally, we used k-fold cross-validation, which is a resampling procedure used
to evaluate machine learning models on a limited data sample, without any
major improvements on our test set in terms of accuracy.

### Training optimization

For the training optimization we assume the given Mask-RCNN loss function as:

$L=\alpha L_{1}+\beta L_{2}+\gamma L_{3}+\delta L_{4}+\epsilon L_{5}$ (4)

where the first loss is the RPN class loss and the second the RPN bounding box
prediction loss. The other three losses correspond to those of the head neural
networks classes, bounding box and mask loss respectively. The training
optimization goal is to find a corresponding configuration such that it
minimizes the overall loss. This is an adaptive process of matching the
results of the loss function with the given dataset and the parameters as a
result of a trial process.

Given a loss $L_{i}$ and equation (4), optimize
$K=(\alpha,\beta,\gamma,\delta,\epsilon)\in R^{5}$ and
$L_{1},L_{2},L_{3},L_{4},L_{5}$ loss functions so that the average precision
of our model so that the average precision of our model is improved with a
threshold of $AP_{5}{}_{0}$ :

$\max\limits_{K,L_{1},L_{2},L_{3},L_{4},L_{5}}AP_{50}$ (5)

Ultimately, the final choice of parameters chosen depends on the structure
configurations discussed in the previous section and also on our input data .
While technically for K any combination of real values can be combined:

$(\alpha,\beta,\gamma,\delta,\epsilon)\sim
n.(\alpha,\beta,\gamma,\delta,\epsilon),\forall n\in\mathcal{R}$ (6)

This means that even though the total loss value changes, our accuracy of the
detection is dependent on the loss gradient and not the loss value.
Furthermore, several other configuration parameters need to be tested and
changed to improve accuracy. The practical approach for tuning loss weights
was to start with a default weight of 1 for each loss and evaluate the model
on the validation set by inferring the model performance image by image. We
had to evaluate the true positives, false positives, and false negatives
combined with IoU of their masks and their score.

Transfer learning

In our approach, we specify that training should start from weights pre-
trained on COCO dataset. In other words, we are using transfer learning, which
means we don’t need to train a new model from scratch and that instead utilize
a lot of the already learned features from the COCO dataset, which contains
around 120K images. After the training for 40 epochs with k-fold cross-
validation have been completed, we save the model’s parameters and evaluate
the new weights (one for each epoch is generated).

Augmentation

To train our model with more training data, we used augmentation on our
existing data as annotating new images would take too long. Since the polygons
are calculated internally this was easy to implement with the imgaug library.

First, we resume training, on the weights with the lowest training loss from
the previous training, with augmentation that flip/mirror each image in the
training set horizontally right/left. Then we again save the model’s
parameters and evaluate the performance of the new weights.

Then we resume training, on the weights with the lowest training loss from the
previous training, with affine which transforms each image in the training set
by rotation (-45, 45) and shearing (-16, 16). Finally, we save the model’s
parameters and evaluate the performance of the new weights with running
inference on the weights that yielded the lowest training loss.

### Inference optimization

In the process of optimizing the trained network, we constantly adjusted and
modified the training set and other parameters. Since the output of the
inference is a list of possible detection boxes and masks, whose probabilities
are larger than our set threshold of detection minimum confidence, we need to
adjust accordingly with regards to the properties of the test set images and
the overall accuracy. We found a detection minimum confidence of 0.9 to yield
as good trade-off of the precision and recall values for our windows.

##  5 Results

We perform our test result experiments on the eTrims dataset and not on the
ECP dataset. The reason being that the ECP dataset doesn’t contain street-view
images and the images are all manually rectified and viewed in a frontal
direction, i.e. making the window detection task relatively easy.

However, to properly use the eTRIMS dataset with our model, we had to refine
their annotation to overcome inaccuracies and mistakes, and also accommodate
our defined window boundary which follows the window frame. We, therefore,
annotated 10 randomly chosen images for our test set to evaluate our model’s
performances. Therefore, any comparisons should note these differences.

Both the training and inference was done on Google Colab with the free Tesla
K80 GPU of about 12GB. We have overall trained and evaluated more than 60
models with different configurations. Out of those we present the best three
models with regards to the highest mAP on our test set (Table 1) and for
visualization we present the inference done on the model achieving the highest
mAP below (Figure 12).

TABLE I:
A comparison of pixel accuracies on the eTRIMS dataset. Accuracies are shown in percentage. Class |  [1] |  [2] |  [3] |  [4] |  [5] |  Our* |  Our** |  Our***
---|---|---|---|---|---|---|---|---
Window[%] |  75 |  73 |  71 |  86 |  90.91 |  91.93 |  94.55 |  94.76
Figure 12:
Window detection results from a model trained on coco-weights, then resumed
with fliplr.

Image Ground truth Prediction Overlap
image green colors random colors green/red colors
image captions: class captions: class/score captions: score/IoU

##  6 CONCLUSIONS

In this paper, we propose to use the state-of-the-art Mask R-CNN framework
[11] as an end-to-end network capable of instance segmentation without further
machinery. Previous facade parsing detection work often only produced semantic
segmentation of a categorical class (e.g. door, balcony, window etc.), while
Mask R-CNN is capable of detecting and producing a bounding box and
segmentation mask for each window instance.

Experimental results show that our suggested approach can with only a
relatively small but precise dataset train the network—only with transfer
learning and augmentation—to produce instance segmentation of windows and a
comparable result with prior state-of-the-art window detection approaches,
even without pre- or post-optimization techniques.

As far as we know, we are the first to have annotated a street-view windows
dataset by consistently trying to follow the window’s frame as the window
boundary and not to include other window parts. We believe that our consequent
and precise annotation of windows helped our model training. We also hope it
can benefit the research community as a free resource for more precise window
detection.

##  7 FUTURE WORK

In this paper, we have got a sense of how to tune the hyper-parameters
effectively with our training data. However, further work is recommended to
investigate this with platforms, such as Wandb or Onepanel Jobs for easier
coordination, synchronization and tracking of configurations.

Another possibility is to extend the dataset with more high-quality
annotations of window types were the model has low accuracy.

Further work could also benefit from training more facade classes, such as
building, balcony, doors, chimney, to reduce false positives, false negatives.
The same goes for the window class, i.e. splitting it into distinct window
type classes, such as double-hung, transom etc. A model capable of a more
complete facade parsing we believe will also make application easier as the
model has a finer granularity of the facades properties. A caution is that
this would require well-reasoned and well-defined class boundaries for each
corner case of every facade structure.

To further refine the results and perhaps mitigate some of the typical
limitations we experienced, such as dealing with occlusion, overconfident
false positives predictions, occasional mask overlaps, future work should
investigate both in pre- and post-optimization methods of previous approaches
or testing their novel approaches. Also, code refactoring and optimization,
e.g. rewriting some Python code in TensorFlow to bring all the benefits of the
TensorFlow, Keras ecosystem to the model are desired. There is also always
room for improvements.

## ACKNOWLEDGMENTS

This paper was the final written report in the course DIT891 Project in Data
Science. The course is offered within the Applied Data Science Master’s
Programme by the University of Gothenburg. We would like to thank Shirin
Tavara, teacher and the course responsible and Richard Johansson, teacher.

A special thanks goes to our supervisors Alexander Hollberg, Sanjay Somanath
and Yinan Yu for their congeniality and support.

## References

  * [1] Waleed Abdulla  “Mask R-CNN for object detection and instance segmentation on Keras and TensorFlow https://github. com/matterport”  In _Mask _ RCNN_ , 2017
  * [2] Waleed Abdulla  “Instance Segmentation with Mask R-CNN and TensorFlow”, https://engineering.matterport.com/splash-of-color-instance-segmentation-with-mask-r-cnn-and-tensorflow-7c761e238b46, 2018
  * [3] Konstantinos Bacharidis, Froso Sarri and Lemonia Ragia  “3D building façade reconstruction using deep learning”  In _ISPRS International Journal of Geo-Information_ 9.5 Multidisciplinary Digital Publishing Institute, 2020, pp. 322
  * [4] Vijay Badrinarayanan, Alex Kendall and Roberto Cipolla  “1SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation”  Citeseer, 2015
  * [5] Shashi Bala and Ajaya Kumar  “A brief review of image segmentation techniques”  In _International Journal of Advanced Research in Electronics and Communication Engineering (IJARECE) Volume_ 5, 2016, pp. 1448–1454
  * [6] Liang-Chieh Chen et al.  “Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs”  In _IEEE transactions on pattern analysis and machine intelligence_ 40.4 IEEE, 2017, pp. 834–848
  * [7] Abhishek Dutta and Andrew Zisserman  “The VIA annotation software for images, audio and video”  In _Proceedings of the 27th ACM international conference on multimedia_ , 2019, pp. 2276–2279
  * [8] Radwa Fathalla and George Vogiatzis  “A deep learning pipeline for semantic facade segmentation”, 2017
  * [9] Ross Girshick  “Fast r-cnn”  In _Proceedings of the IEEE international conference on computer vision_ , 2015, pp. 1440–1448
  * [10] Feng Han and Song-Chun Zhu  “Bottom-up/top-down image parsing with attribute grammar”  In _IEEE transactions on pattern analysis and machine intelligence_ 31.1 IEEE, 2008, pp. 59–73
  * [11] Kaiming He, Georgia Gkioxari, Piotr Dollár and Ross Girshick  “Mask r-cnn”  In _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 2961–2969
  * [12] Han Hu et al.  “Fast and Regularized Reconstruction of Building Fa$\backslash$c $\\{$c$\\}$ ades from Street-View Images using Binary Integer Programming”  In _arXiv preprint arXiv:2002.08549_ , 2020
  * [13] T Karmarkar  “Regional Proposal network (RPN)—Backbone of Faster R-CNN”  In _Aug_ 18, 2018, pp. 6
  * [14] David Koch et al.  “Visual estimation of building condition with patch-level ConvNets”  In _Proceedings of the 2018 ACM Workshop on Multimedia for Real Estate Tech_ , 2018, pp. 12–17
  * [15] Filip Korc and Wolfgang Förstner  “eTRIMS Image Database for interpreting images of man-made scenes”  In _Dept. of Photogrammetry, University of Bonn, Tech. Rep. TR-IGG-P-2009-01_ , 2009
  * [16] Hantang Liu, Jialiang Zhang, Jianke Zhu and Steven CH Hoi  “Deepfacade: A deep learning approach to facade parsing”, 2017
  * [17] Jonathan Long, Evan Shelhamer and Trevor Darrell  “Fully convolutional networks for semantic segmentation”  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 3431–3440
  * [18] Wenguang Ma and Wei Ma  “Deep window detection in street scenes”  In _KSII Transactions on Internet and Information Systems (TIIS)_ 14.2 Korean Society for Internet Information, 2020, pp. 855–870
  * [19] Markus Mathias et al.  “Automatic architectural style recognition”  In _ISPRS-International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences_ 3816, 2011, pp. 171–176
  * [20] Pascal Müller, Gang Zeng, Peter Wonka and Luc Van Gool  “Image-based procedural modeling of facades”  In _ACM Trans. Graph._ 26.3, 2007, pp. 85
  * [21] Keiller Nogueira, Otávio AB Penatti and Jefersson A Dos Santos  “Towards better exploiting convolutional neural networks for remote sensing scene classification”  In _Pattern Recognition_ 61 Elsevier, 2017, pp. 539–556
  * [22] Dhruv Parthasarathy  “A brief history of CNNs in image segmentation: From R-CNN to Mask R-CNN”  In _Available online: blog. athelas. com/a-brief-history-of-cnns-in-image-segmentation-from-r-cnn-to-mask-rcnn-34ea83205de4 (accessed on 24 February 2019)_ , 2017
  * [23] Otávio AB Penatti, Keiller Nogueira and Jefersson A Dos Santos  “Do deep features generalize from everyday objects to remote sensing and aerial scenes domains?”  In _Proceedings of the IEEE conference on computer vision and pattern recognition workshops_ , 2015, pp. 44–51
  * [24] Rahmani and K.  “Facade Interpretation from Images. Fakultät für Informatik at Universität der Bundeswehr München.”, 2020
  * [25] Michal Recky and Franz Leberl  “Windows detection using k-means in cie-lab color space”  In _2010 20th International Conference on Pattern Recognition_ , 2010, pp. 356–359  IEEE
  * [26] Shaoqing Ren, Kaiming He, Ross Girshick and Jian Sun  “Faster R-CNN: towards real-time object detection with region proposal networks”  In _IEEE transactions on pattern analysis and machine intelligence_ 39.6 IEEE, 2016, pp. 1137–1149
  * [27] Matthias Schmitz and Helmut Mayer  “A convolutional network for semantic facade segmentation and interpretation”  In _The International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences_ 41 Copernicus GmbH, 2016, pp. 709
  * [28] F Sultana, A Sufian and P Dutta  “Evolution of Image Segmentation using Deep Convolutional Neural Network: A Survey. arXiv 2020”  In _arXiv preprint arXiv:2001.04074_ , 2020
  * [29] Olivier Teboul, Loic Simon, Panagiotis Koutsourakis and Nikos Paragios  “Segmentation of building facades using procedural shape priors”  In _2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition_ , 2010, pp. 3105–3112  IEEE
  * [30] Zhuowen Tu, Xiangrong Chen, Alan L Yuille and Song-Chun Zhu  “Image parsing: Unifying segmentation, detection, and recognition”  In _International Journal of computer vision_ 63.2 Springer, 2005, pp. 113–140
  * [31] Andreas Wendel, Michael Donoser and Horst Bischof  “Unsupervised facade segmentation using repetitive patterns”  In _Joint Pattern Recognition Symposium_ , 2010, pp. 51–60  Springer
  * [32] Peng Zhao et al.  “Rectilinear parsing of architecture in urban environment”  In _2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition_ , 2010, pp. 342–349  IEEE
  * [33] Song-Chun Zhu, Rong Zhang and Zhuowen Tu  “Integrating bottom-up/top-down for object recognition by data driven Markov chain Monte Carlo”  In _Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No. PR00662)_ 1, 2000, pp. 738–745  IEEE

ren2015faster