# Towards Universal Texture Synthesis by Combining Texton Broadcasting with
Noise Injection in StyleGAN-2

Jue Lin
Northwestern University, ECE Department
2145 Sheridan Rd, Evanston, IL 60208, USA
<EMAIL_ADDRESS>Gaurav Sharma
University of Rochester, ECE Department
Rochester, NY
<EMAIL_ADDRESS>Thrasyvoulos N. Pappas
Northwestern University, ECE Department
2145 Sheridan Rd, Evanston, IL 60208, USA
<EMAIL_ADDRESS>
###### Abstract

We present a new approach for universal texture synthesis by incorporating a
multi-scale texton broadcasting module in the StyleGAN-2 framework. The texton
broadcasting module introduces an inductive bias, enabling generation of
broader range of textures, from those with regular structures to completely
stochastic ones. To train and evaluate the proposed approach, we construct a
comprehensive high-resolution dataset that captures the diversity of natural
textures as well as stochastic variations within each perceptually uniform
texture. Experimental results demonstrate that the proposed approach yields
significantly better quality textures than the state of the art. The ultimate
goal of this work is a comprehensive understanding of texture space.

|  |  |  |  |
---|---|---|---|---|---
|  |

Figure 1: Textures synthesized at $256\times 256$ and $512\times 512$ pixel
resolutions. The proposed model is trained on $256\times 256$ texture crops,
and can produce textures ranging from stochastic to structured with variable
resolutions, conditioned on the latent representation.

##  1 Introduction

Texture is an important visual attribute for human perception and computer
vision, as it provides critical information for material appearance,
understanding, and characterization [2]. Texture understanding, and texture
analysis/synthesis in particular, is important for a variety of applications,
including image analysis and compression, computer graphics, virtual reality,
and human-computer interaction. The study of texture analysis/synthesis must
take into account the stochastic nature of texture and human perception, which
is the ultimate judge of texture quality. Accordingly, a number of authors
have proposed algorithms for texture analysis/synthesis that are based on
multiscale frequency decompositions, which have been used to model early
visual processing in the brain [5, 35, 15, 37, 50, 9, 39]. On the other hand,
the stochastic nature of texture necessitates a statistical approach for
texture analysis. The most complete parametric approach for texture
analysis/synthesis has been proposed by Portilla and Simoncelli [39], who
developed a statistical model for synthesizing a broad set of textures based
on a steerable filter decomposition. Even though their goal was to provide a
universal statistical model that parametrizes the space of visual textures, it
falls short of successfully modeling all textures. Thus, the complete
mathematical and perceptual characterization of texture remains an open
problem.

The resurgence of neural networks has stimulated broad interest in both
academia and industry, promising to push the frontiers in a wide variety of
research areas. One of the most successful models is the generative
adversarial network (GAN), which has yielded impressive results in numerous
applications, such as generation of human faces, anime characters, objects and
scenes, image-to-image translation, image super-resolution, and inpainting.
However, the problem of texture modeling has not received as much attention.
The focus of this work is on utilizing the GAN framework for texture
synthesis, and ultimately, a more complete mathematical and perceptual
characterization of the space of visual textures. We present a new approach
for universal texture synthesis that introduces a multiscale texton
broadcasting module in the StyleGAN-2 framework, which enables the generation
of a wide variety of textures, both regular and stochastic.

A generally accepted definition of visual texture is an image that is
spatially homogeneous and usually contains repeated elements, often with
random variations in position, orientation, and color [38]. The repeated
elements in a texture are commonly referred to as textons, a term introduced
by Bela Julesz, one of the pioneers of texture analysis and perception [21].
Texture appearance can range from completely regular periodic structure to
completely random variations, and typically consists of both periodic
structure and stochastic variations. However, in our experiments we found that
the failure of the StyleGAN-2 framework to capture the periodic aspect of
textures manifests itself in two ways: first, the trained model generates
disproportionately fewer periodic textures when randomly sampled from the
latent space; and second, once we identify a latent vector corresponding to a
periodic texture, injecting the model with different samples of multi-scale
noise cannot produce distinguishable texture crops, that is, the synthesized
periodic structure is often “anchored” in a fixed location. We will analyze
both of these shortcomings and will show that they can be mitigated by the
introduction of the texton broadcasting module.

The main contributions of this work are the following:

  * •

We propose a novel multi-scale texton broadcasting module for the SyleGAN-2
that, in combination with the noise injection, provides appropriate inductive
bias to enable universal high-quality texture synthesis, spanning from regular
to completely stochastic ones.

  * •

For effective training and evaluation, we create a comprehensive dataset of
high-resolution textures, representative of the diversity of natural textures
as well as the variations within individual textures.

  * •

We introduce an intuitive measure to quantify how well a network models
distribution within individual texture, and we emphasize the importance of the
often overlooked multi-scale noise injection and our texton broadcasting in
this regard.

  * •

We demonstrate that unlike conventional analysis-synthesis techniques,
traversing in our trained latent space exhibits smooth transitions between
homogeneous textures instead of incoherent spatial mixtures.

##  2 Related work

###  2.1 Texture analysis and synthesis

As we discussed, traditional parametric approaches for texture
analysis/synthesis have been primarily based on subband decompositions. Heeger
and Bergen [15] match the histograms of a steerable filter decomposition to
achieve impressive texture synthesis results; however, their approach is
limited to stochastic textures. Portilla and Simoncelli [39] developed a more
elaborate model that relies on a wide variety of subband statistics for
synthesizing a much broader set of textures. Their goal is to parametrize the
space of visual textures based on a universal statistical model. Parametric
approaches based on Markov Random Fields (MRFs) have also shown great
potential for texture synthesis [28, 33]. Nonparametric approaches for texture
synthesis build larger textures from seed patches [10, 26]; however, this does
not involve any texture modeling.

Deep learning-based approaches for texture analysis/synthesis are fairly
diversified. Gatys _et al_. [11] use a pretrained VGG-19 network to extract
features from a given texture and then, starting with a random image,
synthesize another texture that matches the Gram-matrix representation of the
original texture. Ulyanov _et al_. [45, 46] use a fast feed-forward generative
network to achieve similar performance. Li _et al_. [29] further develop a
feed-forward generative network to synthesize multiple diverse textures.
Implicit neural representation (INR) methods [6, 30, 34, 43] can be used to
predict RGB values of a texture using raw pixel coordinates as input [16, 32,
36]. However, most non-GAN methods perform texture reconstruction without a
meaningful latent space representation. For that we turn to GANs.

###  2.2 Generative adversarial networks

Goodfellow _et al_. [13] introduced an adversarial formulation for training a
generative model, whereby a second discriminator network provides feedback by
determining whether a generated image comes from the actual data distribution
or not. The WGAN [3] uses the Wasserstein (or earth mover) distance between
probability distributions, which improves stability of learning and alleviates
mode collapse. Further improvements come from alternative formulations of
Lipschitz continuity for the WGAN, e.g., gradient penalty [14] and spectral
normalization [31].

The application of GANs to texture synthesis was introduced by Jetchev _et
al_. [18], who proposed the spatial GAN for synthesis of textures of arbitrary
size. However, like Gatys _et al_. [11], the functionality of the spatial GAN
is limited to producing equivalent textures, that is, it generates one model
per texture. The periodic spatial GAN (PSGAN) by Bergmann _et al_. [4]
represents the first attempt to learn a latent space that is capable of
generating periodic textures by injecting a periodic pattern (with a random
phase term) at the lowest resolution of the generator network. However, they
trained on a very small dataset and, as we will show below, the quality of the
resulting textures is mixed.

###  2.3 StyleGAN models

Building on the progressive-GAN [49], StyleGAN, proposed by Karras _et al_.
[23], introduces an intermediate latent space, which is used to adjust the
style of the image at each convolution layer, and also adds explicit noise
injection at each layer. This allows the disentanglement of global features
(like pose, face shape and human identity) and local stochastic variations
(like hair and skin texture). StyleGAN-2 [24] was proposed to address some
noticeable blob-like visual artifacts in StyleGAN, by redesigning the
normalization and eliminating progressive training.

Following the Karras _et al_. work, which was applied to faces, objects, or
scenes, one line of research sought to interpret the latent space induced by
StyleGAN-like models. Built upon loss functions containing location
information or pretrained attribute classifiers, an input image can be
inverted into a latent code $\mathbf{z}$, and visual property manipulation can
be achieved via navigating in the latent space [1, 40, 41]. However, due to
lack of an equivalent model, a proper GAN inversion technique is still missing
for textures. Although loss functions for faces, objects, or scenes have been
proposed, they rely on pixel-to-pixel correspondences, which are ill-suited
for textures because of their stochastic nature.

Another line of research sought to investigate the impact of the internal
components of the StyleGAN models. Xu _et al_. [48] found that zero-padding
implicitly encodes location, which works for faces, objects, or scenes but is
not desirable for textures. Choi _et al_. [7] addressed the spatial bias in
StyleGAN-2 by adding sinusoidal embeddings, commonly used in transformers [47,
12, 25].

Finally, the recently proposed StyleGAN-3 [22] replaces the bottom tensors
with Fourier features [44] and designs operators to enhance
translation/rotation equivariance. It is interesting to note that, for better
performance on human faces, StyleGAN-3 removes the noise injection, which we
found to be highly beneficial for textures.

##  3 Method

###  3.1 Preliminaries

We denote by $\Omega=\\{\Omega_{1},\Omega_{2},\cdots\\}$ the set of all
textures. Intuitively, each texture $\Omega_{i}$ has a set of basic elements
or textons[20] (_e.g_., a brick in a wall). We can aggregate textons from
$\Omega$ and establish a universal texton codebook $T_{\Omega}$[27]. Any
texture $\Omega_{i}$ can be represented as a spatial repetition of textons
drawn from $T_{\Omega}$, and the selected textons are adjusted to fit certain
properties (_e.g_., shapes). Moreover, stochastic variation is often present
within each $\Omega_{i}$ (_e.g_., layout of bricks). We can therefore model
texture distribution in two parts: inter-texture distribution
$P_{\Omega_{i}\sim\Omega}(\Omega_{i})$ for distinct textures, and intra-
texture distribution $P_{\mathbf{I}\sim\Omega_{i}}(\mathbf{I}_{i})$
conditioned on the same texture $\Omega_{i}$, where we assume $\Omega_{i}$ is
sufficiently large and $\mathbf{I}$ is a random crop from $\Omega_{i}$.

Conceptually, a StyleGAN-2 generator $G_{\boldsymbol{\theta}}$ can approximate
both $P_{\Omega_{i}\sim\Omega}(\Omega_{i})$ and
$P_{\mathbf{I}\sim\Omega_{i}}(\mathbf{I})$ with
$P_{\mathbf{z},\mathbf{n}}(G_{\boldsymbol{\theta}}(\mathbf{z},\mathbf{n}))$
and
$P_{\mathbf{n}|\mathbf{z}}(G_{\boldsymbol{\theta}}(\mathbf{z},\mathbf{n})|\,\mathbf{z})$
respectively, where $\mathbf{z}$ is drawn from a D-dimensional normal
distribution and $\mathbf{n}$ is the multi-scale spatial noise. Due to
randomness in the recurrence of textons, most textures exhibit stochasticity
as well as periodicity. The stochasticity is well captured by $\mathbf{n}$,
producing subtle changes in textured regions, _e.g_., hair and skin. However,
our experiments show that StyleGAN-2 generator is biased towards synthesizing
stochastic textures, even though the regular ones occupy a comparable portion
in the training set, which we refer to as inter-texture mode collapse.
Moreover, the injected noise $\mathbf{n}$ is inadequate for rendering spatial
shifts of the periodic structures, which we refer to as intra-texture mode
collapse. We empirically found that the first problem is relatively easier to
alleviate.

Nonetheless, the intra-texture mode collapse suggests a strong entanglement
between spatial location and latent space. Such entanglement is arguably
acceptable for images in other domains, _e.g_., face positioned in the center,
but the spatial layout of textons should be stochastic. Most operations in
StyleGAN-2 (see Fig. 2) do not explicitly encode spatial information, _e.g_.,
convolutions, upsampling. We have identified the bottom $512\times 4\times 4$
tensor and zero-padding in coarse layers as the causes of spatial anchoring of
visible structures (see Fig. 5(a)).

(a) StyleGAN-2 generator

(b) Multi-Scale Texton Broadcast

Figure 2: Comparison between StyleGAN-2 and our proposed modifications. All
feature maps within the same blue area, enclosed with dashed line, share the
same spatial resolution, indicated on the upper right corner. The ”$\oplus$”
is an element-wise sum.

###  3.2 Texton broadcasting module

To capture the periodic nature of textures, we design a texton broadcasting
(TB) module that simulates the spatial repetition of physical textons, as
illustrated in Fig. 3. First, a trainable texton $\mathbf{v}_{i}$ is
replicated along spatial dimensions. Then the intensity of each
$\mathbf{v}_{i}$ is modulated with respect to a broadcast map (BM), modelled
as a 2D sinusoidal wave:

$\displaystyle\text{BM}_{i}(h,w)=\textbf{A}_{i}\sin(2\pi\varsigma(\mathbf{f}_{i})^{T}\begin{bmatrix}h\\\
w\end{bmatrix}+\boldsymbol{\varphi}_{i}+\Delta)+\textbf{B}_{i}$ (1)

where $\forall\,i\in\\{1,2,\ldots,P\\},\ \textbf{f}_{i}=[f_{ih},f_{iw}]^{T}$,
$\boldsymbol{\varphi}_{i}$, $\textbf{A}_{i}$ and $\textbf{B}_{i}$ represent
the frequency, initial phase, amplitude, and offset of the 2D sine, all of
which are trainable parameters, and $P$ denotes the total number of textons in
a TB module. We use $[h,w]^{T}\in\\{1,2,\ldots,H\\}\times\\{1,2,\ldots,W\\}$
as the spatial coordinate vector; $H$ and $W$ can be dynamically sized but are
fixed during training. We use an element-wise sigmoid function
$\varsigma(\cdot)$ to map $[f_{ih},f_{iw}]^{T}$ into the interval $(0,1)$,
since discrete-time frequency is periodic in $\omega$ with period $2\pi$,
_i.e_., $\sin(\omega n)=\sin((\omega+2\pi)n),\forall n\in\mathbb{Z}$. Note
that $\Delta$ is uniformly sampled from $[0,2\pi)$ to simulate random shift,
and is shared among all $\text{BM}_{i}$ within the same module to have a
unified phase control. The output Y of a module is the sum across all
broadcast/modulated $\mathbf{v}_{i}$:

$\displaystyle\textbf{Y}(c,h,w)=\sum_{i=1}^{P}\textbf{v}_{i}(c)\otimes\text{BM}_{i}(h,w)$
(2)

where $\otimes$ combines replication of all textons $\mathbf{v}_{i}$ with the
modulation by $\text{BM}_{i}$, as illustrated in Fig. 3. Note that the use of
random phase $\Delta$ is critical, otherwise the module is just another
deterministic spatial anchor.

###  3.3 Multi-scale texton broadcasting

Figure 3: Mechanism of Texton Broadcasting. Each cubelet represents a scalar,
the value of which is indicated by its color. For ease of illustraion, we
assume a black cubelet $=0$, a white cubelet $=1$, and other colors are
arbitrary. The ”$\otimes$” duplicates $\textbf{v}_{i}$ spatially and
$\textbf{BM}_{i}$ along channel dimension, then an element-wise multiplication
follows. The ”$\oplus$” is an element-wise sum.

By replacing the bottom $512\times 4\times 4$ tensor with a TB module, the new
model is now capable of producing images of variable sizes. However, the zero
padding at the bottom layers still causes a residual spatial anchoring effect
when synthesizing images of higher resolution. To mitigate this issue, we
couple each noise injection (NI) module with a TB module (see Fig. 2(b)), upto
and including layers with spatial size of $64\times 64$. Such hierarchical
placement of NI and TB is aligned with the multi-scale nature of textures. The
spatial size $H$ and $W$ of each TB module is configured to match the
resolution at its corresponding layer except the bottom one, which is set to
$4\times 4$ to produce a final $256\times 256$ image compatible with the
discriminator during training111Code can be found here.

###  3.4 Training objective functions

The inter-texture mode collapse is in fact closely related to the general
definition of mode collapse in the literature, where a model yields only a few
distinguishable images. We adopt the Wasserstein distance as the loss
function, and impose a gradient penalty [14] to enforce the Lipschitz
continuity on the discriminator network (or critic) $D_{\boldsymbol{\phi}}$,
parametrized by $\boldsymbol{\phi}$. The losses for $D_{\boldsymbol{\phi}}$
and $G_{\boldsymbol{\theta}}$ are given as:

$\displaystyle\mathcal{L}(\boldsymbol{\phi})=$ $\displaystyle\
\mathbb{E}_{(\textbf{z},\textbf{n},\Delta_{G})}\\{D_{\boldsymbol{\phi}}(G_{\boldsymbol{\theta}}(\mathbf{z},\mathbf{n},\Delta_{G}))\\}-$
$\displaystyle\ \ \ \
E_{\Omega_{i}\sim\Omega}\\{\mathbb{E}_{\textbf{I}|\Omega_{i}}\\{D_{\boldsymbol{\phi}}(\textbf{I})\\}\\}$
(3) $\displaystyle\mathcal{L}(\boldsymbol{\theta})=$
$\displaystyle-\mathbb{E}_{(\textbf{z},\textbf{n},\Delta_{G})}\\{D_{\boldsymbol{\phi}}(G_{\boldsymbol{\theta}}(\mathbf{z},\mathbf{n},\Delta_{G}))\\}$
(4)

where $\mathbf{z}$ and $\mathbf{n}$ are drawn from normal distribution,
$\Delta_{G}$ denotes the set of all $\Delta$ injected across
$G_{\boldsymbol{\theta}}$. Note that in (3), multiple crops I are sampled from
the same texture $\Omega_{i}$, which helps $D_{\boldsymbol{\phi}}$ learn
intra-texture distribution explicitly.

(a) PSGAN: suboptimal texture reconstructions

(b) StyleGAN-2: visually competitive, but severe intra-texture mode collapse,
discussed in later sections

(c) Proposed approach: well defined textures, good balance between stochastic
and structured/periodic textures

Figure 4: Lists of textures generated by models trained on our dataset

###  3.5 GAN inversion for textures

Once a reliable generator is obtained, texture analysis can be performed via
GAN inversion. Inverting a latent space trained on textures poses distinct
challenges. Most generative models produce images with well-defined objects of
interest, _e.g_., a face or bed. However, for textures, everything is of
interest and the stochastic placement of textons renders commonly used
location-wise losses, such as L2 loss and content loss [19], inapplicable.
Instead, we use the style loss by Gatys _et al_. [11] who extract the Gram
matrices of feature maps from a pretrained VGG-16 network $\Phi$[42]. The
process of searching for optimal latent code $\mathbf{z}^{*}$ for any texture
$\Omega_{i}$ is given via:

$\displaystyle\mathcal{L}(\mathbf{I}_{1},\mathbf{I}_{2})=\sum_{\mathnormal{l}}[\frac{\Phi_{\mathnormal{l}}(\mathbf{I}_{1})\Phi_{\mathnormal{l}}(\mathbf{I}_{1})^{T}-\Phi_{\mathnormal{l}}(\mathbf{I}_{2})\Phi_{\mathnormal{l}}(\mathbf{I}_{2})^{T}}{C_{\mathnormal{l}}\times
N_{\mathnormal{l}}^{2}}]^{2}$ (5)
$\displaystyle\mathbf{\mathbf{z}}^{*}=\arg\min_{\mathbf{z}}\mathbb{E}_{(\mathbf{n},\Delta_{G},\mathbf{I}\sim\Omega_{i})}[\mathcal{L}(G_{\mathbf{\mathbf{\theta}}}(\mathbf{z},\mathbf{n},\Delta_{G}),\;\mathbf{I})]$
(6)

where $\mathnormal{l}$ is layer index, and $\Phi_{\mathnormal{l}}$,
$N_{\mathnormal{l}}$, $C_{\mathnormal{l}}$ represent feature maps, spatial
size and channel at $\mathnormal{l}$-th layer of $\Phi$, respectively. Note
that a common practice in the literature of GAN inversion is to use
$\mathbf{w}$ or $\mathbf{w}^{+}$ as the optimization variable instead of the
raw latent code $\mathbf{z}$, where $\mathbf{w}$ comes from the style mapping
network with $\mathbf{z}$ as input, and $\mathbf{w}^{+}$ is the aggregation of
all $\mathbf{w}$ across different layers.

| FID $\downarrow$
---|---
PSGAN[4] | 133.72$\pm$2.46
StyleGAN-2[24] | 72.48$\pm$1.86
Proposed | $\mathbf{70.05\pm 1.41}$
Training set | 1.58$\pm 1.10$
Table 1: Quantitative evaluation of different methods

##  4 Experimental results

(a) StyleGAN-2: Exhibits intra-texture mode collapse

(b) Proposed approach: No intra-texture mode collapse

Figure 5: Different crops sampled from the same latent vector, and their
associated standard deviation map $\sigma_{\mathbf{z}}$ on the right. Higher
intensity in the $\sigma_{\mathbf{z}}$ map indicates lower likelihood of
anchoring artifacts, while darker regions indicate severe intra-texture mode
collapse.

###  4.1 Dataset construction

There exist multiple texture datasets (_e.g_. [8]) in the literature.
Unfortunately, most of them either lack inter-texture diversity or sufficient
spatial resolution for intra-texture distribution. To address this limitation,
we collect a more comprehensive dataset of textures. We include a texture
image into the dataset if it fits the following criteria: (1) It is
perceptually uniform; (2) it contains sufficient independent $256\times 256$
crops to learn the intra-texture distribution; (3) each $256\times 256$ crop
contains enough texton repetitions (at least 5 in each dimension) to form a
texture; and (4) it is under either Creative Commons Public Domain license
(CC0) or custom website license for free academic use. After extensive search
on multiple stock image websites, we obtain 500 quality texture images,
ranging from natural to artificial, periodic to stochastic, and fine-grained
to coarse, and each image contains roughly 20 to 50 independent crops.

###  4.2 Training settings

In our experiment, we uniformly sampled 2 crops of $256\times 256$ from each
texture in a mini-batch of 8 distinct textures, to explicitly enforce the
learning of intra-texture distribution on the discriminator network
$D_{\boldsymbol{\phi}}$. The same trick can be applied to the generator
network $G_{\boldsymbol{\theta}}$ by feeding multiple noise samples
$\mathbf{n}$ conditioned on the same latent variable $\mathbf{z}$, but this
resulted in a slower convergence of the generator and no significant
performance gain. We adopted the Wasserstein distance in (3) and (4) as the
losses with gradient penalty $=0.01$ to impose Lipschitz continuity, and the
discriminator parameters $\boldsymbol{\phi}$ were updated twice, followed by
one generator update ($3\times 10^{5}$ generator iterations in total). We
disabled the mixing regularization and the path-length regularization as they
are time-consuming. For other hyperparameters, we followed the default
protocols of StyleGAN-2, including latent space dimensionality $D=512$,
learning rate $=0.002$, Adam optimizer, and exponential moving average of
$G_{\boldsymbol{\theta}}$.

Regarding the settings of TB, each module has $P=16$ learnable texton vectors.
We applied the TB module to all layers of spatial resolution up to $64\times
64$ to prevent high-frequency artifacts. The spatial size of the bottom TB
module was set to $4\times 4$ during training, and the number of channels of
each texton vector was fixed at C$=512$. The spatial size as well as channel
size of all remaining TB modules were designed to match the feature maps of
their preceding blocks of Styled Conv as shown in Fig. 2(b). At test time, the
final output image resolution can be varied by simply modifying the spatial
size of the bottom module.

###  4.3 Comparisons

(a) Original training strategy of StyleGAN-2, _i.e_. non-saturating loss,
shows inter-texture mode collapse

(b) StyleGAN-2 with Wasserstein distance + discriminator noise

Figure 6: Inter-texture mode collapse experiments

We compare our approach with PSGAN, which also aims to model periodic
textures, and the baseline StyleGAN-2. All methods were trained on our
dataset, and we provide a list of textures sampled from each method. As shown
in Fig. 4, the proposed module outperforms PSGAN and StyleGAN-2 in terms of
diversity and image quality. We will provide more samples in the supplementary
material. We also evaluate the FID [17] for each model in Table 1. The
distribution of training set was computed by sampling 20 crops from each
texture. For each method, 1000 latent codes were sampled, and 20 different
texture crops were obtained for each code. Each method was repeated 5 times,
and the 95% confidence intervals are also provided.

#### Intra-texture mode collapse.

Figure 5 shows that the proposed approach yields a significant improvement in
terms of intra-texture diversity, that is, the reconstructed textures do not
all correspond to the same crop. The figure also shows the map of pixel-wise
standard deviations $\sigma_{\mathbf{z}}\in\mathbb{R}^{H\times W}$ conditioned
on $\mathbf{z}$ defined by:

$\sigma_{\mathbf{z}}=\sqrt{\mathbb{E}_{\mathbf{n}|\mathbf{z}}[(G_{\mathbf{\mathbf{\theta}}}(\mathbf{z},\mathbf{n})-\mathbb{E}_{\mathbf{n}|\mathbf{z}}[G_{\mathbf{\mathbf{\theta}}}(\mathbf{z},\mathbf{n})])^{2}]}$
(7)

Pixels with higher intensities in the $\sigma_{\mathbf{z}}$ map are less
likely to suffer from anchoring artifacts, while darker regions in the map
indicate severe intra-texture mode collapse.

To quantify the intra-texture mode collapse of a synthesized texture, we now
introduce a novel and intuitive measure we call thresholded invariant pixel
percentage (TIPP), which is calculated as follows:

$\text{TIPP}_{t}(\mathbf{z})=\frac{1}{H\times
W}\sum_{h}^{H}\sum_{w}^{W}\mathbbm{1}(\sigma_{\mathbf{z}}[h,w]\leq t)$ (8)

where $\sigma_{\mathbf{z}}\in\mathbb{R}^{H\times W}$ is the pixel-wise
standard deviation map defined in (7), $t$ is the threshold, and
$\mathbbm{1}(\cdot)$ is an indicator function that returns 1 if the condition
in parenthesis is met and 0 otherwise. Intuitively, $\text{TIPP}_{t}$
calculates the percentage of pixels with $\sigma_{\mathbf{z}}<t$. Pixels with
low standard deviation have a strong invariance, and thus the higher the TIPP
value the worse the intra-texture mode collapse.

To evaluate each model, we sampled 1000 latent codes $\mathbf{z}$, synthesized
20 crops per $\mathbf{z}$, and calculated TIPP averaged over the codes
$\mathbf{z}$ for different thresholds. TIPP(%) can also be calculated for the
training set, where the averaging is over crops rather than latent codes. For
that, we sampled 20 crops from each texture in the training set. Figure 9
shows that the proposed method consistently outperforms StyleGAN-2 in terms of
intra-texture mode collapse.

(a) Ground truth

(b) L2 loss

(c) Content loss

(d) Gram matrices

Figure 7: Loss functions for texture GAN inversion

#### Inter-texture mode collapse.

We designed an experiment to investigate inter-texture mode collapse. We
selected 8 textures of distinguishable properties, and independently sampled 8
fixed latent codes, from which the generator samples during training. Our
expectation was that StyleGAN-2 would overfit the data, allocating each latent
code to a different texture, which would indicate strong disentanglement
between latent code and noise injection. To our surprise, as shown in Fig.
6(a), the original StyleGAN-2 strategy with non-saturating loss performs
poorly on this small set. We then applied the Wasserstein distance combined
with adding Gaussian noise ($\sigma=0.01$) to the discriminator input and
found that it improves diversity as shown in Fig. 6. Therefore, we adopted the
Wasserstein distance and discriminator noise as the default training
configuration for both the proposed method and StyleGAN-2 (included in the
results shown in Fig. 4).

(a) Traversing in proposed $\mathbf{Z}$ latent space

(b) Based on Portilla and Simoncelli parametrization[38]

Figure 8: Comparison between texture interpolation based on proposed latent
space and Portilla and Simoncelli parametrization.

###  4.4 GAN inversion

In this experiment, we used $relu1\\_2$, $relu2\\_2$, $relu3\\_3$ and
$relu4\\_3$ layers to extract feature maps and compute Gram matrices via (6).
For completeness, we also investigated the efficacy of L2 loss and content
loss [19]. For all losses, the inversion was performed on the same
optimization variable $\mathbf{w}$. We avoided the use of $\mathbf{w}^{+}$ as
we prefer a unified global representation for textures. Other shared
hyperparameters include learning rate of 0.001, a total of 5000 iterations,
and Adam optimizer with default settings. Results are shown in Fig. 7 and are
consistent with our expectations, as the characterization of a texture should
minimize location-wise correspondence due to its stochastic nature.

Figure 9: TIPP measure of intra-texture mode collapse shows that proposed
method outperforms StyleGAN-2, and disabling random phase $\Delta$ causes
significant drop in TIPP.

###  4.5 Latent space interpolation

Interpolation in the latent space can be performed by traversing in the
$\mathbf{Z}$ or $\mathbf{W}$ space, which is mapped from input $\mathbf{z}$
via a MLP. In both learned latent spaces, there is a gradual transition from
one texture to the other via intermediate uniform textures, as shown in Fig.
8. In contrast, the interpolations produced by the classic Portilla and
Simoncelli [39] method consist of a mixture of the endpoint images rather than
homogeneous intermediate textures.

###  4.6 Ablation studies

#### Is random phase noise $\Delta$ needed?

To demonstrate the necessity of $\Delta$, we re-trained the model with fixed
phase. As expected, the module degenerates to another form of spatial
anchoring, severely damaging the intra-texture diversity. As shown in Fig. 9,
such setting degrades TIPP by a considerable margin.

#### Multi-Scale texton broadcasting.

We demonstrated the importance of multi-scale texton broadcasting by removing
all but the bottom TB modules. Such an ablated model is capable of generating
quality textures with the same size as the training images. However, when
generalized to arbitrary sizes, only the 4 corners, shown in Fig. 10(a),
resemble their low-resolution counterpart and the model fails to render the
central area. We attribute this to the zero-padding at bottom layers,
consistent with the study in [48] where zero-padding is shown to have an
implicit encoding of location. By introducing our module in a multi-scale
fashion, the influence of zero-padding can be substantially reduced.

#### Mapping latent code to TB via a MLP?

We also conducted experiments with the trainable parameters in the TB module
linked to the latent code as in PSGAN via a MLP[4], _e.g_.,
$f_{i}=MLP_{f_{i}}(\textbf{z})$. We empirically found that such a model is
unstable to train and struggles to converge. We hypothesize that if the module
parameters are conditioned on the latent code, then the entanglement between
location and latent space is aggravated because the resulting broadcast maps
are directly affected by the latent code, and the latent space no longer works
in a channel-wise manner, contrary to the design philosophy of StyleGAN.

(a) TB Module Only at the Bottom

(b) Multi-scale TB Modules

Figure 10: Multi-scale texton broadcasting experiments

##  5 Limitations

While the proposed approach yields promising results, understanding textures
remains a difficult task as such a powerful model still fails to encompass a
training set of 500 textures. The immense diversity across textures as well as
within textures poses unique challenges, which result in a relatively high FID
compared with images of other domain. For texture analysis, optimization-based
iterative inversion is less time-efficient compared with direct statistics
computation, and systematic procedures of exploring the learned latent space
is still pending. In future work, we aim to expand our dataset as well as
address fundamental limitations of our model for a more thorough study.

##  6 Conclusions

We performed an in-depth analysis of StyleGAN-2, utilized the noise injection
as a means of modeling intra-texture distribution, recognized its shortcoming
of spatial anchoring artifacts, and designed a model that has an architectural
inductive bias more aligned with textures. To demonstrate the effectiveness of
our module, we created a high-resolution dataset, proposed a novel measure
(TIFF) that quantifies the anchoring artifacts, and conducted extensive
experiments to both quantitatively and qualitatively evaluate the proposed
modules. The proposed work facilitates universal textures synthesis and
enables a potentially unified formulation of texture analysis and synthesis
with help of deep neural networks.

## References

  * [1] Rameen Abdal, Yipeng Qin, and Peter Wonka.  Image2StyleGAN: How to embed images into the StyleGAN latent space?  In ICCV, 2019.
  * [2] Edward H. Adelson.  On seeing stuff: The perception of materials by humans and machines.  In Bernice E. Rogowitz and Thrasyvoulos N. Pappas, editors, Human Vision and Electronic Imaging VI, volume 4299 of Proc. SPIE, pages 1–12, San Jose, CA, Jan. 2001.
  * [3] Martin Arjovsky, Soumith Chintala, and Léon Bottou.  Wasserstein generative adversarial networks.  In ICML, 2017.
  * [4] Urs Bergmann, Nikolay Jetchev, and Roland Vollgraf.  Learning texture manifolds with the periodic spatial GAN.  In ICML, 2017.
  * [5] D. Cano and T. H. Minh.  Texture synthesis using hierarchical linear transforms.  Signal Processing, 15:131–148, 1988.
  * [6] Zhiqin Chen and Hao Zhang.  Learning implicit fields for generative shape modeling.  CVPR, 2019.
  * [7] Jooyoung Choi, Jungbeom Lee, Yonghyun Jeong, and Sungroh Yoon.  Toward spatially unbiased generative models.  In ICCV, 2021.
  * [8] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi.  Describing textures in the wild.  In CVPR, 2014.
  * [9] Jeremy S. De Bonet and Paul A. Viola.  A non-parametric multi-scale statistical model for natural images.  Adv. in Neural Info. Processing Systems, 9, 1997.
  * [10] Alexei A. Efros and Thomas K. Leung.  Texture synthesis by non-parametric sampling.  In Proc. Seventh Intl. Conf. Computer Vision (ICCV), volume 2, pages 1033–1038, Kerkyra, Greece, Sept. 1999.
  * [11] Leon Gatys, Alexander S Ecker, and Matthias Bethge.  Texture synthesis using convolutional neural networks.  In NeurIPS, pages 262–270, 2015.
  * [12] Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin.  Convolutional sequence to sequence learning.  In ICML, 2017.
  * [13] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In NeurIPS, 2014.
  * [14] Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville.  Improved training of wasserstein GANs.  In NeurIPS, 2017.
  * [15] David J. Heeger and James R. Bergen.  Pyramid-based texture analysis/synthesis.  In Proc. Int. Conf. Image Processing (ICIP), vol. III, pages 648–651, Washington, DC, Oct. 1995.
  * [16] Philipp Henzler, Niloy J. Mitra, and Tobias Ritschel.  Learning a neural 3D texture space from 2D exemplars.  In CVPR, 2020.
  * [17] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  GANS trained bya two time-scale update rule converge to a local Nash equilibrium.  In NeuIPS, 2017.
  * [18] Nikolay Jetchev, Urs Bergmann, and Roland Vollgraf.  Texture synthesis with spatial generative adversarial networks, 2016.
  * [19] Justin Johnson, Alexandre Alahi, and Li Fei-Fei.  Perceptual losses for real-time style transfer and super-resolution.  In ECCV, 2016.
  * [20] Bela Julesz.  Binocular depth perception of computer-generated patterns.  Bell System Technical Journal, 39:1125–1162, 1960.
  * [21] Bela Julesz.  Textons, the elements of texture perception and their interactions.  Nature, 290:91–97, 1981.
  * [22] Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Alias-free generative adversarial networks.  In NeurIPS, 2021.
  * [23] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In CVPR, 2019.
  * [24] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of StyleGAN.  In CVPR, 2020.
  * [25] Alexander Kolesnikov, Alexey Dosovitskiy, Dirk Weissenborn, Georg Heigold, Jakob Uszkoreit, Lucas Beyer, Matthias Minderer, Mostafa Dehghani, Neil Houlsby, Sylvain Gelly, Thomas Unterthiner, and Xiaohua Zhai.  An image is worth 16x16 words: Transformers for image recognition at scale.  In ICLR, 2021.
  * [26] Vivek Kwatra, Arno Schödl, Irfan Essa, Greg Turk, and Aaron Bobick.  Graphcut textures: Image and video synthesis using graph cuts.  ACM Transactions on Graphics, SIGGRAPH, 22(3):277–286, 2003.
  * [27] Thomas Leung and Jitendra Malik.  Representing and recognizing the visual appearance of materials using thee-dimensional textons.  International Journal of Computer Vision, 43(1):29–44, 2001.
  * [28] Elizaveta Levina and Peter J. Bickel.  Texture synthesis and nonparametric resampling of random fields.  The Annals of Statistics, 34(4):1751 – 1773, 2006.
  * [29] Yijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Xin Lu, and Ming-Hsuan Yang.  Diversified texture synthesis with feed-forward networks.  In CVPR, 2017.
  * [30] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger.  Occupancy networks: Learning 3D reconstruction in function space.  In CVPR, 2019.
  * [31] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida.  Spectral normalization for generative adversarial networks.  In ICLR, 2018.
  * [32] Michael Oechsle, Lars Mescheder, Michael Niemeyer, Thilo Strauss, and Andreas Geiger.  Texture fields: Learning texture representations in function space.  In ICCV, 2019.
  * [33] R. Paget and I.D. Longstaff.  Texture synthesis via a noncausal nonparametric multiscale markov random field.  IEEE Transactions on Image Processing, 7(6):925–931, 1998.
  * [34] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove.  Deepsdf: Learning continuous signed distance functions for shape representation.  In CVPR, 2019.
  * [35] M. Porat and Y. Y. Zeevi.  Localized texture processing in vision: Analysis and synthesis in Gaborian space.  IEEE Trans. Biomed. Eng., 36(1):115–129, 1989.
  * [36] Tiziano Portenier, Siavash Arjomand Bigdeli, and Orcun Goksel.  GramGAN: Deep 3D texture synthesis from 2D exemplars.  In NeurIPS, 2020.
  * [37] Javier Portilla, Rafael Navarro, O. Nestares, and A. Tabernero.  Texture synthesis-by-analysis based on a multiscale early-vision model.  Optical Engineering, 35(8):2403–2417, 1996.
  * [38] Javier Portilla and Eero P. Simoncelli.  Texture modeling and synthesis using joint statistics of complex wavelet coefficients.  In Proc. IEEE Wksp. Statistical and Computational Theories of Vision, Fort Collins, CO, USA, June 1999.
  * [39] Javier Portilla and Eero P. Simoncelli.  A parametric texture model based on joint statistics of complex wavelet coefficients.  Int. J. Computer Vision, 40(1):49–71, Oct. 2000.
  * [40] Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou.  Interpreting the latent space of GANs for semantic face editing.  In CVPR, 2020.
  * [41] Yujun Shen, Ceyuan Yang, Xiaoou Tang, and Bolei Zhou.  InterFaceGAN: Interpreting the disentangled face representation learned by GANs.  TPAMI, 2020.
  * [42] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  In ICLR, 2015.
  * [43] Vincent Sitzmann, Julien N.P. Martel, Alexander W. Bergman, and Gordon Wetzstein.  Implicit neural representations with periodic activation functions.  In NeurIPS, 2020.
  * [44] Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng.  Fourier features let networks learn high frequency functions in low dimensional domains.  NeurIPS, 2020.
  * [45] Dmitry Ulyanov, Vadim Lebedev, Andrea Vedaldi, and Victor Lempitsky.  Texture networks: Feed-forward synthesis of textures and stylized images.  In ICML, 2016.
  * [46] Dmitry Ulyanov, Andrea Vedaldi, and Victor S. Lempitsky.  Improved texture networks: Maximizing quality and diversity in feed-forward stylization and texture synthesis.  In CVPR, 2017.
  * [47] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In NeuIPS, 2017.
  * [48] Rui Xu, Xintao Wang, Kai Chen, Bolei Zhou, and Chen Change Loy.  Positional encoding as spatial inductive bias in GANs.  In CVPR, 2021.
  * [49] Dan Zhang and Anna Khoreva.  Progressive augmentation of GANs.  In NeurIPS, 2019.
  * [50] S. Zhu, Y. N. Wu, and D. Mumford.  Filters, random fields and maximum entropy (FRAME): Towards a unified theory for texture modeling.  In IEEE Conf. Computer Vision and pattern Recognition, pages 693–696, 1996.


#  CVEH: A Dynamic Framework To Profile Vehicle Movements To Mitigate Hit And
Run Cases Using Crowdsourcing

Attiq ur Rehman<EMAIL_ADDRESS>Asad Waqar Malik
<EMAIL_ADDRESS>Anis ur Rahman  Sohail Iqbal
<EMAIL_ADDRESS>Ghalib Ahmed Tahir<EMAIL_ADDRESS>National University Of Science And Technology

###### Abstract

In developed countries like the USA, Germany, and the UK, the security forces
used highly sophisticated equipment, fast vehicles, drones, and helicopters to
catch offenders’ vehicles. Whereas, in developing countries with limited
resources such schemes cannot be utilized due to management cost and other
constraints. In this paper, we proposed a framework called CVEH that enables
developing countries to profile the offender vehicle movements through
crowdsourcing technique and act as an early warning system to the law forcing
agencies. It also engages citizens to play their role in improving security
conditions. The proposed CVEH framework allows Vehicle-to-Infrastructure (V2I)
communication to monitor the movement of the offender’s vehicle and shared its
information with the Command and Control (CC) centre. The CC centre projects
the path and engages nearly located law enforcement agencies. CVEH is
developed and evaluated on android smartphones. Simulations conducted for this
study exhibit the effectiveness of our framework.

##  1 Introduction

Crowdsourcing is an outsourcing model within which people or organizations
obtain contributions from the users (maybe over the internet) to get specific
ideas and services. This kind of outsourcing to attain higher results by
dividing the tasks between the users had already gained success even before
the digital era. However, subcontracting in crowdsourcing is discriminated
against by the statement that in crowdsourcing, an undetermined group of
people do the jobs rather than a specifically tasked cluster as crowdsourcing
uses a mix of top-down and bottom-up approaches. The blessings of
crowdsourcing embrace better prices, agility, fineness, tractability,
measurability, or variety. Figure 1 [28] depicts the design architecture of
crowdsourcing.

Figure 1: Crowdsourcing Architecture

The target of our research is to use crowdsourcing for the advantage of the
community. In underdeveloped nations, it’s impractical to install a gigantic
foundation for the monitoring and tracking systems in this manner, CVEH is one
of the conceivable ways to deal with defeat framework issue and help in
accomplishing social manageability.

###  1.1 Motivation

In the post 9/11 world, the market for camera-based video surveillance
products has expanded manifold with its primary use as an area monitoring
system by public safety and security organizations [1]. Though such systems
are cheap and ubiquitous, there are limitations and underlying challenges in
deploying the practical system. These challenges involve the cost of wiring,
low power hardware and the development of detailed management tools [2]. Once
installed, the human resource required to monitor many real-time feeds
simultaneously is expensive. In many cases, this becomes ineffective due to
the limits of human surveillance [3]. Here automated video surveillance helps
by automatically analyze a scene and generate alarms for the manual supervisor
to take action. No doubt, such systems based on video analytics are the
future, but very few are robust enough to work in variations and unfavourable
conditions [4]. Moreover, existing systems such as VSAM (Video Surveillance
and Monitoring) [5] faces several challenges in their underlying components,
which impacts the overall efficiency.

###  1.2 Problem Statement

With the urban growth in Pakistan, there is a substantial increase in the
number of crimes involving vehicles, ranging from road violations to serious
crimes. In practice, there are many sensor-based solutions primarily used for
road traffic estimation and congestion detection. Moreover, some of these
systems identify minor traffic violations, e.g. an automated parking metering
system issuing a parking ticket on an expired meter but not used by law
enforcement agencies.
For crime detection, the use of automated vehicle surveillance to identify and
track suspicious vehicles is becoming more and more important. Though wide-
area camera-based video surveillance can help perform this task with
increasing workload and other overhead expenses, manual visual tracking is
impossible. Mainly such video surveillance is treated as a post-incident
forensic analysis tool, with investigators going through hordes of recorded
data, which is a serious problem impacting the effectiveness of doing real-
time surveillance. An automated system capable of triggering alarms in this
situation can be useful; however, depending on the nature of the scene, the
system may generate many false alarms that need to be assessed by a human
supervisor [6] [7] [8] [9].
For crime detection and deterrence, as a replacement for the old system of
using a police gazette with rewards, the distributed human intelligence model
can work more effectively. For instance, after the Boston Marathon bombing, an
open call to share the content of the incident helped to apprehend the
culprits within few days [10]. Similarly, they employed the same approach
after the London riots in 2011 [11]. Though there are controversies, from
false accusations to rumours, a moderated platform can be used to put citizens
immediately to work by sifting through video recordings with the investigators
directing the analysis by shutting down misinformation [12]. Moreover, law
enforcement and the public can use collected data to identify areas with high
crime rates.

###  1.3 Contribution Of The Study

In this paper, our contribution is on vehicle identification and reporting
mechanisms through crowdsourcing. We proposed CVEH to profile the movement of
offender vehicles. The command and control centre utilizes its limited
resources efficiently by identifying the offender direction and project its
path through extrapolation. Our approach provides an affordable and easy to
maintain surveillance system to the developing countries which cannot afford
complex and advanced surveillance systems that use advanced and expensive
technologies. Our proposed methodology is capable of providing surveillance
capabilities to small streets and neighbourhoods where the installation of
dedicated surveillance systems is not feasible. It also helps create a sense
of ownership and responsibility among the public and therefore increases the
coherence between law enforcing agencies and the general public.

###  1.4 Outline

The paper is outlined as follows. In section 3 we have reviewed the literature
and related work done in this field. Section 4 describes the framework and
proposed design while conclusion and future work is described in section 5.

##  2 Related Work

Smart cities merge information and communication technology and the Internet
of Things securely. The concept is to attach all the city assets in a grid for
monitoring, controlling and utilizing efficiently. In the recent decade, there
has been significant work in various domains of smart cities.
Ibrar Y. et al. [13] discussed the communication and connectivity problem in
smart cities. The backbone of smart cities is the communication channel, and
in the presence of millions of devices, communication technology cannot
provide flawless connectivity. Moreover, the authors have also presented the
case study of Stratford, Singapore, etc., as a proof of concept.

M. Victoria Moreno et al. [14] discussed the role of big data in smart cities.
The authors proposed a four-layered architecture for smart cities to handle
data coming from static and dynamic sensors to utilize energy efficiently.
Similarly, Z. Ali et al. [15] proposed an automatic voice complaint system to
monitor the resident of the smart home. The concept is detecting emergency,
especially in homes where no one is available to help aged people. The
proposed detection scheme employs the theory of linear prediction analysis.

Smart transport systems are playing a vital role in the last decade.
Significant progress is done in terms of route optimization, congestion
avoidance. Z. Bhutto et. al [16] used smartphones in place of CCTV cameras to
broadcast live videos to help reduce street crimes. It enables users to stream
live videos of street crimes using3G/4G LTE or a Wi-Fi connection. The live
streaming of the video feed into the police servers. Thus, the people
participated in reporting the crimes to the law enforcement agencies and hence
help in capturing the offenders. The Real-Time Streaming Protocol (RTSP) is
used to transmit data in packets to a media server.

S. Mujtaba et al. [17] proposed a framework to report street crimes using
mobile phones. The initiator can generate a crime report by taking a picture
and sending it to the server with a crime tag. They also sent the Geolocation
of the reports to the server. GPS technology built-in in most mobile phones
tagged the location. It uses 3G/4G or Wi-Fi technology to transmit data to and
from the server. After the report generation, all the registered users can see
the crime location on the map in the app and take precautionary measures.

Divya Lal et al. [18] proposed an app that provides the option to report
crimes by using multiple methods. As a first step to use the app, users need
to register themselves. To report a crime, the user can record a voice
message, capture a video or image and initiate the report. This information is
transmitted along with the location to the police station to keep track of
users. The application periodically sends the GPS location of the user to the
server. Similarly, in [19], Alexey Medvedev et al. proposed a framework called
CityWatchers, using vehicle-mounted surveillance cameras to generate alerts in
case of an accident, traffic jams, etc. Drivers initiate the alerts, and users
receive the incentives based on their participation.
W. A. Agangiba et al. [20] implemented the idea of using the computational
capabilities of smartphones to make them useful for society for crime
detection and reporting. The proposed framework helps in the detection,
reporting and eventually tracking the offenders using smartphones. In [21], M.
Eunus et al. introduced SafeStreet, a smartphone application that used
crowdsourcing and GPS location service. This application helps women to report
sexual harassment cases in public places. It enables women to capture and
share their experiences anonymously. Users can also find a path to a safe
location. It stores All the reported cases on the server with the location and
time of the incident. Based on analytics, it can also recommend safe travel
time.

Wisam A. F. et al. [22] proposed a methodology to capture digits from
dissimilar license plates. The authors proposed model is based on fuzzy rules
to identify the fragmented characters and numerical digits from the same input
sets. Thus, the technique is to identify license plates that are not following
the standard template. Using the hybrid method of the Fuzzy template matching
scheme, the authors attained 90.4% accuracy in an outdoor environment.

Similarly, OpenALPR is an open-source Automatic License Plate Recognition
library available in different languages. Sandro Machado et al. [23] ported
the OpenALPR library to Android smartphones. The library analyzes images and
video streams to identify license plates. The output is the text
representation of any license plate characters. Integration of the automatic
license plate recognition (ALPR) system and crime reporting system on
smartphones is the new concept in reporting and tracking crime incidents using
smartphones.

In [21], M. E. Ali et al. have proposed SafeStreet smartphone-based
application that uses crowdsourcing and GPS location service. Women can report
sexual harassment cases through the application. It provides the ability to
capture and share their experiences anonymously. It also helps users to find a
safe path. They stored all the reported harassment on the server with the
location and time of the incident for further analysis.
Alessandro Amoroso et al. [24] present a model in which people establish an ad
hoc network using smartphones. It implements an optimal and lightweight P2P
communication system using GPS and Wi-Fi functionalities of smartphones.
Matthias Pfriem et al. [25] have present the approach to use smartphones as a
multi-sensor platform in a field operational test for naturalistic driving
study. Modern smartphones offer the sensor capability to detect speed and
position as well as accelerations. These sensors of smartphones measure the
mobility of vehicles. Hence, they do the task by crowdsourcing without using
dedicated data loggers.
Eduardo Romero et al. [26] presents a vision-based vehicle detection and
tracking. The approach employs smartphone cameras as an alternative to other
expensive and complex sensors such as LiDAR or other range-based methods. A
multi-scale proposal and simple geometry consideration of the roads based on
the vanishing point are combined to overcome the computational constraints.
They tested their algorithm on a publicly available road data set, thus
demonstrating its real applicability to autonomous driving.
Akhil M. et al. [7447636] propose an inexpensive and effective solution for
traffic signals. The proposed scheme employs vehicle detectors (VD’s) near
intersections. When a vehicle approaches the intersection, it sequentially
passes the VD’s. VD’s then transmit their unique IDs to the smartphone. The
smartphones then send the IDs and relevant kinematic data to a base station
installed at the intersections. Thus, the base station can accurately compute
the expected times of the vehicles at the intersection. In the event of a
potential collision, the base station can transmit a warning to the
smartphones of the concerned drivers.

##  3 CVEH: A dynamic framework to profile vehicle movements to mitigate hit
and chase cases using crowdsourcing

CVEH allows users to participate, in capturing the offenders, generate alerts
and play its role in law and order implementation. In developed countries,
high-speed vehicles in coordination with helicopters, quadcopters and
surveillance cameras placed at various locations used to capture fast-moving
offenders. Whereas, in developing countries, limited resources are available
to maintain the law and order situation. These countries cannot afford high-
speed vehicles due to their cost and maintenance. Moreover, it is also
unwieldy to install cameras at every intersection. Thus, CVEH provides a
platform to maintain law and order under such conditions through the
crowdsourcing paradigm. Without proper surveillance, in a typical hit-and-run
case, vehicles can disappear in urban areas. Therefore, our proposed
frameworks help in identifying the offender’s location and project its
movement through crowdsourcing. This approach gives an early warning to law
enforcement agencies to take preemptive action to capture the offender. Figure
2 shows the high-level architecture of the CVEH framework. It consists of two
main modules user module and the command control centre, which are explained
as follows.

Figure 2: CVEH Architecture

###  3.1 Proposed Design

The proposed framework facilitates end-users to launch complaints through the
web or mobile applications. The timeline diagram in Figure 3 shows the flow of
events between users and the command control module.
CVEH framework facilitates developing countries to arrest offenders through
crowdsourcing. The system consists of two major modules, the user module and
Command Control (CC) Center. The user module is the main part that executes on
the handheld device. As proof of concept, they designed the CVEH user module
for the Android operating system. As a first step, the user needs to download
and register for the CVEH client application. The registration process
authenticates the user through its national security number. This registration
process is mandatory to avoid any fake/fabricated user input. Thus, the user
can use CVEH on any handheld device issued to its name. Similarly, the CVEH
application can also work on Android-based vehicle-mounted surveillance
systems. We also assume that smartphones are mounted on the vehicles’
dashboards and can act as a surveillance system. Moreover, the pedestrian can
use CVEH in manual surveillance mode.

Figure 3: CVEH application flow

After successful authentication, the user can play its role in sensing the
environment and help to implement the law and order situation. In case of any
incident like theft, mobile snatching, etc. the offenders use vehicles to move
out of the crime location. The CVEH user can capture the crime scene through a
snapshot of the offender and its assisted vehicle. The user can also enter the
required information manually. In pedestrian mode, the user can take a
snapshot of a vehicle and CVEH app imaging module extracts the vehicle
registration number from the image using Optical Character Recognition (OCR)
module. Figure 4 depicts the process.

Figure 4: Snapshot Module

The user can enter other details of the vehicle such as colour, make, model,
etc. It sends all the information to the command and control centre. Moreover,
the command and control centre is linked with the online vehicle repository
system. The shared information also contains the location of the incident
extracted through GPS and the time of the incident. After receiving the
complaint, it generates an alert, which is broadcasted to all the users
present in the near vicinity i.e. within 10 km (configurable). When there is
no response within 15 minutes of the alert, the systems broadcast the message
to all the registered users within the city; this is to identify the initial
location of the suspect’s vehicle. Figure 5 illustrates the registration,
reporting and broadcasting functionality.

Figure 5: Images of Different Screens of Our Prototype Application

On receiving the alert from the command and control centre, the CVEH
application configured to work in a surveillance mode mounted on the vehicle’s
dashboard starts taking the snapshot after a regular interval of time. In this
mode, the CVEH surveillance module takes a snapshot of the surroundings,
extract the vehicle number and matches the received alert. Notification pop-up
to inform the users about the alert, in CVEH handheld device configuration
mode. The user can contribute by manually taking a snapshot of its
surroundings. On successful identifying offenders’ vehicle, it sends the
information to the control room. The control room profile the movement of the
offender vehicle based on the crowd input and alert the law enforcing agencies
that are located at an approachable distance from the offender. It calculates
the projected path at run time through extrapolation. The command and control
centre are equipped with computing machines where it displays the user’s input
in the form of offender movement profiling. The control centre also includes a
database where it stores all the information for further analysis. Inside the
command and control centre, the Law Enforcing Agencies (LEA’s) can also see
the location of their registered users, reported incidents on the map and can
also directly communicate with registered users, as shown in Figure 6.
Moreover, the CVEH application provides easy integration with surveillance
cameras installed at various intersections; thus, we can utilize the installed
camera similarly as discussed above.

Figure 6: Command and Control Center, Markers are Crowd sourced Points

The OPEN ALPR (Android) library extract the vehicle information from the
snapshot. Typically, there can be multiple vehicles in the snapshot.
Therefore, correct extraction of all the vehicle information is deemed
necessary. Moreover, to multicast message/alerts, Firebase Cloud Messaging
(FCM) is used. The OpenALPR is an Open source Automatic License Plate
Recognition library written in C++ with bindings available in other languages.
The library analyzes images and video streams to identify license plates, and
the output is the text representation of any license plate characters. The
Fire-based Cloud Messaging (FCM) is a cross-platform reliable messaging
solution. The FCM facilitates generating notification messages to engage and
retain users. It can support message payload up to 4KB.

###  3.2 Case study: ICT Surveillance System

As proof of concept, we have studied the surveillance system installed in
Islamabad, the capital of Pakistan. In Islamabad city, there is a lack of
surveillance infrastructure, especially in the inner-city area. Only partial
infrastructure is available at entries and exits. We used the standard vehicle
number plates, as shown in Figure 7. Our approach allows CVEH to easily
identify vehicles without using complex machine learning algorithms executing
at handheld devices. Moreover, it installs few surveillance cameras at various
locations that can become part of the CVEH grid.

Figure 7: Standard Vehicle Number Plate for Islamabad

####  3.2.1 Prototype evaluation

To evaluate the feasibility of the proposed CVEH framework, we have analyzed
the system based on its resource consumption. The prototype application
implements the core functionality of the system. We have developed a client
(pedestrian and vehicle dashcam surveillance) mode and command and control
centre. Our application uses the OpenALPR library to provide OCR
functionality. The CVEH module at the client end automatically detects the
license plate from the image and extracts the vehicle registration number from
the license plate. The execution of CVEH at the client end requires memory,
processing power, and a data package. In the benchmark study, we kept the
client application running for consecutive 23 days and continuously measure
the resource usage of the proposed framework. Figure 8 (a) shows the time
taken in generating a request and receiving a response from the control centre
(we reported only on 100 requests). In all the communication, the average time
is around 30 secs (we obtain some of these readings during peak hours). We
also observed the memory, battery and CPU consumption during the experiments.
Figure 8 (d) shows the CPU usage is a little high at places where multiple
vehicles detected in a single snapshot. We reported the average CPU
consumption of the entire day at discrete intervals. Moreover, the memory and
battery usage are less than 4 – 8 per cent respectively. To reduce the memory
consumption, we have stored limited information at the CVEH client end. Most
of the data is stored in the form of the tree structure to reduce the query
time. Similarly, the battery is directly related to resource consumption which
keeps all the resource usage below the threshold value as shown in Figure 8.

Figure 8: Resource usage statistics

To extract the vehicle registration number from the image taken, we have used
a variant of OPEN ALPR for Android to tune it for the new Islamabad Capital
Territory (ICT) license format and used FCM to broadcast the message to the
users. Moreover, we evaluated the proposed system through simulations in
Matlab, where different crowdsource points are placed on a google map to
identify the offenders’ vehicle. In the simulation, we increased the source
points from 100 to 1000, and all the source points are movable.
The results show that with an increase in the crowdsource points, the
probability of identifying vehicle also increases, as shown in Figure 9. It
means more volunteers can significantly improve the performance of the
proposed system. Hence, with the increase in the number of crowdsource points,
the system efficiency is increased. It can do the surveillance with the
minimal cost of infrastructure, which is very helpful for developing nations.

Figure 9:

##  4 Conclusion

In this paper, we proposed the CVEH framework to help developing countries for
the sustainability of society. Our proposed framework is robust and scalable.
To evaluate our architecture, we have implemented a prototype that works on
Android smartphones and benchmarks the resource’s consumption. Moreover, we
have also simulated the effect of crowdsourced points on vehicle detection,
which indicate that the efficiency and accuracy of the system are directly
proportional to the number of crowdsource points. The objective of CVEH is to
utilize volunteers for the benefit of society. In developing countries, it is
not feasible to deploy huge infrastructure for surveillance. Therefore, CVEH
is one of the possible approaches to overcome infrastructure problems and help
in achieving social sustainability.

Sustainable Development Goals (SDG’s): The United Nations has declared a set
of 17 goals as the sustainable Development Goals (SDG’s). These goals and
targets consisted of a large collection of sustainable development matter,
e.g. eliminating poverty and famine, fighting climate change, look after the
oceans and forests, etc. Our application facilitates the users to report the
incidents in a short amount of time. As the law enforcement institutes can
provide swift justice and it helps a country to achieve the sustainable
development goal “Peace, Justice and Strong Institutions” by making the
environment safer for the people.

##  5 Future Work

In the future extension of this work, we will improve the algorithm accuracy,
for detection in low visibility environments like nighttime with low lighting
or during thunderstorms where there are hindrances in clear vision. The
extension of the application on iOS smartphones is in the pipeline. Other
proposed extensions include dashcam support, integration with advanced cameras
available at important public places and integration with national citizens
database to regularize the registration process, provide a safety layer from
prank/fake reports and make identification and surveillance easier for the law
enforcement agencies.

## References

  * [1] K. D. Haggerty, The New Politics of Surveillance and Visibility, University of Toronto Press, 2006.
  * [2] A. Hampapur, L. Brown, J. Connell, A. Ekin, N. Haas, M. Lu, H. Merkl, S. Pankanti, Smart video surveillance: exploring the concept of multiscale spatiotemporal tracking, Signal Processing Magazine, IEEE 22 (2005) 38–51.  doi:10.1109/MSP.2005.1406476.
  * [3] A. Dick, M. Brooks, Issues in Automated Visual Surveillance, in: International Conference on Digital Image Computing: Techniques and Applications, 2003, pp. 195–204.
  * [4] E. Boudihir, INTELLIGENT VIDEO SURVEILLANCE SYSTEM ARCHITECTURE FOR ABNORMAL ACTIVITY DETECTION, The International Conference on Informatics and Applications (ICIA2012) (2012) 102–111.
  * [5] T. Kanade, R. Collins, A. Lipton, H. Fujiyoshi, D. Duggins, A System for Video Surveillance and Monitoring CMU VSAM Final Report (1999) 135.
  * [6] D. C. Brabham,  Using Crowdsourcing In Gov- ernment , IBM Center for the Business of Government (2013) 1–42.
URL https://www.cbs.dk/files/cbs.dk/using_crowdsourcing_in_government.pdf

  * [7] D. Tewksbury, et al., Crowdsourcing Homeland Security: The Texas Virtual BorderWatch and Participatory Citizenship, Surveillance and Society 10 (2012) 249–262.  doi:10.24908/ss.v10i3/4.3464.
  * [8] . W. M, Curated Crowdsourcing in UFO Investigations (2017).
URL
https://www.csicop.org/sb/show/curated_crowdsourcing_in_ufo_investigations#footer

  * [9] D. P. Anderson, J. Cobb, E. Korpela, M. Lebofsky, D. Werthimer, SETI@home: an experiment in public-resource computing 45 (11) (2002) 56–61.
  * [10] B. Schafer, Crowdsourcing and cloudsourcing CCTV surveillance, Datenschutz und Datensicherheit - DuD 37 (07 2013).  doi:10.1007/s11623-013-0173-3.
  * [11] D. Wilcock, Police ’Facewatch’ app targets London riot suspects (June 2012).
URL https://www.independent.co.uk/news/uk/crime/police-facewatch-app-targets-
london-riot-suspects-7887778.html

  * [12] I. Rahwan, S. Dsouza, A. Rutherford, V. Naroditskiy, J. Mcinerney, M. Venanzi, N. Jennings, M. Cebrian, Global Manhunt Pushes the Limits of Social Mobilization, Computer 46 (2013) 68–75.  doi:10.1109/MC.2012.295.
  * [13] I. Yaqoob, I. Hashem, Y. Mehmood, A. Gani, S. Mokhtar, S. Guizani, Enabling Communication Technologies for Smart Cities, IEEE Communications Magazine 55 (01 2017).  doi:10.1109/MCOM.2017.1600232CM.
  * [14] M. V. Moreno, F. Terroso-Sáenz, A. González-Vidal, M. Valdés-Vela, A. F. Skarmeta, M. A. Zamora, V. Chang, Applicability of Big Data Techniques to Smart Cities Deployments, IEEE Transactions on Industrial Informatics 13 (2) (2017) 800–809.  doi:10.1109/TII.2016.2605581.
  * [15] Z. Ali, G. Muhammad, M. F. Alhamid, An Automatic Health Monitoring System for Patients Suffering From Voice Complications in Smart Cities, IEEE Access 5 (2017) 3900–3908.  doi:10.1109/ACCESS.2017.2680467.
  * [16] Z. Bhutto, K. Dahri, I. Lakho, S. Memon, Social Video Streaming (SVS): A prototype application for street crime reporting, 2015, pp. 1–4.  doi:10.1109/CyberSA.2015.7166131.
  * [17] S. M. Raza, A Proposed Solution for Crime Reporting and Crime Updates on Maps in Android Mobile Application, International Journal of Computer Applications 124 (08 2015).  doi:10.5120/ijca2015905312.
  * [18] D. Lal, A. Abidin, N. Garg, V. Deep, Advanced Immediate Crime Reporting to Police in India, Procedia Computer Science 85 (2016) 543–549.  doi:10.1016/j.procs.2016.05.216.
  * [19] A. Medvedev, A. Zaslavsky, S. Khoruzhnikov, V. Grudinin, Reporting Road Problems in Smart Cities Using OpenIoT Framework, 2014.  doi:10.1007/978-3-319-16546-2\\_13.
  * [20] W. Agangiba, M. Agangiba, Mobile Solution for Metropolitan Crime Detection and Reporting 4 (2013) 916–921.
  * [21] M. Ali, S. B. Rishta, L. Ansari, T. Hashem, A. Khan, SafeStreet: empowering women against street harassment using a privacy-aware location based application, in: ICTD ’15.
  * [22] W. A. Faqheri, S. Mashohor, S. D. Ehsan, A Real-Time Malaysian Automatic License Plate Recognition (M-ALPR) using Hybrid Fuzzy, 2009.
  * [23] J. A. G. W. Blajer, M. Krawczyk, The Inverse Simulation Study of Aircraft Flight Path Reconstruction, Transport XVII (3) (2002) 103–107.
  * [24] A. Amoroso, M. Roccetti, G. Marfia, Like vehicles like pedestrians, in an interconnected world, in: 2013 IEEE International Conference on Multimedia and Expo Workshops (ICMEW), 2013, pp. 1–5.  doi:10.1109/ICMEW.2013.6618451.
  * [25] M. Pfriem, F. Gauterin, Employing Smartphones as a Low-Cost Multi Sensor Platform in a Field Operational Test with Electric Vehicles, 2014, pp. 1143–1152.  doi:10.1109/HICSS.2014.148.
  * [26] E. Romera, L. Bergasa, R. Arroyo, A Real-Time Multi-scale Vehicle Detection and Tracking Approach for Smartphones, 2015.  doi:10.1109/ITSC.2015.213.
  * [27] M. Akhil, N. Vasudevan, U. Ramanadhan, A. Devassy, D. Krishnaswamy, A. Ramachandran, Collision avoidance at intersections using vehicle detectors and smartphones, in: 2015 International Conference on Connected Vehicles and Expo (ICCVE), 2015, pp. 389–390.  doi:10.1109/ICCVE.2015.39.