# Self-Healing Distributed Swarm Formation Control Using Image Moments

C. Lin Liu1,2, Israel L. Donato Ridgley2,3, Matthew L. Elwin1,2, Michael
Rubenstein1,2,4, Randy A. Freeman2,3,5, and Kevin M. Lynch1,2,5 This work was
supported by NSF CMMI-2024774.All authors are affiliated with Northwestern
University, Evanston, IL 60208 USA. (emails<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>1Department of Mechanical Engineering, 2Center for Robotics and Biosystems,
3Department of Electrical & Computer Engineering, 4Department of Computer
Science, 5Northwestern Institute on Complex Systems We thank Billie Strong and
Marko Vejnovic for their contributions to the Coachbot hardware.

###### Abstract

Human-swarm interaction is facilitated by a low-dimensional encoding of the
swarm formation, independent of the (possibly large) number of robots. We
propose using image moments to encode two-dimensional formations of robots.
Each robot knows the desired formation moments, and simultaneously estimates
the current moments of the entire swarm while controlling its motion to better
achieve the desired group moments. The estimator is a distributed
optimization, requiring no centralized processing, and self-healing, meaning
that the process is robust to initialization errors, packet drops, and robots
being added to or removed from the swarm. Our experimental results with a
swarm of 50 robots, suffering nearly 50% packet loss, show that distributed
estimation and control of image moments effectively achieves desired swarm
formations.

###### Index Terms:

Distributed control, formation control, image moments, multi-agent systems,
robot swarms.

##  I Introduction

Swarms of hundreds or thousands of robots, each implementing simple behaviors,
can generate complex and useful group behaviors [1, 2, 3]. In applications
such as environmental monitoring and search and rescue, humans may need to
control the swarm formation in real time [4]. To facilitate this one-to-many
interaction, the interface should impose a low cognitive burden, and the human
should not need to directly specify each robot’s motion [5]. In other words,
the intended formation should be representable using a small number of
variables, independent of the number of robots.

In this paper, we introduce low-dimensional representations of swarm
formations based on finite sets of moments of a density distribution. Such
moments can describe the center of mass location of a distribution (first-
order moments), the principal axes of the distribution (second-order moments),
and increasingly fine, high-frequency features of the distribution as the
moment order increases. These distributions can be defined on spaces of
arbitrary dimensions; for example, a drone formation can be described as a
distribution over the six-dimensional space of rigid-body configurations. In
this paper, we focus on formations of robots modeled as points in a plane.
This allows us to take advantage of _image moment_ representations of planar
distributions (Section II), an image compression technique pioneered in
computer vision (Figure 1).

(a) (b)

Figure 1: The left image specifies a desired two-dimensional swarm
distribution. White indicates zero robot density and purple indicates a
constant positive density. Top row: Standard computer vision image
reconstruction using representations of the desired distribution based on
Legendre moments of up to fifth, tenth, 15th, and 20th order, respectively.
Bottom row: The final configurations of a swarm of 1000 simulated robots after
moving so their collective Legendre moments, up to fifth, tenth, 15th, and
20th order, respectively, approximately match the desired Legendre moments.

In our setup, image moments, representing a desired formation, are transmitted
to the robots via broadcast. Each robot continuously senses its position and
communicates with neighbors to estimate the moments of the entire swarm. Using
these estimated moments, the robot locally calculates a motion control that
drives the swarm to achieve the desired moments, and therefore the desired
formation. We require the robots’ estimation and control to be (a)
_distributed_ , meaning that there is no centralized controller that can be a
single point of failure; (b) _scalable_ , meaning that the length of the
messages between the robots is constant (independent of the number of robots);
and (c) _self-healing_ [6], which imparts robustness to packet loss and
transient computation errors, and allows robots to be added to or subtracted
from the swarm at any time.

###  I-A Related Work

####  I-A1 Human-Swarm Interaction

A human can influence swarm shape through neighbor interactions by controlling
the motion of one robot [7, 8] or a subgroup of robots using beacons that
broadcast desired local behaviors such as attracting and repulsing motions
[9]. These methods are not scalable since the human’s influence decreases as
the swarm size increases. Kira and Potter describe a bottom-up control scheme
with virtual agents to influence the swarm through local interactions, and a
top-down control scheme that changes global swarm characteristics to influence
individual robot behaviors [10]. Egerstedt et al. describe one approach where
a human controls leader robots to change the configuration of a leader-
follower swarm, and a second approach where swarm robots are treated as
particles suspended in a fluid comprised of regions that a human can influence
[11].

A human can specify regions of interest for the robots rather than individual
robot locations [12]. Prabhakar et al. specified regions of interest by
shading map areas on a tactile tablet for robots to optimally cover using
ergodic control [13]. A human can also specify time-varying density
distributions [14].

####  I-A2 Image Moments

Image moments encode a density distribution over a plane, $f(x,y)\in$, using
the weighted sum of low-degree basis polynomials on the plane [15, 16]. The
weights are called _moments_. Example orthogonal basis polynomials include
Legendre polynomials, Zernike polynomials, and pseudo-Zernike polynomials. The
higher the degree of the polynomials included in the representation, the
greater the detail of the encoded distribution [17].

In general, pseudo-Zernike moments (PZMs) represent greater detail than
Zernike moments (ZMs) of the same order [18, 19, 20]. The performance of PZMs
vs. Legendre moments (LMs) depends on the image. When reconstructing the
letter “E,” PZMs have lower reconstruction error and higher robustness to
image noise [17]. For an eye iris image, LMs are more robust to white Gaussian
noise [20].

####  I-A3 Low-Dimensional Representations of Formations and Distributed
Formation Control

A shape can be defined by goal locations that are assigned to each robot in a
swarm [21]. This approach is not scalable since each robot needs a defined
position. A different, more scalable approach is to use attractive and
repulsive potential functions to create desired formations [22, 23]. Fu et al.
use a consensus algorithm to keep robots in a consistent distribution relative
to a virtual leader that moves along a given trajectory [24].

In previous work, a decentralized swarm of eight robots was used to achieve
first- and second-order inertial moments with convergence guarantees [25, 26].
The robots were also able to achieve a series of goal formations to move past
an obstacle.

###  I-B Contributions

We present simultaneous distributed estimation and control of a swarm
formation using LMs and PZMs. The approach is scalable and self-healing in
theory, simulation, and experiments with up to 50 robots. Compared to previous
work (e.g., [25, 26]), the proposed approach gives moments higher than second-
order and therefore greater detail in specifying the formation; provides
robustness to packet loss; capitalizes on a recent method for distributed
optimization that reduces the amount of robot-robot communication needed for
estimation [6]; and has been verified on larger experimental swarms.

##  II Distributions Using Image Moments

###  II-A Legendre Moments (LMs)

The Legendre polynomials [17, 27, 28] $P_{m}(x)$ of order $m=0,1,...,\infty$
are defined on the domain $[-1,1]$ recursively using Bonnet’s recursion
formula [28],

$\begin{gathered}P_{0}(x)=1,\;P_{1}(x)=x,\\\
P_{m}(x)=\frac{2m-1}{m}xP_{m-1}(x)-\frac{m-1}{m}P_{m-2}(x),\;m\geq
2.\end{gathered}$ (1)

With these polynomials, the orthogonal basis functions of order $n=p+q$ over
the square $(x,y)\in[-1,1]\times[-1,1]$ have the form $P_{p}(x)P_{q}(y)$ where
$p,q=0,1,\ldots,\infty$. Any distribution $f(x,y)$ can be written as a linear
combination of these basis functions

$f(x,y)=\sum_{p=0}^{\infty}\sum_{q=0}^{\infty}M_{pq}P_{p}(x)P_{q}(y),$ (2)

where the _Legendre moments_ (LMs) $M_{pq}$ for a distribution $f(x,y)$ are
calculated as

$M_{pq}=\frac{(2p+1)(2q+1)}{4}\int_{-1}^{1}\int_{-1}^{1}P_{p}(x)P_{q}(y)f(x,y)\,dx\,dy.$
(3)

For a discrete distribution represented by point masses at the centers of
$(R\times C)$ “pixels,” the LMs can be written

$M_{pq}=\frac{(2p+1)(2q+1)}{4}\sum_{i=1}^{C}\sum_{j=1}^{R}P_{p}(x_{i})P_{q}(y_{j})\mu_{ij},$
(4)

where $(x_{i},y_{j})$ is the location of pixel $(i,j)$ and $\mu_{ij}$
represents the mass at pixel $(i,j)$. Similarly, for a set of $N$ point robots
of unit mass distributed over the plane, the LMs can be written as

$M_{pq}=\frac{(2p+1)(2q+1)}{4}\sum_{i=1}^{N}P_{p}(x_{i})P_{q}(y_{i}).$ (5)

A distribution $f(x,y)$ can be approximately reconstructed using its LMs
truncated at order $n$,

$f(x,y)\approx\sum_{p=1}^{n}\sum_{q=0}^{p}M_{p-q,q}P_{p-q}(x)P_{q}(y),$ (6)

where the fidelity of the reconstruction improves as $n$ increases. The single
zeroth-order moment is the total mass of the distribution, the two first-order
moments correspond to the location of the center of mass, and the three
second-order moments relate to the principal axes of the distribution. We do
not use zeroth-order moments in our representations to allow them to be
independent of the number of robots. At each order $n$, there are $n+1$ LMs,
so a formation represented by all moments from first to $n$th has
$\mathfrak{m}=n(n+3)/2$ elements.

###  II-B Pseudo-Zernike Moments (PZMs)

Zernike polynomials form an orthogonal basis for functions defined on the unit
disk $x^{2}+y^{2}\leq 1$ [17]. _Pseudo-Zernike polynomials_ form a related
orthogonal basis on the unit disk while providing improved image
reconstruction properties [17]. A pseudo-Zernike polynomial $W_{pq}$ is
defined as [18]

$W_{pq}(r,\theta)=S_{pq}(r)e^{\mathfrak{j}q\theta},$ (7)

where $p=0,1,\ldots,\infty$ is the order of the radial polynomial,
$q=0,1,\ldots,p$ is the degree of angular repetition, $r\leq 1$ is the radius,
$\theta$ is the angle, and $\mathfrak{j}$ is the imaginary unit. The radial
function $S_{pq}(r)$ is

$S_{pq}(r)=\sum_{k=q}^{p}B_{pqk}r^{k},$ (8)

where the coefficients $B_{pqk}$ are

$B_{pqk}=\frac{(-1)^{(p-k)}(p+k+1)!}{(p-k)!(q+k+1)!(k-q)!}.$ (9)

Any distribution $f(r,\theta)$, $r\leq 1$, can be expressed as a linear
combination of the pseudo-Zernike basis functions $W_{pq}(r,\theta)$

$f(r,\theta)=\sum_{p=0}^{\infty}\sum_{q=0}^{p}M_{pq}W_{pq}(r,\theta),$ (10)

where the _pseudo-Zernike moments_ (PZMs) $M_{pq}$ are

$M_{pq}=\frac{p+1}{\pi}\int_{0}^{2\pi}\int_{0}^{1}W^{*}_{pq}(r,\theta)f(r,\theta)\,r\,dr\,d\theta,$
(11)

and ∗ indicates the complex conjugate [17].

Analogously, the PZMs are

$M_{pq}=\frac{p+1}{L_{p}}\sum_{i=1}^{C}\sum_{j=1}^{R}W^{*}_{pq}(r_{ij},\theta_{ij})\mu_{ij}$
(12)

for an $(R\times C)$-pixel distribution, where
$r_{ij}=(x_{ij}^{2}+y_{ij}^{2})^{\frac{1}{2}}$,
$\theta_{ij}=\operatorname{atan2}(y_{ij},x_{ij})$, the mass $\mu_{ij}$ is zero
for $r_{ij}>1$, and $L_{p}$ is the total number of pixels satisfying
$r_{ij}\leq 1$. For $N$ unit-mass robots, the PZMs are

$M_{pq}=\frac{p+1}{L_{p}}\sum_{i=1}^{N}W^{*}_{pq}(r_{i},\theta_{i}).$ (13)

A distribution $f(r,\theta)$ can be approximately reconstructed using its PZMs
truncated at radial order $n$,

$f(r,\theta)\approx\sum_{p=1}^{n}\sum_{q=0}^{p}M_{pq}W_{pq}(r,\theta),$ (14)

where the fidelity of the reconstruction improves as $n$ increases. Just like
the LMs, there are $n+1$ PZMs at radial order $n$, and the total number of
PZMs from first- to $n$th-order is $\mathfrak{m}=n(n+3)/2$.

##  III Estimation and Control

The position of the $i$th point robot is
$s_{i}=[s_{ix},s_{iy}]^{\top}\in^{2}$, and the swarm configuration is the
column vector
$s=[s_{1}^{\top},s_{2}^{\top},\ldots,s_{N}^{\top}]^{\top}\in^{2N}$. Each robot
senses its own position and knows the desired formation expressed as a finite
set of image moments. All robots estimate the current swarm moments based only
on local measurements and messages from neighbors. The size of these messages
depends on the number of image moments used in the representation, but not the
number of robots. This is important for scaling up to formations of thousands
or millions of robots. Robots calculate their own motions based on their
estimated moments.

The error between distributions can be expressed as:

  * •

Moment vector error: Given $\mathfrak{m}$-vector moments for two formations,
$M_{1}=[M_{11},\ldots,M_{1\mathfrak{m}}]^{\top}$ and
$M_{2}=[M_{21},\ldots,M_{2\mathfrak{m}}]^{\top}$, this error is defined
$M_{1}-M_{2}\in^{\mathfrak{m}}$.

  * •

Mean-square reconstruction error (MSRE): The MSRE is defined using pixel-based
image reconstructions from the moment vectors $M_{1}$ and $M_{2}$ as

$\textrm{MSRE}=\frac{\sum_{i}\sum_{j}(f_{M_{1}}(x_{i},y_{i})-f_{M_{2}}(x_{i},y_{i}))^{2}}{\sum_{i}\sum_{j}f^{2}_{M_{2}}(x_{i},y_{j})},$
(15)

where $M_{2}$ corresponds to the desired moments. To calculate this error, we
evaluate the two reconstructions at $41\times 41$ points corresponding to
increments of $0.05$ in each dimension on the domain
$(x,y)\in[-1,1]\times[-1,1]$. For PZMs, points outside the unit circle are not
used.

######  Assumption 1.

We assume $\mathfrak{m}<2N$ so a moment vector $M\in^{\mathfrak{m}}$ is a
compressed representation of a distribution.

######  Remark 1.

Note that the distribution reconstruction for a given moment vector $M$ is
unique, but since $M$ is a compressed representation of a formation, in
general there are infinitely many robot formations that have the same moments.

Figure 2: Block diagram for robot $i$.

###  III-A Distributed Moment Estimation

The network of $N$ robots in a swarm is represented as a directed graph
consisting of edges $E$ and vertices $V={1,\ldots,N}$. An edge $E_{ij}$ exists
between any two robots $i$ and $j$ when $i$ can send messages to $j$. Each
robot $i$ knows the number of robots it sends a message to, i.e., its _out-
degree_ $d_{i}^{\textrm{out}}$. The out-Laplacian is
$\mathfrak{L}=D^{\textrm{out}}-A_{\mathrm{adj}}$, where the out-degree matrix
$D^{\textrm{out}}$ is the diagonal matrix
$D_{ii}^{\textrm{out}}=d_{i}^{\textrm{out}}$ for all $i\in V$, and an element
$A_{ij}$ of $A_{\mathrm{adj}}$ is $1$ if $E_{ij}$ exists and zero otherwise.

Robot $i$’s estimator and controller is illustrated in Figure 2. The estimator
maintains two local signals, $w_{i}\in^{\mathfrak{m}+1}$ and
$v_{i}\in^{\mathfrak{m}+1}$. The robot broadcasts $w_{i}$ to its neighbors,
receives communications $w_{k1},\ldots,w_{kl}$ from its neighbors, measures
its own position $s_{i}\in^{2}$, and uses $s_{i}$ to construct its input
vector

$u_{i}(s_{i})\equiv\begin{bmatrix}\phi(s_{i})\\\
1\end{bmatrix}\in^{\mathfrak{m}+1},$ (16)

where $\phi(s_{i})\in^{\mathfrak{m}}$ is the _moment-generating function_ that
generates robot $i$’s contributions to the swarm moment vector, i.e.,
$\mathfrak{m}$ components of the form $M_{pq}$ from Equation (5) (LMs) or
Equation (13) (PZMs) for the single robot $i$.

For the purpose of analysis, we model estimation and control as synchronous
discrete-time processes. Based on our recent work on self-healing distributed
optimization [29, 6], the robot runs a push-sum algorithm [30] to estimate the
swarm’s current moment vector. At each time step $t$, $v_{i}[t]$ and
$w_{i}[t]$ are calculated as

$\displaystyle v_{i}[t]$
$\displaystyle=u_{i}[t]-d_{i}^{\textrm{out}}w_{i}[t]+\sum_{k\in\mathcal{N}_{i}^{\textrm{in}}}w_{k}[t]$
(17) $\displaystyle w_{i}[t+1]$ $\displaystyle=w_{i}[t]+\gamma v_{i}[t],$ (18)

where $\mathcal{N}_{i}^{\textrm{in}}$ is the set of in-neighbors of robot $i$,
and the constant gain $\gamma$, which is the same for all robots, is chosen to
satisfy $\gamma d_{i}^{\textrm{out}}<1,\forall i$. Robot $i$’s estimate of the
swarm’s moments is

$\hat{M}_{i}=\left(\frac{1}{v_{i,\mathfrak{m}+1}}\right)v_{i,1\ldots\mathfrak{m}}\in^{\mathfrak{m}},$
(19)

where $v_{i,1\ldots\mathfrak{m}}\in^{\mathfrak{m}}$ refers to the first
$\mathfrak{m}$ components of $v_{i}$ and $v_{i,\mathfrak{m}+1}\in$ refers to
the last component of $v_{i}$.

The individual robots’ signals can be stacked as

$u[t]=\begin{bmatrix}u_{1}[t]\\\ \vdots\\\
u_{N}[t]\end{bmatrix},\;w[t]=\begin{bmatrix}w_{1}[t]\\\ \vdots\\\
w_{N}[t]\end{bmatrix},\;v[t]=\begin{bmatrix}v_{1}[t]\\\ \vdots\\\
v_{N}[t]\end{bmatrix}$ (20)

to write the updates of the whole system as

$\displaystyle v[t]$ $\displaystyle=u[t]-(\mathfrak{L}^{\top}\otimes I)w[t]$
(21) $\displaystyle w[t+1]$ $\displaystyle=w[t]+\gamma v[t],$ (22)

where $\mathfrak{L}$ is the out-Laplacian and $I$ is the identity matrix.

######  Theorem 1.

For a strongly-connected constant digraph network, the estimator of (21)-(22),
and a constant input $u[t]=u$, each $v_{i}$ converges exponentially to

$v_{i}^{*}=\begin{bmatrix}z_{i}\sum_{j=1}^{N}\phi(s_{j})\\\
z_{i}N\end{bmatrix},$ (23)

where $z_{i}$ is the $i$th component of the vector $z\in^{N}$ that satisfies
$\mathfrak{L}^{\top}z=0$ and $\mathbbm{1}^{\top}z=1$. Therefore $\hat{M}_{i}$
(Equation (19)) converges to the correct moments $M(s)$.

Proof.  The proof follows from Theorem 1 of [29]. $\square$

######  Remark 2.

The convergence described in Theorem 1 resumes after transients caused by
network changes, sensing errors, or packet drops. If the input $u$ is
continuously changing, a bound on the convergence error grows with the maximum
rate of change of $u$ [31].

To address the practical issue of packet drops in a fielded system, we can add
memory to the estimator [32]. The memory state $\mathfrak{M}_{ki}$ stores the
last message robot $i$ received from neighbor $k$. If no message was received
from neighbor $k$ this iteration, its previous message is used in estimation.
Equation (17) can be rewritten as

$\displaystyle v_{i}[t]$
$\displaystyle=u_{i}[t]-d_{i}^{\textrm{out}}w_{i}[t]+\sum_{k\in
N_{i}^{\textrm{in}}}\mathfrak{M}_{ki}[t]$ (24)
$\displaystyle\mathfrak{M}_{ki}[t]$
$\displaystyle=\begin{cases}\mathfrak{M}_{ki}[t-1]&\text{if packet lost}\\\
w_{k}[t]&\text{if packet received}.\end{cases}$ (25)

If a robot has not heard from a neighbor after a set number of consecutive
iterations, that neighbor’s value in the memory state is removed. This
forgetting factor allows the estimator to adjust to changes in the
communication network.

To test the performance of moment estimation, we simulated a network of 50
stationary robots randomly distributed over an arena of size
$[-1,1]\times[-1,1]$. The estimators were tested using all-to-all
communication networks and networks defined by a communication radius of 0.5,
and for varying levels of packet loss, with and without memory. A packet loss
rate of 30%, for example, indicates that each individual message has a 30%
chance of being dropped. The estimator gain was $\gamma=1/N$.

Figure 3: Convergence of LM estimation up to eighth-order. Figure 4:
Convergence of PZM estimation up to eighth-order.

Ten trials were conducted for each packet loss level and network type. The
robot positions were the same across all trials. The convergence times of
estimating LMs and PZMs up to eighth-order are reported in Figures 3 and 4,
respectively. Convergence is considered to be achieved when the error relative
to the desired moments is less than $0.01$. If convergence is not reached
within 500,000 steps, the trial is terminated. For trials with memory, a
neighbor’s message is kept in the memory state until $75$ consecutive
iterations passed without receiving a message from the neighbor. The all-to-
all network results are shown in black while the radius-limited results are
shown in red. The point represents the median convergence time of the ten
trials and the vertical line indicates the range of convergence times. Memory
reduces the convergence time by orders of magnitude when packets are lost and
the all-to-all communication network has faster convergence times for both LMs
and PZMs.

###  III-B Control of Swarm Moments

The control goal is to minimize the error between the estimated and desired
moments as given by the quadratic cost

$C(s)=\frac{1}{2}[M(s)-M^{*}]^{\top}\Gamma[M(s)-M^{*}],$ (26)

where $M(s)$ is the vector of moments of the swarm’s current configuration
$s$, $\Gamma\in^{\mathfrak{m}\times\mathfrak{m}}$ is a symmetric positive-
definite gain matrix, and $M^{*}$ is the vector of desired moments.

Each robot moves according to the gradient control law

$\dot{s}_{i}=-[\mathscr{J}(\phi(s_{i}))]^{\top}\Gamma[\hat{M}_{i}-M^{*}],$
(27)

where $\hat{M}_{i}$ is robot $i$’s current estimate of the swarm moments and
$\mathscr{J}(\phi)$ is the Jacobian of the moment-generating function $\phi$
with respect to $s_{i}$.

To analyze the collective behavior of the control law (27), we consider the
case where each robot has a perfect estimate of the current swarm moments,
$\hat{M}_{i}=M(s)$, and we define the moment error $e(s)=M(s)-M^{*}$. Let
$J(s)=\frac{de}{ds}\in^{\mathfrak{m}\times 2N}$. The $N$ copies of (27) can be
stacked to give

$\dot{s}=-J^{\top}(s)\Gamma e(s)\in^{2N}.$ (28)

Clearly every point $s_{e}$ for which $e(s_{e})=0$ is an equilibrium for (28).
We will call these the “good” equilibria, in contrast to equilibria where the
error is nonzero. To analyze the local behavior of the system (28) near a good
equilibrium, we first note that the time derivative of $e$ is

$\frac{de}{dt}=A_{e}(s)e\;\;\textrm{where}\;\;A_{e}(s)=-J(s)J^{\top}(s)\Gamma.$
(29)

######  Theorem 2.

Let $s_{e}$ be a good equilibrium for the dynamics (28), and suppose
$\operatorname{rank}(J(s_{e}))=\mathfrak{m}$. Then $s_{e}$ is stable in the
sense of Lyapunov. Moreover, if the initial configuration $s(0)$ is
sufficiently close to $s_{e}$, then $e$ converges asymptotically to zero.

Proof.  By assumption $\operatorname{rank}(J(s_{e}))=\mathfrak{m}$ and
$\Gamma>0$, which together imply that the matrix $A_{e}(s_{e})$ in (29) is
Hurwitz. Therefore there exists a unique positive definite solution $P$ to the
Lyapunov equation

$PA_{e}(s_{e})+A_{e}(s_{e})^{\top}P=-I.$ (30)

Let $\widetilde{A}(s)=A_{e}(s)-A_{e}(s_{e})$; then we can write the derivative
of the Lyapunov function $V_{e}(e)=e^{\top}Pe$ along solutions to (29) as

$\dot{V_{e}}=-||e||^{2}+e^{\top}[P\widetilde{A}(s)+\widetilde{A}(s)^{\top}P]e.$
(31)

Because $\operatorname{rank}(J(s_{e}))=\mathfrak{m}$, there exists a
$(2N-\mathfrak{m})\times 2N$ matrix $T$ such that
$[T^{\top}\;\;J(s_{e})^{\top}]$ is invertible. This implies that the mapping
from $s$ to $(Ts-Ts_{e},e(s))$ is a diffeomorphism in a neighborhood of
$s_{e}$. Setting $y=Ts-Ts_{e}$, we can write the system (28) in the local
$(y,e)$-coordinates as

$\displaystyle\dot{y}$ $\displaystyle=C_{e}(e,y)e$ (32) $\displaystyle\dot{e}$
$\displaystyle=A_{e}(s_{e})e+B_{e}(e,y)e,$ (33)

where $B_{e}(0,0)=0$. Because $A_{e}(s_{e})$ is Hurwitz, we can use a Lyapunov
function of the form $||y||^{2}+\sqrt{V(e)}$ to show that the point
$(y,e)=(0,0)$ is stable in the sense of Lyapunov for the system (32)–(33) (see
[33, Corollary 8.1] for details). Thus the equilibrium $s_{e}$ for the system
(28) is stable in the sense of Lyapunov, which means if $s(0)$ is sufficiently
close to $s_{e}$ then $\widetilde{A}(s)$ will remain small in forward time.
Thus from (31) we obtain $\dot{V}<0$ for $e\neq 0$, and we conclude that $e$
converges asymptotically to zero. $\square$

To analyze the global behavior of the system (28), we first note that it is a
gradient flow. Hence if $e$ is real analytic and proper, which it is for LMs
and PZMs, then every trajectory of this gradient system converges to an
equilibrium [34]. Thus from Theorem 2 we see that any sufficiently small
perturbation away from a good equilibrium will cause the system to converge to
a (possibly different) good equilibrium. Moreover, it follows from (28) that
$\operatorname{rank}(J(s_{e}))<\mathfrak{m}$ at any “bad” equilibrium $s_{e}$,
i.e., any equilibrium $s_{e}$ for which $e(s_{e})\neq 0$. Thus the bad
equilibria occur only at singular configurations where $J$ loses rank.
Nevertheless, it may be possible for a bad equlilibrium to be a local minimum
of the cost in (26) and thus be a stable equilibrium for the flow (28). While
we have not found this to be an issue in practice, the problem of how to
choose the gain matrix $\Gamma$ to make bad equilibria unstable, or at a
minimum to shrink their regions of attraction, is a topic for future research.

The controller was tested by simulating a network of 50 robots in an arena of
size $[-1,1]\times[-1,1]$. Each robot had access to the correct swarm moments,
$\hat{M}_{i}=M(s)$, and its motion was simulated using a discretized version
of the control law (27):

$s_{i}[t+1]=s_{i}[t]+\dot{s}_{i}.$ (34)

Figure 5: The 2-norm of the moment vector error for varying maximum orders of
LMs and PZMs using the controller with perfect moment estimates.

The gain matrix $\Gamma$ was determined experimentally, with the observation
that higher-order moments should have smaller associated gains for faster
convergence. We define $\Gamma=G$ as

$G=\operatorname{diag}(g)\in^{\mathfrak{m}\times\mathfrak{m}},$ (35)
$g=(\underbrace{\\{{1^{-1.7}},{1^{-1.7}}\\}}_{d=1},\underbrace{\\{{2^{-1.7}},{2^{-1.7}},{2^{-1.7}}\\}}_{d=2},\ldots,\underbrace{\\{{d^{-1.7}}\\}_{2n-1}}_{d=n}),$

i.e., $d+1$ copies of $d^{-1.7}$ for $d=1,\ldots,n$, the maximum degree in the
moment vector. With this definition of the diagonal matrix $G$, we set
$\Gamma=0.1G$.

LMs and PZMs up to orders $2$, $4$, $6$, and $8$ were calculated for the
“bunny head” distribution depicted in Figure 7(a) and were used as the goal
moment vectors. Each control trial consisted of 200,000 iterations from the
same initial robot configuration. Figure 5 shows the robots’ convergence to
the desired distribution.

###  III-C Simultaneous Estimation and Control

Even if each robot’s estimator is guaranteed to converge to the formation’s
actual moments and the controller is guaranteed to converge to the desired
moments using perfect estimates of the swarm moments, when we couple
estimation and control the system may lose these properties and exhibit more
complex behaviors. As an example, for $N=7$ robots ($2N=14$) and third-order
LMs ($\mathfrak{m}=9$ moments), we simulated: the estimator alone (stationary
robots) with estimator gain $\gamma=1/(10N)$, memory state, and no packet
loss; the controller alone with perfect moment estimates and controller gains
$\Gamma_{1}=0.5G$ and $\Gamma_{2}=G$, where $G$ is given by Equation (35); and
combined estimation and control with $\Gamma_{1}=0.5G$ and $\Gamma_{2}=G$.

Figure 6: Comparing estimation alone, control alone, and combined estimation
and control for two different control gain matrices. Top left: The 2-norm of
the moment vector error converges to zero for one robot’s estimates vs.
desired moments. Bottom left: The motions of the robots (from x’s to o’s) for
control gains $\Gamma_{1}=0.5G$ and perfect moment estimates. Middle column:
Performance of the controller with perfect moment estimates. The robots
converge to the desired moments for both control gains $\Gamma_{1}=0.5G$ and
$\Gamma_{2}=G$. Right column: Performance of the controller when each robot
uses estimated moments. The robots converge to the desired moments for the
smaller control gains but not the larger gains.

Figure 6 plots the results of 1000 iterations. The estimator converges to the
correct moment estimates with stationary robots. The controller with perfect
estimates drives the robots to the desired moments for control gains
$\Gamma_{1}$ and $\Gamma_{2}$. The combined estimation and control system is
unstable for control gains $\Gamma_{2}$ but convergent for $\Gamma_{1}$. This
is an example of aggressive control gains coupled with estimator delay causing
instability. Future work may include a stability analysis, e.g., using small-
gain theory or passivity analysis, to better understand how aggressively to
choose motion control gains in light of delays inherent in distributed
estimation.

##  IV Experimental Implementation

###  IV-A Coachbot V2.0 System

Experiments were performed with the Coachbot V2.0 swarm setup, which consists
of 100 differential drive robots and a high-fidelity Docker simulation for
testing robot control algorithms before deployment [21]. The robots sense
their positions and orientations using time-varying infrared signals from a
ceiling mounted HTC VIVE lighthouse and communicate with each other and a
centralized workstation via Wi-Fi with an optional artificially-imposed
limited communication radius. In practice, we found that 40-50% of
communicated packets are dropped due to the high density of communications and
limited buffer sizes.

(a) Left: Coachbot robot. Middle: Uniform-density “bunny head” formation.
Right: Formation described by two identically-sized disks. The darker disk is
twice the density of the lighter disk.

(b) Fifty Coachbots forming the bunny head using eighth-order LMs.

(c) Thirty Coachbot robots forming the disks using sixth-order PZMs.

Figure 7: Experimental desired formations.

Every 0.27 s, each robot executes the control loop of Figure 2: it senses its
position, updates the estimator input (16), checks for received messages,
computes the estimator (17)-(18), broadcasts estimator signals, and calculates
the control velocity of a reference point on the robot away from the wheel
axis (27). This velocity is modified by a collision-avoidance filter (based on
three zones of varying repulsion around each robot that prevents collisions
with neighbors [26]) and then bounded. Velocities below a minimum threshold
are set to zero. The calculated velocity of the robot reference point is
transformed to wheel speeds as described in [26, 35].

The experimental system differs from the model in that (1) there is no global
clock and (2) the robots have nonzero extent and implement collision avoidance
and velocity saturation.

Below we report experimental distributed formation control with two goal
formations: a binary “bunny head” formation, where the desired density inside
the bunny head silhouette is constant and zero outside, and a bimodal
formation consisting of two identically-sized solid disks, with the density of
one disk twice the other’s (Figure 7(a)). The arena is 3 m $\times$ 3 m in
size, for a scale factor of 1.5 m relative to the $[-1,1]^{2}$ LM and PZM
representations. The communication radius was set to 1.5 m. In all
experiments, the robots started in a random configuration centered around the
arena origin.

LMs and PZMs were tested for both goal formations and the results can be seen
in the video [36]. The controller gains were $\Gamma=G$ from (35). The
estimator gain $\gamma$ was chosen to be $1/N$, and memory kept neighbor
messages for up to $1.5N$ consecutive iterations. Desired moments were
calculated from the images of the bunny head and two-disk formations. The
following results show robot 15’s estimates, which are representative of each
robot, as each robot’s moment estimates converged to the actual moments.

###  IV-B Binary “Bunny Head” Formation

Figure 8: Top row: Grayscale image reconstructions using the fourth, sixth,
and eighth-order LM representations of the bunny head formation. Middle row:
Grayscale image reconstructions of the final estimated moments after
distributed estimation and control by 50 robots in simulation (fourth-,
sixth-, and eighth-order moments) and experiment (eighth-order moments).
Bottom row: The final robot configurations.

Figure 7(b) shows snapshots of 50 Coachbots moving toward the eighth-order LMs
of the bunny head formation. Figure 8 shows the distributions reconstructed
from LMs up to fourth order (14 moments), sixth order (27 moments), and eighth
order (44 moments), respectively, as grayscale images; the final simulated
configurations of the robots using the estimator (17)-(18) and control law
(27), as well as the corresponding grayscale representations of the
reconstructions from the LMs; and the final robot configurations and the
reconstructed distribution for a hardware experiment with eighth-order LMs.

The fidelity of the moment representation of the bunny head increases as the
moment order increases. The experiments also show that distributed estimation
and control is effective in driving the robots to achieve the desired moments.
The individual LM coefficient errors are shown in Figure 9. The larger error
in the eighth-order hardware relative to the simulation is primarily
attributed to the velocity deadband in experiment.

Figure 9: Absolute value of the difference between robot 15’s moment
coefficient estimates and the desired moments for increasing orders of LMs
representing the bunny head shape. Figure 10: MSRE vs. control iterations for
the bunny head using LMs of increasing maximum order (fourth-, sixth-, and
eighth-order moments) and experiment 8H (eighth-order moments).

Figure 10 shows the MSRE decreasing as the robots converge. As the error
decreases, the commanded velocities reach the deadband, preventing further
decrease of the MSRE.

###  IV-C Two-Disk Formation

Figure 11: Two-disk formation represented by sixth-order PZMs and the final
experimental configuration of 30 robots.

Figure 7(c) shows snapshots of 30 Coachbots moving toward the sixth-order PZMs
of the two-disk formation, and Figure 11 shows the reconstruction of the
desired moments, the reconstruction of the estimated moments, and the final
configuration achieved by the 30 robots. The robots clearly segregate to the
two regions, with a higher number of robots (19 vs. 11) in the region with
twice the desired density. In the experiment, the size of the robots, coupled
with their collision avoidance, actually prevents 20 robots from fitting
inside the boundaries of the denser disk. Figure 12 shows the robots converged
to their final positions with a small steady-state MSRE.

Figure 12: MSRE of the two-disks using sixth-order PZMs.

##  V Conclusion

This paper presents a method for distributed, scalable, self-healing swarm
formation estimation and control based on image moment representations, where
the level of detail in the desired formation representation is determined by
the number of image moments. In simulation and experiments with up to $50$
robots, the robots approximately achieve specified formations despite nearly
$50$% packet loss.

Further work is needed to determine how to choose gains that ensure good
coupled performance and stability. Beyond a gradient controller, distributed
model predictive control can improve the rate of convergence to desired
moments.

## References

  * Rubenstein et al. [2014] M. Rubenstein, A. Cornejo, and R. Nagpal, “Programmable self-assembly in a thousand-robot swarm,” _Science_ , vol. 345, no. 6198, pp. 795–799, 2014\.
  * [2] International Olympic Committee, “Spectacular Intel drone light show helps bring Tokyo 2020 to life.” [Online]. Available: https://olympics.com/ioc/news/spectacular-intel-drone-light-show-helps-bring-tokyo-2020-to-life-1
  * [3] B. Barrett, “Inside the Olympics opening ceremony world-record drone show.” [Online]. Available: https://www.wired.com/story/olympics-opening-ceremony-drone-show/
  * Schranz et al. [2020] M. Schranz, M. Umlauft, M. Sende, and W. Elmenreich, “Swarm robotic behaviors and current applications,” _Frontiers in Robotics and AI_ , vol. 7, 2020\.
  * Kolling et al. [2016] A. Kolling, P. Walker, N. Chakraborty, K. Sycara, and M. Lewis, “Human interaction with robot swarms: A survey,” _IEEE Transactions on Human-Machine Systems_ , vol. 46, no. 1, pp. 9–26, 2016.
  * Donato Ridgley et al. [2021] I. L. Donato Ridgley, R. A. Freeman, and K. M. Lynch, “Self-healing first-order distributed optimization,” in _2021 60th IEEE Conference on Decision and Control (CDC)_ , 2021, pp. 3850–3856.
  * Bashyal and Venayagamoorthy [2008] S. Bashyal and G. K. Venayagamoorthy, “Human swarm interaction for radiation source search and localization,” in _2008 IEEE Swarm Intelligence Symposium_ , 2008, pp. 1–8.
  * Brown et al. [2014] D. Brown, S. Kerman, and M. Goodrich, “Human-swarm interactions based on managing attractors,” _ACM/IEEE International Conference on Human-Robot Interaction_ , pp. 90–97, 03 2014.
  * Kolling et al. [2013] A. Kolling, K. Sycara, S. Nunnally, and M. Lewis, “Human swarm interaction: An experimental study of two types of interaction with foraging swarms,” _Journal of Human-Robot Interaction_ , vol. 2, no. 2, pp. 103 – 129, June 2013.
  * Kira and Potter [2009] Z. Kira and M. A. Potter, “Exerting human control over decentralized robot swarms,” in _2009 4th International Conference on Autonomous Robots and Agents_ , 2009, pp. 566–571.
  * Egerstedt et al. [2014] M. Egerstedt, J.-P. de la Croix, H. Kawashima, and P. Kingston, _Interacting with Networks of Mobile Agents_. Cham: Springer International Publishing, 2014, pp. 199–224.
  * Saska et al. [2016] M. Saska, V. Vonasek, J. Chudoba, J. Thomas, and G. Loianno, “Swarm distribution and deployment for cooperative surveillance by micro-aerial vehicles,” _Journal of Intelligent and Robotic Systems_ , vol. 84, 12 2016\.
  * Prabhakar et al. [2020] A. Prabhakar _et al._ , “Ergodic specifications for flexible swarm control: From user commands to persistent adaptation,” in _Robotics: Science and Systems_ , 07 2020.
  * Lee et al. [2015] S. G. Lee, Y. Diaz-Mercado, and M. Egerstedt, “Multirobot control using time-varying density functions,” _IEEE Transactions on Robotics_ , vol. 31, no. 2, pp. 489–493, 2015.
  * Pawlak [2006] M. Pawlak, _Image analysis by moments : reconstruction and computational aspects_. Oficyna Wydawnicza Politechniki Wrocławskiej, 2006.
  * Qi et al. [2021] S. Qi, Y. Zhang, C. Wang, J. Zhou, and X. Cao, “A survey of orthogonal moments for image representation: Theory, implementation, and evaluation,” _ACM Computing Surveys_ , vol. 55, 11 2021.
  * Teh and Chin [1988] C.-H. Teh and R. Chin, “On image analysis by the methods of moments,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 10, no. 4, pp. 496–513, 1988.
  * Hosny [2014] K. Hosny, “Fast computation of accurate pseudo Zernike moments for binary and gray-level images,” _International Arab Journal of Information Technology_ , vol. 11, pp. 243–249, 05 2014.
  * Mukundan and Ramakrishnan [1998] R. Mukundan and K. R. Ramakrishnan, _Moment Functions in Image Analysis: Theory and Applications_. World Scientific Publishing Company, 1998.
  * Hosaini et al. [2013] S. J. Hosaini, S. Alirezaee, M. Ahmadi, and S. V.-A. D. Makki, “Comparison of the Legendre, Zernike and pseudo-Zernike moments for feature extraction in iris recognition,” in _2013 5th International Conference and Computational Intelligence and Communication Networks_ , 2013, pp. 225–228.
  * Wang and Rubenstein [2020] H. Wang and M. Rubenstein, “Shape formation in homogeneous swarms using local task swapping,” _IEEE Transactions on Robotics_ , vol. 36, pp. 597–612, 6 2020.
  * Jung and Kim [2014] H. Jung and D. Kim, “Potential-function-based shape formation in swarm simulation,” _International Journal of Control, Automation and Systems_ , vol. 12, pp. 442–449, 04 2014.
  * Yang et al. [2022] H. Yang, Y. Li, X. Duan, G. Shen, and S. Zhang, “A parallel shape formation method for swarm robotics,” _Robotics and Autonomous Systems_ , vol. 151, p. 104043, 2022.
  * Fu et al. [2020] X. Fu, J. Pan, H. Wang, and X. Gao, “A formation maintenance and reconstruction method of uav swarm based on distributed control,” _Aerospace Science and Technology_ , vol. 104, p. 105981, 2020.
  * Yang et al. [2008] P. Yang, R. A. Freeman, and K. M. Lynch, “Multi-agent coordination by decentralized estimation and control,” _IEEE Transactions on Automatic Control_ , vol. 53, no. 11, pp. 2480–2496, 2008.
  * Hwang et al. [2013] M. Hwang, M. Elwin, P. Yang, R. Freeman, and K. Lynch, _Experimental validation of multi-agent coordination by decentralized estimation and control_. Bentham Publishing, 2013, pp. 59–75.
  * Teague [1980] M. R. Teague, “Image analysis via the general theory of moments$\ast$,” _Journal of the Optical Society of America_ , vol. 70, no. 8, pp. 920–930, Aug 1980.
  * Mukundan and Ramakrishnan [1995] R. Mukundan and K. Ramakrishnan, “Fast computation of Legendre and Zernike moments,” _Pattern Recognition_ , vol. 28, no. 9, pp. 1433–1442, 1995.
  * Donato Ridgley et al. [2020] I. L. Donato Ridgley, R. A. Freeman, and K. M. Lynch, “Private and hot-pluggable distributed averaging,” _IEEE Control Systems Letters_ , vol. 4, no. 4, pp. 988–993, 2020.
  * Hadjicostis et al. [2018] C. N. Hadjicostis, A. D. Domínguez-García, and T. Charalambous, _Distributed Averaging and Balancing in Network Systems_ , ser. Foundations and Trends (R) in Systems and Control. Now Publishers, 2018, vol. 13.
  * Kia et al. [2018] S. Kia, B. Van Scoy, J. Cortes, R. Freeman, K. Lynch, and S. Martinez, “Tutorial on dynamic average consensus: The problem, its applications, and the algorithms,” _IEEE Control Systems_ , vol. 39, 03 2018.
  * Hadjicostis et al. [2016] C. N. Hadjicostis, N. H. Vaidya, and A. D. Domínguez-García, “Robust distributed average consensus via exchange of running sums,” _IEEE Transactions on Automatic Control_ , vol. 61, no. 6, pp. 1492–1507, 2016.
  * Khalil [2002] H. K. Khalil, _Nonlinear systems; 3rd ed._ Upper Saddle River, NJ: Prentice-Hall, 2002.
  * Łojasiewicz [1983] S. Łojasiewicz, “Sur les trajectoires du gradient d’une fonction analytique,” in _Seminari di Geometria 1982-1983_. Università di Bologna, 1983, pp. 115–117.
  * Lynch and Park [2017] K. M. Lynch and F. C. Park, _Modern Robotics: Mechanics, Planning, and Control_ , 1st ed. USA: Cambridge University Press, 2017.
  * [36] [Online]. Available: https://www.youtube.com/watch?v=8IjvSkTAUM8


# Preliminary Wildfire Detection: A Dataset and Challenges

Samarth Shah

2021 Summer STEM Institute

###### Abstract

Wildfires are uncontrolled fires in the environment that can be caused by
humans or nature. In 2020 alone, wildfires in California have burned 4.2
million acres, damaged 10,500 buildings/structures, and killed more than 31
people, exacerbated by climate change and a rise in average global
temperatures [1] [2]. This also means there has been an increase in the costs
of extinguishing these treacherous wildfires. The objective of the research is
to detect forest fires in their earlier stages to prevent them from spreading,
prevent them from causing damage to a variety of things, and most importantly,
reduce or eliminate the chances of someone dying from a wildfire. A fire
detection system should be efficient and accurate with respect to
extinguishing wildfires in their earlier stages to prevent the spread of them
along with their consequences. Computer Vision is potentially a more reliable,
fast, and widespread method we need. The current research in the field of
preliminary fire detection has several problems related to unrepresentative
data being used to train models and their existing varied amounts of label
imbalance in the classes (commonly fire and non-fire) of their dataset. We
propose a more representative and evenly distributed data through better
settings, lighting, atmospheres, etc., and class distribution in the entire
dataset. After thoroughly examining the results of this research, it can be
inferred that they supported the dataset’s strengths by being a viable
resource when tested in the real world on unfamiliar data. This is evident
since as the model trains on the dataset, it is able to generalize on it,
hence confirming this is a viable Machine Learning setting that has practical
impact.

##  1 Introduction

Wildfires are uncontrolled fires in the environment that can be caused by
humans or nature. These fires are dangerous to both humans and nature as they
have the potential to destroy a copious amount of property and also damage the
ecosystem. In 2020 alone, wildfires in California have burned 4.2 million
acres, damaged 10,500 buildings/structures, and killed more than 31 people
[1]. The solution must allow for endangering wildfires to be found before they
can spread which can save hundreds of thousands of dollars typically spent on
extinguishing fires, repairing damaged property, and most importantly, it can
save lives. Although there are many applications of Machine Learning in the
field of early wildfire detection, many are too costly to implement and/or not
feasible to work with.

Figure 1: Left: An Image Captured Using A Satellite-based Solution [3]. This
solution is more costly and does not detect wildfires in their earlier stages.
Middle: An Image Captured Using An Aerial-based Solution [4]. This solution is
too costly and not very feasible due to operating expenses and short flight
time. Right: An Image Captured Using A Terrestrial-based Solution [5]. This
solution is relatively more practical since it is cheaper and more feasible,
and can cover large areas of land.

Computer Vision is a necessary tool as part of a larger wildfire solution.
Computer Vision allows for a more efficient and cheap solution to a multitude
of applications as it can repetitively carry out tasks at a constant faster
pace with less human error [6]. They are able to automate tasks that wouldn’t
be feasible for humans to complete at a constant pace on a regular basis.
ALERTWildfire is a large scale PTZ camera network used to monitor landscapes.
Currently, the organization has given humans the responsibility of catching
wildfires during their preliminary stages. Computer Vision can allow for an
improved solution to wildfires by automating the human-dependent process being
carried out today. This Computer Vision solution can be easily expanded and
adopted by others. With ALERTWildfire’s large-scale camera network, I plan to
use their camera data as they are able to look at all of the land on the
western part of The United States that have a risk of sustaining wildfires.
ALERTWildfire is an organization that partners with several other groups
stationed in the western part of The United States. By doing so, they’re able
to provide such a network of cameras in which each partner holds a stake. In
the future, this camera network will be expanded globally to places all over
the world that are susceptible to contain wildfires. These terrestrial-based
solutions make way for an Image Recognition algorithm that can constantly scan
for fires in multiple locations that are likely to contain wildfires with
wide-angle pan-tilt-zoom cameras [5].

Figure 2: Left: Landscape Without Fire, Right: Landscape With Fire. Both
images originate from a wide-angle pan-tilt-zoom camera view provided by
ALERTWildfire. Fires usually happen in the day, and smoke is the biggest
visual characteristic for detection. Evident through comparing each image to
the other, the image on the left does not have a fire despite their being
smoke in the air above the landscape since it is not sprouting from the land.
In contrast, the image on the right does have a fire since there is smoke
coming from the land. Although there are clouds in the background that could
be classified as smoke, they are not considered as smoke caused by a wildfire
as they are not coming from the ground.

On this topic, there are a few other research papers that are based on the
same theme of using Machine Learning and Computer Vision technology to detect
fires on their earlier stages to mitigate the damage they produce [7] [8].
However, these papers have several issues and limitations regarding
practicality due to their failure to consider the Machine Learning obstacles.
This is a very common issue found in these papers simply because their
respective authors tend to ignore the implications of the form of the dataset
they present to their models. Although the accuracies presented by these
papers may seem enticing at first glance, they are not valid representations
of how well their models would perform when given real-world data from common
cameras used for early wildfire detection, such as the ones provided by
ALERTWildfire. For instance, one common issue that circulates through the
research in this field is the issue of Model Robustness–the model’s ability to
identify the important features that distinguish one class from another in the
case of image recognition. This is due to the several papers in the same field
that have models unable to pick up on distinguishing factors to accurately
determine whether or not a given image contains a wildfire. Therefore, this
would result in a not very robust model as the data used for training is
dissimilar to what is seen when the model tests on real-world data.
Consequently, this problem can lead to an issue with Calibration–the model’s
confidence or the probability that its predictions are reliable. If the model
is not very robust, it could negatively affect its Calibration as it would
decrease the probability that its predictions are reliable when applied to a
real-world scenario. The research being carried out will also focus on
mitigating these prevalent issues by using more representative and evenly
distributed data in terms of the image’s setting, lighting, atmosphere, etc.,
and class distribution in the entire dataset. The data was also masked on the
parts that were not correlated with the determination of whether or not there
exists a fire in a given image. The data was split such that 75% of it was
allocated for training purposes and 25% of it was allocated for purposes
testing the accuracy. There were 4000 images in which 2000 of them were images
of fires and 2000 of them were images of landscapes. By doing these things, I
will resolve or mitigate the issues evident in neighboring research papers.
For instance, keeping the dataset with a more representative setting,
lighting, atmosphere, etc. will allow me to train the model on what it will be
expected to input which will increase the validity of the accuracy. By masking
parts of the images in the dataset, I will also lessen the bias present in the
images to also increase the validity of the accuracy. Through carrying out the
experimentation phase of this research and closely observing the results, one
can note the dataset’s attempt to mitigate the bias found in neighboring
papers through its distinguishing features mentioned above. A Convolutional
Neural Network was able to train on this dataset, lessen its loss, and
increase its accuracy.

##  2 Literature Review

I have determined some of the practical problems faced by researchers across
the board regarding preliminary wildfire detection by looking through several
pieces of literature relating to real-world applications using Artificial
Intelligence and collecting data that will be used to train the model. I also
noted common techniques used in the algorithms and general, recurring problems
that show up with the image recognition implemented for early wildfire
detection. Apart from Computer Vision, there are other solutions referred to
by some in the field of preliminary wildfire detection. One of the most
commonly-referred alternatives to using Image Recognition is to implement
terrestrial-based sensors that are stationed at spots likely to contain
wildfires [9]. Some also look at aerial-based solutions using systems such as
drone networks [9]. The issue with both of these approaches is that they are
too costly and not efficient enough to be deployed on a large scale, which is
necessary when there is not one designated location and designated period of
time where and when wildfires ignite. I have included illustrations and
diagrams of some of the more prevalent problems that occur in Computer Vision
research.

Model Robustness refers to the model’s ability to identify the important
features that distinguish one class from another in the case of image
recognition. Oftentimes, data used by researchers would contain images of
fires that were very similar to each other and would often be located on the
same spot of the image, would have the same hue of the flame, etc. For
instance, if an algorithm was trained on images that were pointed at the same
location/landscape with the same features, it could have only learned how to
detect wildfires in that particular setting and for that reason would not be a
very robust model as it would not be using the features that distinguish an
image from having a wildfire as opposed to not having one. Not having a robust
model in the field of wildfire detection would decrease the representativeness
of the displayed accuracy of the model if it refers to its accuracy as its
ability to detect wildfires in general [10]. In turn, this would result in a
decrease in validity of the accuracy showcased by several of the research
papers–in the field of preliminary wildfire detection–that have datasets
filled with skewed images that had several other indicators of fires such as
the orientation of the image, the setting of the image, etc. [11]

Figure 3: Left: Detected Wildfire, Has Wildfire. Right: Detected Wildfire, No
Wildfire. An Illustration of The Model Robustness Problem – The Model’s
Tendency To Perform Poorly From “Subtle Differences” In The Input Image.
Although the left image is a correct prediction as there is a wildfire in the
image, the right image is an incorrect prediction as there is no wildfire in
that image. This is due to an issue with the Model Robustness since a subtle
change in the image resulted in a false prediction. The gray areas of the
smoke, which are often used to determine the existence of a wildfire, look
relatively similar and thus have not changed much between the two images.

Calibration refers to the model’s confidence or the probability that its
predictions are reliable. Calibration is especially important in the field of
early wildfire detection as any individual in the field would want to ensure
that they catch all the wildfires that occur without missing any, but at the
same time, not wanting to overtrain the model to the point where it would
detect false positives [12].

Generalization refers to the ability of a trained model to perform predictions
on unseen/unfamiliar data. As touched on earlier regarding the model’s
robustness, researchers would often find their models not performing at the
accuracy listed when running on unfamiliar data proving their models haven’t
completely generalized on all wildfires [7]. This relates to the robustness of
the model as discussed earlier since a model that is less robust tends to not
be as generalizable. This means generalization and model robustness has a
direct relationship. Relating the previously brought up skewed image with the
topic of generalization, if the model trains on images that have odd
orientations, unrepresentative settings, and other sources of dataset bias, it
will not generalize its weights on realistic data that it will input when
applied to cameras in an outdoor setting where practically all wildfires occur
[11]. Moreover, as a model continues to train on familiar data, it increases
its confidence in it and this will result in increased poor performance on
unfamiliar data. This phenomenon is typically referred to as a Domain Shift
[13].

Figure 4: A Visualization of The Domain Shift Problem [13]. Evident from the
diagram pictured, there is a decrease in performance on real data in
comparison to the model’s performance on synthetic data. This can be
attributed to the Domain Shift phenomenon.

Label Imbalance refers to the problem faced by a model regarding predictions
due to the number of examples for each class in the training set being
unbalanced. In the realm of wildfire detection and mitigation, Label Imbalance
must be taken into account as there is always more data with images of without
wildfires than there are with wildfires. Label Imbalance is difficult because
models will tend to output majority class predictions and will lean in
opposition to the minority class. One way to mitigate the unwanted effects of
Label Imbalance is by implementing a Class Rectification Loss Function to
automatically allow for the model to correct for the “dominant effect” of the
majority classes through observing minority classes and their sampled
boundaries [14]. Through reviewing the literature in the same field of early
wildfire detection, Label Imbalance is evident in several papers through
datasets having relatively more data in the class with wildfires. Due to this,
their models tend to be biased towards detecting wildfires more easily as they
are fine-tuned on more examples of wildfires than images without wildfires
[7].

Figure 5: An Illustration of A Label Imbalance and The Involvement of Class
Rectification Loss [14]. Having more data in one class would introduce a bias
in the model’s judgement of data towards the majority class.

##  3 Research Questions

  1. 1.

How is the dataset collected and why is it comprehensive and inclusive of
wildfire situations?

  2. 2.

Why is the dataset able to mitigate common computer vision problems such as
different atmospheres, model robustness, etc. and what makes it well-suited
for improving not only the model’s accuracy, but also its validity?

  3. 3.

What makes this dataset difficult?

##  4 Methodology

The dataset was consists of 2000 images of wildfires on landscapes susceptible
to containing wildfires and 2000 images of landscapes that are susceptible to
containing wildfires but don’t contain any wildfires at the instance they were
taken. The images of wildfires were collected through website scraping methods
using PyTube, a Python module that can scrape videos from a YouTube channel
and convert them to MP4 files, by using ALERTWildfire’s YouTube channel which
contains hundreds of videos of fires that occurred on landscapes susceptible
to containing wildfires [15] [16]. Once the videos were scraped and converted
into MP4 files, OpenCV, a Python module that has a plethora of methods that
deal with images, was used to capture frame from the video every 10 seconds
[17]. By doing this, the dataset is ultimately able to incorporate more
variety in the images of wildfires as it contains various angles of them while
also maintaining frames that contained wildfires that are representative of
what a real-world input for the model would look like. The images of
landscapes without wildfires were collected through website scraping methods
using Beautiful Soup and PyAutoGUI, two python libraries that are able to
automate website functions such as searching on the website and selecting
buttons and attributes on the website page. ALERTWildfire’s website page was
scraped using these modules by saving each image into the appropriate section
of the dataset by having Beautiful Soup direct the algorithm to the right
website page representing each of the cameras provided by the organization
[18]. PyAutoGUI was used to automatically operate the functions on the site to
save each of the captured images from the cameras [19]. By doing this, the
dataset is also able to incorporate a larger range of data regarding images of
landscapes susceptible to containing wildfires. All in all, what makes this
dataset so comprehensive and inclusive of wildfire situations is that it uses
images of landscapes that have wildfires or are susceptible to containing
wildfires. Therefore, when applied to a real-case scenario on unfamiliar data,
a model trained on this dataset can be more robust and will hence be able to
generalize on more applicable data. As mentioned earlier, this dataset has
been collected through images that are very representative of what the
algorithm would capture as an input in the real world. In addition to this,
the images were also masked on certain parts that would cause bias and did not
correlate with a higher or lower chance of there being a wildfire in those
images. Both of these things definitely help mitigate the bias and
unrepresentativeness present in datasets that are typically showcased in
neighboring research papers in the same field of preliminary wildfire
detection. These neighboring research papers, as mentioned in other segments
of this paper, have issues regarding Model Robustness and Generalization which
often occur from the datasets presented in them that often have a bias due to
the unrepresentative setting, atmosphere, lighting, etc. The dataset formed
from this research mitigates and in some cases solves problems such as the
ones listed above as the masks eliminate the possibility of non-wildfire-
related factors interfering with the model’s weights during training.

##  5 Results and Discussion

Figure 6: The Convolutional Neural Network Structure.

A Convolutional Neural Network was trained on the dataset. Convolutional
Neural Networks are a type of Artificial Neural Network that specializes in
Image Recognition through processing the pixels of an image [20]. This is true
due to its ability to identify key features that distinguish one class from
another–in this case, images containing wildfires from those without them–by
reducing the number of parameters for each consecutive layer. As stated by
Sumit Saha, “The role of the ConvNet [Convolutional Neural Network] is to
reduce the images into a form which is easier to process, without losing
features which are critical for getting a good prediction” [21]. Input images
in Convolutional Networks used for Image Classification are passed through a
filter which performs computations to condense the image’s pixels and
eventually foil down to a single output, which should accurately represent the
input image’s class. The Convolutional Neural Network used for this research
was set to distinguish between images with wildfires and those without them.
It was designed to take images of 128 by 128 pixels, meaning there were a
total of 16384 pixels to start off with. The Convolutional Neural Network was
trained on an initial learning rate of 1e-2, a batch size of 64, 15 epochs,
and ended up having an accuracy of 86%. Below is a graph of the training loss,
validation loss, training accuracy, and validation accuracy.

Figure 7: Training loss and validation loss drop towards 0 while training
accuracy and validation accuracy trend upwards toward 1, demonstrating the
model’s effectiveness through training and its ability to minimize loss and
maximize accuracy.

In comparison to other research performed in the field of preliminary wildfire
detection, the accuracy of this Convolutional Neural Network is a little
lower. However, as mentioned earlier, other research performed in this field
tends to have a heavy load of bias in their datasets and use other factors
that decrease the validity of the accuracy. These neighboring research papers,
as mentioned in other segments of this paper, have issues regarding model
robustness and generalization which often occur from the datasets presented in
them that often have a bias due to the unrepresentative setting, atmosphere,
lighting, etc. The dataset formed from this research mitigates and in some
cases solves problems such as the ones listed above as the masks eliminate the
possibility of non-wildfire-related factors interfering with the model’s
weights during training. One major limitation comes in the form of a lack of
data for the dataset. Although there was a total of 4000 images in the dataset
(2000 images of wildfires and 2000 images of landscapes susceptible to
containing wildfires, but not containing them), more images in each of the
classes would have helped further increase the accuracy of the model since the
features that were associated with wildfires were not very evident in the
images and henceforth adding more data would have helped improve the
robustness of the model. Despite this, the model still showcased an increase
in accuracy and minimizing of loss as it progressed through the training
procedure, which one can note as an initial progression of the Convolutional
Neural Network adapting to and learning the features that determine whether or
not an image contains a wildfire. Therefore, in the future, I can contact
ALERTWildfire and work with the faculty on collecting more data, as that is
very crucial towards the growth of this research.

##  6 Conclusion

The research for this study emphasizes the need for a more representative and
less biased dataset for the purposes of preliminary wildfire detection using
Convolutional Neural Networks that take in data in the form of image data.
After performing background research, it became clear that there was an
abundance of biased and skewed datasets in this field. Problems related to
Label Imbalance and Unrepresentativeness were a common when reading through
past research. However, these were often unmentioned in these studies, leading
to an inaccurate accuracy and validity showcased. Several papers would often
use data where the images of fires, one of the two classes for most models,
would feature orientations and settings that were extremely different from
what an outdoor camera would actually view in the real world. Therefore, this
would result in a not very robust model as the data used for training is
dissimilar to what is actually seen. Through carrying out the experimentation
phase of this study, bias was mitigated from the dataset through inputting
more representative images, masking parts of the image that would not declare
the difference between a wildfire existing or not existing, and inputting an
equal number of images with wildfires and images without them to ensure Label
Imbalance was mitigated. Although there was a total of 4000 images in the
dataset (2000 images of wildfires and 2000 images of landscapes susceptible to
containing wildfires, but not containing them), more images in each of the
classes would have helped further increase the accuracy of the model since the
features that were associated with wildfires were not very evident in the
images and henceforth adding more data would have helped improve the
robustness of the model. Despite this, the model still showcased an increase
in accuracy and minimizing of loss as it progressed through the training
procedure, which one can note as an initial progression of the Convolutional
Neural Network adapting to and learning the features that determine whether or
not an image contains a wildfire.

## References

  * [1] “Facts statistics: Wildfires,” Insurance Information Institute.
  * [2] Cragcrest, “Why california’s wildfires are so destructive, in 5 charts,” Nov 2018\.
  * [3] N. S. D. of the Earth Lab Analytics Hub, “Bringing tech innovation to wildfires: 4 recommendations for smarter firefighting as megafires menace the us,” Jul 2021.
  * [4] P. b. S. Elliott, D. Voican, and A. Singh, “Drones provide eye-in-the-sky to help fight fires,” Jul 2021.
  * [5] A. Faculty, “About alertwildfire,” ALERTWildfire.
  * [6] Simplilearn, “What is computer vision: Applications, benefits and how to learn it,” Simplilearn.com, Apr 2021.
  * [7] J. Sharma, O. Granmo, M. Goodwin, and J. T. Fidje, “Deep convolutional neural networks for fire detection in images,” in Engineering Applications of Neural Networks - 18th International Conference, EANN 2017, Athens, Greece, August 25-27, 2017, Proceedings (G. Boracchi, L. S. Iliadis, C. Jayne, and A. Likas, eds.), vol. 744 of Communications in Computer and Information Science, pp. 183–193, Springer, 2017.
  * [8] Y. Luo, L. Zhao, P. Liu, and D. Huang, “Fire smoke detection algorithm based on motion characteristic and convolutional neural networks,” Multim. Tools Appl., vol. 77, no. 12, pp. 15075–15092, 2018.
  * [9] P. Barmpoutis, P. Papaioannou, K. Dimitropoulos, and N. Grammalidis, “A review on early forest fire detection systems using optical remote sensing,” Sensors, vol. 20, no. 22, p. 6442, 2020.
  * [10] K. Govil, M. L. Welch, J. T. Ball, and C. R. Pennypacker, “Preliminary results from a wildfire detection system using deep learning on remote camera images,” Remote. Sens., vol. 12, no. 1, p. 166, 2020.
  * [11] R. Beneduce, R. Hill, and C. Schelle, “Alertwildfire (group 6),”
  * [12] P. Jain, S. C. P. Coogan, S. G. Subramanian, M. Crowley, S. Taylor, and M. D. Flannigan, “A review of machine learning applications in wildfire science and management,” CoRR, vol. abs/2003.00646, 2020.
  * [13] S. Sankaranarayanan, Y. Balaji, A. Jain, S. Lim, and R. Chellappa, “Learning from synthetic data: Addressing domain shift for semantic segmentation,” in 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pp. 3752–3761, IEEE Computer Society, 2018.
  * [14] Q. Dong, S. Gong, and X. Zhu, “Imbalanced deep learning by minority class incremental rectification,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 41, no. 6, pp. 1367–1381, 2019.
  * [15] “nvseismolab,” YouTube.
  * [16] “pytube.”
  * [17] “Opencv modules.”
  * [18] “Beautiful soup documentation.”
  * [19] “Pyautogui’s documentation.”
  * [20] T. Contributor, “What is convolutional neural network? - definition from whatis.com,” Apr 2018.
  * [21] S. Saha, “A comprehensive guide to convolutional neural networks–the eli5 way,” Dec 2018.