# Fire Threat Detection From Videos with Q-Rough Sets

Debarati B. Chakraborty<EMAIL_ADDRESS>Vinay Detani
<EMAIL_ADDRESS>Shah Parshv Jigneshkumar<EMAIL_ADDRESS>Dept. of
Computer Science and Engineering, Indian Institute of Technology, Jodhpur,
India

###### Abstract

This article defines new methods for unsupervised fire region segmentation and
fire threat detection from video stream. Fire in control serves a number of
purposes to human civilization, but it could simultaneously be a threat once
its spread becomes uncontrolled. There exists many methods on fire region
segmentation and fire non-fire classification. But the approaches to determine
the threat associated with fire is relatively scare, and no such unsupervised
method has been formulated yet. Here we focus on developing an unsupervised
method with which the threat of fire can be quantified and accordingly
generate an alarm in automated surveillance systems in indoor as well as in
outdoors. Fire region segmentation without any manual intervention/ labelled
data set is a major challenge while formulating such a method. Here we have
used rough approximations to approximate the fire region, and to manage the
incompleteness of the knowledge base, due to absence of any prior information.
Utility maximization of Q-learning has been used to minimize ambiguities in
the rough approximations. The new set approximation method, thus developed
here, is named as Q-rough set. It is used for fire region segmentation from
video frames. The threat index of fire flame over the input video stream has
been defined in sync with the relative growth in the fire segments on the
recent frames. All theories and indices defined here have been experimentally
validated with different types of fire videos, through demonstrations and
comparisons, as superior to the state of the art.

###### keywords:

Fire segmentation, video processing , rough sets , Q learning , fire threat
detection , granular computing

##  1 Introduction

Fire is both a threat and a necessity of mankind. We light fires to fulfill
our necessity (e.g., for cooking, generating heat), simultaneously it could be
a threat if it becomes uncontrolled. We know that there are some objects that
are easily flammable (e.g., cloths, plastics) and some that burns out
gradually (e.g., candle, wood or logs in the fireplace etc.). We burn the
latter kind of elements to meet our need in a controlled environment, whereas
fires concerning the former group of elements could be dangerous. Fire spreads
quite fast in the flammable objects, and from one several other objects catch
the fire. Therefore, measuring the spread of the fire could be a solution in
automated fire detection systems to find out the possible threat of the
visible fire. Here we tried to come up with a solution so that the fire threat
can be detected automatically only with a surveillance camera, and no other
sensors will be required. Besides, we tried to make the method fit for both
indoor and outdoor surveillance.

Proper classification of fire region(s) from the non-fire ones and automated
segmentation of the fire region is a very important step while quantifying the
threat of the fire. The aim of this work could be summarized as follows: i) to
develop an unsupervised method for fire pixel classification, ii) to identify
fire regions from different varieties of videos, and iii) to determine the
threat associated with the fire flame over the input video stream. Here in
this article we have formulated a new method of fire region segmentation in
videos using hybridization of rough sets and Q-learning. Besides, we have also
defined a new measure, namely, fire threat index, to quantify the threat
associated with a fire from video streams. Before discussing the other methods
on vision-based fire detection, here in this section we shall roughly discuss
the key concepts of rough set theory and Q-learning relevant to this article.

Theory of rough sets, as explained by Pawlak [1], has become a popular
mathematical framework for granular computing. The focus of the theory is on
the ambiguity caused by limited discernibility of objects in the domain of
discourse. Its key concepts are those of object ’indiscernibility’ and ’set
approximation’. Two major characteristics of the theory that have drawn the
attention of applied researchers are uncertainty handling (using lower and
upper approximations) and granular computing (using information granules).
Theory of rough sets has been proven to be successful in many areas, like,
feature extraction from video streams with information flow [2], on-line
multi-label feature selection from streaming data [3], and information entropy
based fast feature selection for big data analytics [4]. We have aimed to
develop an unsupervised fire detection method here in this article. Since we
have limited access to the knowledge-base while identifying fire in a video
frame, we have approximated the fire region with lower and upper
approximations of rough sets.

Q-learning is a reinforcement learning algorithm [5]. The primary difference
between Q-learning and other reinforcement learning techniques is that it is
model-free [6], i.e., here the agent does not need any state transition model
either for learning or for action selection. Q-learning prefers the best
Q-value from the state reached in the observed transition [7]. The actual
policy, being followed by the agent, is not taken into account in Q-learning,
that is why, it is also known as off-policy control. In Q-learning, the agent
learns the action utility function (or Q-function) to estimate the expected
utility of choosing a certain action when it is in a particular state.

Here in this work we are not considering any prior information on the fire
region or possible spread of the fire pixels. All information is to be
determined by the unsupervised process itself. Therefore, the proposed
solution has an initial rough approximation of fire regions, and then employs
a Q-agent to minimize the ambiguities that is present is the rough
approximation, with utility maximization. This is the broad idea underlying
Q-rough set with utility-based agent. States, in which the Q-agents are
supposed to be in, are different granules (clump of data points [8]) and
actions are insertion or deletion of those granules from fire regions. The
same Q-agents are then employed over the video stream to extract out the fire
regions in each frame. The fire threat index is then formulated using the
segmented fire regions over the video stream.

The novelty of the technique described in the article can be summarized as: i)
definition and application of the Q-rough set (that use unsupervised set
estimations) to minimize ambiguities in rough approximations, ii) segmentation
of fire regions from a video frame with Q-rough set, iii) employing Q-agents
over a sequence of frames to extract fire regions in each frame, and iv)
formulation of a fire threat index to quantify the threat of fire flame in a
video stream. All the theoretical formulations have been experimentally
verified and proven to be effective over the state of the art methods.

The rest of the article is organised as follows. A few state of the art
methods for fire detection are discussed in Section 2. The underlying steps of
the proposed work is described in Section 3. Q-rough set is defined in Section
4 alongwith brief descriptions of rough sets and Q-learning. The method of
fire region segmentation with Q-rough set is developed in Section 5. The
quantification of the threat of fire flame is carried out in Section6. The
qualitative and quantitative experimental results of the proposed methods are
given in Section 7 alongwith suitable comparative studies. The overall
conclusions of this article are drawn in Section 8.

##  2 Related Work

The problem of fire detection from images is being addressed over a decade
[9]. Here we are going to discuss a few bench-mark methods. The problem was
first addressed by Chen _et al._ [10] where fire was detected with RGB values
and rule-base. Ferneds et al. then developed a method for forest fire
detection by classifying lidar signals with neural network committee machine
in [11]. Flame detection by modelling the RGB distributions of fire -pixels
with mixture of Gaussian (MoG) model [12], and with motion-information and
hidden Markov model (HMM) [13] were then done by Toreyin _et al._ Fire pixel
identification in videos was then carried out by Celik [14] where CIE L*a*b
color space was used to identify fire pixels. The method of forest fire
detection by incorporating static and dynamic features in videos in HSV color
space was proposed by Zhao _et al._ in [15]. Chino _et al._ then developed a
way of fire detection from still images by combining color and texture
features in [16]. The method of fire detection with spatio-temporal
consistency energy of each candidate fire region was estimated by
Dimitropoulos _et al._ [17] with prior knowledge about the possible existence
of fire in neighboring blocks from the current and previous video frames, and
an SVM-based classification of fire-non fire regions was also executed there.
Recently a couple of fire detection method has been developed with deep
learning techniques [18]. Muhammad _et al._ [19] came up with a solution by
incorporationg deep features of CNNs in fire-detection and high-priority
cameras based on cognitive radio networks were developed. Kim and Lee [20]
developed a method of detecting fire with faster region grow method using deep
CNN (convolutional neural network). Cai et al. [21] formulated an improved
deep CNN that uses the global average pooling layer instead of the full
connected layer to fuse the acquired depth features and detect fire.

All the methods, we have discussed so far, either requires some initial
information about fire pattern or needs initial manual labelling for training.
Most of the methods are focused on some specific applications, such as, forest
fire, indoor fire, outdoor fire, etc. The method that we developed here is
completely unsupervised and does not need any prior information/ manual
intervention. Besides, it is very effective in classifying fire pixels in any
types of video frame, that is it’s a general method for fire segmentation,
which could be applicable to anywhere for fire detection. The method of
quantification of fire threat is also new to literature. We are going to
describe the basic steps of the method in the following section.

##  3 Proposed Work

Our work can be subdivided into two parts, viz. i) detection of fire from
videos and ii) determination of spreading of fire or quantifying the threat
associated with the fire. The step-wise formulations of the two methods are
shown in Figs. 1(a) and 1(b).

Figure 1: Block diagram of (a) fire region segmentation and (b) threat
detection methods

In Fig. 1(a) proposed fire-flame detection method is shown, that will
accumulate information from different color space and collate the information
judiciously to classify the fire pixels correctly from the video frames. Here
Q-learning is rough because the state-action policy is determined with the
lower approximated regions (obvious fire region here), and the ambiguities
between the two approximations get minimized by utility maximization of the
Q-agents. The same Q-agents are then employed over the rest of the video
sequence to have faster decision making over known states. This part is
described in details in Section 5.

In the second part of the work we aim to quantify the threat associated with
the fire flame. It is shown in Fig. 1(b). Here the relative growth of fire
over recent past of the video stream is considered. The average region of the
fire flame throughout the video stream and over a few recent frames of video
stream are being computed in this part. The fire threat index
($\mathcal{T_{F}}$) is then quantified with these information. This part is
explained in Section 6.

##  4 Formulation of Q-Rough Sets

This section discusses the formulation of Q-rough sets. Prior to that
formulation, the basics of rough set theory and Q-learning have been briefly
described.

###  4.1 Rough Sets

Let there be an information system $S=(\mathcal{U},A)$, where $\mathcal{U}$ is
the universe and $A$ is the set of attributes. For any set $B\subseteq A$,
there is an equivalance relation $IND(B)$ such that
$IND(B)=\\{(x,y)\in\mathcal{U}^{2}|\forall p\in B,p(x)=p(y)\\}$, where $p(x)$
function returns the value of the attribute $p$ for data point $x$. The
relation $IND(B)$ is called B-indiscernibility relation and any two points
$(x,y)\in IND(B)$, i.e., satisfying the B-indiscernibility relation, indicate
that $x$ and $y$ can not be distiguishable using the attribute set $B$. Let
the equivalence class of B-indiscernibility relation be denoted by $[x]_{B}$,
and $\mathcal{U}|B$ denotes all such equivalence classes. Here $[x]_{B}$ is
called a ’granule’ around the data point $x$, created by B-indiscernibility
relation. (As stated before, a granule is a clump of object which can not be
discriminated with a given attribute set.) Let us denote this granulated
information system with $S_{B}=(\mathcal{U},A,[x]_{B})$.

Let $X$ be a set in the universe $\mathcal{U}$ ($X\subseteq\mathcal{U}$) to be
approximated based on the equivalence classes $[x]_{B}$ (i.e., granules)
defined over $B$. Then, $X$ can be approximated in terms of granules from
inner and outer sides as _B-lower approximation_ $\underline{B}X$ and _B-upper
approximation_ $\overline{B}X$ respectively. They are defined as follows:

$\underline{B}X=\\{x\in\mathcal{U}:[x]_{B}\subseteq X\\}$ (1)
$\overline{B}X=\\{x\in\mathcal{U}:[x]_{B}\cap X\neq\emptyset\\}$ (2)

$\underline{B}X$ represents the granules definitely belonging to $X$, while
$\overline{B}X$ means granules definitely and possibly belonging to $X$. That
means all the elements in $\underline{B}X$ can be certainly classified as
member of $X$ on the basis of the knowledge in $B$, while some objects in
$\overline{B}X$ can only be classified as possible members of $X$ on the basis
of $B$.

###  4.2 Q-Learning

Let the state and action in a given environment be denoted as $s$ and $a$
respectively, and the $Q$-value of doing action $a$ in state $s$ be denoted as
$Q(s,a)$. The relation between the direct utility of the state ($U(s)$) and
Q-value is as follows.

$U(s)=\max_{a}Q(s,a)$ (3)

If the Q-values are computed correctly, the following equation (Eqn. 4) will
then only reach to an equilibrium. That is, $LHS=RHS$, iff Q-value is correct
in Eqn. (4).

$Q(s,a)=R(s)+\gamma\sum_{s^{\prime}}P(s^{\prime}|s,a)\max_{a^{\prime}}Q(s^{\prime},a^{\prime})$
(4)

In Eqn. 4 $R(s)$ represents the reward value associated with the state $s$;
$\gamma$ is the discount factor that determines the importance of future
rewards; $P(s^{\prime}|s,a)$ determines the probability of reaching to the
state $s^{\prime}$ from state $s$ given action $a$, it is determined from the
state-transition model; and $Q(s^{\prime},a^{\prime})$ is the $Q$-value of
future action $a^{\prime}$ in future state $s^{\prime}$. As we can see from
Eqn. 4 an estimated state transition model is required here, therefore, it is
not completely model-free. The temporal difference approach or TD Q-learning
is the actually model-free one, i.e., state-transition model is not required
here. The Q-value updation in TD Q-learning is carried out as follows.

$Q(s,a)\leftarrow Q(s,a)+\alpha(R(s)+\gamma(Q(s^{\prime},a^{\prime})-Q(s,a))$
(5)

$\alpha$ is the learning rate in Eqn. 5.

###  4.3 Q-Rough Set: Definition

Here in this article we have focused on unsupervised estimation of a set. The
method is envisioned to be completely unsupervised and only some basic set of
features of the set can be provided apriori. Let us assume the set to be
estimated be denoted as $X:X\in\mathcal{U}^{2}$. The given set of features be
$B:B\subseteq A$, where $A$ is the complete set of features. The lower and
upper approximations of the set: $\underline{B}X$ and $\overline{B}X$ can now
be estimated according to Eqns. 1 and 2 respectively. However, the exact
region of the set cannot be estimated through these equations. To extract the
exact set and minimize the vagueness in the approximation here we employ a
Q-learning agent. Here the set $\underline{B}X$ is used to determine the value
function, policy and the state-transition model in Q-learning. The states
($s$) are treated to be the clump of similar data points (granules $[x]_{B}$)
present in the set $\overline{B}X$, the set of actions, is of cardinality
equals to 2 and it involves: $a=\\{$updation of
$\underline{B}X:\rightarrow\underline{B}X\cup[x]_{B}$ and moving to the next
connected boundary granule to $[x]_{B}$, keeping $\underline{B}X$ the same and
moving to the next boundary granule connected to $\underline{B}_{X}$$\\}$.

The reward function $R(s)$ is chosen based on the similarities between
$[x]_{B}\leavevmode\nobreak\ s$ and $\underline{B}X$ over $n$-dimensional
feature space. The similarities ($D$) between the state ($s\equiv[x]_{B}$) and
model ($M\equiv\underline{B}X$) and reward for that granule ($R([x]_{B})$) are
now defined as per the following equations.

$\displaystyle D(M,s)=||M-s||_{n}$ (6) $\displaystyle R([x]_{B})=1-2D(M,s).$
(7)

In Eqn (6) $R([x]_{B})$ is defined in such that the value of the reward should
be within the range of $[-1,1]$. Maximum reward (+1) will be generated upon
finding total similarity and minimum reward (-1) will be generated upon
finding no similarities.

While determining the Q-value of a boundary granule ($s\equiv[x]_{B}$), it can
be observed that, here the state-action model will lead to two different
future states ($s^{\prime}\equiv[x]_{B}^{\prime}$) based on two different
actions. But the there is a single possible state for a particular action. Let
$[x]_{B1}^{\prime}$ and $[x]_{B2}^{\prime}$ be the two possible future states
that the agent can reach on taking the actions $a_{1}$ and $a_{2}$
respectively. Therefore we are going to get a simple binarized state
transition model as follows.

$\displaystyle P([x]_{B1}^{\prime}|[x]_{B},a1)=1$ $\displaystyle
P([x]_{B2}^{\prime}|[x]_{B},a1)=0$ $\displaystyle
P([x]_{B1}^{\prime}|[x]_{B},a2)=0$ $\displaystyle
P([x]_{B2}^{\prime}|[x]_{B},a2)=1$ (8)

Therefore, Eqn. (4) reduces to Eqn. (9) as follows.

$Q([x]_{B},a)=R([x]_{B})+\gamma\max_{a^{\prime}}Q([x]_{B}^{\prime},a^{\prime})$
(9)

The action, that is maximizing the $Q$-value will be selected. Upon completion
of the same process over all the boundary granules, the final set
$\underline{B}X$, that we are getting is named as Q-rough approximation of the
set $X$. This is how the uncertainty present in the rough approximation of the
set $X$ can be minimized, and exact is found out in the incomplete knowledge-
base.

##  5 Q-Rough Set in Fire Detection

Let the fire region $F$ be approximated in the input frame $f_{t}$, given a
set of features ($B:B\in A$). We have used the set of rules as defined in
[10], for R-G-B and Y-Cr-Cb color space to segment out initial fire regions
from a video frame. Since our method is unsupervised, we are initially
approximating the fire-region in a video frame with these rules. The lower
approximation of fire region is done deploying the information of segmented
regions in Y-Cr-Cb color space and the upper approximation is done considering
the information of segmented region in RGB-color space on the top of the lower
approximated region. The final approximation of the fire region in a frame is
carried out with Q-rough set. A visual illustration of fire segmentation with
Q-rough set is shown in Fig. 2. The process of set formation and approximation
are shown in the following sections.

Figure 2: Pictorial Representation of Fire Segmentation with Q-Rough Set

###  5.1 Formation of Granules

Formation of proper granules plays an important role in approximation in
decision making systems. Here while defining Q-rough set for fire segmentation
we aim to granulate the image frame based on spatio-color similarities. This
is how the separation will be closer to the natural ones. Therefore, we have
used spatio-color granulation technique defined in [2]. That is, a granule
around ($\aleph$) an point $p_{i}$ in the frame $f_{t}$ is formed according to
Eqn. (10).

$\displaystyle\aleph_{sp-clr}(p_{i})$ $\displaystyle=\cup p_{j}\in U$ (10)

where $p_{i}\quad and\quad p_{j}$ are binary connected over the condition,
$|RGB(p_{j})-RGB(p_{i})|<Thr$. The upper and lower approximations of the fire
region ($F$) are carried out over these granules.

###  5.2 Lower Approximation of Fire-Region in a Frame

The lower approximated region will consume the segmented region in Y-Cr-Cb
color space. The rule-base mentioned in [15] for fire segmentation is used
here in Y-Cr-Cb. As we know, in Y-Cr-Cb color space [22], ‘ Y ’ represents the
luma component(very similar to the grayscale conversion of the original image
), whereas ‘ Cb ’ and ’Cr’ represent chroma components of blue-difference red-
difference respectively. The rule-base that is followed for fire segmentation
is in Y-Cr-Cb feature space is as follows.

A pixel $p_{i}$, with Y-Cr-Cb values as $Y(p_{i})$, $Cr(p_{i})$ and
$Cb(p_{i})$ will be treated as a fire pixel if it follows any of the following
rules.

  * •

Rule 1: $Y(p_{i})\geq Cb(p_{i})$ and $Cr(p_{i})\geq Cb(p_{i})$

  * •

Rule 2: $Y(x)>Y_{mean}$ and $Cr(p_{i})>Cr_{mean}$ and $Cb(p_{i})>Cb_{mean}$

  * •

Rule 3: $Cb(p_{i})\leq 120$ and $Cr(p_{i})>150$

$Y_{mean}$, $Cr_{mean}$, and $Cb_{mean}$ are the mean values of Y, Cr and Cb
on that image frame. Let, $F_{YCrCb}$ be the fire-segmented region in the
frame $f_{t}$. Then the lower approximation of the fire region
($\underline{B}F$) is defined according to Eqn. (11). This results in spatio-
color granules that are a subset of $F_{YCrCb}$ are considered to be in the
lower approximation of the set $F$.

$\underline{B}F=\\{\aleph(p):\aleph(p)\in F_{YCrCb}\\}$ (11)

###  5.3 Upper Approximation of Fire Region in a Frame

The upper approximated region will consume both the segmented outputs of Y-Cr-
Cb feature space and RGB feature space. The rules that are defined in [10, 14,
15] for fire detection in RGB feature space as as follows.

A pixel $p_{i}$ will be detected as fire pixel if it follows the following
rules.

  * •

Rule 1: $R(p_{i})\geq R_{mean}$

  * •

Rule 2: $R(p_{i})>G(p_{i})>B(p_{i})$

$R_{mean}$ is the mean value of R over the image frame. Let, $F_{RGB}$ be the
segemnted region of fire in RGB feature space. The upper approximated fire
region will be derived from $F_{RGB}\cup F_{YCrCb}$ region. The upper
approximated fire region ($\overline{B}F$) is defined as according to Eqn.
(12).

$\overline{B}F=\\{\aleph(p):\aleph(p)\cap(F_{RGB}\cup
F_{YCrCb})\neq\emptyset\\}$ (12)

###  5.4 Computation of Reward Function

Proper computation of reward function is another major concern while defining
Q-rough sets. As defined in Eqn. (6), the distance between the lower
approximated region and a boundary granule is to be computed to determine the
reward function. We know that the no. of data points in a $\underline{B}F$ and
a boundary granule $\aleph(p)$ can never be the same. Therefore, instead of
computing point-to-point distance between these two sets, we are computing the
distance between their mean values in different feature space. Therefore, Eqn.
(6) can be re-written here as follows.

$\displaystyle
D(\underline{B}F,\aleph(p))=||mean(\underline{B}F)-mean(\aleph(p))||_{RGBYCrCb}$
(13) $\displaystyle R(\aleph(p))=1-2D(\underline{B}F,\aleph(p)).$ (14)

In Eqn. (13) normalized Euclidean Distance is considered as a distance metric.
The Q-values will now be updated according to Eqn. (15).

$Q(\aleph(p),a)=R(\aleph(p))+\gamma\max_{a^{\prime}}Q(\aleph(p)^{\prime},a^{\prime})$
(15)

###  5.5 Algorithm for Fire Detection with Q-Rough Set

We are going to describe the proposed method step-wise for fire detection from
a video frame $f_{t}$ with Q rough set. The theoretical details are presented
in the preceding sections. The detailed methodology is summarized in the form
of an algorithm in Algorithm 1.

Algorithm 1 Fire Detection from A Frame with Q-Rough Set

INPUT: $f_{t}$

OUTPUT: $f_{t}$ with segmented fire region $F$

INITIALIZE: $\underline{B}F=\overline{B}F$ $\Leftarrow\emptyset$

1: Granulate the frame $f_{t}$ as described in Section 5.1

2: Segment out $F_{YCrCb}$ and $F_{RGB}$ as described in Sections 5.2 and 5.3
respectively.

3: Define $\underline{B}F$ and $\overline{B}F$ following Eqns. (11) and (12).

4: For a boundary granule $\aleph{p}\in\\{\overline{B}F-\underline{B}F\\}$ do
the following. i) Compute $R(\aleph(p))$ with Eqn. (13) ii) Compute
$Q(\aleph(p),a_{1})$ and $Q(\aleph(p),a_{2})$, with Eqn. (15)

if $Q(\aleph(p),a_{1})>Q(\aleph(p),a_{2})$ then

Set $\underline{B}F=\underline{B}F\cup\aleph(p)$ and move to the next
connected granule of $\aleph(p)$

else

Remove $\aleph(p)$ from from $\overline{B}X$ and moving to the granule
connected to $\underline{B}X$.

end if

5: Repeat Step 4 for the next boundary granule.

6: Repeat Step 4 and 5 till $\\{\overline{B}F-\underline{B}F\\}=\emptyset$

7: Set $F=\underline{B}F$

###  5.6 Fire Segmentation Over The Video Sequence

So far we have discussed the method of fire segmentation with Q-rough sets
over a video frame. Since we are dealing with video sequence, we do not need
to repeat the entire process for each frame. Rather, we will employ our
trained Q-agents to explore the fire region in the upcoming frames. That is,
the Q-agents are going to check the spatio-color granules and in rest of the
sequence, and similar granules will automatically be included in the lower
approximated fire region.

Let fire region in the $t^{th}$ frame ($f_{t}$) be approximated as $F_{t}$
with Algorithm 1. Let $\aleph(p_{t+1})$ be a granule under consideration in
the frame $f_{t+1}$. The action over $\aleph(p_{t+1})$ will get automatically
decided by the Q-agent in $f_{t+1}$, since it already knows which action
maximizes the utility from $f_{t}$. The computation of Q-value will only be
required in $f_{t+1}$ if $\aleph(p_{t+1})$ is an unknown state to the Q agent.

##  6 Determination of Threat of Fire

After determination of fire regions in video sequences, we focus to determine
the the threat associated with the fire. Fire is a necessity as well as a
threat in human civilization. Fire is used for light, for heat, to cook etc.
But fire is desired to be used keeping it in control, once it becomes
uncontrolled, it creates threat and damages. Here we plan to develop a
methodology to determine whether the fire is in use or it is becoming a
threat. We have observed that, fire is generally in a controlled position
while the flames are within a certain space, but it becomes a threat when the
flame starts to spread. The faster the flames get spread, greater is the
threat. Here we aim to quantify this phenomenon and accordingly generate an
alarm with the possible threat.

Here we have estimated relative spread in fire regions in a video sequence to
quantify the threat. If the flame is in control, the fire regions may change
from frame to frame, but there will be a flicker effect. That is, the fire
regions may remain within some limits throughout the sequence. But if the fire
becomes uncontrolled, the fire region will spread rapidly compared to its
previous frames. This is why relative spread is considered for the threat
computation. Let $F_{t}$ be the segmented fire region in frame $f_{t}$. Let
the information from $P$ number of previous frames be used to determine the
threat. The average fire segments ($F_{\mu}$) throughout the sequence and the
and recent average fire segments ($F_{\mu P}$) are computed based on the
following equations.

$\displaystyle F_{\mu}=\frac{\sum_{i=1}^{N}F_{i}}{N}$ (16) $\displaystyle
F_{\mu P}=\frac{\sum_{i=N-P}^{N}F_{i}}{P}:P<N$ (17)

The threat index of fire ($\mathcal{T_{F}}$) is now defined with relative
increment of spread in recent frames. It is computed with Eqn. (18).

$\mathcal{T_{F}}=\frac{F_{\mu P}-F_{\mu}}{F_{\mu}}$ (18)

Please note that a signed difference is considered in the numerator of right
hand side of in Eqn. (18). This is because, the threat will only be positive
if there if a relative spread in fire region, if the region is concentrated or
the fire is extinguishing, the threat will become negative. Thereby
eliminating occurrences of false alarms.

Selection of the value of $P$ plays a major role in this index. Here we have
decided to consider the relative growth of fire over the last one second.
Therefore, $P$ is chosen based on the frame rate of video acquisition per
second (fps). That is, if the video is acquired with a frame rate of 30 fps,
$P$ is set to be $30$.

##  7 Experimental Results

Here in this section we have experimentally demonstrated the effectiveness of
the proposed methods in identifying the followings. The unsupervised fire
segmentation method, developed here is proven to be superior in terms of
qualitative and quantitative results over different types of input videos,
from, a few state of the art methods. The threat detection method is also
found to be effective in quantifying the threat associated with the fire.

We have conducted our experiment with 30 different types of videos including
more than 25000 video frames, with fire flame. Ten video sequences out of the
thirty considered videos contain spreading fire or uncontrolled fire, and the
rest contain controlled fire. Ten video sequence sets were acquired from the
sets from ’Firesence’ database [23], and twenty video sequences from different
links freely available on ’YouTube’. To limit the siz of the article we have
shown the qualitative and quantative results obtained over seven different
video sequences. Out of which we have selected only two videos where the fire
region remained to be within a range, fire is either rapidly growing, or
slowly spreading in the rest of the five videos shown here.

The video sequences, over which we have shown the results are described as
follows. ’PosVideo1’ [23] shows a part of a bus that is set on fire, the fire
is gradually growing here. ’PosVideo2’ [23] shows fire is set into a kitchen
and it is growing gradually. In ’PosVideo4’ and ’PosVideo5’ [23] fire is under
control, two different burning fire-places are captured in these videos. Two
men are setting fire in a forest region and fire start to grow is the content
of ’PosVideo6’ [23]. In ’WeddingDress’ [24] video, fire is initially spreading
over the wedding dresses and then becomes constant, whereas in ’ChristmasTree’
[25] video a Christmas tree catches fire that spreads rapidly. The
effectiveness of Q-rough sets in segmenting out the fire regions is shown in
the next section.

###  7.1 Fire Region Segmentation with Q-rough Set

Here in this section we are going to demonstrate the effectiveness of Q-Rough
sets in identifying the fire segments throughout the video sequences. The
visual segmented outputs for four different frames over seven different
sequences, described above, are shown in Figs. 3(1) to 4(7). It can be
concluded from the visual results that the fire regions are well segmented out
with Q-rough sets.

The segmentation accuracy is quantitatively demonstrated in Table 1. Please
note that, there was no ground truth available for the video sequences used
here for the experimentations. Therefore, we have manually annotated ten
frames, selected randomly from each video sequence, and performed the
quantitative analysis. The average $Falsepositive\leavevmode\nobreak\ (FP)$,
$FalseNegative\leavevmode\nobreak\ (FN)$, $Precision$, and $Recall$ values are
given here in Table 1.

Table 1: Fire Segmentation Accuracy of Q-Rough Sets Sequence | $FP(\%)$ | $FN(\%)$ | $Precision$ | $Recall$
---|---|---|---|---
PosVideo1 | 0.5 | 15 | 0.99 | 0.87
PosVideo2 | 3 | 5 | 0.97 | 0.95
PosVideo4 | 4 | 1 | 0.96 | 0.99
PosVideo5 | 3 | 2 | 0.97 | 0.98
PosVideo11 | 2 | 10 | 0.98 | 0.9
WeddingDress | 2 | 8 | 0.98 | 0.91
ChristmasTree | 23 | 5 | 0.77 | 0.93

From Table 1 it can be observed that, the Q-rough set method is found to be
quite efficient in segmenting out the fire regions almost in every video
except ChistmasTree. The fire regions are well segmented even the ones that
are small in size (PosVideo11 and PosVideo1), continuously burning (PosVideo4
and PosVideo5) or spreading rapidly (PosVideo2 and WeddingDress). In the
ChristmassTree video the fire gets reflected in the wall and the wall reflects
almost of similar color as that of the fire. Therefore, some part of the wall
also gets segmented out as fire.

(1)
(2)
(3)
(4)

Figure 3: Fire segmentation results with Q-rough sets for frame nos. (1) 15,
45, 75, 105 of ’PosVideo1’ sequence (2) 25, 45, 85, 105 of ’PosVideo2’
sequence (3) 25, 45, 85, 105 of ’PosVideo4’ sequence, (4) 25, 45, 85, 105 of
’PosVideo5’ sequence

(5)
(6)
(7)

Figure 4: Continuation of Fig. 3 (5) 25, 45, 85, 105 of ’PosVideo11’ sequence,
(6) 25, 45, 85, 105 of ’WeddingDress’ sequence, and (7) 25, 45, 85, 105 of
’CristmassTree’ sequence; (i) input, (ii) segmented

###  7.2 Comparative Study

Here in this section we have demonstrated qualitative and quantitative
comparison of our fire segmentation method with four state of the art methods.
The methods, with which the comparative studies are performed include: i) RGB
and motion feature based method with HMM (RGB-M) [13], ii) HSV model based
method developed for forest fire detection with static and dynamic features
(HSV-SD) [15], iii) spatio-temporal flame modelling method (ST-F) [16], and
iv) fire detection with improved deep learning (F-IDL) [21] method.

The visual comparison over a frame of the seven different video sequences on
with the five different methods are shown in Fig. 5. From the visual results
it can be seen that the Q-rough set method is segmenting out the fire region
most efficiently.

(1)
(2)
(3)
(4)
(5)
(6)
(7)

Figure 5: Visual comparison on fire segmentation on (1) PosVideo1 sequence,
(2) PosVideo2 sequence, (3) PosVideo4 sequence, (4) PosVideo5 Sequence, (5)
PosVideo11 sequence, (6) WeddingDress sequence, and (7) ChristmassTree
Sequence; (a) input, (b) RGB-M method, (c) HSV-SD method, (d) ST-F method, (e)
F-IDL method, and (f) Q-rough set (proposed) method

Please note that, none of the methods used in the comparative study focus on
classifying each fire pixel to the non-fire ones, therefore, the measures of
classification accuracy, like, true positive, false negative is not going to
be a fair parameter here for the comparative study. Therefore, a measure,
namely, RMSE (root mean square error) is taken under consideration here for
quantitative comparison. The root mean square error between the four corner
pixels of the bounding boxes, covering the ground truth regions of fire, and
the segmented regions of fire are considered here for the measurement. The
average RMSE values for the seven videos sequences with five methods are given
in Table 2

Table 2: Fire Region Segmentation: Comparison with Average RMSE Sequence | $RGB-M$ | $HSV-SD$ | $ST-F$ | $F-IDL$ | $Q-RoughSet$
---|---|---|---|---|---
PosVideo1 | 52.3 | 8.2 | 6.6 | 13.4 | 5.8
PosVideo2 | 46.8 | 11.1 | 7.2 | 5.8 | 6.9
PosVideo4 | 9.2 | 6.3 | 5.9 | 6.1 | 5.5
PosVideo5 | 8.8 | 5.2 | 4.7 | 5 | 4.9
PosVideo11 | 48.6 | 11.2 | 9.5 | 7.3 | 4.9
WeddingDress | 19.4 | 17.8 | 8.3 | 12.2 | 5.1
ChristmasTree | 34.6 | 13.5 | 12.6 | 21.2 | 16.3

It can be observed from Fig. 5 and Table 2 that the proposed unsupervised
method performs superior or equally well to those of the state of the art
methods. Besides, it can also be seen from the visual results (Fig. 5), that
Q-rough set method can classify the fire pixels from the non-fire ones better
than the other methods in comparison.

###  7.3 Effectiveness of Fire Threat Index $\mathcal{T_{F}}$

The effectiveness of $\mathcal{T_{F}}$ index in identifying the threat of fire
is validated here with extensive experiments. The values of $\mathcal{T_{F}}$
for each frame throughout the seven video sequences are plotted here in Fig.
6. The $\mathcal{T_{F}}$ values for all the five methods, over which the
comparative studies are performed, are computed here and plotted accordingly
in Fig. 6. The red line represents the values of $\mathcal{T_{F}}$ indices
obtained by Q-rough set method, the green line indicates $RGB-M$ method, blue
line indicated $HSV-SD$ method, gray ’- -’ line indicates $ST-F$ method, and
the solid black line indicates $F-IDL$ method in Fig. 6. The increament or
decrement of fire flame is well reflected by the proposed $\mathcal{T_{F}}$
index. For example, fire is initially increasing and then decreasing in
$PosVideo1$-sequence which is reflected in the curve in Fig. 6(1) almost by
all the methods. The increasing threat of fire of $PosVideo2$-sequence can be
inferred from Fig. 6(2) with the increasing values of $\mathcal{T_{F}}$. Fire
flames have a flicker effect, and no threat in $PosVideo4$ and $PosVideo5$
sequences. It is best reflected by Q rough set method in Figs. 6(3) and 6(4),
where the $\mathcal{T_{F}}$ values are withing a fixed range closed to the
value zero. The gradually spreading fire in $PosVideo11$-sequence can be
inferred from 6(5) with Q-rough set method. Initial spread and then
concentration of fire in $WeddingDress$-sequence is reflected well in Fig.
6(6). The sudden spread of fire in $ChristmasTree$-sequence can be well
detected in Fig. 6(7). Therefore, the proposed $\mathcal{T_{F}}$ index is
successful in identifying the threat of fire correctly.

(1) (2)
(3) (4)
(5) (6)
(7)

Figure 6: $\mathcal{T_{F}}$ index values over 1) PosVideo1 sequence, (2)
PosVideo2 sequence, (3) PosVideo4 sequence, (4) PosVideo5 Sequence, (5)
PosVideo11 sequence, (6) WeddingDress sequence, and (7) ChristmassTree
Sequence

##  8 Conclusions and Future Work

In this article we have primarily addressed two tasks related to fire threat
detection from videos. The first one is unsupervised fire region segmentation
with Q-rough sets and the second one is definition of fire threat index. The
unsupervised method of fire region segmentation with Q-rough set has been
proven to be effective visually and quantitatively in classifying fire pixels
correctly. The performance of Q-rough set based fire segmentation has been
proven to be superior in identifying the correct fire region over different
types of video sequences with respect to different state of the art methods.
The fire threat index is proven to be effective in reflecting the spread of
the fire quickly. This index works well with all the fire segmentation
methods, but it reflects the threat the best with Q-rough set method.

Therefore it can be concluded that the proposed methods can be integrated in
video surveillance systems to generate quick fire alarm. The Q-rough set based
segmentation method could also be used in the other areas of video analysis,
like, object tracking since the Q-agents could learn and explore the object of
interest by themselves alongwith the input stream data. Besides, the lower-
upper approximations with Q-rough set, defined here, could be used other areas
to deal with the incompleteness of knowledge base and stream data.

The fire threat index is defined here assuming that the surveillance videos
will be captured by static cameras. But the method may fail if the
surveillance is carried out with moving cameras or ego-centric videos. The
definition of the index could be modified in future to make it more general
and applicable to any kind of surveillance systems.

## References

  * [1] Z. Pawlak.  Rough Sets: Theoretical Aspects of Reasoning about Data.  Kluwer Academic Publishers, Norwell, MA, 1992.
  * [2] S. K. Pal and D. B. Chakraborty.  Granular flow graph, adaptive rough rule generation and tracking.  IEEE Trans. on Cyberns., 47(12):4096 – 4107, 2017.
  * [3] J. Liu, Y. Lin, Y. Li, W. Weng, and S. Wu.  Online multi-label streaming feature selection based on neighborhood rough set.  Pattern Recognition, Elsevier., 84(12):273–287, 2018.
  * [4] J. Zhaoa, J. Liang, Z. Dong, D. Tang, and Z. Liu.  Accelerating information entropy-based feature selection using rough set theory with classified nested equivalence classes.  Pattern Recognition, Elsevier, 107(11):273–287, 2020.
  * [5] H. V. Hasselt.  Double q-learning.  In J. D. Lafferty, C. K. I. Williams, J. Shawe-Taylor, R. S. Zemel, and A. Culotta, editors, Advances in Neural Information Processing Systems 23, pages 2613–2621. Curran Associates, Inc., 2010.
  * [6] R. S. Sutton and A. G. Barto.  Reinforcement Learning: An Introduction.  The MIT Press, second edition, 2018.
  * [7] S. Russell and P. Norvig.  Artificial Intelligence: A Modern Approach.  Prentice Hall, 4 edition, 2020.
  * [8] Lotfi A. Zadeh.  Toward a theory of fuzzy information granulation and its centrality in human reasoning and fuzzy logic.  Fuzzy Sets Syst., 90(2):111–127, 1997.
  * [9] A. Gaur, A. Singh, A. Kumar, A. Kumar, and K. Kapoor.  Video flame and smoke based fire detection algorithms: A literature review.  Fire Technology, Springer, 56(5):1943–1980, 2020.
  * [10] P. Wu T. Chen and Y. Chiou.  An early fire-detection method based on image processing.  In Proc. IEEE Int. Image Process., pages 1707–1710, 2004.
  * [11] A. M. Fernandesa, A. B.Utkin, A. V.Lavrov, and R. M.Vilar.  Development of neural network committee machines for automatic forest fire detection using lidar.  Pattern Recognition, Elsevier., 37(10):2039–2047, 2004.
  * [12] Y. Dedeoglu B.U. Toreyin and A.E. Cetin.  Flame detection in video using hidden markov models.  In Proc. IEEE Int. Conf. Image Process., pages 1230–1233, 2005\.
  * [13] Y. Dedeoglu B.U. Toreyin and A.E. Cetin.  Computer vision based method for real-time fire and flame detection.  Pattern Recognition Lett., 27:49–58, 2006.
  * [14] T. Celik.  Fast and efficient method for fire detection using image processing.  ETRI Journal, 32:881–890, 2010.
  * [15] J. Zhao, Z. Zhang, S. Han, C. Qu, Z. Yuan, and D. Zhang.  Svm based forest fire detection using static and dynamic features.  Computer Science and Information Systems, 8:821–841, 2011.
  * [16] D. Y. T. Chino, L. P. S. Avalhais, J. F. Rodrigues, and A. J. M. Traina.  Bowfire: Detection of fire in still images by integrating pixel color and texture analysis.  In SIBGRAPI Conference on Graphics, Patterns and Images, pages 95–102, 2015.
  * [17] K. Dimitropoulos, P. Barmpoutis, and N. Grammalidis.  Spatio-temporal flame modeling and dynamic texture analysis for automatic video-based fire detection.  IEEE Trans. on Cir. Sys. Vid. Tech., 25(2):339–351, 2015.
  * [18] P. Li and W. Zhao.  Image fire detection algorithms based on convolutional neural networks.  Case Studies in Thermal Engineering, June 2020 , Elsevier, 19:1–11, 2020.
  * [19] K. Muhammad, J. Ahmad, and S. W. Baik.  Early fire detection using convolutional neural networks during surveillance for effective disaster management.  Neurocomputing, Elsevier, 288:30–42, 2018.
  * [20] B. Kim and J. Lee.  A video-based fire detection using deep learning models.  Appl. Sci., MDPI, 9:2862 – 2874, 2019.
  * [21] Y. Cai, Y. Guo, Y. Li, H. Li, and J. Liu.  Fire detection method based on improved deep convolution neural network.  In ACM ICCPR: Int. Conf. on Computing and Pattern Recognition, pages 466–470, 2019.
  * [22] E. Reinhard, G. Ward, S. Pattanaik, and P. Debevec.  High Dynamic Range Imaging: Acquisition, Display, and Image-Based Lighting (The Morgan Kaufmann Series in Computer Graphics).  Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 2005.
  * [23] N. Grammalidis, K. Dimitropoulos, and E. Cetin.  Firesense database of videos for flame and smoke detection [data set].  In Zenodo, pages 466–470, 2017.
  * [24] Inside Edition.  Daring brides light wedding dresses on fire while wearing them.  In YouTube, Link: https://www.youtube.com/watch?v=yNdaJblEqiU, 2018\.
  * [25] National Fire Protection Association.  Christmas tree fires can turn devastating and deadly within seconds.  In YouTube, Link: https://www.youtube.com/watch?v=xr6b9b8FYKkt=5s, 2015.