# Decision-making of Emergent Incident based on P-MADDPG

Yibo Guo, Lishuo Hou, Mingxin Li, Yue Yuan, Shun Liu, Jingyi Xue, Yafang Han,
Mingliang Xu
_School of Computer and Artificial Intelligence_ ,
_Zhengzhou University_ Yibo Guo, Lishuo Hou, Mingxin Li, Yue Yuan, Shun Liu,
Jingyi Xue, Yafang Han, Mingliang Xu.Decision-making of Emergent Incident
based on P-MADDPG. arxiv:submit/4215882, 2022This paper was produced by NSFC
61602421This paper was produced by China Postdoctoral Science Foundation
2016M600584.

###### Abstract

In recent years, human casualties and damage to resources caused by emergent
incidents have become a serious problem worldwide. In this paper, we model the
emergency decision-making problem and use Multi-agent System (MAS) to solve
the problem that the decision speed cannot keep up with the spreading speed.
MAS can play an important role in the automated execution of these tasks to
reduce mission completion time. In this paper, we propose a P-MADDPG algorithm
to solve the emergency decision-making problem of emergent incidents, which
predicts the nodes where an incident may occur in the next time by GRU model
and makes decisions before the incident occurs, thus solving the problem that
the decision speed cannot keep up with the spreading speed. A simulation
environment was established for realistic scenarios, and three scenarios were
selected to test the performance of P-MADDPG in emergency decision-making
problems for emergent incidents: unmanned storage, factory assembly line, and
civil airport baggage transportation. Simulation results using the P-MADDPG
algorithm are compared with the greedy algorithm and the MADDPG algorithm, and
the final experimental results show that the P-MADDPG algorithm converges
faster and better than the other algorithms in scenarios of different sizes.
This shows that the P-MADDP algorithm is effective for emergency decision-
making in emergent incident.

###### Index Terms:

Emergent incident, MAS, GRU, MADDPG.

##  I Introduction

Principled decision making problem in emergency response management
necessitates the use of statistical models that predict the spatial-temporal
likelihood of incident occurrence due to the influence of complex environment,
high uncertainty, lack of information and other factors. In addition to the
above difficulties, the actual dynamic scheduling is running in the stochastic
environment of inevitable interruption or unpredictable events. These
uncertain disturbances may disturb the pre- established optimal scheduling, or
even make it infeasible. The original task sequence is affected when an
accident occurs in the scheduling process. The traditional solutions under
these circumstances are mostly involved with rescheduling, which will
interrupt the original task sequence in case of emergency. However, such
method are not capable of satisfying the temporal constrains and the
complicated evolution of events. Therefore, the study of emergency decision-
making (dynamic scheduling) is imminent.

In the emergency process, we often encounter high dynamic and timeliness
emergency tasks. When there is an emergency, in order to maximize the
completion of the original task, the optimal decisions have to be compromised.
In recent years, unmanned system can complete difficult tasks such as search
and rescue [1], [2], [3], space underwater detection [4], surveillance and
target tracking [5], [6], transmission and communication in complex
environment, and has the characteristics of autonomy, flexibility, low
operation cost, good availability, more safety and efficiency, while systems
modeled in multi-agents have become a research hotspot in various fields. By
the recent reports and surveys, the unmanned system such as robots or UAVs
have shown significant potential of reducing the cost and time needed for
dynamic scheduling [7]. Compared with the traditional rescue teams, the
unmanned system can substitued human in the rescue task of dangerous
accidents, saving considerable risk and costs.

In this paper, we mainly focus on the dynamic scheduling problems for unmanned
system to solve EDUA problem in the case of time limit. One of the challenges
of using unmanned systems is to coordinate their tasks.Specifically, in an
emergency scenario, there is a group of task sequence. The unmanned system can
solve the accident on the original task sequence without interrupting the
original task sequence, and ultimately ensure the normal operation of the
original task sequence and complete the rescue work, that is, the agent needs
to execute the task before the deadline of the task.The main two goals are: 1)
to maximize the benefits of the original task; 2) Minimize the impact of the
accident. We propose a new framework called T-MADDPG), which combines
transformer and MADDPG to solve EDUA problem. In the dynamic complex
environment [8], multi-agent needs to complete different tasks, such as
rescue, rescue resources. It can effectively rescue the sudden accident in the
confined space, and make the best decision dynamically according to the
accident spread. The flexible deployment of multi-agent team can effectively
avoid collision and cooperate well with the team [9]. The experimental results
verify the ability of the Multi-Agent Reinforcement Learning in the process of
solving emergencies. The new algorithm proposed in this paper is especially
suitable for emergency decision-making, and can also be extended to other
similar scenarios.

The main contributions of this paper can be summarized as follows.

#### I-1

We propose a novel model for evaluating strong temporal-constrained emergency
response tasks. We have In this model, the temporal expand and potential
hazard. The advantage of our model is that it can be applied to the
environment is determined while the emergency response and dispath tasks are
uncertain, which are common in unmanned logistic workshops and factories.

#### I-2

Unlike the majority of forecasting models in literature, we create a general
approach that is flexible to accommodate both the recessive and dominant
events with MADDPG.

#### I-3

We have developed three representative scenarios for simulation and
evaluation. The performance of multi-agent in EDUA is tested, and the results
show the superiority of the proposed Transformer-MADDPG method.

The rest of this paper is organized as follows. Section II provides some
related work about the application of dynamic scheduling and the application
of unmanned system in dynamic scheduling, corresponding to the content of
introduction. Section III gives the definition of EDUA. In section IV, the
problem of multi-agent EDUA is further described.In section V, we propose a
transformer- MADDPG method, which can predict the accident situation and the
trend of accident spread, and finally solve the multi-agent EDUA
problem.Section VI gives the test setup and analyzes the test results.
Finally, Section VII draws the conclusion and suggests the future work.

##  II RELATED WORK

Emergency response management (ERM) is a challenge faced around the globe.
First responders need to respond to a variety of incidents such as
fires,traffic accidents, and medical emergencies. They must respond quickly to
incidents to minimize the risk to humanlife [10, 11]. Consequently,
considerable attention in the last several decades has been devoted to
studying emergency incidents and response. The definition of emergent
incidents in the traditional approach can be divided into two categories [12].
One way to categorize incidents is by the rate at which they occur and how
they affect first responders. For example, some incidents happen often, and
addressing them is part of day-to-day first-responder operations. Examples of
such incidents include crimes, accidents, calls for medical services, and
urban fires. A second category consists of comparatively less frequent
incidents, which include natural calamities like earthquakes, floods, and
cyclones. The emergent incidents we study in this paper are
unconventional,which have the following characteristics: they are time-
sensitive, have a large number of documented cases, and spread in the
environment after they occur.

In many cases, large area disasters are usually caused by small-scale emergent
incidents on a small scale. Large area disasters could be possibly be
prevented if the incipient small-scale emergent incidents are detected in
their early stages. The past decade has seen effective proposals to approach
this problem with the deployment of multiple mobile sensors for monitoring
emergent incidents prone areas. and allow full map coverage. Compared to
static sensor networks, robotic sensor networks offer advantages such as
active sensing, large area coverage and anomaly tracking. In 2015 David
Saldana et al. [13] studied the coordination and control problem of dynamic
anomaly emergent incidents detection of multiple robots in the environment and
proposed a distributed multi-robot dynamic anomaly detection and tracking
method. Using a probabilistic approach that integrates full-map coverage and
tracking of multiple dynamic anomalies with a team of robots. In the searching
phase, robots are guided toward the spots with high probability of the
existence of an emergent incident. The main contribution of this work is the
combination of efficient exploration under uncertain conditions, emergent
incident tracking and autonomous online assignment of intelligent bodies. The
robot explores working areas that maintain the history of the sensed area to
reduce redundancy and allow full map coverage. In [14], the authors proposed
an algorithm for cooperative detection of rapidly changing regions based on
statistical estimation. In [15], a comprehensive emergent incidents anomaly
detection system is proposed.This system uses deep neural networks to monitor
critical infrastructures such as warehouses, airports, and ports. In [16],the
authors proposed an on-line monitoring frame-work for continuous real-time
safety/security in learning-based control systems. In [17], a surveillance
emergent incidents detection method based on UAV-acquired video is proposed.

Mukhopadhyay A et al. [18] proposed that the overall problem of emergency
response to emergent incidents is actually more than just dispatching rescue
personnel to the scene of an incident; specifically, emergency response can be
divided into three sub-processes: (a) incident prediction, (b) resource
allocation, and (c) handling computer aided dispatch to handle the emergency
conditions.

Emergent incidents prediction is necessary to understand the likely demand for
emergency resources in a given region. An important consideration when
designing a prediction model is the frequent dynamic changes in the
environment. Therefore, emergent incidents prediction methods should take into
account the changes in the environment [19]. Recently, researchers have
developed online models for predicting incidents, using input data streams to
continuously update the learned models [20]. In [21], the authors propose a
new emergent incident prediction mechanism that establishes a nonlinear
mathematical procedure that provides maximum coverage of desired events. The
prediction model overcomes the problems of traditional prediction models that
do not consider the priority of incident importance, spatial modeling that
considers discrete regions independently, and learned models that are
homogeneous by combining incident arrival time with incident severity. Ayan
Mukhopadhyay et al. in [22] proposed the use of a predictive statistical model
in emergency management for predicting the spatial and temporal likelihood of
statistical emergent incidents occurring. The model uses a combination of
synthetic resampling, non-spatial clustering, and from data-based methods to
predict the spatio-temporal dynamics of incident occurrence and assign first
responders across a spatial region to reduce response time. In [23], the
authors propose a general approach to emergent incident prediction that is
robust to spatial variation, addressing the problem of events shifting in
space.

For the scheduling processing of emergent incidents, traditional and
intelligent scheduling methods are usually used. Traditional methods such as
optimization methods, heuristic methods, simulation methods. Intelligent
scheduling methods such as expert systems, neural network methods, intelligent
search algorithms, multi-agent methods. The most used traditional methods are
heuristic scheduling algorithms and simulation methods. Intelligent scheduling
methods such as expert systems, neural networks and genetic algorithms.
However, traditional methods for solving scheduling problems are usually not
suitable for emergency decision-making for emergent incident(EDEI) problems,
and most traditional algorithms tend to be too inflexible and slow when faced
with large-scale scenarios in which the above-mentioned accidents occur [24],
and end up providing suboptimal solutions that are not applicable to the
problem studied in this paper.

The Multi-agent method has become one of the hot spots for research because of
its speed, reliability and scalability. Mukhopadhyay A et al. in [25] proposed
two partially decentralized multi-agent planning algorithms for the problem of
myopic and straight-forward decision policies present in ERM systems. In [24],
a dynamically evolving scheduling mechanism is used to combine centralized and
distributed policies. Global optimal scheduling is first implemented and a
fast rescheduling solution is invoked in case of changes. In [26], a mediator
mechanism is used to help agents dynamically find other agents that can
contribute to a given task. Unmanned systems for problems of this type are
scarce in the literature. In 2020, Shaurya et al. [27] proposed the emergency
task processing in task allocation of multi robot team, and proposed a pruning
heuristic method for the original task to add the emergency task. In 2018,
Joanna Turner et al. [28] proposed a distributed task rescheduling algorithm
based on time constrained task allocation optimization for multi robot
systems. The main challenge is to achieve the optimal allocation, maximize the
completion of the original task sequence and minimize the harm of the
accident. Duan T et al. [29] proposed a dynamic fault-tolerant task scheduling
model (DSMFNA) to solve the problems of node failure and inability to
dynamically provide capacity requirements during UAV task execution,and
proposed a flexible network structure dynamic scheduling algorithm(FDSA).And a
flexible network architecture dynamic scheduling algorithm (FDSA) is proposed.
However, these architectures are not suited to the problems of interest in
this paper (EDEI problems)as here.

##  III PRELIMINARIES AND PROBLEM STATEMENT

In this section, we will introduce the formal definition of the multi-agent
emergency decision-making for emergent incident(EDEI) and key concepts . Then,
we define the general problems of multi-agent emergency decision .

###  III-A Concept Definition

Definition 1(Operation Graph). The graph in this paper divides the real
physical space into various regions and abstracts it into a graph
$\mathcal{G}=(V,E)$, where $V$ represents the set of nodes, abstracting the
more important hubs or the area where the assignment needs to be performed as
nodes. The attribute $w_{i}$ of $V$ represents the number of assets for node
$v_{i}$. The nodes are divided into three sets according to whether an
incident occurs and whether the incident is completely destroyed. The set of
normal nodes $V^{n}$ that are not spread by the incident, the set of nodes
$V^{f}$ that are spread by the incident and then have an incident, and the set
of scrapped nodes $V^{s}$ that are completely destroyed by the incident. $E$
denotes the set of edges between nodes that physically connect physical
channels. The attributes $d_{i,j}$ of $E$ represent the distance between nodes
$v_{i}$ and $v_{j}$.

Definition 2(Asset). The assets represent the device assets available to
complete the assignment and are denoted by
$S=\\{(s_{1}^{w_{1}},...,s_{1}^{w_{i}}),(s_{2}^{w_{1}},...,s_{2}^{w_{j}}),...,(s_{z}^{w_{1}},...,s_{z}^{w_{k}})\\}$,
where $z$ denotes node $v_{z}$ and $w_{k}$ denotes the number of assets of
$v_{z}$. According to the damage of the node asset can determine the emergency
situation of the node incident, according to the number of node asset can
determine the importance of the node, so that the emergency recovery task
scheduling of the agent.

Definition 3(Primary assignment). The primary assignment sequence, which
refers to the normal execution of the work process before an incident occurs,
is denoted by $O$. Let
$O=\\{o_{1}(v_{1},{et}_{1},w_{1}),...,o_{m}(v_{m},{et}_{m},w_{m})\\}$
represents the primary assignment sequence, where $v_{i}$ represents the node
of the primary assignment $o_{i}$, ${et}_{i}$ represents the deadline of the
primary assignment, and $w_{i}$ is the number of assets at $v_{i}$. The
primary assignment sequence can form a single queue by the value of
${et}_{i}$. When performing primary assignments, a series of assets are
needed. When an incident occurs, the asset at the node is destroyed, and if
all the assets of a node are destroyed, the assignment of the node is judged
to have failed and $s_{i}^{j}=0$ and
$V^{f}\overset{+}{\leftarrow}\\{v_{i}|v_{i}\in O\\}$.

Definition 4(Emergent incident). Emergent incident denotes a spreadable hazard
that occurs at a node of the operation graph $\mathcal{G}$. The trend of
hazard severity over time at the node is $f(t)$ .

Definition 5(Alleviate anomaly). In this paper, alleviate anomaly means to
move assets at a node from one node to another. As in graph $\mathcal{G}$
moving assets from $v_{i}$ to $v_{j}$, i.e., $s_{k}^{w_{x}}\leftarrow
s_{j}^{w_{y}}$, $s_{j}^{w_{y}}=0$. The set of nodes to which the asset of a
$v_{i}$ node can be moved is defined as
$V_{i}^{R}=\\{v_{i}^{1},v_{i}^{2},...,v_{i}^{k}\\}$, where
$v_{i}^{k}\in\\{V\xleftarrow[]{-}\\{v_{i}\\}\\}$. The asset at $v_{i}$ can be
moved to any node in the set $V_{i}^{R}$.

Definition6(Agent). An agent refers to an unmanned vehicle that performs
emergency recovery tasks, denoted as $A=\\{a_{1},...,a_{n}\\}$. The agent
needs to ensure that the primary assignment sequence proceeds normally without
interrupting the primary assignment sequence and with minimal impact on the
primary assignment sequence. Emergency recovery tasks are specifically
classified as rescue or salvage node assets.

###  III-B Multi-agent Emergency Decision-making

Based on the above definition, the problem of multi-agent emergency decision-
making is as follows: given an undirected graph $\mathcal{G}=(V,E)$ to
represent the operation graph, and the set of $n$ agent
$A=\\{a_{1},...,a_{n}\\}$, a set of ordered primary assignment sequence set
$O=\\{o_{1}(v_{1},{et}_{1},w_{1}),...,o_{m}(v_{m},{et}_{m},w_{m})\\}$, $m>n$,
where $v_{i}$ represents the node of the primary assignment $o_{i}$,
${et}_{i}$ represents the deadline of the primary assignment, and $w_{i}$ is
the number of assets at $v_{i}$. The primary assignment is an unmanned
operation scenario in which the unmanned device traverses a number of nodes on
the operation graph within the primary assignment deadline. When an emergent
incident occurs on graph $\mathcal{G}$, the incident spreads along the node to
the surrounding nodes in the graph. Agents need to perform emergency recovery
tasks and ensure maximum completion of the primary assignment sequence.

The problem is solved by having agents complete emergency recovery tasks
without conflict, so that the primary assignment sequence completes primary
assignment with maximum total value and with minimum impact from incidents.

$J=\text{max}\sum V_{alue}(O)+\text{min}\sum V_{alue}(V^{f})$ (1)

Among them, $V_{alue}(O)$ represents the most value of primary assignment
completion; $V_{alue}(V^{f})$ represents the total value affected by the
incident.

##  IV PROBLEM DESCRIPTIONS

In the real world, emergent incidents often affect a wide range of factors and
matters, such as complex dynamic changes in the environment, time urgency,
limited assets, lack of information, complex situations, and unpredictable
situations. Therefore, it is very difficult to make the right decision in a
short time in an emergent incident situation. There is no way to directly
apply the traditional multi-agent emergency scheduling problem to the real
world. In the above context, we have made some simple modifications to the
traditional multi-agent emergency decision making and further defined a
scenario-specific problem of emergency decision-making for emergent
incident(EDEI) for the type of emergency incident rescue. The specific
amendments are as follows:

#### IV-1

Turn the original operation graph $\mathcal{G}$ into an operation graph
$\mathcal{G}$ and an incident spread graph $\mathcal{G}^{\prime}$. In real
life, the operation graph $\mathcal{G}$ is abstracted from the physical space.
While the incident spreads not according to the physically passable channel to
determine the edge, the incident may spread along the pipeline with the
circuit. Therefore, a connection can be made to obtain the incident spread
graph $\mathcal{G}^{\prime}$ based on the relationship of incident spread.
Each node has materials that need to be salvaged or primary assignment that
need to be completed, and the sequence of primary assignment is sequential
according to the needs of the assignment. The original map information is
known in advance, and the incident will spread over time after it happens.

#### IV-2

Incidents spread, we are targeting incidents that can spread, such as fires.
We consider here the example of fire, where the fire situation may follow the
$f(t)$ function and the spread may follow the $g(f(t))$ function.

#### IV-3

In this paper, an agent refers to a robot or an unmanned cart, which requires
a communication mechanism between them. Existing robots for fire rescue, such
as the fully automated firefighting robot system high-pressure water gun robot
and hose extension robot developed by Mitsubishi Heavy Industries in 2019
[30]. When a fire spot is detected, the water cannon robot can automatically
move into position at speeds of up to 7.2 km/h via GPS and laser sensors. The
problem is that many firefighting robots must carry a heavy water-filled hose
to fire, and the robots do their jobs independently of each other, with no
communication between them. The agent needs to dynamically assign tasks to
complete the emergency recovery tasks at the nodes before the incident
situation $f(t)$ reaches the critical value $m$, or before the primary
assignment deadline $et_{i}$.

###  IV-A Real World Scene Modeling

The typical incident in the emergency rescue is fire. For example, inside an
unmanned storage where a fire occurs, a realistic physical plan of the
warehouse can be abstracted into an incident operation graph
$\mathcal{G}=(V,E)$. Where, $V$ is the set of assignment nodes abstracted
according to different assignment areas in the warehouse, each node contains
the attribute $w_{i}$. When $w_{i}>0$ means that there is a asset placed at
that location, its value number is $w_{i}$, and $w_{i}=0$ means that there is
no asset at that location. $E$ is the set of connectivity relations between
assignment nodes, which are abstracted into connected edges in the graph
according to the specific assignment area. The properties $d_{i,j}$ of $E$
denote the distance of $v_{i}$ from $v_{j}$.

In $\mathcal{G}=(V,E)$, there are a sequence of primary assignments
$O=\\{o_{1}(v_{1},{et}_{1},w_{1}),...,o_{m}(v_{m},{et}_{m},w_{m})\\}$, where
$v_{i}$ represents the node of the primary assignment $o_{i}$, ${et}_{i}$
represents the deadline of the primary assignment, and $w_{i}$ is the number
of assets at $v_{i}$.

###  IV-B Incident Spread

Divide the real physical space into various regions to get the operation graph
$\mathcal{G}$. Determine the incident spreading relationship in the physical
space according to the circuit connection pipeline connection, so that the
incident spreading graph $\mathcal{G}^{\prime}=(V^{\prime},E^{\prime})$ can be
generated, as in Fig.1. where $|V^{\prime}|=|V|$, $V^{\prime}$ contains the
attribute $f(t)$, which indicates the incident condition of the situation of
the spatial node, specifically the severity of the incident.

Figure 1: Operation graph $\mathcal{G}$ and incident spread graph
$\mathcal{G}^{\prime}$

In this paper, we classify incidents into the following categories.

#### IV-B1

Conspicuous incidents. Conspicuous incidents refer to incidents with clear
incident characteristics, such as fires and shelf collapses.

#### IV-B2

Unobtrusive incidents. Unobtrusive incidents refer to incidents that do not
have clear incident characteristics, such as path congestion.

The calculation of the incident condition $f(t)$ for these two incidents has
been studied extensively and accordingly. As in [31] for fires $f(t)$ was
calculated using multiple low-altitude, short endurance (LASE) UAVs to explore
and monitor the forest fire propagation problem, where airborne cameras to
detect fires and image-based techniques were used to detect the extent of the
affected area. An unmanned aircraft system (UAS) for forest fire monitoring is
presented in [32], which is capable of calculating in real time the evolution
of fire front patterns and other potential parameters related to fire
propagation. In the paper, the area where the fire is located is divided into
a rectangular grid, where the state of each cell $k$ is defined by two binary
values:$F_{k,t}\in\\{0,1\\}$, indicating whether the cell has a fire;
$Q_{k,t}\in\\{0,1\\}$, indicating whether the fuel in the cell is completely
depleted. In [33] for the computation of $f(t)$ for shelf collapse, a
cooperative exploration coordination method is proposed to detect and track
multiple dynamic bounds of emergent incidents. In [5 2020] for the calculation
of $f(t)$ for path congestion, a deep neural network architecture for
unsupervised anomaly detection (UAV-AdNet) is proposed to use unmanned aerial
vehicles (UAV ) to monitor critical infrastructure (e.g., warehouses,
airports, and ports).

In this paper, when a emergent incident occurs at a node and the incident
condition $f(t)>\tau$ represents that the node is completely destroyed, the
node is added to the set of $V^{f}$,
$V^{f}\overset{+}{\leftarrow}\\{v_{i}|f_{i}(t)>\tau\\}$. Where $\tau$ is the
threshold value of the severity of the emergent incident $f(t)$.

Each edge $e_{i,j}^{{}^{\prime}}$ in $E^{{}^{\prime}}$ connects two nodes
$v_{i}^{{}^{\prime}}$, $v_{j}^{{}^{\prime}}$ and denotes the probability
$g_{j}(f_{i}(t))$ that the incident at $v_{i}^{{}^{\prime}}$ spreads to
$v_{j}^{{}^{\prime}}$ after an incident at $v_{i}^{{}^{\prime}}$. $|E|$ is not
directly related to $|E^{{}^{\prime}}|$. After an incident, if the node is
spread by the incident, the asset $w_{i}=0$ at this node. If the deadline is
exceeded, the primary assignment is considered failed.

After the incident spread graph is abstracted from the operation graph, the
corresponding incident spread matrix $G^{*}=[g_{i,j}]\in\mathbb{R}^{N\times
N}$ can be derived according to the incident condition $f_{i}(t)$ and incident
spread probability $g_{j}(f_{i}(t))$ of the incident node, as shown in
Equation 2. In $G^{*}$, the ranks and columns represent the nodes in the
incident spread graph. If there are $n$ nodes in the incident spread graph,
$G^{*}$ is an $n*n$ matrix. In $G^{*}$, the diagonal line indicates the
incident condition $f_{i}(t)$ of the current node $v_{i}$, and the other
corresponding positions indicate the probability of incident spread between
two nodes. For example, row $i$, column $j$, corresponding to the value
$g_{j}(f_{i}(t))$, indicates that the probability of the incident at node
$v_{i}$ spreading to node $v_{j}$ is $g_{j}(f_{i}(t))$, which is simplified in
the matrix as $g_{i,j}$. The current incident status of the node and the trend
of incident spread can be clearly seen in the matrix. The rows of the matrix
represent the probability of spreading of the current node to the surrounding
nodes, and the columns of the matrix represent the superimposed influence of
the surrounding nodes on the current node. The specific probability
superimposed impact calculation formula is shown in Equation 3. Update the
incident spread matrix at every time step. The parameters of the response such
as the objective function are updated according to the incident spread matrix.

$G^{*}=\begin{pmatrix}f_{1}&g_{1,2}&\cdots&g_{1,n}\\\
g_{2,1}&f_{2}&\cdots&g_{2,n}\\\ \vdots&\vdots&\ddots&\vdots\\\
g_{n,1}&g_{n,2}&\cdots&f_{n}\\\ \end{pmatrix}$ (2)

$P_{j}=1-\prod_{i=1}^{n}(1-g_{i,j})$ (3)

Example: There are 4 nodes in Figure 2, where nodes $v_{1}$ and $v_{2}$ have
incidents and the probability of spreading from nodes $v_{1}$,$v_{2}$ to nodes
$v_{3}$,$v_{4}$ is labeled in the figure, and the probability spreading matrix
is shown on the right.

Figure 2: Examples of probability superposition formula and incident spread
matrix

According to Figure 2, the probability of an incident at node $v_{3}$,$v_{4}$
is calculated by the probability superposition formula as follows.

$P_{3}=1-(1-g_{1,3})*(1-g_{2,3})=1-(1-0.7)*(1-0.7)=0.91$

$P_{4}=1-(1-g_{1,4})*(1-g_{2,4})=1-(1-0.8)*(1-0.9)=0.98$

In this problem, if the incidents do not spread $g_{j}(f_{i}(t))=0$, the
problem can be abstracted as how to make the total execution of the primary
assignment sequence and the new incoming assignment the most efficient if new
assignments are continuously added to the primary assignment sequence. If
there is no primary assignment sequence, the problem can be abstracted as an
ordinary rescue and relief problem, so that the damage caused by the incident
is minimized.

###  IV-C Agent

Defining a set of $n$ agents $A=\\{a_{1},...,a_{n}\\}$, in which each agent
can move and eliminate the incident nodes in $\mathcal{G}$ and
$\mathcal{G}^{{}^{\prime}}$. The agent starts to move at the initial position,
in case of incident $f_{i}(t)$ reaches the critical value $m$, or completes
the node assignment before the assignment deadline $et_{i}$. The emergency
recovery tasks performed by the agent at the node are specifically divided
into two categories: one is to perform a rescue to eliminate the incident so
that $f_{i}(t)=0$ or $f_{i}(t)<m$, and the other is to carry the assets at the
node to a safe place.

The problem presented in this paper is real-time, where the emergent incident
in the scenario is constantly spreading, so we assume that each agent can
communicate with all other agents in the team by broadcasting messages. At
time $t$, if an incident is detected at a node, the set of incident nodes
$V^{f}$ is updated and propagated through the communication network. The task
execution time of all the agents is inversely related to the number of agents,
and more agents complete the task faster. The time required to handle the
emergency recovery task of an incident node is $CT_{i}$, and the time taken by
n agents to handle the task of the same node is $t_{n}=\frac{CT_{i}}{n}$.

###  IV-D EDEI Problem

Given an incident operation graph $\mathcal{G}$, an incident spreading graph
G’, and an primary assignment sequence $O$, the agent makes emergency
decisions in a conflict-free situation such that the total gain is maximized
when the entire primary assignment $O$ is completed. In the above specific
setup, the global optimization objective is to maximize the sum of the gains
on all primary assignments, defined as follows.

$\varphi=\text{max}\sum(w_{i}(O)+w_{i}(G_{r}))$ (4)

Among them, $w_{i}(O)$ represents the total value of the completed primary
assignment, and $w_{i}(G_{r})$ represents the total value that is not
endangered in the graph $\mathcal{G}$.

##  V PROPOSED SOLUTIONS

The emergency decision algorithm used in this paper is a combination of GRU
and MADDPG, where each agent makes decisions independently and shares
information. The algorithm includes three parts: (1) dynamic perception of
environmental information; (2) Predict the nodes where emergent incidents may
occur in the next step; (3) The agent makes decision based on the predicted
information about the nodes where incidents are likely to occur.

###  V-A Dynamic Perception of Environmental Information

In the first stage of dynamic perception of environment information, agents
transfer three aspects of information: (1) each agent perceives the
information of changes in the surrounding environment, that is, whether there
is a new emergent incident in the surrounding nodes;(2) agents share their
decision information, that is, the location information of their next step;
(3) Predicted node information of possible emergent incidents in the next time
step.

After the agent enters the environment, it senses the incident information of
the surrounding nodes, updates $V^{f}$ and communicates $V^{f}$ information
with other agents. Through the prediction mechanism, the agent predicts the
nodes with possible emergent incidents and puts them into the set
$\mathcal{T}_{t+1}$. Agent communicates decision information of $t+1$ time
with other agents at $t$ time. The main steps of the algorithm are represented
by pseudo code in algorithm 1.

Algorithm 1 Dynamic Perception of Environmental Information Strategy S1.

0: Primary assignment sequence set $O$; incident nodes set $V^{f}$; incident
spread matrix $G^{*}$.

0: $V^{f}_{t+1}$,$\omega(t+1)$.

1: while $t<$ end time $T$ do

2: Agent perceives incident information $f_{i}(t)$of surrounding nodes.

3: Update $V^{f}$, $V^{f}\overset{+}{\leftarrow}\\{v_{i}|f_{i}(t)>0\\}$.

4: Get new predict incident node set $\mathcal{T}_{t+1}$ according to
algorithm2.

5: Communication $\mathcal{T}_{t+1}$ with other agents.

6: Communication assignment execution list $\omega_{t}$ with other agents.

7: if $V^{f}(t)\neq V^{f}(t-1)$ then

8: Communication $V^{f}$ information with other agents.

9: end if

10: if agent make new decision then

11: Communication agent’s own decision $u_{t+1}$with other agents.

12: end if

13: $t\leftarrow t+1$.

14: end while

###  V-B Prediction Mechanism

For the real emergent incident scene, the incident will have an impact on the
surrounding area, spread rapidly to the surrounding area, and spread
exponentially over time. This will cause the environment to change rapidly all
the time, and the speed of decision-making in the later stage may not keep up
with the speed of incident changes. If the agent directly explores the
environment for decision-making, the decision-making efficiency will be low
because of the large search space and the rapid change of the environment.
Therefore, the prediction mechanism is introduced, and the GRU is used to
predict the probability of possible incidents $g_{j}(f_{i}(t))$ at the next
time step node according to a large amount of common sense knowledge. Put the
nodes with $g_{j}(f_{i}(t))>0$ into the set $\mathcal{T}_{t+1}$,
$\mathcal{T}_{t+1}\overset{+}{\leftarrow}\\{v_{i}|g_{j}(f_{i}(t))>0\\}$, where
$\mathcal{T}_{t+1}$ represents the set of nodes predicted to be incident in
the next step, and share $\mathcal{T}_{t+1}$ information with other agents.

GRU can find the coupling relationship between anomalies [34]. When judging,
GRU can not only judge the nodes that may have incidents in the next step, but
also judge the compound impact on the surrounding nodes after the incident,
that is, the occurrence of other types of incidents due to one incident. For
example, a fire at a node may lead to congestion of surrounding roads. After
predicting the node set $\mathcal{T}_{t+1}$ with exception in the next step,
the agent selects the node with the largest $g_{i,j}$ from the
$\mathcal{T}_{t+1}$ set to perform the emergency recovery task. Moreover, the
time of emergent incident emergency recovery task is urgent, so it is
unrealistic to execute the emergency recovery task in serial mode. Multi-agent
executes the emergency recovery task at the same time, and establishes a
comprehensive list $V^{f}$ of all discovered incident.

####  V-B1 Predictive Features

In this paper, we predict incidents based on the following features. The node
that may have an incident in the next step is related to the incident feature
$F_{f(t)}$ in the current environment, the asset feature $F_{r}$ at the node,
and the node vulnerability feature $F_{\xi i}$.

  * •

incident feature $F_{f(t)}$. Node incident status is one of the important
factors affecting the incidents of other nodes. It consists of the incident
state $f_{i}(t)$ of the node where the current incident occurs.

  * •

Asset feature $F_{r}$. $F_{r}(f_{c},w_{i})$ consists of two features, the
asset category $f_{c}$ of the node and the asset quantity $w_{i}$ of the node.
The type and quantity of assets at the node reflect the vulnerability of the
node, so it plays an important role in inferring whether an incident occurs at
the node in the next step. Some types of assets may even have a direct causal
relationship with whether an incident occurs. For example, if a node stores
gasoline, the node is very vulnerable to accidents.

  * •

Node vulnerability feature $F_{\xi i}$. It reflects the degree of
vulnerability of nodes without incidents. $F_{\xi i}$ is related to the
current node asset quantity $w_{i}$, the incident status $f_{i}(t)$ of the
incident node and the distance $d_{i,j}$ between the current node and the
incident node. We calculate the feature according to Equation 5.

$F_{\xi i}=w_{i}\cdot\sum_{j=1}^{n}(f_{j}(t)\cdot\frac{1}{d_{i,j}})$ (5)

Where, $n$ represents the number of incident nodes, $w_{i}$ is the number of
assets of the current node, $f_{j}(t)$ is the incident status of $v_{j}$ node,
and $d_{i,j}$ represents the distance between $v_{i}$ and $v_{j}$.

####  V-B2 GRU Prediction Mechanism

The input of GRU is constructed as an $n*4$ matrix, where $n$ is the number of
nodes. Set each row as a triple, and the elements are composed of
$(F_{f(t)},F_{r},F_{\xi i})$. Output the triplet predicted in the next step
$(F_{f(t)}^{{}^{\prime}},F_{r}^{{}^{\prime}},F_{\xi i}^{{}^{\prime}})$. The
output triplet is mapped into an n*1-dimensional vector through a convolution
neural network(CNN) with a convolution kernel of 1*3, which represents the
incident probability $g_{i,j}$ of the next time node. The specific prediction
framework is shown in Figure 3.

Figure 3: Prediction model framework

Put the nodes with possible incidents predicted by GRU into a set
$\mathcal{T}_{t+1}$,
$\mathcal{T}_{t+1}=\\{<v_{1},g_{1}>,<v_{2},g_{2}>,...,<v_{i},g_{i}>\\}$.
Agents share the incident prediction set $\mathcal{T}_{t+1}$. The main steps
of GRU prediction mechanism are represented in algorithm 2.

Algorithm 2 Prediction mechanism Strategy S2.

0: incident feature$F_{f(t)}$, asset feature $F_{r}$, node vulnerability
feature $F_{\xi i}$.

0: $\mathcal{T}_{t+1}$.

1: while $t<$ end time $T$ do

2: $GRU(F_{f(t)},F_{r},F_{\xi i})\rightarrow\mathcal{T}_{t+k}$.

3: $\mathcal{T}_{t}\leftarrow\mathcal{T}_{t+1}$.

4: $t\leftarrow t+1$.

5: end while

###  V-C MDP Problem Construction

We transform the EDEI problems into a partially observable Markov game for $N$
agents $A=\\{a_{1},...,a_{n}\\}$. Define the Markov game for N agents as a set
of states $S$, a set of observations $x=\\{x_{1},...,x_{n}\\}$, and a set of
actions $U=\\{u_{1},...,u_{n}\\}$. The state set $S$ describes the incident
node set $V^{f}$, the primary assignment set $O$, the primary assignment
completion $\omega(t)$, the location information of each agent $p_{i}(t)$, and
the target location information of each agent $g_{i}(t)$. $X$ are observation
spaces for the agent, and the observation of each agent at step t is a part of
the current state, $s(t)\in S$. $U$ are action spaces for agent. For each
given state $s\in S$, the agent use the policies, $\pi:S\rightarrow U_{m}$, to
choose an action from their action spaces according to their observations
corresponding to $s$.

####  V-C1 Environment State

The state at $t$ is represented by $s(t)$, which consists of the tuple
${I(t),A(t)}$, where $I(t)$ is the set of primary assignments waiting for the
service, ordered according to the relative time of incident occurrence. $I(t)$
is a set $\\{O,\omega(t),V^{f}_{t}\\}$, where $O$ denotes the primary
assignment sequence, $\omega(t)$ denotes the completion of $O$ at moment $t$,
and $V^{f}_{t}$ denotes the set of nodes where an incident occurs at moment
$t$. $A(t)$ corresponds to the information of the agent set at step $t$,
$|A(t)|=|A|$. Each element $a_{i}(t)\in A(t)$ is a set of
$\\{p^{i}(t),g^{i}(t)\\}$, where $p^{i}(t)$ is the current node position of
agent $a_{i}$ and $g^{i}(t)$ is the target node position moved by agent
$a_{i}$. Then, according to the EDEI problems the environment state at step
$t$, $s(t)\in S$, can be given by

$\displaystyle S(t)=\\{I(t),A(t)\\}$ (6a) $\displaystyle
I(t)=\\{O,\omega(t),V^{f}_{t}\\}$ (6b) $\displaystyle|A(t)|=|A|,a_{i}(t)\in
A(t),a_{i}(t)={p^{i}(t),g^{i}(t)}$ (6c)

####  V-C2 Action

In our problem, the action corresponds to directing the agent to a valid node,
either to process the incident or to salvage assets from the primary
assignment node. Valid nodes include nodes that have unprocessed primary
assignments or nodes where incidents occur. The action at moment $t$ is
represented by $u(t)$, which consists of the tuple $\\{U_{O}(t),U_{I}(t)\\}$,
where $U_{O}(t)$ denotes the processing of the primary assignment and
$U_{I}(t)$ the processing of the incident node. According to the current
observation, each agent chooses an action from its own action space. agent’s
behavior at step $t$, $u(t)\in U$, can be described as follow.

$U(t)=\\{U_{O}(t),U_{I}(t)\\}$ (7)

####  V-C3 Reward

As the reward leads each agent to its optimal policy and the policy directly
determines the node where the emergency recovery tasks is to be performed next
time, the reward function should be designed based on the objectives of the
original formulated problems, (1) Minimize the impact of incident spread (2)
Maximize the completion of the primary assignment.

In this paper, the objective function of each agent can be divided into three
parts $W_{succ}$, $W_{is}$ and $W_{r}$, Where $W_{succ}$ represents the
completion of the primary assignment, $W_{is}$ represents the impact of
incident spread, and $W_{r}$ represents the remaining nodes in the graph
$\mathcal{G}$. The objective function is as follows

$r_{i,t}=\sum_{t=1}^{T}(W_{succ}+W_{is}+W_{r})$ (8)

  * •

The first part shows the completion of the primary assignment, specifically
refers to the rescue node assets, and is set as $W_{succ}$. The completion
status of initial task $O$ at each step is $\omega(t)$. Let the set of task
nodes to be completed be $\Lambda=|O|-|\omega(t)|$. When $\Lambda>0$, $t=T$,
there are primary assignments that have not been completed. When $\Lambda=0$
and $t\leqslant T$, it means that all primary assignments are completed.
Specifically, it can be divided into pending tasks $\Lambda$ and completed
tasks $\omega(t)$, $\Lambda\cup\omega(t)=O$. The task to be completed
$\Lambda$ is divided into general task node (i.e. no incident node) and
predicted incident node $\mathcal{T}_{t+1}$. The predicted incident node can
be divided into the incident but not completely damaged node and the predicted
incident node. $W_{succ}$ is calculated as follows

$W_{succ}=\begin{cases}g_{i}\cdot f(t)\cdot
w|_{v_{i}\in\mathcal{T}_{t+1}\bigcap\Lambda\bigcap
V_{f}||0<f(t)<\tau||g_{i}=1},(1)\\\ P_{i}\cdot
w|_{v_{i}\in\mathcal{T}_{t+1}\bigcap\Lambda||f(t)=0||g_{i}>0},(2)\\\
\frac{1}{et_{i}}\cdot w|_{v_{i}\in\Lambda||f_{i}(t)=0||g_{i}=0},(3)\\\
w|_{v_{i}\in\omega_{t}},(4)\\\ \end{cases}$ (9)

Where (1) represents reward for predicting incidents in incidents but not
completely destroyed, (2) represents reward for predicting incidents in tasks
to be completed, (3) represents reward for primary assignments to be completed
without incident, (4) represents completed primary assignment reward.

  * •

The second part represents the impact of incident spread and is set as
$W_{is}$. At each step $t$, there will be incident prediction set
$\mathcal{T}_{t+1}$ and incident set $V^{f}$. Let
$\Gamma=|\mathcal{T}_{t}|-|V_{f(t)}|$, when $\Gamma>0$, it is proved that
there is no incident at the node where the incident originally occurred, that
is, the decision made has an effect on the spread of the incident.

There are three factors affecting the spread of incidents: the primary
assignment deadline $et_{i}$, the node incident status $f_{i}(t)$, and the
distance $d_{j}^{i}$ from the agent $a_{i}$ to the task node $v_{j}$.
According to the above three indicators, the assignments in the primary
assignment sequence $O$ are sorted according to task emergencies to generate a
new task set $O_{et_{i}}$, $O_{f(t)}$, $O_{d_{j}^{i}}$. Note that the more
urgent the tasks are, the higher the ranking. Then, the three sets are grouped
in pairs as Cartesian products, which are represented by relation matrix. The
node corresponding to the lower left corner of the matrix is the most urgent
position, expressed in (0, 0) coordinates. According to the distance
$d_{(0,0)}$ between each task node and (0,0) in the figure, the most serious
task node can be calculated, so that the node to perform the task in the next
step can be selected. If the distance between any node and (0, 0) is equal, go
to the asset quantity $Max\\{v_{i}|_{w_{i}}\\}$ node at the node. If the asset
quantity is equal, randomly select a node for rescue. After selecting the most
serious node in the three Cartesian products, the accident spread degree is
calculated by equation 10.

$W_{is}=\left\\{\begin{matrix}\frac{1}{et_{i}}\cdot f(t)\cdot
w_{i}^{O_{et_{i}}\times O_{f(t)}}\\\
\frac{1}{et_{i}}\cdot\frac{1}{d_{j}^{i}}\cdot w_{i}^{O_{et_{i}}\times
O_{d_{i}}^{v_{j}}}\\\ f(t)\cdot\frac{1}{d_{j}^{i}}\cdot
w_{i}^{O_{d_{i}}^{v_{j}}\times O_{f(t)}}\end{matrix}\right.$ (10)

For example, if the primary assignment set $O=\\{o_{1},o_{2},o_{3},o_{4}\\}$,
it is $O_{et_{i}}=\\{o_{1},o_{2},o_{3},o_{4}\\}$ from small to large according
to the assignment deadline, $O_{f(t)}=\\{o_{2},o_{4},o_{3},o_{1}\\}$ according
to the incident status $f(t)$, and
$O_{d_{j}^{i}}=\\{o_{3},o_{1},o_{2},o_{4}\\}$ from small to large according to
the distance $d_{j}^{i}$ between agent $a_{i}$ and each task node. After that,
the three sets do Cartesian product in pairs to generate the relationship
matrix $O_{et_{i}}\times O_{f(t)}$, $O_{et_{i}}\times O_{d_{j}^{i}}$,
$O_{d_{j}^{i}}\times O_{f(t)}$, as shown in Fig.4. The asterisk in the figure
indicates the most urgent position. It can be seen from the calculation in (a)
that node $v_{2}$ is the most urgent, (b) node $v_{1}$ is the most urgent, (c)
the distance between $v_{2}$ and $v_{3}$ from node (0, 0) is the same. By
comparing the size of $v_{2}$ node asset $w_{2}$ and $v_{3}$ node asset
$w_{3}$, select the node with larger $w$ for rescue. If the number of assets
is equal, select a node at random.

Figure 4: Anomalous spread affects Cartesian product (!!change describe).

  * •

The third part shows the remaining nodes in the graph $\mathcal{G}$, set as
$W_{r}$. Specifically, it refers to the node assets that are not damaged in
the graph $\mathcal{G}$. Let $V^{r}$ represent the set of remaining nodes in
the graph $G$ that are not
destroyed,$V^{r}\overset{-}{\leftarrow}\\{v_{i}|_{et_{i}=0||v_{i}\in
V_{f}}\\}$. The specific calculation formula of $W_{r}$ is as follows:

$W_{r}=w|_{v_{i}\in V^{r}}$ (11)

###  V-D P-MADDPG

####  V-D1 MADDPG

To solve the above Markov game for N agents, For each agent,indicated by
equations 6 and 7, the observation are continuous and action spaces are
discrete. Moreover, to avoid spectrum and time cost on wireless communications
among different agents, because it may not be able to provide a reliable and
good communication channel in the extreme environment of emergent incident, in
order to ensure reliability and security, the algorithm needs to be able to
complete the task well without reliable channel.

MADDPG Framework: The MADDPG framework is composed of the N agents. MADDPG
adopts centralized training and decentralized execution(CTDE). Next, we take
an example to explain how to centrally train the MADDPG model and execute the
learned model in a decentralized way.

In the centralized offline training stage, in addition to the local
observation, extra information, i.e. predicted anomaly information
$\mathcal{T}_{t}$. Namely, at step t, $\mathcal{T}_{t}$ is saved into the
agent’s replay buffer with $\\{V^{f}_{t},O,\omega(t),p^{i}(t),g^{i}(t)\\}$
together. For the i-th transition of the
agent,$(s_{i},u_{i},r_{i},s\textquoteright)$,we have
$s_{i}=\\{\mathcal{T}_{t},V^{f}_{t},O,\omega(t),p^{i}(t),g^{i}(t)\\}$ as shown
in Fig.5. When updating the parameters of the actor and the critic according
to the inputted mini-batch of transitions, the actor chooses an action
according to the local observation $x_{i}$, i.e. , $u_{i}=\mu(x_{i})$, and the
chosen action and $s_{i}$ then are valued by the critic. Moreover, with the
extra information, each agent allows to learn its state-action value function
separately. Also, as aware of all other agents’ actions, the environment is
stationary to each agent during the offline training stage. Thus, the biggest
concern to other multi-agent RL algorithms, the dynamic environment caused by
other agents’ actions, is addressed here. During the execution stage, as only
local observation is required by the actor, each agent can obtain its action
without aware of other agents’ information.

Considering the common objective of the formulated optimization problems, the
agents should maximize the completion of initial tasks and minimize the impact
of anomaly spread. To achieve a cooperative Markov game, we assume the same
immediate reward $r_{i}$ is returned to each agent. The $r_{i}$ of each time
step $t$ is calculated as follows

$r_{i,t}=\sum_{t=1}^{T}(W_{succ}+W_{ia}+W_{r})$ (12)

####  V-D2 P-MADDPG

When considering the decision, the node set $\mathcal{T}_{t}$ predicted by GRU
that may have an incident in the next time step is taken as a part of
information transmission, that is, as the information observed by agent, so as
to make the decision more effective and timely. Fig.5 shows the structural
framework of P-MADDPG. According to the above discussion and Fig.5, the
proposed P-MADDPG management scheme can be summarized in Algorithm 3. In
algorithm 3, first, we perform a series of initialization (line1-4),
initialize the primary assignment $O$, assignment completion $\omega(t)$,
predict the incident node set $\mathcal{T}_{t}$ and incident node set $V^{f}$.
Then initialize a state and random action in (line6-7). (line1-4)
initialization information is shared between (Line8) agents. In (line10-12),
execute the action to update the objective function, update the observation
value, store the $(s,u,R,s^{\prime},V^{f},\omega(t),\mathcal{T}_{t})$
information in the experience pool $D$, and (line13) predict the node set
$\mathcal{T}_{t+1}$ that may have an incident in the next step. (line14-16)
update $\omega(t)$, $\mathcal{T}_{t}$, $V_{f}$, $\Gamma$ information, update
status $s$. (line17-27) update actor network and critical network, and update
target network parameters. The specific steps are shown in algorithm 3.

Figure 5: Structural framework of P-MADDPG. Algorithm 3 P-MADDPG Strategy S3.

0: Incident nodes set $V^{f}$;prediction incident node set
$\mathcal{T}_{t}$;assignment execution$\omega_{t}$.

0: Policy of agent $a_{i}$ $u_{t+1}$.

1: Initialize primary assignment set $O$.

2: Initialize assignment execution$\omega_{t}$.

3: Initialize prediction incident node set $\mathcal{T}_{t}$.

4: Initialize incident node set $V^{f}$.

5: for episode=1 to $M$ do

6: initialize locations of agents and initial incident nodes.

7: receive initialize state $s$.

8: receive $O$,$\omega_{t}$,$\mathcal{T}_{t}$,$V^{f}$ from algorithm 1,2.

9: agents communicate the information of
$O$,$\omega_{t}$,$\mathcal{T}_{t}$,$V^{f}$.

10: for $t$=1 to max-episode-length do

11: For each agent $a_{i}$,select action
$u_{i}=\mu_{\theta_{i}}(s_{i},\mathcal{T}_{i})+\mathcal{N}_{t}$.

12: Execute action $u=(u_{1},...,u_{n})$ and new state $s\textquoteright$.

13: Get new reward $r_{i}$ from Eq. 8.

14: Get new $\mathcal{T}$ from algorithm 2.

15: Get new $\omega_{t}$,$V^{f}$ from algorithm 1.

16: Store $(s,u,r,s^{{}^{\prime}},\omega_{t},V^{f},\mathcal{T}_{t})$ in replay
buffer $D$.

17: $\mathcal{T}\leftarrow\mathcal{T}^{{}^{\prime}}$.

18: $\omega_{t}\leftarrow\omega_{t}^{{}^{\prime}}$.

19: $V_{f}\leftarrow V_{f}^{{}^{\prime}}$.

20: $s\leftarrow s^{{}^{\prime}}$.

21: for agent $i=1$ to $N$ do

22: Sample a random minibatch of $S$ samples
$(s^{j},u^{j},r^{j},s^{{}^{\prime}j},\omega_{t}^{j},V^{fj},\mathcal{T}^{j})$
from $D$.

23: Set
$y^{j}=r_{i}^{j}+\gamma\bar{Q}_{i}^{\mu^{{}^{\prime}}}(s^{{}^{\prime}j},\omega_{t}^{j},V^{fj},\mathcal{T}^{j},u_{1}^{{}^{\prime}},...,u_{n}^{{}^{\prime}})|_{u_{k}^{{}^{\prime}}=\mu_{k}^{{}^{\prime}}(s_{k}^{j})}$.

24: update critic by minimizing the loss
$\mathcal{L}(\theta_{i})=\frac{1}{S}\sum(y^{j}-Q_{i}^{\mu}(s^{j},\omega_{t}^{j},V^{fj},\mathcal{T}^{j},u_{1}^{j},...,u_{N}^{j}))^{2}$.

25: Update actor using the sampled policy
gradient:$\triangledown_{\theta_{i}}J\approx\frac{1}{S}\sum\triangledown_{\theta_{i}}\mu_{i}(s_{i}^{j})\triangledown_{u_{i}}Q_{i}^{\mu}(s^{j},\omega_{t}^{j},V^{fj},\mathcal{T}^{j},u_{1}^{j},...,u_{N}^{j})|_{u_{i}=\mu_{i}(s_{i}^{j})}$.

26: end for

27: Update target network parameters for each agent $i$:
$\theta_{i}^{{}^{\prime}}\leftarrow\tau\theta_{i}+(1-\tau)\theta_{i}^{{}^{\prime}}$.

28: end for

29: end for

In P-MADDPG, $D$ represents the experience replay buffer, and the elements are
composed of $(s,s^{\prime},u_{1},...,u_{n},r_{1},...,r_{n})$, recording all
agent experiences. In order to apply to EDEI problem, the empirical playback
is improved, in which the observed value
$s(t)=\\{V^{f}(t),\mathcal{T}_{t},O,\omega(t),p^{i}(t),g^{i}(t)\\}$.

##  VI Experimental Results and Discussion

This section introduces the experiments and results that test the performance
of the proposed EDEI problem model and P-MADDPG algorithm. In this paper, we
select three experimental scenarios: unmanned storage, factory assembly line,
civil aviation airport. In order to further illustrate the effectiveness of
P-MADDPG, it is compared with greedy algorithm and MADDPG. Greedy algorithm is
easy to implement and very efficient in most cases. MADDPG is the basic method
of P-MADDPG. Therefore, the proposed EDEI problem is evaluated by comparing
the proposed P-MADDPG with greedy algorithm and MADDPG.

In this paper, we use the multi-agent system of unmanned vehicle to simulate
the experiment, and conduct nine groups of experiments. See TableI for the
specific experimental settings. In the three experimental scenarios, different
test conditions can be created by varying the number of tasks and agents. In
case of incident, the emergency recovery tasks to be completed by the agent
include quickly rescuing the materials at the node, eliminating the incident
and carrying out rescue work. Finally, complete the primary assignment
sequence as much as possible and minimize the incident impact. The emergency
recovery task is considered successful if the problem is solved, that is, the
primary assignment is completed on time and on schedule and the incidents in
the environment are eliminated. The algorithm is implemented in Python and
tested on a machine with Intel<EMAIL_ADDRESS>CPU and 16GB RAM. All
algorithms using in this paper are coded or recoded in Python under Windows 10
operating system. All the simulations are run on a single laptop with an Intel
Core i7 CPU 3.40GHz and 8GB of RAM.

TABLE I: Experimental configuration Test scenario | Number of agents | Number of incidents
---|---|---
unmanned storage | 2 | 4
3 | 6
4 | 8
factory assembly line | 2 | 4
3 | 6
4 | 8
civil aviation airport | 2 | 4
3 | 6
4 | 8

###  VI-A A.Unmanned Storage

The experimental background selected for the first experiment is the unmanned
storage, as shown in Fig.6. In the unmanned storage scenario, due to the large
storage area, large number of stacking, imperfect fire safety facilities and
other factors, it is easy to have safety incidents such as fire. If the
materials cannot be transferred in time, it will cause huge economic losses
and affect the normal operation process such as ex warehouse and warehousing.
The working area of unmanned storage is divided into warehouse in area,
warehouse out area, storage area and guarantee area. During normal operation,
the AGV transports the cargo freights from the warehouse in area to the
storage area for storage, and then transports the cargo freights from the
storage area to the warehouse out area for outbound. The guarantee area is
used for some support work such as charging and maintenance of AGVs.

When no exception emergent incidents, the AGV executes its own primary
assignment sequence, that is, warehousing, storage and outbound tasks. After
the incident, the AGV needs to carry out cargo freight and material rescue
tasks. This is very consistent with the EDEI problem proposed in this paper,
so the unmanned storage environment is selected for the experiment.

Figure 6: Unmanned Storage

In the experimental scenario,the following experimental settings were adopted.

#### VI-A1

The environment is defined as a 24 × 36 grid area, which represents the plan
of unmanned storage. The experimental scenario is shown in Fig.6. We limit the
task to 700 steps to emphasize the need for time.

#### VI-A2

The cargo and assignment area are abstracted into nodes. There are 224 nodes
in the scene, and the nodes are evenly distributed in the whole area.

#### VI-A3

The number of resources at the node is divided according to different
assignment areas. The inbound node resource $w_{i}=100$, the outbound node
resource $w_{i}=100$, the support area node resource $w_{i}=100$, and the
storage area node resource $w_{i}=200$.

#### VI-A4

Agents performs the primary assignment sequence $O$. The primary assignment
sequence $O$ includes warehousing and sorting assignment, storage area
placement task and issue buffer assignment.

#### VI-A5

Eight nodes are randomly selected in the scene as the initial incident nodes,
that is, the red nodes in the Fig.6.

#### VI-A6

Using a team of four agents, they are in an environment with 85 primary
assignment nodes. The emergency recovery tasks to be performed shall be judged
according to the node incident. As in real-world scene, agents do not know the
location and number of incident nodes in the environment, and achieve the
conditions of local observability.

#### VI-A7

Each time,agents can move one cell.

#### VI-A8

At the beginning of the assignment, the starting position of the agent
executing the rescue task is the warehousing area (green agent in Fig.6).
There is preliminary information about the location of the primary assignment.

We use greedy algorithm, MADDPG and P-MADDPG to experiment. We evaluate the
algorithms in terms of task completion rate $rate_{s}$, incident damage rate
$rate_{f}$ and reward $r$. The specific calculation formula of evaluation
index is as follows:

$rate_{s}=\frac{(n_{a_{1}}+...+n_{a_{i}})}{n_{O}*(n+k)}$

Where, $n_{a_{i}}$ represents the number of primary assignments completed by
the agent, $n_{O}$ represents the number of assignments in the primary
assignment $O$, $n$ represents the number of initial agents completing the
initial task $O$, $k$ represents the number of agents that change tasks after
the node has an incident.

$rate_{f}=\frac{n_{V_{f}}}{n_{V}*t_{max}}$

Where, $n_{V_{f}}$ represents the number of nodes with incidents, $n_{V}$
represents the number of all nodes in the environment, and $t_{max}$
represents the number of steps of each episode.

For the experiment with 4 agents, the multi-agent adopts greedy algorithm,
MADDPG and P-MADDPG respectively. After 25000 episode training, the
experimental results are shown in Fig.7.

Figure 7: Comparison results of three indexes of greedy algorithm, MADDPG and
P-MADDPG in unmanned storage environment. (a)task completion rate $rate_{s}$.
(b)incident damage rate $rate_{f}$. (c)reward $r$.

In Fig.7(a), among all algorithms, the task completion rate $rate_{s}$ of
P-MADDPG is the highest, up to $90\%$, the task completion rate $rate_{s}$ of
MADDPG is the second, up to $82.9\%$, and the task completion rate $rate_{s}$
of greedy algorithm is too volatile, basically maintained at about $60\%$. In
Fig.7(b), the incident damage rate $rate_{f}$ of P-MADDPG starts to converge
at 1000 episode, and the lowest incident damage rate can reach $11\%$ after
convergence, and among the three algorithms, the incident damage rate
$rate_{f}$ is the lowest. MADDPG followed, with a minimum of $23.5\%$. The
incident damage rate $rate_{f}$ of greedy algorithm is the worst, with a
minimum of $60\%$ and a maximum of $95.1\%$. The objective function is a
unique indicator of reinforcement learning, so only the reward $r$ of P-MADDPG
and MADDPG are compared in Fig.7(c). In Fig.7(c), compared with MADDPG, the
reward of P-MADDPG will converge faster and be greatly improved after adding
the prediction mechanism.

Note that the settings above were arbitrarily chosen and are not necessary for
the algorithm to work; many other settings are possible. Different test
conditions can be created by changing the number of initial incident nodes and
agents. We designed 9 different test questions to verify the algorithm, 3
experimental configurations and 3 algorithms. Fig.8 shows the comparison
results among degree, MADDPG and P-MADDPG under three experimental
configurations (2 agents, 4 incidents, 3 agents, 6 incidents, 4 agents and 8
incidents). Fig.8(a) shows the average task completion rate $\bar{rate_{s}}$
under three configurations. Fig.8(b) shows the average incident damage
$\bar{rate_{f}}$ rate under three configurations. Fig.8(c) shows the average
reward $\bar{r}$ for the three configurations.

Figure 8: Comparison results of P-MADDPG and other two methods in three
different scenarios. (a)average task completion rate $\bar{rate_{s}}$.
(b)average incident damage $\bar{rate_{f}}$. (c)average reward $\bar{r}$.

As shown in Fig.8, each bar chart shows the average task completion rate
$\bar{rate_{s}}$, average incident damage $\bar{rate_{f}}$ and average reward
$\bar{r}$ when episode = 10000. For P-MADDPG, the task completion rate average
task completion rate $\bar{rate_{s}}$, average incident damage
$\bar{rate_{f}}$ and average reward $\bar{r}$ of 9 groups of experiments are
the best results.

TableVI-A8 summarizes the average of each index of all algorithms under the
three configurations, and the best results of each index are highlighted in
bold.

Algorithm |  | Number of
---
agents and
incidents
$\bar{rate_{s}}$ | $\bar{rate_{f}}$ | $\bar{r}$
Greedy | 2,4 | 76.9% | 61.0% | \
3,6 | 76.5% | 77.9% | \
4,8 | 59.5% | 79.9% | \
MADDPG | 2,4 | 79.0% | 43.8% | -745.0
3,6 | 76.1% | 46.1% | -1247.5
4,8 | 80.6% | 29.8% | -1131.9
P-MADDPG | 2,4 | 88.4% | 36.8% | -197.9
3,6 | 87.7% | 41.3% | -295.4
4,8 | 88.6% | 18.2% | -341.3

As can be seen from tableVI-A8, among all algorithms and experimental
settings, for task completion rate $\bar{rate_{s}}$, when agent is 4 and
incident is 8, task completion rate $\bar{rate_{s}}$ of P-MADDPG is the
highest, reaching $88.6\%$. For the incident damage rate $\bar{rate_{f}}$,
when the agent is 4 and the incident is 8, the incident damage rate
$\bar{rate_{f}}$ of P-MADDPG is the lowest, reaching $18.2\%$. For reward $r$,
when the agent is 2 and the incident is 4, the reward $r$ of P-MADDPG is the
highest, reaching $-197.9$. In conclusion, our method offers best results
including the average completion rate $\bar{rate_{s}}$ is the largest, the
average incident damage rate $\bar{rate_{f}}$ is the smallest, and the average
reward $\bar{r}$ is the largest.

The three indexes of MADDPG are better than greedy algorithm, but worse than
P-MADDPG algorithm. When the greedy algorithm faces a complex dynamic
environment and there are many incident nodes, the average anomaly damage rate
$\bar{rate_{f}}$ of the greedy algorithm is the highest among the three
algorithms, up to $76.9\%$, which does not solve the sudden incidents well.
Compared with the other three algorithms, the average completion rate
$\bar{rate_{s}}$ of greedy algorithm is lower, and the lowest is $59.5\%$.

###  VI-B B.Factory Assembly Line

The experimental background selected for the second experiment is the factory
assembly line, as shown in Fig.9. In the factory assembly line scenario,
because in the actual production process, the flow shop is often affected by
factors such as delayed arrival of raw materials, urgent parts insertion,
change of delivery date, machine failure, parts scrapping and so on, which is
easy to lead to incidents. If the incident cannot be handled in time, the
pipeline will stop running and cause huge losses. When there is no incident,
the pipeline completes the primary assignment sequence according to the
primary assignment flow. After an incident, the pipeline will reduce the task
completion efficiency, and in serious cases, it will stop working.

Figure 9: Factory Assembly Line

Compared with experiment 1, the experimental settings are modified as follows.
The environment is defined as a 25×32 grid area. Step is limited to 500. There
are 40 nodes in the scene. Using a team of three agents. Six nodes are
randomly selected in the scene as theinitial anomaly nodes. The experimental
scenario is shown in Fig9.

We use greedy algorithm, MADDPG and P-MADDPG to experiment. We evaluate the
algorithms in terms of though put rate $TP$, incident damage rate $rate_{f}$
and reward $r$. The specific calculation formula of Though put rate $TP$ is as
follows:

$TP=\frac{n_{a_{1}}+...+n_{a_{i}}}{(n_{O}+k-1)*\bigtriangleup t}$

Where, $n_{a_{i}}$ represents the number of primary assignments completed by
the agent, $n_{O}$ represents the number of tasks in the primary assignment
$O$, $k$ is the number of flow line segments, $\bigtriangleup t$ is the time
required to perform an initial task.

For the experiment with 3 agents, the multi-agent adopts greedy algorithm,
MADDPG and P-MADDPG respectively. After 25000 episode training, the
experimental results are shown in Fig.10.

Figure 10: Comparison results of three indexes of greedy algorithm, MADDPG and
P-MADDPG in factory assembly line. (a)though put rate $TP$. (b)incident damage
rate $rate_{f}$. (c)reward $r$.

In Fig.10(a), the through put rate $TP$ of P-MADDPG is the highest. It starts
to converge at 5000 episode, and the maximum throughput can reach $38\%$ after
convergence. The through put rate $TP$ of MADDPG and greedy algorithm is too
volatile. MADDPG is better than greedy algorithm, maintaining at
$23\%$-$28\%$. The worst performance of greedy algorithm is maintained at
$18\%$-$27\%$. In Fig.10(b), the incident damage rate $rate_{f}$ of P-MADDPG
starts to converge at 7500 episode. After convergence, the incident damage
rate can reach as low as $8\%$, and among the three algorithms, the incident
damage rate $rate_{f}$is the lowest. The incident damage rate of MADDPG and
greedy algorithm is too volatile, which is basically maintained between
$35\%$-$50\%$. In Fig.10(c), compared with MADDPG, the reward of P-MADDPG will
converge faster and be greatly improved after adding the prediction mechanism.

Nine different test problems are designed to verify the performance of the
algorithm in the factory assembly line. The experimental results are shown in
Fig.11. Fig.11 shows the comparison results among degree, MADDPG and P-MADDPG
under three experimental configurations. Fig.11(a) shows the average
throughput $\bar{TP}$. Fig.11(b) shows the average incident damage rate
$\bar{rate_{f}}$. Fig.11(c) shows the average reward $\bar{r}$.

Figure 11: Comparison results of P-MADDPG and other two methods in three
different scenarios. (a)average though put rate $\bar{TP}$. (b)average
incident damage $\bar{rate_{f}}$. (c)average reward $\bar{r}$.

TableII summarizes the average of each index of all algorithms under the three
configurations, and the best results of each index are highlighted in bold.

TABLE II: Experimental configuration Algorithm |  | Number of
---
agents and
incidents
$\bar{TP}$ | $\bar{rate_{f}}$ | $\bar{r}$
Greedy | 2,4 | 20.7% | 61.7% | \
3,6 | 19.6% | 48.0% | \
4,8 | 19.9% | 67.4% | \
MADDPG | 2,4 | 22.1% | 58.9% | -9718.2
3,6 | 24.5% | 45.3% | -8956.8
4,8 | 22.1% | 55.8% | -13593.1
P-MADDPG | 2,4 | 23.7% | 56.4% | -2240.6
3,6 | 28.3% | 31.2% | -4182.3
4,8 | 23.3% | 52.6% | -4720.9

As can be seen from Fig.11, for P-MADDPG, the average throughput $\bar{TP}$,
the average incident damage rate $\bar{rate_{f}}$ and the average reward
$\bar{r}$ of the three groups of experiments are the best results. As can be
seen from tableII, among all algorithms and experimental settings, our method
P-MADDPG provides the best results for the four experimental indicators. The
maximum $\bar{TP}$ is $28.3\%$ (3 agents, 6 anomalies), the minimum
$\bar{rate_{f}}$ is $31.2\%$ (3 agents, 6 anomalies), and the maximum
$\bar{r}$ is $-2240.6$ (2 agents, 4 anomalies).

###  VI-C Civil Aviation Airport

The experimental background selected for the third experiment is the baggage
transportation of unmanned logistics trailers in civil airports based on Hong
Kong International Airport and Hunan Airport, as shown in Fig.12. The
transportation and transfer of baggage and goods is an important operation
content of airport logistics. Efficient logistics operation ensures the smooth
and orderly connection of flights and helps to improve passenger satisfaction.
In recent years, with the development of autopilot technology, landing
applications have been welcomed in many scenarios, and airport logistics is
one of the typical scenarios. In the actual transportation process, the
unmanned logistics trailer is vulnerable to major activities, emergencies,
equipment failures and other factors. If the incident is not handled in time,
it will lead to baggage accumulation, reduce the supply capacity of airport
resources, and can not meet the regular or sudden increase of baggage
transportation demand. In serious cases, it may lead to the delay of aircraft
flights and cause huge losses. The use of unmanned vehicles for emergency
rescue can overcome many disadvantages brought by "human factors", can be
flexibly applied to various complex airport environments and bad weather, meet
the needs of all-weather and unmanned, greatly reduce labor costs and improve
rescue efficiency.

In case of no incident, the goods are received in the bulk cargo receiving
area by zones, the conventional goods are stacked on the pallet and enter the
security inspection line. After automatic information collection, the goods
information is transmitted to the security inspection machine to judge whether
there is any harm. After passing the security inspection, the goods are sent
to the storage area by unmanned logistics Trailer after entering the cargo
station. When the board needs to be punched, the freight management system
gives a signal, The unmanned logistics trailer will send the bulk cargo to be
punched to the corresponding punching position, and the punching operation
will be carried out manually. After the punching is completed, call the
unmanned logistics trailer to send the baggage to the aircraft baggage haulage
warehouse. After the incident, the unmanned car shall carry out rescue and
baggage handling tasks.

Figure 12: Civil Aviation Airport

Compared with experiment 1, the experimental settings are modified as follows.
The environment is defined as a 13×21 grid area. Step is limited to 180. There
are 31 nodes in the scene. Using a team of four agents. Eight nodes are
randomly selected in the scene as the initial incident nodes. The experimental
scenario is shown in Fig12.

We use greedy algorithm, MADDPG and P-MADDPG to experiment. We evaluate the
algorithms in terms of transport efficiency $TE$, inventory carry rate $IT$,
incident damage rate $rate_{f}$and reward $r$. The specific calculation
formulas of transport efficiency $TE$ and inventory carry rate $IT$ are as
follows:

$TE=\frac{(n_{a_{1}}+...+n_{a_{i}})}{n_{O}*(n+k)*2}$
$IT=\frac{(n_{a_{1}}+...+n_{a_{i}})}{n_{a}*2}$

Where Where, $n_{a_{i}}$ represents the number of bags transported by agent
$a_{i}$, and $n_{a}$ represents the number of checked baggage.

For the experiment with 3 agents, the multi-agent adopts greedy algorithm,
MADDPG and P-MADDPG respectively. After 25000 episode training, the
experimental results are shown in Fig.13.

Figure 13: Comparison results of three indexes of greedy algorithm, MADDPG and
P-MADDPG in civil aviation airport. (a)transport efficiency $TE$. (b)inventory
carry rate $IT$. (c)reward $r$.

In Fig.13(a), the transport efficiency $TE$ of P-MADDPG and MADDPG is
basically the same. After P-MADDPG converges, the transport efficiency $TE$ is
slightly lower than that of MADDPG. The transport efficiency $TE$ of greedy
algorithm is the lowest, between 0-$5\%$. In Fig.13(b),… In Fig.13(c), the
incident damage rate $rate_{f}$ of P-MADDPG starts to converge at 5000
episode. After convergence, the incident damage rate $tare_{f}$ can reach as
low as $5\%$, and among the three algorithms, the incident damage rate
$rate_{f}$ is the lowest. The incident damage rate $rate_{f}$ of MADDPG is the
worst, reaching about $60\%$ after convergence. The incident damage rate of
greedy algorithm is between MADDPG and P-MADDPG, which is basically maintained
between $20\%$-$40\%$. In Fig.13(d), compared with MADDPG, the reward of
P-MADDPG will converge faster and be greatly improved after adding the
prediction mechanism.

Nine different test problems are designed to verify the performance of the
algorithm in the civil aviation airport. The experimental results are shown in
Fig.14. Fig.14 shows the comparison results among degree, MADDPG and P-MADDPG
under three experimental configurations. Fig.14(a) shows the average transport
efficiency $\bar{TE}$. Fig.14(b) shows the average inventory turnover
$\bar{IT}$. Fig.14(c) shows the average incident damage rate $\bar{rate_{f}}$.
Fig.14(d) shows the average reward $\bar{r}$.

Figure 14: Comparison results of P-MADDPG and other two methods in three
different scenarios. (a)average transport efficiency $\bar{TE}$. (b)average
inventory turnover rate $\bar{IT}$. (c)average incident damage rate
$\bar{rate_{f}}$. (d)average reward $\bar{r}$.

TableVI-C summarizes the average of each index of all algorithms under the
three configurations, and the best results of each index are highlighted in
bold.

Algorithm |  | Number of
---
agents and
incidents
$\bar{TE}$ | $\bar{IT}$ | $\bar{rate_{f}}$ | $\bar{r}$
Greedy | 2,4 | 6.9% | 5.6% | 76.1% | \
3,6 | 4.9% | 4.3% | 89.5% | \
4,8 | 2.5% | 2.5% | 79.3% | \
MADDPG | 2,4 | 8.9% | 12.5% | 71.8% | -515.3
3,6 | 8.7% | 11.7% | 82.5% | -663.9
4,8 | 9.1% | 12.5% | 59.8% | -962.6
P-MADDPG | 2,4 | 9.0% | 12.7% | 67.8% | -195.7
3,6 | 8.9% | 12.2% | 76.1% | -311.8
4,8 | 9.2% | 11.5% | 49.4% | -327.0

As can be seen from Fig.14, for P-MADDPG, the average transport efficiency
$\bar{TE}$, the average inventory turnover $\bar{IT}$, the average incident
damage rate $\bar{rate_{f}}$ and the average reward $\bar{r}$ of the three
groups of experiments are the best results. As can be seen from tableVI-C,
among all algorithms and experimental settings, our method P-MADDPG provides
the best results for the four experimental indicators. The maximum $\bar{TE}$
is $9.2\%$ (4 agents, 8 incidents), the maximum $\bar{IT}$ is $12.7\%$ (2
agnts, 4 incidents), the minimum $\bar{rate_{f}}$ is $49.4\%$ (4 agents, 8
incidents), and the maximum $\bar{r}$ is $-195.7$ (2 agents, 4 incidents).

##  VII CONCLUSION

In this paper, we model the emergency decision-making problem for emergent
incidents and propose a simple P-MADDPG algorithm combining GRU and MADDPG to
solve the emergency decision-making problem for emergent incidents. Compared
with the traditional MARL method, this method uses GRU for incident
prediction, and the prediction results are used as the input of MADDPG to
obtain a better learning efficiency. In the experiments, three different
experimental environments were used to generate training samples, and
comparative experiments were conducted with the greedy algorithm, MADDPG and
P-MADDPG to test the performance of the decision-making methods under
different environments, and the experimental results showed the effectiveness
of the P-MADDPG algorithm.

## References

  * [1] Y. Liu and G. Nejat, “Robotic urban search and rescue: A survey from the control perspective,” _J. Intell. Robotic Syst._ , vol. 72, no. 2, pp. 147–165, 2013. [Online]. Available: https://doi.org/10.1007/s10846-013-9822-x
  * [2] A. Quattrinili, R. Cipolleschi, M. Giusto, and F. Amigoni, “A semantically-informed multirobot system for exploration of relevant areas in search and rescue settings,” _Autonomous Robots_ , vol. 40, no. 4, pp. 581–597, 2016.
  * [3] J. Parker, E. Nunes, J. Godoy, and M. L. Gini, “Exploiting spatial locality and heterogeneity of agents for search and rescue teamwork,” _J. Field Robotics_ , vol. 33, no. 7, pp. 877–900, 2016. [Online]. Available: https://doi.org/10.1002/rob.21601
  * [4] A. C. Kapoutsis, S. A. Chatzichristofis, L. Doitsidis, J. B. de Sousa, J. Pinto, J. Braga, and E. B. Kosmatopoulos, “Real-time adaptive multi-robot exploration with application to underwater map construction,” _Auton. Robots_ , vol. 40, no. 6, pp. 987–1015, 2016. [Online]. Available: https://doi.org/10.1007/s10514-015-9510-8
  * [5] G. Binetti, D. Naso, and B. Turchiano, “Decentralized task allocation for surveillance systems with critical tasks,” _Robotics Auton. Syst._ , vol. 61, no. 12, pp. 1653–1664, 2013. [Online]. Available: https://doi.org/10.1016/j.robot.2013.06.007
  * [6] C. Robin and S. Lacroix, “Multi-robot target detection and tracking: taxonomy and survey,” _Auton. Robots_ , vol. 40, no. 4, pp. 729–760, 2016. [Online]. Available: https://doi.org/10.1007/s10514-015-9491-7
  * [7] K. Dorling, J. Heinrichs, G. G. Messier, and S. Magierowski, “Vehicle routing problems for drone delivery,” _IEEE Trans. Syst. Man Cybern. Syst._ , vol. 47, no. 1, pp. 70–85, 2017. [Online]. Available: https://doi.org/10.1109/TSMC.2016.2582745
  * [8] X. Xu, L. Zuo, X. Li, L. Qian, J. Ren, and Z. Sun, “A reinforcement learning approach to autonomous decision making of intelligent vehicles on highways,” _IEEE Trans. Syst. Man Cybern. Syst._ , vol. 50, no. 10, pp. 3884–3897, 2020. [Online]. Available: https://doi.org/10.1109/TSMC.2018.2870983
  * [9] H. X. Pham, H. M. La, D. Feil-Seifer, and M. C. Deans, “A distributed control framework of multiple unmanned aerial vehicles for dynamic wildfire tracking,” _IEEE Transactions on Systems Man Cybernetics Systems_ , pp. 1–12, 2018.
  * [10] H. Jaldell, “How important is the time factor? saving lives using fire and rescue services,” _Fire Technology_ , vol. 53, no. 2, pp. 695–708, 2017\.
  * [11] Henrik, Jaldell, Prachaksvich, Lebnak, Anurak, and Amornpetchsathaporn, “Time is money, but how much? the monetary value of response time for thai ambulance emergency services,” _Value in Health_ , vol. 17, no. 5, pp. 555–560, 2014.
  * [12] A. Mukhopadhyay, G. Pettet, S. Vazirizade, Y. Vorobeychik, M. J. Kochenderfer, and A. Dubey, “A review of emergency incident prediction, resource allocation and dispatch models,” _CoRR_ , vol. abs/2006.04200, 2020. [Online]. Available: https://arxiv.org/abs/2006.04200
  * [13] D. Saldana, R. Assuncao, and M. Campos, “A distributed multi-robot approach for the detection and tracking of multiple dynamic anomalies,” in _IEEE International Conference on Robotics Automation_ , 2015.
  * [14] J. Cortés, “Cooperative detection of areas of rapid change in spatial fields,” _Pergamon Press, Inc._ , 2012.
  * [15] I. Bozcan and E. Kayacan, “Uav-adnet: Unsupervised anomaly detection using deep neural networks for aerial surveillance,” in _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2020.
  * [16] N. Patel, A. N. Saridena, A. Choromanska, P. Krishnamurthy, and F. Khorrami, “Adversarial learning-based on-line anomaly monitoring for assured autonomy,” 2018.
  * [17] A. Chriki, H. Touati, H. Snoussi, and F. Kamoun, “Uav-based surveillance system: an anomaly detection approach,” in _2020 IEEE Symposium on Computers and Communications (ISCC)_ , 2020.
  * [18] G. Pettet, A. Mukhopadhyay, M. J. Kochenderfer, Y. Vorobeychik, and A. Dubey, “On algorithmic decision procedures in emergency response systems in smart and connected communities,” 2020.
  * [19] A. Mukhopadhyay, G. Pettet, S. Vazirizade, Y. Vorobeychik, M. J. Kochenderfer, and A. Dubey, “A review of emergency incident prediction, resource allocation and dispatch models,” _CoRR_ , vol. abs/2006.04200, 2020. [Online]. Available: https://arxiv.org/abs/2006.04200
  * [20] A. Mukhopadhyay, G. Pettet, C. Samal, A. Dubey, and Y. Vorobeychik, “An online decision-theoretic pipeline for responder dispatch,” in _Proceedings of the 10th ACM/IEEE International Conference on Cyber-Physical Systems, ICCPS 2019, Montreal, QC, Canada, April 16-18, 2019_ , X. Liu, P. Tabuada, M. Pajic, and L. Bushnell, Eds. ACM, 2019, pp. 185–196. [Online]. Available: https://doi.org/10.1145/3302509.3311055
  * [21] A. Mukhopadhyay, Y. Vorobeychik, A. Dubey, and G. Biswas, “Prioritized allocation of emergency responders based on a continuous-time incident prediction model,” in _Sixteenth International Conference on Antonomous Agents and Multiagent Sytems_ , 2017.
  * [22] S. M. Vazirizade, A. Mukhopadhyay, G. Pettet, S. E. Said, H. Baroud, and A. Dubey, “Learning incident prediction models over large geographical areas for emergency response systems,” _CoRR_ , vol. abs/2106.08307, 2021. [Online]. Available: https://arxiv.org/abs/2106.08307
  * [23] A. Mukhopadhyay, K. Wang, A. Perrault, M. J. Kochenderfer, M. Tambe, and Y. Vorobeychik, “Robust spatial-temporal incident prediction,” in _Proceedings of the Thirty-Sixth Conference on Uncertainty in Artificial Intelligence, UAI 2020, virtual online, August 3-6, 2020_ , ser. Proceedings of Machine Learning Research, R. P. Adams and V. Gogate, Eds., vol. 124\. AUAI Press, 2020, pp. 360–369. [Online]. Available: http://proceedings.mlr.press/v124/mukhopadhyay20a.html
  * [24] P. Leitao and F. Restivo, “A holonic approach to dynamic manufacturing scheduling,” _Robotics Computer Integrated Manufacturing_ , 2008.
  * [25] G. Pettet, A. Mukhopadhyay, M. J. Kochenderfer, Y. Vorobeychik, and A. Dubey, “On algorithmic decision procedures in emergency response systems in smart and connected communities,” 2020.
  * [26] L. Nie, Y. Bai, X. Wang, K. Liu, and C. Cai, “An agent-based dynamic scheduling approach for flexible manufacturing systems,” in _IEEE 16th International Conference on Computer Supported Cooperative Work in Design, CSCWD 2012, May 23-25, 2012, Wuhan, China_ , L. Gao, W. Shen, J. A. Barthès, J. Luo, J. Yong, W. Li, and W. Li, Eds. IEEE, 2012, pp. 59–63. [Online]. Available: https://doi.org/10.1109/CSCWD.2012.6221798
  * [27] S. Shriyam and S. K. Gupta, “Incorporation of contingency tasks in task allocation for multirobot teams,” _IEEE Trans Autom. Sci. Eng._ , vol. 17, no. 2, pp. 809–822, 2020. [Online]. Available: https://doi.org/10.1109/TASE.2019.2946688
  * [28] J. Turner, Q. Meng, G. Schaefer, A. Whitbrook, and A. Soltoggio, “Distributed task rescheduling with time constraints for the optimization of total task allocations in a multirobot system,” _IEEE Trans. Cybern._ , vol. 48, no. 9, pp. 2583–2597, 2018. [Online]. Available: https://doi.org/10.1109/TCYB.2017.2743164
  * [29] T. Duan, W. Wang, T. Wang, M. Huang, and H. He, “Dynamic tasks scheduling model for uav cluster flexible network architecture,” in _2020 3rd International Conference on Unmanned Systems (ICUS)_ , 2020.
  * [30] P. D. Zou, “Research of off-line programming and virtual simulation technology based on mitsubishi industrial robot workstation,” _Science Technology Vision_ , 2019.
  * [31] D. W. Casbeer, D. B. Kingston, R. W. Beard, and T. W. McLain, “Cooperative forest fire surveillance using a team of small unmanned air vehicles,” _Int. J. Syst. Sci._ , vol. 37, no. 6, pp. 351–360, 2006. [Online]. Available: https://doi.org/10.1080/00207720500438480
  * [32] L. Merino, F. Caballero, J. Martínez-De-Dios, I. Maza, and A. Ollero, “An unmanned aircraft system for automatic forest fire monitoring and measurement,” _Journal of Intelligent Robotic Systems_ , vol. 65, no. 1-4, pp. 533–548, 2012.
  * [33] D. Saldana, R. Assuncao, and M. Campos, “A distributed multi-robot approach for the detection and tracking of multiple dynamic anomalies,” in _IEEE International Conference on Robotics Automation_ , 2015.
  * [34] A. Inteha _et al._ , “A gru-ga hybrid model based technique for short term electrical load forecasting,” in _2021 2nd International Conference on Robotics, Electrical and Signal Processing Techniques (ICREST)_. IEEE, 2021, pp. 515–519.