# A Virtual Reality-based Training and Assessment System for Bridge Inspectors
with an Assitant Drone

Yu Li, Muhammad Monjurul Karim, Ruwen Qin Y. Li, M.M. Karim, R. Qin are with
the Department of Civil Engineering, Stony Brook University, Stony Brook, NY,
11794, USA.Corresponding author<EMAIL_ADDRESS>2424 Computer
Science Building, Stony Brook University, Stony Brook, NY 11794-4424

##  1 Abstract

Over 600,000 bridges in the U.S. must be inspected every two years to identify
flaws, defects, or potential problems that may need follow-up maintenance. An
aerial robotic technology, Unmanned Aerial Vehicles (drones), has been adopted
for bridge inspection for improving safety, efficiency, and cost-
effectiveness. Although drones have an autonomous operation mode, keeping
inspectors in the loop is still necessary for complex tasks like bridge
inspection. Therefore, inspectors need to develop the skill and confidence in
operating drones in their jobs. This paper presents the design and development
of a virtual reality-based system for training inspectors who are assisted by
a drone in the bridge inspection. The system is composed of four integrated
modules: a simulated bridge inspection developed in Unity, an interface that
allows a trainee to operate the drone in simulation using a remote controller,
monitoring and analysis that analyzes data to provide real-time, in-task
feedback to trainees to assist their learning, and a post-study assessment for
accelerating the learning of trainees. The paper also conducts a small-size
experimental study to illustrate the functionality of this system and its
helpfulness for establishing the inspector-drone partnership. The developed
system has built a modeling and analysis foundation for exploring advanced
solutions to human-drone cooperative inspection and human sensor-based human-
drone interaction.

Keywords: Virtual Reality, Infrastructure Inspection, Unmanned Aerial Vehicle,
Training, Performance Assessment, Sensing, Human-in-the-loop

##  2 Introduction

The U.S. Highway Bridge Inventory has approximately 617,000 bridges. 42% of
them are over 50-years old, and 7.5% are structurally deficient [3]. To avoid
catastrophic incidents, all bridges are required to be inspected every two
years for identifying flaws, defects, or potential problems that may need a
follow-up maintenance. Traditional bridge inspection may require closing the
traffic and the use of heavy equipment such as a snooper truck. Inspecting a
bridge needs a crew of inspectors working at the site for many hours. Some
field operations are dangerous, such as climbing up to high bridge columns.

To make bridge inspection safer, faster, and cheaper, an aerial robotic
technology, Unmanned Aerial Vehicles or drones, has been adopted for use. A
drone can conveniently access various locations of a bridge to capture a large
amount of inspection data efficiently using sensors that it carries, such as a
camera. Then, inspectors can stay in a safe location to analyze the inspection
data. The use of drones for data collection also minimizes the traffic closure
and the use of heavy, expensive equipment. A survey conducted by the American
Association of State Highway and Transportation Officials (AASHTO) shows that
using drones for bridge inspection can reduce the cost by 74% [1]. Besides
bridges, other low-accessible infrastructures have also been adopting drones
for inspection, such as dams and penstocks [23], transmission lines [14], and
railways [4].

Current studies on the bridge inspection with an assistant drone mainly focus
on the drone technology (e.g., [19]) and data analysis using image processing
and computer vision (e.g., [12]). The human factors aspect is largely ignored.
The use of drones in bridge inspection is not to eliminate humans [11].
Instead, drones and inspectors must form a cooperative team to enhance
inspection safety, efficiency, and effectiveness. To inspectors, collaborating
with drones in bridge inspection is a complex activity wherein human factors
significantly contribute to its success [8]. Although drones have an
autonomous operation mode, a ground operating crew needs to monitor the
operation and the bridge condition dynamically. In response to an alarming
situation or a suddenly emerging need, an inspector has to adjust the
autonomous level of the drone, including disengaging it from the autonomous
mode and taking control of it [13]. Keeping human-in-the-loop is necessary for
bridge inspection with a drone.

Training is essential to help inspectors gain the skill and confidence in
inspecting bridges with an assistant drone. Drones specialized for bridge
inspection are usually more expensive than those for entertainment, and
crashes into the traffic will cause severe consequences. Therefore, the
tolerance to drone crashes is low. Like aviation training systems or driving
simulators, simulation-based training and assessment of inspector-drone
cooperation is a cost-effective way to meet the need. A virtual reality (VR)
environment can provide simulated scenes and tasks similar to the real-world
inspection. VR-based training systems have been developed for civil engineers
in pipe maintenance [26], bridge construction process [25], and bridge crane
operation [6]. Some commercial drone flight simulations have been developed.
For example, AeroSim Drone Simulator [2] offers training scenarios of
inspecting wind turbines, power lines, towers, and solar panels. Moud et al.
[20] also developed a first-ever drone flight simulator for construction
sites. To our best knowledge, no simulator has been developed for inspector-
drone cooperative bridge inspection, nor a data-driven framework for assessing
the training performance of bridge inspectors.

A simulator for training bridge inspectors to operate a drone in their jobs is
needed. Some unique features of bridge inspection differentiate it from other
types of inspection, such as the traffic passing the bridge, complex and
diverse structures, narrow irregular spaces between structural elements such
as diaphragms, interlayers to name a few. Moreover, factors that may impact
the bridge inspection with an assistant drone are broad, including job site-
related, drone-related, job-related, and human-related factors. While some
factors are commonly considered by drone simulators, such as the wind speed
and direction, battery level, and task difficulty level (e.g., [8, 2, 20,
24]), those are not sufficient for characterizing the drone assisted bridge
inspection. Skills and confidence developed from simulators for inspecting
other types of infrastructure are not transferred to bridge inspection.

Monitoring and assessing training performance in a data-driven approach are
beneficial. Feedback to inspectors, both in-training and post-training, help
accelerate their learning processes. Effective feedback to an inspector should
be built on measurements of the inspector’s tasks performance and human states
(e.g., cognitive load, physical load, emotion, and other psychological
states). The measurements are from multiple dimensions, include time, quality,
productivity, safety, cost to list several. While task performance
measurements have studied widely in operations management [22], many metrics
are output-based such as the time of completion and productivity, not
applicable to providing in-task feedback. Subjective evaluation of task
performance and human states can be performed by the subject herself/himself,
by peers, or by one or a group of evaluators. Subjective judgement lacks
reliability, timeliness, and accuracy. Despite of these limitations,
subjective evaluation is still commonly used due to the low cost and the ease
to implement. NASA Task Load Index (NASA-TLX) [10] is a questionnaire commonly
used by pilots for reporting their physical demand, time pressure, effort,
performance, mental demand, and frustration level [7, 8]. A data-driven
approach to training performance monitoring and assessment would address the
limitations of existing methods by supplementing them.

This paper presents the design and development of a training and assessment
system for bridge Inspection with an assistant drone (TASBID). Filling gaps in
the literature, contributions of this paper are threefold:

  * •

A factorial framework for creating a VR-based bridge inspection simulator,
which allows the inspector to operate a drone in a virtual inspection using a
remote controller;

  * •

A data-driven approach to providing real-time, in-task feedback, which is
achieved through monitoring and analyzing the inspector’s operation;

  * •

A comprehensive post-study assessment of inspectors’ task performance for
learning acceleration.

The source code of the system is publicly available for download at Github
[15].

The remainder of this paper is organized as the following. The next section
presents the proposed TASBID. Section 4 exhibits the functionality of the
system with the data collected from a small group of participants. In the end,
Section 5 concludes the study and summarizes directions of future work.

##  3 The Training and Assessment System

The architecture of the training and assessment system for bridge inspection
with an assistant drone (TASBID) is illustrated in Figure 1. The system is
designed to consist of four modules: bridge inspection simulation, interfaces
that allows the bridge inspector (the trainee) to interact with the drone in
the simulated inspection, monitoring & data analysis, and post-study
assessment. The inspector, may be equipped with biometric sensors, operates
the drone in the simulated inspection using a remote controller. Streaming
data of the inspector and the drone are monitored to provide real-time in-task
feedback to the inspector. The data on job specifications, the bridge, and the
site are references for monitoring and analysis. Upon completing a simulation
study, a comprehensive assessment based on the collected data is performed to
provide the post-study feedback to the trainee and the management. Below is a
discussion of the four modules in detail.

Figure 1: Architecture of the training and assessment system for bridge
inspection with an assistant drone (TASBID)

###  3.1 Simulated Bridge Inspection

The inspection simulation created in Unity is illustrated in Figure 1. The
simulation provides the visual stimulus of bridge inspection with an assistant
drone for the inspector. To assure that it is close to the real-world work
context, the simulation is designed to include five major elements of bridge
inspection: the ground team, the drone, bridges, the job site, and example
tasks.

####  3.1.1 The Ground Team

The ground team in the simulation comprises an inspector (the virtual
counterpart of the trainee) and a truck. The simulation defines the location
of the ground team where the drone takes off and returns to. The simulation
provides two views to the trainee. The first one is the view of the inspector
in the simulation. The second one is the camera’s view that is moving along
with the drone. The trainee uses these two views alternatively, depending on
the stage of the inspection. For example, the inspector’s view can be useful
during taking-off and landing. The camera’s view is what the trainee would
concentrate on during the inspection and the navigation.

####  3.1.2 The Drone

The simulation adopts a drone simulation model from a Unity drone controller
asset [9] and revises it for the bridge inspection. Parameters for modeling
the drone include the drone model, mass and load, movement force, maximum
forward speed, maximum side-ward speed, rotation speed, slow-down time,
movement sound, the propellers’ rotation, battery capacity, and movement
types. In this simulation, eight types of movement are sufficient for the
bridge inspection. They are forward & backward, right sideward & left
sideward, up & down, and right rotation & left rotation. The trainee controls
the movement of the drone and the speed using the remote controller. The
battery level is a dynamic constraint for the drone operation, which drops
gradually during the inspection. This simulation does not include the return
to home function that can bring the drone to the home point when the battery
level drops to a pre-specified level. Instead, the battery level is displayed
for examining the trainee’s time stress.

####  3.1.3 Bridges

The simulation uses Road Architect [5] to create the bridge models, wherein
multiple types of bridges are available for choice and redesign. The
simulation includes an arch bridge and a suspension bridge to provide trainees
with different experiences in training. For example, the arch bridge in the
simulation has cramped spaces where the drone manipulation is challenging to
the trainee. Road Architect defines the structural elements of the bridges.
Accordingly, the spatial-temporal relationship between the drone and specific
bridge elements during the simulated inspection can be determined. Defects
such as cracks are added to the surface of some bridge elements, to assess the
trainee’s ability to recognize defects during the inspection.

####  3.1.4 The Job Site

Simulation of the bridge inspection site focuses on creating the geographic
context, the environmental condition, and the traffic condition at the
bridges. Bridges to be inspected sit on a lake in a mountain area. Bridge
inspection needs to be conducted in the daytime with a clear weather although
a sudden change in the weather may occur in rare cases. Therefore, only wind
under level five of the Beaufort Wind Scale has been considered as a possible
weather impact in the system. The wind factor is simulated by adding the force
value and direction in Unity. Since most commercial drones can be flown in the
wind between 10 and 30 mph, TASBID considers three levels of wind: light,
gentle, and medium. They correspond to the wind speed around 2mph, 11mph and
22mph, and cause the force of 0.12N, 3N and 12N, respectively. A three-
dimensional vector can set up the wind direction. Light condition is another
common impact factor for the inspection. Some dim areas of bridges, such as
the bridge bottom, are present although the natural light condition is good.
TASBID is designed to include tasks to inspect dim areas of bridges. The drone
in the simulation is equipped with a light. Trainees can turn it on or off
according to their needs by pressing the key “B” on the keyboard. Traffic
volume is considered as well because inspectors may feel pressure when flying
the drone near the traffic. Vehicles moving on the bridges are included in
TASBID using a free traffic simulation asset [18]. The total number of
vehicles can be increased or decreased corresponding to the design of
experiments.

####  3.1.5 Inspection Tasks

Tasks of the simulated inspection are designed based on the specific goal of
training. TASBID includes four tasks that differ in the flight paths, the
space for the drone, and the light condition, and the requirement.

  * •

Task 1 is to inspect the slab of the arch bridge from one side. The flying
path is a straight line, and the open space for the drone is ample. The
trainee is expected to keep the drone 1$\sim$2 meter away from the bridge in
task 1 and also in tasks 2 and 3 below.

  * •

Task 2 is to inspect the bottom of the arch bridge. Different from task 1,
task 2 has a curved flying path and the light condition under the bridge is
darker.

  * •

Task 3 is about inspecting the cramped interlayer area. Compared to tasks 1
and 2, task 3 has a very short flying path and a limited space for the drone,
and the drone requires fine adjustments to let the camera capture both the
upper side and the down side of the interlayer area.

  * •

Task 4 is about harvesting the monitoring data captured by sensing devices
placed under the water near a pier of the suspension bridge. The trainee is
expected to fly the drone around the pier in a lap. The trainee should keep
the drone 5$\sim$7 meters away from the pier and around 3 meters above the
water.

The recommended speed during the inspection is 10 mph. The recommended task
sequence (tasks 1, 2, 3, and 4) is not mandatory. The inspector is expected to
complete all the four tasks timely and return to the ground station,
especially before the drone is running out of power.

###  3.2 Interface between the Trainee and the Drone

The trainee manipulates the drone in the simulated inspection using a remote
controller. TASBID uses a Phantom 2 DJI controller for this purpose. The
controller is connected to Unity, the development environment for creating the
simulation, using the vJoy device driver [21] and the method in mDjiController
[17]. The trainee adjust the joysticks of the controller to control the drone
movement and speed. The trainee’s data on operating the controller are
recorded.

###  3.3 Monitoring & Analysis

TASBID can collect six types of data from the study, as Figure 1 illustrates.
The workplace characteristics, the bridge models, the drone model, and job
specifications are pre-specified static data that do not change during a
study. The flight data of the drone and the trainee’s operation data are the
frame-level streaming data that vary in each time of the study. Although this
paper skips the discussion of biometric sensory data, TASBID has already
integrated a screen-based eye tracker and an EEG device to capture trainees’
eye and brain activities, respectively.

####  3.3.1 Streaming Data

A simulated inspection is captured by a sequence of $N$ frames, indexed by
$i$. Given the frame rate, $f$, the total duration of an inspection is $N/f$.
The starting frame of the inspection is defined as the time when the drone is
taking off. The ending frame corresponds to the time when the drone lands near
the ground team or it is not able to continue and finish the inspection (e.g.,
the battery drains or the drone crashes into the traffic), whichever occurs
the first. Let $O_{i}$ and $D_{i}$ denote the trainee’s operation data and the
drone flight data, respectively, collected at any frame $i$.

The trainee operates the remote controller that has four-axis inputs for
controlling the movement of the drone. Besides, the trainee can use the key
“B” on the keyboard to turn on/off the light and “P” to take snapshots during
the inspection. Therefore, the trainee’s operation data are time series data
in six dimensions:

$O_{i}=[o_{fb,i},o_{rl,i},o_{ud,i},o_{rt,i},o_{b,i},o_{p,i}]$ (1)

where

  * $o_{fb,i}$: Forward(+) $\&$ Backward(-),

  * $o_{rl,i}$: Right(+) $\&$ Left(-) Sideward,

  * $o_{ud,i}$: Up(+) $\&$ Down(-),

  * $o_{rt,i}$: Right(+) $\&$ Left(-) Rotation,

  * $o_{b,i}$: Turning on(true) $\&$ off(false) the light,

  * $o_{p,i}$: Taking a snapshot (true) $\&$ no(false).

The drone flight data include the position, velocity, and the remaining batter
level of the drone:

$D_{i}=[\vec{L}_{i},v_{i},b_{i}]=[l_{x,i},l_{y,i},l_{z,i},v_{i},b_{i}],$ (2)

where $\vec{L}_{i}=(l_{x,i},l_{y,i},l_{z,i})$ are the 3D coordinates of the
drone’s location in the earth reference system, $v_{i}$ is the linear speed of
drone, and $b_{i}$ is the remaining battery level in percentage.

####  3.3.2 Data Analysis

Following the job specifications and focusing on job entities (e.g., the drone
and bridge elements to inspection) contribute to good results from the
inspection. Analysis of the collected data further provides measurements that
reveal if the trainee meets requirements in the inspection. A reference path
for each task is part of the job specifications, which is defined by a
sequence of reference points on the path. Let $t$ be the index of tasks and
$n$ be the index of reference points. $\\{\vec{p}_{t,n}|n=1,\dots,N_{t}\\}$
defines the reference flying path for the drone in task $t$. A speed limit,
$\overline{v}$, is also specified for the drone in performing the inspection
tasks.

On-path Analysis. Denote $X_{t,i}$ as the binary variable indicating if the
drone in frame $i$ is on the reference path of task $t$, for $t=1,\dots,T$ and
$i=1,\dots,N$. $\sum_{t=1}^{T}X_{t,i}\leq 1$ for any $i$, indicating the drone
cannot be on more than one task simultaneously. Using Algorithm 1 below, the
analysis module evaluates if the drone is on the reference path for task $t$.
Specifically, the algorithm uses the reference path of task $t$ and the
location of the drone as inputs to determine the value of the binary variable
$X_{t,i}$. Using the output of the algorithm, the starting frame of task $t$,
$I_{t,s}$, and the ending frame, $I_{t,e}$, are determined accordingly. The
analysis module treats the first frame when $X_{t,i}$ takes one as the
starting frame for task $t$ and the last frame when $X_{t,i}$ takes one as the
ending frame:

$\displaystyle I_{t,s}=\min_{i}\\{i|X_{t,i}=\text{true}\\}$ (3) $\displaystyle
I_{t,e}=\max_{i}\\{i|X_{t,i}=\text{true}\\}$

Algorithm 1 On-path analysis for task $t$ at any frame $i$

// $\\{\vec{p}_{t,n}|n=1,\dots,N_{t}\\}$: reference points that define the
reference path for the drone in task $t$,

// $\vec{L}_{i}$: position of the drone in frame $i$,

// $l(\vec{L}_{i},\vec{p}_{t,n}$): the distance between the drone and the
reference point $p_{t,n}$,

// $n^{\ast}$: the index of the reference point with the shortest distance to
the drone,

// $v_{t,n}$: the segment of the reference path, defined by $\vec{p}_{t,n}$
and its adjacent point(s),

// $l(\vec{L}_{i},v_{t,n})$: the distance from the drone to any location on
the segment $v_{t,n}$,

// $l^{\ast}_{t,i}$: the minimum distance from the drone to the reference
path,

// $\overline{l}_{t}$: the threshold distance for identifying if the drone is
on the reference path for task $t$,

// $X_{t,i}$: binary variable indicating whether the drone in frame $i$ is on
the reference path of task $t$.

Step 1: find the reference point with the shortest distance to the drone,
$\vec{p}_{t,n^{\ast}}$, where

$n^{\ast}:=\operatorname*{arg\,min}_{n}\\{l(\vec{L}_{i},\vec{p}_{t,n})|n=1,\dots,N_{t}\\}$.

Step 2: The shortest distance from the drone to the reference path is computed
as:

$l^{\ast}_{t,i}=\min l(L_{i},v_{t,n^{\ast}})$

Step 3: Determine if the drone in frame $i$ is on path:

$X_{t,i}=\textbf{1}\\{l^{\ast}\leq\overline{l}_{t}\\}$

Return ($X_{t,i}$, $l^{\ast}_{t,i}$)

Analysis of Speed Control. Similarly, a binary variable, $X_{s,i}$, is defined
to indicate if the drone in frame $i$ is speeding when performing inspection
tasks.

$X_{s,i}=\textbf{1}\\{v_{i}>\bar{v},i\in\cup_{t=1}^{T}[I_{t,s},I_{t,e}]\\}.$
(4)

Crash Analysis. A crash in the simulation is defined as an event that the
drone touches traffic agents, the bridges, the terrain, or the waterbody. The
simulation can sense types of objects the drone crash into and track crash
events. $X_{h,i}$ is a binary variable indicating if the drone in frame $i$
touches a human in the traffic. $X_{v,i}$ is another binary variable
indicating if the drone crashes into a vehicle in the traffic. $X_{o,i}$ is a
categorical variable indicating if the drone touches any other objects. A
crash event may last for multiple frames. Therefore, whenever $X_{h,i}$ turns
from “false” to “true”, the simulation identifies the occurrence of a crash
into a human, indicated by a binary variable $X_{h}$:

$X_{h}=\textbf{1}\\{\exists X_{h,i}=\text{true}\\},$ (5)

and another binary variable $X_{v}$ indicates if a crash into a vehicle
happens:

$X_{v}=\textbf{1}\\{\exists X_{v,i}=\text{true}\\}.$ (6)

Crashing into other objects will not terminate the study. At the end, the
total number of crashes into other objects will be

$X_{o}=\sum_{i=2}^{N}\textbf{1}\\{X_{o,i}\neq\text{false}\,\&\,X_{o,i-1}=\text{false}\\}.$
(7)

Visual Detection Analysis. Experienced bridge inspectors should be sensitive
to defects developed on the surface such as cracks, spalling, corrosion to
name a few. To test this hypothesis, the study randomly places $X_{d}$ surface
defects on the bridges to inspect. The trainee takes a snapshot if she/he
believes a defect is found. The total number of snapshots is the count of
times that the trainee pushes the key “P”:

$X_{pd}=\sum_{i=1}^{N}\textbf{1}\\{o_{p,i}=\text{true}\\}.$ (8)

Some of the snapshots may be false detection. The number of true detection is
$X_{td}$.

####  3.3.3 Real-time, In-task Feedback

Using the monitoring data and measurements calculated from the data, real-
time, in-task feedback is provided to the trainee. TASBID is designed to
provide five types of information, illustrated in Figure 2, to raise the
trainee’s attention to job safety and task specifications.

Figure 2: Illustration of in-task feedback

The remaining battery level in percentage of the battery capacity is updated
in real-time and displayed at the upper right corner of the camera view, as
Figure 2 illustrates. The battery icon is in green color when the remaining
power is 70% or higher, yellow if between 30% and 70%, and otherwise in red.
The battery icon starts to flush once the remaining power drops below 30%. The
displayed battery level sets a time constraint to encourage the trainee to
finish the task before the drone running out of power. The drone’s speed is
always displayed at the upper left corner of the camera view. Three types of
messages may appear at the bottom left when certain conditions occur:

  * •

A message about speeding will show up at the bottom left corner if $X_{s,i}$
in Equation (4) is one.

  * •

A message to remind the recommended distance from the bridge elements will
appear if Algorithm 1 returns $X_{t,i}=0$ but $l^{\ast}_{t,i}\leq 8$m for the
inspection task $t$.

  * •

A message appears if the drone senses any object within 2.5m to the center of
the drone or crashes into anything (i.e., whenever $X_{h,i}$, $X_{v,i}$, or
$X_{o,i}$ turns from false to a non-false value).

###  3.4 Post-Study Assessment

After a simulation study ends, data collected from the study are further used
to perform a comprehensive post-study assessment. The assessment covers the
trainee’s task performance and self-assessment through a questionnaire survey.
The trainee’s task performance is evaluated on four dimensions: conformity,
efficiency, safety, and accuracy.

####  3.4.1 Task Performance

The trainee’s ability to conform with task specifications is term conformity,
reflected by the trainee’s ability to keep on the reference paths and the
recommended speed during the inspection. The ability to be on-path in
performing tasks is measured by the percentage of task time when the drone is
on the reference paths of the tasks:

$P_{p}=\sum_{t=1}^{T}\frac{\sum_{i=I_{t,s}}^{I_{t,e}}X_{t,i}}{I_{t,e}-I_{t,s}+1}.\\\
$ (9)

To measure the trainee’s ability to control the drone at the recommended
speed, a weighted sum of times when the drone is speeding is calculated with
the weights being the speed in the unit of speed limit:

$P_{s}=\sum_{t=1}^{T}\frac{\sum_{i=I_{t,s}}^{I_{t,e}}(v_{i}/\bar{v})X_{s,i}}{I_{t,e}-I_{t,s}+1}.$
(10)

Then, the conformity is an aggregation of $P_{p}$ and $P_{s}$,

$P_{C}=\omega_{p}P_{p}+\omega_{s}P_{s},$ (11)

where $\omega_{p}$ is the gain coefficient for on-path and $\omega_{s}$ is the
loss coefficient for speeding. The range of $P_{C}$ in TASBID is [-100,100].
The maximum score occurs if the drone is always on-path and never speeding in
all tasks (e.g., $P_{p}=1$ and $P_{s}=0$). The minimum score occurs when the
drone is never on-path and always flying at its maximum speed (e.g., $P_{p}=0$
and $P_{s}=v_{\text{max}}/\bar{v}$). The maximum speed of the drone in TASBID
is $30$mph and the speed limit for inspection is $10$mph. Therefore,
$\omega_{p}$ and $\omega_{s}$ are set to 25 and -25/3, respectively.

The trainee’s ability to finish the inspection timely is term time efficiency.
Multiple critical values are defined with respect to the time efficiency of
trainees. $\underline{\tau}$ defines the cut-off point of the inspection time
for receiving the highest score and $\bar{\tau}$ is the maximum allowable
flight time for the drone. The battery drains if the inspection would go
beyond $\bar{\tau}$. Let $X_{b}$ be a binary variable indicating if the drone
fails to return to the ground team due to running out of power. $X_{b}$ equals
one if $N/f>\bar{\tau}$, and zero otherwise. Accordingly, The score of time
efficiency, $P_{E}$, is calculated as:

$P_{E}=[\omega_{e0}+\omega_{e1}(N/f-\underline{\tau})^{+}](1-X_{b})+\omega_{b}X_{b}.$
(12)

The range of $P_{E}$ score is [-100,100]. $\omega_{e0}$ in Equation (12) is
set to be 100, representing the highest efficiency score a trainee receives if
the inspection is done by $\underline{\tau}$. $\omega_{b}$ is set to be -100,
indicating the trainee fails to complete the inspection within the maximum
allowed time $\overline{\tau}$ and thus loses 100 points. $P_{E}$ score will
be 0 if the inspection is completed at the defined maximum allowable time
$\overline{\tau}$.
$\omega_{e1}=-\omega_{e0}/(\overline{\tau}-\underline{\tau})$, representing
the score deduction for every additional unit of time exceeding
$\underline{\tau}$. In TASBID, $\overline{\tau}$ is 25 minutes and
$\underline{\tau}$ is 15 minutes.

The trainee’s ability to avoid any crashes is termed job safety. The lack of
ability to keep safe in inspection is measured by the total lost score due to
crashes:

$P_{S}=\max[\omega_{h}X_{h}+\omega_{v}X_{v}+\omega_{o}X_{o},P^{\prime}_{S}]$
(13)

where $\omega_{h}$, $\omega_{v}$, and $\omega_{o}$ are losses from each crash
into a human, a vehicle, and any other object, respectively. In TASBID,
$\omega_{h}$ and $\omega_{v}$ are set to be -100, indicating a crash into a
traffic agent usually has severe consequences such as a fatality or a
hospitalized incident. $\omega_{o}$ is set to be -3, indicating the
consequence of crash into other objects is more related to the drone damage.
$P^{\prime}_{S}$ is set to be -100 in TASBID, meaning that no more points will
be further deducted if the cumulative loss has reached $P^{\prime}_{S}$.
Therefore, the range of $P_{S}$ score is [-100,0]. Limiting the loss due to
crashes by $P^{\prime}_{S}$ can avoid the scenario that $P_{S}$ dominates
scores of other performance measurements.

A trainee’s ability to recognize surface defects correctly during the
inspection is termed accuracy. The assessment module calculates the recall
(the portion of the surface defects that the trainee detected correctly):

$\text{Rc}=X_{td}/X_{d},$ (14)

and the precision (the portion of snapshots that contain a surface defect):

$\text{Pr}=X_{td}/X_{pd},$ (15)

to measure the accuracy. $\text{F}_{\beta}$ further integrates the recall and
the precision as a single metric:

$\text{F}_{\beta}=\frac{(1+\beta^{2})\text{Pr}\text{Rc}}{\beta^{2}\text{Pr}+\text{Rc}},$
(16)

where $\beta$ is a non-negative coefficient indicating the relative importance
of recall with respect to precision. Setting $\beta$ as zero indicates
precision is dominantly important, and setting it as $\infty$ means recall is
dominantly important. $\beta$ is equal to one if precision and recall are
equally important. $\text{F}_{\beta}$ is within [0, 100%]. Accordingly, the
score of accuracy is measured as

$P_{A}=\omega_{f}\text{F}_{\beta}.$ (17)

$\omega_{f}$ in Equation (17) is set to be 100 and so the range of $P_{A}$ is
[0, 100].

The trainee’s scores on conformity, efficiency, safety, and accuracy are
further standardized to be within the range from 0% to 100%. Then the
standardized scores are presented as a Kiviat diagram to show the subject’s
task performance on the four dimensions.

####  3.4.2 Questionnaire-based Workload Assessment

After the simulation study is completed, the trainee is invited to fill out a
questionnaire adopted from [8] and revised for TASBID. Table 1 list the six
aspects the questionnaire asks. “Time Pressure”, “Frustration”, and “In-task
Feedback” are three aspects asked regarding the overall simulation study.
“Performance”, “Mental Demand”, and “Physical Demand” are asked with respect
to each phase or task of the inspection, including calibration, taking-off,
individual tasks 1$\sim$4, and landing. The response to questions are on a
five-point likert scale: strongly agree (1), agree (2), neutral (3), disagree
(4), and strongly disagree (5). “Strongly agree” stands for the most positive
response, and “strongly disagree” stands for the most negative response.

Table 1: Questionnaire Questions | QUESTIONS
---|---
Overall |  Time Pressure: I finished the inspection without stress in regard of the required time.
Frustration: I never felt insecure, irritated, stressed, or discomforted
during this task.
In-task Feedback: The in-task feedback (e.g. battery level, speed, messages)
were helpful for me.
By tasks |  Performance: I finished the task with a good performed.
Mental Demand: It’s easy to finish the task.
Physical Demand: There was no physical activity (including pressing, pulling,
turning, controlling, and holding) required in the task.

Heavy physical or cognitive load may cause frustration and time pressure, and
these psychological states may further influence the task performance. In-task
feedback may mitigate the negative effect of the physical and mental loads
posed on inspectors. The questionnaire can assist in causation analysis of the
aforementioned relationship among causal factors (physical and mental
demands), human states (time pressure and frustration), task performance, and
the moderator (in-task feedback).

####  3.4.3 Continuous Improvement

Repetitive training would help improve a trainee’s task performance and
tolerance to physical and mental demands. The improvement is manifested by
progressive changes in both performance measurements and subjective evaluation
results, which are the learning curve of the trainee. A hypothesis is that the
post-study feedback would accelerate the learning of the trainee.

##  4 Capability Demonstration

A small-scale experimental study is conducted to demonstrate the capabilities
of the developed TASBID. Seven volunteers participated in the study. Among
them, two are female, and five are male. Their ages are from 18 to 45, and
their background is in civil engineering. All participants have no prior
experience with operating drones, but some of them have the experience of
playing video games.

###  4.1 Experiment Protocol

The experimental protocol for the simulation study is the following. In the
beginning, an introduction to TASBID will be presented to the participant
using a few PowerPoint slides. Then, a short tutorial [16] on the simulation
study is given to the participant, which is presented as images and video
clips with annotations. After that, the participant enters a practice session
that offers an opportunity to practice the drone operation using the provided
remote controller. The practice scene has some random variations from the
scene for the simulation study. The inspection simulation starts after the
participant feels she/he has enough practice and is ready for the study. After
the simulation study, the participant will fill out the questionnaire and then
exit the study. The duration of the entire study can last 15$\sim$60 minutes,
depending on the participant’s prior experience with TASBID. Fig. 3
illustrates a participant operating the drone in the simulated inspection. The
display screen is split into two parts. The left portion is the operator’s
view and the right portion is the camera’s view. Vivid videos of the
inspection simulation can be found at the project website [16].

Figure 3: A participant in the simulated study

###  4.2 Task Performance

####  4.2.1 Task Performance of Individuals

The task performance of any participant in a simulation study is calculated
according to the assessment method presented in Section 3.4.1. Figure 4 is a
Kiviat diagram that TASBID generates to show the task performance of
participant #6 on four dimensions. It straightforwardly tells that the
strength of the participant is efficiency, and the weaknesses that the
participant can work on and improve are conformity, safety, and accuracy. The
post-study analysis report further explains the detail of the performance
assessment and suggests ways of improvement. The template of the report is
available for download from the project website [16].

Figure 4: Participant 6 Performance Radar Chart

####  4.2.2 Task Performance of a Group

TASBID can further generate Figure 5 to study the task performance of a group.
Four boxplots respectively show the distribution of the group’s performance on
the four dimensions. The figure can tell how diverse the group is on each
performance measurement. For example, the distribution of the group’s safety
score is the largest and that of time efficiency is the smallest. This figure
can also tell the strength and weakness at the group level. The strength of
this group is the efficiency, and the conformity is what the entire group
needs to improve. Furthermore, individuals, represented by colored dots, are
overlaid with the boxplots to compare the performance of any individual to the
group. For example, participant #4 has a relatively low performance and
probably needs more training. Participant #1 has achieved the top performance
on conformity and accuracy among the group, and she/he can focus on improving
efficiency and safety. Data of participants #2-#6 in Figure 5 are their
initial performance (i.e., from their first time of participation). Both
participants #4 and #6 have low scores on conformity and safety, while others
had a good score on conformity, safety, or both. Participants #4 and #6 have
minimum prior experiences with operating a drone, either in physical or
virtual contexts. Participant #1 performs well because she/he benefits from
practicing the skills using TASBID frequently.

Figure 5: Performance of a Group

####  4.2.3 Conformity Analysis

TASBID also analyzes participants’ ability to follow task specifications.
Figure 6 shows how each participant gained scores (green columns) from keeping
on the reference path of each task and lost scores (red columns) due to
speeding. The conformity score (blue columns) is the aggregation of on-path
scores and speeding scores on the four tasks. For example, participant #1
followed the recommended inspection path and speed well in all tasks and
speeding is a negligible issue of her/his performance, thus achieving the
highest conformity among the seven participants. Participant #7 should
particularly pay attention to keeping on the reference path for task 1, and
participant #2 should avoid speeding in task 3.

Figure 6: The waterfall chart interpreting the conformity score of individual
participants

####  4.2.4 Safety Analysis

TASBID analyzes the likelihood of crashes by tasks and by participants. Figure
7 is a stacked column chart counting the number of crashes from each
participant, split by tasks. There were 44 crashes in total, and 25 (56.8%)
occurred when performing inspection tasks. Among the four tasks, the slab
inspection has the largest chance of crashes, and the inspection of the narrow
interlayer space has the second largest chance. The chance of crashes when
inspecting the pier is the lowest. Participant #4 achieved the lowest safety
score in the group, as seen from Figure 5. Figure 7 further shows this
participant had 13 crashes in total, with 10 (76.9%) occurred when she/he was
not performing tasks. Participant #5 had 11 crashes and achieved the second
lowest safety score. Among the 11 crashes, 6 (54.5%) occurred in inspecting
the narrow interlayer space. Participant #6 is another inspector who had a
relatively low safety score too. She/he had 4 crashes when inspecting the
bridge bottom. The observation suggests further training should be
individualized for every inspector according to specific needs for
improvement.

Figure 7: Count of participants’ crashes by tasks

####  4.2.5 Accuracy Analysis

TASBID also generates Fig. 8 to summarize the accuracy of participants in
visual recognition of surface defects. The group has varied ability to detect
surface defects, but most participants (6 out of 7) are 100% correct about
their detection. The observation confirms that false negatives is a more
severe issue than false positives in human visual inspection.

Figure 8: Participants accuracy performance

####  4.2.6 Continuous Improvement Achieved

Participants can improve their skill to operate drones gradually by repetitive
training in TASBID. The post-study assessment provided to participants may
positively influence their learning outcome. For the illustration purpose,
participants #4 and #6 respectively repeated the training for three times
during a two-week period. The overall inspection scene does not change over
the repeated training, but locations of surface defects are changed from one
training to another. The post-study assessment was provided to participant #4,
but not participant #6. Figure 9 illustrates the task performance of
participants #4 and #6, respectively, achieved from their three times of
training. Comparing to the initial training, participant #4 improved the
conformity, efficiency, safety, and accuracy by 13.24%, 12.36%, 49.18% and
7.69%, respectively, from the third round of training, while the changes of
participant #6 are 18.18%, -9%, 20.55% and -12.5%. From informal interviews
with participants #4 and #6 respectively, participant #4 took the post-study
assessment into consideration in late training, whereas participant #6 focused
on dimensions that she/he felt not performed well from the last round of
training. The comparison suggests the data-driven post-study feedback to
participants is objective and may be more effective than participants’ self
perception in improving their skills of bridge inspection. The effect of post-
study assessment on trainees’ learning curves can be tested statistically
after more samples are collected.

Figure 9: Performance improvement over repetitive training

###  4.3 Questionnaire Result

Figure 10 summarizes the average, the most positive, and the post negative
evaluations by the group of seven participants. The average scores of self-
assessment reveal the between-task variations in performance, mental demand,
and physical demand. The most positive and the most negative evaluations from
the group demonstrate the diverse feelings of the participants.

Figure 10: Participants’ Questionnaire Result

Subjective judgement and objective assessment on task performance can be
consistent or contradict. For example, participant #6 believes her/his
performance on the pier inspection task is poor, and the actual performance
measured by TASBID is consistent with the participant’s subjective judgement.
Another example is that participant #2 believes she/he performed well in
inspecting the bridge bottom and the interlayer, but not the slab and the
pier. The actual performances of the participant are exactly the opposite. The
inconsistency between subjective and objective assessments highlights the
importance of TASBID’s ability to provide the performance measurement.
Moreover, the management can investigate the reason for observed contradiction
and how participants’ feeling influences their ability to finish the remaining
tasks. Participants’ self-assessment of workload, human states, and in-task
feedback can facilitate the development of models for analyzing human sensor
data automatically.

A participant’s responses to the questionnaire may change over her/his
repetitive training. Figure 11 visualizes the self-assessment by participants
#4 and #6 from their repetitive training. The row and column labels are the
same as those in 10. Both participants’ responses to the questionnaire change
over multiple rounds of training, but in different manners. Participant #4
felt time pressure in the first round of training. All of her/his responses
were ranged from neural to strongly agree after the second and third rounds of
training. Frequencies of agree or strongly agree after the third round of
training are higher than the second round. The observation indicates
participant #4’s subjective assessment became more and more positive over
training, indicating the confidence and comfort of her/him in the bridge
inspection with an assistant drone are creasing. The responses of participant
#6 over the three rounds of training have a slightly different pattern. In the
first round of training, participant #6’s self-assessment of performance and
mental demand in performing tasks 1-4 were ranged from strongly disagree to
neural, indicating that these tasks were difficult for her/him. All of her/his
responses are at least neural after the second and third rounds of training,
which means the tasks are less challenging to the participant after two rounds
of training. However, from the second round to the third round of training,
both strongly agree and neural are increasing. The comparison suggests the
effectiveness of repetitive training varies among inspectors and the post-
study assessment provided to inspectors may be an impact factor.

Figure 11: Responses to questionnaire by participants 4 and 6.

##  5 Conclusions and Future Work

This paper introduced a training and assessment system named TASBID by
developing a virtual reality-based simulator for bridge inspectors assisted by
a drone. TASBID monitors and analyzes the trainee’s operation to provide real-
time, in-task feedback, assisting the trainee in the simulated bridge
inspection. A post-study assessment further analyzes and reports the trainees’
task performance on dimensions of conformity, efficiency, safety, and accuracy
to increase the trainee’s learning effectiveness.

The design of the inspection simulation has a multi-factorial framework. It
can be revised to provide a wide range of scenarios for training or factorial
experiments. This paper shares the code of the project with the public.
Prospective users can revise and develop it further to adapt to their own
studies [15]. TASBID monitors trainees’ operation data and the drone’s flight
data to provide real-time, in-task feedback and perform post-study assessment.
The demonstration, although is based on a small-scale experimental study,
shows this data-driven approach is timely, individualized, in detail, and
objective. Therefore, TASBID positively supports inspectors in developing the
skill and confidence in operating a drone in bridge inspection.

A more realistic scenario is that the drone is in the autonomous flying mode,
and the inspector takes control when needed. The mode of inspector-drone
cooperative inspection has been added to TASBID. Both the training and the
assessment introduced in this paper have built a foundation for creating these
for the cooperative inspection as well. Besides, a gap is present between the
simulation created in Unity and the real-world inspection scene. To provide
realistic visual stimuli to inspectors, a Generative Adversarial Network (GAN)
can convert the game-based scene into a more realistic one. Furthermore, this
paper focuses on the system design and development, thus only conducting a
small-size experimental study to demonstrate the system functionality.
Factorial experiments at a larger scale would be necessary for the system
testing and improvement. TASBID can integrate a multi-modal biometric sensor
system composed of EEG, eye tracker, ECG, EMG, and IMU. Advanced methods of
data analytics, such as deep neural networks, need to be developed for
analyzing the multi-modal biometric sensor data to reliably detect and
classify human states and for creating other methods of human-drone
interactions. This paper has built a foundation for exploring the above-
discussed opportunities.

## Acknowledgement

Li, Karim, and Qin receive financial support from National Science Foundation
through the grant ECCS-#2026357 in Future Work at the Human Technology
Frontier Program and CMMI-#1646162 in Cyber-Physical Systems Program.

## References

  * [1] AASHTO.  2019 aashto uas/drone survey of all 50 state dots.  Technical report, American Association of State Highway and Transportation Officials, 2019.  Available at: https://www.transportation.org/wp-content/uploads/2019/05/MissionControl_Drones3.pdf, accessed Mar, 2021.
  * [2] AeroSim.  Aerosim drone simulator, n.d.  Available at: https://www.aerosimrc.com/en/pro.htm, accessed Mar, 2021.
  * [3] ASCE.  2021 report card for america’s infrastructure.  Technical report, American Society of Civil Engineers, 2021.  Available at: https://infrastructurereportcard.org/wp-content/uploads/2020/12/National_IRC_2021-report-2.pdf, accessed Mar, 2021.
  * [4] Milan Banić, Aleksandar Miltenović, Milan Pavlović, and Ivan Ćirić.  Intelligent machine vision based railway infrastructure inspection and monitoring using uav.  Facta Universitatis, Series: Mechanical Engineering, 17(3):357–364, 2019.
  * [5] djohns451.  Road architect for unity, 2019.  Available at: https://github.com/MicroGSD/RoadArchitect, accessed Mar, 2021.
  * [6] Haoming Dong, Shaojun Zou, Dingfang Chen, and Hongliu Xu.  Research and development of distributed bridge crane training system based on virtual reality.  In 2009 Second International Conference on Intelligent Computation Technology and Automation, volume 1, pages 670–673. IEEE, 2009.
  * [7] Cali M Fidopiastis, Julie Drexler, Daniel Barber, Keryl Cosenzo, Michael Barnes, Jessie YC Chen, and Denise Nicholson.  Impact of automation and task load on unmanned system operator’s eye movement patterns.  In International Conference on Foundations of Augmented Cognition, pages 229–238. Springer, 2009.
  * [8] G Gabriel, Miguel A Ramallo, and Elizabeth Cervantes.  Workload perception in drone flight training simulators.  Computers in Human Behavior, 64:449–454, 2016.
  * [9] Mario Haberle.  Drone controller full (pc/joystick/mobile), 2021.  Available at: https://assetstore.unity.com/packages/tools/physics/drone-controller-full-pc-joystick-mobile-111163, accessed Mar, 2021.
  * [10] Sandra G Hart and Lowell E Staveland.  Development of nasa-tlx (task load index): Results of empirical and theoretical research.  In Advances in psychology, volume 52, pages 139–183. Elsevier, 1988\.
  * [11] Robyn Hopcroft, Eleanore Burchat, and Julian Vince.  Unmanned aerial vehicles for maritime patrol: human factors issues.  Technical report, Defence Science and Technology Organisation Edinburgh (Australia) Air Operations DIV., 2006.
  * [12] Muhammad Monjurul Karim, Ruwen Qin, Genda Chen, and Zhaozheng Yin.  A semi-supervised self-training method to develop assistive intelligence for segmenting multiclass bridge elements from inspection videos.  Structural Health Monitoring, page 14759217211010422, 2021.
  * [13] Michael P Kontitsis, Nikos C Tsourveloudis, and Kimon P Valavanis.  A uav based automated airborne surveillance system.  In 11th Mediterranean conference on control and automation, Rhodes, Greece, 2003.
  * [14] Yu Li.  A two-layer routing algorithm for unmanned aerial vehicle in transmission line inspection.  In Proceedings of the International Annual Conference of the American Society for Engineering Management., pages 1–9. American Society for Engineering Management (ASEM), 2020.
  * [15] Yu Li.  Bridge inspection with a drone, 2021.  Available at: https://github.com/yuli1102/Bridge-Inspection-with-a-Drone, accessed May, 2021.
  * [16] Yu Li.  Tasbid experiment, 2021.  Available at: https://sites.google.com/view/yuli1102/projects/bridge-inspection/flight-simulation, accessed May, 2021.
  * [17] Matsemann.  mdjicontroller, 2020.  Available at: https://github.com/Matsemann/mDjiController, accessed Mar, 2021.
  * [18] mchrbn.  Unity-traffic-simulation, 2020.  Available at: https://github.com/mchrbn/unity-traffic-simulation, accessed Mar, 2021.
  * [19] Najib Metni and Tarek Hamel.  A uav for bridge inspection: Visual servoing control law with orientation limits.  Automation in construction, 17(1):3–10, 2007.
  * [20] Hashem Izadi Moud, Mohamad A Razkenari, Ian Flood, and Charles Kibert.  A flight simulator for unmanned aerial vehicle flights over construction job sites.  In Advances in Informatics and Computing in Civil and Construction Engineering, pages 609–616. Springer, 2019.
  * [21] n.d.  vjoy device driver, n.d.  Available at: http://vjoystick.sourceforge.net/site/, accessed Mar, 2021.
  * [22] Andy Neely, Mike Gregory, and Ken Platts.  Performance measurement system design: a literature review and research agenda.  International journal of operations & production management, 1995\.
  * [23] Tolga Özaslan, Shaojie Shen, Yash Mulgaonkar, Nathan Michael, and Vijay Kumar.  Inspection of penstocks and featureless tunnel-like environments using micro uavs.  In Field and Service Robotics, pages 123–136. Springer, 2015.
  * [24] Philippe Rauffet, Assaf Botzer, Alexandre Kostenko, Christine Chauvin, Gilles Coppin, et al.  Eye activity measures as indicators of drone operators’ workload and task completion strategies.  In HFES European Chapter, 2016.
  * [25] A Zita Sampaio and Luís Viana.  Virtual reality used as a learning technology: Visual simulation of the construction of a bridge deck.  In 2013 8th Iberian Conference on Information Systems and Technologies (CISTI), pages 1–5. IEEE, 2013.
  * [26] Yangming Shi, Jing Du, and Darrell A Worthy.  The impact of engineering information formats on learning and execution of construction operations: A virtual reality pipe maintenance experiment.  Automation in Construction, 119:103367, 2020.


* Lozano et al. (2009) Lozano, A. C., H. Li, A. Niculescu-Mizil, Y. Liu, C. Perlich, J. Hosking, and N. Abe, 2009: Spatial-temporal causal modeling for climate change attribution. Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining, 587–596.
  * Marbach et al. (2009) Marbach, D., T. Schaffter, C. Mattiussi, and D. Floreano, 2009: Generating realistic in silico gene networks for performance assessment of reverse engineering methods. Journal of computational biology, 16 (2), 229–239.
  * Markus Kalisch et al. (2012) Markus Kalisch, Martin Mächler, Diego Colombo, Marloes H. Maathuis, and Peter Bühlmann, 2012: Causal inference using graphical models with the R package pcalg. Journal of Statistical Software, 47 (11), 1–26, 10.18637/jss.v047.i11.
  * Massmann et al. (2021) Massmann, A., P. Gentine, and J. Runge, 2021: Causal inference for process understanding in earth sciences. arXiv preprint arXiv:2105.00912.
  * Masson-Delmotte et al. (2021) Masson-Delmotte, V., and Coauthors, 2021: Climate change 2021: the physical science basis. Contribution of working group I to the sixth assessment report of the intergovernmental panel on climate change, 2.
  * McDowall et al. (1980) McDowall, D., R. McCleary, and B. J. Bartos, 1980: Interrupted time series analysis. SAGE Publications.
  * Melkas et al. (2021) Melkas, L., R. Savvides, S. H. Chandramouli, J. Mäkelä, T. Nieminen, I. Mammarella, and K. Puolamäki, 2021: Interactive causal structure discovery in earth system sciences. The KDD’21 Workshop on Causal Discovery, PMLR, 3–25.
  * Menchetti et al. (2021) Menchetti, F., F. Cipollini, and F. Mealli, 2021: Estimating the causal effect of an intervention in a time series setting: the c-arima approach. arXiv preprint arXiv:2103.06740.
  * Meng (2019) Meng, Y., 2019: Estimating granger causality with unobserved confounders via deep latent-variable recurrent neural network. arXiv preprint arXiv:1909.03704.
  * Moraffah et al. (2021) Moraffah, R., P. Sheth, M. Karami, A. Bhattacharya, Q. Wang, A. Tahir, A. Raglin, and H. Liu, 2021: Causal inference for time series analysis: Problems, methods and evaluation. Knowledge and Information Systems, 1–45.
  * Mosedale et al. (2006) Mosedale, T. J., D. B. Stephenson, M. Collins, and T. C. Mills, 2006: Granger causality of coupled climate processes: Ocean feedback on the north atlantic oscillation. Journal of climate, 19 (7), 1182–1194.
  * Muñoz-Marí et al. (2020) Muñoz-Marí, J., G. Mateo, J. Runge, and G. Camps-Valls, 2020: Causeme: An online system for benchmarking causal discovery methods. Inpreparation.
  * Naimi et al. (2017) Naimi, A. I., S. R. Cole, and E. H. Kennedy, 2017: An introduction to g methods. International journal of epidemiology, 46 (2), 756–762.
  * National Centers for Environmental Prediction, National Weather Service, NOAA, U.S. Department of Commerce (2015) National Centers for Environmental Prediction, National Weather Service, NOAA, U.S. Department of Commerce, 2015: Ncep north american mesoscale (nam) 12 km analysis. Research Data Archive at the National Center for Atmospheric Research, Computational and Information Systems Laboratory, Boulder CO, URL https://doi.org/10.5065/G4RC-1N91.
  * Nauta et al. (2019a) Nauta, M., D. Bucur, and C. Seifert, 2019a: Causal discovery with attention-based convolutional neural networks. Machine Learning and Knowledge Extraction, 1 (1), 19.
  * Nauta et al. (2019b) Nauta, M., D. Bucur, and C. Seifert, 2019b: Causal discovery with attention-based convolutional neural networks. Machine Learning and Knowledge Extraction, 1 (1), 312–340.
  * Nogueira et al. (2021) Nogueira, A. R., J. Gama, and C. A. Ferreira, 2021: Causal discovery in machine learning: Theories and applications. Journal of Dynamics & Games, 8 (3), 203.
  * Nowack et al. (2020) Nowack, P., J. Runge, V. Eyring, and J. D. Haigh, 2020: Causal networks for climate model evaluation and constrained projections. Nature Communications, 11 (1), 1415, 10.1038/s41467-020-15195-y.
  * Ombadi et al. (2020) Ombadi, M., P. Nguyen, S. Sorooshian, and K.-l. Hsu, 2020: Evaluation of methods for causal discovery in hydrometeorological systems. Water Resources Research, 56 (7), e2020WR027 251.
  * Osler et al. (2006) Osler, M., R. Lund, M. Kriegbaum, U. Christensen, and A.-M. N. Andersen, 2006: Cohort profile: the metropolit 1953 danish male birth cohort. International Journal of Epidemiology, 35 (3), 541–545.
  * Papadogeorgou et al. (2019) Papadogeorgou, G., C. Choirat, and C. M. Zigler, 2019: Adjusting for unmeasured spatial confounding with distance adjusted propensity score matching. Biostatistics, 20 (2), 256–272.
  * Papadogeorgou et al. (2022) Papadogeorgou, G., K. Imai, J. Lyall, and F. Li, 2022: Causal inference with spatio-temporal data: estimating the effects of airstrikes on insurgent violence in iraq. Journal of the Royal Statistical Society Series B: Statistical Methodology, 84 (5), 1969–1999.
  * Papadogeorgou and Samanta (2023) Papadogeorgou, G., and S. Samanta, 2023: Spatial causal inference in the presence of unmeasured confounding and interference. arXiv preprint arXiv:2303.08218.
  * Pearl (1988) Pearl, J., 1988: Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. Morgan Kaufman Publishers, San Mateo, CA.
  * Pearl (2009) Pearl, J., 2009: Causal inference in statistics: An overview.
  * Pellet and Elisseeff (2008) Pellet, J.-P., and A. Elisseeff, 2008: Finding latent causes in causal networks: an efficient approach based on markov blankets. Advances in Neural Information Processing Systems, 21.
  * Peters et al. (2013) Peters, J., D. Janzing, and B. Schölkopf, 2013: Causal inference on time series using restricted structural equation models. Advances in Neural Information Processing Systems, 26.
  * Pörtner et al. (2022) Pörtner, H.-O., and Coauthors, 2022: Ipcc, 2022: Summary for policymakers.
  * Qu et al. (2021) Qu, Y., C. Montzka, and H. Vereecken, 2021: Causation discovery of weather and vegetation condition on global wildfire using the pcmci approach. 2021 IEEE International Geoscience and Remote Sensing Symposium IGARSS, IEEE, 8644–8647.
  * Ramsey et al. (2018) Ramsey, J. D., and Coauthors, 2018: Tetrad—a toolbox for causal discovery. 8th international workshop on climate informatics.
  * Reich et al. (2021) Reich, B. J., S. Yang, Y. Guan, A. B. Giffin, M. J. Miller, and A. Rappold, 2021: A review of spatial causal inference methods for environmental and epidemiological applications. International Statistical Review, 89 (3), 605–634.
  * Robins et al. (2000) Robins, J. M., M. A. Hernan, and B. Brumback, 2000: Marginal structural models and causal inference in epidemiology. Epidemiology, 550–560.
  * Rubin (2005) Rubin, D. B., 2005: Causal inference using potential outcomes: Design, modeling, decisions. Journal of the American Statistical Association, 100 (469), 322–331.
  * Runge et al. (2023) Runge, J., A. Gerhardus, G. Varando, V. Eyring, and G. Camps-Valls, 2023: Causal inference for time series. Nature Reviews Earth & Environment, 4 (7), 487–505.
  * Runge et al. (2019a) Runge, J., P. Nowack, M. Kretschmer, S. Flaxman, and D. Sejdinovic, 2019a: Detecting and quantifying causal associations in large nonlinear time series datasets. Science advances, 5 (11), eaau4996.
  * Runge et al. (2015) Runge, J., and Coauthors, 2015: Identifying causal gateways and mediators in complex spatio-temporal systems. Nature communications, 6 (1), 1–10.
  * Runge et al. (2019b) Runge, J., and Coauthors, 2019b: Inferring causation from time series in Earth system sciences. Nat Commun, 10 (1), 2553, 10.1038/s41467-019-10105-3.
  * Scheines (1997) Scheines, R., 1997: An introduction to causal inference.
  * Sharma and Kiciman (2020) Sharma, A., and E. Kiciman, 2020: Dowhy: An end-to-end library for causal inference. arXiv preprint arXiv:2011.04216.
  * Sheppard et al. (2023) Sheppard, K., and Coauthors, 2023: bashtage/linearmodels: Release 5.3. Zenodo, URL https://doi.org/10.5281/zenodo.8379085, 10.5281/zenodo.8379085.
  * Sheth et al. (2022) Sheth, P., R. Shah, J. Sabo, K. S. Candan, and H. Liu, 2022: Stcd: A spatio-temporal causal discovery framework for hydrological systems. 2022 IEEE International Conference on Big Data (Big Data), IEEE, 5578–5583.
  * Shimizu et al. (2006) Shimizu, S., P. O. Hoyer, A. Hyvärinen, A. Kerminen, and M. Jordan, 2006: A linear non-gaussian acyclic model for causal discovery. Journal of Machine Learning Research, 7 (10).
  * Smirnov and Mokhov (2009) Smirnov, D. A., and I. I. Mokhov, 2009: From granger causality to long-term causality: Application to climatic data. Physical Review E, 80 (1), 016 208.
  * Smith and Pun (2013) Smith, G., and A. Pun, 2013: How does earth work? physical geology and the process of science: Pearson new international edition. Pearson Higher Ed.
  * Smith et al. (2011) Smith, S. M., K. L. Miller, G. Salimi-Khorshidi, M. Webster, C. F. Beckmann, T. E. Nichols, J. D. Ramsey, and M. W. Woolrich, 2011: Network modelling methods for fmri. Neuroimage, 54 (2), 875–891.
  * Spirtes (2010) Spirtes, P., 2010: Introduction to causal inference. Journal of Machine Learning Research, 11 (5).
  * Spirtes et al. (2000) Spirtes, P., C. N. Glymour, R. Scheines, and D. Heckerman, 2000: Causation, prediction, and search. MIT press.
  * Stone (2004) Stone, J. V., 2004: Independent component analysis: a tutorial introduction.
  * Thams et al. (2022) Thams, N., R. Søndergaard, S. Weichwald, and J. Peters, 2022: Identifying causal effects using instrumental time series: Nuisance iv and correcting for the past. arXiv preprint arXiv:2203.06056.
  * Tian and Pearl (2013) Tian, J., and J. Pearl, 2013: Causal discovery from changes. arXiv preprint arXiv:1301.2312.
  * Tibau et al. (2022) Tibau, X.-A., C. Reimers, A. Gerhardus, J. Denzler, V. Eyring, and J. Runge, 2022: A spatiotemporal stochastic climate model for benchmarking causal discovery methods for teleconnections. Environmental Data Science, 1, e12.
  * Triantafillou and Tsamardinos (2016) Triantafillou, S., and I. Tsamardinos, 2016: Score-based vs constraint-based causal learning in the presence of confounders. Cfa@ uai, 59–67.
  * Van Nes et al. (2015) Van Nes, E. H., M. Scheffer, V. Brovkin, T. M. Lenton, H. Ye, E. Deyle, and G. Sugihara, 2015: Causal feedbacks in climate change. Nature Climate Change, 5 (5), 445–448.
  * VanderWeele (2009) VanderWeele, T. J., 2009: Marginal structural models for the estimation of direct and indirect effects. Epidemiology, 18–26.
  * VanderWeele et al. (2011) VanderWeele, T. J., L. C. Hawkley, R. A. Thisted, and J. T. Cacioppo, 2011: A marginal structural model analysis for loneliness: implications for intervention trials and clinical practice. Journal of consulting and clinical psychology, 79 (2), 225.
  * Varando et al. (2021) Varando, G., M.-A. Fernández-Torres, and G. Camps-Valls, 2021: Learning granger causal feature representations. ICML 2021 Workshop on Tackling Climate Change with Machine Learning.
  * Vowels et al. (2022) Vowels, M. J., N. C. Camgoz, and R. Bowden, 2022: D’ya like dags? a survey on structure learning and causal discovery. ACM Computing Surveys, 55 (4), 1–36.
  * Weichwald et al. (2020) Weichwald, S., M. E. Jakobsen, P. B. Mogensen, L. Petersen, N. Thams, and G. Varando, 2020: Causal structure learning from time series: Large regression coefficients may predict causal links better in practice than small p-values. PMLR, Proceedings of the NeurIPS 2019 Competition and Demonstration Track, Proceedings of Machine Learning Research, Vol. 123, 27–36, URL http://proceedings.mlr.press/v123/weichwald20a.html.
  * Weinberger (2018) Weinberger, N., 2018: Faithfulness, coordination and causal coincidences. Erkenntnis, 83 (2), 113–133.
  * Williamson and Ravani (2017) Williamson, T., and P. Ravani, 2017: Marginal structural models in clinical research: when and how to use them? Nephrology Dialysis Transplantation, 32 (suppl_2), ii84–ii90.
  * Wu et al. (2023) Wu, J., Y. Zhou, H. Wang, X. Wang, and J. Wang, 2023: Assessing the causal effects of climate change on vegetation dynamics in northeast china using convergence cross-mapping. IEEE Access.
  * Yang et al. (2022) Yang, X., Z.-H. Wang, C. Wang, and Y.-C. Lai, 2022: Detecting the causal influence of thermal environments among climate regions in the united states. Journal of Environmental Management, 322, 116 001.
  * Yao et al. (2021) Yao, L., Z. Chu, S. Li, Y. Li, J. Gao, and A. Zhang, 2021: A survey on causal inference. ACM Transactions on Knowledge Discovery from Data (TKDD), 15 (5), 1–46.
  * Ye et al. (2015) Ye, H., E. R. Deyle, L. J. Gilarranz, and G. Sugihara, 2015: Distinguishing time-delayed causal interactions using convergent cross mapping. Scientific reports, 5 (1), 14 750.
  * Zhang and Ning (2023) Zhang, W., and K. Ning, 2023: Spatiotemporal heterogeneities in the causal effects of mobility intervention policies during the covid-19 outbreak: A spatially interrupted time-series (sits) analysis. Annals of the American Association of Geographers, 113 (5), 1112–1134.
  * Zheng et al. (2023) Zheng, Y., and Coauthors, 2023: Causal-learn: Causal discovery in python. arXiv preprint arXiv:2307.16405.
  * Zhou et al. (2014) Zhou, D., Y. Zhang, Y. Xiao, and D. Cai, 2014: Analysis of sampling artifacts on the granger causality analysis for topology extraction of neuronal dynamics. Frontiers in computational neuroscience, 8, 75.
  * Zhu et al. (2017) Zhu, J. Y., C. Zhang, H. Zhang, S. Zhi, V. O. Li, J. Han, and Y. Zheng, 2017: pg-causality: Identifying spatiotemporal causal pathways for air pollutants with urban big data. IEEE Transactions on Big Data, 4 (4), 571–585.


Consider a signed graph $\Gamma^{1}=(G_{1},\sigma_{1},\mu_{1})$ on $n_{1}$
vertices and $m_{1}$ edges. Let $\theta$ be any $r$-orientation of edges of
$\Gamma^{1}$ and $\Gamma^{2}=(G_{2},\sigma_{2},\mu_{2})$ be $\gamma$-regular
signed graph on $n_{2}$ vertices then the characteristic polynomial of
$P(\Gamma^{1}\boxminus\Gamma^{2})_{\theta}$ is

$\displaystyle f_{P(\Gamma^{1}\boxminus\Gamma^{2})_{\theta}}(\lambda)$
$\displaystyle=$
$\displaystyle\lambda^{m_{1}-n_{1}}.\prod_{j=1}^{n_{2}}\left[\lambda-\frac{\gamma}{\gamma+2}\lambda_{j}(P(\Gamma^{2}))\right].\prod_{j=1}^{n_{1}}\left[\lambda^{2}-\frac{1}{2(n_{2}+1)}\left(1+2\Sigma_{A(\Gamma^{2})}(\gamma\lambda+2\lambda)\right)(1+\lambda_{j}(P(\Gamma^{1})))\right]$

###### Proof.

Proof is similar to Theorem 5.7 ∎

######  Corollary 5.11.

Consider a signed graph $\Gamma^{1}=(G_{1},\sigma_{1},\mu_{1})$ on $n_{1}$
vertices and $m_{1}$ edges. Let $\theta$ be any $r$-orientation of edges of
$\Gamma^{1}$ and $\Gamma^{2}=(G_{2},\sigma_{2},\mu_{2})$ be $\gamma$-regular
signed graph on $n_{2}$ vertices then the characteristic polynomial of
$\mathbb{L}(\Gamma^{1}\boxminus\Gamma^{2})_{\theta}$ is

$\begin{split}f_{\mathbb{L}(\Gamma^{1}\boxminus\Gamma^{2})_{\theta}}(\lambda)=&(1-\lambda)^{m_{1}-n_{1}}.\prod_{j=1}^{n_{2}}\left[\frac{2+\gamma\lambda_{j}(\mathbb{L}(\Gamma^{2}))}{\gamma+2}-\lambda\right].\prod_{j=1}^{n_{1}}\bigg{[}(1-\lambda)^{2}-\frac{1}{2(n_{2}+1)}\\\
&\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\
\left(1+2\Sigma_{A(\Gamma^{2})}(\gamma+2-\gamma\lambda-2\lambda)\right)(2-\lambda_{j}(\mathbb{L}(\Gamma^{1})))\bigg{]}\end{split}$

######  Corollary 5.12.

Let $\Gamma^{1}$ be any signed graph and $\theta_{1}$ and $\theta_{2}$ be any
$r$-orientations of edges of $\Gamma^{1}$. If $\Gamma^{2}$ and $\Gamma^{3}$
are two $\gamma$-regular $\mathbb{L}$-cospectral signed graphs such that
$\Sigma_{A(\Gamma^{2})}(\lambda)=\Sigma_{A(\Gamma^{3})}(\lambda)$ then
$(\Gamma^{1}\boxminus\Gamma^{2})_{\theta_{1}}$ and
$(\Gamma^{1}\boxminus\Gamma^{3})_{\theta_{2}}$ are $\mathbb{L}$-cospectral.

## Disclosure statement

No potential conflict of interest.

## Acknowledgements

We would like to acknowledge National Institute of Technology Sikkim for
giving doctoral fellowship to Satyam Guragain and Bishal Sonar.

## References

  * [1] Bibhas Adhikari, Amrik Singh, and Sandeep Kumar Yadav.  Corona product of signed graphs and its application to modeling signed networks.  Discrete Mathematics, Algorithms and Applications, 15(01):2250062, 2023.
  * [2] Ravindra B Bapat.  Graphs and matrices, volume 27.  Springer, 2010.
  * [3] Francesco Belardo and Slobodan K Simić.  On the laplacian coefficients of signed graphs.  Linear Algebra and its Applications, 475:94–113, 2015.
  * [4] Haiyan Chen and Liwen Liao.  The normalized laplacian spectra of the corona and edge corona of two graphs.  Linear and Multilinear Algebra, 65(3):582–592, 2017.
  * [5] Shu-Yu Cui and Gui-Xian Tian.  The spectrum and the signless laplacian spectrum of coronae.  Linear algebra and its applications, 437(7):1692–1703, 2012.
  * [6] Arpita Das and Pratima Panigrahi.  Normalized laplacian spectrum of some subdivision-coronas of two regular graphs.  Linear and Multilinear Algebra, 65(5):962–972, 2017.
  * [7] Jerrold W Grossman, Devadatta M Kulkarni, and Irwin E Schochetman.  Algebraic graph theory without orientation.  Linear Algebra and its Applications, 212:289–307, 1994.
  * [8] Frank Harary.  On the notion of balance of a signed graph.  Michigan Mathematical Journal, 2(2):143–146, 1953.
  * [9] Yaoping Hou, Jiongsheng Li, and Yongliang Pan.  On the laplacian eigenvalues of signed graphs.  Linear and Multilinear Algebra, 51(1):21–30, 2003.
  * [10] Yaoping Hou, Zikai Tang, and Dijian Wang.  On signed graphs with just two distinct laplacian eigenvalues.  Applied Mathematics and Computation, 351:1–7, 2019.
  * [11] Jérôme Kunegis, Stephan Schmidt, Andreas Lommatzsch, Jürgen Lerner, Ernesto W De Luca, and Sahin Albayrak.  Spectral analysis of signed graphs for clustering, prediction and visualization.  In Proceedings of the 2010 SIAM international conference on data mining, pages 559–570. SIAM, 2010.
  * [12] Xiaogang Liu and Pengli Lu.  Spectra of subdivision-vertex and subdivision-edge neighbourhood coronae.  Linear algebra and its applications, 438(8):3547–3559, 2013.
  * [13] Cam McLeman and Erin McNicholas.  Spectra of coronae.  Linear algebra and its applications, 435(5):998–1007, 2011.
  * [14] Nutan G Nayak.  On net-regular signed graphs.  Infinite Study, 2016.
  * [15] D Stephen G Pollock.  Tensor products and matrix differential calculus.  Linear Algebra and its Applications, 67:169–193, 1985.
  * [16] K Shahul Hameed, Viji Paul, and KA Germina.  On co-regular signed graphs.  Australas J Combin, 62(1):8–17, 2015.
  * [17] Amrik Singh, Ravi Srivastava, Bibhas Adhikari, and Sandeep Kumar Yadav.  Structural balance and spectral properties of generalized corona product of signed graphs.  arXiv preprint arXiv:2310.08057, 2023.
  * [18] Thomas Zaslavsky.  Orientation of signed graphs.  European Journal of Combinatorics, 12(4):361–375, 1991.
  * [19] Thomas Zaslavsky.  Orientation embedding of signed graphs.  Journal of graph theory, 16(5):399–422, 1992.
  * [20] Thomas Zaslavsky.  Matrices in the theory of signed simple graphs.  arXiv preprint arXiv:1303.3083, 2013.


######  Lemma 5.

Assume that $y=\left(\rho v,\rho e_{t},C_{1},\ldots,C_{n_{s}}\right)^{T}$ is
in $\mathcal{G}$. Then $\check{y}=y-\alpha^{-1}\Delta\mathcal{F}$, where
$\alpha>0$, is also in $\mathcal{G}$ under the following conditions:

$\alpha>\alpha^{*}\left(y,\Delta\mathcal{F}\right)=\max\left\\{\left.\alpha_{T}\right|_{\left(y,\Delta\mathcal{F}\right)},0\right\\},$
(B.1)

where

$\alpha_{T}=\begin{cases}\frac{-\mathsf{b}+\sqrt{\mathsf{b}^{2}-4\rho^{2}u\mathsf{g}}}{2\rho^{2}u},&\mathsf{b}^{2}-4\rho^{2}u\mathsf{g}\geq
0\\\ 0,&\mathrm{otherwise}\end{cases},$ (B.2)

$\mathsf{b}=-\rho\Delta\mathcal{F}_{\rho e_{t}}+\rho
v\cdot\Delta\mathcal{F}_{\rho v}$, and
$\mathsf{g}=-\frac{1}{2}\left|\Delta\mathcal{F}_{\rho v}\right|^{2}$.

###### Proof.

$\check{y}=y-\alpha^{-1}\Delta\mathcal{F}$ can be expanded as

$\begin{split}\check{y}=&\left(\check{\rho v},\check{\rho
e_{t}},\check{C_{1}},\ldots,\check{C}_{n_{s}}\right)^{T}\\\ =&\left(\rho
v-\alpha\Delta\mathcal{F}_{\rho v},\rho e_{t}-\alpha\Delta\mathcal{F}_{\rho
e_{t}},C_{1}-\alpha\Delta\mathcal{F}_{C_{1}},\ldots,C_{n_{s}}-\alpha\Delta\mathcal{F}_{C_{n_{s}}}\right)^{T}.\end{split}$

Since $\Delta\mathcal{F}_{C_{i}}=0,i=1,\ldots,n_{s}$, we have
$\check{C_{i}}=C_{i}\geq 0$. Density also remains unchanged.

Next, we focus on positivity of temperature. For a given $y=\left(\rho v,\rho
e_{t},C_{1},\ldots,C_{n_{s}}\right)^{T}$, let $Z(y)$ be defined as

$Z(y)=\rho^{2}u^{*}(y)=\rho(y)\rho e_{t}-\left|\rho
v\right|^{2}/2-\rho^{2}u_{0}(y).$ (B.3)

Note that if $Z(y)>0$, then $T(y)>0$. $Z\left(\check{y}\right)$ can be
expressed as

$\displaystyle Z\left(y-\alpha^{-1}\Delta\mathcal{F}\right)=$
$\displaystyle\sum_{i=1}^{n_{s}}W_{i}\left(C_{i}-\alpha^{-1}\Delta\mathcal{F}_{C_{i}}\right)\left(\rho
e_{t}-\alpha^{-1}\Delta\mathcal{F}_{\rho e_{t}}\right)$
$\displaystyle-\frac{1}{2}\left|\rho v-\alpha^{-1}\Delta\mathcal{F}_{\rho
v}\right|^{2}-\left[\sum_{i=1}^{n_{s}}W_{i}\left(C_{i}-\alpha^{-1}\Delta\mathcal{F}_{C_{i}}\right)\right]^{2}u_{0},$

which, after multiplying both sides by $\alpha^{2}$, can be rewritten as

$\displaystyle\alpha^{2}Z\left(y-\alpha^{-1}\Delta\mathcal{F}\right)=$
$\displaystyle\rho^{2}u\alpha^{2}-\mathsf{b}\alpha+\mathsf{g},$ (B.4)

where $\mathsf{b}=-\rho e_{t}\mathsf{M}-\rho\Delta\mathcal{F}_{\rho
e_{t}}+\rho v\cdot\Delta\mathcal{F}_{\rho v}+2\rho u_{0}\mathsf{M}$,
$\mathsf{g}=\mathsf{M}\Delta\mathcal{F}_{\rho
e_{t}}-\frac{1}{2}\left|\Delta\mathcal{F}_{\rho
v}\right|^{2}-u_{0}\mathsf{M}^{2}$, and
$\mathsf{M}=\sum_{i=1}^{n_{s}}W_{i}\Delta\mathcal{F}_{C_{i}}$. Since
$\mathsf{M}=0$, $\mathsf{b}$ and $\mathsf{g}$ reduce to
$\mathsf{b}=-\rho\Delta\mathcal{F}_{\rho e_{t}}+\rho
v\cdot\Delta\mathcal{F}_{\rho v}$ and
$\mathsf{g}=-\frac{1}{2}\left|\Delta\mathcal{F}_{\rho v}\right|^{2}$. Setting
the RHS of Equation (B.4) equal to zero yields a quadratic equation with
$\alpha$ as the unknown. Since $\rho^{2}u$ is positive, the quadratic equation
is convex. As such, if $\mathsf{b}^{2}-4\rho^{2}u\mathsf{g}<0$, then no real
roots exist, and $Z\left(y-\alpha^{-1}\Delta\mathcal{F}\right)>0$ for all
$\alpha\neq 0$; otherwise, at least one real root exists, in which case a
sufficient condition to ensure
$Z\left(y-\alpha^{-1}\Delta\mathcal{F}\right)>0$ is $\alpha>\alpha_{0}$, where
$\alpha_{0}$ is given by

$\alpha_{0}=\max\left\\{\frac{-\mathsf{b}+\sqrt{\mathsf{b}^{2}-4\rho^{2}u\mathsf{g}}}{2\rho^{2}u},0\right\\}.$

∎