Title SpeakEasy Course CS/QTM/LING-329: Computational Linguistics Authors
Hyunbae Jeon, BS in Computer Science<EMAIL_ADDRESS>Rhea Ramachandran,
BS in Computer Science<EMAIL_ADDRESS>Victoria Ploerer, BA in
Computer Science BS in Neuroscience<EMAIL_ADDRESS>Yella Diekmann, BS in
Computer Science<EMAIL_ADDRESS>Max Bagga, BA in Computer Science,
<EMAIL_ADDRESS>
## Abstract

Social interactions and conversation skills separate the successful from the
rest and the confident from the shy. For college students in particular, the
ability to converse can be an outlet for the stress and anxiety experienced on
a daily basis along with a foundation for all-important career skills. In
light of this, we designed SpeakEasy: a chatbot with some degree of
intelligence that provides feedback to the user on their ability to engage in
free-form conversations with the chatbot. SpeakEasy attempts to help college
students improve their communication skills by engaging in a seven-minute
spoken conversation with the user, analyzing the user’s responses with metrics
designed based on previous psychology and linguistics research, and providing
feedback to the user on how they can improve their conversational ability. To
simulate natural conversation, SpeakEasy converses with the user on a wide
assortment of topics that two people meeting for the first time might discuss:
travel, sports, and entertainment. Unlike most other chatbots with the goal of
improving conversation skills, SpeakEasy actually records the user speaking,
transcribes the audio into tokens, and uses macros—e.g., sequences that
calculate the pace of speech, determine if the user has an over-reliance on
certain words, and identifies awkward transitions—to evaluate the quality of
the conversation. Based on the evaluation, SpeakEasy provides elaborate
feedback on how the user can improve their conversations. In turn, SpeakEasy
updates its algorithms based on a series of questions that the user responds
to regarding SpeakEasy’s performance. The source code for SpeakEasy can be
found here: https://github.com/HarryJeon24/SpeakEasy.git. See README for
detailed instructions about setup.

##  1 Team Vision

Communication is essential in all facets of life. The way people talk, express
themselves, and share ideas is at the forefront of our daily lives. Humans are
a social species, which is why communicating effectively is key to maintaining
relationships, getting a job, and increasing self-confidence. For instance,
95.9% of employers found written/oral communication essential for jobs, but
estimated that a mere 41.6% of employees actually possessed this skill [17]
Furthermore, research suggests that improving communication and social skills
can help alleviate development of social anxiety [11]. This is where SpeakEasy
comes into play.

###  1.1 Overall Objective

The goal of SpeakEasy is to help people improve their communication skills
through researched-backed metrics and tips. A dialogue system is perfect to
achieve this goal, as it provides the perfect avenue for the user to practice
and receive feedback on their communication without the social pressure of
interacting with another human being. SpeakEasy first engages in a friendly
conversation with the user. On top of providing great practice for small talk
in a safe environment, this dialogue is used to collect information about how
the user communicates and which areas can be strengthened. Such information
can only be collected through a dialogue system. Then, the chatbot presents
constructive feedback that the user can easily implement into future
conversations. After using SpeakEasy, users will be prepared to build stronger
relationships, land and keep jobs more easily, and build their confidence. As
such, the overarching goal of SpeakEasy is to improve the user’s overall
quality of life.

###  1.2 Target Audience

The target audience for our system is college students, both males, and
females. We decided to cater to this age group because communication skills
are essential for personal growth and development. However, many individuals
struggle to express themselves due to social anxiety or other speaking
disorders. This has especially been heightened through the pandemic with 88%
of students suffering from stress, 44% from anxiety, and 36 % from severe
depression [7]. While an inability to communicate can be attributed to these
disorders, Lee et al. [7] claims that lack of communication can also be one of
the determinants of the aforementioned disorders. In other words, the
inability to communicate can cause anxiety and other disorders which in turn
worsens the student’s communicative skills: a vicious cycle. With this in
mind, we believed we optimized SpeakEasy’s impact by designing it for college
students.

An individual’s experience in college often determines the course of the rest
of their academic and personal lives. Thus, by catering our system to college
students, we can help them become more confident in themselves and their
abilities as they navigate different relationships, job opportunities, and
personal growth. Individuals who can communicate effectively can express their
feelings, needs, and ideas clearly. This, in turn, can improve their
relationships and overall well-being. Studies evaluating students before,
during, and after university experience has established college as the most
profound time in a person’s life in which they can learn cooperative skills
[8]. If learned successfully, this cooperative methodology serves as a
building block from which future corporate and office skills develop.
Consequently, it is imperative that college students successfully develop the
communication skills necessary during the short time they have in university
and as a result, SpeakEasy likely has its most predominant impact when used by
college students.

We specifically catered to university students by incorporating conversational
topics that appeal to college students. Although we directed the conversation
to engage with college students, everyone can benefit from interacting with
our system. The ability to make conversion is a universal skill that people
continue to improve on throughout the course of their lives.

##  2 Challenges

The main challenge with developing SpeakEasy was creating a chatbot that
understood questions the user may ask. With the nature of SpeakEasy, it is
expected that the user will try to have an engaging conversation since they
know their conversational abilities will be judged. The best way to increase
engagement is to ask questions – it is one of the quantitative metrics
measured, after all. However, predicting every possible question the user may
ask and generating a response is near impossible. To overcome this challenge,
SpeakEasy has opinions and a personality. For example, this chatbot wants to
live in Barcelona and loves the beach. Adding depth to SpeakEasy allows it to
answer personal questions the user may ask, increasing engagement and
interactivity. However, there may still be instances where the user’s
questions go unanswered. To guarantee that these instances were not negatively
impacting the user experience too much, one aspect of the evaluation included
how natural the conversation felt. Ideally, SpeakEasy is able to acknowledge
what the user is saying and answer enough questions to resemble real human
conversation.

Furthermore, understanding human emotion became a challenge as well. To ensure
that our conversational AI operated fast enough to simulate as much of a
proper conversation as possible, we attempted to minimize the amount of GPT
API calls. Consequently, we could not always flag the emotion of the user’s
input using the GPT API. Unfortunately, this results in some instances where
SpeakEasy responds positively to something clearly negative which would not
happen in an actual conversation and actually detracts from the natural nature
of the conversation. We attempted to circumnavigate this by designing a
dialogue flow that could have more neutral ts response. Because the entire aim
of our project is to help the user improve their ability to converse in a
realistic environment, maintaining the realism of our conversation was the
most pertinent challenge. With this in mind, we attempted to ensure that the
flow of the conversation was as seamless as possible: i.e., no abrupt
transitions to other topics or off-topic dialogue; however, since we cannot
account for all possible user inputs, ensuring that this was the case was no
easy task. To try to prevent against making abrupt transitions, we tried to
preempt transitions to other topics by priming the user with questions that
could generate user responses that relate to the topic SpeakEasy intends to
switch to.

A technical challenge arose when using Natex with the speech to text
technology. These two technologies are incompatible, so regular expressions
embedded within Macros were used to overcome this. Other technical challenges
came from the differential operating systems of Windows and Mac. For example,
the string inputs for the systems calls accessing the operating system and
thus the speakers differ for each version. Consequently, we had to develop
different versions for each kind of system. Additionally, package installation
is far more difficult on Mac as certain packages require complicated
procedures in order to properly install the right version. Background noise
also played a huge technical challenge. If they system cannot properly pick up
the audio from the background, then the string input for the GPT API calls can
often passed in garbled text—e.g., if it is noisy in the background, sometimes
characters and words from languages are picked up which force the GPT to throw
errors which forces it to terminate early. As a result, we tried to ensure
that some of the error statements force the dialogue flow to ask further
questions as opposed to simply terminating the code. Finally, SpeakEasy relies
on the GPT API to process what the user is saying. Though this increases its
language understanding capabilities immensely, the GPT API experiences major
slowdowns at times which negatively impacts user experience. Thankfully, the
use of an API key mitigated this negative effect.

##  3 Dialogue Overview

Figure 1: A diagram of the dialogue flow

The chatbot’s dialogue flow creates a natural conversation that lasts for at
least seven minutes (averages around ten minutes) and covers topics related to
travel, health, and entertainment. The general idea is that two strangers meet
for the first time and create small talk. Therefore, the dialogue will start
with questions and phrases such as ”Hello, what is your name?”, ”My name is…”,
”How are you doing?”, ”I am doing…”. In the case that the person is a
returning user, the system conducts a different greeting that incorporates a
short evaluation of the user’s prior experience with the bot. After the
introduction, the chatbot guides the conversation into talking about health.
We wanted to design the flow such that the user does not feel forced to talk
about the topics; therefore, the chatbot uses the guiding questions “Is it
part of your everyday routine?” and “Do you make it part of your daily
schedule?” to transition into talking about health. During the conversation
about health, the chatbot asks about and discusses the user’s eating habits,
workout routine, and work-life balance. Following these topics, the chatbot
stirs the conversation either towards talking about entertainment or about
travel. The latter transition, for example, is done by pointing out the
difficulties of maintaining a healthy diet while traveling.

During the travel conversation, the chatbot asks the user questions such as
“Where are you from?”, ”Where would you rather live?” and ”Are there any other
places you would like to live in or visit?”. Furthermore, the chatbot asks the
user for recommendations on what to do in specific travel locations and, if
they prefer, would ever consider solo travel. The chatbot is also able to
share information about its persona. For example, the chatbot is able to tell
the user that it is from Austria and that it wants to live in Barcelona. At
the end of this conversation, the chatbot enters the feedback state of the
conversation.

When talking about entertainment, the chatbot is able to converse about movies
and music. It first asks the user if they have a favorite movie or song, and
if so, ask questions about it. For example, “Who is your favorite artist for
that type of music?”, “What genre is it”?, and “Who is your favorite character
in the movie?”. The chatbot then moves on to recommending a song or movie to
the user. If the user is satisfied, the chatbot either transitions over to
talking about travel or it will end the conversation and enter the feedback
state of the conversation. The travel state can transition back over to the
movie state in the event that a concept from a movie is brought up—e.g., did
you know that this movie was filmed here—in order to prevent the conversation
from being fully linear.

During the feedback state, the chatbot provides information about the user’s
conversation performance and previously determined metrics, such as how many
questions the user asks, word choice, talking speed, and the level of
attention that the user gives to their conversation partner. It also asks the
user if it wants to hear the underlying metrics of the analysis: e.g., the
specific statistics we incorporated from previous research in order to
evaluate the conversation.

Lastly, the chatbot asks the user for feedback on its performance. The user is
be able to do so by answering questions like “Did I say anything wrong during
the conversation?” and “Did you ever have a hard time understanding me?” in
order to assess the chatbots performance. These questions were chosen in line
to the evidence presented in section 5.

Figure 2: Sample dialogue conducted by the chatbot

Figure 2 provides a sample dialogue conducted by the chatbot during the
entertainment dialogue flow. Because the user has a favorite movie, the
chatbot proceeds to ask questions about the movie. After some time, the
chatbot ends the conversation and proceed to giving feedback to the user about
their performance.

##  4 Methodology

For the conversation framework, we are using EmoraSTDM [3], which allows
developers to add custom macros and ontologies. One of the built-in features
in EmoraSTDM is Natex, which functions similarly to regular expressions.
However, we chose not to utilize Natex in our implementation. Instead, we
relied on the GPT API [10] to understand user input. It is important to note
that we did not use GPT to generate responses in this system. To improve
conversational skills, we aimed to mimic a human conversation environment as
closely as possible. Given that more people experience anxiety when talking
rather than texting [13], we decided to implement speech-to-text and text-to-
speech macros.

###  4.1 Audio Function

In this project, we aimed to develop a chatbot capable of taking audio input
and producing audio output. To achieve this, we utilized OpenAI’s Whisper
Automatic Speech Recognition (ASR) API [12] for transcribing audio to text and
the gTTS (Google Text-to-Speech) package in Python for converting text to
audio. This section details the scientific approach and methods employed in
creating the chatbot. To enable the chatbot to accept audio input, we used the
PyAudio library in Python to record the user’s voice. The following parameters
were used for recording:

  * •

Format: 16-bit integer (paInt16)

  * •

Channels: 1 (mono)

  * •

Sample rate: 44,100 Hz

  * •

Chunk size: 1,024 samples

The audio recording was saved as a WAV file, and a separate thread was created
to handle the recording process. This allowed users to stop the recording by
pressing the Enter key. The duration of the recording was calculated by
measuring the time elapsed between the start and end of the recording process.
After obtaining the audio input, we utilized OpenAI’s Whisper API to
transcribe the audio to text. The API was called with the Whisper ASR model
”whisper-1” and the recorded audio file. The API returned a transcript of the
audio, which was then assigned to a variable for further processing. For text-
to-audio conversion, we employed the gTTS package in Python. The gTTS library
converts text to audio using Google’s Text-to-Speech API. To generate audio
output, we passed the chatbot’s response text to the gTTS function, specifying
the following parameters:

  * •

Text: The chatbot’s response

  * •

Language: English (en)

The gTTS function saved the synthesized audio as an MP3 file. To play the
audio output, we used the ”os.system()” function in Python. For Windows, we
executed the ”start” command, followed by the MP3 file’s name, and used the
”time.sleep()” function to pause the execution of the script for the duration
of the audio playback, which was determined using the MP3 library to extract
the length of the audio file. For macOS, we executed the ”afplay” command,
followed by the MP3 file’s name, without the need for a sleep function. The
chatbot’s workflow was as follows:

  1. 1.

Record the user’s audio input using the MacroRecordAudio class.

  2. 2.

Transcribe the audio input to text using OpenAI’s Whisper API.

  3. 3.

Process the text input and generate a response using the chatbot’s logic.

  4. 4.

Convert the text response to audio using the MacrogTTS class.

  5. 5.

Play the audio output for the user.

This approach allowed us to create a chatbot capable of interacting with users
through audio input and output, offering a more natural and accessible user
experience.

###  4.2 Transitions

While creating the introduction state, we used GPT to get the user’s name.
From there we store all user names as a key in the USERS dictionary which we
use to check whether the user has previously used the system. If they are a
returning user, we store their rating for their previous interaction with the
bot as well as the rest of their feedback later in the dialogue flow. We also
use GPT for retrieving information like how the user is doing and what they
have been up to. In addition to this, we use a variety of GET and SET macros
to store GPT responses in various variables which we can use to retrieve the
information later through our .pkl file.

For the Health transition, we again utilize GPT as well as several GET and SET
macros to store information about the user’s health (i.e. the user’s
lifestyle, whether or not they exercise, their eating habits, etc). We also
utilized IF macros to determine whether or not a user indicated yes or no to a
particular question.

The Entertainment section works in a similar way, using GTP and GET/SET macros
to save user information. It also utilizes the Spotify API for song
suggestions and the The Movie Database (TMDB) API for movie recommendations.

Once again, GTP and macros was utilized to build the Travel portion of the
conversation. SpeakEasy prompts the user to talk about where they are from and
where they have traveled or want to travel. Beyond this, SpeakEasy can discuss
favorite activities at these locations and the user’s specific travel habits.
To be able to acknowledge what the user is saying, many SET and GET macros
were employed to store and retrieve responses. GTP was only used to process
what the user is saying, not to generate responses. Beyond this, SpeakEasy’s
personality shines through in the travel portion. We made this chatbot have a
home country, and even specific travel wishes.

###  4.3 Feedback

A set of macros is designed to help SpeakEasy provide advice to users by
evaluating their conversational skills. We focus on five aspects for
evaluation: awkward transition, verbal tic, acknowledgment, number of
questions, and number of words per minute. This evaluation matrix is based on
three papers [4], [16], [15].

The work by Garrels [4] highlights the importance of intonation, smooth
transition, and asking questions for good conversation. Based on these
findings, we created the MacroAwkward, MacroNumQuestions, and MacroAVGToken
classes. The MacroAwkward class checks for awkward transitions by comparing
the user’s transitional words to a predefined list of awkward transitional
words, which were generated by ChatGPT for suggestion and can be expanded [4].
If the user uses less than 10 awkward transitions, they are considered to be
doing well. The MacroNumQuestions class counts the number of question marks in
user responses and advises the user to ask more questions if the ratio of
questions to total utterances is less than 0.39, as this was the high question
rate that showed positive effects on liking towards the conversation partner
[5]. The MacroAVGToken class measures the user’s intonation by calculating the
words per minute (WPM) and providing appropriate feedback based on a range of
120 to 150 WPM, which is the average American talking speed in conversation
according to the cited source [2].

Research by Russell et al. [16] demonstrates the negative impact of overusing
specific words. To address this aspect, we created the MacroTic class, which
calculates the frequency of each token the user used, excluding common words
like articles. The class provides feedback on the overuse of certain words
based on the sorted frequency list of tokens.

Experiments conducted by Rubin and Martin [15] reveal the importance of
acknowledging and showing empathy towards the speaking partner. To measure
these elements, we implemented the MacroAcknow class. One way to show
acknowledgment and empathy is by mentioning what the speaking partner said
previously. The MacroAcknow class calculates the language style matching (LSM)
using a function words list commonly used in conversation, comparing the
user’s responses to SpeakEasy’s responses. If the LSM is greater than or equal
to 0.8, the user is considered to have successfully shown attention to their
partner under symmetric, cooperative, social conditions, where the two people
had to accomplish a task. This is in contrast to asymmetric, competitive, or
negotiation conditions, where the LSM may not hold the same impact [14].
Otherwise, they are advised to show more attention by acknowledging what their
partner said previously.

##  5 Evaluation

Because for Quiz 6, we all designed separate evaluation plans, we chose one of
the evaluation plan’s statistics to present here and detail our reasoning
behind our evaluation. Prior to defining the five categories for the
evaluation, we gathered information about the diversity of the participants
that evaluated our chatbot SpeakEasy. Each member was assigned to gather ten
individuals to evaluate our system. Of the ten total participants in the
context of this evaluation plan, six of them were female and four of them were
male. The age of the participants ranged from 20-22 while the average age was
21.2 years old. All ten participants came from middle to upper-middle class
backgrounds. four of the participants were from Caucasian descent, five were
from Asian descent, and one was from African American descent. The context of
all ten conversations were casual, the purposes of the conversations were to
improve conversational skills, and the power dynamics of the conversation were
that all participants were my peers.

The metric chosen was the Likert scale from 1-5. In class, we went over the
Likert scale from 1-3; however, we decided to expand it to 1-5 because
according to one of the original papers using the Likert scale, the optimal
metrics were (1) Strong Disagree, (2) Disagree, (3) Undecided, (4) Agree, and
(5) Strongly Agree. Jamieson [6] argues that the extra feeling of intensity
(e.g., between disagree and strongly disagree) is important and because it
allows for better statistical analysis. Furthermore, according to their
analysis, the difference in intensity between each category is roughly even
allowing for a well-distributed interval for the variable.

In terms of satisfaction in a conversation with a chatbot, according to
Mohelska and Sokolová [9], the main attributes that must be fulfilled is if
the chatbot met expectation and left an impression. Consequently, the two
questions we asked were “did the chatbot met expectations” and “did the
chatbot leave an impression”. All ten participants gave a five to both these
questions; however, from a qualitative perspective, it was less the dialogue
flow and conversation, and more the voice technology that stunned the user. As
a result, in the future, evaluations of conversation satisfaction should ask
questions specific to dialogue flow. Mohelska and Sokolová [9] also recommends
asking about the conversation duration and number of conversation turns which
we did. All but one participant strongly agreed that the conversation duration
was appropriate while the other user said they agreed that the conversation
duration was appropriate. All ten people evaluating SpeakEasy said that the
number of conversation turns was appropriate.

According to Vanjani et al. [18], the biggest limitation in evaluation
techniques was evaluating how natural the chatbots are. They argue that
chatbots can be conversationally unnatural in two ways: the chatbot can give
garbled responses or give too accurate of an answer. As a result, we decided
to ask, “did the chatbot give garbled responses” and “were the answers the
chatbot gave were unnaturally accurate”. The answers for whether the chatbot
gave garbled responses ranged from 2-4 with the average answer being 2.8. The
biggest reasons for why the participants responded undecided or disagree were
that the chatbot failed to respond accurately to questions or completely
ignored. This is something that the chatbot needs to improve on going forward.
Specifically, we can use the GPT API to first determine whether is asking a
question, then use the GPT API to determine if the user is being on topic, and
finally randomly select a response from an answer bank that we construct.

When generating automatic conversation evaluators, Yi et al. [20] determined
that the four most determiners of chatbot coherence were whether the system
response is comprehensible, on topic, interesting, and if the user wanted to
continue the conversation. As a result, we ask four questions: (1) “Are
SpeakEasy’s responses comprehensible”, (2) “Are SpeakEasy’s responses on
topic”, (3) “Are SpeakEasy’s response interesting”, and (4) “Did you want to
continue the conversation”. The average response for comprehensibility was a
4.8. The average response for on topic was a 3.8 with the answers ranging from
a 2-5. The main qualitative issues were the aforementioned cases in which
SpeakEasy fails to respond to questions and in certain cases SpeakEasy
switches topics with fully closing out the previous topic. The average
response for whether the conversation was interesting was a 4.2. Participants
that did not give a strongly agree mainly stated that while sometimes
SpeakEasy gave a response specific to the participant’s response, there are
some instances in which it could give a more user-centric response. All five
participants that responded 5 to whether they wanted to continue the
conversation gave qualitative feedback mostly stating that they loved being
able to have a conversation in which they could actually speak to the chatbot.

When we designed SpeakEasy, we wanted it to be as empathetic as possible
because we felt that a person cannot be good at conversation without
introducing some degree in empathy into the conversation. The main question
asked by a study evaluating chatbot empathy conducted by Agarwal et al. [1]
was whether the chatbot disregarded any of the feelings of the user. As a
result, we asked the following question “Did SpeakEasy disregard any emotion
you interjected into the conversation” and “Did SpeakEasy reciprocate any
emotions that you conveyed”. For the former question, results varied from 1-5
with the average response being 3.1. A particularly concerning qualitative
response was that in response to the user saying that they were crying,
SpeakEasy responded to “I enjoy crying”. We must fix this in the future as
this error can drive away potential users. To the later question, the average
response was a 3.8 with no specific qualitative concerns.

Wahde and Virgolin [19] argue that the interpret-ability of conversational AI
depends on a variety of factors most importantly if the chatbot can explain
its reasoning, its ability to contain facts that make sense to the user, and
to want unwanted bias. Therefore, we asked the following questions: (1) “was
the chatbot’s reasoning behind its feedback well-explained”, (2) “were the
facts presented in the feedback clear”, and (3) “was there any bias in the
analysis”. All users but one gave a 5 for the first question, with the one
dissenting participant giving it a 4 stating that the LSM score was unclear.
An explanation of this score can be explained better going forward. For
questions 2 and 3, all users gave a 5 saying that the feedback was accurate
and easily understood.

Based on the aforementioned analyses, we need to improve on few specific
areas. Most importantly, we need to redesign SpeakEasy to be able to recognize
and answer questions better. A more minor upgrade would be better designing
our chatbot to recognize specifics of the user response and changing states
(the number of which would increase) to accordingly respond to the user.
Specifically, we need to recognize negative responses better and avoid
responding in a positive tone as that can appear condescending and scare off
users. Finally, we should explain the LSM score better and in future
evaluations we should ask questions more explicitly about the dialogue flow.

##  6 Novelty

During free conversation with SpeakEasy, the chatbot saves the user’s and the
bot’s conversation logs. These logs are then used to evaluate and provide
feedback on the user’s conversational skills, focusing on five aspects:
awkward transition, verbal tic, acknowledgment, number of questions, and
number of words per minute. This evaluation matrix is based on three papers
[4], [16], [15].

The work by Garrels [4] highlights the importance of intonation, smooth
transition, and asking questions for good conversation. Based on these
findings, we created the MacroAwkward, MacroNumQuestions, and MacroAVGToken
classes. The MacroAwkward class checks for awkward transitions by comparing
the user’s transitional words to a predefined list of awkward transitional
words, which were generated by ChatGPT for suggestion and can be expanded [4].
If the user uses less than 10 awkward transitions, they are considered to be
doing well. The MacroNumQuestions class counts the number of question marks in
user responses and advises the user to ask more questions if the ratio of
questions to total utterances is less than 0.39, as this was the high question
rate that showed positive effects on liking towards the conversation partner
[5]. The MacroAVGToken class measures the user’s intonation by calculating the
words per minute (WPM) and providing appropriate feedback based on a range of
120 to 150 WPM, which is the average American talking speed in conversation
according to the cited source [2].

Research by Russell et al. [16] demonstrates the negative impact of overusing
specific words. To address this aspect, we created the MacroTic class, which
calculates the frequency of each token the user used, excluding common words
like articles. The class provides feedback on the overuse of certain words
based on the sorted frequency list of tokens.

Experiments conducted by Rubin and Martin [15] reveal the importance of
acknowledging and showing empathy towards the speaking partner. To measure
these elements, we implemented the MacroAcknow class. One way to show
acknowledgment and empathy is by mentioning what the speaking partner said
previously. The MacroAcknow class calculates the language style matching (LSM)
using a function words list commonly used in conversation, comparing the
user’s responses to SpeakEasy’s responses. If the LSM is greater than or equal
to 0.8, the user is considered to have successfully shown attention to their
partner under symmetric, cooperative, social conditions, where the two people
had to accomplish a task. This is in contrast to asymmetric, competitive, or
negotiation conditions, where the LSM may not hold the same impact [14].
Otherwise, they are advised to show more attention by acknowledging what their
partner said previously.

The idea of improving communication through technology is not new. However,
there is a clear market gap for the use dialogue systems to provide
personalized feedback on conversation. There are two big competitors that
share similarities with SpeakEasy, yet neither provides the same service.
Rocky.AI is a personal development coaching app that provides help on
improving communication. However, its main purpose is to help the user use
self-awareness and reflection to reach their personal goals. Communication is
just a small portion of this. On the other hand, SpeakEasy provides much more
in-depth feedback and is able to narrow down exactly what the user needs to
work on without relying on self-assessment questions. Rather, our metrics are
based on research and use a quantitative approach to assessing communication
skills. Furthermore, SpeakEasy provides the user with the opportunity to
practice small-talk, something Rocky.AI lacks. Rocky.AI is a self-help coach
app that focuses on self-reflection, while SpeakEasy is a dialogue system that
provides personalized feedback in hopes of improving the user’s communication.
Though both systems aim to improve communication, the approach Rocky.AI takes
is entirely different from what SpeakEasy aims to do.

Yoodli is our most similar competitor. It is an AI speech coach that records
the user talking while they are speaking in a professional setting, and then
uses this information to provide personalized feedback on how to the user can
communicate more effectively. SpeakEasy and Yoodli share the same formulaic
approach to generate feedback. That is, both systems assess real dialogue the
user has to give appropriate feedback. The biggest difference is that Yoodli
focuses on improving professional speech, while SpeakEasy aims to improve
communication of all types. While superficially this difference may be minute,
it actually drives the novelty of SpeakEasy. Presentations, which Yoodli
specializes in, relies more on capturing the audiences attention by changing
the tone, volume, and pace of the speaking voice [14]. In other words, the
content of the speaking is less important in public speaking: captivating the
audience trumps all. SpeakEasy will generally improve all forms of
communication, but focuses mainly on conversations. Engaging conversations do
not revolve around the quality of speech but instead build of the creativity
of the talk [4]. As a result, SpeakEasy focuses heavily on developing
extensive ontology’s to capture a wide array of conversations and then perform
a series of analyses in order to analyze the quality of the subjects
discussed. This skill itself is more important than the public speaking skills
developed by Yoodli with conversations being essential to mental health [13].
In short, SpeakEasy assesses communication in a casual conversation setting,
while Yoodli provides feedback on interviews and public speaking.

##  7 Contributions

The development of SpeakEasy can be essentially divided into three main
processes: development of the input audio and transcribing process,
construction of the analysis Macros, and developing each of the main
conversational transition topics. However, in order to abide by the central
scrum tenants of having working software at the end of each sprint and not
working on the software one step of time to avoid integration errors, we
worked on several component simultaneously with different members working on
different elements. Hyunbae Jeon and Max Bagga were primarily responsible for
developing the speech to text process and developing the analysis macros.
Hyunbae developed the windows version of the software while Max developed the
Mac version of the software. Hyunbae contributed primarily by developing the
initial framework for the audio input and output and developing the analysis
Macros. Max worked primarily by refining the audio component, integrating the
dialogue flows, and working on the supplemental Macros needed to make the
dialogue flow work. Victoria Ploerer, Rhea Ramachandran, and Yella Diekmann
developed the conversational transitions. Specifically, Rhea created the
introduction and health transitions, while Victoria completed the Travel
transition, and Yella created the Entertainment transition. Upon completing
each transition, we all worked together to connect them in order to carry out
a smooth transition from one topic to another. Despite the given roles,
everyone was very involved in all aspects, and each of us contributed 20
percent to guarantee the chatbot functioned the best it can.

## References

  * Agarwal et al. [2021] Akshay Agarwal, Shashank Maiya, and Sonu Aggarwal.  Evaluating empathetic chatbots in customer service settings.  1 2021.
  * Barnard [8 Nov. 2022] Dom Barnard.  Average Speaking Rate and Words per Minute — virtualspeech.com.  https://virtualspeech.com/blog/average-speaking-rate-words-per-minute, 8 Nov. 2022.  [Accessed 01-May-2023].
  * Finch and Choi [2020] James D. Finch and Jinho D. Choi.  Emora STDM: A Versatile Framework for Innovative Dialogue System Development.  In _Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue_ , SIGDIAL’20, pages 261–264. Association for Computational Linguistics, 2020.
  * Garrels [2019] Veerle Garrels.  Getting good at small talk: student-directed learning of social conversation skills.  _European Journal of Special Needs Education_ , 34(3):393–402, 2019.  doi: 10.1080/08856257.2018.1458472.  URL https://doi.org/10.1080/08856257.2018.1458472.
  * Huang et al. [2017] Karen Huang, Michael Yeomans, Alison Wood Brooks, Julia Minson, and Francesca Gino.  It doesn’t hurt to ask: Question-asking increases liking.  _Journal of personality and social psychology_ , 113(3):430, 2017.
  * Jamieson [2004] Susan Jamieson.  Likert scales: how to (ab)use them.  _Medical Education_ , 38(12):1217–1218, 2004\.  doi: https://doi.org/10.1111/j.1365-2929.2004.02012.x.  URL https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1365-2929.2004.02012.x.
  * Lee et al. [2021] Jungmin Lee, Hyun Ju Jeong, and Sujin Kim.  Stress, anxiety, and depression among undergraduate students during the COVID-19 pandemic and their use of mental health services.  _Innov. High. Educ._ , 46(5):519–538, April 2021\.
  * Mendo-Lázaro et al. [2018] Santiago Mendo-Lázaro, Benito León-del Barco, Elena Felipe-Castaño, María-Isabel Polo-del Río, and Damián Iglesias-Gallego.  Cooperative team learning and the development of social skills in higher education: The variables involved.  _Frontiers in Psychology_ , 9, 2018.  ISSN 1664-1078.  doi: 10.3389/fpsyg.2018.01536.  URL https://www.frontiersin.org/articles/10.3389/fpsyg.2018.01536.
  * Mohelska and Sokolová [2021] Hana Mohelska and Marcela Sokolová.  Measuring chatbot effectiveness.  pages 589–598, 03 2021.  doi: 10.36689/uhk/hed/2021-01-058.
  * OpenAI [2023] OpenAI.  Gpt-4 technical report, 2023.
  * Pickard et al. [2017] Hannah Pickard, Fruhling Rijsdijk, Francesca Happé, and William Mandy.  Are social and communication difficulties a risk factor for the development of social anxiety?  _Journal of the American Academy of Child & Adolescent Psychiatry_, 56:344–351, 2017.  doi: https://doi.org/10.1016/j.jaac.2017.01.007.
  * Radford et al. [2022] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever.  Robust speech recognition via large-scale weak supervision, 2022.  URL https://arxiv.org/abs/2212.04356.
  * Reid and Reid [2007] Donna J. Reid and Fraser J.M. Reid.  Text or talk? social anxiety, loneliness, and divergent preferences for cell phone use.  _CyberPsychology & Behavior_, 10(3):424–435, 2007.  doi: 10.1089/cpb.2006.9936.  URL https://doi.org/10.1089/cpb.2006.9936.  PMID: 17594267.
  * Richardson et al. [2019] Beth H Richardson, Kathleen C McCulloch, Paul J Taylor, and Helen J Wall.  The cooperation link: Power and context moderate verbal mimicry.  _Journal of experimental psychology: applied_ , 25(1):62, 2019.
  * Rubin and Martin [1994] Rebecca B. Rubin and Matthew M. Martin.  Development of a measure of interpersonal communication competence.  _Communication Research Reports_ , 11(1):33–44, 1994.  doi: 10.1080/08824099409359938.  URL https://doi.org/10.1080/08824099409359938.
  * Russell et al. [2008] Brenda Russell, Jenna Perkins, and Heather Grinnell.  Interviewees’ overuse of the word “like” and hesitations: Effects in simulated hiring decisions.  _Psychological Reports_ , 102(1):111–118, 2008\.  doi: 10.2466/pr0.102.1.111-118.  URL https://doi.org/10.2466/pr0.102.1.111-118.  PMID: 18481672.
  * Staff [2017] NACE Staff.  Job outlook 2018: College hiring to increase by 4 percent.  _National Association of Colleges and Employers_ , 2017.  URL https://www.naceweb.org/job-market/trends-and-predictions/job-outlook-2018-college-hiring-to-increase-by-4-percent/.
  * Vanjani et al. [2019] Mahesh Vanjani, Jamison Posey, and Milam Aiken.  An evaluation of three online chatbots.  volume 18, 2019.
  * Wahde and Virgolin [2022] Mattias Wahde and Marco Virgolin.  The five is: Key principles for interpretable and safe conversational ai.  In _2021 The 4th International Conference on Computational Intelligence and Intelligent Systems_ , CIIS 2021, page 50–54, New York, NY, USA, 2022. Association for Computing Machinery.  ISBN 9781450385930.  doi: 10.1145/3507623.3507632.  URL https://doi.org/10.1145/3507623.3507632.
  * Yi et al. [2019] Sanghyun Yi, Rahul Goel, Chandra Khatri, Alessandra Cervone, Tagyoung Chung, Behnam Hedayatnia, Anu Venkatesh, Raefer Gabriel, and Dilek Hakkani-Tur.  Towards coherent and engaging spoken dialog response generation using automatic conversation evaluators.  volume 12, pages 65–75, october 2019.


# Global Fire Season Severity Analysis and Forecasting111L.N.F conceived and
conducted the experiments. All the authors analyzed the results, wrote and
reviewed the paper. Declarations of interest: none

Leonardo N. Ferreira<EMAIL_ADDRESS>Didier A. Vega-Oliveros
<EMAIL_ADDRESS>Liang Zhao<EMAIL_ADDRESS>
Manoel F. Cardoso<EMAIL_ADDRESS>Elbert E. N. Macau
<EMAIL_ADDRESS>Associated Laboratory for Computing and Applied
Mathematics, National Institute for Space Research,
São José Dos Campos - SP, Brazil.  School of Informatics, Computing and
Engineering, Indiana University, Bloomington, IN, USA  Department of Computing
and Mathematics, University of São Paulo, Ribeirão Preto - SP, Brazil.  Center
for Earth System Science, National Institute for Space Research, Cachoeira
Paulista - SP, Brazil.  Institute of Science and Technology, Federal
University of São Paulo, São José Dos Campos - SP, Brazil.

###### Abstract

Fire activity has a huge impact on human lives. Different models have been
proposed to predict fire activity, which can be classified into global and
regional ones. Global fire models focus on longer timescale simulations and
can be very complex. Regional fire models concentrate on seasonal forecasting
but usually require inputs that are not available in many places. Motivated by
the possibility of having a simple, fast, and general model, we propose a
seasonal fire prediction methodology based on time series forecasting methods.
It consists of dividing the studied area into grid cells and extracting time
series of fire counts to fit the forecasting models. We apply these models to
estimate the fire season severity (FSS) from each cell, here defined as the
sum of the fire counts detected in a season. Experimental results using a
global fire detection data set show that the proposed approach can predict FSS
with a relatively low error in many regions. The proposed approach is
reasonably fast and can be applied on a global scale.

###### keywords:

Global fire activity , Wildfire , Fire season length , Fire severity , Climate
change , Time series prediction

##  1 Introduction

Wildfires have a fundamental role in the environment and a huge impact on
human lives [31, 9, 8]. They affect biodiversity, cause damage to forests and
properties, influence global climate, and pose direct risks to human health
[18, 39, 14]. Forecasting fire activity thus brings many benefits for fire
management and control [31, 17].

Models have been used to predict fire occurrence in global and regional
scales. Global fire models focus on long-term predictions (decades to
centuries) and include a complex set of dynamic factors related to climate,
vegetation, and human activity [7], which are linked to the presence and
flammability of vegetation biomass, and sources of ignitions [3]. Their high
level of detail and precision lead to satisfactory results in long-term
predictions but were not intended for seasonal forecasting [36]. Regional
models are mainly applied for projections at the seasonal timescale. They
provide good forecasting results for specific regions but require input data
like vegetation (fuel), topography and weather, that may not be available for
other regions [37]. This limitation makes their application difficult to
specific regions [43, 38, 11, 32, 39]. Some methods have been developed to
predict seasonal fire activity on a global scale [10, 42]. In general, they
use burned area data, which do not accurately account for smaller real fires
[23, 19, 15, 19]. All these mentioned disadvantages motivate the development
of a simple, accurate, and widely-applicable seasonal forecasting method.

In this paper, we propose a seasonal fire prediction approach, which is
simple, effective, relatively fast, and can be generalized and used on a
global scale. Our methodology is based on applying seven different time series
forecasting methods and evaluating their suitability as seasonal fire
forecasting predictors. It consists of dividing the studied area into grid
cells and extracting time series of fire counts to fit the models. We use
these models to estimate the fire season severity (FSS) defined as the sum of
the fire counts detected in a season.

After applying our approach to a global fire detection data sets, the results
reveal that the forecasting errors are relatively low in the vast majority of
the areas. These results indicate that it is possible to predict worldwide FSS
by employing fire count historical data and time series forecasting methods.
The proposed methodology can be especially useful for landscape managers and
planners that lack the expertise and/or data for building or use some
specialized forecasting model. It can also be applied to quickly estimate fire
activity in large areas.

This manuscript is organized in the following form. We start by presenting in
Section 2 the data we use and the proposed forecasting methodology. In Sec. 3,
we present our results and discussion divided into three analyses: fire season
length, fire season severity, and forecasting. Finally, we present in Sec. 4
some conclusions and future works.

##  2 Material and Methods

###  2.1 Data

In our experiments, we use data from the Moderate Resolution Imaging
Spectroradiometer (MODIS) on-board NASA’s Terra and Aqua satellites.
Specifically, we used the Global Daily Fire Location Product (MCD14ML)
Collection 6 from 2003 to 2017 [20, 21, 27]. This data set comprises active
fire detection and is composed of geographic location, date, detection
confidence, and some additional information for each fire pixel detected by
the Terra and Aqua MODIS sensors. We consider only those fire records in the
data set with detection confidence higher than 75%. We disregard the years of
2001 and 2002 because they have missing data. The data set is freely
accessible [20].

To analyze this spatiotemporal data set, we divide the globe into 65,612
hexagonal grid cells of approximated 7,774 $km^{2}$ each [4]. The main
advantage of the hexagonal grid, when compared to the traditional rectangular
longitude-latitude one, is that the former generates cells with more uniform
coverage area and avoid distortions. Since the vast majority of the grid cells
are located in regions without fire, like in the oceans or poles, we discard
many cells. Only cells with at least one fire record per season were
considered. After selecting only those cells with fire, we end up with 6486
ones. For each of those cells, we construct a time series with the daily fire
records detected by the satellites. We use the term _“fire counts”_ (FC) to
refer to these time series similarly as previous works [11, 16].

Our choice for using fire counts instead of other types of fire data, such as
burned area products, is simply justified by the characteristics of our
analyses. This kind of data is easy to divide into grid cells and to
interpret. Burned area information, on the other hand, would need to be
reprocessed to be spatially reorganized into grid cells, adding complexity and
not contributing with crucial information to our analyses. Another reason for
using an active fire product is the availability of a daily timescale,
required in our analysis. Furthermore, active fire data sets like the MCD14ML
tend to detect smaller fires than burned area products [15, 19], which leads
to a satisfactory level of accuracy for our study.

###  2.2 Fire Season Estimation

To estimate the length of the fire seasons, we propose a simple method: First,
we apply a moving average (window size of seven days) to smooth the historical
time series of fire counts. Then, we count the largest periods without fire.
We remove these periods and consider the other ones as fire seasons. In Fig 1,
we illustrate this process using three FC time series from different regions.

Figure 1:  Fire seasons estimation. Each time series represents the daily fire
counts (FC) detected in grid cells located in three different regions: Brazil,
India, and Mali. After the smoothing, we count the periods without fire (in
blue) and consider the other periods (in red) as fire seasons.

We define fire seasons as the time window centered in the month with the
historical highest occurrence of fire in each cell [11]. We use the fire
season estimation method here proposed to define a single proper time window
length to be used for all the cells. Specifically, this method was applied to
the time series from all grid cells and verified the empirical distribution of
fire season mean lengths. We remove outlier season lengths, i.e., values
higher than the third quartile ($Q_{3}$) + 1.5 $\times$ interquartile range
($IQR$). Outlier lengths appear for multiple reasons. In some cases, these
uncommonly long activities are accounted for false fire detections like in
regions with hot bare soils [33]. In other cases, they represent volcanoes or
gas flares from oil and gas exploration. The outlier removal minimizes the
effect of these uncommon cells. After removing outliers, we choose the 99%
season length percentile as the global season length (see Sec. 2.2). This
method guarantees that the FC time series from all cells have the same length
and well represents the periods with more fire. The fire season severity (FSS)
is the sum of fire counts (FC) in a season [11].

###  2.3 Forecasting methods and evaluation

In our experiments, we discard the first and last season to avoid measuring
broken seasons, resulting in 13 years of data. We also opt for using the
monthly-accumulate fire counts (MA-FC) of each season to fit the forecasting
models. The reason is the lack of FSS samples (13 in total), making it
unfeasible to train the forecasting models. Since every season has seven
months, the MA-FC has 91 values. We opt for using just the MA-FC in the
seasons to train the models because the period outside the seasons is mainly
formed by zeros (see Sec.3).

For each cell, we separate the first ten seasons to train the model and the
last three ones to test. We performed a Box-Cox transformation in both the
train and test time series, and the parameter lambda was calculated using the
method proposed by Guerrero [22]. The FSS forecast is the sum of the MA-FC
predicted for a season. Our forecasting methodology is illustrated in Fig. 2,
using three MA-FC time series (the same from Fig. 1) as examples with
relatively high (Brazil), medium (India), and low (Mali) interannual
variability for the peak months. In the three examples, the forecasting method
(Prophet [41]) can reproduce the seasonal variations and accurately predict
(in red) the test seasons.

Figure 2:  Forecasting methodology. The three time series are the monthly-
accumulate fire counts (MA-FC) for the examples in Fig. 1. The first 10
seasons (blue background) are used to train the forecasting models and the
last three ones (red background) are used to test. The time series in red are
the result of the prediction using one of the methods (Prophet [41]). The
final FSS forecast is the sum of the monthly forecasts from each season.

In the following, we briefly describe the forecasting methods:

  * 1.

Naive forecasting: one of the simplest forecasting methods, it considers the
last observation in a non-seasonal time series as the prediction for the next
value [24]. In a seasonal time series, the seasonal naive forecasting (snaive)
method considers each forecast to be equal to the observed value in the same
period in the previous season. For example, the snaive monthly prediction for
July 2020 is the observed value in July of 2019. Although its simplicity, this
method provides proper results in many cases. We use this method as a
baseline.

  * 2.

Autoregressive Integrated Moving Average models (ARIMA): aims to describe the
autocorrelations on data given the past events, which is an approach widely
employed for time series forecasting. The procedure starts analyzing the lag
regression of the variable, and then, performing a moving average of errors
from the linear combination of the past events. Therefore, the prediction of a
variable is assumed to be a linear function of previous data and random
errors. We employ the function in R that uses the Hyndman-Khandakar algorithm
for automatic ARIMA modeling, which finds the best autoregression (AR) model
and the moving average (MA) of weighted linear combination to obtain the
prediction method [25, 29].

  * 3.

Exponential smoothing (ETS): is a data forecasting method that uses a sliding
window function to specify weights that decay exponentially over time. The
sliding function helps for smoothing the series data, i.e., applying a low-
pass filter to discard outliers or noise [24]. Thus, the previous data events
are weighted with a geometrically decreasing ratio, i.e., the older the event,
the lower its relevance weight. ETS methods can be used in datasets with
seasonality, systematic trend, and other assumptions [29]. Besides, ETSs are
considered a better option than the ARIMA methods because they are non-
stationary and adopt exponentially decreasing weights calculated from previous
events. Conversely, ARIMA methods are stationary and used linear weights of
the past observations according to the moving average technique [24].

  * 4.

Short-term load forecasting (STLF): this approach can be defined as a time-
series decomposition by seasons and trends using a Loess forecasting modeling
[25]. The method assumes that a time series can be separated in error, trend
and seasonality components. Then, the results of the decomposition are used as
input for the forecasting step. The time series is decomposed by a variation
of a seasonal and trend function using Loess (STL) [12], which deals with
multiple seasonality and nonlinear relationships. The returned multiple
seasonal components are employed by a simple method for adjusting the
seasonality of data [25], and the forecasting is the recombination process of
the components. According to the authors, the method produces good forecasts
results in seasonal time series [12].

  * 5.

TBATS: decomposes the time-series data into seasonal components and combines
them in a trigonometric representation [30]. These components are the Fourier
terms of the series smoothed with an exponential Box-Cox transformation [30,
40]. The trigonometric decomposition brings some advantages compared to
traditional linear methods [30]: it is possible to model non-integer seasonal
data or high-frequency series, detecting more refined patterns; the method
works with large parameter space not affecting the forecast results; it can
process nested or not seasonal components with nonlinear behavior, which is
the case of real-world series data. Besides its capabilities, the method is
robust to autocorrelation data, involving a simpler but efficient estimation
procedure.

  * 6.

Generalized linear model for count time series (TSGLM): the method uses the
generalized linear model (GLM) approach for modeling the data events
(observations) according to previous conditional information [29]. For this
purpose, it considers that the data are only composed of positive integers,
i.e., count time series. In this way, TSGLM provides a likelihood-based
estimation capturing the dependence among data events [29]. This linear
predictor provides a regression method over previous observations taking into
consideration the covariance effects. For this point, it employs a probability
distribution function, like Poisson (used in our experiments) or negative
binomial, to estimate the maximum or quasi-likelihood of the distribution for
obtaining the best statistical inference model [29].

  * 7.

Artificial neural networks (ANN): a model inspired by the biological neural
networks constituted in animal brains. They are based on a set of connected
units called artificial neurons [6]. The architecture of ANN frequently
contains multilayer perceptron and sigmoid neurons, organized in layers and
employing the stochastic gradient descent as the standard of the learning
process. The ANNs are trained to learn the input-output relationships through
an iterative process, in which the weight of the neurons are adjusted to
minimize the error between the predicted and the true outputs. As a result, it
is expected that the ANN learns a suitable model that accurately generalize
the response to new data. In our experiments, we used Multi-Layer Perceptron
(MLP) networks with a single hidden layer and lagged inputs [25]. We used just
one seasonal lag as input and the hidden layer has half of the number of input
nodes plus 1. So the number of neurons in the input, hidden, and output layers
are respectively seven, five, and one. We repeat the training process for this
topology 20 times starting with random weights initialization and we report
only the best result.

  * 8.

Prophet: this is an open-source software released by Facebook [40] and used in
several applications by the company for producing reliable forecasts in
planning and goal setting. The algorithm follows an additive model approach
where a non-linear smoother is applied to the regressor by yearly, weekly, and
daily seasonality. The method analyzes three main components [41] trend,
seasonality, and holidays effect, where the components are automatically
detected from the data. According to the authors, Prophet performs better than
any other approach in most of the experimental results. They showed the
robustness of the method in the presence of outliers, missing data, and shifts
in the trend [41]. Besides, the method is suitable for time-series that
present historical strong seasonal effects.

In the experiments, we used the R programming language [35] with the packages:
forecast [25], prophet [40], and tscount [29]. These packages provide
implementations of the forecasting methods and automatically tune the
parameters for fitting the models to the time series. For details of
implementation, we point the interested reader to the packages manuals and our
source code described in Section “Computer Code Availability”.

To find the best forecasting method of a single time series from a grid cell,
we compared the mean absolute error (MAE) [24]. Considering an observed (test)
time series $Y$ of $n$ values and $F$ the predicted values for $Y$, the
prediction error $e_{t}$ in a time $t$ is the difference between an observed
value and its forecast: $e_{t}=Y_{t}-F_{t}$. The MAE is defined as:

$MAE=n^{-1}\sum\limits_{t=1}^{n}|e_{t}|.$ (1)

The advantage of this measure is the easy interpretation of the error for a
specific time series. However, the MAE is a scale-dependent measure that
cannot be used to asses the accuracy in time series with different scales. The
different scales make the accuracy interpretation not so clear in different
scenarios (grid cells). To make the interpretation easier, in our experiments,
we normalize the MAE by mean FSS values in each cell. The non-normalized
results are presented in the A.

We use the mean absolute scaled error (MASE) to compare the accuracies of
different methods in all cells [26]. The scaled forecasting error $q_{t}$ of a
non-seasonal time series $Y$ is the error $e_{t}$ divided by the MAE of the
non-seasonal naive forecast method on the training set:

$q_{t}=\frac{e_{t}}{\frac{1}{n-1}\sum\limits_{t=2}^{n}|Y_{t}-Y_{t-1}|}.$ (2)

The scaled error $q_{t}$ of a time series $Y$ with a seasonal period $m$ is:

$q_{t}=\frac{e_{t}}{\frac{1}{n-m}\sum\limits_{t=m+1}^{n}|Y_{t}-Y_{t-m}|}.$ (3)

If $q_{t}$ < 1, the forecasting method is better than the seasonal naive
(snaive), otherwise snaive is better. As its name indicates, the MASE is
simply the mean of all scaled errors:

$MASE=n^{-1}\sum_{t=1}^{n}|q_{t}|,$ (4)

where $q_{t}$ corresponds to Eq. 2 if the modeled time series $Y$ is not
seasonal or Eq. 3 otherwise. In our experiments, we use the MASE with Eq. 3 to
evaluate the seasonal MA-FCs predictions. We use Eq. 2 to compare the non-
seasonal FSS (sum of MA-FCs in a season) from the best model in each cell with
the result of linear regression. We opt to compare to a linear regression due
to the limited number of FSS values required to train other forecasting
models. When MASE < 1, the forecasting method gives, on average, smaller
errors than the one-step errors from the naive method. The best forecasting
method has the lowest MASE values.

##  3 Results and Discussion

###  3.1 Fire season Analysis

We propose in Sec. 2.2, a simple method to estimate the fire season length. In
our first analysis, we apply this method to all the grid cells. Since we have
15 years of fire data, we extracted the same amount of fire seasons. In Fig.
3, we illustrate the fire season mean length and trend for each cell. Some of
the regions with the longest (more than 7 months) fire season lengths are
Paraguay, Southeast of the USA, north of Australia, and most parts of Africa.
Since 99% of the cells have fire seasons shorter than seven months (in the
mean), we consider seven months a reasonable time window to represent
worldwide fire seasons. More than half of the cells (57%) present a decline in
the fire season lengths but some regions like in Northeast Brazil, Eastern
Russia, and parts of Africa show an increase in the fire season length.

Figure 3: Global Fire season description: (A) length means and (B) linear
trends (slopes). The inset figures show the respective probabilistic density
functions (PDF).

After defining seven months as the fire season length, we find the months with
the historical highest occurrence of fire in each cell. Fig. 4 illustrates the
months with peaks of fire activity and a histogram with the months’
distribution (inset). The months of historical maximum fire activity are
distributed following a bimodal distribution (aligned with previous results
[5]). The two groups in the distribution mainly represent the dry seasons in
the tropical rain belt (northern to the southern tropics). In the tropical
zone, regions below the Equator have peaks of fire activity between August and
October while locations above the Equator have maximums between March and May.
Right on the equator, the fire activity is predominant at the end of the year
(November to January). Part of the high fire activity in April is accounted
for the dry seasons in parts of Russia and South Australia [34].

Figure 4: Months with the highest fire activity in each cell. The inset figure
shows a histogram with the number of cells with the highest fire activity in a
month.

The fire seasons are the periods of seven months centered in the month with
the highest occurrence of fire. In Fig. 5, we show the fire season severity
(FSS) means and trends calculated for all the cells. 50% of the global area
has FSS mean lower than 100 FCs per year and 99% have less than 1200 FCs on
average. The highest number of fire counts concentrate in the tropics. In
general, there is a trend of decline in the mean FSS worldwide (61% of cells).
However, in other regions like Southern Africa, Far Eastern Russia, Eastern
Ukraine, and the West Coast of the USA, we can observe a notably increase
tendency in the mean FSS.

Figure 5: Global Fire season severity (FSS) description: (A) means (B) and
linear trends (slopes). The inset figures show the respective probabilistic
density functions (PDF).

Multiple reasons and factors can explain the trends in the fire seasons’
lengths and severity. Some of the elements related to those variations are the
increase or decrease of factors that lead to fire ignitions, such as less
deforestation or lower lightning occurrence [31, 7, 33]. Climate phenomena,
e.g. the El Niño-Southern Oscillation (ENSO), also influence the fire activity
in critical regions like tropical forests [31]. Some previous works link the
increase in global fire activity and seasonal length caused by climate change
[18, 17, 28]. Similarly to our results, the global decrease in the fire
activity was also observed in previous works that associate it to the
worldwide increase in population densities and cropland areas [1, 2, 16].

###  3.2 Fire season severity forecasting

The forecasting results for the monthly-accumulated fire counts (MA-FC) are
illustrated in Fig. 6. On the top figure (6-A), we show the MAE for the best
method normalized by the FSS mean (Fig 5-A) and the density function for the
MAE (inset). The regions with the higher relative MAEs are Northern Australia,
Ukraine, parts of Russia, and the USA. Although, all the cells present MAE
lower than the FSS mean, even in the African continent whose cells present
that highest FSS means (Fig. 5). These results indicate a low general
forecasting error. Since we defined FSS mean as a baseline, it is difficult to
make general conclusions (especially in large regions) without taking into
account the characteristics of each cell. Therefore, we point the reader
interested in specific regions to the non-normalized MAEs, presented in the A
(Fig. 8).

In the bottom figure (6-B), we present a boxplot of the MASE for the entire
world divided by continent. The TBATS and the ANN models present the lowest
and the highest median error respectively. TBATS performs statistically better
than the other methods except for the ETS ($p\leq$ 0.001 – Friedman and
Nemenyi tests [13]). In 75% of the cells, the forecasting methods provide
results better (MASE < 1) than a seasonal naive forecasting method. Our
results show that MAEs and MASEs for the MA-FC time series are low, which
suggests that it is possible to apply historical detected active fire data to
forecast fire seasons severities.

Figure 6:  Monthly-accumulated fire counts (MA-FC) forecasting. (A) The MAE
from the best model for each cell normalized by the FSS mean (Fig 5-A) and the
PDF (inset). (B) Models comparison using the MASE (Eq. 3) for the seasonal MA-
FC from each cell divided by continent and the global. Outliers removed in
both figures.

Given the MA-FC forecasting errors, we adopt the best model for each cell to
make the FSS prediction. In Fig. 7, we present the normalized MAEs and MASEs
achieved comparing the best model for each cell with the FSS forecasting
results from a linear regression. The non-normalized MAEs are reported in the
A (Fig. 9). In some regions, like north Australia and Ukraine, the normalized
MAE is higher than 1, denoting that the errors are higher than the FSS mean
values. Conversely, in most of the regions (95%), the MAEs are lower than the
FSS mean values. This result indicates worldwide predictability in the FSSs.
The median MASEs are lower than one for the best models, which means that
predictions are better than the naive forecasting method in at least 50% of
the cells (outliers removed). The best models for each cell statistically
outperform the linear regression (according to the Wilcoxon paired test [13]),
except in Europe where the errors are not statistically different. This result
reinforces the worldwide predictability of FSSs in most of the regions.

Figure 7: FSS forecasting. (A) The MAE from the best model for each cell and
the PDF (inset). (B) Forecasting comparison between the best model achieved
with MA-FCs and the linear regression in each cell. Accuracy was measured
using the MASE (Eq. 2) for the non-seasonal FSS from each cell divided by
continent and the global (outliers removed). The significance thresholds are:
not significant (ns), $p<$ 0.05 (*) and $p\leq$ 0.001 (***).

##  4 Conclusions

In this paper, we have analyzed global fire season severity, defined here as
the accumulated fire counts detections in a season. We divided the globe into
hexagonal grid cells and extracted time series of fire counts in each cell. We
propose a very simple method to estimate the fire seasons. Our results show
that 99% of the cells have seasons shorter than seven months. We observed that
the length of fire seasons are decreasing in general. However, in some regions
like Southern Africa, Northeast Brazil, and Eastern Russia the fire seasons
are increasing. Most of the cells (61%) presented a decline in the FSS,
excluding some regions like Southern Africa, Far Eastern Russia, Eastern
Ukraine, and the West Coast of the USA. One possible explanation for the
declining fire activity is the global agricultural expansion and
intensification [1].

We have also shown that it is possible to forecast the FSS in many regions
around the world. Since the FSS time series are very short, we used the
monthly-accumulated fire counts (MA-FC) to train and test seven forecasting
models. In this timescale, all the cells presented mean absolute error (MAE)
lower than the FSS mean. The MAEs for the predicted FSS (sum of predicted MA-
FC in a season) is lower than the FSS mean values in 95% of the cells,
indicating predictability in the global FSS. In general, the TBATS forecasting
model provides the best results. We conclude that time series forecasting
methods can be used to estimate worldwide fire seasons in a simple, fast and
general way.

It is important to mention that our results are based on the historical data
of active fire detections, which is short and limited. The historical data may
not take into account climate change, which may considerably modify the fire
seasons in the future. Our results are aligned with previous results [1, 2,
16] but they might not reflect the future if the climate keeps changing.

This work can be extended in many directions. Here we focused on global
analysis but a natural next step is the thorough analyses of specific regions
and the deep reasons that explain fire activity variations. Other variables,
like socioeconomic projections and climate indices, can be included in the
forecasting methods to improve the prediction. Other forecasting methods and
machine learning tasks can also be explored on this data set. Tasks including
fire seasons classification, clustering, and anomaly detection. These
techniques might reveal new patterns about the interplay between anthropogenic
activity and global fire dynamics.

## Acknowledgments

This research is supported by the Fundação de Amparo à Pesquisa do Estado de
São Paulo (FAPESP) under Grant 2015/50122-0 and the German Research Council
(DFG-GRTK) Grant 1740/2. L.N.F. acknowledges FAPESP Grants 2019/00157-3 and
2017/05831-9. D.A.V.O acknowledges FAPESP Grants 2016/23698-1, 2018/01722-3,
and 2018/24260-5. E.E.N.M acknowledges FAPESP Grant 2018/03211-6. This
research was developed using computational resources from the Center for
Mathematical Sciences Applied to Industry (CeMEAI) funded by FAPESP (grant
2013/07375-0).

## Computer Code Availability

The fire season estimation method and the forecasting algorithms used in this
paper were implemented using the R programming language [35] and the packages:
forecast [25], prophet [40], and tscount [29]. The R software, packages, and
our implementation are open-source and freely available for download. All the
packages are distributed under GPL license, except prophet that is distributed
under BSD-3-Clause license. Our source code can be downloaded in the following
link:

https://lnferreira.github.io/global_fss_analysis_forecasting/

##  Appendix A Forecasting Errors

Figure 8:  Monthly-accumulated fire counts (MA-FC) forecasting. The MAE from
the best model for each cell and the PDF (inset). Figure 9:  FSS forecasting.
The MAE from the best model for each cell and the PDF (inset).

## References

  * Andela et al. [2017] Andela, N., Morton, D.C., Giglio, L., Chen, Y., van der Werf, G.R., Kasibhatla, P.S., DeFries, R.S., Collatz, G.J., Hantson, S., Kloster, S., Bachelet, D., Forrest, M., Lasslop, G., Li, F., Mangeon, S., Melton, J.R., Yue, C., Randerson, J.T., 2017\.  A human-driven decline in global burned area.  Science 356, 1356–1362.
  * Arora and Melton [2018] Arora, V.K., Melton, J.R., 2018\.  Reduction in global area burned and wildfire emissions since 1930s enhances carbon uptake by land.  Nature Communications 9, 1326\.
  * Arroyo et al. [2008] Arroyo, L.A., Pascual, C., Manzanera, J.A., 2008.  Fire models and methods to map fuel types: The role of remote sensing.  Forest Ecology and Management 256, 1239 – 1252.
  * Barnes [2017] Barnes, R., 2017.  dggridR: Discrete Global Grids for R.  R package version 2.0.1.
  * Benali et al. [2017] Benali, A., Mota, B., Carvalhais, N., Oom, D., Miller, L.M., Campagnolo, M.L., Pereira, J.M.C., 2017.  Bimodal fire regimes unveil a global-scale anthropogenic fingerprint.  Global Ecology and Biogeography 26, 799–811.
  * Bishop [1995] Bishop, C.M., 1995.  Neural Networks for Pattern Recognition.  Oxford University Press, Inc., New York, NY, USA.
  * Bowman et al. [2009] Bowman, D.M., Balch, J.K., Artaxo, P., Bond, W.J., Carlson, J.M., Cochrane, M.A., D’Antonio, C.M., DeFries, R.S., Doyle, J.C., Harrison, S.P., Johnston, F.H., Keeley, J.E., Krawchuk, M.A., Kull, C.A., Marston, J.B., Moritz, M.A., Prentice, I.C., Roos, C.I., Scott, A.C., Swetnam, T.W., Van Der Werf, G.R., Pyne, S.J., 2009\.  Fire in the earth system.  Science 324, 481–484.
  * Brando et al. [2014] Brando, P.M., Balch, J.K., Nepstad, D.C., Morton, D.C., Putz, F.E., Coe, M.T., Silvério, D., Macedo, M.N., Davidson, E.A., Nóbrega, C.C., Alencar, A., Soares-Filho, B.S., 2014\.  Abrupt increases in Amazonian tree mortality due to drought-fire interactions.  Proceedings of the National Academy of Sciences of the United States of America 111, 6347–52.
  * Chen et al. [2010] Chen, S.C., Preisler, H.K., Fujioka, F., Benoit, J.W., Roads, J.O., 2010.  Seasonal predictions for wildland fire severity, in: Proceedings of the Third International Symposium on Fire Economics, Planning, and Policy: Common Approaches and Problems, DIANE Publishing Company. p. 57.
  * Chen et al. [2016] Chen, Y., Morton, D.C., Andela, N., Giglio, L., Randerson, J.T., 2016.  How much global burned area can be forecast on seasonal time scales using sea surface temperatures?  Environmental Research Letters 11, 045001.
  * Chen et al. [2011] Chen, Y., Randerson, J.T., Morton, D.C., DeFries, R.S., Collatz, G.J., Kasibhatla, P.S., Giglio, L., Jin, Y., Marlier, M.E., 2011.  Forecasting fire season severity in south america using sea surface temperature anomalies.  Science 334, 787–791.
  * Cleveland et al. [1990] Cleveland, R.B., Cleveland, W.S., McRae, J.E., Terpenning, I., 1990\.  Stl: A seasonal-trend decomposition procedure based on loess.  Journal of Official Statistics 6, 3–73.
  * Demšar [2006] Demšar, J., 2006.  Statistical comparisons of classifiers over multiple data sets.  J. Mach. Learn. Res. 7, 1–30.
  * Dey and Schweitzer [2018] Dey, D., Schweitzer, C., 2018\.  A Review on the Dynamics of Prescribed Fire, Tree Mortality, and Injury in Managing Oak Natural Communities to Minimize Economic Loss in North America.  Forests 9, 461.
  * Earl and Simmonds [2017] Earl, N., Simmonds, I., 2017\.  Variability, trends, and drivers of regional fluctuations in australian fire activity.  Journal of Geophysical Research: Atmospheres 122, 7445–7460.
  * Earl and Simmonds [2018] Earl, N., Simmonds, I., 2018\.  Spatial and temporal variability and trends in 2001–2016 global fire activity.  Journal of Geophysical Research: Atmospheres 123, 2524–2536.
  * Flannigan et al. [2013] Flannigan, M., Cantin, A.S., de Groot, W.J., Wotton, M., Newbery, A., Gowman, L.M., 2013\.  Global wildland fire season severity in the 21st century.  Forest Ecology and Management 294, 54 – 61.
  * Flannigan et al. [2006] Flannigan, M.D., Amiro, B.D., Logan, K.A., Stocks, B.J., Wotton, B.M., 2006.  Forest fires and climate change in the 21 st century.  Mitigation and adaptation strategies for global change 11, 847–859.
  * Fornacca et al. [2017] Fornacca, D., Ren, G., Xiao, W., 2017.  Performance of three MODIS fire products (MCD45A1, MCD64A1, MCD14ML), and ESA Fire_CCI in a mountainous area of northwest yunnan, china, characterized by frequent small fires.  Remote Sensing 9.
  * Giglio [2015] Giglio, L., 2015.  MODIS Collection 6 Active Fire Product User’s Guide Revision A.  http://modis-fire.umd.edu/files/MODIS_C6_Fire_User_Guide_A.pdf.  [Online; accessed 01-March-2019].
  * Giglio et al. [2016] Giglio, L., Schroeder, W., Justice, C.O., 2016.  The collection 6 modis active fire detection algorithm and fire products.  Remote Sensing of Environment 178, 31 – 41.
  * Guerrero [1993] Guerrero, V.M., 1993.  Time-series analysis supported by power transformations.  Journal of Forecasting 12, 37–48.
  * Hantson et al. [2013] Hantson, S., Padilla, M., Corti, D., Chuvieco, E., 2013\.  Strengths and weaknesses of modis hotspots to characterize global fire occurrence.  Remote Sensing of Environment 131, 152 – 159.
  * Hyndman and Athanasopoulos [2018] Hyndman, R., Athanasopoulos, G., 2018\.  Forecasting: principles and practice.  OTexts.
  * Hyndman and Khandakar [2008] Hyndman, R.J., Khandakar, Y., 2008\.  Automatic time series forecasting: the forecast package for R.  Journal of Statistical Software 26, 1–22.
  * Hyndman and Koehler [2006] Hyndman, R.J., Koehler, A.B., 2006\.  Another look at measures of forecast accuracy.  International Journal of Forecasting 22, 679 – 688.
  * Justice et al. [2002] Justice, C., Townshend, J., Vermote, E., Masuoka, E., Wolfe, R., Saleous, N., Roy, D., Morisette, J., 2002\.  An overview of modis land data processing and product status.  Remote Sensing of Environment 83, 3 – 15.
  * Kelly et al. [2013] Kelly, R., Chipman, M.L., Higuera, P.E., Stefanova, I., Brubaker, L.B., Hu, F.S., 2013\.  Recent burning of boreal forests exceeds fire regime limits of the past 10,000 years.  Proceedings of the National Academy of Sciences 110, 13055–13060.
  * Liboschik et al. [2017] Liboschik, T., Fokianos, K., Fried, R., 2017.  tscount: An R package for analysis of count time series following generalized linear models.  Journal of Statistical Software, Articles 82, 1–51.
  * Livera et al. [2011] Livera, A.M.D., Hyndman, R.J., Snyder, R.D., 2011.  Forecasting time series with complex seasonal patterns using exponential smoothing.  Journal of the American Statistical Association 106, 1513–1527.
  * M. C. Pereira et al. [2008] M. C. Pereira, J., Trigo, R., Camara, C., Oom, D., Mota, B., 2008.  Global fire activity patterns (1996-2006) and climatic influence: an analysis using the world fire atlas.  Atmos. Chem. Phys., , 1911–1924.
  * Marcos et al. [2015] Marcos, R., Turco, M., Bedía, J., Llasat, M.C., Provenzale, A., 2015.  Seasonal predictability of summer fires in a mediterranean environment.  International journal of wildland fire 24, 1076–1084.
  * Oom and Pereira [2012] Oom, D., Pereira, J.M., 2012\.  Exploratory spatial data analysis of global MODIS active fire data.  International Journal of Applied Earth Observation and Geoinformation 21, 326–340.
  * Peel et al. [2007] Peel, M.C., Finlayson, B.L., McMahon, T.A., 2007.  Updated world map of the köppen-geiger climate classification.  Hydrology and earth system sciences discussions 4, 439–473.
  * R Core Team [2017] R Core Team, 2017.  R: A Language and Environment for Statistical Computing.  R Foundation for Statistical Computing. Vienna, Austria.
  * Rabin et al. [2017] Rabin, S.S., Melton, J.R., Lasslop, G., Bachelet, D., Forrest, M., Hantson, S., Kaplan, J.O., Li, F., Mangeon, S., Ward, D.S., Yue, C., Arora, V.K., Hickler, T., Kloster, S., Knorr, W., Nieradzik, L., Spessa, A., Folberth, G.A., Sheehan, T., Voulgarakis, A., Kelley, D.I., Prentice, I.C., Sitch, S., Harrison, S., Arneth, A., 2017.  The fire modeling intercomparison project (firemip), phase 1: experimental and analytical protocols with detailed model descriptions.  Geoscientific Model Development 10, 1175–1197.
  * Roads et al. [2005] Roads, J., Fujioka, F., Chen, S., Burgan, R., 2005\.  Seasonal fire danger forecasts for the usa.  International Journal of Wildland Fire 14, 1–18.
  * Roads et al. [2010] Roads, J., Tripp, P., Juang, H., Wang, J., Fujioka, F., Chen, S., 2010\.  Ncep–ecpc monthly to seasonal us fire danger forecasts.  International Journal of Wildland Fire 19, 399–414.
  * Spessa et al. [2015] Spessa, A.C., Field, R.D., Pappenberger, F., Langner, A., Englhart, S., Weber, U., Stockdale, T., Siegert, F., Kaiser, J.W., Moore, J., 2015\.  Seasonal forecasting of fire over kalimantan, indonesia.  Natural Hazards and Earth System Sciences , 429–442.
  * Taylor and Letham [2018a] Taylor, S., Letham, B., 2018a.  Prophet: Automatic Forecasting Procedure.  R package version 0.3.0.1.
  * Taylor and Letham [2018b] Taylor, S.J., Letham, B., 2018b.  Forecasting at scale.  The American Statistician 72, 37–45.
  * Turco et al. [2018] Turco, M., Jerez, S., Doblas-Reyes, F.J., AghaKouchak, A., Llasat, M.C., Provenzale, A., 2018\.  Skilful forecasting of global fire activity using seasonal climate predictions.  Nature Communications 9, 2718\.
  * Westerling et al. [2003] Westerling, A.L., Gershunov, A., Brown, T.J., Cayan, D.R., Dettinger, M.D., 2003.  Climate and wildfire in the western united states.  Bulletin of the American Meteorological Society 84, 595–604.