# Rolling the Dice: Imagining Generative AI as a
Dungeons & Dragons Storytelling Companion

Jose Ma. Santiago III  0002-4730-7865 Paris-Lodron Universität
SalzburgSalzburgAustria Salzburg University of Applied SciencesSalzburgAustria
<EMAIL_ADDRESS>,  Richard Lance Parayno  1234-5678-9012 Paris-
Lodron Universität SalzburgSalzburgAustria Salzburg University of Applied
SciencesSalzburgAustria<EMAIL_ADDRESS>,  Jordan Aiko Deja
0001-9341-6088 De La Salle UniversityManilaPhilippines University of
PrimorskaKoperSlovenia6000<EMAIL_ADDRESS>and  Briane Paul V. Samson
0002-0253-452X De La Salle UniversityManilaPhilippines
<EMAIL_ADDRESS>
(2023)

###### Abstract.

AI Advancements have augmented casual writing and story generation, but their
usage poses challenges in collaborative storytelling. In role-playing games
such as Dungeons & Dragons (D&D), composing prompts using generative AI
requires a technical understanding to generate ideal results, which is
difficult for novices. Thus, emergent narratives organically developed based
on player actions and decisions have yet to be fully utilized. This paper
envisions the use of generative AI in transforming storytelling into an
interactive drama using dynamic and immersive narratives. First, we describe
scenarios where narratives are created and character conversations are
designed within an overarching fantasy disposition. Then, we recommend design
guidelines to help create tools using generative AI in interactive
storytelling. Lastly, we raise questions on its potential impact on player
immersion and cognitive load. Our contributions may be expanded within the
broader interactive storytelling domain, such as speech-conversational AI and
persona-driven chatbots.

Generative AI, AI Storytelling, AI usability, Dungeons and Dragons

††copyright: ccbysa††journalyear: 2023††conference: Generative AI and HCI;
April 28, 2023; Hamburg, Germany††booktitle: Generative AI and HCI ’23
Workshop, April 28, 2023, Hamburg, Germany††ccs: Human-centered computing
Natural language interfaces††ccs: Human-centered computing Collaborative
content creation††ccs: Computing methodologies Discourse, dialogue and
pragmatics

Figure 1. A future where AI works along side dungeon masters as a storytelling
companion in Dungeons & Dragons.

lol

##  1\. Introduction and Background

In recent years, AI-based collaborative writing tools have been rolled out in
various application areas such as argumentative writing (Lee et al., 2022),
code synthesis (Austin et al., 2021), drama (Flowers et al., 2023) and
storytelling (Fotedar et al., 2020; Yuan et al., 2022). These systems are
built using large Language Models (LMs) that take prompts from the user and
return plausible continuations (Brown et al., 2020). With the use of these
tools, authors who struggle most especially during a “writer’s block” may draw
inspiration and benefit from the help of these tools. The writer takes control
of their craft by deciding on their persona, style and identity in their
written work. Most especially in drama and storytelling, conveying a narrative
through spoken or written word requires the presence of culture and tradition
(Greene, 1996), which is usually not seen in most AI-generated content.

Specifically, in the context of tabletop role-playing games (TTRPGs) like
Pathfinder and Dungeons & Dragons (D&D), a dungeon master (DM) directs the
flow of a game’s story and leaves room for the players to interact using
personas that fit the context of the narrative (Walden, 2015). This is done
throughout a campaign divided into game sessions, usually held weekly. This is
different from traditional storytelling, where the practice of conveying a
narrative or message through spoken or written words is a critical component
(Greene, 1996).

While this setup has been the standard practice more recently, despite the
platform undergoing multiple iterations or editions (Walden, 2015), it is
unclear whether this is an ideal or usable approach. In a study by Reinikainen
(Reinikainen, 2008), players encounter difficulty immersing in the game due to
the complexity of the recent editions. Such factors may affect the overall
experience of the game, especially its usability. The players pointed out that
the lack of immersion led to diminished enjoyment and increased frustrations,
which undermined the reason why they played. This leaves more room for
improvement towards better immersive experiences for players and narrators
while reducing cognitive load.

Recent work in generative AI has tried to address these gaps through
collaborative storytelling, from using LMs to spark creativity in authors
(Yuan et al., 2022; Osone et al., 2021) to generative games (Kreminski et al.,
2019). This is further improved by recent developments in Natural Language
Processing (NLP). Storytelling AI aims to generate engaging narratives with
visual illustrations to immerse the user with minimal user input (Fotedar et
al., 2020). The tool creates narrative text and complements images by
combining two generative models. By using new large LMs such as GPT-3, we also
have the added functionality of text-to-image generation (Ramesh et al.,
2021). With the two components set, we can tackle the question, how can
generative AI enhance the creativity and flexibility of D&D storytelling?
Thus, this paper presents the following contributions: (1) We provide sample
scenarios in D&D where we use generative AI (using prompts) as a form of
collaborative storytelling towards improved user engagement. (2) We provide
design guidelines towards expanding the space of immersive collaborative AI-
based storytelling scenarios in D&D and beyond.

##  2\. D&D Scenarios

With the possibilities afforded by large LMs, we imagine storytelling
scenarios in D&D with an AI companion named Avalon. We provide sample prompts
and outputs and discuss them in the form of guidelines in the succeeding
sections.

###  2.1. Scenario 1: Where it begins…

Prompt 1: The adventurers reach a fork in the dungeon, but instead of picking
one of two paths, they make a third path”

As a dungeon master, you run through the dungeon that will be the initial
setting for your first D&D session, as it is your job to control the flow of
the narrative of the session (Walden, 2015). As your players eventually
arrive, you go through your notes one last time as they settle down and get
into character. You elaborately describe the scenario they are in waking up in
a dimly lit dungeon with two paths before them. You can see from their faces
the shock from your revelation, as one of them proposes they should get out as
soon as possible. You ask them what they wish to do, and Gunter the dwarf
replies: I use my bombs to make a third path! You did not prepare for this,
but your observant companion did. You quickly scramble to describe to Avalon
the current situation (see first prompt in quote). Your companion has been
intently listening to the actions of the players and was able to incorporate
the actions of Gunter the dwarf and how it affects the world around him to
match. This way, their story progresses while incorporating narrative
techniques to keep the players engaged as they dredge forward into the
dungeon!

###  2.2. Scenario 2: Continuing where you last left…

Prompt 2: “Can you give me merchants with their own personalities from the
merchant capital in my D&D campaign?”

After the party had defeated the gorgon, the keeper of the treasure of Lord
Nuk’thali, they decided to visit the nearest city to purchase new gear. You
have had the idea of a merchant capital, bustling with trade and commerce,
where merchants and adventurers are free to deal as they please. You know that
your players would like to chat with the locals and barter for goods, but you
need help coming up with names and personalities for each merchant. You rack
your brain, trying to think of familiar-sounding names of humans that you
never considered that a merchant capital would likely have a more diverse
group of locals. You look over to your companion as you describe your vision
of the city to them (see second prompt in quote). You are amazed by the
diversity of the cast that your companion has provided. It has sparked
creativity within you to start writing their backstories and some dialogue
based on their personalities. And with that, you are prepared for the next
session, eager to find out what your players think of the merchants that await
them!

###  2.3. Scenario 3: Here we go again…

Prompt 3: “Can you generate the art for the fierce battle between the
adventuring party and Lord Nuk’thali in the crypt of bones, fantasy style..”

Figure 2. Generated images based on the bottom prompt shown in the quotes.
This image was generated using DALL-E (https://openai.com/dall-e-2/).

Insert image of Dall-e creation

It is the last session of the campaign, as you prepare the scene of the final
battle with Lord Nuk’Thali. To the eyes of a mortal, it is but a crudely made
landscape with cardboard spikes. But to the eyes of the players, it is the
Crypt of Bones, lair of the evil lord itself. Eventually, your players arrive
marvelling at the scenery as the culmination of all of their efforts thus far.
They get into character as you describe what lies before them. Slowly, a
figure steps forward, revealing itself to be none other than Lord Nuk’Thali,
the final boss and master of the crypt. In the battle, the adventurers face
the powerful lich Lord Nuk’Thali and endure a fierce exchange of attacks that
lasts for a while. However, they ultimately manage to destroy his phylactery,
which leads to his defeat and disintegration. The adventurers emerge
victorious but exhausted from the battle. You peek over your DM screen as you
see the faces of your players as they cheer and kick after the the most
exciting battle they ever had. You know they will remember this campaign
forever. To commemorate the event, you look over your companion and asked the
third prompt. Given that LMs are capable of remembering previous prompts and
answers (Brown et al., 2020), your companion was able to generate a scenery (
Figure 2) that incorporates the story so far. Your party gathers to inspect
the illustration as they all decide to frame it, as a remembrance of their
very first Dungeons & Dragons campaign. Eventually, you start a new campaign
with a new group accompanied by their own challenges. Despite having run
multiple sessions with various groups of players, you can still look over your
dining room table and reminisce the experiences you had looking at the framed
portrait on the wall.

##  3\. Discussion

Like other AI collaborative writing tools, AI wears different hats at each
stage of the writing process (Yuan et al., 2022) such as an idea generator,
scene interpolator and copy editor. In D&D, however, the narrative is built on
the spot as the players explore the fantasy world. Thus, some hats require
more attention, precisely that of an idea generator and scene interpolator.
From the scenarios above, it is not feasible for the DM to prepare for
everything that can happen during a D&D session. This is why we turn to tools
and companions to aid the common shortcomings and pitfalls they encounter. By
giving the role to the AI as an idea generator and scene interpolator, we
avoid scenarios where they get stuck on ideas and narratives. We present some
guidelines to prepare developers and designers to help casual players and
writers in similar contexts.

###  3.1. Understanding Context

D&D vastly differs from generative AI platforms such as ChatGPT (Nakano et
al., 2022). Thus, there is a need to have a good understanding of the mode of
communication used in TTRPGs. This can be done by analyzing campaigns such as
Critical Role, an unscripted live stream of voice actors playing D&D led by DM
Matthew Mercer (Role, 2023). The Critical Role Dungeons & Dragons Dataset
(CRD3) analyzed 159 episodes across two campaigns that resulted in 398,682
turns of dialogue (Rameshkumar and Bailey, 2020). By understanding this mode
of communication, designers can empathize with the users they are designing
for. With the current design of LMs, we are limited to $1:1$ interactions with
the user. Incorporating multiple users in a collaborative AI environment may
create responses that align with the users’ expectations. Audio should also be
considered a potential input method, as conversations between players and even
with the DM could be crucial to understanding the context. However, further
improvements with the structure of LMs might be needed to filter out
unnecessary inputs or prompts as chatter is quite frequent in D&D sessions as
seen in the dataset (Rameshkumar and Bailey, 2020).

However, understanding the context of the narrative continues beyond just the
AI (Sidhu and Carter, 2021). Another aspect that should be considered is how
to help the user understand the context. This can be done in one of two ways,
either directly or indirectly. The first method can be done by the narrator,
such as describing the scenery of the landscape the players are exploring.
However, the narrative becomes stale and boring if the players are just given
the information. This is where the second method can be utilized, which can be
done through NPCs and other entities in the world. The players could further
understand the world from a local perspective by conversing with them.
Posters, journals, and even gossip can indirectly give the players more
context. This guideline tackles the issue of understanding the context and the
mode of communication. Since D&D uses a setup that involves multiple users
talking at the same time, an understanding of the mode of communication is
important to grasp the context of the game. With this, we can increase the
chances of generating the desired output when consulting with an AI
collaborator.

###  3.2. Putting Value into Inspirations

It is common practice that DMs turn to existing forms of traditional
storytelling, such as fantasy novels and movies, for inspiration (Slavicsek
and Baker, 2006). DMs often incorporate existing themes, plot hooks, villain
motivations, and even character tropes in the context of their fantasy world.
By utilizing LMs to integrate themes from an extensive collection of fantasy
novels and other forms of media, we can reduce predictable narratives from
occurring frequently. While this does not eliminate these instances, having a
diverse collection of references proves useful for many reasons. Diversity and
inclusion in D&D are other aspects rarely touched upon but have significantly
changed in recent years. A comparative study illustrates how official D&D
content (pre-made adventures, source books, etc.) has struggled with the
presentation of minority groups with each iteration (Long, 2016). This can be
addressed by having a diverse repertoire of literary works. Recently, there
has been a challenge to incorporate diversity in science fiction and fantasy-
based young adult literature (Garcia, 2017). Diversity impacts not only the
nature of the narratives that can be generated but also the overall experience
of the players and DMs.

Another dimension that is worth exploring further is the presence of organic
inspiration. DMs have often used real-life experiences to create diverse
narratives that incorporate their tastes. This is especially important when
envisioning the fantasy world’s NPCs and their dialogue and personalities.
Thus, generative AI must also be able to consider these attributes in their
implementation. Existing LMs can empathize and respond through affective text
(Ghosh et al., 2017). However, emotion encoding is not yet well defined, and
some studies have linked our emotional response to our personality types
(Gilboa and Revelle, 1994). Several models on these can be explored further to
consider these dimensions. This guideline highlights the importance of
inspiration from a diverse collection of works. We aim to limit the occurrence
of stale narratives that have been exhausted in previous works and encourage
the exploration of emotion encoding and how it can be refined in generative
AI.

###  3.3. Maximizing Engagement

Meaningful play is the idea that games can create experiences beyond the
typical leisurely activity (Tekinbas and Zimmerman, 2003). Relationships
between player actions and the outcome must be discernible and integrated into
the larger context of the game. Player agency is a concept in game studies as
the fundamental capability of the player to act in the world (Stang, 2019).
This concept also applies in D&D as players and DMs shape the narrative.
Meaningful play and player agency work together to engage and immerse the
players in the world. One approach that can be used is by emphasizing the
interactive world. In D&D, the theatre of the mind is a concept where players
or the DM would describe a scene in vivid detail using the participants’
imagination to visualize the scenario they are in (Svan and Wuolo, 2021). If
the AI can describe the scenario the players are in and hint about key areas
that players can explore or interact with, then their curiosity would draw
them in. It helps by adding pauses or asking the players what they want to do
in that situation. This ensures that player agency is still present despite
the AI controlling the flow of the narrative.

Another approach would be to highlight the consequences of each player’s
action in the world. This can range from small cause-and-effect narratives,
such as a player cutting down a tree in a forest, causing a loud noise and
wildlife to flee from it to more lasting ones such as the death of a
character. While one can argue that death is superficial in the context of a
fantasy narrative, the actions and experiences that lead to death add weight
to the player’s choices (Flynn-Jones, 2015). A pre-play interview was
conducted to understand meaningful play in D&D and reports that death was the
most common meaningful play experience (Sidhu and Carter, 2021). This aspect
in D&D can potentially transcend the traditional definition of meaningful play
to deeper emotions, thus affecting engagement.

We expand further by incorporating external aids. In one of the scenarios, we
portrayed the final session of a D&D campaign, where players eventually defeat
the villain of the narrative and save the day. This is usually portrayed in an
epic battle of conflicting morals, where the players’ actions coalesce into a
magnificent display in the theatre of the mind. Using this battle, the
companion in the scenario was able to generate illustrations that captured the
essence and emotions of players during the fight. This adds engagement outside
the narrative by using visual aid such as illustrations, audio and even
tactile aid to immerse the players in the narrative. By incorporating these
approaches to the narrative style created by the AI and the DM, we expect
players to feel more connected to the overarching fantasy world. These aspects
can be visualized as threads: the more threads the player has (player agency,
understanding of consequences, emotional ties), the stronger the player’s
connection with the world and the narrative. By ensuring that player agency is
present and has reasonable consequence, we can further immerse the players
within the setting of the overarching fantasy world.

##  4\. Conclusion

As we move towards innovations in the generative AI landscape, we urge
designers and developers to reflect on the users rather than the capabilities
of the tools. We have the technology needed to create engaging experiences
based on existing research. Yet, more work must be done before we deploy them
to our users. In this opinion piece, we used the example of Dungeons & Dragons
as a domain where non-technical users may reside. By envisioning the use of
generative AI, we push the boundaries by designing systems that enable our
users to utilise unfamiliar tools that may be able to help them. The scenarios
follow you as the dungeon master as you prepare and conduct D&D sessions with
your playmates. We highlighted common pain points that DMs encounter and
described how generative AI could help. From these scenarios, we created an
initial list of design guidelines (e.g. understanding context, putting value
into inspirations, maximising engagement) that may help developers and
designers integrate generative AI for interactive storytelling purposes. Our
design guidelines focus on the interaction between AI, the DM and the other
players in the game. We call for future explorations on implementing and
validating whether the design guidelines create better immersive experiences
in broader application areas outside of D&D. We also look at how this aids in
designing collaborative storytelling as a tool for non-technical writers to
utilise in their works. As designers, we have the task of evaluation and
empathy, regardless of the application area. We hope that with the perspective
of D&D dungeon masters and players, we can work towards a user-centered future
in generative AI.

## References

  * (1)
  * Austin et al. (2021) Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. 2021.  Program Synthesis with Large Language Models.  https://doi.org/10.48550/arXiv.2108.07732 arXiv:2108.07732 [cs].
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language Models are Few-Shot Learners. In _Advances in Neural Information Processing Systems_ , Vol. 33. Curran Associates, Inc., 1877–1901.  https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html
  * Flowers et al. (2023) Amanda Flowers, Brian Magerko, and Punya Mishra. 2023\.  Gamemasters and Interactive Story: A Categorization of Storytelling Techniques in Live Roleplaying.  (Feb. 2023).
  * Flynn-Jones (2015) Emily Flynn-Jones. 2015\.  Don’t Forget to Die: A Software Update is Available for the Death Drive.  In _The Dark Side of Game Play_. Routledge.  Num Pages: 14.
  * Fotedar et al. (2020) Sonali Fotedar, Koen Vannisselroij, Shama Khalil, and B. Ploeg. 2020.  Storytelling AI: A Generative Approach to Story Narration.  https://www.semanticscholar.org/paper/Storytelling-AI%3A-A-Generative-Approach-to-Story-Fotedar-Vannisselroij/d78090bdac690c1254ef1725c1c8330dbc76a27a
  * Garcia (2017) Antero Garcia. 2017\.  Worlds of Inclusion: Challenging Reading, Writing, and Publishing Science Fiction– and Fantasy-Based Young Adult Literature.  _Journal of Adolescent & Adult Literacy_ 61, 2 (2017), 221–224.  https://doi.org/10.1002/jaal.676 _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/jaal.676.
  * Ghosh et al. (2017) Sayan Ghosh, Mathieu Chollet, Eugene Laksana, Louis-Philippe Morency, and Stefan Scherer. 2017\.  Affect-LM: A Neural Language Model for Customizable Affective Text Generation.  https://doi.org/10.48550/arXiv.1704.06851 arXiv:1704.06851 [cs].
  * Gilboa and Revelle (1994) Eva Gilboa and William Revelle. 1994.  Personality and the Structure of Affective Responses.  In _Emotions_. Psychology Press.  Num Pages: 26.
  * Greene (1996) Ellin Greene. 1996\.  _Storytelling: Art and Technique: Art and Technique, Third Edition_.  ABC-CLIO.  Google-Books-ID: 5P4KOY1h1KYC.
  * Kreminski et al. (2019) Max Kreminski, Devi Acharya, Nick Junius, Elisabeth Oliver, Kate Compton, Melanie Dickinson, Cyril Focht, Stacey Mason, Stella Mazeika, and Noah Wardrip-Fruin. 2019.  Cozy mystery construction kit: prototyping toward an AI-assisted collaborative storytelling mystery game. In _Proceedings of the 14th International Conference on the Foundations of Digital Games_ _(FDG ’19)_. Association for Computing Machinery, New York, NY, USA, 1–9.  https://doi.org/10.1145/3337722.3341853
  * Lee et al. (2022) Mina Lee, Percy Liang, and Qian Yang. 2022.  CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities. In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_ _(CHI ’22)_. Association for Computing Machinery, New York, NY, USA, 1–19.  https://doi.org/10.1145/3491102.3502030
  * Long (2016) T Long. 2016.  Character creation diversity in gaming art.  _International Journal of Role-Playing_ 1, 7 (2016), 23–29.
  * Nakano et al. (2022) Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. 2022\.  WebGPT: Browser-assisted question-answering with human feedback.  https://doi.org/10.48550/arXiv.2112.09332 arXiv:2112.09332 [cs].
  * Osone et al. (2021) Hiroyuki Osone, Jun-Li Lu, and Yoichi Ochiai. 2021.  BunCho: AI Supported Story Co-Creation via Unsupervised Multitask Learning to Increase Writers’ Creativity in Japanese. In _Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems_ _(CHI EA ’21)_. Association for Computing Machinery, New York, NY, USA, 1–10.  https://doi.org/10.1145/3411763.3450391
  * Ramesh et al. (2021) Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. 2021\.  Zero-Shot Text-to-Image Generation.  https://doi.org/10.48550/arXiv.2102.12092 arXiv:2102.12092 [cs].
  * Rameshkumar and Bailey (2020) Revanth Rameshkumar and Peter Bailey. 2020.  Storytelling with Dialogue: A Critical Role Dungeons and Dragons Dataset. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Online, 5121–5134.  https://doi.org/10.18653/v1/2020.acl-main.459
  * Reinikainen (2008) Arttu Reinikainen. 2008\.  _Role-playing Games and Usability–The Effects of Design on the Experience of Play_.  Master’s thesis.
  * Role (2023) Critical Role. 2023\.  Critical Role.  https://critrole.com/.  [Accessed on February 22, 2023].
  * Sidhu and Carter (2021) Premeet Sidhu and Marcus Carter. 2021.  Pivotal Play: Rethinking Meaningful Play in Games Through Death in Dungeons & Dragons.  _Games and Culture_ 16, 8 (Dec. 2021), 1044–1064.  https://doi.org/10.1177/15554120211005231 Publisher: SAGE Publications.
  * Slavicsek and Baker (2006) Bill Slavicsek and Richard Baker. 2006.  _Dungeon Master For Dummies_.  John Wiley & Sons.  Google-Books-ID: pSG3zxln4FUC.
  * Stang (2019) Sarah Stang. 2019\.  “This Action Will Have Consequences”: Interactivity and Player Agency.  _Game Studies_ 19, 1 (May 2019).  https://gamestudies.org/1901/articles/stang?fbclid=IwAR3-wsiag8BThUFv-CkYjci7qYhpW9wamfDRMNoOjZcq37Jv6_BZUWxT13A
  * Svan and Wuolo (2021) Oscar Svan and Anna Wuolo. 2021.  _Emergent Player-Driven Narrative in Blades in the Dark and Dungeons &amp; Dragons : A Comparative Study_.  http://urn.kb.se/resolve?urn=urn:nbn:se:uu:diva-448012
  * Tekinbas and Zimmerman (2003) Katie Salen Tekinbas and Eric Zimmerman. 2003.  _Rules of Play: Game Design Fundamentals_.  MIT Press.  Google-Books-ID: YrT4DwAAQBAJ.
  * Walden (2015) Corey Ryan Walden. 2015\.  _”A Living and Breathing World…”: examining participatory practices within Dungeons & Dragons_.  Thesis. Auckland University of Technology.  https://openrepository.aut.ac.nz/handle/10292/9100 Accepted: 2015-10-09T03:33:00Z.
  * Yuan et al. (2022) Ann Yuan, Andy Coenen, Emily Reif, and Daphne Ippolito. 2022\.  Wordcraft: Story Writing With Large Language Models. In _27th International Conference on Intelligent User Interfaces_ _(IUI ’22)_. Association for Computing Machinery, New York, NY, USA, 841–852.  https://doi.org/10.1145/3490099.3511105


# Consonant-Vowel Transition Models Based on Deep Learning for Objective
Evaluation of Articulation

Vikram C. Mathad, Julie M. Liss, Kathy Chapman, Nancy Scherer, and Visar
Berisha Vikram C. Mathad is with zapr media labs, Bangalore, India, 560016,
Email<EMAIL_ADDRESS>Julie M. Liss is with the Department of Speech
& Hearing Science, Arizona State University, Tempe, AZ-85281, Email:
<EMAIL_ADDRESS>Kathy Chapman is with the Department of Communication
Sciences and Disorders, University of Utah, Salt Lake City, UT-84112, Email:
<EMAIL_ADDRESS>Nancy Scherer is with the Department of Speech
& Hearing Science, Arizona State University, Tempe, AZ-85281, Email:
<EMAIL_ADDRESS>Visar Berisha is with the College of Health Solutions,
and School of Electrical, Computer, & Energy Engineering, Arizona State
University, Tempe, AZ-85281, Email<EMAIL_ADDRESS>This work is funded in part
by NIH-NIDCR DE026252, NIH-NIDCD R01DC006859, and NIH R21DE023519.

###### Abstract

Spectro-temporal dynamics of consonant-vowel (CV) transition regions are
considered to provide robust cues related to articulation. In this work, we
propose an objective measure of precise articulation, dubbed the objective
articulation measure (OAM), by analyzing the CV transitions segmented around
vowel onsets. The OAM is derived based on the posteriors of a convolutional
neural network pre-trained to classify between different consonants using CV
regions as input. We demonstrate the OAM is correlated with perceptual
measures in a variety of contexts including (a) adult dysarthric speech, (b)
the speech of children with cleft lip/palate, and (c) a database of accented
English speech from native Mandarin and Spanish speakers.

###### Index Terms:

Articulation precision, cleft lip and palate, consonant-vowel transitions,
convolution neural networks, pronunciation scores, dysarthria, and second
language learning.

##  I Introduction

Consonant-vowel (CV) transitions refer to the abrupt movement in articulatory
gestures as a speaker moves from consonants to vowels [1]. While producing CV
transitions the articulators make dynamic movements which result in rapid
changes in the amplitude and spectrum of the acoustic signal. Perceptual and
algorithmic experiments conducted by Stevens concluded that the information in
the CV boundary regions is robust and provides invariant cues related to
articulation [1, 2]. For example, several papers demonstrate that CV
transitions are efficient cues for place of articulation recognition in both
adult and child speakers [3, 4, 5]. The challenging nature of these
articulatory movements amplifies the acoustic manifestation of structural or
functional impairments associated with the speech production mechanism. To
that end, acoustic measures extracted from CV transitions have been analyzed
as correlates of articulatory precision and intelligibility in motor speech
disorders (e.g., apraxia and dysarthria), structural anomalies (e.g., cleft
palate (CP), syndrome-related disorders (e.g., Down syndrome and
galactosemia), and sensory-based disorders (e.g., hearing impairment) [6, 7].

One of the challenges associated with assessment of speech acoustics during CV
transition regions is signal variability. Different error types produce
different acoustic patterns: devoicing and substitution errors in hearing
impairment [8]; compensatory articulation errors (glottal and pharyngeal
substitutions), and nasal substitutions in CP [7]; nasalized consonants with
longer duration in dysarthria [9, 10]; and phoneme substitution errors during
second-language learning [11, 12] all differentially impact speech acoustics
during CV transition regions. We demonstrate this with an example in Fig. 1.
In this figure we analyze the waveform and spectrogram of various types of
articulation errors produced for the target stop consonant /p/. The waveform
and spectrogram (Figs. 1(a) and (b)) of the control speaker show a clear
closure-burst transition for /p/. The dysarthric speakers show the presence of
multiple bursts (Figs. 1(c) and (d)) and spirantization (Figs. 1(e) and (f)).
111Spirantization refers to noise caused by incomplete articulatory closure
during stop production. The substitution error (/p/$\rightarrow$/k/) produced
by the CP speaker shows the presence of a high-energy burst and increased VOT
in Figs. 1(g) and (h). The presence of noise in the consonant region (Figs.
1(i) and (j)) indicates the presence of nasal air emission. The examples in
Fig. 1 reveal that the acoustic characteristics of articulation errors during
CV transitions vary across speech disorders.

Figure 1:  Different articulation errors produced for the target /p/. (a),
(c), (e), (g), and (i) represent the waveforms of CV units containing target
consonant /p/ produced by control, dysarthric case-1, dysarthric case-2, CP
case-1 and CP case-2, and the corresponding spectrograms are shown in (b),
(d), (f), (h), and (j), respectively. For target /p/, the dysarthric case-1
produced multiple bursts and case-2 produced spirantization error. CP case-1
substituted /k/ for /p/ and case-2 produced weak stop.  Figure 2:  An overview
of the system for computation of the objective articulation measure.

This variability challenges the development of disease and speaker-invariant
algorithms that assess CV transitions as objective proxies for precise
articulation and intelligibility. The present work proposes an objective
system for evaluation of CV transitions using an acoustic model pre-trained on
a large normative speech sample. Consistent with existing works that tie CV
transition errors to reduced articulatory precision, we show that the CV
features produced by our model correlate with perceptual measures that capture
articulation across a variety of speech disorders and accented speech.

###  I-A Related work

The importance of CV transition regions, as highlighted by past experiments,
has spurred the development of acoustic features for characterizing relevant
CV features. Traditionally, these features include a set of primitive acoustic
measures including spectral peak, burst energy, spectral energy, spectral
tilt, voice onset time (VOT), and formant transitions trajectories [13, 14,
15, 16, 17]. In many cases, these acoustic measures have been included as
features in machine learning (ML) systems that objectively assess articulatory
precision. These systems involve the supervised training of classifiers (e.g.
support vector machine (SVM), Gaussian mixture model (GMM)) that map the
features to clinical articulation scores. Most of these ML models [18, 19] are
trained on a set of acoustic features extracted from a particular disorder.
However, as we see in Fig. 1, the nature of CV transition errors varies across
speech disorders, consonant categories, and age groups. For example, VOT and
formant transitions are used to assess oral-laryngeal coordination in the case
of motor speech disorders [20], whereas low-frequency energy is used in the
evaluation of nasalized consonants in CP [18, 14]. Furthermore, the validity
of several of these features critically depends on repeatable estimation of
formant frequencies and amplitudes, a notoriously difficult problem [21].
These challenges make it difficult to develop models optimized for one
disorder that generalize to other disorders.

As an alternative to traditional features based on signal processing, joint
spectro-temporal features have been proposed to model CV transition regions in
machine learning models. The two-dimensional discrete cosine transform
(2D-DCT) is one of the most commonly used approaches to modeling the spectro-
temporal dynamics of CV transition regions [22, 23, 24, 19]. Supervised
learning methods using 2D-DCT features as input have been used for
classification of place of articulation in stop consonants [23], to evaluate
the goodness of /t/ and /k/ productions in children with speech sound
disorders [25], to detect stop consonant production errors [24], and to model
perceptual intelligibility ratings in children with CP [19]. Existing models
based on supervised machine learning using the spectrum as input are trained
for a very narrow context and are unlikely to generalize other contexts.
Developing models that generalize to multiple contexts using this approach
will require large and diverse labeled clinical corpora for training [26], a
challenge too steep to overcome in many clinical applications.

###  I-B The present work

Motivated by the significance of CV transitions and the limited availability
of clinical speech data, in this work we propose a system for automated
assessment of CV transition regions and demonstrate its validity by
correlating the extracted features against perceptual ratings that capture
articulatory precision across several clinical and non-clinical application
domains. In contrast to clinical speech corpora, there are several publicly-
available general population speech corpora. In our algorithm, we make use of
the LibriSpeech corpus and train a discriminative CV feature learning model
that learns to classify different consonants using CV regions as input. We use
this pre-trained model as a CV feature extractor on input clinical speech
samples. We further aggregate these features at the level of an utterance or
speaker to generate an objective articulation measure (OAM). We demonstrate
that the OAM correlates strongly ``out-of-the-box" with perceptual measures of
articulation in several clinical speech applications and accentedness in
speech from two different languages. That is, even without any supervised
training using labels or speech samples from an application of interest, these
features encode acoustic characteristics that correlate with human perception
of properly produced speech. The performance of the OAM further improves with
supervision if perceptual ratings are available for a corpus.

##  II Databases

We use adult speech samples from the Librispeech corpus to build our CNN-based
feature extractor and the OAM. We evaluate the OAM on three different corpora,
two clinical corpora, and an accented speech database. Below we describe the
details of the speech samples and perceptual ratings in each corpus.

###  II-A Librispeech

The Librispeech database [27] contains the utterances of adult American
English speakers. In this work, a subset of the Librispeech database, train-
clean-100 and test-clean are used in the development of the CNN. The training
set contains the recordings of 251 adult speakers (125 male and 126 female). A
separate test set, comprised of 5.4 hours of speech, is used to evaluate the
performance of the CNN classifier.

###  II-B Dysarthria

The Dysarthric speech database was collected at the Motor speech laboratory at
Arizona State University [28]. The dysarthric speech database consists of
speech recordings from 80 adult speakers (43 male and 37 female) of varying
levels of articulation impairment: 38 participants with Parkinson's disease
(PD), 6 participants with Huntington's disease (HD), 16 participants with
cerebellar Ataxia (A), and 15 participants with amyotrophic lateral sclerosis
(ALS), and 5 healthy controls. From each subject, a set of 5 sentences were
recorded. The recordings were given to 14 speech-language pathologists (SLPs)
to rate speaker-level articulatory precision on a scale of 1 to 7 (1-precise
articulation, 7-severely imprecise articulation). The average inter-rater
correlation was found to be 0.831, $p<0.001$. For this study, we take the
average articulatory precision ratings of the 14 SLPs as the speaker-level
articulatory precision score.

###  II-C Cleft speech database

The Utah Americelft cleft speech database is comprised of data from 60
children between the ages of 5-7 years old, each repeating 24 sentences from
the Americleft speech evaluation protocol, recorded at the University of Utah
[29]. Speakers with CP produce obligatory (weak and nasalized consonants) and
compensatory (glottal, pharyngeal, palatalized, and nasal fricative sounds)
errors [7]. These errors lead to articulatory imprecision and degrade the
acceptability of the produced speech. Speaker-level speech acceptability was
rated on a 4-point scale (0-speech is acceptable, 1-speech is mildly
unacceptable, 2-speech is moderately unacceptable, 3-speech is very
unacceptable) by 4 Americleft-trained SLPs, and the average correlation among
the 4 raters was 0.776, $p<0.001$. For this study, we take the average
acceptability of the 4 SLPs as the speaker-level speech acceptability score.

###  II-D Accented speech corpus

The accentedness database used in [11] was used as an additional test corpus.
The accentedness database is a subset of the GMU speech accent archive [30].
In this study, native (L1-language) Mandarin, and Spanish speakers speaking
American English (L2) are considered. The speaker set is the same as the
subset used in the experiments of [11], where for each language, 30 speakers,
balanced on age and gender, were considered. Thirteen native American speakers
were recruited as annotators to rate the accentedness on a 4-point rating
scale: 1 = no accent/negligible accent, 2 = mild accent, 3 = strong accent,
and 4 = very strong accent. The average inter-rater correlation coefficient
was found to be 0.73, $p<0.001$. The average of the ratings among the 13
annotators was computed and considered as the final accentedness score for all
analyses in this paper.

##  III OAM computation

In Fig. 2 we show a block diagram of the framework to compute the OAM. The OAM
computation involves (a) the segmentation of CV transition regions around the
vowel onset, (b) training and validation of a CNN-based consonant classifier
using CV transitions as an input, and (c) articulation score computation. The
different stages involved in the proposed system are described in detail in
the sections that follow.

TABLE I: Details of CNN architecture

Sl. No. | Layer | Activation | Dimension
---|---|---|---
1 | Input | - | 40x32x1
2 | Conv_1 | ReLU | 9x5x64
3 | Pool_1 | - | 2x2
4 | Conv_2 | ReLU | 5x3x64
5 | Pool_2 | - | 2x2
6 | FC_1 | ReLU | 1024
7 | FC_2 | ReLU | 1024
8 | FC_3 | ReLU | 1024
9 | Output | Softmax | 20

###  III-A CV Feature Extractor

We use a pre-trained CNN classifier as a CV feature extractor. The CNN
consonant classifier is trained to classify between different consonants
spoken by healthy individuals using CV transition regions as input. The CNN is
a context-independent consonant classifier that aims to classify among 20
consonants, i.e., plosives: /b/, /d/, /g/, /p/, /t/, /k/, fricatives: /z/,
/v/, /s/, /S/, /f/, /h/, /T/, /D/, affricates: //, //, nasals: /n/, /m/, /ng/,
and liquids: /l/, /r/. The hundred hours of healthy speech samples from the
Librispeech database are used to train the CNN. The speech samples and the
corresponding orthographic transcriptions are first passed through the
Montreal forced alignment tool to obtain the phoneme boundaries [31]. From the
forced-alignment, we determine the location of the vowel onsets. Around each
vowel onset, we take a 160 ms window (80 ms to the right and 80 ms to the left
of the vowel onset) to segment the CV transition region. We empirically
justify the 160 ms window size in Section III-B. The CV transition segment is
analyzed using a Hamming window of 20 ms with a shift of 5 ms. From each
frame, a 40-dimensional log Mel-filter bank is used to extract features, where
the 40 filters are separated equally in Mel scale from 100 Hz to 7800 Hz.
Finally, the segmented CV transition is represented by a 2-dimensional
representation of dimension 40x32.

The mel-spectrogram of CV transition regions is used as input to train the CNN
consonant classifier. The CNN architecture is described in Table I. The model
has two convolutional layers with max pooling, where 64 filters of shape 9x5
with 1x1 stride and 64 filters of shape 5x3 with 1x1 stride are used in the
first and second layers respectively. In both layers, we use a rectified
linear unit (ReLU) activation and a Max-pooling operator of shape 2x2 with a
1x1 stride. The output of convolutional layers is flattened and passed to 3
fully connected layers. Each layer has 1024 hidden neurons with a ReLU
activation function. The output layer is comprised of 20 nodes with softmax
activation, with each output corresponding to one of the 20 consonants. The
error between true and predicted output values is evaluated using a
categorical cross-entropy loss ($L$) given by

$L=-\sum_{i=0}^{B-1}\sum_{j=0}^{M-1}y_{j}^{(i)}log(\hat{y}_{j}^{(i)}),$ (1)

where $B$ refers to the batch size, $M$ is the number of classes ($M=20$), and
$y_{j}$ and $\hat{y}_{j}$ correspond to the ground truth and predicted
outputs. In this work, we chose a batch size of 8 sentences (in each batch, we
use the CV transitions from 8 sentences), and the Adam optimizer is used to
minimize the error between ground truth and predicted values. The network is
trained for 10 epochs and the learning rate is set to 0.001.

###  III-B Setting the window length and evaluating the CNN

We analyzed the dependency of the CNN classifier on CV transition duration. We
vary the CV window around the vowel onset from 60 ms to 200 ms in steps of 20
ms and, for each case, we evaluate the consonant classification accuracy on
the Librispeech test samples. Fig. 3 shows the CNN's performance as a function
of the CV transition window size. From 60 to 160 ms, the performance increases
with transition duration and then plateaus to an accuracy of $\sim$ 85%. The
improvement in performance is the result of improved model accuracy among
consonants like affricates, which require a longer duration to produce and,
therefore require longer windows to discriminate from others. For example, for
a 60 ms transition duration, 47.212% of instances of // are falsely detected
as /S/. Both are similar for place of articulation (palatal-alveolar) but
differ in manner of production (fricative vs affricate). After increasing the
window size to 160 ms, the error rate decreases to 13.283%. The affricate
textipa// is a combination of silence and frication noise. If we consider a
very small CV transition window, then only the frication noise is provided as
input to the CNN. This is insufficient to discriminate between // and /S/,
hence // is confused with /S/.

Figure 3:  Consonant detection rate for different CV transition duration.
Figure 4:  Scatter plots of speaker-level averaged OAM vs. perceptual ratings
for (a) dysarthria, (b) Utah Americleft, and (c) Mandarin and (d) Spanish
databases.  Figure 5:  Scatter plots of predicted vs. perceptual ratings for
(a) dysarthria, (b) Utah Americelft, and (c) Mandarin and (d) Spanish
databases.

###  III-C Articulation Score Computation

The posteriors from the pre-trained CNN across the 20 classes are used as
features for assessing consonant-level articulation scores in clinical speech
and L2 speech. As shown in the block diagram Fig. 2, the input speech is first
force-aligned to obtain the vowel onset locations. We extract a 160 ms CV
segment from the input speech sample (as we did during training) and compute a
mel-spectrogram. The mel-spectrogram is passed through the CNN to compute 20
posterior probability values from the pre-trained model. Let $c_{i}$ denote
the segmented CV transition for the $i^{th}$ target consonant and let
$P=\\{p_{0},p_{1},p_{2},...,p_{19}\\}$ denote a vector of softmax layer
outputs corresponding to the posterior probabilities of the 20 consonants. We
define the OAM of consonant $c_{i}$, $OAM(c_{i})$, as

$OAM(c_{i})=\frac{p_{i}}{max(P)}.$ (2)

For each speaker, the OAM is computed for all instances of target consonants.
The vowel onset and target consonant information are obtained by using the
Montreal forced-aligner [31].

Consonant-level and speaker-level OAM scores: For each speaker, OAM scores are
computed for each consonant instance. Further, these scores are averaged
across all instances of a consonant for each speaker to obtain 20 consonant-
level OAM scores for a speaker. These 20 consonant-level scores are averaged
for each speaker to obtain a speaker-level OAM. The OAM score achieves a
maximum value of 1 when $p_{i}=max(P)$ for all consonants produced by a
speaker.

Linear Models: For each speaker, we obtain 20 consonant-level OAM scores. We
develop domain-specific models of articulation using these 20 features to
predict perceptual variables of interest in each domain: imprecise
articulation in dysarthria, speech acceptability in CP, and accentedness in
speech from L2-learners. The linear models are combined with a forward feature
selection algorithm to learn a parsimonious model. The perceptual labels from
each corpus are used as labels during training.

##  IV Experimental Evaluation

Herein we present the details of the experimental evaluation and results.
First, we evaluate the relationship between the OAM and (a) articulation
precision in dysarthria, (b) acceptability in children with CP, (c)
accentedness in Mandarin, and (d) accentedness in Spanish speakers. Next, we
analyze the effect of forced-alignment errors on the OAM and visualize the
internal workings of the pre-trained DNN feature extractor via saliency maps.
Finally, we compare the OAM with the goodness of pronunciation (GOP)
algorithm, a commonly used algorithm for objective articulation assessment.

###  IV-A Comparing the OAM scores to perceptual labels

We demonstrate the validity of the OAM by comparing the acoustic-derived
features with perceptual measures of articulatory precision in dysarthria,
speech acceptability in cleft palate, and accentedness in Mandarin and Spanish
speakers. We evaluate the correlation between acoustics and perceptual
measures for two types of acoustic measures: (a) speaker-level OAM scores and
(b) linear predictors of perceptual labels using the OAM computed at the
consonant level. The scatter plots of speaker-level OAM vs. perceptual ratings
for Dysarthria, Utah Americleft, Mandarin, and Spanish are shown in Fig.
4(a)-(d), respectively. As expected, the higher the OAM, the closer to normal
the articulatory precision ratings (1 on the scale).

The linear model's performance is evaluated using the leave-one-speaker-out
(LOSO) method. In LOSO validation, the consonant-level OAM scores of all
speakers except one are used to train the linear model, and the held-out
speaker is used during testing. A scatter plot between the predicted scores
and perceptual ratings for Dysarthria, Utah Americleft, Mandarin, and Spanish
are shown in Fig. 5(a)-(d), respectively.

The results for the average OAM and the linear model are presented in Table
II. Across all four databases, the averaged OAM scores showed a significant
correlation with the perceptual ratings, despite the fact that the training of
the CNN does not require any perceptual labels. As expected, the linear model-
based approach shows improvement over the average OAM.

TABLE II:  Results of Speaker-level OAM

Database | Measure |  | Average
---
(r)
| Linear
---
(r)
Dysarthria | OAM | -0.829 | 0.836
| $G_{C}$ | -0.795 | 0.796
| $G_{C+V}$ | -0.818 | 0.848
CP | OAM | -0.659 | 0.723
| $G_{C}$ | -0.689 | 0.725
| $G_{C+V}$ | -0.641 | 0.805
Mandarin | OAM | -0.609 | 0.812
| $G_{C}$ | -0.526 | 0.583
| $G_{C+V}$ | -0.597 | 0.730
Spanish | OAM | -0.645 | 0.788
| $G_{C}$ | -0.747 | 0.717
| $G_{C+V}$ | -0.744 | 0.727

###  IV-B The impact of alignment errors on the OAM

Computation of the OAM requires detection of vowel onsets, which are obtained
automatically using forced-alignment. Forced alignment algorithms can be error
prone, i.e., there are timing differences between manual and automatic
alignments [32]. To evaluate the impact of forced alignment error on the OAM,
we evaluate the OAM scores using both automatic and manual alignments. Manual
alignments are obtained by hand-correction of the forced alignments resulting
from the Montreal aligner. The phoneme-boundaries of Praat-textgrid files are
corrected by listening to the audio samples individually using Praat. The
distribution of alignment error for the different databases are shown in the
second column of Table III. This table also shows the correlation values for
the predicted scores obtained for manual and automatic alignments.

The results for automatic alignments are consistently greater than for the
manual case. This is likely because the alignment error increases as the
speech severity increases. Hence, the CV transition region segmented using
incorrectly detected vowel onsets deviates from their target, and the
alignment error contributes to the estimated OAM in addition to acoustic
deviation induced by articulation error [33].

TABLE III:  Comparison of OAM for manual and automatic alignments  Database |  | Alignment error
---
(Mean+std) ms
| Manual
---
(r)
| Automatic
---
(r)
Dysarthria | 22.502+28.501 | 0.766 | 0.836
CP | 32.891+41.429 | 0.707 | 0.712
Mandarin | 18.870+19.263 | 0.824 | 0.868
Spanish | 18.832+21.680 | 0.605 | 0.802

###  IV-C Understanding the spectral characteristics the OAM captures

We used guided backpropagation to obtain the saliency maps for visualizing the
features relevant inside the pre-trained CNN feature detector [34]. Fig. 6
shows the mel-spectrograms and the corresponding saliency maps, obtained from
the first convolutional layer, for healthy controls and dysarthric speakers
for the target /p/. Fig. 6(a) shows the speech waveform produced by a healthy
control for the CV transition /p/-//; the corresponding mel-spectrogram and
the saliency maps are shown in Figs. 6(b) and (c), respectively. The mel
spectrogram (Fig. 6(b)) indicates a sharp increase in energy at the burst
onset followed by formant transitions. The time-frequency components around
the burst onset and the formant transitions around the vowel onset are
highlighted in the saliency map (Fig. 6(c)) as relevant features. In
dysarthric case-1, the target /p/ was detected as /b/. The place of
articulation remains the same in /p/ and /b/, however /p/ contains a silence
indicating the unvoiced manner and /b/ contains a voice bar component
corresponding to voicing. The saliency map indicates the presence of low-
frequency components corresponding to the voice bar component of /b/. In
dysarthric case-2, /p/ is substituted by /v/. Hence, the CNN recognizes the
target /p/ as /v/. The saliency map for /p/$\rightarrow$/v/ does not indicate
the presence of a burst component as in Figs. 6(c) and (f). Instead, the
formant transitions from the frication to the vowel (around 50 ms in Fig.
6(i)) are highlighted. This analysis provides evidence that the CNN learned
the discriminate features across various consonants and hence, is able to
capture the acoustic features associated with articulation errors during these
regions.

Figure 6:  Saliency maps for the target CV transition . (a) waveform, (b) Mel-
spectrogram, and (c) saliency map for control. (d) waveform, (e) Mel-
spectrogram, and (f) saliency map for dysarthric case-1 where /p/ was detected
as /k/ by CNN. (g) waveform, (h) Mel-spectrogram, and (i) saliency map for
dysarthric case-2 where /p/ was detected as /v/ by CNN. The vowel onset is
indicated by red dotted line each waveform plots ((a), (b), and (c)). The
saliency maps high lights the burst onset in control and Dys-1 and formant
transitions in control, Dys-1 and Dys-2 cases. The phonetic symbols in the
plots are in the ARPABET encoded form. ARPABET to IPA mapping:
P$\rightarrow$/p/, B$\rightarrow$/b/, V $\rightarrow$/v/, and
ER$\rightarrow$//

###  IV-D Comparison between the OAM and the GOP

The performance of the proposed approach is compared with the GOP algorithm
[12]. The GOP is a popular method for the assessment of phone-level
articulatory precision in L2-learners. The phone-level likelihood scores
computed from the automatic speech recognizer (ASR) are used for the
computation of GOP, where the ASR is trained for healthy controls. Since, the
OAM is also computed using the acoustic model trained on healthy speakers, we
compared the performance of the OAM with the GOP.

We used a pre-trained DNN-HMM acoustic model
222https://s3.amazonaws.com/kaldi-dnn-ali-gop-models/dnn_model.tar.gz for GOP
computation. The DNN-HMM model uses $i$-vector speaker adaptation. The model
was trained using Kaldi scripts [35] and the CMU pronunciation dictionary
[36]. The GOP scores are computed using the scripts available
at333https://github.com/tbright17/kaldi-dnn-ali-gop.

One important difference between the two algorithms is that the OAM is
intended as a feature extractor for CV transitions only, whereas the GOP is
computed for each phoneme. As a result, we perform the comparison by
separately considering the GOP of consonants and the GOP of vowels and
consonants. $G_{C}$ refers to the GOP scores evaluated for consonants only and
$G_{C+V}$ refers to the GOP computed for consonants and vowels. Similar to the
OAM, the performance of the GOP is evaluated in two ways, i.e., averaged GOP
scores across the different phonemes and a linear model-based prediction built
for each corpus. The performance of the GOP and the OAM for different
databases are shown in Table II. As with the OAM, the GOP also shows improved
performance for the linear model when compared to simple averaging.

The OAM shows better performance than $G_{C}$ in dysarthria, Mandarin and
Spanish databases; however, the performance is comparable in the case of the
CP database. Both GOP and OAM are computed using the acoustic models trained
on the same Librispeech corpus. The presence of an acoustic mismatch between
the adult and CP databases (which is comprised of children) can affect the
estimated GOP scores. However, the DNN acoustic model uses $i$-vector based
speaker adaptation and that can reduce the impact of the adult-child acoustic
mismatch. The OAM scores, on the other hand, are computed directly on the
child speech without any kind of speaker adaptation; still the performance is
comparable to the DNN that uses $i$-vector speaker adaptation. This result
provides further evidence for the invariance argument made in the literature
regarding CV transitions. In the case of both adults and children, the
perception of place of articulation for nasals and stop consonants is more
efficient for CV transitions than consonant regions alone [3].

Figure 7:  Boxplots of coefficient of variation for (a) dysarthria, (b) Utah
Americleft, (c) Mandarin, and (d) Spanish databases ( $**p<0.001$)

Comparison of OAM and GOP variability: Obtaining reliable GOP scores requires
considerable averaging as the individual frame-level scores can be highly
variable. This impacts the length of input speech required to obtain reliable
estimates of the GOP. To evaluate the variability of the OAM relative to the
GOP, we compare the coefficient of variation for the GOP and OAM across the
four different databases considered. The coefficient of variation $\gamma_{c}$
for the phoneme $c$ is given by

$\gamma_{c}=\frac{\sigma_{c}}{|\mu_{c}|}$ (3)

where $\mu_{c}$ and $\sigma_{c}$ are the mean and standard deviation of the
articulation scores estimated for various instances of the consonant $c$ per
speaker. If the consonant category '$c$' occurs $N$ times for a speaker, then
$\mu_{c}$ and $\sigma_{c}$ are estimated as,

$\mu_{c}=\frac{1}{N}\sum_{i=0}^{N-1}A(c_{i})$ (4)
$\sigma_{c}=\sqrt{\frac{1}{N}\sum_{i=0}^{N-1}(A(c_{i})-\mu_{c})^{2}}$ (5)

Under the assumption that the articulation of a speaker is not variable, the
ideal objective articulation score should yield low variance estimates across
different instances of the same phoneme; hence, $\gamma_{c}$ is expected to be
lower for more reliable objective measures. We compute $\gamma_{c}$ for each
consonant across all speakers in the database for the OAM and the GOP. Fig.
7(a)-(d) shows the distributions of $\gamma_{c}$ for the dysarthria, Utah
Americelft, Mandarin, and Spanish databases, respectively. For each database,
we plot the coefficient of variation for OAM-M, OAM-A, and the GOP. The
coefficient of variation values are consistently higher for the GOP measure
compared to the OAM. A statistical evaluation using paired samples $t$-tests
reveals that this difference is statistically significant. Across all
databases, there is no statistically significant difference between the OAM-M
and OAM-A ($p>0.001$).

Computational complexity comparison:  During computation of the GOP, the audio
files and the corresponding orthographic transcriptions are passed through the
pre-trained ASR model. Forced-alignment is used to obtain phoneme boundaries
and phone-level likelihood scores are computed for the expected (target) and
recognized phones to estimate the GOP score. To compute likelihood scores for
each recognized phone, the algorithm needs to traverse across different
triphone states of the acoustic model to estimate a best possible path.
Further, a language model is used to obtain more accurate sentence-level
decoding using the recognized phoneme sequence. In addition, the DNN-based GOP
computes 100-dimensional $i$-vectors and augments them with 40-dimensional
MFCCs for speaker adaptation purposes. The computation of the OAM is much more
straight-forward; it does not use any triphone decoding or language modeling.
Instead, it only uses the posteriors of the expected and observed consonants.
Hence, the computational complexity of the OAM is considerably smaller than
the GOP. This results in a 2X faster run time for the OAM computation relative
to the GOP computation. We computed the execution time for a speech sample of
length 5 sec. For the same file, the DNN-based GOP computation took 11 sec and
the OAM computation took 4.811 sec (1.985 sec for Montreal forced aligner and
2.826 sec for the CNN).

##  V Summary and Conclusion

In this work, we develop an algorithm for extracting features from CV
transitions for the purpose of generating a general-purpose algorithm for the
assessment of articulatory precision. Motivated by previous work which
indicates that CV transitions are largely context- and speaker-invariant, we
develop a CNN consonant classifier to use as a feature extractor. The CNN
developed in this work, captures both time and frequency variations of speech
and is trained using the mel-spectrogram computed from the CV transitions of
healthy adult speakers. Analysis of saliency maps shows that the model has
learned important consonant cues like formant transitions and bursts in the
transition regions to discriminate between the correct and incorrect
articulation. We use the posteriors of the CNN to develop an objective
articulation model (OAM) for prediction of articulatory precision in adult
dysarthric speakers, speech acceptability in children with CP, and
accentedness in L2-learners. The results show that the model output is
significantly correlated with different perceptual scales across all
databases. Furthermore, these correlations are on the order of those obtained
using the GOP algorithm. However, in contrast to the GOP, the OAM model is
less variable and computationally more efficient. One of the limitations of
both the GOP and the OAM is that they require knowledge of the target
transcript and force-alignment to segment individual phonemes (for the GOP) or
CV transitions (for the OAM). Future work should focus on the development of
objective articulation assessment models that are independent of transcription
and forced-alignment operations.

## References

  * [1] K. N. Stevens, ``Evidence for the role of acoustic boundaries in the perception of speech sounds,'' _The Journal of the Acoustical Society of America_ , vol. 69, no. S1, pp. S116–S116, 1981.
  * [2] ——, ``Toward a model for lexical access based on acoustic landmarks and distinctive features,'' _The Journal of the Acoustical Society of America_ , vol. 111, no. 4, pp. 1872–1891, 2002.
  * [3] M. M. Parnell, J. D. Amerman, and C. W. LaRiviere, ``Influence of speaker age on perceptual cue distribution,'' _Journal of Phonetics_ , vol. 6, no. 4, pp. 275–282, 1978.
  * [4] R. N. Ohde, K. L. Haley, and C. W. Barnes, ``Perception of the [m]-[n] distinction in consonant-vowel (cv) and vowel-consonant (vc) syllables produced by child and adult talkers,'' _The Journal of the Acoustical Society of America_ , vol. 119, no. 3, pp. 1697–1711, 2006.
  * [5] M. S. Hedrick and R. N. Ohde, ``Effect of relative amplitude of frication on perception of place of articulation,'' _The Journal of the Acoustical Society of America_ , vol. 94, no. 4, pp. 2005–2026, 1993.
  * [6] F. E. Gibbon, ``Research and practice in developmental phonological disorders,'' in _Phonology in context_. Springer, 2007, pp. 245–273.
  * [7] G. Henningsson, D. P. Kuehn, D. Sell, T. Sweeney, J. E. Trost-Cardamone, and T. L. Whitehill, ``Universal parameters for reporting speech outcomes in individuals with cleft palate,'' _The Cleft Palate-Craniofacial Journal_ , vol. 45, no. 1, pp. 1–17, 2008.
  * [8] M. J. Osberger and N. S. McGarr, ``Speech production characteristics of the hearing impaired,'' in _Speech and Language_. Elsevier, 1982, vol. 8, pp. 221–283.
  * [9] H. Kim, K. Martin, M. Hasegawa-Johnson, and A. Perlman, ``Frequency of consonant articulation errors in dysarthric speech,'' _Clinical linguistics & phonetics_, vol. 24, no. 10, pp. 759–770, 2010.
  * [10] F. L. Darley, A. E. Aronson, and J. R. Brown, ``Differential diagnostic patterns of dysarthria,'' _Journal of speech and hearing research_ , vol. 12, no. 2, pp. 246–269, 1969.
  * [11] M. Tu, A. Grabek, J. Liss, and V. Berisha, ``Investigating the role of l1 in automatic pronunciation evaluation of l2 speech,'' in _Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH_ , vol. 2018, 2018, pp. 1636–1640.
  * [12] S. M. Witt and S. J. Young, ``Phone-level pronunciation scoring and assessment for interactive language learning,'' _Speech communication_ , vol. 30, no. 2-3, pp. 95–108, 2000.
  * [13] R. D. Kent, C. Read, and R. D. Kent, _The acoustic analysis of speech_. Singular Publishing Group San Diego, 1992, vol. 58.
  * [14] B. J. Philips and R. D. Kent, ``Acoustic–phonetic descriptions of speech production in speakers with cleft palate and other velopharyngeal disorders,'' in _Speech and Language_. Elsevier, 1984, vol. 11, pp. 113–168.
  * [15] D. Kewley-Port, D. B. Pisoni, and M. Studdert-Kennedy, ``Perception of static and dynamic acoustic cues to place of articulation in initial stop consonants,'' _The Journal of the Acoustical Society of America_ , vol. 73, no. 5, pp. 1779–1793, 1983.
  * [16] D. J. Sharf and T. Hemeyer, ``Identification of place of consonant articulation from vowel formant transitions,'' _The Journal of the Acoustical Society of America_ , vol. 51, no. 2B, pp. 652–658, 1972.
  * [17] H. M. Sussman, K. A. Hoemeke, and F. S. Ahmed, ``A cross-linguistic investigation of locus equations as a phonetic descriptor for place of articulation,'' _The Journal of the Acoustical Society of America_ , vol. 94, no. 3, pp. 1256–1268, 1993.
  * [18] C. Vikram, N. Adiga, and S. M. Prasanna, ``Detection of nasalized voiced stops in cleft palate speech using epoch-synchronous features,'' _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 27, no. 7, pp. 1189–1200, 2019.
  * [19] S. Kalita, S. Mahadeva Prasanna, and S. Dandapat, ``Intelligibility assessment of cleft lip and palate speech using gaussian posteriograms based on joint spectro-temporal features,'' _The Journal of the Acoustical Society of America_ , vol. 144, no. 4, pp. 2413–2423, 2018.
  * [20] R. D. Kent and Y.-J. Kim, ``Toward an acoustic typology of motor speech disorders,'' _Clinical linguistics & phonetics_, vol. 17, no. 6, pp. 427–445, 2003.
  * [21] G. M. Stegmann, S. Hahn, J. Liss, J. Shefner, S. B. Rutkove, K. Kawabata, S. Bhandari, K. Shelton, C. J. Duncan, and V. Berisha, ``Repeatability of commonly used speech and language features for clinical applications,'' _Digital biomarkers_ , vol. 4, no. 3, pp. 109–122, 2020.
  * [22] V. Karjigi and P. Rao, ``Classification of place of articulation in unvoiced stops with spectro-temporal surface modeling,'' _Speech Communication_ , vol. 54, no. 10, pp. 1104–1120, 2012.
  * [23] Z. B. Nossair and S. A. Zahorian, ``Dynamic spectral shape features as acoustic correlates for initial stop consonants,'' _The Journal of the Acoustical Society of America_ , vol. 89, no. 6, pp. 2978–2991, 1991.
  * [24] V. C. Mathad and S. M. Prasanna, ``Vowel onset point based screening of misarticulated stops in cleft lip and palate speech,'' _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 28, pp. 450–460, 2019.
  * [25] S. Strömbergsson, G. Salvi, and D. House, ``Acoustic and perceptual evaluation of category goodness of/t/and/k/in typical and misarticulated children's speech,'' _The Journal of the Acoustical Society of America_ , vol. 137, no. 6, pp. 3422–3435, 2015.
  * [26] V. Berisha, C. Krantsevich, P. R. Hahn, S. Hahn, G. Dasarathy, P. Turaga, and J. Liss, ``Digital medicine and the curse of dimensionality,'' _NPJ digital medicine_ , vol. 4, no. 1, pp. 1–8, 2021.
  * [27] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, ``Librispeech: an asr corpus based on public domain audio books,'' in _2015 IEEE international conference on acoustics, speech and signal processing (ICASSP)_. IEEE, 2015, pp. 5206–5210.
  * [28] M. Saxon, J. Liss, and V. Berisha, ``Objective measures of plosive nasalization in hypernasal speech,'' in _ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2019, pp. 6520–6524.
  * [29] K. L. Chapman, A. Baylis, J. Trost-Cardamone, K. N. Cordero, A. Dixon, C. Dobbelsteyn, A. Thurmes, K. Wilson, A. Harding-Bell, T. Sweeney _et al._ , ``The americleft speech project: a training and reliability study,'' _The Cleft Palate-Craniofacial Journal_ , vol. 53, no. 1, pp. 93–108, 2016.
  * [30] S. Weinberger, ``Speech accent archive. george mason university,'' _Online: < http://accent. gmu. edu_, 2015.
  * [31] M. McAuliffe, M. Socolof, S. Mihuc, M. Wagner, and M. Sonderegger, ``Montreal forced aligner: Trainable text-speech alignment using kaldi.'' in _Interspeech_ , vol. 2017, 2017, pp. 498–502.
  * [32] T. J. Mahr, V. Berisha, K. Kawabata, J. Liss, and K. C. Hustad, ``Performance of forced-alignment algorithms on children's speech,'' _Journal of Speech, Language, and Hearing Research_ , pp. 1–10, 2021.
  * [33] V. C. Mathad, T. J. Mahr, N. Scherer, K. Chapman, K. C. Hustad, J. Liss, and V. Berisha, ``The impact of forced-alignment errors on automatic pronunciation evaluation,'' 2021.
  * [34] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, ``Grad-cam: Visual explanations from deep networks via gradient-based localization,'' in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 618–626.
  * [35] D. Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glembek, N. Goel, M. Hannemann, P. Motlicek, Y. Qian, P. Schwarz _et al._ , ``The kaldi speech recognition toolkit,'' in _IEEE 2011 workshop on automatic speech recognition and understanding_ , no. CONF. IEEE Signal Processing Society, 2011.
  * [36] R. Weide, ``The cmu pronunciation dictionary, release 0.6,'' 1998.