LLudography

# Player-AI Interaction: What Neural Network Games Reveal About AI as Play

Jichen Zhu<EMAIL_ADDRESS>Drexel UniversityPhiladelphia, PA, USA ,
Jennifer Villareale<EMAIL_ADDRESS>Drexel UniversityPhiladelphia, PA, USA ,
Nithesh Javvaji<EMAIL_ADDRESS>Northeastern UniversityBoston, MA,
USA ,  Sebastian Risi<EMAIL_ADDRESS>IT University CopenhagenCopenhagen, Denmark
,  Mathias Löwe<EMAIL_ADDRESS>IT University CopenhagenCopenhagen, Denmark ,
Rush Weigelt<EMAIL_ADDRESS>Drexel UniversityPhiladelphia, PA, USA and
Casper Harteveld<EMAIL_ADDRESS>Northeastern UniversityBoston,
MA, USA

(2021)

###### Abstract.

The advent of artificial intelligence (AI) and machine learning (ML) bring
human-AI interaction to the forefront of HCI research. This paper argues that
games are an ideal domain for studying and experimenting with how humans
interact with AI. Through a systematic survey of neural network games (n =
38), we identified the dominant interaction metaphors and AI interaction
patterns in these games. In addition, we applied existing human-AI interaction
guidelines to further shed light on player-AI interaction in the context of
AI-infused systems. Our core finding is that AI as play can expand current
notions of human-AI interaction, which are predominantly productivity-based.
In particular, our work suggests that game and UX designers should consider
flow to structure the learning curve of human-AI interaction, incorporate
discovery-based learning to play around with the AI and observe the
consequences, and offer users an invitation to play to explore new forms of
human-AI interaction.

Human-AI Interaction; Neural Networks; User Experience; Game Design

††journalyear: 2021††copyright: acmcopyright††conference: CHI Conference on
Human Factors in Computing Systems; May 8–13, 2021; Yokohama,
Japan††booktitle: CHI Conference on Human Factors in Computing Systems (CHI
’21), May 8–13, 2021, Yokohama, Japan††price: 15.00††doi:
10.1145/3411764.3445307††isbn: 978-1-4503-8096-6/21/05††ccs: Human-centered
computing Human computer interaction (HCI)††ccs: Applied computing Computer
games††ccs: Computing methodologies Artificial intelligence Figure 1.
Selection of neural network (NN) games. (From left to right, Top: How to Train
Your Snake snake, Idle Machine Learning Game idle, Evolution evolution,
EvoCommander jallov2016evocommander, Machine Learning Arena
ferguson2019machine, Hey Robot hey; Middle: Quick, Draw! quick, Semantris
semantris, Dr. Derks Mutant Battlegrounds derks, Forza Car Racing forza,
Democracy 3 democracy, Darwin’s Avatars lessin2015darwin, AudioinSpace
hoover2015audioinspace; Bottom: NERO stanley2005evolving, Black & White
blackandwhite, Creatures grand1997creatures, MotoGP19 (moto), Supreme
Commander 2 rabin2015game, Galactic Arms Race hastings2009evolving, Petalz
risi2015petalz)

##  1\. Introduction

With the recent boom in artificial intelligence (AI) technology, 111Unless
otherwise specified, we use the term AI broadly to include a wide range of
artificial intelligence and machine learning techniques. people are
interacting with a growing number of AI-infused products in many aspects of
everyday life. Already, these AI systems influence our decisions (e.g.,
recommendation systems (Smith and Linden, 2017)), inhabit our households
(e.g., robotic appliances (Sung et al., 2007)), and accompany us in our
playful experiences (Mateas and Stern, 2003; Zhu and Ontanón, 2010; Zhu and
Ontañón, 2013) and educational games (Valls-Vargas et al., 2015; Zhu et al.,
2019).

The technological development has precipitated renewed interest in the Human
Computer Interaction (HCI) community. In addition to improving the usability
of individual products (Myers et al., 2018), HCI researchers synthesized
design guidelines for human-AI interaction from the past decades (Amershi et
al., 2019). There is a growing recognition that, compared to traditional
interactive systems, AI-infused products impose additional challenges (e.g.,
technical barrier, low interpretability) to the current user experience (UX)
design process (Dove et al., 2017). Furthermore, new interdisciplinary
research areas have emerged around topics such as explainable AI (Tintarev and
Masthoff, 2007; Zhu et al., 2018; Binns et al., 2018; Rader et al., 2018),
ethics & fairness (Bryson, 2010; Dove et al., 2017; Holmquist, 2017), and
machine learning (ML) as a design material for UX (Yang et al., 2018b; Yang et
al., 2018a).

Among the fast-growing body of literature on human-AI interaction in the CHI
community, one overlooked area is the context of play. With few exceptions,
most recent literature focuses on productivity-related domains such as
e-commerce, navigation and autocomplete (Amershi et al., 2019). While these
are important domains for human-AI interaction, the history of AI and human-AI
interaction has long been associated with play. For instance, ELIZA, one of
the first AI programs designed to interact with lay users, was a playful
satire of a certain school of psychotherapy (Weizenbaum, 1966). Games such as
Chess, Poker, Go, and StarCraft have continued to serve as benchmarks that
propelled the development of AI since the beginning of the field. Since games
naturally focus on end-user experience, game AI research has accumulated
valuable knowledge related to human-AI interaction (Mateas, 1999; Mateas and
Stern, 2003; Young et al., 2004; Zhu and Harrell, 2008; Risi and Togelius,
2015; Valls-Vargas et al., 2017).

In this paper, we propose the new construct of player-AI interaction to
highlight how people interact with AI in the context of play, especially
through computer games. To provide an overview of existing work in this area,
we conducted the first systematic review of player-AI interaction in the scope
of Neural Network games — computer games in which players interact with an NN
as part of the core gameplay. A neural network (NN) is a computational model
that includes nodes (i.e., neurons) and connections between these nodes that
transmit information (Stanley, 2007). The strengths of these connections
(i.e., weights) are typically adjusted through some learning process. For our
paper, this definition covers both NNs that control agents / non-player
characters (NPCs) in games (Jallov et al., 2016; Vinyals et al., 2019) and
generative NN models that produce game content (Hastings et al., 2009; Risi
and Togelius, 2015).

We chose NN games for two key reasons. First, given the wide adoption of AI in
games, we had to constrain our systematic (qualitative) review. Second, and
more importantly, NN games provide insights into some of the most pressing
open problems in human-AI interaction. For example, NNs are notorious for UX
designers to work with because of NNs’ low interpretability of the underlying
process and the frequent unpredictability of its outcome. Studying NN games
can thus provide valuable information on how game designers have to work with
these challenges.

We collected 38 NN games and applied a two-phased qualitative analysis to
examine them. In the first phase, we use close reading and grounded theory to
identify the overarching interaction metaphors and patterns of how NNs are
represented in the game user interface (UI). In the second phase, we apply
current human-AI interaction design guidelines (Amershi et al., 2019),
compiled from a wide range of productivity-based domains, to our dataset. From
these analyses, we derive design lessons for where games do well and identify
open areas that can expand our current notion of human-AI interaction. A key
design insight is that reframing AI as play offers a useful approach for
considering human-AI interaction in games and beyond.

The core argument of this paper is that games are a rich and currently
overlooked domain for advancing human-AI interaction. The design space
afforded by structuring AI as play, as game designers have been exploring, can
point out new opportunities for AI-infused products in general. At the same
time, insights of the generalized guidelines from other domains can be adapted
to improve player-AI interaction. The key contributions of this paper are as
follows:

  * •

We propose the new research area of player-AI interaction. Through the first
systematic review on player-AI interaction in the context of NN games, we
showcase how player-AI interaction can expand the current productivity-based
discussions around human-AI interaction.

  * •

We adapted existing design guidelines for human-AI interaction to the context
of games. Currently, there are no synthesized metrics to evaluate player-AI
interaction.

  * •

We provide several insights from NN games (e.g., flow, exploration) to improve
current challenges in human-AI interaction (e.g., learnability of AI).

##  2\. Related Work

In this section, we summarize related work in human-AI interaction and AI-
based games research. We aim to bridge these two disconnected areas.

###  2.1. Human-AI Interaction

The HCI community has developed a body of work on how to design user
interactions to improve productivity through AI-based applications (Herlocker
et al., 2000; Horvitz, 1999; Höök, 2000; Steinfeld et al., 2006; Winograd,
2006). Thanks to increasingly sophisticated big data and deep neural networks
(i.e., deep learning), AI-infused products have started to enter the consumer
market, prompting a new surge of interest in human-AI interaction (Amershi et
al., 2019; Bansal et al., 2019; Oulasvirta et al., 2020) and its societal
impact in topics such as explainability and transparency (Tintarev and
Masthoff, 2007; Zhu et al., 2018; Binns et al., 2018; Rader et al., 2018).

An important research area is user-centered design for human-AI interaction.
For instance, in conversational agents, a widely adopted type of AI-infused
products, researchers have reported the wide gap between user expectations and
the user experience (UX) of these systems (Luger and Sellen, 2016) and how
users develop their own strategies to work around the obstacles (Myers et al.,
2018). Recently, researchers synthesized guidelines, principles, and theories
into coherent design frameworks for human-AI interaction (Amershi et al.,
2019; Sukis, 2019; Wang et al., 2019). While user-centered (or player-
centered) design is key in game development (Sweetser and Johnson, 2004), few
works have looked at games as a domain for human-AI interaction, despite the
long history of games and AI. Our work is the first to do so in a systematic
and empirical way. More specifically, we leverage the existing meta-review of
human-AI interaction design principles (Amershi et al., 2019) to investigate
NN games.

There is a growing understanding in recent literature that designing AI and ML
products is especially challenging for UX designers. Dove et al. (Dove et al.,
2017) acknowledged that, despite the regular use of ML in UX products, there
has been little design innovation. Echoing the challenges of using ML as
design material, Yang et al. (Yang et al., 2020) argued that capability
uncertainty and output complexity of AI systems are the two root causes of why
human-AI interaction is uniquely tricky to design. The work presented in this
paper aims to identify player-AI interaction, which is currently separated
from the mainstream human-AI interaction literature, as a rich domain for
further study and experimentation. Lessons from the game research community on
how to structure human-AI interaction in the context of play can help to
expand the current body of work in human-AI interaction.

###  2.2. AI-based Game Design and Player Experience

In AI research, there is an extended history of using games as a rich domain
to motivate algorithmic advancements. Salient examples include Chess in the
era of “Good Old Fashioned AI” (GOFAI) (Campbell et al., 2002), Go (Silver et
al., 2016), classic Atari video games (Mnih et al., 2013), or even the popular
AAA game StarCraft (Ontanón et al., 2013; Vinyals et al., 2019) in the age of
deep learning. The advances in game AI in turn opened new design spaces of
player experience in research (Mateas, 1999; Mateas and Stern, 2003; Young et
al., 2004; Zhu and Harrell, 2008; Valls-Vargas et al., 2017) as well as
commercially released games and game engines (e.g., Versu (Evans and Short,
2013), Left4Dead (Valve, 2008) and Civilization VI (Games, 2016)).

However, with few exceptions, games have only recently started to be used as a
serious domain for human-AI interaction research. For instance, Gomme and
Bartle (Gomme and Bartle, 2020) used strategy games to study players’
expectations for what they consider to be a worthy AI-controlled opponent.
Along those lines, several researchers proposed using games and playful
experiences to help designers and users learn AI (Myers et al., 2020; Fulton
et al., 2020; Pemberton et al., 2019).

Most existing work has focused on high-level metaphors (often referred to as
“design patterns”) of how players and designers can interact with AI. For
instance, Treanor et al. (Treanor et al., 2015) derived nine patterns based on
what players do: AI as role-model, trainee, editable, co-creator, adversary,
villain, or spectacle, and whether AI is visible or guided. Cook et al. (Cook
et al., 2016) further examined design patterns in procedural content
generation (PCG)-based games and derived different AI design patterns. In the
context of assisting the game development process, Riedl and Zook (2013)
proposed that AI plays the role of actor, designer, and producer. While the
above work provides a critical starting point for our work, they are “meant to
be a tool for thinking about creating AI-based games, rather than serve as a
comprehensive taxonomy of methods” (Treanor et al., 2015). Finally, Guzdial et
al. (Guzdial et al., 2019) used the taxonomy of friend, collaborator, student,
and manager to describe the different interaction metaphors for how game
designers interact with an AI-based game level editor. Our work builds on this
tradition of using human-human interaction as metaphors to structure the
interaction between humans and AI. In addition, we extend this literature by
conducting an in-depth empirical analysis through grounded theory instead of
relying on researchers’ domain expertise, as is the case for the above-
mentioned existing work.

Finally, there is a significant body of work in games research to understand
player experience (Nacke et al., 2009; Lucero et al., 2013; Desurvire and
Wiberg, 2009; Denisova and Cairns, 2015; Abeele et al., 2020). For example,
the game engagement questionnaire (GEQ) (Brockmyer et al., 2009) is a widely
used instrument for measuring player engagement, although recently it has been
approached with increasing criticism (Law et al., 2018). Other notable
frameworks include game involvement (Calleja, 2007), game usability (Desurvire
and Wiberg, 2009), and design heuristics (Lucero et al., 2013). While these
frameworks are useful to improve the general player experience, they do not
have sufficient focus on the interaction between players and AI to guide the
human-AI interaction design of games. Thus, our work proposes the first set of
guidelines to design and evaluate player-AI interaction.

##  3\. Dataset: Neural Network Games

This section describes our systematic search process and the resulting dataset
of 38 NN Games. Table 1 provides an overview of this dataset, including the
characteristics described in this section.

###  3.1. Search Strategy and Data Collection

Figure 2. Data collection process.

We searched two popular web gaming portals—Steam and itch.io—and a widely used
game AI book, Artificial Intelligence and Games (Yannakakis and Togelius,
2018). We chose these three sources because they collectively cover a wide
variety of games of different production modes. Steam is the largest digital
distribution platform for PC gaming, offering over 34,000 games with over 95
million monthly active users in 2019 (ste, 2020). itch.io is one of the
largest platforms for indie games, containing nearly 100,000 games with
various metadata. Artificial Intelligence and Games is the most cited book on
game AI that includes examples of games with notable AI innovations. The book
complements the previous two sources for its coverage on AAA commercial games
and research games.

Our inclusion criteria were computer games wherein players can interact with
an NN as part of the core gameplay (i.e., gameplay loop). We use the
definition of core gameplay as “the set of actions the player iterates on the
most while playing the game [and which] should directly influence the outcomes
of the game” (Guardiola, 2016). We further excluded work with no clear win
condition and no clear feedback on how player interaction with the AI impacts
the game, as these games lack the basic elements for meaningful player-AI
interaction. Notice that sandbox games with clear feedback to player
interaction are included (blackandwhite; aidungeon; corral; Grand et al.,
1997; Hastings et al., 2009). For the same reason, we excluded games where the
NN did not interact with players (e.g., ML agents that can automatically play
games (Snodgrass and Ontañón, 2014)). Finally, we excluded digitized versions
of traditional board/card games (e.g., Chess, Go, Poker) to focus on computer
games. Future research is needed for investigating player-AI interaction in
traditional games.

Our search process is summarized in Figure 2. Similar to other systematic
reviews on large game repositories (Alharthi et al., 2018), we used pre-
existing game tags in these systems. On itch.io, we used its tags “neural-
network” and “machine-learning,” and “AI.” On Steam we searched with the terms
“neural network”, “machine learning”, and also used Steam’s own tag
“artificial intelligence.” We acknowledge that not all NN games identify
themselves as such, and this is a limitation of our study. However, it is
possible that the games that advertise their use of NNs are more likely to pay
extra attention to player-AI interaction. In the Artificial Intelligence and
Games book (Yannakakis and Togelius, 2018), we went through the chapter “A
Brief History of Artificial Intelligence and Games” to collect the relevant
games. In order to include as many influential examples as possible, we also
asked on social media in the games and game AI communities for additional
work. The suggested games are included in the category of “additional games”
along with the games the authors were aware of.

After screening the 125 games resulting from the above process for
eligibility, we found 38 games that met the inclusion criteria (Table 1). The
most common reasons for games to be excluded were that 1) they used content
(e.g., music) generated by an external NN, but the NN was not part of the
gameplay loop, and 2) they did not have full human-AI interaction due to the
lack of feedback for player actions. For example, Bird by Example bird is a
single-player RPG where players navigate a forest environment and interact
with their bird offspring that is controlled by an NN. This game was excluded
because the player’s actions (i.e., walk, jump, punch) produced no visible
change in the NN’s behavior. As a result, it was unclear how the player was
intended to interact with the NN and to what end the NN impacted gameplay.
Note that our goal was not to develop a comprehensive list, but rather to
capture a representative sample of NN games to analyze current trends in
player-AI interaction.

###  3.2. Dataset

For each game we included, two researchers collected the following to form our
dataset: 1) screen recording of one researcher playing at least one hour of
the game, 2) game developers’ description of the game and their design intent
(i.e., via the game’s website, developer blog, and academic publications), and
3) technical features of the NN (online vs. offline learning, the types of
output to the NN). We used findings from 1) and 2) to determine 3). If it was
not apparent, the two researchers consulted our NN expert coauthors for
resolution. This section summarizes the key characteristics of our dataset. It
should be noted that seven games did not have playable versions publicly
available (marked with * in Table 1). For those games, the researchers used
existing gameplay footage available on the Internet for our Phase 1 analysis.
However, for Phase 2, directly playing the games is necessary. Thus, we
excluded the unplayable games for our guideline analysis (see Figure 2).

####  3.2.1. Characteristics as Games

As a collection of games, our dataset is diverse in multiple aspects. Indie
and research games make up most of the dataset (74%), while only 26% are AAA
games. Using Heintz and Law’s taxonomy (Heintz and Law, 2015), our games cover
all the genres: 15 simulation games (39%), 7 puzzle games (18%), 5 strategy
games (13%), 4 role-playing games (11%), 3 action games (8%), 3 sports games
(8%) and 1 adventure game (3%). In addition, 16 games are multiplayer (42%),
and the remaining 22 games are single player (58%). This suggests that our
dataset has a good representation of different types of games.

####  3.2.2. Characteristics of NN and Game AI

From the technical point of view, our dataset also covered a wide range of
varieties. There is a relatively even split between NN games with online
learning (58%) and offline learning methods (42%). This technical feature is
associated with different gameplay characteristics. In online learning games,
the network is (further) trained as the player interacts with it. Therefore,
these games can adapt to individual players’ actions in real-time. Offline
learning games, on the other hand, are shipped with fixed NNs and are not
adaptive in the same way. However, offline learning games have the advantage
of handling more complex user input, such as natural language
aidungeon,semantris,guesstheword,hey and images villareale2020innk,quick. The
output of the NN can be divided into behavior (89%) and content (11%).
Behavior output consists of the actions and decisions by NN-controlled
characters. Content output typically take the form of in-game assets such as
flowers risi2015petalz or weapons hastings2009evolving.

A key challenge to our analysis is the black-box nature of AI and NNs,
especially from the players’ point of view. Similar to other AI-infused
products, games often contain complex interactions supported by different
algorithms. It can be difficult to attribute gameplay features to specific
algorithms without access to source code. The authors, including two game AI
researchers, made our best effort to determine whether there were multiple AIs
in a game (e.g., Black and White blackandwhite) and what the NN was
responsible for. We did so by using game developers’ descriptions (e.g.,
conference talks and online articles) and our technical expertise to analyze
the gameplay. Still, many such technical details remain unknown for commercial
games. We acknowledge this as a limitation of our study. However, what makes
NNs uniquely demanding in their human-AI interaction design is their reduced
predictability and low interpretability. We argue that an NN, whether as the
entire game AI or as a component, will introduce these characteristics to the
games they are part of. As a result, NN games can be studied as a whole,
regardless of their technical differences.

Table 1. Overview of the 38 NN games and the results of Phase 1 analysis. (*
Games without available playable versions and excluded for Phase 2.)

Game Characteristics NN Characteristics Player-NN Framework Characteristics
Game Title [Ludography] Publisher Genre Multiple AIs? NN Responsibilities NN
Output Learning Interaction Metaphor UI 2D Walk Evolution 2dwalkevolution
Indie Simulation No controls creature movement Behaviors Offline Teammate NN-
Specific AI Dungeon aidungeon Indie Adventure No creates natural language
responses Behaviors Offline Designer NN-Limited AIvolution aivolution Indie
Simulation Unknown controls creature movement Behaviors Online Teammate NN-
Limited AudioinSpace* hoover2015audioinspace(Hoover et al., 2015) Research
Action Yes creates weapon visuals and audio Content Online Designer NN-
Agnostic Black & White blackandwhite(Wexler, 2002) AAA Role-play Yes creates
creature desires Behaviors Online Apprentice NN-Agnostic UI Blitzkrieg 3
blitzkrieg AAA Strategy Unknown controls ”Boris” battle behavior Behaviors
Offline Competitor NN-Limited UI BrainCrafter* (braincrafter) Research Puzzle
No controls robot movement Behaviors Online Apprentice NN-Specific UI Colin
McRae Rally 2.0* colin AAA Sports Unknown controls the car’s driving
performance Behaviors Offline Competitor NN-Agnostic UI Competitive Snake
competitivesnake Indie Puzzle No controls enemy snake behavior Behaviors
Offline Competitor NN-Agnostic UI Corral corral Indie Simulation Unknown
controls chicken movement and preservation skills Behaviors Online Apprentice
NN-Agnostic UI Creatures (grand1997creatures)(Grand et al., 1997) AAA Role-
play Yes controls the creature’s sensor-motor coordination Behaviors Online
Apprentice NN-Agnostic UI Darwin’s Avatar* lessin2015darwin (Lessin and Risi,
2015) Research Action No controls creature movement Content Offline Designer
NN-Limited UI Democracy 3 democracy AAA Role-play Unknown creates motivations
and desires of the public Behaviors Offline Designer NN-Agnostic UI Dr. Derk’s
Mutant Battlegrounds derks Indie Simulation Unknown controls creature movement
and behavior Behaviors Online Apprentice NN-Limited UI EvoCommander*
jallov2016evocommander (Jallov et al., 2016) Research Simulation No controls
tank movement and shooting behavior Behaviors Online Apprentice NN-Specific UI
Evolution evolution Indie Simulation Unknown controls creature movement
Behaviors Online Teammate NN-Specific UI evolution for beginners evolutionbeg
Indie Simulation Unknown controls creature movement and sensory input
Behaviors Online Apprentice NN-Limited UI Football Evo football Indie
Simulation No controls player movement and behavior Behaviors Online
Apprentice NN-Limited UI Forza Car Racing forza(Takahashi, 2018) AAA Sports
Unknown controls the car’s driving performance Behaviors Online Competitor NN-
Limited UI GAR (hastings2009evolving)(Hastings et al., 2009) Research Action
Yes creates particle weapons Content Online Designer NN-Limited UI Gridworld
grid Indie Simulation Unknown controls creature behavior Behaviors Online
Designer NN-Limited UI Guess the Word guesstheword (Gero et al., 2020)
Research Puzzle Yes creates natural language responses Behaviors Offline
Teammate NN-Limited UI Hey Robot hey Indie Puzzle No controls language
processing Behaviors Offline Teammate NN-Agnostic UI How to Train your Snake
snake Indie Simulation No controls snake movement Behaviors Online Apprentice
NN-Specific UI Idle Machine Learning Game idle Indie Simulation No controls
performance of the vehicle’s movement Behaviors Online Apprentice NN-Specific
UI iNNk villareale2020innk Research Puzzle No identifies sketches drawn by the
player Behaviors Offline Competitor NN-Specific UI Machine Learning Arena*
ferguson2019machine Research Simulation Unknown controls robot behavior
Behaviors Online Teammate NN-Specific UI MotoGP19 moto AAA Sports Unknown
controls the car’s driving performance Behaviors Offline Competitor NN-
Agnostic UI Neat Race neat Indie Simulation No controls car movement Behaviors
Online Apprentice NN-Specific UI NERO stanley2005evolving(Stanley et al.,
2005) Research Simulation No controls robot movement and shooting behavior
Behaviors Online Apprentice NN-Specific UI Oui Chef!! oui(Cimolino et al.,
2019) Research Role-play No controls chef behavior Behaviors Online Apprentice
NN-Agnostic UI Petalz* risi2015petalz(Risi et al., 2015) Research Simulation
No creates flowers Content Offline Designer NN-Agnostic UI Quick, Draw! quick
Research Puzzle No identifies sketches drawn by the player Behaviors Offline
Teammate NN-Limited UI Race for the Galaxy raceforthegalaxy AAA Strategy
Unknown controls opponent behavior Behaviors Offline Competitor NN-Limited UI
Roll for the Galaxy rollforthegalaxy AAA Strategy Unknown controls opponent
behavior Behaviors Offline Competitor NN-Limited UI Semantris semantris
Research Puzzle No controls the classification of words Behaviors Offline
Teammate NN-Limited UI Supreme Commander 2 rabin2015game(Rabin, 2015) AAA
Strategy Yes controls enemy unit flight and fight behavior Behaviors Offline
Competitor NN-Agnostic UI The Abbattoir Intergrade abbattoir Indie Strategy
Unknown controls enemy unit offense behavior Behaviors Online Competitor NN-
Agnostic UI

##  4\. Phase 1: Analyzing Player-AI Interaction in NN Games

The first broad question we attempt to answer is how existing games use neural
networks (NN), especially in terms of human-AI interaction. In particular, we
focused on two subsidiary research questions:

  * •

RQ 1.a: How do NN games structure player-AI interaction?

  * •

RQ 1.b: How visible are NNs in the UI of the core gameplay?

###  4.1. Methods

The overall analysis procedure involved a close reading of the gameplay data
in our dataset of 38 games. We used grounded theory to iteratively develop a
framework for how players interact with the NN (i.e., the interaction
metaphors) and how much the existence of NN is foregrounded in the core
gameplay (i.e., levels of visibility).

After initial observations of the games, two researchers conducted a close
reading of the dataset based on the following questions: 1) What role does the
NN play in the overall game system? 2) How does the player interact with the
NN in the game? 3) Where does the interaction with the NN occur in the
gameplay experience? and 4) How, if at all, are the NNs presented in the UI?

Next, the two researchers conducted a preliminary open coding to label notable
characteristics of the observations. During this step of the analysis, both
researchers first went through each game individually and noted initial labels
(e.g., player input via parameters, NN outputs behavior, NN is represented as
a creature) into a shared document. Then, they discussed this document to
iterate on the labels to form concepts (e.g., player directs NN toward a
desired goal). While constructing the concepts, they re-observed some of the
games and reviewed related literature to refine the classifications.

Once a common concept list was achieved and agreed upon, both researchers
separately re-analyzed the shared document to develop preliminary categories
that fit into a framework. During this step of the analysis, the researchers
presented each other’s framework to one another and then collectively iterated
on the categories to finalize the framework. The result of this phase is a set
of categories that make up the player-NN framework (i.e., the interaction
metaphors discussed in Section 4.2.1 and levels of visibility discussed in
Section 4.2.2). Using the framework, each researcher independently coded the
same 7 games (20%), one from each genre. After a complete agreement on the
codes, they then coded the rest of the games independently. When the codes
were complete, both researchers reviewed each other’s work to ensure there
were no discrepancies. If there was a discrepancy, the game would be discussed
and reviewed again by both researchers.

We opted for this consensual qualitative approach (Hill, 2012) instead of
inter-rater reliability (IRR) because analyzing NN games and their compliance
with guidelines (next section) is complex. This is due to the incredibly
varying contexts and nuances that need to be considered in this emerging but
under-studied area. Consensus coding is generally more suited for small
samples and for considering multiple viewpoints, which fits our study better
than IRR (McDonald et al., 2019). Specifically, in order to get such multiple
viewpoints, the two researchers discussed above were a game researcher and an
AI engineer. A much richer and deeper understanding is thus gained.

###  4.2. Results

This section presents the results of our Phase 1 analysis. All classifications
of each game according to our player-NN framework are presented in Table 1.

####  4.2.1. Interaction Metaphors

The HCI literature shows that interface metaphors (e.g., “Desktop” and “Search
Engine”) are “useful ways to provide familiar entities that enable people
readily to understand the underlying conceptual model [of a system] and know
what to do at the interface” (Sharp et al., 2019, p.78). Critical AI studies
revealed the importance of metaphors to AI (Agre, 1997; Mateas, 2003; Zhu,
2009). Our analysis found four interaction metaphors that provide familiar
structures for players to interact with the AI: NN as Apprentice, Competitor,
Designer, and Teammate. This finding is consistent with recent work in the
game design literature. Based on their expert knowledge and intuition, game
developers discuss how interaction metaphors (often referred to as “design
patterns”) have been used in game design (Treanor et al., 2015; Cook et al.,
2016) and in game production (Riedl and Zook, 2013). Our analysis extends the
existing literature by conducting the first empirical work that uses deep
qualitative analysis to analyze the interaction metaphors.

The largest portion of NN games (34%) adopted what we call Neural Network as
Apprentice. In these games, the player interacts with the NN as its mentor,
and the focus of the gameplay is how player changes the NN over time. The
player’s mentoring of the NN can be achieved by providing direct feedback to
the NN’s behaviors blackandwhite,grand1997creatures,evolutionbeg. For example,
in Creatures, the player provides positive feedback (petting) when the NN-
controlled creature displays desirable behavior (e.g., eat when hungry) and
punishes it (slapping) for the opposite. A second way the player can mentor
the NN is by configuring the right training setting for it
football,hey,braincrafter,stanley2005evolving. The gameplay afforded by this
interaction metaphor focuses on getting the player to train the NN. As shown
in Figure 4, all games in this category use online learning.

Another interaction metaphor our NN games use is Neural Network as Competitor.
The key characteristics of this group, consisting of 26% of the games, is that
player-AI interaction is adversarial. For example, in Supreme Commander 2
rabin2015game, the player fights an NN through their respective army platoons.
As the player customizes their army, the NN weighs the player’s unit
composition against its own and makes tactical battle decisions, such as how
its army will respond, which enemy to target first, or when to retreat. The NN
can exploit players that are over-reliant on a single strategy and counter the
player to create an evolving challenge
forza,moto,rabin2015game,abbattoir,blitzkrieg,raceforthegalaxy,rollforthegalaxy.
In these games, the NN counters the player during gameplay, thus encouraging
them to adapt and try new strategies. A key distinction in this category is
that the NN learns player’s actions to create a more difficult challenge for
the player to overcome. As discussed further in the next section, only one
game villareale2020innk here explicitly highlights the existence of the NN in
their core UI.

For 21% of the games we identified Neural Network as Teammate, which happens
if the interaction between the player and the NN is structured as those
between colleagues. In these games, the player and the NN work together toward
a shared goal. For example, in Evolution evolution, players and the NN create
a stick-figure-like creature together. Players assemble the creature by
placing bones, muscles, and joints in different ways. The NN takes the
player’s creation and improves it through evolving it over many iterations.
This interaction creates a collaborative cycle between the player and the NN.
A unique characteristic of this interaction metaphor is that the player and
the NN have complementary skills. Both are needed to complete the game
objective.

The final 19% of the games used the Neural Network as Designer metaphor. In
these games, the NN acts as a creator and the player as its client. The NN
generates new content lessin2015darwin,aidungeon or customizes content based
on the preferences of the player risi2015petalz,hastings2009evolving, usually
determined passively through players frequently interacting with a particular
game element. For example, in Petalz risi2015petalz, players arrange and
nurture a balcony of flowers, which are generated by an NN. The NN generates
each flower (shape and color) based on the player’s selection of flowers to
breed or cross-pollinate. The NN extends the game’s playability by creating
flowers that match the preferences of the player. Notice that compared to NN
as Teammate, the player here generally has less well-defined goals to
accomplish with the NN.

Figure 3. From left to right, we display _Neat Race_ neat categorized as _NN-
Specific_ , iNNk villareale2020innk categorized as _NN-Specific_ , and
Blitzkrieg 3 blitzkrieg categorized as _NN-Limited_.

####  4.2.2. Visibility of NN in Core UI

For the second research question of “how visible are NNs in the UI of the core
gameplay,” we found 3 levels in which the NN is called into the player’s
attention in the UI: NN-Specific, NN-Limited, and NN-Agnostic.

A significant number of games (26%) foregrounded the existence of its NN
through what we call NN-Specific UI. These UIs highlight the presence of the
NN during core gameplay through linguistic features (e.g., using the term
“neural network” snake,grid,idle,villareale2020innk). For instance, How to
Train your Snake describes each NN-controlled snake as “…hooked up to a Neural
Network” snake. Some games use visual features (e.g., visualizing the
underling NN snake,idle,neat,football,villareale2020innk). In iNNk (Middle,
Figure 3), the word “neural network” is prominently featured in the core game
UI along with the NN’s confidence meter. More interesting, some games
visualize the parameters of the NN training algorithm to make the training
process playable. For example, in Neat Race neat (Left, Figure 3), the game
visualizes the NN’s internal structure (bottom left of the screenshot) and
displays its parameters as sliders (top right).

The majority of our games (40%) used NN-Limited UI. They acknowledge the
presence of the NN in the game, but only through non-essential UI, such as
using technical terminology in tutorials
aidungeon,grand1997creatures,aivolution, menus outside the core gameplay loop
lessin2015darwin,forza,stanley2005evolving,grid, or explicitly referring to
the NN only in title screens semantris,aidungeon. For instance, Blitzkrieg 3
blitzkrieg is a WWII strategy game where players build and command a variety
of units to defeat the opposing NN-controlled enemy. The game’s opening screen
(Right, Figure 3) personifies the NN as an evil-looking person with the text
“Meet Boris, a neural-network AI you can fight against…”

Finally, 34% of the games used NN-Agnostic UI, which does not reference the
NN. By masking the NN, these games maintain the narrative immersion of the
game worlds without revealing the algorithms used to build them.

Figure 4. Distribution of the NN games (n = 38) categorized by interaction
metaphor, online/offline learning, and UI visibility. Each black dot
represents one NN game. Figure 5. Distribution of the NN games by publishing
date.

###  4.3. Discussion

####  4.3.1. NN Is Driving the Experimentation of Novel Gameplay Experiences

We observed a surge of NN games in recent years (Figure 5). NNs have been
adopted in a wide variety of game genres and gameplay experiences. The games
in our collection covered all the common game genres, showing vibrant efforts
in trying different ways of incorporating NN in games. The interaction
metaphor with the longest history is NN as Apprentice, whereas NN as Teammate
has only been published since 2014. Our hypothesis is that the metaphor of NN
as Teammate has the highest technical requirement for the NN because it needs
to sufficiently understand and anticipate player interaction in order to
accomplish a common goal. Thus, they are the most recent interaction metaphor.
Most of the games are experimental, as the game design community has not
formed established ways to use NNs. This is also echoed by the fact that indie
developers and researchers developed the majority of our games (74%).

We noticed that some of the NN games are providing novel gameplay experiences
that would not have been possible without NNs. For example, thanks to
advancements in deep learning, AI Dungeon aidungeon offered a text-based,
human-AI collaborative storytelling experience with few constraints for what
the player could type into the game. This is a drastic improvement over
traditional text-based adventure games, which are infamous for their
intolerance for player input that is slightly different from what the designer
programmed for (Montfort, 2005). More significant is that no matter what story
elements the player enters, AI Dungeon can respond in reasonable ways.

The NN games in our dataset have improved established gameplay experiences
that had been supported by other AI techniques. Compared with traditional AI,
NNs are more adaptive and flexible. In NN as competitor games, we observed
that these games use established game mechanics (e.g., racing forza,moto), but
the NN offers a more challenging competitive experience through more capable
NPCs than other AI techniques.

The most unique gameplay is found in making the training of the NN itself a
playable mechanic. By representing the NN and displaying features of its
internal processes, this may lead to new player experiences, such as using
gameplay to inform their understanding of the system to be successful in the
game. For example, we observed that this occurs in games such as How to Train
your Snake snake and Idle Machine Learning Game idle, which made the NN
explicit in the UI and displayed parameters for the players to use when
steering the NN’s output. The gameplay became a puzzle about how to best
configure the NN to be successful in the game. To do so requires players to
have a basic understanding of the capabilities and limitations of the system,
which are discovered over time through play.

####  4.3.2. Metaphors We Play By

In our analysis, we saw that interaction metaphors based on human
relationships played a powerful role in structuring player-AI interaction. We
did not notice that any games break or even complicate (e.g., a teammate that
back-stabs) the interaction metaphor they use. Our empirical analysis
validates what prior researchers proposed based on intuitions and domain
knowledge (Treanor et al., 2015; Cook et al., 2016; Riedl and Zook, 2013;
Guzdial et al., 2019), all of which uses human relationships. The role of
metaphors has been extensively studied in human cognition (Lakoff and Johnson,
2008; Lakoff and Turner, 2009) and in UI design (Sharp et al., 2019; Lubart,
2005), as it has been more recently in human-AI interaction as well (Dove and
Fayard, 2020).

We believe this full, uncomplicated adoption of interaction metaphors reflects
the early stage of human-AI interaction. While we did notice some laudable
innovations such as those mentioned in the previous section, by and large,
game designers and developers went with familiar concepts and metaphors. This
is consistent with Bolter’s notion of how new technology remediates familiar
forms before taking on its distinctive forms (Bolter, 2016). Compared to ML-
based UX (Dove et al., 2017), an advantage of the games community is that game
AI developers are often game designers themselves or work closely with the
latter. This cross-pollination between algorithm and design makes games a
vibrant domain for new experimentation.

It is also notable that the metaphors are connected with the algorithmic
characteristics of different NNs. As shown in Figure 4, all games that adopted
NN as Apprentice use online learning for player agency, whereas most games
with NN as Competitor use offline learning for opponent competency.

####  4.3.3. The Struggle with Transparency and Interpretability

Similar to most NN-based UX applications (see further discussions in Section
5), NN games struggle with how and what to communicate to the player regarding
the use of the NN. In our initial coding stage, even our NN researcher
coauthors could not always figure out the NN-specific questions by simply
looking at the game. Because the use of NNs may not be apparent in the
gameplay, and since only 26% of our games references NNs (including simply
using the word “neural network”) in their core UI, the game requires the
player to believe they are interacting with an NN.

Even when the players are directly playing with the parameters of the NN, it
is not always clear what these features do. For example, unless the player has
a background in evolutionary algorithms, terms such as “population” and
“retraining” are not necessarily understandable.

Like other AI-infused systems, games also struggle with the lack of
interpretability of NNs. We see a full spectrum from completely blackbox NNs
grand1997creatures,blackandwhite,rabin2015game,democracy (often in NN as
competitors) to attempts to visualize the underlying NN
idle,football,snake,braincrafter,neat,stanley2005evolving. Most notably, NERO
stanley2005evolving, gives insights about the NN and its training in two ways.
First, by visualizing the NN training parameters, players can steer the NN
behavior by tweaking the reward structure for what is preferred behavior
(e.g., approach enemy, attack enemy). Second, players are able to see graphs
of the NN’s internal structure and fitness values across generations for all
robots across various combat stats (e.g., enemy hits).

The strongest designs for making NNs more interpretable come from simulation
games where the player can tweak different training parameters. In these
cases, even though the names of the parameters are sometimes too technical for
players without an AI background, the NN’s behavioral change feedback through
different iterations of trial-and-error gameplay helps the player develop an
intuition. In other words, most games in our dataset manage to reframe the
difficulties of interacting with an NN as a puzzle and thus make it more
engaging.

##  5\. Phase 2: Analyzing NN Games with General Human-AI Interaction
Guidelines

The second broad research question is to explore what neural network (NN)
games can tell us about designing human-AI interaction. Here we focus on the
following subsidiary research questions:

  * •

RQ 2.a: To what extent do NN games comply with contemporary design guidelines
for human-AI interaction?

  * •

RQ 2.b: Using the design guidelines for human-AI interaction, how can NN games
be differentiated according to their characteristics (see Table 1) and in
comparison with other AI-infused products?

###  5.1. Methods

For inferring what NN games can tell us about human-AI interaction, we used
the human-AI design guidelines proposed by Amershi et al. (Amershi et al.,
2019). It is the most recent and comprehensive manner in which the design for
human-AI interaction is documented thus far by the HCI community. As discussed
above (Section 2.2), currently no equivalent guidelines exist specifically for
games. There are 18 guidelines in (Amershi et al., 2019) in total. They are
grouped according to when the user is interacting with the AI: 1) initially,
2) during, 3) when wrong, and 4) overtime. The analysis procedure in adapting
these guidelines to the NN games involved a three-step process: Step 1.
defining guiding questions for NN games, Step 2 establishing codes for
analyzing games, and Step 3 analyzing the games with Step 1 & 2\. Two
researchers performed all steps in close coordination and checked the outcomes
of each step with the other authors for verification and to reach consensus
(Hill et al., 2005; Richards and Hemphill, 2018).

####  5.1.1. Step 1: Guiding Questions for NN games.

Two researchers completed a detailed reading of each guideline to understand
the guideline in the context of the original AI application examples (e.g.,
recommender systems, activity devices, etc.). Then, both researchers explored
how the guidelines may be applied in the context of games. This process led to
the definition of a question for each guideline to help orient the researchers
when observing the games in Step 3. For example, for Guideline 15 “encourage
granular feedback,” we defined the question, “how do players indicate their
feedback such as preference to the NN during gameplay?” Table 2 shows all the
original guidelines and our associated “Guiding questions for NN games.”

From this process, we agreed that guidelines in the “when wrong” category did
not apply in the context of games. Games handle failure differently than other
AI products, where failure is expected and, in fact, part of the main
interaction and resulting experience (Juul, 2013; Anderson et al., 2018).
Additionally, for AI-infused products, humans are consumers of the AI. By
contrast, players in many of our games actively control how the AIs are
trained. This close relationship significantly complicates the notion of
failure in games. As a result, fully unpacking what failure means in games is
out of the scope of this paper. We hence excluded this category and focused
our analysis on the initially, during, and overtime categories. We do offer
some observations in the discussion, but further research is needed in this
important area of player-AI interaction.

####  5.1.2. Step 2: Codes for Analyzing NN Games

Two researchers took an iterative approach to arrive at a set of codes to
analyze the games in Step 3. Amershi et al.’s (Amershi et al., 2019) do not
necessarily specify or recommend how the guidelines should be evaluated, but
in their user study with UX designers (n = 49), they applied a 5-point
semantic differential scale from “clearly violated” to “clearly applied.” We
combined their approach for heuristic evaluation with our guiding questions to
create a 3-point coding scheme for each guideline. In a nutshell, this coding
scheme is used to decide whether a game (A) clearly applies, (B) partially
applies, or (C) violates the guideline. For example, for Guideline 1 “Make
clear what the system can do” we added the guiding question “How does the game
make clear what the NN can do?” and the following three codes: (A) Makes the
NN’s capabilities known; (B) Makes part of NN’s capabilities known; and (C)
Does not make the capabilities known at all. With this coding scheme, the
researchers were able to clearly label in the context of a specific guideline
and NN games. Additionally, the 3-point scale is much more suitable for
qualitative/consensus coding. We aimed to describe rather than only rate or
score the games to extract meaningful insights, hence why we defined the
3-point coding scheme for each guideline in accordance with the associated
guiding question. Table 2 shows the resulting codes.

####  5.1.3. Step 3: Analyzing NN Games

Two researchers applied the codes from Step 2 for the analysis of the 31
playable games from our corpus, the results of which are presented in Section
5.2. When analyzing each game, both researchers reviewed the data collected
from Phase 1 (i.e., gameplay footage and written observations) and played the
game. After reviewing this material, they independently assigned codes for
each guideline per game. Disagreements were resolved through discussion.

After the coding results were agreed upon, scores were assigned for cross-
comparison with the characteristics found in Section 4 (see Table 1). We
assigned a score of 2 for the clearly applies codes (A), 1 for the partially
applies codes (B), and 0 for the violation codes (C). For the comparison of
the guidelines (i.e., comparing G1 with G2, etc.), we then took the sum of
scores of these resulting scores per guidelines and divided them by the
maximum score per guideline to calculate the % per guideline. For the cross-
comparisons of the characteristics (on the interaction metaphors, visibility,
developer, etc.), we normalized the sum of scores as we have a different
number of games per characteristic and then took the average of the normalized
sum of scores to calculate the % per characteristic.

In this section, we do not report the results on the characteristics of NN
input and NN output as they do not provide any meaningful insights. We further
omitted categories with a low number of cases (e.g., there is only one
adventure game). Finally, for the comparison of our outcomes with other AI-
infused products reported by Amershi et al. (Amershi et al., 2019), we
considered the major patterns in the aggregate data for both (i.e., NN games
vs. all other AI-infused products).

Table 2. Summary of the guideline analysis codes.

###  5.2. Results

Figure 6 shows the results of applying the adapted guidelines in the context
of NN games. Below, we discuss per guideline category (i.e., initially,
during, and overtime) the results in more detail. Note that the reported %
here are based on the % of games that were coded as A = clearly applied, B =
partially applied, and C = violation. Following this, we compare the
application of the guidelines across the characteristics discussed in Section
4, and with other AI-infused products. The reported results here are based on
the % derived from the normalized scores.

Figure 6. The distribution of compliance codes (A, B, and C) per guidelines in
count (n = 31) and %.

####  5.2.1. Initially

The initially guidelines (G1–G2) refer to player interaction prior to
gameplay. During the initial interaction of these games, Guideline 1 “makes
clear what the system can do” has the most reported applications: 23 games
(74%) made either full or part of the NN’s capabilities known to the player
prior to gameplay. These games helped the player understand the capabilities
in a variety of ways, such as tutorials, intro screens, or developer notes
prior to gameplay. Some were not as direct and did not provide such textual
content, but still made the capabilities known by immediately providing the
player with an output to observe. For example, in _How to Train your Snake_
snake, players start the game to find the NN already training the snakes to
move and find food, thus, showcasing an immediate result.

While these games are doing well in most cases by communicating the
capabilities of the NN during the initial interaction, they are doing poorly
regarding communicating the limitations of NNs to the players. Guideline 2
“makes clear how well the system can do what it can do” had 27 games (85%) not
making the limitations of the NNs known to the player at all — the second-
highest number of violations among the guidelines. This may be justified in
cases playing against the NNs (i.e., NN as Competitor) to avoid exploiting the
system to win the game. Or in cases where the NN is the focus of gameplay
(i.e., NN as Apprentice), limitations become part of the puzzle and
understanding through play.

####  5.2.2. During

The during guidelines (G3–G6) refer to player interaction at any given
gameplay loop. Guideline 3 “time services based on context” and 5 “match
relevant social norms” had the most reported applications: 21 games’ (68%) NNs
provided timely service, and 25 games (81%) were labeled as an expected
interaction with the NN. The clear application of G3 and G5 suggests how
designers carefully considered how the NN can help assist with the flow of the
player experience by suggesting interactions and providing immediate
consequences that are consistent with player expectations.

Games also performed well in complying with Guideline 4 “show contextually
relevant information,” as none of the games violated showing contextually
relevant information. In the majority of these games, the focus of the
gameplay centers on changing or affecting the output of the NN. We observed
that the games provide information in regards to how the NN was responding to
or utilizing player actions. A common approach is the use of UI elements
(e.g., NN performance stats increasing) accompanied by a continuous animation
or visual change to enable players to observe and then inform their next
gameplay action. For example, in iNNk villareale2020innk, players can observe
the exposed confidence meter that displays as a percentage under the NPC
character in relation to their drawing, thus building a better mental model of
how to subvert the NN in future drawings.

Other games provide additional visual information during this animation, such
as an overlay of the entire NN population. Providing this extra information
allows players to assess the NN’s progress and observe both successful and
failed attempts. For example, in Evolution evolution, players are able to see
an overlay of all the NN attempts at training the same creature
simultaneously. This additional animation shows the highest and the lowest-
performing creatures training at the same time, which enables players to
better understand the progress as a whole and determine if the creature needs
to be tweaked further.

While games are doing well to display relevant information regarding gameplay,
Guideline 6 “mitigate social biases” had 17 games (55%) with a violation.
Examination of these instances revealed that it was unclear if the NN
mitigates undesirable social stereotypes and biases. Additionally, we reported
half of such games as a violation because such biases may emerge directly from
players and are not mitigated. For example, games that made the NN the focus
of gameplay (i.e., training or evolving using an NN) provides a new NN to play
with. Therefore, players may steer the NN with their own personal preferences.
Stereotypes and biases can emerge through the player’s direction and
reinforcement.

####  5.2.3. Overtime

The overtime guidelines (G12–G18) refer to player interaction with the NN over
a longer period of time. Guideline 13 “learn from user behavior” and 14
“update and adapt cautiously” are performing well. Based on the player’s
behavior, 17 games (55%) NN personalize the experience, and 26 games (84%) do
not disrupt the gameplay experience when the NN changes its behavior.
Guideline 15 “encourage granular user feedback” is doing well with 14 games
(45%) that allow players to directly indicate their preferences to the NN
during gameplay. Games are performing moderately in regards to Guideline 17
“provide global controls” with 15 games (48%) providing full or partial global
control to adjust how the NN behaves, respectively.

A common approach to provide players more agency is the ability to adjust the
NN through parameters or the environment it interacts in. We observed these in
setting menus, or in other cases, directly in the core GUI. For example, in
_NERO_ , the game allows players to edit the reward structure of the NNs
during a training session. Further, players are able to edit the training
environment, such as adding barriers and placing particular enemies to
directly influence the NNs training.

Guideline G16 “convey the consequences of user actions related to NN” and G18
“clear notifications of changes in NN capability” had the most violations in
this category: 22 games (71%) did not provide any feedback conveying how
players’ actions will impact the NN, and 28 games (90%) did not notify of any
changes or updated to the NN capabilities. Further, Guideline G12 “remember
recent transactions” was another difficult guideline to apply in games. In
these cases, 10 games (32%) leveraged the history of the player actions to
generating content tailored to the player or a more challenging experience but
did not allow the users to access that memory. Only 2 games (6%) make this
history useful to the player as other AI products do (e.g., navigation
products, search engines) and allowed the users to reference that history.

(a) Developer categories.

(b) UI Visibility categories.

(c) Interaction metaphor categories.

Figure 7. A comparison of guidelines across different characteristics.

####  5.2.4. Comparison

Consistent with the analysis in Section 4, NN-Specific games (60%) outperform
NN-Limited (50%) and NN-Agnostic (40%) as shown in Figure 7(b). G1 “make clear
what the system can do” is understandably what separates NN-Agnostic from the
other two UI categories and G17 “provide global controls” is the most
differentiating guideline for visibility. Consistent with the analysis in
Section 4, we find that the simulation genre complies better (63%) compared to
all others (32–58%), due to G4, G15, and G17. Also, not unexpectedly, NN games
with online learning score higher (57%) than those with offline learning
(39%), specifically in the guidelines in the overtime category.

With the interaction metaphors (see Figure 7(c)), we see that NN as Apprentice
scores the highest (60%) with here too G17 as the most differentiating
guideline, followed by G13 “learn from user behavior.” Most apparent is that
NN as Competitor (35%) scores the lowest, which is not unexpected given that
there are gameplay reasons to not abide by the guidelines. Of note is that NN
as Teammate do reasonably well on G12 “remember recent transactions,” which
makes sense given that players need to build trust with the NN to work with
them.

We further find that the AAA games score lower (38%) compared to the indie
(56%) and research (49%) games (see Figure 7(a)). This is mostly due that 6
out of 9 AAA games are classified as NN as Competitor games. It is interesting
that research games score high on G3 and G14, while indie games score high on
G13, G15, and G17. It indicates that the research games are more focused on
integrating the NN into the game flow, while indie games are more
experimenting in how players can interact with the NN.

Aside from contrasting the outcomes with the game characteristics, we compared
our guideline outcomes with the reported outcomes of AI-infused products by
Amershi et al. (Amershi et al., 2019) to see what key similarities and
differences exist. We find that making the limitations known (G2) is poorly
addressed in both NN games as other AI-infused products. AI-infused products
do tend to perform better on G12, which is not surprising given that this a
feature that is critical for many recommender types of products. NN games,
however, generally perform better on G15 and G17, suggesting that NN games
facilitate granular user feedback through direct interaction and provide more
controls to their users, respectively.

###  5.3. Discussion

By leveraging the design guidelines for human-AI interaction by Amershi et al.
(Amershi et al., 2019), our aim was to explore what NN games tell us about
designing human-AI interaction. In this process, we also learned more about NN
games themselves and were able to verify and confirm the findings reported in
Section 4. In fact, after applying the guidelines, we see more specifically
what NN games do and how they differ from other AI-infused products. Here we
describe the main takeaways from analyzing NN games with general human-AI
interaction guidelines, especially in terms of how the notion of AI as play
can be used to inform human-AI interaction.

####  5.3.1. Learning AI and Its Limitations Through Play

Clearly communicating the affordances and limitations of a system is a long-
established design principle (Norman, 2013) and confirmed in human-AI
interaction (Amershi et al., 2019; De Graaf et al., 2017; Furqan et al.,
2017). While the NN games do overall relatively well in communicating what the
NN does (G1), they do not in communicating what it does not (G2). Other AI-
infused products performed equally poorly on G2 (Amershi et al., 2019).
Through our close analysis of the games as well as developers’ notes, we
noticed that NN games purposely ignore G2 because the point is to learn the
limitations through play.

The developers of aidungeon,quick,semantris all explicitly describe their
games as experiences to explore or test the limits of the NN. For example,
Semantris asked the players to “play around ” and “see what references the AI
understands best.” In such games, the NN cannot directly communicate the
limitations of the system upfront without jeopardizing the gameplay — because
discovering the system’s limitations is the gameplay. By framing the discovery
of NN limitations as play, many NN games are able to foster a sense of
curiosity, discovery, and accomplishment in players.

We also saw creative ways of making AI’s limitations part of the rule of play.
Most notably, we saw a number of projects with online-learning adopted by the
idle game genre, which has a built-in play-wait-play cycle (Alharthi et al.,
2018). This game feature makes the technical requirement of waiting for the NN
to finish the training part of the expected experience and uses the idle
game’s reward system to incentivize players to return to the game after
waiting.

####  5.3.2. Highlighting Failure as Part of Play

As argued above, failures in games are more complex than in other AI-infused
systems. While we removed the “when wrong” category of guidelines in our
analysis, we noticed many interesting uses of failure in the player-AI
interactions in the NN games. Overall, failures in games are used
productively. In many NN games, failure is used to motivate the player to
continue improving their AI. This is consistent with the use of failure in
game design (Juul, 2013).

Most notably, we noticed that failure is highlighted, instead of minimized,
from the beginning. When the player first starts the game, the snake they
control dies in How to Train your Snake snake, or robots run to the edges of
the arena instead of approaching the enemy in NERO stanley2005evolving. This
is a design pattern not commonly observed in non-AI games. We believe that the
game designers used this device in order to re-frame players from “problem
makers” to the AI into “problem fixers”, making controlling the NN a less
intimidating task. We noticed that this design strategy was used particularly
in the NN as Apprentice games.

####  5.3.3. Playing with Different forms of Human-AI Interaction

Our analysis highlights the differences between various types of games, most
notably in games using NN as Competitor. Compared to the other interaction
metaphors, they violate many more guidelines. Through close examination,
however, many of the violations are motivated and intentionally designed to be
so. As argued above, when players compete with the AI, common design
assumptions such as transparency and explainability do not apply directly and
needs to be re-examined and adapted for the context.

Literature on human-AI interaction primarily focuses on the paradigm of AI as
a tool/augmentation to the user. However, an increasing number of AI-infused
products fall outside this assumption. For example, AI in cybersecurity
applications explores AI as an adversary, and AI products with high privacy
concerns (e.g., in healthcare) require a different way to think about
transparency. In these cases, we believe NN games can offer many design
insights and cases for inspiration. We also need to expand current human-AI
interaction guidelines so that it can encompass elements essential to play,
such as engagement, flow, and fun.

##  6\. The Future of Player-AI Interaction

Through the specific case of NN games, we have demonstrated the richness of
player-AI interaction as a research topic. In this section, we intend to
situate player-AI interaction in the broader context of related research
areas. We then distill the design implications of our study for games and for
UX/HCI.

###  6.1. Establishing Player-AI Interaction

The study of players, game design, and game technology (here we focus on AI)
are three main pillars in games research (Figure 8). At the intersection of
Games and AI, there is a well-established research community of game AI. In
fact, game AI has often been driving the world of AI research, in which the
most advanced forms of AI algorithms are often first developed and tested in
games (Risi and Preuss, 2020). Results in domains such as StarCraft 2 and
Quake III now frequently appear in the most prestigious journals (Vinyals et
al., 2019; Jaderberg et al., 2019). Combining the study of the player with
game design, there is the research area of player experience. However, the
intersection between players and AI has so far been relatively under-explored.
Until recently, real players are typically only brought in at the end of game
AI research as the means to evaluate the effectiveness of the algorithms.
Carving out the topic of player-AI interaction will fill in this gap.

Figure 8. Research areas of games.

In this paper, we present the first empirical study on this gap that we call
player-AI interaction. In addition to establishing games as a rich domain for
human-AI interaction, our analyses contribute insights into how we can
classify AI-based games based on interaction metaphors (i.e., apprentice,
designer, teammate, and competitor) and the visibility of the AI system as
part of the core UI (i.e., specific, limited, and agnostic). We further
adapted design guidelines for human-AI interaction to the context of games,
which can be useful for others when designing their AI-based games. However,
we encourage the community to further scrutinize these design guidelines as
our work indicates that AI-based games are different from AI-infused products,
for example, with regards to the role of failure, and to work towards more
formalized and evaluated design guidelines for player-human interaction.

###  6.2. Encouraging Playing with AI

For game designers, UX designers, and HCI researchers interested in human-AI
interaction, one of our key takeaways is that reframing AI as play offers a
useful design approach, complementary to the current instrument-based views on
AI. As as play can offer new human-AI interaction design space where the users
can tinker, explore, and experiment with AI. We propose the following design
considerations:

Use flow to structure the learning curve of human-AI interaction. For many
users, interaction with AI can be overwhelming, especially when they encounter
unexpected output from the algorithm. One important lesson from our study is
that the concept of flow (Csikszentmihalyi, 1990), widely used to balance game
difficulty and player engagement over time, can be useful to design human-AI
interaction. In Section 5.3, we discussed that NN games should better support
players for “Learning AI and Its Limitations” and make experimentation with AI
more acceptable by “Highlighting Failure as Part of Play.” The use of flow can
be useful to structure how to gradually expose users to different AI features
(see also (Cruz and Uresti, 2017)).

Incorporate enhanced discovery-based learning. Many games in our analysis,
especially simulation games, offer discovery-based learning (Alfieri et al.,
2011) with mixed success. Since players come with different background
knowledge and needs, explicit instruction for AI is challenging to design.
Discovery-based learning offers players the opportunity to play around with
the NN at their own pace and observe the consequences of their actions on the
NN and the game world. However, most NN games in our dataset offered very
little scaffold, making it difficult for players without a technical
background to succeed. We suggest that UX designers use enhanced discovery-
based learning and provide feedback, worked examples, scaffolding, and
elicited explanations to further assist their users.

Extend the invitation to play. Finally, for researchers and designers
interested in exploring new forms of human-AI interaction, we believe offering
users an invitation to play can unleash their imagination and empower them to
explore new ways to interact with even the same technology. As we can see from
Hey Robot!, the magic circle of play turns the smart speaker user from the
seeker of information to the provider. The voice assistant’s inability to
understand user command/intent is transformed from failure to perform to the
source of fun.

##  7\. Limitations

We recognize that several limitations impact the scope of our work. First, our
study considered only NN games with specific tags on popular game platforms
and textbooks. Our goal was to find a representative sample of salient NN
games, not a comprehensive list. However, we acknowledge that we may have
missed some relevant games. Second, we omitted the failure-related human-AI
interaction guidelines. While we offered some related observations, further
research is needed to study failure in games, separate from failure outside
the context of play, and how it relates to player-AI interaction. Third, we
used 3-point codes in our qualitative analysis. While it is appropriate for
our purpose of the analysis, future research can adopt a more fine-grained
analysis that can better distinguish the “violations” and “does not apply.”
Finally, we do not have necessary (sufficient) technical information about how
NNs are used in many commercial games. The blackbox nature of game AI limited
our ability to conduct in-depth analyses of specific features of games (see
“Multiple AI?” in Table 1).

##  8\. Conclusion

We introduced the term player-AI interaction to study how human players
interact with AI in the context of games. While we intend to situate it in the
broader context of human-AI interaction, we also highlight the unique
opportunities and challenges presented by re-framing AI as play. Through a
systematic search of existing neural network games, we conducted two deep
qualitative analyses. In the first one, we analyzed the common metaphors that
structure the player-AI interaction and how much the NNs are foregrounded in
the core UI. In the second analysis, we adapted the current human-AI
interaction guidelines to player-AI interaction and applied them to identify
the strengths and weaknesses of NN games. Based on our findings, we proposed
that the notion of AI as play, which is an alternative to the current paradigm
of performance-centric human-AI interaction, can contribute to both game
design and HCI communities.

###### Acknowledgements.

This work is partially supported by the National Science Foundation (NSF)
under Grant Number IIS-1816470 and a DFF-Danish ERC-programme grant
(9145-00003B). The authors would like to thank all past and current members of
the project, especially Evan Freed and Anna Acosta for assistance in
collecting initial data. We want to thank those who suggested additional games
on Twitter. Finally, we thank Robert C. Gray for assistance in editing this
paper.

## References

  * (1)
  * ste (2020) 2020\.  Steam (service) — Wikipedia, The Free Encyclopedia.  https://en.wikipedia.org/w/index.php?title=Steam_(service)&oldid=978516977 [Online].
  * Abeele et al. (2020) Vero Vanden Abeele, Katta Spiel, Lennart Nacke, Daniel Johnson, and Kathrin Gerling. 2020.  Development and validation of the player experience inventory: A scale to measure player experiences at the level of functional and psychosocial consequences.  _International Journal of Human-Computer Studies_ 135 (2020), 102370\.
  * Agre (1997) Philip E Agre. 1997\.  _Computation and human experience_.  Cambridge University Press.
  * Alfieri et al. (2011) Louis Alfieri, Patricia J Brooks, Naomi J Aldrich, and Harriet R Tenenbaum. 2011.  Does discovery-based instruction enhance learning?  _Journal of educational psychology_ 103, 1 (2011), 1\.
  * Alharthi et al. (2018) Sultan A Alharthi, Olaa Alsaedi, Zachary O Toups, Joshua Tanenbaum, and Jessica Hammer. 2018\.  Playing to wait: A taxonomy of idle games. In _Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems_. 1–15.
  * Amershi et al. (2019) Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul N Bennett, Kori Inkpen, et al. 2019\.  Guidelines for human-ai interaction. In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_. 1–13.
  * Anderson et al. (2018) Craig G Anderson, Jen Dalsen, Vishesh Kumar, Matthew Berland, and Constance Steinkuehler. 2018\.  Failing up: How failure in a game environment promotes learning through discourse.  _Thinking Skills and Creativity_ 30 (2018), 135–144.
  * Bansal et al. (2019) Gagan Bansal, Besmira Nushi, Ece Kamar, Walter S Lasecki, Daniel S Weld, and Eric Horvitz. 2019\.  Beyond accuracy: The role of mental models in human-AI team performance. In _Proceedings of the AAAI Conference on Human Computation and Crowdsourcing_ , Vol. 7. 2–11.
  * Binns et al. (2018) Reuben Binns, Max Van Kleek, Michael Veale, Ulrik Lyngs, Jun Zhao, and Nigel Shadbolt. 2018\.  ’It’s Reducing a Human Being to a Percentage’ Perceptions of Justice in Algorithmic Decisions. In _Proceedings of the 2018 Chi conference on human factors in computing systems_. 1–14.
  * Bolter (2016) Jay David Bolter. 2016\.  Remediation.  _The international encyclopedia of communication theory and philosophy_ (2016), 1–11.
  * Brockmyer et al. (2009) Jeanne H Brockmyer, Christine M Fox, Kathleen A Curtiss, Evan McBroom, Kimberly M Burkhart, and Jacquelyn N Pidruzny. 2009.  The development of the Game Engagement Questionnaire: A measure of engagement in video game-playing.  _Journal of Experimental Social Psychology_ 45, 4 (2009), 624–634.
  * Bryson (2010) Joanna J Bryson. 2010\.  Robots should be slaves.  _Close Engagements with Artificial Companions: Key social, psychological, ethical and design issues_ (2010), 63–74.
  * Calleja (2007) Gordon Calleja. 2007\.  Digital game involvement: A conceptual model.  _Games and culture_ 2, 3 (2007), 236–260.
  * Campbell et al. (2002) Murray Campbell, A Joseph Hoane Jr, and Feng-hsiung Hsu. 2002\.  Deep blue.  _Artificial intelligence_ 134, 1-2 (2002), 57–83.
  * Cimolino et al. (2019) Gabriele Cimolino, Sam Lee, Quentin Petraroia, and TC Nicholas Graham. 2019.  Oui, Chef!!: Supervised Learning for Novel Gameplay with Believable AI. In _Extended Abstracts of the Annual Symposium on Computer-Human Interaction in Play Companion Extended Abstracts_. 241–246.
  * Cook et al. (2016) Michael Cook, Mirjam Eladhari, Andy Nealen, Mike Treanor, Eddy Boxerman, Alex Jaffe, Paul Sottosanti, and Steve Swink. 2016\.  PCG-Based Game Design Patterns.  _arXiv preprint arXiv:1610.03138_ (2016).
  * Cruz and Uresti (2017) Christian Arzate Cruz and Jorge Adolfo Ramirez Uresti. 2017\.  Player-centered game AI from a flow perspective: Towards a better understanding of past trends and future directions.  _Entertainment Computing_ 20 (2017), 11–24.
  * Csikszentmihalyi (1990) Mihaly Csikszentmihalyi. 1990\.  _Flow: The psychology of optimal experience_. Vol. 1990.  Harper & Row New York.
  * De Graaf et al. (2017) Maartje De Graaf, Somaya Ben Allouch, and Jan Van Diik. 2017\.  Why do they refuse to use my robot?: Reasons for non-use derived from a long-term home study. In _2017 12th ACM/IEEE International Conference on Human-Robot Interaction (HRI_. IEEE, 224–233.
  * Denisova and Cairns (2015) Alena Denisova and Paul Cairns. 2015.  The Placebo Effect in Digital Games: Phantom Perception of Adaptive Artificial Intelligence. In _Proceedings of the 2015 Annual Symposium on Computer-Human Interaction in Play_ (London, United Kingdom) _(CHI PLAY ’15)_. Association for Computing Machinery, New York, NY, USA, 23–33.  https://doi.org/10.1145/2793107.2793109
  * Desurvire and Wiberg (2009) Heather Desurvire and Charlotte Wiberg. 2009.  Game usability heuristics (PLAY) for evaluating and designing better games: The next iteration. In _International conference on online communities and social computing_. Springer, 557–566.
  * Dove and Fayard (2020) Graham Dove and Anne-Laure Fayard. 2020.  Monsters, Metaphors, and Machine Learning. In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_. 1–17.
  * Dove et al. (2017) Graham Dove, Kim Halskov, Jodi Forlizzi, and John Zimmerman. 2017\.  UX design innovation: Challenges for working with machine learning as a design material. In _Proceedings of the 2017 chi conference on human factors in computing systems_. 278–288.
  * Evans and Short (2013) Richard Evans and Emily Short. 2013.  Versu—a simulationist storytelling system.  _IEEE Transactions on Computational Intelligence and AI in Games_ 6, 2 (2013), 113–130.
  * Fulton et al. (2020) Laura Beth Fulton, Ja Young Lee, Qian Wang, Zhendong Yuan, Jessica Hammer, and Adam Perer. 2020.  Getting Playful with Explainable AI: Games with a Purpose to Improve Human Understanding of AI. In _Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems_. 1–8.
  * Furqan et al. (2017) Anushay Furqan, Chelsea Myers, and Jichen Zhu. 2017.  Learnability through adaptive discovery tools in voice user interfaces. In _Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems_. 1617–1623.
  * Games (2016) Firaxis Games. 2016\.  Sid Meier’s Civilization® VI.  https://civilization.com/
  * Gero et al. (2020) Katy Ilonka Gero, Zahra Ashktorab, Casey Dugan, Qian Pan, James Johnson, Werner Geyer, Maria Ruiz, Sarah Miller, David R Millen, Murray Campbell, et al. 2020\.  Mental Models of AI Agents in a Cooperative Game Setting. In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_. 1–12.
  * Gomme and Bartle (2020) Daniel Gomme and Richard Bartle. 2020.  Strategy Games : The Components of A Worthy Opponent. In _Proceedings of 2020 Foundation of Digital Games_.
  * Grand et al. (1997) Stephen Grand, Dave Cliff, and Anil Malhotra. 1997\.  Creatures: Artificial life autonomous software agents for home entertainment. In _Proceedings of the first international conference on Autonomous agents_. 22–29.
  * Guardiola (2016) Emmanuel Guardiola. 2016\.  The gameplay loop: a player activity model for game design and analysis. In _Proceedings of the 13th International Conference on Advances in Computer Entertainment Technology_. 1–7.
  * Guzdial et al. (2019) Matthew Guzdial, Nicholas Liao, Jonathan Chen, Shao-Yu Chen, Shukan Shah, Vishwa Shah, Joshua Reno, Gillian Smith, and Mark O Riedl. 2019.  Friend, collaborator, student, manager: How design of an ai-driven game level editor affects creators. In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_. 1–13.
  * Hastings et al. (2009) Erin J Hastings, Ratan K Guha, and Kenneth O Stanley. 2009\.  Evolving content in the galactic arms race video game. In _2009 IEEE Symposium on Computational Intelligence and Games_. IEEE, 241–248.
  * Heintz and Law (2015) Stephanie Heintz and Effie Lai-Chong Law. 2015.  The game genre map: a revised game classification. In _Proceedings of the 2015 Annual Symposium on Computer-Human Interaction in Play_. 175–184.
  * Herlocker et al. (2000) Jonathan L Herlocker, Joseph A Konstan, and John Riedl. 2000\.  Explaining collaborative filtering recommendations. In _Proceedings of the 2000 ACM conference on Computer supported cooperative work_. 241–250.
  * Hill (2012) Clara E Hill. 2012\.  _Consensual qualitative research: A practical resource for investigating social science phenomena._ American Psychological Association.
  * Hill et al. (2005) Clara E Hill, Sarah Knox, Barbara J Thompson, Elizabeth Nutt Williams, Shirley A Hess, and Nicholas Ladany. 2005.  Consensual qualitative research: An update.  _Journal of counseling psychology_ 52, 2 (2005), 196\.
  * Holmquist (2017) Lars Erik Holmquist. 2017\.  Intelligence on tap: artificial intelligence as a new design material.  _interactions_ 24, 4 (2017), 28–33.
  * Höök (2000) Kristina Höök. 2000\.  Steps to take before intelligent user interfaces become real.  _Interacting with computers_ 12, 4 (2000), 409–426.
  * Hoover et al. (2015) Amy K Hoover, William Cachia, Antonios Liapis, and Georgios N Yannakakis. 2015.  Audioinspace: Exploring the creative fusion of generative audio, visuals and gameplay. In _International Conference on Evolutionary and Biologically Inspired Music and Art_. Springer, 101–112.
  * Horvitz (1999) Eric Horvitz. 1999\.  Principles of mixed-initiative user interfaces. In _Proceedings of the SIGCHI conference on Human Factors in Computing Systems_. 159–166.
  * Jaderberg et al. (2019) Max Jaderberg, Wojciech M Czarnecki, Iain Dunning, Luke Marris, Guy Lever, Antonio Garcia Castaneda, Charles Beattie, Neil C Rabinowitz, Ari S Morcos, Avraham Ruderman, et al. 2019\.  Human-level performance in 3D multiplayer games with population-based reinforcement learning.  _Science_ 364, 6443 (2019), 859–865.
  * Jallov et al. (2016) Daniel Jallov, Sebastian Risi, and Julian Togelius. 2016\.  EvoCommander: A novel game based on evolving and switching between artificial brains.  _IEEE Transactions on Computational Intelligence and AI in Games_ 9, 2 (2016), 181–191.
  * Juul (2013) Jesper Juul. 2013\.  _The art of failure: An essay on the pain of playing video games_.  MIT press.
  * Lakoff and Johnson (2008) George Lakoff and Mark Johnson. 2008.  _Metaphors we live by_.  University of Chicago press.
  * Lakoff and Turner (2009) George Lakoff and Mark Turner. 2009.  _More than cool reason: A field guide to poetic metaphor_.  University of Chicago press.
  * Law et al. (2018) Effie L-C Law, Florian Brühlmann, and Elisa D Mekler. 2018\.  Systematic review and validation of the game experience questionnaire (geq)-implications for citation and reporting practice. In _Proceedings of the 2018 Annual Symposium on Computer-Human Interaction in Play_. 257–270.
  * Lessin and Risi (2015) Dan Lessin and Sebastian Risi. 2015.  Darwin’s Avatars: A Novel Combination of Gameplay and Procedural Content Generation. In _Proceedings of the 2015 Annual Conference on Genetic and Evolutionary Computation_. 329–336.
  * Lubart (2005) Todd Lubart. 2005\.  How can computers be partners in the creative process: classification and commentary on the special issue.  _International Journal of Human-Computer Studies_ 63, 4-5 (2005), 365–369.
  * Lucero et al. (2013) Andrés Lucero, Jussi Holopainen, Elina Ollila, Riku Suomela, and Evangelos Karapanos. 2013\.  The playful experiences (PLEX) framework as a guide for expert evaluation. In _Proceedings of the 6th International Conference on Designing Pleasurable Products and Interfaces_. 221–230.
  * Luger and Sellen (2016) Ewa Luger and Abigail Sellen. 2016.  ”Like Having a Really Bad PA”: The Gulf between User Expectation and Experience of Conversational Agents. In _Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems_ (San Jose, California, USA) _(CHI ’16)_. Association for Computing Machinery, New York, NY, USA, 5286–5297.  https://doi.org/10.1145/2858036.2858288
  * Mateas (1999) Michael Mateas. 1999\.  An Oz-centric review of interactive drama and believable agents.  In _Artificial intelligence today_. Springer, 297–328.
  * Mateas (2003) Michael Mateas. 2003\.  Expressive AI: A semiotic analysis of machinic affordances. In _3rd Conference on Computational Semiotics for Games and New Media_. 58.
  * Mateas and Stern (2003) Michael Mateas and Andrew Stern. 2003.  Façade: An experiment in building a fully-realized interactive drama. In _Game developers conference_ , Vol. 2. 4–8.
  * McDonald et al. (2019) Nora McDonald, Sarita Schoenebeck, and Andrea Forte. 2019\.  Reliability and inter-rater reliability in qualitative research: Norms and guidelines for CSCW and HCI practice.  _Proceedings of the ACM on Human-Computer Interaction_ 3, CSCW (2019), 1–23.
  * Mnih et al. (2013) Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. 2013.  Playing atari with deep reinforcement learning.  _arXiv preprint arXiv:1312.5602_ (2013).
  * Montfort (2005) Nick Montfort. 2005\.  _Twisty Little Passages: an approach to interactive fiction_.  Mit Press.
  * Myers et al. (2018) Chelsea Myers, Anushay Furqan, Jessica Nebolsky, Karina Caro, and Jichen Zhu. 2018.  Patterns for how users overcome obstacles in voice user interfaces. In _Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems_. 1–7.
  * Myers et al. (2020) Chelsea M. Myers, Jiachi Xie, and Jichen Zhu. 2020.  A Game-Based Approach for Helping Designers Learn Machine Learning Concepts.  arXiv:arXiv:2009.05605
  * Nacke et al. (2009) Lennart Nacke, Anders Drachen, Kai Kuikkaniemi, Joerg Niesenhaus, Hannu J Korhonen, Wouter M Hoogen, Karolien Poels, Wijnand A IJsselsteijn, and Yvonne AW De Kort. 2009\.  Playability and player experience research. In _Proceedings of digra 2009: Breaking new ground: Innovation in games, play, practice and theory_. DiGRA.
  * Norman (2013) Don Norman. 2013\.  _The design of everyday things: Revised and expanded edition_.  Basic books.
  * Ontanón et al. (2013) Santiago Ontanón, Gabriel Synnaeve, Alberto Uriarte, Florian Richoux, David Churchill, and Mike Preuss. 2013.  A survey of real-time strategy game AI research and competition in StarCraft.  _IEEE Transactions on Computational Intelligence and AI in games_ 5, 4 (2013), 293–311.
  * Oulasvirta et al. (2020) A. Oulasvirta, N. R. Dayama, M. Shiripour, M. John, and A. Karrenbauer. 2020.  Combinatorial Optimization of Graphical User Interface Designs.  _Proc. IEEE_ 108, 3 (2020), 434–464.
  * Pemberton et al. (2019) Derrick Pemberton, Zhiguo Lai, Lotus Li, Shitong Shen, Jue Wang, and Jessica Hammer. 2019\.  AI or Nay-I? Making Moral Complexity More Accessible. In _Extended Abstracts of the Annual Symposium on Computer-Human Interaction in Play Companion Extended Abstracts_ (Barcelona, Spain) _(CHI PLAY ’19 Extended Abstracts)_. Association for Computing Machinery, New York, NY, USA, 281–286.  https://doi.org/10.1145/3341215.3358248
  * Rabin (2015) Steven Rabin. 2015\.  _Game AI pro 2: collected wisdom of game AI professionals_.  AK Peters/CRC Press.
  * Rader et al. (2018) Emilee Rader, Kelley Cotter, and Janghee Cho. 2018\.  Explanations as mechanisms for supporting algorithmic transparency. In _Proceedings of the 2018 CHI conference on human factors in computing systems_. 1–13.
  * Richards and Hemphill (2018) K Andrew R Richards and Michael A Hemphill. 2018.  A practical guide to collaborative qualitative data analysis.  _Journal of Teaching in Physical Education_ 37, 2 (2018), 225–231.
  * Riedl and Zook (2013) Mark Owen Riedl and Alexander Zook. 2013.  AI for game production. In _2013 IEEE Conference on Computational Inteligence in Games (CIG)_. IEEE, 1–8.
  * Risi et al. (2015) Sebastian Risi, Joel Lehman, David B D’Ambrosio, Ryan Hall, and Kenneth O Stanley. 2015.  Petalz: Search-based procedural content generation for the casual gamer.  _IEEE Transactions on Computational Intelligence and AI in Games_ 8, 3 (2015), 244–255.
  * Risi and Preuss (2020) Sebastian Risi and Mike Preuss. 2020.  From Chess and Atari to StarCraft and Beyond: How Game AI is Driving the World of AI.  _KI-Künstliche Intelligenz_ 34, 1 (2020), 7–17.
  * Risi and Togelius (2015) Sebastian Risi and Julian Togelius. 2015.  Neuroevolution in games: State of the art and open challenges.  _IEEE Transactions on Computational Intelligence and AI in Games_ 9, 1 (2015), 25–41.
  * Sharp et al. (2019) Helen Sharp, Yvonne Rogers, and Jennifer Preece. 2019\.  _Interaction Design: beyond human-computer interaction_ (5th ed.).  John Wiley & Sons.
  * Silver et al. (2016) David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. 2016\.  Mastering the game of Go with deep neural networks and tree search.  _nature_ 529, 7587 (2016), 484–489.
  * Smith and Linden (2017) Brent Smith and Greg Linden. 2017.  Two decades of recommender systems at Amazon. com.  _Ieee internet computing_ 21, 3 (2017), 12–18.
  * Snodgrass and Ontañón (2014) Sam Snodgrass and Santiago Ontañón. 2014.  Experiments in map generation using Markov chains.. In _FDG_.
  * Stanley (2007) Kenneth O Stanley. 2007\.  Compositional pattern producing networks: A novel abstraction of development.  _Genetic programming and evolvable machines_ 8, 2 (2007), 131–162.
  * Stanley et al. (2005) Kenneth O Stanley, Bobby D Bryant, and Risto Miikkulainen. 2005\.  Evolving neural network agents in the NERO video game.  _Proc. IEEE_ (2005), 182–189.
  * Steinfeld et al. (2006) Aaron Steinfeld, Terrence Fong, David Kaber, Michael Lewis, Jean Scholtz, Alan Schultz, and Michael Goodrich. 2006.  Common metrics for human-robot interaction. In _Proceedings of the 1st ACM SIGCHI/SIGART conference on Human-robot interaction_. 33–40.
  * Sukis (2019) Jennifer Sukis. 2019\.  Ai design & practices guidelines (a review).  https://medium.com/design-ibm/ai-design-guidelines-e06f7e92d864
  * Sung et al. (2007) Ja-Young Sung, Lan Guo, Rebecca E Grinter, and Henrik I Christensen. 2007.  “My Roomba is Rambo”: intimate home appliances. In _International conference on ubiquitous computing_. Springer, 145–162.
  * Sweetser and Johnson (2004) Penelope Sweetser and Daniel Johnson. 2004.  Player-centered game environments: Assessing player opinions, experiences, and issues. In _International Conference on Entertainment Computing_. Springer, 321–332.
  * Takahashi (2018) Dean Takahashi. 2018\.  How Microsoft’s Turn 10 fashioned the A.I. for cars in Forza Motorsport 5 (interview).  https://venturebeat.com/2013/11/06/how-microsofts-turn-10-fashioned-the-ai-for-cars-in-forza-motorsport-5-interview/
  * Tintarev and Masthoff (2007) Nava Tintarev and Judith Masthoff. 2007.  A survey of explanations in recommender systems. In _2007 IEEE 23rd international conference on data engineering workshop_. IEEE, 801–810.
  * Treanor et al. (2015) Mike Treanor, Alexander Zook, Mirjam P Eladhari, Julian Togelius, Gillian Smith, Michael Cook, Tommy Thompson, Brian Magerko, John Levine, and Adam Smith. 2015\.  AI-based game design patterns.  (2015).
  * Valls-Vargas et al. (2015) Josep Valls-Vargas, Santiago Ontanón, and Jichen Zhu. 2015\.  Exploring player trace segmentation for dynamic play style prediction. In _Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment_ , Vol. 11.
  * Valls-Vargas et al. (2017) Josep Valls-Vargas, Jichen Zhu, and Santiago Ontañón. 2017\.  Graph grammar-based controllable generation of puzzles for a learning game about parallel programming. In _Proceedings of the 12th International Conference on the Foundations of Digital Games_. 1–10.
  * Valve (2008) Valve. 2008.  Left 4 Dead.  https://www.l4d.com
  * Vinyals et al. (2019) Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. 2019\.  Grandmaster level in StarCraft II using multi-agent reinforcement learning.  _Nature_ 575, 7782 (2019), 350–354.
  * Wang et al. (2019) Danding Wang, Qian Yang, Ashraf Abdul, and Brian Y Lim. 2019\.  Designing theory-driven user-centric explainable AI. In _Proceedings of the 2019 CHI conference on human factors in computing systems_. 1–15.
  * Weizenbaum (1966) Joseph Weizenbaum. 1966\.  ELIZA—a computer program for the study of natural language communication between man and machine.  _Commun. ACM_ 9, 1 (1966), 36–45.
  * Wexler (2002) James Wexler. 2002\.  Artificial Intelligence in Games.  _Rochester: University of Rochester_ (2002).
  * Winograd (2006) Terry Winograd. 2006\.  Shifting viewpoints: Artificial intelligence and human–computer interaction.  _Artificial intelligence_ 170, 18 (2006), 1256–1258.
  * Yang et al. (2018a) Qian Yang, Nikola Banovic, and John Zimmerman. 2018a.  Mapping machine learning advances from hci research to reveal starting places for design innovation. In _Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems_. 1–11.
  * Yang et al. (2018b) Qian Yang, Alex Scuito, John Zimmerman, Jodi Forlizzi, and Aaron Steinfeld. 2018b.  Investigating how experienced UX designers effectively work with machine learning. In _Proceedings of the 2018 Designing Interactive Systems Conference_. 585–596.
  * Yang et al. (2020) Qian Yang, Aaron Steinfeld, Carolyn Rosé, and John Zimmerman. 2020.  Re-Examining Whether, Why, and How Human-AI Interaction Is Uniquely Difficult to Design. In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_ (Honolulu, HI, USA) _(CHI ’20)_. Association for Computing Machinery, New York, NY, USA, 1–13.  https://doi.org/10.1145/3313831.3376301
  * Yannakakis and Togelius (2018) Georgios N Yannakakis and Julian Togelius. 2018.  _Artificial intelligence and games_. Vol. 2.  Springer.
  * Young et al. (2004) R Michael Young, Mark O Riedl, Mark Branly, Arnav Jhala, RJ Martin, and CJ Saretto. 2004\.  An architecture for integrating plan-based behavior generation with interactive game environments.  _J. Game Dev._ 1, 1 (2004), 1–29.
  * Zhu (2009) Jichen Zhu. 2009\.  _Intentional systems and the artificial intelligence (ai) hermeneutic network: Agency and intentionality in expressive computational systems_.  Ph.D. Dissertation. Georgia Institute of Technology.
  * Zhu et al. (2019) Jichen Zhu, Katelyn Alderfer, Anushay Furqan, Jessica Nebolsky, Bruce Char, Brian Smith, Jennifer Villareale, and Santiago Ontañón. 2019.  Programming in game space: how to represent parallel programming concepts in an educational game. In _Proceedings of the 14th International Conference on the Foundations of Digital Games_. 1–10.
  * Zhu and Harrell (2008) Jichen Zhu and D Fox Harrell. 2008.  Daydreaming with Intention: Scalable Blending-Based Imagining and Agency in Generative Interactive Narrative.. In _AAAI Spring Symposium: Creative Intelligent Systems_ , Vol. 156.
  * Zhu et al. (2018) Jichen Zhu, Antonios Liapis, Sebastian Risi, Rafael Bidarra, and G Michael Youngblood. 2018\.  Explainable AI for designers: A human-centered perspective on mixed-initiative co-creation. In _2018 IEEE Conference on Computational Intelligence and Games (CIG)_. IEEE, 1–8.
  * Zhu and Ontanón (2010) Jichen Zhu and Santiago Ontanón. 2010.  Towards Analogy-Based Story Generation.. In _ICCC_. 75–84.
  * Zhu and Ontañón (2013) Jichen Zhu and Santiago Ontañón. 2013.  Shall I compare thee to another story?—An empirical study of analogy-based story generation.  _IEEE Transactions on Computational Intelligence and AI in Games_ 6, 2 (2013), 216–227.

ACM-Reference-Format references_games *