# Patterns for Representing Knowledge Graphs to Communicate Situational
Knowledge of Service Robots

Shengchen Zhang<EMAIL_ADDRESS>College of Design and
Innovation, Tongji University281 Fuxin RdShanghaiChina ,  Zixuan Wang
<EMAIL_ADDRESS>College of Design and Innovation, Tongji
University281 Fuxin RdShanghaiChina ,  Chaoran Chen<EMAIL_ADDRESS>College of Design and Innovation, Tongji University281 Fuxin RdShanghaiChina ,
Yi Dai<EMAIL_ADDRESS>College of Design and Innovation, Tongji
University281 Fuxin RdShanghaiChina ,  Lyumanshan Ye<EMAIL_ADDRESS>College of Design and Innovation, Tongji University281 Fuxin RdShanghaiChina
and  Xiaohua Sun<EMAIL_ADDRESS>College of Design and Innovation, Tongji
University281 Fuxin RdShanghaiChina

(2021)

###### Abstract.

Service robots are envisioned to be adaptive to their working environment
based on situational knowledge. Recent research focused on designing visual
representation of knowledge graphs for expert users. However, how to generate
an understandable interface for non-expert users remains to be explored. In
this paper, we use knowledge graphs (KGs) as a common ground for knowledge
exchange and develop a pattern library for designing KG interfaces for non-
expert users. After identifying the types of robotic situational knowledge
from the literature, we present a formative study in which participants used
cards to communicate the knowledge for given scenarios. We iteratively coded
the results and identified patterns for representing various types of
situational knowledge. To derive design recommendations for applying the
patterns, we prototyped a lab service robot and conducted Wizard-of-Oz
testing. The patterns and recommendations could provide useful guidance in
designing knowledge-exchange interfaces for robots.

Design Patterns, Interface Design, Knowledge Graph, Human-Robot Interaction

††copyright: acmcopyright††journalyear: 2021††price: 15.00††doi:
10.1145/3411764.3445767††isbn: 978-1-4503-8096-6/21/05††conference: CHI
Conference on Human Factors in Computing Systems; May 8–13, 2021; Yokohama,
Japan††booktitle: CHI Conference on Human Factors in Computing Systems (CHI
’21), May 8–13, 2021, Yokohama, Japan††submissionid: 7721††ccs: Human-centered
computing Graphical user interfaces††ccs: Human-centered computing User
interface design††ccs: Computer systems organization Robotics††ccs: Computing
methodologies Cognitive robotics††ccs: Human-centered computing User studies

Figure 1. In this work, we use observations and analysis of human presentation
of situational knowledge to develop a pattern library for interface design. We
further derive design recommendations by prototyping and testing the interface
of a service robot.

A three-step image illustrating the process of our research framework. We use
observations and analysis of human presentation of situational knowledge to
develop an interface design pattern library. We further derive design
recommendations by prototyping and testing the interface of a service robot.

##  1\. Introduction

Service robots are increasingly pervasive in our daily life and are
anticipated to execute complex tasks based on high-level goals and interact
with users in an easily-understandable way(Kattepur, 2019). This requires
robots to effectively organize and represent situational knowledge — the in-
situ information about humans, objects, places, and events in the robot’s
working environment(Javia and Cimiano, 2016). However, since situational
knowledge is often highly related to users and objects on the spot, robots
need to interact with humans to mediate the mismatch between perception and
comprehension of situational knowledge(Fang et al., 2015).

Knowledge graphs(Hogan et al., 2020) are a common method used in both
artificial intelligence (AI) to incorporate human knowledge into AI models,
and in robotics to build knowledge-enabled robots. Previous research have been
exploring designing interfaces to help domain experts to directly view and
manipulate the knowledge graph in a robot in order to understand and operate
it(Lemaignan et al., 2017a). However, for non-expert users who have little
technical knowledge in ontological models like KGs, it can be hard to
understand and interact with such interfaces. Existing works are developing a
set of human-friendly vocabulary to build robot ontology(Diprose et al.,
2012b), but research on KG interface design for non-expert users is still
relatively few.

Meanwhile, previous works showed that design patterns benefit the field of HCI
and HRI by providing reusable solutions for recurring problems(Kahn et al.,
2008a). For KG interfaces, the study of interface design patterns is
especially useful. A set of patterns can lower the requirement on the
understanding of technical details for the designers, while also accelerating
the development of such interfaces.

This paper describes the process (Figure 1) of developing a library of
patterns for visually presenting situational, robotic knowledge graphs. The
goal of the patterns is to help designing effective interfaces to communicate
the situational knowledge of service robots.

Our contribution is three-fold:

  * •

We developed a pattern library of high-level and low-level patterns of
situational knowledge communication that can be used to design robot
interfaces.

  * •

We described in detail our process of constructing the library from how non-
expert users present KG elements on a canvas.

  * •

We further derived design recommendations for using the pattern library
through prototyping and a Wizard-of-Oz field study.

##  2\. related work

Our work builds on prior research on situational knowledge exchange in humans
and robots, challenges of knowledge graph in cognitive robot, and patterns of
human robot interaction.

###  2.1. Situational knowledge exchange in humans and robots

Situational knowledge exchange has played an increasingly critical role in
human-robot interaction, for both robotic perception and comprehension(Javia
and Cimiano, 2016). Many studies have focused on leverage symbol grounding
techniques (connecting numeric and symbolic representation of real-world
objects and the environment) to make situational knowledge exchange more
accurate and easier to understand (Bastianelli et al., 2013). These techniques
have been used to support situational knowledge representation for
robots(Topp, 2017; Pronobis and Jensfelt, 2012; Nieto-Granda et al., 2010),
situation assessment in human-robot interaction(Riley et al., 2010; Sisbot et
al., 2011), and context-based human-robot collaboration(Zachary et al., 2013,
2015).

Most relevant to our work is the research focused on grounding and
representing situational knowledge for robots in a human-understandable way.
For example, Bastianelli et al.(Bastianelli et al., 2013) proposed using
metric maps and a multi-model interface that allowed users to guide robots to
ground symbolic information in the operational environment. Carpenter et
al.(Carpenter and Zachary, 2017) presented CARIL, a computational architecture
that used declarative situational representation to adapt robots’ behavior to
humans during their collaboration. While several aspects of exchanging
situational knowledge have been studied, such as creating robot task
plans(Paxton et al., 2018; Kattepur, 2019) and programming the Robotic
Operating System (ROS)(Tiddi et al., 2017; Leonardi et al., 2019; Diprose et
al., 2012b), few studies explored a unified form of visual representation for
situational knowledge in robots.

Knowledge graphs can represent heterogeneous and interconnected
knowledge(Jiang, 2020), therefore often used as a unified format of knowledge
representation in robots. Our work aims to identify the patterns and represent
situational knowledge based on the knowledge graph in a service robot.

###  2.2. Challenges of knowledge graph in cognitive robots

Although knowledge graphs have been applied to facilitate human-robot
interaction, it is hindered by three main challenges: static systems,
monotonous interaction modality and intricate interfaces.

Firstly, robots based on static systems are hard to adapt to user preferences
and intuitively interact with users (Jokinen et al., 2019; Javia and Cimiano,
2016). To address this problem and explore human-friendly systems, some recent
research enhanced the accuracy of human-robot knowledge communication by
acquiring multi-modal human behavior data or the AR interface (Kennington and
Shukla, 2017; Liu et al., 2018; Tsiakas et al., 2017).

Secondly, previous research has indicated that uni-modal interaction
(typically voice interaction for robots) might lead to misunderstanding the
users’ requests(Kennington and Shukla, 2017). Liu et al. proposed a graphic
interface that adds synchronous visual feedback of users’ decision sequence
based on a dialogue interaction system, which successfully reduced
misunderstandings(Liu et al., 2010). It demonstrates the effectiveness of GUI-
based interaction in human-robot knowledge communication.

Thirdly, there is still relatively few research focused on enhancing non-
expert users’ understanding of ontological models and developing interactive
systems for this purpose. Some relevant research explored human-readable ways
of describing robot behavior(Diprose et al., 2012b), and designed an interface
using ontological abstraction to help non-experts users to program robots with
low learning cost(Tiddi et al., 2018). These works inspired us to explore how
to generate a systemic and understandable interface for exchanging knowledge
between non-expert users and service robots.

###  2.3. Patterns of human robot interaction

Pattern language was first proposed by Alexander in 1977(Alexander, 1977), and
then thrived in the field of Human computer interaction(HCI)(Pea, 1987;
Seffah, 2010) and human robot interaction(HRI)(Kahn et al., 2008b; Mioch et
al., 2014; Sauppé and Mutlu, 2014). Kahn et al. argued that design patterns
could benefit HRI by providing designers with the necessary knowledge and save
their time when reusing these patterns to solve recurring problems(Kahn et
al., 2008a). Patterns also serve as scaffolding to help future research
explore their relevant direction.

Prior research focused on extracting patterns to guide the design of HRI. For
example, Oliveira et al. yielded social interaction patterns by observing
human and robot players’ behaviors in card games(Oliveira et al., 2018). Oguz
et al. developed an ontological framework from human interaction
demonstrations, which could be transferred to HRI scenarios(Oguz et al.,
2019).

Most relevant to our research are the patterns of presenting robot knowledge
on an graphical interface. Robots have shown the ability to use interface to
effectively communicate various types of information and promote task
efficiency in teleoperating(Barba et al., 2020; Chen et al.,
2007),manufacturing(Marvel et al., 2020) and elderly care(Klakegg et al.,
2017). Although prior research showed interest in exploring the design of
interfaces, few focused on extracting patterns of knowledge presentation on
the robot’s interface. Our work aims to facilitate more design of robot
interface to exchange situational knowledge.

##  3\. Formative Study

Alexander points out in his foundational work(Alexander, 1979) that patterns
“cannot be made, but only be generated, indirectly, by the ordinary actions of
the people.” Regarding applying design patterns to HRI, Freier et al. further
argued that design patterns in nature are “patterns of human interaction with
the physical and social world(Kahn et al., 2008b).” In line with this
reasoning, we grounded our formative study in observations and analysis of
non-expert users’ methods to visually communicate situational knowledge.
Participants are tasked with presenting the knowledge needed in an HRI
scenario using cards on a canvas. We then iteratively coded the results and
identified patterns for presenting various types of robotic situational
knowledge. The following sections describe the process of discovering and
formalizing these patterns.

###  3.1. Scenarios

To create a comprehensive and believable setting to ground our observation, we
first identify three HRI scenarios that span three categories of situational
knowledge exchange interactions: semantic, procedural and episodic. As
knowledge communication involves both robot and human parties, we referenced
research in both human knowledge (Alexander and Judy, 1988; de Jong and
Ferguson-Hessler, 1996) and robotic knowledge(Laird et al., 2017). We adopted
the categorization in (Laird et al., 2017) as the knowledge types are
applicable to both humans and robots, and are more robot-specific.

##### Communicating semantic knowledge

Semantic knowledge is ”semantically abstract facts(Laird et al., 2017).” In
semantic scenarios, the participants are asked to present information about
user information, object ownership, or environment locations on the canvas.

##### Communicating procedural knowledge

Procedural knowledge is ”knowledge about actions, whether internal or
external(Laird et al., 2017).” In procedural scenarios, the participants are
asked to present information about robot tasks or actions in a planned
procedure.

##### Communicating episodic knowledge

Episodic knowledge is ”contextualized experiential knowledge”(Laird et al.,
2017). In episodic scenarios, the participants present knowledge such as a
perceived sequence of events or information related to a certain period of
time.

To ensure our results’ applicability to a wide variety of working
environments, we expanded each scenario into three types of contexts from
private to public: home, office, and elderly care center. This results in nine
scenarios used in the formative study.

We use comic strips to present each scenario to the participants to reduce
narration’s influence on the results. An example of the scenario comic is
shown in Figure 2. The complete collection of comics can be found in the
supplementary materials.

Figure 2. An example of the scenario comic

An example of the scenario comic. It consists of four panels. The first three
depict a scene where a user asks the robot what it has learned on its first
day at home. The fourth panel has instructions for the participant. It asks
the participant to play the role of the robot, and arrange cards on the canvas
to display the information.

Figure 3. The design of the knowledge cards. On the right is an example of a
card in the group “User”, with ontology term “User profile”.

The design of the knowledge cards. The card is hexagonal with shaded outlines.
The ontology group that the card belongs to is written in the shaded area,
near the top and bottom of the card. In the center is the name of the KG node
and its ontology term. There is also a small text-box showing the scenario ID
that the card belongs to. The back of the card features a list of related card
names. On the right of the image is an example of a card in the group “User”,
with ontology term “User profile”.

###  3.2. Knowledge Cards

To achieve a more comprehensive coverage of the types of situational knowledge
in service robots, we conducted a literature survey of works on ontology for
service robots in the Scopus database111https://www.scopus.com. Intially, we
used “situational” and its variations (such as “situated” and “in situ”) in a
pilot search. To the best of our efforts, we found no existing works
categorizing robot situational knowledge types. Therefore, we searched for
works categorizing general knowledge types of robot, using the query shown
below:

        {robot AND knowledge graph} OR {robot AND
        ontology} OR {robot AND semantic network} OR
        {robot AND knowledge model} AND NOT industr*
        AND NOT agricultur* AND NOT farming

The search returned 228 papers. We further excluded works that focused on a
specific type of knowledge and those that focused on specific applications of
robot ontology. In the end, 11 papers were selected. We reviewed each paper
for ontology terms and grouped similar terms to form an initial list of robot
knowledge types. We finally removed non-situational knowledge about robotic
components and capabilities to arrive at Table 1.

Table 1. List of situational knowledge used in our formative study Group | Ontology |  Description |  Examples
---|---|---|---
Objects | Object |  A physical entity in the environment, excluding agents such as users and robots. |  (Olivares-Alarcos et al., 2019) (Lim et al., 2011) (Il Hong Suh et al., 2007) (Chang et al., 2020) (Jeon et al., [n.d.]) (Tenorth and Beetz, 2013) (Azevedo et al., 2019) (Lemaignan et al., 2017b) (Kruijff et al., 2007)
| Affordance |  The possibility of action(Olivares-Alarcos et al., 2019) on an object, as perceived by the robot. |  (Olivares-Alarcos et al., 2019) (Jeon et al., [n.d.]) (Tenorth and Beetz, 2013)
Environment | Environment Map |  A spatial representation of the robot’s working environment(Olivares-Alarcos et al., 2019). |  (Olivares-Alarcos et al., 2019) (Lim et al., 2011) (Il Hong Suh et al., 2007) (Chang et al., 2020) (Jeon et al., [n.d.]) (Tenorth and Beetz, 2013) (Lemaignan et al., 2017b) (Kruijff et al., 2007)
Users | User profile |  The basic information of the users(Chang et al., 2020), such as name and ID in the face recognition module. |  (Lemaignan et al., 2017b) (Chang et al., 2020) (Jeon et al., [n.d.]) (Tenorth and Beetz, 2013) (Azevedo et al., 2019) (Mahieu et al., 2019) (Bruno et al., 2019)
| Social concept |  Social knowledge of the user obtained while interacting with a human.(Chang et al., 2020) |  (Chang et al., 2020) (Mahieu et al., 2019) (Bruno et al., 2019)
| Emotion/Intention |  The emotional state and intention of the user, as detected by the robot. |  (Chang et al., 2020) (Azevedo et al., 2019) (Mahieu et al., 2019)
Action | Action & Task |  A task represents “a piece of work that has to be done by the robot(Olivares-Alarcos et al., 2019)”. An action refers to “a way to execute a task(Olivares-Alarcos et al., 2019)”, which we use to denote a step that the robot takes to complete a task. |  (Olivares-Alarcos et al., 2019) (Lim et al., 2011) (Il Hong Suh et al., 2007) (Tenorth and Beetz, 2013) (Mahieu et al., 2019) (Lemaignan et al., 2017b)
| Activity & Behavior |  The behavior that the robot adopts when carrying out a task. Example behaviors may include scripted interactions, open dialogue, or simply command-and-response. |  (Olivares-Alarcos et al., 2019) (Lim et al., 2011) (Il Hong Suh et al., 2007) (Chang et al., 2020) (Jeon et al., [n.d.]) (Tenorth and Beetz, 2013)
| Plan & Method |  A sequence of actions that the robots would take in order to fulfill a task. This can be either programmed or generated. |  (Olivares-Alarcos et al., 2019) (Chang et al., 2020) (Tenorth and Beetz, 2013)
| Interaction & Communication |  A group of pre-defined actions that involves interacting or communicating with the user. This type of knowledge is often associated with one or more types of behavior. |  (Olivares-Alarcos et al., 2019) (Chang et al., 2020) (Tenorth and Beetz, 2013) (Kruijff et al., 2007)
Context | Spatial & temporal context |  (Lim et al., 2011) used these terms to refer to the spatial and temporal relationship between objects. We expand this definition to include non-object concepts such as events and interactions. |  (Lim et al., 2011) (Lemaignan et al., 2017b)
| Situation |  (Lim et al., 2011) used this terms to refer to the detected spatial status of objects (such as “crowded”). We expand this definition to general social situations. |  (Lim et al., 2011) (Lemaignan et al., 2017b)
| Event |  An event is a notable happening detected by the robot. |  (Tenorth and Beetz, 2013)

We then manually compiled knowledge graph datasets for the nine scenarios,
with each dataset spanning all types of knowledge listed. The total number of
nodes is kept roughly the same for each dataset to maintain similar task
difficulty across the scenarios.

We used hexagonal cards to represent each node in the knowledge graph,
inspired by Padilla et al. (Padilla et al., 2017), as hexagon allows for
versatile arrangement and efficient use of space222The source files we used to
print the cards can be found at https://github.com/tongji-cdi/robot-knowledge-
canvases/tree/master/Cards%20PDF.. An example of the card design is shown in
Figure 3.

###  3.3. Set up

The study was carried out in a controlled environment. As shown in Figure 4,
the set up consisted of a table with a sheet of erasable canvas and hexagonal
cards sorted by knowledge types. We provided markers and erasers for
annotation. There are also two empty boxes for participants to put unused and
unclear cards.

Figure 4. A photo of our study set-up. A researcher plays the role of the
participant to protect the participants’ identity.

An image showing our study set-up. Here, a researcher plays the role of the
participant for anonymity reasons. The image shows a tabletop. On the tabletop
there are several items. In front of the participant is a large canvas with
six stacks of cards on it. The participant is taking some cards from the
stacks and arranging them on the canvas. On the left, there are two boxes for
unused and unclear cards. A scenario comic strip is also there. On the right,
there’s a rectangular frame, along with markers and other tools. Farther away
from the participant is a microphone and a video camera on a tripod. The
camera is out of the picture, and points down at the canvas.

###  3.4. Procedure/Task

After signing a consent form, the participant can browse the material and ask
questions. We then asked the participant to arrange the knowledge cards on the
canvas. They are to arrange the cards in a way that they think best
communicates the information as required in the scenario comic. The researcher
leaves the working area during the process and observes the participant
through a live camera feed. The participant can use the ”question” and
”finished” cards to signal the researcher when needed. There’s no time limit
to complete the task.

After the creation, the participant is asked to communicate the knowledge on
the display to the researcher using a frame. Figure 5 shows an example of the
narration. By defining the task in this manner, it allowed us to identify the
grouping and sequence of the cards presented. Finally, the researcher
interviews the participant about the arrangement logic and other interesting
phenomena noticed during the process.

Figure 5. An example of the narration process of one of the participants (P2).

An example of the narration process of participant P2 for scenario C2. The
participant is moving a rectangular frame across the canvas, stopping here and
there to point to and narrate the cards inside the frame.

In total, each participant would go through three similar sessions with
scenarios communicating different knowledge types (semantic, procedural,
episodic), which were randomly paired with three working environments (home,
office, elderly care center). After that, we will ask participants to compare
the knowledge representations they did with the three finished canvases shown
on a screen.

###  3.5. Data Collection

A total of 12 university students aged from 21 to 27 (M = 23.83, SD = 1.51)
from diverse fields took part in this study. We exclude participants
knowledgeable in KG or graph visualization to avoid interference from their
prior knowledge.

During the study, a camera was used to record the behavior of the
participants. We only recorded the canvas area to maintain as much anonymity
as possible. A photo of the canvas was taken after each session to record the
result333The canvases created by the participants can be found at
https://github.com/tongji-cdi/robot-knowledge-canvases.. All interviews were
recorded.

###  3.6. Data Analysis

Grounded theory is a common method for qualitative pattern generation in
content and thematic analysis. However, it is mostly used to analyze interview
text(Martin and Turner, 1986). Methods for analyzing diagram images and
generating patterns from them are less explored. A relevant approach is visual
grounded theory(Konecki, 2011), which analyzes the results of visual
narrations and iteratively codes data into various categories. Although it is
often used to code images from realistic scenes, the work of Chapman et
al.(Chapman et al., 2017) inspired us. We developed a process that consists of
five steps: positional coding, coding verification, abstraction, relational
coding, and pattern generation. An overview of our methodology can be seen in
Figure 6.

Figure 6. The process of our analysis.

A diagram showing the process of our analysis. The steps are the same as
described in Section 3.6.

Figure 7. An illustration of the analysis process.

An illustration of the abstraction process. This diagram shows four stages.
The first panel show small patches selected from the canvas image, and coded
with different names. The second panel shows abstract representation of the
small patterns using circles and arrows. The third panel shows that the
content of the canvas image is substituted by abstract diagrams from the
second panel. The last panel shows that similar diagrams are merged to become
the final patterns.

##### Positional coding:

Two researchers open coded any interesting phenomenon that emerged from the
canvas photos separately. These phenomena are coded as nodes in NVivo444The
NVivo source file is released under https://github.com/tongji-cdi/robot-
knowledge-canvases/releases/download/1.1/Coding_result.nvp.. As a result of
this step, each researcher generated a set of codes with description and
examples.

##### Coding verification:

The two researchers exchanged the set of codes and reviewed them together. The
review process consisted of merging similar codes and standardizing naming
conventions. This process results in a codebook that consists of code names
and criteria for inclusion. After that, two researchers separately re-coded
the images according to the codebook. Comparison of the two researchers’
coding results shows reliable coding agreement (99% agreement, Cohen’s
$\kappa$ = .79).

##### Abstraction:

The researchers first established a standard graphical library of low-level
codes and used them to transcribe the canvas photos into abstract diagrams. An
example of this process can be seen in Figure 7. By doing so, we abstracted
the relationship between the codes away from the cards’ position and names.

##### Relational coding:

One researcher open coded the diagrams generated from the previous step.
During coding, the researcher focused on discovering patterns in the overall
composition of low-level codes while paying particular attention to what
constitutes the main storyline of the diagram. This coding process resulted in
a set of high-level codes. A second researcher reviewed the coding results and
discussed with the first until a consensus is reached. The two researchers
then compiled the final codebook for high-level patterns.

##### Pattern generation:

The two researchers worked together to review the diagrams of low-level and
high-level codes. They identified diagrams that share similar structures and
merged the codes to arrive at the final patterns. In the end, we identified
eight low-level patterns and four high-level patterns. The relations between
these patterns are analyzed to form a pattern library.

Figure 8. An excerpt of our pattern library. The component and module layer
corresponds to the low-level and high-level patterns we discuss below. The
elements corresponds to ontology groups defined in Table 1. The syntax
described in the legends are used throughout this paper.

An excerpt of our pattern library. There are three layers: element, component,
and module. The component and module layer corresponds to the low-level and
high-level patterns we discuss below. The elements corresponds to ontology
groups defined in Table 1.

##  4\. Pattern Library

The pattern library we proposed consists of three parts: element, component,
and module. The element part includes all types of the robotic situational
knowledge we analyzed in Table 1. The components (low-level patterns) are a
typical combination and representation of certain elements. The modules (high-
level patterns) are further built upon the components and can be easily
applied to an interface. The structure of our pattern library can be seen in
Figure 8, which is an example of Module d (Location as the Main Story Line).

###  4.1. Low-level Patterns

Low-level patterns are found as components that make up the whole canvas and
mostly communicate a single message. These patterns can be employed when
designing interface components that display specific information. An overview
of low-level patterns can be found in Figure 9. The following sections provide
a detailed description of these patterns.

Figure 9. An overview of low-level patterns in our pattern library. “Any” in a
diagram means a node of any type. “Entity” denotes something that physically
exist, such as a user, an object, or a location.

An overview of low-level patterns in our pattern library. Eight patterns are
present, each corresponding to a section below.

##### 1) Action Flowchart:

This pattern occurs when the participants want to show the logical flow of a
series of actions. The inner structure connecting the actions in this pattern
can be sequential, branching, and looping, similar to a flowchart.

##### 2) Action and Required Information:

This pattern is used to show that the robot used pre-existing knowledge in
order to undertake an action. It takes the shape of a central action card
connected to its required information. The knowledge type of the required
information can be any of the semantic types, including environment, object,
and user.

##### 3) Action and Its Results:

This pattern is often similar in shape and content as Action and Required
Information. However, it is used to show that an action of the robot resulted
in the addition of certain new knowledge. A typical example is when the robot
recognizes a new agent in its working place. Then it may add the knowledge
automatically or by asking the user related.

##### 4) Event Sequence:

This pattern is used to show a series of events that happened in chronological
order. However, not all events are necessarily shown. Participants often skip
events that they deem irrelevant.

##### 5) Map:

This pattern is used to showcase a collection of entities (e.g., environment,
users, and objects) in their corresponding locations, similar to a real-world
map. The pattern occurs when the knowledge communicated is highly related to
locations, and there’s no apparent order in the knowledge (e.g., time and
alphabetical order).

##### 6) Entity and Relevant Information:

This pattern is employed to show detailed information about the central
entity, which can be any semantic knowledge. A typical example is when a user
card is placed. The participant will surround it with relevant information
like age and room location.

##### 7) Classification Tree:

This pattern is typically a tree-like structure, depicting the hierarchical
classification of a collection of entities of the same type. An example is
when a participant is trying to show all the people in an office. They used
the department information to sort the people into two groups.

##### 8) Assorted Entities:

This pattern describes a cluster of entities of the same type, without any
specific order.

###  4.2. High Level Patterns

Figure 10. An overview of high-level patterns in our pattern library. Shaded
parts indicate the “main story line” of the high-level pattern.

An overview of high-level patterns in our pattern library. Four patterns are
present, each corresponding to a section below.

High-level patterns communicate a more complex message by combining and re-
arranging low-level patterns. These patterns can be employed when designing
interface modules for different knowledge types and scenarios. Figure 10
presents an overview of the high-level patterns. Paragraphs below detailed
each pattern with description, usage scenario, design considerations.

####  4.2.1. Action as the Main Story Line

##### Pattern description:

An Action Flowchart is the most prominent feature of this pattern, used often
in combination with Action and Required Information or Action and Its Results
to show information that’s relevant to each action in the flowchart.

##### When to use:

This pattern is often employed to explain the logical flow of robotic actions
or to narrate the process of gathering specific information. Therefore, it
mostly occurs in scenarios where the communicated knowledge is procedural or
semantic.

##### Design considerations:

The action flowchart can be naturally represented on-screen as a flowchart,
while relevant information is linked to the actions without affecting the
flowchart structure. Participants used various low-level patterns to organize
the related semantic information, such as Assorted Entities and Classification
Tree. The link between the action and the other cards are depicted in various
ways, including arrows, lines, or simply with proximity. This suggests that
interface designers have a wide range of options when employing this pattern.

In scenarios where mainly procedural knowledge is communicated, some of the
participants neglected the semantic cards altogether and chose to present only
the actions. Therefore, we suggest that the relevant information be collapsed
into the action element in a procedural scenario and remain hidden until the
user specifically asks for details of the action.

####  4.2.2. Time as the Main Story Line

##### Pattern description:

This pattern features an Event Sequence as its main component. Other low-level
patterns, especially Entity and Relevant Information, can be used to provide
context.

##### When to use:

This pattern is most typically found in an episodic scenario, where the robot
needs to show the knowledge related to a specific time in the past.

##### Design considerations:

The Event Sequence can be naturally presented as a timeline. Both horizontal
and vertical timelines have been observed during the study. However, no
participants tried to show the exact time interval between two events on the
timeline, even when such information was given. This suggests that when using
this pattern, the ordering of events is essential while the exact timing of
events is not. Many participants find the naming of the event elements hard to
understand, as they are from the robot’s perspective (e.g. “faces detected”,
“start interacting with user”). Some participants omitted events they deemed
irrelevant, especially those without relevant semantic information. This
suggests that designers should set rules to filter out irrelevant events to
the user and translate the event names to user-friendly language when using
this pattern.

####  4.2.3. Relationship as the Main Story Line

##### Pattern description:

This pattern features a large collection of semantic information, which is
typically sorted into groups using Assorted Entities, or displayed as a
Classification Tree. Sometimes a mix of the two is used, where part of the
knowledge is organized in a tree-shaped structure and others categorized into
groups.

##### When to use:

This pattern is often seen in semantic scenarios when the goal is to quickly
convey a large amount of inter-related semantic knowledge to the user. A
typical example is when the user wants to check the knowledge of a specific
type. An interface that sorts knowledge by type can easily guide the user
towards the goal.

##### Design considerations:

The Classification Tree can be shown as a collapsible tree. However, the
designer may also choose to use hyperlinks between multiple screens to reduce
clutter. Meanwhile, Assorted Entities can be represented most naturally as a
list or group of elements on-screen.

However, care must be taken when using Assorted Entities. A large number of
elements of the same type can easily overwhelm the user. Many participants
preferred Classification Tree and considered it a natural way to organize
information. Some participants used symbols to represent cards frequently
linked to and drew these symbols whenever a connection needs to be shown. This
method can prove useful in the face of a large number of relations.

####  4.2.4. Location as the Main Story Line

##### Pattern description:

This pattern uses a Map as its core, linking other related knowledge to
locations in the Map, using the Entity and Relevant Information pattern.

##### When to use:

This pattern is typical in semantic scenarios and occurs when the knowledge to
be communicated highly related to the location. Such knowledge includes
objects, people, and named locations in space. However, participants also used
this pattern when the robot’s action involved traveling to multiple locations.

##### Design considerations:

A map of the working environment would be a suitable background when using
this pattern. The objects, users, or named locations can be placed on the map
accordingly. When using this pattern to describe the robot’s action, some
participants used an abstract representation of the working environment,
placing the locations in a sequence to show the robot’s route.

##  5\. Prototyping

To ensure our pattern library’s applicability, we further investigated how the
patterns may be applied by prototyping a service robot. We conducted Wizard-
of-Oz testing using our prototype and identified several design challenges and
recommendations by analyzing the questionnaire and interview data.

###  5.1. Design

We prototyped a service robot that took participants on a guided tour of the
lab. The rationale behind choosing this scenario is two-fold. First, it is
natural for the robot to communicate a large amount of knowledge of all three
types (semantic, procedural, and episodic) during a tour. Second, it allowed
us to apply real-world data to test the applicability of our patterns.

Three researchers who were not involved in the pattern generation process were
provided with the pattern library. They were given limited time (4 hours) to
use the patterns to design a series of screens for displaying the necessary
knowledge in a specific scenario. We didn’t expect highly complete interfaces
since the focus of the design session is to gain first-hand experience in
applying the patterns.

###  5.2. Implementation

Figure 11. The Wizard-of-Oz testing system implemented in our study.

The Wizard-of-Oz testing system implemented in our study consists of a control
environment in Node-RED, a graph database, and an Android application that
runs on the Temi robot. The Temi application receives commands from Node-RED
and queries information from the KG database.

We implemented the Wizard-of-Oz testing system on the Temi
robot555https://www.robotemi.com. As shown in Figure 11, our system consists
of a graph database using Neo4j666https://neo4j.com for querying the KG, a
remote controlling environment using Node-RED777http://nodered.org, and an
Android application that runs on Temi that receives control commands and
displays the UI888Available at https://github.com/tongji-cdi/temi-woz-frontend
(dataset and UI) and https://github.com/tongji-cdi/temi-woz-android (Node-RED
remote control and Temi Android application).

We then compiled a knowledge graph dataset according to the types of
situational knowledge (Table 1). To test the applicability of our patterns to
real-world problems, we curated data of the users, projects, objects, and
locations from lab administration documents. We took care to anonymize the
data by replacing person and object names.

In the development process, our patterns demonstrated the ability to translate
to knowledge graph queries quickly. An example of this is given in Figure 12.
The designer specified a task flowchart consisting of task steps and relevant
KG nodes. The patterns were then directly translated to the Neo4j query
language Cypher999https://neo4j.com/developer/cypher to pull the knowledge
from the database.

Figure 12. Turning patterns into interface design and a Cypher query.

The designer used the ”Action as the Main Story Line” pattern, which consists
of an action flowchart connected to the relevant information of the actions.
The patterns are used to design an interface template. The designers specified
which patterns are used. The patterns directly correspond to structures in the
robot knowledge graph, which in turn translates to graph database queries that
matches data in the robot KG. The match results are then used to fill
information into the user interface.

###  5.3. Testing

Ten participants without expert knowledge of KGs performed in a simulated
scenario with six tasks to complete. Participants assumed the role of a new
research assistant and were guided by the robot to tour the lab. Along the
tour, the accompanying researcher gave six tasks to the participant and
recorded their completion status. Each task required the participant to look
for some information using the interface on the robot’s screen. The tasks
cover the three scenarios (communicating semantic, procedural, and episodic
knowledge) as outlined in Section 3.1. Each scenario corresponds to two tasks,
one requiring the participant to look for information on the current robot
screen, and one requiring browsing the KG to search for the information.

We designed a customized questionnaire based on the User Experience
Questionnaire(UEQ) (Schrepp and Thomaschewski, 2019) as few standard
questionnaires target robotic interfaces aimed at communicating knowledge. The
customized questionnaire contains four dimensions: usefulness,
comprehensibility, perspicuity, and clarity. The questionnaire can be found in
the supplementary materials.

Interviews were conducted after participants filled out the questionnaires. We
discussed the overall impression of the interface, the effects of the design,
obstacles for the task, and their opinions on using screen-based interaction
for situational knowledge exchange with the robot. The recordings of the
interviews are transcribed for further analysis.

###  5.4. Results

Figure 13. Questionnaire results from our Wizard-of-Oz testing. Scores range
from -3 to 3, with positive numbers indicating a positive rating.

Boxplot of the questionnaire responses in Usefulness, comprehensibility,
perspicuity, and clarity. Scores range from -3 to 3, with positive numbers
indicating a positive rating. Mean scores are all above 1, while the mean
score for Clarity is above 2. Scores for comprehensibility and perspicuity is
slightly lower with smaller standard deviation.

As seen in Figure 13, participants gave an overall positive rating for all
four aspects of the interface. This indicates that the patterns were able to
support us in designing effective interfaces to communicate situational
knowledge. Participants gave high ratings for the clarity of our interface
(questions 13-16, mean=1.58, sd=1.69). Many consider this interface to be
useful (question 1, mean = 1.90, sd = .88) and can provide help when using the
robot (question 2, mean = 2.00, sd = .94). Participants rate our interface as
easy to learn (question 6, mean = 2.10, sd = .99).

However, scores for comprehensibility and perspicuity is lower. Of note was
that some participants gave lower scores for helpfulness in understanding the
robot (question 7, mean=.70, sd=1.77). Some also think that the information
presented can be confusing (question 12, mean=.80, sd=1.81). For some
participants, it can be hard to find the information they want (question 11,
mean=.80, sd=1.23).

The reasoning for the scores was discussed in the interviews that followed. In
total, we recorded 122 minutes of interview audio. We analyzed the interview
results while focusing on the successful and unsuccessful usage of patterns.
We further discuss design recommendations generated from the analysis and our
work’s limitations in the following sections.

###  5.5. Design Recommendations

Below, we highlight themes found in the interview data, and correlate design
recommendations for the patterns in our library.

##### 1) Use user-friendly language when presenting KG data.

All participants agreed that the classification of robotic knowledge into
objects, users, and environment is intuitive. On the other hand, P10 felt
uncertain what type of information to look for, and P3 proposed that the
naming of knowledge types can be customized.

Therefore, we recommend that when using the Classification Tree pattern,
naming the groups according to users’ habits and customizing name by users can
be considered. Moreover, it is advisable to translate robot ontology terms to
more user-friendly names before display.

##### 2) Avoid loops in the knowledge graph

Repetitive information can confuse the user. P3 indicated that it feels
confusing when entering a loop of often repeated information. They further
commented on how it interfered with information retrieval. P8 thought some
page jump is cyclic and repeated. This is due to the inherent cyclic structure
present in the KG. We recommend to remove cycles when applying the Map
pattern, and organize information in a hierarchical fashion using
Classification Tree.

##### 3) Make it easier to look for information by relationships

Four of the ten participants said the interface hierarchy was too deep. Three
of them went back to the ontology list and looked through all nodes of a
specific ontology type without following the relationships in the KG.
Participants who found the information by following relationships felt it
easier to complete the task than those who did not. Thus we recommend to use
Entity and Relevant Information and enable the users to preview the
information.

##### 4) Use Classification Tree to reduce information overload.

Users tend to reduce the complexity of information. P3 thought that the
interface has a large amount of information and needed to spend more time to
explore; P2 suggested to present less information at the beginning. P5
suggested that the information presented by the interface should be gradually
increased according to the user’s need.

We have two design recommendations. First, Assorted Entities pattern is not
applicable with a large amount of information at the same level. We recommend
combine Assorted Entities to simplify information by common attributes or
concepts. Second, using Classification Tree to present information from a
large scale to a small scale is also recommended.

##  6\. Discussion and future work

In this work, we described our process of developing a pattern library for
designing interfaces to communicate the situational knowledge in service
robots. Results from prototyping and analysis focused on three main findings:
the use of design patterns, the challenges of presenting a knowledge graph to
non-expert users, and human-robot interaction with a KG interface. These
results offer many future directions for improving the pattern library and
designing better interactions for human-robot knowledge exchange.

###  6.1. Design patterns

During prototyping and testing, the pattern library was shown to aid our
design and development process effectively. The patterns can be applied to
real-world data, and interfaces designed using the patterns can easily use
knowledge graph queries to populate its content. Our design session
demonstrated that the patterns enable designers to produce interface designs
without knowing the content of the robotic knowledge graph.

Participants also responded favorably to the interface, highlighting its
clarity in presenting robotic knowledge. Interviews with them provided us with
insights on how to apply the patterns better.

###  6.2. Challenges of designing a KG interface

Our testing also highlighted challenges that surface when designing a KG
interface for non-expert users. We identified several challenges that
originate from the knowledge graph structure, its content, and its size.

Participants often experienced confusion when loops occur in the KG structure.
The patterns we discovered typically use categorization and trees to organize
semantic information, which does not contain loops. This warrants further
investigation into how to support efficient exploration of the KG while
maintaining a hierarchical abstraction.

The naming of knowledge in the KG presents another challenge when displayed to
non-expert users. The ontology of a KG is typically defined to facilitate
effective programming without taking understandability into account. Work is
being done to define an ontology that is understandable for non-expert
users(Diprose et al., 2012a). We believe that this is a necessary foundation
that enables better human-robot knowledge exchange.

The large amount of knowledge to be communicated poses another challenge. We
have discussed how to apply hierarchical patterns to reduce clutter, but
designers must also consider the increased depth of the interface.

###  6.3. Designing better interactions to communicate situational knowledge

Participants suggested various ways to incorporate multimodal interaction when
using our interface. Some noted that using an interface is good when the robot
is showing a large amount of information. Meanwhile, many participants also
noted that it is more natural to use simple voice commands for querying a
smaller amount of knowledge. This points toward a future research direction of
using multimodal interaction (e.g., voice, gesture, and gaze) combined with a
KG interface to enable better communication of situational knowledge.

###  6.4. Limitations and Future Work

The pattern library that we developed relied on our observations and analysis
of twelve participants’ knowledge presentation behavior under nine human-robot
interaction scenarios. While we carefully mapped out these scenarios to cover
as many potential cases as possible and recruited participants from diverse
backgrounds, additional scenarios and testing may lead to the discovery of
more patterns.

Moreover, the datasets we used are compiled specifically for the scenarios,
which may have overlooked many complexities in real-world robot knowledge
graphs. Future work may include building a public, real-world KG dataset. This
will enable comparing the performance of patterns and interfaces, as well as
promote further research into human-understandable robot knowledge ontology.

In the prototyping session, researchers with less practical knowledge in user
interface design were tasked with designing the interfaces. While the
participants gave positive feedback, results may be different had professional
robot interaction and interface designers be employed to conduct the
experiment. However, we hypothesize that this would likely improve participant
feedback, which does not change our conclusions.

Finally, the pattern library we developed demonstrated its ability to apply to
various knowledge communication scenarios. This warrants further research into
developing authoring environments that support designers to design and
prototype robot knowledge interfaces quickly.

##  7\. Conclusion

This paper proposes a pattern library that shows how end-users envisioned
service robots to organize and visually represent situational knowledge. We
then prototyped a service robot based on the patterns and used Wizard-of-Oz
testing to generate a series of design recommendations for knowledge-based
interface design in robots. Future work would include using the patterns to
create more robot interfaces and testing the design in more scenarios. We hope
our work can inspire more researchers and interface designers to explore
diverse approaches in situational knowledge exchange between humans and
robots.

## References

  * (1)
  * Alexander (1977) Christopher Alexander. 1977\.  _A pattern language: towns, buildings, construction_.  Oxford university press.
  * Alexander (1979) Christopher Alexander. 1979\.  _The Timeless Way of Building_.  Oxford University Press, New York, NY.
  * Alexander and Judy (1988) Patricia A. Alexander and Judith E. Judy. 1988.  The Interaction of Domain-Specific and Strategic Knowledge in Academic Performance.  _Review of Educational Research_ 58, 4 (1988), 375–404.  https://doi.org/10.3102/00346543058004375 arXiv:https://doi.org/10.3102/00346543058004375
  * Azevedo et al. (2019) Helio Azevedo, José Pedro R. Belo, and Roseli A. F. Romero. 2019\.  Using Ontology as a Strategy for Modeling the Interface Between the Cognitive and Robotic Systems.  _Journal of Intelligent & Robotic Systems_ (Aug. 2019).  https://doi.org/10.1007/s10846-019-01076-0
  * Barba et al. (2020) Evan Barba, Anthony Lioon, Christopher Miller, and Yasir Majeed Khan. 2020.  Tele-robotic Interface Design in Context: A Case for Recursive Design. In _Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems_. 1–8.
  * Bastianelli et al. (2013) Emanuele Bastianelli, Domenico Bloisi, Roberto Capobianco, Guglielmo Gemignani, Luca Iocchi, and Daniele Nardi. 2013.  Knowledge representation for robots through human-robot interaction.  _arXiv preprint arXiv:1307.7351_ (2013).
  * Bruno et al. (2019) Barbara Bruno, Carmine Tommaso Recchiuto, Irena Papadopoulos, Alessandro Saffiotti, Christina Koulouglioti, Roberto Menicatti, Fulvio Mastrogiovanni, Renato Zaccaria, and Antonio Sgorbissa. 2019\.  Knowledge Representation for Culturally Competent Personal Robots: Requirements, Design Principles, Implementation, and Assessment.  _International Journal of Social Robotics_ 11, 3 (June 2019), 515–538.  https://doi.org/10.1007/s12369-019-00519-w
  * Carpenter and Zachary (2017) Taylor J Carpenter and Wayne W Zachary. 2017.  Using context and robot-human communication to resolve unexpected situational conflicts. In _2017 IEEE Conference on Cognitive and Computational Aspects of Situation Management (CogSIMA)_. IEEE, 1–7.
  * Chang et al. (2020) Doo Soo Chang, Gun Hee Cho, and Yong Suk Choi. 2020\.  Ontology-based knowledge model for human-robot interactive services. In _Proceedings of the 35th Annual ACM Symposium on Applied Computing_. 2029–2038.
  * Chapman et al. (2017) Mimi V Chapman, Shiyou Wu, and Meihua Zhu. 2017.  What is a picture worth? A primer for coding and interpreting photographic data.  _Qualitative Social Work_ 16, 6 (2017), 810–824.
  * Chen et al. (2007) Jessie YC Chen, Ellen C Haas, and Michael J Barnes. 2007\.  Human performance issues and user interface design for teleoperated robots.  _IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews)_ 37, 6 (2007), 1231–1245.
  * de Jong and Ferguson-Hessler (1996) Ton de Jong and Monica G.M. Ferguson-Hessler. 1996\.  Types and qualities of knowledge.  _Educational Psychologist_ 31, 2 (1996), 105–113.  https://doi.org/10.1207/s15326985ep3102_2 arXiv:https://doi.org/10.1207/s15326985ep3102_2
  * Diprose et al. (2012b) James P Diprose, Beryl Plimmer, Bruce A MacDonald, John Hosking, et al. 2012b.  How people naturally describe robot behaviour.  (2012).
  * Diprose et al. (2012a) J P Diprose, B Plimmer, B A MacDonald, and J G Hosking. 2012a.  How People Naturally Describe Robot Behaviour.  _New Zealand._ (2012), 9\.
  * Fang et al. (2015) Rui Fang, Malcolm Doering, and Joyce Y Chai. 2015\.  Embodied collaborative referring expression generation in situated human-robot interaction. In _Proceedings of the Tenth Annual ACM/IEEE International Conference on Human-Robot Interaction_. 271–278.
  * Hogan et al. (2020) Aidan Hogan, Eva Blomqvist, Michael Cochez, Claudia d’Amato, Gerard de Melo, Claudio Gutierrez, José Emilio Labra Gayo, Sabrina Kirrane, Sebastian Neumaier, Axel Polleres, et al. 2020\.  Knowledge graphs.  _arXiv preprint arXiv:2003.02320_ (2020).
  * Il Hong Suh et al. (2007) Il Hong Suh, Gi Hyun Lim, Wonil Hwang, Hyowon Suh, Jung-Hwa Choi, and Young-Tack Park. 2007.  Ontology-based multi-layered robot knowledge framework (OMRKF) for robot intelligence. In _2007 IEEE/RSJ International Conference on Intelligent Robots and Systems_. IEEE, San Diego, CA, USA, 429–436.  https://doi.org/10.1109/IROS.2007.4399082
  * Javia and Cimiano (2016) Binal Javia and Philipp Cimiano. 2016.  A knowledge-based architecture supporting declarative action representation for manipulation of everyday objects. In _Proceedings of the 3rd Workshop on Model-Driven Robot Software Engineering_. 40–46.
  * Jeon et al. ([n.d.]) Hwawoo Jeon, Kyon-Mo Yang, Sungkee Park, Jongsuk Choi, and Yoonseob Lim. [n.d.].  An Ontology-based Home Care Service Robot for Persons with Dementia.  ([n. d.]), 6.
  * Jiang (2020) Meng Jiang. 2020\.  Improving situational awareness with collective artificial intelligence over knowledge graphs. In _Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications II_ , Vol. 11413. International Society for Optics and Photonics, 114130J.
  * Jokinen et al. (2019) Kristiina Jokinen, Satoshi Nishimura, Kentaro Watanabe, and Takuichi Nishimura. 2019.  Human-robot dialogues for explaining activities. In _9th International Workshop on Spoken Dialogue System Technology_. Springer, 239–251.
  * Kahn et al. (2008a) Peter H Kahn, Nathan G Freier, Takayuki Kanda, Hiroshi Ishiguro, Jolina H Ruckert, Rachel L Severson, and Shaun K Kane. 2008a.  Design patterns for sociality in human-robot interaction. In _Proceedings of the 3rd ACM/IEEE international conference on Human robot interaction_. 97–104.
  * Kahn et al. (2008b) Peter H. Kahn, Nathan G. Freier, Takayuki Kanda, Hiroshi Ishiguro, Jolina H. Ruckert, Rachel L. Severson, and Shaun K. Kane. 2008b.  Design patterns for sociality in human-robot interaction. In _Proceedings of the 3rd ACM/IEEE international conference on Human robot interaction_ _(HRI ’08)_. Association for Computing Machinery, New York, NY, USA, 97–104.  https://doi.org/10.1145/1349822.1349836
  * Kattepur (2019) Ajay Kattepur. 2019\.  RoboPlanner: autonomous robotic action planning via knowledge graph queries. In _Proceedings of the 34th ACM/SIGAPP Symposium on Applied Computing_. 953–956.
  * Kennington and Shukla (2017) Casey Kennington and Aprajita Shukla. 2017.  A Graphical Digital Personal Assistant that Grounds and Learns Autonomously. In _Proceedings of the 5th International Conference on Human Agent Interaction_. 353–357.
  * Klakegg et al. (2017) Simon Klakegg, Niels van Berkel, Aku Visuri, Hanna-Leena Huttunen, Simo Hosio, Chu Luo, Jorge Goncalves, and Denzil Ferreira. 2017\.  Designing a context-aware assistive infrastructure for elderly care. In _Proceedings of the 2017 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2017 ACM International Symposium on Wearable Computers_. 563–568.
  * Konecki (2011) Krzysztof Tomasz Konecki. 2011\.  Visual grounded theory: A methodological outline and examples from empirical work.  _Revija za sociologiju_ 41, 2 (2011), 131–160.
  * Kruijff et al. (2007) Geert-Jan M. Kruijff, Hendrik Zender, Patric Jensfelt, and Henrik I. Christensen. 2007.  Situated Dialogue and Spatial Organization: What, Where… and Why?  _International Journal of Advanced Robotic Systems_ 4, 1 (March 2007), 16.  https://doi.org/10.5772/5701
  * Laird et al. (2017) John E. Laird, Christian Lebiere, and Paul S. Rosenbloom. 2017\.  A Standard Model of the Mind: Toward a Common Computational Framework across Artificial Intelligence, Cognitive Science, Neuroscience, and Robotics.  _AI Magazine_ 38, 4 (Dec. 2017), 13–26.  https://doi.org/10.1609/aimag.v38i4.2744
  * Lemaignan et al. (2017a) Séverin Lemaignan, Mathieu Warnier, E Akin Sisbot, Aurélie Clodic, and Rachid Alami. 2017a.  Artificial cognition for social human–robot interaction: An implementation.  _Artificial Intelligence_ 247 (2017), 45–69.
  * Lemaignan et al. (2017b) Séverin Lemaignan, Mathieu Warnier, E. Akin Sisbot, Aurélie Clodic, and Rachid Alami. 2017b.  Artificial cognition for social human–robot interaction: An implementation.  _Artificial Intelligence_ 247 (June 2017), 45–69.  https://doi.org/10.1016/j.artint.2016.07.002
  * Leonardi et al. (2019) Nicola Leonardi, Marco Manca, Fabio Paternò, and Carmen Santoro. 2019.  Trigger-action programming for personalising humanoid robot behaviour. In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_. 1–13.
  * Lim et al. (2011) Gi Hyun Lim, Il Hong Suh, and Hyowon Suh. 2011.  Ontology-Based Unified Robot Knowledge for Service Robots in Indoor Environments.  _IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans_ 41, 3 (May 2011), 492–509.  https://doi.org/10.1109/TSMCA.2010.2076404
  * Liu et al. (2010) Changsong Liu, Jacob Walker, and Joyce Y Chai. 2010\.  Ambiguities in spatial language understanding in situated human robot dialogue. In _2010 AAAI Fall Symposium Series_.
  * Liu et al. (2018) Hangxin Liu, Yaofang Zhang, Wenwen Si, Xu Xie, Yixin Zhu, and Song-Chun Zhu. 2018\.  Interactive robot knowledge patching using augmented reality. In _2018 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 1947–1954.
  * Mahieu et al. (2019) Christof Mahieu, Femke Ongenae, Femke De Backere, Pieter Bonte, Filip De Turck, and Pieter Simoens. 2019\.  Semantics-based platform for context-aware and personalized robot interaction in the internet of robotic things.  _Journal of Systems and Software_ 149 (March 2019), 138–157.  https://doi.org/10.1016/j.jss.2018.11.022
  * Martin and Turner (1986) Patricia Yancey Martin and Barry A Turner. 1986.  Grounded theory and organizational research.  _The journal of applied behavioral science_ 22, 2 (1986), 141–157.
  * Marvel et al. (2020) Jeremy A Marvel, Shelly Bagchi, Megan Zimmerman, and Brian Antonishek. 2020.  Towards Effective Interface Designs for Collaborative HRI in Manufacturing: Metrics and Measures.  _ACM Transactions on Human-Robot Interaction (THRI)_ 9, 4 (2020), 1–55.
  * Mioch et al. (2014) Tina Mioch, Wietse Ledegang, Rosie Paulissen, Mark A Neerincx, and Jurriaan van Diggelen. 2014\.  Interaction design patterns for coherent and re-usable shape specifications of human-robot collaboration. In _Proceedings of the 2014 ACM SIGCHI symposium on Engineering interactive computing systems_. 75–83.
  * Nieto-Granda et al. (2010) Carlos Nieto-Granda, John G Rogers, Alexander JB Trevor, and Henrik I Christensen. 2010.  Semantic map partitioning in indoor environments using regional analysis. In _2010 IEEE/RSJ International Conference on Intelligent Robots and Systems_. IEEE, 1451–1456.
  * Oguz et al. (2019) Ozgur S Oguz, Wolfgang Rampeltshammer, Sebastian Paillan, and Dirk Wollherr. 2019.  An Ontology for Human-Human Interactions and Learning Interaction Behavior Policies.  _ACM Transactions on Human-Robot Interaction (THRI)_ 8, 3 (2019), 1–26.
  * Olivares-Alarcos et al. (2019) Alberto Olivares-Alarcos, Daniel Beßler, Alaa Khamis, Paulo Goncalves, Maki K. Habib, Julita Bermejo-Alonso, Marcos Barreto, Mohammed Diab, Jan Rosell, João Quintas, Joanna Olszewska, Hirenkumar Nakawala, Edison Pignaton, Amelie Gyrard, Stefano Borgo, Guillem Alenyà, Michael Beetz, and Howard Li. 2019.  A review and comparison of ontology-based approaches to robot autonomy.  _The Knowledge Engineering Review_ 34 (2019), e29.  https://doi.org/10.1017/S0269888919000237
  * Oliveira et al. (2018) Raquel Oliveira, Patrícia Arriaga, Patrícia Alves-Oliveira, Filipa Correia, Sofia Petisca, and Ana Paiva. 2018\.  Friends or foes? Socioemotional support and gaze behaviors in mixed groups of humans and robots. In _Proceedings of the 2018 ACM/IEEE international conference on human-robot interaction_. 279–288.
  * Padilla et al. (2017) Stefano Padilla, Thomas S. Methven, David A. Robb, and Mike J. Chantler. 2017.  Understanding Concept Maps: A Closer Look at How People Organise Ideas. In _Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems_. ACM, Denver Colorado USA, 815–827.  https://doi.org/10.1145/3025453.3025977
  * Paxton et al. (2018) Chris Paxton, Felix Jonathan, Andrew Hundt, Bilge Mutlu, and Gregory D Hager. 2018.  Evaluating methods for end-user creation of robot task plans. In _2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 6086–6092.
  * Pea (1987) Roy D Pea. 1987\.  User centered system design: new perspectives on human-computer interaction.  (1987).
  * Pronobis and Jensfelt (2012) Andrzej Pronobis and Patric Jensfelt. 2012.  Large-scale semantic mapping and reasoning with heterogeneous modalities. In _2012 IEEE international conference on robotics and automation_. IEEE, 3515–3522.
  * Riley et al. (2010) Jennifer M Riley, Laura D Strater, Sheryl L Chappell, Erik S Connors, and Mica R Endsley. 2010\.  Situation awareness in human-robot interaction: Challenges and user interface requirements.  _Human-Robot Interactions in Future Military Operations_ (2010), 171–192.
  * Sauppé and Mutlu (2014) Allison Sauppé and Bilge Mutlu. 2014.  Design patterns for exploring and prototyping human-robot interactions. In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_. 1439–1448.
  * Schrepp and Thomaschewski (2019) Martin Schrepp and Jörg Thomaschewski. 2019.  Eine modulare Erweiterung des User Experience Questionnaire.  https://doi.org/10.18420/muc2019-up-0108
  * Seffah (2010) Ahmed Seffah. 2010\.  The evolution of design patterns in HCI: from pattern languages to pattern-oriented design. In _Proceedings of the 1st International Workshop on Pattern-Driven Engineering of Interactive Computing Systems_. 4–9.
  * Sisbot et al. (2011) E Akin Sisbot, Raquel Ros, and Rachid Alami. 2011.  Situation assessment for human-robot interactive object manipulation. In _2011 RO-MAN_. IEEE, 15–20.
  * Tenorth and Beetz (2013) Moritz Tenorth and Michael Beetz. 2013.  KnowRob: A knowledge processing infrastructure for cognition-enabled robots.  _The International Journal of Robotics Research_ 32, 5 (April 2013), 566–590.  https://doi.org/10.1177/0278364913481635
  * Tiddi et al. (2017) Ilaria Tiddi, Emanuele Bastianelli, Gianluca Bardaro, Mathieu d’Aquin, and Enrico Motta. 2017\.  An ontology-based approach to improve the accessibility of ROS-based robotic systems. In _Proceedings of the Knowledge Capture Conference_. 1–8.
  * Tiddi et al. (2018) Ilaria Tiddi, Emanuele Bastianelli, Gianluca Bardaro, and Enrico Motta. 2018.  A User-friendly Interface to Control ROS Robotic Platforms.. In _International Semantic Web Conference (P &D/Industry/BlueSky)_.
  * Topp (2017) Elin Anna Topp. 2017\.  Interaction patterns in human augmented mapping.  _Advanced Robotics_ 31, 5 (2017), 258–267.
  * Tsiakas et al. (2017) Konstantinos Tsiakas, Michalis Papakostas, Michail Theofanidis, Morris Bell, Rada Mihalcea, Shouyi Wang, Mihai Burzo, and Fillia Makedon. 2017.  An interactive multisensing framework for personalized human robot collaboration and assistive training using reinforcement learning. In _Proceedings of the 10th International Conference on PErvasive Technologies Related to Assistive Environments_. 423–427.
  * Zachary et al. (2015) Wayne Zachary, Matthew Johnson, R Hoffman, Travis Thomas, Andrew Rosoff, and Thomas Santarelli. 2015\.  A context-based approach to robot-human interaction.  _Procedia Manufacturing_ 3 (2015), 1052–1059.
  * Zachary et al. (2013) W Zachary, A Rosoff, L Miller, S Read, K Laskey, I Emmons, and P Costa. 2013\.  Context as cognitive process. In _Proc. of_.


# Visual Story Generation Based on Emotion and Keywords

Yuetian Chen, Ruohua Li, Bowen Shi, Peiru Liu, and Mei Si
Rensselaer Polytechnic Institute
110 8th street, Troy, New York 12180
{cheny63, lir11, shib5, liup5<EMAIL_ADDRESS>
###### Abstract

Automated visual story generation aims to produce stories with corresponding
illustrations that exhibit coherence, progression, and adherence to
characters’ emotional development. This work proposes a story generation
pipeline to co-create visual stories with the users. The pipeline allows the
user to control events and emotions on the generated content. The pipeline
includes two parts: narrative and image generation. For narrative generation,
the system generates the next sentence using user-specified keywords and
emotion labels. For image generation, diffusion models are used to create a
visually appealing image corresponding to each generated sentence. Further,
object recognition is applied to the generated images to allow objects in
these images to be mentioned in future story development.

## Introduction

Storytelling has been an important way of communication, entertainment, and
even making sense of the world. Stories can be told with just text or using
the visual median. The use of visualizations can make stories more expressive
and engaging. In this work, we propose a visual story co-creation system that
leverages the power of large language models and diffusion models to create
short visual stories with a user. 111GitHub repository:
https://github.com/Stry233/Visual-Story-Generation-Based-on-Emotional-and-
Keyword-Scheme.

The example short stories we have in mind are from the ROCStories, a popular
dataset for commonsense reasoning and narrative understanding (?) named Story
Cloze Test. Each story is composed of five sentences and follows a character
through a series of events to an ending event or situation. When the data were
collected, the crowdsource workers were instructed that “the story should read
like a coherent story, with a specific beginning and ending, where something
happens in between” (?). These instructions make them good examples of
meaningful short stories. While the stories in the dataset only have text, our
proposed system aims to help people create similar stories with visualization.

To co-create visual stories, the proposed system aims to provide plausible
suggestions for story development while allowing users to override using their
own suggestions. Figure 1 shows the overall pipeline of the system. The system
is iterative in nature. Based on the existing partial story, the system can
suggest the characters’ emotions and keywords included in the next sentence,
such as an object, a location, etc. The user can accept them as defaults or
provide their own. The system will then generate the sentence below based on
the input parameters and the previously generated story. The user will then be
presented with several visualizations for the new sentence and must choose one
to proceed. Finally, the generated images will be subjected to object
recognition so that the system can extract additional keywords and recommend
their use in future storytelling.

Figure 1: Generation Pipeline.

## Related Work

Automatic story generation has long been a challenging task in natural
language processing, and such efforts date back to the 1970s (?). Earlier
attempts, such as symbolic planning systems (?), utilize planning techniques
to create plausible and coherent plots for stories. In addition, case-based or
analogical modeling systems generate new narratives based on adapted existing
stories (?). However, while these approaches can create stories with
impressive coherence and consistency, they often require extensive knowledge
engineering and restrict to a limited domain. To address the issue, large
crowd-sourced corpora of stories are used to help generate stories (?; ?).
Modern approaches based on neural language models have been explored to
generate plot-driven stories (?). Moreover, (?) incorporates commonsense
knowledge into the generator using a novel memory mechanism and utilizes
adversarial training to improve the diversity and originality of essay
generation. (?) leverages reward strategies during the generation to guide the
language model toward generating a coherent story. (?) proposes a character-
centric story-generation model that can produce stories consistent with
characters’ profiles.

In contrast to previous work, we propose an interactive visual story
generation pipeline in which the user can specify keywords and emotions that
will appear in the next sentence. The system generates the sentence and
associated images for the user based on this information.

## Visual Story Co-creation Pipeline

This section describes our approach to story generation pipelines and their
interactive workflow, as shown in Figure 1. The pipeline is divided into two
sections: next-sentence generation and image generation. The next sentence
generation is in charge of producing a plausible story based on specified
keywords and emotions, whereas image generation is in charge of producing
visualization for the generated story. We can mine the generated images for
additional keywords to include in future story development when it operates
interactively.

## Next Sentence Generation

To begin the co-creation process, the pipeline must be fed with the first
sentence of the story. It will then suggest emotions for the characters as
well as keywords for the next sentence, inviting the user to edit them. This
process is repeated iteratively by the pipeline to build the entire story.

We trained a next-sentence emotion prediction model using the Story
Commonsense dataset (?). Here, the existing story text was used as input,
while the emotion labels from the next sentence were used as labels. We fine-
tuned a BERT${}_{\textsc{BASE}}$ model (?), hereinafter noted as
BERT${}_{\textsc{Next Emo}}$, for this task. We provide the objects recognized
from the generated images as the default for keyword suggestions.

To enable the system to generate a sentence with the suggested keywords and
emotions, we fine-tuned a pre-trained T5${}_{\textsc{BASE}}$ (?) model using
the ROCStories dataset(?). The T5 model was pre-trained using the CommonGen
dataset (?), which enabled it to generate a sentence with a set of given
keywords(hereinafter denoted as T5${}_{\textsc{BASE Fine-tuned CommonGen}}$).
Fine-tuning it with the ROCStories dataset enabled the T5 model to further
learn about the story writing styles in ROCStories.

### Keywords Extraction

For training the model and evaluation, we extract keywords from the original
five-sentence stories. We used the SceneGraphParser() from (?) to parse
sentences (in natural language) into scene graphs. The entities in the scene
graphs become the keywords. For example, for the following sentence:

$\textsc{S}_{1}$: I brought the movie home and watched the whole thing.

We are expected to generate the following result:

’I, the movie, the whole thing’

### Categorical Emotion Inference

We use Plutchik’s Wheel of Emotions as our basic model of characters’
emotions. The wheel includes eight emotions in pairs: joy/sadness,
trust/disgust, fear/anger, and surprise/anticipation (?) as shown in Figure 2.
We define emotion entries in a sentence as a real-valued, low-dimensional
vector $\vec{\mathbf{C}}$.

Figure 2: Plutchik basic emotions

$\displaystyle\forall\vec{C}\in\mathbf{D},\vec{C}$
$\displaystyle=\begin{bmatrix}e_{1}\\\ e_{2}\\\ \vdots\\\
e_{8}\end{bmatrix},e\in[0,1]$ (1)

Given the context of a story, i.e., all previous sentences
$\vec{X}=\\{x_{1},x_{2},x_{3},\dots,x_{m}\\}$, the goal is to predict the
emotions $\vec{C_{i}}$ that will appear in the next sentence.

For example, for the following input sequence of sentence $\textsc{S}_{1}$
that serves as context, the model may predict the emotions in the next
sentence to be joy, anticipation, and trust.

$\textsc{S}_{1}$: He was hoping this year to be tall enough for the coaster.

For preparing the training data, we obtained 17,910 pairs of story context and
next-sentence emotions from the Story Commonsense dataset. The dataset was
divided into training and validation sets in the ratio of 6:4. We then fine-
tuned the BERT${}_{\textsc{Next Emo}}$ model using the task of multi-label
classification. The story context is the input, and the next-sentence emotions
are the output. The maximum input length of the tokenizer was set to 120. The
batch size was set to 16, and the learning rate was set to $6e-6$. We ran 16
epochs for training, which took about 45 minutes on a Tesla P100-PCIE-16GB
GPU. The model achieved a Macro ROC-AUC score of 0.69 on prediction. The
prediction results are not perfect. However, they are only used as
suggestions, and the user can always overwrite them.

### Next Sentence Generation with Prompts

Figure 3: Training and inference pipeline for Next Sentence Generation.

An overview of our story generation model is given in Figure 3. We constructed
our training data using the stories from both the ROCStories and Story
Commonsense datasets (denoted as $\mathbf{S_{total}}$ hereinafter).

For a sample story from ROCStories – $\vec{S}\in\mathbf{S_{total}}$, we define
it as a 5-dimensional vector $\vec{S}=\\{s_{1},\dots,s_{5}\\}$ where each
$s_{i}$ represents the corresponding sentence in the story.

For training our system to generate the next sentence, $s_{j+1}$ is fed into
the emotion classifier as well as the keyword extractor for identifying the
emotions and keywords in the next sentence. Based on story context and the
additional knowledge input, T5${}_{\textsc{BASE Fine-tuned CommonGen}}$ is
trained on a triad of data: context, keyword, and emotion labels.

In addition, we used the following prompting structure in both the training as
well as inference processes:

⬇

Generate next sentence based on following

<extra_id_0>KEYWORDS: [...]

<extra_id_0>CONTEXT: [...]

<extra_id_0>EMOTION: [...]

We adopt T5${}_{\textsc{BASE Fine-tuned CommonGen}}$ as the base language
model for the training process. The CommonGen dataset contains pairs of
sentences and keywords. The task is to generate a meaningful sentence that
contains all the given keywords. The T5${}_{\textsc{BASE Fine-tuned
CommonGen}}$ is a T5 model fine-tuned using this dataset. For example, given
the following input:

X1: <’Mike’, ’tree’, ’ground’, ’hole’>

$T5_{\textsc{Fine-tuned CommonGen}}$ may generate:

Y1: $\underset{\textsc{x}_{0}}{\text{[Mike]}}$ is digging a
$\underset{\textsc{x}_{3}}{\text{[hole]}}$ in the
$\underset{\textsc{x}_{2}}{\text{[ground]}}$ near a
$\underset{\textsc{x}_{1}}{\text{[tree]}}$.

By fine-tuning T5${}_{\textsc{BASE Fine-tuned CommonGen}}$ based on the
emotions and the keyword information as prompts, the model can more
effectively capture hidden information in the story and thus improve the
accuracy and variety of sentences. Compared to the baseline model fine-tuned
only with context information, we found that, due to the lack of a priori
knowledge of the link between emotions and events, the baseline model is
highly susceptible to confounding empathy problems arising from different
behavioral motivations. As shown in Table 1, the baseline model could not use
the context effectively and correctly without the prompts. It generated
repeated sentences. Further, it is unclear why the character – Mary – would
feel happy about her situation.

generated - prompt |  generated - no prompt
---|---
Mary had been feeling depressed lately. |  Mary had been feeling depressed lately.
she decided to go see a psychiatrist. |  She decided to go to a psychiatrist.
Psyched, her psychiatrist diagnosed her with depression and sent her to see. |  She was diagnosed with schizophrenia.
Medicant took her to get an antidepressant and prescribed her. |  She was very happy.
Thankfully it eventually made her feel better again. |  She was very happy.
Table 1: An example comparing the baseline model and model with emotion and
keywords prompting

In terms of implementation, we used the previous format to create a dataset
with 18,680 entries. In an 8:2 ratio, we divided it into a training set and a
validation set. The maximum lengths of the source and output text are set to
512 and 50, respectively. To control the diversity of the output content, we
use top-K search in the decoding process, with $k=3$, and set the repetition
penalty and length penalty to 2.6 and 1.0, respectively, for the same reason.
In the training process, we set the batch size and learning rate to 10 and
5e-4, respectively, and introduced the early stop mechanism to prevent
overfitting. We ran 16 epochs, and the training phase of the language model
took about 10 hours on a Tesla P100-PCIE-16GB platform.

## Image Generation

When image generation is added to text-based stories, it increases flexibility
and creates a sense of immersion in the generated story plot. As a result, in
addition to the text pipeline, we implement an image pipeline that generates
additional keywords based on images as suggestions, potentially pushing the
story forward in different directions. We separate the pipeline into two
parts: image generation and object detection. We experimented with both Stable
Diffusion (?) and Disco Diffusion as our primary method for image generation,
as they can produce fairly relevant figures with artistic quality. For object
detection, we chose YOLOv5 (?) and Faster R-CNN (?) as our models. The
generated keywords will serve as suggestions for users to select from, as well
as hints to guide the subsequent generation process.

### Image Generation

To create images for the generated sentences, we utilize Disco Diffusion, a
tool that combines both Diffusion, a mathematical process for removing noise
from an image, and CLIP (?), a model for labeling images. Image generation is
an iterative process in Disco Diffusion, in which CLIP evaluates the
intermediate image based on the text prompt and provides a guideline for
Diffusion to process at each step.

Disco Diffusion has been extensively researched as a means of generating
creative art. In practice, however, we found it difficult to create a good
image based solely on one line of a story. Although Disco Diffusion has a
portrait generator, the model finds it difficult to generate images that
describe actions in the format of ”someone is doing something.” To address
this, we include user-specific keywords in the text prompt, including the
artist’s name and background information. For example, we added ”Carl
Spitzweg” as the artist and ”country view” as the background information in
our example. Disco Diffusion can generate images that are consistent with the
artist’s artistic styles, such as shadow, color, and strokes, by including the
artist’s name in the text prompt. Furthermore, we fill in background
information that can add more relevant elements to the generated image. As
shown in Figure 4, the prompt ”Ray gathered his friends to tell them a funny
joke he heard” will only produce a picture of several people who appear to be
talking, whereas adding ”country view” to the text prompt generates an image
with explicit yet magnificent scenery in the background. Users can change the
artist’s name and background information based on their preferences.

(a) Prompt without adding “country view”

(b) Prompt adding “country view”

Figure 4: Prompt: “Ray gathered his friends to tell them a funny joke he
heard,” generated using Disco Diffusion

Once Disco Diffusion is able to generate images with reasonably appropriate
information based on the text prompt, we use it as a source of common sense
knowledge to suggest additional keywords for the sentence that follows. For
instance, consider the prompt, ”a boy is picking up shells on a beach.” This
generates an image of a boy with a bag picking up shells on a beach with an
ocean backdrop, as shown in Figure 5. While the image contains requested
elements such as ”boy,” ”shells,” and ”beach,” it also introduces new but
highly relevant elements such as ”bag” and ”ocean.” This is similar to how
when people read stories, they frequently fill in details based on their
imagination. The extra elements from the generated images are used to add an
”imagination” component to the story generation pipeline. Object detection
algorithms are used to detect potential keywords from generated images, which
are then presented to users as suggestions for creating the next part of the
story.

Figure 5: Prompt “a boy is picking up shells on a beach,” generated using
Disco Diffusion

#### Implementation

There are about 115 parameters in Disco Diffusion. We mainly focus on settings
that directly impact our output images.

Clip$\\_$guidance$\\_$scale: This variable tells the model how strongly CLIP
will follow the prompt. We set it to 5,000 as recommended.

Steps: Each step (or iteration) involves the model looking at a subset of
images called “cuts” and calculating the “direction” in which the images
should be guided so that the generated results are relevant to the prompt. We
set this to 250.

n$\\_$batches: We generate three images for each prompt. Users can choose
which one is ideal for that story.

Item | Number |  | Confidence
---
level
$\rhd$ horse | 5 |  0.729
$\rhd$ bird | 2 |  0.719
$\rhd$ person | 42 |  0.694
$\rhd$ handbag | 2 |  0.656
$\rhd$ … | … | …
Table 2:  The table includes items detected based on images generated through
Disco Diffusion with their corresponding number detected and confidence level.
Figure 6: Several generated stories based on the same background sentence. The
underlined words in orange indicate the keywords entered

### Key Object Detection

Figure 7: Story fragments generated by the proposed method with two prompt
configurations.

We explored using YOLOv5 and Faster R-CNN (?) for object detection. In
general, YOLOv5 outperforms Faster R-CNN in inference speed due to its smaller
model size. On the other hand, faster R-CNN with ResNet-50 FPN (?) backbone
can provide better performance in terms of mAP (i.e., mean average precision)
improvement. Both models are pre-trained with the COCO dataset (?). Although
these models are only trained on real-world images, they perform impressively
on the paintings generated by Disco Diffusion. We suspect this is due to the
fact that, as long as the artistic style isn’t too abstract, the objects in
the generated images share characteristics with real-world objects.

With the same prompt, Disco Diffusion can generate a batch of images. We
process all images created for object detection and summarize a corresponding
dictionary of the detected object with their confidence levels. As a result,
the user will have a better sense of potential objects associated with the
current story and will be more likely to include them in the text prompt.

#### Implementation

We use YOLOv5x because it produces the best mAP results among the YOLOv5
family. The COCO dataset is used to pre-train YOLOv5 and Faster R-CNN,
resulting in approximately 80 detectable objects. To improve the reliability
of the results, we set the Faster R-CNN threshold to 0.4. The final result is
sorted using the confidence score IOU (Intersection over Union).

Figure 8: Performance distribution of the baseline model and the prompt-
optimized model in 3,748 sets of experiments under different metrics The blue
box on the left side of each figure represents our method, and the orange on
the right side represents the baseline model.

## Example Outputs

Figure 6 demonstrates a story generation example using various keywords and
emotions using our pipeline. The pipeline can generate a reasonable story
based on the provided information by modifying the input prompts of the last
two sentences (i.e., the twist and ending parts of the story) in both
examples. This demonstrates both the capability of our pipeline and its
sensitivity to emotion and keywords. In addition, we demonstrate a full visual
storytelling report in Figure 7 by combining generated images provided by the
image generation pipeline.

## Evaluation and Discussion

To assess the efficacy of a controlled story generation framework based on
categorical emotion and keywords, we compared our approach to a baseline model
fine-tuned solely based on the existing story. We trained another BERT-based
model, hereinafter referred to as BERT${}_{\textsc{Cur Emo}}$, to generate
sensible emotion keywords as input. This model is a fine-tuned emotion
predictor based on the 11,129 emotion-sentence pairs in the Story Commonsense
dataset. This model, unlike the BERT${}_{\textsc{Next Emo}}$ mentioned in the
Categorical Emotion Inference section, aims to detect emotion information for
a given (current) sentence. The macro AUC-ROC score reached 0.78 during
training, leading us to believe that BERT${}_{\textsc{Cur Emo}}$ is adequate
for identifying and extracting emotions from sentences. To ensure that the
models can apply prior knowledge to the inference process of story generation,
both our pipeline and the baseline will use the same T5${}_{\textsc{BASE Fine-
tuned CommonGen}}$ and adapt the inputs to the prompt format used in the
corresponding fine-tuning phase.

In order to obtain the validity of the prompts we added to the input, we used
the average of BLEU (?) scores with n-grams ranging from 1 to 4 and BERT-
scores (?) as basic metrics in our evaluation. In addition, we use METEOR (?)
and SacreBLEU(?) as supplements to compensate for the lack of stemming and
synonym matching, as well as standard exact word matching, when comparing the
generated content with the ground truth.

We present the experimental results in Figure 8 and find that, compared to the
baseline model, which relies solely on story context for inference, our
pipeline shows consistent improvements in all metrics under the same model by
introducing emotions and keywords as prompts.

Story generation methods based on emotion and keywords can present
statistically higher upper bounds, regardless of how the metrics are
implemented or what is tested. As shown in Figure 8, for BLEU-average scores,
our approach improves the story generation task by an average of 62%. It also
reduces the number of samples with zero BERT scores significantly. The average
increase in BERT scores is also 6.7%, and the rate of score fluctuation is
reduced. This implies that our approach has the potential to improve many
aspects of story generation significantly.

## Future Work

We are interested in making several improvements in the future. We hope to
find a better prompt format to fully utilize the potential of language models
for the generation task. We hope that, with the assistance of Chain of Thought
Prompting (?), the model will be able to generate a story with more
sophisticated reasoning and causal relationship. The resulting story will then
feel more natural and logical. Another area that can be improved is the
emotion classifier. At the moment, the model for predicting emotions in the
following sentence can only achieve around 60% accuracy for each of Plutchik’s
emotion categories which can be further improved.

The image pipeline has the potential to improve as well. Disco Diffusion, in
particular, has difficulty producing images of creatures when configured
incorrectly. While we can generate reasonable images by incorporating artists
who frequently include corresponding objects and creatures in their paintings
as a result of CLIP’s recognition capabilities, the Diffusion model will
produce nonsense if we do not provide a ”reference” artist to paint.
Furthermore, because they are only pre-trained with the COCO dataset and real-
world objects, YOLOv5 and Faster R-CNN can detect a limited number of objects.
In the future, we can use YOLOv5 or Faster R-CNN to implement few-shot
learning to increase the number of detected objects (?).

## Conclusion

We present a novel pipeline for creating visual stories based on Plutchik’s
Wheel of Emotions model of emotional knowledge and keyword information. This
method assists visual narrative designers in creating and iterating stories in
a controlled manner. Our method also includes a complete narrative text-to-
image generation pipeline. Experiments show that stories generated by using
keywords as prompts have a higher level of grammatical and logical
consistency, as well as being more consistent with human writing habits. This
work, we believe, is an important step toward a more comprehensive and
functional language model-based storytelling tool.

## References

  * [Banerjee and Lavie 2005] Banerjee, S., and Lavie, A.  2005\.  METEOR: An automatic metric for MT evaluation with improved correlation with human judgments.  In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, 65–72.  Ann Arbor, Michigan: Association for Computational Linguistics.
  * [Devlin et al. 2018] Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K.  2018\.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805.
  * [Fan, Lewis, and Dauphin 2018] Fan, A.; Lewis, M.; and Dauphin, Y.  2018\.  Hierarchical neural story generation.  In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 889–898.  Melbourne, Australia: Association for Computational Linguistics.
  * [Gervás et al. 2005] Gervás, P.; Díaz-Agudo, B.; Peinado, F.; and Hervás, R.  2005\.  Story plot generation based on cbr.  In Knowl. Based Syst.
  * [Ibrahim et al. 2022] Ibrahim, B. I. E.; Eyharabide, V.; Le Page, V.; and Billiet, F.  2022\.  Few-shot object detection: Application to medieval musicological studies.  Journal of Imaging 8(2).
  * [Jocher et al. 2022] Jocher, G.; Chaurasia, A.; Stoken, A.; Borovec, J.; NanoCode012; Kwon, Y.; TaoXie; Fang, J.; imyhxy; Michael, K.; Lorna; V, A.; Montes, D.; Nadar, J.; Laughing; tkianai; yxNONG; Skalski, P.; Wang, Z.; Hogan, A.; Fati, C.; Mammana, L.; AlexWang1900; Patel, D.; Yiwei, D.; You, F.; Hajek, J.; Diaconu, L.; and Minh, M. T.  2022\.  ultralytics/yolov5: v6.1 - TensorRT, TensorFlow Edge TPU and OpenVINO Export and Inference.
  * [Li et al. 2013] Li, B.; Lee-Urban, S.; Johnston, G.; and Riedl, M.  2013\.  Story generation with crowdsourced plot graphs.  Proceedings of the 27th AAAI Conference on Artificial Intelligence, AAAI 2013 598–604.
  * [Lin and Och 2004] Lin, C.-Y., and Och, F. J.  2004\.  ORANGE: a method for evaluating automatic evaluation metrics for machine translation.  In COLING 2004: Proceedings of the 20th International Conference on Computational Linguistics, 501–507.  Geneva, Switzerland: COLING.
  * [Lin et al. 2014] Lin, T.-Y.; Maire, M.; Belongie, S.; Bourdev, L.; Girshick, R.; Hays, J.; Perona, P.; Ramanan, D.; Zitnick, C. L.; and Dollár, P.  2014\.  Microsoft coco: Common objects in context.
  * [Lin et al. 2016] Lin, T.-Y.; Dollár, P.; Girshick, R.; He, K.; Hariharan, B.; and Belongie, S.  2016\.  Feature pyramid networks for object detection.
  * [Lin et al. 2019] Lin, B. Y.; Zhou, W.; Shen, M.; Zhou, P.; Bhagavatula, C.; Choi, Y.; and Ren, X.  2019\.  Commongen: A constrained text generation challenge for generative commonsense reasoning.  arXiv preprint arXiv:1911.03705.
  * [Liu et al. 2020] Liu, D.; Li, J.; Yu, M.-H.; Huang, Z.; Liu, G.; Zhao, D.; and Yan, R.  2020\.  A character-centric neural model for automated story generation.  Proceedings of the AAAI Conference on Artificial Intelligence 34(02):1725–1732.
  * [Meehan 1977] Meehan, J. R.  1977\.  Tale-spin, an interactive program that writes stories.  In IJCAI.
  * [Mostafazadeh et al. 2016] Mostafazadeh, N.; Chambers, N.; He, X.; Parikh, D.; Batra, D.; Vanderwende, L.; Kohli, P.; and Allen, J.  2016\.  A corpus and evaluation framework for deeper understanding of commonsense stories.  arXiv preprint arXiv:1604.01696.
  * [Plutchik 1980] Plutchik, R.  1980\.  A general psychoevolutionary theory of emotion.  In Theories of emotion. Elsevier.  3–33.
  * [Post 2018] Post, M.  2018\.  A call for clarity in reporting BLEU scores.  In Proceedings of the Third Conference on Machine Translation: Research Papers, 186–191.  Belgium, Brussels: Association for Computational Linguistics.
  * [Radford et al. 2021] Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; Krueger, G.; and Sutskever, I.  2021\.  Learning transferable visual models from natural language supervision.
  * [Raffel et al. 2020] Raffel, C.; Shazeer, N.; Roberts, A.; Lee, K.; Narang, S.; Matena, M.; Zhou, Y.; Li, W.; Liu, P. J.; et al.  2020\.  Exploring the limits of transfer learning with a unified text-to-text transformer.  J. Mach. Learn. Res. 21(140):1–67.
  * [Rashkin et al. 2018] Rashkin, H.; Bosselut, A.; Sap, M.; Knight, K.; and Choi, Y.  2018\.  Modeling naive psychology of characters in simple commonsense stories.  In Gurevych, I., and Miyao, Y., eds., Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, 2289–2299.  Association for Computational Linguistics.
  * [Ren et al. 2015] Ren, S.; He, K.; Girshick, R.; and Sun, J.  2015\.  Faster r-cnn: Towards real-time object detection with region proposal networks.
  * [Riedl and Young 2010] Riedl, M. O., and Young, R. M.  2010\.  Narrative planning: Balancing plot and character.  Journal of Artificial Intelligence Research 39:217–268.
  * [Rombach et al. 2022] Rombach, R.; Blattmann, A.; Lorenz, D.; Esser, P.; and Ommer, B.  2022\.  High-resolution image synthesis with latent diffusion models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 10684–10695.
  * [Swanson and Gordon 2012] Swanson, R., and Gordon, A. S.  2012\.  Say anything: Using textual case-based reasoning to enable open-domain interactive storytelling.  ACM Trans. Interact. Intell. Syst. 2(3).
  * [Tambwekar et al. 2019] Tambwekar, P.; Dhuliawala, M.; Martin, L. J.; Mehta, A.; Harrison, B.; and Riedl, M. O.  2019\.  Controllable neural story plot generation via reward shaping.  In IJCAI.
  * [Wei et al. 2022] Wei, J.; Wang, X.; Schuurmans, D.; Bosma, M.; Ichter, B.; Xia, F.; Chi, E.; Le, Q.; and Zhou, D.  2022\.  Chain of thought prompting elicits reasoning in large language models.
  * [Wu et al. 2019] Wu, H.; Mao, J.; Zhang, Y.; Jiang, Y.; Li, L.; Sun, W.; and Ma, W.-Y.  2019\.  Unified visual-semantic embeddings: Bridging vision and language with structured meaning representations.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 6609–6618.
  * [Yang et al. 2019] Yang, P.; Li, L.; Luo, F.; Liu, T.; and Sun, X.  2019\.  Enhancing topic-to-essay generation with external commonsense knowledge.  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2002–2012.  Florence, Italy: Association for Computational Linguistics.
  * [Zhang et al. 2019] Zhang, T.; Kishore, V.; Wu, F.; Weinberger, K. Q.; and Artzi, Y.  2019\.  Bertscore: Evaluating text generation with bert.