# Story Generation from Visual Inputs: Techniques, Related Tasks, and
Challenges

Daniel A. P. Oliveira<EMAIL_ADDRESS>0000-0003-4413-6957 ,
Eugénio Ribeiro<EMAIL_ADDRESS>0000-0001-7147-8675 and  David
Martins de Matos<EMAIL_ADDRESS>0000-0001-8631-2870 INESC-
IDLisbonPortugal Instituto Superior Técnico, Universidade de
LisboaLisbonPortugal

(2024; 20 February 2007; 12 March 2009; 5 June 2009)

###### Abstract.

Creating engaging narratives from visual data is crucial for automated digital
media consumption, assistive technologies, and interactive entertainment. This
survey covers methodologies used in the generation of these narratives,
focusing on their principles, strengths, and limitations.

The survey also covers tasks related to automatic story generation, such as
image and video captioning, and visual question answering, as well as story
generation without visual inputs. These tasks share common challenges with
visual story generation and have served as inspiration for the techniques used
in the field. We analyze the main datasets and evaluation metrics, providing a
critical perspective on their limitations.

Visual Story Generation, Image Captioning, Visual Question Answering,
Storytelling

††copyright: acmlicensed††journalyear: 2024††doi: XXXXXXX.XXXXXXX††journal:
CSUR††journalvolume: 00††journalnumber: 0††article: 000††publicationmonth:
0††ccs: Computing methodologies Natural language generation††ccs: Computing
methodologies Video summarization††ccs: Computing methodologies Scene
understanding

##  1\. Introduction

Stories are fundamental to human experience, serving as a bridge between
imagination and reality (Hühn et al., 2014). They represent the art of telling
tales, whether real or imagined, and present a sequence of events that can
captivate, inform, and provoke thought. Storytelling has been an essential
part of human culture for millennia, serving as a means of communication,
entertainment, education, and moral guidance. The art of storytelling has
evolved over time, beginning with the practice of narrating stories through
spoken words, advancing to written literature, and, more recently,
transitioning to digital media formats. Stories have played a crucial role in
shaping societies, passing down knowledge and values from one generation to
another. Whether told around a campfire, written in books, or shared through
videos and interactive platforms, storytelling continues to captivate and
connect people, transcending time and technology (Abbott, 2008).

The field of automated story generation has gained attention, fueled by
advances in Natural Language Processing (NLP) (Jurafsky and Martin, 2009) and
computer vision (Szeliski, 2010). The goal is to create systems capable of
generating novel and engaging stories, similar to those produced by humans
(Gervas, 2009). With the increasing availability of images and videos, there
is a growing interest in generating stories based on this visual data (Krishna
et al., 2017). We refer to this task as Visual Story Generation (VSG).

###  1.1. Definition and challenges

VSG is the task of automatically creating coherent and engaging stories
grounded on visual inputs, such as images or videos. It extends beyond simple
image captioning or video description, requiring an understanding of complex
relationships, temporal progression, and implicit context within the visual
data. The generated narrative must, not only accurately represent the visual
content, but also encapsulate the essence of the underlying story, thereby
crafting an engaging experience for the audience. VSG serves as a robust
indicator of machine comprehension of visual content and poses several
challenges, some of which are:

  * •

Understanding visual content: The recognition of objects and scenes in the
visual data, as well as understanding the relationships between them is
crucial for generating coherent and meaningful narratives (Xu et al., 2015).

  * •

Temporal coherence: A story is inherently sequential and maintains a temporal
order. Therefore, generated narratives should respect time progression, even
if not in a linear manner (e.g., they may involve flashbacks, foreshadowing,
or other such mechanisms), and ensure that events are narrated in a logically
consistent manner.

  * •

Contextual understanding: Often, elements of a story can be implicit or
inferred from the context. Effective generation approaches must be capable of
understanding and incorporating implicit information in the narrative.

  * •

Engagement and creativity: One of the main goals of a story is to engage the
audience. Therefore, the generated narrative should not only be accurate and
coherent, but also creative and engaging.

  * •

Evaluation: Determining the effectiveness of a VSG system is a challenge in
itself. Metrics originally developed for tasks such as machine translation and
image captioning have been adapted for VSG. However, these metrics are not
appropriate to capture the creativity and engagement aspects of a story.

###  1.2. Background and Motivation

The motivations for VSG ] are numerous. For example, VSG can provide an
interactive component to media consumption. Users could potentially influence
the direction of the story by choosing different sequences of images or
videos, creating a unique, personalized narrative. This could be particularly
beneficial in areas like interactive gaming and movies, educational platforms,
or even in digital marketing, where personalized narratives can greatly
enhance user engagement and satisfaction. VSG can also be used in journalism:
it can be used to summarize large amounts of visual data, giving insights into
the underlying patterns and relationships, and assisting in content creation.

The growing interest in this area is also motivated by its potential to
contribute to the understanding of human cognition and creativity. By
developing models that emulate human-like storytelling abilities, researchers
can gain insights into the cognitive processes involved in story creation, as
well as explore the underlying mechanisms of creativity. These approaches can
further lead to novel applications, such as more engaging virtual experiences
(Fan et al., 2018; Peng et al., 2018).

These use cases represent only a fraction of the potential applications of
VSG. As research in this field progresses, we anticipate that more
applications will emerge, spanning an even broader range of domains.

###  1.3. Scope and structure of the survey

We start by defining the core elements of a story: they are essential for
understanding the structure and dynamics of a story and served as foundational
components for the development of the early story generation systems. Section
2 presents these elements and discusses their significance in storytelling. It
also clarifies the distinctions between story, narrative, and plot,
illustrating how a single story can be narrated in multiple ways to yield
diverse experiences.

We then explore the main datasets used in story generation, and the various
metrics used to assess their performance. We address the limitations that
current evaluation metrics face in such an open-ended task and discuss
potential directions for future research in this regard. Section 3 addresses
these issues.

Since they form the background for VSG approaches, we also explore the methods
for textual story generation: from the early rule-based systems to deep
learning models. We focus on deep learning methodologies, especially
transformer architectures and Large Language Models (LLMs) (Bolaños et al.,
2015; Ammanabrolu et al., 2019), due to their state-of-the-art performance.
Section 4 presents these approaches and emphasizes their contributions and
historical significance.

The connection of VSG with computer vision tasks such as Visual Question
Answering (VQA) and image/video captioning is also acknowledged, since they
share core challenges, such as visual understanding and narrative generation.
Thus, they have contributed with useful techniques to the field. Section 5
explores this convergence between computer vision and NLP.

Concerning VSG proper, we aim to present a balanced perspective on current
methodologies, emphasizing opportunities and challenges for future research.
The inclusion of a detailed analysis of deep learning methodologies with a
broader review of additional and earlier techniques ensures broad coverage of
the field. Section 6 presents a broad perspective on the state-of-the-art of
VSG.

Finally, Section 7 presents the key findings of this survey and proposes
directions for future research.

##  2\. The Elements of Stories

The essence of a story lies in its ability to connect a series of events and
experiences, often conveyed by a narrator to an audience. These can range from
personal intimate stories to larger epics that span cultures and time.

A story can be told in multiple ways using different narratives. These
narratives are more than mere descriptions or dramatic enactments. They
involve a process of narration where events are selected, arranged, and em-
plotted to give meaning to experiences. This structuring of events is what
forms the plot or the story’s synopsis.

The terms story, narrative, and plot are often used interchangeably in
everyday language, but they have distinct meanings. According to the Oxford
English Dictionary (Simpson, 2002) a story is the straightforward,
chronological sequence of events, encompassing characters, settings, and
actions. The story is the “what” – what happened, who it happened to, where
and when it happened. In contrast, the narrative shapes the form or the “how”,
adding context, interpretation, perspective, tone, and style of the
storytelling, and nuance to the same set of events. The plot is connected to
the narrative, referring to the sequence of events as they are presented to
the audience. Thus, a single story can be narrated in multiple ways, each
having a different plot and offering a distinct experience to the audience.

Narratives can be broadly categorized as linear or nonlinear (Lau and Chen,
2008), depending on their structure. Each type offers a different approach to
unfolding a story and has distinct implications for how the audience
experiences and interprets it.

Linear narratives follow a straightforward chronological progression, where
events unfold in a sequential and cause-effect manner. Linear narratives often
have a single protagonist or a group of characters whose actions and decisions
drive the plot forward. This structure is commonly used in traditional
storytelling formats. Linear narratives provide a sense of progression and
closure, making them easy to follow and understand.

Nonlinear narratives deviate from the traditional linear structure by
presenting events out of chronological order. They often involve flashbacks,
flash-forwards, or parallel storylines that disrupt the temporal sequence.
These narratives can create intrigue, suspense, and complexity by challenging
the audience’s expectations and requiring them to piece together the
fragmented storyline. The film “Cloud Atlas” (Mitchell, 2008) is an example.

Story elements – character, conflict, theme, setting, plot, and mode – are the
essential components that form the foundation of a story and are vital for
both crafting and analyzing stories. In the following subsections, we detail
each of these elements, discussing their significance in shaping the story.

###  2.1. Character

Characters are the entities through which the story unfolds (DiBattista,
2011). Their decisions and actions are pivotal in propelling the plot forward.
Characters are typically people or other beings, fictional or based on a real-
life person, and through their actions and speech, they often convey essential
motives and emotions.

The way an audience initially perceives a character significantly influences
their ongoing perception and emotional connection, such as empathy or
antipathy. This initial impression sets the stage for the audience’s
expectations regarding the future behavior of the character. Characters who
deviate unexpectedly from their established patterns of behavior can create
confusion or surprise, challenging the audience’s understanding and
expectations.

Narratives usually center around main characters or protagonists. They are the
focal points of the story, often encountering central conflicts, undergoing
significant personal growth, or acquiring new knowledge. Many stories also
have antagonists, adding complexity and tension to the storyline. In
traditional stories, the protagonist often takes the form of a hero, a
sympathetic figure battling for morally good causes. Conversely, the
antagonist may be portrayed as a villain, acting against these good causes or
even perpetrating evil.

Narratives often employ a diverse range of character types, each contributing
uniquely to the story’s depth and dynamics. The way characters are developed,
presented, and evolve throughout the narrative plays a crucial role in
engaging the audience and driving the story’s progression.

###  2.2. Conflict

Conflict is a fundamental element that generates tension and drives the
thoughts and actions of characters (Abbott, 2008). The manner in which
conflicts are introduced, developed, and resolved plays a significant role in
shaping the trajectory of the narrative and the engagement of the audience
with the story. A well-crafted conflict not only propels the narrative forward
but also enriches the emotional and psychological depth of the characters and
the story itself. The resolution of the conflict, typically towards the end of
the narrative, is a crucial aspect. It marks the culmination of tension and
provides closure to the story’s central problem.

Broadly, conflict encompasses any form of tension that influences characters,
while more narrowly, it may refer to the main challenge faced by the
protagonist of the story. This primary conflict often leads to additional
struggles for the protagonist, such as feelings of anxiety, insecurity,
indecisiveness, or other mental difficulties. These struggles can be
categorized as secondary or internal conflicts, adding layers to the journey
of the character and depth to the story. Longer stories typically involve
multiple conflicts. These can be smaller-scale issues that occur alongside or
as part of the main conflict, contributing to the complexity of the story.
Commonly used conflict types include: “Character vs. Character” (e.g.,
opposition); “Character vs. Nature” (e.g., natural forces); “Character vs.
Society” (e.g., societal norms); “Character vs. Unavoidable Circumstances”
(e.g., fate); and “Character vs. Self” (e.g., dilemmas).

###  2.3. Theme

Themes encompass the major underlying ideas presented by a story, often
abstract and open to the audience’s interpretation (Griffith, 2010). Unlike
more concrete story elements, themes invite discussion and contemplation,
allowing for a range of interpretations and conclusions. They are not
definitive statements but rather exploratory concepts that encourage the
audience to engage with and reflect on the story’s deeper meanings. As the
narrative unfolds, themes can evolve or reveal new dimensions, offering the
audience opportunity to develop their own ideas about the messages or insights
the story conveys. The subjective nature of themes means that different
audience members may arrive at varying interpretations, which might differ
from or even go beyond the creator’s original intent. Thus, understanding and
incorporating themes in narrative generation allows for a dynamic, thought-
provoking storytelling experience that resonates with and challenges the
audience. An example of a theme is the elusive nature of the American Dream,
as portrayed by F. Scott Fitzgerald in “The Great Gatsby” (Fitzgerald and
Reynolds, 1993).

###  2.4. Setting

The setting is an element that encompasses the time, place, and context in
which the story unfolds (Truby, 2007). It includes the physical and temporal
surroundings inhabited by the characters, as well as the social or cultural
conventions that influence their actions and decisions. The setting can act
dynamically, almost like a character itself, possessing specific traits and
undergoing changes that directly affect the plot and contribute to the story’s
development. This dynamic nature of the setting allows it to evolve over the
course of the story, reflecting and impacting the narrative’s progression. The
Hogwarts School of Witchcraft and Wizardry in the “Harry Potter” series
(Rowling, 2015) is an example of an evolving setting that influences both plot
and character development and impacts the progression of the story.

###  2.5. Plot

The plot is the sequence of events, the plot points, that propels the story
from beginning to middle to end (Dibell, 1999). Each plot point is a moment of
change that influences the understanding, decisions, and actions of the
characters. This progression is often closely tied to the protagonists
encountering or realizing a conflict, and their subsequent efforts to resolve
it. This journey creates emotional stakes for both the characters and the
audience, generating a sense of suspense. The plot’s movement is integral to
the narrative, as it not only advances the storyline but also deepens the
audience’s emotional investment in the fate of the characters. A single story
can have multiple plots, each representing a different sequence of events or
perspective. The plot is usually structured through cause and effect, where
actions of the characters and events lead to reactions and consequences,
forming a chain of scenes that collectively move the story forward.

As an example, in “The Lord of the Rings” by J.R.R. Tolkien (Tolkien, 2012),
the plot unfolds as a series of events beginning with Frodo inheriting the One
Ring and learning about its power and dangers, followed by, among other
events, the Council of Elrond and the formation of the Fellowship, Gandalf’s
fall, the breaking of the Fellowship, and Frodo’s decision to, at first,
continue alone. These plot points are influenced by character decisions and
actions, which lead to various developments and challenges, enhancing the
narrative.

###  2.6. Mode

The mode is a set of choices and techniques employed by the author to frame
the narrative (Pinault, 1992). A single story can be narrated using different
modes, each offering a different perspective or experience to the audience.
The mode includes the scope of the presented information, the style and type
of language used, the medium through which the story is conveyed, and the
manner and extent of narrative exposition and commentary.

A key component of the mode is the point of view or perspective from which the
story is told. A first-person narrative is an example where a character –
often the protagonist – narrates the story using personal pronouns like “I”
and “me” directly engaging the audience with their perspective. A third-person
narrative is another example that avoids personal pronouns, offering a more
detached or impersonal viewpoint. These perspectives are examples of
narration, a technique that involves the explicit telling of a story through
spoken or written commentary. While narration is frequent in written
narratives, it can be optional in other forms, like visual or interactive
media.

##  3\. Benchmark Datasets and Evaluation Metrics

This section presents and discusses benchmark datasets used for story
generation and evaluation metrics that assess the performance of story
generation models.

Although not all approaches are data-driven, some of them, especially deep
learning models and other machine learning approaches, rely heavily on data to
guide the learning process. Various story datasets have been employed in the
development of story generation systems by the community.

Table 1 provides a compilation of the datasets documented in the literature
for both VSG and textual story generation. Among these datasets, VIST (Huang
et al., 2016) is prominently utilized by the research community. This dataset
serves as a primary benchmark, and most publications in this field report
their experimental results on it. The Visual Writing Prompts dataset (Hong et
al., 2023) is noteworthy for its focus on stories that are visually grounded
on the characters and objects present in the images. Similarly, the Writing
Prompts dataset (Fan et al., 2018) and STORIUM (Akoury et al., 2020) are
important because they contain long stories derived from the imagination of
the users of the platforms where they were collected.

These datasets allow for the exploration of more complex and imaginative
narrative structures, serving as a resource to train models to generate
fictional stories.

Dataset | No. of Stories | Type |  Notes
---|---|---|---
NY City (Park and Kim, 2015) | 11,863 | visual |  11,863 blog posts / 78,467 images
Disneyland (Park and Kim, 2015) | 7,717 | visual |  5 7,717 blog posts / 60,545 images
ROCStories (Mostafazadeh et al., 2016) | 100,000 | visual |  5 sentences/story.
LSMDC (Rohrbach et al., 2016) | 118,114 | visual |  202 movies.
VIST (Huang et al., 2016) | 50,000 | visual |  210,819 unique photos
Visual Writing Prompts (Hong et al., 2023) | 12,000 | visual |  5 to 10 images/story, grounded on the characters.
Pororo-SV (Kim et al., 2017) | 27,328 | visual |
YouCook2 (Das et al., 2013) | 2,000 | visual |  Average length of 5.26 mins.
Writing Prompts (Fan et al., 2018) | 300,000 | text |  28.4 words/prompt, 734.5 words/story.
CMU movie summary corpus | 42,306 | visual |  Includes box office revenue, genre, release date, runtime, character names and actor names.
Wikipedia’s movie plots | 34,886 | text |  Summary of the movie plot with metadata.
Sci-fi TV plot summaries (Ammanabrolu et al., 2019) | 2,276 | text |  80+ episodes of TV Series.
STORIUM (Akoury et al., 2020) | 5,743 | text |  19,278 words/story. Annotations for scene entries and cards played to guide the story.
Table 1. Story Generation Datasets.

The evaluation metrics, used to assess the performance of VSG models, provide
quantitative measures to gauge the quality and coherence of generated
narratives. Most of these metrics were originally developed for machine
translation or summarization but have been adopted by the community for other
tasks such as image captioning and story generation: instead of comparing a
generated text with a reference translation, the generated text is compared
with a reference caption in the case of image captioning, or with a reference
story in the case of story generation. In the following subsections, we
discuss commonly used metrics, including their strengths and limitations.

###  3.1. Perplexity

Perplexity (Jurafsky and Martin, 2023) evaluates the performance of
autoregressive language models. It measures the model’s ability to predict a
sequence and is defined as the exponential average negative log-likelihood of
a sequence: given a tokenized sequence $X=(x_{1},x_{2},\ldots,x_{N})$, where
$N$ is the length of the sequence, the perplexity of $X$ is given by Eq. 1.
$P(x_{i}|x_{1},\ldots,x_{i-1})$ denotes the probability of token $x_{i}$ given
the preceding token sequence $x_{1},\ldots,x_{i-1}$ as predicted by the model.
Perplexity can be interpreted as the geometric mean of the inverse probability
of the word sequence, indicating how well the model predicts each subsequent
token in the sequence. Lower values indicate better predictive performance,
meaning that the model is more effective in capturing the underlying structure
of the language.

(1)
$\text{Perplexity}(X)=\sqrt[N]{\prod_{i=1}^{N}\frac{1}{P(x_{i}|x_{1},\ldots,x_{i-1})}},$

###  3.2. BLEU

Bilingual Evaluation Understudy (BLEU) (Papineni et al., 2002) evaluates the
quality of generated text by comparing it with one or more reference texts. It
was developed for machine translation tasks, but has been adopted for various
natural language generation tasks including VSG. BLEU outputs a score between
$0$ and $1$, indicating the similarity between the generated and reference
texts. Higher scores reflect better alignment with the reference: a score of
$1$ signifies perfect agreement.

BLEU compares n-grams in the candidate text with the n-grams in the reference
texts, computing the precision of these matches, which are independent of
their position in the text. The general idea is that the greater the number of
matching n-grams, the higher the quality of the text. To mitigate the issue of
favoring shorter texts (since they can potentially have higher precision),
BLEU incorporates a brevity penalty. This penalty balances precision and
recall, ensuring that the metric does not disproportionately favor shorter
stories.

In Eq. 2, BP is the brevity penalty (Eq. 3), $w_{n}$ is the weight for each
n-gram, $p_{n}$ is the n-gram precision, and $N$ is typically up to four,
accounting for the use of unigrams, bigrams, trigrams, and four-grams.

(2) BLEU $\displaystyle=\text{BP}\cdot\exp\left(\sum_{n=1}^{N}w_{n}\log
p_{n}\right)$ (3) BP $\displaystyle=\begin{cases}1&\text{if }c>r\\\
e^{(1-r/c)}&\text{if }c\leq r\end{cases}$

While widely used, BLEU has limitations in evaluating stories involving word
variations, synonyms, and creative text. Its reliance on exact n-gram matches
fails to capture semantic equivalence when different wording or phrasing is
used, potentially penalizing stories that are semantically accurate but
stylistically different. This results in reduced effectiveness in assessing
linguistic richness, idiomatic expressions, and the subtleties of creative
language use.

###  3.3. Self-BLEU

Self-BLEU (Zhu et al., 2018) evaluates diversity within a set of generated
data. While the original BLEU was designed to assess the similarity between
two texts, Self-BLEU adapts this concept to measure how much each sentence
within a generated text resembles the rest of the text. An individual sentence
is treated as the hypothesis and all other companion sentences are considered
as the reference set. The BLEU score is calculated for each sentence against
the aggregated reference set, and the average of these scores constitutes the
Self-BLEU score for the entire document. In Eq. 4, $N$ is the number of
sentences, $s_{i}$ is the $i$-th sentence, and $S\setminus\\{s_{i}\\}$ denotes
the set of all sentences excluding $s_{i}$.

(4) $\text{Self-
BLEU}=\frac{1}{N}\sum_{i=1}^{N}\text{BLEU}(s_{i},S\setminus\\{s_{i}\\}),$

A higher Self-BLEU score indicates a lower diversity among the generated
sentences, suggesting a potential mode collapse issue within the generative
model. Mode collapse occurs when the model produces a narrow range of outputs,
repeatedly generating similar types of data while failing to capture the full
diversity of the target distribution. Conversely, a lower Self-BLEU score
suggests greater diversity and a healthier, more varied generation process.
This is particularly relevant for story generation models where diversity in
narrative output is crucial for creating engaging and unique stories. High
Self-BLEU scores might signal a lack of creativity and variability in the
stories produced, whereas lower scores indicate a model’s effectiveness in
generating rich, varied narratives.

###  3.4. METEOR

Metric for Evaluation of Translation with Explicit ORdering (METEOR) (Banerjee
and Lavie, 2005) was designed to evaluate machine translation outputs,
addressing some limitations of BLEU. It focuses on sentence-level evaluation,
aiming to achieve higher correlation with human judgments. The metric relies
on the harmonic mean of unigram precision and recall, with emphasis placed on
recall to better align with human evaluative tendencies.

The algorithm initiates by aligning the candidate text with the reference
text, focusing on unigram mappings. These mappings are constrained such that
each unigram in the candidate text aligns with zero or one unigram in the
reference text. The alignment process prioritizes configurations with the
fewest crossing lines to ensure clarity and coherence.

Precision $P={m}/{w_{t}}$ and recall $R={m}/{w_{r}}$ are the ratios between
$m$ (the number of matching unigrams in the candidate text) and either $w_{t}$
(the number of unigrams in the candidate text) or $w_{r}$ (the number of
unigrams in the reference text). These metrics are integrated using the
harmonic mean, emphasizing recall $F_{mean}=10\cdot P\cdot R/(R+9\cdot P)$. To
account for the arrangement of words, METEOR introduces a penalty
$p=0.5(c/u_{m})^{3}$, where $c$ is the number of chunks, and $u_{m}$ is the
number of mapped unigrams. The final METEOR score for a candidate text is
given by $M=(1-p)F_{mean}$.

###  3.5. CIDEr

Consensus-based Image Description Evaluation (CIDEr) (Vedantam et al., 2014)
was developed for image captioning evaluation. It evaluates how well a
candidate sentence $c_{i}$ aligns with a consensus among a set of reference
sentences $S_{i}=\\{s_{i1},\ldots,s_{im}\\}$ for an image $I_{i}$. CIDEr
involves stemming words in the sentences to their root forms. Each sentence is
represented by its n-grams, where an n-gram $\omega_{k}$ is an ordered set of
words, and n-grams from one to four words are considered.

CIDEr evaluates the consensus by measuring how frequently n-grams in the
candidate sentence appear in the reference sentences. It also assigns lower
weights to n-grams common across all images in the dataset, as they are likely
less informative. The Term Frequency-Inverse Document Frequency (TF-IDF)
(Leskovec et al., 2014) method is employed to weight each n-gram $\omega_{k}$
based on its informativeness, as shown in Eq. 5, where $h_{k}(s_{ij})$ is the
frequency of $\omega_{k}$ in sentence $s_{ij}$, and $\Omega$ is the set of all
n-grams and $|I|$ is the set of all images in the dataset.

(5)
$g_{k}(s_{ij})=\frac{h_{k}(s_{ij})}{\sum_{\omega_{l}\in\Omega}h_{l}(s_{ij})}\log\left(\frac{|I|}{\sum_{I_{p}\in
I}\min(1,\sum_{q}h_{k}(s_{pq}))}\right),$

The CIDEr score for n-grams of length $n$ is calculated using the average
cosine similarity between the TF-IDF vectors of the candidate and reference
sentences, capturing both precision and recall as shown in Eq. 6, where
$g_{n}(c_{i})$ and $g_{n}(s_{ij})$ are the TF-IDF vectors for n-grams of
length $n$ in the candidate and reference sentences, respectively. $m$ is the
number of reference sentences for the candidate image $i$.

(6) $CIDEr_{n}(c_{i},S_{i})=\frac{1}{m}\sum_{j}\frac{g_{n}(c_{i})\cdot
g_{n}(s_{ij})}{\|g_{n}(c_{i})\|\|g_{n}(s_{ij})\|},$

To derive the final score, CIDEr combines scores across different n-gram
lengths, as shown in Eq. 7, where uniform weights $w_{n}=1/N$ have empirically
been found to perform best, and $N$ is typically set to 4.

(7) $CIDEr(c_{i},S_{i})=\sum_{n=1}^{N}w_{n}CIDEr_{n}(c_{i},S_{i}),$

###  3.6. ROUGE

Recall-Oriented Understudy for Gisting Evaluation (ROUGE) (Lin, 2004) is a
recall-oriented measure of the overlap of n-grams between a candidate text and
reference texts. It was developed for summarization tasks, but has been used
in other natural language generation tasks. ROUGE-N, one of the most common
variations of the metric, is the ratio of the number of n-grams co-occurring
in the candidate text and the reference texts to the total number of n-grams
in the reference texts, as shown in Eq. 8, where $\text{gram}_{n}$ is the
n-gram being considered, and $\text{Count}_{\text{match}}(\text{gram}_{n}(S))$
is the maximum number of n-grams co-occurring in a candidate and a set of
reference texts.

(8)
$\text{ROUGE-N}=\frac{\sum_{S\in\\{\text{ReferenceTexts}\\}}\text{Count}_{\text{match}}(\text{gram}_{n}(S))}{\sum_{S\in\\{\text{ReferenceTexts}\\}}\text{Count}(\text{gram}_{n}(S))}$

As the number of reference texts increases, the denominator in the ROUGE-N
formula also increases, which is intuitive since adding more references
expands the space of acceptable texts, enhancing the robustness of the
evaluation.

For multiple references, ROUGE-N is calculated as the maximum of the pairwise
ROUGE-N scores between the candidate text and each reference text (Eq. 9,
where $r_{i}$ is a reference text, and $s$ is the candidate text).

(9) $\text{ROUGE-N}_{\text{multi}}=\max_{i}\text{ROUGE-N}(r_{i},s)$

This approach is also extended to other ROUGE metrics such as ROUGE-L (based
on the longest common subsequence), ROUGE-W (considers word sequences with
gaps), and ROUGE-S (based on co-occurrence statistics of skip-bigrams, i.e.,
pairs of words in sentence order, allowing arbitrary gaps between them).

###  3.7. BERTScore

BERTScore (Zhang et al., 2019) is a metric that uses contextual embeddings to
evaluate the similarity between a reference sentence
$x=\\{x_{1},\ldots,x_{k}\\}$ and a candidate sentence
$\hat{x}=\\{\hat{x}_{1},\ldots,\hat{x}_{l}\\}$. In story generation, $x$ and
$\hat{x}$ are the reference and candidate stories, respectively. Each token in
$x$ and $\hat{x}$ is a contextual embedding obtained from models trained on
language modeling objectives, enabling the generation of distinct vectors for
the same word in different contexts. The similarity between tokens is computed
using cosine similarity. For pre-normalized vectors, this calculation
simplifies to the inner product of the embeddings, that is,
$\text{similarity}(x_{i},\hat{x}_{j})=x_{i}^{\top}\hat{x}_{j}$.

BERTScore computes recall $R_{\text{BERT}}$, precision $P_{\text{BERT}}$, and
F1 score $F_{\text{BERT}}$ by matching each token in one sentence to the most
similar token in the other sentence. These scores are defined as shown in Eqs.
10, 11, and 12, respectively.

(10) $\displaystyle R_{\text{BERT}}=\frac{1}{|x|}\sum_{x_{i}\in
x}\max_{\hat{x}_{j}\in\hat{x}}x_{i}^{\top}\hat{x}_{j}$ (11) $\displaystyle
P_{\text{BERT}}=\frac{1}{|\hat{x}|}\sum_{\hat{x}_{j}\in\hat{x}}\max_{x_{i}\in
x}x_{i}^{\top}\hat{x}_{j}$ (12) $\displaystyle F_{\text{BERT}}=\frac{2\cdot
P_{\text{BERT}}\cdot R_{\text{BERT}}}{P_{\text{BERT}}+R_{\text{BERT}}}$

BERTScore can incorporate importance weighting using Inverse Document
Frequency (IDF) scores, emphasizing rare words which are often more indicative
of sentence similarity. Eq. 13 shows the recall with IDF weighting.

(13) $R_{\text{BERT}}^{\text{idf}}=\frac{\sum_{x_{i}\in
x}\text{idf}(x_{i})\max_{\hat{x}_{j}\in\hat{x}}x_{i}^{\top}\hat{x}_{j}}{\sum_{x_{i}\in
x}\text{idf}(x_{i})}$

To make BERTScore more interpretable, it is rescaled against an empirical
lower bound $b$, derived from averaging BERTScores computed on randomly paired
sentences from a large corpus. Eq. 14 shows the rescaled BERTScore
$\hat{R}_{\text{BERT}}$.

(14) $\hat{R}_{\text{BERT}}=\frac{R_{\text{BERT}}-b}{1-b}$

###  3.8. Discussion

These metrics guide the development and refinement of visual story generation
models by offering insights into their linguistic fluency, relevance to visual
content, and overall narrative coherence. They focus on specific linguistic
and content-related aspects, providing quantitative measures for precision,
recall, and consensus. They are, however, limited by their focus on surface-
level textual similarities rather than deep narrative alignment: since a story
can be retold in various ways, employing different perspectives while
preserving the original message, the corresponding generated text might
deviate significantly from the reference text in terms of specific wording.

Huang et al. (Huang et al., 2016) conducted an analysis to evaluate how well
BLEU and METEOR correlate with human judgment in the context of VSG according
to different metrics. The results in Table 2 show that METEOR, which places a
strong emphasis on paraphrasing, has higher correlation with human judgment
than BLEU. Nevertheless, in spite of METEOR showing the highest correlation
with human judgment, this value is still low, suggesting that a large gap
still exists between automatic metrics and human judgment.

Metric | METEOR | BLEU
---|---|---
Pearson | 0.22 (2.8e-28) | 0.08 (1.0e-06)
Spearman | 0.20 (3.0e-31) | 0.08 (8.9e-06)
Kendall | 0.14 (1.0e-33) | 0.06 (8.7e-08)
Table 2. Correlations of automatic scores against human judgements, with
p-values in parentheses (Huang et al., 2016).

To evaluate VSG models, additional metrics that capture the creative aspects
of storytelling may be necessary. These metrics might assess the uniqueness,
novelty, and diversity of generated narratives, ensuring that models produce
stories that go beyond mere data memorization. Coherence metrics could delve
deeper into the logical flow of events, ensuring that generated narratives
maintain a consistent and plausible storyline.

The proliferation of diverse LLMs, such as the GPT (OpenAI, 2024c, b, a),
Llama (Touvron et al., 2023; Meta, 2024), Mistral (team, 2023), and Claude
(Anthropic, 2024b, a, 2024) families, has led to the development of systematic
scoring systems, aiming for objective assessments across a wide array of
language understanding tasks. These scoring systems include Multi-turn
Benchmark (MT-Bench) (Zheng et al., 2023) and Chatbot Arena (Zheng et al.,
2023) briefly described below. While they were initially developed for the
evaluation of chatbots by human evaluators, they have been adapted to operate
using a LLM as the evaluator achieving results close to those obtained by
human evaluators.

As LLMs continue to improve, they show potential in replacing human annotators
in many tasks (Gilardi et al., 2023). Metrics to evaluate stories could be
developed based on LLMs (Zheng et al., 2023), possibly following the
underlying principles discussed above.

MT-Bench is a benchmark tailored to test the multi-turn conversation and
instruction-following capabilities of LLM-based chat assistants. It includes
80 high-quality multi-turn questions distributed across diverse categories
such as writing, roleplay, reasoning, and knowledge domains. In the MT-Bench
evaluation process, the user is presented with two distinct conversations,
each generated by a different LLM-based chat assistant and is then tasked with
deciding which assistant, A, B, or indicating a tie, better followed the
instructions and answered the questions.

Chatbot Arena introduces a crowd-sourcing approach, fostering anonymous
battles between LLM-based chatbot models. Users interact with two anonymous
models simultaneously, posing identical questions to both and subsequently
voting for the model that provides the preferred response. The Elo rating
system (Zheng et al., 2023) is employed to rank the models based on their
performance in these battles. This rating system assigns a numerical score to
each model based on their battle outcomes, adjusting these scores higher or
lower after each encounter to reflect their relative ability to satisfy user
queries effectively compared to their opponents.

In summary, while current metrics provide insights into specific dimensions of
VSG, the development of new metrics may be necessary to holistically evaluate
creativity, coherence, and emotional engagement, ensuring a more comprehensive
understanding of the capabilities and limitations of these models. We
anticipate that LLMs can play a role in the development of such metrics.

##  4\. Story Generation

The use of Artificial Intelligence (AI) for automated story generation has
been explored since the 1960s and 1970s. Early approaches include rule-based
systems (Section 4.1), plan-based methods (Section 4.3), grammar-based systems
(Section 4.2), and case-based reasoning systems (Section 4.4). In recent
times, with the availability of large datasets and computational resources the
these methods have been replaced by deep learning models: Recurrent Neural
Networks (RNNs) (Section 4.5) which have been widely explored for story
generation tasks; and LLMs (Section 4.6) which currently achieve state-of-the-
art performance in various natural language generation tasks.

###  4.1. Rule-based Methods

Rule-based methods in narrative generation use a predefined set of rules to
create stories. These rules govern the relationships between narrative
elements, ensuring consistency within the narrative, aligning with the desired
outcomes and narrative styles or genres. However, it is essential to
acknowledge the potential limitations in flexibility and creativity, as the
story’s possibilities are bounded by the predefined rules and the system’s
ability to simulate complex interpersonal dynamics and character development.

TALE-SPIN (1977) (Meehan, 1977) was an influential rule-based method that
initiated its storytelling process by presenting a central problem to the main
character, derived from foundational states. As the narrative unfolds, the
character may encounter various sub-problems and unintended side effects,
which also require resolution. The complexity of these challenges can be
adjusted by the user, influencing the story’s length and intricacy. This
capability to modulate narrative complexity allows for a range of storytelling
from straightforward tales to more complex narratives, depending on the inter-
character dynamics established by the user. A pivotal aspect of narrative
construction of TALE-SPIN is its nuanced handling of character relationships
which are essential for the progression and richness of the story. These
relationships are categorized into various dimensions such as competition,
dominance, familiarity, affection, trust, deceit, and indebtedness. These
relationships not only set preconditions for the actions of the characters but
also emerge as outcomes of their interactions within the narrative. For
instance, the act of one character giving a gift to another can lead to the
recipient feeling a sense of affection towards the giver, thereby influencing
subsequent narrative developments. Fig. 1 shows an example of a story
generated by TALE-SPIN.

Once upon a time there were two bears named Jack and Joe and a bee named Sam.
Jack was very friendly with Sam but very competitive with Joe, who was a
dishonest bear. One day Jack was hungry. He knew that Sam Bee had some honey
and that he might be able to persuade Sam to give him some. He walked from his
cave down the mountain trail, across the valley, over the bridge to the oak
tree where Sam Bee lived. He asked Sam for some honey. Sam gave him some. Then
Joe Bear walked over to the oak tree and saw Jack Bear holding the honey. He
thought that he might get the honey if Jack put it down, so he told him that
he didn’t think Jack could run very fast. Jack accepted this challenge and
decided to run. He put down the honey and ran over the bridge and across the
valley. Joe picked up the honey and went home.

Figure 1. An example story by TALE-SPIN (Meehan, 1977).

IDtension (2003) (Szilas, 2003) merges dramatic storytelling with interactive
fiction. This architecture is designed to balance narrative consistency with
the creative spontaneity essential for immersive interactive experiences.
IDtension is composed of five foundational modules: the world of the story,
narrative logic, narrative sequencer, user model, and the theater. Each module
plays a role in crafting a narrative that responds and evolves based on user
interactions, maintaining an equilibrium between narrative depth and
interactive freedom. The world of the story module serves as the narrative’s
backbone, housing the essential entities and their states, thereby setting the
stage for the unfolding drama. The narrative logic then calculates all
possible character actions derived from the narrative ecosystem. The narrative
sequencer filters through actions to highlight those with the greatest
narrative value, ensuring that each user decision is met with consequences
that propel the story forward. The user model captures the user’s current
narrative context, guiding the sequencer in tailoring the narrative flow to
enhance user engagement. The theater is responsible for displaying the
action(s), and manages the interaction between the computer and the user. Fig.
2 shows an example of a story generated by IDtension.

Anna tells Joe he could try to buy Mr D., the witness / Joe accepts / Bill
tells Joe to kill Mr D. / Joe refuses to do that. / Joe tells Anna he could
kill Mr D. / Anna dissuades him to do so! / Joe tells Anna he wants to buy Mr
D. / Anna incites him to do so / Bill incites Joe to kill Mr D. / Joe meets Mr
D. / He proposes him some money for changing his testimony, but Mr D. wants a
lot of money, and Joe is not rich enough / He then decides to get money / He
tells Anna about it / Joe tells Sylvie he could kill Mr D. / Sylvie dissuades
him to do so! / Joe tells her he is trying to get money / He tells Bill too
about it / Bill inform him that he could rob the bank / Joe accepts. / He
tells Anna he wants to rob the bank / Anna incites him to do it / He tells
Bill about his intention two / Bill incites to do it / He tells Sylvie about
his intention / Sylvie dissuades him to do so / Joe robs the bank, and he is
successful / He pays Mr D. / He obtains the withdrawal of testimony from Mr D.

Figure 2. An example story by IDtension (Szilas, 2003) showing the telegraphic
nature of the text.

###  4.2. Grammar-based Methods

Grammar-based methods in narrative generation, a specialized subset of rule-
based approaches, employ formal grammars, such as context-free grammars, to
systematically construct stories. The vocabulary elements can be narrative
elements like the ones described in the previous section. In contrast with
rule-based methods, which emphasize logical consistency and specific narrative
outcomes through predefined rules guiding character and plot interactions,
these grammars consist of a set of production rules that define how various
elements of a story can be combined to form coherent narratives. This method
emphasizes linguistic correctness and structural coherence, making it suitable
for applications where the narrative needs to adhere to specific stylistic or
syntactic standards. However, the creativity and variety of the narratives
generated are often limited by the constraints of the grammar used. Grammar-
based systems are particularly effective in scenarios where the narrative
needs to follow a well-defined and consistent structure.

Different grammars have been proposed that capture narrative patterns such as
the three-act structure (Aristotle et al., 1997), Propp’s morphology (Propp et
al., 2010), Freytag’s pyramid (Freytag and MacEwan, 1894), the hero’s journey
monomyth (Campbell, 2008), among others. Grammars can be designed for specific
genres or domains.

Examples of grammar-based methods include systems developed by Grimes (Ryan,
2017) and Gervás (Gervás, 2013). Grimes’ system was one of the pioneering
approaches in the 1960s to harness computational methods for automatic story
generation. The system would randomly select elements from Propp’s list
(Propp, 1968) and then order them properly to generate a coherent story. This
was a significant achievement, considering the computational limitations of
the time. This system employed a grammar-based technique, drawing inspiration
from Propp’s morphology, to craft structured narratives. Gervás (2013),
decades after Grimes’ system, computationally modeled Russian folktales. His
approach involved representing Propp’s morphology by analyzing character roles
and narrative functions. This was implemented as a generative grammar that
arranges plot elements according to a chronological sequence of narrative
functions.

###  4.3. Plan-based Methods

Plan-based methods for narrative generation involve using planning algorithms
to create a storyline. In this approach, the system formulates a narrative
goal and then plans a sequence of events or actions that characters can take
to achieve that goal. This method allows for dynamic storytelling, as the
system can adapt to different narrative goals and constraints. However,
incoherent stories can be generated if the planning algorithm is not
sophisticated enough. Plan-based methods are useful for creating stories that
require logical progression and can adapt to changing story requirements.
Examples of plan-based methods include MEXICA (Pérez y Pérez and Sharples,
2001) and Mimesis (Young et al., 2004).

####  4.3.1. MEXICA

MEXICA (1999) (Pérez y Pérez and Sharples, 2001) used creative storytelling to
generate Aztec mythology tales. Character-based planning drove narrative
progression based on changes in character emotions extracted from a case
library.

MEXICA operates through two main processes. Initially, it builds knowledge
from a collection of predefined stories, using this information to create
schemas in its Long-Term Memory (LTM) that encapsulate content and rhetorical
knowledge specific to a genre or form of writing. Users can add or modify
stories in the system’s archive using a specialized language designed for
defining these stories (PSDL) and the actions within them (SADL).

The MEXICA storytelling core is an engagement-reflection cycle. In the
engagement phase, MEXICA generates stories by planning a sequence of actions,
each with associated post-conditions that alter the story world (e.g.,
emotional links between characters or changes in their physical positions).
These post-conditions are not explicitly stated in the story but guide the
narrative’s direction. For instance, the action “princess cures jaguar
knight’s injuries” not only progresses the plot but also establishes an
emotional link (gratitude) between the characters, influencing future actions.

Characters in MEXICA possess individual “story-world contexts”, which are
dynamic records of the post-conditions affecting them. These contexts ensure
that each character’s storyline reflects their unique experiences and
interactions in the narrative, allowing for complex and divergent storylines
based on perspectives and relationships of the characters.

When MEXICA reaches a narrative impasse, that is, when it cannot proceed based
on the current story-world context, it enters the reflection phase. Here, it
analyzes previous stories for inspiration to overcome the block, assessing
novelty and interestingness by comparing the current story’s content and
dramatic tension with those of past narratives. This process allows MEXICA to
break impasses and continue with story generation, thereby simulating a human
writer’s strategy of seeking external inspiration to resolve creative
obstacles. Fig. 3 shows a story generated by MEXICA. This example shows how
limited pronoun variation and repetitive use of character names cause the
narrative to be less fluid.

Jaguar_knight was an inhabitant of the great Tenochtitlan. Princess was an
inhabitant of the great Tenochtitlan. From the first day they met, Princess
felt a special affection for Jaguar_knight. Although at the beginning Princess
did not want to admit it, Princess fell in love with Jaguar_knight. Princess
respected and admired Artist because Artist’s heroic and intrepid behaviour
during the last Flowery-war. For long time Jaguar_knight and Princess had been
flirting. Now, openly they accepted the mutual attraction they felt for each
other. Jaguar_knight was an ambitious person and wanted to be rich and
powerful. So, Jagua_knight kidnapped Artist and went to Chapultepec forest.
Jaguar_knight’s plan was to ask for an important amount of cacauatl (cacao
beans) and quetzalli (quetzal) feathers to liberate Artist. Princess had
ambivalent thoughts towards Jaguar_knight. On one hand princess had strong
feelings towards Jaguar_knight but on the other hand Princess abominated what
Jaguar_knight did. Suddenly, the day turned into night and after seconds the
sun shone again. Princess was scared. The Shaman explained to Princess that
Tonatiuh (the divinity representing the sun) was demanding Princess to rescue
Artist and punish the criminal. Otherwise Princess’s family would die. Early
in the Morning Princess went to Chapultepec forest. Princess thoroughly
observed Jaguar_knight. Then, Princess took a dagger, jumped towards
Jaguar_knight and attacked Jaguar_knight. Jaguar_knight was shocked by
Princess’s actions and for some seconds Jaguar_knight did not know what to do.
Suddenly, Princess and Jaguar_knight were involved in a violent fight. In a
fast movement, Jaguar_knight wounded Princess. An intense haemorrhage arose
which weakened Princess. Jaguar_knight felt panic and ran away. Thus, while
Tlahuizcalpantecuhtli (the god who affected people’s fate with his lance)
observed, Princess cut the rope which bound Artist. Finally, Artist was free
again! Princess was emotionally affected and was not sure if what Princess did
was right. Princess was really confused. The injuries that Princess received
were very serious. So, while praying to Mictlantecuhtli (the lord of the land
of the dead) Princess died.

Figure 3. An example story by MEXICA (Pérez y Pérez and Sharples, 2001).

####  4.3.2. Mimesis

Mimesis (2004) (Young et al., 2004) addresses a fundamental challenge in game
development: creating compelling and context-sensitive behaviors for
characters and objects. Its architecture employs a structured approach to
generate and execute narrative plans within interactive game environments. The
process begins when the world planner receives a request from the game engine,
detailing a narrative problem that requires resolution, such as achieving
specified narrative objectives using a predetermined set of actions. In
response, the world planner formulates a plan, a structured dataset that
sequences actions for characters and other game entities to fulfill the
narrative goals outlined in the request. Following the creation of the story
world plan, the discourse planner manages non-character elements like camera
movements and background music, enhancing the narrative unfolding within the
game. The discourse planner then uses the world plan to craft a narrative plan
that encapsulates all system and user activities planned in response to the
game engine’s initial request. The final step involves the execution manager,
which receives the narrative plan and builds a Directed Acyclic Graph (DAG) to
map temporal dependencies among the actions. It then navigates through the
DAG, initiating ready actions, commanding the game engine to execute them, and
monitoring their outcomes to ensure successful completion or to record any
failures. We do not include an example story of Mimesis as it is a system
designed to generate stories in interactive game environments.

###  4.4. Case-based Reasoning Systems

Case-based reasoning approaches generate new stories by retrieving and
adapting prior story plots stored in a memory of narrative cases. When a new
story situation is defined, including elements like setting, characters, and
author goals, this situation is matched against the narrative case library to
find the most similar prior plots. One or more retrieved plots are adapted to
the new situation by modifying characters, events and descriptions, creating a
new plot that becomes the framework for the new story (it can also be added to
the library for future reuse). This approach allows generating novel
narratives by reusing and building upon successful past story structures.

Case-based narrative systems must define how to represent, index, match,
adapt, and learn from story cases. Large case libraries, similarity metrics,
and adaptation rules enable generating a wide range of creative plot
variations.

Examples of case-based reasoning systems include MINSTREL (1993) (Turner,
1993) and Story Plot Generation (2004) (Gervás et al., 2004). The latter uses
existing stories and case-based reasoning to create new stories based on user
queries. MINSTREL is a creative storytelling system that generated original
stories about King Arthur. It combines case-based goal planning with models of
drama and emotional responses. Fig. 4 shows an example of a story created by
the system.

It was the spring of 1089, and a knight named Lancelot returned to Camelot
from elsewhere. Lancelot was hot tempered. Once, Lancelot lost a joust.
Because he was hot tempered, Lancelot wanted to destroy his sword. Lancelot
struck his sword. His sword was destroyed.

One day, a lady of the court named Andrea wanted to have some berries. Andrea
went to the woods. Andrea had some berries because Andrea picked some berries.
Lancelot’s horse moved Lancelot to the woods. This unexpectedly caused him to
be near Andrea. Because Lancelot was near Andrea, Lancelot saw Andrea.
Lancelot loved Andrea.

Some time later, Lancelot’s horse moved Lancelot to the woods unintentionally,
again causing him to be near Andrea. Lancelot knew that Andrea kissed with a
knight named Frederick because Lancelot saw that Andrea kissed with Frederick.
Lancelot believed that Andrea loved Frederick. Lancelot loved Andrea. Because
Lancelot loved Andrea, Lancelot wanted to be the love of Andrea. But he could
not because Andrea loved Frederick. Lancelot hated Frederick. Because Lancelot
was hot tempered, Lancelot wanted to kill Frederick. Lancelot went to
Frederick. Lancelot fought with Frederick. Frederick was dead.

Andrea went to Frederick. Andrea told Lancelot that Andrea was siblings with
Frederick. Lancelot believed that Andrea was siblings with Frederick. Lancelot
wanted to take back that he wanted to kill Frederick, but he could not because
Frederick was dead. Lancelot hated himself. Lancelot became a hermit.
Frederick was buried in the woods. Andrea became a nun.

Moral: “Done in haste is done forever.”

Figure 4. An example story by MINSTREL (Turner, 1993).

MINSTREL narratives, while coherent, lack several elements that would make
them more engaging. The story’s phrasing is repetitive, which, along with the
lack of descriptive detail, hampers the reader’s ability to fully immerse
themselves in the world or connect with the characters. It also misses
opportunities to demonstrate the emotions and motivations of the characters
through their actions or dialogues, rather than through plain statements.

###  4.5. Recurrent Neural Networks

Recurrent Neural Network (RNN) are a deep learning models for sequential data
processing, with applications ranging from language modeling to image
captioning and text generation, that influenced later models. Sequential
processing in RNNs involves handling sentences word by word, which, while
effective for capturing short-term dependencies, poses challenges in learning
long-range dependencies.

RNNs encounter difficulties with long-range dependencies due to issues like
vanishing or exploding gradients. This poses a limitation in capturing complex
dependencies over long sequences. Architectures such as Long Short-Term Memory
network (LSTM) (Hochreiter and Schmidhuber, 1997) and Gated Recurrent Units
(GRUs) (Chung et al., 2014) were introduced to address these issues.

Choi et al. (Choi et al., 2016) used RNNs to generate stories by predicting
the next sentence vector based on the current input, ensuring the continuity
of the narrative flow within a story. Jain et al.(Jain et al., 2017) used GRUs
to encode sequences of variable length input descriptions and to decode them
as story-like summaries. Fig. 5(a) shows an example input provided to the
model, and Fig. 5(b) shows an example story generated by the model. RNNs base
their narratives generation decisions on statistical associations derived from
the datasets on which they are trained, meaning that narrative elements are
learned implicitly from the training data, leading to outputs that can
sometimes be unpredictable. In the example we can see this unpredictability in
the repetition of the phrase “we also saw some lily pads .”.

the leaf has begun to change it ‘s color . close up of a bush with small pink
flower clusters. purple and white flowers growing outdoors together .

(a) Input captions.

we decided to take a trip to the botanical gardens . there was a beautiful
view of the water . we also saw some lily pads . there was a lot of flowers on
the ground . we also saw some lily pads .

(b) Generated story.

Figure 5. A story generated by an RNN (Jain et al., 2017).

###  4.6. Transformers and Large Language Models

Transformers (Vaswani et al., 2017) are a class of deep-learning models that
represent a paradigm shift in sequential data processing and are the basis of
the Large Language Models (LLMs) revolution. In contrast with RNNs, which
process words sequentially in a recurrent way, transformers process the entire
sentence simultaneously, eliminating the risk of “forgetting” past information
and improving the training process efficiency.

LLMs are composed of multiple transformer layers, which enable them to attend
to information at the word, sentence, and paragraph levels. This multi-layered
approach allows for the construction of narratives that are not only coherent
but also perceived as richly detailed and creatively varied. LLMs also
leverage their extensive training data to produce stories across a wide range
of styles and genres. However, as with RNNs, LLMs do not explicitly learn
narrative elements or the underlying structure of storytelling. Their ability
to generate stories originates from the statistical patterns and associations
found within their training corpora. Nevertheless, the transformer
architecture’s ability to attend to information across different parts and
levels of the text enables LLMs to maintain context and coherence over diverse
textual structures, contributing to their effectiveness in generating
narrative content.

Bidirectional Encoder Representations from Transformers (BERT) (Devlin et al.,
2019) is one of the first LLMs. It was widely adopted and established several
state-of-the-art results for downstream tasks, even though it is now
considered a small model: the base version has 110 million parameters, while
current state-of-the-art models, such as Generative Pre-trained Transformer
(GPT)-4 (OpenAI, 2024b) and Mixtral (team, 2023), have hundreds of billions or
even trillions of parameters. Current LLMs, such as GPT (OpenAI, 2024b), have
demonstrated proficiency in understanding context, generating coherent text,
and even simulating human-like conversational abilities.

Some of the key aspects that characterize a LLM and are crucial for their
effectiveness in narrative generation are their size (number of parameters),
the nature of the training data, and the context window size. Regarding size,
larger models tend to perform better, capturing more complex language nuances.
However, increasing model size also requires more computational resources for
training and inference. Regarding training data, LLMs benefit from diverse and
extensive training datasets: a broader range of topics and writing styles in
the training data helps the model generalize well to various tasks. The
context window size refers to the number of preceding tokens the model
considers when generating the next token. This is a critical feature, as it
determines how much of the previous text the model can “remember” and use for
making predictions about what comes next. A larger context window allows the
model to understand longer dependencies in the text, improving coherence and
context awareness. This is especially important for tasks that involve lengthy
discussions or complex narratives. However, larger context windows come with
increased computational costs and this relationship is quadratic (Vaswani et
al., 2017), posing significant challenges in terms of memory and processing
power, especially for very large models and has been a key focus of recent
research efforts (Ding et al., 2023).

Several works using LLMs have been proposed for story generation. Tang et al.
(Tang et al., 2022) proposed TextBox 2.0, a library that uses LLMs to generate
stories. We do not include an example of a story generated by TextBox 2.0 as
the authors did not provide one in their paper. Yang et al. (Yang et al.,
2022) proposed Re3, a framework that prompts a general-purpose LLM to build a
story plan consisting of a setting, characters, and a plot. This structured
plan is generated from a short sentence describing the story, called premise.
Story passages are then generated by repeatedly injecting contextual
information from both the plan and current story state via prompt. This
approach is one of the first approaches to explicitly model the elements
character, setting, and plot in the story generation process. Fig. 6 shows the
initial setup and outline for an example premise. The resulting story
incorporates all the elements from the premise, including character
interactions, setting descriptions, and plot developments. We did not include
the generated story as it occupies several pages and is available in the
original paper (Yang et al., 2022).

Premise: A woman discovers that her husband has been cheating on her and
decides to get revenge.

Setting: The story is set in the present day, in a small town.

Characters:
1\. Diane Chambers is a beautiful woman in her early thirties. She has long
dark hair and blue eyes. She is married to Mark Chambers and is a stay-at-home
mom.
2\. Mark Chambers is a successful businessman in his early forties. He is tall
and handsome, with brown hair and green eyes. He is married to Diane Chambers
and is the father of two young children.
3\. Karen Johnson is a beautiful blonde women in her early thirties. She is
Mark Chambers’ mistress and is also married with two young children.

Outline:
1\. Diane Chambers discovers that her husband has been cheating on her with
another woman.
2\. Diane Chambers decides to take revenge on her husband by having an affair
of her own.
3\. Diane Chambers’ affair is discovered by her husband, leading to a
confrontation between the two.

Figure 6. Re3 initial setup and outline for the shown premise (Yang et al.,
2022).

###  4.7. Discussion

A critical limitation of rule-based, grammar-based, plan-based, and case-based
methods is that they struggle with language variation, especially (but not
only) with pronoun variation. This results in the frequent and repetitive use
of certain structures and constituents, such as character names, leading to
less fluid and less engaging narratives. The principle of “Show, Don’t Tell”
(Dietrich and Sundell, 1983) is often underused: showing is more powerful than
telling because it allows the reader to experience the story directly, rather
than being told what to feel. However, these systems tend to rely on direct
exposition to convey character emotions and story developments, rather than
implying them through actions and interactions. This failure is not because
the methods cannot solve these issues. Rather, they would require an
impractically large number of conditions and rules to be manually encoded.
RNNs are not immune to these challenges, but their inherent adaptability and
capacity for learning from extensive datasets make them more capable of
capturing the complexities of narrative storytelling. Likewise, LLMs are able
to create fluid narratives with varied sentence structures, dialogues between
characters, rich and descriptive details about the setting and characters,
because they are able to learn aspects of the world from their training data.
This allows them to generate details about the setting and characters that are
not explicitly provided in the prompt. LLMs, however, are not without their
limitations: they may suffer from hallucinations, that is, generating details
that completely deviate from the original prompt, or generate contradictory or
nonsensical information.

##  5\. From NLP to computer vision

This section provides an overview of areas at the intersection of computer
vision and Natural Language Processing (NLP) that are relevant to the topic of
VSG, namely image and video captioning and Visual Question Answering (VQA).
These tasks share a common challenge: they require the understanding of visual
content and the ability to generate text based on that visual content. Some of
the techniques used in these tasks have been adapted to VSG. Special attention
is given to approaches based on deep learning architectures.

###  5.1. Image and Video Captioning

The transition from image and video captioning to VSG represents a shift from
describing individual images or video segments to narrating image sequences or
video segments. While the former focus on what is in images or videos, the
latter also considers why and how, capturing the underlying narrative that
links a series of images or video frames.

VSG models need to understand the temporal progression of events, causal
relationships, and the ability to construct cohesive and engaging stories.
Therefore, they build upon the foundational principles of image and video
captioning, incorporating both the interpretation of visual data and the
generation of coherent, contextually appropriate narratives.

The tasks of image and video captioning consist in generating descriptive,
human-readable sentences that accurately represent the content of an image or
video, identifying the main components in a visual scene and their
relationships, alongside the ability to express these details in natural
language. Initial efforts in image and video captioning relied on template-
based and rule-based approaches. Current deep learning models are capable of
end-to-end training, resulting in a substantial enhancement in the quality of
the generated captions.

One of the most notable contributions in this domain was the application of
encoder-decoder architectures, known as the ‘show and tell’ approach (Vinyals
et al., 2014; Yao et al., 2015; Anderson et al., 2017). This technique employs
Convolutional Neural Networks (CNNs) as encoders to extract visual features.
Specifically, 2D-CNNs are employed for image processing where the model
interprets two dimensions: height and width. On the other hand, for video
processing, 3D-CNNs are typically used where an additional dimension, time or
depth, is taken into account to perceive the temporal dynamics in video
sequences. Then, RNNs or their improved variants, LSTMs (Hochreiter and
Schmidhuber, 1997) and GRUs (Chung et al., 2014), known for their ability to
effectively model sequential data, are commonly used as decoders in image and
video captioning tasks, enabling the generation of coherent and contextually
meaningful textual captions and descriptions from the visual features
extracted by the CNNs. This combination of CNNs and RNNs effectively bridges
the gap between vision and language.

In the realm of image captioning, transformers have been used as encoders to
process visual features extracted from images. Leveraging the self-attention
mechanism inherent in transformers, these models are able to capture
relationships between visual elements, attending to relevant regions in the
image, and highlighting visual cues and their interdependencies (Olimov et
al., 2021; Radford et al., 2021; Liu et al., 2021a; Yu et al., 2019).
Transformer models also demonstrate the ability to generate precise and
elaborate textual descriptions, significantly improving the overall quality of
image captions.

Similarly, transformers have made contributions to video captioning,
addressing the challenges associated with understanding and describing
temporal dynamics in videos. By treating video frames as a sequential series
of images, transformers can leverage their attention mechanism to capture
long-range dependencies across frames (Liu et al., 2021b; Lin et al., 2021)
This enables the models to map the temporal progression of events, infer
causal relationships, and grasp the underlying narrative structure within the
video.

The integration of transformers in image and video captioning has yielded
improvements that surpass the performance of traditional encoder-decoder
architectures (Liu et al., 2021a, b). Transformers not only generate
semantically accurate captions but also exhibit enhanced coherence and
contextual relevance. Their attention mechanism allows for a incorporation of
local and global context, enabling a more holistic understanding of the visual
content and facilitating caption generation.

The use of transformers in image and video captioning holds great promise, but
there are challenges to be addressed. Efficient fusion of visual features with
textual information (Radford et al., 2021), handling long-term dependencies in
video sequences (Liu et al., 2021b), and ensuring temporal coherence in
caption generation for longer videos are among the ongoing research topics.

###  5.2. Visual Question Answering

Visual Question Answering (VQA) combines computer vision and natural language
processing to enable the extraction of information from visual content,
identifying objects, scenes, actions, and comprehending context, and provide
meaningful answers to questions about that content. Both VQA and VSG involve
understanding visual content and generating meaningful responses and the
developments in VQA can be directly applied in VSG tasks. Through the use of
VQA techniques, VSG models can gain valuable insights into the content and
context of images and videos, improving the overall quality of their visual
narratives.

Encoder-classifier deep-learning VQA models encode visual and textual inputs
and select an answer from a predefined set using a classification process.
Image encoders, typically pretrained CNNs, extract visual features from the
input images. Question encoders, often RNNs, processes the input questions,
converting them into fixed-length latent representations. Fusion techniques
combine the visual and textual features, creating a fused representation that
captures their interactions that is then classified using fully connected
layers, mapping them to possible answer classes. Traditional VQA models are
trained using supervised learning with large-scale datasets, where image-
question-answer triplets are provided as training examples. The parameters are
optimized to minimize the loss between predicted answers and ground truth
answers during training. Some examples are BUTD (Anderson et al., 2017), that
uses an attention-based model that combines bottom-up and top-down mechanisms
to improve performance in image captioning and visual question answering
tasks, and “Show, Ask, Attend, and Answer” (Kazemi and Elqursh, 2017), that
combines LSTMs with multiple attention layers.

The transition from encoder-classifier methods to transformer methods in VQA
was driven by the transformer’s ability to capture long-range dependencies and
global contextual information. Transformers offered the potential to more
efficiently capture the interactions between visual and textual modalities,
enabling a more comprehensive understanding of the relationship between images
and questions. In these VQA models, images and questions are encoded using
transformer encoders and self-attention mechanisms capture the dependencies
between words and the visual features, allowing the model to focus on relevant
image regions while processing the question. The fused encoded representations
are then fed into classification layers to generate answers (Bao et al., 2022;
Wang et al., 2023a, 2022, b). Recent advances in VQA also explore new models
that approach the task as a generation problem rather than a classification
one, allowing for a more flexible and diverse range of answers (Chen et al.,
2022).

##  6\. Visual Story Generation

The current state-of-the-art and baseline methods employed in VSG have evolved
from their origins in textual story generation. These methods have served as
foundations for the development of more sophisticated techniques and continue
to provide a comparative baseline for new models and approaches.

###  6.1. Visual Storytelling with Convolutional and Recurrent Neural Networks

Huang et al. (Huang et al., 2016) proposed a dataset of sequential vision-to-
language mappings, SIND v.1 (later renamed to VIST), with 81,743 images in
20,211 sequences, with both descriptive (captions) and story language, and
aiming at facilitating research, from understanding individual images to
comprehending image sequences that narrate events over time.

This work also proposed models for generating stories based on image
sequences, using a sequence-to-sequence RNN approach, which extends the image
captioning technique. It extracts features from each image using a pretrain
CNN called AlexNET (Krizhevsky et al., 2012). It then encodes image sequences
by using a GRU over the features of each individual image. The story decoder
model consists of another GRU and generates narratives word by word. Four
different decoding methods, shown in Table 3, were considered: beam search,
greedy search, greedy search with removal of duplicates, and greedy search
with removal of duplicates and incorporation of grounded words.

Decoding Method | METEOR
---|---
Beam=10 | 23.13
Greedy (Beam=1) | 27.76
Greedy - Duplicates | 30.11
Greedy - Duplicates + Grounded | 31.42
Table 3. Results of each decoding approach using evaluated using METEOR scores
on the VIST (Huang et al., 2016) dataset.

Results indicate that beam search alone does not perform well in generating
high-quality stories. This contrasts with the results for image captioning,
where beam search is more effective than greedy search. The reason is that
generating stories with beam search results in generic high-level descriptions
such as “This is a picture of a dog” originating from the label bias problem.
On the other hand, by using a beam size of 1 (greedy search) there is an
improvement in the quality of the generated stories.

Since stories generated using a greedy search may sometimes contain repeated
words or phrases that can adversely affect textual quality, a duplicate-
removal heuristic was added to ensure that content words cannot be produced
more than once on a given story. Finally, to further improve quality, the
previous approach was extended to include “visually grounded” words. These are
words that are authorized for use by the story model only if they are licensed
by the caption model: for a word to be included in the story, it must have
appeared with some frequency in the captions generated by the caption model
for the sequence of images. Fig. 7 shows an example of the generated stories
for an image sequence.

Figure 7. Sequence of images used to generate stories using different decoding
methods. Beam search: This is a picture of a family. This is a picture of a
cake. This is a picture of a dog. This is a picture of a beach. This is a
picture of a beach. Greedy: The family gathered together for a meal. The food
was delicious. The dog was excited to be there. The dog was enjoying the
water. The dog was happy to be in the water. Greedy - duplicates: The family
gathered together for a meal. The food was delicious. The dog was excited to
be there. The kids were playing in the water. The boat was a little too much
to drink. Greedy - duplicates + grounded words: The family got together for a
cookout. They had a lot of delicious food. The dog was happy to be there. They
had a great time on the beach. They even had a swim in the water. Adapted from
(Huang et al., 2016).

###  6.2. Visual Storytelling with Convolutional and Bidirectional Recurrent
Networks Networks

Park et al. (Park and Kim, 2015) proposed Coherent Recurrent Convolutional
Network (CRCN), integrating various neural network components to bridge the
gap between visual content and textual narrative: VGGNet (Simonyan and
Zisserman, 2015) as image encoder and Bidirectional Recurrent Networks (BRNN)
to capture contextual information from both the forward and backward
directions within a given text sequence. CRCN also introduces a local
coherence model that focuses on maintaining coherence and context within the
generated sentence sequences. It also utilizes parse trees to detect
grammatical roles and structural attributes, ensuring that the generated
sentences integrate with the broader narrative context. During training,
correctly aligned image-sentence pairs are given higher scores than misaligned
pairs.

The evaluation of CRCN considered several automatic metrics and several
baseline works, including one that excludes the entity coherence model. Table
4 shows the results for the NY City (Park and Kim, 2015) and Disneyland (Park
and Kim, 2015) datasets. Both datasets were introduced by Park et al. (Park
and Kim, 2015) and consist of image sequences with the corresponding blog
posts. In these datasets, results were superior to all baselines. The
exception, where it was a close second, was with the baseline without entity
coherence and in only certain metrics. The work was further validated by user
studies conducted via Amazon Mechanical Turk (AMT), showing that human
evaluators prefer CRCN-generated sequences over several baselines.

Metric | B-1 | B-2 | B-3 | B-4 | C | M | R@1 | R@5 | R@10
---|---|---|---|---|---|---|---|---|---
NY City | 26.83 | 5.37 | 2.57 | 2.08 | 30.9 | 7.69 | 11.67 | 31.19 | 43.57
Disneyland | 28.40 | 6.88 | 4.11 | 3.49 | 52.7 | 8.78 | 14.29 | 31.29 | 43.20
Table 4. Performance Metrics for CRCN on New York City and Disneyland
Datasets. Bold values represent state-of-the-art results at the time. B stands
for BLEU, C for CIDEr, M for METEOR and R for Recall (Park and Kim, 2015).

Fig. 8 shows an example of a sequence of images and the corresponding
generated sentences.

Figure 8.  Sequence of images used by CRCN a story. (1) One of the hallway
arches inside of the library (2) As we walked through the library I noticed an
exhibit called lunch hour nyc it captured my attention as I had also taken a
tour of nyc food carts during my trip (3) Here is the top of the Chrysler
building everyone’s favorite skyscraper in new york. (4) After leaving the
nypl we walked along 42nd st. (5) We walked down fifth avenue from rockefeller
centre checking out the windows in saks the designer stores and eventually
making our way to the impressive new york public library. Adapted from (Park
and Kim, 2015).

Besides, Park et al. (Park and Kim, 2015) which is the first approach to use a
bidirectional RNN for visual storytelling, to the best of our knowledge, XE
(Wang et al., 2018) and AREL (Wang et al., 2018) have also proposed models for
visual storytelling that use bidirectional RNNs on the VIST dataset.

###  6.3. Transitional Adaptation of Pretrained Models for Visual Storytelling

Yu et al. (Yu et al., 2021) proposed Transitional Adaptation of Pretrained
Model (TAPM), aimed at refining the generation of textual descriptions for
visual content, particularly in the context of VSG tasks. It aims at bridging
the gap between pretrained language models and visual encoders. In contrast
with previous models for vision-to-language generation tasks, which typically
pretrain a visual encoder and a language generator separately and then jointly
fine-tune them for the target task, TAPM proposes a transitional adaptation
task to harmonize the visual encoder and language model for downstream tasks
like VSG, mitigating discord between visual specificity and language fluency
arising from separate training on large corpora of visual and text data. TAPM
introduces an approach that adapts multimodal modules through a simpler
alignment task focusing solely on visual inputs, eliminating the need for text
labels.

TAPM components aim at enhancing the quality of textual descriptions for
visual content in storytelling tasks. They are a visual encoder, a language
generator, adaptation training, sequential coherence loss, training with
adaptation loss, and a fine-tuning and inference process. The visual encoder,
a pretrained model, extracts features from images or videos. In TAPM, it
becomes integral during the adaptation phase, where it integrates with the
language generator to fuse visual and textual information. The pretrained
language generator model is responsible for converting visual information into
textual descriptions. During the adaptation phase, it generates video and text
embeddings, aligning textual representations with the corresponding visual
features, based on a sequential coherence loss function.

The loss function divides sequential coherence into three components: past,
current, and future matching losses. The past matching loss uses a Fully
Connected (FC) layer $f_{p}$ to project the text representation $b_{s_{i}}$ of
video $i$, drawing it nearer to the visual representation $v_{b_{i-1}}$ of the
preceding video $i-1$ and distancing it from those of non-sequential videos.
The future matching loss projects $b_{s_{i}}$ via a distinct FC layer $f_{f}$,
aligns with the subsequent visual representation $v_{b_{i+1}}$. The current
matching loss then aligns the current visual representation $v_{b_{i}}$ with
$b_{s_{i}}$ through another FC layer $f_{c}$. These components are unified by
the FC layer projections in their respective visual spaces, pulling the
embeddings of correct matches closer and pushing incorrect matches further
apart. Margin ranking losses are utilized to implement this concept,
contrasting correct matches with incorrect ones. The final sequential
coherence loss for a given video $i$ is formulated as shown in Eq. 15, where
$\cos$ is the cosine similarity, and $j$ represents indices of incorrect
matches.

(15) $\displaystyle L_{i}=$ $\displaystyle\sum_{j\neq
i-1}\max(0,1+\cos(v_{b_{j}},f_{p}(b_{s_{i}}))-\cos(v_{b_{i-1}},f_{p}(b_{s_{i}})))$
$\displaystyle+$ $\displaystyle\sum_{j\neq
i}\max(0,1+\cos(v_{b_{j}},f_{c}(b_{s_{i}}))-\cos(v_{b_{i}},f_{c}(b_{s_{i}})))$
$\displaystyle+$ $\displaystyle\sum_{j\neq
i+1}\max(0,1+\cos(v_{b_{j}},f_{f}(b_{s_{i}}))-\cos(v_{b_{i+1}},f_{f}(b_{s_{i}}))),$

TAPM uses a split-training strategy to optimize model performance. Initially,
the visual encoder undergoes adaptation training with the adaptation loss,
while the text encoder and language generator remain fixed. Subsequently, all
components are jointly updated with the generation loss, allowing the model to
optimize the adaptation task before addressing the more challenging generation
objective. After the adaptation and split-training phases, TAPM undergoes
fine-tuning. The model is then ready for the inference phase, generating
captions for unseen visual inputs.

Fig. 9(a) shows results for TAPM against selected baselines on the LSMDC 2019
and VIST datasets. Fig. 9(b) and 9(c) show human evaluation results in which
TAPM surpasses adversarial baselines on LSMDC 2019. On VIST it surpasses the
XE (Wang et al., 2018) and AREL (Wang et al., 2018) baselines in relevance,
expressiveness, and concreteness. These results highlight the strengths in
word choice and contextual accuracy, showcasing its ability to capture causal
relationships between images. However, the score is still far from human
performance, indicating that there is still room for improvement.

Models | Public Test | Blind Test
---|---|---
| C | R | C | R
Official Baseline (Park et al., 2018) | 7.0 | 12.0 | 6.9 | 11.9
XE (Wang et al., 2018) | 7.2 | 11.5 | - | -
AREL (Wang et al., 2018) | 7.3 | 11.4 | - | -
TAPM | 10.0 | 12.3 | 8.8 | 12.4
(a) Results on LSMDC 2019 public and blind test set. C stands for CIDEr and R
for ROUGE.

Models | Scores
---|---
Human | 1.085
Official Baseline (Park et al., 2018) | 4.015
TAPM | 3.670
(b) Human evaluation results on the LSMDC 2019 blind test set according to a
Likert scale from 5 (worst) to 1 (best), where lower is better.

|  TAPM vs XE |  TAPM vs AREL
---|---|---
Choice (%) | TAPM | XE | Tie | TAPM | AREL | Tie
Relevance | 59.9 | 34.1 | 6.0 | 61.3 | 32.8 | 5.9
Expressiveness | 57.3 | 32.3 | 10.4 | 57.3 | 34.0 | 8.7
Concreteness | 59.1 | 30.3 | 10.7 | 59.6 | 30.4 | 10.0
(c) Human evaluation results on VIST. Higher is better.

Figure 9. TAPM evaluation results. Adapted from (Yu et al., 2021).

###  6.4. Combining LLMs (GPT-2) and Character Features

Hong et al. (Hong et al., 2023) proposed CharGrid (Character-Grid
Transformer), a transformer model that integrates diverse input features, from
global image attributes to character-specific details. It takes a sequence of
input tokens, including global image features obtained using Swin
Transformer’s (Liu et al., 2021a), character features extracted by cropping
character instances, a character grid for coherence assessment, and text
features introduced incrementally during processing. The trainable image and
character encoders process the image feature inputs. The character grid is
flattened and fed to a FC layer. The transformer module then processes these
inputs, generating an output as a probability distribution over potential
tokens using a pre-trained GPT-2 tokenizer. The model architecture is shown in
Fig. 10.

Figure 10.  CharGrid architecture: the blue boxes are pre-trained components
where the parameters are fixed. Adapted from (Hong et al., 2023).

During training, model parameters, excluding the vision backbone, are randomly
initialized. The training objective involves maximizing the likelihood of
image sequence-story pairs through backpropagation. Nucleus sampling (Holtzman
et al., 2019) is employed for validation, and the METEOR score is used for
evaluation.

CharGrid has two additional variants: ObjGrid (Object-Grid Transformer) and
EntiGrid (Entity-Grid Transformer). ObjGrid replaces character features with
object features. This model incorporates global features and object features,
with an object grid representing coherence based on the similarity between
global image features and object features. It is similar to CharGrid but
includes both character and object features. EntiGrid incorporates global,
character, and object features, with an entity grid representing coherence
through the similarity between global image features and both character and
object features. These variants explore the impact of different input feature
combinations on model performance. CharGrid, with its emphasis on character
coherence, serves as the primary model, while ObjGrid and EntiGrid address the
contributions of object-related features.

The results in Table 5 show that CharGrid outperforms TAPM with character
features and GPT-2 with character features on various metrics, emphasizing the
effectiveness of character grid representations for coherence in VSG. A crowd-
sourcing experiment was conducted with 28 workers to obtain binary judgments
on grammaticality, coherence, diversity, and visual groundedness of the
generated stories. The results in Table 6 show that TAPM with character
features excels in visual groundedness over plain TAPM, while CharGrid
surpasses TAPM with character features across all metrics. Two-sided binomial
tests support that the character grid representation yields superior stories,
affirming the reference-based metrics findings.

Model | B-1 | B-2 | B-3 | B-4 | M | R-L | C
---|---|---|---|---|---|---|---
GPT-2 | 38.65 | 20.28 | 9.78 | 4.68 | 31.64 | 24.24 | 1.66
GPT-2 + obj | 40.65 | 21.35 | 10.2 | 4.87 | 31.69 | 24.05 | 1.85
GPT-2 + char | 39.95 | 21.04 | 10.11 | 4.92 | 31.85 | 24.19 | 1.57
GPT-2 + obj,char | 40.41 | 21.44 | 10.56 | 5.06 | 32.03 | 24.38 | 1.87
TAPM | 39.85 | 21.7 | 10.72 | 5.19 | 32.38 | 25.09 | 1.48
TAPM \+ obj | 40.86 | 22.13 | 10.83 | 5.25 | 32.34 | 24.91 | 1.82
TAPM \+ char | 40.03 | 21.68 | 10.66 | 5.18 | 32.42 | 24.88 | 1.4
TAPM \+ obj,char | 40.87 | 21.99 | 10.72 | 5.06 | 32.48 | 24.87 | 1.59
ObjGrid + obj | 47.66 | 25.26 | 11.95 | 5.42 | 32.83 | 24.42 | 4.68
EntityGrid + obj,char | 45.83 | 24.85 | 12.11 | 5.7 | 32.68 | 24.89 | 3.53
CharGrid + char | 47.71 | 25.33 | 11.95 | 5.42 | 33.03 | 25.01 | 4.83
Table 5. Results using different input features on the test set of VWP. Global features are always included. “+ obj” and “+ char” represent that object and character features are also included, respectively. BLEU (B), METEOR (M), ROUGE-L (R-L), and CIDEr (C) values are averages of three runs with different random seeds. Adapted from (Hong et al., 2023). Model | Grammaticality | Coherence | Visual Groundedness | Diversity
---|---|---|---|---
TAPM \+ char vs. TAPM | +2.45 | +1.99 | +3.99* | +1.69
CharGrid vs. TAPM \+ char | +6.49** | +8.41** | +6.25* | +11.06**
Table 6. Human binary judgments in percentage of generated stories between
TAPM and TAPM with character features (TAPM \+ char), TAPM \+ char and
CharGrid on the test set of VWP across four criteria: Grammaticality,
Coherence, Visually Groundedness, and Diversity. The numbers are percentages.
* means p-value $<$ 0.05. ** means p-value $<$ 0.01. Adapted from (Hong et
al., 2023).

###  6.5. Visual Story Generation using Graphs for Event Ratiocination

Zheng et al. (Zheng et al., 2021) introduced Hypergraph-Enhanced Graph
Reasoning (HEGR) for “visual event ratiocination”, a task that involves
generating interpretations and narratives for events that precede, coincide
with, or follow a visual scene, as well as discerning the intents of depicted
characters. The work addresses several downstream tasks, including VSG. The
work introduces a framework to improve the integration and interpretation of
visual and textual data, using hypergraphs to address challenges in multimodal
interactions and temporal dynamics. Figure 11 shows this architecture.

Figure 11.  Architecture of HEGR by Zheng et al. Adapted from (Zheng et al.,
2021).

The method begins with feature extraction. Visual features are extracted using
Mask R-CNN (He et al., 2017) to detect objects and their spatial
characteristics within images. Textual features are extracted from associated
descriptions or captions using BERT (Devlin et al., 2019), capturing semantic
information. In the second phase, semantic graphs are constructed for each
modality: nodes represent detected features or semantic tokens, while edges
depict relationships within the same modality. To capture dependencies between
modalities and across various temporal segments, hypergraphs are employed,
linking semantic graphs based on contextual and temporal relevance, forming a
high-dimensional interaction map. In the third phase, graph convolution is
used to update node representations within each graph, allowing contextual
information aggregation through learned adjacency matrices. Hyper-graph
convolution extends this to include higher-order relationships across multiple
graphs and temporal instances. In the fourth phase, a self-attention mechanism
refines graph representations by dynamically adjusting the significance of
nodes within and across modalities. This process focuses on the most relevant
features for narrative generation. Finally, the event ratiocination generator
uses the enriched feature representations to generate narratives using BART
(Lewis et al., 2019).

The method was evaluated on the VIST dataset, achieving state-of-the-art
performance in terms of BLEU-4, ROUGE-L and METEOR scores achieving 16.4,
37.4, and 37.8, respectively.

###  6.6. Discussion

The works presented in this section have made substantial contributions to the
field of VSG, establishing robust benchmarks for the assessment of emerging
models and methodologies.

Specifically, the first three techniques, presented in Sections 6.1, 6.2, and
6.3 employed a CNN encoder for visual feature extraction and a RNN decoder for
language generation. The methods detailed in Sections 6.1 and 6.3, offer
comparative analyses on the VIST dataset, with METEOR scores of 31.4 and 37.2,
respectively. Notably, the latter technique surpasses the former, which can be
attributed to its transitional training approach.

The study outlined in Section 6.4, CharGrid, does not present findings on the
VIST dataset. However, this method is benchmarked against the method detailed
in Section 6.3, TAPM, on the VWP dataset, where it demonstrates superior
performance across all evaluated metrics. This comparison is shown in Table 5,
specifically within the rows labeled “TAPM \+ obj,char” and “CharGrid”. This
method differentiates itself from the previous by focusing on the extraction
of object-level features from visual inputs rather than employing a CNN to
encode the entire visual input. Furthermore, it leverages the capabilities of
an LLM, using its extensive world knowledge to enhance language modeling,
illustrating a novel integration of visual and linguistic elements in story
generation.

Finally, the method discussed in Section 6.5, HEGR, introduces a different
approach that uses hypergraphs to model the contextual and temporal
relationships between visual and textual data. This approach is explicitly
able to model some story elements, namely the characters and the setting.

Table 7 summarizes the performance of the methods described above, along with
other methods from the literature that contributed to the field of VSG. The
listed models range from earlier methods using CNNs and RNNs to more recent
advances that integrate object detection models like Faster R-CNN (Ren et al.,
2015) and Mask-RCNN (He et al., 2017), and LLMs like GPT-2 (Radford et al.,
2019) and BART (Lewis et al., 2019). The first two methods, Park et al. (2015)
(Park and Kim, 2015) and Park et al. (2019) (Park et al., 2018) are not
comparable to the other methods as they do not use the VIST dataset. The
remaining methods are evaluated on the VIST dataset with the exception of
CharGrid (2023) (Hong et al., 2023) that is evaluated on the VWP dataset. The
authors of CharGrid trained and evaluated the performance of TAPM on the VWP
(Hong et al., 2023) dataset, making the two methods directly comparable and it
is likely that CharGrid would perform slightly better than TAPM on the VIST
dataset. It is however not possible to know if CharGrid would perform better
than HEGR on the VIST dataset without further experiments.

Model | Encoding | Decoding | Dataset | B-4 | R-L | M
---|---|---|---|---|---|---
Park et al. (2015) (Park and Kim, 2015) | CNN, Bi-RNN | RNN | Blog Posts | 3.49 | - | 8.78
Park et al. (2019) (Park et al., 2018) | Resnet-101/152, LSTM | LSTM | ActivityNet | 9.91 | - | 16.48
Huang et al. (2015) (Huang et al., 2016) | CNN, GRU | GRU | VIST | - | - | 31.4
Xu et al. (2015) (Xu et al., 2015) | VGGNet | LSTM | VIST | - | 28.94 | 32.98
Yu et al. (2017) (Yu et al., 2017) | ResNet-101, Bi-GRU | GRU | VIST | - | 29.53 | 34.12
CST (2018) (Gonzalez-Rico and Pineda, 2018) | CNN, LSTM | LSTM | VIST | 12.7 | 29.2 | 34.4
AREL (2018) (Wang et al., 2018) | ResNet-152, Bi-GRU | GRU | VIST | 14.1 | 29.5 | 35.0
HSRL (2019) (Huang et al., 2018) | ResNet-152 | LSTM | VIST | 12.32 | 30.84 | 35.23
GLACNet (2019) (Kim et al., 2018) | ResNet-152, Bi-LSTM | LSTM | VIST | - | - | 30.63
StoryAnchor (2020) (Modi and Parde, 2019) | ResNet-152, Bi-GRU | GRU | VIST | 14.0 | 30.0 | 35.5
INET (2020) (Jung et al., 2020) | CNN, GRU | GRU | VIST | 14.7 | 29.7 | 35.5
AOG-LSTM (2023) (Liu et al., 2023) | RestNet-152 | LSTM | VIST | 12.9 | 30.1 | 36.0
CoVS (2023) (Gu et al., 2023) | RestNet-152, GRU | GRU | VIST | 15.2 | 30.8 | 36.5
TAPM (2021) (Yu et al., 2021) 1 | Faster R-CNN, ResNet-152 | GPT-2 | VIST | - | 33.1 | 37.2
HEGR (2021) (Zheng et al., 2021) 1,2 | Mask-RCNN, BERT | BART | VIST | 16.4 | 37.4 | 37.8
TAPM (2021) (Hong et al., 2023) | Faster R-CNN, ResNet-152 | GPT-2 | VWP | 5.19 | 25.09 | 32.38
CharGrid (2023) (Hong et al., 2023)2 | Mask-RCNN, SwinTransformer | GPT-2 | VWP | 5.42 | 25.01 | 33.03
Table 7.  Analysis of Visual Story Generation works. The columns “B-4”, “R-L”,
and “M” represent the BLEU-4, ROUGE-L, and METEOR scores, respectively. All
works, except Huang et al. (2015) (Huang et al., 2016) and CST (2018)
(Gonzalez-Rico and Pineda, 2018), use attention mechanisms. All works use
images as input, except ActivityNet (Fabian Caba Heilbron and Niebles, 2015)
(it uses videos). The mark 1 indicates works that used extra training data
(direct comparison may not be fair). The mark 2 indicated works that
explicitly model some story elements, namely the characters by the detection
of objects in the case of CharGrid (2023) and HEGR (2021) and the setting in
the case of HEGR (2021).

##  7\. Conclusions

This survey reviewed the field of story generation, explaining its development
and current methods. It started by explaining basic story elements and early
techniques, which mostly involved manually created rules. We looked at the
datasets and evaluation metrics used to judge the quality of the stories. The
discussion pointed out the need for better metrics to assess complex tasks
like story generation accurately. The document also covered related areas like
image and video captioning and VQA, discussing how improvements in these
fields have helped enhance VSG.

The survey also looked at the latest developments in VSG and discussed various
models that use deep learning. It started with basic models using simple
encoder-decoder setups and went on to more advanced systems that use
transformers. Each model was described with a focus on how they handle issues
like making the story coherent, keeping it relevant to the context, and
combining detailed visual and textual information.

Despite the advances, challenges remain for VSG. In the following subsections,
we discuss some of the key challenges and opportunities in the field that
could guide future research directions. We start with the ones that are more
concrete and could influence the field in the short term and end with the ones
that are more abstract and could potentially influence the field in the long
term.

###  7.1. Exploring Large Language Models

The field of VSG using LLMs is under-explored, with only a few works delving
into this domain. To the best of our knowledge the existing works
predominantly rely on GPT-2, which is an outdated LLM when compared to models
such as Mixtral (team, 2023) which is the best performing open-source model on
MT-Bench or GPT-4 which is the best performing proprietary model. The scarcity
of the integration of LLMs with visual elements presents an opportunity for
further research, exploring the potential of these models in generating more
coherent and contextually relevant stories.

###  7.2. Automatic Evaluation Metrics

The need for robust and efficient automatic evaluation metrics is emphasized
in our survey. While existing metrics like METEOR and CIDEr have been widely
adopted, they do not fully capture the nuances of story quality. The metrics
mostly focus on the similarity between the generated text and the reference
text. A story can be retold in various ways, employing different perspectives
while preserving the original message. Consequently, a generated text might
deviate significantly from the reference text in terms of specific wording and
still deliver the same story in a coherent way. Thus, while these metrics
provide valuable insights into certain dimensions of text generation, they did
not fully encompass the multifaceted nature of storytelling. As LLMs continue
to improve, they show potential in replacing human annotators in many tasks
(Gilardi et al., 2023) and so metrics to evaluate stories could be developed
based on LLMs. We advance that these metrics could be based on the existing
systematic scoring systems (Zheng et al., 2023) that were developed to
evaluate the performance of LLM-based chat bots.

###  7.3. Decomposition of Story Generation

Addressing the complexity of stories, we underscore the importance of
decomposing the story generation process into modular components. Each module
can focus on specific attributes such as author goals, character interactions,
consistency, etc. This decomposition has already been explored with modeling
characters in CharGrid (Yu et al., 2021) and the modeling of characters and
setting in HEGR (Zheng et al., 2021). We believe that explicit modeling of
other story elements and abstract concepts such as emotions and culture,
possibly using hypergraphs or similar structures could further enhance the
quality of generated stories.

###  7.4. Hybrid Systems and Knowledge Integration

Building on the decomposition concept, hybrid systems, combining planning-
based techniques and machine learning approaches, present an intriguing
direction. Hyper-graphs, as seen in HEGR (Zheng et al., 2021), could possibly
be a starting point as a deep learning approach to model the planning process
replacing the traditional handcrafted planning systems. The integration of
multiple knowledge sources can enrich the diversity and quality of generated
stories, paving the way for more robust and context-aware narrative creation.

###### Acknowledgements.

Daniel A. P. Oliveira is supported by a scholarship granted by Fundação para a
Ciência e Tecnologia (FCT), with reference 2021.06750.BD. Additionally, this
work was supported by Portuguese national funds through FCT, with reference
UIDB/50021/2020.

## References

  * (1)
  * Abbott (2008) H. Porter Abbott. 2008\.  _The Cambridge Introduction to Narrative_ (2nd ed.).  Cambridge University Press.  https://doi.org/10.1017/CBO9780511816932
  * Akoury et al. (2020) Nader Akoury, Shufan Wang, Josh Whiting, Stephen Hood, Nanyun Peng, and Mohit Iyyer. 2020\.  STORIUM: A Dataset and Evaluation Platform for Machine-in-the-Loop Story Generation. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (Eds.). Association for Computational Linguistics, Online, 6470–6484.  https://doi.org/10.18653/v1/2020.emnlp-main.525
  * Ammanabrolu et al. (2019) Prithviraj Ammanabrolu, Ethan Tien, Wesley Cheung, Zhaochen Luo, William Ma, Lara J. Martin, and Mark O. Riedl. 2019\.  Story Realization: Expanding Plot Events into Sentences.  _ArXiv_ abs/1909.03480 (2019).  https://api.semanticscholar.org/CorpusID:202537394
  * Anderson et al. (2017) Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2017.  Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering. In _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 6077–6086.  https://api.semanticscholar.org/CorpusID:3753452
  * Anthropic (2024a) Anthropic. 2024a.  Claude-2: Overview and Specifications.  https://www.anthropic.com/index/claude-2.  Accessed: 2024-01-03.
  * Anthropic (2024b) Anthropic. 2024b.  Claude-2.1: Overview and Specifications.  https://www.anthropic.com/index/claude-2-1.  Accessed: 2024-01-03.
  * Anthropic (2024) Anthropic. 2024\.  Introducing Claude.  https://www.anthropic.com/index/introducing-claude.  Accessed: 2024-01-03.
  * Aristotle et al. (1997) Aristotle, G. Whalley, J. Baxter, and P. Atherton. 1997\.  _Aristotle’s Poetics_.  McGill-Queen’s University Press.  https://books.google.pt/books?id=14gTwJMEl7UC
  * Banerjee and Lavie (2005) Satanjeev Banerjee and Alon Lavie. 2005.  METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments. In _Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization_ , Jade Goldstein, Alon Lavie, Chin-Yew Lin, and Clare Voss (Eds.). Association for Computational Linguistics, Ann Arbor, Michigan, 65–72.  https://aclanthology.org/W05-0909
  * Bao et al. (2022) Hangbo Bao, Wenhui Wang, Li Dong, Qiang Liu, Owais Khan Mohammed, Kriti Aggarwal, Subhojit Som, Songhao Piao, and Furu Wei. 2022.  VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts. In _Advances in Neural Information Processing Systems_ , S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Vol. 35. Curran Associates, Inc., 32897–32912.  https://proceedings.neurips.cc/paper_files/paper/2022/file/d46662aa53e78a62afd980a29e0c37ed-Paper-Conference.pdf
  * Bolaños et al. (2015) Marc Bolaños, Mariella Dimiccoli, and Petia Radeva. 2015\.  Toward Storytelling From Visual Lifelogging: An Overview.  _IEEE Transactions on Human-Machine Systems_ 47 (2015), 77–90.  https://api.semanticscholar.org/CorpusID:499315
  * Campbell (2008) J. Campbell. 2008\.  _The Hero with a Thousand Faces_.  New World Library.  https://books.google.pt/books?id=I1uFuXlvFgMC
  * Chen et al. (2022) Xi Chen, Xiao Wang, Soravit Changpinyo, A. J. Piergiovanni, Piotr Padlewski, Daniel M. Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish V. Thapliyal, James Bradbury, Weicheng Kuo, Mojtaba Seyedhosseini, Chao Jia, Burcu Karagol Ayan, Carlos Riquelme, Andreas Steiner, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, and Radu Soricut. 2022.  PaLI: A Jointly-Scaled Multilingual Language-Image Model.  _ArXiv_ abs/2209.06794 (2022).  https://api.semanticscholar.org/CorpusID:252222320
  * Choi et al. (2016) YunSeok Choi, SuAh Kim, and Jee-Hyong Lee. 2016.  Recurrent Neural Network for Storytelling. In _2016 Joint 8th International Conference on Soft Computing and Intelligent Systems (SCIS) and 17th International Symposium on Advanced Intelligent Systems (ISIS)_. 841–845.  https://doi.org/10.1109/SCIS-ISIS.2016.0182
  * Chung et al. (2014) Junyoung Chung, Çaglar Gülçehre, Kyunghyun Cho, and Yoshua Bengio. 2014.  Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling.  _ArXiv_ abs/1412.3555 (2014).  https://api.semanticscholar.org/CorpusID:5201925
  * Das et al. (2013) Pradipto Das, Chenliang Xu, Richard Doell, and Jason Corso. 2013\.  A Thousand Frames in Just a Few Words: Lingual Description of Videos through Latent Topics and Sparse Object Stitching.  _Proceedings / CVPR, IEEE Computer Society Conference on Computer Vision and Pattern Recognition. IEEE Computer Society Conference on Computer Vision and Pattern Recognition_ , 2634–2641.  https://doi.org/10.1109/CVPR.2013.340
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _North American Chapter of the Association for Computational Linguistics_.  https://api.semanticscholar.org/CorpusID:52967399
  * DiBattista (2011) M. DiBattista. 2011\.  _Novel Characters: A Genealogy_.  Wiley, 14–20.  https://books.google.pt/books?id=5uATBJUxMqEC
  * Dibell (1999) A. Dibell. 1999\.  _Elements of Fiction Writing - Plot_.  F+W Media, 5–6.  https://books.google.pt/books?id=vEpjuQEACAAJ
  * Dietrich and Sundell (1983) R.F. Dietrich and R.H. Sundell. 1983.  _The Art of Fiction_.  Holt, Rinehart and Winston.  https://books.google.pt/books?id=iSD900urA8QC
  * Ding et al. (2023) Jiayu Ding, Shuming Ma, Li Dong, Xingxing Zhang, Shaohan Huang, Wenhui Wang, and Furu Wei. 2023.  LongNet: Scaling Transformers to 1,000,000,000 Tokens.  _ArXiv_ abs/2307.02486 (2023).  https://api.semanticscholar.org/CorpusID:259341682
  * Fabian Caba Heilbron and Niebles (2015) Bernard Ghanem Fabian Caba Heilbron, Victor Escorcia and Juan Carlos Niebles. 2015\.  ActivityNet: A Large-Scale Video Benchmark for Human Activity Understanding. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 961–970.
  * Fan et al. (2018) Angela Fan, Mike Lewis, and Yann Dauphin. 2018.  Hierarchical Neural Story Generation. In _Annual Meeting of the Association for Computational Linguistics_.  https://api.semanticscholar.org/CorpusID:44134226
  * Fitzgerald and Reynolds (1993) F.S. Fitzgerald and G. Reynolds. 1993.  _The Great Gatsby_.  Wordsworth.  https://books.google.pt/books?id=HestSXO362YC
  * Freytag and MacEwan (1894) G. Freytag and E.J. MacEwan. 1894.  _Freytag’s Technique of the Drama_.  Scott, Foresman.  https://books.google.pt/books?id=nD8PAQAAMAAJ
  * Gervas (2009) Pablo Gervas. 2009\.  Computational Approaches to Storytelling and Creativity.  _AI Magazine_ 30, 3 (Jul. 2009), 49\.  https://doi.org/10.1609/aimag.v30i3.2250
  * Gervás (2013) Pablo Gervás. 2013\.  Propp’s Morphology of the Folk Tale as a Grammar for Generation. In _Workshop on Computational Models of Narrative_.  https://api.semanticscholar.org/CorpusID:18856498
  * Gervás et al. (2004) Pablo Gervás, Belén Díaz-Agudo, Federico Peinado, and Raquel Hervás. 2004\.  Story plot generation based on CBR. In _Knowledge-Based Systems_.  https://api.semanticscholar.org/CorpusID:2126816
  * Gilardi et al. (2023) Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023\.  ChatGPT outperforms crowd workers for text-annotation tasks.  _Proceedings of the National Academy of Sciences of the United States of America_ 120 (2023).  https://api.semanticscholar.org/CorpusID:257766307
  * Gonzalez-Rico and Pineda (2018) Diana Gonzalez-Rico and Gibran Fuentes Pineda. 2018.  Contextualize, Show and Tell: A Neural Visual Storyteller.  _ArXiv_ abs/1806.00738 (2018).  https://api.semanticscholar.org/CorpusID:46936117
  * Griffith (2010) K. Griffith. 2010\.  _Writing Essays About Literature_.  Cengage Learning, 40.  https://books.google.pt/books?id=wwQJzgEACAAJ
  * Gu et al. (2023) Jinjing Gu, Hanli Wang, and Ruichao Fan. 2023.  Coherent Visual Storytelling via Parallel Top-Down Visual and Topic Attention.  _IEEE Transactions on Circuits and Systems for Video Technology_ 33, 1 (2023), 257–268.  https://doi.org/10.1109/TCSVT.2022.3199603
  * He et al. (2017) Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick. 2017.  Mask R-CNN.  https://api.semanticscholar.org/CorpusID:54465873
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long Short-Term Memory.  _Neural Computation_ 9, 8 (1997), 1735–1780.
  * Holtzman et al. (2019) Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019.  The Curious Case of Neural Text Degeneration.  _ArXiv_ abs/1904.09751 (2019).  https://api.semanticscholar.org/CorpusID:127986954
  * Hong et al. (2023) Xudong Hong, Asad Sayeed, Khushboo Mehra, Vera Demberg, and Bernt Schiele. 2023.  Visual Writing Prompts: Character-Grounded Story Generation with Curated Image Sequences.  _Transactions of the Association for Computational Linguistics_ 11 (06 2023), 565–581.  https://doi.org/10.1162/tacl_a_00553 arXiv:https://direct.mit.edu/tacl/article-pdf/doi/10.1162/tacl_a_00553/2134487/tacl_a_00553.pdf
  * Huang et al. (2018) Qiuyuan Huang, Zhe Gan, Asli Celikyilmaz, Dapeng Oliver Wu, Jianfeng Wang, and Xiaodong He. 2018\.  Hierarchically Structured Reinforcement Learning for Topically Coherent Visual Story Generation. In _AAAI Conference on Artificial Intelligence_.  https://api.semanticscholar.org/CorpusID:29160111
  * Huang et al. (2016) Ting-Hao Kenneth Huang, Francis Ferraro, Nasrin Mostafazadeh, Ishan Misra, Aishwarya Agrawal, Jacob Devlin, Ross Girshick, Xiaodong He, Pushmeet Kohli, Dhruv Batra, C. Lawrence Zitnick, Devi Parikh, Lucy Vanderwende, Michel Galley, and Margaret Mitchell. 2016\.  Visual Storytelling. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Association for Computational Linguistics, San Diego, California, 1233–1239.  https://doi.org/10.18653/v1/N16-1147
  * Hühn et al. (2014) P. Hühn, J.C. Meister, J. Pier, and W. Schmid. 2014\.  _Handbook of Narratology_.  De Gruyter.  https://books.google.pt/books?id=0q_nBQAAQBAJ
  * Jain et al. (2017) Parag Jain, Priyanka Agrawal, Abhijit Mishra, Mohak Sukhwani, Anirban Laha, and Karthik Sankaranarayanan. 2017.  Story Generation from Sequence of Independent Short Descriptions.  _ArXiv_ abs/1707.05501 (2017).  https://api.semanticscholar.org/CorpusID:643611
  * Jung et al. (2020) Yunjae Jung, Dahun Kim, Sanghyun Woo, Kyungsu Kim, Sungjin Kim, and In-So Kweon. 2020\.  Hide-and-Tell: Learning to Bridge Photo Streams for Visual Storytelling. In _AAAI Conference on Artificial Intelligence_.  https://api.semanticscholar.org/CorpusID:211010817
  * Jurafsky and Martin (2023) D. Jurafsky and J.H. Martin. 2023.  _Speech and Language Processing. An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition_.  Youcanprint.  https://books.google.pt/books?id=b3PuzwEACAAJ
  * Jurafsky and Martin (2009) Dan Jurafsky and James H. Martin. 2009.  _Speech and language processing : an introduction to natural language processing, computational linguistics, and speech recognition_.  Pearson Prentice Hall, Upper Saddle River, N.J.  http://www.amazon.com/Speech-Language-Processing-2nd-Edition/dp/0131873210/ref=pd_bxgy_b_img_y
  * Kazemi and Elqursh (2017) Vahid Kazemi and Ali Elqursh. 2017.  Show, Ask, Attend, and Answer: A Strong Baseline For Visual Question Answering.  _ArXiv_ abs/1704.03162 (2017).  https://api.semanticscholar.org/CorpusID:12446195
  * Kim et al. (2017) Kyung-Min Kim, Min-Oh Heo, Seong-Ho Choi, and Byoung-Tak Zhang. 2017.  DeepStory: video story QA by deep embedded memory networks. In _Proceedings of the 26th International Joint Conference on Artificial Intelligence_ (Melbourne, Australia) _(IJCAI’17)_. AAAI Press, 2016–2022.
  * Kim et al. (2018) Taehyeong Kim, Min-Oh Heo, Seonil Son, Kyoung-Wha Park, and Byoung-Tak Zhang. 2018.  GLAC Net: GLocal Attention Cascading Networks for Multi-image Cued Story Generation.  _ArXiv_ abs/1805.10973 (2018).  https://api.semanticscholar.org/CorpusID:44064084
  * Krishna et al. (2017) Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. 2017.  Dense-Captioning Events in Videos. In _2017 IEEE International Conference on Computer Vision (ICCV)_. 706–715.  https://api.semanticscholar.org/CorpusID:1026139
  * Krizhevsky et al. (2012) Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012\.  ImageNet Classification with Deep Convolutional Neural Networks. In _Advances in Neural Information Processing Systems_ , F. Pereira, C.J. Burges, L. Bottou, and K.Q. Weinberger (Eds.), Vol. 25. Curran Associates, Inc.  https://proceedings.neurips.cc/paper_files/paper/2012/file/c399862d3b9d6b76c8436e924a68c45b-Paper.pdf
  * Lau and Chen (2008) Siew-Yung Lau and Chwen Jen Chen. 2008.  Designing a Virtual Reality (VR) Storytelling System for Educational Purposes.  _Technological Developments in Education and Automation_ , 135–138.  https://doi.org/10.1007/978-90-481-3656-8_26
  * Leskovec et al. (2014) J. Leskovec, J. Leskovec, A. Rajaraman, and J.D. Ullman. 2014\.  _Mining of Massive Datasets_.  Cambridge University Press.  https://books.google.pt/books?id=16YaBQAAQBAJ
  * Lewis et al. (2019) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdel rahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2019.  BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. In _Annual Meeting of the Association for Computational Linguistics_.  https://api.semanticscholar.org/CorpusID:204960716
  * Lin (2004) Chin-Yew Lin. 2004\.  ROUGE: A Package for Automatic Evaluation of Summaries. In _Text Summarization Branches Out_. Association for Computational Linguistics, Barcelona, Spain, 74–81.  https://aclanthology.org/W04-1013
  * Lin et al. (2021) Kevin Lin, Linjie Li, Chung-Ching Lin, Faisal Ahmed, Zhe Gan, Zicheng Liu, Yumao Lu, and Lijuan Wang. 2021\.  SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioning. In _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 17928–17937.  https://api.semanticscholar.org/CorpusID:244709307
  * Liu et al. (2023) Hanqing Liu, Jiacheng Yang, Chia-Hao Chang, Wei Wang, Hai-Tao Zheng, Yong Jiang, Hui Wang, Rui Xie, and Wei Wu. 2023.  AOG-LSTM: An adaptive attention neural network for visual storytelling.  _Neurocomputing_ 552 (2023), 126486.  https://doi.org/10.1016/j.neucom.2023.126486
  * Liu et al. (2021a) Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021a.  Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. In _2021 IEEE/CVF International Conference on Computer Vision (ICCV)_. 9992–10002.  https://api.semanticscholar.org/CorpusID:232352874
  * Liu et al. (2021b) Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. 2021b.  Video Swin Transformer. In _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 3192–3201.  https://api.semanticscholar.org/CorpusID:235624247
  * Meehan (1977) James R. Meehan. 1977\.  TALE-SPIN, An Interactive Program that Writes Stories. In _International Joint Conference on Artificial Intelligence_.  https://api.semanticscholar.org/CorpusID:2372981
  * Meta (2024) Meta. 2024.  Introducing Meta Llama 3: The most capable openly available LLM to date.  https://ai.meta.com/blog/meta-llama-3/.  Accessed: 2024-05-28.
  * Mitchell (2008) D. Mitchell. 2008\.  _Cloud Atlas: A Novel_.  Random House Publishing Group.  https://books.google.pt/books?id=-sapXk7dWcAC
  * Modi and Parde (2019) Yatri Modi and Natalie Parde. 2019.  The Steep Road to Happily Ever after: an Analysis of Current Visual Storytelling Models. In _Proceedings of the Second Workshop on Shortcomings in Vision and Language_ , Raffaella Bernardi, Raquel Fernandez, Spandana Gella, Kushal Kafle, Christopher Kanan, Stefan Lee, and Moin Nabi (Eds.). Association for Computational Linguistics, Minneapolis, Minnesota, 47–57.  https://doi.org/10.18653/v1/W19-1805
  * Mostafazadeh et al. (2016) N. Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James F. Allen. 2016\.  A Corpus and Cloze Evaluation for Deeper Understanding of Commonsense Stories. In _North American Chapter of the Association for Computational Linguistics_.  https://api.semanticscholar.org/CorpusID:1726501
  * Olimov et al. (2021) Farrukh Olimov, Shikha Dubey, Labina Shrestha, Tran Trung Tin, and Moongu Jeon. 2021.  Image Captioning using Multiple Transformers for Self-Attention Mechanism.  _ArXiv_ abs/2103.05103 (2021).  https://api.semanticscholar.org/CorpusID:232168674
  * OpenAI (2024a) OpenAI. 2024a.  GPT-3.5: Models Documentation.  https://platform.openai.com/docs/models/gpt-3-5.  Accessed: 2024-01-03.
  * OpenAI (2024b) OpenAI. 2024b.  GPT-4 and GPT-4 Turbo: Models Documentation.  https://platform.openai.com/docs/models/gpt-4-and-gpt-4-turbo.  Accessed: 2024-01-03.
  * OpenAI (2024c) OpenAI. 2024c.  New Models and Developer Products Announced at DevDay.  https://openai.com/blog/new-models-and-developer-products-announced-at-devday.  Accessed: 2024-01-03.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002\.  Bleu: a Method for Automatic Evaluation of Machine Translation. In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , Pierre Isabelle, Eugene Charniak, and Dekang Lin (Eds.). Association for Computational Linguistics, Philadelphia, Pennsylvania, USA, 311–318.  https://doi.org/10.3115/1073083.1073135
  * Park and Kim (2015) Cesc C Park and Gunhee Kim. 2015.  Expressing an Image Stream with a Sequence of Natural Sentences. In _Advances in Neural Information Processing Systems_ , C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (Eds.), Vol. 28. Curran Associates, Inc.  https://proceedings.neurips.cc/paper_files/paper/2015/file/17e62166fc8586dfa4d1bc0e1742c08b-Paper.pdf
  * Park et al. (2018) Jae Sung Park, Marcus Rohrbach, Trevor Darrell, and Anna Rohrbach. 2018.  Adversarial Inference for Multi-Sentence Video Description. In _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 6591–6601.  https://api.semanticscholar.org/CorpusID:55702273
  * Peng et al. (2018) Nanyun Peng, Marjan Ghazvininejad, Jonathan May, and Kevin Knight. 2018.  Towards Controllable Story Generation. In _Proceedings of the First Workshop on Storytelling_. Association for Computational Linguistics, New Orleans, Louisiana, 43–49.  https://doi.org/10.18653/v1/W18-1505
  * Pinault (1992) D. Pinault. 1992\.  _Story-Telling Techniques in the Arabian Nights_.  Brill.  https://books.google.pt/books?id=guHmLGJMbg4C
  * Propp (1968) V. Propp. 1968.  _Second Edition_.  University of Texas Press, New York, USA.  https://doi.org/doi:10.7560/783911
  * Propp et al. (2010) V. Propp, L. Scott, and S. Pirkova-Jakobson. 2010.  _Morphology of the Folk Tale_.  University of Texas Press.  https://books.google.pt/books?id=aY7rtbIOC5EC
  * Pérez y Pérez and Sharples (2001) Rafael Pérez y Pérez and Mike Sharples. 2001.  MEXICA: A computer model of a cognitive account of creative writing.  _J. Exp. Theor. Artif. Intell._ 13 (04 2001), 119–139.  https://doi.org/10.1080/09528130010029820
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021\.  Learning Transferable Visual Models From Natural Language Supervision. In _International Conference on Machine Learning_.  https://api.semanticscholar.org/CorpusID:231591445
  * Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019\.  Language Models are Unsupervised Multitask Learners.  https://api.semanticscholar.org/CorpusID:160025533
  * Ren et al. (2015) Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. 2015\.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 39 (2015), 1137–1149.  https://api.semanticscholar.org/CorpusID:10328909
  * Rohrbach et al. (2016) Anna Rohrbach, Atousa Torabi, Marcus Rohrbach, Niket Tandon, Christopher Joseph Pal, H. Larochelle, Aaron C. Courville, and Bernt Schiele. 2016\.  Movie Description.  _International Journal of Computer Vision_ 123 (2016), 94 – 120.  https://api.semanticscholar.org/CorpusID:18217052
  * Rowling (2015) J.K. Rowling. 2015\.  _Harry Potter and the Sorcerer’s Stone_.  Pottermore Publishing.  https://books.google.pt/books?id=wrOQLV6xB-wC
  * Ryan (2017) James Owen Ryan. 2017\.  Grimes’ Fairy Tales: A 1960s Story Generator. In _International Conference on Interactive Digital Storytelling_.  https://api.semanticscholar.org/CorpusID:3929953
  * Simonyan and Zisserman (2015) Karen Simonyan and Andrew Zisserman. 2015.  Very Deep Convolutional Networks for Large-Scale Image Recognition. In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_ , Yoshua Bengio and Yann LeCun (Eds.).  http://arxiv.org/abs/1409.1556
  * Simpson (2002) J.A. Simpson. 2002\.  _Oxford English Dictionary: Version 3.0 : Upgrade Version_.  Oxford University Press.  https://books.google.pt/books?id=HvjcMgEACAAJ
  * Szeliski (2010) R. Szeliski. 2010\.  _Computer Vision: Algorithms and Applications_.  Springer London.  https://books.google.pt/books?id=bXzAlkODwa8C
  * Szilas (2003) Nicolas Szilas. 2003\.  IDtension: a narrative engine for Interactive Drama. In _Technologies for Interactive Digital Storytelling and Entertainment (TIDSE’03)_.  https://api.semanticscholar.org/CorpusID:15972197
  * Tang et al. (2022) Tianyi Tang, Junyi Li, Z. Chen, Yiwen Hu, Zhuohao Yu, Wen-Dao Dai, Zican Dong, Xiaoxue Cheng, Yuhao Wang, Wayne Xin Zhao, Jianyun Nie, and Ji rong Wen. 2022\.  TextBox 2.0: A Text Generation Library with Pre-trained Language Models. In _Conference on Empirical Methods in Natural Language Processing_.  https://api.semanticscholar.org/CorpusID:255125466
  * team (2023) Mistral AI team. 2023\.  Mixtral of Experts.  https://mistral.ai/news/mixtral-of-experts/.  Mistral AI continues its mission to deliver open models to the developer community, introducing Mixtral 8x7B, a high-quality sparse mixture of experts model..
  * Tolkien (2012) J.R.R. Tolkien. 2012\.  _The Lord Of The Rings: One Volume_.  HarperCollins.  https://books.google.pt/books?id=yl4dILkcqm4C