# AWGUNet: Attention-aided Wavelet Guided U-net for nuclei segmentation in
histopathology images

###### Abstract

Accurate nuclei segmentation in histopathological images is crucial for cancer
diagnosis. Automating this process offers valuable support to clinical
experts, as manual annotation is time-consuming and prone to human errors.
However, automating nuclei segmentation presents challenges due to uncertain
cell boundaries, intricate staining, and diverse structures. In this paper, we
present a segmentation approach that combines the U-Net architecture with a
DenseNet-121 backbone, harnessing the strengths of both to capture
comprehensive contextual and spatial information. Our model introduces the
Wavelet-guided channel attention module to enhance cell boundary delineation,
along with a learnable weighted global attention module for channel-specific
attention. The decoder module, composed of an upsample block and convolution
block, further refines segmentation in handling staining patterns. The
experimental results conducted on two publicly accessible histopathology
datasets, namely Monuseg and TNBC, underscore the superiority of our proposed
model, demonstrating its potential to advance histopathological image analysis
and cancer diagnosis. The code is made available at:
https://github.com/AyushRoy2001/AWGUNET

Index Terms—  Nuclei segmentation, Wavelet guided network, Deep learning,
U-Net, Histopathology images

##  1 Introduction

Nuclei and cell micro-environments provide vital diagnostic information,
particularly as cancer transforms normal cells, altering characteristics such
as nuclei count, size, and morphology. Accurate segmentation of nuclei in
stained pathological images is crucial for pathological research, which is
typically a time-consuming task for radiologists. Automating this segmentation
becomes valuable given the high density of entire slide images. However, the
diversity in uncertain cell boundaries continues to challenge automated nuclei
segmentation in clinical pathology analysis. While machine learning techniques
have shown progress in medical image segmentation, such methods struggle to
extract highly representative features and suffer from limited neighborhood
receptive fields. In recent years, deep learning-based methods [1, 2, 3] have
advanced rapidly, finding widespread application in medical image
segmentation. Convolutional Neural Networks (CNNs) based models, in
particular, have demonstrated exceptional performance in this domain. However,
due to the presence of uncertain cell boundaries, complex staining, and
intricate and heterogeneous histopathological structure, it becomes
challenging to segment nuclei cells accurately. Accurate localization of
nuclei cell boundaries and handling intricate staining patterns remain
fundamental challenges for this task. Additionally, ensuring the model’s
robustness across a wide spectrum of cell morphologies and image variations
presents another significant challenge.

Motivation. With the emergence of the U-Net[4] model, the encoder-decoder
structure has become the go-to choice for image segmentation, including
medical images. U-Net utilizes skip connections to effectively integrate
context information during network training, achieving deep feature fusion
across multiple scales. To enhance U-Nets’ ability, various approaches have
emerged, such as the 3D U-Net for 3D image segmentation, the residual-based
ResUnet++, etc. Attention mechanisms have been a key focus in enhancing the
performance of U-Net for medical image segmentation, leading to the
development of models like the attention U-Net[5]. These methods leverage
attention mechanisms to extract discriminative features, allowing the model to
concentrate on the region of interest and improve image segmentation. While
most existing medical image segmentation networks based on U-Net aim to
enhance contextual information for feature extraction, the importance of
spatial features, particularly edge information, should not be overlooked.

Contributions. To capture both the contextual and spatial information, we
propose a model that adopts a U-Net architecture with a DenseNet-121 backbone
in this work. This combination leverages the strengths of both structures,
offering a powerful foundation for accurate segmentation in histopathological
images. To capture the edge information for spatial guidance, we introduce a
Wavelet Guided Channel Attention Module (WGCAM), which improves the model’s
ability to delineate cell boundaries with precision. We further enhance the
model performance with a learnable weighted Global Average Pooling (lw-GAP),
which provides channel-specific attention. Additionally, the decoder module,
featuring the upsample block and convolution block, refines segmentation,
especially regarding staining patterns and extremely small regions of
interest.

##  2 Methodology

The proposed segmentation model combines the structure of U-Net with the
feature-rich DenseNet-121 as a backbone network to learn hierarchical features
from input data. The WGCAM module captures edge information for spatial
guidance, followed by a lw-GAP module for providing attention to specific
channels. The decoder module comprises the upsample block and the convolution
block. The upsampling block merges locally extracted features from the
transposed convolution layer with upsampled features generated through
Gaussian and Lanczos filters, resulting in a noise-suppressed and anti-aliased
upsampled feature map. This upsampled feature map, achieved by concatenating
the attention-guided features, serves as input for the convolution block. In
this block, features from multiple receptive fields are extracted using
convolution layers with kernels of different sizes. The pipeline of the
proposed model is shown in Fig. 1.

Fig. 1: Our segmentation model reinforces decoder features with edge
information-enhanced features using WGCAM.

###  2.1 Wavelet-guided Channel Attention Module

The WGCAM captures the edge information using wavelet transform [6]. A Haar
wavelet is applied to the input feature, $F_{inp}$ of dimensions $H\times
W\times C$ to decompose it to form $F_{wav}$ of dimensions $H/2\times
W/2\times 4C$. $F_{wav}$ is operated with a transposed convolution to modify
its dimensions to $H\times W\times 4C$. A separable convolution layer is then
utilized to generate the attention weights $F^{\prime}_{wav}$ of dimensions
$H\times W\times C$ as shown in Eq. 1, where $f_{sc}$ and $f_{ct}$ are the
separable convolution and transposed convolution, respectively.

Fig. 2: Wavelet-guided Channel Attention Module

$F^{\prime}_{wav}=f_{sc}^{1\times 1}(f_{ct}^{1\times 1}(DWT_{haar}(F_{inp})))$
(1)

$F^{\prime}_{wav}$ is then added with $F^{\prime}_{inp}$ (generated by
applying separable convolution on $F_{inp}$). A separable convolution layer
followed by a sigmoid activation function is utilized to generate the
attention-aided feature. This attention-aided feature is then element-wise
multiplied with $F_{inp}$ followed by a channel-wise multiplication with
attention weights $F_{c}$ across the channel dimension as shown in Eq. 2 where
$\times$ is element-wise multiplication and $\otimes$ is the multiplication
across the channel dimension.

$F^{\prime}_{wav}=F_{c}\otimes(Sigmoid(f_{sc}^{1\times
1}(F^{\prime}_{inp}+F^{\prime}_{wav}))\times F_{inp})$ (2)

$F_{c}$ is obtained by a learnable weighted Global Average Pooling (lw-GAP)
where learnable weights $\alpha_{1}$, $\alpha_{2}$,… $\alpha_{C}$ are assigned
to each channel. This flattened 1D tensor of dimension C then passes through
two dense layers to generate $F_{c}$ as shown in Fig. 2.

###  2.2 The Decoder Module

In digital images, noise and aliasing can be problematic, especially in
histopathological images, given their complex staining patterns and extremely
small regions of interest. To address these challenges, we have applied the
Gaussian filter [7] to suppress noise and the Lanczos filter [8] with a
$5\times 5$ kernel size for its anti-aliasing properties in the upsampling
layers. The upsampled features are combined to create $F_{up}$, and the
feature $F_{ct}$ from the transposed convolution layer offers detailed local
information to compensate for any data loss during interpolation in the
upsampling layer. The final feature $F_{up-ct}$ is generated by fusing
features from both techniques, as illustrated in Eq. 3, where $\oplus$ denotes
concatenation.

$F_{up-ct}=(f_{c}^{1\times 1}(F_{up})\oplus F_{ct})$ (3)

The convolution block takes the input feature map $F_{up-ct}$ and extracts
information from three different receptive fields. It uses convolution layers
with kernel sizes of $5\times 5$ ($F_{5}$), $3\times 3$ ($F_{3}$), and
$1\times 1$ ($F_{1}$). These features ($F_{5}$, $F_{3}$, and $F_{1}$) are
normalized using Instance Normalization (IN) [9] to account for variations
among images in a dataset, followed by a ReLU activation. The normalized
features are then concatenated and passed through convolution layers, instance
normalization, and Rectified Linear Unit (ReLU) activation to produce the
decoded feature map $F_{dec}$ as shown in Fig. 3.

Fig. 3: A block diagram of the Decoder module. The two key components are the
upsample and convolution blocks

##  3 Results

###  3.1 Experimental Setup

We have evaluated our model on two datasets: MonuSeg [10] and TNBC [1].
MonuSeg contains $512\times 512$ Hematoxylin and Eosin-stained tissue images,
with 30 training images (22,000 annotations) and 14 test images (7000
annotations). The TNBC dataset focuses on Triple-Negative Breast Cancer
tissues, with 50 images (4022 annotated cells). The original image size
($512\times 512$) was used as input for both datasets. We trained using a
70-20-10 % train-validation-test split, using a learning rate of 0.0001, the
Adam optimizer, batch size of 2, and training for 100 epochs. We have used
dice loss and binary cross-entropy (BCE) loss [11] for training and evaluated
using dice, Intersection over Union (IoU), precision and recall as standard
metrics.

###  3.2 Ablation study

To figure out the optimal setup and parameters for our model, we have
performed an extensive ablation study on the MonuSeg dataset. The experiments
are listed below:

(i) U-Net with DenseNet-121 as the backbone

(ii) (i) + WGCAM with GAP

(iii) (i) + lw-WGCAM with GAP

(iv) The proposed model: (iii) + the decoder module

Table LABEL:ablation1 highlights the substantial impact of WGCAM attention
along with the decoder and the upsample module on the performance enhancement
of U-Net with DenseNet-121 as the backbone. The denoising and anti-aliasing
capacity of the upsample layers proves instrumental in tackling the noisy and
complex straining patterns of the images. Also, arbitrary shapes are easily
detected with the help of edge information obtained from the wavelet features
of the WGCAM. Fig. 4 shows the segmentation results and the impact of each
module of the proposed model on both datasets. We compare our method with
state-of-the-art (SOTA) methods on both datasets and tabulate the results in
Table 2.

Table 1: Performance of the segmentation models. All values are in %. Bold values indicate superior performance. Model | Dice | Precision | Recall | IoU
---|---|---|---|---
(i) | 77.39 | 72.56 | 83.17 | 63.17
(ii) | 78.77 | 73.67 | 84.89 | 65.03
(iii) | 78.89 | 75.75 | 82.62 | 65.20
(iv) | 79.46 | 76.26 | 84.91 | 66.57

(a) TNBC

(b) Monuseg

Fig. 4: Segmentation results on MonuSeg and TNBC datasets. GT represents ground truth, ELS, BOT, and DLS are the Encoder Last Layer, BOTtleneck, and Decoder Last Layer, respectively. DWT-1, DWT-2, and DWT-3 are the wavelet features of WGCAM between the second, third and fourth encoder and decoder layers, respectively. Table 2: Performance comparison of the proposed model with SOTA methods. All values are in %. Bold values indicate superior performance. Model | Monuseg | TNBC
---|---|---
Dice | IoU | Dice | IoU
U-Net[4] | 74.67 | 60.89 | 68.61 | 52.92
Attention U-Net[5] | 78.67 | 66.51 | 71.43 | 54.21
DIST[1] | 77.31 | 63.77 | 70.51 | 56.34
MMPSO-S [2] | 72.00 | 56.00 | 65.00 | 49.00
Deep-Fuzz [3] | 79.10 | 66.10 | 77.80 | 64.20
Proposed Method | 79.46 | 66.57 | 81.65 | 69.18

##  4 Conclusion

We present a novel histopathological image segmentation model, which comprises
three essential modules: WGCAM, lw-GAP and the decoder. WGCAM effectively
captures edge information, improving precise cell boundary delineation. The
introduction of lw-GAP enhances performance by providing channel-specific
attention. Within the decoder module, the upsample block reduces noise and
aliasing in the upsampled feature map by combining locally extracted features
with upsampled features generated through Gaussian and Lanczos filters. This
noise-reduced feature map is further strengthened by concatenating attention-
guided features, significantly improving boundary delineation. The convolution
block utilizes convolution layers with varying kernel sizes, adapting to
diverse image variations and enhancing feature extraction. Future work will
explore the model’s performance across other medical image modalities.

## References

  * [1] Peter Naylor, Marick Laé, Fabien Reyal, and Thomas Walter,  “Segmentation of nuclei in histopathology images by deep regression of the distance map,”  IEEE transactions on medical imaging, vol. 38, no. 2, pp. 448–459, 2018.
  * [2] Anusree Kanadath, J Angel Arul Jothi, and Siddhaling Urolagin,  “Multilevel multiobjective particle swarm optimization guided superpixel algorithm for histopathology image detection and segmentation,”  Journal of Imaging, vol. 9, no. 4, pp. 78, 2023.
  * [3] Nirmal Das, Satadal Saha, Mita Nasipuri, Subhadip Basu, and Tapabrata Chakraborti,  “Deep-fuzz: A synergistic integration of deep learning and fuzzy water flows for fine-grained nuclei segmentation in digital pathology,”  Plos one, vol. 18, no. 6, pp. e0286862, 2023.
  * [4] Ronneberger Olaf, Philipp Fischer, and Thomas Brox,  “U-net: Convolutional networks for biomedical image segmentation,”  in Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18. Springer, 2015, pp. 234–241.
  * [5] Ozan Oktay, Jo Schlemper, Loic Folgoc, Matthew Lee, Mattias Heinrich, Kazunari Misawa, Kensaku Mori, Steven McDonagh, Nils Hammerla, Bernhard Kainz, et al.,  “Attention u-net: Learning where to look for the pancreas,”  arXiv preprint arXiv:1804.03999, 2018.
  * [6] Kamarul Hawari Ghazali, Mohd Fais Mansor, and Hussain Aini Mustafa, Mohd Marzuki,  “Feature extraction technique using discrete wavelet transform for image classification,”  in 2007 5th Student Conference on Research and Development. IEEE, 2007, pp. 1–4.
  * [7] Yeon Ju Lee and Jungho Yoon,  “Nonlinear image upsampling method based on radial basis function interpolation,”  IEEE Transactions on Image Processing, vol. 19, no. 10, pp. 2682–2692, 2010.
  * [8] Jagyanseni Panda and Sukadev Meher,  “A new residual image sharpening scheme for image up-sampling,”  in 2022 8th International Conference on Signal Processing and Communication (ICSC). IEEE, 2022, pp. 244–249.
  * [9] Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky,  “Instance normalization: The missing ingredient for fast stylization,”  arXiv preprint arXiv:1607.08022, 2016.
  * [10] Neeraj Kumar, Ruchika Verma, Deepak Anand, Yanning Zhou, Omer Fahri Onder, Efstratios Tsougenis, Hao Chen, Pheng-Ann Heng, Jiahui Li, Zhiqiang Hu, et al.,  “A multi-organ nucleus segmentation challenge,”  IEEE transactions on medical imaging, vol. 39, no. 5, pp. 1380–1391, 2019.
  * [11] Shruti Jadon,  “A survey of loss functions for semantic segmentation,”  in 2020 IEEE conference on computational intelligence in bioinformatics and computational biology (CIBCB). IEEE, 2020, pp. 1–7.


# “Kurosawa”: A Script Writer’s Assistant

Prerak Gandhi∗, Vishal Pramanik∗, Pushpak Bhattacharyya
Department of Computer Science and Engineering
Indian Institute of Technology Bombay, Mumbai
<EMAIL_ADDRESS>
###### Abstract

Storytelling is the lifeline of the entertainment industry- movies, TV shows,
and stand-up comedies, all need stories. A good and gripping script is the
lifeline of storytelling and demands creativity and resource investment. Good
scriptwriters are rare to find and often work under severe time pressure.
Consequently, entertainment media are actively looking for automation. In this
paper, we present an AI-based script-writing workbench called KUROSAWA which
addresses the tasks of plot generation and script generation. Plot generation
aims to generate a coherent and creative plot (600–800 words) given a prompt
(15–40 words). Script generation, on the other hand, generates a scene
(200–500 words) in a screenplay format from a brief description (15–40 words).
Kurosawa needs data to train. We use a 4-act structure of storytelling to
annotate the plot dataset manually. We create a dataset of 1000 manually
annotated plots and their corresponding prompts/storylines and a gold-standard
dataset of 1000 scenes with four main elements — scene headings, action lines,
dialogues, and character names — tagged individually. We fine-tune GPT-3 with
the above datasets to generate plots and scenes. These plots and scenes are
first evaluated and then used by the scriptwriters of a large and famous media
platform ErosNow111https://erosnow.com/. We release the annotated datasets and
the models trained on these datasets as a working benchmark for automatic
movie plot and script generation.

**footnotetext: These authors contributed equally to this work

##  1 Introduction

Movies are one of the most popular sources of entertainment for people
worldwide and can be a strong medium for education and social awareness. The
impact and influence of film industries can be gauged from the fact that
Hollywood movies invest 100s of millions of dollars and often make box-office
collections of billions of dollars. The first motion picture The Great Train
Robbery, 1903— black & white with no sound— was created at the beginning of
the 20th century. Since then, the art has gone through several
transformations, and now people can instantly access 4K HD movies of their
liking on any smart device.

Throughout the history of film, two of the contributors to a film’s
blockbuster success have been the quality of its plot and the manner of
storytelling. The appeal of the movie decreases drastically if the viewers
find the plot drably predictable. Writing a creative and exciting script is,
therefore, a critical necessity and is extremely challenging. Add to this the
constraints of time and budget, and the need for (at least partial) automation
in script writing becomes obvious.

AI-based story generation has been used before. Based on the engagement-
reflection cognitive explanation of writing, the computer model MEXICA Pérez
and Sharples (2001) generates frameworks for short tales. BRUTUS Bringsjord
and Ferrucci (1999) creates short stories with predetermined themes like
treachery. With the arrival of pre-trained transformer models, automatic story
generation has got a shot in the arm. Transformer models like GPT-2 and GPT-3
are extensively used for text generation. These models have shown the
capability of generating creative text, albeit sometimes with hallucinations
Zhao et al. (2020). Text generated by these models also sometimes lacks
coherence and cohesiveness. On the other hand, template-based models can
generate coherent text but lack creativity in generating new characters and
events in the plot Kale and Rastogi (2020).

The process of creating a movie generally starts with an idea which is then
used to create a plot which is used as the base to build the movie script
(Figure 1).

Novel datasets are an important feature of this paper. We closely studied the
plots and prompts of movies from Bollywood and Hollywood. Such plots and
prompts were scraped from Wikipedia222https://www.wikipedia.org/ and
IMDb333https://www.imdb.com/, respectively. The plots are then annotated using
the 4-act story structure- an extension of the well-known 3-act structure
Field (1979). The 4-act structure and the annotation methods are explained in
detail in appendix A.5 and section 4, respectively.

We introduce a dataset of 1000 Hollywood movie scenes and their short
descriptions. The scripts are scraped from IMSDb444https://www.imsdb.com/. The
scenes are annotated with the four major components of a screenplay:
sluglines, action lines, character names and dialogues, described in details
in appendix A.4

We introduce a workbench which we call “Kurosawa”, consisting of datasets and
a pair of GPT-3 Brown et al. (2020) models fine-tuned with the said datasets.
One GPT-3 model generates a movie plot given a short description of the
storyline (15–40 words), while the other creates a scene based on a short
description of the required scene.

Importantly, we have provided the “Kurosawa” platform to one of the biggest
media platforms engaged in the business of making movies and TV shows,
producing music and soundtrack etc.- to help script and content writers from
different film industries create new movie plots.

Figure 1: The thought process a scriptwriter follows in creating a movie
script. An idea (storyline) leads to a plot which is then converted into a
movie script.

Our contributions in this work are as follows:

  * •

To the best of our knowledge, this is the first work on generating movie
scenes from a scene description.

  * •

We create and publicly release two datasets: (a) a parallel dataset of 1000
movie storylines and their corresponding plots, (b) a parallel dataset of 1000
movie scenes and their corresponding descriptions. In (a), we link available
movie storylines from IMDb with available corresponding movie plots from
Wikipedia. In (b), we link available movie scenes from IMSDb with
corresponding descriptions from IMDb.

  * •

We manually annotate movie plots according to a 4-act structure which is an
extension of the well-known 3-act structure Field (1979). Professional
scriptwriters from the media and entertainment industry guided us very
closely.

  * •

We manually annotate movie scenes with four major components of a scene:
sluglines, action lines, character names and dialogues, along with a short
description of the scene.

  * •

We introduce “Kurosawa”: a workbench that consists of multiple datasets and
models which can assist script and scene writers in the film industry.

##  2 Motivation

Movies are a form of visual media and can have a huge influence on life and
society. Movie scripts are often 30,000 words long, comparable to a 100-page
book. Though scripts can be diverse, they have fixed and oft-repeated
structures, e.g., scene heading, transition, character name, etc.. This fixity
and repetition can be dull and time-consuming and can be handed over to AI.
However, a surprising fact is that AI-based models can be creative in
generating novel characters and stories. These reasons have motivated the film
industry to seriously consider harnessing AI for various aspects of movie
making, script and scene writing being one of them.

Los Angeles Times, 19 December 2022, asks, "AI is here, and it’s making
movies. Is Hollywood ready?". The newspaper edition reports mainly movie
editing efforts ongoing at various places using AI. Our task in the paper is
allied but different in the sense that we aim to provide a "script-writers’
assistant".

##  3 Related Work

###  3.1 Automatic Story Generation

Neural models have been able to produce stories by conditioning on different
contents like visuals Huang et al. (2016) and succinct text descriptions Jain
et al. (2017). Work on plot controllable, plan-driven story generation abounds
Riedl and Young (2010); Fan et al. (2019); Pérez and Sharples (2001); Rashkin
et al. (2020). A related kind of work is automatic poetry generation based on
keywords or descriptions Yan (2016); Wang et al. (2016).

###  3.2 Plot Generation

Plot Machines Rashkin et al. (2020) generate multi-paragraph stories based on
some outline phrases. Fan et al. (2018) introduce a hierarchical sequence-to-
sequence fusion model to generate a premise and condition that in turn
generate stories of up to 1000 words. This work- unlike ours- is non-neural
and template-driven and is, therefore, much less creative and novel compared
to what we generate.

###  3.3 Scene Generation

Automatic scene or script generation has received comparatively less
attention. Dialogue generation Li et al. (2016); Huang et al. (2018); Tang et
al. (2019); Wu et al. (2019) with a semblance of scene generation has been
done. There has recently been some work focusing on guiding dialogues with the
help of a narrative Zhu et al. (2020). We generate scenes in which the main
elements come from a small prompt as input.

##  4 Dataset

For movie plot generation, we have taken the plots from Wikipedia. The prompts
for this task have been taken from IMDb. In IMDb, this prompt can be of two
types. The first is a short description (15–40 words) of the movie, while the
second is a long storyline, which varies from 30–200 words and contains much
more details about the different characters and events of the movie. We have
also collected the genres of each film from IMDb. We then divide the plots
using a 4-act structure. For scene generation, we take the scripts from IMSDb
and annotate them with the key elements of a scene.

###  4.1 Plot Generation Dataset

We have created a dataset of 1000 plots consisting of both Bollywood and
Hollywood plots, extracted from Wikipedia using the _wikipedia_ module in
python. The plots collected are around 700 words long on average.

####  4.1.1 Annotation Guidelines

We annotate the plots by manually dividing them into 4 parts using the 4-act
structure described in appendix A.5. We place a single tag at the end of each
act: ⟨one⟩ (Act 1), ⟨two-a⟩ (Act 2 Part A), ⟨two-b⟩ (Act 2 Part B) and ⟨three⟩
(Act 3) as delimiters. An example for plot annotation is given in the appendix
(Figure 6).

####  4.1.2 Movie Genres

To bring some controllability to the plots generated by the model, we have
introduced the genres of the movies in the dataset along with the storyline.
We concatenate the genres at the beginning of the storyline. Figure 2 shows
the distributions of genres in the dataset.

Figure 2: Genre distribution within the plot dataset

###  4.2 Scene Generation Dataset

Movie scripts are very long. A 2-hour movie corresponds to around 30,000
words. Language models used for creative text generation, like GPT-2 and
GPT-3, have token limits of 1024 and 2048, respectively, making it impossible
to handle an entire script in one go. Hence, we divided the scripts into
scenes and manually created their short descriptions. This allows training the
scenes independently instead of relying on any previous scenes.

Movie scripts comprise of multiple elements described in appendix A.4. The
different elements increase the difficulty models face in learning to
distinguish each element. To overcome this obstacle, we tag four major
elements throughout the script — _sluglines, action lines, dialogues and
character names_.

####  4.2.1 Annotation Guidelines

We keep the four major elements present in every script — _sluglines, action
lines, character name and dialogues_ — and remove any other type of
information such as page number, transitions or scene dates. The tagging of
the four major elements is done using beginning and ending tags that are
wrapped around the elements, as shown below:

  * •

Sluglines: ⟨bsl⟩…⟨esl⟩

  * •

Action Lines: ⟨bal⟩…⟨eal⟩

  * •

Character Name: ⟨bcn⟩…⟨ecn⟩

  * •

Dialogue:⟨bd⟩…⟨ed⟩

An example of an annotated scene is seen in Fig 3.

Figure 3: The image depicts a portion of a movie scene with the four major
elements annotated.

##  5 Experiments and Evaluation

We fine-tune GPT3 with our datasets (refer appendix A.6).

###  5.1 Plot Generation

We have created 5 models by fine-tuning GPT-3 with our movie plot dataset in
the following manner, (i) original (without annotation) (O): input- short
storylines, output- plots without any annotations, (ii) annotation and short
input (AS): input- short storylines, output- plots annotated with 4-act
structure, (iii) annotation and long input (AL): input- long, more descriptive
storylines, output- plots annotated with 4-act structure, (iv) annotation and
short input with genres included (ASG): input- short storylines and genre,
output- plots annotated with 4-act structure, (v) annotation and long input
with genres included (ALG): input- long and more descriptive storylines along
with the genre, output- plots annotated with 4-act structure.

For automatic evaluation we use BLEU Papineni et al. (2002), Perplexity
Jelinek et al. (1977), ROUGE Lin (2004). We also use human evaluation in the
form of a five-point Likert Scale Likert (1932). The rating system has 1->
Strongly Disagree, 2-> Disagree, 3-> Neutral, 4-> Agree, 5-> Strongly Agree.
Human-written stories are assumed to have a rating of 5 for each of the
following 5 features: (1) Fluency: grammatical correctness; (2) Coherence:
logical ordering of sentences and paragraphs; (3) Relevance: Whether the key
points from the prompt have been highlighted in the output; (4) Likability:
The measure of how much the story is enjoyable; (5) Creativity: If the output
introduced any new events, character profiles, or relationships.

For plot generation, we generate 50 plots from 50 test prompts. We divide the
stories into five groups of 10 and assign three evaluators to each group.

For scene generation, we generate ten scenes from 10 test prompts. We assign
five evaluators to rate these ten stories.

##  6 Results and Analysis

We present our observations and evaluations. The nature of our task makes
human evaluation take precedence over automatic evaluation (it is for
automatic movie script generation, after all!). The qualitative analysis of
our generated plots and scenes is based on feedback from 5 professional
scriptwriters of our industry partner, the well-known media platform.

###  6.1 Plot Generation

####  6.1.1 Automatic Evaluation

Table 1 shows auto-evaluation scores for the multiple GPT-3 plot generation
models.

Figure 4: The above paragraph is a partial example of a movie plot generated by the model fine-tuned with input as short storyline and output as plot annotated with the 4-act structure. Models | O | AS | ASG | AL | ALG
---|---|---|---|---|---
Metrics |  |  |  |  |
Perplexity | 2.48 | 1.84 | 2.43 | 2.33 | 2.63
BLEU-2 (%) | 12.95 | 12.01 | 12.51 | 13.08 | 14.52
BLUE-3 (%) | 4.70 | 4.21 | 4.55 | 4.84 | 5.59
BLUE-4 (%) | 2.14 | 1.92 | 2.13 | 2.27 | 2.59
ROUGE-L (%) | 22.67 | 21.72 | 23 | 24.02 | 24.88
Distinct 3-gram (%) | 97.55 | 97.61 | 97.39 | 97.28 | 98.09
Repetition 3-gram (%) | 1.99 | 2.02 | 1.72 | 1.89 | 1.74
Table 1: Scores from common evaluation metrics for 5 Hollywood plot generation
models fine-tuned on GPT-3 as O, AS, ASG, AL, ALG (5.1)

####  6.1.2 Human Rating

We conducted human evaluation on Hollywood annotated short input model. The
evaluation was done by five groups of 3 people, with each group having been
assigned 10 unique plots. The ratings given for the 5 features are in Figure
5. The average scores for fluency, creativity, likability, coherence and
relevance are 3.98, 3.29, 2.97, 2.65 and 2.55, respectively. Fluency of almost
4 is an indicator of the power of GPT-3 as a language model. Creativity and
likeability are respectable at a value of around 3.0. The low BLEU scores
support the average creativity score (Table 1). Figure 5 indicates that
coherence and relevance still have major room for improvement.

The MAUVE (Pillutla et al., 2021) value measures the gap between neural text
and human text. We have separately calculated the MAUVE scores for 20 plots
and 50 plots. The weighted average of the MAUVE scores for the two experiments
is 0.48 which is reasonably good.

Figure 5: Boxplot graphs for Human Evaluation of the plot and scene generation
models.

####  6.1.3 Qualitative Observations

Professional scriptwriters from our industry partner have given the following
observations:

Non-annotated Hollywood Plots

  * •

The build-up is creative and interesting, but the ending becomes incoherent.

  * •

Some characters which are introduced in the beginning are never mentioned
again.

  * •

The output is not portraying the key points or the theme mentioned in the
input.

Annotated Hollywood Plots

  * •

The plots are much more coherent, and the endings are logical.

  * •

There is still hallucination present (a common feature of all models).

  * •

The longer inputs made the plots more attentive to the key points.

Annotated Hollywood Plots with Genres included

  * •

Along with the above points, now the plots generated are more tilted towards
the genre or genres of the movie the writer wants to create.

  * •

Addition of genre gives some control over the kind of plot generated by the
model.

Annotated Bollywood plots

  * •

The outputs show incoherence in the last two paragraphs and repetition of the
same characters throughout the plot.

  * •

The flow of the plot is not fast enough, i.e., the plot does not move ahead
much.

  * •

Many of the outputs have a 1990s theme around them, where the characters are
separated and then find each other later. This is due to a skewed dataset with
lesser modern plots.

###  6.2 Scene Generation

We fine-tuned GPT-3 for scene generation with our dataset. We generated ten
scenes using the models mentioned in 5.1. Figure 7 in the appendix. shows an
example of a completely generated scene.

####  6.2.1 Human Ratings

We conducted a human evaluation on 10 scenes generated by the above model. 5
people evaluated the scenes using the Likert Scale. The ratings for the five
features can be seen in Figure 5. The average scores for fluency, creativity,
likability, coherence, and relevance are 4.48, 3.9, 3.48, 3.46 and 3.86,
respectively. All of the values are above the neutral mark and imply that the
generated scenes are close to human-written scenes.

####  6.2.2 Qualitative Observations

In this section, we analyze the quality of the scenes generated by the GPT-3
model. This analysis has been done by professional scriptwriters from the
previously mentioned media company.

  * •

The model produces a well-structured scene.

  * •

It can create new characters and fabricate dialogues even when they are
unimportant.

  * •

The key points from the input can be found in the output.

  * •

There are some lines that are repetitive.

  * •

The output is not completely coherent.

##  7 Conclusion and Future Work

In this paper, we have reported a first-of-its-kind work on automatic plot and
script generation from prompts. Automatic evaluation, human rating using the
Likert scale, and qualitative observations by professional scriptwriters from
our industry partner (a large and well-reputed media platform)- all vindicate
the power of our rich dataset and GPT3 in script generation. We hope our work
will help television show writers, game show writers, and so on.

There are several future directions: (i) the imbalance in the Bollywood plot
dataset needs to be rectified; (ii) there is a lot of variation in Indian
script because of multilingualism, which needs addressing; (iii) the most
obvious weakness of GPT-3 is not being able to handle factual data and
numbers, causing hallucination and preventing the automatic generation of
documentaries and biographies. Detection and resolution of hallucination is
anyway a growing need for language models.

##  8 Limitations

  * •

In the plot generation dataset, the Wikipedia plots are sometimes not written
by professional content writers from the film industry. Therefore these plots
may fail to include the main events of the movie.

  * •

In a few cases, the model fails to generate coherent events along with the
abrupt introduction of characters in the plots and scenes.

  * •

Although it has been noticed only a few times, the plot or scene generated
contains repeated clauses or phrases.

  * •

The model hallucinates and generates factually incorrect things, making it
incapable of generating biographies or documentaries.

  * •

The plot or scene may not abide by the theme of the input or genre mentioned
along with the prompt.

## References

  * Bringsjord and Ferrucci (1999) Selmer Bringsjord and David Ferrucci. 1999.  _Artificial intelligence and literary creativity: Inside the mind of brutus, a storytelling machine_.  Psychology Press.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Fan et al. (2018) Angela Fan, Mike Lewis, and Yann Dauphin. 2018.  Hierarchical neural story generation.  _arXiv preprint arXiv:1805.04833_.
  * Fan et al. (2019) Angela Fan, Mike Lewis, and Yann Dauphin. 2019.  Strategies for structuring story generation.  _arXiv preprint arXiv:1902.01109_.
  * Field (1979) S. Field. 1979.  _Screenplay: The Foundations of Screenwriting_.  A Delta book. Dell Publishing Company.
  * Huang et al. (2018) Chenyang Huang, Osmar R Zaiane, Amine Trabelsi, and Nouha Dziri. 2018.  Automatic dialogue generation with expressed emotions.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 49–54.
  * Huang et al. (2016) Ting-Hao Huang, Francis Ferraro, Nasrin Mostafazadeh, Ishan Misra, Aishwarya Agrawal, Jacob Devlin, Ross Girshick, Xiaodong He, Pushmeet Kohli, Dhruv Batra, et al. 2016.  Visual storytelling.  In _Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: Human language technologies_ , pages 1233–1239.
  * Jain et al. (2017) Parag Jain, Priyanka Agrawal, Abhijit Mishra, Mohak Sukhwani, Anirban Laha, and Karthik Sankaranarayanan. 2017.  Story generation from sequence of independent short descriptions.  _arXiv preprint arXiv:1707.05501_.
  * Jelinek et al. (1977) Frederick Jelinek, Robert L. Mercer, Lalit R. Bahl, and J. Baker. 1977.  Perplexity—a measure of the difficulty of speech recognition tasks.  _Journal of the Acoustical Society of America_ , 62.
  * Kale and Rastogi (2020) Mihir Kale and Abhinav Rastogi. 2020.  Template guided text generation for task-oriented dialogue.  _arXiv preprint arXiv:2004.15006_.
  * Li et al. (2016) Jiwei Li, Will Monroe, Alan Ritter, Michel Galley, Jianfeng Gao, and Dan Jurafsky. 2016.  Deep reinforcement learning for dialogue generation.  _arXiv preprint arXiv:1606.01541_.
  * Likert (1932) Rensis Likert. 1932.  _A technique for the measurement of attitudes / by Rensis Likert._ Archives of psychology ; no. 140. [s.n.], New York.
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_ , pages 311–318.
  * Pérez and Sharples (2001) Rafael Pérez Ý Pérez and Mike Sharples. 2001.  Mexica: A computer model of a cognitive account of creative writing.  _Journal of Experimental & Theoretical Artificial Intelligence_, 13(2):119–139.
  * Pillutla et al. (2021) Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. 2021.  Mauve: Measuring the gap between neural text and human text using divergence frontiers.  _Advances in Neural Information Processing Systems_ , 34:4816–4828.
  * Rashkin et al. (2020) Hannah Rashkin, Asli Celikyilmaz, Yejin Choi, and Jianfeng Gao. 2020.  Plotmachines: Outline-conditioned generation with dynamic plot state tracking.  _arXiv preprint arXiv:2004.14967_.
  * Riedl and Young (2010) Mark O Riedl and Robert Michael Young. 2010.  Narrative planning: Balancing plot and character.  _Journal of Artificial Intelligence Research_ , 39:217–268.
  * Tang et al. (2019) Jianheng Tang, Tiancheng Zhao, Chenyan Xiong, Xiaodan Liang, Eric Xing, and Zhiting Hu. 2019.  Target-guided open-domain conversation.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 5624–5634, Florence, Italy. Association for Computational Linguistics.
  * Wang et al. (2016) Zhe Wang, Wei He, Hua Wu, Haiyang Wu, Wei Li, Haifeng Wang, and Enhong Chen. 2016\.  Chinese poetry generation with planning based neural network.  _arXiv preprint arXiv:1610.09889_.
  * Wu et al. (2019) Wenquan Wu, Zhen Guo, Xiangyang Zhou, Hua Wu, Xiyuan Zhang, Rongzhong Lian, and Haifeng Wang. 2019.  Proactive human-machine conversation with explicit conversation goal.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3794–3804, Florence, Italy. Association for Computational Linguistics.
  * Yan (2016) Rui Yan. 2016.  i, poet: Automatic poetry composition through recurrent neural networks with iterative polishing schema.  In _IJCAI_ , volume 2238, page 2244.
  * Zhao et al. (2020) Zheng Zhao, Shay B Cohen, and Bonnie Webber. 2020.  Reducing quantity hallucinations in abstractive summarization.  _arXiv preprint arXiv:2009.13312_.
  * Zhu et al. (2020) Yutao Zhu, Ruihua Song, Zhicheng Dou, Jian-Yun Nie, and Jin Zhou. 2020.  ScriptWriter: Narrative-guided script generation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8647–8657, Online. Association for Computational Linguistics.

##  Appendix A Appendix

###  A.1 Ethics Consideration

We have taken all the scripts from IMDB and IMSDb databases. The website has a
disclaimer regarding using its scripts for research, which can be found at
this link https://imsdb.com/disclaimer.html. We have used the scripts fairly
and without copyright violation.

###  A.2 Annotator Profiles

We required the help of external annotators in two cases: (i) Manually
Annotating the Scripts and (ii) Creating scenes and their descriptions from
the scripts. For the first task, we took the help of 10 annotators. Their ages
ranged from 21-28, and all were Asian. They were given detailed guidelines
with examples for annotating. There were also periodic sessions to confirm
their understanding and solve their doubts and mistakes. For the second task,
we took the help of two annotators. Both of them are Asian females aged
between 21-23. Both of them were given detailed guidelines for the scene-
writing task. A few data points were picked randomly and checked to find out
and correct conceptual mistakes. The annotators had bachelors and masters
degree in STEM and Arts.

Figure 6: Example of manual annotation of the plot of the movie _Music of the
Heart_ using the 4-act structure

Figure 7: An example of a complete scene generated given a short input.

###  A.3 Evaluation Metrics

The evaluation metrics are described below:

  * •

Perplexity (PPL): Perplexity is one of the most common metrics for evaluating
language models. They are computed as exponential of entropy. The smaller the
value of the PPL, the greater the fluency of the generated text.

  * •

BLEU: BiLingual Evaluation Understudy is a common metric in many NLP tasks,
especially in the field of Machine Translation. It measures the overlap
between the generated output and gold standard data. Although this metric does
not consider the model’s creativity, we can deduce the difference between the
candidate text and the reference text using BLEU. The higher the BLEU measure,
the better it is.

  * •

ROUGE: Recall-Oriented Understudy for Gisting Evaluation is typically used for
evaluating automatic summarization. In our case, it measures the longest
overlapping sequence between the generated and original plots. The higher the
ROUGE measure, the better it is.

  * •

N-grams: We measure the redundancy and diversity of the movie plots by
computing the repetition and distinction n-gram scores.

###  A.4 Screenplay Structure

A movie script or a screenplay has a different format than a story. A script
is a group of scenes. Each of these scenes consists of a few major components,
which are discussed below:

Scene Headings/Sluglines\- This component describes the when and where of the
scene. It can be thought of as the first shot that a camera takes of a new
scene. For example, INT. - RESTAURANT - NIGHT indicates that the scene starts
inside a restaurant at night. Sluglines are normally written in capital
letters and are left-aligned.

Character Names\- they are mentioned every time a character is going to utter
a dialogue. The name of each character is mentioned in uppercase and is centre
aligned.

Dialogues\- dialogues are the lines that the characters say. They appear right
after the character name in a script and are centrally aligned.

Action Lines\- action lines describe almost everything about a scene. They can
be described as the narration of each script. Action lines can be present
after either dialogues or sluglines and are left-aligned.

Transitions\- a transition marks the change from one scene to the next. They
also depict how a scene is ended. For example, DISSOLVE, FADE, and CUT are
different keywords used to indicate a transition. They are usually in upper
case and are right-aligned.

Figure 8 shows an example of the screenplay elements.

Figure 8: The elements of a screenplay

###  A.5 Story Templates

Over time various templates have been developed that help to create stories.
One of the most famous templates is the 3-act structure Field (1979). This
structure divides a story into a setup, confrontation, and resolution. In this
work, we have used the 4-act structure which we now describe in detail.

Act 1\- This is the opening/introduction act. It describes the protagonist’s
character and briefly introduces the movie’s theme. The act ends with the
start of a new journey for the protagonist.

Act 2A\- Due to the vast span of Act 2, it can be divided into two acts. This
act usually contains the start of a love story. It also entertains the
audience as the protagonist tries to adapt to their new journey. The act ends
as the movie’s midpoint, one of the film’s critical moments, with either a
very positive or negative scene.

Act 2B\- This act usually contains the protagonist’s downfall. The villain or
antagonist starts to gain an advantage, and the protagonist loses something or
someone significant. The act ends with the protagonist realizing their new
mission after reaching rock bottom.

Act 3— The protagonist has realized the change required in them and sets out
to defeat the antagonist in a thrilling finale. The movie then ends by
displaying a welcome change in the protagonist that was lacking in the
beginning.

###  A.6 Fine-Tuning GPT-3

GPT-3 was deemed publicly available last year by OpenAI Brown et al. (2020).
Its best model has 175B parameters, which is much more than GPT-2’s 2.9B
parameters. We have fine-tuned multiple plot generation models with GPT-3
along with a scene generation model. The multiple combinations of plot
generation models are short or long prompts and with or without genres. The
GPT-3 model and hyperparameters remain the same for all the above
combinations. We have fine-tuned the GPT-3 Curie model for four epochs. For
generating text, GPT-3 offers various hyperparameters to tune and get closer
to our desired results. For testing, we set other hyperparameters as follows:
the temperature as 0.7, top-p as 1, frequency penalty as 0.1, presence penalty
as 0.1, and max tokens as 900.