#  CollabStory:
Multi-LLM Collaborative Story Generation and Authorship Analysis

Saranya Venkatraman, Nafis Irtiza Tripto, Dongwon Lee
The Pennsylvania State University, USA
{saranyav, nit5154<EMAIL_ADDRESS>
###### Abstract

The rise of unifying frameworks that enable seamless interoperability of Large
Language Models (LLMs) has made LLM-LLM collaboration for open-ended tasks a
possibility. Despite this, there have not been efforts to explore such
collaborative writing. We take the next step beyond human-LLM collaboration to
explore this multi-LLM scenario by generating the first exclusively LLM-
generated collaborative stories dataset called CollabStory. We focus on
single-author ($N=1$) to multi-author (up to $N=5$) scenarios, where multiple
LLMs co-author stories. We generate over 32k stories using open-source
instruction-tuned LLMs. Further, we take inspiration from the PAN tasks
Bevendorff et al. (2023) that have set the standard for human-human multi-
author writing tasks and analysis. We extend their authorship-related tasks
for multi-LLM settings and present baselines for LLM-LLM collaboration. We
find that current baselines are not able to handle this emerging scenario.
Thus, CollabStory is a resource that could help propel an understanding as
well as the development of techniques to discern the use of multiple LLMs.
This is crucial to study in the context of writing tasks since LLM-LLM
collaboration could potentially overwhelm ongoing challenges related to
plagiarism detection, credit assignment, maintaining academic integrity in
educational settings, and addressing copyright infringement concerns. We make
our dataset and code available at https://github.com/saranya-
venkatraman/multi_llm_story_writing.

CollabStory:
Multi-LLM Collaborative Story Generation and Authorship Analysis

Saranya Venkatraman, Nafis Irtiza Tripto, Dongwon Lee The Pennsylvania State
University, USA {saranyav, nit5154<EMAIL_ADDRESS>

##  1 Introduction

Figure 1: CollabStory contains over $32k$ creative stories written
collaboratively by up to 5 LLMs. Each story segment is generated by a single
author, that then passes the narrative baton to the next, completing the
storyline part by part in a sequential manner.

Generative Large Language Models (LLMs) are being used more widely and
becoming ubiquitous in real-world scenarios. There is particular interest in
understanding the use of such LLMs in various writing tasks as writing
assistants or collaborators in machine-in-the-loop settings Yeh et al. (2024);
Kim et al. (2023); Zhong et al. (2023); Singh et al. (2023); Yang et al.
(2022); Yuan et al. (2022); Lee et al. (2022); Clark et al. (2018). So far
though, this has only been explored in the case where a human is present.
However given the rise of unifying frameworks that bring together and make
LLMs from different sources interoperable, such as
vLLM111https://docs.vllm.ai/en/stable/,
LangChain222https://www.langchain.com/langchain, and
HuggingFace333https://huggingface.co/, the prospect of LLMs seamlessly
collaborating and even handing off tasks to one another without external
routing algorithms is on the horizon. This is particularly immediately
possible with open-source models that are already being used by over 100K
users per month (according to the number of downloads reported by
HuggingFace). Despite the ease of interoperability of such LLMs, so far,
automated writing assistants have been used only in collaboration with human
authors or with a single LLM. Therefore, this study explores collaborative
creative story-writing scenarios involving multiple LLMs, i.e. LLM-LLM
collaboration.

Dataset | # Stories | # Authors |  Avg words | M-M Collaboration | Available
---|---|---|---|---|---
STORIUM Akoury et al. (2020) | 5,743 | 30,119 | ~19k tokens | H-H X | ✓
CoAuthor Lee et al. (2022) | 830 | 58 | 418 | H-M X | ✓
StoryWars Du and Chilton (2023) | 40,135 | 9,494 | 367 | H-H X | X
CollabStory [Ours] | 32,503 | 5 | 725 | M-M ✓ | ✓

Table 1: Comparison of CollabStory with other existing collaborative creative
story datasets. Here, “M-M” collaboration refers to “Machine-Machine”
collaboration, while “H-H” refers to “Human-Human” collaboration, where “H”
$\,\to\,$ “Human” and “M”$\,\to\,$ Machine. Ours is the largest publicly
available dataset to present creative stories written collaboratively by
different LLMs.

Collaborative creative story writing entails multiple authors contributing
separate segments to form a coherent storyline (see Figure 1 for our dataset
schema). Although individual LLMs excel at generating story plots,
collaborative writing presents unique hurdles. Models must seamlessly continue
the existing storylines generated so far by other models, even if they do not
align perfectly with their own language distribution. The rise of multi-agent
Artificial Intelligence (AI) underscores the potential for combining the
expertise of agents specialized in various tasks. While previous mixture-of-
experts scenarios focused on agents proficient in task-oriented settings Zhang
et al. (2024); Pan et al. (2024); Liu et al. (2023); Li et al. (2023b, a), the
emergence of LLMs conversing for continuous generative tasks in open domains
is noteworthy. Imagine the possibilities when multiple LLMs collaborate; one
LLM can generate compelling stories, but what if we put them together?

In this study, we attempt to address this question through a collaborative
creative story-writing scenario involving multiple open-source LLMs. This is a
crucial setting to study in the context of writing tasks since LLM-LLM
collaboration could potentially overwhelm ongoing challenges related to
plagiarism detection, credit assignment, maintaining academic integrity in
educational settings, and addressing copyright infringement concerns.

We focus on single-author ($N=1$) to multi-author (up to $N=5$) scenarios,
where multiple LLMs co-author creative stories. This exploration is novel, as
previous studies have primarily focused on human-LLM collaboration. Towards
this goal, we generate the first multi-LLM collaborative story dataset called
CollabStory using open-source LLMs. We select 5 frequently used LLMs (with
number of downloads on HuggingFace for May 2024 provided in parenthesis):
Meta’s Llama ($>540k$ downloads, Touvron et al. (2023)), Mistral.ai’s Mistral
($>1000k$ downloads, Jiang et al. (2023)), Google’s Gemma ($>180k$ downloads,
Team et al. (2024)), AllenAI’s Olmo ($>26k$ downloads, Groeneveld et al.
(2024)) and Microsoft’s Orca ($>22k$ downloads, Mukherjee et al. (2023)) to
replicate a scenario in which commonly used LLMs from different organizations
are being used in conjunction towards a single task. We demonstrate how one
such dataset can be developed and the considerations involved in building an
iterative Multi-LLM story-writer. We take inspiration from the PAN tasks
Bevendorff et al. (2023) that have set the standard for multi-author writing
tasks and analysis for human-human collaboration for over 15 years. We
replicate their task settings and present baselines of different authorship-
related tasks such as authorship verification and attribution for LLM-LLM
collaboration and demonstrate that current baselines are challenged by this
emerging scenario. CollabStory is the first resource that could help propel an
understanding as well as the development of new techniques to discern the use
of multiple LLMs in text.

Our work is motivated by the implications of Multi-LLM settings for different
stakeholders (LLM developers, end-users) and considerations (such as credit
assignment, legality of usage) arising in the generative AI landscape. As one
example, a malicious actor might assemble texts from different LLMs together
in one document to evade current detectors and successfully spread
misinformation. Our discussion will further elaborate on the tasks our dataset
enables and why it is crucial to develop methods to tackle the incoming
challenges of machine-machine collaboration.

##  2 Related Work

#### LLMs as Collaborative Writers

LLMs are being increasingly used as writing assistants or to paraphrase, edit
or enhance human-written written texts in machine-in-the-loop settings Kim et
al. (2023); Singh et al. (2023); Yang et al. (2022); Clark et al. (2018).
GhostWriter Yeh et al. (2024) and Wordcraft Yuan et al. (2022) are tools that
enable users to co-write stories using instructions Yuan et al. (2022). Zhong
et al. (2023) use “writing modes” as a control signal to better align the
machine during co-writing with humans. CoAuthor positions GPT3.5 as a writing
collaborator for over 50 human participants to co-write creative and
argumentative stories Lee et al. (2022).

#### Datasets

Despite such emerging tools, only a handful have developed datasets that can
be leveraged to understand collaborative story writing. One such resource is
the STORIUM dataset Akoury et al. (2020) that contains over 5k creative
stories written and obtained from human-human collaboration. In terms of
human-machine co-writing, CoAuthor Lee et al. (2022) and CoPoet Chakrabarty et
al. (2022) remain one of the few publically available datasets of human-
machine collaborative creative story and poem writing, respectively. Beyond
creative writing, Zeng et al. (2024) developed the first machine-human
academic essay dataset as a means to study boundary detection for academic
settings. A comparison of CollabStory with existing datasets is provided in
Table 1.

# Number | # Words per Author / | # Author Order | # Prompts per | # Stories | Authors | HuggingFace distribution of LLMs used
---|---|---|---|---|---|---
of Authors ($N$) | # Total Words | Permutations | Author Order |  |  |
1 | 900 / 900 | 4 | 1800 | 7200 | Gemma | google/gemma-1.1-7b-it888https://huggingface.co/google/gemma-1.1-7b-it
2 | 450 / 900 | 12 | 600 | 7200 | Llama | meta-llama/Llama-2-13b-chat-hf999https://huggingface.co/meta-llama/Llama-2-13b-chat-hf
3 | 300 / 900 | 15 | 480 | 7200 | Mistral | mistralai/Mistral-7B-Instruct-v0.2101010https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2
4 | 225 / 900 | 15 | 480 | 7200 | Orca | microsoft/Orca-2-13b111111https://huggingface.co/microsoft/Orca-2-13b
5 | 180 / 900 | 15 | 480 | 7200 | Olmo | allenai/OLMo-7B-Instruct121212https://huggingface.co/allenai/OLMo-7B-Instruct

Table 2: Summary of Data Collection Statistics

##  3 Methodology

###  3.1 CollabStory: Dataset Creation

We generate a dataset of creative stories using five open-source instruction-
tuned LLMs: Llama2 Touvron et al. (2023), Olmo Groeneveld et al. (2024), Gemma
Team et al. (2024), Mistral Jiang et al. (2023) and Orca Mukherjee et al.
(2023) (model details are provided in Table 2). The main focus of our data
generation is to simulate a scenario where LLMs from different sources
(organizations) collaboratively work on a storyline, handing off control of
the story from one LLM to the next. The stories in our dataset vary in the
number of authors/LLMs involved, from being written entirely by a single LLM
to written collaboratively by between 2 to up to all 5 LLMs. In this document,
we refer to each of the LLMs as “authors". For cases where we refer to the
human author, we specifically mention “human” author/writer. We generate our
dataset by prompting various LLMs using creative writing prompts from an
existing dataset called the Writing Prompts (WP) Dataset. The Writing Prompts
Dataset was collected by Fan et al. (2018) using Reddit’s r/WritingPrompts/
forum that contains premises or prompts for stories. The WP dataset consists
of a cleaned subset of story prompts and corresponding human-written stories
using filtration criteria such as removal of stories that are bot-generated,
less than 30 words long, contain profanity, general announcements, and so on.
We used the test
split444https://www.kaggle.com/code/ratthachat/writingprompts-combine-one-
line-data-for-gpt2/input?select=writingPrompts of this dataset as the source
of prompts for LLM generated stories. We also filter out prompts that do not
have at least one corresponding human-written story that is at least 800 words
long. We do this to ensure that the prompt itself does not preclude longer
storylines. We chose 800 words as a criteria as a means to include stories
that are slightly longer than the average of the dataset. The average length
(number of words) of articles in the test set is 675.75 words. Out of 15138
total prompt-story pairs, this left us with 4623 data points. For each prompt,
we divide the total goal article length (800-900 words) by the number of
authors ($N$) to calculate the length of each part or story chunk to be
written by each author, such that the writing load is distributed roughly
uniformly amongst the LLM authors. We also generate different permutations of
LLM authorship order such that every author can contribute to random parts of
the story and we ensure that our dataset does not have any spurious
correlations between LLM/author and story sections such as the beginning, or
ending. For each value of the numbers of authors i.e. $N\in\\{1,2,3,4,5\\}$,
we generate all possible permutations of author orders. For example, for N =
3, two examples of author order permutations could be:

$\displaystyle Olmo\rightarrow Mistral\rightarrow Llama$ $\displaystyle
Gemma\rightarrow Llama\rightarrow Mistral$

From all such possible permutations, we sample the minimum of either total
possible orders or 15 as the number of author orders. For each author order,
we then generate stories using each of the prompts from a unique set of
prompts per N. Our goal number of stories for each N was set to 7200 stories.
A summary of the words written by each author, author order permutations, and
prompts per author, as well as the pool of 5 authors and their corresponding
model checkpoints used for generating all story parts is shown in Table 2.

###  3.2 LLM prompting

For each value of $N$, we used different prompts to generate story parts
sequentially, as detailed in Table 3. Utilizing the vLLM
library555https://docs.vllm.ai/en/stable/, we accessed and generated text from
various LLMs. Initially, we conducted a pilot study to refine our prompts by
generating and reviewing 100 articles. For the "Beginning" prompt, the first
LLM used only the original r/WritingPrompts/ input. For subsequent parts, we
found that longer input prompts reduced story length, so we used Falcon.ai
summarizer666https://huggingface.co/Falconsai to condense the story so far
into under 80 words, allowing LLMs to generate longer sequences. We also
included the last sentence of the story so far for smooth continuity. Prompts
for different sections only varied in their instructions to “begin”,
“continue”, or “conclude” the story. Additionally, we added an instruction to
prevent LLMs from generating extraneous instructions. More details are
provided in Section A.2.

Story Part |  Prompt Template
---|---
Beginning |  You are a creative story writer. Write a story that starts with the prompt {starting prompt} in around {n} words. Do not add any instructions. Start the story as follows:
Middle |  Write {n} words to continue this storyline: {summary of story so far}. Continue from this sentence: {last sentence from previous part}
Ending |  Write {n} words to conclude this storyline: {summary of story so far}. Do not add any instructions. Continue from this sentence: {last sentence from previous part}

Table 3: Prompt templates for different parts of the story. {n} here denotes
the number of target words for each author.

Part 1 | Part 2 | Part 3 | Part 4 | Part 5
---|---|---|---|---
Mistral | Gemma | Llama | Olmo | Orca
Years ago, in the depths of a dark and desolate forest, I made a promise to a witch. Desperate for her help to save my beloved firstborn son, I agreed to offer her my firstborn in return for her healing magic. The old crone’s cackling laughter still echoed in my ears as I made that fateful vow. But time passed, and my wife bore us three more children. With each birth, I was filled with relief and gratitude, knowing that our family had been spared the tragedy that had once threatened to tear us apart. Yet no matter how hard I tried, I couldn’t bring myself to fulfill my promise to the witch. Now, years have passed, and the witch has grown tired of waiting. Her messages, delivered by raven messenger, are becoming more and more urgent. The ground beneath my feet seems to shake with her anger, and I can’t help but fear for the consequences of my broken vow. Desperate for a solution, I turn to the only person who might be able to help me. But will she be able to save me from the wrath of the vengeful witch? |  Her pale, ethereal face held a cruel smile, her voice like the rustling of dry leaves in a desolate forest. "Your son’s life hangs in the balance, dear one," she hissed, her voice dripping with malice. "Yet, you have failed to fulfill your part of the bargain." Panic surged through me as I realized the witch was right. I had never presented her with a child. The burden of my desperation had become a heavier chains, tethering me to an unspoken debt. Desperation fueled the flames of my fear, as I knew that if I failed, not only would my son succumb to his fate, but the wrath of the vengeful witch would descend upon me. Time seemed to warp, stretching into an agonizing eternity as I considered my options. Could I find a way to appease her, to escape the consequences of my foolish decision? |  I paced back and forth in my chambers, my mind racing with thoughts of the witch and the debt I owed her. I had been so desperate to save my son, I had agreed to give her my firstborn child in exchange for her healing magic. Now, years had passed and the witch had grown tired of waiting. She had come to me, demanding that I fulfill my end of the bargain. I knew that I could not give her my son. He was all I had left of my beloved husband, and I would rather die than lose him. But I also knew that I could not defy the witch. She had the power to take my life, and the lives of my children, with just a wave of her hand. I spent hours pacing, trying to think of a way to appease her. But no matter what I did, I could not come up with a solution. I was trapped, and I knew it. The witch had me in her grasp, and she would not let me go. Finally, in desperation, I decided to seek out the help of a wise old woman who lived in the village. |  A few days ago, I came to her, my heart heavy with grief and my soul wounded. She, in turn, looked at me with a kind and gentle expression. Then, with a wave of her hand, she began her spell. I watched as the darkness in my heart began to fade, and my love for my children grew stronger than ever. She whispered words of power and love, and the bond between my wife and I was restored. From that day forward, we were a family once more, united by the magic of the wise old woman. |  The wise old woman had always been a renowned healer in our village. Her knowledge of herbs and potions was unmatched, and many came to her seeking relief from various ailments. I was one of those who sought her help, when my firstborn son fell gravely ill. His fever raged, and his tiny body was wracked with pain. Desperate for her help to save my beloved firstborn son, I agreed to give her my firstborn in exchange for her healing magic. I returned to my wife, heartbroken and ashamed, to tell her of the bargain I had made. But time passed, and my wife bore us three more children. With each birth, I was filled with relief and gratitude, knowing that our family had been spared the tragedy of losing a child. The wise old woman had honored her part of the bargain, and my son grew strong and healthy under her care. He became a skilled healer in his own right, following in the footsteps of the wise old woman who had saved his life.

Table 4: Example of a 5-part LLM story from CollabStory dataset for the
prompt: “Years ago, you promised your firstborn to a witch. Since then,
despite your best efforts, you can’t seem to give him away. The witch is
starting to get pretty mad.”

###  3.3 Post-processing and filtering

Though we used instruction-tuned LLMs, they do not follow instructions
perfectly. Though our goal number of words per story was 800-900 words, we
used the upper limit to calculate the number of words each LLM should
generate. From our pilot study, we found that most LLMs were undershooting
their target number of words in the instruction. We also filtered out all
stories in which at least one part was under 50 words long. We also removed
all extra spaces from the stories and any repetitions of the instructions in
rare cases. We also filtered for some additional types of noises detailed in
Section A.3. After this filtration, we were left with the following number of
stories per $N\in[1,5]:{7164,7070,6093,6955,5221}$ for a total of $32,503$
stories. An example of one such story from our dataset can be read in detail
in Table 4.

Feature | N=1 (H) | N=1 (M) | N=2 | N=3 | N=4 | N=5
---|---|---|---|---|---|---
# Words | 1352.26 ± 425.11 | 725.03 ± 288.32 | 1090.67 ± 207.43 | 1154.44 ± 112.24 | 1091.99 ± 85.67 | 995.42 ± 74.20
# Sentences | 84.23 ± 34.85 | 41.90 ± 61.94 | 60.49 ± 14.49 | 64.38 ± 11.57 | 59.29 ± 11.24 | 53.86 ± 9.41
# Words in sentence | 17.82 ± 28.18 | 18.27 ± 3.92 | 18.42 ± 2.85 | 18.30 ± 2.45 | 18.81 ± 2.47 | 18.88 ± 2.51
Vocabulary richness | 0.34 ± 0.05 | 0.40 ± 0.09 |  0.36 ± $0.05^{*}$ |  0.36 ± $0.03^{*}$ |  0.37 ± $0.03^{*}$ | 0.39 ± 0.03
% of stopwords | 31.26 ± 4.76 | 37.00 ± 4.77 | 37.77 ± 3.11 | 37.39 ± 2.67 | 37.71 ± 2.39 | 37.41 ± 2.39
Readability | 80.28 ± 33.26 | 75.16 ± 9.83 | 75.34 ± 8.13 | 75.25 ± 7.80 | 74.37 ± 8.00 | 74.80 ± 8.14
Entropy | 38.49 ± 12.25 | 26.03 ± 13.10 | 38.42 ± 7.93 | 40.30 ± 4.75 | 37.97 ± 4.07 | 34.69 ± 3.58
Coherence | 0.38 ± 0.05 | 0.46 ± 0.06 | 0.46 ± 0.06 | 0.46 ± 0.04 | 0.46 ± 0.04 | 0.46 ± 0.04

Table 5: Comparison of descriptive features for articles with number of
authors $N\in[1,5]$, as compared with human-written single-author stories for
the same prompts. Here, “H” $\to$ “Human”; “M”$\to$ Machine. * represents
statistical significance ($\textit{p}<0.01$) compared to human-written
stories.

###  3.4 Descriptive Statistics

We report the average and standard deviation of number of words, sentences,
vocabulary richness, readability scores, and coherence scores using
TextDescriptives Library777https://github.com/HLasse/TextDescriptives for all
parts of the story, as well as for different numbers of authors ($N$) in Table
5. There is a statistically significant difference between LLM-coauthored and
human-written stories in vocabulary richness measured by type-token-ratio
(TTR) for $N\in[2,3,4]$. For all other measures, our dataset’s machine-
generated stories follow similar distributions to human-written stories. We
thus ensure the quality of our dataset using automated measures of readability
and coherence by using human-generated stories as the reference text. Detailed
statistics for each LLM are provided in the Appendix (Table 12).

##  4 Authorship Analysis: Extending PAN tasks for multi-LLM scenario

Plagiarism Analysis, Authorship Identification, and Near-Duplicate Detection,
known as PAN tasks Bevendorff et al. (2023), have presented a persistent
challenge, establishing benchmarks for analyzing multi-authored text among
humans for more than 15 years. We extend the most common and repeated
authorship-related tasks from the PAN multi-human-author task suite to the
multi-LLM scenario. We then fine-tune and report performance using the
following 5 baseline methods: Multinomial Naive Bayes (MNB) Losada and
Azzopardi (2008), Support Vector Machine (SVM) Vapnik (1998), BERT Kenton and
Toutanova (2019), ALBERT Lan et al. (2019), and RoBERTa Liu et al. (2019).

###  4.1 Task 1: Is a story written by multiple authors or not?

We randomly sample articles from the single-LLM authored stories i.e. $N=1$ as
the negative class v/s articles from the multi-authored settings where
$N\in[2,3,4,5]$ as the positive class. We sample from the single-LLM stories
to keep the class distribution equal, based on the number of articles for each
$N$. From Table 6, we see that for all methods, the performance at $N=5$ is
higher than for $N=2$, gradually increasing with the value of $N$. Stories
that have a higher number of authors are more distinct from single-authored
ones. We conjecture that introducing more authors in the article might lead to
more variations in the text, making stories with $N=5$ authors most easily
distinguishable from stories without any such variations i.e. $N=1$.

Task 1: Multi-author or not
---
Method | N=2 | N=3 | N=4 | N=5
MNB | 0.83 | 0.86 | 0.86 | 0.87
SVM | 0.78 | 0.8 | 0.83 | 0.82
BERT | 0.85 | 0.85 | 0.83 | 0.87
ALBERT | 0.83 | 0.89 | 0.9 | 0.92
RoBERTa | 0.88 | 0.92 | 0.9 | 0.93
AVG | 0.83 | 0.86 | 0.86 | 0.88

Table 6: Performance is shown as F1-scores. AVG denotes average F1-score for
each $N$. For each $N\geq 2$ we evaluate the classifiers on their ability to
distinguish the stories from those written with $N=1$. Best performing method
is in bold and second highest underlined for each N.

###  4.2 Task 2: How many authors have written a story?

The second task is to predict the number of authors involved in generating a
story. For the CollabStory dataset, this means that class labels $\in[1,5]$.
From Table 7 we see that the task of predicting exactly how many authors have
co-written a story is easiest for $N=1$ in conjunction with findings from Task
1 that showed that multi-authored text can be more easily distinguished from
single-authored text. Thus, here too it seems to be easiest to separate the
single-authored texts from $N\geq 2$. However, for multi-authored stories,
only BERT and RoBERTa perform better than other baselines (>0.72 F1),
especially for $N\in[4,5]$. Overall, the performance across this task is low.

Task 2: Predict Number of Authors (N)
---
Method | N=1 | N=2 | N=3 | N=4 | N=5
MNB | 0.72 | 0.50 | 0.49 | 0.48 | 0.51
SVM | 0.68 | 0.43 | 0.52 | 0.40 | 0.54
BERT | 0.79 | 0.70 | 0.64 | 0.75 | 0.81
ALBERT | 0.70 | 0.57 | 0.55 | 0.53 | 0.65
RoBERTa | 0.76 | 0.68 | 0.63 | 0.72 | 0.74
AVG | 0.73 | 0.58 | 0.57 | 0.58 | 0.65

Table 7: All scores are F1-scores. AVG denotes average F1-score for each $N$.
This task is particularly challenging with only the single-authored stories
($N=1$) being correctly classified. For all multi-authored texts, BERT and
RoBERTa perform better than others. Best performing method is in bold and
second highest underlined for each N.

###  4.3 Task 3: Authorship Verification

This is a pair-wise sentence classification task where the goal is to predict
if two adjacent sentences are written by the same author or not. For this
task, we used all the sentences at LLM-LLM boundaries, that is the last
sentence of part $i$ and the first sentence of part $i+1$. The negative class
data samples were sampled as random pairs of consecutive sentences within each
story part.

Task 3: Authorship Verification
---
Method | N=2 | N=3 | N=4 | N=5
MNB | 0.65 | 0.64 | 0.60 | 0.63
SVM | 0.63 | 0.62 | 0.62 | 0.63
BERT | 0.73 | 0.73 | 0.73 | 0.71
ALBERT | 0.92 | 0.89 | 0.89 | 0.89
RoBERTa | 0.91 | 0.91 | 0.90 | 0.89
AVG | 0.76 | 0.75 | 0.74 | 0.75

Table 8: All scores are F1-scores and AVG denotes average F-1 scores for each
$N$ for the task of detecting authorship boundaries between sentence pairs. We
see that generally, performance is slightly higher for $N=2$ across all
classifiers. Best performing method is in bold and second highest underlined
for each N.

From Table 8, we see that transformers-based fine-tuned methods perform well
at this task. We also note that detecting sentence authorship boundaries seems
to be slightly easier for the 2-author case than for $N\geq 3$.

###  4.4 Task 4: Authorship Attribution

Authorship Attribution involves predicting exactly who the author of a text
article is. In the case of multi-LLM text, we design this task such that each
data sample is homogeneous or each part is written by a single author and the
classifier’s task is to identify its author.

Task 4: Authorship Attribution
---
N=1
Method | Orca | Olmo | Llama | Mistral | Gemma | AVG
MNB | - | 0.70 | 0.71 | 0.64 | 0.99 | 0.76
SVM | - | 0.61 | 0.68 | 0.58 | 0.97 | 0.71
BERT | - | 0.70 | 0.71 | 0.64 | 0.99 | 0.76
ALBERT | - | 0.78 | 0.73 | 0.70 | 0.99 | 0.80
RoBERTa | - | 0.73 | 0.70 | 0.68 | 0.99 | 0.78
N=2
Method | Orca | Olmo | Llama | Mistral | Gemma | AVG
MNB | 0.49 | 0.51 | 0.52 | 0.51 | 0.92 | 0.62
SVM | 0.51 | 0.55 | 0.54 | 0.59 | 0.79 | 0.62
BERT | 0.54 | 0.54 | 0.63 | 0.58 | 0.95 | 0.68
ALBERT | 0.56 | 0.58 | 0.63 | 0.59 | 0.96 | 0.69
RoBERTa | 0.49 | 0.62 | 0.60 | 0.56 | 0.94 | 0.68
N=3
Method | Orca | Olmo | Lama | Mistral | Gemma | AVG
MNB | - | 0.60 | 0.67 | 0.63 | 0.94 | 0.71
SVM | - | 0.57 | 0.65 | 0.57 | 0.82 | 0.65
BERT | - | 0.58 | 0.69 | 0.67 | 0.95 | 0.72
ALBERT | - | 0.64 | 0.71 | 0.68 | 0.95 | 0.75
RoBERTa | - | 0.71 | 0.71 | 0.67 | 0.96 | 0.76
N=4
Method | Orca | Olmo | Llama | Mistral | Gemma | AVG
MNB | - | 0.58 | 0.65 | 0.63 | 0.91 | 0.69
SVM | - | 0.58 | 0.67 | 0.59 | 0.80 | 0.66
BERT | - | 0.59 | 0.70 | 0.68 | 0.93 | 0.73
ALBERT | - | 0.66 | 0.73 | 0.70 | 0.93 | 0.75
RoBERTa | - | 0.66 | 0.68 | 0.64 | 0.93 | 0.73
N=5
Method | Orca | Olmo | Llama | Mistral | Gemma | AVG
MNB | 0.54 | 0.54 | 0.56 | 0.53 | 0.86 | 0.61
SVM | 0.56 | 0.61 | 0.60 | 0.54 | 0.79 | 0.62
BERT | 0.60 | 0.57 | 0.62 | 0.54 | 0.93 | 0.65
ALBERT | 0.58 | 0.55 | 0.65 | 0.62 | 0.92 | 0.66
RoBERTa | 0.56 | 0.69 | 0.61 | 0.58 | 0.92 | 0.67

Table 9: F1-scores for identifying the author of story parts across articles
written by different numbers of authors. The 5 columns show each of the labels
or authors. AVG denotes average F1-scores across all authors. Best performing
method is in bold and second highest underlined.

From Table 9, we see that most of the authors seem hard to identify
irrespective of the value of $N$, except for Gemma. We were expecting
attribution to be easier the fewer the number of authors since the length of
the parts contributed by each author would be longer. But there does not seem
to be any such correlation in our dataset i.e. length contributed by each
author does not correlate with their detection. To further explore why Gemma
was easily identifiable, we manually inspected a random sample of 100 articles
for discernible features or peculiarities in story parts written by Gemma v/s
all other authors. We provide examples of story parts generated by Gemma and
other LLMs in the Appendix (Table 11). We found that the text generated by
Gemma is not noticeably different from that of other authors. We leave a
deeper analysis of potential factors, such as training data-induced biases,
story part, or author order for future work.

##  5 Discussion

PAN Task equivalent | Task Description: Multi-LLM scneario | Real-world Implications
---|---|---
Predict multi-author or not |  To determine if a text includes content from multiple LLMs or not |  Credit Assignment and Intellectual Property (IP) regulation
Predict number of authors |  To predict the number of LLMs involved in writing an article |  Keeping track of LLM-LLM agent interactions in growing open-source market
Author Verification |  To detect when authorship switches between LLMs |  To detect perjury, misinformation injection, falsifying editing in news articles, and text obfuscation
Authorship Attribution |  Predicting who wrote each text segment? |  Plagiarism detection
Style Change Detection and Attribution |  Finding all positions in the text where authorship changes and who wrote each segment |  Classroom settings: Academic Integrity, detecting use of multiple open-source and free-to-use LLMs to surpass detection methods

Table 10: Real-world implications of the tasks involved in understanding LLM-
LLM collaboration for writing tasks

Recent developments have significantly advanced LLM-assisted writing, sparking
widespread discussions about the nature of authorship. Beyond using LLMs for
paraphrasing, editing, and enhancing text, there exists an extreme scenario
where text is generated entirely by multiple LLMs. Our work addresses this
extreme case, raising several nuanced authorship concerns: Who should be
considered the true creative source in such a situation? Should all LLMs
involved be credited? Or should the human developers designing the prompts be
acknowledged as the primary authors? Moreover, should the LLM that contributed
the most—whether in terms of word count, narrative depth, or plot twists—be
granted greater ownership? There have been recent works addressing this
question of ownership. For example, Joshi and Vogel (2024) and Dhillon et al.
(2024) found that humans felt a sense of higher ownership when they wrote
longer prompts and when the AI assistant-generated text length was shorter,
respectively. The question of authorship when text is repeatedly paraphrased
using LLMs has also been deeply considered by Tripto et al. (2023), who find
that the notion of authorship is task-dependent and cannot be generalized.

These questions have profound implications for various stakeholders in the
burgeoning socio-technical system of generative AI. Our research introduces
authorship-related tasks using CollabStory, which can help address these
concerns by accurately discerning the usage of multiple LLMs in texts. Our
extension of PAN-inspired authorship tasks is closely linked to real-world
implications, as follows:

#### Task 1: Predict multi-author or not

In the rapidly expanding and fiercely competitive market for LLMs, the
ownership of content and the ability to prove the origins of creative work are
becoming increasingly crucial. As the market evolves, closed-source LLMs are
implementing stricter regulations and demanding credit assignment under
various distribution licensing norms. In this context, the capacity to
demonstrate that a text incorporates generated output from multiple LLMs is
essential. This capability can effectively prevent any single stakeholder or
developer from erroneously claiming exclusive rights to the content, thereby
bolstering the defense against wrongful intellectual property (IP) claims.

#### Task 2: Predict number of authors

Predicting the exact number of LLMs involved in the writing process can help
keep track of the frequency and extent to which LLMs are used collaboratively,
as more and more models enter the open-source market. This is essential to
understand whether such usage improves task performance or introduces
inefficiencies beyond a certain threshold. Understanding the optimal number of
LLMs or the degree to which LLMs can leverage each other’s strengths in
writing tasks is vital. It ensures effective collaboration without unnecessary
complexity, maximizing the benefits of combined model capabilities while
avoiding overkill and collaboration for its own sake.

#### Task 3: Author Verification

With LLMs increasingly paraphrasing and editing each other’s texts, it becomes
crucial to identify which spans were generated by different LLMs. Consider a
scenario where a news article is paraphrased by one LLM and subsequently
edited by another, with the latter introducing fallacies or misinformation. In
such cases, discerning the contributions of each LLM is essential for
identifying malicious LLM agents or the infiltration of critical content, such
as media and news articles. This capability has significant applications,
including detecting perjury and combating the adversarial obfuscation of text,
thus maintaining the integrity and reliability of information.

#### Task 4: Authorship Attribution

Identifying the exact LLMs responsible for authoring a text is crucial for
detecting and addressing plagiarism. This is particularly important in
academic settings, where students might use closed-source LLMs without
complying with content ownership and usage declaration regulations. This is
possible also in cases where content from one LLM is being posed as that from
another to claim higher ability or quality. An example of such a situation
might be in a bid to motivate financial investors hoping to monetize and
utilize LLMs for specific domains (such as medical applications, educational
tools, and creative content generation).

##  6 Conclusion

In this work, we present CollabStory, the first exclusively LLM-LLM or
machine-machine collaborative story dataset, and demonstrate the tasks it
enables. We elaborate on why multi-LLM authorship tasks are crucial to study
by discussing their real-world implications (summarized in Table 10). We will
soon need “Catch As Catch Can” methods to not only find all points where
authorship changes within an article (style change detection) but also
simultaneously attribute each independent segment to the specific LLM author
(attribution). As more and more LLMs are becoming easier to access, malicious
actors could combine texts from different LLMs to evade automated and in-built
misinformation flaggers, or students might circumvent credibility checks by
having different LLMs write different sections of an academic article. Thus,
CollabStory was developed as a resource with long-form stories written by
multiple LLMs to support the development and expansion of tasks and methods
that can help address incoming challenges brought by LLM-LLM interactions.

## Limitations

Our work demonstrates one way of collecting a collaborative multi-LLM dataset.
However, several variants are possible. Of course, as the LLM space is ever
evolving, newer LLMs (e.g. Llama 3) became available as we were already
collecting this dataset. Another aspect is that our dataset was collected in a
uniform manner such that all LLMs contributed somewhat equal portions of text
to a story. The next step would be to train a routing algorithm or a
randomizer that could generate non-uniform collaborative texts. Our current
analysis is unable to account for this setting and we leave this for future
work. Additionally, the iterative generation process is resource-intensive and
not easily scalable. We also acknowledge that LLM tasks beyond story writing
are essential for a deeper understanding of how LLMs collaborate in open-ended
generation tasks.

## Ethics Statement

Using LLMs for creative story writing could relay some of the biases and
harmful stereotypes present in the LLMs original training data since all our
LLMs are trained on data from the internet. This is an important consideration
before or during the dissemination of any such generated texts or stories.
Transparency of the source of generated articles is important to avoid
deception or wrongful content attribution. With creative writing tasks, it is
also important to address any impact on creative professionals and guidelines
to ensure that LLMs help enhance rather than undermine human creativity. We
study LLM story-writing as a means to better prepare for a future of LLM-
generated creative texts that might be misused in classroom settings, to
manipulate public opinion on social media forums, and also to protect human
writers against plagiarism amongst many other potential non-ethical usages.

## References

  * Akoury et al. (2020) Nader Akoury, Shufan Wang, Josh Whiting, Stephen Hood, Nanyun Peng, and Mohit Iyyer. 2020.  STORIUM: A Dataset and Evaluation Platform for Machine-in-the-Loop Story Generation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6470–6484, Online. Association for Computational Linguistics.
  * Bevendorff et al. (2023) Janek Bevendorff, Ian Borrego-Obrador, Mara Chinea-Ríos, Marc Franco-Salvador, Maik Fröbe, Annina Heini, Krzysztof Kredens, Maximilian Mayerl, Piotr Pęzik, Martin Potthast, et al. 2023.  Overview of pan 2023: Authorship verification, multi-author writing style analysis, profiling cryptocurrency influencers, and trigger detection: Condensed lab overview.  In _International Conference of the Cross-Language Evaluation Forum for European Languages_ , pages 459–481. Springer.
  * Chakrabarty et al. (2022) Tuhin Chakrabarty, Vishakh Padmakumar, and He He. 2022.  Help me write a poem - instruction tuning as a vehicle for collaborative poetry writing.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 6848–6863, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Clark et al. (2018) Elizabeth Clark, Anne Spencer Ross, Chenhao Tan, Yangfeng Ji, and Noah A Smith. 2018.  Creative writing with a machine in the loop: Case studies on slogans and stories.  In _23rd International Conference on Intelligent User Interfaces_ , pages 329–340.
  * Dhillon et al. (2024) Paramveer S Dhillon, Somayeh Molaei, Jiaqi Li, Maximilian Golub, Shaochun Zheng, and Lionel Peter Robert. 2024.  Shaping human-ai collaboration: Varied scaffolding levels in co-writing with language models.  In _Proceedings of the CHI Conference on Human Factors in Computing Systems_ , pages 1–18.
  * Du and Chilton (2023) Yulun Du and Lydia Chilton. 2023.  Storywars: A dataset and instruction tuning baselines for collaborative story understanding and generation.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3044–3062.
  * Fan et al. (2018) Angela Fan, Mike Lewis, and Yann Dauphin. 2018.  Hierarchical neural story generation.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 889–898.
  * Groeneveld et al. (2024) Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, et al. 2024.  Olmo: Accelerating the science of language models.  _arXiv preprint arXiv:2402.00838_.
  * Jiang et al. (2023) Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023.  Mistral 7b.  _arXiv preprint arXiv:2310.06825_.
  * Joshi and Vogel (2024) Nikhita Joshi and Daniel Vogel. 2024.  Writing with ai lowers psychological ownership, but longer prompts can help.  _arXiv preprint arXiv:2404.03108_.
  * Kenton and Toutanova (2019) Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of NAACL-HLT_ , pages 4171–4186.
  * Kim et al. (2023) Jeongyeon Kim, Sangho Suh, Lydia B Chilton, and Haijun Xia. 2023.  Metaphorian: Leveraging large language models to support extended metaphor creation for science writing.  In _Proceedings of the 2023 ACM Designing Interactive Systems Conference_ , pages 115–135.
  * Lan et al. (2019) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019.  Albert: A lite bert for self-supervised learning of language representations.  In _International Conference on Learning Representations_.
  * Lee et al. (2022) Mina Lee, Percy Liang, and Qian Yang. 2022.  Coauthor: Designing a human-ai collaborative writing dataset for exploring language model capabilities.  In _Proceedings of the 2022 CHI conference on human factors in computing systems_ , pages 1–19.
  * Li et al. (2023a) Huao Li, Yu Chong, Simon Stepputtis, Joseph Campbell, Dana Hughes, Charles Lewis, and Katia Sycara. 2023a.  Theory of mind for multi-agent collaboration via large language models.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 180–192, Singapore. Association for Computational Linguistics.
  * Li et al. (2023b) Yuan Li, Yixuan Zhang, and Lichao Sun. 2023b.  Metaagents: Simulating interactions of human behaviors for llm-based task-oriented coordination via collaborative generative agents.  _arXiv preprint arXiv:2310.06500_.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Liu et al. (2023) Zijun Liu, Yanzhe Zhang, Peng Li, Yang Liu, and Diyi Yang. 2023.  Dynamic llm-agent network: An llm-agent collaboration framework with agent team optimization.  _arXiv preprint arXiv:2310.02170_.
  * Losada and Azzopardi (2008) David E Losada and Leif Azzopardi. 2008.  Assessing multivariate bernoulli models for information retrieval.  _ACM Transactions on Information Systems (TOIS)_ , 26(3):1–46.
  * Mukherjee et al. (2023) Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. 2023.  Orca: Progressive learning from complex explanation traces of gpt-4.  _arXiv preprint arXiv:2306.02707_.
  * Pan et al. (2024) Bo Pan, Jiaying Lu, Ke Wang, Li Zheng, Zhen Wen, Yingchaojie Feng, Minfeng Zhu, and Wei Chen. 2024.  Agentcoord: Visually exploring coordination strategy for llm-based multi-agent collaboration.  _arXiv preprint arXiv:2404.11943_.
  * Singh et al. (2023) Nikhil Singh, Guillermo Bernal, Daria Savchenko, and Elena L Glassman. 2023.  Where to hide a stolen elephant: Leaps in creative writing with multimodal machine intelligence.  _ACM Transactions on Computer-Human Interaction_ , 30(5):1–57.
  * Team et al. (2024) Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. 2024.  Gemma: Open models based on gemini research and technology.  _arXiv preprint arXiv:2403.08295_.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Tripto et al. (2023) Nafis Irtiza Tripto, Saranya Venkatraman, Dominik Macko, Robert Moro, Ivan Srba, Adaku Uchendu, Thai Le, and Dongwon Lee. 2023.  A ship of theseus: Curious cases of paraphrasing in llm-generated texts.  _arXiv preprint arXiv:2311.08374_.
  * Vapnik (1998) VN Vapnik. 1998.  Statistical learning theory.
  * Yang et al. (2022) Daijin Yang, Yanpeng Zhou, Zhiyuan Zhang, Toby Jia-Jun Li, and LC Ray. 2022.  Ai as an active writer: Interaction strategies with generated text in human-ai collaborative fiction writing.  In _Joint International Conference on Intelligent User Interfaces Workshops: APEx-UI, HAI-GEN, HEALTHI, HUMANIZE, TExSS, SOCIALIZE (IUI-WS 2022)_ , pages 56–65. CEUR-WS Team.
  * Yeh et al. (2024) Catherine Yeh, Gonzalo Ramos, Rachel Ng, Andy Huntington, and Richard Banks. 2024.  Ghostwriter: Augmenting collaborative human-ai writing experiences through personalization and agency.  _arXiv preprint arXiv:2402.08855_.
  * Yuan et al. (2022) Ann Yuan, Andy Coenen, Emily Reif, and Daphne Ippolito. 2022.  Wordcraft: story writing with large language models.  In _27th International Conference on Intelligent User Interfaces_ , pages 841–852.
  * Zeng et al. (2024) Zijie Zeng, Lele Sha, Yuheng Li, Kaixun Yang, Dragan Gašević, and Guangliang Chen. 2024.  Towards automatic boundary detection for human-ai collaborative hybrid essay in education.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 38, pages 22502–22510.
  * Zhang et al. (2024) Yang Zhang, Shixin Yang, Chenjia Bai, Fei Wu, Xiu Li, Xuelong Li, and Zhen Wang. 2024.  Towards efficient llm grounding for embodied multi-agent collaboration.  _arXiv preprint arXiv:2405.14314_.
  * Zhong et al. (2023) Wenjie Zhong, Jason Naradowsky, Hiroya Takamura, Ichiro Kobayashi, and Yusuke Miyao. 2023.  Fiction-writing mode: An effective control for human-machine collaborative writing.  In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 1752–1765.

##  Appendix A Appendix

###  A.1 Examples of story parts written by Gemma followed by other authors

We inspected 100 randomly sampled stories and inspected the parts written by
Gemma in search of visibly discernible features or peculiarities that might
have explained the ease of its detection as compared to other LLMs. To the
naked eye, this text seemed to be similar to all the other parts of the story
since we subjected all the story parts to the same filtration process. We
suspect that other factors might be in play here such as the author order or
tone differences, and leave this exploration to future attempts (Table 11 with
examples of story samples in next page).

###  A.2 LLM Prompting

For for all $N\geq 2$, we provided the summary of the story so far as an input
in the prompt. To make sure that our story parts had smooth continuity, we
also used the last sentences of the story so far as input. This made sure that
the generating LLM has access to the last sentence in addition to the overall
storyline to continue the story as seamlessly as possible. This second input
is denoted as “last sentence from previous part” in Table 3. Other than this,
our prompt for the three types of story sections only differed in the
instruction of writing either the “beginning”, “continue”, or “conclude” the
storyline so far. We also had to add an instruction to stop the LLMs from
generating any additional instructions as from our pilot study, we found that
some LLMs (Orca and Llama) would often first generate a rephrasing or more
detailed version of our instruction before generating the actual story
content.

###  A.3 Dataset Cleaning

For each prompt, we gave each LLM 20 maximum attempts to re-generate that
particular story part if it fell 15 or more words shorter than the goal length
in the previous iteration. Despite this, we had instances of very short story
parts that would have made the average article length too short or led to a
very skewed representation of one LLM v/s the rest. Thus, we discarded such
stories. Additionally, we were able to notice two formatting peculiarities for
Gemma and LLama. Particularly, Gemma’s story parts often began with a short
title for the section it was to generate surrounded by “###” for example “###
The return of the Jedi ###”. Llama on the other hand was appending a “The end”
whenever it was its turn to write the ending part of a story. We removed all
cases of these two substrings using regular expressions search and deletion as
a means to unify the flow of the story across all LLMs and to make sure
particular LLMs weren’t identifiable only due to such formatting details. We
also removed all extra spaces from the stories and any repetitions of the
instructions in rare cases.

Gemma |  Not Gemma
---|---
The chase has begun, and adrenaline floods his veins. The blinding headlights pierce through the night, illuminating the frantic dance of tires on asphalt. The adrenaline-fueled chase weaves through the labyrinth of city streets, sirens screaming like tormented spirits. The driver, hands white on the wheel, desperately maneuvers, adrenaline masking the fear gnawing at his conscience. Every corner, every alleyway, feels like a potential trap, each sudden movement a potential assault. The police officer, unwavering in his pursuit, his seasoned instincts guiding him through the chaos. He maneuvers his cruiser with precision, anticipating the driver’s next move. His heart pounds in his chest, adrenaline battling with the growing sense of urgency. He knows the city like the back of his hand, weaving through traffic, his eyes glued to the fleeing vehicle. As the chase intensifies, the driver’s desperation escalates. He risks collisions, weaving through pedestrians, pushing the limits of his vehicle’s performance. The officer remains calm, focused, his training taking over. He knows he must bring this chase to an end, but he also understands the risks involved. |  Swallowing his fear, he decides to take the next exit and lead his pursuers through the winding, narrow streets of the old town. The streets are alive with the sound of market vendors calling out their wares, children laughing, and the faint sounds of a distant band playing music. The narrow, cobblestone alleys are lined with ancient buildings, each one more beautiful than the last. He knows that this labyrinth of streets will give him an advantage, but it also increases the risk of colliding with a pedestrian or another car. Despite the danger, he continues to navigate the maze of alleys, turning sharply at each intersection. He can hear the revving of engines and the screeching of tires behind him, but he refuses to look back. He knows that every twist and turn could be his last, but he is determined to outsmart his pursuers and escape. As he rounds the final corner, he sees the safety of the forest beyond the old town walls. He grits his teeth, puts the pedal to the metal, and bursts through the gates, leaving the chaos of the chase behind him.
Emily stood frozen, her heart pounding in her chest. A sense of awe washed over her as she approached the base of the tree and noticed a faint glow emanating from beneath the ground. Curiosity overwhelmed her, and she cautiously lowered a hand into the soft earth. As her fingers brushed against something cold and smooth, a surge of energy flowed through her body. A deep voice rumbled in her mind, "You have awakened. The spirit of the wolf within you has been awakened." The earth trembled beneath her feet, and a swirling mist began to envelop Emily. She felt her bones shifting, her muscles growing taut. Her reflection in the nearby pond transformed from a girl to a magnificent silver wolf. She barked, a primal call echoing through the woods. The glow from beneath the oak tree intensified, and a majestic wolf spirit emerged, its eyes glowing with wisdom and power. The spirit guided Emily through the forest, teaching her the secrets of her newfound lineage. She learned of the ancient bond between wolves and humans, the importance of loyalty and bravery. |  As she embraced her new identity, she found herself a part of an intricate web of relationships. Her new pack, made up of both humans and wolves, accepted her with open arms. Together, they faced challenges, both natural and supernatural. Emily’s courage and quick thinking saved the pack on more than one occasion. Her transformation brought her closer to her true self, and she no longer felt lost or alone. The bonds of loyalty and friendship grew stronger as they worked together to protect their territory. Emily felt a deep sense of belonging, and she knew that she had found her place in the world. The woods, once a place of solitude, had become her home, a place of love and acceptance. Emily’s life as a wolf shifter was not without its difficulties, but she faced each challenge with the same bravery and determination that had always defined her. She had found her true identity, and nothing would ever shake her sense of belonging or her commitment to her pack. As the sun set over the glade, Emily looked up at the stars, her heart filled with gratitude and love.
In the unspoken words hanging between us, Sarah and I understand that the journey to healing will be arduous. There will be moments of doubt, days when the weight feels unbearable, when the waves of sadness threaten to pull us under. But we are not in this together. We are stronger than our individual struggles. Together, we will learn to navigate the storms, to embrace the sunshine, to rediscover who we are beyond the baggage we once carried. |  Sarah’s smile brightens my day, and I know that this new chapter in our lives is going to be filled with adventure, growth, and most importantly, love. We’ll stumble, we’ll trip, but we’ll always have each other to pick us back up and move forward, stronger than before. I take Sarah’s hand and squeeze it tightly.
Her words seemed to hang in the air like acrid smoke, but I knew better. The voice, the chilling certainty in its delivery, had belonged to something far more sinister. I glanced at the others, their faces pale, their eyes wide with disbelief. "There’s no way this is real," one whispered, her voice trembling. But the undeniable presence of death in our living room told us otherwise. The figure shifted, its skeletal hands brushing the floor with an unholy whisper. "Your time has come," it hissed, its voice like nails on a chalkboard. Panic surged through me, my limbs frozen in terror. The others stumbled around, scrambling for escape, but their movements were clumsy, their screams muted by fear. In that moment, I realized that the only way out was through the front door, directly into the path of the creature that had come for me. With a surge of adrenaline, I broke free from the group, sprinting towards the exit. |  As I reached the door, it swung open to reveal Tia, the figure from my nightmares, standing there. But instead of fear, I felt relief. She had a warm smile on her face, and her eyes were full of compassion. "You have been chosen to bear the mark of the guardian," she explained, holding out the hourglass. "I have protected this town for generations, and now it’s your turn to carry on the tradition." I hesitated for a moment, but then I accepted the hourglass with a sense of pride and purpose. Together, Tia and I faced the darkness that lurked outside, ready to protect our town and its people from the evil that sought to harm them. From that day on, I became the guardian of my town, using the powers granted to me by Tia to keep the peace and protect those around me. The people of the town were grateful, and I knew that I had found a new purpose in life.

Table 11: As can be seen from a few examples, the text written by Gemma is not
necessarily different from that of other authors (in the second column). Thus,
from our manual inspection, we were unable to attribute any noticeable
features to the higher authorship attribution performance for Gemma.

Feature | Author | K=1 | K=2 | K=3 | K=4 | K=5
---|---|---|---|---|---|---
# Words | Gemma | 172.97 ± 16.47 | 157.17 ± 36.22 | 124.51 ± 46.13 | 129.76 ± 47.16 | 133.75 ± 46.90
| Llama | 172.51 ± 19.91 | 170.88 ± 13.28 | 173.35 ± 15.29 | 174.76 ± 16.18 | 172.23 ± 19.33
| Mistral | 177.25 ± 12.61 | 182.24 ± 12.88 | 178.09 ± 22.71 | 178.82 ± 19.47 | 178.15 ± 22.69
| Olmo | 168.01 ± 8.69 | 197.91 ± 18.93 | 194.89 ± 23.64 | 192.64 ± 26.60 | 191.41 ± 30.28
| Orca | 174.45 ± 22.20 | 175.61 ± 10.42 | 178.11 ± 15.62 | 178.01 ± 14.37 | 177.89 ± 16.57
Lexical Diversity | Gemma | 0.67 ± 0.04 | 0.67 ± 0.05 | 0.70 ± 0.07 | 0.69 ± 0.07 | 0.68 ± 0.06
| Llama | 0.60 ± 0.05 | 0.61 ± 0.05 | 0.59 ± 0.05 | 0.58 ± 0.05 | 0.58 ± 0.06
| mistral | 0.64 ± 0.04 | 0.62 ± 0.04 | 0.61 ± 0.05 | 0.61 ± 0.04 | 0.61 ± 0.05
| Olmo | 0.62 ± 0.05 | 0.57 ± 0.06 | 0.58 ± 0.06 | 0.58 ± 0.06 | 0.57 ± 0.07
| Orca | 0.62 ± 0.05 | 0.62 ± 0.04 | 0.61 ± 0.04 | 0.61 ± 0.05 | 0.60 ± 0.05
Readability | Gemma | 75.95 ± 8.53 | 77.13 ± 9.70 | 75.13 ± 12.29 | 74.11 ± 11.81 | 73.28 ± 12.96
| Llama | 83.11 ± 8.35 | 82.61 ± 8.50 | 82.75 ± 9.00 | 80.88 ± 10.10 | 80.13 ± 9.19
| Mistral | 81.04 ± 8.58 | 83.99 ± 8.40 | 81.59 ± 10.07 | 82.02 ± 9.09 | 79.91 ± 9.44
| Olmo | 80.78 ± 9.01 | 83.31 ± 9.87 | 81.41 ± 9.81 | 80.55 ± 10.73 | 80.45 ± 10.20
| Orca | 83.08 ± 8.54 | 82.51 ± 8.45 | 80.86 ± 9.65 | 79.95 ± 9.97 | 79.51 ± 9.80
Coherence | Gemma | 0.49 ± 0.07 | 0.47 ± 0.08 | 0.47 ± 0.08 | 0.47 ± 0.08 | 0.48 ± 0.08
| Llama | 0.44 ± 0.08 | 0.47 ± 0.08 | 0.47 ± 0.08 | 0.47 ± 0.08 | 0.49 ± 0.08
| mistral | 0.46 ± 0.07 | 0.44 ± 0.07 | 0.45 ± 0.07 | 0.45 ± 0.07 | 0.46 ± 0.07
| Olmo | 0.47 ± 0.08 | 0.43 ± 0.07 | 0.45 ± 0.08 | 0.45 ± 0.08 | 0.45 ± 0.08
| Orca | 0.44 ± 0.07 | 0.45 ± 0.08 | 0.45 ± 0.08 | 0.46 ± 0.08 | 0.47 ± 0.07

Table 12: Descriptive Statistics or Features for stories generated by
different authors for different parts of the stories. Here, “K” represents the
part of the story written, i.e. K=1 corresponds to the first part of the
story, K=2 referees to the second part, and so on.