# Benchmarking LLMs on the Semantic Overlap Summarization Task

John Salvador, Naman Bansal, Mousumi Akter, Souvika Sarkar,
Anupam Das, and Shubhra Kanti Karmaker (“Santu”)
Big Data Intelligence Lab
Department of Computer Science and Software Engineering
College of Engineering, Auburn University
{jms0256, nbansal, mza0170, szs0239<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

Semantic Overlap Summarization (SOS) is a constrained multi-document
summarization task, where the constraint is to capture the common/overlapping
information between two alternative narratives. While recent advancements in
Large Language Models (LLMs) have achieved superior performance in numerous
summarization tasks, a benchmarking study of the SOS task using LLMs is yet to
be performed. As LLMs’ responses are sensitive to slight variations in prompt
design, a major challenge in conducting such a benchmarking study is to
systematically explore a variety of prompts before drawing a reliable
conclusion. Fortunately, very recently, the TELeR taxonomy has been proposed
which can be used to design and explore various prompts for LLMs. Using this
TELeR taxonomy and $15$ popular LLMs, this paper comprehensively evaluates
LLMs on the SOS Task, assessing their ability to summarize overlapping
information from multiple alternative narratives. For evaluation, we report
well-established metrics like ROUGE, BERTscore, and SEM-$F_{1}$ on two
different datasets of alternative narratives. We conclude the paper by
analyzing the strengths and limitations of various LLMs in terms of their
capabilities in capturing overlapping information111The code and datasets used
to conduct this study are available at
https://anonymous.4open.science/r/llm_eval-E16D.

##  1 Introduction

Large Language Models (LLMs) represent a groundbreaking advancement in the
research landscape of Natural Language Processing (NLP) and Artificial
Intelligence (AI). Trained on large bodies of text data, LLMs excel in
generating coherent and human-like text. These models have been evaluated in a
wide range of NLP tasks Bubeck et al. (2023); Dai et al. (2022); Du et al.
(2022); Smith et al. (2022) across several areas including software
development, law, and medicine Schäfer et al. (2024); School (2023);
Thirunavukarasu et al. (2023). However, there are still areas and tasks where
LLMs have yet to be rigorously evaluated. One such task is Semantic Overlap
Summarization (SOS) Bansal et al. (2022c); Karmaker Santu et al. (2018), where
the goal is to summarize the common/overlapping information between two
alternative narratives.

In this paper, we conduct a comprehensive benchmarking study of the SOS task
using 15 popular LLMs to perform this task. Conducting such a benchmarking
study is challenging because of the large variations in LLMs’ performance when
different prompt types/styles are used, and different degrees of detail are
provided in the prompts. Indeed, Rodriguez et al. (2023) shows varying
performance on the CM1 dataset Hayes et al. (2006) across many different
prompts with F1-scores ranging from $0.21$ to $0.54$, exhibiting a $0.33$
point difference. Fortunately, To address this issue, Santu and Feng (2023)
recently proposed a general taxonomy that can be used to design diverse
prompts with specific properties in order to perform a wide range of complex
tasks. Using this TELeR taxonomy, we devised 5 distinct types of prompts with
different degrees of detail to perform the SOS task on two different
alternative narratives data sets. One data set is the previously introduced
AllSides dataset released by Bansal et al. (2022c) and the second one is our
original contribution which we name as the PrivacyPolicyPairs (3P) dataset.

Figure 1: Example of a SOS task using two alternative privacy policy
narratives.

Figure 1 illustrates an example of the SOS task when comparing two alternative
privacy policies in the 3P data set, where the green text denotes the output
(common information) from two input privacy policies, one from Google and one
from Apple. In this case, we have two competing platforms that provide similar
types of services (e.g. Cloud Storage or Streaming Services) and each company
lays out its practices when handling your private information. These documents
contain important information regarding your privacy but can be long and
cumbersome to read. SOS helps by briefly identifying the common practices
followed by each company.

For evaluation, we report well-established metrics like ROUGE, BERTscore, and
SEM-$F_{1}$ on both Allsides and 3P data sets for each combination of LLMs and
prompt level ($15$ LLMs $\times 5$ prompts$=105$ combinations in total). We
conclude the paper by analyzing the strengths and limitations of various LLMs
in performing the Semantic Overlap Task, offering valuable insights into their
capabilities in capturing and synthesizing overlapping information from
multiple narratives.

##  2 Related Work

Text Summarization: SOS is essentially a summarization task. Over the past two
decades, many document summarization approaches have been investigated Zhong
et al. (2019). The two most popular among them are extractive approaches (Cao
et al., 2018; Narayan et al., 2018; Wu and Hu, 2018; Zhong et al., 2020) and
abstractive approaches (Bae et al., 2019; Liu et al., 2017; Nallapati et al.,
2016). Some researchers have also tried combining extractive and abstractive
approaches (Chen and Bansal, 2018; Hsu et al., 2018; Zhang et al., 2019).

The SOS Task: Technically, Semantic Overlap Summarization can be framed as a
multi-document summarization task, i.e., multi-seq-to-seq task Goldstein et
al. (2000); Yasunaga et al. (2017); Zhao et al. (2020); Ma et al. (2020);
Meena et al. (2014); Lebanoff et al. (2018); Fabbri et al. (2019). However,
unlike typical multi-document summarizing tasks, the SOS task aims to
summarize multiple alternative narratives with an overlapping constraint
Bansal et al. (2022d), i.e., the output should only contain the common
information from both input narratives Santu et al. (2018). The availability
of data for this task is relatively small so recently Bansal et al. (2022a)
proposed a method for training models by utilizing synthetically generated
data.

Transformers and LLMs: Encoder-decoder-based transformer models have recently
gained a lot of attraction, especially for abstractive summarization tasks,
Rush et al. (2015); Chopra et al. (2016); Zhou et al. (2017); Paulus et al.
(2017). Training a generic language model on a large corpus of data and then
transferring/fine-tuning it for the summarization job has become a standard
approach Radford et al. (2019); Devlin et al. (2019); Lewis et al. (2019);
Xiao et al. (2020); Yan et al. (2020); Zhang et al. (2019); Raffel et al.
(2019). Transformer architecture features more parallelizable training,
better-scaling properties, and a built-in attention mechanism, allowing large
language models (LLMs) to emerge. Made up of billions of parameters, many LLMs
like GPT-2 Radford et al. , and LLaMA Touvron et al. (2023a) have showcased
strong abilities at generating text. Then, with the introduction of
Reinforcement Learning From Human Feedback Ouyang et al. (2022), LLMs became
even more powerful, allowing users to interact with them as data with natural
language queries.

Prompt Engineering for LLMs: “Prompt Engineering” is a crucial technique for
maximizing the utility of LLMs in various tasks Zhou et al. (2022). It
involves crafting and revising the query or context to elicit the desired
response or behavior from LLMs Brown et al. (2022). Prompt engineering is an
iterative process requiring multiple trial and error runs Shao et al. (2023).
In fact, differences in prompts along several key factors can significantly
impact the accuracy and performance of LLMs in complex tasks. To address this
issue, Santu and Feng (2023) recently proposed the TELeR taxonomy, which can
serve as a unified standard for benchmarking LLMs’ performances by exploring a
wide variety of prompts in a structured manner.

Figure 2: TELeR Taxonomy proposed by Santu and Feng (2023): (<Turn,
Expression, Level of Details, Role>)

##  3 Background

###  3.1 Semantic Overlap Summarization Task

Semantic Overlap Summarization (SOS) is a task aimed at extracting and
condensing shared information between two input documents, $D_{A}$ and
$D_{B}$. The output, denoted as $D_{O}$, is generated in natural language and
only includes information present in both input documents. The task is framed
as a constrained multi-seq-to-seq (text generation) task, where brevity is
emphasized to minimize the repetition of overlapping content. The output can
be extractive summaries, abstractive summaries, or a combination of both
Karmaker Santu et al. (2018).

Furthermore, SOS adheres to the commutative property, meaning the order of
input documents doesn’t affect the output summary;
$D_{A}\cap_{O}D_{B}=D_{B}\cap_{O}D_{A}$. To facilitate research in this area,
Bansal et al. (2022c) introduced the AllSides dataset for training and
evaluation, which we also used for evaluation in this work.

###  3.2 Prompting With TELeR Taxonomy

As shown in Figure 2, the TELeR taxonomy introduced by Santu and Feng (2023)
categorizes complex task prompts based on four criteria.

  1. 1.

Turn: This refers to the number of turns or shots used while prompting an LLM
to accomplish a complex task. In general, prompts can be classified as either
single or multi-turn.

  2. 2.

Expression: This refers to the style of expression for interacting with the
LLM, such as questioning or instructing.

  3. 3.

Level of Details: This dimension of prompt style deals with the granularity or
depth of question or instruction. Prompts with higher levels of detail provide
more granular instructions. For example, Level 0 has no directive, i.e., ,
questions or instructions. On the other hand, Level 4 prompts have to specify
the particular sub-tasks, an explanation of the output, and the basis for
evaluation of the output generated by the LLM.

  4. 4.

Role: Some LLMs provide users with the option of specifying the role of the
system. The response of LLM can vary due to changes in role definitions in
spite of the fact that the prompt content remains unchanged.

The taxonomy outlines $7$ distinct levels starting from level 0 to level 6.
With each increase in level comes an increase in complexity of the prompt. In
level 0, only data/context is provided with no further instruction. Level 1
extends level 0 by providing single-sentence instruction. Then level 2 extends
level 1, and so on, until level 6, where all characteristics of previous
levels are provided along with the additional instruction for the LLM to
explain its output. For more details on the TELeR taxonomy and its
applications, see Santu and Feng (2023).

##  4 The Benchmark Datasets

We experimented on two datasets designed specifically for the Semantic Overlap
Summarization task.

###  4.1 The AllSides Data

The AllSides dataset is collected from AllSides.com, a third-party online news
forum known for presenting news and information from various political
perspectives. Bansal et al. (2022c) crawled news articles from AllSides.com to
build the dataset, focusing on narratives covering $2,925$ events. These
articles provide contrasting perspectives from media outlets affiliated with
“Left” and “Right” political leanings. Additionally, each event includes a
factual description labeled as “Theme” by AllSides, which can serve as a
neutral perspective for readers (ground truth for common facts).

The creation of the testing dataset involved human annotation with the goal of
generating several comprehensive reference summaries for narrative pairs. A
total of $150$ narrative pairs, each comprising divergent perspectives from
the “Left” and “Right” wing media, were randomly sampled, narrating $150$
unique events. Three human annotators were asked to write a summary of common
information present in both narratives for each of the $150$ events.
Initially, there was a discrepancy among the annotators regarding the
definition of “common/overlapping information” resulting in some annotations
containing few or no words. To address the issue, a filtering criterion was
set to retain only the narrative pairs where at least two of the annotators
wrote a minimum of 15 words as their reference summaries. Using this strategy,
the authors finally curated a test set with $137$ narrative pairs where each
sample had $4$ reference summaries: one from AllSides and three from human
annotators, resulting in a total of $548$ reference summaries.

###  4.2 The PrivacyPolicyPairs (3P) Data

We introduce the PrivacyPolicyPairs (3P) dataset, an additional evaluation set
for the SOS task, which contains $135$ thoroughly annotated samples. Our (3P)
dataset is built on the OPP-115 Corpus introduced by Wilson et al. (2016),
which comprises $115$ privacy policies (267K words) spanning $15$ sectors
(Arts, Shopping, News, etc.).

The OPP-115 Corpus: When building the OPP-115 Corpus, Wilson et al. (2016)
wanted to ensure that the corpus contained privacy policy samples that
appropriately represented the diverse set of areas where these types of
documents were used. To that end, the authors employ relevance-based website
pre-selection and sector-based sub-sampling. For relevance-based pre-
selection, the authors monitored and selected websites from the top Google
trends, resulting in $1,799$ unique websites. Then, they further organized the
websites into their $15$ sectors, ranked the websites in each sector according
to their frequency in the retrieved search results, and then selected eight
websites from each sector by randomly choosing two websites from each rank
quartile. Following this procedure, they were able to build a final dataset of
115 privacy policies across $15$ sectors.

Table 1: A single sample from the 3P dataset. For each sample, you are given
the category name, company names, the corresponding policy subsections, the
count of words in each policy, and the $3$ reference summaries. The
highlighted text shows the overlapping information.

To enhance the quality of their data, the authors developed an annotation
scheme to highlight the types of practices covered in each policy. Their
annotations consist of $10$ categories:

  * •

First Party Collection/Use

  * •

Third Party Sharing/Collection

  * •

User Choice/Control

  * •

User Access, Edit, & Deletion

  * •

Data Retention

  * •

Data Security

  * •

Policy Change

  * •

Do Not Track

  * •

International & Specific Audiences

  * •

Other

These annotations were also associated with a text span in the privacy policy
to denote where the labels were relevant. As a result, we are left with 23K
annotated practices across the 115 privacy policies.

####  4.2.1 Constructing the 3P Dataset

The 3P dataset includes pairs of passages taken from the OPP-115 corpus and
tasks the annotator with finding the semantically overlapping information
between them. A data sample is shown in Table 1. Each sample comprises $2$
source documents (two alternative privacy policy narratives), the category
they fall under, and $3$ reference overlap summaries. The company names and
count of words in each document are also included.

When curating this dataset, we wanted to ensure each passage pair had some
degree of overlap. To facilitate this goal, we reversed the process followed
by the original authors and grouped the documents back into their respective
sectors. Then, we built pairs of passages for each document in each sector
according to the practices they were originally labeled with. This process
resulted in $6110$ passage pairs across all sectors.

3P Dataset Statistics
---
# Samples | 135
Avg. # Words per Document | 331.00
Avg. # Words per Document Pair | 662.01
Avg. # Sentences per Document | 14.96
Avg. # Sentences per Document Pair | 28.99
Avg. # Words per Reference | 22.46
Avg. # Sentences per Reference | 1.75
Table 2: Dataset statistics for the 3P evaluation set. The set consists of 135
samples of document pairs with 3 references each.

Of the sectors, we chose to focus on three diverse sectors: eCommerce,
Technology, and Food and Drink. From these sectors, we have $346$ passage
pairs to annotate. Finally, we employed the same annotation task covered in
section 4.1. After annotating, resolving conflicts, and removing samples with
no overlap, the process yielded us $3$ annotations per privacy pair for a
total of $405$ annotations for $135$ high-quality samples. The final dataset
statistics are listed in Table 2.

##  5 Methodology

###  5.1 Evaluated Large Language Models

We choose to test our datasets using $6$ families of instruction-tuned LLMs,
totaling $15$ models. All evaluated models are listed in Table 3. For the
commercial LLMs (OpenAI and Google), we used their provided APIs for summary
generation, but for open-source LLMs, we used the huggingface transformers
library Wolf et al. (2020) to access model weights and perform generation on a
server with $4XA4500$ 20GB GPUs. For additional speedup, we leveraged the vLLM
library Kwon et al. (2023).

LLM Family | Model
---|---
Google PaLM2 | chat-bison-001 (May 2023)
Anil et al. (2023) |
OpenAI | gpt-3.5-turbo-0613
OpenAI (2023) | gpt-4-0613
| mosaicml/mpt-7b-chat (7B)
MosaicML MPT | mosaicml/mpt-30b-chat (30B)
Team (2023) | mosaicml/mpt-7b-instruct (7B)
| mosaicml/mpt-30b-instruct (30B)
| lmsys/vicuna-7b-v1.5 (7B)
LMSYS Vicuna | lmsys/vicuna-13b-v1.5 (13B)
Zheng et al. (2023) | lmsys/vicuna-7b-v1.5-16k (7B)
| lmsys/vicuna-13b-v1.5-16k (13B)
MistralAI | mistralai/Mistral-7B-Instruct-v0.1 (7B)
Jiang et al. (2023) | mistralai/Mistral-7B-Instruct-v0.2 (7B)
MetaAI Llama2 | meta-llama/Llama-2-7b-chat-hf (7B)
Touvron et al. (2023b) | meta-llama/Llama-2-13b-chat-hf (13B)

Table 3:  The list of models evaluated in this paper. We use $6$ families of
models, $2$ of which are closed source, and $4$ open source. The OpenAI models
are dated June 13, 2023 and chat-bison-001 is dated May 2023. Parameter counts
of open source models are included in parentheses. OpenAI and Google have not
reported the parameter counts of their models.

Zero-shot approaches to NLP tasks have gained popularity as LLM capabilities
continue to grow. For example, works from Sarkar et al. (2023, 2022) explore
their use cases in topic inference and text classification. To a similar
degree, we use LLMs in a zero-shot setting with TELeR.

For this study, we used TELeR levels $0$ through $4$ ($5$ out of the $7$). We
chose not to prompt using levels 5 and 6 because their use of retrieval
augmented prompting does not necessarily apply to the SOS task due to all
relevant context being present, i.e., the two source narratives are already
provided as part of the prompt. Furthermore, requirement number 5 for level 6
also specifies asking the LLM to explain its own output, which would
negatively affect the generated summaries during evaluation.

###  5.2 Designed TELeR Prompts

For each level, we use the following outline for our prompt design.

  * •

Level 0: {Document 1} {Document 2}

  * •

Level 1:

> Document 1: {Document 1 Content}
> Document 2: {Document 2 Content}
>
> Summarize the overlapping information between these two documents

  * •

Level 2:

> {Level 1 Prompt Text}
>
> This information must keep in mind the 5W1H facets of the documents. Do not
> include any uncommon information.

  * •

Level 3:

> {Level 1 Prompt Text}
>
>     * –
>
> This information must keep in mind the 5W1H facets of the documents.
>
>     * –
>
> Do not include uncommon information.

  * •

Level 4:

> {Level 3 Prompt Text}.
> Your response will be evaluated against a set of reference summaries. Your
> score will depend on how semantically similar your response is to the
> reference.

To ensure comprehensive prompt engineering, we created $5$ variations for each
TELeR level Santu and Feng (2023) from level 0 to level 4, i.e., we created
$5\times 5=25$ different unique prompts. To avoid confusion, we label these
variations A to E. Each variation set follows a similar layout to what was
previously mentioned but with variations in word choice, directive styles,
etc.

After designing our prompts, we evaluated them using $15$ LLMs for both
datasets. To ensure fairness, we report results only for the set of prompts
with the highest average score for each metric across all the evaluated LLMs.
Table 4 shows that prompt group A scored the highest averages across all
metrics, so we used that set for final evaluations.

Prompt Group | A | B | C | D | E
---|---|---|---|---|---
Sem-F1 (USE) | 0.37 | 0.35 | 0.34 | 0.34 | 0.34
Sem-F1 (Distil) | 0.41 | 0.40 | 0.39 | 0.39 | 0.39
Sem-F1 (RoBERTa) | 0.44 | 0.43 | 0.41 | 0.42 | 0.41
Rouge-1 | 0.24 | 0.23 | 0.22 | 0.22 | 0.22
Rouge-2 | 0.10 | 0.09 | 0.08 | 0.09 | 0.09
Rouge-L | 0.17 | 0.17 | 0.16 | 0.16 | 0.16
Rouge-L Sum | 0.18 | 0.17 | 0.16 | 0.17 | 0.17
BERTscore | 0.11 | 0.09 | 0.05 | 0.07 | 0.06

Table 4:  Average evaluation scores for each prompt trial. Each of these
trials are made up from $1$ of each level of TeLER prompt from levels $0$ to
$4$

###  5.3 Evaluation Metrics

ROUGE: ROUGE Lin (2004) is a family of metrics that score the lexical overlap
between the generated text and the reference text. We used 3 variations, R-1,
R-2, and R-L, which are widely adopted for evaluating text summarizing tasks.
However, despite its popularity, works like Akter et al. (2022) and Bansal et
al. (2022b) show that ROUGE is an unsuitable metric for comparing semantics.
For this reason we also evaluate using metrics that have been designed with
semantic awareness in mind.

Figure 3:  Best scores over each TeLER prompt level for the largest model of
each family of LLMs and for each dataset. Yellow shows BERTscore, green shows
ROUGE, and blue shows Sem-F1.

Dataset | Level | BERT | R-1 | R-2 | R-L | R-L- | Sem-F1 | Sem-F1 | Sem-F1
---|---|---|---|---|---|---|---|---|---
|  | score |  |  |  | Sum | (Distil) | (RoBERTa) | (USE)
Privacy | 0 | -0.069 | 0.119 | 0.035 | 0.085 | 0.095 | 0.398 | 0.444 | 0.288
Policy | 1 | 0.139 | 0.235 | 0.073 | 0.181 | 0.185 | 0.438 | 0.473 | 0.323
Pairs (3P) | 2 | 0.160 | 0.223 | 0.059 | 0.158 | 0.166 | 0.447 | 0.501 | 0.330
| 3 | 0.135 | 0.209 | 0.053 | 0.149 | 0.155 | 0.452 | 0.504 | 0.333
| 4 | 0.145 | 0.214 | 0.058 | 0.152 | 0.157 | 0.462 | 0.511 | 0.339
AllSides | 0 | 0.105 | 0.255 | 0.123 | 0.177 | 0.190 | 0.475 | 0.493 | 0.481
1 | 0.265 | 0.365 | 0.199 | 0.290 | 0.292 | 0.511 | 0.526 | 0.525
2 | 0.227 | 0.327 | 0.154 | 0.243 | 0.249 | 0.443 | 0.473 | 0.475
3 | 0.239 | 0.331 | 0.152 | 0.243 | 0.252 | 0.437 | 0.474 | 0.472
4 | 0.249 | 0.332 | 0.159 | 0.245 | 0.251 | 0.453 | 0.489 | 0.486

Table 5:  Average scores per metric broken down by level and dataset. highest
of each metric and dataset are in bold.

BERTscore: While ROUGE can only convey information about lexical overlap,
BERTscore is a metric that utilizes contextual embeddings from transformer
models like BERT to evaluate the semantic similarity between the generated
text and reference text. For this study, we compute BERTscore with the
hashcode roberta-large_ L17_ no-idf_ version=0.3.12(hug_
trans=4.36.2)-rescaled.

SEM-F1: While ROUGE and BERTscore are useful and powerful metrics, SEM-F1 was
specifically designed for the SOS task. SEM-F1 leverages rigorously fine-tuned
sentence encoders to evaluate the SOS task using sentence-level similarity. It
differs from BERTscore as BERTscore computes token-level similarity. For this
study, we compute SEM-F1 with $3$ underlying models: USE Cer et al. (2018),
RoBERTa Zhuang et al. (2021), and DistilRoBERTa Sanh et al. (2019).

##  6 Results

For clarity, we show our results for the largest or newest models of each
family in Figure 3. This Figure shows the highest scores achieved by each
model over the set of given TELeR prompts. The first two columns are for the
commercial LLMs (GPT-4 and PaLM2), and the $4$ columns on the right show our
open-source LLMs. In general, we observe that GPT-4 and PaLM2 beat the open-
source models across all benchmarks but surprisingly, Mistral-7B-Instruct-v0.2
exhibits competitive performance across all metrics despite being only a
7-billion parameter model. For a comprehensive breakdown of the model scores
achieved by each LLM and for each metric, refer to appendix (Figure 5).

AllSides Vs. 3P: In Table 5, we show average scores across all LLMs for each
TELeR level and dataset. The highest scores for each column in the table are
in bold. It is evident that TELeR level $1$ consistently performs the best
across the AllSides dataset, while on the 3P dataset, BERTscore and SEM-F1
suggest better performance in TELeR level $2$ and $4$ respectively. From this
table, we also observe that scores on the 3P dataset tend to trail ~$0.12$
points behind the AllSides dataset with the exception of the SEM-F1 (Distil)
and SEM-F1 (RoBERTa) metrics where the difference falls below $0.05$.

Figure 4:  Percentage of top prompts by level for each combination of model
and metric (out of 240 variations). Level 1 prompts generally outperform the
other TeLER levels for these datasets. The number $240$ comes from the
different combinations of LLMs ($15$), metrics ($8$), and datasets ($2$),
giving us $\text{\\# LLMs}\times\text{\\# Metrics}\times\text{\\#
Datasets}=15\times 8\times 2=240$

Best TELeR Prompt: Figure 4 shows the percentage of best scoring prompt levels
for each pair of metric and model totaling $240$ cases. TELeR level $1$
generally performs the best, beating other prompt levels $61.7\%$ of the time.
This preference towards level 1 is also seen in Table 5, where level 1 is
unanimously preferred across all metrics on the AllSides dataset and preferred
by half the metrics on the 3P dataset. It is also worth mentioning that SEM-F1
consistently favors the level 4 prompts on the 3P dataset, which was really
interesting.

##  7 Discussion

The main takeaways from our study are as follows:

Finding-1: “3P” dataset is harder than “AllSides”.

To elaborate, Table 5 shows a ~$0.12\%$ difference in evaluation scores
between the AllSides data and the 3P data. Given that the average document
length/word count for both datasets was comparable ($504.51$ for dataset
document pairs from AllSides and $662.01$ for the same from 3P), the
difference may arise due to the abundance of news data in the pertaining
corpus in comparison to privacy policy data, which is of limited quantity
compared to the real world event narratives. Interestingly, the average length
of the generated summary for the AllSides data was $39.51$ words, while for
the 3P data, it was $22.46$, indicating LLMs perhaps have seen more news data
than privacy policy narratives.

Finding-2: TELeR Level 1 generally outperforms other levels for the SOS task.

This was a little puzzling to us as we expected more detailed/specific prompts
would improve LLM’s performance. But, looking retrospectively, this result may
be explained by the fact the overlap summarizing task is still not very
complex, and a single one-sentence instruction (Level 1) is sufficient for
LLMs to understand the task properly.

Therefore, we hypothesized that maybe the models struggled to follow complex
directions (Levels 3 and 4) compared to the single-direction given at level
$1$. However, upon further inspection of a few samples, we found it to not
necessarily be the case. For example, with gpt-3.5-turbo, a sample with the
level 1 prompt resulted in the following:

However, in the case of the level 4 prompt, where explicit instructions are
given to capture the common entities, events, and locations, we got:

Such an example could suggest limitations in the ability of the metrics to
properly score responses with extra information such as "The common subjects
are …" or a more common response such as "Sure, here is an example of …".

##  8 Conclusion

In this study, we provide a comprehensive look into the capability of LLMs for
the Semantic Overlap Summarization (SOS) task. To facilitate robust
evaluation, we test on a previously created dataset and additionally introduce
the PrivacyPolicyPairs (3P) dataset. We leverage the TELeR prompting taxonomy
to devise a set of hand-crafted prompts that generate the highest scores we
could achieve with pre-trained instruction-tuned LLMs and found that: 1) TELeR
level $1$ prompts perform the best on average, and 2) generated summaries were
more accurate for the Allsides data-set than the 3P data-set in terms of
popular evaluation metrics.

##  9 Limitations

The primary limitation of this work is the size of the dataset. At only $135$
samples, it is not feasible to train a model. However while the size of the
dataset is small, there is a large amount of time and resource that is
required to build a dataset of this nature. Firstly, this dataset requires
that for each sample, we find two documents that share an overlapping
narrative. Second, each sample is annotated manually by $3$ people which for
this dataset results in $405$ annotations. That is without considering the
other annotations where no overlap was found. Third, there have been several
instances where disagreements need to be resolved which requires further
discussion among annotators. Nevertheless, this work effectively doubles the
amount of samples to evaluate on the SOS task. In the future, a larger scale
effort will be needed to increase the space of data for the SOS task.

Another limitation is that we did not perform any fine-tuning on these models.
All scores were obtained using the pre-trained weights for each model. This
means that it’s possible for additional performance to be gained using methods
like LoRA Hu et al. (2021). However the main goal of this study was to
benchmark LLMs to set new baselines for the SOS task. In that regard we
believe this to be an appropriate setup.

## References

  * Akter et al. (2022) Mousumi Akter, Naman Bansal, and Shubhra Kanti Karmaker Santu. 2022.  Revisiting automatic evaluation of extractive summarization task: Can we do better than rouge? In _Findings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland, May 22-27, 2022_ , pages 1547–1560. Association for Computational Linguistics.
  * Anil et al. (2023) Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernandez Abrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan Botha, James Bradbury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christopher A. Choquette-Choo, Aakanksha Chowdhery, Clément Crepy, Shachi Dave, Mostafa Dehghani, Sunipa Dev, Jacob Devlin, Mark Díaz, Nan Du, Ethan Dyer, Vlad Feinberg, Fangxiaoyu Feng, Vlad Fienber, Markus Freitag, Xavier Garcia, Sebastian Gehrmann, Lucas Gonzalez, Guy Gur-Ari, Steven Hand, Hadi Hashemi, Le Hou, Joshua Howland, Andrea Hu, Jeffrey Hui, Jeremy Hurwitz, Michael Isard, Abe Ittycheriah, Matthew Jagielski, Wenhao Jia, Kathleen Kenealy, Maxim Krikun, Sneha Kudugunta, Chang Lan, Katherine Lee, Benjamin Lee, Eric Li, Music Li, Wei Li, YaGuang Li, Jian Li, Hyeontaek Lim, Hanzhao Lin, Zhongtao Liu, Frederick Liu, Marcello Maggioni, Aroma Mahendru, Joshua Maynez, Vedant Misra, Maysam Moussalem, Zachary Nado, John Nham, Eric Ni, Andrew Nystrom, Alicia Parrish, Marie Pellat, Martin Polacek, Alex Polozov, Reiner Pope, Siyuan Qiao, Emily Reif, Bryan Richter, Parker Riley, Alex Castro Ros, Aurko Roy, Brennan Saeta, Rajkumar Samuel, Renee Shelby, Ambrose Slone, Daniel Smilkov, David R. So, Daniel Sohn, Simon Tokumine, Dasha Valter, Vijay Vasudevan, Kiran Vodrahalli, Xuezhi Wang, Pidong Wang, Zirui Wang, Tao Wang, John Wieting, Yuhuai Wu, Kelvin Xu, Yunhan Xu, Linting Xue, Pengcheng Yin, Jiahui Yu, Qiao Zhang, Steven Zheng, Ce Zheng, Weikang Zhou, Denny Zhou, Slav Petrov, and Yonghui Wu. 2023.  Palm 2 technical report.  ArXiv:2305.10403 [cs].
  * Bae et al. (2019) Sanghwan Bae, Taeuk Kim, Jihoon Kim, and Sang-goo Lee. 2019.  Summary level training of sentence rewriting for abstractive summarization.  _arXiv preprint arXiv:1909.08752_.
  * Bansal et al. (2022a) Naman Bansal, Mousumi Akter, and Shubhra Kanti Karmaker Santu. 2022a.  Learning to generate overlap summaries through noisy synthetic data.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , page 11765–11777, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Bansal et al. (2022b) Naman Bansal, Mousumi Akter, and Shubhra Kanti Karmaker Santu. 2022b.  Sem-f1: an automatic way for semantic evaluation of multi-narrative overlap summaries at scale.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , page 780–792, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Bansal et al. (2022c) Naman Bansal, Mousumi Akter, and Shubhra Kanti Karmaker Santu. 2022c.  Semantic overlap summarization among multiple alternative narratives: An exploratory study.  In _Proceedings of the 29th International Conference on Computational Linguistics_ , page 6195–6207, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
  * Bansal et al. (2022d) Naman Bansal, Mousumi Akter, and Shubhra Kanti Karmaker Santu. 2022d.  Semantic overlap summarization among multiple alternative narratives: An exploratory study.  In _Proceedings of the 29th International Conference on Computational Linguistics_ , pages 6195–6207, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
  * Brown et al. (2022) Hannah Brown, Katherine Lee, Fatemehsadat Mireshghallah, Reza Shokri, and Florian Tramèr. 2022.  What does it mean for a language model to preserve privacy?  In _2022 ACM Conference on Fairness, Accountability, and Transparency_ , pages 2280–2292.
  * Bubeck et al. (2023) Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. 2023.  Sparks of artificial general intelligence: Early experiments with gpt-4.  _arXiv preprint arXiv:2303.12712_.
  * Cao et al. (2018) Ziqiang Cao, Wenjie Li, Sujian Li, and Furu Wei. 2018.  Retrieve, rerank and rewrite: Soft template based neural summarization.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 152–161, Melbourne, Australia. Association for Computational Linguistics.
  * Cer et al. (2018) Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Brian Strope, and Ray Kurzweil. 2018.  Universal sentence encoder for english.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , page 169–174, Brussels, Belgium. Association for Computational Linguistics.
  * Chen and Bansal (2018) Yen-Chun Chen and Mohit Bansal. 2018.  Fast abstractive summarization with reinforce-selected sentence rewriting.  _arXiv preprint arXiv:1805.11080_.
  * Chopra et al. (2016) Sumit Chopra, Michael Auli, and Alexander M Rush. 2016.  Abstractive sentence summarization with attentive recurrent neural networks.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 93–98.
  * Dai et al. (2022) Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Zhifang Sui, and Furu Wei. 2022.  Why can gpt learn in-context? language models secretly perform gradient descent as meta optimizers.  _arXiv preprint arXiv:2212.10559_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Du et al. (2022) Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. 2022.  Glam: Efficient scaling of language models with mixture-of-experts.  In _International Conference on Machine Learning_ , pages 5547–5569. PMLR.
  * Fabbri et al. (2019) Alexander R Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir R Radev. 2019.  Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model.  _arXiv preprint arXiv:1906.01749_.
  * Goldstein et al. (2000) Jade Goldstein, Vibhu O Mittal, Jaime G Carbonell, and Mark Kantrowitz. 2000.  Multi-document summarization by sentence extraction.  In _NAACL-ANLP 2000 Workshop: Automatic Summarization_.
  * Hayes et al. (2006) J.H. Hayes, A. Dekhtyar, and S.K. Sundaram. 2006.  Advancing candidate link generation for requirements tracing: the study of methods.  _IEEE Transactions on Software Engineering_ , 32(1):4–19.
  * Hsu et al. (2018) Wan-Ting Hsu, Chieh-Kai Lin, Ming-Ying Lee, Kerui Min, Jing Tang, and Min Sun. 2018.  A unified model for extractive and abstractive summarization using inconsistency loss.  _arXiv preprint arXiv:1805.06266_.
  * Hu et al. (2021) Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021.  Lora: Low-rank adaptation of large language models.
  * Jiang et al. (2023) Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023.  Mistral 7b.  ArXiv:2310.06825 [cs].
  * Karmaker Santu et al. (2018) Shubhra Kanti Karmaker Santu, Chase Geigle, Duncan Ferguson, William Cope, Mary Kalantzis, Duane Searsmith, and Chengxiang Zhai. 2018.  Sofsat: Towards a setlike operator based framework for semantic analysis of text.  _ACM SIGKDD Explorations Newsletter_ , 20(2):21–30.
  * Kwon et al. (2023) Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023.  Efficient memory management for large language model serving with pagedattention.  In _Proceedings of the 29th Symposium on Operating Systems Principles_ , SOSP ’23, page 611–626, New York, NY, USA. Association for Computing Machinery.
  * Lebanoff et al. (2018) Logan Lebanoff, Kaiqiang Song, and Fei Liu. 2018.  Adapting the neural encoder-decoder framework from single to multi-document summarization.  _arXiv preprint arXiv:1808.06218_.
  * Lewis et al. (2019) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019.  Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  _arXiv preprint arXiv:1910.13461_.
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  * Liu et al. (2017) Linqing Liu, Yao Lu, Min Yang, Qiang Qu, Jia Zhu, and Hongyan Li. 2017.  Generative adversarial network for abstractive text summarization.  _arXiv preprint arXiv:1711.09357_.
  * Ma et al. (2020) Congbo Ma, Wei Emma Zhang, Mingyu Guo, Hu Wang, and Quan Z Sheng. 2020.  Multi-document summarization via deep learning techniques: A survey.  _arXiv preprint arXiv:2011.04843_.
  * Meena et al. (2014) Yogesh Kumar Meena, Ashish Jain, and Dinesh Gopalani. 2014.  Survey on graph and cluster based approaches in multi-document text summarization.  In _International Conference on Recent Advances and Innovations in Engineering (ICRAIE-2014)_ , pages 1–5. IEEE.
  * Nallapati et al. (2016) Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. 2016.  Abstractive text summarization using sequence-to-sequence rnns and beyond.  _arXiv preprint arXiv:1602.06023_.
  * Narayan et al. (2018) Shashi Narayan, Shay B Cohen, and Mirella Lapata. 2018.  Ranking sentences for extractive summarization with reinforcement learning.  _arXiv preprint arXiv:1802.08636_.
  * OpenAI (2023) OpenAI. 2023.  Gpt-4 technical report.  ArXiv:2303.08774 [cs].
  * Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022.  Training language models to follow instructions with human feedback.  ArXiv:2203.02155 [cs].
  * Paulus et al. (2017) Romain Paulus, Caiming Xiong, and Richard Socher. 2017.  A deep reinforced model for abstractive summarization.  _arXiv preprint arXiv:1705.04304_.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Dario Amodei, Daniela Amodei, Jack Clark, Miles Brundage, and Ilya Sutskever. 2019.  Better language models and their implications.  _OpenAI Blog https://openai. com/blog/better-language-models_.
  * (37) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  Language models are unsupervised multitask learners.
  * Raffel et al. (2019) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _arXiv preprint arXiv:1910.10683_.
  * Rodriguez et al. (2023) Alberto D. Rodriguez, Katherine R. Dearstyne, and Jane Cleland-Huang. 2023.  Prompts matter: Insights and strategies for prompt engineering in automated software traceability.  In _2023 IEEE 31st International Requirements Engineering Conference Workshops (REW)_ , page 455–464.
  * Rush et al. (2015) Alexander M Rush, Sumit Chopra, and Jason Weston. 2015.  A neural attention model for abstractive sentence summarization.  _arXiv preprint arXiv:1509.00685_.
  * Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019.  Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.  _ArXiv_ , abs/1910.01108.
  * Santu and Feng (2023) Shubhra Kanti Karmaker Santu and Dongji Feng. 2023.  Teler: A general taxonomy of llm prompts for benchmarking complex tasks.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , page 14197–14203, Singapore. Association for Computational Linguistics.
  * Santu et al. (2018) Shubhra Kanti Karmaker Santu, Chase Geigle, Duncan Ferguson, William Cope, Mary Kalantzis, Duane Searsmith, and Chengxiang Zhai. 2018.  Sofsat: Towards a setlike operator based framework for semantic analysis of text.  _ACM SIGKDD Explorations Newsletter_ , 20(2):21–30.
  * Sarkar et al. (2022) Souvika Sarkar, Dongji Feng, and Shubhra Kanti Karmaker Santu. 2022.  Exploring universal sentence encoders for zero-shot text classification.  In _Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , page 135–147, Online only. Association for Computational Linguistics.
  * Sarkar et al. (2023) Souvika Sarkar, Dongji Feng, and Shubhra Kanti Karmaker Santu. 2023.  Zero-shot multi-label topic inference with sentence encoders and llms.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , page 16218–16233, Singapore. Association for Computational Linguistics.
  * School (2023) Stanford Law School. 2023.  Large language models as fiduciaries: A case study toward robustly communicating with artificial intelligence through legal standards.
  * Schäfer et al. (2024) Max Schäfer, Sarah Nadi, Aryaz Eghbali, and Frank Tip. 2024.  An empirical evaluation of using large language models for automated unit test generation.  _IEEE Transactions on Software Engineering_ , 50(1):85–105.
  * Shao et al. (2023) Nan Shao, Zefan Cai, Chonghua Liao, Yanan Zheng, Zhilin Yang, et al. 2023.  Compositional task representations for large language models.  In _The Eleventh International Conference on Learning Representations_.
  * Smith et al. (2022) Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, et al. 2022.  Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model.  _arXiv preprint arXiv:2201.11990_.
  * Team (2023) MosaicML NLP Team. 2023.  Introducing mpt-7b: A new standard for open-source, commercially usable llms.  Accessed: 2024-01-30.
  * Thirunavukarasu et al. (2023) Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. 2023.  Large language models in medicine.  _Nature Medicine_ , 29(88):1930–1940.
  * Touvron et al. (2023a) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a.  Llama: Open and efficient foundation language models.  ArXiv:2302.13971 [cs].
  * Touvron et al. (2023b) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023b.  Llama 2: Open foundation and fine-tuned chat models.  ArXiv:2307.09288 [cs].
  * Virtanen et al. (2020) Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C J Carey, İlhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. 2020.  SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python.  _Nature Methods_ , 17:261–272.
  * Wilson et al. (2016) Shomir Wilson, Florian Schaub, Aswarth Abhilash Dara, Frederick Liu, Sushain Cherivirala, Pedro Giovanni Leon, Mads Schaarup Andersen, Sebastian Zimmeck, Kanthashree Mysore Sathyendra, N. Cameron Russell, Thomas B. Norton, Eduard Hovy, Joel Reidenberg, and Norman Sadeh. 2016.  The creation and analysis of a website privacy policy corpus.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , page 1330–1340, Berlin, Germany. Association for Computational Linguistics.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020.  Huggingface’s transformers: State-of-the-art natural language processing.  ArXiv:1910.03771 [cs].
  * Wu and Hu (2018) Yuxiang Wu and Baotian Hu. 2018.  Learning to extract coherent summary via deep reinforcement learning.  _arXiv preprint arXiv:1804.07036_.
  * Xiao et al. (2020) Dongling Xiao, Han Zhang, Yukun Li, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. 2020.  Ernie-gen: An enhanced multi-flow pre-training and fine-tuning framework for natural language generation.  _arXiv preprint arXiv:2001.11314_.
  * Yan et al. (2020) Yu Yan, Weizhen Qi, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei Zhang, and Ming Zhou. 2020.  Prophetnet: Predicting future n-gram for sequence-to-sequence pre-training.  _arXiv preprint arXiv:2001.04063_.
  * Yasunaga et al. (2017) Michihiro Yasunaga, Rui Zhang, Kshitijh Meelu, Ayush Pareek, Krishnan Srinivasan, and Dragomir Radev. 2017.  Graph-based neural multi-document summarization.  _arXiv preprint arXiv:1706.06681_.
  * Zhang et al. (2019) Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J Liu. 2019.  Pegasus: Pre-training with extracted gap-sentences for abstractive summarization.  _arXiv preprint arXiv:1912.08777_.
  * Zhao et al. (2020) Jinming Zhao, Ming Liu, Longxiang Gao, Yuan Jin, Lan Du, He Zhao, He Zhang, and Gholamreza Haffari. 2020.  Summpip: Unsupervised multi-document summarization with sentence graph compression.  In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pages 1949–1952.
  * Zheng et al. (2023) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023.  Judging llm-as-a-judge with mt-bench and chatbot arena.  ArXiv:2306.05685 [cs].
  * Zhong et al. (2020) Ming Zhong, Pengfei Liu, Yiran Chen, Danqing Wang, Xipeng Qiu, and Xuanjing Huang. 2020.  Extractive summarization as text matching.  _arXiv preprint arXiv:2004.08795_.
  * Zhong et al. (2019) Ming Zhong, Pengfei Liu, Danqing Wang, Xipeng Qiu, and Xuanjing Huang. 2019.  Searching for effective neural extractive summarization: What works and what’s next.  _arXiv preprint arXiv:1907.03491_.
  * Zhou et al. (2017) Qingyu Zhou, Nan Yang, Furu Wei, and Ming Zhou. 2017.  Selective encoding for abstractive sentence summarization.  _arXiv preprint arXiv:1704.07073_.
  * Zhou et al. (2022) Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2022.  Large language models are human-level prompt engineers.  _arXiv preprint arXiv:2211.01910_.
  * Zhuang et al. (2021) Liu Zhuang, Lin Wayne, Shi Ya, and Zhao Jun. 2021.  A robustly optimized bert pre-training approach with post-training.  In _Proceedings of the 20th Chinese National Conference on Computational Linguistics_ , page 1218–1227, Huhhot, China. Chinese Information Processing Society of China.

##  Appendix A Appendix

###  A.1 Additional Figures

Figure 5 shows a comprehensive breakdown of the best scores obtained by each
model for each dataset. Figure 6 shows Pearson’s correlation scores between
all metrics on both datasets. The Pearson scores were computed using the SciPy
library Virtanen et al. (2020)

Figure 5:  Best scores over each TeLER prompt level for all $15$ evaluated
LLMs and for each dataset. Yellow shows BERTscore, green shows ROUGE, and blue
shows Sem-F1.  Figure 6:  Correlation scores between all evaluation metrics.

###  A.2 Detailed Results

Tables 6 and 7 contain the full breakdown of evaluation scores for our
finalized set of TELeR prompts.

Model | Dataset | Level | Sem-F1 | Sem-F1 | Sem-F1 | R-1 | R-2 | R-L | R-L- | BERTscore
---|---|---|---|---|---|---|---|---|---|---
|  |  | (USE) | (Distil) | (RoBERTa) |  |  |  | Sum |
lmsys/vicuna-13b-v1.5-16k | all_sides | 0 | 0.25 | 0.3 | 0.3 | 0.16 | 0.07 | 0.12 | 0.13 | -0.29
lmsys/vicuna-13b-v1.5-16k | all_sides | 1 | 0.17 | 0.23 | 0.23 | 0.14 | 0.07 | 0.12 | 0.12 | -0.36
lmsys/vicuna-13b-v1.5-16k | all_sides | 2 | 0.18 | 0.24 | 0.23 | 0.15 | 0.07 | 0.12 | 0.13 | -0.33
lmsys/vicuna-13b-v1.5-16k | all_sides | 3 | 0.42 | 0.41 | 0.43 | 0.36 | 0.17 | 0.26 | 0.28 | 0.28
lmsys/vicuna-13b-v1.5-16k | all_sides | 4 | 0.47 | 0.45 | 0.48 | 0.37 | 0.19 | 0.28 | 0.29 | 0.3
lmsys/vicuna-13b-v1.5 | all_sides | 0 | 0.24 | 0.29 | 0.28 | 0.17 | 0.08 | 0.13 | 0.14 | -0.29
lmsys/vicuna-13b-v1.5 | all_sides | 1 | 0.2 | 0.26 | 0.25 | 0.16 | 0.08 | 0.13 | 0.13 | -0.3
lmsys/vicuna-13b-v1.5 | all_sides | 2 | 0.42 | 0.41 | 0.42 | 0.36 | 0.17 | 0.26 | 0.28 | 0.28
lmsys/vicuna-13b-v1.5 | all_sides | 3 | 0.42 | 0.41 | 0.43 | 0.35 | 0.16 | 0.26 | 0.27 | 0.27
lmsys/vicuna-13b-v1.5 | all_sides | 4 | 0.45 | 0.43 | 0.45 | 0.37 | 0.18 | 0.27 | 0.29 | 0.28
lmsys/vicuna-7b-v1.5-16k | all_sides | 0 | 0.38 | 0.39 | 0.4 | 0.24 | 0.12 | 0.17 | 0.19 | -0.04
lmsys/vicuna-7b-v1.5-16k | all_sides | 1 | 0.5 | 0.49 | 0.5 | 0.34 | 0.17 | 0.26 | 0.26 | 0.24
lmsys/vicuna-7b-v1.5-16k | all_sides | 2 | 0.43 | 0.42 | 0.44 | 0.35 | 0.17 | 0.27 | 0.28 | 0.25
lmsys/vicuna-7b-v1.5-16k | all_sides | 3 | 0.38 | 0.38 | 0.39 | 0.34 | 0.16 | 0.25 | 0.27 | 0.26
lmsys/vicuna-7b-v1.5-16k | all_sides | 4 | 0.36 | 0.35 | 0.37 | 0.29 | 0.14 | 0.22 | 0.23 | 0.2
lmsys/vicuna-7b-v1.5 | all_sides | 0 | 0.32 | 0.33 | 0.34 | 0.18 | 0.08 | 0.13 | 0.15 | -0.16
lmsys/vicuna-7b-v1.5 | all_sides | 1 | 0.5 | 0.5 | 0.5 | 0.35 | 0.19 | 0.26 | 0.27 | 0.25
lmsys/vicuna-7b-v1.5 | all_sides | 2 | 0.44 | 0.43 | 0.45 | 0.39 | 0.21 | 0.3 | 0.32 | 0.33
lmsys/vicuna-7b-v1.5 | all_sides | 3 | 0.4 | 0.39 | 0.41 | 0.34 | 0.17 | 0.26 | 0.27 | 0.28
lmsys/vicuna-7b-v1.5 | all_sides | 4 | 0.35 | 0.35 | 0.37 | 0.27 | 0.13 | 0.2 | 0.21 | 0.19
meta-llama/Llama-2-13b-chat-hf | all_sides | 0 | 0.58 | 0.57 | 0.59 | 0.26 | 0.13 | 0.17 | 0.2 | 0.21
meta-llama/Llama-2-13b-chat-hf | all_sides | 1 | 0.44 | 0.44 | 0.46 | 0.42 | 0.23 | 0.33 | 0.34 | 0.34
meta-llama/Llama-2-13b-chat-hf | all_sides | 2 | 0.54 | 0.49 | 0.52 | 0.35 | 0.16 | 0.25 | 0.26 | 0.29
meta-llama/Llama-2-13b-chat-hf | all_sides | 3 | 0.46 | 0.4 | 0.46 | 0.32 | 0.14 | 0.22 | 0.25 | 0.13
meta-llama/Llama-2-13b-chat-hf | all_sides | 4 | 0.5 | 0.46 | 0.51 | 0.32 | 0.14 | 0.22 | 0.24 | 0.2
meta-llama/Llama-2-7b-chat-hf | all_sides | 0 | 0.59 | 0.58 | 0.6 | 0.26 | 0.13 | 0.17 | 0.19 | 0.21
meta-llama/Llama-2-7b-chat-hf | all_sides | 1 | 0.6 | 0.57 | 0.59 | 0.42 | 0.23 | 0.32 | 0.32 | 0.38
meta-llama/Llama-2-7b-chat-hf | all_sides | 2 | 0.54 | 0.48 | 0.52 | 0.35 | 0.16 | 0.25 | 0.26 | 0.29
meta-llama/Llama-2-7b-chat-hf | all_sides | 3 | 0.5 | 0.45 | 0.49 | 0.33 | 0.14 | 0.24 | 0.24 | 0.26
meta-llama/Llama-2-7b-chat-hf | all_sides | 4 | 0.49 | 0.45 | 0.48 | 0.33 | 0.14 | 0.24 | 0.24 | 0.26
mistralai/Mistral-7B-Instruct-v0.1 | all_sides | 0 | 0.53 | 0.52 | 0.54 | 0.32 | 0.13 | 0.21 | 0.21 | 0.25
mistralai/Mistral-7B-Instruct-v0.1 | all_sides | 1 | 0.65 | 0.62 | 0.62 | 0.47 | 0.29 | 0.39 | 0.39 | 0.47
mistralai/Mistral-7B-Instruct-v0.1 | all_sides | 2 | 0.53 | 0.47 | 0.52 | 0.36 | 0.17 | 0.26 | 0.26 | 0.29
mistralai/Mistral-7B-Instruct-v0.1 | all_sides | 3 | 0.52 | 0.46 | 0.5 | 0.35 | 0.17 | 0.26 | 0.27 | 0.27
mistralai/Mistral-7B-Instruct-v0.1 | all_sides | 4 | 0.52 | 0.47 | 0.51 | 0.37 | 0.18 | 0.27 | 0.27 | 0.29
mistralai/Mistral-7B-Instruct-v0.2 | all_sides | 0 | 0.48 | 0.44 | 0.47 | 0.18 | 0.07 | 0.11 | 0.13 | 0.06
mistralai/Mistral-7B-Instruct-v0.2 | all_sides | 1 | 0.64 | 0.64 | 0.63 | 0.45 | 0.25 | 0.36 | 0.36 | 0.45
mistralai/Mistral-7B-Instruct-v0.2 | all_sides | 2 | 0.53 | 0.48 | 0.53 | 0.37 | 0.18 | 0.28 | 0.28 | 0.33
mistralai/Mistral-7B-Instruct-v0.2 | all_sides | 3 | 0.52 | 0.48 | 0.54 | 0.38 | 0.19 | 0.29 | 0.29 | 0.32
mistralai/Mistral-7B-Instruct-v0.2 | all_sides | 4 | 0.54 | 0.49 | 0.54 | 0.38 | 0.19 | 0.29 | 0.29 | 0.32
mosaicml/mpt-7b-chat | all_sides | 0 | 0.41 | 0.41 | 0.44 | 0.25 | 0.11 | 0.17 | 0.18 | 0.14
mosaicml/mpt-7b-chat | all_sides | 1 | 0.54 | 0.5 | 0.54 | 0.32 | 0.15 | 0.23 | 0.23 | 0.26
mosaicml/mpt-7b-chat | all_sides | 2 | 0.5 | 0.45 | 0.5 | 0.32 | 0.14 | 0.23 | 0.23 | 0.25
mosaicml/mpt-7b-chat | all_sides | 3 | 0.47 | 0.44 | 0.49 | 0.29 | 0.12 | 0.21 | 0.21 | 0.22
mosaicml/mpt-7b-chat | all_sides | 4 | 0.5 | 0.46 | 0.5 | 0.31 | 0.14 | 0.23 | 0.23 | 0.23
mosaicml/mpt-7b-instruct | all_sides | 0 | 0.59 | 0.57 | 0.6 | 0.28 | 0.2 | 0.23 | 0.24 | 0.25
mosaicml/mpt-7b-instruct | all_sides | 1 | 0.6 | 0.56 | 0.59 | 0.36 | 0.18 | 0.26 | 0.26 | 0.3
mosaicml/mpt-7b-instruct | all_sides | 2 | 0.52 | 0.46 | 0.5 | 0.28 | 0.12 | 0.2 | 0.2 | 0.18
mosaicml/mpt-7b-instruct | all_sides | 3 | 0.49 | 0.44 | 0.48 | 0.27 | 0.11 | 0.18 | 0.19 | 0.14
mosaicml/mpt-7b-instruct | all_sides | 4 | 0.53 | 0.5 | 0.53 | 0.28 | 0.15 | 0.2 | 0.22 | 0.17
mosaicml/mpt-30b-chat | all_sides | 0 | 0.57 | 0.56 | 0.58 | 0.27 | 0.14 | 0.18 | 0.19 | 0.2
mosaicml/mpt-30b-chat | all_sides | 1 | 0.52 | 0.51 | 0.54 | 0.25 | 0.14 | 0.19 | 0.2 | 0.17
mosaicml/mpt-30b-chat | all_sides | 2 | 0.46 | 0.44 | 0.47 | 0.22 | 0.09 | 0.15 | 0.16 | 0.11
mosaicml/mpt-30b-chat | all_sides | 3 | 0.44 | 0.42 | 0.47 | 0.2 | 0.08 | 0.14 | 0.15 | 0.07
mosaicml/mpt-30b-chat | all_sides | 4 | 0.45 | 0.44 | 0.48 | 0.23 | 0.1 | 0.16 | 0.16 | 0.09
mosaicml/mpt-30b-instruct | all_sides | 0 | 0.59 | 0.56 | 0.58 | 0.35 | 0.19 | 0.26 | 0.27 | 0.31
mosaicml/mpt-30b-instruct | all_sides | 1 | 0.6 | 0.56 | 0.57 | 0.39 | 0.19 | 0.32 | 0.32 | 0.41
mosaicml/mpt-30b-instruct | all_sides | 2 | 0.44 | 0.43 | 0.43 | 0.29 | 0.13 | 0.24 | 0.24 | 0.22
mosaicml/mpt-30b-instruct | all_sides | 3 | 0.48 | 0.47 | 0.47 | 0.33 | 0.14 | 0.28 | 0.27 | 0.27
mosaicml/mpt-30b-instruct | all_sides | 4 | 0.47 | 0.44 | 0.47 | 0.33 | 0.14 | 0.25 | 0.25 | 0.26
chat-bison-001 | all_sides | 0 | 0.55 | 0.52 | 0.54 | 0.22 | 0.09 | 0.15 | 0.17 | 0.15
chat-bison-001 | all_sides | 1 | 0.63 | 0.59 | 0.61 | 0.46 | 0.27 | 0.39 | 0.39 | 0.47
chat-bison-001 | all_sides | 2 | 0.5 | 0.46 | 0.49 | 0.35 | 0.16 | 0.26 | 0.26 | 0.28
chat-bison-001 | all_sides | 3 | 0.49 | 0.43 | 0.49 | 0.33 | 0.15 | 0.24 | 0.25 | 0.21
chat-bison-001 | all_sides | 4 | 0.55 | 0.5 | 0.54 | 0.37 | 0.18 | 0.28 | 0.28 | 0.3
gpt-3.5-turbo | all_sides | 0 | 0.53 | 0.52 | 0.54 | 0.32 | 0.13 | 0.21 | 0.21 | 0.26
gpt-3.5-turbo | all_sides | 1 | 0.63 | 0.59 | 0.62 | 0.48 | 0.28 | 0.41 | 0.41 | 0.46
gpt-3.5-turbo | all_sides | 2 | 0.51 | 0.47 | 0.51 | 0.36 | 0.18 | 0.28 | 0.28 | 0.3
gpt-3.5-turbo | all_sides | 3 | 0.51 | 0.47 | 0.51 | 0.37 | 0.18 | 0.27 | 0.28 | 0.28
gpt-3.5-turbo | all_sides | 4 | 0.51 | 0.47 | 0.52 | 0.36 | 0.18 | 0.26 | 0.27 | 0.29
gpt-4 | all_sides | 0 | 0.6 | 0.57 | 0.6 | 0.37 | 0.17 | 0.25 | 0.25 | 0.32
gpt-4 | all_sides | 1 | 0.65 | 0.61 | 0.64 | 0.47 | 0.26 | 0.38 | 0.38 | 0.44
gpt-4 | all_sides | 2 | 0.58 | 0.51 | 0.56 | 0.41 | 0.2 | 0.29 | 0.29 | 0.33
gpt-4 | all_sides | 3 | 0.58 | 0.5 | 0.55 | 0.4 | 0.2 | 0.29 | 0.29 | 0.33
gpt-4 | all_sides | 4 | 0.6 | 0.53 | 0.58 | 0.4 | 0.2 | 0.3 | 0.3 | 0.35

Table 6: All data collected on the AllSides dataset for our finalized set of
prompts

Model | Dataset | Level | Sem-F1 | Sem-F1 | Sem-F1 | R-1 | R-2 | R-L | R-L- | BERTscore
---|---|---|---|---|---|---|---|---|---|---
|  |  | (USE) | (Distil) | (RoBERTa) |  |  |  | Sum |
lmsys/vicuna-13b-v1.5-16k | privacy_policy | 0 | 0.17 | 0.26 | 0.27 | 0.06 | 0.02 | 0.05 | 0.05 | -0.38
lmsys/vicuna-13b-v1.5-16k | privacy_policy | 1 | 0.15 | 0.24 | 0.23 | 0.1 | 0.03 | 0.08 | 0.08 | -0.32
lmsys/vicuna-13b-v1.5-16k | privacy_policy | 2 | 0.25 | 0.37 | 0.41 | 0.19 | 0.05 | 0.14 | 0.15 | 0.1
lmsys/vicuna-13b-v1.5-16k | privacy_policy | 3 | 0.23 | 0.34 | 0.36 | 0.14 | 0.04 | 0.1 | 0.1 | -0.12
lmsys/vicuna-13b-v1.5-16k | privacy_policy | 4 | 0.29 | 0.4 | 0.44 | 0.18 | 0.05 | 0.13 | 0.14 | 0.08
lmsys/vicuna-13b-v1.5 | privacy_policy | 0 | 0.2 | 0.29 | 0.3 | 0.07 | 0.02 | 0.05 | 0.06 | -0.3
lmsys/vicuna-13b-v1.5 | privacy_policy | 1 | 0.19 | 0.29 | 0.29 | 0.11 | 0.04 | 0.09 | 0.09 | -0.22
lmsys/vicuna-13b-v1.5 | privacy_policy | 2 | 0.25 | 0.37 | 0.41 | 0.2 | 0.06 | 0.14 | 0.16 | 0.14
lmsys/vicuna-13b-v1.5 | privacy_policy | 3 | 0.25 | 0.38 | 0.41 | 0.17 | 0.05 | 0.12 | 0.13 | 0.08
lmsys/vicuna-13b-v1.5 | privacy_policy | 4 | 0.27 | 0.39 | 0.42 | 0.18 | 0.05 | 0.13 | 0.14 | 0.1
lmsys/vicuna-7b-v1.5-16k | privacy_policy | 0 | 0.22 | 0.32 | 0.33 | 0.06 | 0.02 | 0.04 | 0.05 | -0.27
lmsys/vicuna-7b-v1.5-16k | privacy_policy | 1 | 0.27 | 0.39 | 0.4 | 0.2 | 0.05 | 0.15 | 0.15 | 0.11
lmsys/vicuna-7b-v1.5-16k | privacy_policy | 2 | 0.24 | 0.36 | 0.39 | 0.2 | 0.05 | 0.14 | 0.16 | 0.13
lmsys/vicuna-7b-v1.5-16k | privacy_policy | 3 | 0.28 | 0.4 | 0.45 | 0.18 | 0.05 | 0.13 | 0.14 | 0.11
lmsys/vicuna-7b-v1.5-16k | privacy_policy | 4 | 0.3 | 0.41 | 0.44 | 0.18 | 0.05 | 0.13 | 0.13 | 0.1
lmsys/vicuna-7b-v1.5 | privacy_policy | 0 | 0.21 | 0.31 | 0.31 | 0.06 | 0.02 | 0.04 | 0.05 | -0.26
lmsys/vicuna-7b-v1.5 | privacy_policy | 1 | 0.25 | 0.37 | 0.39 | 0.17 | 0.05 | 0.12 | 0.13 | 0.06
lmsys/vicuna-7b-v1.5 | privacy_policy | 2 | 0.24 | 0.36 | 0.39 | 0.2 | 0.05 | 0.14 | 0.16 | 0.13
lmsys/vicuna-7b-v1.5 | privacy_policy | 3 | 0.24 | 0.37 | 0.4 | 0.17 | 0.04 | 0.13 | 0.14 | 0.09
lmsys/vicuna-7b-v1.5 | privacy_policy | 4 | 0.24 | 0.37 | 0.4 | 0.18 | 0.05 | 0.13 | 0.14 | 0.1
meta-llama/Llama-2-13b-chat-hf | privacy_policy | 0 | 0.29 | 0.4 | 0.44 | 0.1 | 0.03 | 0.07 | 0.09 | -0.07
meta-llama/Llama-2-13b-chat-hf | privacy_policy | 1 | 0.29 | 0.38 | 0.41 | 0.23 | 0.07 | 0.17 | 0.18 | 0.16
meta-llama/Llama-2-13b-chat-hf | privacy_policy | 2 | 0.3 | 0.41 | 0.46 | 0.18 | 0.05 | 0.13 | 0.14 | 0.11
meta-llama/Llama-2-13b-chat-hf | privacy_policy | 3 | 0.29 | 0.41 | 0.45 | 0.18 | 0.04 | 0.13 | 0.13 | 0.1
meta-llama/Llama-2-13b-chat-hf | privacy_policy | 4 | 0.28 | 0.39 | 0.43 | 0.16 | 0.04 | 0.11 | 0.12 | 0.07
meta-llama/Llama-2-7b-chat-hf | privacy_policy | 0 | 0.29 | 0.39 | 0.43 | 0.09 | 0.03 | 0.06 | 0.07 | -0.08
meta-llama/Llama-2-7b-chat-hf | privacy_policy | 1 | 0.32 | 0.44 | 0.46 | 0.23 | 0.07 | 0.17 | 0.17 | 0.16
meta-llama/Llama-2-7b-chat-hf | privacy_policy | 2 | 0.31 | 0.42 | 0.46 | 0.19 | 0.04 | 0.13 | 0.14 | 0.11
meta-llama/Llama-2-7b-chat-hf | privacy_policy | 3 | 0.31 | 0.41 | 0.46 | 0.18 | 0.04 | 0.13 | 0.13 | 0.11
meta-llama/Llama-2-7b-chat-hf | privacy_policy | 4 | 0.3 | 0.42 | 0.44 | 0.18 | 0.05 | 0.14 | 0.14 | 0.11
mistralai/Mistral-7B-Instruct-v0.1 | privacy_policy | 0 | 0.3 | 0.4 | 0.45 | 0.14 | 0.04 | 0.1 | 0.1 | 0.02
mistralai/Mistral-7B-Instruct-v0.1 | privacy_policy | 1 | 0.28 | 0.4 | 0.41 | 0.22 | 0.06 | 0.17 | 0.17 | 0.17
mistralai/Mistral-7B-Instruct-v0.1 | privacy_policy | 2 | 0.31 | 0.42 | 0.46 | 0.2 | 0.05 | 0.14 | 0.14 | 0.12
mistralai/Mistral-7B-Instruct-v0.1 | privacy_policy | 3 | 0.3 | 0.42 | 0.45 | 0.19 | 0.05 | 0.14 | 0.14 | 0.12
mistralai/Mistral-7B-Instruct-v0.1 | privacy_policy | 4 | 0.31 | 0.42 | 0.46 | 0.2 | 0.05 | 0.14 | 0.14 | 0.12
mistralai/Mistral-7B-Instruct-v0.2 | privacy_policy | 0 | 0.24 | 0.37 | 0.4 | 0.05 | 0.01 | 0.03 | 0.04 | -0.17
mistralai/Mistral-7B-Instruct-v0.2 | privacy_policy | 1 | 0.33 | 0.45 | 0.48 | 0.24 | 0.09 | 0.19 | 0.19 | 0.21
mistralai/Mistral-7B-Instruct-v0.2 | privacy_policy | 2 | 0.31 | 0.41 | 0.47 | 0.21 | 0.05 | 0.15 | 0.15 | 0.15
mistralai/Mistral-7B-Instruct-v0.2 | privacy_policy | 3 | 0.31 | 0.42 | 0.47 | 0.21 | 0.05 | 0.14 | 0.15 | 0.14
mistralai/Mistral-7B-Instruct-v0.2 | privacy_policy | 4 | 0.31 | 0.43 | 0.47 | 0.21 | 0.06 | 0.15 | 0.15 | 0.15
mosaicml/mpt-7b-chat | privacy_policy | 0 | 0.24 | 0.36 | 0.4 | 0.12 | 0.03 | 0.09 | 0.09 | -0.05
mosaicml/mpt-7b-chat | privacy_policy | 1 | 0.27 | 0.39 | 0.41 | 0.19 | 0.04 | 0.14 | 0.14 | 0.09
mosaicml/mpt-7b-chat | privacy_policy | 2 | 0.26 | 0.39 | 0.43 | 0.18 | 0.03 | 0.12 | 0.13 | 0.08
mosaicml/mpt-7b-chat | privacy_policy | 3 | 0.27 | 0.39 | 0.42 | 0.17 | 0.04 | 0.12 | 0.13 | 0.08
mosaicml/mpt-7b-chat | privacy_policy | 4 | 0.27 | 0.39 | 0.43 | 0.17 | 0.04 | 0.12 | 0.13 | 0.06
mosaicml/mpt-7b-instruct | privacy_policy | 0 | 0.32 | 0.43 | 0.47 | 0.13 | 0.04 | 0.09 | 0.1 | -0.03
mosaicml/mpt-7b-instruct | privacy_policy | 1 | 0.28 | 0.38 | 0.41 | 0.18 | 0.05 | 0.13 | 0.14 | 0.07
mosaicml/mpt-7b-instruct | privacy_policy | 2 | 0.27 | 0.38 | 0.41 | 0.15 | 0.04 | 0.11 | 0.12 | 0.03
mosaicml/mpt-7b-instruct | privacy_policy | 3 | 0.27 | 0.37 | 0.41 | 0.16 | 0.04 | 0.11 | 0.13 | 0.02
mosaicml/mpt-7b-instruct | privacy_policy | 4 | 0.27 | 0.38 | 0.41 | 0.14 | 0.03 | 0.1 | 0.11 | 0
mosaicml/mpt-30b-chat | privacy_policy | 0 | 0.29 | 0.41 | 0.45 | 0.12 | 0.04 | 0.08 | 0.09 | -0.01
mosaicml/mpt-30b-chat | privacy_policy | 1 | 0.31 | 0.42 | 0.45 | 0.18 | 0.05 | 0.14 | 0.14 | 0.08
mosaicml/mpt-30b-chat | privacy_policy | 2 | 0.31 | 0.41 | 0.45 | 0.11 | 0.03 | 0.08 | 0.09 | -0.02
mosaicml/mpt-30b-chat | privacy_policy | 3 | 0.31 | 0.41 | 0.45 | 0.13 | 0.04 | 0.09 | 0.1 | 0
mosaicml/mpt-30b-chat | privacy_policy | 4 | 0.31 | 0.42 | 0.45 | 0.14 | 0.04 | 0.11 | 0.11 | 0.02
mosaicml/mpt-30b-instruct | privacy_policy | 0 | 0.27 | 0.38 | 0.42 | 0.15 | 0.05 | 0.11 | 0.12 | 0.03
mosaicml/mpt-30b-instruct | privacy_policy | 1 | 0.28 | 0.38 | 0.4 | 0.22 | 0.06 | 0.18 | 0.18 | 0.21
mosaicml/mpt-30b-instruct | privacy_policy | 2 | 0.29 | 0.4 | 0.44 | 0.19 | 0.05 | 0.14 | 0.14 | 0.11
mosaicml/mpt-30b-instruct | privacy_policy | 3 | 0.28 | 0.4 | 0.44 | 0.19 | 0.05 | 0.14 | 0.15 | 0.11
mosaicml/mpt-30b-instruct | privacy_policy | 4 | 0.28 | 0.39 | 0.41 | 0.18 | 0.04 | 0.14 | 0.14 | 0.08
chat-bison-001 | privacy_policy | 0 | 0.26 | 0.36 | 0.4 | 0.09 | 0.02 | 0.07 | 0.08 | -0.1
chat-bison-001 | privacy_policy | 1 | 0.33 | 0.44 | 0.47 | 0.26 | 0.1 | 0.21 | 0.22 | 0.23
chat-bison-001 | privacy_policy | 2 | 0.32 | 0.42 | 0.47 | 0.22 | 0.06 | 0.16 | 0.16 | 0.15
chat-bison-001 | privacy_policy | 3 | 0.33 | 0.44 | 0.48 | 0.2 | 0.06 | 0.15 | 0.16 | 0.14
chat-bison-001 | privacy_policy | 4 | 0.33 | 0.44 | 0.48 | 0.22 | 0.07 | 0.17 | 0.17 | 0.16
gpt-3.5-turbo | privacy_policy | 0 | 0.34 | 0.44 | 0.48 | 0.14 | 0.05 | 0.1 | 0.11 | 0.04
gpt-3.5-turbo | privacy_policy | 1 | 0.34 | 0.46 | 0.49 | 0.27 | 0.1 | 0.22 | 0.22 | 0.26
gpt-3.5-turbo | privacy_policy | 2 | 0.33 | 0.44 | 0.49 | 0.21 | 0.06 | 0.15 | 0.15 | 0.15
gpt-3.5-turbo | privacy_policy | 3 | 0.34 | 0.44 | 0.49 | 0.22 | 0.07 | 0.16 | 0.16 | 0.16
gpt-3.5-turbo | privacy_policy | 4 | 0.34 | 0.45 | 0.49 | 0.23 | 0.07 | 0.16 | 0.16 | 0.17
gpt-4 | privacy_policy | 0 | 0.31 | 0.42 | 0.47 | 0.13 | 0.04 | 0.09 | 0.1 | 0.01
gpt-4 | privacy_policy | 1 | 0.34 | 0.47 | 0.49 | 0.26 | 0.08 | 0.2 | 0.2 | 0.22
gpt-4 | privacy_policy | 2 | 0.33 | 0.45 | 0.5 | 0.22 | 0.06 | 0.15 | 0.15 | 0.17
gpt-4 | privacy_policy | 3 | 0.33 | 0.45 | 0.5 | 0.21 | 0.05 | 0.14 | 0.14 | 0.15
gpt-4 | privacy_policy | 4 | 0.34 | 0.46 | 0.5 | 0.22 | 0.06 | 0.15 | 0.15 | 0.17

Table 7: All data collected on the PrivacyPolicy dataset using our finalized
set of prompts

###  A.3 More on Prompt Design

The prompt designs for each group mostly follow the format covered in section
3.2. Each prompt group included one prompt each for levels $0$ through $4$.
the data was consistently presented to the models at the begging of the prompt
in the format:

> Document 1:
> {{Document_1}}
>
>
> Document 2:
> {{Document_2}}

The differences occurred after the data was presented and also in the system
role definitions. Depending on the TELeR level, the prompt would change but
the overall goal remained the same. For example, the prompt for level $1$ in
group B says "In one sentence, please tell me the overlapping information
between Document 1 and Document 2".

The system role definitions were made to emphasize that the LLM would respond
according to the documents given and that responses should be brief. For
example, prompt group A used the system role: "You are to read several
documents and briefly provide information according to the user’s needs".

Each group were written as variations of the same goal to sweep over the
diverse set of ways to prompt the LLMs.


# Understanding Gamma-ray Burst Jet Structures from Afterglow Light Curves

Celia R. Tandon Stanford University, Palo Alto, CA 94305  CCS-2, Computational
Physics and Methods, Los Alamos National Lab, Los Alamos NM 87544  Nicole M.
Lloyd-Ronning CCS-2, Computational Physics and Methods, Los Alamos National
Lab, Los Alamos NM 87544  Department of Math, Science and Engineering,
University of New Mexico, Los Alamos, Los Alamos NM 87544

(Received June 1, 2019; Revised January 10, 2019)

###### Abstract

Gamma-ray bursts (GRBs), associated with the collapse of massive stars or the
collisions of compact objects, are the most luminous events in our universe.
However, there is still much to learn about the nature of the relativistic
jets launched from the central engines of these objects. We examine how jet
structure—that is, the energy and velocity distribution as a function of
angle—affects observed GRB afterglow light curves. Using the package
afterglowpy, we compute light curves arising from an array of possible jet
structures, and present the suite of models that can fit the coincident
electromagnetic observations of GW190814 (which is likely due to a background
AGN). Our work emphasizes not only the need for broadband spectral and timing
data to distinguish among jet structure models, but also the necessity for
high resolution radio follow-up to help resolve background sources that may
mimic a GRB afterglow.

††journal: RNAAS

##  1 Introduction

Gamma-Ray Bursts (GRBs) are brief (less than a second to hundreds of seconds)
but extremely luminous flashes of gamma-rays created by internal dissipation
processes in ultra relativistic jets launched from central compact object-disk
systems. As the jet front ploughs into the external medium and sweeps up
matter, it decelerates and produces long-lived afterglow emission across the
entire electromagnetic spectrum Piran (2004). Many important details of
relativistic jets—how they are launched, the internal processes that lead to
the gamma-ray emission, the degree of magnetization, the physics of their
collimation—are yet to be understood.

For short GRBs in particular, given the increased detection of coincident
gravitational wave (GW) emission from their progenitor systems (the merger of
two compact objects), the broadband GRB afterglow combined with gravitational
wave emission can provide deep insight into the physics of their progenitors.

One of the major open questions in understanding the emission from
electromagnetic counterparts to gravitational wave events is getting a handle
on the so-called structure of the jet. It has long been recognized that a GRB
jet will have some sort of angular structure—that is a distribution of the jet
outflow velocity and energy as a function of angle from the jet axis111The jet
will also have radial structure, but here we consider emission from the jet
front at roughly a singular radius. Urrutia et al. (2021) and Murguia-Berthier
et al. (2021), for example, both show that the jet interaction with the wind
can produce complicated jet structures, including an inverted like structure.

With the propsect of continued, improved LIGO observations as well as a number
of additional GW detectors slated to come online in the next few years, the
need for a solid understanding of potential electromagnetic counterparts to GW
events is stronger than ever. Below, we show how the angular structures of a
jet affect the observed afterglow light curves. We use the electromagnetic
emission coincident with GW190814 (likely a background AGN (Alexander et al.,
2021b)) to illustrate how a number of physically plausible GRB jet models can
reproduce the data. Our work emphasizes the need not only for broadband
spectral and timing modelling, but the necessity of high resolution radio
follow-up to rule out contamination from background sources.

##  2 Jet Models

###  2.1 Jet Structures

A structured jet is a GRB jet model where the isotropic-equivalent energy,
$E_{iso}$, of the blast wave is a function of the angle from the jet axis:

$E_{iso}=4\pi\frac{dE}{d\Omega}$ (1)

The particular angular structure of the jet is first determined by the jet
launching mechanism and then modified as the jet burrows out of the
surrounding ejecta debris or stellar envelope (Ryan et al., 2020).

Below are some common jet structures found in the literature. These models are
parameterized by a normalization $E_{iso}$, a width $\theta_{c}$, and a
truncation angle $\theta_{wing}$ outside of which the energy is zero.

  * •

Gaussian with Core: a Gaussian with a top-hat core of width $\theta_{c}$.

$E(\theta)=E_{iso}e^{-\frac{\theta^{2}}{2\theta^{2}_{c}}}$ (2)

  * •

Cone: a ”hollow” top-hat with constant isotropic-equivalent energy $E_{iso}$
between inner angle $\theta_{c}$ and outer angle $\theta_{wing}$.

$E(\theta)=\begin{cases}0&\text{if $\theta<\theta_{c}$}\\\ E_{iso}&\text{if
$\theta_{c}\leq\theta\leq\theta_{wing}$}\\\ \end{cases}$ (3)

  * •

Top Hat: a top-hat with constant isotropic-equivalent energy $E_{iso}$ inside
$\theta_{c}$.

$E(\theta)=\begin{cases}E_{iso}&\text{if $\theta\leq\theta_{c}$}\\\ 0&\text{if
$\theta>\theta_{c}$}\\\ \end{cases}$ (4)

###  2.2 Jet Parameters

Important parameters that define the structure of a jet and the resulting
afterglow light curve include:

  * •

$E_{iso}$: Isotropic equivalent kinetic energy (erg)

  * •

$n_{0}$: Circumburst density ($cm^{-3}$)

  * •

$\epsilon_{B}$: Fraction of energy in the magnetic fields

  * •

$\epsilon_{e}$: Fraction of energy in the electrons

  * •

$p$: Electron energy distribution index

  * •

$\theta_{obs}$: Viewing angle (radians)

The latter in particular plays an important role in how we see the light curve
and therefore interpret the jet structure. This is where the gravitational
wave emission is additionally important as it can provide some constraint on
this value.

##  3 GW 190814

The compact object binary merger GW190814 was detected in gravitational waves
by Advanced LIGO/Virgo on August 14, 2019. This merger is notable due to the
uncertain nature of the binary’s lighter-mass counterpart–either the heaviest
known neutron star or the lightest known black hole (Alexander et al., 2021b).
Since two small black holes merging would likely not produce electromagnetic
emission, we can help constrain the nature of the merger by searching for
electromagnetic emission. Alexander et al. (2021b) and Thakur et al. (2020),
among others, have searched near the merger event for a GRB afterglow
counterpart of GW190814 in the radio and optical.

Alexander et al. (2021b) found a promising candidate (Candidate 1) for the
radio counterpart of GW190814. Candidate 1 includes a non-detection at 38 days
as well as detections at 238 days and 266 days (black triangle and black
points, respectively, on Figure 1).

Given the viewing angle of $\theta_{obs}=46^{+17}_{-11}$ degrees from the
gravitational wave signal and the distance of $d_{L}=241^{+41}_{-45}$ Mpc
(Abbott et al., 2020), we produce 6.0 GHz light curves of Gaussian jets with
cores, top hat jets, and cone jets that match the flux evolution of Candidate
1 (Figure 1).

Figure 1: The 5.0 GHz light curve of Candidate 1 (black points, error bars are
$1\sigma$). Afterglow light curves of Gaussian jets with cores (top), top hat
jets (middle), and cone jets (bottom) with $\theta_{obs}=46$ degrees and
$\theta_{c}=15$ degrees are fitted to the data. We show the suitable range of
$\epsilon_{e}$, $\epsilon_{B}$, $p$, $n_{0}$, and $E_{iso}$ for each jet
structure while all other parameters are held constant (Gaussian:
$E_{iso}=10^{52}$ ergs, $n_{0}=0.01cm^{-3}$, $\epsilon_{e}=0.1$,
$\epsilon_{b}=0.01$, $p=2.2$; Top Hat: $E_{iso}=10^{52}$ ergs,
$n_{0}=0.1cm^{-3}$, $\epsilon_{e}=0.1$, $\epsilon_{b}=0.01$, $p=2.2$; Cone:
$E_{iso}=10^{53}$ ergs, $n_{0}=0.1cm^{-3}$, $\epsilon_{e}=0.1$,
$\epsilon_{b}=0.01$, $p=2.2$).

Our results indicate that a number of physically reasonable jet structures fit
this data222Beniamini et al. (2020) show how this degeneracy can be broken to
some extent and how jet structure can be better constrained by considering the
shape (rather than simply the flux and time normalizations) of the light
curves, when enough data are available.. However, a high resolution radio
image taken 266 days post merger by Very Large Array (VLA) of this source
indicates a double-lobed structure of this source that fairly confidently
rules out Candidate 1 as coming from the jet of a GRB associated with
GW190814. The emission is partially resolved into two distinct components
separated by $\sim 3"$ (Alexander et al., 2021a). At 271 Mpc (the distance of
ESO 471-035), that would correspond to a physical separation of $\sim 4$ kpc.
This is orders of magnitude too large for a newly formed GRB jet ($<1$ pc).
This implies that Candidate 1 is an older and larger source, likely a
background double-lobbed radio AGN undergoing a flaring event, although we
have not examined the possibility that one lobe is associated with the GRB
while the other may be a background source (Wagoner, 2021).

##  4 Conclusions

With the end of the third LIGO-Virgo observing run (O3), it becomes
particularly urgent to continue the search for electromagnetic counterparts to
GW emission, particularly in light of 2 new NS-BH mergers having been
discovered (the xLIGO Scientific Collaboration et al., 2021). Much of the
previous GRB afterglow analysis has relied solely on flux evolution. Yet, this
study reveals how closely a background AGN jet could mimic the behavior of a
GRB jet and appear to be a GW EM counterpart. Consequently, obtaining high-
resolution radio data is crucial in future follow-up efforts, as that may be
the only way to distinguish background sources like AGN jets from true GW
counterparts.

##  5 Acknowledgements

We thank Kate Alexander, Wen-fai Fong, Genevieve Schroeder, Kerry Paterson,
for very helpful discussions and for sharing their data. We also are very
grateful to Greg Ryan for his help with afterglowpy, and Bob Wagoner for
insightful comments and discussions.

## References

  * Abbott et al. (2020) Abbott, R., Abbott, T., Abraham, S., et al. 2020, ApJ, 896, doi: 10.3847/2041-8213/ab960f
  * Alexander et al. (2021a) Alexander, K. D., Schroeder, G., Paterson, K., & Fong, W. 2021a, Private Communication
  * Alexander et al. (2021b) Alexander, K. D., Schroeder, G., Paterson, K., Fong, W., & Cowperthwaite, P. 2021b, ApJS.  https://arxiv.org/abs/2102.08957v1
  * Beniamini et al. (2020) Beniamini, P., Granot, J., & Gill, R. 2020, MNRAS, 493, 3521, doi: 10.1093/mnras/staa538
  * Murguia-Berthier et al. (2021) Murguia-Berthier, A., Ramirez-Ruiz, E., De Colle, F., et al. 2021, ApJ, 908, 152, doi: 10.3847/1538-4357/abd08e
  * Piran (2004) Piran, T. 2004, Reviews of Modern Physics, 76, 1143, doi: 10.1103/RevModPhys.76.1143
  * Ryan et al. (2020) Ryan, G., van Eerten, H., Piro, L., & Troja, E. 2020, ApJ, 896, 166, doi: 10.3847/1538-4357/ab93cf
  * Thakur et al. (2020) Thakur, A. L., Dichiara, S., Troja, E., et al. 2020, MNRAS, doi: 10.1093/mnras/staa2798
  * the xLIGO Scientific Collaboration et al. (2021) the xLIGO Scientific Collaboration, the Virgo Collaboration, & the KAGRA Collaboration. 2021, ApJ, 915, doi: 10.3847/2041-8213/ac082e
  * Urrutia et al. (2021) Urrutia, G., De Colle, F., Murguia-Berthier, A., & Ramirez-Ruiz, E. 2021, MNRAS, 503, 4363, doi: 10.1093/mnras/stab723
  * Wagoner (2021) Wagoner, R. 2021, Private Communication