# Hierarchical Indexing for Retrieval-Augmented Opinion Summarization

Tom Hosking Hao Tang Mirella Lapata
Institute for Language, Cognition and Computation
School of Informatics, University of Edinburgh
10 Crichton Street, Edinburgh EH8 9AB
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

We propose a method for unsupervised abstractive opinion summarization, that
combines the attributability and scalability of extractive approaches with the
coherence and fluency of Large Language Models (LLMs). Our method, HIRO,
learns an index structure that maps sentences to a path through a semantically
organized discrete hierarchy. At inference time, we populate the index and use
it to identify and retrieve clusters of sentences containing popular opinions
from input reviews. Then, we use a pretrained LLM to generate a readable
summary that is grounded in these extracted evidential clusters. The
modularity of our approach allows us to evaluate its efficacy at each stage.
We show that HIRO learns an encoding space that is more semantically
structured than prior work, and generates summaries that are more
representative of the opinions in the input reviews. Human evaluation confirms
that HIRO generates more coherent, detailed and accurate summaries that are
significantly preferred by annotators compared to prior work.

##  1 Introduction

Online review websites are a useful resource when choosing which hotel to
visit or which product to buy, but it is impractical for a user to read
hundreds of reviews. Automatic opinion summarization aims to aggregate a large
and diverse set of customer reviews about a particular entity into a single,
easy to read summary. A good summary should accurately reflect the balance of
opinions in the input reviews, highlighting the most common or popular
opinions, while omitting unnecessary details. A useful summary should also
help compare between competing options, and include points that differentiate
the current entity from others.

Early work on opinion summarization extracted reviewers’ sentiment about
specific features Hu and Liu (2004) or identified salient sentences based on
centrality Erkan and Radev (2004), while more recent methods have proposed
_extractive_ models that use learned feature spaces Angelidis et al. (2021);
Basu Roy Chowdhury et al. (2022). Prior work on _abstractive_ opinion
summarization has almost exclusively either required costly supervision
Bražinskas et al. (2021); Cattan et al. (2023) or has assumed that the number
of input reviews is limited Coavoux et al. (2019); Bražinskas et al. (2020);
Amplayo et al. (2021a, b); Iso et al. (2021). This defeats the point of a
summary: a user could feasibly read 8 reviews in a reasonable period of time.
A good summarization system should be scalable, since popular products online
may receive thousands of reviews. It should also be attributable, offering
some evidence to justify its output. Finally, it should generate summaries
that are coherent and faithful to the input reviews.

Large Language Models (LLMs) have been shown to generate highly fluent
summaries in the news domain Bhaskar et al. (2023) but are a flawed solution
because current instruction-tuned models are not attributable, and because
their context windows limit the number of reviews they are able to base their
summaries on. Models with long context windows have been proposed Beltagy et
al. (2020); Gu et al. (2022) but these are not currently instruction-tuned,
and it has been shown that LLMs are biased toward information at the start and
end of the input Liu et al. (2023).

Our approach, Hierarchical Indexing for Retrieval-Augmented Opinion
Summarization (HIRO), identifies informative sentences using hierarchical
indexing and then passes the selected sentences as input to a LLM, similar to
retrieval-augmented generation (RAG, Lewis et al., 2020). This separation of
concerns (selection and generation) allows us to combine the attributability
and scalability of the discrete representation with the strong generative
abilities of LLMs, leading both to a higher quality index and to more
informative and coherent output summaries.

HIRO consists of three modules, allowing for increased control, flexibility
and interpretability. The Hierarchical Indexer is an encoder that maps
sentences from reviews to paths through a hierarchical discrete latent space.
The Retriever uses the index to identify clusters of sentences for each entity
that contain popular and informative opinions. These sentence clusters are
passed to a Generator, a pretrained LLM, that generates coherent summaries
that are grounded in the retrieved sentences.

Our contributions are as follows:

  * •

We propose a method for learning an encoder that maps sentences to a path
through a semantically structured discrete hierarchy.

  * •

We show how to exploit this discrete hierarchy at inference time to identify
clusters of related and prevalent sentences from input reviews.

  * •

We introduce an automatic metric that measures whether generated summaries
reflect the input reviews, while penalizing overly generic statements.

  * •

Through extensive experiments on two English datasets from different product
domains, we demonstrate that passing these retrieved sentences in a zero-shot
manner to a pretrained LLM generates summaries that better reflect the
distribution of opinions within the input reviews. Human evaluation shows that
summaries generated by HIRO are significantly more coherent and accurate than
prior work, and are preferred by annotators.

Our code and dataset splits are available at
https://github.com/tomhosking/hiro.

##  2 Related Work

#### Opinion Summarization

Prior approaches to generating summaries of reviews broadly fall into two
categories. Extractive methods generate summaries by selecting representative
sentences from input reviews to use as the summary Erkan and Radev (2004);
Angelidis et al. (2021); Basu Roy Chowdhury et al. (2022). These types of
approach are scalable and inherently attributable, but result in summaries
that are overly detailed and not coherent. Abstractive methods ‘read’ input
reviews and generate novel language to use as the summary Bražinskas et al.
(2020); Iso et al. (2021). The resulting summaries are more fluent and
coherent, but most prior abstractive methods are only able to consider a
limited number of input reviews, whether because of combinatoric complexity or
a maximum input length.

Hosking et al. (2023b) propose a hybrid method that represents sentences from
reviews as paths through a learned discrete hierarchy, then generates output
sentences based on frequent paths through this hierarchy. While their method
is abstractive as well as attributable and scalable, the highly compressed
bottleneck leads to frequent hallucination and generic output with poor
coherence.

Louis and Maynez (2023) use a Natural Language Inference (NLI) model to
construct ‘silver standard’ summaries to use as training data for fine-tuning
a pretrained language model (T5, Raffel et al., 2020). However, their approach
is computationally very expensive, calculating over 1B pairwise entailment
relations between sentences in the training data, before fine-tuning a LLM. By
contrast, HIRO uses a lightweight indexing encoder, combined with an off-the-
shelf LLM that is prompted in a zero-shot manner.

#### Structured Encodings and Indexes

Prior work has investigated how to learn representations of data with richer
structure and interpretability. Vendrov et al. (2016) propose learning an
embedding space where the ordering of two pairs of samples could be inferred
from their relative positions, but their method requires supervision of the
correct ordering. Opper et al. (2023) describe a method for learning
embeddings that explicitly include structural information, but they focus on
representing structures, rather than on learning an ordering within the
embedding space itself. Li et al. (2023) learn a tree-based index for passage
retrieval concurrently with the dense embedding space, showing that this leads
to improved retrieval performance. However, they focus on retrieving passages
relevant to a query, rather than aggregating opinions.

#### Content Selection

The idea of first selecting relevant parts of an input before generating
output has been well studied (Kedzie et al., 2018; Puduppully et al., 2019;
Amplayo et al., 2021b; Narayan et al., 2023, inter alia), and has been shown
to be very effective when used in conjunction with LLMs in the form of
retrieval-augmented generation (RAG, Lewis et al., 2020). Xu et al. (2023)
found that retrieval augmentation is beneficial even when using models that
can accept long inputs.

#### Evaluation of Summaries

Automatic evaluation of generated summaries is extremely challenging. Prior
work has shown that ROUGE Lin (2004) scores correlate poorly with human
assessments of summary quality Shen and Wan (2023); Clark et al. (2023);
Aharoni et al. (2023). Some datasets are created automatically, with
references that are not directly based on the input reviews Bražinskas et al.
(2021). Modern summarization system outputs are now often preferred to human-
written references Goyal et al. (2022); Bhaskar et al. (2023).

SummaC Laban et al. (2022) is a reference-free metric that uses an NLI model
to evaluate the degree of support between a summary and the input documents,
but it overly rewards trivial statements; using the obvious statement “The
hotel was a building” as a summary for every entity achieves a near-perfect
SummaC score of 99.9% on Space, a dataset of hotel reviews Angelidis et al.
(2021).

Malon (2023) propose a metric that uses a NLI model to evaluate prevalence,
i.e. how many input reviews contain supporting evidence for each sentence in a
summary, and explicitly penalizes trivial or redundant output. However, we
found it has a similar failure mode to SummaC, with the statement “The rooms
are clean and comfortable” achieving a prevalence score of 72% on Space. We
propose a modification to prevalence in Section 6.2 that penalizes overly
generic summaries.

Figure 1: HIRO uses three modules to generate summaries of customer reviews.
First, we use our encoder to index all sentences from input review into a
learned hierarchy. Then we identify paths within this index that occur
frequently, and retrieve the corresponding clusters of sentences. Finally, we
pass these clusters to an LLM to generate an output summary.

##  3 Overview

Let $\mathcal{R}_{e}$ be a set of reviews about an entity $e\in\mathcal{E}$,
where each review $\mathbf{R}\in\mathcal{R}_{e}$ is composed of a number of
sentences $\mathbf{x}$. The goal is to generate a textual summary that
includes the most informative opinions from $\mathcal{R}_{e}$, while
abstracting away the details specific to any one review.

HIRO generates summaries following a modular approach, depicted in Figure 1.
We learn an index structure that maps each sentence $\mathbf{x}$ from the
input reviews to a path $q_{1:D}$ through a discrete hierarchy. We choose a
hierarchical representation so that sentences are grouped at a useful level of
abstraction; the upper levels of the hierarchy should partition sentences by
topic, while the lowest levels should group together sentences with equivalent
meaning.

At inference time, we encode all sentences from the reviews, then identify the
paths or subpaths $q_{1:d}$ within the index that are particularly popular,
and retrieve the corresponding sentences. This selection process is query-
free, instead relying on properties of the hierarchical index to determine the
frequency of different opinions. By indexing sentences hierarchically
according to their semantics, we can easily identify opinions or topics that
occur frequently by simply counting their occurrence in the index.

Finally, we generate a summary by passing the retrieved clusters of sentences
as input to a LLM. This retrieval-augmented usage of LLMs allows us to benefit
from the fluency and coherence of LLMs, while retaining the attributability
and scalability of extractive opinion summarization methods.

We now detail the three modules that comprise HIRO, evaluating each of them in
turn to confirm their efficacy compared to previous methods.

##  4 Learning a Hierarchical Indexing

Our goal is to learn an encoding space where sentences with similar meanings
are grouped together. The space should be discretized so that frequent
opinions can be easily identified by counting the membership of each part of
the index, and it should be hierarchical so that opinions may be aggregated at
an appropriate level of granularity, rather than by details or phrasings
specific to a particular review. Finally, the encoding space should be
structured semantically, to enable accurate aggregation of opinions; sentences
with equivalent meaning should clearly be indexed to the same point in the
hierarchy, while sentences that are topically related but not equivalent
should be grouped together at a higher level.

We base our encoder on Hercules Hosking et al. (2023b). Hercules uses an
encoder-decoder architecture with a discrete hierarchical bottleneck to
generate summaries. It is trained as a denoising autoencoder, and therefore
needs to learn a representation that is both compressed enough to enable
aggregation, but also expressive enough for the decoder to be able to generate
meaningful output. These factors are in direct competition, with the
compressed bottleneck leading to output that is generic and contains
hallucinations. By contrast, HIRO uses an external LLM as the ‘decoder’,
allowing us to focus on learning a representation that is useful for
identifying informative opinions.

###  4.1 Method

The HIRO encoder module maps a single sentence $\mathbf{x}$ to a path
$q_{1:D}$ through a discrete hierarchy, using a technique based on residual
vector quantization Chen et al. (2010); Zeghidour et al. (2022); Hosking et
al. (2023b).

First, we use a Transformer encoder followed by attention pooling Vaswani et
al. (2017); Liu and Lapata (2019) to map a sequence of tokens $\mathbf{x}$ to
a single dense embedding $\mathbf{z}\in\mathbb{R}^{\mathbb{D}}$. Then, we
decompose $\mathbf{z}$ into a path through a latent discrete hierarchy
$q_{1:D}$, where $q_{d}\in[1,K]$ are discrete ‘codes’ at each level $d$.
Briefly, we induce a distribution over codes at each level $p(q_{d})$,
parameterised by a softmax with scores $s_{d}$ given by the Euclidean distance
from learned codebook embeddings to the residual error between the input and
the cumulative embedding from all previous levels,

$\displaystyle\vspace{-.2cm}s_{d}(q)=-\left(\left[\mathbf{z}-\sum\limits_{d^{\prime}=1}^{d-1}\mathbf{C}_{d^{\prime}}(q_{d^{\prime}})\right]-\mathbf{C}_{d}(q)\right)^{2},\vspace{-.2cm}$
(1)

where $\textbf{C}_{d}\in\mathbb{R}^{K\times\mathbb{D}}$ is a codebook which
maps each discrete code to a continuous embedding
$\textbf{C}_{d}(q_{d})\in\mathbb{R}^{\mathbb{D}}$. During training, we use the
Gumbel reparameterization Jang et al. (2017); Maddison et al. (2017); Sønderby
et al. (2017) to sample from the distribution $p(q_{d})$. During inference, we
set $q_{d}=\operatorname*{arg\,max}s_{d}$.

Since our goal is to learn a representation where semantically similar
sentences are grouped together, we use a training objective that explicitly
induces this arrangement in encoding space. We train the encoder with a
contrastive learning objective, bringing representations of semantically
similar sentences (i.e., positive pairs) together, and pushing dissimilar ones
apart.

For each sentence in the training data, we construct positive pairs of
semantically related sentences $\mathbf{x},\mathbf{x}_{+}$ as follows: given a
random ‘query’ sentence $\mathbf{x}$ from the training data, we identify
possible candidate ‘targets’ based on tf-idf similarity; then, we check
whether each candidate is entailed by the query using an NLI model (DeBERTa
v3, trained on Debiased NLI; He et al., 2021; Wu et al., 2022), and use the
sentences which are labelled as entailed as positive targets $\mathbf{x}_{+}$.
An example is shown in Figure 2. We do not use any ‘hard’ negatives; instead,
during training we calculate the pairwise tf-idf similarity between all
samples in a batch, and include only those samples with similarity below a
threshold of $0.3$ as negatives $\mathbf{X}_{-}$. This prevents ‘false
negatives’ being wrongly forced apart, and allows sentences that are topically
related but not strictly equivalent to remain in the same high-level grouping
within the index.

Figure 2: An example of the process for constructing the positive pairs used
to train our model. Given a query sentence, we first use tf-idf to identify
possible candidates from the training data, keeping only those sentences with
similarity over a specified threshold. Then we check for entailment using an
NLI model, and use any sentences labelled as ‘entailed’ as positive targets.

We found that it was crucial to include sufficient exploration in the process
of constructing positive pairs. The candidate sentences retrieved using tf-idf
should be sufficiently similar that we are likely to find ones that are
entailed, but not so similar that they only have minor lexical differences.

We use a modified version of the InfoNCE training objective van den Oord et
al. (2018) designed to bring the representations of a query $\mathbf{x}$
closer to those of a positive target $\mathbf{x}_{+}$, while pushing them
apart from negative samples $\mathbf{X}_{-}$,

$\displaystyle\mathcal{L}=$ $\displaystyle-\rho(\mathbf{x},\mathbf{x}_{+})\log
f,$ (2) $\displaystyle f=$
$\displaystyle\frac{\exp\big{(}s(\mathbf{x},\mathbf{x}_{+})\big{)}}{\exp\big{(}s(\mathbf{x},\mathbf{x}_{+})\big{)}+\frac{\omega}{|\mathbf{X}_{-}|}\sum\limits_{\mathbf{x}_{-}\in\mathbf{X}_{-}}\exp\big{(}s(\mathbf{x},\mathbf{x}_{-})\big{)}},$

where $\rho(\mathbf{x},\mathbf{x}_{+})$ is the tf-idf similarity between
$\mathbf{x}$ and $\mathbf{x}_{+}$ that weights the confidence of the positive
pairs, inspired by MarginMSE Hofstätter et al. (2021), and $\omega$ is a
constant that controls the strength of the negative examples. The similarity
function $s(\cdot,\cdot)$ is given by the mean dot product between the
embedding of all subpaths $q_{1:d}$ for $d\leq D$,

$\displaystyle
s(\mathbf{x},\mathbf{x}^{\prime})=\frac{1}{D}\sum\limits_{d=1}^{D}\max\big{(}\mathbf{C}(q_{1:d})^{T}\mathbf{C}(q^{\prime}_{1:d}),0\big{)},$
(3)

where $\mathbf{C}(q_{1:d})=\sum_{d}\mathbf{C}_{d}(q_{d})$ is the full
embedding of path $q_{1:d}$. Intuitively, this brings together the
representations of the positive pairs at each level in the hierarchy, while
penalizing any overlap with the representations of negative examples.

The similarity is floored at zero, to prevent the model from being able to
‘game’ the loss by pushing negative examples further and further apart.
Although the positive pairs are selected based on a directional entailment
label, we do not exploit this directionality in our training objective.

We employ the techniques to induce a hierarchical encoding space proposed by
Hosking et al. (2023b), including depth dropout, initialization decay, and
norm loss. We additionally include the entropy of the distribution over codes,
$-\sum_{d,q_{d}}\log\big{(}p(q_{d})\big{)}$, as an additional term in the
objective, to encourage exploration of the latent space during training.

###  4.2 Evaluation

We now evaluate whether the combination of discrete hierarchical encoding and
contrastive objective leads to a more semantically distributed representation
than previous methods.

#### Experimental Setup

We experiment using Space Angelidis et al. (2021), which consists of hotel
reviews from TripAdvisor with 100 reviews per entity, as well as reference
summaries created by annotators. We encode all the review sentences from the
Space validation set, then calculate both the purity (the mean intra-cluster
similarity) and colocation (the mean inter-cluster similarity) for the
clusters of sentences assigned to each subpath $q_{1:d}$ for $d\leq 4$.
Finally, we take the difference between the purity and colocation as an
overall measure of the quality of the representation space.

We compare to Hercules Hosking et al. (2023b). We also compare to a baseline
using recursive $k$-means over embeddings from a pretrained embeddings model
(MiniLM, Reimers and Gurevych, 2019): we apply $k$-means to the embeddings,
then calculate the residual errors between cluster centroids and embeddings,
apply $k$-means to those errors, and repeat. All methods use the same number
of clusters at each level ($k=12$).

Figure 3: Cluster quality by depth, as measured by the difference between
cluster purity and colocation, according to NLI (solid line) and tf-idf
(dashed line) similarity measures. HIRO learns a higher quality encoding space
than comparison methods.

#### Model Configuration

We use a 6 layer Transformer, with token embeddings initialized from BERT base
Devlin et al. (2019).111We experimented with using BERT as the encoder but
found no significant improvement, since the discrete encoding is the main
bottleneck in the model. We set the codebook size $K=12$, with the number of
levels $D=12$, based on development set performance. Other hyperparameters are
given in Appendix B.

#### HIRO learns a higher quality encoding space

Figure 3 shows the overall quality (using both NLI and tf-idf measures of
similarity), indicating that HIRO learns a more semantically distributed space
at all levels of depth than comparison approaches, according to both
similarity measures. The separation between clusters increases with depth,
confirming that the encoder learns a semantic hierarchy. We believe these
results indicate that our method could potentially be used for more general
purpose document retrieval (similar to Li et al., 2023), which is beyond the
scope of this paper.

##  5 Retrieving Popular Opinions

Recall that a good summary should include opinions that occur repeatedly in
the input reviews, as well as opinions that differentiate the current entity
from others. In Section 4 we showed how the HIRO encoder maps a single
sentence $\mathbf{x}$ to a path $q_{1:D}$. We now exploit the discrete,
hierarchical nature of the representation space to index a large number of
review sentences, then identify informative sentences to use to generate a
summary. We hypothesize that our content selection method leads to clusters
that better represent the balance of opinions in the input reviews.

###  5.1 Method

For each review $\mathbf{R}\in\mathcal{R}_{e}$ about an entity
$e\in\mathcal{E}$, we separately encode each sentence within the review to its
path $q_{1:D}$, giving an indexed review $\mathbf{Q}(\mathbf{R})$.

Our content selection method identifies the parts of the hierarchy that are
particularly popular for each entity, and extracts the corresponding clusters
of sentences. This process is query-free; instead, we assign each subpath in
the hierarchy $q_{1:d}$ a score based on its popularity within the indexed
input reviews, and ‘retrieve’ all sentences that were mapped to the $k$
highest-scoring subpaths.

Our scoring function is inspired by tf-idf Jones (1972), which is designed to
measure the importance of a particular term with respect to a set of baseline
documents. We define the term popularity $\operatorname*{tp}(q_{1:d},e)$ of a
path as the fraction of indexed reviews for entity $e$ which contain the
subpath $q_{1:d}$,

$\displaystyle\operatorname*{tp}(q_{1:d},e)=$
$\displaystyle\frac{1}{|\mathcal{R}_{e}|}\sum_{\textbf{R}\in\mathcal{R}_{e}}\mathbb{I}\big{(}q_{1:d}\in\mathbf{Q}(\textbf{R})\big{)},$
(4)

where $\mathbb{I}$ is the indicator function. We define the inverse baseline
popularity $\operatorname*{ibp}$ as the reciprocal of the mean term popularity
across all entities $\mathcal{E}$,

$\displaystyle\operatorname*{ibp}(q_{1:d})=$
$\displaystyle\biggl{(}\frac{\alpha+\sum_{e\in\mathcal{E}}\operatorname*{tp}(q_{1:d},e)}{\alpha+|\mathcal{E}|}\biggr{)}^{-1},$
(5)

where the smoothing constant $\alpha$ allows us to balance between absolute
and comparative popularity. The overall score is then

$\displaystyle\text{score}(q_{1:d},e)=$
$\displaystyle\operatorname*{tp}(q_{1:d},e)\times\operatorname*{ibp}(q_{1:d}).$
(6)

Intuitively, the score represents the relative popularity within the current
entity of a path $q_{1:d}$ compared to all entities in the dataset.

Our scoring scheme conveniently accounts for the fact that short paths are
more common;222The presence of a path $q_{1:D}$ for a review also implies the
presence of all subpaths $q_{1:d},d<D$. short paths will also have a higher
baseline popularity, leading to an overall score that is effectively
normalized for the depth of the subpath.

After retrieving the clusters of sentences corresponding to the top-$k$
highest scoring subpaths, we filter out sentences with very low lexical
similarity to other cluster members, and combine any clusters of sentences
with high lexical overlap.

###  5.2 Evaluation

We evaluate whether our method successfully selects clusters of sentences
containing informative opinions, using reviews from Space, as in Section 4.2.
First, we measure the overlap between retrieved clusters and oracle clusters
constructed by selecting sentences with high overlap to the reference
summaries,333Space includes multiple reference summaries for each entity; we
randomly select one when determining the oracle clusters. using the Adjusted
Rand Index (ARI, Hubert and Arabie, 1985). Second, we evaluate the average
prevalence Malon (2023) of sentences in retrieved clusters, which uses a NLI
model (ALBERT, trained on VitC; Lan et al., 2020; Schuster et al., 2021) to
evaluate how many input reviews support each retrieved sentence.

We compare HIRO to the clusters extracted by Hercules Hosking et al. (2023b),
and the oracle upper bound. As a baseline, we apply $k$-means to MiniLM
embeddings Reimers and Gurevych (2019), then extract the 25 sentences whose
embeddings are closest to each centroid. For HIRO, we select the top $k=8$
subpaths for each entity, and set the smoothing constant $\alpha=6$.

#### HIRO selects higher prevalence sentences

Table 1 confirms that HIRO retrieves clusters that more closely match the
oracle clusters, and contain opinions that are more prevalent in the input
reviews compared to prior work. The oracle ARI score is less than 1 because
some sentences appear multiple times in different clusters.

System | Prevalence | ARI
---|---|---
$k$-means | 38.1 | 0.59
Hercules | 32.3 | 0.50
HIRO | 46.5 | 0.69
(Oracle) | 48.1 | 0.73
Table 1: Evaluation of our cluster selection method, compared to a range of
baseline approaches. HIRO selects clusters of sentences that more closely
match the references and contain more prevalent opinions.

##  6 Generating Coherent Summaries

Given the retrieved clusters of sentences for each entity, we want to generate
a coherent and fluent textual summary. LLMs are well suited to this
constrained rewriting task, and we leverage the zero-shot abilities of
instruction-tuned models to map clusters of sentences to a readable summary.

###  6.1 Method

We generate a summary from the retrieved clusters in three ways, with varying
trade-offs between coherence and attributability.

#### HIROext

We generate extractive summaries by selecting the centroid of each cluster; we
compute all pairwise ROUGE-2 scores between sentences in each cluster, and
choose the sentence with highest average similarity to other cluster members.
This approach is fully attributable, since each summary sentence is extracted
verbatim from a review.

#### HIROsent

We generate summaries one sentence at a time by passing the contents of a
single cluster as input to an instruction-tuned LLM with a simple prompt that
requests a single sentence that summarizes the main points. This leads to more
fluent output that is still attributable, since each output sentence has an
associated cluster of evidential sentences used to generate it.

#### HIROdoc

We generate summaries as a single document, by passing the sentences from all
retrieved clusters for an entity to the LLM in one go. This gives summaries
that are more coherent and less redundant, since the LLM has control over the
whole summary. However, it is not possible to identify which cluster was used
to generate each part of the summary, so the attributability is less granular.

The ideal balance between coherence and attributability is likely to vary by
application.

#### Experimental Setup

For the sentence-wise and document variants, we use Mistral 7B Instruct v0.2
to generate summaries from retrieved clusters. The LLM is queried in a zero-
shot manner, and the prompts used are given in Appendix C. We sample with a
temperature of $0.7$, and report the mean and standard deviation scores based
on 3 samples.

###  6.2 Automatic Evaluation

Human evaluation is the gold standard (Section 6.3), but automatic metrics
remain useful for model development. ROUGE scores are no longer reliable
(Section 2), but we nonetheless report them for consistency with prior work.
Malon (2023) propose a prevalence metric, that uses an NLI model to determine
how many input reviews contain supporting evidence for each sentence in the
summary, but this suffers from a failure mode that overly rewards generic
statements. A good summary should include common opinions, but should also
help a user to differentiate between multiple entities.

To counteract this pathology, we propose a modified version of prevalence,
that explicitly penalizes generic summaries. First, we define the genericness
of a summary as the average number of summaries from other entities that
support each sentence in a given summary, as measured by an NLI model (ALBERT,
trained on VitC; Lan et al., 2020; Schuster et al., 2021). Then, we define the
Specificity Adjusted Prevalence score (SAP) as

$\texttt{SAP}=\texttt{prevalence}-\alpha~{}\texttt{genericness},$ (7)

where $\alpha$ is a constant that controls the balance between absolute
prevalence and specificity. In practice, the ideal summary is unlikely to be
entirely unique and a user may want to allow some degree of overlap between
generated summaries. We report the prevalence and genericness, as well as the
combined SAP with $\alpha=0.5$.

|  | Space | AmaSum (4 domains)
---|---|---|---
| System |  R-2 $\uparrow$ |  R-L $\uparrow$ |  Prev. $\uparrow$ |  Gen. $\downarrow$ |  SAP $\uparrow$ |  R-2 $\uparrow$ |  R-L $\uparrow$ |  Prev. $\uparrow$ |  Gen. $\downarrow$ |  SAP $\uparrow$
Extractive | Random | 6.2 | 17.1 | 18.0 | 12.5 | 11.8 | 1.0 | 9.5 | 16.3 | 8.0 | 12.3
$k$-means | 9.5 | 19.8 | 27.9 | 25.0 | 15.4 | 2.3 | 12.0 | 14.9 | 11.4 | 9.2
LexRank | 5.9 | 16.4 | 18.2 | 4.4 | 16.0 | 2.7 | 12.2 | 9.0 | 3.0 | 7.5
QT | 10.3 | 21.5 | 24.9 | 23.3 | 13.3 | 1.5 | 11.4 | 10.9 | 7.3 | 7.3
SemAE | 11.1 | 23.5 | 29.2 | 17.1 | 20.6 | 1.6 | 11.3 | 8.7 | 4.1 | 6.7
Herculesext | 13.2 | 24.4 | 30.2 | 25.2 | 17.6 | 3.0 | 12.5 | 9.5 | 6.7 | 6.2
HIROext | 11.7 | 22.1 | 36.3 | 20.5 | 26.1 | 2.7 | 12.6 | 19.4 | 9.5 | 14.6
Abstractive | CopyCat | 12.1 | 22.9 | 48.3 | 70.9 | 12.9 | 1.5 | 11.2 | 15.8 | 21.0 | 5.3
BiMeanVAE | 13.7 | 27.1 | 45.0 | 61.4 | 14.2 | 2.0 | 12.5 | 14.7 | 24.1 | 2.7
COOP | 14.2 | 27.2 | 46.1 | 63.2 | 14.5 | 2.8 | 14.1 | 18.8 | 30.3 | 3.7
Herculesabs | 14.8 | 27.2 | 32.2 | 36.1 | 14.1 | 2.0 | 11.8 | 8.5 | 9.2 | 3.9
Mistral 7B |  5.3$\pm$0.1 |  19.6$\pm$0.4 |  41.3$\pm$1.3 |  34.3$\pm$3.3 |  24.2$\pm$0.8 |  1.9$\pm$0.0 |  12.6$\pm$0.0 |  17.3$\pm$0.2 |  17.6$\pm$0.4 |  8.5$\pm$0.2
Llama 2 7B |  4.4$\pm$0.2 |  17.6$\pm$0.1 |  32.6$\pm$3.0 |  25.7$\pm$3.7 |  19.7$\pm$2.3 |  1.5$\pm$0.0 |  11.5$\pm$0.0 |  17.7$\pm$0.4 |  17.6$\pm$0.3 |  8.9$\pm$0.5
Llama 2 13B |  6.5$\pm$0.5 |  17.6$\pm$0.4 |  22.9$\pm$1.7 |  21.9$\pm$1.8 |  12.0$\pm$0.8 |  2.9$\pm$0.1 |  12.9$\pm$0.0 |  10.7$\pm$0.1 |  13.9$\pm$0.1 |  3.8$\pm$0.2
HIROsent |  4.5$\pm$0.1 |  18.2$\pm$0.0 |  36.2$\pm$0.8 |  20.1$\pm$0.5 |  26.2$\pm$0.9 |  3.5$\pm$0.0 |  14.1$\pm$0.1 |  14.6$\pm$0.3 |  6.9$\pm$0.1 |  11.2$\pm$0.3
HIROdoc |  7.0$\pm$0.2 |  20.5$\pm$0.3 |  44.0$\pm$3.0 |  28.8$\pm$2.1 |  29.6$\pm$2.1 |  4.0$\pm$0.0 |  15.1$\pm$0.1 |  15.3$\pm$0.1 |  9.4$\pm$0.3 |  10.6$\pm$0.1
| (References) | — | — | 44.3 | 50.2 | 19.2 | — | — | 9.3 | 7.0 | 5.8
| (Oracle) | 45.0 | 53.3 | 41.0 | 38.5 | 21.7 | 14.4 | 26.0 | 12.3 | 9.0 | 7.8
Table 2: Results for automatic evaluation of summary generation. R-2 and R-L
represent ROUGE-2/L F1 scores. Prev. refers to Prevalence, Gen. refers to
Genericness, and SAP refers to Specificity-Adjusted Prevalence. We consider
SAP to be our primary metric. The best scores for extractive and abstractive
systems are shown in bold. Results for systems involving LLMs are based on 3
samples, with the mean and standard deviation shown. HIRO generates summaries
with the best balance between prevalent opinions and specificity.

#### Datasets

We evaluate summary generation using Space (Section 4.2), which includes
multiple reference summaries created by human annotators for each entity. We
also include AmaSum Bražinskas et al. (2021), to evaluate summary generation
on a wide range of categories of Amazon products, with an average of 560
reviews per entity. The reference summaries were collected from professional
review websites, and therefore are not necessarily grounded in the input
reviews. We use the same splits, based on four product categories, as released
by Hosking et al. (2023b). Further dataset statistics are given in
LABEL:appx:datasets.

#### Comparison Systems

We select a random review from the inputs as a lower bound. We include an
extractive oracle as an upper bound, by selecting the input sentence with
highest ROUGE-2 similarity to each reference sentence.444When multiple
references are available, we select one at random. For a k-means baseline, we
run $k$-means on MiniLM sentence embeddings Reimers and Gurevych (2019), then
extract the nearest sentence to the cluster centroids. We set $k=8$ to match
the average sentence length of the reference summaries. Lexrank Erkan and
Radev (2004) is an unsupervised extractive method using graph-based centrality
scoring of sentences. QT Angelidis et al. (2021) uses vector quantization to
map sentences to a discrete encoding space, then generates extractive
summaries by selecting representative sentences from clusters. SemAE Basu Roy
Chowdhury et al. (2022) is an extractive method that extends QT, relaxing the
discretization and encoding sentences as mixtures of learned embeddings.
CopyCat Bražinskas et al. (2020) is an abstractive approach that models
sentences as observations of latent variables representing entity opinions,
trained in a ‘leave one out’ manner. BiMeanVAE and COOP Iso et al. (2021) are
abstractive methods that encode full reviews as continuous latent vectors
using an autoencoder, and take the average (BiMeanVAE) or an optimised
combination (COOP) of review encodings. We compare to a recent open weight
instruction-tuned LLM, specifically Mistral Instruct v0.2 7B Jiang et al.
(2023). To evaluate the effect of model size, we compare to Llama 2 Chat 7B
and 13B Touvron et al. (2023), which are equivalent models apart from
parameter count. The LLMs are prompted zero-shot using the prompt in Appendix
C, and sampled with temperature $0.7$. We report the mean and standard
deviation scores based on 3 samples.

Most of the abstractive methods are not scalable and have upper limits on the
number of input reviews. CopyCat and the LLMs have a maximum input sequence
length, while COOP exhaustively searches over combinations of input reviews.
We use 8 randomly selected reviews as input to CopyCat, COOP, Llama 2, and
Mistral 7B.

#### HIRO gives the best balance between prevalence and specificity

The results in Table 2 show that HIRO achieves the highest SAP scores across
both datasets, indicating that it generates summaries with the best balance
between absolute prevalence and specificity. While CopyCat and COOP achieve
the highest prevalence scores on Space and AmaSum respectively, they also
display some of the highest genericness; qualitatively, the outputs are very
similar to each other, and include no specific details about each entity. We
give example outputs in Tables LABEL:tab:output and LABEL:tab:output2, and an
example of selected subpaths, sentence clusters and corresponding HIRO outputs
in LABEL:tab:evidence_example.

#### Increasing model size does not help

The SAP scores decrease for the larger Llama 2 13B model compared to Llama 2
7B, indicating that increasing model size does not lead to improved
performance. The short context window of current instruction-tuned LLMs limits
the number of reviews they can take as input, preventing them from accurately
reflecting the true distribution of opinions.

#### References are not the upper bound

While the oracle summaries score highly in terms of specificity-adjusted
prevalence, some systems (including HIRO) outperform them. This indicates the
difficulty with constructing reference summaries for entities with large
numbers of reviews; it is not feasible for human annotators to reliably
summarize such a large amount of information.

Figure 4: Distribution of selected subpaths by depth. A significant fraction of the extracted clusters come from paths deeper than the top level. System | % Partial | % Majority
---|---|---
Herculesabs | 85.4 | 27.6
HIROsent | 81.6 | 21.6
HIROdoc | 91.8 | 28.4
Table 3: Results for automatic evaluation of the evidence supplied by
attributable systems, showing the percentage of summary sentences that have
support from at least one sentence in the evidence set (partial support) and
from at least half the sentences in the evidence (majority support). HIRO
generates summaries that have strong partial support from the associated
evidence sets, with improved majority support compared to Hercules. Figure 5:
Results of our human evaluation. Crowdworkers were asked for pairwise
preferences between generated summaries in terms of their Accuracy, Detail,
and Coherence & Fluency, as well as an overall preference. HIRO generates more
coherent and detailed summaries that better represent the opinions in the
input reviews than comparison systems, and are preferred by human annotators.

#### HIRO is more attributable

To evaluate how faithful the generated summaries are to the retrieved sentence
clusters or evidence sets, we use an NLI model to determine how many sentences
in each cluster either entail or are entailed by the corresponding sentence in
the output summary, and take the mean. Considering both forward and backward
entailment in this manner accounts for the different levels of granularity
between the inputs and summary Zhang et al. (2024); input reviews are likely
to be more specific than summary sentences, but concise summary sentences are
likely to contain multiple assertions, e.g. “The food was good and the rooms
were clean”. HIROdoc does not align the evidence sets with each generated
sentence, so we calculate the maximum support score over all sets for each
summary sentence. Most abstractive systems are not attributable, and so we
only compare with Hercules. Table 3 shows the percentage of summary sentences
that have support from at least one sentence in the evidence (partial support)
and from at least half the sentences in the evidence (majority support).
HIROdoc generates summaries that have strong partial support from the
associated evidence sets, with improved majority support compared to Hercules
despite also being significantly more detailed.

#### HIRO makes use of the hierarchy

We confirm that HIRO exploits the hierarchical nature of the representation
space. Figure 4 shows the distribution of selected subpath depths for both
datasets, indicating that HIRO takes advantage of the hierarchy and selects
clusters deeper than the top level. This is particularly true for AmaSum,
where there is a wider range of product types, causing the selection process
to search deeper in the tree.

###  6.3 Human Evaluation

We conduct a human evaluation to verify that HIRO generates summaries that are
coherent and accurately reflect the opinions in the input reviews. We recruit
crowdworkers through Prolific, and show them a set of 50 reviews, followed by
two generated summaries. We solicit pairwise preferences along three
dimensions, as well as an overall preference:

  * •

Accuracy — Which summary accurately reflects the balance of opinion in the
reviews?

  * •

Detail — Which summary includes more specific details?

  * •

Coherence — Which summary is easy to read and avoids contradictions?

  * •

Overall — Which summary do you think is better, overall?

The full instructions are reproduced in Appendix A. We gather annotations for
10 entities each from the Space and AmaSum test sets, with 3 annotations for
each.

We compare HIROdoc to the top performing extractive and abstractive systems
from Table 2, SemAE and Herculesabs. HIROdoc uses Mistral 7B as the generator,
so we also compare to Mistral 7B without HIRO (i.e., prompted directly with
reviews). Finally, we include a random review as a lower bound, and the
references as an upper bound.

#### Humans prefer summaries generated by HIRO

The results in Figure 5 show that HIROdoc produces summaries that outperform
comparison systems across all dimensions, producing summaries that coherently
and accurately represent the opinions in the input reviews. Differences
between HIROdoc and other systems are significant in all cases (using a one-
way ANOVA with post-hoc Tukey HSD test, $p<0.05$), except for coherence versus
Mistral 7B. Both Mistral 7B and HIROdoc outperform the reference summaries,
supporting findings from prior work Bhaskar et al. (2023); Hosking et al.
(2023a).

Figure 6: Results of a human evaluation comparing the three variants of HIRO:
extractive, sentence-wise and document. Overall, annotators prefer the
coherent of the document approach, but the sentence-wise variant generates
more detailed summaries that are also more attributable. The preferred
tradeoff between coherence and attribution will vary depending on the
application.

#### Comparing HIRO variants

We run an additional human evaluation to evaluate the tradeoff between quality
and attribution between the three HIRO variants, collecting annotations for 5
entities from each dataset from 3 annotators each. Figure 6 shows that
annotators prefer the output of HIROdoc, although the sentence-wise variant
generates more detailed summaries. The preferred tradeoff between attribution
and coherence will likely vary depending on the application.

##  7 Conclusion

We propose HIRO, a modular method for unsupervise opinion summarization that
uses a hierarchical index over sentences to select clusters of prevalent
opinions. Our approach leverages pretrained Large Language Models to generate
coherent and fluent summaries that are attributable and accurately reflect the
popularity of opinions in the input reviews. Extensive experiments show that,
as well as generating higher quality summaries, HIRO learns a more
semantically distributed representation than competitive baselines. While we
limit our experiments to opinion summarization, we believe that HIRO could be
usefully applied to a wide range of other retrieval-augmented generation
tasks.

## References

  * Aharoni et al. (2023) Roee Aharoni, Shashi Narayan, Joshua Maynez, Jonathan Herzig, Elizabeth Clark, and Mirella Lapata. 2023.  Multilingual summarization with factual consistency evaluation.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 3562–3591, Toronto, Canada. Association for Computational Linguistics.
  * Amplayo et al. (2021a) Reinald Kim Amplayo, Stefanos Angelidis, and Mirella Lapata. 2021a.  Aspect-controllable opinion summarization.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 6578–6593, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Amplayo et al. (2021b) Reinald Kim Amplayo, Stefanos Angelidis, and Mirella Lapata. 2021b.  Unsupervised opinion summarization with content planning.  In _Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021_ , pages 12489–12497. AAAI Press.
  * Angelidis et al. (2021) Stefanos Angelidis, Reinald Kim Amplayo, Yoshihiko Suhara, Xiaolan Wang, and Mirella Lapata. 2021.  Extractive opinion summarization in quantized transformer spaces.  _Transactions of the Association for Computational Linguistics_ , 9:277–293.
  * Basu Roy Chowdhury et al. (2022) Somnath Basu Roy Chowdhury, Chao Zhao, and Snigdha Chaturvedi. 2022.  Unsupervised extractive opinion summarization using sparse coding.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1209–1225, Dublin, Ireland. Association for Computational Linguistics.
  * Beltagy et al. (2020) Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020.  Longformer: The long-document transformer.  _arXiv:2004.05150_.
  * Bhaskar et al. (2023) Adithya Bhaskar, Alex Fabbri, and Greg Durrett. 2023.  Prompted opinion summarization with GPT-3.5.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 9282–9300, Toronto, Canada. Association for Computational Linguistics.
  * Bražinskas et al. (2020) Arthur Bražinskas, Mirella Lapata, and Ivan Titov. 2020.  Unsupervised opinion summarization as copycat-review generation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5151–5169, Online. Association for Computational Linguistics.
  * Bražinskas et al. (2021) Arthur Bražinskas, Mirella Lapata, and Ivan Titov. 2021.  Learning opinion summarizers by selecting informative reviews.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 9424–9442, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Cattan et al. (2023) Arie Cattan, Lilach Eden, Yoav Kantor, and Roy Bar-Haim. 2023.  From key points to key point hierarchy: Structured and expressive opinion summarization.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 912–928, Toronto, Canada. Association for Computational Linguistics.
  * Chen et al. (2010) Yongjian Chen, Guan Tao, and Cheng Wang. 2010.  Approximate nearest neighbor search by residual vector quantization.  _Sensors (Basel, Switzerland)_ , 10:11259–73.
  * Clark et al. (2023) Elizabeth Clark, Shruti Rijhwani, Sebastian Gehrmann, Joshua Maynez, Roee Aharoni, Vitaly Nikolaev, Thibault Sellam, Aditya Siddhant, Dipanjan Das, and Ankur Parikh. 2023.  SEAHORSE: A multilingual, multifaceted dataset for summarization evaluation.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 9397–9413, Singapore. Association for Computational Linguistics.
  * Coavoux et al. (2019) Maximin Coavoux, Hady Elsahar, and Matthias Gallé. 2019.  Unsupervised aspect-based multi-document abstractive summarization.  In _Proceedings of the 2nd Workshop on New Frontiers in Summarization_ , pages 42–47, Hong Kong, China. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Erkan and Radev (2004) G. Erkan and D. R. Radev. 2004.  LexRank: Graph-based lexical centrality as salience in text summarization.  _Journal of Artificial Intelligence Research_ , 22:457–479.
  * Goyal et al. (2022) Tanya Goyal, Junyi Jessy Li, and Greg Durrett. 2022.  News summarization and evaluation in the era of GPT-3.  _arXiv preprint_.
  * Gu et al. (2022) Albert Gu, Karan Goel, and Christopher Ré. 2022.  Efficiently modeling long sequences with structured state spaces.  In _The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022_. OpenReview.net.
  * He et al. (2021) Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021.  Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing.
  * Hofstätter et al. (2021) Sebastian Hofstätter, Sophia Althammer, Michael Schröder, Mete Sertkan, and Allan Hanbury. 2021.  Improving efficient neural ranking models with cross-architecture knowledge distillation.
  * Hosking et al. (2023a) Tom Hosking, Phil Blunsom, and Max Bartolo. 2023a.  Human feedback is not gold standard.
  * Hosking et al. (2023b) Tom Hosking, Hao Tang, and Mirella Lapata. 2023b.  Attributable and scalable opinion summarization.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 8488–8505, Toronto, Canada. Association for Computational Linguistics.
  * Hu and Liu (2004) Minqing Hu and Bing Liu. 2004.  Mining and summarizing customer reviews.  In _Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , KDD ’04, page 168–177, New York, NY, USA. Association for Computing Machinery.
  * Hubert and Arabie (1985) Lawrence Hubert and Phipps Arabie. 1985.  Comparing partitions.  _Journal of Classification_ , 2(1):193–218.
  * Iso et al. (2021) Hayate Iso, Xiaolan Wang, Yoshihiko Suhara, Stefanos Angelidis, and Wang-Chiew Tan. 2021.  Convex Aggregation for Opinion Summarization.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 3885–3903, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Jang et al. (2017) Eric Jang, Shixiang Gu, and Ben Poole. 2017.  Categorical reparameterization with Gumbel-softmax.  In _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings_. OpenReview.net.
  * Jiang et al. (2023) Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023.  Mistral 7b.
  * Jones (1972) Karen Spärck Jones. 1972.  A statistical interpretation of term specificity and its application in retrieval.  _Journal of Documentation_ , 28:11–21.
  * Kedzie et al. (2018) Chris Kedzie, Kathleen McKeown, and Hal Daumé III. 2018.  Content selection in deep learning models of summarization.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 1818–1828, Brussels, Belgium. Association for Computational Linguistics.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
  * Laban et al. (2022) Philippe Laban, Tobias Schnabel, Paul N. Bennett, and Marti A. Hearst. 2022.  SummaC: Re-visiting NLI-based models for inconsistency detection in summarization.  _Transactions of the Association for Computational Linguistics_ , 10:163–177.
  * Lan et al. (2020) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020.  ALBERT: A lite BERT for self-supervised learning of language representations.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
  * Lewis et al. (2020) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020.  Retrieval-augmented generation for knowledge-intensive nlp tasks.  In _Proceedings of the 34th International Conference on Neural Information Processing Systems_ , NIPS’20, Red Hook, NY, USA. Curran Associates Inc.
  * Li et al. (2023) Haitao Li, Qingyao Ai, Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Zheng Liu, and Zhao Cao. 2023.  Constructing tree-based index for efficient and effective dense retrieval.  In _Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , SIGIR ’23, page 131–140, New York, NY, USA. Association for Computing Machinery.
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  * Liu et al. (2023) Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2023.  Lost in the middle: How language models use long contexts.
  * Liu and Lapata (2019) Yang Liu and Mirella Lapata. 2019.  Hierarchical transformers for multi-document summarization.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 5070–5081, Florence, Italy. Association for Computational Linguistics.
  * Louis and Maynez (2023) Annie Louis and Joshua Maynez. 2023.  OpineSum: Entailment-based self-training for abstractive opinion summarization.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 10774–10790, Toronto, Canada. Association for Computational Linguistics.
  * Maddison et al. (2017) Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. 2017.  The concrete distribution: A continuous relaxation of discrete random variables.  In _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings_. OpenReview.net.
  * Malon (2023) C. Malon. 2023.  Automatically evaluating opinion prevalence in opinion summarization.  In _The 6th Workshop on e-Commerce and NLP (KDD 2023)_.
  * Narayan et al. (2023) Shashi Narayan, Joshua Maynez, Reinald Kim Amplayo, Kuzman Ganchev, Annie Louis, Fantine Huot, Anders Sandholm, Dipanjan Das, and Mirella Lapata. 2023.  Conditional generation with a question-answering blueprint.  _Transactions of the Association for Computational Linguistics_ , 11:974–996.
  * Opper et al. (2023) Mattia Opper, Victor Prokhorov, and Siddharth N. 2023.  StrAE: Autoencoding for pre-trained embeddings using explicit structure.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 7544–7560, Singapore. Association for Computational Linguistics.
  * Puduppully et al. (2019) Ratish Puduppully, Li Dong, and Mirella Lapata. 2019.  Data-to-text generation with content selection and planning.  In _The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019_ , pages 6908–6915. AAAI Press.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21(140):1–67.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence embeddings using Siamese BERT-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3982–3992, Hong Kong, China. Association for Computational Linguistics.
  * Schuster et al. (2021) Tal Schuster, Adam Fisch, and Regina Barzilay. 2021.  Get your vitamin C! robust fact verification with contrastive evidence.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 624–643, Online. Association for Computational Linguistics.
  * Shen and Wan (2023) Yuchen Shen and Xiaojun Wan. 2023.  Opinsummeval: Revisiting automated evaluation for opinion summarization.
  * Sønderby et al. (2017) Casper Kaae Sønderby, Ben Poole, and Andriy Mnih. 2017.  Continuous relaxation training of discrete latent variable image models.  In _Beysian DeepLearning workshop, NIPS_ , volume 201.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023.  Llama 2: Open foundation and fine-tuned chat models.
  * van den Oord et al. (2018) Aäron van den Oord, Yazhe Li, and Oriol Vinyals. 2018.  Representation learning with contrastive predictive coding.  _CoRR_ , abs/1807.03748.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_ , pages 5998–6008.
  * Vendrov et al. (2016) Ivan Vendrov, Ryan Kiros, Sanja Fidler, and Raquel Urtasun. 2016.  Order-embeddings of images and language.  In _4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings_.
  * Wu et al. (2022) Yuxiang Wu, Matt Gardner, Pontus Stenetorp, and Pradeep Dasigi. 2022.  Generating data to mitigate spurious correlations in natural language inference datasets.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2660–2676, Dublin, Ireland. Association for Computational Linguistics.
  * Xu et al. (2023) Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, and Bryan Catanzaro. 2023.  Retrieval meets long context large language models.
  * Zeghidour et al. (2022) Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. 2022.  Soundstream: An end-to-end neural audio codec.  _IEEE/ACM Trans. Audio, Speech and Lang. Proc._ , 30:495–507.
  * Zhang et al. (2024) Huajian Zhang, Yumo Xu, and Laura Perez-Beltrachini. 2024.  Fine-grained natural language inference based faithfulness evaluation for diverse summarisation tasks.  In _EACL 2024_.

##  Appendix A Annotation Instructions

> Instructions
>
> In this task you will be presented with some reviews of a product/hotel,
> followed by two summaries produced by different automatic systems. Your task
> is to rate the system summaries based on the criteria listed below.
>
> First, please skim read through the reviews, to try to get an overall idea
> of what opinions are mentioned frequently. Then, read the system summaries
> carefully, and rate each one according to the criteria.
>
> Please read the criteria descriptions and system summaries carefully, and
> whenever is necessary re-read the summaries or reviews. You might want to
> use your browser’s search function to help find parts of reviews that are
> relevant.
>
> Criteria
>
> Accuracy – Which system summary accurately reflects the balance of opinion
> in the input reviews? Statements in the summary should be backed up by
> multiple reviews.
>
> Detail – Which system summary includes more specific details?
>
> Coherence & Fluency – Which system summary is easy to read and avoids
> contradictions?
>
> Overall – Which system summary do you think is better, overall?

##  Appendix B Hyperparameters

Parameter | Value
---|---
Embedding dim. $\mathbb{D}$ | 768
Encoder layers | 5
Feedforward dim. | 2048
Transformer heads | 8
Depth $D$ | 12
Codebook size $K$ | 12
Optimizer |  Adam Kingma and Ba (2015)
Learning rate | 1e-4
Batch size | 384
$\omega$ | 150
$\alpha_{init}$ | 0.5
$\tau_{0}$ | 1.0
$\tau_{min}$ | 0.5
$\gamma_{temp}$ | 33333
$\beta_{KL}$ | 0.0025
$\beta_{NL}$ | 0.05
$\gamma_{NL}$ | 1.5
top-$k$ subpaths | 8
$\operatorname*{tp}-\operatorname*{ibp}$ smoothing $\alpha$ |  6 (Space), 3 (AmaSum)
Table 4: Hyperparameter values used for our experiments.

##  Appendix C LLM prompts

⬇

Review:

[...] (x8)

Write a summary in 70 words or less:\end{lstlisting}

\begin{lstlisting}[caption=HIRO\textsubscript{sent},captionpos=b]

Here is a list of sentences taken from reviews of the {entity name}:

[...]

In no more than 10 words, write a single concise sentence that includes the
main point:

Prompt 1: Baseline LLMs

⬇

Here is a list of sentences taken from reviews of the {entity name}:

[...]

In no more than 60 words, write a concise summary that includes the main
points:\end{lstlisting}

\section{Dataset Statistics}

\label{appx:datasets}

% See \Cref{tab:dataset_stats_training} and \Cref{tab:dataset_stats_eval}.

\begin{table}[!h]

\centering

\small

\begin{tabular}{r|rr}

& \textsc{Space} & AmaSum \\\

\hline\hline

Entities & 8350 & 7255 \\\

Reviews & 303,357 & 533,972 \\\

% Reviews per entity & 36.3 & 73.6 \\\

Training pairs $(\mathbf{x},\mathbf{x}_{+})$ & 1,373,079 & 2,991,478 \\\

\end{tabular}

\caption{Statistics for the training datasets. }

\label{tab:dataset_stats_training}

\end{table}

\begin{table}[ht]

\centering

\small

\begin{tabular}{r|rr}

& \textsc{Space} & AmaSum \\\

\hline\hline

Entities & 25 & 200 \\\

Reviews per entity & 100 & 560.4 \\\

Review length (words) & 162.6 & 49.9 \\\

Ref. summaries (words) & 82.0 & 80.1 \\\

\end{tabular}

\caption{Statistics for the evaluation datasets.}

\label{tab:dataset_stats_eval}

\end{table}

\section{Example Outputs}

\label{appx:examples}

See \Cref{tab:output}, \Cref{tab:output2} and \Cref{tab:evidence_example}.

\begin{table*}[ht]

\renewcommand{\arraystretch}{1.2}

\centering

\small

\begin{tabular}{@{}m{2cm}@{~}|@{~}m{13.2cm}@{}}

\textbf{System} & \textbf{Output} \\\

\hline\hline

% ix=20

\textit{Reference} &

The staff were very welcoming and were always happy to help you with whatever
was needed. The comfortably-sized rooms were very nice and clean, and the
bathrooms were exceptionally clean and well equipped. Breakfast was
continental and included in the price, offering baguettes, croissants, cereal,
yogurts, etc.

\\\

\hline

\textit{Random \newline Review} &

A real parisienne experience. The hotel is a beautiful building and the
location perfect. We walked everywhere and the local area is a delight with
many bars and FRIENDLY restaurants. We are an older couple and went to Paris
to get engaged - it was all perfect and the hotel’s ambience added to the
experience. Highly recommended if you want a small, local hotel as opposed to
a large modern one. Many thanks - we would return and recommend you to
friends. Margaret and David

\\\

\hline

\textit{SemAE} &

We were able to walk to all the tourists sights, except Montmarte and the
Eiffel Tower. The hotel staff was very friendly and helpful during our stay.
The staff is very friendly and helpful and the breakfast is a perfect start to
the day. The rooms was sufficent in size, the bed and pillows were very
comfortable. The ladies in the breakfast area were very friendly and
courteous. The bathroom was clean and the shower

\\\

\hline

\textit{CopyCat} &

We stayed at this hotel for 3 nights. The hotel was very clean and the staff
very helpful. The location is perfect for exploring the city and the metro
station is a short walk away. We will be back!

\\\

\hline

\textit{COOP} &

The hotel is in a great location, close to the metro and the metro. The staff
were very friendly and helpful. The rooms were clean and comfortable, and the
breakfast was very good. The hotel is in a great location, close to the Metro
and the metro. The only downside is that the rooms are a little on the small
side, but they are very clean and the staff is very friendly and helpful.

\\\

\hline

\textit{Hercules\textsubscript{abs}} &

The room was clean and comfortable. The staff was very friendly and helpful.
walking distance to everything. The room was clean and comfortable. breakfast
was good. The hotel is in a great location, just a few minutes walk from the
train station. breakfast was fine. The room and bathroom were very clean. The
staff spoke English and were very helpful. There is also a small restaurant on
the ground floor.

\\\

\hline

\textit{Mistral 7B} &

The St. Christophe Hotel in Paris offers small, clean, and cozy rooms in a
lively neighborhood near the Place Contrascarpe and Rue Mouffetard. The staff
are helpful and friendly, and the location is convenient with easy access to
metro stations and nearby restaurants. The hotel lacks air conditioning, so it
may not be ideal during warm weather. Free Wi-Fi is available in the lobby.
Overall, the St. Christophe Hotel is a charming and elegant choice for
travelers to Paris.

\\\

\hline

\textit{HIRO\textsubscript{ext}} &

Nearby, and very close to 2 Metro stations. The hotel is 1 block to the Metro
(line 7) between the North and South exits of the "place Monge" stop. The
bathroom was clean and functional. The staff were friendly and helpful at all
times. Breakfast was Plentiful-Cold cereal, great coffee, yogurt and a
wonderful basket of Croissants, breads and Pastries. The shower excellent.

\\\

\hline

\textit{HIRO\textsubscript{sent}} &

The Hotel Saint Christophe is conveniently located near multiple metro
stations. The Hotel Saint Christophe is centrally located near the Metro and
restaurants in the Latin Quarter. The Hotel Saint Christophe offers clean,
functional bathrooms in small rooms. The Hotel Saint Christophe’s friendly and
helpful staff provided excellent service. Breakfast offers cereal, yogurt,
pastries, and hot beverages. Guests praised the Hotel Saint Christophe for its
excellent showers.

\\\

\hline

\textit{HIRO\textsubscript{doc}} &

The Hotel Saint Christophe is centrally located in the Latin Quarter, close to
multiple Metro stations and attractions such as Notre Dame and Jardin
Luxembourg. Rooms are clean and adequate, though small, and breakfast offers a
variety of options. The staff are friendly and helpful, making for a pleasant
stay. The hotel is conveniently located near shops, restaurants, and markets.
The bathrooms are clean and modern, though some may be smaller than expected
for Paris.

\end{tabular}

% \vspace*{-.2cm}

\caption{Examples of generated summaries for an entity from \textsc{Space}.
HIRO\textsubscript{doc} generates concise, informative and coherent summaries
that accurately reflect the balance of opinions from input reviews. Mistral 7B
prompted directly with reviews also generates coherent summaries, but are
based on a small sample of reviews and do not accurately reflect the overall
balance of opinions. Previous abstractive systems, CopyCat, COOP and
Hercules\textsubscript{abs}, generate generic summaries that lack detail and
are fairly generic.}

% \vspace*{-.2cm}

\label{tab:output}

\end{table*}

\begin{table*}[ht]

\renewcommand{\arraystretch}{1.2}

\centering

\small

\begin{tabular}{@{}m{2cm}@{~}|@{~}m{13.2cm}@{}}

\textbf{System} & \textbf{Output} \\\

\hline\hline

% ix=18

\textit{Reference} &

These headphones are your best option if you’re looking for noise-cancelling
earbuds, but they are pricey. Noise-cancelling earbuds. Available in black or
white. Stay Hear tips keep earbuds in your ears while maximizing comfort.
Impressive sound quality. Carrying case included. These are expensive for
earbuds, so you should only buy them if you’re going to use them often

\\\

\hline

\textit{Random \newline Review} &

Excellent. I’m comparing this to the Bose on ear headphones and this cancels
noise better. I’m a frequent flyer and like to disappear during flights. This
helps me get into another zone whether it’s listening to electro-house or
watching the latest movies while flying. Highly recommend it!

\\\

\hline

\textit{CopyCat} &

So far so good. The only issue I have is that the volume is not as loud as my
old one, but other than that I’m happy with this purchase.

\\\

\hline

\textit{COOP} &

These are the best ear buds out of the box. The sound quality is great, the
bass is very good, and the noise cancelling feature is amazing. The only
downside is that it doesn’t have a USB-C port on the back of the device. It’s
not a big deal, but if you want to use it for work, you can’t go wrong with
this product.

\\\

\hline

\textit{Hercules\textsubscript{abs}} &

Love these Earbuds. These are great for listening to music. Noise cancelling
works well. Long battery life. These are my favorite Headphones. The sound
quality is very clear. The sound quality is great and the noise cancellation
is good. Got this for my wife for Christmas. The sound quality is great and
the Mic is clear. The Bass is a bit lacking. The ear pads are soft and
comfortable. These Headphones are the best ever. The Headphones are very
comfortable.

\\\

\hline

\textit{Mistral 7B} &

These earbuds have received positive reviews from users who travel frequently,
find them a game changer, and have even convinced friends to buy them. Some
users have purchased them for their children who are also satisfied. The
earbuds have received five-star ratings from both the buyers and their sons.
One user’s wife loves them and uses them when the user snores at night.
However, some users have noted that the noise-cancelling feature is not as
effective as over-ear headphones.

\\\

\hline

\textit{HIRO\textsubscript{ext}} &

The noise cancelling is fantastic. These are the best Headphones I’ve ever
owned. Well worth the money! The batteries last a long time and charge
quickly. I have the over the ears pair and also these in the ears.

\\\

\hline

\textit{HIRO\textsubscript{sent}} &

The Bose QuietComfort 20 headphones offer exceptional noise cancellation and
comfort. These Bose QuietComfort 20 headphones are exceptional, providing
excellent noise cancellation and sound quality. These headphones are worth the
investment due to their exceptional noise-cancelling capabilities. Battery
lasts long for noise cancellation, charging quickly. Some users report over 10
hours usage. Some prefer Bose QuietComfort 20 earbuds for flights and noisy
environments, others find them dizzying.

\\\

\hline

\textit{HIRO\textsubscript{doc}} &

The Bose QuietComfort 20 Acoustic Noise Cancelling Headphones are widely
praised for their excellent noise cancellation capability, making them a
popular choice for frequent flyers and those working in noisy environments.
They are also praised for their comfort, sound quality, and long battery life.
However, some customers have expressed concerns about the non-replaceable
battery and the price. Overall, these headphones are considered a worthwhile
investment for their impressive noise cancellation and sound quality.

\end{tabular}

% \vspace*{-.2cm}

\caption{Examples of generated summaries for an entity from AmaSum.}

\vspace*{-.2cm}

\label{tab:output2}

\end{table*}

\begin{table*}[ht]

\renewcommand{\arraystretch}{1.2}

\centering

\small

\begin{tabular}{@{}m{2cm}@{~}|@{~}m{13.2cm}@{}}

\hline\hline

\textit{$q_{1:d}$} & (6,2,10) \\\

\hline

\multirow{16}{*}{\textit{Evidence}} & The pool area was very nice. \\\

& The staff was very Friendly and helpful. \\\

& the pool area was very enjoyable \\\

& The pool area is very nice. \\\

& The pool area is fantastic \\\

& The pool area was nice \\\

& the pool area is very attractive \\\

& The staff was so friendly and helpful. \\\

& The pool was nice \\\

& The staff was friendly and accommodating. \\\

& The pool was very nice and the lobby inviting \\\

& the staff was welcoming and helpful \\\

& The hotel staff was very friendly and accommodating. \\\

& Staff was friendly and helpful. \\\

& The pool area is very nice and large with several water features. \\\

& The staff was really great and helpful \\\

\hline

\textit{HIRO\textsubscript{sent}} & The pool area and friendly staff make this
hotel a enjoyable stay. \\\

\hline\hline

\textit{$q_{1:d}$} & (10, 8, 6) \\\

\hline

\multirow{6}{*}{\textit{Evidence}} &

Free parking is also available to the rear of the hotel. \\\

& Free parking in the hotel yard. \\\ \

& A big plus was the free parking and large selection for continental
breakfast. \\\

& The free plentiful parking (some even under cover) is also one of the better
features of the hotel. \\\

& It has free parking (a big deal in Key West), an awesome pool, and the best
free continental breakfast I’ve ever seen. \\\

& Parking is free with some covered parking. \\\

\hline

\textit{HIRO\textsubscript{sent}} & Free parking and continental breakfast are
notable features of this hotel. \\\

\hline\hline

\textit{$q_{1:d}$} & (9,) \\\

\hline

\multirow{9}{*}{\textit{Evidence}} & I was extremely disappointed in the rooms
\\\

& Only one mirror in the room. \\\

& The rooms are loud. \\\

& the rooms are awful \\\

& the room was small and shabby \\\

& we were a little disappointed because the room was a lot smaller than we
expected \\\

& Extremely disappointed in the room, although the help was very nice as was
the outdoor area. \\\

& i suspect we were in the older part of the hotel with a double room. \\\

& Since the parking is directly under the rooms, it was Very loud espcially
from 12-3 am. \\\

\hline

\textit{HIRO\textsubscript{sent}} & Rooms were small, loud, and in need of
renovation with poor housekeeping. \\\

\hline\hline

\textit{HIRO\textsubscript{doc}} & The Fairfield Inn and Suites Key West
received positive reviews for its friendly and helpful staff, attractive and
nice pool area, and free parking. However, some guests were disappointed with
the small and shabby rooms, lack of storage space, and noise from parking and
adjacent rooms. The continental breakfast was also mentioned as a plus. \\\

\hline\hline

\end{tabular}

% \vspace*{-.2cm}

\caption{Examples of selected subpaths $q_{1:d}$, the corresponding evidential
clusters, the resulting HIRO\textsubscript{sent} output sentences, and the
overall HIRO\textsubscript{doc} summary for a single entity from
\textsc{Space}. We show only three out of five input clusters, and a subset of
all evidence sentences, due to space constraints.}

\vspace*{-.2cm}

\label{tab:evidence_example}

\end{table*}

\end{document}

Prompt 2: HIROdoc