# A variant of the Bonabeau model

Hsin-Lun Li<EMAIL_ADDRESS>
###### Abstract.

The Bonabeau model consists of a finite number of agents, each of which
occupies only one site on a square lattice. One agent is randomly selected and
chooses a neighboring site at random. If the neighboring site is vacant, the
agent moves to that site. However, if the site is occupied, a fight is
triggered. If the agent wins the fight, the two agents switch sites.
Otherwise, they remain in their original position. We consider a variant of
the Bonabeau model where the initial number of wins for all agents is
distributed between two absorbing states. We study various properties of the
model, including finite-time convergence. We demonstrate that achieving an
egalitarian society is impossible and explore the circumstances under which
the number of clusters in the distribution of wins of all agents is either two
or more than two.

###### Key words and phrases:

Social network, Bonabeau model, egalitarian society, polarization,
fragmentation

###### 1991 Mathematics Subject Classification:

91C20, 91D25, 91D30, 94C15

Hsin-Lun Li

1National Sun Yat-sen University, Kaohsiung 804, Taiwan

##  1\. Introduction

The Bonabeau model consists of a finite set of agents occupying the sites of a
two-dimensional square lattice with linear dimension $L$. Each agent occupies
a single site, and the density of agents on the lattice is denoted by $\rho$.
At each time step, an agent is uniformly randomly selected and chooses a
neighboring site uniformly at random. If the neighboring site is unoccupied,
the agent moves to that site. However, if the neighboring site is occupied, a
fight occurs. Let agents $i$ and $j$ be the attacking agent and the agent
being forced to fight, respectively. If agent $i$ defeats agent $j$, they
switch sites. Otherwise, they remain unchanged. The probability $Q_{ij}(t)$
that agent $i$ beating agent $j$ at time $t$ is defined by a Fermi function,

$Q_{ij}(t)=\frac{1}{1+\exp[{-\eta_{t}(F_{i}(t)-F_{j}(t))]}},$

where

$\begin{array}[]{rcl}\eta_{t}&=&\hbox{inverse temperature at time}\
t,\vspace{2pt}\\\ F_{i}(t)&=&\hbox{number of wins of agent}\ i\ \hbox{at
time}\ t.\end{array}$

$F_{i}(t)$ is an integer: $F_{i}(t)<0$ indicates that agent $i$ loses
$|F_{i}(t)|$ fights. Inverse temperature $\eta_{t}$ controls the level of
randomness in the dynamics: given $F_{i}(t)>F_{j}(t)$, agent $i$ almost surely
wins if $\eta_{t}\to\infty$, almost surely loses if $\eta_{t}\to-\infty$, and
wins with probability $1/2$ as agent $j$ if $\eta_{t}\to 0.$ The Bonabeau
model can be applied to interpret the rise of hierarchies from an initial
egalitarian society. The Bonabeau model, as described in [2, 3, 9, 1], differs
from the Deffuant model in [4, 5], the Hegselmann-Krause model in [6], and the
mixed Hegselmann-Krause model in [7, 8]. In the Bonabeau model, the
interactions between agents are fights, resulting in a win-or-loss outcome. In
contrast, in the Deffuant model, Hegselmann-Krause model, and mixed
Hegselmann-Krause model, the interactions between agents lead them to move
closer to each other in opinions.

Now, we consider a variant of the Bonabeau model by letting

  * •

$[n]=\\{1,2,\ldots,n\\}$ be the collection of all agents,

  * •

$G$ be a finite site graph,

  * •

$\eta_{t}\geq 0$ for all $t\geq 0,$

  * •

$(F_{i}(0))_{i\in[n]}\subset(\ell,h)$ for integers $\ell$ and $h$, and

  * •

$\ell$ and $h$ be absorbing states.

The number of wins serves as a representation of an agent’s power. The
condition $\eta_{t}>0$ at all times implies that powerful agents have a higher
likelihood of winning fights. $\eta_{t}=0$ indicates two fighting agents have
an equal chance to win the fight. When an agent enters an absorbing state, it
ceases to engage in further fights. Namely, a fight occurs at time $t$ if and
only if the two selected agents’ number of wins at time $t$ is in $(\ell,h).$
Therefore, an agent’s power cannot indefinitely increase or decrease. Agents
in absorbing states may obstruct the encounters between other agents, impeding
their interactions. An egalitarian society is achieved when
$(F_{i})_{i\in[n]}$ forms a single cluster, polarization arises when
$(F_{i})_{i\in[n]}$ forms two clusters, and fragmentation occurs when
$(F_{i})_{i\in[n]}$ exhibits more than two clusters. Let $|G|$ denote the
order of graph $G$, i.e., the number of vertices in $G$.

##  2\. Main results

We present an argument that the number of wins of all agents exhibits a
finite-time convergence property. Namely, all fights end in finite time.

######  Theorem 1.

$(F_{i}(t))_{t\geq 0}$ converges almost surely in finite time for all
$i\in[n].$

Let $T$ be the earliest time that $(F_{i}(t))_{t\geq 0}$, $i\in[n]$ converge
over time, i.e.,

$T=\inf\\{t\geq 0:F_{i}(s)=F_{i}(t)\ \hbox{for all $s\geq t$ and
$i\in[n]$}\\}.$

From Theorem 1, $T$ is almost surely finite. Corollary 2 states that agents
whose number of wins does not fall into the absorbing states at time T cannot
encounter each other from time $T$ on.

######  Corollary 2.

From time $T$ onwards, it is almost surely impossible for any agents whose
number of wins at time $T$ falls in $(\ell,h)$ to encounter each other.

Corollary 3 demonstrates that it is impossible to achieve an egalitarian
society at time $T$ on a connected site graph when there is more than one
agent. Even if the initial distribution represents an egalitarian society, a
minor difference at the next time step can lead to an increased disparity in
the number of wins among agents, rendering the achievement of an egalitarian
society unattainable. In particular, having an equal chance of winning a fight
does not guarantee the achievement of an egalitarian society.

######  Corollary 3.

It is almost surely impossible to achieve an egalitarian society at time $T$
for $n\geq 2$ and the site graph connected.

Corollary 4 presents the circumstances under which polarization or
fragmentation occurs. If $\ell=-h$ and the initial wins of all agents add up
to zero on the complete site graph with more than one agent, polarization
occurs at time $T$ if and only if there is an even number of agents.

######  Corollary 4.

Assume that $n\geq 2$, $\ell=-h$, $\sum_{i\in[n]}F_{i}(0)=0,$ and the site
graph is complete. Then, $(F_{i}(T))_{i\in[n]}$ is almost surely polarization
if and only if $n$ is even, i.e., $(F_{i}(T))_{i\in[n]}$ is almost surely
fragmentation if and only if $n$ is odd.

If the initial wins of all agents are in $(-h,h)$ and add up to zero, then the
distribution of wins exhibits more than two clusters at time $T$ if $G$ is a
cycle and the number of agents is odd and greater than one.

######  Corollary 5.

Assume that $\ell=-h$ and $\sum_{i\in[n]}F_{i}(0)=0$. Then,
$(F_{i}(T))_{i\in[n]}$ is fragmentation if $G$ is a cycle and $n>1$ is odd.

##  3\. The model

The crucial part is to find a submartingale or supermartingale as follows.

######  Lemma 6.

Let $Z_{t}=\sum_{i,j\in[n]}(F_{i}(t)-F_{j}(t))^{2}.$ Then, $(Z_{t})_{t\geq 0}$
is a submartingale. In particular,

$\mathbf{E}[Z_{t}-Z_{t+1}|\hbox{a fight at time}\ t]\leq-4n\ \hbox{if there is
a fight at time}\ t.$

###### Proof.

If there is no fight at time $t$, then $Z_{t}=Z_{t+1}$. Else, there is a fight
between some agents $p$ and $q$. Let $F_{i}=F_{i}(t)$ and
$F_{i}^{\star}=F_{i}(t+1)$. For
$(F_{i}-F_{j})^{2}-(F_{i}^{\star}-F_{j}^{\star})^{2}$, we consider the
following cases:

  1. (1)

Only one of $i$ and $j$ belongs to $\\{p,q\\}$,

  2. (2)

$\\{i,j\\}=\\{p,q\\}$.

Observe that

$\begin{array}[]{rcl}\displaystyle(F_{i}-F_{p})^{2}-(F_{i}-F_{p}^{\star})^{2}&=&(F_{p}^{\star}-F_{p})^{2}+2(F_{i}-F_{p}^{\star})(F_{p}^{\star}-F_{p})\vspace{2pt}\\\
\displaystyle(F_{i}-F_{q})^{2}-(F_{i}-F_{q}^{\star})^{2}&=&(F_{q}^{\star}-F_{q})^{2}+2(F_{i}-F_{q}^{\star})(F_{q}^{\star}-F_{q}).\end{array}$

Therefore,

$\displaystyle(F_{i}-F_{p})^{2}-(F_{i}-F_{p}^{\star})^{2}+(F_{i}-F_{q})^{2}-(F_{i}-F_{q}^{\star})^{2}$
$\displaystyle\hskip
28.45274pt=(F_{p}^{\star}-F_{p})^{2}+(F_{q}^{\star}-F_{q})^{2}-2(F_{p}^{\star}-F_{p})(F_{p}^{\star}-F_{q}^{\star}).$
$\displaystyle
Z_{t}-Z_{t+1}=2\bigg{\\{}\sum_{i\in[n]-\\{p,q\\}}\big{[}(F_{i}-F_{p})^{2}-(F_{i}-F_{p}^{\star})^{2}+(F_{i}-F_{q})^{2}-(F_{i}-F_{q}^{\star})^{2}\big{]}$
$\displaystyle\hskip
56.9055pt+(F_{p}-F_{q})^{2}-(F_{p}^{\star}-F_{q}^{\star})^{2}\bigg{\\}}$

Let $A_{t}$ be the event of a fight at time $t$. Then,

$\displaystyle\mathbf{E}[Z_{t}-Z_{t+1}|A_{t}]=\mathbf{E}\bigg{[}Q_{pq}\mathbf{E}\big{[}Z_{t}-Z_{t+1}|p\
\hbox{wins},(F_{i})_{i\in[n]}\big{]}+Q_{qp}\mathbf{E}\big{[}Z_{t}-Z_{t+1}|p\
\hbox{loses},(F_{i})_{i\in[n]}\big{]}|A_{t}\bigg{]}$ $\displaystyle\hskip
14.22636pt=2\bigg{\\{}\mathbf{E}\big{[}-2Q_{pq}(n-2)(F_{p}-F_{q}+1)-4Q_{pq}(F_{p}-F_{q}+1)$
$\displaystyle\hskip
28.45274pt+2Q_{qp}(n-2)(F_{p}-F_{q}-1)+4Q_{qp}(F_{p}-F_{q}-1)|A_{t}\big{]}\bigg{\\}}$
$\displaystyle\hskip
14.22636pt=4n\mathbf{E}\big{[}-Q_{pq}(F_{p}-F_{q}+1)+Q_{qp}(F_{p}-F_{q}-1)|A_{t}\big{]}$
$\displaystyle\hskip
14.22636pt=-4n\bigg{\\{}\mathbf{E}\big{[}(Q_{pq}-Q_{qp})(F_{p}-F_{q})|A_{t}\big{]}+1\bigg{\\}}\leq-4n$

∎

###### Proof of Theorem 1.

According to Lemma 6, $(-Z_{t})_{t\geq 0}$ is a supermartingale bounded by
$n^{2}(\ell-h)^{2}.$ By martingale convergence theorem, $-Z_{t}$ converges
almost surely to a random variable $-Z_{\infty}$ with finite expectation,
therefore

$\lim_{t\to\infty}(Z_{t}-Z_{t+1})=Z_{\infty}-Z_{\infty}=0\ \hbox{almost
surely.}$

So $|Z_{t}-Z_{t+1}|<1/2$ for some $N\geq 0$ and for all $t\geq N$. Since
$Z_{t}-Z_{t+1}\in\mathbbm{Z}$, $Z_{t}-Z_{t+1}=0$ almost surely for all $t\geq
N.$ Via Lemma 6,

$\mathbf{E}[Z_{t}-Z_{t+1}|\hbox{a fight at time}\ t]\leq-4n\quad\hbox{if there
is a fight at time $t$.}$

Hence, there is no fight after time $N$. This completes the proof. ∎

###### Proof of Corollary 2.

Assume that this is not the case. Let $F_{t}$ be the event that a fight
happens at some time $t\geq T$ and $E_{t}$ be the event that the agents whose
number of wins at time $t$ is in $(\ell,h)$ meet each other. By Theorem 1,
$P(F_{t})=0$, therefore

$0=P(F_{t})\geq P(F_{t}|E_{t})P(E_{t})>\frac{2}{|G|^{2}}P(E_{t})>0,\ \hbox{a
contradiction.}$

∎

###### Proof of Corollary 3.

Given $F_{i}(0),\ i\in[n]$ in $(\ell,h)$, since the total number of wins is
conservative over time, $F_{i}(T)$ is the average number of winnings at time
0, which is in $(\ell,h)$, contradicting Corollary 2. ∎

###### Proof of Corollary 4.

Given $n\geq 2$ and the site graph complete. Say there are $n_{1}$ agents
whose number of wins at time $T$ is $h$ and $n_{2}$ agents whose number of
wins at time $T$ is $-h$. Via Corollary 2, there is at most one agent whose
number of wins at time $T$ is in $(-h,h)$. Namely, either there is only one
agent whose number of wins is not in absorbing states or all agents’ number of
wins is in absorbing states. If there is only one agent whose number of wins
is $x\in(-h,h)$ at time $T$. Then,

$0=n_{1}h-n_{2}h+x=(n_{1}-n_{2})h+x,\ \hbox{therefore}\ x=(n_{2}-n_{1})h.$

Since $x\in(-h,h)$, we get $n_{1}=n_{2}$, $x=0$ and $n=n_{1}+n_{2}+1$ odd. If
all agents’ number of wins is in absorbing states, then

$(n_{1}-n_{2})h=0\ \hbox{so}\ n_{1}=n_{2}\ \hbox{and}\ n=n_{1}+n_{2}\
\hbox{even}.$

Hence, polarization occurs at time $T$ when $n$ is even and fragmentation
occurs at time $T$ when $n$ is odd. ∎

###### Proof.

By Corollary 2, one of the clusters at time $T$ is in an absorbing state. If
polarization occurs at time $T$, say $n_{1}$ agents in state $h$ and $n_{2}$
agents in state $x$. If $x\in(-h,h)$, then $n_{2}\geq n_{1}+1$. So there are
two agents in state $x$ that encounter each other on a cycle, contradicting
Corollary 2. Given $G$ a cycle and $n>1$,

$\displaystyle\hskip 28.45274pt\hbox{polarization occurs at time}\ T$
$\displaystyle\iff\hbox{the states of all agents are $\pm h$ at time}\ T$
$\displaystyle\iff(n_{1}-n_{2})h=0$ $\displaystyle\iff n_{1}=n_{2}\implies n\
\hbox{even}$

By Corollary 3, $n$ odd implies fragmentation occurs at time $T.$ ∎

##  4\. Statements and Declarations

###  4.1. Competing Interests

The author is funded by the National Science and Technology Council in Taiwan.

###  4.2. Data availability

No associated data was used.

## References

  * [1] E. Bonabeau.  Agent-based modeling: Methods and techniques for simulating human systems.  Proceedings of the national academy of sciences, 99(suppl_3):7280–7287, 2002.
  * [2] E. Bonabeau, M. Dorigo, and G. Theraulaz.  Swarm intelligence: from natural to artificial systems.  Number 1. Oxford university press, 1999.
  * [3] J.-L. Deneubourg, S. Goss, N. Franks, A. Sendova-Franks, C. Detrain, and L. Chrétien.  The dynamics of collective sorting robot-like ants and ant-like robots.  In From animals to animats: proceedings of the first international conference on simulation of adaptive behavior, pages 356–365, 1991\.
  * [4] N. Lanchier.  The critical value of the Deffuant model equals one half.  ALEA Lat. Am. J. Probab. Math. Stat., 9(2):383–402, 2012.
  * [5] N. Lanchier and H.-L. Li.  Probability of consensus in the multivariate Deffuant model on finite connected graphs.  Electronic Communications in Probability, 25:1–12, 2020.
  * [6] N. Lanchier and H.-L. Li.  Consensus in the Hegselmann–Krause model.  Journal of Statistical Physics, 187(3):1–13, 2022.
  * [7] H.-L. Li.  Mixed Hegselmann-Krause dynamics.  Discrete and Continuous Dynamical Systems - B, 27(2):1149–1162, 2022.
  * [8] H.-L. Li.  Mixed Hegselmann-Krause dynamics II.  Discrete and Continuous Dynamical Systems - B, 28(5):2981–2993, 2023.
  * [9] G. Theraulaz and E. Bonabeau.  A brief history of stigmergy.  Artificial life, 5(2):97–116, 1999.


# Fine-Grained Opinion Summarization with Minimal Supervision

Suyu Ge, Jiaxin Huang, Yu Meng, Sharon Wang, Jiawei Han

###### Abstract

Opinion summarization aims to profile a target by extracting opinions from
multiple documents. Most existing work approaches the task in a semi-
supervised manner due to the difficulty of obtaining high-quality annotation
from thousands of documents. Among them, some uses aspect and sentiment
analysis as a proxy for identifying opinions. In this work, we propose a new
framework, _FineSum_ , which advances this frontier in three aspects: (1)
_minimal supervision_ , where only aspect names and a few aspect/sentiment
keywords are available; (2) _fine-grained opinion analysis_ , where sentiment
analysis drills down to the sub-aspect level; and (3) _phrase-based
summarization_ , where opinion is summarized in the form of phrases. FineSum
automatically identifies opinion phrases from the raw corpus, classifies them
into different aspects and sentiments, and constructs multiple fine-grained
opinion clusters under each aspect/sentiment. Each cluster consists of
semantically coherent phrases, expressing uniform opinions towards certain
sub-aspect or characteristics (e.g., positive feelings for “burgers” in the
“food” aspect). An opinion-oriented spherical word embedding space is trained
to provide weak supervision for the phrase classifier, and phrase clustering
is performed using the aspect-aware contextualized embedding generated from
the phrase classifier. Both automatic evaluation on the benchmark and
quantitative human evaluation validate the effectiveness of our approach.

##  1 Introduction

Opinion summarization is the task of aggregating user opinions towards a
single target from multiple documents (e.g., profiling a restaurant from
online reviews). It benefits intelligent decision making by succinctly
displaying diverse opinions to users and reducing the information overload.

Different from generic multi-document summarization, the large volumes of
reviews and the inherent subjectivity within them pose challenges to curating
golden annotation for this task, rendering end-to-end training infeasible. A
majority of work focuses on developing weakly-supervised or unsupervised
summarization approaches (Elsahar et al. 2021; Suhara et al. 2020). To further
handle the diversity and conflicts in user opinions, some approaches perform
aspect extraction and sentiment polarization at first, then generate summaries
for different aspects and sentiments in either extractive or abstractive forms
(Angelidis and Lapata 2018; Krishna and Srinivasan 2018).

Figure 1: An example of fine-grained opinion summarization (Opinion clusters
are separated by dot lines).

Though previous methods partly consider the heterogeneity in user opinions, we
argue that they still summarize at a coarse level for two reasons: (1)
Opinions in the same $\langle$aspect, sentiment$\rangle$ category may target
at different subjects (e.g., in the $\langle$food, good$\rangle$ category of
Fig. 1, one set of opinions can be about “burger” but the other about “fish”),
and (2) different opinions may focus on different characteristics of the same
subject (e.g., one may praise waiters for their kind service but the other may
comment on their slowness). Motivated by this, we propose to drill down and
aggregate similar opinions for each sub-aspect and characteristic.

Further, traditional extractive and abstractive methods may be sub-optimal for
fine-grained summarization. It is common to see multiple fine-grained opinions
entangled in the same sentence. In this scenario, extractive summarization
will either bring outlier or cause information loss; whereas abstractive
summarization usually suffers from hallucinations (Maynez et al. 2020) and
text degeneration (Holtzman et al. 2019). Therefore, instead of extracting or
generating sentences, we seek to extract and aggregate smaller semantic units,
such as phrases, as our fine-grained summarization.

In this paper, we propose FineSum, a fine-grained opinion summarization
approach, which leverages only aspect names and a few related keywords as
supervision. It consists of the following three stages: (1) _Extract
meaningful candidate phrases_ by performing syntactic analysis towards the raw
corpus. (2) _Classify each candidate phrase_ into a particular aspect and
sentiment. We leverage two modules as phrase classifiers: (i) the opinion-
oriented spherical embedding and (ii) the contextualized BERT classifier. The
opinion-oriented embedding explicitly represents aspects/sentiments along with
word semantics in a distinctive sphere space, and classifies phrases according
to their directional similarities with the aspect/sentiment embedding.
Meanwhile, it outputs aspect-related and sentiment-related sentences as weak
supervision to train the BERT classifier. To enhance model performance, we
additionally fine-tune BERT with jointly agreed phrases from the two
classifiers. (3) _Aggregate phrases within each aspect and sentiment to obtain
fine-grained opinion clusters_ so that phrases in the same cluster convey
coherent meanings.

We summarize our contributions as follows: (1) a minimally supervised approach
is proposed for fine-grained opinion summarization, which relies only on
aspect names and a few keywords; (2) extracted candidate phrases are
classified by training an opinion-oriented spherical embedding and leveraging
it to provide weak supervision for a contextualized classifier, and similar
phrases are then aggregated within each aspect and sentiment to obtain fine-
grained opinion clusters; and (3) Extensive automatic and human evaluations
verify the superiority of this approach.

##  2 Related Work

As the primary goal is to reduce redundancy, previous work on unsupervised or
weakly-supervised opinion summarization mainly adopts a popularity-based
approach (i.e., extract or generate sentences containing the most salient
opinions in the original corpus) (Di Fabbrizio, Stent, and Gaizauskas 2014;
Ganesan, Zhai, and Han 2010). A majority of early methods focus on extractive
summarization. Ku et al. (2006) define popular opinions using TF-IDF and use
pre-defined keyword sets to retrieve the most relevant and opinionated
sentences. Paul, Zhai, and Girju (2010) extract opinions according to a
variety of lexical and syntactic features, and calculate salience and
contrastiveness of sentences using random walk. Recently, with the
proliferation of end-to-end training, abstractive summarization receives much
more attention. A typical practice is to encode salient information using an
aggregated representation, then generate novel sentences by reconstructing
this representation. A representative method is MeanSum (Chu and Liu 2019),
which generates summaries by training an auto-decoder to reconstruct the
averaged input representation. Similarly, Amplayo and Lapata (2021) condense
review documents into multiple dense vectors and use a multi-source fusion
module to generate summaries. Though effective, extractive and abstractive
summarizations suffer from information loss since they only consider popular
opinions. Moreover, they overlook the heterogeneity and conflicts in opinions
by generating uniform summaries for all types of opinions.

Only a few unsupervised methods propose to summarize according to different
aspects and sentiments (Krishna and Srinivasan 2018; Frermann and Klementiev
2019). Angelidis and Lapata (2018) first couple extractive opinion
summarization with the tasks of aspect identification and sentiment
polarization. It uses an aspect extractor trained under a multi-task objective
and a sentiment predictor based on multiple instance learning. Following this,
Angelidis et al. (2021) represent aspects as discrete latent codes in the
quantized transformer space, and encode sentences to the aspect space using a
variational autoencoder. However, the above methods remain coarse-grained
because they are not designed to capture sub-aspects and specific
characteristics within each aspect. On the contrary, we propose to generate
fine-grained phrase clusters for sub-aspects. To our knowledge, this is the
first work attempting to generate opinion summarization in the form of phrase
clusters.

Figure 2: An overview of FineSum: Our proposed fine-grained opinion
summarization approach.

##  3 Problem Definition

Given a corpus $T$ containing reviews for targets $\\{t_{1},t_{2},\ldots\\}$
from a single domain (e.g., restaurants), we define a domain-related aspect
set $A=\\{a_{1},a_{2},\ldots\\}$ and sentiment set
$S=\\{s_{1},s_{2},\ldots\\}$ and input a keyword list as $L_{a}$ or $L_{s}$
for each $a$ or $s$. For every target $t$, we define its review sentence set
as $R=\\{r_{1},r_{2},\ldots\\}$, where each review $r$ consists of multiple
sentences $(x_{1},x_{2},\ldots)$. Each phrase $p$ is defined as a non-
overlapped word sequence $(w_{1},w_{2},\ldots)$ in one sentence. For each
target, our final model outputs are a set of clusters
$C=\\{c_{1},c_{2},\ldots\\}$ for every aspect-sentiment pair
$\langle$$a,s$$\rangle$, where each cluster $c$ contains multiple semantically
coherent phrases $(p_{1},p_{2},\ldots)$.

##  4 Approach

Fig. 2 illustrates the overall workflow of our approach, which decomposes the
task of aspect-based fine-grained opinion summarization to three stages. The
first stage, candidate phrase extraction, performs syntactic analysis to bring
up consecutive and informative multi-word sequences from raw corpus as
candidate phrases (Section 4.1). The following stage, opinion phrase
classification, aims to classify extracted phrases into different aspects and
sentiments (Section 4.2). The last stage, opinion phrase clustering, generates
fine-grained clusters within each aspect and sentiment by gathering
semantically coherent phrases into the same cluster (Section 4.3). We
introduce them as follows.

###  4.1 Candidate Phrase Extraction

Figure 3: An illustration of phrases extracted by dependency parsing and
constituency analysis. We highlight the subject and its associated
descriptions inside each [phrase].

As phrases form concise but complete semantic units in the original review
sentences, we propose to extract phrases as the basic component of
summarization, instead of the whole sentences. However, existing phrase mining
methods are usually designed to extract entity-like structures, which differs
from our goal of extracting opinions.

To mine opinions, we need to simultaneously discover a subject, which is
usually in the noun form, along with its associated descriptions, which are
usually in the form of adjective, adverb, or verb. We achieve this by
performing syntactic analysis towards each sentence using custom parsing
tools. Specifically, we consider two types of syntactic structure: (1)
dependency parsing which discovers a noun subject and its descriptive
adjectives/adverbs: If a noun is in one-hop relation with any adjective or
adverb, we count them together as a phrase; and (2) constituency analysis
which identifies a noun subject and its consecutive verb phrases: If a noun
component and a verb component exist at the same level in the constituency
parsing tree, they are considered as a phrase. We illustrate the results of
both methods in Fig. 3. More technical details of it can be found in the
Appendix. From the figure, dependency parsing usually mines concise phrases
with similar structures, whereas constituency analysis can identify phrases in
more diverse expression forms. We take the union of their results as our final
candidate phrase set, resulting in a phrase extractor with high recall.
According to our human evaluation on 50 reviews, the recall of this method
reaches 0.92, suggesting it as an intuitive yet effective approach. Note that
this semantic-agnostic method will inevitably introduce noisy phrases such as
“last night”. However, this will be corrected by the subsequent classification
stage. At this stage, our goal is to bring up as many candidates as possible.

###  4.2 Opinion Phrase Classification

##### Learning opinion-oriented embedding.

Given the aspect/sentiment set and keyword lists, we leverage them as seeds to
learn text embeddings tailored for aspects/sentiments. For simplicity, we take
aspect categories as an example to introduce our phrase embedding learning and
classification method, and sentiment categories follow the same procedure. To
learn an aspect-distinctive embedding space, we jointly embed text and aspects
in the spherical space, where directional similarity is used to effectively
characterize semantic correlations among words, sentences, and aspects. Each
aspect is surrounded by its representative keywords on the sphere. In this
way, the aspect information of a word is explicitly measured by its
directional similarity with different aspects.

The opinion-oriented spherical embedding is implemented via hierarchical topic
mining (Meng et al. 2020). We simplify it to the flat case and the learning
objectives are introduced as below:

$\displaystyle\forall a,x,$ $\displaystyle
w,\quad||\mathbf{a}||=||\mathbf{x}||=||\mathbf{w}||=1,$ (1)
$\displaystyle\mathcal{L}_{inter}=\sum_{a_{i}\in A}$
$\displaystyle\sum_{a_{j}\in
A\setminus\\{a_{i}\\}}min(0,1-\textbf{a}_{i}^{T}\textbf{a}_{j}-m_{inter}).$
$\displaystyle\mathcal{L}_{intra}=$ $\displaystyle\sum_{a_{i}\in
A}\sum_{w_{j}\in L_{a_{i}}}min(0,\mathbf{w}_{j}^{T}\textbf{a}_{i}-m_{intra}).$
$\displaystyle\mathcal{L}_{aspect}=$ $\displaystyle\sum_{x\in T}\log
p(\mathbf{x}|\mathbf{a}_{x})+$ $\displaystyle\sum_{x\in T}\sum_{w_{i}\in
x}\log p(\mathbf{w}_{i}|\mathbf{x})+$ $\displaystyle\sum_{x\in
T}\sum_{w_{i}\in x}\sum_{|j|\leq h,j\notin 0}\log
p(\mathbf{w}_{i+j}|\mathbf{w}_{i}).$ $\displaystyle\mathcal{L}=$
$\displaystyle-\mathcal{L}_{inter}-\mathcal{L}_{intra}-\mathcal{L}_{aspect},$

where $m_{inter}$ and $m_{intra}$ are two learnable parameters, and $h$ is the
context window length. The first objective $\mathcal{L}_{inter}$ encourages
inter-aspect distinctiveness across different aspects by enforcing the cosine
distance between any two aspects to be larger than $m_{inter}$. The second
objective $\mathcal{L}_{intra}$ requires the embeddings of aspect keywords to
be placed near the aspect center direction within a local region $m_{intra}$.
The third objective $\mathcal{L}_{aspect}$ models the corpus generation
process conditioned on the aspects in a three-step process: (1)
$p(\mathbf{x}|\mathbf{a}_{x})$ conditions each sentence $\mathbf{x}$ on an
aspect $\mathbf{a}_{x}$, (2) $p(\mathbf{w}_{i}|\mathbf{x})$ models the
semantic coherence between a word $\mathbf{w}_{i}$ and the sentence
$\mathbf{x}$ it appears, and (3) $p(\mathbf{w}_{i+j}|\mathbf{w}_{i})$ models
co-occurring words within local contexts. Note that all three steps use
directional similarity to model correlations. Namely,
$p(\mathbf{x}|\mathbf{a}_{x})\propto\mathbf{x}^{T}\mathbf{a}_{x}$. More
details on modeling and optimization can be found in the original topic mining
paper.

##### Knowledge distillation to contextualized classifier.

Context is crucial for phrase aspect classification. The context-free
spherical embedding space mainly captures word-level discriminative signals
but is insufficient to model sequential information from ordering of words.
Therefore, we propose to distill knowledge from the aspect-regularized
embedding space by generating confident soft predictions for high-quality
sentences, and fine-tune a pre-trained language model. Specifically, we
leverage soft predictions given by the directional similarity between sentence
embeddings and aspect embeddings as weak supervision to fine-tune the BERT-
base model (Devlin et al. 2019) for aspect classification111Similar with
aspect classification, we fine-tune another BERT-base model to classify
sentiments..

To provide high-quality supervision, for each aspect $a_{i}$, we select
top-$K$ sentences from the entire corpus with the highest
$\mathbf{x}^{T}\mathbf{a}_{i}$ score. We transform the directional similarity
to pseudo training labels $\mathbf{l}_{x}$ as below:

$l_{xi}=\frac{\exp(\alpha\cdot\mathbf{x}^{T}\mathbf{a}_{i})}{\sum\limits_{a_{i}\in
A}\exp(\alpha\cdot\mathbf{x}^{T}\mathbf{a}_{i})},$ (2)

where $l_{xi}$ is the probability of sentence $x$ belonging to the $i_{th}$
aspect. $\alpha$ is the temperature to control how greedy we want to learn
from the embedding-based prediction. We then train BERT on the pseudo training
sentences by minimizing the cross entropy $H$ between the embedding-based
prediction $\mathbf{l}$ and the output prediction $\mathbf{y}$ of BERT, namely

$H(x)=\sum_{x}\sum_{i}l_{xi}\log\frac{l_{xi}}{y_{xi}}.$ (3)

##### Fine-tuning classifier on jointly agreed phrases.

To identify the aspect of candidate phrases, one direct method is to leverage
the sentence-level classifier fine-tuned in previous stage. However, the
sentence-level model may not perform as well on phrases because phrases are
usually shorter, thus provide less context for accurate classification. Thus,
we propose to further enhance the BERT model by fine-tuning it for phrase-
level aspect classification. Note that the candidate phrase extraction stage
may include phrases not belonging to any aspect. To exclude them, we
additionally require the model to identify such “background” phrases by
training BERT to output a uniform aspect distribution on them.

To generate high-quality pseudo training samples for BERT, we exploit the
wisdom from both models (i.e., the opinion-oriented embedding and the weakly
trained BERT). We select their jointly agreed predictions as pseudo training
phrases, and generate pseudo labels $l^{\prime}_{s}$ for phrase $s$ as:

$l^{\prime}_{si}=\left\\{\begin{array}[]{ll}\frac{\exp(\alpha
y_{si})}{\sum\limits_{i\in|A|}\exp(\alpha
y_{si})}&{y_{si}\geq\theta_{1},\overline{\mathbf{w}}^{T}_{s}\mathbf{a}_{i}\geq\theta_{2}}\\\
\frac{1}{|A|}&{y_{si}<\theta_{1},\overline{\mathbf{w}}^{T}_{s}\mathbf{a}_{i}<\theta_{2}}\end{array},\right.$
(4)

where $\overline{\mathbf{w}}_{s}$ is the averaged embeddings of words in the
phrase $s$, $y_{si}$ is the predicted probability from BERT, and
$\overline{\mathbf{w}}^{T}_{s}\mathbf{a}_{i}$ is the directional similarity
between phrase $s$ and the $i^{th}$ aspect in the embedding space.
$\theta_{1}$ and $\theta_{2}$ are two probability thresholds. During
inference, we also use $\theta_{2}$ for BERT to decide whether a phrase
belongs to a certain aspect.

###  4.3 Fine-Grained Opinion Clustering

Given predictions from the previous stage, we can organize all extracted
phrases according to their aspects and sentiments. However, there are two
problems with this practice: (1) Phrases located in the same aspect and
sentiment may still cover diverse and heterogeneous opinions, varied by their
subjects and targeted characteristics; and (2) some phrases express similar
meanings using different words, which leads to unwanted semantic redundancy.
To solve these problems, we propose to represent fine-grained opinions by
automatically forming clusters under each aspect and sentiment. To guarantee
that phrases belonging to the same cluster convey consistent and coherent
meanings, we require them to locate near each other in the semantic space. We
achieve this by clustering in the fine-tuned BERT embedding space, as it
explicitly encodes aspect and sentiment information after training and fine-
tuning in the previous stage.

Specifically, we use the bottom-up hierarchical agglomerative clustering
(Dubitzky et al. 2013) which treats each phrase as a singleton cluster at the
outset, and then successively merges pairs of phrases until the euclidean
distance between phrases in the same cluster exceeds a pre-defined threshold
$T_{c}$.

The final output of FineSum is the fine-grained clusters under each aspect and
sentiment. Considering the semantic coherence in each cluster, one could
further reduce redundancy by selecting salient phrase(s) to represent the
whole cluster, or even selecting salient cluster(s) to represent each aspect-
sentiment pair. In this work, we stay on the original complete clusters to
offer comprehensive summarization. In real application, the best form of
summarization should be adjusted according to user needs, which is beyond the
scope of this paper.

##  5 Experiments

As no off-the-shelf evaluation framework exists, we evaluate model performance
quantitatively and qualitatively on two major tasks in our approach: (i)
opinion phrase classification and (ii) fine-grained opinion clustering. For
each task, we create extra human annotation and use them for performance
analysis. We will release the source code and test data together with human-
curated annotations.

Restaurant | # Training Sentences | # Training Phrases | # Test Reviews
---|---|---|---
10,000 | 297,210 | 643
Aspects | Sentiments
location, drinks, food, ambience, service | good, bad
Laptop | # Training Sentences | # Training Phrases | # Test Reviews
16,000 | 83,540 | 307
Aspects | Sentiments
| support, os, display, battery, company,
---
mouse, software, keyboard
good, bad

Table 1: Dataset Statistics.

Model | Restaurant-aspect | Restaurant-sentiment | Laptop-aspect | Laptop-sentiment
---|---|---|---|---
Acc. | Pre. | Rec. | macro-F1 | Acc. | Pre. | Rec. | macro-F1 | Acc. | Pre. | Rec. | macro-F1 | Acc. | Pre. | Rec. | macro-F1
CosSim | 64.20 | 54.55 | 47.82 | 49.85 | 70.19 | 68.73 | 63.57 | 63.87 | 55.39 | 59.21 | 58.72 | 54.33 | 70.03 | 69.88 | 70.50 | 70.84
W2VLDA* | 71.05 | 61.98 | 61.46 | 54.30 | 77.36 | 79.69 | 73.02 | 70.30 | 66.21 | 67.83 | 68.45 | 65.02 | 71.69 | 71.33 | 71.51 | 71.66
BERT | 75.98 | 62.30 | 80.26 | 60.83 | 79.50 | 77.72 | 77.99 | 77.85 | 69.12 | 68.13 | 68.74 | 67.38 | 71.24 | 70.92 | 71.08 | 71.17
JASen* | 84.99 | 74.09 | 77.50 | 74.85 | 83.64 | 83.45 | 80.76 | 81.74 | 73.02 | 70.31 | 70.67 | 72.19 | 76.47 | 76.67 | 76.46 | 76.53
FineSum w/o BERT | 81.18 | 71.31 | 61.53 | 64.52 | 67.19 | 75.66 | 56.55 | 51.96 | 67.34 | 68.42 | 68.71 | 67.43 | 68.12 | 67.94 | 69.47 | 68.33
FineSum w/o joint | 84.91 | 69.72 | 76.74 | 68.89 | 84.76 | 84.92 | 83.94 | 84.10 | 71.01 | 72.18 | 71.27 | 70.42 | 78.83 | 78.38 | 78.74 | 78.82
FineSum | 87.67 | 69.48 | 78.31 | 69.90 | 86.16 | 87.21 | 85.82 | 84.86 | 76.98 | 76.27 | 76.96 | 75.93 | 79.15 | 79.64 | 79.02 | 79.15
|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |

Table 2: Quantitative evaluation of aspect identification and sentiment
polarity on sentence level tasks. * denotes that the model learns aspect and
sentiment jointly.

Model | Restaurant-phrase
---|---
Acc. | Pre. | Rec. | macro-F1
CosSim | 55.97 | 56.22 | 53.64 | 53.84
W2VLDA* | 63.34 | 61.64 | 61.37 | 60.93
BERT | 72.39 | 63.54 | 74.13 | 67.55
JASen* | 82.20 | 85.32 | 79.70 | 79.64
FineSum w/o BERT | 63.80 | 62.76 | 60.19 | 58.21
FineSum w/o joint | 76.82 | 66.41 | 78.98 | 70.43
FineSum | 88.60 | 85.44 | 86.98 | 84.57
|  |  |  |

Table 3: Quantitative evaluation of restaurant aspect identification on phrase
level task.

###  5.1 Datasets and Experimental Settings

#### Datasets.

We experiment on reviews from two domains: restaurant and laptop. Details of
dataset statistics can be found in Table 1. We additionally provide four seed
keywords for each aspect and sentiment as weak supervision, as listed in the
Appendix. We also display the robustness of our methods on seed keywords in
Section 5.4.

Restaurant: We collect restaurant reviews from the Yelp Dataset
Challenge222https://www.kaggle.com/yelp-dataset/yelp-dataset as our in-domain
training and summarization corpus. We gather reviews from 42 businesses, where
each business has at least 100 reviews and includes the keyword “restaurant”
in its meta business type list. The average number of reviews for each
restaurant is 599. For evaluation, we use an external benchmark dataset in the
restaurant domain of SemEval-2016 (Pontiki et al. 2016), which provides
sentence-level aspect category and sentiment polarity of each review.
Following Huang et al. (2020), we remove sentences with multiple labels or
with a neutral sentiment polarity to simplify the problem.

Laptop: We collect reviews of 50 laptop products from the Laptop domain of
Amazon Review Dataset (Ni, Li, and McAuley 2019). Each product has at least
100 reviews. We also use the laptop domain of SemEval-2016 for evaluation and
remove sentences with multiple labels .

#### Experimental Settings.

For text prepossessing, we use NLTK tokenizers and the Stanford CoreNLP
(Manning et al. 2014)
parser333https://stanfordnlp.github.io/CoreNLP/parse.html,444https://stanfordnlp.github.io/CoreNLP/depparse.html.
We set the hyperparameters as follows: $h=5$, $K=2000$, $\theta_{1}=0.35$,
$\theta_{2}=0.30$, $T_{c}=7$, embedding dimension $=100$, learning rate
$=1e-5$, batch size = $64$. We train and fine-tune BERT with AdamW optimizer
for only one epoch to avoid overfitting on noisy pseudo labels. Details on
hyperparameter selection can be found in the Appendix.

###  5.2 Opinion Classification

We first evaluate the phrase classifier on sentence-level aspect
identification and sentiment polarization tasks using the benchmark test set.
However, a strong sentence-level classifier may not perform as well on phrase-
level task, so we additionally evaluate the model on phrase-level aspect
classification task. Due to the shortage of phrase-level annotation, we
manually collect data by randomly sampling 500 extracted phrases from the
Restaurant domain. We ask three human annotators to label their aspects and
split them by half into validation and test sets. As the phrase extraction is
aspect-agnostic, the collected set includes noisy phrases that do not belong
to any aspect. To handle this, we allow multiple and none aspect assignment.
Final labels were obtained using a majority vote among annotators. The three
annotators provided the same labels on 85.60% phrases, guaranteeing the high
quality of our annotation. More annotation and evaluation details can be found
in the Appendix.

We compare our approach with a series of weakly-supervised baselines and two
variants of our own approach.

  * •

CosSim: a Word2Vec embedding based model, which classifies according to the
cosine similarity between the averaged word embedding and topic vectors. Topic
vectors are calculated as the average of seed keywords.

  * •

W2VLDA (García-Pablos, Cuadros, and Rigau 2018): an aspect-based sentiment
analysis model, which leverages aspect and sentiment keywords as seeds to
perform joint topic modeling.

  * •

BERT (Devlin et al. 2019): a language model fine-tuning model. We incorporate
sentences containing seed keywords as pseudo training samples to fine-tune
BERT-base.

  * •

JASen (Huang et al. 2020): a state-of-the-art aspect based sentiment analysis
model. It first learns aspect-sentiment joint word embedding, then generalizes
word knowledge to neural models through weakly-supervised training and
iterative self-training.

  * •

FineSum w/o BERT: A context-free ablation of our model. It uses the opinion-
oriented spherical embedding as the only classifier.

  * •

FineSum w/o joint: A BERT-based ablation without fine-tuning on the jointly
agreed phrases (but trained on sentence-level classification).

Spherical Embedding | Vanilla BERT | BERT in FineSum
---|---|---
allergic reactions | severe allergies | severe allergies
severe allergies | broken leg | intestinalgastro issues
mild reactions | severe food poisoning | severe food poisoning
no allergic reactions | suffer severe migraine | crazy allergies
|  |

Table 4: Qualitative evaluation of fine-grained clustering on the restaurant
dataset. We compare opinion clusters from different embeddings that show
similar semantics. Conflicting and irrelevant phrases are denoted with red and
blue.

(a) Spherical Embedding

(b) Vanilla BERT

(c) Fine-tuned BERT in FineSum

Figure 4: Visualization of opinion clusters on Restaurant. Phrases assigned to
the same cluster are denoted with the same color.

The standard Accuracy, Precision, Recall and macro-F1 are used as evaluation
metrics. We run experiments for 5 times and report average performances. For
each method, we set a lowest threshold for their output classification
probability. We classify phrases with a probability larger than the threshold
into the corresponding aspect, otherwise the “none” aspect. The threshold is
selected when the model performs best on the validation set. Sentence- and
phrase-level evaluation results are shown in Table 2 and Table 3,
respectively. We have the following observations:

Sentence Classification. (1) FineSum outperforms all the baselines on
sentiment polarization, and achieves competitive performances on aspect
identification. Benefiting from both word-level and contextualized knowledge,
it achieves better performances than models only based on topic modeling
(W2VLDA), language model (BERT), or word embedding (CosSim, JASen). (2)
FineSum outperforms its two variants by a large margin. This is intuitive
since FineSum w/o BERT works in a context-free manner, and FineSum w/o joint
is only trained on sentence classification task. It may not perform as well on
phrases where less contexts are presented. (3) On restaurant aspect
identification, FineSum shows relatively lower macro-F1 and precision than
JASen. This happens because JASen is designed for aspect-based sentiment
analysis and takes advantage of the mutual information in aspects and
sentiments. However, FineSum views them as independent for simplicity. In
fact, we can easily extend FineSum to the joint setting by modifying the word
embedding approach (e.g., substitute the spherical embedding with the
embedding in JASen), which may lead to even better performance.

Phrase Classification. (1) Most baselines perform worse on phrases than on
sentences. This is because fewer informative contexts are provided in phrases,
thus calling for the need of phrase-level training. (2) Despite fewer
contexts, FineSum gains a larger edge over other baselines. This observation
validates the effectiveness of fine-tuning BERT on jointly agreed phrases. We
also qualitatively compare FineSum with its two variants in the Appendix.

Domain: Restaurant Aspect: Food | Good |  |  delicious dark chocolate dipping sauce; it was a good ratio of chocolate to vanilla; yummy chocolate
---
brownie; just ok chocolate berry dessert; amazing chocolate pudding dessert
| this is a fantastic change while keeping the integrity of so many familiar
ingredients; all of the ingredients
---
went really well together; appreciate how all the ingredients come together;
everything was well seasoned
Bad |  | left the rest of the fish untouched; the salmon was hastily created; the fish had a weird texture; the fish
---
seemed a little oily; the fish was n’t that large; i hate raw fish; awful raw
fish
| the dish disappointed my companion; the food was absolutely underwhelming;
they were really disappointed
---
with the dish; my fault for ordering their food, not theirs for making it; I
wasn’t as impressed with the food
Domain: Laptop Aspect: Mouse | Good |  | i love the trackman wheel; the trackman wheel is practically perfect just as it is; the trackman wheel
---
optical is the best of those; more compact trackman wheel; the trackman wheel
feels good in the hand
| the two little buttons are useful if you install the logitech software; love
the fact that i could program
---
the buttons for what i needed; the buttons can be programed to perform various
functions
Bad |  | sometimes the mouse becomes unresponsive and sluggish; the movement of the cursor with the trackball
---
gets slow and erratic at times; possible nobody think the mouse is too slow
like me?
| experienced wrist/arm pain from mouses; my mouse arm tends to be constantly
sore; using a mouse 8
---
hours i began to get pains in my wrist; i got sick of my optical mouse messing
up on me
|

Table 5: A case study of the final output from FineSum on both datasets. For
simplicity, we display one aspect from each domain and randomly select two
clusters under each aspect and sentiment. For a quick understanding, we
manually italicize and bold fine-grained sub-aspects, and highlight sentiment-
indicating words with red (good) and blue (bad).

###  5.3 Fine-Grained Opinion Clustering

We compare our fine-tuned BERT embedding with two ablations on clustering
performance: (i) Opinion-oriented Spherical Embedding introduced in Section
4.2 and (ii) Last layer outputs from Vanilla BERT, which are not trained on
the aspect classification task.

Qualitative Evaluation. To intuitively understand the difference between
embedding methods, we display semantically similar clusters from them in Table
4. We observe that Spherical Embedding relies excessively on overlapped
surface words. It tends to gather look-alike phrases even if their meanings
are converse. For instance, “severe allergies” and “no allergic reactions” are
in the same cluster. Besides, we also find that Vanilla BERT, although less
reliant on overlapped words, sometimes suffers from semantic drifts. For
example, “broken leg” and “severe migraine” are grouped together. On the
contrary, BERT in FineSum forms clusters that are both coherent in meaning and
diverse in expression. We further visualize phrase distribution in Fig. 4 to
intuitively understand how clusters distribute in the fine-tuned BERT
embedding space. Compared with Spherical Embedding and Vanilla BERT, BERT in
FineSum displays a clearer and better-separated cluster space.

Quantitative Evaluation. We evaluate the coherence and diversity of generated
clusters quantitatively. Coherence measures the semantic consistency of
phrases within the same cluster, whereas diversity measures how their
expressions differ from each other. In principle, a cluster of high quality
should be in both high coherence and diversity, indicating that the model can
gather phrases with similar semantic meanings regardless of their expression
forms.

Figure 5: Quantitative evaluation of fine-grained clustering on the restaurant
dataset. Higher scores indicates better performances.

Figure 6: Influence of seed keywords.

We define the two metrics as follows: (i) Coherence: Given an opinion cluster,
we inject an intrusion phrase that is randomly chosen from another cluster.
Then three human annotators are asked to identify the intruded phrase.
Empirically, we observe that phrases within the same cluster usually share
common words, making it easy to identify the intruded phrase. Hence, we
require the intruded phrase to have at least one overlapped word with other
phrases. We provide an example of this in the Appendix, along with other
evaluation details. We compute the ratio of correctly identified intrusion
instances as the coherence score. (ii) Diversity: The percentage of unique
words in each cluster. Fig. 5 showcases that BERT in FineSum significantly
outperforms the other two ablations on coherence, validating that the fine-
tuned BERT embeddings generate semantically coherent clusters. This finding
indicates that fine-tuned BERT for aspect classification not only benefits the
task itself, but also leads the model to better distinguish fine-grained sub-
aspects within each aspect. Moreover, BERT in FineSum achieves slightly higher
diversity score than the other two methods, indicating that the high coherence
score of our approach is not brought by simply gathering phrases with similar
words.

###  5.4 Parameter Study

To investigate whether FineSum is sensitive to different choices of seed
keywords, we initiate the opinion-oriented embedding with different numbers of
seeds. Figure 6 shows sentence-level classification results on the Restaurant
dataset. As can be observed from the figure, the classification accuracy and
macro-F1 remain relatively stable when we alter the number of seeds. This
result indicates that the opinion-oriented embedding can learn well-separated
semantic space with little human guidance, which validates the robustness of
FineSum and opens up possibilities to apply it to diverse domains in the
future.

###  5.5 Case Study

Table 5 shows an example of our system output. We observe that different
opinion phrase clusters are well-separated by their aspects and sentiments,
which is guaranteed by the opinion phrase classification stage. Probing into
each aspect and sentiment, we discover that the model automatically forms
clusters which represent concrete sub-aspects or describes particular traits.
For example, under the aspect-sentiment pair $\langle$food, bad$\rangle$, we
find one cluster expressing overall disappointment for their food and two
clusters complaining about specific food types. The coherent and meaningful
clusters under each aspect validate the effectiveness of clustering with fine-
tuned BERT embedding.

##  6 Conclusion

In this paper we propose FineSum, a minimally supervised approach for fine-
grained opinion summarization. FineSum works by first extracting candidate
phrases, then classifying them into aspects and sentiments using the opinion-
oriented spherical embedding and the weakly-supervised BERT. We further
propose to aggregate similar phrases using the fine-tuned BERT embedding to
obtain fine-grained opinion clusters. Comprehensive automatic and human
evaluation demonstrate that our approach generates high-quality phrase-level
summarization.

## References

  * Amplayo and Lapata (2021) Amplayo, R. K.; and Lapata, M. 2021.  Informative and Controllable Opinion Summarization.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , 2662–2672.
  * Angelidis et al. (2021) Angelidis, S.; Amplayo, R. K.; Suhara, Y.; Wang, X.; and Lapata, M. 2021.  Extractive Opinion Summarization in Quantized Transformer Spaces.  _Transactions of the Association for Computational Linguistics_ , 9: 277–293.
  * Angelidis and Lapata (2018) Angelidis, S.; and Lapata, M. 2018.  Summarizing Opinions: Aspect Extraction Meets Sentiment Prediction and They Are Both Weakly Supervised.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , 3675–3686.
  * Chu and Liu (2019) Chu, E.; and Liu, P. 2019.  MeanSum: a neural model for unsupervised multi-document abstractive summarization.  In _International Conference on Machine Learning_ , 1223–1232. PMLR.
  * Devlin et al. (2019) Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , 4171–4186.
  * Di Fabbrizio, Stent, and Gaizauskas (2014) Di Fabbrizio, G.; Stent, A.; and Gaizauskas, R. 2014.  A hybrid approach to multi-document summarization of opinions in reviews.  In _Proceedings of the 8th International Natural Language Generation Conference (INLG)_ , 54–63.
  * Dubitzky et al. (2013) Dubitzky, W.; Wolkenhauer, O.; Yokota, H.; and Cho, K.-H. 2013.  _Encyclopedia of systems biology_.  Springer Publishing Company, Incorporated.
  * Elsahar et al. (2021) Elsahar, H.; Coavoux, M.; Rozen, J.; and Gallé, M. 2021.  Self-Supervised and Controlled Multi-Document Opinion Summarization.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , 1646–1662.
  * Frermann and Klementiev (2019) Frermann, L.; and Klementiev, A. 2019.  Inducing Document Structure for Aspect-based Summarization.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , 6263–6273. Florence, Italy: Association for Computational Linguistics.
  * Ganesan, Zhai, and Han (2010) Ganesan, K.; Zhai, C.; and Han, J. 2010.  Opinosis: A Graph Based Approach to Abstractive Summarization of Highly Redundant Opinions.  In _Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010)_.
  * García-Pablos, Cuadros, and Rigau (2018) García-Pablos, A.; Cuadros, M.; and Rigau, G. 2018.  W2VLDA: almost unsupervised system for aspect based sentiment analysis.  _Expert Systems with Applications_ , 91: 127–137.
  * Holtzman et al. (2019) Holtzman, A.; Buys, J.; Du, L.; Forbes, M.; and Choi, Y. 2019.  The Curious Case of Neural Text Degeneration.  In _International Conference on Learning Representations_.
  * Huang et al. (2020) Huang, J.; Meng, Y.; Guo, F.; Ji, H.; and Han, J. 2020.  Aspect-Based Sentiment Analysis by Aspect-Sentiment Joint Embedding.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , 6989–6999.
  * Krishna and Srinivasan (2018) Krishna, K.; and Srinivasan, B. V. 2018.  Generating topic-oriented summaries using neural attention.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , 1697–1705.
  * Ku et al. (2006) Ku, L.-W.; Liang, Y.-T.; Chen, H.-H.; et al. 2006.  Opinion extraction, summarization and tracking in news and blog corpora.  In _AAAI spring symposium: Computational approaches to analyzing weblogs_ , volume 100107, 1–167.
  * Manning et al. (2014) Manning, C. D.; Surdeanu, M.; Bauer, J.; Finkel, J. R.; Bethard, S.; and McClosky, D. 2014.  The Stanford CoreNLP natural language processing toolkit.  In _Proceedings of 52nd annual meeting of the association for computational linguistics: system demonstrations_ , 55–60.
  * Maynez et al. (2020) Maynez, J.; Narayan, S.; Bohnet, B.; and McDonald, R. 2020.  On Faithfulness and Factuality in Abstractive Summarization.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , 1906–1919.
  * Meng et al. (2020) Meng, Y.; Zhang, Y.; Huang, J.; Zhang, Y.; Zhang, C.; and Han, J. 2020.  Hierarchical topic mining via joint spherical tree and text embedding.  In _Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining_, 1908–1917.
  * Ni, Li, and McAuley (2019) Ni, J.; Li, J.; and McAuley, J. 2019.  Justifying recommendations using distantly-labeled reviews and fine-grained aspects.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , 188–197.
  * Paul, Zhai, and Girju (2010) Paul, M.; Zhai, C.; and Girju, R. 2010.  Summarizing contrastive viewpoints in opinionated text.  In _Proceedings of the 2010 conference on empirical methods in natural language processing_ , 66–76.
  * Pontiki et al. (2016) Pontiki, M.; Galanis, D.; Papageorgiou, H.; Androutsopoulos, I.; Manandhar, S.; AL-Smadi, M.; Al-Ayyoub, M.; Zhao, Y.; Qin, B.; De Clercq, O.; Hoste, V.; Apidianaki, M.; Tannier, X.; Loukachevitch, N.; Kotelnikov, E.; Bel, N.; Jiménez-Zafra, S. M.; and Eryiğit, G. 2016.  SemEval-2016 Task 5: Aspect Based Sentiment Analysis.  In _Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016)_ , 19–30. San Diego, California: Association for Computational Linguistics.
  * Suhara et al. (2020) Suhara, Y.; Wang, X.; Angelidis, S.; and Tan, W.-C. 2020.  OpinionDigest: A Simple Framework for Opinion Summarization.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , 5789–5798.