# A Variational Approach to Unsupervised Sentiment Analysis

Ziqian Zeng<EMAIL_ADDRESS>
Department of CSE, Hong Kong University of Science and Technology
Wenxuan Zhou<EMAIL_ADDRESS>
Department of CS, University of Southern California, CA, USA
Xin Liu<EMAIL_ADDRESS>
Zizheng Lin<EMAIL_ADDRESS>
Yangqiu Song<EMAIL_ADDRESS>
Department of CSE, Hong Kong University of Science and Technology
Michael David Kuo<EMAIL_ADDRESS>
Wan Hang Keith Chiu<EMAIL_ADDRESS>
Department of Diagnostic Radiology, University of Hong Kong

###### Abstract

In this paper, we propose a variational approach to unsupervised sentiment
analysis. Instead of using ground truth provided by domain experts, we use
target-opinion word pairs as a supervision signal. For example, in a document
snippet “the room is big,” (room, big) is a target-opinion word pair. These
word pairs can be extracted by using dependency parsers and simple rules. Our
objective function is to predict an opinion word given a target word while our
ultimate goal is to learn a sentiment classifier. By introducing a latent
variable, i.e., the sentiment polarity, to the objective function, we can
inject the sentiment classifier to the objective function via the evidence
lower bound. We can learn a sentiment classifier by optimizing the lower
bound. We also impose sophisticated constraints on opinion words as
regularization which encourages that if two documents have similar
(dissimilar) opinion words, the sentiment classifiers should produce similar
(different) probability distribution. We apply our method to sentiment
analysis on customer reviews and clinical narratives. The experiment results
show our method can outperform unsupervised baselines in sentiment analysis
task on both domains, and our method obtains comparable results to the
supervised method with hundreds of labels per aspect in customer reviews
domain, and obtains comparable results to supervised methods in clinical
narratives domain.

##  1 Introduction

Sentiment analysis is a task of identifying the sentiment polarity expressed
in textual material. Sentiment Analysis is an important and useful application
in natural language processing (?). For example, sentiment analysis on
customer reviews can help business owners to improve their products and help
other customers to purchase products wisely. Sentiment analysis on clinical
narratives which contains personal observations and attitudes from nurses,
radiologists, or therapists, can provide automated decision support for
physicians. Most existing sentiment analysis methods are supervised methods
which requires labeled training data. In practice, naturally labeled data is
scarce and annotating data could be expensive. Although crowd-sourcing is a
cost-saving way to annotate data, some tasks still require domain experts such
as doctors, lawyers, or financial consultants. Hence, sentiment analysis in
the unsupervised setting is a more realistic scenario than supervised one.

We aim to tackle the challenge of unsupervised sentiment analysis. A
traditional way to perform unsupervised sentiment analysis is lexicon-based
methods (?, ?). A lexicon consists of a list of opinion words and
corresponding sentiment orientation (SO) scores capturing polarity (positive
or negative) and strength (degree to which the opinion word is positive or
negative). Then, for any given text, all opinion words are extracted and
annotated with their SO scores using a lexicon. The SO scores are aggregated
into a single score for the text. There are two disadvantages of lexicon-based
methods. First, unsupervised lexicon-based methods do not involve any training
process. Second, the performance of these methods is highly dependent on the
quality of pre-defined lexicons. Most of lexicons suffer from either high
coverage/low precision or low coverage/high precision problem (?).

We propose an unsupervised method to train a sentiment classifier without
relying on the quality of the lexicon. The supervision signal of our method is
pairs of words. We observe that normally there is a syntactic dependency
between a opinion word and a target word. A target word is usually related to
a subjective. An opinion word is related to a sentiment polarity. For example,
given a document snippet “the hotel is comfortable,” in sentiment
classification task which aims to classify the sentiment polarity of a hotel,
“comfortable” is an opinion word and “hotel” is a target word. The objective
function of our algorithm is to maximize the likelihood of an opinion word
given a target word. By introducing a latent variable, i.e., the sentiment
polarity of a document, to the objective function, we can inject a sentiment
classifier to the objective function via the evidence lower bound (ELBO).
There are two classifiers in the ELBO, i.e, a sentiment classifier and an
opinion word classifier. The input of the sentiment classifier is a document
representation, and the inputs of opinion word classifier are a possible value
of sentiment polarity and a target word. The sentiment classifier produces a
probability distribution, which is used to approximate the true posterior
distribution in ELBO. Then, we can learn a sentiment classifier (an
approximated posterior distribution) by optimizing the lower bound. Further to
make use of opinion words, we also apply a regularization term which
encourages that if two documents have similar (dissimilar) opinion words, the
sentiment classifiers should produce similar (different) probability
distribution. We apply our algorithm on two domains, i.e., sentiment analysis
on customer reviews and radiology reports. In customer reviews domain, our
task is document-level multi-aspect sentiment classification (DMSC). The goal
of DMSC is to predict the sentiment polarity (e.g., positive or negative) of
each aspect given a document in which there are several sentences and each
sentence describes one or more aspects. The objective function is to predict
an opinion word such as “good” given a target word such as “price.” In
clinical domain, our task is hip fracture classification. The goal of hip
fracture classification is to predict the heath status (e.g., fracture or non-
fracture) given a textual radiology report which describes the health status
of the hip region. The objective function predict an opinion word such
“fractured” given a target word such as “trochanter”.

Our framework has two advantages. First, in realistic scenario, we could
manually define target words to perform sentiment analysis with different
granularity. For example, if we want to do a coarse-grained sentiment
classification on hotel reviews, target words would be {price, view, room,
location, …}. While if a fine-grained classification is needed, e.g., the
Internet quality, then target words would be {Internet, WiFi, access,
connection, …}. The supervision signal is very flexible in realistic scenario
where the sentiment classification tasks are in different granularity. Second,
under our framework, the sentiment classifier would be any neural network
architectures which have demonstrated high capacity in supervised learning
setting. The upper bound of the performance of our sentiment classifier is
that of the same model in supervised settings. Our framework will benefit from
powerful neural networks.

Our contribution can be summarized as follows:

$\bullet$ We propose an unsupervised approach to solve sentiment analysis.

$\bullet$ We propose to learn a sentiment classifier by injecting it into
another relevant objective via the evidence lower bound. This framework is
flexible to adopt different neural network architectures as sentiment
classifiers and perform classification tasks with different granularity.

$\bullet$ The experiment results show our method can outperform unsupervised
baselines in both tasks and obtains comparable results to the supervised
method with hundreds of labels per aspect in DMSC task, and comparable results
to supervised methods in hip fracture classification task.

A preliminary version of this work (?) appeared in the proceedings of NAACL
2019. This journal version has made several major improvements. First, our
method is generalized to sentiment classification task. Previous work (?)
focuses on document-level multi-aspect sentiment classification task. Second,
we add more details of the methodology section. We discuss different forms of
training objective when applying different assumptions. We also introduce
approximation techniques to deal with the situations that the opinion
vocabulary is large or the number of categories is large. Third, we add a
regularization term to encourage that if two documents have similar
(dissimilar) opinion words, the sentiment classifiers should produce similar
(different) probability distribution. Last, we add more experiments on
clinical sentiment analysis and conduct a case study.

The remainder of the paper is organized as follows. We first introduce our
method including sentiment classifier, opinion word classifier, training
objective and regularization term in Section 2. Then we shows our experiments
including target opinion word pars extraction, compared methods, result and
error analysis in Section 3. Finally, we introduce the related work in Section
4 followed by the conclusion of this paper in Section 5.

##  2 Methodology

In this section, we introduce our variational approach to unsupervised
sentiment analysis. We will introduce sentiment classifier, opinion word
classifier, training objective, and regularization of opinion word
respectively in the following subsections.

Figure 1: Our model consists of a sentiment classifier and an opinion word
classifier.

###  2.1 Overview

Our model consists of a sentiment classifier and an opinion word classifier.
Our goal is to learn a sentiment classifier to predict the sentiment polarity
of a document. It could be the sentiment polarity of an aspect given a user
review or the health status of the hip given textual radiology report. The
input of the sentiment classifier is the text representation of a user review
or a radiology report.

Figure 1 shows the relation between two classifiers. The input $\mathbf{x}$ of
the sentiment classifier is the text representation of a user review or a
radiology report, e.g., bag-of-words or a representation learned by neural
networks. Let $C$ be a random variable, indicating the sentiment polarity of a
document, e.g., the sentiment polarity of an aspect or the health status of
the hip. The sentiment classifier takes $\mathbf{x}$ as input and produces a
probability distribution, denoted as $q(C|\mathbf{x})$. For example, if $C$
has two possible values, i.e., positive and negative, then outputs of the
classifier are $q(positive|\mathbf{x})$ and $q(negative|\mathbf{x})$
respectively. The opinion word classifier of DMSC task takes a target word
(“price”) and a possible value of sentiment polarity $c$ as input, and
estimates $p(\text{``good''}|c,\text{``price''})$. The opinion word classifier
of hip fracture classification task takes a target word (“trochanter”) and a
possible value of health status $c$ as input, and estimates
$p(\text{``fractured''}|c,\text{``trochanter''})$. Our training objective
function is to maximize the log-likelihood of an opinion word given a target
word, e.g., $p(\text{``good''}|\text{``price''})$ or
$p(\text{``fractured''}|\text{``trochanter''})$. The ELBO of log-likelihood
consists of a sentiment classifier and an opinion word classifier.

###  2.2 Sentiment Classifier

The sentiment classifier aims to estimate a distribution $q(C|\mathbf{x})$,
where $C$ is a discrete random variable representing the sentiment polarity of
a document, e.g., the sentiment polarity or the health status of the hip, and
$\mathbf{x}$ is a feature representation of a document. Let $c$ denote a
possible value of the random variable $C$, representing a possible value of
sentiment polarity, e.g., positive, negative, fractured or non-fractured. The
sentiment classifier estimates the probability as

$q(C=c|\mathbf{x})=\frac{\exp\big{(}\mathbf{w}_{c}^{T}\mathbf{x}\big{)}}{\sum_{c^{\prime}}{\exp\big{(}\mathbf{w}_{c^{\prime}}^{T}\mathbf{x}\big{)}}}\;,$
(1)

where $\mathbf{w}_{c}$ is a vector associated with a sentiment polarity $c$.

The representation of a document $\mathbf{x}$ can be various. Traditional
document representations of text classification would be bag-of-words, n-gram,
or averaged word embeddings. Recently, end-to-end neural network based models
demonstrate a powerful capacity to extract features of a document (?). Our
model will benefit growingly powerful neural network based feature extraction
methods. We use bag-of-words, convolutional neural networks, or long short-
term memory networks as the document representation methods in our
experiments.

###  2.3 Opinion Word Classifier

The opinion word classifier aims to estimate the probability of an opinion
word $w_{o}$ given a target word $w_{t}$ and a possible value of sentiment
polarity $c$:

$p(w_{o}|c,w_{t})=\frac{\exp\big{(}\varphi({w_{o}},{w_{t}},c)\big{)}}{\sum_{{w_{o}^{\prime}}}{\exp\big{(}\varphi({w_{o}^{\prime}},{w_{t}},c)\big{)}}}\;,$
(2)

where $\varphi(\cdot)$ is a scoring function which takes opinion word $w_{o}$,
target word $w_{t}$, and a possible value of sentiment polarity $c$ as inputs.
The nature of the score function is about the frequency of occurrence. If an
opinion word, a target word, and a possible value of sentiment polarity co-
occur frequently, the score will be high, otherwise, it will be low. The
scoring function could be various, and here we use the simplest one (dot
product) as the scoring function:

$\varphi({w_{o}},{w_{t}},c)=I(w_{t}\in\mathcal{K})\cdot\mathbf{a}_{c}^{T}\mathbf{w}_{o}\;,$
(3)

where $\mathbf{w}_{o}$ is the word embedding of opinion word $w_{o}$,
$\mathbf{a}_{c}$ is a vector associated with $c$, $\mathcal{K}$ is the set of
relevant target words, and $I(\cdot)$ is an indicator function where
$I(true)=1$ and $I(false)=0$. The scoring function could be various, e.g.,
multilayer perceptron (MLP). Here we only introduce the simplest case.

Given a target word $w_{t}$ and a possible value of sentiment polarity $c$, we
aim to maximize the probability of opinion words highly related to them. For
example, the opinion word “good” is usually related to the target word “price”
for the aspect value with sentiment polarity positive, and the opinion word
“terrible” is usually related to the target word “traffic” for the aspect
location with sentiment polarity negative. The opinion word “fractured” is
usually related to the target word “trochanter” with the health status of hip
positive, and the opinion word “intact” is usually related to target word
“trochanter” with the health status of hip negative.

###  2.4 Training Objective

The objective function is to maximize the log-likelihood of an opinion word
$w_{o}$ given a target word $w_{t}$. After introducing a latent variable
(i.e., the sentiment polarity of a document) to the objective function, we can
derive an evidence lower bound (ELBO) of the log-likelihood which can
incorporate two classifiers. The first one corresponds to the sentiment
classifier. The second one corresponds to the opinion word classifier. The
ELBO of log-likelihood is shown as follows,

$\displaystyle{\mathcal{L}}$ $\displaystyle=\sum_{x\in
X}\sum_{(w_{o},w_{t})\in{\mathcal{P}}_{x}}\log p(w_{o}|w_{t})$
$\displaystyle=\sum_{x\in
X}\sum_{(w_{o},w_{t})\in{\mathcal{P}}_{x}}\log\sum_{c}p(w_{o},c|w_{t})$
$\displaystyle=\sum_{x\in
X}\sum_{(w_{o},w_{t})\in{\mathcal{P}}_{x}}\log\sum_{c}q(c|\mathbf{x})\Big{[}\frac{p(w_{o},c|w_{t})}{q(c|\mathbf{x})}\Big{]}$
$\displaystyle\geq\sum_{x\in
X}\sum_{(w_{o},w_{t})\in{\mathcal{P}}_{x}}\sum_{c}q(c|\mathbf{x})\Big{[}\log\frac{p(w_{o},c|w_{t})}{q(c|\mathbf{x})}\Big{]}$
$\displaystyle=\sum_{x\in
X}\sum_{(w_{o},w_{t})\in{\mathcal{P}}_{x}}\operatorname{\mathbb{E}}_{q(C|\mathbf{x})}\big{[}\log
p(w_{o}|c,w_{t})p(c|w_{t})\big{]}+H(q(C|\mathbf{x})),$ (4)

where $X$ is the training set containing all documents, and
${\mathcal{P}}_{x}$ is the set of all word pairs extracted from a document
$x$, $H(\cdot)$ refers to the Shannon entropy, and $q(c|\mathbf{x})$ is short
for $q(C=c|\mathbf{x})$. By applying Jensen’s inequality, the log-likelihood
is lower-bounded by Eq. (4).

The equality holds if and only if the KL-divergence of two distributions,
$q(C|\mathbf{x})$ and $p(C|w_{t},w_{o})$, equals to zero. Maximizing the
evidence lower bound is equivalent to minimizing the KL-divergence. The proof
is shown as follows. To be concise, the proof only consider one training
sample.

$\displaystyle\log p(w_{o}|w_{t})$ $\displaystyle=\sum_{c}q(c|\mathbf{x})\log
p(w_{o}|w_{t})$
$\displaystyle=\sum_{c}q(c|\mathbf{x})\log\frac{p(w_{o}|w_{t})q(c|\mathbf{x})p(w_{o},c|w_{t})}{p(w_{o},c|w_{t})q(c|\mathbf{x})}$
$\displaystyle=\sum_{c}q(c|\mathbf{x})\log\frac{p(w_{o},c|w_{t})}{q(c|\mathbf{x})}+\sum_{c}q(c|\mathbf{x})\log\frac{p(w_{o}|w_{t})q(c|\mathbf{x})}{p(w_{o},c|w_{t})}$
$\displaystyle=LB+\sum_{c}q(c|\mathbf{x})\log\frac{q(c|\mathbf{x})}{p(c|w_{t},w_{o})}$
$\displaystyle=LB+KL\big{(}q(C|\mathbf{x})||p(C|w_{t},w_{o})\big{)},$ (5)

where $LB$ is short for Eq. (4) with one training sample. Hence, we can learn
a sentiment classifier that can produce a similar distribution to the true
posterior $p(C|w_{t},w_{o})$. Compared with $p(C|w_{t},w_{o})$,
$q(C|\mathbf{x})$ is more flexible since it can take any kind of feature
representations of whole document as input. It is possible for
$q(C|\mathbf{x})$ to approximate the true posterior distribution, because a
document includes the target word and the opinion word.

We assume that a target word $w_{t}$ and a possible value of sentiment
polarity $c$ are independent, i.e., $p(c|w_{t})$ = $p(c)$, since the polarity
assignment is not influenced by the target word. For example, only given a
target word “price”, the probability of positive and negative are equal. By
this assumption, the ELBO becomes,

$\displaystyle{\mathcal{L}}_{1}$ $\displaystyle=\sum_{x\in
X}\sum_{(w_{o},w_{t})\in{\mathcal{P}}_{x}}\operatorname{\mathbb{E}}_{q(C|\mathbf{x})}\big{[}\log
p(w_{o}|c,w_{t})p(c)\big{]}+H(q(C|\mathbf{x})),$ (6)

For piror distribution of sentiment polarity, i.e., $p(C)$, we could use a
neural network to parameterize it. Or we could assume $C$ follows a uniform
distribution, which means $p(c)$ is a constant. If we assume $C$ follows a
uniform distribution, we would remove $p(c)$ in Eq. (4) to get a new objective
function as follows,

$\displaystyle{\mathcal{L}}_{2}$ $\displaystyle=\sum_{x\in
X}\sum_{(w_{o},w_{t})\in{\mathcal{P}}_{x}}\operatorname{\mathbb{E}}_{q(C|\mathbf{x})}\left[\log
p(w_{o}|c,w_{t})\right]+H(q(C|\mathbf{x}))\;.$ (7)

###  2.5 Approximation

The partition function in Eq. (2) requires the summation over all opinion
words in the vocabulary. If the size of the opinion word vocabulary is large,
we could use the negative sampling technique (?) to approximate Eq. (2).
Specifically, we approximate $p(w_{o}|c,w_{t})$ in the objective (2) with the
following objective function:

$\log\sigma\big{(}\varphi({w_{o}},{w_{t}},c)\big{)}+\sum_{w_{o}^{\prime}\in{\mathcal{N}}}\log\big{(}1-\sigma\big{(}\varphi({w_{o}^{\prime}},{w_{t}},c)\big{)}\big{)}\;,$
(8)

where $w_{o}^{\prime}$ is a negative sample in opinion words vocabulary,
${\mathcal{N}}$ is the set of negative samples and $\sigma(\cdot)$ is the
sigmoid function. If the approximation is used, we should multiply a hyper-
parameter $\alpha$ to the entropy term to ensure that the approximation part
and entropy term into are on the same scale (?). The objective function
becomes,

$\displaystyle{\mathcal{L}}_{3}$ $\displaystyle=\sum_{x\in
X}\sum_{(w_{o},w_{t})\in{\mathcal{P}}_{x}}\operatorname{\mathbb{E}}_{q(C|\mathbf{x})}\big{[}\log\sigma\big{(}\varphi({w_{o}},{w_{t}},c)\big{)}+\sum_{w_{o}^{\prime}\in{\mathcal{N}}}\log\big{(}1-\sigma\big{(}\varphi({w_{o}^{\prime}},{w_{t}},c)\big{)}\big{)}+\log
p(c|w_{t})\big{]}$ $\displaystyle+\sum_{x\in
X}\sum_{(w_{o},w_{t})\in{\mathcal{P}}_{x}}\alpha H(q(C|\mathbf{x}))\;.$ (9)

The expectation operation in Eq. (4) requires the summation over all possible
sentiment polarities. If the number of polarities is large, we could use
likelihood ratio gradient estimators (?) to approximate the expectation part.
Using the likelihood ratio estimation, the gradient of ${\mathcal{L}}$ with
respect to parameters $\theta$ in $q(C|\mathbf{x})$ can be computed as
follows. To be concise, we only consider one training sample.

$\displaystyle\nabla_{\theta}{\mathcal{L}}=\operatorname{\mathbb{E}}_{q(C|\mathbf{x};\theta)}[A(w_{o},w_{t},c)\nabla\log
q(c|\mathbf{x})],$ (10)

where $A(w_{o},w_{t},c)=\log p(w_{o}|c,w_{t})+\log p(c|w_{t})-\log
q(c|\mathbf{x})$ can be treated as the learning signal in policy gradient. The
gradient in Eq. (10) can be approximated by the Monte Carlo method using $K$
samples of the latent variable:

$\displaystyle\nabla_{\theta}{\mathcal{L}}\approx\frac{1}{K}\sum_{j=1}^{K}A(w_{o},w_{t},j)\nabla\log
q(C=j|\mathbf{x}).$ (11)

###  2.6 Regularization on Opinion Words

The motivation of regularization is that if the opinion words extracted from
two documents are similar semantically, then these two documents probably are
in the same cluster, and if the opinion words are opposite semantically, then
these two documents are probably not in the same cluster. For example, in the
sentiment classification task, if one document contains a word pair (staff,
friendly) and the other one contains a word pair (waiter, unfriendly), it is
highly possible that these two documents do not belong to the same cluster
because “friendly” and “unfriendly” are opposite semantically. In hip fracture
classification, if one document contains a word pair (trochanter, displaced)
and the other one contains a word pair (hip, fractured), it is highly possible
that these two documents belong to the same cluster because “displaced” and
“fractured” are similar semantically.

First, we will measure the semantic similarity between two opinion words by
introducing a score function. The score function takes the cosine similarity
between the embeddings of two words $w^{i}_{d}$ and $w^{j}_{d}$ as inputs,

$\displaystyle
s(w^{i}_{d},w^{j}_{d})=\tau\big{(}\cos(\mathbf{w}^{i}_{d},\mathbf{w}^{j}_{d});\gamma\big{)},$
(12)

where $w^{i}_{d}$ and $w^{j}_{d}$ are the opinion word in document $x^{i}$ and
$x^{j}$ respectively, $\mathbf{w}^{i}_{d}$ and $\mathbf{w}^{j}_{d}$ are
corresponding embeddings, ${\gamma}\in[0,1]$ is a threshold, and
$\tau(z;\gamma)=z$ if $|z|\geq{\gamma}$ otherwise $\tau(z;\gamma)=0$. There
are two cases that the score function is zero. The first one is that the
cosine value is positive and is less than a threshold $\gamma$. The second one
is that the cosine value is negative and is greater than a threshold
$-\gamma$. The first case means the two opinion words are not similar enough.
The second case means that two opinion words are not dissimilar enough. In
both cases, the regularization will not be applied.

Then we maximize the following objective function:

$\displaystyle R(x^{i},x^{j})$
$\displaystyle=-d\big{(}q(C|\mathbf{x}^{i}),q(C|\mathbf{x}^{j})\big{)}s(w^{i}_{d},w^{j}_{d}),$
(13)

where $d\left(\cdot,\cdot\right)$ is squared Euclidean distance, i.e.,
$d\big{(}q(C|\mathbf{x}^{i}),q(C|\mathbf{x}^{j})\big{)}=\sum_{c}\big{(}q(c|\mathbf{x}^{i})-q(c|\mathbf{x}^{j})\big{)}^{2}$
, and $\mathbf{x}^{i}$ and $\mathbf{x}^{j}$ are documents representation of
$x^{i}$ and $x^{j}$ respectively. If the posterior distributions
$q(C|\mathbf{x}^{i})$ and $q(C|\mathbf{x}^{j})$ are different from each other,
i.e., $d\big{(}q(C|\mathbf{x}^{i}),q(C|\mathbf{x}^{j})\big{)}$ is large, but
opinion words suggest that these two documents should be in the same cluster
i.e., $s(w^{i}_{d},w^{j}_{d})$ is large, then
$d\big{(}q(C|\mathbf{x}^{i}),q(C|\mathbf{x}^{j})\big{)}$ will be encouraged to
be small by applying the regularization. If applying the regularization, the
objective function becomes,

$\displaystyle{\mathcal{J}}$ $\displaystyle={\mathcal{L}}+\beta\sum_{x^{i}\in
X}\sum_{x^{j}\in X}R(x^{i},x^{j})\;,$ (14)

where $x^{i}$ and $x^{j}$ are documents in the training set $X$. The
constraints are defined in a $|X|\times|X|$ space. In practice, we train our
model batch by batch. So we only apply the constraints within a mini-batch.
There are at most $|X_{b}|\times|X_{b}|$ constraints in a mini-batch, where
$|X_{b}|$ is the number of samples in a mini-batch.

##  3 Experiment

In this section, we report average sentiment classification accuracies over
all aspects on binary document level multi-aspect sentiment classification
task and the accuracy on binary hip fracture classification task.

###  3.1 Datasets

####  3.1.1 Document Level Multi-aspect Sentiment Analysis

We evaluate our model on TripAdvisor (?) and BeerAdvocate (?, ?, ?) datasets,
which contain seven aspects (value, room, location, cleanliness, check-in or
front desk, service, and business) and four aspects (feel, look, smell, and
taste) respectively. We run the same preprocessing steps as (?). Both datasets
are split into train/development/test sets with proportions 8:1:1. All methods
can use development set to tune their hyper-parameters.

Ratings of TripAdvisor and BeerAdvocate datasets are on scales of $1$ to $5$
and $0$ to $5$ respectively. But in BeerAdvocate, $0$ star is rare, so we
treat the scale as $1$ to $5$. We convert original scales to binary scales as
follows: $1$ and $2$ stars are treated as negative, $3$ is ignored, and $4$
and $5$ stars are treated as positive. In BeerAdvocate, most reviews have
positive polarities, so to avoid the unbalanced issue, we perform data
selection according to overall polarities. After data selection, the number of
reviews with negative overall polarities and that with positive overall
polarities are equal.

####  3.1.2 Hip Fracture Classification

The dataset is collected from Hong Kong Hospital Authority which is a
territory-wise and multi-center database storing abundant medical data such as
clinical summary, radiology report, and radiology image. Hong Kong Hospital
Authority has collected medical data since 1999. This dataset is collected
during the period of 2008 – 2017. In this dataset, each sample consists of
multiple radiology images and a textual radiology report. We only use textual
radiology reports in our experiment. All samples come from two patient groups,
one of which comprises $|X_{p}|$ (number of positive samples) patients with
hip fracture(s), and the other group consists of $|X_{n}|$ (number of negative
samples) patients aged $65$ or above with hip or pelvic X-ray conducted and
with no hip fracture diagnosed. The whole dataset is highly unbalanced. The
number of negative samples (no hip fracture) is much larger than that of
positive samples. We extracted all positive samples and partial negative
samples to get a balanced dataset to evaluate our model.

In the dataset, most textual radiology reports consist of examinations of
different body parts. Each examination is normally written in the following
format. It starts with the type of examination, e.g., computed tomography (CT)
of the abdomen. It is followed by some of the following terms such as clinical
history, comparison (to previous examinations), technique, and findings. To
eliminate textual inputs of unrelated body parts, we extract only the text
segment(s) related to hip or pelvis through keywords matching for those
documents which follow the format described earlier. Afterward, we perform
standard text preprocessing steps such as removing punctuation and stopwords
as well as converting all alphabetic letters to lower case.

The dataset is shuffled and then divided into training/development/test sets
with 8:1:1 proportion. All methods can use development set to tune their
hyper-parameters.

###  3.2 Target Opinion Word Pairs Extraction

####  3.2.1 Document-level Multi-aspect Sentiment Classification

Dataset | TripAdvisor | BeerAdvocate | HAHip
---|---|---|---
# docs | 28,543 | 27,583 | 5,878
# target words | 3,737 | 3,088 | 12
# opinion words | 12,406 | 9,166 | 19
# word pairs | 208,676 | 249,264 | 11,230
Table 1: Statistics of datasets, the vocabulary size of target words and
opinion words, and the number of extracted target-opinion word pairs.

In sentiment analysis task, target-opinion word pairs extraction is a well
studied problem in sentiment analysis (?, ?, ?, ?). We designed five rules to
extract potential target-opinion word pairs. Our method relies on Stanford
Dependency Parser (?). We describe our rules as follows.

Rule 1: We extract pairs satisfying the grammatical relation amod (adjectival
modifier) (?). For example, in phrase “very good price,” we extract “price”
and “good” as a target-opinion pair.

Rule 2: We extract pairs satisfying the grammatical relation nsubj (nominal
subject), and the head word is an adjective and the tail word is a noun. For
example, in a sentence “The room is small,” we can extract “room” and “small”
as a target-opinion pair.

Rule 3: Some verbs are also opinion words and they are informative. We extract
pairs satisfying the grammatical relation dobj (direct object) when the head
word is one of the following four words: “like”, “dislike”, “love”, and
“hate”. For example, in the sentence “I like the smell,” we can extract
“smell” and “like” as a target-opinion pair.

Rule 4: We extract pairs satisfying the grammatical relation xcomp (open
clausal complement), and the head word is one of the following word:
“seem”,“look”, “feel”, “smell”, and “taste”. For example, in the sentence
“This beer tastes spicy,” we can extract “taste” and “spicy” as a target-
opinion pair.

Rule 5: If the sentence contains some adjectives that can implicitly indicate
aspects, we manually assign them to the corresponding aspects. According to
(?), some adjectives serve both as target words and opinion words. For
example, in the sentence “very tasty, and drinkable,” the previous rules fail
to extract any pair. But we know it contains a target-opinion pair, i.e.,
(taste, tasty). Most of these adjectives have the same root form with the
aspects they indicated, e.g., “clean” (cleanliness), and “overpriced” (price).
This kind of adjective can be extracted first and then we can obtain more
similar adjectives using word similarities. For example, given “tasty,” we
could get “flavorful” by retrieving similar words.

Table 1 shows the statistics of the rule-based extraction on our two datasets.
The first four rules can be applied to any dataset while the last one is
domain dependent which requires human effort to identify these special
adjectives. In practice, rule 5 can be removed to save human effort. The
effect of removing rule 5 is shown in experiments.

After extracting potential target-opinion word pairs, we need to assign them
to different aspects as supervision signals. We select some seed words to
describe each aspect, and then calculate similarities between the extracted
target (or opinion) word and seed words, and assign the pair to the aspect
where one of its seed words has the highest similarity. The similarity we used
is the cosine similarity between two word embeddings trained by word2vec (?).
For example, suppose seed words $\\{$“room”, “bed”$\\}$ and $\\{$“business”,
“Internet”$\\}$ are used to describe the aspect room and business
respectively, and the candidate pair “pillow - soft” will be assigned to the
aspect room if the similarity between “pillow” and “bed” is highest among all
combinations.

####  3.2.2 Hip Fracture Classification

Opinion Words | Target Words
---|---
fracture | fractured | fractures | hip | hips
displaced | dislocated | dislocation | joint | joints
rotated | replacement | destruction | femur | femo
normal | unremarkable | degenerative | inter-trochanter | trochanter
abnormality | suspicious | impacted | angulation | erosion
injury | pain | olique | femoral | pelvis
intact |  |  |  |
Table 2: Opinion words and target words in HAHip.

We do not use dependency parsers to extract word pairs. There is a huge number
of medical terminology in radiology reports, the accuracy of dependency
parsers drops significantly. Radiology reports often comprise abbreviations,
acronyms, and grammatical errors (?). For example, in a report snippet “r hip
fracture noted,” they omit the predicate “is,” and use “r” to replace “right.”
It is difficult to use dependency parsers to extract satisfactory word pairs.
Hence, we manually define target words and opinion words lexicons. The opinion
words and target words used in our method are shown in Table 2. The statistics
of extracted word pairs are shown in Table 1. The opinion words mostly are
adjectives and target words are nouns. These words are selected manually
because of their high frequencies and high relevance.

We do not use dependency parsers, but we assume every pair of two words within
a sentence has a syntactic dependency. If within a sentence, there is a word
which is in the target words lexicon and a word is in the opinion words
lexicon, then the pair of two words should be extracted. The idea is shown as
follows, Within a sentence, if there are one opinion word and one target word,
they are extracted as a word pair. If there is more than one opinion word, we
select the opinion word that is nearest to the target words to form the pair.
If there is more than one target word, we select the target word that is
nearest to the opinion words to form the pair. In our dataset, most sentences
in reports are very short. For the sentence which is longer than $20$ words,
we manually cut it into several sentences and each of them is no longer than
$20$ words.

###  3.3 Compared Methods

####  3.3.1 Document-level Multi-aspect Sentiment Classification

The goal of DMSC is to predict the sentiment polarity (e.g., positive or
negative) of each aspect given a document in which there are several sentences
and each sentence describes one or more aspects. In the document-level multi-
aspect sentiment classification task, we compare our model with the following
baselines:

Majority uses the majority of sentiment polarities in training sets as
predictions.

Lexicon means using an opinion lexicon to assign sentiment polarity to an
aspect (?, ?). We combine two popular opinion lexicons used by (?) and (?) to
get a new one. If an opinion word from extracted pairs is in the positive
(negative) lexicon, it votes for positive (negative). When the opinion word is
with a negation word, its polarity will be flipped. Then, the polarity of an
aspect is determined by using majority voting among all opinion words
associated with the aspect. When the number of positive and negative words is
equal, we adopt two different ways to resolve it. For Lexicon-R, it randomly
assigns a polarity. For Lexicon-O, it uses the overall polarity as the
prediction. Since overall polarities can also be missing, for both Lexicon-R
and Lexicon-O, we randomly assign a polarity in uncertain cases and report
both mean and std based on five trials of random assignments.

Assign-O means directly using the overall polarity of a review in the
development set or test set as the prediction for each aspect.

LRR assumes the overall polarity is a weighted sum of the polarity of each
aspect (?). LRR can be regarded as the only existing weakly supervised
baseline where both algorithm and source code are available.

BoW-ALL is a simple softmax classifier using all annotated training data where
the input is a bag-of-words feature vector of a document.

NDMSC-ALL is the state-of-the-art neural network based model (?) (NDMSC) in
DMSC task using all annotated training data, which serves an upper bound to
our method.

NDMSC-O is to use overall polarities as a supervision signal to train an NDMSC
and apply it to the classification task of each aspect at the test time.

NDMSC-{50,100,200,500,1000} is the NDMSC algorithm (?) using partial data. In
order to see our method is comparable to supervised methods using how many
labeled data, we use $\\{50,100,200,500,1000\\}$ annotated data of each aspect
to train NDMSC and compare them to our method. In addition to annotated data
for training, there are extra $20\%$ annotated data for validation. Since the
sampled labeled data may vary for different trials, we perform five trials of
random sampling and report both mean and std of the results.

For our method, denoted as VUSC, we assume that a target word and a possible
value of sentiment polarity $c$ are independent, and assume $C$ follows a
uniform distribution. Since the vocabulary size of opinion words is large, we
use the negative sampling technique to approximate the partition in Eq. (2).
Since there are two sentiment polarities, we do not need to use likelihood
ratio estimation to approximate the expectation part.

The document representation we used is obtained from NDMSC (?). They proposed
a novel hierarchical iterative attention model in which documents and pseudo
aspect related questions are interleaved at both word and sentence-level to
learn an aspect-aware document representation. The pseudo aspect related
questions are represented by aspect related keywords. In order to benefit from
their aspect-aware representation scheme, we train an NDMSC to extract the
document representation using only overall polarities. In the iterative
attention module, we use the pseudo aspect related keywords of all aspects
released by (?). One can also use document-to-document autoencoders (?) to
generate the document representation. In this way, our method can get rid of
using overall polarities to generate the document representation. Hence,
unlike LRR, it is not necessary for our method to use overall polarities.
Here, to have a fair comparison with LRR, we use the overall polarities to
generate document representation. For our method, we do not know which state
is positive and which one is negative at training time, so the Hungarian
algorithm (?) is used to resolve the assignment problem at the test time.

K-means can be a baseline, but in DMSC task, the inputs of K-means are the
same for different aspects, hence, the averaged results of different aspects
are very bad. Hence, we do not compare with it in DMSC task.

####  3.3.2 Hip Fracture Classification

In hip fracture classification task, we compare our model with following
baselines:

K-means++ (?) is identical to standard K-means clustering algorithm (?) except
centroids initialization strategy. K-means++ chooses initial centroids one
after the other in a special manner such that every new centroid is likely to
be far away from the existing centroids. The number of centroids is set to
two.

Lexicon refers to an unsupervised approach similar to the Lexicon method in
DMSC task. The idea is as follows. Each of the extracted opinion words would
vote for positive if it belongs to {fractures, fracture, fractured}, or it
would vote for negative if it is in {no fractures, no fracture, no fractured}.
If neither of the above two cases happens, the word would vote for either
fracture or no-fracture with equal probability. When the numbers of the two
types of votes are equal, the sentiment polarity of the document is randomly
predicted. Finally, the majority voting among all corresponding opinion words
would determine the classification result.

LR Logistic Regression uses a logistic function to model the probability of
one category in binary classification. The training objective is to find a
weight vector such that the overall loss such as binary cross-entropy is
minimized, with regularization techniques such as Ridge regularization (?).
The inputs are BoW (tf-idf) representation. This method is one of the
supervised upper bounds to our approach.

SVM Support Vector Machine (?) algorithm constructs a hyperplane that
maximizes the margin between the hyperplane itself and the nearest training
data point of any category. The inputs are BoW (tf-idf) representation. We
compare our approach with linear support vector machine in the experiment,
which serves as a supervised upper-bound to our model.

CNN Convolutional Neural Network (?) is a supervised approach for text
classification. It implicitly extracts informative local features from input
document matrix composed of pre-trained word vectors through operations such
as convolution and max-pooling. Three different filter sizes $\\{3,4,5\\}$ are
applied in our implementation, and a max pooling layer is applied to each
convolutional layer, and each convolutional layer has 100 filters. Results of
both trainable embeddings setting and fixed embeddings setting are reported.

LSTM Long Short-Term Memory (?) model processes data sequentially through
incorporating the information of previous steps. In the experiment, we adopt
vanilla LSTM implementation (?), and the hidden dimension is set to $50$.
Similar to CNN, we report the performance of LSTM with trainable embeddings
and fixed embeddings.

For our method, denoted as VUSC, we assume that a target word and a possible
value of sentiment polarity $c$ are independent, and assume $C$ follows a
uniform distribution. Since the vocabulary size of opinion words is small, we
do not use the negative sampling technique to approximate the partition in Eq.
(2). Since there are two sentiment polarities, we do not use likelihood ratio
estimation to approximate the expectation part. The document representations
of our model are BoW, CNN, and LSTM, which are called VUSC(BoW), VUSC(CNN),
and VUSC(LSTM) respectively.

###  3.4 Results and Analysis

####  3.4.1 Document-level Multi-aspect Sentiment Classification

Dataset | TripAdvisor | BeerAdvocate
---|---|---
| DEV | TEST | DEV | TEST
| Mean | Std | Mean | Std | Mean | Std | Mean | Std
Majority | 0.6286 | – | 0.6242 | – | 0.6739 | – | 0.6726 | –
Lexicon-R | 0.5914 | 0.0021 | 0.5973 | 0.0018 | 0.5895 | 0.0020 | 0.5881 | 0.0025
Lexicon-O | 0.7153 | 0.0012 | 0.7153 | 0.0015 | 0.6510 | 0.0023 | 0.6510 | 0.0021
Assign-O | 0.7135 | 0.0016 | 0.7043 | 0.0020 | 0.6652 | 0.0028 | 0.6570 | 0.0034
NDMSC-O | 0.7091 | – | 0.7064 | – | 0.6386 | – | 0.6493 | –
LRR | 0.6915 | 0.0045 | 0.6947 | 0.0024 | 0.5976 | 0.0110 | 0.5941 | 0.0113
VUSC | 0.7577 | 0.0016 | 0.7561 | 0.0012 | 0.7502 | 0.0058 | 0.7538 | 0.0066
NDMSC-50 | 0.7255 | 0.0231 | 0.7270 | 0.0204 | 0.7381 | 0.0143 | 0.7442 | 0.0157
NDMSC-100 | 0.7482 | 0.0083 | 0.7487 | 0.0069 | 0.7443 | 0.0126 | 0.7493 | 0.0145
NDMSC-200 | 0.7531 | 0.0040 | 0.7550 | 0.0043 | 0.7555 | 0.0096 | 0.7596 | 0.0092
NDMSC-500 | 0.7604 | 0.0028 | 0.7616 | 0.0040 | 0.7657 | 0.0066 | 0.7713 | 0.0070
NDMSC-1000 | 0.7631 | 0.0054 | 0.7638 | 0.0042 | 0.7708 | 0.0066 | 0.7787 | 0.0053
NDMSC-All | 0.8281 | – | 0.8334 | – | 0.8576 | – | 0.8635 | –
BoW-All | 0.8027 | – | 0.8029 | – | 0.8069 | – | 0.8089 | –
Table 3: Averaged accuracies on DMSC of unsupervised, weakly supervised, and
supervised methods on TripAdvisor and BeerAdvocate.

We show all results in Table 3, which consists of three blocks, namely,
unsupervised, weakly supervised, and supervised methods.

For unsupervised methods, our method can outperform the majority on both
datasets consistently. But other weakly supervised methods cannot outperform
the majority on BeerAdvocate dataset, which shows these baselines cannot
handle unbalanced data well since BeerAdvocate is more unbalanced than
TripAdvisor. Our method outperforms Lexicon-R and Lexicon-O, which shows that
predicting an opinion word based on a target word may be a better way to use
target-opinion pairs, compared with performing a lexicon lookup using opinion
words from extract pairs. Good performance of Lexicon-O and Assign-O
demonstrates the usefulness of overall polarities in development/test sets.
NDMSC-O trained with the overall polarities cannot outperform Assign-O since
NDMSC-O can only see overall polarities in training set while Assign-O can see
overall polarities for both development and test sets and does not involve
learning and generalization.

For weakly supervised methods, LRR is the only open-source baseline in the
literature on weakly supervised DMSC, and our method (VUSC) outperforms LRR by
6% and 16% on TripAdvisor and BeerAdvocate datasets. NDMSC-O can also be
considered as a weakly supervised method because it only uses overall
polarities as a supervision signal, and we still outperform it significantly.
It is interesting that LRR is worse than NDMSC-O. We guess that assuming that
the overall polarity is a weighted sum of all aspect polarities may not be a
good strategy to train each aspect’s polarity or the document representation
learned by NDMSC is better than the bag-of-words representation.

For supervised block methods, BoW-All and NDMSC-All are both supervised
methods using all annotated data, which can be seen as the upper bound of our
algorithm. NDMSC-All outperforms BoW-All, which shows that the document
representation based on neural network is better than the bag-of-words
representation. Hence, we use the neural networks based document
representation as the input of the sentiment polarity classifier. Our results
are comparable to NDMSC-200 on TripAdvisor and NDMSC-100 on BeerAdvocate.

####  3.4.2 Hip Fracture Classification

Methods | Dev | Test
---|---|---
| Mean | Std | Mean | Std
VUSC(BoW) | 0.9086 | 0.0066 | 0.9109 | 0.0056
VUSC(CNN) | 0.9521 | 0.0073 | 0.9383 | 0.0069
VUSC(LSTM) | 0.6627 | 0.0365 | 0.6599 | 0.0247
Lexicon | 0.8457 | 0.0104 | 0.8476 | 0.0089
K-means++ | 0.8700 | 0.0005 | 0.8576 | 0.0000
CNN (trainable) | 0.9626 | 0.0013 | 0.9647 | 0.0045
CNN (fixed) | 0.9609 | 0.0007 | 0.9647 | 0.0025
LSTM (trainable) | 0.9366 | 0.0102 | 0.9302 | 0.0042
LSTM (fixed) | 0.9322 | 0.0058 | 0.9213 | 0.0025
LR | 0.9675 | 0.0000 | 0.9678 | 0.0000
SVM | 0.9675 | 0.0000 | 0.9695 | 0.0000
Table 4: Accuracies of unsupervised and supervised methods on hip fracture
classification task. Trainable or fixed means the embeddings are trainable or
fixed.

Regarding unsupervised models, our two models i.e., VUSC(BoW) and VUSC(CNN)
outperform other unsupervised methods by a large margin. For instance, the
accuracy of VUSC(CNN) on the test set is 9% higher than that of Lexicon and 8%
higher than that of K-means++. For our methods, document representation has a
large influence on results. The best document representation is CNN. CNN
representation is slightly better than BoW. The possible reason is that BoW is
a fixed representation while CNN a is trainable representation. CNN has a
better capacity for feature extraction. The results of VUSC(LSTM) are very
low. One possible reason is that training LSTM is more difficult compared with
CNN, and the supervised signal from target-opinion word pairs is not strong
enough to supervise the LSTM.

In terms of supervised methods, the results of CNN, LR, SVM are comparable.
The results of LSTM are $3\%$ lower than other supervised methods, which also
suggests that LSTM is difficult to train well in this dataset. Both our two
models, i.e., the accuracies of VUSC(BoW) and VUSC(CNN) are slightly ($5\%$
and $3\%$) lower than CNN, LR, and LSTM, which indicates that our approach is
very promising. For example, the accuracy of VUSC(CNN) (i.e., $0.9383$) is
very close to the best score (i.e., $0.9695$ of SVM) and it is even higher
than that of both LSTM models with or without embeddings trainable (i.e.,
$0.9302$ and $0.9213$).

###  3.5 Ablation Study of Extraction Rules

To evaluate the effects of extraction rules, we performed an ablation study on
DMSC task. We run our algorithm VUSC with each rule kept or removed over two
datasets. If no pairs extracted for one aspect in the training set, the
accuracy of this aspect will be 0.5, which is a random guess. From the Table 5
we can see that the rule R1 is the most effective rule for both datasets.
Rules R3/R4/R5 are less effective on their own. However, as a whole, they can
still improve the overall performance. When considering removing each rule, we
found that our algorithm is quite robust, which indicates missing one of the
rules may not hurt the performance much. Hence, if human labor is a major
concern, rule 5 can be discarded. We found that sometimes removing one rule
may even result in better accuracy (e.g., “-R3” for BeerAdvocate dataset).
This means this rule may introduce some noises into the objective function.
However, “-R3” can result in worse accuracy for TripAdvisor, which means it is
still complementary to the other rules for this dataset.

Dataset | TripAdvisor | BeerAdvocate
---|---|---
Rule | Dev | Test | Dev | Test
R1 | 0.7215 | 0.7174 | 0.7220 | 0.7216
R2 | 0.7172 | 0.7180 | 0.6864 | 0.6936
R3 | 0.6263 | 0.6187 | 0.6731 | 0.6725
R4 | 0.6248 | 0.6279 | 0.6724 | 0.6717
R5 | 0.5902 | 0.5856 | 0.7095 | 0.7066
\- R1 | 0.7538 | 0.7481 | 0.7458 | 0.7474
\- R2 | 0.7342 | 0.7368 | 0.7504 | 0.7529
\- R3 | 0.7418 | 0.7397 | 0.7565 | 0.7558
\- R4 | 0.7424 | 0.7368 | 0.7518 | 0.7507
\- R5 | 0.7448 | 0.7440 | 0.7550 | 0.7548
All | 0.7577 | 0.7561 | 0.7502 | 0.7538
Table 5: Averaged accuracies on DMSC. “R1 – R5” means only using a rule while
“-R1 – -R5” means removing a rule from all the rules.

###  3.6 Case Study

We carry out two case studies below to empirically uncover more details of our
methods on hip fracture classification task.

####  3.6.1 Analysis of Mis-classified Samples

We present two misclassified samples of VUSC(BoW) and VUSC(CNN), where all
sensitive information and irrelevant sentences are removed and replaced by
***.

Regarding VUSC(BoW), it predicts the report saying “*** pelvis and r hip
fracture. *** pelvic ring is intact. *** hip joint are unremarkable .” to be
_negative_ , which turns out to be wrong. According to the report, pelvis and
right hip are fractured, but the pelvic ring and hip joint are not. The model
might focus on the latter and thus give a wrong prediction. Regarding
VUSC(BoW), it predicts the reports saying “*** no fracture. *** femur possibly
healed old injury to right inferior pubic ramus. ***” to be _positive_ , which
is incorrect. According to the report, the patient has no fracture but has an
old injury. In most samples, hip injury comes along with hip fracture. This
method might capture this pattern. As for VUSC(CNN), it predicts the report
saying “*** there is a displaced left intertrochanteric fracture. no other
fracture or dislocation identified. ***” as _negative_ , which is also very
likely to be caused by the misleading sentence “no other fracture or
dislocation identified”. The model may consider it as “no fracture”. In other
report stating that “*** right hip pain ***”, the model predicts it to be
_positive_ , which is wrong. According to the report, the patient feels pain
in the right hip. In most samples, pain comes along with hip fracture. The
probability of fracture will increase if there is “pain” in the report. The
model misclassifies the report. As can be seen from the above two cases, if
the report states many areas of the body, and some areas are fine but some are
suggesting fractures, then the model could be confused.

####  3.6.2 Analysis of Word Weights

Positive | Negative
---|---
1\. hip | 2\. left | 1\. hip | 2\. no
3\. clinical | 4\. right | 3\. fracture | 4\. pelvis
5\. fracture | 6\. femur | 5\. seen | 6\. right
7\. noted | 8\. pain | 7\. left | 8\. noted
9\. injury | 10\. pelvis | 9\. xr | 10\. clinical
11\. neck | 12\. xr | 11\. bony | 12\. diagnosis
13\. ap | 14\. ring | 13\. ap | 14\. history
15\. pelvic | 16\. diagnosis | 15\. femur | 16\. lesion
17\. history | 18\. seen | 17\. bone | 18\. intact
19\. intact | 20\. inter-trochanteric | 19\. pelvic | 20\. definite
Table 6: Top 20 largest weighted words for postive weight and negative weight
in VUSC(BoW).

To have a deeper understanding of how our approach elicits crucial information
hidden in unstructured text, we investigate the weight vectors
(i.e.,$\mathbf{w}_{c}$ in Eq. (1)) of the hip fracture classifier in VUSC(BoW)
model. In Table 6, we show top $20$ largest weighted words in
$\mathbf{w}_{\textit{positive}}$ and $\mathbf{w}_{\textit{negative}}$. Large
values in the $\mathbf{w}_{\textit{positive}}$
($\mathbf{w}_{\textit{negative}}$) weight vector mean the associated words
having stronger influence to encourage the model to predict positive
(negative) category, as they contribute more in the dot product of logit. As
for words in the left column, some words including “fracture”, “injury” and
“pain” also have a strong positive correlation with hip fracture, which also
aligns well with intuition. For other words which seem to be rather neutral
(e.g., “hip” and “left”), it is possible that their high frequency leads to
their high value in the weight vector. Regarding words in the right column,
some words such as “no” and “intact” intuitively make sense as they are
powerful indicators of having no hip fracture. Nonetheless, it is worthwhile
to notice that these words might also induce the model to make false
predictions. For instance, lots of false negative predictions made by
VUSC(BoW) model have both “no” and “fracture”(or “fractured”) inside the text,
which causes the model to categorize them as negative. But the keyword “no”
actually does not syntactically modify “fracture”, just like the sentence “no
lesion, there is fracture” in one of the false negative samples. Therefore, it
is essential to design a more sophisticated mechanism to accurately capture
the relationships among words in documents, so that the model can understand
the document in a more comprehensive and precise way.

###  3.7 Implementation Details

We implemented our models using TensorFlow (?). For NDMSC and LRR, we used
code released by (?) and (?) respectively, and followed their preprocessing
steps and optimal settings.

Parameters are updated by using ADADELTA (?), an adaptive learning rate
method. To avoid overfitting, we impose weight decay and drop out on both
classifiers. The regularization coefficient and drop out rate are set to
$10^{-3}$ and $0.3$ respectively. In DMSC task, the number of negative samples
and $\alpha$ in our model are set to $10$ and $0.1$ respectively. For both
datasets, $\alpha$ is set to $0.1$. In the hip fracture classification task,
the number of negative samples is set to $10$. For VUSC(BoW), $\beta$ and
$\gamma$ are set to $0.2$ and $0.8$ respectively. For VUSC(CNN), $\beta$ and
$\gamma$ are set to $0.9$ and $0.6$ respectively. For VUSC(LSTM), $\beta$ and
$\gamma$ are set to $0.2$ and $0.4$ respectively. The range of $\beta$ and
$\gamma$ are $[0.1,0.2,...1.0]$.

For each document (and each aspect), multiple target-opinion pairs are
extracted. The opinion word classifier associated with an aspect will predict
five target-opinion pairs at a time. These five target-opinion pairs are
selected with bias. The probability of a pair being selected is proportional
to the frequency of the opinion word to the power of $-0.25$. In this way,
opinion words with low frequency are more likely to be selected compared to
uniform sampling. In DMSC task, in order to initialize both classifiers
better, the word embeddings are retrofitted (?) using PPDB (?) semantic
lexicons. The embeddings size is set $200$. In the hip fracture classification
task, we train word embeddings using word2vec algorithm (?) on the whole
dataset. The embedding size is set to $50$. For our method and K-means++, we
do not know which state is positive and which one is negative at training
time, so the Hungarian algorithm (?) is used to resolve the assignment problem
at the test time.

##  4 Related Work

In this section, we review the related work on unsupervised sentiment
analysis, document-level multi-aspect sentiment classification, clinical
sentiment analysis, target-opinion word pairs extraction, and variational
methods.

##### Unsupervised Sentiment Analysis.

Lexicon-based approaches are typical ways to perform unsupervised sentiment
analysis. These methods use a sentiment lexicon consists of a list of opinion
words along with their sentiment orientation scores to determine the overall
sentiment of a given text. Some methods (?, ?) used sentiment orientation
scores in existing lexicons directly, and aggregated them within a document to
determine polarity. Some methods developed their own semantic orientation
estimation algorithm. For example, (?) first identified phrases in the review
and then estimated the semantic orientation of each extracted phrases. The
semantic orientation of a given phrase is calculated by comparing its
similarity to a positive reference word (“excellent”) with its similarity to a
negative reference word (“poor”). This method determined the sentiment
polarity based on the average semantic orientation of the phrases extracted
from the review. (?) used the minimum path distance between a phrase and pivot
words (“good” and “bad”) in WordNet to estimate the semantic orientation of
extracted phrases. Another line of works (?, ?) proposed a constrained non-
negative matrix tri-factorization approach to sentiment analysis, and used a
sentiment lexicon as prior knowledge. In these models, a term-document matrix
is approximated by three factors that specify soft membership of terms and
documents in one of $k$ classes. All three factors are non-negative matrices.
The first factor is a matrix representing knowledge in the word space, i.e.,
each row represents the posterior probability of a word belonging to the $k$
classes. The second factor is a matrix providing a condensed view of the term-
document matrix. The third factor is a matrix representing knowledge in
document space, i.e., each row represents the posterior probability of a
document belonging to the $k$ classes. (?) applied a regularization to
encourage that the first factor is close to prior knowledge. (?) applied a
regularization based on an intuition that if two words (or documents) are
sufficiently close to each other, they tend to share the same sentiment
polarity.

##### Document-level Multi-Aspect Sentiment Classification.

(?) proposed an LRR model to solve this problem. LRR assumes the overall
polarity is a weighted sum of all aspect polarities which are represented by
word frequency features. LRR needs to use aspect keywords to perform sentence
segmentation to generate the representation of each aspect. To address the
limitation of using aspect keywords, LARAM (?) assumes that the text content
describing a particular aspect is generated by sampling words from a topic
model corresponding to the latent aspect. Both LRR and LARAM can only access
to overall polarities in the training data, but not gold standards of aspect
polarities. (?) proposed a weakly supervised text classification method that
can take label surface names, class-related keywords, or a few labeled
documents as supervision. (?) developed a weakly supervised joint model to
identify aspects and the corresponding sentiment polarities in online courses.
They treat aspect (sentiment) related seed words as weak supervision. In the
DMSC task which is a fine-grained text classification task, the label surface
names or keywords for some aspects would be very similar. Given that the
inputs are the same and the supervisions are similar, weakly supervised models
cannot distinguish them. So we do not consider them as our baselines. (?)
modeled this problem as a machine comprehension problem under a multi-task
learning framework. It also needs aspect keywords to generate aspect-aware
document representations. Moreover, it can access gold standards of aspect
polarities and achieved state-of-the-art performance on this task. Hence, it
can serve as an upper bound. Some sentence-level aspect based sentiment
classification methods (?, ?) can be directly applied to the DMSC task,
because they can solve aspect category sentiment classification task. For
example, given a sentence “the restaurant is expensive,” the aspect category
sentiment classification task aims to classify the polarity of the aspect
category “price” to be negative. The aspect categories are predefined which
are the same as the DMSC task. Some of them (?, ?, ?, ?) cannot because they
are originally designed for aspect term sentiment classification task. For
example, given a sentence “I loved their fajitas,” the aspect term sentiment
classification task aims to classify the polarity of the aspect term “fajitas”
to be positive. The aspect terms appearing in the sentence should be provided
as inputs.

##### Clinical Sentiment Analysis.

Clinical sentiment analysis aims to determine patients’ health status given
clinical narratives such as nurse letters, discharge summary, and radiology
reports or effectiveness of a treatment or medication given medical social
media text (?, ?, ?). Here we focus on clinical narratives. This task is
useful because it could assist physicians to have a complete evaluation of
patients’ health status or performance diagnosis automatically. (?) proposed a
regular expression discovery (RED) algorithm and implemented two text
classifiers based on RED to predict smoking status and pain status. (?)
proposed a rule-based feature engineering algorithm to identify trigger
phrases and then trained knowledge-guided convolutional neural networks to
predict patient disease status with respect to obesity and 15 of its
comorbidities. (?) annotated psychiatric electronic health records (EHRs)
texts at the sentence level and train lexicon-based, semi-supervised, and
supervised machine learning models to predict patients’ mental health status.
(?) developed a rule-based NLP algorithm to automatically generate labels for
the training data, and then trained machine learning models including Support
Vector Machine (SVM), Random Forrest (RF), Multilayer Perceptron Neural
Networks (MLPNN), and Convolutional Neural Networks (CNN) to predict the
smoking status and proximal femur (hip) fracture condition.

##### Target Opinion Word Pairs Extraction.

There are two kinds of methods, namely, rule-based methods and learning based
methods to solve this task. Rule-based methods extract target-opinion word
pairs by mining the dependency tree paths between target words and opinion
words. Learning based methods treat this task as a sequence labeling problem,
mapping each word to one of the following categories: target, opinion, and
other. (?) is one of the earliest rule-based methods to extract target-opinion
pairs. An opinion word is restricted to be an adjective. Target words are
extracted first, and then an opinion word is linked to its nearest target word
to form a pair. (?) and (?) manually designed dependency tree path templates
to extract target-opinion pairs. If the path between a target word candidate
and an opinion word candidate belongs to the set of path templates, the pair
will be extracted. (?) identified dependency paths that link opinion words and
targets via a bootstrapping process. This method only needs an initial opinion
lexicon to start the bootstrapping process. (?) adopted a supervised learning
algorithm to learn valid dependency tree path templates, but it requires
target-opinion pairs annotations. Learning based methods require lots of
target-opinion pairs annotations. They trained conditional random fields (CRF)
(?) based models (?, ?, ?) or deep neural networks (?, ?, ?) to predict the
label (target, opinion or other) of each word. (?) and (?) extracted target-
opinion pairs without using any labeled data in the domain of interest, but it
needs lots of labeled data in another related domain. In this paper, we only
use very simple rules to extract target-opinion pairs to validate the
effectiveness of our approach. If better pairs can be extracted, we can
further improve our results.

##### Variational Methods.

Variational autoencoders (?, ?) (VAEs) use a neural network to parameterize a
probability distribution. VAEs consist of an encoder that parameterizes
posterior probabilities and a decoder which parameterizes the reconstruction
likelihood given a latent variable. VAEs inspire many interesting works (?, ?,
?, ?, ?, ?) which are slightly different from VAEs. Their encoders produce a
discrete distribution while the encoder in VAEs yields a continuous latent
variable. (?) aimed to solve the semantic role labeling problem. The encoder
is essentially a semantic role labeling model that predicts roles given a rich
set of syntactic and lexical features. The decoder reconstructs argument
fillers given predicted roles. (?) aimed to solve unsupervised open domain
relation discovery. The encoder is a feature-rich relation extractor, which
predicts a semantic relation between two entities. The decoder reconstructs
entities relying on the predicted relation. (?) tried to learn multi-sense
word embeddings. The encoder uses bilingual context to choose a sense for a
given word. The decoder predicts context words based on the chosen sense and
the given word. (?) aimed to solve knowledge graph powered question answering.
Three neural networks are used to parameterize probabilities of a topic entity
given a query and an answer, an answer based on a query and a predicted topic,
and the topic given the query. (?) aimed to infer missing links in a knowledge
graph. Three neural networks are used to parameterize probabilities of a
latent path given two entities and a relation, a relation based on two
entities and the chosen latent path, and the relation given the latent path.
(?) aims to solve document-level multi-aspect sentiment classification task.
It uses a neural network to parameterize a discrete distribution which is
severed as sentiment classifier and a neural network to parameterize
probabilities of opinioned words given target words and possible sentiment
polarity.

##  5 Conclusion

In this paper, we propose a variational approach to unsupervised sentiment
analysis. We extract many target-opinion word pairs by using dependency
parsers and simple rules. These pairs can be supervision signals to predict
sentiment polarity. Our objective function is to predict an opinion word given
a target word. After introducing the sentiment polarity of a document as a
latent variable, we can learn a sentiment classifier by optimizing the
variational lower bound. We also impose sophisticated constraints on extracted
words as regularization. The experiment results show our method can outperform
unsupervised baselines in sentiment analysis task on both domains, and our
method obtains comparable results to the supervised method with hundreds of
labels per aspect in customer reviews domain, and obtains comparable results
to supervised methods in clinical narratives domain.

## References

  * doi
  * Abadi et al. Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., Kudlur, M., Levenberg, J., Monga, R., Moore, S., Murray, D. G., Steiner, B., Tucker, P., Vasudevan, V., Warden, P., Wicke, M., Yu, Y., & Zheng, X. (2016).  Tensorflow: A system for large-scale machine learning.  In Proceedings of OSDI, pp. 265–283.
  * Abualigah et al. Abualigah, L., Alfar, H. E., Shehab, M., & Hussein, A. M. A. (2020).  Sentiment analysis in healthcare: A brief review.  In Recent Advances in NLP: The Case of Arabic Language, pp. 129–141. Springer.
  * Arthur & Vassilvitskii Arthur, D., & Vassilvitskii, S. (2007).  k-means++: The advantages of careful seeding.  In Proceedings of SODA, pp. 1027–1035.
  * Bloom et al. Bloom, K., Garg, N., Argamon, S., et al. (2007).  Extracting appraisal expressions..  In Proceedings of NAACL-HLT, pp. 308–315.
  * Bui & Zeng-Treitler Bui, D. D. A., & Zeng-Treitler, Q. (2014).  Learning regular expressions for clinical text classification.  Journal of the American Medical Informatics Association, 21(5), 850–857.
  * Chen & Manning Chen, D., & Manning, C. (2014).  A fast and accurate dependency parser using neural networks.  In Proceedings of EMNLP, pp. 740–750.
  * Chen et al. Chen, P., Sun, Z., Bing, L., & Yang, W. (2017).  Recurrent attention network on memory for aspect sentiment analysis.  In Proceedings of EMNLP, pp. 452–461.
  * Chen et al. Chen, W., Xiong, W., Yan, X., & Wang, W. (2018).  Variational knowledge graph reasoning.  In Proceedings of NAACL-HLT, pp. 1823–1832.
  * Cortes & Vapnik Cortes, C., & Vapnik, V. (1995).  Support-vector networks.  Machine learning, 20(3), 273–297.
  * De Marneffe & Manning De Marneffe, M.-C., & Manning, C. D. (2008).  Stanford typed dependencies manual.  Tech. rep., Technical report, Stanford University.
  * Denecke & Deng Denecke, K., & Deng, Y. (2015).  Sentiment analysis in medical settings: New opportunities and challenges.  Artificial intelligence in medicine, 64(1), 17–27.
  * Devlin et al. Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019).  Bert: Pre-training of deep bidirectional transformers for language understanding., 4171–4186.
  * Faruqui et al. Faruqui, M., Dodge, J., Jauhar, S. K., Dyer, C., Hovy, E., & Smith, N. A. (2015).  Retrofitting word vectors to semantic lexicons.  In Proceedings of NAACL-HLT, pp. 1606–1615.
  * Ganitkevitch et al. Ganitkevitch, J., Van Durme, B., & Callison-Burch, C. (2013).  Ppdb: The paraphrase database.  In Proceedings of NAACL-HLT, pp. 758–764.
  * Gers et al. Gers, F. A., Schmidhuber, J., & Cummins, F. (1999).  Learning to forget: Continual prediction with lstm..
  * Glynn Glynn, P. W. (1987).  Likelilood ratio gradient estimation: an overview.  In Proceedings of CWS, pp. 366–375.
  * Holderness et al. Holderness, E., Cawkwell, P., Bolton, K., Pustejovsky, J., & Hall, M. (2019).  Distinguishing clinical sentiment: The importance of domain adaptation in psychiatric patient health records.  CoRR, abs/1904.03225.
  * Hu & Liu Hu, M., & Liu, B. (2004).  Mining and summarizing customer reviews.  In Proceedings of SIGKDD, pp. 168–177.
  * Jakob & Gurevych Jakob, N., & Gurevych, I. (2010).  Extracting opinion targets in a single- and cross-domain setting with conditional random fields.  In Proceedings of EMNLP, pp. 1035–1045.
  * Kamps et al. Kamps, J., Marx, M., Mokken, R. J., De Rijke, M., et al. (2004).  Using wordnet to measure semantic orientations of adjectives..  In Proceedings of LREC, Vol. 4, pp. 1115–1118. Citeseer.
  * Kim Kim, Y. (2014).  Convolutional neural networks for sentence classification., 1746–1751.
  * Kingma & Welling Kingma, D. P., & Welling, M. (2014).  Auto-encoding variational bayes.  In Proceedings of ICLR.
  * Kuhn Kuhn, H. W. (1955).  The hungarian method for the assignment problem.  Naval Research Logistics, 2(1-2), 83–97.
  * Lafferty et al. Lafferty, J. D., McCallum, A., & Pereira, F. C. N. (2001).  Conditional random fields: Probabilistic models for segmenting and labeling sequence data.  In Proceedings of ICML, pp. 282–289.
  * Lakkaraju et al. Lakkaraju, H., Socher, R., & Manning, C. (2014).  Aspect specific sentiment analysis using hierarchical deep learning.  In Proceedings of NeurIPS workshop on Deep Learning and Representation Learning.
  * Lei et al. Lei, T., Barzilay, R., & Jaakkola, T. (2016).  Rationalizing neural predictions.  In Proceedings of EMNLP, pp. 107–117.
  * Li et al. Li, F., Pan, S. J., Jin, O., Yang, Q., & Zhu, X. (2012).  Cross-domain co-extraction of sentiment and topic lexicons.  In Proceedings of ACL, pp. 410–419.
  * Li et al. Li, J., Luong, M.-T., & Jurafsky, D. (2015).  A hierarchical neural autoencoder for paragraphs and documents.  In Proceedings of ACL, pp. 1106–1115.
  * Li et al. Li, T., Zhang, Y., & Sindhwani, V. (2009).  A non-negative matrix tri-factorization approach to sentiment classification with lexical prior knowledge.  In Proceedings of ACL, pp. 244–252.
  * Li & Lam Li, X., & Lam, W. (2017).  Deep multi-task learning for aspect term extraction with memory interaction.  In Proceedings of EMNLP, pp. 2886–2892.
  * Liu Liu, B. (2012).  Sentiment analysis and opinion mining.  Synthesis lectures on human language technologies, 5(1), 1–167.
  * Liu et al. Liu, P., Joty, S., & Meng, H. (2015).  Fine-grained opinion mining with recurrent neural networks and word embeddings.  In Proceedings of EMNLP, pp. 1433–1443.
  * Ma et al. Ma, D., Li, S., Zhang, X., & Wang, H. (2017).  Interactive attention networks for aspect-level sentiment classification.  In Proceedings of IJCAI, pp. 4068–4074.
  * MacQueen et al. MacQueen, J., et al. (1967).  Some methods for classification and analysis of multivariate observations.  In Proceedings of the fifth Berkeley symposium on mathematical statistics and probability, Vol. 1, pp. 281–297. Oakland, CA, USA.
  * Marcheggiani & Titov Marcheggiani, D., & Titov, I. (2016).  Discrete-state variational autoencoders for joint discovery and factorization of relations.  Transactions of the Association for Computational Linguistics, 4, 231–244.
  * McAuley et al. McAuley, J., Leskovec, J., & Jurafsky, D. (2012).  Learning attitudes and attributes from multi-aspect reviews.  In Proceedings of ICDM, pp. 1020–1025.
  * Meng et al. Meng, Y., Shen, J., Zhang, C., & Han, J. (2018).  Weakly-supervised neural text classification.  In Proceedings of CIKM, pp. 983–992.
  * Mikolov et al. Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., & Dean, J. (2013).  Distributed representations of words and phrases and their compositionality.  In Proceedings of NeurIPS, pp. 3111–3119.
  * Missen & Boughanem Missen, M. M. S., & Boughanem, M. (2009).  Using wordnet’s semantic relations for opinion detection in blogs.  In Proceedings of ECIR, pp. 729–733. Springer.
  * Ng et al. Ng, V., Dasgupta, S., & Arifin, S. N. (2006).  Examining the role of linguistic knowledge sources in the automatic identification and classification of reviews.  In Proceedings of ACL, pp. 611–618.
  * Nguyen & Patrick Nguyen, H., & Patrick, J. (2016).  Text mining in clinical domain: Dealing with noise.  In Krishnapuram, B., Shah, M., Smola, A. J., Aggarwal, C. C., Shen, D., & Rastogi, R. (Eds.), Proceedings of SIGKDD, pp. 549–558.
  * Pablos et al. Pablos, A. G., Cuadros, M., & Rigau, G. (2015).  V3: Unsupervised aspect based sentiment analysis for semeval2015 task 12.  In Proceedings of SemEval, pp. 714–718.
  * Popescu & Etzioni Popescu, A.-M., & Etzioni, O. (2005).  Extracting product features and opinions from reviews.  In Proceedings of EMNLP-HLT, pp. 339–346.
  * Qiu et al. Qiu, G., Liu, B., Bu, J., & Chen, C. (2011).  Opinion word expansion and target extraction through double propagation.  Computational linguistics, 37, 9–27.
  * Ramesh et al. Ramesh, A., Kumar, S. H., Foulds, J., & Getoor, L. (2015).  Weakly supervised models of aspect-sentiment for online course discussion forums.  In Proceedings of ACL, pp. 74–83.
  * Read & Carroll Read, J., & Carroll, J. (2009).  Weakly supervised techniques for domain-independent sentiment classification.  In Proceedings of CIKM workshop on Topic-sentiment Analysis for Mass Opinion, pp. 45–52.
  * Rezende et al. Rezende, D. J., Mohamed, S., & Wierstra, D. (2014).  Stochastic backpropagation and approximate inference in deep generative models.  In Proceedings of ICML, pp. 1278–1286.
  * Šuster et al. Šuster, S., Titov, I., & van Noord, G. (2016).  Bilingual learning of multi-sense embeddings with discrete autoencoders.  In Proceedings of NAACL-HLT, pp. 1346–1356.
  * Taboada et al. Taboada, M., Brooke, J., Tofiloski, M., Voll, K. D., & Stede, M. (2011).  Lexicon-based methods for sentiment analysis.  Computational Linguistics, 37(2), 267–307.
  * Tang et al. Tang, D., Qin, B., Feng, X., & Liu, T. (2016a).  Effective lstms for target-dependent sentiment classification.  In Proceedings of COLING, pp. 3298–3307.
  * Tang et al. Tang, D., Qin, B., & Liu, T. (2016b).  Aspect level sentiment classification with deep memory network.  In Proceedings of EMNLP, pp. 214–224.
  * Titov & Khoddam Titov, I., & Khoddam, E. (2015).  Unsupervised induction of semantic roles within a reconstruction-error minimization framework.  In Proceedings of NAACL-HLT, pp. 1–10.
  * Tsytsarau et al. Tsytsarau, M., Palpanas, T., & Denecke, K. (2010).  Scalable discovery of contradictions on the web.  In Proceedings of WWW, pp. 1195–1196.
  * Turney Turney, P. D. (2002).  Thumbs up or thumbs down?: semantic orientation applied to unsupervised classification of reviews.  In Proceedings of ACL, pp. 417–424.
  * Wang et al. Wang, H., Lu, Y., & Zhai, C. (2010).  Latent aspect rating analysis on review text data: a rating regression approach.  In Proceedings of SIGKDD, pp. 783–792.
  * Wang et al. Wang, H., Lu, Y., & Zhai, C. (2011).  Latent aspect rating analysis without aspect keyword supervision.  In Proceedings of SIGKDD, pp. 618–626.
  * Wang et al. Wang, J., Li, J., Li, S., Kang, Y., Zhang, M., Si, L., & Zhou, G. (2018).  Aspect sentiment classification with both word-level and clause-level attention networks..  In Proceedings of IJCAI, pp. 4439–4445.
  * Wang et al. Wang, W., Pan, S. J., Dahlmeier, D., & Xiao, X. (2016).  Recursive neural conditional random fields for aspect-based sentiment analysis.  In Proceedings of EMNLP, pp. 616–626.
  * Wang et al. Wang, W., Pan, S. J., Dahlmeier, D., & Xiao, X. (2017).  Coupled multi-layer attentions for co-extraction of aspect and opinion terms.  In Proceedings of AAAI, pp. 3316–3322.
  * Wang et al. Wang, Y., Sohn, S., Liu, S., Shen, F., Wang, L., Atkinson, E. J., Amin, S., & Liu, H. (2019).  A clinical text classification paradigm using weak supervision and deep representation.  BMC Medical Informatics and Decision Making, 19(1), 1–13.
  * Wang et al. Wang, Y., Huang, M., Zhao, L., et al. (2016).  Attention-based lstm for aspect-level sentiment classification.  In Proceedings of EMNLP, pp. 606–615.
  * Wilson et al. Wilson, T., Wiebe, J., & Hoffmann, P. (2005).  Recognizing contextual polarity in phrase-level sentiment analysis.  In Proceedings of EMNLP-HLT, pp. 347–354.
  * Yang & Cardie Yang, B., & Cardie, C. (2012).  Extracting opinion expressions with semi-markov conditional random fields.  In Proceedings of EMNLP-CoNLL, pp. 1335–1345.
  * Yao et al. Yao, L., Mao, C., & Luo, Y. (2019).  Clinical text classification with rule-based features and knowledge-guided convolutional neural networks.  BMC Medical Informatics and Decision Making, 19-S(3), 31–39.
  * Yin et al. Yin, Y., Song, Y., & Zhang, M. (2017).  Document-level multi-aspect sentiment classification as machine comprehension.  In Proceedings of EMNLP, pp. 2034–2044.
  * Zeiler Zeiler, M. D. (2012).  Adadelta: an adaptive learning rate method.  arXiv preprint arXiv:1212.5701.
  * Zeng et al. Zeng, Z., Zhou, W., Liu, X., & Song, Y. (2019).  A variational approach to weakly supervised document-level multi-aspect sentiment classification.  In Proceedings of NAACL-HLT, pp. 386–396.
  * Zhang et al. Zhang, Y., Dai, H., Kozareva, Z., Smola, A. J., & Song, L. (2018).  Variational reasoning for question answering with knowledge graph.  In Proceedings of AAAI, pp. 6069–6076.
  * Zhou et al. Zhou, G., Zhao, J., & Zeng, D. (2014).  Sentiment classification with graph co-regularization.  In Proceedings of COLING, pp. 1331–1340.
  * Zhuang et al. Zhuang, L., Jing, F., & Zhu, X.-Y. (2006).  Movie review mining and summarization.  In Proceedings of CIKM, pp. 43–50.
  * Zunic et al. Zunic, A., Corcoran, P., & Spasic, I. (2020).  Sentiment analysis in health and well-being: Systematic review.  JMIR Medical Informatics, 8(1), e16023.