# Aspect Sentiment Triplet Extraction Using
Reinforcement Learning

Samson Yu Bai Jian  SUTD, Singapore8 Somapah Road<EMAIL_ADDRESS>,
Tapas Nayak  IIT KGP, India<EMAIL_ADDRESS>,  Navonil Majumder  SUTD,
Singapore<EMAIL_ADDRESS>and  Soujanya Poria  SUTD, Singapore
<EMAIL_ADDRESS>
(2021)

###### Abstract.

Aspect Sentiment Triplet Extraction (ASTE) is the task of extracting triplets
of aspect terms, their associated sentiments, and the opinion terms that
provide evidence for the expressed sentiments. Previous approaches to ASTE
usually simultaneously extract all three components or first identify the
aspect and opinion terms, then pair them up to predict their sentiment
polarities. In this work, we present a novel paradigm, ASTE-RL, by regarding
the aspect and opinion terms as arguments of the expressed sentiment in a
hierarchical reinforcement learning (RL) framework. We first focus on
sentiments expressed in a sentence, then identify the target aspect and
opinion terms for that sentiment. This takes into account the mutual
interactions among the triplet’s components while improving exploration and
sample efficiency. Furthermore, this hierarchical RL setup enables us to deal
with multiple and overlapping triplets. In our experiments, we evaluate our
model on existing datasets from laptop and restaurant domains and show that it
achieves state-of-the-art performance. The implementation of this work is
publicly available at https://github.com/declare-lab/ASTE-RL.

††journalyear: 2021††copyright: acmcopyright††conference: Proceedings of the
30th ACM International Conference on Information and Knowledge Management;
November 1–5, 2021; Virtual Event, Australia.††booktitle: Proceedings of the
30th ACM Int’l Conf. on Information and Knowledge Management (CIKM ’21),
November 1–5, 2021, Virtual Event, Australia††price: 15.00††isbn:
978-1-4503-8446-9/21/11††doi: 10.1145/XXXXXX.XXXXXX

##  1\. Introduction

Aspect-based sentiment analysis (ABSA) or target-based sentiment analysis
(TBSA) is an important research area in natural language processing (NLP)
(Poria et al., 2020; Duan et al., 2021). It consists of various fine-grained
sentiment analysis tasks (Nasukawa and Yi, 2003; Liu et al., 2010; Liu, 2012),
with the three most fundamental being aspect/target term extraction, opinion
term extraction and aspect/target term sentiment classification. Aspect
Sentiment Triplet Extraction (ASTE) is a relatively new subtask of ABSA
introduced by Li et al. (2019a); Peng et al. (2020). In ASTE, the task is to
extract triplets containing aspect terms, their associated sentiment
polarities, and the opinion terms that express those sentiments. A sentence
may contain multiple such triplets where the aspect terms or opinion terms
across triplets may overlap with each other. We include an example of such
triplets in Table 1.

Table 1. An Example of ASTE Triplets Present in a Sentence

Sentence |  | Appetizers are excellent ; you can make a great
---
( but slightly expensive ) meal out of them .
Triplets |  | [Aspect ; Opinion ; Sentiment]
---
(1) Appetizer ; excellent ; positive
(2) meal ; great ; positive
(3) meal ; slightly expensive ; negative

Existing methods, such as, CMLA+ (Wang et al., 2017), RINANTE+ (Dai and Song,
2019), Li-unified-R (Li et al., 2019a), WhatHowWhy (Peng et al., 2020), OTE-
MTL (Zhang et al., 2020), GTS (Wu et al., 2020), JET (Xu et al., 2020), TOP
(Huang et al., 2021) and BMRC (Chen et al., 2021) are mainly divided into
simultaneous and sequential methods. Early works (Wang et al., 2017; Dai and
Song, 2019; Li et al., 2019a; Peng et al., 2020; Zhang et al., 2020; Wu et
al., 2020) usually employ a two-staged approach where they simultaneously
extract aspect terms with sentiments and opinion terms. These triplets are
subsequently decoded through triplet classification or pairwise matching.
Recent works (Huang et al., 2021; Chen et al., 2021) have shifted towards a
more multi-stage, restrictive and sequential process during the extraction
stage that can potentially capture more mutual dependencies and correlations
among the triplet’s components while forgoing the triplet decoding stage.

In this work, we tackle the ASTE task using a novel paradigm ASTE-RL where we
consider the aspect and opinion terms as arguments of the sentiments expressed
in a sentence. Previous approaches usually simultaneously extract all three
components or first identify the aspect and opinion terms, then pair them up
to predict their sentiment polarities. Unlike previous approaches, we propose
a hierarchical reinforcement learning (RL) framework (Takanobu et al., 2019;
Duan et al., 2020) where we first consider the sentiment polarities, then
identify their associated opinion and aspect terms using separate RL
processes. This process is repeated to extract all triplets present in a
sentence. With this hierarchical RL setup, the model handle multiple triplets
and overlapping triplets, and model interactions between the three components
effectively. Inspired by the recent success of the multi-turn machine reading
comprehension (MRC) framework (Li et al., 2019b; Chen et al., 2021), we
incorporate ideas to further improve mutual interactions.

##  2\. Proposed Framework

###  2.1. Overview

We divide our framework ASTE-RL into three components: 1) aspect-oriented
sentiment classification, 2) opinion term extraction and 3) aspect term
extraction. For the sentiment classification component, the sentiment is
expressed towards the aspect term, and has four possible labels:
$L^{C}=\\{\mathrm{none,positive,negative,neutral}\\}$. Our opinion and aspect
extraction components are sequence labeling models with a BIO tagging scheme
(Ramshaw and Marcus, 1999). With this BIO scheme, we have three different
labels to tag an input sequence for the opinion/aspect terms:
$L^{O,A}=\\{\mathrm{B,I,O}\\}$. For a given sentence
$X=\\{x_{1},x_{2},...,x_{J}\\}$ with $J$ tokens, ASTE-RL aims to output a set
of labels $\\{(Y^{O},Y^{A},C)^{|K|}\\}$ where $|K|$ is the number of labels,
$Y^{O}=\\{y_{1}^{O},y_{2}^{O},...,y_{J}^{O}\\}$ represents the tagging labels
for the opinion term in a predicted triplet,
$Y^{A}=\\{y_{1}^{A},y_{2}^{A},...,y_{J}^{A}\\}$ represents the tagging labels
for the aspect term, and $C=\\{c\\}$ represents the sentiment polarity.

The three components are structured in a two-level hierarchy (Takanobu et al.,
2019). In the higher level, we have the sentiment indicator. During the
sequential scan of a sentence, an agent will decide at each position $j$ in a
sentence at the token $x_{j}$ if it has gathered sufficient information to
mark the position as indicative of a sentiment that is expressed towards an
aspect term. If not, the agent will mark it as $\mathrm{none}$. Otherwise, it
will mark it as either $\mathrm{positive,negative}$ or $\mathrm{neutral}$. In
the latter case, the agent launches two subtasks in the lower level for the
opinion and aspect extractions to identify the terms as arguments of the
sentiment and engages in sequence labeling. Upon completion, the agent will
return to the high-level sentiment indication process and continue the
sequential scan of the sentence. This process is well-suited to be formulated
as a semi-Markov decision process (Sutton et al., 1999b): 1) a high-level RL
process that detects a sentiment indicator in a sentence; 2) two low-level RL
processes that identify the opinion and aspect terms separately for the
corresponding sentiment.

###  2.2. Aspect-Oriented Sentiment Classification with High-Level RL

The high-level RL policy $\pi_{u}$ aims to detect the aspect-oriented
sentiments in a sentence. This can be seen as a RL policy over options, where
options are high-level actions (Sutton et al., 1999b).

Option: The option $\mathrm{\textbf{o}}_{t}$ is selected from
$L^{C}=\\{\mathrm{none,positive,}$ $\mathrm{negative,neutral}\\}$ where
$\mathrm{none}$ indicates no sentiment indicated towards any aspect term.

State: The state $\mathrm{\textbf{s}}_{t}^{u}$ at each time step $t$ is
represented by: 1) the current hidden state $\mathrm{\textbf{h}}_{t}$, 2) the
current part-of-speech (POS) tag $\mathrm{\textbf{p}}_{t}$, 3) the sentiment
polarity vector $\mathrm{\textbf{v}}_{t}^{c}$, and 4) the high-level state for
the previous time step $\mathrm{\textbf{s}}_{t-1}^{u}$ . To obtain
$\mathrm{\textbf{p}}_{t}$ for each token in a sentence, we pass the sentence
into the spaCy (https://spacy.io/) POS tagger. The sentiment polarity vector
$\mathrm{\textbf{v}}_{t}^{c}$ is the embedding of the latest option
$\mathrm{\textbf{o}}_{t^{\prime}}$ where
$\mathrm{\textbf{o}}_{t^{\prime}}\neq\mathrm{none}$. Both the POS tag and
sentiment embeddings are learned parameters in the model. Hence, the state
$\mathrm{\textbf{s}}_{t}^{u}$ is formally represented by:

(1)
$\mathrm{\textbf{s}}_{t}^{u}=f^{u}_{s}(\mathrm{\textbf{W}}_{s}^{u}[\mathrm{\textbf{h}}_{t};\mathrm{\textbf{p}}_{t};\mathrm{\textbf{v}}_{t}^{c};\mathrm{\textbf{s}}_{t-1}^{u}]),$

where $f^{u}_{s}(\cdot)$ is a non-linear function implemented by a MLP. The
hidden state $\mathrm{\textbf{h}}_{t}$ is obtained from a pre-trained BERT
model (Devlin et al., 2018) with Whole Word Masking and fine-tuned on the
SQuAD v1.1 training set (Rajpurkar et al., 2016). Specifically, we first
combine the query ” _Which tokens indicate sentiments relating pairs of aspect
spans and opinion spans?_ ” and the review sentence $X$ into the BERT
tokenizer to get a final input
$I=\\{[CLS],q^{u}_{1},...,q^{u}_{|q|},[SEP],x_{1},x_{2},...,x_{J}\\}$. We then
pass this input into the BERT model, and $\mathrm{\textbf{h}}_{t}$ represents
the output vector from the BERT model that corresponds to the token $x_{t}$.
The initial state $\mathrm{\textbf{s}}_{0}^{u}$ is initialized as:
$\mathrm{\textbf{s}}_{0}^{u}=\textbf{0}$.

Policy: The stochastic policy for sentiment detection $\pi_{u}$ specifies a
probability distribution over the options:

(2)
$\mathrm{\textbf{o}}_{t}\sim\pi_{u}(\mathrm{\textbf{o}}_{t}|\mathrm{\textbf{s}}_{t}^{u})=softmax(\mathrm{\textbf{W}}_{\pi}^{u}\mathrm{\textbf{s}}_{t}^{u}).$

Reward: At every time step, when $\mathrm{\textbf{o}}_{t}$ is executed, the
intermediate reward $\mathrm{\textbf{r}}_{t}^{u}$ provided by the environment
follows this:

(3) $\mathrm{\textbf{r}}_{t}^{u}=\begin{cases}1,\quad\ \ \ \mathrm{if\
\textbf{o}_{t}\ in}\ X\\\ 0,\quad\ \ \ \mathrm{if\
\textbf{o}_{t}}=\mathrm{none}\\\ -1,\quad\mathrm{if\ \textbf{o}_{t}\ not\ in}\
X.\end{cases}$

If a sentiment that is expressed towards an aspect term is detected at a time
step (i.e. $\mathrm{\textbf{o}}_{t}\neq\mathrm{none}$), the agent will launch
two subtasks as low-level RL processes. When the subtasks are completed, the
agent will return to the high-level RL process. Otherwise, the agent continues
its sequential scan of $X$ until the last option $\mathrm{\textbf{o}}_{J}$
about the last word $x_{J}$ of $X$ is sampled. When all options are sampled
(i.e. at the end of the combined hierarchical RL process), there is a final
reward $\mathrm{\textbf{r}}^{u}_{final}$ for the high-level process:
$\mathrm{\textbf{r}}^{u}_{final}=F_{1}(X)$ where $F_{1}$ is the harmonic mean
of the precision and recall in terms of the sentiment(s) in a sentence $X$.

###  2.3. Opinion and Aspect Extractions with Low-Level RL

Every time the high-level policy detects an aspect-oriented sentiment, two
low-level policies $\pi_{l}^{O},\pi_{l}^{A}$ will extract the corresponding
opinion and aspect terms respectively and separately for the sentiment. In
this subsection, we will generalize the RL elements such that they apply for
both low-level RL processes, unless otherwise stated.

Action: The action at every time step $t$ is to assign a tag to the current
word. The action $\mathrm{\textbf{a}}_{t}$ is selected from
$L^{O,A}=\\{\mathrm{B,I,O}\\}$, following a BIO tagging scheme. The
$\mathrm{B/I}$ symbols represent the beginning and inside of an opinion/aspect
term respectively, while the $\mathrm{O}$ symbol represents the unmarked
label.

State: Similar to the high-level policy, the state
$\mathrm{\textbf{s}}_{t}^{l}$ at each time step $t$ is represented by: 1) the
current hidden state $\mathrm{\textbf{h}}_{t}$, 2) the current POS tag
$\mathrm{\textbf{p}}_{t}$, 3) the opinion/aspect tag vector
$\mathrm{\textbf{v}}_{t}^{O,A}$, 4) the low-level state for the previous time
step $\mathrm{\textbf{s}}_{t-1}^{l}$. To enhance the interactions between the
sentiment and its associated opinion/aspect terms, we add a context vector
$\mathrm{\textbf{v}}_{t^{\prime}}^{ctx}$ to the state
$\mathrm{\textbf{s}}_{t}^{l}$ at each time step $t$, using the sentiment state
representation assigned to the latest option
$\mathrm{\textbf{o}}_{t^{\prime}}$:

(4)
$\mathrm{\textbf{v}}_{t^{\prime}}^{ctx}=g^{ctx}(\mathrm{\textbf{W}}^{ctx}_{v_{t^{\prime}}}\mathrm{\textbf{s}}_{t^{\prime}}^{u}).$

We also add the output vector from the BERT model for the [CLS] token, while
computing the output vectors for the hidden states. Hence, the state
$\mathrm{\textbf{s}}_{t}^{l}$ is formally represented by:

(5)
$\mathrm{\textbf{s}}_{t}^{l}=f^{l}_{s}(\mathrm{\textbf{W}}_{s}^{l}[\mathrm{\textbf{h}}_{t};\mathrm{\textbf{p}}_{t};\mathrm{\textbf{v}}_{t}^{O,A};\mathrm{\textbf{s}}_{t-1}^{l};\mathrm{\textbf{v}}_{t^{\prime}}^{ctx};\mathrm{\textbf{h}}_{t}^{CLS}]).$

Note that the representations used to compute the first low-level states for
the opinion and aspect extractions are different. The representation used to
compute the first low-level state for opinion term extraction
$\mathrm{\textbf{s}}_{1}^{l,O}$ is initialized using
$\mathrm{\textbf{s}}^{u}_{t^{\prime}}$ as:
$\mathrm{\textbf{s}}_{0}^{l,O}=g^{l}(\mathrm{\textbf{W}}^{l}_{s_{0}}\mathrm{\textbf{s}}_{t^{\prime}}^{u})$.
The representation used to compute the first low-level state for aspect term
extraction $\mathrm{\textbf{s}}_{1}^{l,A}$ is initialized using
$\mathrm{\textbf{s}}_{J}^{l,O}$ as:
$\mathrm{\textbf{s}}_{0}^{l,A}=\mathrm{\textbf{s}}_{J}^{l,O}$. These
initializations help us capture interactions between the triplet’s components.
$f^{l}_{s}(\cdot)$ is a non-linear function implemented by a MLP, while
$g^{ctx}(\cdot)$ and $g^{l}(\cdot)$ are linear functions that are implemented
by a single linear layer. The hidden state $\mathrm{\textbf{h}}_{t}$ is
obtained in the same way as in the high-level RL process. However, the queries
are changed. The query is ” _What is the opinion span for the_
$\mathrm{\textbf{o}}_{t^{\prime}}$ _sentiment indicated at_
$x_{t^{\prime}}$_?_ ” for opinion term extraction and ” _What is the aspect
span for the_ $\mathrm{\textbf{o}}_{t^{\prime}}$ _sentiment indicated at_
$x_{t^{\prime}}$_?_ ” for aspect term extraction.

Policy: The stochastic policy for opinion/aspect extraction $\pi_{l}$
specifies a probability distribution over the actions given the low-level
state $\mathrm{\textbf{s}}_{t}^{l}$ and the high-level option
$\mathrm{\textbf{o}}_{t^{\prime}}$ that launches the current subtask:

(6)
$\mathrm{\textbf{a}}_{t}\sim\pi_{l}(\mathrm{\textbf{a}}_{t}|\mathrm{\textbf{s}}_{t}^{l};\mathrm{\textbf{o}}_{t^{\prime}})=softmax(\mathrm{\textbf{W}}_{\pi}^{l}[\mathrm{\textbf{o}}_{t^{\prime}}]\mathrm{\textbf{s}}_{t}^{l}).$

Reward: At every time step, when $\mathrm{\textbf{a}}_{t}$ is executed, the
intermediate reward $r^{l}_{t}$ is computed as the prediction error over the
gold labels:

(7) $r^{l}_{t}=\begin{cases}\lambda,\quad\ \ \ \ \ \ \mathrm{if\ tag}\
y_{t}\mathrm{\ is\ predicted\ correctly}\\\
-0.5,\quad\mathrm{otherwise},\end{cases}$

where $\lambda\geq 0$ and $\lambda$ depends on the aspect/opinion tag type.
This enables the model to learn a policy that emphasizes the prediction of B
and I tags and avoids only predicting O tags in a trivial manner. When all
actions are sampled, there is a final reward $r^{l}_{final}$ for the low-level
processes, represented by:

(8) $r^{l}_{final}=\begin{cases}1,\quad\ \ \ \mathrm{if\ all\ tags\ are\
predicted\ correctly}\\\ -1,\quad\mathrm{otherwise}.\end{cases}$

There will also be negative rewards in the cases where the low-level processes
produce impossible predictions, namely cases where there are no or more than
one $\mathrm{B}$ tag present, and no or more than one opinion/aspect term
identified for each predicted triplet. Note that the low-level rewards are
non-zero only in the case where the option $\mathrm{\textbf{o}}_{t^{\prime}}$
from the high-level process is correctly predicted.

###  2.4. Hierarchical Policy Learning

We learn the high-level policy by maximizing the expected total reward at each
time step $t$ as the agent samples trajectories following the high-level
policies $\pi_{u}$. Likewise, we learn the low-level policies by maximizing
the expected total reward at each time step $t$ as the agent samples
trajectories following the low-level policies $\pi_{l}^{A},\pi_{l}^{O}$. We
then optimize all policies using policy gradient methods (Sutton et al.,
1999a) with the REINFORCE algorithm (Williams, 1992; Takanobu et al., 2019).

###  2.5. Training Procedure

We pre-train our ASTE-RL models for 40 epochs with a learning rate of 2e-5.
During pre-training, we give our model the ground-truth options or actions at
every time step to limit the exploration of the agent due to the high-
dimensional state space in our setup. This prevents the agent from exploring
too many unreasonable cases, e.g. an I tag preceding a B tag, and learning too
slowly. We then fine-tune the best model (chosen based on the Dev $F_{1}$
score) with RL policy for 15 epochs with a learning rate of 5e-6. We sample 5
trajectories for each data point during RL fine-tuning.

We initialize the BERT parameters from pre-trained weights (Devlin et al.,
2018) and update them during training for this task. We set the dimension of
sentiment polarity and opinion/aspect tag embeddings at 300. For POS
embeddings, we set the dimension at 25. We randomly initialize these
embeddings and update them during training. We set the state vector dimension
for $\mathrm{\textbf{s}}^{u}_{t}$ and $\mathrm{\textbf{s}}^{l}_{t}$ at 300. We
apply dropout (Srivastava et al., 2014) after the non-linear activations in
$f^{u}_{s}(\cdot)$ and $f^{l}_{s}(\cdot)$ during training and set the dropout
rate at 0.5. We train our models in mini-batches of size 16 and optimize the
model parameters using the Adam optimizer (Kingma and Ba, 2014).

##  3\. Experiments and Analysis

###  3.1. Datasets & Evaluation Metrics

Table 2. ASTE-Data-V2 Dataset Statistics

| 14Lap | 14Rest | 15Rest | 16Rest
---|---|---|---|---
| Train | Dev | Test | Train | Dev | Test | Train | Dev | Test | Train | Dev | Test
#sentence | 906 | 219 | 328 | 1266 | 310 | 492 | 605 | 148 | 322 | 857 | 210 | 326
#positive | 817 | 169 | 364 | 1692 | 404 | 773 | 783 | 185 | 317 | 1015 | 252 | 407
#negative | 517 | 141 | 116 | 480 | 119 | 155 | 205 | 53 | 143 | 329 | 76 | 78
#neutral | 126 | 36 | 63 | 166 | 54 | 66 | 25 | 11 | 25 | 50 | 11 | 29

We use the ASTE-Data-V2 dataset111https://github.com/xuuuluuu/SemEval-Triplet-
data curated by Xu et al. (2020) to show the effectiveness of ASTE-RL in two
different domains of English reviews, namely the laptop and restaurant
domains. 14Rest, 15Rest, 16Rest are the datasets of the restaurant domain and
14Lap is of the laptop domain. We include the statistics of the four datasets
in ASTE-Data-V2 in Table 2, where #sentence represents the number of
sentences, and #positive, #negative, and #neutral represent the numbers of
triplets with positive, negative, and neutral sentiment polarities
respectively.

We process the sentences with BERT’s WordPiece tokenizer (Wu et al., 2016) to
make them work for ASTE-RL. Since the WordPiece tokenization may break down
the tokens in the original dataset into subwords, we need to align the
opinion/aspect term annotations and our BIO tagging scheme. We tag every token
that corresponds to the opinion/aspect term tokens in the original annotations
with $\mathrm{I}$, except for the first token, which we tag with $\mathrm{B}$.

We follow the evaluation metrics of Xu et al. (2020) for our experiments. An
extracted triplet is correct if the entire aspect term, opinion term, and
sentiment polarity match with a ground-truth triplet. We report precision,
recall and $\mathbf{F_{1}}$ score based on this.

Table 3. Results of ASTE-RL and Previous Methods on the ASTE-Data-V2 Dataset

| 14Lap | 14Rest | 15Rest | 16Rest
---|---|---|---|---
Model |  Dev $F_{1}$ | Prec | Rec | $F_{1}$ |  Dev $F_{1}$ | Prec | Rec | $F_{1}$ |  Dev $F_{1}$ | Prec | Rec | $F_{1}$ |  Dev $F_{1}$ | Prec | Rec | $F_{1}$
WhatHowWhy (Peng et al., 2020) | - | 37.38 | 50.38 | 42.87 | - | 43.24 | 63.66 | 51.46 | - | 48.07 | 57.51 | 52.32 | - | 46.96 | 64.24 | 54.21
OTE-MTL (Zhang et al., 2020) | - | 54.26 | 41.07 | 46.75 | - | 63.07 | 58.25 | 60.56 | - | 60.88 | 42.68 | 50.18 | - | 65.65 | 54.28 | 59.42
$\mathrm{GTS}_{\mathrm{BiLSTM}}$ (Wu et al., 2020) | - | 58.02 | 40.11 | 47.43 | - | 71.41 | 53.00 | 60.84 | - | 64.57 | 44.33 | 52.57 | - | 70.17 | 55.95 | 62.26
$\mathrm{JET}_{\mathrm{BiLSTM}}^{t}$ (Xu et al., 2020) | 48.26 | 54.84 | 34.44 | 42.31 | 53.14 | 66.76 | 49.09 | 56.58 | 55.06 | 59.77 | 42.27 | 49.52 | 58.45 | 63.59 | 50.97 | 56.59
$\mathrm{JET}_{\mathrm{BiLSTM}}^{o}$ (Xu et al., 2020) | 45.83 | 55.98 | 35.36 | 43.34 | 53.54 | 61.50 | 55.13 | 58.14 | 60.97 | 64.37 | 44.33 | 52.50 | 60.90 | 70.94 | 57.00 | 63.21
$\mathrm{GTS}_{\mathrm{BERT}}$ (Wu et al., 2020) | - | 57.12 | 53.42 | 55.21 | - | 71.76 | 59.09 | 64.81 | - | 54.71 | 55.05 | 54.88 | - | 65.89 | 66.27 | 66.08
$\mathrm{JET}_{\mathrm{BERT}}^{t}$ (Xu et al., 2020) | 50.40 | 53.53 | 43.28 | 47.86 | 56.00 | 63.44 | 54.12 | 58.41 | 59.86 | 68.20 | 42.89 | 52.66 | 60.67 | 65.28 | 51.95 | 57.85
$\mathrm{JET}_{\mathrm{BERT}}^{o}$ (Xu et al., 2020) | 48.84 | 55.39 | 47.33 | 51.04 | 56.89 | 70.56 | 55.94 | 62.40 | 64.78 | 64.45 | 51.96 | 57.53 | 63.75 | 70.42 | 58.37 | 63.83
TOP (Huang et al., 2021) | - | 57.84 | 59.33 | 58.58 | - | 63.59 | 73.44 | 68.16 | - | 54.53 | 63.30 | 58.59 | - | 63.57 | 71.98 | 67.52
BMRC (Chen et al., 2021) | 56.08 | 65.91 | 52.15 | 58.18 | 62.83 | 72.17 | 65.43 | 68.64 | 72.47 | 62.48 | 55.55 | 58.79 | 70.91 | 69.87 | 65.68 | 67.35
ASTE-RL | 58.14 | 64.80 | 54.99 | 59.50 | 64.40 | 70.60 | 68.65 | 69.61 | 74.01 | 65.45 | 60.29 | 62.72 | 72.11 | 67.21 | 69.69 | 68.41
\- Pre-training only | 57.35 | 62.00 | 55.84 | 58.73 | 64.50 | 69.70 | 69.23 | 69.47 | 72.84 | 63.31 | 61.61 | 62.44 | 71.50 | 64.76 | 70.74 | 67.57

###  3.2. Baselines

We compare the performance of ASTE-RL against the following baselines: (i)
WhatHowWhy: Peng et al. (2020) proposed a multi-layer LSTM neural architecture
for co-extraction of aspect terms with sentiments, and opinion terms, with a
Graph Convolutional Network (Kipf and Welling, 2016) component to capture
dependency information to enhance the co-extraction. (ii) OTE-MTL: Zhang et
al. (2020) proposed a multi-task learning framework to jointly extract aspect
and opinion terms while parsing word-level sentiment dependencies, before
conducting a triplet decoding process. We use results from Huang et al. (2021)
for OTE-MTL’s performance on ASTE-Data-V2. (iii) GTS: Wu et al. (2020)
proposed an end-to-end grid tagging framework and a grid inference strategy to
exploit mutual indication between opinion factors. We use results from Huang
et al. (2021) for GTS’ performance on ASTE-Data-V2, and report them for two
variants: bidirectional LSTM (BiLSTM) and BERT. (iv) JET: Xu et al. (2020)
proposed a position-aware tagging scheme for triplet extraction. They encode
information about sentiment polarities and distances between the start
position of aspect term and the opinion term’s start and end positions
($\mathrm{JET}^{t}$) or vice versa ($\mathrm{JET}^{o}$). We report the results
for two variants: BiLSTM and BERT. (v) TOP: Huang et al. (2021) proposed a
two-stage method to enhance correlations between aspect and opinion terms.
Aspect and opinion terms are first extracted with sequence labeling, and
artificial tags are added to each pair to establish correlation. A sentiment
polarity is then identified for each pair using the resulting representations.
(vi) BMRC: Chen et al. (2021) proposed a transformation of the ASTE task into
a multi-turn MRC task and a bidirectional MRC framework to address it. They
use non-restrictive, restrictive and sentiment classification queries in a
three-turn process to extract triplets. We train and test BMRC on ASTE-Data-V2
over 5 runs with different random seeds.

Table 4. $F_{1}$ Scores for Multiple and Overlapping Triplets

| 14Lap | 14Rest | 15Rest | 16Rest
---|---|---|---|---
| ASTE-RL | BMRC | ASTE-RL | BMRC | ASTE-RL | BMRC | ASTE-RL | BMRC
Single | 62.46 | 63.04 | 67.44 | 66.51 | 61.33 | 59.59 | 66.31 | 67.83
Multiple | 57.70 | 53.77 | 70.23 | 69.17 | 63.95 | 56.37 | 69.84 | 64.11
No Overlap | 62.24 | 62.80 | 72.16 | 70.81 | 62.17 | 59.75 | 70.13 | 69.23
Overlap | 55.94 | 50.18 | 67.23 | 63.96 | 63.83 | 55.48 | 65.20 | 58.59

###  3.3. Experimental Results

The experimental results are shown in Table 3. We observe that BERT-based
models (their results are in the row above ASTE-RL’s results in Table 3)
generally perform better than the non-BERT models. Hence, we only experiment
with BERT for our ASTE-RL model. We select our best model for each dataset
based on its Dev $F_{1}$ score. For reproducibility, we report the testing
results averaged over 5 runs with different random seeds. ASTE-RL outperforms
existing baselines on all four datasets, and significantly outperforms
existing baselines on the 15Rest dataset. When compared to the second-best
performance for each dataset, we observe an average improvement of 1.68%
$F_{1}$ score across all four datasets, and an improvement of 3.93% on 15Rest.
We also observe that our model strikes a balance between the TOP and BMRC
models in terms of precision and recall, and hypothesize that this balance can
be flexibly shifted depending on $\beta$ to fit dataset requirements, if we
generalize $\mathrm{\textbf{r}}^{u}_{final}=F_{\beta}(X)$, where
$F_{\beta}(X)$ is the weighted harmonic mean of precision and recall.

###  3.4. Effect of RL Fine-tuning

In Table 3, we report our results for ASTE-RL without the RL fine tuning step.
In this setting, we pre-train our ASTE-RL for 40 epochs as usual and after
that we run for another 15 epochs with a learning rate of 5e-6 (as used in RL
fine-tuning step). As compared to the RL fine-tuning setting with multinomial
sampling, this setting has lower $F_{1}$ scores with an average decrease of
0.51% over 5 runs with different random seeds. In this setting, our model
achieves slightly higher recall, but precision is significantly lower across
all four datasets. This might be because multinomial sampling encourages more
exploration after the initial pre-training of 40 epochs.

###  3.5. Analysis on Multiple & Overlapping Triplet Extraction

We show the results of ASTE-RL and BMRC in complex situations where there are
multiple and overlapping triplets in a sentence in Table 4. For the multiple
triplet scenario, we observe that there is a performance increase for 14Rest,
15Rest and 16Rest and a decrease for 14Lap as compared to the case where only
one triplet is present in a sentence. For the overlapping triplet scenario, we
observe a performance increase for for 15Rest and a decrease for 14Lap, 14Rest
and 16Rest.

In general, we observe that ASTE-RL can handle multiple and overlapping
triplets in a sentence consistently well due to its hierarchical RL setup, as
compared to BMRC. There is a total $F_{1}$ decrease of 4.76% for multiple
triplet extraction for ASTE-RL across all four datasets as compared to 16.21%
for BMRC, and a total $F_{1}$ decrease for overlapping triplet extraction of
16.16% for ASTE-RL as compared to 34.38% for BMRC.

##  4\. Conclusion

In this work, we propose a novel ASTE-RL model based on hierarchical
reinforcement learning (RL) paradigm for aspect sentiment triplet extraction
(ASTE). In this paradigm, we treat the aspect and opinion terms as arguments
of the sentiment polarities. We decompose the ASTE task into a hierarchy of
three subtasks: high-level sentiment polarity extraction, and low-level
opinion and aspect term extractions. This approach is good at modeling the
interactions between the three tasks and handling multiple and overlapping
triplets. We incorporate the multi-turn MRC elements in our model to further
improve these interactions. Our proposed model achieves state-of-the-art
performance on four challenging datasets for the ASTE task.

## Acknowledgments

This project is supported by the DSO grant no. RTDST190702 awarded to SUTD
titled Complex Question Answering.

## References

  * (1)
  * Chen et al. (2021) Shaowei Chen, Yu Wang, Jie Liu, and Yuelin Wang. 2021\.  Bidirectional Machine Reading Comprehension for Aspect Sentiment Triplet Extraction.  _arXiv preprint arXiv:2103.07665_ (2021).
  * Dai and Song (2019) Hongliang Dai and Yangqiu Song. 2019.  Neural Aspect and Opinion Term Extraction with Mined Rules as Weak Supervision. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_.  https://www.aclweb.org/anthology/P19-1520
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ (2018).
  * Duan et al. (2020) Jiafei Duan, Samson Yu, Hui Li Tan, and Cheston Tan. 2020\.  Actionet: An Interactive End-To-End Platform For Task-Based Data Collection And Augmentation In 3D Environment. In _2020 IEEE International Conference on Image Processing (ICIP)_. IEEE, 1566–1570.
  * Duan et al. (2021) Jiafei Duan, Samson Yu, Hui Li Tan, Hongyuan Zhu, and Cheston Tan. 2021.  A Survey of Embodied AI: From Simulators to Research Tasks.  _arXiv preprint arXiv:2103.04918_ (2021).
  * Huang et al. (2021) Lianzhe Huang, Peiyi Wang, Sujian Li, Tianyu Liu, Xiaodong Zhang, Zhicong Cheng, Dawei Yin, and Houfeng Wang. 2021\.  First Target and Opinion then Polarity: Enhancing Target-opinion Correlation for Aspect Sentiment Triplet Extraction.  _arXiv preprint arXiv:2102.08549_ (2021).
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ (2014).
  * Kipf and Welling (2016) Thomas N Kipf and Max Welling. 2016.  Semi-supervised classification with graph convolutional networks.  _arXiv preprint arXiv:1609.02907_ (2016).
  * Li et al. (2019a) Xin Li, Lidong Bing, Piji Li, and Wai Lam. 2019a.  A Unified Model for Opinion Target Extraction and Target Sentiment Prediction. In _The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI_.  https://doi.org/10.1609/aaai.v33i01.33016714
  * Li et al. (2019b) Xiaoya Li, Fan Yin, Zijun Sun, Xiayu Li, Arianna Yuan, Duo Chai, Mingxin Zhou, and Jiwei Li. 2019b.  Entity-Relation Extraction as Multi-Turn Question Answering. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Florence, Italy, 1340–1350.  https://doi.org/10.18653/v1/P19-1129
  * Liu (2012) Bing Liu. 2012.  Sentiment analysis and opinion mining.  _Synthesis lectures on human language technologies_ (2012).
  * Liu et al. (2010) Bing Liu et al. 2010\.  Sentiment analysis and subjectivity.  _Handbook of natural language processing_ (2010).
  * Nasukawa and Yi (2003) Tetsuya Nasukawa and Jeonghee Yi. 2003.  Sentiment analysis: Capturing favorability using natural language processing. In _Proceedings of the 2nd international conference on Knowledge capture_.
  * Peng et al. (2020) Haiyun Peng, Lu Xu, Lidong Bing, Fei Huang, Wei Lu, and Luo Si. 2020\.  Knowing what, how and why: A near complete solution for aspect-based sentiment analysis. In _Proceedings of the AAAI Conference on Artificial Intelligence_.  https://aaai.org/ojs/index.php/AAAI/article/view/6383
  * Poria et al. (2020) S. Poria, D. Hazarika, N. Majumder, and R. Mihalcea. 2020\.  Beneath the Tip of the Iceberg: Current Challenges and New Directions in Sentiment Analysis Research.  _IEEE Transactions on Affective Computing_ 01 (nov 2020), 1–29.  https://doi.org/10.1109/TAFFC.2020.3038167
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  Squad: 100,000+ questions for machine comprehension of text.  _arXiv preprint arXiv:1606.05250_ (2016).
  * Ramshaw and Marcus (1999) Lance A Ramshaw and Mitchell P Marcus. 1999.  Text chunking using transformation-based learning.  In _Natural language processing using very large corpora_. Springer, 157–176.
  * Srivastava et al. (2014) Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014.  Dropout: a simple way to prevent neural networks from overfitting.  _The journal of machine learning research_ 15, 1 (2014), 1929–1958.
  * Sutton et al. (1999a) Richard S Sutton, David A McAllester, Satinder P Singh, Yishay Mansour, et al. 1999a.  Policy gradient methods for reinforcement learning with function approximation.. In _NIPs_ , Vol. 99. Citeseer, 1057–1063.
  * Sutton et al. (1999b) Richard S Sutton, Doina Precup, and Satinder Singh. 1999b.  Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning.  _Artificial intelligence_ (1999).
  * Takanobu et al. (2019) Ryuichi Takanobu, Tianyang Zhang, Jiexi Liu, and Minlie Huang. 2019\.  A hierarchical framework for relation extraction with reinforcement learning. In _Proceedings of the AAAI Conference on Artificial Intelligence_.  https://doi.org/10.1609/aaai.v33i01.33017072
  * Wang et al. (2017) Wenya Wang, Sinno Jialin Pan, Daniel Dahlmeier, and Xiaokui Xiao. 2017.  Coupled Multi-Layer Attentions for Co-Extraction of Aspect and Opinion Terms. In _Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence_.  http://aaai.org/ocs/index.php/AAAI/AAAI17/paper/view/14441
  * Williams (1992) Ronald J Williams. 1992\.  Simple statistical gradient-following algorithms for connectionist reinforcement learning.  _Machine learning_ 8, 3-4 (1992), 229–256.
  * Wu et al. (2016) Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016\.  Google’s neural machine translation system: Bridging the gap between human and machine translation.  _arXiv preprint arXiv:1609.08144_ (2016).
  * Wu et al. (2020) Zhen Wu, Chengcan Ying, Fei Zhao, Zhifang Fan, Xinyu Dai, and Rui Xia. 2020\.  Grid Tagging Scheme for Aspect-oriented Fine-grained Opinion Extraction.  _arXiv preprint arXiv:2010.04640_ (2020).
  * Xu et al. (2020) Lu Xu, Hao Li, Wei Lu, and Lidong Bing. 2020\.  Position-Aware Tagging for Aspect Sentiment Triplet Extraction. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing_.  https://doi.org/10.18653/v1/2020.emnlp-main.183
  * Zhang et al. (2020) Chen Zhang, Qiuchi Li, Dawei Song, and Benyou Wang. 2020\.  A Multi-task Learning Framework for Opinion Triplet Extraction.  _arXiv preprint arXiv:2010.01512_ (2020).


# Stochastic Economic Dispatch Considering Demand Response and Endogenous
Uncertainty

Nasrin Bayat, Qifeng Li  Dept. of Electrical and Computer Engineering
University of Central Florida
Orlando, USA
<EMAIL_ADDRESS>Joon-Hyuk Park  Dept. of Mechanical and Aerospace
Engineering
University of Central Florida
Orlando, USA
<EMAIL_ADDRESS>
###### Abstract

This paper considers endogenous uncertainty (EnU) in the stochastic economic
dispatch (SED) problem, where the endogenous uncertainty means decision
dependent uncertainty. In this problem, demand response (DR) commitment is the
source of the EnU. Nevertheless, EnU is not well considered in existing
literature. Our first contribution is to build up an optimization model of DR-
involved SED under EnU (SED-DR-EnU). This is a computational challenging
problem due to the EnU. Our second contribution is introducing a coupled
learning enabled optimization algorithm which can effectively solve the
proposed SED-DR-EnU problem. This strategy is tested on the IEEE 14 bus, and
IEEE 39 bus systems, and the results showed the importance of considering EnU
in the DR-involved SED problem.

###### Index Terms:

Decision dependent uncertainty, Endogenous uncertainty, Stochastic economic
dispatch, Exogenous uncertainty, Demand response

##  I Introduction

Economic dispatch (ED) is the process of allocating output (active and
reactive power) among available generation at the lowest cost and meeting the
operational restrictions of the generation and the transmission system [1].
Demand response (DR) is a tariff or program that aims to persuade end-user
consumers to change their electricity consumption in response to changes in
the cost of electricity over time, or to offer incentive payments designed to
persuade consumers to use less electricity during times of high market prices
or when the stability of the grid is in danger [2].

In reality, several variables, including inconvenience costs, incentive
policies, and level of education, may affect how responsive consumers are. We
must consider how dependent the consumers’ uncertain behavior is on these
elements. The assumption that consumers’ demand responsiveness or demand
response commitment, is a constant that can be comprehended or perfectly
projected by the utility company in advance, is made in the majority of recent
publications [3], [4], [5], [6]. consumers may have different consumption
patterns and preferences, making it challenging for utility companies to
understand each DR consumer’s unique characteristics in real-world settings.
Additionally, even if such data were available, other factors, such as unusual
events, might still affect people’s responsiveness. A deterministic model
would find it difficult to forecast the effects of these factors. As a result,
while the system is in operation, the utility companies should be completely
unaware of the system’s actual DR capacity. However, while plans were being
created, these DR ambiguity issues were rarely taken into consideration.

Exogenous uncertainty or decision-independent uncertainty is the term used in
practice to describe an uncertain variable that follows a particular
statistical regularity that may be known in advance and does not change over
time. On the other hand, endogenous uncertainty (EnU) or decision-dependent
uncertainty refers to uncertain variables whose probability distribution is
not known and is affected by the decision variables. Neglecting EnU in DR
modeling could lead utility companies to make poor system strategic planning
decisions by vastly overestimating the capacity value of DR resources.

In the prior works, the majority of the DR uncertainties were modeled as fixed
probability distributions that could be fully described before the evaluation.
In other words, existing models have considered consumer demand responsiveness
as an exogenous uncertain quantity that is unrelated to the operator’s choice
of control method. In practice, consumer participation in DR might benefit
them financially but adversely affect their comfort and accessibility of
electricity service [7]. Ineffective scheduling strategies could reduce the
level of compliance of the consumers with DR calls [8]. As a result, the
current distribution of consumer responsiveness is influenced by grid
operating decisions. DR presents uncertainty and is dynamic in nature. Thus,
underestimating these aspects is likely to yield unreliable utilization of DR-
based economic dispatch (ED) [9]. For example, the operational and control
tactics may impact some stochastic features (such as response probability and
temperature preference); however, these kinds of uncertainties, which are
called EnU are disregarded and considered to be static and known probability
distributions [10].

Stochastic programming (SP) methods use the discretization process with
scenarios of uncertainty without knowing the precise probability distribution
of the uncertainty. Scenarios are frequently regarded as a priori in SP
approaches when the uncertainty is decision independent, which is fair because
they may be simulated using either an exogenous prediction model or from a
historical data collection [11]. Decisions taken before the realization of
uncertainty affect EnU. By altering its parameters or information structures,
decisions might have an impact on uncertainty. A few examples include
operating budgets, equipment switch status, and investment plans for renewable
energy sources (RES). End-consumers must adapt their loads when the demand
response set point changes. Some examples include reducing PV output,
switching the state of domestic appliances, and rescheduling electric vehicle
charging. Decision-related uncertainty loads can change as a result of
amplification or reduction of uncertainty sources [12].

In this paper, the decision variable influences the probability distribution
of the random variable. In addition, the learning step of predicting the
latent dependency and the optimization step of computing a candidate decision
are carried out interactively. The contributions of this work are as follows:

  * •

Considering both the exogenous and endogenous uncertainties of Demand Response
in the stochastic economic dispatch (SED-DR-EnU) problem, which are a result
of demand-side physical factors and the behavior of consumers.

  * •

A coupled learning enabled optimization (CLEO) algorithm [14] is introduced to
effectively solve the DR problem under both exogenous and endogenous
uncertainties without making any assumptions about the probability
distribution of decision-dependent uncertainty.

  * •

Comparative analysis to demonstrate how the demand-side participation
uncertainty may have a significant impact on the effectiveness of DR programs.

##  II Optimization Model of SED-DR-EnU

This section describes the optimization model we developed for the stochastic
economic dispatch considering demand response under both exogenous and
endogenous uncertainties in detail. Incorporating demand response commitments
$P^{rd}_{j,max}$ from end-consumers utilizing a variety of incentives, a
demand response provider (DRP) aggregates a group of residential/commercial
consumers and sets its offer price $\pi^{DR}_{j}$ in the day-ahead market. The
cost function of the unit i is presented as:

$C_{i}(P_{i}^{G})=\sum_{i=1}^{N_{G}}\left(a_{i}{P_{i}^{G}}^{2}+b_{i}P^{G}_{i}\right)$
(1)

Where, $a_{i}$ and $b_{i}$ are coefficients of the cost function and
$P_{i}^{G}$ is power generation of unit i. Each RES output’s uncertainty
interval is given by $\zeta_{i}\in\left[-P_{Ri},P_{Ri}\right]\times(r/100)$,
where $P_{Ri}$ is the $i_{th}$ RES nominal output and $r$ determines the
uncertainty level of $P_{Ri}$. It is assumed that the predicted RES generation
is such that the average value of its uncertainty is zero. The parameter $r$
shows the confidence interval for the RES output forecast error; that means,
the closer $r$ is to 100, the power forecast would be the less accurate. The
power generation is made up of the nominal conditions output $P^{G}_{i(\text{
base })}$, and the shift in generation brought on by RES output changes. Let
$\alpha_{i}$ stand for the unit $i$’s participation factor, which represents
the contribution of generator $i$ to the uncertain component of total load,
i.e., the difference between the real and forecasted total load. It is subject
to the constraints (2) [15].

$\alpha_{i}=\frac{1/a_{i}}{\sum_{j=1}^{N_{G}}1/a_{j}},\quad
0\leq\alpha_{i}\leq 1,\quad\sum_{j=1}^{N_{G}}\alpha_{i}=1,\quad
i=1,\ldots,N_{G}$

(2)

Then $P^{G}_{i}$ is calculated as:

$P^{G}_{i}=P^{G}_{i(\text{ base
})}-\alpha_{i}\times\sum_{j=1}^{N_{R}}\zeta_{j},\quad i=1,\ldots,N_{G}$ (3)

$N_{G}$ is the total number of generators and $N_{R}$ is the number of
available RES. If $\lambda$ is covariance matrix of the predicted RES
generation uncertainty and the expected prediction error of the uncertainty is
zero, the mathematical formulation of the expectation of supply-side cost,
which is the sum of the generation costs, and the cost of DRPs will be as
follows:

$\displaystyle\min\quad
f\left(P^{rd}\right)=\mathbb{E}[C_{i}(P_{i}^{G})+\sum_{i=1}^{N_{G}}a_{i}^{\prime}\alpha_{i}^{2}+\sum_{j=1}^{N_{DRP}}P_{j,EnU}^{rd}\pi_{j}^{DR}]$
(4)

Where

$a_{i}^{\prime}=\left[\sum_{j=1}^{N_{R}}\sum_{k=1}^{N_{R}}\lambda_{(j,k)}\right]\times
a_{i},i=1,\ldots,N_{G}$ (5) $P_{j,EnU}^{rd}=h(P_{j}^{rd},\varepsilon_{j})$ (6)

where $P_{j}^{rd}$ is the decision variable which represents the accepted DR
commitment of the j-th DRP, $\varepsilon_{j}$ is a random variable that is
independent of $P_{j}^{rd}$. $P_{j,EnU}^{rd}$ is the actual DR commitment
considering EnU that shows the responsiveness of consumers and is a function
of the independent system operator’s (ISO) decisions (accepted DR
commitments). $\pi^{DR}_{j}$ is the j-th DRP’s offer price, and $N_{DRP}$ is
the number of DRPs for a group of consumers. Subject to the following
constraints:
1- Energy adequacy constraint :

$\sum_{i=1}^{N_{G}}P^{G}_{i}+\sum_{i=1}^{N_{R}}P^{R}_{i}+\sum_{j=1}^{N_{DRP}}P_{j,EnU}^{rd}>\sum_{i=1}^{N_{B}}P^{L}_{i}$

(7)

2- Power flow limit:

$L\times\left(I_{1}\sum_{i=1}^{N_{G}}P^{G}_{i}+I_{2}\sum_{j=1}^{N_{DRP}}P_{j,EnU}^{rd}+I_{3}\sum_{i=1}^{N_{R}}\left(P^{R}_{i}+\zeta_{i}\right)-I_{4}\sum_{i=1}^{N_{B}}P^{L}_{i}\right)\leq
F_{l}^{\max}$

(8)

where $L$ is an $N_{L}\times\left(N_{B}-1\right)$ matrix of power transfer
distribution factors. $N_{B}$ is the number of buses and $N_{L}$ is the number
of lines. $P^{L}_{i}$ represents the load at bus $i$. $\epsilon$ indicates the
upper limit of violation probability in (7). $F_{l}^{\max}$ is the maximum
line power flow vector. $I_{1},I_{2},I_{3}$, and $I_{4}$ are matrices that
respectively map the vectors $P^{G},P^{rd},P^{R}$, $\zeta$, and $P^{L}$ into
$\left(N_{B}-1\right)\times 1$ vectors, with the nonzero elements denoting
connections to one of the $N_{B}$ buses other than the slack bus.
3- Power generation limit:

$P^{G}_{i,min}\leq P^{G}_{i}\leq P^{G}_{i,max},i=1,2,\ldots,N_{G}$ (9)

4- DR commitment limit:

$0\leq P_{\mathrm{j}}^{\mathrm{rd}}\leq
P_{\mathrm{j},max}^{\mathrm{rd}},j=1,2,\ldots,N_{DRP}$ (10)

Assuming a linear aggregated demand curve, $P_{\mathrm{j},max}^{\mathrm{rd}}$
can be calculated as

$P_{\mathrm{j},max}^{\mathrm{rd}}=\min\left(P_{\text{base
},j},\frac{\pi_{s,j}}{\pi_{j}^{\max}-\pi^{RR}}P_{\text{base },j}\right)$ (11)

where $\pi_{s,j}$ is DRP’s incentive price to end-consumers, $P_{base}$ is the
baseline for end-consumers, $\pi_{j}^{max}$ is the demand curve’s Y-axis
intercept. $\pi^{RR}$ is the retail price. This problem is very hard to solve,
because $P_{j,EnU}^{rd}$ and the distribution of the random variable
$\varepsilon$ are unknowns before optimization. In addition because
realizations of $\varepsilon$ cannot be observed, traditional SP techniques
cannot be used. Data pairs of decisions and uncertainties should be gathered
together to understand the link between uncertainty and the decision variable
in order to successfully address such SP problems. This paper introduces a
coupled learning enabled optimization (CLEO) algorithm [14] to solve this
problem.

##  III Solution Method

This section is devoted to providing a detailed description of the presented
framework. In this work, it is considered that the responsiveness of consumers
is a function of the independent system operator’s (ISO) decisions as below:

$P_{j,EnU}^{rd}=\psi(P_{j}^{rd})+\varepsilon$ (12)
$\psi(P_{j}^{rd})=\left(A^{1}\right)^{\top}P_{j}^{rd}+\left(A^{0}\right)^{\top}$
(13)

The derivative-free trust region method is the foundation of the CLEO
algorithm, which is used to solve SED with latently decision-dependent
uncertainty. In the proposed solution process, the SED-DR-EnU problem (1)-(9)
is solved in two steps, learning and optimization. At each iteration, a region
around the current best solution is defined that is trusted to be a sufficient
approximation of the objective function and hence the model is called trust
region. Using local linear regression (LLR) centered at the current iteration,
the local latent dependency is predicted in the learning step. During the
optimization phase, a potential answer to the trust region subproblem is
sought, which is made up of the random LLR model. At the $k_{th}$ iteration,
the current point is $P_{j,k}^{rd}$, $\delta_{k}$ is the radius of the current
trust region and the data set is
$T_{k}=\left\\{\left(P_{j,i}^{rd},P_{j,i,EnU}^{rd}\right)\right\\}$ of size
$N_{k}$. The estimation of parameters $A^{1}$ and $A^{0}$ can be found by
minimizing the sum of square errors between the actual and acctepted DR
commitment.

$\displaystyle\left\\{\widehat{A}^{k,0},\widehat{A}^{k,1}\right\\}\in\underset{A^{0},A^{1}}{\operatorname{argmin}}\sum_{\left(P_{j,i}^{rd},P_{j,i,EnU}^{rd}\right)\in
T_{k}}$ (14)
$\displaystyle\left\|P_{j,i,EnU}^{rd}-\left(A^{1}\right)^{\top}P_{j,i}^{rd}-\left(A^{0}\right)^{\top}\right\|^{2}$

$\widehat{A}^{k,0}$ and $\widehat{A}^{k,1}$ are the estimation of parameters
$A^{0}$ and $A^{1}$ respectively. By subtracting the actual DR commitment from
the estimated one, the error $e^{k,i}$ can be calculated as shown in (15).

$e^{k,i}\leftarrow
P_{j,i,EnU}^{rd}-\left(\widehat{A}^{k,1}\right)^{\top}P_{j,i}^{rd}-\left(\widehat{A}^{k,0}\right)^{\top}\quad
i=1,2,..,N_{k}$ (15)

A random variable with the empirical probability distribution of
$\left\\{e^{k,i}\right\\}_{i=1}^{N_{k}}$ is denoted by the symbol
$\varepsilon^{k}$. The $\mathrm{LLR}$ model $m_{k}$ is constructed as follows.

$m_{k}\left(P_{k}^{rd}+s^{k},\varepsilon^{k}\right)\triangleq\left(\widehat{A}^{k,1}\right)^{\top}\left(P_{k}^{rd}+s^{k}\right)+\left(\widehat{A}^{k,0}\right)^{\top}+\varepsilon^{k}$
(16)

Where $s^{k}$ is the step which minimizes the objective function in the trust
region and satisfies a ”sufficient” decrease requirement. The mathematical
formulation of the DR problem under both exogenous and endogenous
uncertainties (1)-(8) will change to:

$\displaystyle\min\quad
f\left(P_{k}^{rd}+s^{k}\right)=\mathbb{E}[C_{k,i}(P_{k,i}^{G})+$ (17)
$\displaystyle\sum_{i=1}^{N_{G}}a_{k,i}^{\prime}\alpha_{k,i}^{2}+\sum_{j=1}^{N_{DRP}}m_{j,k}\left(P_{j,k}^{rd}+s^{k},\varepsilon^{j,k}\right)\pi^{DR}_{j,k}]$

subject to the following constraints:
1- Energy adequacy constraint: We reformulate the constraint (7) to (18) by
substituting $P_{j,EnU}^{rd}$ with
$m_{j,k}\left(P_{j,k}^{rd}+s^{k},\varepsilon^{j,k}\right)$.

$\sum_{i=1}^{N_{G}}P^{G}_{k,i}+\sum_{i=1}^{N_{R}}P^{R}_{k,i}+\sum_{j=1}^{N_{DRP}}m_{j,k}\left(P_{j,k}^{rd}+s^{k},\varepsilon^{j,k}\right)>\sum_{i=1}^{N_{B}}P^{L}_{i,k}$

(18)

2- Power flow limit:

$L\times\left(I_{1}\sum_{i=1}^{N_{G}}P^{G}_{k,i}+I_{2}\sum_{j=1}^{N_{DRP}}P^{rd}_{k,j}+I_{3}\sum_{i=1}^{N_{R}}\left(P^{R}_{k,i}+\zeta_{i}\right)-I_{4}\sum_{i=1}^{N_{B}}P^{L}_{i,k}\right)\leq
F_{l}^{\max}$

(19)

3- Power generation limit:

$P^{G}_{i,min}\leq P^{G}_{k,i}\leq P^{G}_{i,max},i=1,2,\ldots,N_{G}$ (20)

4- DR commitmen limit:

$0\leq P_{\mathrm{k},\mathrm{j}}^{\mathrm{rd}}\leq
P_{\mathrm{j},max}^{\mathrm{rd}},j=1,2,\ldots,N_{DRP}$ (21)

5-

$\lVert s\rVert\leq\delta_{k}$ (22)

Actual to predicted reduction ratio of the objective function is used to
determine whether $s_{k}$ is a descent step. The problem has a decision-
dependent uncertainty. This dependency is unknown. As a result, it is
impossible to compute the objective function’s true value. In this case, local
linear regression models can be used to estimate the value of the objective
function. Particularly, $u_{k}$ and $u_{k+1/2}$ are used to estimate the
values of the functions $f\left(p^{k}\right)$ and $f\left(p^{k}+s^{k}\right)$.

$\displaystyle u_{k}$
$\displaystyle\triangleq\left[f\left(p^{k},m_{k,1}\left(p^{k},\varepsilon^{k,1}\right)\right)\right]$
(23) $\displaystyle u_{k+1/2}$
$\displaystyle\triangleq\left[f\left(p^{k}+s^{k},m_{k,2}\left(p^{k}+s^{k},\varepsilon^{k,2}\right)\right)\right]$
(24)

The estimated ratio between actual and predicted decrease in objective
function is then roughly represented as

$\rho_{k}\triangleq\left(u_{k}-u_{k+1/2}\right)/\left(f_{k}\left(p^{k}\right)-f_{k}\left(p^{k}+s^{k}\right)\right)$
(25)

To determine if the new step $p^{k}+s^{k}$ will be approved or denied, the
ratio $\rho_{k}$ needs to be large enough. In addition, $\lVert
s\rVert\leq\delta_{k}$ should be satisfied and the generalized gradient and
the TR radius $\delta_{k}$ should be large enough in relation to one another.
After applying the two steps of the algorithm, cost and total demand response
commitment are predicted.

##  IV Results and Discussion

In this section, IEEE 14 bus and 39 bus power systems with DRPs and RESs are
examined to evaluate the performance of the proposed stochastic economic
dispatch model. The SED-DR-ENU formulations are coded in Python. Looking back
on (11), a linear aggregated inherent demand curve for end-consumers is
assumed, with baseline $Pbase=60$, retail price $\pi^{RR}=100$, the j-th DRP’s
offer price $\pi^{DR}_{j}=100$, and Y-axis intercept $\pi^{max}_{j}=400$. In
total, 3 cases are considered to analyze the effect of EnU on DR commitment,
as it is described below:
Case 1: Demand response considering both endogenous and exogenous
uncertainties (uncertainty of load and RES)
Case 2: Demand response without considering uncertainties.
Case 3: Demand response considering just exogenous uncertainties (uncertainty
of RES)

###  IV-A IEEE 14 bus system

This subsection analyzes a more complex IEEE 14-bus system that has two DRPs
and two RESs as shown in Fig.1. In order to lower the demand, it is supposed
that DRPs are located in buses 3 and 4 that have the highest loads
$(N_{DRP}=2)$. It is assumed that two RESs, each with a 40 MW capacity, are
connected to buses 2 and 3, with $r=20\%$ $(N_{R}=2)$. Fig. 2 shows the result
of the simulation for the three cases. The amount of required load reduction
for case 1 is higher than for cases 2 and 3 because case 1 is more accurate.
Based on the first row of Table I case 1 needs more DR commitment compared
with cases 2 and 3. This result represents the importance of considering EnU
in the SED problem.

Figure 1: IEEE 14 Bus system with 2 DRPs and 2 RESs Figure 2: IEEE 14 Bus
system results

###  IV-B IEEE 39 bus system

In this subsection, IEEE 39 bus system with two DRPs is studied as shown in
Fig. 3. Buses 20 and 39 are presumed to have DRPs $(N_{DRP}=2)$. RES
integration is simulated using three wind farms connected to buses 30, 34 and
37. Total contribution of wind farms is assumed to be around 20$\%$ of the
system’s total generating capacity, $r$ is equal to 20$\%$ and $N_{R}=3$. Fig.
4 represents the result of the simulation for the three cases. Required load
reduction for case 1 is higher than cases 2 and 3, this shows the inaccuracy
of the case 2 and 3. The second row of Table I compares the DR commitment
values for the 39 bus system.

Figure 3: IEEE 39 Bus system with 2 DRPs and 2 RESs Figure 4: IEEE 39 Bus
system results

Cases 2 and 3 represent the existing research where case 2 does not consider
any uncertainties and case 3 considers just exogenous uncertainties
(uncertainty of RES). Case 1 is the proposed scenario of this paper which is
closer to real-world situations since it considers both endogenous and
exogenous uncertainties (uncertainty of load and RES). Given this, the
simulation results shown in 2 and 4 indicate that cases 2 and 3 (i.e.,
existing research) are not accurate enough to represent the real-world. An
important conclusion we can make from the simulation results is that
endogenous uncertainty should be considered in the economic dispatch problems
which involve demand response. To evaluate the performance of the stochastic
ED model, the convergence curve of a deterministic ED which does not take into
account any uncertainties is compared with our model in Fig. 5. Both
approaches converge after about 210 iterations. In addition, a comparison
between stochastic and deterministic approaches is represented in Fig. 5.
Minimum objective function vlues are 2109.73 and 4890 for stochastic and
deterministic algorithms respectively. The SED problem is also solved with SVR
and linear regression in addition to CLEO algorithm, to evaluate CLEO
performance. Table II compares the standard deviation of the objective
function and simulation time of all three algorithms. Based on these results
CLEO performs as well as the SVR algorithm.

Figure 5: Comparison of convergence curves of stochastic (blue) and deterministic (orange) ED (top). Deterministic (red plot) VS. Stochastic (green plot) results. Vertical axis shows objective function vlue which has minimum value of 2109.73 and 4890 for stochastic and deterministic algorithms respectively. X, Y and Z axes show accepted DR commitment, total power generation of the generators and objective function value respectively (bottom). TABLE I: DR commitment value for 3 cases and IEEE 14 bus and 39 bus systems  Value | Case 1 | Case 2 | Case 3
---|---|---|---
DR commitment(14Bus) | 28.05 pu | 6.48e-08 pu | 5.30e-08 pu
DR commitment (39Bus) | 100.44 pu | 100.22 pu | 100.26 pu
TABLE II: Comparison of standard deviation of the supply-side cost and the simulation time between SVR, CLEO and Linear Regression models. Value | SVR | CLEO | LinearRegression
---|---|---|---
Obj func standard deviation | 0 | 1.13e-13 | 0.10
Simulation time(1000 samples) | 0.004 | 0.02 | 0.007

##  V Conclusion

SED-DR-EnU framework is presented in this paper, to account for the importance
of considering endogenous uncertainty in the day-ahead market. Demand response
is provided by DRPs using bottom-up aggregation, which has a significant
degree of DR uncertainty because of end-consumer behavior. The proposed
framework’s ability to take into account both of the exogenous and endogenous
uncertainties of DR in the SED problem—which are caused by physical factors
and behavior of consumers on the demand side—represents its key innovation in
comparison to past research. CLEO model has been presented to take into
account the unpredictability of consumer responsiveness, and it is considered
that the responsiveness of consumers is a function of accepted DR commitments,
to show the dependence of consumer behavior with DR operations. The simulation
results show that DR commitment is influenced by factors like consumer
consumption patterns and the grid’s DR operation decisions. Additionally, a
comparative study demonstrates that demand-side participation uncertainty may
have a high impact on how effectively DR programs operate. Thus, in actuality,
effective calculation of DR commitment should take into account the impact of
endogenous uncertainty.

## References

  * [1] Younes, Z., Alhamrouni, I., Mekhilef, S. and Reyasudin, M.J.A.S.E.J., 2021. A memory-based gravitational search algorithm for solving economic dispatch problem in micro-grid. Ain Shams Engineering Journal, 12(2), pp.1985-1994.
  * [2] Qdr, Q. Benefits of demand response in electricity markets and recommendations for achieving them. US Dept. Energy, Washington, DC, USA, Tech. Rep 2006, 2006.
  * [3] Syrri, A.L. and Mancarella, P., 2016. Reliability and risk assessment of post-contingency demand response in smart distribution networks. Sustainable Energy, Grids and Networks, 7, pp.1-12.
  * [4] Rahmani-Andebili, M. Nonlinear demand response programs for residential customers with nonlinear behavioral models. Energy and Buildings 2016, 119, 352–362.
  * [5] Bayat, N., 2022. Particle Swarm Optimization Based Demand Response Using Artificial Neural Network Based Load Prediction. arXiv preprint arXiv:2204.13990.
  * [6] Ming, H.; Xie, L.; Campi, M.C.; Garatti, S.; Kumar, P. Scenario-based economic dispatch with uncertain demand response. IEEE Transactions on Smart Grid 2017, 10, 1858–1868.
  * [7] Chen, Z., Wu, L. and Fu, Y., 2012. Real-time price-based demand response management for residential appliances via stochastic optimization and robust optimization. IEEE transactions on smart grid, 3(4), pp.1822-1831.
  * [8] Joung, M.; Kim, J. Assessing demand response and smart metering impacts on long-term electricity market prices and system reliability. Applied Energy 2013, 101, 441–448.
  * [9] Li, G.; Bie, Z.; Xie, H.; Lin, Y. Customer satisfaction based reliability evaluation of active distribution networks. Applied energy 2016, 162, 1571–1578.
  * [10] Nolan, S.; O’Malley, M. Challenges and barriers to demand response deployment and evaluation. Applied Energy 2015, 152, 1–10.
  * [11] Zeng, B.; Wei, X.; Zhao, D.; Singh, C.; Zhang, J. Hybrid probabilistic-possibilistic approach for capacity credit evaluation of demand response considering both exogenous and endogenous uncertainties. Applied energy 2018, 229, 186–200.
  * [12] Qi, N.; Pinson, P.; Cheng, L.; Wan, Y.; Zhuang, Y. Chance Constrained Economic Dispatch of Generic Energy Storage under Decision-Dependent Uncertainty. arXiv preprint arXiv:2201.06407 2022.
  * [13] Zhang, Y.; Liu, F.; Wang, Z.; Su, Y.; Wang, W.; Feng, S. Robust scheduling of virtual power plant under exogenous and endogenous uncertainties. IEEE Transactions on Power Systems 2021, 37, 1311–1325.
  * [14] Liu, J.; Li, G.; Sen, S. Coupled learning enabled stochastic programming with endogenous uncertainty. Mathematics of Operations Research 2022, 47, 1681–1705.
  * [15] Jabr, R.A. Adjustable robust OPF with renewable energy sources. IEEE Transactions on Power Systems 2013, 28, 4742–4751.