# ForeSeer: Product Aspect Forecasting Using Temporal Graph Embedding

Zixuan Liu<EMAIL_ADDRESS>0000-0002-5730-9987 University of
WashingtonP.O. Box 1212SeattleWAUSA98105 ,  Gaurush Hiranadani
<EMAIL_ADDRESS>AmazonPalo AltoCAUSA 0009-0002-5718-8959 ,  Kun Qian
<EMAIL_ADDRESS>AmazonPalo AltoCAUSA 0000-0002-9063-102X ,  Edward W. Huang
<EMAIL_ADDRESS>AmazonPalo AltoCAUSA 0000-0002-4461-8545 ,  Yi Xu
<EMAIL_ADDRESS>AmazonSeattleWAUSA 0000-0002-0604-8481 ,  Belinda Zeng
<EMAIL_ADDRESS>AmazonSeattleWAUSA 0009-0002-5446-0523 ,  Karthik Subbian
<EMAIL_ADDRESS>AmazonPalo AltoCAUSA 0000-0002-9023-2248 and  Sheng Wang
<EMAIL_ADDRESS>University of WashingtonSeattleWAUSA
0000-0002-0439-5199

(2023)

###### Abstract.

Developing text mining approaches to mine aspects from customer reviews has
been well-studied due to its importance in understanding customer needs and
product attributes. In contrast, it remains unclear how to predict the future
emerging aspects of a new product that currently has little review
information. This task, which we named product aspect forecasting, is critical
for recommending new products, but also challenging because of the missing
reviews. Here, we propose ForeSeer, a novel textual mining and product
embedding approach progressively trained on temporal product graphs for this
novel product aspect forecasting task. ForeSeer transfers reviews from similar
products on a large product graph and exploits these reviews to predict
aspects that might emerge in future reviews. A key novelty of our method is to
jointly provide review, product, and aspect embeddings that are both time-
sensitive and less affected by extremely imbalanced aspect frequencies. We
evaluated ForeSeer on a real-world product review system containing 11,536,382
reviews and 11,000 products over 3 years. We observe that ForeSeer
substantially outperformed existing approaches with at least 49.1% AUPRC
improvement under the real setting where aspect associations are not given.
ForeSeer further improves future link prediction on the product graph and the
review aspect association prediction. Collectively, Foreseer offers a novel
framework for review forecasting by effectively integrating review text,
product network, and temporal information, opening up new avenues for online
shopping recommendation and e-commerce applications.

Aspect forecasting, textual mining, temporal graph embedding, multi-time
forecasting, information extraction, contrastive learning

††journalyear: 2023††copyright: rightsretained††conference: Proceedings of the
32nd ACM International Conference on Information and Knowledge Management;
October 21–25, 2023; Birmingham, United Kingdom.††booktitle: Proceedings of
the 32nd ACM International Conference on Information and Knowledge Management
(CIKM ’23), October 21–25, 2023, Birmingham, United Kingdom††isbn:
979-8-4007-0124-5/23/10††doi: 10.1145/3583780.3614887††ccs: Information
systems Data mining††ccs: Information systems Information retrieval††ccs:
Information systems Language models††ccs: Computing methodologies Machine
learning††ccs: Computing methodologies Information extraction††ccs: Computing
methodologies Semantic networks††ccs: Computing methodologies Temporal
reasoning††ccs: Computing methodologies Transfer learning

##  1\. Introduction

Customer reviews reflect the properties of the product and the needs of the
customer on online shopping systems (Kim and Kang, 2018; Zhao et al., 2019b).
As a result, one widely studied task is to extract descriptive keywords from
customer reviews, such as ”not greasy” and ”soft inside”. These descriptive
keywords, which are referred to as aspects in the literature, are later used
as key features for various applications that are beneficial to buyers making
decisions and sellers improving the market, such as customer behavior
prediction (Rana et al., 2022), sentiment analysis (Mittal and Agrawal, 2022),
opinion mining (Xu et al., 2011; Kumar et al., 2016), product rating (Liu et
al., 2019b), and marketing refinement (Verma et al., 2021). To tackle this
problem, many text mining methods have been proposed to mine such aspects from
customer reviews (Xu et al., 2013; Yi and Oh, 2022; Sharma et al., 2014;
Dadhich et al., 2021; Kpiebaareh et al., 2022; Jeyapriya and Selvi, 2015;
Chakraborty, 2022; Zhang et al., 2021b, 2022).

Despite its usefulness, however, customer reviews are also subjective and
sometimes biased, therefore mining effective aspects from reviews for a
product requires adequate numbers of reviews. For a new product with limited
customer reviews, aspect mining methods will suffer from starting cold and
mining inaccurate and noisy aspects. In this paper, we aim at a novel product
aspect forecasting task, by forecasting the top aspects in the future for
products with inadequate reviews (Fig. 1), i.e., what aspects will the
customer use to describe a new product after six months or three years? To the
best of our knowledge, product aspect forecasting has not been studied in the
literature and is one step further than traditional aspect extraction tasks.
The most related task is zero-shot aspect-based sentiment analysis (Jebbara
and Cimiano, 2019; Shu et al., 2022; Deng et al., 2022), but their frameworks
are restricted to sentiment analysis or still require multilingual
supervision. The recent progress in large language models such as ChatGPT
(Ouyang et al., 2022) or GPT-4 (Bubeck et al., 2023) may also open a door for
improved zero-shot aspect extraction performance, but besides the heavy
computational costs using them, the limited availability of reviews of new
products still restricts the model from forecasting their top aspects in the
future.

Figure 1. Problem setting of aspect forecasting. The product graph is evolving
because new product nodes and edges are added to the graph. The edge weights
also change over time.

In this paper, we formulate the product aspect forecasting task as a multi-
future top-K aspect retrieving problem. An intuitive solution to the problem
is to find similar products with longer histories and infer based on their
received feedback, just like a customer would search for the reviews of
similar older products that have the same functionality, brand, or same style
before making the decision. Motivated by this, in this paper, we propose a
three-stage framework ForeSeer to address it. ForeSeer is a flexible textual
mining and product embedding framework progressively trained on temporal
product graphs. The key idea is to find similar older products from the
temporal product graph and propagate potential aspects mined from their
reviews to the new item. This enables forecasting currently under-estimated
aspects which might be frequently mentioned in the future (Fig. 2). ForeSeer
has three core steps, contrastive review-aspect association learning,
progressive temporal graph-based product embedding, and aspect-guided product
embedding refinement with temporal information. The first step efficiently
captures the static semantic relationship between review texts and aspects,
and the second step together captures the evolving graphical product
similarity information and how products aggregate reviews over time. In the
third step, we design a novel product-aspect temporal attention (PATA) module
to help adjust the product embeddings guided by aspect embeddings for multi-
length future times forecasting.

We evaluated ForeSeer on a real-world customer review dataset, which contains
11,000 products, 11,536,382 reviews, and a product similarity graph
constructed using user queries and clicks. Our dataset contains the timestamp
for each review and product over three years. We observed substantial
improvement against comparison approaches on product aspect forecasting in
both settings with accessible estimated or ground truth association. We found
the product embeddings derived from ForeSeer show more visible patterns when
the timestamp is increasing, suggesting an accurate incorporation of temporal
information. In addition to aspect forecasting, ForeSeer achieved prominent
performance on multi-time future link prediction and review-aspect association
prediction, demonstrating its wide applicability in modeling temporal review
information. ForeSeer is developed as, to our knowledge, the first approach to
product aspect forecasting and can be broadly applied to other temporal graph
mining tasks.

Figure 2. The key idea of ForeSeer is to find and embed similar products
dynamically and propagate their top aspects mined from reviews to the new item
to forecast future aspects.

##  2\. Product aspect forecasting

###  2.1. Problem Definition: A new task

In principle, products receive and aggregate reviews and aspects are mined
from the review text pieces, enabling the products to extract and aggregate
aspects associations. We then understand relationships among the three core
components, products, aspects, and reviews in detail, and formally define the
product aspect mining and forecasting problem as a top-$K_{l}$ aspect
retrieval problem.

####  2.1.1. Product aspect mining from review texts

A descriptive aspect $A_{l}$, is a descriptive textual span extracted from a
review text sequence $r_{j}$ that summarizes a particular attribute or feature
of $P_{i}$, the product $r_{j}$ belongs to (examples and illustration in Table
1). Given an aspect list $L=\\{A_{1},A_{2},\dots,A_{|L|}\\}$ and $r_{j}$, we
define the review-aspect association vector
$\mathbf{v}_{j}\in\\{0,1\\}^{|L|}$, where $v^{l}_{j}=1$ if $r_{j}$ and $A_{l}$
are associated. The aspect list can be obtained by supervised or unsupervised
mining from reviews (Zhang et al., 2022; Yang et al., 2022). The longer the
list, the more descriptive and diverse aspects can be included in the list.

A product network $\mathcal{G}=\\{\mathcal{P},\mathcal{E}\\}$ is given with a
set of product nodes $\mathcal{P}$ and an edge set $\mathcal{E}$ representing
the similarities between products. $\mathbf{A}^{w}$ is the weighted adjacency
matrix of $\mathcal{E}$. Each product receives reviews and maintains a review
list $R_{i}=\\{r_{i,1},\dots,r_{i,|R_{i}|}\\}$. Let $\mathbf{v}_{i,k}$ be the
review-aspect association vector of $r_{i,k}$ and
$V^{l}_{i}=\\{v^{l}_{i,k}\\}_{k=1}^{|R_{i}|}$ be the collected $A_{l}$’s
association list of $P_{i}$. The product-aspect association vector
$\mathbf{u}_{i}\in\mathbb{R}^{|L|}$ can then be obtained by aggregating all
the review-aspect association vectors in the review list through $Agg(.)$, a
permutation-invariant aggregation function:

Table 1. Three review-aspect association examples are presented to show the
different difficulty levels mining these aspects. The mining difficulty for an
aspect depends on its relationship to the related corpus (More challenging
from top to bottom).

Relation with corpus | Product type | Aspect | Related corpus | Raw review texts
---|---|---|---|---
Exact match | Table | Very sturdy | very sturdy | This table is very sturdy.
Synonym | Candle | Looks great | looks perfect | I bought this product for my wife and it looks perfect for me.
Summary | Carpet cleaning | Great carpet machine | did a great job cleaner cleaning my carpet | This Compact Carpet Cleaner did a great job cleaning my carpet !

(1) $\mathbf{u}_{i}=[Agg(V_{i}^{1});\dots;Agg(V_{i}^{|L|})].$

The binarized top-$K_{l}$ aspect label vector
$\mathbf{y}^{K_{l}}_{i}=\mathbf{y}_{i}$ can then be constructed by ranking
$\mathbf{u}_{i}$ and picking up the top-$K_{l}$ product-aspect associations of
$P_{i}$ to be positive. Let $\mathbf{x}_{i}\in\mathbb{R}^{d_{P}}$ be the node
feature of $P_{i}$, the product aspect mining task is using $\mathbf{x}_{i}$
and graphical information in $\mathcal{G}$ such as $\mathbf{A}^{w}$ to predict
$\mathbf{y}_{i}$. Note that the number of positive labels can be less than
$K_{l}$, resulting in the multi-label nature of $\mathbf{y}_{i}$.

Table 2. Head and tail aspect examples with frequency information over a
three-year period.

Head aspect | 3-year frequency | Tail aspect | 3-year frequency
---|---|---|---
Looks great | 1,955,762 | Buttons are good quality | 145
Works great | 1,796,908 | Great fit and finish | 145
Very sturdy | 1,199,984 | Simple plug and play | 145

####  2.1.2. Predicting future aspect trends for products

In practice, products are released at different times and start receiving
reviews over time. Consider in a discrete time period
$\mathcal{T}=\\{0,\dots,T\\}$, the product network is evolving and
$\mathcal{G}_{t}=\\{\mathcal{P}_{t},\mathcal{E}_{t}\\}$ is the product network
snapshot at time $t\in\mathcal{T}$. $P_{i}\in\mathcal{P}_{t>t^{start}_{i}}$
only occurs after its proposed time $t^{start}_{i}$ in the monotonically
increasing product node list $\mathcal{P}_{t}$ over time. The product
similarities edge set $\mathcal{E}_{t}$ and its weighted adjacency matrix
$\mathbf{A}^{w}_{t}$are also changing as $t$ changes. Products begin receiving
reviews once become available to customers, therefore the corresponding review
lists $R_{i}(t;t^{start}_{i})=R_{i,t}$ are also expanding over time. Any
review $r_{i,k}$ in $R_{i,t}=\\{r_{i,1},\dots,r_{i,|R_{i,t}|}\\}$ also has a
proposed time $t^{prop}_{i,k},t_{i}^{start}\leq t^{prop}_{i,k}\leq t$, For
simplicity, assume $R_{i,t}$ is ordered by ascending review proposed time. Now
$V^{l}_{i}$ will be $V^{l}_{i,t}$ with reviews up to $R_{i,t}$, and the
product-aspect association vector becomes $\mathbf{u}_{i,t}$ by replacing the
$V_{i}^{l}$ to $V^{l}_{i,t}$ in Eqn (1).

Let the above $t$ be the present time, and $\mathbf{x}_{i,t}$ be the node
feature of $P_{i}$ in $\mathcal{G}_{t}$. Suppose at a future time
$t^{\prime}=t+\Delta t$, the product aspect vector and the top-$K_{l}$ aspect
label vector is $\mathbf{u}_{i,t^{\prime}}$ and $\mathbf{y}_{i,t^{\prime}}$.
The $\Delta t$-future product aspect forecasting problem at time $t$ is to
predict $\mathbf{y}_{i,t^{\prime}}$ using $\mathbf{x}_{i,t}$ and
$\mathcal{G}_{t}$. Moreover, we are interested in forecasting top product
aspects $\mathbf{y}_{i,t+\Delta t_{n}}$ at multiple future with
$N_{\Delta_{T}}$ different time lengths $\Delta T\triangleq\\{\Delta
t_{n}\\}^{N_{\Delta_{T}}}_{n=1}$ simultaneously using information till $t$. In
practice, the accessible information is till the present $t$ (referred to as
‘now’). Yet setting ‘$t$ at now’ to an earlier stage will therefore meet with
the old product that is earlier ‘new’, together with a smaller product network
that has fewer products with shorter review lists.

###  2.2. Motivations and challenges of the problem

####  2.2.1. Motivation

We illustrate the motivation of the product aspect forecasting problem. In a
large e-commerce system, existing products keep receiving reviews from buyers,
and sellers continue to release more and more new products (Fig. 1). The high-
level target of this problem is predicting the customer feedback for products
in the long future: e.g., what will be the top aspects of a new product six
months or three years later? This problem might be easier to answer for
products with long histories - they have many customer reviews, thus simply
extracting all aspects till now and counting the most frequent ones can
already obtain statistically significantly satisfying results. However, for
the larger and more important group of newly released products with limited
reviews, such a strategy will give a much less reliable aspect forecasting, as
aspects that currently have low frequency might be frequently mentioned in
future reviews. On one hand, customers cannot trust products with less
accurate aspects prediction; on the other hand, sellers worry about long-
history products monopolizing the market. Therefore, a robust and reliable
solution clearly relieves the dilemma that new products have limited reviews
to help forecast on both the seller and customer side despite the non-trivial
problem nature.

####  2.2.2. Challenges

Forecasting aspects for ‘new’ products can not be effectively achieved by
trivially aggregating mined results from aspect mining methods, as the number
of reviews is limited besides their subjectiveness. Also, the scale of reviews
can be very large and aspect list length $L$ can also be large in practice,
leading to high sparsity of positive associations and extremely imbalanced
aspect frequencies (Table 2). Therefore the top-K aspect retrieving
formulation also introduces association sparsity and extremely imbalanced
aspect frequencies, making it more challenging.

Another challenge is that the ‘new’ products are distributed at different time
points, resulting in unaligned review list series evolving processes. It is
because reviews will keep occurring for products once they are released, and
all the products will receive different reviews at different times. Since
forecasting aspect trends for the ‘new’ products with limited reviews is more
important and challenging, trivially training a graph neural network from a
single product graph is also not effective. The evolving product-review
associations and product similarities need to be efficiently extracted,
aggregated, and embedded across the whole time range to avoid over-fitting
effects.

The third challenge is that the aspect evolving trends will be quite different
in shorter and longer time periods, as $\mathbf{y}_{i,t^{\prime}}$ will
reflect the genuine converged customer feedback when $\Delta
t\rightarrow\infty$ and the recent popular thoughts from customers when
$\Delta t$ is small. Therefore predicting top aspect trends for products at
different $\Delta t$ future indicates capturing different stages of product
evolving from the entire ‘new’ (with limited reviews) to ‘old’ (with adequate
reviews) evolving process from time to time. Trivially using one single
product embedding at current time $t$ without further future-length aware
adjustment will result in unsatisfactory performance.

##  3\. Methods

Figure 3. ForeSeer first resamples review-aspect associations and co-embed
aspects and reviews. It then progressively embeds products leveraging temporal
product graphs. Finally, it exploits temporal information to adjust product
and aspect embeddings to provide multi-time future forecasting.

We propose ForeSeer, a textual mining and product embedding framework
progressively trained on temporal product graph to predict the dynamic product
embeddings at a future time point $t+\Delta t$ using all information till the
current time point $t$. ForeSeer then exploits the product embeddings to
predict the top aspects of that future time. ForeSeer is a three-stage
framework (Fig. 3). In the first stage, we exploit contrastive learning to co-
embed reviews and aspects, leveraging the aspect semantic information to
acquire better review embeddings. We resample review-aspect associations to
increase the presence of rare aspects so that the review embeddings can be
less biased to frequently-appeared aspects. In the second stage, we
progressively train the product graph embedding network on the temporal
product graph. This helps ForeSeer see more new products and the new reviews
they received, as well as evolving product similarity information, resulting
in an efficient product embedding alignment. In the third stage, we develop a
novel Product-Aspect Temporal Attention (PATA) module to further adjust
product embeddings with the help of learned aspect embeddings. PATA module
helps to forecast future product-aspect associations with explicit
consideration of temporal information, thus providing time-sensitive
forecasting.

###  3.1. A base model

We first introduce a base model that extends aspect mining methods for aspect
forecasting and later contrast this base model with Foreseer to clarify the
key technical ideas of ForSeer. Let
$\mathbf{R}=\\{r_{j}\\}^{|\mathbf{R}|}_{j=1}$ be the list of reviews collected
from all products. The base model builds a BERT-based textual encoder that is
pretrained on all reviews with the mask language modeling (MLM) objective
(Devlin et al., 2018). The model can then be used to embed $r_{j}$ as
$\mathbf{h}_{j}=\text{BERT}(r_{j})$. At present time $t$ (‘now’), the learned
review embedding $\mathbf{h}_{j}$ can then be aggregated to the product
$P_{i}$ they belong to as $\mathbf{f}_{i}$ via an aggregator function. Let
$R_{i}$ be the review list of $P_{i}$ at time $t$ and $r_{i,k}\in R_{i}$ be
one of the review of $P_{i}$, the aggregation process is:

(2) $\displaystyle\mathbf{f}_{i}=Agg(\\{\mathbf{h}_{i,k}|r_{i,k}\in
R_{i}\\}).$

The base model then leverages
$\mathbf{x}_{i}=[\mathbf{f}_{i};\textbf{SE}_{i}]$ as a node feature of $P_{i}$
and trains a graph neural network on the last observed product graph
$\mathcal{G}_{last}$ to further get a product embedding $Z_{last}$. Here,
$\textbf{SE}_{i}$ is the static product-specific embedding that encodes other
product-related information, such as seller descriptions, which will not
change over time. Let $\mathbf{A}_{last}^{w}$ be the weighted adjacency matrix
of $\mathcal{G}_{last}$, and $X$ be the product node feature matrix of all
products that make up $\mathbf{x}_{i}$, where $P_{i}\in\mathcal{G}_{last}$.
Let $\sigma$ be an activation function and $H^{(0)}=X$, for $H^{(l_{p})}$, the
output of the $l_{p}$-th GNN layer we have:

(3) $\displaystyle
H^{(l_{p})}=\sigma([\mathbf{A}^{w}_{last}H^{(l_{p}-1)}W^{l_{p}}]).$

Let $Z_{last}=H^{(L_{p})}$ be the resulting embedding matrix of all products
after the final layer, its $i$-th row $\mathbf{z}_{i}$ is the embedding of
$P_{i}$ and is used to get the prediction of top aspects in the future.
Suppose we are interested in multiple futures $\Delta T\triangleq\\{\Delta
t_{n}\\}^{N_{\Delta T}}_{n=1}$, we use $N_{\Delta T}$ classifier heads
$\\{\text{CLS}_{P\rightarrow A,n}\\}^{N_{\Delta T}}_{n=1}$ to map $Z_{last}$
and different $\Delta t_{n}$ to the prediction $\hat{\mathbf{y}}_{i,t+\Delta
t_{n}}$ for different future $t+\Delta t_{n}$. They are finally optimized with
a multi-time multi-label cross-entropy objective:

(4) $\displaystyle\hat{\textbf{y}}_{i,t+\Delta t_{n}}$
$\displaystyle=\text{CLS}_{P\rightarrow A,n}(\mathbf{z}_{i},\Delta t_{n}),$
$\displaystyle\mathcal{L}^{(i)}_{P\rightarrow A}$
$\displaystyle=\frac{1}{N_{\Delta T}\cdot|\mathbf{L}|}\sum^{N_{\Delta
T}}_{n=1}\sum^{|L|}_{l=1}-y^{l}_{i,t+\Delta t_{n}}\log\hat{y}^{l}_{i,t+\Delta
t_{n}}.$

###  3.2. Mining aspect to enhance review embedding

The major limitation of the base model is it only pre-trains the language
model using review texts instead of clearly capturing the review-aspect
association. Because reviews are usually long and subjective (Table 1), it is
very inefficient to capture signals of potential aspects from the embedding of
the entire review text. Also, to have more descriptive and diverse aspects,
the aspect list can be very long. Under this situation, the positive aspect
association in review texts will be very sparse, and the aspect distribution
can be really imbalanced and long-tailed. We thus propose to refine the review
embedding by fine-tuning the review encoder to mine aspect associations. The
key intuition is to embed the review focusing on representing sparse aspect
associations.

Specifically, given a review text $r_{j}$, we fine-tune its embedding
$\mathbf{h}_{j}$ to capture the review-aspect association vector
$\hat{\textbf{v}}_{j}$ with a classifier head $\text{CLS}_{r\rightarrow a}(.)$
and a multi-label cross-entropy loss objective:

(5) $\displaystyle\hat{\textbf{v}}_{j}=\text{CLS}_{r\rightarrow
a}(\mathbf{h}_{j});\ \mathcal{L}^{(j)}_{cls,r\rightarrow
a}=\frac{1}{|\mathbf{L}|}\sum^{|\mathbf{L}|}_{l=1}-v^{l}_{j}\log\hat{v}^{l}_{j}.$

Note that here the association between the review texts and the resulted
aspects will not change with time; it is therefore compatible to combine this
objective with the MLM objectives (Devlin et al., 2018) in the BERT pre-
training stage. The resulting prediction of the association of the review
aspect $\hat{\mathbf{v}}$ can be viewed as part of the refined review
embeddings. For instance, the updated representation
$\mathbf{h}^{\prime}_{j}=[\mathbf{h}_{j};\hat{\mathbf{v}}_{j}]$ can be the
concatenation of BERT embedding and predicted association vectors. Having it
can thus better capture sparse aspect association information during review
aggregation using Eqn. (2).

###  3.3. Aspect-guided review embedding matching

However, the limitation of the above refinement is that it has no idea that
some aspects might describe similar meanings. It thus lacks the ability to
efficiently capture the tail aspects that might have a similar meaning to the
top frequent aspects but are rare and more diverse. Incorporating class names
is useful when classes have their own description (Meng et al., 2020).
Therefore, our next key improvement is to jointly embed reviews and aspects
through cross-domain contrastive learning. The key intuition is that by also
embedding aspects, tail aspects can be closer to head popular aspects if they
have similar semantic meanings. The resulting aspect embeddings can therefore
guide two review embeddings to be closer if they point to aspects with similar
meanings. Moreover, if we treat the base model as a review feature extractor,
training a cascaded contrastive learner will enable an effective way to train
multiple review feature extractors and incorporate resampling and ensemble
techniques.

Specifically, we train multiple feature extractors by instantiating $M$ base
aspect extraction replicas with different reweighting factors
$\alpha_{1},...,\alpha_{M}$ in addition to the standard aspect extraction
instance ($\alpha_{0}=0$). We use $\textbf{h}^{(\alpha_{m})}_{j}$ and
$\hat{\textbf{v}}_{j}^{(\alpha_{m})}$ to denote the trained features output by
the $m$-th replica follows Eqn (5). We then downsample frequent aspects to
encourage the model to focus on review-aspect associations from infrequent
aspects. For replica $m$, we first construct a bin for each aspect by
assigning reviews that have positive associations with that aspect. Here, a
review might be allocated to multiple aspect bins. We thus apply the
resampling strategy hierarchically, which chooses the $l$-th aspect bin with
probability $p_{l}$ by reweighting and then normalizing the importance of the
aspect based on the reciprocal of the $\alpha_{m}$-reweighted aspect
frequency:

(6) $\displaystyle p_{l}=\frac{w_{l}^{(m)}}{\sum_{l}w_{l}^{(m)}},\
w_{l}^{(m)}=(\frac{1}{freq(A_{l})})^{\alpha_{m}},\ \alpha_{m}\geq 0.$

The reweight factor $\alpha_{m}$ leads the model to focus more on the aspects
of tail (large $\alpha_{m}$) or head (small $\alpha_{m}$). $\alpha_{m}=0$
corresponds to the standard aspect extractor that conduct uniform sampling.
Then, a review $r_{j}$ is sampled from the chosen aspect bin with an assigned
importance weight based on the number of review-aspect associations and the
reweighted aspect importance.

This resampling strategy enables the possibility to ensemble the features from
all $M$ feature extractors and perform mixture-of-experts (MoE) learning in
the contrastive learner. We can now exploit contrastive learning to co-embed
reviews and aspects so that the embedding of reviews can pay more attention to
rare aspects. Specifically, we first build review and aspect embedding
networks $E_{R}$ and $E_{A}$ with the same output dimension $d$. $E_{R}$
encodes the embeddings of the review $r_{j}$ and the predicted associations
from all replicas of the extraction of aspects simultaneously and matches them
into a calibrated embedding of the review $\mathbf{z}^{re}_{j}$:

(7)
$\displaystyle\mathbf{z}^{re}_{j}=E_{R}(\text{CONCAT}([\textbf{h}_{j};\hat{\textbf{v}}_{j};\textbf{h}^{(\alpha_{1})}_{j};\hat{\textbf{v}}^{(\alpha_{1})}_{j};...;\textbf{h}^{(\alpha_{M})}_{j};\hat{\textbf{v}}^{(\alpha_{M})}_{j}]).$

For aspect $A_{l}$, we use SpanBERT (Joshi et al., 2020) to get the pre-
trained features and feed it into $E_{A}$ to get the aspect embedding
$\mathbf{z}^{asp}_{l}$:

(8) $\displaystyle\mathbf{z}^{asp}_{l}=E_{A}(\text{SpanBERT}(A_{l})).$

We then utilize the aspect embeddings $\mathbf{z}^{asp}_{l}$ of $A_{l}$ to
guide the optimization of the calibrated review embedding
$\mathbf{z}^{re}_{j}$ of review $r_{j}$ using the inner product of
$\mathbf{z}^{re}_{j}$ and $\mathbf{z}^{asp}_{l}$:

(9) $\displaystyle s^{l}_{j}$
$\displaystyle=(\mathbf{z}^{re}_{j})^{T}\mathbf{z}^{asp}_{l};Z^{asp}=[\mathbf{z}^{asp}_{1};\dots;\mathbf{z}^{asp}_{|L|}];$
$\displaystyle\mathbf{s}_{j}$
$\displaystyle=(\mathbf{z}_{j}^{re})^{T}Z^{asp}=[s^{1}_{j};\dots;s^{|L|}_{j}]\in\mathbb{R}^{|L|}.$

We finally employ a multi-label cross-domain contrastive loss similar to NT-
Xent loss (Sohn, 2016) to maximize positive association and minimize negative
association denoted by $\mathbf{v}_{j}$:

(10)
$\displaystyle\mathcal{L}_{CL}(\mathbf{s}_{j},\mathbf{v}_{j};\tau)=-\log\sum_{l}\frac{v^{l}_{j}exp(s^{l}_{j}/\tau)}{\sum_{k}exp(s^{k}_{j}/\tau)}.$

By jointly learning an informative, low-dimensional embedding space for
reviews and aspects, the resulting $\mathbf{s}_{j}$ has two advantages. First,
it will efficiently ensemble learnings from multiple aspect extraction
replicas, thus more effectively capturing tail aspects. Second, by minimizing
Eqn (10), $\mathbf{s}_{j}$ will encode the ranking information of the aspects
for $r_{j}$. This indicates a more flexible way to retrieve $K_{a}$ positive
associations. By simply tuning $K_{a}$, more (recall-inclined) or less
(precision-inclined) positive associations can be given, resulting in a more
flexible aspect extraction module.

Figure 4. UMAP plots visualizing product embeddings at day 0 (a), day 30 (b),
day 300 (c), and aggregation of product embeddings at all of the three time
points (d).

###  3.4. Progressive temporal graph-based product embedding

The major limitation of the graph-based product embedding module of the base
model is that it is only trained on the last observed product graph snapshot
with the product features at that time. In fact, only a small ratio of
products are ‘new’ in the last observed graph snapshot. This leads the model
to have severe overfitting effects. We thus propose to progressively train the
model on the temporal product graph snapshot series rather than only on the
last observed graph. The key intuition is that the model can not only see more
‘new’ products during training but also be aware of the other temporal
evolving information, including product similarities and how a product becomes
from ‘new’ to ‘old’ by receiving more and more reviews.

Without loss of generality, we now let $t\in\mathcal{T}$ be a time point we
want to train our model at and let the graph be the snapshot $\mathcal{G}_{t}$
at that time. To enable progressive training, multiple fixations are needed.
First, the feature vectors of product nodes need to be aware of the evolving
temporal information, such as the related information of the progressive
product-aspect associations until time $t$. Given a review $r_{i,k}\in
R_{i,t}$, the review list of $P_{i}$ at time $t$, we aggregate the learnt
embedding $\mathbf{z}_{i,k}$ and the predicted review-aspect association
$\mathbf{s}_{i,k}$ for all reviews in $R_{i,t}$ to acquire the product feature
$\textbf{f}_{i,t}$ of $P_{i}$ at time $t$:

(11)
$\displaystyle\textbf{f}_{i,t}=Agg(\\{\text{CONCAT}([\mathbf{z}_{i,k},\mathbf{s}_{i,k}])|r_{i,k}\in
R_{i,t}\\}).$

We then incorporate the related temporal information of $P_{i}$ at time $t$.
Similar to the positional embedding (Vaswani et al., 2017), the 2d-dimensional
temporal embedding of a discrete-time index $t_{0}$ is:

(12) $\displaystyle
Emb_{t_{0}}=[\sin(\omega_{1}t_{0}),\cos(\omega_{1}t_{0}),\dots,\sin(\omega_{d}t_{0}),\cos(\omega_{d}t_{0})].$

For $P_{i}$ at time $t$, there are three pieces of temporal information that
need to be incorporated: its proposed time $t_{i}^{start}$, the training time
$t$ and the gap between these two times $t_{i}^{gap}=t-t_{i}^{start}$. The
resulting temporal embedding therefore has three sub-parts:

(13) $\displaystyle
T^{emb}_{i,t}=\text{CONCAT}([Emb_{t_{i}^{start}},Emb_{t},Emb_{t^{gap}_{i}}]).$

We then compose the overall node feature vector $\mathbf{x}_{i,t}$ for $P_{i}$
at training time $t$ with three parts of features mentioned before:

(14)
$\displaystyle\mathbf{x}_{i,t}=\text{CONCAT}([\mathbf{f}_{i,t};T^{emb}_{i,t};\textbf{SE}_{i}]).$

Let $\mathbf{A}^{w}_{t}$ be the normalized weighted adjacency matrix of
$\mathcal{G}_{t}$, to enable progressive training, we sequentially train the
graph neural network at a subset of graph snapshots
$\\{\mathcal{G}_{t_{1}},\dots,\mathcal{G}_{t_{Q}}|t_{q}\in\mathcal{T}\\}$. We
further refine the GNN layer that contains an extra multi-layer perceptron
(MLP) module that focuses on learning product-specific information.
Specifically, let $W^{l}_{t_{q-1}}$ and $W^{l}_{MLP,t_{q-1}}$ be the weight
matrix of $l$-th GNN layer and the extra MLP module after training the network
on $\mathcal{G}_{t_{q-1}}$ and let $H^{(0)}_{t_{q}}=\mathbf{x}_{i,t_{q}}$, the
product embedding process in Eqn (3) will be updated at $t_{q}$ as:

(15) $\displaystyle
H^{(l_{p})}_{t_{q}}=\sigma(\text{CONCAT}([\mathbf{A}^{w}_{t_{q}}H^{(l_{p-1})}_{t_{q}}W^{l_{p}}_{t_{q-1}};W^{l_{p}}_{MLP,t_{q-1}}H^{(l_{p-1})}_{t_{q}}])).$

At present time $t$, we take $Z^{P}_{t}=H^{(L_{p})}_{t}$ got on
$\mathcal{G}_{t}$ as the resulting product embedding matrix. By progressively
updating the network, ForeSeer can see more new products and how they ‘grow’
from ‘new’ to ‘old’, which is helpful in efficiently aligning the product
embedding.

###  3.5. PATA: Aspect-guided product embedding adjustment for multi-future
forecasting

One limitation of the base model is it uses only one product embedding for all
$\Delta t_{n}$ future predictions without any refinement when capturing future
product-aspect associations. We thus develop a novel multi-head product-aspect
temporal attention (PATA) module to adjust product embedding based on learned
aspect embedding $Z^{asp}$. Our intuition is to adjust product embedding with
a designated attention module guided by aspect embedding. This novel design
empowers ForeSeer with the ability to predict both short and long-future time
ranges with lower computational costs.

Specifically, given $N_{\Delta T}$ target futures, for each $\Delta t_{n}$, we
set up a query network $EP^{Q}_{\Delta t_{n}}$ and a key network
$EA^{K}_{\Delta t_{n}}$ to get the product query matrix from $Z_{t}^{P}$ and
aspect key matrix from $Z^{asp}$:

(16) $\displaystyle Q^{P}_{t+\Delta t_{n}}$ $\displaystyle=EP^{Q}_{\Delta
t_{n}}(Z^{P}_{t}).$ $\displaystyle K^{asp}_{t+\Delta t_{n}}$
$\displaystyle=EA^{K}_{\Delta t_{n}}(Z^{asp})$

We then calculate cross-attention score $Attn_{t+\Delta t_{n}}$ between
product query and aspect key for $\Delta t_{n}$ future:

(17) $\displaystyle Attn_{t+\Delta t_{n}}=Softmax(\frac{Q^{P}_{t+\Delta
t_{n}}(K^{asp}_{t+\Delta
t_{n}})^{T}}{\sqrt{d}})\in\mathbb{R}^{|\mathcal{P}_{t}|\times|L|}.$

We then set up a shared value network $EV^{val}$ that takes $Z^{P}_{t}$ as
input and outputs value matrix $Val^{P}_{t}$. Let $val_{i,t}$ and
$attn_{i,t+\Delta t_{n}}$, the $i$-th row of $Val^{P}_{t}$ and $Attn_{t+\Delta
t_{n}}$, be the value and attention score vector for $P_{i}$, we obtain the
final product embedding for $\Delta t_{n}$ future as:

(18) $\displaystyle\mathbf{z}^{final}_{i,t+\Delta t_{n}}$
$\displaystyle=attn_{i,t+\Delta t_{n}}\circ val_{i,t}.$

We next exploit the final product embedding to get the final prediction using
Eqn (4). With the above design, we can acquire different product embedding for
different $\Delta t_{n}$ future prediction tasks. Using a shared value network
is especially efficient when $N_{\Delta_{T}}$ is large and $d\ll|L|$. This
also regularizes the model by sharing weights for learning multiple imbalanced
top-$K_{l}$ aspect retrieving tasks.

##  4\. Experiment

In this section, we evaluated ForeSeer on a large scale e-commerce dataset
with a variety of tasks, aiming to answer the following research questions
(RQs):

  * •

RQ1: How does ForeSeer perform on product aspect forecasting without or with
genuine annotated aspects for reviews?

  * •

RQ2: Is ForeSeer sensitive to aspects with long trends?

  * •

RQ3: How does different components contributes to the success of ForeSeer on
capturing review-aspect associations?

  * •

RQ4: Is the PATA module in ForeSeer able to capture temporal information for
different lengths of future?

  * •

RQ5: How does the product embedding captured by ForeSeer evolve over time?

Table 3. Performance of ForeSeer and comparison approaches on aspect
forecasting at three different time gaps (3 months, 6 months, 3 years) under
settings of with annotated aspects and without annotated aspects.

$\Delta t$ | Method | Without annotated aspects | With annotated aspects
---|---|---|---
AUPRC | AUROC | Kappa | Max F1 | AUPRC | AUROC | Kappa | Max F1
3 months | Frequency baseline | 21.0 ±5.7 | 84.8±4.1 | 28.6 ±6.2 | 37.4 ±5.6 | 91.2 ±4.2 | 96.9± 2.0 | 86 ±5.4 | 90.0± 4.2
MLP | 13.1± 7.2 | 89.8± 3.2 | 4.5 ±4.3 | 25.1± 6.5 | 14.6± 7.7 | 91.3 ±2.9 | 4.9 ±4.1 | 28.5 ±7.1
LSTM | 17.0 ±6.0 | 93.7± 2.5 | 6.5 ±5.1 | 29.6 ±6.1 | 17.4 ±6.1 | 93.8 ±2.6 | 7.3 ±5.5 | 30.1 ±6.2
GRU | 17.3± 6.2 | 93.6± 2.6 | 7.0 ±5.5 | 30.1± 5.5 | 18.2± 6.2 | 94.0 ±2.6 | 5.9 ±5.3 | 31.0 ±6.3
GNN | 19.2 ± 6.6 | 95.1 ± 3.8 | 25.1± 8.1 | 32.3± 6.8 | 20.9 ± 6.5 | 97.2± 3.9 | 24.9± 7.0 | 34.2 ±6.6
ForeSeer (Ours) | 84.1 ±7.2 | 99.9± 0.2 | 75.8± 7.9 | 83.9 ±5.9 | 80.2 ± 8.4 | 99.9 ±0.1 | 72.3 ±8.2 | 80.0± 7.2
6 months | Frequency baseline | 20.2± 5.6 | 84.3± 4.2 | 27.5± 6.2 | 36.1± 5.6 | 87.4 ±5.1 | 96.4 ±2.2 | 81.8 ±6.0 | 86.3 ±4.9
MLP | 11.1± 7.1 | 88.4± 3.5 | 4.3 ±4.1 | 23.5± 6.7 | 13.9± 7.5 | 90.9 ±2.7 | 5.1 ±4.0 | 27.6 ±6.8
LSTM | 14.9± 5.7 | 93± 2.8 | 6.6 ±5.2 | 27.5± 6.0 | 15.3 ±5.8 | 93.0 ±2.9 | 7.2 ±5.5 | 27.8± 6.1
GRU | 15.4± 5.9 | 92.9 ±2.8 | 6.7 ±5.3 | 28.1± 6.2 | 16.2± 6.0 | 93.3± 5.6 | 28.9 ±6.3 | 26.3 ±5.3
GNN | 18.1 ± 6.5 | 93.8 ± 3.5 | 23.0± 7.7 | 29.4± 6.7 | 19.3 ± 6.4 | 95.9± 3.3 | 22.9± 6.9 | 31.9 ±6.6
ForeSeer (Ours) | 81.8± 7.7 | 99.8± 0.2 | 74.2± 8.1 | 82.2 ±6.3 | 82.3±8.6 | 99.7 ±0.3 | 73.6 ±8.5 | 82.6±6.9
3 years | Frequency baseline | 18.6 ±5.4 | 83.8 ±4.2 | 25.4± 6.1 | 34.0± 5.6 | 81.0± 6.1 | 96.2± 2.3 | 75.8 ±6.5 | 81.2± 5.3
MLP | 10.2± 4.9 | 86.3 ±3.0 | 3.1± 3.0 | 21.2± 5.3 | 11.7± 5.3 | 89.7 ±4.2 | 5.5± 5.2 | 23.3± 6.1
LSTM | 11.4± 5.0 | 90.3 ±3.3 | 5.8± 4.9 | 23.4± 5.7 | 12.0± 5.2 | 90.3 ±3.5 | 6.6± 5.3 | 24.1± 6.0
GRU | 12.1 ±5.2 | 90.3 ±3.3 | 6.0 ±5.1 | 24.4 ±5.9 | 13.1± 5.4 | 90.7±6.7 | 5.9±5.1 | 25.1±6.0
OA-Mine | 5.9 ± 3.9 | 78.6 ± 5.9 | 3.4 ± 2.7 | 6.1 ± 2.6 | 5.9 ± 3.9 | 78.6 ± 5.9 | 3.4 ± 2.7 | 6.1 ± 2.6
GNN | 16.1 ± 6.3 | 92.1 ± 3.1 | 20.9± 7.1 | 28.6± 6.6 | 17.3 ± 6.4 | 93.0± 2.7 | 20.6± 6.9 | 29.7 ±6.5
ForeSeer (Ours) | 68.5 ± 8.9 | 98.9± 1.1 | 65.6 ± 8.5 | 73.2 ± 6.9 | 83.6 ±7.9 | 99.9± 0.1 | 74.7 ± 7.7 | 82.8± 6.4

###  4.1. Experimental setup

####  4.1.1. Dataset

We evaluated our ForeSeer framework on a large-scale e-commerce dataset that
contains a dynamic product network with 11,536,382 time-stamped review events
in a three-year period. The product network is a series of homogeneous dynamic
product similarity graphs with 1,096 daily time-stamped snapshots. The final
graph has 11,000 product nodes from 418 different product types based on their
name, high-level property, and usage, such as ”Lamp”, ”TABLE”, and ”Caddy”.
The unpublished product nodes will not be added to the graph until their
proposed times are reached. The average start date of products is 77. The
first snapshot has 999,614 edges and the final snapshot has 2,400,404 edges.
Each edge represents a similarity ranging from 0 and 1 between the two
products. We calculate the similarities based on multiple factors such as user
click information. We obtained an aspect list with 10,000 popular aspects
extracted via sequential extraction tools (Yan et al., 2021; Yang et al.,
2022) and 30,217,638 review-aspect associations from these reviews. We
constructed the top-$K_{l}$ ($K_{l}=10$) aspects label for every product node
at all timesteps by aggregating all the review-aspect associations it has and
picking up the top-ranked aspects after normalization. Example head and tail
aspects with their frequency over three years are shown in Table 2. While the
top 3 aspects have more than 1,000,000 counts, the tail aspects only have 150
counts from more than 11 million review texts, demonstrating extreme aspect
imbalances.

####  4.1.2. Tasks

We studied ForeSeer with three challenging tasks: single review-aspect
association prediction, multi-time aspect forecasting, and multi-time link
prediction.

Single review-aspect association prediction: We model the single review-aspect
association prediction as a multi-label classification problem. We split 80%
of the reviews as the training set and leave the remaining 20% as the test
set.

Multi-time aspect forecasting: To further assess the effectiveness of the
learned review associations, we tested the multi-time aspect forecasting task
with two settings: (i) ground truth association accessible, (ii) only learned
approximate association accessible. For this task, we chose to predict $\Delta
t=3,6$ months and 3 years (always predicting the last observed label) at every
test timestep $t$. We use the label at the final time if the target future
time is out of range. We aimed to test the inductive performance by randomly
splitting 10% nodes out as the test set.

Multi-time link prediction: For multi-time link prediction, we chose to
predict the link at the current time, 180 days later, and at the final time at
every test time $t$. The positive edge ratio is 50%.

####  4.1.3. Comparison approaches

There is no existing framework that directly handles the multi-time aspect
forecasting problem. We therefore designed and compared our method with four
types of baselines. Frequency baseline directly aggregates the association
learnt in section 3.2 at time $t$ and uses it as the prediction. It can not
incorporate other information and the performance will highly depend on the
quality of learned aspect associations from reviews. We aim to assess the
effectiveness of the multi-time forecasting PATA module. MLP baseline directly
predicts the multi-time future with product features without graphical
information. Recurrent-based baselines (GRU, LSTM) predict multi-time future
step-by-step by using an autoregressive network structure and also ignore
graphical similarity information. We thus aim to assess the importance of
using graphical information. Graph-based baseline (GNN) leverages only the
final graphical information $\mathcal{G}_{last}$. We aim to assess the
effectiveness of our progressive training process in multiple $\Delta
t$-future. Weakly supervised baseline (OA-Mine) directly mines candidate
aspects from reviews and aggregates the candidates as the final predictions.
We aim to assess the effectiveness of learned review-aspect associations and
the importance of using learning-based models as this baseline does not
leverage review-aspect association information.

For multi-time link prediction tasks for different $\Delta t$-future, we only
compare our method with graph-based baselines, since other baselines can not
handle this task. We built an extra lightweight GNN for multi-task link
prediction for our methods and a GNN baseline with the same network structure.
For our method, we used features learned from our multi-head product aspect
temporal attention module for different futures as the input for different
future time predictions. For the graph-based baseline, we used the same
features for different future time predictions.

Table 4. Case study showing the aspected we predicted for skin moisturizer.
Our method identified time-sensitive phrases (red) while the frequency
baseline failed to identify them.

Product type | Skin moisturizer
---|---
Count baseline prediction | Smooth, Feels great, Product is great, Clean, Light weight, Soothing, Love this product, Dries quickly,
with one month reviews | Works great, Great price
ForeSeer prediction with | Smooth, Clean, Great price, Product is great, Soothing, A little goes a long way, Sensitive skin, Light
one month reviews | weight, Not greasy, Good quality
Final Label (Ground truth) | Smooth, Love the texture, Light weight, Product is great, Soothing, Great price, A little goes a long way,
Sensitive skin , Fragrance free, Not greasy

####  4.1.4. Implementation details

For the aggregation function $Agg(.)$ used to construct future aspect labels,
we used $sum(.)$ with per-aspect z-score normalization operation.
Specifically, for each aspect, we got the sum of its occurred association for
every product at the end of the time period and calculated the mean and
standard deviation of them. We then performed z-score normalization on the sum
of the associations of this aspect for every product at time $t$. We used
$mean(.)$ as the aggregator to get the aggregated feature $f_{i,t}$ of product
$P_{i}$ at time $t$. We maintained both the predicted logits and the binarized
association prediction from $\mathbf{s}_{i,k}$. We incorporated a pre-trained
product embedding obtained from a multi-modality neural model that encodes
information from the product title and descriptions to construct the auxiliary
product embedding $\mathbf{SE}_{i}$.

For simplicity, we only trained one resampling replica with $\alpha=0.5$ and
match its embedding with embedding learned by standard aspect extraction
replica ($\alpha=0$). We used AdamW as the optimizer and $10^{-4}$ as the
learning rate to train the aspect-guided embedding match model for 10 epochs
on four 16GB Nvidia-V100 GPUs and set the number of retrieved positive
associations $K_{a}=5$ for better precision performance. We used a two-layer
MLP with ReLU for all the embedding networks and a linear projection layer for
all query and key blocks. All the embedding, query, and key vector output
dimensions are 100. To avoid the over-smoothing problem, we used a GNN with
one designed graph neural layer and two cascaded MLP layers. We used the same
network structure for the GNN baseline. For the MLP baseline, we used a two-
layer MLP with a hidden size of 128, for GRU and LSTM baseline we set the
number of the layer as 2 with a hidden size of 128. For the lightweight GNN
for multi-task link prediction task, we used a standard GraphSAGE layer and
one cascaded MLP layer. We used Adam as the optimizer and $10^{-3}$ as the
learning rate and trained all aspect forecasting and link prediction tasks for
200 epochs on a 16GB Nvidia-V100 GPU.

###  4.2. Experimental results

####  4.2.1. Improvement on aspect forecasting without annotated aspects (RQ1)

We first investigated the performance of ForeSeer on aspect forecasting when
aspects are not annotated in the review (Table 3). We found that our method
achieved the best performance on all three pieces of time gaps (3 months, 6
months, and 3 years), indicating that our temporal graph embedding framework
can effectively model the dynamics of aspects and products. For instance,
ForeSeer obtains 63.1% and 49.1% improvements on 90-day gap forecasting and
3-year gap forecasting, respectively. We found that the frequency baseline
didn’t perform well on this task, especially when the time gap is larger,
further confirming the importance of aligning products temporally. We found
that the GNN baseline has undesirable performance, demonstrating the
effectiveness of progressive training on avoiding over-fitting. We found that
the MLP, GRU, and LSTM baselines performed badly on all future forecasting,
necessitating the importance of graphical information. We also found that two
temporal baselines outperform the MLP baseline, suggesting the importance to
adjust product embedding for multi-time forecasting. We found that OA-Mine,
one of the state-of-the-art approaches in aspect mining, obtained a less
prominent performance, indicating that aspect mining methods cannot be applied
to this novel aspect forecasting task. Finally, we noticed that graph-based
approaches GNN, in general, performed better than methods that do not consider
graphs, necessitating the consideration of graph dynamics in this task.

####  4.2.2. Improvement on aspect forecasting with annotated aspects (RQ1,
RQ2)

We next evaluated an easier setting where aspects are annotated in each
review. In this setting, the frequency baseline achieved very good
performances since it simply counts the aspects in current reviews and uses it
to predict future aspects. For aspects that are not time-sensitive, the
frequency baseline can be regarded as an upper bound. Nevertheless, we found
that ForeSeer still achieved good performance in this setting, where ForeSeer
showed comparable performance with the frequency baseline on a 6-month gap and
even outperformed the frequency baseline on a 3-year gap. All other comparison
approaches obtained a much less promising performance. This indicates that
ForeSeer can obtain a comparable performance with the frequency baseline on
aspects that are not time-sensitive and substantially better performance on
aspects that are time-sensitive. To further examine this (RQ2), we illustrated
one case study showing how ForeSeer can effectively recognize long-term
aspects (Table 4). ForeSeer corrected forecasted time-sensitive aspects such
as ”a little goes a long way” and ”sensitive skin”, while the frequency
baseline failed to identify, reassuring ForeSeer’s ability to forecast long-
term future aspects.

Table 5. Comparison of ForeSeer and GNN on predicting future links at three
different time points (current, 6 months, 3 years).

$\Delta t-$future | Approaches | Accuracy | Precision | Recall | F1 score | AUPRC | AUROC
---|---|---|---|---|---|---|---
Current | Baseline GNN | 88.32± 0.54 | 85.29 ± 0.37 | 91.28 ± 0.72 | 88.13 ± 0.61 | 94.64 ± 0.88 | 94.73 ± 0.76
(0 day) | ForeSeer (Ours) | 92.64 ± 0.53 | 92.68 ± 0.35 | 92.66 ± 0.69 | 92.65 ± 0.58 | 97.54 ± 0.83 | 97.66 ± 0.75
6 months | Baseline GNN | 18.36 ± 0.32 | 10.93 ± 0.24 | 89.02 ± 0.72 | 19.49 ± 0.23 | 89.13 ± 0.84 | 44.79 ± 0.24
ForeSeer (Ours) | 85.12 ± 0.51 | 92.96 ± 0.38 | 91.87 ± 0.78 | 92.37 ± 0.55 | 92.21 ± 0.87 | 74.91 ± 0.49
3 years | Baseline GNN | 80.28 ± 0.48 | 77.95 ± 0.35 | 87 ± 0.63 | 82.03 ± 0.77 | 92.24 ± 0.88 | 89.34 ± 0.81
ForeSeer (Ours) | 84.25 ± 0.51 | 92.95 ± 0.44 | 87.31 ± 0.88 | 86.39 ± 0.64 | 94.08 ± 0.96 | 91.75 ± 1.02

####  4.2.3. Improvement on predicting review-aspect association with
embedding matching (RQ3)

Next, we examined the importance of the aspect-guided embedding matching
strategy by performing an ablation study on the review-aspect association
prediction (Fig. 5). We observed the limited precision performance of using
either with or without resampling multi-label classification instances,
leading to poor ability or focusing too much on capturing tail aspect
associations. Instead, we observed substantial improvements from our aspect-
guided embedding framework by successfully exploiting the advantage of two
multi-label classification instances. For example, the precision of our
precision-inclined embedding matching model ($K_{a}=5$) is 22% and 34% higher
than the with or without resampling multi-label classification instances. The
superior precision performance suggests the learned review embeddings guided
by aspects concentrate less-biased features for both head and tail aspects,
which is critical to their success.

Figure 5. Bar plot showing the performance of our embedding match approach and
comparison approaches on review-aspect association prediction evaluated using
precision, F1 score, accuracy, and recall.

####  4.2.4. Improvement on predicting future links (RQ4)

After confirming the superior performance of ForeSeer on aspect forecasting,
we next investigated whether ForeSeer’s PATA module is able to capture
temporal information for multiple futures by letting it predict product-
product edges that might emerge in the multiple future times (Table 5). We
observed a consistent improvement of our methods over different time gaps
compared to GNN. We adopted a multi-task prediction setting for efficient
inference and training where predictions for different time gaps are treated
as different tasks. Specifically, we found that the GNN baseline cannot
generalize well on the intermediate-long future range (6 months) as it didn’t
consider the dynamics when simultaneously predicting multiple time gaps. In
contrast, our model addressed this by explicitly modeling the temporal
dynamics of products, confirming that ForeSeer can effectively capture both
product and aspect dynamics.

####  4.2.5. Contrasting static and temporal embeddings (RQ5)

We further visualized the dynamic product embedding space colored by their
product types at different time steps to assess how ForeSeer captures the
temporal pattern (Fig. 4). As the time step becomes larger, we observed more
visible clustering patterns among products. This suggests that the quality of
the product embeddings becomes better as the data is accumulated by modeling
the temporal dynamics. We also noticed the product embedding similarity
reflects edges in the product-product graphs (”shelf” and ”caddy”, ”lamp”,
”light fixture” and ”string light”), indicating these product embeddings
successfully encode graphical information.

##  5\. Related works

Product aspect mining. Few work that tries to forecast aspects for products
and most previous works only aim at product aspect mining tasks. Mining
product aspects from large-scale commercial data has been a well-studied
problem (Hu and Liu, 2004; Sharma et al., 2014; Dadhich et al., 2021;
Kpiebaareh et al., 2022; Yang et al., 2022). Previous works target the problem
using rule-based (Sharma et al., 2014; Dadhich et al., 2021; Kpiebaareh et
al., 2022), supervised learning based extractor (Jeyapriya and Selvi, 2015;
Chakraborty, 2022) or propagation-based methods (Qiu et al., 2011; Lu et al.,
2022). These works are less generalizable to product aspect forecasting, as
they require either domain-specific features or supervision signals from
downstream tasks such as sentiment or opinion labels. Recently, other works
similarly extract descriptive product attributes from seller-provided product
profiles by using sequential supervised textual span labeling information (Xu
et al., 2019; Zhu et al., 2020; Wang et al., 2020b; Li et al., 2020; Yan et
al., 2021) or distant supervision (Zhang et al., 2021b; Wang et al., 2022).
OA-Mine (Zhang et al., 2022) leverage weakly-supervised seed set to mine
aspects from product titles. These methods need either more costly massive
sequential labeled datasets or predesigned hierarchical attribute taxonomy,
which is hard to obtain for customer-centered aspects. Zero-shot aspect
extraction is another recently emerged related topic that aims to extract
aspects in new domains without annotated data by leveraging transfer learning
(Jebbara and Cimiano, 2019), natural language inference (Shu et al., 2022),
document sentiment classification (Deng et al., 2022) and the recent emerged
super-scale ChatGPT (Ouyang et al., 2022) and GPT-4 (Bubeck et al., 2023).
However, the zero-shot indicates that no prior knowledge of the new domain is
needed instead of the number of reviews being limited for products. They are
thus not directly applicable to forecasting tasks. In summary, all the methods
above focus on text sequence level extraction and do not efficiently aggregate
learned attributes at the product level. They are thus useful to help
construct the aspect list for product aspect forecasting tasks, but are not
able to forecast future aspect trends. Unlike these methods, our methods are
the first framework that can forecast aspects that might be mentioned in
future reviews for a new product with a limited number of reviews.

Label-guided classification. Label-guided text classification has been studied
in fields such as social recommendation and document classification (Liu et
al., 2017; Chien et al., 2021). It has been proven to be beneficial for
classification performance (Xiao et al., 2019; Ma et al., 2021). While these
methods focus on classification, our method provides an efficient cross-domain
contrastive learning framework that can easily ensemble these models as expert
multi-label classification instances. Compared to (Wang et al., 2023), which
also proposes to mix multiple encoder instances, our representation learning
scheme allows the model to also produce informative reviews and aspect
representations to help downstream tasks.

Temporal graph learning. Temporal graph-based representation learning has been
intensively studied (Rossi et al., 2020; Xue et al., 2022; Guo et al., 2022).
Previous works either incline on graph structure (Li et al., 2017; Zhu et al.,
2018; Nguyen et al., 2018; Zhao et al., 2019a; Schaefer et al., 2022) or
temporal dynamics (Trivedi et al., 2017, 2019; Ma et al., 2020; Min et al.,
2021; Xu et al., 2022; Jin et al., 2022). The dynamic graph structure is
extracted by graph adjacency dynamics (Li et al., 2018; Goyal et al., 2020),
skip-gram-based modeling (Du et al., 2018; Zuo et al., 2018; Mahdavi et al.,
2018), leveraging clique information (Yu et al., 2018; Zhou et al., 2018;
Huang et al., 2020), step-by-step embedding updating (Zhu et al., 2016; Rahman
et al., 2018; Goyal et al., 2018; Singer et al., 2019; Liu et al., 2019a; Cai
et al., 2021; Liu et al., 2022). These methods only focus on capturing
implicit or short graphical dynamics changing and can not efficiently model
large-scale review-aspect associations and evolving product-review
associations. Some methods are designed specifically for heterogeneous
networks (Zhang et al., 2017; Wang et al., 2020a; Zhang et al., 2021a; Ji et
al., 2021; Zhan and Jia, 2022). Temporal-dynamic focused methods leverage
recurrent-based models (Dai et al., 2016; Zhu et al., 2017; Wu et al., 2017;
Beutel et al., 2018; Kazemi et al., 2019), continuous point process modeling
(Sajadmanesh et al., 2019; Chang et al., 2020) and self-supervised graph
representation learning (Tian et al., 2021; Sun et al., 2022). These methods
focus on modeling interactive events between nodes, but are not suitable for
modeling dynamic product similarities and massive product-review associations.
Some methods leverage self-attention, hierarchical, or temporal attention
(Sankar et al., 2018; Zheng et al., 2019; Lu et al., 2019; Yang et al., 2020;
Xu et al., 2020; Xue et al., 2021; Jiao et al., 2021), but only for temporal
information aggregation, while our PATA module leverages attention to predict
multiple time range future simultaneously. Other recent works such as
EvolveGCN (Pareja et al., 2020) and Roland (You et al., 2022) incorporate
recurrent modules and efficient graph learning. These frameworks can be easily
adapted to our progressive training framework, while we aim to clearly show
that progressive training is the key success to in aspect forecasting with a
simple yet clear formulation. JODIE (Kumar et al., 2019) is the closest work
that can also predict multiple time-range futures with a time projection
module. However, it is based on interaction events and is designed for
heterogeneous user-item networks, making it impractical for our use case. In
summary, most of the existing works can not be directly applied to the product
aspect forecasting task because of the training efficiency and inability to
handle large-scale reviews.

##  6\. Conclusion

In this paper, we have studied a novel task of product aspect forecasting,
which aims to predict aspects that users might mention in future reviews. We
have proposed a novel framework ForeSeer to solve this task by dynamically
embedding products, reviews, and aspects. We have evaluated our method on a
large-scale real-world dataset and observed ForeSeer’s superior performance on
aspect forecasting and link prediction. In the future, we are interested in
boosting ForeSeer with more advanced progressive training strategies. We are
interested in applying ForeSeer to other temporal graph embedding frameworks,
such as modeling biological signaling pathways. We are further interested in
exploring how ForeSeer can assist the classic task of aspect mining, allowing
us to apply ForeSeer to advance a larger number of e-commerce applications,
such as integrating it with recommendation systems to enhance the product
suggestions based on anticipated aspect preferences.

## References

  * (1)
  * Beutel et al. (2018) Alex Beutel, Paul Covington, Sagar Jain, Can Xu, Jia Li, Vince Gatto, and Ed H Chi. 2018.  Latent cross: Making use of context in recurrent recommender systems. In _Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining_. 46–54.
  * Bubeck et al. (2023) Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. 2023\.  Sparks of artificial general intelligence: Early experiments with gpt-4.  _arXiv preprint arXiv:2303.12712_ (2023).
  * Cai et al. (2021) Lei Cai, Zhengzhang Chen, Chen Luo, Jiaping Gui, Jingchao Ni, Ding Li, and Haifeng Chen. 2021.  Structural temporal graph neural networks for anomaly detection in dynamic graphs. In _Proceedings of the 30th ACM international conference on Information & Knowledge Management_. 3747–3756.
  * Chakraborty (2022) Abir Chakraborty. 2022\.  Aspect Based Sentiment Analysis Using Spectral Temporal Graph Neural Network.  _arXiv preprint arXiv:2202.06776_ (2022).
  * Chang et al. (2020) Xiaofu Chang, Xuqin Liu, Jianfeng Wen, Shuang Li, Yanming Fang, Le Song, and Yuan Qi. 2020.  Continuous-time dynamic graph learning via neural interaction processes. In _Proceedings of the 29th ACM International Conference on Information & Knowledge Management_. 145–154.
  * Chien et al. (2021) Eli Chien, Wei-Cheng Chang, Cho-Jui Hsieh, Hsiang-Fu Yu, Jiong Zhang, Olgica Milenkovic, and Inderjit S Dhillon. 2021.  Node Feature Extraction by Self-Supervised Multi-scale Neighborhood Prediction.  _arXiv preprint arXiv:2111.00064_ (2021).
  * Dadhich et al. (2021) Anjali Dadhich, Blessy Thankachan, and Blessy Thankachan. 2021\.  Social & Juristic challenges of AI for Opinion Mining Approaches on Amazon & Flipkart Product Reviews Using Machine Learning Algorithms.  _SN Computer Science_ 2, 3 (2021), 1–21.
  * Dai et al. (2016) Hanjun Dai, Yichen Wang, Rakshit Trivedi, and Le Song. 2016\.  Deep coevolutionary network: Embedding user and item features for recommendation.  _arXiv preprint arXiv:1609.03675_ (2016).
  * Deng et al. (2022) Pengfei Deng, Jianhua Yuan, Yanyan Zhao, and Bing Qin. 2022\.  Zero-shot Aspect-level Sentiment Classification via Explicit Utilization of Aspect-to-Document Sentiment Composition.  _arXiv preprint arXiv:2209.02276_ (2022).
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ (2018).
  * Du et al. (2018) Lun Du, Yun Wang, Guojie Song, Zhicong Lu, and Junshan Wang. 2018.  Dynamic Network Embedding: An Extended Approach for Skip-gram based Network Embedding.. In _IJCAI_ , Vol. 2018. 2086–2092.
  * Goyal et al. (2020) Palash Goyal, Sujit Rokka Chhetri, and Arquimedes Canedo. 2020\.  dyngraph2vec: Capturing network dynamics using dynamic graph representation learning.  _Knowledge-Based Systems_ 187 (2020), 104816.
  * Goyal et al. (2018) Palash Goyal, Nitin Kamra, Xinran He, and Yan Liu. 2018\.  Dyngem: Deep embedding method for dynamic graphs.  _arXiv preprint arXiv:1805.11273_ (2018).
  * Guo et al. (2022) Jin Guo, Zhen Han, Zhou Su, Jiliang Li, Volker Tresp, and Yuyi Wang. 2022\.  Continuous Temporal Graph Networks for Event-Based Graph Data.  _arXiv preprint arXiv:2205.15924_ (2022).
  * Hu and Liu (2004) Minqing Hu and Bing Liu. 2004.  Mining and summarizing customer reviews. In _Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining_. 168–177.
  * Huang et al. (2020) Hong Huang, Zixuan Fang, Xiao Wang, Youshan Miao, and Hai Jin. 2020.  Motif-Preserving Temporal Network Embedding.. In _IJCAI_. 1237–1243.
  * Jebbara and Cimiano (2019) Soufian Jebbara and Philipp Cimiano. 2019.  Zero-shot cross-lingual opinion target extraction.  _arXiv preprint arXiv:1904.09122_ (2019).
  * Jeyapriya and Selvi (2015) A Jeyapriya and CS Kanimozhi Selvi. 2015.  Extracting aspects and mining opinions in product reviews using supervised learning algorithm. In _2015 2nd international conference on electronics and communication systems (ICECS)_. IEEE, 548–552.
  * Ji et al. (2021) Yugang Ji, MingYang Yin, Yuan Fang, Hongxia Yang, Xiangwei Wang, Tianrui Jia, and Chuan Shi. 2021.  Temporal heterogeneous interaction graph embedding for next-item recommendation. In _Joint European Conference on Machine Learning and Knowledge Discovery in Databases_. Springer, 314–329.
  * Jiao et al. (2021) Pengfei Jiao, Xuan Guo, Xin Jing, Dongxiao He, Huaming Wu, Shirui Pan, Maoguo Gong, and Wenjun Wang. 2021\.  Temporal network embedding for link prediction via vae joint attention mechanism.  _IEEE Transactions on Neural Networks and Learning Systems_ (2021).
  * Jin et al. (2022) Guangyin Jin, Min Wang, Jinlei Zhang, Hengyu Sha, and Jincai Huang. 2022.  STGNN-TTE: Travel time estimation via spatial–temporal graph neural network.  _Future Generation Computer Systems_ 126 (2022), 70–81.
  * Joshi et al. (2020) Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. 2020\.  Spanbert: Improving pre-training by representing and predicting spans.  _Transactions of the Association for Computational Linguistics_ 8 (2020), 64–77.
  * Kazemi et al. (2019) Seyed Mehran Kazemi, Rishab Goel, Sepehr Eghbali, Janahan Ramanan, Jaspreet Sahota, Sanjay Thakur, Stella Wu, Cathal Smyth, Pascal Poupart, and Marcus Brubaker. 2019.  Time2vec: Learning a vector representation of time.  _arXiv preprint arXiv:1907.05321_ (2019).
  * Kim and Kang (2018) Sung Guen Kim and Juyoung Kang. 2018.  Analyzing the discriminative attributes of products using text mining focused on cosmetic reviews.  _Information Processing & Management_ 54, 6 (2018), 938–957.
  * Kpiebaareh et al. (2022) Michael Y Kpiebaareh, Wei-Ping Wu, Brighter Agyemang, Charles R Haruna, and Tandoh Lawrence. 2022\.  A Generic Graph-Based Method for Flexible Aspect-Opinion Analysis of Complex Product Customer Feedback.  _Information_ 13, 3 (2022), 118.
  * Kumar et al. (2016) KL Santhosh Kumar, Jayanti Desai, and Jharna Majumdar. 2016\.  Opinion mining and sentiment analysis on online customer review. In _2016 IEEE International Conference on Computational Intelligence and Computing Research (ICCIC)_. IEEE, 1–4.
  * Kumar et al. (2019) Srijan Kumar, Xikun Zhang, and Jure Leskovec. 2019\.  Predicting dynamic embedding trajectory in temporal interaction networks. In _Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining_. 1269–1278.
  * Li et al. (2017) Jundong Li, Harsh Dani, Xia Hu, Jiliang Tang, Yi Chang, and Huan Liu. 2017.  Attributed network embedding for learning in a dynamic environment. In _Proceedings of the 2017 ACM on Conference on Information and Knowledge Management_. 387–396.
  * Li et al. (2020) Ning Li, Chi-Yin Chow, and Jia-Dong Zhang. 2020.  EMOVA: A semi-supervised end-to-end moving-window attentive framework for aspect mining. In _Pacific-Asia Conference on Knowledge Discovery and Data Mining_. Springer, 811–823.
  * Li et al. (2018) Taisong Li, Jiawei Zhang, S Yu Philip, Yan Zhang, and Yonghong Yan. 2018.  Deep dynamic network embedding for link prediction.  _IEEE Access_ 6 (2018), 29219–29230.
  * Liu et al. (2022) Hao Liu, Shuwang Zhou, Changfang Chen, Tianlei Gao, Jiyong Xu, and Minglei Shu. 2022\.  Dynamic knowledge graph reasoning based on deep reinforcement learning.  _Knowledge-Based Systems_ 241 (2022), 108235.
  * Liu et al. (2017) Jingzhou Liu, Wei-Cheng Chang, Yuexin Wu, and Yiming Yang. 2017\.  Deep learning for extreme multi-label text classification. In _Proceedings of the 40th international ACM SIGIR conference on research and development in information retrieval_. 115–124.
  * Liu et al. (2019a) Xi Liu, Ping-Chun Hsieh, Nick Duffield, Rui Chen, Muhe Xie, and Xidao Wen. 2019a.  Real-time streaming graph embedding through local actions. In _Companion Proceedings of The 2019 World Wide Web Conference_. 285–293.
  * Liu et al. (2019b) Yao Liu, Cuiqing Jiang, and Huimin Zhao. 2019b.  Assessing product competitive advantages from the perspective of customers by mining user-generated content on social media.  _Decision Support Systems_ 123 (2019), 113079.
  * Lu et al. (2022) Guangquan Lu, Jiecheng Li, and Jian Wei. 2022.  Aspect sentiment analysis with heterogeneous graph neural networks.  _Information Processing & Management_ 59, 4 (2022), 102953\.
  * Lu et al. (2019) Yuanfu Lu, Xiao Wang, Chuan Shi, Philip S Yu, and Yanfang Ye. 2019.  Temporal network embedding with micro-and macro-dynamics. In _Proceedings of the 28th ACM international conference on information and knowledge management_. 469–478.
  * Ma et al. (2021) Qianwen Ma, Chunyuan Yuan, Wei Zhou, and Songlin Hu. 2021\.  Label-specific dual graph neural network for multi-label text classification. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_. 3855–3864.
  * Ma et al. (2020) Yao Ma, Ziyi Guo, Zhaocun Ren, Jiliang Tang, and Dawei Yin. 2020.  Streaming graph neural networks. In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_. 719–728.
  * Mahdavi et al. (2018) Sedigheh Mahdavi, Shima Khoshraftar, and Aijun An. 2018\.  dynnode2vec: Scalable dynamic network embedding. In _2018 IEEE international conference on big data (Big Data)_. IEEE, 3762–3765.
  * Meng et al. (2020) Yu Meng, Yunyi Zhang, Jiaxin Huang, Chenyan Xiong, Heng Ji, Chao Zhang, and Jiawei Han. 2020.  Text classification using label names only: A language model self-training approach.  _arXiv preprint arXiv:2010.07245_ (2020).
  * Min et al. (2021) Shengjie Min, Zhan Gao, Jing Peng, Liang Wang, Ke Qin, and Bo Fang. 2021\.  STGSN—A Spatial–Temporal Graph Neural Network framework for time-evolving social networks.  _Knowledge-Based Systems_ 214 (2021), 106746.
  * Mittal and Agrawal (2022) Divya Mittal and Shiv Ratan Agrawal. 2022.  Determining banking service attributes from online reviews: text mining and sentiment analysis.  _International Journal of Bank Marketing_ (2022).
  * Nguyen et al. (2018) Giang Hoang Nguyen, John Boaz Lee, Ryan A Rossi, Nesreen K Ahmed, Eunyee Koh, and Sungchul Kim. 2018.  Continuous-time dynamic network embeddings. In _Companion proceedings of the the web conference 2018_. 969–976.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022\.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ 35 (2022), 27730–27744.
  * Pareja et al. (2020) Aldo Pareja, Giacomo Domeniconi, Jie Chen, Tengfei Ma, Toyotaro Suzumura, Hiroki Kanezashi, Tim Kaler, Tao Schardl, and Charles Leiserson. 2020.  Evolvegcn: Evolving graph convolutional networks for dynamic graphs. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 34. 5363–5370.
  * Qiu et al. (2011) Guang Qiu, Bing Liu, Jiajun Bu, and Chun Chen. 2011\.  Opinion word expansion and target extraction through double propagation.  _Computational linguistics_ 37, 1 (2011), 9–27.
  * Rahman et al. (2018) Mahmudur Rahman, Tanay Kumar Saha, Mohammad Al Hasan, Kevin S Xu, and Chandan K Reddy. 2018\.  Dylink2vec: Effective feature representation for link prediction in dynamic networks.  _arXiv preprint arXiv:1804.05755_ (2018).
  * Rana et al. (2022) Jyoti Rana, Loveleen Gaur, and KC Santosh. 2022.  Classifying Customers’ Journey from Online Reviews of Amazon Fresh via Sentiment Analysis and Topic Modelling. In _2022 3rd International Conference on Computation, Automation and Knowledge Management (ICCAKM)_. IEEE, 1–6.
  * Rossi et al. (2020) Emanuele Rossi, Ben Chamberlain, Fabrizio Frasca, Davide Eynard, Federico Monti, and Michael Bronstein. 2020.  Temporal graph networks for deep learning on dynamic graphs.  _arXiv preprint arXiv:2006.10637_ (2020).
  * Sajadmanesh et al. (2019) Sina Sajadmanesh, Sogol Bazargani, Jiawei Zhang, and Hamid R Rabiee. 2019.  Continuous-time relationship prediction in dynamic heterogeneous information networks.  _ACM Transactions on Knowledge Discovery from Data (TKDD)_ 13, 4 (2019), 1–31.
  * Sankar et al. (2018) Aravind Sankar, Yanhong Wu, Liang Gou, Wei Zhang, and Hao Yang. 2018.  Dynamic graph representation learning via self-attention networks.  _arXiv preprint arXiv:1812.09430_ (2018).
  * Schaefer et al. (2022) Simon Schaefer, Daniel Gehrig, and Davide Scaramuzza. 2022\.  AEGNN: Asynchronous Event-based Graph Neural Networks. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 12371–12381.
  * Sharma et al. (2014) Richa Sharma, Shweta Nigam, and Rekha Jain. 2014.  Mining of product reviews at aspect level.  _arXiv preprint arXiv:1406.3714_ (2014).
  * Shu et al. (2022) Lei Shu, Hu Xu, Bing Liu, and Jiahua Chen. 2022\.  Zero-Shot Aspect-Based Sentiment Analysis.  _arXiv preprint arXiv:2202.01924_ (2022).
  * Singer et al. (2019) Uriel Singer, Ido Guy, and Kira Radinsky. 2019.  Node embedding over temporal graphs.  _arXiv preprint arXiv:1903.08889_ (2019).
  * Sohn (2016) Kihyuk Sohn. 2016\.  Improved deep metric learning with multi-class n-pair loss objective.  _Advances in neural information processing systems_ 29 (2016).
  * Sun et al. (2022) Li Sun, Junda Ye, Hao Peng, and Philip S Yu. 2022\.  A Self-supervised Riemannian GNN with Time Varying Curvature for Temporal Graph Learning. In _Proceedings of the 31st ACM International Conference on Information & Knowledge Management_. 1827–1836.
  * Tian et al. (2021) Sheng Tian, Ruofan Wu, Leilei Shi, Liang Zhu, and Tao Xiong. 2021.  Self-supervised representation learning on dynamic graphs. In _Proceedings of the 30th ACM International Conference on Information & Knowledge Management_. 1814–1823.
  * Trivedi et al. (2017) Rakshit Trivedi, Hanjun Dai, Yichen Wang, and Le Song. 2017\.  Know-evolve: Deep temporal reasoning for dynamic knowledge graphs. In _international conference on machine learning_. PMLR, 3462–3471.
  * Trivedi et al. (2019) Rakshit Trivedi, Mehrdad Farajtabar, Prasenjeet Biswal, and Hongyuan Zha. 2019.  Dyrep: Learning representations over dynamic graphs. In _International conference on learning representations_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ 30 (2017).
  * Verma et al. (2021) Sanjeev Verma, Rohit Sharma, Subhamay Deb, and Debojit Maitra. 2021.  Artificial intelligence in marketing: Systematic review and future research direction.  _International Journal of Information Management Data Insights_ 1, 1 (2021), 100002.
  * Wang et al. (2023) Jiyao Wang, Zijie Chen, Yang Qin, Dengbo He, and Fangzhen Lin. 2023.  Multi-Aspect co-Attentional Collaborative Filtering for extreme multi-label text classification.  _Knowledge-Based Systems_ 260 (2023), 110110.
  * Wang et al. (2020b) Qifan Wang, Li Yang, Bhargav Kanagal, Sumit Sanghai, D Sivakumar, Bin Shu, Zac Yu, and Jon Elsas. 2020b.  Learning to extract attribute value from product via question answering: A multi-task approach. In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 47–55.
  * Wang et al. (2022) Xin Wang, Jiaxiu He, David J Curry, and Jun Hyun Ryoo. 2022\.  Attribute embedding: Learning hierarchical representations of product attributes from consumer reviews.  _Journal of Marketing_ 86, 6 (2022), 155–175.
  * Wang et al. (2020a) Xiao Wang, Yuanfu Lu, Chuan Shi, Ruijia Wang, Peng Cui, and Shuai Mou. 2020a.  Dynamic heterogeneous information network embedding with meta-path based proximity.  _IEEE Transactions on Knowledge and Data Engineering_ (2020).
  * Wu et al. (2017) Chao-Yuan Wu, Amr Ahmed, Alex Beutel, Alexander J Smola, and How Jing. 2017.  Recurrent recommender networks. In _Proceedings of the tenth ACM international conference on web search and data mining_. 495–503.
  * Xiao et al. (2019) Lin Xiao, Xin Huang, Boli Chen, and Liping Jing. 2019\.  Label-specific document representation for multi-label text classification. In _Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP)_. 466–475.
  * Xu et al. (2020) Da Xu, Chuanwei Ruan, Evren Korpeoglu, Sushant Kumar, and Kannan Achan. 2020.  Inductive representation learning on temporal graphs.  _arXiv preprint arXiv:2002.07962_ (2020).
  * Xu et al. (2019) Huimin Xu, Wenting Wang, Xinnian Mao, Xinyu Jiang, and Man Lan. 2019.  Scaling up open tagging from tens to thousands: Comprehension empowered attribute value extraction from product title. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_. 5214–5223.
  * Xu et al. (2011) Kaiquan Xu, Stephen Shaoyi Liao, Jiexun Li, and Yuxia Song. 2011\.  Mining comparative opinions from customer reviews for competitive intelligence.  _Decision support systems_ 50, 4 (2011), 743–754.
  * Xu et al. (2022) Mengjia Xu, Apoorva Vikram Singh, and George Em Karniadakis. 2022\.  DynG2G: An Efficient Stochastic Graph Embedding Method for Temporal Graphs.  _IEEE Transactions on Neural Networks and Learning Systems_ (2022).
  * Xu et al. (2013) Xueke Xu, Xueqi Cheng, Songbo Tan, Yue Liu, and Huawei Shen. 2013.  Aspect-level opinion mining of online customer reviews.  _China Communications_ 10, 3 (2013), 25–41.
  * Xue et al. (2022) Guotong Xue, Ming Zhong, Jianxin Li, Jia Chen, Chengshuai Zhai, and Ruochen Kong. 2022\.  Dynamic network embedding survey.  _Neurocomputing_ 472 (2022), 212–223.
  * Xue et al. (2021) Hansheng Xue, Luwei Yang, Wen Jiang, Yi Wei, Yi Hu, and Yu Lin. 2021.  Modeling dynamic heterogeneous network for link prediction using hierarchical attention with temporal rnn. In _Joint European Conference on Machine Learning and Knowledge Discovery in Databases_. Springer, 282–298.
  * Yan et al. (2021) Jun Yan, Nasser Zalmout, Yan Liang, Christan Grant, Xiang Ren, and Xin Luna Dong. 2021\.  Adatag: Multi-attribute value extraction from product profiles with adaptive decoding.  _arXiv preprint arXiv:2106.02318_ (2021).
  * Yang et al. (2022) Li Yang, Qifan Wang, Zac Yu, Anand Kulkarni, Sumit Sanghai, Bin Shu, Jon Elsas, and Bhargav Kanagal. 2022\.  Mave: A product dataset for multi-source attribute value extraction. In _Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining_. 1256–1265.
  * Yang et al. (2020) Luwei Yang, Zhibo Xiao, Wen Jiang, Yi Wei, Yi Hu, and Hao Wang. 2020.  Dynamic heterogeneous graph embedding using hierarchical attentions. In _European Conference on Information Retrieval_. Springer, 425–432.
  * Yi and Oh (2022) Jisu Yi and Yun Kyung Oh. 2022.  The informational value of multi-attribute online consumer reviews: a text mining approach.  _Journal of Retailing and Consumer Services_ 65 (2022), 102519.
  * You et al. (2022) Jiaxuan You, Tianyu Du, and Jure Leskovec. 2022.  ROLAND: graph learning framework for dynamic graphs. In _Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_. 2358–2366.
  * Yu et al. (2018) Wenchao Yu, Wei Cheng, Charu C Aggarwal, Kai Zhang, Haifeng Chen, and Wei Wang. 2018\.  Netwalk: A flexible deep embedding approach for anomaly detection in dynamic networks. In _Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining_. 2672–2681.
  * Zhan and Jia (2022) Ling Zhan and Tao Jia. 2022\.  Coarsas2hvec: Heterogeneous information network embedding with balanced network sampling.  _Entropy_ 24, 2 (2022), 276.
  * Zhang et al. (2021b) Hanlei Zhang, Hua Xu, Ting-En Lin, and Rui Lyu. 2021b.  Discovering new intents with deep aligned clustering. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 35. 14365–14373.
  * Zhang et al. (2022) Xinyang Zhang, Chenwei Zhang, Xian Li, Xin Luna Dong, Jingbo Shang, Christos Faloutsos, and Jiawei Han. 2022.  OA-Mine: Open-World Attribute Mining for E-Commerce Products with Weak Supervision. In _Proceedings of the ACM Web Conference 2022_. 3153–3161.
  * Zhang et al. (2017) Yao Zhang, Yun Xiong, Xiangnan Kong, and Yangyong Zhu. 2017\.  Learning node embeddings in interaction graphs. In _Proceedings of the 2017 ACM on Conference on Information and Knowledge Management_. 397–406.
  * Zhang et al. (2021a) Zhen Zhang, Jiajun Bu, Zhao Li, Chengwei Yao, Can Wang, and Jia Wu. 2021a.  TigeCMN: On exploration of temporal interaction graph embedding via Coupled Memory Neural Networks.  _Neural Networks_ 140 (2021), 13–26.
  * Zhao et al. (2019a) Yifeng Zhao, Xiangwei Wang, Hongxia Yang, Le Song, and Jie Tang. 2019a.  Large Scale Evolving Graphs with Burst Detection.. In _IJCAI_. 4412–4418.
  * Zhao et al. (2019b) Yabing Zhao, Xun Xu, and Mingshu Wang. 2019b.  Predicting overall customer satisfaction: Big data evidence from hotel online textual reviews.  _International Journal of Hospitality Management_ 76 (2019), 111–121.
  * Zheng et al. (2019) Li Zheng, Zhenpeng Li, Jian Li, Zhao Li, and Jun Gao. 2019.  AddGraph: Anomaly Detection in Dynamic Graph Using Attention-based Temporal GCN.. In _IJCAI_. 4419–4425.
  * Zhou et al. (2018) Lekui Zhou, Yang Yang, Xiang Ren, Fei Wu, and Yueting Zhuang. 2018.  Dynamic network embedding by modeling triadic closure process. In _Proceedings of the AAAI conference on artificial intelligence_ , Vol. 32.
  * Zhu et al. (2018) Dingyuan Zhu, Peng Cui, Ziwei Zhang, Jian Pei, and Wenwu Zhu. 2018.  High-order proximity preserved embedding for dynamic networks.  _IEEE Transactions on Knowledge and Data Engineering_ 30, 11 (2018), 2134–2144.
  * Zhu et al. (2016) Linhong Zhu, Dong Guo, Junming Yin, Greg Ver Steeg, and Aram Galstyan. 2016.  Scalable temporal latent space inference for link prediction in dynamic social networks.  _IEEE Transactions on Knowledge and Data Engineering_ 28, 10 (2016), 2765–2777.
  * Zhu et al. (2020) Tiangang Zhu, Yue Wang, Haoran Li, Youzheng Wu, Xiaodong He, and Bowen Zhou. 2020\.  Multimodal joint attribute prediction and value extraction for e-commerce product.  _arXiv preprint arXiv:2009.07162_ (2020).
  * Zhu et al. (2017) Yu Zhu, Hao Li, Yikang Liao, Beidou Wang, Ziyu Guan, Haifeng Liu, and Deng Cai. 2017.  What to Do Next: Modeling User Behaviors by Time-LSTM.. In _IJCAI_ , Vol. 17. 3602–3608.
  * Zuo et al. (2018) Yuan Zuo, Guannan Liu, Hao Lin, Jia Guo, Xiaoqian Hu, and Junjie Wu. 2018\.  Embedding temporal network via neighborhood formation. In _Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining_. 2857–2866.