# Context-Aware Timewise VAEs for Real-Time Vehicle Trajectory Prediction

Pei Xu, Jean-Bernard Hayet and Ioannis Karamouzas * This work was supported by
the National Science Foundation under Grant No. IIS-2047632 and by an
unrestricted gift from Roblox. Pei Xu and Ioannis Karamouzas are with the
School of Computing at Clemson University, South Carolina, USA. {peix,
<EMAIL_ADDRESS>Hayet is with the Department of Computer
Science at CIMAT, A.C., Mexico<EMAIL_ADDRESS>
###### Abstract

Real-time, accurate prediction of human steering behaviors has wide
applications, from developing intelligent traffic systems to deploying
autonomous driving systems in both real and simulated worlds. In this paper,
we present ContextVAE, a context-aware approach for multi-modal vehicle
trajectory prediction. Built upon the backbone architecture of a timewise
variational autoencoder, ContextVAE employs a dual attention mechanism for
observation encoding that accounts for the environmental context information
and the dynamic agents’ states in a unified way. By utilizing features
extracted from semantic maps during agent state encoding, our approach takes
into account both the social features exhibited by agents on the scene and the
physical environment constraints to generate map-compliant and socially-aware
trajectories. We perform extensive testing on the nuScenes prediction
challenge [1], Lyft Level 5 dataset [2] and Waymo Open Motion Dataset [3] to
show the effectiveness of our approach and its state-of-the-art performance.
In all tested datasets, ContextVAE models are fast to train and provide high-
quality multi-modal predictions in real-time.

##  I Introduction

Predicting human behaviors is crucial in the development of autonomous systems
operating in human-populated spaces. Specifically, trajectory prediction for
traffic agents is fundamental for the development of self-driving systems and
intelligent traffic control systems. In this paper, we focus on the problem of
trajectory prediction for vehicles in scenes populated with heterogeneous
neighbors, including pedestrians, cyclists and other vehicles. Accurate
trajectory prediction for such traffic agents requires the predictive model to
fully take into account the external (observable) and internal (intentions)
characteristics of the agents, in addition to the contextual influence from
the agent’s neighbors and the scene environment.

Recently, variational autoencoder (VAE) architectures have been used for human
trajectory prediction in homogeneous, human-human interaction settings,
leading to state-of-the-art performance. Our work is in line with this family
of predictive methods and builds upon the _timewise_ variational autoencoder
(VAE) trajectory prediction architecture introduced in [4]. The resulting
SocialVAE approach has been shown to effectively capture human-human
interactions in complex scenarios and achieve state-of-the-art performance on
several benchmarks of human trajectory prediction. However, it only considers
agent states as the input for trajectory prediction and ignores the influence
from the environmental context on the agent’s steering decision making. This
limits its applicability on vehicle trajectory prediction tasks, where the
environmental context (roads, lanes, etc.) is critical to generate map-
compliant trajectories.

Figure 1:  Trajectory predictions on the Lyft dataset using our proposed
approach. ContextVAE employs a timewise VAE architecture and a context-aware
observation encoding scheme that accounts for environmental (map) and social
(neighbor) features in a unified manner. Top: Corresponding attention that the
target vehicle is paying to its neighbors within a radius of $30m$. Bottom:
Predicted trajectories and map regions with high activation in the attention
model. Red regions are considered more visually salient than blue regions. The
white car identifies the vehicle agent under prediction. Observed trajectories
are shown in blue, and the ground-truth future trajectories are shown in red.
Other colored dots and lines denote stationary and moving neighbors
respectively. Light red polygons on the map denote crosswalks.

In this paper, we introduce ContextVAE as a novel approach for vehicle
trajectory prediction. While keeping a timewise VAE architecture [4],
ContextVAE adopts a dual attention mechanism for observation encoding, which
simultaneously takes into account the environmental context extracted from the
map features and the social context extracted from the neighboring agents’
states. Previous VAE-based works for vehicle trajectory prediction employ a
_decoupled_ observation encoding scheme, where environmental and social
contexts are encoded separately and then are simply concatenated to produce
the final encoding used for prediction [5, 6, 7, 8]. In contrast, we propose
to encode the environmental and social contexts in a _unified_ way, based on
the intuition that humans account for environmental features dynamically and
instantaneously during navigation decision making, rather than using such
features to evaluate different navigation paths after the fact. Instead of
directly plotting trajectories on rasterized maps, our approach uses an RNN-
based encoding of the perceived neighbors along with the extracted local map
features as the context for encoding the local state of the agent under
prediction, As such, we avoid processing a sequence of rasterized maps for
faster inference. Meanwhile, the extracted environmental features directly
participate in the agent’s state encoding process, leading to a fully,
context-aware scheme for observation encoding that allows for a better
inference of an agent’s navigation strategy.

We demonstrate the effectiveness of our proposed ContextVAE and its underlying
encoding scheme on vehicle trajectory prediction tasks from large-scale,
heterogeneous datasets including the nuScenes prediction challenge [1], Lyft
Level 5 dataset [2] and Waymo Open Motion Dataset [3]. As shown in Fig. 1, our
approach accurately captures the relevant map features to generate map-
compliant trajectories while simultaneously taking into account the nearby
traffic agents and the self-state of the agent under prediction. Since our
approach works directly with vectorized map features and agent states, it can
provide multi-modal predictions in real-time. We show that our approach’s
performance is rather insensitive to the nature of the employed map feature
extraction module. Overall, models trained with ContextVAE lead to state-of-
the-art performance on the tested datasets. They are fast to train, have a
small memory footprint, and provide high-quality multi-modal predictions in
real-time.

##  II Related Work

#### Leveraging environmental context

To produce map-compliant predictions, early works directly plot the observed
trajectories (agent of interest and neighbors) on the rasterized scene maps,
performing prediction mainly by using convolutional neural networks (CNNs) to
extract map features without fully exploiting the agent states like velocity
and acceleration [9, 10, 11, 2]. More recent works exploit both the
environmental context and vectorized neighbors’ states as inputs to the
prediction model for the sake of better synthesizing the agent’s motion states
and context [9, 12, 13, 14, 15, 16, 6, 17, 7]. Beyond rasterized maps, some
approaches leverage the graph representation of high-definition (HD) maps to
avoid information loss during rasterization [18, 19]. Systems such as
VectorNet [18] and AutoBots [14] have shown that CNNs applied on rasterized
maps can be replaced by graph neural networks (GNNs) using graph-represented
maps as the input without changing the prediction model’s overall
architecture.

#### Generating sequential predictions

CoverNet [9] and MultiPath [11], as early works, generate trajectories by
picking them from a predefined trajectory set and convert the generative task
to a classification problem. WIMP [20], Trajectron++[6] and DiversityGAN [16]
leverage recurrent neural networks (RNNs) to perform prediction sequentially.
MTP [12], AutoBots [14] and LaPred [21] output fixed-length predicted
trajectories at once. WIMP [20], CXX [22], TNT [23], GoalNet [15] and SGNet
[17] first predict goal positions and then generate trajectories between the
agent’s last observed position and the predicted goals. IMAP [5] and M2I [24],
instead of predicting trajectories only for target agents, perform prediction
sequentially for all the agents. P2T [13] and PGP [25] take a reinforcement
learning approach, with a reward-based model upon which a policy is trained to
perform prediction. P2T uses a goal-directed way to generate trajectories and
PGP generates paths by traversing nodes in the road graph.

#### Handling multi-modality

To model the uncertainty during human driving and account for multi-modal
decision making, Trajectron++[6], BiTrap [7] and SGNet [17] exploit a
conditional variational autoencoder (CVAE) architecture where the prior on the
latent variable is modeled as a mixture of Gaussians. AutoBots [14] uses
Gaussian mixtures as well but considers an encoder-decoder architecture
without variational inference. DiversityGAN [16] takes the architecture of
generative adversarial networks to produce stochastic results. LaPred [21]
employs $K$ blocks of fully connected layers with random initialization to
produce $K$ predictions parallelly. P2T [13] and PGP [25] rely on a discrete
policy under the framework of reinforcement learning to make predictions.

#### Introducing prior knowledge

Besides leveraging vectorized HD maps, recent works introduce prior knowledge
[26] into the process of vehicle trajectory prediction. For example, CXX [22],
WIMP [20], P2T [13], LaPred [21], LaneRCNN [27] and GOHOME [28] select
interesting paths traversing the road graph or specific lanes in the graph as
the basis to generate numeric coordinates. TNT [23] and GoalNet [15] select
goal positions based on the road graph and then perform goal-directed
prediction. Though achieving impressive results on some benchmarks, such
approaches heavily rely on the assumption that vehicles will move between
nodes in the road graph or along specific lanes, which is not always true in
real-life scenarios. In this work, therefore, we focus on comparison with
works that do not rely on this assumption.

##  III Approach

Figure 2: System overview of ContextVAE for vehicle trajectory prediction. Our
approach’s backbone architecture is a timewise VAE shown in the gray block at
the upper right. The proposed map encoding module is linked to the backbone
timewise VAE architecture through the context-aware observation encoder
(dashed box), which processes the vectorized agent states in a unified way
with the map features. M-ATTN and S-ATTN represent the map and social
attention respectively, where $\odot$ represents the dot product operator,
$\otimes$ represents the element-wise multiplication operator, and $\bigcirc$
represents the concatenation operator. Red parts for posterior estimation are
used only during training.

###  III-A Problem Formulation

Given a local, $T$-frame observation $\mathcal{O}_{i}^{1:T}$ gathered from a
target agent $i$, we seek to estimate the agent’s trajectory predictive
distribution over the future $H$ frames, i.e.
$p(\mathbf{x}_{i}^{T+1:T+H}|\mathcal{O}_{i}^{1:T})$, where
$\mathbf{x}_{i}^{T+1:T+H}$ are the agent’s future position coordinates from
frame $T$+1 to frame $T$+$H$. To perform context-aware prediction, we consider
observations that include both the $N_{i}$ observed neighboring agent states
(positions) and environmental context information, i.e.
$\mathcal{O}_{i}^{t}:=\\{\mathbf{M}_{i}^{t},\\{\mathbf{x}_{j}^{t}\\}_{j}\\}$,
where $j=1,\cdots,N_{i}$ and $||\mathbf{x}_{j}^{t}-\mathbf{x}_{i}^{t}||\leq
r_{i}$ given $r_{i}$ as the observation radius of agent $i$.
$\mathbf{M}_{i}^{t}$ is a representation of the local environmental context,
that we model as a vector of features extracted from rasterized semantic maps
via a CNN module, as shown in Fig. 2. To train our models in a scene-invariant
way, we do not predict directly the global coordinates
$\mathbf{x}_{i}^{T+1:T+H}$ but rather the _displacement_ distribution
$p(\mathbf{d}_{i}^{T+1:T+H}|\mathcal{O}_{i}^{1:T})$, where
$\mathbf{d}_{i}^{t}:=\mathbf{x}_{i}^{t}-\mathbf{x}_{i}^{t-1}$ is the
displacement between two frames. Future trajectories are reconstructed as
$\mathbf{x}_{i}^{T+\tau}=\mathbf{x}_{i}^{T}+\sum_{t=T+1}^{T+\tau}\mathbf{d}_{i}^{\tau}$
by sampling $\mathbf{d}_{i}^{\tau}$ from the predicted distributions.

In the following, we present our proposed ContextVAE approach for context-
aware trajectory prediction.

###  III-B Model Architecture and Training Objective

The backbone architecture of ContextVAE is a timewise VAE with a conditional
prior $p_{\theta}$ and a posterior $q_{\phi}$ estimated bidirectionally (see
Fig. 2). The training objective is to maximize a timewise evidence lower
bound:

$\begin{split}\mathbb{E}_{i}\Biggl{[}\frac{1}{H}\sum_{t=T+1}^{T+H}&\mathbb{E}_{\mathbf{z}_{i}^{t}\sim
q_{\phi}(\cdot|\mathbf{b}_{i}^{t},\mathbf{h}_{i}^{t-1})}\biggl{[}\log
p_{\xi}(\mathbf{d}_{i}^{t}|\mathbf{z}_{i}^{t},\mathbf{h}_{i}^{t-1})\\\
&-D_{KL}\left[q_{\phi}(\mathbf{z}_{i}^{t}|\mathbf{b}_{i}^{t},\mathbf{h}_{i}^{t-1})||p_{\theta}(\mathbf{z}_{i}^{t}|\mathbf{h}_{i}^{t-1})\right]\biggr{]}\Biggr{]},\end{split}$
(1)

where $\mathbf{z}_{i}^{t}$ is the latent variable sampled timewisely, and
$\mathbf{b}_{i}^{t}$ and $\mathbf{h}_{i}^{t-1}$ are the RNN hidden states,
which are updated backward in the encoder and forward in the decoder,
respectively. We parameterize the output distribution $p_{\xi}$, the posterior
$q_{\phi}$ and the prior $p_{\theta}$ as Gaussians, using neural networks.

The timewise VAE decoder hidden state $\mathbf{h}_{i}^{t}$ for $t>T$ is
updated recurrently based on the sampled $\mathbf{z}_{i}^{t}$ and the
displacement $\mathbf{d}_{i}^{t}$, i.e.
$\mathbf{h}_{i}^{t}=\overrightarrow{g}(\psi_{\mathbf{zd}}(\mathbf{z}_{i}^{t},\mathbf{d}_{i}^{t}),\mathbf{h}_{i}^{t-1})$
The initial state $\mathbf{h}_{i}^{T}$ encodes the observation via
$\mathbf{h}_{i}^{T}=\psi_{\mathbf{h}}(\mathcal{O}_{i}^{1:T})$, where
$\psi_{\mathbf{zd}}$ and $\psi_{\mathbf{h}}$ are embedding networks. This
leads to a prior conditioned by the context observation
$\mathcal{O}_{i}^{1:T}$ (see III-C). The timewise VAE encoder hidden state
$\mathbf{b}_{i}^{t}$ is updated backwards, i.e.
$\mathbf{b}_{i}^{t}=\overleftarrow{g}(\mathcal{O}_{i}^{t},\mathbf{b}_{i}^{t+1})$,
given the ground-truth observation $\mathcal{O}_{i}^{t}$ at the future frame
$t>T$ and a zero initial state $\mathbf{b}_{i}^{T+H+1}$.

This way, during training, the output distribution $p_{\xi}$ relies on the
latent variable $\mathbf{z}_{t}$ drawn from the backward posterior
$q_{\phi}(\cdot|\mathbf{b}_{i}^{t})$ and $\mathbf{h}_{t-1}$ given by the
forward prior $p_{\theta}$, and thus uses features provided by the whole
trajectory $\mathcal{O}_{i}^{1:T+H}$ for better extracting the agent’s
navigation strategy. At inference, predictions are generated through latents
drawn from the prior sequentially: $\mathbf{d}_{i}^{t}\sim
p_{\xi}(\cdot|\mathbf{z}_{i}^{t},\mathbf{h}_{i}^{t-1})$ where
$\mathbf{z}_{i}^{t}\sim p_{\theta}(\cdot|\mathbf{h}_{i}^{t-1})$, thereby
relying on the historical observation $\mathcal{O}_{i}^{1:T}$ only.

###  III-C Context-Aware Observation Encoding

for observation encoding, where the environmental context extracted from the
semantic map and the social context extracted from the observed agent states
are exploited simultaneously. Our scheme leverages a dual attention mechanism
that combines _map attention_ with _social attention_ as shown at the bottom
of Fig. 2.

Following prior literature [6, 7, 29, 4], our observation encoder employs an
RNN structure $g$ (blue box in Fig. 2) that encodes the agent state
sequentially. The RNN is updated as

$\mathbf{q}_{i}^{t+1}=g(\mathcal{O}_{i}^{t},\mathbf{q}_{i}^{t}),$ (2)

where, from the local observation $\mathcal{O}_{i}^{t}$, we use the agent’s
self states $\mathbf{s}_{i}^{t}$, including the agent’s velocity and
acceleration, and the observed neighbors’ state $\\{\mathbf{n}_{j|i}^{t}\\}$,
including the relative position and velocity of each neighbor $j$ to the agent
$i$. The final $\mathbf{q}_{i}^{T}$ is used to initialize the decoder’s hidden
states in the timewise VAE backbone, i.e.,
$\psi_{\mathbf{h}}(\mathcal{O}_{i}^{1:T})\equiv\mathbf{q}_{i}^{T}$ (Section
III-B).

To account for how the environmental context influences the agent’s steering
actions, we propose to initialize the hidden state $\mathbf{q}_{i}^{1}$ using
features extracted from the local semantic map, rather than encode the map and
the agent’s state independently. However, instead of directly plugging the map
features into the observation encoder, we introduce a _map attention_ (M-ATTN)
mechanism and initialize the encoder via

$\mathbf{q}_{i}^{1}=\textsc{Concat}\left(\mathbf{M}_{i}^{1},\text{M-ATTN}(\\{\mathbf{n}_{j|i}^{1}\\})\right),$
(3)

where

$\begin{split}\text{M-ATTN}(\\{\mathbf{n}_{j|i}^{1}\\})&\triangleq\textsc{Attn}(\mathbf{M}_{i}^{1},\\{\mathbf{n}_{j|i}^{1}\\},\\{\mathbf{n}_{j|i}^{1}\\})\\\
&=\sum\nolimits_{j}w_{j|i}f_{\mathbf{n}_{\text{val}}^{1}}(\mathbf{n}_{j|i}^{1}).\end{split}$
(4)

Here, $\mathbf{M}_{i}^{1}$ denotes vectorized features extracted from agent
$i$’s local map at the _first_ observation frame, and
$f_{\mathchoice{\mathbin{\vbox{\hbox{\scalebox{0.6}{$\displaystyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.6}{$\textstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.6}{$\scriptstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.6}{$\scriptscriptstyle\bullet$}}}}}}$
are embedding networks. $\mathbf{M}_{i}^{1}$, $\\{\mathbf{n}_{j|i}^{1}\\}$ and
$\\{\mathbf{n}_{j|i}^{1}\\}$ correspond to the query, key, and value vectors,
respectively, of the attention mechanism. The attention weights $w_{j|i}$ are
obtained through dot-product operation between $\mathbf{M}_{i}^{1}$ and
$\mathbf{n}_{j|i}^{1}$ followed by a softmax operation, i.e.

$w_{j|i}=\frac{\exp\left(f_{\mathbf{M}^{1}}(\mathbf{M}_{i}^{1})\odot
f_{\mathbf{n}_{\text{key}}^{1}}(\mathbf{n}_{j|i}^{1})\right)}{\sum_{j^{\prime}\neq
i}\exp\left(f_{\mathbf{M}^{1}}(\mathbf{M}_{i}^{1})\odot
f_{\mathbf{n}_{\text{key}}^{1}}(\mathbf{n}_{j^{\prime}|i}^{1})\right)}.$ (5)

The resulting M-ATTN mechanism can help identify neighbors that are located
nearby the target agent but have less influence in the given context, e.g.,
vehicles that are spatially close to the agent but drive on a lane that is
inaccessible to it. We refer to Section IV-E and Fig. 1 for related
qualitative results and to Section IV-C for a quantitative analysis.

During agent state encoding, we synthesize the social context represented by
the neighbors’ states $\\{\mathbf{n}_{j|i}^{t}\\}$ via a _social attention_
(S-ATTN) mechanism as:

$\begin{split}\textsc{S-ATTN}(\\{\mathbf{n}_{j|i}^{t}\\})&\triangleq\textsc{Attn}(\mathbf{q}_{i}^{t},\\{\mathbf{k}_{j|i}^{t}\\},\\{\mathbf{n}_{j|i}^{t}\\})\\\
&=\sum\nolimits_{j}\omega_{j|i}f_{\mathbf{n}_{\text{val}}}(\mathbf{n}_{j|i}^{t}),\end{split}$
(6)

where, $\mathbf{q}_{i}^{t}$, $\\{\mathbf{k}_{j|i}^{t}\\}$, and
$\\{\mathbf{n}_{j|i}^{t}\\}$ correspond to the query, key and value vectors of
the attention, and each weight $\omega_{j|i}$ of the attention graph is
computed as the dot product between $\mathbf{q}_{i}^{t}$ and
$\mathbf{k}_{j|i}^{t}$ after obtaining their fixed-length embeddings. Here,
$\mathbf{k}_{j|i}^{t}$ denotes the social features exhibited by the neighbor
$j$ and observed by the agent $i$. As introduced in [29, 4], we include the
distance, bearing angle and minimal predicted distance from agent $i$ to $j$
as the social features $\mathbf{k}_{j|i}^{t}$.

By choosing a proper size of the local map, we can have $\mathbf{M}_{i}^{1}$
covering the whole trajectory of the agent during the observation and
prediction horizon from $t=1$ to $t=T+H$. This allows us to sidestep
introducing $\mathbf{M}_{i}^{t}$ timewisely, and update the hidden state in
Eq. 2 by representing $\mathcal{O}_{i}^{t}$ as

$\mathcal{O}_{i}^{t}\equiv\textsc{Concat}\left(\mathbf{s}_{i}^{t},\textsc{S-ATTN}(\\{\mathbf{n}_{j|i}^{t}\\})\right).$
(7)

This way, we perform fast observation encoding directly on the environmental
and social contexts after just one pass of the map encoding module, and avoid
the computationally expensive processing a sequence of rasterized maps. We
refer to Section IV-C for our model’s real-time prediction performance.

As shown in Fig. 2, we exploit rasterized semantic maps to capture contextual
information and extract the related features $\mathbf{M}_{i}^{1}$ using a CNN
module. However, we note that there is no constraint about the form of
$\mathbf{M}_{i}^{1}$. For example, $\mathbf{M}_{i}^{1}$ can also be
represented using graphs processed by a GNN [18, 19, 23, 25, 28], or using a
lane attention block [22, 21]. Since map rasterization is the common way to
generate semantic maps for contextual information and given that most open-
source vehicle datasets do not provide an API to directly generate road
graphs, in our experiments we rely on CNNs to extract contextual features from
rasterized maps.

##  IV Experiments

We evaluate our approach on vehicle trajectory prediction tasks using the
nuScenes prediction challenge (nuScenes) [1], Lyft Level 5 Prediction Dataset
(Lyft) [2] and Waymo Open Dataset (Waymo) [3]. These datasets handle
heterogeneous types of agents, including various vehicles, cyclists,
pedestrians. We focus our prediction on vehicles’ trajectories and all other
types of agents are considered as neighbors.

###  IV-A Setup

Figure 3: Examples of rasterized semantic maps for each dataset. From left to
right: nuScenes, Lyft and Waymo. Blue identifies drivable areas and
crosswalks; green identifies road edges; red identifies lane dividers. In the
Waymo maps, we enhance the drivable area with lane center lines colored in
white. The drawn cars identify the first observed position of the agent under
prediction, which also indicates the map’s local origin and $x$-axis
direction. The cars are not part of the rasterized maps.

#### Implementation details

We use a rasterized, $224{\times}224$ local semantic map to represent the
environmental context. The map is translated and rotated such that the vehicle
under prediction heads towards the positive $x$-axis and stands at the 122nd
row and 51st column of the map in the 1st frame of observation. To be
consistent, the coordinates of all agents are expressed using the same local
system. Fig. 3 shows examples of the extracted maps for the three datasets.
Since each dataset has its own semantic map definition, the generated
rasterized maps are slightly different. We refer to the supplementary material
for details on semantic map rasterization.

#### Evaluation Metrics and Baselines

We use the minimum average displacement error over $k$ predictions (minADEk)
and the minimum final displacement error over $k$ predictions (minFDEk) to
assess a model’s performance. We consider both the most likely prediction
($k=1$) and the top-5 predictions ($k=5$) as our key metrics measured in
meters. We provide comparisons to two deterministic baselines: Constant
Velocity assumes target vehicles always move at the velocity defined between
the last two observed frames; Kalman Filter applies an extended Kalman filter
while assuming constant longitudinal speed and heading direction. We also
consider state-of-the-art, data-driven baselines for each dataset. Since our
approach does not use prior knowledge to constrain the movement of vehicles,
we limit our comparison to approaches without restricting the accessibility of
vehicles via maps or refining predicted trajectories at a post-processing
stage.

###  IV-B Quantitative Results

TABLE I: Performance on nuScenes Prediction Challenge. nuScenes (Validation Set) | minADEk (m) | minFDEk (m)
---|---|---
$k=1$ | $k=5$ | $k=1$ | $k=5$
Constant Velocity | 4.61 | - | 11.21 | -
Kalman Filter | 4.17 | - | 10.99 | -
Trajectron++ | 4.08 | 2.41 | 9.67 | 5.63
P2T | 3.82 | 1.86 | 8.95 | 4.08
AutoBots-Ego | 3.86 | 1.70 | 8.89 | 3.40
ContextVAE | 3.54 | 1.59 | 8.24 | 3.28
TABLE II: Performance on Lyft Level 5 Prediction Dataset. Lyft | minADEk (m) | minFDEk (m)
---|---|---
$k=1$ | $k=5$ | $k=1$ | $k=5$
Constant Velocity | 0.70 | - | 1.64 | -
Kalman Filter | 0.60 | - | 1.30 | -
SAMPP [5] | 0.39 | - | 0.83 | -
IMAP [5] | 0.28 | - | 0.65 | -
Trajectron++ | 0.38 | 0.26 | 0.89 | 0.57
ContextVAE | 0.24 | 0.16 | 0.54 | 0.32
TABLE III: Performance on Waymo Open Dataset. Waymo (Validation Set) | minADEk (m) | minFDEk (m)
---|---|---
$k=1$ | $k=5$ | $k=1$ | $k=5$
Constant Velocity | 2.04 | - | 5.25 | -
Kalman Filter | 1.99 | - | 4.07 | -
SAMPP [5] | 1.26 | - | 2.80 | -
IMAP [5] | 0.97 | - | 2.03 | -
Trajectron++ | 0.88 | 0.56 | 2.37 | 1.41
MotionCNN | 0.83 | 0.40 | 1.99 | 0.81
M2I | 0.67 | 0.42 | 1.60 | 0.85
ContextVAE | 0.59 | 0.30 | 1.49 | 0.68

#### nuScenes

We use the official training and validation sets of nuScenes prediction
challenge to perform model training and performance evaluation. An observation
window of up to 2s is used and the prediction horizon is 6s. As the data was
recorded at 2FPS, this leads to a 12-frame prediction and a varying
observation window between 2 and 5 frames. Additionally to Constant Velocity
and Kalman Filter, we introduce three other baselines for comparison. The
Trajectron++ [6] model is trained from scratch with two known bugs fixed as
indicated in the official repository. P2T [13] is tested using their provided
pre-trained model. Since no pre-trained models are provided, we train the
AutoBots-Ego model using the official implementation. These three baselines
exploit CNN modules for map encoding. We report the results in Table I. Among
the considered baselines, Constant Velocity and Kalman Filter perform the
worst. As recently published works, P2T and AutoBots-Ego work significantly
better than Trajectron++, with AutoBots-Ego outperfoming P2T. However, its
training is very time-consuming. AutoBots-Ego needed more than three days to
perform 70-epochs training using a V100 GPU, despite the fact that nuScenes
has only 32,186 trajectories for training. Our approach not only achieves the
best performance on all four metrics, but it is also extremely fast to train,
needing only about 2 hours to train on a single V100 GPU.

#### Lyft

This dataset is much larger than _nuScenes_ and has more than 1,000 hours of
data with 170,000 scenes. We use only the first 16,265 scenes of the training
data, with each scene being about 24s long. For evaluation, we use the full
validation dataset (16,220 scenes). We downsample the data from 10FPS to 5FPS
and train models with an observation window of 1s and prediction horizon of
3s. In Table II, we report the performance obtained by our approach and two
newly introduced baseline models: SAMPP and IMAP [5]. These are deterministic
models that rely on lane graphs rather than rasterized semantic maps. They use
the same observation window and prediction horizon as ours, and we list their
reported numbers in the table. Trajectron++ is introduced as a multimodal
baseline, trained using the same data as ours. As it can be seen, IMAP
achieves better deterministic results (minADE1/minFDE1) compared to
Trajectron++, though these results are slightly worse compared to
Trajectron++’s multimodal performance. Our approach outperforms IMAP and
Trajectron++ on both the deterministic and multimodal results, with an
improvement around 15% on minADE1/minFDE1 and around 40% on minADE5/minFDE5.

#### Waymo

This dataset has two types of training data formats, consisting of 20s and 9s
sequential data, respectively. We use the 9s data for training, which contains
487,002 scenes, while the validation set has 44,097 scenes. In addition to
performing downsampling from 10FPS to 5FPS, we apply the data filter from the
Lyft dataset to filter out invalid data. Similar to our Lyft implementation we
use a 1s window for observation and a 3s horizon for prediction. Following the
Waymo motion prediction challenge design, we train and evaluate our model
using only the vehicle agents labeled as objects of interest. Table III shows
the performance of our approach along with two new baselines: MotionCNN [10]
and M2I [24]. We run the pre-trained models from the two baselines and keep
the first 3s prediction for evaluation. As shown in Table III, MotionCNN and
M2I outperform Trajectron++ with a significant improvement on minADE5/minFDE5.
M2I also brings a large improvement on the most likely prediction. Compared
with these two strong baselines, our approach achieves more than 15%
improvement overall, and a 29% and 20% improvement on minADE5 and minFDE5,
respectively.

The default Waymo motion prediction challenge setup uses an 8s prediction
horizon with $k=6$. The performance of our model trained for 8s prediction on
the interactive validation set is: minADE${}_{6}=1.59$ and
minFDE${}_{6}=3.67$. As a comparison, the Waymo LSTM baseline achieves
minFDE${}_{6}=6.07$ for 8s prediction [30], and M2I [24] reports
minFDE${}_{6}=5.49$.

###  IV-C Sensitivity Analysis

TABLE IV: Prediction errors with Different CNN Modules for map feature extraction. |  |  | nuScenes |  | Lyft |  | Waymo
---|---|---|---|---|---|---|---
| Infer. | # of | minADEk (m) | minFDEk (m) |  | minADEk (m) | minFDEk (m) |  | minADEk (m) | minFDEk (m)
CNN Module | Time | Params. | $k=1$ | $k=5$ | $k=1$ | $k=5$ |  | $k=1$ | $k=5$ | $k=1$ | $k=5$ |  | $k=1$ | $k=5$ | $k=1$ | $k=5$
ResNet18 | 0.23s | 11.7M | 3.543 | 1.586 | 8.244 | 3.277 |  | 0.247 | 0.165 | 0.548 | 0.324 |  | 0.600 | 0.306 | 1.532 | 0.709
ResNet152 | 1.33s | 60.2M | 3.728 | 1.728 | 8.773 | 3.657 |  | 0.244 | 0.164 | 0.544 | 0.321 |  | 0.598 | 0.306 | 1.530 | 0.708
EfficientNet-B0 | 0.25s |  05.3M | 3.715 | 1.740 | 8.733 | 3.676 |  | 0.246 | 0.165 | 0.548 | 0.325 |  | 0.585 | 0.299 | 1.491 | 0.686
MobileNet-V2 | 0.20s |  03.5M | 3.780 | 1.738 | 8.888 | 3.671 |  | 0.251 | 0.165 | 0.560 | 0.321 |  | 0.585 | 0.298 | 1.492 | 0.684

#### CNN Modules

We evaluated the performance of ContextVAE with several popular CNN modules
for map encoding. In Table IV, we report the corresponding results along with
the number of parameters and inference time taken by each CNN module. The
inference time refers to the time needed to perform $k=5$ predictions for one
batch of 512 input samples. The reported time is measured on a machine
equipped with a V100 GPU and averaged over 10,000 test trials. As can be seen
in the table, there are no large differences regarding minADEk/minFDEk among
the tested CNN modules on Lyft and Waymo. We can therefore safely choose a
small module, like EfficientNet-B0 or MobileNet-V2, for the sake of faster
inference. In nuScenes, the best performance is obtained with ResNet18, but
the differences among the CNN modules are larger. A possible reason is the
small size of the nuScenes dataset, which makes the model more sensitive to
the choice of CNN modules. Overall, using ResNet18 on _nuScenes_ meets the
real-time inference requirements while guaranteeing high prediction accuracy.
For a complete list of all the tested CNN modules, we refer to the
supplementary material.

Figure 4: Minimal final displacement error (in meters) as a function of the
number $k$ of predicted trajectories for different prediction horizons.

#### Prediction Horizon and Choice of $k$

Intuitively, increasing the prediction horizon leads to more uncertainty,
making prediction harder. This is consistent with results shown in Fig. 4,
where larger horizons lead to higher minFDE values. For a given horizon, an
important question is how many prediction samples $k$ are enough to lead to a
sufficiently low error? For a 1s prediction horizon (blue line), the error
converges around $k=10$. For a 3s horizon (red line), $k=20$ roughly leads to
convergence. However, for a 6s horizon (green line), there is still a
decreasing trend at $k=20$. We refer to the supplementary material for more
numeric results of model performance with different prediction horizons.

#### Ablation Study

In Table V, we perform ablation studies on _Waymo_ , related to our proposed
unified observation encoding scheme and its underlying dual attention
mechanism (M-ATTN in Eq. 4 and S-ATTN in Eq. 6). When neither map nor social
context is exploited, the neighbors’ states are simply added together for
observation encoding, i.e.
$\mathcal{O}_{i}^{t}\equiv\textsc{Concat}(\mathbf{s}_{i}^{t},\sum_{j}f_{\mathbf{n}}(\\{\mathbf{n}_{j|i}^{t}\\}))$
instead of Eq. 7. When only S-ATTN is used, the observation encoder is
initialized as
$\mathbf{q}_{i}^{1}=\sum_{j}f_{\mathbf{n}^{1}}(\mathbf{n_{j|i}^{1}})$ instead
of Eq. 3. When the map is introduced independently (“Indie”),
$\mathbf{q}_{i}^{1}$ is initialized similarly to S-ATTN and the output
$\mathbf{q}_{i}^{T}$ of the encoder is concatenated with the map features
$\mathbf{M}_{i}^{1}$ to provide the initial state for the timewise VAE
decoder, i.e.
$\mathbf{h}_{i}^{T}\equiv\textsc{Concat}(\mathbf{q}_{i}^{T},\mathbf{M}_{i}^{1})$.
In contrast, in our proposed observation encoding scheme, the map features are
integrated into the encoder (“Integrated”), with $\mathbf{M}_{i}^{1}$ being
used for initializing $\mathbf{q}_{i}^{1}$ and
$\mathbf{h}_{i}^{T}\equiv\mathbf{q}_{i}^{T}$. This allows for map and agents’
states to be encoded in a unified manner. When M-ATTN is employed,
$\mathbf{q}_{i}^{1}$ is initialized as shown in Eq. 3. Without M-ATTN, we
simply integrate the map features and the agent’s neighbors as
$\mathbf{q}_{i}^{1}=\textsc{Concat}(\mathbf{M}_{i}^{1},\sum_{j}f_{\mathbf{n}^{1}}(\\{\mathbf{n}_{j|i}^{1}\\}))$.
As can be seen from Table V, S-ATTN brings about 4% improvement over the basic
timewise VAE backbone. Handling the semantic map independently from the social
context brings an extra 3% improvement on the deterministic predictions and
around 15% on the multimodal predictions. Integrating the map into the encoder
without M-ATTN results in even better performance, highlighting its potential
over Indie maps. M-ATTN further boosts the performance with an improvement
around 35% on minADE5/minFDE5, and around 15% on minADE1/minFDE1 compared to
the basic backbone architecture. Similar conclusions can be drawn from the
nuScenes and Lyft datasets, as described in the supplementary material.

TABLE V: Model Ablation on Waymo. |  |  | minADEk (m) | minFDEk (m)
---|---|---|---|---
S-ATTN | MAP | M-ATTN | $k=1$ | $k=5$ | $k=1$ | $k=5$
- | - | - | 0.68 | 0.45 | 1.78 | 1.11
✓ | - | - | 0.65 | 0.43 | 1.71 | 1.06
✓ | Indie | - | 0.63 | 0.37 | 1.68 | 0.85
✓ | Integrated | - | 0.62 | 0.33 | 1.65 | 0.70
✓ | Integrated | ✓ | 0.59 | 0.30 | 1.49 | 0.68
Figure 5: Performance improvement when using map context on Waymo dataset.

Figure 6: Qualitative examples from Lyft. Observed trajectories are shown in
blue. The ground-truth future trajectories are shown in red. 5 predictions are
shown in orange. Other colored dots and lines denote stationary and moving
neighbors respectively.

Prediction with S-ATTN + M-ATTN

Prediction w/o Map (S-ATTN only)

Figure 7: Case studies with (top) and without (bottom) considering rasterized
maps in the observation encoder. The trajectory distributions heatmaps are
generated using 2,000 predictions.

###  IV-D Integrated vs. Independent Maps for Observation Encoding

To further highlight the effectiveness of using Integrated rather than Indie
maps for RNN-based observation encoding, we consider experiments with
additional VAE-based approaches that by default decouple environmental context
from social context. Fig. 5 compares the performance of Trajectron++ [6] when
Indie maps and Integrated maps are employed to its performance when no
semantic maps are used. The Indie scheme follows the original implementation
of Trajectron++ that accounts for map features independently from the agents’
states. In the Integrated model, the map features are used to initialize the
RNN hidden state of the observation encoder. We also extend the BiTraP [7]
method to work with rasterized maps by using the same map encoding module as
Trajectron++ and report the related comparisons in the figure. As can be seen,
models with Integrated maps significantly improve the performance of
Trajectron++ and Bitrap on both deterministic and multimodal predictions. In
contrast, Indie maps do not provide any noticeable performance gain on
deterministic predictions and only bring a small improvement on multimodal
ones. We refer to the supplementary material for complete results on all the
tested datasets.

###  IV-E Case Studies

Fig. 6 shows predictions obtained with ContextVAE on the Lyft dataset. As can
be seen, using maps makes the model aware of the lanes and results in covering
the ground-truth trajectories closely, while also accounting for other
possible driving maneuvers that could be taken. In the first three scenes, for
example, ContextVAE accurately assigns samples to different lanes at the
multi-lane intersections. In the fourth scene, the model predicts the target
vehicle’s turning-left behavior along with a potential U-turn behavior. The
last two cases exemplify non map-compliant behaviors, where the vehicle drives
from and to the off-map region, respectively. While graph-based methods
typically struggle with off-road cases, our approach accurately predicts the
vehicles’ movement.

Prediction with S-ATTN Social Attention Map Prediction w/o S-ATTN

Figure 8:  Case study from _Lyft_ with/without accounting for S-ATTN in
observation encoding. The left and middle figures show the heatmap of the
predicted trajectory distribution and the social attention of the target
vehicle, respectively, given neighbors observed within a radius of 30m. The
right figure shows the predictions without using the S-ATTN mechanism, i.e.,
ignoring the pink box from the observation encoder in Fig. 2.

In Fig. 7, we highlight additional examples comparing the prediction results
of ContextVAE to the results obtained when the map and its related attention
mechanism are disabled from the observation encoder. Ignoring maps can lead to
myopic as well as wrong predictions. In the first two cases, for example, the
model predicts that the target vehicle will keep driving straight and does not
account for the multi-modal decisions at intersections. While in the third
scene, the model predicts multi-modal driving trajectories, such trajectories
are invalid as the vehicle can only take a right turn. In contrast, accounting
for map attention leads to map-compliant paths and helps to predict potential
turning behaviors at intersections.

As discussed in the ablation study, the social attention mechanism of the
observation encoder is equally important, allowing the model to infer the
interactions between agents in the scene. Fig. 8 depicts an indicative example
where ContextVAE performs prediction with and without using the S-ATTN
mechanism. In the latter case, we rely only on the agent’s local state
$\mathbf{s}_{i}^{t}$ to sequentially update the RNN hidden state
$\mathbf{q}_{i}^{t}$ of the observation encoder after it has been initialized
using our M-ATTN mechanism. Our S-ATTN mechanism pays attention to the stopped
car in front of the target vehicle and helps the model to make predictions
that are compliant with human driving behavior. Note how the predictive
trajectory distribution accurately captures the motion pattern of the target
vehicle that slows down to avoid colliding with its neighbor as opposed to
when no social attention is used.

To highlight how our unified observation encoding scheme (Section III-C) helps
the model make accurate predictions, we consider three urban traffic examples.
In Fig. 1, we visualize the corresponding activation maps of the map encoder
obtained using the GradCAM technique [31] along with the social attention
weights of the perceived neighbors. In the first example, the model focuses on
the challenging intersection area around the target vehicle and pays high
attention to the leading car along its lane. In the second example, the most
salient map area is the crosswalk (light red polygons) in front of the target
vehicle, where a number of agents are waiting for the traffic lights to turn
green. Hence, the vehicle’s attention focuses on these neighbors while
ignoring irrelevant agents such as the three stopped cars on the outer right
lane. In the third example, the target vehicle is static and the model relies
only on the observed neighbors’ states to make predictions with little
attention paid to the map. These examples demonstrate that our approach can
flexibly utilize the environmental map information and the agents social
states to make predictions.

We refer to the supplementary material and video for additional examples and
results on all three tested datasets.

##  V Conclusion

We have introduced ContextVAE, a fast and lightweight approach for context-
aware vehicle trajectory prediction. ContextVAE relies on a timewise VAE
architecture and employs a map encoding module that performs observation
encoding in a unified and socially-aware way. We show that our approach
provides high-fidelity, map-compliant predictions on a variety of
heterogeneous datasets, capturing the multi-modal nature of vehicle motions
and their interactions with neighbor agents. We note that the model
performance can be further improved by leveraging post-processing techniques
like final position clustering [4] and bootstrap aggregation [32], at the cost
of higher running times. An interesting avenue for future work is to account
for dynamic environmental information like traffic lights in the observation
encoding scheme by introducing local maps timewisely. It is also worth
exploring how map representation influences context encoding and model
performance, including trying alternative ways of map rasterization and using
vectorized road graphs to capture contextual features.

## References

  * [1] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom, “nuscenes: A multimodal dataset for autonomous driving,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 11 621–11 631.
  * [2] J. Houston, G. Zuidhof, L. Bergamini, Y. Ye, L. Chen, A. Jain, S. Omari, V. Iglovikov, and P. Ondruska, “One thousand and one hours: Self-driving motion prediction dataset,” in _Conference on Robot Learning_ , 2021, pp. 409–418.
  * [3] S. Ettinger, S. Cheng, B. Caine, C. Liu, H. Zhao, S. Pradhan, Y. Chai, B. Sapp, C. R. Qi, Y. Zhou, Z. Yang, A. Chouard, P. Sun, J. Ngiam, V. Vasudevan, A. McCauley, J. Shlens, and D. Anguelov, “Large scale interactive motion forecasting for autonomous driving: The Waymo open motion dataset,” in _IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 9710–9719.
  * [4] P. Xu, J.-B. Hayet, and I. Karamouzas, “SocialVAE: Human trajectory prediction using timewise latents,” in _European Conference on Computer Vision_. Springer, 2022.
  * [5] J. L. V. Espinoza, A. Liniger, W. Schwarting, D. Rus, and L. Van Gool, “Deep interactive motion prediction and planning: Playing games with motion prediction models,” in _Learning for Dynamics and Control Conference_ , 2022, pp. 1006–1019.
  * [6] T. Salzmann, B. Ivanovic, P. Chakravarty, and M. Pavone, “Trajectron++: Dynamically-feasible trajectory forecasting with heterogeneous data,” in _European Conference on Computer Vision_ , 2020, pp. 683–700.
  * [7] Y. Yao, E. Atkins, M. Johnson-Roberson, R. Vasudevan, and X. Du, “BiTraP: Bi-directional pedestrian trajectory prediction with multi-modal goal estimation,” _IEEE Robotics and Automation Letters_ , vol. 6, no. 2, pp. 1463–1470, 2021.
  * [8] A. Sadeghian, V. Kosaraju, A. Sadeghian, N. Hirose, H. Rezatofighi, and S. Savarese, “SoPhie: An attentive GAN for predicting paths compliant to social and physical constraints,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 1349–1358.
  * [9] T. Phan-Minh, E. C. Grigore, F. A. Boulton, O. Beijbom, and E. M. Wolff, “CoverNet: Multimodal behavior prediction using trajectory sets,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 14 074–14 083.
  * [10] S. Konev, K. Brodt, and A. Sanakoyeu, “MotionCNN: A strong baseline for motion prediction in autonomous driving,” in _Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_ , 2021.
  * [11] Y. Chai, B. Sapp, M. Bansal, and D. Anguelov, “Multipath: Multiple probabilistic anchor trajectory hypotheses for behavior prediction,” _arXiv preprint arXiv:1910.05449_ , 2019.
  * [12] H. Cui, V. Radosavljevic, F.-C. Chou, T.-H. Lin, T. Nguyen, T.-K. Huang, J. Schneider, and N. Djuric, “Multimodal trajectory predictions for autonomous driving using deep convolutional networks,” in _IEEE International Conference on Robotics and Automation_ , 2019, pp. 2090–2096.
  * [13] N. Deo and M. M. Trivedi, “Trajectory forecasts in unknown environments conditioned on grid-based plans,” _arXiv preprint arXiv:2001.00735_ , 2020\.
  * [14] R. Girgis, F. Golemo, F. Codevilla, M. Weiss, J. A. D’Souza, S. E. Kahou, F. Heide, and C. Pal, “Latent variable sequential set transformers for joint multi-agent motion prediction,” in _International Conference on Learning Representations_ , 2021.
  * [15] L. Zhang, P.-H. Su, J. Hoang, G. C. Haynes, and M. Marchetti-Bowick, “Map-adaptive goal-based trajectory prediction,” in _Conference on Robot Learning_ , 2021, pp. 1371–1383.
  * [16] X. Huang, S. G. McGill, J. A. DeCastro, L. Fletcher, J. J. Leonard, B. C. Williams, and G. Rosman, “DiversityGAN: Diversity-aware vehicle motion prediction via latent semantic sampling,” _IEEE Robotics and Automation Letters_ , vol. 5, no. 4, pp. 5089–5096, 2020.
  * [17] C. Wang, Y. Wang, M. Xu, and D. Crandall, “Stepwise goal-driven networks for trajectory prediction,” _IEEE Robotics and Automation Letters_ , 2022.
  * [18] J. Gao, C. Sun, H. Zhao, Y. Shen, D. Anguelov, C. Li, and C. Schmid, “VectorNet: Encoding HD maps and agent dynamics from vectorized representation,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 11 525–11 533.
  * [19] M. Liang, B. Yang, R. Hu, Y. Chen, R. Liao, S. Feng, and R. Urtasun, “Learning lane graph representations for motion forecasting,” in _European Conference on Computer Vision_. Springer, 2020, pp. 541–556.
  * [20] S. Khandelwal, W. Qi, J. Singh, A. Hartnett, and D. Ramanan, “What-if motion prediction for autonomous driving,” _arXiv preprint arXiv:2008.10587_ , 2020\.
  * [21] B. Kim, S. H. Park, S. Lee, E. Khoshimjonov, D. Kum, J. Kim, J. S. Kim, and J. W. Choi, “LaPred: Lane-aware prediction of multi-modal future trajectories of dynamic agents,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 14 636–14 645.
  * [22] C. Luo, L. Sun, D. Dabiri, and A. Yuille, “Probabilistic multi-modal trajectory prediction with lane attention for autonomous vehicles,” in _IEEE/RSJ International Conference on Intelligent Robots and Systems_ , 2020, pp. 2370–2376.
  * [23] H. Zhao, J. Gao, T. Lan, C. Sun, B. Sapp, B. Varadarajan, Y. Shen, Y. Shen, Y. Chai, C. Schmid _et al._ , “TNT: Target-driven trajectory prediction,” in _Conference on Robot Learning_ , 2021, pp. 895–904.
  * [24] Q. Sun, X. Huang, J. Gu, B. C. Williams, and H. Zhao, “M2I: From factored marginal trajectory prediction to interactive prediction,” in _Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 6543–6552.
  * [25] N. Deo, E. Wolff, and O. Beijbom, “Multimodal trajectory prediction conditioned on lane-graph traversals,” in _5th Annual Conference on Robot Learning_ , 2021.
  * [26] S. Casas, C. Gulino, S. Suo, and R. Urtasun, “The importance of prior knowledge in precise multimodal prediction,” in _IEEE/RSJ International Conference on Intelligent Robots and Systems_ , 2020, pp. 2295–2302.
  * [27] W. Zeng, M. Liang, R. Liao, and R. Urtasun, “LaneRCNN: Distributed representations for graph-centric motion forecasting,” in _IEEE/RSJ International Conference on Intelligent Robots and Systems_ , 2021, pp. 532–539.
  * [28] T. Gilles, S. Sabatini, D. Tsishkou, B. Stanciulescu, and F. Moutarde, “Gohome: Graph-oriented heatmap output for future motion estimation,” in _IEEE International Conference on Robotics and Automation_ , 2022, pp. 9107–9114.
  * [29] J. Amirian, J.-B. Hayet, and J. Pettré, “Social Ways: Learning multi-modal distributions of pedestrian trajectories with GANs,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_ , 2019.
  * [30] J. Ngiam, V. Vasudevan, B. Caine, Z. Zhang, H.-T. L. Chiang, J. Ling, R. Roelofs, A. Bewley, C. Liu, A. Venugopal _et al._ , “Scene Transformer: A unified architecture for predicting future trajectories of multiple agents,” in _International Conference on Learning Representations_ , 2021.
  * [31] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, “Grad-CAM: Visual explanations from deep networks via gradient-based localization,” in _IEEE International Conference on Computer Vision_ , 2017, pp. 618–626.
  * [32] B. Varadarajan, A. Hefny, A. Srivastava, K. S. Refaat, N. Nayakanti, A. Cornman, K. Chen, B. Douillard, C. P. Lam, D. Anguelov _et al._ , “Multipath++: Efficient information fusion and trajectory aggregation for behavior prediction,” in _IEEE International Conference on Robotics and Automation_ , 2022, pp. 7814–7821.

See pages - of appendix_pdf


# Zero-shot Bilingual App Reviews Mining with Large Language Models

Jialiang Wei∗, Anne-Lise Courbis∗, Thomas Lambolais∗,
Binbin Xu∗, Pierre Louis Bernard∗∗ and Gérard Dray∗
∗: EuroMov Digital Health in Motion, Univ Montpellier, IMT Mines Ales, Ales,
France
∗∗: EuroMov Digital Health in Motion, Univ Montpellier, IMT Mines Ales,
Montpellier, France
∗<EMAIL_ADDRESS>∗∗<EMAIL_ADDRESS>
###### Abstract

App reviews from app stores are crucial for improving software requirements. A
large number of valuable reviews are continually being posted, describing
software problems and expected features. Effectively utilizing user reviews
necessitates the extraction of relevant information, as well as their
subsequent summarization. Due to the substantial volume of user reviews,
manual analysis is arduous. Various approaches based on natural language
processing (NLP) have been proposed for automatic user review mining. However,
the majority of them requires a manually crafted dataset to train their
models, which limits their usage in real-world scenarios.

In this work, we propose Mini-BAR, a tool that integrates large language
models (LLMs) to perform zero-shot mining of user reviews in both English and
French. Specifically, Mini-BAR is designed to (i) classify the user reviews,
(ii) cluster similar reviews together, (iii) generate an abstractive summary
for each cluster and (iv) rank the user review clusters. To evaluate the
performance of Mini-BAR, we created a dataset containing 6,000 English and
6,000 French annotated user reviews and conducted extensive experiments.
Preliminary results demonstrate the effectiveness and efficiency of Mini-BAR
in requirement engineering by analyzing bilingual app reviews.

##  I Introduction

App stores, such as Google Play and Apple App Store, allow users to express
their feedback on downloaded apps. This feedback is in form of rating scores
and text reviews. The latter contains praise and dispraise, user experience,
problem reports, and feature requests [1]. App reviews are important for app
success. As evidenced in prior literature, high-user rating scores have
positive effects on apps’ sustainability [2]. Consequently, the design and
development teams should take the users’ feedback into consideration during
the evolution of their application.

Due to the large amount and the redundancy of app reviews, manual analysis is
laborious. Various approaches based on natural language processing (NLP) have
been proposed to reduce the efforts in analyzing user feedback, including the
classification, clustering and summarization. Classification models are
commonly employed in the first approach to categorize app reviews into
predefined groups, such as feature requests and problem reports [3, 4, 5, 6,
7]. However, even after classification, the volume of reviews within each
category remains substantial, making direct analysis impractical. To tackle
this issue, some researchers have proposed grouping reviews that pertain to
the same topic [8, 9, 10, 11]. As the obtained clusters still contain a
relatively large number of user reviews, the manual analysis of each cluster
continues to be time-consuming. Certain techniques attempt to overcome this
challenge by selecting the most representative phrases or sentences as
summaries for groups of app reviews [9, 10, 12, 13, 14]. Nevertheless, this
extractive summarization approach may not capture all the crucial information
present within a given group. Moreover, existing approaches in user review
analysis mainly focus on the English language, with few works on analyzing
reviews in other languages [15, 16]. Furthermore, most existing approaches
requires manually crafted dataset for training their classification models.
The creation of dataset is costly and time consuming, which limits their usage
in real-word scenarios. The objective of this article is therefore to address
these gaps with large language models.

Pre-trained language models (PTMs) are deep neural networks previously trained
on a vast corpus. Researchers have observed that large-sized PTMs display
different behaviors from smaller PTMs and show surprising abilities (called
emergent abilities) in solving a series of complex tasks. Thus, the research
community coins the term “large language models (LLMs)” for these large-sized
PTMs [17]. A remarkable application of LLMs is
ChatGPT111https://openai.com/blog/chatgpt/, it is fine-tuned from the GPT-3.5
[18] using Reinforcement Learning from Human Feedback (RLHF), which optimizes
the model by interacting with human and learning from human preference. The
Guanaco model is an open-source, finely-tuned LLM, derived through the
application of QLoRa’s 4-bit tuning approach [19] on LLaMA base models [20].
QLoRA is an efficient fine-tuning approach that reduces memory usage. Guanaco
is available in various parameter sizes, including 7B, 13B, 33B and 65B.

In this paper, we propose Mini-BAR, a bilingual approach based on LLMs to: (i)
classify the user reviews into three categories: feature request, problem
report and irrelevant; (ii) cluster similar reviews for feature request and
problem report; (iii) generate a summary for each cluster of user reviews; and
(iv) rank the user review clusters. Figure 1 depicts an overview of the
workflow of Mini-BAR. We use the same pipeline to process the bilingual app
reviews, eliminating the necessity of deploying separate models for each
language. By combining these functionalities, Mini-BAR provides a
comprehensive approach for analyzing bilingual app reviews, which can yield
valuable insights for app developers and marketers. We validate the key steps
of Mini-BAR by conducting an extensive set of experiments on 12000 annotated
user reviews from three Health & Fitness apps. The results indicate that Mini-
BAR has a satisfactory performance in both classification and clustering
tasks, and produced high-quality summaries. We provide a replication
package222https://github.com/Jl-wei/mini-bar containing the code, dataset, and
experiment setups.

##  II Approach

Mini-BAR provides support to developers for the analysis of mobile app user
reviews through a four-step process. First, it applies a pre-trained
classifier to categorize the user reviews (Section II-A). The second step
clusters the user reviews based on their semantic similarity (Section II-B).
The third step summarizes the user reviews belonging to the same cluster
(Section II-C). The last step is to determine the importance of user review
clusters and rank them accordingly (Section II-D). In the following
subsections, we will detail each step of Mini-BAR.

Figure 1: Overview of Mini-BAR

###  II-A Classification

The objective of this step is to automatically classify English and French
user reviews into three categories: (F) feature request, (B) problem report,
and (I) irrelevant. A user review may belong to either one of the three
categories or both feature request and problem report. A user review is
considered as problem report if it mentions the issues the users have
experienced while using the app (e.g., “Can’t sync sleep data since last
update”). Feature requests reflect users’ needs for new functions, new
content, or improvements (e.g., “Please bring a feature to add some custom
watch faces …”). All the other user reviews are irrelevant (e.g., “Best app
ever!”). Classifying the reviews can aid to redirect them to the appropriate
software project members. For instance, feature requests can be delivered to
requirements analysts, while problem reports can be directed to developers and
testers [4].

The classifier of Mini-BAR is based on ChatGPT, the model we use is
gpt-3.5-turbo333https://platform.openai.com/docs/models/gpt-3-5. We use the
following prompt to classify the app reviews.

Classify the following {lang} app review into problem report, feature request
or irrelevant. Be concise.
‘‘‘
{review}
‘‘‘

Given a user review, its language is detected automatically with
Lingua444https://github.com/pemistahl/lingua-py, which is an accurate language
detector. The detected language, which could be English, French among others,
replaces the $\\{lang\\}$ variable in the prompt. And the $\\{review\\}$ in
the prompt is replaced by the user review. The response of ChatGPT contains a
single-phrase label name. We parse the response with regular expressions to
automatically obtain the predicted labels.

###  II-B Clustering

The objective of this step is to group English and French user reviews based
on their semantic similarity, ensuring that reviews within a group are related
to the same topic. Through clustering analysis, texts are divided into
clusters such that those within a cluster exhibit semantic similarity.
Currently, the RE community predominantly focuses on clustering English user
reviews, leaving little attention to non-English user reviews. To address this
gap, we propose a bilingual clustering approach that allows the creation of
clusters comprising reviews from different languages that share common topics.

Figure 2: Overview of clustering

In this step, we perform bilingual clustering analysis on user reviews that
belong to the same category, namely feature request or problem report, which
were identified in the previous step. It is worth noting that in the
clustering process, the categories of feature requests and problem reports are
processed separately, in order to obtain distinct clusters for each.
Conversely, user reviews labeled as irrelevant are excluded from the
clustering analysis.

The user reviews cannot be directly used as input for the clustering
algorithm, as they are in a textual format. Therefore, it is necessary to
convert them into embeddings — numerical vectors in a high-dimensional space —
to enable effective processing. In this space, similar inputs in different
languages are mapped close together. For example, the embeddings of “Problème
de serveur récurrent” and “Connection issues to the main server” are in
proximity to each other. We used Instructor [21] to embed English and French
app reviews due to its high performance proven in Section III-B3. Instructor
is an instruction-finetuned text embedding model that can generate text
embeddings tailored to any task (e.g. clustering) by simply providing the task
instruction, without any finetuning. In our case, we have utilized the
instruction ”Represent the app user review for clustering”. This model
generates 768-dimensional embeddings for each app review.

The high dimension of embeddings causes high computation costs. Dimension
reduction techniques can transforms data from a high-dimensional space into a
low-dimensional space and keeps meaningful information of the original data.
As in Stanik et al. [9], we reduced the embeddings’ dimension with Uniform
Manifold Approximation and Projection (UMAP). The implementation of Mini-BAR
utilizes the UMAP Python package555https://github.com/lmcinnes/umap, the UMAP
parameters are as follows: output dimensionality of $20$, number of neighbors
set to $100$, and minimum distance of $0$.

The reduced embeddings of the feature requests and problem reports are then
clustered by HDBSCAN [22], which has been proven efficient by Devine et al.
[10] and Stanik et al. [9]. We use the HDBCAN666https://github.com/scikit-
learn-contrib/hdbscan Python package for the implementation of Mini-BAR. The
HDBSCAN parameters include a minimum cluster size of 5. The minimum cluster
size is the smallest grouping size considered as a cluster. In our study, we
chose a minimum cluster size of 5, as we were interested in identifying
problems or features that were reported by at least 5 users.

###  II-C Summarization

Given the potential magnitude of reviews within clusters, the process of
summarization becomes imperative, enabling developers to efficiently grasp the
cluster’s contents without the need to peruse every individual review. Large
language models (LLMs), such as ChatGPT, has achieved a state-of-the-art
performance in cross-lingual summarization [23]. In this step, we utilized
ChatGPT (gpt-3.5-turbo) to generate abstractive English summaries for clusters
containing bilingual user reviews. Our evaluation in Section III-C proves that
ChatGPT outperforms extractive summarization method, and is able to generate
high-quality summaries.

To generate the summaries, we utilized the following prompt within ChatGPT.
The $\\{reviews\>list\\}$ is replace by a list of user reviews separated by
new line break. The response generated by ChatGPT represents the summary for
that list of user reviews. Table I presents an example of a manual created
cluster and its generated summary.

Please summarize all following app reviews into one English sentence:
‘‘‘
{reviews list}
‘‘‘

Figure 3: Overview of summarization for large clusters

However, in case the user review clusters contain a large number of reviews
and exceed the input length limitation of ChatGPT, it will issue a warning
message indicating that “The message you submitted was too long, please reload
the conversation and submit something shorter”. To address this issue, as
illustrated in Figure 3, we adopted a hierarchical summarization approach
consisting of the following steps: (i) dividing the reviews belonging to one
cluster into multiple groups, each with a maximum of 4000
tokens777https://platform.openai.com/docs/guides/gpt/chat-completions-vs-
completions, (ii) generating a sub-summary for each group of user reviews, and
(iii) obtaining the final summary by summarizing the sub-summaries. Since the
length of all sub-summaries may also exceed the input limit of ChatGPT, we
used a recursive procedure for steps (i) and (ii) by replacing the user
reviews with sub-summaries.

TABLE I: Example of a user review cluster and its generated summary User
reviews:
---
\- Dommage que la connexion 4g soit indispensable pour fonctionner.
\- Please for god sake make it to work offline also.
\- Is not work offline
\- It used to work offline. Now I have to log in just to see my old data.
\- Useless without internet.
Summary:
Users are disappointed that the app requires an internet connection to
function and wish it could work offline like it used to.

###  II-D Ranking

Given the clusters with summaries, the next step is to rank them by their
importance. The goal of this step is to aid in the release planning of app
developers. The importance of a cluster ($\mathit{ClusterScore}$) is
determined based on the following characters:

  * •

The number of reviews present within the cluster ($|\mathit{reviews}|$).
problems or feature requests reported by more users should be given higher
priority compared to those reported by fewer users.

  * •

The average rating of the cluster ($\overline{\mathit{rating}}$). Clusters
with lower average ratings should be given higher priority, as they may
indicate users’ greater dissatisfaction with specific aspects of the app.

  * •

The number of “thumbs up” inside the cluster ($|\mathit{thumbsup}|$). Users on
Google Play have the option to click the “thumbs up” button on reviews that
they find helpful. We posit that the number of “thumbs up” and the importance
of a cluster are positively correlated as the review liked by more users
should have a higher priority.

Given the weight of $w_{rev}$, $w_{th}$ and $w_{ra}$, which are assigned
default values of $1$, $0.1$, and $1$, respectively, the calculation of
$\mathit{ClusterScore}$ is defined as follows:

$\displaystyle\mathit{ClusterScore}$
$\displaystyle=\frac{w_{\mathit{rev}}\cdot|\mathit{reviews}|+w_{th}\cdot|\mathit{thumbsup}|}{w_{\mathit{ra}}\cdot\overline{\mathit{rating}}}$
(1)

The clusters are ranked in decreasing order of $\mathit{ClusterScore}$.

##  III Empirical Evaluation

The objective of this study is to assess the performance of Mini-BAR with
respect to three criteria: (i) its accuracy in classifying user reviews into
one of three predefined categories, namely feature request, problem report,
and irrelevant; (ii) its ability to cluster related user reviews that fall
into the same topic; (iii) its ability to provide high-quality summaries of
user reviews clusters. To achieve this goal, we evaluated Mini-BAR’s
performance on a dataset of 6000 English and 6000 French reviews from three
health-related mobile apps.

###  III-A Evaluation of Classification

In this section, we aim to answer the following research question (RQ1): How
accurate is Mini-BAR in classifying bilingual user reviews ?

####  III-A1 Dataset

The training and evaluation of the classifier require a large number of
labeled user reviews. We relabelled the $6000$ French reviews of three
applications (Garmin Connect, Huawei Health and Samsung Health) on Google Play
from our previous work [16]. Besides, we collected $365,967$ English user
reviews from these three applications. For each application, $2000$ English
are randomly sampled for annotation. In this work, we have labeled $6000$
English reviews and $6000$ French reviews.

We used Prodigy888https://prodi.gy/ from spaCy to annotate the user reviews.
We created an annotation guide to clarify the definition of feature request,
problem report, and irrelevant. Four authors of this paper annotated the
sampled user reviews and they are finally reviewed by the first author of this
paper. Table II shows the details of the annotated dataset. The sum of each
category does not equal the total of reviews, as some reviews have been
assigned to more than one label.

####  III-A2 Evaluation Metrics

The performance of the classifiers is evaluated by precision, recall, and F1
as presented in related work [15, 4].

####  III-A3 Experiments

In this experiment, we compared the performance of ChatGPT and Guanaco-33B
with ML models (Random Forest, Support Vector Machine), as well as various
PTMs (BERT [24], CamemBERT[25], XLM-R [26]), on the classification of app
reviews.

The ML models are trained using batch gradient descent, while PTMs employed
mini-batch gradient descent, with a batch size of 12 and AdamW optimizer with
a learning rate of $2e^{-5}$. They are trained on 3 epochs on a machine with a
NVIDIA Tesla T4 GPU with 16 GB VRAM. The user reviews from the three apps of
both languages were split using an 80:20 (training and test sets) ratio in a
stratified manner, as illustrated in Figure 4. We trained the classifiers
using a combination of en_train and fr_train. Subsequently, the classifiers
were individually tested on the en_test and fr_test. We performed ten-fold
cross-validation by randomly splitting the training and test sets ten times,
and computed the average performance across these runs.

TABLE II: Overview of the dataset for classification App | Language | Total |  Feature request |  problem report | Irrelevant
---|---|---|---|---|---
| en | $2000$ | $223$ | $579$ | $1231$
Garmin Connect | fr | $2000$ | $217$ | $772$ | $1051$
| en | $2000$ | $415$ | $876$ | $764$
Huawei Health | fr | $2000$ | $387$ | $842$ | $817$
| en | $2000$ | $528$ | $500$ | $990$
Samsung Health | fr | $2000$ | $496$ | $492$ | $1047$

Figure 4: Overview of dataset split for training and testing

We conduct classifications utilizing LLMs, specifically ChatGPT and
Guanaco-33B, on all 12,000 user reviews. This is executed under zero-shot
setting, implying that no prior training is involved. We have also assessed
the performance of Guanaco-13B and Guanaco-65B. However, the responses
generated by Guanaco-13B are disorganized, thereby hindering the extraction of
predicted labels using regular expressions. The inference of Guanaco-65B is
intolerably slow, even on advanced hardware such as the NVIDIA A100, making
its usage impractical.

TABLE III: Classification accuracy on user reviews of three apps |  | Feature Request | Problem Report | Irrelevant | Average Weight
---|---|---|---|---|---
| $\mathit{Precision}$ | $\mathit{Recall}$ | $\mathit{F1}$ | $\mathit{Precision}$ | $\mathit{Recall}$ | $\mathit{F1}$ | $\mathit{Precision}$ | $\mathit{Recall}$ | $\mathit{F1}$ | $\mathit{Precision}$ | $\mathit{Recall}$ | $\mathit{F1}$
| Random Forest | 0.75 | 0.453 | 0.564 | 0.797 | 0.82 | 0.808 | 0.898 | 0.885 | 0.891 | 0.837 | 0.782 | 0.802
| SVM | 0.86 | 0.438 | 0.58 | 0.86 | 0.806 | 0.832 | 0.931 | 0.893 | 0.912 | 0.895 | 0.778 | 0.823
| BERT | 0.814 | 0.782 | 0.797 | 0.897 | 0.914 | 0.905 | 0.972 | 0.954 | 0.963 | 0.918 | 0.909 | 0.913
| CamemBERT | 0.811 | 0.743 | 0.775 | 0.883 | 0.894 | 0.888 | 0.966 | 0.951 | 0.958 | 0.91 | 0.893 | 0.901
| XLM-R | 0.823 | 0.811 | 0.816 | 0.902 | 0.917 | 0.909 | 0.979 | 0.958 | 0.968 | 0.925 | 0.917 | 0.92
| ChatGPT | 0.768 | 0.5 | 0.606 | 0.762 | 0.972 | 0.854 | 0.983 | 0.911 | 0.945 | 0.871 | 0.853 | 0.852
En | Guanaco-33B | 0.361 | 0.62 | 0.456 | 0.662 | 0.951 | 0.781 | 0.983 | 0.817 | 0.893 | 0.763 | 0.823 | 0.774
| Random Forest | 0.8 | 0.528 | 0.635 | 0.798 | 0.834 | 0.816 | 0.902 | 0.869 | 0.885 | 0.848 | 0.796 | 0.817
| SVM | 0.895 | 0.459 | 0.606 | 0.86 | 0.828 | 0.844 | 0.956 | 0.89 | 0.922 | 0.912 | 0.791 | 0.838
| BERT | 0.766 | 0.725 | 0.744 | 0.871 | 0.866 | 0.869 | 0.947 | 0.931 | 0.939 | 0.888 | 0.872 | 0.88
| CamemBERT | 0.852 | 0.823 | 0.837 | 0.922 | 0.925 | 0.923 | 0.977 | 0.96 | 0.968 | 0.936 | 0.924 | 0.929
| XLM-R | 0.819 | 0.833 | 0.825 | 0.917 | 0.921 | 0.919 | 0.982 | 0.949 | 0.965 | 0.93 | 0.919 | 0.924
| ChatGPT | 0.853 | 0.473 | 0.608 | 0.782 | 0.973 | 0.868 | 0.978 | 0.935 | 0.956 | 0.888 | 0.866 | 0.863
Fr | Guanaco-33B | 0.296 | 0.576 | 0.391 | 0.624 | 0.97 | 0.759 | 0.985 | 0.756 | 0.856 | 0.737 | 0.797 | 0.739

####  III-A4 Results

The experiment results presented in Table III demonstrate that the ChatGPT
exhibited good overall performance, which is comparable to that of ML models.
Its comparatively lower performance in classifying feature requests can be
attributed to the inherent complexity associated with such requests. In
certain instances, users may express their desire for new features by
criticizing existing ones or complaining about missing functionalities, rather
than straightforwardly stating ”I need…”. Among all the models, XLM-R archived
the best performance in bilingual classification. Although the performance of
ChatGPT does not match up to PTMs, it is noteworthy that these LLMs have
achieved such performance without the utilization of any reviews during their
training phase. This suggests that ChatGPT can achieve satisfactory
performance in user reviews of other application categories.

###  III-B Evaluation of Clustering

In this section, our objectives are to address the $RQ_{2}$: How semantically
meaningful are the clusters generated by Mini-BAR?

####  III-B1 Dataset

In order to evaluate the performance of clustering algorithms, we created a
dataset with 1200 user reviews. We randomly selected 100 problem reports and
100 feature requests from each of the three apps in each of the two languages
present in the dataset created in Section III-A1. Then the authors employed
manual clustering for each collection of 200 bilingual reviews, all of which
pertained to an identical category. Reviews sharing the same topic were
subsequently grouped into a single cluster. In instances where a user’s review
mentioned multiple topics, the assignment of the cluster was determined by the
initial topic that was reported. Two authors independently performed the
clustering of the 1200 reviews, and their individual results were later merged
through discussion. The resulting clusters were then considered as the ground
truth for subsequent evaluation.

Table IV shows the number of manually created clusters and the number of
clusters whose size is greater than or equal to 5 in each category and app.
The feature requests encompass enhancements such as the modification of a
Graphical User Interface (GUI), support for additional languages, increased
activity options, integration with other applications, customization of
permissions, and improvement of sleep tracking function. The problem reports
predominantly center on a series of issues, notably the application’s
unexpected crashes, errors encountered during the login process, difficulties
in pairing with smartwatches, challenges with data synchronization,
inconsistencies in the notification system, and complications related to
Bluetooth connectivity, among others.

TABLE IV: Overview of manually created clusters Bilingual |  Garmin Connect |  Huawei Health |  Samsung Health
---|---|---|---
#clusters in feature request |  89 |  74 |  69
#clusters($size\geq 5$) in feature request |  7 |  9 |  11
#clusters in problem report |  45 |  44 |  41
#clusters($size\geq 5$) in problem report |  10 |  13 |  12

####  III-B2 Evaluation Metrics

Following previous work [11], we use two commonly used indices, Normalized
Mutual Information (NMI) [27] and Adjusted Rand Index (ARI) [28], to quantify
the similarity between the automatic clustering and the ground truth. NMI
ranges from 0 to 1, while ARI ranges from -1 to 1. A higher NMI or ARI
indicates that the clustering method is more effective in producing clusters
that align with the ground truth. Note that the NMI and ARI are computed for
clusters with a size of 5 or greater, as the minimum cluster size of HDBSCAN
is set to 5.

####  III-B3 Experiments and Results

In this section, we evaluate the performance of different text representation
methods (including traditional frequency-based methods, bag of words (BOW),
and TF-IDF, as well as PTM-based methods, Universal Sentence Encoder [29],
MiniLM [30], MPNet [31], E5 [32] and Instructor [21]) on the dataset created
in Section III-B1. We performed three distinct experiments on English-only,
French-only and bilingual user reviews. In each experiment, clustering is
executed separately for the two categories (feature requests and problem
reports) within each of the three applications.

####  III-B4 Results

The average NMI and ARI of the three distinct experiments are shown in Table
V. Results show that PTM-based methods outperformed traditional methods. Among
all text representation methods, Instructor demonstrated the highest level of
performance in clustering. The results from the English user reviews
clustering were superior to those derived from the bilingual and French user
reviews clustering. We attribute this variation to the relatively small French
corpus employed for training the PTMs.

TABLE V: Evaluation on user reviews clustering Embedding | NMI | ARI
---|---|---
Methods | en | fr | bi | en | fr | bi
BOW | 0.450 | 0.405 | 0.417 | 0.191 | 0.155 | 0.103
TF-IDF | 0.452 | 0.449 | 0.460 | 0.253 | 0.216 | 0.149
USE | 0.575 | 0.501 | 0.552 | 0.337 | 0.330 | 0.236
MiniLM | 0.548 | 0.567 | 0.541 | 0.323 | 0.380 | 0.219
MPNet | 0.616 | 0.575 | 0.593 | 0.400 | 0.346 | 0.278
E5 | 0.465 | 0.401 | 0.436 | 0.248 | 0.190 | 0.139
Instructor | 0.713 | 0.587 | 0.603 | 0.597 | 0.357 | 0.308

While the proposed approach appears to have some degree of validity, the
results do not appear to be particularly encouraging at this stage. The
primary reason for this is that our evaluation dataset is relatively small.
Clustering algorithms require a sufficient amount of data to discover
underlying patterns and structures. With a limited amount of data, it becomes
difficult to identify meaningful groupings of text. Moreover, text clustering
is a challenging task, particularly given the informal nature of the
terminology employed in user reviews and the prevalence of spelling errors.
Additionally, based on our empirical analysis, it appears that longer user
reviews tend to result in less accurate clustering. This presents an
opportunity for potential improvement by applying sentence-level clustering to
user reviews.

###  III-C Evaluation of Summarization

In this section, we aim to investigate the $RQ_{3}$: : How effectively does
ChatGPT perform in summarizing bilingual user reviews?

####  III-C1 Dataset

In Section III-B1, we carried out manual clustering for a total of 1200 user
reviews. Among these, we utilized clusters with a size of 5 or greater to
assess the performance of ChatGPT on summarization.

####  III-C2 Evaluation Metrics

As outlined in Fabbri et al. [33], human evaluators rate the generated
summaries based on four dimensions: relevance (the degree to which crucial
information from the source has been included), consistency (how well the
summary aligns with the factual details of the source), fluency (the quality
of individual sentences), and coherence (the overall quality and coherence of
all the sentences in the summary). Each dimension is scored on a Likert scale
ranging from 1 to 5, with higher scores indicating superior performance.

####  III-C3 Experiments

To evaluate the proficiency of ChatGPT in producing succinct English summaries
of bilingual app reviews, we compare it with baseline approaches: Extractive
summarization of Devine et al. [10], which selects the most representative
sentence from a cluster of app reviews as the summary, the sentence is chosen
by calculating the similarity with all other sentences of that cluster.
Abstractive summarization with Guanaco models (13B, 33B and 65B version of
Guanaco are used in our experiments) [19]. ChatGPT and Guanaco were instructed
to synthesize clusters of user reviews into a single English sentence with the
same prompt, as presented in Section II-C. Subsequently, human evaluators
assessed the generated summaries. Given the pairs of user reviews and
corresponding summaries, two authors of this paper were asked to evaluate the
summaries on the Likert scale in the four dimensions that were previously
mentioned.

TABLE VI: Human evaluation on generated summaries | Relevance | Consistency | Fluency | Coherence
---|---|---|---|---
Devine et al. [10] | 4.23 | 4.83 | 4.62 | 4.71
Guanaco-13B | 3.67 | 3.68 | 4.92 | 4.91
Guanaco-33B | 4.65 | 4.58 | 4.91 | 4.88
Guanaco-65B | 4.79 | 4.77 | 4.95 | 4.94
ChatGPT | 4.81 | 4.84 | 4.95 | 4.94

####  III-C4 Results

Table VI presents the average results of hand-made evaluations. Results show
that abstractive approaches (ChatGPT, Guanaco-33B, and Guanaco-65B) generate
very high quality sentences, and they are highly coherent when viewed in
conjunction with one another. On the other hand, the extractive approach
excels at extracting the most significant information from a cluster; however,
it falls short in capturing all the essential details. During our manual
evaluation, we found that Guanaco-13B tends to retrieve all available
information available in the cluster without selectively focusing on crucial
elements. In contrast, Guanaco-33B performs significantly better in this
regard by effectively filtering out non-essential information. The results
highlight the impressive performance of ChatGPT and Guanaco-65B in generating
highly satisfactory summaries of user reviews.

##  IV Threats to validity

This section aims to identify potential threats to the validity of our study.

####  IV-1 App reviews from one category

All reviews studied in this paper are collected from three Health & Fitness
apps (Garmin Connect, Huawei Health, and Samsung Health), mainly due to the
context of health activity monitoring project. Instead of analyzing small
number of reviews in many apps, we choose to annotate 12,000 reviews on three
apps to create a larger dataset to evaluate clustering and summarization.
However, these three apps may not be representative of apps in other
categories. In the future, we will alleviate this threat by investigating user
reviews of apps in various categories.

####  IV-2 Subjectivity in manual annotation

The annotation of user reviews is a straightforward task, people without
specific training can well classify or cluster the reviews. However,
subjectivity can still arise during manual annotation, leading to variations
in how different annotators interpret and label the reviews. To mitigate this
threat, we (i) created a annotation guideline to detail the definition of each
label following with examples, (ii) reviewed the final label through
discussion and consensus.

####  IV-3 Issues of using ChatGPT

ChatGPT has been selected as the classification and summarization component of
Mini-BAR, owing to its superior capabilities. However, it is noteworthy to
mention that certain countries have imposed prohibitions on the use of
ChatGPT. Some users may refrain from utilizing ChatGPT owing to concerns
pertaining to data privacy. And the cost of analyzing user reviews using
ChatGPT cannot be overlooked, particularly in light of the large volume of
user reviews. To mitigate these challenges, we have implemented alternative
strategies. For classification, Mini-BAR users can utilize our XLM-R
checkpoint, which has been trained on 12,000 reviews, or they can opt for
Guanaco-33B. For summarization tasks, users have the option to use other large
language models, specifically Guanaco-33B or Guanaco-65B.

##  V Conclusion

This paper introduces Mini-BAR, a mobile app review mining tool designed to
assist app developers in extracting and summarizing user-reported issues and
requests from a huge number of app reviews. This tool is based on LLMs and
operates under zero-shot setting. Our empirical evaluation on the key steps of
Mini-BAR resulted in numerous positive outcomes: (i) it accurately classified
bilingual user reviews with an F1 score of $0.85$; (ii) it created meaningful
clusters of bilingual user reviews with a NMI greater than $0.6$; (iii) it
produced highly satisfactory summaries of bilingual user reviews.

In our future research, we intend to: (i) conduct a comparative analysis of
various prompts utilized for classification and summarization, (ii) implement
alternative large-scale language models for classification and summarization,
(iii) execute classification and clustering at the sentence level as opposed
to the review level, (iv) undertake evaluations using user reviews sourced
from applications across a diverse range of categories.

## References

  * [1] D. Pagano and W. Maalej, “User feedback in the appstore: An empirical study,” in _2013 21st IEEE International Requirements Engineering Conference (RE)_ , 2013, pp. 125–134.
  * [2] G. Lee and T. S. Raghu, “Determinants of Mobile Apps’ Success: Evidence from the App Store Market,” _Journal of Management Information Systems_ , vol. 31, no. 2, pp. 133–170, 2014.
  * [3] N. Chen, J. Lin, S. C. Hoi _et al._ , “AR-miner: Mining informative reviews for developers from mobile app marketplace,” _Proceedings - International Conference on Software Engineering_ , no. 1, pp. 767–778, 2014.
  * [4] W. Maalej, Z. Kurtanović, H. Nabil, and C. Stanik, “On the automatic classification of app reviews,” _Requirements Engineering_ , vol. 21, no. 3, pp. 311–331, 2016.
  * [5] R. R. Mekala, A. Irfan, E. C. Groen _et al._ , “Classifying User Requirements from Online Feedback in Small Dataset Environments using Deep Learning,” in _2021 IEEE 29th International Requirements Engineering Conference (RE)_ , 2021, pp. 139–149.
  * [6] P. R. Henao, J. Fischbach, D. Spies _et al._ , “Transfer Learning for Mining Feature Requests and Bug Reports from Tweets and App Store Reviews,” in _2021 IEEE 29th International Requirements Engineering Conference Workshops (REW)_ , 2021, pp. 80–86.
  * [7] J. Zhang, Y. Chen, N. Niu, and C. Liu, “A Preliminary Evaluation of ChatGPT in Requirements Information Retrieval,” no. 2022, pp. 1–16. [Online]. Available: http://arxiv.org/abs/2304.12562 2023\.
  * [8] S. Scalabrino, G. Bavota, B. Russo _et al._ , “Listening to the Crowd for the Release Planning of Mobile Apps,” _IEEE Transactions on Software Engineering_ , vol. 45, no. 1, pp. 68–86, 2019.
  * [9] C. Stanik, T. Pietz, and W. Maalej, “Unsupervised Topic Discovery in User Comments,” in _2021 IEEE 29th International Requirements Engineering Conference (RE)_ , 2021, pp. 150–161.
  * [10] P. Devine, J. Tizard, H. Wang _et al._ , “What’s Inside a Cluster of Software User Feedback: A Study of Characterisation Methods,” in _2022 IEEE 30th International Requirements Engineering Conference (RE)_ , 2022, pp. 189–200.
  * [11] Y. Wang, J. Wang, H. Zhang _et al._ , “Where is Your App Frustrating Users?” _Proceedings - International Conference on Software Engineering_ , vol. 2022-May, pp. 2427–2439, 2022.
  * [12] A. D. Sorbo, S. Panichella, C. V. Alexandru _et al._ , “What would users change in my App? Summarizing app reviews for recommending software changes,” _Proceedings of the ACM SIGSOFT Symposium on the Foundations of Software Engineering_ , vol. 13-18-Nove, pp. 499–510, 2016.
  * [13] M. Alshangiti, W. Shi, E. Lima _et al._ , “Hierarchical Bayesian multi-kernel learning for integrated classification and summarization of app reviews,” _ESEC/FSE 2022 - Proceedings of the 30th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ , pp. 558–569, 2022.
  * [14] C. Gao, Y. Li, S. Qi _et al._ , “Listening to Users’ Voice: Automatic Summarization of Helpful App Reviews,” _IEEE Transactions on Reliability_ , pp. 1–13, 2022.
  * [15] C. Stanik, M. Haering, and W. Maalej, “Classifying Multilingual User Feedback using Traditional Machine Learning and Deep Learning,” in _2019 IEEE 27th International Requirements Engineering Conference Workshops (REW)_ , 2019, pp. 220–226.
  * [16] J. Wei, A.-L. Courbis, T. Lambolais _et al._ , “Towards a Data-Driven Requirements Engineering Approach: Automatic Analysis of User Reviews,” in _7th National Conference on Practical Applications of Artificial Intelligence_ , 2022. [Online]. Available: http://arxiv.org/abs/2206.14669
  * [17] W. X. Zhao, K. Zhou, J. Li _et al._ , “A Survey of Large Language Models,” pp. 1–85. [Online]. Available: http://arxiv.org/abs/2303.18223 2023\.
  * [18] T. Brown, B. Mann, N. Ryder _et al._ , “Language Models are Few-Shot Learners,” in _Advances in Neural Information Processing Systems_ , H. Larochelle, M. Ranzato, R. Hadsell _et al._ , Eds., vol. 33. Curran Associates, Inc., 2020, pp. 1877–1901.
  * [19] T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, “QLoRA: Efficient Finetuning of Quantized LLMs.” [Online]. Available: http://arxiv.org/abs/2305.14314 2023\.
  * [20] H. Touvron, T. Lavril, G. Izacard _et al._ , “Llama: Open and efficient foundation language models,” _arXiv preprint arXiv:2302.13971_ , 2023.
  * [21] H. Su, W. Shi, J. Kasai _et al._ , “One Embedder, Any Task: Instruction-Finetuned Text Embeddings.” [Online]. Available: http://arxiv.org/abs/2212.09741 2022\.
  * [22] L. McInnes, J. Healy, and S. Astels, “hdbscan: Hierarchical density based clustering,” _The Journal of Open Source Software_ , vol. 2, no. 11, p. 205, 2017.
  * [23] J. Wang, Y. Liang, F. Meng _et al._ , “Cross-Lingual Summarization via ChatGPT.” [Online]. Available: http://arxiv.org/abs/2302.14229 2023\.
  * [24] J. Devlin, M. W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics_ , vol. 1, oct 2019, pp. 4171–4186.
  * [25] L. Martin, B. Muller, P. J. Ortiz Suárez _et al._ , “CamemBERT: a Tasty French Language Model,” in _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , 2020, pp. 7203–7219.
  * [26] S. Ruder, A. Søgaard, and I. Vulic, “Unsupervised cross-lingual representation learning,” in _ACL 2019_ , nov 2019, pp. 31–38.
  * [27] N. X. Vinh, J. Epps, and J. Bailey, “Information theoretic measures for clusterings comparison: Is a correction for chance necessary?” _ACM International Conference Proceeding Series_ , vol. 382, pp. 1073–1080, 2009.
  * [28] L. Hubert and P. Arabie, “Comparing partitions,” _Journal of Classification_ , vol. 2, no. 1, pp. 193–218, 1985.
  * [29] D. Cer, Y. Yang, S. yi Kong _et al._ , “Universal sentence encoder for English,” _EMNLP 2018 - Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Proceedings_ , pp. 169–174, 2018.
  * [30] W. Wang, F. Wei, L. Dong _et al._ , “MINILM: Deep self-attention distillation for task-agnostic compression of pre-trained transformers,” _Advances in Neural Information Processing Systems_ , 2020.
  * [31] K. Song, X. Tan, T. Qin _et al._ , “MPNet: Masked and permuted pre-training for language understanding,” _Advances in Neural Information Processing Systems_ , vol. 2020-Decem, no. NeurIPS, pp. 1–14, 2020\.
  * [32] L. Wang, N. Yang, X. Huang _et al._ , “Text Embeddings by Weakly-Supervised Contrastive Pre-training,” pp. 1–17. [Online]. Available: http://arxiv.org/abs/2212.03533 2022\.
  * [33] A. R. Fabbri, W. Kryściński, B. McCann _et al._ , “Summeval: Re-evaluating summarization evaluation,” _Transactions of the Association for Computational Linguistics_ , vol. 9, pp. 391–409, 2021.