# Embracing Uncertainty: Adaptive Vague Preference Policy Learning for Multi-
round Conversational Recommendation

Gangyi Zhang1, Chongming Gao1, Wenqiang Lei2, Xiaojie Guo3, Shijun Li1,
Lingfei Wu3, Hongshen Chen4, Zhuozhi Ding4, Sulong Xu4 and Xiangnan He1
1University of Science and Technology of China, 2 Sichuan University, 3 JD.COM
Silicon Valley Research Center, 4 JD.COM

(2018)

###### Abstract.

Conversational recommendation systems (CRS) effectively address information
asymmetry by dynamically eliciting user preferences through multi-turn
interactions. Existing CRS widely assumes that users have clear preferences,
i.e., users have a firm belief about the fine-grained preference for one or
multiple target items. Under this assumption, the agent will completely trust
the user feedback and treat the accepted or rejected signals as strong
indicators to filter items and reduce the candidate space, which may lead to
the problem of over-filtering. However, in reality, users’ preferences are
often vague and volatile, with uncertainty about their desires and changing
decisions during interactions.

To address this issue, we introduce a novel scenario called Vague Preference
Multi-round Conversational Recommendation (VPMCR), which considers users’
vague and volatile preferences in CRS. VPMCR employs a soft estimation
mechanism to assign a non-zero confidence score for all candidate items to be
displayed, naturally avoiding the over-filtering problem. In the VPMCR
setting, we introduce an solution called Adaptive Vague Preference Policy
Learning (AVPPL), which consists of two main components: Uncertainty-aware
Soft Estimation (USE) and Uncertainty-aware Policy Learning (UPL). USE
estimates the uncertainty of users’ vague feedback and captures their dynamic
preferences using a choice-based preferences extraction module and a time-
aware decaying strategy. UPL leverages the preference distribution estimated
by USE to guide the conversation and adapt to changes in users’ preferences to
make recommendations or ask for attributes.

Our extensive experiments demonstrate the effectiveness of our method in the
VPMCR scenario, highlighting its potential for practical applications and
improving the overall performance and applicability of CRS in real-world
settings, particularly for users with vague or dynamic preferences.

Conversational Recommendation; Vague Preference; Policy Learning

††copyright: acmcopyright††journalyear: 2018††doi:
XXXXXXX.XXXXXXX††conference: Make sure to enter the correct conference title
from your rights confirmation emai; June 03–05, 2018; Woodstock, NY††price:
15.00††isbn: 978-1-4503-XXXX-X/18/06††ccs: Information systems Users and
interactive retrieval††ccs: Information systems Recommender systems††ccs:
Information systems Personalization††ccs: Human-centered computing Interactive
systems and tools Figure 1. A realistic user simulation example

##  1\. Introduction

Conversational recommendation systems (CRS) have drawn a lot of research
attention recently. These systems interact with users to elicit preferences,
understand motivations, and address the long-standing information asymmetry
problem (Gao et al., 2021). Despite considerable progress, CRS is far from
mature, and researchers have focused on specific scenarios (Sun and Zhang,
2018; Lei et al., 2020a; Zhang et al., 2022b) to address particular
challenges.

One widely adopted scenario (Lei et al., 2020a; Lei et al., 2020b; Xu et al.,
2021) is Multi-round Conversational Recommendation (MCR), where the system can
ask for attributes or make recommendations multiple times, and the user
accepts or rejects accordingly. However, MCR assumes that users have a clear
single preferred item in mind, which may not be realistic, as users may have
more than one preferred item in mind. To address this, the Multi-Interest
Multi-round Conversational Recommendation (MIMCR) scenario (Zhang et al.,
2022b) was proposed, allowing users to have multiple preferences. In this
setting, a user may accept multiple attribute instances (e.g., red and black)
of an attribute type (e.g., color).

Despite the improvement, MIMCR can still fall short because it assumes that
users have clear preferences in mind during the conversation. This can be
impractical as users’ preferences can be vague or change dynamically over
time, leading to randomness in their answers and potential regret for previous
choices.

In practical applications, users exhibit vague or dynamic preferences, but
MIMCR (or MCR) fails to account for the uncertainty in users’ feedback,
treating it as a hard indicator to filter the candidate item set. This results
in over-filtering, as numerous potential items are removed when the user
selects or does not select corresponding attributes. In Fig. 1 (a), we
illustrate a toy example showing a conversation (tailored for vague settings)
under the MIMCR scenario. The CRS incorrectly interprets the user’s non-
clicking attributes (i.e., “plaid” in the first turn) and removes potential
target items (i.e., “item-1” in the first turn), causing the user’s preference
distribution over items to collapse suddenly as shown in the left side of Fig.
1 (b). This wrong inference will naturally affect the reasoning of the
subsequent conversation, leading to the wrong preference estimation (i.e., in
Fig. 1 (a), the “black” color of “item-1” was not displayed in the third
turn).

To address over-filtering in MIMCR and MCR and maintain diversity and accuracy
in the CRS, we propose a new scenario called Vague Preference Multi-round
Conversational Recommendation (VPMCR). This scenario uses a soft estimation
mechanism to account for users’ vague or dynamic preferences by assigning non-
zero confidence scores to all candidate items, avoiding the rigid filtering
strategy of MIMCR and MCR. Fig. 1 (c) shows an example of the VPMCR, which, in
contrast to MIMCR, captures changes in preference distribution of the entire
item space as shown in the right side of Fig. 1 (b).

In the VPMCR scenario, several challenges need to be addressed, including
estimating the uncertainty of the user’s vague feedback, capturing the user’s
dynamic preference throughout the conversation, and making conversational
decisions that consider the user’s vague or dynamic preferences.

To tackle these challenges, we propose an enhanced solution called Adaptive
Vague Preference Policy Learning (AVPPL), which consists of:

1\. Uncertainty-aware Soft Estimation (USE): USE estimates the uncertainty of
the user’s vague feedback in each turn using a choice-based preference
extraction method. It captures both explicit and implicit preferences
(distinguished based on whether the user explicitly clicks the item),
effectively estimating the uncertainty of users’ vague feedback. To capture
users’ dynamic preferences, USE employs a time-aware preference decay
strategy, which gives more weight to recent preferences while gradually
reducing the influence of historical preferences.

2\. Uncertainty-aware Policy Learning (UPL): Leveraging the preference
distribution estimated by USE, UPL implements a unified policy learning
framework to guide the conversation and adapt to changes in the user’s
preferences to make recommendations or ask for attributes. The soft estimation
scores from USE’s preference distribution are utilized as edge weights to
construct a dynamic heterogeneous graph of the conversation. We also introduce
a preference-guided action pruning strategy to expedite the RL sampling
process. To address the challenges in the VPMCR scenario, particularly
considering the uncertainty of users’ vague feedback, we employ a Deep
Q-Network (DQN) algorithm for UPL.

In summary, our contributions are as follows:

  * •

We identify the limitations of existing CRS settings and introduce the VPMCR
scenario, which accounts for users’ vague and volatile preferences in CRS.

  * •

We propose the AVPPL solution for the VPMCR setting, utilizing a unified
policy learning framework to make decisions that consider users’ current vague
preferences and account for their fading historical preferences.

  * •

Our extensive experiments on four real-world datasets demonstrate the
effectiveness of AVPPL in the VPMCR scenario, highlighting its potential for
practical applications.

##  2\. Related Work

We briefly introduce the related works in conversational recommendation,
reinforcement learning, and graph learning.

###  2.1. Conversational recommendation system

(CRSs) is a novel solution to recommendation that leverage natural language to
effectively elicit dynamic user preferences that align with their real needs
through multiple rounds of real-time interaction. CRS is considered to be a
cutting-edge discipline that incorporates dialogue systems, recommendation
systems, and interactive systems (Gao et al., 2021). According to the focus on
different functions and settings, existing CSR methods can be roughly divided
into two types: dialogue-based recommendation (Li et al., 2018; Zhou et al.,
2020c; Chen et al., 2019; Zhou et al., 2022; Wu et al., 2019) and multi-round
conversational recommendation (MCR) (Lei et al., 2020b; Deng et al., 2021; Xu
et al., 2021; He et al., 2022; Gao et al., 2022b; Li et al., 2021). In this
work, we focus on the MCR setting.

MCR is considered to be the most realistic setting in CRS. Unlike dialogue-
based recommenders that need to extract information or generate responses
through raw natural language (Wang et al., 2022b), MCR focuses on the core
logic of the interaction strategy which involves asking questions (Zou et al.,
2020, 2020; Ren et al., 2022) and making recommendations. The traditional MCR
setting allows users to select only one preferred attribute value at a time,
which restricts users’ expression in the interaction. To overcome this issue,
Zhang et al. (2022b) propose the MIMCR setting, where a user is allowed to
select multiple options for a certain attribute. Though effective, they follow
the recommendation philosophy in MCR to directly filter out the items that the
user has not mentioned by attributes, which leads to failure as users may not
be sure what they want precisely. In our proposed VPMCR setting, we
specifically consider users’ vague preferences and adjust the recommendation
mechanism to consider the items with unmentioned attributes, which better
reflect users’ needs.

###  2.2. RL-based Recommendation

Reinforcement Learning (RL) is a type of Machine Learning. It considers how an
agent (e.g., a machine) should automatically make decisions within a specific
context to pursue a long-term goal. The agent learns and adjusts its policy
based on the reward feedback (i.e., reinforcement signals) given by the
environment. Recently, RL has shown its effectiveness in recommendation (Afsar
et al., 2022; Deffayet et al., 2023; Gao et al., 2023). As fitting user
interest is not a bottleneck for now, recommenders care more about users’
long-term satisfaction (Xue et al., 2023; Zhang et al., 2022a; Wang et al.,
2022a). For instance, Montazeralghaem and Allan (2022) use RL to generate the
proper questions that can maximally make the system help users search desired
products. Gao et al. (2022a) integrate causal inference into offline RL to
maximize users’ long-term satisfaction by removing filter bubbles. Sadeghi
Eshkevari et al. (2022) propose an RL-based dispatching solution for ride-
hailing platforms that can conduct robust and efficient on-policy learning and
inference while being adaptable for full-scale deployment. In this work, we
use RL to learn a policy that can automate question-asking and item
recommendation.

###  2.3. Graph-based Recommendation

Graph-based recommender systems have drawn a lot of research attention (Chen
et al., 2022; Liu et al., 2022; Guo et al., 2021, 2021). By arranging the
various entities (e.g., users, items, and attributes) in a heterogeneous
graph, we can leverage lots of properties in modeling the collaborative
signals. In CRS, the knowledge graph is utilized in enriching the system with
additional knowledge (Lei et al., 2020b; Xu et al., 2020; Zhou et al., 2020b;
Moon et al., 2019; Zhou et al., 2020a). For example, to better understand
concepts that a user mentioned, Zhou et al. (2020b) propose to incorporate two
external knowledge graphs (KGs): a word-oriented KG providing relations (e.g.,
synonyms, antonyms, or co-occurrence) between words and an item-oriented KG
carrying structured facts regarding the attributes of items. With the
increasing of nodes, the computational overhead is too large to satisfy the
requirement of real-time interaction. Hence, we propose a pruning strategy to
overcome this work.

##  3\. Problem Definition

Vague Preference Multi-round Conversational Recommendation (VPMCR). In the
VPMCR scenario, we consider a dynamic conversation between a user and a
conversational recommendation system (CRS). The user has a clear preference
space, denoted as $\mathcal{C}_{CI}$ (e.g., ”style” in Fig. 1), and a vague
preference space, denoted as $\mathcal{C}_{VI}$ (e.g., ”color” and ”pattern”
in Fig. 1).

The conversation begins with the user specifying a query attribute $p_{0}$
(e.g., ”T-shirt”), which initializes the candidate item set containing all
relevant items (e.g., all ”T-shirts”) and the candidate attribute set
containing all attributes of those items.

During the conversation, the CRS can either ask questions about attributes or
provide recommendations. When the CRS asks questions, the user responds
accordingly with their behavior depending on whether the attribute type $c$
belongs to their clear or vague preference space. If $c\in\mathcal{C}_{CI}$,
the user _honestly_ accepts or rejects the displayed attributes. However, if
$c\in\mathcal{C}_{VI}$, the user may _randomly_ accept or reject a potentially
preferred attribute. When the CRS provides recommendations, the user can
accept or reject one or more items from the recommended set
$\mathcal{V}_{rec}$.

The conversation proceeds through multiple iterations of the CRS
asking/recommending and the user responding, until a successful recommendation
is made or the maximum number of turns is reached. The VPMCR scenario differs
from previous MCR or MIMCR settings in that it does not filter
$\mathcal{V}_{cand}$ based on the user’s clicking or non-clicking attributes.
Instead, it only removes $\mathcal{V}_{rec}$ from $\mathcal{V}_{cand}$ when
the recommendation fails. Additionally, all candidate attributes linked to
candidate items are maintained in $\mathcal{P}_{cand}$.

The main challenges in the VPMCR scenario include estimating the uncertainty
of the user’s vague feedback, capturing the user’s dynamic preference
throughout the conversation, and making conversational decisions that consider
the user’s vague or dynamic preferences.

Figure 2. Adaptive Vague Preference Policy Learning (AVPPL) solution for VPMCR
scenario.

##  4\. METHODOLOGY

To address the challenges in the Vague Preference Multi-round Conversational
Recommendation (VPMCR) scenario, we propose the _Adaptive Vague Preference
Policy Learning (AVPPL)_ solution. AVPPL consists of two main components:
Uncertainty-aware Soft Estimation (USE) and Uncertainty-aware Policy Learning
(UPL). The USE component estimates the uncertainty of users’ vague feedback
and captures their dynamic preferences, while the UPL component leverages the
preference distribution estimated by USE to guide the conversation and adapt
to changes in users’ preferences. By incorporating the VPMCR scenario and the
AVPPL solution, we aim to improve the overall performance and applicability of
conversational recommendation systems in real-world settings, particularly for
users with vague or dynamic preferences.

###  4.1. Uncertainty-aware Soft Estimation

Uncertainty-aware Soft Estimation (USE) aims to estimate the uncertainty of
the user’s vague feedback in each turn by considering both explicit and
implicit preferences. USE focuses on understanding users’ decision-making
processes (Bettman et al., 1998), which reflect the trade-offs they make when
providing non-binary feedback. To capture users’ dynamic preferences
throughout the conversation, USE employs a time-aware preference decay
strategy that combines users’ recent preferences with fading historical
preferences.

In the VPMCR setting, we model the signals of clicking and non-clicking
separately based on the decision-making consciousness of users in choice-based
questions. For each turn, preference implied by clicking and non-clicking
choices is extracted, then the decay mechanism is used to weaken the
preference of historical turns. Finally, in the soft estimation, we derive the
user’s preference distribution toward items and attributes.

####  4.1.1. Preference Extraction with Choice-based Approach

In each turn of interaction, user preference can be divided into personalized
user preference and choice-based preference. We adopt a common personalization
modeling strategy (Lei et al., 2020a) to represent the static preference of
user $u$ for item $v$ as:

(1) $w_{v\mbox{-}u}=e_{u}^{\top}e_{v},$

where $e_{u}$ and $e_{v}$ denote the embedding vectors of user $u$ and item
$v$, respectively.

To model users’ decision-making processes, USE employs a choice-based
preference extraction method that considers the trade-offs users make when
providing non-binary feedback. This approach captures both _explicit
preferences_ (when users actively select an attribute) and _implicit
preferences_ (when users do not select an attribute but may still have some
preference for it) by estimating the importance of clicking choices and non-
clicking choices separately.

For item $v$, we estimate the importance of clicking choices and non-clicking
choices, respectively. In turn $t$, the formula for capturing the user’s
explicit preference towards clicking choices
$\mathcal{P}_{\text{click}}^{(t)}$ and implicit preference towards non-
clicking choices $\mathcal{P}_{\text{noclick}}^{(t)}$ are shown as follows:

(2)
$\begin{split}w_{v\mbox{-}click}^{(t)}=\frac{1}{\lvert\mathcal{P}_{\text{click}}^{(t)}\rvert}\sum_{p\in\mathcal{P}_{\text{click}}^{(t)}}(e_{v}^{\top}e_{p}-w_{v\mbox{-}avg}^{(t)}),\\\
w_{v\mbox{-}noclick}^{(t)}=\frac{1}{\lvert\mathcal{P}_{\text{noclick}}^{(t)}\rvert}\sum_{p\in\mathcal{P}_{\text{noclick}}^{(t)}}(e_{v}^{\top}e_{p}-w_{v\mbox{-}avg}^{(t)}),\end{split}$

where $\lvert\mathcal{P}_{\text{click}}\rvert$ and
$\lvert\mathcal{P}_{\text{noclick}}\rvert$ indicates the number of attributes
related to clicked items and non-clicked items, respectively.
$w_{v\mbox{-}avg}^{(t)}$ measures the average preference towards all unshown
attribute types and is used to mitigate over-estimation of the system-
displayed choices, which is defined as:

(3)
$w_{v\mbox{-}avg}^{(t)}=\sum_{p\in\mathcal{P}_{\text{noshow}}^{(t)}}e_{v}^{\top}e_{p}\bigg{/}\lvert\mathcal{P}_{\text{noshow}}^{(t)}\rvert,$

where $e_{v}$ and $e_{p}$ represent the embedding vectors of item $v$ and
attribute $p$, respectively, and $\mathcal{P}_{\text{noshow}}^{(t)}$ refers to
the set of all unshown attributes associated with the specified attribute type
in turn $t$.

By considering both the personalized preferences and the choice-based
preference in turn $t$, the users’ preference for item $v$ in turn $t$ can be
calculated as:

(4)
$w_{v}^{(t)}=\sigma(w_{v\mbox{-}u}+\lambda_{1}w_{v\mbox{-}click}^{(t)}+\lambda_{2}w_{v\mbox{-}noclick}^{(t)}),$

where $\sigma$ is the sigmoid function. $\lambda_{1}$ and $\lambda_{2}$
represent the information intensity coefficients of the information contained
in the user’s clicked attribute and the user’s unclicked attribute,
respectively.

####  4.1.2. Time-aware Preference Decay

In dynamic conversation interactions, the user’s global preferences should be
viewed as a combination of preferences across all turns. We employ a decay
mechanism to adjust the influence of historical preferences, enabling the
model to focus more on the user’s real-time feedback in the current turn and
mitigating the over-emphasized impact related to the user’s clicking behavior.

To combine the user’s current preference with historical decay preferences,
the user’s global preference toward the item is estimated as follows:

(5) $w_{v}^{(t)}=w_{v}^{(t)}+\gamma w_{v}^{(t-1)},$

which can be unfolded as:

(6) $w_{v}^{(t)}=\sum_{i=0}^{t-1}\gamma^{t-i-1}w_{v}^{(i)},$

where $\gamma$ is a decay factor satisfying $0\leq\gamma\leq 1$. The farther
the interaction history is from the current turn, the less impact it will have
on the current turn. $\gamma$ should be carefully chosen to balance the
influence of historical preferences and the user’s real-time feedback.

Finally, for turn $t$, the user’s global preference distribution for items
$f_{u}^{(t)}(v)$ can be calculated by estimating the user’s global preference
$w$ for each item $v$ in the candidate item set $\mathcal{V}_{\text{cand}}$.
When the size of the candidate item set is $n$, the soft estimation
distribution for items is shown as follows:

(7) $f_{u}^{(t)}(v)=\\{w_{v_{1}}^{(t)},w_{v_{2}}^{(t)},...,w_{v_{n}}^{(t)}\\}$

Similarly, by replacing items with attributes in the aforementioned equations,
we derive the user’s global preference distribution towards the candidate
attribute set $\mathcal{P}_{\text{cand}}$. When the size of the candidate
attribute set is $m$, the soft estimation for attributes is depicted by the
following distribution:

(8) $f_{u}^{(t)}(p)=\\{w_{p_{1}}^{(t)},w_{p_{2}}^{(t)},...,w_{p_{m}}^{(t)}\\}$

###  4.2. Uncertainty-aware Policy Learning (UPL)

In the Uncertainty-aware Policy Learning (UPL) module, we address the
challenge of making conversational decisions that consider users’ vague or
dynamic preferences in a Conversational Recommendation System (CRS). We
utilize the preference distribution estimated by the Uncertainty-aware Soft
Estimation (USE) module to guide the conversation and adapt to preference
changes. By constructing a dynamic heterogeneous graph and employing a
preference-guided action pruning strategy, we streamline the Reinforcement
Learning (RL) sampling process. We adopt a Deep Q-Network (DQN) algorithm for
UPL, which is effective in learning action policies in dynamic environments.
The UPL module, as part of the Adaptive Vague Preference Policy Learning
(AVPPL) solution, aims to enhance CRS performance for users with vague or
dynamic preferences.

####  4.2.1. Graph-based Conversation Modeling

In the Graph-based Conversation Modeling section, we represent the current
state of the conversation at turn $t$ using a dynamic undirected graph
$\mathcal{G}_{u}^{(t)}=(\mathcal{N}^{(t)},\mathbf{A}^{(t)})$. This graph is a
subgraph of the heterogeneous graph, which consists of users, items, and
attributes. The dynamic graph is constructed based on the preference
distribution estimated by the Uncertainty-aware Soft Estimation (USE) module,
which sets it apart from previous work (Deng et al., 2021; Zhang et al.,
2022b).

The nodes in the graph, $\mathcal{N}^{(t)}$, are defined as follows:

(9)
$\mathcal{N}^{(t)}=\\{u\\}\cup\mathcal{P}_{\text{click}}\cup\mathcal{P}_{n\mbox{-}\text{click}}\cup\mathcal{P}_{\text{cand}}^{(t)}\cup\mathcal{V}_{\text{sample}}^{(t)}$

Here, $\mathcal{P}_{\text{click}}$ and $\mathcal{P}_{n\mbox{-}\text{click}}$
represent the user’s historical clicking and non-clicking attributes
throughout the conversation. $\mathcal{P}_{\text{cand}}^{(t)}$ and
$\mathcal{V}_{\text{sample}}^{(t)}$ indicate the candidate attribute set and
the randomly sampled candidate item set at turn $t$, respectively.

The weighted adjacency matrix, $\mathbf{A}^{(t)}$, is defined as:

(10)
$\begin{array}[]{l}A_{i,j}^{(t)}=\left\\{\begin{array}[]{ll}w_{v}^{(t)},&\text{
if }n_{i}=u,n_{j}\in\mathcal{V}\\\ 1,&\text{ if
}n_{i}\in\mathcal{V},n_{j}\in\mathcal{P}\\\ 0,&\text{ otherwise
}\end{array}\right.\end{array}$

The weight $w_{v}^{(t)}$ denotes the user’s estimated preference for the item
$v$, which is calculated via Eq. (6) within the USE module. The weights of the
edge between the item and its associated attributes are set to $1$.

To address the issue of a large number of candidate items in the VPMCR
setting, we implement a sampling strategy for candidate items
$\mathcal{V}_{\text{sample}}^{(t)}$ by randomly selecting from the candidate
items in each turn $t$. This node sampling strategy is similar to node dropout
(Wu et al., 2021) in graph learning and helps reduce the scale of the dynamic
graph while enhancing training convergence and the robustness of graph
learning (Wu et al., 2021).

We employ a Graph Convolutional Network (GCN) (Kipf and Welling, 2016) to
refine all node representations $\mathcal{E}{node}$ by capturing the
information of changing interrelationships for the current conversation state
$\mathcal{G}_{u}^{(t)}$:

(11) $\mathcal{E}_{\text{node}}=\text{GCN}(\mathcal{G}_{u}^{(t)}).$

Following the design from Deng et al. (Deng et al., 2021), the explicit
clicking history session $\mathcal{P}_{\text{click}}$ is encoded by a
Transformer (Vaswani et al., 2017) to learn the sequence information of the
conversation history $\mathcal{I}_{his}^{(t)}$:

(12)
$\mathcal{I}_{\text{his}}^{(t)}=\text{Transformer}(e_{\text{click}}^{1},e_{\text{click}}^{2},...e_{\text{click}}^{l}).$

Here, $l$ denotes the length of the sequence and
$\lvert\mathcal{P}_{\text{click}}\rvert$ is the number of clicking attributes
in a whole conversation. The input to the Transformer is an embedding sequence
corresponding to a sequence of clicking attributes
$\mathcal{P}_{\text{click}}$, where each embedding $e_{\text{click}}$ is
learned from the embeddings in $\mathcal{E}_{\text{node}}$.

Finally, the final conversation state representation $S_{\text{conv}}^{(t)}$
is obtained by a mean polling layer.

(13) $s_{\text{conv}}^{(t)}=\text{MeanPool}(\mathcal{I}_{\text{his}}^{(t)}).$

####  4.2.2. Preference-guided Action Pruning

In the unified policy learning framework (Deng et al., 2021), the action space
includes all candidate attributes and all candidate items. Such a large action
space in reinforcement learning can negatively impact sampling efficiency. To
address this issue, we propose an effective action-pruning strategy based on
user preferences.

As described in Section 4.1, we can estimate the user’s preference
distribution $f_{u}$. Item $v$ or attribute $p$ with higher confidence values
are more likely to be preferred by the user.

To construct the pruning action space $\mathcal{A}_{\text{action}}^{(t)}$, we
first calculate the user’s preference distribution over items using Eq. (7) in
USE. Then, we select the top-N items $\mathcal{V}_{\text{top}}^{(t)}$ with the
highest confidence and include them in the pruning action space. Additionally,
we select the top-N attributes $\mathcal{P}_{\text{top}}^{(t)}$ with the
highest confidence and add them to the pruning action space. The pruning
action space is defined as:

(14)
$\mathcal{A}_{\text{action}}^{(t)}=\mathcal{V}_{\text{top}}^{(t)}+\mathcal{P}_{\text{top}}^{(t)}$

####  4.2.3. Deep Q-Network for Policy Learning

Following UNICORN (Deng et al., 2021), we introduce a unified policy learning
framework that can systematically integrate the conversation and
recommendation component to solve the decision making problem in CRS.

We employ a Deep Q-Network (DQN) algorithm to address the challenge of making
conversational decisions that consider users’ vague or dynamic preferences in
CRS. The DQN algorithm has been proven effective in learning action policies
in dynamic environments, such as Markov Decision Processes (MDPs), making it
well-suited for predicting the next decision based on a series of historical
choices.

The Q-value function $Q\left(s_{t},a_{t}\right)$ of a policy $\pi$ is defined
to measure the expectation of the accumulated rewards based on the state $s$
and the action $a$. We adopt the same Dueling DQN and prioritized experience
replay as in UNICORN (Deng et al., 2021) to optimize the Q-function
$Q^{\ast}\left(s_{t},a_{t}\right)$:

(15)
$Q^{*}(s_{t},a_{t})=\max_{\pi}\mathbb{E}[R_{t+1}+\gamma\max_{a}Q^{\pi}(s_{t+1},a)|s_{t},a_{t}]$

where $\pi$ is the policy, $R_{t+1}$ is the reward at turn $t+1$, $\gamma$ is
the discount factor, and $Q^{\pi}(s_{t+1},a)$ is the estimated action-value
function for the next state and action.

For policy learning, the input conversation state $s_{\text{conv}}^{(t)}$ is
learned by the graph-based conversation modeling module. The pruning action
space $\mathcal{A}_{\text{action}}^{(t)}$ is determined by employing a
preference-guided action pruning strategy, which expedites the RL sampling
process. The reward $R$ follows the previous MCR setting (Lei et al., 2020b),
and the detailed settings will be described in the experimental section.

##  5\. Experiments

In this section, we evaluate the proposed method in VPMCR. We use the
following research questions (RQs) to guide our experiment.

  * •

RQ1. How does our AVPPL method perform in comparison to state-of-the-art CRS
methods in the VPMCR scenario?

  * •

RQ2. How do the key components contribute to the overall performance of our
AVPPL method?

  * •

RQ3. How do the hyperparameters of our method affect its performance?

  * •

RQ4. Can AVPPL effectively make recommendations based on users’ vague
preferences during the conversation?

###  5.1. Dataset Description

Table 1. Statistics of datasets.

Dataset | Yelp | LastFM | Amazon-Book | MovieLens
---|---|---|---|---
#Users | 27,675 | 1,801 | 30,291 | 20,892
#Items | 70,311 | 7,432 | 17,739 | 16,482
#Interactions | 1,368,609 | 76,693 | 478,099 | 454,011
#Attributes | 590 | 8,438 | 988 | 1,498
#Attribute-types | 29 | 34 | 40 | 24
#Entities | 98,576 | 17,671 | 49,018 | 38,872
#Relations | 3 | 4 | 2 | 2
#Triplets | 2,533,827 | 228,217 | 565,068 | 380,016

We introduce four datasets, whose statistics are shown in table 1.

  * •

Yelp and LastFM (Lei et al., 2020a): Yelp111https://www.yelp.com/dataset/ and
LastFM222https://grouplens.org/datasets/hetrec-2011/ datasets are used for
business and music artist recommendations, respectively. We follow the
multiple attribute question settings, retaining the original attribute
instances in LastFM and Yelp, and extracting the attribute types they depend
on. In Yelp, we utilize the 2-layer taxonomy designed by (Lei et al., 2020a),
resulting in 29 categories in the first layer as attribute types and 590
attributes in the second layer as attribute instances. For LastFM, we follow
(Zhang et al., 2022b), retaining the original 8,438 attributes as attribute
instances and employing clustering to obtain 34 attribute types.

  * •

Amazon-Book (Wang et al., 2019): Amazon
Book333http://jmcauley.ucsd.edu/data/amazon. is a widely used product
recommendation dataset. We retain users and items with at least 10 interaction
records and consider entities (e.g., science fiction) and relations (e.g.,
genre) in the knowledge graph as attribute instances and attribute types,
respectively.

  * •

MovieLens: Movielens is a movie rating dataset. We adopt
MovieLens-20M444https://grouplens.org/datasets/movielens/ dataset, following
(Zhang et al., 2022b), and retain interactions with ratings greater than 3. We
select entities and relations in the knowledge graph (KG) as attribute
instances and attribute types, respectively.

###  5.2. Experimental Setup

####  5.2.1. User Simulator in VPMCR

Conversational recommendation systems (CRSs) are interactive and require
training and evaluation through user interactions. However, obtaining data
directly from users in a research lab is impractical, so employing a user
simulator is a common practice (Chandramohan et al., 2011). The user simulator
simulates users’ interaction records in the training and test sets.

In the VPMCR scenario, we adopt a user simulation strategy similar to that in
MIMCR (Zhang et al., 2022b), considering the reasonableness of the multi-
interest setting. For a given observed user-items interaction pair
$(u,\mathcal{V}_{u})$, we simulate a conversation session. Each item $v$ in
$\mathcal{V}_{u}$ is treated as a ground-truth target item, and the union of
attribute types and attributes associated with each item are considered as the
user’s ground-truth intent space $\mathcal{C}_{u}$ and ground-truth attribute
space $\mathcal{P}$, respectively. The conversation session is initialized
when the user specifies a common attribute $p_{0}$ to all $\mathcal{V}_{u}$,
and the user’s clear preference space $\mathcal{C}_{CI}$ and user’s vague
preference space $\mathcal{C}_{VI}$ are randomly initialized from the ground-
truth intent space $\mathcal{C}_{u}$.

During the interaction, we use the ground-truth attribute space $\mathcal{P}$
as a criterion for the user simulator’s acceptance or rejection. The detailed
interaction process follows the “system asks or recommends and user responds”
rules outlined in Section 3.

####  5.2.2. Action Inference

The action inference involves either recommending items or asking an
attribute-related question.

(1) Recommendation: If an item $v$ in the action space has the highest
Q-value, the CRS make a recommendation, resulting in a new action space
$\mathcal{A}^{(t)}=\mathcal{V}_{top}^{(t)}$.

(2) Questioning: If an attribute $p$ in the action space has the highest
Q-value, the CRS asks a question. In a multiple-choice setting, a two-level
decision process is employed: first selecting an attribute type, then
presenting several attributes within that type. A sum-based strategy (Zhang et
al., 2022b) is used to determine the attribute type for questioning.
Specifically, Q-values of all attributes within the attribute action space
$\mathcal{P}_{top}^{(t)}$ are summed and allocated to their respective
attribute types. The attribute type with the highest total value is selected
for questioning, and the top $K$ attributes with the highest Q-values within
that type are presented to the user.

####  5.2.3. Baselines

We use the following baselines. For fairness, all baselines are compared in
the VPMCR scenario.

  * •

Max Entropy. It selects the attribute with the maximum information entropy and
inversely relates the probability of making a recommendation to the length of
candidate items.

  * •

CRM (Sun and Zhang, 2018). It employs a belief tracker to record user
preferences as conversation state representation vectors and applies them to a
reinforcement learning decision module and factorization machine (FM)
recommendation modules.

  * •

EAR (Lei et al., 2020a). This method adopts the three-stage solution framework
to enhance the interaction between the conversation component and the
recommendation component.

  * •

SCPR (Lei et al., 2020b). SCPR leverages graph-based path reasoning to prune
useless candidate attributes. It separates attribute selection from
reinforcement learning, which is only used for determining when to ask and
recommend.

  * •

UNICORN (Deng et al., 2021). A state-of-the-art method for the MCR scenario
that proposes a unified policy learning framework using dynamic graphs to
model conversation states and employs a preference-based scoring to reduce
reinforcement learning action space.

  * •

MCMIPL (Zhang et al., 2022b). It considers the user’s multi-interest space and
extends the MCR scenario to a more realistic MIMCR scenario. This method also
follows the graph-based unified reinforcement learning framework and employs
the multi-interest encoder to learn the conversation state.

####  5.2.4. Training Details

We divide each dataset into training, validation, and testing sets using a
7:1.5:1.5 ratio. In the user simulator, we set the maximum conversation turn
$T$ to 15 and the number of target item sets $\mathcal{V}_{u}$ for the user to
2. We initialize the user’s vague preference space and clear preference space
using uniform sampling.

In the Uncertainty-aware Soft Estimation (USE) module, we set the information
intensity coefficients $\lambda_{1}$ and $\lambda_{2}$ to 0.1 and 0.01,
respectively, and the decay discount factor to 0.1.

In the Uncertainty-aware Policy Learning (UPL) module, when constructing the
dynamic graph, random sampling is employed to select candidate items when the
available number of candidates exceeds 5000. The graph-based conversation
modeling architecture consists of two GNN layers and one Transformer layer. We
fix the embedding size and hidden size at 64 and 100, respectively. For action
pruning in RL, we set the size of the item space and attribute space to 10
(i.e., $N=10$). For action inference, we set the number of attributes
displayed to the user to 2 (i.e., $K=2$). Following (Deng et al., 2021), we
use TransE (Bordes et al., 2013), implemented throughKE (Han et al., 2018), to
pre-train the graph node embeddings. During DQN training, we ensure a fair
comparison with other benchmarks by conducting online training for 10,000
episodes and adopting the same reward setting with $r_{\text{rec-suc}}=1$,
$r_{\text{rec-fail}}=-0.01$, $r_{{ask-suc}}=-0.1$, $r_{\text{ask-fail}}=-0.1$,
and $r_{\text{quit}}=-0.3$. We set the experience replay buffer to 50,000 and
the mini-batch size to 128. The learning rate is fixed at 1e-4 with an L2
regularization of 1e-6, using the Adam optimization algorithm.

####  5.2.5. Evaluation Metrics

This study employs success rate (SR@$T$) and average turn (AT) to evaluate the
recommendation performance. SR@$T$ measures the percentage of successful
recommendations within $T$ turns. A higher SR@$T$ indicates better
performance. AT measures the average number of turns in a conversation. A
lower AT demonstrates greater efficiency.

We also use hierarchical normalized discounted cumulative gain (hDCG@($T,K$))
to evaluate the ranking performance of the top-$K$ recommendations within $T$
turns. hDCG assigns higher scores to recommendations that are more relevant to
the user. A higher nDCG@($T,K$) indicates a better ranking performance.

Table 2. Performance comparison of different models in VPMCR scenario. hDCG stands for hDCG@(15, 10). Models | Yelp |  | LastFM |  | Amazon-Book |  | MovieLens
---|---|---|---|---|---|---|---
SR@15 | AT | hDCG |  | SR@15 | AT | hDCG |  | SR@15 | AT | hDCG |  | SR@15 | AT | hDCG
Max Entropy | 0.062 | 14.44 | 0.030 |  | 0.376 | 11.25 | 0.189 |  | 0.180 | 12.91 | 0.107 |  | 0.448 | 9.93 | 0.315
CRM | 0.212 | 13.27 | 0.070 |  | 0.372 | 12.26 | 0.126 |  | 0.296 | 12.34 | 0.109 |  | 0.780 | 5.96 | 0.341
EAR | 0.232 | 13.05 | 0.080 |  | 0.414 | 11.61 | 0.146 |  | 0.324 | 12.14 | 0.119 |  | 0.792 | 5.50 | 0.361
SCPR | 0.322 | 12.34 | 0.115 |  | 0.596 | 10.18 | 0.206 |  | 0.374 | 11.62 | 0.139 |  | 0.806 | 4.90 | 0.387
UNICORN | 0.314 | 12.11 | 0.140 |  | 0.632 | 9.17 | 0.280 |  | 0.396 | 11.05 | 0.193 |  | 0.810 | 4.81 | 0.548
MCMIPL | 0.322 | 12.16 | 0.136 |  | 0.634 | 9.52 | 0.267 |  | 0.412 | 10.90 | 0.205 |  | 0.820 | 4.39 | 0.579
AVPPL | 0.398 | 11.26 | 0.175 |  | 0.686 | 8.58 | 0.306 |  | 0.424 | 10.75 | 0.206 |  | 1.000 | 1.60 | 0.689

Figure 3. SR* of compared methods at different turns on four datasets (RQ1)

###  5.3. Performance comparison of AVPPL with existing models (RQ1)

Table 2 reports the SR@15, AT and hDCG@($15,10$) for AVPPL and baseline
models. AVPPL achieved significantly higher scores on all metrics and
datasets, demonstrating its effectiveness in the VPMCR scenario. The
performance gap was largest on MovieLens, likely because movie recommendations
are a relatively simple task and AVPPL better models user preferences for
items.

Fig. 3 shows the relative success rate (SR*) of each model at every turn
compared to the MCMIPL baseline (represented by the dark green line at $y=0$).
Observing the variation trend of curves in Fig. 3, we have the following
findings:

  * •

AVPPL almost consistently and substantially surpassed all baselines over the
entire conversation session across datasets. Specifically, AVPPL achieved a
high recommendation success rate in the first a few turns on MovieLens,
demonstrating its ability to precisely capture users’ preferences.

  * •

As the conversation continues, the performance gap between AVPPL and other
baselines widened, especially compared to Max Entropy. The lack of an adaptive
policy caused Max Entropy to require excessive turns, while AVPPL dynamically
predicts the best action at each turn based on the user responses and the
personalized recommendation policy learned via reinforcement learning.

  * •

Reinforcement learning-based methods like CRM and EAR lag behind more advanced
models, as they directly apply RL to a large decision space without
effectively representing the conversation state, hindering optimal policy
learning. In contrast, graph-based models such as SCPR, UNICORN, and MCMIPL
leverage graph structures to achieve state-of-the-art performance on some
datasets, but still fall short of AVPPL’s performance.

###  5.4. Evaluating Key Design in AVPPL (RQ2)

Table 3. Ablation study of AVPPL in VPMCR (top) and comparison of AVPPL with
other baselines in MIMCR (bottom).

| Yelp |  | LastFM |  | Amazon-Book |  | MovieLens
---|---|---|---|---|---|---|---
SR@15 | AT | hDCG |  | SR@15 | AT | hDCG |  | SR@15 | AT | hDCG |  | SR@15 | AT | hDCG
AVPPL - (VPMCR) | 0.398 | 11.26 | 0.175 |  | 0.686 | 8.58 | 0.306 |  | 0.424 | 10.75 | 0.206 |  | 1.000 | 1.60 | 0.689
(a) - w/o USE Item.Score | 0.328 | 12.04 | 0.144 |  | 0.618 | 9.35 | 0.271 |  | 0.386 | 11.17 | 0.189 |  | 0.852 | 3.84 | 0.593
(b) - w/o USE Attr.Score | 0.354 | 11.88 | 0.149 |  | 0.614 | 9.44 | 0.267 |  | 0.412 | 10.91 | 0.199 |  | 1.000 | 1.75 | 0.663
(c) - w/o Personalized Preference | 0.142 | 13.84 | 0.060 |  | 0.444 | 10.79 | 0.211 |  | 0.284 | 12.10 | 0.142 |  | 0.858 | 5.22 | 0.492
(d) - w/o Average Preference | 0.368 | 11.38 | 0.169 |  | 0.630 | 9.24 | 0.269 |  | 0.416 | 10.84 | 0.199 |  | 1.000 | 1.77 | 0.668
(e) - w/o Decaying Preference | 0.382 | 11.56 | 0.163 |  | 0.628 | 9.15 | 0.280 |  | 0.410 | 11.05 | 0.190 |  | 1.000 | 1.49 | 0.708
AVPPL - (MIMCR) | 0.636 | 10.68 | 0.210 |  | 0.840 | 7.33 | 0.350 |  | 0.610 | 9.81 | 0.251 |  | 0.988 | 2.42 | 0.640
MCMIPL - (MIMCR) | 0.552 | 10.95 | 0.204 |  | 0.856 | 7.21 | 0.342 |  | 0.544 | 10.32 | 0.239 |  | 0.838 | 4.23 | 0.602
UNICORN - (MIMCR) | 0.454 | 11.01 | 0.188 |  | 0.832 | 7.42 | 0.350 |  | 0.530 | 10.23 | 0.231 |  | 0.832 | 4.35 | 0.567
SCPR - (MIMCR) | 0.452 | 12.52 | 0.136 |  | 0.688 | 10.27 | 0.220 |  | 0.450 | 11.10 | 0.167 |  | 0.834 | 4.80 | 0.392

####  5.4.1. Key Components of AVPPL

We examine the effectiveness of Uncertainty-aware Soft Estimation (USE), our
framework’s main design, in guiding conversations and adapting to user
preference changes in VPMCR scenarios. We separately remove the USE module for
items and attributes (Section 4.1) and replace them with a preference-based
scoring strategy (Deng et al., 2021; Zhang et al., 2022b), which models user
preferences using historical click or non-click attributes as mixed signals.

Table 3 rows (a-b) display the ablation study results. Removing the USE module
for both items and attributes significantly degrades performance across all
datasets, emphasizing the importance of considering user preference
uncertainty. The USE module allows our model to learn a sophisticated
conversational state representation and prune a more reasonable action space
for the Unified Policy Learning (UPL) module, enhancing the upper bound for
unified policy learning.

We also find that the USE component is more effective in measuring user
preferences for items than attributes in VPMCR scenarios, suggesting that
click behavior provides more direct item-related information.

####  5.4.2. Key Components of USE

Table 3 rows (c-e) present the ablation experiments for the USE component. Row
(c) shows that personalized information for user modeling is crucial; without
it, the model cannot capture personalized preferences, severely limiting
performance. Removing the average preference in Equation 3 (Row (d)) degrades
performance across all datasets, with LastFM suffering the most. This may be
due to LastFM’s numerous attributes and the significant impact of non-
displayed attribute information on user preference estimation. Additionally,
we remove the historical decay preference in time-aware preference decay (Row
(e)), leading to performance degradation on three datasets except for
MovieLens. On MovieLens, USE without decaying information reliably estimates
preferences in the current turn, and recommendations succeed within 1-2
rounds. Thus, introducing historical decay preference in short interactive
rounds may weaken preference inference on MovieLens.

Overall, the results confirm the USE module’s importance and the proposed
AVPPL framework’s effectiveness.

####  5.4.3. VPMCR vs. MIMCR Scenarios

To comprehensively evaluate AVPPL’s effectiveness in modeling user preferences
based on click behaviors, we relax the scenario assumption and employ the
MIMCR scenario involving multi-choice question interactions. In MIMCR, user
feedback signals are treated as strong indicators to filter items.

Table 3 compares AVPPL’s performance with advanced baselines in the MIMCR
scenario. Our method shows significant advantages on Yelp, Amazon-book, and
Movielens datasets. On LastFM, although slightly inferior to MCMIPL in SR and
AT, AVPPL outperforms all w.r.t. hDCG. These results confirm AVPPL’s
effectiveness in eliciting user preferences in multi-choice question
scenarios, demonstrating its universality and effectiveness in handling both
VPMCR and MIMCR scenarios.

Figure 4. Comparative performance analysis of Success Rate with varying decay
factor (left) and proportion of vague preference (right)
hyperparameters.(RQ3).

###  5.5. Model Parameter Analysis RQ3

Table 4. The impact of the coefficient of information intensity w.r.t. SR@15. Dataset | Yelp |  | Amazon-Book
---|---|---|---
$\lambda_{2}$ | 0.01 | 0.1 | 1 |  | 0.01 | 0.1 | 1
$\lambda_{1}$ | 0.01 | 0.414 | 0.408 | 0.328 |  | 0.424 | 0.430 | 0.400
0.1 | 0.398 | 0.410 | 0.344 |  | 0.424 | 0.414 | 0.384
1 | 0.394 | 0.370 | 0.302 |  | 0.420 | 0.398 | 0.406

The previous work on graph-based policy learning (Deng et al., 2021), has
conducted relevant hyperparameter analysis regarding policy learning. Here we
focus on the analysis of the hyperparameter impact of the core module (USE) in
AVPPL in the VPMCR scenario. Due to the limited space, we only present results
for Yelp and Amazon-Book, but note that LastFM and Movielens exhibit similar
trends.

####  5.5.1. Hyperparameter Analysis in USE

We identified two key hyperparameters: (1) The information intensity
coefficients $\lambda_{1}$ and $\lambda_{2}$ control the importance of
explicit versus implicit preferences. The results presented in Table 4 show
that larger $\lambda_{1}$ and smaller $\lambda_{2}$ resulted in higher success
rates, indicating that explicit preferences ($\lambda_{1}$) are more crucial
than implicit preferences ($\lambda_{2}$) in VPMCR. Notably, performance
decreases when both $\lambda_{1}$ and $\lambda_{2}$ are large, especially for
sparser datasets like Yelp, posing a challenge to the model’s robustness. (2)
The decay factor $\gamma$ controls the trade-off between recent and historical
preferences. Fig. 4 shows that a moderate decay factor (0.6-0.8) performs
best, suggesting that a balance between recent and historical preferences is
optimal. Extreme values (0.1 and 1.0) perform poorly, indicating that
disregarding historical preferences or solely relying on recent ones is
suboptimal.

####  5.5.2. Proportion of Vague Preferences

We conducted experiments with varying vague preference proportions (0.1 to 1).

In Fig. 4, higher success rates occurred at moderate vague preference
proportions. With a moderate level of vague preferences (around 40-50%), the
model balances the ability to utilize both vague and explicit preferences,
resulting in better recommendations. However, when vague preferences dominated
(over 70-80%), the model struggled to accurately determine user needs,
hampering performance.

###  5.6. Case Study RQ4

Figure 5. The left figure displays a sample conversation generated by our
AVPPL and the right figure illustrates the changes in the user preference
distribution during the conversation.

In this case study from the Yelp dataset (Fig. 5), the user initiated a
conversation with a clear preference of finding a beverage shop, prompting the
initialization of the user’s distribution space across all potential
locations. The user had a clear preference for “ _tea $\&$ coffee_” but was
vague about their preferences for “ _price_ ” and “ _leisure food_ ”. Our
proposed method takes into account the user’s click/non-click behavior to
update the user’s preference distribution on all beverage establishments
accordingly. This is in contrast to the traditional approach of filtering out
items based on click/non-click signals.

After the third turn of conversation, the combination of the user’s immediate
feedback (clicking on “ _dessert_ ” and not clicking on “ _smoothies_ ” and
historical feedback (“ _price_ ” and “ _tea $\&$ coffee_”) resulted in
identifying two target items, “ID:69761” and “ID:25587”, with the highest
preference estimate.

##  6\. conclusion

We propose a realistic Vague Preference Multi-round Conversational
Recommendation (VPMCR), which considers the user’s vague and volatile
preferences. By addressing the limitations of existing CRS scenarios and
incorporating the VPMCR scenario and AVPPL solution, we aim to improve the
overall performance and applicability of CRS in real-world settings,
particularly for users with vague or dynamic preferences. We hope the findings
will provide valuable insights into developing user-centric CRSs that can
handle users’ vague and dynamic preferences. In future work, we plan to
explore more sophisticated vague preference modeling and more efficient policy
learning techniques to further enhance the performance and generalizability of
AVPPL in VPMCR.

## References

  * (1)
  * Afsar et al. (2022) M Mehdi Afsar, Trafford Crump, and Behrouz Far. 2022\.  Reinforcement Learning based Recommender Systems: A Survey.  _Comput. Surveys_ 55, 7 (2022), 1–38.
  * Bettman et al. (1998) James R Bettman, Mary Frances Luce, and John W Payne. 1998\.  Constructive consumer choice processes.  _Journal of consumer research_ 25, 3 (1998), 187–217.
  * Bordes et al. (2013) Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013\.  Translating embeddings for modeling multi-relational data.  _Advances in neural information processing systems_ 26 (2013).
  * Chandramohan et al. (2011) Senthilkumar Chandramohan, Matthieu Geist, Fabrice Lefevre, and Olivier Pietquin. 2011.  User simulation in dialogue systems using inverse reinforcement learning. In _Interspeech 2011_. 1025–1028.
  * Chen et al. (2019) Qibin Chen, Junyang Lin, Yichang Zhang, Ming Ding, Yukuo Cen, Hongxia Yang, and Jie Tang. 2019.  Towards Knowledge-Based Recommender Dialog System. In _EMNLP-IJCNLP_. 1803–1813.
  * Chen et al. (2022) Yankai Chen, Huifeng Guo, Yingxue Zhang, Chen Ma, Ruiming Tang, Jingjie Li, and Irwin King. 2022.  Learning Binarized Graph Representations with Multi-Faceted Quantization Reinforcement for Top-K Recommendation. In _Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_ _(KDD ’22)_. Association for Computing Machinery, New York, NY, USA, 168–178.  https://doi.org/10.1145/3534678.3539452
  * Deffayet et al. (2023) Romain Deffayet, Thibaut Thonet, Jean-Michel Renders, and Maarten de Rijke. 2023.  Offline Evaluation for Reinforcement Learning-based Recommendation: A Critical Issue and Some Alternatives.  _arXiv preprint arXiv:2301.00993_ (2023).
  * Deng et al. (2021) Yang Deng, Yaliang Li, Fei Sun, Bolin Ding, and Wai Lam. 2021.  Unified conversational recommendation policy learning via graph-based reinforcement learning. In _Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_. 1431–1441.
  * Gao et al. (2023) Chongming Gao, Kexin Huang, Jiawei Chen, Yuan Zhang, Biao Li, Peng Jiang, Shiqi Wang, Zhong Zhang, and Xiangnan He. 2023.  Alleviating Matthew Effect of Offline Reinforcement Learning in Interactive Recommendation. In _Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval_ _(SIGIR ’23)_. 11.  https://doi.org/10.1145/3539618.3591636
  * Gao et al. (2022a) Chongming Gao, Wenqiang Lei, Jiawei Chen, Shiqi Wang, Xiangnan He, Shijun Li, Biao Li, Yuan Zhang, and Peng Jiang. 2022a.  CIRS: Bursting Filter Bubbles by Counterfactual Interactive Recommender System.  _arXiv preprint arXiv:2204.01266_ (2022).
  * Gao et al. (2021) Chongming Gao, Wenqiang Lei, Xiangnan He, Maarten de Rijke, and Tat-Seng Chua. 2021.  Advances and Challenges in Conversational Recommender Systems: A Survey.  _AI Open_ 2 (2021), 100–126.  https://doi.org/10.1016/j.aiopen.2021.06.002
  * Gao et al. (2022b) Chongming Gao, Shijun Li, Wenqiang Lei, Jiawei Chen, Biao Li, Peng Jiang, Xiangnan He, Jiaxin Mao, and Tat-Seng Chua. 2022b.  KuaiRec: A Fully-observed Dataset and Insights for Evaluating Recommender Systems. In _Proceedings of the 31st ACM International Conference on Information and Knowledge Management_ _(CIKM ’22)_. 11.
  * Guo et al. (2021) Wei Guo, Rong Su, Renhao Tan, Huifeng Guo, Yingxue Zhang, Zhirong Liu, Ruiming Tang, and Xiuqiang He. 2021\.  Dual Graph Enhanced Embedding Neural Network for CTR Prediction. In _Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining_ _(KDD ’21)_. Association for Computing Machinery, New York, NY, USA, 496–504.  https://doi.org/10.1145/3447548.3467384
  * Han et al. (2018) Xu Han, Shulin Cao, Xin Lv, Yankai Lin, Zhiyuan Liu, Maosong Sun, and Juanzi Li. 2018.  Openke: An open toolkit for knowledge embedding. In _Proceedings of the 2018 conference on empirical methods in natural language processing: system demonstrations_. 139–144.
  * He et al. (2022) Zhankui He, Handong Zhao, Tong Yu, Sungchul Kim, Fan Du, and Julian McAuley. 2022\.  Bundle MCR: Towards Conversational Bundle Recommendation. In _Proceedings of the 16th ACM Conference on Recommender Systems_ _(RecSys ’22)_. Association for Computing Machinery, New York, NY, USA, 288–298.  https://doi.org/10.1145/3523227.3546755
  * Kipf and Welling (2016) Thomas N Kipf and Max Welling. 2016.  Semi-supervised classification with graph convolutional networks.  _arXiv preprint arXiv:1609.02907_ (2016).
  * Lei et al. (2020a) Wenqiang Lei, Xiangnan He, Yisong Miao, Qingyun Wu, Richang Hong, Min-Yen Kan, and Tat-Seng Chua. 2020a.  Estimation–Action–Reflection: Towards Deep Interaction Between Conversational and Recommender Systems. In _WSDM_. 304–312.
  * Lei et al. (2020b) Wenqiang Lei, Gangyi Zhang, Xiangnan He, Yisong Miao, Xiang Wang, Liang Chen, and Tat-Seng Chua. 2020b.  Interactive Path Reasoning on Graph for Conversational Recommendation. In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_ _(KDD ’20)_. 2073–2083.
  * Li et al. (2018) Raymond Li, Samira Ebrahimi Kahou, Hannes Schulz, Vincent Michalski, Laurent Charlin, and Chris Pal. 2018.  Towards Deep Conversational Recommendations. In _Proceedings of the 32nd International Conference on Neural Information Processing Systems_ _(NeurIPS ’18)_. 9748–9758.
  * Li et al. (2021) Shijun Li, Wenqiang Lei, Qingyun Wu, Xiangnan He, Peng Jiang, and Tat-Seng Chua. 2021\.  Seamlessly Unifying Attributes and Items: Conversational Recommendation for Cold-Start Users.  _ACM Trans. Inf. Syst._ 39, 4, Article 40 (aug 2021), 29 pages.  https://doi.org/10.1145/3446427
  * Liu et al. (2022) Dugang Liu, Mingkai He, Jinwei Luo, Jiangxu Lin, Meng Wang, Xiaolian Zhang, Weike Pan, and Zhong Ming. 2022\.  User-Event Graph Embedding Learning for Context-Aware Recommendation. In _Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_ _(KDD ’22)_. Association for Computing Machinery, New York, NY, USA, 1051–1059.  https://doi.org/10.1145/3534678.3539458
  * Montazeralghaem and Allan (2022) Ali Montazeralghaem and James Allan. 2022.  Learning Relevant Questions for Conversational Product Search Using Deep Reinforcement Learning. In _Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining_ _(WSDM ’22)_. Association for Computing Machinery, New York, NY, USA, 746–754.  https://doi.org/10.1145/3488560.3498526
  * Moon et al. (2019) Seungwhan Moon, Pararth Shah, Anuj Kumar, and Rajen Subba. 2019\.  OpenDialKG: Explainable Conversational Reasoning with Attention-based Walks over Knowledge Graphs. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ _(ACL ’19)_. 845–854.
  * Ren et al. (2022) Zhaochun Ren, Zhi Tian, Dongdong Li, Pengjie Ren, Liu Yang, Xin Xin, Huasheng Liang, Maarten de Rijke, and Zhumin Chen. 2022.  Variational Reasoning about User Preferences for Conversational Recommendation. In _Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval_ _(SIGIR ’22)_. Association for Computing Machinery, New York, NY, USA, 165–175.  https://doi.org/10.1145/3477495.3532077
  * Sadeghi Eshkevari et al. (2022) Soheil Sadeghi Eshkevari, Xiaocheng Tang, Zhiwei Qin, Jinhan Mei, Cheng Zhang, Qianying Meng, and Jia Xu. 2022\.  Reinforcement Learning in the Wild: Scalable RL Dispatching Algorithm Deployed in Ridehailing Marketplace. In _Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_ _(KDD ’22)_. Association for Computing Machinery, New York, NY, USA, 3838–3848.  https://doi.org/10.1145/3534678.3539095
  * Sun and Zhang (2018) Yueming Sun and Yi Zhang. 2018.  Conversational Recommender System. In _SIGIR_. 235–244.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ 30 (2017).
  * Wang et al. (2022a) Shiqi Wang, Chongming Gao, Min Gao, Junliang Yu, Zongwei Wang, and Hongzhi Yin. 2022a.  Who Are the Best Adopters? User Selection Model for Free Trial Item Promotion.  _IEEE Transactions on Big Data_ (2022).
  * Wang et al. (2019) Xiang Wang, Xiangnan He, Yixin Cao, Meng Liu, and Tat-Seng Chua. 2019.  Kgat: Knowledge graph attention network for recommendation. In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 950–958.
  * Wang et al. (2022b) Xiaolei Wang, Kun Zhou, Ji-Rong Wen, and Wayne Xin Zhao. 2022b.  Towards Unified Conversational Recommender Systems via Knowledge-Enhanced Prompt Learning. In _Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_ _(KDD ’22)_. Association for Computing Machinery, New York, NY, USA, 1929–1937.  https://doi.org/10.1145/3534678.3539382
  * Wu et al. (2021) Jiancan Wu, Xiang Wang, Fuli Feng, Xiangnan He, Liang Chen, Jianxun Lian, and Xing Xie. 2021.  Self-supervised graph learning for recommendation. In _Proceedings of the 44th international ACM SIGIR conference on research and development in information retrieval_. 726–735.
  * Wu et al. (2019) Wenquan Wu, Zhen Guo, Xiangyang Zhou, Hua Wu, Xiyuan Zhang, Rongzhong Lian, and Haifeng Wang. 2019.  Proactive human-machine conversation with explicit conversation goals.  _arXiv preprint arXiv:1906.05572_ (2019).
  * Xu et al. (2020) Hu Xu, Seungwhan Moon, Honglei Liu, Bing Liu, Pararth Shah, Bing Liu, and Philip Yu. 2020.  User Memory Reasoning for Conversational Recommendation. In _Proceedings of the 28th International Conference on Computational Linguistics_ _(COLING ’20)_. 5288–5308.
  * Xu et al. (2021) Kerui Xu, Jingxuan Yang, Jun Xu, Sheng Gao, Jun Guo, and Ji-Rong Wen. 2021.  Adapting User Preference to Online Feedback in Multi-Round Conversational Recommendation. In _Proceedings of the 14th ACM International Conference on Web Search and Data Mining_ _(WSDM ’21)_. 364–372.
  * Xue et al. (2023) Wanqi Xue, Qingpeng Cai, Ruohan Zhan, Dong Zheng, Peng Jiang, and Bo An. 2023\.  ResAct: Reinforcing Long-term Engagement in Sequential Recommendation with Residual Actor. In _International Conference on Learning Representations_ _(ICLR ’23)_.
  * Zhang et al. (2022a) Qihua Zhang, Junning Liu, Yuzhuo Dai, Yiyan Qi, Yifan Yuan, Kunlun Zheng, Fan Huang, and Xianfeng Tan. 2022a.  Multi-Task Fusion via Reinforcement Learning for Long-Term User Satisfaction in Recommender Systems. In _Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_ _(KDD ’22)_. Association for Computing Machinery, New York, NY, USA, 4510–4520.  https://doi.org/10.1145/3534678.3539040
  * Zhang et al. (2022b) Yiming Zhang, Lingfei Wu, Qi Shen, Yitong Pang, Zhihua Wei, Fangli Xu, Bo Long, and Jian Pei. 2022b.  Multiple Choice Questions based Multi-Interest Policy Learning for Conversational Recommendation. In _Proceedings of the ACM Web Conference 2022_. 2153–2162.
  * Zhou et al. (2020b) Kun Zhou, Wayne Xin Zhao, Shuqing Bian, Yuanhang Zhou, Ji-Rong Wen, and Jingsong Yu. 2020b.  Improving Conversational Recommender Systems via Knowledge Graph based Semantic Fusion. In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_ _(SIGKDD’ 20)_. 1006–1014.
  * Zhou et al. (2020c) Kun Zhou, Yuanhang Zhou, Wayne Xin Zhao, Xiaoke Wang, and Ji-Rong Wen. 2020c.  Towards Topic-Guided Conversational Recommender System. In _Proceedings of the 28th International Conference on Computational Linguistics_ _(COLING ’2020)_.
  * Zhou et al. (2020a) Sijin Zhou, Xinyi Dai, Haokun Chen, Weinan Zhang, Kan Ren, Ruiming Tang, Xiuqiang He, and Yong Yu. 2020a.  Interactive Recommender System via Knowledge Graph-Enhanced Reinforcement Learning. In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_ _(SIGIR’ 20)_. 179–188.
  * Zhou et al. (2022) Yuanhang Zhou, Kun Zhou, Wayne Xin Zhao, Cheng Wang, Peng Jiang, and He Hu. 2022\.  C²-CRS: Coarse-to-Fine Contrastive Learning for Conversational Recommender System. In _Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining_ _(WSDM ’22)_. Association for Computing Machinery, New York, NY, USA, 1488–1496.  https://doi.org/10.1145/3488560.3498514
  * Zou et al. (2020) Jie Zou, Yifan Chen, and Evangelos Kanoulas. 2020.  Towards Question-Based Recommender Systems. In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_ _(SIGIR ’20)_. 881–890.


Creating new air combat tactics and discovering novel maneuvers can require numerous hours of expert pilots' time. Additionally, for each different combat scenario, the same strategies may not work since small changes in equipment performance may drastically change the air combat outcome. For this reason, we created a reinforcement learning environment to help investigate potential air combat tactics in the field of beyond-visual-range (BVR) air combat: the BVR Gym. This type of air combat is important since long-range missiles are often the first weapon to be used in aerial combat. Some existing environments provide high-fidelity simulations but are either not open source or are not adapted to the BVR air combat domain. Other environments are open source but use less accurate simulation models. Our work provides a high-fidelity environment based on the open-source flight dynamics simulator JSBSim and is adapted to the BVR air combat domain. This article describes the building blocks of the environment and some use cases.

Reinforcement Learning, Beyond Visual Range Air Combat

\(h\) Agents altitude
\(v\) Agents air speed
\(v_{D}\) Agents down velocity
\(\psi\) Agents heading
\(\rho\) Distance to the firing position
\(\nu\) Initial launch velocity
\(\tau\) Time since missile launch
\(\eta\) Relative angle to firing position
\(\beta\) Altitude of the firing position
\(MD\)Miss-Distance
\(a_{Head}\)Set heading
\(a_{Alt}\)Set altitude
\(a_{Thr}\)Set thrust

§ INTRODUCTION

The nature of air combat has changed dramatically in the past half a century. Pilots can engage hostile aircraft at increasing distances due to sensors, armaments, and communication improvements. This development allows pilots to switch from within-visual range (WVR) combat to beyond-visual range (BVR)  combat. Given all these technological advancements, BVR air warfare is currently the most effective type of air combat [1]. When pilots train, it is vital that they are exposed to a large variety of situations and opponent tactics.
Manually creating such situations and tactics can be difficult and time-consuming.
One possible way to alleviate this problem is to use Reinforcement Learning (RL).

RL has been applied to a large variety of problem domains. A recent work [2] focuses on providing a customizable environment where the agent can learn team strategies for football. Other environments present challenges in agriculture, traffic management, and product recommendations [3, 4, 5]. In  [6], the authors noted that there is a lack of standard environments for aerospace problems. For this reason, the latter work developed an Aerospace SafeRL Framework that includes environments for aircraft formation flight and spacecraft docking in both 2D and 3D environments.

Below, we discuss some noteworthy recent high-fidelity flight dynamics simulation engines available. AirSim [7] is one of the more cited flight dynamics simulation environments used for AI research in the aerospace domain. It is an open-source platform built upon the game platform Unreal Engine and aims to narrow the gap between simulation and reality. Gazebo [8] is another high-fidelity simulation framework popular among robotics researchers and extends to the aerospace domain, with a focus on multi-rotor drones. X-plane [<https://github.com/nasa/XPlaneConnect>] from Nasa is a high-fidelity simulation environment. In [9], authors used an X-plane flight dynamics engine for data collection, which was later used to train an autopilot in the form of a feed-forward neural network (FNN).  The authors of [10] used the Double Deep Q-Network (DDQN) approach to train an agent for attitude control. This work used X-plane to verify the trained agents' ability to deal with complex environments. Another high-fidelity flight dynamics simulation environment is JSBSim [11]. One important recent event within the air combat domain was the DARPA AlphaDogfight Trials [<https://www.darpa.mil/news-events/2020-08-26>], where teams competed against each other on algorithms that are capable of performing simulated WVR air combat maneuvering and finally competing against experienced Air Force F-16 pilots. The authors of [12] participated in the trials and used RL to train an agent for this specific competition. The simulation environment used within this competition was based on the JSBSim flight dynamics engine, operating within a WVR air combat setting.

Given the currently available RL environments, as seen in Table <ref>, there is a need for an open-source high-fidelity environment to explore tactics within the BVR air combat domain.

Overview of Simulation environments

Simulation Environment     High Fidelity     Open source     BVR

AirSim [7]

Gazebo [8]

X-plane [<https://github.com/nasa/XPlaneConnect>]

WUKONG [13]

JSBSim [11]

General Motion Model[14]

BVRGym [15] (our approach)

The main contributions of this paper are as follows.
We propose a BVR air combat environment based on the high-fidelity flight dynamics simulator JSBSim. Key contributions are listed below:

* it is open source

* it provides a set of BVR scenarios with easy integration into different RL algorithms

* it provides a BVR missile model with a Proportional Navigation (PN) guidance law

* it gives the ability to customize and create new scenarios

The library and additional documentation are available here[<https://github.com/xcwoid/BVRGym>], and Table <ref> above provides a comparison to related simulation environments.

§ BACKGROUND

This framework consists of the following components: (i) Tactical units that are used to conduct BVR air combat. We use a military aircraft model and a long-range missile for this work. These models are explicitly adapted for BVR air combat. (ii) To enable the participation of units applying manually designed policies in the scenarios, we include a behavior tree implementing a simple but extendable BVR policy.  (iii) To facilitate the use of a wide range of RL algorithms, we developed a simple OpenAI-Gym-like interface [16].
Additional effort has been made to make it similar to an actual BVR air combat training. In such scenarios, the teams are usually split into two groups: the blue and red teams, the red team being the adversary. In these scenarios, aircraft use radars to track the position of the opposing team since the distances implied within BVR are generally up to 100 km. To detect an aircraft at such distances, pilots use onboard radars. When the opposing team launches a missile, it is possible to detect the launch; when the missile engine ignites, it might be captured by the Infrared-Search and Track (IRST) sensors, and the detection can be associated with the tracked aircraft. In this case, estimating from which adversarial aircraft the missile has been launched is possible. Tracking the missile, on the other hand, is a much more challenging task since the missile is much smaller than the size of the aircraft that launched it. For this reason, our training environments only enable using the knowledge of from where the missile was launched and not the current position of the missile.
Additionally, BVR air combat evolves on a slower time scale than WVR air combat, which makes typical RL rewards sparse and training for RL agents more challenging, as fewer things happen during more extended periods of time and the exploration space is large.
As noted above, the BVR Gym enables the manual design of policies using a behavior tree (BT), a switching structure that has been shown to be optimally modular [17] and used to create extendable hierarchical control structures in robotics [18, 19].
Below, we describe the basic concepts of RL and the BTs.

§.§ Reinforcement learning

Reinforcement learning (RL) is a subfield of machine learning that obtains knowledge by dynamic interaction with an environment, and it offers a powerful method to train an agent for intelligent decision-making. The agent is the learning entity's representative in this scenario, and a strategy that directs the agent's decision-making process is at the core of the agent. Unlike supervised learning, where the trained model is learned on a labeled dataset, the field of RL focuses on unsupervised learning, where the model focuses on discovering patterns without explicit guidance through a set of discrete time steps. At each time step $t$, the agent perceives the environment through state representation $s_t$, can select an action $a_t$ from a set of possible actions, and receives a numerical reward $r_{t}$ provided by the environment [20].

Depending on the agent's condition, the obtained reward might be used to assess how good or bad a particular action was. Thus, the agent's goal is to find a $\pi(a|s)$ policy that maximizes these long-term rewards given the agent's current state. The problem can be mathematically formulated as

\pi^* = \arg\max_\pi \mathbb{E}\left(\sum_{t=0}^{\infty} \gamma^t r_{t+1} \mid \pi \right),

where $\gamma$ is the discount factor to balance immediate rewards against future rewards, the expectation $\mathbb{E}$ is taken over all possible sequences of states, actions, and rewards under the policy $\pi$.

The two main methods for solving RL problems are on-policy and off-policy learning. Q-learning is one form of off-policy learning, which separates the learning policy from the policy used to investigate the provided environment [20]. On the other hand, policies are updated by on-policy reinforcement learning algorithms, like REINFORCE, based on their experiences interacting with the environment while utilizing current policy. Off-policy approaches also benefit from a broader range of experiences brought along by various policies. Finding the right balance between exploitation and exploration is crucial in reinforcement learning. To maximize immediate benefits, the agent must leverage its present knowledge while exploring to the fullest extent possible to uncover optimum behaviors. In this work, we use on-policy optimization, such as the Proximal Policy Optimization (PPO) algorithm [21]. One of the reasons for selecting PPO is its robustness of hyper-parameter selections for different tasks and the reported performance of the algorithm when used to address real-world issues like attitude control for fixed-wing and quad-rotor aircraft [22], [23].

§.§ Behavior Trees

Behavior Trees (BT) are a hierarchical and modular [17] method used in robotics, artificial intelligence, and video games to describe the policy of autonomous entities. BTs were first created by video game programmers, but they have since been used in many other fields where complex decision-making and accomplishment of tasks are crucial [19].

BT structure controlling the F16 aircraft belonging to  Team $\mathcal{R}$ .

A behavior tree is a hierarchical structure depicting an agent's decision-making process. Unlike alternative frameworks, BTs offer a more adaptable and modular method of describing and classifying behaviors. We use BT in our work to create manual policies, and modularity is useful since BVR air combat can be decomposed into a number of complex behaviors. An example of a simple BT is illustrated in Figure <ref>. Each node in the tree structure represents a specific action or decision-making measure. The directed graph, which is made up of interconnected nodes, directs the agent through a series of decisions and actions. Control and task nodes are the two basic categories of nodes that comprise BTs [24]. Control nodes oversee the execution flow, choosing when and how to execute child nodes. In contrast, task nodes stand for discrete actions or decision points. There are two types of execution nodes (Action and Condition) and three primary categories of control nodes (Sequence, Fallback, and Parallel); however, for this work, we do not use the Parallel) node. Below, we will briefly describe the nodes of a given BT.

Sequences, illustrated by a box containing the label $\rightarrow$, execute its child nodes in sequence until one fails. Fallbacks, illustrated by a box containing the label $?$, executes its child nodes in sequence until one succeeds. Action nodes take actions, such as avoiding a missile, engaging with an enemy, or guiding your missile to the target, and Condition nodes check if a given condition is satisfied or not.  BTs have the advantage of being easily readable and modifiable. Designers may easily manipulate the tree structure to visualize and change the decision-making process. Because of this, BTs are a very useful tool in fields where quick iterations and prototyping are crucial.

§ TACTICAL UNITS

Two critical components of the BVR air combat are military aircraft, such as jet fighters with long-range detection systems, and long-range missiles. This section briefly describes the tactical units used within this simulation environment, namely the F-16 aircraft and the BVR missile and their properties.

§.§.§ F-16 Aircraft

We utilize the JSBsim F-16 flight dynamics model[<https://github.com/JSBSim-Team/jsbsim/tree/master/aircraft/f16>] for this training environment. While the F-16 model has its own predefined controllers to keep the inherently unstable aircraft stable, we added an additional high-level controller to adapt the unit for BVR air combat. In general, BVR air combat does not include aggressive maneuvering since pilots conserve the energy of their aircraft; for this reason, we have added an auto-pilot controller to steer the aircraft in the desired direction.  This allows the agent to set the desired heading, altitude, and throttle instead of controlling the attitude rates, reducing the RL search space and promoting faster convergence.

Thus, if the agent chooses to set a desired direction, the lower-level controllers automatically roll the aircraft and turn it to the desired direction. Similarly, if the agent chooses to change altitude, lower-level controllers automatically stabilize the aircraft, adjust the pitch angle, and execute the maneuver to achieve the desired altitude.

Since aircraft are complex systems, it is helpful to see the exact behavior of the unit of interest. For this reason, we added the possibility of studying aircraft behavior before deploying it to an RL environment. Figure <ref> captures aircraft dynamics while performing an evasive maneuver, including a decrease in altitude (to increase air density to promote missile deceleration) and a change in direction (to maximize the distance from the missile).

F-16 evasive maneuver. A drop in altitude initiates the maneuver, followed by a turn to evade the incoming missile.

§.§.§ BVR Missile

The capacity to interact with enemy fleets at great distances, up to 100 [km], is crucial in BVR air combat. For this reason, we developed a BVR Missile model. Since the exact performance of real missiles is highly classified, our missile model was inspired by performance estimates available from open sources[<https://en.wikipedia.org/wiki/AIM-120_AMRAAM>]. After being launched, a stage of acceleration is followed by the missile's ascent to a higher altitude. Since the air is less thick at higher altitudes, it is possible to increase the flight range by keeping a high altitude for as long as possible. To keep the missile model simple but realistic, we implemented a Proportional Navigation (PN) guidance law [25]. This law navigates the missile toward the target after reaching the desired cruise velocity and altitude.

The PN guidance law's solution provides the desired acceleration to change the missile heading to intercept a moving target. The acceleration is converted to the appropriate velocity vector and then sent to a lower-level controller to execute the turn. Like the F-16 unit, the user can change the missile's characteristics, including its precise launching location, initial velocity, altitude, cruise altitude, and target. More tools are available to aid with aligning the missile's initial heading in the direction of the target.

Similar to aircraft models, missiles are complex systems; hence, observing the missile's behavior before deploying it in an RL training environment is beneficial. Figure <ref> depicts the missile's flight dynamics properties while the target performs an evasive maneuver away from the missile.

Missile response to F-16 evasive maneuver. Initialized by acceleration stage to Mach 4 and an ascent in altitude.

§ SCENARIOS

This section introduces a set of example BVR problems for the agent to solve. We start by introducing a problem where a single aircraft is faced with a single incoming missile. Afterward, we present a more challenging problem: the aircraft has to evade two incoming missiles launched from different locations. Finally, we present a one vs one BVR air combat scenario, where the agent needs to figure out how to defeat an adversarial aircraft.

§.§ Evading a BVR Missile

Consider a situation where there is one unit on each team: a single F16 aircraft on the blue team $\mathcal{B}$ and a launched missile from the red team $\mathcal{R}$. In this environment, the agent aims to find a policy that maximizes the miss distance (MD) between itself and the incoming missile.
The following observations are available to the agent at each time step
s_t = (h, v_{D}, v, \psi, \nu, \tau, \eta, \beta, \rho).

These observations are chosen to represent parts of a realistic air combat scenario. In such cases, when a missile is launched, the knowledge of the current missile location is usually not available since the missile is too small for the radar to detect at long ranges. However, tracking the adversary aircraft that launched it is much less complicated. Thus, an assumption can be made: if we track an aircraft and detect a sudden flash (usually representing a missile launch), we can assume that the missile was launched from the aircraft where the flash occurred. Modern military aircraft are equipped with Missile Approach Warning systems (MAW) that may detect the flash associated with the missile launch.

When a missile launch has occurred, pilots tend to perform an evasive maneuver to evade the incoming missile. In BVR air combat, super maneuverability is not a must; thus, in most cases, complex maneuvers are not used in order to preserve aircraft momentum. For this reason, the action space can be broken down into the following actions.
a_t = (a_{Head}, a_{Alt}, a_{Thr}).
At each time step, the agent receives a reward $r_t = R(s_t,a_t)$ that is $r_t = 0$ except for the last step, when the missile has either hit the target or has depleted all fuel and speed so that it cannot get closer to it. In the case of a successful missile evasion, the agent then receives a reward $r_T > 0$ equivalent to the MD, which is equal to the smallest encountered distance between the agent and the missile.
If the missile hits the agent, this distance is zero, resulting in a zero reward.

Missile response to F-16 evasive maneuver. Initialized by acceleration stage to Mach 4 and an ascent in altitude.

§.§ Evading two BVR Missiles

We now consider an agent subjected to two incoming missiles launched simultaneously from different locations. This is an interesting problem to study since pilots in such situations must carefully consider their options when making maneuvers to avoid missiles. Focusing purely on evading one of the two missiles can expose the aircraft to the other one. Thus, multiple threats require splitting the focus and finding solutions that can avoid both. Additionally, more missiles usually mean more maneuvering, which makes controlling energy depletion challenging.

The observation assumptions about the two missile cases are similar to the single missile case. The agent has access to both launch locations for missiles M1 and M2 and the state observation
s_t = (h, v_{D}, v, \psi, \nu^{M1}, \tau^{M1}, \eta^{M1}, \beta^{M1}, \rho^{M1}, \nu^{M2}, \tau^{M2}, \eta^{M2}, \\ \beta^{M2}, \rho^{M2})
$, with the same action space available as in the problem with a single incoming missile. The reward is provided in a similar manner, where it is proportional to the smallest encountered distance between the agent and any of the two missiles. At every time step, the agent receives a reward $r_t^{M1} = R(s_t,a_t)$ and $r_t^{M2} = R(s_t,a_t)$ which both are equal to $0$ except for the last step. In the case of a successful missile evasion, the agent then receives a reward $min(r_{T}^{M1}, r_{T}^{M2} ) > 0$ equivalent to the MD, which is equal to the smallest encountered distance between the agent and the incoming missiles.

§.§ BVR DogFight

The aircraft's radar and other sensors play a critical role in the efficacy of BVR engagements. Challenges may arise from a sensor's limited precision, range, or sensitivity to electronic countermeasures. To simplify this, we consider that the location of the enemy aircraft is known without any interference. One of the crucial decisions that have to be made during air combat is the timing of when to fire and when to hold fire. Engaging in combat too soon could disclose the aircraft's location, while waiting too long could make you the target of the initial attack. BVR engagements also involve controlling the aircraft's altitude and speed to maximize missile performance after launch.

This scenario aims to find possible counterattack strategies against an enemy with known behavior.
To capture the enemy policy, we use BTs, which have shown to be an effective tool for creating sophisticated behaviors and have been used to define behaviors within the air combat domain [19]. Since BVR combat is rarely observable in practice, with low availability of historical data, much of its possibilities must be assessed through simulation [26]. For this reason, you might want to study potential solutions to a given adversary behavior.

We have equipped the adversary aircraft with a BT that dictates the actions to take during air combat. The strategy is visualized in Figure <ref>. The main focus of this BT is to prioritize one's own safety; for this reason, missile evasion tactics are placed on the left-hand side of the tree, while offensive tactics are located on the right-hand side. The following state observation is available to the agent $s_t = \\ (
\rho^{\mathcal{B}\mathcal{R}},
\nu^{\mathcal{B}\mathcal{R}},
\psi^{\mathcal{B}},
\rho^{\mathcal{B}\mathcal{M}}_{0},
\nu^{\mathcal{B}\mathcal{M}}_{0},

The components $(\rho^{\mathcal{B}\mathcal{M}}_{0}, \nu^{\mathcal{B}\mathcal{M}}_{0}, v^{\mathcal{M}}_{0}, h^{\mathcal{M}}_{0})$ indicating observations with respect to the launch location. If there are no active missiles launched by the $\mathcal{R}$ team aircraft, then $(\rho^{\mathcal{B}\mathcal{M}}_{0}, \nu^{\mathcal{B}\mathcal{M}}_{0}, v^{\mathcal{M}}_{0}, h^{\mathcal{M}}_{0})$ are equivalent to the $(\rho^{\mathcal{B}\mathcal{R}}, \nu^{\mathcal{B}\mathcal{R}}, v^{\mathcal{R}}, h^{\mathcal{R}})$, indicating that the missile is located at the same place as the aircraft carrying it. The following action space is available to the agent $a_t = (a_{Head}, a_{Alt}, a_{Thr}, a_{l})$, with an additional change being the missile launch capability $a_{l}$.

In this environment, the agent receives a reward $r_t = R(s_t,a_t)$, which is equal to $0$ except for the last step. In the case of a successful adversary kill, the agent then receives a reward $r_{T} = 1$. If the adversary successfully shoots down the agent or the agent hits the ground, or the scenario time runs out, then a reward of $r_{T} = -1$; is provided.

§ NUMERICAL RESULTS

This section presents results obtained from training an agent in different environments. We first consider the environment where the agent is faced with one and two incoming missiles, followed by a BVR air combat scenario.

§.§ One and two missile scenario

Figure <ref> shows the values obtained from the training with both one and two incoming missiles. The initial conditions of the agent and the missiles are both randomized. Both the missile's initial launch conditions and the agent's initial state are presented in Table <ref>.

Table of initial conditions.

Parameter     Value

Initial Velocity: Agent      $300 - 365 $ [m/s]

Initial Velocity: M1,M2       $280 - 320 $ [m/s]

Initial Altitude: Agent      $6000 - 10000 $ [m]

Initial Altitude: M1,M2       $9000 - 11000 $ [m]

Firing Distance: M1,M2       $40 - 80$ [km]

Initial heading: Agent      $0-360 $ [deg]

Initial pitch/roll: Agent      $0 , 0$ [deg]

A closer look at Figure <ref> reveals that the agent typically achieves a greater separation with a single missile scenario because it is easier to establish a strategy without conflicting objectives than in a two-missile scenario. The trained model was able to improve the evasive maneuver in both situations, increasing the distance between the missile and the aircraft and preventing a missile hit. The primary tactic employed is the same as what pilots practice in training: lowering the aircraft's altitude to be surrounded by denser air, which promotes the missile to slow down and travel in the opposite direction from its launch point. Air combat problems with high-fidelity models typically require large computational budgets, as in [12]. We decrease the search space by setting the throttle to maximum, leaving the agent with the action space $a_t = (a_{Head}, a_{Alt})$. Such changes, in combination with a lower-level flight controller, reduce the need for such a budget. Additional changes have been made compared to our previous work in [27] to speed up convergence, and all the parameters can be found in [15]. One day's worth of computing is needed to solve the problem using an Intel Core<EMAIL_ADDRESS>ten CPUs operating in parallel, and one NVIDIA GeForce GTX 1080 GPU for neural network optimization. Changing the default settings, such as step time, the ability to control thrust, or increasing observation space, may significantly increase computation time.

Training results for evading one and two missile scenarios. The Y-axis shows the average distance by which the missile misses its target in kilometers.

§.§ BVR Dogfight

In this scenario, we let two aircraft face each other with two BVR missiles each. Each scenario lasts up to 16 minutes, slightly longer than in a related work done by [26] where the authors used a 12-minute time cap. The agent's goal in this scenario is to explore tactical policies to defeat the opposing aircraft that behaves according to the BT formulation in Figure <ref>. The training progress results are shown in Figure <ref>.

One vs One BVR air combat training results. The Figure illustrates the average obtained reward (Accumulated Reward) of ten in parallel executed episodes.

An illustration of the success rate of utilizing the first missile.

An illustration of the success rate of utilizing the second missile.

Upon examining Figure <ref>, we can observe that the agent is consistently shot down by the adversary at the beginning. Following a training period, we observe that the agent begins to behave better and wins more battles than it loses. A closer examination of the issue may also be done by examining how weapons are used in Figures <ref> - <ref>. Figure <ref> shows the frequency of the agent's missile usage. The missiles were not fired since the agent did not first approach the enemy to get within firing range. Following a period of training, we observed that the first missile was launched periodically and then, in the later phases of the training, a second missile. Looking at the adversary's actions, we can observe that it consistently, and with a high likelihood of success, used the first missile successfully. But when the agent became adept at dodging the first missile, the enemy began to use the second missile even more frequently.

BVR air combat, incorporating several tactical units, often leads to computationally intensive problems. To speed up the process, we introduced the following steps: (i) reduce the search space concerning the previous environments, as described in Section <ref>, (ii) let the agent make decisions once in 10 seconds, (iii) no variation in starting position for the agent and the adversary (iv) the missile launch is automated. This implies that the missile will be launched automatically when the launch conditions are satisfied. The main reason is to limit exploration space since missile launch action is a one-time action lasting for approximately 2-4 minutes of simulation time, depending on the distance. If the missile launch is made outside its range, the missile can be considered lost. The same computational resources were used in this scenario as in  Section <ref> above, and all the parameters and the code can be found in [15].

§ CONCLUSION

In this work, we presented a high-fidelity environment to investigate tactics with Beyond Visual Range air combat. We showed three case study scenarios to explore different aspects of BVR air combat. We also suggested some configuration parameters that enable users with limited computational resources to investigate the problems.

§ ACKNOWLEDGMENT

The authors gratefully acknowledge funding from Vinnova, NFFP7, dnr 2017-04875.

[1]
John Stillion.
Trends in air-to-air combat: Implications for future air
Center for Strategic and Budgetary Assessments., 2015.

[2]
Karol Kurach, Anton Raichuk, Piotr Stańczyk, Michał Zając, Olivier
Bachem, Lasse Espeholt, Carlos Riquelme, Damien Vincent, Marcin Michalski,
Olivier Bousquet, et al.
Google research football: A novel reinforcement learning environment.
In Proceedings of the AAAI conference on artificial
intelligence, volume 34, pages 4501–4510, 2020.

[3]
Hiske Overweg, Herman NC Berghuijs, and Ioannis N Athanasiadis.
Cropgym: a reinforcement learning environment for crop management.
arXiv preprint arXiv:2104.04326, 2021.

[4]
Huichu Zhang, Siyuan Feng, Chang Liu, Yaoyao Ding, Yichen Zhu, Zihan Zhou,
Weinan Zhang, Yong Yu, Haiming Jin, and Zhenhui Li.
Cityflow: A multi-agent reinforcement learning environment for large
scale city traffic scenario.
In The world wide web conference, pages 3620–3624, 2019.

[5]
David Rohde, Stephen Bonner, Travis Dunlop, Flavian Vasile, and Alexandros
Recogym: A reinforcement learning environment for the problem of
product recommendation in online advertising.
arXiv preprint arXiv:1808.00720, 2018.

[6]
Umberto J Ravaioli, James Cunningham, John McCarroll, Vardaan Gangal, Kyle
Dunlap, and Kerianne L Hobbs.
Safe reinforcement learning benchmark environments for aerospace
control systems.
In 2022 IEEE Aerospace Conference (AERO), pages 1–20. IEEE,

[7]
Shital Shah, Debadeepta Dey, Chris Lovett, and Ashish Kapoor.
Airsim: High-fidelity visual and physical simulation for autonomous
In Field and Service Robotics: Results of the 11th International
Conference, pages 621–635. Springer, 2018.

[8]
Nathan Koenig and Andrew Howard.
Design and use paradigms for gazebo, an open-source multi-robot
In 2004 IEEE/RSJ international conference on intelligent robots
and systems (IROS)(IEEE Cat. No. 04CH37566), volume 3, pages 2149–2154.
IEEE, 2004.

[9]
Jérémy Pinguet, Philippe Feyel, and Guillaume Sandou.
A neural autopilot training platform based on a matlab and x-plane
In 2021 International Conference on Unmanned Aircraft Systems
(ICUAS), pages 1200–1209. IEEE, 2021.

[10]
David J Richter and Ricardo A Calix.
Using double deep q-learning to learn attitude control of fixed-wing
In 2022 16th International Conference on Signal-Image Technology
& Internet-Based Systems (SITIS), pages 646–651. IEEE, 2022.

[11]
Jon Berndt.
Jsbsim: An open source flight dynamics model in c++.
In AIAA Modeling and Simulation Technologies Conference and
Exhibit, page 4923, 2004.

[12]
Adrian P Pope, Jaime S Ide, Daria Mićović, Henry Diaz, Jason C Twedt,
Kevin Alcedo, Thayne T Walker, David Rosenbluth, Lee Ritholtz, and Daniel
Hierarchical reinforcement learning for air combat at darpa's
alphadogfight trials.
IEEE Transactions on Artificial Intelligence, 2022.

[13]
Haiyin Piao, Zhixiao Sun, Guanglei Meng, Hechang Chen, Bohao Qu, Kuijun Lang,
Yang Sun, Shengqi Yang, and Xuanqi Peng.
Beyond-visual-range air combat tactics auto-generation by
reinforcement learning.
In 2020 international joint conference on neural networks
(IJCNN), pages 1–8. IEEE, 2020.

[14]
Zhen Yang, Deyun Zhou, Haiyin Piao, Kai Zhang, Weiren Kong, and Qian Pan.
Evasive maneuver strategy for ucav in beyond-visual-range air combat
based on hierarchical multi-objective evolutionary algorithm.
IEEE Access, 8:46605–46623, 2020.

[15]
Edvards Scukins, Markus Klein, and Lars Kroon.
<https://github.com/xcwoid/BVRGym>, 2024.

[16]
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman,
Jie Tang, and Wojciech Zaremba.
Openai gym.
arXiv preprint arXiv:1606.01540, 2016.

[17]
Oliver Biggar, Mohammad Zamani, and Iman Shames.
On modularity in reactive control architectures, with an application
to formal verification.
ACM Transactions on Cyber-Physical Systems (TCPS), 6(2):1–36,

[18]
Petter Ögren and Christopher I Sprague.
Behavior trees in robot control systems.
Annual Review of Control, Robotics, and Autonomous Systems,
5:81–107, 2022.

[19]
Matteo Iovino, Edvards Scukins, Jonathan Styrud, Petter Ögren, and
Christian Smith.
A survey of behavior trees in robotics and ai.
Robotics and Autonomous Systems, 154:104096, 2022.

[20]
Richard S Sutton and Andrew G Barto.
Reinforcement learning: An introduction.
MIT press, 2018.

[21]
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.
Proximal policy optimization algorithms.
preprint arXiv:1707.06347, 2017.

[22]
William Koch, Renato Mancuso, Richard West, and Azer Bestavros.
Reinforcement learning for uav attitude control.
ACM Transactions on Cyber-Physical Systems, 3(2):1–21, 2019.

[23]
Eivind Bøhn, Erlend M Coates, Signe Moe, and Tor Ame Johansen.
Deep reinforcement learning attitude control of fixed-wing uavs using
proximal policy optimization.
IEEE, 2019.

[24]
Michele Colledanchise and Petter Ögren.
Behavior trees in robotics and AI: An introduction.
CRC Press, 2018.

[25]
Rafael T Yanushevsky.
Modern missile guidance.
CRC Press, 2018.

[26]
Joao PA Dantas, Andre N Costa, Diego Geraldo, Marcos ROA Maximo, and Takashi
Engagement decision support for beyond visual range air combat.
In 2021 Latin American Robotics Symposium (LARS), 2021 Brazilian
Symposium on Robotics (SBR), and 2021 Workshop on Robotics in Education
(WRE), pages 96–101. IEEE, 2021.

[27]
Edvards Scukins, Markus Klein, and Petter Ögren.
Enhancing situation awareness in beyond visual range air combat with
reinforcement learning-based decision support.
In 2023 International Conference on Unmanned Aircraft Systems
(ICUAS), pages 56–62. IEEE, 2023.