# Confined and eruptive catastrophes of solar magnetic flux ropes caused by
mass loading and unloading

Quanhao Zhang11affiliation: CAS Key Laboratory of Geospace Environment,
Department of Geophysics and Planetary Sciences, University of Science and
Technology of China, Hefei 230026, China 22affiliation: CAS Center for
Excellence in Comparative Planetology, University of Science and Technology of
China, Hefei 230026, China 33affiliation: Mengcheng National Geophysical
Observatory, School of Earth and Space Sciences, University of Science and
Technology of China, Hefei 230026, China , Rui Liu11affiliation: CAS Key
Laboratory of Geospace Environment, Department of Geophysics and Planetary
Sciences, University of Science and Technology of China, Hefei 230026, China
22affiliation: CAS Center for Excellence in Comparative Planetology,
University of Science and Technology of China, Hefei 230026, China
44affiliation: Collaborative Innovation Center of Astronautical Science and
Technology, Hefei, Anhui 230026, China , Yuming Wang11affiliation: CAS Key
Laboratory of Geospace Environment, Department of Geophysics and Planetary
Sciences, University of Science and Technology of China, Hefei 230026, China
22affiliation: CAS Center for Excellence in Comparative Planetology,
University of Science and Technology of China, Hefei 230026, China
33affiliation: Mengcheng National Geophysical Observatory, School of Earth and
Space Sciences, University of Science and Technology of China, Hefei 230026,
China , Xiaolei Li11affiliation: CAS Key Laboratory of Geospace Environment,
Department of Geophysics and Planetary Sciences, University of Science and
Technology of China, Hefei 230026, China 33affiliation: Mengcheng National
Geophysical Observatory, School of Earth and Space Sciences, University of
Science and Technology of China, Hefei 230026, China , Shaoyu
Lyu11affiliation: CAS Key Laboratory of Geospace Environment, Department of
Geophysics and Planetary Sciences, University of Science and Technology of
China, Hefei 230026, China 33affiliation: Mengcheng National Geophysical
Observatory, School of Earth and Space Sciences, University of Science and
Technology of China, Hefei 230026, China<EMAIL_ADDRESS>
###### Abstract

It is widely accepted that coronal magnetic flux ropes are the core structures
of large-scale solar eruptive activities, which inflict dramatic impacts on
the solar-terrestrial system. Previous studies have demonstrated that varying
magnetic properties of a coronal flux rope system could result in a
catastrophe of the rope, which may trigger solar eruptive activities. Since
the total mass of a flux rope also plays an important role in stabilizing the
rope, we use 2.5-dimensional magnetohydrodynamic (MHD) numerical simulations
in this letter to investigate how a flux rope evolves as its total mass
varies. It is found that an unloading process that decreases the total mass of
the rope could result in an upward (eruptive) catastrophe in the flux rope
system, during which the rope jumps upward and the magnetic energy is
released. This indicates that mass unloading processes could initiate the
eruption of the flux rope. Moreover, when the system is not too diffusive,
there is also a downward (confined) catastrophe that could be caused by mass
loading processes, via which the total mass accumulates. The magnetic energy,
however, is increased during the downward catastrophe, indicating that mass
loading processes could cause confined activities that may contribute to the
storage of energy before the onset of coronal eruptions.

Solar activity—Solar flares—Solar prominences—Solar magnetic fields—Solar
coronal mass ejections—Solar filament eruptions

##  1 Introduction

Large-scale solar eruptions include filament/prominence eruptions, coronal
mass ejections (CMEs), and flares. They are intimately related to each other,
and are usually believed to be different manifestations of coronal magnetic
flux rope eruptions (Zhang et al., 2001; Vršnak et al., 2005; Jiang et al.,
2016; Yan et al., 2020; Liu, 2020). Since large-scale solar eruptions are the
major trigger of extreme space weather (Švestka, 2001; Cheng et al., 2014;
Lugaz et al., 2017; Gopalswamy et al., 2018), it is significant to investigate
the eruptive mechanism of coronal flux ropes. To shed light on the physical
processes of solar eruptive activities, many efforts have been made in
modeling the eruptions of coronal flux ropes in previous theoretical studies,
in which distinctive physical mechanisms are invoked, e.g., ideal
magnetohydrodynamic(MHD) instabilities (Török & Kliem, 2003; Aulanier et al.,
2010; Guo et al., 2010; Savcheva et al., 2012; Keppens et al., 2019), magnetic
reconnections (Antiochos et al., 1999; Chen & Shibata, 2000; Moore et al.,
2001; Sterling & Moore, 2004; Archontis & Hood, 2008; Inoue et al., 2015), and
flux rope catastrophes.

The flux rope catastrophe theory investigates the onset of flux rope eruptions
from the perspective of loss of equilibrium. Before the onset of the eruption,
the flux rope should be in equilibrium (static or quasi-static), i.e., with no
net force on the rope; after the onset of the eruption, however, the flux rope
evolves in a dynamic process, during which the state of motion must be very
different from that before the onset. The varying state of motion implies that
a loss of equilibrium occurs at the onset of eruption, so that the net force
resulting from the loss of equilibrium initiates the eruption of the rope.
This is the physical scenario of flux rope catastrophes, which was first
proposed by Van Tend & Kuperus (1978): they analytically derived the
equilibrium manifold as a function of the filament current, and found that the
catastrophe occurs if the current exceeds a critical value, which is called
the catastrophic point. Here the equilibrium manifold is the set of all the
equilibrium states of the flux rope system; based on the equilibrium manifold,
one is able to tell when and how the catastrophe occurs. Similar results were
obtained by many other analytical studies (Forbes & Isenberg, 1991; Lin &
Forbes, 2000; Démoulin & Aulanier, 2010; Longcope & Forbes, 2014; Kliem et
al., 2014), suggesting that the catastrophes of coronal flux ropes are
promising candidates for coronal eruptions.

Numerical simulations are also widely used to investigate flux rope
catastrophes (e.g., Forbes, 1990; Chen et al., 2007; Zhang et al., 2017). In
those studies, the authors calculated equilibrium solutions of the rope system
with different values of a characteristic physical parameter, and all the
equilibrium solutions constitute the equilibrium manifold versus this
parameter. For example, by simulating the equilibrium manifold as a function
of the magnetic field strength within the flux rope, Hu (2001) concluded that
a catastrophe occurs if the strength exceeds a critical value. Similar
conclusions are reached in many other studies (e.g., Su et al., 2011; Zhang et
al., 2017). In most of the previous studies, the flux rope jumps upward when
the catastrophe occurs, so that this kind of catastrophe is termed “upward
catastrophe”. Recently, Zhang et al. (2016) found its downward counterpart
caused by reducing magnetic fluxes of coronal flux ropes; during this
catastrophe, the flux rope suspended in the corona falls down to the
photosphere. This new kind of catastrophe is thus termed “downward
catastrophe”, and Zhang et al. (2016) also demonstrated that the magnetic
energy is released during not only the upward but also the downward
catastrophe. Observational studies suggested that, apart from coronal
eruptions, there are also various confined activities, with no portion of the
flux rope escaping the corona (e.g. Gilbert et al., 2000; Jiang et al., 2016).
Comparing with the observations, Zhang et al. (2016) inferred that the
downward catastrophe could be responsible for, e.g., confined flares.

Previous numerical studies mainly focus on flux rope catastrophes associated
with the magnetic parameters of the rope; whether the variation of the mass
could cause flux rope catastrophes, however, has hardly been touched upon
previously. In fact, as concluded by Low (1996), the total mass of a flux rope
should play an important role in stabilizing the rope, which is further
confirmed by some recent studies (Hillier & van Ballegooijen, 2013; Jenkins et
al., 2019; Tsap et al., 2019; Fan, 2020). Moreover, the total mass of a flux
rope is usually not conserved but highly dynamic (Zhang et al., 2017; Gibson,
2018): it either accumulates via, e.g., coronal condensation (Liu et al.,
2012; Xia & Keppens, 2016; Jenkins & Keppens, 2021), or decreases as a result
of, e.g., mass unloading (Low, 1996) and mass drainage processes (Bi et al.,
2014; Zhang et al., 2017; Jenkins et al., 2019). Therefore, to understand the
influence of the varying mass on the catastrophic behaviors of coronal flux
ropes is not only important to improve flux rope catastrophe theory, but also
shed light on the physical scenario of different kinds of flux rope
activities. To achieve this, we carry out numerical simulations to seek for
the flux rope catastrophes associated with the rope mass. For simplicity, we
use “mass loading” and “mass unloading” to refer to the processes that
increase and decrease the mass hereafter, respectively. The rest of this
letter is arranged as follows. The simulating procedures are introduced in
Sect. 2. The simulations are presented in Sect. 3. Our conclusion and
discussion are given in Sect. 4.

##  2 Simulating procedures

In our 2.5-dimensional simulations, all the quantities satisfy
$\partial/\partial z=0$. The magnetic field can be written as:

$\displaystyle\textbf{B}=\triangledown\psi\times\hat{\textbf{\emph{z}}}+B_{z}\hat{\textbf{\emph{z}}},$
(1)

Here $\psi$ is the magnetic flux function; the subscript $z$ denotes the
component in $z-$dirention. Basic equations and numerical procedures to
construct the initial state are introduced in Appendix A. The magnetic
configuration of the background field is plotted in 1(a): it is a partially
open bipolar field, with a negative and a positive surface magnetic charges
located at the lower base within $-30~{}\mathrm{Mm}<x<-10~{}\mathrm{Mm}$ and
$10~{}\mathrm{Mm}<x<30~{}\mathrm{Mm}$, respectively. The initial state,
illustrated in 1(b), is an equilibrium state consisting of a flux rope
embedded in the background field. Anomalous resistivity is used:

$\displaystyle\eta=\begin{cases}0,&~{}j\leq j_{c}\\\
\eta_{m}\mu_{0}v_{0}L_{0}(\frac{j}{j_{c}}-1)^{2}.&~{}j>j_{c}\end{cases}$ (2)

Here $\eta_{m}=10^{-4}$, $L_{0}=10^{7}$ m, $v_{0}=128.57$ km s-1, and
$j_{c}=2.37\times 10^{-4}$ A m-2; $\mu_{0}$ is the vacuum magnetic
permeability. The physical properties of a flux rope can be characterized by
its axial magnetic flux, $\Phi_{z}$, its poloidal magnetic flux per unit
length along the $z$-direction, $\Phi_{p}$, and its total mass per unit length
along the $z$-direction, $M$. For the flux rope in the initial state, the
values of the these properties are: $\Phi_{z0}=4.10\times
10^{19}~{}\mathrm{Mx}$, $\Phi_{p0}=1.49\times
10^{12}~{}\mathrm{Mx}~{}\mathrm{m}^{-1}$, and $M_{0}=5.01\times 10^{3}$ kg
m-1. Assuming the length of the flux rope to be 100 Mm, its total mass is of
the order $M_{0}=5.0\times 10^{11}$ kg, which falls within the observed mass
range of typical prominences/filaments (Parenti, 2014).

Figure 1: Simulations of the catastrophes caused by the varing mass of the
rope. Panel (a) is the magnetic configuration of the background field, and
panel (b) that of the initial state; panels (c) and (d) are the equilibrium
states right before and after the upward catastrophe, and panels (e) and (f)
are those for the downward catastrophe. In panels (a)-(f), the blue lines
represent the magnetic field lines; the green curve mark the boundary of the
flux rope; the red color depicts the distribution of the mass density. The
dots in panel (g) plot the variation of the height of the rope axis for the
simulated equilibrium states as a function of the rope mass, and the
corresponding magnetic energies of the equilibrium states are plotted by the
dots in panel (h). The upward (eruptive) and the downward (confined)
catastrophes are marked by the red and blue arrows, respectively.

To investigate the evolution of the rope versus its total mass, we simulate
the equilibrium manifold as a function of the mass. Starting from the initial
state, we first adjust the mass density at each grid within the rope by

$\displaystyle\rho=\alpha\rho_{0},~{}\alpha=\frac{M_{1}}{M_{0}},$ (3)

where $\rho_{0}$ is the density in the initial state. With this procedure, the
total mass of the rope is adjusted from $M_{0}$ to a certain target value
$M_{1}$. Second, we let the rope system relax to an equilibrium state, during
which the total mass of the rope is maintained to be conserved at $M_{1}$ by
similar procedures to those in Hu et al. (2003). Moreover, relaxation method
(e.g., Zhang et al., 2016) is used in the region outside the flux rope: reset
the temperature and density to their initial values, so that the pressure
gradient force is always balanced by the gravitational force; when the rope
system eventually evolves to the equilibrium state, force-free condition is
satisfied in the region outside the flux rope. The region inside the flux rope
is still non-force-free. With the simulating procedures introduced above, we
obtain the equilibrium state with the rope mass to be $M_{1}$. Similar
procedures are repeated for different target values of the mass (e.g,.
$M_{2}$, $M_{3}$, …), and all the calculated equilibrium states constitute the
equilibrium manifold. During the whole simulation, $\Phi_{z}$ and $\Phi_{p}$
are always maintained to be conserved at $\Phi_{z0}$ and $\Phi_{p0}$ by
similar procedures to those in Hu et al. (2003), i.e., the characteristic
magnetic parameters of all the simulated equilibrium states are the same, so
that we could focus on the influence of the mass on the rope system.

In our simulation, the quantities at the lower boundary are fixed, indicating
that it corresponds to the photosphere; increment equivalent extrapolation is
used at the other boundaries (e.g., Zhang et al., 2020):

$\displaystyle U^{n+1}_{b}=U^{n+1}_{b-1}+U^{n}_{b}-U^{n}_{b-1}.$

Here $U$ represents the quantities (e.g. $\rho$, v, $\psi$); the quantities at
the current and the next time steps are indicated by the superscript n and
n+1, respectively; $U_{b}$ represents the quantities at the boundary, and
$U_{b-1}$ the quantities at the location next to the boundary. The boundary
quantities at the next time step, $U^{n+1}_{b}$, are then evaluated.

##  3 Simulation results

The simulated equilibrium manifold of the flux rope system is plotted in 1(g).
The evolution of the equilibrium states versus the mass is described by the
variation of the height of the rope axis, $H$. The black dot in 1(g)
corresponds to the initial state 1(b), starting from which we change the mass
of the rope and calculate the corresponding equilibrium states, as plotted by
the red dots. There is a critical value of the mass, $M^{u}=2.96\times 10^{3}$
kg m-1: the flux rope keeps sticking to the photosphere if its total mass $M$
is larger than $M^{u}$, as shown in, e.g., 1(c); if $M$ decreases to reach
$M^{u}$, however, the flux rope jumps upward, and evolves to an equilibrium
state with the flux rope suspended in the corona, as shown in 1(d).
Consequently, the equilibrium manifold is discontinuous at $M^{u}$ (marked by
the vertical red line in 1(g)), i.e., an upward catastrophe occurs at $M^{u}$
if $M$ decreases to reach $M^{u}$, as marked by the red arrow in 1(g); $M^{u}$
is the upward catastrophic point. 1(c) and 1(d) illustrate the equilibrium
states right before and after the upward catastrophe occurs, respectively;
they are also marked by “c” and “d” in 1(g). We note that the resistivity in
our simulation is not very large ($\eta_{m}=10^{-4}$), so that the rope does
not escape from the corona after the upward catastrophe occurs, but keeps
suspended in the corona (1(d)). If the system is strongly diffusive (e.g.,
$\eta_{m}=0.1$), the rope could keep rising after the upward catastrophe
occurs, resulting in a full eruption. In addition, it could be recognized in
1(g) that $H$ slowly increases with the decreasing $M$ before the upward
catastrophe occurs, which may correspond to the slow-rising phase before the
onset of the eruption. Our simulation result of the upward catastrophe caused
by the decreasing mass demonstrates that mass unloading processes could
initiate flux rope eruptions. This result is consistent with the analytical
analysis in Jenkins et al. (2019). Moreover, by using the analytical model in
Jenkins et al. (2019), we could estimate the critical height at which loss of
equilibrium should occur in our model. The input parameters include: the mass
$m=M^{u}=2.96\times 10^{3}$ kg m-1, half of the distance between the two
photospheric polarities $D=10$ Mm, the magnetic field strength at the
photosphere $B_{Phot}=B(x=0,y=0)=5.97$ G. The critical height is then
calculated as 17.32 Mm. In our simulations, $H$ of the state right before the
upward catastrophe occurs (1(c)) is 23.84 Mm. The gap between the critical
heights should result from the difference in the models: the flux rope in our
simulation has a finite radius, whereas a thin-rope model is used in Jenkins
et al. (2019).

Restarting from the equilibrium state with the rope suspended in the corona
(1(d)), we increase the total mass of the rope and simulate the corresponding
equilibrium states, which are plotted by the blue dots in 1(g). Obviously, the
flux rope does not immediately evolve back to the state marked as “c” when we
start to decrease the mass of the rope, but keeps suspended in the corona, as
shown in, e.g., 1(e). This indicates that the upward catastrophe is an
irreversible process. As demonstrated by the blue dots in 1(g), there is
another critical mass $M^{d}=6.81\times 10^{3}$ kg m-1, at which the flux rope
falls down until reaching an equilibrium state with the rope once again
sticking to the photosphere (1(f)); the eventual magnetic configuration is
similar to the initial state. Therefore, a downward catastrophe occurs if the
mass of the rope increases to reach $M^{d}$. The equilibrium states right
before and after the downward catastrophe, which are illustrated in 1(e) and
1(f), are also marked by “e” and “f” in 1(g), and the downward catastrophe is
marked by the blue arrow. We have also tried to decrease the mass from the
equilibrium state after the downward catastrophe occurs (1(f)), and verified
that the downward catastrophe is also an irreversible process, similar to the
upward catastrophe; if we decrease the mass to the upward catastrophic point,
$M^{u}$, the upward catastrophe occurs again. Our results suggest that mass
loading processes could cause confined activities in flux rope systems before
the onset of the eruption. From 1(g), we may conclude that the equilibrium
states are bifurcated into two branches: the states with the rope sticking to
the photosphere belong to the lower branch, and those with the rope suspended
in the corona belong to the upper branch. These two branches are separated
from each other, and are only connected by the upward and the downward
catastrophes. Obviously, the upward and the downward catastrophe should occur
in flux rope systems with different topologies: the equilibrium state right
before the upward catastrophe occurs (1(c)) is in the lower branch, belonging
to the bald patch separatrix surface (BPSS) topology (Titov et al., 1993),
whereas that for the downward catastrophe (1(e)) is in the upper branch, which
should correspond to the hyperbolic flux tube (HFT) topology (Titov et al.,
2003). As shown in 1(g), the rope will keep rising (descending) if its total
mass decreases (increases) with time until reaching the catastrophic points in
both the upper and the lower branches. Since faster variation of the mass
corresponds to faster variation of the rope height, higher mass loss (gain)
rate should result in larger rising (descending) velocity of the rope. It is
noteworthy that the downward catastrophe could only exist when the magnetic
system is not too diffusive; otherwise, as explicated above, the flux rope
will escape the corona after the upward catastrophe occurs, so that the upper
branch as well as the downward catastrophe vanishes.

To roughly estimate the variation of the magnetic energy versus the total mass
of the rope, we calculate the magnetic energy within the domain for each
simulated equilibrium state, which are plotted in 1(h). Although the axial and
poloidal magnetic fluxes of the flux ropes in all the simulated equilibrium
states are the same (Sect. 2), the corresponding magnetic energies are quite
different. The equilibrium states belonging to the lower branch tend to have
larger magnetic energies than those belonging to the upper branch; the two
branches are also well separated in the parameter space spanned by the
magnetic energy and the mass. This should be caused by the different
topologies of the states in different branches. As a result, magnetic energy
is released during the upward catastrophe, via which the flux rope system
evolves from the high-energy state (lower branch) to the low-energy state
(upper branch); in contrast, magnetic energy is stored during the downward
catastrophe. The amount of the magnetic energy released during the upward
catastrophe is of the order $E_{m}^{u}=13.64\times 10^{10}$ J m-1, and that
stored during the downward catastrophe is of the order $E_{m}^{d}=5.72\times
10^{10}$ J m-1. Moreover, to further investigate the energy budget during the
catastrophes, we also calculate the variations of the gravitational energy
during the catastrophes: it increases by $E_{g}^{u}=6.89\times 10^{10}$ J m-1
during the upward catastrophe, and decreases by $E_{g}^{u}=6.23\times 10^{10}$
J m-1 during the downward catastrophe. Comparing with $E_{m}^{u}$ and
$E_{m}^{d}$, we may infer that: (1) during the upward catastrophe, part (about
50$\%$) of the released magnetic energy is converted to the gravitational
energy; (2) the stored magnetic energy during the downward catastrophe might
originate from the gravitational energy in the flux rope system. Moreover,
comparing 1(g)) and 1(h), we may infer that the increase of the magnetic
energy as a result of the increasing mass in the upper branch should counter
the according decrease of the rope height. This result might presumably be
useful for the analysis of, e.g., quiescent prominences.

##  4 Discussion and conclusion

In this letter, we investigate how the varying mass of coronal flux ropes
affects their catastrophic evolutions. Our simulation results discover both
the upward (eruptive) and the downward (confined) catastrophes associated with
the varying mass of the rope. For the flux rope system with the rope sticking
to the photosphere, an upward catastrophe will occur if the mass of the rope
decreases to reach a critical value. During the upward catastrophe, the flux
rope jumps upward, and the magnetic energy is released. Therefore, the upward
catastrophe should initiate the eruption of the flux rope. This result
provides a promising theoretical scenario for triggering the eruption of
coronal flux ropes in the BPSS configuration caused by mass unloading
processes. Moreover, if the total mass of a flux rope suspended in the corona
increases to reach another critical value, a downward catastrophe will occur,
during which the flux rope falls down to the photosphere, and the magnetic
energy is, however, stored in the rope system. This indicates that mass
loading processes could cause confined activities of coronal flux ropes in the
HFT configuration, which may contribute to the storage of energy before the
onset of solar eruptions. These two catastrophes appear as discontinuous jumps
in the equilibrium manifold (as marked by the arrows in 1(g)); they connect
the separated upper and lower branches of the equilibrium states of the rope
system, and both of the catastrophes are irreversible processes. We note once
more that the resistivity in our simulation is not very large; if the magnetic
system is too diffusive, the upper branch as well as the downward catastrophe
vanishes, indicating that the downward catastrophe could only exist in, e.g.,
quiescent regions.

As simulated in Zhang et al. (2016), increasing and decreasing magnetic fluxes
are also able to cause upward and downward catastrophes, respectively; the
magnetic energy, however, is released during both the upward and the downward
catastrophes caused by varying magnetic fluxes. Comparing with our simulation
results, we may conclude that increasing magnetic fluxes of a coronal flux
rope should play a similar role as the decreasing mass of the rope: they are
in favor of coronal eruptions. Nevertheless, although both decreasing magnetic
fluxes and increasing mass could cause confined activities, the corresponding
energetic evolutions are quite different: the downward catastrophe caused by
decreasing magnetic fluxes provides a possible scenario for confined flares,
whereas that caused by the increasing mass corresponds to the storage process
of the magnetic energy. This implies that downward catastrophes, regulated by
different physical parameters, may contribute to various confined activities
observed in the solar corona. Efforts are still needed to seek for
observational counterparts of these phenomena.

In this letter, our analysis is mainly based on the parameter space of the
mass, rather than the temporal evolution of the rope. Since there are various
kinds of non-linear processes that could change the mass of the rope, the
temporal evolution of the rope associated with these processes should reveal
the evolutionary scenario of coronal flux ropes in the actual solar
atmosphere. We will extend our model and investigate this topic in our future
studies.

The authors appreciate the anonymous referee for the comments and suggestions
that helped to improve this work. This research is supported by the National
Natural Science Foundation of China (NSFC 41804161, 42174213, 41774178,
41761134088, 41774150, 41842037 and 41574165), the Strategic Priority Program
of CAS (XDB41000000 and XDA15017300), and the fundamental research funds for
the central universities. Quanhao Zhang acknowledges for the support from
National Space Science Data Center, National Science `&` Technology
Infrastructure of China (www.nssdc.ac.cn).

##  Appendix A Basic equations and initial preparations

With the form in Equation (1), the MHD equations could then be cast in the
non-dimensional form as:

$\displaystyle\frac{\partial\rho}{\partial
t}+\triangledown\cdot(\rho\textbf{\emph{v}})=0,$ (A1)
$\displaystyle\frac{\partial\textbf{\emph{v}}}{\partial
t}+\frac{2}{\rho\beta_{0}}(\vartriangle\psi\triangledown\psi+B_{z}\triangledown
B_{z}+\triangledown\psi\times\triangledown
B_{z})+\textbf{\emph{v}}\cdot\triangledown\textbf{\emph{v}}$
$\displaystyle~{}~{}~{}+\triangledown
T+\frac{T}{\rho}\triangledown\rho+g\hat{\textbf{\emph{y}}}=0,$ (A2)
$\displaystyle\frac{\partial\psi}{\partial
t}+\textbf{\emph{v}}\cdot\triangledown\psi-\eta\vartriangle\psi=0,$ (A3)
$\displaystyle\frac{\partial B_{z}}{\partial
t}+\triangledown\cdot(B_{z}\textbf{\emph{v}})+(\triangledown\psi\times\triangledown
v_{z})\cdot\hat{\textbf{\emph{z}}}-\eta\vartriangle B_{z}=0,$ (A4)
$\displaystyle\frac{\partial T}{\partial t}-\frac{\eta(\gamma-1)}{\rho
R}\left[(\vartriangle\psi)^{2}+|\triangledown\times(B_{z}\hat{\textbf{\emph{z}}})|^{2}\right]$
$\displaystyle~{}~{}~{}+\textbf{\emph{v}}\cdot\triangledown
T+(\gamma-1)T\triangledown\cdot\textbf{\emph{v}}=0,$ (A5)

where

$\displaystyle\vartriangle\psi=\frac{\partial^{2}\psi}{\partial
x^{2}}+\frac{\partial^{2}\psi}{\partial y^{2}},~{}~{}\vartriangle
B_{z}=\frac{\partial^{2}B_{z}}{\partial
x^{2}}+\frac{\partial^{2}B_{z}}{\partial y^{2}}.$ (A6)

Here $\rho$ and $T$ are the density and the temperature;
$\textit{{v}}=v_{x}\hat{\textbf{\emph{x}}}+v_{y}\hat{\textbf{\emph{y}}}+v_{z}\hat{\textbf{\emph{z}}}$
is the velocity; $g$ is the normalized gravity, which is a constant; $\eta$ is
the resistivity; $\gamma=5/3$ is the polytropic index; $\beta_{0}=0.1$ is the
characteristic ratio of the gas pressure to the magnetic pressure. The
characteristic values of the quantities are: $\rho_{0}=3.34\times
10^{-13}\mathrm{~{}kg~{}m^{-3}}$, $T_{0}=10^{6}\mathrm{~{}K}$,
$\psi_{0}=3.73\times 10^{3}\mathrm{~{}Wb~{}m^{-1}}$, $v_{0}=128.57$ km s-1,
$t_{0}=77.8$ s, $B_{0}=3.37\times 10^{-4}$ T, $g_{0}=1.65\times 10^{3}$ m s-2.
The numerical domain is $0<x<300$ Mm, $0<y<300$ Mm; it is discretized into
300$\times$150 uniform meshes. Symmetric boundary condition is used at the
left side of the domain ($x=0$). The radiation and the heat conduction in the
energy equation are neglected.

To investigate the catastrophic behaviors of coronal flux rope systems versus
the mass of the rope, we must first construct a typical flux rope system. Here
we first use the complex variable method (e.g., Zhang et al., 2020) to
construct the partially open bipolar background field, which can be cast in
the complex variable form

$\displaystyle f(\omega)\equiv
B_{x}-iB_{y}=\frac{(\omega+iy_{N})^{1/2}(\omega-
iy_{N})^{1/2}}{F(a,b,y_{N})}\mathrm{ln}\left(\frac{\omega^{2}-a^{2}}{\omega^{2}-b^{2}}\right),$
(A7)

where $\omega=x+iy$, and

$\displaystyle
F(a,b,y_{N})=\frac{1}{b-a}\int_{a}^{b}(x^{2}+y_{N}^{2})^{1/2}dx=\frac{1}{2(b-a)}\times$
$\displaystyle\left[b(b^{2}+y_{N}^{2})^{1/2}-a(a^{2}+y_{N}^{2})^{1/2}+y_{N}^{2}\mathrm{ln}\left(\frac{b+(b^{2}+y_{N}^{2})^{1/2}}{a+(a^{2}+y_{N}^{2})^{1/2}}\right)\right],$
(A8)

with $a=10$ Mm, $b=30$ Mm, and ($y=y_{N}=26.3$ Mm, $x=0$) is the position of
the neutral point in the background field; a neutral current sheet is located
at $(x=0,~{}y\geq y_{N})$. The magnetic flux function $\psi$ could then be
obtained by:

$\displaystyle\psi(x,y)=\mathrm{Im}\left\\{\int f(\omega)d\omega\right\\}.$
(A9)

The flux function at the lower base ($y=0$), $\psi_{i}$, is

$\psi_{i}=\psi(x,0)=\left\\{\begin{array}[]{ll}{\psi_{c}},&{|x|<a}\\\
{\psi_{c}F(|x|,b,y_{N})/F(a,b,y_{N})},&{a\leqslant|x|\leqslant b}\\\
{0},&{|x|>b}\end{array}\right.$ (A10)

where $\psi_{c}=2\pi\psi_{0}$, and that at the neutral point $y=y_{N}$ is:

$\displaystyle\psi_{N}=\frac{\pi(b^{2}-a^{2})}{2F(a,b,y_{N})}.$ (A11)

With the obtained $\psi$, and letting $B_{z}=0$ in the background field, the
distribution of the magnetic field within the domain could be calculated. The
initial corona is isothermal and static:

$\displaystyle T_{c}\equiv T(t=0,x,y)=1\times 10^{6}~{}\mathrm{K},\ \
\rho_{c}\equiv\rho(t=0,x,y)=\rho_{0}\mathrm{e}^{-gy}.$ (A12)

Starting from the initial condition and the background configuration,
equations (A1) to (A5) are simulated by the multi-step implicit scheme (Hu,
1989). With similar simulating procedures to those in Zhang et al. (2020), we
let a flux rope emerge from the lower base at a constant speed. The emergence
begins at $t=0$ and ends at $t=\tau_{E}=174$ s, so that at time $t$
($0\leqslant t\leqslant\tau_{e}$), the emerged part of the rope is within
$-x_{E}\leqslant x\leqslant x_{E}$ at the lower base, where
$x_{E}=(a^{2}-h_{E}^{2})^{1/2}$, $h_{E}=a(2t/\tau_{E}-1)$. Thus the emergence
is achieved by adjusting the quantities at the lower base ($y=0$) within
$-x_{E}\leqslant x\leqslant x_{E}$

$\displaystyle\psi(t,x,y=0)=\psi_{i}(x,y=0)+\psi_{E}(t,x),$ (A13)
$\displaystyle\psi_{E}(t,x)=\frac{C_{E}}{2}\mathrm{ln}\left(\frac{2a^{2}}{a^{2}+x^{2}+h_{E}^{2}}\right),$
(A14) $\displaystyle B_{z}(t,x,y=0)=C_{E}a(a^{2}+x^{2}+h_{E}^{2})^{-1},$ (A15)
$\displaystyle v_{y}(t,x,y=0)=2a/\tau_{E},~{}v_{x}(t,x,y=0)=v_{z}(t,x,y=0)=0,$
(A16) $\displaystyle T(t,x,y=0)=2\times
10^{5}\mathrm{~{}K},~{}\rho(t,x,y=0)=1.67\times
10^{-12}\mathrm{~{}kg~{}m^{-3}},$ (A17)

where $C_{E}$=3.0. With these procedures, we obtain a flux rope system
consisting of a magnetic flux rope embedded in the background field, as shown
in 1(b). We note that the emergence of the rope is merely a numerical approach
to construct a coronal flux rope system, so that the detailed evolution during
the emergence should not be related to the physics discussed in this paper.

## References

  * Antiochos et al. (1999) Antiochos, S. K., DeVore, C. R., & Klimchuk, J. A. 1999, ApJ, 510, 485
  * Archontis & Hood (2008) Archontis, V., & Hood, A. W. 2008, ApJ, 674, L113
  * Aulanier et al. (2010) Aulanier, G., Török, T., Démoulin, P., & DeLuca, E. E. 2010, ApJ, 708, 314
  * Bi et al. (2014) Bi, Y., Jiang, Y., Yang, J., et al. 2014, ApJ, 790, 100
  * Chen & Shibata (2000) Chen, P. F., & Shibata, K. 2000, ApJ, 545, 524
  * Chen et al. (2007) Chen, Y., Hu, Y. Q., & Sun, S. J. 2007, ApJ, 665, 1421
  * Cheng et al. (2014) Cheng, X., Ding, M. D., Zhang, J., et al. 2014, ApJ, 789, 93
  * Démoulin & Aulanier (2010) Démoulin, P., & Aulanier, G. 2010, ApJ, 718, 1388
  * Fan (2020) Fan, Y. 2020, ApJ, 898, 34
  * Forbes (1990) Forbes, T. G. 1990, J. Geophys. Res., 95, 11919
  * Forbes & Isenberg (1991) Forbes, T. G., & Isenberg, P. A. 1991, ApJ, 373, 294
  * Gibson (2018) Gibson, S. E. 2018, Living Reviews in Solar Physics, 15, 7
  * Gilbert et al. (2000) Gilbert, H. R., Holzer, T. E., Burkepile, J. T., & Hundhausen, A. J. 2000, ApJ, 537, 503
  * Gopalswamy et al. (2018) Gopalswamy, N., Akiyama, S., Yashiro, S., & Xie, H. 2018, Journal of Atmospheric and Solar-Terrestrial Physics, 180, 35
  * Guo et al. (2010) Guo, Y., Ding, M. D., Schmieder, B., et al. 2010, ApJ, 725, L38
  * Hillier & van Ballegooijen (2013) Hillier, A., & van Ballegooijen, A. 2013, ApJ, 766, 126
  * Hu (1989) Hu, Y. Q. 1989, Journal of Computational Physics, 84, 441
  * Hu (2001) Hu, Y. Q. 2001, Solar Physics, 200, 115
  * Hu et al. (2003) Hu, Y. Q., Li, G. Q., & Xing, X. Y. 2003, Journal of Geophysical Research (Space Physics), 108, 1072
  * Inoue et al. (2015) Inoue, S., Hayashi, K., Magara, T., Choe, G. S., & Park, Y. D. 2015, ApJ, 803, 73
  * Jenkins et al. (2019) Jenkins, J. M., Hopwood, M., Démoulin, P., et al. 2019, ApJ, 873, 49
  * Jenkins & Keppens (2021) Jenkins, J. M., & Keppens, R. 2021, A&A, 646, A134
  * Jiang et al. (2016) Jiang, C., Wu, S. T., Yurchyshyn, V., et al. 2016, ApJ, 828, 62
  * Keppens et al. (2019) Keppens, R., Guo, Y., Makwana, K., et al. 2019, Reviews of Modern Plasma Physics, 3, 14
  * Kliem et al. (2014) Kliem, B., Lin, J., Forbes, T. G., Priest, E. R., & Török, T. 2014, ApJ, 789, 46
  * Lin & Forbes (2000) Lin, J., & Forbes, T. G. 2000, J. Geophys. Res., 105, 2375
  * Liu (2020) Liu, R. 2020, Research in Astronomy and Astrophysics, 20, 165
  * Liu et al. (2012) Liu, W., Berger, T. E., & Low, B. C. 2012, ApJ, 745, L21
  * Longcope & Forbes (2014) Longcope, D. W., & Forbes, T. G. 2014, Sol. Phys., 289, 2091
  * Low (1996) Low, B. C. 1996, Sol. Phys., 167, 217
  * Lugaz et al. (2017) Lugaz, N., Farrugia, C. J., Winslow, R. M., et al. 2017, ApJ, 848, 75
  * Moore et al. (2001) Moore, R. L., Sterling, A. C., Hudson, H. S., & Lemen, J. R. 2001, ApJ, 552, 833
  * Parenti (2014) Parenti, S. 2014, Living Reviews in Solar Physics, 11, 1
  * Savcheva et al. (2012) Savcheva, A. S., van Ballegooijen, A. A., & DeLuca, E. E. 2012, ApJ, 744, 78
  * Sterling & Moore (2004) Sterling, A. C., & Moore, R. L. 2004, ApJ, 602, 1024
  * Su et al. (2011) Su, Y., Surges, V., van Ballegooijen, A., DeLuca, E., & Golub, L. 2011, ApJ, 734, 53
  * Titov et al. (2003) Titov, V. S., Galsgaard, K., & Neukirch, T. 2003, ApJ, 582, 1172
  * Titov et al. (1993) Titov, V. S., Priest, E. R., & Demoulin, P. 1993, A&A, 276, 564
  * Török & Kliem (2003) Török, T., & Kliem, B. 2003, A&A, 406, 1043
  * Tsap et al. (2019) Tsap, Y. T., Filippov, B. P., & Kopylova, Y. G. 2019, Sol. Phys., 294, 35
  * Švestka (2001) Švestka, Z. 2001, Space Sci. Rev., 95, 135
  * Van Tend & Kuperus (1978) Van Tend, W., & Kuperus, M. 1978, Sol. Phys., 59, 115
  * Vršnak et al. (2005) Vršnak, B., Sudar, D., & Ruždjak, D. 2005, A&A, 435, 1149
  * Xia & Keppens (2016) Xia, C., & Keppens, R. 2016, ApJ, 823, 22
  * Yan et al. (2020) Yan, X., Xue, Z., Cheng, X., et al. 2020, ApJ, 889, 106
  * Zhang et al. (2001) Zhang, J., Dere, K. P., Howard, R. A., Kundu, M. R., & White, S. M. 2001, ApJ, 559, 452
  * Zhang et al. (2016) Zhang, Q., Wang, Y., Hu, Y., & Liu, R. 2016, ApJ, 825, 109
  * Zhang et al. (2017) Zhang, Q., Wang, Y., Hu, Y., Liu, R., & Liu, J. 2017, ApJ, 835, 211
  * Zhang et al. (2020) Zhang, Q., Wang, Y., Liu, R., et al. 2020, ApJ, 898, L12
  * Zhang et al. (2017) Zhang, Q. M., Li, T., Zheng, R. S., Su, Y. N., & Ji, H. S. 2017, ApJ, 842, 27


exampleloeList of Examples

# Bayesian Optimization with LLM-Based Acquisition Functions for Natural
Language Preference Elicitation

David Eric Austin<EMAIL_ADDRESS>University of
WaterlooWaterlooOntarioCanada ,  Anton Korikov<EMAIL_ADDRESS>,  Armin Toroghi  and  Scott Sanner  University of TorontoTorontoOntarioCanada

(2018)

###### Abstract.

Designing preference elicitation (PE) methodologies that can quickly ascertain
a user’s top item preferences in a cold-start setting is a key challenge for
building effective and personalized conversational recommendation (ConvRec)
systems. While large language models (LLMs) constitute a novel technology that
enables fully natural language (NL) PE dialogues, we hypothesize that
monolithic LLM NL-PE approaches lack the multi-turn, decision-theoretic
reasoning required to effectively balance the NL exploration and exploitation
of user preferences towards an arbitrary item set. In contrast, traditional
Bayesian optimization PE methods define theoretically optimal PE strategies,
but fail to use NL item descriptions or generate NL queries, unrealistically
assuming users can express preferences with direct item ratings and
comparisons. To overcome the limitations of both approaches, we formulate NL-
PE in a Bayesian Optimization (BO) framework that seeks to actively elicit
natural language feedback to reduce uncertainty over item utilities to
identify the best recommendation. Key challenges in generalizing BO to deal
with natural language feedback include determining (a) how to leverage LLMs to
model the likelihood of NL preference feedback as a function of item utilities
and (b) how to design an acquisition function that works for language-based BO
that can elicit in the infinite space of language. We demonstrate our
framework in a novel NL-PE algorithm, PEBOL, which uses Natural Language
Inference (NLI) between user preference utterances and NL item descriptions to
maintain preference beliefs and BO strategies such as Thompson Sampling (TS)
and Upper Confidence Bound (UCB) to guide LLM query generation. We numerically
evaluate our methods in controlled experiments, finding that PEBOL achieves up
to 131% improvement in MAP@10 after 10 turns of cold start NL-PE dialogue
compared to monolithic GPT-3.5, despite relying on a much smaller 400M
parameter NLI model for preference inference.

Conversational Recommendation, Preference Elicitation, Bayesian Optimization,
Online Recommendation, Query Generation

††copyright: acmlicensed††journalyear: 2018††doi: XXXXXXX.XXXXXXX††conference:
Make sure to enter the correct conference title from your rights confirmation
emai; June 03–05, 2018; Woodstock, NY††isbn: 978-1-4503-XXXX-X/18/06††ccs:
Information systems Recommender systems††ccs: Information systems
Personalization††ccs: Information systems Language models

##  1\. Introduction

Personalized conversational recommendation (ConvRec) systems require effective
natural language (NL) preference elicitation (PE) strategies that can
efficiently learn a user’s top item preferences in cold start settings,
ideally requiring only an arbitrary set of NL item descriptions. While the
advent of large language models (LLMs) has introduced the technology to
facilitate NL-PE conversations (Handa et al., 2024; Li et al., 2023) we
conjecture that monolithic LLMs have limited abilities to strategically
conduct active, multi-turn NL-PE dialogues about a set of arbitrary items.
Specifically, we hypothesize that LLMs lack the sequential decision-theoretic
reasoning to interactively generate queries that avoid over-exploitation or
over-exploration of user-item preferences, thus risking over-focusing on
already revealed item preferences or wastefully exploring preferences over
low-value items. Further challenges faced by monolithic LLM NL-PE approaches
include the need to jointly reason over large, potentially unseen sets of item
descriptions, and the lack of control and interpretability in system behaviour
even after prompt engineering or fine-tuning (Maynez et al., 2020).

Figure 1. PEBOL’s belief updates over a cold-start user’s item utilities
during three turns of NL dialogue. Bayesian preference beliefs not only
facilitate recommendation, but also enable Bayesian optimization policies to
guide LLM query generation, avoiding over-exploration (asking about clearly
low-value items) and over-exploitation (over-focusing on known preferences).

In contrast, conventional PE algorithms (Li et al., 2021; Zhang et al., 2020;
Lei et al., 2020; Lee et al., 2019; Zhao et al., 2013), including Bayesian
optimization methods (Guo and Sanner, 2010; Yang et al., 2021; Christakopoulou
et al., 2016; Vendrov et al., 2019; Boutilier, 2002), establish formal
decision-theoretic policies such as Thompson Sampling (TS) and Upper
Confidence Bound (UCB) (Karamanolakis et al., 2018) to balance exploration and
exploitation with the goal of quickly identifying the user’s most preferred
items. However, these techniques typically assume a user can express
preferences via direct item ratings or comparisons – an unrealistic
expectation when users are unfamiliar with most items (Biyik et al., 2023).
While recent work has extended Bayesian PE to a fixed set of template-based
queries over pre-defined keyphrases (Yang et al., 2021), no existing work
extends Bayesian methodologies to generative NL-PE over a set of generic NL
item descriptions.

In this paper, we make the following contributions:

  * •

We introduce the first Bayesian optimization formalization of NL-PE for
arbitrary NL dialogue over a generic set of NL item descriptions –
establishing a framework for research on augmenting LLMs with decision-
theoretic reasoning.

  * •

We present PEBOL (Preference Elicitation with Bayesian Optimization augmented
LLMs), a novel NL-PE algorithm which 1) infers item preferences via Natural
Language Inference (NLI) (Yin et al., 2019) between dialogue utterances and
item descriptions to maintain Bayesian preference beliefs and 2) introduces
LLM-based acquisition functions, where NL query generation is guided by
decision-theoretic strategies such as TS and UCB over the preference beliefs.

  * •

We numerically evaluate PEBOL against monolithic GPT-3.5 NL-PE methods via
controlled NL-PE dialogue experiments over multiple NL item datasets and
levels of user noise.

  * •

We observe that PEBOL achieves up to 131% improvement in MAP@10 after 10 turns
compared to GPT-3.5, despite using a much smaller 400M NLI model for
preference inference.

##  2\. Background and Related Work

###  2.1. Bayesian Optimization

Given an objective function $f:\mathcal{X}\rightarrow\mathbb{R}$, (standard)
optimization systematically searches for a point $x^{*}\in\mathcal{X}$ that
maximizes111We take the maximization direction since this paper searches for
items with maximum utility for a person. $f$. Bayesian optimization focuses on
settings where $f$ is a black-box function which does not provide gradient
information and cannot be evaluated exactly – rather, $f$ must be evaluated
using indirect or noisy observations which are expensive to obtain (Garnett,
2023; Shahriari et al., 2016). To address these challenges, Bayesian
optimization maintains probabilistic beliefs over $f(x)$ and its observations
to guide an uncertainty-aware optimization policy which decides where to next
observe $f(x)$.

Bayesian optimization begins with a prior $p(f)$ which represents the beliefs
about $f$ before any observations are made. Letting $y_{i}$ represent a noisy
or indirect observation of $f(x_{i})$, and collecting a sequence of
observations into a dataset $\mathcal{D}=(\mathbf{x},\mathbf{y})$, an
observation model defines the likelihood $p(\mathcal{D}|f)$. We then use the
observed data and Bayes theorem to update our beliefs and obtain the posterior

(1) $p(f|\mathcal{D})=\frac{p(f)p(\mathcal{D}|f)}{p(\mathcal{D})}.$

This posterior informs an acquisition function $\gamma(x|\mathcal{D})$ which
determines where to next observe $f(x)$ in a way that balances exploitation
(focusing observations where $f$ is likely near its maximum) with exploration
(probing areas where $f$ has high uncertainty).

###  2.2. Preference Elicitation

PE has witnessed decades of research, and includes approaches based on
Bayesian optimization (e.g., (Eric et al., 2007; Guo et al., 2010; Brochu et
al., 2010; González et al., 2017; Khajah et al., 2016)), Bandits (e.g., (Li et
al., 2010, 2011; Zhao et al., 2013; Christakopoulou et al., 2016)),
constrained optimization (Rossi and Sperduti, 2004), and POMDPs (Boutilier,
2002). In the standard PE setting, a user is assumed to have some hidden
utilities $\mathbf{u}=[u_{1},...,u_{N}]$ over a set $\mathcal{I}$ of $N$
items, where item $i$ is preferred to item $j$ if $u_{i}>u_{j}$. The goal of
PE is typically to search for an item $i^{*}\in\arg\max_{i}u_{i}$ that
maximizes user utility in a minimal number of PE queries, which most often ask
a user to express item preferences as item ratings (e.g., (Li et al., 2010,
2011; Brochu et al., 2010; Zhao et al., 2013; Christakopoulou et al., 2016))
or relative preferences between item pairs or sets (e.g., (Boutilier, 2002;
Guo and Sanner, 2010; Handa et al., 2024; Eric et al., 2007; González et al.,
2017; Vendrov et al., 2019)). An alternative form of PE asks users to express
preferences over predefined item features, also through rating- or comparison-
based queries (Li et al., 2021; Zhang et al., 2020; Lei et al., 2020). Central
to the above PE methods are query selection strategies that balance the
exploration and exploitation of user preferences, with TS and UCB algorithms
(cf. Sec. 4.2) often exhibiting strong performance (Zhao et al., 2013;
Christakopoulou et al., 2016; Yang et al., 2021; Li et al., 2021; Zhang et
al., 2020). However, none of these methods are able to interact with users
through NL dialogue.

Figure 2. The PEBOL NL-PE algorithm, which maintains a Bayesian belief state
over a user’s item preferences given an arbitrary set of NL item descriptions
$\mathbf{x}$. This belief is used by a decision-theoretic policy to balance
the exploration and exploitation of preferences by strategically selecting an
item description $x_{i^{t}}$ as the basis for LLM query generation. Belief
updates are computed through Bayesian inference with NLI entailment scores
between item descriptions and query-response pairs.

###  2.3. Language-Based Preference Elicitation

Yang et al. (Yang et al., 2021) introduce Bayesian PE strategies using TS and
UCB for keyphrase rating queries, where keyphrases are first mined from NL
item reviews and then co-embedded with user-item preferences in a
recommendation system. Handa et al. (Handa et al., 2024) propose using LLMs to
interface with a conventional Bayesian PE system, suggesting a preprocessing
step to extract features from NL descriptions and a verbalization step to
fluidly express pairwise item comparison queries. Li et al. (Li et al., 2023)
prompt an LLM to generate PE queries for some specific domain (e.g., news
content, morals), observe user responses, and evaluate LLM relevance
predictions for a single item. While these works make progress towards NL-PE,
they do not study how LLM query generation can strategically explore user
preferences towards an arbitrary item set outside the realm of item-based or
category-based feedback.

###  2.4. Conversational Recommendation

Recent work on ConvRec uses language models222Earlier systems (e.g. (Li et
al., 2018; Chen et al., 2019)) use relatively small RNN-based language models.
to facilitate NL dialogue while integrating calls to a recommender module
which generates item recommendations based on user-item interaction history
(Li et al., 2018; Chen et al., 2019; Wang et al., 2022; Yang et al., 2022). He
et al. (He et al., 2023) report that on common datasets, zero-shot GPT-3.5/4
outperforms these ConvRec methods, which generally use older language models
and require user-item interaction history for their recommendation modules.

###  2.5. Natural Language Inference

Binary Natural Language Inference (NLI) (Yin et al., 2019) models predict the
likelihood that one span of text called a premise is entailed by (i.e., can be
inferred from) a second span called the hypothesis. For example, an effective
NLI model should predict a high likelihood that the premise “I want to watch
Iron Man” entails the hypothesis “I want to watch a superhero movie”. As
illustrated by this example, the hypothesis typically must be more general
than the premise. NLI models are trained by fine-tuning encoder-only LLMs on
NLI datasets (Williams et al., 2017; Dagan et al., 2005; Thorne et al., 2018),
which typically consist of short text spans for the premise and hypothesis –
thus enabling relatively efficient performance on similar tasks with a fairly
small number LLM parameters.

Figure 3. Cherry-picked system-generated dialogues from our NL-PE experiments.
The Monolithic GPT-3.5 dialogue (left) demonstrates over-exploitation, with
$q^{3}$ directly extending $q^{2}$ after a positive user preference is
observed and leading to the extreme case of query repetition ($q^{4}$ =
$q^{3}$). In contrast, PEBOL (right) continues exploring even after a positive
response, while focusing on promising aspects (three out of four queries
elicit a positive response) by using UCB-guided query generation.

##  3\. Problem Definition

We now present a Bayesian optimization formulation of NL-PE. The goal of NL-PE
is to facilitate a NL dialogue which efficiently discovers a user’s most
preferred items out of a set of $N$ items. Each item $i\in\mathcal{I}$ has a
NL description $x_{i}$, which might be a title, long-form description, or even
a sequence of reviews, with the item set $\mathcal{I}$ collectively
represented by $\mathbf{x}\in\mathcal{X}$ with $\mathbf{x}=[x_{1},...,x_{N}]$.
We assume the user has some (unknown) utility function
$f:\mathcal{X}\rightarrow\mathbb{R}$ establishing hidden utilities
$\mathbf{u}=f(\mathbf{x})$ so that item $i$ is preferred to item $j$ if
$u_{i}>u_{j}$. Our goal is to find the most preferred item(s):

(2) $i^{*}\in\arg\max_{i\in\mathcal{I}}u_{i}.$

In contrast to standard Bayesian PE formalisms (c.f. Sec 2.2), we do not
assume that the user can effectively convey direct item-level preferences by
either: 1) providing item ratings (i.e., utilities) or 2) pairwise or listwise
item comparisons. Instead, we must infer user preferences by observing
utterances during a NL system-user dialogue. At turn $t$ of a dialogue, we let
$q^{t}$ and $r^{t}$ be the system and user utterance, respectively, with
$\mathbf{q}^{t}=[q^{1},...,q^{t}]$ and $\mathbf{r}^{t}=[r^{1},...,r^{t}]$
representing all system and user utterances up to $t$. In this paper, we call
$q^{t}$ the query and $r^{t}$ the response, though extensions to more generic
dialogues (e.g., when users can also ask queries) are discussed in Section 7.
We let $\mathcal{H}^{t}=(\mathbf{q}^{t},\mathbf{r}^{t})$ be the conversation
history at turn $t$.

To formulate NL-PE as a Bayesian optimization problem, we place a prior belief
on the user’s utilities $p(\mathbf{u}|\mathbf{x})$, conditioned on item
descriptions since they are available before the dialogue begins. We then
assume an observation model that gives the likelihood
$p(\mathbf{r}^{t}|\mathbf{x},\mathbf{u},\mathbf{q}^{t}$), letting us define
the posterior utility belief as

(3) $p(\mathbf{u}|\mathbf{x},\mathcal{H}^{t})\propto
p(\mathbf{r}^{t}|\mathbf{x},\mathbf{u,\mathbf{q}^{t}})p(\mathbf{u}|\mathbf{x}).$

This posterior informs an acquisition function
$\gamma(\mathbf{x},\mathcal{H}^{t})$ which generates333To represent the
generative acquisition of NL outputs, we deviate from the conventional
definition of acquisition functions as mapping to $\mathbb{R}.$ a new NL query

(4) $q^{t+1}=\gamma(\mathbf{x},\mathcal{H}^{t}),$

to systematically search for $i^{*}$. The preference beliefs also let us
define an Expected Utility (EU) $\mu_{i}^{t}$ for every item as

(5)
$\mu_{i}^{t}=\mathbb{E}_{p(\mathbf{u}|\mathbf{x},\mathcal{H}^{t})}[u_{i}],$

which allows the top-$k$ items to be recommended at any turn based on their
expected utilities.

Our Bayesian optimization NL-PE paradigm lets us formalize several key
questions, including:

  1. (1)

How do we represent beliefs $p(\mathbf{u}|\mathbf{x},\mathcal{H}^{t})$ in
user-item utilities $\mathbf{u}$, given NL item descriptions $\mathbf{x}$ and
a dialogue $\mathcal{H}^{t}$?

  2. (2)

What are effective models for the likelihood
$p(\mathbf{r}^{t}|\mathbf{x},\mathbf{u},\mathbf{q}^{t})$ of observed responses
$\mathbf{r}^{t}$ given $\mathbf{x}$, $\mathbf{q}^{t}$, and user utilities
$\mathbf{u}$?

  3. (3)

How can our beliefs inform the generative acquisition of NL queries $q^{t+1}$
given $\mathcal{H}^{t}$ to strategically search for $i^{*}$?

These questions reveal a number of novel research directions discussed further
in Section 7. In this paper, we present PEBOL, a NL-PE algorithm based on the
above Bayesian optimization NL-PE formalism, and numerically evaluate it
against monolithic LLM alternatives through controlled, simulated NL dialogues
(cf. Sec. 6).

##  4\. Methodology

##### Limitations of Monolithic LLM Prompting

An obvious NL-PE approach, described further as baseline in Section 5.1, is to
prompt a monolithic LLM with all item descriptions $\mathbf{x}$, dialogue
history $\mathcal{H}^{t}$, and instructions to generate a new query at each
turn. However, providing all item descriptions $[x_{1},...,x_{N}]$ in the LLM
context window is very computationally expensive for all but the smallest item
sets. While item knowledge could be internalized through fine-tuning, each
item update would imply system retraining. Critically, an LLM’s preference
elicitation behaviour cannot be controlled other than by prompt-engineering or
further fine-tuning, with neither option offering any guarantees of
predictable or interpretable behaviour that balances the exploitation and
exploration of user preferences.

##### PEBOL Overview

We propose to addresses these limitations by augmenting LLM reasoning with a
Bayesian Optimization procedure in a novel algorithm, PEBOL, illustrated in
Figure 2. At each turn $t$, our algorithm maintains a probabilistic belief
state over user preferences as a Beta belief state (cf. Sec. 4.1). This belief
state guides an LLM-based acquisition function to generate NL queries
explicitly balancing exploration and exploitation to uncover the top user
preferences (cf. Sec. 4.2). In addition, our acquisition function reduces the
context needed to prompt the LLM in each turn from all $N$ item descriptions
$\mathbf{x}$ to a single strategically selected item description $x_{i^{t}}$.
PEBOL then uses NLI over elicited NL preferences and item descriptions to map
dialogue utterances to numerical observations (c.f. Sec 4.3).

Figure 4. MAP@10 for MonoLLM and PEBOL-P with uncertainty-informed policies
(UCB, TS, ER) – all methods show preference learning over time. MonoLLM is
outperformed by PEBOL on Yelp and MovieLens, but remains competitive with
PEBOL up to the fourth turn on Recipe-MPR until demonstrating a sharp
performance drop that takes several turns to recover from.

###  4.1. Utility Beliefs

####  4.1.1. Prior Beliefs

Before any dialogue, PEBOL establishes a prior belief
$p(\mathbf{u}|\mathbf{x})$ on user-item utilities. We assume item utilities
are independent so that

(6) $p(\mathbf{u}|\mathbf{x})=\prod_{i=1}^{N}p(u_{i}|x_{i}),$

and that the prior for each utility $u_{i}$ is a Beta distribution

(7) $p(u_{i}|x_{i})=\text{Beta}(\alpha_{i}^{0},\beta_{i}^{0}).$

Since this paper focuses on fully cold start settings, we assume a uniform
Beta prior with $(\alpha_{i}^{0},\beta_{i}^{0})=(1,1)$. Beta distributions,
illustrated in Figure 1, lie in the domain $[0,1]$ – a normalized interval for
bounded ratings in classical recommendation systems. We can thus interpret
utility values of $u_{i}=1$ or $u_{i}=0$ to represent a complete like or
dislike of item $i$, respectively, while values $u_{i}\in(0,1)$ provide a
strength of preference between these two extremes.

####  4.1.2. Observation Model

To perform a posterior update on our utility beliefs given observed responses
$\mathbf{r}^{t}$, we need an observation model that represents the likelihood
$p(\mathbf{r}^{t}|\mathbf{x},\mathbf{u},\mathbf{q}^{t})$. Modelling the
likelihood of $\mathbf{r}^{t}$ is a challenging task, so we will require some
simplifying assumptions. Firstly, we assume that the likelihood of a single
response $r^{t}$ is independent from any previous dialogue history
$\mathcal{H}^{t-1}$, so that:

(8)
$p(\mathbf{r}^{t}|\mathbf{x},\mathbf{u},\mathbf{q}^{t})=\prod_{t^{\prime}=1}^{t}p(r^{t^{\prime}}|\mathbf{x},\mathbf{u},q^{t^{\prime}}).$

Note that this independence assumption will allow incremental posterior belief
updates, so that

(9) $p(\mathbf{u}|\mathbf{x},\mathcal{H}^{t})\propto
p(r^{t}|\mathbf{x},\mathbf{u},q^{t})p(\mathbf{u}|\mathbf{x},\mathcal{H}^{t-1}).$

####  4.1.3. Binary Item Response Likelihoods and Posterior Update

With the factorized distributions over item utilities and observational
likelihood history now defined, we simply have to provide a concrete
observational model of the response likelihood conditioned on the query, item
descriptions, and latent utility: $p(r^{t}|\mathbf{x},\mathbf{u},q^{t})$.

Because the prior is factorized over conditionally independent $u_{i}$ (cf.
(6)), we can likewise introduce individual per-item factorized binary
responses $r^{t}_{i}\in\\{0(\mathit{dislike}),1(\mathit{like})\\}$ to
represent the individual relevance of each item $i$ to the preference elicited
at turn $t$. Critically, we won’t actually require an individual response per
item — this will be computed by a natural language inference (NLI) model
(Dagan et al., 2005) to be discussed shortly — but we’ll begin with an
individual binary response model for $r_{i}^{t}$ for simplicity:

(10) $p(r_{i}^{t}|x_{i},u_{i},q^{t})=\mbox{Bernoulli}(u_{i}).$

With our response likelihood defined, this now leads us to our first pass at a
full posterior utility update that we term PEBOL-B for observed _Binary_
rating feedback. Specifically, given observed binary ratings $r^{t}_{i}$, the
update at $t=1$ uses the Beta prior (7) with the Bernoulli likelihood (10) to
form a standard Beta-Bernoulli conjugate pair and compute the posterior
utility belief

(11) $\displaystyle p(u_{i}|x_{i},\mathcal{H}^{1})$ $\displaystyle\propto
p(u_{i}|x_{i})p(r^{1}_{i}|x_{i},u_{i},q^{t})$ (12)
$\displaystyle=\text{Beta}(\alpha_{i}^{1},\beta_{i}^{1}),$

where $\alpha_{i}^{1}=\alpha_{i}^{0}+r_{i}^{1}$,
$\beta_{i}^{1}=\beta_{i}^{0}+(1-r_{i}^{1})$. Subsequent incremental updates
updates follow Eq. (9) and use the same conjugacy to give

(13) $\displaystyle
p(u_{i}|x_{i},\mathcal{H}^{t})=\text{Beta}(\alpha_{i}^{t},\beta_{i}^{t}),$

where $\alpha_{i}^{t}=\alpha_{i}^{t-1}+r_{i}^{t}$,
$\beta_{i}^{t}=\beta_{i}^{t-1}+(1-r_{i}^{t})$.

####  4.1.4. Natural Language Inference and Probabilistic Posterior Update

As hinted above, effective inference becomes slightly more nuanced since we
don’t need to observe an explicit binary response _per item_ in our PEBOL
framework. Rather, we receive general preference feedback $r^{t}$ on whether a
user generically prefers a text description $q^{t}$ and then leverage an NLI
model (Dagan et al., 2005) to _infer_ whether the description $x_{i}$ of item
$i$ would be preferred according to this feedback. For instance, for a
$(q^{t},r^{t})$ pair (“Want to watch a children’s movie?”,“Yes”), NLI should
infer a rating of $r_{1}^{t}=1$ for $x_{1}=$ “The Lion King” and $r_{2}^{t}=0$
for $x_{2}=$ “Titanic”.

To deal with the fact that NLI models actually return an _entailment
probability_ , our probabilistic observation variant, PEBOL-P leverages the
entailment probability that $r_{i}^{t}=1$ (cf. Sec 4.3), which we denote as
$w_{i}^{t}\in[0,1]$. We then simply relax our previous binary observations
$r_{i}^{t}$ to continuous $w_{i}^{t}$, thus leading to soft posterior updates
$\alpha_{i}^{t}=\alpha_{i}^{t-1}+w_{i}^{t}$,
$\beta_{i}^{t}=\beta_{i}^{t-1}+(1-w_{i}^{t})$.

To visually illustrate how this posterior inference process works in practice,
Figure 1 shows the effect of PEBOL’s posterior utility belief updates based on
NLI for three query-response pairs – we can see the system gaining statistical
knowledge about useful items for the user from the dialogue.

Figure 5. MAP@10 for PEBOL-P with various context acquisition policies.

###  4.2. LLM-Based Acquisition Functions

Recall from Sec. 2.1 that in Bayesian optimization, the posterior informs an
acquisition function which determines where to make the next observation.
PEBOL generates a new query $q^{t}$ with a two-step acquisition function
$\gamma$, first using Bayesian Optimization policies (step 1) based on the
posterior utility beliefs $p(\mathbf{u}|\mathbf{x},\mathcal{H}^{t})$ to select
NL context, and then using this selected context to guide LLM prompting (step
2). We express the overall acquisition function
$\gamma=\gamma^{G}\circ\gamma^{C}$ as a composition of a context acquisition
function $\gamma^{C}$ (cf. Sec. 4.2.1) and a NL generation function
$\gamma^{G}$ (cf. Sec. 4.2.2).

####  4.2.1. Context Acquisition via Bayesian Optimization Policies

First, PEBOL harnesses Bayesian optimization policies to select an item
description $x_{i^{t}}$ which will be used to prompt an LLM to generate a
query about an aspect described by $x_{i^{t}}$ (cf. Sec. 4.2.2). Selecting an
item $i^{t}$ whose utility $u_{i^{t}}$ is expected to be near the maximum,
$u_{i^{*}}$, will generate exploitation queries asking about properties of
items that are likely to be preferred by the user. In contrast, selecting an
item $i^{t}$ associated with high uncertainty in its utility $u_{i}^{t}$ will
generate exploration queries that probe into properties of items for which
user preferences are less known. Thus, strategically selecting $x_{i^{t}}$
allows PEBOL to balance the exploration and exploitation behaviour of NL
queries, decreasing the risks of becoming stuck in local optima (over-
exploitation) or wasting resources exploring low utility item preferences
(over-exploration). We define the item selected by the context acquisition
function as

(14) $i^{t}=\gamma^{C}(\mathbf{x},\mathcal{H}^{t}),$

and list several alternatives for $\gamma^{C}$, including the well-known
strategies of TS and UCB (Shahriari et al., 2016):

  1. (1)

Thompson Sampling (TS): First, a sample of each item’s utility
$\hat{u}^{t}_{i}$ is taken from the posterior, $\hat{u}^{t}_{i}\sim
p(u_{i}|x_{i},\mathcal{H}^{t}).$ Then, the item with the highest sampled
utility is selected:

(15) $i^{t}=\arg\max_{i}\hat{u}^{t}_{i}.$

TS explores more when beliefs have higher uncertainty and exploits more as the
system becomes more confident.

  2. (2)

Upper Confidence Bound (UCB): Let $P_{k}(\alpha,\beta)$ represent the $k$’th
percentile of $\text{Beta}(\alpha,\beta)$, which provides a confidence bound
on the posterior. UCB selects the item with the highest confidence bound

(16) $i^{t}=\arg\max_{i}P_{k}(p(u_{i}|x_{i},\mathcal{H}^{t})),$

following a balanced strategy because confidence bounds are increased by both
high utility and high uncertainty.

  3. (3)

Entropy Reduction (ER): An explore-only strategy that selects the item with
the most uncertain utility:

(17) $i^{t}=\arg\max_{i}\text{Var}(p(u_{i}|x_{i},\mathcal{H}^{t})).$

  4. (4)

Greedy: An exploit-only strategy that selects the item with the highest
expected utility $\mu_{i}^{t}$ (Eq. 5):

(18) $i^{t}=\arg\max_{i}\mu_{i}^{t}.$

  5. (5)

Random: An explore-only heuristic that selects the next item randomly.

####  4.2.2. Generating Short, Aspect-Based NL Queries

Next, PEBOL prompts an LLM to generate a NL query $q^{t}$ based on the
selected item description $x_{i^{t}}$ while also using the dialogue history
$\mathcal{H}^{t}$ to avoid repetitive queries. We choose to generate “yes-or-
no” queries asking if a user prefers items with some aspect $a^{t}$, which is
a short text span extracted dynamically from $x_{i^{t}}$ to be different from
any previously queried aspects $a^{1},...,a^{t-1}$. We adopt this query
generation strategy to: 1) reduce cognitive load on the user, who may be
frustrated by long and specific queries about unfamiliar items and 2) better
facilitate NLI through brief, general phrases (Yin et al., 2019). Letting
$\phi$ represent the query generation prompt, we let

(19) $q^{t},a^{t}=\gamma^{G}(x_{i^{t}},\mathcal{H}^{t},\phi)$

be the LLM generated query and aspect at turn $t$, with prompting details
discussed in Section 5.2.2. An example of such a query and aspect (bold) is
“Are you interested in movies with patriotic themes?”, generated by PEBOL in
our movie recommendation experiments and shown in Figure 2.

###  4.3. NL Item-Preference Entailment

####  4.3.1. Preference Descriptions from Query Response Pairs

Next, PEBOL receives a NL user response $r^{t}$, which it must convert to
individual item preference observations. Since the LLM is instructed to
generate ”yes-or-no” queries $q^{t}$ asking a user if they like aspect
$a^{t}$, we assume the user response will be a ”yes” or a ”no”, and create a
NL description of the users preference $\rho^{t}$, letting $\rho^{t}=a^{t}$ if
$r^{t}=$“yes”, and $\rho^{t}=\text{concat({``not ''}},a^{t})$ if $r^{t}=$
“no”. For example, given a query that asks if the user prefers the aspect
“patriotism” in an item, if the user response is “yes”, then the user
preference $\rho^{t}$ is “patriotism”, and “not patriotism” otherwise. This
approach produces short, general preference descriptions that are well suited
for NLI models (Yin et al., 2019).

####  4.3.2. Inferring Item Ratings from NL Preferences

Given a NL preference $\rho^{t}$, PEBOL must infer whether the user would like
an item described by $x_{i}$. Specifically, PEBOL acquires ratings
$\mathbf{w}^{t}=[w^{t}_{1},...,w^{t}_{N}]$ (cf. Sec. 4.1.4) by using NLI to
predict whether an item description $x_{i}$ entails (i.e., implies) the
preference $\rho^{t}$. For example, we expect that an NLI model would predict
that $x_{i}=$“The Lion King” entails $\rho^{t}=$“animated” while
$x_{j}=$“Titanic” does not, inferring that a user who expressed preference
$\rho^{t}$ would like item $i$ but not $j$. We use an NLI model
$g_{\theta}(x_{i},\rho^{t})$ to predict the probability $w_{i}^{t}$ that
$x_{i}$ entails $\rho^{t}$, and return $r_{i}^{t}=\lfloor w_{i}^{t}\rceil$ in
the case of binary observations (PEBOL-B) and $w_{i}^{t}$ in the case of
probabilistic observations (PEBOL-P).

###  4.4. The Complete PEBOL System

This concludes the PEBOL specification – the entire process from prior utility
belief to the LLM-based acquisition function generation of a query to the
posterior utility update is illustrated in Figure 2.

Figure 6. MAP@10 for PEBOL using binary vs. probabilistic entailment scores.
PEBOL-P with the best policy (TS on Yelp and MovieLens, UCB on Recipe-MPR)
generally outperforms PEBOL-B.

##  5\. Experimental Methods

We numerically evaluate our PEBOL variations through controlled NL-PE dialogue
experiments across multiple datasets and response noise levels – comparing
against a monolithic GPT-3.5-turbo NL-PE system, which we refer to as MonoLLM.
We do not compare against ConvRec methods (Li et al., 2018; Chen et al., 2019;
Wang et al., 2022; Yang et al., 2022) because they are not cold-start systems,
requiring observed user-item interactions data to drive their recommendation
modules. We also do not base our experiments on ConvRec datasets such as
ReDIAL (Li et al., 2018), since they are made up of pre-recorded conversation
histories and cannot be used to evaluate active, cold-start NL-PE systems.

###  5.1. MonoLLM Baseline

A major challenge of using MonoLLM for NL-PE is that item descriptions
$\mathbf{x}$ either need to be internalized through training or be provided in
the context window (cf. Sec. 4). Since we focus on fully cold-start settings,
we test the latter approach as a baseline. Specifically, we prompt the LLM to
generate new queries to elicit preferences given the full conversation history
$\mathcal{H}^{t}$ and $\mathbf{x}$ (all prompts are shown in the Supplementary
Materials). We evaluate recommendation performance after each turn by using
another prompt to recommend a list of ten item names from $\mathbf{x}$ given
$\mathcal{H}^{t}$. Due to context window limits, this MonoLLM approach is only
feasible for small item sets with short item descriptions; thus, we have to
limit $|\mathcal{I}|$ to 100 for fair comparison to the MonoLLM baseline.

###  5.2. Simulation Details

We test PEBOL and MonoLLM through NL-PE dialogues with LLM-simulated users
whose item preferences are hidden from the system, evaluating recommendation
performance over 10 turns of dialogue, and using GPT-3.5-turbo-0613 for all
text generation.

####  5.2.1. User Simulation

For each experiment, we simulate 100 users, each of which likes a single item
$i\in\mathcal{I}$. Each user is simulated by GPT-3.5-turbo-0613, which is
given item description $x_{i}$ and instructed to provide only “yes” or “no”
responses to a query $q^{t}$ as if it was a user who likes item $i$.

####  5.2.2. PEBOL Query Generation

Given an item description $x_{i}$ and previously generated aspects
$(a^{1},...,a^{t-1})$, an LLM is prompted to generate an aspect $a^{t}$
describing the item $i$ that is no more than 3 words long. The LLM is then
prompted again to generate a “yes-or-no” query asking if a user prefers
$a^{t}$.

####  5.2.3. NLI

We use the 400M FAIR mNLI444https://huggingface.co/facebook/bart-large-mnli
model to predicts logits for entailment, contradiction, and neutral, and
divide these logits by an MNLI temperature $T\in\\{1,10,100\\}$ As per the
FAIR guidelines, we pass the temperature-scaled entailment and contradiction
scores through a softmax layer and take the entailment probabilities.

####  5.2.4. User Response Noise

We test three user response noise levels $\in\ ${0,0.25,0.5} corresponding to
the proportion or user responses that are randomly selected between ”yes” and
”no”.

####  5.2.5. Omitting Query History Ablation

We test how tracking query history in PEBOL effects performance with an
ablation study that removes previously generated aspects $(a^{1},...,a^{t-1})$
from the aspect extraction prompt.

###  5.3. Datasets

We obtain item descriptions from three real-world datasets:
MovieLens25M555https://grouplens.org/datasets/movielens/25m/,
Yelp666https://www.yelp.com/dataset, and Recipe-MPR (Zhang et al., 2023)
(example item descriptions from each shown in Table 1 in the Supplementary
Materials). After the filtering steps below for Yelp and MovieLens, we
randomly sample 100 items to create $\mathbf{x}$. For Yelp, we filter
restaurant descriptions to be from a single major North American city and to
have at least 50 reviews and five or more category labels. For
MovieLens,777For all experiments with MovieLens, we use the 16k version of
GPT-3.5-turbo-0613, due to MonoLLM requiring extra context length for
$\mathbf{x}.$ we filter movies to be in the 10% by rating count with at least
20 tags, and let movie descriptions use the title, genre labels, and 20 most
common user-assigned tags.

Figure 7. The effect of including the generated aspect history in the aspect
generation prompt. Including the history improves performance, which we
hypothesize is due to reducing repeated or uninformative queries. Figure 8.
The effect of user response noise on MAP@10 – error bars are 95% confidence
intervals. At all noise levels, MonoLLM is outperformed by all PEBOL-P methods
on Yelp and MovieLens and by UCB, Greedy, and ER on Recipe-MPR.

###  5.4. Research Questions

Our experiments explore the following research questions (RQs):

  * •

RQ1: How does PEBOL perform against the GPT-3.5 Mono-LLM baseline?

  * •

RQ2: Does PEBOL perform better with binary or probabilistic observations, and
how sensitive is the latter to temperature?

  * •

RQ3: How do PEBOL and MonoLLM perform under user response noise?

  * •

RQ4: How do the context selection policies of TS, UCB, ER, Greedy, and Random
effect PEBOL performance?

  * •

RQ5: How much does PEBOL performance depend on access to the query history
during query generation?

##  6\. Experimental Results

###  6.1. RQ1 - PEBOL vs. MonoLLM

Figure 4 shows MAP@10 over 10 dialogue turns for MonoLLM and PEBOL
(UCB,TS,ER), with all methods starting near random guessing (reflecting a cold
start) and showing clear preference learning over time.888For each policy, we
use the MNLI temperature that performed best on the most datasets (see
Supplementary Materials) with continous responses. PEBOL achieves a 131%
improvement over MonoLLM in MAP@10 after 10 dialogue turns on Yelp, an 88%
improvement on MovieLens, and a 55% improvement on RecipeMPR. MonoLLM also
displays a large drop in MAP@10 between turns 4 and 5 on Recipe-MPR which it
takes an additional five turns to recover from. In contrast, we do not see any
comparably large drops in our methods, indicating that the incremental
Bayesian belief updating mechanism of PEBOL may make it less prone to
catastrophic mistakes.

###  6.2. RQ2 - Binary vs. Probabilistic Responses

Figure 6 compares PEBOL performance for PEBOL-B versus PEBOL-P. In general,
PEBOL-P performed better, likely due to PEBOL-B discarding valuable
information from entailment probabilities. Specifically Figure 6 compares
PEBOL-B with each policy against PEBOL-P with the best performing policy on
each dataset. On MovieLens, we see at least a 34% improvement in MAP from
probabilistic entailment over binary entailment starting from turn 2, though
we note that TS with binary updates performs comparably to the best
probabilistic methods on Yelp.

To identify our best continuous method, we first compare the performance of
PEBOL with PEBOL-P using MNLI temperatures 1, 10, and 100 with each policy. We
keep the temperature with the highest MAP@10 at turn 10 for the most datasets:
for Yelp and MovieLens, this is TS with MNLI temperature 1; for Recipe-MPR, it
is UCB with MNLI temperature 10.

###  6.3. RQ3 - Effect of User Response Noise

Figure 8 shows the impact of user response noise on MAP@10 at turn 10 – PEBOL
generally continues to outperform MonoLLM under user response noise. At turn
10 for all noise levels, MonoLLM is the worst performing method on Yelp and
MovieLens, and on RecipeMPR it is always outperformed by UCB, Greedy, and ER.
Methods typically maintain their relative performance under various noise
levels – though an exception is MovieLens where TS is the best method under
zero noise and nearly the worst one at 25% noise (though the margin between
the best and worst method in this case is small).

###  6.4. RQ4 - Comparison of Context Acquisition Policies

Figure 5 compares the performance of various PEBOL context acquisition
policies – all policies show active preference learning, other than random
item selection on RecipeMPR. There is considerable overlap between methods,
however for most turns TS does well on Yelp and MovieLens while being beaten
by Greedy, ER, and UCB on Recipe-MPR. As expected due to the randomness in
sampling, TS performance is correlated with random item selection, while UCB
performs quite similarly to greedy.

###  6.5. RQ5 - Effect of Aspect History in Query Generation

We see an improvement in performance from including a list of previously
generated aspects in the aspect generation prompt. In Figure 7, the average
improvement across all turns and all datasets for including the aspect history
is 50% for TS and 49% for UCB. Including the aspect generation history helps
to avoid repeat queries, which gain no information and would frustrate a user.

##  7\. Conclusion and Future Work

This paper presents a novel Bayesian optimization formalization of natural
language (NL) preference elicitation (PE) over arbitrary NL item descriptions,
as well as introducing and evaluating PEBOL, an algorithm for NL Preference
Elicitation with Bayesian Optimization augmented LLMs – and establishing many
new research directions. For instance, while PEBOL uses a pointwise query
generation strategy that selects one item description at a time for LLM
context, future work can explore LLM-based acquisition functions with pairwise
and setwise context selection. Such multi-item context selection would enable
contrastive query generation that could better discriminate between item
preferences. Another direction for future research is the integration of NL-PE
methodologies such as PEBOL into conversational recommendation (ConvRec)
system architectures (e.g. (Friedman et al., 2023; Wang et al., 2023)), which
must balance many tasks including recommendation, explanation, and
personalized question answering. PE in ConvRec systems thus implies that
algorithms need to elicit preferences given arbitrary pairs of system-user
utterances, in addition to queries and responses.

## References

  * (1)
  * Biyik et al. (2023) Erdem Biyik, Fan Yao, Yinlam Chow, Alex Haig, Chih wei Hsu, Mohammad Ghavamzadeh, and Craig Boutilier. 2023.  Preference Elicitation with Soft Attributes in Interactive Recommendation.  _ArXiv_ abs/2311.02085 (2023).  https://api.semanticscholar.org/CorpusID:265034238
  * Boutilier (2002) Craig Boutilier. 2002\.  A POMDP formulation of preference elicitation problems. In _AAAI/IAAI_. Edmonton, AB, 239–246.
  * Brochu et al. (2010) Eric Brochu, Tyson Brochu, and Nando De Freitas. 2010\.  A Bayesian interactive optimization approach to procedural animation design. In _Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation_. 103–112.
  * Chen et al. (2019) Qibin Chen, Junyang Lin, Yichang Zhang, Ming Ding, Yukuo Cen, Hongxia Yang, and Jie Tang. 2019.  Towards knowledge-based recommender dialog system.  _arXiv preprint arXiv:1908.05391_ (2019).
  * Christakopoulou et al. (2016) Konstantina Christakopoulou, Filip Radlinski, and Katja Hofmann. 2016\.  Towards Conversational Recommender Systems. In _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ (San Francisco, California, USA) _(KDD ’16)_. Association for Computing Machinery, New York, NY, USA, 815–824.
  * Dagan et al. (2005) Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005.  The pascal recognising textual entailment challenge. In _Machine learning challenges workshop_. Springer, 177–190.
  * Eric et al. (2007) Brochu Eric, Nando Freitas, and Abhijeet Ghosh. 2007\.  Active preference learning with discrete choice data.  _Advances in neural information processing systems_ 20 (2007).
  * Friedman et al. (2023) Luke Friedman, Sameer Ahuja, David Allen, Terry Tan, Hakim Sidahmed, Changbo Long, Jun Xie, Gabriel Schubiner, Ajay Patel, Harsh Lara, et al. 2023\.  Leveraging Large Language Models in Conversational Recommender Systems.  _arXiv preprint arXiv:2305.07961_ (2023).
  * Garnett (2023) Roman Garnett. 2023\.  _Bayesian optimization_.  Cambridge University Press.
  * González et al. (2017) Javier González, Zhenwen Dai, Andreas Damianou, and Neil D Lawrence. 2017.  Preferential bayesian optimization. In _International Conference on Machine Learning_. PMLR, 1282–1291.
  * Guo and Sanner (2010) Shengbo Guo and Scott Sanner. 2010.  Real-time multiattribute Bayesian preference elicitation with pairwise comparison queries. In _Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics_. JMLR Workshop and Conference Proceedings, 289–296.
  * Guo et al. (2010) Shengbo Guo, Scott Sanner, and Edwin V Bonilla. 2010\.  Gaussian process preference elicitation.  _Advances in neural information processing systems_ 23 (2010).
  * Handa et al. (2024) Kunal Handa, Yarin Gal, Ellie Pavlick, Noah Goodman, Jacob Andreas, Alex Tamkin, and Belinda Z. Li. 2024.  Bayesian Preference Elicitation with Language Models.  arXiv:2403.05534 [cs.CL]
  * He et al. (2023) Zhankui He, Zhouhang Xie, Rahul Jha, Harald Steck, Dawen Liang, Yesu Feng, Bodhisattwa Prasad Majumder, Nathan Kallus, and Julian McAuley. 2023\.  Large language models as zero-shot conversational recommenders.  _arXiv preprint arXiv:2308.10053_ (2023).
  * Karamanolakis et al. (2018) Giannis Karamanolakis, Kevin Raji Cherian, Ananth Ravi Narayan, Jie Yuan, Da Tang, and Tony Jebara. 2018.  Item recommendation with variational autoencoders and heterogeneous priors. In _Proceedings of the 3rd Workshop on Deep Learning for Recommender Systems_. 10–14.
  * Khajah et al. (2016) Mohammad M. Khajah, Brett D. Roads, Robert V. Lindsey, Yun-En Liu, and Michael C. Mozer. 2016\.  Designing Engaging Games Using Bayesian Optimization. In _Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems_ (San Jose, California, USA) _(CHI ’16)_. Association for Computing Machinery, New York, NY, USA, 5571–5582.  https://doi.org/10.1145/2858036.2858253
  * Lee et al. (2019) Hoyeop Lee, Jinbae Im, Seongwon Jang, Hyunsouk Cho, and Sehee Chung. 2019.  MeLU: Meta-Learned User Preference Estimator for Cold-Start Recommendation.  arXiv:1908.00413 [cs.IR]
  * Lei et al. (2020) Wenqiang Lei, Gangyi Zhang, Xiangnan He, Yisong Miao, Xiang Wang, Liang Chen, and Tat-Seng Chua. 2020.  Interactive Path Reasoning on Graph for Conversational Recommendation. In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_ (Virtual Event, CA, USA) _(KDD ’20)_. Association for Computing Machinery, New York, NY, USA, 2073–2083.  https://doi.org/10.1145/3394486.3403258
  * Li et al. (2023) Belinda Z. Li, Alex Tamkin, Noah Goodman, and Jacob Andreas. 2023.  Eliciting Human Preferences with Language Models.  arXiv:2310.11589 [cs.CL]
  * Li et al. (2010) Lihong Li, Wei Chu, John Langford, and Robert E Schapire. 2010\.  A contextual-bandit approach to personalized news article recommendation. In _Proceedings of the 19th international conference on World wide web_. 661–670.
  * Li et al. (2011) Lihong Li, Wei Chu, John Langford, and Xuanhui Wang. 2011\.  Unbiased offline evaluation of contextual-bandit-based news article recommendation algorithms. In _Proceedings of the fourth ACM international conference on Web search and data mining_. 297–306.
  * Li et al. (2018) Raymond Li, Samira Ebrahimi Kahou, Hannes Schulz, Vincent Michalski, Laurent Charlin, and Chris Pal. 2018.  Towards deep conversational recommendations.  _Advances in neural information processing systems_ 31 (2018).
  * Li et al. (2021) Shijun Li, Wenqiang Lei, Qingyun Wu, Xiangnan He, Peng Jiang, and Tat-Seng Chua. 2021\.  Seamlessly Unifying Attributes and Items: Conversational Recommendation for Cold-start Users.  _ACM Trans. Inf. Syst._ 39, 4, Article 40 (aug 2021), 29 pages.  https://doi.org/10.1145/3446427
  * Maynez et al. (2020) Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020.  On faithfulness and factuality in abstractive summarization.  _arXiv preprint arXiv:2005.00661_ (2020).
  * Rossi and Sperduti (2004) Francesca Rossi and Allesandro Sperduti. 2004.  Acquiring both constraint and solution preferences in interactive constraint systems.  _Constraints_ 9, 4 (2004), 311–332.
  * Shahriari et al. (2016) Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P. Adams, and Nando de Freitas. 2016.  Taking the Human Out of the Loop: A Review of Bayesian Optimization.  _Proc. IEEE_ 104, 1 (2016), 148–175.  https://doi.org/10.1109/JPROC.2015.2494218
  * Thorne et al. (2018) James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018.  FEVER: a large-scale dataset for fact extraction and VERification.  _arXiv preprint arXiv:1803.05355_ (2018).
  * Vendrov et al. (2019) Ivan Vendrov, Tyler Lu, Qingqing Huang, and Craig Boutilier. 2019\.  Gradient-based Optimization for Bayesian Preference Elicitation.  _CoRR_ abs/1911.09153 (2019).  arXiv:1911.09153 http://arxiv.org/abs/1911.09153
  * Wang et al. (2023) Xiaolei Wang, Xinyu Tang, Wayne Xin Zhao, Jingyuan Wang, and Ji-Rong Wen. 2023.  Rethinking the evaluation for conversational recommendation in the era of large language models.  _arXiv preprint arXiv:2305.13112_ (2023).
  * Wang et al. (2022) Xiaolei Wang, Kun Zhou, Ji-Rong Wen, and Wayne Xin Zhao. 2022\.  Towards unified conversational recommender systems via knowledge-enhanced prompt learning. In _Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_. 1929–1937.
  * Williams et al. (2017) Adina Williams, Nikita Nangia, and Samuel R Bowman. 2017\.  A broad-coverage challenge corpus for sentence understanding through inference.  _arXiv preprint arXiv:1704.05426_ (2017).
  * Yang et al. (2022) Bowen Yang, Cong Han, Yu Li, Lei Zuo, and Zhou Yu. 2022.  Improving Conversational Recommendation Systems’ Quality with Context-Aware Item Meta-Information. In _Findings of the Association for Computational Linguistics: NAACL 2022_. 38–48.
  * Yang et al. (2021) Hojin Yang, Scott Sanner, Ga Wu, and Jin Peng Zhou. 2021\.  Bayesian Preference Elicitation with Keyphrase-Item Coembeddings for Interactive Recommendation. In _Proceedings of the 29th ACM Conference on User Modeling, Adaptation and Personalization_. 55–64.
  * Yin et al. (2019) Wenpeng Yin, Jamaal Hay, and Dan Roth. 2019.  Benchmarking zero-shot text classification: Datasets, evaluation and entailment approach.  _arXiv preprint arXiv:1909.00161_ (2019).
  * Zhang et al. (2023) Haochen Zhang, Anton Korikov, Parsa Farinneya, Mohammad Mahdi Abdollah Pour, Manasa Bharadwaj, Ali Pesaranghader, Xi Yu Huang, Yi Xin Lok, Zhaoqi Wang, Nathan Jones, and Scott Sanner. 2023.  Recipe-MPR: A Test Collection for Evaluating Multi-aspect Preference-based Natural Language Retrieval. In _Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval_ (¡conf-loc¿, ¡city¿Taipei¡/city¿, ¡country¿Taiwan¡/country¿, ¡/conf-loc¿) _(SIGIR ’23)_. Association for Computing Machinery, New York, NY, USA, 2744–2753.  https://doi.org/10.1145/3539618.3591880
  * Zhang et al. (2020) Xiaoying Zhang, Hong Xie, Hang Li, and John C.S. Lui. 2020\.  Conversational Contextual Bandit: Algorithm and Application. In _Proceedings of The Web Conference 2020_ (Taipei, Taiwan) _(WWW ’20)_. Association for Computing Machinery, New York, NY, USA, 662–672.  https://doi.org/10.1145/3366423.3380148
  * Zhao et al. (2013) Xiaoxue Zhao, Weinan Zhang, and Jun Wang. 2013.  Interactive collaborative filtering. In _Proceedings of the 22nd ACM international conference on Information & Knowledge Management_. 1411–1420.