# An investigation of type-I superconductivity in single crystal of Pb2Pd

Arushi  Department of Physics, Indian Institute of Science Education and
Research Bhopal, Bhopal, 462066, India  K. Motla  Department of Physics,
Indian Institute of Science Education and Research Bhopal, Bhopal, 462066,
India  A. Kataria  Department of Physics, Indian Institute of Science
Education and Research Bhopal, Bhopal, 462066, India  S. Sharma  Department of
Physics and Astronomy, McMaster University, Hamilton, Ontario L8S 4M1, Canada
J. Beare  Department of Physics and Astronomy, McMaster University, Hamilton,
Ontario L8S 4M1, Canada  M. Pula  Department of Physics and Astronomy,
McMaster University, Hamilton, Ontario L8S 4M1, Canada  M. Nugent  Department
of Physics and Astronomy, McMaster University, Hamilton, Ontario L8S 4M1,
Canada  G. M. Luke  Department of Physics and Astronomy, McMaster University,
Hamilton, Ontario L8S 4M1, Canada  TRIUMF, Vancouver, British Columbia V6T
2A3, Canada  R. P. Singh<EMAIL_ADDRESS>Department of Physics, Indian
Institute of Science Education and Research Bhopal, Bhopal, 462066, India

###### Abstract

We have investigated the superconducting properties in a single crystal of a
new superconductor Pb2Pd via various techniques including magnetization, AC
transport, transverse field muon spin rotation and relaxation (TF-$\mu$SR),
and heat capacity. Pb2Pd crystallizes in a body-centred tetragonal structure
with space group I4/$mcm$. All measurements confirm the superconducting
transition temperature, TC = 3.0 $\pm$ 0.1 K. Electronic specific heat data
are well described by the BCS fitting, suggesting that Pb2Pd opens an
isotropic gap on entering the superconducting state. The specific heat jump
and $\lambda_{e-ph}$ value categorize Pb2Pd as a moderately coupled
superconductor. Magnetization and transverse field muon spin rotation
measurements along with Ginzburg-Landau parameter, $\kappa$ < 1/$\sqrt{2}$
strongly infers that Pb2Pd is a type I superconductor.

##  I Introduction

In recent years, the discovery and study of exotic phases of matter such as
Weyl and Dirac semimetal weyl&dirac , topological insulators topIns  and,
topological superconductors topSC  continues to interest both theorists and
experimentalists. Among these phases, topological superconductivity (TSC) is
one of the most studied phenomena and has become a rapidly developing field in
which significant advances have been recently made topSC . It not only
enriches the existing theoretical framework of physics but also provides a
background for investigations on low energy excitations, Majorana fermions
which have potential applications in fault-tolerant quantum computation Majo1
; Majo2 . Apart from computation application, TSC itself is an intriguing
topological phase of matter, namely, a new type of unconventional
superconductivity. In order to realize a topological superconducting state,
strong spin-orbital coupling (SOC) is considered one of the necessary
ingredients SOCtop . Superconducting compounds containing heavy atomic number
(Z) elements are proposed to have strong SOC since the strength of SOC is
proportional to Z4. These superconducting systems with strong SOC often show
various exotic properties such as robustness of superconductivity beyond Pauli
limit Pauli1 ; Pauli2 ; Pauli3 , nodes and multiple gaps in the
superconducting state nod&mult1 ; nod&mult2 ; nod&mult3 , time-reversal
symmetry breaking (TRSB) TRSB1 ; TRSB2 ; TRSB3 , as well as being a platform
to realize topological superconductivity. Intrinsic topological
superconductivity is still a rare phenomenon though various methods have been
employed in its study.

To date, Pb rich compounds have not been investigated as potential candidates
to exhibit the exotic superconducting features although they display a range
of other interesting electronic properties. Recent work on some of the Pb rich
compounds (MPb2 where M = Au, Rh, Er) predicted them to be a Dirac semimetal,
topological superconductor, and coexistence of superconductivity with
antiferromagnetic order, respectively SOCtop3&AuPb2 ; AuPb2 ; RhPb2 ; ErPb2 .
The existence of topological phases of matter in Pb rich compounds inspires us
to study the superconducting and normal state properties of a theoretically
proposed topological material Pb2Pd Pb2Pd_theor ; Pb2Pd .

In this paper, we report the superconductivity in a single crystal of a new
type I superconductor Pb2Pd. This behaviour is surprising because only a few
binary and ternary compounds are known to exhibit type I superconductivity.
Electrical transport, magnetization and specific heat measurements confirm
bulk superconductivity at a transition temperature, TC = 3.0 $\pm$ 0.1 K.
Magnetization and TF-$\mu$SR measurements reveal that Pb2Pd is a type I
superconductor. Specific heat measurement indicates a moderate electron-phonon
coupling.

##  II Experimental Details

A Polycrystalline sample of Pb2Pd was first synthesized by adding starting
materials, Pb and Pd in a ratio of 2:1 in an evacuated sealed quartz tube. It
was then heated beyond the melting point (462$\degree$C). A single crystal was
prepared by the Bridgeman technique in which the polycrystalline sample was
placed in a conical quartz ampoule, heated to 570$\degree$C, dwelled for 30
hours and then slowly cooled to 456$\degree$C, followed by water quenching.
Phase purity and crystal structure characterization were performed using a
CuKα (1.5406 Å) equipped PANalytical powder X-ray diffractometer. The quality
control and orientation of single crystals were determined using a white beam
X-ray diffraction Laue instrument. The data was collected using North Star
software. Magnetization and AC susceptibility measurements were carried out in
a range of 1.8-10 K and 0-30 mT using a Quantum Design magnetic property
measurement system (MPMS 3). Electrical resistivity using a four-probe ac
technique in the range 1.9-300 K and specific heat (two-tau relaxation method)
measurements were performed on a Quantum Design Physical Property Measurement
System (PPMS).

Transverse field muon spin rotation (TF-$\mu$SR) measurements were carried out
to investigate the magnetic field distribution inside the sample. The data
were collected using the M15 beamline at TRIUMF, Centre for Molecular and
Material Science, Vancouver, Canada. The spectrometer incorporates a dilution
refrigerator which allows for measurements in the low-temperature range till
0.025 K and has a time resolution of 0.4 ns. The single crystal sample used
for our measurement was cut into plates before mounting on the cold finger so
that a large fraction of muon beam is covered. In TF geometry, the magnetic
field was applied parallel to the direction of muon beam whereas measurements
were taken with the initial muon spin arranged perpendicular to the field. The
direction of the applied magnetic field was perpendicular to the c-axis of
crystal. The $\mu$SRfit software package was used to analyze the $\mu$SR data.

##  III Results and Discussion

###  a Crystal Structure characterization

Figure 1:  Powder XRD pattern of Pb2Pd obtained at room temperature is shown
by red circles whereas black line represents the Rietveld refinement. Crystal
image (left-up) and crystal structure of Pb2Pd where blue circles represent Pb
atoms, whereas grey circles represent Pd atom (right-up). Back Laue X-ray
diffraction pattern of Pb2Pd single crystal (right most).

Room-temperature X-ray diffraction (XRD) pattern on powdered crystals is shown
in Fig. 1. Rietveld refinement confirms the phase purity, and crystal
structure as a body-centred tetragonal with space group I4/mcm. The obtained
lattice parameters are as follows: a = b = 6.859 Å and c = 5.838 Å P2P_PD1 .
All parameters including atomic and Wyckoff positions, cell volume etc.
obtained from refinement are summarized in the Table I. Fig. 1(right) also
shows the Laue pattern where the symmetric and bright spots indicate the
single-crystalline nature of Pb2Pd and correspond to (001) growth direction.

Table 1: Parameters obtained from Rietveld refinement

Structure | Tetragonal
---|---
Space group | I4/$mcm$
Lattice parameters |
a = b (Å) | 6.859(3)
c (Å) | 5.838(3)

Atom | Wyckoff position | x | y | z |
---|---|---|---|---|---
Pd1 | 4a | 0 | 0 | 0.250 |
Pb1 | 8h | 0.165 | 0.665 | 0 |

Figure 2:  Zero-field temperature variation of electrical resistivity in the
range 1.9 K $\leq T\leq$ 300 K where solid red line is a fit to parallel
resistor model. Inset (bottom right): A sharp superconducting transition at TC
= 3.0 K whereas top left inset shows the hall resistivity w.r.t. field ($\pm$
9T) at T = 50 K. Figure 3:  a) Temperature dependence of moment collected via
ZFCW and FCC under an applied magnetic field of 1 mT. b) Magnetization as a
function of applied field for 1.8 K $\leq$ T $\leq$ 2.8 K. c) Magnetization vs
magnetic field at 1.8 K of Pb2Pd.

###  b Normal and superconducting state properties

Fig. 2 shows the electrical resistivity of Pb2Pd as a function of temperature
obtained at zero field in the range 1.9-300 K. Inset shows a drop in
resistivity at a temperature Tc,onset = 3.0 $\pm$ 0.1 K with a transition
width $\Delta$T = 0.1 K which confirms superconductivity in this compound. The
residual resistivity ratio, (RRR = $\rho(300K)$/$\rho(6K)$) = 19 and the low
value of $\rho_{0}$ imply a low amount of disorder in the present sample. The
saturating type behaviour of $\rho(T)$ instead of linear T dependence at high
temperatures, allows this data to be fit by the parallel resistor model. This
possibility arises when the order of mean free path becomes short, equal to a
few inter-atomic spacing InterSpac . In such a case, the scattering cross-
section will no longer be linear in scattering perturbation. This leads to the
dominant temperature-dependent scattering mechanism is of electron-phonon type
interaction at high temperatures. Hence the resistivity will rise less rapidly
with temperature showing a saturating behaviour. This type of behaviour in
$\rho(T)$ was described by Wiesmann et al. Parallel  by the following
expression

$\rho(T)=\left[\frac{1}{\rho_{s}}+\frac{1}{\rho_{i}(T)}\right]^{-1}$ (1)

where $\rho_{s}$ is the temperature-independent saturation resistivity
attained at higher temperatures and $\rho_{i}(T)$ is given by

$\rho_{i}(T)=\rho_{i,0}+C\left(\frac{T}{\Theta_{D}}\right)^{5}\int_{0}^{\Theta_{D}/T}\frac{x^{5}}{(e^{x}-1)(1-e^{-x})}dx$
(2)

where $\rho_{i,0}$ represent temperature-independent residual resistivity due
to scattering from defects in the crystal structure whereas the second term is
temperature-dependent generalized BG resistivity in which C is a material-
dependent quantity and $\Theta_{D}$ is the Debye temperature obtained from
resistivity measurements. The red dashed line in Fig. 2 shows the best fit to
the data and yields $\rho_{0}$ = 2.14 $\pm$ 0.03 $\mu\Omega$ cm, C = 63 $\pm$
1 $\mu\Omega$ cm, $\rho_{0,s}$ = 378 $\pm$ 12 $\mu\Omega$ cm, and $\Theta_{D}$
= 116 $\pm$ 2 K which is close with the $\Theta_{D}$ obtained from specific
heat measurement (discussed later).

In order to extract the information regarding the carrier density of Pb2Pd,
hall resistivity, $\rho_{xy}$ as a function of magnetic field in the range
$\pm$ 9 T and at T = 50 K was measured as shown in the top left panel of Fig.
2. We have determined normal state hall coefficient, RH form the slope of a
linear fit to the $\rho_{xy}$(H) data in both field directions. The value
obtained for RH = -(2.11 $\pm$ 0.02)$\times 10^{-10}\Omega$ m T-1 and the
negative sign indicate towards the electron as the charge carriers. By using
RH = $-1/ne$, where $n$ is the carrier density, and e is the electronic
charge, we obtained $n$ = (2.96 $\pm$ 0.02)$\times 10^{28}m^{-3}$.
Theoretically, the carrier density is defined as n = Z/Vcell, where Z = 8 is
the number of conduction electrons per unit cell and Vcell = 0.275$\times
10^{-27}m^{3}$ for Pb2Pd, yielding a value of n = 2.90 $\times 10^{28}$ m-3
and is close to the experimentally obtained value.

Magnetization measurements has been performed on the single crystal oriented
along the (001) direction in both parallel and perpendicular magnetic field
configuration. It was obtained via two modes: zero-field cooled cooling (ZFCW)
and field cooled cooling (FCC) under an applied magnetic field of 1.0 mT which
confirms the superconducting nature of Pb2Pd displayed in Fig. 3(a). A strong
diamagnetic signal on entering the superconducting state at a transition
temperature of Tc,onset = 2.85 $\pm$ 0.10 K is observed, in agreement with
previous report P2P_Tc . There was no anisotropy in the observed TC in both
configurations so rest of the measurements were performed with magnetic field
perpendicular to the (001) direction. The Meissner volume fraction indicates
the complete expulsion of flux in the superconducting state of this compound.
Fig. 3(c) represents the M-H curve obtained at 1.8 K where it exhibits a
diamagnetic signal on re-entering the superconducting state. This type of
magnetic behaviour is expected for type I superconductors and has been
observed in other type I superconductors including AuBe AuBe , Ir2Ga9 IrGa
and LaRhSi3 LaRhSi . The absence of a sharp transition to the normal state is
accounted for the demagnetization factor. Low-field magnetization curves
obtained for different temperatures (as shown in Fig. 3(b)) together with
Specific heat, and resistivity data at various fields are used to depict the
critical field value at 0 K and fit according to

$H_{C}(T)=H_{C}(0)\left[1-\left(\frac{T}{T_{C}}\right)^{2}\right].$ (3)

The green solid curve shows the Ginzburg-Landau fit and the obtained value for
HC(0) = 33.6 $\pm$ 1.2 mT.

Figure 4:  Temperature dependence of the critical field from various
measurements. Inset: $\rho$(T) at different fields starting from 0 mT to 25
mT.

Specific heat measurements on the single crystal of Pb2Pd is carried out in
zero-field from 1.9 K to 10 K and are displayed in the inset of Fig. 5(a). The
jump in the specific heat at Tc,midpoint = 2.95 $\pm$ 0.10 K which is in
agreement with the other measurements confirms the bulk nature of the
superconductivity. Above the transition temperature, data is well described by
$\frac{C}{T}=\gamma_{n}+\beta_{3}T^{2}+\beta_{5}T^{4}$, shown by the solid
pink line in Fig. 5(a) and yields a value of Sommerfeld coefficient
$\gamma_{n}$ = 5.23 $\pm$ 0.39 mJ mol-1K-2, $\beta_{3}$ = 1.65 $\pm$ 0.03 mJ
mol-1K-4, and $\beta_{5}$ = 0.0129 $\pm$ 0.0005 mJ mol-1K-6. The Debye
temperature $\theta_{D}$ thetaD  is given by
$({12\pi^{4}RN}/{5\beta_{3}})^{1/3}$ where using R = 8.314 J mol-1K-2, N = 3,
and $\beta_{3}$ = 1.76 $\pm$ 0.10 mJ mol-1K-4, we have determined $\theta_{D}$
= 152 $\pm$ 1 K. Further, the Sommerfeld coefficient is related to the density
of states at the Fermi energy by $(\pi^{2}k_{B}^{2}D_{C}(E_{F})/{3})$ which
yields D${}_{C}(E_{F})$ = 2.2 $\pm$ 0.1 states eV-1f.u.-1. The value of
$\theta_{D}$ = 152 $\pm$ 1 K and Tc,midpoint = 2.95 $\pm$ 0.10 K are used to
estimate $\lambda_{e-ph}$ using McMillian’s relation given below McMillian

$\lambda_{e-ph}=\frac{1.04+\mu^{*}\mathrm{ln}(\theta_{D}/1.45T_{C})}{(1-0.62\mu^{*})\mathrm{ln}(\theta_{D}/1.45T_{C})-1.04}$
(4)

where $\mu^{*}$ represents screened repulsive Coulomb potential and usually
taken to be 0.13, yielding $\lambda_{e-ph}$ = 0.67 $\pm$ 0.08 which is
slightly higher than the other weakly coupled superconductors.

The electronic specific heat (Cel) contribution has been calculated by
subtracting the lattice and phononic contribution from the total specific heat
(C). The magnitude of specific heat jump $\Delta C_{el}$/$\gamma_{n}T_{C}$ =
1.7, which is higher than the BCS weak coupling limit 1.43. The value of the
same together with $\lambda_{e-ph}$ suggests a moderately coupled
superconductivity in Pb2Pd. The BCS expression SH_BCS  for the specific heat
below and above TC is described well with the normalized electronic specific
heat data and gives a value of $\alpha$ = $\Delta(0)/k_{B}T_{C}$ = 1.87 which
is also a bit higher than the BCS value, $\alpha$ = 1.76. Thus, the electronic
specific heat data provide compelling evidence of a moderately coupled
isotropic s-wave BCS superconductivity in Pb2Pd.

Figure 5:  Zero-field C/T variation with respect to T2 where solid line
represents a fit to $\frac{C}{T}=\gamma_{n}+\beta_{3}T^{2}+\beta_{5}T^{4}$.
Inset: Specific heat data exhibiting a superconducting transition at TC = 2.95
K. b) s-wave fitting to the normalized electronic specific heat data shown by
the solid line.

We now evaluate various superconducting parameters: $l$ (mean free path),
$\lambda_{L}$, $\xi_{0}$, and $\kappa$. Residual resistivity value along with
electron carrier density is used to estimate the electronic mean free path,
$l$ = v${}_{F}\tau$ m*&Drude  where vF and $\tau$ are the Fermi velocity and
the scattering time. Based on the Drude model, the Fermi velocity is given by
$\hbar k_{F}$/m∗ and scattering time $\tau^{-1}$ = ne${}^{2}\rho_{0}$/m∗ where
m∗ is the effective mass and n is the carrier density. The Fermi wave vector
is calculated by assuming a spherical Fermi surface, using the following
relation kF = (3$\pi^{2}$n)1/3, where $n$ = (2.96 $\pm$ 0.02)$\times
10^{28}m^{-3}$ was used and kF is estimated to be 0.957 Å-1. The Sommerfeld
coefficient from the specific heat ($\gamma_{n}$ = 5.33 mJmol-1K-2) and
carrier density are used to estimate the effective mass m∗ = ($\hbar
k_{F})^{2}\gamma_{n}$/$\pi^{2}nk_{B}^{2}$ m*&Drude , and it provides a value
of 2.59 me. By using all the calculated values of n, m∗, $\rho_{0}$ and kF, we
determined vF = 4.29$\times 10^{5}$ ms-1 and $l$ = 638 Å.

In order to confirm type I or type II superconductivity, we further
investigate London’s penetration depth ($\lambda_{L}$), BCS coherence length
($\xi_{0}$) and Ginzburg-Landau parameter ($\kappa_{GL}$). The penetration
depth $\lambda_{L}$ = $({m^{*}}/{\mu_{0}ne^{2}})^{1/2}$ is estimated to be 497
Å by using the value of m∗ = 2.59 me and n = 2.96$\times 10^{28}$ m-3. Within
BCS theory, the coherence length ($\xi_{0}$) is given by 0.18$\hbar
v_{F}$/kBTC, yielding a value of 2008 Å. The ratio $l$/$\xi_{0}$ = 0.32
concludes moderately dirty limit superconductivity in Pb2Pd. The value of
$\kappa_{GL}$ distinguishes between a type I and type II superconductor. In
dirty limit, $\kappa_{GL}$ is given by 0.715$\lambda_{L}$(0)/$l$ kGL  which is
found out to be 0.55, less than 1/$\sqrt{2}$, classifies Pb2Pd as a type I
superconductor. $\kappa_{GL}$ is further determined using the relation
7.49$\times 10^{3}\rho_{0}\sqrt{\gamma}_{nV}$ and found out to be 0.56 by
considering $\gamma_{nV}$, $\rho_{0}$ in cgs units, consistent with the value
obtained before. To determine Ginzburg-Landau coherence length $\xi(0)$ which
is defined as $\lambda_{eff}$/$\kappa_{GL}$ where $\lambda_{eff}$ =
$\lambda_{L}({1+\xi_{0}/l})^{1/2}$ = 1011.2 Å. Considering $\gamma_{nV}$,
$\rho_{0}$ and $\kappa_{GL}$ = 0.55, $\xi(0)$ is evaluated to be 1817 Å.

Figure 6:  The Fourier transform of $\mu$SR asymmetry representing the local
field distribution and asymmetry spectra at different fields, and
temperatures. (a)-(c) shows field distribution in the Meissner, intermediate
and normal state while (d)-(f) consists of corresponding asymmetry spectra.
The solid lines are fit to the data using Eq. (5). Figure 7:  a) Internal
field as a function of temperature in the normal region of Pb2Pd. b)
Superconducting volume fraction with respect to temperature at two different
fields.

Figure 6 shows the TF-$\mu$SR spectra and the corresponding internal field
distribution in i) Meissner state, ii) Intermediate state, and iii) normal
state. The data was collected after field cooling the sample from normal to
superconducting state in two different applied fields. The Fourier transform
extracted from TF-$\mu$SR spectra provides the information regarding the
probability distribution of an internal field in the intermediate state of a
type-I or a vortex state of a type-II superconductor. The presence of only one
peak at H = Happ = 100 G and T = 0.02 K corresponds to complete
superconducting or Meissner state where all the magnetic flux gets expelled
from the inside of a superconductor. The peak at H = 100 G represents the
background signal originating due to muons stopping in the sample holder. At T
= 2.25 K and Happ = 100 G shows the intermediate state of Pb2Pd. The asymmetry
spectra were analyzed using the following equation:

$\begin{split}A(t)=A_{0}[F(1-F_{S})\cos(\gamma_{\mu}H_{N}+\phi)\exp\left(-\frac{1}{2}\sigma_{N}^{2}t^{2}\right)\\\
+(1-F)\cos(\gamma_{\mu}H_{bg}+\phi)\exp\left(-\lambda_{bg}t\right)],\end{split}$
(5)

where A0 is the initial asymmetry, FS and F are the superconducting and total
fraction coming from the sample, HN is the internal field in the normal
regions of the sample, Hbg is the contribution from the background field which
originates due to muon stopping in the sample holder, $\phi$ is the phase
shift, $\sigma_{N}$ is the normal region relaxation rate, and $\lambda_{bg}$
is the background relaxation rate. The fit using Eq. (5) at T = 2.25 K and H =
100 G provide two values of field: 100 G and 123 G. The first value
corresponds to the applied field while the second one is taken as an estimate
of the critical field in the intermediate state of a type I superconductor
which is induced by the non-zero demagnetization effects. This causes some
part of the superconductor to experience a field greater than the applied
field and gives rise to the coexistence of superconducting and normal regions
even if the applied magnetic field is less than the critical field. Muons
landing in the superconducting region will not precess whereas muons implanted
in the normal region will precess with a magnetic field $\sim$ HC. The Fourier
transform shown in Fig. 6 (b) consists of two peaks in which one peak at H =
100 G corresponds to the muon stopping in the silver cold finger as well as
the normal state regions of the superconductor whereas the short peak centered
at Hint = 123 G corresponds to the intermediate state in Pb2Pd. This
unambiguously demonstrates that Pb2Pd is a type-I superconductor as the mixed
state in type-II superconductor will have a lower field value than the applied
field due to the establishment of flux line lattice.

The $\mu$SR asymmetry spectrum was also analyzed as a function of temperature
at 100 and 200 G fields. Figure 7 (a) shows the temperature dependence of the
internal field in the normal region of Pb2Pd. At T << TC and Happ < HC, the
internal field is equal to the critical field and as the temperature increases
it approaches the Happ value. Superconducting volume fraction with respect to
temperature is shown in Fig. 7 (b) and it shows an increasing behaviour as the
temperature decreases as it is expected for a type I superconductor.

Figure 8:  The Uemura plot showing the superconducting transition temperature $T_{c}$ vs the effective Fermi temperature $T_{F}$, where Pb2Pd is shown as a solid red marker. Other data points plotted between the blue solid lines is the different families of unconventional superconductors Umera_1 ; Umera_2 ; Umera_3 . Table 2: Superconducting and normal state parameters of Pb2Pd Parameters | unit | Pb2Pd
---|---|---
$T_{C}$ | K | 3.0 (1)
$H_{C}(0)$ | mT | 34 (1)
$\Delta C_{el}/\gamma_{n}T_{C}$ |  | 1.7 (1)
$\theta_{D}$ | K | 152 (1)
$\lambda_{e-ph}$ |  | 0.7 (1)
$\Delta(0)/k_{B}T_{C}$ |  | 1.9 (2)
$\xi(0)$ | Å | 1817 (280)
$m^{*}/m_{e}$ |  | 2.6 (1)
n | 1028m-3 | 2.96 (2)
$l$ | Å | 638 (60)
$\xi_{0}$ | Å | 2008 (220)
$\lambda_{L}$ | Å | 497 (20)
$l/\xi_{0}$ |  | 0.32
$k_{GL}$ |  | 0.55 (2)
$v_{f}$ | 105ms-1 | 4.3 (1)
$T_{F}$ | K | 13629
$T_{C}/T_{F}$ |  | 0.00019

To classify a superconductor as a conventional or unconventional, Uemura et
al. Umera_1 ; Umera_2 ; Umera_3  provides a classification based on their
$\frac{T_{C}}{T_{F}}$ ratio. This ratio falls in the range 0.01 $\leq$
$\frac{T_{C}}{T_{F}}$ $\leq$ 0.1 for many unconventional such as heavy-
fermion, iron-based superconductors and high TC cuprates shown between the two
blue solid lines. To calculate TF for a 3D system, the following relation is
used: $k_{B}T_{F}$ = ($\hbar k_{F})^{2}/{2m^{*}}$ where kF is the Fermi wave
vector. This gives TF = 13629 K and the ratio $\frac{T_{C}}{T_{F}}$ = 0.00019.
It places Pb2Pd away from the band of unconventional superconductors (as shown
by a green symbol). But at the same time, Pb2Pd is also away from the type I
conventional superconductors which are mostly pure elemental superconductors.

##  IV CONCLUSION

In this work, we have examined the superconducting properties of a new
superconductor Pb2Pd using specific heat, electric transport, magnetization,
and TF $\mu$SR measurements and found that it undergoes a sharp
superconducting transition at TC = 2.95 K. The value of residual resistivity,
full Meissner fraction and small width of superconducting transition in all
measurements indicate towards the high sample purity and crystallographic
order. The field dependence of magnetization together with other
superconducting parameters listed in Table II obtained from Sommerfeld
coefficient and the residual resistivity value within the framework of BCS
theory strongly revealed a type I and moderately dirty limit superconductivity
in Pb2Pd. Low-temperature specific heat measurements suggest isotropic s-wave
superconductivity with a moderate electron-phonon coupling. The microscopic
study of Pb2Pd using TF $\mu$SR measurement confirms type I superconductivity.
To establish a relation between the topological nature of the material and
superconducting ground-state properties, further microscopic measurements need
to be done.

##  V Acknowledgments

R. P. S. acknowledges Science and Engineering Research Board, Government of
India for the Core Research Grant CRG/2019/001028.

## References

  * (1) N. P. Armitage, E. J. Mele, and A. Vishwanath, Rev. Mod. Phys. 90, 015001 (2018).
  * (2) M. Z. Hasan and C. L. Kane, Rev. of Mod. Phys. 82, 3045 (2010).
  * (3) Y. Li and Z.-A. Xu, Adv. Quantum Technol. 2, 1800112 (2019).
  * (4) Jason Alicea, Yuval Oreg, Gil Refael, Felix von Oppen, Matthew P. A. Fisher, Nature Physics 7, 412-417 (2011).
  * (5) X.-J. Liu, C. L. M. Wong, K. T. Law, Phys. Rev. X 4, 021018 (2014).
  * (6) M. Sato and Y. Ando, Rep. Prog. Phys. 80, 076501 (2017).
  * (7) M. Uchida, M. Ide, M. Kawamura, K. S. Takahashi, Y. Kozuka, Y. Tokura, M. Kawasaki, Phys. Rev. B 99, 161111(R) (2019).
  * (8) N. Kimura, K. Ito, H. Aoki, S. Uji, T. Terashima, Phys. Rev. Lett. 98, 197001 (2007).
  * (9) E. M. Carnicom, W. W. Xie, T. Klimczuk, J. J. Lin, K. Górnicka, Z. Sobczak, N. P. Ong, and R. J. Cava, Sci. Adv. 4, 7969 (2018).
  * (10) K. Maki, S. Haas, D. Parker, H. Won, K. Izawa and Y. Matsuda, Europhys. Lett. 68, 720-725 (2004).
  * (11) K. Izawa, Y. Nakajima, J. Goryo, Y. Matsuda, S. Osaki, H. Sugawara, H. Sato, P. Thalmeier, K. Maki, Phys. Rev. Lett. 90, 117001 (2003).
  * (12) Z. F. Weng, J. L. Zhang, M. Smidman, T. Shang, J. Quintanilla, J. F. Annett, M. Nicklas, G. M. Pang, L. Jiao, W. B. Jiang, Y. Chen, F. Steglich, and H. Q. Yuan, Phys. Rev. Lett. 117, 027001 (2016).
  * (13) Y. Aoki, A. Tsuchiya, T. Kanayama, S. R. Saha, H. Sugawara, H. Sato, W. Higemoto, A. Koda, K. Ohishi, K. Nishiyama, and R. Kadono, Phys. Rev. Lett. 91, 067003 (2003).
  * (14) G. M. Luke, Y. Fudamoto, K. M. Kojima, M. I. Larkin, J. Merrin, B. Nachumi, Y. J. Uemura, Y. Maeno, Z. Q. Mao, Y. Mori, H. Nakamura and M. Sigrist, Nature 396, 658-660 (1998).
  * (15) A.D. Hillier, J. Quintanilla, B. Mazidian, J. F. Annett, and R. Cywinski, Phys. Rev. Lett. 109, 097001 (2012).
  * (16) Y. Xing, H. Wang, C.-K. Li, X. Zhang, J. Liu, Y. Zhang, J. Luo, Z. Wang, Y. Wang, L. Ling, M. Tian, S. Jia, J. Feng, X.-J. Liu, J. Wei, and J. Wang, npj Quantum Materials 1, 16005 (2016).
  * (17) L. M. Schoop, L. S. Xie, R. Chen, Q. D. Gibson, S. H. Lapidus, I.r Kimchi, M. Hirschberger, N. Haldolaarachchige, M. N. Ali, C. A. Belvin, T. Liang, J. B. Neaton, N. P. Ong, A. Vishwanath, and R. J. Cava, Phys. Rev. B 91, 214517 (2015).
  * (18) J.-F. Zhang, P.-J. Guo, M. Gao, K. Liu, and Z.-Y. Lu, Phys. Rev. B 99, 045110 (2019).
  * (19) Y. Hattori, K. Fukamichi, T. Goto, K. Suzuki, J. Phys. Soc. Jpn. 61, pp. 3845-3848 (1992).
  * (20) T. Zhang, Y. Jiang, Z. Song, H. Huang, Y. He, Z. Fang, H. Weng, C. Fang, arXiv:1807.08756v1 (2018).
  * (21) M.F. Gendron, R.E. Jones, J. Phys. Chem. Solids 23 405-406 (1962).
  * (22) E. E. Havinga, H. Damsma, P. Hokkeling, Journal of the Less-Common Metals 27, 169-186 (1972).
  * (23) Z. Fisk and G. W. Webb, Phys. Rev. Lett. 36, 1084 (1976).
  * (24) H. Wiesmann, M. Gurvitch, H. Lutz, A. K. Ghosh, B. Schwarz, M. Strongin, P. B. Allen, and J. W. Halley, Phys. Rev. Lett. 38, 782 (1977).
  * (25) E. E. Havinga, H. Damsma, P. Hokkeling, Journal of the Less-Common Metals 27, 281-291 (1972).
  * (26) D. Singh, A. D. Hillier, R. P. Singh, Phys. Rev. B 99, 134509 (2019)
  * (27) T. Shibayama, M. Nohara, H. A. Katori, Y. Okamoto, Z. Hiroi, and H. Takagi, J. Phys. Soc. Jpn. 76, 073708 (2007).
  * (28) V. K. Anand, A. D. Hillier, D. T. Adroja, Phys. Rev. B 83, 064522 (2011).
  * (29) D. Singh, J. A. T. Barker, A. Thamizhavel, D. McK. Paul, A. D. Hillier, and R. P. Singh, Phys. Rev. B 96, 180501(R) (2017).
  * (30) W. L. McMillan, Phys. Rev. 167, 331 (1968).
  * (31) D. Singh, A. D. Hillier, A. Thamizhave, R. P. Singh, Phys. Rev. B 94, 054515 (2016).
  * (32) M. Tinkham, Introduction to Superconductivity (McGraw-Hill, New York, 1996).
  * (33) M. Tinkham, Introduction to Superconductivity, 2nd ed. (Dover, Mineola, NY, 1996).
  * (34) Y. J. Uemura, V. J. Emery, A. R. Moodenbaugh, M. Suenaga, D. C. Johnston, A. J. Jacobson, J. T. Lewandowski, J. H. Brewer, R. F. Kiefl, S. R. Kreitzman, G. M. Luke, T. Riseman, C. E. Stronach, W. J. Kossler, J. R. Kempton, X. H. Yu, D. Opie, H. E. Schone, Phys. Rev. B 38, 909(R) (1988).
  * (35) Y. J. Uemura, G. M. Luke, B. J. Sternlieb, J. H. Brewer, J. F. Carolan, W. N. Hardy, R. Kadono, J. R. Kempton, R. F. Kiefl, S. R. Kreitzman, P. Mulhern, T. M. Riseman, D. Ll. Williams, B. X. Yang, S. Uchida, H. Takagi, J. Gopalakrishnan, A. W. Sleight, M. A. Subramanian, C. L. Chien, M. Z. Cieplak, Gang Xiao, V. Y. Lee, B. W. Statt, C. E. Stronach, W. J. Kossler, X. H. Yu, Phys. Rev. Lett. 62, 2317 (1989).
  * (36) Y. J. Uemura, L. P. Le, G. M. Luke, B. J. Sternlieb, W. D. Wu, J. H. Brewer, T. M. Riseman, C. L. Seaman, M. B. Maple, M. Ishikawa, D. G. Hinks, J. D. Jorgensen, G. Saito, H. Yamochi, Phys. Rev. Lett. 66, 2665 (1991).


# Retrieval-Augmented Neural Response Generation
Using Logical Reasoning and Relevance Scoring

Nicholas Thomas Walker
Norwegian Computing Center
Oslo, Norway
<EMAIL_ADDRESS>
&Stefan Ultes
University of Bamberg
Bamberg, Germany
<EMAIL_ADDRESS>
&Pierre Lison
Norwegian Computing Center
Oslo, Norway
<EMAIL_ADDRESS>

###### Abstract

Constructing responses in task-oriented dialogue systems typically relies on
information sources such the current dialogue state or external databases.
This paper presents a novel approach to knowledge-grounded response generation
that combines retrieval-augmented language models with logical reasoning. The
approach revolves around a knowledge graph representing the current dialogue
state and background information, and proceeds in three steps. The knowledge
graph is first enriched with logically derived facts inferred using
probabilistic logical programming. A neural model is then employed at each
turn to score the conversational relevance of each node and edge of this
extended graph. Finally, the elements with highest relevance scores are
converted to a natural language form, and are integrated into the prompt for
the neural conversational model employed to generate the system response.

We investigate the benefits of the proposed approach on two datasets (KVRET
and GraphWOZ) along with a human evaluation. Experimental results show that
the combination of (probabilistic) logical reasoning with conversational
relevance scoring does increase both the factuality and fluency of the
responses.

##  1 Introduction

Although Large Language Models (LLMs) are widely used for conversational
response generation, they still suffer from a number of shortcomings,
including their propensity to produce hallucinated content Ji et al. (2023).
Recent work has demonstrated how to exploit external information sources such
as knowledge bases (KBs) to improve the output of LLMs in various downstream
tasks (Yu et al., 2022a), including dialogue systems (Wang et al., 2021). A
promising approach is Retrieval-Augmented Generation (RAG), which operates by
first retrieving relevant information from external sources and then
augmenting the input provided to the LLM with this retrieved content (Lewis et
al., 2020). While RAG has been demonstrated to reduce hallucinations (Shuster
et al., 2021), LLMs are nonetheless easily distracted by irrelevant
information (Shi et al., 2023). For this reason, one should strike a balance
between providing the model with potentially useful information and avoiding
overloading it with too many spurious or irrelevant facts.

Moreover, while LLMs have recently shown some success at reasoning benchmarks
Bubeck et al. (2023), their ability to engage in multi-step reasoning remains
poor. In particular, Dziri et al. (2023) provide a systematic investigation of
the performance of LLMs on several compositional reasoning tasks, and find
that those models largely rely on pattern matching shortcuts and fall short of
exhibiting generic problem-solving skills.

Figure 1: General sketch of the proposed approach. The starting point is a
dialogue state represented as a knowledge graph that combines both background
information and various features extracted from the dialogue turns
(transcriptions, speakers, entity mentions). In Step 1, the facts representing
the nodes and edges of the graph are first extended with derived facts using
probabilistic logical programming. Those facts are then verbalized (converted
into natural language sentences) in Step 2, and a neural model is employed to
score their conversational relevance with regard to the current dialogue
context. Finally, the $k$ most relevant facts are included in Step 3 as part
of the prompt for the neural language model responsible for producing the
actual system response.

This paper presents a novel approach to retrieval augmented generation in
task-oriented dialogue systems that seeks to address those challenges.
Following Walker et al. (2022), we represent the background knowledge of the
system as a graph of dynamically updated facts representing the dialogue
state. This initial graph is first enriched at each turn with derived facts
inferred through probabilistic logical programming using a limited number of
rules, using ProbLog as a framework Fierens et al. (2015). The conversational
relevance of each fact is then scored using a neural model, based on various
features expressing both the conversational saliency of each entity and
semantic similarity between the fact and the recent dialogue history. The most
relevant facts are then converted into sentences and incorporated into the
input of the response generation model. Crucially, the relevance scoring model
and the response generation model are optimised jointly based on dialogue
examples. Figure 1 provides a general sketch of the approach.

The paper makes the following contributions:

  1. 1.

The use of probabilistic logical programming to extend the knowledge graph
representing the current dialogue state with derived facts.

  2. 2.

A neural scoring model that relies on both dialogue-level features (such as
recency) and semantic similarity to determine the most relevant nodes and
edges of this augmented graph.

  3. 3.

An empirical evaluation of the above approach using two dialogue datasets
(KVRET and GraphWOZ) and a human evaluation.

##  2 Related Work

Several papers have investigated the use of neural models to retrieve relevant
information from knowledge sources and integrate their results in response
generation. Dinan et al. (2018) distinguish between the tasks of knowledge
selection and response generation for knowledge-grounded dialogue agents. This
information may be structured or unstructured (Young et al., 2018; Zhao et
al., 2020), and generally consists of documents describing entities which may
be relevant to the dialogue. For an open-domain dialogue model, this
background information can be drawn from sources such as Wikipedia. In task-
oriented dialogue, relevant information will depend more heavily on the domain
of the dialogues. Previous work has also demonstrated the effectiveness of
jointly learning a language model with a knowledge retrieval model (Zhang et
al., 2021), simplifying the task of identifying relevant items without
labelled data.

External information, often in the form of KBs, is crucial to many dialogue
models (Ghazvininejad et al., 2018; Parthasarathi and Pineau, 2018; Zhang et
al., 2018; Madotto et al., 2018). Multiple approaches exist for combining
retrieval and generation to yield higher quality responses, such as by first
generating a response and subsequently refining it (Weston et al., 2018). The
model used by Peng et al. (2023) queries an LLM and evaluates the output for
factuality, and re-queries the system with feedback to elicit a more factual
response.

Thulke et al. (2021) propose an approach which samples a subset of the
background knowledge rather than optimizing over the entirety of it, a process
that we also integrate in our model training process. The Global-to-Local
Knowledge Selection model is an alternative which pre-selects information
across the whole of the background knowledge using topic transition vectors
(Ren et al., 2020). Meanwhile, He et al. (2021) proposed a model which
integrated information about system API calls to the retrieval model.

Numerous models make use of large, static, knowledge bases to augment language
models. The KETOD model (Chen et al., 2022) used Wikipedia data to enhance a
task-oriented dialogue system’s responses with information about entities in
the dialogue. Likewise, Kim et al. (2020) and Zhan et al. (2021) modelled
knowledge selection with a latent variable model, which have also shown strong
results for RAG in a zero-shot setting (Li et al., 2020). Moon et al. (2019)
investigated a graph decoder model using random walks over a knowledge graph
containing dialogue relevant information. Paranjape et al. (2021) made use of
a "guide" retriever model to use posterior information from responses to help
the retriever model learn from relevance in both the input and reference
output responses. Cai et al. (2019) made of of a "skeleton-guided" response
generator in a dialogue system.

RAG has also been used to improve common sense reasoning (Yu et al., 2022b),
or incorporate graphs of commonsense knowledge to the model (Zhang et al.,
2019a). Common sense knowledge in the SenticNet KB has also been used as a
source of knowledge for a dialogue model (Young et al., 2018), albeit without
logical reasoning over the graph. Liu et al. (2019b) explored a model which
used multi-hop reasoning to identify a relevant vertex in a graph of
"factoids" which are each associated with unstructured sentences. Other models
have made use of linguistic rule-based components to combine semantic
representations of the dialogue state with background knowledge to improve
empathetic responses and dialogue flow in task-oriented dialogue Smith et al.
(2011).

A hierarchical approach to knowledge grounded task-oriented dialogue was
presented by Lee and Jeong (2023), where the pipeline is composed of domain
identification, entity extraction, and a pre-trained language model to rank
relevant documents. Other work proposed a novel factuality-specific sampling
algorithm to improve LLM output (Lee et al., 2022), while Bonetta et al.
(2021) used k-nearest neighbors to find relevant information.

##  3 Approach

As illustrated in Figure 1, the approach proceeds in three steps.
Probabilistic logical programming is first employed to extend the initial
knowledge graph with new facts based on a small set of rules. A neural scoring
model then determines the relevance of those facts in the current dialogue
context. The most relevant facts are then included as part of the input to the
second neural model, which is responsible for the actual response generation.
The next sections describe those steps.

###  3.1 Dialogue state representation

Following Walker et al. (2022), we represent the current dialogue state (along
with other background information that might be relevant for response
generation) as a knowledge graph consisting of multiple entities connected by
relations. The graph is always grounded in a specific dialogue and
continuously evolves during the interaction, with new nodes and edges
representing dialogue turns, speakers, or entity mentions. The dynamic and
dialogue-specific nature of this knowledge graph stands in contrast with the
static KBs (based on e.g. Wikipedia or similar sources) typically used in
knowledge-grounded generation.

To account for uncertainties associated with noisy or partial observations
(such as ASR transcriptions of user utterances or ambiguous referential
links), both node attributes and labelled edges may be associated with
probabilities.

###  3.2 Probabilistic logical programming

To explicitly reason over this graph, we rely on the probabilistic logical
programming language ProbLog Kimmig et al. (2011); Fierens et al. (2015). We
assign each node to a unique identifier and represent the node attributes and
edges as (ground) logical predicates, as illustrated in Figure 1. Node
attributes and edges associated with a probability $<1$ are expressed as
probabilistic facts.

####  3.2.1 ProbLog

A ProbLog program consists of two parts: a set of ground probabilistic facts,
and a logic program, expressed as a set of logical clauses. The clauses may be
themselves associated with probabilities. ProbLog also allows for the
definition of “annotated disjunctions” where mutually exclusive facts are
coupled with a discrete probability distribution. Syntax-wise, ProbLog is a
probabilistic extension of Prolog and supports both probabilistic and
inductive reasoning. Given a set of logical rules and ground facts, ProbLog
provides inference algorithms to efficiently query the probability of one or
more predicates. This inference is done by converting the facts and logical
program to a compact encoding such as Sentential Decision Diagrams Vlasselaer
et al. (2014) and then running weighted model counting Chavira and Darwiche
(2008) on this compiled representation.

####  3.2.2 Entity linking rules

An important task in goal-oriented dialogues is to connect entity mentions to
the actual entities present in the KB. For instance, if “Jill Martinez” is
mentioned by the user, this mention must be linked to the actual node
($p_{1}$) for that person in the KB. Entity mentions may correspond to named
entities, but may also take the form of pronouns (”she”) or generic noun
phrases (“the meeting”).

We first detect entity mentions in user utterances using a neural sequence
labelling model fine-tuned on labelled, in-domain data from a pretrained
ROBERTA model Liu et al. (2019a). A small set of probabilistic ProbLog rules
is then employed to determine the most likely reference among the entities in
the knowledge graph. Those rules take advantage of both edit distance metrics
and recency measures Walker et al. (2022). Each rule is attached to a
probability reflecting its strength. Those probabilities are estimated
empirically from partial interpretations on the training data, following the
approach described in Gutmann et al. (2011). After applying those entity
linking rules, the outcome is then written back to the knowledge graph as
probabilistic refers_to edges linking each observed mention to the entity it
refers to.

####  3.2.3 Commonsense rules

Consider a scenario where a task-oriented dialogue system must answer a user
question:

> "What events do I have today?"

Assuming the knowledge graph contains basic information about calendar events
such as their date, time and attendees, answering this question rests upon
multiple reasoning steps. As multi-step reasoning remains a challenging task
for language models Liu et al. (2023), we specify a small number of
commonsense reasoning rules to automatically derive new facts from the current
dialogue state. For the above example, the connection between dates and events
in context can be made explicit with the following rule:

> person(P), event(E), attendee(E,P), date(E,D), date(today,D)
> $\implies$ attending_today(E,P)

A second example of a logical rule is as follows:

> room(R), $\neg$(event(E), location(E,R), date(E,D), date(today,D),
> start_time(E,ST), end_time(E,ET), time_between(T,ST,ET,1))
> $\implies$ room_available_today(R,T)

The above rule simply states that a room $R$ is available today at a given
time $T$ if no event is scheduled at that time in that room.

The goal of those commonsense rules is to deduce facts that may provide useful
information to the response generation model. Those logically derived facts
will typically correspond to information that may be queried by the users,
such as a person’s agenda for today or the availability of a room at a given
time. To avoid deriving too many spurious or irrelevant facts, we only query
ProbLog for facts pertaining to entities recently mentioned in the dialogue
history. For our experiments, we query entities mentioned in the current turn.

After applying both entity linking and commonsense rules, the facts are
converted to a natural language verbalization. Each predicate is associated
with a handcrafted template which creates a natural language form of the fact.
For example, a person defined by the fact person(p_123) with a name
name(p_123, "Lisa Wilson") can be verbalized as Lisa Wilson is a person.

###  3.3 Relevance Scoring

The second component of the proposed approach is a neural model that scores
the relevance of the verbalized facts (including both the intial ones as well
as the ones derived through logical reasoning). Given a dialogue history
$x=[u_{1},...u_{n}]$ corresponding to a list of utterances and a set of
verbalized facts $Z$, the relevance scoring model expresses the probability
$P(z|x)$ that the fact $z\in Z$ is relevant for responding to $x$.

The model is expressed as a simple feedforward neural network based on the
following inputs:

  1. 1.

Semantic similarity measures between $z$ and $x$, using the cosine similarity
between the embedding of the verbalized fact $z$ and the embedding of the most
recent $k$ utterances in the dialogue history $x$ (concatenated if $k>1$):

$sim(z,x)=\frac{\textit{Enc}(z)\cdot\textit{Enc}(x_{[n-k:n]})}{\|\textit{Enc}(z)\|\
\|\textit{Enc}(x_{[n-k:n]}\|}$

The Enc embeddings are obtained with a sentence-BERT model (Reimers and
Gurevych, 2019) optimized for semantic search and question
answering111https://huggingface.co/sentence-transformers/multi-qa-
MiniLM-L6-cos-v1. The cosine similarity is computed for $k=1$ and 2.

  2. 2.

BM25 information retrieval scores Robertson et al. (2009) using the verbalized
facts as database and the user utterance as a query.

  3. 3.

Recency score expressing whether the fact $z$ pertains to a recently mentioned
entity. This score relies on the refers_to predicates derived from entity
linking and captures the conversational saliency of entities and facts related
to them. For instance, facts related to $p_{1}$ in Fig. 1 are salient since
the person is mentioned in the last utterance.

The relevance model $P(z|x)$ is trained jointly with the response generation.
Concretely, we define the probability of a response $y$ given a dialogue
history $x$ as:

$P(y|x)=\sum_{z\in Z}P(y|x,z)P(z|x)$ (1)

where $P(y|x,z)$ is provided by the response generation model (see below), and
express the probability of a response $y$ given a prompt concatenating the
dialogue history $x$ and fact $z$, and $P(z|x)$ express the relevance of $z$
for $x$. The relevance model $P(z|x)$ is then optimized by back-propagating
the cross-entropy loss of Eq. (1) using a training set of dialogue examples.
Intuitively, a fact will therefore be deemed as relevant if its inclusion in
the prompt makes it relatively easier for the generation model to produce the
correct response. To ensure the inference remains efficient, Eq. (1) is
simplified by sampling the $K$ most relevant facts instead of marginalizing
over all possible facts.

###  3.4 Response Generation

The final step of our approach is to generate a response $y$ based on both the
current dialogue history $x$ and a set of relevant facts $z_{1}:z_{K}$, where
$K$ denotes the number of facts (sorted by relevance) to include in the input
prompt. Any pretrained language model can be employed for this task. We rely
for our experiments on both the GODEL model (Peng et al., 2022) which is
specifically designed for goal-oriented dialog as well the generic GPT 3.5
model Brown et al. (2020).

##  4 Evaluation

We evaluate the performance of the proposed approach on two existing dialogue
datasets along with a human evaluation. We present below the experimental
design, and discuss the results.

###  4.1 Datasets

Model | Dev | Test
---|---|---
BLEU | METEOR | BERTScore | BLEU | METEOR | BERTScore
$\textrm{GODEL}_{\textrm{NoFacts}}$ | 0.17 | 0.37 | 0.89 | 0.11 | 0.36 | 0.88
$\textrm{GODEL}_{\textrm{AllFacts}}$ | 0.14 | 0.38 | 0.88 | 0.13 | 0.33 | 0.88
$\textrm{GODEL}_{\textrm{Relevance}}$ | 0.18 | 0.38 | 0.89 | 0.14 | 0.33 | 0.88
$\textrm{GODEL}_{\textrm{Relevance+Logic}}$ | 0.17 | 0.37 | 0.89 | 0.16 | 0.35 | 0.88
$\textrm{GPT}_{\textrm{NoFacts}}$ | 0.08 | 0.35 | 0.88 | 0.06 | 0.32 | 0.87
$\textrm{GPT}_{\textrm{AllFacts}}$ | 0.07 | 0.36 | 0.88 | 0.06 | 0.32 | 0.87
$\textrm{GPT}_{\textrm{Relevance}}$ | 0.07 | 0.35 | 0.88 | 0.06 | 0.35 | 0.87
$\textrm{GPT}_{\textrm{Relevance+Logic}}$ | 0.07 | 0.37 | 0.88 | 0.07 | 0.36 | 0.87
Table 1: Results with reference-based metrics on the development and test set of GraphWOZ. Model | Dev | Test
---|---|---
BLEU | METEOR | BERTScore | BLEU | METEOR | BERTScore
$\textrm{GODEL}_{\textrm{NoFacts}}$ | 0.18 | 0.45 | 0.91 | 0.11 | 0.36 | 0.91
$\textrm{GODEL}_{\textrm{Relevance}}$ | 0.18 | 0.42 | 0.91 | 0.16 | 0.41 | 0.91
$\textrm{GODEL}_{\textrm{Relevance+Logic}}$ | 0.20 | 0.43 | 0.91 | 0.17 | 0.42 | 0.91
Table 2: Results with reference-based metrics on the development and test set
of KVRET.

#### GraphWOZ Walker et al. (2022)

GraphWOZ contains task-oriented dialogue with dialogues discussing people and
places in a fictional organization to schedule meetings and discover
information. Each dialogue is paired with synthetically generated calendar
events. The graphs contain fictive people, rooms, and events along with
dialogue information such as utterances and mentions of entities in
utterances.

Although the calendar information is synthetically generated, generation of
new dialogue utterances with calendar information may not accurately reflect
real system-human interaction. In consideration of this factor and to
compensate for the small amount of training data, we augment the GraphWOZ
training set with modified versions of the original dialogues where entities
are replaced in both the knowledge base and dialogue history. We replace
entity names with randomly sampled replacements, and the dates and times of
the dialogues and events in the KBs are replaced such that relative terms such
as "today", "tomorrow", "morning", and "afternoon" remain consistent in the
modified dialogue.

The entity linking and commonsense rules for this dataset are provided in the
Appendix.

#### KVRET Eric et al. (2017)

This dataset contains task-oriented dialogue in three domains: weather,
navigation, and calendar scheduling. Each type of dialogue contains associated
KB information representing objects of interest which are discussed in the
dialogue. The knowledge bases in KVRET were created by randomly sampling
attribute values for defined slots according to the domain. We convert these
KBs into a ProbLog program along with the user utterances and mentions of
objects. For simplicity, we take a string equality match of a substring in the
utterance to an object in the KB as a refers_to relation.

We rely on three simple ProbLog rules for this dataset. The weather domain has
a rule which determines "today" along the weather for a particular day and
location. In the calendar scheduling domain, we provide a rule to handle
location names with multiple potential referents. Lastly, for the navigation
domain we define a rule comparing the distance from the user to two points of
interests and determining which one is closest. Each of these rules therefore
makes information explicitly available to the system which would be
unavailable from context or otherwise require logical inference that an LLM is
not optimized to perform.

Model | Coherence | Groundedness | Naturalness | Understandability
---|---|---|---|---
$\textrm{GODEL}_{\textrm{NoFacts}}$ | 0.946 | 0.908 | 0.871 | 0.864
$\textrm{GODEL}_{\textrm{AllFacts}}$ | 0.975 | 0.943 | 0.903 | 0.896
$\textrm{GODEL}_{\textrm{Relevance}}$ | 0.916 | 0.878 | 0.862 | 0.855
$\textrm{GODEL}_{\textrm{Relevance+Logic}}$ | 0.979 | 0.951 | 0.868 | 0.861
$\textrm{GPT}_{\textrm{NoFacts}}$ | 0.951 | 0.880 | 0.943 | 0.938
$\textrm{GPT}_{\textrm{AllFacts}}$ | 0.952 | 0.878 | 0.931 | 0.925
$\textrm{GPT}_{\textrm{Relevance}}$ | 0.969 | 0.912 | 0.935 | 0.931
$\textrm{GPT}_{\textrm{Relevance+Logic}}$ | 0.949 | 0.883 | 0.928 | 0.922
Table 3: UniEval Score (Reference-free) on the test set of GraphWOZ. Model | Dev | Test
---|---|---
Hallucinations | Retrieval Errors | Hallucinations | Retrieval Errors
$\textrm{GPT}_{\textrm{NoFacts}}$ |  34 $(18\%)$ | 17 $(9\%)$ | 32 $(17\%)$ | 16 $(8\%)$
$\textrm{GPT}_{\textrm{AllFacts}}$ |  23 $(13\%)$ | 11 $(6\%)$ | 23 $(13\%)$ |  20 $(11\%)$
$\textrm{GPT}_{\textrm{Relevance}}$ |  21 $(12\%)$ | 13 $(7\%)$ | 24 $(13\%)$ | 16 $(8\%)$
$\textrm{GPT}_{\textrm{Relevance+Logic}}$ | 15 $(8\%)$ | 14 $(8\%)$ | 25 $(14\%)$ |  9 $(5\%)$
Table 4: Turns containing hallucinations and retrieval errors (GraphWOZ, 181
turns in Dev, 180 in Test)

###  4.2 Models

We experiment with the four following types of response generation models:

NoFacts

Generation model that does not use the knowledge graph at all and produce a
response based on the current dialogue history.

AllFacts+Logic

Generation model using all verbalized facts (including logically derived
ones), without relevance scoring. These facts are shuffled and truncated to
fit into the context window of the generation model.

Relevance

Generation model using the initial facts from the knowledge graph (but without
logically derived ones) ranked using the relevance scoring model. The 10 most
relevant facts are then prepended to the prompt.

Relevance+Logic

Generation model using both the initial facts and the logically derived ones,
along with the relevance scoring model to select the 10 most relevant facts.

We experiment with two generative models: the encoder-decoder GODEL (Peng et
al., 2022), which is pre-trained on large volumes of multi-turn dialogues, and
the recent GPT-3.5 model Brown et al. (2020). We first test the response
generation capabilities of GPT-3.5 with the three different approaches on
GraphWOZ. For each turn, we provide the system with the dialogue history up to
the current turn. When using all facts, background knowledge is added as a
single document in the initial prompt, as repeating the entire document of the
facts at each turn would result in truncation of the dialogue history without
adding additional information.

###  4.3 Metrics

For both GraphWOZ and KVRET, we use standard evaluation metrics such BLEU,
METEOR and the averaged BERTScore F1 Zhang et al. (2019b). We also use the
recently introduced UniEval Zhong et al. (2022), a reference-free metric which
has been shown to correlate well with human judgments.

We also evaluate the factuality of the responses by manually annotating them
with two types of error. The first error type are hallucinations, which we
define as either (a) a statement that contradicts the KB, including
contradictions implied by the dialogue context ; (b) a statement referring to
a nonexistent entity in the KB ; or (c) a statement describing a calendar
action that would create a calendar conflict if enacted.

The second type of error occurs when the system fails to retrieve information
it has access to or should have access to. We denote this type of error as a
retrieval error, and we annotate a turn as containing a retrieval error if it
contains:

  * •

A statement denying having access to information which exists in the calendar,
or does not answer a question while answering another ;

  * •

A confirmation of inviting a person to a meeting without specifying whom ;

  * •

A request to confirm a piece of information that was already confirmed.

###  4.4 Results

Tables 1 and 2 respectively provide the results of the reference-based metrics
on GraphWOZ and KVRET. We observe modest differences between the models,
particularly on the development set. This is expected as those metrics are not
really suited to measure the extent to which a language model produces factual
answers. We do nevertheless observe a slight increase in BLEU score when
logic-derived facts are added. This result is consistent in both GraphWOZ and
KVRET. The differences in UniEval, shown in Table 3 were also small, with the
model using all facts performing comparably to the Relevance+Logic model.

However, for the manually annotated hallucinated and retrieval errors in Table
4 (only annotated for the GPT models), we do observe a substantial reduction
in overall errors for the Relevance+Logic model compared to the baselines.
Particularly in the development set, the model using all facts (without
relevance scoring) frequently produce hallucinations, such as incorrect
attributes of various people and events.

###  4.5 Error Analysis

Qualitatively, the errors produced by the models also differed. The GPT models
using selected facts were more likely to provide all available information,
whether requested or not. In some cases, the model produced statements
connecting unrelated facts and implying an error, for example:

> User: Hi robot, my name is Curtis Williams, please let me know when the
> meeting called, deliverables team meeting. [sic]
> System: Hello Curtis! The deliverables team meeting will be held on Friday,
> April 08 at 11:30 AM in the Beta conference room. Annette Harding is
> organizing the meeting on behalf of the Mathematics group. You and Angela
> Jimenez are scheduled to attend the meeting. The meeting is scheduled to
> last for 30 minutes until the Niches team meeting starts at 12:00. Michael
> Glover is attending the Niches team meeting and Annette Harding is also
> attending the Deliverables team meeting with you. Is there anything else you
> need assistance with?

Although the general content is factual, the niches team meeting was on a
different day despite the implication of the phrasing. The GPT model with all
facts responded erroneously at the same turn as:

> Alright Curtis, I have checked your calendar and it looks like there is a
> "Deliverables Team Meeting" scheduled for tomorrow at 2:00 pm in room 619.
> Would you like me to add any notes or reminders for the meeting?"

While more concise, the system incorrectly states the time of the event
(11:30) and date (today), as well as the location (Beta). This apparently
arises from misattribution of background facts relating to other events
scheduled across times and locations.

###  4.6 Human Evaluation

#### Experimental setup

To confirm the performance of the approach in actual interactions, we also
conduct a human evaluation in the context of a receptionist scenario similar
to GraphWOZ, where the participant interacts with the system to find
information about entities and schedule events. We recruited 16 participants
including students from the university and employees to interact with the
dialogue systems through text. Users were instructed to interact with the
system to accomplish a task, mark the conversation as finished when either the
task appeared complete or the dialogue system unrecoverably failed. After each
dialogue, the users were prompted to rate the dialogue on a scale of 1 to 5
for two statements, where 1 is "Never", 2 is "Mostly Not", 3 "Sometimes", 4
"Mostly", and 5 "Always":

  * •

The system responded to me in a conversationally relevant way.

  * •

The system successfully completed my task and gave me the information I asked
for.

Users were instructed to repeat this process for 30 minutes, with priority
given to conversation quality. For each dialogue, a model was randomly
selected and a random dialogue state similar to the GraphWOZ dataset was
generated for the dialogue. A task was then randomly generated from a set of
task templates involving fictive entities. The collected dialogues were then
manually annotated for both hallucinations and retrieval errors. Because the
total number of turns varied from model to model, we evaluate the proportion
of turns which contain hallucinations and retrieval errors.

#### Results

Model | Task | Appropriateness
---|---|---
$\textrm{GODEL}_{\textrm{None}}$ | 3.35 | 3.07
$\textrm{GODEL}_{\textrm{All}}$ | 3.75 | 3.63
$\textrm{GODEL}_{\textrm{Logic}}$ | 4.08 | 3.75
$\textrm{GPT}_{\textrm{None}}$ | 4.18 | 4.59
$\textrm{GPT}_{\textrm{All}}$ | 4.09 | 4.32
$\textrm{GPT}_{\textrm{Logic}}$ | 4.37 | 4.11
Table 5: Average participant scores for the model task completion and appropriateness criteria. Model | Hallucinations | Retrieval | #
---|---|---|---
|  | errors |
$\textrm{GODEL}_{\textrm{None}}$ | 0.17 | 0.41 | 105
$\textrm{GODEL}_{\textrm{All}}$ | 0.24 | 0.32 | 84
$\textrm{GODEL}_{\textrm{Logic}}$ | 0.22 | 0.21 | 67
$\textrm{GPT}_{\textrm{None}}$ | 0.32 | 0.14 | 88
$\textrm{GPT}_{\textrm{All}}$ | 0.22 | 0.39 | 117
$\textrm{GPT}_{\textrm{Logic}}$ | 0.20 | 0.12 | 132
Table 6: Proportion of system responses containing either hallucinations or
retrieval errors in the human evaluation experiments. The last column
indicates the total number of system utterances from all dialogues with that
model.

As for the GraphWOZ results, the human interaction experiments indicate a
reduction in the proportion of turns with hallucinations or retrieval errors,
as shown in Table 6. This reduction is observed for both model types when the
logic-enhanced relevance scoring model was used.

The participant scores in Table 5 ranked the models which used the relevance
scored facts highest. While the GODEL model using the relevant facts scored
higher in appropriateness, the opposite pattern is observable in the GPT
models, although the scores remain relatively high. As not every participant
interacted with every model, differences in scoring between individual
participants cannot be discounted as a factor impacting these results, thus a
larger study would be beneficial.

##  5 Conclusion

This paper presented a novel approach to retrieval-augmented response
generation in task-oriented dialogue systems. The approach relies a dynamic
knowledge graph representing the dialogue state, which is enriched at each
turn with facts derived from a small set of rules specified in the ProbLog
language. Those facts are then ranked by relevance using a dedicated scoring
model which accounts for both the semantic similarity and conversational
saliency of each fact. The most relevant facts are then incorporated to the
background knowledge provided as input to the response generation model.

We provide experimental results showing that the combination of logical
reasoning with a relevance scoring model leads to more factual responses. In
particular, the logical rules seem to assist the generation model’s ability to
provide responses grounded in multi-step reasoning based on the available
background knowledge.

The proportion of errors remains, however, relatively high, likely due to the
very limited number of dialogues available for training in GraphWOZ and KVRET.
Future work will focus on evaluating the potential of this approach in other
(and potentially broader) dialogue domains.

## References

  * Bonetta et al. (2021) Giovanni Bonetta, Rossella Cancelliere, Ding Liu, Paul Vozila, et al. 2021.  Retrieval-augmented transformer-xl for close-domain dialog generation.  In _The International FLAIRS Conference Proceedings_ , volume 34, pages 1–4. Florida Online Journals.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Bubeck et al. (2023) Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. 2023.  Sparks of artificial general intelligence: Early experiments with gpt-4.  _arXiv preprint arXiv:2303.12712_.
  * Cai et al. (2019) Deng Cai, Yan Wang, Wei Bi, Zhaopeng Tu, Xiaojiang Liu, and Shuming Shi. 2019.  Retrieval-guided dialogue response generation via a matching-to-generation framework.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 1866–1875, Hong Kong, China. Association for Computational Linguistics.
  * Chavira and Darwiche (2008) Mark Chavira and Adnan Darwiche. 2008.  On probabilistic inference by weighted model counting.  _Artificial Intelligence_ , 172(6-7):772–799.
  * Chen et al. (2022) Zhiyu Chen, Bing Liu, Seungwhan Moon, Chinnadhurai Sankar, Paul Crook, and William Yang Wang. 2022.  KETOD: Knowledge-enriched task-oriented dialogue.  In _Findings of the Association for Computational Linguistics: NAACL 2022_ , pages 2581–2593, Seattle, United States. Association for Computational Linguistics.
  * Dinan et al. (2018) Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2018.  Wizard of wikipedia: Knowledge-powered conversational agents.  In _International Conference on Learning Representations_.
  * Dziri et al. (2023) Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jian, Bill Yuchen Lin, Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena D Hwang, et al. 2023.  Faith and fate: Limits of transformers on compositionality.  _arXiv preprint arXiv:2305.18654_.
  * Eric et al. (2017) Mihail Eric, Lakshmi Krishnan, Francois Charette, and Christopher D. Manning. 2017\.  Key-value retrieval networks for task-oriented dialogue.  In _Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue_ , pages 37–49, Saarbrücken, Germany. Association for Computational Linguistics.
  * Fierens et al. (2015) Daan Fierens, Guy Van den Broeck, Joris Renkens, Dimitar Shterionov, Bernd Gutmann, Ingo Thon, Gerda Janssens, and Luc De Raedt. 2015.  Inference and learning in probabilistic logic programs using weighted boolean formulas.  _Theory and Practice of Logic Programming_ , 15(3):358–401.
  * Ghazvininejad et al. (2018) Marjan Ghazvininejad, Chris Brockett, Ming-Wei Chang, Bill Dolan, Jianfeng Gao, Wen-tau Yih, and Michel Galley. 2018.  A knowledge-grounded neural conversation model.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 32.
  * Gutmann et al. (2011) Bernd Gutmann, Ingo Thon, and Luc De Raedt. 2011.  Learning the parameters of probabilistic logic programs from interpretations.  In _Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2011, Athens, Greece, September 5-9, 2011. Proceedings, Part I 11_ , pages 581–596. Springer.
  * He et al. (2021) Huang He, Hua Lu, Siqi Bao, Fan Wang, Hua Wu, Zhengyu Niu, and Haifeng Wang. 2021\.  Learning to select external knowledge with multi-scale negative sampling.  _arXiv preprint arXiv:2102.02096_.
  * Ji et al. (2023) Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023.  Survey of hallucination in natural language generation.  _ACM Computing Surveys_ , 55(12):1–38.
  * Kim et al. (2020) Byeongchang Kim, Jaewoo Ahn, and Gunhee Kim. 2020.  Sequential latent knowledge selection for knowledge-grounded dialogue.  In _International Conference on Learning Representations_.
  * Kimmig et al. (2011) Angelika Kimmig, Bart Demoen, Luc De Raedt, Vitor Santos Costa, and Ricardo Rocha. 2011.  On the implementation of the probabilistic logic programming language problog.  _Theory and Practice of Logic Programming_ , 11(2-3):235–262.
  * Lee and Jeong (2023) Hayoung Lee and Okran Jeong. 2023.  A knowledge-grounded task-oriented dialogue system with hierarchical structure for enhancing knowledge selection.  _Sensors_ , 23(2):685.
  * Lee et al. (2022) Nayeon Lee, Wei Ping, Peng Xu, Mostofa Patwary, Pascale N Fung, Mohammad Shoeybi, and Bryan Catanzaro. 2022.  Factuality enhanced language models for open-ended text generation.  _Advances in Neural Information Processing Systems_ , 35:34586–34599.
  * Lewis et al. (2020) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020.  Retrieval-augmented generation for knowledge-intensive nlp tasks.  _Advances in Neural Information Processing Systems_ , 33:9459–9474.
  * Li et al. (2020) Linxiao Li, Can Xu, Wei Wu, Yufan Zhao, Xueliang Zhao, and Chongyang Tao. 2020.  Zero-resource knowledge-grounded dialogue generation.  _Advances in Neural Information Processing Systems_ , 33:8475–8485.
  * Liu et al. (2023) Hanmeng Liu, Ruoxi Ning, Zhiyang Teng, Jian Liu, Qiji Zhou, and Yue Zhang. 2023\.  Evaluating the logical reasoning ability of chatgpt and gpt-4.  _arXiv preprint arXiv:2304.03439_.
  * Liu et al. (2019a) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019a.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Liu et al. (2019b) Zhibin Liu, Zheng-Yu Niu, Hua Wu, and Haifeng Wang. 2019b.  Knowledge aware conversation generation with explainable reasoning over augmented graphs.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 1782–1792, Hong Kong, China. Association for Computational Linguistics.
  * Madotto et al. (2018) Andrea Madotto, Chien-Sheng Wu, and Pascale Fung. 2018.  Mem2Seq: Effectively incorporating knowledge bases into end-to-end task-oriented dialog systems.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1468–1478, Melbourne, Australia. Association for Computational Linguistics.
  * Moon et al. (2019) Seungwhan Moon, Pararth Shah, Anuj Kumar, and Rajen Subba. 2019.  OpenDialKG: Explainable conversational reasoning with attention-based walks over knowledge graphs.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 845–854, Florence, Italy. Association for Computational Linguistics.
  * Paranjape et al. (2021) Ashwin Paranjape, Omar Khattab, Christopher Potts, Matei Zaharia, and Christopher D Manning. 2021.  Hindsight: Posterior-guided training of retrievers for improved open-ended generation.  In _International Conference on Learning Representations_.
  * Parthasarathi and Pineau (2018) Prasanna Parthasarathi and Joelle Pineau. 2018.  Extending neural generative conversational model using external knowledge sources.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 690–695, Brussels, Belgium. Association for Computational Linguistics.
  * Peng et al. (2022) Baolin Peng, Michel Galley, Pengcheng He, Chris Brockett, Lars Liden, Elnaz Nouri, Zhou Yu, Bill Dolan, and Jianfeng Gao. 2022.  Godel: Large-scale pre-training for goal-directed dialog.  _arXiv preprint arXiv:2206.11309_.
  * Peng et al. (2023) Baolin Peng, Michel Galley, Pengcheng He, Hao Cheng, Yujia Xie, Yu Hu, Qiuyuan Huang, Lars Liden, Zhou Yu, Weizhu Chen, et al. 2023.  Check your facts and try again: Improving large language models with external knowledge and automated feedback.  _arXiv preprint arXiv:2302.12813_.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-bert: Sentence embeddings using siamese bert-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics.
  * Ren et al. (2020) Pengjie Ren, Zhumin Chen, Christof Monz, Jun Ma, and Maarten de Rijke. 2020.  Thinking globally, acting locally: Distantly supervised global-to-local knowledge selection for background based conversation.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pages 8697–8704.
  * Robertson et al. (2009) Stephen Robertson, Hugo Zaragoza, et al. 2009.  The probabilistic relevance framework: Bm25 and beyond.  _Foundations and Trends® in Information Retrieval_ , 3(4):333–389.
  * Shi et al. (2023) Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed Chi, Nathanael Schärli, and Denny Zhou. 2023.  Large language models can be easily distracted by irrelevant context.
  * Shuster et al. (2021) Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. 2021.  Retrieval augmentation reduces hallucination in conversation.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 3784–3803, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Smith et al. (2011) Cameron Smith, Nigel Crook, Daniel Charlton, Johan Boye, Raul Santos De La Camara, Markku Turunen, David Benyon, Björn Gambäck, Oli Mival, Nick Webb, et al. 2011.  Interaction strategies for an affective conversational agent.  _Presence_ , 20(5):395–411.
  * Thulke et al. (2021) David Thulke, Nico Daheim, Christian Dugast, and Hermann Ney. 2021.  Efficient retrieval augmented generation from unstructured knowledge for task-oriented dialog.  _arXiv preprint arXiv:2102.04643_.
  * Vlasselaer et al. (2014) Jonas Vlasselaer, Joris Renkens, Guy Van den Broeck, and Luc De Raedt. 2014.  Compiling probabilistic logic programs into sentential decision diagrams.  In _Proceedings Workshop on Probabilistic Logic Programming (PLP)_ , pages 1–10.
  * Walker et al. (2022) Nicholas Thomas Walker, Stefan Ultes, and Pierre Lison. 2022.  Graphwoz: Dialogue management with conversational knowledge graphs.  _arXiv preprint arXiv:2211.12852_.
  * Wang et al. (2021) Hao Wang, Bin Guo, Wei Wu, Sicong Liu, and Zhiwen Yu. 2021.  Towards information-rich, logical dialogue systems with knowledge-enhanced neural models.  _Neurocomputing_ , 465:248–264.
  * Weston et al. (2018) Jason Weston, Emily Dinan, and Alexander Miller. 2018.  Retrieve and refine: Improved sequence generation models for dialogue.  In _Proceedings of the 2018 EMNLP Workshop SCAI: The 2nd International Workshop on Search-Oriented Conversational AI_ , pages 87–92, Brussels, Belgium. Association for Computational Linguistics.
  * Young et al. (2018) Tom Young, Erik Cambria, Iti Chaturvedi, Hao Zhou, Subham Biswas, and Minlie Huang. 2018.  Augmenting end-to-end dialogue systems with commonsense knowledge.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 32.
  * Yu et al. (2022a) Wenhao Yu, Chenguang Zhu, Zaitang Li, Zhiting Hu, Qingyun Wang, Heng Ji, and Meng Jiang. 2022a.  A survey of knowledge-enhanced text generation.  _ACM Computing Surveys_ , 54(11s):1–38.
  * Yu et al. (2022b) Wenhao Yu, Chenguang Zhu, Zhihan Zhang, Shuohang Wang, Zhuosheng Zhang, Yuwei Fang, and Meng Jiang. 2022b.  Retrieval augmentation for commonsense reasoning: A unified approach.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 4364–4377, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Zhan et al. (2021) Haolan Zhan, Lei Shen, Hongshen Chen, and Hainan Zhang. 2021.  CoLV: A collaborative latent variable model for knowledge-grounded dialogue generation.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 2250–2261, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Zhang et al. (2019a) Houyu Zhang, Zhenghao Liu, Chenyan Xiong, and Zhiyuan Liu. 2019a.  Grounded conversation generation as guided traverses in commonsense knowledge graphs.  _arXiv preprint arXiv:1911.02707_.
  * Zhang et al. (2018) Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018.  Personalizing dialogue agents: I have a dog, do you have pets too? In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2204–2213, Melbourne, Australia. Association for Computational Linguistics.
  * Zhang et al. (2019b) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019b.  Bertscore: Evaluating text generation with bert.  In _International Conference on Learning Representations_.
  * Zhang et al. (2021) Yizhe Zhang, Siqi Sun, Xiang Gao, Yuwei Fang, Chris Brockett, Michel Galley, Jianfeng Gao, and Bill Dolan. 2021.  Joint retrieval and generation training for grounded text generation.  _arXiv preprint arXiv:2105.06597_.
  * Zhao et al. (2020) Xueliang Zhao, Wei Wu, Can Xu, Chongyang Tao, Dongyan Zhao, and Rui Yan. 2020.  Knowledge-grounded dialogue generation with pre-trained language models.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 3377–3390, Online. Association for Computational Linguistics.
  * Zhong et al. (2022) Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng Ji, and Jiawei Han. 2022.  Towards a unified multi-dimensional evaluator for text generation.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 2023–2038, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

##  Appendix A Appendix

###  A.1 Entity linking rules

The ProbLog rules employed for entity linking are given below. The
probabilities attached to the rules are estimated empirically using the
parameter estimation approach implement in ProbLog library, based on Learning
from Interpretations Gutmann et al. (2011).

⬇

0.60838635::refers_to(M,E) :- new(U), mention(U,M), string(M,S),

is_processable_time(S,0), name(E,N), jw_similarity(N,S,O), O>0.9.

0::refers_to(M,E) :- new(U), mention(U,M), string(M,S),

is_processable_time(S,0), name(E,N), jw_similarity(N,S,O), O>0.8.

0::refers_to(M,E) :- new(U), mention(U,M), string(M,S),

is_processable_time(S,0), name(E,N), jw_similarity(N,S,O), O>0.7.

0.72255423::refers_to(M,E) :- new(U), mention(U,M), string(M,S),

is_processable_time(S,0), name(E,N), lev_distance(N,S,O), O < 2.

0.30394455::refers_to(M,E) :- new(U), mention(U,M), string(M,S),

is_processable_time(S,0), name(E,N), lev_distance(N,S,O), O < 3.

0::refers_to(M,E) :- new(U), mention(U,M), string(M,S),

is_processable_time(S,0), name(E,N), lev_distance(N,S,O), O < 6.

0.0019686::refers_to(M,E) :- new(U), mention(U,M), string(M,S),

is_processable_time(S,0), name(E,N), lcs(N,S,O), O > 3.

0::refers_to(M,E) :- new(U), mention(U,M), string(M,S),

is_processable_time(S,0), name(E,N), string(M,S), lcs(N,S,O), O > 6.

0::refers_to(M,E) :- new(U), mention(U,M), string(M,S),

is_processable_time(S,0), name(E,N), nb_common_words(N,S,O), O > 0.

0::refers_to(M,E) :- new(U), mention(U,M), string(M,S),

is_processable_time(S,0), name(E,N), nb_common_words(N,S,O), O > 1.

0::refers_to(M,E) :- new(U), mention(U,M), string(M,S),

is_processable_time(S,0), name(E,N), nb_common_words(N,S,O), O > 2.

0.27142172::refers_to(M,E) :- new(U), mention(U,M), respond_to(U,AR1),

mention(AR1,PM1), refers_to(PM1, E).

0.12752306::refers_to(M,E) :- new(U), mention(U,M), respond_to(U,AR1),

respond_to(AR1,PU1), mention(PU1,PM1), refers_to(PM1, E).

0.07429096::refers_to(M,E) :- new(U), mention(U,M), respond_to(U,AR1),

respond_to(AR1,PU1), respond_to(PU1,AR2), mention(AR2,PM1), refers_to(PM1, E).

0.01403269::refers_to(M,E) :- new(U), mention(U,M), respond_to(U,AR1),

respond_to(AR1,PU1), respond_to(PU1,AR2), respond_to(AR2,PU2),

mention(PU2,PM2), refers_to(PM1, E).

###  A.2 Commonsense rules

The rules employed for commonsense reasoning on the GraphWOZ dialogues are
provided below.

⬇

event_today(E,T) :- event(E), start_time(E,T), date(at_today,D), date(E,D).

event_tomorrow(E,T) :- event(E), start_time(E,T), date(at_tomorrow,D),
date(E,D).

person_group(P,G) :- people(P), group(P,G).

group_members(G,L) :- group(G), findall(P, person_group(P,G), L).

count_members(G,N) :- group(G), refers_to(M,G), group_members(G,L),
list_length(L,N).

room_available_today(R,T) :- room(R), \\+(location(E,R), date(E,D),
date(at_today,D),

start_time(E,ST), end_time(E,ET), time_between(T,ST,ET,1)).

room_available_tomorrow(R,T) :- room(R), \\+(location(E,R), date(E,D),

date(at_tomorrow,D), start_time(E,ST), end_time(E,ET),
time_between(T,ST,ET,1)).

room_available_now(P) :- room(P), \\+(room_busy_now(P)).

room_busy_now(P) :- room(P), time(at_now,T), attendee(E,P), date(E,D),

date(at_today,D), start_time(E,ST), end_time(E,ET), time_between(T,ST,ET,1).

person_available_today(P,T) :- refers_to(M,P), people(P),

string(_,T), is_time_expression(T,1), \\+(person_busy_today(P,T)).

person_busy_today(P,T) :- refers_to(M,P), people(P), string(_,T),

is_time_expression(T,1), attendee(E,P), date(E,D), date(at_today,D),

start_time(E,ST), end_time(E,ET), time_between(T,ST,ET,1).

person_available_tomorrow(P,T) :- refers_to(M,P), people(P), string(_,T),

is_time_expression(T,1), \\+(person_busy_tomorrow(P,T).

person_busy_tomorrow(P,T) :- refers_to(M,P), people(P), string(_,T),

is_time_expression(T,1), attendee(E,P), date(E,D), date(at_tomorrow,D),

start_time(E,ST), end_time(E,ET), time_between(T,ST,ET,1).

person_available_now(P) :- refers_to(M,P), people(P), time(at_now,T),

\\+(person_busy_now(P)).

person_busy_now(P) :- refers_to(M,P), people(P), time(at_now,T),
attendee(E,P),

date(E,D), date(at_today,D), start_time(E,ST), end_time(E,ET),

time_between(T,ST,ET,1).

attending_today(E,P) :- attendee(E,P), date(E,D), date(at_today,D).

person_events_today(P,L) :- refers_to(M,P), people(P),

findall(X, attending_today(X,P), L).

attending_tomorrow(E,P) :- attendee(E,P), date(E,D), date(at_tomorrow,D).

person_events_tomorrow(P,L) :- refers_to(M,P), people(P),

findall(X, attending_tomorrow(X,P), L).

available_rooms_now(L) :- findall(R, room_available_now(R), L).

available_rooms_today(L,T) :- string(_,M), morning_time(M,1), between(8,11,T),

findall(R, room_available_today(R,T), L).

available_rooms_tomorrow(L,T) :- string(_,M), morning_time(M,1),
between(8,11,T),

findall(R, room_available_tomorrow(R,T), L).

available_rooms_today(L,T) :- string(_,M), afternoon_time(M,1),
between(12,17,T),

findall(R, room_available_today(R,T), L).

available_rooms_tomorrow(L,T) :- string(_,M), afternoon_time(M,1),
between(12,17,T),

findall(R, room_available_tomorrow(R,T), L).

time_place(E,D,T) :- refers_to(M,E), event(E), date(E,D), start_time(E,T).