# IRS Phase-Shift Feedback Overhead-Aware Model Based on Rank-One Tensor
Approximation
††thanks: This work was supported by the Ericsson Research, Sweden, and
Ericsson Innovation Center, Brazil, under UFC.48 Technical Cooperation
Contract Ericsson/UFC. This study was financed in part by the Coordenação de
Aperfeiçoamento de Pessoal de Nível Superior - Brasil (CAPES)-Finance Code
001, and CAPES/PRINT Proc. 88887.311965/2018-00. André L. F. de Almeida
acknowledges CNPq for its financial support under the grant 312491/2020-4. G.
Fodor was partially supported by the Digital Futures project PERCy.

Bruno Sokal  Wireless Research Teleco Group (GTEL)
Federal University of Ceara (UFC
Fortaleza, Brazil
<EMAIL_ADDRESS>Paulo R. B. Gomes  Wireless Research Teleco Group
(GTEL)
Federal University of Ceara (UFC
Fortaleza, Brazil
<EMAIL_ADDRESS>André L. F. de Almeida  Wireless Research Teleco Group
(GTEL)
Federal University of Ceara (UFC
Fortaleza, Brazil
<EMAIL_ADDRESS>Behrooz Makki  Ericsson Research
Ericsson
Göteborg, Sweden
<EMAIL_ADDRESS>Gabor Fodor  Ericsson Research and KTH Royal
Institute of Technology
Ericsson
Stockholm, Sweden
<EMAIL_ADDRESS>
###### Abstract

In this paper, we propose a rank-one tensor modeling approach that yields a
compact representation of the optimum intelligent reconfigurable surface (IRS)
phase-shift vector for reducing the feedback overhead. The main idea consists
of factorizing the IRS phase-shift vector as a Kronecker product of smaller
vectors, namely factors.  The proposed phase-shift model allows the network to
trade-off between achievable data rate and feedback reduction by controling
the factorization parameters.  Our simulations show that the proposed phase-
shift factorization drastically reduces the feedback overhead, while improving
the data rate in some scenarios, compared to the state-of-the-art schemes.

###### Index Terms:

IRS, feedback, tensor modeling.

##  I Introduction

IRS is a possible candidate technology for beyond fifth generation (B5G) and
sixth generation (6G) networks due to the ability to control the
electromagnetic properties of the radio-frequency propagated waves by
performing an intelligent phase-shift to the desired direction [1, 2, 3, 4, 5,
6]. Usually, the IRS is defined as a planar ($2$-D) surface with a large
number of independent reflective elements in which they can be fully passive
or with some elements active [7, 8, 9]. The IRS is connected to a smart
controller that sets the desired phase-shift for each reflective element, by
applying some bias voltage at the elements e.g., PIN diodes. One main
advantage of the fully passive IRSs is its full-duplex nature, i.e., no noise
amplification is observed since no signal processing is possible. However,
this fully passive nature makes the channel state information acquisition
process difficult, since no pilots are processed, thus only the cascade
channel can be estimated. Another advantage of an IRS with fully passive
elements is that the power consumption is concentrated at the controller. This
makes the IRS a more attractive technology in terms of energy efficient
compared to others, e.g., amplify-and-forward and decode-and-forward relays
[10].

The work of [11] proposes a protocol design to maximize the transmission rate
in IRS-assisted multiple-input multiple-output (MIMO)-orthogonal frequency
division multiplexing (OFDM) systems. Also, [12] proposes a framework for a
feedback overhead-aware resource allocation in IRS-assisted MIMO systems.

In this work, we propose a new overhead-aware model for designing the IRS
phase-shifts. Our idea is to represent the IRS phase-shift vector according to
a rank-one model. This is achieved by factorizing a _tensorized_ version of
the IRS phase-shift vector into a rank-one tensor, which is modelled as the
Kronecker product of a predefined number of factors.  These factors are
estimated using a closed-form solution to rank-one tensor approximation. After
the estimation process, the phases of the factors are quantized and fed back
to the IRS controller, which will reconstruct the IRS phase-shift vector based
on its rank-one tensor model. The main contributions of the proposed
factorization are the following:

$1)$ The proposed IRS phase-shift factorization allows to save network
resources. In other words, the network can perform IRS phase-shift feedback
more often, or with a reduced overhead, which significantly improves the end-
to-end latency.

$2)$ The proposed method allows a trade-off between the feedback duration and
achievable data rate (ADR) performance by properly choosing the parameters of
the rank-one model that represents the IRS phase-shift vector. This is an
important feature of our proposed method, specially for a limited feedback
control link.

$3)$ Our approach relies on the optimum IRS phase-shift vector, which means
that, it can be implemented in different IRS-assisted systems and in multiple
communication links, i.e., downlink or uplink, in single-input single-output
(SISO), multiple-input single-output (MISO), as well in MIMO systems.

###  I-A Notation and Properties

Scalars are represented as non-bold lower-case letters $a$, column vectors as
lower-case boldface letters $\bm{a}$, matrices as upper-case boldface letters
$\bm{A}$, and tensors as calligraphic upper-case letters $\bm{\mathcal{A}}$.
The superscripts $\\{\cdot\\}^{\text{T}}$, $\\{\cdot\\}^{\text{*}}$,
$\\{\cdot\\}^{\text{H}}$ and $\\{\cdot\\}^{+}$ stand for transpose, conjugate,
conjugate transpose and pseudo-inverse operations, respectively. The operator
$\Arrowvert\cdot\Arrowvert_{\text{F}}$ denotes the Frobenius norm of a matrix
or tensor, $\mathbb{E}\\{\cdot\\}$ is the expectation operator. The operator
$\text{diag}\left(\bm{a}\right)$ converts $\bm{a}$ into a diagonal matrix.
Moreover, $\text{vec}\left(\bm{A}\right)$ converts
$\bm{A}\in\mathbb{C}^{I_{1}\times R}$ to a column vector
$\bm{a}\in\mathbb{C}^{I_{1}R\times 1}$ by stacking its columns on top of each
other, while the unvec($\cdot$) operator is the inverse of the vec operation.
Also, $\bm{a}_{r}\in\mathbb{C}^{I\times 1}$ represents the $r$-th column of
$\bm{A}\in\mathbb{C}^{I\times R}$. The operators $\otimes$ and $\circ$,
defines the Kronecker and the outer products, respectively. We make use of the
property

$\displaystyle\bm{a}\otimes\bm{b}=\text{vec}\left(\bm{b}\circ\bm{a}\right).$
(1)

##  II Tensor Pre-Requisites

Consider a set of matrices $\\{\bm{X}_{i_{3}}\\}\in\mathbb{C}^{I_{1}\times
I_{2}}$, for $i_{3}=1,\ldots,I_{3}$. By concatenating all $I_{3}$ matrices to
form the third-order tensor
$\bm{\mathcal{X}}=[\bm{X}_{1}\sqcup_{3}\bm{X}_{2}\sqcup_{3}\ldots\sqcup_{3}\bm{X}_{I_{3}}]\in\mathbb{C}^{I_{1}\times
I_{2}\times I_{3}}$, where $\sqcup_{3}$ indicates a concatenation in the third
dimension. We can interpret $\bm{X}_{i_{3}}$ as the $i_{3}$-th frontal slice
of $\bm{\mathcal{X}}$, defined as $\bm{\mathcal{X}}_{..i_{3}}=\bm{X}_{i_{3}}$
where the “$..$” indicates that the dimensions $I_{1}$ and $I_{2}$ are fixed.
The tensor $\bm{\mathcal{X}}$ can be matricized by letting one dimension vary
along the rows and the remaining two dimensions along the columns. From
$\bm{\mathcal{X}}$, we can form three different matrices, referred to as the
$n$-mode unfoldings (for $n=\\{1,2,3\\}$ in this case),

$\displaystyle\left[\bm{\mathcal{X}}\right]_{\left(1\right)}$
$\displaystyle=[\bm{\mathcal{X}}_{..1},\ldots,\bm{\mathcal{X}}_{..I_{3}}]\in\mathbb{C}^{I_{1}\times
I_{2}I_{3}},$ (2)
$\displaystyle\left[\bm{\mathcal{X}}\right]_{\left(2\right)}$
$\displaystyle=[\bm{\mathcal{X}}_{..1}^{\text{T}},\ldots,\bm{\mathcal{X}}_{..I_{3}}^{\text{T}}]\in\mathbb{C}^{I_{2}\times
I_{1}I_{3}}$ (3) $\displaystyle\left[\bm{\mathcal{X}}\right]_{\left(3\right)}$
$\displaystyle=[\text{vec}(\bm{\mathcal{X}}_{..1}),\ldots,\text{vec}(\bm{\mathcal{X}}_{..I_{3}})]^{\text{T}}\in\mathbb{C}^{I_{3}\times
I_{1}I_{2}}.$ (4)

###  II-A Tensorization

The tensorization operation consists of mapping the elements of a vector into
high-order tensors. Let us define the vector $\bm{y}\in\mathbb{C}^{N\times
1}$, in which $N=\prod\limits_{p=1}^{P}{N_{p}}$. By applying the tensorization
operator, defined as $\bm{\mathcal{T}}\\{\cdot\\}$, we can form the tensor
$\bm{\mathcal{Y}}=\bm{\mathcal{T}}\\{\bm{y}\\}\in\mathbb{C}^{N_{1}\times
N_{2}\times\ldots\times N_{P}}$. The mapping of elements from $\bm{y}$ to
$\bm{\mathcal{Y}}$ is defined as

$\displaystyle\bm{\mathcal{Y}}_{n_{1},n_{2},\ldots,n_{P}}=\bm{y}_{n_{1}+(n_{2}-1)N_{1}+\ldots+(n_{P}-1)N_{P-1}\cdots
N_{2}N_{1}},$ (5)

where $n_{p}=\\{1,\ldots,N_{p}\\}$, for $p=\\{1,\ldots,P\\}$. This operator
plays a key role on the proposed feedback-aware method where we reshape the
elements of the IRS phase-shift vector into a $P$-order tensor.

###  II-B Rank-One Tensors

A rank-one matrix can be described as the outer product of two vectors.
Similarly to the matrix case, a rank-one tensor is given as the outer product
of three or more vectors. For a $P$-order rank-one tensor $\bm{\mathcal{Y}}$,
it can be written as

$\displaystyle\bm{\mathcal{Y}}=\bm{a}^{(1)}\circ\bm{a}^{(2)}\circ\ldots\circ\bm{a}^{(P)}\in\mathbb{C}^{I_{1}\times
I_{2}\times\ldots\times I_{P}},$ (6)

where $\bm{a}^{(p)}\in\mathbb{C}^{I_{p}\times 1}$ is the $p$-th factor of
$\bm{\mathcal{Y}}$, for $p=\\{1,\ldots,P\\}$. The $p$-th rank-one matrix
unfolding of $\bm{\mathcal{Y}}$, defined as
$\left[\bm{\mathcal{Y}}\right]_{\left(P\right)}\in\mathbb{C}^{I_{p}\times
I_{1}\cdots I_{p-1}I_{p+1}\cdots I_{P}}$, is given by

$\displaystyle\left[\bm{\mathcal{Y}}\right]_{\left(p\right)}=\bm{a}^{(p)}\left(\bm{a}^{(P)}\otimes\ldots\otimes\bm{a}^{(p+1)}\otimes\bm{a}^{(p-1)}\ldots\otimes\bm{a}^{(1)}\right)^{\text{T}}.$
(7)

Defining
$\bm{y}=\text{vec}\left(\bm{\mathcal{Y}}\right)\in\mathbb{C}^{I_{1}\cdots
I_{P}\times 1}$ and using (1), we have

$\displaystyle\bm{y}=\bm{a}^{(P)}\otimes\ldots\otimes\bm{a}^{(1)}.$ (8)

##  III System Model

We consider an IRS-assisted MIMO system, where the transmitter (TX) is
equipped with a uniform linear array (ULA) with $M_{T}$ antenna elements, the
receiver (RX) is equipped with ULA with $M_{R}$ antenna elements and the IRS
has $N$ reflective elements. To simplify the discussions, let us consider a
single stream transmission, assuming that there is no direct link between the
TX and RX. First, the TX sends a pilot signal to the RX with the aid of the
IRS. Since there is no signal processing at the IRS, the channel estimation
and the IRS phase-shifts optimization are performed at the RX. The received
signal after processing the pilots is given by

$\displaystyle
y=\bm{w}^{\text{H}}\bm{G}\bm{S}\bm{H}\bm{q}+\bm{w}^{\text{H}}\bm{b},$ (9)

where $\bm{b}\in\mathbb{C}^{M_{R}\times 1}$ is the additive noise at the
receiver with
$\mathbb{E}[\bm{b}\bm{b}^{\text{H}}]=\sigma^{2}_{b}\bm{I}_{M_{r}}$,
$\bm{w}\in\mathbb{C}^{M_{R}\times 1}$ and $\bm{q}\in\mathbb{C}^{M_{T}\times
1}$ are the receiver and transmitter combiner and precoder, respectively.
$\bm{H}\in\mathbb{C}^{N\times M_{T}}$ and $\bm{G}\in\mathbb{C}^{M_{R}\times
N}$ are the TX-IRS and IRS-RX involved channels, and
$\bm{S}=\text{diag}(\bm{s})\in\mathbb{C}^{N\times N}$ where
$\bm{s}=[e^{j\theta_{1}},\ldots,e^{j\theta_{N}}]\in\mathbb{C}^{N\times 1}$ is
the IRS phase-shift vector, and $\theta_{n}$ is the phase-shift applied to the
$n$-th IRS element.

After the channel estimation step, an optimization step of the precoder and
combiner (active beamformers) vectors $\bm{w}$ and $\bm{q}$, and the IRS
phase-shift vector $\bm{s}$ (passive beamformer) is performed. Later, the RX
needs to feedback to the IRS controller the designed phase-shift so that the
elements can reconfigure the elements to have the optimal phase-shift. From
the fact that this feedback may occur through a limited control channel and
that the IRS may contain several hundreds to thousand of elements, the
feedback of each phase-shift with a certain resolution implies in a signaling
overhead.

Related to the issue of the overhead IRS phase-shift signaling feedback, the
work of [12] models the feedback duration as

$T_{\text{F}}=\frac{Nb_{\text{F}}}{B_{\text{F}}\text{log}\left(1+\frac{p_{\text{F}}|{g}_{\text{F}}|^{2}}{B_{\text{F}}N_{0}}\right)},$
(10)

where $N$ is the total number of phase-shifts of the IRS to be fed back,
$B_{\text{F}}$, $p_{\text{F}}$ are the feedback bandwith and power,
$g_{\text{F}}$ is the scalar control channel used, $b_{\text{F}}$ is the
resolution of each phase-shift, and $N_{0}$ is the noise power density.

##  IV Proposed Feedback-Aware Method

In this section, we describe our proposed feedback-aware method that focuses
on reducing the feedback duration $T_{\text{F}}$, given in (10). Suppose that
the channels $\bm{H}$ and $\bm{G}$ are estimated at the RX. Then, the $N$
phase-shifts of the IRS are determined based on state-of-the-art algorithm
[12], and are represented in a vector format as
$\bm{s}=[e^{j\theta_{1}},e^{j\theta_{2}},\ldots,e^{j\theta_{N}}]\in\mathbb{C}^{N\times
1}$. Our idea is to factorize $\bm{s}$ as the Kronecker product of $P$
factors, i.e.,

$\bm{s}=\bm{s}^{(P)}\otimes\ldots\otimes\bm{s}^{(1)}\in\mathbb{C}^{N_{P}\cdots
N_{1}\times 1}$ (11)

where $\bm{s}^{(p)}\in\mathbb{C}^{N_{p}\times 1}$ $p$-th factor. From (11), we
have that $N=\prod\limits_{p=1}^{P}N_{p}$. The main idea of the proposed model
is to feedback to the IRS controller only the phase-shifts of the factors,
$\bm{s}^{(p)}$, for $p=\\{1,\ldots,P\\}$, which results into the feedback of
$\sum\limits_{p=1}^{P}N_{p}$ phase-shifts, instead of
$N=\prod\limits_{p=1}^{P}$ phase-shifts in the case with no factorization.

Example: To illustrate the impact of this factorization on the IRS phase-shift
feedback duration, let us consider the simple scenario, illustrated in Figure
1, where we have an IRS with $N=1024$ elements. Normally, supposing $N$
independent elements, in Figure 1 (a), $N=1024$ phase-shifts would have to be
determined at some system node and conveyed to the IRS controller. On the
other hand, considering that, as one example, $P=3$ with the following factors
$\bm{s}^{(1)}=[e^{j\theta^{(1)}_{1}},\ldots,e^{j\theta^{(1)}_{64}}]\in\mathbb{C}^{64\times
1}$,
$\bm{s}^{(2)}=[e^{j\theta^{(2)}_{1}},\ldots,e^{j\theta^{(2)}_{8}}]\in\mathbb{C}^{8\times
1}$ and
$\bm{s}^{(3)}=[e^{j\theta^{(3)}_{1}},e^{j\theta^{(3)}_{2}}]\in\mathbb{C}^{2\times
1}$, i.e., $N_{1}=64$, $N_{2}=8$ and $N_{3}=2$, only $N_{1}+N_{2}+N_{3}=74$
phase-shifts has to be reported to the IRS controller, drastically reducing
the feedback overhead compared to the traditional scenario (a), by a factor of
$\approx 1400\%$. $\blacksquare$

With the factorization, an additional complexity is introduced on the IRS
controller, which will have to build the phase-shift vector, which in this
case is given as
$\bm{s}=\bm{s}^{(3)}\otimes\bm{s}^{(2)}\otimes\bm{s}^{(1)}\in\mathbb{C}^{1024\times
1}$. Physically, the Kronecker product in (11) represents a summation of the
factors phase-shifts,  as shown in Figure 1 (b). The proposed feedback
duration is given by

Figure 1: (a) IRS with $N$ phase-shifts, without the factorization, (b) IRS
with $N$ phase-shifts factorized into $P=3$ factors.

$T_{\text{F}}^{\text{prop}}=\frac{T_{\text{PR}}+\sum\limits_{p=1}^{P}N_{p}b^{(p)}_{\text{F}}}{B_{\text{F}}\text{log}\left(1+\frac{p_{\text{F}}|{g}_{\text{F}}|^{2}}{B_{\text{F}}N_{0}}\right)},$
(12)

where $T_{PR}$ is the duration of the preamble that informs the values of the
factorization parameters, such as the number of factors $P$, the size of each
factor $N_{p}$, and the number of quantization bits used for each factor
$b^{(p)}_{\text{F}}$, for $p=\\{1,\ldots,P\\}$.

The physical implications on the choice of the proposed rank-one tensor
approximation parameters, i.e., the number of the factors $P$ and the size of
each factor $N_{p}$ are discussed in Section IV-C. In a general view, the
proposed factorization consists of three steps:

$1)$ Rearrangement of elements: In this step, the optimum phase-shift vector
$\bm{s}\in\mathbb{C}^{N\times 1}$ is rearranged into a $P$-th order tensor
$\bm{\mathcal{S}}\in\mathbb{C}^{N_{1}\times N_{2}\times\ldots\times N_{P}}$,
with $N=\prod\limits_{p=1}^{P}N_{p}$. This is accomplished by mapping the
elements of the IRS phase-shift vector $\bm{s}$ into the tensor $\bm{S}$,
using the tensorization operator, given in (5).

$2)$ Rank-One tensor approxmation: In this step, the RX estimates the factors
of the IRS phase-shift tensor $\bm{\mathcal{S}}$. For this, a prior art rank-
one tensor estimation algorithmz can be employed, such as the high order
singular value decomposition (HOSVD) [13], which is described in Algorithm 1.

$3)$ Normalization: The estimated factors have their entries normalized to
ensure the unitary modulus constraint of the IRS phase-shift vector. This can
be achieved by only taking the angles of the estimated factors.

###  IV-A Rank-One IRS model

After the tensorization step, the RX will approximate the optimum phase-shift
tensor $\bm{\mathcal{S}}$ using a rank-one model, i.e.,

$\bm{\mathcal{S}}\approx\bm{s}^{(1)}\circ\ldots\circ\bm{s}^{(P)}\in\mathbb{C}^{N_{1}\times\cdots\times
N_{P}}.$ (13)

Please note that, the approximation comes from the fact that we are fitting
independent phase-shifts as a combination of $P$ sets of phase-shifts, thus an
approximation error is expected. However, as it will be explained in Section
V, for scenarios with moderate/strong line of sight (LOS) components
(approximated rank-one channels), which is the case of interest in IRS
networks, this fitting error is negligible on the ADR performance.

The RX estimates the factor components solving the following problem

$\displaystyle\left[\bm{\hat{s}}^{(1)},\ldots,\bm{\hat{s}}^{(P)}\right]=\underset{\bm{s}^{(1)},\ldots,\bm{s}^{(P)}}{\text{argmin}}\left|\left|\bm{\mathcal{S}}-\bm{s}^{(1)}\circ\ldots\circ\bm{s}^{(P)}\right|\right|^{2}_{\text{F}},$
(14)

where $\bm{s}^{(p)}\in\mathbb{C}^{N_{p}\times 1}$ is the $p$-th factor. From
(7), the $p$-mode unfolding of $\bm{\mathcal{S}}$, defined as
$\left[\bm{\mathcal{S}}\right]_{\left(p\right)}\in\mathbb{C}^{N_{p}\times
N_{1}\cdots N_{p-1}N_{p+1}\cdots N_{P}}$, is the approximated rank-one matrix,
given as

$\left[\bm{\mathcal{S}}\right]_{\left(p\right)}\approx\bm{s}^{(p)}\left(\bm{s}^{(P)}\otimes\ldots\otimes\bm{s}^{(p+1)}\otimes\bm{s}^{(p-1)}\otimes\ldots\otimes\bm{s}^{(1)}\right)^{\text{T}}.$
(15)

1:Inputs: Tensor $\bm{\mathcal{S}}$

2:for $p=1:P$ do

3: Define the SVD of $\left[\bm{\mathcal{S}}\right]_{\left(p\right)}$ as
$\bm{U}^{(p)}\bm{\Sigma}^{(p)}\bm{V}^{(p)\text{H}}$, compute an estimate of
the $p$-th factor $\bm{s}^{(p)}$ as
$\displaystyle\bm{\hat{s}}^{(p)}=e^{j\angle\bm{u}_{.1}^{(p)}}$

4:end for

5:Return $\bm{\hat{s}}^{(1)}$, $\ldots$, $\bm{\hat{s}}^{(P)}$.

Algorithm 1 Feedback-Aware Rank-One Approximation

To solve the problem in (14), we make use of the rank-one HOSVD, which
consists of computing, a singular value decomposition (SVD) 111In fact, since
we want the left dominant singular vector of each unfolding, the power method
[14] can be used instead of computing the whole SVD.  for each unfolding of
$\bm{\mathcal{S}}$ to extract the left dominant singular vector, i.e.,
considering the SVD of the $p$-th unfolding of $\bm{\mathcal{S}}$,
$\left[\bm{\mathcal{S}}\right]_{\left(p\right)}$ as
$\bm{U}^{(p)}\bm{\Sigma}^{(p)}\bm{V}^{(p)\text{H}}$, the algorithm set an
estimation for the $p$-th factor $\bm{s}^{(p)}$ as
$\bm{\hat{s}}^{(p)}=e^{j\angle\bm{u}_{.1}^{(p)}}\in\mathbb{C}^{N_{p}\times
1}$, where $\bm{u}_{.1}^{(p)}$ is the dominant left singular vector of
$\bm{s}^{(p)}$.

###  IV-B Phase-shift Quantization and Feedback

It is important to mention that, after the rank-one factor estimation
procedure in Algorithm 1, the RX quantizes the phase-shifts of each factor
with $b_{F}^{(p)}$ bits. Let us define
$\bm{\tilde{s}}^{(p)}\in\mathbb{C}^{N_{p}\times 1}$ as the $p$-factor after
the quantization process, i.e.,
$\bm{\tilde{s}}^{(p)}=\mathcal{Q}\left\\{\bm{\hat{s}}^{(p)},b_{\text{F}}^{(p)}\right\\}$.
Then, the phase-shifts of the $P$ factors are conveyed to the IRS controller
via a control link channel. Finally, the IRS controller reconstructs the IRS
phase-shift vector as

$\displaystyle\bm{s}=\bm{\tilde{s}}^{(P)}\otimes\ldots\otimes\bm{\tilde{s}}^{(1)}\in\mathbb{C}^{N\times
1}.$ (16)

As observed, the proposed method allows the system to adapt the phase-shift
resolution of the factors depending on the control link availability. This
feature is very important in order to minimize the performance loss, in the
case of limited control link.

###  IV-C On the Effect of the Factorization Parameters

In this section, we discuss the choice of the factorization parameters and the
system performance implications.

Number of factors $P$: This parameter defines the total number of factors used
in the rank-one tensor approximation. Its minimum value for the proposed
factorization is $P=2$, i.e., the value $P=1$ means that no factorization is
employed. By increasing the value of $P$, the number of factors of the
factorization model is increased, allowing to reduce the size of the factor
components $N_{p}$, consequentially, increasing $P$ reduces the phase-shift
feedback overhead.

Size of factor components $N_{p}$: The size of the factor components indicates
the total number of independent phase-shifts in the proposed solution, which
it also affects the performance. For example, for $N=256$ and choosing $P=2$,
two possible configurations are $N_{1}=128$, $N_{2}=2$ and $N_{1}=N_{2}=16$.
For the first choice, the system has more independent phase-shifts ($130$),
thus a higher spectral efficiency. However, its feedback overhead is higher
than that of second configuration that requires only $32$ phase-shifts to be
reported in the feedback channel.

##  V Simulation Results

In this section, we evaluate the performance of the proposed IRS phase-shift
overhead-aware feedback model in terms of feedback duration and ADR. For a
fair comparison between the proposed rank-one model and the state-of-the-art,
we optimize the precoder ($\bm{q}$), combiner ($\bm{w}$), and the IRS phase-
shifts ($\bm{s}$) using the upper-bound algorithm of the state-of-the-art
[12]. In this case, they are given as

$\displaystyle\bm{w}=\bm{U}^{(\text{G})}_{.1},\hskip
2.84544pt\bm{q}=\bm{V}^{(\text{H})}_{.1},\hskip
2.84544pt\bm{s}_{n}=e^{-\angle\left(\bm{V}^{\text{G})}_{n,1}\cdot\bm{U}^{(\text{H})}_{n,1}\right)},n=\\{1,\ldots,N\\},$

where $\bm{U}^{(\text{G})}_{.1}\in\mathbb{C}^{M_{R}\times 1}$,
$\bm{V}^{\text{G})}_{.1}\in\mathbb{C}^{N\times 1}$ are the dominant left and
right singular vectors of $\bm{G}$, while
$\bm{U}^{(\text{H})}_{.1}\in\mathbb{C}^{N\times 1}$,
$\bm{V}^{\text{H})}_{.1}\in\mathbb{C}^{M_{T}\times 1}$ are the dominant left
and right singular vectors of $\bm{H}$. The ADR is given by

$\displaystyle\text{ADR}=\text{log}_{2}\left(1+\frac{\left|\bm{w}^{\text{H}}\bm{G}\bm{S}\bm{H}\bm{q}\right|^{2}}{\sigma^{2}_{b}}\right),$
(17)

where, the noise variance $\sigma^{2}_{b}=0.1$,
$\bm{S}=\text{diag}(\bm{s})\in\mathbb{C}^{N\times 1}$ is the matrix that
contains in its diagonal the optimized feedback phase-shifts, which, in the
proposed approach is given in (16). Regarding the involved channels, in (9),
they are modelled as

$\displaystyle\bm{H}$
$\displaystyle=\sqrt{\alpha_{H}\frac{K_{H}}{K_{H}+1}}\bm{H}_{\text{LOS}}+\sqrt{\frac{1}{K_{H}+1}}\bm{H}_{\text{NLOS}},$
(18) $\displaystyle\bm{G}$
$\displaystyle=\sqrt{\alpha_{G}\frac{K_{G}}{K_{G}+1}}\bm{G}_{\text{LOS}}+\sqrt{\frac{1}{K_{G}+1}}\bm{G}_{\text{NLOS}},$
(19)

where $\alpha_{H}$ and $\alpha_{G}$ are the path-loss components of the TX-IRS
and IRS-RX links, respectively. $K_{H}$ and $K_{G}$ are the Rician factors for
the channels $\bm{H}$ and $\bm{G}$, respectively. $\bm{H}_{\text{LOS}}$,
$\bm{G}_{\text{LOS}}$ are modelled as geometric-based channels, while the
entries of $\bm{H}_{\text{NLOS}}$, $\bm{G}_{\text{NLOS}}$ are modelled as
circularly symmetric complex Gaussian random variables, with zero mean an unit
variance, i.e., $\bm{H}_{\text{NLOS}}\sim\mathcal{CN}(0,\bm{I}_{M_{T}})$ and
$\bm{G}_{\text{NLOS}}\sim\mathcal{CN}(0,\bm{I}_{M_{R}})$.  We consider
$\alpha_{H}=\alpha_{G}=1$ for experimental purposes, to observe the
performance impact of the proposed IRS phase-shift factorization. Also, we
consider $K_{H}=K_{G}=K$ as metric for performance evaluation. The details of
the channel model are given in Appendix A.

Let us define the vector
$\bm{N}_{\text{P}}=\left[N_{1},\ldots,N_{P}\right]^{\text{T}}\in\mathbb{R}^{P\times
1}$ that contains the size of all individual factors, for a certain $P$.
Likewise, let us define the vector
$\bm{b}^{(\text{P})}_{\text{F}}=\left[b^{(1)}_{\text{F}},\ldots,b^{(P)}_{\text{F}}\right]^{\text{T}}\in\mathbb{R}^{P\times
1}$ that contains the number of bits used for quantization in each factor, for
a certain $P$.

Figure 2 illustrates the payload ratio (PR) between the proposed feedback
overhead model and state-of-the-art, given by
$\text{PR}=N/\left(\sum\limits_{p=1}^{P}N_{p}\right)$. In this figure, we can
observe the role of the $P$ in the factorization model. It allows to decrease
the size of the factors, consequentially, allows a massive feedback overhead
reduction. For example, in this case where $N=1024$, for $P=2$ we observe a
maximum feedback overhead reduction when $\bm{N}_{2}=[32,32]$, i.e., we
feedback $64$ phase-shifts against the total of $1024$ phase-shifts that the
state-of-the-art does, in other words, a feedback duration $16$ times smaller.
As $P$ increases, we can improve the feedback overhead reduction, at the point
that, for $P=10$, the proposed feedback overhead is more than $50$ times
smaller than the state-of-the-art [12].

Figure 2: Feedback Payload Ratio for different factors and size
configurations. $N=1024$ and $b_{\text{F}}=b^{(p)}_{\text{F}}=3$ bits, for
$p=\\{1,\ldots,P\\}$. Figure 3: For an IRS with $N=1024$, TX and RX with
$M_{R}=M_{T}=2$ and $b^{(p)}_{\text{F}}=b_{\text{F}}=3$ bits, for the IRS
phase-shift quantization resolution, for $p=\\{1,\ldots,P\\}$. Figure 4: ADR
of the state-of-the-art and the proposed factorization for $P=3$ with
different size configurations and resolution (bits), in a fixed control link
scenario of $1024$ bits.

In Figure 3, we study the achievable rate versus the Rician factor $K$ for
different factorization methods. Although the proposed IRS phase-shift
factorization has a performance degradation in non-line of sight (NLOS)
scenarios ($K<0$ dB), the required number of bits for the feedback of the IRS
phase-shifts is reduced drastically, as also concluded in Figure 2. We also
observe that, as the number of factors $P$ increases the ADR decreases. This
is explained by the fact that $P$ is the factorization parameter with the role
to control the size of each factor, recalling that
$N=\prod\limits_{p=1}^{P}N_{p}$. In other words, for a higher $P$, the number
of independent phase-shifts can be reduced to provide a low feedback payload,
while for a smaller $P$ the focus is the performance ADR with feedback
reduction. Thus, there is a trade-off between the achievable rate and the
feedback overhead, based on which one can select the proper factorization
parameters. However, as we move towards LOS scenarios ($K>5$ dB), we can
observe that, for all proposed parameter configurations ($P=\\{2,3,4,10\\}$)
we achieve approximately the same data rate while for a higher number of
factors ($P=10$) we have a negligible amount of bits to be fed back, compared
to the state-of-the-art. This is explained by the fact that the IRS phase-
shift vector $\bm{s}\in\mathbb{C}^{N\times 1}$ is optimized based on the
involved channels $\bm{H}$ and $\bm{G}$ which are geometric-based channels
with a separable structure in the IRS (azimuth and elevation), also, every
Vandermond vector can be written as the Kronecker product of multiple factors
[15]. Nonetheless, in LOS scenarios, a higher $P$ should be prioritized since
it has the lowest feedback overhead with a similar ADR than the state-of-the-
art [12].

In Figure 4 we compare the proposed factorization with the state-of-the-art
for the case of a fixed feedback control link of $1024$ bits. We consider, for
the proposed case, $P=3$ factors, with different size configurations and,
consequentially, resolution, i.e.,
$\bm{N}_{\text{P}}^{\text{T}}\cdot\bm{b}^{(\text{P})}_{\text{F}}\leq 1024$. As
upper bound, we plot the state-of-the-art with continuous phase-shift (solid
curve) to compare with the effect of the quantization. We can observe that,
since $N=1024$ phase-shifts, the state-of-the-art can only perform a one-bit
quantization, while the proposed method can apply different resolution per
factor, in order to meet the control link limit. In this case, for $K\geq-5$
dB, the proposed factorization achieves a better performance than the state-
of-the-art, while as $K$ grows, the proposed factorization method achieves a
similar performance as in the state-of-the-art with continuous phase-shift.

To summarize the results illustrated in Figures 2-4, the control of the
factorization parameters ($P$, $\bm{N}_{\text{P}}$ and
$\bm{b}^{(\text{P})}_{\text{F}}$) creates a trade-off between ADR and feedback
reduction (in NLOS scenarios), thus RX can decide, for example, to feedback
the IRS phase-shifts more often, feedback the IRS phase-shifts of multiple
users, or in a limited control link channel, can improve the data rate by
increase the resolution of the individual factors.  Meanwhile, in the
scenarios with a moderate/strong LOS, it is preferable factorize the IRS
phase-shift vector with the maximum number of factors $P$, which depends of
$N$, since $N=\prod\limits_{p=1}^{P}{N_{p}}$.

##  VI Conclusion and Perspectives

In this paper, we have proposed a new feedback overhead-aware method by
factorizing the optimum IRS phase-shift factor into smaller factors. We have
shown the trade-off between the ADR and the feedback overhead by varying the
factorization parameters. In a future work, we will investigate different low-
rank approximations models and their trade-off between ADR and feedback
overhead.

##  Appendix A LOS Channel Model

The LOS components are given as

$\displaystyle\bm{H}_{\text{LOS}}$
$\displaystyle=\alpha_{H}\bm{b}_{\text{IRS}}\cdot\bm{a}^{\text{H}}_{\text{TX}}\in\mathbb{C}^{N\times
M_{T}},$ $\displaystyle\bm{G}_{\text{LOS}}$
$\displaystyle=\alpha_{G}\bm{b}_{\text{RX}}\cdot\bm{a}^{\text{H}}_{\text{IRS}}\in\mathbb{C}^{M_{R}\times
N},$

where $\alpha_{H}$ and $\alpha_{G}$ are the path-loss components of the links
TX-IRS and IRS-RX, respectively. By considering that the TX and RX are
equipped with ULAs, in which, the spacing between the antennas is $\lambda/2$,
their steering vectors, $\bm{a}_{\text{TX}}$ and $\bm{b}_{\text{RX}}$ can be
written as

$\displaystyle\bm{a}_{\text{TX}}=\left[1,e^{j\pi\text{sin}\theta_{\text{TX}}},\ldots,e^{j\pi(M_{T}-1)\text{sin}\theta_{\text{TX}}}\right]^{\text{T}}\in\mathbb{C}^{M_{T}\times
1},$ (20)
$\displaystyle\bm{b}_{\text{RX}}=\left[1,e^{j\pi\text{sin}\theta_{\text{RX}}},\ldots,e^{j\pi(M_{R}-1)\text{sin}\theta_{\text{RX}}}\right]^{\text{T}}\in\mathbb{C}^{M_{R}\times
1},$ (21)

where $\theta_{\text{TX}}$ and $\theta_{\text{RX}}$ are the TX and RX angle of
departure (AOD) and angle of arrival (AOA), respectively. Regarding to the IRS
steering vectors on the AOA and AOD directions, they are given, as
$\bm{b}_{\text{IRS}}=\bm{b}^{(v)}_{\text{IRS}}\otimes\bm{b}^{(h)}_{\text{IRS}}\in\mathbb{C}^{N_{h}N_{v}\times
1}$ and
$\bm{a}_{\text{IRS}}=\bm{a}^{(v)}_{\text{IRS}}\otimes\bm{a}^{(h)}_{\text{IRS}}\in\mathbb{C}^{N_{h}N_{v}\times
1}$, $\bm{b}^{(h)}_{\text{IRS}}\in\mathbb{C}^{N_{h}\times 1}$ and
$\bm{b}^{(v)}_{\text{IRS}}\in\mathbb{C}^{N_{v}\times 1}$ are the AOA steering
vectors in the azimuth and elevation directions, respectively, and
$N=N_{h}N_{v}$. Likewise, $\bm{a}^{(h)}_{\text{IRS}}\in\mathbb{C}^{N_{h}\times
1}$ and $\bm{a}^{(v)}_{\text{IRS}}\in\mathbb{C}^{N_{v}\times 1}$. The arrival
steering vectors are given as

$\displaystyle\bm{b}^{(h)}_{\text{IRS}}$
$\displaystyle=[1,e^{j\pi\text{sin}\psi^{\text{AOA}}_{\text{IRS}}\text{cos}\phi^{\text{AOA}}_{\text{IRS}}},\ldots,e^{j\pi(N_{h}-1)\text{sin}\psi^{\text{AOA}}_{\text{IRS}}\text{cos}\phi^{\text{AOA}}_{\text{IRS}}}],$
$\displaystyle\bm{b}^{(v)}_{\text{IRS}}$
$\displaystyle=[1,e^{j\pi\text{cos}\phi^{\text{AOA}}_{\text{IRS}}},\ldots,e^{j\pi(N_{h}-1)\text{cos}\phi^{\text{AOA}}_{\text{IRS}}}],$

In the same way for the AOD steering vectors $\bm{a}^{(h)}_{\text{IRS}}$ and
$\bm{a}^{(v)}_{\text{IRS}}$. All the azimuth AOA and AOD are generated from a
uniform random distribution in $[-\pi,\pi]$, while the elevation AOA and AOD
follows a uniform random distribution within $[0,\pi/2]$.

## References

  * [1] Q. Wu, S. Zhang, B. Zheng, C. You, and R. Zhang, “Intelligent reflecting surface aided wireless communications: A tutorial,” _IEEE Transactions on Communications_ , pp. 1–1, May 2021.
  * [2] M. Jian, G. C. Alexandropoulos, E. Basar, C. Huang, R. Liu, Y. Liu, and C. Yuen, “Reconfigurable intelligent surfaces for wireless communications: Overview of hardware designs, channel models, and estimation techniques,” _arXiv preprint arXiv:2203.03176_ , 2022.
  * [3] N. Rajatheva, I. Atzeni, S. Bicais, E. Bjornson, A. Bourdoux, S. Buzzi, C. D’Andrea, J.-B. Dore, S. Erkucuk, M. Fuentes _et al._ , “Scoring the terabit/s goal: Broadband connectivity in 6G,” _arXiv preprint arXiv:2008.07220_ , 2020.
  * [4] G. T. de Araújo, P. R. B. Gomes, A. L. F. de Almeida, G. Fodor, and B. Makki, “Semi-Blind Joint Channel and Symbol Estimation in IRS-Assisted Multi-User MIMO Networks,” _arXiv preprint arXiv:2202.11087_ , 2022.
  * [5] H. Guo, B. Makki, M. Åström, M.-S. Alouini, and T. Svensson, “Dynamic Blockage Pre-Avoidance using Reconfigurable Intelligent Surfaces,” _arXiv preprint arXiv:2201.06659_ , 2022.
  * [6] K. B. d. A. Benício, B. Sokal, and A. L. F. de Almeida, “Channel Estimation and Performance Evaluation of Multi-IRS Aided MIMO Communication System,” in _2021 Workshop on Communication Networks and Power Systems (WCNPS)_ , 2021, pp. 1–6.
  * [7] A. Taha, M. Alrabeiah, and A. Alkhateeb, “Enabling large intelligent surfaces with compressive sensing and deep learning,” 2019, [Online]. Available: https://arxiv.org/abs/1904.10136.
  * [8] M. H. Khoshafa, T. M. Ngatched, M. H. Ahmed, and A. R. Ndjiongue, “Active reconfigurable intelligent surfaces-aided wireless communication system,” _IEEE Communications Letters_ , vol. 25, no. 11, pp. 3699–3703, 2021.
  * [9] G. C. Alexandropoulos and E. Vlachos, “A hardware architecture for reconfigurable intelligent surfaces with minimal active elements for explicit channel estimation,” in _Proc. ICASSP_. Barcelona, Spain: IEEE, 2020, pp. 9175–9179.
  * [10] E. Björnson, Ö. Özdogan, and E. G. Larsson, “Intelligent reflecting surface versus decode-and-forward: How large surfaces are needed to beat relaying?” _IEEE Wireless Communications Letters_ , vol. 9, no. 2, pp. 244–248, 2019.
  * [11] Y. Yang, B. Zheng, S. Zhang, and R. Zhang, “Intelligent reflecting surface meets OFDM: Protocol design and rate maximization,” _IEEE Transactions on Communications_ , 2020.
  * [12] A. Zappone, M. Di Renzo, F. Shams, X. Qian, and M. Debbah, “Overhead-aware design of reconfigurable intelligent surfaces in smart radio environments,” _IEEE Transactions on Wireless Communications_ , vol. 20, no. 1, pp. 126–141, 2021.
  * [13] L. De Lathauwer, B. De Moor, and J. Vandewalle, “On the best rank-1 and rank-(r 1, r 2,…, rn) approximation of higher-order tensors,” _SIAM journal on Matrix Analysis and Applications_ , vol. 21, no. 4, pp. 1324–1342, 2000\.
  * [14] G. H. Golub and C. F. Van Loan, “Matrix computations.” JHU press, 2013, ch. 10.
  * [15] B. Sokal, A. L. de Almeida, and M. Haardt, “Semi-blind receivers for MIMO multi-relaying systems via rank-one tensor approximations,” _Signal Processing_ , vol. 166, p. 107254, 2020.


# Towards Fine-Grained Citation Evaluation in Generated Text:
A Comparative Analysis of Faithfulness Metrics

Weijia Zhang1 Mohammad Aliannejadi1 Yifei Yuan2
Jiahuan Pei3 Jia-Hong Huang1 Evangelos Kanoulas1
1University of Amsterdam
2University of Copenhagen
3Centrum Wiskunde & Informatica
<EMAIL_ADDRESS>
###### Abstract

Large language models (LLMs) often produce unsupported or unverifiable
information, known as “hallucinations.” To mitigate this, retrieval-augmented
LLMs incorporate citations, grounding the content in verifiable sources.
Despite such developments, manually assessing how well a citation supports the
associated statement remains a major challenge. Previous studies use
faithfulness metrics to estimate citation support automatically but are
limited to binary classification, overlooking fine-grained citation support in
practical scenarios. To investigate the effectiveness of faithfulness metrics
in fine-grained scenarios, we propose a comparative evaluation framework that
assesses the metric effectiveness in distinguishing citations between three-
category support levels: full, partial, and no support. Our framework employs
correlation analysis, classification evaluation, and retrieval evaluation to
measure the alignment between metric scores and human judgments
comprehensively. Our results show no single metric consistently excels across
all evaluations, revealing the complexity of assessing fine-grained support.
Based on the findings, we provide practical recommendations for developing
more effective metrics.

Towards Fine-Grained Citation Evaluation in Generated Text:
A Comparative Analysis of Faithfulness Metrics

Weijia Zhang1 Mohammad Aliannejadi1 Yifei Yuan2 Jiahuan Pei3 Jia-Hong Huang1
Evangelos Kanoulas1 1University of Amsterdam 2University of Copenhagen
3Centrum Wiskunde & Informatica<EMAIL_ADDRESS>

Figure 1: An example of partial support in citation evaluation. Inconsistent
metric scores are observed when assessing the statement with three
faithfulness metrics.

##  1 Introduction

Large language models (LLMs) often generate content known as “hallucinations”
Zhang et al. (2023), which contradicts established knowledge or lacks
verification from reliable sources. Mainstream studies Bohnet et al. (2022);
Gao et al. (2023b); Li et al. (2023) aim to mitigate this by using retrieval-
augmented LLMs to generate responses with in-line citations that provide
supporting evidence. One primary challenge is to assess how well a citation
supports its statement, as manual evaluation is labor-intensive and time-
consuming. Automated citation evaluation has been explored to reduce reliance
on human assessments Bohnet et al. (2022); Li et al. (2023). To this end,
faithfulness evaluation metrics are employed as proxies to automatically
estimate the citation support Gao et al. (2023a, b). These metrics measure the
faithfulness between model-generated and sourced text, which aligns closely
with the objectives of automated citation evaluation.

Prior studies in faithfulness metrics have primarily limited this task to a
binary classification problem Gao et al. (2023a, b), where faithfulness
metrics are leveraged to determine whether a citation supports the associated
statement. However, this binary approach fails to capture the fine-grained
citation support encountered in real-world applications. For instance, in
Figure 1, a retrieval-augmented LLM generates a response with multiple
citations given a query. A human assessor labels the first citation as
“partial support” since it only supports “the most humid place in Australia is
Macquarie Island” but not “which is located in the Southern Ocean off the
coast of Tasmania.” This partial support scenario causes noticeable
inconsistencies across three different faithfulness metrics. Therefore, there
is a significant research need to evaluate the effectiveness of faithfulness
metrics in accurately distinguishing citations in fine-grained support
scenarios.

To address this issue, we propose a comparative evaluation framework for
assessing the effectiveness of faithfulness metrics in fine-grained support
scenarios. In our framework, we define “support levels” as the extent to which
a citation supports the corresponding statement Liu et al. (2023); Yue et al.
(2023). We evaluate the effectiveness of faithfulness metrics in a three-
category support level scenario: full support, partial support, and no
support, indicating whether a citation provides full, partial, or no support
to the associated statement. To assess the metric effectiveness, we measure
the alignment between metric scores and human judgments with three types of
evaluation protocols:  1) Correlation analysis:we employ it to measure how
well metric scores correlate with human judgments.  2) Classification
evaluation:we conduct a classification evaluation to assess the metrics’
capability to distinguish citations based on their support levels.  3)
Retrieval evaluation:we undertake a retrieval evaluation to assess the
effectiveness of metrics in ranking citations according to their support
levels. This is motivated by the observation that the previous two evaluation
protocols assume citations are within statements, which is not always valid in
practice Liu et al. (2023). In such cases, faithfulness metrics are adapted to
perform post-hoc retrieval, aiming to retrieve potential citations from a
candidate pool Gao et al. (2023a, b). Thus, retrieval evaluation is crucial
for determining the practical utility of these metric adaptations.

In our experiments, we assess various widely used faithfulness metrics,
categorizing them into similarity-based, entailment-based, and LLM-based
metrics. We find that:  1) No single faithfulness metric consistently
outperforms others across three evaluation protocols, suggesting that these
protocols are complementary and should be integrated for a comprehensive
evaluation of metric performance;  2) The best-performing metrics show promise
in distinguishing some support scenarios but struggle with some others. This
highlights the inherent complexities of automated citation evaluation.  3)
Similarity-based metrics surpass best-performing entailment-based metrics in
retrieval evaluation. This indicates that entailment-based metrics exhibit
higher sensitivity to noisy data, which is introduced by irrelevant documents
in such scenarios.

Our contributions can be summarized as follows:

  * •

To the best of our knowledge, we are the first to systematically investigate
the effect of fine-grained support levels on faithfulness metrics in the task
of automated citation evaluation.

  * •

We propose a comparative evaluation framework to assess the alignment between
metric scores and human judgments. This framework includes three evaluation
protocols to comprehensively evaluate the metric performance.

  * •

Our experimental results demonstrate the best-performing faithfulness metrics
still struggle to identify partially supporting citations, underscoring the
inherent challenges of automated citation evaluation. Based on our findings,
we offer practical recommendations for the development of more effective
metrics.

##  2 Related Work

##### Faithfulness Evaluation Metrics

Faithfulness evaluation metrics are crucial for assessing the factual
consistency of text generated by models relative to the source text. It
receives great interest within the field of natural language generation (NLG)
Li et al. (2022); Ji et al. (2022), particularly in abstractive text
summarization Maynez et al. (2020); Kryscinski et al. (2020). In general,
faithfulness metrics are categorized into three types: entailment-based,
similarity-based, and QA-based metrics. Entailment-based metrics employ
natural language inference (NLI) models to determine if the source text
entails the generated text Falke et al. (2019); Kryscinski et al. (2020);
Laban et al. (2022); Honovich et al. (2022); Zha et al. (2023). Similarity-
based metrics, such as BERTScore Zhang et al. (2020) and BARTScore Yuan et al.
(2021), quantify text similarity and have demonstrated robust performance in
faithfulness evaluation Pagnoni et al. (2021); Honovich et al. (2022). QA-
based metrics utilize a combination of question generation and question
answering to estimate faithfulness levels Durmus et al. (2020); Wang et al.
(2020); Scialom et al. (2021); Honovich et al. (2021); Fabbri et al. (2022).
In this work, we exclude QA-based metrics from our work, following recent
works suggesting the challenging limitations in these metrics Kamoi et al.
(2023). We focus on the extrinsic evaluation of faithfulness metrics against
human judgments in scenarios requiring fine-grained citation support.

Figure 2: The overview of the proposed comparative evaluation framework. A
faithfulness metric assigns scores to given statements and their corresponding
citations. Subsequently, our framework comprehensively assesses the alignment
between these metric scores and human judgments by employing correlation
analysis, classification evaluation, and retrieval evaluation.

##### Citation Evaluation

Citation evaluation seeks to enhance the trustworthiness of retrieval-
augmented LLMs by verifying the support provided by citations to the generated
statements Rashkin et al. (2021); Bohnet et al. (2022); Gao et al. (2023a, b);
Yue et al. (2023). Given the labor-intensive nature of manual citation
evaluation, there has been a shift towards automated approaches to reduce
dependence on human evaluation. Since the goals of automated citation
evaluation align closely with faithfulness evaluation in NLG, faithfulness
metrics are employed to verify whether a citation supports the corresponding
statement Bohnet et al. (2022); Gao et al. (2023a, b); Li et al. (2024).
Despite their widespread usage, the effectiveness of these metrics in more
practical fine-grained citation support scenarios, such as those involving
partial support by citations, has not been adequately addressed. Questions
remain about the metrics’ capability to differentiate citations in these fine-
grained scenarios. This work addresses these gaps by examining the
effectiveness of faithfulness metrics across three distinct levels of citation
support: full, partial, and no support.

##  3 Evaluation Framework

In this section, we introduce the proposed comparative evaluation framework.
We begin by formalizing the task of automated citation evaluation.
Subsequently, we detail three distinct evaluation protocols within this
framework, ensuring a comprehensive assessment in alignment between
faithfulness metrics and human judgments. Our framework is demonstrated in
Figure 2.

###  3.1 Task Formulation

The objective of automated citation evaluation is to automatically quantify
the support level of a citation based on the citation and its associated
statement. In this work, we assume access to a dataset for automated citation
evaluation, comprising pairs of statements and their corresponding citations,
denoted as $(s_{i},c_{i})$. Each $s_{i}$ is a statement from the set $S$ of
all statements produced by an LLM and each $c_{i}$ is a citation from a set
$C$ of citations returned by the LLM. We categorize the citations into three
distinct levels of support: full, partial, and no support. We adopt the
definition of these support levels from Liu et al. (2023):

  * •

Full Support (FS): the citation fully supports every detail in the statement.

  * •

Partial Support (PS): the citation supports certain aspects of the statement,
while other details remain unsupported or are contradicted.

  * •

No Support (NS): none of the content in the statement is supported by the
citation. For instance, the citation is entirely irrelevant or contradicts the
statement.

To this end, without loss of generality, we define a faithfulness metric as a
scoring function, denoted as $F(s_{i},c_{i})\rightarrow R^{+}$. For any given
statement $s_{i}$ and its associated citation $c_{i}$, this scoring function
provides a numeric score that indicates the extent of support provided by the
citation to the statement.

###  3.2 Evaluation Protocols

The objective of evaluation protocols is to comprehensively assess the extent
to which metric scores align with human judgments. In this work, we assess
this alignment across three distinct dimensions: correlation, classification
performance, and retrieval effectiveness.

####  3.2.1 Correlation Analysis

The correlation analysis measures the general trend in the relationship
between metric scores and human judgments. Previous research Kryscinski et al.
(2020); Pagnoni et al. (2021) has employed correlation analysis to meta-
evaluate faithfulness metrics in abstractive text summarization. They involve
measuring the extent to which metric scores align with binary levels of
faithfulness, which are annotated by human assessors as either faithful ($1$)
or unfaithful ($0$). Inspired by them, we adapt correlation analysis to the
task of automated citation evaluation. Specifically, given the statements and
their associated citations, we assess how well predicted metric scores
correlate with human-annotated support levels. To facilitate correlation
analysis, we assign support levels $\\{\textsc{FS},\textsc{PS},\textsc{NS}\\}$
to values $\\{0,1,2\\}$. We then utilize standard correlation metrics to
assess metric performance. The details are shown in Section 5.2.

####  3.2.2 Classification Evaluation

In addition to correlation analysis, we perform classification evaluation to
determine the effectiveness of faithfulness metrics in discriminating
citations based on their support level. Specifically, the metrics need to
categorize a citation into one of three support levels: FS, PS, NS. Notably,
existing faithfulness metrics do not apply to this three-way classification
scenario, as they are unable to accurately determine the extent to which a
statement is partially supported by its corresponding citation Honovich et al.
(2022). To address this issue, we adopt a one-vs-one strategy, by effectively
decomposing the three-way classification into three binary classification task
settings:  (i) Full Support vs. No Support (FS-vs-NS),  (ii) Full Support vs.
Partial Support (FS-vs-PS), and  (iii) Partial Support vs. No Support (PS-vs-
NS).  For each binary classification task setting, we construct a specialized
dataset comprising only instances with the corresponding binary support levels
derived from the original dataset. We assess the performance of metrics on
these tailored binary datasets using standard binary classification evaluation
metrics. The overall metric performance is then computed by averaging the
results across all binary tasks.

####  3.2.3 Retrieval Evaluation

The objective of retrieval evaluation is to measure the effectiveness of
metrics in ranking citations according to their support levels. This
evaluation is motivated by the observation that previous correlation and
classification evaluations presuppose the presence of citations within
generated statements. However, real-world scenarios frequently present
instances where citations are absent or irrelevant, highlighting the need for
post-hoc retrieval to enhance citation quality Liu et al. (2023); Huang and
Chang (2023); Huang et al. (2024). In post-hoc retrieval, candidate documents
are retrieved to form a pool of potential citations using information
retrieval techniques Karpukhin et al. (2020). Faithfulness metrics are then
employed to rank citations based on their predicted metric scores, aiming to
identify the citation with the highest support level Gao et al. (2023a);
Bohnet et al. (2022); Gao et al. (2023b). Ideally, a faithfulness metric
should rank fully supporting citations at the top, followed by partially
supporting citations, and finally non-supporting citations. Similar to
correlation analysis, we assign support levels
$\\{\textsc{FS},\textsc{PS},\textsc{NS}\\}$ to relevance labels $\\{2,1,0\\}$.
The metric effectiveness is assessed using standard information retrieval
evaluation metrics. This evaluation also provides a deeper understanding of
metric performance in post-hoc citation retrieval scenarios.

##  4 Faithfulness Metrics

In our experiments, we evaluate diverse faithfulness evaluation metrics,
dividing them into similarity-based, entailment-based, and LLM-based metrics.
Similarity-based metrics assess the support levels mainly based on the degree
of similarity between the citation and the associated statement. Entailment-
based metrics leverage natural language inference (NLI) models Williams et al.
(2018) to estimate the support levels. LLM-based metrics directly prompt LLMs
to measure the support levels.

###  4.1 Similarity-Based Metrics

BERTScore Zhang et al. (2020) adopts BERT Devlin et al. (2019) to measure
semantic similarity between a pair of text by aggregating cosine similarity
among token-level BERT representation without further fine-tuning. We report
the precision version of BERTScore since it correlates more with human
judgments in faithfulness evaluation Pagnoni et al. (2021), We use recommended
deberta-xlarge-mnli He et al. (2021) as the backbone of BERTScore.

BARTScore Yuan et al. (2021) adopts BART Lewis et al. (2020) to measure the
similarity between two texts based on conditional log-likelihood of generating
target text from source text. In our experiments, we leverage the faithfulness
version of BARTScore, in which we treat the citation and the statement as the
source and target text, respectively. We use the BART model fine-tuned on the
CNN/DailyMail dataset Hermann et al. (2015) as the backbone of BARTScore.

Human Judgment | # Statement-Citation Pair
---|---
Full Support | 6,616
Partial Support | 1,445
No Support | 4,620
Total | 12,681
Table 1: Data statistics of the GenSearch dataset. Each pair has been
annotated by human assessors based on three categories: full, partial, and no
support.

###  4.2 Entailment-Based Metrics

FactCC Kryscinski et al. (2020) is a BERT-based model to verify whether a
generated text is faithful to a source text, which is fine-tuned on synthetic
training data containing simulated examples with different factual errors
Kryscinski et al. (2020).

SummaC Laban et al. (2022) is a RoBERTa-based model Liu et al. (2019) fine-
tuned on NLI datasets. This metric splits source and generated texts into
sentences, computes entailment scores for each pair and aggregates these
scores to obtain the final faithfulness score. It has two variants:  (i)
SummaCZS is a zero-shot version that is only pre-trained on NLI datasets;
(ii) SummaCConv adds extra convolutional layers and is further fine-tuned on
synthetic training data proposed in Kryscinski et al. (2020).

AutoAIS Honovich et al. (2022); Gao et al. (2023a) is a T5-11B Raffel et al.
(2020) model trained on a collection of NLI datasets, which is commonly used
in recent automated citation evaluation Bohnet et al. (2022); Gao et al.
(2023a). As the original output of AutoAIS is a numeric, either “1” (faithful)
or “0” (unfaithful), we use the generated token probability of “1” as the
predicted metric score.

AlignScore Zha et al. (2023) further fine-tunes a RoBERTa-based model Liu et
al. (2019) with a unified alignment loss function. To this end, a unified
dataset containing a variety of related natural language processing datasets
has been collected. In this work, we adapt the large version as it
demonstrates the best performance.

###  4.3 LLM-Based Metrics

In addition to established faithfulness metrics, we utilize large language
models (LLMs) as faithfulness evaluators for comparison. Specifically, we
introduce two prompting methods as follows:  (i) Discrete scoringprompts the
LLM to assign discrete scores from the set ${0,1,2}$ for a given statement and
its citation, where $0$, $1$, and $2$ indicate no support, partial support,
and full support, respectively;  (ii) Continuous scoringprompts the LLM to
assign continuous scores in the range $[0,1]$ for a given statement and its
citation. Here, $1$ indicates full support, $0$ indicates no support, and
values between 0 and 1 indicate partial support.

In the experiments, we employ the latest version of GPT-3.5
(gpt-3.5-turbo-0125) as the base model. Moreover, we utilize the chain of
thought (CoT) method Kojima et al. (2022) to enhance the reasoning
capabilities of the LLM. We use GPT-3.5-DIS and GPT-3.5-CON to denote GPT-3.5
using discrete and continuous scoring methods, respectively. The detailed
prompts are shown in Appendix A.

Metric Pearson Spearman Kendall LLM-based GPT-3.5-CON 0.023 0.057 0.035
GPT-3.5-DIS 0.101 0.181 0.128 Entailment-based FactCC 0.121 0.199 0.140
SummaCZS 0.364 0.180 0.137 SummaCConv 0.565 0.444 0.342 AlignScore 0.585 0.488
0.393 AutoAIS 0.638 0.639 0.547 Similarity-based BERTScore 0.542 0.227 0.170
BARTScore 0.598 0.235 0.176

Table 2: Correlation coefficients between human-annotated support levels and metric scores on the GenSearch dataset. The best and second-best correlations are marked in bold and underline. Category | Metric | FS-vs-NS | FS-vs-PS | PS-vs-NS | Overall
---|---|---|---|---|---
LLM-based | GPT-3.5-CON | 54.80 | 54.13 | 51.60 | 53.51
GPT-3.5-DIS | 57.84 | 52.79 | 55.48 | 55.37
1-6[1pt/1.5pt] Entailment-based | FactCC | 68.45 | 62.58 | 56.39 | 62.47
SummaCZS | 78.60 | 72.96 | 58.67 | 70.08
SummaCConv | 85.01 | 78.74 | 61.84 | 75.20
AlignScore | 90.79 | 81.41 | 69.78 | 80.66
AutoAIS | 92.61 | 82.31 | 73.90 | 82.94
1-6[1pt/1.5pt] Similarity-based | BARTScore | 87.43 | 75.42 | 71.34 | 78.07
BERTScore | 91.55 | 75.94 | 78.72 | 82.07
Table 3: Classification performance of faithfulness metrics regarding ROC-AUC
score (%) on the GenSearch dataset. The overall performance is the macro-
averaged performance of three binary classification settings. The best and
second-best scores are marked in bold and underline, respectively.

##  5 Experiments

###  5.1 Datasets

##### Data Statistics

In the experiments, we employ the dataset Liu et al. (2023) collected from
generative search engines (e.g., BingChat) as our evaluation benchmark,
referred to as GenSearch. This dataset comprises a total of $12,681$
statement-citation pairs. For each pair, human assessors categorize the
citation into one of three categories of support levels: full, partial, or no
support. The details of data statistics are shown in Table 1. Notably, for
citations classified under the full or partial support categories, human
assessors additionally extract explicit evidence from the citation that
supports the associated statement.

##### Data Processing

While the GenSearch dataset aligns well with our research objectives, we
encounter a significant challenge: the extensive length of most citations
within the dataset. These citations often comprise a web document with
thousands of words, far exceeding the maximum input capacity of most
faithfulness metrics, which is limited to $512$ words. This limitation
necessitates input truncation, potentially compromising the reliability of
faithfulness metrics. To mitigate this issue, we adopt a strategy similar to
previous studies Zha et al. (2023). Specifically, we segment each cited
document into shorter text chunks, with a maximum length of $150$ words per
chunk. These text chunks, along with their corresponding statements, serve as
the inputs for faithfulness metrics to predicted metric scores. Furthermore,
to determine human judgments for the text chunks, we employ the Jaccard
similarity index to identify text chunks containing human-annotated evidence,
classifying them as either fully or partially supporting text chunks.

###  5.2 Evaluation Metrics

We report Pearson, Spearman, and Kendall coefficients for correlation
analysis, as recommended by previous research Pagnoni et al. (2021). In terms
of classification evaluation, following previous studies Honovich et al.
(2022); Ma et al. (2023), we report the macro-averaged Receiver Operating
Characteristic-Area Under Curve (ROC-AUC) score, as it obviates the need for
manual threshold setting for each binary classification task. For retrieval
evaluation, we report standard normalized discounted cumulative gain (NDCG@n)
scores where $n\in\\{5,10,20\\}$.

Figure 3: Retrieval performance of faithfulness metrics regarding NDCG@n
scores on the GenSearch dataset. Note that we assign relevance labels $2$,
$1$, and $0$ to full, partial, and no support, respectively (shown in the
color).

##  6 Results and Analyses

In this section, we discuss the performance of faithfulness metrics across
three distinct evaluation protocols. Subsequently, we conduct a qualitative
analysis through case studies.

###  6.1 Correlation Results

The correlation results are demonstrated in Table 2. The following
observations can be made:  1) The best-performing metrics reveal moderate
correlations when analyzed using the Pearson coefficient. For instance,
AutoAIS achieves the highest Pearson coefficient, recording a value of
$0.638$, largely surpassing the second-best BARTScore, which posts a
coefficient of $0.598$.  2) There is notable variation in correlation trends
among high-performing metrics. BARTScore shows the second-best Pearson
correlation but much lower Spearman and Kendall correlations. This divergence
likely arises from the Pearson coefficient’s assumption of linear
relationships between two variables, which is often invalid in automated
citation evaluation.  3) Similarity-based metrics generally show lower
Spearman and Kendall correlations compared to Pearson. For instance, BERTScore
has a substantial Pearson correlation of $0.542$ but lower Spearman and
Kendall correlations of $0.227$ and $0.170$. This indicates that similarity-
based metrics do not align well with human judgments, highlighting their
limitations in fine-grained support scenarios.  4) LLM-based metrics show
little correlation with human judgments among all correlation coefficients,
with the correlation of the GPT-3.5-CON metric being almost zero. This finding
suggests a negligible relationship between LLM-based metric scores and human
judgments. Furthermore, the GPT-3.5-DIS metric significantly outperforms
GPT-3.5-CON, highlighting that more fine-grained support levels present
greater challenges in correlation analysis.

Error Reason |  Example
---|---
The citation does not explicitly mention coreference. |  Statement: Others believe that performance-enhancing drugs should be allowed in sports. Citation: However, if children are allowed to train as professional athletes, then they should be allowed to take the same drugs, provided that they are no more dangerous than their training is … Human Judgment: full support Metric Score: 0.055 (no support)
The complex statement includes independent claims. |  Statement: Love leads to growth while being in love is about ownership … Citation: “Growing to love the real person and accepting who they are, with both strengths and weaknesses, can make a wonderful difference in your relationship,” McCoy says … Human Judgment: partial support Metric Score: 0.0004 (no support)
The citation is semantically similar but non-supporting. |  Statement: Carpal tunnel syndrome can be treated with various methods, including wrist splinting, anti-inflammatory medication, and surgery. Citation: If diagnosed and treated early, the symptoms of carpal tunnel syndrome can often be relieved without surgery. If your diagnosis is uncertain or if your symptoms are mild, your doctor will recommend nonsurgical treatment first … Human Judgment: no support Metric Score: 0.52 (partial support)

Table 4: Case study of the faithfulness metric AutoAIS. Green phrases indicate
supported content in the statement and corresponding supporting evidence. Red
phrases indicate unsupported content in the statement and corresponding
misleading information in the citation.

###  6.2 Classification Results

Table 3 presents the results of the classification evaluation. The
observations can be summarized as follows:  1) Among all three binary
classification task settings, most faithfulness metrics demonstrate superior
performance in the FS-vs-NS setting. Notably, entailment-based AutoAIS
achieves the highest ROC-AUC score of $92.61$, which shows significant
discriminability between full support and no support instances. This can be
attributed to its much more extensive parameters compared to other entailment-
based metrics.  2) We observe the performance decline across the other two
settings (i.e. FS-vs-PS and PS-vs-NS). For instance, when comparing the FS-vs-
NS and PS-vs-NS settings, the ROC-AUC score of AutoAIS diminishes from $92.61$
to $73.90$. This decline indicates that even the best-performing metric
struggles with granular sensitivity to varying levels of support.  3) While
entailment-based AutoAIS generally surpasses other metrics, it is outperformed
by similarity-based BERTScore in the PS-vs-NS setting. Interestingly, while
most metrics perform worst in this setting, BERTScore shows its least
effectiveness in FS-vs-PS. This highlights the unique prediction behaviors of
different metrics across binary classification settings.  4) The performance
of LLM-based metrics significantly lags behind other metrics. For instance,
GPT-3.5-DIS achieves only a ROC-AUC score of $57.84$ in the FS-vs-NS setting,
markedly lower than the best-performing AutoAIS, which achieves a ROC-AUC
score of $92.61$. Furthermore, the overall performance of LLM-based metrics
approaches random guessing. This underscores the inefficacy of LLM-based
metrics in distinguishing fine-grained support levels.

###  6.3 Retrieval Results

Figure 3 presents the results of the retrieval evaluation. The key findings
are as follows:  1) Similarity-based metrics, BARTScore and BERTScore,
outperform other entailment-based metrics in all NDCG@n scores. For instance,
entailment-based AutoAIS exhibits weaker NDCG@5 scores than BARTScore. This is
likely because entailment-based metrics are more sensitive to noisy
information than similarity-based metrics, as many irrelevant documents exist
in retrieval scenarios. It suggests the need for the robustness improvements
of metrics in post-hoc retrieval scenarios.  2) The best-performing BERTScore
achieves more than twice the NDCG@n scores compared to LLM-based metrics. This
result suggests that LLM-based metrics are ineffective in ranking documents
with higher support levels. A plausible explanation is that LLM-based metrics
lack fine-grained sensitivity to variations in support levels. Interestingly,
our observations reveal that GPT-3.5-CON surpasses GPT-3.5-DIS, highlighting
the advantage of fine-grained scoring methods in retrieval evaluation.  3)
NDCG@n scores effectively capture the performance variations as the number of
text chunks increases. For instance, as the chunk count increases, BARTScore
shows a marginal performance improvement, while FactCC exhibits a more
pronounced enhancement.

###  6.4 Case Study

Table 4 presents three cases of AutoAIS. In the first example, where human
judgment indicates full support. AutoAIS incorrectly assigns a very low score.
This may be due to the lack of explicit mention of drug coreference in the
cited text chunk. This indicates coreference resolution remains a significant
challenge in automated citation evaluation. In the second example, where human
judgment indicates partial support. The complex statement implicitly contains
two independent claims that require verification. However, the provided
citation fails to offer sufficient evidence, resulting in an almost zero
metric score. In the third example, where human judgment indicates no support.
The given citation is semantically similar to the statement, leading to a
metric score of partial support. Despite this semantic similarity, specific
treatments mentioned in the statement, such as wrist splinting, are not
explicitly referenced in the citation.

##  7 Discussions

Overall, our results indicate that the evaluation protocols are complementary
and should be integrated for a comprehensive assessment of metrics. Based on
the evaluation results, we further propose the following practical
recommendations to develop more effective metrics for automated citation
evaluation:  1) Development of training resources:motivated by the observation
that the best-performing metrics still struggle with identifying partial
support, we recommend the development of training resources that include fine-
grained support level annotations. These resources could significantly enhance
the metrics’ fine-grained sensitivity to varying support levels;  2)
Introduction of contrastive learning:to improve the robustness of metrics in
post-hoc retrieval scenarios, we recommend fine-tuning metrics using
contrastive learning frameworks. This method has demonstrated effectiveness
across various information retrieval tasks Izacard et al. (2022).  3)
Development of more explainable metrics:traditional faithfulness metrics often
only provide final scores without sufficient explainability. This limitation
hinders a deeper understanding of the models’ behavior. Therefore, it is
crucial to develop more explainable faithfulness metrics, potentially using
large language models (LLMs).

##  8 Conclusion

We propose a comparative evaluation framework to explore the efficacy of
faithfulness metrics beyond the binary scenario by examining three levels of
citation support. Our framework employs correlation analysis, classification
evaluation, and retrieval evaluation to measure the alignment between metric
scores and human judgments. Experimental results reveal that no single metric
consistently excels across all evaluation protocols, indicating the complexity
of automated citation evaluation and the limitations of existing faithfulness
metrics. We provide practical suggestions based on the findings.

## Limitations

In this work, we consider a citation that explicitly contains human-annotated
evidence as the fully supporting citation for each statement. However, for
some complex statements, their evidence is distributed among multiple
citations. For instance, about $2\%$ statements on the GenSearch dataset
require multiple citations to be fully supported. Also, we focus on statement-
level citation evaluation. Since answer-level citation evaluation is much more
complicated and requires proper aggregation methods, we leave this exploration
as future work. We do not evaluate QA-based faithfulness metrics as a recent
study shows that such metrics have some fundamental issues, such as failing to
localize errors Kamoi et al. (2023). However different findings could be
explored with QA-based metrics.

## Ethical Considerations

We realized there are some risks in exploring citation evaluation for LLM-
generated text. Since we have used publicly available datasets and open-source
implementation of faithfulness metrics, we carefully avoid potential ethical
problems caused by datasets or open-source codes. As we address the issue of
the effectiveness of faithfulness metrics for LLM-generated text, concerning
hallucination. We acknowledged the hallucinated text generated by LLMs may
contain potentially harmful or misleading information. Our final goal is to
mitigate such hallucination issues, which should support the discussion around
hallucinations of LLMs and all ethical aspects around them.

## References

  * Bohnet et al. (2022) Bernd Bohnet, Vinh Q Tran, Pat Verga, Roee Aharoni, Daniel Andor, Livio Baldini Soares, Jacob Eisenstein, Kuzman Ganchev, Jonathan Herzig, Kai Hui, Tom Kwiatkowski, Ji Ma, Jianmo Ni, Tal Schuster, William W Cohen, Michael Collins, Dipanjan Das, Donald Metzler, Slav Petrov, and Kellie Webster. 2022.  Attributed Question Answering: Evaluation and Modeling for Attributed Large Language Models.  abs/2212.08037.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_ , pages 4171–4186. Association for Computational Linguistics.
  * Durmus et al. (2020) Esin Durmus, He He, and Mona Diab. 2020.  FEQA: A question answering evaluation framework for faithfulness assessment in abstractive summarization.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5055–5070, Online. Association for Computational Linguistics.
  * Fabbri et al. (2022) Alexander Fabbri, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong. 2022.  QAFactEval: Improved QA-Based factual consistency evaluation for summarization.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2587–2601, Seattle, United States. Association for Computational Linguistics.
  * Falke et al. (2019) Tobias Falke, Leonardo F. R. Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. 2019.  Ranking generated summaries by correctness: An interesting but challenging application for natural language inference.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 2214–2220, Florence, Italy. Association for Computational Linguistics.
  * Gao et al. (2023a) Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, and Kelvin Guu. 2023a.  RARR: Researching and revising what language models say, using language models.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 16477–16508, Toronto, Canada. Association for Computational Linguistics.
  * Gao et al. (2023b) Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. 2023b.  Enabling large language models to generate text with citations.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 6465–6488, Singapore. Association for Computational Linguistics.
  * He et al. (2021) Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021.  Deberta: Decoding-Enhanced Bert with Disentangled Attention.  In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net.
  * Hermann et al. (2015) Karl Moritz Hermann, Tomás Kociský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015.  Teaching machines to read and comprehend.  In _Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada_ , pages 1693–1701.
  * Honovich et al. (2022) Or Honovich, Roee Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kukliansy, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and Yossi Matias. 2022.  TRUE: Re-evaluating factual consistency evaluation.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3905–3920, Seattle, United States. Association for Computational Linguistics.
  * Honovich et al. (2021) Or Honovich, Leshem Choshen, Roee Aharoni, Ella Neeman, Idan Szpektor, and Omri Abend. 2021.  Q2: Evaluating factual consistency in knowledge-grounded dialogues via question generation and question answering.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 7856–7870, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Huang et al. (2024) Chengyu Huang, Zeqiu Wu, Yushi Hu, and Wenya Wang. 2024.  Training language models to generate text with citations via fine-grained rewards.  _CoRR_ , abs/2402.04315.
  * Huang and Chang (2023) Jie Huang and Kevin Chen-Chuan Chang. 2023.  Citation: A key to building responsible and accountable large language models.  _CoRR_ , abs/2307.02185.
  * Izacard et al. (2022) Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022.  Unsupervised dense information retrieval with contrastive learning.  2022\.
  * Ji et al. (2022) Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Andrea Madotto, and Pascale Fung. 2022.  Survey of hallucination in natural language generation.
  * Kamoi et al. (2023) Ryo Kamoi, Tanya Goyal, and Greg Durrett. 2023.  Shortcomings of question answering based factuality frameworks for error localization.  In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 132–146, Dubrovnik, Croatia. Association for Computational Linguistics.
  * Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020.  Dense passage retrieval for open-domain question answering.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6769–6781. Association for Computational Linguistics.
  * Kojima et al. (2022) Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022.  Large language models are zero-shot reasoners.  In _NeurIPS_.
  * Kryscinski et al. (2020) Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020.  Evaluating the factual consistency of abstractive text summarization.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 9332–9346, Online. Association for Computational Linguistics.
  * Laban et al. (2022) Philippe Laban, Tobias Schnabel, Paul N. Bennett, and Marti A. Hearst. 2022.  SummaC: Re-visiting NLI-based models for inconsistency detection in summarization.  _Transactions of the Association for Computational Linguistics_ , 10:163–177.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_ , pages 7871–7880. Association for Computational Linguistics.
  * Li et al. (2022) Wei Li, Wenhao Wu, Moye Chen, Jiachen Liu, Xinyan Xiao, and Hua Wu. 2022.  Faithfulness in Natural Language Generation: A Systematic Survey of Analysis, Evaluation and Optimization Methods.  abs/2203.05227.
  * Li et al. (2023) Xiaonan Li, Changtai Zhu, Linyang Li, Zhangyue Yin, Tianxiang Sun, and Xipeng Qiu. 2023.  LLatrieval: LLM-Verified retrieval for verifiable generation.  _CoRR_ , abs/2311.07838.
  * Li et al. (2024) Yifei Li, Xiang Yue, Zeyi Liao, and Huan Sun. 2024.  AttributionBench: How hard is automatic attribution evaluation? _CoRR_ , abs/2402.15089.
  * Liu et al. (2023) Nelson Liu, Tianyi Zhang, and Percy Liang. 2023.  Evaluating verifiability in generative search engines.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 7001–7025, Singapore. Association for Computational Linguistics.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  RoBERTa: A robustly optimized BERT pretraining approach.  abs/1907.11692.
  * Ma et al. (2023) Liang Ma, Shuyang Cao, Robert L Logan IV, Di Lu, Shihao Ran, Ke Zhang, Joel Tetreault, and Alejandro Jaimes. 2023.  BUMP: A benchmark of unfaithful minimal pairs for meta-evaluation of faithfulness metrics.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 12788–12812, Toronto, Canada. Association for Computational Linguistics.
  * Maynez et al. (2020) Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020.  On faithfulness and factuality in abstractive summarization.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1906–1919, Online. Association for Computational Linguistics.
  * Pagnoni et al. (2021) Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. 2021.  Understanding factuality in abstractive summarization with FRANK: A benchmark for factuality metrics.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4812–4829, Online. Association for Computational Linguistics.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, and et al. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  21:140:1–140:67.
  * Rashkin et al. (2021) Hannah Rashkin, Vitaly Nikolaev, Matthew Lamm, Michael Collins, Dipanjan Das, Slav Petrov, Gaurav Singh Tomar, Iulia Turc, and David Reitter. 2021.  Measuring Attribution in Natural Language Generation Models.  abs/2112.12870.
  * Scialom et al. (2021) Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, Alex Wang, and Patrick Gallinari. 2021.  QuestEval: Summarization asks for fact-based evaluation.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 6594–6604, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Wang et al. (2020) Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020.  Asking and answering questions to evaluate the factual consistency of summaries.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5008–5020, Online. Association for Computational Linguistics.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1112–1122. Association for Computational Linguistics.
  * Yuan et al. (2021) Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021.  BARTScore: Evaluating Generated Text as Text Generation.  In _Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, Virtual_ , pages 27263–27277.
  * Yue et al. (2023) Xiang Yue, Boshi Wang, Ziru Chen, Kai Zhang, Yu Su, and Huan Sun. 2023.  Automatic evaluation of attribution by large language models.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 4615–4635, Singapore. Association for Computational Linguistics.
  * Zha et al. (2023) Yuheng Zha, Yichi Yang, Ruichen Li, and Zhiting Hu. 2023.  AlignScore: Evaluating factual consistency with A unified alignment function.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 11328–11348, Toronto, Canada. Association for Computational Linguistics.
  * Zhang et al. (2020) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2020.  BERTScore: Evaluating Text Generation with BERT.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
  * Zhang et al. (2023) Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, Longyue Wang, Anh Tuan Luu, Wei Bi, Freda Shi, and Shuming Shi. 2023.  Siren’s song in the AI ocean: A survey on hallucination in large language models.  _CoRR_ , abs/2309.01219.

##  Appendix A Details of Prompts

Details of Prompts are shown in Table 5.

Prompt Name | Prompt Content
---|---
Discrete Scoring |  Instruction: Your task is to quantify how well a provided citation supports a given statement. You should predict a discrete score from the set $\\{0,1,2\\}$, where $0$, $1$, $2$ represent that the statement is not supported, partially supported, and fully supported, respectively. Let’s think step by step. Statement: statement Citation: cited text chunk Prediction:
Continuous Scoring |  Instruction: Your task is to quantify how well a provided citation supports a given statement. You should predict a continuous score between 0 and 1 (inclusive), where 0 is not supported, 1 is fully supported, and a float value between 0 and 1 is partially supported. Let’s think step by step. Statement: statement Citation: cited text chunk Prediction:

Table 5: Prompts for discrete and continuous scoring methods.