Review

# Offline Metrics for Evaluating Explanation Goals in Recommender Systems

André Levi Zanon<EMAIL_ADDRESS>0003-0526-2678 ,  A. L. Zanon
<EMAIL_ADDRESS>Universidade de São PauloAvenida Trabalhador São-carlense,
400 - CentroSão CarlosSão PauloBrasil13566-590 Samsung R&D InstituteAv.
Cambacicas, 1200 - Bloco I CampinasSão PauloBrasil13097-104 ,  Leonardo Rocha
<EMAIL_ADDRESS>Universidade Federal de São João del-ReiPraça Frei
Orlando, 170São João del ReiMinas GeraisBrasil36307-352 and  Marcelo Garcia
Manzato<EMAIL_ADDRESS>Universidade de São PauloAvenida Trabalhador
São-carlense, 400 - CentroSão CarlosSão PauloBrasil13566-590

(2018)

###### Abstract.

Explanations are crucial for improving users’ transparency, persuasiveness,
engagement, and trust in Recommender Systems (RSs). However, evaluating the
effectiveness of explanation algorithms regarding those goals remains
challenging due to existing offline metrics’ limitations. This paper
introduces new metrics for the evaluation and validation of explanation
algorithms based on the items and properties used to form the sentence of an
explanation. Towards validating the metrics, the results of three state-of-art
post-hoc explanation algorithms were evaluated for six RSs, comparing the
offline metrics results with those of an online user study. The findings show
the proposed offline metrics can effectively measure the performance of
explanation algorithms and highlight a trade-off between the goals of
transparency and trust, which are related to popular properties, and the goals
of engagement and persuasiveness, which are associated with the
diversification of properties displayed to users. Furthermore, the study
contributes to the development of more robust evaluation methods for
explanation algorithms in RSs.

Recommender Systems, Explainability, Recommendation explanation, Evaluation,
Recommendation evaluation

††copyright: acmcopyright††journalyear: 2018††doi: XXXXXXX.XXXXXXX††journal:
JDS††journalvolume: 37††journalnumber: 4††article: 111††publicationmonth: 8

##  1\. Introduction

Recommender systems (RSs) provide personalized suggestions based on users’
past interactions and preferences. As the amount of information increases, RSs
architectures have become increasingly more complex regarding the modeling of
underlying relations among users, items, and metadata (Nunes and Jannach,
2017). Consequently, the reason why a recommendation is provided can be
unclear to the users. In this sense, explaining the output of RSs gained
significance among researchers towards a more personalized experience to users
(Tintarev and Masthoff, 2015).

Explanation goals rely on the persuasiveness and transparency of the system
for enhancing users’ trust and engagement (Balog and Radlinski, 2020; Tintarev
and Masthoff, 2015). However, measuring these four explanation goals is not
trivial, since they should reflect the extent to which explanations reveal how
a recommendation is generated (Zhang et al., 2020). Transparency is defined as
a user’s understanding of a recommendation; persuasiveness is the influence of
an explanation over making the user interact with an item; engagement is
related to the discovery of new information about a suggestion; and trust is
the user’s perception of the reliability of the RS.

In this context, the first research question investigated in this paper was:
(RQ1): How are explanations in RSs evaluated in the literature?. According to
our literature review, the explanations’ quality is indirectly evaluated in
most studies, since they rely on other contributions, such as improvements in
ranking accuracy and diversity (Nunes and Jannach, 2017). Moreover, some
explanation goals are evaluated through online studies, which can be time-
consuming and resource-intensive, requiring the development of a proposal, an
explainable algorithm, a user application, and an experimental protocol for
the control of user and system bias.

The second research question was (RQ2): Can an offline evaluation be provided
for explanations in RSs?. To answer this question, we propose 11 offline
metrics that consider persuasiveness, transparency, engagement, and trust and
use them to compare three state-of-art algorithms to generate explanations in
RSs. Two of these algorithms were also evaluated in an online experiment,
where their results were compared to those obtained from the offline metrics,
showing that the offline metrics provide a comparison between explanation
algorithms regarding explanation goals. We also identify a trade-off between
the goals of transparency and trust, related to popular properties, and
engagement and persuasiveness, related to diverse properties that form an
explanation.

Based on our comprehensive literature review and to the best of our knowledge,
this paper is the first to propose and compare offline metrics for explanation
goals such as transparency, engagement, trust, and persuasiveness. The
findings highlight the need for more robust and efficient evaluation methods
that consider explanation goals for providing a reliable measure of the
quality of explainable recommendations (Zhang et al., 2020; Balog and
Radlinski, 2020). Therefore, the main contributions of this paper are:

  * •

A survey on current offline metrics for evaluations of explanations in offline
settings;

  * •

A proposal of offline explanation metrics for assessing explanation goals in
recommender systems;

  * •

A discussion on explanation metrics in offline and online settings and a
robust evaluation of those explanations; and

  * •

Guidelines for the evaluation of explanations in recommender systems with
offline experiments and open research directions in the field.

The paper is structured as follows: Section 2 provides a literature review of
offline metrics for the evaluation of explanations in recommender systems;
Section 3 introduces the metrics proposed along with their motivation; Section
4 describes the methodology for their validation; Section 5 reports the
results; finally, Section 6 is devoted to conclusions and open directions for
offline metrics to explain recommender systems.

##  2\. Literature Review

The literature review presented in this section aims to answer (RQ1). The
search query was designed to retrieve papers that contained the keywords
‘explainable recommender systems’, ‘explanations’, and ‘explainability’ from
major conferences and journals in the RS community, indexed by distinguished
publishers such as ACM, IEEE, and Elsevier. The criteria for the selection of
papers were the number of citations and recency, since recently published
papers may have few citations.

Algorithms that generate explanations in RSs are divided into three main
approaches, namely explainable recommendation engines (Melchiorre et al.,
2022; Zhao et al., 2020; Wang et al., 2018; Xian et al., 2019; Balloccu et
al., 2023; Tai et al., 2021), explainable reordering approaches (Balloccu et
al., 2022; Zanon et al., 2022), and post-hoc explanations (Musto et al.,
2019b; Hada and Shevade, 2021; Musto et al., 2016; Du et al., 2022). Whereas
explainable recommendation engines justify the generation of a suggestion by a
recommendation algorithm, post-hoc algorithms consider relations between a
user’s past interactions and recommendations to justify the relevance of a
suggestion to the user. On the other hand, explainable reordering approaches
reorder recommendations from an engine towards prioritizing recommendations
with more convincing explanations.

Concerning explainable recommendation engines, we highlight the works of
(Melchiorre et al., 2022) and (Zhao et al., 2020). (Melchiorre et al., 2022)
created ProtoMF, a collaborative filtering algorithm that extends Matrix
Factorization methods by extracting prototypes, which are representative
entities from users and items, and are used in the explanation based on their
influence on the user-item pair. (Zhao et al., 2020) adopted an adversarial
actor-critic reinforcement learning over a Knowledge-Graph (KG) to obtain the
best paths based on user interactions and accurate recommendations and
explanations. In both papers, the proposals are evaluated according to their
recommendation ability rather than the power of explanation. Explanations are
only exemplified demonstrating the engine’s ability to provide them.
Consequently, similarly to recommendation engines that suffer from long-tail
distribution by recommending only the most interacted items, explanation
algorithms from explainable engines may also learn specific paths for the most
popular items or properties. This issue is recurrent in several explainable
recommendation engines (Wang et al., 2018; Xian et al., 2019; Balloccu et al.,
2023; Tai et al., 2021).

For explainable reordering approaches, (Balloccu et al., 2022) created a KG
reordering one based on three explanation properties, namely recency of
interacted items, i.e., explanations based on the last user interactions,
popularity of entities, and diversification of explanation types, with a
weighted optimization, to measure the quality of explanations. (Zanon et al.,
2022) also developed a reordering approach based on the best explanation
considering weighted paths between the items the user interacted with and
those recommended. The weights of the paths were measured according to a Term
Frequency–Inverse Document Frequency (TF-IDF) of KG properties extracted from
Wikidata. The evaluations of both algorithms considered accuracy and beyond-
accuracy metrics such as diversity and fairness; however, the explanations
were not evaluated with online experiments, since they only presented how
explanations could be generated.

Regarding post-hoc explainable approaches, (Musto et al., 2019b) used Natural
Language Processing (NLP) sentiment analysis and aspect extraction to provide
a summarized text from previous reviews and justify explanations. On the other
hand, (Hada and Shevade, 2021) generated personalized reviews for users as
explanations with an attention-based parallel network called cross-attention
for the selection of candidate users and item reviews for the construction of
the final sentence. The authors compared the ground-truth text with the
generated review to evaluate the explanations. (Tan et al., 2021) produced a
counterfactual explanation for the recommendation algorithm as a black box
with a soft optimization method sensitive to changes to the item’s aspects via
solving a counterfactual optimization. Using KG-based explanations, rather
than text justifications, (Musto et al., 2016, 2019b) and (Du et al., 2022)
generated explanations based on ranking properties from a KG of the users’
interacted items. Whereas (Musto et al., 2016, 2019b) proposed a score based
on the number of links between recommended and interacted items and the IDF of
the property, (Du et al., 2022) calculated a relevance score for each property
by dividing the number of interacted items that have that property by the
total number of items with the same property. The authors then applied a
logarithmic function to the score to penalize uncommon properties and rank the
most relevant ones. Post-hoc explanations are usually evaluated with online
studies under transparency, persuasiveness, engagement, and trust (Du et al.,
2022; Musto et al., 2016, 2019b, 2019a). For NLP approaches, another metric is
to compare users’ reviews with the output explanation of the proposed
algorithm based on precision and recall of extracted properties (Hada and
Shevade, 2021; Tan et al., 2021). However, this metric is exclusive to NLP
explanations and limits the justification to the set of properties extracted
from a user. Consequently, users that have not evaluated a large number of
items are defined only by little information, limiting the discovery of new
information about an item.

Our search also identified a work (Coba et al., 2022) that proposed a
framework for the development and evaluation of explanations in RSs. The
implemented metrics were Mean Explainability Precision, which measures the
number of explainable items for a user, Model Fidelity, which evaluates
recommendations with proxy predictions, and Explanation Score, which measures
the number of interactions that support an explanation. However, all such
metrics regard the algorithm’s robustness in generating explanations in
contrast to its quality and user perception.

According to the literature review explanations and to answer RQ1, explainable
algorithms have not been evaluated regarding the quality and goals of
explanations. As a result, current explanation metrics do not account for
explanation goals or are not evaluated with an online study, which can be
complex, since they require the development of a user application and
recruiting of participants for evaluating and comparing explanations. The next
section introduces the proposal for filling this literature gap.

##  3\. Proposal of Offline Metrics

Metric | Equation | Metric | Equation
---|---|---|---
(1) Total Item Diversity | $TID(S)=\left|\bigcup_{e\in E}L_{i_{e}}^{S}\right|$ | (7) Item Gini | $IGini(S)=\frac{1}{|I|-1}\sum_{k=1}^{|I|}(2k-|I|-1)p(i_{k}|L_{i_{e}}^{S})$
(2) Total Property Diversity | $TPD(S)=\left|\bigcup_{e\in E}L_{p_{e}}^{S}\right|$ | (8) Property Gini | $PGini(S)=\frac{1}{|P|-1}\sum_{k=1}^{|P|}(2k-|P|-1)p(i_{k}|L_{p_{e}}^{S})$
(3) Mean Item Diversity | $MID(S)=mean(\forall_{e\in E_{u}}L_{i_{e}}^{S})$ | (9) Linking Interaction Recency | $LIR(p^{i},t^{i})=(1-\beta)*LIR(p^{i-1},t^{i-1})+\beta*t^{i}$
(4) Mean Property Diversity | $MPD(S)=mean(\forall_{e\in E_{u}}L_{i_{p}}^{S}))$ | (10) Shared Entity Popularity | $SEP(e^{i},v^{i})=(1-\beta)*SEP(e^{i-1},v^{i-1})+\beta*v^{i}$
(5) Item Entropy | $IEnt(S)=-\sum_{e\in E}p(e|L_{i_{e}}^{S})\log p(i|L_{i_{e}}^{S})$ | (11) Explanation Type Diversity | $ETD(S)=\frac{|\omega_{L_{u}}|}{min(k,|\omega_{L}|)}$
(6) Property Entropy | $PEnt(S)=-\sum_{p_{e}\in E}p(e|L_{p_{e}}^{S})\log p(i|L_{p_{e}}^{S})$ |  |
Table 1. Table of proposed metrics to evaluate explanation goals with offline
experiments.

Explanations in RSs typically consist of three components, namely, (i)
recommended items produced by the recommendation engine; (ii) the user’s
interacted items associated with the suggestion; and (iii) the property that
links the recommendation and the user’s interacted items. Towards filling the
gap in the evaluation of the effectiveness of explanation algorithms according
to the goals of explanations, 11 metrics, shown in Table 1, are proposed to
evaluate the diversity and popularity of properties and items of explanations
in an offline setting, since they play a critical role in the users’
perception of explanations (Balloccu et al., 2022; Kouki et al., 2019).

Because the engagement goal of explanations accounts for the discovery of new
information (Tintarev and Masthoff, 2015), diversifying explanations can
effectively increase user engagement. Firstly proposed by (Adomavicius and
Kwon, 2011), the aggregate diversity of an RS algorithm $S$ accounts for the
number of items exposed to all users. In that regard, adapting this metric, we
propose the Total Items Aggregate Diversity ($TID$) and Total Property
Aggregate Diversity ($TPD$), defined by Equations (1) and (2) of Table 1,
respectively. Term $E$ is the set of all explanations for all users and $e$ is
an explanation in $E$, $L_{i_{e}}^{S}$ is the set of profile items used for
the explanations, and $L_{p_{e}}^{S}$ is the set of properties used for the
explanations. Similarly to aggregate diversity in items, the idea behind $TID$
and $TPD$ is to verify the total number of items/properties shown in
explanations, and consequently, if the explanation algorithm is biased toward
a few popular items and properties from the dataset to all users, which may
contain relevant information to most users.

Complementing those metrics, even if the system can provide a large or low
number of items/properties to explanations across all users, it can also be
biased towards a single user’s popular properties and interacted items. This
happens when different items/properties are shown to different users, but for
a single user, the same item/property is shown multiple times due to its
popularity in the user’s historical items. Such an aspect can lower the goals
of persuasiveness and transparency, since explanations will be repetitive and
will not reflect the user’s different tastes. Therefore, it is difficult,
hence, less convincing, for the user to understand the reason for a
recommendation. To this end, we also propose Mean Item Diversity ($MID$) and
Mean Property Diversity ($MPD$) metrics as the mean quantity of items and
properties shown in the set of $E_{u}$ explanations shown for each user $u$ in
the set $U$ of all users. The metrics are represented by Equations (3) and (4)
of Table 1. Moreover, the use of the standard deviation is proposed for those
distributions for verifying the algorithm’s consistency in diversifying items
and properties in explanations across users.

Other metrics for the evaluation of fairness are Entropy and Gini (Zanon et
al., 2022; Kunaver and Požrl, 2017), which measure bias in the distribution
provided by $S$ for the distribution of items ($L_{i_{e}}^{S}$) and properties
($L_{p_{e}}^{S}$). When items/properties are not equally distributed across
explanations, Entropy is low and Gini is high, meaning that few
items/properties are shown many times, and many items/properties are shown
very often. Such aspects can negatively affect engagement since
diversification is not achieved due to an unfair distribution. Therefore, we
propose to adapt Entropy and Gini based on i) interacted items ($IEnt$ and
$IGini$), represented, respectively by Equations (5) and (7) in Table 1, and
ii) properties of the user ($PEnt$ and $PGini$), represented respectively by
Equations (6) and (8) of Table 1. Term $I$ refers to the set of items and $P$
refers to the set of properties.

Since popular properties and recently interacted items are more likely to be
known and consequently increase the trust in RSs, three optimization metrics
used in (Balloccu et al., 2022) as evaluation metrics, namely Shared Entity
Popularity ($SEP$), Explanation Type Diversity ($ETD$), and Linking
Interaction Recency ($LIR$) were adapted. $LIR$ and $SEP$ metrics, displayed,
respectively, in Equations (9) and (10) of Table 1, are the mean of the
normalized exponentially weighted moving average for timestamp ($t$) of the
interacted item ($p$) shown in the explanations; and number of times ($v$) a
property ($e$) is referenced in the graph. Term $i$ is the index of the
item/property of the ordered array based on the timestamp and popularity of
items and properties and $\beta$ is a parameter set as 0.3, according to
(Balloccu et al., 2022). $ETD$ metric (Equation (11) in Table 1) should be
highly correlated with our diversity metrics and is the number of unique
properties in explanations ($\omega_{L_{u}}$) divided by the minimum between
the size of the recommendation list $k$ and possible explanations properties
$\omega_{L}$. They account, respectively, for the recency of items, the
popularity of attributes, and the diversity of attributes in explanations and
are important aspects for relevant explanations according to users (Balloccu
et al., 2022).

##  4\. Evaluation Methodology

###  4.1. Offline Experiments

Three post-hoc algorithms, ExpLOD, entitled and proposed by (Musto et al.,
2016), ExpLOD v2, proposed in (Musto et al., 2019b), and Property Based
Explanation Model (PEM), developed by (Du et al., 2022) were implemented for
validating the metrics. All of them are similar and the explanation sentence
is based on the ranking of attributes of common KG properties between
historical and recommended items. KG post-hoc algorithms were chosen because
differently from NLP, KG approaches do not require user data, which would
limit the number of users in the dataset, thus affecting evaluation, and the
item recommended does not interfere with the explanation algorithm.
Consequently, an explanation algorithm can be directly compared to another
without the effect of a re-ranking. Furthermore, according to the results of
online experiments from the literature, the state-of-the-art in knowledge
graph post-hoc explanation algorithms has significantly evolved and shows a
well-defined order of proposals: ExpLOD v2 was developed as an improvement
over ExpLOD, validating the results through online experiments. Similarly,
PEM, the most recent algorithm, builds upon ExpLOD v2 and was validated
through online experiments. Results indicate PEM offers enhanced transparency,
persuasiveness, engagement, trust, and effectiveness and, is, therefore, the
current state-of-the-art

The methods ExpLOD and ExpLOD v2 (Musto et al., 2016, 2019b) rank properties
of the KG based on Equation 1 ($score\\_explod$), where the number of links of
the property $p$ to historic items ($I_{u}$) and recommended items ($I_{r}$),
are represented, respectively, by the terms $n_{p,I_{u}}$ and $n_{p,I_{r}}$.
Such factors are weighted by $\alpha$ and $\beta$, which are arbitrary and
multiplied by the inverse-document frequency of $p$ ($IDF(p)$). The only
difference between the first and second approaches is that broader properties
from the KG hierarchy are also considered in the latter as the sum of Equation
1 to child properties of the broader properties. The method PEM is proposed in
(Du et al., 2022) and represented by $score\\_pem$ (Equation 2). In comparison
to ExpLOD, it changes the formula to account for the number of items instead
of the number of links for the sets of user historical and catalog items
($I(p,I_{u})$ and $I(p,C)$, respectively). It also changes the $IDF(p)$ term
to the logarithmic function on the catalog ($I(p,C)$). The term
$\overline{I(p,I_{u})}$ accounts for the number of items described directly or
by the hierarchy of an item, and $I(p,I_{u})$ accounts only for direct
connections.

(1)
$score\\_explod(p,I_{u},I_{r})=(\alpha\frac{{n_{p,I_{u}}}}{|I_{u}|})+(\beta\frac{n_{p,I_{r}}}{{|I_{r}|}})*IDF(p)$

(2)
$score\\_pem(p,I_{u},I_{r},C)=\frac{|\overline{I(p,I_{u})}|/|I_{u}|}{|\overline{I(p,C)}|/|C|}*\log(|I(p,C)|)$

The methodology and reproducibility guidelines of (Ferrari Dacrema et al.,
2021) guided the evaluation of explanations with the proposed metrics.
Therefore, the three explanation algorithms were run with the following
recommendation algorithms using 10-fold cross-validation: the Most Popular
(Cremonesi et al., 2010) algorithm was adopted for a non-personalized
recommender, Personalized Page Rank algorithm (Musto et al., 2016) with
Wikidata graph enriched with the dataset users as nodes were used for graph-
based approach; for a KNN algorithm it was used the User-KNN (Resnick et al.,
1994); for non-neural algorithms, the Embarrassingly Shallow AutoEncoder
(EASE) (Steck, 2019) and the Bayesian Personalized Ranking Matrix
Factorization (BPR-MF) (Rendle et al., 2009) were adopted; and for neural-
based architectures, Neural Collaborative Filtering (NeuMF) algorithm was
chosen (He et al., 2017).

Most Popular, User-KNN, and BPR-MF were executed using the library proposed by
(da Costa et al., 2018); whereas EASE, NeuMF, and PageRank were implemented
according to the corresponding papers and are available in our public
repository111https://github.com/andlzanon/lod-personalized-recommender. All
parameters used were the original ones and the explanation was formed with the
top-ranked property connecting a maximum of three interacted items to each
recommendation. MovieLens 100k (Harper and Konstan, 2015) dataset was chosen
for the offline experiments due to the high computational cost for computing
the metrics of all explanation algorithms for the top-1 and top-5
recommendations of every algorithm on every fold. The data extracted from
movies is from Wikidata222https://www.wikidata.org since it is more up-to-date
and has more completeness of data compared to the
DBPedia333https://www.dbpedia.org/ (Pillai et al., 2019). Descriptive (e.g.,
box office data) and identification links (e.g., id on IMDb) were removed for
the retrieval of data from Linked Open Data (LOD) since they are unique to
items. The coverage of items that data was obtained was 97%.

###  4.2. Online Experiments

To validate the metrics based on the user perception of explanation goals and
associate them with the proposed offline metrics, the framework proposed in
(Knijnenburg and Willemsen, 2015) for evaluating RSs with online experiments
was adopted. The experiment was designed as a within-subjects experiment and
all participants went through the same steps and compared explanations
produced by ExpLOD v2 (Musto et al., 2019a) and PEM (Du et al., 2022) to
validate the proposed metrics. In the first step, participants read the
consent term from the ethical committee and fill in personal information such
as nationality, level of education, age, gender, and previous usage of RSs.
The user was then asked to add ten liked films to build his/her profile. In
this step, the findings of (Rashid et al., 2002) were followed, and the top
100 items were shown in random order, ranked according to the function
$\forall i\in I\log_{10}(popularity*entropy)$, in which $I$ is the set of
items from the dataset.

The participants received five recommendations with two explanations, one from
(Musto et al., 2019a) and another from (Du et al., 2022). The movie was in the
center and separated the two columns of explanations, called A on the left and
B on the right. Positions A and B for each algorithm were chosen randomly in
run-time to prevent positional bias. The recommendation algorithm used was
EASE (Steck, 2019) because it provided the best accuracy results from the
algorithms used in the offline experiments [Blinded]. Every explanation was
built as in the offline experiment to facilitate the user evaluation of items
and properties.

Finally, after all recommendations and explanations had been analyzed,
participants were asked to answer six questions on a Likert Scale with the
following options Much More A, More A, Equal, More B, or Much More B. The
questions were: (1) Which explanation group (A or B) has more diverse
explanations?; (2) Which explanation group (A or B) has more familiar
explanations?; (3) Which explanation group (A or B) is more convincing?; (4)
Which explanation group (A or B) made you understand better why the
recommendation was suggested to you?; (5) Which explanation group (A or B)
made you discover new information about the movie?; (6) Which explanation
group (A or B) made you trust more in the recommendation system?. Questions
(1) and (2) regard explanations’ perceived diversity and popularity and should
reflect directly on the diverse and popular offline metrics. Questions (3) to
(6) evaluate the goals of Persuasiveness, Transparency, Engagement, and Trust
of explanations provided by (Tintarev and Masthoff, 2015), respectively.

##  5\. Results

###  5.1. Offline Results

|  | MID | MPD | TID | TPD | IEnt | PEnt | IGini | PGini | LIR | ETD | SEP
---|---|---|---|---|---|---|---|---|---|---|---|---
MostPOP | ExpLOD |  | 2,9956
---
$\pm$ 0.14
1 | 678,2 | 35,2 | 6,0205 | 2,4896 | 0,9622 | 0,9999 | 0,0827 | 1 | 0,6611
ExpLOD v2 |  | 2,9787
---
$\pm$ 0.21
1 | 766,4 | 31,8 | 6,2061 | 2,3880 | 0,9541 | 0,9999 | 0,0893 | 1 | 0,6488
PEM |  | 1,94721
---
$\pm$ 0.93
1 | 436,8 | 120,8 | 5,5532 | 4,1084 | 0,9758 | 0,9993 | 0,0299 | 1 | 0,1418
UserKNN | ExpLOD |  | 2,9733
---
$\pm$ 0.22
1 | 810,1 | 56,8 | 6,2553 | 3,0040 | 0,9511 | 0,9998 | 0,0984 | 1 | 0,5212
ExpLOD v2 |  | 2,9046
---
$\pm$ 0.39
1 | 812,4 | 67,0 | 6,2920 | 3,2084 | 0,9497 | 0,9997 | 0,0889 | 1 | 0,5011
PEM |  | 2,0370
---
$\pm$ 0.92
1 | 531,1 | 272,1 | 5,8368 | 5,0789 | 0,9684 | 0,9979 | 0,0271 | 1 | 0,1171
PageRank | ExpLOD |  | 2,9756
---
$\pm$ 0.22
1 | 714,8 | 54,9 | 6,1025 | 2,7356 | 0,9588 | 0,9998 | 0,0815 | 1 | 0,6003
ExpLOD v2 |  | 2,9525
---
$\pm$ 0.29
1 | 801,9 | 54,0 | 6,2941 | 2,6932 | 0,9500 | 0,9998 | 0,0855 | 1 | 0,6006
PEM |  | 2,0388
---
$\pm$ 0.93
1 | 443,3 | 152,0 | 5,5675 | 4,2832 | 0,9758 | 0,9991 | 0,0253 | 1 | 0,1274
BPRMF | ExpLOD |  | 2,9789
---
$\pm$ 0.21
1 | 835,7 | 59,4 | 6,2947 | 3,1173 | 0,9488 | 0,9997 | 0,0990 | 1 | 0,5771
ExpLOD v2 |  | 2,9197
---
$\pm$ 0.37
1 | 845,1 | 74,4 | 6,3358 | 3,3408 | 0,9469 | 0,9997 | 0,0948 | 1 | 0,5555
PEM |  | 1,9623
---
$\pm$ 0.92
1 | 587,1 | 317,8 | 6,0072 | 5,3188 | 0,9623 | 0,9973 | 0,0316 | 1 | 0,1446
EASE | ExpLOD |  | 2,9679
---
$\pm$ 0.25
1 | 786,8 | 60,2 | 6,2235 | 2,9745 | 0,9527 | 0,9998 | 0,0950 | 1 | 0,5961
ExpLOD v2 |  | 2,8944
---
$\pm$ 0.25
1 | 805,7 | 73,3 | 6,2875 | 3,2331 | 0,9498 | 0,9997 | 0,0891 | 1 | 0,5590
PEM |  | 2,0674
---
$\pm$ 0.90
1 | 529,5 | 257,5 | 5,8045 | 4,9957 | 0,9689 | 0,9981 | 0,0254 | 1 | 0,1264
NCF | ExpLOD |  | 2,9592
---
$\pm$ 0.27
1 | 967,6 | 79,7 | 6,5508 | 3,5096 | 0,9335 | 0,9996 | 0,1125 | 1 | 0,5217
ExpLOD v2 |  | 2,8727
---
$\pm$ 0.47
1 | 951,5 | 105,4 | 6,5487 | 3,7789 | 0,9339 | 0,9995 | 0,0996 | 1 | 0,5266
PEM |  | 1,9243
---
$\pm$ 0.92
1 | 671,7 | 456,0 | 6,2403 | 5,9436 | 0,9516 | 0,9948 | 0,0387 | 1 | 0,1558
Table 2. Offline results for the proposed metrics for the top-1
recommendations. Bold results are the best values considering the three
explanation algorithms for a recommendation algorithm. Two underlined values
represent Wilcoxon’s p-value above 0.05 between them, meaning non-significant
statistical differences.

To analyze the offline metric results in line with the online experiments
conducted in (Musto et al., 2019b), which compared ExpLOD to ExpLOD v2, and
(Du et al., 2022), which compared ExpLOD v2 to PEM, Table 2 presents the
values of the proposed metrics for the top-1 recommendation, since in both
online studies only one recommendation was provided for participants to
compare explanation algorithms. Each column in the table represents one of the
metrics described in Section 3, and each row corresponds to a recommendation
algorithm with the three explanation algorithms. Regarding metrics related to
items, ExpLOD outperformed its updated version ExpLOD v2 in the MID metric,
which then outperformed PEM in the same metric. This means the number of items
in an explanation may exert a lower effect on the explanation in comparison to
the properties that connect historical and recommended items, since the
results of explanation goals in (Musto et al., 2019a) and (Du et al., 2022)
were in the opposite direction, favoring ExPLOD v2 and PEM, over their
baselines, ExpLOD, and ExPLOD v2 respectively. Similarly, the total number of
items shown to users (TID) was highest for the ExpLOD v2 algorithm, which was
also reflected in the Gini and Entropy item metrics (IGini and IEnt,
respectively). This suggests that the algorithms were not biased toward some
items, creating a long-tail distribution of the items displayed. The recency
of items (LIR metric) also had the same effects as other item metrics, but
because the user studies were conducted with participants interacting only one
time with the system, the effect of recency in explanation goals, despite
theoretically important (Balloccu et al., 2022), could not be reflected
directly into the explanation goals evaluated by the participants in (Musto et
al., 2019a) and (Du et al., 2022).

|  | MID | MPD | TID | TPD | IEnt | PEnt | IGini | PGini | LIR | ETD | SEP
---|---|---|---|---|---|---|---|---|---|---|---|---
MostPop | ExpLOD |  | 6.4674
---
$\pm$ 2.31
| 2.8620
---
$\pm$ 1.00
1320.4 | 66.8 | 6.2508 | 2.6229 | 0.9518 | 0.9998 | 0.0890 | 0.5724 | 0.6504
ExpLOD v2 |  | 6.7446
---
$\pm$ 2.59
| 2.8890
---
$\pm$ 1.02
1360.8 | 40.6 | 6.2418 | 2.4383 | 0.9511 | 0.9999 | 0.0797 | 0.5778 | 0.5822
PEM |  | 7.7393
---
$\pm$ 2.68
| 4.6907
---
$\pm$ 0.62
1040.1 | 378.6 | 6.0865 | 4.9386 | 0.9598 | 0.9982 | 0.0313 | 0.9381 | 0.1427
UserKNN | ExpLOD |  | 7.3926
---
$\pm$ 2.73
| 3.2666
---
$\pm$ 1.00
1578.8 | 113.0 | 6.5531 | 3.1517 | 0.9352 | 0.9997 | 0.0976 | 0.6533 | 0.5427
ExpLOD v2 |  | 7.4649
---
$\pm$ 2.53
| 3.1754
---
$\pm$ 1.00
1630.1 | 104.8 | 6.6503 | 3.2199 | 0.9297 | 0.9997 | 0.0917 | 0.6351 | 0.4233
PEM |  | 8.2190
---
$\pm$ 2.90
| 4.7149
---
$\pm$ 0.57
1285.0 | 844.3 | 6.3368 | 5.8462 | 0.9474 | 0.9952 | 0.0318 | 0.9430 | 0.1333
PageRank | ExpLOD |  | 6.4330
---
$\pm$ 2.30
| 2.8056
---
$\pm$ 1.03
1349.5 | 112.7 | 6.3085 | 2.7679 | 0.9491 | 0.9998 | 0.0892 | 0.5611 | 0.6099
ExpLOD v2 |  | 7.1271
---
$\pm$ 2.50
| 2.9839
---
$\pm$ 0.99
1485.5 | 95.0 | 6.4895 | 2.7751 | 0.9401 | 0.9998 | 0.0819 | 0.5968 | 0.5688
PEM |  | 7.9168
---
$\pm$ 2.68
| 4.6677
---
$\pm$ 0.62
1061.6 | 509.2 | 6.0613 | 5.2385 | 0.9600 | 0.9976 | 0.0305 | 0.9335 | 0.1177
BPRMF | ExpLOD |  | 7.7752
---
$\pm$ 2.39
| 3.4538
---
$\pm$ 0.98
1698.8 | 118.3 | 6.6252 | 3.2585 | 0.9293 | 0.9997 | 0.0970 | 0.6908 | 0.6007
ExpLOD v2 |  | 7.9518
---
$\pm$ 2.49
| 3.3939
---
$\pm$ 0.97
1774.5 | 119.3 | 6.7395 | 3.3650 | 0.9223 | 0.9997 | 0.0890 | 0.6788 | 0.5353
PEM |  | 8.1582
---
$\pm$ 2.84
| 4.7710
---
$\pm$ 0.5
1415.5 | 1033.1 | 6.4937 | 6.1062 | 0.9384 | 0.9935 | 0.0328 | 0.9542 | 0.1452
EASE | ExpLOD |  | 7.0517
---
$\pm$ 2.28
| 3.0882
---
$\pm$ 1.00
1530.7 | 125.8 | 6.5268 | 3.1507 | 0.9366 | 0.9997 | 0.0960 | 0.6176 | 0.6009
ExpLOD v2 |  | 7.3943
---
$\pm$ 2.41
| 3.1391
---
$\pm$ 0.98
1630.0 | 121.3 | 6.6621 | 3.2874 | 0.9286 | 0.9997 | 0.0891 | 0.6278 | 0.5272
PEM |  | 8.2949
---
$\pm$ 2.88
| 4.7023
---
$\pm$ 0.60
1296.1 | 863.5 | 6.3391 | 5.8200 | 0.9468 | 0.9951 | 0.0309 | 0.9405 | 0.1335
NCF | ExpLOD |  | 9.4064
---
$\pm$ 2.11
| 4.1977
---
$\pm$ 0.78
2062.9 | 185.6 | 7.0301 | 3.6775 | 0.8980 | 0.9995 | 0.1163 | 0.8395 | 0.5453
ExpLOD v2 |  | 9.2497
---
$\pm$ 2.21
| 4.0482
---
$\pm$ 0.84
2077.2 | 235.6 | 7.0707 | 3.7976 | 0.8949 | 0.9994 | 0.1026 | 0.8096 | 0.5145
PEM |  | 8.5089
---
$\pm$ 2.85
| 4.9363
---
$\pm$ 0.28
1749.1 | 1796.3 | 6.8649 | 7.1184 | 0.9114 | 0.9830 | 0.0375 | 0.9873 | 0.1605
Table 3. Offline results for the proposed metrics for the top-5
recommendations. Bold results are the best values considering the three
explanation algorithms for a recommendation algorithm. Two underlined values
represent Wilcoxon’s p-value above 0.05 between them, meaning non-significant
statistical differences.

On the other hand, regarding the effects of property metrics, the progression
of TPD, PEnt, and PGini was similar to that of the explanation goals reported
in the studies, indicating the property that connects historical and
recommended items influences the user perception of the quality of
explanations. Similarly to items, PEnt and PGini were highest for the
algorithms with higher TPD, which indicates properties are distributed equally
across algorithms. Nevertheless, because the popularity of the items followed
the opposite trend of their diversity, in which ExpLOD, the simplest baseline,
outperformed ExpLOD v2 and PEM, SEP may impact user perception since the
disparity of values of TPD, PEnt, and PGini was not as high as the values of
explanation goals in the user experiments in (Musto et al., 2019a) and (Du et
al., 2022).

Finally, since MPD and ETD metrics measure the diversity of properties in an
explanation list of recommendations for a single user, value 1 is due to the
fact only one explanation was shown to the user. Sentence explanations are
based on connections of historical and recommended items by common attributes.
When users evaluate only one explanation, a possible algorithmic bias can be
overlooked since the algorithm can generate explanations towards only a set of
properties and items that are popular among the user profile and other items.
As a result, participants must be provided with a larger list when evaluating
the explanation of an algorithm in online experiments, so that they can
analyze whether justifications are not repetitive among historical items or
properties linked to the recommendation. Therefore, in order to analyze the
effects of diversification of properties, which could not be measured by the
metrics MID and ETD metrics by the top recommendation, Table 3 shows the
results of the offline experiments for the top-5 recommendations.

According to the results of the three algorithms, PEM achieved the highest
user mean diversity of items and properties (MID and MPD) for all
recommendation algorithms. Due to the high diversification, it also showed the
highest total property diversity (TPD), and lower popularity of the properties
in explanations (SEP). In contrast, it achieved the lowest total item
diversity (TID), i.e., despite the diversification of properties, the items to
which those properties are connected are more common across users. Differently
from a situation in which the number of recommendations was 1, PEM achieved
better results in MID, which means that, for multiple recommendations, the
ExpLOD algorithms are biased toward items with popular properties.

Regarding ExpLOD and ExpLOD v2, in the original paper (Musto et al., 2019a),
the authors performed online experiments comparing both approaches with three
recommendation algorithms. The overall results indicated EXPLOD v2 showed a
statistically non-significant decrease in persuasion, non-significant
improvements in transparency and engagement, and significant improvements in
trust from the first to the second version. Such findings are aligned with our
offline metrics since our experiments revealed an improvement in ExpPLOD v2
over ExpLOD in MID and MPD metrics (except in User-KNN and NCF algorithms). On
the other hand, our metrics also showed lower diversity of the second version,
as indicated by TPD, Gini, Entropy, and ETD, meaning those metrics exert a
lower effect than MID and MPD on users. The popularity of the properties of
ExpLOD is also higher than that of the second version since it does not use
the hierarchical properties connected to fewer items compared to directly
connected properties.

The online study reported in (Du et al., 2022) showed the PEM algorithm
significantly improved persuasiveness, engagement, and trust, with
transparency slightly improved, but not significantly over ExpLOD v2
algorithm. Our offline metrics (Table 3) indicate that the PEM algorithm
showed higher property and item explanations (MPD and MID), despite lower
total item diversity (TID). Such aspects highlight the importance of user
diversity in explanations since improvements in MPD and MID were reflected in
improved explanation goals in the online experiment from the first to the
second version of ExpLOD and from ExpLOD v2 to PEM. In contrast, because the
total and user property diversities (TPD and MPD) of PEM are much higher, it
is also reflected in the lower popularity of properties (SEP) compared to
ExpLOD v2, which could indicate why transparency was non-significantly
improved in (Du et al., 2022).

According to the comparison results between the proposed offline metrics and
online experiments reported in (Musto et al., 2019a; Du et al., 2022) item and
property total diversity (TID and TPD) did not affect user perception
significantly, indicating showing more items and properties does not influence
user perception. On the other hand, user item and property diversity (MID and
MPD) significantly impacted the user perception of transparency, directly
reflecting the evolution of algorithms. However, a comparison of the results
of PEM over ExpLOD v2 for explanation goals in the online experiments in (Du
et al., 2022) and our diversity offline metric results suggests the disparity
in TID and MID, as perceived by users, was reduced due to the improved
popularity (SEP) of ExpLOD v2 over PEM.

###  5.2. Online Results

| Persuasiveness | Trust | Transparency | Engagement | Diversity | Popularity
---|---|---|---|---|---|---
Much More PEM | 3.6% (2) | 3.6% (2) | 9.0% (5) | 7.2% (4) | 9.0% (5) | 3.6% (2)
More PEM | 36.6% (20) | 29.1% (16) | 23.6% (13) | 25.4% (14) | 36.6% (20) | 25.4% (14)
Equal | 27.2% (15) | 30.9% (17) | 27.2% (15) | 41.8% (23) | 27.2% (15) | 27.2% (15)
More ExpLOD v2 | 23.6% (13) | 30.9% (17) | 36.6% (20) | 21.8% (12) | 25.4% (14) | 30.9% (17)
Much More ExpLOD v2 | 9.0% (5) | 5.4% (3) | 3.6% (2) | 3.6% (2) | 1.8% (1) | 12.7% (7)
Table 4. User preference for each explanation goal. Bold values are the highest. | Persuasiveness | Trust | Transparency | Engagement | Diversity | Popularity
---|---|---|---|---|---|---
Much More PEM | 5.2% (1) | 5.2% (1) | 15.7% (3) | 5.2% (1) | 5.2% (1) | 5.2% (1)
More PEM | 21.0% (4) | 21.0% (4) | 15.7% (3) | 26.3% (5) | 36.8% (7) | 21.0% (4)
Equal | 26.3% (5) | 15.7% (3) | 15.7% (3) | 42.1% (8) | 26.3% (5) | 26.3% (5)
More ExpLOD v2 | 36.8% (7) | 52.6% (10) | 52.6% (10) | 21.0% (4) | 31.5% (6) | 36.8% (7)
Much More ExpLOD v2 | 10.5% (2) | 5.2% (1) | 0% (0) | 5.2% (1) | 0% (0) | 10.5% (2)
Table 5. User preference for each explanation goal when both ExpLOD v2 and PEM
are diverse. Bold values are the highest.

Our online within-subjects experiments were conducted with 55 participants, a
sufficient number according to (Knijnenburg and Willemsen, 2015), and composed
of different profiles - mostly between 25 and 50 years old (58%), male (71%),
with an understanding of RSs (96%), and a bachelor’s degree (38%). Users
compared five explanations between the Explod v2 and PEM algorithms in order
to assess multiple explanations from both algorithms.

Table 4 shows the overall results considering the four explanation goals in
addition to the perceived popularity and diversity of properties in
explanations to validate the offline experiments. The results from the offline
metrics regarding diversity and popularity (3) are aligned with those with
diversity and popularity of the online setting. Specifically, for both
diversity and popularity questions, users rated PEM as much more diverse or
more diverse than ExpLOD v2, which was rated as much more popular or more
popular than PEM. In fact, according to the measurement of average user
diversity in the online experiments, PEM showed an average diversity of 4.8
properties per user in contrast to 3.09 for ExpLOD v2.

In addition, Table 4 shows that PEM obtained better preference from
participants regarding persuasiveness and engagement, which align with the
results reported in (Du et al., 2022). However, for persuasiveness, more users
responded that they preferred Much More ExpLOD v2 over PEM, which indicates
that these participants were more confident of their response. Concerning
engagement, despite more votes for PEM, most participants did not perceive
differences among explanations.

Contrary to the results reported in (Du et al., 2022), ExpLOD v2 obtained more
votes than PEM regarding trust and transparency. The main reason for this is
because, in the online experiments of (Du et al., 2022), participants only
evaluated the explanation for the top item and not a list of items,
furthermore, because a between-subjects online experiment was conducted, users
evaluated an explanation for only a single method, without comparing it to
another. In our context, a within-subjects experiment with more than one
explanation is more appropriate for accurately validating the metrics, since
users can evaluate the difference in multiple explanations among methods.

Participants that received explanations with a user property diversity of 4
were filtered so that the reason why trust and transparency were better with
ExpLOD v2 could be verified. The analysis aimed mainly at investigating the
effects of ExpLOD v2 and PEM when both show high average diversity. As a
result, the impact of the improved property popularity of ExpLOD v2 when no
diversity difference is observed towards recommendation goals would be
compared. Table 5 shows the results of 19 users, whose $MPD$ for ExpLOD v2 and
PEM were 4.1 and 4.9, respectively.

According to the results, when both algorithms increase the number of
properties shown (MPD is similar), users can still identify that PEM displays
more diverse and less popular properties than ExpLOD v2, meaning that the
online results align with the scores obtained by SEP and LIR metrics, which
were higher for ExpLOD and ExpLOD v2 than for PEM. However, when MPD is
similar for both algorithms, even in metrics such as persuasiveness and
engagement, in which PEM outperformed ExpLOD v2 in Table 4, the scenario
changed with ExpLOD v2 improving its results over PEM, as seen in Table 5.

As a result, to answer the (RQ2), transparency and trust are related to
explanations with popular properties since they were not affected by the
increase of properties shown in ExpLOD v2. In contrast, persuasiveness and
engagement changed when ExpLOD v2 explained recommendations with more
properties, meaning that these goals are influenced by diversity. The main
reason for this behavior is that engagement is defined as how convincing an
explanation is and persuasiveness is defined as the discovery of new
information about an item. Therefore, to achieve such goals, it is necessary
to show properties that are unfamiliar and will try to encourage the user to
interact with an item based on new information.

In contrast, trust and transparency are directed toward familiarity since
trust is defined by reliability, and transparency is determined by
understanding why a recommendation was made, which is achieved when the user
relates to the information shown. As a result, there is a trade-off between
popularity and diversity when analyzing explanations for recommendations:
diversifying explanations will provide new and interesting information about
recommendations that persuade the user to interact. However, it can come with
a cost of displaying unfamiliar information, lowering the reason why a
recommendation was shown to the user and, consequently, the trust in the RSs.

###  5.3. Offline Metrics Discussion

The proposed offline metrics reflected the evolution of explanation goals
presented by (Musto et al., 2016, 2019a; Du et al., 2022) with between-
subjects experiments and were confirmed by the online experiments of within-
subjects experiments described in this manuscript. The ExpLOD method (Musto et
al., 2016) yielded similar but outperformed results in the offline metrics
compared to its evolution proposed by the same authors in (Musto et al.,
2019a). In addition, the most recent method, proposed by (Du et al., 2022),
also outperformed (Musto et al., 2019a) in the proposed metrics and the
within-subjects experiment proposed. These observations mean that the proposed
metrics are reliable in measuring explanation goals, highlighted mainly by the
MPD and SEP property that reflects the diversity and popularity of the
properties shown in explanations.

The metrics also effectively reflected the user perception under explanation
goals and can help researchers and the industry assess the necessity of a user
trial. Similarly to ranking, in which offline experiments are valuable tools
to analyze the competitiveness of a new recommendation algorithm compared to
the state-of-the-art and the one currently employed in production, the
proposed metrics for explanations can also be resourceful for researchers to
assess the quality of the explanation algorithm before building an online
trial. For academia, user trials are demanding because they require the
development of a web platform to compare the performance of algorithms. For
the industry, it can affect the user experience with the platform if the
explanation algorithm that will be validated does not have competitive results
with the current in production. Therefore, we hope the metrics proposed can
help researchers with the current lack of offline metrics to assess the
quality of explanations of their algorithms.

In sentence explanations formulated by a set of properties that connect
historical and recommended items, explanation goals of transparency and trust
are related to familiar and popular properties and engagement and
persuasiveness to diverse and novel properties, characterizing a similar
trade-off to popularity and diversity regarding the ranking of items in
recommender systems. However, similarly to the ranking evaluation in
recommender systems, the metrics are susceptible to a different evaluation in
the online experiment. In (Du et al., 2022), PEM had an increase in the goal
of trust, when explanations were shown only for one user and without
comparison to another method, however, the conducted offline test and online
in-subject test with five recommendations /explanations, trust was higher for
the baseline ExpLOD v2 method. Similarly, when evaluating the ranking of items
in recommender systems, deep learning approaches have reportedly presented the
same mismatch between offline and online metrics (Steck et al., 2021).

Regarding historical items shown in explanations, diversifying the items
displayed as historical in a sentence for a single user improves explanation
goals, as demonstrated by the MID metric results in Table 3. Therefore,
explanations biased towards repetitive properties and items can lead to a
lower experience of users with explanations. On the other hand, regarding
metrics that consider the diversification of items across all users (TID,
IEnt, and IGini), the same did not occur since the two versions of ExpLOD
outperformed PEM. Therefore, item diversity in explanations for single users
has a more significant impact on explanation goals than diversification of
items in explanations for a set of users.

In the context of properties, diversifying them was also perceived by users in
the explanation goals. This finding was consistent with the offline and online
experiments conducted in the literature, and it was corroborated by our
proposed online experiment, which measured the MPD. Unlike items, where the
diversification of historical items for a single user did not affect the
diversification for a set of users, the diversification of properties for a
single user had a positive impact on the diversification for the entire user
set. As a result, an increase in the MPD reflected the user experience with
the diversification of explanations and had a positive impact on metrics TPD,
PEnt, PGini, and ETD, which account for all users. However, it’s worth noting
that metrics of diversification for a set of users, such as item aggregate
diversity (Adomavicius and Kwon, 2011), may not fully reflect the difference
in user perception, as users are only exposed to a limited number of items.
Therefore, MPD reflects user experience, while TPD, PEnt, and PGini indicate
the ability of the explanation algorithm to display different properties
across users. Finally, the LIR metric, despite its relevance in the literature
for the impact of recent items in explanations (Balloccu et al., 2022), our
experiments, where users generated recommendations and explanations in one
interaction, did not allow for validation under explanation goals.

##  6\. Conclusions and Future Directions

This paper introduced several metrics for the evaluation of explanation goals
in an offline setting. The explanations of three post-hoc KG algorithms were
assessed and offline and online results were compared considering the
diversification and popularity aspects of items and properties shown to users.
According to the results, the explanation goals of transparency and trust are
related to the popularity of items shown to users, and persuasiveness and
engagement are associated with diversity, characterizing a trade-off similar
to the evaluation of ranking in recommendations.

According to the literature review conducted, this is the first paper that
analyzes and proposes metrics for explanation goals. Our aim is to draw
attention to the crucial aspect of evaluating explanations to identify and
address potential bias in explainable recommendation algorithms. When
comparing algorithms, it is essential to analyze metrics across large
databases, different ranking sizes, and various types of recommendation
engines to assess the consistency and robustness of the algorithms (Coba et
al., 2022). These findings can provide valuable guidance in order to fine-tune
explanation algorithms and evaluate them under explanation goals.

There are many open directions for offline evaluation of explanation in
recommender systems: the proposed diversity and popularity metrics captured
the user perception regarding explanation goals. Nevertheless, these metrics
can be susceptible to the same problems of overfitting, offline and online
mismatch, bias, and proper selection of test data that also regard the
evaluation of ranking in recommender systems (Castells and Moffat, 2022; Chen
et al., 2017; Steck et al., 2021). This means that similar approaches to
improving ranking in recommender systems could also be applied to
explanations, such as counterfactual explanations (Ghazimatin et al., 2020;
Yao et al., 2022).

The proposed metrics indicate users prefer explanations that balance diversity
with the popularity of the item’s attributes. Since, in current systems,
explanations are provided by categories that are limited and shown repeatedly
to the user, an interesting research topic is the relation of user engagement
with changes in explanation in multiple visits to the system. The LIR metric
should also be evaluated in an online experiment, where users evaluate the
explanation in a scenario of time dependency on the addition of items to the
user profile. Finally, although the metrics can be adapted to other contexts
and scenarios, their validation by reordering algorithms and other techniques
(e.g., NLP) and domains such as e-commerce and music, can also be explored.

## Ethical Considerations

This paper has no potential negative societal impact regarding fairness,
privacy, security, safety, misuse of the technology by malicious actors, or
harm.

## References

  * (1)
  * Adomavicius and Kwon (2011) Gediminas Adomavicius and YoungOk Kwon. 2011.  Improving aggregate recommendation diversity using ranking-based techniques.  _IEEE Transactions on Knowledge and Data Engineering_ 24, 5 (2011), 896–911.
  * Balloccu et al. (2022) Giacomo Balloccu, Ludovico Boratto, Gianni Fenu, and Mirko Marras. 2022.  Post processing recommender systems with knowledge graphs for recency, popularity, and diversity of explanations. In _Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval_. 646–656.
  * Balloccu et al. (2023) Giacomo Balloccu, Ludovico Boratto, Gianni Fenu, and Mirko Marras. 2023.  Reinforcement recommendation reasoning through knowledge graphs for explanation path quality.  _Knowledge-Based Systems_ 260 (2023), 110098.
  * Balog and Radlinski (2020) Krisztian Balog and Filip Radlinski. 2020.  Measuring recommendation explanation quality: The conflicting goals of explanations. In _Proceedings of the 43rd international ACM SIGIR conference on research and development in information retrieval_. 329–338.
  * Castells and Moffat (2022) Pablo Castells and Alistair Moffat. 2022.  Offline recommender system evaluation: Challenges and new directions.  _AI Magazine_ 43, 2 (2022), 225–238.
  * Chen et al. (2017) Hung-Hsuan Chen, Chu-An Chung, Hsin-Chien Huang, and Wen Tsui. 2017.  Common pitfalls in training and evaluating recommender systems.  _ACM SIGKDD Explorations Newsletter_ 19, 1 (2017), 37–45.
  * Coba et al. (2022) Ludovik Coba, Roberto Confalonieri, and Markus Zanker. 2022.  RecoXplainer: A Library for Development and Offline Evaluation of Explainable Recommender Systems.  _IEEE Computational Intelligence Magazine_ 17, 1 (2022), 46–58.  https://doi.org/10.1109/mci.2021.3129958
  * Cremonesi et al. (2010) Paolo Cremonesi, Yehuda Koren, and Roberto Turrin. 2010.  Performance of Recommender Algorithms on Top-n Recommendation Tasks. In _Proceedings of the Fourth ACM Conference on Recommender Systems_ (Barcelona, Spain) _(RecSys ’10)_. Association for Computing Machinery, New York, NY, USA, 39–46.  https://doi.org/10.1145/1864708.1864721
  * da Costa et al. (2018) Arthur da Costa, Eduardo Fressato, Fernando Neto, Marcelo Manzato, and Ricardo Campello. 2018.  Case recommender: a flexible and extensible python framework for recommender systems. In _Proceedings of the 12th ACM Conference on Recommender Systems_. 494–495.
  * Du et al. (2022) Yu Du, Sylvie Ranwez, Nicolas Sutton-Charani, and Vincent Ranwez. 2022.  Post-hoc recommendation explanations through an efficient exploitation of the DBpedia category hierarchy.  _Knowledge-Based Systems_ 245 (2022), 108560.
  * Ferrari Dacrema et al. (2021) Maurizio Ferrari Dacrema, Simone Boglio, Paolo Cremonesi, and Dietmar Jannach. 2021.  A troubling analysis of reproducibility and progress in recommender systems research.  _ACM Transactions on Information Systems (TOIS)_ 39, 2 (2021), 1–49.
  * Ghazimatin et al. (2020) Azin Ghazimatin, Oana Balalau, Rishiraj Saha Roy, and Gerhard Weikum. 2020.  PRINCE: Provider-side interpretability with counterfactual explanations in recommender systems. In _Proceedings of the 13th International Conference on Web Search and Data Mining_. 196–204.
  * Hada and Shevade (2021) Deepesh V Hada and Shirish K Shevade. 2021.  Rexplug: Explainable recommendation using plug-and-play language model. In _Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_. 81–91.
  * Harper and Konstan (2015) F Maxwell Harper and Joseph A Konstan. 2015.  The movielens datasets: History and context.  _Acm transactions on interactive intelligent systems (tiis)_ 5, 4 (2015), 1–19.
  * He et al. (2017) Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. 2017.  Neural Collaborative Filtering. In _Proceedings of the 26th International Conference on World Wide Web_ (Perth, Australia) _(WWW ’17)_. International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, CHE, 173–182.  https://doi.org/10.1145/3038912.3052569
  * Knijnenburg and Willemsen (2015) Bart P Knijnenburg and Martijn C Willemsen. 2015.  Evaluating recommender systems with user experiments.  _Recommender systems handbook_ (2015), 309–352.
  * Kouki et al. (2019) Pigi Kouki, James Schaffer, Jay Pujara, John O’Donovan, and Lise Getoor. 2019.  Personalized explanations for hybrid recommender systems. In _Proceedings of the 24th International Conference on Intelligent User Interfaces_. 379–390.
  * Kunaver and Požrl (2017) Matevž Kunaver and Tomaž Požrl. 2017.  Diversity in recommender systems–A survey.  _Knowledge-based systems_ 123 (2017), 154–162.
  * Melchiorre et al. (2022) Alessandro B Melchiorre, Navid Rekabsaz, Christian Ganhör, and Markus Schedl. 2022.  ProtoMF: Prototype-based Matrix Factorization for Effective and Explainable Recommendations. In _Proceedings of the 16th ACM Conference on Recommender Systems_. 246–256.
  * Musto et al. (2016) Cataldo Musto, Fedelucio Narducci, Pasquale Lops, Marco De Gemmis, and Giovanni Semeraro. 2016.  ExpLOD: a framework for explaining recommendations based on the linked open data cloud. In _Proceedings of the 10th ACM Conference on Recommender Systems_. 151–154.
  * Musto et al. (2019a) Cataldo Musto, Fedelucio Narducci, Pasquale Lops, Marco de Gemmis, and Giovanni Semeraro. 2019a.  Linked open data-based explanations for transparent recommender systems.  _International Journal of Human-Computer Studies_ 121 (2019), 93–107.
  * Musto et al. (2019b) Cataldo Musto, Gaetano Rossiello, Marco de Gemmis, Pasquale Lops, and Giovanni Semeraro. 2019b.  Combining text summarization and aspect-based sentiment analysis of users’ reviews to justify recommendations. In _Proceedings of the 13th ACM conference on recommender systems_. 383–387.
  * Nunes and Jannach (2017) Ingrid Nunes and Dietmar Jannach. 2017.  A systematic review and taxonomy of explanations in decision support and recommender systems.  _User Modeling and User-Adapted Interaction_ 27 (2017), 393–444.
  * Pillai et al. (2019) Sini Govinda Pillai, Lay-Ki Soon, and Su-Cheng Haw. 2019.  Comparing DBpedia, Wikidata, and YAGO for web information retrieval. In _Intelligent and Interactive Computing: Proceedings of IIC 2018_. Springer, 525–535.
  * Rashid et al. (2002) Al Mamunur Rashid, Istvan Albert, Dan Cosley, Shyong K Lam, Sean M McNee, Joseph A Konstan, and John Riedl. 2002.  Getting to know you: learning new user preferences in recommender systems. In _Proceedings of the 7th international conference on Intelligent user interfaces_. 127–134.
  * Rendle et al. (2009) Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. 2009.  BPR: Bayesian Personalized Ranking from Implicit Feedback. In _Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence_ (Montreal, Quebec, Canada) _(UAI ’09)_. AUAI Press, Arlington, Virginia, USA, 452–461.
  * Resnick et al. (1994) Paul Resnick, Neophytos Iacovou, Mitesh Suchak, Peter Bergstrom, and John Riedl. 1994.  GroupLens: An Open Architecture for Collaborative Filtering of Netnews. In _Proceedings of the 1994 ACM Conference on Computer Supported Cooperative Work_ (Chapel Hill, North Carolina, USA) _(CSCW ’94)_. Association for Computing Machinery, New York, NY, USA, 175–186.  https://doi.org/10.1145/192844.192905
  * Steck (2019) Harald Steck. 2019.  Embarrassingly Shallow Autoencoders for Sparse Data. In _The World Wide Web Conference_ (San Francisco, CA, USA) _(WWW ’19)_. Association for Computing Machinery, New York, NY, USA, 3251–3257.  https://doi.org/10.1145/3308558.3313710
  * Steck et al. (2021) Harald Steck, Linas Baltrunas, Ehtsham Elahi, Dawen Liang, Yves Raimond, and Justin Basilico. 2021.  Deep learning for recommender systems: A Netflix case study.  _AI Magazine_ 42, 3 (2021), 7–18.
  * Tai et al. (2021) Chang-You Tai, Liang-Ying Huang, Chien-Kun Huang, and Lun-Wei Ku. 2021.  User-centric path reasoning towards explainable recommendation. In _Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_. 879–889.
  * Tan et al. (2021) Juntao Tan, Shuyuan Xu, Yingqiang Ge, Yunqi Li, Xu Chen, and Yongfeng Zhang. 2021.  Counterfactual Explainable Recommendation. In _Proceedings of the 30th ACM International Conference on Information & Knowledge Management_ (Virtual Event, Queensland, Australia) _(CIKM ’21)_. Association for Computing Machinery, New York, NY, USA, 1784–1793.  https://doi.org/10.1145/3459637.3482420
  * Tintarev and Masthoff (2015) Nava Tintarev and Judith Masthoff. 2015.  Explaining recommendations: Design and evaluation.  _Recommender systems handbook_ (2015), 353–382.
  * Wang et al. (2018) Hongwei Wang, Fuzheng Zhang, Jialin Wang, Miao Zhao, Wenjie Li, Xing Xie, and Minyi Guo. 2018.  Ripplenet: Propagating user preferences on the knowledge graph for recommender systems. In _Proceedings of the 27th ACM international conference on information and knowledge management_. 417–426.
  * Xian et al. (2019) Yikun Xian, Zuohui Fu, Shan Muthukrishnan, Gerard De Melo, and Yongfeng Zhang. 2019.  Reinforcement knowledge graph reasoning for explainable recommendation. In _Proceedings of the 42nd international ACM SIGIR conference on research and development in information retrieval_. 285–294.
  * Yao et al. (2022) Yuanshun Yao, Chong Wang, and Hang Li. 2022.  Counterfactually Evaluating Explanations in Recommender Systems.  _arXiv preprint arXiv:2203.01310_ (2022).
  * Zanon et al. (2022) André Levi Zanon, Leonardo Chaves Dutra da Rocha, and Marcelo Garcia Manzato. 2022.  Balancing the trade-off between accuracy and diversity in recommender systems with personalized explanations based on Linked Open Data.  _Knowledge-Based Systems_ 252 (2022), 109333.
  * Zhang et al. (2020) Yongfeng Zhang, Xu Chen, et al. 2020\.  Explainable recommendation: A survey and new perspectives.  _Foundations and Trends® in Information Retrieval_ 14, 1 (2020), 1–101.
  * Zhao et al. (2020) Kangzhi Zhao, Xiting Wang, Yuren Zhang, Li Zhao, Zheng Liu, Chunxiao Xing, and Xing Xie. 2020.  Leveraging demonstrations for reinforcement recommendation reasoning over knowledge graphs. In _Proceedings of the 43rd international ACM SIGIR conference on research and development in information retrieval_. 239–248.


# Towards a quantum notion of covariance in spherically symmetric loop quantum
gravity

Rodolfo Gambini1, Javier Olmedo2 Jorge Pullin3 1\. Instituto de Física,
Facultad de Ciencias, Iguá 4225, esq. Mataojo, 11400 Montevideo, Uruguay.
2\. Departamento de Física teórica y del cosmos, Universidad de Granada,
Granada-18071, Spain
3\. Department of Physics and Astronomy, Louisiana State University, Baton
Rouge, LA 70803-4001, USA.

###### Abstract

The covariance of loop quantum gravity studies of spherically symmetric space-
times has recently been questioned. This is a reasonable worry, given that
they are formulated in terms of slicing-dependent variables. We show
explicitly that the resulting space-times, obtained from Dirac observables of
the quantum theory, are covariant in the usual sense of the way —they preserve
the quantum line element— for any gauge that is stationary (in the exterior,
if there is a horizon). The construction depends crucially on the details of
the Abelianized quantization considered, the satisfaction of the quantum
constraints and the recovery of standard general relativity in the classical
limit and suggests that more informal polymerization constructions of possible
semi-classical approximations to the theory can indeed have covariance
problems. This analysis is based on the understanding of how slicing dependent
quantities as the metric arise in a quantum context in terms of parameterized
observables. It has implications beyond loop quantum gravity that hold for
general approaches to quantum space time theories.

The application of loop quantum gravity techniques to spherically symmetric
space-times has led to insights about how the singularity inside black holes
could be eliminated by quantum effects. We refer specifically to the
construction that uses inhomogeneous slices and enforces the constraint
algebra at the quantum level through Abelianization us . The construction is
based on canonical quantum gravity and, as such, is based on three dimensional
objects that are slicing-dependent. A reasonable worry bojo  is if the
constructions lead to covariant quantizations of the space-times. An
encouraging sign is that they enforce the constraint algebra, which in the
canonical theory is the guarantor of slicing independence of the construction,
and of reproducing the standard general relativistic results in the classical
limit. However, technical aspects, like the fact that the algebra is
Abelianized, may lead to questions about the covariance of the procedure. We
would like to show explicitly that the resulting space-times are indeed
covariant at any space time region in the usual sense of the word: the
invariant line element is indeed invariant at the quantum level for any
stationary foliation (stationary in the exterior if there is a horizon).
Although this is not a definitive extension of the notion of covariance to the
quantum realm, the result arises in a non-trivial way and suggests that with
more work this notion could be achieved.

We start with a spherically symmetric space-time. Following the discussion in
us , the line element can be written as $dS^{2}=ds^{2}+|E^{x}|d\omega^{2}$,
where $d\omega^{2}$ is the line element of the unit 2-sphere, and,

$ds^{2}=-(N^{2}-N_{x}N^{x})dt^{2}+2N_{x}dtdx+\frac{(E^{\varphi})^{2}}{|E^{x}|}dx^{2},$
(1)

where $N$ and $N_{x}$ are suitable lapse and shift functions, with
$N^{x}=g^{xx}N_{x}$, and $E^{\varphi}$ and $E^{x}$ triad variables, conjugate
to the extrinsic curvature components of the foliation, $K_{\varphi}$ and
$K_{x}$, with Poisson brackets,

$\displaystyle\\{K_{x}(x),E^{x}(x^{\prime})\\}=G\delta(x-x^{\prime}),$
$\displaystyle\\{{K}_{\varphi}(x),E^{\varphi}(x^{\prime})\\}=G\delta(x-x^{\prime}).$
(2)

We take the Immirzi parameter $\gamma=1$ and $G$ is Newton’s constant.

We re-define the lapse and the shift in order to make the constraints Abelian
as shown in us-improv

$\displaystyle\overline{N}^{x}$ $\displaystyle=$ $\displaystyle
N^{x}-\frac{2NK_{\varphi}\sqrt{|E^{x}|}}{|E^{x}|^{\prime}},$ (3)
$\displaystyle\overline{N}$ $\displaystyle=$
$\displaystyle-\frac{1}{E^{\varphi}}\left(N\frac{E^{\varphi}}{|E^{x}|^{\prime}}\right)^{\prime},$
(4)

(prime denotes derivative with respect to $x$ with which the smeared
Hamiltonian constraint takes the form,

$\tilde{H}(\tilde{N}):=\frac{1}{G}\int
dx\tilde{N}\sqrt{|E^{x}|}E^{\varphi}\bigg{[}K_{\varphi}^{2}-\frac{[|E^{x}|^{\prime}]^{2}}{4(E^{\varphi})^{2}}+\left(1-\frac{2GM}{\sqrt{|E^{x}|}}\right)\bigg{]},$
(5)

where $M$ is the ADM mass.111Classically, the variational problem is well-
posed once a boundary term is introduced kuchar ; bh-review . The boundary
term explicitly introduces the mass $M$ and its conjugate variable $\tau$ (the
proper time of an asymptotic observer) as the global physical degree of
freedom that characterizes classical solutions. It can be checked that this
constraint has an Abelian algebra with itself. The momentum constraint, on the
other hand, keeps its original form.

Let us recall the construction of the physical Hilbert space. The elements of
a basis of quantum states are one dimensional spin networks with integer
valences $k_{j}$ at each link $j\in[-S,-S+1,\ldots,0,\ldots,S-1,S]$, with
$2S+1$ nodes in total. We have three kinds of Dirac observables, one
corresponding to the ADM mass $M$, the total number of vertices $S$ and the
set of integers $\vec{k}$,

$\hat{M}|M,\vec{k}\rangle=M|M,\vec{k}\rangle,$ (6)

and the other two not having a classical counterpart,

$\hat{O}(z)|M,\vec{k}\rangle=\ell^{2}_{\rm Planck}k_{{\rm
Int}(Sz)}|M,\vec{k}\rangle$ (7)

where ${\rm Int}$ means the integer part and $z$ is a real parameter in the
interval $[-1,1]$, so $O(z)$ constitutes a one-parameter family of
observables. $M$, $O(z)$ and $S$ are the Dirac observables. $\ell_{\rm
Planck}$ is Planck’s length. Having identified the physical space, we will
describe the metric by introducing gauge fixings that allow to write the
metric components in specific gauges in terms of evolving observables defined
on the physical space of states.

The action of the triads and their derivatives on physical states is,

$\displaystyle\hat{|E^{x}}(x_{j})||M,\vec{k}\rangle$ $\displaystyle=$
$\displaystyle\hat{O}\left(z\left(x_{j}\right)\right)|M,\vec{k}\rangle=\ell_{\rm
Planck}^{2}k_{j}|M,\vec{k}\rangle,$ (8)
$\displaystyle|\hat{E}^{x}(x_{j})|^{\prime}|M,\vec{k}\rangle$ $\displaystyle=$
$\displaystyle\frac{1}{\ell_{\rm
Planck}}(|\hat{E}^{x}|(x_{j+1})-|\hat{E}^{x}|(x_{j}))|M,\vec{k}\rangle.$ (9)

Here, and for simplicity, we choose a particular class of spin networks, with
given $M$ and $k_{j}$ (no superpositions) and gauge fixing such that
$x_{j}^{2}=\ell_{\rm Planck}^{2}k_{j}$, where

$x_{j}=\ell_{\rm Planck}\left(|j|+j_{0}\right).$ (10)

This is motivated in that in the classical theory the condition
$|E^{x}|=x^{2}$ corresponds to having $x$ be the radius of the spheres of
symmetry. This includes many popular coordinate systems for studying spherical
space-times. We will later relax this assumption, allowing arbitrary
stationary changes in the radial coordinate.

Within the improved dynamics of us-improv , $j_{0}$ is the minimum integer
greater than $\left(\frac{2GM\Delta}{4\pi l^{3}_{\rm Planck}}\right)^{1/3}$
and $\Delta$ is the loop quantum gravity area gap us-improv . This implies,

$\displaystyle|\hat{E}^{x}(x_{j})||M,\vec{k}\rangle$ $\displaystyle=$
$\displaystyle x_{j}^{2}|M\,\vec{k}\rangle,$ (11)
$\displaystyle|\hat{E}^{x}(x_{j})|^{\prime}|M,\vec{k}\rangle$ $\displaystyle=$
$\displaystyle{\rm sign}(j)\left(2x_{j}+\ell_{\rm
Planck}\right)|M,\vec{k}\rangle,$ (12)

where we take ${\rm sign}(j)=1,\,\forall j\geq 0$ and ${\rm
sign}(j)=-1,\,\forall j<0$.

The quantization of the constraints involves,

$K^{\varphi}\longrightarrow\frac{\sin\left(\rho_{j}K_{\varphi}(x_{j})\right)}{\rho_{j}},$
(13)

with $\hat{\rho}_{j}^{2}=\Delta/(4\pi\hat{E}^{x}(x_{j}))$. The quantum gauge
fixings that we adopt here leave $K_{\varphi}(x_{j})$ as either a c-number
function or a function of the Dirac observables $M$, $S$ or $O(z)$. These
quantum gauge fixings correspond to a choice of a slicing in the quantum
theory.

Moreover, the conjugate variable to $K_{\varphi}(x_{j})$ on the physical
Hilbert space is obtained by solving the Abelianized constraint (5), and
amounts to,

$\left[\hat{E}^{\varphi}\left(x_{j}\right)\right]^{2}={\left[|\hat{E}^{x}|(x_{j})^{\prime}\right]^{2}}\left[4\left(1+\frac{\widehat{\sin^{2}\left(\rho_{j}K_{\varphi}(x_{j})\right)}}{\hat{\rho}_{j}}-\frac{2G\hat{M}}{\sqrt{|\hat{E}^{x}|(x_{j})}}\right)\right]^{-1}.$
(14)

From now on we will use the notation
$\hat{E}^{x}_{j}\equiv|\hat{E}^{x}|(x_{j})$,
$(\hat{E}^{x}_{j})^{\prime}\equiv|\hat{E}^{x}|(x_{j})^{\prime}$,
$\hat{E}^{\varphi}_{j}\equiv\hat{E}^{\varphi}(x_{j})$ and $K_{\varphi,j}\equiv
K_{\varphi}(x_{j})$. The metric of space-time can be implemented as a quantum
operator acting on the physical space of states by writing it as a
parameterized observable given by the gauge fixing conditions. Details are in
us .

Let us consider stationary slices, that is, the gauge fixing conditions do not
depend on time. Preservation of these conditions us-improv  correspond to
$\overline{N}^{x}=\overline{N}=0$ and imply that
$NE^{\varphi}/(E^{x})^{\prime}$ is a constant that we take equal to $1/2$.
This in turns means that $N=(E^{x})^{\prime}/(2E^{\varphi})$ and this implies
that $N^{x}=K^{\varphi}\sqrt{E^{x}}/E^{\varphi}$. The gauge fixing determines
the Lagrange multipliers that were mentioned before.

We will first analyze what happens outside the “bounce” that replaces the
classical singularity and later study the covariance at the bounce. For the
physical states $|M,{\hat{k}}\rangle$ under consideration (with no
superpositions in $M$ and $k_{j}$), the Schwarzschild metric can be readily
obtained by fixing $K_{\varphi,j}=0$. It is given by,222Note that the line
element of the 2-spheres is determined by $g_{\theta\theta}=x_{j}^{2}$ and
$g_{\varphi\varphi}=x_{j}^{2}\sin^{2}\theta$.

$\displaystyle\hat{g}^{S}_{tt}(x_{j})$ $\displaystyle=$
$\displaystyle-\frac{\left[\left(\hat{E}^{x}_{j}\right)^{\prime}\right]^{2}}{4\left({E}^{\varphi}_{j}\right)^{2}}=-\hat{N}^{2}=-\left(1-\frac{\hat{r}_{S}}{\sqrt{\hat{E}^{x}_{j}}}\right),$
(15) $\displaystyle\hat{g}^{S}_{tt}(x_{j})|M,\vec{k}\rangle$ $\displaystyle=$
$\displaystyle-\left(1-\frac{\hat{r}_{S}}{x_{j}}\right)|M,\vec{k}\rangle,$
(16) $\displaystyle\hat{g}^{S}_{tx}(x_{j})$ $\displaystyle=$ $\displaystyle
0,\,\,\textrm{since}\,\,K_{\varphi,j}=0,$ (17)
$\displaystyle\hat{g}^{S}_{xx}(x_{j})$ $\displaystyle=$
$\displaystyle\frac{\left(E^{\varphi}_{j}\right)^{2}}{\hat{E}^{x}_{j}}=\frac{\left[\left(\hat{E}^{x}_{j}\right)^{\prime}\right]^{2}}{4\hat{E}^{x}_{j}}\frac{1}{1-\frac{\hat{r}_{S}}{\sqrt{\hat{E}^{x}_{j}}}}$
(18) $\displaystyle\hat{g}^{S}_{xx}(x_{j})|M,\vec{k}\rangle$ $\displaystyle=$
$\displaystyle\frac{\left(2x_{j}+\ell_{\rm
Planck}\right)^{2}}{4x_{j}^{2}}\frac{1}{1-\frac{{r}_{S}}{x_{j}}}|M,\vec{k}\rangle$
(19) $\displaystyle=$ $\displaystyle\left(1+\frac{\ell_{\rm
Planck}}{x_{j}}+\frac{\ell_{\rm
Planck}^{2}}{4x_{j}}\right)\left(1-\frac{{r}_{S}}{x_{j}}\right)^{-1}|M,\vec{k}\rangle$
$\displaystyle=$ $\displaystyle{\left(1+\frac{\ell_{\rm
Planck}}{2x_{j}}\right)^{2}}\left({1-\frac{{r}_{S}}{x_{j}}}\right)^{-1}|M,\vec{k}\rangle,$

with $\hat{r}_{S}=2G\hat{M}$. It should be noted that the calculation is
exact. Let us proceed to compare the result with the action of the metric with
a generic choice of functional parameter of the observable $K_{\varphi,j}$
only restricting to stationary foliations (independent on time) that will
include usual ones like the Painlevé–Gullstrand and Eddington–Finkelstein
ones.

On generic stationary foliation, lapse, shift and $E^{\varphi}_{j}$ can be
written as,

$\displaystyle N_{j}$ $\displaystyle=$
$\displaystyle\frac{1}{2}\frac{\left(E^{x}_{j}\right)^{\prime}}{E^{\varphi}_{j}},\qquad
N^{x}_{j}=\frac{\sin\left(\rho_{j}K_{\varphi,j}\right)}{\rho_{j}}\frac{\sqrt{E^{x}_{j}}}{E^{\varphi}_{j}},$
(20) $\displaystyle E^{\varphi}_{j}$ $\displaystyle=$
$\displaystyle\left(E^{x}_{j}\right)^{\prime}\left(2\sqrt{1+\frac{\sin^{2}\left(\rho_{j}K_{\varphi,j}\right)}{\rho_{j}^{2}}-\frac{2GM}{\sqrt{E^{x}_{j}}}}\right)^{-1}$
(21)

with the parameter of the observable $K_{\varphi,j}$ generic but time
independent. These are the quantum versions of the classical expressions
discussed above.

The choice of $K_{\varphi,j}$ completes the prescription for the gauge fixing
that characterizes the foliation. We recall that it may be considered as the
functional parameter of the parameterized observable that defines
$E_{j}^{\varphi}$, and through it the metric components, and therefore can be
chosen at will. Each choice determines a different system of coordinates. One
way of doing this is to introduce a function $F(x_{j})$ such that
$\sin\left(\rho_{j}K_{\varphi,j}\right)=F(x_{j})$ with
$F(x_{j})\in[-1,1]\,\,\forall x_{j}$ and therefore, with the notation
$F(x_{j})\equiv F_{j}\in[-1,1]$. Each choice of $F_{j}$ leads to a different
foliation, for instance $F(x_{j})=\rho_{j}\sqrt{r_{S}/\sqrt{E^{x}_{j}}}$ leads
to ingoing Painlevé–Gullstrand form of the metric extension  and
$F(x_{j})=\rho_{j}{r_{S}}/\sqrt{E^{x}_{j}(1+r_{S}/\sqrt{E^{x}_{j}})}$ to
ingoing Eddington-Finkelstein coordinates us-improv .333Outgoing coordinates
in both cases are defined similarly with a minus sign in the right hand side
of these choices of $F(x_{j})$. Note that as we explained for $K_{\varphi,j}$,
$F(x_{j})$ it can either be a c-number function or an operator, function of
the Dirac observables, and should be treated accordingly.

For a generic stationary foliation given by $F(x_{j})$ we have,

$\displaystyle\hat{N}_{F}(x_{j})$ $\displaystyle=$
$\displaystyle\sqrt{1+\frac{F_{j}^{2}}{\rho_{j}^{2}}-\frac{r_{S}}{\sqrt{\hat{E}^{x}_{j}}}},$
(22) $\displaystyle\hat{N}^{x}_{F}(x_{j})$ $\displaystyle=$
$\displaystyle\frac{2F_{j}}{\rho_{j}}\frac{\sqrt{\hat{E}^{x}_{j}}}{\left(\hat{E}^{x}_{j}\right)^{\prime}}\hat{N}_{F}(x_{j}),$
(23) $\displaystyle\hat{g}^{F}_{xx}(x_{j})$ $\displaystyle=$
$\displaystyle\frac{\left(\left(\hat{E}^{x}_{j}\right)^{\prime}\right)^{2}}{4\hat{E}^{x}_{j}}\hat{N}_{F}(x_{j})^{-2},$
(24) $\displaystyle\hat{g}^{F}_{tt}(x_{j})$ $\displaystyle=$
$\displaystyle-1+\frac{\hat{r}_{S}}{\sqrt{\hat{E}^{x}_{j}}},$ (25)
$\displaystyle\hat{g}^{F}_{tx}(x_{j})$ $\displaystyle=$
$\displaystyle\hat{g}_{xx}\hat{N}^{x}_{F}(x_{j}).$ (26)

We would like to show that the length of a space-time curve
$\left(t(x),x\right)$ is invariant. If the state of the black hole system is
given by the basis element $|M,{\hat{k}}\rangle$ defined in Eqs. (6-10), to
each function $t(x)$ corresponds, in Schwarzchild coordinates, a polygonal
curve in the plane $(t,x)$ described by a discrete set of points
$[...(t_{j},x_{j}),(t_{j+1},x_{j+1})...]$ where
$\sqrt{(}\hat{E}^{x}_{j})|M,{\hat{k}}\rangle=\ell_{\rm
Planck}\sqrt{k_{j}}|M,{\hat{k}}\rangle=(|j|+j_{0})\ell_{\rm
Planck}|M,{\hat{k}}\rangle=x_{j}|M,{\hat{k}}\rangle$ and
${\widehat{t}(x_{j})}|M,{\hat{k}}\rangle=t(x_{j})|M,{\hat{k}}\rangle$. More
general polygonal curves may be defined by composition of these curves. To be
able to discuss changes of slicings in a situation where space is discrete it
is necessary to consider polygonal curves in space-time

We assume that the quantum version of the invariant interval between two
successive points of the polygonal curve acting on a basis element
$|M,{\vec{k}}\rangle$ of the physical space of states may be written as

${(\widehat{\Delta
s_{j}}})^{2}={\widehat{g_{ab}(t_{j},x_{j})}}\widehat{\Delta{x^{a}_{j}}}\widehat{\Delta{x^{b}_{j}}},$
(27)

with ${\widehat{\Delta
x^{0}_{j}}}={\hat{t}}_{j+1}-{\hat{t}}_{j}={\widehat{\Delta t}_{j}}$ and
${\widehat{\Delta x^{1}_{j}}}={\hat{x}}_{j+1}-{\hat{x}}_{j}={\widehat{\Delta
x}_{j}}$.

In particular, the invariant interval between two successive points of the
polygonal in Schwarzschild coordinates $\Delta s^{S}_{j}$ and generic
stationary coordinates $\Delta s^{F}_{j}$ is,

$\displaystyle(\Delta s^{S}_{j})^{2}$ $\displaystyle=$
$\displaystyle-\left(1-\frac{r_{S}}{\sqrt{E^{x}_{j}}}\right)\Delta
t^{2}_{j}+\frac{\left(\left(E^{x}_{j}\right)^{\prime}\right)^{2}}{4E^{x}_{j}}\frac{1}{1-\frac{r_{S}}{\sqrt{E^{x}_{j}}}}\Delta{x_{j}}^{2},$
(28) $\displaystyle(\Delta s^{F}_{j})^{2}$ $\displaystyle=$
$\displaystyle-\left(1-\frac{r_{S}}{\sqrt{E^{x}_{j}}}\right)\Delta
t_{F,j}^{2}+2\frac{F_{j}}{\rho_{j}}\frac{\left(E^{x}_{j}\right)^{\prime}}{2\sqrt{E^{x}_{j}}}\frac{1}{N_{F}(x_{j})}\Delta
t_{F,j}\Delta{x_{j}}$ (29)
$\displaystyle+\frac{\left(\left(E^{x}_{j}\right)^{\prime}\right)^{2}}{4E^{x}_{j}}\frac{1}{N_{F}(x_{j})^{2}}\Delta{x_{j}}^{2},$

where we have omitted the hats indicating that $E^{x}_{j}$, $N_{j}$, $\Delta
s_{j}^{S,F}$, $r_{S}$ and $F_{j}$ are operators. By now the reader should
notice by context what we are referring to with the expressions. So setting
${\hat{t}_{F}(x_{j})}=\hat{t}(x_{j})-{\hat{a}(x_{j})}$ and taking into account
[28],

$\displaystyle(\Delta s^{F}_{j})^{2}$ $\displaystyle=$
$\displaystyle-\left(1-\frac{r_{S}}{\sqrt{E^{x}_{j}}}\right)\Delta
t_{F,j}^{2}+2\frac{\Delta a_{j}}{\Delta
x_{j}}\left(1-\frac{r_{S}}{\sqrt{E^{x}_{j}}}\right)\Delta t_{F,j}\Delta x_{j}$
(30)
$\displaystyle+\left(\frac{\left(E^{x}_{j}\right)^{\prime}}{4E^{x}_{j}}\frac{1}{1-\frac{r_{S}}{\sqrt{E^{x}_{j}}}}-\left(1-\frac{r_{S}}{\sqrt{E^{x}_{j}}}\right)\left(\frac{\Delta
a_{j}}{\Delta x_{j}}\right)^{2}\right)\Delta x^{2}_{j}$

and setting,

$\Delta
a_{j}=\frac{F_{j}}{\rho_{j}}\frac{\left(E^{x}_{j}\right)^{\prime}}{2\sqrt{E^{x}_{j}}}\frac{1}{N_{j}}\frac{1}{1-\frac{r_{S}}{\sqrt{E^{x}_{j}}}}\ell_{\rm
Planck},$ (31)

one can see that the intervals $\Delta s^{F}_{j}$ computed in (29, 31)
coincide and therefore we have shown that the line element is invariant.

This allows to construct $a(x_{j})$ from $a(x_{S})$ and one recovers the
classical change of coordinates when $\ell_{\rm Planck}$ is taken to be
infinitesimally small. The polygonal line element is invariant up to all
orders in terms of Planck’s length.

This illustrates the invariance when one changes coordinates that imply a
change of slicing. It is clear that changes that preserve the foliation keep
the line element invariant, provided they are well defined. For example, let
us consider the “tortoise” coordinate defined as

$\displaystyle\left(1-\frac{r_{S}}{x_{j}}\right)\left(x^{*}_{j+1}-x^{*}_{j}\right)^{2}$
$\displaystyle=$ $\displaystyle\left(1+\frac{\ell_{\rm
Planck}}{2x_{j}}\right)^{2}\left(1-\frac{r_{S}}{x_{j}}\right)^{-1}\left(x_{j+1}-x_{j}\right)^{2},$
(32)

or equivalently,

$x^{*}_{j+1}-x^{*}_{j}=\left(1+\frac{\ell_{\rm
Planck}}{2x_{j}}\right)\left(1-\frac{r_{S}}{x_{j}}\right)^{-1}\ell_{\rm
Planck},$ (33)

where we replaced $x_{j+1}-x_{j}=\ell_{\rm Planck}$ for all $j$. This equation
determines all $x^{*}_{j}$ provided the value of, for instance, $x^{*}_{S}$.
Again, we note that this change of radial coordinate leaves invariant the line
element, by construction.

Up to now we have analyzed the case $k_{j}=(|j|+j_{0})^{2}$ and ignoring
superpositions in the quantum states. It is easy to extend the analysis to a
general case of $|M,\vec{k}\rangle$. If we define $x_{j}=\sqrt{k_{j}}\ell_{\rm
Planck}$ with a non-uniform spacing, with the discrete interval defined as
$(\Delta s_{j})^{2}=g_{ab}(x_{j})\Delta x_{j}^{a}\Delta x_{j}^{b}$ and $\Delta
x_{j}=\sqrt{k_{j+1}}-\sqrt{k_{j}}$ in the gauge $E^{x}_{j}={\rm
sig}(j)x^{2}_{j}$, the previous proof can be extended easily.

For a generic superposition states,

$\int dM\sum_{k}c\left(\vec{k},M\right)|M,\vec{k}\rangle,$ (34)

the proof can be extended since $\hat{M},\hat{E}^{x}$ commute and therefore
$\hat{g}_{ab}$ can be defined without ordering ambiguities. One can see that
the intervals $\Delta s^{S}_{j}$ and $\Delta s^{F}_{j}$ (29,31) coincide for
any element of the physical basis $|M,\vec{k}\rangle$ and therefore their
expectation value coincide for any element of the physical space of states.
However, due to fluctuations, as it is usual in quantum mechanics, even though
the expectation value $\langle{(\widehat{\Delta
s_{j}}})^{2}\rangle=\langle{\widehat{g_{ab}(t_{j},x_{j})}}\widehat{\Delta{x^{a}_{j}}}\widehat{\Delta{x^{b}_{j}}}\rangle$
is invariant,
$\langle{\widehat{g_{ab}(t_{j},x_{j})}}\rangle\langle\widehat{\Delta{x^{a}_{j}}}\rangle\langle\widehat{\Delta{x^{b}_{j}}}\rangle$
it is not. Thus in a highly quantum regime the length of a curve is gauge
invariant but there will be correction to the tensorial behavior of the
metric. The previous analysis then provides an explicit and operational notion
of quantum covariance.

Let us now address the covariance of the framework at the bounce that replaces
the singularity in extension . It is important to remark that the bounce
occurs at a point that may be identified in a way that is invariant under
changes of foliation and radial coordinates and is given by the infimum of
$|E_{j}^{x}|$. Thus, the bounce hypersurface is slicing independent and
covariantly defined being the geometric description unique up to this point
and, as we shall see, beyond. We start from Schwarzschild’s metric given in
Eqs. (15), (17) and (18). Note that, for this metric, the region
$j<r_{S}/\ell_{\rm Planck}-j_{0}$ is foliated by $x_{j}=const$ hypersurfaces,
i.e. a nonstationary slicing. However, as we will see, our discussion about
covariance is still valid. 444Let us note that it is not difficult to carry
out the proof of covariance at the bounce starting from horizon-penetrating
coordinates stationary at the exterior. We adopt the nonstationary
Schwarschild’s metric for the sake of simplicity. At the bounce
$x_{0}=j_{0}\ell_{\rm Planck}$, we have that,

$\displaystyle g^{S}_{tt}\left(x_{0}\right)$ $\displaystyle=$
$\displaystyle-\left(1-\frac{r_{S}}{j_{0}\ell_{\rm Planck}}\right),$ (35)
$\displaystyle g_{tx}^{S}\left(x_{0}\right)$ $\displaystyle=$ $\displaystyle
0,$ (36) $\displaystyle g_{xx}^{S}\left(x_{0}\right)$ $\displaystyle=$
$\displaystyle\left(\frac{\left(E^{x}_{0}\right)^{\prime}}{4E^{x}_{0}}\right)^{2}\left({1-\frac{r_{S}}{j_{0}\ell_{\rm
Planck}}}\right)^{-1}.$ (37)

Explicitly, acting on a state,

$\left(E^{x}_{0}\right)^{\prime}=\frac{\left(j_{0}\ell_{\rm Planck}+\ell_{\rm
Planck}\right)^{2}-j_{0}^{2}\ell_{\rm Planck}^{2}}{\ell_{\rm
Planck}}=\left(2j_{0}+1\right)\ell_{\rm Planck},$ (38)

and therefore,

$\frac{\left(\left(E^{x}_{0}\right)^{\prime}\right)^{2}}{4E^{x}_{0}}=\frac{\left(2j_{0}+1\right)^{2}}{4j_{0}}=\left(1+\frac{\ell_{\rm
Planck}}{2j_{0}\ell_{\rm Planck}}\right),$ (39)

and

$g^{S}_{xx}\left(x_{0}\right)=\left(1+\frac{1}{2j_{0}}\right)^{2}\left(1-\frac{r_{S}}{j_{0}\ell_{\rm
Planck}}\right)^{-1}.$ (40)

Whereas at $x_{-1}$, the point beyond where the classical singularity would
have been (recall (10)),

$\displaystyle g^{S}_{tt}\left(x_{-1}\right)$ $\displaystyle=$
$\displaystyle-\left(1-\frac{r_{S}}{\left(j_{0}+1\right)\ell_{\rm
Planck}}\right),$ (41) $\displaystyle g^{S}_{tx}\left(x_{-1}\right)$
$\displaystyle=$ $\displaystyle 0,$ (42)

and

$\displaystyle{\left(E^{x}_{-1}\right)^{\prime}}$ $\displaystyle=$
$\displaystyle\frac{j_{0}^{2}\ell_{\rm
Planck}^{2}-\left(j_{0}+1\right)^{2}\ell_{\rm Planck}^{2}}{\ell_{\rm
Planck}}=-\left(2j_{0}+1\right)\ell_{\rm Planck},$ (43)
$\displaystyle\frac{\left(\left(E^{x}_{-1}\right)^{\prime}\right)^{2}}{4E^{x}_{-1}}$
$\displaystyle=$
$\displaystyle\frac{\left(2j_{0}+1\right)^{2}}{4\left(j_{0}+1\right)^{2}}=\frac{4j_{0}^{2}+4j_{0}+1}{4\left(j_{0}+1\right)^{2}}=\left(1-\frac{1}{2\left(j_{0}+1\right)}\right)^{2},$
(44)

and as a consequence,

$g_{xx}^{S}\left(x_{-1}\right)=\left(1-\frac{1}{2\left(j_{0}+1\right)}\right)^{2}\left(1-\frac{r_{S}}{\left(j_{0}+1\right)\ell_{\rm
Planck}}\right)^{-1}.$ (45)

Let us now consider the generic stationary metric $g^{F}_{ab}$ with
$F(x_{j})\in[-1,1]$. We will see that for the system to describe correctly the
bounce $F(x_{0})$ must be close to one (as usual, not all gauge choices allow
to reach the singularity, in this case, the bounce). We start with the general
expression of the metric Eqs. (24), (25) and (26), and evaluate them at the
bounce, namely,

$\displaystyle g_{tt}^{F}\left(x_{0}\right)$ $\displaystyle=$
$\displaystyle-\left(1-\frac{r_{S}}{j_{0}\ell_{\rm Planck}}\right),$ (46)
$\displaystyle g_{tx}^{F}\left(x_{0}\right)$ $\displaystyle=$
$\displaystyle\sqrt{\frac{\pi}{\Delta}}{\left(E^{x}_{0}\right)^{\prime}\sqrt{[F(x_{0})]^{2}}}\left(1-\frac{r_{S}}{j_{0}\ell_{\rm
Planck}}+\frac{4\pi j_{0}^{2}\ell_{\rm
Planck}^{2}[F(x_{0})]^{2}}{\Delta}\right)^{-1/2}$ (47) $\displaystyle=$
$\displaystyle\sqrt{\frac{\pi}{\Delta}}\left(2j_{0}+1\right)\ell_{\rm
Planck}\sqrt{[F(x_{0})]^{2}}\left(1-\frac{r_{S}}{j_{0}\ell_{\rm
Planck}}+\frac{4\pi j_{0}^{2}\ell_{\rm
Planck}^{2}[F(x_{0})]^{2}}{\Delta}\right)^{-1/2},$ $\displaystyle
g_{xx}^{F}\left(x_{0}\right)$ $\displaystyle=$
$\displaystyle\left(1+\frac{1}{2j_{0}}\right)^{2}\left(1-\frac{r_{S}}{j_{0}\ell_{\rm
Planck}}+\frac{4\pi j_{0}^{2}\ell_{\rm
Planck}^{2}[F(x_{0})]^{2}}{\Delta}\right)^{-1},$ (48)

with $[F(x_{0})]^{2}>\left(r_{S}-j_{0}\ell_{\rm Planck}\right)\Delta/(4\pi
j_{0}^{3}\ell_{\rm Planck}^{3})$.

The metric at $x_{-1}$, i.e. $j=-1$ is,

$\displaystyle g_{tt}\left(x_{-1}\right)$ $\displaystyle=$
$\displaystyle-\left(1-\frac{r_{S}}{(j_{0}+1)\ell_{\rm Planck}}\right),$ (49)
$\displaystyle g_{tx}\left(x_{-1}\right)$ $\displaystyle=$
$\displaystyle-\sqrt{\frac{\pi}{\Delta}}\left(2j_{0}+1\right)\ell_{\rm
Planck}\sqrt{[F(x_{-1})]^{2}}$ (50)
$\displaystyle\times\left(1-\frac{r_{S}}{(j_{0}+1)\ell_{\rm
Planck}}+\frac{4\pi(j_{0}+1)^{2}\ell_{\rm
Planck}^{2}[F(x_{-1})]^{2}}{\Delta}\right)^{-1/2},$

now with $[F(x_{-1})]^{2}>\left(r_{S}-(j_{0}+1)\ell_{\rm
Planck}\right)\Delta/(4\pi(j_{0}+1)^{3}\ell_{\rm Planck}^{3})$. Notice that
$g_{tx}^{F}$ changes sign at the bounce since
$\left(E^{x}_{0}\right)^{\prime}$ is positive and
$\left(E^{x}_{-1}\right)^{\prime}$ is negative. This does not introduce
singularities in the curvature, as we have shown explicitly in extension ,
where we proved that it is of order Planck at the bounce.

For the spatial component we have that,

$g_{xx}^{F}\left(x_{-1}\right)=\left(1-\frac{1}{2\left(j_{0}+1\right)}\right)^{2}\left(1-\frac{r_{S}}{(j_{0}+1)\ell_{\rm
Planck}}+\frac{4\pi\ell_{\rm
Planck}^{2}(j_{0}+1)^{2}[F(x_{-1})]^{2}}{\Delta}\right)^{-1}.$ (51)

Making the substitution $t_{F}(x_{0})=t(x_{0})-a(x_{0})$, the invariant line
element is,

$\Delta
a(x_{0})=\frac{F(x_{0})}{\rho_{0}}\frac{\left(E^{x}_{0}\right)^{\prime}}{2\sqrt{E^{x}_{0}}}\left(1+\frac{[F(x_{0})]^{2}}{\rho_{0}^{2}}-\frac{r_{S}}{\sqrt{E^{x}_{0}}}\right)^{-1/2}\left(1-\frac{r_{S}}{\sqrt{E^{x}_{0}}}\right)^{-1}\ell_{\rm
Planck},$ (52)

and $\Delta a(x_{-1})$ is identical substituting $x_{0}\to x_{-1}$. $\Delta a$
changes sign but is continuous when $\ell_{\rm Planck}$ is taken to be
infinitesimally small.

Following the arguments discussed above, one can easily show that this notion
of quantum covariance are immediately applicable for all nodes with $j<0$. The
only difference arise in a global sign in (31), which indicates that this
region is covered by outgoing coordinates if one starts with ingoing
coordinates at $j>0$ (and viceversa).

We have also studied the covariance of several curvature scalars: the Ricci
and the Kretschmann scalars, and the scalar obtained by contracting the Weyl
tensor with itself. We checked that in the approximation where $x_{j}$ is
treated as a continuous variable, which allows to use derivatives instead of
finite differences, these scalars do not depend on the choice of the gauge
function $F(x)$. This gives robustness to our model regarding its covariance.
It remains to be checked if the discrete version of these scalars is also
slicing independent. Nevertheless, the ideas presented in this manuscript
regarding the invariance of the spacetime line element of a discrete quantum
geometry opens the possibility of studying the covariance of discrete versions
of curvature operators and the invariance of curvature scalars.

Given the granularity of space time at the Planck scale, quantum gravity
should provide a new principle that replaces general covariance. But it must
still obey certain consistency conditions related to independence of physical
effects on the frames we are using, provided these frames are realizable in
the quantum theory. Reference frames are associated to physical observations,
by a system of observers: at rest, free falling or others. In a quantum theory
of gravity not all reference frames will be physically implementable. The
holonomization condition that takes an extrinsic curvature of the form
$\sin({\rho_{j}}{K_{\varphi}(x_{j})})=F(x_{j})$ with $|F(x_{j})|\leq 1$
provides for each $F$ an explicit definition for the realizable foliations.
Also notice that covariance allows to eliminate some ambiguities. For instance
in principle it could be possible to choose different polimerizations for the
shift and the spatial metric as we did in us-improv  that would not lead to a
quantum covariant formulation. The covariant version of the improved
quantization appears in extension .

We have shown here that there exists a quantum operator extension of the line
element whose expectation value in any state is independent on the quantum
stationary foliation chosen. This provides an explicit and operational notion
of quantum covariance that reproduces the usual one at the classical limit.
When quantum reference frames are considered, the relation among coordinates
associated to two different frames have quantum nature and depend on the
observables and c-number functions that describe the change of reference
frame. In the explicit case of spherical symmetry in which the radial
coordinate is quantized by $x_{j}$ once the stationary foliation $F(x_{j})$ is
specified, the description of a given curve whose invariant length we want to
evaluate in two different coordinate systems is given in terms of an operator
${\hat{t}}(O,M,S,j)$ whose form we have determined. The use of parameterized
observables for the coordinate dependent quantities and operatorial change of
coordinates as considered in this paper should be present in any approach to
quantum gravity. Although we have only shown covariance for the line element
for generic stationary slicings, it is likely that it can also be shown for
other non-stationary foliations of space-time and scalar quantities that are
functions of the geometry. It opens the possibility of discussing covariance
in the presence of a discrete geometry. These ideas are not restricted to loop
quantum gravity or spherical symmetry.

We wish to thank Martin Bojowald for many useful comments. This work was
supported in part by Grant NSF-PHY-1903799, funds of the Hearne Institute for
Theoretical Physics, CCT-LSU, Pedeciba, Fondo Clemente Estable FCE 1 2019 1
155865 and the Spanish Government through the projects FIS2017-86497-C2-2-P,
PID2019-105943GB-I00 (with FEDER contribution), and the “Operative Program
FEDER2014-2020 Junta de Andalucía-Consejería de Economía y Conocimiento” under
project E-FQM-262-UGR18 by Universidad de Granada.

## References

  * (1) R. Gambini and J. Pullin, Phys. Rev. Lett. 110, no.21, 211301 (2013) doi:10.1103/PhysRevLett.110.211301 [arXiv:1302.5265 [gr-qc]]; R. Gambini, J. Olmedo and J. Pullin, Class. Quant. Grav. 31, 095009 (2014) doi:10.1088/0264-9381/31/9/095009 [arXiv:1310.5996 [gr-qc]] and references therein.
  * (2) M. Bojowald, Universe 6 (2020) no.8, 125 doi:10.3390/universe6080125 [arXiv:2009.13565 [gr-qc]]; Universe 7, no.7, 251 (2021) doi:10.3390/universe7070251 [arXiv:2108.11936 [gr-qc]]; M. Bojowald, S. Brahma and J. D. Reyes, Phys. Rev. D 92, no.4, 045043 (2015) doi:10.1103/PhysRevD.92.045043 [arXiv:1507.00329 [gr-qc]].
  * (3) R. Gambini, J. Olmedo and J. Pullin, Class. Quant. Grav. 37 (2020) no.20, 205012 doi:10.1088/1361-6382/aba842 [arXiv:2006.01513 [gr-qc]].
  * (4) K. Kuchar, Phys. Rev. D 50, 3961-3981 (1994). doi:10.1103/PhysRevD.50.3961 [arXiv:gr-qc/9403003]
  * (5) J. Olmedo, Universe 2, 12 (2016). doi:10.3390/universe2020012 [arXiv:1606.01429 [gr-qc]]
  * (6) R. Gambini, J. Olmedo and J. Pullin, Front. Astron. Space Sci. 8, 74 (2021) doi:10.3389/fspas.2021.647241 [arXiv:2012.14212 [gr-qc]].