# Graphing else matters: exploiting aspect opinions and ratings in explainable
graph-based recommendations

Iván Cantador  Universidad Autónoma de MadridSpain ,  Andrés Carvallo
Pontificia Universidad Católica de ChileChile ,  Fernando Diez  Universidad
Autónoma de MadridSpain and  Denis Parra  Pontificia Universidad Católica de
ChileChile

###### Abstract.

The success of neural network embeddings has entailed a renewed interest in
using knowledge graphs for a wide variety of machine learning and information
retrieval tasks. In particular, current recommendation methods based on graph
embeddings have shown state-of-the-art performance. These methods commonly
encode latent rating patterns and content features. Different from previous
work, in this paper, we propose to exploit embeddings extracted from graphs
that combine information from ratings and aspect-based opinions expressed in
textual reviews. We then adapt and evaluate state-of-the-art graph embedding
techniques over graphs generated from Amazon and Yelp reviews on six domains,
outperforming baseline recommenders. Our approach has the advantage of
providing explanations which leverage aspect-based opinions given by users
about recommended items. Furthermore, we also provide examples of the
applicability of recommendations utilizing aspect opinions as explanations in
a visualization dashboard, which allows obtaining information about the most
and least liked aspects of similar users obtained from the embeddings of an
input graph.

knowledge graphs, graph embeddings, aspect-based opinions

††journal: POMACS††journalvolume: 37††journalnumber: 4††article:
111††publicationmonth: 8††ccs: Information systems Recommender systems††ccs:
Computing methodologies Neural networks

##  1\. Introduction

In recent years, knowledge graphs (KG) have been widely used in recommendation
(He et al., 2015; Wang et al., 2018b). A KG is a graph data structure
containing information about semantic entities (or concepts) expressed as
nodes, and semantic relations between entities expressed as edges. The
relations can be represented as ¡subject, property, object¿ triples whose
elements may belong to structured knowledge bases or ontologies. Subject and
object entities (nodes) can refer to users, items, item metadata, and content
features, among others. Compared to recommendation approaches that treat
relations between users and items independently, exploiting a KG graph allows
capturing deeper relations valuable to generate better recommendations (Wang
et al., 2019a). Another benefit of a KG-based recommender is its capacity to
deal with cold-start situations for new users or unrated items by encoding
sequential dependency among them (Hu et al., 2018; Huang et al., 2018).
Moreover, they enable the generation of human-interpretable recommendations,
making the systems more reliable and transparent for the end-user (Wang et
al., 2019b; He et al., 2015; Xian et al., 2019).

The broad majority of graph-based recommendation methods makes use of
information underlying rating relations and connections between items with
similar attributes (Wang et al., 2019b). In general, these methods encode
latent rating patterns and content features using graph embeddings, since the
information stored in KG triples can be too computationally expensive to
manipulate. Knowledge graph embeddings (Cai et al., 2018) consist of learned
lower dimension, continuous vectors for entities and relations that preserve
the original structure of the input KG.

In this context, a source of information that has been barely explored for KG
is the textual content of user reviews. In the research literature, it has
been shown that both ratings and reviews can give support and trust to users
in decision making tasks (Lackermair et al., 2013). In particular, approaches
have been proposed to automatically extract personal opinions about item
aspects (i.e., features, components and functionalities) from reviews (Poria
et al., 2016), and aspect-based opinions have been used to enrich sentiment
analysis models (de Kok et al., 2018), document classifiers (Ostendorff et
al., 2020), and recommender systems (He et al., 2015). Concerning the
exploitation of aspect-based opinions by recommendation methods, Musto et al.
(Musto et al., 2019b) made use of aspects to justify recommendations using
sentiment analysis and text summarization, whereas Wu and Ester (Wu and Ester,
2015) proposed a probabilistic approach that combines aspect-based opinion
mining and collaborative filtering.

Aspect-based opinions have also been included as additional features for
latent factor (Qiu et al., 2016) and neural network (Chin et al., 2018; Guan
et al., 2019) models. To the best of our knowledge, the approach by He et al.
(He et al., 2015) is the only one that takes advantage of a KG with
information from aspect-based opinions extracted from reviews, by enriching
user-item binary relation to an heterogeneous tripartite graph composed of
user-item-aspect ternary relations.

Unlike previous approaches, our method combines rating and aspect-based
opinion interactions in a KG embedding framework to obtain a richer
representation of users, items and aspects in the same latent space for making
recommendations. Furthermore, we also provide a use case of the applicability
of recommendations using aspect opinions as explanations in a visualization
dashboard that allows exploring information about the most and least liked
aspects of similar users obtained from the embeddings of an input graph.

Therefore, the main contributions of our work are as follows:

  1. (1)

Presenting a graph model that represents and relates users, items and aspects
in the same latent space for its exploitation by recommender systems.

  2. (2)

Using the graph model to adapt recommendation methods based on neural network
graph embeddings, showing empirical performance improvements over state-of-
the-art baselines.

  3. (3)

Proposing a method that uses the graph model to provide explanations of
generated recommendations at the aspect opinion level.

##  2\. Related work

In this section, we survey relevant literature on four research lines: graph-
based recommendations (subsection 2.1), review and text-based recommendations
(subsection 2.2), aspect-based recommendations (subsection 2.3), and graph-
based recommendation explanations (subsection 2.4).

###  2.1. Graph-based recommendations

We can distinguish between three main types of approaches that use graphs for
recommendation purposes: path-based, embedding-based, and unified. Path-based
methods take advantage of users and items connectivity score functions to
generate recommendations. For instance, Luo et al. (Luo et al., 2014) proposed
Hete-CF, which makes use of path similarity combined with common paths between
two entities in a social network to make recommendations. Concerning
embedding-based methods, a KG is used to obtain latent representations of
users and items, with which finding closest users and items. Examples of this
type of methods are the approach by Wang et al. (Wang et al., 2018c) for news
recommendation and Huang et al. (Huang et al., 2018), which incorporate
sequential information through a recurrent neural network, and the recent KGAT
(Wang et al., 2019a) method, which models high order relations in a KG.
Finally, unified methods exploit information from embedding- and path-based
forms together. Wang et al. (Wang et al., 2018b) proposed RippleNet, which
introduces the concept of preference propagation to refine user and item
representations obtained with embedding-based methods.

In our work, we advocate for embedding-based graph recommendations, since they
allow better addressing the cold start problem and efficiently representing
entities and relations, preserving the graph structure (Wang et al., 2019a;
Wang et al., 2018b). We also chose the embedding-based approach since the
state-of-the-art KGAT method (Wang et al., 2019a) achieved better performance
results than other embedding-based, path-based and unified approaches. This
led us to empirically compare our method with KGAT in the experimental
section.

###  2.2. Review and text-based recommendations

Several works have used textual information extracted from reviews to improve
the modeling of user preferences and recommendation performance. Shet et al.
(Shalom et al., 2019) proposed to use review text as input for a generative
recommendation model. Similarly, Chuang et al. proposed the TPR model (Chuang
et al., 2020), which makes use of user-item interaction combined with
relations between item and associated text for improving the modeling of user
preferences.

Another usage of user-generated content is done by DAML (Liu et al., 2019), a
dual attention model that learns from ratings and reviews texts inputs. RNS, a
review neural recommender model proposed by Li et al. (Li et al., 2019), uses
attention mechanisms over the users’ historical purchases and past reviews for
collaborative filtering. Lastly, Wang et al. (Wang et al., 2021) proposed
using other properties extracted from reviews different from text, such as
user age and review length, for learning user preferences and item attributes.

A major benefit of including textual data into a recommendation model is the
capability of providing human-readable explanations. One example of this is
the work done by Zhang et al. (Zhang et al., 2014), which incorporates
explainable sentence-level sentiment information extracted from reviews. In
the same context, other approaches have modeled the recommendation problem as
a multitask learning of user preferences and item features extracted from
reviews, allowing the generation of explanations of model predictions (Wang et
al., 2018a; Bansal et al., 2016).

Even though these models have taken advantage of information extracted from
text reviews, yielding competitive results, none of them have incorporated
such information into a KG to enrich the representation of users, items, and
their relations. Moreover, using the whole text can often contain redundant
and noisy information that can worsen the performance of the model (Agarwal et
al., 2007), compared to extracting relevant information in the form of
aspects, as proposed in this work. Furthermore, there have been approaches to
automatize text content analysis in the field of medicine(Carvallo et al.,
2020), along with stress tests(Aspillaga et al., 2020; Araujo et al., 2021) to
evaluate the models performance under different situations.

###  2.3. Aspect-based recommendations

Although many aspect-based recommenders exist in the research literature, a
few consider incorporating opinions about item aspects into a KG. In general,
aspect-based opinions have been used to justify recommendations, including
sentiment analysis and text summarization from reviews (Musto et al., 2019b).

Several works have exploited aspects to improve the performance of recommender
systems. For instance, the work proposed by Chin et al. (Chin et al., 2018)
provides additional knowledge by giving more attention to reviews aspects by
means of a neural model with attention mechanisms. Another work done by Musto
et al. (Musto et al., 2019a) proposed to justify recommendations through
aspects sentiment analysis on users reviews. In terms of combining ratings and
text data, a work done by Wu et al. (Wu and Ester, 2015) proposed a
probabilistic framework that exploits aspect-based opinion mining and
collaborative filtering. Moreover, using aspects as features of latent factors
(Qiu et al., 2016) and neural network models (Chin et al., 2018; Guan et al.,
2019) have also been investigated.

Regarding the incorporation of aspect-based opinion information into a KG,
TriRank (He et al., 2015) makes use of a tripartite graph representing
relationships between users, items, and aspects to generate recommendations.
Differently from TriRank, our method combines ratings and aspect-based
opinions in a KG to learn richer vector representations for both users and
items, and exploits the generated vectors to provide effective personalized
recommendations.

###  2.4. Graph-based recommendation explanations

The vast majority of works on graph-based recommenders aims to produce path-
based interpretations of generated recommendations, although it has been
stated that they require domain knowledge (Ma et al., 2019). Fu et al. (Fu et
al., 2020) made use of paths from users to items, mitigating unfairness
problems related to undesired recommendations from inactive users. Xian et al.
(Xian et al., 2019) proposed a reinforcement learning approach for
recommendation, where explanations are delivered by a policy-guided path
reasoning on a KG. Wang et al. (Wang et al., 2019b) suggested recommendation
explanations by showing items shared features and mining association rules.
Lastly, Huan et al. (Huang et al., 2019) proposed a method to provide path-
based explanation through user-rating interactions incorporating dynamic user
preferences.

Our recommendation method brings entities and relationships in the same latent
space, and presents embedding-based explanations, which have been proven to be
effective (Ma et al., 2019). Furthermore, we propose a visualization to give
more information at the aspect opinion level to final users using the
embeddings representations learned from an input KG.

##  3\. Proposed method

In this section, we first introduce our KG model, which incorporates ratings
and aspect-based opinions (subsection 3.1). Next, we explain the graph
embedding learning process conducted over our KG model (subsection 3.2).

###  3.1. Rating and aspect opinion knowledge graph

Figure 1. Example of ratings and aspect-based opinions graph. Users may like,
dislike or doesNotCare about item aspects, or may assign highRating or
lowRating values to items. An aspect belongsTo an item. Circle, triangle and
square nodes represent users, aspects and items, respectively. Dotted lines
correspond to rating relations, while solid lines are aspect-based opinion
relations.

Our KG representation model is depicted in Figure 1. Let $G=(N,R,E)$ be a
graph where N are nodes representing users, items and item aspects; R is the
set of possible relations between such entities: like, doesNotCare and dislike
for user-aspect relations, belongsTo for aspect-item relations, and highRating
and lowRating for user-to-item relations; and E represents the graph edges,
whose elements are semantic triples $<s,r,d>$ –source, relation and
destination–, where $s,d\in N$ and $r\in R$. More specifically, to represent
ratings given by users to items as triples $<s,r,d>$, we use two types of
relations: highRating and lowRating. Considering a 5-point Likert scale, if a
rating is lower than or equal to 3, then it is transformed into a lowRating
relation; otherwise, it is represented as highRating relation. An aspect-based
opinion, by contrast, is represented as two triples $<s,r,d>$: one that
relates the user and the aspect, and other that relates the aspect and the
item. The former relation can be a likes, doesNotCare or dislikes relation
depending on the polarity (positive, neutral or negative) of the opinion given
to the aspect. This polarity is computed through the opinion mining technique
proposed by Hernández-Rubio et al. (Hernández-Rubio et al., 2019). If the
extracted polarity is $0$, it means the user doesnotCare about the aspect for
that particular item; if the polarity is greater than $0$, the user likes the
item’s aspect, and finally, if the polarity is lower than $0$, the user
dislikes the item’s aspect. The latter relation is always belongsTo, stating
that the aspect is associated to the item. An illustrative KG example is shown
in Figure 1, where user1 likes aspect1, which belongs to item1, despite the
fact that the item receives a low rating from the user; user2 dislikes the
above aspect of item1, but assigns a high rating to the item; and user3 does
not care about aspect2 of item1, and does not rate the item.

###  3.2. Knowledge graph embedding-based recommendations

Table 1. Notation used in the KG embedding learning process. Symbol | Description
---|---
$\theta_{s}$ | learnable parameters for source node.
$\theta_{d}$ | learnable parameters for destination node.
$\theta_{r}^{(i)}$ | learnable parameters for relations types $r^{(i)}$
$sim(.)$ | similarity function.
$g(.)$ | transformation function.
$f(.)$ | fitness function that combines similarity ($sim(.)$) and transformation function ($g(.)$).
$f(e)$ | fitness function over original training edges.
$f(e^{\prime})$ | fitness function over negative sampled edges.
$\lambda$ | margin meta-parameter.
$Re(.)$ | real part of complex vector operation.
$conj(x)$ | conjugate of vector x.
$<\theta_{a},\theta_{b},\theta_{c}>$ | trilinear product between embeddings a,b and c.
$\theta_{a}\odot\theta_{b}$ | element-wise multiplication between embeddings a and b.

On the proposed KG, we used PyTorch BigGraph (PBG), a graph embedding learning
framework released by Facebook AI (Lerer et al., 2019) for training embeddings
on huge graphs. We use this framework since it supports including aspect and
rating interactions in a graph-based recommendation system, allowing us to
train embeddings for each entity and relations in the same latent space
efficiently due to parallelization of the learning process into multiple
partitions of the graph.

To explain the PBG representation learning process, we define $E$ as a set of
edges represented as triples $<s,r^{(i)},d>$ that connect source (s) and
destination (d) nodes trough one or many types of relations ($r^{(i)}$). As
shown in the previous section, source nodes can be users, items, and aspects
depending on type of relations, where highRating/lowRating relations directly
connect users and items, like/dislike relations connect users to aspects, and
belongsTo relation connects aspects to items (for more details on notation see
Table 1).

For each triple, the PBG model have parameters
$(\theta_{s},\theta_{r}^{(i)},\theta_{d})$ representing source nodes ($s$),
relations ($r^{(i)}$), and destination nodes ($d$), and a fitness function $f$
whose purpose is to reach a maximum in the cases where the triple $e$ belongs
to $E$, and to minimize cases when the edge does not exists in the graph.

To achieve this, $f$ encodes a latent representation learned for $s$, $d$
through different types of relations expressed as $r^{(i)}$, the function is
the following:

$f(\theta_{s},\theta_{r}^{(i)},\theta_{d})=sim(g_{s}(\theta_{s},\theta_{r}^{(i)}),g_{s}(\theta_{d},\theta_{r}^{(i)})$

where the transformation function $g$ encodes the relation between nodes and
relations $r^{(i)}$ vectors, and the similarity function ($sim$) computes a
similarity score between both transformed vectors.

PBG combines similarity ($sim$) and transformation ($g$) functions to learn
embeddings for source nodes, destination nodes, and relations. This
combination of functions varies depending on the type of relations being
modeled by minimizing the following triple loss function:

$L=\sum_{e\in E}\sum_{e^{\prime}\in
S^{\prime}_{e}}max(f(e)-f(e^{\prime})+\lambda,0)$

where $S^{\prime}_{e}$ is a set of edges obtained using negative sampling,
which samples half of them from corrupting one node from training edges and
the other half from triples randomly created, and $\lambda$ is a margin
hyperparameter.

In this case, as we have multiple relations and different types of nodes, we
use $Re(<\theta_{a},conj(\theta_{b})>)$ as similarity function. Re(.) denotes
the real part of a complex vector, $conj(.)$ is the complex conjugate of the
vector and $<.,.,.,>$ denotes the trilinear product.

Then, as transformation function, we use element-wise multiplication ($\odot$)
between the resulting transformed vector and the relation vector $\theta_{r}$.
Given this setup, the function f is defined as follows:

$f(\theta_{s},\theta_{r}^{(i)},\theta_{d})=Re(<\theta_{s}\odot\theta_{r}^{(i)},conj(\theta_{d}\odot\theta_{r}^{(i)})>)$

which corresponds to a complEX (Trouillon et al., 2016) learning
representation strategy, since we empirically obtained that it was the best
combination of transformation and similarity functions, in terms of
recommendation performance compared to transE (Bordes et al., 2013), transR
(Lin et al., 2015), RESCAL (Nickel et al., 2011) and Distmult (Yang et al.,
2014).

Finally, to generate the recommendations, as user and item embeddings are in
the same latent space, we look for the item embeddings closest to the target
user’s embedding using cosine similarity.

##  4\. Experiments

In this section, we report the conducted experiments, describing the used
datasets (subsection 4.1), presenting the evaluated recommendation methods
(subsection 4.2), and discussing the achieved results (subsection 4.3).

###  4.1. Datasets

To evaluate the proposed method, we used the well-known Amazon Reviews
dataset111https://jmcauley.ucsd.edu/data/amazon/ (AMZ) with reviews and
ratings for Movies & TV (MVT), Videogames (VGM), and Cellphones (CPH). For
this dataset, we considered only users with more than $10$ ratings. We also
experimented with other domains –Restaurants (RST), Hotels (HTL), and
Beauty&Spa (BTY)– using the Yelp Challenge
dataset222https://www.yelp.com/dataset (YLP), which, as the Amazon dataset,
contains both ratings and reviews.

Additionally, enriching the above datasets, we used the aspect opinions
dataset333http://ir.ii.uam.es/aspects/ provided by Hernández et al.
(Hernández-Rubio et al., 2019), which complements the ratings with aspect-
based opinions (i.e., [user, item, aspect, opinion polarity] tuples) extracted
from the Amazon and Yelp reviews.

Table 2 shows statistics of the datasets. It can be seen that the number of
ratings varies from 25 thousand to more than one million and three thousand
interactions. It can also be observed that the Yelp dataset for HTL and BTY
domains has fewer interactions compared to others. Regarding aspect-based
opinions, the number of relations varies from 13 thousand to more than 2
millions for the MVT domain. Furthermore, in general, users are more active
for rating interactions than aspect opinions, except for the MVT domain.

Table 2. Datasets statistics Dataset | Domain | Users | Items | Ratings | Aspect | Rating
---|---|---|---|---|---|---
|  |  |  |  | opinions | sparsity
AMZ | MVT | 40,058 | 131,638 | 1,349,351 | 2,132,927 | $2.4\cdot 10^{-4}$
VGM | 6,686 | 25,795 | 575,136 | 209,755 | $8.3\cdot 10^{-4}$
CPH | 7,237 | 47,183 | 464,684 | 85,930 | $3.4\cdot 10^{-4}$
YLP | RST | 36,471 | 4,503 | 791,865 | 547,892 | $9.6\cdot 10^{-4}$
HTL | 4,148 | 284 | 25,170 | 16,935 | $4.2\cdot 10^{-3}$
BTY | 4,270 | 764 | 27,885 | 13,217 | $1.7\cdot 10^{-3}$

###  4.2. Evaluated methods

We report experimental results from the following methods:

  * •

Non-personalized recommendation methods: random rating (RDM) and most popular
item (POP) recommendation methods.

  * •

Matrix Factorization optimized with alternate least squares (MF): a
traditional baseline; a matrix factorization model optimized with alternate
least-squares (Koren et al., 2009) using 200 latent factors, which yielded
better results than other traditional baselines.

  * •

Graph-based recommendation baseline (KGAT):  as a graph-based recommendation
state-of-the-art, Wang et al.’s KG Attention Network recommendation method
(Wang et al., 2019a) .

  * •

Graph embeddings from ratings (GER): a version of our proposed method that
considers only rating-based relations between users and items.

  * •

Graph embeddings from aspect-based opinions (GEA): a version of our proposed
method that considers likes, dislikes, and doesNotCare relations between users
and item aspects, and belongsTo relation between item aspects and items.

  * •

Graph embeddings from ratings and aspect-based opinions (GERA): our proposed
method considering both rating- and aspect opinion-based relations.

We note that content-based filtering, user-based collaborative filtering
(Schafer et al., 2007), item-based collaborative filtering (Schafer et al.,
2007), matrix factorization using alternated least-squares (Koren et al.,
2009), and matrix factorization using Bayesian personalized ranking (Rendle et
al., 2012) were also evaluated, performing worse.

###  4.3. Achieved results

In an experimental setup phase, we performed a grid search testing a range of
parameters, obtaining best performance results for embeddings of dimension
400, a learning rate of 0.01, 5 epochs, and a hinge loss margin of 0.1.

For the evaluation, we used the OrdRec methodology (Koren and Sill, 2013),
where a recommendation method is evaluated on the test ratings at the user
level. We employed a 5-fold cross-validation strategy, and computed the F1@k
(k = 10,20,30) metric. Other metrics such as nDCG@k (k = 10,20,30), recall@k
(k = 10,20,30), and precision@k (k = 10,20,30) were also calculated, but they
did not give additional insights to the analysis we present next.

The results in Tables 3 and 4 indicate that in most of the domains, our method
(GERA) outperforms the baselines in terms of F1@k (10,20, 30). For a top-N
recommendation task, GERA outperforms state-of-the-art traditional and graph-
based recommendation methods. However, for the CPH domain, in terms of F1@10,
MF achieves the highest value (0.915), whereas, for F1@20 and F1@30, the best-
performing method was the aspect-based opinion graph (GEA) version of our
proposal, with values of 0.935 and 0.939, respectively. The worse results of
GERA in the CPH domain can be explained because, as shown in Table 2 this
domain does not have enough aspect opinions to improve the results by
combining them with ratings in a KG.

Besides, when comparing graph-based models that use only ratings (GER) with
the ones that consider only aspects (GEA), we observe that in the CPH, RST,
HTL and BTY domains, the GEA model performs better than GER. This can be
explained because, in addition to the fact that these domains have a
considerable amount of aspect interactions (cf. 2).

Furthermore, the model that uses only ratings (GER) performs better on the MVT
and VGM domains than those that use only aspects (GEA). This behavior can be
due to the vast amount of rating interactions in these particular domains (cf.
2). However, an interesting phenomenon is that when combining both models into
one that considers both rating and aspect opinion interactions (GERA), we have
better results than GEA and GER separately.

Table 3. Evaluation results on the Amazon dataset. Values in bold are the best
ones for each domain and metric. * indicates statistical significance best
result by multiple t-test using Bonferroni correction.

Dataset | Domain | Model | F1@10 | F1@20 | F1@30
---|---|---|---|---|---
|  | RDM | .801 | .812 | .813
| POP | .033 | .031 | .031
| MF | .905 | .914 | .917
MVT | KGAT | .924 | .938 | .942
| GER | .921 | .936 | .940
| GEA | .915 | .932 | .936
|  | GERA | .928* | .941* | .945*
|  | RDM | .800 | .807 | .806
|  | POP | .045 | .045 | .045
|  | MF | .909 | .914 | .915
AMZ | VGM | KGAT | .791 | .903 | .920
|  | GER | .791 | .901 | .920
|  | GEA | .781 | .898 | .918
|  | GERA | .934* | .943* | .945*
|  | RDM | .819 | .819 | .823
|  | POP | .043 | .043 | .043
|  | MF | .915* | .916 | .917
| CPH | KGAT | .888 | .929 | .933
|  | GER | .890 | .930 | .934
|  | GEA | .898 | .935* | .939*
|  | GERA | .880 | .928 | .933

Table 4. Evaluation results on the Yelp dataset. Values in bold are the best
ones for each domain and metric. * indicates statistical significance best
result by multiple t-test using Bonferroni correction.

Dataset | Domain | Model | F1@10 | F1@20 | F1@30
---|---|---|---|---|---
|  | RDM | .647 | .646 | .650
| POP | .107 | .107 | .107
| MF | .827 | .829 | .830
| RST | KGAT | .913 | .932 | .938
|  | GER | .914 | .931 | .936
|  | GEA | .927 | .940 | .944
|  | GERA | .933* | .944* | .947*
|  | RDM | .592 | .589 | .571
|  | POP | .468 | .468 | .468
|  | MF | .816 | .816 | .816
YLP | HTL | KGAT | .986 | .986 | .986
|  | GER | .983 | .983 | .983
|  | GEA | .985 | .985 | .985
|  | GERA | .987* | .987* | .987*
|  | RDM | .650 | .634 | .622
|  | POP | .263 | .263 | .263
|  | MF | .846 | .848 | .850
| BTY | KGAT | .984 | .984 | .984
|  | GER | .983 | .984 | .984
|  | GEA | .983 | .983 | .983
|  | GERA | .987* | .988* | .988*

##  5\. Recommendation explanations

In this section, we present a simple technique to generate explanations about
recommendations provided by our method. To obtain interpretable
recommendations at aspect level, we first recommend 30 (a chosen cut-off)
items whose embeddings are closer to the target user’s embedding. From them,
we search for the similar users, and gather the opinions these users made
about aspects from the recommended items. We first present our proposed
explainable recommendation dashboard (subsection 5.1), and then we report
statistics associated to explanations on each of the addressed datasets and
domains (subsection 5.2).

###  5.1. Explainable dashboard

In order to provide a concrete usage scenario about how our model can be
leveraged to produce a rich interactive user experience, we designed a
visualization dashboard. We followed Munzer nested visualization design
framework (Munzner, 2014) in order to reduce the design space and prevent
ineffective visualizations. We then start by answering the three questions:
What (input data for the dashboard), Why (visual tasks) and How (the actual
visualization and interactive encodings).

What. The input data for the dashboard are: A given user $u$, the list of
movies recommended with their corresponding score (score is data of type
quantitative), 2-dimensional UMAP (McInnes et al., 2018) embeddings for each
user in the dataset, set of aspects liked and disliked by each user on each
movie reviewed (with the corresponding frequency per user), record of ratings
per user, review text per user with aspects identified, pre-computed.

Why. The visual tasks in Munzner framework are presented in the the form of
pairs {action, target}. We identify the following visual tasks for designing
the dashboard:

  * •

{present, recommendation}: a visual element must let users see the list of
recommended items and trigger actions if the user selects an item.

  * •

{browse, neighborhood}: the user should be able to select an arbitrary group
users in order to conduct other visual tasks.

  * •

{compare, aspect distribution}: The user must the able to compare her own
distribution of aspects (liked and disliked) with the distribution of aspects
from other users arbitrarily selected.

  * •

{locate, outliers}: The user should be able to identify outlier users or
neighborhoods in order to help them interpret her own recommendations.

  * •

{identify, features}: The user should be able to identify features (aspects in
our case) in the textual description that she and other people have written.

  * •

{identify, aspect distribution}: the user must be able to visualize her own
liked and disliked aspects.

  * •

{identify, rating distribution}: the user must be able to visualize the rating
distribution of arbitrary people selected.

  * •

{identify, extremes}: The user should be able to identify extreme users in
terms of aspects they care about reviewing.

How. Once the visual tasks are identified, it is time to choose the actual
visual and interactive elements. The first decision we made here is actually
designing several views, where in each view we can see a different visual
element which will support the visual tasks identified in the previous step.
In terms of interactive design, all these views will be connected in order to
update the plots once the user selects certain elements in the dashboard.
Figure 2 shows an illustrative example where our proposed visualization
dashboard presents generated aspect-level explainable recommendations
consisting of:

Figure 2. Visual dashboard for interactive model exploration. (A)
recommendations where user can select one movie M, (B) this panel shows all
the users who watched movie M, each user represented by 2D UMAP embedding, the
active user is presented by a star. (C) distribution of liked and disliked
aspects for users selected in previous (B) step, (D) distribution of liked
items and disliked items for active user, (E) sampled review texts by people
selected in (B) with highlighted words corresponding to aspects, and (F)
rating distribution of users chosen in (B).

  1. A)

Recommended items and associated scores given by our recommendation method.
The user can select one of the recommended items to display other associated
visualizations.

  2. B)

2D chart of users that interacted with a chosen item. The dimensionality of
the graph-learned user embeddings is reduced by UMAP. In the dashboard
interface, one can select a neighborhood, where the current user is symbolized
with a star.

  3. C)

Barplots for most liked and most disliked aspects from the chosen neighborhood
given a selected recommended item.

  4. D)

Barplots for most liked and most disliked aspects from the current user.

  5. E)

Sampled reviews text for a selected item, indicating the liked aspects in
green, and the disliked aspects in red. It is worth mentioning that the model
captures important aspects and aspect synonyms in the reviews. Also, it is
possible to explore an item’s reviews with a previous and next button.

  6. F)

Distribution of ratings given by the subset of selected users to the chosen
item.

Use case. Following the example from Figure 2, an active user was recommended
ten movies, of which he chooses “The Avengers” (see Figure 2A). By choosing
this movie, all the users who have commented on the movie are displayed in a
2D plane (Figure 2B). The active user can then select a subgroup of closest
neighbors, and obtain the aspects they liked and disliked the most. In the
example, the most-liked aspect of the chosen neighborhood is “locations,” with
six liked interactions, and the most unliked aspect is “costumes,” with eight
dislike interactions (Figure 2C). Then, the active user can be compared with
others based on their most liked and disliked aspect distributions (Figure
2D); in the example, the most liked aspect of the active user is
“photography,” and the most disliked aspects are “cast,” “costume” and
“director”. Exploring the distribution of ratings (Figure 2F) of the chosen
users for the movie “Avengers,” nine users gave ratings 4 and 5. The other
rating values are distributed among other users, showing that the
recommendation score is similar to the nearest users. In the dashboard, one
can also explore examples of reviews with the positive aspect-based opinions
highlighted in green and the negatives aspect-based opinions highlighted in
red (Figure 2E).

###  5.2. Statistics of recommendation explanations

In Table 5, we report Coverage –percentage of items with aspect opinion-based
explanations–, lk/other –the proportion of likes over dislikes and
doesNotCare) relations–, #aspects –the average number of unique aspects–, and
asp/item –the average number of aspects per item.

Table 5. Statistics of the recommendation explanation technique for each domain. Results in bold are the best ones for each metric. Dataset | Domain | Coverage | lk/other | #aspects | asp/item
---|---|---|---|---|---
AMZ | MVT | 47.90% | .663 | 10.30 | .716
VGM |  90.23% | .584 | 18.52 | .684
CPH | 70.38% | .592 | 12.27 | .581
YLP | RST | 11.11% | .508 | 6.22 | 1.87
HTL | 21.64% | .796 | 22.34 | 3.44
BTY | 20.02% | .653 | 17.55 | 2.88

These results indicate that the proposed method has better coverage in the VGM
domain with a 90.23% value; this shows that of the 30 recommended items, 27
have an interpretation at the aspect level. Regarding the balance of likes
with respect to other interactions, the proposed method in the HTL domain has
the highest ratio (0.796). This means that, in general, the aspects of hotels
are valued higher than in other domains. The method explanations in that
domain cover a greater variety of aspects, with an average of 22.34 unique
aspects per recommendation. More evidence of this is given by the high number
of aspects per item (3.44). However, despite the diversity in explanations,
the method in the HTL domain has the third worst coverage, followed by RST and
BTY. Comparing the results on the two datasets, it can be seen that in terms
of coverage, in general, AMZ surpasses YLP.

An interesting behavior to discuss is if there is a relationship between
models’ interpretability in terms of coverage and their recommendation
performance, where both seem to have a negative interdependence. For example,
although the proposed model in the RST domain has the best performance
results, it has the lowest coverage (11.11%), showing that there seems to be a
trade-off between explainability and models’ performance. A similar response
is observed for HTL and BTY domains.

##  6\. Conclusions

In this paper, we have empirically shown that latent representations of users
and items extracted from a knowledge graph by means of rating-based embeddings
can be exploited to effectively outperform the performance of graph-based and
traditional recommendation approaches. Moreover, we have shown that richer
embeddings integrating ratings and aspect-based opinions in the knowledge
graph allow surpassing a state-of-the-art neural network recommendation model.

In addition to performance improvements, an advantage of the proposed method
is the interpretabilty of its recommendations, since users, items, and aspects
are modeled in a common latent space. Differently from previous approaches
that generate explanations at rating level or according to graph paths that
require domain knowledge, our method provides aspect-level explanations
extracted from item reviews.

In the paper, we have provided an example of an explanatory visualization
dashboard that presents generated aspect-level explanations for
recommendations, showing the potential that the graph-based method can have in
terms of interpretability. A user study aimed to evaluate the potential
benefits (e.g., system transparency, and user trust and satisfaction) of the
recommendation explanations is left as future work.

Our work has shown that to improve state-of-the-art graph-based
recommendations, it is not always necessary to change the recommendation
algorithm, but preferably seek new ways of incorporating existing user
preference information like ratings and aspect-based opinions into a reference
knowledge graph. Furthermore, other information that will be relevant to add
to the recommendation are the embeddings learned from aspects since, in this
work, we only use those of users and items.

Insightful results were achieved when testing two variants of the proposed
graph-based recommendation method: one that considers only ratings (GER) and
other that only uses aspects (GEA). We observe that GEA has better
recommendation performance on domains that have a large amount of aspect
opinion interactions. In contrast, when the number of rating interactions was
relatively high, GER outperformed GEA. Interestingly, when combining rating
and aspect opinion information in a graph model (GERA), we significantly
outperformed both GEA and GER models separately, among other baselines.

As future work, a user study will be performed to demonstrate whether the
proposed dashboard that incorporates explanations on aspects is helpful for
end-users.

## References

  * (1)
  * Agarwal et al. (2007) Sumeet Agarwal, Shantanu Godbole, Diwakar Punjani, and Shourya Roy. 2007.  How much noise is too much: A study in automatic text classification. In _Seventh IEEE International Conference on Data Mining (ICDM 2007)_. IEEE, 3–12.
  * Araujo et al. (2021) Vladimir Araujo, Andrés Carvallo, Carlos Aspillaga, Camilo Thorne, and Denis Parra. 2021\.  Stress Test Evaluation of Biomedical Word Embeddings. In _Proceedings of the 20th Workshop on Biomedical Language Processing_. 119–125.
  * Aspillaga et al. (2020) Carlos Aspillaga, Andrés Carvallo, and Vladimir Araujo. 2020.  Stress Test Evaluation of Transformer-based Models in Natural Language Understanding Tasks.  _arXiv preprint arXiv:2002.06261_ (2020).
  * Bansal et al. (2016) Trapit Bansal, David Belanger, and Andrew McCallum. 2016\.  Ask the gru: Multi-task learning for deep text recommendations. In _proceedings of the 10th ACM Conference on Recommender Systems_. 107–114.
  * Bordes et al. (2013) Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013\.  Translating embeddings for modeling multi-relational data. In _Neural Information Processing Systems (NIPS)_. 1–9.
  * Cai et al. (2018) Hongyun Cai, Vincent W Zheng, and Kevin Chen-Chuan Chang. 2018\.  A comprehensive survey of graph embedding: Problems, techniques, and applications.  _IEEE Transactions on Knowledge and Data Engineering_ 30, 9 (2018), 1616–1637.
  * Carvallo et al. (2020) Andres Carvallo, Denis Parra, Hans Lobel, and Alvaro Soto. 2020\.  Automatic document screening of medical literature using word and text embeddings in an active learning setting.  _Scientometrics_ 125, 3 (2020), 3047–3084.
  * Chin et al. (2018) Jin Yao Chin, Kaiqi Zhao, Shafiq Joty, and Gao Cong. 2018\.  ANR: Aspect-based neural recommender. In _Proceedings of the 27th ACM International Conference on Information and Knowledge Management_. 147–156.
  * Chuang et al. (2020) Yu-Neng Chuang, Chih-Ming Chen, Chuan-Ju Wang, Ming-Feng Tsai, Yuan Fang, and Ee-Peng Lim. 2020\.  TPR: Text-aware Preference Ranking for Recommender Systems. In _Proceedings of the 29th ACM International Conference on Information & Knowledge Management_. 215–224.
  * de Kok et al. (2018) Sophie de Kok, Linda Punt, Rosita van den Puttelaar, Karoliina Ranta, Kim Schouten, and Flavius Frasincar. 2018.  aggregated aspect-based sentiment analysis with ontology features.  _Progress in Artificial Intelligence_ 7, 4 (2018), 295–306.
  * Fu et al. (2020) Zuohui Fu, Yikun Xian, Ruoyuan Gao, Jieyu Zhao, Qiaoying Huang, Yingqiang Ge, Shuyuan Xu, Shijie Geng, Chirag Shah, Yongfeng Zhang, et al. 2020\.  Fairness-aware explainable recommendation over knowledge graphs. In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_. 69–78.
  * Guan et al. (2019) Xinyu Guan, Zhiyong Cheng, Xiangnan He, Yongfeng Zhang, Zhibo Zhu, Qinke Peng, and Tat-Seng Chua. 2019.  Attentive aspect modeling for review-aware recommendation.  _ACM Transactions on Information Systems (TOIS)_ 37, 3 (2019), 1–27.
  * He et al. (2015) Xiangnan He, Tao Chen, Min-Yen Kan, and Xiao Chen. 2015\.  Trirank: Review-aware explainable recommendation by modeling aspects. In _Proceedings of the 24th ACM International on Conference on Information and Knowledge Management_. 1661–1670.
  * Hernández-Rubio et al. (2019) María Hernández-Rubio, Iván Cantador, and Alejandro Bellogín. 2019.  A comparative analysis of recommender systems based on item aspect opinions extracted from user reviews.  _User Modeling and User-Adapted Interaction_ 29, 2 (2019), 381–441.
  * Hu et al. (2018) Binbin Hu, Chuan Shi, Wayne Xin Zhao, and Philip S Yu. 2018\.  Leveraging meta-path based context for top-n recommendation with a neural co-attention model. In _Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 1531–1540.
  * Huang et al. (2018) Jin Huang, Wayne Xin Zhao, Hongjian Dou, Ji-Rong Wen, and Edward Y Chang. 2018.  Improving sequential recommendation with knowledge-enhanced memory networks. In _The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval_. 505–514.
  * Huang et al. (2019) Xiaowen Huang, Quan Fang, Shengsheng Qian, Jitao Sang, Yan Li, and Changsheng Xu. 2019\.  Explainable interaction-driven user modeling over knowledge graph for sequential recommendation. In _Proceedings of the 27th ACM International Conference on Multimedia_. 548–556.
  * Koren et al. (2009) Yehuda Koren, Robert Bell, and Chris Volinsky. 2009\.  Matrix factorization techniques for recommender systems.  _Computer_ 42, 8 (2009), 30–37.
  * Koren and Sill (2013) Yehuda Koren and Joseph Sill. 2013.  Collaborative filtering on ordinal user feedback. In _Twenty-third international joint conference on artificial intelligence_.
  * Lackermair et al. (2013) Georg Lackermair, Daniel Kailer, and Kenan Kanmaz. 2013\.  Importance of online product reviews from a consumer’s perspective.  _Advances in economics and business_ 1, 1 (2013), 1–5.
  * Lerer et al. (2019) Adam Lerer, Ledell Wu, Jiajun Shen, Timothee Lacroix, Luca Wehrstedt, Abhijit Bose, and Alex Peysakhovich. 2019.  Pytorch-biggraph: A large-scale graph embedding system.  _arXiv preprint arXiv:1903.12287_ (2019).
  * Li et al. (2019) Chenliang Li, Xichuan Niu, Xiangyang Luo, Zhenzhong Chen, and Cong Quan. 2019.  A review-driven neural model for sequential recommendation.  _arXiv preprint arXiv:1907.00590_ (2019).
  * Lin et al. (2015) Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. 2015.  Learning entity and relation embeddings for knowledge graph completion. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 29.
  * Liu et al. (2019) Donghua Liu, Jing Li, Bo Du, Jun Chang, and Rong Gao. 2019.  Daml: Dual attention mutual learning between ratings and reviews for item recommendation. In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 344–352.
  * Luo et al. (2014) Chen Luo, Wei Pang, Zhe Wang, and Chenghua Lin. 2014\.  Hete-cf: Social-based collaborative filtering recommendation using heterogeneous relations. In _2014 IEEE International Conference on Data Mining_. IEEE, 917–922.
  * Ma et al. (2019) Weizhi Ma, Min Zhang, Yue Cao, Woojeong Jin, Chenyang Wang, Yiqun Liu, Shaoping Ma, and Xiang Ren. 2019\.  Jointly learning explainable rules for recommendation with knowledge graph. In _The World Wide Web Conference_. 1210–1221.
  * McInnes et al. (2018) Leland McInnes, John Healy, and James Melville. 2018\.  Umap: Uniform manifold approximation and projection for dimension reduction.  _arXiv preprint arXiv:1802.03426_ (2018).
  * Munzner (2014) Tamara Munzner. 2014\.  _Visualization analysis and design_.  CRC press.
  * Musto et al. (2019a) Cataldo Musto, Pasquale Lops, Marco de Gemmis, and Giovanni Semeraro. 2019a.  Justifying recommendations through aspect-based sentiment analysis of users reviews. In _Proceedings of the 27th ACM Conference on User Modeling, Adaptation and Personalization_. 4–12.
  * Musto et al. (2019b) Cataldo Musto, Gaetano Rossiello, Marco de Gemmis, Pasquale Lops, and Giovanni Semeraro. 2019b.  Combining text summarization and aspect-based sentiment analysis of users’ reviews to justify recommendations. In _Proceedings of the 13th ACM Conference on Recommender systems_. 383–387.
  * Nickel et al. (2011) Maximilian Nickel, Volker Tresp, and Hans-Peter Kriegel. 2011\.  A three-way model for collective learning on multi-relational data. In _Icml_.
  * Ostendorff et al. (2020) Malte Ostendorff, Terry Ruas, Till Blume, Bela Gipp, and Georg Rehm. 2020.  Aspect-based Document Similarity for Research Papers.  _arXiv preprint arXiv:2010.06395_ (2020).
  * Poria et al. (2016) Soujanya Poria, Erik Cambria, and Alexander Gelbukh. 2016\.  Aspect extraction for opinion mining with a deep convolutional neural network.  _Knowledge-Based Systems_ 108 (2016), 42–49.
  * Qiu et al. (2016) Lin Qiu, Sheng Gao, Wenlong Cheng, and Jun Guo. 2016\.  Aspect-based latent factor model by integrating ratings and reviews for recommender system.  _Knowledge-Based Systems_ 110 (2016), 233–243.
  * Rendle et al. (2012) Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. 2012.  BPR: Bayesian personalized ranking from implicit feedback.  _arXiv preprint arXiv:1205.2618_ (2012).
  * Schafer et al. (2007) J Ben Schafer, Dan Frankowski, Jon Herlocker, and Shilad Sen. 2007.  Collaborative filtering recommender systems.  In _The adaptive web_. Springer, 291–324.
  * Shalom et al. (2019) Oren Sar Shalom, Guy Uziel, and Amir Kantor. 2019\.  A generative model for review-based recommendations. In _Proceedings of the 13th ACM Conference on Recommender Systems_. 353–357.
  * Trouillon et al. (2016) Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. 2016.  Complex embeddings for simple link prediction. In _International Conference on Machine Learning_. PMLR, 2071–2080.
  * Wang et al. (2018b) Hongwei Wang, Fuzheng Zhang, Jialin Wang, Miao Zhao, Wenjie Li, Xing Xie, and Minyi Guo. 2018b.  Ripplenet: Propagating user preferences on the knowledge graph for recommender systems. In _Proceedings of the 27th ACM International Conference on Information and Knowledge Management_. 417–426.
  * Wang et al. (2018c) Hongwei Wang, Fuzheng Zhang, Xing Xie, and Minyi Guo. 2018c.  DKN: Deep knowledge-aware network for news recommendation. In _Proceedings of the 2018 world wide web conference_. 1835–1844.
  * Wang et al. (2018a) Nan Wang, Hongning Wang, Yiling Jia, and Yue Yin. 2018a.  Explainable recommendation via multi-task learning in opinionated text data. In _The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval_. 165–174.
  * Wang et al. (2019a) Xiang Wang, Xiangnan He, Yixin Cao, Meng Liu, and Tat-Seng Chua. 2019a.  Kgat: Knowledge graph attention network for recommendation. In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 950–958.
  * Wang et al. (2021) Xi Wang, Iadh Ounis, and Craig Macdonald. 2021.  Leveraging Review Properties for Effective Recommendation.  (2021).
  * Wang et al. (2019b) Xiang Wang, Dingxian Wang, Canran Xu, Xiangnan He, Yixin Cao, and Tat-Seng Chua. 2019b.  Explainable reasoning over knowledge graphs for recommendation. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 33. 5329–5336.
  * Wu and Ester (2015) Yao Wu and Martin Ester. 2015.  Flame: A probabilistic model combining aspect based opinion mining and collaborative filtering. In _Proceedings of the Eighth ACM International Conference on Web Search and Data Mining_. 199–208.
  * Xian et al. (2019) Yikun Xian, Zuohui Fu, S Muthukrishnan, Gerard De Melo, and Yongfeng Zhang. 2019.  Reinforcement knowledge graph reasoning for explainable recommendation. In _Proceedings of the 42nd international ACM SIGIR conference on research and development in information retrieval_. 285–294.
  * Yang et al. (2014) Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. 2014.  Embedding entities and relations for learning and inference in knowledge bases.  _arXiv preprint arXiv:1412.6575_ (2014).
  * Zhang et al. (2014) Yongfeng Zhang, Guokun Lai, Min Zhang, Yi Zhang, Yiqun Liu, and Shaoping Ma. 2014\.  Explicit factor models for explainable recommendation based on phrase-level sentiment analysis. In _Proceedings of the 37th international ACM SIGIR conference on Research & development in information retrieval_. 83–92.


Figure 25: Full set of amplitudes for $\vec{p}_{f}=(2,1,0)$ and
$\vec{p}_{i}=(0,1,2)$. See a discussion of the features of the plot in the
text. The first row presents the real parts of $\mathcal{A}_{1}$ (left),
$\mathcal{A}_{4}$ (middle), and $\mathcal{A}_{6}$ (right), and the second row
their imaginary parts. The third row presents the real parts of
$\mathcal{A}_{5}$ (left), $\mathcal{A}_{7}$ (middle), and
$\mathcal{A}_{2,3,8}$ at once (right), and the fourth row their imaginary
parts. In all cases but the grouped plot of $\mathcal{A}_{2,3,8}$, we show
three extractions of the amplitudes: using a cut of $t_{s}=3$ (blue, see
section 4) and an $l^{2}$ minimization (or SVD, see section 3.3), using a cut
of $t_{s}=4$ and the $l^{2}$ minimization (green), or using a cut of 3 and an
$l^{1}$ minimization (orange). In the last plot, we only use a cut of 3 with
the $l^{2}$ minimization for clarity.

#### Figure 25

  * •

For the five amplitudes that enter the construction of light-cone GPDs, we
compare a solution for the extraction of the amplitudes from the matrix
elements using either the SVD ($l^{2}$ minimization) or an $l^{1}$
minimization. We find that the $l^{1}$ minimization produces generally larger
uncertainties but with central values in excellent agreement with the $l^{2}$
extraction. We consider this as a validation of the robustness of the SVD
method.

  * •

For those five amplitudes, we also show the comparison between the SVD on the
data with a cut in lattice time separation of 3 and 4. The larger cut produces
more uncertainty, but also in a good consistency with the smaller cut. The
results of this paper are jointly extracted using the cut in 3 and 4 to give
an account of excited state contribution.

  * •

$\mathcal{A}_{5}$ and $\mathcal{A}_{7}$ do not show a strong deviation from 0,
a situation generally observed in the dataset. $\mathcal{A}_{5}$ contains
signal of the $D$-term, the kinematic higher-twist contamination $Y$ discussed
in Radyushkin (2024), and lattice systematic errors. As for $\mathcal{A}_{7}$,
it enters the construction of the light-cone GPD $E$, but is generally small
in magnitude and barely different from 0.

  * •

The kinematic factors of the amplitudes $\mathcal{A}_{2,3,6,7,8}$ cancel in
the limit $z=0$. Therefore, we cannot evaluate the amplitudes in this limit.
$\mathcal{A}_{6}$ grows noticeably at small $z$. It is in fact necessary to
counter-act the discontinuity of $\mathcal{A}_{4}$ between the value computed
at $z=0$ (the form factor $F_{2}$), and the value at $z=1$. Adding
$\nu\mathcal{A}_{6}$ to $\mathcal{A}_{4}$ to form the light-cone GPD $E$
alleviates the discontinuity, as we demonstrate in Figure 26.

Figure 26: Effect of adding $\nu\mathcal{A}_{6}$ to $\mathcal{A}_{4}$ for
$\vec{p}_{f}=(2,1,0)$ and $\vec{p}_{i}=(0,1,2)$.  Figure 27: Full set of
amplitudes for $\vec{p}_{f}=(1,-1,0)$ and $\vec{p}_{i}=(-1,1,0)$.

#### Figure 27

The choice of momentum pair $\vec{p}_{f}=(1,-1,0)$ and $\vec{p}_{i}=(-1,1,0)$
means that $\nu$ and $\nu\xi$ are both 0 for every value of $z$. Therefore, we
should have $\mathcal{A}_{1}=F_{1}+\mathcal{O}(z^{2})$ and
$\mathcal{A}_{4}=F_{2}+\mathcal{O}(z^{2})$. $\mathcal{A}_{1}$ presents a
distinctive evolution with $z$, which we discuss in section 5.
$\mathcal{A}_{4}$ on the other hand is compatible with a constant within its
uncertainty. The general features identified in the previous figure remain
valid.

##  Appendix B To $\gamma_{3}$ or not to $\gamma_{3}$

In the lattice calculation of PDFs, the choice of matrix elements with Dirac
structure $\gamma^{3}$ has been considered disfavored. In the Lorentz
decomposition, $M^{\mu}=p^{\mu}\mathcal{M}+z^{\mu}\mathcal{N}$, choosing $\mu$
parallel to the Wilson line will introduce a contribution from $\mathcal{N}$
which does not appear in the light cone definition of the PDF Radyushkin
(2017). In the work of Constantinou and Panagopoulos (2017), the matrix
element of the operator $\bar{\psi}(z)\Gamma W(z;0)\psi(0)$ is calculated at
1-loop lattice perturbation theory. The matrix elements contain terms
proportional to $\bar{u}\Gamma u$ and $\bar{u}\\{\Gamma,\not{z}\\}u$,
describing a finite mixing with operators which occurs due to the breaking of
chiral symmetry. As discussed in this appendix for the case of the vector
structures, this pattern can be consolidated with another contribution we
typically neglect or discard. For the vector current, the mixing comes from
$\bar{u}\\{\gamma^{\mu},\not{z}\\}u=-2z^{\mu}\bar{u}u$ which can only impact
the $\mathcal{N}(\nu,z^{2})$ or $A_{2}(\nu,t,z^{2})$ for PDFs and GPDs
respectively. Since these do not contain the leading order contributions we
are interested in, this contamination is benign. Similarly for axial
$\Gamma=\gamma^{\mu}\gamma^{5}$ structure in the case of the pion DA Kovner
_et al._ (2024), the mixing tensor structure can only generate effects of
$O(z^{2})$ which will not ultimately effect the asymptotic short distance
limit of the relevant amplitude. For the vector case, the inclusion of
$\gamma_{3}$ data has a negligible effect on the amplitudes $A_{1,4,5}$ when
$z/a>0$.

First, for $z=0$, the first $\mathcal{A}_{1,4}$ amplitudes correspond to the
form factors $F_{1,2}$ while $\mathcal{A}_{5}$ should be 0 in the continuum by
the Ward Identity $q_{\mu}M^{\mu}=0$. The rest of the amplitudes have a factor
of $z$ in the kinematic factor and do not belong here. Using the SVD procedure
in section 3.2, $\mathcal{A}_{1,4,5}$ were extracted while including and while
neglecting the $M^{3}$ matrix element, shown in Fig. 28. The amplitudes remain
quite similar with a few percent deviations in $A_{1,4}$, relevant at our
statistical precision while the quality of $\mathcal{A}_{5}=0$ changes
slightly. After binning of the $t$-dependence (see section 5), the elastic
form factors are indiscernible whether $M^{z}$ is used or not. Remembering the
duplicity relating half helicity combinations to the other half, for $z=0$,
there were either 8 or 6 independent matrix elements used to obtain 3 unknown
functions. Adding in more data without increasing the number of unknowns means
there are new constraints being applied when $M^{3}$ is added.

Figure 28: (Left) The amplitudes $\mathcal{A}_{1,4,5}$ at $z=0$ calculated by
using data with and without $\gamma_{3}$. The sans $\gamma_{3}$ data (circles)
are shifted to the right slightly from the with $\gamma_{3}$ data (crosses).
(Right) The difference of the amplitudes calculated with and without
$\gamma_{3}$. The difference of $\mathcal{A}_{5}$ which has significant
changes is not shown.

While studying the real component of $z=a$ data, which shows the same patterns
as larger $z$ and with the imaginary component, if the $M^{3}$ data is
neglected then the $\mathcal{A}_{2,8}$ terms are dropped since they always
contribute 0 to $M^{t,x,y}$. The two sets of results are remarkably identical,
shown in Fig. 29. In fact, as we have already mentioned in section 3.3, at
$z\neq 0$, the $M^{3}$ data brings exactly two constraints for two new unknown
amplitudes, so it exactly serves to determine $\mathcal{A}_{2,8}$ without any
incidence on the other amplitudes. The only deviations on the other amplitudes
come from numerical noise, at a level of $10^{-6}$ corresponding to the
accuracy at which we computed the kinematic matrix.

Figure 29: Same as Fig. 28 but for amplitudes $\mathcal{A}_{1,3,4,5,6,7}$ at
$z=1$.

##  Appendix C Effect of the range of separations $z$

We compare in Fig. 30 the extraction of the moment $A_{3,0}$ in one bin in $t$
depending on the range in $z$ one chooses to consider. The standard analysis
in the main body of the article is performed with $z\in[a,6a]$. We observe
that varying this range gives statistically similar results, not least because
of the very high correlation between the values in $z$. The very inaccurate
result at small $z$ stems from the fact that $A_{3,0}$ is a subdominant moment
in the real part of the Ioffe-time distribution, which can only be captured
accurately when the Ioffe time is large enough.

Figure 30: The red points are the matched $A_{3,0}(t=-0.33\textrm{
GeV}^{2},\mu=2$ GeV) depending on the non-local separation $z$ for a cut
$t_{s}=4$. The bands represent the correlated fit by a constant of a subset of
the red points. It appears that the correlated fit result is very stable
whereas an uncorrelated fit would show a significant variation. We observed a
similar robustness of the correlated constant fits for the non-local
extraction of the EFF.

Figure 31: $A_{2,0}(t,\mu=2$ GeV) (left) and $B_{2,0}(t,\mu=2$ GeV) (right)
extracted from the lattice cut of 3 by fitting the matched values for
$z\in[a,6a]$ (red) or $z\in[a,4a]$.

Figure 32: $A_{3,0}(t,\mu=2$ GeV) (left) and $B_{3,0}(t,\mu=2$ GeV) (right)
extracted from the lattice cut of 3 by fitting the matched values for
$z\in[a,6a]$ (red) or $z\in[a,4a]$.

In Figs. 31 to 36, we show systematically the extraction of the moments using
$z\in[a,6a]$ and $z\in[a,4a]$. It appears that there is no significant
statistical difference between using the two methods, and no clear sign of
higher twist contributions within the precision of this study.

Figure 33: $A_{4,0}(t,\mu=2$ GeV) (left) and $B_{4,0}(t,\mu=2$ GeV) (right)
extracted from the lattice cut of 3 by fitting the matched values for
$z\in[a,6a]$ (red) or $z\in[a,4a]$.

Figure 34: $A_{3,2}(t,\mu=2$ GeV) (left) and $B_{3,2}(t,\mu=2$ GeV) (right)
extracted from the lattice cut of 3 by fitting the matched values for
$z\in[a,6a]$ (red) or $z\in[a,4a]$.

Figure 35: $A_{4,2}(t,\mu=2$ GeV) (left) and $B_{4,2}(t,\mu=2$ GeV) (right)
extracted from the lattice cut of 3 by fitting the matched values for
$z\in[a,6a]$ (red) or $z\in[a,4a]$. Figure 36: $C_{2}(t,\mu=2$ GeV) extracted
from the lattice cut of 3 by fitting the matched values for $z\in[a,6a]$ (red)
or $z\in[a,4a]$.

## References

  * Müller  _et al._ (1994) D. Müller, D. Robaschik, B. Geyer, F. M. Dittes, and J. Hořejši, Fortsch. Phys. 42, 101 (1994), arXiv:hep-ph/9812448 .
  * Ji (1997a) X.-D. Ji, Phys. Rev. D 55, 7114 (1997a), arXiv:hep-ph/9609381 .
  * Radyushkin (1996a) A. V. Radyushkin, Phys. Lett. B 380, 417 (1996a), arXiv:hep-ph/9604317 .
  * Radyushkin (1997) A. V. Radyushkin, Phys. Rev. D 56, 5524 (1997), arXiv:hep-ph/9704207 .
  * Ji (1997b) X.-D. Ji, Phys. Rev. Lett. 78, 610 (1997b), arXiv:hep-ph/9603249 .
  * Polyakov (2003) M. V. Polyakov, Phys. Lett. B 555, 57 (2003), arXiv:hep-ph/0210165 .
  * Polyakov and Schweitzer (2018) M. V. Polyakov and P. Schweitzer, Int. J. Mod. Phys. A 33, 1830025 (2018), arXiv:1805.06596 [hep-ph] .
  * Radyushkin (1996b) A. V. Radyushkin, Phys. Lett. B 385, 333 (1996b), arXiv:hep-ph/9605431 .
  * Collins  _et al._ (1997) J. C. Collins, L. Frankfurt, and M. Strikman, Phys. Rev. D 56, 2982 (1997), arXiv:hep-ph/9611433 .
  * Dudek  _et al._ (2012) J. Dudek _et al._ , Eur. Phys. J. A 48, 187 (2012), arXiv:1208.1244 [hep-ex] .
  * Accardi  _et al._ (2016) A. Accardi _et al._ , Eur. Phys. J. A 52, 268 (2016), arXiv:1212.1701 [nucl-ex] .
  * Abdul Khalek  _et al._ (2022) R. Abdul Khalek _et al._ , Nucl. Phys. A 1026, 122447 (2022), arXiv:2103.05419 [physics.ins-det] .
  * Qiu and Yu (2023a) J.-W. Qiu and Z. Yu, Phys. Rev. D 107, 014007 (2023a), arXiv:2210.07995 [hep-ph] .
  * Qiu and Yu (2023b) J.-W. Qiu and Z. Yu, Phys. Rev. Lett. 131, 161902 (2023b), arXiv:2305.15397 [hep-ph] .
  * Bertone  _et al._ (2021) V. Bertone, H. Dutrieux, C. Mezrag, H. Moutarde, and P. Sznajder, Phys. Rev. D 103, 114019 (2021), arXiv:2104.03836 [hep-ph] .
  * Freund (2000) A. Freund, Phys. Lett. B 472, 412 (2000), arXiv:hep-ph/9903488 .
  * Dutrieux  _et al._ (2023) H. Dutrieux, M. Winn, and V. Bertone, Phys. Rev. D 107, 114019 (2023), arXiv:2302.07861 [hep-ph] .
  * Moffat  _et al._ (2023) E. Moffat, A. Freese, I. Cloët, T. Donohoe, L. Gamberg, W. Melnitchouk, A. Metz, A. Prokudin, and N. Sato, Phys. Rev. D 108, 036027 (2023), arXiv:2303.12006 [hep-ph] .
  * Radyushkin (1999) A. V. Radyushkin, Phys. Rev. D 59, 014030 (1999), arXiv:hep-ph/9805342 .
  * Pire  _et al._ (1999) B. Pire, J. Soffer, and O. Teryaev, Eur. Phys. J. C 8, 103 (1999), arXiv:hep-ph/9804284 .
  * Diehl  _et al._ (2001) M. Diehl, T. Feldmann, R. Jakob, and P. Kroll, Nucl. Phys. B 596, 33 (2001), [Erratum: Nucl.Phys.B 605, 647–647 (2001)], arXiv:hep-ph/0009255 .
  * Pobylitsa (2002) P. V. Pobylitsa, Phys. Rev. D 65, 114015 (2002), arXiv:hep-ph/0201030 .
  * Dutrieux  _et al._ (2022) H. Dutrieux, H. Dutrieux, O. Grocholski, O. Grocholski, H. Moutarde, H. Moutarde, P. Sznajder, and P. Sznajder, Eur. Phys. J. C 82, 252 (2022), [Erratum: Eur.Phys.J.C 82, 389 (2022)], arXiv:2112.10528 [hep-ph] .
  * Burkert  _et al._ (2018) V. D. Burkert, L. Elouadrhiri, and F. X. Girod, Nature 557, 396 (2018).
  * Kumerički (2019) K. Kumerički, Nature 570, E1 (2019).
  * Dutrieux  _et al._ (2021) H. Dutrieux, C. Lorcé, H. Moutarde, P. Sznajder, A. Trawiński, and J. Wagner, Eur. Phys. J. C 81, 300 (2021), arXiv:2101.03855 [hep-ph] .
  * Hagler  _et al._ (2003) P. Hagler, J. W. Negele, D. B. Renner, W. Schroers, T. Lippert, and K. Schilling (LHPC, SESAM), Phys. Rev. D 68, 034505 (2003), arXiv:hep-lat/0304018 .
  * Hagler  _et al._ (2008) P. Hagler _et al._ (LHPC), Phys. Rev. D 77, 094502 (2008), arXiv:0705.4295 [hep-lat] .
  * Bali  _et al._ (2019) G. S. Bali, S. Collins, M. Göckeler, R. Rödl, A. Schäfer, and A. Sternbeck, Phys. Rev. D 100, 014507 (2019), arXiv:1812.08256 [hep-lat] .
  * Alexandrou  _et al._ (2020a) C. Alexandrou _et al._ , Phys. Rev. D 101, 034519 (2020a), arXiv:1908.10706 [hep-lat] .
  * Hackett  _et al._ (2024) D. C. Hackett, D. A. Pefkou, and P. E. Shanahan, Phys. Rev. Lett. 132, 251904 (2024), arXiv:2310.08484 [hep-lat] .
  * Guidal and Vanderhaeghen (2003) M. Guidal and M. Vanderhaeghen, Phys. Rev. Lett. 90, 012001 (2003), arXiv:hep-ph/0208275 .
  * Belitsky and Mueller (2003) A. V. Belitsky and D. Mueller, Phys. Rev. Lett. 90, 022001 (2003), arXiv:hep-ph/0210313 .
  * Deja  _et al._ (2023) K. Deja, V. Martinez-Fernandez, B. Pire, P. Sznajder, and J. Wagner, Phys. Rev. D 107, 094035 (2023), arXiv:2303.13668 [hep-ph] .
  * Pedrak  _et al._ (2020) A. Pedrak, B. Pire, L. Szymanowski, and J. Wagner, Phys. Rev. D 101, 114027 (2020), arXiv:2003.03263 [hep-ph] .
  * Grocholski  _et al._ (2022) O. Grocholski, B. Pire, P. Sznajder, L. Szymanowski, and J. Wagner, Phys. Rev. D 105, 094025 (2022), arXiv:2204.00396 [hep-ph] .
  * Boussarie  _et al._ (2017) R. Boussarie, B. Pire, L. Szymanowski, and S. Wallon, JHEP 02, 054 (2017), [Erratum: JHEP 10, 029 (2018)], arXiv:1609.03830 [hep-ph] .
  * Duplančić  _et al._ (2018) G. Duplančić, K. Passek-Kumerički, B. Pire, L. Szymanowski, and S. Wallon, JHEP 11, 179 (2018), arXiv:1809.08104 [hep-ph] .
  * Qiu and Yu (2022) J.-W. Qiu and Z. Yu, JHEP 08, 103 (2022), arXiv:2205.07846 [hep-ph] .
  * Guo  _et al._ (2022) Y. Guo, X. Ji, and K. Shiells, JHEP 09, 215 (2022), arXiv:2207.05768 [hep-ph] .
  * Guo  _et al._ (2023) Y. Guo, X. Ji, M. G. Santiago, K. Shiells, and J. Yang, JHEP 05, 150 (2023), arXiv:2302.07279 [hep-ph] .
  * Riberdy  _et al._ (2024) M. J. Riberdy, H. Dutrieux, C. Mezrag, and P. Sznajder, Eur. Phys. J. C 84, 201 (2024), arXiv:2306.01647 [hep-ph] .
  * Liu and Dong (1994) K.-F. Liu and S.-J. Dong, Phys. Rev. Lett. 72, 1790 (1994), arXiv:hep-ph/9306299 .
  * Detmold and Lin (2006) W. Detmold and C. J. D. Lin, Phys. Rev. D 73, 014501 (2006), arXiv:hep-lat/0507007 .
  * Braun and Müller (2008) V. Braun and D. Müller, Eur. Phys. J. C 55, 349 (2008), arXiv:0709.1348 [hep-ph] .
  * Ji (2013) X. Ji, Phys. Rev. Lett. 110, 262002 (2013), arXiv:1305.1539 [hep-ph] .
  * Ji (2014) X. Ji, Sci. China Phys. Mech. Astron. 57, 1407 (2014), arXiv:1404.6680 [hep-ph] .
  * Radyushkin (2017) A. V. Radyushkin, Phys. Rev. D 96, 034025 (2017), arXiv:1705.01488 [hep-ph] .
  * Chambers  _et al._ (2017) A. J. Chambers, R. Horsley, Y. Nakamura, H. Perlt, P. E. L. Rakow, G. Schierholz, A. Schiller, K. Somfleth, R. D. Young, and J. M. Zanotti, Phys. Rev. Lett. 118, 242001 (2017), arXiv:1703.01153 [hep-lat] .
  * Ma and Qiu (2018) Y.-Q. Ma and J.-W. Qiu, Phys. Rev. Lett. 120, 022003 (2018), arXiv:1709.03018 [hep-ph] .
  * Bali  _et al._ (2018) G. S. Bali, P. C. Bruns, L. Castagnini, M. Diehl, J. R. Gaunt, B. Gläßle, A. Schäfer, A. Sternbeck, and C. Zimmermann, JHEP 12, 061 (2018), arXiv:1807.03073 [hep-lat] .
  * Ji  _et al._ (2015) X. Ji, A. Schäfer, X. Xiong, and J.-H. Zhang, Phys. Rev. D 92, 014039 (2015), arXiv:1506.00248 [hep-ph] .
  * Liu  _et al._ (2019) Y.-S. Liu, W. Wang, J. Xu, Q.-A. Zhang, J.-H. Zhang, S. Zhao, and Y. Zhao, Phys. Rev. D 100, 034006 (2019), arXiv:1902.00307 [hep-ph] .
  * Radyushkin (2019) A. V. Radyushkin, Phys. Rev. D 100, 116011 (2019), arXiv:1909.08474 [hep-ph] .
  * Ma  _et al._ (2022) J. P. Ma, Z. Y. Pang, and G. P. Zhang, JHEP 08, 130 (2022), arXiv:2202.07116 [hep-ph] .
  * Chen  _et al._ (2020) J.-W. Chen, H.-W. Lin, and J.-H. Zhang, Nucl. Phys. B 952, 114940 (2020), arXiv:1904.12376 [hep-lat] .
  * Lin (2021) H.-W. Lin, Phys. Rev. Lett. 127, 182001 (2021), arXiv:2008.12474 [hep-ph] .
  * Alexandrou  _et al._ (2020b) C. Alexandrou, K. Cichy, M. Constantinou, K. Hadjiyiannakou, K. Jansen, A. Scapellato, and F. Steffens, Phys. Rev. Lett. 125, 262001 (2020b), arXiv:2008.10573 [hep-lat] .
  * Lin (2022) H.-W. Lin, Phys. Lett. B 824, 136821 (2022), arXiv:2112.07519 [hep-lat] .
  * Alexandrou  _et al._ (2022) C. Alexandrou, K. Cichy, M. Constantinou, K. Hadjiyiannakou, K. Jansen, A. Scapellato, and F. Steffens, Phys. Rev. D 105, 034501 (2022), arXiv:2108.10789 [hep-lat] .
  * Bhattacharya  _et al._ (2022) S. Bhattacharya, K. Cichy, M. Constantinou, J. Dodson, X. Gao, A. Metz, S. Mukherjee, A. Scapellato, F. Steffens, and Y. Zhao, Phys. Rev. D 106, 114512 (2022), arXiv:2209.05373 [hep-lat] .
  * Bhattacharya  _et al._ (2023a) S. Bhattacharya, K. Cichy, M. Constantinou, X. Gao, A. Metz, J. Miller, S. Mukherjee, P. Petreczky, F. Steffens, and Y. Zhao, Phys. Rev. D 108, 014507 (2023a), arXiv:2305.11117 [hep-lat] .
  * Bhattacharya  _et al._ (2024a) S. Bhattacharya _et al._ , Phys. Rev. D 109, 034508 (2024a), arXiv:2310.13114 [hep-lat] .
  * Bhattacharya  _et al._ (2023b) S. Bhattacharya, K. Cichy, M. Constantinou, J. Dodson, A. Metz, A. Scapellato, and F. Steffens, Phys. Rev. D 108, 054501 (2023b), arXiv:2306.05533 [hep-lat] .
  * Lin (2023) H.-W. Lin, Phys. Lett. B 846, 138181 (2023), arXiv:2310.10579 [hep-lat] .
  * Bhattacharya  _et al._ (2024b) S. Bhattacharya, K. Cichy, M. Constantinou, A. Metz, N. Nurminen, and F. Steffens, (2024b), arXiv:2405.04414 [hep-lat] .
  * Hannaford-Gunn  _et al._ (2024) A. Hannaford-Gunn, K. U. Can, J. A. Crawford, R. Horsley, P. E. L. Rakow, G. Schierholz, H. Stüben, R. D. Young, and J. M. Zanotti, (2024), arXiv:2405.06256 [hep-lat] .
  * Egerer  _et al._ (2021) C. Egerer, R. G. Edwards, C. Kallidonis, K. Orginos, A. V. Radyushkin, D. G. Richards, E. Romero, and S. Zafeiropoulos (HadStruc), JHEP 11, 148 (2021), arXiv:2107.05199 [hep-lat] .
  * Ji (1998) X.-D. Ji, J. Phys. G 24, 1181 (1998), arXiv:hep-ph/9807358 .
  * Ishikawa  _et al._ (2017) T. Ishikawa, Y.-Q. Ma, J.-W. Qiu, and S. Yoshida, Phys. Rev. D 96, 094019 (2017), arXiv:1707.03107 [hep-ph] .
  * Radyushkin (2024) A. V. Radyushkin, Phys. Rev. D 109, 014514 (2024), arXiv:2311.06007 [hep-ph] .
  * Edwards  _et al._ (2023) R. G. Edwards _et al._ (HadStruc), JHEP 03, 086 (2023), arXiv:2211.04434 [hep-lat] .
  * Balitsky  _et al._ (2022) I. Balitsky, W. Morris, and A. Radyushkin, JHEP 02, 193 (2022), arXiv:2112.02011 [hep-ph] .
  * Egerer  _et al._ (2022a) C. Egerer _et al._ (HadStruc), Phys. Rev. D 106, 094511 (2022a), arXiv:2207.08733 [hep-lat] .
  * Braun (2023) V. M. Braun, JHEP 10, 134 (2023), arXiv:2308.04270 [hep-ph] .
  * Anikin  _et al._ (2000) I. V. Anikin, B. Pire, and O. V. Teryaev, Phys. Rev. D 62, 071501 (2000), arXiv:hep-ph/0003203 .
  * Penttinen  _et al._ (2000) M. Penttinen, M. V. Polyakov, A. G. Shuvaev, and M. Strikman, Phys. Lett. B 491, 96 (2000), arXiv:hep-ph/0006321 .
  * Belitsky and Mueller (2000) A. V. Belitsky and D. Mueller, Nucl. Phys. B 589, 611 (2000), arXiv:hep-ph/0007031 .
  * Radyushkin and Weiss (2000) A. V. Radyushkin and C. Weiss, Phys. Lett. B 493, 332 (2000), arXiv:hep-ph/0008214 .
  * Radyushkin and Weiss (2001) A. V. Radyushkin and C. Weiss, Phys. Rev. D 63, 114012 (2001), arXiv:hep-ph/0010296 .
  * Kivel and Polyakov (2001) N. Kivel and M. V. Polyakov, Nucl. Phys. B 600, 334 (2001), arXiv:hep-ph/0010150 .
  * Braun  _et al._ (2024) V. M. Braun, M. Koller, and J. Schoenleber, Phys. Rev. D 109, 074510 (2024), arXiv:2401.08012 [hep-ph] .
  * Radyushkin (2018) A. Radyushkin, Phys. Rev. D98, 014019 (2018), arXiv:1801.02427 [hep-ph] .
  * Zhang  _et al._ (2018) J.-H. Zhang, J.-W. Chen, and C. Monahan, Phys. Rev. D97, 074508 (2018), arXiv:1801.03023 [hep-ph] .
  * Izubuchi  _et al._ (2018) T. Izubuchi, X. Ji, L. Jin, I. W. Stewart, and Y. Zhao, Phys. Rev. D98, 056004 (2018), arXiv:1801.03917 [hep-ph] .
  * Dall’Olio  _et al._ (2024) P. Dall’Olio, F. De Soto, C. Mezrag, J. M. Morgado Chávez, H. Moutarde, J. Rodríguez-Quintero, P. Sznajder, and J. Segovia, Phys. Rev. D 109, 096013 (2024), arXiv:2401.12013 [hep-ph] .
  * Bertone  _et al._ (2022) V. Bertone, H. Dutrieux, C. Mezrag, J. M. Morgado, and H. Moutarde, Eur. Phys. J. C 82, 888 (2022), arXiv:2206.01412 [hep-ph] .
  * Joó  _et al._ (2019a) B. Joó, J. Karpie, K. Orginos, A. Radyushkin, D. Richards, and S. Zafeiropoulos, JHEP 12, 081 (2019a), arXiv:1908.09771 [hep-lat] .
  * Sufian  _et al._ (2020) R. S. Sufian, C. Egerer, J. Karpie, R. G. Edwards, B. Joó, Y.-Q. Ma, K. Orginos, J.-W. Qiu, and D. G. Richards, Phys. Rev. D 102, 054508 (2020), arXiv:2001.04960 [hep-lat] .
  * Dutrieux  _et al._ (2024) H. Dutrieux, J. Karpie, C. Monahan, K. Orginos, and S. Zafeiropoulos (HadStruc), JHEP 04, 061 (2024), arXiv:2310.19926 [hep-lat] .
  * Gao  _et al._ (2021) X. Gao, K. Lee, S. Mukherjee, C. Shugert, and Y. Zhao, Phys. Rev. D 103, 094504 (2021), arXiv:2102.01101 [hep-ph] .
  * Ohrndorf (1982) T. Ohrndorf, Nucl. Phys. B 198, 26 (1982).
  * Chase (1980) M. Chase, Nuclear Physics B 174, 109 (1980).
  * Orginos  _et al._ (2017) K. Orginos, A. Radyushkin, J. Karpie, and S. Zafeiropoulos, Phys. Rev. D 96, 094503 (2017), arXiv:1706.05373 [hep-ph] .
  * Karpie  _et al._ (2018) J. Karpie, K. Orginos, and S. Zafeiropoulos, JHEP 11, 178 (2018), arXiv:1807.10933 [hep-lat] .
  * Joó  _et al._ (2019b) B. Joó, J. Karpie, K. Orginos, A. V. Radyushkin, D. G. Richards, R. S. Sufian, and S. Zafeiropoulos, Phys. Rev. D 100, 114512 (2019b), arXiv:1909.08517 [hep-lat] .
  * Joó  _et al._ (2020) B. Joó, J. Karpie, K. Orginos, A. V. Radyushkin, D. G. Richards, and S. Zafeiropoulos, Phys. Rev. Lett. 125, 232003 (2020), arXiv:2004.01687 [hep-lat] .
  * Egerer  _et al._ (2022b) C. Egerer _et al._ (HadStruc), Phys. Rev. D 105, 034507 (2022b), arXiv:2111.01808 [hep-lat] .
  * Khan  _et al._ (2021) T. Khan _et al._ (HadStruc), Phys. Rev. D 104, 094516 (2021), arXiv:2107.08960 [hep-lat] .
  * Peardon  _et al._ (2009) M. Peardon, J. Bulava, J. Foley, C. Morningstar, J. Dudek, R. G. Edwards, B. Joo, H.-W. Lin, D. G. Richards, and K. J. Juge (Hadron Spectrum), Phys. Rev. D 80, 054506 (2009), arXiv:0905.2160 [hep-lat] .
  * Edwards  _et al._ (2016) R. Edwards, B. Joó, K. Orginos, D. Richards, and F. Winter, “U.S. 2+1 flavor clover lattice generation program,” (2016), unpublished.
  * Yoon  _et al._ (2016) B. Yoon _et al._ , Phys. Rev. D 93, 114506 (2016), arXiv:1602.07737 [hep-lat] .
  * Yoon  _et al._ (2017) B. Yoon _et al._ , Phys. Rev. D 95, 074508 (2017), arXiv:1611.07452 [hep-lat] .
  * Bernard  _et al._ (1986) C. Bernard, T. Draper, G. Hockney, and A. Soni, “Calculation of weak matrix elements: Some technical aspects,” in  _Lattice Gauge Theory: A Challenge in Large-Scale Computing_, edited by B. Bunk, K. H. Mütter, and K. Schilling (Springer US, Boston, MA, 1986) pp. 199–207.
  * Kilcup  _et al._ (1985) G. W. Kilcup, S. R. Sharpe, R. Gupta, G. Guralnik, A. Patel, and T. Warnock, Phys. Lett. B 164, 347 (1985).
  * Moore and Fleming (2006) D. C. Moore and G. T. Fleming, Phys. Rev. D 73, 014504 (2006), [Erratum: Phys.Rev.D 74, 079905 (2006)], arXiv:hep-lat/0507018 .
  * Edwards  _et al._ (2011) R. G. Edwards, J. J. Dudek, D. G. Richards, and S. J. Wallace, Phys. Rev. D 84, 074508 (2011), arXiv:1104.5152 [hep-ph] .
  * Dudek and Edwards (2012) J. J. Dudek and R. G. Edwards, Phys. Rev. D 85, 054016 (2012), arXiv:1201.2349 [hep-ph] .
  * Thomas  _et al._ (2012) C. E. Thomas, R. G. Edwards, and J. J. Dudek, Phys. Rev. D 85, 014507 (2012), arXiv:1107.1930 [hep-lat] .
  * Egerer  _et al._ (2019) C. Egerer, D. Richards, and F. Winter, Phys. Rev. D 99, 034506 (2019), arXiv:1810.09991 [hep-lat] .
  * Penrose (1956) R. Penrose, Mathematical Proceedings of the Cambridge Philosophical Society 52, 17–19 (1956).
  * Constantinou and Panagopoulos (2017) M. Constantinou and H. Panagopoulos, Phys. Rev. D 96, 054506 (2017), arXiv:1705.11193 [hep-lat] .
  * LAD (2008) “Least absolute deviation regression,” in  _The Concise Encyclopedia of Statistics_ (Springer New York, New York, NY, 2008) pp. 299–302.
  * Maiani  _et al._ (1987) L. Maiani, G. Martinelli, M. L. Paciello, and B. Taglienti, Nucl. Phys. B 293, 420 (1987).
  * Capitani  _et al._ (2012) S. Capitani, M. Della Morte, G. von Hippel, B. Jager, A. Juttner, B. Knippschild, H. B. Meyer, and H. Wittig, Phys. Rev. D 86, 074502 (2012), arXiv:1205.0180 [hep-lat] .
  * Park  _et al._ (2022) S. Park, R. Gupta, B. Yoon, S. Mondal, T. Bhattacharya, Y.-C. Jang, B. Joó, and F. Winter (Nucleon Matrix Elements (NME)), Phys. Rev. D 105, 054505 (2022), arXiv:2103.05599 [hep-lat] .
  * Lee  _et al._ (2015) G. Lee, J. R. Arrington, and R. J. Hill, Phys. Rev. D 92, 013013 (2015), arXiv:1505.01489 [hep-ph] .
  * Burkardt (2003) M. Burkardt, Int. J. Mod. Phys. A 18, 173 (2003), arXiv:hep-ph/0207047 .
  * Yuan (2004) F. Yuan, Phys. Rev. D 69, 051501 (2004), arXiv:hep-ph/0311288 .
  * Goeke  _et al._ (2001) K. Goeke, M. V. Polyakov, and M. Vanderhaeghen, Prog. Part. Nucl. Phys. 47, 401 (2001), arXiv:hep-ph/0106012 .
  * Towns  _et al._ (2014) J. Towns, T. Cockerill, M. Dahan, I. Foster, K. Gaither, A. Grimshaw, V. Hazlewood, S. Lathrop, D. Lifka, G. D. Peterson, R. Roskies, J. Scott, and N. Wilkins-Diehr, Computing in Science & Engineering 16, 62 (2014).
  * Nystrom  _et al._ (2015) N. A. Nystrom, M. J. Levine, R. Z. Roskies, and J. R. Scott,  XSEDE ’15, 30:1 (2015).
  * Edwards and Joo (2005) R. G. Edwards and B. Joo, Nucl. Phys. Proc. Suppl. 140, 832 (2005), arXiv:hep-lat/0409003 [hep-lat] .
  * Clark  _et al._ (2010) M. A. Clark, R. Babich, K. Barros, R. C. Brower, and C. Rebbi, Comput. Phys. Commun. 181, 1517 (2010), arXiv:0911.3191 [hep-lat] .
  * Babich  _et al._ (2010) R. Babich, M. A. Clark, and B. Joo, in  _SC 10 (Supercomputing 2010)_ (2010) arXiv:1011.0024 [hep-lat] .
  * Joó  _et al._ (2016) B. Joó, D. D. Kalamkar, T. Kurth, K. Vaidyanathan, and A. Walden (Springer International Publishing, Cham, 2016) pp. 415–427.
  * Chen  _et al._ (2023) J. Chen, R. G. Edwards, and W. Mao, in  _Platform for Advanced Scientific Computing_ (2023).
  * Stanzione  _et al._ (2020) D. Stanzione, J. West, R. T. Evans, T. Minyard, O. Ghattas, and D. K. Panda, in  _Practice and Experience in Advanced Research Computing_, PEARC ’20 (Association for Computing Machinery, New York, NY, USA, 2020) p. 106–111.
  * Kovner  _et al._ (2024) D. Kovner, J. Karpie, K. Orginos, A. Radyushkin, and S. Zafeiropoulos (HadStruc), PoS LATTICE2023, 300 (2024), arXiv:2401.06858 [hep-lat] .