# GradXKG: A Universal Explain-per-use Temporal Knowledge Graph Explainer

Chenhan Yuan  The University of ManchesterManchesterUK
<EMAIL_ADDRESS>and  Hoda Eldardiry  Virginia TechBlacksburgU.S.
<EMAIL_ADDRESS>
(2018; 20 February 2007; 12 March 2009; 5 June 2009)

###### Abstract.

Temporal knowledge graphs (TKGs) have shown promise for reasoning tasks by
incorporating a temporal dimension to represent how facts evolve over time.
However, existing TKG reasoning (TKGR) models lack explainability due to their
black-box nature. Recent work has attempted to address this through customized
model architectures that generate reasoning paths, but these recent approaches
have limited generalizability and provide sparse explanatory output. To enable
interpretability for most TKGR models, we propose GradXKG, a novel two-stage
gradient-based approach for explaining Relational Graph Convolution Network
(RGCN)-based TKGR models. First, a Grad-CAM-inspired RGCN explainer tracks
gradients to quantify each node’s contribution across timesteps in an
efficient ”explain-per-use” fashion. Second, an integrated gradients explainer
consolidates importance scores for RGCN outputs, extending compatibility
across diverse TKGR architectures based on RGCN. Together, the two explainers
highlight the most critical nodes at each timestep for a given prediction. Our
extensive experiments demonstrated that, by leveraging gradient information,
GradXKG provides insightful explanations grounded in the model’s logic in a
timely manner for most RGCN-based TKGR models. This helps address the lack of
interpretability in existing TKGR models and provides a universal explanation
approach applicable across various models.

Temporal knowledge graph, explainable AI, temporal knowledge graph reasoning,
event forecasting, gradients-based explanation

††copyright: acmcopyright††journalyear: 2018††doi:
XXXXXXX.XXXXXXX††conference: Make sure to enter the correct conference title
from your rights confirmation emai; June 03–05, 2018; Woodstock, NY††price:
15.00††isbn: 978-1-4503-XXXX-X/18/06††ccs: Computing methodologies Temporal
reasoning

##  1\. Introduction

Since knowledge graphs are dynamic in nature, i.e., evolve over time, Temporal
Knowledge Graphs (TKG) have a promising potential in the fields of question
answering (Saxena et al., 2021; Jia et al., 2021), event forecasting (Deng et
al., 2020; Li et al., 2021b), and information retrieval (Gottschalk and
Demidova, 2018). Unlike conventional static knowledge graphs, which represent
each fact with a triplet $(subject,relation,object)$, temporal knowledge
graphs incorporate a temporal dimension to represent how facts and relations
evolve over time. In general, a temporal knowledge graph represents each fact
using a quadruple $(subject,relation,object,timestamp)$. Conventionally, a
temporal knowledge graph is represented by decomposing it into a sequence of
static knowledge graphs, each of which contains all facts at the corresponding
timestamp.

TKGs provide new perspectives and insights for many downstream applications,
e.g., disease diagnosis aid(Diao et al., 2021) and stock prediction (Deng et
al., 2019). The unique promise of TKGs has sparked a growing interest in
reasoning over TKG. TKG reasoning (TKGR) is a task to validate whether a query
relationship between two entities is true, given the context provided by the
TKG. With the rise of graph neural networks (GNN), most existing TKGR methods
first leverage Relational Graph Convolutional Networks (RGCNs) (Schlichtkrull
et al., 2018), a type of GNN well-suited for multi-relational graphs like
TKGs, to encode the local graph structure into dense vector representations.
Various neural architectures are then applied on top of the RGCN encodings to
score the validity of the query relationship. For instance, curriculum
learning and neural ordinary differential equations have been used to enhance
TKGR performance (Han et al., 2021b; Li et al., 2022). While RGCN-based TKGR
models have achieved significant improvements over general neural network
models, RGCN-based TKGR models still lack explainability due to their end-to-
end black-box nature. The reasoning process behind their predictions is
opaque. This lack of explainability is a critical limitation because
explainability is crucial for trustworthy AI systems.

Recent work has attempted to address this limitation by designing customized
model architectures that can generate reasoning paths along with the
prediction results. For example, TimeTraveler (Sun et al., 2021) proposed a
reinforcement learning-based explainable TKGR model that works as follows.
Given a query: _(Governor (Cote d’Ivoire), Make an appeal or request, ?,
2018/10/14)_ , it generates the reasoning path: _(Governor (Cote d’Ivoire),
Praise or endorse, Party Member, 2018/10/12) $\rightarrow$(Party Member, Make
an appeal or request, Citizen, 2018/9/29)_. However, existing explainable TKGR
models have two critical problems: 1) They are constrained by requiring
custom-designed model architectures to enable reasoning path generation. This
leads to a lack of generalizability, which is a major drawback, as it prevents
easy application to the myriad of most non-explainable TKGR models, i.e.,
RGCN-based models. 2) As illustrated in Fig 1, current explainable TKGR models
only provide reasoning paths with very few nodes per static knowledge graph.
This sparse explanatory output fails to capture the complex interdependencies
within the broader knowledge graph context. In particular, it is difficult to
fully trust or evaluate the faithfulness of explanations without the ability
to account for the potential impact of other entities.

To address these challenges, it is necessary to design a universal explainer
that can work for most RGCN-based TKGR models. A simple and intuitive possible
approach is to follow the GNNExplainer method, by masking a node/edge in the
TKG, and observing its impact on the TKGR model prediction performance. (Ying
et al., 2019). However, to obtain the importance of all nodes across all
timestamps, the GNNExplainer-based method requires repeating the test at least
$\mathcal{O}(n*m)$ times; where $n$ is the number of nodes and $m$ is the
number of timestamps for various inputs; as shown in Fig 2. This is an
exhaustive process that scales poorly as the knowledge graph grows. We propose
a more efficient approach to design a universal explainer. More specifically,
we propose a universal TKGR model explainer that works by analyzing the
model’s gradients. In doing so, it provides real-time rationales for each
individual prediction, elucidating the model’s logic in an accessible
“explain-per-use” fashion.

Figure 1. Comparison between the proposed GradXKG and conventional explainable
TKGR models (xTKGR). Node color shade represents node importance; darker
yellow is more important than lighter yellow. Dashed arrows denote the same
entities in different timestamps. $e?$ is the query entity. Figure 2.
Comparison between the proposed GradXKG and perturbation-based methods.
$g(\cdot)$ denotes TKGR models. Perturbation-based methods require at least
$n$ times running to determine node importance in TKG.

In this paper, we propose GradXKG: a novel two-stage approach for generating
explanations of RGCN-based temporal knowledge graph reasoning (TKGR) models.
Our proposed GradXKG approach bridges a critical gap in universal explainable
temporal knowledge graph reasoning. Our method leverages gradient information
to highlight the most influential nodes across timesteps. First, a Grad-CAM-
inspired RGCN explainer tracks gradients flowing into each node at each
timestep, quantifying the contribution of that node to the RGCN output tensors
in each run. Secondly, to extend the compatibility of our proposed approach to
TKGR architectures that utilize RGCN, we propose an integrated gradients-based
top-layer explainer. This proposed explainer layer tracks the contribution of
RGCN output tensors toward the final prediction in the top layer of these TKGR
models. Moreover, the top-layer explainer associates RGCN output tensors with
prediction scores regardless of the model specifics. This allows our approach
to be applicable to any neural network-based RGCN TKGR model. Together, the
two explainers produce a heat map spotlighting the most critical nodes at each
timestep for a given prediction. Going beyond perturbation-based approaches,
our proposed “explain-per-use” mechanism provides explanations that are
tailored to each individual prediction in a time-efficient manner.
Furthermore, GradXKG enables interpretability within most RGCN-based TKGR
models. This is achieved by generating explanations that are informed by
gradient information.

Our main contributions can be summarized as follows:

  * •

We proposed the first novel gradient-based two-stage RGCN-based TKGR
explanation approach (GradXKG) that can explain all RGCN-based TKGR models. To
the best of our knowledge, this is the first effort in this research area.

  * •

We proposed a Grad-CAM-based RGCN-explainer in an “explain-per-use” fashion,
which significantly reduces time complexity compared to perturbation methods.

  * •

We proved that our proposed explainer can be approximated to the Grad-CAM GCN
model if both are unsigned.

  * •

Our extensive experiments demonstrate that the proposed GradXKG can outperform
other explanation methods in terms of explanation sufficiency and accuracy.

##  2\. Related Work

###  2.1. Temporal Knowledge Graph Reasoning Models

Temporal knowledge graph reasoning is an emerging field that aims to model the
evolution of real-world events and their relationships over time. This task
involves two key settings: interpolation, which focuses on completing
knowledge graphs over a given time span, and extrapolation, which forecasts
future facts based on historical data (Li et al., 2021b, 2022). Early work on
extrapolation like Know-Evolve utilized temporal point processes to capture
continuous-time dynamics (Trivedi et al., 2017). However, graph neural
networks (GNNs) have since become the dominant modeling approach due to their
ability to encode both structural and temporal dependencies. In particular,
relational GCNs (RGCNs) have emerged as a powerful tool for temporal knowledge
graph reasoning (Schlichtkrull et al., 2018). RE-NET pioneered the use of
RGCNs with an autoregressive encoder to model long-term temporal patterns (Jin
et al., 2020). RE-GCN further improved modeling by emphasizing graph
dependency learning (Li et al., 2021b). Later, Zhang et al. proposed a
hierarchical RGCN to learn both long global and short local representations of
temporal knowledge graphs (Zhang et al., 2020). Han et al. adapted neural
ordinary differential equations on RGCN so that they can be applied in
continuous space as temporal knowledge graphs vary continuously over time (Han
et al., 2021b). To better forecast unseen entities and relations, MTKGE
utilized meta-learning to sample from existing temporal knowledge graphs to
simulate future scenarios (Xia et al., 2022). CENET learned both historical
and non-historical dependency and distinguished the dependency type of
predicted events using contrastive learning (Xu et al., 2023). Li et
al.consider the length, diversity, and time-variability of evolutional
patterns by introducing curriculum learning and online learning (Li et al.,
2022). Overall, the ubiquity of RGCNs in TGKR models demonstrates their
importance for encoding local graph structures, on top of which more complex
temporal reasoning techniques can be developed.

Several other approaches generate predictions along with validated reasons.
These explainable techniques can be categorized into three main types: logic
rule-based, reinforcement learning-based, and attention-based. Logic rule
mining is a popular technique for explainable forecasting on temporal
knowledge graphs. TLogic extracts logic rules from the graph via temporal walk
estimation and then applies the rules to make predictions (Liu et al., 2022).
Lin et al. train encoders to incorporate both graph structures and logic
rules(Lin et al., 2023). Reinforcement learning has also been leveraged for
explainable reasoning. Sun et al. developed an RL agent that travels optimal
paths on the knowledge graph to predict future events (Sun et al., 2021). The
agent’s trajectory explains its decision-making. Similarly, Li et al. first
find relevant event clusters and then use RL search to forecast those
subgroups (Li et al., 2021a). Attention mechanisms are another way to enable
explainability. Some methods learn to expand an initial query by attending to
important neighboring nodes (Han et al., 2021a). The attended subgraphs
indicate the influential regions for prediction. Jung et al. also apply graph
attention to iteratively propagate attention weights towards target nodes and
use these weights to provide model interpretability (Jung et al., 2021).

###  2.2. Graph Neural Networks Explanation

GNN explainers can be broadly classified into two distinct categories:
perturbation-based methods and gradient-based methods (Yuan et al., 2022).
Perturbation-based explainers identify important graph components (nodes,
edges, or features) by removing or masking components and measuring the
resulting impact on model predictions. A significant change in predictions
when a component is masked indicates high relevance. For example, GNNExplainer
identifies compact subgraph structures that are important for a prediction by
using mutual information to quantify the difference between predictions on the
original versus perturbed graph (Ying et al., 2019). SubgraphX also masks
substructures but uses Shapley values to measure subgraph importance based on
contribution to the model output (Yuan et al., 2021). Another perturbation-
based method called PGM-Explainer illustrates dependencies between important
features and provides interpretations of a model’s reasoning using Bayesian
network concepts (Vu and Thai, 2020).

In contrast, gradient-based explainers focus directly on analyzing gradient
information flow through the neural network model itself. The key idea is that
components with high gradient magnitude likely have high relevance or impact
on predictions. For example, GNN-LRP utilizes layer-wise relevance
propagation, a technique that redistributes relevance scores based on neural
network activations, to generate detailed explanations of predictions (Bach et
al., 2015; Schnake et al., 2021). GraphLIME adapts a popular local explanation
method called LIME, which stands for Local Interpretable Model-Agnostic
Explanations, to graph domains using nonlinear feature selection (Ribeiro et
al., 2016; Huang et al., 2022). Some methods extend gradient visualization
techniques like Grad-CAM, originally designed for convolutional neural
networks, to graph neural networks to improve explainability (Pope et al.,
2019; Selvaraju et al., 2017; Oquab et al., 2015).

Figure 3. General architecture of an RGCN-based TKGR model. We propose to
explain an RGCN-based TKGR model by decomposing it into two sub-modules: RGCN
explainer and Top-layer explainer.

##  3\. Proposed GradXKG Method

Conventionally, as shown in Fig. 3, an RGCN-based temporal knowledge graph
reasoning model utilizes the Relational Graph Convolutional Network (RGCN) as
the base model to encode graph information in each individual timestamp. On
top of RGCN outputs, various reasoning methods are deployed, such as
curriculum learning (Li et al., 2022), and autoregressive encoder (Jin et al.,
2020), to yield a score ranging from 0-1 that indicates if the query triplet
will happen in the future or not. The process can be formally defined as
follows: Suppose a set
$\mathcal{G}=\\{\mathcal{G}_{1},\mathcal{G}_{2},\cdots.\mathcal{G}_{T}\\}$
represents the individual static knowledge graph in each timestamp $t$, a
regular temporal knowledge graph reasoning model can be defined as:

(1) $\displaystyle s$
$\displaystyle=f(\mathcal{H}_{1},\mathcal{H}_{2},\cdots,\mathcal{H}_{t})$
$\displaystyle\mathcal{H}_{i}$ $\displaystyle=RGCN(\mathcal{G}_{i})$

where $RGCN(\cdot)$ is the RGCN model that takes each static knowledge graph
as input. $f(\cdot)$ represents the complex temporal knowledge graph reasoning
model that outputs a scalar score $s$. As shown in the Eq. 1, our explanation
method can also be decomposed into two parts: Grad-CAM-based RGCN explanation,
i.e., $RGCN(\cdot)$ explainer, and a universal top-layer explanation, i.e.,
$f(\cdot)$ explainer.

###  3.1. Gradient-based RGCN Explainer

We start deriving our method from a simple setting: Assume that only the RGCN
model itself is used in the temporal knowledge graph reasoning model. Then we
have $s=RGCN(\mathcal{G}_{i})$ following Eq. 1. That is, the output of RGCN is
a scalar that indicates the prediction of the whole model. We propose to
design a gradient-based RGCN explainer method to explain this simple
architecture.

Currently, the most popular gradient-based convolutional neural network (CNN)
explanation method, Grad-CAM, has been successfully extended to graph
convolutional networks (GCN) as GCN shares the similar convolutional layer as
CNN (Pope et al., 2019; Selvaraju et al., 2017). The key idea of GCN Grad-CAM
is to use the gradients of the target classification flowing into the final
convolutional layer to produce a coarse localization map highlighting the
important nodes for predicting the classification result. Formally, the first
step of GCN-based Grad-CAM is to determine the $k$’th graph convolutional
feature map at layer $l$:

(2)
$M^{l}_{k}(X,A)=\sigma(\tilde{D}^{-\frac{1}{2}}\tilde{A}\tilde{D}^{-\frac{1}{2}}M^{l-1}(X,A)W_{k}^{l})$

where $X\in\mathbf{R}^{N\times d_{x}}$ is the representation of the nodes, $N$
is the number of nodes in the graph, and $A\in\mathbf{R}^{d_{x}\times
d_{x}}$is the adjacency matrix. $\tilde{A}=A+I_{N}$ is the adjacency matrix of
the undirected graph with added self-connections. $I_{N}$ is the identity
matrix, $\tilde{D}_{ii}=\sum_{j}\tilde{A}_{ij}$, and $W_{k}^{l}$ is the $k$-th
column of matrix $W_{k}$. Then the importance score $I_{n}$ can be used to
generate a heat map for a node $n$ defined as follows:

(3) $\displaystyle\alpha_{k}^{l}$
$\displaystyle=\frac{1}{N}\sum_{n=1}^{N}\frac{\partial y}{\partial
M^{l}_{k,n}(X,A)}$ $\displaystyle I_{n}^{GCN}$
$\displaystyle=\frac{1}{L}\sum_{l=1}^{L}ReLU(\sum_{k}\alpha_{k}^{l}M^{l}_{k,n}(X,A))$

where $M^{l}_{k,n}(X,A))$ denotes the $k$’th feature at the $l$’th layer of
node $n$. That is, the importance score of each node is a weighted sum over
all feature columns associated with each node. However, despite the success of
this simple yet effective gradient-based approach, it cannot be directly
applied to the RGCN model. This is because the RGCN model binds different
relational edges with different matrices so no $W_{k}^{l}$ can be defined for
the RGCN model. This can be formally observed in the following definition of
the RGCN model:

(4)
$h_{i}^{(l+1)}=\sigma\left(\sum_{r\in\mathcal{R}}\sum_{j\in\mathcal{N}^{r}_{i}}\frac{1}{c_{i,r}}W_{r}^{(l)}h_{j}^{(l)}+W_{0}^{(l)}h_{i}^{(l)}\right),$

where $\mathcal{N}^{r}_{i}$ denotes the set of neighbor indices of node $i$
under relation $r\in\mathcal{R}$ and $h_{i}^{(}l)$ is the hidden state of node
$i$ in layer $l$. $c_{i,r}$ is a normalization constant. As shown in the Eq.
4, the hidden representation of node $i$ in layer $l+1$ is contributed by a
set of relation matrices $W_{r}^{l}$ instead of one $W^{l}$. Therefore, Eq. 3
cannot be directly applied to the RGCN model.

To extend the compatibility of GCN Grad-CAM to RGCN, we first convert Eq. 4
into matrix graph representation form:

(5) $H^{(l+1)}=\sigma\left(\sum_{r\in
R}A_{r}H^{(l)}W_{r}^{(l)}+I_{N}H^{(l)}W_{0}^{(l)}\right)$

where $A_{r}$ is a relation-specific adjacency matrix as shown in the
following equation.

(6) $A_{r,i,j}=\left\\{\begin{array}[]{rcl}1&&{E(i,j)\in r}\\\ 0&&{else}\\\
\end{array}\right.$

where $E(i,j)\in\mathcal{r}$ means the edge between nodes $i$ and $j$
represents relation type $r$. In this way, we can define the $k$-th RGCN
feature map at layer $l$ under each relation individually. Formally, combining
Eq. 5 and Eq. 3, the $k$-th RGCN feature map under relation $r$ can be derived
as follows:

(7)
$H^{(l+1)}_{k,r}=\sigma\left(A_{r}H^{(l)}W_{r,k}^{(l)}+I_{N}H^{(l)}W_{0,k}^{(l)}\right)$

Then the relation-dependent gradient-based weight $\alpha_{k,r}^{l}$ for
$k$-th feature in $l$-th layer under $r$-th relation can be derived as
follows:

(8) $\displaystyle\alpha_{k,r}^{l}$
$\displaystyle=\frac{1}{N}\sum_{n=1}^{N}\frac{\partial s}{\partial
H^{l}_{k,n,r}(X,A)}$ $\displaystyle=\frac{1}{N}\sum_{n=1}^{N}\frac{\partial
y}{\partial\sigma(\cdot)}\frac{\partial\sigma(\cdot)}{\partial
H^{l-1}_{k,n,r}(X,A)}\left(A_{r}W_{r,k}^{(l))}+I_{N}W_{0,k}^{(l)}\right)$

where $s$ is the final score. In this way, the weights derived here are always
associated with one specific relation.

Following a similar procedure as in Eq. 3, the importance score $I_{n,r,l}$
for node $n$ from layer $l$ under relation $r$ is defined as follows:

(9) $I_{n,r,l}^{RGCN}=ReLU(\sum_{k}\alpha_{k,r}^{l}H^{l}_{k,n,r}(X,A))$

We use the average $I_{n,r,l}$ as the final importance score:

(10) $I_{n,r}^{RGCN}=\frac{1}{L}\sum_{l=1}^{L}I_{n,r,l}^{RGCN}$

Note that the implementation complexity of Eq. 9 increases as the number of
relations increases. Therefore, we propose the following approximation that
averages over the relation $r$ dimension:

(11)
$I_{n}^{RGCN}\approx\frac{1}{R}\sum_{r=1}^{R}\frac{1}{L}\sum_{l=1}^{L}I_{n,r,l}^{RGCN}$

###  3.2. Theoretical Analysis of RGCN Grad-CAM

We provide a theoretical analysis of the proposed Grad-CAM-based RGCN
explainer in this section. First, note that the derived RGCN explainer is
equivalent to the original GCN Grad-CAM if the output is unsigned. Formally,
we define the equivalence theorem as follows:

###### Theorem 1.

$I_{n}^{RGCN}\approx I_{n}^{GCN}$ iff. both $I_{n}^{RGCN}$ and $I_{n}^{GCN}$
are unsigned.

###### Proof.

To prove the equivalence theorem, we first convert Eq. 3 into unsigned format
by removing the $ReLU(\cdot)$ function:

(12)
$I_{n}^{GCN}=\frac{1}{L}\sum_{l=1}^{L}\sum_{k}\alpha_{k}^{l}M^{l}_{k,n}(X,A)$

Then the $I_{n}^{RGCN}$ can be derived as follows:

(13) $\displaystyle I_{n}^{RGCN}$
$\displaystyle=\frac{1}{R}\sum_{r=1}^{R}\frac{1}{L}\sum_{l=1}^{L}I_{n,r,l}$
$\displaystyle=\frac{1}{R}\sum_{r=1}^{R}\frac{1}{L}\sum_{l=1}^{L}ReLU(\sum_{k}\alpha_{k,r}^{l}H^{l}_{k,n,r}(X,A))$
$\displaystyle\Rightarrow\frac{1}{L}\sum_{l=1}^{L}\sum_{k}\frac{1}{R}\sum_{r=1}^{R}\alpha_{k,r}^{l}H^{l}_{k,n,r}(X,A)\;\text{(unsigned)}$
$\displaystyle=\frac{1}{L}\sum_{l=1}^{L}\sum_{k}\beta_{k,n}^{l}(X,A)$

Comparing $I_{n}^{RGCN}$ with $I_{n}^{GCN}$, the only difference is between
$\beta_{k,n}^{l}(X,A)$ and $\alpha_{k}^{l}M^{l}_{k,n}(X,A)$. However, we
consider that $\beta_{k,n}^{l}(X,A)$ represents the averaged $k$-th feature
along $r$ dimension, which is equivalent $\alpha_{k}^{l}M^{l}_{k,n}(X,A)$.
Therefore, we have $I_{n}^{RGCN}\approx I_{n}^{GCN}$. ∎

Directly applying Eq. 4 to relational graphs in practice can lead to over-
fitting on rare relations as the number of relations can rapidly grow.
Therefore, the original RGCN provides a linear combination of basis
transformations to represent relation matrices $W_{r}$. It is defined as
follows:

(14) $W_{r}^{(l)}=\sum_{b=1}^{B}a_{rb}^{(l)}V_{b}^{(l)}$

where $V_{b}^{l}\in\mathbf{R}^{d_{r}\times d_{o}}$ is the $b$-th basis in
layer $l$. Note that this decomposition method is consistent with Theorem 1.

###### Theorem 2.

THEOREM 1 holds when $W_{r}^{(l)}=\sum_{b=1}^{B}a_{rb}^{(l)}V_{b}^{(l)}$

###### Proof.

To prove Theorem 2, we first convert the decomposition equation into a matrix
form:

(15) $W_{r}^{(l)}=U_{r}^{l}B^{(l)}$

where $B^{l}\in\mathbf{R}^{B\times d_{r}\times d_{o}}$ is the stack of the set
of all basis. $U_{r}^{l}\in\mathbf{R}^{1\times B}$ is a linear combination of
all coefficients $a_{rb}^{(l)}$. We can therefore derive the hidden layer of
RGCN under basis decomposition as follows:

(16) $H^{(l+1)}=\sigma\left(\sum_{r\in
R}A_{r}H^{(l)}[U_{r}^{l}B^{(l)}]+I_{N}H^{(l)}W_{0}^{(l)}\right)$

Note that $[U_{r}^{l}B^{(l)}]$ is equivalent to the original $W_{r}^{l}$ in
terms of matrix shape. Therefore, the computation of $\alpha_{k,r}^{l}$ and
$H_{k,n,r}^{l}(X,A)$ (Eq. 8$\sim$11)remains unchanged. As the final
$I_{n}^{RGCN}$ preserves the same format, Theorem 1 still holds. ∎

###  3.3. Top-Layer Integration

####  3.3.1. Top-Layer Explanation Method

We proposed a top-layer explanation method that is model-agnostic so it can
work with any model type in the layers above the RGCN. In particular, we
utilize the integrated gradients method to quantify the contribution of each
tensor to the model prediction for the layers above the RGCN. Integrated
gradients is an explanation method by attributing a prediction to individual
features (Sundararajan et al., 2017). The integrated gradients method then
calculates the contribution of each feature by integrating the gradients of
the model’s output with respect to that feature. This calculation is carried
out along a path from a baseline input to the actual input. Formally, the
integrated gradients for a one-dimensional tensor $x$ can be defined as
follows:

(17) $IG(x)::=(x-x^{\prime})\int_{\alpha=0}^{1}\frac{\partial
f(x^{\prime}+\alpha(x-x^{\prime}))}{\partial x}\,d\alpha$

where $f(\cdot)$ denotes the general neural network function and $x^{\prime}$
is the base input. In practice, $x^{\prime}$ is set to random or zeros.

Note that the integral part of Eq. 17 is not applicable in neural networks.
Therefore, we approximate the Eq. 17 by summing over small intervals. Then we
have:

(18) $IG(x)\approx(x-x^{\prime})\sum_{\alpha=1}^{p}\frac{\partial
f(x^{\prime}+\frac{p}{\alpha}(x-x^{\prime}))}{\partial x}\frac{1}{p}$

where $p$ is the sampling times.

####  3.3.2. Explanation Integration

Our proposed RGCN explainer provides node-level explanations by quantifying
each node’s contribution to the final predicted scalar score $s$, as shown in
Eq. 8. However, this formulation assumes that the RGCN model outputs $s$
directly, which is not always true, since many models integrate RGCN as an
intermediate component. In this case, additional processing is applied to the
RGCN outputs before producing the final prediction. Our proposed RGCN
explainer further considers cases when models include intermediate RGCN
components by propagating explanations through intermediate components to
subsequent layers. For example, given an intermediate tensor $t$ generated by
the RGCN, we compute explanations in two steps. First, we compute the
importance of each node to generate $t$. Second, we compute the importance of
$t$ to generate the final prediction. Finally, each node’s final explanation
is the sum of the nodes contribution to $t$ combined with $t$’s contribution
to the final prediction. More formally, for a model defined as in Eq. 1, the
derivation of the final predicted score for any node $n$ can be expressed as:

(19) $\displaystyle I_{n}$ $\displaystyle=\frac{\partial s}{\partial n}$
$\displaystyle=\frac{\partial s}{\partial f}\frac{\partial f}{\partial
RGCN(\mathcal{G}_{t})}$
$\displaystyle=f^{\prime}(\mathcal{H}_{t})\frac{\partial f}{\partial
RGCN(\mathcal{G}_{t})}$ $\displaystyle=IG(\mathcal{H}_{t})\cdot I_{n}^{RGCN}$

##  4\. Experiments

Models | N=5
---|---
ICEWS14 | ICEWS0515 | ICEWS18
Fidelity |  Stability |  Time Cost |  Fidelity |  Stability |  Time Cost |  Fidelity |  Stability |  Time Cost
TLogic |  0.50 |  0.67 |  1.0 |  0.58 |  0.55 |  1.1 |  0.55 |  0.58 |  1.3
TimeTraveler |  0.59 |  0.69 |  1.6 |  0.57 |  0.54 |  1.5 |  0.55 |  0.68 |  1.7
per-RE-NET |  0.77 |  0.85 |  76.6 |  0.76 |  0.87 |  70.9 |  0.86 |  0.82 |  78.0
per-CEM |  0.81 |  0.95 |  97.8 |  0.88 |  0.94 |  92.2 |  0.83 |  0.89 |  98.5
IG-RE-NET |  0.48 |  0.55 |  2.5 |  0.47 |  0.57 |  3.2 |  0.41 |  0.45 |  2.4
IG-CEM |  0.57 |  0.53 |  1.3 |  0.44 |  0.46 |  1.5 |  0.50 |  0.45 |  1.9
xRE-NET |  0.61 |  0.79 |  1.2 |  0.73 |  0.77 |  1.1 |  0.65 |  0.73 |  1.3
xCEM |  0.70 |  0.82 |  1.0 |  0.71 |  0.78 |  1.0 |  0.74 |  0.76 |  1.0
Table 1. Automatic evaluation scores of each explainable TKGR model under
three datasets. $N$ denotes the number of nodes output by the models as an
explanation.

In this section, we conduct extensive experiments to evaluate our proposed
GradXKG’s performance on RGCN-based TKGR models. Specifically, we investigate
four research questions:

  * •

_Does GradXKG provide more sufficient explanations than model-specific
approaches for all evaluated RGCN-based TKGR models?_

  * •

_Compared to perturbation-based methods, does GradXKG offer comparable or
superior explanation quality with lower time costs?_

  * •

_Does the proposed RGCN explainer in GradXKG contribute to explanation
quality?_

  * •

_Can GradXKG provide relevant, validated explanations?_

###  4.1. Experimental Settings

####  4.1.1. Datasets

Currently, there are no TKGR datasets designed specifically for explainable
TKGR model evaluation that contain explanations associated with each quadruple
(fact). Therefore, we first utilize the standard ICEWS14 (Garcia-Duran et al.,
2018), ICEWS18 (Jin et al., 2020), and ICEWS0515 (Garcia-Duran et al., 2018)
datasets from the Integrated Crisis Early Warning System to evaluate our
proposed GradXKG method implemented on TKGR models, following previous work.
We then designed an automatic evaluation approach that is based on
faithfulness and trustworthiness criteria and does not require a golden
standard testing set with explanations. Additionally, we conducted a human
evaluation to further assess the performance of the generated explanations.

####  4.1.2. Baselines

We implemented our approach on two state-of-the-art RGCN-based TKGR models:
CEM (Li et al., 2022) and RE-NET (Jin et al., 2020). Since there are currently
no universal or gradient-based explanation methods for TKGR models to compare
against, we implemented two model-specific explainable TKGR methods as
baselines: TimeTraveler (Sun et al., 2021) and TLogic (Liu et al., 2022). We
also implemented a simple perturbation-based explanation method on both CEM
and RE-NET as additional baselines to benchmark GradXKG. To analyze the
effectiveness of the proposed RGCN explainer, we conducted an ablation study
by replacing RGCN explainer with integrated gradients. In total, we compared
eight methods: xCEM: A curriculum learning-based RGCN TKGR model. xCEM denotes
that GradXKG is implemented on CEM. xRE-NET: A autoregressive RGCN TKGR model.
xRE-NET means GradXKG implemented version. per-CEM: CEM model with
perturbation-based method-generated explanations. per-RE-NET: RE-NET model
with perturbation-based method-generated explanations IG-CEM: The whole CEM
model is explained by integrated gradients. IG-RE-NET: The whole RE-NET model
is explained by integrated gradients. TimeTraveler: A RL-based explainable
TKGR model. This model does not contain RGCN architecture. TLogic: A temporal
random walk-based explainable TKGR model that does not use RGCN.

Models | N=9
---|---
ICEWS14 | ICEWS0515 | ICEWS18
Fidelity |  Stability |  Time Cost |  Fidelity |  Stability |  Time Cost |  Fidelity |  Stability |  Time Cost
TLogic |  0.44 |  0.53 |  1.3 |  0.47 |  0.51 |  1.7 |  0.40 |  0.54 |  2.1
TimeTraveler |  0.50 |  0.49 |  3.0 |  0.51 |  0.52 |  2.7 |  0.40 |  0.56 |  3.2
per-RE-NET |  0.78 |  0.87 |  76.5 |  0.88 |  0.83 |  70.9 |  0.71 |  0.90 |  78.1
per-CEM |  0.83 |  0.93 |  97.6 |  0.75 |  0.95 |  92.3 |  0.80 |  0.94 |  98.5
IG-RE-NET |  0.41 |  0.53 |  2.4 |  0.43 |  0.55 |  3.1 |  0.44 |  0.54 |  2.5
IG-CEM |  0.47 |  0.52 |  1.5 |  0.46 |  0.40 |  1.3 |  0.36 |  0.42 |  1.9
xRE-NET |  0.63 |  0.77 |  1.4 |  0.71 |  0.75 |  1.3 |  0.70 |  0.78 |  1.8
xCEM |  0.74 |  0.85 |  1.0 |  0.70 |  0.78 |  1.0 |  0.72 |  0.74 |  1.0
Table 2. Fidelity and stability scores of each explainable TKGR model under
three datasets. $N$ denotes the number of nodes output by the models as an
explanation.

####  4.1.3. Evaluation Metrics

Given the lack of explainable TKGR model evaluation datasets, we assessed the
quality of generated explanations through both automatic and human evaluation.
We utilized three automatic evaluation metrics, namely fidelity, stability,
and time complexity. We also used three human evaluation criteria, namely
validity, relevance, and sufficiency.

Following previous work on GNN explanation (Yuan et al., 2022), we considered
the following two automatic criteria. 1) Fidelity, which measures whether the
explanations are faithfully important to the model’s predictions. Formally,
fidelity can be defined by measuring the prediction gap between the original
graph input and graph input that lacks important nodes:

(20)
$Fidelity=\frac{1}{N}\sum_{i=1}^{N}(f(\mathcal{G}_{i})-f(\mathcal{G}_{i}^{m_{i}}))$

where $f(\cdot)$ denotes the TKGR model, $\mathcal{G}_{i}$ is the input graph,
and $\mathcal{G}_{i}^{m_{i}}$ is the graph with important node $m_{i}$
removed. 2) Stability, which measures the ability of the explainable TKGR
model to produce a consistent explanation when the input graph is slightly
altered or perturbed. We measure stability as follows:

(21)
$Stability=\frac{|\mathcal{N}_{p}\cap\mathcal{N}_{o}|}{|\mathcal{N}_{o}|}$

where $\mathcal{N}_{p}$ is the set of important nodes generated when the input
graph is altered and $\mathcal{N}_{o}$ is the set of important nodes generated
when the input graph is the original one. In addition to fidelity and
stability, we also report the time complexity of each explanation method.

In addition to the automatic evaluation, we also included three human
evaluation metrics as follows. Two annotators evaluated explanations based on
three metrics: validity, relevance, and sufficiency criteria using a 3-point
scale (1=low, 3=high). Validity measures whether the explanation is logically
valid based on the annotator’s knowledge, indicating if the model learned the
expected features. Relevance measures whether selected nodes are connected
meaningfully to the target node. Sufficiency measures if the explanation
contains enough nodes to sufficiently understand the reasoning behind the
prediction, since explanations with too few nodes may lack context. Note that
there may be a trade-off between relevance and sufficiency, since more nodes
may increase sufficiency but decrease relevance. The detailed evaluation
guideline is provided in Appendix A.

###  4.2. Automatic Evaluation Results

We conduct two experiments where the number of nodes output in the explanation
is constrained to 5 and 9, representing reasoning paths of lengths 3 and 4,
respectively. This allows us to evaluate the performance of GradXKG compared
to TimeTravler and TLogic, which have constraints on the length of the
reasoning path they can output. The results in Table 1 demonstrate that with 5
node explanations, GradXKG achieves higher fidelity and stability scores than
TimeTravler and TLogic on all datasets. Specifically, on the ICEWS05015
dataset, the xCEM and xRE-NET variants of GradXKG obtain fidelity scores of
0.73 and 0.71, and stability scores of 0.77 and 0.78; compared to 0.58 and
0.57 fidelity, and 0.55 and 0.54 stability for TimeTravler and TLogic,
respectively. This shows that despite being a universal explainer without
special architectural constraints, GradXKG can generate more trustworthy
explanations than previous constrained models.

We also compare GradXKG to perturbation-based methods, which intrinsically
optimize explanation fidelity and stability. As expected, these methods
slightly outperform GradXKG but at a drastically higher computational cost.
For instance, per-CEM takes 97.8 times longer than xCEM on the ICEWS14
dataset, demonstrating the efficiency of GradXKG’s “explain-per-use” approach.

Note that TLogic is not theoretically runnable when the reasoning path length
is longer than 5. Therefore, we also set the generated number of nodes
requirement to be 9 and show the results in Table. 2. Compared to the 5-node
experiment, the 9-node experiment shows that most perturbation and gradient-
based methods maintain a similar performance across all three automatic
criteria. This is because they score all nodes and select the top ones.
However, TimeTravler and TLogic slow down considerably and have lower fidelity
and stability as they are constrained by needing to traverse explanation
trajectories. Overall, the experiments demonstrate GradXKG’s ability to
generate trustworthy explanations efficiently despite its model-agnostic
nature.

###  4.3. Human Evaluation and Qualitative Analysis

Human Evaluation To further evaluate the quality of the generated
explanations, we randomly selected 50 generated explanations from each
compared method and asked two experienced annotators to rate the quality of
the explanations based on the aforementioned criteria. As shown in Fig. 5, the
saliency map explanations generated by GradXKG achieved higher sufficiency
scores compared to the other methods. This demonstrates that compared to
reasoning path explanations, like those from TLogic and TimeTraveler, the
proposed GradXKG method can provide more contextual information by assigning
importance scores to all nodes in the TKG. Furthermore, the relevance scores
of xCEM are comparable to those of TLogic, indicating that the importance
scores of each node are aligned with its contribution to the prediction. We
also observed differences in the evaluation results between xRE-NET and xCEM
even when using the same GradXKG method. This discrepancy can be attributed to
the fact that the explainer’s performance also heavily depends on the
capabilities of the original model.

Qualitative Analysis We include two saliency maps generated by xCEM, the best
variant of GradXKG, when the following query is submitted to the model:
_(National Council for Peace and Order of Thailand, Make Statement, ?,
2014-07-24)_ and _(John Kerry, Engage in negotiation, ?, 2014-07-24)_. The
correct answers to these two queries are _Thailand_ and _Benjamin Netanyahu_ ,
respectively. As shown in Fig. 4, GradXKG revealed that the CEM model relied
on nodes/entities that are highly related to the queries in order to make the
correct predictions. For instance, for the query _(John Kerry, Engage in
negotiation, ?, 2014-07-24)_ , CEM attended to other U.S. government officials
like Barack Obama and international political organizations such as “UN
Security Council” as well as politicians in the Middle East region. This
demonstrates that with the proposed GradXKG method, RGCN-based TKGR models’
predictions can be accurately interpreted and explained.

Figure 4. The saliency maps generated by xCEM, where big nodes are the
selected important nodes by GradXKG. We omit the rest of the TKGs and
timestamps for better illustration. Figure 5. The box plots of human
evaluation for each criterion of each baseline method. The dashed line denotes
the mean value and the bold line indicates the median value.

###  4.4. Ablation Study

To evaluate the effectiveness of the proposed RGCN explainer, we also
conducted an ablation study by replacing the RGCN explainer with integrated
gradients (IG). The node importance score is calculated by averaging the IG
scores across the whole vector representation of that node. As shown in Tables
1 and 2, the IG-based RE-NET and CEM methods perform much worse in terms of
fidelity and stability. This demonstrates the effectiveness and
trustworthiness of the proposed RGCN explainer. Furthermore, as illustrated in
Fig. 5, the human evaluation also suggests that IG-only RE-NET or CEM methods
cannot provide a saliency map with high validity, relevance, and sufficiency.
Together, this shows the necessity and effectiveness of the proposed RGCN
explainer.

##  5\. Conclusion

In this paper, we propose GradXKG, a universal explain-per-use temporal
knowledge graph explainer. The proposed approach can be applied to most RGCN-
based TKGR models to generate saliency maps that indicate node importance
(contribution) towards the prediction as the explanation. Our proposed
explainer model tracks the gradient flow in the whole model by introducing,
and combining, two explainers: a Grad-CAM-based RGCN explainer and an
integrated gradient-based top-layer explainer. Our extensive experiments
demonstrate that with GradXKG, most state-of-the-art TKGR models can be
explained with high fidelity and stability in a time-efficient manner.

## References

  * (1)
  * Bach et al. (2015) Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojciech Samek. 2015.  On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation.  _PloS one_ 10, 7 (2015), e0130140.
  * Deng et al. (2020) Songgaojun Deng, Huzefa Rangwala, and Yue Ning. 2020\.  Dynamic knowledge graph based multi-event forecasting. In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 1585–1595.
  * Deng et al. (2019) Shumin Deng, Ningyu Zhang, Wen Zhang, Jiaoyan Chen, Jeff Z Pan, and Huajun Chen. 2019\.  Knowledge-driven stock trend prediction and explanation via temporal convolutional network. In _Companion Proceedings of The 2019 World Wide Web Conference_. 678–685.
  * Diao et al. (2021) Lijuan Diao, Wei Yang, Penghua Zhu, Gaofang Cao, Shoujun Song, and Yang Kong. 2021\.  The research of clinical temporal knowledge graph based on deep learning.  _Journal of Intelligent & Fuzzy Systems_ 41, 3 (2021), 4265–4274.
  * Garcia-Duran et al. (2018) Alberto Garcia-Duran, Sebastijan Dumančić, and Mathias Niepert. 2018.  Learning Sequence Encoders for Temporal Knowledge Graph Completion. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_. 4816–4821.
  * Gottschalk and Demidova (2018) Simon Gottschalk and Elena Demidova. 2018.  Eventkg: A multilingual event-centric temporal knowledge graph. In _The Semantic Web: 15th International Conference, ESWC 2018, Heraklion, Crete, Greece, June 3–7, 2018, Proceedings 15_. Springer, 272–287.
  * Han et al. (2021a) Zhen Han, Peng Chen, Yunpu Ma, and Volker Tresp. 2021a.  Explainable Subgraph Reasoning for Forecasting on Temporal Knowledge Graphs. In _International Conference on Learning Representations_.  https://openreview.net/forum?id=pGIHq1m7PU
  * Han et al. (2021b) Zhen Han, Zifeng Ding, Yunpu Ma, Yujia Gu, and Volker Tresp. 2021b.  Learning neural ordinary equations for forecasting future links on temporal knowledge graphs. In _Proceedings of the 2021 conference on empirical methods in natural language processing_. 8352–8364.
  * Huang et al. (2022) Qiang Huang, Makoto Yamada, Yuan Tian, Dinesh Singh, and Yi Chang. 2022.  Graphlime: Local interpretable model explanations for graph neural networks.  _IEEE Transactions on Knowledge and Data Engineering_ (2022).
  * Jia et al. (2021) Zhen Jia, Soumajit Pramanik, Rishiraj Saha Roy, and Gerhard Weikum. 2021.  Complex temporal question answering on knowledge graphs. In _Proceedings of the 30th ACM international conference on information & knowledge management_. 792–802.
  * Jin et al. (2020) Woojeong Jin, Meng Qu, Xisen Jin, and Xiang Ren. 2020\.  Recurrent Event Network: Autoregressive Structure Inferenceover Temporal Knowledge Graphs. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. 6669–6683.
  * Jung et al. (2021) Jaehun Jung, Jinhong Jung, and U Kang. 2021.  Learning to walk across time for interpretable temporal knowledge graph completion. In _Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining_. 786–795.
  * Li et al. (2022) Zixuan Li, Saiping Guan, Xiaolong Jin, Weihua Peng, Yajuan Lyu, Yong Zhu, Long Bai, Wei Li, Jiafeng Guo, and Xueqi Cheng. 2022\.  Complex Evolutional Pattern Learning for Temporal Knowledge Graph Reasoning. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_. 290–296.
  * Li et al. (2021a) Zixuan Li, Xiaolong Jin, Saiping Guan, Wei Li, Jiafeng Guo, Yuanzhuo Wang, and Xueqi Cheng. 2021a.  Search from History and Reason for Future: Two-stage Reasoning on Temporal Knowledge Graphs. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_. 4732–4743.
  * Li et al. (2021b) Zixuan Li, Xiaolong Jin, Wei Li, Saiping Guan, Jiafeng Guo, Huawei Shen, Yuanzhuo Wang, and Xueqi Cheng. 2021b.  Temporal knowledge graph reasoning based on evolutional representation learning. In _Proceedings of the 44th international ACM SIGIR conference on research and development in information retrieval_. 408–417.
  * Lin et al. (2023) Qika Lin, Jun Liu, Rui Mao, Fangzhi Xu, and Erik Cambria. 2023.  TECHS: Temporal Logical Graph Networks for Explainable Extrapolation Reasoning. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. 1281–1293.
  * Liu et al. (2022) Yushan Liu, Yunpu Ma, Marcel Hildebrandt, Mitchell Joblin, and Volker Tresp. 2022.  Tlogic: Temporal logical rules for explainable link forecasting on temporal knowledge graphs. In _Proceedings of the AAAI conference on artificial intelligence_ , Vol. 36. 4120–4127.
  * Oquab et al. (2015) Maxime Oquab, Léon Bottou, Ivan Laptev, and Josef Sivic. 2015\.  Is object localization for free?-weakly-supervised learning with convolutional neural networks. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 685–694.
  * Pope et al. (2019) Phillip E Pope, Soheil Kolouri, Mohammad Rostami, Charles E Martin, and Heiko Hoffmann. 2019\.  Explainability methods for graph convolutional neural networks. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 10772–10781.
  * Ribeiro et al. (2016) Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016\.  ” Why should i trust you?” Explaining the predictions of any classifier. In _Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining_. 1135–1144.
  * Saxena et al. (2021) A Saxena, S Chakrabarti, and P Talukdar. 2021.  Question answering over temporal knowledge graphs. In _ACL-IJCNLP 2021-59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, Proceedings of the Conference_. Association for Computational Linguistics (ACL), 6663–6676.
  * Schlichtkrull et al. (2018) Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling. 2018.  Modeling relational data with graph convolutional networks. In _The Semantic Web: 15th International Conference, ESWC 2018, Heraklion, Crete, Greece, June 3–7, 2018, Proceedings 15_. Springer, 593–607.
  * Schnake et al. (2021) Thomas Schnake, Oliver Eberle, Jonas Lederer, Shinichi Nakajima, Kristof T Schütt, Klaus-Robert Müller, and Grégoire Montavon. 2021.  Higher-order explanations of graph neural networks via relevant walks.  _IEEE transactions on pattern analysis and machine intelligence_ 44, 11 (2021), 7581–7596.
  * Selvaraju et al. (2017) Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. 2017.  Grad-cam: Visual explanations from deep networks via gradient-based localization. In _Proceedings of the IEEE international conference on computer vision_. 618–626.
  * Sun et al. (2021) Haohai Sun, Jialun Zhong, Yunpu Ma, Zhen Han, and Kun He. 2021.  TimeTraveler: Reinforcement Learning for Temporal Knowledge Graph Forecasting. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_. 8306–8319.
  * Sundararajan et al. (2017) Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017.  Axiomatic attribution for deep networks. In _International conference on machine learning_. PMLR, 3319–3328.
  * Trivedi et al. (2017) Rakshit Trivedi, Hanjun Dai, Yichen Wang, and Le Song. 2017\.  Know-evolve: Deep temporal reasoning for dynamic knowledge graphs. In _international conference on machine learning_. PMLR, 3462–3471.
  * Vu and Thai (2020) Minh Vu and My T Thai. 2020\.  Pgm-explainer: Probabilistic graphical model explanations for graph neural networks.  _Advances in neural information processing systems_ 33 (2020), 12225–12235.
  * Xia et al. (2022) Yuwei Xia, Mengqi Zhang, Qiang Liu, Shu Wu, and Xiao-Yu Zhang. 2022.  MetaTKG: Learning Evolutionary Meta-Knowledge for Temporal Knowledge Graph Reasoning. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_. 7230–7240.
  * Xu et al. (2023) Yi Xu, Junjie Ou, Hui Xu, and Luoyi Fu. 2023\.  Temporal knowledge graph reasoning with historical contrastive learning. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 37. 4765–4773.
  * Ying et al. (2019) Zhitao Ying, Dylan Bourgeois, Jiaxuan You, Marinka Zitnik, and Jure Leskovec. 2019.  Gnnexplainer: Generating explanations for graph neural networks.  _Advances in neural information processing systems_ 32 (2019).
  * Yuan et al. (2022) Hao Yuan, Haiyang Yu, Shurui Gui, and Shuiwang Ji. 2022\.  Explainability in graph neural networks: A taxonomic survey.  _IEEE transactions on pattern analysis and machine intelligence_ 45, 5 (2022), 5782–5799.
  * Yuan et al. (2021) Hao Yuan, Haiyang Yu, Jie Wang, Kang Li, and Shuiwang Ji. 2021.  On explainability of graph neural networks via subgraph explorations. In _International conference on machine learning_. PMLR, 12241–12252.
  * Zhang et al. (2020) Zhao Zhang, Fuzhen Zhuang, Hengshu Zhu, Zhiping Shi, Hui Xiong, and Qing He. 2020\.  Relational graph neural network with hierarchical attention for knowledge graph completion. In _Proceedings of the AAAI conference on artificial intelligence_ , Vol. 34. 9612–9619.

##  Appendix A Annotation Guideline

We provide detailed evaluation guidelines for human evaluations in this
section. Overview

You will be shown explanations generated by a model for temporal knowledge
graph reasoning tasks. Your task is to evaluate the quality of the explanation
based on 3 criteria using a 1-3 rating scale:

  * •

Validity (1=low, 3=high) - Does the explanation seem logically valid based on
your own knowledge? Note that invalid explanations indicate the model may not
have learned expected features.

  * •

Relevance (1=low, 3=high) - Do the nodes selected seem relevant for explaining
the model’s prediction? Relevant nodes should meaningfully connect to the
target node being explained.

  * •

Sufficiency (1=low, 3=high) - Does the explanation contain enough nodes to
sufficiently understand the reasoning behind the prediction? Too few nodes may
lack the needed context.

The detailed criteria for each score is described below: Note there is a
tradeoff between sufficiency and relevance - more nodes increase sufficiency
but may reduce relevance.

Validity

  * •

Assign a score of 1 if the explanation is illogical or contains factual errors
based on your knowledge.

  * •

Assign a score of 2 if you are uncertain of the validity or logic of the
explanation.

  * •

Assign a score of 3 if the explanation seems factually and logically valid.

Relevance

  * •

Assign a score of 1 if the nodes seem unrelated to explaining the prediction.

  * •

Assign a score of 2 if some nodes are relevant while others are not.

  * •

Assign a score of 3 if all nodes clearly help explain the prediction through
their connections.

Sufficiency

  * •

Assign a score of 1 if the explanation does not provide enough context to
understand the reasoning.

  * •

Assign a score of 2 if the explanation provides some context but is still
lacking.

  * •

Assign a score of 3 if the explanation provides ample context to understand
the reasoning.


are hired as regular faculty, but typically 50% of their time is covered by a
national laboratory through tenure. with national laboratories. This could be
an effective tool for encouraging new positions, but we should be aware of the
additional demands on the researchers for serving two departments/institutions
instead of one. The creation of cross-cutting tenure-track staff positions at
national laboratories is another effective strategy.

Second, and equally important, we need to eliminate barriers for researchers
with diverse backgrounds to work on S&C and we need to create an environment
that will retain this workforce. To this end, the CPSC should establish a
committee to study the status and challenges related to diversity, equity, and
inclusion within S&C in HEP. There will be some challenges that are common
with HEP more generally, but there may also be properties of S&C that require
specific attention. The committee should present a list of actionable items to
the CPSC leadership for implementation in coordination with the broader S&C
community.

## 12 Conclusions and Outlook

Software and computing technologies are evolving rapidly and it is hard to
imagine what the state of the art will be at the time of the next Snowmass in
about a decade. However, it is clear that the trend towards more heterogeneity
and a diversity of software tools will continue, triggering the need for a HEP
workforce with a correspondingly diverse set of skills and backgrounds.

S&C traverses all areas of HEP. While we have organized the Computational
Frontier along cross-cutting topics, there are strong connections to all other
Snowmass frontiers. Experimental projects, large and small, in the Energy,
Neutrino, Rare Processes and Precision, and Cosmic Frontiers will continue to
rely on S&C to deliver the best physics output from our instruments at a
variety of latencies and bandwidths. Many areas of HEP simulations and theory
more generally also have stringent computational requirements. There are also
strong and growing connections to the other Snowmass Frontiers, including the
Accelerator Frontier (e.g., accelerator control and modeling), the
Instrumentation Frontier (e.g., experimental design), the Underground
Facilities Frontier (e.g., quantum computing), and the Community Engagement
Frontier (e.g., training and industry connections).

It is our hope that the establishment of a Coordinating Panel for Software and
Computing (CPSC) will serve the community on timescales commensurate with the
rapid evolution of S&C technologies. An important function of this panel is to
provide guidance to the funding agencies with input from the domestic and
international HEP communities. Advances in S&C have been essential to progress
in both experiment and theory. Working together as a unified community, we can
ensure this continues in the future.

## References

  * [1] HEP Software Foundation collaboration, _A Roadmap for HEP Software and Computing R &D for the 2020s_, _Comput. Softw. Big Sci._ 3 (2019) 7 [1712.06982].
  * [2] GEANT4 collaboration, _GEANT4–a simulation toolkit_ , _Nucl. Instrum. Meth. A_ 506 (2003) 250.
  * [3] J. Allison et al., _Geant4 developments and applications_ , _IEEE Trans. Nucl. Sci._ 53 (2006) 270.
  * [4] J. Allison et al., _Recent developments in Geant4_ , _Nucl. Instrum. Meth. A_ 835 (2016) 186.
  * [5] T. Sjostrand, S. Mrenna and P.Z. Skands, _PYTHIA 6.4 Physics and Manual_ , _JHEP_ 05 (2006) 026 [hep-ph/0603175].
  * [6] L.A.T. Bauerdick et al., _Planning the Future of U.S. Particle Physics (Snowmass 2013): Chapter 9: Computing Frontier_ , in _Community Summer Study 2013: Snowmass on the Mississippi_ , 1, 2014 [1401.6117].
  * [7] A. Connolly et al., _Working Group Report: Computing for the Cosmic Frontier_ , in _Community Summer Study 2013: Snowmass on the Mississippi_ , 2013 [1311.2841].
  * [8] I. Fisk and J. Shank, _Working Group Report: Computing for the Energy Frontier_ , in _Community Summer Study 2013: Snowmass on the Mississippi_ , 2014 [1401.1840].
  * [9] B. Rebel, M.C. Sanchez and S. Wolbers, _Working Group Report: Computing for the Intensity Frontier_ , in _Community Summer Study 2013: Snowmass on the Mississippi_ , 2013 [1310.6964].
  * [10] P. Spentzouris et al., _Working Group Report: Computing for Accelerator Science_ , in _Community Summer Study 2013: Snowmass on the Mississippi_ , 2013 [1310.2203].
  * [11] T. Blum et al., _Working Group Report: Lattice Field Theory_ , in _Community Summer Study 2013: Snowmass on the Mississippi_ , 2013 [1310.6087].
  * [12] S. Hoche et al., _Working Group Report: Computing for Perturbative QCD_ , in _Community Summer Study 2013: Snowmass on the Mississippi_ , 2013 [1309.3598].
  * [13] K. Bloom and R. Gerber, _Working Group Report: Distributed Computing and Facilities Infrastructure_ , in _Community Summer Study 2013: Snowmass on the Mississippi_ , 2013 [1311.2208].
  * [14] G. Bell and M. Ernst, _Working Group Report: Networking_ , in _Community Summer Study 2013: Snowmass on the Mississippi_ , 2013 [1311.2478].
  * [15] D. Brown et al., _Working Group Report: Software Development, Personnel, and Training_ , in _Community Summer Study 2013: Snowmass on the Mississippi_ , 2013 [1311.2567].
  * [16] M. Butler, R. Mount and M. Hildreth, _Working Group Report: Storage and Data Management_ , in _Community Summer Study 2013: Snowmass on the Mississippi_ , 2013 [1311.4580].
  * [17] G. Cerati, K. Heitmann, W. Hopkins et al., _Snowmass Computational Frontier: Topical Group Report on Experimental Algorithm Parallelization_ , 2209.07356.
  * [18] P. Boyle, K. Pedro, J. Qiang, _CompF2: Theoretical Calculations and Simulation Topical Group Report_ , 2209.08177.
  * [19] P. Shanahan, K. Terao, D. Whiteson, _Snowmass 2021 Computational Frontier CompF03 Topical Group Report: Machine Learning_ , 2209.07559.
  * [20] W. Bhimji, M. Lin, F. Würthwein et al., _Snowmass 2021 Computational Frontier CompF4 Topical Group Report: Storage and Processing Resource Access_ , 2209.08868.
  * [21] G. Davies, P. Onyisi, A. Roberts, _CompF5: End User Analysis Topical Group Report_ , 2209.14984.
  * [22] R. Brun and F. Rademakers, _ROOT: An object oriented data analysis framework_ , _Nucl. Instrum. Meth. A_ 389 (1997) 81.
  * [23] O. Shadura, B. Krikler and G.A. Stewart, _HL-LHC Computing Review Stage 2, Common Software Projects: Data Science Tools for Analysis_ , in _LHCC Review of HL-LHC Computing_ , J. Pivarski, E. Rodrigues and K. Pedro, eds., 2022 [2202.02194].
  * [24] P. Virtanen, R. Gommers, T.E. Oliphant, M. Haberland, T. Reddy, D. Cournapeau et al., _SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python_ , _Nature Methods_ 17 (2020) 261.
  * [25] T. Humble, G. Perdue, M. Savage, _Snowmass Computational Frontier: Topical Group Report on Quantum Computing_ , 2209.06786.
  * [26] S. Bailey, K. Cranmer et al., _Reinterpretation and Long-Term Preservation of Data and Code_ , 2209.08054.
  * [27] R.K. Ellis et al., _Physics Briefing Book: Input for the European Strategy for Particle Physics Update 2020_ , 1910.11775.
  * [28] _2020 Update of the European Strategy for Particle Physics (Brochure)_ , _CERN-ESU-015_ (2020) .
  * [29] HEP Software Foundation collaboration, _HL-LHC Computing Review: Common Tools and Community Software_ , 2008.13636.
  * [30] A.E. Bauer, E.C. Bellm, A.S. Bolton, S. Chaudhuri, A.J. Connolly, K.L. Cruz et al., _Petabytes to Science_ , 1905.05116.
  * [31] B.J. Mount et al., _LUX-ZEPLIN (LZ) Technical Design Report_ , 1703.09144.
  * [32] DUNE collaboration, _Deep Underground Neutrino Experiment (DUNE), Far Detector Technical Design Report, Volume II: DUNE Physics_ , 2002.03005.
  * [33] X. Qian, C. Zhang, B. Viren and M. Diwan, _Three-dimensional Imaging for Large LArTPCs_ , _JINST_ 13 (2018) P05032 [1803.04850].
  * [34] C. Andreopoulos et al., _Software and Computing for Small HEP Experiments_ , 2203.07645.
  * [35] M. Alvarez et al., _Snowmass2021 Computational Frontier White Paper: Cosmological Simulations and Modeling_ , in _2022 Snowmass Summer Study_ , 2022 [2203.07347].
  * [36] A. Banerjee et al., _Snowmass2021 Cosmic Frontier White Paper: Cosmological Simulations for Dark Matter Physics_ , in _2022 Snowmass Summer Study_ , 2022 [2203.07049].
  * [37] A. Huebl, R. Lehe, J.-L. Vay, D.P. Grote, I. Sbalzarini, S. Kuschel et al., “openPMD: A meta data standard for particle and mesh based data.” https://github.com/openPMD, 2015.  10.5281/zenodo.591699.
  * [38] J. Qiang, S. Lidia, R.D. Ryne and C. Limborg-Deprey, _Three-dimensional quasistatic model for high brightness beam dynamics simulation_ , _Phys. Rev. ST Accel. Beams_ 9 (2006) 044204.
  * [39] J. Qiang, R.D. Ryne, M. Venturini, A.A. Zholents and I.V. Pogorelov, _High resolution simulation of beam dynamics in electron linacs for x-ray free electron lasers_ , _Phys. Rev. ST Accel. Beams_ 12 (2009) 100702.
  * [40] A. Edelen et al., _Machine learning for orders of magnitude speedup in multiobjective optimization of particle accelerator systems_ , _Phys. Rev. Accel. Beams_ 23 (2020) 044601.
  * [41] J. Apostolakis et al., _Detector Simulation Challenges for Future Accelerator Experiments_ , _Front. Phys._ 10 (2022) 913510.
  * [42] S. Banerjee et al., _Detector and Beamline Simulation for Next-Generation High Energy Physics Experiments_ , _2022 Snowmass Summer Study_ (2022) [2203.07614].
  * [43] HEP Software Foundation, _HEP Software Foundation Community White Paper Working Group - Detector Simulation_ , 1803.04165.
  * [44] D. Müller, M. Clemencic, G. Corti and M. Gersabeck, _ReDecay: A novel approach to speed up the simulation at LHCb_ , _Eur. Phys. J. C_ 78 (2018) 1009 [1810.10362].
  * [45] LHCb Computing Project, _LHCb computing resources: 2023 requests_ , Tech. Rep. LHCb-PUB-2022-010, CERN-LHCb-PUB-2022-010, CERN, Geneva (2022).
  * [46] CMS Collaboration, “CMS Offline and Computing Public Results .” https://twiki.cern.ch/twiki/bin/view/CMSPublic/CMSOfflineComputingResults, Nov., 2021.
  * [47] ATLAS Collaboration, _ATLAS Software and Computing HL-LHC Roadmap_ , Tech. Rep. CERN-LHCC-2022-005, LHCC-G-182, CERN, Geneva (Mar, 2022).
  * [48] Y. Kahn et al., _Snowmass2021 Cosmic Frontier: Modeling, statistics, simulations, and computing needs for direct dark matter detection_ , in _2022 Snowmass Summer Study_ , 2022 [2203.07700].
  * [49] A. Roberts et al., _Dark-matter And Neutrino Computation Explored (DANCE) Community Input to Snowmass_ , 2203.08338.
  * [50] _https://github.com/apt-sim/adept_ , accessed March 9, 2022.
  * [51] S. Johnson et al., _Novel features and GPU performance analysis for EM particle transport in the Celeritas code_ , _EPJ Web of Conferences_ 251 (2021) .
  * [52] S.C. Tognini et al., _Celeritas: GPU-accelerated particle transport for detector simulation in High Energy Physics experiments_ , in _2022 Snowmass Summer Study_ , 3, 2022, https://arxiv.org/abs/2203.09467 [2203.09467].
  * [53] A. Adelmann et al., _New directions for surrogate models and differentiable programming for High Energy Physics detector simulation_ , 2203.08806.
  * [54] P. Harris et al., _Physics Community Needs, Tools, and Resources for Machine Learning_ , 2203.16255.
  * [55] HSF Physics Event Generator WG, _Challenges in Monte Carlo Event Generator Software for High-Luminosity LHC_ , _Comput. Softw. Big Sci._ 5 (2021) 12 [2004.13687].
  * [56] S. Höche, S. Prestel and H. Schulz, _Simulation of Vector Boson Plus Many Jet Final States at the High Luminosity LHC_ , _Phys. Rev. D_ 100 (2019) 014024 [1905.05120].
  * [57] J.M. Campbell et al., _Event Generators for High-Energy Physics Experiments_ , 2203.11110.
  * [58] F.F. Cordero, A. von Manteuffel and T. Neumann, _Computational challenges for multi-loop collider phenomenology_ , 2204.04200.
  * [59] A. Huss, J. Huston, S. Jones, M. Pellen, _Les Houches 2021: Physics at TeV Colliders: Report on the Standard Model Precision Wishlist_ , 2022.
  * [60] Muon g-2 collaboration, _Measurement of the Positive Muon Anomalous Magnetic Moment to 0.46 ppm_ , _Phys. Rev. Lett._ 126 (2021) 141801 [2104.03281].
  * [61] T.-J. Hou, H.-W. Lin, M. Yan and C.P. Yuan, _Impact of lattice $s(x)-\overline{s}(x)$ data in the CTEQ-TEA global analysis_, 2204.07944.
  * [62] M. Constantinou et al., _Lattice QCD calculations of parton physics_ , 2202.07193.
  * [63] T. Blum et al., _Discovering new physics in rare kaon decays_ , 2203.10998.
  * [64] Y. Aoki et al., _FLAG Review 2021_ , 2111.09849.
  * [65] P. Boyle et al., _Lattice QCD and the Computational Frontier_ , 2204.00039.
  * [66] P. Boyle, G. Cossu, G. Filaci, C. Lehner, A. Portelli and A. Yamaguchi, _Grid: OneCode and FourAPIs_ , in _38th International Symposium on Lattice Field Theory_ , 2022 [2203.06777].
  * [67] P.A. Boyle, G. Cossu, A. Yamaguchi and A. Portelli, _Grid: A next generation data parallel C++ QCD library_ , _PoS_ LATTICE2015 (2016) 023.
  * [68] P.A. Boyle, M. Clark, C. DeTar, M. Lin, V. Rana and A.V. Avilés-Casco, _Performance Portability Strategies for Grid C++ Expression Templates_ , _EPJ Web Conf._ 175 (2018) 09006 [1710.09409].
  * [69] C. DeTar, S. Gottlieb, R. Li and D. Toussaint, _MILC Code Performance on High End CPU and GPU Supercomputer Clusters_ , _EPJ Web Conf._ 175 (2018) 02009.
  * [70] S.A. Gottlieb and S. Tamhankar, _Benchmarking MILC code with OpenMP and MPI_ , _Nucl. Phys. B Proc. Suppl._ 94 (2001) 841 [hep-lat/0011037].
  * [71] RBC, UKQCD collaboration, _Overview of Columbia Physics System_ , _PoS_ LATTICE2013 (2014) 417.
  * [72] SciDAC, LHPC, UKQCD collaboration, _The Chroma software system for lattice QCD_ , _Nucl. Phys. B Proc. Suppl._ 140 (2005) 832 [hep-lat/0409003].
  * [73] M.A. Clark, R. Babich, K. Barros, R.C. Brower and C. Rebbi, _Solving Lattice QCD systems of equations using mixed precision solvers on GPUs_ , _Comput. Phys. Commun._ 181 (2010) 1517 [0911.3191].
  * [74] A. Hocker et al., _TMVA - Toolkit for Multivariate Data Analysis_ , physics/0703039.
  * [75] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro et al., _TensorFlow: Large-scale machine learning on heterogeneous systems_ , 2015\.
  * [76] A.e.a. Paszke, _Pytorch: An imperative style, high-performance deep learning library_ , _Advances in Neural Information Processing Systems_ 32 (2019) 8024.
  * [77] J. Preskill, _Quantum computing in the NISQ era and beyond_ , _Quantum_ 2 (2018) 79.
  * [78] C.W. Bauer et al., _Quantum Simulation for High Energy Physics_ , 2204.03381.
  * [79] M.D. Wilkinson, M. Dumontier, I.J. Aalbersberg, G. Appleton, M. Axton, A. Baak et al., _The fair guiding principles for scientific data management and stewardship_ , _Scientific Data_ 3 (2016) 160018.
  * [80] S. Bailey et al., _Data and Analysis Preservation, Recasting, and Reinterpretation_ , 2203.10057.
  * [81] “American physical society.” https://www.aps.org/programs/minorities/resources/statistics.cfm.
  * [82] “Computing research association.” https://cra.org/wp-content/uploads/2020/05/2019-Taulbee-Survey.pdf.
  * [83] “National public radio.” https://www.npr.org/sections/money/2014/10/21/357629765/when-women-stopped-coding. Source: National Science Foundation, American Bar Association, American Association of Medical Colleges.
  * [84] K. Harmon, E. Walden, _Comparing Three Theories of the Gender Gap in Information Technology Careers: The Role of Salience Differences_ , _Journal of the Association for Information Systems_ 22 (2021) 1099.
  * [85] A. Master, A. Meltzoff, S. Cheryan in _Gender stereotypes about interests start early and cause gender disparities in computer science and engineering._ , vol. 118, p. e2100030118, 2021.
  * [86] O. Egbue, A. Khan, R. Al-Hammoud, “Analysis of workplace climate for female faculty of color in computer sci- ence and engineering.” https://peer.asee.org/analysis-of-workplace-climate-for-female-faculty-of-color-in-computer-science-and-engineering.pdf.
  * [87] HEPAP Subcommittee collaboration, S. Ritz et al., _Building for Discovery: Strategic Plan for U.S. Particle Physics in the Global Context_ , 2014.
  * [88] G. Agarwal, J.L. Barrow, M.F. Carneiro, E. Conley, M.E. da Silva Pereira, S. Hedges et al., _Snowmass 2021 Community Survey Report_ , in _2022 Snowmass Summer Study_ , 3, 2022 [2203.07328].