#  Transferring Neural Potentials For High Order Dependency Parsing

Farshad Noravesh111Email<EMAIL_ADDRESS>
###### Abstract

High order dependency parsing leverages high order features such as siblings
or grandchildren to improve state of the art accuracy of current first order
dependency parsers. The present paper uses biaffine scores to provide an
estimate of the arc scores and is then propagated into a graphical model. The
inference inside the graphical model is solved using dual decomposition. The
present algorithm propagates biaffine neural scores to the graphical model and
by leveraging dual decomposition inference, the overall circuit is trained
end-to-end to transfer first order informations to the high order
informations.

##  1 Introduction

Dependency parsing is the basis of many complex pipelines for problems in
natural language processing such as machine summarization, machine
translation, event extraction, semantic parsing ,semantic role labeling(SRL),
emotion analysis, dialogue systems and information processing. Thus, any error
in dependency parsing could propagate to downstream task and therefore any
advance in this field could lead to major improvement in NLP tasks. There are
two main approaches to dependency parsing.

The first approach is transition based which has incremental local inference
and involves using datastructures such as buffer and stack (Nivre, 2008),(Buys
& Blunsom, 2015). This approach has the limitation of resolving relatively
short sentences and is a trade-off between speed and accuracy.

The second approach is graph based and can handle any long sentence but the
inference time has usually long time complexity.

There are many technical issues for improving state of the art dependency
parsers. Thus these research directions include:

  1. 1.

nonprojective cases

  2. 2.

high order features

  3. 3.

faster inference algorithms

  4. 4.

training data scarcity and the need for few shot learning

  5. 5.

span-span modeling

  6. 6.

reranking

The amount of nonprojective examples of dependency parsing varies from one
language to another. At inference time, Eisner algorithm which is a bottom up
CKY-like algorithm can not resolve nonprojective cases. For nonprojective
parsing, many algorithms based on maximum spanning tree are used such as
(Zmigrod et al., 2021),(McDonald et al., 2005),(McDonald et al., 2006),(Levi
et al., 2015) or leveraging maximum subgraph parsing for the case of nontrees
in semantic dependency parsing as is shown in (Kuhlmann & Jonsson, 2015). An
alternative approach to finding maximum spanning tree is to formulate it as an
integer linear programming(ILP) like (Riedel et al., 2006). One advantage of
ILP formulation is the capability to model many constraint like each verb has
a single subject in a direct way. The other advantage is that it can be used
for nonprojective cases (Riedel & Clarke, 2006).

Most articles in the literature are devoted to first order parsing (Dozat &
Manning, 2016) which only use one edge and node as sources of input features
and neglect the richness of language structure. Although (Li et al., 2022)
uses graph neural networks(GNN) and creates the graph dynamically, but it
still does not model high order features. A good way to leverage GNN to
consider higher order features like grandparents, grandchildren and siblings,
is described in (Ji et al., 2019) which recursively aggregates the neighbors’
information and the graph at each layer appears as a soft parse tree, and
finally it uses a MST algorithm for decoding. The novelty of this GNN is that
it represents both the head representation of each node, and dependent
representation of each node. The drawback is that the number of parameters to
learn is quite large and it also suffers from the curse of dimensionality and
therefore needs many data to train efficiently in this high dimensional vector
space. An interesting idea to get around this difficulty in GNN, is
considering each node as an edge in the dependency structure, which is
explained in (Yang & Tu, 2022). An alternative for GNN is stack-pointer
networks in (Ma et al., 2018) and siblings and grandchildren features have
been modeled. Although this model is fast but has all the limitations of deep
learning such as interpretability, being data hungry and struggling with the
curse of dimensionality.

One way to generalize to high order features is using ILP and see it as a
structural prediction problem (Martins et al., 2009b). In order to unify all
approaches, graphical models are used as a promising paradigm as is shown in
(Niculae & Martins, 2020). Prior knowledge could be encoded as hard constraint
(Martins et al., 2009a) and keeps polynomial number of constraints in general.
(Martins et al., 2009a) uses nonlocal(high order) features. Another idea that
can be combined is dual decomposition which is inspired by optimization
(Martins, Smith, Figueiredo & Aguiar, 2011),(Martins, Figueiredor, Aguiar,
Smith & Xing, 2011). A good approach to unify the loopy belief propagation
parser of (Smith & Eisner, 2008) and the relaxed linear program (Martins et
al., 2009a) is explained in (Martins et al., 2010) that considers model
assumptions in a factor graph. (Gormley et al., 2015) considers Feed-forward
topology of inference as a differentiable circuit and considers high order
interactions in a factor graph. (Gormley et al., 2015) models each potential
function as a loglinear form.

Although high order features are crucial to obtain state of the art models for
dependency parsing but there is another factor which is even more important
and is described in (Gan et al., 2021). The basic idea is to measure the
relation between the spans in contrast to measuring the relations between
words in classical dependency parsing. This approach is a proper
generalization since each word is always a span of length one, and subspans
could be evaluated from spans recursively which could be considered as a
dynamic programming paradigm.

##  2 Related Works

An inspiring and natural approach to high order dependency parsing is
described in the seminal work of (Smith & Eisner, 2008) that formulates it as
an approximate learning and inference over a graphical model and the global
constraints are encoded inside the model and Loopy Belief Propagation(LBP) is
a simple approximation that is used for that. (Smith & Eisner, 2008)
incrementally adjusts the numerical edge weights that are fed to a fast first-
order parser. One of the main difficulties is satisfy hard constraints such as
tree constraint which ensures the resulting graph is a tree. The probability
distribution of all configurations(all assigments $\mathcal{A}$) is defined by
the following Markov random field(MRF)

$p(\mathcal{A})=\frac{1}{\mathcal{Z}}\underset{m}{\prod}F_{m}(\mathcal{A})$
(2.1)

where $F_{m}$ is the m-th factor function which could be unary, binary,
ternary, or global. From a different classification angle, these factors could
be either hard or soft. A hard factor has a value 0 on violated constraint
parses, acting as a constraint to rule them out such as TREE constraint which
ensures the final graph is a tree or even a harder constraint such as PTREE
which ensures trees to be projective. Another important hard constraint is
EXACTLY1 which does not allow any word to have more than one parent. Soft
factors in (2.1) can easily be modeled by the following loglinear model:

$F_{m}(\mathcal{A})=\exp\sum_{h\in
features(F_{m})}\theta_{h}f_{h}(\mathcal{A},W,m)$ (2.2)

Nine types of soft constraints and seven hard constraints are described in
(Smith & Eisner, 2008). An interesting experimental and combinatorial
exploration is discovering which set of soft and hard constraints are
sufficient for a reasonable accuracy which experimentally measures the degree
of sensitivity of each of these constraints to the final accuracy.

The main difficulty in training this model is the fact that the normalizing
constant in the denominator depends implicitly on the learning parameters and
therefore can not be neglected but Belief Propagation(BP) provides an estimate
of that marginal distribution. Thus the gradient of the normalizing constant
can easily be computed as follows.

$\nabla_{\theta}\log\mathcal{Z}=\sum_{m}\mathbb{E}_{p(\mathcal{A})}[\nabla_{\theta}F_{m}(\mathcal{A})]$
(2.3)

(Gormley et al., 2015) considers approximations and parsing in (Smith &
Eisner, 2008) as a differentiable circuit to improve accuracy. It uses a
different objective function which is based on the $L2$ distance between the
approximate marginals and the gold marginals.

##  3 Main Results

###  3.1 Terminology

Let $W=W_{0},\ldots,W_{n}$ denote the input sentence where $W_{0}$ is the
root. The corresponding part of speech(POS) tags are $T_{1},\ldots,T_{n}$.
There are $O(n^{2})$ links in the dependency parse that can be enumerated by
$\\{L_{ij}:0\leq i\leq n,1\leq j\leq n\\}$

###  3.2 Transferring Neural Potentials

By borrowing from (Dozat & Manning, 2016), the scores can easily be calculated
as follows:

$\begin{split}h_{i}^{(arc-dep)}&=MLP^{(arc-dep)}(r_{i})\\\ h_{j}^{(arc-
head)}&=MLP^{(arc-head)}(r_{j})\\\ s_{i}^{(arc)}&=H^{(arc-
head)}U^{(1)}h_{i}^{(arc-dep)}+H^{(arc-head)}u^{(2)}\end{split}$ (3.1)

Unary and binary potentials could be defined as follows:

$\begin{split}\psi_{Y_{k}}&=\exp{s_{i(k)j(k)}}\\\
\psi_{Y_{k},Y_{k^{\prime}}}&=\psi_{Y_{k}}+\psi_{Y_{k^{\prime}}}+\phi_{Y_{k},Y_{k^{\prime}}}\end{split}$
(3.2)

where $i(k)$ and $j(k)$ is the simple lookup table mapping from the actual
dependency graph to the graphical model. Please note that the score of the
labels are also defined similarly. The best way to understand the idea of
transferring neural potentials is to imagine that the cheap and fast first
order parser is the baseline and the goal is to perturb these edge scores to
be adjusted to the global constraints through high order features. There are
two paradigms that can resolve this issue. The first paradigm says that the
weights of the first order parser does not receive any feedback from high
order features and the misalignment is modeled by a new term
$\phi_{Y_{k},Y_{k^{\prime}}}$ that is small only for cases that high order
features do not have any conflict with first order features and we call it a
perfect alignment case. The second paradigm couples first order with high
order features in a bidirectional way and allows to change the scores of first
order parser by end to end training and the error is propagated all the way
downstream to influence and tune the first order parser. The first paradigm
can be used as a warm start of the second paradigm to increase the speed of
training process since the initial weights are at a reasonable space and just
a perturbation of it could satisfy the high order dependency constraints. The
present paper assumes that $\psi_{Y_{k}}+\psi_{Y_{k^{\prime}}}$ can
sufficiently model the interactions of edges and there is no need to model the
mutual interaction $\phi_{Y_{k},Y_{k^{\prime}}}$ explicitly, since the model
is trained end to end and all potentials are based on neural networks and the
mutual interaction is implicitly considered. There are two main approaches to
inference for the best parse. The first one is based on sum-product algorithm
also known as belief propagation algorithm. After calculating the beliefs from
final message passing iteration, the marginal probability can be approximated.
This should be done for all variable nodes of the factor graph to get all
parts of the parse. The second approach simultaneously maximizes the objective
by finding the best assignment which is also called the MAP assignment task.
The second approach is mathematically richer since the integrality gap can be
evaluated in contrast to loopy belief propagation that only can hope to reach
the convergence and the evaluation is hard. These two approaches are explained
here:

####  3.2.1 Loopy Belief Propagation

After sending messages iteratively from variables, $y_{i}$, to factors,
$\alpha$ and, from factors to variables, the algorithm will eventually
converge:

$\begin{split}m^{(t)}_{i\rightarrow\alpha}(y_{i})&\propto\prod_{\beta\in\mathcal{N}(i)\backslash\alpha}m_{\beta\rightarrow
i}^{(t-1)}(y_{i})\\\ m^{(t)}_{\alpha\rightarrow
i}(y_{i})&\propto\sum_{y_{\alpha}\sim
y_{i}}\psi_{\alpha}(y_{\alpha})\prod_{j\in\mathcal{N}(\alpha)\backslash
i}m_{j\rightarrow\alpha}^{(t-1)}(y_{i})\end{split}$ (3.3)

where $\mathcal{N}(i)$ and $\mathcal{N}(\alpha)$ are the neighbors of $y_{i}$
and $alpha$ respectively. Beliefs at each variable and factor are computed as
follows:

$\begin{split}b_{i}(y_{i})&\propto\prod_{\alpha\in\mathcal{N}(i)}m_{\alpha\rightarrow
i}^{(t_{max})}(y_{i})\\\
b_{\alpha}(y_{\alpha})&\propto\psi_{\alpha}(y_{\alpha})\prod_{i\in\mathcal{N}(\alpha)}m_{i\rightarrow\alpha}^{(t_{max})}(y_{i})\end{split}$
(3.4)

This approach is used in (Gormley et al., 2015) in the inference step.

####  3.2.2 MAP Inference

In the present work, this approach is chosen since it is fast, parallelizable
and has a rich mathematical analysis. Linear programming(LP) relaxation is
used to solve MAP inference as is explained in (Jaakkola & Sontag, 2010). The
factor graph has an equivalent Markov random field(MRF) and thus the objective
is as follows:

$\begin{split}\text{MAP}(\theta)&=\underset{\mu}{\max}\sum_{i\in
V}\sum_{x_{i}}\theta_{i}(x_{i})\mu_{i}(x_{i})+\sum_{ij\in
E}\sum_{x_{i},x_{j}}\theta_{ij}(x_{i},x_{j})\mu_{ij}(x_{i},x_{j})\\\
&=\underset{\mu}{\max}\ \theta.\mu\end{split}$ (3.5)

subject to :

$\begin{split}\mu_{i}(x_{i})&\in\\{0,1\\}\ \forall i\in V,x_{i}\\\
\underset{x_{i}}{\sum}\mu_{i}(x_{i})&=1\ \forall i\in V\\\
\mu_{i}(x_{i})&=\underset{x_{j}}{\sum}\mu_{ij}(x_{i},x_{j})\ \forall ij\in
E,x_{i}\\\ \mu_{j}(x_{j})&=\underset{x_{i}}{\sum}\mu_{ij}(x_{i},x_{j})\
\forall ij\in E,x_{j}\end{split}$ (3.6)

where $\theta_{i}$ and $\theta_{ij}$ are unary and binary potentials
respectively. This is a pairwise relaxation. We can tighten the relaxation by
enforcing the joint consistency of edges in a cluster of variables using the
framework of lift-and-project methods but is out of the scope of the present
paper since a fast algorithm is more preferred to a highly accurate algorithm.
The lifting refers to introducing the new high level variables and the
projection refers to projecting to the original variables. An alternative
framework is to use cutting plane algorithms. When using these higher order
methods, the number of constraints and the variables, grows exponentially in
the size of the clusters considered and is therefore prohibitive. The
constraints in (3.6) can be generalized to the cluster based constraint as is
done in (Batra et al., 2011),(Sontag et al., 2008) to have tighter relaxation.
A different LP relaxation for MAP assignment problem is by reducing it to an
instance of a Bipartite Multi-cut problem as is shown in (J. Reddi et al.,
2010). A good survey of all LP relaxations for MAP inference in discrete
Markov random fields is described in (Kannan et al., 2019). A cutting-plane
algorithm is used in the present paper as follows: After solving the pairwise
LP relaxation, there are two cases. The first case is that the solution is
integer, the MAP assignment is done, and algorithm is terminated. To handle
the second case, one can add a valid constraint to the relaxation. Valid
constraint is a constraint that does not cut off any of the integral vertices.
Solving (3.6) is computationally expensive and is not efficient. Thus, a
natural approach is to use dual decomposition which is explained in (Martins,
Figueiredor, Aguiar, Smith & Xing, 2011), (Koo et al., 2010),(Martins, Smith,
Figueiredo & Aguiar, 2011). Block coordinate descent is used for dual
decomposition in (Belanger et al., 2014) while the present paper used ADMM as
is leveraged in (Martins, Smith, Figueiredo & Aguiar, 2011).

####  3.2.3 Dual Decomposition

Following the ADMM approach of (Martins, Smith, Figueiredo & Aguiar, 2011) to
dual decomposition, first the primal problem is defined as follows:

$\begin{split}P:\underset{z_{s}\in y_{s}}{\max}\sum_{s=1}^{S}f_{s}(z_{s})\\\
<u(r)>_{r\in R}\in\mathbb{R}^{|R|}\\\ s.t.\ z_{s}(r)=u(r)\ \forall
s,r\in\bar{R}_{s}\end{split}$ (3.7)

After doing relaxation and writing the dual form, the master problem is:

$\begin{split}D:\underset{\lambda=<\lambda_{1},\ldots,\lambda_{S}>}{\min}\sum_{s=1}^{S}g_{s}(\lambda_{s})\\\
s.t.\sum_{s:r\in\bar{R_{s}}}\lambda_{s}(r)=0\ \forall r\in R\end{split}$ (3.8)

where $g_{s}(\lambda_{s})$ are the solution to the following slaves

$\underset{z_{s}\in Z_{s}}{\max}\
f_{s}(z_{s})+\sum_{r\in\bar{R_{s}}}\lambda_{s}(r)z_{s}(r)-\frac{\rho}{2}\sum_{r\in\bar{R}_{s}}(z_{s}(r)-u^{t}(r))^{2}$
(3.9)

Since the scores $f_{s}(z_{s})$ in (3.9) is modeled by a linear form like
$f_{s}(z_{s})=\sum_{r\in R_{s}}\theta_{s}(r)z_{s}(r)$, the slaves can be
written as:

$\underset{z_{s}\in Z_{s}}{\max}\
\sum_{r\in\bar{R_{s}}}(\theta_{s}(r)+\lambda_{s}(r))z_{s}(r)-\frac{\rho}{2}\sum_{r\in\bar{R}_{s}}(z_{s}(r)-u^{t}(r))^{2}$
(3.10)

Note that $\theta_{s}(r)$ in (3.10) are neural potentials that are estimated
from the deep learning module and these coefficients vary at each iteration of
the overall circuit. To solve (3.10) using a generic quadratic solver, it is
written in the following form:

$\underset{z_{s}\in Z_{s}}{\max}\
\sum_{r\in\bar{R_{s}}}(\theta_{s}(r)+\lambda_{s}(r)+\rho
u^{t}(r))z_{s}(r)-\frac{\rho}{2}\sum_{r\in\bar{R}_{s}}(z_{s}(r))^{2}$ (3.11)

The Lagrange variables can be updated as follows:

$\lambda_{s}^{t+1}(r)=\lambda_{s}^{t}(r)-\eta_{t}(z_{s}^{t+1}(r)-u^{t+1}(r))$
(3.12)

where $\eta_{t}$ is the step size. Applying ADMM algorithm, u has a closed
form solution as a simple average which is obtained by projected subgradient
method:

$u^{t+1}(r)=\frac{1}{\delta(r)}\sum_{s:r\in\bar{R_{s}}}z_{s}^{t+1}(r)$ (3.13)

where $\delta(r)$ is the cardinality of set $\\{s:r\in R_{s}\\}$. The loop
iterates until primal and dual residuals defined in (Martins, Smith,
Figueiredo & Aguiar, 2011) violate the constraint. To fully perceive the
details of these variables, consider a sentence with 5 tokens as follows:
$w_{1},\ldots,w_{5}$

Figure 1: right and left minimal dependencies

$z^{gp}_{3f}=\\{y_{34},y_{45}\\}$

The minimal dependencies are defined as all second order dependencies that are
either consecutive siblings or grand parents and are shown in Figure 1 with
constraints that are shown in Figure 2 .

Figure 2: forward constraints

Figure 3 shows the factor graph generated for 5 token sentences which includes
6 constraints and 7 overlapping basic components. Nots that only two types of
higher order constraints namely grandparent and consecutive siblings are used
in the present paper since there is always a tradeoff between computational
complexity and exactness of the solutions. These two types of constraints are
more essential than the rest of them and have more impact in any selection
process.

Figure 3: factor graph

In order to connect the dual decomposition to the first order deep learning
model, a mapping is defined that assigns a score to each of the components.
The weighted combinations of these first order scores supports how solution of
the $z_{s}$ variables can shape the global dependency parsing graph:

$f_{s}(z_{s})=\sum_{r}z_{(s,r)}\theta(s,r)$ (3.14)

Once the $z$ variables are solved, the dependency graph can be read. Note that
the number of components of each slave is fixed but each basic component can
be connected to any number of constraints. The equality constraint in equation
(3.7) ensures the consistency of the selection.

###  3.3 Training And Prediction

By drawing inspiration from (Gormley et al., 2015), everything is trained end
to end like a circuit and the inference mechanism of the factor graph is
coupled with the neural estimation of edge scores. A drawback of the present
approach like any other deep learning model is the lack of sufficient training
data since deep learning models are data hungry and suffer from curse of
dimensionality. There is a compromise between speed of convergence and the
accuracy of modeling. This is the reason that very few slaves(constraints) are
considered for high order modeling. The more constraint added to the model,
the longer it takes to converge. Minimum Bayes risk(MBR) is used to produce a
tree that minimizes the expected loss as follows:

$\begin{split}h_{\theta}(x)&=\underset{\hat{y}}{\operatorname*{arg\,min}}\mathbb{E}_{y\sim
p_{\theta}(.|x)}l(\hat{y},y)\\\
&=\underset{\hat{y}}{\operatorname*{arg\,max}}\sum_{i:\hat{y_{i}}=ON}p_{\theta}(y_{i}=ON|x)\end{split}$
(3.15)

At test time, maximum spanning tree automatically ensures that the resulting
graph is a tree.

Algorithm 1 outputs dependency parsing tree

1:Input: batch of sentences
2:repeat
3: calculate word embedding
4: calculate the scores using (3.1)
5: calculate the neural potentials using first order estimation (3.2)
6: repeat
7: for each slave $s=1,\ldots,S$ do
8: make an update for slave $z_{s}$ using (3.11)
9: end for
10: update u using (3.13)
11: update $\lambda$ using (3.12)
12: $t\leftarrow t+1$
13: until primal and dual residuals are below a threshold
14: round $u,z,\lambda$
15: backpropagate the loss in (3.15) to adjust the neural scores
16: until maxIter
17: if not tree
18: use maximum spanning tree algorithm
19: return the dependency parsing tree

###  3.4 Experiments

Universal dependency dataset for English language is used for all of the
experiments in the present paper. The maximum length of the sentence is 71
tokens but at most 60 token sentences are used for training since few
datapoints for longer sentences are not enough for an adequate training and
makes some noise.

Table 1: different experiments for dependency parsing experiment | opt | epochs | batch size | maxseq | highorder | accuracy
---|---|---|---|---|---|---
1 | Adam | 10 | 5 | 20 | False | 93.2
2 | Adam | 10 | 5 | 40 | False | 91.2
3 | SGD | 10 | 5 | 60 | False | 90.1
4 | Adam | 10 | 20 | 60 | False | 89.7
5 | Adam | 10 | 5 | 60 | False | 88.6
6 | Adam | 10 | 5 | 60 | True | 65.3
7 | SGD | 10 | 5 | 60 | True | 66.2

When highorder in Table reffig:experiments is True, it means that the high
order dependency parsing is used while when it is False, it corresponds to the
first order modeling that uses biaffine attention and factor graph and dual
decomposition algorithm are not involved in training. One reason that the
results of highorder in Table 1 are lower than the first order counterpart, is
that very few slaves are used in these experiments for faster convergence.
Even the backward constraint slaves are not used and there is always a
tradeoff between speed and accuracy.

##  4 Conclusion

The present algorithm combines first order information with high order
information to represent a richer feature representation to obtain relations
and labels for dependency parsing. The contribution of the present paper is on
joining deep biaffine scores in traditional deep learning with high order
constraints that are best represented by graphical models. By combining the
strength of deep learning and graphical model inferences, a unified approach
to high order parsing is suggested. One can also analyze that the high order
parser is a perturbation of first order parser. The analysis could also be
used for further investigation on which structures in languages have high
deviations between first order and high order parsers.

## References

  * (1)
  * Batra et al. (2011) Batra, D., Nowozin, S. & Kohli, P. (2011), ‘Tighter relaxations for map-mrf inference: A local primal-dual gap based separation algorithm.’, Journal of Machine Learning Research - Proceedings Track 15, 146–154.
  * Belanger et al. (2014) Belanger, D., Passos, A., Riedel, S. & McCallum, A. (2014), ‘Message passing for soft constraint dual decomposition’, Uncertainty in Artificial Intelligence - Proceedings of the 30th Conference, UAI 2014 pp. 62–71.
  * Buys & Blunsom (2015) Buys, J. & Blunsom, P. (2015), ‘Generative incremental dependency parsing with neural networks’, 2, 863–869.
  * Dozat & Manning (2016) Dozat, T. & Manning, C. (2016), ‘Deep biaffine attention for neural dependency parsing’.
  * Gan et al. (2021) Gan, L., Meng, Y., Kuang, K., Sun, X., Fan, C., Wu, F. & Li, J. (2021), ‘Dependency parsing as mrc-based span-span prediction’.
  * Gormley et al. (2015) Gormley, M. R., Dredze, M. & Eisner, J. (2015), ‘Approximation-aware dependency parsing by belief propagation’, CoRR abs/1508.02375.
  * J. Reddi et al. (2010) J. Reddi, S., Sarawagi, S. & Vishwanathan, S. (2010), Map estimation in binary mrfs via bipartite multi-cuts., pp. 955–963.
  * Jaakkola & Sontag (2010) Jaakkola, T. & Sontag, D. A. (2010), Approximate inference in graphical models using lp relaxations.
  * Ji et al. (2019) Ji, T., Wu, Y. & Lan, M. (2019), Graph-based dependency parsing with graph neural networks, in ‘Annual Meeting of the Association for Computational Linguistics’.
  * Kannan et al. (2019) Kannan, H., Komodakis, N. & Paragios, N. (2019), Tighter continuous relaxations for MAP inference in discrete MRFs: A survey, pp. 351–400.
  * Koo et al. (2010) Koo, T., Rush, A., Collins, M., Jaakkola, T. & Sontag, D. (2010), Dual decomposition for parsing with non-projective head automata., pp. 1288–1298.
  * Kuhlmann & Jonsson (2015) Kuhlmann, M. & Jonsson, P. (2015), ‘Parsing to noncrossing dependency graphs’, Transactions of the Association for Computational Linguistics 3, 559–570.
  * Levi et al. (2015) Levi, E., Reichart, R. & Rappoport, A. (2015), ‘Edge-linear first-order dependency parsing with undirected minimum spanning tree inference’.
  * Li et al. (2022) Li, B., Gao, M., Fan, Y., Sataer, Y., Gao, Z. & Gui, Y. (2022), Dyngl-sdp: Dynamic graph learning for semantic dependency parsing, in ‘Proceedings of the 29th International Conference on Computational Linguistics’, International Committee on Computational Linguistics, Gyeongju, Republic of Korea.
  * Ma et al. (2018) Ma, X., Hu, Z., Liu, J., Peng, N., Neubig, G. & Hovy, E. H. (2018), Stack-pointer networks for dependency parsing, in ‘Annual Meeting of the Association for Computational Linguistics’.
  * Martins, Figueiredor, Aguiar, Smith & Xing (2011) Martins, A. F. T., Figueiredor, M. A. T., Aguiar, P. M. Q., Smith, N. A. & Xing, E. P. (2011), An augmented lagrangian approach to constrained map inference, in ‘Proceedings of the 28th International Conference on International Conference on Machine Learning’, ICML’11, Omnipress, Madison, WI, USA, p. 169–176.
  * Martins et al. (2009a) Martins, A. F. T., Smith, N. A. & Xing, E. P. (2009a), Concise integer linear programming formulations for dependency parsing, in ‘Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 1 - Volume 1’, ACL ’09, Association for Computational Linguistics, USA, p. 342–350.
  * Martins et al. (2009b) Martins, A. F. T., Smith, N. A. & Xing, E. P. (2009b), Polyhedral outer approximations with application to natural language parsing, in ‘Proceedings of the 26th Annual International Conference on Machine Learning’, ICML ’09, Association for Computing Machinery, New York, NY, USA, p. 713–720.
  * Martins, Smith, Figueiredo & Aguiar (2011) Martins, A., Smith, N., Figueiredo, M. & Aguiar, P. (2011), Dual decomposition with many overlapping components., pp. 238–249.
  * Martins et al. (2010) Martins, A., Smith, N., Xing, E., Aguiar, P. & Figueiredo, M. (2010), Turbo parsers: Dependency parsing by approximate variational inference, pp. 34–44.
  * McDonald et al. (2005) McDonald, R., Pereira, F., Ribarov, K. & Hajič, J. (2005), Non-projective dependency parsing using spanning tree algorithms, in ‘Proceedings of the Conference on Human Language Technology and Empirical Methods in Natural Language Processing’, HLT ’05, Association for Computational Linguistics, USA, p. 523–530.
  * McDonald et al. (2006) McDonald, R. T., Crammer, K. & Pereira, F. (2006), Spanning tree methods for discriminative training of dependency parsers.
  * Niculae & Martins (2020) Niculae, V. & Martins, A. (2020), Lp-sparsemap: Differentiable relaxed optimization for sparse structured prediction, in H. D. III & A. Singh, eds, ‘Proceedings of the 37th International Conference on Machine Learning’, Vol. 119 of Proceedings of Machine Learning Research, PMLR, pp. 7348–7359.
  * Nivre (2008) Nivre, J. (2008), ‘Algorithms for deterministic incremental dependency parsing’, Comput. Linguist. 34(4), 513–553.
  * Riedel et al. (2006) Riedel, S., cakici, R. & Meza-Ruiz, I. (2006), Multi-lingual dependency parsing with incremental integer linear programming, in ‘Proceedings of the Tenth Conference on Computational Natural Language Learning’, CoNLL-X ’06, Association for Computational Linguistics, USA, p. 226–230.
  * Riedel & Clarke (2006) Riedel, S. & Clarke, J. (2006), Incremental integer linear programming for non-projective dependency parsing, in ‘Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing’, EMNLP ’06, Association for Computational Linguistics, USA, p. 129–137.
  * Smith & Eisner (2008) Smith, D. & Eisner, J. (2008), Dependency parsing by belief propagation, pp. 145–156.
  * Sontag et al. (2008) Sontag, D., Globerson, A. & Jaakkola, T. (2008), Clusters and coarse partitions in lp relaxations, in ‘Proceedings of the 21st International Conference on Neural Information Processing Systems’, NIPS’08, Curran Associates Inc., Red Hook, NY, USA, p. 1537–1544.
  * Yang & Tu (2022) Yang, S. & Tu, K. (2022), Semantic dependency parsing with edge gnns, in ‘Findings of the Association for Computational Linguistics: EMNLP 2022’, Association for Computational Linguistics, Abu Dhabi, United Arab Emirates, pp. 6096–6102.
  * Zmigrod et al. (2021) Zmigrod, R., Vieira, T. & Cotterell, R. (2021), On finding the k-best non-projective dependency trees, in ‘Annual Meeting of the Association for Computational Linguistics’.


# LMExplainer: a Knowledge-Enhanced Explainer for Language Models

Zichen Chen, Ambuj K Singh, Misha Sra
University of California, Santa Barbara
{zichen_chen, ambuj<EMAIL_ADDRESS>
###### Abstract

Large language models (LMs) such as GPT-4 are very powerful and can process
different kinds of natural language processing (NLP) tasks. However, it can be
difficult to interpret the results due to the multi-layer nonlinear model
structure and millions of parameters. Lack of understanding of how the model
works can make the model unreliable and dangerous for everyday users in real-
world scenarios. Most recent works exploit the weights of attention to provide
explanations for model predictions. However, pure attention-based explanation
is unable to support the growing complexity of the models, and cannot reason
about their decision-making processes. Thus, we propose LMExplainer, a
knowledge-enhanced interpretation module for language models that can provide
human-understandable explanations. We use a knowledge graph (KG) and a graph
attention neural network to extract the key decision signals of the LM. We
further explore whether interpretation can also help AI understand the task
better. Our experimental results show that LMExplainer outperforms existing
LM+KG methods on CommonsenseQA and OpenBookQA. We also compare the explanation
results with generated explanation methods and human-annotated results. The
comparison shows our method can provide more comprehensive and clearer
explanations. LMExplainer demonstrates the potential to enhance model
performance and furnish explanations for the reasoning processes of models in
natural language.

##  1 Introduction

Pre-trained language models (PLMs) have recently gained significant attention
due to their impressive performance on various natural language processing
tasks (Brown et al., 2020; Liu et al., 2023; Wei et al., ; Zhou et al., 2022).
PLMs are a type of Artificial Intelligence (AI) that is trained on large
amounts of data in order to perform various natural language processing tasks.
These tasks can include language translation (Conneau and Lample, 2019), text
generation (Mireshghallah et al., 2022), and text classification (Raffel et
al., 2020), among others. One of the main advantages of PLMs is their ability
to capture the nuances and the complexity of human languages, allowing them to
achieve state-of-the-art performance on different tasks (Li et al., 2022).

However, a major limitation of PLMs is the lack of interpretability (Meng et
al., 2022), as they are unable to provide explanations for their decision-
making processes. The decision-making process is often referred to as a “black
box”. PLMs use techniques such as attention mechanisms, which allow the model
to focus on specific parts of the input data when making decisions (Vaswani et
al., 2017; Devlin et al., 2019; Liu et al., 2019a). These mechanisms are not
transparent and are difficult for humans to interpret (Jain and Wallace,
2019). For example, model’s embedding may capture relationships and meanings
that are not immediately apparent to humans due to transmission through
millions of neurons. This lack of interpretability poses a challenge in
critical domains, such as healthcare (Loh et al., 2022) and online education
(Zytek et al., 2022), as it limits the trust that users can place in the
inference made by the models. In addition to explainability, improving model
interpretability can help address issues of fairness, privacy, and safety.
Therefore, exploring methods that explain the behaviors of PLMs can help
overcome the black-box nature of neural networks.

Many recent approaches have focused on providing model-intrinsic and post-hoc
methods to address the challenge of model explanations (Ribeiro et al., 2016;
Shrikumar et al., 2017; Štrumbelj and Kononenko, 2014; Ying et al., 2019;
Tigas et al., 2022; Situ et al., 2021). Model-intrinsic methods consider
models that are considered interpretable due to their simple structure, such
as linear models. Post-hoc methods provide explanations that are obtained
after the model training, such as feature selection. Interpreting and
explaining the decision-making process of PLMs is a challenging task. To
overcome this challenge, Thorne et al. propose the Multiple Instance Learning
method for generating token-level explanations for natural language inference
(NLI), which is based on the attention matrix of a neural model. Chen et al.
utilize counterfactual examples to generate contrastive explanations. Zhan et
al. incorporate structured information from external knowledge to explain the
model. However, these works only provide simple textual explanations,
neglecting the reasoning process and thereby, do not provide a complete
understanding of the model’s reasoning process. We argue that in order to
provide more informative explanations that allow humans to understand and
trust the model in crucial domains, it is necessary to not only provide
insight into how the model reasons but also offer textual explanations. By
doing so, we can help humans to gain a more comprehensive understanding of the
decision-making process and interpret the model’s output in a human-readable
format.

In this paper, we present LMExplainer, a novel approach for explaining the
predictions made by large PLMs. Our approach uses retrieved knowledge and
Graph Attention Networks (GAT), and is able to provide explanations of the
rationale behind the PLM’s predictions. As an example, we apply our approach
to the task of question answering (QA). In addition to addressing the
challenge of explaining the decision-making process, we also investigate the
potential impact of interpretation on model performance. Specifically, we
explore the potential of using explanations to serve a dual purpose: helping
humans in comprehending the model, and enhancing the model’s understanding of
the task at hand through interpretation during the explanation process. In
this paper, explanation refers to explaining the model’s decision-making in a
human-understandable way, while interpretation refers to understanding the
internal workings of the model.

We evaluate the effectiveness of LMExplainer on the CommonsenseQA (Talmor et
al., 2019) and OpenBookQA datasets (Mihaylov et al., 2018). Our experimental
results demonstrate that LMExplainer outperforms state-of-the-art LM+KG QA
methods on CommonsenseQA, while exhibiting competitive performance on
OpenBookQA. These outcomes indicate that our method can enhance overall
performance. Furthermore, we demonstrate that LMExplainer is capable of
providing valuable reasoning insights for humans in a natural manner,
surpassing prior explanation methods. To the best of our knowledge,
LMExplainer is the first work to utilize graph-based knowledge in generating
natural language explanations to comprehend the rationale behind model
behaviors.

##  2 Related Work

###  2.1 Post-hoc Explanation

Post-hoc explanation methods have gained significant attention in NLP research
in recent years. These include feature-based, concept-based and example-based
explanations. One popular approach of feature-based methods is called LIME
(Ribeiro et al., 2016). They generate explanations by approximating the
original model with a local sample. The model then uses the approximations to
highlight the most important features for a given prediction. LIME is extended
by Guidotti et al., which uses a decision tree classifier to approximate non-
linear models. However, they cannot guarantee that the approximations are
accurate representations of the original model due to inherent limitations.
Compared to feature-based explanations, the concept-based approaches use
single or multiple phrases to explain the model, which is more understandable
to humans. Thorne et al. generate tokens of classifiers operating on pairs of
sentences, while Yu et al. generate _aspects_ as explanations for search
results. Example-based approaches similarly explain using natural language.
Kumar and Talukdar use positive labels to generate candidate explanations,
while Chen et al. use contrastive examples in the format of “Why A not B" to
distinguish between confusing candidates. Different from prior work, we
integrate the reasoning features and concepts to explain the model’s behavior.

###  2.2 Large Language Models

Language models are used in a wide range of tasks across NLP, such as
sentiment analysis, machine translation, and question answering. Conventional
n-gram language models (Brown et al., 1992) are based on conditional
probability, and are interpretable. Recently, large language models such as
RoBERTa (Liu et al., 2019a) and GPT-4 (OpenAI, 2023) have achieved impressive
results. For example, they are able to generate coherent stories (Nye et al.,
2021), perform as AI teachers (Tack and Piech, 2022), and chat with humans
using natural language (Lin et al., 2020). However, these models are often
criticized for their lack of interpretability, which can hinder their adoption
in real-world applications. Previous interpretable frameworks (Ribeiro et al.
(2016), Sundararajan et al. (2017), Smilkov et al. (2017), Ding and Koehn
(2021)) could be applied on the large language models, but they often rely on
approximations and simplifications of the original models, which may cause
discrepancies between the model and the explanation. Swamy et al. propose a
framework to explain the language models during training, but they focus on
the comparison metrics between models and different stages of the same model,
and do not consider the effect of their methods on performance. In this paper,
we explain language models by utilizing the model reasoning process, and
evaluate the impact of the proposed method on model performance.

###  2.3 Explainability with Knowledge-Graph

Knowledge Graphs (KGs) are increasingly being used as a means to improve the
interpretability and explainability of language models (Huang et al., 2022;
Yasunaga et al., 2021; Huang et al., 2019; Liu et al., 2019b). KGs are
structured representations of knowledge, and can be used to capture the
complex semantic relationships that are difficult to represent in traditional
language models (Ji et al., 2021). We divide KG-based methods into two types:
graph-based and graph embedding-based. Graph-based explanation identifies the
most relevant paths or subgraphs in a KG that support a given model
prediction. Zhan et al. retrieve explainable reasoning paths from a KG and use
path features to predict the answers. However, their path explanations are
difficult for humans to understand. Graph embedding-based explanation uses an
encoder to embed the structure and the relations of a graph, and then the
embedding is used to generate explanations by identifying the most relevant
nodes in the KG. Yasunaga et al. integrate the KG into the model, enabling the
model to reason over structured knowledge and generate more interpretable
predictions. However, these generated explanations may not consistently and
accurately represent the model’s reasoning, and they might be challenging for
humans to comprehend because they are presented in a graph-based format. By
drawing upon the insights from prior works, we employ graph embedding as our
fundamental component to generate explanations.

##  3 Task Definition

In this paper, we aim to address two research questions related to the
interpretability of PLMs: (1) how can we provide interpretable and naturally
understandable explanations for the decision-making process of PLMs, and (2)
how does the provision of explanations impact the performance of PLMs?

We now define the task of generating reasoning-level explanation for inference
made by PLMs. As an example, we use a QA task. Given a pre-trained PLM
$f_{LM}$ with input context $z$ and predicted answer $a$, the goal is to
generate an explanation $E$ for why $f_{LM}$ makes prediction $a$ for input
context $z$. The explanation $E$ should provide insight into the reasoning
behind the prediction and be presented in a format that is understandable to
humans. This task can be written as:

$E\leftarrow Generate\\_Explanation(f_{LM},z,a)$ (1)

##  4 Approach

The LMExplainer architecture is shown in Figure 1. It consists of three main
steps: (1) key element extraction and building (Section 4.1), (2) element-
graph interpretation (Section 4.2), and (3) explanation generation (Section
4.3). In the first step, we extract the relevant elements from the input data
and the retrieved knowledge, and build an element-graph representation. In the
second step, we use a Graph Attention Network (GAT) to interpret the element-
graph and identify the reason-elements behind the model’s prediction. Finally,
in the third step, we use a prompt-based method to generate a textual
explanation of the decision-making process based on the identified reasoning
elements. Our approach in LMExplainer is flexible and applicable to a range of
PLMs, e.g., BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019a), and GPT-2
(Radford et al., ).

Figure 1: The LMExplainer architecture. Given a question context $z$ (which
includes question $q$ and the set $\mathcal{A}$ of answers), we first generate
language embeddings using a PLM. Simultaneously, it retrieves relevant
knowledge from a KG to construct a subgraph (Sec. 4.1). The language
embeddings and subgraph are then combined to obtain GNN embeddings. This
combined representation is then passed through a GAT to obtain the attention
scores (Sec. 4.2). Our attention scores serve a dual purpose. Firstly, they
weigh the importance of the GNN embeddings and are used with the language
embeddings for the final prediction (Sec. 4.4). Secondly, they are used to
generate explanations by highlighting the most important parts of the
reasoning process (Sec. 4.3).

###  4.1 Key Elements Extraction and Building

Certain key elements can significantly influence the reasoning process of
PLMs. To capture these essential elements, we use the context of input data,
represented as $z$. Each token $x$ in the input question $q$ is treated as a
content element. We use $\mathcal{A}$ to denote the set of all candidate
answers, and $a\in\mathcal{A}$ to denote one candidate answer. Referring to
Figure 1, the example demonstrates the $q$ and $a$ in “Input Context [$z$]".

We connect the context $z$ to each token $x$ and potential answer $a$ to
construct a multi-relational graph, following the approach from Yasunaga et
al.. Specifically, we incorporate external knowledge to construct the
knowledge graph. The graph is constructed by retrieving paths in the knowledge
graph from ConceptNet (Speer et al., 2017). This allows us to prepare the
essential elements that play a key role in the final results and analyze the
relations among them. In addition to connecting relations in the knowledge
graph, we also have two relationships to capture the content and the potential
answers: one is the relationship between question $q$ and context $z$, and
another is the relationship between answer $a$ and context $z$. We denote them
with $r_{z,q}$ and $r_{z,a}$, respectively. Apart from this, we integrate the
knowledge from PLMs as an instructive embedding to the KG. We denote a node in
KG with $n$. The instructive embedding is added as a probability into the node
embedding. The instructive score of $n$ can be computed as:

$\hat{n}_{score}=f_{score}(f_{enc}(n|z))$ (2)
$f_{score}=sigmoid(\text{MLP}(\cdot))$ (3)

where $f_{enc}$ is used to get the embedding from PLM, and $f_{score}$ is a
multi-layer perceptron (MLP) followed by sigmoid activation. This score is
used to instruct the KG to capture the node that is the key for reasoning.

The constructed graph typically has a large number of paths at the initial
building stage, which can lead to a large reasoning space. Since the score
$\hat{n}$ can represent the correlation between the node $n$ and context $z$,
we will utilize it to prune the graph by removing the irrelevant nodes. The
extracted graph is our key element-graph $G_{e}$.

###  4.2 Interpreting graph

Given the element-graph, we follow (Yasunaga et al., 2021) to extract the
representation for graph reasoning. The method is based on the Graph Attention
Network (GAT) (Veličković et al., 2018), and uses a Graph Convolutional
Network (GCN) approach (Kipf and Welling, 2017). The nodes in the graph
provide feature vectors, while the edges provide paths for the transfer and
aggregation of features between nodes. In each layer of a GCN, a single node
transfers its own feature vectors to all of its neighboring nodes and
aggregates the feature vectors transmitted by its neighbors as its updated
node features. This process of “pass-aggregate-update" allows both GCN and GAT
to preserve part of the structure and context of the original data through the
connections between the nodes.

Given the element-graph $G_{e}$, for any node $V_{e}$, we define its neighbor
node $s$, where $s\in\mathcal{N}(V_{e})$, and $\mathcal{N}(V_{e})$ refers to
all the neighbors of $V_{e}$. We use $V_{es}$ to denote the connecting between
node $V_{e}$ and node $s$. The updated node feature $h^{k+1}_{V_{e}}$ is then
calculated as shown in Equation 4.

$h^{k+1}_{V_{e}}=f_{\delta}(\sum\limits_{s\in\mathcal{N}_{V_{e}}\cup\left\\{V_{e}\right\\}}\alpha_{es}m_{es})+h^{k}_{V_{e}}$
(4)

where $f_{\delta}$ is a two-layer MLP, and $\alpha_{es}$ and $m_{es}$ are the
attention coefficients and the message, respectively. The node feature
$h^{k+1}_{V_{e}}$ is the updated node feature at layer $k+1$ in our GNN, while
$h^{k}_{V_{e}}$ is the node feature at layer $k$. After all the neighbor nodes
of node $V_{e}$ in the $k$-th layer have passed information to node $V_{e}$,
the feature of node $V_{e}$ in the $(k+1)$-th layer, $h^{k+1}_{V_{e}}$, is
obtained through an additive aggregation and a non-linear transformation, in
addition to the previous feature of itself. This process allows the GAT model
to capture the structural and contextual information of the graph and use it
to update the node features at each layer. The updated node features at the
final layer are then used as the representation of the graph for reasoning.

In order to filter the important connections of the graph $G_{e}$, we
incorporate attention weights $\alpha_{es}$ when aggregating the message
passing $m_{es}$ from neighbor node $s$ (Veličković et al., 2018). The message
passing $m_{es}$ is computed using the node type embedding $u_{s}$, its
feature embedding $h^{k}_{s}$, and the relation embedding $r_{es}$ between
nodes $V_{e}$ and $s$:

$m_{es}=f_{n}(h^{k}_{s},u_{s},r_{es})$ (5)

where $f_{n}$ is a linear transformation. The relation embedding $r_{es}$ is
calculated by a two-layer MLP $f_{\theta}$:

$r_{es}=f_{\theta}(\hat{r}_{es},u_{es})$ (6)

where $\hat{r}_{es}$ is a one-hot embedding for the relation connecting
$V_{e}$ and $s$, $u_{es}$ is the contacted node type embedding of $V_{e}$, and
$s$.The attention weight $\alpha_{es}$ is calculated from the query vector
$\bm{q}_{s}$ and key vector $\bm{k}_{e}$:

$\alpha_{es}=softmax(\frac{\bm{q}_{s}^{\top}\bm{k}_{e}}{\sqrt{D}})$ (7)

where $D$ refers to the feature dimension. The query and key vectors are
derived from the element-graph as follows:

$\bm{q}_{s}=f_{q}(h^{k}_{s},u_{s},\hat{n})$ (8)
$\bm{k}_{e}=f_{k}(h^{k}_{e},u_{e},\hat{n},r_{es})$ (9)

Both $f_{q}$ and $f_{k}$ are linear transformations. The attention weight
$\alpha_{es}$ captures the importance of the connection between nodes $V_{e}$
and $s$, while the message passing $m_{es}$ captures the information being
passed between the nodes. By multiplying these two quantities, the GAT model
is able to filter the important connections in the graph and use them to
update the node features. This process allows our method to discern the
underlying rationale in decision-making processes, and obtain the
interpretative embedding."

After the interpretation embedding has been obtained, we can generate
explanations by extracting the most important nodes and edges in the graph. We
use a linear transformation to map the final node features to the output
space, and use a softmax function to compute the probability distribution over
the potential answers. The most probable answers are selected as the final
answer, and the corresponding nodes and edges in the element-graph are used to
generate the textual explanations for the decision-making process of the PLMs.

###  4.3 Attention-aware Explanation Generation

In prior work, Chen et al. proposed a counterfactual-based explanation
generator that pairs input text with qualified counterfactual examples to
fine-tune the LM to generate explanations in the format of "why A and not B".
However, this approach only provides a possible explanation that may reveal
the reasoning behind the model’s decision, rather than faithfully interpreting
the inner workings of the neural network.

To generate more accurate explanations, we utilize a template-based approach
that interprets the decision-making process of the LM, as described in Section
4.2. The generated explanations are based on the final answer, corresponding
nodes of reason-elements, and edges in the interpretation. Our explanation
generator consists of two steps: key explanation element extraction and
prompt-based explanation generation.

####  4.3.1 Explanation Components Extraction

We first extract the key components that are essential to the decision-making
process of the PLMs. These key components consist of the final answer,
corresponding nodes of reason-elements, edges, and attention weights
($\alpha$) obtained in Section 4.2. The final answer, corresponding nodes, and
edges are used to trace the important explanation nodes, and the attention
weights are used to sort the nodes and select the top $w$ nodes that are most
relevant to the decision-making. Each node represents an element, so we have
$w$ most important components to interpret the explanation. We use $l$ to
represent the extracted key component. We denote the output by $E$. $E$ is a
natural language explanation.

####  4.3.2 Prompt-based Explanation Generation

We integrate the key component set $\\{l\\}$ into our prompt-based explanation
generator. The prompt-based explanation generator is based on a set of
predefined structures that guide the generation of explanations. The generator
includes input context $z$, model predicted output $y^{\prime}$, the trigger
sentence $\mathrm{Z}$, and the extracted key components $\\{l\\}$. Our
explanation generation has two stages: (1) why choose this explanation, (2)
why not choose other explanations. In the “why choose" stage, we generate the
explanation for explaining why the model chose the specific answer. The
template we used is “Q: [$z$], A: [$y^{\prime}$], T: [$\mathrm{Z}$], R:
[$\\{l\\}$]". The output $E$ of the first stage is used in the second stage to
explain why the model did not choose other answers. The template we used in
the second stage is: “P: [$E$], T: [$\mathrm{\hat{Z}}$]". We use the
GPT-3.5-turbo (Ouyang et al., 2022) model to provide a literal interpretation
of the reasoning process of the PLMs. The output of the generator is a natural
language explanation in the form of a sentence or a paragraph. As demonstrated
in Figure 1, our approach uses the top 5 reason-elements. These elements are
then passed to Stage 1, which produces the “why choose" explanation. Once the
“why choose" explanation is obtained, it is used to generate the corresponding
“why not choose" explanation.

###  4.4 Learning and Inference

In our task, each question $q$ is associated with a set of answer choices
$\mathcal{A}$, with only one being the correct answer. We utilize the
information from LM embedding and interpretation embedding. Specifically, we
define the probability of choosing an answer with $P(a|q)\propto
exp(MLP(\mathbb{H}^{LM},\mathbb{H}^{itp}))$, where
$\mathbb{H}^{itp}=h_{V}^{K}$, and $\mathbb{H}^{LM}$ is the representation
embedding from LM. We optimize the model by using the cross-entropy loss.

##  5 Experiments

###  5.1 Dataset

In our experiments, we use the CommonsenseQA (Talmor et al., 2019) and
OpenBookQA (Mihaylov et al., 2018) datasets to evaluate the performance of the
candidate approaches. CommonsenseQA consists of 12,247 questions created by
crowd-workers, which are designed to test commonsense knowledge through a
5-way multiple choice QA task. OpenBookQA consists of 5,957 questions each
requiring the task of 4-way multiple choice question answering. The questions
are designed to assess the ability of models to reason with elementary science
knowledge.

###  5.2 Baselines

Our evaluation can be divided into two parts. In the first part, we focus on
model performance. We compare LMExplainer with three sets of baseline models
on the CommonsenseQA and OpenBookQA datasets. The first set of baseline models
consists of fine-tuned language model RoBERTa-large (Liu et al., 2019a), which
demonstrates the capabilities of language models without interpretation. The
second set of baseline models includes KG augmented versions of RoBERTa-large,
using ConceptNet as the source of common sense knowledge and following the
approach in (Lin et al., 2019). The third set of baseline models is the
current state-of-the-art common sense reasoning method on CommonsenseQA, MHGRN
(Feng et al., 2020), QA-GNN (Yasunaga et al., 2021), GreaseLM (Zhang et al.,
2022). The PLM we used is from Huggingface111https://huggingface.co/.

In the second part, we evaluate LMExplainer on explanation ability. To
establish a baseline for comparison, two prior works, namely PathReasoner
(Zhan et al., 2022b) and Explanations for CommonsenseQA (Aggarwal et al.,
2021), were employed as benchmarks. These works are recognized for providing
natural and comprehensible explanations.

###  5.3 Experimental Settings

We set our GNN module to have 200 dimensions and 5 layers, where a dropout
rate of 0.2 was applied to each layer. We trained the model using the RAdam
optimizer on a single NVIDIA A100 GPU, with a training process of
approximately 3 hours. A batch size of 64 was employed during the training
process, and the learning rate for the language model and the GNN module were
set to 1e-5 and 1e-3, respectively. These settings were adopted in the first
part of the evaluation to investigate the performance of the GNN module.

We employ ConceptNet (Speer et al., 2017) as our external knowledge source for
CommonsenseQA and OpenBookQA. ConceptNet contains a vast amount of information
with 799,273 nodes and 2,487,810 edges, which provides a valuable resource for
improving the accuracy of QA systems. We extract a subgraph with a hop size of
3, and subsequently prune the obtained graph to retain only the top 200 nodes.

For explanation generation, the example prompts we used in the first stage are
Q=“Question context is", A=“The predicted choice is", T=“According to the
model top reason-elements" + K; “explain the model reasoning process with
“since…, ….", K is the reason-elements of the model. In the second stage,
P=“According to", and T=“explain why the model doesn’t choose other answers".

###  5.4 Experimental Results

We present our experimental results in Table 1 and Table 2, where the accuracy
of our proposed LM approach is evaluated on the CommonsenseQA and OpenBookQA
datasets. Our empirical findings indicate that our approach leads to
consistent improvements in performance compared to existing baseline methods
on both datasets. Specifically, the test performance on CommonsenseQA is
improved by 4.71% over the prior best LM+KG method, GreaseLM, 5.35% over the
included KG augmented LMs, and 7.12% over fine-tuned LMs. The test performance
achieves comparable results to the prior best LM+KG method, GreaseLM, on
OpenBookQA. It is worth noting that GreaseLM is specifically designed to
improve accuracy for QA tasks, while our LMExplainer model focuses on
providing explanations for the reasoning process. Despite this difference in
focus, our LMExplainer model not only offers insight into the underlying
reasoning but also demonstrates an improvement in performance. This finding
highlights the potential benefits of incorporating explainability into the
model design, as it may lead to enhanced performance in addition to fostering
a better understanding of the decision-making process.

Method | IHdev-Acc. | IHtest-Acc.
---|---|---
Baselines (Feng et al., 2020) |  |
MHGRN (2020) | 73.69% | 71.08%
KagNet (2019) | 73.47% | 69.01%
GconAttn (2019) | 72.61% | 68.59%
RGCN (2018) | 72.69% | 68.41%
RN (2017) | 74.57% | 69.08%
Baselines (our implementation) |  |
GreaseLM (2022) | 76.17% | 72.60%
QA-GNN (2021) | 74.94% | 72.36%
LMExplainer (ours) | 77.97% | 77.31%

Table 1: Performance comparison of our proposed LMExplainer model against
various baselines on Commonsense QA in-house split. Our model outperforms all
the other methods, achieving 77.97% and 77.31% accuracy on IHdev and IHtest,
respectively. As the official test is hidden, here we report the in-house Dev
(IHdev) and Test (IHtest) accuracy, following the data split of (Lin et al.,
2019).

Method | Dev-Acc. | Test-Acc.
---|---|---
Baselines (Feng et al., 2020) |  |
MHGRN (2020) | 68.10% | 66.85%
GconAttn (2019) | 64.30% | 61.90%
RGCN (2018) | 64.65% | 62.45%
RN (2017) | 67.00% | 65.20%
Baselines (our implementation) |  |
GreaseLM (2022) | 71.80% | 70.80%
QA-GNN (2021) | 63.00% | 59.80%
LMExplainer (ours) | 69.20% | 68.00%

Table 2: Performance comparison of our proposed LMExplainer model against
various baselines on OpenBookQA. Our LMExplainer model exhibits competitive
performance in relation to the top-performing model, GreaseLM. It is worth
noting that GreaseLM is specifically tailored to enhance accuracy for QA
tasks, whereas our LMExplainer model emphasizes providing explanations for the
underlying reasoning process. We use the official data splits.

###  5.5 Explanation Results

Input Questions |  | Q: What is someone doing if he or she is sitting quietly and his or her eyes are moving?
---
A. reading B. meditate C. fall asleep D. bunk E. think
Label | A. reading
| Results of Our Approach - LM
Ranked Reasoning Keys |  | 1\. quiet chattering mind, 2. not making sound, 3. mind focuses,
---
4\. glasses for people with poor eyesight, 5. war
Explanation (why choose) |  |  Since the person is described as sitting quietly and their eyes are moving,
---
it is likely that they are engaged in a visual activity.
Based on the keyword "glasses for people with poor eyesight",
option "A. reading" is the most likely answer,
as reading is a common visual activity that requires focusing one’s eyes on a
page
and is often aided by glasses for people with poor eyesight.
Explanation (why not choose) |  |  The other options, such as "B. meditate" or "C. fall asleep",
---
involve closing one’s eyes or having a still mind,
so it is unlikely that the person is doing either of those activities if their
eyes are moving.
Similarly, "D. bunk" and "E. think" do not seem to be related to the visual
activity of
having one’s eyes move while sitting quietly.
| Explanation of Others
PathReasoner (Zhan et al., 2022b) |  | quietly [related to] quiet [at location] a library [used for] reading
---
eyes [used for] reading
eyes [form of] eye [related to] glasses [used for] reading
sitting [related to] sit [related to] relaxing [has subevent] reading
Explanations for CommonsenseQA (Aggarwal et al., 2021) |  | Positive examples:
---
\- When we read, our eyes move.
\- While reading, a person sits quietly,
Negative examples:
\- While meditating, eyes don’t move, eyes are closed,
\- While sleeping, eyes are closed and they don’t move,
\- When a person bunks, he/she doesn’t sit quietly,
\- Eyes don’t move when you think about something.
Explanation:
When we read, our eyes move.
While reading, a person sits quietly.
While meditating and sleeping, eyes don’t move, eyes are closed.
When a person bunks, he/she doesn’t sit quietly.
Eyes don’t move when you think about something.

Table 3: Explanation examples of LMExplainer, PathReasoner and Explanations
for CommonsenseQA dataset. We show the different types of explanations,
including ranked reasoning keys, explanations for why choose and explanations
for why not choose. The explanations for “why choose‘", presents the model
reasoning process in a logical way, while for “not choose" shows the model why
does not choose other answers, which enhances the transparency and
interpretability of the reasoning process for humans. We use green and blue to
highlight the logical connectives and reasoning framework, respectively.

Our explanation results in Table 3 provide evidence of the reasoning ability
of our proposed LMExplainer. To further demonstrate the effectiveness of our
approach, we compare it with two other state-of-the-art methods, PathReasoner
(Zhan et al., 2022a) and Explanations for CommonsenseQA (Aggarwal et al.,
2021). PathReasoner utilizes structured information to explain the reasoning
path, while Explanations for CommonsenseQA first are created by human-
annotated explanations and then leverage a generation model to organize the
final explanation. In Table 3, we present the inputs of our model, as well as
the results of our approach, which include ranked reasoning keywords and
natural language explanations of the reasoning process. These examples
highlight the ability of our LMExplainer approach in generating comprehensive
and interpretable explanations for the models.

In comparison to PathReasoner explanations, which only provide structured
reasoning paths that are non-informative and require manual selection of a
specific path, our proposed approach not only offers a complete reasoning path
but also provides a justification for the predicted answer. As illustrated in
Table 3, PathReasoner presents four reasoning paths, including redundant
paths, making it difficult to identify the faithful reasoning path. In
contrast, our method provides a clear and concise natural language explanation
for the chosen answer (why choose explanation), which greatly enhances the
understandability and smoothness of the explanation.

The Explanations for CommonsenseQA dataset consists of human-annotated
explanations that provide highly accurate descriptions of the reasoning
process. However, as shown in Table 3, its explanations are simply a
combination of positive and negative examples provided by humans. While this
approach can generate high-quality explanations from a human perspective, it
fails to illustrate the actual reasoning process of the model. In contrast,
the explanations generated by LMExplainer are not a mere combination of
sentences but are inferred and logically derived. Our approach provides a more
comprehensive and accurate depiction of the reasoning process and improves the
overall interpretability and usefulness of the generated explanations. In
addition, the why not choose explanation explains why the model does not
choose other answers, which gives people a better understanding of the model’s
predictions and increases the transparency of the model. These results
highlight the effectiveness of quantifying the influence of tokens on
determining the reasoning process and provide a literal representation of the
information flow during the inference process. This is important because it
allows us to understand the rationale behind the decision-making process of
the LM and identify key factors that contribute to its predictions.

###  5.6 Ablation Studies

Table 4, Table 5 and Table 6 summarize the ablation study, examining the
impact of different components on the performance of our LMExplainer model. We
evaluated the effects of varying PLM sizes, knowledge components, and
interpreting components on the CommonsenseQA IHdev and IHtest sets.

####  5.6.1 PLM Size

PLM | IHdev-Acc. | IHtest-Acc.
---|---|---
RoBERTa-large (final) | 77.97% | 77.31%
RoBERTa | 66.26% | 63.01%

Table 4: Ablation study on the effect of PLM size on model accuracy.

Table 4 shows the impact of the size of PLM on our proposed method. We
evaluate the performance of two different PLM sizes: RoBERTa-large (340M
parameters) and RoBERTa (110M parameters). Our results indicate that using a
larger PLM leads to a significant improvement in performance, with an increase
of 11.71% and 14.30% on the IHdev and IHtest sets, respectively. These
findings suggest that the size of the PLM plays a critical role in the
performance of our model, and using a larger PLM can result in better
performance.

####  5.6.2 Knowledge Component

Method | IHdev-Acc. | IHtest-Acc.
---|---|---
RoBERTa-large only | 74.28% | 70.19%
RoBERTa only | 62.65% | 60.27%
w/ external knowledge (final) | 77.97% | 77.31%

Table 5: Ablation study on the effect of knowledge component on model
accuracy.

Table 5 shows the impact of the knowledge component on our method. We compare
the performance of the LM-only model with and without external knowledge from
ConceptNet. Model only means we only use the LM to predict the answer. W/
external knowledge means we incorporate the external knowledge. We observe
that incorporating external knowledge can significantly improve the accuracy
of prediction, especially on the test set. By incorporating the knowledge, the
accuracy of IHdev and IHtest is increased by at least 3.69% and 7.12%,
respectively. This shows that external knowledge plays an important role in
enhancing the reasoning ability of the model.

####  5.6.3 Interpreting Component

Method | IHdev-Acc. | IHtest-Acc.
---|---|---
RoBERTa-large w/o itp | 73.05% | 71.96%
RoBERTa w/o itp | 68.63% | 64.54%
RoBERTa-large w/ itp (final) | 77.97% | 77.31%

Table 6: Ablation study on the effect of interpreting component on model
accuracy.

In Table 6, we analyze the impact of the interpreting component on the model’s
performance. Model w/o itp indicates that the interpreting component was not
incorporated in the prediction, whereas the Model w/ itp row indicates its
presence. We observe that removing the interpreting component leads to a clear
decrease in accuracy of at least 4.92% and 5.35% on IHdev and IHtest,
respectively. Furthermore, comparing the results of RoBERTa-large only,
RoBERTa-large w/o itp, and Final, we find that the interpreting component has
a greater impact on accuracy than the other components.

The ablation highlights the positive contributions of each component of our
method. Specifically, we find that the interpreting component has a crucial
role in our method’s accuracy and generalizability on unseen questions.

##  6 Conclusion

In this paper, we propose LMExplainer, a novel model that incorporates an
interpretation module to enhance the performance of language models while also
providing clear and trustworthy explanations of the model’s reasoning. Our
model utilizes both interpretation and explanation to achieve these goals. The
explanation results are presented in a logical and comprehensive manner,
making it easier for people to understand the model’s reasoning in natural
language. Our experimental results demonstrate superior performance compared
to prior state-of-the-art works across standard datasets in the commonsense
domain. Our analysis shows that LMExplainer not only improves the model’s
performance but also provides humans with a better understanding of the model.

## References

  * Aggarwal et al. (2021) Shourya Aggarwal, Divyanshu Mandowara, Vishwajeet Agrawal, Dinesh Khandelwal, Parag Singla, and Dinesh Garg. 2021.  Explanations for CommonsenseQA: New Dataset and Models.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 3050–3065, Online. Association for Computational Linguistics.
  * Brown et al. (1992) Peter F. Brown, Vincent J. Della Pietra, Peter V. deSouza, Jenifer C. Lai, and Robert L. Mercer. 1992.  Class-based n-gram models of natural language.  _Computational Linguistics_ , 18(4):467–480.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Chen et al. (2021) Qianglong Chen, Feng Ji, Xiangji Zeng, Feng-Lin Li, Ji Zhang, Haiqing Chen, and Yin Zhang. 2021.  Kace: Generating knowledge aware contrastive explanations for natural language inference.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 2516–2527.
  * Conneau and Lample (2019) Alexis Conneau and Guillaume Lample. 2019.  Cross-lingual language model pretraining.  _Advances in neural information processing systems_ , 32.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Ding and Koehn (2021) Shuoyang Ding and Philipp Koehn. 2021.  Evaluating saliency methods for neural language models.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5034–5052, Online. Association for Computational Linguistics.
  * Feng et al. (2020) Yanlin Feng, Xinyue Chen, Bill Yuchen Lin, Peifeng Wang, Jun Yan, and Xiang Ren. 2020.  Scalable multi-hop relational reasoning for knowledge-aware question answering.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1295–1309.
  * Guidotti et al. (2018) Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Dino Pedreschi, Franco Turini, and Fosca Giannotti. 2018.  Local rule-based explanations of black box decision systems.  _arXiv preprint arXiv:1805.10820_.
  * Huang et al. (2022) Jie Huang, Kerui Zhu, Kevin Chen-Chuan Chang, Jinjun Xiong, and Wen-mei Hwu. 2022\.  DEER: Descriptive knowledge graph for explaining entity relationships.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 6686–6698, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Huang et al. (2019) Xiaowen Huang, Quan Fang, Shengsheng Qian, Jitao Sang, Yan Li, and Changsheng Xu. 2019.  Explainable interaction-driven user modeling over knowledge graph for sequential recommendation.  In _proceedings of the 27th ACM international conference on multimedia_ , pages 548–556.
  * Jain and Wallace (2019) Sarthak Jain and Byron C Wallace. 2019.  Attention is not explanation.  In _Proceedings of NAACL-HLT_ , pages 3543–3556.
  * Ji et al. (2021) Shaoxiong Ji, Shirui Pan, Erik Cambria, Pekka Marttinen, and S Yu Philip. 2021.  A survey on knowledge graphs: Representation, acquisition, and applications.  _IEEE transactions on neural networks and learning systems_ , 33(2):494–514.
  * Kipf and Welling (2017) Thomas N. Kipf and Max Welling. 2017.  Semi-supervised classification with graph convolutional networks.  In _International Conference on Learning Representations (ICLR)_.
  * Kumar and Talukdar (2020) Sawan Kumar and Partha Talukdar. 2020.  NILE : Natural language inference with faithful natural language explanations.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8730–8742, Online. Association for Computational Linguistics.
  * Li et al. (2022) Belinda Z Li, Jane Yu, Madian Khabsa, Luke Zettlemoyer, Alon Halevy, and Jacob Andreas. 2022.  Quantifying adaptability in pre-trained language models with 500 tasks.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4696–4715.
  * Lin et al. (2019) Bill Yuchen Lin, Xinyue Chen, Jamin Chen, and Xiang Ren. 2019.  Kagnet: Knowledge-aware graph networks for commonsense reasoning.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2829–2839.
  * Lin et al. (2020) Zhaojiang Lin, Peng Xu, Genta Indra Winata, Farhad Bin Siddique, Zihan Liu, Jamin Shin, and Pascale Fung. 2020.  Caire: An end-to-end empathetic chatbot.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 34, pages 13622–13623.
  * Liu et al. (2023) Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023.  Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing.  _ACM Computing Surveys_ , 55(9):1–35.
  * Liu et al. (2019a) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019a.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Liu et al. (2019b) Zhibin Liu, Zheng-Yu Niu, Hua Wu, and Haifeng Wang. 2019b.  Knowledge aware conversation generation with explainable reasoning over augmented graphs.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 1782–1792.
  * Loh et al. (2022) Hui Wen Loh, Chui Ping Ooi, Silvia Seoni, Prabal Datta Barua, Filippo Molinari, and U Rajendra Acharya. 2022.  Application of explainable artificial intelligence for healthcare: A systematic review of the last decade (2011–2022).  _Computer Methods and Programs in Biomedicine_ , page 107161.
  * Meng et al. (2022) Chuizheng Meng, Loc Trinh, Nan Xu, James Enouen, and Yan Liu. 2022.  Interpretability and fairness evaluation of deep learning models on mimic-iv dataset.  _Scientific Reports_ , 12(1):7166.
  * Mihaylov et al. (2018) Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018.  Can a suit of armor conduct electricity? a new dataset for open book question answering.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2381–2391.
  * Mireshghallah et al. (2022) Fatemehsadat Mireshghallah, Kartik Goyal, and Taylor Berg-Kirkpatrick. 2022.  Mix and match: Learning-free controllable text generationusing energy language models.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 401–415.
  * Nye et al. (2021) Maxwell Nye, Michael Tessler, Josh Tenenbaum, and Brenden M Lake. 2021.  Improving coherence and consistency in neural sequence models with dual-system, neuro-symbolic reasoning.  _Advances in Neural Information Processing Systems_ , 34:25192–25204.
  * OpenAI (2023) OpenAI. 2023.  Gpt-4 technical report.  _ArXiv_ , abs/2303.08774.
  * Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022\.  Training language models to follow instructions with human feedback.  _arXiv preprint arXiv:2203.02155_.
  * (29) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _The Journal of Machine Learning Research_ , 21(1):5485–5551.
  * Ribeiro et al. (2016) Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016.  " why should i trust you?" explaining the predictions of any classifier.  In _Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining_ , pages 1135–1144.
  * Shrikumar et al. (2017) Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. 2017.  Learning important features through propagating activation differences.  In _International conference on machine learning_ , pages 3145–3153. PMLR.
  * Situ et al. (2021) Xuelin Situ, Ingrid Zukerman, Cecile Paris, Sameen Maruf, and Gholamreza Haffari. 2021.  Learning to explain: Generating stable explanations fast.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 5340–5355, Online. Association for Computational Linguistics.
  * Smilkov et al. (2017) Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda B. Viégas, and Martin Wattenberg. 2017.  Smoothgrad: removing noise by adding noise.  _ArXiv_ , abs/1706.03825.
  * Speer et al. (2017) Robyn Speer, Joshua Chin, and Catherine Havasi. 2017.  Conceptnet 5.5: An open multilingual graph of general knowledge.  In _Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence_ , AAAI’17, page 4444–4451. AAAI Press.
  * Štrumbelj and Kononenko (2014) Erik Štrumbelj and Igor Kononenko. 2014.  Explaining prediction models and individual predictions with feature contributions.  _Knowledge and information systems_ , 41:647–665.
  * Sundararajan et al. (2017) Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017.  Axiomatic attribution for deep networks.  In _Proceedings of the 34th International Conference on Machine Learning - Volume 70_ , ICML’17, page 3319–3328. JMLR.org.
  * Swamy et al. (2021) Vinitra Swamy, Angelika Romanou, and Martin Jaggi. 2021.  Interpreting language models through knowledge graph extraction.  In _Advances in Neural Information Processing Systems (NeurIPS), 1st Workshop on eXplainable AI Approaches for Debugging and Diagnosis_.
  * Tack and Piech (2022) Anaïs Tack and Chris Piech. 2022.  The ai teacher test: Measuring the pedagogical ability of blender and gpt-3 in educational dialogues.  In _Proceedings of the 15th International Conference on Educational Data Mining_ , page 522.
  * Talmor et al. (2019) Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019.  CommonsenseQA: A question answering challenge targeting commonsense knowledge.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4149–4158, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Thorne et al. (2019) James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2019\.  Generating token-level explanations for natural language inference.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 963–969.
  * Tigas et al. (2022) Panagiotis Tigas, Yashas Annadani, Andrew Jesson, Bernhard Schölkopf, Yarin Gal, and Stefan Bauer. 2022.  Interventions, where and how? experimental design for causal models at scale.  _arXiv preprint arXiv:2203.02016_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Veličković et al. (2018) Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. 2018.  Graph Attention Networks.  _International Conference on Learning Representations_.  Accepted as poster.
  * (45) Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le.  Finetuned language models are zero-shot learners.  In _International Conference on Learning Representations_.
  * Yasunaga et al. (2021) Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Percy Liang, and Jure Leskovec. 2021.  Qa-gnn: Reasoning with language models and knowledge graphs for question answering.  _arXiv preprint arXiv:2104.06378_.
  * Ying et al. (2019) Zhitao Ying, Dylan Bourgeois, Jiaxuan You, Marinka Zitnik, and Jure Leskovec. 2019\.  Gnnexplainer: Generating explanations for graph neural networks.  _Advances in neural information processing systems_ , 32.
  * Yu et al. (2022) Puxuan Yu, Razieh Rahimi, and James Allan. 2022.  Towards explainable search results: A listwise explanation generator.  In _Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pages 669–680.
  * Zhan et al. (2022a) Xunlin Zhan, Yinya Huang, Xiao Dong, Qingxing Cao, and Xiaodan Liang. 2022a.  Pathreasoner: Explainable reasoning paths for commonsense question answering.  _Knowledge-Based Systems_ , 235:107612.
  * Zhan et al. (2022b) Xunlin Zhan, Yinya Huang, Xiao Dong, Qingxing Cao, and Xiaodan Liang. 2022b.  Pathreasoner: Explainable reasoning paths for commonsense question answering.  _Knowledge-Based Systems_ , 235:107612.
  * Zhang et al. (2022) Xikun Zhang, Antoine Bosselut, Michihiro Yasunaga, Hongyu Ren, Percy Liang, Christopher D Manning, and Jure Leskovec. 2022.  Greaselm: Graph reasoning enhanced language models for question answering.  _arXiv preprint arXiv:2201.08860_.
  * Zhou et al. (2022) Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. 2022.  Learning to prompt for vision-language models.  _International Journal of Computer Vision_ , 130(9):2337–2348.
  * Zytek et al. (2022) Alexandra Zytek, Ignacio Arnaldo, Dongyu Liu, Laure Berti-Equille, and Kalyan Veeramachaneni. 2022.  The need for interpretable features: Motivation and taxonomy.  _ACM SIGKDD Explorations Newsletter_ , 24(1):1–13.

## References

  * Aggarwal et al. (2021) Shourya Aggarwal, Divyanshu Mandowara, Vishwajeet Agrawal, Dinesh Khandelwal, Parag Singla, and Dinesh Garg. 2021.  Explanations for CommonsenseQA: New Dataset and Models.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 3050–3065, Online. Association for Computational Linguistics.
  * Brown et al. (1992) Peter F. Brown, Vincent J. Della Pietra, Peter V. deSouza, Jenifer C. Lai, and Robert L. Mercer. 1992.  Class-based n-gram models of natural language.  _Computational Linguistics_ , 18(4):467–480.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Chen et al. (2021) Qianglong Chen, Feng Ji, Xiangji Zeng, Feng-Lin Li, Ji Zhang, Haiqing Chen, and Yin Zhang. 2021.  Kace: Generating knowledge aware contrastive explanations for natural language inference.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 2516–2527.
  * Conneau and Lample (2019) Alexis Conneau and Guillaume Lample. 2019.  Cross-lingual language model pretraining.  _Advances in neural information processing systems_ , 32.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Ding and Koehn (2021) Shuoyang Ding and Philipp Koehn. 2021.  Evaluating saliency methods for neural language models.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5034–5052, Online. Association for Computational Linguistics.
  * Feng et al. (2020) Yanlin Feng, Xinyue Chen, Bill Yuchen Lin, Peifeng Wang, Jun Yan, and Xiang Ren. 2020.  Scalable multi-hop relational reasoning for knowledge-aware question answering.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1295–1309.
  * Guidotti et al. (2018) Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Dino Pedreschi, Franco Turini, and Fosca Giannotti. 2018.  Local rule-based explanations of black box decision systems.  _arXiv preprint arXiv:1805.10820_.
  * Huang et al. (2022) Jie Huang, Kerui Zhu, Kevin Chen-Chuan Chang, Jinjun Xiong, and Wen-mei Hwu. 2022\.  DEER: Descriptive knowledge graph for explaining entity relationships.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 6686–6698, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Huang et al. (2019) Xiaowen Huang, Quan Fang, Shengsheng Qian, Jitao Sang, Yan Li, and Changsheng Xu. 2019.  Explainable interaction-driven user modeling over knowledge graph for sequential recommendation.  In _proceedings of the 27th ACM international conference on multimedia_ , pages 548–556.
  * Jain and Wallace (2019) Sarthak Jain and Byron C Wallace. 2019.  Attention is not explanation.  In _Proceedings of NAACL-HLT_ , pages 3543–3556.
  * Ji et al. (2021) Shaoxiong Ji, Shirui Pan, Erik Cambria, Pekka Marttinen, and S Yu Philip. 2021.  A survey on knowledge graphs: Representation, acquisition, and applications.  _IEEE transactions on neural networks and learning systems_ , 33(2):494–514.
  * Kipf and Welling (2017) Thomas N. Kipf and Max Welling. 2017.  Semi-supervised classification with graph convolutional networks.  In _International Conference on Learning Representations (ICLR)_.
  * Kumar and Talukdar (2020) Sawan Kumar and Partha Talukdar. 2020.  NILE : Natural language inference with faithful natural language explanations.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8730–8742, Online. Association for Computational Linguistics.
  * Li et al. (2022) Belinda Z Li, Jane Yu, Madian Khabsa, Luke Zettlemoyer, Alon Halevy, and Jacob Andreas. 2022.  Quantifying adaptability in pre-trained language models with 500 tasks.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4696–4715.
  * Lin et al. (2019) Bill Yuchen Lin, Xinyue Chen, Jamin Chen, and Xiang Ren. 2019.  Kagnet: Knowledge-aware graph networks for commonsense reasoning.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2829–2839.
  * Lin et al. (2020) Zhaojiang Lin, Peng Xu, Genta Indra Winata, Farhad Bin Siddique, Zihan Liu, Jamin Shin, and Pascale Fung. 2020.  Caire: An end-to-end empathetic chatbot.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 34, pages 13622–13623.
  * Liu et al. (2023) Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023.  Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing.  _ACM Computing Surveys_ , 55(9):1–35.
  * Liu et al. (2019a) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019a.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Liu et al. (2019b) Zhibin Liu, Zheng-Yu Niu, Hua Wu, and Haifeng Wang. 2019b.  Knowledge aware conversation generation with explainable reasoning over augmented graphs.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 1782–1792.
  * Loh et al. (2022) Hui Wen Loh, Chui Ping Ooi, Silvia Seoni, Prabal Datta Barua, Filippo Molinari, and U Rajendra Acharya. 2022.  Application of explainable artificial intelligence for healthcare: A systematic review of the last decade (2011–2022).  _Computer Methods and Programs in Biomedicine_ , page 107161.
  * Meng et al. (2022) Chuizheng Meng, Loc Trinh, Nan Xu, James Enouen, and Yan Liu. 2022.  Interpretability and fairness evaluation of deep learning models on mimic-iv dataset.  _Scientific Reports_ , 12(1):7166.
  * Mihaylov et al. (2018) Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018.  Can a suit of armor conduct electricity? a new dataset for open book question answering.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2381–2391.
  * Mireshghallah et al. (2022) Fatemehsadat Mireshghallah, Kartik Goyal, and Taylor Berg-Kirkpatrick. 2022.  Mix and match: Learning-free controllable text generationusing energy language models.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 401–415.
  * Nye et al. (2021) Maxwell Nye, Michael Tessler, Josh Tenenbaum, and Brenden M Lake. 2021.  Improving coherence and consistency in neural sequence models with dual-system, neuro-symbolic reasoning.  _Advances in Neural Information Processing Systems_ , 34:25192–25204.
  * OpenAI (2023) OpenAI. 2023.  Gpt-4 technical report.  _ArXiv_ , abs/2303.08774.
  * Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022\.  Training language models to follow instructions with human feedback.  _arXiv preprint arXiv:2203.02155_.
  * (29) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _The Journal of Machine Learning Research_ , 21(1):5485–5551.
  * Ribeiro et al. (2016) Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016.  " why should i trust you?" explaining the predictions of any classifier.  In _Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining_ , pages 1135–1144.
  * Shrikumar et al. (2017) Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. 2017.  Learning important features through propagating activation differences.  In _International conference on machine learning_ , pages 3145–3153. PMLR.
  * Situ et al. (2021) Xuelin Situ, Ingrid Zukerman, Cecile Paris, Sameen Maruf, and Gholamreza Haffari. 2021.  Learning to explain: Generating stable explanations fast.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 5340–5355, Online. Association for Computational Linguistics.
  * Smilkov et al. (2017) Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda B. Viégas, and Martin Wattenberg. 2017.  Smoothgrad: removing noise by adding noise.  _ArXiv_ , abs/1706.03825.
  * Speer et al. (2017) Robyn Speer, Joshua Chin, and Catherine Havasi. 2017.  Conceptnet 5.5: An open multilingual graph of general knowledge.  In _Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence_ , AAAI’17, page 4444–4451. AAAI Press.
  * Štrumbelj and Kononenko (2014) Erik Štrumbelj and Igor Kononenko. 2014.  Explaining prediction models and individual predictions with feature contributions.  _Knowledge and information systems_ , 41:647–665.
  * Sundararajan et al. (2017) Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017.  Axiomatic attribution for deep networks.  In _Proceedings of the 34th International Conference on Machine Learning - Volume 70_ , ICML’17, page 3319–3328. JMLR.org.
  * Swamy et al. (2021) Vinitra Swamy, Angelika Romanou, and Martin Jaggi. 2021.  Interpreting language models through knowledge graph extraction.  In _Advances in Neural Information Processing Systems (NeurIPS), 1st Workshop on eXplainable AI Approaches for Debugging and Diagnosis_.
  * Tack and Piech (2022) Anaïs Tack and Chris Piech. 2022.  The ai teacher test: Measuring the pedagogical ability of blender and gpt-3 in educational dialogues.  In _Proceedings of the 15th International Conference on Educational Data Mining_ , page 522.
  * Talmor et al. (2019) Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019.  CommonsenseQA: A question answering challenge targeting commonsense knowledge.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4149–4158, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Thorne et al. (2019) James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2019\.  Generating token-level explanations for natural language inference.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 963–969.
  * Tigas et al. (2022) Panagiotis Tigas, Yashas Annadani, Andrew Jesson, Bernhard Schölkopf, Yarin Gal, and Stefan Bauer. 2022.  Interventions, where and how? experimental design for causal models at scale.  _arXiv preprint arXiv:2203.02016_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Veličković et al. (2018) Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. 2018.  Graph Attention Networks.  _International Conference on Learning Representations_.  Accepted as poster.
  * (45) Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le.  Finetuned language models are zero-shot learners.  In _International Conference on Learning Representations_.
  * Yasunaga et al. (2021) Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Percy Liang, and Jure Leskovec. 2021.  Qa-gnn: Reasoning with language models and knowledge graphs for question answering.  _arXiv preprint arXiv:2104.06378_.
  * Ying et al. (2019) Zhitao Ying, Dylan Bourgeois, Jiaxuan You, Marinka Zitnik, and Jure Leskovec. 2019\.  Gnnexplainer: Generating explanations for graph neural networks.  _Advances in neural information processing systems_ , 32.
  * Yu et al. (2022) Puxuan Yu, Razieh Rahimi, and James Allan. 2022.  Towards explainable search results: A listwise explanation generator.  In _Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pages 669–680.
  * Zhan et al. (2022a) Xunlin Zhan, Yinya Huang, Xiao Dong, Qingxing Cao, and Xiaodan Liang. 2022a.  Pathreasoner: Explainable reasoning paths for commonsense question answering.  _Knowledge-Based Systems_ , 235:107612.
  * Zhan et al. (2022b) Xunlin Zhan, Yinya Huang, Xiao Dong, Qingxing Cao, and Xiaodan Liang. 2022b.  Pathreasoner: Explainable reasoning paths for commonsense question answering.  _Knowledge-Based Systems_ , 235:107612.
  * Zhang et al. (2022) Xikun Zhang, Antoine Bosselut, Michihiro Yasunaga, Hongyu Ren, Percy Liang, Christopher D Manning, and Jure Leskovec. 2022.  Greaselm: Graph reasoning enhanced language models for question answering.  _arXiv preprint arXiv:2201.08860_.
  * Zhou et al. (2022) Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. 2022.  Learning to prompt for vision-language models.  _International Journal of Computer Vision_ , 130(9):2337–2348.
  * Zytek et al. (2022) Alexandra Zytek, Ignacio Arnaldo, Dongyu Liu, Laure Berti-Equille, and Kalyan Veeramachaneni. 2022.  The need for interpretable features: Motivation and taxonomy.  _ACM SIGKDD Explorations Newsletter_ , 24(1):1–13.