# Inferring Implicit Relations in Complex Questions with Language Models

Uri Katz1 Mor Geva2 Jonathan Berant1
1The Blavatnik School of Computer Science, Tel-Aviv University
2Allen Institute for Artificial Intelligence
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

A prominent challenge for modern language understanding systems is the ability
to answer implicit reasoning questions, where the required reasoning steps for
answering the question are not mentioned in the text explicitly. In this work,
we investigate why current models struggle with implicit reasoning question
answering (QA) tasks, by decoupling inference of reasoning steps from their
execution. We define a new task of implicit relation inference and construct a
benchmark, ImplicitRelations, where given a question, a model should output a
list of concept-relation pairs, where the relations describe the implicit
reasoning steps required for answering the question. Using ImplicitRelations,
we evaluate models from the GPT-3 family and find that, while these models
struggle on the implicit reasoning QA task, they often succeed at inferring
implicit relations. This suggests that the challenge in implicit reasoning
questions does not stem from the need to plan a reasoning strategy alone, but
to do it while also retrieving and reasoning over relevant information.

##  1 Introduction

A longstanding goal of language understanding has been to develop systems that
can reason, i.e., integrate multiple pieces of information to reach a
conclusion McCarthy (1959); Clark et al. (2021). This has sparked interest in
question answering (QA) benchmarks that require such reasoning Welbl et al.
(2018); Yang et al. (2018); Talmor and Berant (2018). One particularly
challenging case is questions that require _implicit reasoning_ , that is,
where the evidence for answering the question is not mentioned explicitly.
Consider the question “Does Santa Claus work during summer?”. This question
requires implicit reasoning since it involves knowing when the holiday
associated with Santa occurs, but this is not evident from the question.

Figure 1: An example _implicit reasoning question_ , where answering requires inferring _implicit relations_ that are not explicitly mentioned. Besides implicit relations, answering the question also requires reasoning over the relevant retrieved facts. In this work, we focus on the first step of inferring implicit relations.  Source Dataset |  Question and Implicit Relation Pairs | Answer
---|---|---
StrategyQA |  Did the 40th president of the United States forward lolcats to his friends?
$\langle$40th president of the United States, year of death$\rangle$ , $\langle$lolcats, year of creation$\rangle$ | False
Could $1 for each 2009 eclipse buy a copy of TIME magazine in 2020?
$\langle$2009 eclipse, number$\rangle$ , $\langle$TIME magazine in 2020, retail price$\rangle$ | True
Creak |  Aziz Ansari has never performed in front of a crowd.
$\langle$Aziz Ansari, profession$\rangle$ | False
Pantera made music with distorted guitars.
$\langle$Pantera, music genre$\rangle$ | True
CSQA2.0 |  None of the mail in a person’s Yahoo inbox has a stamp on it.
$\langle$Yahoo inbox, type of mailbox$\rangle$ | True
If you play a cello you cannot join the marching band.
$\langle$cello, playing posture$\rangle$ | True
Table 1: Example annotations of concept-relation pairs from ImplicitRelations
along with the question source dataset and answer. Each source exhibits
different facets of implicit reasoning questions.

Recent advances in QA Tafjord and Clark (2021); Lourie et al. (2021) have
steered attention towards implicit reasoning QA benchmarks such as StrategyQA
Geva et al. (2021), OpenCSR Lin et al. (2021), CommonsenseQA 2.0 Talmor et al.
(2021), Creak Onoe et al. (2021), and RealFP Kalyan et al. (2021), which span
a wide range of domains and reasoning skills. Still, implicit reasoning
remains an open challenge, even for large language models (LMs) such as GPT-3
and PaLM BIG-bench collab. (2021); Talmor et al. (2021); Rae et al. (2021);
Chowdhery et al. (2022).

Answering implicit reasoning questions can be viewed as a two-step process:
(a) inferring simple sub-questions necessary for answering the question, and
(b) retrieving the relevant knowledge pieces (i.e., answering sub-questions)
and reasoning over them to derive the answer. Figure 1 illustrates this
decoupling. To answer the shown question, we need to use knowledge about the
_Lusotitan dinosaur_ and _almond trees_ to infer that the relevant sub-
questions concern their _heights_. We refer to the relation _height_ , which
is not mentioned in the question as an _implicit relation_. Once implicit
relations are inferred, we can retrieve the relevant facts and deduce that the
answer is ‘False’, as a Lusotitan is much higher than an almond tree.

In this work, we put a spotlight on _implicit relations_ and investigate the
ability of language models to infer them as a necessary (albeit insufficient)
step for answering implicit reasoning questions. We first define implicit
relations, and show that they can be reliably annotated through crowdsourcing
(example annotated implicit relations are in Figure 1 and Table 1). To show
implicit relations are common, we curate and annotate implicit reasoning
questions from three existing datasets, StrategyQA, Creak, and CommonsenseQA
2.0, which results in ImplicitRelations, a new evaluation benchmark containing
615 questions and 2,673 annotations of implicit relations.

We use our benchmark to evaluate the ability of large LMs to infer implicit
relations, since they are known to acquire substantial amounts of knowledge
and common sense with scale Roberts et al. (2020); Liu et al. (2021); Smith et
al. (2022), but struggle with implicit reasoning questions. Specifically, we
evaluate models from the GPT-3 family using _in-context learning_ , where the
model is fixed and only a few examples are given as context.

We find that large LMs perform well on this task, with a 175B parameter model
recovering 0.53-0.59 of the implicit relations across datasets, outperforming
a baseline by 21-40 points. This is robust across methods for sampling in-
context examples, and even in cross-data scenarios where in-context examples
are sampled from a _different_ dataset than the target question. However,
inferring implicit relations does _not_ improve accuracy on the downstream QA
task, even when gold relations are provided. This suggests that the challenge
of implicit reasoning questions is not primarily due to implicit relation
inference, but possibly due to the need to also retrieve information and
reason over it.

To conclude, in this work we propose the notion of _implicit relations_ , and
construct the ImplicitRelations evaluation benchmark for testing the ability
of models to infer them from questions. We evaluate large LMs and show that
they infer implicit relations fairly well, while still falling short of
answering implicit reasoning questions. Our work facilitates future work on
improving implicit relation inference, and sheds light on the factors relevant
for developing models that can handle implicit reasoning. From a broader
perspective, our work joins recent community efforts to highlight the ubiquity
of missing and implicit elements in natural language Cheng and Erk (2018);
Pyatkin et al. (2020); Elazar et al. (2022). 111Our benchmark
ImplicitRelations and relevant code can be downloaded from
github.com/katzurik/ImplicitRelations.

##  2 Implicit Relations

Figure 2: An example of explicit and implicit reasoning questions that share
the same question decomposition, along with the implicit relations derived
from the retrieval steps in the decomposition.

We now define the notion of implicit relations in the context of complex
question answering.

Complex questions are questions that require multiple steps of reasoning in
order to be answered Yang et al. (2018); Talmor and Berant (2018); Welbl et
al. (2018); Khot et al. (2021). For example, the question _“Was Linnaeus alive
when Darwin published Origin of Species?”_ involves fetching two dates and
then comparing them (Figure 2). A prominent challenge in complex QA, that
attracted substantial attention recently Mihaylov et al. (2018); Khashabi et
al. (2020); Lin et al. (2021); Geva et al. (2021); Yasunaga et al. (2021); Wei
et al. (2022), is cases where the reasoning steps are _implicit_ and should be
inferred from the question. For instance, “Did Linnaeus edit Darwin’s draft of
Origin of Species?” involves the same reasoning steps, but they are not
mentioned explicitly. Thus, the former question is an _explicit_ reasoning
question, while the latter is an _implicit_ one (Figure 2).

Wolfson et al. (2020) proposed QDMR as a meaning representation for complex
questions, where a complex question $q$ is decomposed into a sequence of $m$
reasoning steps $D=(s_{1},\dots,s_{m})$, and each step $s_{i}$ corresponds to
a simple natural language question. Answering the simple questions one-by-one
yields the final answer (see decomposition in Figure 2). Geva et al. (2021)
collected decompositions for implicit reasoning questions as part of the
StrategyQA dataset, where importantly, inferring the sequence of reasoning
steps is challenging due to their implicit nature. In addition to generating
effective decompositions for implicit reasoning questions, we find an
additional challenge in evaluating these decompositions when represented as a
sequence of sub-questions. Specifically, Geva et al. (2021) distinguished two
types of reasoning steps in a decomposition – _retrieval steps_ , which
require retrieval of facts ($s_{1}$ and $s_{2}$ in Figure 2), and _logical
steps_ , which perform logical reasoning over previous results ($s_{3}$ in
Figure 2).

In this work, we observe that a key ingredient in inferring decompositions is
to identify the _implicit relations_ that are necessary for answering the
question. Concretely, each retrieval step in a question decomposition can
typically be represented as concept-relation pair $\langle c,r\rangle$, where
$c$ is a sequence of tokens from the question that refer to a concept, and $r$
is a relation of that concept. For example, the concept $c$ in step $s_{2}$ in
Figure 2 is _“Origin of Species”_ , and the relation $r$ is its publication
year.

Based on this observation, we provide the following definition for implicit
relations in complex QA. Let $q$ be an implicit reasoning question, and denote
by $D=(s_{1},...,s_{m})$ its decomposition into a sequence of $m$ reasoning
steps. Let $\\{s_{i_{1}},\dots,s_{i_{n}}\\}$ be the subset of retrieval steps
in the decomposition $D$. We define the implicit relations for answering $q$
as the set $\mathcal{I}=\\{\langle c_{1},r_{1}\rangle,...,\langle
c_{n},r_{n}\rangle\\}$ of concept-relation pairs, where each concept-relation
pair corresponds to a particular retrieval step.

In the next sections, we will use this definition to construct a task for
probing the ability of models to infer implicit relations (§3-§5), and
investigate why they struggle on the downstream QA task (§6).

##  3 The ImplicitRelations Benchmark

In this section, we describe the process of creating ImplicitRelations, a
benchmark for evaluating the ability of models to infer implicit relations.

###  3.1 Data Collection

We curate questions that require inferring implicit relations from three
recent datasets:

  * •

StrategyQA Geva et al. (2021): A dataset of yes/no questions that require
implicit multi-step reasoning. StrategyQA (StgQA) questions can be answered
from Wikipedia, and are diverse in terms of the required reasoning skills and
question topic. Large language models, such as GPT-3 Brown et al. (2020), were
shown to struggle on StgQA BIG-bench collab. (2021).

  * •

Creak Onoe et al. (2021): A dataset containing true/false statements that
require common sense and knowledge about real-world entities.

  * •

CSQA2 Talmor et al. (2021): A dataset containing yes/no questions and
true/false statements. CSQA2 questions involve generic commonsense reasoning.
Most questions do not require knowledge about particular entities.

Examples from each dataset are shown in Table 1.

Collecting questions from three sources serves two purposes. First, it
demonstrates that inferring implicit relations is necessary for many question
types: in multi-step questions (StgQA) and single-step questions (Creak), in
entity-focused questions (Creak) and generic common sense questions (CSQA2).
Second, these datasets were created using different protocols: StgQA and CSQA2
use a model-in-the-loop during data collection, while Creak does not. StgQA
and Creak ask annotators to author questions freely, while CSQA2 employs a
gamification mechanism. Having questions from different data collection
pipelines increases the likelihood that empirical conclusions are not tied to
a particular dataset.

#### Question curation

We chose questions that satisfy two properties: (a) answering the question
requires inferring an implicit relation, and (b) the question is _feasible_ ,
that is, it can be answered using real-world facts (provided as part of the
benchmark in StgQA and Creak) or using generic common sense (in CSQA2). We
sampled examples from the training set of each of the three datasets and kept
questions that satisfy the two properties.

#### Annotating implicit relations

We use Amazon Mechanical Turk to annotate implicit relations over curated
questions. We qualify 15 crowdworkers to identify concepts, which are token
sequences from the question, relevant for answering the question, and the
corresponding relation for each concept (see annotation guidelines in Appendix
C). Annotators can specify up to four concept-relation pairs per question, but
in practice, 98.9% consist of $\leq 2$ pairs. Concepts must be extracted
directly from the input questions, and relations are phrased using concise
natural language phrases.

For StgQA and Creak, which often require uncommon knowledge about entities, we
provided additional context from the original data source. For StgQA, we
provided facts along with the full question decomposition. For Creak, we
provided an explanation for why the claim is true or false.

We collected 5 annotations per example in Creak and CSQA2, and 3 annotations
in StgQA. Due to the availability of facts and question decompositions, StgQA
showed high agreement between annotators (see Table 2). Creak and CSQA2 showed
more variability, and thus, we collected more annotations per example. To
ensure quality, we manually verified all examples created during the
annotation process, filtering out annotations that do not fit the task
requirements.

###  3.2 Data Analysis

| StgQA | Creak | CSQA2
---|---|---|---
# of questions | 201 | 205 | 209
# of unique concepts | 399 | 309 | 303
# of concept-relation pairs | 1139 | 1272 | 1285
Concept agreement | 87% | 71% | 74%
Relation agreement: |  |  |
Lexical variability | 85% | 65% | 80%
Multiple reasoning paths | 15% | 35% | 20%
Table 2: Statistics on ImplicitRelations.

Table 2 provides statistics on the collected data. ImplicitRelations consists
of 615 annotated questions, $\sim 200$ per dataset, where exactly 100 examples
from each data source are used as a test set, and the rest are used as a
development set. Development set examples are in Appendix A.

#### Annotator Agreement

We manually analyzed 20 random examples from each data source to evaluate
agreement between annotators for concepts and relations. We declared concept
agreement when at least three annotators identified the same concept in an
example. We found that annotators agreed on 77% of the concepts, and less than
10% of concepts were extracted only by a single annotator. See Table 2 for
break-down by data source.

Assessing agreement on relations is more challenging, since relations are
short phrases that can differ lexically. We marked for each example whether
annotated relations differed only lexically or due to multiple reasoning
strategies for answering the question. In 76% of the examples, the relations
across _all_ annotators were identical or differed lexically, e.g., the
relations _“average price”_ and _“cost”_. In 24% of the examples, multiple
reasoning strategies were used, which would result in different reasoning and
retrieval steps (see Table 2).

Overall, our analysis suggests that implicit relations can be annotated
reliably.

##  4 Experimental Setting

We now turn to evaluate the ability of large LMs to infer implicit relations
in questions. To this end, we use examples from ImplicitRelations in a few-
shot _in-context learning_ setting Brown et al. (2020), where given several
input-output examples and a test input, the LM is expected to generate the
required output. We focus on this setup following the recent progress in in-
context learning, specifically for tasks that involve general commonsense
reasoning Da et al. (2021); Chowdhery et al. (2022).

###  4.1 Task and Model Specification

Given a test question $q^{*}$, we prepend $k$ annotated examples $\\{\langle
q^{(i)},\mathcal{I}^{(i)}\rangle\\}_{i=1}^{k}$ to it, where $q^{(i)}$ is the
$i$-th question in this set and $\mathcal{I}^{(i)}=\\{\langle
c^{(i)}_{1},r^{(i)}_{1}\rangle,...,\langle
c^{(i)}_{n_{i}},r^{(i)}_{n_{i}}\rangle\\}$ is the set of corresponding
concept-relation pairs. Specifically, we use the following input format:

$\displaystyle\textbf{Question:}\;\;q_{1}$ $\displaystyle\textbf{Implicit
Reasoning:}\;\;\langle c^{(1)}_{1},r^{(1)}_{1}\rangle,...,\langle
c^{(1)}_{n_{1}},r^{(1)}_{n_{1}}\rangle$ $\displaystyle\dots$
$\displaystyle\textbf{Question:}\;\;q_{k}$ $\displaystyle\textbf{Implicit
Reasoning:}\;\;\langle c^{(k)}_{1},r^{(k)}_{1}\rangle,...,\langle
c^{(k)}_{n_{k}},r^{(k)}_{n_{k}}\rangle$
$\displaystyle\textbf{Question:}\;\;q^{*}$ Implicit Reasoning:

Example inputs are given in Appendix A. The prefixes “Question” and “Implicit
Reasoning” were chosen arbitrarily and remained fixed throughout all
experiments. For each test question, $k$ examples are randomly sampled from
the development set, and for each example, a single random annotation is
selected. In our experiments, we use $k=16$.222We experimented with
$k\in\\{8,16,32$} but found it does not dramatically change performance, see
Appendix B.

#### Models

We evaluate models from the GPT-3 family,333We use the API at
https://openai.com/api/. which are known to exhibit broad factual knowledge
Brown et al. (2020). In particular, we use text-davinci-002, a 175B-parameter
LM, which to the best of our knowledge was not trained on any of the target
benchmarks. Furthermore, to assess the scaling behaviour of models, we
experiment with other models from the GPT-3 family, see §5.2 for details. In
all experiments, outputs are predicted using greedy decoding.

#### Baseline

To account for correlations originating from the concepts that appear in the
question or reasoning shortcuts, we define a ‘Concept-only’ baseline, where
instead of testing whether the model can infer the implicit relations from the
full question, we test its ability to infer them from the set of _gold_
concepts that appear in the question. For this baseline, we use the same
inputs as before, but replace every question $q_{i}$ and test question $q^{*}$
with its set of annotated concepts.

$\displaystyle\textbf{Question:}\;\;q_{i}\;\rightarrow\;\textbf{Question:}\;\;c^{(i)}_{1}\;;\;...\;;\;c^{(i)}_{n_{i}}$

While the identity of the gold concepts provides useful information for
inferring implicit relations, we expect models that have access to the full
question to perform better.

###  4.2 Evaluation

Inferring implicit relations involves identifying concepts and relations. We
now define evaluation metrics for this task.

#### Concept extraction

Our output is a set of conept-relation pairs. Let $\mathcal{C}_{\text{pred}}$
be the set of concepts predicted by the LM, and let
$\mathcal{C}_{\text{gold}}^{i}$ be the gold set annotated by annotator $i$.
Given that annotated concepts are tokens from the original question, we can
use edit-distance to match each predicted concept
$c\in\mathcal{C}_{\text{pred}}$ with a concept from
$\mathcal{C}_{\text{gold}}^{i}$, declaring a match when the edit distance of
the best match is above 0.8. Following concept matching, we compute recall and
precision in the typical fashion and take the $\max$ over all annotators. A
post-hoc manual analysis validated that no incorrect concept matching occurred
due to the use of edit distance.

Figure 3:  Given predicted concept-relation pairs (P) and gold concept-
relation pairs (G), we evaluate relations by aligning predicted and gold
concepts using edit distance (1), and computing the cosine similarity between
matched relation embeddings (2). Relation coverage is the fraction of gold
relations with similarity $>\tau$ (3).

#### Relation coverage

Since relations are short phrases with high lexical variability, string
matching is not a viable strategy. To overcome this, we leverage our ability
to align concepts and use relation embeddings rather than relation strings.
Figure 3 depicts our evaluation procedure for two sets of predicted
($\mathcal{R}_{\text{pred}}$) and annotated ($\mathcal{R}_{\text{gold}}$)
relations. First (Figure 3, step 1), we align predicted and gold concept-
relation pairs, using the concepts (as done for concept evaluation). Then
(Figure 3, step 2), we embed every relation phrase, using the sentence
transformer all-mpnet-base-v2 Reimers and Gurevych (2019), and compute cosine
similarity between the embeddings of matched relations (defined it as zero if
no relation was matched). Last (Figure 3, step 3), we consider a gold relation
$r_{\text{gold}}$ _covered_ if cosine similarity is higher than a threshold
$\tau$, and compute _relation coverage_ , that is, the fraction of gold
relations that were covered. With this procedure, we evaluate model
predictions against each annotation and take the maximum as the final relation
coverage.

We focus on coverage (rather than precision) since we care about whether a
model can reveal implicit relations, but predicting additional ones is mostly
harmless. Moreover, since we use in-context learning, the average number of
concept-relation pairs generated is similar to the average number of gold
concept-relation pairs (Table 3).

| StgQA | Creak | CSQA2
---|---|---|---
# of gold pairs | 1.9 | 1.2 | 1.2
# of generated pairs | $2.0\pm 0.04$ | $1.2\pm 0.02$ | $1.2\pm 0.01$
Table 3: The mean number of concept-relation pairs in the development set of
each data source. For model-generated pairs, we report an average over 3
seeds.

To set a threshold $\tau$ on relation embedding similarity, we annotated
whether a predicted relation is semantically equivalent to a matched gold
relation for 100 development examples. We choose $\tau=0.51$, which results in
a 5% false-positive rate (predicting that two different relations are
equivalent) and 12% false-negative rate (predicting that two equivalent
relations are different). All reported results are an average over three
random seeds.

##  5 Large LMs Can Infer Implicit Relations

|  | Concept | Concept | Relation
---|---|---|---|---
|  | Recall | Precision | Coverage
StgQA | CO | 0.99$\pm$0.01 | 0.95$\pm$0.02 | 0.32$\pm$0.02
FQ | 0.97$\pm$0.01 | 0.89$\pm$0.02 | 0.53$\pm$0.02
Creak | CO | 1$\pm$0.0 | 0.96$\pm$0.0 | 0.33$\pm$0.03
FQ | 0.98$\pm$0.0 | 0.95$\pm$0.01 | 0.54$\pm$0.05
CSQA2 | CO | 1$\pm$0.0 | 0.98$\pm$0.02 | 0.19$\pm$0.02
FQ | 0.93$\pm$0.02 | 0.94$\pm$0.01 | 0.59$\pm$0.01
Table 4: Test set performance for concept-only (CO) and full-question (FQ) for
all datasets.

Table 4 shows results on implicit relation inference. First, the model
successfully identifies the relevant concepts in the question, achieving high
concept recall and precision across all datasets. This is not limited to named
entities but is also achieved when concepts are more abstract, as in CSQA2
(Table 1). More importantly, GPT-3 infers the implicit relations well,
achieving relation coverage scores of 0.53, 0.54, and 0.59 on StgQA, Creak and
CSQA2, respectively. Moreover, a model exposed to the full question
dramatically outperforms a model exposed only to the gold concepts by 21, 21,
and 40 points on the three datasets. This indicates that concepts contain
relevant information, but access to the full question allows the LM to infer
the reasoning strategy and in turn the implicit relations. We provide examples
for predicted vs. gold concept-relation pairs along with additional
qualitative analysis in Appendix A.

Next, we perform additional experiments to (a) further substantiate the
ability of LMs to infer implicit relations (§5.1), and (b) test the effect of
model scale on performance (§5.2).

###  5.1 Effect of In-Context Examples

While the aforementioned results are encouraging, there are two potential
(non-disjoint) causes for them: (a) the LM “understands” the task of implicit
relation inference, or (b) the LM observes in-context examples and uses them
to guess implicit relations for the target question (“soft copying”). We study
the effect of these causes.

#### Similar vs. dissimilar in-context examples

To quantify the effect of in-context examples, rather than choosing them
randomly, we use examples that are similar or dissimilar to the target
question in terms of their implicit relations.

We first represent each example as an embedding vector, by (a) concatenating
all annotated relations, i.e., $r_{1},\dots,r_{n}$, and computing a vector
representation using a sentence transformer, and (b) averaging the embedding
vectors of all annotators. Then, for each example, we select two sets of in-
context examples: (a) _Similar_ : the top-$k$ most similar examples (using
cosine similarity), and (b) _Dissimilar_ : we discard the 33% most similar
examples, and randomly sample from the rest. In both cases, we use gold
implicit relations at test time, and thus this experiment is for analysis
only.

|  | Relation | Copying
---|---|---|---
|  | Coverage |
StgQA | Dissimilar | 0.46$\pm$0.01 | 0.01$\pm$0.01
Random | 0.52$\pm$0.02 | 0.13$\pm$0.02
Similar | 0.54$\pm$0.02 | 0.31$\pm$0.02
| Concept-only | 0.28$\pm$0.04 | 0.10$\pm$0.04
Creak | Dissimilar | 0.60$\pm$0.02 | 0.02$\pm$0.02
Random | 0.59$\pm$0.03 | 0.08$\pm$0.02
Similar | 0.67$\pm$0.02 | 0.33$\pm$0.03
| Concept-only | 0.34$\pm$0.02 | 0.13$\pm$0.02
CSQA2 | Dissimilar | 0.63$\pm$0.02 | 0.02$\pm$0.02
Random | 0.67$\pm$0.01 | 0.07$\pm$0.02
Similar | 0.73$\pm$0.01 | 0.21$\pm$0.03
| Concept-only | 0.20$\pm$0.02 | 0.10$\pm$0.02
Table 5: Development set performance. Controlling the set of in-context
examples with Dissimilar, Similar, Random relations, and Concept-only
baseline.

Table 5 shows relation coverage for the different sets of in-context examples
and the fraction of cases where one of the implicit relations predicted by the
LM is copied from the in-context examples. When _Dissimilar_ examples are
presented, there is a slight performance degradation, most notably in StgQA.
However, results are still dramatically higher compared to Concept-only.
Moreover, the model succeeds in predicting implicit relations while hardly
copying from in-context examples.

In the _Similar_ setting, performance increases across all datasets, along
with a much higher rate of copying. This hints that designing methods for
retrieving similar prompts can lead to gains in performance Rubin et al.
(2022).

To further investigate the relation between copying and performance, we label
every example for whether the model copies from in-context examples and the
coverage of the inferred implicit relations. We then compute the point-
biserial correlation Tate (1954) to check if copying is correlated with
performance and find that correlation is low ($<0.1$ for all datasets),
showing that copying does not explain model performance.

Overall, this experiment suggests that while models can leverage examples from
context to improve performance, the LM does more than copy and execute
implicit relation inference.

#### Cross-dataset in-context examples

If LMs can infer implicit relations, we should expect high performance even
when in-context examples and target questions are taken from different
datasets.

To test this, we evaluate performance on questions from Creak and CSQA2 when
in-context examples originate from all 3 datasets. Testing on StgQA does not
work well because the number of implicit relations in an example is typically
two, while in Creak and CSQA2 it is typically one (see Table 3), and thus the
LM output a single implicit relation, leading to poor relation coverage.

Table 6 shows that, overall, relation coverage remains high for all sources,
suggesting that the LM indeed infers implicit relations regardless of the
question and reasoning types in the source dataset. Concept recall and
precision are also relatively stable, except when using StgQA for in-context
examples, since the model tends to output two concept-relation pairs, reducing
precision. Thus, the LM is sensitive to _the number_ of output concept-
relation pairs that appear in in-context examples, but succeeds in inferring
implicit relations.

Inference/Context | Concept | Concept | Relation
---|---|---|---
Source | Recall | Precision | Coverage
Creak/StgQA | 0.97$\pm$0.0 | 0.69$\pm$0.01 | 0.62$\pm$0.03
Creak/Creak | 0.93$\pm$0.02 | 0.91$\pm$0.01 | 0.59$\pm$0.03
Creak/CSQA2 | 0.93$\pm$0.01 | 0.90$\pm$0.02 | 0.60$\pm$0.0
CSQA2/StgQA | 0.98$\pm$0.01 | 0.77$\pm$0.02 | 0.62$\pm$0.0
CSQA2/Creak | 0.96$\pm$0.02 | 0.94$\pm$0.02 | 0.59$\pm$0.02
CSQA2/CSQA2 | 0.95$\pm$0.01 | 0.96$\pm$0.0 | 0.67$\pm$0.01
Table 6: Development set performance in the cross-dataset setup where
inference on example from one dataset is done with in-context examples from
another dataset.

###  5.2 Effect of Model Size

Recent work Kaplan et al. (2020); Smith et al. (2022); Chowdhery et al. (2022)
has shown that reasoning abilities of LMs improve with model size. We evaluate
this effect on models from the GPT-3 family: ada, babbage, curie, and davinci,
which are estimated to have 350M, 1.3B, 6.7B, and 175B parameters,
respectively Gao (2021); Black et al. (2022). text-davinci, the model
evaluated thus far, is a more recent LM that Ouyang et al. (2022) was trained
differently.444text-davinci has 175B parameters like davinci, but its relation
coverage on StgQA is higher: 0.43$\rightarrow$0.52. This indicates that its
training procedure improves inference of implicit relations.

Table 7 presents results on StgQA. Increasing model size improves relation
coverage and concept recall, but does not significantly change concept
precision. Moving from curie to davinci leads to a modest gain in relation
coverage. Comparing this to the order of magnitude difference in parameters
between curie and davinci suggests that inferring implicit relations does not
explain performance improvement in many reasoning and commonsense QA
benchmarks. The smallest model, babbage, tends to produce structural errors,
indicating it did not properly learn the task.

Parameters | Concept | Concept | Relation
---|---|---|---
| Recall | Precision | Coverage
350M | 0.83$\pm$0.01 | 0.89$\pm$0.01 | 0.21$\pm$0.01
1.3B | 0.93$\pm$0.01 | 0.84$\pm$0.01 | 0.37$\pm$0.01
6.7B | 0.92$\pm$0.01 | 0.83$\pm$0.02 | 0.42$\pm$0.02
175B | 0.97$\pm$0.0 | 0.88$\pm$0.0 | 0.43$\pm$0.03
Table 7: Model size and performance comparison on the development set of
StgQA. The relation coverage improves as model size is increased.

##  6 Implicit Relations for QA

Given that LMs infer implicit relations well, a natural question is whether
they improve performance on answering implicit reasoning questions.

To examine this, we created three experimental setups: Question + predicted:
in-context examples are triples of the question, the implicit relations, and
the True/False answer; the model is given a question and asked to return the
implicit relations and the answer. Question + gold: Similar to Question +
predicted except that the model is given the target question and gold implicit
relations and asked to return the answer. Question only: in-context examples
are pairs of questions and answers, and the model is given a question and
asked to provide the answer. We report an average over 7 seeds. See Figure 4
for results.

Figure 4: Test QA accuracy under all conditions, averaged over 7 seeds.
Providing the gold implicit relations did not contribute to model performance.

Overall, access to either gold or predicted relations does not improve
accuracy. This suggests that additional capabilities are missing from LMs to
handle implicit reasoning questions, such as retrieval and reasoning. This
agrees with work on chain-of-thoughts prompting Wei et al. (2022), which found
that adding an explanation of the reasoning process to a question does not
improve performance on StgQA for both GPT-3 and LaMDA Thoppilan et al. (2022).
Nevertheless, recently Chowdhery et al. (2022) achieved improvements on StgQA
using chain-of-thought prompting, but with the larger 540B-parameter PaLM.

On StgQA, adding gold relations (Question + gold) does not improve QA
performance.555paired t-test with $p$-value $>0.05$. Additionally, no
significant differences were observed when the model inferred implicit
relations on its own (Question + predicted).666paired t-test with $p$-value
$>0.05$. For Creak, adding gold implicit relations did not improve accuracy
compared to Question only, and none of the experiments showed any significant
difference. Last, in CSQA2 adding gold implicit relations (Question + gold)
did not improve the QA performance, but we observed a statistically
significant accuracy improvement of 4.5% when the model inferred the implicit
relations (Question + predicted).777paired t-test with $p$-value $<0.05$,
Cohen’s $d=1.7$.

To further analyze the results, we computed the point-biserial correlation
coefficient between the relation coverage score and the binary outcome
(correct/incorrect) for each question. We found that relation coverage score
and answer accuracy are entirely not correlated with a $r_{\text{pb}}$
coefficient of $0.03$, $-0.02$ and $0.06$ for StgQA, CSQA2 and Creak
respectively. Overall, our results indicate that inferring implicit relations
correctly is not sufficient to answer implicit reasoning questions.

##  7 Related Work

Recent work utilized the ability of large LMs to generate intermediate
reasoning steps for improving performance on QA tasks Wei et al. (2022); Wang
et al. (2022); Zelikman et al. (2022); Nye et al. (2022). Wei et al. (2022)
introduced ‘chain-of-thought’ prompting to elicit intermediate reasoning steps
along with answers from LMs, which improved performance on several reasoning
tasks. Conversely, we propose a task and benchmark for evaluating the ability
of LMs to infer the intermediate reasoning steps themselves.

Prior work has dealt with reasoning abilities in LMs Talmor et al. (2020);
Khashabi et al. (2020); Gu et al. (2021) by fine-tuning LMs to generate
additional knowledge for reasoning questions. We contribute to this effort by
evaluating the in-context ability to infer implicit relations with large LMs.

Implicit relations are closely related to question decomposition, which have
been used in past work to improve performance on questions that require
reasoning Min et al. (2019); Wolfson et al. (2020); Perez et al. (2020); Khot
et al. (2021). We contribute to this research direction by defining implicit
relations pairs, which provide a structured representation of the decomposed
sub-questions and allow us to examine how language models infer reasoning
steps. Several works in narrative understanding Rajpurkar et al. (2018);
Mostafazadeh et al. (2020); Lal et al. (2021) have attempted to assess a
model’s implicit reasoning capabilities using different methods, such as
assessing the solution path to unanswerable questions and and narrative
understanding through question-answering. Despite their different approaches,
these studies are relevant to our cause.

##  8 Conclusion

We propose the task of implicit relation inference, which decouples inference
of reasoning steps from their execution. We introduce ImplicitRelations, a
benchmark that includes more than 2,000 annotated implicit relations. We show
large LMs can infer implicit relations across multiple types of questions and
reasoning skills, but this success does not translate to an improvement in
answering implicit reasoning questions. Our work sheds light on capabilities
missing from large LMs for addressing implicit reasoning questions, and
provides a valuable resource for improving the ability of models to infer
implicit relations.

## Limitations

This research has some limitations, which are typical for work on text
generation with large language models.

First, we demonstrated that large LMs can infer implicit relations from
complex questions, but we also showed that they may fail to answer those
questions correctly. It is unclear how LMs can use implicit relations to
improve QA accuracy or what is the path that leads from inferring implicit
relations to actually answering the questions.

Second, evaluating relation coverage requires comparing to free texts, and
therefore may be prone to error. Despite the fact that an analysis performed
manually exhibited a high degree of consistency with the automatic one, we
cannot guarantee the same result for datasets or parameters that have not been
tested.

Finally, our research was conducted utilizing OpenAI’s GPT-3 family of models,
which are not publicly available. Despite our best efforts to eliminate
confounding factors, there is a lack of transparency regarding the training
methods and the data composition used for pretraining those models.

## Acknowledgements

We thank Itay Levy for useful feedback. This research was partially supported
by the Computer Science Scholarship granted by the Séphora Berrebi Foundation,
the Yandex Initiative for Machine Learning, and the European Research Council
(ERC) under the European Union Horizons 2020 research and innovation programme
(grant ERC DELPHI 802800).

## References

    * BIG-bench collab. (2021) BIG-bench collab. 2021.  Beyond the imitation game: Measuring and extrapolating the capabilities of language models.  _In preparation_.
    * Black et al. (2022) Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. 2022.  GPT-NeoX-20B: An open-source autoregressive language model.  In _Proceedings of the ACL Workshop on Challenges & Perspectives in Creating Large Language Models_.
    * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
    * Cheng and Erk (2018) Pengxiang Cheng and Katrin Erk. 2018.  Implicit argument prediction with event knowledge.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 831–840, New Orleans, Louisiana. Association for Computational Linguistics.
    * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022.  Palm: Scaling language modeling with pathways.  _arXiv preprint arXiv:2204.02311_.
    * Clark et al. (2021) Peter Clark, Oyvind Tafjord, and Kyle Richardson. 2021.  Transformers as soft reasoners over language.  In _Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence_ , pages 3882–3890.
    * Da et al. (2021) Jeff Da, Ronan Le Bras, Ximing Lu, Yejin Choi, and Antoine Bosselut. 2021.  Analyzing commonsense emergence in few-shot knowledge models.  _arXiv preprint arXiv:2101.00297_.
    * Elazar et al. (2022) Yanai Elazar, Victoria Basmov, Yoav Goldberg, and Reut Tsarfaty. 2022.  Text-based np enrichment.  _Transactions of the Association for Computational Linguistics_.
    * Gao (2021) Leo Gao. 2021.  On the sizes of openai api models.  EleutherAI Blog.
    * Geva et al. (2021) Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021.  Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies.  _Transactions of the Association for Computational Linguistics_ , 9:346–361.
    * Gu et al. (2021) Yuling Gu, Bhavana Dalvi, and Peter Clark. 2021.  Dream: Uncovering mental models behind language models.  _ArXiv_ , abs/2112.08656.
    * Kalyan et al. (2021) Ashwin Kalyan, Abhinav Kumar, Arjun Chandrasekaran, Ashish Sabharwal, and Peter Clark. 2021.  How much coffee was consumed during EMNLP 2019? fermi problems: A new reasoning challenge for AI.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 7318–7328, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
    * Kaplan et al. (2020) Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020.  Scaling laws for neural language models.  _arXiv preprint arXiv:2001.08361_.
    * Khashabi et al. (2020) Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. 2020.  UNIFIEDQA: Crossing format boundaries with a single QA system.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 1896–1907, Online. Association for Computational Linguistics.
    * Khot et al. (2021) Tushar Khot, Daniel Khashabi, Kyle Richardson, Peter Clark, and Ashish Sabharwal. 2021.  Text modular networks: Learning to decompose tasks in the language of existing models.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1264–1279, Online. Association for Computational Linguistics.
    * Lal et al. (2021) Yash Kumar Lal, Nathanael Chambers, Raymond Mooney, and Niranjan Balasubramanian. 2021.  TellMeWhy: A dataset for answering why-questions in narratives.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 596–610, Online. Association for Computational Linguistics.
    * Lin et al. (2021) Bill Yuchen Lin, Haitian Sun, Bhuwan Dhingra, Manzil Zaheer, Xiang Ren, and William Cohen. 2021.  Differentiable open-ended commonsense reasoning.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4611–4625, Online. Association for Computational Linguistics.
    * Liu et al. (2021) Jiachen Liu, Alisa Liu, Ximing Lu, Sean Welleck, Peter West, Ronan Le Bras, Yejin Choi, and Hannaneh Hajishirzi. 2021.  Generated knowledge prompting for commonsense reasoning.  _ArXiv_ , abs/2110.08387.
    * Lourie et al. (2021) Nicholas Lourie, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021.  Unicorn on rainbow: A universal commonsense reasoning model on a new multitask benchmark.  In _AAAI_.
    * McCarthy (1959) John McCarthy. 1959.  Programs with common sense.  In _Proceedings of the Teddington Conference on the Mechanization of Thought Processes_ , pages 75–91, London. Her Majesty’s Stationary Office.
    * Mihaylov et al. (2018) Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018.  Can a suit of armor conduct electricity? a new dataset for open book question answering.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2381–2391, Brussels, Belgium. Association for Computational Linguistics.
    * Min et al. (2019) Sewon Min, Victor Zhong, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2019.  Multi-hop reading comprehension through question decomposition and rescoring.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 6097–6109, Florence, Italy. Association for Computational Linguistics.
    * Mostafazadeh et al. (2020) Nasrin Mostafazadeh, Aditya Kalyanpur, Lori Moon, David Buchanan, Lauren Berkowitz, Or Biran, and Jennifer Chu-Carroll. 2020.  GLUCOSE: GeneraLized and COntextualized story explanations.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4569–4586, Online. Association for Computational Linguistics.
    * Nye et al. (2022) Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. 2022.  Show your work: Scratchpads for intermediate computation with language models.  In _Deep Learning for Code Workshop_.
    * Onoe et al. (2021) Yasumasa Onoe, Michael JQ Zhang, Eunsol Choi, and Greg Durrett. 2021.  CREAK: A dataset for commonsense reasoning over entity knowledge.  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)_.
    * Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022\.  Training language models to follow instructions with human feedback.  _arXiv preprint arXiv:2203.02155_.
    * Perez et al. (2020) Ethan Perez, Patrick Lewis, Wen-tau Yih, Kyunghyun Cho, and Douwe Kiela. 2020.  Unsupervised question decomposition for question answering.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 8864–8880, Online. Association for Computational Linguistics.
    * Pyatkin et al. (2020) Valentina Pyatkin, Ayal Klein, Reut Tsarfaty, and Ido Dagan. 2020.  QADiscourse - Discourse Relations as QA Pairs: Representation, Crowdsourcing and Baselines.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 2804–2819, Online. Association for Computational Linguistics.
    * Rae et al. (2021) Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. 2021.  Scaling language models: Methods, analysis & insights from training gopher.  _arXiv preprint arXiv:2112.11446_.
    * Rajpurkar et al. (2018) Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018.  Know what you don’t know: Unanswerable questions for SQuAD.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 784–789, Melbourne, Australia. Association for Computational Linguistics.
    * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-bert: Sentence embeddings using siamese bert-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics.
    * Roberts et al. (2020) Adam Roberts, Colin Raffel, and Noam Shazeer. 2020.  How much knowledge can you pack into the parameters of a language model? In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 5418–5426, Online. Association for Computational Linguistics.
    * Rubin et al. (2022) Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2022.  Learning to retrieve prompts for in-context learning.  In _North American Association for Computational Linguistics (NAACL)_.
    * Smith et al. (2022) Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, et al. 2022.  Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model.  _arXiv preprint arXiv:2201.11990_.
    * Tafjord and Clark (2021) Oyvind Tafjord and Peter Clark. 2021.  General-purpose question-answering with Macaw.  _ArXiv_ , abs/2109.02593.
    * Talmor and Berant (2018) Alon Talmor and Jonathan Berant. 2018.  The web as a knowledge-base for answering complex questions.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 641–651, New Orleans, Louisiana. Association for Computational Linguistics.
    * Talmor et al. (2020) Alon Talmor, Oyvind Tafjord, Peter Clark, Yoav Goldberg, and Jonathan Berant. 2020\.  Leap-of-thought: Teaching pre-trained models to systematically reason over implicit knowledge.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 20227–20237. Curran Associates, Inc.
    * Talmor et al. (2021) Alon Talmor, Ori Yoran, Ronan Le Bras, Chandra Bhagavatula, Yoav Goldberg, Yejin Choi, and Jonathan Berant. 2021.  Commonsenseqa 2.0: Exposing the limits of ai through gamification.  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1)_.
    * Tate (1954) Robert F Tate. 1954.  Correlation between a discrete and a continuous variable. point-biserial correlation.  _The Annals of mathematical statistics_ , 25(3):603–607.
    * Thoppilan et al. (2022) Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam M. Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, Yaguang Li, Hongrae Lee, Huaixiu Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Yanqi Zhou, Chung-Ching Chang, I. A. Krivokon, Willard James Rusch, Marc Pickett, Kathleen S. Meier-Hellstern, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Hartz Søraker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravindran Rajakumar, Alena Butryna, Matthew Lamm, V. O. Kuzmina, Joseph Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera-Arcas, Claire Cui, Marian Croak, Ed Chi, and Quoc Le. 2022.  Lamda: Language models for dialog applications.  _ArXiv_ , abs/2201.08239.
    * Wang et al. (2022) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou. 2022.  Self-consistency improves chain of thought reasoning in language models.  _arXiv preprint arXiv:2203.11171_.
    * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022.  Chain of thought prompting elicits reasoning in large language models.  _ArXiv_ , abs/2201.11903.
    * Welbl et al. (2018) Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2018.  Constructing datasets for multi-hop reading comprehension across documents.  _Transactions of the Association for Computational Linguistics_ , 6:287–302.
    * Wolfson et al. (2020) Tomer Wolfson, Mor Geva, Ankit Gupta, Matt Gardner, Yoav Goldberg, Daniel Deutch, and Jonathan Berant. 2020.  Break it down: A question understanding benchmark.  _Transactions of the Association for Computational Linguistics_ , 8:183–198.
    * Yang et al. (2018) Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018.  HotpotQA: A dataset for diverse, explainable multi-hop question answering.  In _Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
    * Yasunaga et al. (2021) Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Percy Liang, and Jure Leskovec. 2021.  QA-GNN: Reasoning with language models and knowledge graphs for question answering.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 535–546, Online. Association for Computational Linguistics.
    * Zelikman et al. (2022) Eric Zelikman, Yuhuai Wu, and Noah D Goodman. 2022.  Star: Bootstrapping reasoning with reasoning.  _arXiv preprint arXiv:2203.14465_.

##  Appendix A Examples and Qualitative Analysis

###  A.1 Qualitative analysis of evaluation

In addition to the evaluation described in §4 and used throughout our study,
we performed manual qualitative analysis to assess the relation coverage
metric model outputs. We sampled 50 examples randomly from StgQA together with
GPT-3 predictions and manually labeled if the implicit relations output by the
model are semantically correct.

| Relation coverage
---|---
Manual evaluation | 0.6
Automatic evaluation | 0.53
Table 8: Manual and automatic evaluation of 50 examples from StgQA.

Results in Table 8 show that our relation coverage was slightly more
conservative than our manual evaluation – 0.53 vs. 0.6. Out of the 20 examples
that we marked as incorrect, the automatic evaluation scored 13 examples (65%)
with 0 relation coverage, 6 (30%) examples with partial coverage (scored 0.5),
which indeed we included a partially correct prediction but not enough to to
cover the needed reasoning process. Only one example out of the 20 was a false
positive.

###  A.2 ImplicitRelations Examples

Tables 9,10,11 present examples from each source dataset accompanied by the
question, gold annotated implicit relation pairs, predicted pairs generated by
the model and the answer to the question.

Examples from StrategyQA
---
Q: Could Eric Clapton’s children play a regulation game of basketball among
themselves?
G: $\langle$Eric Clapton, number of children$\rangle$ , $\langle$regulation
game of basketball, number of players$\rangle$
P: $\langle$eric clapton, number of children$\rangle$ , $\langle$regulation
game of basketball, number of players$\rangle$
A: False
Q: Can you purchase General Motors products at a movie theater?
G: $\langle$General Motors, Types of products$\rangle$ , $\langle$Movie
theater, products sold$\rangle$
P: $\langle$general motors, type of product$\rangle$ , $\langle$movie theater,
type of product$\rangle$
A: False
Q: Does Rupert Murdoch’s alma mater have more history than the USA?
G: $\langle$Rupert Murdoch’s alma mater, year founded$\rangle$ , $\langle$USA,
year founded$\rangle$
P: $\langle$rupert murdoch’s alma mater, date of founding$\rangle$ ,
$\langle$usa, date of founding$\rangle$
A: True
Q: Do Squidward Tentacles and Alan Greenspan have different musical passions?
G: $\langle$Squidward Tentacles, instruments played$\rangle$ , $\langle$Alan
Greenspan, instruments played$\rangle$
P: $\langle$squidward tentacles, musical instrument$\rangle$ , $\langle$alan
greenspan, musical instrument$\rangle$
A: False
Q: Is the most expensive color in the world Blue?
G: $\langle$Blue, cost of pigment$\rangle$ , $\langle$most expensive color in
the world, cost of pigment$\rangle$
P: $\langle$blue, color$\rangle$ , $\langle$most expensive color,
color$\rangle$
A: True
Table 9: Examples of gold annotations (G) and predicted (P) concept-relation
pairs from ImplicitRelations along with the question (Q) and answer (A) from
StrategyQA. Examples from Creak
---
Q: People celebrate Cinco de Mayo before summer starts.
G: $\langle$Cinco de Mayo, date$\rangle$ , $\langle$summer, starting
date$\rangle$
P: $\langle$cinco de Mayo, date$\rangle$
A: True
Q: Arthur C. Clarke wrote a book set in outer space.
G: $\langle$Arthur C. Clarke, writing genre$\rangle$
P: $\langle$arthur c. clarke, book genre$\rangle$
A: True
Q: The local chamber music company hired a Harp seal to join the group.
G: $\langle$Harp seal, musical ability$\rangle$
P: $\langle$harp seal, natural habitat$\rangle$
A: False
Q: Nicole Kidman has never heard of Batman.
G: $\langle$Nicole Kidman, filmography$\rangle$
P: $\langle$nicole kidman, knowledge$\rangle$
A: False
Q: Competitors know how to drive when they are competing in Formula One
racing.
G: $\langle$Formula One racing, type of sport$\rangle$
P: $\langle$formula one racing, skills required$\rangle$
A: True
Table 10: Examples of gold annotations (G) and predicted (P) concept-relation
pairs from ImplicitRelations along with the question (Q) and answer (A) from
Creak. Examples from CSQA2
---
Q: Trees are never part of forests.
G: $\langle$forests, type of flora$\rangle$
P: $\langle$trees, location$\rangle$
A: False
Q: You can only win a medal at the Olympics if you place in the top 3?
G: $\langle$Olympics, medal awarding rules$\rangle$
P: $\langle$medal, requirements for winning$\rangle$
A: True
Q: Is it true that a hospital is capable of treating Dutch Elm disease?
G: $\langle$Dutch Elm disease, targeted organism$\rangle$ , $\langle$hospital,
species treated$\rangle$
P: $\langle$hospital, type of diseases treated$\rangle$
A: False
Q: at least one letter in the word walnut is q.
G: $\langle$walnut, spelling$\rangle$
P: $\langle$word walnut, letters$\rangle$
A: False
Q: Is the temperature usually quite hot before Christmas in the southern
hemisphere?
G: $\langle$Christmas, month$\rangle$ , $\langle$southern hemisphere, summer
months$\rangle$
P: $\langle$southern hemisphere, seasons$\rangle$
A: True
Table 11: Examples of gold annotations (G) and predicted (P) concept-relation
pairs from ImplicitRelations along with the question (Q) and answer (A) from
CSQA2.

##  Appendix B Number of Prompt Examples

We investigate how different number of examples influence implicit relation
inference. We run the original experiment with $k\in\\{8,16,32\\}$ examples in
the prompt for 3 random seeds. The results (Table 12) show that the number of
examples in the prompt has little effect on all evaluation metrics, justifying
our choice to use $k=16$ in all experiments.

| $k$ | Concept | Concept | Relation
---|---|---|---|---
|  | Recall | Precision | Coverage
StgQA | 8 | 0.94$\pm$0.02 | 0.87$\pm$0.01 | 0.48$\pm$0.02
16 | 0.97$\pm$0.01 | 0.88$\pm$0.01 | 0.53$\pm$0.02
32 | 0.97$\pm$0.01 | 0.88$\pm$0.01 | 0.50$\pm$0.01
Creak | 8 | 0.94$\pm$0.02 | 0.91$\pm$0.02 | 0.56$\pm$0.0
16 | 0.93$\pm$0.02 | 0.91$\pm$0.01 | 0.59$\pm$0.03
32 | 0.95$\pm$0.01 | 0.90$\pm$0.02 | 0.62$\pm$0.02
CSQA2 | 8 | 0.94$\pm$0.0 | 0.95$\pm$0.02 | 0.61$\pm$0.02
16 | 0.95$\pm$0.01 | 0.96$\pm$0.0 | 0.67$\pm$0.01
32 | 0.95$\pm$0.01 | 0.96$\pm$0.02 | 0.66$\pm$0.02
Table 12: Development set results for 8/16/32 examples in the prompt, averaged
for 3 seeds.

##  Appendix C Annotation Task Instruction

###  C.1 Task instruction

See Figures/mturk_instruction_1.pdf

###  C.2 Task instruction cont.

See Figures/mturk_instruction_2.pdf

###  C.3 Concept-relation pairs

See Figures/mturk_instruction_3.pdf


Amid a discussion about Green AI in which we see explainability neglected, we explore the possibility to efficiently approximate computationally expensive explainers. To this end, we propose feature attribution modelling with Empirical Explainers. Empirical Explainers learn from data to predict the attribution maps of expensive explainers. We train and test Empirical Explainers in the language domain and find that they model their expensive counterparts surprisingly well, at a fraction of the cost. They could thus mitigate the computational burden of neural explanations significantly, in applications that tolerate an approximation error.

§ INTRODUCTION

In recent years, important works were published on the ecological impacts of artificial intelligence and deep learning in particular, e.g. Strubell et al., 2019, Schwartz et al., 2020,  Henderson et al., 2020. Research is focused on the energy hunger of model training and subsequent inference in production. Besides training and in-production inference, explainability has become an integral phase of many neural systems.

In the ongoing discussion about Green AI we see explainability neglected.
Conversely, in the explainability community, even though research on efficiency is an active area, apparently the discussion is currently shaped by other aspects, such as faithfulness and plausibility [Jacovi and Goldberg, 2020].
This is surprising because to explain a single model output, many prominent explanation methods, in particular many feature attribution methods (cf. below), require a multiple of computing power when compared to the prediction step.

§.§ Motivation: Expensive Explainers

Take, for instance, the demonstrative but arguably realistic case of a classifier that was trained on $100k$ instances for 10 epochs. The training thus amounts to at least $1M$ forward passes and $1M$ backward passes. To produce explanations, in this paper, we consider feature attribution methods and focus on Integrated Gradients (IG) [Sundararajan et al., 2017] and Shapley Values (SV) [Castro et al., 2009], which are popular and established but also computationally expensive. To compute the exact IG or SV is virtually intractable, which is why sampling-based approximations were devised. For IG

\begin{equation*}
\label{eq:ig}
\phi_{f,i}(x) = \frac{x_{i} - \bar{x}_{i}}{s}\sum_{k=1}^{s} \frac{\partial f(\bar{x} + \frac{k}{s}(x - \bar{x}))}{\partial x_{i}}
\end{equation*}

is computed, where $x$ is the input to model $f$, $\bar{x}$ is a user-defined baseline, $s$ denotes the number of samples (a hyperparameter), and $\phi_{f,i}(x)$ denotes the attribution score of feature $i$. For SV, $s$ permutations of the input data $O_{1}, O_{2}, \dots O_{s}$ are drawn and then features from $x$ are added to a user-defined baseline,[There are several variants of Shapley Value Sampling. This sampling method is based on PyTorch's Captum [Kokhlikyan et al., 2020] library, that we also use for our experiments: <https://captum.ai/api/shapley_value_sampling.html>, last accessed March 26, 2021.] in the order they occur in the permutation. Let Pre$^{i}(O)$ denote the baseline including the features that were added to the baseline prior to $i$. The Shapley value can then be approximated by

\begin{equation*}
\phi_{f,i}(x) =\frac{1}{s}\sum_{k=1}^{s}f(\text{Pre}^{i}(O_{k})\cup x_{i}) - f(\text{Pre}^{i}(O_{k}))
\end{equation*}

Sundararajan et al., 2017 report that $s$ between $20$ and $300$ is usually enough to approximate IG. Let us set $s:=20$. This requires 40 passes (forward and backward) through model $f$ to explain a single instance in production and furthermore, after only 50k explanations the computational costs of training are also already surpassed. In the case of SV, again setting $s:=20$ and assuming only $512$ input features (i.e. tokens to an NLP model), one already needs to conduct $20 * 512 = 10240$ passes to generate an input attribution map for a single classification decision.
This means that SV surpasses the training costs specified above after only $195$ explanations.

This may only have a small impact if the number of required explanations is low. However, there are strong indications that explainability will take (or retain) an important role in many neural systems: For example, there are legal regulations, such as
the EU's GDPR which hints at a “right to explanation” [Goodman and Flaxman, 2017]. For such cases, a 1:1 ratio in production between model outputs and explanations is not improbable.
If the employed explainability method requires more than one additional pass through the model (as many do, cf. below), there then is a tipping point at which the energy need of explanations exceeds the energy needs of both model training and in-production inference.

IG and SV are not the only tipping point methods. Other expensive prominent and recent methods and variants are proposed by Zeiler and Fergus, 2014, Ribeiro et al., 2016, Lundberg and Lee, 2017, Smilkov et al., 2017, Chen et al., 2019, Dhamdhere et al., 2020, Erion et al., 2019, Covert and Lee, 2020, Schwarzenberg and Castle, 2020, Schulz et al., 2020, Harbecke and Alt, 2020.

All of the above listed explainers require more than one additional pass through the model. This is why in general the following

should hold across methods: The smaller the model, the greener the explanation. In terms of energy efficiency, explainability therefore benefits from model compression, distillation, or quantization. These are dynamic fields with a lot of active research which is why in the remainder of this paper we instead focus on something else: The mitigation of the ecological impact of tipping-point methods that dominate the cost term in the example cited in this section.

These are our main contributions in this paper:

* We propose to utilize the task of feature attribution modelling to efficiently model the attribution maps of expensive explainers.

* We address feature attribution modelling with trainable explainers that we coin Empirical Explainers.

* We evaluate their performance qualitatively and quantitatively in the language domain and establish them as an efficient alternative to computationally expensive explainers in applications where an approximation error is tolerable.

§ FRAMEWORK: EMPIRICAL EXPLAINERS

Informally, an empirical explainer is a model that has learned from data to efficiently model the feature attribution maps of an expensive explainer. For training, one collects sufficiently many attribution maps from the expensive explainer and then maximizes the likelihood of these target attributions under the Empirical Explainer.

An expensive explainer may, for instance, be a costly attribution method such as Integrated Gradients that is used to return attributions for the decisions of a classifier, say, a BERT-based [Devlin et al., 2019]
sentiment classifier. The corresponding Empirical Explainer could be a separate neural network, similar in size to the sentiment classifier, consuming the same input tokens as the sentiment model, but instead of predicting the sentiment class, it is trained to predict the integrated gradients for each token.

Whereas the original Integrated Gradients explainer requires multiple passes through the classifier, producing the empirical integrated gradients
requires just one pass through the similarly sized Empirical Explainer.
Empirical explanations come with an accuracy-efficiency trade-off that we discuss in the course of a more formal definition of Empirical Explainers.

For the more formal definition, we need to fix notation first. Let $E_{f}: {\rm I\!R}^{d} \rightarrow {\rm I\!R}^{d}$ be the expensive explainer that maps inputs onto attributions.
Furthermore, let an Empirical Explainer be a function $e_{\theta}:  {\rm I\!R}^{d} \rightarrow {\rm I\!R}^{d}$, parametrized by $\theta$, which also returns attribution maps. Let $||\cdot||$ be a penalty for the inefficiency of a computation, e.g. a count of floating point operations, energy consumption or number of model passes needed. Furthermore, let us assume, without the loss of generality, that $||E_{f}(x)|| >= ||e_{\theta}(x)||$ always holds; i.e., the Empirical Explainer – which we develop and train – is
never more inefficient than the original, expensive explainer. Let $D: {\rm I\!R}^{d} \times {\rm I\!R}^{d} \rightarrow [0,1]$ be a similarity measure, where $D(l,m) = 0$ if $l=m$, for $l,m \in {\rm I\!R}^{d}$ and $\alpha, \beta \in [0,1]$ with $\alpha + \beta = 1$. For data $\mathbf{X}$, we define an $\alpha$-optimal Empirical Explainer by the $\argmin_{\theta \in \Theta}$

\begin{align}
\label{eq:maxlike}
\frac{1}{|X|}\sum_{x\in\mathbf{X}}\overbrace{\alpha D(E_{f}(x),e_{\theta}(x))}^{\text{accuracy}} + \overbrace{\beta \left(\frac{||e_{\theta}(x)||}{||E_{f}(x)||}\right) }^{\text{ efficiency}}.
\end{align}

§.§ Properties

The first term describes how accurately the Empirical Explainer $e_{\theta}$ models the expensive explainer $E_{f}$. The second term compares the efficiency of the two explainers. For $\alpha = 1$, efficiency is considered unimportant and $e_{\theta} := E_{f}$ can be set to minimize Eq. <ref>. $\alpha < 1$ allows to optimize efficiency at the cost of accuracy, which brings about the trade off: One may not succeed in increasing
efficiency while maintaining accuracy.
In fact, there is generally no exact guarantee for how accurately $e_{\theta}$ models $E_{f}$ for new data.

Furthermore, while several expensive explainers, such as Integrated Gradients or Shapley Values, were developed axiomatically to have desirable properties, Empirical Explainers are derived from data – empirically. Consequently, the evidence and guarantees Empirical Explainers offer for their faithfulness to the downstream model are empirical in nature and upper-bound by the faithfulness of the expensive explainer used to train them.

We point this out explicitly because we would like to emphasize that we do not regard an Empirical Explainer a new explainability method, nor do we argue that it can be used to replace the original expensive explainer everywhere. There are certainly situations for which Empirical Explainers are unsuitable for any $\alpha \neq 1$; critical cases in which explanations must have guaranteed properties.

Nevertheless, we still see a huge potential for Empirical Explainers where approximation errors are tolerable: Consider, for instance, a search engine powered by a neural model in the back-end. Without the need to employ the expensive explainer, Empirical Explainers can efficiently provide the user with clues about what the model probably considers relevant in their query (according to the expensive explainer).

§ EXPERIMENTS

In this section, we report on the performance of Empirical Explainers that we trained and tested in the language domain.
We conducted tests with two prominent and expensive explainers, Integrated Gradients and Shapley Value Samples, varying the experiments across four state-of-the-art language classifiers, trained on four different tasks.

The experiments address the question of whether or not it is feasible – in principle – to train efficient Empirical Explainers while achieving significant accuracy. All experiments, code, models and data are open source and can be retrieved following <https://github.com/DFKI-NLP/emp-exp>.
The most important choices are documented in the following paragraphs. Before going into greater detail, it is noteworthy that Eq. <ref> provides a theoretical framework which one does not have use directly for explicit optimization. For example, in this work, we address the first objective, accuracy, by fitting Empirical Explainers to the attribution maps of expensive explainers. However, the second objective, efficiency, is addressed implicitly by design, i.e. the Empirical Explainers, in contrast to their expensive counterparts, are designed (and trained) in a way s.t. only a single forward pass is required through a model similar in size to the downstream model. In the future, it would be very interesting to fully incorporate Eq. <ref> in a differential setting, i.e. also optimize for efficiency automatically, rather than by manual design.

We trained four Empirical Explainers. All explainers consume only the input tokens to the downstream model and return an attribution score for each token. The first one (EmpExp-BERT-IG) was trained to predict integrated gradients w.r.t. the input tokens to a BERT-based IMDB movie review  [Maas et al., 2011]
classifier. For the second Empirical Explainer (EmpExp-XLNet-SV), we varied the downstream model architecture, task and target explainer: EmpExp-XLNet-SV predicts the Shapley Values (as returned by the expensive Shapley Value Sampling explainer) for the inputs of an XLNet-based [Yang et al., 2019] natural language inference classifier that was trained on the SNLI [Bowman et al., 2015]
dataset. The third (EmpExp-RoBERTa-IG) and fourth (EmpExp-ELECTRA-SV) empirical explainers again approximate IG and SV, but for a RoBERTa-based news topic classifier [Liu et al., 2019] trained on the AG News dataset [Zhang et al., 2015] and an ELECTRA (small)-based model [Clark et al., 2020] that detects paraphrases, trained on the PAWS dataset (subset “labelled_final”) [Zhang et al., 2019], respectively.

The Empirical Explainers were trained on target attributions that we generated with IG and SV with $s:=20$ samples. For EmpExp-RoBERTa-IG, we used $s:=25$ due to a slower convergence rate (cf. below). Explanations were generated for the output neuron with the maximal activation. EmpExp-BERT-IG was trained with early stopping using the IG attribution maps for the full IMDB train split (25k). EmpExp-XLNet-SV was trained with around 100k SV attribution maps for the SNLI train split with early stopping, for which we used the 10k attribution maps for the validation split. We did not use all training instances in the split for EmpExp-XLNet-SV, due to the computational costs of Shapley Value Sampling. In case of EmpExp-RoBERTa-IG and EmpExp-ELECTRA-SV it was possible to train with the full train splits again, 108k (12k held out) and around 50k instances, respectively.

As mentioned above, the expensive explainers require user-defined baselines. For the baselines, we replaced all non-special tokens in the input sequence with pad tokens. For the expensive IG, we produced attribution maps for the embedding layer and projected the attribution scores onto tokens by summing them over the token dimension. For the expensive SV, the input IDs were perturbed. During perturbation, we grouped and treated special tokens (CLS, SEP, PAD, ...) in the original input as one feature to accelerate the computation.

In architectural terms, the Empirical Explainers are very similar to the downstream models: We heuristically decided to copy the fine-tuned BERT, XLNet, RoBERTa and ELECTRA encoders from the classifiers and instead of the classification layers on top, we initialized new fully connected layers with $T$ output neurons. $T$ was lower bound by the maximum input token sequence length to the downstream model in the respective dataset: $T=512$ for BERT/IMDB and RoBERTa/AG News, $T=130$ for XLNet/SNLI, and $T=145$ for ELECTRA/PAWS. All input sequences were padded to $T$ and we did not treat padding tokens different from other tokens, when training the Empirical Explainers. Please note that the sequence length has a considerable impact on the runtime of the SV explainer in particular, which is why limiting $T$ increases comparability.

We trained the Empirical Explainers to output the right (in accordance with the expensive explainers) attribution scores for the input tokens, using an MSE loss between $E_{f}(x_{1})\dots E_{f}(x_{T})$ and $e_{\theta}(x_{1})\dots e_{\theta}(x_{T})$
where $x = x_{1}, x_{2}, \dots x_{T}$ is a sequence of input tokens.[Even though we do not solve for Eq. <ref> directly, please note that for evaluation we can normalize the attribution scores to $[0,1]$ prior to computing the MSE and this way force the MSE into the interval $[0,1]$ to comply with the constraints for Eq. <ref>.]

To put the performance of an Empirical Explainer into perspective, we propose the following baseline, which is the strongest we can think of: We take the original expensive explainer with a reduced number of samples as the baseline. To position the Empirical Explainer against this alternative energy saving strategy, we compute convergence curves. Starting with $s=1$, we incrementally increase the number of samples until $s=19$ ($s=24$ in the case of EmpExp-RoBERTa-IG) and collect attribution maps from the expensive explainer for the different choices of $s$. We then compute the MSEs of these attribution maps when compared to the target attributions (with $s=20$ or $s=25$). We average the MSEs across the test split. The same is done for the Empirical Explainer.

§ RESULTS & DISCUSSION

In the following, we report the experimental results, divided into the aspects of task performance, explanation efficiency and explanation accuracy.

Task Performance  On the test splits, the classifiers we trained achieved  weighted F$_{1}$ scores of 0.93 (BERT $\cdot$ IMDB), 0.90 (XLNet $\cdot$ SNLI), 0.94 (RoBERTa $\cdot$ AG News) and 0.92 (ELECTRA $\cdot$ PAWS).

Explanation Efficiency  Regarding the efficiency term in Eq. <ref>, in terms of model passes, the Empirical Explainers have a clear advantage over their expensive counterparts. For IG with $s=20$, 40 model passes are required, for $s=25$, 50 passes. For SV with $s=20$, assuming a token sequence length of 100 for the purpose of discussion, 2000 model passes are required. For the empirical explanations, only one (additional) forward pass through a similarly sized model (the Empirical Explainer) is necessary.

Contrary to runtime (and energy consumption) measures, the number of required model passes is largely invariant of available hardware and implementation details. For the sake of completeness, we nevertheless also report our runtimes in appendix <ref>. In summary, generating the expensive explanations for the test splits took between around 02:15 and 48 hours, whereas the empirical explanations only required between 02:05 and 07:14 minutes.

The runtimes are not definitive, however. We were unable to establish a fair game for the explainers. For example, due to implementation details and memory issues we explained the data instance-wise with the expensive explainers while our Empirical Explainers easily allowed batch processing. We expect that the expensive explainers can be accelerated but due to the larger number of model passes required, they will very likely not outperform their empirical counterparts.

The Empirical Explainers come with additional training costs, which we also report in appendix <ref>. Training took between 02:15 and 07:00 hours. These additional training costs are thus quickly outweighed by the expensive explainers, in particular in a continuous in-production setting.

Explanation Accuracy  Regarding the accuracy term in Eq. <ref>, Figs. <ref>, <ref>, <ref> and <ref> provide anecdotal qualitative evidence that the Empirical Explainers are capable of modelling their expensive counterparts well, with varying degrees of approximation errors. Alongside this paper, we provide four files (see repository) with around 25k (IMDB), 10k (SNLI), 7.6k (AG News) and 8k (PAWS) lines, each of which contains an HTML document that depicts a target attribution and its empirical counterpart from the test set. The heatmaps in the figures mentioned above are taken from the accompanying files.

Figs. <ref> and <ref> are instances of what we consider surprisingly accurate approximations of the expensive target attribution maps, despite challenging inputs. Let us first consider Fig. <ref> in greater detail. Consider the tokens  and  that are not attributed much importance by the expensive explainer (IG, top) but could be considered signal words for the positive and negative class, respectively and thus pose a challenge for the Empirical Explainer. Nevertheless, in accordance with the expensive target explainer, the Empirical Explainer (bottom) does not attribute the classifier output primarily to these tokens but instead accurately assigns a lot of weight to .

A similar phenomenon can be observed in Fig. <ref> for the token . The approximation in this figure, however, also contains a prominent approximation error. The Empirical Explainer erroneously attributes a salient negative score to the token  while the target explainer does not highlight that token. Similarly, in Fig. <ref> the Empirical Explainer returns a negative score for , whereas the expensive target explainer has returned a positive score.

We suspect that such errors may result from global priors that the Empirical Explainers have learned and that sometimes outweigh the instantaneous information. For instance, in Fig. <ref> the verb  in the premise in conjunction with the (conjugated) verb  in the hypothesis may statistically be indicative of an entailment in the training data. This is because to produce a contradiction the verb sometimes is simply replaced by another one (cf. Fig. <ref>:  vs. ). In this instance, however, the verb is not replaced. Thus, here the prior knowledge of the Empirical Explainer may outweigh the local information in favor of the error that we observe: The Empirical Explainer may signal that  is evidence against the class  since it finds it in the premise and hypothesis. A similar argument can be put forward for the case of  in Fig. <ref>.

The above points are rather speculative. A more objective and quantitative analysis of the efficiency/accuracy trade-off is provided in Fig. <ref>. The left column depicts the MSE lines of IG for an increasing number of samples in $x$-direction. We observe that IG converges fast in case of BERT/IMDB. (This may be due to saturation effects in Integrated Gradients, reported on by Miglani et al., 2020.) In case of RoBERTa/AG News we found a slower convergence rate, which is why we increased the number of samples for the target explainer. We observe that in both cases, the empirical integrated gradients (dashed lines) perform favourably: To outperform the Empirical Explainer by decreasing $s$, in case of BERT/IMDB one needs to set $s>5$ which entails $10$ model passes as opposed to the single additional pass through the Empirical Explainer for the empirical explanations.  Furthermore, the approximation error is already marginal at the intersection of expensive and empirical line. In case of RoBERTa/AG News, one even needs to set $s:=18$ to be closer to the target than the Empirical Explainer.

A similar trend can be observed for the (empirical) Shapley Values in the right column of Fig. <ref>. In case of XLNet/SNLI, however, the intersection occurs only after $s=10$ which means that the Empirical Explainer needs only $\frac{1}{11*100} = 0.9\%$ of model passes (plus the pass through its output layer) when compared to the next best expensive explainer, again assuming 100 input tokens for the purpose of discussion. In case of ELECTRA/PAWS, the Empirical Explainer even beats the expensive explainer with just one sample less than the target.

In summary, we take the experimental results as a strong indication that Empirical Explainers could become an efficient alternative to expensive explainers (in the language domain) where approximation errors are tolerable.

§ RELATED WORK

The computational burden of individual explainability methods was addressed in numerous works. As mentioned above, Integrated Gradients can only be computed exactly in limit cases and for all other cases, the community relies on the approximate method proposed by Sundararajan et al., 2017. Similarly, Shapley Values can rarely be computed precisely which is why Shapley Value Sampling was investigated, e.g. by  Castro et al., 2009, Štrumbelj and Kononenko, 2010. Shapley Value Sampling was later unified with other methods under the SHAP framework [Lundberg and Lee, 2017] which yielded the method KernelSHAP that showed improved sample efficiency. Covert and Lee, 2020 then analysed the convergence behaviour of KernelSHAP and again further improved runtime. Chen et al., 2019 introduced L-Shapley and C-Shapley which accelerate Shapley Value Sampling for structured data, such as dependency trees in NLP.

Thus, computational feasibility appears to be a driving force in the research community, already. To the best of our knowledge, however, we are the first to propose the task of feature attribution modelling to mitigate the computational burden of expensive explainers with Empirical Explainers.

Technically, self-explaining models [Alvarez-Melis and Jaakkola, 2018] are related to our approach in that they also generate explanations in a forward pass (alongside their classification decision). Contrary to self-explaining models, Empirical Explainers can be employed after training for a variety of black box and white box classifiers and explainers.

A source of inspiration for our method was the work by Camburu et al., 2018. The authors train self-explaining models that return a natural language rationale alongside their classification. Thus, they, too, train an explainer. However, their target explanations (natural language) differ substantially from the ones Empirical Explainers are trained with (attribution scores).

Furthermore, related to our work is the technique of gradient matching for which a network's (integrated) gradients are compared to a target attribution, i.e. a human prior, and then the network's parameters are updated, s.t. the gradients move closer to the target, as done e.g. by Ross et al., 2017, Erion et al., 2019, Liu and Avci, 2019. Apart from the loss on an alignment with target attributions, our method and goals diverge from theirs significantly.

Human priors and expensive target explanations have recently also been used for explanatory interactive learning (XIL). Like Empirical Explainers, XIL is motivated by the expensiveness of a target explainer; in the case of XIL this is a human in the loop. Schramowski et al., 2020 present humans with informative (cf. active learning) instances, the model prediction and an explanation for the prediction

The expensive human feedback is then used to improve the model.
Apart from the expensive explainer assumption, their approach differs substantially from ours. Very recently, Behrens et al., 2021 contributed to XIL by introducing a method that learns to explain from explanations and in this respect is close to the setting of Empirical Explainers. One fundamental difference between ours and their work is that they propose and focus on a specific class of self-explainable models  whereas Empirical Explainers make no assumptions about the underlying predictor and are intended for a variety of model classes, as already mentioned above.

Very recently again, Rajagopal et al., 2021 proposed local interpretable layers as a means to generate concept attributions which in parts aligns with our method, even though their target attributions and task objectives are very different again.

Lastly, Empirical Explainers can be viewed as a form of knowledge distillation [Hinton et al., 2015]. However, contrary to the established approach, we do not assume a parametric teacher network that knowledge is distilled from. Very recently we became aware of the work by
Pruthi et al., 2020 who boost the accuracy of a student learner with explanations in the form of a subset of tokens that are relevant to the teacher decision, determined by an explainer. In a sequence classification task, the student is trained to identify the relevant tokens and could thus be considered an Empirical Explainer. The task, however, is not to predict the original attribution map and the overall objective differs significantly from ours again.

§ CONCLUSION & FUTURE DIRECTIONS

In this paper, we take a step towards greener XAI by again reviving energy efficiency as an additional criterion by which to judge an explainability method, alongside important aspects such as faithfulness and plausibility. In this context, we propose feature attribution modelling with efficient Empirical Explainers. In the language domain, we investigate the efficiency/accuracy trade-off and find that it is possible to generate empirical explanations with significant accuracy, at a fraction of the costs of the expensive counterparts. We take this as a strong indication that Empirical Explainers could be a viable alternative to expensive explainers where approximation errors are tolerable.

Regarding future directions: The Empirical Explainers we trained are our concrete model choices. The framework we propose allows for many other approaches. For instance, one could provide the Empirical Explainers with additional information, such as the gradient w.r.t. the inputs. This would require an additional pass through the model but may possibly further boost accuracy.

We would like to note that we trained and tested our Empirical Explainers only on in-domain data but their behaviour on out-of-domain data should be investigated, too. Fortunately, since we explain the model decision (the maximum output activation), no gold labels are required to train Empirical Explainers which facilitates data collection immensely. Finally, there are some more sample efficient explainers that should be considered, too.

AcknowledgementsWe would like to thank Christoph Alt, Steffen Castle, Jonas Mikkelsen and David Harbecke for their helpful feedback. This work has been supported by the German Federal Ministry of Education and Research as part of the projects XAINES (01IW20005) and CORA4NLP (01IW20010).

[Alvarez-Melis and Jaakkola, 2018]
David Alvarez-Melis and Tommi Jaakkola. 2018.
Towards robust interpretability with self-explaining neural networks.
Advances in Neural Information Processing Systems,

[Behrens et al., 2021]
Freya Behrens, Stefano Teso, and Davide Mottin. 2021.
Bandits for learning to explain from explanations.
arXiv preprint arXiv:2102.03815.

[Bowman et al., 2015]
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning.
A large annotated corpus for learning natural language inference.
In EMNLP.

[Camburu et al., 2018]
Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom.
e-snli: Natural language inference with natural language
Advances in Neural Information Processing Systems,

[Castro et al., 2009]
Javier Castro, Daniel Gómez, and Juan Tejada. 2009.
Polynomial calculation of the shapley value based on sampling.
Computers & Operations Research, 36(5):1726–1730.

[Chen et al., 2019]
Jianbo Chen, Le Song, Martin J Wainwright, and Michael I Jordan. 2019.
L-shapley and c-shapley: Efficient model interpretation for
structured data.
International Conference on Learning Representations 2019.

[Clark et al., 2020]
Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. 2020.
Electra: Pre-training text encoders as discriminators rather than
arXiv preprint arXiv:2003.10555.

[Covert and Lee, 2020]
Ian Covert and Su-In Lee. 2020.
Improving kernelshap: Practical shapley value estimation via linear
arXiv preprint arXiv:2012.01536.

[Devlin et al., 2019]
J. Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.
Bert: Pre-training of deep bidirectional transformers for language
In NAACL-HLT.

[Dhamdhere et al., 2020]
Kedar Dhamdhere, Ashish Agarwal, and Mukund Sundararajan. 2020.
The shapley taylor interaction index.
In ICML 2020.

[Erion et al., 2019]
Gabriel Erion, Joseph D Janizek, Pascal Sturmfels, Scott Lundberg, and Su-In
Lee. 2019.
Learning explainable models using attribution priors.
arXiv preprint arXiv:1906.10670.

[Goodman and Flaxman, 2017]
Bryce Goodman and Seth Flaxman. 2017.
European union regulations on algorithmic decision-making and a
“right to explanation”.
AI magazine, 38(3):50–57.

[Harbecke and Alt, 2020]
David Harbecke and Christoph Alt. 2020.
Considering likelihood in nlp classification explanations with
occlusion and language modeling.
ACL 2020 SRW.

[Henderson et al., 2020]
P. Henderson, Jie-Ru Hu, Joshua Romoff, Emma Brunskill, Dan Jurafsky, and
Joelle Pineau. 2020.
Towards the systematic reporting of the energy and carbon footprints
of machine learning.
ArXiv, abs/2002.05651.

[Hinton et al., 2015]
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015.
Distilling the knowledge in a neural network.
arXiv preprint arXiv:1503.02531.

[Jacovi and Goldberg, 2020]
Alon Jacovi and Y. Goldberg. 2020.
Towards faithfully interpretable nlp systems: How should we define
and evaluate faithfulness?
ACL 2020.

[Kokhlikyan et al., 2020]
Narine Kokhlikyan, Vivek Miglani, Miguel Martin, Edward Wang, Bilal Alsallakh,
Jonathan Reynolds, Alexander Melnikov, Natalia Kliushkina, Carlos Araya, Siqi
Yan, et al. 2020.
Captum: A unified and generic model interpretability library for
arXiv preprint arXiv:2009.07896.

[Liu and Avci, 2019]
Frederick Liu and Besim Avci. 2019.
Incorporating priors with feature attribution on text classification.
In Proceedings of the 57th Annual Meeting of the Association
for Computational Linguistics, pages 6274–6283.

[Liu et al., 2019]
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer
Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.
Roberta: A robustly optimized bert pretraining approach.
arXiv preprint arXiv:1907.11692.

[Lundberg and Lee, 2017]
Scott M Lundberg and Su-In Lee. 2017.
A unified approach to interpreting model predictions.
In Advances in neural information processing systems, pages

[Maas et al., 2011]
Andrew L. Maas, Raymond E. Daly, P. T. Pham, D. Huang, A. Ng, and Christopher
Potts. 2011.
Learning word vectors for sentiment analysis.
In ACL 2011.

[Miglani et al., 2020]
Vivek Miglani, Narine Kokhlikyan, Bilal Alsallakh, Miguel Martin, and Orion
Reblitz-Richardson. 2020.
Investigating saturation effects in integrated gradients.
arXiv preprint arXiv:2010.12697.

[Pruthi et al., 2020]
Danish Pruthi, Bhuwan Dhingra, Livio Baldini Soares, Michael Collins, Zachary C
Lipton, Graham Neubig, and William W Cohen. 2020.
Evaluating explanations: How much do explanations from the teacher
aid students?
arXiv preprint arXiv:2012.00893.

[Rajagopal et al., 2021]
Dheeraj Rajagopal, Vidhisha Balachandran, E. Hovy, and Yulia Tsvetkov. 2021.
Selfexplain: A self-explaining architecture for neural text
arXiv preprint arXiv:2103.12279.

[Ribeiro et al., 2016]
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016.
" why should i trust you?" explaining the predictions of any
In Proceedings of the 22nd ACM SIGKDD international conference
on knowledge discovery and data mining, pages 1135–1144.

[Ross et al., 2017]
Andrew Slavin Ross, Michael C Hughes, and Finale Doshi-Velez. 2017.
Right for the right reasons: Training differentiable models by
constraining their explanations.
In IJCAI.

[Schramowski et al., 2020]
Patrick Schramowski, Wolfgang Stammer, Stefano Teso, Anna Brugger, Franziska
Herbert, Xiaoting Shao, Hans-Georg Luigs, Anne-Katrin Mahlein, and Kristian
Kersting. 2020.
Making deep neural networks right for the right scientific reasons by
interacting with their explanations.
Nature Machine Intelligence, 2(8):476–486.

[Schulz et al., 2020]
Karl Schulz, Leon Sixt, Federico Tombari, and Tim Landgraf. 2020.
Restricting the flow: Information bottlenecks for attribution.
ICLR 2020.

[Schwartz et al., 2020]
Roy Schwartz, Jesse Dodge, N. A. Smith, and Oren Etzioni. 2020.
Green ai.
Communications of the ACM, 63:54 – 63.

[Schwarzenberg and Castle, 2020]
Robert Schwarzenberg and Steffen Castle. 2020.
Pattern-guided integrated gradients.
ICML 2020 Workshop on Human Interpretability in Machine
Learning (WHI) arXiv preprint arXiv:2007.10685.

[Smilkov et al., 2017]
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin
Wattenberg. 2017.
Smoothgrad: removing noise by adding noise.
arXiv preprint arXiv:1706.03825.

[Strubell et al., 2019]
Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019.
Energy and policy considerations for deep learning in NLP.
In Proceedings of the 57th Annual Meeting of the Association
for Computational Linguistics, pages 3645–3650, Florence, Italy.
Association for Computational Linguistics.

[Štrumbelj and Kononenko, 2010]
E. Štrumbelj and I. Kononenko. 2010.
An efficient explanation of individual classifications using game
J. Mach. Learn. Res., 11:1–18.

[Sundararajan et al., 2017]
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017.
Axiomatic attribution for deep networks.
In Proceedings of the 34th International Conference on Machine
Learning-Volume 70, pages 3319–3328.

[Yang et al., 2019]
Z. Yang, Zihang Dai, Yiming Yang, J. Carbonell, R. Salakhutdinov, and Quoc V.
Le. 2019.
Xlnet: Generalized autoregressive pretraining for language
In NeurIPS.

[Zeiler and Fergus, 2014]
Matthew D Zeiler and Rob Fergus. 2014.
Visualizing and understanding convolutional networks.
In 13th European Conference on Computer Vision, ECCV 2014,
pages 818–833. Springer Verlag.

[Zhang et al., 2015]
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015.
Character-level convolutional networks for text classification.
Advances in neural information processing systems,

[Zhang et al., 2019]
Yuan Zhang, Jason Baldridge, and Luheng He. 2019.
Paws: Paraphrase adversaries from word scrambling.
arXiv preprint arXiv:1904.01130.

§ RUNTIMES

Generating the expensive target explanations for the official IMDB test split (25k instances, $T=512$, $s=25$, BERT, Titan V) with Integrated Gradients took us 7:17 hours

(6:22 hours for the 22500 training instances). Generating the expensive Shapley Values for the SNLI test split ($\sim$ 10k instances, $T=130$, $s=20$, XLNet, Quadro P5000) took us 48:22 hours

(and over 600 GPU hours for under 100k training instances).
It took us over 2:15 hours to explain the 7600 instances in the test split of the AG News dataset with IG ($T=512$, $s=25$, RoBERTa, RTX2080Ti; over 31 hours for the train split). For the PAWS test split ($T=145$, $s=20$, 8k instances, ELECTRA (small), RTX6000) we needed over 18 GPU hours

(over 126 GPU hours for the 49401 instances in the train split, using NVIDIA's RTX3090 and RTX2080Ti).

In contrast, generating the empirical explanations took us only 07:14 minutes for the IMDB test split on the Titan GPU and only

02:05 minutes for SNLI test split on the Quadro P5000 GPU.  The AG News test split took 03:16 minutes to explain (RTX3090) and the PAWS test split was explained empirically in only 02:19 minutes (RTX3090).

The training of EmpExp-BERT-IG terminated after 10 epochs (Titan V), which took less than 4 hours. EmpExp-XLNet-SV (Quadro P5000), EmpExp-RoBERTa-IG (RTX3090), and EmpExp-ELECTRA-SV (RTX3090) terminated after 7, 3, and 8 epochs, respectively (<7 hours, < 2:15 hours, and <1 hours).