#  SciTab: A Challenging Benchmark for Compositional Reasoning and Claim
Verification on Scientific Tables

Xinyuan Lu$\>\,{}^{1,2}$ Liangming Pan††footnotemark: $\>\,{}^{3}$ Qian Liu4
Preslav Nakov5 Min-Yen Kan2
1ISEP Program, NUS Graduate School 2 National University of Singapore
3University of California, Santa Barbara 4Sea AI Lab 5MBZUAI
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>Equal Contribution.

###### Abstract

Scientific fact-checking is crucial for ensuring the accuracy, reliability,
and trustworthiness of scientific claims. However, existing benchmarks are
limited in terms of their claim diversity, reliance on text-based evidence,
and oversimplification of scientific reasoning. To address these gaps, we
introduce SciTab, a novel dataset comprising of 1,225 challenging scientific
claims requiring compositional reasoning with scientific tables. The claims in
SciTab are derived from actual scientific statements, and the evidence is
presented as tables, closely mirroring real-world fact-checking scenarios. We
establish benchmarks on SciTab using state-of-the-art models, revealing its
inherent difficulty and highlighting limitations in existing prompting
methods. Our error analysis identifies unique challenges, including ambiguous
expressions and irrelevant claims, suggesting future research directions. The
code and the data are publicly available at
https://github.com/XinyuanLu00/SciTab.

##  1 Introduction

Scientific fact-checking is a crucial process that involves validating the
accuracy of scientific claims by cross-referencing them with established
scientific literature, research, or data Guo et al. (2022). This process is
crucial for preserving the integrity of scientific information, preventing the
spread of misinformation, and fostering public trust in research findings.
However, the sheer volume of scientific data and claims can be overwhelming
for manual fact-checking, making automated scientific fact-checking an
imperative research area of Natural Language Processing (NLP).

Scientific fact-checking has advanced significantly with benchmarks including
Sci-Fact Wadden et al. (2020), Sci-Fact Open Wadden et al. (2022a), and COVID-
Fact Saakyan et al. (2021). However, these datasets still exhibit several
limitations. First, the claims are crowd-sourced rather than collected from
real scientific papers. This leads to problems such as bias in human
annotation, a lack of diversity, and shallow claims that do not reflect the
complexity of scientific reasoning. For example, most claims in Sci-Fact can
be validated by a single sentence in a paper’s abstract, which oversimplifies
the scientific discourse. Second, the claims in the existing benchmarks are
solely validated against text-based evidence, primarily paper abstracts. In
much scientific processes, claims are intrinsically tied to quantative
experimental data, commonly presented in tables and figures. This disparity
highlights a significant gap between the existing benchmarks and real-world
scientific fact-checking needs. To bridge these gaps, a dataset that 1)
compiles real-world claims from scientific papers, and 2) includes original
scientific data such as tables and figures, is needed.

Figure 1: An example of our SciTab dataset (left) and its corresponding
reasoning graph (right). Each data entry contains paper, paper id, table, and
one of the {Supported, Refuted, Not Enough Info} claims and its label.

In this paper, we propose a novel dataset SciTab, which fulfills these stated
criteria. It contains 1,225 challenging scientific claims, each demanding
compositional reasoning for verification using scientific tables. Our data is
derived from the SciGen dataset Moosavi et al. (2021), a resource that
includes scientific tables and claims crawled from arXiv.org. We first
manually filter out the check-worthy scientific claims from the raw data.
Following this, we employ a strategy of human–model collaboration, as depicted
in Figure 2, to generate claims that are either contradicted or unverifiable
based on the table’s content. Figure 1 shows a claim from SciTab and the
corresponding reasoning process to verify it. Compared with existing
benchmarks, SciTab is closer to real-world scientific fact-checking in terms
of both claim and evidence. The claims are constructed from actual scientific
statements authored by researchers, while the evidence is presented in the
form of scientific tables as opposed to abstracts. Through data analysis, we
show that the claims in SciTab necessitate a more comprehensive and nuanced
set of reasoning skills for verification.

With the $\sim$1.2K challenging, expert-annotated scientific claims, we employ
SciTab as a diagnostic dataset for benchmarking the zero-shot and in-context
learning performance for a wide range of state-of-the-art models, including
table-based pretraining models, encoder–decoder models, decoder-only language
models, and API-based language models. We observe that all models, with the
exception of GPT-4, can only achieve marginally superior $F_{1}$ scores than a
random guess, which underscores the challenging nature of SciTab.
Additionally, established prompting methods like Chain-of-Thought Wei et al.
(2022) and Program-of-Thought Wei et al. (2022) which typically enhance
performance across most reasoning tasks, do not bring performance gain on
SciTab. Our error analysis sheds light on several unique challenges in SciTab
that may lead to this, such as dealing with ambiguous expressions and
irrelevant claims. In the spirit of advancing future research in scientific
fact-checking, we make our dataset fully accessible to the research community.

Figure 2: The human–model collaboration construction process of SciTab, which
contains three steps: 1) data preparation (including data preprocessing and
claim extraction) 2) automatic claim generation (including refuted and Not
Enough Info claim generation) and 3) manual claim verification.

##  2 Related Work

#### Fact-Checking.

Automated fact-checking has emerged as an active research area for combating
the spread of misinformation. Various fact-checking datasets have been
proposed, including human-crafted claims based on Wikipedia Thorne et al.
(2018); Sathe et al. (2020); Schuster et al. (2021) and naturally occurring
claims in the political and scientific domain Saakyan et al. (2021); Gupta and
Srikumar (2021); Wadden et al. (2020, 2022a, 2022b); Wright et al. (2022).
Most of these datasets rely on text evidence to verify claims. Among those few
table-based fact-checking datasets, TabFact Chen et al. (2020) contains
Wikipedia tables as evidence for natural language statements, while FEVEROUS
Aly et al. (2021) uses Wiki articles and annotates evidence in the form of
sentences or cells from tables. SEM-TAB-FACTS Wang et al. (2021) is the only
existing dataset for scientific table fact verification, but it is limited to
simple, crowd-sourced claims. To address these limitations, we construct a new
dataset, SciTab, that contains complex claims from original scientific papers
with table-based evidence.

#### Scientific Document Processing.

Scientific document processing involves various tasks in the analysis and
manipulation of scientific documents Kashyap et al. (2023). One task is
citation contextualization, which aims to identify relevant sections in a
cited document for a given citation in a citing document Cohan et al. (2015);
Ramesh Kashyap and Kan (2020). Another task is biomedical question-answering,
which focuses on answering factual questions based on biomedical literature,
requiring a deep understanding of the relationship between the query and the
relevant information within the scientific documents Jin et al. (2019); Gu et
al. (2020); Nori et al. (2023). Scientific document summarization aims to
generate abstractive or extractive summaries containing background
information, results and the document’s contributions Cachola et al. (2020);
Beltagy et al. (2021); Cohan et al. (2022). These related tasks highlight the
importance of constructing diverse datasets and systems for scientific
document processing. In this context, our paper introduces the scientific
fact-checking dataset SciTab, which focuses on fact-checking claims made in
scientific articles against the tables presented in the same articles,
bridging the gap between textual claims and tabular data.

##  3 The SciTab Dataset

We present a human–model collaboration construction process of SciTab, as
shown in Figure 2. We describe the steps involved in data preparation (Section
3.1), automatic claim generation (Section 3.2), and subsequent human
verification (Section 3.3).

###  3.1 Data Preparation

#### Data Source.

The SciTab is derived from the publicly available SciGen dataset Moosavi et
al. (2021), which consists of computer science papers containing original
tables and descriptions. However, our focus is specifically on claims that
worth verifying, distinct from the provided descriptions. These claims serve
to highlight and comment on key data within the papers. To identify these
claims, we tokenized the descriptions into individual sentences using the
NLTK111https://www.nltk.org/, resulting in 1,301 sentences. Appendix A.1
provides detailed information on the definition of claims in academic writing,
where we adhere to the conventions of highlighting and commenting on key data,
referred to as claims.

#### Claim Extraction.

We then conduct a 3-class classification task where each sentence is manually
assigned one of the labels: {Descriptions, Background, Claim}. Descriptions
and Background are commonly found in academic reading and provide contextual
information. By manually labeling these sentences, we filter out the noise and
extract claims worth verifying from the original dataset. After manually
labeling 200 sentences using the tool (User Interface in Appendix A.2), we
employed a BERT model Devlin et al. (2019) to automatically classify the
remaining sentences. The classification results can be found in Appendix A.3,
with a total of 872 extracted claims.

###  3.2 Automatic Claim Generation

We consider those extracted claims to be Supported claims by default as they
are from the papers themselves. To build a fact verification dataset, we also
need to generate Refuted claims and Not Enough Info (NEI) claims. We now
describe how we generate them.

#### Refuted Claims.

To avoid annotation artifacts and conditional stylistic patterns biases and to
ensure that we had enough Refuted claims, we design an automatic refuted claim
rewriting strategy. We input the Supported claims C into InstructGPT(text-
davinci-003) Ouyang et al. (2022) and edit C with the following prompts:
Please change the original claims to opposite meanings with minimum edits.
After editing, we obtain the generated claims C’.To ensure correct grammar and
semantics, we validate all the generated C’ claims. Finally, we obtain 872
generated claims.

#### NEI Claims.

To generate additional NEI claims, we employ the InstructGPT (text-
davinci-003) model, capitalizing on its ability to generate text based on the
given input. Specifically, we feed the original table evidence into the model,
accompanied by the prompt: Generate 10 claims based on the given table. This
approach produces 900 free-formed claims, contributing to the diversity and
richness of SciTab since they are based on the hallucinations or speculative
responses provided by the LLM. Similar to the validation process for refuted
claims, we ensure that all the generated NEI claims adhere to correct grammar
and semantics.

###  3.3 Manual Claim Verification

We employ a human verification process to ensure the accuracy and reliability
of the generated claims.Due to the relatively low number of claims available,
we employed domain experts to manually verify the generated refuted and NEI
claims.

#### Annotation Task Design.

To verify the generated NEI claims, we ask annotators to verify the claims
using the given table and assign one of the labels from the options
{Supported, Refuted, NEI}. They also need to provide the NEI reasons with one
sentence for each claim. For each generated refuted claim, annotators are
provided with the original supported claim and the generated refuted claim.
They need to verify 1) the original supported claim and 2) the generated
refuted claim by the given table and then choose one of the following three
options: (A) The original claim is not supported. (B) The original claim is
supported by the table, but the generated refuted claim is not refuted by the
table. (C) The original claim is supported by the table, and the generated
refuted claim is refuted by the table.

#### Recruiting Annotators.

Since the tables are from computer science papers, to ensure the annotators
could understand the tables, we recruit domain experts – twelve university
students majoring in computer science with basic math and programming
backgrounds. We ask each annotator to fill in a questionnaire, including their
age, department, maximum workload per week, etc. After that, we provide a
training session to ensure they understand the task and can use the annotation
interfaces (Appendix B.1 and B.2). We also give them three samples to test
their understanding. Once they pass the training session, we assign them
official batches to work on.

To ensure the quality of annotations in SciTab, we apply strict quality
control procedures following the guidelines outlined in the Dataset Statement
Bender and Friedman (2018). We assign two different annotators to perform a
two-round annotation for each claim, while two authors review and address any
identified errors or issues. This rigorous quality control framework
guarantees reliable and accurate annotations while excluding author
involvement in the annotation process. In compliance with ethical guidelines,
we ensure fair compensation for the annotators, as the Dataset Statement
recommends. We hired a total of 12 annotators for the annotation process. Each
claim annotation was reimbursed at a rate of 0.37 USD, resulting in an hourly
wage of 11.2 USD222The compensation provided to the annotators was fair and
aligned with guidelines for dataset creation Bender and Friedman (2018).. It
is important to note that the inter-annotator agreement was calculated after
the authors’ checking process.

#### Inter-annotator Agreement.

To measure the consistency of the annotations, we use Cohen’s Kappa Cohen
(1960), which takes into account both the observed agreement $p_{o}$ and the
expected agreement by chance $p_{e}$, and is calculated as
$k=\frac{p_{o}-p_{e}}{1-p_{e}}$. Our inter-annotator agreement is 0.630 for
872 samples for the refuted claim verification task and 0.719 for 900 samples
for the NEI claim verification task. Both values indicate substantial
agreement among the annotators, with 12 annotators for refuted claims and 10
annotators for NEI claims.

Statistics | TabFact | FEVEROUS | SEM-TAB-FACTS | SciTab
---|---|---|---|---
Domain | Wiki Tables | Wiki Tables | Scientific Articles | Scientific Articles
Annotator | AMT | AMT | AMT | Experts
Max. Reasoning Hops | 7 | 2 | 1 | 11
Veracity Labels |  |  |  |
$\hookrightarrow$ Supported | 54% | 56% | 58% | 37%
$\hookrightarrow$ Refuted | 46% | 39% | 38% | 34%
$\hookrightarrow$ NEI | — | 5% | 4% | 29%
Total # of Claims | 117,854 | 87,026 | 5,715 | 1,225
Avg. claims per table | 7.11 | 0.07 | 5.27 | 6.16
Table 1: Comparison of SciTab to three recent table fact verification
datasets: TabFact Chen et al. (2020), FEVEROUS Aly et al. (2021), and SEM-TAB-
FACTS Wang et al. (2021). The table presents statistics related to the domain,
annotator (AMT represents Amazon Mechanical Turk), maximum reasoning hops,
veracity labels percentage of each dataset, the total number of claims, and
average claims per table.

##  4 Data Analysis

We compared our SciTab dataset it with three recent table fact verification
datasets: TabFact Chen et al. (2020), FEVEROUS Aly et al. (2021), and SEM-TAB-
FACTS Wang et al. (2021). Table 1 summarized the comparison. SciTab is unique
in several ways. It focuses on scientific tables rather than general knowledge
(Wiki tables), it was annotated by domain experts rather than crowd-sourced
workers (Amazon Mechanical Turk), and it requires up to 11 reasoning steps for
claim verification, making it more challenging. Additionally, it has a more
balanced 3-class distribution of veracity labels and a higher percentage of
NEI claims.

To further demonstrate the unique features of our dataset, we conduct an
analysis of the reasoning types, depths, program structures (Section 4.1), and
refuted and NEI reasons (Section 4.2) as follows.

###  4.1 Reasoning Structure Analysis

To gain a better understanding of the challenges and to show the complexity of
the claims presented in SciTab, we conduct an analysis of the types, depths,
and program structure of reasoning required to answer the claims.

#### Reasoning Types.

We have adopted and refined the 14 reasoning types (shown in Table 2) proposed
by INFOTABS Gupta et al. (2020), including simple lookup, numerical reasoning,
check value, trend same/different, and set check. In addition, we have
introduced two new types of reasoning, namely “Closed-domain knowledge” and
“Open-domain knowledge” that are designed specifically for scientific table
fact-checking. Closed-domain knowledge refers to information that is available
within the paper itself, e.g., from table caption or the article, while Open-
domain knowledge refers to domain-specific knowledge that the experts need to
know but which is not explained in the article, e.g., BLEU score. By analyzing
100 samples in SciTab, we manually annotate the required operation type, the
number of required reasoning steps, and the reasoning graph for each claim as
described in the following paragraphs.

Function Names | Descriptions | Prop. (%)
---|---|---
Simple lookup | Retrieve the value for a specific cell. | 20.6
Comparison | Compare two numbers. | 19.5
Closed-domain knowledge | Extract information from context sentences in the table caption or article. | 12.1
Open-domain knowledge | Extract additional information required by domain experts. | 5.3
Commonsense knowledge | Extract commonsense knowledge necessary for claim verification. | 5.3
Subtract | Perform subtraction of two numbers. | 5.3
Divide | Perform division of two numbers. | 5.3
Rank | Determine the rank of a set of numbers. | 5.3
Different / Same | Determine if two numbers are different or the same. | 5.3
Add | Calculate the sum of two numbers. | 4.0
Max / Min | Retrieve the maximum or minimum number from a set of numbers. | 3.1
Col / Rowname | Retrieve the column or row name from the table. | 3.1
Trend same/different | Determine the trend for two columns or rows, whether they are the same or different. | 2.9
Set check | Verify if a value belongs to a set of numbers. | 2.9

Table 2: The function names, descriptions, and their proportions in our SciTab
dataset.

#### Reasoning Depth.

We measure the reasoning depth as the number of required operations in each
claim. Figure 3 shows the reasoning depth distribution with an average of 4.76
and a maximum depth of 11. The claims are classified into shallow (1–2
reasoning steps) and deep (3 or more steps) categories. Most of the claims
(86%) fall into the deep claims category, as shown in Figure 3. This showcases
the reasoning difficulty in SciTab.

$1$$2$$3$$4$$5$$6$$7$$8$$9$$10$$11$$5$$10$$15$$20$$25$$6$$8$$15$$18$$20$$15$$7$$5$$3$$2$$1$Reasoning
StepsPercentages (%) Figure 3: The distribution histogram of reasoning steps
in our SciTab dataset. The x-axis is the reasoning steps in each claim, and
the y-axis is the frequency for each reasoning step. The shallow claims (with
1–2 reasoning steps) are highlighted in red, while the deep claims (with 3+
reasoning steps) are highlighted in blue.

#### Reasoning Graph.

In order to highlight the diverse and complex nature of the reasoning involved
in the claims of SciTab, we constructed reasoning graphs, as depicted in the
right part of Figure 1. By analyzing these reasoning graphs, several
noteworthy findings emerge. For instance, in the example claim, “A’s
productivity of 57.5% expresses that it appears in 7.5% more often than
expected by random chance”, we observe that various types and levels of
reasoning paths are involved. Firstly, the utilization of Closed-domain
knowledge’ from the table caption enables the capture of productivity
corresponding to the ‘Prod.’ column in the table. This initial reasoning step
provides essential context for further analysis. Subsequently, a ‘simple
lookup’ reasoning approach is applied to ascertain that A’s productivity is
indeed 57.5%. Furthermore, the reasoning process involves ‘commonsense
knowledge’ reasoning, which aids in the interpretation of random chance as
50%. This demonstrates the incorporation of external knowledge beyond the
immediate context provided by the table. Notably, the integration of multiple
reasoning paths is facilitated by a ‘subtraction’ reasoning type, which allows
for the calculation of the difference between 57.5% and 50%, yielding the
result of 7.5%. Consequently, this comprehensive reasoning framework provides
strong support for the claim under consideration. The analysis of reasoning
graphs provides valuable insights into the complexity and variety of reasoning
strategies employed in SciTab, underscoring the significance of incorporating
different forms of knowledge and reasoning paths to substantiate claims
effectively.

###  4.2 Refuted and NEI Claims Analysis

To ensure the quality of both the machine-generated and human-collaborated
refuted and NEI claims in our SciTab dataset, we conduct an in-depth analysis
by randomly selecting 60 claims from each category. Table 3 (top half)
presents the distribution of claims in each category based on the type of
knowledge required to refute them. Our analysis shows that the distribution of
refuted types in our SciTab dataset is consistent with other fact-checking
datasets, such as incorrect calculation results (41.7%), values that do not
match (8.3%), and wrong operation type (6.3%). More examples of refuted claims
are shown in Appendix C. However, two unique refuted types in our SciTab
dataset are incorrect approximation words (33.33%) and the claim is generally
correct with some exceptions (10.4%). These findings indicate that our dataset
is more representative of real-world scientific claims, where ambiguity and
half-truth cases are frequent in scientific fact-checking.

The NEI claims (bottom half; Table 3) exhibit diverse reasoning types. For
instance, except for not having enough matching evidence, open-domain
knowledge is necessary for claims that require familiarity with specific
terminologies such as BLEU. Lack of closed-domain knowledge is another reason
for NEI, where a clear understanding of the model architecture mentioned
elsewhere in the paper is necessary to verify the claim. Other reasons include
the use of vague pronouns like “it” or “this” leading to ambiguity in the
claim. These distinct refuted and NEI reasoning types underscore the unique
features of SciTab, making it a more comprehensive and realistic
representation of the challenges faced in real-world scientific fact-checking.

Refuted Reasons | Prop. (%)
---|---
The calculation result is wrong. | 41.7
The approximation word is wrong. | 33.3
The claim is partially right. | 10.0
The values in the claim do not match. | 8.3
The operation type is wrong. | 6.7
NEI Reasons | Prop. (%)
The claim does not have enough matching evidence. | 33.3
The claim lacks open-domain knowledge. | 25.0
The claim lacks closed-domain knowledge. | 15.0
The claim refers to another table. | 11.7
The claim contains vague pronouns. | 8.3
The claim omits specific information. | 6.7

Table 3: Refuted and NEI reasons and their proportions in SciTab.

##  5 Experiment

We then conduct a comprehensive evaluation of the state-of-the-art models on
SciTab.

Models | # of Para. | Zero-shot | In-Context
---|---|---|---
2-class | 3-class | 2-class | 3-class
|  TAPAS-large (Tabfact) Herzig et al. (2020) | 340M | 50.30 | — | — | —
I. Table-based |  TAPEX-large (Tabfact) Liu et al. (2022) | 400M | 56.06 | — | — | —
LLMs |  TAPEX-Zero-large Liu et al. (2023) | 780M | 48.28 | 29.72 | 42.44 | 23.47
|  TAPEX-Zero-XL Liu et al. (2023) | 3B | 49.77 | 34.30 | 42.12 | 25.62
|  Flan-T5-base Chung et al. (2022) | 250M | 47.38 | 26.56 | 44.82 | 24.09
II. Encoder–Decoder |  Flan-T5-large Chung et al. (2022) | 780M | 51.58 | 32.55 | 49.62 | 27.30
LLMs |  FLan-T5-XL Chung et al. (2022) | 3B | 52.41 | 38.05 | 48.05 | 29.21
|  Flan-T5-XXL Chung et al. (2022) | 11B | 59.60 | 34.91 | 60.48 | 34.04
|  Alpaca-7B Taori et al. (2023) | 7B | 37.22 | 27.59 | 40.46 | 28.95
III. Decoder-Only |  Vicuna-7B Chiang et al. (2023) | 7B | 63.62 | 32.47 | 50.35 | 34.26
LLMs |  Vicuna-13B Chiang et al. (2023) | 13B | 41.82 | 29.63 | 55.11 | 35.16
|  LLaMA-7B Touvron et al. (2023) | 7B | 49.05 | 32.26 | 45.24 | 27.17
|  LLaMA-13B Touvron et al. (2023) | 13B | 53.97 | 37.18 | 44.39 | 32.66
|  InstructGPT Ouyang et al. (2022) | 175B | 68.44 | 41.41 | 68.10 | 41.58
IV. API-based |  InstructGPT+CoT Ouyang et al. (2022) | 175B | — | — | 68.46 | 42.60
LLMs |  PoT Chen et al. (2022) | 175B | — | — | 63.79 | —
|  GPT-4 OpenAI (2023) | — | 78.22 | 64.80 | 77.98 | 63.21
|  GPT-4+CoT OpenAI (2023) | — | — | — | 76.85 | 62.77

Table 4: Macro-$F_{1}$ scores of baselines on SciTab for different settings.
The TAPAS-large and TAPEX-large models are fine-tuned on the TabFact dataset,
while others perform zero-shot learning. IV represents API-based LLMs, which
are black-box, while III corresponds to decoder-only LLMs, which are open-
sourced and replicable. The bold text indicates the best performance among I
to III, while the underlined text indicates the overall best performance among
all the models.

###  5.1 Task Formulation

We formally define the task of scientific table-based fact-checking as
follows. A scientific table $\mathcal{T}$ consists of a table caption $P$ and
the table content $(\\{T_{i,j}|i\leq R_{T},j\leq C_{T}\\}$ with $R_{T}$ rows
and $C_{T}$ columns, where $T_{i,j}$ is the content in the $(i,j)$th cell.
Given a claim $C$ describing a fact to be verified against the table
$\mathcal{T}$, a table fact-checking model $\mathcal{F}$ predicts a label
$\mathcal{Y}$ to verify whether $\mathcal{C}$ is supported, refuted, or can
not be verified by the information in $\mathcal{T}$.

Considering the real-world situation that large-scale training data is only
not available or expensive to collect, we focus on the zero-shot/in-context
evaluation where the model can only access zero/few in-domain data from
SciTab. To this end, we randomly hold out 5 tables with 25 claims as model-
accessible data and use the rest of the data as the unseen test set. This also
prevents the model from learning spurious features that lead to over-estimated
performance Schuster et al. (2019).

###  5.2 Models

We conduct a comprehensive evaluation of SciTab for various models, including
table-based pretraining models, encoder–decoder models, decoder-only LLMs, and
API-based LLMs. Notably, API-based LLMs are black-box models, making them non-
replicable, whereas decoder-only LLMs are open-sourced and offer replicability
for further research.

#### Table-based LLMs.

These are pretrained transformer models further fine-tuned on tabular data. We
choose three different models: 1) TAPAS Herzig et al. (2020), a BERT-based
model fine-tuned on millions of tables from English Wikipedia and
corresponding texts, 2) TAPEX Liu et al. (2022), a model that fine-tunes BART
Lewis et al. (2019) on a large-scale synthetic dataset generated by
automatically synthesizing executable SQL queries and their execution outputs,
and 3) TAPEX-Zero Liu et al. (2023), an enlarged version of TAPEX. For TAPAS
and TAPEX, we use their fine-tuned version on the TabFact Chen et al. (2020)
dataset for table fact-checking.

#### Encoder–Decoder LLMs.

We also use encoder–decoder models where both the input and output are
sequences of tokens. To adapt the model to take the table as input, we flatten
the table as a sequence following Chen et al. (2020). The input is then
formulated as $[\tilde{T};P;C;Q]$, where $\tilde{T}$ is the linearized table,
and $Q$ is a question template “Based on the information in the table, is the
above claim true? A) True B) False C) Unknown?”. We choose FLAN-T5 Chung et
al. (2022), an improved T5 model Raffel et al. (2020) pretrained on more than
1.8K tasks with instruction tuning, which has achieved strong zero-shot/in-
context performance on other fact-checking benchmarks.

#### Decoder-Only LLMs.

We also evaluate the performance of state-of-the-art decoder-only LLMs,
including: 1) LLaMA Touvron et al. (2023), the first open-source model by Meta
AI; 2) Alpaca Taori et al. (2023), an instruction-following language model
fine-tuned on LLaMA; and 3) Vicuna Chiang et al. (2023), the arguably best-
performed open-source LLMs that claimed to achieve 90% quality compared to
OpenAI ChatGPT and Google Bard. We use the same input format as in the
encoder-decoder model.

#### API-based LLMs.

Finally, we evaluate closed-source language models that require API calls for
inference, including InstructGPT (text-davinci-003) Ouyang et al. (2022) and
GPT-4 OpenAI (2023). We evaluate the setting that directly predicts the label
and the Chain-of-Thought (CoT) Wei et al. (2022) setting, which generates
explanations before predicting the final label. We also include the Program-
of-Thoughts (PoT) Chen et al. (2022) model that has shown strong ability in
solving complex numerical reasoning tasks. It first parses the reasoning steps
as Python programs and then executes them on a Python interpreter to derive
accurate answers. Since most claims in SciTab also require numerical
reasoning, we want to test whether program-guided reasoning can be extended to
table-based fact-checking.

###  5.3 Main Results

We evaluate all models under both zero-shot and in-context settings. In the
zero-shot setting, the model does not have access to any in-domain data. In
the in-context setting, we provide three hold-out examples as demonstrations
for in-context learning. We report two sets of results: the 2-class case,
where examples labeled as ‘Not Enough Information’ (NEI) are excluded (since
some models cannot process NEI claims), and the 3-class case including all
three labels. The results are shown in Table 4. We have five major
observations.

1\. In general, all decoder-only LLMs do not achieve promising results on
SCITAB. The best result is 63.62 for the 2-class setting (Vicuna-7B and 38.05
for the 3-class setting (FLAN-T5-XL). Both results are only moderately better
(+13.62 and +4.72) than the random guess. This reveals the challenging nature
of SciTab and its potential to be the future benchmark for scientific fact-
checking.

2\. Counter-intuitively, table-based LLMs do not outperform models pretrained
on pure texts, for example, FLAN-T5. We speculate that this discrepancy may be
attributed to the dissimilarity between the distribution of tables in
scientific literature and publicly available table corpus. For example,
scientific tables commonly include both row and column headers, whereas most
tables in Wikipedia lack row headers. Meanwhile, the claims in our dataset are
usually much longer than those in previous works, raising challenges to table-
based LLMs.

3\. The results in the 3-class setting are notably poorer than those in the
2-class setting. This discrepancy underscores the challenges that most models
face when confronted with the NEI class. One plausible explanation could be
the inherent difficulty in distinguishing between ‘refuted’ and ‘NEI’ claims —
a task that even trained human annotators to struggle with, as noted by Jiang
et al. (2020). Our forthcoming error analysis will further demonstrate that
the inclusion of the NEI class tends to diminish the models’ confidence,
causing a shift in their predictions from ‘supported/refuted’ to ‘NEI’.

4\. Interestingly, the provision of in-context examples does not result in
improved performance for the majority of models. This observation is somewhat
expected for decoder-only LLMs as they have not been reported to possess in-
context learning capabilities. Nonetheless, it is surprising to find that even
with chain-of-thought prompting, in-context demonstrations do not yield
positive effects for InstructGPT and GPT-4. Our error analysis on the PoT
offers some insight into this phenomenon and will be discussed in the next
section.

5\. API-based LLMs perform better than decoder-only LLMs, with GPT-4 achieving
78.22 macro-$F_{1}$ for the 2-class setting and 64.80 for the 3-class setting.
This observation aligns with the assertion in GPT-4 technical report OpenAI
(2023) that GPT-4 has a strong ability to perform complex reasoning. And we
show that this ability can generalize to reason over tabular data as well.
However, the black-box nature of OpenAI models restricts our further analysis
of its behavior.

###  5.4 Error Analysis

To better understand the challenges of our SciTab dataset, we conduct an error
analysis for InstructGPT, GPT-4, and PoT.

#### InstructGPT and GPT-4.

The confusion matrices for InstructGPT and GPT-4 under the zero-shot 3-class
setting are shown in Figure 4. We find that both models displayed difficulty
in accurately predicting the NEI class. InstructGPT displays a pattern of
“less confident”, frequently classifying supported and refuted claims as
‘NEI’. In contrast, GPT-4 exhibits a pattern of “overconfidence”, incorrectly
categorizing NEI claims as either supported or refuted. This corroborates our
earlier observation that distinguishing whether a claim is verifiable is one
of the key challenges for SciTab. Further, we also examine individual error
instances, with typical examples provided in Figure 13 and Figure 14 within
Appendix E. The majority of ‘supported’ claims that were incorrectly
classified as ‘refuted’ (Case 6) typically involve numerical reasoning or
comparisons. Conversely, when ‘refuted’ claims are inaccurately predicted as
‘supported’ (Case 3), we find that LLMs often overlook claims containing
negation, indicating a lack of deep comprehension. For cases where ‘supported’
or ‘refuted’ claims are erroneously predicted as ‘NEI’ (Case 1 and Case 2),
such claims typically demand extensive reasoning and a nuanced understanding
of the research findings. Interestingly, when faced with these complex cases,
the model tends to default to the safer choice of ‘uncertain’ (NEI). This
underscores the importance of NEI claims and highlights the more balanced
distribution of NEI claims in our SciTab dataset.

#### PoT.

Unexpectedly, incorporating a Python interpreter does not confer any advantage
on our dataset (as shown in Table 4), despite its positive impacts on other
numerical reasoning tasks. In order to understand this, we randomly selected
50 claims wherein the PoT incorrectly predicted the final veracity labels and
evaluated the quality of the generated Python programs. The errors, as
assessed by human annotators, can be divided into four categories: (_i_)
Grounding errors, where the program incorrectly associates data with the
respective cells in the table; (_ii_) Program errors, which encompass mistakes
such as incorrect or missing arguments/variables, and erroneous operations;
(_iii_) Ambiguity errors, where the claim contains ambiguous expressions that
the program fails to represent; and (_iv_) Calculation errors, where incorrect
float digits in the Python code lead to inaccurate calculation results. We
present the error analysis in Table 5, and examples of program errors can be
found in Figure 11 and Figure 12 in Appendix D. Compared to other datasets,
categories (_i_) and (_iii_) present unique challenges in our dataset.
Category (_i_) underlines the difficulty in accurately referencing the
specific cells to which a claim refers. Category (_iii_), on the other hand,
emphasizes the difficulties posed by the ambiguous nature of scientific
claims, such as “A is significantly better than B”, to program-based methods.
This connection further emphasizes the contribution of our work in addressing
the mismatches between reasoning types and the occurrence of grounding errors.

Figure 4: Confusion matrix for InstructGPT (left) and GPT-4 (right) on zero-
shot 3-class classification.

Error Type | Prop. (%)
---|---
I. Grounding errors | 50
II. Program errors | 8
III. Ambiguity errors | 22
IV. Calculation errors | 20

Table 5: Error types and their proportions for incorrectly-predicted samples
in PoT.

##  6 Conclusion and Future Work

In this paper, we have presented SciTab, a novel dataset for scientific fact-
checking that addresses the limitations of existing benchmarks. By
incorporating real-world scientific claims and their corresponding evidence in
the form of tables, SciTab offers a more comprehensive and nuanced
representation of scientific reasoning. The challenging nature of SciTab is
evident from the performance of state-of-the-art models, highlighting the need
for further research in this field. First, developing more sophisticated
models specifically tailored for SciTab could improve the accuracy of
scientific claim verification. Besides, addressing the challenges posed by
ambiguous claims represents a crucial direction for future research in
scientific fact-checking. One potential approach is to explore methods that
enhance the understanding and disambiguation of ambiguous claims. This could
involve leveraging contextual information from the surrounding text, utilizing
external knowledge sources, or incorporating domain-specific ontologies to
provide additional context and disambiguate the claim.

## Ethics Statement

#### IRB (Institutional Review Board) Application

We have received approval from the Institutional Review Board
(IRB)444https://www.nus.edu.sg/research/irb. The NUS-IRB Reference Code is
NUS-IRB-2022-599 for our data collection. The IRB reviewed our experimental
design and research procedures to ensure that they do not pose more than
minimal risks to research participants. We take steps to protect research
participants’ privacy and the confidentiality of their data. The review
process took two months to complete.

## References

  * Aly et al. (2021) Rami Aly, Zhijiang Guo, Michael Sejr Schlichtkrull, James Thorne, Andreas Vlachos, Christos Christodoulopoulos, Oana Cocarascu, and Arpit Mittal. 2021.  FEVEROUS: fact extraction and verification over unstructured and structured information.  In _Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual_.
  * Beltagy et al. (2021) Iz Beltagy, Arman Cohan, Guy Feigenblat, Dayne Freitag, Tirthankar Ghosal, Keith Hall, Drahomira Herrmannova, Petr Knoth, Kyle Lo, Philipp Mayr, Robert Patton, Michal Shmueli-Scheuer, Anita de Waard, Kuansan Wang, and Lucy Lu Wang. 2021.  Overview of the second workshop on scholarly document processing.  In _Proceedings of the Second Workshop on Scholarly Document Processing_ , pages 159–165, Online. Association for Computational Linguistics.
  * Bender and Friedman (2018) Emily M. Bender and Batya Friedman. 2018.  Data statements for natural language processing: Toward mitigating system bias and enabling better science.  _Trans. Assoc. Comput. Linguistics_ , 6:587–604.
  * Cachola et al. (2020) Isabel Cachola, Kyle Lo, Arman Cohan, and Daniel S. Weld. 2020.  TLDR: extreme summarization of scientific documents.  _CoRR_ , abs/2004.15011.
  * Chen et al. (2022) Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. 2022.  Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks.
  * Chen et al. (2020) Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. 2020.  Tabfact: A large-scale dataset for table-based fact verification.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
  * Chiang et al. (2023) Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.
  * Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Y. Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022.  Scaling instruction-finetuned language models.  _CoRR_ , abs/2210.11416.
  * Cohan et al. (2022) Arman Cohan, Guy Feigenblat, Tirthankar Ghosal, and Michal Shmueli-Scheuer. 2022\.  Overview of the first shared task on multi perspective scientific document summarization (MuP).  In _Proceedings of the Third Workshop on Scholarly Document Processing_ , pages 263–267, Gyeongju, Republic of Korea. Association for Computational Linguistics.
  * Cohan et al. (2015) Arman Cohan, Luca Soldaini, and Nazli Goharian. 2015.  Matching citation text and cited spans in biomedical literature: a search-oriented approach.  In _Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1042–1048, Denver, Colorado. Association for Computational Linguistics.
  * Cohen (1960) Jacob Cohen. 1960.  A coefficient of agreement for nominal scales.  _Educational and Psychological Measurement_ , 20:37 – 46.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_ , pages 4171–4186. Association for Computational Linguistics.
  * Gu et al. (2020) Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2020.  Domain-specific language model pretraining for biomedical natural language processing.  _CoRR_ , abs/2007.15779.
  * Guo et al. (2022) Zhijiang Guo, Michael Sejr Schlichtkrull, and Andreas Vlachos. 2022.  A survey on automated fact-checking.  _Trans. Assoc. Comput. Linguistics_ , 10:178–206.
  * Gupta and Srikumar (2021) Ashim Gupta and Vivek Srikumar. 2021.  X-fact: A new benchmark dataset for multilingual fact checking.  In _Annual Meeting of the Association for Computational Linguistics (ACL)_ , pages 675–682.
  * Gupta et al. (2020) Vivek Gupta, Maitrey Mehta, Pegah Nokhiz, and Vivek Srikumar. 2020.  INFOTABS: inference on tables as semi-structured data.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_ , pages 2309–2324. Association for Computational Linguistics.
  * Herzig et al. (2020) Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Martin Eisenschlos. 2020.  Tapas: Weakly supervised table parsing via pre-training.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_ , pages 4320–4333. Association for Computational Linguistics.
  * Jiang et al. (2020) Yichen Jiang, Shikha Bordia, Zheng Zhong, Charles Dognin, Maneesh Kumar Singh, and Mohit Bansal. 2020.  Hover: A dataset for many-hop fact extraction and claim verification.  In _Findings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 3441–3460.
  * Jin et al. (2019) Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W. Cohen, and Xinghua Lu. 2019\.  Pubmedqa: A dataset for biomedical research question answering.  _CoRR_ , abs/1909.06146.
  * Kashyap et al. (2023) Abhinav Ramesh Kashyap, Yajing Yang, and Min-Yen Kan. 2023.  Scientific document processing: challenges for modern learning methods.  _International Journal on Digital Libraries_ , pages 1432–1300.
  * Lee et al. (2009) W.Y. Lee, L. Ho, and M.E.T. Ng. 2009.  _Research Writing: A Workbook for Graduate Students_.  Prentice Hall.
  * Lewis et al. (2019) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019.  Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  _arXiv preprint arXiv:1910.13461_.
  * Liu et al. (2022) Qian Liu, Bei Chen, Jiaqi Guo, Morteza Ziyadi, Zeqi Lin, Weizhu Chen, and Jian-Guang Lou. 2022.  TAPEX: table pre-training via learning a neural SQL executor.  In _The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022_. OpenReview.net.
  * Liu et al. (2023) Qian Liu, Fan Zhou, Zhengbao Jiang, Longxu Dou, and Min Lin. 2023.  From zero to hero: Examining the power of symbolic tasks in instruction tuning.
  * Moosavi et al. (2021) Nafise Sadat Moosavi, Andreas Rücklé, Dan Roth, and Iryna Gurevych. 2021\.  Scigen: a dataset for reasoning-aware text generation from scientific tables.  In _Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual_.
  * Nori et al. (2023) Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz. 2023.  Capabilities of gpt-4 on medical challenge problems.  _arXiv preprint arXiv:2303.13375_.
  * OpenAI (2023) OpenAI. 2023.  GPT-4 technical report.  _CoRR_ , abs/2303.08774.
  * Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022.  Training language models to follow instructions with human feedback.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research (JMLR)_ , 21:140:1–140:67.
  * Ramesh Kashyap and Kan (2020) Abhinav Ramesh Kashyap and Min-Yen Kan. 2020.  SciWING– a software toolkit for scientific document processing.  In _Proceedings of the First Workshop on Scholarly Document Processing_ , pages 113–120, Online. Association for Computational Linguistics.
  * Saakyan et al. (2021) Arkadiy Saakyan, Tuhin Chakrabarty, and Smaranda Muresan. 2021.  Covid-fact: Fact extraction and verification of real-world claims on COVID-19 pandemic.  In _Annual Meeting of the Association for Computational Linguistics (ACL)_ , pages 2116–2129.
  * Sathe et al. (2020) Aalok Sathe, Salar Ather, Tuan Manh Le, Nathan Perry, and Joonsuk Park. 2020.  Automated fact-checking of claims from wikipedia.  In _Proceedings of The 12th Language Resources and Evaluation Conference (LREC)_ , pages 6874–6882.
  * Schuster et al. (2021) Tal Schuster, Adam Fisch, and Regina Barzilay. 2021.  Get your vitamin c! robust fact verification with contrastive evidence.  In _Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT)_ , pages 624–643.
  * Schuster et al. (2019) Tal Schuster, Darsh J. Shah, Yun Jie Serene Yeo, Daniel Filizzola, Enrico Santus, and Regina Barzilay. 2019.  Towards debiasing fact verification models.  In _Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 3417–3423.
  * Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023.  Stanford alpaca: An instruction-following llama model.  https://github.com/tatsu-lab/stanford_alpaca.
  * Thorne et al. (2018) James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018\.  FEVER: a large-scale dataset for fact extraction and verification.  In _Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT)_ , pages 809–819.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023.  Llama: Open and efficient foundation language models.
  * Wadden et al. (2020) David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020.  Fact or fiction: Verifying scientific claims.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020_ , pages 7534–7550. Association for Computational Linguistics.
  * Wadden et al. (2022a) David Wadden, Kyle Lo, Bailey Kuehl, Arman Cohan, Iz Beltagy, Lucy Lu Wang, and Hannaneh Hajishirzi. 2022a.  Scifact-open: Towards open-domain scientific claim verification.  In _Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022_ , pages 4719–4734. Association for Computational Linguistics.
  * Wadden et al. (2022b) David Wadden, Kyle Lo, Lucy Lu Wang, Arman Cohan, Iz Beltagy, and Hannaneh Hajishirzi. 2022b.  MultiVerS: Improving scientific claim verification with weak supervision and full-document context.  In _Findings of the Association for Computational Linguistics: NAACL 2022_ , pages 61–76, Seattle, United States. Association for Computational Linguistics.
  * Wang et al. (2021) Nancy Xin Ru Wang, Diwakar Mahajan, Marina Danilevsky, and Sara Rosenthal. 2021\.  Semeval-2021 task 9: Fact verification and evidence finding for tabular data in scientific documents (SEM-TAB-FACTS).  In _Proceedings of the 15th International Workshop on Semantic Evaluation, SemEval@ACL/IJCNLP 2021, Virtual Event / Bangkok, Thailand, August 5-6, 2021_ , pages 317–326. Association for Computational Linguistics.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed H. Chi, Quoc Le, and Denny Zhou. 2022.  Chain of thought prompting elicits reasoning in large language models.  _CoRR_ , abs/2201.11903.
  * Wright et al. (2022) Dustin Wright, David Wadden, Kyle Lo, Bailey Kuehl, Arman Cohan, Isabelle Augenstein, and Lucy Lu Wang. 2022.  Generating scientific claims for zero-shot scientific fact checking.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2448–2460, Dublin, Ireland. Association for Computational Linguistics.

##  Appendix A Claim Extraction Procedure

###  A.1 Description, Background and Claim Definition

In academic writing Lee et al. (2009), when writing the accompanying text for
data (table, figure), the following three elements are typically included. Our
focus is the KD and COM that shows the highlight and commenting on key data
(or what we call claims).

#### Location of results (LOC)

Statements that locate where the figure/table is found, e.g., Figure 7.2
displays the mean percentile scores

#### Highlighting of key data (KD)

Statements that highlight the important data, e.g.,(1) Highest or lowest
values (2) Overall trend or pattern in the data (3) Points that do not seem to
fit the pattern or trend, etc. (4) Results which provide answers to your
research questions

#### Commenting on key data (COM)

Statements that interpret the data. There are three types of comments: (1)
Generalization (deductions and implications drawn from the results), e.g.,
This indicates that…(2) Comparison of results with those from prior studies,
e.g., Different from … (3) Explanation or speculation (possible reasons or
cause-effect relationships for the results), e.g., The possible reason is
that…

###  A.2 Claim Extraction Interface

Figure 5 shows the User Interface for the claim extraction task.

Figure 5: User interface for the claim extraction task.

###  A.3 Automatic Classification Results

By using BERT Devlin et al. (2019) classifier, the 2-class classification and
3-class classification results are shown in Table 6.

Tasks | Predict Accuracy | Macro $F_{1}$
---|---|---
2-class | 0.8814 | 0.8535
3-class | 0.8376 | 0.7315

Table 6: BERT in claim classification tasks. 2-class task contains the labels
of {Claim, Non-claim}, while the 3-class tasks contain the labels of
{Descriptions, Background, Claims}

##  Appendix B Human Verification Procedure

The human verifies the generated NEI claims and the generated Refuted claims.

###  B.1 Annotation Interface for NEI Claims Verification

Figure 6 shows the User Interface for the NEI claim verification task.

Figure 6: User interface for the NEI claim verification task.

###  B.2 Annotation Interface for Refuted Claims Verification

Figure 6 shows the User Interface for the refuted claim verification task.

Figure 7: User interface for the refuted claim verification task

###  B.3 Annotation Post-Survey

Figure 8 shows the examples of post-annotation survey questions and the
annotator’s answers.

Figure 8: The examples of annotators’ post-survey questions and their answers.

##  Appendix C Case Study for Refuted Claims

Figure 9 and Figure 10 show five examples of refuted cases. We provide
explanations for each of the error cases below:

#### 1\. The calculation result is wrong.

It generates wrong calculation results. The correct result should be 27.9-21.7
= 6.2.

#### 2\. The approximation word is wrong.

It generates wrong approximation words since compared to 23.3, 19.4 is not
“significantly” lower.

#### 3\. The claim is partially right.

The claim is right with the exceptions. For example, BShift column doesn’t
satisfy the claim.

#### 4\. The values in the claim do not match.

The value in the claim does not match the value in the table, i.e., the
correct value should be 27.9.

#### 5\. The operation type is wrong.

It generates the wrong operation type. For example, GCN+RC+LA (9) is not
better than DCGCN1 since 22.9 > 22.0 and 53.0 > 52.6.

Figure 9: The refuted claims cases 1-3, including 1) the calculation result is
wrong. 2) The approximation word is wrong. 3) The claim is partially right.
Figure 10: The refuted claims cases 4 and 5, including 4) The values in the
claim do not match. and 5) The operation type is wrong.

##  Appendix D Error Cases for Program-of-Thoughts

Figure 11 and Figure 12 shows five error examples of Program-of-Thoughts when
tested on our SciTab dataset. We provide explanations for each of the error
cases below:

#### Error Case 1

It has the wrong entity linking (Grounding error) and the wrong operation
(Program error). The “winograd_baseline = 73.06” and “winocoref_baseline =
88.48” should be “IlliCons_winograd = 53.26” and “IlliCons_winocoref = 74.32’.
The “>” operation should be “>=”.

#### Error Case 2

It parses incomplete entity linking (Grounding error). The program should
parse the other baseline results, such as “SFEGAN_WER = 14.9”.

#### Error Case 3

It fails to generate a correct program (Program error). The variables and the
logical functions in the programs are wrong. For example, the
“G2S_GAT_BLEU_LDC2015E86” should be “G2S_GIN_BLEU_LDC2015E86”. The “and”
function should be “or”.

#### Error Case 4

It fails to generate a fine-grained program for the approximation word
“comparable” (Ambiguity error). Currently, the program defined “comparable” as
“larger than”, which is not precise.

#### Error Case 5

It generates the right program, but the calculation result is wrong due to the
incorrect float digits in the Python code (Calculation error). For example,
Python will output ‘1.9499999’ which is not equal to ‘1.95’.

Figure 11: Error Cases 1-3 for Program-of-Thought. Figure 12: Error Cases 4
and 5 for Program-of-Thought.

##  Appendix E Error Cases for InstructGPT

Figure 13 and Figure 14 show six error examples of InstructGPT on zero-shot
settings when tested on our SciTab dataset.

Figure 13: Error Cases 1-3 for InstructGPT on 3-class classification in the
zero-shot setting. Figure 14: Error Cases 4-6 for InstructGPT on 3-class
classification in the zero-shot setting.


# Fast rare events in exit times distributions of jump processes

Alessandro Vezzani  Istituto dei Materiali per l’Elettronica ed il Magnetismo
(IMEM-CNR), Parco Area delle Scienze, 37/A-43124 Parma, Italy  Dipartimento di
Scienze Matematiche, Fisiche e Informatiche, Università degli Studi di Parma,
Parco Area delle Scienze, 7/A 43124 Parma, Italy  INFN, Gruppo Collegato di
Parma, Parco Area delle Scienze 7/A, 43124 Parma, Italy  Raffaella Burioni
Dipartimento di Scienze Matematiche, Fisiche e Informatiche, Università degli
Studi di Parma, Parco Area delle Scienze, 7/A 43124 Parma, Italy  INFN, Gruppo
Collegato di Parma, Parco Area delle Scienze 7/A, 43124 Parma, Italy

###### Abstract

Rare events in the first-passage distributions of jump processes are capable
of triggering anomalous reactions or series of events. Estimating their
probability is particularly important when the jump probabilities have broad-
tailed distributions, and rare events are therefore not so rare. We study
three jump processes that are used to model a wide class of stochastic
processes ranging from biology to transport in disordered systems, ecology and
finance. We consider discrete time random-walks, continuous time random-walks
and the Lévy-Lorentz gas and determine the exact form of the scaling function
for the probability distribution of fast rare events, in which the jump
process exits in a very short time at a large distance opposite to the
starting point. For this estimation we use the so called big jump principle,
and we show that in the regime of fast rare events the exit time distributions
are not exponentially suppressed, even in the case of normal diffusion. This
implies that fast rare events are actually much less rare than predicted by
the usual estimates of large deviations and can occur on timescales orders of
magnitude shorter than expected. Our results are confirmed by extensive
numerical simulations.

A stochastic process that reaches a certain threshold value for the first time
can trigger many events: a chemical reaction occurs [1], a target is reached
[2, 3, 4, 5, 6, 7], the dynamics of a financial process starts [8], biological
and ecological processes take place [9, 10]. The study of these triggering
events and the probability of their occurrence is based on the knowledge of
first passage probabilities [11]. As history-dependent quantities, first-
passage probabilities are difficult to determine: general results are
typically available for the average first-passage time and its lower-order
moments [11, 12], but knowledge of complete first-passage distributions is
generally limited. Given the ’trigger’ property of exit times, the tails of
these distributions are particularly important as they allow the probability
of rare anomalous events to be estimated, such as, for example, an exit in a
very short time [13]. These estimates are particularly relevant in the case of
broad-tailed distributions, where rare events are not so rare.

In particular, a few results are available for first-passage probabilities of
jump processes, which are stochastic processes that involve random jumps
between different states or positions, occurring at random times and with
random magnitudes [14]. Jump processes are so important in modelling the
dynamics of stochastic processes in many fields that even their one-
dimensional formulation is of great relevance [15, 16, 17]. In the one-
dimensional case, exit time probabilities refer to the process leaving a
particular state or interval within a specified time. In stochastic processes
with two alternative outcomes, the exit side can also be of importance [10],
for example to quantify observables such as transmission or backscattering
probabilities [18].

A well know examples of jump processes is that of random walks (RW)s, in
discrete and continuous time [19, 20], for which recent results have been
obtained [21, 22, 23]. An interesting question is to determine the exit
probabilities from the side of a domain opposite to the starting point. A rare
events in this case corresponds to a fast walker exiting in a very short time,
that is in a time for which the dynamical scaling length of the process is
still much shorter than the size of the domain. To exit in a short time, the
trajectory of the walker should correspond to a long jump, allowing it to
travel far from the starting point. When fat-tailed distributions for jumps
are present in the stochastic process, these very fast events may not be
exponentially suppressed and may occur in timescales orders of magnitude
shorter than expected, proving crucial in predicting anomalous events.

Recently, we have investigated the role played in rare events by the so called
big jump principle [24]. The principle explains extreme events in a wide class
of systems with heavy tailed distributions not in terms of an accumulation of
many small subevents but as an effect of only the biggest event, the big jump
[25]. The principle has been successfully applied to characterize the tail of
the probability distribution, at distances much larger than the scaling
length, for a wide class of jump processes for RWs involving Lévy and sub-
exponential statistics for space and time [26], even in disordered settings
such as the Lévy-Lorentz gas [27, 28]. This is a system of scatterers placed
randomly in one dimension, with a broad distributed spacing. A walker moves
ballistically and is randomly transmitted or reflected when it reaches a
scatterer. The Lévy-Lorentz gas is a relevant model in the study of transport
in disordered media [29, 30], and there is currently no known first-passage
probability estimate for this system.

In this letter, we consider three well known models of one-dimensional jump
processes: the discrete time RW, the continuous time RW, and the Lévy-Lorentz
gas. Taking advantage of the big jump approach, we estimate the exit
probability from one side of a domain. In particular, for a walker starting at
the origin, we analytically determine the form of the probability distribution
to reach for the first time a distance $X$ without being absorbed at the
origin, in the asymptotic limit where $X$ is much larger than the scaling
length of the process (i.e. in the limit of very fast atypical events). We
show that in this regime the three exit time distributions are not
exponentially suppressed, even when the scaling length of the process is
diffusive. This implies that, in these processes where jumps follows a
subexponential statistics, fast rare events are actually much less rare than
predicted by the usual estimates of large deviations. We test our results by
comparing them with extensive numerical simulations, with very good agreement.

The discrete time random walk. In discrete time RWs [20], at each discrete
time step the walker moves with probability $1/2$ to the right or to the left,
with the step length $r$ drawn from the probability density function (PDF)
$p(r)$. A finite second moment of $p(r)$ implies standard diffusion while, if
the second moment diverges, the walker performs a symmetric Lévy flight. We
consider a PDF $p(r)$ with a power law decay at large $r$ and a cut off at
short $r$:

$p(r)=\begin{cases}\frac{\alpha r_{0}^{\alpha}}{\alpha r^{1+\alpha}}&\mbox{if
}r>r_{0}\\\ 0&\mbox{if }r<r_{0}\\\ \end{cases}$ (1)

In this case, standard diffusion or a Lévy process are observed for $\alpha>2$
and $0<\alpha<2$, respectively. Our results apply to any distribution that
features the same behaviour at large $r$.

We denote with $x(n)$ the position of the walker after $n$ steps ($x(0)=0$)
and we consider the exit time probability $P_{X}(n)$, that is the probability
that at step $n$ the walker reaches for the first time a distance larger that
$X$ without being absorbed at the origin ($x(n)>X$ and $0<x(k)<X$ for
$0<k<n$).

Numerical results in Figure 1 show $P_{X}(n)$ as a function of the rescaled
exit time $n/X^{2}$. For at $\alpha=2.5$ (i.e. RWs with finite variance of the
step length) the typical exit time is driven by the growth of the scaling
length $\ell(n)$ of standard diffusion: $X\sim\ell(n)\sim n^{1/2}$. The
scaling of $P_{X}(n)$ and the analytic results displayed by the dashed line
have been obtained in [23] using a continuous limit. However, very fast
events, which occur in much shorter timescales (i.e. for $X\gg\ell(n)\sim
n^{1/2}$) are also observed in the figure with non-negligible probability.
These are the rare events we wish to describe.

Figure 1: Exit probability $P_{X}(n)$ for a Lévy flight. $P_{X}(n)$ is
multiplied by $X^{3}$ and plotted as a function of the rescaled exit time
$n/X^{2}$. For $\alpha>2$, the typical dynamics is diffusive and, at large
number of steps $n$, converges slowly to the theoretical prediction in [23],
with no free parameters. Very fast events at short times are however
significantly present.

If $X$ is much larger than the typical length scale $\ell(n)$, we expect that
$P_{X}(n)$ can be evaluated using the single big jump principle [25, 24, 26,
31, 32]. The principle states that for sub-exponential $p(r)$, a distance
$X\gg\ell(n)$ in a time $n$ is reached by a single event involving a large
jump of order $X$, while smaller distances of order $\ell(n)$ are reached in
multiple steps. For the process we are considering here, the scaling lengths
are known, and read $\ell(n)\sim n^{1/2}$ for diffusive processes and
$\ell(n)\sim n^{1/\alpha}$ for the Lévy flight. However, we stress again that
the big jump principle is expected to be valid for any sub-exponential step
length PDF $p(r)$ [26].

In this case, the RW exits at time $n$ at a distance $X\gg\ell(n)$ if it is
not absorbed at the origin in the first $n-1$ steps, and if exactly at step
$n$ the walker performs a jump in the positive direction, of length larger
than $X$. In the spirit of the big jump approach, the distance covered in the
previous $n-1$ steps can be neglected, as it is of the order of $\ell(n)$. The
asymptotic behavior of the survival probability $\pi(n)$ after $n-1$ steps in
a semi infinite-line is analytically known by the Sparre Andersen theorem [33,
34, 35] i.e. $\pi(n-1)\sim(\pi n)^{-1/2}$, while the probability to perform a
jump larger than $X$ is directly obtained from the single step length
distribution. Therefore the asymptotic behavior of the exit time probability
for $n\gg 1$ and $X\gg\ell(n)$ is given by:

$P_{X}(n)\sim\frac{1}{2}\pi(n-1)\int_{X}^{\infty}drp(r)\sim\frac{1}{2(\pi
n)^{1/2}}\cdot\frac{r_{0}^{\alpha}}{X^{\alpha}}$ (2)

where the factor $1/2$ takes into account the fact that the last long jump is
covered in the positive direction with probability $1/2$.

The results in Eq. (2) are compared with numerical simulations in Figure 2
showing an excellent agreement in the asymptotic regime at large $X$, both for
the Lévy flights $\alpha<2$ and for normal diffusion $\alpha>2$. On the other
hand, the peaks of the distributions at shorter $X$ are expected to occur when
the system size is comparable to the scaling length of the system $\ell(n)$,
according to the analytical predictions in [23], as shown in Figure 1 for
$\alpha>2$.

Figure 2: Exit probability $P_{X}(n)$ for a Lévy flight. $P_{X}(n)$ is
multiplied by $n^{1/2}$ and plotted as a function of the distance $X$. Upper
panel: $\alpha=1.2$ lower panel $\alpha=2.5$. The dashed line shows the
asymptotic analytical prediction in Eq. (2).

The continuous time random-walk. The continuous time RW [20, 36] is a process
in which a walker performs random steps of duration $t$ at constant velocity
$v$. In one dimension, each step is covered with equal probability in the
positive and the negative direction, and the duration of the step follows the
PDF $p(t)$, with

$p(t)=\begin{cases}\frac{\alpha}{t^{1+\alpha}}&\mbox{if }t>1\\\ 0&\mbox{if
}t<1\\\ \end{cases}$ (3)

The scaling length of the stochastic process is $\ell(t)\sim t^{\gamma}$, with
$\gamma=1$ for $\alpha<1$, $\gamma=1/\alpha$ for $1<\alpha<2$ and $\gamma=1/2$
for $\alpha>2$ [36]. Moreover, the single step contains another scaling length
that grows linearly with time, due to the constant velocity motion. Therefore,
for $\alpha<1$ in a single step the walker cannot reach a distance much larger
than $\ell(t)$ and the single big jump approach cannot be applied. On the
other hand, for $\alpha>1$ as well as for other sub-exponential PDF (e.g.
Weibull), the principle is valid and can be used efficiently [24, 26].

Analogously to the discrete case, $P_{X}(T)dT$ represents the probability to
reach for the first time a distance $X$ from the origin in a time between $T$
and $T+dT$, without being absorbed at the starting point $x=0$. For
$X>>\ell(T)$ according to the big jump, this process is realized by a single
stochastic event, a big jump carrying the walker at large distance. In
particular, the last step should be taken at time $T_{w}=T-X/v$ and the
duration of this last step should be larger than $X/v$, so that the walker
reaches the distance $X$ exactly at time $T$. Once again, the motion up to
time $T_{w}$ can be neglected. Moreover, the survival probability can again be
evaluated using the Sparre Andersen theorem. In particular the number of steps
performed by the walker in a time $T_{w}$ is $T_{w}/\langle t\rangle$, where
$\langle t\rangle=\int dt^{\prime}t^{\prime}p(t^{\prime})$ is the average
duration of the steps, so that the survival probability in a time $T_{w}$ is
$\pi(T_{w})\sim(T_{w}/\langle t\rangle)^{-1/2}$. Summing up we obtain:

$\displaystyle P_{X}(T)$ $\displaystyle\sim$ $\displaystyle\frac{1}{2\langle
t\rangle}\pi((T-X/v)/\langle t\rangle)\int_{X/v}^{\infty}dtp(t)$ (4)
$\displaystyle\sim$ $\displaystyle\frac{1}{(vT)^{\alpha+1/2}}\frac{\langle
t\rangle^{-1/2}t_{0}^{\alpha}}{2(\pi(1-y))^{1/2}y^{\alpha}}.$

In the first expression the prefactor $1/2$ takes into accounts the
probability that the big jump is covered in the positive direction, while
$1/\langle t\rangle$ is the jump rate of the process [24, 26] (in the discrete
time process, the rate is naturally fixed to one). Notice that in Eq. (4),
$y=X/(vT)$ represents the natural scaling of the distance, defined by the
ballistic motion performed in the big jump. Notice also that $vT$ is the
maximum distance that is covered in a time $T$. This means that $y<1$. In
Figure 3 we shown that $P_{X}(T)$ plotted against $X/(vT)$ indeed converges to
Eq. (4) at large times, so that the big jump estimate is correct.

Figure 3: Exit probability $P_{X}(T)$ for a continuos time RW. $P_{X}(T)$ is
multiplied by $(vT)^{1/2+\alpha}$ and plotted as a function of the rescaled
distance $X/(vT)$, $\alpha=1.5$. The continuous line shows the asymptotic
analytical prediction in Eq. (4).

The Lévy Lorentz gas. The Lévy Lorentz gas introduced in [27, 28] is a system
of scatterers randomly placed in one dimension. The distances between
scatterers are drawn again from a power law PDF $p(l)$:

$p(l)=\begin{cases}\frac{\alpha l_{0}^{\alpha}}{l^{1+\alpha}}&\mbox{if
}l>l_{0}\\\ 0&\mbox{if }l<l_{0}\\\ \end{cases}.$ (5)

A RW is naturally defined on the Lévy Lorentz gas: the walker moves at
constant speed $v$ and it is reflected with probability $\epsilon$
($0<\epsilon<1$) when it hits a scatterer [37]. We will focus on the case
where the walker starts at $t=0$ in a scattering site placed at $x=0$ [27,
28]. For this model, the PDF of the walker position $x$ as a function of time
has been recently studied [28, 37, 38, 39, 40], and its asymptotic behaviour
at large distances has been estimated using the big jump principle [28, 24,
41]. However, no results are yet known for the exit time probabilities.

Figure 4: Exit probability $P_{X}(n)$ for a Lévy Lorentz gas. $P_{X}(n)$ is
multiplied by $(vT)^{1+\alpha}$ and plotted a function of the distance
$X/(vT)$. Upper panel: $\alpha=0.6$ lower panel $\alpha=1.5$. The continuous
line shows the asymptotic analytical prediction in Eq. (6). In this case a
multiplicative constant has been optimized in order to reproduce numerical
data.

In the Lévy Lorentz gas, if a walker crosses a scatterer that has never been
crossed before, the choice of the distance of the subsequent scattering point
is a random process independent of the RW history. Hence in this event we can
successfully apply the big jump principle. The scaling length $\ell(T)$ is the
typical distance covered by the walker up to time $T$. We have $\ell(T)\sim
T^{\frac{1}{1+\tilde{\alpha}}}$ where $\tilde{\alpha}=\alpha$ if $\alpha<1$
and $\tilde{\alpha}=1$ if $\alpha>1$ [28]. According to the principle, the
walker exits from the boundary at a distance $X\gg\ell(T)$, only if at a time
$T_{w}=T-X/v$ the walker crosses a scattering point separated from the
subsequent scatterer by a distance larger than $X$. The definition of $T_{w}$
takes into account the finite velocity. In order to evaluate $P_{X}(T)$ we
need the probability of not being absorbed at the origin before $T_{w}$ and
the rate at which at time $T_{w}$ new scattering sites are crossed by the
walker. The rate has been calculated in [24] and it decays with $T_{w}$ as
$T_{w}^{-\frac{1}{1+\tilde{\alpha}}}$. The surviving probability can be
estimated using again the Sparre Andersen theorem, by evaluating the number of
scattering events $N(T_{w})$ occurring up to time $T_{w}$. According to [42],
the number of different scattering sites visited up to time $T_{w}$ is
$S(T_{w})\sim\ell(T_{w})^{\tilde{\alpha}}\sim
T_{w}^{\frac{\tilde{\alpha}}{1+\tilde{\alpha}}}$. Since the discrete process
occurring on the scatterers, without taking into account the distances among
the sites, is a simple diffusion on a lattice, we obtain $N(T_{w})\sim
S(T_{w})^{2}\sim T_{w}^{\frac{2\tilde{\alpha}}{1+\tilde{\alpha}}}$ (for
$\alpha>1$ the average distance between sites is finite and we recover the
expected result that the number of scattering events is proportional to the
elapsed time).

Now we can evaluate the exit time probability:

$\displaystyle P_{X}(T)$ $\displaystyle\sim$
$\displaystyle\frac{1}{N(T_{w})^{1/2}}\cdot\frac{1}{T_{w}^{\frac{1}{1+\tilde{\alpha}}}}\cdot\int_{X}^{\infty}dlp(l)$
(6) $\displaystyle\sim$
$\displaystyle\frac{1}{(T-X/v)^{\frac{\tilde{\alpha}}{1+\tilde{\alpha}}}}\cdot\frac{1}{(T-X/v)^{\frac{1}{1+\tilde{\alpha}}}}\cdot\int_{X}^{\infty}dlp(l)$
$\displaystyle\sim$
$\displaystyle\frac{1}{T^{\alpha+1}}\frac{1}{(1-y)y^{\alpha}}.$

In the first line the term $N(T_{w})^{-1/2}$ represents the decay of the
surviving probability at $T_{w}$ according to Sparre Andersen result; the term
$T_{w}^{-\frac{1}{1+\tilde{\alpha}}}$ is the jumping rate at time $T_{w}$ and
the integral is the probability to have a jump larger than $X$. We also
introduced the scaled distance $y=X/(vT)$ according to the ballistic motion
characterizing the single jump evolution. Eq (6) contains a multiplicative
constant that in this case cannot be simply evaluated since the rate and the
number of jumps have been calculated only by means of a scaling arguments. Eq.
(6) is compared with numerical simulation in Figure 4, showing an excellent
agreement.

Conclusions. A correct estimation of rare events in first-passage
distributions is crucial for predicting the occurrence of reactions, the
attainment of a target, the extinction of a population or a financial asset.
In this letter, using the big jump principle, we derived the analytical form
of the tail in the exit distributions on the opposite side of an interval from
the starting point for three fundamental jump processes: the discrete and the
continuous time RWs and the Lévy-Lorentz gas. In a nutshell, our result shows
that, in the presence of power law jump distributions, anomalous exit events
can occur on time scales that are orders of magnitude smaller than typical
exit times, even when the process is Gaussian. The result is heuristic and
based on the application of the big jump principle, yet comparison with
detailed numerical simulations suggests that the estimate is essentially
correct, thus opening the way to a rigorous derivation. We obtained our result
for a power law distribution but we expect such estimates to hold for a large
class of sub-exponential jump processes, providing a boost to the study of
rare events in first-passage probabilities.

###### Acknowledgements.

We warmly thank Olivier Bénichou and Jéremie Klinger for interesting
discussions.

## References

  * Gardiner [2004] C. W. Gardiner,  _Handbook of Stochastic Methods: for Physics, Chemistry and the Natural Sciences (3rd ed.)_ (Springer Berlin, 2004).
  * Bénichou  _et al._ [2005] O. Bénichou, M. Coppey, M. Moreau, P.-H. Suet, and R. Voituriez, Optimal search strategies for hidden targets, Phys. Rev. Lett. 94, 198101 (2005).
  * Agliari  _et al._ [2010] E. Agliari, R. Burioni, and A. Manzotti, Effective target arrangement in a deterministic scale-free graph, Phys. Rev. E 82, 011118 (2010).
  * Bénichou  _et al._ [2011] O. Bénichou, C. Loverdo, M. Moreau, and R. Voituriez, Intermittent search strategies, Rev. Mod. Phys. 83, 81 (2011).
  * Godec and Metzler [2016] A. Godec and R. Metzler, Universal proximity effect in target search kinetics in the few-encounter limit, Phys. Rev. X 6, 041037 (2016).
  * Thiel  _et al._ [2018] F. Thiel, E. Barkai, and D. A. Kessler, First detected arrival of a quantum walker on an infinite line, Phys. Rev. Lett. 120, 040502 (2018).
  * Meyer and Rieger [2021] H. Meyer and H. Rieger, Optimal non-markovian search strategies with $n$-step memory, Phys. Rev. Lett. 127, 070601 (2021).
  * Chicheportiche and Bouchaud [2014] R. Chicheportiche and J.-P. Bouchaud, Some applications of first-passage ideas to finance, in  _First-Passage Phenomena and Their Applications_, edited by R. Metzler, G. Oshanin, and S. Redner (World Scientific, 2014) Chap. 1, pp. 447–476.
  * Pulkkinen and Metzler [2013] O. Pulkkinen and R. Metzler, Distance matters: The impact of gene proximity in bacterial gene regulation, Phys. Rev. Lett. 110, 198101 (2013).
  * Arani  _et al._ [2021] B. M. S. Arani, S. R. Carpenter, L. Lahti, E. H. van Nes, and M. Scheffer, Exit time as a measure of ecological resilience, Science 372, eaay 4895 (2021).
  * Redner [2013] S. Redner,  _A Guide to First-Passage Processes_ (Cambridge University Press, 2013).
  * Bénichou and Voituriez [2014] O. Bénichou and R. Voituriez, From first-passage times of random walks in confinement to geometry-controlled kinetics, Physics Reports 539, 225 (2014).
  * Lawley [2020] S. D. Lawley, Extreme first-passage times for random walks on networks, Phys. Rev. E 102, 062118 (2020).
  * Van Kampen [2013] N. G. Van Kampen,  _Stochastic Processes in Physics and Chemistry, Third Edition_ (North-Holland Personal Library, 2013).
  * Chechkin  _et al._ [2008] A. V. Chechkin, R. Metzler, J. Klafter, and V. Y. Gonchar, Introduction to the theory of Lévy Flights, in  _Anomalous Transport_, edited by R. Klages, G. Radons, and I. M. Sokolov (John Wiley & Sons, Ltd, 2008) Chap. 5, pp. 129–162.
  * Dudko  _et al._ [2006] O. K. Dudko, G. Hummer, and A. Szabo, Intrinsic rates and activation free energies from single-molecule pulling experiments, Phys. Rev. Lett. 96, 108101 (2006).
  * Marconi  _et al._ [2008] U. M. B. Marconi, A. Puglisi, L. Rondoni, and A. Vulpiani, Fluctuation-dissipation: Response theory in statistical physics, Physics Reports 461, 111 (2008).
  * Rotter and Gigan [2017] S. Rotter and S. Gigan, Light fields in complex media: Mesoscopic scattering meets wave control, Rev. Mod. Phys. 89, 015005 (2017).
  * Weiss [1994] G. H. Weiss,  _Aspects and Applications of the Random Walk._ (North Holland, Amsterdam, 1994).
  * Klafter and Sokolov [2011] J. Klafter and I. Sokolov,  _First Steps in Random Walks: From Tools to Applications_ (Oxford University Press, 2011).
  * Levernier  _et al._ [2021] N. Levernier, O. Bénichou, and R. Voituriez, Universality classes of hitting probabilities of jump processes, Phys. Rev. Lett. 126, 100602 (2021).
  * Klinger  _et al._ [2022] J. Klinger, R. Voituriez, and O. Bénichou, Splitting probabilities of symmetric jump processes, Phys. Rev. Lett. 129, 140603 (2022).
  * Klinger  _et al._ [2023] J. Klinger, R. Voituriez, and O. Bénichou, Leftward, rightward, and complete exit-time distributions of jump processes, Phys. Rev. E 107, 054109 (2023).
  * Vezzani  _et al._ [2019] A. Vezzani, E. Barkai, and R. Burioni, Single-big-jump principle in physical modeling, Phys. Rev. E 100, 012108 (2019).
  * Foss  _et al._ [2013] S. Foss, D. Korshunov, and S. Zachary,  _An Introduction to Heavy-Tailed and Subexponential Distributions_ (Springer New York, NY, 2013).
  * Burioni and Vezzani [2020] R. Burioni and A. Vezzani, Rare events in stochastic processes with sub-exponential distributions and the big jump principle, J. Stat. Mech. , 034005 (2020).
  * Barkai  _et al._ [2000] E. Barkai, V. Fleurov, and J. Klafter, One-dimensional stochastic Lévy-Lorentz gas, Phys. Rev. E 61, 1164 (2000).
  * Burioni  _et al._ [2010] R. Burioni, L. Caniparoli, and A. Vezzani, Lévy walks and scaling in quenched disordered media, Phys. Rev. E 81, 060101 (2010).
  * Ślȩzak and Burov [2021] J. Ślȩzak and S. Burov, From diffusion in compartmentalized media to non-Gaussian random walks, Scientific Reports 11, 5101 (2021).
  * Bousige  _et al._ [2021] C. Bousige, P. Levitz, and B. Coasne, Bridging scales in disordered porous media by mapping molecular dynamics onto intermittent Brownian motion, Nature Communications 12, 1043 (2021).
  * Vezzani  _et al._ [2020] A. Vezzani, E. Barkai, and R. Burioni, Rare events in generalized Lévy Walks and the Big Jump principle, Scientific Reports 10, 2732 (2020).
  * Wang  _et al._ [2019] W. Wang, A. Vezzani, R. Burioni, and E. Barkai, Transport in disordered systems: The single big jump approach, Phys. Rev. Res. 1, 033172 (2019).
  * Sparre Andersen [1954] E. Sparre Andersen, On the fluctuations of sums of random variables ii, Mathematica Scandinavica 2, 195 (1954).
  * Sparre Andersen [1953] E. Sparre Andersen, On sums of symmetrically dependent random variables, Scandinavian Actuarial Journal 1953, 123 (1953).
  * Majumdar  _et al._ [2017] S. N. Majumdar, P. Mounaix, and G. Schehr, Survival probability of random walks and Lévy flights on a semi-infinite line, Journal of Physics A: Mathematical and Theoretical 50, 465002 (2017).
  * Zaburdaev  _et al._ [2015] V. Zaburdaev, S. Denisov, and J. Klafter, Lévy walks, Rev. Mod. Phys. 87, 483 (2015).
  * Burioni  _et al._ [2012] R. Burioni, S. di Santo, S. Lepri, and A. Vezzani, Scattering lengths and universality in superdiffusive Lévy materials, Phys. Rev. E 86, 031125 (2012).
  * Artuso  _et al._ [2018] R. Artuso, G. Cristadoro, M. Onofri, and M. Radice, Non-homogeneous persistent random walks and Lévy-Lorentz gas, Journal of Statistical Mechanics: Theory and Experiment 2018, 083209 (2018).
  * Bianchi  _et al._ [2020a] A. Bianchi, G. Cristadoro, M. Lenci, and M. Ligabò, Random walks in a one-dimensional levy random environment, Journal of Statistical Physics 163, 22 (2020a).
  * Bianchi  _et al._ [2020b] A. Bianchi, M. Lenci, and F. Pène, Continuous-time random walk between lévy-spaced targets in the real line, Stochastic Processes and their Applications 130, 708 (2020b).
  * Zamparo [2023] M. Zamparo, Large fluctuations and transport properties of the Lévy-Lorentz gas, Annales de l’Institut Henri Poincaré, Probabilités et Statistiques 59, 621 (2023).
  * Beenakker  _et al._ [2009] C. W. J. Beenakker, C. W. Groth, and A. R. Akhmerov, Nonalgebraic length dependence of transmission through a chain of barriers with a Lévy spacing distribution, Phys. Rev. B 79, 024204 (2009).