# Enhancing Temporal Understanding in LLMs for Semi-structured Tables

Irwin Deng, Kushagra Dixit , Vivek Gupta , Dan Roth
University of Pennsylvania
{ideng, dixitk, gvivek<EMAIL_ADDRESS>Work done during internship at
UPenn. corresponding author

###### Abstract

Temporal reasoning over tabular data presents substantial challenges for large
language models (LLMs), as evidenced by recent research. In this study, we
conduct a comprehensive analysis of temporal datasets to pinpoint the specific
limitations of LLMs. Our investigation leads to enhancements in TempTabQA, a
dataset specifically designed for tabular temporal question answering. We
provide critical insights for improving LLM performance in temporal reasoning
tasks with tabular data. Furthermore, we introduce a novel approach, C.L.E.A.R
to strengthen LLM capabilities in this domain. Our findings demonstrate that
our method significantly improves evidence-based reasoning across various
models. Additionally, our experimental results reveal that indirect
supervision with auxiliary data substantially boosts model performance in
these tasks. This work contributes to a deeper understanding of LLMs’ temporal
reasoning abilities over tabular data and promotes advancements in their
application across diverse fields.

##  1 Introduction

Q1 : How many MLB wins did Al McBean have before turning 20? A: 0 (evidence:
Born, MLB statistics)
Q2 : For how many years did Al McBean play for the Pittsburgh Pirates? A: 9
(evidence: Teams)
Q3 : What was the duration of Al McBean’s Major League Baseball career? A: 10
(evidence: MLB debut, Last MLB appearance)

Figure 1: A semi-structured table of Baseball pitcher Al McBean with follow up
question answers.

Large Language Models (LLMs) have exhibited remarkable proficiency across
various natural language processing tasks. However, recent investigations
reveal a notable deficiency in their ability to reason effectively over
_tabular_ data, particularly when _temporal_ relationships are involved Chen
(2023); Sui et al. (2024). This discrepancy between model performance and
human-level understanding underscores the pressing need for innovative
approaches to enhance the capabilities of LLMs in this domain.

To explore the root causes behind the limitations in reasoning about
structured or semi-structured data, we conducted a thorough analysis of
TempTabQA Gupta et al. (2023) dataset, an important and only datasets in this
domain. Through meticulous examination, we identified discrepancies in model
comprehension and reasoning, leading to the development of an enhanced
evaluation set. Our investigation also delved into the mechanisms through
which LLMs tackle temporal reasoning tasks using conventional prompting
techniques, leasing in the introduction of a novel approach, C.L.E.A.R
(Comprehend, Locate, Examine, Analyze, Resolve), designed to considerably
enhance temporal reasoning in LLMs.

While prompts have been instrumental in guiding models to better reason about
tasks through explicit instructions, they do not _inherently improve_ task
understanding. In our research, we demonstrate that despite the implementation
of our novel C.L.E.A.R approach, models, though improved, still overlook
crucial context—such as relevant rows in tables—and suffer from data leakage,
relying more on memorization rather than grounding their decisions on the
evidence. Additionally, since C.L.E.A.R lacks temporal aspects in its prompt,
its effectiveness is limited. This observation necessitate for more robust
solutions, where model inherently improve i.e. it’s parameters changes.

To address these challenges, we advocate for learning through fine-tuning
processes. Fine tuning on temporal data can inherently improve the model’s
temporal reasoning ability. We utilized the TRAM Wang and Zhao (2024) dataset
to engage in an indirectly supervised fine-tuning process by using Auxiliary
Out of Domain(OOD) data. This approach significantly improved model _cross-
generalization_ performance on temporal questions on tabular data. Our
findings indicate that combining our novel C.L.E.A.R prompting approach with
fine-tuning on the TempTabQA test-set yields optimal results. This combination
effectively addresses the limitations in LLMs’ temporal reasoning
capabilities, paving the way for future advancements in the field. In summary,
this paper advances the understanding and enhancement of Large Language
Models’ (LLMs) abilities to reason about temporal relationships in tabular
data. Our key contributions are as follows:

  * •

We conducted a thorough analysis of the TempTabQA test set to assess the
limitations of current approaches. This analysis also led to an improved test
set, enhancing the evaluation of models in temporal tabular reasoning.

  * •

We introduce an novel approach, C.L.E.A.R (Comprehend, Locate, Examine,
Analyze, Resolve), designed to significantly enhance temporal reasoning
capabilities in LLMs for tabular data. Our approach grounds models in
evidence, thus reducing memorization.

  * •

Through indirect supervision technique a.k.a. auxiliary task training, we
inherently improve model performance. Fine-tuning models with the auxiliary
TRAM dataset leads to substantial enhancements in handling temporal questions
on tabular data.

These contributions greatly enhance LLMs’ temporal reasoning capabilities,
paving the way for future research and applications in temporal reasoning over
tabular data. _Our code and enhanced evaluation set will be released for
public use._

##  2 Temporal Reasoning on Tables

Tables organize and record diverse types of information, making them useful
for studying an entity’s timeline. They provide a chronological sequence of
events, facilitating the analysis of the progression of positions, marital
status changes, and awards, thereby serving as reliable sources for temporal
reasoning. Entity-centric tables, like Wikipedia Infoboxes, significantly
differ from unstructured data and fully structured data (e.g., SQL tables and
knowledge graphs).

Recent studies show that Large Language Models (LLMs) struggle with reasoning
over tabular data, especially concerning temporal aspects. Various datasets
have been used to explore LLMs’ understanding of tabular data, including
TempTabQA Gupta et al. (2023), Table2vec Zhang et al. (2019), TAPAS Herzig et
al. (2020), TaBERT Yin et al. (2020), TabStruc Zhang et al. (2020a), TABBIE
Iida et al. (2021), TabGCN Pramanick and Bhattacharya (2021), and RCI Glass et
al. (2021). These datasets provide various contexts and challenges, aiding in
the development and evaluation of models for tabular data processing.

Among these datasets, we focus on TempTabQA, a recent and prominent dataset
for temporal reasoning over tabular data. TempTabQA features temporal
question-answer pairs derived from Wikipedia Infobox tables and is notable for
its two distinct eval sets: the "Head" set, with pairs from popular and
frequent domains, and the "Tail" set, with pairs from less common and rare
domains. The eval sets include approximately 2,900 question-answer pairs, with
around 1,900 in the "Head" set and 1,000 in the "Tail" set. This bifurcation
enables comprehensive evaluation, assessing models’ performance across both
frequent and rare domains.

##  3 Where do LLMs Fails?

We used Chain of Thought prompting with GPT-3.5 to analyze the TempTabQA test
set and evaluate current models’ performance and limitations. Out of 1,038
examples, 339 were incorrect responses. Of these, 159 errors were due to data
issues, while 180 were due to model limitations. These errors fall into the
following categories:

1\. Tabular Data Issues (75 examples): These errors were related to
hallucinations, incomplete evidence extraction, missing evidence, or incorrect
information extraction.

2\. Temporal Calculation Errors (84 examples): These involved difficulties
with calculations related to time, such as determining age, calculating the
time between dates in different months, or assessing whether a value fell
within a specified range.

3\. Other Errors (31 examples): This category included errors stemming from
common-sense reasoning, arithmetic, and other miscellaneous issues.

Our observations indicate that even with Chain of Thought reasoning, models
generate incorrect responses consistently. The models not only produce
incorrect answers but also exhibit hallucinations, struggling with temporal
calculations and common-sense reasoning. This emphasizes the need for enhance
models performance in this domain. The 159 data-related issues fall into these
categories:

1\. Tables Requiring External Knowledge to Answer (75 examples): These
questions could not be answered correctly without additional information not
present in the table, indicating a gap in the information provided in the
context.

2\. Wrong Human Annotation or Multiple Correct Answers (42 examples): These
instances involved incorrect annotations by humans or questions that had
multiple valid answers, but the annotations provided only one correct answer.

3\. Ambiguous or Incomplete Questions (14 examples): These questions are
either vague or lacked sufficient detail required for correct answer.

4\. Other Issues (28 examples): This category included various problems such
as questions relying on images within the HTML table or missing rows in the
JSON table etc.

Through our analysis, we addressed all these categories of errors and created
a new evaluation set designed to better assess model performance. This refined
dataset eliminates the noise caused by the aforementioned issues, providing a
more accurate benchmark for evaluating model capabilities. In this paper, when
evaluating a model’s performance, we primarily use the model’s accuracy on the
full test dataset as a basis for comparison, unless otherwise stated. This
approach ensures consistency and allows for a clear assessment of improvements
made through the new evaluation set.

##  4 Methodology

Improving the temporal reasoning capabilities of Large Language Models is
crucial for enhancing their performance on tasks involving time-based data.
Current models often struggle with accurately interpreting and reasoning about
temporal information. This limitation reduces their effectiveness in real-
world applications.

Our analysis revealed several key areas where current models underperform in
reasoning over tabular data, especially in temporal contexts. To address these
challenges, we explored two approaches. First, we developed a novel method
called C.L.E.A.R (Comprehend, Locate, Examine, Analyze, Resolve). This method
was inspired by our detailed examination of the TempTabQA test set. C.L.E.A.R
instructs models specifically for tabular data reasoning tasks. It employs
advanced prompting techniques to guide models more effectively, aiming to
reduce errors and enhance accuracy in temporal question answering. Secondly,
we fine-tuned models using auxiliary temporal data from unrelated domains to
enhance cross-generalization. This approach boosts temporal understanding on
complex temporal tasks.

###  4.1 C.L.E.A.R Prompting

In this section, we introduce the C.L.E.A.R (Comprehend, Locate, Examine,
Analyze, Resolve) technique, designed to enhance temporal reasoning over semi-
structured data. C.L.E.A.R is a structured, step-by-step approach that guides
the process of understanding, identifying, examining, analyzing, and resolving
questions involving temporal reasoning. It ensures a comprehensive and logical
extraction of information from tables (refer Fig. 2).

(a.) Comprehend: The first step, Comprehend, involves applying domain
knowledge to understand the given question. For example, if the question
pertains to calculating time differences, it is essential to recognize that
this involves subtracting the earlier year from the later year. This step
ensures the correct interpretation of the question and sets the foundation for
correct structural solution analysis.

(b.) Locate: The Locate step focuses on identifying and extracting the
relevant rows from the table that directly pertain to the question. This
involves explaining the rationale behind the selection of these rows to
provide transparency and clarity. For instance, if the question asks for
events between specific years, only the rows corresponding to those years
should be selected. The output of this step includes listing the relevant rows
on new lines for clarity and ease of reference.

(c.) Examine: In the Examine phase, the main question is broken down into
smaller, more manageable sub-questions. Each sub-question aims at extracting a
specific piece of information from the table that is necessary to answer the
main question. This decomposition allows for a systematic approach to solving
the question, ensuring that no critical piece of information is overlooked.

(d.) Analyze: The Analyze step is multifaceted, involving several sub-steps:

  1. 1.

Mark Evidence for Each Sub-Question: For each sub-question, identify the
specific evidence from the table that will be used to answer it. Explain the
relevance of this evidence to the sub-question in detail.

  2. 2.

Determine Dependencies: This sub-step assesses whether the answer to a sub-
question depends on information from previous sub-questions. If dependencies
exist, they are stated to maintain logical coherence.

  3. 3.

Reasoning for Each Sub-Question: A logical explanation is provided for how the
evidence leads to the answer for each sub-question. This includes detailed
reasoning for transparency and to support the answer’s validity.

(e.) Resolve: The final step, Resolve, involves combining and applying the
answers from the sub-questions to formulate the final answer to the main
question. This step includes explaining the reasoning process leading to the
final answer, which may involve necessary calculations or logical deductions.
This synthesis ensures the final answer is well-supported and logically
derived from evidence gathered in preceding steps. Refer Figure 3 in Appendix
A for step-by-step process of C.L.E.A.R.

###  4.2 Fine Tuning with Auxiliary Data

Fine-tuning models can significantly boost their capabilities by adjusting
parameters through training on task-specific examples. In this section, we
demonstrate how models’ temporal reasoning can be enhanced inherently, not
only through fine-tuning on specific data but also by integrating auxiliary
data sources.

Auxiliary data, such as temporal unstructured data, does not directly relate
to the main task but contains relevant logical structures and principles. This
type of data enhances the model’s understanding of underlying logic, improving
the overall performance. The TRAM dataset, illustrated in Figure 5 in Appendix
A, serves as an auxiliary source to boost model performance. It includes
temporal questions that aid the model in grasping temporal relationships
across diverse contexts.

The TRAM dataset, though unrelated to tabular data temporal reasoning,
enhances model temporal reasoning skills. Exposing the model to diverse
temporal questions improves its ability to handle temporal relationships,
boosting performance. This approach shows leveraging auxiliary data can
effectively address temporal reasoning complexities and enhance model
robustness in diverse scenarios.

##  5 Experimental Setup

Models: In this paper, we experimented with several state-of-the-art large
language models (LLMs), including GPT-3.5-Turbo, GPT-4, PaLM-2, Mistral-2-7B,
LLaMA-2-7B-chat, and Gemini 1.5 Pro Flash. These models represent the
forefront in both open-source and closed-model applications, showcasing
advancements in natural language understanding and generation capabilities.

Prompts: Prompting models with detailed instructions enhances their
understanding of tasks, leading to improved responses. These prompts may
include demonstrations for the model’s reference. We explore the following
prompting techniques:

\- Chain of Thought (CoT) Wei et al. (2023): CoT prompts guide the model
through a series of interconnected thoughts or reasoning steps, encouraging
coherent and structured responses. We apply CoT in both zero-shot (Z.S) and
few-shot (F.S) settings to evaluate its impact on model performance.

\- Faithful Chain of Thought (F-CoT) Lyu et al. (2023): F-CoT extends CoT by
emphasizing fidelity to the initial prompt throughout response generation to
maintain consistency and accuracy in model outputs. We apply F-CoT in both
zero-shot and few-shot scenarios to evaluate its effectiveness.

\- Program of Thought (PoT) Chen et al. (2023): PoT provides the model with a
predefined sequence of operations, akin to a program, to generate structured
and task-specific responses. Like CoT and F-CoT, PoT is evaluated in zero-shot
and few-shot contexts to enhance model performance.

Auxiliary Data: In our experiments, we utilize several unstructured temporal
reasoning datasets to assess the temporal reasoning capabilities of language
models:

\- DATE Understanding Srivastava et al. (2023): This dataset evaluates a
model’s ability to comprehend, manipulate, and reason about dates in diverse
formats and contexts. Tasks include Date Format Conversion, Date Arithmetic,
Date Recognition, Relative Date Interpretation, and Time Reasoning.

\- Temporal Sequences Srivastava et al. (2023): This dataset tests language
models on logical deduction tasks. Models are given a series of events and
their durations, and they deduce the possible timing of additional activities.

\- TRAM dataset Wang and Zhao (2024): This benchmark dataset comprises ten
distinct datasets, each focusing on different temporal aspects: order,
arithmetic, frequency, and duration. The TRAM dataset aims to evaluate
language models’ temporal reasoning capabilities comprehensively. Table 5
shows examples of questions from the TRAM dataset, highlighting the variety
and complexity of temporal reasoning tasks included.

##  6 Results and Analysis

In this section, we present the results of enhancing temporal reasoning in
Large Language Models (LLMs) for tabular data tasks. We evaluate the
effectiveness of two strategies: C.L.E.A.R (Comprehend, Locate, Examine,
Analyze, Resolve), a novel method tailored for tabular data reasoning, and the
integration of out-of-domain temporal data for model fine-tuning.

###  6.1 C.L.E.A.R Prompting

We analyzed C.L.E.A.R prompting against effective techniques like CoT, F-CoT,
and PoT in zero-shot and few-shot settings. Our goal was to evaluate their
performance across various models and determine the most effective approach.

Prompt | No FT | TRAM | TempTabQA
---|---|---|---
100 | 1000 | 100 | 1000
| GPT-3.5 turbo
C.L.E.A.R | 77.99% | 77.92% | 76.12% | 80.89% | 82.04%
F.S. CoT | 73.49% | 75.53% | 76.19% | 77.81% | 79.47%
F.S. F-CoT | 70.54% | 71.13% | 72.72% | 77.85% | 76.57%
F.S. PoT | 70.13% | 70.58% | 69.75% | 79.37% | 77.57%
Z.S. CoT | 65.84% | 68.64% | 69.61% | 73.73% | 72.72%
Z.S. F-CoT | 64.38% | 67.32% | 65.91% | 71.27% | 71.69%
Z.S. PoT | 65.39% | 67.39% | 68.05% | 72.17% | 72.76%
| LLAMA 7B
C.L.E.A.R | 59.33% | 61.65% | 59.85% | 63.69% | 63.17%
F.S. CoT | 61.51% | 64.87% | 63.66% | 67.22% | 68.64%
F.S. F-CoT | 61.96% | 61.54% | 64.24% | 66.77% | 67.32%
F.S. PoT | 62.69% | 64.04% | 66.08% | 67.43% | 65.94%
Z.S. CoT | 61.65% | 62.48% | 61.79% | 63.07% | 65.21%
Z.S. F-CoT | 63.93% | 66.98% | 69.40% | 63.38% | 64.31%
Z.S. PoT | 63.34% | 63.69% | 64.76% | 65.07% | 65.42%
| PALM 2
C.L.E.A.R | 80.06% | 81.97% | 83.28% | 85.01% | 85.01%
F.S. CoT | 77.40% | 81.14% | 82.24% | 80.17% | 81.59%
F.S. F-CoT | 75.63% | 75.35% | 78.82% | 82.45% | 82.21%
F.S. PoT | 76.01% | 75.98% | 76.25% | 84.56% | 80.27%
Z.S. CoT | 70.13% | 73.97% | 74.42% | 78.89% | 76.74%
Z.S. F-CoT | 69.75% | 74.04% | 74.90% | 75.22% | 75.94%
Z.S. PoT | 71.24% | 74.32% | 76.29% | 79.09% | 79.13%
| MISTRAL-2 7B
C.L.E.A.R | 69.33% | 73.52% | 72.41% | 74.80% | 73.87%
F.S. CoT | 69.23% | 70.61% | 71.13% | 73.31% | 75.63%
F.S. F-CoT | 68.19% | 71.06% | 70.02% | 74.84% | 75.25%
F.S. PoT | 67.60% | 70.51% | 69.99% | 74.35% | 73.59%
Z.S. CoT | 65.25% | 66.22% | 70.37% | 71.62% | 71.48%
Z.S. F-CoT | 66.56% | 71.13% | 69.51% | 67.08% | 71.10%
Z.S. PoT | 64.31% | 66.63% | 66.98% | 71.69% | 71.65%
Table 1: Model’s performance on various prompts without fine-tuning, with
fine-tuning on auxiliary data (TRAM), and with fine-tuning on the TempTabQA
dataset using 100 and 1000 examples. The best result is highlighted in bold.

As shown in Table 1, C.L.E.A.R consistently outperforms other techniques
across various models without fine-tuning, except for LLaMA-2. For GPT-3.5
Turbo, our method achieves a 4.5% performance boost compared to the next best
technique. For PaLM-2, it leads to a 2.7% improvement.

These findings highlight C.L.E.A.R prompting’s effectiveness in enhancing
model performance, particularly in scenarios where fine-tuning is not
feasible. Significant performance gains observed with GPT-3.5 Turbo and PaLM-2
demonstrate its potential to improve temporal reasoning and overall task
comprehension in large language models.

###  6.2 Efficacy of C.L.E.A.R

The results in Section 6.1 demonstrate that our method is more effective than
other prompting techniques. However, is C.L.E.A.R _trustworthy?_ Does it
really enhance the model’s _evidence-based reasoning capabilities?_ Gupta et
al. (2021). Our experiments compare C.L.E.A.R with Zero Shot and Few Shot
Chain of Thought (CoT) approaches across tasks highlighting model
deficiencies. Below are each of task descriptions:

  1. 1.

Original Table: The model uses the original table to answer the question,
testing its ability to effectively utilize the provided data.

  2. 2.

Without Table: The model is asked to answer the question without access to the
table, testing whether the model has memorized the answers or can deduce them
independently. Here, an ideal model’s performance should suffer as it doesn’t
use pre-trained knowledge.

  3. 3.

Altered Entity Name: The table and question are provided to the model with an
altered entity name. This checks the model’s reasoning ability without relying
on memorized data.

  4. 4.

Missing Relevant Rows: The model is given the original table with relevant
rows deleted. This evaluates whether the model can use external knowledge to
answer questions. Responses are evaluated against gold answers.

  5. 5.

Information Absence Detection: The model receives the original table with
relevant rows deleted. This task evaluates the model’s accuracy in identifying
missing information.

For the last two tasks i.e. Missing Relevant Rows and Information Absence
Detection, we perform evaluations in two settings:

  * •

Original Prompt: The model is tested with the original prompt to see if it can
detect missing information or use external knowledge.

  * •

Updated Prompt: The model is explicitly instructed that the information may or
may not be present, to see if explicit instructions improve performance.

We aim to assess if C.L.E.A.R prompting improves evidence-based reasoning in
models beyond accuracy. This analysis evaluates its effectiveness in
addressing specific reasoning challenges and enhancing model reliability
across diverse contexts.

#### Results and Analysis.

We tested zero-shot Chain of Thought (CoT), few-shot CoT, and C.L.E.A.R
prompting techniques across various tasks. In Table 2, C.L.E.A.R consistently
outperforms other methods on most tasks, except for LLaMA-2 and Mistral-2
models, which show varied results. GPT 4O achieves the highest accuracy at
85.08%, marking a 2.39% improvement over few-shot CoT.

(a) Without Table. In this task, when evaluated without table access,
C.L.E.A.R’s performance declines across all models using original labels. This
indicates that it mitigates memorization issues seen in other techniques by
emphasizing contextual reliance. Specifically, we observe a decrease of ~4%
for GPT-4O and GPT-3.5 Turbo, and an 11.04% decrease for Llama.

(b) Altered Table Entity. C.L.E.A.R performs best overall in this task, except
for LLaMA-2, PaLM, and Mistral-2. GPT 4O provides best accuracy of 82.62%
which is a bosst of 6.26% in comparision of few shot CoT. LLaMA-2 7B and
Mistral-2 7B show weaker performance mainly due to their smaller size, which
generally leads to less effective adherence to instructions compared to larger
models. For models GPT 3.5 turbo and Gemini 1.5 pro flash, we observe a gain
of 4.15% and 8.51% respectively.

Models | Task (expected) | Prompts
---|---|---
Z.S CoT | F.S CoT | C.L.E.A.R
GPT 4O | Original Table ($\uparrow$) | 75.80% | 82.69% | 85.08%
Without Table ($\downarrow$) | 75.91% | 73.31% | 69.44%
Altered Name Entity ($\uparrow$) | 74.94% | 76.36% | 82.62%
GPT 3.5 | Original Table ($\uparrow$) | 65.56% | 73.35% | 78.05%
Without Table ($\downarrow$) | 62.27% | 62.10% | 58.22%
Altered Name Entity($\uparrow$) | 61.09% | 67.95% | 72.10%
Llama | Original Table ($\uparrow$) | 62.31% | 62.41% | 59.12%
Without Table ($\downarrow$) | 54.69% | 42.85% | 31.81%
Altered Name Entity | 66.87% | 60.78% | 58.15%
Palm 2 | Original Table ($\uparrow$) | 69.75% | 77.29% | 79.58%
Without Table ($\downarrow$) | 66.63% | 63.24% | 60.26%
Altered Name Entity ($\uparrow$) | 82.10% | 71.34% | 74.97%
MISTRAL-2 | Original Table ($\uparrow$) | 64.87% | 69.37% | 69.30%
Without Table ($\downarrow$) | 56.87% | 52.58% | 49.36%
Altered Name Entity ($\uparrow$) | 74.87% | 63.55% | 64.83%
GEMINI 1.5 | Original Table ($\uparrow$) | 73.17% | 80.34% | 83.28%
Without Table ($\downarrow$) | 70.85% | 69.19% | 68.64%
Altered Name Entity ($\uparrow$) | 79.02% | 71.24% | 79.75%
Table 2: This table provides a comparison among C.L.E.A.R, zero-shot CoT, and
few-shot CoT prompts across Original Table, No Table, and Altered Named Entity
tasks in evidence-based reasoning. We evaluate performance using GPT-4o,
GPT-3.5 Turbo, LLAMA-7B-Chat, PaLM-2, MISTRAL-7B, and Gemini 1.5 Pro Fash
models in our experiments.

(c) Missing Relevant Row. We observe in Table 3, C.L.E.A.R outperforms others
across all models with the original prompt. It shows the largest performance
decline compared to zero-shot and few-shot CoT. This trend continues with the
modified prompt, where our method remains superior, except for GPT-4, where
zero-shot performs slightly better. Prompting creates an additional drop of
~1-5% in all models. Explicitly informing the model about the availability of
information promotes context-driven responses over reliance on memorized data.

(d) Information Absence Detection In Table 3, our method outperforms zero-shot
and few-shot CoT with the original prompt across all models. Llama achieves
the highest accuracy at 40.05%, followed by GPT-3.5 at 31.71%. With the
updated prompt, our technique consistently performs best for all models.
Gemini 1.5 Pro Flash achieves the highest accuracy at 91.45%, with GPT-4O
close behind at 90.65

Models | Original Prompt | Updated Prompt
---|---|---
| CoT |  | CoT |
| Z.S | F.S | C.L.E.A.R | Z.S | F.S | C.L.E.A.R
| Missing Relevant Rows (Lower ($\downarrow$) is better)
GPT 4O | 69.71% | 70.89% | 62.82% | 6.89% | 8.34% | 7.30%
GPT 3.5 | 60.06% | 56.59% | 49.46% | 11.60% | 13.05% | 11.39%
LLAMA 7B | 64.42% | 37.69% | 24.92% | 26.10% | 16.13% | 11.18%
PALM 2 | 78.40% | 61.68% | 55.90% | 10.90% | 8.27% | 7.58%
MISTRAL-2 | 69.92% | 50.85% | 39.84% | 16.44% | 13.08% | 10.76%
GEMINI 1.5 | 76.25% | 68.05% | 57.29% | 9.07% | 8.45% | 7.34%
| Information Absence Detection (Higher ($\uparrow$) is better
GPT 4O | 19.45% | 22.71% | 25.16% | 89.89% | 88.85% | 90.65%
GPT 3.5 | 27.83% | 29.73% | 31.71% | 79.75% | 79.47% | 82.07%
LLAMA 7B | 25.37% | 34.96% | 40.05% | 61.72% | 61.96% | 62.34%
PALM 2 | 19.21% | 23.68% | 29.11% | 86.43% | 86.78% | 88.44%
MISTRAL-2 | 21.05% | 29.42% | 32.71% | 75.42% | 74.49% | 77.09%
GEMINI 1.5 | 15.82% | 22.33% | 25.23% | 89.48% | 87.61% | 91.45%
Table 3: This table compares C.L.E.A.R, zero-shot CoT, and few-shot CoT
prompts for tasks involving Missing Relevant Rows and Information Absence
Detection in evidence-based reasoning, considering both original and changed
prompts. Experimental evaluations are conducted using GPT-4o, GPT-3.5 Turbo,
LLAMA-7B-Chat, PaLM-2, MISTRAL-7B, and Gemini 1.5 Pro Fash models.

###  6.3 Auxiliary Data Fine Tuning

C.L.E.A.R enhances the model’s capacity to process and reason with context. It
facilitates effective instruction for models to handle temporal questions and
improve evidence-based reasoning. However, prompting alone does not adjust
their inherent parameters, so models do not intrinsically improve. Innate
enhancement in models requires fine-tuning, so that their parameters better
reflect improved temporal reasoning capabilities. We suggest fine-tuning the
model on simple auxiliary data to enhance its ability to reason across diverse
data formats and improve overall reasoning capabilities.

#### Why TRAM dataset?

We evaluated the impact of fine-tuning GPT-3.5 Turbo using the auxiliary
datasets discussed in Section 5, along with the TempTabQA dataset. For this
evaluation, we fine-tuned with 100 examples from each dataset.

Train Dataset | Exact Match
---|---
No fine-tuning | 73.49%
DATE | 74.28%
Temporal Sequences | 74.42%
TRAM | 75.53%
TempTabQA | 77.81%
Table 4: This table showcases the performance of GPT-3.5 Turbo following fine-
tuning on auxiliary datasets within a zero-shot chain-of-thought (CoT)
setting."

Analysis. Table 4 shows that the TRAM dataset offers the largest performance
improvement among auxiliary datasets. While DATE and Temporal Sequences
datasets provide gains of less than 1.01%, TRAM achieves a significant 2.04%
boost compared to the base model without fine-tuning. These findings emphasize
TRAM’s unique advantages. DATE focuses solely on date-based reasoning, and
Temporal Sequences on temporal reasoning across events. In contrast, TRAM
includes 10 sub-datasets covering diverse temporal reasoning tasks. It
includes tasks such as _Ordering, Frequency, Duration, Typical time, Ambiguity
Resolution, Arithmetic, Relations, Temporal NLI, Causality and Storytelling_.
Each task have multiple question type including _Commonsense, Facts, Reading
Comprehension, Application, Computation, Direct & Multi-step Comparison,
Interpretation, Calender shift, Long-term, Mid-term & Short-term shift, Date
Computation, 12 & 24 hour adjustment, Week Identification etc_. (For more
information on TRAM, refer to Table 6 in Appendix A). This diversity enhances
performance by exposing models to varied temporal scenarios, improving overall
comprehension in temporal domain.

The TRAM dataset’s significant gains highlight the importance of diverse
auxiliary data for fine-tuning. By covering a wide range of temporal reasoning
tasks, TRAM helps models grasp nuanced temporal relationships, enhancing
performance across various challenges. This finding suggests that future
efforts in model fine-tuning should consider leveraging similarly diverse
datasets to maximize performance improvements.

#### Fine-tuning on TRAM.

We analyzed fine-tuning models on subsets of the TRAM dataset and TempTabQA,
each with 100 and 1000 examples (evenly sampled across tasks), as shown in
Table 1.

Analysis. Our findings reveal that, on the TRAM dataset with fine-tuning on
100 examples, C.L.E.A.R outperforms all models except for LLAMA-2 7B. This
trend is consistent with 1000 examples, where it slightly trails behind the
few-shot CoT for GPT-3.5 turbo by a negligible margin of 0.07%.When fine-
tuning models on TempTabQA with 100 examples and 1000 examples, our method
demonstrates superior performance for GPT 3.5 turbo and PaLM-2, but
underperforms for LLAMA-2 7B and Mistral-2 7B.

Our findings show that fine-tuning models with auxiliary data significantly
improves performance, close to that of task-specific fine-tuning. This method
enhances models’ ability to reason about temporal information and can be
applied across various tasks. Additionally, auxiliary data typically provides
richer resources compared to task-specific datasets. This scalability enables
fine-tuning on larger datasets beyond traditional methods. 111Due to
computational limits, especially with GPT and Gemini models, we fine-tuned
LLMs on just 1000 examples.

##  7 Comparison with Related Work

#### Tabular Reasoning

Recent research has extensively explored large language models’ applications
with semi-structured tabular data Chen et al. (2020b); Gupta et al. (2020);
Zhang and Balog (2019), covering areas such as question answering and semantic
parsing Zhang et al. (2020b); Zhang and Balog (2020); Pasupat and Liang
(2015); Krishnamurthy et al. (2017); Abbas et al. (2016); Sun et al. (2016);
Chen et al. (2020c); Lin et al. (2020); Zayats et al. (2021); Oguz et al.
(2020); Chen et al. (2021b); Iyyer et al. (2017), as well as table-to-text
generation Parikh et al. (2020); Li et al. (2021); Nan et al. (2021); Yoran et
al. (2021); Chen et al. (2020a). Various datasets and models have emerged to
handle semi-structured data, including Table2vec Zhang et al. (2019), TAPAS
Herzig et al. (2020), TaBERT Yin et al. (2020), TabStruc Zhang et al. (2020a),
TABBIE Iida et al. (2021), TabGCN Pramanick and Bhattacharya (2021), and RCI
Glass et al. (2021), all aimed at improving understanding and representation
of tabular data. Research has also focused on fine-tuning models for enhanced
inference on tabular data, as demonstrated by studies Yu et al. (2018);
Eisenschlos et al. (2020); Neeraja et al. (2021), showcasing targeted
techniques to boost model performance.

Recently, Srivastava et al. (2024) introduced EEDP, a novel approach for
financial document Question Answering. EEDP retrieves finance domain
information, extracts relevant table rows, and decomposes mathematical tasks
into atomic operations. Our method similarly involves understanding table data
and extracting relevant rows, but we differentiate by systematically
addressing sub-problems based on extracted evidence. This targeted approach
makes our method well-suited for temporal tabular reasoning.

#### Temporal Reasoning

In temporal question answering, recent datasets like TIME-SENSITIVEQA (Chen et
al., 2021c) and TORQUE (Ning et al., 2020) focus on entity-specific reading
comprehension with time-sensitive questions. Other datasets like TEMPQA-WD
(Neelam et al., 2022), CRONQUESTIONS (Saxena et al., 2021), and TEMPQUESTIONS
(Jia et al., 2018a) explore temporal links in knowledge graph embeddings.
There are also open-domain (Zhang and Choi, 2021) and cloze-form (Dhingra et
al., 2022) question-answering tasks, and event-centric datasets (Ning et al.,
2018; Wen et al., 2021; Chen et al., 2021a). Models like CRONKBQA (Saxena et
al., 2021), TEQUILA (Jia et al., 2018b), and others (Jia et al., 2021; Shang
et al., 2021; Mavromatis et al., 2021; Kannen et al., 2022) have been tailored
for knowledge-based question answering with temporal considerations, often
integrating temporal aspects during pre-training (Dhingra et al., 2022; Iv et
al., 2022). Additionally, temporal reasoning across structured and
unstructured data has seen advancements with works such as TempTabQA Gupta et
al. (2023), TRAM Wang and Zhao (2024), Temporal Sequencescite Srivastava et
al. (2023), and DATE understanding Srivastava et al. (2023), all addressing
the challenge of handling temporal information effectively.

Our work focuses on enhancing temporal reasoning in tabular data. We introduce
a novel prompting method to improve language models’ temporal reasoning
abilities with tabular data. Additionally, we demonstrate the effectiveness of
augmenting models’ temporal understanding using auxiliary data, a novel
approach in the field. This combined approach significantly enhances temporal
inference, paving the way for future advancements.

##  8 Conclusion and Future Work

Our results demonstrate that C.L.E.A.R (Comprehend, Locate, Examine, Analyze,
Resolve) prompting significantly enhances model performance, particularly in
understanding tabular data and tasks requiring temporal reasoning. It promotes
grounding LLMs in evidence context rather than relying solely on pre-trained
knowledge. Moreover, fine-tuning LLMs with auxiliary temporal data has proven
highly effective in enhancing temporal understanding. This study validates
that integrating auxiliary data strengthens models’ temporal reasoning
capabilities, thereby enhancing the robustness of large language models. For
Future, we propose (a.) Generation of Synthetic Data: Creating synthetic
training data from temporal aspects of tabular data enhances model performance
through diverse temporal exposure. (b.) Neuro-symbolic Learning: Integrating
neural networks with symbolic reasoning improves models’ understanding of
temporal information for more effective solutions. (c.) Expanding C.L.E.A.R
Prompting Applications: Applying C.L.E.A.R prompting to diverse tasks and
domains validates its versatility in natural language processing, enhancing
reasoning with temporal information. (d.) Integration with Existing Models:
Seamlessly integrating C.L.E.A.R prompting and auxiliary data into existing
models maximizes benefits without architectural changes.

## Limitations

The experiments in this paper have been conducted exclusively on the English
language. This study can be extended to a multilingual setting to evaluate the
approach’s effectiveness across different languages. Additionally, the
temporal datasets used in our study are limited to simple, entity-centric
tables. Since structured data can exist in more complex forms, such as
hierarchical tables, further research is necessary to assess the impact of our
methods on these more complex structures.

Moreover, our computational limitations restricted us to fine-tuning models on
only 1000 samples of auxiliary data. To fully understand the potential
improvements from fine-tuning on auxiliary data, it is essential to explore
the effects of fine-tuning on larger datasets. Future work should focus on
overcoming these limitations to provide a comprehensive evaluation of our
approach.

## Ethics Statement

We confirm that our work adheres to the highest ethical standards in research
and publication. We will publicly release our code and enhanced evaluation set
to enable the research community to validate and build upon our findings. We
are committed to the responsible and fair use of computational linguistics
methodologies. The claims in our paper accurately reflect the experimental
results. While using black-box large language models introduces some
stochasticity, we mitigate this by maintaining a fixed temperature. We utilize
an AI assistive tools for writing while ensuring absence of bias. We provide
comprehensive details on annotations, dataset splits, models used, and
prompting methods tried, ensuring the reproducibility of our work.

##  9 Acknowledgement

Research was sponsored by the Army Research Office and was accomplished under
Grant Number W911NF-20-1-0080. The views and conclusions contained in this
document are those of the authors and should not be interpreted as
representing the official policies, either expressed or implied, of the Army
Research Office or the U.S. Government. The U.S. Government is authorized to
reproduce and distribute reprints for Government purposes notwithstanding any
copyright notation herein. This work was partially funded by ONR Contract
N00014-23-1-2365.

## References

  * Abbas et al. (2016) Faheem Abbas, M. K. Malik, M. Rashid, and Rizwan Zafar. 2016.  Wikiqa — a question answering system on wikipedia using freebase, dbpedia and infobox.  _2016 Sixth International Conference on Innovative Computing Technology (INTECH)_ , pages 185–193.
  * Chen et al. (2021a) Muhao Chen, Hongming Zhang, Qiang Ning, Manling Li, Heng Ji, Kathleen McKeown, and Dan Roth. 2021a.  Event-centric natural language understanding.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics_.
  * Chen (2023) Wenhu Chen. 2023.  Large language models are few(1)-shot table reasoners.  In _Findings of the Association for Computational Linguistics: EACL 2023_ , pages 1120–1130, Dubrovnik, Croatia. Association for Computational Linguistics.
  * Chen et al. (2021b) Wenhu Chen, Ming-Wei Chang, Eva Schlinger, William Yang Wang, and William W. Cohen. 2021b.  Open question answering over tables and text.  In _International Conference on Learning Representations_.
  * Chen et al. (2020a) Wenhu Chen, Jianshu Chen, Yu Su, Zhiyu Chen, and William Yang Wang. 2020a.  Logical natural language generation from open-domain tables.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7929–7942, Online. Association for Computational Linguistics.
  * Chen et al. (2023) Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. 2023.  Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks.
  * Chen et al. (2020b) Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. 2020b.  Tabfact: A large-scale dataset for table-based fact verification.  In _International Conference on Learning Representations_.
  * Chen et al. (2021c) Wenhu Chen, Xinyi Wang, and William Yang Wang. 2021c.  A dataset for answering time-sensitive questions.  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)_.
  * Chen et al. (2020c) Wenhu Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong, Hong Wang, and William Yang Wang. 2020c.  HybridQA: A dataset of multi-hop question answering over tabular and textual data.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 1026–1036, Online. Association for Computational Linguistics.
  * Dhingra et al. (2022) Bhuwan Dhingra, Jeremy R. Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William W. Cohen. 2022.  Time-aware language models as temporal knowledge bases.  _Transactions of the Association for Computational Linguistics_ , 10:257–273.
  * Eisenschlos et al. (2020) Julian Eisenschlos, Syrine Krichene, and Thomas Müller. 2020.  Understanding tables with intermediate pre-training.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 281–296, Online. Association for Computational Linguistics.
  * Glass et al. (2021) Michael Glass, Mustafa Canim, Alfio Gliozzo, Saneem Chemmengath, Vishwajeet Kumar, Rishav Chakravarti, Avi Sil, Feifei Pan, Samarth Bharadwaj, and Nicolas Rodolfo Fauceglia. 2021.  Capturing row and column semantics in transformer based question answering over tables.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1212–1224, Online. Association for Computational Linguistics.
  * Gupta et al. (2021) Vivek Gupta, Riyaz A. Bhat, Atreya Ghosal, Manish Srivastava, Maneesh Singh, and Vivek Srikumar. 2021.  Is my model using the right evidence? systematic probes for examining evidence-based tabular reasoning.  _CoRR_ , abs/2108.00578.
  * Gupta et al. (2023) Vivek Gupta, Pranshu Kandoi, Mahek Vora, Shuo Zhang, Yujie He, Ridho Reinanda, and Vivek Srikumar. 2023.  TempTabQA: Temporal question answering for semi-structured tables.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 2431–2453, Singapore. Association for Computational Linguistics.
  * Gupta et al. (2020) Vivek Gupta, Maitrey Mehta, Pegah Nokhiz, and Vivek Srikumar. 2020.  INFOTABS: Inference on tables as semi-structured data.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 2309–2324, Online. Association for Computational Linguistics.
  * Herzig et al. (2020) Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Eisenschlos. 2020.  Tapas: Weakly supervised table parsing via pre-training.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics.
  * Iida et al. (2021) Hiroshi Iida, Dung Thai, Varun Manjunatha, and Mohit Iyyer. 2021.  Tabbie: Pretrained representations of tabular data.
  * Iv et al. (2022) Robert Iv, Alexandre Passos, Sameer Singh, and Ming-Wei Chang. 2022.  FRUIT: Faithfully reflecting updated information in text.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3670–3686, Seattle, United States. Association for Computational Linguistics.
  * Iyyer et al. (2017) Mohit Iyyer, Wen-tau Yih, and Ming-Wei Chang. 2017.  Search-based neural structured learning for sequential question answering.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1821–1831, Vancouver, Canada. Association for Computational Linguistics.
  * Jia et al. (2018a) Zhen Jia, Abdalghani Abujabal, Rishiraj Saha Roy, Jannik Strötgen, and Gerhard Weikum. 2018a.  Tempquestions: A benchmark for temporal question answering.  In _Companion Proceedings of the The Web Conference 2018_ , WWW ’18, page 1057–1062, Republic and Canton of Geneva, CHE. International World Wide Web Conferences Steering Committee.
  * Jia et al. (2018b) Zhen Jia, Abdalghani Abujabal, Rishiraj Saha Roy, Jannik Strötgen, and Gerhard Weikum. 2018b.  TEQUILA: Temporal Question Answering over Knowledge Bases.  In _Proceedings of the 27th ACM International Conference on Information and Knowledge Management_ , CIKM ’18, pages 1807–1810, New York, NY, USA. ACM.
  * Jia et al. (2021) Zhen Jia, Soumajit Pramanik, Rishiraj Saha Roy, and Gerhard Weikum. 2021.  Complex temporal question answering on knowledge graphs.  In _Proceedings of the 30th ACM International Conference on Information & Knowledge Management_, pages 792–802.
  * Kannen et al. (2022) Nithish Kannen, Udit Sharma, Sumit Neelam, Dinesh Khandelwal, Shajith Ikbal, Hima Karanam, and L Venkata Subramaniam. 2022.  Targeted extraction of temporal facts from textual resources for improved temporal question answering over knowledge bases.  _arXiv preprint arXiv:2203.11054_.
  * Krishnamurthy et al. (2017) Jayant Krishnamurthy, Pradeep Dasigi, and Matt Gardner. 2017.  Neural semantic parsing with type constraints for semi-structured tables.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 1516–1526, Copenhagen, Denmark. Association for Computational Linguistics.
  * Li et al. (2021) Tongliang Li, Lei Fang, Jian-Guang Lou, and Zhoujun Li. 2021.  TWT: Table with written text for controlled data-to-text generation.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 1244–1254, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Lin et al. (2020) Xi Victoria Lin, Richard Socher, and Caiming Xiong. 2020.  Bridging textual and tabular data for cross-domain text-to-SQL semantic parsing.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 4870–4888, Online. Association for Computational Linguistics.
  * Lyu et al. (2023) Qing Lyu, Shreya Havaldar, Adam Stein, Li Zhang, Delip Rao, Eric Wong, Marianna Apidianaki, and Chris Callison-Burch. 2023.  Faithful chain-of-thought reasoning.  In _Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 305–329, Nusa Dua, Bali. Association for Computational Linguistics.
  * Mavromatis et al. (2021) Costas Mavromatis, Prasanna Lakkur Subramanyam, Vassilis N. Ioannidis, Soji Adeshina, Phillip R. Howard, Tetiana Grinberg, Nagib Hakim, and George Karypis. 2021.  Tempoqr: Temporal question reasoning over knowledge graphs.
  * Nan et al. (2021) Linyong Nan, Dragomir Radev, Rui Zhang, Amrit Rau, Abhinand Sivaprasad, Chiachun Hsieh, Xiangru Tang, Aadit Vyas, Neha Verma, Pranav Krishna, Yangxiaokang Liu, Nadia Irwanto, Jessica Pan, Faiaz Rahman, Ahmad Zaidi, Mutethia Mutuma, Yasin Tarabar, Ankit Gupta, Tao Yu, Yi Chern Tan, Xi Victoria Lin, Caiming Xiong, Richard Socher, and Nazneen Fatema Rajani. 2021.  DART: Open-domain structured data record to text generation.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 432–447, Online. Association for Computational Linguistics.
  * Neelam et al. (2022) Sumit Neelam, Udit Sharma, Hima Karanam, Shajith Ikbal, Pavan Kapanipathi, Ibrahim Abdelaziz, Nandana Mihindukulasooriya, Young-Suk Lee, Santosh Srivastava, Cezar Pendus, Saswati Dana, Dinesh Garg, Achille Fokoue, G P Shrivatsa Bhargav, Dinesh Khandelwal, Srinivas Ravishankar, Sairam Gurajada, Maria Chang, Rosario Uceda-Sosa, Salim Roukos, Alexander Gray, Guilherme Lima, Ryan Riegel, Francois Luus, and L Venkata Subramaniam. 2022.  A benchmark for generalizable and interpretable temporal question answering over knowledge bases.
  * Neeraja et al. (2021) J. Neeraja, Vivek Gupta, and Vivek Srikumar. 2021.  Incorporating external knowledge to enhance tabular reasoning.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2799–2809, Online. Association for Computational Linguistics.
  * Ning et al. (2020) Qiang Ning, Hao Wu, Rujun Han, Nanyun Peng, Matt Gardner, and Dan Roth. 2020.  TORQUE: A reading comprehension dataset of temporal ordering questions.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1158–1172, Online. Association for Computational Linguistics.
  * Ning et al. (2018) Qiang Ning, Ben Zhou, Zhili Feng, Haoruo Peng, and Dan Roth. 2018.  CogCompTime: A tool for understanding time in natural language.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 72–77, Brussels, Belgium. Association for Computational Linguistics.
  * Oguz et al. (2020) Barlas Oguz, Xilun Chen, Vladimir Karpukhin, Stan Peshterliev, Dmytro Okhonko, Michael Schlichtkrull, Sonal Gupta, Yashar Mehdad, and Scott Yih. 2020.  Unified open-domain question answering with structured and unstructured knowledge.  _arXiv preprint arXiv:2012.14610_.
  * Parikh et al. (2020) Ankur Parikh, Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang, and Dipanjan Das. 2020.  ToTTo: A controlled table-to-text generation dataset.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1173–1186, Online. Association for Computational Linguistics.
  * Pasupat and Liang (2015) Panupong Pasupat and Percy Liang. 2015.  Compositional semantic parsing on semi-structured tables.  In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 1470–1480, Beijing, China. Association for Computational Linguistics.
  * Pramanick and Bhattacharya (2021) Aniket Pramanick and Indrajit Bhattacharya. 2021.  Joint learning of representations for web-tables, entities and types using graph convolutional network.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 1197–1206, Online. Association for Computational Linguistics.
  * Saxena et al. (2021) Apoorv Saxena, Soumen Chakrabarti, and Partha Talukdar. 2021.  Question answering over temporal knowledge graphs.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 6663–6676, Online. Association for Computational Linguistics.
  * Shang et al. (2021) Chao Shang, Peng Qi, Guangtao Wang, Jing Huang, Youzheng Wu, and Bowen Zhou. 2021.  Open temporal relation extraction for question answering.  In _3rd Conference on Automated Knowledge Base Construction_.
  * Srivastava et al. (2023) Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, and Abubakar Abid. 2023.  Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.
  * Srivastava et al. (2024) Pragya Srivastava, Manuj Malik, Vivek Gupta, Tanuja Ganu, and Dan Roth. 2024.  Evaluating llms’ mathematical reasoning in financial document question answering.
  * Sui et al. (2024) Yuan Sui, Mengyu Zhou, Mingjie Zhou, Shi Han, and Dongmei Zhang. 2024.  Table meets llm: Can large language models understand structured table data? a benchmark and empirical study.  In _Proceedings of the 17th ACM International Conference on Web Search and Data Mining_ , pages 645–654.
  * Sun et al. (2016) Huan Sun, Hao Ma, Xiaodong He, Scott Wen-tau Yih, Yu Su, and Xifeng Yan. 2016.  Table cell search for question answering.  In _Proceedings of the companion publication of the 25th international conference on World Wide Web_. ACM - Association for Computing Machinery.
  * Wang and Zhao (2024) Yuqing Wang and Yun Zhao. 2024.  Tram: Benchmarking temporal reasoning for large language models.
  * Wei et al. (2023) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023.  Chain-of-thought prompting elicits reasoning in large language models.
  * Wen et al. (2021) Haoyang Wen, Yanru Qu, Heng Ji, Qiang Ning, Jiawei Han, Avi Sil, Hanghang Tong, and Dan Roth. 2021.  Event time extraction and propagation via graph attention networks.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 62–73, Online. Association for Computational Linguistics.
  * Yin et al. (2020) Pengcheng Yin, Graham Neubig, Wen tau Yih, and Sebastian Riedel. 2020.  Tabert: Pretraining for joint understanding of textual and tabular data.
  * Yoran et al. (2021) Ori Yoran, Alon Talmor, and Jonathan Berant. 2021.  Turning tables: Generating examples from semi-structured tables for endowing language models with reasoning skills.  _arXiv preprint arXiv:2107.07261. Version 1._
  * Yu et al. (2018) Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018.  Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 3911–3921, Brussels, Belgium. Association for Computational Linguistics.
  * Zayats et al. (2021) Vicky Zayats, Kristina Toutanova, and Mari Ostendorf. 2021.  Representations for question answering from documents with tables and text.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 2895–2906, Online. Association for Computational Linguistics.
  * Zhang et al. (2020a) Hongzhi Zhang, Yingyao Wang, Sirui Wang, Xuezhi Cao, Fuzheng Zhang, and Zhongyuan Wang. 2020a.  Table fact verification with structure-aware transformer.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1624–1629, Online. Association for Computational Linguistics.
  * Zhang et al. (2019) Li Zhang, Shuo Zhang, and Krisztian Balog. 2019.  Table2vec: Neural word and entity embeddings for table population and retrieval.  In _Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval_ , SIGIR ’19. ACM.
  * Zhang and Choi (2021) Michael Zhang and Eunsol Choi. 2021.  SituatedQA: Incorporating extra-linguistic contexts into QA.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 7371–7387, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Zhang and Balog (2019) Shuo Zhang and Krisztian Balog. 2019.  Auto-completion for data cells in relational tables.  In _Proceedings of the 28th ACM International Conference on Information and Knowledge Management_ , CIKM ’19, pages 761–770, New York, NY, USA. ACM.
  * Zhang and Balog (2020) Shuo Zhang and Krisztian Balog. 2020.  Web table extraction, retrieval, and augmentation: A survey.  _ACM Trans. Intell. Syst. Technol._ , 11(2):13:1–13:35.
  * Zhang et al. (2020b) Shuo Zhang, Zhuyun Dai, Krisztian Balog, and Jamie Callan. 2020b.  Summarizing and exploring tabular data in conversational search.  In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_ , SIGIR ’20, pages 1537–1540, New York, NY, USA. Association for Computing Machinery.

##  Appendix A Appendix

Ordering |  Q: Arrange the following events in chronological order: (1) Brusilov Offensive by Russia. (2) Kamehameha I of the Island of Hawaii defeats the Oahuans at the Battle of Nu’uanu. (3) The Kuomintang, the Chinese nationalist party, is founded. (4) Emperor Claudius dies and is succeeded by his grand nephew Nero. (5) St. Norbert and 29 companions make their solemn vows marking the beginning of the Premonstratensian Order.
---|---
(Facts) |  A. (1), (2), (4), (5), (3) $\times$ B. (4), (5), (2), (3), (1) ✓ C. (3), (1), (2), (4), (5) $\times$
Frequency |  Q: It is also a love story, between Ace and Tobio, a trans woman. How often do they break up?
(Commonsense) |  A. Once ✓ B. Always $\times$ C. Once per week $\times$
Duration |  Q: While Yoga Session gave attendees time to plant an entire garden, Jazz Concert was enough to water a few plants, and Board Game Night was merely smelling a flower. Which event was the longest?
(Analogy Inference) |  A. Jazz Concert $\times$ B. Board Game Night $\times$ C. Yoga Session ✓
Typical Time |  Q: Which event typically happens earlier: morning yoga or farmer starting their day?
(Comparison) |  A. Morning yoga $\times$ B. Farmer starting their day ✓ C.Around the same time$\times$
Ambiguity Resolution |  Q: The dynasty which fell in 1830 had risen to power roughly 90 years earlier. When was its establishment?
(Long-term Shift) |  A. 1742 $\times$ B. 1745 $\times$ C 1740 ✓
Arithmetic |  Q: In which week of year 2007 does the date 10-12-2007 occur?
(Week Identification) |  A.Week41 ✓ B.Week28 $\times$ C.Week5 $\times$ D.Week10 $\times$
Temporal Relation |  Q: It added that the Ministry of Economic Affairs and Finance was assigned to draw up practical procedure for the ceding, while the Ministry of Welfare and Social Security would be responsible for identifying the beneficiaries in two months. What is the relationship between the event ‘added’ and the event ‘ceding’?
|  A. IS_INCLUDED $\times$ B. SIMULTANEOUS $\times$ C. AFTER ✓
Temporal NLI |  Q: Premise: Two guys playing football on a campus green. Hypothesis: They are practicing before the big game tomorrow
|  A. Entailment $\times$ B. Neutral ✓ C. Contradiction$\times$
Temporal Causality |  Q: The seasons changed from summer to autumn. What’s the more plausible RESULT?
(Effect) |  A. People evacuated their homes. $\times$ B. Leaves fell from the trees. ✓
Temporal Storytelling |  Q: There is a huge clock in my living room. I turned the clock back one hour for daylight savings. My wife also turned the clock back one hour for daylight savings. Our 2 kids each turned the clock back one hour for daylight savings. Which of the two endings is the most plausible correct ending to the story?
|  A. Then we wondered why it got so dark so early. ✓ B. The kids were not
happy $\times$
Table 5: Examples Questions in TRAM Dataset. Task |  Data Size |  $\\#$ Problem Types |  Metrics |  Answer Type |  Text Sources
---|---|---|---|---|---
Foundational Temporal Understanding Tasks
Ordering |  29,462 |  Commonsense, Facts |  Acc. |  3-Way MC |  MCTACO1, Wikipedia, Misc.
Frequency |  4,658 |  Commonsense, Reading Comprehension, Application, Computation, Comparison, Facts |  Acc. |  3-Way MC |  MCTACO1, SQuAD2, Misc.
Duration |  7,232 |  Commonsense, Reading Comprehension, Analogy Inference, Computation, Direct Comparison, Multi-step Comparison, Facts |  Acc. |  3-Way MC |  Same
Typical Time |  13,018 |  Commonsense, Comparison, Facts, Reading Comprehension |  Acc. |  3-Way MC |  Same
Temporal Interpretation and Computation Tasks
Amb. Res. |  3,649 |  Interpretation, Calendar shift, Long-term shift, Mid-term shift, Short-term shift |  Acc. |  3-Way MC |  Misc.
Arithmetic |  15,629 |  Application, Date Computation, 12-hour Adjustment, 24-hour Adjustment, Month Shift, Week Identification, Year Shift, Time Computation, Time Zone Conversion |  Acc. |  4-Way MC |  Same
Advanced Temporal and Conceptual Understanding Tasks
Relation |  102,462 |  - |  Acc./F1 |  3-Way MC |  TempEval-33
Temporal NLI |  282,144 |  - |  Acc./F1 |  3-Way MC |  MNLI4, SNLI5
Causality |  1,200 |  Cause, Effect |  Acc. |  2-Way MC |  COPA6, Misc.
Storytelling |  67,214 |  - |  Acc. |  2-Way MC |  ROC7, SCT8
Table 6: Overview of tasks in TRAM. Figure 2: Prompt Example Figure 3: The figure illustrates the step-by-step process of C.L.E.A.R instruction. The reference table is provided in Figure 1 Input : Given an entity-centric table and corresponding question, follow the steps below exactly to answer the question: Step 1. Comprehend Information: Apply domain knowledge to understand how to approach and answer the question. Step 2. Locate Relevant Rows: Identify and extract any rows from the table that could be relevant to the question. Explain why these rows are selected. Also output the relevant rows verbatim. Step 3. Examine the Question: Determine whether the original question can be solved by a simple table lookup. If not, break down the original question into between 2 and 4 smaller, more manageable sub-questions. Assume each sub-problem can be solved using the provided evidence. Explain how one could approach solving each sub-problem. Step 4. Analyze Sub-Questions: Answer each sub-question using evidence from the table. Explain the step-by-step reasoning process that leads to each answer. Include any calculations or logical deduction to arrive at each conclusion, no matter how simple. Step 5. Resolve to Form the Final Answer: Using the answers to each sub-question, answer the original question. Explain the step-by-step reasoning process that leads to this final answer. Include any calculations or logical deduction to arrive at each conclusion, no matter how simple. Clearly state the final answer using "Final Answer:", providing the answer as concisely as possible without unnecessary information. Each table-question pair is presented as a table (identified by "Table:") followed by a question (identified by "Q:"). Tables are presented in a linear format, with columns separated by tabs, rows separated by newlines, and subsections separated by double newlines. If necessary, assume the current date is December, 2022. ======================== Table: Title Bette Davis Died October 6, 1989 | (1989-10-06) | (aged 81) | Neuilly-sur-Seine, France Resting place Forest Lawn Memorial Park Occupation Actress Years active 1929-1989 Political party Democratic Spouse(s) Harmon Oscar Nelson | ( | married | 1932; | divorced | 1938) | Arthur Farnsworth | ( | married | 1940; died 1943) | William Grant Sherry | ( | married | 1945; | divorced | 1950) | Gary Merrill | ( | married | 1950; | divorced | 1960) Children 3, including B. D. Hyman Bette Davis [’BETTEDavis (cropped).jpg | Publicity photo, 1935’] Question: How many years after Davis’s career began did she take the photo? Figure 4: Example 1 from TempTabQA head set with C.L.E.A.R prompting on GPT 3.5 turbo : Input Response : Step 1. Comprehend Information Domain Knowledge: \- The question asks "how many years", so the answer should be a length of time in years \- To determine the time between two years, take the absolute value of the difference between the two years. Step 2. Locate Relevant Rows \- "Title Bette Davis": This indicates that the information in the table is about Bette Davis. \- "Years active 1929-1989": This provides the start and end year of Bette Davis’s career. \- "Bette Davis [’BETTEDavis (cropped).jpg | Publicity photo, 1935’]": This provides the year the photo was taken. Step 3. Examine the Question OQ) How many years after Davis’s career began did she take the photo? The table doesn’t provide this information. To determine the time between these two events, we need to know the following: • SQ1) When did Davis’s career begin? • SQ2) When did Davis take the photo? : Davis took the photo in 1935 SQ1) When did Davis’s career begin? The table gives the range years spanning Davis’s career. We can determine the begin year of her career by looking at the start year of this range. SQ2) When did Davis take the photo? The table provides the year the photo was taken. Step 4. Answer Sub-Questions SQ1) Davis’s career lasted from 1929 to 1989. Therefore, her career began in 1929. SQ2) The photo was taken in 1935. Step 5. Synthesize to Form the Final Answer OQ) How many years after Davis’s career began did she take the photo? Davis’s career began in 1929 (SQ1), and she took the photo in 1935 (SQ2). To get the number of years between these two events, take the absolute value of the difference between the two years. The absolute value of (1929-1935) is 6. Therefore, the photo was taken 6 years after Davis’s career began. Final Answer: 6 Figure 5: Example 1 from TempTabQA head set with C.L.E.A.R prompting on GPT 3.5 turbo : Response Input : Given an entity-centric table and corresponding question, follow the steps below exactly to answer the question: Step 1. Comprehend Information: Apply domain knowledge to understand how to approach and answer the question. Step 2. Locate Relevant Rows: Identify and extract any rows from the table that could be relevant to the question. Explain why these rows are selected. Also output the relevant rows verbatim. Step 3. Examine the Question: Determine whether the original question can be solved by a simple table lookup. If not, break down the original question into between 2 and 4 smaller, more manageable sub-questions. Assume each sub-problem can be solved using the provided evidence. Explain how one could approach solving each sub-problem. Step 4. Analyze Sub-Questions: Answer each sub-question using evidence from the table. Explain the step-by-step reasoning process that leads to each answer. Include any calculations or logical deduction to arrive at each conclusion, no matter how simple. Step 5. Resolve to Form the Final Answer: Using the answers to each sub-question, answer the original question. Explain the step-by-step reasoning process that leads to this final answer. Include any calculations or logical deduction to arrive at each conclusion, no matter how simple. Clearly state the final answer using "Final Answer:", providing the answer as concisely as possible without unnecessary information. Each table-question pair is presented as a table (identified by "Table:") followed by a question (identified by "Q:"). Tables are presented in a linear format, with columns separated by tabs, rows separated by newlines, and subsections separated by double newlines. If necessary, assume the current date is December, 2022. ======================== Table: Title Dwight Bernard Dwight Bernard [’Dwight Bernard.jpg’] Pitcher Born | ( | 1952-05-31 | ) | May 31, 1952 | (age 70) | Mount Vernon, Illinois Batted | Right | Threw | Right MLB debut MLB debut [’June 29, 1978, for the New York Mets’] Last MLB appearance Last MLB appearance [’October 2, 1982, for the Milwaukee Brewers’] MLB statistics Win-loss record 4-8 Earned run average 4.14 Strikeouts 92 Teams Teams [’New York Mets (1978-1979) | Milwaukee Brewers (1981-1982)’] Question: In what year did Dwight Bernard start playing with the last team he was associated with? Figure 6: Example 2 from TempTabQA tail set with C.L.E.A.R prompting on GPT 3.5 turbo : Input Response : Step 1. Comprehend Information Domain Knowledge: \- The question asks "In what year", so the answer should be a year. \- To determine the last team someone was associated with, identify the one that occurred most recently. \- Given an event spanning a range of years, the start is the first number in that range. Step 2. Locate Relevant Rows \- "Title Dwight Bernard": This indicates that the information in the table is about Dwight Bernard. \- "Teams [’New York Mets (1978-1979) | Milwaukee Brewers (1981-1982)’]": This provides a list of all teams Dwight Bernard played with and the years he played with each. Step 3. Examine the Question OQ) In what year did Dwight Bernard start playing with the last team he was associated with? The table doesn’t provide this information. To determine this, we need to know the following: • SQ1) What was the last team Dwight Bernard was associated with? • SQ2) When did Dwight Bernard start playing for this team? SQ1) What was the last team Dwight Bernard was associated with? The table doesn’t provide this information. However, the table provides a list of all teams Bernard was associated with and the years he was associated with each team. We can determine the last team Dwight Bernard was associated with by comparing the last year he was associated with each of his teams. SQ2) When did Dwight Bernard start playing for this team? The table provides the range of years Bernard was associated with each of his teams. We can determine the year he started playing for that team by looking at the start year in the range. Step 4. Analyze Sub-Questions SQ1) The only teams Dwight Bernard has played with are the New York Mets and Milwaukee Brewers. Bernard played for the Mets from 1978 to 1979 and the Brewers from 1981 to 1982. He was associated with the Brewers in 1982, which is more recent than the Mets in 1979; therefore, the last team he was associated with is the Brewers. SQ2) He played with the Brewers from 1981 to 1982, so he started playing for the Brewers in 1981. Step 5. Resolve to Form the Final Answer OQ) In what year did Dwight Bernard start playing with the last team he was associated with? The last team Dwight Bernard was associated with was the Brewers (SQ1), which he started playing for in 1981 (SQ2). Therefore, Dwight Bernard started playing with the last team he was associated with in 1981. Final Answer: 1981 Figure 7: Example 2 from TempTabQA tail set with C.L.E.A.R prompting on GPT 3.5 turbo : Response