11institutetext: RIAG, India.

# Domain knowledge assisted cyst segmentation in OCT retinal images

Karthik Gopinath  Jayanthi Sivaswamy

###### Abstract

3D imaging modalities are becoming increasingly popular and relevant in
retinal imaging owing to their effectiveness in highlighting structures in
sub-retinal layers. OCT is one such modality which has great importance in the
context of analysis of cystoid structures in subretinal layers. Signal to
noise ratio(SNR) of the images obtained from OCT is less and hence automated
and accurate determination of cystoid structures from OCT is a challenging
task. We propose an automated method for detecting/segmenting cysts in 3D OCT
volumes. The proposed method is biologically inspired and fast aided by the
domain-knowledge about the cystoid structures. An ensemble learning method-
Random forests is learnt for classification of detected region into cyst
region. The method achieves detection and segmentation in a unified setting.
We believe the proposed approach with further improvements can be a promising
starting point for more robust approach. This method is validated against the
training set achieves a mean dice coefficient of 0.3893 with a standard
deviation of 0.2987

##  1 Introduction

Optical coherence tomography (OCT) is a non-invasive imaging modality that
captures 3D projections of the retinal layers using low-coherence light waves
[2]. In assessing retinal diseases, OCT is a handy tool. It has a potential to
provide 3D information and hence analyze subretinal layers which are not
captured by more conventional techniques such as color fundus imaging. Early
versions of OCT were of limited use owing to low image quality and acquisition
time. Fourier transformation of the optical spectrum of the low-coherence
interferometry refers to Spectral-domain OCT (SD-OCT) or Fourier-domain OCT.
SD-OCT is less noisy and significantly faster than the previous technology.
The increased speed and number of scans renders greater image detail and
clarity [3]. The higher SNR of SD-OCT allows assessment of smaller
pathological changes. SD-OCT is becoming one of the most important ancillary
test for the diagnosis of sight degrading diseases today. 3D cross sectional
volumetric images of the retina and the sub-retinal layers are imaged from SD-
OCT to detect pathologies such as cysts in addition to retinal pathologies.

Cystoid macular edema (CME) [4] [5] represents a common pathological
appearance in retina and co-occurs in a variety of pathological conditions
such as intra ocular inflammation, central or branch retinal vein occlusion
[6], diabetic retinopathy [7] and most commonly following cataract extraction
[8] . CME appearing in one eye, increases the likelihood of similar appearance
in the second eye. CME causes, multiple cyst-like (cystoid) regions filled
with fluid appearing in macula. Cysts cause retinal swelling or edema. The
areas of retina in which the cells are displaced are cysts. CME leads to many
complications such as loss of vision. Persistent CME for more than 6 to 9
months leads to chronic macular changes, with permanent impairment of central
vision.

While automated detection of CME in SD-OCT images is relevant for early
diagnosis and prevention of vision-loss, this task is nontrivial and
challenging. SD-OCT images often suffer from noise, apart from the demarcation
between subretinal layers not being very clear. While long and high power
exposure to LASER can alleviate some of these difficulties, patient health and
other medical considerations often impose some necessary bounds on such
approaches. Thus automated CME detection from noisy SD-OCT data is a
challenging and relevant problem in retinal image analysis

This work proposes a biologically inspired and simple methodology for an
automatic segmentation of candidate cyst regions, in SD-OCT retinal images.
This paper is organized as follows: in Section 2 the methodology for automatic
cyst detection is exposed. Section 3 shows the results obtained with this
technique and in Section 4 conclusions and future lines of work.

##  2 Methodology

The detection or segmentation of cyst candidate regions is a complex task. Our
method to accomplish this task is composed of different stages (Fig. 1) a
preprocessing stage: to perform size normalizations of individual slices, A
total variance denoising, enhancement using center surround; a candidate
selection phase: segmentation of ILM and RPE layers to reduce the search
region, MSER features for region selection; finally, false positive rejection
stage.

Figure 1: Outline of the proposed methodInputPre ProcessingCanidate
seletionFalsepositiverejectionCyst regions

###  2.1 Preprocessing

The training data has 15 SD-OCT volumes from 4 different scanners. The number
of slices across the training set differ from 49 to 200. The size of each
slice is neither constant for an individual scanner nor across scanners. For
further processing, the size of the image is standardised for all volumes. All
the scanned volumes are resized to a 512X256 pixels.
SD-OCT images are degraded by the speckle noise. Speckle noise is signal
dependent and the pattern of the speckle depends on the structure of the
imaging tissue.The speckle noise is a multiplicative noise and can be modelled
using Rayleigh distribution. Speckle corrupted SD-OCT images has a have high
total variation i.e. the integral of the absolute gradient of these images is
high. We use total variational denoising [9] technique which has advantages
over the traditional denoising methods such as linear smoothing and median
filtering. The Total Variational approach will reduce the texture content
resulting in a smooth piecewise constant images preserving the edges [10] [11]
[12].
Center-surround difference is a biologically inspired technique of finding
local extrema in images [13]. As a saliency mechanism, its effectiveness has
been demonstrated in a variety of applications. Since cyst region candidates
are locally dark, we use center-surround difference in a multi-scale setting
to detect the locally minimum regions as cyst candidates.

###  2.2 Candidate selection

Selection of region of interest is a vital task as it helps in reducing the
search region for identifying cysts and computation time. Region of interest
for cyst detection task are the retinal layers between Internal Limiting
Membrane (ILM) and Retinal pigment epithelium (RPE). Areas like vitreous
cavity and area below choroid layers are not of interests in detection of cyst
regions.Various segmentation algorithms have been proposed for segmentaion of
retinal layers. Methods like pixel intensity variation based ILM and RPE
segmentation[14], active contour with a two-step kernel-based optimization
scheme [15], complex diffusion filtering with combined structure tensor
replacing thresholding [16]. In this work we use a conceptually simpler, yet
accurate, graph theory based segmentation approach [17].
Candidate regions are selected using Maximally stable extremal regions(MSER).
This feature computation gives a set of stable extremal region in an image.
This feature is used to detect the multi-scale objects without any smoothing
involved. Both fine and large structure can be detected using this feature.

###  2.3 False positive rejection

The regions picked from the MSER features contain both cyst and non cyst
regions. A bounding box around the candidate region is taken as the input for
feature extraction. A local descriptor based on the texture of pattern [18] is
calculated for the patch. This feature is extracted for all detected regions
and a random forest with 50 trees is trained for the classification task. The
detected cyst regions is considered as the segmentation result and the
performance of the system is evaluated.

##  3 Experimental Results and Discussion

The performance of our method is evaluated using OPTIMA Cyst Segmentation
Challenge MICCAI 2015 training dataset. The dataset includes 15 volumes of SD-
OCT scans from 4 different scanners containing a wide variety of retinal
cysts. The details of the dataset is shown in Table 1

Table 1: Dataset used Scanner |  Spectralis |  Cirrus |  Topcon |  Nidek |  Total
---|---|---|---|---|---
Training |  4 |  4 |  4 |  3 |  15

Segmentation results are quantitatively measured using Dice coefficient (DC)
defined below.

$Dice\ coefficient=2{\frac{|Detected\cap GT|}{|Detected|+|GT|}}$ (1)

The maximum possible value for DC is 1(indicating a perfect match between the
result and Ground Truth).

Validation on the training set is performed using a leave-one-out cross
validation approach. The mean and standard deviation for all the volumes with
ground truth taken as union of Graders 1 and 2 is mentioned in Table 2 below.

Table 2: Dice coefficient with GT as union of two graders Dice coefficient | Spectralis | Cirrus | Topcon | Nidek
---|---|---|---|---
Mean | 0.3024 | 0.3728 | 0.3736 | 0.4613
Maximum | 0.8596 | 0.9246 | 0.8638 | 0.8367
Standard deviation | 0.3436 | 0.2510 | 0.2849 | 0.2823

The DC is zero for a slice when no cyst region is detected. The mean DC for a
volume is lowered when there are few slices with zero cyst detection. The mean
DC values for the given volumes across 3 scanners is tabulated in Table 3.

Table 3: Dice coefficient excluding zero DC slices Dice coefficient | Spectralis | Cirrus | Topcon | Nidek
---|---|---|---|---
Mean | 0.6652 | 0.5606 | 0.5357 | 0.5957
Maximum | 0.8596 | 0.9246 | 0.8638 | 0.8367
Standard deviation | 0.1195 | 0.1851 | 0.1754 | 0.1496

The analysis is discussed module-wise. Table 4 indicates the percentage of
small ($<$200 pixels), medium (200 to 2000 pixels) and large($>$2000 pixels)
sized cysts detected. The SNR was observed to vary across the four scanners.
Detection fails due to noise, largely for small sized-cysts and since they
constitute a large proportion of the cysts the effect is an deterioration the
net DC value. Specifically, the MSER feature is unable to detect many small
cyst regions due to the presence of noise.

Table 4: Detection of cysts based on size from MSER features Size of cysts | No. Cyst Present | Percent detected correctly
---|---|---
Small | 2514 | 61.76
Medium | 2109 | 80.02
Large | 243 | 88.04

Along with the cyst regions, there are a lot of non-cyst regions picked up by
the candidate selection stage. These constitute false positives. The
supervised classifier will aid in reducing these false positives. The random
forest ensemble classifier performance with respect to cysts of different
sizes is listed below in Table 5. It is apparent that the detection is once
again worst for small-sized cysts compared to medium and large ones due to the
aggressive rejection by the random forest stage.

Table 5: Detection of cysts after random forest based on size Size of cysts | Percent detected correctly
---|---
Small | 27.83
Medium | 51.18
Large | 76.10

##  4 Conclusion

This paper presented a method to detect and segment the retinal cysts in a SD-
OCT volume. The failure in detection of small cyst regions (indistinguishable
from noise) is a major contributor to poor performance. Other contribution is
not having an explicit segmentation stage. The MSER regions are often only
partially overlapping with true cyst regions which leads to a lowering of DC
value of even true detections. This was found to be the case in medium-sized
cysts. Based on the above observations, future investigations can adopt
different detection strategies depending on the size of the cysts in the SD-
OCT volumes to improve the results. An advanced denoising algorithm might also
help in segmenting small sized regions. Inclusion of a final segmentation
stage (post rejection) will also aid in enhancing the performance.

## References

  * [1] Joshi, G.D., Sivaswamy, J., Krishnadas, S.:  Optic disk and cup segmentation from monocular color retinal images for glaucoma assessment.  Medical Imaging, IEEE Transactions on 30(6) (2011) 1192–1205
  * [2] Costa, R.A., Skaf, M., Melo, L.A., Calucci, D., Cardillo, J.A., Castro, J.C., Huang, D., Wojtkowski, M.:  Retinal assessment using optical coherence tomography.  Progress in retinal and eye research 25(3) (2006) 325–353
  * [3] Schuman, J.S.:  Spectral domain optical coherence tomography for glaucoma (an aos thesis).  Transactions of the American Ophthalmological Society 106 (2008) 426
  * [4] Rotsos, T.G., Moschos, M.M.:  Cystoid macular edema.  Clinical ophthalmology (Auckland, NZ) 2(4) (2008) 919
  * [5] Scholl, S., Kirchhof, J., Augustin, A.J.:  Pathophysiology of macular edema.  Ophthalmologica 224(Suppl. 1) (2010) 8–15
  * [6] Hayreh, S.S., et al.:  Retinal vein occlusion.  Indian journal of ophthalmology 42(3) (1994) 109
  * [7] Browning, D.J., Glassman, A.R., Aiello, L.P., Bressler, N.M., Bressler, S.B., Danis, R.P., Davis, M.D., Ferris, F.L., Huang, S.S., Kaiser, P.K., et al.:  Optical coherence tomography measurements and analysis methods in optical coherence tomography studies of diabetic macular edema.  Ophthalmology 115(8) (2008) 1366–1371
  * [8] Gass, J., Norton, E.:  Follow-up study of cystoid macular edema following cataract extraction.  Transactions-American Academy of Ophthalmology and Otolaryngology. American Academy of Ophthalmology and Otolaryngology 73(4) (1969) 665
  * [9] Boyd, S., Vandenberghe, L.:  Convex optimization.  Cambridge university press (2004)
  * [10] Yang, R., Yin, L., Gabbouj, M., Astola, J., Neuvo, Y.:  Optimal weighted median filtering under structural constraints.  Signal Processing, IEEE Transactions on 43(3) (1995) 591–604
  * [11] Hamza, A.B., Luque-Escamilla, P.L., Martínez-Aroza, J., Román-Roldán, R.:  Removing noise and preserving details with relaxed median filters.  Journal of Mathematical Imaging and Vision 11(2) (1999) 161–177
  * [12] Buades, A., Coll, B., Morel, J.M.:  A review of image denoising algorithms, with a new one.  Multiscale Modeling & Simulation 4(2) (2005) 490–530
  * [13] Itti, L., Koch, C., Niebur, E.:  A model of saliency-based visual attention for rapid scene analysis.  IEEE Transactions on Pattern Analysis & Machine Intelligence (11) (1998) 1254–1259
  * [14] Fabritius, T., Makita, S., Miura, M., Myllylä, R., Yasuno, Y.:  Automated segmentation of the macula by optical coherence tomography.  Optics express 17(18) (2009) 15659–15669
  * [15] Garvin, M.K., Abràmoff, M.D., Kardon, R., Russell, S.R., Wu, X., Sonka, M.:  Intraretinal layer segmentation of macular optical coherence tomography images using optimal 3-d graph search.  Medical Imaging, IEEE Transactions on 27(10) (2008) 1495–1505
  * [16] Cabrera Fernández, D., Salinas, H.M., Puliafito, C.A.:  Automated detection of retinal layer structures on optical coherence tomography images.  Optics Express 13(25) (2005) 10200–10216
  * [17] Chiu, S.J., Li, X.T., Nicholas, P., Toth, C.A., Izatt, J.A., Farsiu, S.:  Automatic segmentation of seven retinal layers in sdoct images congruent with expert manual segmentation.  Optics express 18(18) (2010) 19413–19428
  * [18] Medathati, N., Sivaswamy, J.:  Local descriptor based on texture of projections.  In: Proceedings of the Seventh Indian Conference on Computer Vision, Graphics and Image Processing, ACM (2010) 398–404


# Are Machines Better at Complex Reasoning? Unveiling Human-Machine Inference
Gaps in Entailment Verification

Soumya Sanyal$1$ Tianyi Xiao$1$ Jiacheng Liu$2$
Wenya Wang$3$ Xiang Ren$1$
$1$University of Southern California $2$University of Washington
$3$Nanyang Technological University, Singapore
<EMAIL_ADDRESS>
###### Abstract

Making inferences in text comprehension to understand the meaning is essential
in language processing. This work studies the entailment verification (EV)
problem of multi-sentence premises that requires a system to make multiple
inferences implicitly. Studying EV for such complex premises is important
because modern NLP problems, such as detecting inconsistent model-generated
rationales, require complex multi-hop reasoning. However, current textual
inference datasets mostly contain short premises that only partially focus on
these challenges. To address this, we compile an EV benchmark that includes
datasets from three NLP domains (NLI, contextual QA, and rationales)
containing multi-sentence premises. On benchmarking humans and LLMs, we find
that LLMs are better than humans in multi-hop reasoning across extended
contexts, while humans perform better in simple deductive reasoning tasks. We
also finetune a Flan-T5
model111https://huggingface.co/soumyasanyal/entailment-verifier-xxl for EV
using two training objectives to obtain a strong open-source model that
outperforms GPT-3.5 and rivals GPT-4. Finally, we use this model to filter out
inconsistent model-generated rationales in self-consistency decoding,
resulting in a 6% accuracy improvement on average across three MCQ datasets.

Are Machines Better at Complex Reasoning? Unveiling Human-Machine Inference
Gaps in Entailment Verification

Soumya Sanyal$1$ Tianyi Xiao$1$ Jiacheng Liu$2$ Wenya Wang$3$ Xiang Ren$1$
$1$University of Southern California $2$University of Washington $3$Nanyang
Technological University, Singapore<EMAIL_ADDRESS>

[sections/figures.tex]motivation

[sections/tables.tex]datasetComparisons

##  1 Introduction

A prevailing notion in cognitive psychology exists that humans make numerous
inferences to understand discourse and text Garnham (1989). These inferences
play a crucial role in linking information from disparate sections of a text
to establish its literal meaning and are closely associated with reasoning.
With the recent applications of Large Language Models (LLMs) Devlin et al.
(2019); Raffel et al. (2020); Brown et al. (2020); Chung et al. (2022) in NLP
tasks that require inference skills Minaee et al. (2021), it is thus essential
to understand and improve upon the limitations of LLMs concerning different
aspects of language inferences.

In this work, we focus on the task of entailment verification (EV) that
classifies whether a given context supports a hypothesis. To assert the
validity of a hypothesis, a system has to make multiple inferences from the
given context and its internal knowledge, which requires complex multi-hop
reasoning. Verifying the entailment of such complex premises has important
applications in rationale-generating LLMs, such as chain-of-thoughts (CoT) Wei
et al. (2022), that typically generate multi-sentence rationales with
incomplete information and suffer from inconsistent reasoning Ye and Durrett
(2022). While EV is similar to Natural Language Inference (NLI) Dagan et al.
(2006); Manning and MacCartney (2009), some interesting challenges distinguish
it from NLI. Many existing textual inference datasets, such as SNLI Bowman et
al. (2015), MNLI Williams et al. (2018), etc., mostly contain short sentence
premises that only partially encapsulate the challenges of multi-sentence
premises requiring complex reasoning. Predicting the entailment of such
complex premise-hypothesis pairs often requires multi-hop reasoning, inferring
missing information, etc., which is lacking from standard NLI datasets
Gururangan et al. (2018); McCoy et al. (2019). Thus, studying EV in the
context of modern LLMs is essential and is currently missing.

To this end, we first compile an evaluation benchmark to study the EV problem
by selecting multiple datasets across three categories: NLI, contextual QA,
and rationales. As shown in Table LABEL:tab:dataset_comparisons, the datasets
typically contain multi-sentence premises that require inferring different
types of knowledge to predict the entailment. Thus, this benchmark is more
complex than standard NLI datasets and can be used as a new evaluation
benchmark.

Next, we evaluate LLMs and humans on this benchmark and make some interesting
observations. Cognitive studies Buschman et al. (2011); Cowan (2001) have
shown that an average human brain has a limited capacity to retain only four
chunks in short-term memory, indicating a limitation of human inference
abilities over long contexts. Our analysis shows that LLMs are indeed stronger
than humans at tasks that involve multi-hop reasoning across long contexts,
reinforcing this human limitation. In contrast, humans are better in cases
that require simple deductive reasoning using substitutions, negations, etc.,
indicating that current LLMs lack consistency along these reasoning aspects.
Further, humans and LLMs perform comparably in instances requiring inferring
missing knowledge. These findings are depicted in Figure LABEL:fig:motivation
with motivating examples.

Additionally, on comparison between LLMs, we find that models finetuned on a
specific dataset category are usually strong within the category but don’t
generalize well on unseen categories. In contrast, instruction-finetuned
models are better on average, with the best performing model, GPT-4, achieving
0.79 macro-F1 and outperforming the best-open-sourced model Flan-T5-xxl by
0.08 macro-F1 on average. In order to bridge this gap, we finetune a Flan-T5
Chung et al. (2022) model on a training subset containing datasets from each
category of the above data collection. To this end, we explore two different
training approaches: a classification-based finetuning that learns to directly
predict the label and a ranking-based finetuning that learns to rank the most
supported hypothesis from a given pair of hypotheses for a given premise.
Ranking-based finetuning is better than classification, specifically in
contextual QA datasets, as it can learn a softer decision boundary. Overall,
our fine-tuned Flan-T5 outperforms GPT-3.5, baseline Flan-T5, and performs
comparably to GPT-4 on the benchmark, thus providing a strong open-sourced
model for entailment verification.

Finally, we demonstrate the utility of our finetuned models on a downstream
application of filtering unfaithful model-generated explanations. Self-
consistency (SC) Wang et al. (2023) decoding samples multiple model-generated
reasoning paths from the LLM decoder and aggregates them to predict the most
consistent answer. We use our finetuned Flan-T5 to filter out non-entailed
reasoning chains before aggregating the final prediction, which leads to 6%
performance improvement on average across three MCQ datasets.

##  2 Entailment Verification using LLMs

In this section, we formally define the task of entailment verification (EV),
the datasets used to create the evaluation benchmark, and the evaluation
procedure for evaluating different LLM baselines.

For a given premise $p$ and a hypothesis (or claim) $h$, the task of
entailment verification is to determine whether the context has information
that directly confirms the hypothesis or not, i.e., whether the hypothesis
follows from the information present in the context. This is a binary
classification task defined as $f(p,h)=\\{\textrm{{support}},\textrm{{not
support}}\\}$, where $f$ is a classifier (human/LLM).

###  2.1 Evaluation Benchmark

In this section, we list some desirable properties we want to include in the
EV benchmark, followed by details of the dataset categories we select.

  * •

Type of Premise: Typically, NLI datasets, such as SNLI, MNLI, etc., do not
contain more than one sentence in the premise, potentially leading to shortcut
learning. In contrast, we focus more on multi-sentence premises that require
complex reasoning. We also consider datasets where the premise is a rationale,
i.e., the premise is not just a logical precursor to the hypothesis but rather
an explanation. This tests the ability to evaluate model-generated rationales
Wei et al. (2022).

  * •

Type of Knowledge: Often, one or more information in the premise needs to be
used to predict support. We categorize this information as entity-grounded,
commonsense, or localized. Entity-grounded knowledge consists of information
about entities and other general knowledge verifiable on the internet. These
can be facts about general science, history, etc., or details of some known
person, event, etc. It is possible to infer this information even if not
mentioned in the premise. The commonsense knowledge is typically all
information about everyday life that humans use implicitly but cannot always
be verified online. This information is often missing from the premise and has
to be inferred implicitly. Lastly, localized information is all other
knowledge provided for understanding the events, people, or items mentioned in
the premise that are not grounded to any known entity. This information
depends on the premise’s specific context and, thus, is impossible to infer
unless stated explicitly. Please refer to Table LABEL:tab:knowledge_examples
for examples of each knowledge type.

[sections/tables.tex]knowledgeExamples

We consider three data sources for creating the entailment verification
benchmark, amounting to 10 datasets in total. In Table
LABEL:tab:dataset_comparisons, we compare these datasets across the desirable
characteristics mentioned earlier. Please refer to Appendix A for more details
on the datasets used.

##### Natural Language Inference

Given the close connection between NLI and EV, it is an obvious choice to
consider appropriate NLI datasets for the benchmark. To convert an NLI dataset
for EV, we merge the neutral and contradict labels to the not support label.
We use the following NLI datasets in our benchmark: WaNLI Liu et al. (2022),
FEVER Nie et al. (2019), and ANLI Nie et al. (2020).

[sections/tables.tex]humanResults

##### Contextual QA

Next, we consider multiple-choice question-answering datasets where the task
is to answer a question based on a given context and some options. We use a
QA-to-statement converter model Chen et al. (2021) to generate a hypothesis
statement for each question option pair. Then, the hypothesis corresponding to
the correct choice is marked as “support”, while the rest are marked as “not
support”. This process is depicted in Figure LABEL:fig:contextual_qa in
Appendix A. We include the following datasets from this category: Cosmos QA
(CosQA) Huang et al. (2019), SocialIQA (SIQA) Sap et al. (2019), DREAM Sun et
al. (2019), BoolQ Clark et al. (2019), and RACE Lai et al. (2017).

##### Rationale

Lastly, we consider data sources where human-annotated explanations are
available that justify the original hypothesis. In this case, we use the
rationales as the premise. We use the following datasets: Entailer Tafjord et
al. (2022), and ECQA Aggarwal et al. (2021).

###  2.2 Evaluation Metric

We use the macro-F1 score as the primary evaluation metric for comparing LLMs
on the entailment verification task because there are label imbalances in our
evaluation datasets. The macro-F1 score computes the unweighted mean of F1
scores for each class, ensuring equal importance for each class irrespective
of the label statistics. Please refer to Appendix C for more discussions on
the label imbalance of each dataset.

###  2.3 LLM Evaluation Setup

We evaluate two types of LLMs on the task of EV, as categorized below:

##### Task-finetuned LLMs

In this, the models considered are already finetuned on some subset of the
benchmark. We evaluate two models: RoBERTa Liu et al. (2019) (finetuned on NLI
datasets) and Entailer-11B Tafjord et al. (2022) (finetuned on Entailer
dataset). Refer to Appendix B.1 for more details on the evaluation setup for
these models.

Premise: {premise} Hypothesis: {hypothesis} Question: Given the premise, is
the hypothesis correct? Answer:

List of Boxes 1  Prompt used to evaluate instruction-finetuned LLMs for
entailment verification.

##### Instruction-finetuned LLMs

These are language models trained on a collection of NLP tasks described using
instructions, leading to generalization abilities to solve unseen tasks
described using new instructions. Here, we evaluate Flan-T5-xxl Chung et al.
(2022), GPT-3.5 Brown et al. (2020), and GPT-4 OpenAI (2023) models. To
compute the label, we first modify a given premise-hypothesis pair $(p,h)$
into a prompted input $\mathcal{P}$ using the prompt template as shown in Box
1. Next, we compute a score $s$ as defined below:

$s(p,h)=\frac{p_{LLM}({\textrm{``Yes''}|\mathcal{P}})}{p_{LLM}({\textrm{``Yes''}|\mathcal{P}})+p_{LLM}({\textrm{``No''}|\mathcal{P}})},$
(1)

where $p_{LLM}(\cdot|\mathcal{P})$ is the model’s probability distribution
over the vocabulary. If the score $s$ is higher than a threshold (typically
set to $0.5$ in all our experiments), we assign the label support, else we
assign the label not support. For GPT-4 evaluation, we directly check for the
“Yes” / “No” label prediction as the token probabilities are not accessible
via the API. Please refer to Appendix B.2 for more details about the models
and ablations on prompts.

##  3 Evaluation of Humans and LLMs

First, we randomly sample 100 instances from each dataset (i.e., 1000
instances in total) and conduct a human evaluation on this subset to estimate
average human performance. Please refer to Appendix E.1 for more details on
the annotation procedure. Additionally, we evaluate the above LLMs on this
sampled subset and report those numbers for fair comparisons with humans.
Table LABEL:tab:human_results shows the overall evaluation results.

###  3.1 Comparison among LLMs

We observe that the task-finetuned models (rows 1-2 in Table
LABEL:tab:human_results) are weaker on average compared to the instruction-
finetuned models (rows 3-5 in Table LABEL:tab:human_results). However, there
are some interesting exceptions. RoBERTa, which is finetuned on NLI datasets,
performs at par with GPT-4 on FEVER and outperforms it on WaNLI but falls
behind on contextual QA and rationale datasets. On the other hand,
Entailer-11B, which is finetuned on the Entailer dataset, outperforms GPT-4 on
Entailer but lags on most of the other datasets. This demonstrates that
finetuning an LLM using datasets from one of these categories is ineffective
in outperforming models trained on more general data. Overall, we observe that
GPT-4 is the best-performing entailment verification model on average and
Flan-T5-xxl is the best open-source LLM for this task.

###  3.2 Comparison between Humans and LLMs

In Table LABEL:tab:human_results, we report the human performance and the
corresponding annotation agreement222We use a pairwise agreement ratio that
computes the fraction of matched annotations over all pairs of annotations.
(in parenthesis) for each dataset. We note that the agreement ratios are
lowest for Contextual QA datasets, especially CosQA and SIQA. Questions in
these datasets are often based on commonsense scenarios, and sometimes, the
support for the wrong hypothesis can be debatable. Please refer to Appendix D
for examples. Between humans and LLMs, we find that humans beat all the
baseline LLMs, except GPT-4. This shows that existing open-sourced LLMs are
subpar with humans on this task. Additionally, although humans and GPT-4
perform comparably on average, we note that large misalignments exist in
different individual datasets. Specifically, we observe that ANLI, CosQA,
SIQA, and Entailer are the four datasets with $>0.1$ absolute macro-F1
difference. We analyze these four datasets in Section 3.3.

[sections/figures.tex]reasoningAnalysis2

[sections/tables.tex]finetuneResults

###  3.3 Effect of Reasoning Type

We design an analysis to understand further the misalignments between GPT-4
and humans. First, we categorize the type of reasoning required to predict an
entailment into the following four categories:

  * •

Simple Deductive (R1): The premise contains sentences that can be minimally
combined in one step to predict the support for the hypothesis. This typically
tests skills such as substitution, understanding negations, word meanings,
etc.

  * •

Complex Deductive (R2): More than one step of reasoning is required to solve
the task. Typically, this tests skills like mathematical reasoning, combining
multiple information in context, etc.

  * •

Missing Entity-grounded/Commonsense Knowledge (R3): In this, some essential
commonsense or entity-grounded knowledge is missing in the premise. Both
humans and LLMs can implicitly invoke such information from the memory or the
parametric knowledge obtained from pertaining, respectively.

  * •

Missing Localized Knowledge (R4): In this, information very specific to the
premise is missing. Typically, this is information about the subjects in the
context and is not grounded in any known entities. It is practically
impossible for humans or LLMs to infer such information.

We note that these categories are mutually exclusive.333Deductive reasoning
implies that the premise has all the necessary information. Thus, any missing
knowledge instance falls under inductive reasoning. Figure
LABEL:fig:reasoning_analysis2 depicts the aggregated results. The top plot
shows the percentage of each reasoning type among 400 samples from ANLI,
CosQA, SIQA, and Entailer, and the bottom plot compares the human and GPT-4
macro-F1 scores. Please refer to Appendices E.2, E.3, and E.4 for details on
the annotation setup, examples of each reasoning type, and detailed analysis,
respectively.

The first type in Figure LABEL:fig:reasoning_analysis2 is simple deductive
reasoning ($\sim$ 25% data). Here, humans perform better than GPT-4 by a small
margin. Instances that require simple deductive reasoning usually use
substitutions, negations, paraphrasing, etc., to prove entailment. We find
that humans are more robust than GPT-4 in performing such simple deductive
reasoning tasks, which is also observed in prior works Sanyal et al. (2022);
Nguyen et al. (2023).

Next, we find that GPT-4 significantly outperforms the humans on complex
reasoning that constitutes $\sim$ 20% data. This usually requires two skills:
understanding multiple relevant information in the premise and combining them
for reasoning. GPT-4 is likely a stronger context processor, especially for
long premises, since it has been trained on long-context data sources OpenAI
(2023). In contrast, cognitive studies Buschman et al. (2011); Cowan (2001)
have shown that an average human brain can retain only four chunks in short-
term memory, thus limiting the long-context processing abilities of humans.

Lastly, we observe that approximately 30% of the data has some missing entity-
grounded or commonsense information while $\sim$ 25% of the data has missing
localized information. To correctly predict entailment in such cases, a system
needs to infer some missing grounded knowledge while not hallucinating
specific localized information not mentioned in the premise. We find that both
humans and models are comparable across the reasoning types R3 and R4, with R4
being more challenging. This shows that both models and humans tend to
hallucinate missing localized information.

##  4 Training LLMs for Entailment Verification

In Section 3.2, we observed that open-sourced LLMs lack performance compared
to humans and close-sourced models such as GPT-4. From Section 3.1, we know
that task-finetuned models perform well on a category when finetuned on data
from the same category. Using this insight, we finetune the Flan-T5-xxl model
on the train splits of datasets from each category, resulting in using ANLI
Nie et al. (2020), RACE Lai et al. (2017), and ECQA Aggarwal et al. (2021) for
finetuning. Please refer to Appendix F.1 for more details on our training
dataset selection criteria. Next, we describe the finetuning approaches and
our key findings.

###  4.1 Finetuning Formulations

This section describes the two finetuning formulations explored in this work.

##### Classification

This is the standard training paradigm where we finetune a Flan-T5-xxl model
using the training data. We follow the same steps as the evaluation setup to
create a prompted input using the prompt format in Box 1 and then define the
cross-entropy loss over the “Yes” and “No” token logits. We refer to this
finetuned model as “Flan-T5-xxl + Class”.

[sections/figures.tex]cotFlowchart

##### Ranking

In this approach, for a given premise and hypothesis pair $(p,h)$, we define a
weaker hypothesis $h^{\prime}$ as a statement such that the premise $p$
supports $h$ more strongly than $h^{\prime}$. Then, for a given triplet
$(p,h,h^{\prime})$, we formulate the ranking task as predicting the hypothesis
that is more supported by the premise. Given the triplet $(p,h,h^{\prime})$,
we define the margin ranking loss as follows:

$\mathcal{L}_{ranking}=\max\\{0,s(p,h)-s(p,h^{\prime})+m\\},$ (2)

where $s(p,h)$ is the entailment score as defined in Equation 1. The key
advantage of this formulation over the classification is that ranking, by
design, learns a softer decision boundary between the two labels. This can
lead to better generalization, especially for contextual QA datasets.
Sometimes, the wrong choice can be relatively less favorable compared to the
best option in QA instead of being incorrect. As discussed in Section 3.2,
this is indicated by the low agreement between human raters. Training to hard-
classify the hypothesis for such options can be avoided by ranking them with
the best hypothesis (corresponding to the right choice), thus learning a
softer classification boundary. We refer to the finetuned model using the
ranking objective as “Flan-T5-xxl + Rank”. Please refer to Appendix F.2 for
more details on the training data collection process for ranking.

###  4.2 Findings

Table LABEL:tab:finetune_results shows the evaluation results on the complete
evaluation set (i.e., we use all the data points instead of 100 samples per
dataset, which was used in Table LABEL:tab:human_results). For our models, we
separately average the results for the datasets already seen in training
(namely, ANLI, RACE, and ECQA) and unseen during training into two columns,
seen and unseen, respectively. First, we observe that all our finetuned models
are consistently better across nine of ten datasets than the baseline
Flan-T5-xxl. Finetuning improves 0.07 macro-F1 on average over Flan-T5-xxl.
This shows that finetuning is overall beneficial in training the model on the
task of entailment verification.

Next, we observe that compared to classification, the ranking formulation is
beneficial for the contextual QA datasets CosQA, SIQA, and DREAM. This
demonstrates that the ranking objective improves contextual QA datasets’
generalization, which is expected. Additionally, our ranking model outperforms
GPT-3.5 and performs comparably to GPT-4, with stronger performance on
contextual QA datasets and weaker performance on NLI datasets. Thus,
Flan-T5-xxl + Rank is a strong open-sourced model for entailment verification
and can be used as an alternative to GPT-4.

[sections/figures.tex]cot

##  5 Application: Filtering CoT Rationales

Recently, Wang et al. (2023) proposed self-consistency (SC), a decoding
technique to improve over chain-of-though (CoT) reasoning Wei et al. (2022) in
LLMs, whereby multiple CoT rationales are sampled for a given input instance
and a majority voting overall predicted labels is considered as the final
prediction. However, generative LLMs can potentially output rationales that
are inconsistent Ye and Durrett (2022), i.e., the rationale does not support
the corresponding model prediction. Such inconsistency can, in turn, degrade
the overall self-consistency results. Evaluating the consistency between an
explanation and the corresponding prediction can be framed as an entailment
verification (EV) task, as described below. Please refer to Appendix G.2 for
examples of consistent and inconsistent CoTs.

##### Approach

As shown in Figure LABEL:fig:cot_application, we can use a verifier as an
intermediate filtering step to filter out the inconsistent rationales before
computing the majority vote. For this, we define the generated CoT rationale
as the premise and use the QA-to-statement model Chen et al. (2021) as defined
in Section 2.1 to convert the question and model’s prediction into a
hypothesis. Next, we calculate the entailment score of all the premise-
hypothesis pairs using a verifier (Equation 1). Finally, we select the top-$k$
rationales for majority voting, discarding the rest. We set $k=5$ for all our
experiments.

##### Findings

In figure LABEL:fig:cot_comparison, we compare the vanilla SC with the
filtering+SC approach described above. Following Wang et al. (2023), we
compute the average performance of these methods across three MCQ datasets for
four different base CoT models (UL2 Tay et al. (2023), Codex-001 Brown et al.
(2020), LaMDA-137B Thoppilan et al. (2022), and ChatGPT444ChatGPT refers to
the gpt-3.5-turbo-0613 model. OpenAI (2022)). Please refer to Appendix G for
details on the datasets and more comparisons with Flan-T5-xxl. We observe that
filtering leads to a consistent performance gain over SC across all CoT base
models. This demonstrates the advantage of the filtering approach. Next, we
find that the improvements are more prominent for weaker base models such as
UL2 than the stronger ones (ChatGPT). For instance, filtering UL2 generated-
rationales can even achieve comparable performance with vanilla SC over
LaMDA-137B. In comparison, the gains for filtering ChatGPT CoTs are $\sim$ 1.7
%. This shows that weaker models are prone to generating inconsistent CoTs and
thus benefit more from this approach. But at the same time, even stronger
models such as ChatGPT can still benefit from consistency checks. Please refer
to Appendix G.2 for examples of filtered CoT rationales.

##  6 Related Works

##### Natural Language Inference

NLI Dagan et al. (2006); Manning and MacCartney (2009) is one of the core NLP
problems in which the relationship between a premise and hypothesis is
classified as either entailment, contradiction, or neutral. Prior works have
mainly trained LLMs and evaluated them on standard NLI datasets Bowman et al.
(2015); Williams et al. (2018); Wang et al. (2019); Nie et al. (2020, 2019);
Liu et al. (2022). Another line of work Mishra et al. (2020); Chen et al.
(2021) has used question-answer-to-NLI conversion Demszky et al. (2018) to
transform QA datasets into NLI format and solve them. In fact verification
literature Bekoulis et al. (2021); Thorne et al. (2018), retrieved pieces of
evidence have been used to verify the claim using an entailment verifier Nie
et al. (2019); Guan et al. (2023). Recently, NLI models have been used to
verify the entailment of model-generated explanations Tafjord et al. (2022);
Jung et al. (2022); Mitchell et al. (2022). In this work, we curate a diverse
NLI benchmark for evaluating LLMs and humans by using datasets from all the
above NLI applications.

##### Reasoning in LLMs

With the advent of general-purpose LLMs Brown et al. (2020); Chung et al.
(2022); OpenAI (2023), many prompting strategies have been proposed to
generate a natural language reasoning along with the model’s prediction Wei et
al. (2022); Zhou et al. (2023); Yao et al. (2023); Huang and Chang (2023).
Recently, Ye and Durrett (2022) have found that such generations can sometimes
be unreliable due to non-factual and inconsistent reasoning, while Huang et
al. (2023) have argued that LLMs struggle to self-correct such issues without
external feedback.

Prior works have addressed this limitation by oversampling reasoning chains
and marginalizing Wang et al. (2023), using the LLMs itself to recheck their
reasoning Madaan et al. (2023); Miao et al. (2023), leveraging external
knowledge source to verify factuality Zhao et al. (2023), using deterministic
solvers to improve faithfulness Lyu et al. (2023), decomposing the reasoning
steps into smaller steps Ling et al. (2023), etc. While the progress is
impressive, some of these are either specialized approaches for math-specific
datasets or heavily rely on close-sourced LLMs (GPT-3.5, GPT-4, etc.) for
verification. In contrast, here we focus on natural language datasets and
develop an open-sourced model for the verification task.

##  7 Conclusion

We studied the EV problem in the context of LLMs. Specifically, we sourced
datasets across three different categories (NLI, contextual QA, and
rationales) and analyzed human and LLM performance on the benchmark. We found
that LLMs are better than humans in complex reasoning, while humans are more
consistent on simpler reasoning tasks. We also explored different training
objectives to finetune open-sourced LLMs for EV. Our finetuned model
outperforms GPT-3.5, baseline Flan-T5-xxl, and is comparable to GPT-4.
Finally, we applied the EV model to filter out inconsistent model-generated
CoTs in self-consistency decoding, achieving improvements over the baseline
self-consistency approach.

## Limitations

Even though our work demonstrates exciting results on entailment verification
tasks by finetuning LLMs, several limitations can be potentially improved.

##### Data Processing

Our strategy to convert a QA pair into a statement using the QA-to-statement
converter model can have errors that can cascade both in the evaluation
dataset and our fine-tuned models. Better strategies for this pipeline would
help with data quality.

##### Finetuning

We only tried encoder-decoder-based models for finetuning. However, other
models with different architectures (like decoder-only) can also be considered
in the future. For computing the entailment score in Equation 1, we only
considered the probability of “Yes” and “No” tokens in the entire vocabulary.
Other alternative expressions like “YES”/“NO”, “True”/“False”, etc., can also
be considered to make the score more robust. Finally, our training objective
outputs entailment scores instead of directly generating answers. Answer
generation as a training objective can be more robust since it is a stricter
objective than our scoring technique.

##### Potential Risks

We using existing datasets, with some post-processing, to train our models.
Thus, any issues in the existing dataset in terms of bias, toxicity, etc., can
potentially affect our model. Also, since we use a pretrained checkpoint, we
also inherit any existing biases in the baseline model. The model is trained
to always output scores, even if the data is well outside the training
distribution. This is an existing issue with most NLP models and can be
mitigated by additional checks for domain shift.

## Acknowledgments

This research is supported in part by the Office of the Director of National
Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA),
via the HIATUS Program contract #2022-22072200006, the DARPA MCS program under
Contract No. N660011924033, the Defense Advanced Research Projects Agency with
award W911NF-19-20271, NSF IIS 2048211, and gift awards from Google and
Amazon. The views and conclusions contained herein are those of the authors.
They should not be interpreted as necessarily representing the official
policies, either expressed or implied, of ODNI, IARPA, or the U.S. Government.
We would like to thank all the USC INK research lab collaborators for their
constructive feedback on the work.

## References

  * Aggarwal et al. (2021) Shourya Aggarwal, Divyanshu Mandowara, Vishwajeet Agrawal, Dinesh Khandelwal, Parag Singla, and Dinesh Garg. 2021.  Explanations for CommonsenseQA: New Dataset and Models.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 3050–3065, Online. Association for Computational Linguistics.
  * Bekoulis et al. (2021) Giannis Bekoulis, Christina Papagiannopoulou, and Nikos Deligiannis. 2021.  A review on fact extraction and verification.  _ACM Comput. Surv._ , 55(1).
  * Bowman et al. (2015) Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015\.  A large annotated corpus for learning natural language inference.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 632–642, Lisbon, Portugal. Association for Computational Linguistics.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems_ , volume 33.
  * Buschman et al. (2011) Timothy J. Buschman, Markus Siegel, Jefferson E. Roy, and Earl K. Miller. 2011.  Neural substrates of cognitive capacity limitations.  _Proceedings of the National Academy of Sciences_ , 108(27):11252–11255.
  * Chen et al. (2021) Jifan Chen, Eunsol Choi, and Greg Durrett. 2021.  Can NLI models verify QA systems’ predictions? In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 3841–3854, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022.  Scaling instruction-finetuned language models.
  * Clark et al. (2019) Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019.  BoolQ: Exploring the surprising difficulty of natural yes/no questions.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 2924–2936, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Clark et al. (2018) Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018.  Think you have solved question answering? try arc, the ai2 reasoning challenge.
  * Cowan (2001) Nelson Cowan. 2001.  The magical number 4 in short-term memory: A reconsideration of mental storage capacity.  _Behavioral and Brain Sciences_ , 24(1):87–114.
  * Crowston (2012) Kevin Crowston. 2012.  Amazon mechanical turk: A research tool for organizations and information systems scholars.  In _Shaping the Future of ICT Research. Methods and Approaches_ , pages 210–221, Berlin, Heidelberg. Springer Berlin Heidelberg.
  * Dagan et al. (2006) Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006.  The pascal recognising textual entailment challenge.  In _Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Tectual Entailment_ , pages 177–190, Berlin, Heidelberg. Springer Berlin Heidelberg.
  * Dalvi et al. (2021) Bhavana Dalvi, Peter Jansen, Oyvind Tafjord, Zhengnan Xie, Hannah Smith, Leighanna Pipatanangkura, and Peter Clark. 2021.  Explaining answers with entailment trees.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 7358–7370, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Demszky et al. (2018) Dorottya Demszky, Kelvin Guu, and Percy Liang. 2018.  Transforming question answering datasets into natural language inference datasets.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Fleiss (1971) Joseph L Fleiss. 1971.  Measuring nominal scale agreement among many raters.  _Psychological bulletin_ , 76(5):378.
  * Garnham (1989) Alan Garnham. 1989.  Inference in language understanding: What, when, why and how.  In RAINER DIETRICH and CARL F. GRAUMANN, editors, _Language Processing in Social Context_ , volume 54 of _North-Holland Linguistic Series: Linguistic Variations_ , pages 153–172. Elsevier.
  * Geva et al. (2021) Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021.  Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies.  _Transactions of the Association for Computational Linguistics (TACL)_.
  * Guan et al. (2023) Jian Guan, Jesse Dodge, David Wadden, Minlie Huang, and Hao Peng. 2023.  Language models hallucinate, but may excel at fact verification.
  * Gururangan et al. (2018) Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018.  Annotation artifacts in natural language inference data.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 107–112, New Orleans, Louisiana. Association for Computational Linguistics.
  * Huang and Chang (2023) Jie Huang and Kevin Chen-Chuan Chang. 2023.  Towards reasoning in large language models: A survey.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 1049–1065, Toronto, Canada. Association for Computational Linguistics.
  * Huang et al. (2023) Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. 2023.  Large language models cannot self-correct reasoning yet.
  * Huang et al. (2019) Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019.  Cosmos QA: Machine reading comprehension with contextual commonsense reasoning.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2391–2401, Hong Kong, China. Association for Computational Linguistics.
  * Jung et al. (2022) Jaehun Jung, Lianhui Qin, Sean Welleck, Faeze Brahman, Chandra Bhagavatula, Ronan Le Bras, and Yejin Choi. 2022.  Maieutic prompting: Logically consistent reasoning with recursive explanations.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 1266–1279, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Lai et al. (2017) Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017.  RACE: Large-scale ReAding comprehension dataset from examinations.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 785–794, Copenhagen, Denmark. Association for Computational Linguistics.
  * Ling et al. (2023) Zhan Ling, Yunhao Fang, Xuanlin Li, Zhiao Huang, Mingu Lee, Roland Memisevic, and Hao Su. 2023.  Deductive verification of chain-of-thought reasoning.
  * Liu et al. (2022) Alisa Liu, Swabha Swayamdipta, Noah A. Smith, and Yejin Choi. 2022.  WANLI: Worker and AI collaboration for natural language inference dataset creation.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 6826–6847, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  RoBERTa: a robustly optimized BERT pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Lyu et al. (2023) Qing Lyu, Shreya Havaldar, Adam Stein, Li Zhang, Delip Rao, Eric Wong, Marianna Apidianaki, and Chris Callison-Burch. 2023.  Faithful chain-of-thought reasoning.
  * Madaan et al. (2023) Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Sean Welleck, Bodhisattwa Prasad Majumder, Shashank Gupta, Amir Yazdanbakhsh, and Peter Clark. 2023.  Self-refine: Iterative refinement with self-feedback.
  * Manning and MacCartney (2009) Christopher D. Manning and Bill MacCartney. 2009.  _Natural language inference_.  Stanford University.
  * McCoy et al. (2019) Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019.  Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3428–3448, Florence, Italy. Association for Computational Linguistics.
  * Miao et al. (2023) Ning Miao, Yee Whye Teh, and Tom Rainforth. 2023.  Selfcheck: Using llms to zero-shot check their own step-by-step reasoning.
  * Minaee et al. (2021) Shervin Minaee, Nal Kalchbrenner, Erik Cambria, Narjes Nikzad, Meysam Chenaghlu, and Jianfeng Gao. 2021.  Deep learning–based text classification: A comprehensive review.  _ACM Comput. Surv._ , 54(3).
  * Mishra et al. (2020) Anshuman Mishra, Dhruvesh Patel, Aparna Vijayakumar, Xiang Li, Pavan Kapanipathi, and Kartik Talamadupula. 2020.  Reading comprehension as natural language inference:a semantic analysis.  In _Proceedings of the Ninth Joint Conference on Lexical and Computational Semantics_ , pages 12–19, Barcelona, Spain (Online). Association for Computational Linguistics.
  * Mitchell et al. (2022) Eric Mitchell, Joseph Noh, Siyan Li, Will Armstrong, Ananth Agarwal, Patrick Liu, Chelsea Finn, and Christopher Manning. 2022.  Enhancing self-consistency and performance of pre-trained language models through natural language inference.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 1754–1768, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Nguyen et al. (2023) Ha Thanh Nguyen, Randy Goebel, Francesca Toni, Kostas Stathis, and Ken Satoh. 2023\.  A negation detection assessment of gpts: analysis with the xnot360 dataset.
  * Nie et al. (2019) Yixin Nie, Haonan Chen, and Mohit Bansal. 2019.  Combining fact extraction and verification with neural semantic matching networks.  In _Association for the Advancement of Artificial Intelligence (AAAI)_.
  * Nie et al. (2020) Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020.  Adversarial nli: A new benchmark for natural language understanding.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics.
  * OpenAI (2022) OpenAI. 2022.  Introducing chatgpt.  https://openai.com/blog/chatgpt.
  * OpenAI (2023) OpenAI. 2023.  Gpt-4 technical report.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21(140):1–67.
  * Sanyal et al. (2022) Soumya Sanyal, Zeyi Liao, and Xiang Ren. 2022.  RobustLR: A diagnostic benchmark for evaluating logical robustness of deductive reasoners.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 9614–9631, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Sap et al. (2019) Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. 2019.  SocialIQA: Commonsense reasoning about social interactions.  In _EMNLP_.
  * Sun et al. (2019) Kai Sun, Dian Yu, Jianshu Chen, Dong Yu, Yejin Choi, and Claire Cardie. 2019.  DREAM: A challenge data set and models for dialogue-based reading comprehension.  _Transactions of the Association for Computational Linguistics_ , 7:217–231.
  * Tafjord et al. (2022) Oyvind Tafjord, Bhavana Dalvi Mishra, and Peter Clark. 2022.  Entailer: Answering questions with faithful and truthful chains of reasoning.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 2078–2093, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Talmor et al. (2019) Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019.  CommonsenseQA: A question answering challenge targeting commonsense knowledge.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4149–4158, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Tay et al. (2023) Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier Garcia, Jason Wei, Xuezhi Wang, Hyung Won Chung, Siamak Shakeri, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Denny Zhou, Neil Houlsby, and Donald Metzler. 2023.  Ul2: Unifying language learning paradigms.
  * Thoppilan et al. (2022) Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Vincent Zhao, Yanqi Zhou, Chung-Ching Chang, Igor Krivokon, Will Rusch, Marc Pickett, Pranesh Srinivasan, Laichee Man, Kathleen Meier-Hellstern, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Soraker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera-Arcas, Claire Cui, Marian Croak, Ed Chi, and Quoc Le. 2022.  Lamda: Language models for dialog applications.
  * Thorne et al. (2018) James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018\.  FEVER: a large-scale dataset for fact extraction and VERification.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 809–819, New Orleans, Louisiana. Association for Computational Linguistics.
  * Wang et al. (2019) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In the Proceedings of ICLR.
  * Wang et al. (2023) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023.  Self-consistency improves chain of thought reasoning in language models.  In _The Eleventh International Conference on Learning Representations_.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. 2022.  Chain of thought prompting elicits reasoning in large language models.  In _Advances in Neural Information Processing Systems_.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1112–1122. Association for Computational Linguistics.
  * Wolf et al. (2020a) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020a.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Wolf et al. (2020b) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020b.  Huggingface’s transformers: State-of-the-art natural language processing.
  * Yao et al. (2023) Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. 2023.  Tree of thoughts: Deliberate problem solving with large language models.
  * Ye and Durrett (2022) Xi Ye and Greg Durrett. 2022.  The unreliability of explanations in few-shot prompting for textual reasoning.  In _Advances in Neural Information Processing Systems_.
  * Zhao et al. (2023) Ruochen Zhao, Xingxuan Li, Shafiq Joty, Chengwei Qin, and Lidong Bing. 2023.  Verify-and-edit: A knowledge-enhanced chain-of-thought framework.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 5823–5840, Toronto, Canada. Association for Computational Linguistics.
  * Zhou et al. (2023) Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V Le, and Ed H. Chi. 2023.  Least-to-most prompting enables complex reasoning in large language models.  In _The Eleventh International Conference on Learning Representations_.

##  Appendix A Evaluation Datasets

In this section, we describe the datasets used in our evaluation. We mention
some important challenges that make these datasets useful for benchmarking the
entailment verification task. Please refer to Table
LABEL:tab:dataset_statistics for these datasets’ train/dev/test statistics.

[sections/tables.tex]dataStatistic

##### Natural Language Inference datasets

NLI is an obvious choice of data source as it is a more general case of the
entailment verification problem. While converting an NLI dataset for our task,
we merge the neutral and contradict labels to the not support label. We use
the following NLI datasets in our benchmark:

  * •

WaNLI Liu et al. (2022): This is a new NLI dataset built using worker and AI
collaboration. This challenging dataset improves over the existing NLI dataset
MultiNLI Williams et al. (2018). The test split is used when doing the
evaluation.

  * •

FEVER Nie et al. (2019): This is a modification of the original FEVER dataset
Thorne et al. (2018) in which the claim is paired with textual evidence from
Wikipedia to convert it into an NLI format dataset. This pairing uses existing
state-of-the-art evidence extraction systems to find relevant evidence for
each claim. Premises in this dataset typically contain multiple sentences,
which is one of our focus areas. As the test split is unavailable, we report
results on the dev split for evaluation.

  * •

ANLI Nie et al. (2020): This is a large-scale NLI dataset that was collected
using an adversarial human-and-model-in-the-loop procedure. Like FEVER, this
dataset tests factual knowledge, and the premises typically contain multiple
sentences. During evaluation, the test split is considered.

[sections/figures.tex]contextualQA

##### Contextual QA datasets

Next, we consider QA datasets where the task is to answer a question based on
a given context and some options. We use an off-the-shelf QA-to-statement
converter model Chen et al. (2021) to generate a hypothesis statement for each
question option pair. Then, the hypothesis corresponding to the correct choice
is marked as “support”, while the rest are marked as “not support” to create
the entailment verification dataset. We depict this process in Figure
LABEL:fig:contextual_qa. The green box is the valid hypothesis and the red
ones are the invalid hypothesis corresponding to the given context. Overall,
we include the following datasets from this category:

  * •

Cosmos QA (CosQA) Huang et al. (2019): This dataset contains multiple-choice
questions (MCQs) that require an understanding of commonsense-based reading
comprehension to answer a question. The key challenge in this dataset is
understanding people’s everyday narratives described in the context that can
have some missing commonsense knowledge that needs to be inferred implicitly.
Since the test split is missing, we evaluate models on the dev split instead.

  * •

SocialIQA (SIQA) Sap et al. (2019): Similar to CosQA, this is another MCQ
benchmark for commonsense reasoning about social situations that probes
emotional and social intelligence in a variety of everyday situations. This
dataset has more nuanced commonsense knowledge requirements, which makes it a
challenging dataset for our task. Similarly, results on the dev split are
reported, given that the test is missing.

  * •

DREAM Sun et al. (2019): This is a dialogue-based reading comprehension MCQ
dataset that focuses on multi-turn dialogue understanding. Here, the unique
challenge is inferring the events discussed across long, multi-turn dialogues.
During evaluation, we use the test split as it is available.

  * •

BoolQ Clark et al. (2019): This is a True/False QA dataset consisting of
aggregated queries to the Google search engine. Questions in this dataset
require complex and difficult entailment-like inference to solve, making it a
good set for evaluation. The test split is lacking for this dataset, and we
can only report results on the dev split.

  * •

RACE Lai et al. (2017): The reading comprehension dataset from examinations
(RACE) is one of the most popular machine reading comprehension datasets
containing questions from English exams for middle and high school students.
These questions are designed by domain experts for testing specific human
reading skills, thus making it a good evaluation set for our task. We report
evaluating results on test split for this dataset.

##### Rationale datasets

Lastly, we consider data sources where human-annotated explanations are
available that justify the original hypothesis (or the correct option, in the
case of QA datasets). In this case, we use the rationales as the premise. We
use the following datasets:

  * •

Entailer Tafjord et al. (2022): This dataset contains entailment-style
statements and corresponding rationales obtained from EntailmentBank dataset
Dalvi et al. (2021) and crowdsourcing. The dataset mainly contains science
domain statements and tests simple deductive reasoning skills whereby
sentences from the premise must be combined to either support or refute the
hypothesis. The test split is also missing for this dataset, and we can only
evaluate it on the dev split.

  * •

ECQA Aggarwal et al. (2021): This is a human-annotated explanation dataset for
CommonsenseQA Talmor et al. (2019). We only use the explanations for the
correct choice as the explanations for the incorrect choices are often
trivial. It is a complete dataset and by convention, we use the test split for
evaluation.

##  Appendix B Details on LLM Evaluation

We evaluate two types of LLMs on the task of entailment verification, as
categorized below:

###  B.1 Task-Finetuned models

In this category, the models considered are already finetuned for either NLI
or the exact entailment verification task itself. We evaluate two models in
this category.

##### RoBERTa

Nie et al. (2020); Liu et al. (2019) This is a strong pre-trained RoBERTa-
Large model with the corresponding model card on
HuggingFace555https://huggingface.co/models Wolf et al. (2020b) called
“ynie/roberta-large-snli_mnli_fever_anli_R1_R2_R3-nli”. It is a specifically
pre-trained RoBERTa-Large for NLI task and includes the combination of SNLI
Bowman et al. (2015), MNLI Williams et al. (2018), FEVER Nie et al. (2019) and
ANLI Nie et al. (2020) datasets as the training data. Hence, this incurs a
potential data leakage problem, as we also test it on FEVER and ANLI. To some
extent, it explains the strong performance of RoBERTa on NLI datasets in Table
LABEL:tab:human_results.

The model is used as a classifier in evaluation, and three classes are
available. Class 0 corresponds to “entail”, class 1 corresponds to “neutral”
and class 2 means “not entail”. In our experiment setting, we only regard
class 0 as the “Yes” label and combine the remaining two classes to be the
“No” label.

##### Entailer

Tafjord et al. (2022) Entailer is a T5-based model Raffel et al. (2020)
trained to answer hypotheses by building proof trees containing chains of
reasoning. It can either generate valid premises for a given hypothesis or
predict a score for a given premise and hypothesis. We evaluate Entailer-11B
(model name “allenai/entailer-11b” on HuggingFace) in our experiments.
Similarly, Entailer dataset Dalvi et al. (2021) is in the training set, making
this model very competitive when evaluating the same dataset.

We strictly follow the official implementation of the model to acquire class
labels.666https://github.com/allenai/entailment_bank/blob/main/entailer.md The
“entailment_verifier” is called to decide if the hypothesis can be implied
from the premise. If the answer is “True”, then the class label will be “Yes”
and vice versa.

[sections/tables.tex]promptSensitivity

[sections/tables.tex]fewshotResults

###  B.2 Instruction-Finetuned models

These are the more recent “general-purpose” language models trained on a
collection of NLP tasks described using instructions, leading to
generalization abilities to solve unseen tasks described using new
instructions. The models included in our evaluation from this category are
described below.

##### Flan-T5-xxl

Chung et al. (2022) It is instruction-tuned from T5 Raffel et al. (2020) on
1.8K+ tasks. We adopt a publicly available version on HuggingFace Wolf et al.
(2020a) with model card name “google/flan-t5-xxl”. Flan-T5-xxl is also exposed
to data leakage issues. BoolQ Clark et al. (2019), ECQA Aggarwal et al.
(2021), and ANLI Nie et al. (2020) have appeared in its training data.
However, this is not a serious problem in the finetuning stage because we
transform original datasets into entailment verification format before using
them for model finetuning.

We extract labels from the model by focusing on the output probabilities of
two words, “Yes” and “No”. After applying the softmax function to those two
probabilities, we finalize the label as the word with a probability larger
than a given threshold, as described in Equation 1.

##### GPT-3.5

Brown et al. (2020) It is a general-purpose autoregressive decoder-only LLM
accessible via the OpenAI Completions
API.777https://platform.openai.com/docs/api-reference/completions We utilize
“text-davinci-003” in OpenAI’s API for evaluation, and the label determination
procedure is similar to the one in Flan-T5-xxl.

##### GPT-4

OpenAI (2023) It is the latest generative model published by OpenAI, which is
optimized for creativity and long context inputs. It is accessible via the
OpenAI Chat API.888https://platform.openai.com/docs/api-reference/chat We
adopt plain “gpt-4” in OpenAI’s API for our experiments. Unlike other models,
the output probabilities are not accessible. Thus, we constrain the model to
predict a single token as its generated prediction. In the ideal case, we can
directly use the output text as the label if “Yes” or “No” is produced. If
some other token is generated, we choose one of the labels at random. We note
that this occurrence is rare in our GPT-4 evaluation runs.

####  B.2.1 Prompt Robustness Evaluation

To assess the robustness of the model in section 2.3, we design different
prompt formats but hold the order of premise and hypothesis in the prompt
unchanged. Table LABEL:tab:prompt_sensitivity presents all prompts we tested
with Flan-T5-xxl and their corresponding averaged results across datasets. The
results suggest that the Flan-T5-xxl model is robust to the variation in the
prompt and yields relatively consistent results. This characteristic is
maintained in Flan-T5-xxl + Class and Flan-T5-xxl + Rank as well since they
are developed over based on Flan-T5-xxl.

####  B.2.2 Few-Shot Evaluation

Few-shot is an effective and promising strategy when testing the performance
of a model Brown et al. (2020). We also include this analysis by randomly
picking two examples from Entailer as demonstrations and incorporating them
into the prompt. We test Flan-T5-xxl, Flan-T5-xxl + Rank, and GPT-4 in this
setting and represent results in Table LABEL:tab:fewshot_results. The few-shot
setting yields promising improvement for Flan-T5-xxl, substantiating that the
few-shot is a beneficial approach to teaching the prompt to the model.
However, it is not as helpful as our finetuning strategies, which give even
better performance. On the other hand, few-shot does not bring significant
gains for Flan-T5-xxl + Rank, suggesting that finetuning has already helped
the model have a comprehensive understanding of the prompt, and extra
demonstrations are unnecessary. As for the GPT-4, simply using examples from
Entailer and applying the same prompt for all datasets seem detrimental.

[sections/tables.tex]majorityResults

##  Appendix C Majority Prediction and Label Imbalance

In Table LABEL:tab:majority_results, we show the performance of an oracle
model that predicts the most frequent label in a dataset. For a label-balanced
dataset, the macro-F1 score of such a majority prediction model would be 0.67
(precision 0.5 and recall 1.0). The datasets in the evaluation set have some
label imbalance, as evidenced by the lower majority label prediction scores.
Since we convert existing 3-class NLI and multi-choice QA datasets into our
binary classification task format, it inherently has more not support labels.
We have more support instances for the rationale datasets since the dataset
creators usually only annotate the rationale for the right choice.
Specifically, the ECQA dataset only has positive instances, leading to a 1.0
macro-F1 score for majority prediction (support label). Since it has all
support labels, any model predicting even a single non support label gets
penalized severely, as is seen in ECQA results. Because of this label
imbalance in the datasets, we report the macro-F1 scores instead of accuracy
or micro-F1 metrics.

##  Appendix D Debatable Cases in Contextual QA

We include two examples of SIQA dataset in Table LABEL:tab:contextual_examples
to illustrate the debatable hypotheses in Contextual QA datasets. These
examples demonstrate that some less probable hypotheses can also be considered
plausible in certain situations. In the first example, option “run screaming”
best describes the audience’s reactions after listening to a scary ghost
story. However, there are also situations where people are so terrified and
start to cry, making the option “cry” somehow debatable. Similarly, in the
second example, it is common sense that people will express gratefulness after
someone else builds a house for them. But it is reasonable to say that Quinn
is intrigued by the house after seeing the amazing architecture. In both
cases, apart from the most adequate one, there are still options leading to
some partially supported hypotheses, which explains the low human annotation
agreement in these cases.

[sections/tables.tex]contextualExamples

[sections/figures.tex]reasoningAnalysis1

##  Appendix E Human Evaluation and Analysis

We adopted Amazon Mechanical Turk (MTurk) Crowston (2012) for data collection.
Two annotation formats (Figure LABEL:fig:eval_annotate and Figure
LABEL:fig:reason_annotate) were devised for the human evaluation task and
reasoning type analysis task, respectively. During the annotations, each
annotator was compensated according to $15/hour per the U.S. minimum wage.

###  E.1 Human Evaluation Details

In each HIT, the annotator was presented with a format exactly like Figure
LABEL:fig:eval_annotate, including detailed task descriptions and label
explanations. Annotators were expected to read the premise and claim first,
then determine the supportiveness of the claim based on the premise and choose
the corresponding label. Initially, we only provided three labels — “support”,
“irrelevant” and “contradict”. But later, we realized that annotators could
not explicitly identify labels for some ambiguous instances where the premise
only partially supported or contradicted the claim. Hence, we introduced two
weak labels (“partially support” and “partially contradict”) to remedy this
issue. When collating results for analysis, we internally combined “support”
and “partially support” to be “support”, and the rest to be “not support”,
aligning to the standard entailment verification (EV) setup. Each instance was
annotated by 3 MTurk annotators, and a majority verdict determined the label.
The Fleiss’s kappa score Fleiss (1971) we got was 0.6, indicating a moderate
level of agreement among annotators.

[sections/figures.tex]evalAnnotate

###  E.2 Reasoning Type Annotation Details

We annotate the reasoning type of the 100 sampled instances for each dataset
with absolute macro-F1 difference $>0.1$ in Table LABEL:tab:human_results. We
make this choice intending to attribute the largely misaligned datasets
(namely, ANLI, CosQA, SIQA, and Entailer) since some random noise in the
annotation and sampling process can potentially also cause some misalignment.

In every HIT, we used Figure LABEL:fig:reason_annotate as the reasoning type
annotation format. The instructions and label explanations were explicitly
stated at the beginning of the HIT. For task 1, after reading the premise and
the claim, annotators had to decide whether the supportiveness of the claim
could be decided by only referring to the information in the premise. If yes,
annotators needed to choose the corresponding difficulty level of reasoning
about the supportiveness of the claim in task 2. Otherwise, the type of
missing information had to be decided in task 1, and task 2 did not apply to
these cases. We aggregated answers from the two tasks and categorized them
into four types. Both R1 and R2 were types where the premise contained all
necessary information. The difference was that the difficulty level of
reasoning for R1 was Easy while for R2 was Moderate. We combined Missing
Entity-grounded Information and Missing Commonsense Information to be R3.
Finally, the Missing Localized Information label corresponded to R4.

Given that some strong NLP background knowledge was required to understand
task descriptions and label explanations, we recruited Computer Science
graduate students instead of the general mTurk workers to finish this
annotation. Every instance was assigned to 2 students, and the majority vote
determined the label. The Fleiss’s kappa score for this job was 0.62, showing
a substantial inter-annotator agreement.

[sections/figures.tex]reasonAnnotate

###  E.3 Reasoning Type Examples

Table LABEL:tab:reasoning_examples incorporates three examples from each
reasoning type, providing more insight into those types.

In the third example of R1, the first sentence in the premise states that
“iron oxide” comes from “oxygen” and “rust”. The second sentence shows those
two substances are “gases” at room temperature. Therefore, combining them will
be sufficient to entail the hypothesis.

However, unlike R1, the third example of R2 requires three steps of reasoning
that “De Baandert was a multi-use stadium.”, “It was mostly used for football
matches.”, and “The stadium was able to hold 22,000 people.”. The hypothesis
can be disproved with those steps because “22,000 people” is just the maximum
capacity.

As for the first example of R3, some missing commonsense information like “it
is not wise to give more money to a person who keeps playing in a detrimental
situation.” should be combined with the premise to disprove the hypothesis.

In the first example of R4, the next movement of “Addison” is the missing
information specific to the context depicted by the premise. The hypothesis
can not be directly disproved without that piece of information.

###  E.4 Reasoning Type Analysis

We depict the aggregated results of the reasoning type annotation for each
dataset in Figure LABEL:fig:reasoning_analysis1. Here, the first row shows the
frequency of each reasoning type in that dataset, and the corresponding plot
in the second row compares the human and GPT-4 macro-F1 scores. We fade out
the columns with data percentages less than 5% because such low frequency
might not lead to conclusive observations.

We note that type R1 is most prominent in Entailer dataset. Here, we observe
that humans are significantly better than models. This shows that humans are
usually more consistent with simple deductive reasoning. Similar findings
about consistency in human deductive reasoning skills have been reported in
prior works Sanyal et al. (2022); Nguyen et al. (2023).

Reasoning type R2, which requires more complex reasoning, is dominant in ANLI
and CosQA. For this type, we find that models are superior to humans. Complex
reasoning requires two skills: understanding multiple relevant information in
the premise and then using them for reasoning. We hypothesize that models are
stronger context processors than humans because they have been trained on
long-context data OpenAI (2023).

The reasoning type R3 is present in ANLI, CosQA, and SIQA. From Table
LABEL:tab:dataset_comparisons, we know that ANLI mostly require entity-
grounded knowledge, whereas CosQA and SIQA specifically test commonsense
knowledge. Here, we find that humans are stronger than models in commonsense
knowledge (CosQA and SIQA), whereas models are better in ANLI that requires
entity-grounded knowledge. This shows that humans can infer missing
social/commonsense knowledge more easily since these are inherently known to
humans. In contrast, models can retrieve the entity-grounded knowledge stored
in their parameters more efficiently.

Lastly, we find that the reasoning type R4, indicating missing localized
knowledge, is also prominent in ANLI, CosQA, and SIQA. Here, we find that the
trends are a bit mixed. We find that for SIQA, humans are better at
recognizing missing localized knowledge, but in CosQA, models outperform
humans. This is likely because SIQA typically contains short contexts based on
everyday social situations that are easier for humans to understand. In
contrast, CosQA has longer contexts with rarer situations requiring more
complex understanding.

Overall, we conclude that GPT-4 outperforms humans in complex deductive
reasoning and situations involving entity-grounded knowledge, whereas humans
are more consistent at simple reasoning and situations requiring commonsense
knowledge.

##  Appendix F Finetuning LLMs

In this section, we describe more details about the training dataset used for
finetuning, our negative data collection strategy that was used in the ranking
formulation, and other finetuning details.

###  F.1 Training Dataset Selection

To train the Flan-T5-xxl model, we create a training dataset using
representative datasets from each category. We pick the ANLI, RACE, and ECQA
datasets to represent NLI, contextual QA, and rationale categories,
respectively. We select the datasets with diverse entailment challenges and
aim to maximize the total training data. We note that the amount of training
data is quite low for the rationale category. Thus, we also include the
StrategyQA Geva et al. (2021) dataset in the training set to alleviate this.
Similar to BoolQ, StrategyQA has a Yes/No type of questions and their
corresponding explanations. We convert the question-answer pair into a
hypothesis using the QA-to-statement converter Chen et al. (2021) as described
in Section 2.1.

For a given premise and a valid hypothesis, generate five alternate hypotheses
contradicted by the premise. Try to avoid using the negation words such as
“not”, “never”, etc. The output should be numbered from 1 to 5. Premise:
{premise} Hypothesis: {hypothesis}

List of Boxes 2  Prompt format for generating alternate negative hypothesis
for a given premise-hypothesis pair. Please refer to Section F.2 for details.

[sections/tables.tex]cotResults

###  F.2 Negative Data Collection for Ranking

In the ranking formulation, for a given premise and hypothesis pair $(p,h)$,
we need to find some weaker hypothesis $h^{\prime}$ to use the ranking loss
defined by Equation 2. We collect such weaker hypotheses in two ways and then
combine them to form the training data. The two techniques are described
below:

  * •

Using incorrect options: The contextual QA category has naturally occurring
negative data. For a given question and choices, we pair the hypothesis
corresponding to the correct option with all other hypotheses corresponding to
the wrong options to create the ranked data.

  * •

GPT-3.5 prompting: The other way we generate negative data is by prompting
GPT-3.5. Specifically, we use the prompt format shown in Box 2 to generate
alternate hypotheses contradicted by the original premise. We only select
premise and hypothesis pairs that originally have support label. GPT-3.5
generated hypotheses are then considered negative samples and paired with the
original hypothesis. We repeat this for all the training datasets (ANLI, RACE,
ECQA, and StrategyQA).

###  F.3 Hyperparameters and other details

During training, we select the learning rate from the set
$\\{7e^{-5},1e^{-4},2e^{-4}\\}$, per GPU batch size from the set $\\{6,8\\}$,
margin $m$ in Equation 2 from the set $\\{0.2,0.3,0.5\\}$, and warmup ratio
$0.1$. The model is trained for 1400 steps on a cluster of 8 A6000 GPUs. We
evaluate the model every 200 steps and save the checkpoint if the model shows
improvements on a held-out development set.

##  Appendix G Chain-of-Thought Filtering

We study three variants of CoT Filtering as mentioned below:

  * •

$\mathcal{B}+\textrm{SC}$: This is the self-consistency baseline. Here,
$\mathcal{B}$ is the base model used to sample CoTs. We sample 40 CoTs for
each instance before computing the majority predicted label.

  * •

$\mathcal{B}+\textrm{{Flan-T5-xxl}{} + SC}$: In this, we use a pre-trained
Flan-T5-xxl for filtering out the inconsistent rationales before the majority
voting. We keep the top-5 rationales after scoring them using Flan-T5-xxl.

  * •

$\mathcal{B}+\textrm{{Flan-T5-xxl + {Rank}}{} + SC}$: This is the same as
above, but instead, we use our ranking-finetuned Flan-T5-xxl model for
filtering.

Following Wang et al. (2023), we use four different base CoT model: UL2 Tay et
al. (2023), Codex-001 Brown et al. (2020), LaMDA-137B Thoppilan et al. (2022),
and ChatGPT OpenAI (2022). Further, we compute the CoTs and analyze the
performance of the above methods for three multi-choice QA datasets, namely,
CommonsenseQA Talmor et al. (2019) and AI2 Reasoning Challenge Clark et al.
(2018) (easy (ARC-e) and challenge (ARC-c) variants). Please refer to Wang et
al. (2023); Wei et al. (2022) and the associated
code999https://openreview.net/attachment?id=1PL1NIMMrw&name=supplementary_material
for details on the CoT prompt formats. The results are shown in Table
LABEL:tab:cot_results. We note a consistent improvement between the three
variants, with Flan-T5-xxl + Rank model performing the best. This demonstrates
the advantage of our entailment finetuning approach. Please refer to Section 5
for more findings.

[sections/figures.tex]k-selection

###  G.1 Ablation of Top-k Filtering

We examine our top-$k$ selecting strategy by choosing different values of $k$
and compare their results. We pick up $k$ from $\\{3,5,10,20,30\\}$ and
present the corresponding aggregated results in Figure LABEL:fig:k-selection.
The general trend is that, as $k$ increases, the model performance increases
first and then starts to decrease, reaching the peak at $k=5$. When $k$ is
small, there are only a limited number of CoT rationals, and the majority
voting process is vulnerable to potential outliers. On the other hand, if k is
too large, then many noisy results are included, leading to poorer
performance.

###  G.2 Examples of Filtered CoTs

Table LABEL:tab:filter_examples presents three CoT reasoning examples, each
including two outputs that are kept and three that are filtered out by
ranking. According to the table, outputs supported by strong rationales are
ranked highly and kept. On the other hand, if the rationale is irrelevant to
the prediction (like rationale 3 in example 1), the rationale itself is
incomplete (like rationale 5 in example 1), or the rationale supports another
option rather than the prediction (like rationale 4 in the example 1), then
such output has a low entailment score leading to a lower ranking and getting
filtered out.

[sections/tables.tex]reasoningExamples

[sections/tables.tex]filterExamples