# Preconditioned Visual Language Inference with Weak Supervision

Ehsan Qasemi† Amani R. Maina-Kilaas 1 Devadutta Dash 2
Khalid Alsaggaf 3 Muhao Chen†
$\dagger$Department of Computer Science, University of Southern California
1Harvey Mudd College, 2Indian Institute of Technology, 3University of
Wisconsin-Madison
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS>

###### Abstract

Humans can infer the affordance of objects by extracting related contextual
preconditions for each scenario. For example, upon seeing an image with a
broken cup, we can infer that this precondition prevents the cup from being
used for drinking. Reasoning with preconditions of commonsense is studied in
NLP where the model explicitly gets the contextual precondition. However, it
is unclear if SOTA visual language models (VLMs) can extract such
preconditions and infer the affordance of objects with them. In this work, we
introduce the task of preconditioned visual language inference and
rationalization (PVLIR). We propose a learning resource based on three
strategies to retrieve weak supervision signals for the task and develop a
human-verified test set for evaluation. Our results reveal the shortcomings of
SOTA VLM models in the task and draw a road map to address the challenges
ahead in improving them.

††$1$ Work performed during REU Program at USC-ISI.††$2$ Work performed during
IUSSTF Viterbi India Program Internship at USC-ISI.††$3$ Work performed during
IKAUST Exchange Student Internship at USC-ISI.

##  1 Introduction

According to the _Theory of Affordance_ Gibson (2000); Chemero (2003),
understanding the preconditions in which an action or statement is possible or
impossible is a key aspect of human intelligence. For example, a glass may be
used for drinking water, under an implicit assumption that the water is at
normal temperature, but may not be if the glass is shattered. From the
cognitive perspective, understanding the affordance of objects, or simply
preconditions of actions Qasemi et al. (2022a), is part of the commonsense
knowledge that constitutes what distinguishes humans from a machine to make
inference Lenat (1998). From an applications perspective, it also has huge
implications such as robotics Ahn et al. (2022), transportations Prakken
(2017); Seff and Xiao (2016); Kothawade et al. (2021), and general artificial
intelligence Nguyen et al. (2021).

Figure 1: Preconditioned Visual Language Inference (PVLI) and Preconditioned
Visual Language Reasoning (PVLR) tasks. The “H” and “P” are the input
hypothesis and premise. The outputs, label (letter “L”) and rationale (letter
“R”), are highlighted.

Reasoning with preconditions of commonsense knowledge (i.e. preconditioned
inference), is proposed as a benchmarking task for evaluation of the theory of
affordance Qasemi et al. (2022a). Multiple studies have formulated the
preconditioned natural language inference (PNLI) as variations of the Natural
Language Inference (NLI) Williams et al. (2018); Bowman et al. (2015a);
Condoravdi et al. (2003) task and contributed learning resources that are
gathered through crowdsouring Rudinger et al. (2020); Qasemi et al. (2022a);
Hwang et al. (2020); Do and Pavlick (2021); Jiang et al. (2021b) or weak
supervision data Qasemi et al. (2022b). In PNLI, the models rely on the
contextual information (i.e. textual preconditions as premise) as input and
have to decide whether the hypothesis is true (entailment), false
(contradiction), or undetermined (neutral) given the premise (first row in
Figure 1). However, humans reason about affordance using information beyond
text Barsalou (2010); Andrews et al. (2009) and extract the contextual meaning
representations for cognitive tasks (such as PNLI) from the pool of available
information in various modalities. For example, upon getting the query “can
this person run?” and seeing a picture of a person in a full leg cast, one can
imply the contextual information from the image that “the person is injured
and incapable of running” and use it to answer the query accordingly. Thus, a
visual variation of the PNLI task is cognitively more realistic to benchmark
artificial intelligence models.

In this work, we propose PVLIR, to expand the preconditioned inference and
reasoning to the visual-language realm by considering the interaction between
linguistic and visual information in common sense. This work presents three
contributions. First, we introduce the Preconditioned Visual Language
Inference (PVLI) and Rationalization (PVLR) tasks (2nd and 3rd rows in Figure
1), which evaluate the visual-language models’ (VLM) capabilities to reason
with preconditions associated with commonsense knowledge. In PVLI, the
precondition is represented as an image that further constrains the context in
which the model has to decide the “prevent” or “allow” labels. In PVLR, the
model has to provide the rationale for the choice between the labels as well.
For example, say the model is given a commonsense statement such as “a glass
is used for drinking water” and/or an image of “a person drinking water” as
the hypothesis and an image of a “broken glass” as the premise. Then, in PVLI,
the model has to decide whether there is a prevented or allowed relation
between them, and in PVLR, it has to provide a rationale for its decision,
such as the glass is broken. In addition, to foster further research, we
created a crowd-verified evaluation dataset to benchmark future models.

Second, we propose three strategies for retrieving a rich amount of cheap and
allowably noisy supervision signals for inference and rationalization. Similar
to Parcalabescu et al. (2021), PVLIR’s three strategies rely on the available
image captioning datasets (e.g. Changpinyo et al. (2021); Sharma et al.
(2018); Gurari et al. (2020a); Lin et al. (2014a)) that are readily available
as a result of years of research in the field and maturity of resources. In
the first strategy, _Extraction from Captions_ , we utilize the PInKS Qasemi
et al. (2022b) method to extract PNLI instances from image captions. PInKS
uses a combination of linguistic patterns (e.g. “{action} unless
{precondition}”) and generative augmentation to extract large quantities of
instances from raw text. In the second strategy, _Caption Querying_ , we use
the existing crowdsourced PNLI instances (e.g. Rudinger et al. (2020); Qasemi
et al. (2022a); Hwang et al. (2020); Do and Pavlick (2021); Jiang et al.
(2021b)) and find an image caption that is semantically identical to them. The
third strategy, _Image Querying_ , focuses solely on the PNLI instances and
devises queries (such as “you are in a desert”) to search directly for
corresponding images on the web using image search engines (e.g. Google
Images).

Our third contribution is an extensive benchmarking of VLMs based on PVLIR. We
benchmark 4 SOTA VLMs, FLAVA Singh et al. (2022a), VisualBERT Li et al.
(2019), ViLBERT Lu et al. (2019) and ViLT Kim et al. (2021) in inference
(Section 4.1) and Ayyubi et al. (2020) in rationalization (Section 4.4). We
show how an effective rationalization will improve inference in the VLM models
(Section 4.4). We further investigate the fine-tuning (learning) process of
VLMs in the inference task (Section 4.2) and study their exploitation of the
spurious correlation in our dataset (Section 4.3).

##  2 Construction of PVLIR and Test Set

This section gives an overview of PVLIR (summarized in Figure 2), describing
our strategies for obtaining the data, and quality control. Details associated
with generating the human-verified test subset are moved to Appendix B. The
implementation details of each strategy are discussed in Section A.1.

Figure 2: Overview of weak supervision methods for constructing PVLIR.

#### Datasets:

The construction of PVLIR uses existing text-only PNLI and image-captioning
datasets as building blocks. For the text-only PNLI datasets, we require that
they contain a precondition (e.g. premise, context), an action (e.g.
hypothesis, question), and a binary label indicating whether the precondition
allows or prevents the action. For the image captioning datasets, we simply
require images (typically the URL) and their captions. Any datasets that meet
these requirements can be used for the following steps.

#### Preprocessing:

The PNLI preconditions and actions (collectively referred to as statements)
often use varying conventions for referring to people. We standardize them by
replacing these identifiers with “the person”, “another person”, “a third
person”, and so on. For example, the sentence “Alice helps Bob” would become
“the person helps another person”. This ensures that the specific names or
traditionally-associated genders are not mistaken as a focus of the
statements. We also encourage further preprocessing of the data as seen fit,
as not all sources contain clean text. Of the image captions, some may be very
long and consist of multiple sentences. In these cases, we split the captions
into individual examples, pairing each sentence with the same image. Using
these preprocessed resources, we then obtain PVLIR instances with three
different strategies: extraction from captions, caption querying, and image
querying.

#### Extraction from Captions (EC):

Our first strategy focuses solely on the image captions, finding the few that
contain preconditions and actions and extracting them. By nature, the
resulting statements are already grounded in the images. We use the minimally-
supervised approach described in PInKS Qasemi et al. (2022b), where linguistic
patterns are used to extract preconditions and actions from raw corpora. This
strategy constructs labeling functions (LF) based on common conjunctions such
as “only if” and “unless”. For example, the sentence “Swimming pools have cold
water in the winter unless they are heated” is matched by the pattern
“{action} unless {precondition}”, and therefore we can infer that “they are
heated” is a precondition that prevents the action “Swimming pools have cold
water in the winter”. Such labeling functions can be refined and added to as
desired. In cases where the conjunction can be used in multiple senses, part-
of-speech tagging can be utilized to filter out irrelevant senses. After
applying the labeling functions to the image captions, we have a dataset
consisting of preconditions and actions, where both are grounded in the
associated images. To control for quality, we annotate a sample of matches
from each labeling function (precision of each LF) to record whether the
relation between precondition and action makes sense. Based on the results, we
choose a precision threshold and only include labeling functions that meet
this minimum.

#### Caption Querying (CQ):

Our second strategy bridges the PNLI statements and image captions by
grounding preconditions and actions in images that have semantically similar
captions. We begin by limiting the statements and captions to those whose
length is within one standard deviation of the mean (rounded to the nearest
integer) in order to remove outliers. We then encode the PNLI statements and
image captions in high-dimensional vector embeddings using multiple models.
Next, using a PNLI precondition as a query, we find the most similar captions
through approximate nearest neighbors. This returns multiple rankings of the
closest captions, one for each model’s encoding. We then aggregate the
rankings and select the first-place caption. This strategy of including
multiple models in the decision-making process helps make it more robust to
model differences and to the approximate nature of the nearest neighbors. The
number of models incorporated depends on balancing the desired robustness and
time or computational constraints. Likewise, the number of similar captions
returned for the rankings can vary but should be chosen such that the rankings
typically contain some overlap. Note that the quality of data produced from
this approach is dependent on the range of concepts covered by the image
caption datasets, as it assumes that each PNLI precondition has a fairly
similar caption. As such, it benefits from a very large corpus of captions
(more discussion in Limitations).

To control for quality, when we select the best caption for a given query, we
additionally record two values: perplexity and model agreement. The perplexity
is the distance (cosine, dot, etc.) between the query and caption, averaged
over the models. In the case when one of the models did not include the chosen
caption in its ranking, the distance of the last caption is used for the
average. By nature, the perplexity measures how good the models believe the
match to be. In contrast, the model agreement is not specific to the chosen
caption but instead measures how well-aligned the models’ rankings are. Using
a ranking similarity metric, we compute the similarity between pairs of
rankings and then average the scores for the model agreement. Since a high
model agreement indicates that the models agree on which are the closest
captions, but does not speak to the actual proximity of the match, it can be
thought of as a measure of confidence.

#### Image Querying (IQ):

Our third strategy focuses solely on the PNLI statements and utilizes advances
in image search engines to directly find the relevant images on the internet.
Like the caption querying strategy, we limit the statements to those whose
length is within one standard deviation of the mean. For this approach, we
recommend excluding any PNLI datasets that deal with largely abstract concepts
(e.g. the person is responsible, the person will be grateful), as searching
directly for images is unlikely to yield good results. To ground the PNLI
statements, we find the top images on the internet, using the statement with
any commas removed as the search query. Since each of the top images becomes
its own example, this strategy can quickly generate very large amounts of
noisy training data.

##  3 Data Analysis

In this section, we investigate different aspects of the weak supervision data
and evaluate the quality of the generated resource. Implementation details and
experimental setup details are moved to Section A.1 (for data acquisition),
and Appendix B (for human annotations) to conserve space.

Figure 3: Distribution of instances extracted from captions (log-scale), for
each source of the caption. Figure 4: Cumulative distribution of the data with
respect to the precision of the generating labeling function.

We then use a random sample of 3% of the preprocessed preconditions and
actions to perform Caption Querying (roughly 13k each) and Image Querying
(roughly 80k each) to provide a representative sample of the full dataset.

#### Extraction from Captions Results:

After preprocessing, we have a resource of 17 million captions. From this, we
utilize the Extraction from Captions method that results in 34K extracted
instances. Figure 3 illustrates the percentage of matches that come from each
labeling function, separated by dataset. General statements such as “if”
unsurprisingly make up a large percentage of the data, but interestingly, some
datasets have very different distributions. Among the sources of captions,
VizWiz has disproportionately high counts of “but” and “so that”, while MS
COCO is high in “in order to” and “as if”.

Figure 4 shows the percent of the data and the percent of “allow” examples for
varying precision thresholds. For the results in Section 4, we use the
threshold of $0.6$ to have a good balance between the quality and quantity of
the final resource.

#### Caption Quering Results:

With Figure 5(a) and Figure 5(b), we look at where our matches in the CQ
method come from. This tells us that the majority of our queries come from
Anion while the majority of captions come from CC12M, which is unsurprising
given they are our largest datasets. Figure 5(c) and Figure 5(d) take the
ratio of the observed percentages to the percentages we would expect based
purely on the sizes of the datasets. While we always use every query, the
captions can be taken from any of the image datasets. This means that some may
be over or underrepresented based on their fit for a given NLI dataset. For
example, in Figure 5(d) it appears that MS COCO captions are not good matches
for PaCo actions.

(a)

(b)

(c)

(d)

Figure 5: 5(a)) Observed distribution of matches for preconditions. 5(b))
Observed distribution of matches for actions. 5(c)) Deviation from the
expected distribution of matches for preconditions. 5(d)) Deviation from the
expected distribution of matches for actions.

We also report more results on Image Querying in Section A.2.

##  4 Evaluation and Discussion

In this section, we focus on the PVLIR tasks. We first benchmark state-of-the-
art visual language models on the inference (Section 4.1) and reasoning tasks
(Section 4.4). Then, we focus on sources of bias in the data through
counterfactual analysis in the inference task (Section 4.3).

###  4.1 Inference Benchmarking Results

Here, as the main results, we benchmark the SoTA VLM models in the PVLI task.

#### Experimental Setup:

We used 4 SOTA vision-language models: ViLBERT Lu et al. (2019), ViLT Kim et
al. (2021), FLAVA Singh et al. (2022a), and CLIP Radford et al. (2021).

For the ViLBERT Lu et al. (2019) model, we used the pre-trained model provided
by the authors111https://github.com/facebookresearch/vilbert-multi-task. For
the rest of the models, we use the pre-trained weights from the Hugging Face
library Wolf et al. (2020). ViLBERT has separate encoders for processing
images and text, hence we feed the text and image separately. The model then
predicts one of the 2 outcomes in the PVLI, that is whether the precondition
(image) allows or prevents the commonsense statement. The ViLBERT model,
provided by the authors, is originally fine-tuned on the Visual Natural
Language Inference (VSNLI) task Vu et al. (2018); hence it is familiar with
the structure of the PVLI task and we can use it for a zero-shot evaluation as
well as the fine-tined evaluation. We fine-tune ViLBERT on the PVLI training
set with a batch size of 32 for 5 epochs, with the Adam Optimiser to optimize
the cross entropy loss between the actual and the predicted labels. For all
other hyperparameters, we used the default values by authors. Finally, we
report the accuracy of the ViLBERT model on the PVLI test set in both zero-
shot and fine-tuned setups.

The Hugging Face library contains the ViLT Kim et al. (2021) pre-trained on
the Visual Question-Answering task, in which the model has to find an answer
from a predefined set of tokens including yes and no. So for zero-shot and
fine-tuned results, we format the PVLI dataset into a question-answering
format with binary yes/no answers. The statement is converted into a question
format by appending the phrase “Is this possible?” to the statement. This
question is then fed into the model along with the associated image, which
acts as the premise. The model then outputs one of the 2 labels - yes or no,
which we use to compute its accuracy on the task.

FLAVA Singh et al. (2022a) and CLIP Radford et al. (2021) are multi-modal
vision and language models that can be used for tasks such as image-text
similarity or zero-shot image classification. Similar to ViLT, the hugging
face library does not provide CLIP and FLAVA models, pre-trained on binary or
multi-label classification tasks. For the fine-tuned results of FLAVA model,
we extract the multi-modal embeddings it generates and feed them to a
classification head. This classification head is fine-tuned on the VSNLI
before using in our experiments. For the CLIP model, we utilize the similarity
scores between the visual and the textual features. Similarly, we feed the
features through a classification head to output the label which indicates
whether the precondition “allows” or “prevents” the common sense statement. We
then report the accuracy of the resulting models on the PVLI tasks in zero-
shot and fine-tuned, w.r.t. PVLI, setups.

From the weak supervision data, we randomly sample 16K for tuning and 6K as
noisy test set. For the clean test set we used the $261$ human-verified
samples obtained through AMT experiments in Appendix B. The experiments are
conducted on a commodity workstation with an Intel Xeon Gold 5217 CPU and an
NVIDIA RTX 8000 GPU.

| 0-shot | Finetuned
---|---|---
Model | Noisy Test | Clean Test | Noisy Test | Clean Test
ViLBERT | 52.02 | 48.48 | 78.75 | 55.68
ViLT | 50.88 | 45.83 | 77.92 | 55.68
CLIP | 30.15 | 42.80 | 73.13 | 56.82
FLAVA | 47.38 | 53.78 | 80.43 | 59.47
Random | 63.47 | 56.08 |  |

Table 1: Results of SoTA Visual Language Models on the PVLI task.

#### Discussion

Table 1 summarizes the results of SoTA VLMs on the PVLI task. In the zero-shot
setup, all the models perform below the random baseline, showing the
difficulty and novelty of the task for the models. After fine-tuning, the
models’ performance improves above the random guess, where the FLAVA’s Singh
et al. (2022a) performance elevates by $33.05$ points of accuracy to $80.43\%$
on the noisy-test. However, it still is not mastering the task. Overall, this
shows that SOTA methods generally fall behind human-level performance,
therefore indicating the need for further research in order to improve the
comprehension of preconditions by commonsense visual reasoners.

###  4.2 Analysis with Fine-tuning

In the above evaluation on PVLI, we observe that all models get higher scores
after a full fine-tuning process. Here, we dissect the fine-tuning process to
find at what point the model understands the task’s requirements.

#### Experimental Setup

Here we focus on FLAVA Singh et al. (2022a) as one of the top-performing
models in PVLI. We carry the setup from Section 4.1 and evaluate FLAVA on the
noisy test set in fine-grained intervals during fine-tuning.

Figure 6: Accuracy difference from the peak value of fine-tuning FLAVA (lower
is better) with increasing amounts of tuning data from PVLI. The batch size is
64.

#### Discussion

Figure 6 illustrates the progression of the FLAVA model toward its peak
accuracy performance. As illustrated, the model’s performance saturates after
600 iterations of fine-tuning (or 38K instances). The slow saturation of the
accuracy score here suggests that the instances in PVLI are not trivial for
the model and it has to see a substantial number of instances to be able to
perform the task. Considering that the FLAVA has been pre-trained on a vast
corpus, our result shows the novelty and uniqueness of the PVLI task. This
result is consistent with the similar analysis in Qasemi et al. (2022a), for
comparing MNLI task with PNLI (text-only).

###  4.3 Identifying Sources of Bias

Large LMs (and by extension VLMs) tend to learn to solve the dataset rather
than the task Bras et al. (2020), by overfitting spurious correlations in the
data Xu et al. (2022). To quantify and eliminate such biases in the
data/model, recent studies conduct counterfactual inference to debias textual
resources used in text classification Qian et al. (2021a) and information
extraction tasks Wang et al. (2022c, 2023). Internally, debiasing through
counterfactuals works on the model trained or fine-tuned on the biased
classification data. During inference, this technique creates counterfactuals
where parts or all of the input are wiped out to observe what the model would
give by seeing only the biasing factors. In this way, bias terms can be
distilled from the model, which can be further deducted from the original
prediction for debiasing. Specifically, Qian et al. (2021a) design two types
of counterfactual variations of the input to produce two counterfactual output
distributions that model label bias and keyword bias in the model.

#### Experimental Setup

Since our data contain both images and text, we modified the counterfactuals
in Qian et al. (2021a) to fit the task. We create four counterfactual variants
of the inputs to consider, visual-token bias, textual-token bias, image bias,
and text bias. In the visual-token bias and textual-token bias, we partially
mask the input image ($50\%$) and text ($67\%$ as in Qian et al. (2021b))
respectively with no change to the other modality of input. In the image bias
and text bias we blind the model in one modality by fully masking their
respective modalities. Here, we focus on the FLAVA Singh et al. (2022a) model
and carry over the setup from Section 4.1 on the noisy test set.

#### Discussion

Our results show that the visually blind FLAVA Singh et al. (2022a) model is
performing on par with the original model ($79.88$ accuracy on noisy test).
This shows that the model may overly rely on the text modality as a shortcut
in most of the instances rather than utilizing both image and text. This
result further motivates the need for further research in multi-modal
debiasing techniques for both data and models.

###  4.4 Utilizing Rationale for Inference Task

Here, we try to answer the question “How can the rationales contribute to the
inference task?”. In other words, we show how the generated rationales can
become a piece of useful evidence for inference.

As discussed in Section 5 (under “Free-Text Rationale Generation”), even
though there exists a rich body of literature on the free-text rationale
generation models in the text-only tasks, there are limited publicly available
models for the visual language tasks. We implement the architecture proposed
in Ayyubi et al. (2020) for visually-guided rationale generation222At the time
of this writing, the original implementation of Ayyubi et al. (2020) was not
public. The architecture feeds the visual embeddings from a VLM to the decoder
of a LM and jointly trains both in an end-to-end fashion.

#### Experimental Setup

We do an experiment similar to Section 4.1, except that the VLM model is
trained with both the textual hypothesis and rationale plus the visual premise
as input. To contain the length of this experiment we only focus on the FLAVA
Singh et al. (2022a) VLM, and evaluate the performance on the noisy test set
in a fully fine-tuned setup. We separately experiment with two types of
rationales as input: the FLAVA-rationale-gen gets the generated rationale, and
the FLAVA-rationale-gold gets the ground-truth rationale from PVLIR .

For our implementation of Ayyubi et al. (2020) to generate the rationale, we
use a separate FLAVA Singh et al. (2022a) as the VLM to embed the multi-modal
input and use GPT-2 Radford et al. (2019) as a decoder-only LM to generated
the rationale. We initialize both models, from pre-trained weights on Hugging
Face Wolf et al. (2020) library and fine-tune them on PVLIR data for the
rationale generation task given the input (text and image).

#### Discussion

The inference accuracy of the FLAVA-rationale-gold and FLAVA-rationale-gen is
$94.2$ and $80.56$ respectively. First, the significant jump in the
performance of FLAVA-rationale-gold (from the base of $80.43$ in Table 1)
shows that in the presence of a competent rationalization model, the generated
rationales can significantly contribute to the inference task. Second, we
observe that a rationale model as simple as FLAVA-rationale-gen, can also
contribute to the performance (although slightly) of the visual preconditioned
inference task. This further motivates the need for further research in multi-
modal rationalization models.

##  5 Related Works

#### Preconditions of Commonsense Knowledge

reasoning with preconditions of common sense has been studied in the context
of affordance in different fields from cognitive sciences Garbarini and
Adenzato (2004) to robotics Ahn et al. (2022) but was recently brought up in
natural language understanding. In NLP, the focus has been mainly on proposing
human-verified learning resources Qasemi et al. (2022a); Rudinger et al.
(2020); Hwang et al. (2020); Sap et al. (2019); Heindorf et al. (2020); Do and
Pavlick (2021); Jiang et al. (2021a). Among them, Qasemi et al. (2022a) and
Rudinger et al. (2020) propose variations of the canonical NLI task for
preconditioned inference in common sense. Qasemi et al. (2022b) propose a
combination of weak-supervision strategy and biased masking to improve LMs’
performance in the task.

#### Visual Language Inference

With the advent of visual language models (VLMs; Li et al. 2022b; Liu et al.
2021; Li et al. 2019; Cho et al. 2021; Huang et al. 2022) that can
simultaneously process visual and linguistic information, there is growing
attention to enrich text-only tasks with visual context Parcalabescu et al.
(2021); Xie et al. (2018); Vu et al. (2018). Vu et al. (2018) propose a
visually-grounded version of the textual entailment task, supported by the
cognitive science view of enriching meaning representations with multiple
modalities. According to how Visual Language Inference (VLI; Xie et al. 2018;
Vu et al. 2018) is defined, the task is regarded as a visual extension of the
NLI task. In VLI, the premise is substituted with an image with visual context
instead of the text in NLI Xie et al. (2018). Instead of relying on
crowdsourcing, both works augment the Stanford NLI (SNLI) dataset Bowman et
al. (2015b). Since the textual premises of SNLI are extracted from image
captions on Flickr, each premise can be easily replaced with its respective
image. Our proposed PVLI task is a variation of the proposed VLI that focuses
on the preconditions (affordance) of tasks/objects (similar relation exists
between general NLI, e.g. MNLI versus the PNLI task).

#### Weak Supervision

Instead of using direct supervision from annotated data, weak supervision in
NLP tasks typically use linguistic patterns to infer large-scale “noisy” or
“imperfect” labels on unlabelled corpora Rekatsinas et al. (2017); Zhang et
al. (2017); Dehghani et al. (2017); Singh et al. (2022b), e.g. using heuristic
rules. Models fine-tuned on weak supervision data have shown considerable
improvements across NLU tasks lacking direct supervision, including temporal
commonsense reasoning Zhou et al. (2020), rationale generation Brahman et al.
(2020), document ranking Dehghani et al. (2017), and preconditioned inference
Qasemi et al. (2022b). Dai et al. (2021); Choi et al. (2018) use weak
supervision from linguistically mined or LM-generated noisy data to enhance
ultra-fine entity typing. Choi et al. (2018) also uses human annotators to
create a small ground-truth test set for testing.

#### Free-Text Rationale Generation

There is a large body of research on free-text rationale generation toward
faithful and explainable NLP. Work like this typically fine-tunes a single LM
to generate the task output and rationale Narang et al. (2020); Marasović et
al. (2021); Zelikman et al. (2022), or uses a separate LM to generate the
rationale that another LM uses to generate the output Wang et al. (2022a); Wei
et al. (2022); Kumar and Talukdar (2020); Rajani et al. (2019). In the visual-
language realm, free-text rationale generation is limited, where based on our
observation it can be due to the lack of large-scale learning resources. Dua
et al. (2021) and Ayyubi et al. (2020) repurpose the VCR Zellers et al. (2019)
data and propose VL models to generate free-text rationale (instead of picking
one as is in the VCR) for it. Other works, e.g. Su et al. (2022); Li et al.
(2022a), use visual inputs for text generation, but they are not focused on
the rationale generation.

##  6 Conclusion and Future Work

We introduce the Preconditioned Visual Language Inference and Rationalization
tasks to measure how SOTA VLMs can extract preconditions and infer the
affordance of the objects. We propose three strategies for obtaining and
retrieving a rich amount of cheap and allowably noisy supervision signals for
the task and use them to create a crowd-verified evaluation dataset to
benchmark future models. Our experiments show that SOTA VLMs largely fall
behind human performance in the proposed task raw a road map to address the
challenges ahead in improving them.

## Limitations

Image captioning datasets are limited both in breadth and depth. We have not
investigated the use of automatically generated captions, e.g. Wang et al.
(2022b), in our weak-supervised pipeline, but it is a viable path for future
extensions of this work. Alternatively, automatic text-to-image generation
techniques, e.g. stable diffusion Rombach et al. (2022) or Dall-E Ramesh et
al. (2022), are gaining a lot of attention and are promising but require a lot
of prompt engineering that is challenging on a large scale. In addition, the
lack of access to a large number of free-text rationale generation models
(through libraries such as Huggingface Wolf et al. (2020)) limited the
evaluation of our PVLR tasks. We hope the availability of resources, such as
ours, elicits more research effort in the field.

## Ethical Concerns

We started from publicly available data that is both crowd-verified and
neutralized, however, multiple studies have shown the existence of bias and
ethical issues in such resources, e.g. Mehrabi et al. (2021). Since our work
is based on weak supervision, we have no additional filter on the acquired
instances, hence our resource exacerbates the bias in models by reinforcing it
with biased evidence, e.g. results from the query “fat person” will only
return images of obese white males. In addition, there is a combination of
well-studied biases in the large models trained on raw text, e.g. Bender et
al. (2021).

Finally, in this work, we have only relied on English resources. In addition,
we have only used English-speaking annotators. Hence the judgments and design
decisions are heavily skewed culturally which will aggravate the bias issues
of our work.

## References

  * Ahn et al. (2022) Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al. 2022.  Do as i can, not as i say: Grounding language in robotic affordances.  _arXiv preprint arXiv:2204.01691_.
  * Akbik et al. (2018) Alan Akbik, Duncan Blythe, and Roland Vollgraf. 2018.  Contextual string embeddings for sequence labeling.  In _COLING 2018, 27th International Conference on Computational Linguistics_ , pages 1638–1649.
  * Andrews et al. (2009) Mark Andrews, Gabriella Vigliocco, and David Vinson. 2009.  Integrating experiential and distributional data to learn semantic representations.  _Psychological review_ , 116(3):463.
  * Ayyubi et al. (2020) Hammad A Ayyubi, Md Tanjim, Julian J McAuley, Garrison W Cottrell, et al. 2020.  Generating rationales in visual question answering.  _arXiv preprint arXiv:2004.02032_.
  * Barsalou (2010) Lawrence W Barsalou. 2010.  Grounded cognition: Past, present, and future.  _Topics in cognitive science_ , 2(4):716–724.
  * Bender et al. (2021) Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021.  On the dangers of stochastic parrots: Can language models be too big?  In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_ , pages 610–623.
  * Bird et al. (2009) Steven Bird, Ewan Klein, and Edward Loper. 2009.  _Natural language processing with Python: analyzing text with the natural language toolkit_.  " O’Reilly Media, Inc.".
  * Bowman et al. (2015a) Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015a.  A large annotated corpus for learning natural language inference.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 632–642, Lisbon, Portugal. Association for Computational Linguistics.
  * Bowman et al. (2015b) Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015b.  A large annotated corpus for learning natural language inference.
  * Brahman et al. (2020) Faeze Brahman, Vered Shwartz, Rachel Rudinger, and Yejin Choi. 2020.  Learning to rationalize for nonmonotonic reasoning with distant supervision.  _arXiv preprint arXiv:2012.08012_.
  * Bras et al. (2020) Ronan Le Bras, Swabha Swayamdipta, Chandra Bhagavatula, Rowan Zellers, Matthew E. Peters, Ashish Sabharwal, and Yejin Choi. 2020.  Adversarial filters of dataset biases.  In _Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event_ , volume 119 of _Proceedings of Machine Learning Research_ , pages 1078–1088. PMLR.
  * Changpinyo et al. (2021) Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. 2021.  Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts.
  * Chemero (2003) Anthony Chemero. 2003.  An outline of a theory of affordances.  _Ecological psychology_ , 15(2):181–195.
  * Cho et al. (2021) Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. 2021.  Unifying vision-and-language tasks via text generation.  In _International Conference on Machine Learning_ , pages 1931–1942. PMLR.
  * Choi et al. (2018) Eunsol Choi, Omer Levy, Yejin Choi, and Luke Zettlemoyer. 2018.  Ultra-fine entity typing.  _arXiv preprint arXiv:1807.04905_.
  * Condoravdi et al. (2003) Cleo Condoravdi, Dick Crouch, Valeria De Paiva, Reinhard Stolle, and Daniel Bobrow. 2003.  Entailment, intensionality and text understanding.  In _Proceedings of the HLT-NAACL 2003 workshop on Text meaning_ , pages 38–45.
  * Copeland (1951) Arthur H Copeland. 1951.  A reasonable social welfare function.  Technical report, Mimeo, University of Michigan USA.
  * Crowston (2012) Kevin Crowston. 2012.  Amazon mechanical turk: A research tool for organizations and information systems scholars.  In _Shaping the Future of ICT Research. Methods and Approaches_ , pages 210–221. Springer.
  * Dai et al. (2021) Hongliang Dai, Yangqiu Song, and Haixun Wang. 2021.  Ultra-fine entity typing with weak supervision from a masked language model.  _arXiv preprint arXiv:2106.04098_.
  * Dehghani et al. (2017) Mostafa Dehghani, Hamed Zamani, Aliaksei Severyn, Jaap Kamps, and W Bruce Croft. 2017.  Neural ranking models with weak supervision.  In _Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pages 65–74.
  * Do and Pavlick (2021) Nam Do and Ellie Pavlick. 2021.  Are rotten apples edible? challenging commonsense inference ability with exceptions.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 2061–2073.
  * Dua et al. (2021) Radhika Dua, Sai Srinivas Kancheti, and Vineeth N Balasubramanian. 2021.  Beyond vqa: Generating multi-word answers and rationales to visual questions.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 1623–1632.
  * Garbarini and Adenzato (2004) Francesca Garbarini and Mauro Adenzato. 2004.  At the root of embodied cognition: Cognitive science meets neurophysiology.  _Brain and cognition_ , 56(1):100–106.
  * Gibson (2000) Eleanor J Gibson. 2000.  Where is the information for affordances?  _Ecological Psychology_ , 12(1):53–56.
  * Gurari et al. (2020a) Danna Gurari, Yinan Zhao, Meng Zhang, and Nilavra Bhattacharya. 2020a.  Captioning images taken by people who are blind.  In _European Conference on Computer Vision_ , pages 417–434. Springer.
  * Gurari et al. (2020b) Danna Gurari, Yinan Zhao, Meng Zhang, and Nilavra Bhattacharya. 2020b.  Captioning images taken by people who are blind.
  * Heindorf et al. (2020) Stefan Heindorf, Yan Scholten, Henning Wachsmuth, Axel-Cyrille Ngonga Ngomo, and Martin Potthast. 2020.  Causenet: Towards a causality graph extracted from the web.  In _CIKM ’20: The 29th ACM International Conference on Information and Knowledge Management, Virtual Event, Ireland, October 19-23, 2020_ , pages 3023–3030. ACM.
  * Huang et al. (2022) Luyang Huang, Guocheng Niu, Jiachen Liu, Xinyan Xiao, and Hua Wu. 2022.  Du-vlg: Unifying vision-and-language generation via dual sequence-to-sequence pre-training.  _arXiv preprint arXiv:2203.09052_.
  * Hwang et al. (2020) Jena D Hwang, Chandra Bhagavatula, Ronan Le Bras, Jeff Da, Keisuke Sakaguchi, Antoine Bosselut, and Yejin Choi. 2020.  Comet-atomic 2020: On symbolic and neural commonsense knowledge graphs.  _arXiv preprint arXiv:2010.05953_.
  * Jiang et al. (2021a) Liwei Jiang, Antoine Bosselut, Chandra Bhagavatula, and Yejin Choi. 2021a.  " i’m not mad": Commonsense implications of negation and contradiction.  _arXiv preprint arXiv:2104.06511_.
  * Jiang et al. (2021b) Liwei Jiang, Antoine Bosselut, Chandra Bhagavatula, and Yejin Choi. 2021b.  “i’m not mad”: Commonsense implications of negation and contradiction.
  * Kim et al. (2021) Wonjae Kim, Bokyung Son, and Ildoo Kim. 2021.  Vilt: Vision-and-language transformer without convolution or region supervision.  In _International Conference on Machine Learning_ , pages 5583–5594. PMLR.
  * Kothawade et al. (2021) Suraj Kothawade, Vinaya Khandelwal, Kinjal Basu, Huaduo Wang, and Gopal Gupta. 2021\.  Auto-discern: Autonomous driving using common sense reasoning.  _arXiv preprint arXiv:2110.13606_.
  * Kumar and Talukdar (2020) Sawan Kumar and Partha Talukdar. 2020.  Nile: Natural language inference with faithful natural language explanations.  _arXiv preprint arXiv:2005.12116_.
  * Lenat (1998) Doug Lenat. 1998.  The dimensions of context-space.
  * Li et al. (2022a) Bin Li, Yixuan Weng, Ziyu Ma, Bin Sun, and Shutao Li. 2022a.  Scene-aware prompt for multi-modal dialogue understanding and generation.  In _CCF International Conference on Natural Language Processing and Chinese Computing_ , pages 179–191. Springer.
  * Li et al. (2022b) Kunchang Li, Yali Wang, Peng Gao, Guanglu Song, Yu Liu, Hongsheng Li, and Yu Qiao. 2022b.  Uniformer: Unified transformer for efficient spatiotemporal representation learning.  _arXiv preprint arXiv:2201.04676_.
  * Li et al. (2019) Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2019\.  Visualbert: A simple and performant baseline for vision and language.  _arXiv preprint arXiv:1908.03557_.
  * Lin et al. (2014a) Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014a.  Microsoft coco: Common objects in context.  In _European conference on computer vision_ , pages 740–755. Springer.
  * Lin et al. (2014b) Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014b.  Microsoft coco: Common objects in context.  In _Computer Vision – ECCV 2014_ , pages 740–755, Cham. Springer International Publishing.
  * Liu et al. (2021) Alexander H Liu, SouYoung Jin, Cheng-I Jeff Lai, Andrew Rouditchenko, Aude Oliva, and James Glass. 2021.  Cross-modal discrete representation learning.  _arXiv preprint arXiv:2106.05438_.
  * Lu et al. (2019) Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  _Advances in neural information processing systems_ , 32.
  * Marasović et al. (2021) Ana Marasović, Iz Beltagy, Doug Downey, and Matthew E Peters. 2021.  Few-shot self-rationalization with natural language prompts.  _arXiv preprint arXiv:2111.08284_.
  * Mehrabi et al. (2021) Ninareh Mehrabi, Pei Zhou, Fred Morstatter, Jay Pujara, Xiang Ren, and Aram Galstyan. 2021.  Lawyers are dishonest? quantifying representational harms in commonsense knowledge resources.  _arXiv preprint arXiv:2103.11320_.
  * Narang et al. (2020) Sharan Narang, Colin Raffel, Katherine Lee, Adam Roberts, Noah Fiedel, and Karishma Malkan. 2020.  Wt5?! training text-to-text models to explain their predictions.  _arXiv preprint arXiv:2004.14546_.
  * Nguyen et al. (2021) Tuan-Phong Nguyen, Simon Razniewski, and Gerhard Weikum. 2021.  Advanced semantics for commonsense knowledge extraction.  In _Proceedings of the Web Conference 2021_ , pages 2636–2647.
  * Parcalabescu et al. (2021) Letitia Parcalabescu, Michele Cafagna, Lilitta Muradjan, Anette Frank, Iacer Calixto, and Albert Gatt. 2021.  Valse: A task-independent benchmark for vision and language models centered on linguistic phenomena.
  * Prakken (2017) Henry Prakken. 2017.  On the problem of making autonomous vehicles conform to traffic law.  _Artificial Intelligence and Law_ , 25(3):341–363.
  * Qasemi et al. (2022a) Ehsan Qasemi, Filip Ilievski, Muhao Chen, and Pedro Szekely. 2022a.  Paco: Preconditions attributed to commonsense knowledge.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing-Findings_ , page 6781–6796, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Qasemi et al. (2022b) Ehsan Qasemi, Piyush Khanna, Qiang Ning, and Muhao Chen. 2022b.  Pinks: Preconditioned commonsense inference with minimal supervision.  In _AACL-IJCNLP 2022_.
  * Qian et al. (2021a) Chen Qian, Fuli Feng, Lijie Wen, Chunping Ma, and Pengjun Xie. 2021a.  Counterfactual inference for text classification debiasing.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 5434–5445.
  * Qian et al. (2021b) Chen Qian, Fuli Feng, Lijie Wen, Chunping Ma, and Pengjun Xie. 2021b.  Counterfactual inference for text classification debiasing.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 5434–5445, Online. Association for Computational Linguistics.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021.  Learning transferable visual models from natural language supervision.  In _International Conference on Machine Learning_ , pages 8748–8763. PMLR.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Rajani et al. (2019) Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. 2019.  Explain yourself! leveraging language models for commonsense reasoning.  _arXiv preprint arXiv:1906.02361_.
  * Ramesh et al. (2022) Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. 2022.  Hierarchical text-conditional image generation with clip latents.  _arXiv preprint arXiv:2204.06125_.
  * Ratner et al. (2017) Alexander Ratner, Stephen H. Bach, Henry Ehrenberg, Jason Fries, Sen Wu, and Christopher Ré. 2017.  Snorkel: Rapid training data creation with weak supervision.  _Proc. VLDB Endow._ , 11(3):269–282.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence embeddings using Siamese BERT-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3982–3992, Hong Kong, China. Association for Computational Linguistics.
  * Rekatsinas et al. (2017) Theodoros Rekatsinas, Xu Chu, Ihab F Ilyas, and Christopher Ré. 2017.  Holoclean: Holistic data repairs with probabilistic inference.  _arXiv preprint arXiv:1702.00820_.
  * Rombach et al. (2022) Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 10684–10695.
  * Rudinger et al. (2020) Rachel Rudinger, Vered Shwartz, Jena D. Hwang, Chandra Bhagavatula, Maxwell Forbes, Ronan Le Bras, Noah A. Smith, and Yejin Choi. 2020.  Thinking like a skeptic: Defeasible inference in natural language.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 4661–4675, Online. Association for Computational Linguistics.
  * Sap et al. (2019) Maarten Sap, Ronan Le Bras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A. Smith, and Yejin Choi. 2019.  ATOMIC: an atlas of machine commonsense for if-then reasoning.  In _AAAI 2019_ , pages 3027–3035. AAAI Press.
  * Sap et al. (2018) Maarten Sap, Ronan LeBras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A. Smith, and Yejin Choi. 2018.  Atomic: An atlas of machine commonsense for if-then reasoning.
  * Seff and Xiao (2016) Ari Seff and Jianxiong Xiao. 2016.  Learning from maps: Visual common sense for autonomous driving.  _arXiv preprint arXiv:1611.08583_.
  * Sharma et al. (2018) Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2556–2565.
  * Singh et al. (2022a) Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. 2022a.  Flava: A foundational language and vision alignment model.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 15638–15650.
  * Singh et al. (2022b) Shikhar Singh, Ehsan Qasemi, and Muhao Chen. 2022b.  Viphy: Probing" visible" physical commonsense knowledge.  _arXiv preprint arXiv:2209.07000_.
  * Su et al. (2022) Yixuan Su, Tian Lan, Yahui Liu, Fangyu Liu, Dani Yogatama, Yan Wang, Lingpeng Kong, and Nigel Collier. 2022.  Language models can see: Plugging visual controls in text generation.  _arXiv preprint arXiv:2205.02655_.
  * Vu et al. (2018) Hoa Trong Vu, Claudio Greco, Aliia Erofeeva, Somayeh Jafaritazehjan, Guido Linders, Marc Tanti, Alberto Testoni, Raffaella Bernardi, and Albert Gatt. 2018\.  Grounded textual entailment.  _arXiv preprint arXiv:1806.05645_.
  * Wang et al. (2023) Haoyu Wang, Hongming Zhang, Yuqian Deng, Jacob R Gardner, Muhao Chen, and Dan Roth. 2023.  Extracting or guessing? improving faithfulness of event temporal relation extraction.  In _EACL_.
  * Wang et al. (2022a) Peifeng Wang, Aaron Chan, Filip Ilievski, Muhao Chen, and Xiang Ren. 2022a.  Pinto: Faithful language reasoning using prompt-generated rationales.  _arXiv preprint arXiv:2211.01562_.
  * Wang et al. (2022b) Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. 2022b.  Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework.  In _International Conference on Machine Learning_ , pages 23318–23340. PMLR.
  * Wang et al. (2022c) Yiwei Wang, Muhao Chen, Wenxuan Zhou, Yujun Cai, Yuxuan Liang, Dayiheng Liu, Baosong Yang, Juncheng Liu, and Bryan Hooi. 2022c.  Should we rely on entity mentions for relation extraction? debiasing relation extraction with counterfactual analysis.  In _NAACL_.
  * Webber et al. (2010) William Webber, Alistair Moffat, and Justin Zobel. 2010.  A similarity measure for indefinite rankings.  _ACM Trans. Inf. Syst._ , 28(4).
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022.  Chain of thought prompting elicits reasoning in large language models.  _arXiv preprint arXiv:2201.11903_.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1112–1122, New Orleans, Louisiana. Association for Computational Linguistics.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _EMNLP 2020: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Xie et al. (2018) Ning Xie, Farley Lai, Derek Doran, and Asim Kadav. 2018.  Visual entailment task for visually-grounded language learning.  _arXiv preprint arXiv:1811.10582_.
  * Xu et al. (2022) Nan Xu, Fei Wang, Bangzheng Li, Mingtao Dong, and Muhao Chen. 2022.  Does your model classify entities reasonably? diagnosing and mitigating spurious correlations in entity typing.  _arXiv preprint arXiv:2205.12640_.
  * Zelikman et al. (2022) Eric Zelikman, Yuhuai Wu, and Noah D Goodman. 2022.  Star: Bootstrapping reasoning with reasoning.  _arXiv preprint arXiv:2203.14465_.
  * Zellers et al. (2019) Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019.  From recognition to cognition: Visual commonsense reasoning.  In _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Zhang et al. (2017) Ce Zhang, Christopher Ré, Michael Cafarella, Christopher De Sa, Alex Ratner, Jaeho Shin, Feiran Wang, and Sen Wu. 2017.  Deepdive: Declarative knowledge base construction.  _Communications of the ACM_ , 60(5):93–102.
  * Zhou et al. (2020) Ben Zhou, Qiang Ning, Daniel Khashabi, and Dan Roth. 2020.  Temporal common sense acquisition with minimal supervision.  _arXiv preprint arXiv:2005.04304_.

Appendices

##  Appendix A Weak Supervision Methods

###  A.1 Implementation Details and Experimental Setup

This section discusses the experimental setup and implementation details for
the results in Section 3.

#### Precondition Resources:

For our P-NLI datasets, we pull from Anion Jiang et al. (2021b), Atomic Sap et
al. (2018), PaCo Qasemi et al. (2022a), $\delta$-NLI Rudinger et al. (2020)
and WinoVenti Do and Pavlick (2021). For the image caption datasets, we use
CC12M Changpinyo et al. (2021), CC3M Sharma et al. (2018), MS COCO Lin et al.
(2014b), and VizWiz Gurari et al. (2020b).

#### Preprocessing Setup:

Since Anion and Atomic use fixed identifiers (Alice/Bob, PersonX/PersonY), we
rely on regex rules to replace them with “the person” and “another person”.
WinoVenti uses random first names, and so we utilize Flair’s ner-english-fast
model Akbik et al. (2018) to identify and replace the spans that are
identified as people with greater than 90% confidence. PaCo does not have any
such identifiers to replace.

For the image caption data, we break the captions into multiple lines using
Natural Language Toolkit’s sentence tokenizer Bird et al. (2009) in
combination with splitting on newline characters. We also notice that some
contained “<PERSON>” tags and use regex to replace them.

As the last step, we leverage regex to fix whitespace issues and replace “the
person’s” with “their” to increase fluidity. We also found that datasets were
easier to clean after lowercasing, particularly as some contained inconsistent
capitalization.

#### Extraction from Captions Setup:

We modify some of the original labeling functions from PInKS Qasemi et al.
(2022b) and add eight new ones after inspecting our caption corpus. In PInKS,
the authors also calculate precision values for each of their labeling
functions by sampling 20 examples from each function. The samples are then
marked as relevant (score of 1) or irrelevant (score of 0) to the task by two
human annotators. The average score of each labeling function provides an
estimate of the quality that each labeling function returns and is used for
tie-breaking matches or filtering out low-quality functions. We follow their
lead and do the same, computing these precision values specifically on our
caption corpus. Table 2 summarizes all the labeling functions, patterns, their
precision, and other details associated with them.

Label | Conjunction | Precision | Regex Pattern
---|---|---|---
enables | so that | 0.689 | {P} so that {A}
| in order to | 0.650 | {P} in order to {A}
| because | 0.625 | {A} because (?!of$\backslash$b){P}
| due to | 0.550 | {A} due to {P}
| in case | 0.475 | {A} in case (?!of$\backslash$b){P}
| as if | 0.400 | {A} as if {P}
| as long as | 0.375 | {A} as long as {P}
| if | 0.150 | {A}(?<!$\backslash$bas) if (?!not$\backslash$b){P}
| in the event | 0.100 | {A} in the event {P}
| on condition | 0.045 | {A} on condition (?!of anonymity$\backslash$b){P}
| supposing | 0.000∗ | {A} supposing {P}
| on the assumption | 0.000∗ | {A} on the assumption {P}
| in the case that | 0.000∗ | {A} in the case that {P}
| contingent upon | 0.000∗ | {A} contingent upon {P}
| with the proviso | — | {A} with the proviso {P}
| to understand event | — | to understand the event "{E}", it is important to know that {P}$\backslash$.
| statement is true | — | the statement "{E}" is true because {P}$\backslash$.
| only if | — | {A} only if {P}
| on these terms | — | {A} on these terms {P}
| makes possible | — | {P} makes {A} possible$\backslash$.
disables | unless | 0.750 | {A} unless {P}
| even though | 0.550 | {A} even though {P}
| despite | 0.475 | {A} despite {P}
| if not | 0.300 | {A}(?<!$\backslash$bas) if not (?!(more|most|many|all)$\backslash$b){P}
| without | 0.257 | {A} without {P}
| but | 0.175 | {A} but {NP}
| except | 0.075 | {A} except {P}
| lest | 0.045∗ | {A} lest {P}
| excepting that | — | {A} excepting that {P}
| except for | — | {A} except for {P}
Table 2: Regex patterns for the labeling functions. A=action, E=event,
P=precondition, NP=negative precondition. Patterns with fewer than 20 examples
in the corpora are marked with asterisks, and those with no examples are left
empty. Bolded conjunctions were followed with part-of-speech tagging to
confirm that they were used as conjunctions.

Balancing quality and quantity in our data, we select a threshold of $0.60$
and only use the labeling functions that meet this minimum. The labeling
functions are applied using Snorkel Ratner et al. (2017), a SOTA framework for
algorithmically labeling data—see the original PInKS paper for more detail on
the setup for Snorkel. Finally, We noticed that not all of our patterns were
being used in the sentence as conjunctions, and utilized Flair’s pos-english-
fast model to remove some examples for select patterns.

#### Caption Querying Setup:

For our models, we use the sentence transformers Reimers and Gurevych (2019)
all-distilroberta-v1, all-MiniLM-L12-v2, and all-mpnet-base-v2 from
HuggingFace Wolf et al. (2020). When forming the rankings, we retrieve the 50
closest captions, as that provides a decent overlap and completes within a
reasonable amount of time. To aggregate our rankings and select the best
caption, we use Copeland’s method Copeland (1951). To compare our rankings for
model agreement, we utilize the extrapolated form of rank-biased overlap
Webber et al. (2010).

Furthermore, we annotate a subset of our data to assess the ability of
perplexity and model agreement to separate good training examples from poor
ones. When we rate examples for quality of match between the statement
(precondition/action) and the fetched image caption, on a scale from 1 (worst)
to 4 (best), we find that our measures are reasonably successful (See Figure
7). However, when we ask Amazon Mechanical Turk workers to vote on examples
for overall quality, requiring that both statements and images be cohesive,
the measures are unable to isolate better examples (See Figure 8). Given that
the measures are based purely on the textual match, it makes sense that it
would perform better without the incorporation of the image. Unfortunately,
matching with a caption is not always sufficient for matching with the
associated image. Further work is needed to develop useful heuristics for the
overall quality of a training example.

Figure 7: Heatmap graph comparing the measures of perplexity and model
agreement with expert human evaluation in the caption querying method. Bins
are computed using 6-quantiles for each axis. Figure 8: Heatmap graph
comparing the measures of perplexity and model agreement with human evaluation
from Amazon Mechanical Turk in the caption querying method. Bins are computed
using 6-quantiles for each axis.

#### Image Querying Setup:

To find the top images on the internet, we use Google Images Download
333https://github.com/Joeclinton1/google-images-download to retrieve the URLs
of images. We obtain the top 10 images for each query as it is large enough to
generate lots of data while keeping them relevant to the query.

###  A.2 Image Search Results

While less information is available for the Image Querying data, we can look
at the websites most frequently drawn from for the matches. Table 3 and Table
4 display the top 10 websites for each NLI dataset for preconditions and
actions, respectively. If desired, it is possible to remove images from
unwanted websites.

WinoVenti | PaCo | Anion
---|---|---
m.media-amazon.com (89) | quotefancy.com (102) | quotefancy.com (4721)
i.ytimg.com (85) | i.ytimg.com (80) | thumbs.dreamstime.com (2668)
cdn.shopify.com (67) | thumbs.dreamstime.com (73) | i0.wp.com (2074)
upload.wikimedia.org (64) | i0.wp.com (72) | i.pinimg.com (1662)
media.istockphoto.com (52) | media.istockphoto.com (46) | www.wikihow.com (1597)
thumbs.dreamstime.com (50) | m.media-amazon.com (46) | www.verywellmind.com (1546)
i0.wp.com (47) | i.pinimg.com (39) | media.istockphoto.com (1251)
images.squarespace-cdn.com (35) | c8.alamy.com (36) | miro.medium.com (997)
i5.walmartimages.com (34) | upload.wikimedia.org (35) | www.incimages.com (967)
c8.alamy.com (33) | www.wikihow.com (33) | previews.123rf.com (900)
Table 3: Top 10 websites for preconditions by NLI dataset. There are a total of 10,975 unique websites for 50,729 unique images belonging to 82,740 examples. WinoVenti | PaCo | Anion
---|---|---
m.media-amazon.com (54) | i0.wp.com (79) | thumbs.dreamstime.com (3164)
i.ytimg.com (25) | www.verywellmind.com (62) | quotefancy.com (2943)
i5.walmartimages.com (24) | upload.wikimedia.org (45) | i0.wp.com (2050)
thumbs.dreamstime.com (21) | post.healthline.com (34) | c8.alamy.com (1964)
cdn.shopify.com (21) | media.cheggcdn.com (24) | media.istockphoto.com (1962)
i0.wp.com (20) | quotefancy.com (19) | www.wikihow.com (1469)
c8.alamy.com (20) | qph.cf2.quoracdn.net (19) | www.verywellmind.com (1262)
i.etsystatic.com (19) | www.helpguide.org (17) | i.insider.com (1213)
upload.wikimedia.org (18) | media.self.com (15) | previews.123rf.com (1075)
media.istockphoto.com (17) | images.squarespace-cdn.com (15) | i.ytimg.com (1050)
Table 4: Top 10 websites for actions by NLI dataset. There are a total of
9,700 unique websites for 48,305 unique images belonging to 80,170 examples.

###  A.3 Model Sizes and Run-times

For results in Table 1, the runtimes are FLAVA=4hr, VilBERT=4hr, Clip=5hr,
ViLT=4hr; the model sizes for VLMs and LMs are identical to their respective
implementations from the source (e.g. gpt2 has 1.5B parameters on Wolf et al.
(2020)). The classification head added to VLMs (e.g. FLAVA) has $1.4k$
parameters.

##  Appendix B Data Annotation Details

We used Amazon Mechanical Turk (AMT) Crowston (2012) to evaluate the quality
of extracted PVLIR instances through our proposed weak supervision methods.
This enabled us to coordinate the study and access a large pool of English-
speaking participants as our study population. The AMT is especially suitable
for this study as it can facilitate accessing a diverse population of
participants which is necessary for any notion of common sense. Our study on
AMT consists of two parts: a tutorial, which also serves as a qualification
test, and the main survey. In addition, we implemented two levels of quality
control: in the first one we use a response checker code and in the second we
use human annotators to ensure only high-quality responses wind up in the
final data.

###  B.1 Main AMT Survey

In the main survey, the participants are given a set of question units each
consisting of a prompt question, an image, and the radio buttons with three
options. We then ask participants to select their responses for each prompt
question from the available options in the unit (e.g. “true” “false” “not
sure” sample in Fig. 9). We create a question until through the PVLI instances
with image and text, that was discussed in Section 2.

Since our annotated images are not perfect, there are a lot of possible points
of failure that can render the question units to be impossible to understand.
For example, some of the annotations may not be correct, the automatic
conversion of meta-data to a sentence can be wrong in corner cases, or the
image links be corrupted. Hence some of the question units may have odd
grammar (e.g. "An net is used for catch fish"). Consequently, some of the
question units may be hard to understand or just be wrong. To help us find
those question units and ignore them in future iterations, each question unit
has a checkbox in front of it with the label "not sure/does not make sense".
The participant may choose to select the option and skip answering that
prompt. To make the payment structure fair for the participants, they will get
paid regardless of their responses. We keep the right reserved to block the
participants who abuse this option using the annotator agreement metric.

Figure 9: A sample question-unit used in the main survey on the AMT.

###  B.2 Qualifying Participants

In the tutorial, first, we have prepared detailed instructions that explain to
the participants what they need to do and what are the criteria for a good vs
bad response. For example, in the instructions, we ask participants to avoid
answering “correct”/“incorrect” when they are not sure or when there is
something wrong with the image or text of the question unit. The instruction
is <500 words with an expected reading time of <7 mins. Additionally, we have
prepared a set of good/bad examples associated with each rule that can also be
accessed in the tutorial. Each one of the good/bad examples comes with a short
explanation that discusses the reason for the good/bad rating of the response.
The participants are then asked to give the qualification test as a check on
whether they have read and understood the instructions. The qualification test
contains $\sim$10 question units similar to the ones they will see in the
original survey (due to AMT limitations the qualification question units have
a different visual layout but contain the same information). We have carefully
designed each qualification question unit such that it tests the participants’
understanding of the rules individually and give them feedback on their wrong
answers. For example, for the rule discouraging the use of
“correct”/“incorrect” when the question unit is invalid, we have two question
units where first the image is not visible, and second, the text is gibberish.
After successfully passing the test, participants with acceptable scores are
granted a qualification badge that allows them to engage in the main survey.
It must be noted that the detailed instructions and the good/bad examples are
both available in the main survey as a memory refresher for the participants.

To coordinate the study and access a large pool of participants we use
Amazon’s Mechanical Turk (AMT) service to hire English-speaking people with no
specific background as our study population. As part of AMT’s service design,
the main survey can be divided into thousands of micro-tasks that each is
related to a handful of unique question units. In this setup, the participants
may choose their amount of participation in the study by accepting micro-task
jobs whenever they want or fits with their schedule. Our goal is that each
micro-task takes a short time to complete (less than 1 min) so we can attract
a larger group of participants. It must be noted that participants can quit at
any time and they will be compensated for their submitted work up until that
point. To ensure the quality of the responses, the AMT service allows us to
review and accept the responses from each participant individually, this
allows us to pinpoint workers with low-quality responses (e.g. disagreement on
more than 50 percent of the tasks with other participants) and ban them from
future participation. Even after being banned, the participants with low-
quality responses will be compensated for their previous accepted works.

###  B.3 Mechanical Turk Results

Asked annotators to go through 500 instances of PVLI. Each instance was
annotated by 3 randomly-selected workers from mainly English-speaking
countries: U.S., Canada, England, India, and Australia. We selected the
instances that are found correct by at least 2 annotators and use them as the
clean-test set. The final clean-test set, consists of 261 human-verified
instances with 151 allow labels. The inter-annotator agreement (Fleiss’ Kappa)
measure between our annotators is 0.78, showing good agreement among them.


# Empowering Multi-step Reasoning across Languages via Tree-of-Thoughts

Leonardo Ranaldi (†), Fabio Massimo Zanzotto(∗)
${(\dagger)}$ Idiap Research Institute, Martigny, Switzerland
<EMAIL_ADDRESS>
${(*)}$Human-Centric ART Group,
Dep. of Enterprise Engineering, University of Rome Tor Vergata, Italy
<EMAIL_ADDRESS>

###### Abstract

Chain-of-Thought (CoT) prompting empowers the reasoning abilities of Large
Language Models (LLMs), eliciting them to solve complex reasoning tasks step-
by-step. However, with the success of CoT methods, the ability to deliver
multi-step reasoning remains limited to English due to the imbalance in the
distribution of the pre-training data, making the other languages a barrier.

In this work, we propose a Cross-lingual multi-step reasoning approach, aiming
to align reasoning processes across different languages. In particular, our
method, through a Self-consistent Cross-lingual prompting mechanism inspired
by the Tree-of-Thoughts approach, delivers multi-step reasoning paths in
different languages that, during the steps, lead to the final solution. Our
experimental evaluations show that our method significantly outperforms
existing prompting methods, reducing the number of interactions and achieving
state-of-the-art performance.

Empowering Multi-step Reasoning across Languages via Tree-of-Thoughts

Leonardo Ranaldi (†), Fabio Massimo Zanzotto(∗) ${(\dagger)}$ Idiap Research
Institute, Martigny, Switzerland<EMAIL_ADDRESS>${(*)}$Human-Centric ART Group, Dep. of Enterprise Engineering, University of
Rome Tor Vergata, Italy<EMAIL_ADDRESS>

##  1 Introduction

Chain-of-Thought (CoT) prompting elicits Large Language Models (LLMs) to break
down a reasoning task towards a sequence of intermediate steps Wei et al.
(2022). Previous works have demonstrated that LLMs achieve impressive
performances in zero-shot learning scenarios without the need to modify the
model parameters during the training and testing process. In particular, by
appending to the prompt “Let’s think step by step!” Kojima et al. (2023) LLMs
with at least several billions of parameters, such as GPTs family OpenAI
(2023) or PaLM Chowdhery et al. (2022), deliver multi-step controlled
reasoning, achieving promising results across commonsense Bubeck et al.
(2023), symbolic and mathematical reasoning datasets Gaur and Saunshi (2023);
Liu et al. (2023).

Although the performances seem promising, they are only firmly established in
English. This poses a barrier to generalizing current CoT techniques to
different languages. Hence, despite the remarkable success of zero-shot CoT
techniques, the reasoning abilities of LLMs still struggle to generalize to
different languages. Shi et al. (2022) introduced the first multilingual
benchmark to assess LLMs’ mathematical reasoning abilities using prompts in
different languages. Qin et al. (2023) propose task-specific solver prompting,
using a succession of prompts, elicit the LLMs to understand questions and
deliver CoT answers in different languages. However, these strategies require
two-step prompts, which goes against the zero-shot approach.

In this paper, we propose Cross-lingual Tree-of-Thoughts (Cross-ToT), a method
for aligning Cross-lingual CoT reasoning across languages by proposing a
Cross-lingual Alignment prompt to elicit the model to deliver a Self-
consistent Chain-of-Thougt. Our method is inspired by the Tree-of-Thoughts
(ToT) prompting Yao et al. (2023) that delivers cross-lingual alignment via
single prompts in zero-shot scenarios. In particular, we use ToT-style
prompting to deliver the reasoning process in different languages that, step-
by-step, converge to a single final solution. The inherent insight is that as
the different paths of thought evolve, the relationships between the different
languages are inherently grasped via Self-consistent Chains-of-Thougt. This
leads to the target research questions, which are the focus of this paper:

RQ1: Are LLMs able to deliver Cross-lingual multi-step reasoned answers?

RQ2: Are the different paths of ToT evolving Self-correcting each other?

RQ3: What is the role of English in Cross-linguistic scenarios?

Figure 1: Our Cross-ToT elicits the LLM to generate step-by-step Cross-lingual
reasoning. Furthermore, different pathways are developed during these
reasoning steps. This mechanism develops the Chains-of-Thoughts in a Self-
consistent way, streaming with the different pathways.

To answer these questions, we propose Cross-ToT, a novel Cross-lingual
prompting strategy that aims to bridge the gap across different languages. In
particular, using the prompt shown in Figure 1, we elicit the model to deliver
different CoT reasoning steps in different languages that converge to the
final solution step-by-step. We test our method on GPT-3.5 and conduct an
extensive analysis using Multilingual Grade School Math (MGSM) benchmark Shi
et al. (2022) in six languages. Experimental results reveal that our method,
based on a single prompt, outperforms the baselines and achieves the SOTA
performance in four tasks on six. The main contributions of this work are
concluded as follows:

  * •

We introduce Cross-ToT, which is a novel Cross-lingual prompting mechanism
that stimulates the model to produce parallel CoT reasoning processes across
different languages;

  * •

We show that our Cross-ToT is Self-consistent and allows the integration of
reasoning paths between different languages;

  * •

Extensive evaluations on different languages demonstrate that our Cross-ToT
can effectively improve the performance of cross-lingual CoTs and achieve SOTA
performance.

  * •

Finally, we show that introducing English in our prompting technique plays a
beneficial role in improving downstream performance.

##  2 Cross-lingual Multi-step Reasoning

To elicit the multi-step reasoning abilities of LLMs in Cross-lingual
scenarios, we propose Cross-ToT that is a Cross-lingual Alignment Chain-of-
Thought as a solution. In particular, our method overcomes the Multi-lingual
and Cross-lingual approaches introduced in Section 2.1. In fact, our approach
elicits the LLMs to deliver Self-consistent Parallel Chain-of-Thougts,
introduced in Section 2.2.

Native-CoT in this example in Chinese
---
问题: 利亚有 32 块巧克力，她妹妹有 42 块。如果她们吃了 35 块，她们一共还剩下多少块？
答案: 让我们一步步思考

En-CoT
---
问题: 利亚有 32 块巧克力，她妹妹有 42 块。如果她们吃了 35 块，她们一共还剩下多少块？
Answer: Let’s think step by step

Translated-CoT (is the Native translated in En)
---
Question: Leah has 32 chocolates and her sister has 42. If they ate 35 pieces,
how many pieces do they have left?
Answer: Let’s think step by step

Table 1: Different types of input prompts in order to elicit Chain-of-Thought
reasoning process. Specifically, given a problem in Chinese, the following
prompts are Native-CoT and En-CoT, the original question in Chinese with
elicitation in Chinese and English; for Translated-CoT, the question is in
English and consequently a step-by-step solution in English.

###  2.1 Chain-of-Thought Across Languages

The Cross-lingual Alignment is a core challenge for cross-lingual transfer.
Shi et al. (2022) proposed a series of prompts to elicit models to generate
CoT answers in specific language Native-CoT, and in English En-CoT and
Translate-CoT (more detailed in Table 1).

Later, Qin et al. (2023) proposed a method based on two phases: Cross-lingual
alignment prompt and task-specific solver prompting. This approach uses two
separate steps, as shown in Table 2, in order to handle input and output in
different languages.

Cross-CoT First-Step
---
Please act as an expert in multi-lingual understanding in [Specific Language
$L_{s}$].
Question: [Given sentence $X$ in $L_{s}$]
Let’s understand the task in [Target Language $L_{t}$] step-by-step!

Cross-CoT Second-Step
---
After understanding, you should act as an expert in mathematics in [Language
$L_{t}$].
Let’s resolve the task you understand above step-by-step!

Table 2: Cross-lingual Prompt proposed in Qin et al. (2023). By setting an
input language and a target language, the prompt is divided into two phases:
in phase one, there is the alignment of the different languages, and in phase
two, there is the solving mechanism for the specific language.

Although this second approach overcomes the limitations of Shi et al. (2022)’s
work, the two-step prompting could be more laborious and challenging, and
there is no exchange of information during the multi-step reasoning process
between the different chains as the final outputs are estimated using a voting
heuristic.

###  2.2 Self-consistent Parallel Chain-of-Thougts

In our work, we propose Cross-ToT, a prompting method that can handle
different languages in a parallel way. Furthermore, through a mechanism
inspired by Tree-of-Thoughts prompting techniques Yao et al. (2023), our
method elicits the LLM to deliver the generation of the answer in a sequence
of intermediate steps that do not provide independent parallel answers but
deliver collaborative Self-consistent reasoned steps until arriving at a final
answer.

Our Proposal
---
Simulate the collaboration of six mathematicians answering a question in their
mother tongue: $L_{1}$, $L_{2}$, ... and $L_{6}$. They all start Step1 from a
separate thought process, step by step, each explaining their thought process.
Following Step1, each expert refines and develops their thought process by
comparing themselves with others. This process continues until a definitive
answer to the question is obtained.
Question: [Question in Language $L_{1}$]
Answer: [num].
Table 3: In our prompting approach, we elicit the model to produce multi-step
reasoning processes in different languages. We specifically prompt to start
from separate reasoning and collaborate step-by-step.

Our Cross-ToT shown in Table 3 elicits the LLM to generate different paths as
shown in Figure 1, achieving significant improvements in accuracy as discussed
in Section 4.

Model | de | zh | fr | ru | sw | es | Average
---|---|---|---|---|---|---|---
GPT-3 (text-davinci-002)* |  |  |  |  |  |  |
Direct Shi et al. (2022) | 14.8 | 18.0 | 16.8 | 12.4 | 8.8 | 17.2 | 14.67
Native-CoT Shi et al. (2022) | 36.0 | 40.0 | 37.6 | 28.4 | 11.2 | 40.4 | 32.27
En-CoT Shi et al. (2022) | 44.0 | 40.8 | 46.0 | 28.4 | 20.8 | 44.8 | 37.47
Translate-En Shi et al. (2022) | 46.4 | 47.2 | 46.4 | 48.8 | 37.6 | 51.6 | 46.33
GPT-3.5 (gpt-3.5-turbo) |  |  |  |  |  |  |
Direct Qin et al. (2023) | 56.0 | 60.0 | 62.0 | 62.0 | 48.0 | 61.2 | 58.20
Native-CoT Qin et al. (2023) | 70.0 | 59.6 | 64.4 | 62.4 | 54.0 | 70.4 | 63.47
En-CoT Qin et al. (2023) | 73.6 | 63.2 | 70.0 | 65.6 | 55.2 | 69.6 | 66.20
Translate-En Qin et al. (2023) | 75.6 | 71.6 | 72.4 | 72.8 | 69.6 | 74.4 | 72.73
Cross-CoT Qin et al. (2023) | 86.8 | 77.2 | 82.0 | 87.6 | 76.0 | 84.8 | 82.40
Cross-ToT | 87.6 | 83.5 | 84.3 | 86.5 | 75.4 | 86.2 | 83.91
Table 4: Accuracies (%) on MGSM using the "Direct" prompt, i.e., question and
answer in the original language; the "Native-CoT" prompt, i.e., question and
answer CoT in the original language; the "En-CoT" prompt specific language
question and answer CoT in English, the "Translate-En" prompt where the
specific input is translated into English and the answer accordingly is in
English. Moreover, Cross-CoT, as proposed by Qin et al. (2023), questions in a
specific language and answers in different languages. Finally, Cross-ToT is
explained in Section 2.2. (* the results stem from 6-shot input-prompt)

##  3 Experiments

###  3.1 Data

In order to observe the Cross-lingual abilities of LLMs, we used the GSM8K
Cobbe et al. (2021). In particular, we used the extension proposed by Shi et
al. (2022), i.e., Multilingual Grade School Math (MGSM). Initially, Cobbe et
al. (2021) proposed a benchmark of mathematical problems in English in GSM8K.
Each example has the following structure: a mathematical problem in natural
language and a target answer in Arabic number. Shi et al. (2022), in their
contribution, i.e., MGSM, selected the first 250 examples from the official
list of examples in GSM8K and translated them manually into 11 different
languages, maintaining the structure of the input and output.

In our experiments, we propose an analysis of six different versions. The
selected languages are Chinese (zh), French (fr), Spanish (es), German (de),
Swedish (sw), and Russian (ru). Furthermore, as a further experiment, we test
the introduction of a seventh language, which should be the strongest, namely
English (see the prompt example in Table 6.). We plan to extend the
experiments to the missing languages in future developments.

###  3.2 Experimental Setup

In order to conduct our study on robust models and have a term of comparison
with the work proposed in Shi et al. (2022); Qin et al. (2023), we use
GPT-3.5; however, in future developments, we plan to scale the method to
different models. We then systematically defined the input prompt in Table 3
by modifying the appropriate languages with $L_{1}$, $L_{2}$, etc. as showed
in Table 6 for the German 111Although we do not observe perceptible changes in
the order of languages present in the input prompt, we set as a first the
language-related subset of the benchmark.

Following Wei et al. (2022); Kojima et al. (2023), we evaluate performance
using the accuracy score. In particular, we compute the string matching
between the final answers (see Figure 1 where the final outputs have the form
of Answer:[num]) and the target values. The top-p parameter is set to 1 in all
processes. We select the Prompting temperature [0, 1].

##  4 Results & Discussion

Mechanisms for delivering multistep-reasoned answers across languages can be
empowered via Cross-ToT that align languages’ Chain-of-Thoughts (CoT). Our
approach based on a Tree-of-Thoughts-inspired prompting mechanism (see Figure
1) outperforms state-of-the-art prompting techniques, as shown in Table 4. In
particular, Cross-ToT elicit LLMs to produce different reasoning pathways that
share the ”Thoughts” during the steps and, at the same time, promote Self-
correction of mistaken paths. In fact, during the steps of the CoT,
information is swapped between the paths. This interaction delivers Self-
consistent paths. Furthermore, in the prompt, we exeplified that the different
paths must arrive at a shared and, consequently, unique by sharing the
”thought process” (see the prompt in Table 3).

Figure 2: Accuracies (%) on MGSM using ”Cross-ToT” and ”Cross-ToT \+ English”
as proposed in Table 6.

In particular, our approach outperforms the approaches proposed in Shi et al.
(2022) that yet surpassed by the Cross-CoT proposed by Qin et al. (2023). In
particular, our approach outperforms in four languages out of six (i.e.,
German, Chinese, French, Spanish). In particular, we observe a marked
improvement for Chinese 4 points and improvements of about 2 points for other
languages (see Table 4). Although there have been stable improvements for
Russian and Swedish, we will better study the reasoning branches for these two
languages in future developments.

In conclusion, in order to observe the impact of English in our prompting
approach, we performed the setting proposed in Section 3.2 by modifying the
prompt as in Table 6. From the results obtained in Figure 2, we can observe
that the English path contributed significantly to a sustained increase in
performance. This result suggests that the presence of one robust path, in
this case, the English path, may influence the others in the final reasoning
process. Indeed, assuming that the production of the intermediate steps is
self-consistent, i.e., the paths do not disagree with each other, the
additional language seems to influence performance positively.

##  5 Limitations & Future Works

Our proposal proved tangible in terms of performance. However, as mentioned in
Section 4, to confirm the results obtained in future developments, we plan to
test in a systematic way our approach on further benchmarks.

In addition, we would like to scale the approach to different LLMs to observe
whether the actual trend in performance is related to the LLM taken into
consideration in our work (i.e., GPT-3.5) or whether there are other factors
as the pre-training languages or the refinement methods.

Finally, we plan to conduct more detailed analysis and studies in order to:

  * •

investigate the reasoning process and the average number of intermediate
steps;

  * •

analyze of pathways (and thus languages) that lead to a target solution;

  * •

estimate number of paths (minimum or maximum languages supported) and
consequently language configuration via an ablation study.

##  6 Related Work

Large Language Models (LLMs) with billions of parameters demonstrate in-
context learning and few-shot learning abilities Brown et al. (2020); Wei et
al. (2022); Min et al. (2022) to guide LLMs to generate desired task
responses, marking the advent of the prompting era and surpassing the age of
the intermediate steps in algorithmic and structured reasoning Roy and Roth
(2015); Ling et al. (2017). Nevertheless, early works challenged the efficacy
of few-shot techniques for empowering the prompting phase and downstream
performances. In particular, Yao et al. (2023) refined the original idea of
Chain-of-Thought (CoT) Wei et al. (2022) by considering various reasoning
paths as well known as Tree-of-Thought.

The traditional and derivated CoT mechanisms have achieved considerable
success but are limited to generating answers within a single language (i.e.,
English). Shi et al. (2022) proposed a multilingual evaluation that Qin et al.
(2023) extended to cross-lingual scenarios. In particular, Qin et al. (2023)
proposed a prompt mechanism to handle requests in any language and generate
CoT specifically in English. Although the mechanism achieves state-of-the-art
cross-linguistic reasoning steps, a series of cascading prompts is supported
far away from the zero-shot chain-of-thought concept.

In our work, we propose a method of CoT reasoning inspired. Specifically, we
elicit the cross-lingual generation of a series of parallel Cross-lingual
reasoning paths using a single prompt. In fact, our method is inspired by the
Tree-of-Thoughts approach proposed by Yao et al. (2023). Hence, in a different
way from previous approaches, our technique generates shared parallel
reasoning paths that share the ”thoughts process” delivering Self-consistent
answers and reducing reasoning steps.

##  7 Conclusion

Chain-of-Thought is an outstanding prompting technique. However, due to the
imbalance of languages in pre-training data, it does not always produce robust
results, as demonstrated in Shi et al. (2022). Different state-of-the-art
works have proposed multilingual and cross-lingual techniques to align
performances obtained in different languages. However, the present approaches
are limited to handling one language at a time or proposing multiple prompting
stages, making them difficult to manage. In this paper, we propose Cross-ToT
which is a prompting technique to elicit multi-step reasoning abilities in
Cross-lingual scenarios. Hence, we elicit models to deliver answers in a Self-
consistent way, collaborating to the final answer. We have shown the
functionality of our Cross-ToT through performance improvements obtained on a
multilingual benchmark of mathematical problems.

In future works, we plan to extend the studies by scaling the approach to
different models and testing more benchmarks and languages. Our contribution
aims to propose more robust models that can break down issues arising from
language barriers and provide more reliable results.

## References

  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.
  * Bubeck et al. (2023) Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. 2023.  Sparks of artificial general intelligence: Early experiments with gpt-4.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022.  Palm: Scaling language modeling with pathways.
  * Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021.  Training verifiers to solve math word problems.
  * Gaur and Saunshi (2023) Vedant Gaur and Nikunj Saunshi. 2023.  Reasoning in large language models through symbolic math word problems.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 5889–5903, Toronto, Canada. Association for Computational Linguistics.
  * Kojima et al. (2023) Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2023.  Large language models are zero-shot reasoners.
  * Ling et al. (2017) Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. 2017.  Program induction by rationale generation: Learning to solve and explain algebraic word problems.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 158–167, Vancouver, Canada. Association for Computational Linguistics.
  * Liu et al. (2023) Hanmeng Liu, Ruoxi Ning, Zhiyang Teng, Jian Liu, Qiji Zhou, and Yue Zhang. 2023.  Evaluating the logical reasoning ability of chatgpt and gpt-4.
  * Min et al. (2022) Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022.  Rethinking the role of demonstrations: What makes in-context learning work? In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 11048–11064, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * OpenAI (2023) OpenAI. 2023.  Gpt-4 technical report.
  * Qin et al. (2023) Libo Qin, Qiguang Chen, Fuxuan Wei, Shijue Huang, and Wanxiang Che. 2023.  Cross-lingual prompting: Improving zero-shot chain-of-thought reasoning across languages.
  * Roy and Roth (2015) Subhro Roy and Dan Roth. 2015.  Solving general arithmetic word problems.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 1743–1752, Lisbon, Portugal. Association for Computational Linguistics.
  * Shi et al. (2022) Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. 2022.  Language models are multilingual chain-of-thought reasoners.
  * Wei et al. (2022) Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022.  Emergent abilities of large language models.
  * Yao et al. (2023) Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. 2023.  Tree of thoughts: Deliberate problem solving with large language models.

##  Appendix A Cross-ToT with German as $L_{1}$

---

Table 5: Our Cross-ToT with a German Question and Tree-of-Thought answer.

##  Appendix B Cross-ToT Empowered with English

---

Table 6: We test our prompting approach empowered with English. We
specifically reproduce the same prompt by adding additional language and
additional path.