# CARETS: A Consistency And Robustness Evaluative Test Suite for VQA

Carlos E. Jimenez Olga Russakovsky Karthik Narasimhan
Princeton University
<EMAIL_ADDRESS>
###### Abstract

We introduce CARETS, a systematic test suite to measure consistency and
robustness of modern VQA models through a series of six fine-grained
capability tests. In contrast to existing VQA test sets, CARETS features
balanced question generation to create pairs of instances to test models, with
each pair focusing on a specific capability such as rephrasing, logical
symmetry or image obfuscation. We evaluate six modern VQA systems on CARETS
and identify several actionable weaknesses in model comprehension, especially
with concepts such as negation, disjunction, or hypernym invariance.
Interestingly, even the most sophisticated models are sensitive to aspects
such as swapping the order of terms in a conjunction or changing the number of
answer choices mentioned in the question. We release CARETS to be used as an
extensible tool for evaluating multi-modal model robustness.111Source code,
data, and additional resources may be found at https://github.com/princeton-
nlp/CARETS

##  1 Introduction

Figure 1: Our consistency and robustness test suite (CARETS) consists of six
tests, corresponding to six identified phenomena that VQA models should be
robust to. Tests evaluate models’ predictions between pairs of instances
(changing $q_{1}\rightarrow q_{2}$ or changing $I_{1}\rightarrow I_{2}$).
rephrase-inv for robustness to simple rephrasings; order-inv for robustness to
changed argument order in lists, conjunctions, and disjunctions; ontological-
inv for understanding of ontology; visual-inv for robustness to visual context
perturbations; negation-dir for robustness to negative clauses; antonym-dir
for understanding of mutually exclusive attributes.

The task of visual question answering integrates the domains of computer
vision and NLP by probing models’ understanding of images through natural
language queries. After the introduction of the Visual Question Answering
(VQA) benchmark Antol et al. (2015), subsequent work identified the presence
of several superficial correlations and other weaknesses latent in the VQA
question gathering process Goyal et al. (2017); Agrawal et al. (2018), which
lead to potentially optimistic evaluations when considering accuracy alone.
More recently developed benchmarks Hudson and Manning (2019); Goyal et al.
(2017); Agrawal et al. (2018) explicitly avoid these weaknesses by introducing
question, answer, and image balancing, or distributional shifts. While these
efforts provide more difficult benchmarks, a thorough evaluation of model
capabilities requires a deeper and more detailed approach.

To this end, we introduce CARETS – a Consistency And Robustness Evaluative
Test Suite for visual question answering. Inspired by recent work in NLP that
generates ‘unit tests’ for models Ribeiro et al. (2020b), CARETS contains
systematically generated VQA tests that evaluate six different capabilities
that any VQA model should handle – robustness to question rephrasings,
ontological reasoning, symmetrical logic, visual perturbations, question
negation, and attribute antonymy. Each test point in CARETS consists of a pair
of instances which are small but strategic variations of each other either
visually or in the question’s text. This allows us to conduct fine-grained
capability evaluations beyond just measuring high-level accuracy scores.

Across tests, we generate over 190,000 instance pairs in total using a
programmatic approach that fills in templates (from nearly 200 templates in
total) using ground-truth scene graphs Krishna et al. (2017) from the GQA
Hudson and Manning (2019) validation split. We then evaluate six modern VQA
models on each test using metrics of overall accuracy, self-consistency and
comprehensive accuracy. Self-consistency measures models’ ability to maintain
their answer across question variations, while comprehensive accuracy
estimates their ability to answer all instance variants correctly.

Our experiments reveal several interesting findings: (1) most modern VQA
systems achieve only middling self-consistency ($\sim$60-80%) which is further
not always correlated with their accuracy, (2) all models struggle to
comprehend the concept of negation (self-consistency of 18-28% and
comprehensive accuracy $<$17%), and (3) even simple perturbations like
changing the order of choices in the question text can induce a substantial
drop (10-15%) in performance. Moreover, even state-of-the-art models like
LXMERT Tan and Bansal (2020) are highly sensitive to the type of questions
(binary vs multi-choice) and the number of choices provided. These results
reveal several shortcomings in modern VQA systems and hint at potential areas
for future improvement. Going beyond our current discoveries, CARETS is an
extensible framework and may be easily extended by adding new capability tests
for fine-grained evaluation of future models.

##  2 Related Work

#### VQA evaluation.

The textual, visual, and answer biases discovered in the VQA dataset Antol et
al. (2015) spurred on recent work seeking to improve model evaluation for the
task by eliminating these biases Goyal et al. (2017), introducing
distributional shifts Agrawal et al. (2018), requiring model explanations Li
et al. (2018), thoroughly analyzing biases in datasets and models Manjunatha
et al. (2019), or evaluating on different recognition subtasks Kafle and Kanan
(2017). While debiased and challenging benchmarks are important, their focus
on accuracy as the sole evaluation metric leaves much to be desired Ribeiro et
al. (2020a); Kervadec et al. (2020). In contrast, our testbed provides
question or image pairs that compares models’ predictions _between_ questions;
measuring their accuracy, consistency, and robustness to a variety of text and
image perturbations.

#### Synthetic Dataset Generation for VQA.

One way in which we can generate diverse and balanced datasets is to generate
them synthetically, as is done by Johnson et al. (2015); Zhang et al. (2016);
Hudson and Manning (2019). Synthetically generating questions, images, or
both, allows fine control over the distribution of questions, answers, and
images. Additionally for our case, synthetic generation allows us to control
not just the particular semantics of one question, but also how one question
relates to another question in a precisely defined way (e.g. one question is a
negation of another) while also remaining relevant and grounded in the image.
As both the CLEVR Johnson et al. (2015) and GQA Hudson and Manning (2019)
datasets use image scene graphs for question and label generation, they
contain questions combining a variety of required capabilities, including
compositionality. We feature real-world images with synthetically generated
questions as well, but in contrast to GQA, our evaluation has instance pairs
to systematically test a focused set of capabilities, showing that models may
still struggle with simpler, non-compositional questions.

#### Consistency as Model Comprehension.

Some recent work has sought to evaluate models using consistency and other
metrics Hudson and Manning (2019); Shah et al. (2019); Ribeiro et al. (2020a);
Selvaraju et al. (2020); Bitton et al. (2021). These evaluations often
evaluate consistency through question entailment and implication, or simply
contrasting examples in the case of Bitton et al. (2021). While we consider
such methods important for evaluating model comprehension, they often combine
question types and capabilities, changing the kind of expected answer, or
evaluating consistency on a tree or set of entailed questions. While models
would ideally be consistent and robust for these more complex types of tests,
our approach reveals that models can fail even on simple implications.

##  3 Fine-grained capability tests

Our goal is to provide a testbed for fine-grained evaluation of VQA models’
capabilities. To do so, we generate multiple tests, each corresponding to a
specific model capability. In contrast to standard VQA benchmarks Antol et al.
(2015); Goyal et al. (2017); Hudson and Manning (2019), our test sets consist
of a _pair_ of _original_ and _perturbed_ instances
$\langle(I_{1},q_{1},a_{1}),(I_{2},q_{2},a_{2})\rangle$, each with an image, a
question and an answer. The two instances within a pair differ from each other
in a minimal yet carefully constructed way to hone in on a particular
capability, similar to BLiMP Warstadt et al. (2020). A model is then evaluated
on its overall _accuracy_ , _self-consistency_ (ability to produce consistent,
even if wrong, answers to both instances within a pair), and _comprehensive
accuracy_ (ability to answer consistently and correctly for an instance pair).

Overall, we create six tests corresponding to key capabilities. We provide a
high-level description of each test here and describe generation details in
Section 4. First, we create four _invariance_ tests222Reusing terminology from
Ribeiro et al. (2020b). that use variations of the question phrasing and
expect the model to produce the _same_ answer to both questions within an
instance pair:

  1. 1.

Rephrasing invariance (rephrase-inv) measures the model’s understanding of
minor, meaning-preserving textual changes, e.g.: “What color is the bottle on
the shelf, white or blue?” and “Does the color of the bottle on the shelf seem
more white or blue?”

  2. 2.

Ontological invariance (ontological-inv) measures understanding of ontology,
e.g. changing a hyponym in: “Do you see a green jacket?” to a hypernym “Do you
see any green _clothes_?”

  3. 3.

Order invariance (order-inv) measures understanding of logically equivalent
questions containing different argument orders, e.g.: “Is the black vehicle a
van or a truck?” and “Is the black vehicle a _truck_ or a _van_?”

  4. 4.

Visual obfuscation invariance (visual-inv) measures the model’s answering
ability when parts of the image not directly relevant to the visual question
are removed. Specifically, we explore blurring, masking and cropping
techniques to modify the image.

We also create _directional expectation_ tests to measure model behavior on
instance pairs where the answer is expected to _change_ :

  1. 5.

Attribute antonym directional expectation (antonym-dir) measures the model’s
understanding of antonyms, e.g., “Do you think that the wood table is short?”
and “Do you think that the wood table is _long_?”

  2. 6.

Negation directional expectation (negation-dir) measures a model’s grasp of
negation, e.g., “ Are there any apples in this picture?” and “Are there _no_
apples in this picture?

##  4 Dataset generation

Each of the six test datasets start with the generation of ‘original’,
unperturbed instances $(I_{1},q_{1},a_{1})$ (Section 4.1). Then, for each such
instance, we generate a variation $(I_{2},q_{2},a_{2})$ by either perturbing
the original question $q_{1}$ or image $I_{1}$ (Section 4.2). Further, each
test set is composed of a diverse set of questions. These may broadly be
grouped into verification (or binary) questions, with expected answers being
either yes or no, and multi-choice questions, with expected answers derived
from a list of objects or attributes provided in the question.

###  4.1 Original instance generation

Questions for each test are generated from question templates (examples for
each are provided in Appendix A.1) grouped into the following types.

  * Q1:

Object verification (54 templates): e.g., “Is there a <obj> in the image?”

  * Q2:

Conjunctive verification (18 templates): e.g., “Is there a <obj1> and a <obj2>
in the image?”

  * Q3:

Disjunctive verification (18 templates): e.g., “Is there a <obj1> or a <obj2>
in the image?”

  * Q4:

Attribute verification (25 templates): e.g., “Is the <obj> in the image
<attr>?”

  * Q5:

Object multi-choice (25 templates): e.g., “Is the <obj-class>, <choices>?”

  * Q6:

Attribute multi-choice (39 templates): e.g., “What sort of <attr-class> is the
<obj>, <choices>?”

  * Q7:

Action multi-choice (28 templates): e.g., “What is the <action-class> that the
<obj> doing, <choices>?”

Words in typewriter font represent template arguments. Generally, each <obj>
argument can be filled by a singular object (“cup”) or an attribute+object
(“red cup”) while <attr> arguments are filled with singular attributes
(“shiny”). For object verification (Q1), attribute verification (Q4),
attribute multi-choice (Q6), and action multi-choice (Q7) questions, some
templates let <obj> arguments be filled with an object related to another
object (e.g. “red cup on the table”); this type is excluded from conjunctive
verification (Q2) and disjunctive verification (Q3) questions to prevent the
generation of verbose questions.

For the multi-choice questions (Q5, Q6, Q7), <choices> are replaced with a
list of 2 or 3 singular objects, attributes, or actions respectively (e.g.
“cow or horse” or “wood, metal, or plastic”). The <obj-class> argument is
filled with a hypernym of all object choices and always appears with either an
attribute or a related object (“black animal”, “animal eating grass”). The
<attr-class> argument is filled with the attribute category of all attribute
choices (e.g. “material”). Finally, the <action-class> argument is filled with
the action category of all action choices (e.g. “sport”).

#### Question argument generation.

The question arguments are generated using images from the validation split of
the GQA dataset Hudson and Manning (2019) which contains 10,696 images
manually annotated with 1,536 different objects and 603 different object
attributes.

To generate questions, we sample objects and attributes directly from an
image’s scene graph annotation to populate a question type’s arguments. For
binary question types this results in questions with solely affirmative
answers. To produce an answer balanced dataset, we run a second stage of
question argument generation for binary questions to generate _plausible_
negative questions with false objects or attributes. We sample false objects
from a distribution conditioned on an image’s objects, and optionally sample
object attributes from a distribution conditioned on the chosen object.

For <choices> arguments, false choices are again generated from a distribution
conditioned on the object’s hypernym for Q5 questions, the attribute category
for Q6 questions, or the action category for the Q7 questions. We additionally
ensure that the generated choices are mutually exclusive (e.g. “tan or beige”
would be an invalid generation). To get more diverse multi-choice questions,
we first generate a large pool of question candidates, and then select only a
small number of questions sampled from this pool with sample probabilities
inversely proportional to the count of the questions’ hypernym, attribute
class, or action class, and the count of the generated answer.

#### Question argument refinement.

To improve the reliability of generated questions, we apply a variety of
checks and constraints. For example, when sampling false objects from the
conditional distribution, we filter out all objects (and their hypernyms333We
use an ontology with hypernym paths generated with WordNet Miller (1995). We
manually review and revise the default synset annotations from Visual Genome
Krishna et al. (2017) for the entire object vocabulary, and compare to a
sample of annotated images to ensure general validity.) present in the scene
graph in order to guarantee that the sampled object is truly not present. We
also filter out question arguments that are not included in the image scene
graph but are sub-parts of objects that are annotated (e.g., “tire” when a
“car” is annotated). Finally, we enforce various logical constraints on
question arguments to prevent trivial or malformed questions. For example, for
conjunctive and disjunctive questions (Q2, Q3), we apply a _hypernym
exclusion_ constraint to prevent questions like “Is there a black cat and a
black animal in the image?”.

###  4.2 Perturbed pair generation

We now describe our procedure for creating perturbed instances
$(I_{2},q_{2},a_{2})$ for the six tests. In all tests except visual
obfuscation, the image remains unchanged, i.e. $I_{2}=I_{1}$.

#### (a) Rephrasing invariance.

Since each original question $q_{1}$ was generated using a text template, we
simply use a different template of the same type to generate a valid
rephrasing $q_{2}$. The image and answer remain the same, i.e.
$I_{1}=I_{2},a_{1}=a_{2}$ and the model is expected to be invariant to this
rephrasing. We apply this to Q1, Q2, Q3, Q5, Q6 and Q7.

#### (b) Ontological invariance.

Here, we use object verification questions (Q1) only and perform two types of
transformations. For positive questions (i.e. $a_{1}$ = _yes_), we filter
question arguments to only include objects that are valid hyponyms (using
WordNet again) and use those to generate a perturbed question $q_{2}$ by
changing the hyponym to a hypernym. For example, $q_{1}$ = “Do you see a
jogging _woman_?” with $a_{1}$ = _yes_ is paired with $q_{2}$ = “Do you see a
jogging _person_?” containing a hypernym. Similarly, for negative questions
($a_{1}$ = _no_), we filter question arguments to only include valid hypernyms
and generate a $q_{2}$: thus for example, $q_{1}$ = “Do you see a jogging
_person_ ” with $a_{1}$ = _no_ is paired with $q_{2}$ = “Do you see a jogging
_woman_?” containing a hyponym, $a_{2}$ = _no_ also.

#### (c) Order invariance.

Order invariance tests apply to conjunctive verification, disjunctive
verification, and all multi-choice question types; models are expected to be
invariant to the logical order of arguments. We perturb conjunctive
verification and disjunctive verification questions by swapping the questions’
first and second arguments (<obj1>, <obj2>). For multi-choice question types,
we perturb instances by generating the <choices> argument with different
orders. The answer remains the same in both cases by construction.

#### (d) Visual obfuscation invariance.

For this test, we let $q_{1}=q_{2}$ and $a_{1}=a_{2}$ but generate a perturbed
image $I_{2}$ by obscuring parts of $I_{1}$ that are irrelevant to the
question at hand using bounding box annotations from Visual Genome Krishna et
al. (2017). For all true objects in a question, we consider the bounding boxes
around these object(s) to be the _foreground_ and all other pixels in the
image to be the _background_. For negative verification questions asking about
object(s) not present in the image, we select one (or two) random object
bounding box(es) as the foreground and consider everything else to be the
background, since focusing on any image region should not affect the model’s
answer.444We choose $32\times 32$ as a minimum bounding box size, shown to be
reliably recognized by humans Torralba et al. (2008).  We then apply five
types of perturbations to obscure the background: (i-iii) Gaussian blurring
using the soft masking method of Yang et al. (2021) with light ($\sigma=3$),
medium ($\sigma=6$), or heavy ($\sigma=9$) blurring, (iv) Masking with the
channel-wise averaged pixel value from the GQA Hudson and Manning (2019)
training dataset, entirely obscuring the context, and (v) Cropping to the
smallest rectangular region including the foreground. Example images are shown
in Appendix A.2.

#### (e) Negation directional expectation.

For the negation directional test, we use object verification, conjunctive
verification, and disjunctive verification questions. Each question $q_{1}$ is
perturbed by substituting the original’s text template with a paired negated
text template to create $q_{2}$. Since each perturbed question represents the
negation of the original, the expected answers $a_{1}\neq a_{2}$.

#### (f) Attribute antonym directional expectation.

We perturb the generated attribute verification questions by changing the
<attr> question argument to its antonym using WordNet. All attribute antonym
relations are manually curated to remove unintuitive examples; questions with
arguments without a valid antonym are discarded. The original and perturbed
questions of a pair have opposite answers $a_{1}\neq a_{2}$.

##  5 Experimental Setup

###  5.1 Human baseline

To assess the quality, difficulty and validity of the generated tests, we
sample 100 question pairs (200 questions) from each question type for the 6
tests and procure 5 annotations per question from workers on Amazon Mechanical
Turk. Workers are vetted for a 97% approval rating and minimum of 500
completed HITs. Workers take $\sim 2$ minutes per task on average and are
compensated $\$0.50$ per task and thus $\sim\$15$ per hour. Each HIT include
24 questions total, including 4 verification questions555Tasks are also
interspersed with binary or multi-choice “gold-standard” questions with
perfect annotator agreement from the VQA dataset Antol et al. (2015) which are
required to be answered correctly before a HIT can be submitted. Workers are
given the opportunity to correct answers before submitting if a “gold-
standard” question has been answered incorrectly., and typically include a
variety of question types from each of our tests.

#### Human agreement.

In addition to “yes” and “no” for binary questions and the appropriate choices
for multi-choice questions, all questions offer an ambiguous option. Human
answers are the majority vote among the 5 workers; questions failing to reach
majority or with ambiguous as the majority are always counted against
accuracy. This process is inspired by the human evaluations of implied
question pairs in Ribeiro et al. (2020a). We report both human and model
performance in Section 6.

###  5.2 Evaluated models

We evaluate six recent models on our tests, and compare them to human
accuracy. Models are trained on the GQA Hudson and Manning (2019) balanced
training split (using hyperparameters suggested from the original papers). All
models, except LXMERT666For LXMERT we use the authors’ open source repository
at https://github.com/airsplay/lxmert, are trained and finetuned using the MMF
Singh et al. (2020) library and region of interest (RoI) features from Faster
R-CNN Ren et al. (2015) with a ResNeXt-152 Xie et al. (2017) backbone pre-
trained on the Visual Genome Krishna et al. (2017) dataset for object-based
models. More details are provided in Appendix A.3.

#### Model initialization and pre-training.

Of the six models evaluated, a defining characteristic of each model relates
to their initializations, image-encoding choice, and the use of multi-modal
pre-training. Our most basic model (CNN+LSTM) is randomly initialized and uses
no pre-trained components; however, GloVe Pennington et al. (2014) word
embeddings are used for representing tokens. Another class of models use pre-
trained image encoders to extract object features from images. Of our models,
BAN Kim et al. (2018) is randomly initialized prior to training but ingests
pre-trained Faster R-CNN features which should provide the model with enhanced
visual capabilities over the CNN+LSTM model. MultiModal BiTransformer (MMBT)
Kiela et al. (2019), uses similar pre-trained image features as BAN but is
further initialized with pre-trained BERT Devlin et al. (2019) weights prior
to training on GQA. The last class of models are multi-modal pre-trained
models; those that use pre-trained image features and are pre-trained on
multi-modal tasks, such as image-based masked language modeling. Models in
this class include LXMERT Tan and Bansal (2020), ViLBERT Lu et al. (2019), and
VisualBERT Li et al. (2019). Similar to MMBT, ViLBERT and VisualBERT are also
initialized with pre-trained weights from BERT.

###  5.3 Metrics

Accuracy (acc). On our test datasets with $K$ paired instances, we define
accuracy as:

$\frac{1}{2K}\sum_{i=1}^{K}\mathbbm{1}[\hat{a}_{1}^{i}=a_{1}^{i}]+\mathbbm{1}[\hat{a}_{2}^{i}=a_{2}^{i}]$

where the model answers $\hat{a}_{a}^{i},\hat{a}_{2}^{i}$ on the original and
perturbed questions respectively are compared to the ground truth answers
$a_{1}^{i},a_{2}^{i}$.
Self-consistency (cons). We measure self-consistency of the model predictions
across the original and perturbed questions as

$\displaystyle\frac{1}{K}\sum_{i=1}^{K}\left\\{\begin{array}[]{ll}\mathbbm{1}[\hat{a}_{1}^{i}=\hat{a}_{2}^{i}]&\text{on
invariance tests}\\\ \mathbbm{1}[\hat{a}_{1}^{i}\neq\hat{a}_{2}^{i}]&\text{on
directional exp. tests}\end{array}\right.$

Note that these metrics only measure the internal consistency of the model and
do not include the ground truth answers $a_{1}^{i},a_{2}^{i}$.
Comprehensive accuracy (c-acc). We define comprehensive accuracy as:

$\frac{1}{K}\sum_{i=1}^{K}\mathbbm{1}[\hat{a}_{1}^{i}=a_{1}^{i}\land\hat{a}_{2}^{i}=a_{2}^{i}]$

measuring whether model predictions are both accurate and self-consistent
across perturbations.

##  6 Results

Figure 2: acc and c-acc across all six tests. All models perform worse than
humans and exhibit consistent drops in c-acc from acc, especially struggling
on negation-dir and antonym-dir tests. Best viewed in color.

#### (R1) Modern VQA models are not robust to invariance and directionality
tests.

Fig 2 details the performance of various models under our suite of tests. Each
bar in the figure shows both acc and c-acc for the model, with the arrow
representing the gap between the two. We first observe that all models achieve
significantly lower performance (at least a 8% drop) compared to humans
(grey). Even simple tests such as rephrase-inv (Figure 2(a)) prove to be quite
challenging, with models managing $<$ 68% acc compared to humans’ 86%. On
tests like negation-dir (Figure 2(e)), models only get about 50% accuracy,
substantially worse than human scores of 78%.

Moreover, c-acc is considerably lower than acc across the board, with as much
as a 35% gap on negation-dir and 14.5% on antonym-dir tests, even for a state-
of-the-art model like LXMERT.777Other modern systems like ViLBERT and
VisualBERT have even worse c-acc. Even though this gap is smaller on other
tests like rephrase-inv or visual-inv, the performance drop is still at least
6-7% in most cases. This means that models are not invariant to textual
rephrasings of questions and do not have a strong grasp of concepts like
attributes and negation, despite negation of attributes appearing in the GQA
training dataset.

| cons
---|---
| Reph. | Order | Onto. | Vis. | Neg. | Attr.
BAN | 82.3 | 65.2 | 77.4 | 82.0 | 56.8 | 19.3
CNN+LSTM | 68.9 | 62.6 | 73.4 | 80.0 | 10.5 | 17.8
LXMERT | 83.8 | 71.0 | 83.5 | 78.6 | 71.1 | 28.1
MMBT | 81.7 | 58.6 | 70.5 | 81.4 | 54.6 | 23.1
ViLBERT | 86.4 | 66.2 | 76.5 | 83.4 | 55.3 | 18.7
VisualBERT | 83.6 | 60.8 | 74.1 | 83.8 | 59.8 | 18.3
1-7 Human | 96.6 | 98.8 | 94.0 | 89.0 | 89.0 | 88.5

Table 1: Self-consistency scores for all tests. Models only achieve middling
self-consistency, compared to >89% human self-consistency.

#### (R2) VQA systems are not self-consistent in their predictions.

Table 6 shows the self-consistency scores for all models under our different
tests. While humans achieve cons > 88% in all the tests, VQA models are much
worse – at least 6% lower cons in all cases, with the best performing model
(LXMERT) still 26% lower than human performance on average across tests and
models. Scores are especially low on the directional tests (antonym and
negation), which means that models are confused in their decisions simply with
the addition of negation words – this hints at issues of overfitting to
spurious feature without understanding the presence or absence of specific
concepts, corroborating the findings of Bitton et al. (2021). Interestingly,
the best performing model (LXMERT) is not always the most consistent.
Furthermore, there is no single model that is the most self-consistent, with
LXMERT, ViLBERT and VisualBERT each returning the highest consistency scores
on different tests.

Table 2: Ontological invariance results breakdown: original vs perturbed
accuracy, and hyponym vs hypernym self-consistency.

| Orig. | Pert. | Hyper- | Hypo-
---|---|---|---|---
BAN | 79 | 76 | 75 | 79
CNN+LSTM | 49 | 56 | 75 | 72
LXMERT | 89 | 82 | 80 | 87
MMBT | 82 | 66 | 61 | 80
ViLBERT | 85 | 72 | 69 | 84
Visual BERT | 81 | 69 | 68 | 80
1-5 Human | 96 | 96 | 91 | 96

#### (R3) Models are more robust to hyponym than hypernym variations.

Breaking out the results on the ontological invariance test (Figure 2 (c)) in
the last two columns of Table 6, we see that self-consistency is higher on the
hyponym perturbations (on negative answer questions) than on hypernym
perturbations (positive questions); this effect is particularly noticeable for
MMBT and ViLBERT with a 19% and 15% difference, respectively. Thus, when an
object is not detected in an image its hyponym elicits a negative response as
expected; however when an object (like “steak”) is detected, the hypernym
question (“Is there any _meat_ in the image?”) may trip the model to generate
a negative response. This points to the need for more structured, hierarchical
grounding of concepts in these models.

#### (R4) Models perform better on conjunctive rather than disjunctive tests.

From Table 3, we note that models generally have higher accuracy on
conjunctive rather than disjunctive tests, with the largest discrepancy for
LXMERT at 81% accuracy on conjunctive tests vs only 62% on disjunctive. Many
models seem to exhibit a strong positive bias for disjunctive questions,
suggesting they may just be short-cutting to answering ‘yes’ for disjunctive
questions. LXMERT also seems to frequently confuse disjunctive questions for
an open-ended or multi-choice question.

Table 3: Conjunctive (Con) vs disjunctive (Dis) comprehensive accuracy on order-inv, along with a breakdown of response rates for yes (Y), no (N) and other than yes/no (O). | Conjunction | Disjunction
---|---|---
| acc | Y | N | O | acc | Y | N | O
BAN | 52 | 53 | 47 | 0 | 52 | 71 | 28 | 0
CNN+LSTM | 39 | 53 | 47 | 0 | 35 | 65 | 35 | 0
LXMERT | 78 | 49 | 51 | 0 | 56 | 59 | 32 | 9
MMBT | 56 | 50 | 50 | 0 | 55 | 63 | 34 | 2
ViLBERT | 58 | 54 | 46 | 0 | 56 | 79 | 21 | 0
VisualBERT | 59 | 49 | 51 | 0 | 57 | 70 | 30 | 0
Table 4: Comparison of LXMERT’s behavior for questions with binary answer types and multi-choice answers. Metrics are reported as averages over rephrasing invariance and order invariance tests. Human acc reports human performance across test sets. | acc | cons | c-acc |  | Human
---
acc
Binary | 74.5 | 89.3 | 69.8 | 82.8
Multi-choice | 59.0 | 69.7 | 47.3 | 83.6
2-choice | 62.4 | 70.8 | 49.9 | 83.4
3-choice | 55.4 | 68.4 | 44.5 | 83.9

#### (R5) Models are sensitive to answer types and the number of choices in a
question.

Table 4 provides a breakdown of LXMERT’s scores for binary and multi-choice
questions. It is evident that multi-choice questions are harder for the model,
with self-consistency dropping by 16% between binary and multi-choice
questions, and c-acc dropping by 33%. This is surprising since the multi-
choice questions only include two or three choices and hence are quite similar
to the binary (yes/no) questions. This may indicate a bias in the models
towards binary questions with simple answers. Furthermore, Table 4 also shows
that models consistently perform worse on 3-choice questions than 2-choice
ones, with even the top-performing LXMERT having a 7% drop from 62% to 55%.
This hints that there may be some effect of randomness in the way these models
pick their answers. In contrast and as expected, humans are robust to the
number of choices.

Table 5:  We augment the LXMERT model by adding 95,000 questions (9.9% of the
training dataset) to the training dataset and keep all other training
procedures the same. The added questions are generated using the same
methodology as CARETS but using images from the GQA training set. GQA
validation accuracy stays comparable at 70.60% for LXMERT (Augmented) vs.
70.67% for LXMERT (GQA).

| LXMERT (Augmented)
---|---
| acc | cons |  | c-acc
---
rephrase-inv | 84.0 $\uparrow 16.2$ | 94.1 $\uparrow 10.3$ | 81.5 $\uparrow 19.6$
order-inv | 82.1 $\uparrow 18.8$ | 92.3 $\uparrow 21.3$ | 78.8 $\uparrow 27.8$
ontological-inv | 95.3 $\uparrow$ 9.6 | 94.0 $\uparrow 10.5$ | 92.4 $\uparrow 14.9$
negation-dir | 88.3 $\uparrow 36.0$ | 90.5 $\uparrow 62.4$ | 83.6 $\uparrow 66.6$
antonym-dir | 65.0 $\uparrow$ 5.5 | 83.5 $\uparrow 12.4$ | 76.7 $\uparrow 11.6$

#### (R6) Visual perturbations are easier for models to deal with.

From Figure 2 and Table 6, we notice that the models are slightly more robust
to visual perturbations on average compared to the lexical ones. All models
only have a drop of 4-8% from acc to c-acc, while self-consistency of all
models is also 78% or higher. Appendix A.2 provides a more detailed breakdown
of all the different visual perturbation tests we performed.

#### (R7) Direct data augmentation improves CARETS evaluation.

We show the feasibility of high performance on CARETS through data
augmentation. We add 95,000 questions generated from CARETS question
templates, and using a similar distribution of question types, to the original
training split of GQA and re-train the LXMERT model. Table 5 shows that this
dramatically improves the model on all three metrics (acc, self-consistency
and c-acc), with the LXMERT(Augmented) model achieving near human performance
on tests like order-inv and antonym-dir. Since CARETS is designed to be an
evaluation suite, these numbers show that CARETS questions should generally be
within the capabilities of existing SOTA models, provided that they are able
to generalize appropriately.

##  7 Conclusion & Future Work

In this work, we have developed CARETS – a new test suite for capability-
focused robustness testing and comprehension evaluation of visual question
answering (VQA) models. CARETS consists of six different tests that use
instance pairs to evaluate models on their understanding of various linguistic
and visual concepts. Using this test suite, we evaluated six modern VQA
systems to reveal several inconsistencies in state-of-the-art models and
provide a fine-grained view of their comprehension of different visuo-
linguistic concepts. Quite surprisingly, we find that even state-of-the-art
models struggle with concepts like negation, disjunction, order invariance and
multi-choice questions. CARETS can also support the addition of more tests in
the future and we view it as a platform for continually stress testing and
improving VQA models.

CARETS emulates previous work in using text templates to generate questions
and their textual perturbations Hudson and Manning (2019); Johnson et al.
(2017); Ribeiro et al. (2020b). The use of templates to generate perturbations
is motivated by the desire to maintain the grounded integrity of generations,
ensuring that they remain relevant and that the generated label is true in the
context of the subject image. While we have sought to generate a diverse
language set through using a large number of templates (nearly 200 in total),
there are some limitations to this approach. An improvement to our approach
may be able to generate more sophisticated questions and perturbations through
conditional text generation Schick and Schütze (2021); Madaan et al. (2021);
Wu et al. (2021) while also preserving our other motivations, such as
atomicity and grounded relevancy.

## Acknowledgments

This material is based upon work supported by the National Science Foundation
under Grant No. 2107048. Any opinions, findings, and conclusions or
recommendations expressed in this material are those of the author(s) and do
not necessarily reflect the views of the National Science Foundation.

## References

  * Agrawal et al. (2018) Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Aniruddha Kembhavi. 2018.  Don’t Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering.  _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Antol et al. (2015) Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015.  VQA: Visual Question Answering.  In _International Conference on Computer Vision (ICCV)_.
  * Bitton et al. (2021) Yonatan Bitton, Gabriel Stanovsky, Roy Schwartz, and Michael Elhadad. 2021.  Automatic generation of contrast sets from scene graphs: Probing the compositional consistency of gqa.  _ArXiv_ , abs/2103.09591.
  * Devlin et al. (2019) J. Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _NAACL-HLT_.
  * Goyal et al. (2017) Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017\.  Making the v in vqa matter: Elevating the role of image understanding in visual question answering.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * He et al. (2016) Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. 2016.  Deep residual learning for image recognition.  _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Hudson and Manning (2019) Drew A. Hudson and Christopher D. Manning. 2019.  GQA: A new dataset for real-world visual reasoning and compositional question answering.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Johnson et al. (2017) Justin Johnson, Li Fei-Fei, Bharath Hariharan, C. Lawrence Zitnick, Laurens Van Der Maaten, and Ross Girshick. 2017.  CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning.  _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Johnson et al. (2015) Justin Johnson, Ranjay Krishna, Michael Stark, Li Jia Li, David A. Shamma, Michael S. Bernstein, and Fei Fei Li. 2015.  Image retrieval using scene graphs.  _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Kafle and Kanan (2017) Kushal Kafle and Christopher Kanan. 2017.  An analysis of visual question answering algorithms.  In _IEEE International Conference on Computer Vision (ICCV)_.
  * Kervadec et al. (2020) Corentin Kervadec, Grigory Antipov, Moez Baccouche, and Christian Wolf. 2020.  Roses are red, violets are blue… but should vqa expect them to?
  * Kiela et al. (2019) Douwe Kiela, Suvrat Bhooshan, Hamed Firooz, Ethan Perez, and Davide Testuggine. 2019\.  Supervised multimodal bitransformers for classifying images and text.
  * Kim et al. (2018) Jin-Hwa Kim, Jaehyun Jun, and Byoung-Tak Zhang. 2018.  Bilinear attention networks.  In _Advances in Neural Information Processing Systems (NeurIPS)_. Curran Associates, Inc.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  _CoRR_ , abs/1412.6980.
  * Krishna et al. (2017) Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, and et al. 2017.  Visual genome: Connecting language and vision using crowdsourced dense image annotations.  _International Journal of Computer Vision_.
  * Li et al. (2019) Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2019\.  Visualbert: A simple and performant baseline for vision and language.
  * Li et al. (2018) Qing Li, Qingyi Tao, Shafiq Joty, Jianfei Cai, and Jiebo Luo. 2018.  Vqa-e: Explaining, elaborating, and enhancing your answers for visual questions.  _Proceedings of the European Conference on Computer Vision (ECCV)_.
  * Lin et al. (2014) Tsung-Yi Lin, M. Maire, Serge J. Belongie, James Hays, P. Perona, D. Ramanan, Piotr Dollár, and C. L. Zitnick. 2014.  Microsoft coco: Common objects in context.  In _Proceedings of the European Conference on Computer Vision (ECCV)_.
  * Lu et al. (2019) Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  In _NeurIPS_.
  * Madaan et al. (2021) Nishtha Madaan, Inkit Padhi, Naveen Panwar, and Diptikalyan Saha. 2021.  Generate your counterfactuals: Towards controlled counterfactual generation for text.  In _AAAI_.
  * Manjunatha et al. (2019) Varun Manjunatha, Nirat Saini, and Larry S. Davis. 2019.  Explicit bias discovery in visual question answering models.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Miller (1995) George A. Miller. 1995.  Wordnet: A lexical database for english.  _Commun. ACM_ , 38(11):39–41.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014.  Glove: Global vectors for word representation.  In _Empirical Methods in Natural Language Processing (EMNLP)_.
  * Ren et al. (2015) Shaoqing Ren, Kaiming He, Ross B. Girshick, and J. Sun. 2015.  Faster r-cnn: Towards real-time object detection with region proposal networks.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_.
  * Ribeiro et al. (2020a) Marco Tulio Ribeiro, Carlos Guestrin, and Sameer Singh. 2020a.  Are red roses red? Evaluating consistency of question-answering models.  _ACL 2019 - 57th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference_.
  * Ribeiro et al. (2020b) Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020b.  Beyond accuracy: Behavioral testing of NLP models with CheckList.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , Online. Association for Computational Linguistics.
  * Schick and Schütze (2021) Timo Schick and Hinrich Schütze. 2021.  Generating datasets with pretrained language models.  In _EMNLP_.
  * Selvaraju et al. (2020) Ramprasaath R. Selvaraju, Purva Tendulkar, Devi Parikh, E. Horvitz, Marco Túlio Ribeiro, Besmira Nushi, and Ece Kamar. 2020.  Squinting at vqa models: Introspecting vqa models with sub-questions.  _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Shah et al. (2019) Meet Shah, Xinlei Chen, Marcus Rohrbach, and Devi Parikh. 2019.  Cycle-consistency for robust visual question answering.  _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Singh et al. (2020) Amanpreet Singh, Vedanuj Goswami, Vivek Natarajan, Yu Jiang, Xinlei Chen, Meet Shah, Marcus Rohrbach, Dhruv Batra, and Devi Parikh. 2020.  Mmf: A multimodal framework for vision and language research.  https://github.com/facebookresearch/mmf.
  * Tan and Bansal (2020) Hao Tan and Mohit Bansal. 2020.  LXMert: Learning cross-modality encoder representations from transformers.  _Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_.
  * Torralba et al. (2008) A. Torralba, R. Fergus, and W. Freeman. 2008.  80 million tiny images: A large data set for nonparametric object and scene recognition.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _ArXiv_ , abs/1706.03762.
  * Warstadt et al. (2020) Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020.  BLiMP: The Benchmark of Linguistic Minimal Pairs for English.  _Transactions of the Association for Computational Linguistics (TACL)_.
  * Wu et al. (2021) Tongshuang Wu, Marco Tulio Ribeiro, Jeffrey Heer, and Daniel Weld. 2021.  Polyjuice: Generating counterfactuals for explaining, evaluating, and improving models.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 6707–6723, Online. Association for Computational Linguistics.
  * Xie et al. (2017) Saining Xie, Ross B. Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. 2017\.  Aggregated residual transformations for deep neural networks.  _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Yang et al. (2021) Kaiyu Yang, Jacqueline Yau, Li Fei-Fei, Jia Deng, and Olga Russakovsky. 2021.  A study of face obfuscation in imagenet.
  * Zhang et al. (2016) P. Zhang, Yash Goyal, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2016.  Yin and yang: Balancing and answering binary visual questions.  _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.

##  Appendix A Appendix

###  A.1 Test Dataset Examples

We provide examples of templates for each question type here. In Section 4.1,
we simplified the template arguments for readability. Here, we show template
examples with their arguments as they are actually represented for generation.

  * Q1:

Object verification (54 templates)

⬇

"Does it look like there [1:Is] [1:DET] <attrs1> <obj1> anywhere?"

"[1:Is] there [1:DET] <attrs1> <obj1> anywhere?"

  * Q2:

Conjunctive verification (18 templates):

⬇

"[1:Is] there both [1:DET] <attrs1> <obj1> and [2:DET] <attrs2> <obj2>?"

"Do you see both [1:DET] <attrs1> <obj1> and [2:DET] <attrs2> <obj2> in this
photo?"

  * Q3:

Disjunctive verification (18 templates):

⬇

"Do you see either [1:DET] <attrs1> <obj1> or [2:DET] <attrs2> <obj2>?"

"Does it look like there is either [1:DET] <attrs1> <obj1> or [2:DET] <attrs2>
<obj2> anywhere"

  * Q4:

Attribute verification (25 templates):

⬇

"Does it seem that the <obj1> that [1:Is] <2rel1> the <attrs2> <obj2> [1:Is]
<attrs1>?"

"Would you say that the <obj1> <2rel1> the <attrs2> <obj2> [1:Is] <attrs1>?"

  * Q5:

Object multi-choice (25 templates):

⬇

"What would you call the <obj-category1> that [1:Is] <2rel1> the <attrs2>
<obj2>, <obj-category-options1>?"

"Do you think the <attrs1> <obj-category1> [1:Is] <obj-category-options1>?"

  * Q6:

Attribute multi-choice (39 templates):

⬇

"Which sort of <category1> [1:Is] the <attrs1> <obj1>, <category-options1>?"

"What type of <category1> [1:Is] the <attrs1> <obj1>, <category-options1>?"

  * Q7:

Action multi-choice (28 templates):

⬇

"What kind of <category1> [1:Is] the <attrs1> <obj1> doing, <category-
options1>?"

"What would you call the sort of <category1> that the <attrs1> <obj1> [1:Is]
doing, <category-options1>?"

Table 6: Test dataset statistics: answer distributions for _original_ (non-perturbed) questions in each test dataset. Test Dataset | Binary |  | Multi-
---
choice
Total
rephrase-inv | 10,000 | 9,412 | 19,412
order-inv | 5,000 | 9,412 | 14,412
ontological-inv | 13,952 | - | 13,952
visual-inv | 18,000 | 8,272 | 26,272
negation-dir | 10,000 | - | 10,000
antonym-dir | 5,000 | - | 5,000

###  A.2 Visual Obfuscation Invariance Details

Table 3 shows examples of context blurring, masking, and cropping. Five
perturbations are done in total, blurring (with $\sigma\in\\{3,6,9\\}$),
masking context by replacing pixel values with the channel-wise average
computed from the GQA training data, and cropping around the tightest bounding
box containing the question’s objects.

|  |
---|---|---
Gaussian blur ($\sigma=9$) | Masking | Cropping
Figure 3: Visual obfuscation for the question “Is there both a banana and a black camera in this photo?” Table 7: Visual Accuracy | O | M | C | G3 | G6 | G9
---|---|---|---|---|---|---
BAN | 48.83 | 49.38 | 47.70 | 49.10 | 49.29 | 49.42
CNN+LSTM | 40.28 | 39.29 | 40.14 | 39.45 | 39.18 | 39.35
LXMERT | 69.64 | 66.33 | 61.82 | 66.59 | 66.89 | 67.06
MMBT | 50.37 | 50.00 | 49.17 | 50.44 | 50.27 | 50.18
ViLBERT | 51.60 | 51.12 | 49.99 | 51.69 | 51.36 | 51.22
VisualBERT | 53.81 | 53.25 | 52.53 | 53.67 | 53.39 | 53.44

###  A.3 Training Details

We provide greater detail on training environments and hyperparameter choices.

All models are trained using the GQA Hudson and Manning (2019) balanced
training set and validated on the balanced validation set (with minimal
parameter tuning, aiming to stay faithful to the original implementation). All
non-LXMERT models are trained and finetuned withing the MMF Singh et al.
(2020) framework, trained using binary cross-entropy loss for a set maximum
number of epochs, taking the epoch checkpoint that best performs on the
validation set, and using features from Faster R-CNN Ren et al. (2015) with a
ResNext-152 Xie et al. (2017) backbone pre-trained on the Visual Genome
Krishna et al. (2017) dataset for object-based models.

#### BAN.

The Bilinear Attention Network Kim et al. (2018) (BAN) uses pre-trained Faster
R-CNN features Ren et al. (2015) and GloVe Pennington et al. (2014) embeddings
with an attention model and early bilinear fusion mechanism. We train “four
glimpse” (with 4 attention heads) BAN Kim et al. (2018) (BAN) for 13 epochs
using the Adamax Kingma and Ba (2015) optimizer with an initial learning rate
of $1e^{-3}$ and batch size of 256, decaying the learning rate at epochs 11
and 13. As in the original hyperparameter configuration, we perform gradient
clipping at 0.25.

#### LXMERT.

LXMERT Tan and Bansal (2020) is a transformer-based architecture Vaswani et
al. (2017) with pre-trained Faster R-CNN features. It undergoes an extended
pre-training procedure using 5 different pre-training tasks, including image
question answering. We first pre-train a version of LXMERT from scratch with
all GQA validation instances removed from the pre-training data to prevent
direct leakage. We use all the default hyperparameters used in the author’s
GitHub repository.888github.com/airsplay/lxmert We then finetune LXMERT base
on the GQA training dataset for 4 epochs with the same hyperparameter
configuration as the original implementation, using a batch size of 32, and
initial learning rate of $1e^{-5}$. LXMERT base is pre-trained using the MS
COCO Lin et al. (2014) and Visual Genome datasets. LXMERT also uses a Faster
R-CNN Ren et al. (2015) with a ResNet-101 He et al. (2016) backbone.

#### VisualBERT.

VisualBERT Li et al. (2019) is similar in architecture and pre-training method
to BERT Devlin et al. (2019). It performs an early fusion of text and image
features immediately before several transformer layers. It uses Faster R-CNN
features, is initialized using weights from BERT, and pre-trained on 2
different tasks using the MS COCO dataset. We finetune an MS COCO pre-trained
version of VisualBERT using the same hyper parameters and training scheme of
the original implementation for the VQA task. We use the Adam W optimizer with
an initial learning rate of $2e^{-5}$ and a batch size of 64 for a maximum of
20 epochs.

#### CNN+LSTM.

This model uses a 6 layer CNN module and a bidirectional LSTM module before
concatenating the output of each module and passing the combined output to a
FC classifier. The LSTM module uses GloVe word embeddings Pennington et al.
(2014). Model weights are randomly initialized. We train the model for 25
epochs using the Adam W optimizer with an initial learning rate of $1e^{-4}$
and batch size of 256. This model uses a 6 layer CNN module and a
bidirectional LSTM module with a hidden size of 128, and concatenates the
output of each module before passing the combined output to 2 layer MLP
classifier with a ReLU activation. The LSTM module uses GloVe word embeddings
Pennington et al. (2014) to represent questions.

#### MMBT.

The MultiModal BiTransformer (MMBT) Kiela et al. (2019) is an early fusion
model, which uses Faster R-CNN features projected to a common space and
concatenated with contextual BERT embeddings before being passed to
transformer layers. MMBT uses pre-trained Faster R-CNN features and is
initialized with BERT pre-trained weights. We finetune MMBT with the Adam W
Kingma and Ba (2015) optimizer with an initial learning rate of $5e^{-5}$ with
a batch size of 64, for a maximum of 15 epochs.

#### ViLBERT.

ViLBERT Lu et al. (2019) uses two parallel transformer “streams” for vision
and language separately. These streams interact using multi-modal co-
attentional transformer blocks. It uses Faster R-CNN features and is
initialized using some weights from BERT. ViLBERT is pre-trained using 2
different tasks, using the MS COCO dataset. We finetune the MS COCO pre-
trained version of ViLBERT in a similar manner to the finetuning scheme used
for the VQA task of the original implementation. We use the Adam W Kingma and
Ba (2015) optimizer with an initial learning rate of $4e^{-5}$ and batch size
64, for a maximum of 20 epochs.

###  A.4 Test Results

We provide fuller test results for each test dataset, including accuracy on
the original instances and perturbed instances in tables A.4, A.4, A.4, A.4,
and A.4. These results supplement the primary results reported in Section 6.

Table 8: Full results for rephrase-inv.

| acc | Original acc | Perturbed acc | cons | c-acc
---|---|---|---|---|---
BAN | 55.86 | 55.90 | 55.83 | 82.32 | 50.20
CNN+LSTM | 37.58 | 37.61 | 37.56 | 68.93 | 28.90
LXMERT | 67.80 | 67.70 | 67.90 | 83.83 | 61.89
MMBT | 56.91 | 56.79 | 57.02 | 81.72 | 50.76
ViLBERT | 56.44 | 56.54 | 56.35 | 86.44 | 51.63
VisualBERT | 59.81 | 60.13 | 59.48 | 83.62 | 53.87
1-5 Human | 86.15 | 86.74 | 85.56 | 96.58 | 85.28

Table 9: Full results for order-inv.

| acc | Original acc | Perturbed acc | cons | c-acc
---|---|---|---|---|---
BAN | 51.69 | 51.81 | 51.57 | 65.22 | 38.43
CNN+LSTM | 33.25 | 33.24 | 33.26 | 62.56 | 22.29
LXMERT | 63.26 | 63.22 | 63.30 | 71.02 | 51.01
MMBT | 51.52 | 51.30 | 51.75 | 58.62 | 36.65
ViLBERT | 51.02 | 51.01 | 51.04 | 66.21 | 38.27
VisualBERT | 55.87 | 55.80 | 55.93 | 60.75 | 40.60
1-5 Human | 79.10 | 79.38 | 78.83 | 98.82 | 78.76

Table 10: Full results for ontological-inv.

| acc | Original acc | Perturbed acc | cons | c-acc
---|---|---|---|---|---
BAN | 77.67 | 79.25 | 76.10 | 77.36 | 66.36
CNN+LSTM | 52.37 | 48.70 | 56.04 | 73.37 | 39.06
LXMERT | 85.69 | 88.96 | 82.42 | 83.48 | 77.43
MMBT | 73.73 | 81.84 | 65.62 | 70.46 | 58.97
ViLBERT | 78.30 | 84.96 | 71.65 | 76.48 | 66.56
VisualBERT | 74.82 | 80.53 | 69.10 | 74.13 | 61.88
1-5 Human | 96.00 | 96.00 | 96.00 | 94.00 | 94.00

Table 11: Full results for antonym-dir.

| acc | Original acc | Perturbed acc | cons | c-acc
---|---|---|---|---|---
BAN | 71.02 | 81.70 | 60.34 | 56.80 | 49.42
CNN+LSTM | 51.98 | 70.64 | 33.32 | 10.50 | 7.24
LXMERT | 79.47 | 85.14 | 73.80 | 71.12 | 65.04
MMBT | 69.83 | 82.74 | 56.92 | 54.58 | 47.54
ViLBERT | 72.41 | 88.94 | 55.88 | 55.30 | 50.06
VisualBERT | 73.76 | 82.16 | 65.36 | 59.84 | 53.68
1-5 Human | 87.50 | 86.00 | 89.00 | 89.00 | 84.00

| acc | Original acc | Perturbed acc | cons | c-acc
---|---|---|---|---|---
BAN | 50.10 | 64.64 | 35.57 | 19.30 | 9.73
CNN+LSTM | 49.67 | 48.51 | 50.82 | 17.85 | 8.59
LXMERT | 52.31 | 76.08 | 28.55 | 28.11 | 17.00
MMBT | 50.13 | 67.41 | 32.86 | 23.07 | 11.90
ViLBERT | 49.15 | 67.58 | 30.72 | 18.73 | 8.61
VisualBERT | 51.10 | 68.67 | 33.53 | 18.26 | 10.23
1-5 Human | 78.25 | 82.50 | 74.00 | 88.50 | 74.50

Table 12: Full results for negation-dir.

###  A.5 Model comparison results

We provide additional model comparison results for each test dataset in Tables
13, 14, 15, 16, 17.

| B | C+L | L | M | ViL | Vis
---|---|---|---|---|---|---
BAN | - | 75.70 | 69.42 | 75.58 | 76.88 | 73.97
CNN+LSTM | 48.58 | - | 44.02 | 49.11 | 49.55 | 47.43
LXMERT | 83.84 | 80.09 | - | 83.23 | 84.40 | 83.01
MMBT | 76.53 | 76.22 | 69.77 | - | 79.47 | 77.58
ViLBERT | 77.00 | 76.27 | 70.12 | 78.74 | - | 76.87
VisualBERT | 78.95 | 76.93 | 73.39 | 81.89 | 81.80 | -

Table 13: rephrase-inv model comparison: Entries ($C_{i,j}$) show the
proportion of model $j$’s correct predictions that are also predicted
correctly by model $i$. E.g. here we see that BAN correctly predicts 65.39% of
the instance pairs correctly predicted by LXMERT, but LXMERT correctly
predicts 79.85% of the correct instance pairs captured by BAN. Table 14: Model
coverage results for order-inv.

| B | C+L | L | M | ViL | Vis
---|---|---|---|---|---|---
BAN | - | 76.20 | 67.90 | 75.41 | 76.28 | 72.81
CNN+LSTM | 46.63 | - | 41.31 | 47.71 | 47.94 | 45.95
LXMERT | 82.49 | 78.80 | - | 82.10 | 82.86 | 81.14
MMBT | 74.95 | 75.90 | 67.06 | - | 78.11 | 76.10
ViLBERT | 74.88 | 75.72 | 67.02 | 77.31 | - | 74.11
VisualBERT | 78.42 | 78.36 | 71.91 | 82.56 | 81.12 | -

Table 15: Model coverage results for ontological-inv.

| B | C+L | L | M | ViL | Vis
---|---|---|---|---|---|---
BAN | - | 81.71 | 81.91 | 84.95 | 83.92 | 83.85
CNN+LSTM | 55.09 | - | 53.05 | 54.57 | 53.64 | 53.70
LXMERT | 90.36 | 86.81 | - | 91.20 | 91.81 | 90.89
MMBT | 80.64 | 76.83 | 78.48 | - | 82.13 | 83.12
ViLBERT | 84.60 | 80.21 | 83.90 | 87.22 | - | 88.14
VisualBERT | 80.77 | 76.71 | 79.36 | 84.34 | 84.22 | -

Table 16: Model coverage results for antonym-dir.

| B | C+L | L | M | ViL | Vis
---|---|---|---|---|---|---
BAN | - | 79.06 | 78.60 | 82.55 | 82.89 | 82.21
CNN+LSTM | 57.94 | - | 55.13 | 59.96 | 59.20 | 57.48
LXMERT | 87.98 | 84.30 | - | 87.19 | 88.06 | 87.63
MMBT | 81.07 | 80.03 | 76.56 | - | 83.04 | 82.15
ViLBERT | 84.48 | 82.22 | 80.20 | 86.21 | - | 84.84
VisualBERT | 85.38 | 81.47 | 81.32 | 86.88 | 86.45 | -

Table 17: Model coverage results for negation-dir.

| B | C+L | L | M | ViL | Vis
---|---|---|---|---|---|---
BAN | - | 55.26 | 66.06 | 68.95 | 71.67 | 68.69
CNN+LSTM | 54.73 | - | 52.40 | 54.11 | 54.66 | 53.57
LXMERT | 68.91 | 55.16 | - | 70.95 | 72.17 | 70.60
MMBT | 68.99 | 54.64 | 68.08 | - | 73.13 | 71.83
ViLBERT | 70.24 | 54.08 | 67.76 | 71.69 | - | 73.48
VisualBERT | 70.04 | 55.16 | 68.96 | 73.19 | 76.43 | -


# Enhancing Large Language Models with Climate Resources

Mathias Kraus,1 Julia Anna Bingler,2,5 Markus Leippold,3 Tobias Schimanski,3
Chiara Colesanti Senni,3,5 Dominik Stammbach,4 Saeid Ashraf Vaghefi,3 Nicolas
Webersinke1

###### Abstract

Large language models (LLMs) have significantly transformed the landscape of
artificial intelligence by demonstrating their ability in generating human-
like text across diverse topics. However, despite their impressive
capabilities, LLMs lack recent information and often employ imprecise
language, which can be detrimental in domains where accuracy is crucial, such
as climate change. In this study, we make use of recent ideas to harness the
potential of LLMs by viewing them as agents that access multiple sources,
including databases containing recent and precise information about
organizations, institutions, and companies. We demonstrate the effectiveness
of our method through a prototype agent that retrieves emission data from
ClimateWatch (https://www.climatewatchdata.org/) and leverages general Google
search. By integrating these resources with LLMs, our approach overcomes the
limitations associated with imprecise language and delivers more reliable and
accurate information in the critical domain of climate change. This work paves
the way for future advancements in LLMs and their application in domains where
precision is of paramount importance.

##  1 Introduction

Figure 1: Setup for LLM agents that can access multiple sources, such as
databases or general internet search.

Motivation. Large language models have revolutionized the field of artificial
intelligence (AI) in recent years (Brown et al. 2020; Radford et al. 2019;
Ouyang et al. 2022). Models such as T0 (Sanh et al. 2021), LLama (Touvron et
al. 2023), Palm (Chowdhery et al. 2022), GPT-3 (Brown et al. 2020), GPT-4
(OpenAI 2023a), or instruction finetuned models, such as ChatGPT (OpenAI
2023b) have demonstrated their exceptional capabilities in generating human-
like text across various domains, including language translation,
summarization, and question answering. However, despite their impressive
performance, LLMs are not without limitations. First, LLMs by themselves only
have access to information from their training data (i.e., data from the
past). Thus LLMs can not access the most recent information which prohibits
usage in the domain of climate change to a large extend. Another critical
drawback is their tendency to use imprecise language, which can lead to
unreliable or inaccurate predictions in domains where precision and
responsibility are crucial. Thus, criticism of these systems has grown, with a
recent letter from several experts in the field asking, ”Should we let
machines flood our information channels with propaganda and untruth?” (Future
of Life Institute 2023).

One such domain where precision and accuracy are paramount is research on
climate reporting (Financial Stability Board 2017a). Accurate data and
information are essential for policymakers, scientists, and the general public
to understand the magnitude and urgency of the climate crisis and take
necessary actions to mitigate its impact (Financial Stability Board 2017b).
Inaccurate information, on the other hand, can lead to inappropriate policies
or delayed actions, exacerbating the consequences of climate change (Stern
2007). However, retrieving and interpreting accurate data from a vast amount
of climate-related information is a challenging task (Bingler et al. 2022).
One example of a valuable source of relevant information for understanding a
firm’s climate-related performance is annual reports, which contain both
quantitative and qualitative information (Edmans, Heinle, and Huang 2016).
However, this type of information is not naturally represented in LLMs and
must be retrieved separately.

Contribution. To address this challenge, we use recent developments in LLMs as
agents that access multiple sources, including databases containing precise
information about organizations, institutions, and companies. By integrating
these resources with LLMs, our approach mitigates the limitations associated
with imprecise language, providing more reliable and accurate information in
the critical domain of climate change. We present a prototype LLM agent that
retrieves emission data from ClimateWatch (https://www.climatewatchdata.org/)
and leverages general Google search. Through two exemplary experiments, we
demonstrate the potential of LLMs in the area of climate change. We anticipate
that our approach will encourage further research on integrating LLMs with
accurate data sources, ultimately fostering more responsible and reliable AI
systems.

Findings. Our study demonstrates the potential of LLMs as agents that access
multiple sources of information to deliver reliable and accurate information
in the domain of climate change. We have shown how our prototype agent can
retrieve and process data from ClimateWatch, a database containing precise
information on emissions from countries. Additionally, our approach leverages
general Google search to complement the information retrieved from
ClimateWatch and provide additional context to the generated text. Our
experiments and analyses showcase how LLM agents can be used to access
multiple datasources and combine this information to provide accurate and
reliable responses.

Implications. The implications of our study are significant for both practice
and research. In practice, this work is not limited to the field of climate
change but can be used to improve the accuracy and reliability of AI systems
in other domains where precision and responsibility are crucial. Policymakers,
scientists, and the general public can benefit from more accurate and reliable
information to make informed decisions and take necessary actions to mitigate
the impact of climate change. In research, our study paves the way for future
advancements in LLMs and their application in domains where precision is of
paramount importance. Further research is needed to explore the potential of
integrating LLMs with other data sources, such as scientific papers, reports,
and databases.

The remainder of this paper is structured as follows. Section 2 gives a brief
overview over recent developments in the field of LLMs, LLM agents, NLP in
climate change, and use of multiple sources in the context of LLMs. Section 3
describes the setting in which we run our experiments, Section 4 showcases
example prompts that need to access information from ClimateWatch, General
Internet, or a combination thereof. Section 5 discusses these results and
implications of this work. Section 6 concluces this work.

##  2 Background

### Large Language Models

Large language models are a type of artificial neural network that have
demonstrated remarkable ability to generate human-like text across diverse
topics (Radford et al. 2019; Brown et al. 2020; Ouyang et al. 2022). These
models are primarily transformer models and are trained on massive amounts of
text data to learn the patterns and relationships within the data (Vaswani et
al. 2017). The recent breakthroughs with models like T0 (Sanh et al. 2021),
LLama (Touvron et al. 2023), Palm (Chowdhery et al. 2022), GPT-3 (Brown et al.
2020), and GPT-4 (OpenAI 2023a) have further highlighted the potential of
LLMs, with applications ranging from natural language processing to chatbots
(OpenAI 2023b) and virtual assistants (Jo 2023). LLMs have also demonstrated
the ability to complete tasks that they were not explicitly trained on, making
them a versatile tool for a variety of applications. However, despite their
impressive capabilities, LLMs are also limited in several ways. One of the
primary limitations is their tendency to employ imprecise language, which can
be detrimental in domains where accuracy and responsibility are crucial. This
has led to concerns about the ethical implications of using LLMs for certain
applications, such as automated content creation or decision-making. In this
view, multiple experts from the field demand a pause in development of large
AI systems in their letter ”Pause Giant AI Experiments: An Open Letter”
(Future of Life Institute 2023).

### LLM Agents

An LLM agent employs the capabilities of state-of-the-art language models to
perform complex tasks and make informed decisions. These agents can
autonomously determine which actions to take, including utilizing various
tools and observing their outputs or providing responses to user queries
(Schick et al. 2023). By leveraging the LLM’s vast knowledge and understanding
of natural language, agents can efficiently navigate through an array of tools
and select the most appropriate one based on the given context. This enables
the LLM agent to provide reliable, accurate, and contextually relevant
solutions in diverse applications and domains.

In contrast to a basic LLM, an LLM agent does not only rely on the trained
weights in the network, thus it is capable of accessing data generated after
the point of training. This is crucial in many contexts, such as in climate
change, as there is constantly new information presented from companies,
institutions, and organizations that needs to be included into the AI system
in order to provide an informed response.

### NLP in Climate Change

Climate change is one of the most pressing challenges facing humanity today,
with implications for everything from the environment to the economy (Diaz and
Moore 2017). In order to effectively address this challenge, it is critical to
have accurate and reliable data about greenhouse gas emissions and other
climate-related factors (Mac Dowell et al. 2017). In the field of climate
change, accurate and up-to-date information is of paramount importance to
ensure that decision-makers, researchers, and the public can make informed
choices and develop effective strategies (Financial Stability Board 2017b).
Recent transformer models have started to be employed within the climate
change domain, witnessing improved accuracy in typical classification tasks
(Kölbel et al. 2020; Bingler et al. 2022; Callaghan et al. 2021; Webersinke et
al. 2022; Stammbach et al. 2022). These models are capable of accounting for
the context of words, enabling them to detect complex and implicit topic
patterns in addition to many trivial cases. However, recent advances in LLMs
have pushed the boundary towards more general models which are not restricted
to typical classification tasks, but which can serve as systems that can
potentially be used to make verdicts about quality of climate reporting etc.

### Use of Multiple Data Sources

Given the challenges associated with obtaining accurate and reliable data, the
use of multiple data sources has become increasingly common in research and
decision-making in topics related to climate change (Brunetti et al. 2021). By
combining data from multiple sources, it is possible to enhance the accuracy
and reliability of the results, as well as identify patterns and relationships
that might not be apparent from a single data source alone. However, the
integration of different data sources can be a challenging task, as the data
may be in different formats, have different levels of granularity, or be
subject to different biases. This has led to the development of various tools
and techniques for data integration (Liang et al. 2023). The potential of
combining different data sources to enhance the accuracy and reliability of
the results has significant implications for a wide range of applications,
from scientific research to business decision-making.

##  3 Setting

### LLM Agent

In this work, we build upon the LangChain
package.111https://github.com/hwchase17/langchain In the LangChain ecosystem,
the LLM agent operates by utilizing the ReAct framework. This framework
enables the agent to select the most suitable tool based on the tool’s
description, given that each tool has an associated description provided. The
ReAct framework allows the LLM agent to work with any number of tools. Our LLM
agent specifically employs the ”text-davinci-003” model from OpenAI with a
temperature setting of 0. This particular configuration maximizes precision,
ensuring that the agent generates appropriate responses while maintaining a
high degree of accuracy. The combination of the ReAct framework and the LLM’s
vast knowledge base empowers the agent to autonomously navigate through
diverse tasks, making it an invaluable asset in a wide range of applications.

The use of LLM agents in LangChain offers several advantages, including their
ability to process vast amounts of information, understand context, and adapt
to new tasks quickly. By harnessing the power of LLMs, agents can efficiently
handle complex challenges and provide precise solutions that cater to the
specific needs of users. This adaptability enables the LLM agent to
continually evolve its understanding and decision-making processes, staying
up-to-date with the latest developments and trends in various fields. As a
result, the LLM agent becomes a valuable tool for users, researchers, and
organizations alike, offering insights and solutions that might otherwise be
difficult or time-consuming to obtain.

In conclusion, LLM agents represent a significant advancement in the field of
artificial intelligence, particularly in the context of LangChain and the
ReAct framework. By leveraging the extensive knowledge and comprehension
abilities of LLMs, these agents can autonomously navigate a diverse array of
tools and generate contextually relevant, accurate, and reliable solutions.
The versatility and adaptability of LLM agents make them an essential asset in
various applications and domains, highlighting the immense potential for their
future development and integration into increasingly complex and sophisticated
AI systems.

### LLM Tools

In our setting, we use two tools. Note that this only serves as an exemplary
use case and we plan to extend this setting to include multiple data sources.

The first tool employed by our LLM agent is a Python-based module that
utilizes the powerful pandas library to access and manipulate data stored in
dataframes. This tool enables the agent to seamlessly interface with
structured data. In our case this tool allows access to data from ClimateWatch
and perform complex data processing tasks using Python code. By leveraging the
capabilities of pandas, the agent can efficiently filter, sort, aggregate, and
transform the data, allowing it to extract precise and relevant information as
needed. The integration of this tool with the LLM agent ensures that the AI
system has direct access to accurate and up-to-date data, which is vital for
generating reliable and responsible insights in the domain of climate change.

The second tool, ”google-seper,” is designed to facilitate general Google
searches within the context of the LLM agent. This tool allows the agent to
perform targeted queries on the web, tapping into the vast repository of
knowledge available through Google’s search engine. By leveraging ”google-
seper,” the LLM agent can complement the information retrieved from structured
data sources, such as ClimateWatch, with additional context and insights from
various web sources. This integration enables the agent to generate richer,
more comprehensive responses to

### Input Prompt

The prompt used in our study consists of a series of instructions that guide
the agent on how to answer a given question. The prompt is structured in a way
that allows the agent to access multiple sources of information, including
ClimateWatch and Google search. Listing 1 shows this input prompt.

The prompt begins with a section that describes the available tools to the
agent, which are ClimateWatch and Serper Search. The agent is instructed to
prioritize the use of ClimateWatch for answering general questions about
emissions and to only use Google search if necessary for answering questions
about current events. The ClimateWatch tool is described to handle pandas
dataframes in Python, which suggests that the agent should be able to read and
manipulate data in this format.

The prompt then presents a format for the agent to follow, which consists of
five sections: Question, Thought, Action, Action Input, and Observation. The
Question section presents the input question that the agent must answer. The
Thought section provides guidance for the agent to think about what to do, and
the Action section describes the action that the agent should take. The Action
Input section specifies the input to the action, and the Observation section
presents the result of the action. These five sections can repeat multiple
times until the agent is confident that it has the final answer, at which
point it is instructed to provide the Final Answer.

The prompt concludes with a reminder to the agent to prioritize the use of
ClimateWatch and to only use Google Search if the needed information is not
provided by ClimateWatch. This ensures that the agent uses reliable and
accurate data sources for generating its response.222In our experiments, we
noticed that the agent tends to use Google search, even in cases when the
information is available in the more reliable source of ClimateWatch.

Overall, the prompt is designed to guide the agent in accessing multiple
sources of information and to prioritize the use of precise and reliable data
sources for generating its response. By following the format presented in the
prompt, the agent is able to systematically retrieve and process information
to provide accurate and reliable answers to the input questions.

Listing 1: Input prompt.

⬇

Answer the following questions as best you can. Prioritize the "ClimateWatch"
tool and only use other tools if necessary. You have access to the following
tools:

ClimateWatch: useful for when you need to answer general questions about
emissions. You are working with a pandas dataframe in Python.

Serper Search: A low-cost Google Search API. Useful for when you need to
answer questions about current events. Input should be a search query.

Use the following format:

Question: the input question you must answer

Thought: you should always think about what to do

Action: the action to take, should be one of [ClimateWatch, Serper Search]

Action Input: the input to the action

Observation: the result of the action

... (this Thought/Action/Action Input/Observation can repeat N times)

Thought: I now know the final answer

Final Answer: the final answer to the original input question

Begin! Remember to prioritize the "ClimateWatch" tool and only use other tools
if the needed information is not provided by "ClimateWatch"."

Question: {input}

{agent_scratchpad}"

##  4 Experiments

In the following we showcase two experiments, in which the LLM agent should
once access only the ClimateWatch tool, and once has to combine information
from ClimateWatch and google-seper.

### Example with Single Data Source

Input: What is the average emission of Italy between 2010 and 2015?

The example with a single data source illustrates how the LLM agent uses the
prompt to answer a specific question about the average emissions of Italy
between 2010 and 2015. Listing 2 shows an extract of the chain of thought of
the LLM agent. The steps can be broken down into several steps:

  1. 1.

Upon receiving the input question, the LLM agent first identifies the goal of
finding the average emission of Italy between 2010 and 2015.

  2. 2.

The agent then determines that the best data source for answering this
question is the ClimateWatch tool, which is a pandas dataframe in Python.

  3. 3.

The agent uses the python_repl_ast action to print the columns of the
dataframe to gain an understanding of the available data.

  4. 4.

With this information, the agent determines the relevant columns for answering
the question, which are the ones corresponding to Italy and the years 2010 to
2015.

  5. 5.

The agent then calculates the average emission of Italy between 2010 and 2015
and provides the final answer.

Overall, the LLM agent follows a systematic approach to answering the input
question, starting with identifying the goal and determining the best data
source to use. The agent then processes the data to provide the final answer,
with each step of the process being captured in the prompt response. This
approach highlights the potential of LLMs to effectively retrieve and process
information from data sources and provide accurate and reliable answers to
complex questions.

Listing 2: Chain of thought for input ”What is the average emission of Italy
between 2010 and 2015?.”

⬇

Thought: I need to find the average emission of Italy between 2010 and 2015

Action: ClimateWatch

Action Input: pandas dataframe

Thought: I need to find out what the dataframe contains

Action: python_repl_ast

Action Input: print(df.columns)Index([’Country’, ’Data source’, ’Sector’,
’Gas’, ’Unit’, ’2019’, ’2018’,

...,

’1990’],

dtype=’object’)

Observation: None

Thought: I now know the columns of the dataframe

...

Thought: I need to filter the dataframe for Italy and select the columns from
2010 to 2015

Action: python_repl_ast

Action Input: df[df[’Country’] == ’Italy’][[’2010’, ’2011’, ’2012’, ’2013’,
’2014’, ’2015’]]

Observation: 2010 2011 2012 2013 2014 2015

448.37 437.82 419.63 387.64 367.92 379.58

Thought: I now know the final answer

Final Answer: Italy’s emissions from 2010 to 2015 were 448.37 MtCO2e in 2010,
437.82 MtCO2e in 2011, 419.63 MtCO2e in 2012, 387.64 MtCO2e in 2013, 367.92
MtCO2e in 2014, and 379.58 MtCO2e in 2015.

Thought: I now know the average emission of Italy between 2010 and 2015

Final Answer: The average emission of Italy between 2010 and 2015 was 406.83
MtCO2e.’

### Example with Combined Data Sources

Input: ”Which European country has the most ambitious net zero plans? How did
the emissions of this country develop over the last 10 years? Remember to only
include single countries.”

The example with combined data sources showcases how the LLM agent can
effectively combine information from multiple data sources to answer a complex
question. In this case, the question asks for the European country with the
most ambitious net zero plans and how the emissions of this country developed
over the last 10 years. Listing 3 shows an extract of the chain of thought of
the LLM agent. The steps can be broken down into several steps:

  1. 1.

Upon receiving the input question, the LLM agent determines that it needs to
find the European country with the most ambitious net zero plans and then look
at the emissions data for that country.

  2. 2.

The agent decides to use the Google Search tool to search for information on
the most ambitious net zero plans in Europe.

  3. 3.

The agent retrieves information from Google Search that Finland has passed
arguably the world’s most ambitious climate target into law, aiming to be the
first developed country to reach net zero in 2035 and net negative by 2040.

  4. 4.

The agent then determines that the best data source to look at Finland’s
emissions data is the ClimateWatch tool.

  5. 5.

The agent uses the python_repl_ast action to filter the ClimateWatch tool’s
pandas dataframe to only include data for Finland.

  6. 6.

The agent then observes that Finland’s emissions have decreased from 38.54
MtCO2e in 1990 to 58.42 MtCO2e in 2019, providing the final answer to the
question.

Overall, the LLM agent demonstrates the potential to effectively combine
information from multiple data sources to answer complex questions. Note,
however that the LLM agent provides information about the emissions from 1990
to 2019 and not over the last 10 years as asked for. However, with the recent
developments, we belief such shortcomings to be solved very soon.333Note also,
that we did not use the most recent LLM models in these experiments.

Listing 3: Chain of thought for input ”Which european country has the most
ambitious net zero plans? How did the emissions of this country develop over
the last 10 years? Remember to only include single countries.”

⬇

Thought: I should look for the country with the most ambitious net zero plans
and then look at the emissions data for that country.

Action: Serper Search

Action Input: "European country with most ambitious net zero plans"

Observation: Finland has passed arguably the world’s most ambitious climate
target into law. It aims to be the first developed country to reach net zero,
in 2035, and net negative - absorbing more CO2 than it emits - by 2040. May
31, 2022

Thought: I should now look at the emissions data for Finland.

Action: ClimateWatch

Action Input: Finland emissions data

Thought: I need to find the emissions data for Finland

Action: python_repl_ast

Action Input: df[df[’Country’] == ’Finland’]

Observation: Country Data source Sector Gas Unit 2019 \

82 Finland ClimateWatch Total including LUCF All GHG MtCO2e 58.42

2018 2017 2016 2015 ... 1999 1998 1997 1996 1995 1994 \

82 62.43 60.86 63.65 -0.28 ... 39.51 40.18 43.79 45.69 39.23 43.21

1993 1992 1991 1990

...

Final Answer: Finland has the most ambitious net zero plans and its emissions
have decreased from 38.54 MtCO2e in 1990 to 58.42 MtCO2e in 2019.’

##  5 Discussion

Limitations. While the described approach offers a promising solution for
leveraging the potential of LLMs in domains where accuracy and responsibility
are crucial, there are also limitations to the current state-of-the-art. One
of the primary limitations of current LLMs is their limited context length,
which can make it difficult for them to accurately capture complex
relationships and patterns that require a broader context. However, progress
is being made in this area, with recent models like GPT-4 demonstrating
significantly improved performance on longer sequences of text (OpenAI 2023a).
Another limitation of the LLM agents is the potential for the agent to
misunderstand the question such as seen in our second experiment. This
highlights the importance of developing robust methods for evaluating the
accuracy and reliability of the results. Finally, in this work, we merely used
two data sources. In practice, hundreds or thousands of data sources should be
used to combine the most precise information on each topic. While this
potentially raises technical issues, recent efforts by companies such as
Microsoft, which recently published a paper on the use of thousands of tools
for NLP applications, are promising and offer potential solutions to overcome
these limitations. (Liang et al. 2023).

Implications for NLP in Climate Change. The described approach has significant
implications for the field of NLP in the context of climate change research
and policy-making. By leveraging the potential of LLMs to generate human-like
text and integrating multiple data sources, it is possible to enhance the
accuracy and reliability of the results, as well as facilitate more informed
decision-making. This has significant implications for a wide range of
applications, from climate modeling to environmental policy-making.
Additionally, the used approach offers a solution for addressing some of the
limitations associated with LLMs, such as their outdated information stored in
the model itself and their tendency to employ imprecise language. By
incorporating precise data sources and developing methods for evaluating the
accuracy and reliability of the results, it is possible to overcome some of
these limitations and enhance the applicability of LLMs in domains where
precision and responsibility are crucial.

Implications for Research. The described approach has implications beyond the
specific application of climate change research and policy-making. It
represents a novel solution for leveraging the potential of LLMs in domains
where precision and accuracy are paramount, and highlights the potential of
combining multiple data sources to enhance the reliability and accuracy of the
results. Potential use cases could include for instance applications in
healthcare or legal reasoning. Here, this work can serve as a starting point
for future research in this area. While there are still limitations to the
current state-of-the-art, the development is of incredible pace and it is only
a matter of time until the majority of these limitations are also solved.
Overall, the described approach represents a significant step forward for the
field of NLP and has the potential to drive advancements in both research and
practice.

Carbon Footprint. While LLMs have demonstrated impressive capabilities in
generating human-like text across diverse topics, they come with a significant
energy consumption cost (Hershcovich et al. 2022). Training these models
requires massive amounts of computational resources, which in turn generates
considerable greenhouse gas emissions. Recent estimates suggest that training
a single large LLM, like GPT-3, can emit as much as 626,000 pounds of carbon
dioxide equivalent, which is roughly equivalent to the lifetime emissions of
five average cars (Hao 2019). These emissions can have a significant impact on
the environment, particularly in light of the urgent need to reduce global
carbon emissions. As such, there is a need for further research into methods
for optimizing the training and inference of LLMs to minimize their
environmental impact. For example, recent approaches like LoRA (Hu et al.
2021) or low-precision optimizations (Dettmers et al. 2022a, b) for reducing
the energy consumption and carbon emissions associated with LLMs. Further
research in this area is crucial to ensure that the benefits of LLMs are not
outweighed by their environmental costs, and to enable their widespread
adoption in a sustainable manner.

##  6 Conclusion

In conclusion, this paper demonstrates the potential of LLMs in the field of
climate change by employing an approach that integrates multiple sources of
information to correctly answer to questions that can not be answered by using
only the original model weights. Our prototype LLM agent retrieves information
from general Google search and and emission data from ClimateWatch to provide
reliable and accurate information. Through two exemplary experiments, we
showcase how such an LLM agent can operate to enhance the accuracy and
reliability of climate-related text generation. This work contributes to the
exploration of LLM applications in domains where up-to-date and accurate
information is critical, and encourages further research on integrating LLMs
with external data sources for more responsible and reliable AI systems.

## References

  * Bingler et al. (2022) Bingler, J. A.; Kraus, M.; Leippold, M.; and Webersinke, N. 2022.  Cheap talk and cherry-picking: What ClimateBert has to say on corporate climate risk disclosures.  _Finance Research Letters_ , 102776.
  * Brown et al. (2020) Brown, T.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J. D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33: 1877–1901.
  * Brunetti et al. (2021) Brunetti, C.; Dennis, B.; Gates, D.; Hancock, D.; Ignell, D.; Kiser, E. K.; Kotta, G.; Kovner, A.; Rosen, R. J.; and Tabor, N. K. 2021.  Climate Change and Financial Stability.  _Federal Reserve Notes_ , (2021-03): 19–3.
  * Callaghan et al. (2021) Callaghan, M.; Schleussner, C.-F.; Nath, S.; Lejeune, Q.; Knutson, T. R.; Reichstein, M.; Hansen, G.; Theokritoff, E.; Andrijevic, M.; Brecha, R. J.; et al. 2021.  Machine-learning-based evidence and attribution mapping of 100,000 climate impact studies.  _Nature climate change_ , 11(11): 966–972.
  * Chowdhery et al. (2022) Chowdhery, A.; Narang, S.; Devlin, J.; Bosma, M.; Mishra, G.; Roberts, A.; Barham, P.; Chung, H. W.; Sutton, C.; Gehrmann, S.; et al. 2022.  Palm: Scaling language modeling with pathways.  _arXiv preprint arXiv:2204.02311_.
  * Dettmers et al. (2022a) Dettmers, T.; Lewis, M.; Belkada, Y.; and Zettlemoyer, L. 2022a.  LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale.  _arXiv preprint arXiv:2208.07339_.
  * Dettmers et al. (2022b) Dettmers, T.; Lewis, M.; Shleifer, S.; and Zettlemoyer, L. 2022b.  8-bit Optimizers via Block-wise Quantization.  _9th International Conference on Learning Representations, ICLR_.
  * Diaz and Moore (2017) Diaz, D.; and Moore, F. 2017.  Quantifying the economic risks of climate change.  _Nature Climate Change_ , 7(11): 774–782.
  * Edmans, Heinle, and Huang (2016) Edmans, A.; Heinle, M. S.; and Huang, C. 2016.  The real costs of financial efficiency when some information is soft.  _Review of Finance_ , 20(6): 2151–2182.
  * Financial Stability Board (2017a) Financial Stability Board. 2017a.  Recommendations of the Task Force on Climate-Related Financial Disclosures.  _Task force on climate-related financial disclosures_.
  * Financial Stability Board (2017b) Financial Stability Board. 2017b.  Task force on climate-related financial disclosures: Overview.  _Task force on climate-related financial disclosures_.
  * Future of Life Institute (2023) Future of Life Institute. 2023.  Pause Giant AI Experiments: An Open Letter.  https://futureoflife.org/pause-giant-ai-experiments/.
  * Hao (2019) Hao, K. 2019.  Training a single AI model can emit as much carbon as five cars in their lifetimes.  _MIT technology Review_ , 75: 103.
  * Hershcovich et al. (2022) Hershcovich, D.; Webersinke, N.; Kraus, M.; Bingler, J. A.; and Leippold, M. 2022\.  Towards Climate Awareness in NLP Research.  _arXiv preprint arXiv:2205.05071_.
  * Hu et al. (2021) Hu, E. J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W. 2021.  Lora: Low-rank adaptation of large language models.  _arXiv preprint arXiv:2106.09685_.
  * Jo (2023) Jo, A. 2023.  The Promise and Peril of Generative AI.  _Nature_ , 614.
  * Kölbel et al. (2020) Kölbel, J. F.; Leippold, M.; Rillaerts, J.; and Wang, Q. 2020.  Ask BERT: How regulatory disclosure of transition and physical climate risks affects the CDS term structure.  _Available at SSRN 3616324_.
  * Liang et al. (2023) Liang, Y.; Wu, C.; Song, T.; Wu, W.; Xia, Y.; Liu, Y.; Ou, Y.; Lu, S.; Ji, L.; Mao, S.; Wang, Y.; Shou, L.; Gong, M.; and Duan, N. 2023.  TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIs.  _arXiv preprint arXiv:2303.16434_.
  * Mac Dowell et al. (2017) Mac Dowell, N.; Fennell, P. S.; Shah, N.; and Maitland, G. C. 2017.  The role of CO2 capture and utilization in mitigating climate change.  _Nature climate change_ , 7(4): 243–249.
  * OpenAI (2023a) OpenAI. 2023a.  GPT-4 Technical Report.  Technical report.
  * OpenAI (2023b) OpenAI. 2023b.  InstructGPT: AI for Generating Instructions.  https://openai.com/research/instructgpt/.
  * Ouyang et al. (2022) Ouyang, L.; Wu, J.; Jiang, X.; Almeida, D.; Wainwright, C.; Mishkin, P.; Zhang, C.; Agarwal, S.; Slama, K.; Ray, A.; et al. 2022.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ , 35: 27730–27744.
  * Radford et al. (2019) Radford, A.; Wu, J.; Child, R.; Luan, D.; Amodei, D.; and Sutskever, I. 2019.  Language Models are Unsupervised Multitask Learners.  _OpenAI Blog_.
  * Sanh et al. (2021) Sanh, V.; Webson, A.; Raffel, C.; Bach, S. H.; Sutawika, L.; Alyafeai, Z.; Chaffin, A.; Stiegler, A.; Scao, T. L.; Raja, A.; Dey, M.; Bari, M. S.; Xu, C.; Thakker, U.; Sharma, S. S.; Szczechla, E.; Kim, T.; Chhablani, G.; Nayak, N.; Datta, D.; Chang, J.; Jiang, M. T.-J.; Wang, H.; Manica, M.; Shen, S.; Yong, Z. X.; Pandey, H.; Bawden, R.; Wang, T.; Neeraj, T.; Rozen, J.; Sharma, A.; Santilli, A.; Fevry, T.; Fries, J. A.; Teehan, R.; Biderman, S.; Gao, L.; Bers, T.; Wolf, T.; and Rush, A. M. 2021.  Multitask Prompted Training Enables Zero-Shot Task Generalization.  arXiv:2110.08207.
  * Schick et al. (2023) Schick, T.; Dwivedi-Yu, J.; Dessì, R.; Raileanu, R.; Lomeli, M.; Zettlemoyer, L.; Cancedda, N.; and Scialom, T. 2023.  Toolformer: Language models can teach themselves to use tools.  _arXiv preprint arXiv:2302.04761_.
  * Stammbach et al. (2022) Stammbach, D.; Webersinke, N.; Bingler, J. A.; Kraus, M.; and Leippold, M. 2022\.  A Dataset for Detecting Real-World Environmental Claims.  _arXiv preprint arXiv:2209.00507_.
  * Stern (2007) Stern, N. 2007.  _The Economics of Climate Change: The Stern Review_.  Cambridge University Press.
  * Touvron et al. (2023) Touvron, H.; Lavril, T.; Izacard, G.; Martinet, X.; Lachaux, M.-A.; Lacroix, T.; Rozière, B.; Goyal, N.; Hambro, E.; Azhar, F.; et al. 2023.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_.
  * Vaswani et al. (2017) Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems_ , 5998–6008.
  * Webersinke et al. (2022) Webersinke, N.; Kraus, M.; Bingler, J.; and Leippold, M. 2022.  ClimateBERT: A Pretrained Language Model for Climate-Related Text.  In _Proceedings of AAAI 2022 Fall Symposium: The Role of AI in Responding to Climate Challenges_.