# An Analysis of Negation in Natural Language Understanding Corpora

Md Mosharaf Hossain,8 Dhivya Chinnappa,U Eduardo Blanco7
8Department of Computer Science and Engineering, University of North Texas
UThomson Reuters
7School of Computing and Augmented Intelligence, Arizona State University
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

This paper analyzes negation in eight popular corpora spanning six natural
language understanding tasks. We show that these corpora have few negations
compared to general-purpose English, and that the few negations in them are
often unimportant. Indeed, one can often ignore negations and still make the
right predictions. Additionally, experimental results show that state-of-the-
art transformers trained with these corpora obtain substantially worse results
with instances that contain negation, especially if the negations are
important. We conclude that new corpora accounting for negation are needed to
solve natural language understanding tasks when negation is present.

##  1 Introduction

Natural language understanding (NLU) is an umbrella term used to refer to any
task that requires text understanding. For example, question answering
Rajpurkar et al. (2016), information extraction Stanovsky et al. (2018),
coreference resolution Wu et al. (2020), and machine reading Yang et al.
(2019), among many others, are tasks that fall under natural language
understanding. The threshold for claiming that a system understands natural
language is ever-moving. New corpora are often justified by pointing out that
state-of-the-art models do not obtain good results. After years of steady
improvements, more powerful models eventually obtain so-called human
performance, and at that point new, more challenging corpora are created.

Many corpora for natural language understanding tasks contain language
generated by annotators rather than retrieved from texts written independently
of the corpus creation process. These corpora are certainly useful and have
facilitated tremendous progress. Annotator-generated examples, however, carry
the risk of evaluating systems with synthetic language that is not
representative of language in the wild. For example, annotators are likely to
use negation when asked to write a text that contradicts something despite
contradictions in the wild need not have a negation Gururangan et al. (2018).
Recently, Kwiatkowski et al. (2019) present a large corpus for question
answering that consists of natural questions (i.e., asked by somebody with a
real information need) in order to encourage research in a more realistic
scenario. This contrasts with previous corpora, where the questions were
written by annotators after being told the answer Rajpurkar et al. (2016).

In this paper, we explore the role of negation in eight corpora for six
popular natural language understanding tasks. Our goal is to check whether
negation plays the role it deserves in these tasks. To our surprise, we
conclude that negation is virtually ignored by answering the following
questions:111Code and data available at
https://github.com/mosharafhossain/negation-and-nlu.

  1. 1.

Do NLU corpora contain as many negations as general-purpose texts? (they
don’t);

  2. 2.

Do the (few) negations in NLU corpora play a role in solving the tasks? (they
don’t); and

  3. 3.

Do state-of-the-art transformers trained with NLU corpora face challenges with
instances that contain negation? (they do, especially if the negation is
important).

##  2 Background and Related Work

We work with the eight corpora covering six tasks summarized below and
exemplified in Table 2.

We select two corpora for question answering: CommonsenseQA Talmor et al.
(2019) and COPA Roemmele et al. (2011). CommonsenseQA consists of multi-choice
questions (5 candidate answers) that require some degree of commonsense. COPA
presents a premise (e.g., _The man broke his toe_) and a question (e.g., _What
was the cause of this?_) and the system must choose between two plausible
alternatives (e.g. _He got a hole in his sock_ or _He dropped a hammer on his
foot_).

For textual similarity and paraphrasing, we select
QQP222https://www.quora.com/q/quoradata/First-Quora-Dataset-Release-Question-
Pairs and STS-B Cer et al. (2017). QQP consists of pairs of questions and the
task is to determine whether they are paraphrases. STS-B consists of pairs of
texts and the task is to determine how semantically similar they are with a
score from 0 to 5.

We select one corpus for the remaining tasks. For inference, we work with QNLI
Rajpurkar et al. (2016), which consists in determining whether a text is a
valid answer to a question. We use WiC Pilehvar and Camacho-Collados (2019)
for word sense disambiguation. WiC consists in determining whether two
instances of the same word (in two sentences; italicized in Table 2) are used
with the same meaning. For coreference resolution, we choose WSC Levesque et
al. (2012), which consists in determining whether a pronoun and a noun phrase
are co-referential (italicized in Table 2). Finally, we work with SST-2 Socher
et al. (2013) for sentiment analysis. The task consists in determining whether
a sentence from a collection of movie reviews has positive or negative
sentiment.

For convenience, we work with the formatted versions of these corpora in the
GLUE Wang et al. (2018) and SuperGLUE Wang et al. (2019) benchmarks. The only
exception is CommonsenseQA, which is not part of these benchmarks.

Related Work Previous work has shown that SNLI Bowman et al. (2015) and MNLI
Williams et al. (2018) have annotation artifacts (e.g., negation is a strong
indicator of _contradictions_) Gururangan et al. (2018). The literature has
also shown that simple adversarial attacks including negation cues are very
effective Naik et al. (2018); Wallace et al. (2019). Kovatchev et al. (2019)
analyze 11 paraphrasing systems and show that they obtain substantially worse
results when negation is present.

More recently, Ribeiro et al. (2020) show that negation is one of the
linguistic phenomena commercial sentiment analysis struggle with. Several
previous works have investigated the (lack of) ability of transformers to make
inferences when negation is present. For example, Ettinger (2020) conclude
that BERT is unable to complete sentences when negation is present. BERT also
faces challenges solving the task of natural language inference (i.e.,
identifying entailments and contradictions) with monotonicity and negation
Geiger et al. (2020); Yanaka et al. (2019). Warstadt et al. (2019) show the
limitations of BERT making acceptability judgments with sentences that contain
negative polarity items. Most related to out work, Hossain et al. (2020)
analyze the role of negation in three natural language inference corpora: RTE
Dagan et al. (2006); Bar-Haim et al. (2006); Giampiccolo et al. (2007);
Bentivogli et al. (2009), SNLI and MNLI. In this paper, we present a similar
analysis, but we move beyond natural language inference and work with eight
corpora spanning six natural language understanding tasks.

| #sents. | % w/ neg.
---|---|---
Question Answering |  |
CommonsenseQA | 12,102 | 14.5
COPA | 1,000 | 0.8
Similarity and Paraphrasing |  |
QQP | 1,590,482 | 8.1
STS-B | 17,256 | 7.1
Inference |  |
QNLI | 231,338 | 8.7
Word Sense Disambiguation |  |
WiC | 14,932 | 8.2
Coreference Resolution |  |
WSC | 804 | 26.2
Sentiment Analysis |  |
SST-2 | 70,042 | 16.0
General-purpose English |  |
all sentences | 8,300,000 | 22.6–29.9
only questions | 456,214 | 15.8–20.2
Table 1: Number of sentences and percentage of sentences containing negation in natural language understanding corpora. All but WSC contain substantially fewer negations than general-purpose English texts.  |  Example |  | Important?
---|---|---|---
CmmsnsQA |  […] he (John) never saw the lady before. They were what? | C | ✓
A) pay debts, B) slender, C) unacquainted, D) free flowing, E) sparse
When you travel you should what in case of unexpected costs? | E | ✗
A) go somewhere, B) energy, C) spend frivilously, D) fly in airplane, E) have
money
QQP |  What are some not-so-boring baby shower games ? | yes | ✓
What are some baby shower games that are actually fun?
Who was philosophical guru of Shivaji Maharaj? | no | ✗
What are the unknown facts of shivaji maharaj?
STS-B |  Colin Powell, the Secretary of State, said contacts with Iran would not stop. | 4.3 | ✓
Secretary of State Colin Powell said yesterday that contacts with Iran would
continue.
Well for one a being could have a non-physical existance and yet not even be in your mind. | 3.4 | ✗
The difference is huge, as not all non-physical things exist in minds.
QNLI |  Who did BSkyB team up with as it was not part of consortium? | yes | ✓
While BSkyB had been excluded from being a part of the […], BSkyB was able to
join ITV Digital’s free-to-air replacement, Freeview, in which it holds an
equal stake […]
In what year did Lavoisier publish his work on combustion? | no | ✗
In one experiment, Lavoisier observed that there was no overall increase in
weight when tin and air were heated in a closed container.
SST-2 |  It’s not the ultimate depression-era gangster movie. | neg. | ✓
Whaley’s determination to immerse you in sheer, unrelenting wretchedness is exhausting. | neg. | ✗
WiC |  The _intention_ of this legislation is to boost the economy. | same | ✗
Good _intentions_ are not enough.
WSC |  _Sam and Amy_ are passionately in love, but Amy’s parents are unhappy about it, because _they_ are only fifteen. | yes | ✗
Table 2: Examples containing negation (underlined) from the validation
datasets of the natural language understanding corpora we work with. The third
column presents the expected answer for the example (a choice, judgment, or
score depending on the task). The last column indicates whether the negation
is important.

##  3 Research Questions and Analysis

Q1: Do natural language understanding corpora contain as many negations as
general-purpose English texts? In order to automatically identify negation
cues, we train a negation cue detector with the largest corpus available,
ConanDoyle-neg Morante and Daelemans (2012). The cue detector is based on the
RoBERTa pretrained language model Liu et al. (2019); we provide details about
the architecture and training process in Appendix A. Our cue detector obtains
the best results to date: F1: 93.79 vs. 92.94 Khandelwal and Sawant (2020).
ConanDoyle-neg (and thus our cue detector) identifies common negation cues
such as _no_ , _not_ , _n’t_ and _never_ , affixal negation cues such as
_impossible_ and _careless_ , and lexical negations such as _deny_ and
_avoid_.

Table 1 presents the percentage of sentences that contain negation in (a) the
eight corpora we work with and (b) general-purpose English. We take the latter
percentage (all sentences) from Hossain et al. (2020), who run a negation cue
detector in online reviews, conversations, and books. Additionally, we also
present the percentages in questions. Negation is much less common in all
natural language understanding corpora but WSC (0.8%–16%) than in general-
purpose English (22.6%–29.9%). Note that negation is also underrepresented in
corpora that primarily contain questions (general-purpose: 15.8%–20.2%; COPA:
0.8%, QQP: 8.1%).

|  |  Example |  | Important?
---|---|---|---|---
CommonsenseQA |  Syntactic |  Where would a person live if they wanted no neighbors? | D | ✓
A) housing estate, B) neighborhood, C) mars, D) woods, E) suburbs |  |
The teacher doesn’t tolerate noise during a test in their what? | E | ✗
A) movie theatre, B) bowling alley, C) factory, D) store, E) classroom |  |
Morpho. |  What might result in an unsuccessful suicide attempt? | B | ✓
A) die, B) interruption, C) bleed, D) hatred, E) dying |  |
How are the conditions for someone who is living in a homeless shelter? | A | ✗
A) sometimes bad, B) happy, C) respiration, D) growing older, E) death |  |
STS-2 |  Syntactic |  Despite the evocative aesthetics evincing the hollow state of modern love life, the film never percolates beyond a monotonous whine. | neg. | ✓
Even if you don’t think (kissinger’s) any more guilty of criminal activity than most contemporary statesmen, he’d sure make a courtroom trial great fun to watch. | pos. | ✗
Morpho. |  Makes for a pretty unpleasant viewing experience. | neg. | ✓
For anyone unfamiliar with pentacostal practices in general and theatrical phenomenon of hell houses in particular, it’s an eye-opener . | pos. | ✗
Table 3: Examples containing syntactic and morphological negation (underlined) from the validation datasets of CommonsenseQA and SST-2.  | CmmnsnsQA | COPA | QQP | STS-B | QNLI | WiC | WSC | SST-2
---|---|---|---|---|---|---|---|---
validation w/o neg | 0.60 | 0.73 | 0.90 | 0.92 / 0.91 | 0.93 | 0.67 | 0.63 | 0.94
validation w/ neg | 0.53 | n/a | 0.91 | 0.85 / 0.84 | 0.91 | 0.64 | 0.59 | 0.93
important (sample from Q2) | 0.47 | n/a | 0.73 | 0.57 / 0.62 | 0.67 | n/a | n/a | 0.86
unimportant (sample from Q2) | 0.62 | n/a | 0.92 | 0.85 / 0.84 | 0.92 | 0.64 | 0.59 | 0.95
Table 4:  Results obtained with RoBERTa evaluating against (a) all instances
with and without negation, and (b) the sample of instances with negation we
analyze in detail (important and unimportant). Since the datasets are
unbalanced, we report macro F1-score for all tasks except STS-B, for which we
report Pearson and Spearman correlations. Results are slightly lower with
negation, and substantially lower with _important_ negations.

Q2: Do the (few) negations in natural language understanding corpora play a
role in solving the tasks? After showing that negation in underrepresented in
natural language understanding corpora, we explore whether the few negations
they contain are important. Given an instance from any of the corpora, we
consider a negation _important_ if removing it changes the ground truth. In
other words, a negation is _unimportant_ if one can ignore it and still solve
the task at hand. Table 2 presents examples of important and unimportant
negations.

We manually examine the negations in all instances containing negation from
the validation split of each corpus except QQP, for which we examine 1,000
(out of 5,196). Note that COPA does not have any negations in the validation
split, and many corpora have few instances containing negation (CommonsenseQA:
184, STS-B: 225, QNLI: 852, WiC: 99, WSC: 52, and SST-2: 263). We choose to
work with the validation set because we want to compare results when negation
is and is not important (Q3), and the ground truth for the test splits of some
corpora are not publicly available.

We observe that (a) all negations in WiC and WSC are unimportant, and (b) the
percentages of unimportant negations in CommonsenseQA, SST-2, QQP, STS-B, and
QNLI are substantial: 45.1%, 63%, 97.4%, 95.6%, and 97.7%, respectively. These
percentages indicate that one can safely ignore (almost) all negations and
still solve the benchmarks. Despite the fact that negations are not important
in WSC and WiC, they do affect the experimental results (details in Q3).

We also analyze the role of two major types of negation: syntactic (_not_ ,
_no_ , _never_ , etc.) and morphological (i.e., affixes such as _un-_ , _im-_
, and _-less_). To this end, we work with CommonsenseQA and SST-2, which have
lower percentages of unimportant negations (45.1% and 63%) than the other
corpora we use (97.4%–100%). Table 3 provides examples of these two negation
types. Perhaps unsurprisingly, syntactic negations are much more common than
morphological negations (CommonsenseQA: 88.6% vs 11.4%, SST-2: 71.9% vs
28.1%). More importantly, syntactic negations are more often important in
SST-2 (42.3% vs 23%), but both syntactic and morphological negation are
roughly equaly important in CommonsenseQA (55.2% vs 52.4%).

Q3: Do state-of-the-art transformers trained with NLU corpora face challenges
with instances that contain negation? We conduct experiments with RoBERTa Liu
et al. (2019). More specifically, we use the implementation by Phang et al.
(2020) and train a model with the training split of each corpus. We refer the
readers to the Appendix B for the details about these models and
hyperparameters. We chose RoBERTa over other transformers because 4 out of the
10 best submissions to the SuperGLUE benchmark use
it.333https://super.gluebenchmark.com/leaderboard

Table 4 presents the results evaluating the models with the corresponding
validation splits. RoBERTa obtains slightly worse results with the validation
instances that have negation in all corpora; the only exception is QQP (F1:
0.90 vs. 0.91). These results lead to the conclusion that negation _may_ only
pose a small challenge to state-of-the-art transformers.

The results obtained evaluating with the important and unimportant negations
from the samples analyzed in Question 2, however, provide a different picture.
Indeed, we observe substantial drops in results in all tasks that have both
kinds of negations. More specifically, we obtain 27% lower results with
instances containing important negations in QNLI (F1: 0.92 vs. 0.67), 33%/26%
lower in STS-B, 24% lower in CommonsenseQA, 21% lower in QQP, and 9% lower in
SST. Further, even though all negations are unimportant in WiC and WSC, we
observe a drop in performance for the instances with negation compared to the
instances without negation (WiC: 0.64 vs 0.67 and WSC: 0.59 vs 0.63). We
conclude that transformers trained with existing NLU corpora face challenges
with instances that contain negation. These results raise two important
questions for future research: Is negation an inherently challenging
phenomenon for RoBERTa? How many instances with negation are required to solve
a natural language understanding task?

##  4 Conclusions

We have analyzed the role of negation in eight natural language understanding
corpora covering six tasks. Our analyses show that (a) all but WSC contain
almost no negations or around 31%–54% of the negations found in general-
purpose texts, (b) the few negations in these corpora are usually unimportant,
and (c) RoBERTa obtains substantially worse results when negation is
important.

Our analyses also provide some evidence that creating models to properly deal
with negation may require both new corpora and more powerful models. The need
for new corpora stems from the answers to Questions 1 and 2. The justification
for powerful models is more subtle. We point out that the percentage of
unimportant negations (Section 3) is only a weak indicator of the drop in
results with important negations (Table 4). For example, we observe a 24% and
21% drop in results with important negations from CommonsenseQA and QQP
despite 45% and 97% of negations are unimportant.

Negation reverses truth values thus solutions to any natural language
understanding task should be robust when negation is present and important. To
this end, our future work includes two lines of research. First, we plan to
create benchmarks for the six tasks consisting of instances containing
negation (50/50 split important/unimportant). Second, we plan to conduct
probing experiments to investigate whether (and where) pretrained transformers
capture the meaning of negation. Doing so may help us discover potential
solutions to understand negation and make inferences.

## Acknowledgements

This material is based upon work supported by the National Science Foundation
under Grant No. 1845757. Any opinions, findings, and conclusions or
recommendations expressed in this material are those of the authors and do not
necessarily reflect the views of the NSF. The Titan Xp used for this research
was donated by the NVIDIA Corporation. Computational resources were also
provided by the UNT office of High-Performance Computing. Further, we utilized
computational resources from the Chameleon platform Keahey et al. (2020). We
also thank the reviewers for insightful comments.

## References

  * Bar-Haim et al. (2006) Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. 2006.  The second pascal recognising textual entailment challenge.  In _Proceedings of the second PASCAL challenges workshop on recognising textual entailment_ , volume 6, pages 6–4. Venice.
  * Bentivogli et al. (2009) Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. 2009.  The fifth pascal recognizing textual entailment challenge.
  * Bowman et al. (2015) Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015\.  A large annotated corpus for learning natural language inference.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 632–642, Lisbon, Portugal. Association for Computational Linguistics.
  * Cer et al. (2017) Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. 2017\.  SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation.  In _Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)_ , pages 1–14, Vancouver, Canada. Association for Computational Linguistics.
  * Dagan et al. (2006) Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006.  The pascal recognising textual entailment challenge.  In _Proceedings of the First International Conference on Machine Learning Challenges: Evaluating Predictive Uncertainty Visual Object Classification, and Recognizing Textual Entailment_ , MLCW’05, pages 177–190, Berlin, Heidelberg. Springer-Verlag.
  * Ettinger (2020) Allyson Ettinger. 2020.  What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models.  _Transactions of the Association for Computational Linguistics_ , 8:34–48.
  * Geiger et al. (2020) Atticus Geiger, Kyle Richardson, and Christopher Potts. 2020.  Neural natural language inference models partially embed theories of lexical entailment and negation.  In _Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP_ , pages 163–173, Online. Association for Computational Linguistics.
  * Giampiccolo et al. (2007) Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. 2007.  The third PASCAL recognizing textual entailment challenge.  In _Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing_ , pages 1–9, Prague. Association for Computational Linguistics.
  * Gururangan et al. (2018) Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018.  Annotation artifacts in natural language inference data.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 107–112, New Orleans, Louisiana. Association for Computational Linguistics.
  * Hossain et al. (2020) Md Mosharaf Hossain, Venelin Kovatchev, Pranoy Dutta, Tiffany Kao, Elizabeth Wei, and Eduardo Blanco. 2020.  An analysis of natural language inference benchmarks through the lens of negation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 9106–9118, Online. Association for Computational Linguistics.
  * Keahey et al. (2020) Kate Keahey, Jason Anderson, Zhuo Zhen, Pierre Riteau, Paul Ruth, Dan Stanzione, Mert Cevik, Jacob Colleran, Haryadi S. Gunawi, Cody Hammock, Joe Mambretti, Alexander Barnes, François Halbach, Alex Rocha, and Joe Stubbs. 2020.  Lessons learned from the chameleon testbed.  In _Proceedings of the 2020 USENIX Annual Technical Conference (USENIX ATC ’20)_. USENIX Association.
  * Khandelwal and Sawant (2020) Aditya Khandelwal and Suraj Sawant. 2020.  NegBERT: A transfer learning approach for negation detection and scope resolution.  In _Proceedings of the 12th Language Resources and Evaluation Conference_ , pages 5739–5748, Marseille, France. European Language Resources Association.
  * Kovatchev et al. (2019) Venelin Kovatchev, M. Antonia Marti, Maria Salamo, and Javier Beltran. 2019.  A qualitative evaluation framework for paraphrase identification.  In _Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019)_ , pages 568–577, Varna, Bulgaria. INCOMA Ltd.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019.  Natural questions: A benchmark for question answering research.  _Transactions of the Association for Computational Linguistics_ , 7:452–466.
  * Levesque et al. (2012) Hector J. Levesque, Ernest Davis, and Leora Morgenstern. 2012.  The winograd schema challenge.  In _Proceedings of the Thirteenth International Conference on Principles of Knowledge Representation and Reasoning_ , KR’12, page 552–561. AAAI Press.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Morante and Daelemans (2012) Roser Morante and Walter Daelemans. 2012.  ConanDoyle-neg: Annotation of negation cues and their scope in conan doyle stories.  In _Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12)_ , pages 1563–1568, Istanbul, Turkey. European Language Resources Association (ELRA).
  * Naik et al. (2018) Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig. 2018.  Stress test evaluation for natural language inference.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , pages 2340–2353, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
  * Phang et al. (2020) Jason Phang, Phil Yeres, Jesse Swanson, Haokun Liu, Ian F. Tenney, Phu Mon Htut, Clara Vania, Alex Wang, and Samuel R. Bowman. 2020.  jiant 2.0: A software toolkit for research on general-purpose text understanding models.  http://jiant.info/.
  * Pilehvar and Camacho-Collados (2019) Mohammad Taher Pilehvar and Jose Camacho-Collados. 2019.  WiC: the word-in-context dataset for evaluating context-sensitive meaning representations.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 1267–1273, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  SQuAD: 100,000+ questions for machine comprehension of text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
  * Ribeiro et al. (2020) Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020.  Beyond accuracy: Behavioral testing of NLP models with CheckList.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4902–4912, Online. Association for Computational Linguistics.
  * Roemmele et al. (2011) Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. 2011.  Choice of plausible alternatives: An evaluation of commonsense causal reasoning.  In _AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning_ , pages 90–95.
  * Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , pages 1631–1642, Seattle, Washington, USA. Association for Computational Linguistics.
  * Stanovsky et al. (2018) Gabriel Stanovsky, Julian Michael, Luke Zettlemoyer, and Ido Dagan. 2018.  Supervised open information extraction.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 885–895, New Orleans, Louisiana. Association for Computational Linguistics.
  * Talmor et al. (2019) Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019.  CommonsenseQA: A question answering challenge targeting commonsense knowledge.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4149–4158, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Wallace et al. (2019) Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019.  Universal adversarial triggers for attacking and analyzing NLP.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2153–2162, Hong Kong, China. Association for Computational Linguistics.
  * Wang et al. (2019) Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019.  Superglue: A stickier benchmark for general-purpose language understanding systems.  In _Advances in Neural Information Processing Systems 32_ , pages 3261–3275. Curran Associates, Inc.
  * Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 353–355, Brussels, Belgium. Association for Computational Linguistics.
  * Warstadt et al. (2019) Alex Warstadt, Yu Cao, Ioana Grosu, Wei Peng, Hagen Blix, Yining Nie, Anna Alsop, Shikha Bordia, Haokun Liu, Alicia Parrish, Sheng-Fu Wang, Jason Phang, Anhad Mohananey, Phu Mon Htut, Paloma Jeretic, and Samuel R. Bowman. 2019.  Investigating BERT’s knowledge of language: Five analysis methods with NPIs.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2877–2887, Hong Kong, China. Association for Computational Linguistics.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1112–1122. Association for Computational Linguistics.
  * Wu et al. (2020) Wei Wu, Fei Wang, Arianna Yuan, Fei Wu, and Jiwei Li. 2020.  CorefQA: Coreference resolution as query-based span prediction.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 6953–6963, Online. Association for Computational Linguistics.
  * Yanaka et al. (2019) Hitomi Yanaka, Koji Mineshima, Daisuke Bekki, Kentaro Inui, Satoshi Sekine, Lasha Abzianidze, and Johan Bos. 2019.  HELP: A dataset for identifying shortcomings of neural models in monotonicity reasoning.  In _Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM 2019)_ , pages 250–255, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Yang et al. (2019) An Yang, Quan Wang, Jing Liu, Kai Liu, Yajuan Lyu, Hua Wu, Qiaoqiao She, and Sujian Li. 2019.  Enhancing pre-trained language representations with rich knowledge for machine reading comprehension.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 2346–2357, Florence, Italy. Association for Computational Linguistics.

##  Appendix A Negation Cue Detection

We develop a negation cue detector (Section 3 in the paper) by utilizing the
RoBERTa (base architecture; 12 layers) pre-trained model Liu et al. (2019). We
fine-tune the system on ConanDoyle-neg Morante and Daelemans (2012) corpus.
While fine-training, the negation cues are marked with BIO (B: Beginning of
cue, I: Inside of cue, O: Outside of cue) tagging scheme. The contextualized
representations from the last layer of RoBERTa are passed to a fully connected
(FC) layer. Finally, a conditional random field (CRF) layer produces the
output sequence for the labels.

Our model yields the following results on the test set: 93.26 Precision, 94.32
Recall, and 93.79 F1. The neural model takes about two hours on average to
train on a single GPU of NVIDIA Tesla K80. A list of the tuned hyperparameters
that the model requires to achieve the above results is provided in Table 5.
The code is available at https://github.com/mosharafhossain/negation-and-nlu.

Hyperparameter |
---|---
Max Epochs | 50
Batch Size | 10
Learning Rate (RoBERTa) | 1e-5
Learning Rate (FC, CRF) | 1e-3
Weight Decay (RoBERTa) | 0.00001
Weight Decay (FC) | 0.001
Grad Clipping | 5.0
Warmup Epochs | 5
Patience | 15
Dropout | 0.5
Table 5: Hyperparameters used to fine-tune the cue detector with ConanDoyle-neg Morante and Daelemans (2012) corpus. FC and CRF refers to fully connected and conditional random field layers, respectively.  | Hp-1 | Hp-2 | Hp-3
---|---|---|---
CmmnsnsQA | 10 | 16 | 1e-5
COPA | 50 | 16 | 1e-5
QQP | 3 | 16 | 1e-5
STS-B | 10 | 16 | 1e-5
QNLI | 3 | 8 | 1e-5
WiC | 10 | 16 | 1e-5
WSC | 200 | 16 | 1e-6
SST-2 | 3 | 16 | 1e-5
Table 6: Hyperparameters used to fine-tune RoBERTa individually for each
corpus. Hp-1, Hp-2, and Hp-3 refer to the number of epochs, batch size, and
learning rate used in the training procedure. We use default settings for the
other hyperparameters when we use the implementation by Phang et al. (2020).

##  Appendix B Hyperparameters to Fine-tune the System for Each of the NLU
Tasks

We use an implementation by Phang et al. (2020) and fine-tune RoBERTa (base
architecture; 12 layers) Liu et al. (2019) model separately for each of the
eight corpora. We use the default settings of the hyperparameters, except for
a few, when fine-tuning the model on each benchmark. Table 6 shows tuned
hyperparameters for each benchmark.


# The Illusion of Competence: Evaluating the Effect of Explanations on Users’
Mental Models of Visual Question Answering Systems

Judith Sieker†, Simeon Junker†, Ronja Utescher†, Nazia Attari†,
Heiko Wersing‡, Hendrik Buschmeier∥, Sina Zarrieß†
†Computational Linguistics Group, Linguistics Department, Bielefeld University
‡Honda Research Institute Europe
∥Digital Linguistics Lab, Linguistics Department, Bielefeld University

###### Abstract

We examine how users perceive the limitations of an AI system when it
encounters a task that it cannot perform perfectly and whether providing
explanations alongside its answers aids users in constructing an appropriate
mental model of the system’s capabilities and limitations. We employ a visual
question answer and explanation task where we control the AI system’s
limitations by manipulating the visual inputs: during inference, the system
either processes full-color or grayscale images. Our goal is to determine
whether participants can perceive the limitations of the system. We
hypothesize that explanations will make limited AI capabilities more
transparent to users. However, our results show that explanations do not have
this effect. Instead of allowing users to more accurately assess the
limitations of the AI system, explanations generally increase users’
perceptions of the system’s competence – regardless of its actual performance.

The Illusion of Competence: Evaluating the Effect of Explanations on Users’
Mental Models of Visual Question Answering Systems

Judith Sieker†, Simeon Junker†, Ronja Utescher†, Nazia Attari†, Heiko
Wersing‡, Hendrik Buschmeier∥, Sina Zarrieß† †Computational Linguistics Group,
Linguistics Department, Bielefeld University ‡Honda Research Institute Europe
∥Digital Linguistics Lab, Linguistics Department, Bielefeld University

##  1 Introduction

Machine learning-based technologies (often called ‘artificial intelligence’,
AI) are now commonly being deployed and used in real-world applications,
influencing human decision-making (or automating decision-making altogether)
with implications for societies, organizations, and individuals. Despite
continuous advances and impressive performance on many tasks, these
technologies are not always accurate and will likely never be. Machine
learning models depend on curation of the data they are trained on, they are
optimized according to criteria that may not do justice to the complexity of
reality, and the context in which they are used cannot be fully modeled, to
name a few reasons for their limitations. In addition, the underlying
algorithms themselves have inherent weaknesses. Large language models (LLMs),
e.g., are well known to hallucinate, i.e., to make predictions that are
inconsistent with facts or themselves (Ji et al., 2023), or to be highly
sensitive to spurious variations in their inputs/prompts (Sclar et al., 2023).

Many machine learning models also suffer from their own complexity: consisting
of millions, billions, or even trillions of parameters, they are black-boxes,
opaque to human understanding. However, in order to reliably use machine
learning models and AI systems based on such models, human users must be able
to assess their limitations and deficiencies, and to understand the decisions
that such systems make and why (codified, for example, as the right “to obtain
an explanation of the decision reached” in the legal framework of the General
Data Protection Regulation of the European Union; GDPR, 2016, Recital 71).
Research in Explainable AI (XAI) addresses this need, and recent years have
seen an explosion of explainability methods that aim to make the internal
knowledge and reasoning of AI systems transparent and explicit, and thus
interpretable and accessible to users. Explainability of model predictions is
thus seen as a solution, and it is assumed that they enable users to construct
functional ‘mental models’ (Norman, 1983) of AI systems, i.e., models that
closely correspond to the actual capabilities of the systems.

Whether this is the case is an active research question and there is evidence
that explainability comes with new challenges. Important questions in XAI are
what actually makes a good explanation, which criteria it needs to satisfy,
and how the quality of explanations can be measured (Alshomary et al., 2024).
Furthermore, recent perspectives emphasize that explanations should be social
(Miller, 2019) and constructed interactively, taking into account the user’s
explanation needs (Rohlfing et al., 2021). Jacovi and Goldberg (2020) argue
that evaluations of explanations should carefully distinguish plausibility
(does it seem plausible to users) and faithfulness (does it reflect the
model’s internal reasoning) and that non-faithful, but plausible, explanations
can be dangerous in that they let users construct faulty, and eventually
dysfunctional, mental models that can lead to unwarranted trust (Jacovi et
al., 2021).

In this paper, we investigate the effects of providing natural language
explanations on users’ mental models of an AI system in terms of its
capabilities, and whether these explanations allow them to diagnose system
limitations. We present the results of a study in the visual question
answering and explanation (VQA/X) domain, artificially inducing a simple
limitation by providing two VQA/X systems with images stripped of color
information, i.e., in grayscale (see Figure 1). Participants, unaware of the
manipulation, see the unmanipulated full color image, the question, the
system’s answer, and its explanation for the answer, and have to judge various
system capabilities (including its ability to recognize colors) and its
competence. This visual domain does not require participants to understand the
internal processes of the system but should still enable them to estimate what
it can and cannot do. The comparison of judgments to responses to non-
manipulated system input and judgments of responses without explanations sheds
light on participants’ difficulties in using (natural language) XAI
explanations to build accurate mental models, even for such a simple case.
This raises the question of how effective explanations can be in real-world
applications of XAI technology that involve more complex reasoning and
problems.

##  2 Background

Our work is related to previous studies that have examined whether
explanations enhance users’ trust in AI systems. Kunkel et al. (2019), for
example, compared trust in personal (human) versus impersonal (recommender
system) recommendation sources and examined the impact of explanation quality
on trust. Their results showed that users rated human explanations higher than
system-generated ones and that the quality of explanations significantly
influenced trust in the recommendation source. Bansal et al. (2021)
investigated whether explanations help humans anticipate when an AI system is
potentially incorrect. They used scenarios where an AI system helps
participants to solve a task (text classification or question answering),
providing visual explanations (highlighted words) under certain conditions.
Their findings revealed that explanations increased the likelihood of the
participants to accept the AI system’s recommendations, irrespective of their
accuracy. Thus, rather than fostering appropriate reliance on AI systems,
explanations tended to foster blind trust. Similarly, (Kim et al., 2021)
conducted a large-scale user study for visual explanations, showing that these
do not allow users to distinguish correct from incorrect predictions.
Dhuliawala et al. (2023) investigated how users develop and regain trust in AI
systems in human–AI collaborations. They found that NLP systems that
confidently make incorrect predictions harm user trust, and that even a few
incorrect instances can damage trust, with slow recovery. While these studies
evaluate the influence of system explanations on users’ trust in the system’s
output (a proxy for its perceived competence), they do not investigate users’
understanding of the systems’ reasoning processes and capabilities. In our
study, we specifically address this issue and investigate the users’ mental
model of the systems’ capabilities and limitations.

Figure 1: Items from our study: Answers and explanations generated with NLX-
GPT for color/grayscale images in VQA-X (top) and CLEVR-X (bottom).
Explanations in the grayscale condition refer to colors that were not
available in the system inputs (_green, yellow_).

While the studies above found that nonverbal explanations can be misleading to
users, natural language explanations are assumed to be more transparent or
less difficult to interpret (Park et al., 2018; Salewski et al., 2022). Verbal
explanations also offer the advantage that they can be collected from humans,
which has led to the development of explanation benchmarks, particularly in
multimodal domains (Kayser et al., 2021; Salewski et al., 2022). Thus, the
dominant approach to verbal explanation generation currently is to leverage
human explanations during model training (Park et al., 2018; Wu and Mooney,
2019; Kayser et al., 2021; Plüster et al., 2023; Sammani and Deligiannis,
2023). While Lyu et al. (2024) discuss potential faithfulness issues related
to supervising explanation generation with human explanations, we are not
aware of work that explicitly tests these supervised models in a user-centered
setting similar to ours.

##  3 Approach

Figure 2: Illustration of our experimental design and hypotheses. In Exp.A, we
do not expect users to spot the system defect (no color recognition due to
grayscale input) since only answers are provided. In Exp.X, the system
provides explanations which should help users in building a better mental
model.

We conduct a study to investigate how users of an AI system perceive its
limitations when it encounters tasks that it cannot perform perfectly. We aim
to investigate whether providing explanations alongside model responses helps
users build an appropriate mental model of the AI system’s capabilities and
limitations. We control the AI system’s limitations by systematically
manipulating its inputs. We design a questionnaire for users to judge specific
aspects of the AI system’s capabilities. This allows us to measure whether
users can diagnose which capabilities of the AI system have been perturbed
through our explicit input manipulations. The design of our study is
summarized in Figure 2 and will be explained in detail below.

#### VQA Task and Abilities

We employ a visual question answering and explanation task: the input to the
AI system is an image and a question in natural language, and its task is to
generate an answer and a natural language explanation that justifies the
answer. We select a visual question-answering setting as it is a rather simple
task for humans and, at the same time, a task that involves distinguishable
semantic-visual reasoning capabilities. This is important for our setting
since we want to test whether users can differentiate specific system
capabilities, based on generated explanations. Thus, inspired by Salewski et
al.’s (2022) CLEVR-X benchmark for explainable VQA, we assume that these
capabilities involve the abilities to process objects’ (i) color, (ii) shape,
(iii) material, and (iv) scene composition (e.g., spatial relations, relative
size). In our study participants are asked to rate the AI system’s
capabilities along these four dimensions, next to other, more general criteria
for competence and fluency (see Figures 8 and 9 in Appendix A.4). In the
CLEVR-X benchmark, these dimensions are given by construction: the visual
scenes are synthetically generated and composed of objects defined by
attributes for color, material, and shape. The corresponding questions
explicitly relate to one or multiple of these dimensions. In real-world image
benchmarks, such as VQA-X (Park et al., 2018), these abilities are often more
implicit, but still highly relevant (see examples in Figure 1). We run our
study on items from both benchmarks.

#### Color vs. Grayscale Input

Our goal is to investigate whether explanations help users in diagnosing
system limitations. To introduce these limitations in a controlled way, we
manipulate the input of the VQA systems. Out of the four VQA capabilities
explained above (color, shape, material, and scene), the color dimension lends
itself to straightforward manipulation: during inference, systems either
receive the image (i) in full color or (ii) in grayscale. This induced
limitation resembles a situation where a multimodal AI model was trained on
colored images but, at run-time, a camera/visual sensor is broken such that
model inputs are perturbed. To make sure that this manipulation induces an
incorrect model response, we only include items that are correctly answered
with the full color image input _but_ incorrectly answered with the grayscale
image input. This item selection accounts for the fact that VQA models can be
assumed to have further limitations that we cannot explicitly control for and
exclude items (i) where the VQA does not generate the correct ground-truth
answer for the colored image, and (ii) where the VQA generates the correct
answer for the grayscale image. This gives us a clean set of items where the
limitations of the AI system can be attributed to a particular error source.
The participants in our study were unaware of the underlying color–grayscale
manipulation: they saw images in color, along with the models’ answers and
explanations. Our goal was to determine whether participants were able
perceive the limitations of the model, i.e., whether they could identify the
system’s lack of color recognition ability. See Figure 2 for an illustration
of this set-up.

#### Experiments A and X

To investigate the effect of providing generated explanations alongside the
system answers, we conduct two separate studies: In Experiment X, participants
were shown both the answer and its explanation, whereas in Experiment A
participants were shown only the answer without an explanation. In both
studies, we ask participants to rate each item for the system’s capabilities
(color, shape, material, scene), the overall system competence, answer
correctness, the consistency of answer/explanation, the consistency of
explanation/image, and the explanation’s fluency.

Importantly, participants in both Experiments A and X received mixed sets of
items from all systems, data sets, and color conditions, and we collected
judgments for each item. In this way, we wanted to prevent them from becoming
“conditioned” to a particular setting, i.e., getting used to certain ways of
answering or explaining and becoming overly sensitive to changes in patterns.

If explanations lead users to build more appropriate mental models,
participants should, generally speaking, be able to differentiate items where
systems processed grayscale vs. full color images. We approached this broad
expectation with five hypotheses specific to our set-up (see Table 2 for a
brief summary). First, hypotheses H1A and H1X relate to the differences in
competence scores between color and grayscale conditions. Here, we expect that
explanations help participants to differentiate between different system
capabilities.

H1A

In Exp.A, competence and all capability scores are lower in the grayscale
condition than in the color condition.

H1X

In Exp.X, competence and color capability scores are lower in the grayscale
condition than in the color condition, but other capability scores are more
stable.

Hypotheses H2A and H2X are concerned with the comparison between individual
competence scores in the grayscale condition. Again, explanations should help
users to identify system deficiencies.

H2A

In the grayscale condition of Exp.A, participants give similar scores for all
capabilities.

H2X

In the grayscale condition of Exp.X, participants rate the color capability
lower relative to the other capabilities.

Hypothesis H3A/X pertains to the comparison of competence scores between Exp.A
and X. If explanations make defects in color processing transparent, grayscale
inputs should specifically affect scores for this dimension.

H3A/X

In Exp.X the overall competence is rated higher than in Exp.A. In Exp.X, color
competence is rated lower or the same as in Exp.A.

##  4 Experimental Setup

#### Data

We use two datasets in our study: VQA-X (Park et al., 2018) and CLEVR-X
(Salewski et al., 2022). VQA-X is extensively utilized in Visual Question
Answering (VQA) tasks, as an extension of the well-established Visual Question
Answering v1 (Antol et al., 2015) and v2 (Goyal et al., 2017) datasets. The
images within VQA-X originate from MSCOCO (Lin et al., 2015), and the
questions are open-ended (see Figure 1, top). The style of the ground-truth
explanations in VQA-X varies widely, ranging from simple image descriptions to
detailed reasoning (Salewski et al., 2022).

CLEVR-X expands the synthetic dataset CLEVR (Johnson et al., 2017),
incorporating synthetic natural language explanations. Each image in the CLEVR
dataset depicts three to ten objects, each possessing distinct properties
including size, color, material, and shape (see Figure 1, bottom). For each
image–question pair in the CLEVR dataset, CLEVR-X contains multiple structured
textual explanations. These explanations are constructed from the underlying
scene graph, ensuring their accuracy without necessitating additional prior
knowledge.

#### Models

For each dataset, we used two vision and language models: (i) NLX-GPT (Sammani
et al., 2022) and PJ-X (Park et al., 2018) for VQA-X, and (ii) NLX-GPT and
Uni-NLX (Sammani and Deligiannis, 2023) for CLEVR-X111We tried to obtain model
outputs from other explainable VQA-X models such as, e.g., OFA-X (Plüster et
al., 2023), FME (Wu and Mooney, 2019), or e-UG (Kayser et al., 2021), but
encountered significant reproducibility issues: code was unavailable or not
running, authors were unavailable to provide model outputs, etc. . We did not
use vanilla generative AI systems (such as ChatGPT) in this study, as we
wanted to investigate models that were specifically constructed to provide
explanations alongside their outputs.

NLX-GPT is an encoder–decoder model, which combines CLIP (Radford et al.,
2021) as the visual encoder with a distilled GPT-2 model (Radford et al.,
2019). Importantly, this model jointly predicts answers and explanations,
i.e., it generates a single response string of the form “the answer is
<answer> because <explanation>”, given a question and image. For VQA-X, we use
the model from Sammani et al. (2022), which is pre-trained on image-caption
pairs and fine-tuned on the VQA-X data. For CLEVR-X, we use the published pre-
trained weights and fine-tune the model on this dataset. Uni-NLX relies on the
same architecture as NLX-GPT, but the model is trained on various datasets for
natural language explanations (including VQA-X), to leverage shared
information across diverse tasks and increase flexibility in both answers and
explanations. We take the trained model from Sammani and Deligiannis (2023)
and fine-tune it on CLEVR-X. While NLX-GPT and Uni-NLX generate answers and
explanations simultaneously, the PJ-X model takes a two-step approach. It
first predicts the answer with an answering model and, subsequently, generates
visual and textual explanations based on the question, image, and answer222We
could not replicate Salewski et al.’s (2022) PJ-X results on CLEVR-X, and the
authors could not provide model outputs. Therefore, we only report PJ-X on
VQA-X..

For each model, we utilize the recommended model weights and fine-tune them on
the two datasets. During fine-tuning, we supply each model with the original,
i.e., full color images along with the questions, answers, and explanations
for both datasets. During inference, images are presented in color alongside
the question, or in grayscale.

#### User Study

We conducted the study online, using Prolific, and obtained ratings from 160
participants (80 each in Exp.A and X) who were native English speakers with
normal color vision (selected using Prolific’s filters). In both experiments,
we utilized identical experimental items, differing only in the presence or
absence of explanations. All items consisted of instances where the model
provided correct answers for colored images and incorrect answers for
grayscale images. We selected a total of 128 items, evenly distributed across
the datasets and models, comprising 64 for each dataset and 32 for each model,
equally split between 16 colored and 16 grayscale items (for NLX-GPT, a total
of 64 items were selected, with 32 items from CLEVR-X and 32 items from
VQA-X). The items were distributed over four experimental lists, with each
participant evaluating 32 individual items. We gathered 2560 judgments per
experiment and 5120 overall.

We designed the evaluation as a rating task. We informed participants that we
are assessing an AI system’s ability to answer questions about images (and,
for Exp.X, to generate explanations). The image, question, and answer for each
item were presented at the top of the page, and, in Exp.X, the generated
explanation was displayed below the answer. Each item had several questions
and statements for the participants to assess. First, they were asked to
evaluate the correctness of the answer. In Exp.X, participants were further
asked to assess whether the explanation was (i) consistent with the answer,
(ii) consistent with the picture, and (iii) overall fluent. Additionally,
participants in both experiments were asked to judge whether they believed
that the AI system correctly identifies (iv) shapes, (v) colors, and (vi)
materials, as well as whether it (vii) understands the general scene in the
image. Finally, (viii) participants judged the overall competence of the
system. Participants indicated their agreement on five-point Likert scales,
ranging from 1 (‘strongly disagree’) to 5 (‘strongly agree’). For each
criterion, we also offered the option of selecting “I don’t know”. Before
providing ratings, participants received instructions and viewed an example
item illustrating the evaluation criteria. They were paid at a rate of £9.00
per hour. See Appendix A.3 for example trials of the experiment.

##  5 Results

We organize the discussion of results based on the hypotheses outlined in
Section 3. Since we ask whether explanations help participants determine that
the systems could not recognize color, the following discussion concentrates
on the grayscale condition and the differences between the grayscale and color
conditions (see Appendix A.3 for detailed results of the color condition).

(a) Exp.A – colored images

(b) Exp.A – grayscale images

(c) Exp.X – colored images

(d) Exp.X – grayscale images

Figure 3: Human ratings on the evaluation criterion “Ability of the AI to
recognize colors”. Participants indicated their judgment on a scale from 1
(strongly disagree; here in red) to 5 (strongly agree; here in green).

All systems received high ratings in all competency and capability dimensions
when tested in the color condition of Exp.A and X, on both datasets (see Table
9 in Appendix A.3). These ratings decreased in very similar ways in the
grayscale condition. Therefore, we were able to use all items from all systems
to test our hypotheses, generalizing over minor system differences. We discuss
differences between datasets and models in Appendix A.3, since these were not
essential for testing our hypotheses. Summaries of hypotheses and results are
given in Table 2.

#### Hypotheses H1A and H1X

state our expectations on distinctions between the grayscale and color
conditions in Exp.A and X, respectively. Figure 3 shows the distribution of
participant ratings for the AI system’s ability to recognize colors, for the
grayscale and color conditions in both experiments (see Figures 4, 5, 6, and 7
in Appendix A.3 for results on the other capabilities). In Exp.A and X, there
is a consistent trend of better assessments when systems have been seen the
color images compared to grayscale images, across different systems, datasets,
and all capabilities. Most users rate the color capability with the highest
rating in the color condition (Figure 3a/c) and with the lowest rating in the
grayscale condition (Figure 3b/d). The same holds for all other capabilities
and competency (Figures 4, 5, 6, and 7). This confirms hypothesis H1A, i.e.,
ratings for all capabilities decrease when the system does not see color.
However, this does not support H1X, as we expected that only overall
competence and capability to recognize colors would be rated lower in the
grayscale condition when explanations were given, and not all capabilities.
This suggests that the AI’s explanations did not help users diagnose the
system’s limitation in the grayscale condition, as all capability dimensions
are similarly affected in Exp.X.

|  |  | Colors | Shapes | Materials | General Scene | Competency
---|---|---|---|---|---|---|---
Experiment | Dataset | Model |  med |  mean |  med |  mean |  med |  mean |  med |  mean |  med |  mean
Exp.A | CLEVR-X | NLX-GPT |  1.0 |  1.69 |  1.0 |  2.08 |  1.0 |  1.94 |  1.5 |  1.97 |  1.0 |  1.68
|  | Uni-NLX |  1.0 |  1.84 |  2.0 |  2.31 |  1.0 |  2.11 |  2.0 |  2.16 |  1.0 |  1.91
| VQA-X | NLX-GPT |  1.0 |  1.73 |  2.0 |  2.23 |  1.0 |  1.71 |  1.0 |  1.87 |  1.0 |  1.64
|  | PJ-X |  1.0 |  1.71 |  2.0 |  2.08 |  1.0 |  1.74 |  1.0 |  1.83 |  1.0 |  1.60
Exp.X | CLEVR-X | NLX-GPT |  1.0 |  1.93 |  3.0 |  2.95 |  2.0 |  2.62 |  2.5 |  2.61 |  2.0 |  2.13
|  | Uni-NLX |  2.0 |  2.27 |  3.0 |  2.89 |  3.0 |  2.82 |  2.0 |  2.61 |  2.0 |  2.21
| VQA-X | NLX-GPT |  2.0 |  2.36 |  3.0 |  2.70 |  2.0 |  2.32 |  2.0 |  2.29 |  2.0 |  1.96
|  | PJ-X |  2.0 |  2.25 |  2.0 |  2.53 |  2.0 |  2.32 |  2.0 |  2.23 |  2.0 |  1.88
Table 1: Human ratings on system capabilities for the grayscale condition of
Exp.A (no explanations) and Exp.X (with explanations), as median and mean
scores across raters.

#### Hypotheses H2A and H2X

state our expectations for the grayscale condition. Table 1 presents the human
evaluation results in Exp.A and X. Starting with Exp.A, Table 1 shows that all
evaluation criteria in the grayscale condition receive relatively low scores.
Interestingly, the manipulated capability, i.e., to recognize colors, does
have slightly worse ratings than the other criteria (for most models and
datasets). This outcome does not align with our expectation (H2A) as
participants in Exp.A solely viewed the answers without access to
explanations, making it difficult to discern which specific ability or
(limitation) influenced the model’s answer. Results from Mann-Whitney U tests
(see Table 4 in Appendix A.2) show significant differences between the ability
to recognize colors and the ability to recognize other criteria for Exp.A
(except for the models’ overall competence), contradicting hypothesis (H2A).
This suggests that users in Exp.A were able to interpret incorrect system
answers more than we expected.

H1A |  competence and all capabilities rated lower in grayscale cond. than in color cond. in Exp.A | ✔
---|---|---
H1X |  competence and color capability rated lower in grayscale cond. than in color cond. in Exp.X | ✗
H2A |  similar ratings for color compared to other capabilities, in grayscale cond. in Exp.A | ✗
H2X |  lower ratings for color compared to other capabilities, in grayscale cond. in Exp.X | (✓)
H3A/X |  competence rated higher for grayscale cond. in Exp.X than in Exp.A, color rated lower | (✓/✗)
Table 2: Overview of the validity of the hypotheses formulated in Section 3.

For Exp.X, the results in Table 1 suggest a very similar trend to Exp.A: the
ability to recognize colors is rated slightly lower than the other
capabilities. The Mann-Whitney U tests for Exp.X (reported in the lower part
of Table 4 in Appendix A.2 ), again confirms significant differences between
the perceived ability to recognize colors and the other abilities (except the
systems’ overall competence). Looking at Exp.X in isolation, these results
seem to speak in favor of our hypothesis H2X: users were indeed able to
diagnose the system defect, at least to some extent. However, in light of our
findings on H2A, these results have to be interpreted with care: even without
model explanations, users rated the color capability lower than others. This
trend is a bit stronger in Exp.X but, overall, the differences between
perceived capabilities are still rather small. The strongest expected trend in
favor of H2X can be found for NLX-GPT on the CLEVR-X data: here, the median if
the color rating is $1.0$ and $3.0$ or $2.0$ for the other capabilities. For
the other combinations of models and datasets in Exp.X, there is no clear
difference in the median ratings for the perceived capabilities. We conclude
that there is weak evidence in favor of H2X, as explanations do not
substantially improve users’ assessments of system capabilities.

#### Hypothesis H3A/X

states our expectations regarding the differences between Exp.A and X for
overall competency and color recognition ability.

Once again, consider Table 1. As expected, in Exp.A, i.e., without
explanations, the overall competency of the models was rated low (with median
values of $1.0$ only). In Exp.X, although the values remain low at 2.0, there
is a noticeable improvement relative to Exp.A. Thus, despite the answers being
incorrect, the addition of the models’ explanations enhances the perception of
the models’ overall competency. This could suggest that the explanations
reveal other capabilities of the models, consistent with our hypothesis H3A/X.
However, contrary to H3A/X, we also see a general increase in the ratings for
the systems’ color recognition ability in Exp.X compared to Exp.A. We expected
that the explanations would make the color limitation explicit, which would
result in color ability being rated worse or at least as poorly as in Exp.A.
This also holds for _all_ other model capabilities: all capability ratings are
comparatively higher in Exp.X than in Exp.A (even if lower than in the color
condition). This observation is supported by the Mann-Whitney U tests (see the
upper part of Table 4 in Appendix A.2), which show significant differences
between Exp.A and X for all evaluation criteria. This suggests that users rate
all system capabilities significantly higher when explanations are provided.
From this we conclude that, instead of making systems’ limitations more
transparent, the explanations contribute to an overall more positive
perception of the system, regardless of its capabilities. In other words, the
AI system’s explanations seem to create an illusion of the system’s competence
that does not correspond to its actual performance.

#### Automatic Evaluation

In the VQA-X domain, automatic measures for evaluating similarity or overlap
with human ground-truth explanations are commonly used (cf. Salewski et al.,
2022; Sammani and Deligiannis, 2023). To assess the construct validity of a
representative automatic evaluation method, we compute BERTScores, measuring
the similarity of ground truth explanations from both datasets to human
evaluation scores. Table 3 reports the results of the BERTscore metric,
showing that they do not exhibit any notable differences between the grayscale
and color conditions, which clearly contradicts the results of our human
investigation. Thus, while user ratings between the grayscale and color
condition are located on opposite ends on the Likert scale, BERTscores show
marginal differences across the board. Yet, when comparing the two datasets,
the BERTScores for the CLEVR-X dataset show improved values (in both the
grayscale and color conditions), aligning with the human results from Exp.X
(see Table 1 and 9 in Appendix A.3).

|  | BERTScore
---|---|---
Dataset |  Model |  color |  grayscale
CLEVR-X |  NLX-GPT |  0.76 |  0.74
|  Uni-NLX |  0.75 |  0.74
VQA-X |  NLX-GPT |  0.72 |  0.72
|  PJ-X |  0.71 |  0.70
Table 3: BERTScores for explanations by condition.

#### Summary

Table 2 provides an overview of the validity of our hypotheses. Generally, our
results show that explanations do not have a desirable effect on users’
assessment of the system’s competency and capabilities. They do not help users
construct a more accurate mental model of the system and its capabilities and
limitations, but simply lead to more positive user assessment overall. Our
results are strikingly consistent across models and datasets. Even systems
fine-tuned on the CLEVR-X benchmark, where explanations were designed to
systematically mention the capabilities we assessed in our study (including
color), do not address these limitations. Figure 1 shows representative
examples of why this might be the case: rather than avoiding color words or
using incorrect colors, systems seem to be able to guess the correct color
from the question or the general context (e.g., _green_ in the context of
_tree_). This behavior is well-known in multimodal language models but should
be avoided in explanation tasks since it counteracts transparency and
appropriate user assessment.

##  6 Discussion of Implications

It is still not well understood how XAI can bridge the gap between highly
complex black-box models with largely opaque internal reasoning processes and
users’ intuitive understanding of these. Generally, our study provides
evidence that explanations generated by state-of-the-art systems do not always
lead to the expected effects of XAI and that explanations may even further
obstruct AIs’ reasoning processes and trick users into believing that the AI
is more competent than it actually is. This result is particularly noteworthy
in light of the fact that the manipulation employed in our study introduced an
obvious error that should be easy to spot for users (defects in systems’ color
recognition).

#### XAI Models

Our study underlines the great importance of prioritizing faithfulness over
plausibility in explanation methods (Jacovi and Goldberg, 2020). With today’s
AI systems and LLMs, users face the challenging situation that these systems
present fluent outputs projecting confidence and competence. Yet, this
confidence may not be grounded in actual system capabilities and reliability
(Guo et al., 2017). Our findings suggest that this also holds, to some extent,
for state-of-the-art approaches to natural language explanation generation.
Looking at the architecture of these models, this is by no means surprising.
At least within the domain of VQA-X, which we focused on in this paper,
explanation generation approaches largely follow common language modeling
architectures and prioritize generating fluent, human-like outputs. Despite
the fact that the importance of faithfulness in XAI has been recognized for
some time and it continues to be a challenge (Lyu et al., 2024).

#### Evaluation of XAI

Our study also highlights the importance of evaluating explanation methods in
thorough, detailed, and user-centered ways (cf. Lopes et al., 2022). In the
domain of VQA-X, automatic, benchmark-based evaluations still seem to be in
focus and widely accepted in the community. All systems we tested in our study
have been assessed mainly in automatic evaluations (cf. Park et al., 2018;
Kayser et al., 2021; Sammani et al., 2022; Sammani and Deligiannis, 2023).
This stands in stark contrast to research showing that XAI evaluations often
have little construct validity, i.e., do not assess the intended properties of
explanations (Doshi-Velez and Kim, 2017; van der Waa et al., 2021). Our
BERTscore-results lend further support to this argument.

##  7 Conclusion

This paper investigates the effects of providing natural language explanations
on users’ ability to construct accurate mental models of AI systems’
capabilities, and whether these explanations allow them to diagnose system
limitations. Results from two experiments show that natural language
explanations generated by state-of-the-art VQA-X systems may actually hinder
users from accurately reflecting capabilities and limitations of AI systems.
Participants who received natural language explanations projected more
competence onto the system and rated its limited capabilities higher than
those who did not receive explanations.

## Limitations

We identify the following limitations in our work:

The addition of further models and data sets might have provided additional
insights into our experiments. Unfortunately, recently research on generating
natural language explanations has not been very active. The best known
approaches are models like PJ-X (Park et al., 2018) or e-UG (Kayser et al.,
2021), which have older code bases with reproducibility issues. We have tried
to include other models (see Section 4, footnotes 1 and 2).

For the grayscale condition, we remove color information at the inference
level for models trained on colored input. An alternative approach would be
altering inputs during model training, possibly leading to deficiencies that
are harder to identify for participants. Similarly, other kinds of
perturbations such as altering relative object sizes or scene layouts might
affect different dimensions of perceived system capabilities than color
recognition. Here, we focused on color, as this property is easier to control
and less intertwined with other properties than, e.g., object size (which
might also change how relative positions are described).

## Ethics Statement

Our study focuses on user-centered evaluation of XAI systems and on
understanding whether these systems fulfill the promise of making black-box AI
systems more transparent for users. Therefore, we believe that our study
contributes to understanding and improving the social and ethical implications
of recent work in NLP, and Language & Vision. In our study, we collect ratings
from Prolific users but, other than that, did not record any personal
information on these users.

## References

  * Alshomary et al. (2024) Milad Alshomary, Felix Lange, Meisam Booshehri, Meghdut Sengupta, Philipp Cimiano, and Henning Wachsmuth. 2024.  Modeling the quality of dialogical explanations.  In _Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation_ , pages 11523–11536, Torino, Italy.
  * Antol et al. (2015) Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015.  VQA: Visual question answering.  In _Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV)_ , pages 2425–2433, Santiago, Chile.
  * Bansal et al. (2021) Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel Weld. 2021.  Does the whole exceed its parts? The effect of AI explanations on complementary team performance.  In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_ , Yokohama, Japan.
  * Dhuliawala et al. (2023) Shehzaad Dhuliawala, Vilém Zouhar, Mennatallah El-Assady, and Mrinmaya Sachan. 2023.  A diachronic perspective on user trust in AI under uncertainty.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 5567–5580, Singapore.
  * Doshi-Velez and Kim (2017) Finale Doshi-Velez and Been Kim. 2017.  Towards a rigorous science of interpretable machine learning.  _Preprint_ , arxiv:1702.08608.
  * GDPR (2016) GDPR. 2016.  Regulation (EU) 2016/679 of the European Parliament and of the Council.  _Official Journal of the European Union_ , L 119:1–88.
  * Goyal et al. (2017) Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017.  Making the V in VQA matter: Elevating the role of image understanding in visual question answering.  In _Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 6325–6334, Honolulu, HI, USA.
  * Guo et al. (2017) Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017.  On calibration of modern neural networks.  In _Proceedings of the 34th International Conference on Machine Learning_ , volume 70, pages 1321–1330, Syndney, Australia.
  * Jacovi and Goldberg (2020) Alon Jacovi and Yoav Goldberg. 2020.  Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness? In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4198–4205, Online.
  * Jacovi et al. (2021) Alon Jacovi, Ana Marasović, Tim Miller, and Yoav Goldberg. 2021.  Formalizing trust in artificial intelligence: Prerequisites, causes and goals of human trust in AI.  In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_ , pages 624–635, Virtual, Canada.
  * Ji et al. (2023) Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023.  Survey of hallucination in natural language generation.  _ACM Computing Surveys_ , 55(12):248:1–248:38.
  * Johnson et al. (2017) Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. 2017.  CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning.  In _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 1988–1997, Honolulu, HI.
  * Kayser et al. (2021) Maxime Kayser, Oana-Maria Camburu, Leonard Salewski, Cornelius Emde, Virginie Do, Zeynep Akata, and Thomas Lukasiewicz. 2021.  e-ViL: A dataset and benchmark for natural language explanations in vision-language tasks.  In _2021 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 1224–1234, Montreal, Canada.
  * Kim et al. (2021) Sunnie S. Y. Kim, Nicole Meister, Vikram V. Ramaswamy, Ruth C. Fong, and Olga Russakovsky. 2021.  Hive: Evaluating the human interpretability of visual explanations.  In _European Conference on Computer Vision_.
  * Kunkel et al. (2019) Johannes Kunkel, Tim Donkers, Lisa Michael, Catalin-Mihai Barbu, and Jürgen Ziegler. 2019.  Let me explain: Impact of personal and impersonal explanations on trust in recommender systems.  In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_ , Glasgow, UK.
  * Lin et al. (2015) Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. 2015.  Microsoft COCO: Common objects in context.  _Preprint_ , arxiv:1405.0312.
  * Lopes et al. (2022) Pedro Lopes, Eduardo Silva, Cristiana Braga, Tiago Oliveira, and Luís Rosado. 2022.  XAI systems evaluation: A review of human and computer-centred methods.  _Applied Sciences_ , 12(19).
  * Lyu et al. (2024) Qing Lyu, Marianna Apidianaki, and Chris Callison-Burch. 2024.  Towards faithful model explanation in NLP: A survey.  _Computational Linguistics_ , 50(2):1–67.
  * Miller (2019) Tim Miller. 2019.  Explanation in artificial intelligence: Insights from the social sciences.  _Artificial Intelligence_ , 267:1–38.
  * Norman (1983) Donald A. Norman. 1983.  Some observations on mental models.  In Dedre Gentner and Albert L. Stevens, editors, _Mental Models_ , pages 7–14. Psychology Press, New York, NY, USA.
  * Park et al. (2018) Dong Huk Park, Lisa Anne Hendricks, Zeynep Akata, Anna Rohrbach, Bernt Schiele, Trevor Darrell, and Marcus Rohrbach. 2018.  Multimodal explanations: Justifying decisions and pointing to the evidence.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 8779–8788, Salt Lake City, UT, USA.
  * Plüster et al. (2023) Björn Plüster, Jakob Ambsdorf, Lukas Braach, Jae Hee Lee, and Stefan Wermter. 2023.  Harnessing the power of multi-task pretraining for ground-truth level natural language explanations.  _Preprint_ , arxiv:2212.04231.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021.  Learning transferable visual models from natural language supervision.  In _Proceedings of the 38th International Conference on Machine Learning_ , pages 8748–8763, Virtual.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.  _OpenAI Blog_ , 1(8):9.
  * Rohlfing et al. (2021) Katharina Rohlfing, Philipp Cimiano, Ingrid Scharlau, Tobias Matzner, Heike Buhl, Hendrik Buschmeier, Angela Grimminger, Barbara Hammer, Reinhold Häb-Umbach, Ilona Horwath, Eyke Hüllermeier, Friederike Kern, Stefan Kopp, Kirsten Thommes, Axel-Cyrille Ngonga Ngomo, Carsten Schulte, Henning Wachsmuth, Petra Wagner, and Britta Wrede. 2021.  Explanation as a social practice: Toward a conceptual framework for the social design of ai systems.  _IEEE Transactions on Cognitive and Developmental Systems_ , 13:717–728.
  * Salewski et al. (2022) Leonard Salewski, A Sophia Koepke, Hendrik P A Lensch, and Zeynep Akata. 2022.  CLEVR-X: A visual reasoning dataset for natural language explanations.  In Andreas Holzinger, Randy Goebel, Ruth Fong, Taesup Moon, Klaus-Robert Müller, and Wojciech Samek, editors, _xxAI – Beyond Explainable AI_ , pages 69–88. Springer, Cham, Switzerland.
  * Sammani and Deligiannis (2023) Fawaz Sammani and Nikos Deligiannis. 2023.  Uni-NLX: Unifying textual explanations for vision and vision-language tasks.  In _Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW)_ , pages 4636–4641, Paris, France.
  * Sammani et al. (2022) Fawaz Sammani, Tanmoy Mukherjee, and Nikos Deligiannis. 2022.  NLX-GPT: A model for natural language explanations in vision and vision-language tasks.  In _Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 8312–8322, New Orleans, LA, USA.
  * Sclar et al. (2023) Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2023.  Quantifying language models’ sensitivity to spurious features in prompt design or: How I learned to start worrying about prompt formatting.  _Preprint_ , arxiv:2310.11324.
  * van der Waa et al. (2021) Jasper van der Waa, Elisabeth Nieuwburg, Anita Cremers, and Mark Neerincx. 2021.  Evaluating XAI: A comparison of rule-based and example-based explanations.  _Artificial Intelligence_ , 291:103404.
  * Wu and Mooney (2019) Jialin Wu and Raymond Mooney. 2019.  Faithful multimodal explanation for visual question answering.  In _Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 103–112, Florence, Italy.

##  Appendix A Appendix

###  A.1 Materials Availability Statement

We used the following public resources in our work:

  * •

Source code for NLX-GPT is available from GitHub at
https://github.com/fawazsammani/nlxgpt

  * •

Source code for Uni-NLX is available from GitHub at
https://github.com/fawazsammani/uni-nlx/

  * •

Source code for PJ-X and VQA-X data is available from GitHub at
https://github.com/Seth-Park/MultimodalExplanations

  * •

COCO Images for VQA-X are available here: https://cocodataset.org/

  * •

CLEVR-X data is available from GitHub at
https://github.com/ExplainableML/CLEVR-X

  * •

CLEVR images for CLEVR-X are available here:
https://cs.stanford.edu/people/jcjohns/clevr/

Our source code and the data from the human evaluation study will be made
available in form of an accompanying data publication.

###  A.2 Statistical Tests

Table 4 shows the results of Mann-Whitney U tests in the grayscale condition.
The upper half of the table reports the differences in user ratings of system
capabilities (color, shape, material, scene) and overall competence between
Exp.A and X, all differences are highly statistically significant. The lower
half of the Table reports the differences in ratings with Exp.A and X. Table 5
reports the same tests for the color condition. Here, only the difference
between overall competence is statistically significant between Exp.A and X
while all system capabilities are rated similarly with or without
explanations. This further supports our finding that explanations enhance
user’s perception of system competence, regardless of the correctness of
system answers.

Criterion | U-statistic |  $p$-value
---|---|---
Colors | 488421.0 | $\mathbf{4.09\times 10^{-15}}$
Shapes | 460501.0 | $\mathbf{5.81\times 10^{-21}}$
Materials | 428263.0 | $\mathbf{3.06\times 10^{-32}}$
General Scene | 457629.0 | $\mathbf{3.38\times 10^{-22}}$
Competency | 464419.5 | $\mathbf{3.01\times 10^{-21}}$
Color / Shape (Exp.A) | 452212.0 | $\mathbf{1.64\times 10^{-15}}$
Color / Shape (Exp.X) | 506384.0 | $\mathbf{4.70\times 10^{-21}}$
Color / Material (Exp.A) | 510967.5 | $\mathbf{6\times 10^{-04}}$
Color / Material (Exp.X) | 548762.5 | $\mathbf{3.43\times 10^{-11}}$
Color / Gen. Scene (Exp.A) | 486718.0 | $\mathbf{1.70\times 10^{-06}}$
Color / Gen. Scene (Exp.X) | 557231.0 | $\mathbf{4.54\times 10^{-09}}$
Color / Comp. (Exp.A) | 538178.0 | 0.52
Color / Comp. (Exp.X) | 640143.5 | 0.73
Table 4: Mann-Whitney U test results for the grayscale conditions of Experiments A and X. In the upper part of the table, we measure whether the ratings of one evaluation criterion (e.g., the ability to recognize _colors_) of Exp.A differs significantly from the ratings of the same evaluation criterion from Exp.X. In the lower part of the table, we measure whether the ratings of the color criterion differ significantly from the ratings of the other evaluation criteria. $p$-values in bold indicate statistical significance ($p<0.001$), the smallest $p$-value is underlined. Criterion | U-statistic |  $p$-value
---|---|---
Colors | 627628.0 | 0.77510
Shapes | 632776.5 | 0.49522
Materials | 606350.0 | 0.17573
General Scene | 647675.0 | 0.06266
Competency | 678234.5 | 0.00003
Colors / Shapes (Exp.A) | 594055.5 | 0.23511
Colors / Shapes (Exp.X) | 706324.0 | 0.14946
Colors / Materials (Exp.A) | 626865.0 | 0.00012
Colors / Materials (Exp.X) | 717614.5 | 0.02390
Colors / Gen. Scene (Exp.A) | 569399.0 | 0.84294
Colors / Gen. Scene (Exp.X) | 710226.5 | 0.08423
Colors / Competency (Exp.A) | 572890.5 | 0.61815
Colors / Competency (Exp.X) | 746006.5 | 0.00002
Table 5: Mann-Whitney U test results for the color conditions of Experiments A
and X. In the upper part of the table, we measure whether the ratings of one
evaluation criterion (e.g. the ability to recognize _colors_) of Exp.A differs
significantly from the ratings of the same evaluation criterion from Exp.X. In
the lower part of the table, we measure whether the ratings of the color
criterion differs significantly from the ratings of the other evaluation
criteria. $p$-Values in bold indicate significance ($p<0.05$), the smallest
$p$-values are underlined.

###  A.3 Additional Results

#### Answer Correctness

First, recall that we only included cases where the models generated incorrect
answers for grayscale images and correct answers for full-color images,
according to ground-truth answers in the datasets. Table 6 displays frequency
distributions of correctness ratings in our user study: ‘no’ ratings
predominated in the grayscale condition, whereas ‘yes’ ratings were more
prevalent in the color condition across both datasets. We also conducted a
chi-squared test of independence on this evaluation criterion
($\chi^{2}=2.3617,df=2,p=0.67$), finding no statistically significant
difference between Exp.A and X regarding the evaluation of the answers’
correctness. These results replicate and confirm the correctness of ground-
truth answers in VQA-X and CLEVR-X.

| Exp.A | Exp.X
---|---|---
Condition |  no |  unsure |  yes |  no |  unsure |  yes
grayscale |  1129 |  51 |  99 |  1157 |  36 |  86
color |  82 |  67 |  1131 |  59 |  48 |  1172
Table 6: Frequency distributions of ratings regarding correctness of system
answers for Exp.A and X.

#### Differences between Datasets and Models

If we first look at Exp.A (Table 1), only minimal distinctions are evident
between datasets or models, particularly concerning the models’ ability to
recognize colors, materials, and their overall competency. While slight
variations exist in the other evaluation criteria, none are notably
remarkable. For instance, regarding their understanding of the general scene,
the models exhibit slightly better performance with the CLEVR-X dataset. In
Exp.X (Table 1), on the other hand, the results exhibit some more variation
between models and datasets. For example, only for the models’ overall
competency, do we find the same (median) value across models and datasets.
Overall, it also appears that the items based on CLEVR-X data perform slightly
better in Exp.X, specifically in terms of the models’ ability to recognize
shapes and materials, as well as their general scene understanding and overall
competence.

Table 7 shows the frequency of questions in the human evaluation study that
contain the word “color[s]” or specific color terms like “red” or “blue” etc.,
categorized by dataset. It is evident that almost all questions in the CLEVR-X
dataset contain color terms, with about half explicitly mentioning the word
“color”. Conversely, in the VQA-X dataset, only three out of 64 questions
include the word “color[s]”. Hence, the observed distinctions between the
datasets may be attributed to this contrast.

| “Color[s]” in question | Color term in question
---|---|---
Dataset |  yes |  no |  yes |  no
CLEVR-X |  34 |  30 |  59 |  5
VQA-X |  3 |  61 |  3 |  61
Table 7: Occurrence of questions in the human evaluation study containing the word “color[s]” or specific color terms like “red” or “blue”, differentiated by dataset (color terms include any instance of “color”, a specific color term, or both). |  |  | Consist. of Expl. & Answ. | Consist. of Expl. & Img. | Fluency of Expl.
---|---|---|---|---|---
Condition | Dataset | Model |  median |  mean |  median |  mean |  median |  mean
grayscale | CLEVR-X | NLX-GPT |  4.0 |  3.26 |  1.0 |  1.53 |  4.0 |  3.27
|  | Uni-NLX |  4.0 |  3.17 |  1.0 |  1.74 |  4.0 |  3.46
| VQA-X | NLX-GPT |  2.0 |  2.67 |  1.0 |  1.85 |  4.0 |  3.42
|  | PJ-X |  1.0 |  2.20 |  1.0 |  2.02 |  4.0 |  3.35
color | CLEVR-X | NLX-GPT |  5.0 |  4.58 |  5.0 |  4.53 |  5.0 |  4.52
|  | Uni-NLX |  5.0 |  4.61 |  5.0 |  4.59 |  5.0 |  4.54
| VQA-X | NLX-GPT |  5.0 |  4.42 |  5.0 |  4.53 |  5.0 |  4.34
|  | PJ-X |  4.0 |  3.56 |  4.0 |  3.63 |  5.0 |  3.85
Table 8: Human ratings for the additional evaluation criteria of Exp.X. We
asked the participants to rate the _consistency of the explanation with the
answer_ , the _consistency of the explanation with the image_ , and the
_fluency of the explanation_. We report the median and mean scores across
raters as the final scores, with bold values indicating conditions with the
best (mean) values for that evaluation criteria.

(a) Exp.A – colored images

(b) Exp.A – grayscale images

(c) Exp.X – colored images

(d) Exp.X – grayscale images

Figure 4: Human ratings on the evaluation criterion “Ability of the AI system
to understand the general scene”. Participants indicated their judgment on a
scale from 1 (strongly disagree; here in red) to 5 (strongly agree; here in
green).

(a) Exp.A – colored images

(b) Exp.A – grayscale images

(c) Exp.X – colored images

(d) Exp.X – grayscale images

Figure 5: Human ratings on the evaluation criterion “Overall competency of the
AI system”. Participants indicated their judgment on a scale from 1 (strongly
disagree; here in red) to 5 (strongly agree; here in green).

(a) Exp.A – colored images

(b) Exp.A – grayscale images

(c) Exp.X – colored images

(d) Exp.X – grayscale images

Figure 6: Human ratings on the evaluation criterion “Ability of the AI system
to recognize shapes”. Participants indicated their judgment on a scale from 1
(strongly disagree; here in red) to 5 (strongly agree; here in green).

(a) Exp.A – colored images

(b) Exp.A – grayscale images

(c) Exp.X – colored images

(d) Exp.X – grayscale images

Figure 7: Human ratings on the evaluation criterion “Ability of the AI system
to recognize materials”. Participants indicated their judgment on a scale from
1 (strongly disagree; here in red) to 5 (strongly agree; here in green).

#### Analysis of the Color Condition

Table 9 shows the human evaluation results for the color condition in Exp. A
and X. In contrast to the results of the grayscale condition (Table 1), with
respect to all the evaluation criteria, the evaluation for both Exp.A and
Exp.X is very good. This corresponds to our expectation because only items
with correct model answers were included in the color condition.

Furthermore, we can see that in both Exp.A and Exp.X, there are no remarkable
differences between the ability to recognize colors and the other tested
abilities. This is also evident from the Mann-Whitney U Test results in Table
5, especially when compared to the Mann-Whitney U results for the grayscale
condition in Table 4.

However, it is notable that, with respect to all evaluation criteria, the PJ-X
model receives lower ratings in Exp.X compared to Exp.A. In other words,
including explanations in Exp.X results in a decline in performance for the
PJ-X model. For the other models, we do not observe this difference between
the two Experiments; instead, their evaluation remains fairly consistent in
the color condition across both experiments. Consequently, the explanations
produced by the PJ-X model seem inferior to those of the other models. This
discrepancy may be due to the unique architecture of the PJ-X model, which,
unlike the other models, generates answers and explanations in two separate
steps rather than one.

|  |  | Colors | Shapes | Materials | General Scene | Competency
---|---|---|---|---|---|---|---
Experiment | Dataset | Model |  med |  mean |  med |  mean |  med |  mean |  med |  mean |  med |  mean
Exp. A. | CLEVR-X | NLX-GPT |  5.0 |  4.55 |  5.0 |  4.57 |  5.0 |  4.34 |  5.0 |  4.43 |  5.0 |  4.47
|  | Uni-NLX |  5.0 |  4.33 |  5.0 |  4.38 |  5.0 |  4.20 |  5.0 |  4.23 |  5.0 |  4.28
| VQA-X | NLX-GPT |  5.0 |  4.55 |  5.0 |  4.50 |  5.0 |  4.45 |  5.0 |  4.67 |  5.0 |  4.66
|  | PJ-X |  5.0 |  4.38 |  5.0 |  4.30 |  5.0 |  4.30 |  5.0 |  4.57 |  5.0 |  4.50
Exp.X | CLEVR-X | NLX-GPT |  5.0 |  4.65 |  5.0 |  4.66 |  5.0 |  4.58 |  5.0 |  4.57 |  5.0 |  4.52
|  | Uni-NLX |  5.0 |  4.74 |  5.0 |  4.61 |  5.0 |  4.56 |  5.0 |  4.58 |  5.0 |  4.56
| VQA-X | NLX-GPT |  5.0 |  4.54 |  5.0 |  4.54 |  5.0 |  4.54 |  5.0 |  4.58 |  5.0 |  4.38
|  | PJ-X |  4.0 |  3.80 |  4.0 |  3.86 |  4.0 |  3.84 |  4.0 |  3.86 |  4.0 |  3.71
Table 9: Human ratings on the different evaluation criteria for the color
condition of Exp.A (i.e., no model explanations were shown to the
participants) and Exp.B (i.e., model explanations were shown to the
participants). For _Colors_ , _Shapes_ and _Materials_ , we asked the
participants to rate the AI system’s ability to recognize the respective
capability. Further, we asked the participants to rate the AI system’s
understanding of the _General Scene_ as well as it’s overall _Competency_. We
report the median and mean scores across raters as the final scores. Bold
values indicate conditions with the best (mean) values for that evaluation
criteria.

#### Correlations between BERTscore and human judgements

Table 10 shows Pearson’s correlation coefficients ($\rho$) between the
automatic and human evaluation metrics for the CLEVR-X and VQA-X datasets.
Interestingly, we find large differences between the datasets. While all human
metrics show statistically significant correlations with BERTScore for the
VQA-X dataset, we find no statistically significant correlations for the
CLEVR-X dataset. However, one commonality between the two datasets is the lack
of differentiation between various criteria. The fact that all skills either
correlate or show no correlation suggests that the automatic BERTScore metric
is not able to capture the nuanced distinctions that human evaluation can
discern.

|  | CLEVR-X | VQA-X
---|---|---|---
Automatic metric | Human metric |  $\rho$ |  $p$-value |  $\rho$ |  $p$-value
BERTScore | Consist. of Expl. & Answ. |  -0.090 |  0.31 |  0.251 |  0.008
Consist. of Expl. & Img. |  -0.020 |  0.82 |  0.278 |  0.003
Fluency of Expl. |  -0.033 |  0.71 |  0.304 |  0.001
Shapes |  -0.068 |  0.44 |  0.231 |  0.02
Colors |  -0.023 |  0.80 |  0.201 |  0.04
Materials |  -0.056 |  0.53 |  0.248 |  0.009
General Scene |  -0.051 |  0.57 |  0.251 |  0.008
Competency |  -0.051 |  0.57 |  0.252 |  0.008
Table 10: Pearson’s correlation coefficient ($\rho$) between BERTScore results
and human evaluation metrics for CLEVR-X and VQA-X data. $p$-values in bold
indicate statistical significance ($p<0.05$).

###  A.4 Online Experiment

Figures 8 and 9 show screenshots of the study, example items and evaluation
criteria.

Figure 8: A training item used in the online experiment to familiarize
participants with the task and rating scales. This item comes from the VQA-X
dataset and from Exp.A, i.e., the study without explanations. Figure 9: An
experimental item used in the online experiment. This item comes from the
CLEVR-X dataset and from Exp.X, i.e., the experiment with explanations.