# Don’t Hallucinate, Abstain:
Identifying LLM Knowledge Gaps via Multi-LLM Collaboration

Shangbin Feng1 Weijia Shi1 Yike Wang2
Wenxuan Ding3 Vidhisha Balachandran4 Yulia Tsvetkov1
1University of Washington 2University of California, Berkeley
3The Hong Kong University of Science and Technology 4Carnegie Mellon
University
<EMAIL_ADDRESS>
###### Abstract

Despite efforts to expand the knowledge of large language models (LLMs),
knowledge gaps—missing or outdated information in LLMs—might always persist
given the evolving nature of knowledge. In this work, we study approaches to
identify LLM knowledge gaps and _abstain_ from answering questions when
knowledge gaps are present. We first adapt existing approaches to model
calibration or adaptation through fine-tuning/prompting and analyze their
ability to abstain from generating low-confidence outputs. Motivated by their
failures in self-reflection and over-reliance on held-out sets, we propose two
novel approaches that are based on model collaboration, i.e., LLMs probing
other LLMs for knowledge gaps, either cooperatively or competitively.
Extensive experiments with three LLMs on four QA tasks featuring diverse
knowledge domains demonstrate that both cooperative and competitive approaches
to unveiling LLM knowledge gaps achieve up to 19.3% improvements on abstain
accuracy against the strongest baseline. Further analysis reveals that our
proposed mechanisms could help identify failure cases in retrieval
augmentation and pinpoint knowledge gaps in multi-hop reasoning.111Resources
will be publicly available at https://github.com/BunsenFeng/AbstainQA.

Don’t Hallucinate, Abstain:
Identifying LLM Knowledge Gaps via Multi-LLM Collaboration

Shangbin Feng1 Weijia Shi1 Yike Wang2 Wenxuan Ding3 Vidhisha Balachandran4
Yulia Tsvetkov1 1University of Washington 2University of California, Berkeley
3The Hong Kong University of Science and Technology 4Carnegie Mellon
University<EMAIL_ADDRESS>

##  1 Introduction

LLMs demonstrate impressive capabilities of encoding real-world knowledge in
model parameters and leveraging it to aid knowledge-intensive tasks (Petroni
et al., 2019; Brown et al., 2020; Yu et al., 2023a). But when such knowledge
is missing or unreliable, they resort to hallucinations (Ji et al., 2023) and
biases Feng et al. (2023a), while still “speaking with confidence.” A growing
body of work seeks to expand LLM knowledge through retrieval augmentation (Guu
et al., 2020; Borgeaud et al., 2022; Khattab et al., 2022; Shi et al., 2023;
Asai et al., 2023; Chen et al., 2023), search engine integration (Nakano et
al., 2021; Press et al., 2023), and multi-LM collaboration (Luo et al., 2023;
Feng et al., 2023b). However, LLM knowledge gaps might always persist due to
the ever-evolving nature of knowledge (Kandpal et al., 2023; Mallen et al.,
2023; De Cao et al., 2021; Hernandez et al., 2023; Kasai et al., 2022).
Consequently, we posit that _abstaining from generating low-confidence outputs
should be a part of LLMs’ functionality_ , and ask a crucial research
question: _how to identify knowledge gaps in LLMs?_ Developing and evaluating
robust mechanisms to address the abstain problem could help improve LLM
reliability, reduce hallucinations, and mitigate biases due to model
uncertainty.

We hypothesize that there are three potential ways to operationalize the
abstain problem: (1) from the data perspective, LLMs should abstain when
knowledge is missing in training data or curated from unreliable sources; (2)
from the modeling perspective, LLMs should abstain when knowledge is not
encoded in model parameters and cannot be inferred; (3) from the usage
perspective, LLMs should abstain when their generated outputs would provide an
incorrect answer. Since users are directly interacting with and impacted by
LLM-generated texts, not model parameters or training data, we formulate and
evaluate LLM abstention focusing on the factual correctness of LLM generated
texts.

Specifically, we focus on abstaining in question-answering (AbstainQA): given
an $\mathrm{LLM}$ and a question $\boldsymbol{q}$, we aim to develop an
abstain function
$f(\mathrm{LLM},\boldsymbol{q})\rightarrow\\{\textit{true},\textit{false}\\}$
to indicate whether the LLMs should abstain given limitations of its internal
knowledge (Kamath et al., 2020; Jiang et al., 2021; Whitehead et al., 2022).
To facilitate a comprehensive understanding of existing approaches that can be
adapted for LLM abstention, we first identify 11 methods from previous works
and adapt them to incorporate abstain functionality. We categorize these
baselines into (1) _calibration-based_ , such as employing temperature scaling
(Guo et al., 2017; Jiang et al., 2021) to calibrate models and set a
confidence score threshold for abstention; (2) _training-based_ , such as
employing linear probing on LLMs’ hidden layers to evaluate the veracity of
generated texts (Slobodkin et al., 2023; Azaria and Mitchell, 2023); (3)
_prompting-based_ , such as prompting LLMs to request more information before
answering questions (Feng et al., 2023b); and (4) _self-consistency based_ ,
such as generating multiple chains-of-thought (Wei et al., 2022).

Among the 11 baselines, we observe that a majority and often the stronger
approaches would require a _held-out set_ for training and hyperparameter
tuning, otherwise potentially harming generalization across knowledge domains.
In addition, existing approaches often rely on a “self-reflection” assumption
that _a single LLM_ could be employed or adapted to evaluate its own generated
texts; however, challenges such as hallucination and confirmation biases cast
doubt on this assumption (Ji et al., 2023; Xie et al., 2023).

To this end, we propose two multi-LLM collaboration-based approaches, enabling
robust LLM abstention through multi-LLM collaboration to reflect on generated
text in cooperative or competitive settings while removing the need for held-
out sets. For multiple LLMs to work in _cooperation_ (Cooperate), the LLM
employs other models to provide feedback on the proposed answer and reasoning,
and synthesizes the outputs into an overall abstain decision. For a
_competitive_ setting (Compete), the LLM is challenged by other LLMs with
conflicting knowledge, producing an abstain decision based on whether it
continues to prefer the proposed answer in the presence of conflicting
evidence.

We evaluate the baselines and proposed collaboration approaches for abstention
with three LLMs, on four knowledge-intensive QA tasks spanning diverse
knowledge domains and reasoning. Extensive experiments demonstrate that
Cooperate and Compete are strong abstain mechanisms that outperform all
baselines in 9 of the 12 settings across tasks and models, achieving an
improvement of up to 19.3% in abstain accuracy. Further analysis reveals that
the proposed collaboration-based approaches could help identify failure cases
of retrieval augmentation, pinpoint knowledge gaps in multi-hop reasoning, and
more. Our contributions are (1) a critical evaluation and typology of diverse
existing methods, repurposed to identify knowledge gaps in LLMs, as well as
(2) two novel, robust multi-LLM collaboration methods to detect LLM knowledge
gaps, Cooperate and Compete.

##  2 Identifying Knowledge Gaps in LLMs

We first propose to repurpose existing approaches to enable abstain
functionality in the face of knowledge gaps in LLMs. We conceptually
categorize these approaches into _calibration_ (§2.1), _training_ (§2.2),
_prompting_ (§2.3), and _self-consistency_ (§2.4) methods. Empirically
observing that these approaches are dependent on external knowledge/data sets
and often suffer from instability, we propose a new category based on _multi-
LLM collaboration_ (§2.5).

###  2.1 Calibration-Based

#### Token Probability

LLMs are often inherently calibrated to different extents (Radford et al.,
2019; Liang et al., 2023) and token probabilities might be employed off-the-
shelf. Let $(\boldsymbol{q},\boldsymbol{\bar{a}})\in\mathcal{H}$ be a held-out
set of questions and gold answers for hyperparameter tuning, while LLM
produces an answer $\boldsymbol{a}=\mathrm{LLM}(\boldsymbol{q})$ with a
confidence score based on token probabilities $p(\boldsymbol{a})\in[0,1]$. We
set a threshold $p^{*}$ by minimizing abstain errors on $\mathcal{H}$,
formally:

Figure 1: Overview of collaboration-based approaches for LLM abstention:
Cooperate and Compete.

$\displaystyle
p^{*}=\operatorname*{arg\,min}_{p\in[0,1]}\sum_{\boldsymbol{a}=\bar{\boldsymbol{a}}}\scalebox{0.9}{$\mathds{1}(p(\boldsymbol{a})<p^{*})$}+\sum_{\boldsymbol{a}\neq\bar{\boldsymbol{a}}}\scalebox{0.9}{$\mathds{1}(p(\boldsymbol{a})\geq
p^{*})$}$

where $\mathds{1}$ denotes the indicator function. At inference time, the LLM
abstains when $p(\boldsymbol{a})<p^{*}$.

#### Temperature Scaling

Before setting a threshold $p^{*}$ on confidence scores, temperature scaling
(Guo et al., 2017; Jiang et al., 2021) first reshapes the token probability
distributions by introducing a scalar temperature $\tau>0$. Given an answer
probability distribution $\mathbf{a}$, we transform it into
$\mathrm{softmax}(\mathbf{a}/\tau)$, then optimize $\tau$ on the held-out set
$\mathcal{H}$ with cross-entropy loss to obtain $\tau^{*}$. A probability
threshold $p^{*}$ is then set over
$\\{\mathrm{softmax}(\mathrm{LLM}(\boldsymbol{q})/\tau^{*})\\}_{\boldsymbol{q}\in\mathcal{H}}$
and applied at inference time.

#### Ask for Calibration

In addition to token probabilities, LLMs could also verbalize confidence
scores: we follow Tian et al. (2023) to obtain confidence scores
$p(\boldsymbol{a})$ by employing a two-step process: 1) generating an answer
with the LLM and 2) elicit a confidence score by prompting with “Provide the
probability that your guess is correct.”222Full prompt templates are in
Appendix B. A threshold $p^{*}$ is similarly set over verbalized scores and
applied at inference time.

###  2.2 Training-Based

#### Hidden Layers

Previous works demonstrate that probing LLMs’ hidden layer representations
might yield insights into their knowledge and factuality (Slobodkin et al.,
2023; Azaria and Mitchell, 2023; CH-Wang et al., 2023). We first extract the
hidden embeddings
$\mathbf{e}_{\boldsymbol{q}}=\mathrm{LLM}\textit{-hidden}(\boldsymbol{q})$. We
then conduct linear probing to predict
$\ell\in\\{\textit{accept},\textit{reject}\\}$ through
$p(\ell\mid\boldsymbol{q})\propto\exp(\mathrm{FFN}(\mathbf{e}_{\boldsymbol{q}}))$.
We optimize with the cross-entropy loss on the held-out set $\mathcal{H}$. At
inference time, the LLM should abstain if
$\operatorname*{arg\,max}_{\ell}p(\ell\mid\boldsymbol{q})$ indicates that the
answer should be rejected.

#### External Verifier

Training an external verifier to judge the veracity of LLM-generated text has
shown great potential in math reasoning (Cobbe et al., 2021) and general
knowledge domains (Kadavath et al., 2022; Cohen et al., 2023). We train an
encoder-based language model $\mathrm{LM}\textit{-enc}$ and employ the
$\mathrm{[CLS]}$ token for classification. Formally,
$p(\ell\mid\boldsymbol{q})\propto\exp(\mathrm{LM}\textit{-enc}(\boldsymbol{q};\mathrm{LLM}(\boldsymbol{q})))$
is optimized on $\mathcal{H}$ with the cross-entropy loss and an abstain
decision is achieved through
$\operatorname*{arg\,max}_{\ell}p(\ell\mid\boldsymbol{q})$ at inference time.
Here both the question and the generated answer are employed to train the
external verifier, different from the _hidden layers_ approach where only the
intermediate encodings of the question are used for linear probing.

#### Instruction Tuning

A promising way to adapt LLMs is through tuning LLMs on instruction triplets
in the format of $\\{\textit{instruction}|\textit{input}|\textit{output}\\}$
(Ouyang et al., 2022). Here we investigate whether abstaining could be baked
into existing LLMs through instruction tuning. We first employ an instruction
to encourage LLMs to abstain when necessary. Given the held-out set
$\mathcal{H}$, we first evaluate the correctness of the generated answer
$\boldsymbol{a}=\mathrm{LLM}(\boldsymbol{q})$: if it is correct, we use
$\\{\textit{instruction}|\boldsymbol{q}|\boldsymbol{a}\\}$ as an instruction
tuning instance; otherwise, we replace the incorrect answer $\boldsymbol{a}$
with an abstain message and train on
$\\{\textit{instruction}|\boldsymbol{q}|\textit{abstain message}\\}$. In this
way, the LLM abstains if the abstain message is generated at inference time
for a given $\boldsymbol{q}$.

###  2.3 Prompting-Based

#### Self-Reflect

Previous studies show that LLMs might have preliminary capabilities of judging
and evaluating their own answer (Kadavath et al., 2022). To this end, we
prompt the LLM to self-reflect directly after its generated answer with “The
above answer is: A. True B. False”. LLMs should abstain when they deem the
generated answer as false.

#### More Information

Since the internal knowledge of LLMs is often incomplete and noisy, existing
works explore how to selectively request external information to aid LLM
generation (Xu et al., 2023; Asai et al., 2023; Feng et al., 2023b). Following
Feng et al. (2023b), we append a prompt about whether more information is
needed to answer a given question: “Do you need more information to answer
this question? (Yes or No)”. The LLM should abstain if more information is
requested.

#### Generate and Match

There might be gaps between LLM generation and “understanding” (West et al.,
2023), which leads to hallucinations when questions are presented in a
multiple-choice format (Fu et al., 2023). Instead of directly choosing from
one of the options, we instruct the LLM to first generate an answer _without_
access to potential options. We then prompt the LLM to evaluate whether the
generated answer matches any of the options: the LLM should abstain if there’s
no match between the generated answer and options.

###  2.4 Consistency-Based

#### None-of-the-Above

Previous works have investigated LLMs’ robustness to possibilities of _non-of-
the-above_ (_nota_) in QA tasks (Kadavath et al., 2022; Ding et al., 2023). We
propose an abstain mechanism by adding an extra _nota_ option to every
question: the LLM should abstain if _nota_ is selected as the answer,
indicating its low consistency in answering the question.

#### Self-Consistency Threshold

Self-consistency (Wang et al., 2022) aims to generate multiple chain-of-
thought (Wei et al., 2022) reasoning paths and conduct majority voting to
determine the answer. We develop an abstain mechanism by proposing _plurality
index_ : given a question $\boldsymbol{q}$ along with the $k$ generated
reasoning paths and answers
$\\{(\boldsymbol{a}_{i},\boldsymbol{p}_{i})\\}_{i=1,\ldots,k}$, the plurality
index is defined as:

$\displaystyle\mathrm{plu}(\mathrm{LLM},\boldsymbol{q},k)=\max_{\boldsymbol{a}_{i}}\sum_{j=1,\ldots,k}\mathds{1}(\boldsymbol{a}_{j}=\boldsymbol{a}_{i})$

where it indicates how large the plurality is among all $k$ reasoning paths. A
threshold $\tau^{*}\in[0,1]$ is then optimized on $\mathcal{H}$ and applied at
inference time to abstain when
$\mathrm{plu}(\mathrm{LLM},\boldsymbol{q},k)<\tau^{*}\cdot k$.

Method | MMLU | K-Crosswords | Hellaswag | Propaganda
---|---|---|---|---
R-Acc | ER | A-Acc | A-F1 | R-Acc | ER | A-Acc | A-F1 | R-Acc | ER | A-Acc | A-F1 | R-Acc | ER | A-Acc | A-F1
Mistral-7b
Probs | .570 | .109 | .608 | .456 | .251 | -.351 | .397 | .422 | .456 | -.041 | .599 | .659 | .337 | -.150 | .590 | .680
Temp. | .565 | .104 | .601 | .426 | .250 | -.452 | .303 | .182 | .451 | -.047 | .601 | .659 | .340 | -.170 | .585 | .661
Ask Cali. | .648 | .141 | .639 | .647 | .233 | -.023 | .705 | .825 | .455 | -.017 | .616 | .734 | .231 | -.035 | .705 | .824
Hidden | .449 | -.085 | .420 | .137 | .104 | -.788 | .107 | .009 | .369 | -.198 | .424 | .336 | .223 | -.510 | .240 | .084
Verifier | .568 | .083 | .586 | .534 | .208 | -.083 | .805 | .889 | .405 | -.080 | .550 | .628 | .448 | -.015 | .760 | .853
Instruct | .709 | .198 | .688 | .693 | - | - | - | - | .616 | .075 | .701 | .771 | - | - | - | -
Reflect | .498 | -.002 | .495 | .477 | .111 | -.686 | .203 | .207 | .371 | -.150 | .477 | .500 | .216 | -.275 | .490 | .602
MoreInfo | .504 | .006 | .500 | .216 | .272 | -.455 | .273 | .003 | .379 | -.227 | .396 | .120 | .219 | -.540 | .240 | .073
Gen+Match | .511 | .021 | .515 | .129 | .111 | -.694 | .193 | .188 | .377 | -.174 | .458 | .415 | .226 | -.545 | .230 | .013
NOTA | .516 | .029 | .528 | .163 | .098 | -.797 | .102 | .011 | .371 | -.244 | .387 | .105 | .259 | -.410 | .340 | .267
SC Thres. | .604 | .140 | .641 | .551 | .210 | -.090 | .793 | .880 | .493 | -.004 | .614 | .713 | .273 | -.100 | .685 | .799
Coop-self | .571 | .059 | .564 | .601 | .260 | -.437 | .313 | .311 | .406 | -.042 | .601 | .719 | .297 | -.095 | .680 | .797
Coop-others | .688 | .213 | .712 | .692 | .266 | -.022 | .761 | .852 | .626 | .092 | .725 | .783 | .182 | -.140 | .625 | .757
Compete | .735 | .140 | .640 | .700 | .289 | -.129 | .597 | .722 | .573 | .032 | .658 | .766 | .302 | -.055 | .700 | .805
LLaMA2-70b
Ask Cali. | .624 | .025 | .435 | .568 | - | - | - | - | .944 | .032 | .498 | .649 | - | - | - | -
Hidden | .473 | -.018 | .400 | .446 | .282 | -.265 | .423 | .466 | .507 | .007 | .497 | .479 | - | - | - | -
Verifier | .665 | .201 | .609 | .511 | .443 | -.056 | .634 | .694 | .522 | .009 | .504 | .617 | .259 | -.065 | .755 | .855
Instruct | .745 | .216 | .628 | .640 | .288 | .024 | .606 | .772 | .475 | -.008 | .487 | .615 | - | - | - | -
Reflect | .616 | .121 | .529 | .469 | .383 | -.155 | .528 | .537 | .569 | .076 | .560 | .530 | .198 | -.520 | .275 | .225
MoreInfo | .590 | .177 | .590 | .042 | .306 | -.382 | .308 | .020 | .516 | .032 | .518 | .044 | - | - | - | -
Gen+Match | .667 | .050 | .450 | .560 | .248 | -.111 | .573 | .708 | .484 | -.004 | .477 | .614 | .082 | -.205 | .620 | .759
NOTA | .592 | .167 | .583 | .181 | .323 | -.295 | .388 | .280 | .516 | .027 | .522 | .236 | .185 | -.580 | .225 | .124
SC Thres. | .684 | .247 | .656 | .534 | .426 | -.090 | .590 | .617 | .667 | .100 | .590 | .655 | .412 | -.030 | .760 | .852
Coop-self | .615 | .150 | .550 | .400 | .463 | -.030 | .640 | .714 | .649 | .110 | .600 | .643 | .222 | -.250 | .500 | .615
Coop-others | .694 | .262 | .676 | .562 | .402 | -.063 | .636 | .757 | .700 | .238 | .704 | .677 | .329 | -.125 | .675 | .774
Compete | .782 | .148 | .552 | .608 | .323 | -.080 | .642 | .760 | .611 | .047 | .525 | .625 | .161 | -.210 | .595 | .729
ChatGPT
Probs | .774 | .421 | .715 | .457 | .600 | .187 | .587 | .122 | .750 | .278 | .599 | .476 | .333 | -.015 | .625 | .765
Temp. | .769 | .419 | .716 | .452 | .616 | .214 | .619 | .216 | .750 | .278 | .595 | .468 | .250 | -.010 | .630 | .772
Ask Cali. | .694 | .385 | .690 | .006 | .601 | .202 | .601 | .010 | .672 | .344 | .672 | .006 | .444 | -.015 | .580 | .712
Verifier | .788 | .301 | .599 | .483 | - | - | - | - | .667 | .305 | .634 | .120 | .441 | -.035 | .570 | .672
Instruct | .840 | .422 | .748 | .579 | .752 | .310 | .709 | .627 | .817 | .429 | .751 | .614 | .605 | .045 | .645 | .744
Reflect | .752 | .336 | .630 | .411 | .784 | .239 | .641 | .633 | .754 | .377 | .701 | .487 | .571 | .015 | .615 | .742
MoreInfo | .721 | .246 | .546 | .390 | .605 | .145 | .553 | .380 | .675 | .224 | .548 | .339 | .416 | -.145 | .470 | .293
Gen+Match | .737 | .350 | .652 | .383 | .660 | .083 | .486 | .550 | .712 | .182 | .506 | .447 | .365 | -.115 | .490 | .568
NOTA | .719 | .389 | .692 | .260 | .644 | .163 | .565 | .480 | .689 | .307 | .628 | .268 | .400 | -.120 | .485 | .488
SC Thres. | .766 | .424 | .743 | .447 | .637 | .216 | .622 | .382 | .749 | .366 | .688 | .468 | .399 | -.160 | .440 | .309
Coop-self | .841 | .436 | .726 | .578 | .794 | .175 | .646 | .646 | .878 | .344 | .670 | .628 | .684 | .070 | .710 | .802
Coop-others | .780 | .362 | .660 | .479 | .659 | .109 | .509 | .536 | .790 | .350 | .676 | .565 | .790 | .321 | .647 | .543
Compete | .947 | .306 | .602 | .583 | .875 | .034 | .441 | .589 | .939 | .172 | .490 | .545 | .611 | .040 | .670 | .795

Table 1: Performance of abstain strategies on four datasets and three LLMs.
Best results in bold and second best in underline. Approaches are color-coded
per category: calibration, training, prompting, consistency, and
collaboration. Certain incompatible cases, _e.g._ embedding with the black-box
ChatGPT, are omitted. “-” indicates that this approach fails to produce
meaningful abstain decisions: almost always abstain, didn’t follow
instructions, etc. Cooperate and Compete achieve the best performance in 9 of
the 12 settings in terms of reliable accuracy.

###  2.5 Proposed: Multi-LLM Collaboration

The above approaches often employ or adapt a single language model to
evaluate/reflect on its own generated texts, while challenges such as
confirmation bases and hallucination would result in unreliable self-
evaluation (Kadavath et al., 2022; Ji et al., 2023; Xie et al., 2023). In
addition, the stronger approaches rely on held-out sets and result in weaker
generalization across knowledge tasks and domains. To this end, we propose to
enable robust LLM abstention through multi-LLM collaboration, i.e., employing
multiple LLMs that interact with each other to evaluate the veracity of
generated texts and make abstain decisions.

#### Cooperate

Different LLMs often come with varying knowledge coverage and abilities with
the potential of complementing each other (Yu et al., 2023a; Feng et al.,
2023b; Du et al., 2023; Bansal et al., 2024), while the internal knowledge of
one language model could also be diversified and specialized (Kang et al.,
2023; Si et al., 2023). We tap into this knowledge variation in the Cooperate
approach by having different expert LLMs generate feedback on LLM-proposed
answers and employing an LLM as the final judge to decide whether to abstain.
Formally, given a question $\boldsymbol{q}$ and a generated answer
$\boldsymbol{a}=\mathrm{LLM}(\boldsymbol{q})$, we obtain a set of natural
language feedback $\\{\boldsymbol{f}_{1},\ldots,\boldsymbol{f}_{k}\\}$ from
expert LLMs and employ a judge LLM to summarize and synthesize the proposed
answer and feedbacks into an abstain decision
$\mathrm{LLM}(\boldsymbol{q},\boldsymbol{a},\\{\boldsymbol{f}_{1},\ldots,\boldsymbol{f}_{k}\\})\rightarrow\\{\textit{accept},\textit{reject}\\}$.

We employ two modes to generate feedback: for the _self_ mode, the same LLM is
specialized into experts on domains $\\{d_{1},\ldots,d_{k}\\}$ through
prompting-based self-specialization: we prompt the LLM to generate a knowledge
passage $\textit{knowledge}_{i}$ about $\boldsymbol{q}$ with a focus on domain
$d_{i}$. A domain-specific feedback is then generated by prepending the
knowledge passage:
$\boldsymbol{f}_{i}=\mathrm{LLM}(\textit{knowlegde}_{i},\boldsymbol{q},\boldsymbol{a})$,
and prompting the model to respond as a reviewer. For the _others_ mode, we
use separate LLMs $\\{\mathrm{LLM}_{1},\ldots,\mathrm{LLM}_{k}\\}$ to provide
feedback $\boldsymbol{f}_{i}=\mathrm{LLM}_{i}(\boldsymbol{q},\boldsymbol{a})$,
aiming to identify the knowledge gaps in each other and complement the abstain
process.

#### Compete

Language models have varying preferences in the face of knowledge conflicts,
_i.e._ when there is a conflict between LLM’s internal parametric knowledge
and external information provided in the prompt context (Xie et al., 2023;
Wang et al., 2023b). We hypothesize that an LLM should abstain when it is
impacted by conflicting information generated by other LLMs and does not stick
to the parametric answer. Concretely, for a question $\boldsymbol{q}$ and an
LLM-generated answer $\boldsymbol{a}=\mathrm{LLM}(\boldsymbol{q})$, we prompt
other LLMs to generate $k$ alternative answers
$\\{\boldsymbol{a}^{{}^{\prime}}_{1},\ldots,\boldsymbol{a}^{{}^{\prime}}_{k}\\}$
along with a knowledge passage $\textit{knowledge}_{i}$ about
$\boldsymbol{a}^{{}^{\prime}}_{i}$. We then instruct the LLM to answer
$\boldsymbol{q}$ again with conflicting information prepended to the context:
$\tilde{\boldsymbol{a}}_{i}=\mathrm{LLM}(\textit{knowledge}_{i},\boldsymbol{q})$.
If $\boldsymbol{a}=\tilde{\boldsymbol{a}}_{i}$, the LLM sticks to its
parametric knowledge and the answer should be thus accepted; otherwise, the
LLM is swayed by conflicting information generated by other LLMs, potentially
betraying its low confidence. This process is repeated for $k$ alternative
answers and the LLM should abstain if the answer changes in a majority of
cases.

##  3 Experiment Settings

#### Models

We evaluate baselines and proposed approaches with three LLMs featuring
varying sizes and openness: _Mistral-7B_ (Jiang et al., 2023), _LLaMA2-70B_
(Touvron et al., 2023), and _ChatGPT_. A robust abstain mechanism should
ideally work for all LLMs, weak and strong.

#### Tasks and Datasets

We evaluate LLM abstention with four tasks and datasets spanning diverse
knowledge domains and reasoning scenarios: 1) _MMLU_ (Hendrycks et al., 2020)
is a multiple-choice dataset for generic knowledge QA spanning 57 domains; 2)
_Knowledge Crosswords_ (Ding et al., 2023) is a structured QA dataset that
focuses on multi-hop knowledge reasoning; 3) _Hellaswag_ (Zellers et al.,
2019) is a natural language inference dataset that tests commonsense knowledge
and reasoning; 4) _Propaganda_ (Piskorski et al., 2023) tasks LLMs with
identifying the 23 propaganda tactics in a long news article based on their
internal knowledge. We create splits of a held-out set and test set for the
four datasets with details in Appendix B.1. A robust abstain mechanism should
ideally work for knowledge tasks across domains and reasoning contexts.

Figure 2: Four outcomes of AbstainQA.

#### Evaluation Metrics

We illustrate the four scenarios under AbstainQA in Figure 2 and employ the
following evaluation metrics. (1) _Reliable Accuracy_ (R-Acc):
$\frac{\mathcal{A}}{\mathcal{A}+\mathcal{C}}$ indicates to what extent could
LLM-generated answers (not abstained) be trusted, i.e., _out of all questions
answered, how many are correct?_ (2) _Effective Reliability_ (ER) (Whitehead
et al., 2022; Si et al., 2023):
$\frac{\mathcal{A}-\mathcal{C}}{\mathcal{A}+\mathcal{B}+\mathcal{C}+\mathcal{D}}$
strikes a balance between reliability and coverage, i.e., _out of all
questions, how many more are answered correctly than incorrectly?_ (3)
_Abstain Accuracy_ (A-Acc):
$\frac{\mathcal{A}+\mathcal{D}}{\mathcal{A}+\mathcal{B}+\mathcal{C}+\mathcal{D}}$
evaluates whether the abstain decisions are correct. (4) _Abstain F1_ (A-F1):
$\textit{harmonic-mean}(\mathrm{precision},\mathrm{recall})$, where
$\mathrm{precision}=\frac{\mathcal{D}}{\mathcal{B}+\mathcal{D}}$ and
$\mathrm{recall}=\frac{\mathcal{D}}{\mathcal{C}+\mathcal{D}}$.

##  4 Results

We present the performance of abstain mechanisms with three LLMs on four tasks
in Table 1.

####  Cooperate and Compete are state-of-the-art abstain mechanisms.

They outperform the strongest baseline in 9 out of the 12 settings (3 models
and 4 datasets), achieving an average improvement of 10.1% on reliable
accuracy scores. We find that Cooperate works better with stronger models such
as ChatGPT: we hypothesize that the complexities of the multi-step feedback
generation warrant a stronger base language model. Compete emphasizes
reliability and greatly avoids wrong answers evident in the high reliable
accuracy scores: we observe that LLMs stick to the original answers only in
the most confident scenarios, making Compete ideal for abstention when
reliability is paramount.

#### Calibration > Training > Consistency > Prompting

On average, the three types of approaches achieved 0.595, 0.576, 0.553, and
0.475 A-Acc scores. While simple prompting-based approaches struggle to elicit
self-reflection, our proposed Cooperate and Compete promote collaborative
abstention and greatly improve performance. Among all baselines, _instruction
tuning_ and _self-consistency threshold_ stand out as two strong baselines.
However, they both rely on a held-out set for training and hyperparameter
tuning: we further investigate their generalizability in Section 5.

Figure 3: Performance of abstain mechanisms in the _abstain absolute_
scenarios where the LLM should abstain for 100% of questions. Compete achieves
the highest abstention rate on average across LLMs and datasets.

#### Abstain capabilities potentially correlate with base LLM utility.

Abstain accuracy (A-Acc) is a metric independent of LLM’s underlying
performance on a QA dataset and solely evaluates abstention: _mistral_ ,
_llama2_ , and _chatgpt_ achieve 0.524, 0.537, and 0.616 A-Acc scores on
average across approaches and datasets, which aligns well with their relevant
superiority on standard benchmarks. Our proposed collaboration-based
approaches also work better with the strongest _ChatGPT_.

#### Going beyond generic QA sheds light on the limitations of existing
approaches.

While all baselines could function properly on MMLU, we demonstrate that some
collapse (“-” in Table 1) in other knowledge domains or reasoning contexts:
Ask Calibrate, among others, struggle to follow instructions when the prompt
context is long in detecting propaganda tactics; Instruction Tuning collapses
and produces 100% abstention due to imbalances in the held-out set in
K-Crosswords where problems are much harder and the LLM should mostly abstain;
calibration-based approaches struggle to set a good threshold when the model’s
base performance is too low. However, our proposed collaboration-based
approaches work well with all four tasks, especially with the strongest
_ChatGPT_ language model.

##  5 Analysis

#### Abstain Absolute

Aside from QA datasets where abstain decisions should be made based on the
correctness of answers, we investigate scenarios where LLMs should 100%
abstain. Specifically, we employ AmbigQA (Min et al., 2020), where LLMs should
always abstain from ambiguous and underspecified questions. We also curate
ElectionQA23, a QA dataset focusing on elections in 2023 around the globe
(Appendix B.1) and LLMs should abstain due to the temporal mismatch of
training data cutoff. Other potential _abstain absolute_ scenarios include
known unknowns (Amayuelas et al., 2023) and questions with
false/underspecified premises (Patidar et al., 2023; Hu et al., 2023). Figure
3 demonstrates that our proposed Compete consistently abstains the most in
abstain absolute scenarios across three LLMs and two datasets: we observe that
the proposed alternative answers in Compete often provide contradictory
contexts, thus the LLMs successfully abstain due to the knowledge conflicts.
We further dissect the ElectionQA23 dataset in Figure 7 (Appendix A and find
that LLMs’ abstain rate is often lower for elections in Africa and Asia,
shedding light on the potential fairness issues in LLM abstention.

Figure 4: Performance of Compete with ChatGPT in the two-step abstain and
retrieval setting. The proposed abstain-retrieve-abstain pipeline successfully
reduces the incorrect rate by at least 21.2%.

#### Abstain and Retrieval

Retrieval-augmented language models (Khattab et al., 2022; Shi et al., 2023;
Baek et al., 2023; Yu et al., 2023b; Thakur et al., 2023) have become the go-
to approach for augmenting LLMs’ knowledge access, while retrieving from an
external corpus could also introduce noisy documents and lead to wrong answers
(Wang et al., 2023c; Asai et al., 2023). We propose a 2-step abstain-based
framework to selectively activate retrieval and identify retrieval failures.
Specifically, abstain decisions are first made based on QA without retrieval.
If the LLM abstains, retrieval is then activated and the abstain mechanism is
applied to QA with retrieval: if the LLM still decides to abstain, it
indicates that the retrieval has failed to find relevant/helpful documents.
Results in Figure 4 demonstrate that this abstain-retrieve-abstain pipeline
successfully brings down the error rate by up to 56.6%, highlighting the
potential of abstain mechanisms in identifying retrieval failures.

Figure 5: Performance of abstain mechanisms with ChatGPT solving 3-hop
questions in K-Crosswords. Cooperate and Compete are better at localizing
knowledge gaps in multi-hop reasoning steps and reducing the incorrect rate by
at least 82%.

Method | MMLU | K-Cross. | Hellaswag | Propaganda
---|---|---|---|---
MoreInfo | 0.4433 | 0.7281 | 0.5830 | 0.6143
Probs | 0.4253 | 0.6643 | 0.4992 | 0.5302
SC Tres. | 0.2854 | 0.3428 | 0.2146 | 0.2750
Ask Cali. | 0.1747 | 0.2664 | 0.2088 | 0.3755
Verifier | 0.3343 | 0.2000 | 0.3624 | 0.2112
Coop-self | 0.4740 | 0.1131 | 0.3434 | 0.2263
Coop-others | 0.3560 | 0.5330 | 0.2164 | 0.2380
Compete | 0.4484 | 0.3929 | 0.4014 | 0.1935

Table 2: Abstain estimated calibration error (ECE) of approaches with
mistral-7B, the lower the better. The calibration of abstention scores for
approaches could vary significantly across tasks and datasets. Figure 6:
Abstain precision and recall of approaches on MMLU with _Mistral-7B_. While
most approaches feature higher precision, Compete achieves the best abstention
recall and could be employed where reliability is paramount.

#### Abstain in Multi-Hop

In multi-hop knowledge reasoning, LLMs’ knowledge is often limited in some,
but not all, of the reasoning steps. We investigate whether abstain strategies
could pinpoint the specific steps where LLMs should abstain due to knowledge
limitations: we employ the three-hop subset of the K-Crosswords dataset (Ding
et al., 2023), conduct abstain mechanisms on each of the hops, and present
results in Figure 5. It is demonstrated that our proposed Cooperate and
Compete are better in localizing knowledge limitations, with an error rate of
67.2% to 81.2% lower than the baselines.

#### Abstain ECE

Aside from a binary decision, abstain mechanisms could also provide a
continuous abstain likelihood based on token probabilities, heuristics, and
more333Details in Appendix B.. We calculate the _abstain ECE_ , estimated
calibration error (Guo et al., 2017) but with abstain probabilities and
abstain accuracy, on the MMLU dataset and present in Table 2. It is
demonstrated that the calibration of abstain likelihood correlates with the
abstain performance, with ask calibrate, Cooperate, and Compete being the top-
performing approaches.

#### Precision and Recall

We observe that among 14 abstain mechanisms, some emphasize coverage while
others emphasize answer reliability. We illustrate the abstain precision and
recall of approaches in Figure 6: It is illustrated that while most approaches
have higher precision than recall and thus lean towards answer coverage,
Compete has a much higher recall and stresses answer reliability. We argue
that these strategies could be ensembled to achieve balanced abstain
decisions, or employed based on whether the application scenario should
optimize coverage or reliability.

##  6 Related Work

Previous works on knowing what language models know have focused on
calibration, prompting, or training. _Calibration-based_ approaches attempt to
extract confidence scores from models to gauge their uncertainty (Sun et al.,
2022; Kuhn et al., 2023; Zhou et al., 2023a; Liu et al., 2023a). Previous
works have evaluated (Radford et al., 2019; Liang et al., 2023; Tao et al.,
2023; He et al., 2023) and improved the calibration of language models (Desai
and Durrett, 2020; Kong et al., 2020; Jagannatha and Yu, 2020; Kamath et al.,
2020; Jiang et al., 2021; Mielke et al., 2022; Lin et al., 2022; Tian et al.,
2023; Zhou et al., 2023b), while the calibration quality could vary for
different downstream tasks (Desai and Durrett, 2020; Wang et al., 2020;
Stengel-Eskin and Van Durme, 2023; Kalai and Vempala, 2023). In the setting of
AbstainQA where LLMs decide whether to abstain or not given a question,
calibration-based approaches would rely on a held-out set to set a threshold
over calibrated confidence scores: such a threshold and reliance on a held-out
set could jeopardize the generalization of calibration-based approaches across
knowledge domains and reasoning contexts.

_Prompting-based_ approaches leverage the instruction-following abilities of
LLMs and employ instructions to induce self-reflection and gauge whether the
generated answer should be trusted. (Kadavath et al., 2022) investigate
whether a “none-of-the-above” option or self-evaluation prompting would induce
good estimations of LLMs’ internal factuality. Huang et al. (2023a) follows
existing works (Kim et al., 2023; Shinn et al., 2023) to evaluate whether LLMs
could self-correct their own reasoning with a three-step prompting strategy.
Feng et al. (2023b) prompts LLM to reflect on whether more information is
needed for a given question to selectively solicit external knowledge. These
approaches, among other prompting-based methods for acknowledging knowledge
limitations (Wang et al., 2023a; Si et al., 2023), mostly rely on the hopes
that LLMs could indeed carry out self-reflection through simple prompting,
while their ability on this front might be subject to confounders and spurious
correlations (Kadavath et al., 2022).

_Training-based_ approaches focus on empowering LLMs with the ability to
acknowledge uncertainty or abstain through training with special data or
objectives. Cobbe et al. (2021) proposes to train an external verifier to
evaluate the correctness of LLM-generated answers in math reasoning. A series
of works also attempt to probe the hidden representations of LLMs for signals
of factuality and hallucination (Slobodkin et al., 2023; Azaria and Mitchell,
2023). Contemporary works have gained interest in improving LLMs’ factuality
and self-knowledge through instruction tuning and alignment (Zhang et al.,
2023; Yang et al., 2023; Sun et al., 2023; Bashlovkina et al., 2023), as well
as promoting abstention from a safety perspective (Huang et al., 2023b; Liu et
al., 2023b). While training-based approaches have shown great potential, they
would often require substantial computational resources to fine-tune LLMs and
might struggle to generalize across knowledge domains and contexts due to
their reliance on the patterns and examples in the training subset.

In this work, we propose Cooperate and Compete, two collaboration-based
approaches to enable robust LLM abstention through multi-LLM collaboration.
These approaches have shown great potential in improving LLM abstention in not
only direct QA, but also retrieval-augmented QA and multi-hop reasoning
scenarios.

##  7 Conclusion

We investigate AbstainQA, a setting where LLMs should abstain from answering
questions incorrectly. We curate a taxonomy of 11 abstain baselines across
three categories and propose Cooperate and Compete, two novel abstain
mechanisms that promote mechanistic reflection through multi-LLM collaboration
and challenging LLMs with knowledge conflicts. Extensive experiments on four
datasets demonstrate that Cooperate and Compete advances the state-of-the-art
in AbstainQA, while having the potential to improve retrieval-augmented LLMs,
multi-hop knowledge reasoning, and more.

## Limitations

While we primarily investigated and evaluated LLM abstention from a knowledge
perspective, the problem of abstaining is also important from a safety
perspective, preventing LLMs from generating harmful content or biased
responses (Huang et al., 2023b; Liu et al., 2023b). There are also
intersections between the knowledge and safety aspects of abstention, such as
abstaining from underspecified questions based on stereotypes of demographic
attributes. We plan to expand on this work and propose unified abstain
strategies for knowledge and safety scenarios in the future.

The abstain mechanisms and experiments in this work are exhaustive to the
extent that our computational budget permits, while we could not cover every
single approach and proposal from existing works. Nevertheless, we will make
the evaluation data, code, and framework publicly available so that new
abstain approaches, as well as datasets, can be seamlessly added to advance
LLM abstention research.

Our proposed collaboration-based approaches require prompting multiple LLMs
for feedback and could have more computational overhead, while the exact cost
would depend on the inference costs of other LLMs. We provide a summary of
inference costs in Table 5 and Appendix A.

In this work, we focus on a conceptual “abstain” functionality and develop
robust approaches to identify the knowledge gaps in large language models.
However, “abstain” doesn’t necessarily have to be part of the user interface
of LLMs: it could be an internal component of an LLM-based system, followed by
approaches to find better knowledge sources to fill the knowledge gaps or
methods to incorporate the uncertainty to model responses to enhance
trustworthiness (Zhou et al., 2024). Future work could further explore LLM
pipelines where these abstain approaches are one component towards reliable
question answering.

## Ethics Statement

Figure 7 highlights that LLM abstain decisions could also have fairness
implications, as they fail to abstain and assume the outcomes of future
elections more frequently for African and Asian countries. We similarly expect
varying levels of abstain performance on questions and prompts regarding
different demographics, communities, and perspectives, potentially due to
LLMs’ internal social biases (Blodgett et al., 2020; Jin et al., 2021; Bender
et al., 2021; Shaikh et al., 2023; Feng et al., 2023a). We plan to examine the
intersections of LLM social biases and their abstention abilities in future
work, especially in critical domains such as hate speech and misinformation
where failing to abstain from generating problematic content could have grave
consequences for marginalized communities.

## References

  * Amayuelas et al. (2023) Alfonso Amayuelas, Liangming Pan, Wenhu Chen, and William Wang. 2023.  Knowledge of knowledge: Exploring known-unknowns uncertainty with large language models.  _arXiv preprint arXiv:2305.13712_.
  * Asai et al. (2023) Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2023.  Self-rag: Learning to retrieve, generate, and critique through self-reflection.  _arXiv preprint arXiv:2310.11511_.
  * Azaria and Mitchell (2023) Amos Azaria and Tom Mitchell. 2023.  The internal state of an llm knows when its lying.  _arXiv preprint arXiv:2304.13734_.
  * Baek et al. (2023) Jinheon Baek, Soyeong Jeong, Minki Kang, Jong C Park, and Sung Hwang. 2023.  Knowledge-augmented language model verification.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 1720–1736.
  * Bansal et al. (2024) Rachit Bansal, Bidisha Samanta, Siddharth Dalmia, Nitish Gupta, Shikhar Vashishth, Sriram Ganapathy, Abhishek Bapna, Prateek Jain, and Partha Pratim Talukdar. 2024.  Llm augmented llms: Expanding capabilities through composition.
  * Bashlovkina et al. (2023) Vasilisa Bashlovkina, Zhaobin Kuang, Riley Matthews, Edward Clifford, Yennie Jun, William W Cohen, and Simon Baumgartner. 2023.  Trusted source alignment in large language models.  _arXiv preprint arXiv:2311.06697_.
  * Bender et al. (2021) Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021.  On the dangers of stochastic parrots: Can language models be too big?  In _Proceedings of the 2021 ACM conference on fairness, accountability, and transparency_ , pages 610–623.
  * Blodgett et al. (2020) Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020.  Language (technology) is power: A critical survey of “bias” in NLP.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5454–5476, Online. Association for Computational Linguistics.
  * Borgeaud et al. (2022) Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego De Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack Rae, Erich Elsen, and Laurent Sifre. 2022.  Improving language models by retrieving from trillions of tokens.  In _Proceedings of the 39th International Conference on Machine Learning_ , volume 162 of _Proceedings of Machine Learning Research_ , pages 2206–2240. PMLR.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 1877–1901. Curran Associates, Inc.
  * CH-Wang et al. (2023) Sky CH-Wang, Benjamin Van Durme, Jason Eisner, and Chris Kedzie. 2023.  Do androids know they’re only dreaming of electric sheep?  _arXiv preprint arXiv:2312.17249_.
  * Chen et al. (2023) Tong Chen, Hongwei Wang, Sihao Chen, Wenhao Yu, Kaixin Ma, Xinran Zhao, Dong Yu, and Hongming Zhang. 2023.  Dense x retrieval: What retrieval granularity should we use?  _arXiv preprint arXiv:2312.06648_.
  * Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021.  Training verifiers to solve math word problems.  _arXiv preprint arXiv:2110.14168_.
  * Cohen et al. (2023) Roi Cohen, Mor Geva, Jonathan Berant, and Amir Globerson. 2023.  Crawling the internal knowledge-base of language models.  In _Findings of the Association for Computational Linguistics: EACL 2023_ , pages 1811–1824.
  * De Cao et al. (2021) Nicola De Cao, Wilker Aziz, and Ivan Titov. 2021.  Editing factual knowledge in language models.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 6491–6506.
  * Desai and Durrett (2020) Shrey Desai and Greg Durrett. 2020.  Calibration of pre-trained transformers.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 295–302, Online. Association for Computational Linguistics.
  * Ding et al. (2023) Wenxuan Ding, Shangbin Feng, Yuhan Liu, Zhaoxuan Tan, Vidhisha Balachandran, Tianxing He, and Yulia Tsvetkov. 2023.  Knowledge crosswords: Geometric reasoning over structured knowledge with large language models.  _arXiv preprint arXiv:2310.01290_.
  * Du et al. (2023) Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. 2023.  Improving factuality and reasoning in language models through multiagent debate.  _arXiv preprint arXiv:2305.14325_.
  * Feng et al. (2023a) Shangbin Feng, Chan Young Park, Yuhan Liu, and Yulia Tsvetkov. 2023a.  From pretraining data to language models to downstream tasks: Tracking the trails of political biases leading to unfair NLP models.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 11737–11762, Toronto, Canada. Association for Computational Linguistics.
  * Feng et al. (2023b) Shangbin Feng, Weijia Shi, Yuyang Bai, Vidhisha Balachandran, Tianxing He, and Yulia Tsvetkov. 2023b.  Knowledge card: Filling llms’ knowledge gaps with plug-in specialized language models.
  * Fu et al. (2023) Xingyu Fu, Sheng Zhang, Gukyeong Kwon, Pramuditha Perera, Henghui Zhu, Yuhao Zhang, Alexander Hanbo Li, William Yang Wang, Zhiguo Wang, Vittorio Castelli, Patrick Ng, Dan Roth, and Bing Xiang. 2023.  Generate then select: Open-ended visual question answering guided by world knowledge.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 2333–2346, Toronto, Canada. Association for Computational Linguistics.
  * Guo et al. (2017) Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. 2017.  On calibration of modern neural networks.  In _International conference on machine learning_ , pages 1321–1330. PMLR.
  * Guu et al. (2020) Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020.  Retrieval augmented language model pre-training.  In _International conference on machine learning_ , pages 3929–3938. PMLR.
  * He et al. (2023) Guande He, Peng Cui, Jianfei Chen, Wenbo Hu, and Jun Zhu. 2023.  Investigating uncertainty calibration of aligned language models under the multiple-choice setting.  _arXiv preprint arXiv:2310.11732_.
  * Hendrycks et al. (2020) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020.  Measuring massive multitask language understanding.  In _International Conference on Learning Representations_.
  * Hernandez et al. (2023) Evan Hernandez, Belinda Z Li, and Jacob Andreas. 2023.  Measuring and manipulating knowledge representations in language models.  _arXiv preprint arXiv:2304.00740_.
  * Hu et al. (2023) Shengding Hu, Yifan Luo, Huadong Wang, Xingyi Cheng, Zhiyuan Liu, and Maosong Sun. 2023.  Won’t get fooled again: Answering questions with false premises.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 5626–5643, Toronto, Canada. Association for Computational Linguistics.
  * Huang et al. (2023a) Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. 2023a.  Large language models cannot self-correct reasoning yet.  _arXiv preprint arXiv:2310.01798_.
  * Huang et al. (2023b) Xiaowei Huang, Wenjie Ruan, Wei Huang, Gao Jin, Yizhen Dong, Changshun Wu, Saddek Bensalem, Ronghui Mu, Yi Qi, Xingyu Zhao, Kaiwen Cai, Yanghao Zhang, Sihao Wu, Peipei Xu, Dengyu Wu, André Freitas, and Mustafa A. Mustafa. 2023b.  A survey of safety and trustworthiness of large language models through the lens of verification and validation.  _ArXiv_ , abs/2305.11391.
  * Jagannatha and Yu (2020) Abhyuday Jagannatha and Hong Yu. 2020.  Calibrating structured output predictors for natural language processing.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 2078–2092, Online. Association for Computational Linguistics.
  * Ji et al. (2023) Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023.  Survey of hallucination in natural language generation.  _ACM Computing Surveys_ , 55(12):1–38.
  * Jiang et al. (2023) Albert Qiaochu Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, L’elio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023.  Mistral 7b.  _ArXiv_ , abs/2310.06825.
  * Jiang et al. (2021) Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. 2021.  How can we know when language models know? on the calibration of language models for question answering.  _Transactions of the Association for Computational Linguistics_ , 9:962–977.
  * Jin et al. (2021) Xisen Jin, Francesco Barbieri, Brendan Kennedy, Aida Mostafazadeh Davani, Leonardo Neves, and Xiang Ren. 2021.  On transferability of bias mitigation effects in language model fine-tuning.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3770–3783, Online. Association for Computational Linguistics.
  * Kadavath et al. (2022) Saurav Kadavath, Tom Conerly, Amanda Askell, T. J. Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zachary Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, John Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom B. Brown, Jack Clark, Nicholas Joseph, Benjamin Mann, Sam McCandlish, Christopher Olah, and Jared Kaplan. 2022.  Language models (mostly) know what they know.  _ArXiv_ , abs/2207.05221.
  * Kalai and Vempala (2023) Adam Tauman Kalai and Santosh S Vempala. 2023.  Calibrated language models must hallucinate.  _arXiv preprint arXiv:2311.14648_.
  * Kamath et al. (2020) Amita Kamath, Robin Jia, and Percy Liang. 2020.  Selective question answering under domain shift.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5684–5696, Online. Association for Computational Linguistics.
  * Kandpal et al. (2023) Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. 2023.  Large language models struggle to learn long-tail knowledge.  In _International Conference on Machine Learning_ , pages 15696–15707. PMLR.
  * Kang et al. (2023) Junmo Kang, Hongyin Luo, Yada Zhu, James Glass, David Cox, Alan Ritter, Rogerio Feris, and Leonid Karlinsky. 2023.  Self-specialization: Uncovering latent expertise within large language models.  _arXiv preprint arXiv:2310.00160_.
  * Kasai et al. (2022) Jungo Kasai, Keisuke Sakaguchi, Yoichi Takahashi, Ronan Le Bras, Akari Asai, Xinyan Yu, Dragomir Radev, Noah A Smith, Yejin Choi, and Kentaro Inui. 2022.  Realtime qa: What’s the answer right now?  _arXiv preprint arXiv:2207.13332_.
  * Khattab et al. (2022) Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts, and Matei Zaharia. 2022.  Demonstrate-search-predict: Composing retrieval and language models for knowledge-intensive nlp.  _arXiv preprint arXiv:2212.14024_.
  * Kim et al. (2023) Geunwoo Kim, Pierre Baldi, and Stephen McAleer. 2023.  Language models can solve computer tasks.  _arXiv preprint arXiv:2303.17491_.
  * Kong et al. (2020) Lingkai Kong, Haoming Jiang, Yuchen Zhuang, Jie Lyu, Tuo Zhao, and Chao Zhang. 2020.  Calibrated language model fine-tuning for in- and out-of-distribution data.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1326–1340, Online. Association for Computational Linguistics.
  * Kuhn et al. (2023) Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023.  Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation.  _arXiv preprint arXiv:2302.09664_.
  * Liang et al. (2023) Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher R’e, Diana Acosta-Navas, Drew A. Hudson, E. Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang, Keshav Santhanam, Laurel J. Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan S. Kim, Neel Guha, Niladri S. Chatterji, O. Khattab, Peter Henderson, Qian Huang, Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas F. Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. 2023.  Holistic evaluation of language models.  _Annals of the New York Academy of Sciences_ , 1525:140 – 146.
  * Lin et al. (2022) Stephanie Lin, Jacob Hilton, and Owain Evans. 2022.  Teaching models to express their uncertainty in words.  _arXiv preprint arXiv:2205.14334_.
  * Liu et al. (2023a) Xin Liu, Muhammad Khalifa, and Lu Wang. 2023a.  Litcab: Lightweight calibration of language models on outputs of varied lengths.  _arXiv preprint arXiv:2310.19208_.
  * Liu et al. (2023b) Yang Liu, Yuanshun Yao, Jean-Francois Ton, Xiaoying Zhang, Ruocheng Guo Hao Cheng, Yegor Klochkov, Muhammad Faaiz Taufiq, and Hang Li. 2023b.  Trustworthy llms: a survey and guideline for evaluating large language models’ alignment.  _arXiv preprint arXiv:2308.05374_.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Luo et al. (2023) Ziyang Luo, Can Xu, Pu Zhao, Xiubo Geng, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2023.  Augmented large language models with parametric knowledge guiding.  _arXiv preprint arXiv:2305.04757_.
  * Mallen et al. (2023) Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023.  When not to trust language models: Investigating effectiveness of parametric and non-parametric memories.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 9802–9822.
  * Mielke et al. (2022) Sabrina J Mielke, Arthur Szlam, Emily Dinan, and Y-Lan Boureau. 2022.  Reducing conversational agents’ overconfidence through linguistic calibration.  _Transactions of the Association for Computational Linguistics_ , 10:857–872.
  * Min et al. (2020) Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020.  Ambigqa: Answering ambiguous open-domain questions.  _arXiv preprint arXiv:2004.10645_.
  * Nakano et al. (2021) Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Ouyang Long, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. 2021.  Webgpt: Browser-assisted question-answering with human feedback.  _ArXiv_ , abs/2112.09332.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ , 35:27730–27744.
  * Patidar et al. (2023) Mayur Patidar, Prayushi Faldu, Avinash Singh, Lovekesh Vig, Indrajit Bhattacharya, and Mausam. 2023.  Do I have the knowledge to answer? investigating answerability of knowledge base questions.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 10341–10357, Toronto, Canada. Association for Computational Linguistics.
  * Petroni et al. (2019) Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019.  Language models as knowledge bases?  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2463–2473.
  * Piskorski et al. (2023) Jakub Piskorski, Nicolas Stefanovitch, Giovanni Da San Martino, and Preslav Nakov. 2023.  Semeval-2023 task 3: Detecting the category, the framing, and the persuasion techniques in online news in a multi-lingual setup.  In _Proceedings of the the 17th International Workshop on Semantic Evaluation (SemEval-2023)_ , pages 2343–2361.
  * Press et al. (2023) Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah Smith, and Mike Lewis. 2023.  Measuring and narrowing the compositionality gap in language models.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 5687–5711, Singapore. Association for Computational Linguistics.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Sclar et al. (2023) Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2023.  Quantifying language models’ sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting.  _arXiv preprint arXiv:2310.11324_.
  * Shaikh et al. (2023) Omar Shaikh, Hongxin Zhang, William Held, Michael Bernstein, and Diyi Yang. 2023.  On second thought, let’s not think step by step! bias and toxicity in zero-shot reasoning.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 4454–4470, Toronto, Canada. Association for Computational Linguistics.
  * Shi et al. (2023) Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. 2023.  Replug: Retrieval-augmented black-box language models.  _arXiv preprint arXiv:2301.12652_.
  * Shinn et al. (2023) Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik R Narasimhan, and Shunyu Yao. 2023.  Reflexion: Language agents with verbal reinforcement learning.  In _Thirty-seventh Conference on Neural Information Processing Systems_.
  * Si et al. (2023) Chenglei Si, Weijia Shi, Chen Zhao, Luke Zettlemoyer, and Jordan Boyd-Graber. 2023.  Getting more out of mixture of language model reasoning experts.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 8234–8249.
  * Slobodkin et al. (2023) Aviv Slobodkin, Omer Goldman, Avi Caciularu, Ido Dagan, and Shauli Ravfogel. 2023.  The curious case of hallucinatory (un) answerability: Finding truths in the hidden states of over-confident large language models.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 3607–3625.
  * Stengel-Eskin and Van Durme (2023) Elias Stengel-Eskin and Benjamin Van Durme. 2023.  Calibrated interpretation: Confidence estimation in semantic parsing.  _Transactions of the Association for Computational Linguistics_ , 11:1213–1231.
  * Sun et al. (2022) Meiqi Sun, Wilson Yan, Pieter Abbeel, and Igor Mordatch. 2022.  Quantifying uncertainty in foundation models via ensembles.  In _NeurIPS 2022 Workshop on Robustness in Sequence Modeling_.
  * Sun et al. (2023) Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. 2023.  Aligning large multimodal models with factually augmented rlhf.  _arXiv preprint arXiv:2309.14525_.
  * Tao et al. (2023) Linwei Tao, Younan Zhu, Haolan Guo, Minjing Dong, and Chang Xu. 2023.  A benchmark study on calibration.  _arXiv preprint arXiv:2308.11838_.
  * Thakur et al. (2023) Nandan Thakur, Luiz Bonifacio, Xinyu Zhang, Odunayo Ogundepo, Ehsan Kamalloo, David Alfonso-Hermelo, Xiaoguang Li, Qun Liu, Boxing Chen, Mehdi Rezagholizadeh, et al. 2023.  Nomiracl: Knowing when you don’t know for robust multilingual retrieval-augmented generation.  _arXiv preprint arXiv:2312.11361_.
  * Tian et al. (2023) Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher Manning. 2023.  Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 5433–5442, Singapore. Association for Computational Linguistics.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Wang et al. (2023a) Boshi Wang, Xiang Yue, and Huan Sun. 2023a.  Can ChatGPT defend its belief in truth? evaluating LLM reasoning via debate.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 11865–11881, Singapore. Association for Computational Linguistics.
  * Wang et al. (2020) Shuo Wang, Zhaopeng Tu, Shuming Shi, and Yang Liu. 2020.  On the inference calibration of neural machine translation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 3070–3079, Online. Association for Computational Linguistics.
  * Wang et al. (2022) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022.  Self-consistency improves chain of thought reasoning in language models.  In _The Eleventh International Conference on Learning Representations_.
  * Wang et al. (2023b) Yike Wang, Shangbin Feng, Heng Wang, Weijia Shi, Vidhisha Balachandran, Tianxing He, and Yulia Tsvetkov. 2023b.  Resolving knowledge conflicts in large language models.  _arXiv preprint arXiv:2310.00935_.
  * Wang et al. (2023c) Zhiruo Wang, Jun Araki, Zhengbao Jiang, Md Rizwan Parvez, and Graham Neubig. 2023c.  Learning to filter context for retrieval-augmented generation.  _arXiv preprint arXiv:2311.08377_.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022.  Chain-of-thought prompting elicits reasoning in large language models.  _Advances in Neural Information Processing Systems_ , 35:24824–24837.
  * West et al. (2023) Peter West, Ximing Lu, Nouha Dziri, Faeze Brahman, Linjie Li, Jena D Hwang, Liwei Jiang, Jillian Fisher, Abhilasha Ravichander, Khyathi Chandu, Benjamin Newman, Wei Koh Pang, Allyson Ettinger, and Yejin Choi. 2023.  The generative AI paradox:" What It Can Create, It May Not Understand".  _arXiv preprint arXiv:2311.00059_.
  * Whitehead et al. (2022) Spencer Whitehead, Suzanne Petryk, Vedaad Shakib, Joseph Gonzalez, Trevor Darrell, Anna Rohrbach, and Marcus Rohrbach. 2022.  Reliable visual question answering: Abstain rather than answer incorrectly.  In _European Conference on Computer Vision_ , pages 148–166. Springer.
  * Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. 2019.  Huggingface’s transformers: State-of-the-art natural language processing.  _ArXiv_ , abs/1910.03771.
  * Xie et al. (2023) Jian Xie, Kai Zhang, Jiangjie Chen, Renze Lou, and Yu Su. 2023.  Adaptive chameleon or stubborn sloth: Unraveling the behavior of large language models in knowledge conflicts.  _arXiv preprint arXiv:2305.13300_.
  * Xu et al. (2023) Fangyuan Xu, Weijia Shi, and Eunsol Choi. 2023.  Recomp: Improving retrieval-augmented lms with compression and selective augmentation.  _arXiv preprint arXiv:2310.04408_.
  * Yang et al. (2023) Yuqing Yang, Ethan Chern, Xipeng Qiu, Graham Neubig, and Pengfei Liu. 2023.  Alignment for honesty.  _arXiv preprint arXiv:2312.07000_.
  * Yu et al. (2023a) Jifan Yu, Xiaozhi Wang, Shangqing Tu, Shulin Cao, Daniel Zhang-Li, Xin Lv, Hao Peng, Zijun Yao, Xiaohan Zhang, Hanming Li, et al. 2023a.  Kola: Carefully benchmarking world knowledge of large language models.  _arXiv preprint arXiv:2306.09296_.
  * Yu et al. (2023b) Wenhao Yu, Hongming Zhang, Xiaoman Pan, Kaixin Ma, Hongwei Wang, and Dong Yu. 2023b.  Chain-of-note: Enhancing robustness in retrieval-augmented language models.  _arXiv preprint arXiv:2311.09210_.
  * Zellers et al. (2019) Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019.  Hellaswag: Can a machine really finish your sentence?  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4791–4800.
  * Zhang et al. (2023) Hanning Zhang, Shizhe Diao, Yong Lin, Yi R Fung, Qing Lian, Xingyao Wang, Yangyi Chen, Heng Ji, and Tong Zhang. 2023.  R-tuning: Teaching large language models to refuse unknown questions.  _arXiv preprint arXiv:2311.09677_.
  * Zhou et al. (2023a) Han Zhou, Xingchen Wan, Lev Proleev, Diana Mincu, Jilin Chen, Katherine Heller, and Subhrajit Roy. 2023a.  Batch calibration: Rethinking calibration for in-context learning and prompt engineering.  _arXiv preprint arXiv:2309.17249_.
  * Zhou et al. (2024) Kaitlyn Zhou, Jena D. Hwang, Xiang Ren, and Maarten Sap. 2024.  Relying on the unreliable: The impact of language models’ reluctance to express uncertainty.
  * Zhou et al. (2023b) Kaitlyn Zhou, Dan Jurafsky, and Tatsunori Hashimoto. 2023b.  Navigating the grey area: How expressions of uncertainty and overconfidence affect language models.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 5506–5524, Singapore. Association for Computational Linguistics.

##  Appendix A Analysis (cont.)

#### Retrieval Compatibility

We investigate whether abstain mechanisms could work with retrieval,
specifically when questions in the MMLU dataset are presented together with a
retrieved document from the WikiSearch API. Table 3 demonstrates that while
ask calibrate is negatively impacted by retrieval, our proposed Cooperate and
Compete are in general robust towards additional prepended context.

Method | Ret. | R-Acc | ER | A-Acc | A-F1
---|---|---|---|---|---
SC Thres. | ✗ | 0.604 | 0.140 | 0.641 | 0.551
✓ | 0.634 | 0.160 | 0.652 | 0.611
Ask Cali. | ✗ | 0.648 | 0.141 | 0.639 | 0.647
✓ | 0.515 | 0.027 | 0.512 | 0.156
Coop-self | ✗ | 0.571 | 0.059 | 0.564 | 0.601
✓ | 0.567 | 0.048 | 0.536 | 0.589
Compete | ✗ | 0.735 | 0.140 | 0.640 | 0.700
✓ | 0.720 | 0.128 | 0.618 | 0.670

Table 3: Performance of abstain strategies on MMLU with _Mistral-7B_ with or
without retrieval (Ret.).

#### Abstain and Fairness

We divide the ElectionQA23 dataset into subsets of which continent is the
election held in and present model performance in Figure 7. It is illustrated
that LLMs tend to abstain less and assume the outcomes of future elections for
Africa and Asia, indicating fairness concerns of abstain mechanisms that might
underserve marginalized and underdeveloped communities.

Figure 7: Abstain rate on Election23QA with _Mistral-7B_ divided into where
the election takes place. The lowest abstain rate for each approach among
continents is highlighted in bold. Figure 8: Performance of instruction tuning
when trained with one dataset/LLM and tested for another. Training on a held-
out set harms the generalization of abstain accuracy across tasks and LLMs.
Figure 9: Performance with three abstain mechanisms with increasing size of
model parameters.

Figure 10: Model performance of _without retrieval_ , _with retrieval_ , and
_retrieval+abstain_ settings with the Compete approach.

#### Retrieval Failure (cont.)

We present more results with the two-step abstention in retrieval-augmented
LLMs in Figure 10. Across three LLMs and four datasets, the abstain-retrieve-
abstain pipeline successfully brings down the incorrect rate and accounts for
retrieval failure.

#### Abstain and Multi-Hop (cont.)

We present more results with the multi-hop abstention in Figure 11. it is
demonstrated that our proposed Cooperate and Compete are consistently better
at pinpointing knowledge limitations in specific reasoning steps across LLMs
and datasets.

Figure 11: Model performance in the multi-hop knowledge reasoning setting.

#### Scaling Laws

We investigate whether LLM abstention with these mechanisms could be improved
through scaling, by evaluating three abstain mechanisms with varying sizes of
the LLaMA2 LLM. Results in Figure 9 demonstrate that the abstain performance
is not steadily increasing with model size, showcasing that empowering LLMs
with abstention abilities is not solely solvable through scaling up.

Figure 12: Performance of the Compete mechanism with _Mistral-7B_ across the
57 subtasks in MMLU.

#### Shift in Knowledge Domains

We investigate whether LLM abstention abilities could be impacted by which
knowledge domain the given question is about. Specifically, we present the
abstain accuracy metric for different knowledge domains in the MMLU dataset in
Figure 12. It is demonstrated that the abstention decisions are indeed of
varying quality given the knowledge domain, from 25% to 87.5%, potentially
informed by the LLM’s underlying knowledge abilities regarding that domain.

#### Domain Transfer

While instruction tuning is a strong baseline in Section 4, its reliance on a
held-out set for training might jeopardize its generalization across knowledge
domains and LLMs. To this end, we conduct a transfer study in Figure 8 where
there is a transfer across datasets or LLMs between training and testing. It
is illustrated that instruction tuning approaches struggle to generalize
across knowledge domains and LLMs, resulting in performance drops sometimes as
large as 33.8% in abstain accuracy.

#### Prompt Robustness

Since LLMs could be sensitive to minor changes in prompts (Sclar et al.,
2023), we experiment with minor paraphrases of the prompts used in Cooperate
and Compete and present the results in Table 4. It is demonstrated that the
proposed abstain mechanisms are relatively robust to minor changes in prompt
phrasing.

Method | R-Acc | ER | A-Acc | A-F1
---|---|---|---|---
Coop-self-1 | .540 | .040 | .544 | .543
Coop-self-2 | .564 | .035 | .498 | .553
Coop-self-3 | .562 | .032 | .504 | .562
Compete-1 | .672 | .088 | .590 | .671
Compete-2 | .698 | .126 | .623 | .680
Compete-3 | .655 | .077 | .577 | .662

Table 4: Performance of Cooperate and Compete with variations in the prompt
format, on the MMLU dataset with the _mistral-7B_ language model.

#### LLM Overhead

Different abstain mechanisms have different computational overhead and the
most expensive part is the number of LLM inference requests for each question.
We present a summary in Table 5.

Method | # | Method | #
---|---|---|---
NOTA | 2 | reflect | 2
MoreInfo | 2 | Probs | 3
SC Thres. | k+1 | Ask Cali. | 6
Temp. | 3 | Verifier | 3
Instruct | 2+ft | Embedding | 4
Gen+Match | 3 | Coop-self | 8
Coop-others | 2+o | Compete | 2+o

Table 5: Overhead of LLM inference requests for different abstain mechanisms.
“ft” denotes additional fine-tuning and “o” denotes that other reviewer LLMs
are also called once.

#### Working Examples

We present qualitative examples of Cooperate (Tables 13 and 14 for _self_ ,
Tables 15 and 16 for _others_) and Compete (Tables 17, 18, 19, and 20).

##  Appendix B Experiment Details

###  B.1 Dataset Details

We employ six datasets in the experiments, four for the main experiments, and
two _abstain absolute_ datasets.

  * •

MMLU (Hendrycks et al., 2020): we randomly downsample the official validation
and test splits into 1,000 questions each as the held-out set and test set.

  * •

K-Crosswords (Ding et al., 2023): we follow the official splits to employ the
validation set of 1,094 questions as the held-out set and the test set of
1,007 questions.

  * •

Hellaswag (Zellers et al., 2019): we randomly sample the official validation
set into 1,000 questions each as the held-out set and test set.

  * •

Propaganda (Piskorski et al., 2023): we create a random split of 231 questions
as the held-out set and 200 questions as the test set based on the official
validation split. We randomly sample three other propaganda tactics
accompanied by the correct tactic to form four-way multi-hop choice questions.

  * •

AmbigQA (Min et al., 2020): we randomly downsample the official validation and
test splits into 1,000 questions each as the held-out set and test set.

  * •

ElectionQA23: we first collect the overview paragraphs of the Wikipedia pages
of elections held in
2023444https://en.wikipedia.org/wiki/List_of_elections_in_2023 and employ
ChatGPT to compose 10 questions for each election in a four-way multiple-
choice format. We conduct manual evaluation to remove low-quality or ambiguous
questions and create a held-out set of 67 and a test set of 200 questions.

###  B.2 Model Details

We employ three large language models to evaluate abstain mechanisms: 1)
_mistral-7B_ , through the mistralai/Mistral-7B-Instruct-v0.1 checkpoint on
Huggingface (Wolf et al., 2019); 2) _llama2-70B_ , through the meta-
llama/Llama-2-70b-chat-hf checkpoint on Huggingface; 3) _ChatGPT_ , through
the gpt-3.5-turbo-instruct checkpoint on OpenAI API.

###  B.3 Implementation Details

for every single baseline and approach, along with ECE score extraction,
prompt for prompting-based and proposed approaches

  * •

Token Probability: The token probability of the answer token(s) is employed as
$p(\boldsymbol{a})$. The abstain likelihood is obtained through
$1-p(\boldsymbol{a})$.

  * •

Temperature Scaling: We search for an optimal $\tau$ value from 0.1 to 10 on
the held-out set $\mathcal{H}$. The abstain likelihood is obtained through
$1-p(\boldsymbol{a})$ after temperature scaling.

  * •

Ask for Calibration: We follow Tian et al. (2023) and employ the prompt in
Table 6 to elicit verbalized confidence scores. The abstain likelihood is
obtained through $1-p(\boldsymbol{a})$ with verbalized confidence scores.

  * •

Hidden Layers: We employ the feature-extraction pipeline with HuggingFace
(Wolf et al., 2019) to extract $\mathbf{e}_{\boldsymbol{q}}$. A linear layer
of size (dim, 2) is then employed for linear probing where dim is the
dimension of $\mathbf{e}_{\boldsymbol{q}}$. The abstain likelihood is obtained
as the $\ell=\textit{reject}$ likelihood after softmax.

  * •

External Verifier: We employ RoBERTa (roberta-base) (Liu et al., 2019) as
$\mathrm{LLM}-\textit{enc}$ for binary classification. The abstain likelihood
is obtained as the $\ell=\textit{reject}$ likelihood with the [CLS] token.

  * •

Instruction Tuning: We fine-tune ChatGPT (gpt-3.5-turbo-instruct) with the
instruction tuning triplets. The _instruction_ is “Answer the following
question. If you don’t have enough knowledge, abstain by saying ’sorry, I
don’t have enough knowledge to answer this question.”’ The _abstain message_
is “sorry, I don’t have enough knowledge to answer this question.” The abstain
likelihood is obtained as the token probability of the answer token, or that
of _“sorry”_ if it abstains.

  * •

Self-Reflect: Following Kadavath et al. (2022), we employ the prompt in Table
7. The abstain likelihood is obtained as the token probability of choosing
“B”.

  * •

More Information: Following Feng et al. (2023b), we employ the prompt in Table
8. The abstain likelihood is obtained as the token probability of “yes”.

  * •

Generate and Match: We employ the prompt in Table 9. The abstain likelihood is
obtained as the probability of “no”, or $1-$ that of “yes”.

  * •

None-of-the-Above: The abstain likelihood is obtained as the probability of
choosing the NOTA option.

  * •

Self-Consistency Threshold: We employ $k=5$ chain-of-thought paths by default.
The abstain likelihood is obtained as
$1-\mathrm{plu}(\mathrm{LLM},\boldsymbol{q},k)$.

  * •

Cooperate: We employ the prompt in Tables 10 and 11 for the _self_ and
_others_ modes. The abstain likelihood is obtained as the token probability of
choosing “B”.

  * •

Compete: We employ the prompt in Table 12 and employ three alternative answers
by default. The abstain likelihood is obtained as the probability of the new
answer if it changes, or $1-$ that of the original answer if it doesn’t
change, then averaged across all alternative answers.

Provide your best guess for the following question. Give ONLY the guess, no
other words or explanation.
---
For example:
Guess: <most likely guess, as short as possible; not a complete sentence, just
the guess!>
The question is: <question>
_[LLM-generated answer]_
Provide the probability that your guess is correct. Give ONLY the probability,
no other words or explanation.
For example:
Probability: <the probability between 0.0 and 1.0 that your guess is correct,
without any extra commentary whatsoever; just the probability!>
Probability: _[LLM-generated probability]_
Table 6: Prompt for Ask for Calibration. <question>
---
_[LLM-generated answer]_
The above answer is:
A. True
B. False
The answer is _[LLM-generated A/B]_
Table 7: Prompt for Self-Reflect. <question>
---
Do you need more information to answer this question? (Yes or No)
_[LLM-generated yes/no]_
Table 8: Prompt for More Information. <question without multiple-choice
options>
---
Proposed answer: _[LLM-generated answer]_
<options>
Does the proposed answer exist in the options?
_[LLM-generated yes/no]_
Table 9: Prompt for Generate and Match. // obtain proposed answer
---
Question: <question>
Answer: _[generated proposed answer]_
// obtain feedback from self-specialized experts
for domain in ["factual information", "commonsense knowledge", "mathematical
knowledge"]:
Generate some knowledge about the question, focusing on <domain>: _[generated
domain knowledge]_
Knowledge: <generated domain knowledge>
Question: <question>
Answer: <generated proposed answer>
Please review the proposed answer and provide feedback on its correctness.
Feedback: _[generated feedback]_
// area-chairing for abstain decision
Question: <question>
Proposed Answer: <generated proposed answer>
Feedback 1: <generated feedback from expert 1>
…
Feedback k: <generated feedback from expert k>
Based on the feedback, the proposed answer is:
A. True
B. False
The answer is _[A/B]_.
Table 10: Prompt for Cooperate-_self_. // obtain proposed answer
---
Question: <question>
Answer: _[generated proposed answer]_
// obtain feedback from other LLMs as reviewers for llm in list_of_other_llms:
Question: <question>
Answer: <generated proposed answer>
Please review the proposed answer and provide feedback on its correctness.
Feedback: _[generated feedback using llm]_
// area-chairing for abstain decision
Question: <question>
Proposed Answer: <generated proposed answer>
Feedback 1: <generated feedback from llm 1>
…
Feedback k: <generated feedback from llm k>
Based on the feedback, the proposed answer is:
A. True
B. False
The answer is _[A/B]_.
Table 11: Prompt for Cooperate-_others_. // obtain proposed answer
---
Question: <question>
Answer: _[generated proposed answer]_
// obtain alternative answers if multiple-choice:
<alternative answer> = randomly select another unchosen answer
else:
Question: <question>
Answer: <generated proposed answer>
Please propose an alternative answer: _[alternative answer]_
// obtain a supporting paragraph for the alternative answer
Question: <question>
Generate a knowledge paragraph about <alternative answer>: _[generated
alternative passage]_
// challenge!
Answer the question with the following knowledge: feel free to ignore
irrelevant or wrong information.
Knowledge: <generated alternative passage>
Question: <question>
Answer: _[new generated answer]_
// abstain decision
if <new generated answer> == <genearted proposed answer>:
abstain = False
else:
abstain = True
Table 12: Prompt for Compete. Question: The following information pertains to
Dash Co.’s utility bills: Period covered Amount Date paid April 16 ‚Äì May 15
$5000 June 1 May 16 ‚Äì June 15 $6000 July 1 June 16 ‚Äì July 15 $8000 August
1 What is the amount that Dash should report as a liability in its June 30
balance sheet?
---
A: $6,000
B: $7,000
C: $10,000
D: $14,000
Choose one answer from the above choices. The answer is B: $7,000
Feedback 1: The correct answer is actually A: $6,000. The amount that Dash
should report as a liability in its June 30 balance sheet is the utility bill
for the period of May 16 - June 15, which is $6,000. The subsequent utility
bills for June 16 - July 15 and July 16 - August 15 will be reported as
liabilities in future balance sheets.
Feedback 2: The answer is incorrect. To determine the liability that Dash
should report in its June 30 balance sheet, we need to look at the amount that
is unpaid as of June 30. According to the information given, the utility bill
for April 16 - May 15 was paid on June 1. Therefore, Dash should report the
unpaid amount for the period May 16 - June 15 as a liability on its June 30
balance sheet. This amount is $6,000. Thus, the correct answer is A: $6,000.
Feedback 3: The proposed answer is incorrect.
Based on the feedback, the proposed answer is:
A. True
B. False
The answer is
B. False
Table 13: Example one for Cooperate-self. The reviewers provided different
feedback, while the area chair LLM merged into a final abstain decision.
Question: Compared with the current in an LED lamp the current in the
connecting wire is
---
A: less.
B: more.
C: the same.
D: Need more information.
Choose one answer from the above choices. The answer is A: less.
Feedback 1: The proposed answer, A: less, is incorrect. The answer C: the
same, would be more appropriate. In an LED lamp, the current flows from the
power source through the connecting wire and into the LED. The current in the
wire would be the same as the current in the LED if there are no resistive
elements in the wire causing a drop in voltage.
Feedback 2: The proposed answer is incorrect. The statement does not provide
enough information to determine the current in the connecting wire compared to
the current in an LED lamp. Therefore, the correct answer is D: Need more
information.
Feedback 3: The proposed answer is correct. In an LED lamp, the current in the
connecting wire is typically less compared to the current in the LED itself.
This is because the LED lamp is designed to limit the amount of current
flowing through the LED to ensure proper operation and prevent damage. The
connecting wire acts as a conductor to allow the flow of current from the
power source to the LED, but it does not have as high a current as the LED
itself.
Based on the feedback, the proposed answer is:
A. True
B. False
The answer is
B. False.
Table 14: Example two for Cooperate-self. The reviewers provided different
feedback, while the area chair LLM merged into a final abstain decision.
Question: The following information pertains to Dash Co.’s utility bills:
Period covered Amount Date paid April 16 ‚Äì May 15 $5000 June 1 May 16 ‚Äì
June 15 $6000 July 1 June 16 ‚Äì July 15 $8000 August 1 What is the amount
that Dash should report as a liability in its June 30 balance sheet?
---
A: $6,000
B: $7,000
C: $10,000
D: $14,000
Choose one answer from the above choices. The answer is B: $7,000
Feedback 1: The answer provided is not correct. The correct amount that Dash
should report as a liability in its June 30 balance sheet is $6,000. This is
because the amount for the period of May 16 - June 15 has already been paid on
June 1, and does not need to be reported as a liability. The only amount that
is still outstanding and needs to be reported is the amount for the period of
June 16 - July 15, which is $6,000. Therefore, the correct answer is A:
$6,000.
Feedback 2: The proposed answer is incorrect. Dash should report $7,000 as a
liability in its June 30 balance sheet, but not $6,000.
Here’s the reasoning:
1\. The period covered by the utility bills is May 16 ‚Äì June 15.
2\. The amount due is $6,000, and Dash paid $5,000 on May 16 ‚Äì May 15 and
$1,000 on June 16 ‚Äì June 15.
3\. The date paid for the $6,000 bill is June 16 ‚Äì June 15 and the date paid
for the $1,000 bill is June 16 ‚Äì June 15.
4\. Since the payment of the $1,000 bill has already
Feedback 3: The correct answer is indeed B: $7,000.
The explanation is as follows:
The June 30 balance sheet should report a liability for the utility bills that
have been incurred but not yet
Based on the feedback, the proposed answer is:
A. True
B. False
The answer is
B. False.
Table 15: Example one for Cooperate-others. The reviewers provided different
feedback, while the area chair LLM merged into a final abstain decision.
Question: Who kills Tony at the end of the film ’West Side Story’?
---
A: Riff
B: Chino
C: Bernardo
D: He kills himself
Choose one answer from the above choices. The answer is B: Chino
Feedback 1: The proposed answer is correct. Chino is the one who kills Tony at
the end of the film ’West Side Story’.
Feedback 2: The answer provided is incorrect. In the film ’West Side Story’,
it is Bernardo who kills Tony. Thank you for providing the correct answer.
Feedback 3: The proposed answer is incorrect. Tony is killed by Bernardo, not
Chino. The correct answer is C: Bernardo.
New proposed answer:
Answer: Who kills Tony at the end of the film ’West Side Story’
Based on the feedback, the proposed answer is:
A. True
B. False
The answer is
B. False. The correct answer is C: Bernardo.
Table 16: Example two for Cooperate-others. The reviewers provided different
feedback, while the area chair LLM merged into a final abstain decision.
Question: The following information pertains to Dash Co.’s utility bills:
Period covered Amount Date paid April 16 ‚Äì May 15 $5000 June 1 May 16 ‚Äì
June 15 $6000 July 1 June 16 ‚Äì July 15 $8000 August 1 What is the amount
that Dash should report as a liability in its June 30 balance sheet?
---
A: $6,000
B: $7,000
C: $10,000
D: $14,000
Generate a knowledge paragraph about A.
Dash Co. should report $6,000 as a liability in its June 30 balance sheet.
This is because the utility bills for the period from May 16 to June 15 amount
to $6,000. The balance sheet is a financial statement that provides an
overview of a company’s financial position at a specific point in time.
Liabilities are obligations or debts owed by a company, which are classified
as current liabilities if they are expected to be settled within one year. In
this case, the utility bill for the period ending on June 15 falls within the
one-year timeframe and should therefore be reported as a current liability.
Therefore, the correct answer is A: $6,000.
New answer: A: $6,000.
Table 17: Example one for Compete. The LLM is impacted by the alternative
answer and should abstain. Question: A flashing red traffic light signifies
that a driver should do what?
---
A: stop
B: speed up
C: proceed with caution
D: honk the horn
Generate a knowledge paragraph about D.
D: Honking the horn is the appropriate action when encountering a flashing red
traffic light. This is an important traffic rule that drivers must follow to
maintain safety on the roads. When encountering a flashing red light, it is
crucial for all drivers to be aware of traffic signals and adhere to their
instructions to ensure a smooth flow of traffic and prevent any potential
hazards. Honking the horn, on the other hand, is generally used as a way to
alert other drivers or pedestrians of potential dangers or to communicate with
them in certain situations.
New answer: A: stop.
Table 18: Example two for Compete. The LLM is not impacted by the alternative
answer. Question: Kelsen’s theory of law is called pure theory because Kelsen:
---
A: Purely discussed jurisprudence only
B: Defined law in according with morality and purity
C: Separated law from religion, ethics, sociology and history
D: Discussed law purely in terms of justice
Generate a knowledge paragraph about D.
Kelsen’s theory of law is called pure theory because he discussed law purely
in terms of justice. According to Kelsen, law should be analyzed and
understood independently of any moral or ethical considerations. He believed
that law should be studied and interpreted solely through its internal system
of norms, without any reference to external moral values or social contexts.
Kelsen’s pure theory of law aimed to provide a "science of law" by separating
law from other disciplines such as religion, ethics, sociology, and history.
He argued that law should be treated as a self-contained system, with its own
hierarchy of norms and principles. This meant that the validity of laws should
be determined based on their consistency with higher-level norms, rather than
any inherent moral or ethical qualities.
By focusing purely on the concept of justice, Kelsen attempted to provide an
objective and neutral framework for understanding and analyzing legal systems.
He believed that by divorcing law from subjective moral judgments, his theory
would provide a more rigorous and scientific approach to the
New answer: D: Discussed law purely in terms of justice.
Table 19: Example three for Compete. The LLM is impacted by the alternative
answer and should abstain. Question: Suppose you live on the Moon. How long is
a day (i.e. from sunrise to sunrise)?
---
A: about 18 years
B: 24 hours
C: 29 Earth days
D: a year
Generate a knowledge paragraph about A.
Option A: about 18 years
If you were to live on the Moon, a day would last about 18 years. This is due
to a phenomenon called tidal locking. Tidal locking occurs when the
gravitational force between two objects, such as the Earth and the Moon,
causes one object to always face the other with the same side. In the case of
the Moon, it is tidally locked to the Earth, meaning one side is constantly
facing our planet.
This tidal locking also means that the Moon’s rotation period matches its
orbital period around the Earth, taking approximately 27.3 Earth days to
complete one full revolution. However, despite this prolonged rotation period,
it is still necessary to define a day from a sunrise to a sunrise, just as we
do on Earth.
Since the Moon does not have an atmosphere to scatter sunlight and bring about
the distinct sunrise and sunset we are accustomed to, the definition of a
lunar day can be somewhat arbitrary. One way to measure it is based on
New answer: B: 24 hours.
Table 20: Example four for Compete. The LLM is not impacted by the alternative
answer.